2024-03-28T22:10:38.388 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-28T22:10:38.389 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_teuthology@teuthology --name teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627121 --verbose --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} -- /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627121/orig.config.yaml 2024-03-28T22:10:38.448 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T22:10:38.539 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627121 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7627121' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request sha1: 323d23dc228bbcb7f16f188fdc9c0dcb801844be subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: 323d23dc228bbcb7f16f188fdc9c0dcb801844be kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - nowsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 55f5138942aa4866e49329c9103468889cec3f73 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 7214 sha1: 323d23dc228bbcb7f16f188fdc9c0dcb801844be sleep_before_teardown: 0 subset: 39/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_55f5138942aa4866e49329c9103468889cec3f73/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 55f5138942aa4866e49329c9103468889cec3f73 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi043.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBYr1Ssw9eRynMasJZ901KkWuiYsR9v3BvnaY7jH8jUWsowQft6ZQwe+wl3S5s9+68Pq4XNaDYR9725IG+q+Bo8= smithi082.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAbC5P4o+dEy/zQHqjuWbzSn6SgvjeI7xm29CErx33cJhzMgXCvGcQOvoOyjUq4GLYIGu2mcsT4ob00QaD8nnpI= smithi172.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIsh76rEIablDRipA8ysrOA1thB3z3+kbKqxrMns7d3K5WxYO5qyNVd6cbx0W3sOD7LwMS+s3QFlx5yayY4CYGk= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-27_21:24:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-28T22:10:38.540 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_55f5138942aa4866e49329c9103468889cec3f73/qa; will attempt to use it 2024-03-28T22:10:38.540 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_55f5138942aa4866e49329c9103468889cec3f73/qa/tasks 2024-03-28T22:10:38.540 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-28T22:10:38.542 INFO:teuthology.task.internal:Checking packages... 2024-03-28T22:10:38.565 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '323d23dc228bbcb7f16f188fdc9c0dcb801844be' 2024-03-28T22:10:38.565 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-28T22:10:38.565 INFO:teuthology.packaging:ref: None 2024-03-28T22:10:38.565 INFO:teuthology.packaging:tag: None 2024-03-28T22:10:38.565 INFO:teuthology.packaging:branch: squid 2024-03-28T22:10:38.565 INFO:teuthology.packaging:sha1: 323d23dc228bbcb7f16f188fdc9c0dcb801844be 2024-03-28T22:10:38.565 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-28T22:10:38.819 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1959.g8da23624 2024-03-28T22:10:38.821 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-28T22:10:38.827 INFO:teuthology.task.internal:no buildpackages task found 2024-03-28T22:10:38.827 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-28T22:10:38.843 INFO:teuthology.task.internal:Saving configuration 2024-03-28T22:10:38.865 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-28T22:10:38.873 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T22:10:38.899 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi043.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627121', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 22:05:11.800096', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBYr1Ssw9eRynMasJZ901KkWuiYsR9v3BvnaY7jH8jUWsowQft6ZQwe+wl3S5s9+68Pq4XNaDYR9725IG+q+Bo8='} 2024-03-28T22:10:38.920 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi082.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627121', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 22:05:11.803401', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAbC5P4o+dEy/zQHqjuWbzSn6SgvjeI7xm29CErx33cJhzMgXCvGcQOvoOyjUq4GLYIGu2mcsT4ob00QaD8nnpI='} 2024-03-28T22:10:38.943 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627121', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 22:05:11.802100', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIsh76rEIablDRipA8ysrOA1thB3z3+kbKqxrMns7d3K5WxYO5qyNVd6cbx0W3sOD7LwMS+s3QFlx5yayY4CYGk='} 2024-03-28T22:10:38.943 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-28T22:10:38.951 INFO:teuthology.task.internal:roles: ubuntu@smithi043.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-28T22:10:38.951 INFO:teuthology.task.internal:roles: ubuntu@smithi082.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-28T22:10:38.951 INFO:teuthology.task.internal:roles: ubuntu@smithi172.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-28T22:10:38.951 INFO:teuthology.run_tasks:Running task console_log... 2024-03-28T22:10:39.062 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4e98a8d160>, signals=[15]) 2024-03-28T22:10:39.062 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-28T22:10:39.071 INFO:teuthology.task.internal:Opening connections... 2024-03-28T22:10:39.071 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi043.front.sepia.ceph.com 2024-03-28T22:10:39.073 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T22:10:39.220 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi082.front.sepia.ceph.com 2024-03-28T22:10:39.221 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T22:10:39.311 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi172.front.sepia.ceph.com 2024-03-28T22:10:39.312 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T22:10:39.400 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-28T22:10:39.405 DEBUG:teuthology.orchestra.run.smithi043:> uname -m 2024-03-28T22:10:39.424 INFO:teuthology.orchestra.run.smithi043.stdout:x86_64 2024-03-28T22:10:39.425 DEBUG:teuthology.orchestra.run.smithi043:> cat /etc/os-release 2024-03-28T22:10:39.482 INFO:teuthology.orchestra.run.smithi043.stdout:NAME="CentOS Stream" 2024-03-28T22:10:39.483 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION="9" 2024-03-28T22:10:39.483 INFO:teuthology.orchestra.run.smithi043.stdout:ID="centos" 2024-03-28T22:10:39.483 INFO:teuthology.orchestra.run.smithi043.stdout:ID_LIKE="rhel fedora" 2024-03-28T22:10:39.483 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION_ID="9" 2024-03-28T22:10:39.483 INFO:teuthology.orchestra.run.smithi043.stdout:PLATFORM_ID="platform:el9" 2024-03-28T22:10:39.483 INFO:teuthology.orchestra.run.smithi043.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T22:10:39.483 INFO:teuthology.orchestra.run.smithi043.stdout:ANSI_COLOR="0;31" 2024-03-28T22:10:39.483 INFO:teuthology.orchestra.run.smithi043.stdout:LOGO="fedora-logo-icon" 2024-03-28T22:10:39.483 INFO:teuthology.orchestra.run.smithi043.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T22:10:39.483 INFO:teuthology.orchestra.run.smithi043.stdout:HOME_URL="https://centos.org/" 2024-03-28T22:10:39.483 INFO:teuthology.orchestra.run.smithi043.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T22:10:39.483 INFO:teuthology.orchestra.run.smithi043.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T22:10:39.484 INFO:teuthology.orchestra.run.smithi043.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T22:10:39.484 INFO:teuthology.lock.ops:Updating smithi043.front.sepia.ceph.com on lock server 2024-03-28T22:10:39.507 DEBUG:teuthology.orchestra.run.smithi082:> uname -m 2024-03-28T22:10:39.525 INFO:teuthology.orchestra.run.smithi082.stdout:x86_64 2024-03-28T22:10:39.525 DEBUG:teuthology.orchestra.run.smithi082:> cat /etc/os-release 2024-03-28T22:10:39.582 INFO:teuthology.orchestra.run.smithi082.stdout:NAME="CentOS Stream" 2024-03-28T22:10:39.582 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION="9" 2024-03-28T22:10:39.582 INFO:teuthology.orchestra.run.smithi082.stdout:ID="centos" 2024-03-28T22:10:39.582 INFO:teuthology.orchestra.run.smithi082.stdout:ID_LIKE="rhel fedora" 2024-03-28T22:10:39.582 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION_ID="9" 2024-03-28T22:10:39.582 INFO:teuthology.orchestra.run.smithi082.stdout:PLATFORM_ID="platform:el9" 2024-03-28T22:10:39.583 INFO:teuthology.orchestra.run.smithi082.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T22:10:39.583 INFO:teuthology.orchestra.run.smithi082.stdout:ANSI_COLOR="0;31" 2024-03-28T22:10:39.583 INFO:teuthology.orchestra.run.smithi082.stdout:LOGO="fedora-logo-icon" 2024-03-28T22:10:39.583 INFO:teuthology.orchestra.run.smithi082.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T22:10:39.583 INFO:teuthology.orchestra.run.smithi082.stdout:HOME_URL="https://centos.org/" 2024-03-28T22:10:39.583 INFO:teuthology.orchestra.run.smithi082.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T22:10:39.583 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T22:10:39.583 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T22:10:39.583 INFO:teuthology.lock.ops:Updating smithi082.front.sepia.ceph.com on lock server 2024-03-28T22:10:39.607 DEBUG:teuthology.orchestra.run.smithi172:> uname -m 2024-03-28T22:10:39.624 INFO:teuthology.orchestra.run.smithi172.stdout:x86_64 2024-03-28T22:10:39.625 DEBUG:teuthology.orchestra.run.smithi172:> cat /etc/os-release 2024-03-28T22:10:39.681 INFO:teuthology.orchestra.run.smithi172.stdout:NAME="CentOS Stream" 2024-03-28T22:10:39.682 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION="9" 2024-03-28T22:10:39.682 INFO:teuthology.orchestra.run.smithi172.stdout:ID="centos" 2024-03-28T22:10:39.682 INFO:teuthology.orchestra.run.smithi172.stdout:ID_LIKE="rhel fedora" 2024-03-28T22:10:39.682 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_ID="9" 2024-03-28T22:10:39.682 INFO:teuthology.orchestra.run.smithi172.stdout:PLATFORM_ID="platform:el9" 2024-03-28T22:10:39.682 INFO:teuthology.orchestra.run.smithi172.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T22:10:39.682 INFO:teuthology.orchestra.run.smithi172.stdout:ANSI_COLOR="0;31" 2024-03-28T22:10:39.682 INFO:teuthology.orchestra.run.smithi172.stdout:LOGO="fedora-logo-icon" 2024-03-28T22:10:39.682 INFO:teuthology.orchestra.run.smithi172.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T22:10:39.682 INFO:teuthology.orchestra.run.smithi172.stdout:HOME_URL="https://centos.org/" 2024-03-28T22:10:39.683 INFO:teuthology.orchestra.run.smithi172.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T22:10:39.683 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T22:10:39.683 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T22:10:39.683 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server 2024-03-28T22:10:39.698 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-28T22:10:39.709 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-28T22:10:39.715 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-28T22:10:39.716 DEBUG:teuthology.orchestra.run.smithi043:> test '!' -e /home/ubuntu/cephtest 2024-03-28T22:10:39.718 DEBUG:teuthology.orchestra.run.smithi082:> test '!' -e /home/ubuntu/cephtest 2024-03-28T22:10:39.720 DEBUG:teuthology.orchestra.run.smithi172:> test '!' -e /home/ubuntu/cephtest 2024-03-28T22:10:39.737 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-28T22:10:39.744 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-28T22:10:39.745 DEBUG:teuthology.orchestra.run.smithi043:> test -z $(ls -A /var/lib/ceph) 2024-03-28T22:10:39.775 DEBUG:teuthology.orchestra.run.smithi082:> test -z $(ls -A /var/lib/ceph) 2024-03-28T22:10:39.777 DEBUG:teuthology.orchestra.run.smithi172:> test -z $(ls -A /var/lib/ceph) 2024-03-28T22:10:39.801 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-28T22:10:39.884 INFO:teuthology.run_tasks:Running task kernel... 2024-03-28T22:10:39.898 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-03-28T22:10:39.899 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-03-28T22:10:39.899 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-03-28T22:10:39.899 DEBUG:teuthology.orchestra.run.smithi043:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T22:10:39.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T22:10:39.917 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-28T22:10:40.117 DEBUG:teuthology.task.kernel:sha1 for client.0 is 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-28T22:10:40.118 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "1df0c7e8a2354c42d738e5454ba83ea7332deac2"... 2024-03-28T22:10:40.118 DEBUG:teuthology.orchestra.run.smithi043:> uname -r 2024-03-28T22:10:40.136 INFO:teuthology.orchestra.run.smithi043.stdout:5.14.0-361.el9.x86_64 2024-03-28T22:10:40.136 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-28T22:10:40.136 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-28T22:10:40.137 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum upgrade -y linux-firmware 2024-03-28T22:10:42.355 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-28T22:10:43.863 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS-9 - BaseOS 5.4 MB/s | 2.0 MB 00:00 2024-03-28T22:10:52.994 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - BaseOS 937 kB/s | 8.0 MB 00:08 2024-03-28T22:10:55.421 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-28T22:11:00.116 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - CRB 5.8 MB/s | 6.1 MB 00:01 2024-03-28T22:11:01.489 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - Extras packages 84 kB/s | 16 kB 00:00 2024-03-28T22:11:02.968 INFO:teuthology.orchestra.run.smithi043.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-28T22:11:08.132 INFO:teuthology.orchestra.run.smithi043.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-28T22:11:10.740 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-03-28T22:11:10.751 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-28T22:11:10.751 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-03-28T22:11:10.751 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-28T22:11:10.751 INFO:teuthology.orchestra.run.smithi043.stdout:Upgrading: 2024-03-28T22:11:10.751 INFO:teuthology.orchestra.run.smithi043.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-28T22:11:10.751 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:11:10.751 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-03-28T22:11:10.751 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-03-28T22:11:10.751 INFO:teuthology.orchestra.run.smithi043.stdout:Upgrade 1 Package 2024-03-28T22:11:10.751 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:11:10.752 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 375 M 2024-03-28T22:11:10.752 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-03-28T22:11:19.083 INFO:teuthology.orchestra.run.smithi043.stdout:linux-firmware-20240219-143.el9.noarch.rpm 45 MB/s | 375 MB 00:08 2024-03-28T22:11:19.084 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-03-28T22:11:19.084 INFO:teuthology.orchestra.run.smithi043.stdout:Total 45 MB/s | 375 MB 00:08 2024-03-28T22:11:19.084 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-03-28T22:11:19.169 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-03-28T22:11:19.169 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-03-28T22:11:21.693 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-03-28T22:11:21.693 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-03-28T22:11:28.151 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-03-28T22:11:28.447 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-28T22:11:28.544 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-28T22:11:29.942 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-28T22:11:29.942 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-28T22:11:30.357 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-28T22:11:30.357 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:11:30.357 INFO:teuthology.orchestra.run.smithi043.stdout:Upgraded: 2024-03-28T22:11:30.357 INFO:teuthology.orchestra.run.smithi043.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-28T22:11:30.357 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:11:30.357 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-03-28T22:11:30.653 INFO:teuthology.task.kernel:Downloading kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on client.0... 2024-03-28T22:11:30.654 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-28T22:11:30.799 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/ 2024-03-28T22:11:30.799 DEBUG:teuthology.orchestra.run.smithi043:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-28T22:11:30.801 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi043.front.sepia.ceph.com to complete... 2024-03-28T22:11:44.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28 22:11:44 URL:https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm [336317861/336317861] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-28T22:11:44.235 INFO:teuthology.orchestra.run.smithi043.stderr:FINISHED --2024-03-28 22:11:44-- 2024-03-28T22:11:44.235 INFO:teuthology.orchestra.run.smithi043.stderr:Total wall clock time: 13s 2024-03-28T22:11:44.235 INFO:teuthology.orchestra.run.smithi043.stderr:Downloaded: 1 files, 321M in 13s (24.6 MB/s) 2024-03-28T22:11:44.238 INFO:teuthology.task.kernel:Installing kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on client.0... 2024-03-28T22:11:44.238 DEBUG:teuthology.orchestra.run.smithi043:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-28T22:11:46.265 INFO:teuthology.orchestra.run.smithi043.stdout:Verifying... ######################################## 2024-03-28T22:11:46.331 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing... ######################################## 2024-03-28T22:11:46.415 INFO:teuthology.orchestra.run.smithi043.stdout:Updating / installing... 2024-03-28T22:11:54.270 INFO:teuthology.orchestra.run.smithi043.stdout:kernel-6.8.0_g1df0c7e8a235-1 ######################################## 2024-03-28T22:12:10.515 INFO:teuthology.orchestra.run.smithi043.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-28T22:12:33.732 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi043.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-28T22:12:33.733 DEBUG:teuthology.orchestra.run.smithi043:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-28T22:12:33.809 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235 2024-03-28T22:12:33.810 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/System.map 2024-03-28T22:12:33.810 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/config 2024-03-28T22:12:33.810 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel 2024-03-28T22:12:33.810 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch 2024-03-28T22:12:33.810 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86 2024-03-28T22:12:33.810 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto 2024-03-28T22:12:33.810 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-28T22:12:33.810 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-28T22:12:33.810 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-28T22:12:33.810 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-28T22:12:33.810 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-28T22:12:33.810 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-28T22:12:33.811 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-28T22:12:33.811 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-28T22:12:33.811 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-28T22:12:33.811 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-28T22:12:33.811 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-28T22:12:33.811 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-28T22:12:33.811 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-28T22:12:33.811 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-28T22:12:33.811 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-28T22:12:33.811 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-28T22:12:33.811 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-28T22:12:33.811 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-28T22:12:33.811 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-28T22:12:33.812 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-28T22:12:33.812 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel 2024-03-28T22:12:33.812 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu 2024-03-28T22:12:33.812 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce 2024-03-28T22:12:33.812 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-28T22:12:33.812 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm 2024-03-28T22:12:33.812 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-28T22:12:33.812 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-28T22:12:33.812 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm.ko 2024-03-28T22:12:33.812 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto 2024-03-28T22:12:33.812 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ansi_cprng.ko 2024-03-28T22:12:33.812 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/anubis.ko 2024-03-28T22:12:33.812 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/arc4.ko 2024-03-28T22:12:33.812 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx 2024-03-28T22:12:33.813 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_memcpy.ko 2024-03-28T22:12:33.813 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_pq.ko 2024-03-28T22:12:33.813 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-28T22:12:33.813 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_tx.ko 2024-03-28T22:12:33.813 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_xor.ko 2024-03-28T22:12:33.813 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/raid6test.ko 2024-03-28T22:12:33.813 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authenc.ko 2024-03-28T22:12:33.813 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authencesn.ko 2024-03-28T22:12:33.813 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blake2b_generic.ko 2024-03-28T22:12:33.813 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_common.ko 2024-03-28T22:12:33.813 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_generic.ko 2024-03-28T22:12:33.813 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/camellia_generic.ko 2024-03-28T22:12:33.813 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast5_generic.ko 2024-03-28T22:12:33.813 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast6_generic.ko 2024-03-28T22:12:33.814 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast_common.ko 2024-03-28T22:12:33.814 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ccm.ko 2024-03-28T22:12:33.814 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cmac.ko 2024-03-28T22:12:33.814 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crc32_generic.ko 2024-03-28T22:12:33.814 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_engine.ko 2024-03-28T22:12:33.814 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_user.ko 2024-03-28T22:12:33.814 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cts.ko 2024-03-28T22:12:33.814 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/des_generic.ko 2024-03-28T22:12:33.814 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/dh_generic.ko 2024-03-28T22:12:33.814 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/echainiv.ko 2024-03-28T22:12:33.815 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/essiv.ko 2024-03-28T22:12:33.815 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/fcrypt.ko 2024-03-28T22:12:33.815 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/khazad.ko 2024-03-28T22:12:33.815 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4.ko 2024-03-28T22:12:33.815 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4hc.ko 2024-03-28T22:12:33.815 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/md4.ko 2024-03-28T22:12:33.815 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/michael_mic.ko 2024-03-28T22:12:33.815 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcbc.ko 2024-03-28T22:12:33.815 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcrypt.ko 2024-03-28T22:12:33.815 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/rmd160.ko 2024-03-28T22:12:33.815 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/seed.ko 2024-03-28T22:12:33.815 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/serpent_generic.ko 2024-03-28T22:12:33.815 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tcrypt.ko 2024-03-28T22:12:33.815 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tea.ko 2024-03-28T22:12:33.816 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_common.ko 2024-03-28T22:12:33.816 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_generic.ko 2024-03-28T22:12:33.816 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/vmac.ko 2024-03-28T22:12:33.816 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/wp512.ko 2024-03-28T22:12:33.816 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xcbc.ko 2024-03-28T22:12:33.816 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xor.ko 2024-03-28T22:12:33.816 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xxhash_generic.ko 2024-03-28T22:12:33.816 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers 2024-03-28T22:12:33.816 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi 2024-03-28T22:12:33.816 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-28T22:12:33.816 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_pad.ko 2024-03-28T22:12:33.816 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/custom_method.ko 2024-03-28T22:12:33.816 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/ec_sys.ko 2024-03-28T22:12:33.816 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/platform_profile.ko 2024-03-28T22:12:33.816 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbs.ko 2024-03-28T22:12:33.817 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbshc.ko 2024-03-28T22:12:33.817 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/video.ko 2024-03-28T22:12:33.817 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata 2024-03-28T22:12:33.817 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/acard-ahci.ko 2024-03-28T22:12:33.817 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ahci_platform.ko 2024-03-28T22:12:33.817 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ata_generic.ko 2024-03-28T22:12:33.817 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/libahci_platform.ko 2024-03-28T22:12:33.817 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_acpi.ko 2024-03-28T22:12:33.817 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ali.ko 2024-03-28T22:12:33.817 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_amd.ko 2024-03-28T22:12:33.817 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_artop.ko 2024-03-28T22:12:33.817 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atiixp.ko 2024-03-28T22:12:33.817 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atp867x.ko 2024-03-28T22:12:33.817 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd640.ko 2024-03-28T22:12:33.817 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd64x.ko 2024-03-28T22:12:33.818 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cypress.ko 2024-03-28T22:12:33.818 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_efar.ko 2024-03-28T22:12:33.818 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt366.ko 2024-03-28T22:12:33.818 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt37x.ko 2024-03-28T22:12:33.818 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-28T22:12:33.818 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-28T22:12:33.818 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it8213.ko 2024-03-28T22:12:33.818 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it821x.ko 2024-03-28T22:12:33.818 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_jmicron.ko 2024-03-28T22:12:33.818 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_marvell.ko 2024-03-28T22:12:33.818 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_mpiix.ko 2024-03-28T22:12:33.818 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_netcell.ko 2024-03-28T22:12:33.818 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ninja32.ko 2024-03-28T22:12:33.818 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87410.ko 2024-03-28T22:12:33.819 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87415.ko 2024-03-28T22:12:33.819 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_oldpiix.ko 2024-03-28T22:12:33.819 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_opti.ko 2024-03-28T22:12:33.819 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_optidma.ko 2024-03-28T22:12:33.819 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pcmcia.ko 2024-03-28T22:12:33.819 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-28T22:12:33.819 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-28T22:12:33.819 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_piccolo.ko 2024-03-28T22:12:33.819 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_rdc.ko 2024-03-28T22:12:33.819 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sch.ko 2024-03-28T22:12:33.819 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_serverworks.ko 2024-03-28T22:12:33.819 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sil680.ko 2024-03-28T22:12:33.819 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sis.ko 2024-03-28T22:12:33.819 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sl82c105.ko 2024-03-28T22:12:33.820 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_triflex.ko 2024-03-28T22:12:33.820 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_via.ko 2024-03-28T22:12:33.820 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pdc_adma.ko 2024-03-28T22:12:33.820 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_inic162x.ko 2024-03-28T22:12:33.820 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_mv.ko 2024-03-28T22:12:33.820 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_nv.ko 2024-03-28T22:12:33.820 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_promise.ko 2024-03-28T22:12:33.820 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_qstor.ko 2024-03-28T22:12:33.820 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil.ko 2024-03-28T22:12:33.820 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil24.ko 2024-03-28T22:12:33.820 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sis.ko 2024-03-28T22:12:33.820 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_svw.ko 2024-03-28T22:12:33.820 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sx4.ko 2024-03-28T22:12:33.820 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_uli.ko 2024-03-28T22:12:33.820 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_via.ko 2024-03-28T22:12:33.821 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_vsc.ko 2024-03-28T22:12:33.821 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay 2024-03-28T22:12:33.821 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay/ks0108.ko 2024-03-28T22:12:33.821 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base 2024-03-28T22:12:33.821 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap 2024-03-28T22:12:33.821 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-28T22:12:33.821 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-28T22:12:33.821 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma 2024-03-28T22:12:33.821 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma/bcma.ko 2024-03-28T22:12:33.821 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block 2024-03-28T22:12:33.821 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe 2024-03-28T22:12:33.821 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe/aoe.ko 2024-03-28T22:12:33.821 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/brd.ko 2024-03-28T22:12:33.821 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd 2024-03-28T22:12:33.821 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd/drbd.ko 2024-03-28T22:12:33.822 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/floppy.ko 2024-03-28T22:12:33.822 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/loop.ko 2024-03-28T22:12:33.822 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx 2024-03-28T22:12:33.822 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-28T22:12:33.822 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/nbd.ko 2024-03-28T22:12:33.822 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk 2024-03-28T22:12:33.822 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk/null_blk.ko 2024-03-28T22:12:33.822 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/pktcdvd.ko 2024-03-28T22:12:33.822 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/rbd.ko 2024-03-28T22:12:33.822 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/virtio_blk.ko 2024-03-28T22:12:33.822 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback 2024-03-28T22:12:33.822 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-28T22:12:33.822 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkfront.ko 2024-03-28T22:12:33.822 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram 2024-03-28T22:12:33.822 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram/zram.ko 2024-03-28T22:12:33.823 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char 2024-03-28T22:12:33.823 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hangcheck-timer.ko 2024-03-28T22:12:33.823 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random 2024-03-28T22:12:33.823 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-28T22:12:33.823 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-28T22:12:33.823 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-28T22:12:33.823 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/via-rng.ko 2024-03-28T22:12:33.823 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-28T22:12:33.823 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi 2024-03-28T22:12:33.823 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-28T22:12:33.825 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-28T22:12:33.825 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-28T22:12:33.825 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-28T22:12:33.825 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-28T22:12:33.825 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-28T22:12:33.825 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/lp.ko 2024-03-28T22:12:33.826 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave 2024-03-28T22:12:33.826 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave/mwave.ko 2024-03-28T22:12:33.826 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ppdev.ko 2024-03-28T22:12:33.826 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tlclk.ko 2024-03-28T22:12:33.826 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm 2024-03-28T22:12:33.826 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm.ko 2024-03-28T22:12:33.826 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-28T22:12:33.826 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-28T22:12:33.826 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-28T22:12:33.826 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-28T22:12:33.826 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-28T22:12:33.826 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/uv_mmtimer.ko 2024-03-28T22:12:33.826 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/virtio_console.ko 2024-03-28T22:12:33.826 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq 2024-03-28T22:12:33.827 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-28T22:12:33.827 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-28T22:12:33.827 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-28T22:12:33.827 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-28T22:12:33.827 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-28T22:12:33.827 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-28T22:12:33.827 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto 2024-03-28T22:12:33.827 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp 2024-03-28T22:12:33.827 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-28T22:12:33.827 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp.ko 2024-03-28T22:12:33.827 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel 2024-03-28T22:12:33.827 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat 2024-03-28T22:12:33.827 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common 2024-03-28T22:12:33.827 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-28T22:12:33.827 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-28T22:12:33.828 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-28T22:12:33.828 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-aes.ko 2024-03-28T22:12:33.828 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-sha.ko 2024-03-28T22:12:33.828 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio 2024-03-28T22:12:33.828 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-28T22:12:33.828 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca 2024-03-28T22:12:33.828 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca/dca.ko 2024-03-28T22:12:33.828 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq 2024-03-28T22:12:33.828 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-28T22:12:33.828 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma 2024-03-28T22:12:33.828 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw 2024-03-28T22:12:33.828 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-28T22:12:33.828 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat 2024-03-28T22:12:33.828 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-28T22:12:33.828 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac 2024-03-28T22:12:33.828 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/amd64_edac.ko 2024-03-28T22:12:33.829 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/e752x_edac.ko 2024-03-28T22:12:33.829 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/edac_mce_amd.ko 2024-03-28T22:12:33.829 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3000_edac.ko 2024-03-28T22:12:33.829 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3200_edac.ko 2024-03-28T22:12:33.829 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5100_edac.ko 2024-03-28T22:12:33.829 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5400_edac.ko 2024-03-28T22:12:33.829 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7300_edac.ko 2024-03-28T22:12:33.829 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7core_edac.ko 2024-03-28T22:12:33.829 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i82975x_edac.ko 2024-03-28T22:12:33.829 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/ie31200_edac.ko 2024-03-28T22:12:33.829 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/sb_edac.ko 2024-03-28T22:12:33.829 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/x38_edac.ko 2024-03-28T22:12:33.829 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware 2024-03-28T22:12:33.829 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/edd.ko 2024-03-28T22:12:33.830 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-28T22:12:33.830 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio 2024-03-28T22:12:33.830 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-ich.ko 2024-03-28T22:12:33.830 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-28T22:12:33.830 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid 2024-03-28T22:12:33.830 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-appleir.ko 2024-03-28T22:12:33.830 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-aureal.ko 2024-03-28T22:12:33.830 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-axff.ko 2024-03-28T22:12:33.830 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-dr.ko 2024-03-28T22:12:33.830 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elecom.ko 2024-03-28T22:12:33.830 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elo.ko 2024-03-28T22:12:33.830 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-emsff.ko 2024-03-28T22:12:33.830 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gaff.ko 2024-03-28T22:12:33.830 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gt683r.ko 2024-03-28T22:12:33.831 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gyration.ko 2024-03-28T22:12:33.831 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-28T22:12:33.831 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-28T22:12:33.831 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtekff.ko 2024-03-28T22:12:33.831 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-hyperv.ko 2024-03-28T22:12:33.831 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-icade.ko 2024-03-28T22:12:33.831 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-keytouch.ko 2024-03-28T22:12:33.831 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-kye.ko 2024-03-28T22:12:33.831 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lcpower.ko 2024-03-28T22:12:33.831 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-led.ko 2024-03-28T22:12:33.831 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lenovo.ko 2024-03-28T22:12:33.831 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-28T22:12:33.831 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-28T22:12:33.831 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-multitouch.ko 2024-03-28T22:12:33.831 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-ortek.ko 2024-03-28T22:12:33.831 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-penmount.ko 2024-03-28T22:12:33.832 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-petalynx.ko 2024-03-28T22:12:33.832 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-picolcd.ko 2024-03-28T22:12:33.832 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-pl.ko 2024-03-28T22:12:33.832 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-plantronics.ko 2024-03-28T22:12:33.832 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-primax.ko 2024-03-28T22:12:33.832 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-rmi.ko 2024-03-28T22:12:33.832 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-28T22:12:33.832 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-common.ko 2024-03-28T22:12:33.832 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-28T22:12:33.832 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-28T22:12:33.832 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-28T22:12:33.832 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-28T22:12:33.832 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-28T22:12:33.832 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-28T22:12:33.833 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-28T22:12:33.833 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-28T22:12:33.833 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-28T22:12:33.833 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat.ko 2024-03-28T22:12:33.833 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-saitek.ko 2024-03-28T22:12:33.833 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-samsung.ko 2024-03-28T22:12:33.833 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-28T22:12:33.833 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sjoy.ko 2024-03-28T22:12:33.833 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sony.ko 2024-03-28T22:12:33.833 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-speedlink.ko 2024-03-28T22:12:33.833 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-steelseries.ko 2024-03-28T22:12:33.833 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sunplus.ko 2024-03-28T22:12:33.833 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-28T22:12:33.833 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tivo.ko 2024-03-28T22:12:33.833 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tmff.ko 2024-03-28T22:12:33.834 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-topseed.ko 2024-03-28T22:12:33.834 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-twinhan.ko 2024-03-28T22:12:33.834 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-uclogic.ko 2024-03-28T22:12:33.834 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-waltop.ko 2024-03-28T22:12:33.834 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-wiimote.ko 2024-03-28T22:12:33.834 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-xinmo.ko 2024-03-28T22:12:33.834 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zpff.ko 2024-03-28T22:12:33.834 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zydacron.ko 2024-03-28T22:12:33.834 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/uhid.ko 2024-03-28T22:12:33.834 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/wacom.ko 2024-03-28T22:12:33.834 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv 2024-03-28T22:12:33.834 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_balloon.ko 2024-03-28T22:12:33.834 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_utils.ko 2024-03-28T22:12:33.834 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_vmbus.ko 2024-03-28T22:12:33.835 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon 2024-03-28T22:12:33.835 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru.ko 2024-03-28T22:12:33.835 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru3.ko 2024-03-28T22:12:33.835 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-28T22:12:33.835 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7414.ko 2024-03-28T22:12:33.835 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7418.ko 2024-03-28T22:12:33.835 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adc128d818.ko 2024-03-28T22:12:33.835 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1025.ko 2024-03-28T22:12:33.835 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1026.ko 2024-03-28T22:12:33.835 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1029.ko 2024-03-28T22:12:33.835 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1031.ko 2024-03-28T22:12:33.835 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm9240.ko 2024-03-28T22:12:33.835 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ads7828.ko 2024-03-28T22:12:33.835 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7410.ko 2024-03-28T22:12:33.835 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7411.ko 2024-03-28T22:12:33.835 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7462.ko 2024-03-28T22:12:33.836 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7470.ko 2024-03-28T22:12:33.836 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7475.ko 2024-03-28T22:12:33.836 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7x10.ko 2024-03-28T22:12:33.836 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/amc6821.ko 2024-03-28T22:12:33.836 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/applesmc.ko 2024-03-28T22:12:33.836 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asb100.ko 2024-03-28T22:12:33.836 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asc7621.ko 2024-03-28T22:12:33.836 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-28T22:12:33.836 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/atxp1.ko 2024-03-28T22:12:33.836 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/coretemp.ko 2024-03-28T22:12:33.836 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-28T22:12:33.836 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dme1737.ko 2024-03-28T22:12:33.836 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds1621.ko 2024-03-28T22:12:33.836 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds620.ko 2024-03-28T22:12:33.837 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc1403.ko 2024-03-28T22:12:33.837 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc6w201.ko 2024-03-28T22:12:33.837 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71805f.ko 2024-03-28T22:12:33.837 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71882fg.ko 2024-03-28T22:12:33.837 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f75375s.ko 2024-03-28T22:12:33.837 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fam15h_power.ko 2024-03-28T22:12:33.837 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fschmd.ko 2024-03-28T22:12:33.837 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g760a.ko 2024-03-28T22:12:33.837 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g762.ko 2024-03-28T22:12:33.837 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl518sm.ko 2024-03-28T22:12:33.837 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl520sm.ko 2024-03-28T22:12:33.837 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-28T22:12:33.837 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5500_temp.ko 2024-03-28T22:12:33.837 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5k_amb.ko 2024-03-28T22:12:33.837 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmaem.ko 2024-03-28T22:12:33.837 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmpex.ko 2024-03-28T22:12:33.838 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina209.ko 2024-03-28T22:12:33.838 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina2xx.ko 2024-03-28T22:12:33.838 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/it87.ko 2024-03-28T22:12:33.838 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k10temp.ko 2024-03-28T22:12:33.838 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k8temp.ko 2024-03-28T22:12:33.838 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lineage-pem.ko 2024-03-28T22:12:33.838 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm63.ko 2024-03-28T22:12:33.838 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm73.ko 2024-03-28T22:12:33.838 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm75.ko 2024-03-28T22:12:33.838 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm77.ko 2024-03-28T22:12:33.838 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm78.ko 2024-03-28T22:12:33.838 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm80.ko 2024-03-28T22:12:33.838 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm83.ko 2024-03-28T22:12:33.838 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm85.ko 2024-03-28T22:12:33.839 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm87.ko 2024-03-28T22:12:33.839 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm90.ko 2024-03-28T22:12:33.839 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm92.ko 2024-03-28T22:12:33.839 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm93.ko 2024-03-28T22:12:33.839 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95234.ko 2024-03-28T22:12:33.839 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95241.ko 2024-03-28T22:12:33.839 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95245.ko 2024-03-28T22:12:33.839 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc2945.ko 2024-03-28T22:12:33.839 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4151.ko 2024-03-28T22:12:33.839 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4215.ko 2024-03-28T22:12:33.839 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4222.ko 2024-03-28T22:12:33.839 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4245.ko 2024-03-28T22:12:33.839 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4260.ko 2024-03-28T22:12:33.839 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4261.ko 2024-03-28T22:12:33.839 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max16065.ko 2024-03-28T22:12:33.840 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1619.ko 2024-03-28T22:12:33.840 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1668.ko 2024-03-28T22:12:33.840 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max197.ko 2024-03-28T22:12:33.840 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6639.ko 2024-03-28T22:12:33.840 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6650.ko 2024-03-28T22:12:33.840 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6697.ko 2024-03-28T22:12:33.840 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/mcp3021.ko 2024-03-28T22:12:33.840 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6683.ko 2024-03-28T22:12:33.840 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775-core.ko 2024-03-28T22:12:33.840 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775.ko 2024-03-28T22:12:33.840 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct7802.ko 2024-03-28T22:12:33.840 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87360.ko 2024-03-28T22:12:33.840 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87427.ko 2024-03-28T22:12:33.840 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pcf8591.ko 2024-03-28T22:12:33.840 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus 2024-03-28T22:12:33.841 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-28T22:12:33.841 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-28T22:12:33.841 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-28T22:12:33.841 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-28T22:12:33.841 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-28T22:12:33.841 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-28T22:12:33.841 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-28T22:12:33.841 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-28T22:12:33.841 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-28T22:12:33.841 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-28T22:12:33.841 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-28T22:12:33.841 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-28T22:12:33.841 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/powr1220.ko 2024-03-28T22:12:33.841 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5627.ko 2024-03-28T22:12:33.841 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5636.ko 2024-03-28T22:12:33.842 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-28T22:12:33.842 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht15.ko 2024-03-28T22:12:33.842 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht21.ko 2024-03-28T22:12:33.842 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/shtc1.ko 2024-03-28T22:12:33.842 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sis5595.ko 2024-03-28T22:12:33.842 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47b397.ko 2024-03-28T22:12:33.842 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m1.ko 2024-03-28T22:12:33.842 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m192.ko 2024-03-28T22:12:33.842 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/thmc50.ko 2024-03-28T22:12:33.842 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp102.ko 2024-03-28T22:12:33.842 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp103.ko 2024-03-28T22:12:33.842 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp401.ko 2024-03-28T22:12:33.842 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp421.ko 2024-03-28T22:12:33.842 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via-cputemp.ko 2024-03-28T22:12:33.843 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via686a.ko 2024-03-28T22:12:33.843 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt1211.ko 2024-03-28T22:12:33.843 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt8231.ko 2024-03-28T22:12:33.843 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627ehf.ko 2024-03-28T22:12:33.843 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627hf.ko 2024-03-28T22:12:33.843 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83781d.ko 2024-03-28T22:12:33.843 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83791d.ko 2024-03-28T22:12:33.843 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83792d.ko 2024-03-28T22:12:33.843 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83793.ko 2024-03-28T22:12:33.843 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83795.ko 2024-03-28T22:12:33.843 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l785ts.ko 2024-03-28T22:12:33.843 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l786ng.ko 2024-03-28T22:12:33.843 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c 2024-03-28T22:12:33.843 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos 2024-03-28T22:12:33.844 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-28T22:12:33.844 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-28T22:12:33.844 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses 2024-03-28T22:12:33.844 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-28T22:12:33.844 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-28T22:12:33.844 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-28T22:12:33.844 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-28T22:12:33.844 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-28T22:12:33.844 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-28T22:12:33.844 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-28T22:12:33.844 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-28T22:12:33.844 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-28T22:12:33.844 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-28T22:12:33.844 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-28T22:12:33.844 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-28T22:12:33.845 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-28T22:12:33.845 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-28T22:12:33.845 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-28T22:12:33.845 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-28T22:12:33.845 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-28T22:12:33.845 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-28T22:12:33.845 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-28T22:12:33.845 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-28T22:12:33.845 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-28T22:12:33.845 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-28T22:12:33.845 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-28T22:12:33.845 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-dev.ko 2024-03-28T22:12:33.845 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-mux.ko 2024-03-28T22:12:33.845 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-28T22:12:33.845 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-smbus.ko 2024-03-28T22:12:33.846 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-stub.ko 2024-03-28T22:12:33.846 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband 2024-03-28T22:12:33.846 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core 2024-03-28T22:12:33.846 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-28T22:12:33.846 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_core.ko 2024-03-28T22:12:33.846 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-28T22:12:33.846 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-28T22:12:33.846 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-28T22:12:33.846 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-28T22:12:33.846 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-28T22:12:33.846 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw 2024-03-28T22:12:33.846 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4 2024-03-28T22:12:33.846 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-28T22:12:33.846 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4 2024-03-28T22:12:33.846 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-28T22:12:33.847 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5 2024-03-28T22:12:33.847 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-28T22:12:33.847 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca 2024-03-28T22:12:33.847 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-28T22:12:33.847 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp 2024-03-28T22:12:33.847 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib 2024-03-28T22:12:33.847 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-28T22:12:33.847 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser 2024-03-28T22:12:33.847 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-28T22:12:33.847 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert 2024-03-28T22:12:33.847 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-28T22:12:33.847 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp 2024-03-28T22:12:33.847 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-28T22:12:33.847 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt 2024-03-28T22:12:33.847 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-28T22:12:33.848 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input 2024-03-28T22:12:33.848 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport 2024-03-28T22:12:33.848 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-28T22:12:33.848 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-28T22:12:33.848 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/gameport.ko 2024-03-28T22:12:33.848 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/joydev.ko 2024-03-28T22:12:33.848 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard 2024-03-28T22:12:33.848 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-28T22:12:33.848 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse 2024-03-28T22:12:33.848 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/appletouch.ko 2024-03-28T22:12:33.848 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/bcm5974.ko 2024-03-28T22:12:33.848 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/cyapatp.ko 2024-03-28T22:12:33.848 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-28T22:12:33.848 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/sermouse.ko 2024-03-28T22:12:33.849 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-28T22:12:33.849 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-28T22:12:33.849 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-28T22:12:33.849 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4 2024-03-28T22:12:33.849 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-28T22:12:33.849 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio 2024-03-28T22:12:33.849 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/altera_ps2.ko 2024-03-28T22:12:33.849 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/arc_ps2.ko 2024-03-28T22:12:33.849 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-28T22:12:33.849 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/serio_raw.ko 2024-03-28T22:12:33.849 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/sparse-keymap.ko 2024-03-28T22:12:33.849 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds 2024-03-28T22:12:33.849 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-blinkm.ko 2024-03-28T22:12:33.849 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm3530.ko 2024-03-28T22:12:33.849 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lp3944.ko 2024-03-28T22:12:33.850 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-ss4200.ko 2024-03-28T22:12:33.850 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger 2024-03-28T22:12:33.850 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-28T22:12:33.850 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-28T22:12:33.850 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-28T22:12:33.850 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-28T22:12:33.850 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-28T22:12:33.850 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-28T22:12:33.850 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-28T22:12:33.850 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-28T22:12:33.850 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-28T22:12:33.850 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md 2024-03-28T22:12:33.850 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache 2024-03-28T22:12:33.850 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache/bcache.ko 2024-03-28T22:12:33.850 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-bio-prison.ko 2024-03-28T22:12:33.851 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache-smq.ko 2024-03-28T22:12:33.851 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache.ko 2024-03-28T22:12:33.851 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-crypt.ko 2024-03-28T22:12:33.851 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-delay.ko 2024-03-28T22:12:33.851 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-flakey.ko 2024-03-28T22:12:33.851 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-log-userspace.ko 2024-03-28T22:12:33.851 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-multipath.ko 2024-03-28T22:12:33.851 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-queue-length.ko 2024-03-28T22:12:33.851 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-raid.ko 2024-03-28T22:12:33.851 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-round-robin.ko 2024-03-28T22:12:33.851 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-service-time.ko 2024-03-28T22:12:33.851 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-switch.ko 2024-03-28T22:12:33.851 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-thin-pool.ko 2024-03-28T22:12:33.851 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-verity.ko 2024-03-28T22:12:33.852 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data 2024-03-28T22:12:33.852 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-28T22:12:33.852 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid0.ko 2024-03-28T22:12:33.852 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid1.ko 2024-03-28T22:12:33.852 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid10.ko 2024-03-28T22:12:33.852 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid456.ko 2024-03-28T22:12:33.852 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message 2024-03-28T22:12:33.852 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion 2024-03-28T22:12:33.852 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptbase.ko 2024-03-28T22:12:33.852 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptctl.ko 2024-03-28T22:12:33.852 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptfc.ko 2024-03-28T22:12:33.852 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptlan.ko 2024-03-28T22:12:33.852 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptsas.ko 2024-03-28T22:12:33.852 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptscsih.ko 2024-03-28T22:12:33.852 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptspi.ko 2024-03-28T22:12:33.853 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd 2024-03-28T22:12:33.853 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_ich.ko 2024-03-28T22:12:33.853 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_sch.ko 2024-03-28T22:12:33.853 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/mfd-core.ko 2024-03-28T22:12:33.853 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/sm501.ko 2024-03-28T22:12:33.853 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/viperboard.ko 2024-03-28T22:12:33.853 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/vx855.ko 2024-03-28T22:12:33.853 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/wl1273-core.ko 2024-03-28T22:12:33.853 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc 2024-03-28T22:12:33.853 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl 2024-03-28T22:12:33.853 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-28T22:12:33.853 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds9802als.ko 2024-03-28T22:12:33.853 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds990x.ko 2024-03-28T22:12:33.853 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/bh1770glc.ko 2024-03-28T22:12:33.853 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710 2024-03-28T22:12:33.853 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710/cb710.ko 2024-03-28T22:12:33.854 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom 2024-03-28T22:12:33.854 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/at24.ko 2024-03-28T22:12:33.854 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-28T22:12:33.854 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/max6875.ko 2024-03-28T22:12:33.854 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/enclosure.ko 2024-03-28T22:12:33.854 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/hpilo.ko 2024-03-28T22:12:33.854 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29003.ko 2024-03-28T22:12:33.854 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29020.ko 2024-03-28T22:12:33.854 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d 2024-03-28T22:12:33.854 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-28T22:12:33.854 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-28T22:12:33.854 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei 2024-03-28T22:12:33.854 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-me.ko 2024-03-28T22:12:33.855 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-txe.ko 2024-03-28T22:12:33.855 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei.ko 2024-03-28T22:12:33.855 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru 2024-03-28T22:12:33.855 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-28T22:12:33.855 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp 2024-03-28T22:12:33.855 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-28T22:12:33.855 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-28T22:12:33.855 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-28T22:12:33.855 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_7xx1.ko 2024-03-28T22:12:33.855 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_core.ko 2024-03-28T22:12:33.855 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tsl2550.ko 2024-03-28T22:12:33.855 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_balloon.ko 2024-03-28T22:12:33.855 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci 2024-03-28T22:12:33.855 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-28T22:12:33.855 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net 2024-03-28T22:12:33.856 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding 2024-03-28T22:12:33.856 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding/bonding.ko 2024-03-28T22:12:33.856 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa 2024-03-28T22:12:33.856 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53 2024-03-28T22:12:33.856 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-28T22:12:33.856 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-28T22:12:33.856 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-28T22:12:33.856 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx 2024-03-28T22:12:33.856 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-28T22:12:33.856 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dummy.ko 2024-03-28T22:12:33.856 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/eql.ko 2024-03-28T22:12:33.856 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet 2024-03-28T22:12:33.856 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com 2024-03-28T22:12:33.856 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-28T22:12:33.856 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-28T22:12:33.857 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-28T22:12:33.857 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-28T22:12:33.857 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390 2024-03-28T22:12:33.857 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-28T22:12:33.857 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-28T22:12:33.857 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-28T22:12:33.857 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-28T22:12:33.857 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec 2024-03-28T22:12:33.857 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-28T22:12:33.857 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere 2024-03-28T22:12:33.857 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-28T22:12:33.857 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon 2024-03-28T22:12:33.857 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-28T22:12:33.857 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera 2024-03-28T22:12:33.857 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-28T22:12:33.858 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd 2024-03-28T22:12:33.858 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-28T22:12:33.858 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-28T22:12:33.858 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-28T22:12:33.858 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros 2024-03-28T22:12:33.858 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx 2024-03-28T22:12:33.858 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-28T22:12:33.858 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-28T22:12:33.858 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-28T22:12:33.858 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-28T22:12:33.858 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-28T22:12:33.858 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx 2024-03-28T22:12:33.858 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-28T22:12:33.858 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-28T22:12:33.859 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom 2024-03-28T22:12:33.859 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-28T22:12:33.859 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-28T22:12:33.859 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-28T22:12:33.859 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-28T22:12:33.859 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-28T22:12:33.859 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet 2024-03-28T22:12:33.859 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-28T22:12:33.859 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-28T22:12:33.859 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade 2024-03-28T22:12:33.859 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna 2024-03-28T22:12:33.859 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-28T22:12:33.859 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio 2024-03-28T22:12:33.859 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-28T22:12:33.859 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-28T22:12:33.860 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-28T22:12:33.860 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-28T22:12:33.860 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-28T22:12:33.860 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-28T22:12:33.860 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-28T22:12:33.860 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-28T22:12:33.860 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-28T22:12:33.860 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-28T22:12:33.860 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco 2024-03-28T22:12:33.860 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic 2024-03-28T22:12:33.860 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-28T22:12:33.860 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec 2024-03-28T22:12:33.860 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip 2024-03-28T22:12:33.860 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-28T22:12:33.860 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-28T22:12:33.861 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-28T22:12:33.861 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-28T22:12:33.861 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-28T22:12:33.861 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-28T22:12:33.861 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink 2024-03-28T22:12:33.861 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-28T22:12:33.861 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-28T22:12:33.861 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dnet.ko 2024-03-28T22:12:33.861 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex 2024-03-28T22:12:33.861 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet 2024-03-28T22:12:33.861 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-28T22:12:33.861 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ethoc.ko 2024-03-28T22:12:33.861 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/fealnx.ko 2024-03-28T22:12:33.861 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel 2024-03-28T22:12:33.861 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-28T22:12:33.862 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000 2024-03-28T22:12:33.862 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-28T22:12:33.862 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e 2024-03-28T22:12:33.862 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-28T22:12:33.862 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k 2024-03-28T22:12:33.862 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-28T22:12:33.862 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e 2024-03-28T22:12:33.862 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-28T22:12:33.862 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf 2024-03-28T22:12:33.862 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-28T22:12:33.862 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb 2024-03-28T22:12:33.862 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-28T22:12:33.862 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf 2024-03-28T22:12:33.862 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-28T22:12:33.863 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-28T22:12:33.863 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-28T22:12:33.863 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-28T22:12:33.863 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-28T22:12:33.863 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/jme.ko 2024-03-28T22:12:33.863 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell 2024-03-28T22:12:33.863 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-28T22:12:33.863 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-28T22:12:33.863 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-28T22:12:33.863 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox 2024-03-28T22:12:33.863 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-28T22:12:33.863 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-28T22:12:33.863 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-28T22:12:33.863 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-28T22:12:33.863 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-28T22:12:33.864 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-28T22:12:33.864 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel 2024-03-28T22:12:33.864 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-28T22:12:33.864 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom 2024-03-28T22:12:33.864 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-28T22:12:33.864 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-28T22:12:33.864 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi 2024-03-28T22:12:33.864 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-28T22:12:33.864 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-28T22:12:33.864 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion 2024-03-28T22:12:33.864 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-28T22:12:33.864 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia 2024-03-28T22:12:33.864 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-28T22:12:33.864 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines 2024-03-28T22:12:33.865 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-28T22:12:33.865 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-28T22:12:33.865 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic 2024-03-28T22:12:33.865 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-28T22:12:33.865 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-28T22:12:33.865 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-28T22:12:33.865 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-28T22:12:33.865 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-28T22:12:33.865 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc 2024-03-28T22:12:33.865 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-28T22:12:33.865 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek 2024-03-28T22:12:33.866 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-28T22:12:33.866 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-28T22:12:33.866 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-28T22:12:33.866 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-28T22:12:33.866 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker 2024-03-28T22:12:33.866 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-28T22:12:33.866 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc 2024-03-28T22:12:33.866 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-28T22:12:33.866 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan 2024-03-28T22:12:33.866 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-28T22:12:33.866 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis 2024-03-28T22:12:33.866 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-28T22:12:33.866 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-28T22:12:33.866 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc 2024-03-28T22:12:33.867 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-28T22:12:33.867 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-28T22:12:33.867 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-28T22:12:33.867 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-28T22:12:33.867 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro 2024-03-28T22:12:33.867 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-28T22:12:33.867 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-28T22:12:33.867 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-28T22:12:33.867 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun 2024-03-28T22:12:33.867 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-28T22:12:33.867 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-28T22:12:33.867 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-28T22:12:33.867 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-28T22:12:33.867 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti 2024-03-28T22:12:33.867 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-28T22:12:33.868 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti 2024-03-28T22:12:33.868 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-28T22:12:33.868 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via 2024-03-28T22:12:33.868 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-28T22:12:33.868 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-28T22:12:33.868 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet 2024-03-28T22:12:33.868 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-28T22:12:33.868 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-28T22:12:33.868 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom 2024-03-28T22:12:33.868 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-28T22:12:33.868 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/geneve.ko 2024-03-28T22:12:33.868 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv 2024-03-28T22:12:33.868 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-28T22:12:33.868 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154 2024-03-28T22:12:33.868 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-28T22:12:33.869 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ifb.ko 2024-03-28T22:12:33.869 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan 2024-03-28T22:12:33.869 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-28T22:12:33.869 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvlan.ko 2024-03-28T22:12:33.869 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvtap.ko 2024-03-28T22:12:33.869 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio 2024-03-28T22:12:33.869 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio.ko 2024-03-28T22:12:33.869 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-28T22:12:33.869 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-28T22:12:33.869 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-28T22:12:33.869 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mii.ko 2024-03-28T22:12:33.869 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/net_failover.ko 2024-03-28T22:12:33.869 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/netconsole.ko 2024-03-28T22:12:33.869 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/nlmon.ko 2024-03-28T22:12:33.869 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs 2024-03-28T22:12:33.870 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-28T22:12:33.870 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-28T22:12:33.870 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy 2024-03-28T22:12:33.870 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/amd.ko 2024-03-28T22:12:33.870 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-28T22:12:33.870 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-28T22:12:33.870 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-28T22:12:33.870 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm87xx.ko 2024-03-28T22:12:33.870 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/broadcom.ko 2024-03-28T22:12:33.870 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/cicada.ko 2024-03-28T22:12:33.870 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/davicom.ko 2024-03-28T22:12:33.870 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/dp83640.ko 2024-03-28T22:12:33.870 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/et1011c.ko 2024-03-28T22:12:33.870 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/icplus.ko 2024-03-28T22:12:33.871 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/lxt.ko 2024-03-28T22:12:33.871 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/marvell.ko 2024-03-28T22:12:33.871 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/micrel.ko 2024-03-28T22:12:33.871 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/national.ko 2024-03-28T22:12:33.871 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/phylink.ko 2024-03-28T22:12:33.871 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/qsemi.ko 2024-03-28T22:12:33.871 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/realtek.ko 2024-03-28T22:12:33.871 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/smsc.ko 2024-03-28T22:12:33.871 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/ste10Xp.ko 2024-03-28T22:12:33.871 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/vitesse.ko 2024-03-28T22:12:33.871 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp 2024-03-28T22:12:33.871 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-28T22:12:33.871 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_async.ko 2024-03-28T22:12:33.871 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-28T22:12:33.871 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-28T22:12:33.872 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-28T22:12:33.872 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-28T22:12:33.872 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppoe.ko 2024-03-28T22:12:33.872 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppox.ko 2024-03-28T22:12:33.872 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pptp.ko 2024-03-28T22:12:33.872 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip 2024-03-28T22:12:33.872 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slhc.ko 2024-03-28T22:12:33.872 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slip.ko 2024-03-28T22:12:33.872 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/sungem_phy.ko 2024-03-28T22:12:33.872 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tap.ko 2024-03-28T22:12:33.872 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team 2024-03-28T22:12:33.872 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team.ko 2024-03-28T22:12:33.872 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-28T22:12:33.872 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-28T22:12:33.873 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-28T22:12:33.873 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_random.ko 2024-03-28T22:12:33.873 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-28T22:12:33.873 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tun.ko 2024-03-28T22:12:33.873 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/veth.ko 2024-03-28T22:12:33.873 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/virtio_net.ko 2024-03-28T22:12:33.873 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3 2024-03-28T22:12:33.873 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-28T22:12:33.873 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan 2024-03-28T22:12:33.873 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan/vxlan.ko 2024-03-28T22:12:33.873 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback 2024-03-28T22:12:33.873 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-28T22:12:33.873 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netfront.ko 2024-03-28T22:12:33.873 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb 2024-03-28T22:12:33.873 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb/ntb.ko 2024-03-28T22:12:33.873 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme 2024-03-28T22:12:33.874 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host 2024-03-28T22:12:33.874 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme-core.ko 2024-03-28T22:12:33.874 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme.ko 2024-03-28T22:12:33.874 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport 2024-03-28T22:12:33.874 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport.ko 2024-03-28T22:12:33.874 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_cs.ko 2024-03-28T22:12:33.874 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_pc.ko 2024-03-28T22:12:33.874 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_serial.ko 2024-03-28T22:12:33.874 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci 2024-03-28T22:12:33.874 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug 2024-03-28T22:12:33.874 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-28T22:12:33.874 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie 2024-03-28T22:12:33.874 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-28T22:12:33.874 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/xen-pcifront.ko 2024-03-28T22:12:33.874 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia 2024-03-28T22:12:33.875 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/i82092.ko 2024-03-28T22:12:33.875 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pd6729.ko 2024-03-28T22:12:33.875 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-28T22:12:33.875 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl 2024-03-28T22:12:33.875 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel 2024-03-28T22:12:33.875 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-28T22:12:33.875 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform 2024-03-28T22:12:33.875 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86 2024-03-28T22:12:33.875 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-28T22:12:33.875 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acerhdf.ko 2024-03-28T22:12:33.875 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-28T22:12:33.875 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-28T22:12:33.875 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-28T22:12:33.875 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-28T22:12:33.875 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-28T22:12:33.875 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-28T22:12:33.876 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-28T22:12:33.876 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-28T22:12:33.876 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-28T22:12:33.876 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/hdaps.ko 2024-03-28T22:12:33.876 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel 2024-03-28T22:12:33.876 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-28T22:12:33.876 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel_ips.ko 2024-03-28T22:12:33.876 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-28T22:12:33.876 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-28T22:12:33.876 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-28T22:12:33.876 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-28T22:12:33.876 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-28T22:12:33.876 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-28T22:12:33.876 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-28T22:12:33.876 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-28T22:12:33.876 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-28T22:12:33.877 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi.ko 2024-03-28T22:12:33.877 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap 2024-03-28T22:12:33.877 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-28T22:12:33.877 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-28T22:12:33.877 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps 2024-03-28T22:12:33.877 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients 2024-03-28T22:12:33.877 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-28T22:12:33.877 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-28T22:12:33.877 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps_parport.ko 2024-03-28T22:12:33.877 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/pps_core.ko 2024-03-28T22:12:33.877 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp 2024-03-28T22:12:33.877 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp.ko 2024-03-28T22:12:33.877 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp_kvm.ko 2024-03-28T22:12:33.877 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc 2024-03-28T22:12:33.877 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-28T22:12:33.877 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-28T22:12:33.878 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-28T22:12:33.878 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-28T22:12:33.878 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-28T22:12:33.878 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-28T22:12:33.878 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-28T22:12:33.878 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-28T22:12:33.878 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-28T22:12:33.878 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-28T22:12:33.878 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-em3027.ko 2024-03-28T22:12:33.878 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-28T22:12:33.878 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-28T22:12:33.878 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-28T22:12:33.878 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-28T22:12:33.878 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-28T22:12:33.879 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-28T22:12:33.879 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max6900.ko 2024-03-28T22:12:33.879 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-28T22:12:33.879 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-28T22:12:33.879 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-28T22:12:33.879 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-28T22:12:33.879 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-28T22:12:33.879 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-28T22:12:33.879 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-28T22:12:33.879 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-28T22:12:33.879 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-28T22:12:33.879 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-28T22:12:33.879 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-28T22:12:33.879 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-28T22:12:33.879 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-x1205.ko 2024-03-28T22:12:33.879 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi 2024-03-28T22:12:33.880 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-9xxx.ko 2024-03-28T22:12:33.880 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-sas.ko 2024-03-28T22:12:33.880 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-xxxx.ko 2024-03-28T22:12:33.880 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/BusLogic.ko 2024-03-28T22:12:33.880 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/a100u2w.ko 2024-03-28T22:12:33.880 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid 2024-03-28T22:12:33.880 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-28T22:12:33.880 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/advansys.ko 2024-03-28T22:12:33.880 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx 2024-03-28T22:12:33.880 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-28T22:12:33.880 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-28T22:12:33.880 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/am53c974.ko 2024-03-28T22:12:33.880 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr 2024-03-28T22:12:33.880 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-28T22:12:33.881 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/atp870u.ko 2024-03-28T22:12:33.881 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi 2024-03-28T22:12:33.881 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-28T22:12:33.881 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa 2024-03-28T22:12:33.881 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa/bfa.ko 2024-03-28T22:12:33.881 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc 2024-03-28T22:12:33.881 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-28T22:12:33.881 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i 2024-03-28T22:12:33.881 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-28T22:12:33.881 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ch.ko 2024-03-28T22:12:33.881 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor 2024-03-28T22:12:33.881 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-28T22:12:33.881 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi 2024-03-28T22:12:33.881 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-28T22:12:33.881 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-28T22:12:33.882 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-28T22:12:33.882 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-28T22:12:33.882 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-28T22:12:33.882 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dc395x.ko 2024-03-28T22:12:33.882 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler 2024-03-28T22:12:33.882 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-28T22:12:33.882 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-28T22:12:33.882 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-28T22:12:33.882 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-28T22:12:33.882 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dmx3191d.ko 2024-03-28T22:12:33.882 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r 2024-03-28T22:12:33.882 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-28T22:12:33.882 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esp_scsi.ko 2024-03-28T22:12:33.882 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe 2024-03-28T22:12:33.882 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-28T22:12:33.883 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-28T22:12:33.883 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic 2024-03-28T22:12:33.883 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic/fnic.ko 2024-03-28T22:12:33.883 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hpsa.ko 2024-03-28T22:12:33.883 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hptiop.ko 2024-03-28T22:12:33.883 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hv_storvsc.ko 2024-03-28T22:12:33.883 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/initio.ko 2024-03-28T22:12:33.883 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ipr.ko 2024-03-28T22:12:33.883 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ips.ko 2024-03-28T22:12:33.883 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci 2024-03-28T22:12:33.883 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci/isci.ko 2024-03-28T22:12:33.883 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-28T22:12:33.883 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-28T22:12:33.883 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc 2024-03-28T22:12:33.883 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc/libfc.ko 2024-03-28T22:12:33.883 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi.ko 2024-03-28T22:12:33.884 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-28T22:12:33.884 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas 2024-03-28T22:12:33.884 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas/libsas.ko 2024-03-28T22:12:33.884 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc 2024-03-28T22:12:33.884 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-28T22:12:33.884 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid 2024-03-28T22:12:33.884 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid.ko 2024-03-28T22:12:33.884 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-28T22:12:33.884 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-28T22:12:33.884 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-28T22:12:33.884 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas 2024-03-28T22:12:33.884 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-28T22:12:33.884 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas 2024-03-28T22:12:33.884 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-28T22:12:33.884 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvumi.ko 2024-03-28T22:12:33.884 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001 2024-03-28T22:12:33.885 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-28T22:12:33.885 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pmcraid.ko 2024-03-28T22:12:33.885 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla1280.ko 2024-03-28T22:12:33.885 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx 2024-03-28T22:12:33.885 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-28T22:12:33.885 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-28T22:12:33.885 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx 2024-03-28T22:12:33.885 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-28T22:12:33.885 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/raid_class.ko 2024-03-28T22:12:33.885 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_debug.ko 2024-03-28T22:12:33.885 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-28T22:12:33.885 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-28T22:12:33.885 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-28T22:12:33.885 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-28T22:12:33.885 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-28T22:12:33.886 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ses.ko 2024-03-28T22:12:33.886 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/st.ko 2024-03-28T22:12:33.886 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/stex.ko 2024-03-28T22:12:33.886 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2 2024-03-28T22:12:33.886 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-28T22:12:33.886 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/virtio_scsi.ko 2024-03-28T22:12:33.886 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-28T22:12:33.886 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/wd719x.ko 2024-03-28T22:12:33.886 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb 2024-03-28T22:12:33.886 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb/ssb.ko 2024-03-28T22:12:33.886 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target 2024-03-28T22:12:33.886 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi 2024-03-28T22:12:33.886 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-28T22:12:33.886 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback 2024-03-28T22:12:33.886 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-28T22:12:33.887 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_file.ko 2024-03-28T22:12:33.887 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_iblock.ko 2024-03-28T22:12:33.887 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_mod.ko 2024-03-28T22:12:33.887 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_pscsi.ko 2024-03-28T22:12:33.887 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_user.ko 2024-03-28T22:12:33.887 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc 2024-03-28T22:12:33.887 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-28T22:12:33.887 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal 2024-03-28T22:12:33.887 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel 2024-03-28T22:12:33.887 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal 2024-03-28T22:12:33.887 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-28T22:12:33.887 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-28T22:12:33.887 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-28T22:12:33.887 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-28T22:12:33.888 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-28T22:12:33.888 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-28T22:12:33.888 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-28T22:12:33.888 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-28T22:12:33.888 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-28T22:12:33.888 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-28T22:12:33.888 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-28T22:12:33.888 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-28T22:12:33.888 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-28T22:12:33.888 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-28T22:12:33.888 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-28T22:12:33.888 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-28T22:12:33.888 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-28T22:12:33.888 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-28T22:12:33.888 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty 2024-03-28T22:12:33.889 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless 2024-03-28T22:12:33.889 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-28T22:12:33.889 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_gsm.ko 2024-03-28T22:12:33.889 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_hdlc.ko 2024-03-28T22:12:33.889 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/nozomi.ko 2024-03-28T22:12:33.889 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial 2024-03-28T22:12:33.889 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250 2024-03-28T22:12:33.889 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-28T22:12:33.889 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/arc_uart.ko 2024-03-28T22:12:33.889 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm 2024-03-28T22:12:33.889 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-28T22:12:33.889 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs 2024-03-28T22:12:33.889 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core 2024-03-28T22:12:33.889 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-28T22:12:33.889 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host 2024-03-28T22:12:33.890 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-28T22:12:33.890 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio 2024-03-28T22:12:33.890 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio.ko 2024-03-28T22:12:33.890 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_aec.ko 2024-03-28T22:12:33.890 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_cif.ko 2024-03-28T22:12:33.890 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_pci_generic.ko 2024-03-28T22:12:33.890 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_sercos3.ko 2024-03-28T22:12:33.890 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb 2024-03-28T22:12:33.890 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class 2024-03-28T22:12:33.890 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-acm.ko 2024-03-28T22:12:33.890 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-28T22:12:33.890 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usblp.ko 2024-03-28T22:12:33.890 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usbtmc.ko 2024-03-28T22:12:33.890 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host 2024-03-28T22:12:33.891 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-28T22:12:33.891 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image 2024-03-28T22:12:33.891 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/mdc800.ko 2024-03-28T22:12:33.891 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/microtek.ko 2024-03-28T22:12:33.891 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc 2024-03-28T22:12:33.891 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/adutux.ko 2024-03-28T22:12:33.891 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/appledisplay.ko 2024-03-28T22:12:33.891 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi26.ko 2024-03-28T22:12:33.891 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi62.ko 2024-03-28T22:12:33.891 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ezusb.ko 2024-03-28T22:12:33.891 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/idmouse.ko 2024-03-28T22:12:33.891 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/iowarrior.ko 2024-03-28T22:12:33.891 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-28T22:12:33.891 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ldusb.ko 2024-03-28T22:12:33.891 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/legousbtower.ko 2024-03-28T22:12:33.891 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga 2024-03-28T22:12:33.892 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-28T22:12:33.892 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-28T22:12:33.892 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usb3503.ko 2024-03-28T22:12:33.892 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usblcd.ko 2024-03-28T22:12:33.892 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-28T22:12:33.892 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/uss720.ko 2024-03-28T22:12:33.892 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/yurex.ko 2024-03-28T22:12:33.892 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy 2024-03-28T22:12:33.892 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-generic.ko 2024-03-28T22:12:33.892 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial 2024-03-28T22:12:33.892 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/aircable.ko 2024-03-28T22:12:33.892 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ark3116.ko 2024-03-28T22:12:33.892 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-28T22:12:33.892 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ch341.ko 2024-03-28T22:12:33.892 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cp210x.ko 2024-03-28T22:12:33.893 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cyberjack.ko 2024-03-28T22:12:33.893 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-28T22:12:33.893 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-28T22:12:33.893 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/empeg.ko 2024-03-28T22:12:33.893 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-28T22:12:33.893 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-28T22:12:33.893 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-28T22:12:33.893 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_ti.ko 2024-03-28T22:12:33.893 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipaq.ko 2024-03-28T22:12:33.893 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipw.ko 2024-03-28T22:12:33.893 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ir-usb.ko 2024-03-28T22:12:33.893 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-28T22:12:33.893 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan.ko 2024-03-28T22:12:33.893 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-28T22:12:33.893 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-28T22:12:33.894 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-28T22:12:33.894 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mct_u232.ko 2024-03-28T22:12:33.894 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7720.ko 2024-03-28T22:12:33.894 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7840.ko 2024-03-28T22:12:33.894 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/navman.ko 2024-03-28T22:12:33.894 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/omninet.ko 2024-03-28T22:12:33.894 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/opticon.ko 2024-03-28T22:12:33.894 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/option.ko 2024-03-28T22:12:33.894 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/oti6858.ko 2024-03-28T22:12:33.894 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/pl2303.ko 2024-03-28T22:12:33.894 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcaux.ko 2024-03-28T22:12:33.894 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcserial.ko 2024-03-28T22:12:33.894 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/quatech2.ko 2024-03-28T22:12:33.894 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/safe_serial.ko 2024-03-28T22:12:33.894 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/sierra.ko 2024-03-28T22:12:33.895 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-28T22:12:33.895 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ssu100.ko 2024-03-28T22:12:33.895 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/symbolserial.ko 2024-03-28T22:12:33.895 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-28T22:12:33.895 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-28T22:12:33.895 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_debug.ko 2024-03-28T22:12:33.895 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-28T22:12:33.895 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/visor.ko 2024-03-28T22:12:33.895 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/whiteheat.ko 2024-03-28T22:12:33.895 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-28T22:12:33.895 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage 2024-03-28T22:12:33.895 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/uas.ko 2024-03-28T22:12:33.895 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-28T22:12:33.895 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-28T22:12:33.895 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-28T22:12:33.896 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-28T22:12:33.896 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-28T22:12:33.896 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-28T22:12:33.896 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-28T22:12:33.896 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-karma.ko 2024-03-28T22:12:33.896 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-28T22:12:33.896 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-28T22:12:33.896 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-28T22:12:33.896 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-28T22:12:33.896 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-28T22:12:33.896 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/usb-storage.ko 2024-03-28T22:12:33.896 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip 2024-03-28T22:12:33.896 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-28T22:12:33.896 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-28T22:12:33.896 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-28T22:12:33.897 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio 2024-03-28T22:12:33.897 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci 2024-03-28T22:12:33.897 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-28T22:12:33.897 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-28T22:12:33.897 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio.ko 2024-03-28T22:12:33.897 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-28T22:12:33.897 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost 2024-03-28T22:12:33.897 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost.ko 2024-03-28T22:12:33.897 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-28T22:12:33.897 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_net.ko 2024-03-28T22:12:33.897 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_scsi.ko 2024-03-28T22:12:33.897 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video 2024-03-28T22:12:33.897 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight 2024-03-28T22:12:33.897 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/apple_bl.ko 2024-03-28T22:12:33.897 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lcd.ko 2024-03-28T22:12:33.897 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-28T22:12:33.898 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio 2024-03-28T22:12:33.898 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio.ko 2024-03-28T22:12:33.898 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_balloon.ko 2024-03-28T22:12:33.898 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_mmio.ko 2024-03-28T22:12:33.898 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci.ko 2024-03-28T22:12:33.898 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-28T22:12:33.898 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-28T22:12:33.898 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_ring.ko 2024-03-28T22:12:33.898 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1 2024-03-28T22:12:33.898 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters 2024-03-28T22:12:33.898 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2482.ko 2024-03-28T22:12:33.898 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2490.ko 2024-03-28T22:12:33.898 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves 2024-03-28T22:12:33.898 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-28T22:12:33.899 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-28T22:12:33.899 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-28T22:12:33.899 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-28T22:12:33.899 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-28T22:12:33.899 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-28T22:12:33.899 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-28T22:12:33.899 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-28T22:12:33.899 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-28T22:12:33.899 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-28T22:12:33.899 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-28T22:12:33.899 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/wire.ko 2024-03-28T22:12:33.899 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog 2024-03-28T22:12:33.899 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-28T22:12:33.899 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-28T22:12:33.899 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-28T22:12:33.900 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/hpwdt.ko 2024-03-28T22:12:33.900 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/i6300esb.ko 2024-03-28T22:12:33.900 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-28T22:12:33.900 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-28T22:12:33.900 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ib700wdt.ko 2024-03-28T22:12:33.900 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ibmasr.ko 2024-03-28T22:12:33.900 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-28T22:12:33.900 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-28T22:12:33.900 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it87_wdt.ko 2024-03-28T22:12:33.900 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/machzwd.ko 2024-03-28T22:12:33.900 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/nv_tco.ko 2024-03-28T22:12:33.900 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-28T22:12:33.900 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-28T22:12:33.900 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-28T22:12:33.900 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-28T22:12:33.901 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/softdog.ko 2024-03-28T22:12:33.901 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-28T22:12:33.901 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/via_wdt.ko 2024-03-28T22:12:33.901 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-28T22:12:33.901 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-28T22:12:33.901 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-28T22:12:33.901 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wdt_pci.ko 2024-03-28T22:12:33.901 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/xen_wdt.ko 2024-03-28T22:12:33.901 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen 2024-03-28T22:12:33.901 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-28T22:12:33.901 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-evtchn.ko 2024-03-28T22:12:33.901 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntalloc.ko 2024-03-28T22:12:33.901 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntdev.ko 2024-03-28T22:12:33.901 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback 2024-03-28T22:12:33.901 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-28T22:12:33.901 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-privcmd.ko 2024-03-28T22:12:33.902 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs 2024-03-28T22:12:33.902 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-28T22:12:33.902 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs 2024-03-28T22:12:33.902 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p 2024-03-28T22:12:33.902 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p/9p.ko 2024-03-28T22:12:33.902 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs 2024-03-28T22:12:33.902 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs/affs.ko 2024-03-28T22:12:33.902 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs 2024-03-28T22:12:33.902 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs/befs.ko 2024-03-28T22:12:33.902 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/binfmt_misc.ko 2024-03-28T22:12:33.902 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs 2024-03-28T22:12:33.902 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs/btrfs.ko 2024-03-28T22:12:33.902 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles 2024-03-28T22:12:33.902 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles/cachefiles.ko 2024-03-28T22:12:33.902 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph 2024-03-28T22:12:33.903 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph/ceph.ko 2024-03-28T22:12:33.903 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda 2024-03-28T22:12:33.903 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda/coda.ko 2024-03-28T22:12:33.903 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs 2024-03-28T22:12:33.903 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs/cramfs.ko 2024-03-28T22:12:33.903 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm 2024-03-28T22:12:33.903 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm/dlm.ko 2024-03-28T22:12:33.903 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs 2024-03-28T22:12:33.903 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-28T22:12:33.903 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs 2024-03-28T22:12:33.903 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs/f2fs.ko 2024-03-28T22:12:33.903 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat 2024-03-28T22:12:33.903 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/fat.ko 2024-03-28T22:12:33.903 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/msdos.ko 2024-03-28T22:12:33.903 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/vfat.ko 2024-03-28T22:12:33.903 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse 2024-03-28T22:12:33.904 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/cuse.ko 2024-03-28T22:12:33.904 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/fuse.ko 2024-03-28T22:12:33.904 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2 2024-03-28T22:12:33.904 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2/gfs2.ko 2024-03-28T22:12:33.904 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs 2024-03-28T22:12:33.904 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs/hfs.ko 2024-03-28T22:12:33.904 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus 2024-03-28T22:12:33.904 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus/hfsplus.ko 2024-03-28T22:12:33.904 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs 2024-03-28T22:12:33.904 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs/isofs.ko 2024-03-28T22:12:33.904 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs 2024-03-28T22:12:33.904 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs/jfs.ko 2024-03-28T22:12:33.904 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd 2024-03-28T22:12:33.904 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd/lockd.ko 2024-03-28T22:12:33.904 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix 2024-03-28T22:12:33.905 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix/minix.ko 2024-03-28T22:12:33.905 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs 2024-03-28T22:12:33.905 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs/netfs.ko 2024-03-28T22:12:33.905 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs 2024-03-28T22:12:33.905 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout 2024-03-28T22:12:33.905 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-28T22:12:33.905 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout 2024-03-28T22:12:33.905 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-28T22:12:33.905 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout 2024-03-28T22:12:33.905 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-28T22:12:33.905 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfs.ko 2024-03-28T22:12:33.905 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv3.ko 2024-03-28T22:12:33.905 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv4.ko 2024-03-28T22:12:33.905 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common 2024-03-28T22:12:33.905 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/grace.ko 2024-03-28T22:12:33.905 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/nfs_acl.ko 2024-03-28T22:12:33.906 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd 2024-03-28T22:12:33.906 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd/nfsd.ko 2024-03-28T22:12:33.906 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2 2024-03-28T22:12:33.906 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2/nilfs2.ko 2024-03-28T22:12:33.906 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls 2024-03-28T22:12:33.906 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-celtic.ko 2024-03-28T22:12:33.906 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-centeuro.ko 2024-03-28T22:12:33.906 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-croatian.ko 2024-03-28T22:12:33.906 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-cyrillic.ko 2024-03-28T22:12:33.906 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-gaelic.ko 2024-03-28T22:12:33.906 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-greek.ko 2024-03-28T22:12:33.906 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-iceland.ko 2024-03-28T22:12:33.906 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-inuit.ko 2024-03-28T22:12:33.906 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-roman.ko 2024-03-28T22:12:33.907 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-romanian.ko 2024-03-28T22:12:33.907 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-turkish.ko 2024-03-28T22:12:33.907 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1250.ko 2024-03-28T22:12:33.907 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1251.ko 2024-03-28T22:12:33.907 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1255.ko 2024-03-28T22:12:33.907 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp737.ko 2024-03-28T22:12:33.907 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp775.ko 2024-03-28T22:12:33.907 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp850.ko 2024-03-28T22:12:33.907 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp852.ko 2024-03-28T22:12:33.907 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp855.ko 2024-03-28T22:12:33.907 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp857.ko 2024-03-28T22:12:33.907 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp860.ko 2024-03-28T22:12:33.907 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp861.ko 2024-03-28T22:12:33.907 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp862.ko 2024-03-28T22:12:33.907 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp863.ko 2024-03-28T22:12:33.908 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp864.ko 2024-03-28T22:12:33.908 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp865.ko 2024-03-28T22:12:33.908 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp866.ko 2024-03-28T22:12:33.908 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp869.ko 2024-03-28T22:12:33.908 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp874.ko 2024-03-28T22:12:33.908 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp932.ko 2024-03-28T22:12:33.908 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp936.ko 2024-03-28T22:12:33.908 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp949.ko 2024-03-28T22:12:33.908 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp950.ko 2024-03-28T22:12:33.908 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_euc-jp.ko 2024-03-28T22:12:33.908 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-1.ko 2024-03-28T22:12:33.908 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-13.ko 2024-03-28T22:12:33.908 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-14.ko 2024-03-28T22:12:33.908 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-15.ko 2024-03-28T22:12:33.908 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-2.ko 2024-03-28T22:12:33.908 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-3.ko 2024-03-28T22:12:33.909 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-4.ko 2024-03-28T22:12:33.909 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-5.ko 2024-03-28T22:12:33.909 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-6.ko 2024-03-28T22:12:33.909 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-7.ko 2024-03-28T22:12:33.909 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-9.ko 2024-03-28T22:12:33.909 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-r.ko 2024-03-28T22:12:33.909 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-ru.ko 2024-03-28T22:12:33.909 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-u.ko 2024-03-28T22:12:33.909 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-28T22:12:33.909 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_utf8.ko 2024-03-28T22:12:33.909 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2 2024-03-28T22:12:33.909 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster 2024-03-28T22:12:33.909 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-28T22:12:33.909 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm 2024-03-28T22:12:33.909 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-28T22:12:33.910 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs 2024-03-28T22:12:33.910 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-28T22:12:33.910 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2.ko 2024-03-28T22:12:33.910 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-28T22:12:33.910 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-28T22:12:33.910 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-28T22:12:33.910 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs 2024-03-28T22:12:33.910 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs/overlay.ko 2024-03-28T22:12:33.910 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore 2024-03-28T22:12:33.910 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore/ramoops.ko 2024-03-28T22:12:33.910 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs 2024-03-28T22:12:33.910 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs/reiserfs.ko 2024-03-28T22:12:33.910 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs 2024-03-28T22:12:33.911 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs/romfs.ko 2024-03-28T22:12:33.911 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb 2024-03-28T22:12:33.911 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client 2024-03-28T22:12:33.911 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client/cifs.ko 2024-03-28T22:12:33.911 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common 2024-03-28T22:12:33.911 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_arc4.ko 2024-03-28T22:12:33.911 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_md4.ko 2024-03-28T22:12:33.911 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs 2024-03-28T22:12:33.911 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs/squashfs.ko 2024-03-28T22:12:33.911 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv 2024-03-28T22:12:33.911 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv/sysv.ko 2024-03-28T22:12:33.911 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf 2024-03-28T22:12:33.911 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf/udf.ko 2024-03-28T22:12:33.911 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs 2024-03-28T22:12:33.911 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs/ufs.ko 2024-03-28T22:12:33.912 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs 2024-03-28T22:12:33.912 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs/xfs.ko 2024-03-28T22:12:33.912 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel 2024-03-28T22:12:33.912 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu 2024-03-28T22:12:33.912 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu/rcutorture.ko 2024-03-28T22:12:33.912 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/torture.ko 2024-03-28T22:12:33.912 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace 2024-03-28T22:12:33.912 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-28T22:12:33.912 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib 2024-03-28T22:12:33.912 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/asn1_encoder.ko 2024-03-28T22:12:33.912 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc-itu-t.ko 2024-03-28T22:12:33.912 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc8.ko 2024-03-28T22:12:33.912 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto 2024-03-28T22:12:33.912 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libarc4.ko 2024-03-28T22:12:33.912 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libdes.ko 2024-03-28T22:12:33.913 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/libcrc32c.ko 2024-03-28T22:12:33.913 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lru_cache.ko 2024-03-28T22:12:33.913 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4 2024-03-28T22:12:33.913 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4_compress.ko 2024-03-28T22:12:33.913 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4hc_compress.ko 2024-03-28T22:12:33.913 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math 2024-03-28T22:12:33.913 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math/cordic.ko 2024-03-28T22:12:33.913 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6 2024-03-28T22:12:33.913 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6/raid6_pq.ko 2024-03-28T22:12:33.913 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon 2024-03-28T22:12:33.913 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-28T22:12:33.913 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_bm.ko 2024-03-28T22:12:33.913 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_fsm.ko 2024-03-28T22:12:33.913 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_kmp.ko 2024-03-28T22:12:33.913 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd 2024-03-28T22:12:33.914 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd/zstd_compress.ko 2024-03-28T22:12:33.914 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm 2024-03-28T22:12:33.914 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm/hwpoison-inject.ko 2024-03-28T22:12:33.914 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net 2024-03-28T22:12:33.914 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan 2024-03-28T22:12:33.914 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/6lowpan.ko 2024-03-28T22:12:33.914 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_dest.ko 2024-03-28T22:12:33.914 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_fragment.ko 2024-03-28T22:12:33.914 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_hop.ko 2024-03-28T22:12:33.914 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-28T22:12:33.914 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_mobility.ko 2024-03-28T22:12:33.914 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_routing.ko 2024-03-28T22:12:33.914 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_udp.ko 2024-03-28T22:12:33.914 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802 2024-03-28T22:12:33.914 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/garp.ko 2024-03-28T22:12:33.915 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/mrp.ko 2024-03-28T22:12:33.915 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/p8022.ko 2024-03-28T22:12:33.915 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/psnap.ko 2024-03-28T22:12:33.915 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/stp.ko 2024-03-28T22:12:33.915 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q 2024-03-28T22:12:33.915 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q/8021q.ko 2024-03-28T22:12:33.915 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p 2024-03-28T22:12:33.915 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet.ko 2024-03-28T22:12:33.915 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_fd.ko 2024-03-28T22:12:33.915 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_rdma.ko 2024-03-28T22:12:33.915 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_virtio.ko 2024-03-28T22:12:33.915 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv 2024-03-28T22:12:33.915 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv/batman-adv.ko 2024-03-28T22:12:33.915 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge 2024-03-28T22:12:33.915 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/br_netfilter.ko 2024-03-28T22:12:33.916 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/bridge.ko 2024-03-28T22:12:33.916 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter 2024-03-28T22:12:33.916 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-28T22:12:33.916 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-28T22:12:33.916 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-28T22:12:33.916 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-28T22:12:33.916 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-28T22:12:33.916 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-28T22:12:33.916 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-28T22:12:33.916 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-28T22:12:33.916 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-28T22:12:33.916 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-28T22:12:33.916 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-28T22:12:33.916 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-28T22:12:33.916 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-28T22:12:33.917 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-28T22:12:33.917 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-28T22:12:33.917 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-28T22:12:33.917 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-28T22:12:33.917 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-28T22:12:33.917 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-28T22:12:33.917 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-28T22:12:33.917 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtables.ko 2024-03-28T22:12:33.917 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-28T22:12:33.917 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-28T22:12:33.917 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-28T22:12:33.917 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph 2024-03-28T22:12:33.917 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph/libceph.ko 2024-03-28T22:12:33.917 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core 2024-03-28T22:12:33.917 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/failover.ko 2024-03-28T22:12:33.917 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/pktgen.ko 2024-03-28T22:12:33.918 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp 2024-03-28T22:12:33.918 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp.ko 2024-03-28T22:12:33.918 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_diag.ko 2024-03-28T22:12:33.918 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv4.ko 2024-03-28T22:12:33.918 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv6.ko 2024-03-28T22:12:33.918 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver 2024-03-28T22:12:33.918 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver/dns_resolver.ko 2024-03-28T22:12:33.918 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa 2024-03-28T22:12:33.918 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/dsa_core.ko 2024-03-28T22:12:33.918 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_brcm.ko 2024-03-28T22:12:33.918 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_dsa.ko 2024-03-28T22:12:33.918 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_none.ko 2024-03-28T22:12:33.918 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_trailer.ko 2024-03-28T22:12:33.919 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154 2024-03-28T22:12:33.919 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan 2024-03-28T22:12:33.919 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-28T22:12:33.919 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154.ko 2024-03-28T22:12:33.919 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-28T22:12:33.919 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4 2024-03-28T22:12:33.919 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ah4.ko 2024-03-28T22:12:33.919 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/esp4.ko 2024-03-28T22:12:33.919 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/fou.ko 2024-03-28T22:12:33.919 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/gre.ko 2024-03-28T22:12:33.919 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/inet_diag.ko 2024-03-28T22:12:33.919 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_gre.ko 2024-03-28T22:12:33.919 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_tunnel.ko 2024-03-28T22:12:33.919 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_vti.ko 2024-03-28T22:12:33.919 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipcomp.ko 2024-03-28T22:12:33.920 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipip.ko 2024-03-28T22:12:33.920 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter 2024-03-28T22:12:33.920 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-28T22:12:33.920 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-28T22:12:33.920 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-28T22:12:33.920 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-28T22:12:33.920 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-28T22:12:33.920 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-28T22:12:33.920 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-28T22:12:33.920 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-28T22:12:33.920 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-28T22:12:33.920 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-28T22:12:33.920 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-28T22:12:33.920 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-28T22:12:33.920 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-28T22:12:33.921 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-28T22:12:33.921 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-28T22:12:33.921 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-28T22:12:33.921 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-28T22:12:33.921 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-28T22:12:33.921 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-28T22:12:33.921 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-28T22:12:33.921 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-28T22:12:33.921 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_bic.ko 2024-03-28T22:12:33.921 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_dctcp.ko 2024-03-28T22:12:33.921 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_diag.ko 2024-03-28T22:12:33.921 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_highspeed.ko 2024-03-28T22:12:33.921 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_htcp.ko 2024-03-28T22:12:33.921 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_hybla.ko 2024-03-28T22:12:33.921 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_illinois.ko 2024-03-28T22:12:33.922 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_lp.ko 2024-03-28T22:12:33.922 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_scalable.ko 2024-03-28T22:12:33.922 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_vegas.ko 2024-03-28T22:12:33.922 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_veno.ko 2024-03-28T22:12:33.922 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_westwood.ko 2024-03-28T22:12:33.922 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_yeah.ko 2024-03-28T22:12:33.922 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tunnel4.ko 2024-03-28T22:12:33.922 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_diag.ko 2024-03-28T22:12:33.922 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_tunnel.ko 2024-03-28T22:12:33.922 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-28T22:12:33.922 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6 2024-03-28T22:12:33.922 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ah6.ko 2024-03-28T22:12:33.922 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/esp6.ko 2024-03-28T22:12:33.922 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/fou6.ko 2024-03-28T22:12:33.922 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_tunnel.ko 2024-03-28T22:12:33.923 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-28T22:12:33.923 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_vti.ko 2024-03-28T22:12:33.923 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ipcomp6.ko 2024-03-28T22:12:33.923 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter 2024-03-28T22:12:33.923 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-28T22:12:33.923 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-28T22:12:33.923 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-28T22:12:33.923 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-28T22:12:33.923 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-28T22:12:33.923 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-28T22:12:33.923 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-28T22:12:33.924 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-28T22:12:33.924 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-28T22:12:33.924 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-28T22:12:33.924 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-28T22:12:33.924 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-28T22:12:33.924 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-28T22:12:33.924 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-28T22:12:33.924 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-28T22:12:33.924 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-28T22:12:33.924 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-28T22:12:33.924 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-28T22:12:33.924 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-28T22:12:33.924 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-28T22:12:33.924 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-28T22:12:33.925 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-28T22:12:33.925 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-28T22:12:33.925 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-28T22:12:33.925 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/sit.ko 2024-03-28T22:12:33.925 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/tunnel6.ko 2024-03-28T22:12:33.925 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-28T22:12:33.925 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key 2024-03-28T22:12:33.925 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key/af_key.ko 2024-03-28T22:12:33.925 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp 2024-03-28T22:12:33.925 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_core.ko 2024-03-28T22:12:33.925 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-28T22:12:33.925 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_eth.ko 2024-03-28T22:12:33.925 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip.ko 2024-03-28T22:12:33.925 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip6.ko 2024-03-28T22:12:33.925 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_netlink.ko 2024-03-28T22:12:33.926 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ppp.ko 2024-03-28T22:12:33.926 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc 2024-03-28T22:12:33.926 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc/llc.ko 2024-03-28T22:12:33.926 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154 2024-03-28T22:12:33.926 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154/mac802154.ko 2024-03-28T22:12:33.926 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls 2024-03-28T22:12:33.926 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls/mpls_gso.ko 2024-03-28T22:12:33.926 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter 2024-03-28T22:12:33.926 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset 2024-03-28T22:12:33.926 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set.ko 2024-03-28T22:12:33.926 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-28T22:12:33.926 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-28T22:12:33.926 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-28T22:12:33.926 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-28T22:12:33.927 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-28T22:12:33.927 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-28T22:12:33.927 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-28T22:12:33.927 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-28T22:12:33.927 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-28T22:12:33.927 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-28T22:12:33.927 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-28T22:12:33.927 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-28T22:12:33.927 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-28T22:12:33.927 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-28T22:12:33.927 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-28T22:12:33.927 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs 2024-03-28T22:12:33.927 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-28T22:12:33.927 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-28T22:12:33.928 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-28T22:12:33.928 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-28T22:12:33.928 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-28T22:12:33.928 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-28T22:12:33.928 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-28T22:12:33.928 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-28T22:12:33.928 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-28T22:12:33.928 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-28T22:12:33.928 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-28T22:12:33.928 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-28T22:12:33.928 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-28T22:12:33.928 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-28T22:12:33.928 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conncount.ko 2024-03-28T22:12:33.928 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack.ko 2024-03-28T22:12:33.928 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-28T22:12:33.928 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-28T22:12:33.929 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-28T22:12:33.929 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-28T22:12:33.929 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-28T22:12:33.929 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-28T22:12:33.929 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-28T22:12:33.929 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-28T22:12:33.929 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-28T22:12:33.929 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-28T22:12:33.929 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-28T22:12:33.929 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-28T22:12:33.929 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-28T22:12:33.929 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table.ko 2024-03-28T22:12:33.929 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-28T22:12:33.929 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_log_syslog.ko 2024-03-28T22:12:33.929 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat.ko 2024-03-28T22:12:33.929 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-28T22:12:33.930 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-28T22:12:33.930 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_irc.ko 2024-03-28T22:12:33.930 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_sip.ko 2024-03-28T22:12:33.930 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-28T22:12:33.930 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-28T22:12:33.930 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_tables.ko 2024-03-28T22:12:33.930 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink.ko 2024-03-28T22:12:33.930 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-28T22:12:33.930 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_log.ko 2024-03-28T22:12:33.930 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-28T22:12:33.930 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-28T22:12:33.930 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_chain_nat.ko 2024-03-28T22:12:33.930 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_compat.ko 2024-03-28T22:12:33.930 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_ct.ko 2024-03-28T22:12:33.930 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-28T22:12:33.931 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fib.ko 2024-03-28T22:12:33.931 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-28T22:12:33.931 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_hash.ko 2024-03-28T22:12:33.931 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_limit.ko 2024-03-28T22:12:33.931 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_log.ko 2024-03-28T22:12:33.931 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_masq.ko 2024-03-28T22:12:33.931 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_nat.ko 2024-03-28T22:12:33.931 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_numgen.ko 2024-03-28T22:12:33.931 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_queue.ko 2024-03-28T22:12:33.931 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_quota.ko 2024-03-28T22:12:33.931 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_redir.ko 2024-03-28T22:12:33.931 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject.ko 2024-03-28T22:12:33.931 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject_inet.ko 2024-03-28T22:12:33.931 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_socket.ko 2024-03-28T22:12:33.931 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_tunnel.ko 2024-03-28T22:12:33.932 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_AUDIT.ko 2024-03-28T22:12:33.932 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-28T22:12:33.932 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-28T22:12:33.932 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-28T22:12:33.932 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CT.ko 2024-03-28T22:12:33.932 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_DSCP.ko 2024-03-28T22:12:33.932 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HL.ko 2024-03-28T22:12:33.932 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HMARK.ko 2024-03-28T22:12:33.932 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-28T22:12:33.932 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LED.ko 2024-03-28T22:12:33.932 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LOG.ko 2024-03-28T22:12:33.932 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-28T22:12:33.932 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NETMAP.ko 2024-03-28T22:12:33.932 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFLOG.ko 2024-03-28T22:12:33.932 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-28T22:12:33.932 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_RATEEST.ko 2024-03-28T22:12:33.933 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-28T22:12:33.933 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_SECMARK.ko 2024-03-28T22:12:33.933 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-28T22:12:33.933 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-28T22:12:33.933 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TEE.ko 2024-03-28T22:12:33.933 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TPROXY.ko 2024-03-28T22:12:33.933 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TRACE.ko 2024-03-28T22:12:33.933 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_addrtype.ko 2024-03-28T22:12:33.933 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_bpf.ko 2024-03-28T22:12:33.933 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cgroup.ko 2024-03-28T22:12:33.933 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cluster.ko 2024-03-28T22:12:33.933 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_comment.ko 2024-03-28T22:12:33.933 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connbytes.ko 2024-03-28T22:12:33.933 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlabel.ko 2024-03-28T22:12:33.933 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlimit.ko 2024-03-28T22:12:33.933 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connmark.ko 2024-03-28T22:12:33.934 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_conntrack.ko 2024-03-28T22:12:33.934 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cpu.ko 2024-03-28T22:12:33.934 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dccp.ko 2024-03-28T22:12:33.934 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_devgroup.ko 2024-03-28T22:12:33.934 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dscp.ko 2024-03-28T22:12:33.934 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ecn.ko 2024-03-28T22:12:33.934 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_esp.ko 2024-03-28T22:12:33.934 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hashlimit.ko 2024-03-28T22:12:33.934 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_helper.ko 2024-03-28T22:12:33.934 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hl.ko 2024-03-28T22:12:33.934 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipcomp.ko 2024-03-28T22:12:33.934 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_iprange.ko 2024-03-28T22:12:33.934 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipvs.ko 2024-03-28T22:12:33.934 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_l2tp.ko 2024-03-28T22:12:33.935 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_length.ko 2024-03-28T22:12:33.935 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_limit.ko 2024-03-28T22:12:33.935 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mac.ko 2024-03-28T22:12:33.935 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mark.ko 2024-03-28T22:12:33.935 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_multiport.ko 2024-03-28T22:12:33.935 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nat.ko 2024-03-28T22:12:33.935 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nfacct.ko 2024-03-28T22:12:33.935 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_osf.ko 2024-03-28T22:12:33.935 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_owner.ko 2024-03-28T22:12:33.935 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_physdev.ko 2024-03-28T22:12:33.935 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_pkttype.ko 2024-03-28T22:12:33.935 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_policy.ko 2024-03-28T22:12:33.935 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_quota.ko 2024-03-28T22:12:33.935 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_rateest.ko 2024-03-28T22:12:33.935 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_realm.ko 2024-03-28T22:12:33.935 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_recent.ko 2024-03-28T22:12:33.936 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_sctp.ko 2024-03-28T22:12:33.936 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_set.ko 2024-03-28T22:12:33.936 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_socket.ko 2024-03-28T22:12:33.936 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_state.ko 2024-03-28T22:12:33.936 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_statistic.ko 2024-03-28T22:12:33.936 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_string.ko 2024-03-28T22:12:33.936 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_tcpmss.ko 2024-03-28T22:12:33.936 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_time.ko 2024-03-28T22:12:33.936 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_u32.ko 2024-03-28T22:12:33.936 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink 2024-03-28T22:12:33.936 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink/netlink_diag.ko 2024-03-28T22:12:33.936 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh 2024-03-28T22:12:33.936 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh/nsh.ko 2024-03-28T22:12:33.936 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch 2024-03-28T22:12:33.936 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/openvswitch.ko 2024-03-28T22:12:33.937 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-geneve.ko 2024-03-28T22:12:33.937 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-gre.ko 2024-03-28T22:12:33.937 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-vxlan.ko 2024-03-28T22:12:33.937 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet 2024-03-28T22:12:33.937 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet/af_packet_diag.ko 2024-03-28T22:12:33.937 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds 2024-03-28T22:12:33.937 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds.ko 2024-03-28T22:12:33.937 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_rdma.ko 2024-03-28T22:12:33.937 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_tcp.ko 2024-03-28T22:12:33.937 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched 2024-03-28T22:12:33.937 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_csum.ko 2024-03-28T22:12:33.937 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_gact.ko 2024-03-28T22:12:33.937 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_mirred.ko 2024-03-28T22:12:33.937 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_nat.ko 2024-03-28T22:12:33.937 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_pedit.ko 2024-03-28T22:12:33.937 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_police.ko 2024-03-28T22:12:33.938 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_simple.ko 2024-03-28T22:12:33.938 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_skbedit.ko 2024-03-28T22:12:33.938 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_vlan.ko 2024-03-28T22:12:33.938 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_basic.ko 2024-03-28T22:12:33.938 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_bpf.ko 2024-03-28T22:12:33.938 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_flow.ko 2024-03-28T22:12:33.938 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_fw.ko 2024-03-28T22:12:33.938 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_route.ko 2024-03-28T22:12:33.938 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_u32.ko 2024-03-28T22:12:33.938 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_cmp.ko 2024-03-28T22:12:33.938 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_ipset.ko 2024-03-28T22:12:33.938 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_meta.ko 2024-03-28T22:12:33.938 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_nbyte.ko 2024-03-28T22:12:33.938 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_text.ko 2024-03-28T22:12:33.938 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_u32.ko 2024-03-28T22:12:33.939 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_choke.ko 2024-03-28T22:12:33.939 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_codel.ko 2024-03-28T22:12:33.939 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_drr.ko 2024-03-28T22:12:33.939 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_fq.ko 2024-03-28T22:12:33.939 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_gred.ko 2024-03-28T22:12:33.939 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hfsc.ko 2024-03-28T22:12:33.939 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hhf.ko 2024-03-28T22:12:33.939 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_htb.ko 2024-03-28T22:12:33.939 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_ingress.ko 2024-03-28T22:12:33.939 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio.ko 2024-03-28T22:12:33.939 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio_lib.ko 2024-03-28T22:12:33.939 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_multiq.ko 2024-03-28T22:12:33.939 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_netem.ko 2024-03-28T22:12:33.939 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_pie.ko 2024-03-28T22:12:33.939 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_plug.ko 2024-03-28T22:12:33.940 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_prio.ko 2024-03-28T22:12:33.940 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_qfq.ko 2024-03-28T22:12:33.940 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_red.ko 2024-03-28T22:12:33.940 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfb.ko 2024-03-28T22:12:33.940 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfq.ko 2024-03-28T22:12:33.940 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_tbf.ko 2024-03-28T22:12:33.940 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_teql.ko 2024-03-28T22:12:33.940 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp 2024-03-28T22:12:33.940 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp.ko 2024-03-28T22:12:33.940 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp_diag.ko 2024-03-28T22:12:33.940 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc 2024-03-28T22:12:33.940 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss 2024-03-28T22:12:33.940 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-28T22:12:33.940 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-28T22:12:33.940 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/sunrpc.ko 2024-03-28T22:12:33.941 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma 2024-03-28T22:12:33.941 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-28T22:12:33.941 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc 2024-03-28T22:12:33.941 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/diag.ko 2024-03-28T22:12:33.941 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/tipc.ko 2024-03-28T22:12:33.941 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix 2024-03-28T22:12:33.941 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix/unix_diag.ko 2024-03-28T22:12:33.941 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock 2024-03-28T22:12:33.941 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-28T22:12:33.941 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-28T22:12:33.941 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock.ko 2024-03-28T22:12:33.941 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-28T22:12:33.941 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-28T22:12:33.941 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm 2024-03-28T22:12:33.941 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-28T22:12:33.942 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security 2024-03-28T22:12:33.942 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys 2024-03-28T22:12:33.942 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys 2024-03-28T22:12:33.942 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-28T22:12:33.942 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys 2024-03-28T22:12:33.942 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys/trusted.ko 2024-03-28T22:12:33.942 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt 2024-03-28T22:12:33.942 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib 2024-03-28T22:12:33.942 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib/irqbypass.ko 2024-03-28T22:12:33.942 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias 2024-03-28T22:12:33.942 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias.bin 2024-03-28T22:12:33.942 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin 2024-03-28T22:12:33.942 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.alias.bin 2024-03-28T22:12:33.942 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.bin 2024-03-28T22:12:33.942 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.modinfo 2024-03-28T22:12:33.942 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep 2024-03-28T22:12:33.943 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep.bin 2024-03-28T22:12:33.943 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.devname 2024-03-28T22:12:33.943 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.order 2024-03-28T22:12:33.943 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.softdep 2024-03-28T22:12:33.943 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols 2024-03-28T22:12:33.943 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols.bin 2024-03-28T22:12:33.943 INFO:teuthology.orchestra.run.smithi043.stdout:/lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-03-28T22:12:33.947 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g1df0c7e8a235 2024-03-28T22:12:33.947 DEBUG:teuthology.orchestra.run.smithi043:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-28T22:12:33.970 INFO:teuthology.orchestra.run.smithi043.stdout:postinstall scriptlet (using /bin/sh): 2024-03-28T22:12:33.970 INFO:teuthology.orchestra.run.smithi043.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-28T22:12:33.970 INFO:teuthology.orchestra.run.smithi043.stdout: /usr/bin/kernel-install add 6.8.0-g1df0c7e8a235 /lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-03-28T22:12:33.970 INFO:teuthology.orchestra.run.smithi043.stdout:fi 2024-03-28T22:12:33.971 INFO:teuthology.orchestra.run.smithi043.stdout:for file in vmlinuz System.map config; do 2024-03-28T22:12:33.971 INFO:teuthology.orchestra.run.smithi043.stdout: if ! cmp --silent "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235"; then 2024-03-28T22:12:33.971 INFO:teuthology.orchestra.run.smithi043.stdout: cp "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235" 2024-03-28T22:12:33.971 INFO:teuthology.orchestra.run.smithi043.stdout: fi 2024-03-28T22:12:33.971 INFO:teuthology.orchestra.run.smithi043.stdout:done 2024-03-28T22:12:33.971 INFO:teuthology.orchestra.run.smithi043.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-28T22:12:33.971 INFO:teuthology.orchestra.run.smithi043.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-28T22:12:33.971 INFO:teuthology.orchestra.run.smithi043.stdout:new-kernel-pkg --remove 6.8.0-g1df0c7e8a235 --rminitrd --initrdfile=/boot/initramfs-6.8.0-g1df0c7e8a235.img 2024-03-28T22:12:33.971 INFO:teuthology.orchestra.run.smithi043.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-28T22:12:33.971 INFO:teuthology.orchestra.run.smithi043.stdout:kernel-install remove 6.8.0-g1df0c7e8a235 2024-03-28T22:12:33.971 INFO:teuthology.orchestra.run.smithi043.stdout:fi 2024-03-28T22:12:33.971 INFO:teuthology.orchestra.run.smithi043.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-28T22:12:33.971 INFO:teuthology.orchestra.run.smithi043.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-28T22:12:33.971 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/update-bootloader --remove 6.8.0-g1df0c7e8a235 2024-03-28T22:12:33.971 INFO:teuthology.orchestra.run.smithi043.stdout:fi 2024-03-28T22:12:33.972 DEBUG:teuthology.orchestra.run.smithi043:> sudo rpm -qi grub2-tools 2024-03-28T22:12:34.050 INFO:teuthology.orchestra.run.smithi043.stdout:Name : grub2-tools 2024-03-28T22:12:34.051 INFO:teuthology.orchestra.run.smithi043.stdout:Epoch : 1 2024-03-28T22:12:34.051 INFO:teuthology.orchestra.run.smithi043.stdout:Version : 2.06 2024-03-28T22:12:34.051 INFO:teuthology.orchestra.run.smithi043.stdout:Release : 68.el9 2024-03-28T22:12:34.051 INFO:teuthology.orchestra.run.smithi043.stdout:Architecture: x86_64 2024-03-28T22:12:34.051 INFO:teuthology.orchestra.run.smithi043.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-28T22:12:34.051 INFO:teuthology.orchestra.run.smithi043.stdout:Group : Unspecified 2024-03-28T22:12:34.051 INFO:teuthology.orchestra.run.smithi043.stdout:Size : 8264254 2024-03-28T22:12:34.051 INFO:teuthology.orchestra.run.smithi043.stdout:License : GPLv3+ 2024-03-28T22:12:34.051 INFO:teuthology.orchestra.run.smithi043.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-28T22:12:34.051 INFO:teuthology.orchestra.run.smithi043.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-28T22:12:34.051 INFO:teuthology.orchestra.run.smithi043.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-28T22:12:34.051 INFO:teuthology.orchestra.run.smithi043.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-28T22:12:34.051 INFO:teuthology.orchestra.run.smithi043.stdout:Packager : builder@centos.org 2024-03-28T22:12:34.052 INFO:teuthology.orchestra.run.smithi043.stdout:Vendor : CentOS 2024-03-28T22:12:34.052 INFO:teuthology.orchestra.run.smithi043.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-28T22:12:34.052 INFO:teuthology.orchestra.run.smithi043.stdout:Summary : Support tools for GRUB. 2024-03-28T22:12:34.052 INFO:teuthology.orchestra.run.smithi043.stdout:Description : 2024-03-28T22:12:34.052 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:12:34.052 INFO:teuthology.orchestra.run.smithi043.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-28T22:12:34.052 INFO:teuthology.orchestra.run.smithi043.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-28T22:12:34.052 INFO:teuthology.orchestra.run.smithi043.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-28T22:12:34.052 INFO:teuthology.orchestra.run.smithi043.stdout:hardware devices. 2024-03-28T22:12:34.052 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:12:34.052 INFO:teuthology.orchestra.run.smithi043.stdout:This subpackage provides tools for support of all platforms. 2024-03-28T22:12:34.053 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-28T22:12:34.053 INFO:teuthology.task.kernel:Updating grub on smithi043 to boot 6.8.0-g1df0c7e8a235 2024-03-28T22:12:34.053 DEBUG:teuthology.orchestra.run.smithi043:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-28T22:12:34.708 INFO:teuthology.orchestra.run.smithi043.stderr:Generating grub configuration file ... 2024-03-28T22:12:36.209 INFO:teuthology.orchestra.run.smithi043.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-28T22:12:36.255 INFO:teuthology.orchestra.run.smithi043.stderr:done 2024-03-28T22:12:36.255 DEBUG:teuthology.orchestra.run.smithi043:> mktemp 2024-03-28T22:12:36.270 INFO:teuthology.orchestra.run.smithi043.stdout:/tmp/tmp.ZUX5RphPAz 2024-03-28T22:12:36.271 DEBUG:teuthology.orchestra.run.smithi043:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ZUX5RphPAz 2024-03-28T22:12:36.343 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0666 /tmp/tmp.ZUX5RphPAz 2024-03-28T22:12:36.495 DEBUG:teuthology.orchestra.remote:smithi043:/tmp/tmp.ZUX5RphPAz is 6KB 2024-03-28T22:12:36.543 DEBUG:teuthology.orchestra.run.smithi043:> rm -fr /tmp/tmp.ZUX5RphPAz 2024-03-28T22:12:36.558 DEBUG:teuthology.orchestra.run.smithi043:> sudo /bin/ls /boot/loader/entries || true 2024-03-28T22:12:36.627 INFO:teuthology.orchestra.run.smithi043.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-28T22:12:36.627 INFO:teuthology.orchestra.run.smithi043.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-28T22:12:36.627 INFO:teuthology.orchestra.run.smithi043.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-28T22:12:36.627 INFO:teuthology.orchestra.run.smithi043.stdout:15949e3c88704300882482eddfd7baae-6.8.0-g1df0c7e8a235.conf 2024-03-28T22:12:36.628 DEBUG:teuthology.orchestra.run.smithi043:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-g1df0c7e8a235 2024-03-28T22:12:36.776 DEBUG:teuthology.orchestra.run.smithi043:> sudo shutdown -r now 2024-03-28T22:13:06.781 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-28T22:13:06.782 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-03-28T22:13:06.783 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T22:13:25.150 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.43 2024-03-28T22:13:34.152 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-03-28T22:13:34.153 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T22:13:52.542 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.43 2024-03-28T22:14:04.544 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-03-28T22:14:04.545 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T22:14:07.614 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.43 2024-03-28T22:14:22.616 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-03-28T22:14:22.617 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T22:14:23.059 DEBUG:teuthology.orchestra.run.smithi043:> true 2024-03-28T22:14:23.594 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-03-28T22:14:23.594 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-g1df0c7e8a235"... 2024-03-28T22:14:23.594 DEBUG:teuthology.orchestra.run.smithi043:> uname -r 2024-03-28T22:14:23.631 INFO:teuthology.orchestra.run.smithi043.stdout:6.8.0-g1df0c7e8a235 2024-03-28T22:14:23.631 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g1df0c7e8a235 vs 6.8.0-g1df0c7e8a235 2024-03-28T22:14:23.631 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-28T22:14:23.631 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-28T22:14:24.632 DEBUG:teuthology.parallel:result is None 2024-03-28T22:14:24.633 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-28T22:14:24.641 INFO:teuthology.task.internal:Creating test directory... 2024-03-28T22:14:24.641 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T22:14:24.648 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T22:14:24.651 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T22:14:24.694 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-28T22:14:24.701 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-28T22:14:24.709 INFO:teuthology.task.internal:Creating archive directory... 2024-03-28T22:14:24.709 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T22:14:24.716 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T22:14:24.718 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T22:14:24.851 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-28T22:14:24.857 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-28T22:14:24.857 DEBUG:teuthology.orchestra.run.smithi082:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T22:14:24.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T22:14:24.873 DEBUG:teuthology.orchestra.run.smithi172:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T22:14:24.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T22:14:24.890 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T22:14:24.897 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T22:14:24.916 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T22:14:24.967 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T22:14:24.984 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T22:14:24.998 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T22:14:25.020 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T22:14:25.045 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T22:14:25.116 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T22:14:25.121 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-28T22:14:25.129 INFO:teuthology.task.internal:Configuring sudo... 2024-03-28T22:14:25.129 DEBUG:teuthology.orchestra.run.smithi043:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T22:14:25.133 DEBUG:teuthology.orchestra.run.smithi082:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T22:14:25.136 DEBUG:teuthology.orchestra.run.smithi172:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T22:14:25.222 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-28T22:14:25.231 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-28T22:14:25.232 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T22:14:25.236 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T22:14:25.238 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T22:14:25.282 DEBUG:teuthology.orchestra.run.smithi043:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T22:14:25.358 DEBUG:teuthology.orchestra.run.smithi043:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T22:14:25.479 DEBUG:teuthology.orchestra.run.smithi043:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T22:14:25.551 DEBUG:teuthology.orchestra.run.smithi043:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T22:14:25.637 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-28T22:14:25.637 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T22:14:25.719 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T22:14:25.761 DEBUG:teuthology.orchestra.run.smithi082:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T22:14:25.817 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T22:14:25.901 DEBUG:teuthology.orchestra.run.smithi082:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T22:14:25.948 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-28T22:14:25.948 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T22:14:26.017 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T22:14:26.063 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T22:14:26.114 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T22:14:26.197 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T22:14:26.243 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-28T22:14:26.243 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T22:14:26.312 DEBUG:teuthology.orchestra.run.smithi043:> sudo service rsyslog restart 2024-03-28T22:14:26.317 DEBUG:teuthology.orchestra.run.smithi082:> sudo service rsyslog restart 2024-03-28T22:14:26.320 DEBUG:teuthology.orchestra.run.smithi172:> sudo service rsyslog restart 2024-03-28T22:14:26.355 INFO:teuthology.orchestra.run.smithi082.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T22:14:26.399 INFO:teuthology.orchestra.run.smithi172.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T22:14:26.459 INFO:teuthology.orchestra.run.smithi043.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T22:14:26.868 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-28T22:14:26.876 INFO:teuthology.task.internal:Starting timer... 2024-03-28T22:14:26.876 INFO:teuthology.run_tasks:Running task pcp... 2024-03-28T22:14:26.885 INFO:teuthology.run_tasks:Running task selinux... 2024-03-28T22:14:26.893 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-28T22:14:26.894 DEBUG:teuthology.orchestra.run.smithi043:> sudo service auditd rotate 2024-03-28T22:14:27.112 INFO:teuthology.orchestra.run.smithi043.stdout:Rotating logs: 2024-03-28T22:14:27.119 DEBUG:teuthology.orchestra.run.smithi082:> sudo service auditd rotate 2024-03-28T22:14:27.179 INFO:teuthology.orchestra.run.smithi082.stdout:Rotating logs: 2024-03-28T22:14:27.180 DEBUG:teuthology.orchestra.run.smithi172:> sudo service auditd rotate 2024-03-28T22:14:27.231 INFO:teuthology.orchestra.run.smithi172.stdout:Rotating logs: 2024-03-28T22:14:27.234 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-28T22:14:27.234 DEBUG:teuthology.orchestra.run.smithi043:> /usr/sbin/getenforce 2024-03-28T22:14:27.320 INFO:teuthology.orchestra.run.smithi043.stdout:Permissive 2024-03-28T22:14:27.320 DEBUG:teuthology.orchestra.run.smithi082:> /usr/sbin/getenforce 2024-03-28T22:14:27.344 INFO:teuthology.orchestra.run.smithi082.stdout:Permissive 2024-03-28T22:14:27.344 DEBUG:teuthology.orchestra.run.smithi172:> /usr/sbin/getenforce 2024-03-28T22:14:27.362 INFO:teuthology.orchestra.run.smithi172.stdout:Permissive 2024-03-28T22:14:27.362 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi043.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi082.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi172.front.sepia.ceph.com': 'permissive'} 2024-03-28T22:14:27.363 DEBUG:teuthology.orchestra.run.smithi043:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T22:14:27.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T22:14:27.495 DEBUG:teuthology.orchestra.run.smithi082:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T22:14:27.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T22:14:27.524 DEBUG:teuthology.orchestra.run.smithi172:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T22:14:27.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T22:14:27.551 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-28T22:14:27.552 DEBUG:teuthology.orchestra.run.smithi043:> sudo /usr/sbin/setenforce permissive 2024-03-28T22:14:27.696 DEBUG:teuthology.orchestra.run.smithi082:> sudo /usr/sbin/setenforce permissive 2024-03-28T22:14:27.737 DEBUG:teuthology.orchestra.run.smithi172:> sudo /usr/sbin/setenforce permissive 2024-03-28T22:14:27.773 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-28T22:14:27.783 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-28T22:14:27.799 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-28T22:14:27.800 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi043.front.sepia.ceph.com,smithi082.front.sepia.ceph.com,smithi172.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-28T22:20:33.482 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi043.front.sepia.ceph.com'), Remote(name='ubuntu@smithi082.front.sepia.ceph.com'), Remote(name='ubuntu@smithi172.front.sepia.ceph.com')] 2024-03-28T22:20:33.484 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-03-28T22:20:33.486 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T22:20:33.637 DEBUG:teuthology.orchestra.run.smithi043:> true 2024-03-28T22:20:33.799 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-03-28T22:20:33.800 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-03-28T22:20:33.800 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T22:20:33.890 DEBUG:teuthology.orchestra.run.smithi082:> true 2024-03-28T22:20:33.957 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-03-28T22:20:33.958 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-28T22:20:33.958 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T22:20:34.047 DEBUG:teuthology.orchestra.run.smithi172:> true 2024-03-28T22:20:34.117 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-28T22:20:34.118 INFO:teuthology.run_tasks:Running task clock... 2024-03-28T22:20:34.128 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-28T22:20:34.129 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T22:20:34.129 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T22:20:34.136 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T22:20:34.137 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T22:20:34.139 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T22:20:34.139 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T22:20:34.170 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T22:20:34.188 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T22:20:34.189 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T22:20:34.206 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T22:20:34.217 INFO:teuthology.orchestra.run.smithi082.stderr:sudo: ntpd: command not found 2024-03-28T22:20:34.231 INFO:teuthology.orchestra.run.smithi082.stdout:506 Cannot talk to daemon 2024-03-28T22:20:34.236 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: ntpd: command not found 2024-03-28T22:20:34.240 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T22:20:34.248 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T22:20:34.249 INFO:teuthology.orchestra.run.smithi172.stdout:506 Cannot talk to daemon 2024-03-28T22:20:34.264 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T22:20:34.266 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T22:20:34.282 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T22:20:34.298 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T22:20:34.308 INFO:teuthology.orchestra.run.smithi082.stderr:bash: line 1: ntpq: command not found 2024-03-28T22:20:34.312 INFO:teuthology.orchestra.run.smithi082.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T22:20:34.312 INFO:teuthology.orchestra.run.smithi082.stdout:=============================================================================== 2024-03-28T22:20:34.312 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T22:20:34.312 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T22:20:34.312 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T22:20:34.312 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T22:20:34.326 INFO:teuthology.orchestra.run.smithi172.stderr:bash: line 1: ntpq: command not found 2024-03-28T22:20:34.329 INFO:teuthology.orchestra.run.smithi172.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T22:20:34.329 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================== 2024-03-28T22:20:34.329 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T22:20:34.329 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T22:20:34.329 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T22:20:34.329 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T22:20:34.404 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: ntpd: command not found 2024-03-28T22:20:34.451 INFO:teuthology.orchestra.run.smithi043.stdout:506 Cannot talk to daemon 2024-03-28T22:20:34.507 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T22:20:34.564 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T22:20:34.699 INFO:teuthology.orchestra.run.smithi043.stderr:bash: line 1: ntpq: command not found 2024-03-28T22:20:34.708 INFO:teuthology.orchestra.run.smithi043.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T22:20:34.708 INFO:teuthology.orchestra.run.smithi043.stdout:=============================================================================== 2024-03-28T22:20:34.708 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T22:20:34.708 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T22:20:34.708 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T22:20:34.708 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T22:20:34.710 INFO:teuthology.run_tasks:Running task install... 2024-03-28T22:20:34.719 DEBUG:teuthology.task.install:project ceph 2024-03-28T22:20:34.719 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '323d23dc228bbcb7f16f188fdc9c0dcb801844be'}} 2024-03-28T22:20:34.720 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '323d23dc228bbcb7f16f188fdc9c0dcb801844be'} 2024-03-28T22:20:34.720 INFO:teuthology.task.install:Using flavor: default 2024-03-28T22:20:34.726 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-28T22:20:34.726 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-28T22:20:34.727 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '323d23dc228bbcb7f16f188fdc9c0dcb801844be', 'tag': None, 'wait_for_package': False} 2024-03-28T22:20:34.727 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=323d23dc228bbcb7f16f188fdc9c0dcb801844be 2024-03-28T22:20:34.729 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '323d23dc228bbcb7f16f188fdc9c0dcb801844be', 'tag': None, 'wait_for_package': False} 2024-03-28T22:20:34.730 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=323d23dc228bbcb7f16f188fdc9c0dcb801844be 2024-03-28T22:20:34.731 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '323d23dc228bbcb7f16f188fdc9c0dcb801844be', 'tag': None, 'wait_for_package': False} 2024-03-28T22:20:34.731 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=323d23dc228bbcb7f16f188fdc9c0dcb801844be 2024-03-28T22:20:34.872 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/ 2024-03-28T22:20:34.873 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1899.g323d23dc 2024-03-28T22:20:34.944 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/ 2024-03-28T22:20:34.944 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1899.g323d23dc 2024-03-28T22:20:34.948 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/ 2024-03-28T22:20:34.948 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1899.g323d23dc 2024-03-28T22:20:35.047 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T22:20:35.047 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-28T22:20:35.047 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T22:20:35.060 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T22:20:35.060 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-28T22:20:35.060 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T22:20:35.078 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T22:20:35.078 DEBUG:teuthology.orchestra.run.smithi172:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/323d23dc228bbcb7f16f188fdc9c0dcb801844be/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T22:20:35.090 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T22:20:35.090 DEBUG:teuthology.orchestra.run.smithi082:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/323d23dc228bbcb7f16f188fdc9c0dcb801844be/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T22:20:35.115 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T22:20:35.115 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-28T22:20:35.115 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T22:20:35.148 DEBUG:teuthology.orchestra.run.smithi172:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T22:20:35.160 DEBUG:teuthology.orchestra.run.smithi082:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T22:20:35.222 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T22:20:35.223 DEBUG:teuthology.orchestra.run.smithi043:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/323d23dc228bbcb7f16f188fdc9c0dcb801844be/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T22:20:35.230 DEBUG:teuthology.orchestra.run.smithi172:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T22:20:35.241 DEBUG:teuthology.orchestra.run.smithi082:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T22:20:35.297 INFO:teuthology.orchestra.run.smithi172.stdout:check_obsoletes = 1 2024-03-28T22:20:35.299 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum clean all 2024-03-28T22:20:35.309 INFO:teuthology.orchestra.run.smithi082.stdout:check_obsoletes = 1 2024-03-28T22:20:35.310 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum clean all 2024-03-28T22:20:35.364 DEBUG:teuthology.orchestra.run.smithi043:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T22:20:35.509 DEBUG:teuthology.orchestra.run.smithi043:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T22:20:35.537 INFO:teuthology.orchestra.run.smithi172.stdout:74 files removed 2024-03-28T22:20:35.544 INFO:teuthology.orchestra.run.smithi082.stdout:74 files removed 2024-03-28T22:20:35.564 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T22:20:35.571 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T22:20:35.607 INFO:teuthology.orchestra.run.smithi043.stdout:check_obsoletes = 1 2024-03-28T22:20:35.615 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum clean all 2024-03-28T22:20:35.994 INFO:teuthology.orchestra.run.smithi043.stdout:74 files removed 2024-03-28T22:20:36.034 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T22:20:36.200 INFO:teuthology.orchestra.run.smithi172.stdout:ceph packages for x86_64 242 kB/s | 83 kB 00:00 2024-03-28T22:20:36.339 INFO:teuthology.orchestra.run.smithi082.stdout:ceph packages for x86_64 177 kB/s | 83 kB 00:00 2024-03-28T22:20:36.646 INFO:teuthology.orchestra.run.smithi172.stdout:ceph noarch packages 29 kB/s | 11 kB 00:00 2024-03-28T22:20:36.731 INFO:teuthology.orchestra.run.smithi082.stdout:ceph noarch packages 33 kB/s | 11 kB 00:00 2024-03-28T22:20:37.077 INFO:teuthology.orchestra.run.smithi172.stdout:ceph source packages 4.7 kB/s | 1.9 kB 00:00 2024-03-28T22:20:37.164 INFO:teuthology.orchestra.run.smithi043.stdout:ceph packages for x86_64 137 kB/s | 83 kB 00:00 2024-03-28T22:20:37.173 INFO:teuthology.orchestra.run.smithi082.stdout:ceph source packages 4.6 kB/s | 1.9 kB 00:00 2024-03-28T22:20:37.765 INFO:teuthology.orchestra.run.smithi043.stdout:ceph noarch packages 23 kB/s | 11 kB 00:00 2024-03-28T22:20:37.813 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-9 - AppStream 9.5 MB/s | 6.7 MB 00:00 2024-03-28T22:20:37.909 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS-9 - AppStream 9.5 MB/s | 6.7 MB 00:00 2024-03-28T22:20:38.315 INFO:teuthology.orchestra.run.smithi043.stdout:ceph source packages 4.0 kB/s | 1.9 kB 00:00 2024-03-28T22:20:39.243 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS-9 - AppStream 7.9 MB/s | 6.7 MB 00:00 2024-03-28T22:20:39.513 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-9 - BaseOS 3.3 MB/s | 2.0 MB 00:00 2024-03-28T22:20:39.534 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-28T22:20:41.219 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS-9 - BaseOS 3.0 MB/s | 2.0 MB 00:00 2024-03-28T22:20:41.983 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - BaseOS 3.8 MB/s | 8.0 MB 00:02 2024-03-28T22:20:42.037 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - BaseOS 3.7 MB/s | 8.0 MB 00:02 2024-03-28T22:20:43.340 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.0 MB 00:01 2024-03-28T22:20:44.827 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-03-28T22:20:45.171 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - AppStream 9.4 MB/s | 19 MB 00:02 2024-03-28T22:20:48.751 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-28T22:20:49.163 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-28T22:20:50.143 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 16 kB 00:00 2024-03-28T22:20:50.757 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 16 kB 00:00 2024-03-28T22:20:52.158 INFO:teuthology.orchestra.run.smithi172.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-28T22:20:52.927 INFO:teuthology.orchestra.run.smithi082.stdout:Extra Packages for Enterprise Linux 7.8 MB/s | 21 MB 00:02 2024-03-28T22:20:57.099 INFO:teuthology.orchestra.run.smithi172.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-28T22:20:57.810 INFO:teuthology.orchestra.run.smithi082.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-03-28T22:20:59.630 INFO:teuthology.orchestra.run.smithi172.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T22:20:59.630 INFO:teuthology.orchestra.run.smithi172.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T22:20:59.645 INFO:teuthology.orchestra.run.smithi172.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T22:20:59.646 INFO:teuthology.orchestra.run.smithi172.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T22:20:59.647 INFO:teuthology.orchestra.run.smithi172.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T22:20:59.648 INFO:teuthology.orchestra.run.smithi172.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T22:20:59.652 INFO:teuthology.orchestra.run.smithi172.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T22:20:59.655 INFO:teuthology.orchestra.run.smithi172.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T22:20:59.656 INFO:teuthology.orchestra.run.smithi172.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T22:20:59.738 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-28T22:20:59.763 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================================ 2024-03-28T22:20:59.763 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-03-28T22:20:59.763 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================================ 2024-03-28T22:20:59.763 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-28T22:20:59.763 INFO:teuthology.orchestra.run.smithi172.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T22:20:59.763 INFO:teuthology.orchestra.run.smithi172.stdout: ceph x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 6.6 k 2024-03-28T22:20:59.763 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 5.3 M 2024-03-28T22:20:59.763 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-fuse x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 889 k 2024-03-28T22:20:59.763 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 145 k 2024-03-28T22:20:59.764 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 1.5 M 2024-03-28T22:20:59.764 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 138 k 2024-03-28T22:20:59.764 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 3.5 M 2024-03-28T22:20:59.764 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 7.4 M 2024-03-28T22:20:59.764 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-rook noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 49 k 2024-03-28T22:20:59.764 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-radosgw x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 9.8 M 2024-03-28T22:20:59.764 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-test x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 47 M 2024-03-28T22:20:59.764 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-volume noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 264 k 2024-03-28T22:20:59.764 INFO:teuthology.orchestra.run.smithi172.stdout: cephadm noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 748 k 2024-03-28T22:20:59.764 INFO:teuthology.orchestra.run.smithi172.stdout: cephfs-mirror x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 221 k 2024-03-28T22:20:59.764 INFO:teuthology.orchestra.run.smithi172.stdout: cephfs-shell x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 27 k 2024-03-28T22:20:59.764 INFO:teuthology.orchestra.run.smithi172.stdout: cephfs-top noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 18 k 2024-03-28T22:20:59.764 INFO:teuthology.orchestra.run.smithi172.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T22:20:59.764 INFO:teuthology.orchestra.run.smithi172.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T22:20:59.765 INFO:teuthology.orchestra.run.smithi172.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T22:20:59.765 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-devel x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 32 k 2024-03-28T22:20:59.765 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs2 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 746 k 2024-03-28T22:20:59.765 INFO:teuthology.orchestra.run.smithi172.stdout: librados-devel x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 127 k 2024-03-28T22:20:59.765 INFO:teuthology.orchestra.run.smithi172.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T22:20:59.765 INFO:teuthology.orchestra.run.smithi172.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T22:20:59.765 INFO:teuthology.orchestra.run.smithi172.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T22:20:59.765 INFO:teuthology.orchestra.run.smithi172.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-03-28T22:20:59.765 INFO:teuthology.orchestra.run.smithi172.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-03-28T22:20:59.765 INFO:teuthology.orchestra.run.smithi172.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-03-28T22:20:59.765 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cephfs x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 162 k 2024-03-28T22:20:59.765 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rados x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 321 k 2024-03-28T22:20:59.765 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rbd x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 299 k 2024-03-28T22:20:59.766 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rgw x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 100 k 2024-03-28T22:20:59.766 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-fuse x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 86 k 2024-03-28T22:20:59.766 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-mirror x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 3.1 M 2024-03-28T22:20:59.766 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-nbd x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 171 k 2024-03-28T22:20:59.766 INFO:teuthology.orchestra.run.smithi172.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T22:20:59.766 INFO:teuthology.orchestra.run.smithi172.stdout:Upgrading: 2024-03-28T22:20:59.766 INFO:teuthology.orchestra.run.smithi172.stdout: NetworkManager x86_64 1:1.47.2-1.el9 CentOS-BaseOS 2.3 M 2024-03-28T22:20:59.766 INFO:teuthology.orchestra.run.smithi172.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 CentOS-BaseOS 1.8 M 2024-03-28T22:20:59.766 INFO:teuthology.orchestra.run.smithi172.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 CentOS-BaseOS 41 k 2024-03-28T22:20:59.766 INFO:teuthology.orchestra.run.smithi172.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 CentOS-BaseOS 247 k 2024-03-28T22:20:59.766 INFO:teuthology.orchestra.run.smithi172.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T22:20:59.766 INFO:teuthology.orchestra.run.smithi172.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T22:20:59.766 INFO:teuthology.orchestra.run.smithi172.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T22:20:59.767 INFO:teuthology.orchestra.run.smithi172.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T22:20:59.767 INFO:teuthology.orchestra.run.smithi172.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T22:20:59.767 INFO:teuthology.orchestra.run.smithi172.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T22:20:59.767 INFO:teuthology.orchestra.run.smithi172.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T22:20:59.767 INFO:teuthology.orchestra.run.smithi172.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T22:20:59.767 INFO:teuthology.orchestra.run.smithi172.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T22:20:59.767 INFO:teuthology.orchestra.run.smithi172.stdout: librados2 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 3.4 M 2024-03-28T22:20:59.767 INFO:teuthology.orchestra.run.smithi172.stdout: librbd1 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 3.1 M 2024-03-28T22:20:59.767 INFO:teuthology.orchestra.run.smithi172.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T22:20:59.767 INFO:teuthology.orchestra.run.smithi172.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T22:20:59.767 INFO:teuthology.orchestra.run.smithi172.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T22:20:59.767 INFO:teuthology.orchestra.run.smithi172.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T22:20:59.768 INFO:teuthology.orchestra.run.smithi172.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T22:20:59.768 INFO:teuthology.orchestra.run.smithi172.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T22:20:59.768 INFO:teuthology.orchestra.run.smithi172.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T22:20:59.768 INFO:teuthology.orchestra.run.smithi172.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T22:20:59.768 INFO:teuthology.orchestra.run.smithi172.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T22:20:59.768 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-03-28T22:20:59.768 INFO:teuthology.orchestra.run.smithi172.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T22:20:59.768 INFO:teuthology.orchestra.run.smithi172.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T22:20:59.768 INFO:teuthology.orchestra.run.smithi172.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T22:20:59.768 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-common x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 21 M 2024-03-28T22:20:59.768 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 31 k 2024-03-28T22:20:59.768 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mds x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 2.4 M 2024-03-28T22:20:59.769 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 248 k 2024-03-28T22:20:59.769 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mon x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 4.7 M 2024-03-28T22:20:59.769 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-osd x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 17 M 2024-03-28T22:20:59.769 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 15 k 2024-03-28T22:20:59.769 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-selinux x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 25 k 2024-03-28T22:20:59.769 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T22:20:59.769 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T22:20:59.769 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T22:20:59.769 INFO:teuthology.orchestra.run.smithi172.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T22:20:59.770 INFO:teuthology.orchestra.run.smithi172.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T22:20:59.770 INFO:teuthology.orchestra.run.smithi172.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T22:20:59.770 INFO:teuthology.orchestra.run.smithi172.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T22:20:59.770 INFO:teuthology.orchestra.run.smithi172.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T22:20:59.770 INFO:teuthology.orchestra.run.smithi172.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T22:20:59.770 INFO:teuthology.orchestra.run.smithi172.stdout: libcephsqlite x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 163 k 2024-03-28T22:20:59.770 INFO:teuthology.orchestra.run.smithi172.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T22:20:59.770 INFO:teuthology.orchestra.run.smithi172.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T22:20:59.770 INFO:teuthology.orchestra.run.smithi172.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T22:20:59.770 INFO:teuthology.orchestra.run.smithi172.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T22:20:59.770 INFO:teuthology.orchestra.run.smithi172.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T22:20:59.770 INFO:teuthology.orchestra.run.smithi172.stdout: libradosstriper1 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 501 k 2024-03-28T22:20:59.771 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T22:20:59.771 INFO:teuthology.orchestra.run.smithi172.stdout: librgw2 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 5.1 M 2024-03-28T22:20:59.771 INFO:teuthology.orchestra.run.smithi172.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T22:20:59.771 INFO:teuthology.orchestra.run.smithi172.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T22:20:59.771 INFO:teuthology.orchestra.run.smithi172.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T22:20:59.771 INFO:teuthology.orchestra.run.smithi172.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T22:20:59.771 INFO:teuthology.orchestra.run.smithi172.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T22:20:59.771 INFO:teuthology.orchestra.run.smithi172.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T22:20:59.771 INFO:teuthology.orchestra.run.smithi172.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T22:20:59.771 INFO:teuthology.orchestra.run.smithi172.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T22:20:59.771 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T22:20:59.771 INFO:teuthology.orchestra.run.smithi172.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T22:20:59.771 INFO:teuthology.orchestra.run.smithi172.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-03-28T22:20:59.771 INFO:teuthology.orchestra.run.smithi172.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T22:20:59.772 INFO:teuthology.orchestra.run.smithi172.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T22:20:59.772 INFO:teuthology.orchestra.run.smithi172.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T22:20:59.772 INFO:teuthology.orchestra.run.smithi172.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T22:20:59.772 INFO:teuthology.orchestra.run.smithi172.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T22:20:59.772 INFO:teuthology.orchestra.run.smithi172.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T22:20:59.772 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T22:20:59.772 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 45 k 2024-03-28T22:20:59.772 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 131 k 2024-03-28T22:20:59.772 INFO:teuthology.orchestra.run.smithi172.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T22:20:59.772 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T22:20:59.772 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T22:20:59.772 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T22:20:59.772 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T22:20:59.772 INFO:teuthology.orchestra.run.smithi172.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T22:20:59.772 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T22:20:59.773 INFO:teuthology.orchestra.run.smithi172.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T22:20:59.773 INFO:teuthology.orchestra.run.smithi172.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T22:20:59.773 INFO:teuthology.orchestra.run.smithi172.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T22:20:59.773 INFO:teuthology.orchestra.run.smithi172.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T22:20:59.773 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T22:20:59.773 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T22:20:59.773 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T22:20:59.773 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T22:20:59.773 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T22:20:59.774 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T22:20:59.774 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T22:20:59.774 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T22:20:59.774 INFO:teuthology.orchestra.run.smithi172.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T22:20:59.774 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T22:20:59.774 INFO:teuthology.orchestra.run.smithi172.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T22:20:59.774 INFO:teuthology.orchestra.run.smithi172.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T22:20:59.774 INFO:teuthology.orchestra.run.smithi172.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T22:20:59.774 INFO:teuthology.orchestra.run.smithi172.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T22:20:59.774 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T22:20:59.775 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T22:20:59.775 INFO:teuthology.orchestra.run.smithi172.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T22:20:59.775 INFO:teuthology.orchestra.run.smithi172.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T22:20:59.775 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T22:20:59.775 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T22:20:59.775 INFO:teuthology.orchestra.run.smithi172.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T22:20:59.775 INFO:teuthology.orchestra.run.smithi172.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T22:20:59.775 INFO:teuthology.orchestra.run.smithi172.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T22:20:59.775 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T22:20:59.775 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T22:20:59.775 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T22:20:59.775 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T22:20:59.775 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T22:20:59.776 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T22:20:59.776 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T22:20:59.776 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T22:20:59.776 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T22:20:59.776 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T22:20:59.776 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T22:20:59.776 INFO:teuthology.orchestra.run.smithi172.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T22:20:59.776 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T22:20:59.776 INFO:teuthology.orchestra.run.smithi172.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T22:20:59.776 INFO:teuthology.orchestra.run.smithi172.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T22:20:59.776 INFO:teuthology.orchestra.run.smithi172.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T22:20:59.776 INFO:teuthology.orchestra.run.smithi172.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T22:20:59.777 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T22:20:59.777 INFO:teuthology.orchestra.run.smithi172.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T22:20:59.777 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T22:20:59.777 INFO:teuthology.orchestra.run.smithi172.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T22:20:59.777 INFO:teuthology.orchestra.run.smithi172.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-28T22:20:59.777 INFO:teuthology.orchestra.run.smithi172.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-28T22:20:59.777 INFO:teuthology.orchestra.run.smithi172.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T22:20:59.777 INFO:teuthology.orchestra.run.smithi172.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T22:20:59.777 INFO:teuthology.orchestra.run.smithi172.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T22:20:59.777 INFO:teuthology.orchestra.run.smithi172.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T22:20:59.777 INFO:teuthology.orchestra.run.smithi172.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T22:20:59.777 INFO:teuthology.orchestra.run.smithi172.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T22:20:59.777 INFO:teuthology.orchestra.run.smithi172.stdout:Installing weak dependencies: 2024-03-28T22:20:59.778 INFO:teuthology.orchestra.run.smithi172.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-28T22:20:59.778 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-28T22:20:59.778 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-28T22:20:59.778 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================================ 2024-03-28T22:20:59.778 INFO:teuthology.orchestra.run.smithi172.stdout:Install 150 Packages 2024-03-28T22:20:59.778 INFO:teuthology.orchestra.run.smithi172.stdout:Upgrade 24 Packages 2024-03-28T22:20:59.778 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-28T22:20:59.778 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 226 M 2024-03-28T22:20:59.779 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-28T22:21:00.325 INFO:teuthology.orchestra.run.smithi082.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T22:21:00.326 INFO:teuthology.orchestra.run.smithi082.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T22:21:00.336 INFO:teuthology.orchestra.run.smithi082.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T22:21:00.337 INFO:teuthology.orchestra.run.smithi082.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T22:21:00.338 INFO:teuthology.orchestra.run.smithi082.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T22:21:00.339 INFO:teuthology.orchestra.run.smithi082.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T22:21:00.340 INFO:teuthology.orchestra.run.smithi082.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T22:21:00.343 INFO:teuthology.orchestra.run.smithi082.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T22:21:00.344 INFO:teuthology.orchestra.run.smithi082.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T22:21:00.424 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-28T22:21:00.440 INFO:teuthology.orchestra.run.smithi082.stdout:============================================================================================ 2024-03-28T22:21:00.440 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-03-28T22:21:00.440 INFO:teuthology.orchestra.run.smithi082.stdout:============================================================================================ 2024-03-28T22:21:00.440 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-28T22:21:00.440 INFO:teuthology.orchestra.run.smithi082.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T22:21:00.440 INFO:teuthology.orchestra.run.smithi082.stdout: ceph x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 6.6 k 2024-03-28T22:21:00.440 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-base x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 5.3 M 2024-03-28T22:21:00.440 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-fuse x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 889 k 2024-03-28T22:21:00.440 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 145 k 2024-03-28T22:21:00.441 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 1.5 M 2024-03-28T22:21:00.441 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 138 k 2024-03-28T22:21:00.441 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 3.5 M 2024-03-28T22:21:00.441 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 7.4 M 2024-03-28T22:21:00.441 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-rook noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 49 k 2024-03-28T22:21:00.441 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-radosgw x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 9.8 M 2024-03-28T22:21:00.441 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-test x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 47 M 2024-03-28T22:21:00.441 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-volume noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 264 k 2024-03-28T22:21:00.441 INFO:teuthology.orchestra.run.smithi082.stdout: cephadm noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 748 k 2024-03-28T22:21:00.441 INFO:teuthology.orchestra.run.smithi082.stdout: cephfs-mirror x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 221 k 2024-03-28T22:21:00.441 INFO:teuthology.orchestra.run.smithi082.stdout: cephfs-shell x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 27 k 2024-03-28T22:21:00.441 INFO:teuthology.orchestra.run.smithi082.stdout: cephfs-top noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 18 k 2024-03-28T22:21:00.441 INFO:teuthology.orchestra.run.smithi082.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T22:21:00.441 INFO:teuthology.orchestra.run.smithi082.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T22:21:00.441 INFO:teuthology.orchestra.run.smithi082.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T22:21:00.442 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs-devel x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 32 k 2024-03-28T22:21:00.442 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs2 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 746 k 2024-03-28T22:21:00.442 INFO:teuthology.orchestra.run.smithi082.stdout: librados-devel x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 127 k 2024-03-28T22:21:00.442 INFO:teuthology.orchestra.run.smithi082.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T22:21:00.442 INFO:teuthology.orchestra.run.smithi082.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T22:21:00.442 INFO:teuthology.orchestra.run.smithi082.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T22:21:00.442 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-03-28T22:21:00.442 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-03-28T22:21:00.442 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-03-28T22:21:00.442 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cephfs x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 162 k 2024-03-28T22:21:00.442 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rados x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 321 k 2024-03-28T22:21:00.442 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rbd x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 299 k 2024-03-28T22:21:00.442 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rgw x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 100 k 2024-03-28T22:21:00.443 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-fuse x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 86 k 2024-03-28T22:21:00.443 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-mirror x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 3.1 M 2024-03-28T22:21:00.443 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-nbd x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 171 k 2024-03-28T22:21:00.443 INFO:teuthology.orchestra.run.smithi082.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T22:21:00.443 INFO:teuthology.orchestra.run.smithi082.stdout:Upgrading: 2024-03-28T22:21:00.443 INFO:teuthology.orchestra.run.smithi082.stdout: NetworkManager x86_64 1:1.47.2-1.el9 CentOS-BaseOS 2.3 M 2024-03-28T22:21:00.443 INFO:teuthology.orchestra.run.smithi082.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 CentOS-BaseOS 1.8 M 2024-03-28T22:21:00.443 INFO:teuthology.orchestra.run.smithi082.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 CentOS-BaseOS 41 k 2024-03-28T22:21:00.443 INFO:teuthology.orchestra.run.smithi082.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 CentOS-BaseOS 247 k 2024-03-28T22:21:00.443 INFO:teuthology.orchestra.run.smithi082.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T22:21:00.443 INFO:teuthology.orchestra.run.smithi082.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T22:21:00.443 INFO:teuthology.orchestra.run.smithi082.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T22:21:00.443 INFO:teuthology.orchestra.run.smithi082.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T22:21:00.443 INFO:teuthology.orchestra.run.smithi082.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T22:21:00.443 INFO:teuthology.orchestra.run.smithi082.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T22:21:00.444 INFO:teuthology.orchestra.run.smithi082.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T22:21:00.444 INFO:teuthology.orchestra.run.smithi082.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T22:21:00.444 INFO:teuthology.orchestra.run.smithi082.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T22:21:00.444 INFO:teuthology.orchestra.run.smithi082.stdout: librados2 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 3.4 M 2024-03-28T22:21:00.444 INFO:teuthology.orchestra.run.smithi082.stdout: librbd1 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 3.1 M 2024-03-28T22:21:00.444 INFO:teuthology.orchestra.run.smithi082.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T22:21:00.444 INFO:teuthology.orchestra.run.smithi082.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T22:21:00.444 INFO:teuthology.orchestra.run.smithi082.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T22:21:00.444 INFO:teuthology.orchestra.run.smithi082.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T22:21:00.444 INFO:teuthology.orchestra.run.smithi082.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T22:21:00.444 INFO:teuthology.orchestra.run.smithi082.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T22:21:00.444 INFO:teuthology.orchestra.run.smithi082.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T22:21:00.444 INFO:teuthology.orchestra.run.smithi082.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T22:21:00.444 INFO:teuthology.orchestra.run.smithi082.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T22:21:00.445 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-03-28T22:21:00.445 INFO:teuthology.orchestra.run.smithi082.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T22:21:00.445 INFO:teuthology.orchestra.run.smithi082.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T22:21:00.445 INFO:teuthology.orchestra.run.smithi082.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T22:21:00.445 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-common x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 21 M 2024-03-28T22:21:00.445 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 31 k 2024-03-28T22:21:00.445 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mds x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 2.4 M 2024-03-28T22:21:00.445 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 248 k 2024-03-28T22:21:00.445 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mon x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 4.7 M 2024-03-28T22:21:00.445 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-osd x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 17 M 2024-03-28T22:21:00.445 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 15 k 2024-03-28T22:21:00.445 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-selinux x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 25 k 2024-03-28T22:21:00.445 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T22:21:00.445 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T22:21:00.446 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T22:21:00.446 INFO:teuthology.orchestra.run.smithi082.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T22:21:00.446 INFO:teuthology.orchestra.run.smithi082.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T22:21:00.446 INFO:teuthology.orchestra.run.smithi082.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T22:21:00.446 INFO:teuthology.orchestra.run.smithi082.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T22:21:00.446 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T22:21:00.446 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T22:21:00.446 INFO:teuthology.orchestra.run.smithi082.stdout: libcephsqlite x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 163 k 2024-03-28T22:21:00.446 INFO:teuthology.orchestra.run.smithi082.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T22:21:00.446 INFO:teuthology.orchestra.run.smithi082.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T22:21:00.446 INFO:teuthology.orchestra.run.smithi082.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T22:21:00.446 INFO:teuthology.orchestra.run.smithi082.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T22:21:00.446 INFO:teuthology.orchestra.run.smithi082.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T22:21:00.446 INFO:teuthology.orchestra.run.smithi082.stdout: libradosstriper1 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 501 k 2024-03-28T22:21:00.447 INFO:teuthology.orchestra.run.smithi082.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T22:21:00.447 INFO:teuthology.orchestra.run.smithi082.stdout: librgw2 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 5.1 M 2024-03-28T22:21:00.447 INFO:teuthology.orchestra.run.smithi082.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T22:21:00.447 INFO:teuthology.orchestra.run.smithi082.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T22:21:00.447 INFO:teuthology.orchestra.run.smithi082.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T22:21:00.447 INFO:teuthology.orchestra.run.smithi082.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T22:21:00.447 INFO:teuthology.orchestra.run.smithi082.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T22:21:00.447 INFO:teuthology.orchestra.run.smithi082.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T22:21:00.447 INFO:teuthology.orchestra.run.smithi082.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T22:21:00.447 INFO:teuthology.orchestra.run.smithi082.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T22:21:00.447 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T22:21:00.447 INFO:teuthology.orchestra.run.smithi082.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T22:21:00.447 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-03-28T22:21:00.447 INFO:teuthology.orchestra.run.smithi082.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T22:21:00.448 INFO:teuthology.orchestra.run.smithi082.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T22:21:00.448 INFO:teuthology.orchestra.run.smithi082.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T22:21:00.448 INFO:teuthology.orchestra.run.smithi082.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T22:21:00.448 INFO:teuthology.orchestra.run.smithi082.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T22:21:00.448 INFO:teuthology.orchestra.run.smithi082.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T22:21:00.448 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T22:21:00.448 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-argparse x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 45 k 2024-03-28T22:21:00.448 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-common x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 131 k 2024-03-28T22:21:00.448 INFO:teuthology.orchestra.run.smithi082.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T22:21:00.448 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T22:21:00.448 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T22:21:00.448 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T22:21:00.448 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T22:21:00.448 INFO:teuthology.orchestra.run.smithi082.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T22:21:00.448 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T22:21:00.449 INFO:teuthology.orchestra.run.smithi082.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T22:21:00.449 INFO:teuthology.orchestra.run.smithi082.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T22:21:00.449 INFO:teuthology.orchestra.run.smithi082.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T22:21:00.449 INFO:teuthology.orchestra.run.smithi082.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T22:21:00.449 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T22:21:00.449 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T22:21:00.449 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T22:21:00.449 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T22:21:00.449 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T22:21:00.449 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T22:21:00.449 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T22:21:00.449 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T22:21:00.449 INFO:teuthology.orchestra.run.smithi082.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T22:21:00.449 INFO:teuthology.orchestra.run.smithi082.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T22:21:00.450 INFO:teuthology.orchestra.run.smithi082.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T22:21:00.450 INFO:teuthology.orchestra.run.smithi082.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T22:21:00.450 INFO:teuthology.orchestra.run.smithi082.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T22:21:00.450 INFO:teuthology.orchestra.run.smithi082.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T22:21:00.450 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T22:21:00.450 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T22:21:00.450 INFO:teuthology.orchestra.run.smithi082.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T22:21:00.450 INFO:teuthology.orchestra.run.smithi082.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T22:21:00.450 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T22:21:00.450 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T22:21:00.450 INFO:teuthology.orchestra.run.smithi082.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T22:21:00.450 INFO:teuthology.orchestra.run.smithi082.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T22:21:00.450 INFO:teuthology.orchestra.run.smithi082.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T22:21:00.450 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T22:21:00.451 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T22:21:00.451 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T22:21:00.451 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T22:21:00.451 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T22:21:00.451 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T22:21:00.451 INFO:teuthology.orchestra.run.smithi082.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T22:21:00.451 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T22:21:00.451 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T22:21:00.451 INFO:teuthology.orchestra.run.smithi082.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T22:21:00.451 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T22:21:00.451 INFO:teuthology.orchestra.run.smithi082.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T22:21:00.451 INFO:teuthology.orchestra.run.smithi082.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T22:21:00.451 INFO:teuthology.orchestra.run.smithi082.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T22:21:00.451 INFO:teuthology.orchestra.run.smithi082.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T22:21:00.451 INFO:teuthology.orchestra.run.smithi082.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T22:21:00.452 INFO:teuthology.orchestra.run.smithi082.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T22:21:00.452 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T22:21:00.452 INFO:teuthology.orchestra.run.smithi082.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T22:21:00.452 INFO:teuthology.orchestra.run.smithi082.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T22:21:00.452 INFO:teuthology.orchestra.run.smithi082.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T22:21:00.452 INFO:teuthology.orchestra.run.smithi082.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-28T22:21:00.452 INFO:teuthology.orchestra.run.smithi082.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-28T22:21:00.452 INFO:teuthology.orchestra.run.smithi082.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T22:21:00.452 INFO:teuthology.orchestra.run.smithi082.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T22:21:00.452 INFO:teuthology.orchestra.run.smithi082.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T22:21:00.452 INFO:teuthology.orchestra.run.smithi082.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T22:21:00.452 INFO:teuthology.orchestra.run.smithi082.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T22:21:00.452 INFO:teuthology.orchestra.run.smithi082.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T22:21:00.452 INFO:teuthology.orchestra.run.smithi082.stdout:Installing weak dependencies: 2024-03-28T22:21:00.452 INFO:teuthology.orchestra.run.smithi082.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-28T22:21:00.453 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-28T22:21:00.453 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-28T22:21:00.453 INFO:teuthology.orchestra.run.smithi082.stdout:============================================================================================ 2024-03-28T22:21:00.453 INFO:teuthology.orchestra.run.smithi082.stdout:Install 150 Packages 2024-03-28T22:21:00.453 INFO:teuthology.orchestra.run.smithi082.stdout:Upgrade 24 Packages 2024-03-28T22:21:00.453 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-28T22:21:00.453 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 226 M 2024-03-28T22:21:00.453 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-28T22:21:01.116 INFO:teuthology.orchestra.run.smithi172.stdout:(1/174): ceph-19.0.0-1899.g323d23dc.el9.x86_64. 22 kB/s | 6.6 kB 00:00 2024-03-28T22:21:01.392 INFO:teuthology.orchestra.run.smithi172.stdout:(2/174): ceph-base-19.0.0-1899.g323d23dc.el9.x8 9.4 MB/s | 5.3 MB 00:00 2024-03-28T22:21:01.442 INFO:teuthology.orchestra.run.smithi172.stdout:(3/174): ceph-immutable-object-cache-19.0.0-189 2.8 MB/s | 145 kB 00:00 2024-03-28T22:21:01.444 INFO:teuthology.orchestra.run.smithi082.stdout:(1/174): ceph-19.0.0-1899.g323d23dc.el9.x86_64. 36 kB/s | 6.6 kB 00:00 2024-03-28T22:21:01.467 INFO:teuthology.orchestra.run.smithi172.stdout:(4/174): ceph-fuse-19.0.0-1899.g323d23dc.el9.x8 2.5 MB/s | 889 kB 00:00 2024-03-28T22:21:01.651 INFO:teuthology.orchestra.run.smithi172.stdout:(5/174): ceph-mds-19.0.0-1899.g323d23dc.el9.x86 11 MB/s | 2.4 MB 00:00 2024-03-28T22:21:01.909 INFO:teuthology.orchestra.run.smithi172.stdout:(6/174): ceph-mgr-19.0.0-1899.g323d23dc.el9.x86 3.4 MB/s | 1.5 MB 00:00 2024-03-28T22:21:02.044 INFO:teuthology.orchestra.run.smithi082.stdout:(2/174): ceph-fuse-19.0.0-1899.g323d23dc.el9.x8 1.4 MB/s | 889 kB 00:00 2024-03-28T22:21:02.068 INFO:teuthology.orchestra.run.smithi172.stdout:(7/174): ceph-mon-19.0.0-1899.g323d23dc.el9.x86 11 MB/s | 4.7 MB 00:00 2024-03-28T22:21:02.111 INFO:teuthology.orchestra.run.smithi082.stdout:(3/174): ceph-immutable-object-cache-19.0.0-189 2.1 MB/s | 145 kB 00:00 2024-03-28T22:21:03.035 INFO:teuthology.orchestra.run.smithi172.stdout:(8/174): ceph-radosgw-19.0.0-1899.g323d23dc.el9 10 MB/s | 9.8 MB 00:00 2024-03-28T22:21:03.069 INFO:teuthology.orchestra.run.smithi082.stdout:(4/174): ceph-mds-19.0.0-1899.g323d23dc.el9.x86 2.5 MB/s | 2.4 MB 00:00 2024-03-28T22:21:03.085 INFO:teuthology.orchestra.run.smithi172.stdout:(9/174): ceph-selinux-19.0.0-1899.g323d23dc.el9 506 kB/s | 25 kB 00:00 2024-03-28T22:21:03.494 INFO:teuthology.orchestra.run.smithi082.stdout:(5/174): ceph-mgr-19.0.0-1899.g323d23dc.el9.x86 3.5 MB/s | 1.5 MB 00:00 2024-03-28T22:21:03.635 INFO:teuthology.orchestra.run.smithi172.stdout:(10/174): ceph-common-19.0.0-1899.g323d23dc.el9 7.3 MB/s | 21 MB 00:02 2024-03-28T22:21:03.744 INFO:teuthology.orchestra.run.smithi172.stdout:(11/174): cephfs-mirror-19.0.0-1899.g323d23dc.e 2.0 MB/s | 221 kB 00:00 2024-03-28T22:21:03.836 INFO:teuthology.orchestra.run.smithi172.stdout:(12/174): cephfs-shell-19.0.0-1899.g323d23dc.el 294 kB/s | 27 kB 00:00 2024-03-28T22:21:03.928 INFO:teuthology.orchestra.run.smithi172.stdout:(13/174): libcephfs-devel-19.0.0-1899.g323d23dc 346 kB/s | 32 kB 00:00 2024-03-28T22:21:03.978 INFO:teuthology.orchestra.run.smithi082.stdout:(6/174): ceph-base-19.0.0-1899.g323d23dc.el9.x8 2.0 MB/s | 5.3 MB 00:02 2024-03-28T22:21:04.120 INFO:teuthology.orchestra.run.smithi172.stdout:(14/174): libcephfs2-19.0.0-1899.g323d23dc.el9. 3.8 MB/s | 746 kB 00:00 2024-03-28T22:21:04.237 INFO:teuthology.orchestra.run.smithi172.stdout:(15/174): libcephsqlite-19.0.0-1899.g323d23dc.e 1.4 MB/s | 163 kB 00:00 2024-03-28T22:21:04.345 INFO:teuthology.orchestra.run.smithi082.stdout:(7/174): ceph-mon-19.0.0-1899.g323d23dc.el9.x86 5.5 MB/s | 4.7 MB 00:00 2024-03-28T22:21:04.345 INFO:teuthology.orchestra.run.smithi172.stdout:(16/174): librados-devel-19.0.0-1899.g323d23dc. 1.1 MB/s | 127 kB 00:00 2024-03-28T22:21:04.471 INFO:teuthology.orchestra.run.smithi172.stdout:(17/174): libradosstriper1-19.0.0-1899.g323d23d 3.9 MB/s | 501 kB 00:00 2024-03-28T22:21:05.088 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - AppStream 943 kB/s | 19 MB 00:20 2024-03-28T22:21:05.405 INFO:teuthology.orchestra.run.smithi172.stdout:(18/174): librgw2-19.0.0-1899.g323d23dc.el9.x86 5.5 MB/s | 5.1 MB 00:00 2024-03-28T22:21:05.504 INFO:teuthology.orchestra.run.smithi172.stdout:(19/174): python3-ceph-argparse-19.0.0-1899.g32 452 kB/s | 45 kB 00:00 2024-03-28T22:21:05.605 INFO:teuthology.orchestra.run.smithi172.stdout:(20/174): python3-ceph-common-19.0.0-1899.g323d 1.3 MB/s | 131 kB 00:00 2024-03-28T22:21:05.714 INFO:teuthology.orchestra.run.smithi172.stdout:(21/174): python3-cephfs-19.0.0-1899.g323d23dc. 1.5 MB/s | 162 kB 00:00 2024-03-28T22:21:05.830 INFO:teuthology.orchestra.run.smithi172.stdout:(22/174): python3-rados-19.0.0-1899.g323d23dc.e 2.7 MB/s | 321 kB 00:00 2024-03-28T22:21:05.870 INFO:teuthology.orchestra.run.smithi082.stdout:(8/174): ceph-radosgw-19.0.0-1899.g323d23dc.el9 6.4 MB/s | 9.8 MB 00:01 2024-03-28T22:21:05.914 INFO:teuthology.orchestra.run.smithi172.stdout:(23/174): python3-rbd-19.0.0-1899.g323d23dc.el9 3.5 MB/s | 299 kB 00:00 2024-03-28T22:21:05.981 INFO:teuthology.orchestra.run.smithi172.stdout:(24/174): python3-rgw-19.0.0-1899.g323d23dc.el9 1.5 MB/s | 100 kB 00:00 2024-03-28T22:21:06.065 INFO:teuthology.orchestra.run.smithi172.stdout:(25/174): rbd-fuse-19.0.0-1899.g323d23dc.el9.x8 1.0 MB/s | 86 kB 00:00 2024-03-28T22:21:06.104 INFO:teuthology.orchestra.run.smithi082.stdout:(9/174): ceph-common-19.0.0-1899.g323d23dc.el9. 4.3 MB/s | 21 MB 00:04 2024-03-28T22:21:06.128 INFO:teuthology.orchestra.run.smithi082.stdout:(10/174): ceph-selinux-19.0.0-1899.g323d23dc.el 97 kB/s | 25 kB 00:00 2024-03-28T22:21:06.229 INFO:teuthology.orchestra.run.smithi082.stdout:(11/174): cephfs-mirror-19.0.0-1899.g323d23dc.e 2.2 MB/s | 221 kB 00:00 2024-03-28T22:21:06.321 INFO:teuthology.orchestra.run.smithi082.stdout:(12/174): cephfs-shell-19.0.0-1899.g323d23dc.el 295 kB/s | 27 kB 00:00 2024-03-28T22:21:06.421 INFO:teuthology.orchestra.run.smithi082.stdout:(13/174): libcephfs-devel-19.0.0-1899.g323d23dc 317 kB/s | 32 kB 00:00 2024-03-28T22:21:06.573 INFO:teuthology.orchestra.run.smithi172.stdout:(26/174): rbd-mirror-19.0.0-1899.g323d23dc.el9. 6.1 MB/s | 3.1 MB 00:00 2024-03-28T22:21:06.588 INFO:teuthology.orchestra.run.smithi082.stdout:(14/174): libcephfs2-19.0.0-1899.g323d23dc.el9. 4.4 MB/s | 746 kB 00:00 2024-03-28T22:21:06.665 INFO:teuthology.orchestra.run.smithi172.stdout:(27/174): rbd-nbd-19.0.0-1899.g323d23dc.el9.x86 1.8 MB/s | 171 kB 00:00 2024-03-28T22:21:06.749 INFO:teuthology.orchestra.run.smithi172.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1899.g 378 kB/s | 31 kB 00:00 2024-03-28T22:21:06.841 INFO:teuthology.orchestra.run.smithi172.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1899.g323d23d 1.5 MB/s | 138 kB 00:00 2024-03-28T22:21:06.921 INFO:teuthology.orchestra.run.smithi082.stdout:(15/174): libcephsqlite-19.0.0-1899.g323d23dc.e 490 kB/s | 163 kB 00:00 2024-03-28T22:21:07.022 INFO:teuthology.orchestra.run.smithi082.stdout:(16/174): librados-devel-19.0.0-1899.g323d23dc. 1.2 MB/s | 127 kB 00:00 2024-03-28T22:21:07.164 INFO:teuthology.orchestra.run.smithi082.stdout:(17/174): libradosstriper1-19.0.0-1899.g323d23d 3.5 MB/s | 501 kB 00:00 2024-03-28T22:21:07.199 INFO:teuthology.orchestra.run.smithi172.stdout:(30/174): ceph-osd-19.0.0-1899.g323d23dc.el9.x8 3.2 MB/s | 17 MB 00:05 2024-03-28T22:21:07.483 INFO:teuthology.orchestra.run.smithi172.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1899.g323d2 5.5 MB/s | 3.5 MB 00:00 2024-03-28T22:21:07.591 INFO:teuthology.orchestra.run.smithi172.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1899.g32 2.2 MB/s | 248 kB 00:00 2024-03-28T22:21:07.675 INFO:teuthology.orchestra.run.smithi172.stdout:(33/174): ceph-mgr-rook-19.0.0-1899.g323d23dc.e 591 kB/s | 49 kB 00:00 2024-03-28T22:21:07.734 INFO:teuthology.orchestra.run.smithi172.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1899.g3 264 kB/s | 15 kB 00:00 2024-03-28T22:21:07.772 INFO:teuthology.orchestra.run.smithi082.stdout:(18/174): ceph-osd-19.0.0-1899.g323d23dc.el9.x8 4.4 MB/s | 17 MB 00:03 2024-03-28T22:21:07.809 INFO:teuthology.orchestra.run.smithi172.stdout:(35/174): ceph-volume-19.0.0-1899.g323d23dc.el9 3.4 MB/s | 264 kB 00:00 2024-03-28T22:21:07.856 INFO:teuthology.orchestra.run.smithi082.stdout:(19/174): python3-ceph-argparse-19.0.0-1899.g32 542 kB/s | 45 kB 00:00 2024-03-28T22:21:07.956 INFO:teuthology.orchestra.run.smithi082.stdout:(20/174): python3-ceph-common-19.0.0-1899.g323d 1.3 MB/s | 131 kB 00:00 2024-03-28T22:21:07.984 INFO:teuthology.orchestra.run.smithi172.stdout:(36/174): cephadm-19.0.0-1899.g323d23dc.el9.noa 4.2 MB/s | 748 kB 00:00 2024-03-28T22:21:08.056 INFO:teuthology.orchestra.run.smithi082.stdout:(21/174): librgw2-19.0.0-1899.g323d23dc.el9.x86 5.7 MB/s | 5.1 MB 00:00 2024-03-28T22:21:08.059 INFO:teuthology.orchestra.run.smithi172.stdout:(37/174): cephfs-top-19.0.0-1899.g323d23dc.el9. 243 kB/s | 18 kB 00:00 2024-03-28T22:21:08.082 INFO:teuthology.orchestra.run.smithi082.stdout:(22/174): python3-cephfs-19.0.0-1899.g323d23dc. 1.3 MB/s | 162 kB 00:00 2024-03-28T22:21:08.115 INFO:teuthology.orchestra.run.smithi082.stdout:(23/174): python3-rados-19.0.0-1899.g323d23dc.e 5.4 MB/s | 321 kB 00:00 2024-03-28T22:21:08.182 INFO:teuthology.orchestra.run.smithi082.stdout:(24/174): python3-rgw-19.0.0-1899.g323d23dc.el9 1.5 MB/s | 100 kB 00:00 2024-03-28T22:21:08.202 INFO:teuthology.orchestra.run.smithi172.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 4.6 MB/s | 662 kB 00:00 2024-03-28T22:21:08.207 INFO:teuthology.orchestra.run.smithi082.stdout:(25/174): python3-rbd-19.0.0-1899.g323d23dc.el9 2.3 MB/s | 299 kB 00:00 2024-03-28T22:21:08.233 INFO:teuthology.orchestra.run.smithi082.stdout:(26/174): rbd-fuse-19.0.0-1899.g323d23dc.el9.x8 1.7 MB/s | 86 kB 00:00 2024-03-28T22:21:08.260 INFO:teuthology.orchestra.run.smithi172.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 15 MB/s | 920 kB 00:00 2024-03-28T22:21:08.294 INFO:teuthology.orchestra.run.smithi172.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-28T22:21:08.299 INFO:teuthology.orchestra.run.smithi082.stdout:(27/174): rbd-nbd-19.0.0-1899.g323d23dc.el9.x86 2.5 MB/s | 171 kB 00:00 2024-03-28T22:21:08.332 INFO:teuthology.orchestra.run.smithi172.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.8 MB/s | 307 kB 00:00 2024-03-28T22:21:08.366 INFO:teuthology.orchestra.run.smithi172.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 950 kB/s | 32 kB 00:00 2024-03-28T22:21:08.380 INFO:teuthology.orchestra.run.smithi082.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1899.g 389 kB/s | 31 kB 00:00 2024-03-28T22:21:08.450 INFO:teuthology.orchestra.run.smithi082.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1899.g323d23d 1.9 MB/s | 138 kB 00:00 2024-03-28T22:21:08.466 INFO:teuthology.orchestra.run.smithi172.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2024-03-28T22:21:08.492 INFO:teuthology.orchestra.run.smithi172.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 683 kB/s | 17 kB 00:00 2024-03-28T22:21:08.525 INFO:teuthology.orchestra.run.smithi172.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 302 kB/s | 10 kB 00:00 2024-03-28T22:21:08.559 INFO:teuthology.orchestra.run.smithi172.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.7 MB/s | 159 kB 00:00 2024-03-28T22:21:08.593 INFO:teuthology.orchestra.run.smithi172.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-28T22:21:08.643 INFO:teuthology.orchestra.run.smithi172.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-28T22:21:08.685 INFO:teuthology.orchestra.run.smithi172.stdout:(49/174): libtool-2.4.6-45.el9.x86_64.rpm 14 MB/s | 578 kB 00:00 2024-03-28T22:21:08.719 INFO:teuthology.orchestra.run.smithi172.stdout:(50/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-28T22:21:08.761 INFO:teuthology.orchestra.run.smithi172.stdout:(51/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-28T22:21:08.803 INFO:teuthology.orchestra.run.smithi172.stdout:(52/174): lua-5.4.4-4.el9.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2024-03-28T22:21:08.836 INFO:teuthology.orchestra.run.smithi172.stdout:(53/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 6.4 MB/s | 217 kB 00:00 2024-03-28T22:21:08.870 INFO:teuthology.orchestra.run.smithi172.stdout:(54/174): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-28T22:21:08.909 INFO:teuthology.orchestra.run.smithi082.stdout:(30/174): rbd-mirror-19.0.0-1899.g323d23dc.el9. 4.4 MB/s | 3.1 MB 00:00 2024-03-28T22:21:08.967 INFO:teuthology.orchestra.run.smithi082.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1899.g323d2 6.8 MB/s | 3.5 MB 00:00 2024-03-28T22:21:09.004 INFO:teuthology.orchestra.run.smithi172.stdout:(55/174): openblas-openmp-0.3.21-2.el9.x86_64.r 36 MB/s | 4.7 MB 00:00 2024-03-28T22:21:09.042 INFO:teuthology.orchestra.run.smithi082.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1899.g32 3.2 MB/s | 248 kB 00:00 2024-03-28T22:21:09.062 INFO:teuthology.orchestra.run.smithi172.stdout:(56/174): postgresql-13.14-1.el9.x86_64.rpm 26 MB/s | 1.5 MB 00:00 2024-03-28T22:21:09.109 INFO:teuthology.orchestra.run.smithi082.stdout:(33/174): ceph-mgr-rook-19.0.0-1899.g323d23dc.e 740 kB/s | 49 kB 00:00 2024-03-28T22:21:09.112 INFO:teuthology.orchestra.run.smithi172.stdout:(57/174): postgresql-contrib-13.14-1.el9.x86_64 16 MB/s | 817 kB 00:00 2024-03-28T22:21:09.151 INFO:teuthology.orchestra.run.smithi082.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1899.g3 367 kB/s | 15 kB 00:00 2024-03-28T22:21:09.155 INFO:teuthology.orchestra.run.smithi172.stdout:(58/174): postgresql-private-libs-13.14-1.el9.x 3.1 MB/s | 132 kB 00:00 2024-03-28T22:21:09.210 INFO:teuthology.orchestra.run.smithi082.stdout:(35/174): ceph-volume-19.0.0-1899.g323d23dc.el9 4.4 MB/s | 264 kB 00:00 2024-03-28T22:21:09.297 INFO:teuthology.orchestra.run.smithi172.stdout:(59/174): postgresql-server-13.14-1.el9.x86_64. 40 MB/s | 5.7 MB 00:00 2024-03-28T22:21:09.327 INFO:teuthology.orchestra.run.smithi082.stdout:(36/174): cephadm-19.0.0-1899.g323d23dc.el9.noa 6.3 MB/s | 748 kB 00:00 2024-03-28T22:21:09.347 INFO:teuthology.orchestra.run.smithi172.stdout:(60/174): protobuf-3.14.0-13.el9.x86_64.rpm 20 MB/s | 1.0 MB 00:00 2024-03-28T22:21:09.385 INFO:teuthology.orchestra.run.smithi082.stdout:(37/174): cephfs-top-19.0.0-1899.g323d23dc.el9. 312 kB/s | 18 kB 00:00 2024-03-28T22:21:09.397 INFO:teuthology.orchestra.run.smithi172.stdout:(61/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 78 kB 00:00 2024-03-28T22:21:09.539 INFO:teuthology.orchestra.run.smithi172.stdout:(62/174): python3-babel-2.9.1-2.el9.noarch.rpm 41 MB/s | 5.8 MB 00:00 2024-03-28T22:21:09.545 INFO:teuthology.orchestra.run.smithi082.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 4.1 MB/s | 662 kB 00:00 2024-03-28T22:21:09.581 INFO:teuthology.orchestra.run.smithi172.stdout:(63/174): python3-jsonpatch-1.21-16.el9.noarch. 585 kB/s | 24 kB 00:00 2024-03-28T22:21:09.603 INFO:teuthology.orchestra.run.smithi082.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 15 MB/s | 920 kB 00:00 2024-03-28T22:21:09.615 INFO:teuthology.orchestra.run.smithi172.stdout:(64/174): python3-jsonpointer-2.0-4.el9.noarch. 544 kB/s | 18 kB 00:00 2024-03-28T22:21:09.636 INFO:teuthology.orchestra.run.smithi082.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-28T22:21:09.657 INFO:teuthology.orchestra.run.smithi172.stdout:(65/174): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-28T22:21:09.679 INFO:teuthology.orchestra.run.smithi082.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-28T22:21:09.691 INFO:teuthology.orchestra.run.smithi172.stdout:(66/174): python3-markupsafe-1.1.1-12.el9.x86_6 947 kB/s | 32 kB 00:00 2024-03-28T22:21:09.712 INFO:teuthology.orchestra.run.smithi082.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 954 kB/s | 32 kB 00:00 2024-03-28T22:21:09.824 INFO:teuthology.orchestra.run.smithi172.stdout:(67/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-28T22:21:09.829 INFO:teuthology.orchestra.run.smithi082.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 26 MB/s | 3.0 MB 00:00 2024-03-28T22:21:09.863 INFO:teuthology.orchestra.run.smithi082.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 514 kB/s | 17 kB 00:00 2024-03-28T22:21:09.896 INFO:teuthology.orchestra.run.smithi082.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 304 kB/s | 10 kB 00:00 2024-03-28T22:21:09.908 INFO:teuthology.orchestra.run.smithi172.stdout:(68/174): ceph-mgr-diskprediction-local-19.0.0- 2.7 MB/s | 7.4 MB 00:02 2024-03-28T22:21:09.930 INFO:teuthology.orchestra.run.smithi082.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.7 MB/s | 159 kB 00:00 2024-03-28T22:21:09.933 INFO:teuthology.orchestra.run.smithi172.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.0 MB/s | 217 kB 00:00 2024-03-28T22:21:09.959 INFO:teuthology.orchestra.run.smithi172.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-28T22:21:09.963 INFO:teuthology.orchestra.run.smithi082.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-28T22:21:09.984 INFO:teuthology.orchestra.run.smithi172.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-28T22:21:10.005 INFO:teuthology.orchestra.run.smithi082.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 16 MB/s | 661 kB 00:00 2024-03-28T22:21:10.009 INFO:teuthology.orchestra.run.smithi172.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 824 kB/s | 41 kB 00:00 2024-03-28T22:21:10.034 INFO:teuthology.orchestra.run.smithi172.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-28T22:21:10.047 INFO:teuthology.orchestra.run.smithi082.stdout:(49/174): libtool-2.4.6-45.el9.x86_64.rpm 14 MB/s | 578 kB 00:00 2024-03-28T22:21:10.068 INFO:teuthology.orchestra.run.smithi172.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2024-03-28T22:21:10.081 INFO:teuthology.orchestra.run.smithi082.stdout:(50/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-28T22:21:10.093 INFO:teuthology.orchestra.run.smithi172.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.5 MB/s | 211 kB 00:00 2024-03-28T22:21:10.110 INFO:teuthology.orchestra.run.smithi172.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-28T22:21:10.115 INFO:teuthology.orchestra.run.smithi082.stdout:(51/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 8.3 MB/s | 282 kB 00:00 2024-03-28T22:21:10.127 INFO:teuthology.orchestra.run.smithi172.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 43 kB 00:00 2024-03-28T22:21:10.174 INFO:teuthology.orchestra.run.smithi082.stdout:(52/174): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2024-03-28T22:21:10.208 INFO:teuthology.orchestra.run.smithi082.stdout:(53/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 6.3 MB/s | 217 kB 00:00 2024-03-28T22:21:10.241 INFO:teuthology.orchestra.run.smithi082.stdout:(54/174): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-28T22:21:10.342 INFO:teuthology.orchestra.run.smithi082.stdout:(55/174): ceph-mgr-diskprediction-local-19.0.0- 5.2 MB/s | 7.4 MB 00:01 2024-03-28T22:21:10.426 INFO:teuthology.orchestra.run.smithi082.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 26 MB/s | 4.7 MB 00:00 2024-03-28T22:21:10.476 INFO:teuthology.orchestra.run.smithi082.stdout:(57/174): postgresql-13.14-1.el9.x86_64.rpm 12 MB/s | 1.5 MB 00:00 2024-03-28T22:21:10.510 INFO:teuthology.orchestra.run.smithi082.stdout:(58/174): postgresql-contrib-13.14-1.el9.x86_64 9.8 MB/s | 817 kB 00:00 2024-03-28T22:21:10.535 INFO:teuthology.orchestra.run.smithi082.stdout:(59/174): postgresql-private-libs-13.14-1.el9.x 2.3 MB/s | 132 kB 00:00 2024-03-28T22:21:10.544 INFO:teuthology.orchestra.run.smithi172.stdout:(78/174): ceph-test-19.0.0-1899.g323d23dc.el9.x 6.4 MB/s | 47 MB 00:07 2024-03-28T22:21:10.578 INFO:teuthology.orchestra.run.smithi172.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 86 kB/s | 39 kB 00:00 2024-03-28T22:21:10.669 INFO:teuthology.orchestra.run.smithi082.stdout:(60/174): postgresql-server-13.14-1.el9.x86_64. 36 MB/s | 5.7 MB 00:00 2024-03-28T22:21:10.702 INFO:teuthology.orchestra.run.smithi082.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.1 MB/s | 1.0 MB 00:00 2024-03-28T22:21:10.728 INFO:teuthology.orchestra.run.smithi082.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-28T22:21:10.853 INFO:teuthology.orchestra.run.smithi082.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 39 MB/s | 5.8 MB 00:00 2024-03-28T22:21:10.878 INFO:teuthology.orchestra.run.smithi082.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 165 kB/s | 24 kB 00:00 2024-03-28T22:21:10.903 INFO:teuthology.orchestra.run.smithi082.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 360 kB/s | 18 kB 00:00 2024-03-28T22:21:10.920 INFO:teuthology.orchestra.run.smithi172.stdout:(80/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 19 MB/s | 16 MB 00:00 2024-03-28T22:21:10.928 INFO:teuthology.orchestra.run.smithi082.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-28T22:21:10.954 INFO:teuthology.orchestra.run.smithi082.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 634 kB/s | 32 kB 00:00 2024-03-28T22:21:11.037 INFO:teuthology.orchestra.run.smithi172.stdout:(81/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 84 kB/s | 41 kB 00:00 2024-03-28T22:21:11.062 INFO:teuthology.orchestra.run.smithi172.stdout:(82/174): qatlib-24.02.0-1.el9.x86_64.rpm 456 kB/s | 220 kB 00:00 2024-03-28T22:21:11.087 INFO:teuthology.orchestra.run.smithi172.stdout:(83/174): qatlib-service-24.02.0-1.el9.x86_64.r 209 kB/s | 35 kB 00:00 2024-03-28T22:21:11.121 INFO:teuthology.orchestra.run.smithi172.stdout:(84/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-28T22:21:11.146 INFO:teuthology.orchestra.run.smithi172.stdout:(85/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 554 kB/s | 46 kB 00:00 2024-03-28T22:21:11.171 INFO:teuthology.orchestra.run.smithi172.stdout:(86/174): uuid-1.6.2-55.el9.x86_64.rpm 677 kB/s | 56 kB 00:00 2024-03-28T22:21:11.197 INFO:teuthology.orchestra.run.smithi172.stdout:(87/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 835 kB/s | 63 kB 00:00 2024-03-28T22:21:11.222 INFO:teuthology.orchestra.run.smithi172.stdout:(88/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 884 kB/s | 44 kB 00:00 2024-03-28T22:21:11.256 INFO:teuthology.orchestra.run.smithi172.stdout:(89/174): fuse-2.9.9-15.el9.x86_64.rpm 933 kB/s | 79 kB 00:00 2024-03-28T22:21:11.282 INFO:teuthology.orchestra.run.smithi172.stdout:(90/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.8 MB/s | 241 kB 00:00 2024-03-28T22:21:11.307 INFO:teuthology.orchestra.run.smithi172.stdout:(91/174): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-28T22:21:11.341 INFO:teuthology.orchestra.run.smithi172.stdout:(92/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-28T22:21:11.363 INFO:teuthology.orchestra.run.smithi082.stdout:(68/174): ceph-test-19.0.0-1899.g323d23dc.el9.x 9.0 MB/s | 47 MB 00:05 2024-03-28T22:21:11.366 INFO:teuthology.orchestra.run.smithi172.stdout:(93/174): python3-pycparser-2.20-6.el9.noarch.r 1.4 MB/s | 124 kB 00:00 2024-03-28T22:21:11.402 INFO:teuthology.orchestra.run.smithi172.stdout:(94/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.0 MB/s | 187 kB 00:00 2024-03-28T22:21:11.466 INFO:teuthology.orchestra.run.smithi082.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 8.9 MB/s | 4.8 MB 00:00 2024-03-28T22:21:11.494 INFO:teuthology.orchestra.run.smithi172.stdout:(95/174): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-28T22:21:11.496 INFO:teuthology.orchestra.run.smithi082.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 403 kB/s | 217 kB 00:00 2024-03-28T22:21:11.521 INFO:teuthology.orchestra.run.smithi082.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.1 MB/s | 174 kB 00:00 2024-03-28T22:21:11.527 INFO:teuthology.orchestra.run.smithi172.stdout:(96/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 195 kB 00:00 2024-03-28T22:21:11.546 INFO:teuthology.orchestra.run.smithi082.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 1.4 MB/s | 69 kB 00:00 2024-03-28T22:21:11.571 INFO:teuthology.orchestra.run.smithi082.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 552 kB/s | 41 kB 00:00 2024-03-28T22:21:11.594 INFO:teuthology.orchestra.run.smithi172.stdout:(97/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.5 MB/s | 803 kB 00:00 2024-03-28T22:21:11.597 INFO:teuthology.orchestra.run.smithi082.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-28T22:21:11.619 INFO:teuthology.orchestra.run.smithi172.stdout:(98/174): python3-requests-2.25.1-8.el9.noarch. 996 kB/s | 125 kB 00:00 2024-03-28T22:21:11.623 INFO:teuthology.orchestra.run.smithi082.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-28T22:21:11.645 INFO:teuthology.orchestra.run.smithi172.stdout:(99/174): elfutils-libelf-devel-0.190-2.el9.x86 183 kB/s | 21 kB 00:00 2024-03-28T22:21:11.647 INFO:teuthology.orchestra.run.smithi082.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.8 MB/s | 211 kB 00:00 2024-03-28T22:21:11.673 INFO:teuthology.orchestra.run.smithi082.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 635 kB/s | 47 kB 00:00 2024-03-28T22:21:11.695 INFO:teuthology.orchestra.run.smithi172.stdout:(100/174): libuuid-devel-2.37.4-18.el9.x86_64.r 254 kB/s | 25 kB 00:00 2024-03-28T22:21:11.698 INFO:teuthology.orchestra.run.smithi082.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 856 kB/s | 43 kB 00:00 2024-03-28T22:21:11.840 INFO:teuthology.orchestra.run.smithi082.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 235 kB/s | 39 kB 00:00 2024-03-28T22:21:11.862 INFO:teuthology.orchestra.run.smithi172.stdout:(101/174): python3-jinja2-2.11.3-5.el9.noarch.r 1.5 MB/s | 248 kB 00:00 2024-03-28T22:21:11.912 INFO:teuthology.orchestra.run.smithi172.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-03-28T22:21:11.937 INFO:teuthology.orchestra.run.smithi172.stdout:(103/174): python3-devel-3.9.17-2.el9.x86_64.rp 840 kB/s | 245 kB 00:00 2024-03-28T22:21:12.007 INFO:teuthology.orchestra.run.smithi082.stdout:(80/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2024-03-28T22:21:12.032 INFO:teuthology.orchestra.run.smithi082.stdout:(81/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 124 kB/s | 41 kB 00:00 2024-03-28T22:21:12.057 INFO:teuthology.orchestra.run.smithi082.stdout:(82/174): qatlib-24.02.0-1.el9.x86_64.rpm 1.0 MB/s | 220 kB 00:00 2024-03-28T22:21:12.082 INFO:teuthology.orchestra.run.smithi082.stdout:(83/174): qatlib-service-24.02.0-1.el9.x86_64.r 702 kB/s | 35 kB 00:00 2024-03-28T22:21:12.108 INFO:teuthology.orchestra.run.smithi082.stdout:(84/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 621 kB/s | 46 kB 00:00 2024-03-28T22:21:12.113 INFO:teuthology.orchestra.run.smithi172.stdout:(104/174): openssl-devel-3.0.7-27.el9.x86_64.rp 8.4 MB/s | 4.1 MB 00:00 2024-03-28T22:21:12.133 INFO:teuthology.orchestra.run.smithi082.stdout:(85/174): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 300 kB 00:00 2024-03-28T22:21:12.138 INFO:teuthology.orchestra.run.smithi172.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 99 kB/s | 22 kB 00:00 2024-03-28T22:21:12.158 INFO:teuthology.orchestra.run.smithi082.stdout:(86/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-28T22:21:12.183 INFO:teuthology.orchestra.run.smithi082.stdout:(87/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 838 kB/s | 63 kB 00:00 2024-03-28T22:21:12.209 INFO:teuthology.orchestra.run.smithi082.stdout:(88/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 586 kB/s | 44 kB 00:00 2024-03-28T22:21:12.234 INFO:teuthology.orchestra.run.smithi082.stdout:(89/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-28T22:21:12.259 INFO:teuthology.orchestra.run.smithi082.stdout:(90/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-28T22:21:12.301 INFO:teuthology.orchestra.run.smithi082.stdout:(91/174): python3-cryptography-36.0.1-4.el9.x86 13 MB/s | 1.1 MB 00:00 2024-03-28T22:21:12.326 INFO:teuthology.orchestra.run.smithi082.stdout:(92/174): python3-pycparser-2.20-6.el9.noarch.r 1.9 MB/s | 124 kB 00:00 2024-03-28T22:21:12.351 INFO:teuthology.orchestra.run.smithi082.stdout:(93/174): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-28T22:21:12.376 INFO:teuthology.orchestra.run.smithi082.stdout:(94/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2024-03-28T22:21:12.380 INFO:teuthology.orchestra.run.smithi172.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 2.0 MB/s | 548 kB 00:00 2024-03-28T22:21:12.405 INFO:teuthology.orchestra.run.smithi172.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.1 MB/s | 309 kB 00:00 2024-03-28T22:21:12.430 INFO:teuthology.orchestra.run.smithi172.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 390 kB/s | 19 kB 00:00 2024-03-28T22:21:12.497 INFO:teuthology.orchestra.run.smithi172.stdout:(109/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.5 MB/s | 863 kB 00:00 2024-03-28T22:21:12.589 INFO:teuthology.orchestra.run.smithi172.stdout:(110/174): libarrow-9.0.0-10.el9.x86_64.rpm 24 MB/s | 4.4 MB 00:00 2024-03-28T22:21:12.615 INFO:teuthology.orchestra.run.smithi172.stdout:(111/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 142 kB/s | 26 kB 00:00 2024-03-28T22:21:12.618 INFO:teuthology.orchestra.run.smithi082.stdout:(95/174): libquadmath-11.4.1-3.el9.x86_64.rpm 809 kB/s | 195 kB 00:00 2024-03-28T22:21:12.640 INFO:teuthology.orchestra.run.smithi172.stdout:(112/174): liboath-2.6.7-2.el9.x86_64.rpm 349 kB/s | 49 kB 00:00 2024-03-28T22:21:12.644 INFO:teuthology.orchestra.run.smithi082.stdout:(96/174): jq-1.6-16.el9.x86_64.rpm 591 kB/s | 187 kB 00:00 2024-03-28T22:21:12.665 INFO:teuthology.orchestra.run.smithi172.stdout:(113/174): libunwind-1.6.2-1.el9.x86_64.rpm 897 kB/s | 67 kB 00:00 2024-03-28T22:21:12.677 INFO:teuthology.orchestra.run.smithi082.stdout:(97/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 803 kB 00:00 2024-03-28T22:21:12.699 INFO:teuthology.orchestra.run.smithi172.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-03-28T22:21:12.703 INFO:teuthology.orchestra.run.smithi082.stdout:(98/174): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-03-28T22:21:12.741 INFO:teuthology.orchestra.run.smithi172.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.2 MB/s | 838 kB 00:00 2024-03-28T22:21:12.765 INFO:teuthology.orchestra.run.smithi172.stdout:(116/174): python3-asyncssh-2.13.2-5.el9.noarch 5.4 MB/s | 548 kB 00:00 2024-03-28T22:21:12.769 INFO:teuthology.orchestra.run.smithi082.stdout:(99/174): elfutils-libelf-devel-0.190-2.el9.x86 171 kB/s | 21 kB 00:00 2024-03-28T22:21:12.791 INFO:teuthology.orchestra.run.smithi172.stdout:(117/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 484 kB/s | 43 kB 00:00 2024-03-28T22:21:12.803 INFO:teuthology.orchestra.run.smithi082.stdout:(100/174): libuuid-devel-2.37.4-18.el9.x86_64.r 203 kB/s | 25 kB 00:00 2024-03-28T22:21:12.816 INFO:teuthology.orchestra.run.smithi172.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 643 kB/s | 32 kB 00:00 2024-03-28T22:21:12.841 INFO:teuthology.orchestra.run.smithi172.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 200 kB/s | 15 kB 00:00 2024-03-28T22:21:12.866 INFO:teuthology.orchestra.run.smithi172.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-28T22:21:12.900 INFO:teuthology.orchestra.run.smithi172.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 6.0 MB/s | 358 kB 00:00 2024-03-28T22:21:12.934 INFO:teuthology.orchestra.run.smithi172.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-03-28T22:21:12.951 INFO:teuthology.orchestra.run.smithi172.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 424 kB/s | 35 kB 00:00 2024-03-28T22:21:12.976 INFO:teuthology.orchestra.run.smithi172.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 2.7 MB/s | 203 kB 00:00 2024-03-28T22:21:13.028 INFO:teuthology.orchestra.run.smithi082.stdout:(101/174): python3-devel-3.9.17-2.el9.x86_64.rp 949 kB/s | 245 kB 00:00 2024-03-28T22:21:13.034 INFO:teuthology.orchestra.run.smithi172.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 25 MB/s | 2.0 MB 00:00 2024-03-28T22:21:13.054 INFO:teuthology.orchestra.run.smithi082.stdout:(102/174): python3-jinja2-2.11.3-5.el9.noarch.r 990 kB/s | 248 kB 00:00 2024-03-28T22:21:13.068 INFO:teuthology.orchestra.run.smithi172.stdout:(126/174): python3-grpcio-tools-1.46.7-10.el9.x 1.2 MB/s | 144 kB 00:00 2024-03-28T22:21:13.087 INFO:teuthology.orchestra.run.smithi082.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.6 MB/s | 94 kB 00:00 2024-03-28T22:21:13.093 INFO:teuthology.orchestra.run.smithi172.stdout:(127/174): python3-jaraco-8.2.1-3.el9.noarch.rp 92 kB/s | 11 kB 00:00 2024-03-28T22:21:13.119 INFO:teuthology.orchestra.run.smithi172.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 211 kB/s | 18 kB 00:00 2024-03-28T22:21:13.121 INFO:teuthology.orchestra.run.smithi082.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 333 kB/s | 22 kB 00:00 2024-03-28T22:21:13.144 INFO:teuthology.orchestra.run.smithi172.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 463 kB/s | 23 kB 00:00 2024-03-28T22:21:13.169 INFO:teuthology.orchestra.run.smithi172.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 259 kB/s | 19 kB 00:00 2024-03-28T22:21:13.194 INFO:teuthology.orchestra.run.smithi172.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 262 kB/s | 20 kB 00:00 2024-03-28T22:21:13.205 INFO:teuthology.orchestra.run.smithi082.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 7.2 MB/s | 863 kB 00:00 2024-03-28T22:21:13.219 INFO:teuthology.orchestra.run.smithi172.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 930 kB/s | 46 kB 00:00 2024-03-28T22:21:13.253 INFO:teuthology.orchestra.run.smithi172.stdout:(133/174): python3-kubernetes-26.1.0-2.el9.noar 12 MB/s | 1.0 MB 00:00 2024-03-28T22:21:13.278 INFO:teuthology.orchestra.run.smithi172.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 942 kB/s | 79 kB 00:00 2024-03-28T22:21:13.304 INFO:teuthology.orchestra.run.smithi172.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 707 kB/s | 58 kB 00:00 2024-03-28T22:21:13.329 INFO:teuthology.orchestra.run.smithi172.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 330 kB/s | 16 kB 00:00 2024-03-28T22:21:13.354 INFO:teuthology.orchestra.run.smithi172.stdout:(137/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-28T22:21:13.379 INFO:teuthology.orchestra.run.smithi172.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-03-28T22:21:13.405 INFO:teuthology.orchestra.run.smithi172.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 377 kB/s | 28 kB 00:00 2024-03-28T22:21:13.430 INFO:teuthology.orchestra.run.smithi172.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 616 kB/s | 31 kB 00:00 2024-03-28T22:21:13.455 INFO:teuthology.orchestra.run.smithi172.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.5 MB/s | 188 kB 00:00 2024-03-28T22:21:13.480 INFO:teuthology.orchestra.run.smithi172.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 787 kB/s | 59 kB 00:00 2024-03-28T22:21:13.505 INFO:teuthology.orchestra.run.smithi172.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 718 kB/s | 36 kB 00:00 2024-03-28T22:21:13.531 INFO:teuthology.orchestra.run.smithi172.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 680 kB/s | 51 kB 00:00 2024-03-28T22:21:13.556 INFO:teuthology.orchestra.run.smithi172.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-28T22:21:13.581 INFO:teuthology.orchestra.run.smithi172.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.8 MB/s | 90 kB 00:00 2024-03-28T22:21:13.615 INFO:teuthology.orchestra.run.smithi172.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-03-28T22:21:13.640 INFO:teuthology.orchestra.run.smithi172.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 238 kB/s | 20 kB 00:00 2024-03-28T22:21:13.665 INFO:teuthology.orchestra.run.smithi172.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.3 MB/s | 192 kB 00:00 2024-03-28T22:21:13.699 INFO:teuthology.orchestra.run.smithi172.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-03-28T22:21:13.713 INFO:teuthology.orchestra.run.smithi082.stdout:(106/174): openssl-devel-3.0.7-27.el9.x86_64.rp 4.1 MB/s | 4.1 MB 00:01 2024-03-28T22:21:13.746 INFO:teuthology.orchestra.run.smithi082.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 571 kB/s | 309 kB 00:00 2024-03-28T22:21:13.771 INFO:teuthology.orchestra.run.smithi082.stdout:(108/174): abseil-cpp-20211102.0-3.el9.x86_64.r 843 kB/s | 548 kB 00:00 2024-03-28T22:21:13.799 INFO:teuthology.orchestra.run.smithi172.stdout:(151/174): NetworkManager-1.47.2-1.el9.x86_64.r 23 MB/s | 2.3 MB 00:00 2024-03-28T22:21:13.883 INFO:teuthology.orchestra.run.smithi172.stdout:(152/174): NetworkManager-libnm-1.47.2-1.el9.x8 22 MB/s | 1.8 MB 00:00 2024-03-28T22:21:13.942 INFO:teuthology.orchestra.run.smithi172.stdout:(153/174): NetworkManager-team-1.47.2-1.el9.x86 704 kB/s | 41 kB 00:00 2024-03-28T22:21:13.989 INFO:teuthology.orchestra.run.smithi082.stdout:(109/174): grpc-data-1.46.7-10.el9.noarch.rpm 71 kB/s | 19 kB 00:00 2024-03-28T22:21:13.992 INFO:teuthology.orchestra.run.smithi172.stdout:(154/174): NetworkManager-tui-1.47.2-1.el9.x86_ 4.8 MB/s | 247 kB 00:00 2024-03-28T22:21:14.034 INFO:teuthology.orchestra.run.smithi172.stdout:(155/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-28T22:21:14.039 INFO:teuthology.orchestra.run.smithi082.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 97 kB/s | 26 kB 00:00 2024-03-28T22:21:14.092 INFO:teuthology.orchestra.run.smithi172.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-03-28T22:21:14.159 INFO:teuthology.orchestra.run.smithi172.stdout:(157/174): librbd1-19.0.0-1899.g323d23dc.el9.x8 6.4 MB/s | 3.1 MB 00:00 2024-03-28T22:21:14.193 INFO:teuthology.orchestra.run.smithi172.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 362 kB/s | 36 kB 00:00 2024-03-28T22:21:14.252 INFO:teuthology.orchestra.run.smithi172.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.0 MB/s | 192 kB 00:00 2024-03-28T22:21:14.285 INFO:teuthology.orchestra.run.smithi172.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.7 MB/s | 256 kB 00:00 2024-03-28T22:21:14.339 INFO:teuthology.orchestra.run.smithi082.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 139 kB/s | 49 kB 00:00 2024-03-28T22:21:14.344 INFO:teuthology.orchestra.run.smithi172.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 250 kB/s | 23 kB 00:00 2024-03-28T22:21:14.386 INFO:teuthology.orchestra.run.smithi172.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 108 kB 00:00 2024-03-28T22:21:14.389 INFO:teuthology.orchestra.run.smithi082.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 192 kB/s | 67 kB 00:00 2024-03-28T22:21:14.486 INFO:teuthology.orchestra.run.smithi172.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 155 kB 00:00 2024-03-28T22:21:14.537 INFO:teuthology.orchestra.run.smithi172.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 904 kB/s | 136 kB 00:00 2024-03-28T22:21:14.570 INFO:teuthology.orchestra.run.smithi172.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 765 kB/s | 64 kB 00:00 2024-03-28T22:21:14.595 INFO:teuthology.orchestra.run.smithi172.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 491 kB/s | 29 kB 00:00 2024-03-28T22:21:14.615 INFO:teuthology.orchestra.run.smithi082.stdout:(113/174): libarrow-9.0.0-10.el9.x86_64.rpm 5.1 MB/s | 4.4 MB 00:00 2024-03-28T22:21:14.663 INFO:teuthology.orchestra.run.smithi172.stdout:(167/174): librados2-19.0.0-1899.g323d23dc.el9. 3.3 MB/s | 3.4 MB 00:01 2024-03-28T22:21:14.773 INFO:teuthology.orchestra.run.smithi082.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 348 kB/s | 151 kB 00:00 2024-03-28T22:21:14.913 INFO:teuthology.orchestra.run.smithi172.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 3.5 MB/s | 1.2 MB 00:00 2024-03-28T22:21:14.998 INFO:teuthology.orchestra.run.smithi082.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.3 MB/s | 838 kB 00:00 2024-03-28T22:21:15.113 INFO:teuthology.orchestra.run.smithi172.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 4.2 MB/s | 2.2 MB 00:00 2024-03-28T22:21:15.123 INFO:teuthology.orchestra.run.smithi082.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 124 kB/s | 43 kB 00:00 2024-03-28T22:21:15.163 INFO:teuthology.orchestra.run.smithi172.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 1.8 MB/s | 465 kB 00:00 2024-03-28T22:21:15.205 INFO:teuthology.orchestra.run.smithi172.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 4.2 MB/s | 2.3 MB 00:00 2024-03-28T22:21:15.230 INFO:teuthology.orchestra.run.smithi172.stdout:(172/174): util-linux-user-2.37.4-18.el9.x86_64 266 kB/s | 31 kB 00:00 2024-03-28T22:21:15.232 INFO:teuthology.orchestra.run.smithi082.stdout:(117/174): python3-asyncssh-2.13.2-5.el9.noarch 888 kB/s | 548 kB 00:00 2024-03-28T22:21:15.266 INFO:teuthology.orchestra.run.smithi082.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 120 kB/s | 32 kB 00:00 2024-03-28T22:21:15.297 INFO:teuthology.orchestra.run.smithi172.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 811 kB/s | 74 kB 00:00 2024-03-28T22:21:15.397 INFO:teuthology.orchestra.run.smithi172.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 4.6 MB/s | 1.1 MB 00:00 2024-03-28T22:21:15.404 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-28T22:21:15.404 INFO:teuthology.orchestra.run.smithi172.stdout:Total 14 MB/s | 226 MB 00:15 2024-03-28T22:21:15.416 INFO:teuthology.orchestra.run.smithi082.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 51 kB/s | 15 kB 00:00 2024-03-28T22:21:15.658 INFO:teuthology.orchestra.run.smithi082.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 403 kB/s | 172 kB 00:00 2024-03-28T22:21:15.767 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-28T22:21:15.775 INFO:teuthology.orchestra.run.smithi082.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 704 kB/s | 358 kB 00:00 2024-03-28T22:21:15.917 INFO:teuthology.orchestra.run.smithi082.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 606 kB/s | 303 kB 00:00 2024-03-28T22:21:15.942 INFO:teuthology.orchestra.run.smithi082.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 124 kB/s | 35 kB 00:00 2024-03-28T22:21:16.040 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-28T22:21:16.040 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-28T22:21:16.201 INFO:teuthology.orchestra.run.smithi082.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 476 kB/s | 203 kB 00:00 2024-03-28T22:21:16.368 INFO:teuthology.orchestra.run.smithi082.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 340 kB/s | 144 kB 00:00 2024-03-28T22:21:16.393 INFO:teuthology.orchestra.run.smithi082.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 56 kB/s | 11 kB 00:00 2024-03-28T22:21:16.635 INFO:teuthology.orchestra.run.smithi082.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 66 kB/s | 18 kB 00:00 2024-03-28T22:21:16.660 INFO:teuthology.orchestra.run.smithi082.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 87 kB/s | 23 kB 00:00 2024-03-28T22:21:16.710 INFO:teuthology.orchestra.run.smithi082.stdout:(129/174): python3-grpcio-1.46.7-10.el9.x86_64. 2.6 MB/s | 2.0 MB 00:00 2024-03-28T22:21:16.902 INFO:teuthology.orchestra.run.smithi082.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 73 kB/s | 19 kB 00:00 2024-03-28T22:21:16.927 INFO:teuthology.orchestra.run.smithi082.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 74 kB/s | 20 kB 00:00 2024-03-28T22:21:17.252 INFO:teuthology.orchestra.run.smithi082.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 132 kB/s | 46 kB 00:00 2024-03-28T22:21:17.277 INFO:teuthology.orchestra.run.smithi082.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 225 kB/s | 79 kB 00:00 2024-03-28T22:21:17.386 INFO:teuthology.orchestra.run.smithi082.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 1.5 MB/s | 1.0 MB 00:00 2024-03-28T22:21:17.611 INFO:teuthology.orchestra.run.smithi082.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 161 kB/s | 58 kB 00:00 2024-03-28T22:21:17.653 INFO:teuthology.orchestra.run.smithi082.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 61 kB/s | 16 kB 00:00 2024-03-28T22:21:17.713 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - CRB 723 kB/s | 6.1 MB 00:08 2024-03-28T22:21:17.740 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-28T22:21:17.741 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-28T22:21:17.787 INFO:teuthology.orchestra.run.smithi082.stdout:(137/174): python3-pecan-1.4.2-3.el9.noarch.rpm 535 kB/s | 272 kB 00:00 2024-03-28T22:21:17.929 INFO:teuthology.orchestra.run.smithi082.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 102 kB/s | 28 kB 00:00 2024-03-28T22:21:17.962 INFO:teuthology.orchestra.run.smithi082.stdout:(139/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 256 kB/s | 90 kB 00:00 2024-03-28T22:21:18.063 INFO:teuthology.orchestra.run.smithi082.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 112 kB/s | 31 kB 00:00 2024-03-28T22:21:18.329 INFO:teuthology.orchestra.run.smithi082.stdout:(141/174): python3-tempora-5.0.0-2.el9.noarch.r 134 kB/s | 36 kB 00:00 2024-03-28T22:21:18.355 INFO:teuthology.orchestra.run.smithi082.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 151 kB/s | 59 kB 00:00 2024-03-28T22:21:18.380 INFO:teuthology.orchestra.run.smithi082.stdout:(143/174): python3-routes-2.5.1-5.el9.noarch.rp 420 kB/s | 188 kB 00:00 2024-03-28T22:21:18.672 INFO:teuthology.orchestra.run.smithi082.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 149 kB/s | 51 kB 00:00 2024-03-28T22:21:18.722 INFO:teuthology.orchestra.run.smithi082.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 262 kB/s | 90 kB 00:00 2024-03-28T22:21:18.856 INFO:teuthology.orchestra.run.smithi082.stdout:(146/174): python3-webob-1.8.7-6.el9.noarch.rpm 459 kB/s | 230 kB 00:00 2024-03-28T22:21:18.981 INFO:teuthology.orchestra.run.smithi082.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 77 kB/s | 20 kB 00:00 2024-03-28T22:21:19.265 INFO:teuthology.orchestra.run.smithi082.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 721 kB/s | 427 kB 00:00 2024-03-28T22:21:19.290 INFO:teuthology.orchestra.run.smithi082.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 442 kB/s | 192 kB 00:00 2024-03-28T22:21:19.583 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-03-28T22:21:19.667 INFO:teuthology.orchestra.run.smithi082.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 2.3 MB/s | 1.6 MB 00:00 2024-03-28T22:21:19.717 INFO:teuthology.orchestra.run.smithi082.stdout:(151/174): librbd1-19.0.0-1899.g323d23dc.el9.x8 7.3 MB/s | 3.1 MB 00:00 2024-03-28T22:21:19.768 INFO:teuthology.orchestra.run.smithi082.stdout:(152/174): NetworkManager-1.47.2-1.el9.x86_64.r 23 MB/s | 2.3 MB 00:00 2024-03-28T22:21:19.809 INFO:teuthology.orchestra.run.smithi082.stdout:(153/174): NetworkManager-libnm-1.47.2-1.el9.x8 20 MB/s | 1.8 MB 00:00 2024-03-28T22:21:19.835 INFO:teuthology.orchestra.run.smithi082.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 636 kB/s | 41 kB 00:00 2024-03-28T22:21:19.860 INFO:teuthology.orchestra.run.smithi082.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 4.8 MB/s | 247 kB 00:00 2024-03-28T22:21:19.885 INFO:teuthology.orchestra.run.smithi082.stdout:(156/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-28T22:21:20.052 INFO:teuthology.orchestra.run.smithi082.stdout:(157/174): librados2-19.0.0-1899.g323d23dc.el9. 4.3 MB/s | 3.4 MB 00:00 2024-03-28T22:21:20.103 INFO:teuthology.orchestra.run.smithi082.stdout:(158/174): acl-2.3.1-4.el9.x86_64.rpm 295 kB/s | 71 kB 00:00 2024-03-28T22:21:20.153 INFO:teuthology.orchestra.run.smithi082.stdout:(159/174): elfutils-debuginfod-client-0.190-2.e 136 kB/s | 36 kB 00:00 2024-03-28T22:21:20.211 INFO:teuthology.orchestra.run.smithi082.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 3.0 MB/s | 256 kB 00:00 2024-03-28T22:21:20.245 INFO:teuthology.orchestra.run.smithi082.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 252 kB/s | 23 kB 00:00 2024-03-28T22:21:20.287 INFO:teuthology.orchestra.run.smithi082.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 108 kB 00:00 2024-03-28T22:21:20.321 INFO:teuthology.orchestra.run.smithi082.stdout:(163/174): elfutils-libelf-0.190-2.el9.x86_64.r 720 kB/s | 192 kB 00:00 2024-03-28T22:21:20.346 INFO:teuthology.orchestra.run.smithi082.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.5 MB/s | 155 kB 00:00 2024-03-28T22:21:20.388 INFO:teuthology.orchestra.run.smithi082.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 957 kB/s | 64 kB 00:00 2024-03-28T22:21:20.413 INFO:teuthology.orchestra.run.smithi082.stdout:(166/174): libmount-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 136 kB 00:00 2024-03-28T22:21:20.438 INFO:teuthology.orchestra.run.smithi082.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 312 kB/s | 29 kB 00:00 2024-03-28T22:21:20.514 INFO:teuthology.orchestra.run.smithi082.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 9.6 MB/s | 1.2 MB 00:00 2024-03-28T22:21:20.674 INFO:teuthology.orchestra.run.smithi082.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 8.3 MB/s | 2.2 MB 00:00 2024-03-28T22:21:20.839 INFO:teuthology.orchestra.run.smithi082.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 1.4 MB/s | 465 kB 00:00 2024-03-28T22:21:20.890 INFO:teuthology.orchestra.run.smithi082.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 5.1 MB/s | 2.3 MB 00:00 2024-03-28T22:21:20.915 INFO:teuthology.orchestra.run.smithi082.stdout:(172/174): util-linux-user-2.37.4-18.el9.x86_64 129 kB/s | 31 kB 00:00 2024-03-28T22:21:20.948 INFO:teuthology.orchestra.run.smithi082.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 9.9 MB/s | 1.1 MB 00:00 2024-03-28T22:21:21.015 INFO:teuthology.orchestra.run.smithi082.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 595 kB/s | 74 kB 00:00 2024-03-28T22:21:21.022 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-28T22:21:21.022 INFO:teuthology.orchestra.run.smithi082.stdout:Total 11 MB/s | 226 MB 00:20 2024-03-28T22:21:21.096 INFO:teuthology.orchestra.run.smithi043.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-28T22:21:21.398 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-28T22:21:21.670 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-28T22:21:21.670 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-28T22:21:23.379 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-28T22:21:23.379 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-28T22:21:24.573 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-28T22:21:24.874 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T22:21:24.986 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T22:21:25.024 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T22:21:25.035 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T22:21:25.076 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T22:21:25.139 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T22:21:25.179 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T22:21:25.204 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T22:21:25.248 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T22:21:25.575 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T22:21:25.638 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : librados2-2:19.0.0-1899.g323d23dc.el9.x86_64 10/198 2024-03-28T22:21:25.759 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados2-2:19.0.0-1899.g323d23dc.el9.x86_64 10/198 2024-03-28T22:21:25.789 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephfs2-2:19.0.0-1899.g323d23dc.el9.x86_64 11/198 2024-03-28T22:21:25.880 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephfs2-2:19.0.0-1899.g323d23dc.el9.x86_64 11/198 2024-03-28T22:21:25.918 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rados-2:19.0.0-1899.g323d23dc.el9.x86_64 12/198 2024-03-28T22:21:25.952 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T22:21:26.132 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T22:21:26.199 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T22:21:26.235 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephsqlite-2:19.0.0-1899.g323d23dc.el9.x86_64 16/198 2024-03-28T22:21:26.330 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1899.g323d23dc.el9.x86_64 16/198 2024-03-28T22:21:26.361 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libradosstriper1-2:19.0.0-1899.g323d23dc.el9.x86 17/198 2024-03-28T22:21:26.408 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1899.g323d23dc.el9.x86 17/198 2024-03-28T22:21:26.448 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T22:21:26.471 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T22:21:26.486 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T22:21:26.629 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T22:21:26.668 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T22:21:26.708 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T22:21:26.814 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T22:21:26.936 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T22:21:26.996 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T22:21:27.076 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T22:21:27.131 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T22:21:27.165 INFO:teuthology.orchestra.run.smithi043.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-28T22:21:27.362 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T22:21:27.473 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T22:21:27.564 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T22:21:27.611 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T22:21:27.638 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T22:21:27.682 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T22:21:27.756 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T22:21:27.809 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T22:21:27.856 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T22:21:27.920 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T22:21:27.955 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T22:21:27.999 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T22:21:28.032 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T22:21:28.059 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T22:21:28.128 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T22:21:28.164 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ceph-common-2:19.0.0-1899.g323d23dc.el9. 39/198 2024-03-28T22:21:28.207 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ceph-argparse-2:19.0.0-1899.g323d23dc.el 40/198 2024-03-28T22:21:28.236 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cephfs-2:19.0.0-1899.g323d23dc.el9.x86_6 41/198 2024-03-28T22:21:28.313 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T22:21:28.367 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T22:21:28.441 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T22:21:28.599 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T22:21:28.653 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T22:21:29.172 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T22:21:29.256 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T22:21:29.327 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T22:21:29.393 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T22:21:29.403 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T22:21:29.433 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T22:21:29.474 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T22:21:29.511 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librados-devel-2:19.0.0-1899.g323d23dc.el9.x86_6 53/198 2024-03-28T22:21:29.549 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T22:21:29.777 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T22:21:29.967 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T22:21:30.027 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T22:21:30.063 INFO:teuthology.orchestra.run.smithi043.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T22:21:30.065 INFO:teuthology.orchestra.run.smithi043.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T22:21:30.071 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T22:21:30.088 INFO:teuthology.orchestra.run.smithi043.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T22:21:30.090 INFO:teuthology.orchestra.run.smithi043.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T22:21:30.091 INFO:teuthology.orchestra.run.smithi043.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T22:21:30.093 INFO:teuthology.orchestra.run.smithi043.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T22:21:30.095 INFO:teuthology.orchestra.run.smithi043.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T22:21:30.099 INFO:teuthology.orchestra.run.smithi043.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T22:21:30.101 INFO:teuthology.orchestra.run.smithi043.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T22:21:30.194 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-03-28T22:21:30.214 INFO:teuthology.orchestra.run.smithi043.stdout:============================================================================================ 2024-03-28T22:21:30.214 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-03-28T22:21:30.214 INFO:teuthology.orchestra.run.smithi043.stdout:============================================================================================ 2024-03-28T22:21:30.214 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-03-28T22:21:30.214 INFO:teuthology.orchestra.run.smithi043.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T22:21:30.214 INFO:teuthology.orchestra.run.smithi043.stdout: ceph x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 6.6 k 2024-03-28T22:21:30.214 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-base x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 5.3 M 2024-03-28T22:21:30.214 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-fuse x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 889 k 2024-03-28T22:21:30.214 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 145 k 2024-03-28T22:21:30.214 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 1.5 M 2024-03-28T22:21:30.215 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 138 k 2024-03-28T22:21:30.215 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 3.5 M 2024-03-28T22:21:30.215 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 7.4 M 2024-03-28T22:21:30.215 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-rook noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 49 k 2024-03-28T22:21:30.215 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-radosgw x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 9.8 M 2024-03-28T22:21:30.215 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-test x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 47 M 2024-03-28T22:21:30.215 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-volume noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 264 k 2024-03-28T22:21:30.215 INFO:teuthology.orchestra.run.smithi043.stdout: cephadm noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 748 k 2024-03-28T22:21:30.215 INFO:teuthology.orchestra.run.smithi043.stdout: cephfs-mirror x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 221 k 2024-03-28T22:21:30.215 INFO:teuthology.orchestra.run.smithi043.stdout: cephfs-shell x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 27 k 2024-03-28T22:21:30.215 INFO:teuthology.orchestra.run.smithi043.stdout: cephfs-top noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 18 k 2024-03-28T22:21:30.215 INFO:teuthology.orchestra.run.smithi043.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T22:21:30.215 INFO:teuthology.orchestra.run.smithi043.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T22:21:30.216 INFO:teuthology.orchestra.run.smithi043.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T22:21:30.216 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs-devel x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 32 k 2024-03-28T22:21:30.216 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs2 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 746 k 2024-03-28T22:21:30.216 INFO:teuthology.orchestra.run.smithi043.stdout: librados-devel x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 127 k 2024-03-28T22:21:30.216 INFO:teuthology.orchestra.run.smithi043.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T22:21:30.216 INFO:teuthology.orchestra.run.smithi043.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T22:21:30.216 INFO:teuthology.orchestra.run.smithi043.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T22:21:30.216 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-03-28T22:21:30.216 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-03-28T22:21:30.216 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-03-28T22:21:30.216 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cephfs x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 162 k 2024-03-28T22:21:30.216 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rados x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 321 k 2024-03-28T22:21:30.217 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rbd x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 299 k 2024-03-28T22:21:30.217 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rgw x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 100 k 2024-03-28T22:21:30.217 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-fuse x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 86 k 2024-03-28T22:21:30.217 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-mirror x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 3.1 M 2024-03-28T22:21:30.217 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-nbd x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 171 k 2024-03-28T22:21:30.217 INFO:teuthology.orchestra.run.smithi043.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T22:21:30.217 INFO:teuthology.orchestra.run.smithi043.stdout:Upgrading: 2024-03-28T22:21:30.217 INFO:teuthology.orchestra.run.smithi043.stdout: NetworkManager x86_64 1:1.47.2-1.el9 CentOS-BaseOS 2.3 M 2024-03-28T22:21:30.217 INFO:teuthology.orchestra.run.smithi043.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 CentOS-BaseOS 1.8 M 2024-03-28T22:21:30.217 INFO:teuthology.orchestra.run.smithi043.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 CentOS-BaseOS 41 k 2024-03-28T22:21:30.217 INFO:teuthology.orchestra.run.smithi043.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 CentOS-BaseOS 247 k 2024-03-28T22:21:30.217 INFO:teuthology.orchestra.run.smithi043.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T22:21:30.217 INFO:teuthology.orchestra.run.smithi043.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T22:21:30.218 INFO:teuthology.orchestra.run.smithi043.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T22:21:30.218 INFO:teuthology.orchestra.run.smithi043.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T22:21:30.218 INFO:teuthology.orchestra.run.smithi043.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T22:21:30.218 INFO:teuthology.orchestra.run.smithi043.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T22:21:30.218 INFO:teuthology.orchestra.run.smithi043.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T22:21:30.218 INFO:teuthology.orchestra.run.smithi043.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T22:21:30.218 INFO:teuthology.orchestra.run.smithi043.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T22:21:30.218 INFO:teuthology.orchestra.run.smithi043.stdout: librados2 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 3.4 M 2024-03-28T22:21:30.218 INFO:teuthology.orchestra.run.smithi043.stdout: librbd1 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 3.1 M 2024-03-28T22:21:30.218 INFO:teuthology.orchestra.run.smithi043.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T22:21:30.218 INFO:teuthology.orchestra.run.smithi043.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T22:21:30.218 INFO:teuthology.orchestra.run.smithi043.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T22:21:30.218 INFO:teuthology.orchestra.run.smithi043.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T22:21:30.218 INFO:teuthology.orchestra.run.smithi043.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T22:21:30.219 INFO:teuthology.orchestra.run.smithi043.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T22:21:30.219 INFO:teuthology.orchestra.run.smithi043.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T22:21:30.219 INFO:teuthology.orchestra.run.smithi043.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T22:21:30.219 INFO:teuthology.orchestra.run.smithi043.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T22:21:30.219 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-03-28T22:21:30.219 INFO:teuthology.orchestra.run.smithi043.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T22:21:30.219 INFO:teuthology.orchestra.run.smithi043.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T22:21:30.219 INFO:teuthology.orchestra.run.smithi043.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T22:21:30.219 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-common x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 21 M 2024-03-28T22:21:30.219 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 31 k 2024-03-28T22:21:30.219 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mds x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 2.4 M 2024-03-28T22:21:30.219 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 248 k 2024-03-28T22:21:30.220 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mon x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 4.7 M 2024-03-28T22:21:30.220 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-osd x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 17 M 2024-03-28T22:21:30.220 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1899.g323d23dc.el9 ceph-noarch 15 k 2024-03-28T22:21:30.220 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-selinux x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 25 k 2024-03-28T22:21:30.220 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T22:21:30.220 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T22:21:30.220 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T22:21:30.220 INFO:teuthology.orchestra.run.smithi043.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T22:21:30.220 INFO:teuthology.orchestra.run.smithi043.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T22:21:30.220 INFO:teuthology.orchestra.run.smithi043.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T22:21:30.220 INFO:teuthology.orchestra.run.smithi043.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T22:21:30.220 INFO:teuthology.orchestra.run.smithi043.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T22:21:30.220 INFO:teuthology.orchestra.run.smithi043.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T22:21:30.221 INFO:teuthology.orchestra.run.smithi043.stdout: libcephsqlite x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 163 k 2024-03-28T22:21:30.221 INFO:teuthology.orchestra.run.smithi043.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T22:21:30.221 INFO:teuthology.orchestra.run.smithi043.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T22:21:30.221 INFO:teuthology.orchestra.run.smithi043.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T22:21:30.221 INFO:teuthology.orchestra.run.smithi043.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T22:21:30.221 INFO:teuthology.orchestra.run.smithi043.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T22:21:30.221 INFO:teuthology.orchestra.run.smithi043.stdout: libradosstriper1 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 501 k 2024-03-28T22:21:30.221 INFO:teuthology.orchestra.run.smithi043.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T22:21:30.221 INFO:teuthology.orchestra.run.smithi043.stdout: librgw2 x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 5.1 M 2024-03-28T22:21:30.221 INFO:teuthology.orchestra.run.smithi043.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T22:21:30.221 INFO:teuthology.orchestra.run.smithi043.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T22:21:30.221 INFO:teuthology.orchestra.run.smithi043.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T22:21:30.221 INFO:teuthology.orchestra.run.smithi043.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T22:21:30.221 INFO:teuthology.orchestra.run.smithi043.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T22:21:30.222 INFO:teuthology.orchestra.run.smithi043.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T22:21:30.222 INFO:teuthology.orchestra.run.smithi043.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T22:21:30.222 INFO:teuthology.orchestra.run.smithi043.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T22:21:30.222 INFO:teuthology.orchestra.run.smithi043.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T22:21:30.222 INFO:teuthology.orchestra.run.smithi043.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T22:21:30.222 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-03-28T22:21:30.222 INFO:teuthology.orchestra.run.smithi043.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T22:21:30.222 INFO:teuthology.orchestra.run.smithi043.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T22:21:30.222 INFO:teuthology.orchestra.run.smithi043.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T22:21:30.222 INFO:teuthology.orchestra.run.smithi043.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T22:21:30.222 INFO:teuthology.orchestra.run.smithi043.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T22:21:30.222 INFO:teuthology.orchestra.run.smithi043.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T22:21:30.223 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T22:21:30.223 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-argparse x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 45 k 2024-03-28T22:21:30.223 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-common x86_64 2:19.0.0-1899.g323d23dc.el9 ceph 131 k 2024-03-28T22:21:30.223 INFO:teuthology.orchestra.run.smithi043.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T22:21:30.223 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T22:21:30.223 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T22:21:30.223 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T22:21:30.223 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T22:21:30.223 INFO:teuthology.orchestra.run.smithi043.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T22:21:30.223 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T22:21:30.223 INFO:teuthology.orchestra.run.smithi043.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T22:21:30.223 INFO:teuthology.orchestra.run.smithi043.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T22:21:30.223 INFO:teuthology.orchestra.run.smithi043.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T22:21:30.223 INFO:teuthology.orchestra.run.smithi043.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T22:21:30.223 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T22:21:30.224 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T22:21:30.224 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T22:21:30.224 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T22:21:30.224 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T22:21:30.224 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T22:21:30.224 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T22:21:30.224 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T22:21:30.224 INFO:teuthology.orchestra.run.smithi043.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T22:21:30.224 INFO:teuthology.orchestra.run.smithi043.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T22:21:30.224 INFO:teuthology.orchestra.run.smithi043.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T22:21:30.224 INFO:teuthology.orchestra.run.smithi043.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T22:21:30.224 INFO:teuthology.orchestra.run.smithi043.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T22:21:30.225 INFO:teuthology.orchestra.run.smithi043.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T22:21:30.225 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T22:21:30.225 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T22:21:30.225 INFO:teuthology.orchestra.run.smithi043.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T22:21:30.225 INFO:teuthology.orchestra.run.smithi043.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T22:21:30.225 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T22:21:30.226 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T22:21:30.226 INFO:teuthology.orchestra.run.smithi043.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T22:21:30.226 INFO:teuthology.orchestra.run.smithi043.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T22:21:30.226 INFO:teuthology.orchestra.run.smithi043.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T22:21:30.226 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T22:21:30.226 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T22:21:30.226 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T22:21:30.226 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T22:21:30.226 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T22:21:30.226 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T22:21:30.226 INFO:teuthology.orchestra.run.smithi043.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T22:21:30.226 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T22:21:30.226 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T22:21:30.227 INFO:teuthology.orchestra.run.smithi043.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T22:21:30.227 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T22:21:30.227 INFO:teuthology.orchestra.run.smithi043.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T22:21:30.227 INFO:teuthology.orchestra.run.smithi043.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T22:21:30.227 INFO:teuthology.orchestra.run.smithi043.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T22:21:30.227 INFO:teuthology.orchestra.run.smithi043.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T22:21:30.227 INFO:teuthology.orchestra.run.smithi043.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T22:21:30.227 INFO:teuthology.orchestra.run.smithi043.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T22:21:30.227 INFO:teuthology.orchestra.run.smithi043.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T22:21:30.227 INFO:teuthology.orchestra.run.smithi043.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T22:21:30.227 INFO:teuthology.orchestra.run.smithi043.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T22:21:30.227 INFO:teuthology.orchestra.run.smithi043.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T22:21:30.227 INFO:teuthology.orchestra.run.smithi043.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-28T22:21:30.227 INFO:teuthology.orchestra.run.smithi043.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-28T22:21:30.227 INFO:teuthology.orchestra.run.smithi043.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T22:21:30.228 INFO:teuthology.orchestra.run.smithi043.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T22:21:30.228 INFO:teuthology.orchestra.run.smithi043.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T22:21:30.228 INFO:teuthology.orchestra.run.smithi043.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T22:21:30.228 INFO:teuthology.orchestra.run.smithi043.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T22:21:30.228 INFO:teuthology.orchestra.run.smithi043.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T22:21:30.228 INFO:teuthology.orchestra.run.smithi043.stdout:Installing weak dependencies: 2024-03-28T22:21:30.228 INFO:teuthology.orchestra.run.smithi043.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-28T22:21:30.228 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:21:30.228 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-03-28T22:21:30.228 INFO:teuthology.orchestra.run.smithi043.stdout:============================================================================================ 2024-03-28T22:21:30.228 INFO:teuthology.orchestra.run.smithi043.stdout:Install 150 Packages 2024-03-28T22:21:30.228 INFO:teuthology.orchestra.run.smithi043.stdout:Upgrade 24 Packages 2024-03-28T22:21:30.228 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:21:30.229 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T22:21:30.229 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 226 M 2024-03-28T22:21:30.229 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-03-28T22:21:30.264 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T22:21:30.328 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T22:21:30.408 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T22:21:30.427 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-28T22:21:30.535 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T22:21:30.569 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T22:21:30.600 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T22:21:30.609 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T22:21:30.624 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T22:21:30.659 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T22:21:30.661 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T22:21:30.707 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T22:21:30.711 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T22:21:30.743 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T22:21:30.745 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T22:21:30.771 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T22:21:30.787 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T22:21:30.805 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T22:21:30.828 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T22:21:30.962 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T22:21:31.017 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T22:21:31.047 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T22:21:31.088 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T22:21:31.133 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T22:21:31.180 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : librados2-2:19.0.0-1899.g323d23dc.el9.x86_64 10/198 2024-03-28T22:21:31.192 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T22:21:31.230 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T22:21:31.270 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T22:21:31.291 INFO:teuthology.orchestra.run.smithi043.stdout:(1/174): ceph-19.0.0-1899.g323d23dc.el9.x86_64. 24 kB/s | 6.6 kB 00:00 2024-03-28T22:21:31.295 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T22:21:31.301 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librados2-2:19.0.0-1899.g323d23dc.el9.x86_64 10/198 2024-03-28T22:21:31.320 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T22:21:31.331 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephfs2-2:19.0.0-1899.g323d23dc.el9.x86_64 11/198 2024-03-28T22:21:31.401 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T22:21:31.423 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libcephfs2-2:19.0.0-1899.g323d23dc.el9.x86_64 11/198 2024-03-28T22:21:31.452 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rados-2:19.0.0-1899.g323d23dc.el9.x86_64 12/198 2024-03-28T22:21:31.483 INFO:teuthology.orchestra.run.smithi043.stdout:(2/174): ceph-fuse-19.0.0-1899.g323d23dc.el9.x8 4.5 MB/s | 889 kB 00:00 2024-03-28T22:21:31.487 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T22:21:31.521 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T22:21:31.533 INFO:teuthology.orchestra.run.smithi043.stdout:(3/174): ceph-immutable-object-cache-19.0.0-189 3.0 MB/s | 145 kB 00:00 2024-03-28T22:21:31.621 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T22:21:31.652 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T22:21:31.670 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T22:21:31.700 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T22:21:31.727 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T22:21:31.728 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephsqlite-2:19.0.0-1899.g323d23dc.el9.x86_64 16/198 2024-03-28T22:21:31.775 INFO:teuthology.orchestra.run.smithi043.stdout:(4/174): ceph-mds-19.0.0-1899.g323d23dc.el9.x86 9.8 MB/s | 2.4 MB 00:00 2024-03-28T22:21:31.783 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T22:21:31.815 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1899.g323d23dc.el9.x86_64 16/198 2024-03-28T22:21:31.837 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libradosstriper1-2:19.0.0-1899.g323d23dc.el9.x86 17/198 2024-03-28T22:21:31.893 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1899.g323d23dc.el9.x86 17/198 2024-03-28T22:21:31.932 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T22:21:31.956 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T22:21:31.966 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T22:21:31.971 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T22:21:31.984 INFO:teuthology.orchestra.run.smithi043.stdout:(5/174): ceph-base-19.0.0-1899.g323d23dc.el9.x8 5.6 MB/s | 5.3 MB 00:00 2024-03-28T22:21:32.034 INFO:teuthology.orchestra.run.smithi043.stdout:(6/174): ceph-mgr-19.0.0-1899.g323d23dc.el9.x86 5.8 MB/s | 1.5 MB 00:00 2024-03-28T22:21:32.041 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T22:21:32.147 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T22:21:32.170 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T22:21:32.179 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T22:21:32.202 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T22:21:32.215 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T22:21:32.247 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T22:21:32.277 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T22:21:32.305 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T22:21:32.307 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T22:21:32.348 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T22:21:32.387 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T22:21:32.397 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T22:21:32.419 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T22:21:32.448 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T22:21:32.456 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T22:21:32.468 INFO:teuthology.orchestra.run.smithi043.stdout:(7/174): ceph-mon-19.0.0-1899.g323d23dc.el9.x86 9.7 MB/s | 4.7 MB 00:00 2024-03-28T22:21:32.524 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T22:21:32.527 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T22:21:32.559 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T22:21:32.591 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T22:21:32.592 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T22:21:32.634 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T22:21:32.668 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T22:21:32.757 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T22:21:32.800 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T22:21:32.848 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T22:21:32.891 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T22:21:32.913 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T22:21:32.922 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T22:21:32.969 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T22:21:33.027 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T22:21:33.063 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T22:21:33.069 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T22:21:33.089 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T22:21:33.125 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T22:21:33.132 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T22:21:33.198 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T22:21:33.253 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T22:21:33.291 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T22:21:33.355 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T22:21:33.390 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T22:21:33.443 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T22:21:33.475 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T22:21:33.484 INFO:teuthology.orchestra.run.smithi043.stdout:(8/174): ceph-osd-19.0.0-1899.g323d23dc.el9.x86 12 MB/s | 17 MB 00:01 2024-03-28T22:21:33.503 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T22:21:33.573 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T22:21:33.609 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ceph-common-2:19.0.0-1899.g323d23dc.el9. 39/198 2024-03-28T22:21:33.643 INFO:teuthology.orchestra.run.smithi043.stdout:(9/174): ceph-radosgw-19.0.0-1899.g323d23dc.el9 8.3 MB/s | 9.8 MB 00:01 2024-03-28T22:21:33.651 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ceph-argparse-2:19.0.0-1899.g323d23dc.el 40/198 2024-03-28T22:21:33.660 INFO:teuthology.orchestra.run.smithi043.stdout:(10/174): ceph-selinux-19.0.0-1899.g323d23dc.el 144 kB/s | 25 kB 00:00 2024-03-28T22:21:33.680 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cephfs-2:19.0.0-1899.g323d23dc.el9.x86_6 41/198 2024-03-28T22:21:33.718 INFO:teuthology.orchestra.run.smithi043.stdout:(11/174): cephfs-mirror-19.0.0-1899.g323d23dc.e 3.8 MB/s | 221 kB 00:00 2024-03-28T22:21:33.749 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T22:21:33.785 INFO:teuthology.orchestra.run.smithi043.stdout:(12/174): cephfs-shell-19.0.0-1899.g323d23dc.el 411 kB/s | 27 kB 00:00 2024-03-28T22:21:33.795 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T22:21:33.844 INFO:teuthology.orchestra.run.smithi043.stdout:(13/174): libcephfs-devel-19.0.0-1899.g323d23dc 552 kB/s | 32 kB 00:00 2024-03-28T22:21:33.870 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T22:21:33.870 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T22:21:33.912 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T22:21:33.946 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T22:21:34.007 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T22:21:34.022 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T22:21:34.053 INFO:teuthology.orchestra.run.smithi043.stdout:(14/174): libcephfs2-19.0.0-1899.g323d23dc.el9. 3.5 MB/s | 746 kB 00:00 2024-03-28T22:21:34.056 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T22:21:34.273 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T22:21:34.511 INFO:teuthology.orchestra.run.smithi043.stdout:(15/174): ceph-common-19.0.0-1899.g323d23dc.el9 5.9 MB/s | 21 MB 00:03 2024-03-28T22:21:34.514 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T22:21:34.545 INFO:teuthology.orchestra.run.smithi043.stdout:(16/174): libcephsqlite-19.0.0-1899.g323d23dc.e 333 kB/s | 163 kB 00:00 2024-03-28T22:21:34.569 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T22:21:34.578 INFO:teuthology.orchestra.run.smithi043.stdout:(17/174): librados-devel-19.0.0-1899.g323d23dc. 1.9 MB/s | 127 kB 00:00 2024-03-28T22:21:34.629 INFO:teuthology.orchestra.run.smithi043.stdout:(18/174): libradosstriper1-19.0.0-1899.g323d23d 5.9 MB/s | 501 kB 00:00 2024-03-28T22:21:34.642 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T22:21:34.696 INFO:teuthology.orchestra.run.smithi043.stdout:(19/174): python3-ceph-argparse-19.0.0-1899.g32 689 kB/s | 45 kB 00:00 2024-03-28T22:21:34.702 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T22:21:34.713 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T22:21:34.736 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T22:21:34.771 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T22:21:34.779 INFO:teuthology.orchestra.run.smithi043.stdout:(20/174): python3-ceph-common-19.0.0-1899.g323d 1.6 MB/s | 131 kB 00:00 2024-03-28T22:21:34.781 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T22:21:34.811 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T22:21:34.814 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T22:21:34.851 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T22:21:34.871 INFO:teuthology.orchestra.run.smithi043.stdout:(21/174): python3-cephfs-19.0.0-1899.g323d23dc. 1.7 MB/s | 162 kB 00:00 2024-03-28T22:21:34.890 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librados-devel-2:19.0.0-1899.g323d23dc.el9.x86_6 53/198 2024-03-28T22:21:34.919 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T22:21:35.022 INFO:teuthology.orchestra.run.smithi043.stdout:(22/174): librgw2-19.0.0-1899.g323d23dc.el9.x86 12 MB/s | 5.1 MB 00:00 2024-03-28T22:21:35.064 INFO:teuthology.orchestra.run.smithi043.stdout:(23/174): python3-rados-19.0.0-1899.g323d23dc.e 1.6 MB/s | 321 kB 00:00 2024-03-28T22:21:35.106 INFO:teuthology.orchestra.run.smithi043.stdout:(24/174): python3-rbd-19.0.0-1899.g323d23dc.el9 3.5 MB/s | 299 kB 00:00 2024-03-28T22:21:35.131 INFO:teuthology.orchestra.run.smithi043.stdout:(25/174): python3-rgw-19.0.0-1899.g323d23dc.el9 1.5 MB/s | 100 kB 00:00 2024-03-28T22:21:35.150 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T22:21:35.173 INFO:teuthology.orchestra.run.smithi043.stdout:(26/174): rbd-fuse-19.0.0-1899.g323d23dc.el9.x8 1.3 MB/s | 86 kB 00:00 2024-03-28T22:21:35.248 INFO:teuthology.orchestra.run.smithi043.stdout:(27/174): rbd-nbd-19.0.0-1899.g323d23dc.el9.x86 2.3 MB/s | 171 kB 00:00 2024-03-28T22:21:35.303 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T22:21:35.332 INFO:teuthology.orchestra.run.smithi043.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1899.g 379 kB/s | 31 kB 00:00 2024-03-28T22:21:35.364 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T22:21:35.408 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T22:21:35.416 INFO:teuthology.orchestra.run.smithi043.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1899.g323d23d 1.6 MB/s | 138 kB 00:00 2024-03-28T22:21:35.462 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T22:21:35.541 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T22:21:35.594 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T22:21:35.598 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T22:21:35.649 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T22:21:35.722 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T22:21:35.741 INFO:teuthology.orchestra.run.smithi043.stdout:(30/174): rbd-mirror-19.0.0-1899.g323d23dc.el9. 5.1 MB/s | 3.1 MB 00:00 2024-03-28T22:21:35.753 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T22:21:35.785 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T22:21:35.831 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T22:21:35.849 INFO:teuthology.orchestra.run.smithi043.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1899.g323d2 8.1 MB/s | 3.5 MB 00:00 2024-03-28T22:21:35.867 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T22:21:35.910 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T22:21:35.925 INFO:teuthology.orchestra.run.smithi043.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1899.g32 3.3 MB/s | 248 kB 00:00 2024-03-28T22:21:35.953 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T22:21:35.992 INFO:teuthology.orchestra.run.smithi043.stdout:(33/174): ceph-mgr-rook-19.0.0-1899.g323d23dc.e 751 kB/s | 49 kB 00:00 2024-03-28T22:21:36.067 INFO:teuthology.orchestra.run.smithi043.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1899.g3 207 kB/s | 15 kB 00:00 2024-03-28T22:21:36.142 INFO:teuthology.orchestra.run.smithi043.stdout:(35/174): ceph-volume-19.0.0-1899.g323d23dc.el9 3.5 MB/s | 264 kB 00:00 2024-03-28T22:21:36.251 INFO:teuthology.orchestra.run.smithi043.stdout:(36/174): cephadm-19.0.0-1899.g323d23dc.el9.noa 6.8 MB/s | 748 kB 00:00 2024-03-28T22:21:36.251 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T22:21:36.318 INFO:teuthology.orchestra.run.smithi043.stdout:(37/174): cephfs-top-19.0.0-1899.g323d23dc.el9. 277 kB/s | 18 kB 00:00 2024-03-28T22:21:36.475 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T22:21:36.501 INFO:teuthology.orchestra.run.smithi043.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 3.5 MB/s | 662 kB 00:00 2024-03-28T22:21:36.505 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T22:21:36.546 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T22:21:36.560 INFO:teuthology.orchestra.run.smithi043.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 16 MB/s | 920 kB 00:00 2024-03-28T22:21:36.602 INFO:teuthology.orchestra.run.smithi043.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 106 kB 00:00 2024-03-28T22:21:36.641 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T22:21:36.644 INFO:teuthology.orchestra.run.smithi043.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.4 MB/s | 307 kB 00:00 2024-03-28T22:21:36.669 INFO:teuthology.orchestra.run.smithi043.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 1.3 MB/s | 32 kB 00:00 2024-03-28T22:21:36.680 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T22:21:36.721 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T22:21:36.747 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T22:21:36.771 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T22:21:36.778 INFO:teuthology.orchestra.run.smithi043.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 28 MB/s | 3.0 MB 00:00 2024-03-28T22:21:36.803 INFO:teuthology.orchestra.run.smithi043.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 712 kB/s | 17 kB 00:00 2024-03-28T22:21:36.813 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T22:21:36.828 INFO:teuthology.orchestra.run.smithi043.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 419 kB/s | 10 kB 00:00 2024-03-28T22:21:36.842 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T22:21:36.870 INFO:teuthology.orchestra.run.smithi043.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.8 MB/s | 159 kB 00:00 2024-03-28T22:21:36.895 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T22:21:36.926 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T22:21:36.957 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T22:21:36.980 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T22:21:36.984 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T22:21:36.996 INFO:teuthology.orchestra.run.smithi043.stdout:(47/174): ceph-mgr-diskprediction-local-19.0.0- 5.9 MB/s | 7.4 MB 00:01 2024-03-28T22:21:37.013 INFO:teuthology.orchestra.run.smithi043.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 313 kB/s | 44 kB 00:00 2024-03-28T22:21:37.055 INFO:teuthology.orchestra.run.smithi043.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-28T22:21:37.096 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T22:21:37.122 INFO:teuthology.orchestra.run.smithi043.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 5.5 MB/s | 578 kB 00:00 2024-03-28T22:21:37.146 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T22:21:37.147 INFO:teuthology.orchestra.run.smithi043.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 539 kB/s | 46 kB 00:00 2024-03-28T22:21:37.181 INFO:teuthology.orchestra.run.smithi043.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2024-03-28T22:21:37.194 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T22:21:37.214 INFO:teuthology.orchestra.run.smithi043.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 2.9 MB/s | 187 kB 00:00 2024-03-28T22:21:37.248 INFO:teuthology.orchestra.run.smithi043.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.3 MB/s | 217 kB 00:00 2024-03-28T22:21:37.250 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T22:21:37.273 INFO:teuthology.orchestra.run.smithi043.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 612 kB/s | 34 kB 00:00 2024-03-28T22:21:37.296 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T22:21:37.331 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : librbd1-2:19.0.0-1899.g323d23dc.el9.x86_64 124/198 2024-03-28T22:21:37.422 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librbd1-2:19.0.0-1899.g323d23dc.el9.x86_64 124/198 2024-03-28T22:21:37.424 INFO:teuthology.orchestra.run.smithi043.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 27 MB/s | 4.7 MB 00:00 2024-03-28T22:21:37.433 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T22:21:37.452 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rbd-2:19.0.0-1899.g323d23dc.el9.x86_64 125/198 2024-03-28T22:21:37.474 INFO:teuthology.orchestra.run.smithi043.stdout:(57/174): postgresql-13.14-1.el9.x86_64.rpm 8.3 MB/s | 1.5 MB 00:00 2024-03-28T22:21:37.722 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T22:21:37.723 INFO:teuthology.orchestra.run.smithi043.stdout:(58/174): postgresql-contrib-13.14-1.el9.x86_64 8.8 MB/s | 817 kB 00:00 2024-03-28T22:21:37.723 INFO:teuthology.orchestra.run.smithi043.stdout:(59/174): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 132 kB 00:00 2024-03-28T22:21:37.723 INFO:teuthology.orchestra.run.smithi043.stdout:(60/174): postgresql-server-13.14-1.el9.x86_64. 33 MB/s | 5.7 MB 00:00 2024-03-28T22:21:37.723 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T22:21:37.723 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T22:21:37.723 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T22:21:37.723 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T22:21:37.723 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T22:21:37.733 INFO:teuthology.orchestra.run.smithi043.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.7 MB/s | 1.0 MB 00:00 2024-03-28T22:21:37.748 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T22:21:37.782 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T22:21:37.792 INFO:teuthology.orchestra.run.smithi043.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-28T22:21:37.821 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T22:21:37.853 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T22:21:37.883 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T22:21:37.926 INFO:teuthology.orchestra.run.smithi043.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 30 MB/s | 5.8 MB 00:00 2024-03-28T22:21:37.950 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T22:21:37.951 INFO:teuthology.orchestra.run.smithi043.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 171 kB/s | 24 kB 00:00 2024-03-28T22:21:37.959 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T22:21:37.984 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librgw2-2:19.0.0-1899.g323d23dc.el9.x86_64 128/198 2024-03-28T22:21:37.993 INFO:teuthology.orchestra.run.smithi043.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 447 kB/s | 18 kB 00:00 2024-03-28T22:21:38.004 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T22:21:38.018 INFO:teuthology.orchestra.run.smithi043.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 144 kB 00:00 2024-03-28T22:21:38.034 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T22:21:38.038 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librgw2-2:19.0.0-1899.g323d23dc.el9.x86_64 128/198 2024-03-28T22:21:38.060 INFO:teuthology.orchestra.run.smithi043.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 780 kB/s | 32 kB 00:00 2024-03-28T22:21:38.075 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rgw-2:19.0.0-1899.g323d23dc.el9.x86_64 129/198 2024-03-28T22:21:38.076 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T22:21:38.110 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T22:21:38.194 INFO:teuthology.orchestra.run.smithi043.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 28 MB/s | 4.8 MB 00:00 2024-03-28T22:21:38.199 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T22:21:38.218 INFO:teuthology.orchestra.run.smithi043.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 217 kB 00:00 2024-03-28T22:21:38.261 INFO:teuthology.orchestra.run.smithi043.stdout:(70/174): python3-packaging-20.9-5.el9.noarch.r 1.7 MB/s | 69 kB 00:00 2024-03-28T22:21:38.282 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T22:21:38.286 INFO:teuthology.orchestra.run.smithi043.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 2.6 MB/s | 174 kB 00:00 2024-03-28T22:21:38.311 INFO:teuthology.orchestra.run.smithi043.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 859 kB/s | 41 kB 00:00 2024-03-28T22:21:38.333 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T22:21:38.345 INFO:teuthology.orchestra.run.smithi043.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.1 MB/s | 238 kB 00:00 2024-03-28T22:21:38.365 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T22:21:38.370 INFO:teuthology.orchestra.run.smithi043.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 132 kB 00:00 2024-03-28T22:21:38.404 INFO:teuthology.orchestra.run.smithi043.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.6 MB/s | 211 kB 00:00 2024-03-28T22:21:38.412 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T22:21:38.421 INFO:teuthology.orchestra.run.smithi043.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 995 kB/s | 47 kB 00:00 2024-03-28T22:21:38.438 INFO:teuthology.orchestra.run.smithi043.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 43 kB 00:00 2024-03-28T22:21:38.510 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T22:21:38.557 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T22:21:38.622 INFO:teuthology.orchestra.run.smithi043.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 231 kB/s | 39 kB 00:00 2024-03-28T22:21:38.814 INFO:teuthology.orchestra.run.smithi043.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 40 MB/s | 16 MB 00:00 2024-03-28T22:21:38.830 INFO:teuthology.orchestra.run.smithi043.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 202 kB/s | 41 kB 00:00 2024-03-28T22:21:38.864 INFO:teuthology.orchestra.run.smithi043.stdout:(81/174): qatlib-24.02.0-1.el9.x86_64.rpm 4.4 MB/s | 220 kB 00:00 2024-03-28T22:21:38.881 INFO:teuthology.orchestra.run.smithi043.stdout:(82/174): qatlib-service-24.02.0-1.el9.x86_64.r 733 kB/s | 35 kB 00:00 2024-03-28T22:21:38.906 INFO:teuthology.orchestra.run.smithi043.stdout:(83/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-28T22:21:38.940 INFO:teuthology.orchestra.run.smithi043.stdout:(84/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 300 kB 00:00 2024-03-28T22:21:38.965 INFO:teuthology.orchestra.run.smithi043.stdout:(85/174): uuid-1.6.2-55.el9.x86_64.rpm 1.0 MB/s | 56 kB 00:00 2024-03-28T22:21:38.991 INFO:teuthology.orchestra.run.smithi043.stdout:(86/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-28T22:21:39.016 INFO:teuthology.orchestra.run.smithi043.stdout:(87/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 902 kB/s | 44 kB 00:00 2024-03-28T22:21:39.041 INFO:teuthology.orchestra.run.smithi043.stdout:(88/174): fuse-2.9.9-15.el9.x86_64.rpm 1.6 MB/s | 79 kB 00:00 2024-03-28T22:21:39.075 INFO:teuthology.orchestra.run.smithi043.stdout:(89/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.1 MB/s | 241 kB 00:00 2024-03-28T22:21:39.150 INFO:teuthology.orchestra.run.smithi043.stdout:(90/174): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.1 MB 00:00 2024-03-28T22:21:39.175 INFO:teuthology.orchestra.run.smithi043.stdout:(91/174): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-28T22:21:39.201 INFO:teuthology.orchestra.run.smithi043.stdout:(92/174): python3-pycparser-2.20-6.el9.noarch.r 2.5 MB/s | 124 kB 00:00 2024-03-28T22:21:39.234 INFO:teuthology.orchestra.run.smithi043.stdout:(93/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.2 MB/s | 187 kB 00:00 2024-03-28T22:21:39.271 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T22:21:39.322 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T22:21:39.356 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T22:21:39.404 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T22:21:39.485 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T22:21:39.532 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T22:21:39.709 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T22:21:39.726 INFO:teuthology.orchestra.run.smithi043.stdout:(94/174): ceph-test-19.0.0-1899.g323d23dc.el9.x 7.8 MB/s | 47 MB 00:06 2024-03-28T22:21:39.738 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T22:21:39.815 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T22:21:40.326 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 130/198 2024-03-28T22:21:40.357 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 130/198 2024-03-28T22:21:40.444 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T22:21:40.551 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T22:21:40.568 INFO:teuthology.orchestra.run.smithi043.stdout:(95/174): jq-1.6-16.el9.x86_64.rpm 137 kB/s | 187 kB 00:01 2024-03-28T22:21:40.669 INFO:teuthology.orchestra.run.smithi043.stdout:(96/174): libquadmath-11.4.1-3.el9.x86_64.rpm 208 kB/s | 195 kB 00:00 2024-03-28T22:21:40.761 INFO:teuthology.orchestra.run.smithi043.stdout:(97/174): elfutils-libelf-devel-0.190-2.el9.x86 240 kB/s | 21 kB 00:00 2024-03-28T22:21:40.819 INFO:teuthology.orchestra.run.smithi043.stdout:(98/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 445 kB/s | 25 kB 00:00 2024-03-28T22:21:40.896 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 130/198 2024-03-28T22:21:40.935 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-base-2:19.0.0-1899.g323d23dc.el9.x86_64 131/198 2024-03-28T22:21:41.000 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-base-2:19.0.0-1899.g323d23dc.el9.x86_64 131/198 2024-03-28T22:21:41.000 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T22:21:41.000 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T22:21:41.000 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-28T22:21:41.028 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-selinux-2:19.0.0-1899.g323d23dc.el9.x86_64 132/198 2024-03-28T22:21:41.444 INFO:teuthology.orchestra.run.smithi043.stdout:(99/174): libgfortran-11.4.1-3.el9.x86_64.rpm 364 kB/s | 803 kB 00:02 2024-03-28T22:21:41.478 INFO:teuthology.orchestra.run.smithi043.stdout:(100/174): python3-requests-2.25.1-8.el9.noarch 137 kB/s | 125 kB 00:00 2024-03-28T22:21:41.687 INFO:teuthology.orchestra.run.smithi043.stdout:(101/174): openssl-devel-3.0.7-27.el9.x86_64.rp 4.8 MB/s | 4.1 MB 00:00 2024-03-28T22:21:41.712 INFO:teuthology.orchestra.run.smithi043.stdout:(102/174): python3-devel-3.9.17-2.el9.x86_64.rp 932 kB/s | 245 kB 00:00 2024-03-28T22:21:41.745 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T22:21:41.770 INFO:teuthology.orchestra.run.smithi043.stdout:(103/174): python3-jinja2-2.11.3-5.el9.noarch.r 851 kB/s | 248 kB 00:00 2024-03-28T22:21:41.795 INFO:teuthology.orchestra.run.smithi043.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.1 MB/s | 94 kB 00:00 2024-03-28T22:21:41.834 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T22:21:41.861 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T22:21:41.884 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T22:21:41.904 INFO:teuthology.orchestra.run.smithi043.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 117 kB/s | 22 kB 00:00 2024-03-28T22:21:41.912 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T22:21:41.938 INFO:teuthology.orchestra.run.smithi043.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 3.8 MB/s | 548 kB 00:00 2024-03-28T22:21:41.980 INFO:teuthology.orchestra.run.smithi043.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 7.4 MB/s | 309 kB 00:00 2024-03-28T22:21:42.005 INFO:teuthology.orchestra.run.smithi043.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 296 kB/s | 19 kB 00:00 2024-03-28T22:21:42.047 INFO:teuthology.orchestra.run.smithi043.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 637 kB/s | 26 kB 00:00 2024-03-28T22:21:42.105 INFO:teuthology.orchestra.run.smithi043.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 846 kB/s | 49 kB 00:00 2024-03-28T22:21:42.139 INFO:teuthology.orchestra.run.smithi043.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-03-28T22:21:42.225 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T22:21:42.259 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : librbd1-2:19.0.0-1899.g323d23dc.el9.x86_64 124/198 2024-03-28T22:21:42.340 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librbd1-2:19.0.0-1899.g323d23dc.el9.x86_64 124/198 2024-03-28T22:21:42.348 INFO:teuthology.orchestra.run.smithi043.stdout:(112/174): libarrow-9.0.0-10.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-03-28T22:21:42.371 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rbd-2:19.0.0-1899.g323d23dc.el9.x86_64 125/198 2024-03-28T22:21:42.373 INFO:teuthology.orchestra.run.smithi043.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 648 kB/s | 151 kB 00:00 2024-03-28T22:21:42.408 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T22:21:42.423 INFO:teuthology.orchestra.run.smithi043.stdout:(114/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.3 MB/s | 863 kB 00:00 2024-03-28T22:21:42.474 INFO:teuthology.orchestra.run.smithi043.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.6 MB/s | 838 kB 00:00 2024-03-28T22:21:42.516 INFO:teuthology.orchestra.run.smithi043.stdout:(116/174): python3-asyncssh-2.13.2-5.el9.noarch 3.8 MB/s | 548 kB 00:00 2024-03-28T22:21:42.533 INFO:teuthology.orchestra.run.smithi043.stdout:(117/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 402 kB/s | 43 kB 00:00 2024-03-28T22:21:42.549 INFO:teuthology.orchestra.run.smithi043.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 459 kB/s | 15 kB 00:00 2024-03-28T22:21:42.566 INFO:teuthology.orchestra.run.smithi043.stdout:(119/174): python3-cachetools-4.2.4-1.el9.noarc 352 kB/s | 32 kB 00:00 2024-03-28T22:21:42.600 INFO:teuthology.orchestra.run.smithi043.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-03-28T22:21:42.642 INFO:teuthology.orchestra.run.smithi043.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-03-28T22:21:42.667 INFO:teuthology.orchestra.run.smithi043.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 534 kB/s | 35 kB 00:00 2024-03-28T22:21:42.709 INFO:teuthology.orchestra.run.smithi043.stdout:(123/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.1 MB/s | 303 kB 00:00 2024-03-28T22:21:42.743 INFO:teuthology.orchestra.run.smithi043.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 2.0 MB/s | 203 kB 00:00 2024-03-28T22:21:42.776 INFO:teuthology.orchestra.run.smithi043.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 2.1 MB/s | 144 kB 00:00 2024-03-28T22:21:42.802 INFO:teuthology.orchestra.run.smithi043.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 185 kB/s | 11 kB 00:00 2024-03-28T22:21:42.827 INFO:teuthology.orchestra.run.smithi043.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 358 kB/s | 18 kB 00:00 2024-03-28T22:21:42.854 INFO:teuthology.orchestra.run.smithi043.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 470 kB/s | 23 kB 00:00 2024-03-28T22:21:42.862 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T22:21:42.877 INFO:teuthology.orchestra.run.smithi043.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 394 kB/s | 19 kB 00:00 2024-03-28T22:21:42.895 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librgw2-2:19.0.0-1899.g323d23dc.el9.x86_64 128/198 2024-03-28T22:21:42.903 INFO:teuthology.orchestra.run.smithi043.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 398 kB/s | 20 kB 00:00 2024-03-28T22:21:42.936 INFO:teuthology.orchestra.run.smithi043.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 1.4 MB/s | 46 kB 00:00 2024-03-28T22:21:42.965 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librgw2-2:19.0.0-1899.g323d23dc.el9.x86_64 128/198 2024-03-28T22:21:42.978 INFO:teuthology.orchestra.run.smithi043.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 1.9 MB/s | 79 kB 00:00 2024-03-28T22:21:42.988 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rgw-2:19.0.0-1899.g323d23dc.el9.x86_64 129/198 2024-03-28T22:21:43.020 INFO:teuthology.orchestra.run.smithi043.stdout:(133/174): python3-kubernetes-26.1.0-2.el9.noar 7.2 MB/s | 1.0 MB 00:00 2024-03-28T22:21:43.049 INFO:teuthology.orchestra.run.smithi043.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 831 kB/s | 58 kB 00:00 2024-03-28T22:21:43.082 INFO:teuthology.orchestra.run.smithi043.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 4.4 MB/s | 272 kB 00:00 2024-03-28T22:21:43.108 INFO:teuthology.orchestra.run.smithi043.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 284 kB/s | 16 kB 00:00 2024-03-28T22:21:43.133 INFO:teuthology.orchestra.run.smithi043.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-03-28T22:21:43.167 INFO:teuthology.orchestra.run.smithi043.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 486 kB/s | 28 kB 00:00 2024-03-28T22:21:43.192 INFO:teuthology.orchestra.run.smithi043.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 530 kB/s | 31 kB 00:00 2024-03-28T22:21:43.251 INFO:teuthology.orchestra.run.smithi043.stdout:(140/174): python3-grpcio-1.46.7-10.el9.x86_64. 3.5 MB/s | 2.0 MB 00:00 2024-03-28T22:21:43.276 INFO:teuthology.orchestra.run.smithi043.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 714 kB/s | 59 kB 00:00 2024-03-28T22:21:43.302 INFO:teuthology.orchestra.run.smithi043.stdout:(142/174): python3-routes-2.5.1-5.el9.noarch.rp 1.4 MB/s | 188 kB 00:00 2024-03-28T22:21:43.327 INFO:teuthology.orchestra.run.smithi043.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 478 kB/s | 36 kB 00:00 2024-03-28T22:21:43.352 INFO:teuthology.orchestra.run.smithi043.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 681 kB/s | 51 kB 00:00 2024-03-28T22:21:43.411 INFO:teuthology.orchestra.run.smithi043.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-28T22:21:43.436 INFO:teuthology.orchestra.run.smithi043.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 834 kB/s | 90 kB 00:00 2024-03-28T22:21:43.469 INFO:teuthology.orchestra.run.smithi043.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-03-28T22:21:43.486 INFO:teuthology.orchestra.run.smithi043.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 407 kB/s | 20 kB 00:00 2024-03-28T22:21:43.520 INFO:teuthology.orchestra.run.smithi043.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.3 MB/s | 192 kB 00:00 2024-03-28T22:21:43.587 INFO:teuthology.orchestra.run.smithi043.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-03-28T22:21:43.679 INFO:teuthology.orchestra.run.smithi043.stdout:(151/174): NetworkManager-1.47.2-1.el9.x86_64.r 25 MB/s | 2.3 MB 00:00 2024-03-28T22:21:43.763 INFO:teuthology.orchestra.run.smithi043.stdout:(152/174): NetworkManager-libnm-1.47.2-1.el9.x8 22 MB/s | 1.8 MB 00:00 2024-03-28T22:21:43.804 INFO:teuthology.orchestra.run.smithi043.stdout:(153/174): NetworkManager-team-1.47.2-1.el9.x86 1.0 MB/s | 41 kB 00:00 2024-03-28T22:21:43.866 INFO:teuthology.orchestra.run.smithi043.stdout:(154/174): NetworkManager-tui-1.47.2-1.el9.x86_ 4.0 MB/s | 247 kB 00:00 2024-03-28T22:21:43.914 INFO:teuthology.orchestra.run.smithi043.stdout:(155/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.8 MB/s | 321 kB 00:00 2024-03-28T22:21:44.047 INFO:teuthology.orchestra.run.smithi043.stdout:(156/174): librbd1-19.0.0-1899.g323d23dc.el9.x8 6.0 MB/s | 3.1 MB 00:00 2024-03-28T22:21:44.139 INFO:teuthology.orchestra.run.smithi043.stdout:(157/174): librados2-19.0.0-1899.g323d23dc.el9. 5.2 MB/s | 3.4 MB 00:00 2024-03-28T22:21:44.789 INFO:teuthology.orchestra.run.smithi043.stdout:(158/174): acl-2.3.1-4.el9.x86_64.rpm 82 kB/s | 71 kB 00:00 2024-03-28T22:21:44.931 INFO:teuthology.orchestra.run.smithi043.stdout:(159/174): elfutils-debuginfod-client-0.190-2.e 41 kB/s | 36 kB 00:00 2024-03-28T22:21:45.169 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 130/198 2024-03-28T22:21:45.227 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 130/198 2024-03-28T22:21:45.281 INFO:teuthology.orchestra.run.smithi043.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 168 kB/s | 192 kB 00:01 2024-03-28T22:21:45.607 INFO:teuthology.orchestra.run.smithi043.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 34 kB/s | 23 kB 00:00 2024-03-28T22:21:45.765 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 130/198 2024-03-28T22:21:45.796 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-base-2:19.0.0-1899.g323d23dc.el9.x86_64 131/198 2024-03-28T22:21:45.824 INFO:teuthology.orchestra.run.smithi043.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 247 kB/s | 256 kB 00:01 2024-03-28T22:21:45.861 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-base-2:19.0.0-1899.g323d23dc.el9.x86_64 131/198 2024-03-28T22:21:45.861 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T22:21:45.861 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T22:21:45.861 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-28T22:21:45.889 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-selinux-2:19.0.0-1899.g323d23dc.el9.x86_64 132/198 2024-03-28T22:21:46.174 INFO:teuthology.orchestra.run.smithi043.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 122 kB/s | 108 kB 00:00 2024-03-28T22:21:46.591 INFO:teuthology.orchestra.run.smithi043.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 158 kB/s | 155 kB 00:00 2024-03-28T22:21:46.724 INFO:teuthology.orchestra.run.smithi043.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 151 kB/s | 136 kB 00:00 2024-03-28T22:21:47.075 INFO:teuthology.orchestra.run.smithi043.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 71 kB/s | 64 kB 00:00 2024-03-28T22:21:47.350 INFO:teuthology.orchestra.run.smithi043.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 38 kB/s | 29 kB 00:00 2024-03-28T22:21:49.108 INFO:teuthology.orchestra.run.smithi043.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 520 kB/s | 1.2 MB 00:02 2024-03-28T22:21:50.325 INFO:teuthology.orchestra.run.smithi043.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 383 kB/s | 465 kB 00:01 2024-03-28T22:21:50.509 INFO:teuthology.orchestra.run.smithi043.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 656 kB/s | 2.2 MB 00:03 2024-03-28T22:21:51.068 INFO:teuthology.orchestra.run.smithi043.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 629 kB/s | 2.3 MB 00:03 2024-03-28T22:21:51.093 INFO:teuthology.orchestra.run.smithi043.stdout:(172/174): util-linux-user-2.37.4-18.el9.x86_64 41 kB/s | 31 kB 00:00 2024-03-28T22:21:51.185 INFO:teuthology.orchestra.run.smithi043.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 822 kB/s | 74 kB 00:00 2024-03-28T22:21:52.696 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1899.g323d23dc.el9.x86_64 132/198 2024-03-28T22:21:52.696 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /sys 2024-03-28T22:21:52.696 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /proc 2024-03-28T22:21:52.696 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /mnt 2024-03-28T22:21:52.696 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /var/tmp 2024-03-28T22:21:52.696 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /home 2024-03-28T22:21:52.697 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /root 2024-03-28T22:21:52.697 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /tmp 2024-03-28T22:21:52.697 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-28T22:21:52.734 INFO:teuthology.orchestra.run.smithi043.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 494 kB/s | 1.1 MB 00:02 2024-03-28T22:21:52.743 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-03-28T22:21:52.744 INFO:teuthology.orchestra.run.smithi043.stdout:Total 10 MB/s | 226 MB 00:22 2024-03-28T22:21:52.857 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-volume-2:19.0.0-1899.g323d23dc.el9.noarch 133/198 2024-03-28T22:21:52.872 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-volume-2:19.0.0-1899.g323d23dc.el9.noarch 133/198 2024-03-28T22:21:52.872 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T22:21:52.872 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T22:21:52.872 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-28T22:21:53.962 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-03-28T22:21:54.363 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-03-28T22:21:54.364 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-03-28T22:21:54.368 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-osd-2:19.0.0-1899.g323d23dc.el9.x86_64 134/198 2024-03-28T22:21:54.393 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-osd-2:19.0.0-1899.g323d23dc.el9.x86_64 134/198 2024-03-28T22:21:54.393 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T22:21:54.393 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T22:21:54.393 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T22:21:54.393 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T22:21:54.394 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-28T22:21:54.621 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mds-2:19.0.0-1899.g323d23dc.el9.x86_64 135/198 2024-03-28T22:21:54.643 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mds-2:19.0.0-1899.g323d23dc.el9.x86_64 135/198 2024-03-28T22:21:54.643 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T22:21:54.643 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T22:21:54.643 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T22:21:54.643 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T22:21:54.643 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-28T22:21:55.088 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mon-2:19.0.0-1899.g323d23dc.el9.x86_64 136/198 2024-03-28T22:21:55.111 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mon-2:19.0.0-1899.g323d23dc.el9.x86_64 136/198 2024-03-28T22:21:55.111 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T22:21:55.111 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T22:21:55.111 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T22:21:55.111 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T22:21:55.111 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-28T22:21:55.198 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T22:21:55.557 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 138/198 2024-03-28T22:21:55.587 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 138/198 2024-03-28T22:21:55.619 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1899.g323d23dc.e 139/198 2024-03-28T22:21:55.679 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1899.g323d23dc. 140/198 2024-03-28T22:21:55.702 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1899.g323d23dc.el9.noa 141/198 2024-03-28T22:21:56.738 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1899.g323d23dc.el9.noa 141/198 2024-03-28T22:21:56.779 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1899.g323d23dc.el9.n 142/198 2024-03-28T22:21:56.918 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-03-28T22:21:56.920 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-03-28T22:21:57.740 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1899.g323d23dc.el9.n 142/198 2024-03-28T22:21:57.771 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1899.g323 143/198 2024-03-28T22:21:57.837 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1899.g323d23dc.el9.x86_64 132/198 2024-03-28T22:21:57.838 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /sys 2024-03-28T22:21:57.838 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /proc 2024-03-28T22:21:57.838 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /mnt 2024-03-28T22:21:57.838 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /var/tmp 2024-03-28T22:21:57.838 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /home 2024-03-28T22:21:57.838 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /root 2024-03-28T22:21:57.838 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /tmp 2024-03-28T22:21:57.838 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-28T22:21:57.869 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1899.g323 143/198 2024-03-28T22:21:58.014 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-volume-2:19.0.0-1899.g323d23dc.el9.noarch 133/198 2024-03-28T22:21:58.025 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1899.g323d23dc.el 144/198 2024-03-28T22:21:58.029 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-volume-2:19.0.0-1899.g323d23dc.el9.noarch 133/198 2024-03-28T22:21:58.029 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T22:21:58.029 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T22:21:58.030 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-28T22:21:58.057 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-2:19.0.0-1899.g323d23dc.el9.x86_64 145/198 2024-03-28T22:21:58.080 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1899.g323d23dc.el9.x86_64 145/198 2024-03-28T22:21:58.080 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T22:21:58.080 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T22:21:58.080 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T22:21:58.080 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T22:21:58.080 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-28T22:21:58.124 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-rook-2:19.0.0-1899.g323d23dc.el9.noarch 146/198 2024-03-28T22:21:58.134 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1899.g323d23dc.el9.noarch 146/198 2024-03-28T22:21:58.180 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-2:19.0.0-1899.g323d23dc.el9.x86_64 147/198 2024-03-28T22:21:59.035 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T22:21:59.068 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-radosgw-2:19.0.0-1899.g323d23dc.el9.x86_64 149/198 2024-03-28T22:21:59.090 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1899.g323d23dc.el9.x86_64 149/198 2024-03-28T22:21:59.090 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T22:21:59.090 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T22:21:59.091 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T22:21:59.091 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T22:21:59.091 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-28T22:21:59.143 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1899.g323d2 150/198 2024-03-28T22:21:59.164 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1899.g323d2 150/198 2024-03-28T22:21:59.165 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T22:21:59.165 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T22:21:59.165 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-28T22:21:59.219 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : cephfs-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 151/198 2024-03-28T22:21:59.240 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 151/198 2024-03-28T22:21:59.240 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T22:21:59.241 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T22:21:59.241 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T22:21:59.241 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T22:21:59.241 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-28T22:21:59.514 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-osd-2:19.0.0-1899.g323d23dc.el9.x86_64 134/198 2024-03-28T22:21:59.539 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-osd-2:19.0.0-1899.g323d23dc.el9.x86_64 134/198 2024-03-28T22:21:59.539 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T22:21:59.539 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T22:21:59.540 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T22:21:59.540 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T22:21:59.540 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-28T22:21:59.544 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 152/198 2024-03-28T22:21:59.567 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 152/198 2024-03-28T22:21:59.567 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T22:21:59.567 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T22:21:59.567 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T22:21:59.567 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T22:21:59.567 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-28T22:21:59.764 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mds-2:19.0.0-1899.g323d23dc.el9.x86_64 135/198 2024-03-28T22:21:59.787 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mds-2:19.0.0-1899.g323d23dc.el9.x86_64 135/198 2024-03-28T22:21:59.787 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T22:21:59.787 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T22:21:59.787 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T22:21:59.787 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T22:21:59.787 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-28T22:22:00.226 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mon-2:19.0.0-1899.g323d23dc.el9.x86_64 136/198 2024-03-28T22:22:00.248 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mon-2:19.0.0-1899.g323d23dc.el9.x86_64 136/198 2024-03-28T22:22:00.248 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T22:22:00.248 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T22:22:00.248 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T22:22:00.248 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T22:22:00.248 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-28T22:22:00.336 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T22:22:00.661 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 138/198 2024-03-28T22:22:00.692 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 138/198 2024-03-28T22:22:00.724 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1899.g323d23dc.e 139/198 2024-03-28T22:22:00.776 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1899.g323d23dc. 140/198 2024-03-28T22:22:00.807 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1899.g323d23dc.el9.noa 141/198 2024-03-28T22:22:01.834 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1899.g323d23dc.el9.noa 141/198 2024-03-28T22:22:01.884 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1899.g323d23dc.el9.n 142/198 2024-03-28T22:22:02.844 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1899.g323d23dc.el9.n 142/198 2024-03-28T22:22:02.876 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1899.g323 143/198 2024-03-28T22:22:02.974 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1899.g323 143/198 2024-03-28T22:22:03.139 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1899.g323d23dc.el 144/198 2024-03-28T22:22:03.170 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-2:19.0.0-1899.g323d23dc.el9.x86_64 145/198 2024-03-28T22:22:03.193 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1899.g323d23dc.el9.x86_64 145/198 2024-03-28T22:22:03.193 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T22:22:03.193 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T22:22:03.193 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T22:22:03.193 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T22:22:03.193 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-28T22:22:03.237 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-rook-2:19.0.0-1899.g323d23dc.el9.noarch 146/198 2024-03-28T22:22:03.248 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1899.g323d23dc.el9.noarch 146/198 2024-03-28T22:22:03.293 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-2:19.0.0-1899.g323d23dc.el9.x86_64 147/198 2024-03-28T22:22:03.998 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-test-2:19.0.0-1899.g323d23dc.el9.x86_64 153/198 2024-03-28T22:22:04.039 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 154/198 2024-03-28T22:22:04.070 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-nbd-2:19.0.0-1899.g323d23dc.el9.x86_64 155/198 2024-03-28T22:22:04.097 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T22:22:04.150 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T22:22:04.181 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-radosgw-2:19.0.0-1899.g323d23dc.el9.x86_64 149/198 2024-03-28T22:22:04.203 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1899.g323d23dc.el9.x86_64 149/198 2024-03-28T22:22:04.203 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T22:22:04.204 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T22:22:04.204 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T22:22:04.204 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T22:22:04.204 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-28T22:22:04.248 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1899.g323d2 150/198 2024-03-28T22:22:04.270 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1899.g323d2 150/198 2024-03-28T22:22:04.270 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T22:22:04.271 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T22:22:04.271 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-28T22:22:04.287 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : cephfs-shell-2:19.0.0-1899.g323d23dc.el9.x86_64 157/198 2024-03-28T22:22:04.315 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : cephfs-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 151/198 2024-03-28T22:22:04.338 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 151/198 2024-03-28T22:22:04.338 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T22:22:04.338 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T22:22:04.338 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T22:22:04.338 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T22:22:04.338 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-28T22:22:04.405 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T22:22:04.453 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 159/198 2024-03-28T22:22:04.472 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-03-28T22:22:04.501 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T22:22:04.570 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T22:22:04.583 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T22:22:04.613 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T22:22:04.631 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T22:22:04.646 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephfs-devel-2:19.0.0-1899.g323d23dc.el9.x86_ 163/198 2024-03-28T22:22:04.658 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 152/198 2024-03-28T22:22:04.670 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T22:22:04.680 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 152/198 2024-03-28T22:22:04.680 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T22:22:04.680 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T22:22:04.680 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T22:22:04.680 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T22:22:04.680 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-28T22:22:04.686 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T22:22:04.714 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T22:22:04.716 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T22:22:04.779 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T22:22:04.806 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T22:22:04.850 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T22:22:04.890 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T22:22:04.910 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T22:22:04.964 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T22:22:05.002 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T22:22:05.104 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T22:22:05.421 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T22:22:05.454 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T22:22:05.474 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T22:22:05.529 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T22:22:05.592 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : librados2-2:19.0.0-1899.g323d23dc.el9.x86_64 10/198 2024-03-28T22:22:05.760 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librados2-2:19.0.0-1899.g323d23dc.el9.x86_64 10/198 2024-03-28T22:22:05.793 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libcephfs2-2:19.0.0-1899.g323d23dc.el9.x86_64 11/198 2024-03-28T22:22:05.912 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libcephfs2-2:19.0.0-1899.g323d23dc.el9.x86_64 11/198 2024-03-28T22:22:05.950 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rados-2:19.0.0-1899.g323d23dc.el9.x86_64 12/198 2024-03-28T22:22:06.017 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T22:22:06.306 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : cephfs-top-2:19.0.0-1899.g323d23dc.el9.noarch 170/198 2024-03-28T22:22:06.310 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T22:22:06.382 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T22:22:06.422 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T22:22:06.498 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T22:22:06.503 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T22:22:06.523 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libcephsqlite-2:19.0.0-1899.g323d23dc.el9.x86_64 16/198 2024-03-28T22:22:06.544 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T22:22:06.577 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T22:22:06.603 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T22:22:06.629 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T22:22:06.629 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T22:22:06.699 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1899.g323d23dc.el9.x86_64 16/198 2024-03-28T22:22:06.726 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T22:22:06.732 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libradosstriper1-2:19.0.0-1899.g323d23dc.el9.x86 17/198 2024-03-28T22:22:06.755 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T22:22:06.755 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T22:22:06.832 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T22:22:06.842 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1899.g323d23dc.el9.x86 17/198 2024-03-28T22:22:06.858 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T22:22:06.900 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T22:22:06.904 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T22:22:06.925 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T22:22:06.936 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T22:22:06.950 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T22:22:06.982 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T22:22:07.011 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T22:22:07.030 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T22:22:07.071 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T22:22:07.105 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T22:22:07.130 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T22:22:07.155 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T22:22:07.180 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T22:22:07.186 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T22:22:07.187 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T22:22:07.328 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T22:22:07.367 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T22:22:07.367 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T22:22:07.380 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T22:22:07.433 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T22:22:07.436 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T22:22:07.458 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T22:22:07.490 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T22:22:07.526 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T22:22:07.551 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T22:22:07.614 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T22:22:07.727 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T22:22:07.921 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T22:22:08.006 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T22:22:08.195 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T22:22:08.255 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T22:22:08.599 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T22:22:08.789 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T22:22:08.909 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T22:22:08.943 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T22:22:08.971 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T22:22:09.036 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T22:22:09.129 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-test-2:19.0.0-1899.g323d23dc.el9.x86_64 153/198 2024-03-28T22:22:09.151 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T22:22:09.169 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 154/198 2024-03-28T22:22:09.200 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-nbd-2:19.0.0-1899.g323d23dc.el9.x86_64 155/198 2024-03-28T22:22:09.227 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T22:22:09.228 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T22:22:09.278 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T22:22:09.383 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T22:22:09.405 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : cephfs-shell-2:19.0.0-1899.g323d23dc.el9.x86_64 157/198 2024-03-28T22:22:09.432 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T22:22:09.498 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T22:22:09.518 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T22:22:09.546 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T22:22:09.559 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 159/198 2024-03-28T22:22:09.578 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T22:22:09.598 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T22:22:09.659 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T22:22:09.697 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T22:22:09.702 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T22:22:09.727 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephfs-devel-2:19.0.0-1899.g323d23dc.el9.x86_ 163/198 2024-03-28T22:22:09.739 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-ceph-common-2:19.0.0-1899.g323d23dc.el9. 39/198 2024-03-28T22:22:09.767 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T22:22:09.797 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T22:22:09.800 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-ceph-argparse-2:19.0.0-1899.g323d23dc.el 40/198 2024-03-28T22:22:09.839 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cephfs-2:19.0.0-1899.g323d23dc.el9.x86_6 41/198 2024-03-28T22:22:09.843 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T22:22:09.912 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T22:22:09.954 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T22:22:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T22:22:10.028 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T22:22:10.141 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T22:22:10.327 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T22:22:10.386 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T22:22:10.572 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T22:22:10.610 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T22:22:10.630 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T22:22:11.107 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T22:22:11.202 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T22:22:11.316 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T22:22:11.383 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T22:22:11.413 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T22:22:11.448 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : cephfs-top-2:19.0.0-1899.g323d23dc.el9.noarch 170/198 2024-03-28T22:22:11.458 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T22:22:11.531 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T22:22:11.536 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T22:22:11.571 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T22:22:11.593 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librados-devel-2:19.0.0-1899.g323d23dc.el9.x86_6 53/198 2024-03-28T22:22:11.654 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T22:22:11.663 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T22:22:11.684 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T22:22:11.709 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T22:22:11.735 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T22:22:11.761 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T22:22:11.761 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T22:22:11.832 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T22:22:11.862 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T22:22:11.862 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T22:22:11.947 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T22:22:11.973 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T22:22:12.006 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T22:22:12.031 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T22:22:12.048 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T22:22:12.081 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T22:22:12.121 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T22:22:12.128 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T22:22:12.153 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T22:22:12.178 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T22:22:12.204 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T22:22:12.229 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T22:22:12.254 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T22:22:12.261 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T22:22:12.261 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T22:22:12.326 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T22:22:12.396 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T22:22:12.440 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T22:22:12.440 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T22:22:12.454 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T22:22:12.515 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T22:22:12.540 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T22:22:12.572 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T22:22:12.599 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T22:22:12.624 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T22:22:12.666 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T22:22:12.688 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T22:22:12.717 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T22:22:12.798 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T22:22:12.902 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T22:22:12.941 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T22:22:12.981 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T22:22:12.985 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T22:22:12.986 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-2:19.0.0-1899.g323d23dc.el9.x86_64 1/198 2024-03-28T22:22:12.986 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-base-2:19.0.0-1899.g323d23dc.el9.x86_64 2/198 2024-03-28T22:22:12.986 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 3/198 2024-03-28T22:22:12.986 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 4/198 2024-03-28T22:22:12.986 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1899.g323d2 5/198 2024-03-28T22:22:12.986 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mds-2:19.0.0-1899.g323d23dc.el9.x86_64 6/198 2024-03-28T22:22:12.986 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-2:19.0.0-1899.g323d23dc.el9.x86_64 7/198 2024-03-28T22:22:12.986 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mon-2:19.0.0-1899.g323d23dc.el9.x86_64 8/198 2024-03-28T22:22:12.986 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-osd-2:19.0.0-1899.g323d23dc.el9.x86_64 9/198 2024-03-28T22:22:12.986 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-radosgw-2:19.0.0-1899.g323d23dc.el9.x86_64 10/198 2024-03-28T22:22:12.986 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-selinux-2:19.0.0-1899.g323d23dc.el9.x86_64 11/198 2024-03-28T22:22:12.986 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-test-2:19.0.0-1899.g323d23dc.el9.x86_64 12/198 2024-03-28T22:22:12.987 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : cephfs-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 13/198 2024-03-28T22:22:12.987 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : cephfs-shell-2:19.0.0-1899.g323d23dc.el9.x86_64 14/198 2024-03-28T22:22:12.987 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephfs-devel-2:19.0.0-1899.g323d23dc.el9.x86_ 15/198 2024-03-28T22:22:12.987 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephfs2-2:19.0.0-1899.g323d23dc.el9.x86_64 16/198 2024-03-28T22:22:12.987 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephsqlite-2:19.0.0-1899.g323d23dc.el9.x86_64 17/198 2024-03-28T22:22:12.987 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados-devel-2:19.0.0-1899.g323d23dc.el9.x86_6 18/198 2024-03-28T22:22:12.987 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libradosstriper1-2:19.0.0-1899.g323d23dc.el9.x86 19/198 2024-03-28T22:22:12.987 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librgw2-2:19.0.0-1899.g323d23dc.el9.x86_64 20/198 2024-03-28T22:22:12.987 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1899.g323d23dc.el 21/198 2024-03-28T22:22:12.987 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ceph-common-2:19.0.0-1899.g323d23dc.el9. 22/198 2024-03-28T22:22:12.987 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cephfs-2:19.0.0-1899.g323d23dc.el9.x86_6 23/198 2024-03-28T22:22:12.987 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rados-2:19.0.0-1899.g323d23dc.el9.x86_64 24/198 2024-03-28T22:22:12.987 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rbd-2:19.0.0-1899.g323d23dc.el9.x86_64 25/198 2024-03-28T22:22:12.988 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rgw-2:19.0.0-1899.g323d23dc.el9.x86_64 26/198 2024-03-28T22:22:12.988 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 27/198 2024-03-28T22:22:13.004 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 28/198 2024-03-28T22:22:13.004 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-nbd-2:19.0.0-1899.g323d23dc.el9.x86_64 29/198 2024-03-28T22:22:13.004 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1899.g323d23dc. 30/198 2024-03-28T22:22:13.004 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1899.g323d23dc.el9.noa 31/198 2024-03-28T22:22:13.004 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1899.g323d23dc.el9.n 32/198 2024-03-28T22:22:13.004 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1899.g323 33/198 2024-03-28T22:22:13.005 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1899.g323d23dc.el 34/198 2024-03-28T22:22:13.005 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1899.g323d23dc.el9.noarch 35/198 2024-03-28T22:22:13.005 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1899.g323d23dc.e 36/198 2024-03-28T22:22:13.005 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-volume-2:19.0.0-1899.g323d23dc.el9.noarch 37/198 2024-03-28T22:22:13.005 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 38/198 2024-03-28T22:22:13.005 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : cephfs-top-2:19.0.0-1899.g323d23dc.el9.noarch 39/198 2024-03-28T22:22:13.005 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T22:22:13.005 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T22:22:13.005 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T22:22:13.005 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T22:22:13.005 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T22:22:13.005 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T22:22:13.005 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T22:22:13.005 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T22:22:13.005 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T22:22:13.006 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T22:22:13.006 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T22:22:13.006 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T22:22:13.006 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T22:22:13.006 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T22:22:13.006 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T22:22:13.006 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T22:22:13.006 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T22:22:13.006 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T22:22:13.006 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-03-28T22:22:13.006 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-03-28T22:22:13.006 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-03-28T22:22:13.006 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-03-28T22:22:13.007 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-28T22:22:13.007 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-28T22:22:13.007 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-28T22:22:13.007 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-28T22:22:13.007 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-28T22:22:13.007 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-28T22:22:13.007 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-28T22:22:13.007 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-28T22:22:13.007 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-28T22:22:13.007 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-28T22:22:13.007 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-28T22:22:13.007 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-28T22:22:13.007 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-28T22:22:13.007 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-28T22:22:13.007 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-28T22:22:13.008 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-28T22:22:13.008 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-28T22:22:13.008 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-28T22:22:13.008 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-28T22:22:13.008 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-28T22:22:13.008 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-03-28T22:22:13.008 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-03-28T22:22:13.008 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-03-28T22:22:13.008 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-03-28T22:22:13.008 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-03-28T22:22:13.008 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-03-28T22:22:13.008 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-03-28T22:22:13.008 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T22:22:13.008 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-03-28T22:22:13.009 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-03-28T22:22:13.009 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-03-28T22:22:13.009 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-03-28T22:22:13.009 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-03-28T22:22:13.009 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-03-28T22:22:13.009 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-03-28T22:22:13.009 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-03-28T22:22:13.009 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-03-28T22:22:13.009 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-03-28T22:22:13.009 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-03-28T22:22:13.009 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-03-28T22:22:13.009 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-03-28T22:22:13.009 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-03-28T22:22:13.010 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T22:22:13.010 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T22:22:13.010 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T22:22:13.010 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T22:22:13.010 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T22:22:13.010 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T22:22:13.010 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T22:22:13.010 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T22:22:13.010 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T22:22:13.010 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T22:22:13.010 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T22:22:13.010 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T22:22:13.010 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T22:22:13.010 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T22:22:13.011 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T22:22:13.011 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T22:22:13.011 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T22:22:13.011 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T22:22:13.011 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T22:22:13.011 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T22:22:13.011 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T22:22:13.011 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T22:22:13.011 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T22:22:13.011 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T22:22:13.011 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T22:22:13.011 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T22:22:13.011 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T22:22:13.011 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T22:22:13.012 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T22:22:13.012 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T22:22:13.012 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T22:22:13.012 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T22:22:13.012 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T22:22:13.012 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T22:22:13.012 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T22:22:13.012 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T22:22:13.012 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T22:22:13.012 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T22:22:13.012 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T22:22:13.012 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T22:22:13.012 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T22:22:13.012 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T22:22:13.012 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T22:22:13.013 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T22:22:13.013 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T22:22:13.013 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T22:22:13.013 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T22:22:13.013 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados2-2:19.0.0-1899.g323d23dc.el9.x86_64 151/198 2024-03-28T22:22:13.013 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T22:22:13.013 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librbd1-2:19.0.0-1899.g323d23dc.el9.x86_64 153/198 2024-03-28T22:22:13.013 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T22:22:13.013 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 155/198 2024-03-28T22:22:13.013 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-03-28T22:22:13.013 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T22:22:13.013 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T22:22:13.013 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T22:22:13.014 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T22:22:13.014 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T22:22:13.014 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T22:22:13.014 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-03-28T22:22:13.014 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-03-28T22:22:13.014 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T22:22:13.014 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T22:22:13.014 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T22:22:13.014 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T22:22:13.014 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T22:22:13.014 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T22:22:13.014 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T22:22:13.014 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T22:22:13.014 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T22:22:13.015 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T22:22:13.015 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T22:22:13.015 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T22:22:13.015 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T22:22:13.015 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T22:22:13.015 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T22:22:13.015 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T22:22:13.015 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T22:22:13.015 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T22:22:13.015 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T22:22:13.015 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T22:22:13.015 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T22:22:13.015 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T22:22:13.015 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T22:22:13.016 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T22:22:13.016 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T22:22:13.016 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T22:22:13.016 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T22:22:13.016 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T22:22:13.016 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T22:22:13.016 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T22:22:13.016 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T22:22:13.016 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T22:22:13.016 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T22:22:13.049 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T22:22:13.089 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T22:22:13.154 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T22:22:13.220 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T22:22:13.437 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T22:22:13.496 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T22:22:13.535 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T22:22:13.595 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T22:22:13.739 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T22:22:13.773 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T22:22:13.819 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T22:22:13.856 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T22:22:13.880 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T22:22:14.041 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T22:22:14.219 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T22:22:14.357 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T22:22:14.407 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T22:22:14.470 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T22:22:14.545 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T22:22:14.798 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T22:22:14.861 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T22:22:15.045 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T22:22:15.092 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T22:22:15.141 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T22:22:15.179 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T22:22:15.228 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T22:22:15.267 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T22:22:15.324 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T22:22:15.376 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T22:22:15.423 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T22:22:15.555 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T22:22:15.594 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T22:22:15.629 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T22:22:15.682 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T22:22:15.723 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T22:22:15.872 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T22:22:15.992 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T22:22:16.054 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T22:22:16.090 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T22:22:16.148 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T22:22:16.189 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T22:22:16.189 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-28T22:22:16.189 INFO:teuthology.orchestra.run.smithi172.stdout:Upgraded: 2024-03-28T22:22:16.189 INFO:teuthology.orchestra.run.smithi172.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T22:22:16.189 INFO:teuthology.orchestra.run.smithi172.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T22:22:16.189 INFO:teuthology.orchestra.run.smithi172.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T22:22:16.189 INFO:teuthology.orchestra.run.smithi172.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T22:22:16.189 INFO:teuthology.orchestra.run.smithi172.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T22:22:16.190 INFO:teuthology.orchestra.run.smithi172.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T22:22:16.190 INFO:teuthology.orchestra.run.smithi172.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T22:22:16.190 INFO:teuthology.orchestra.run.smithi172.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T22:22:16.190 INFO:teuthology.orchestra.run.smithi172.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T22:22:16.190 INFO:teuthology.orchestra.run.smithi172.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T22:22:16.190 INFO:teuthology.orchestra.run.smithi172.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T22:22:16.190 INFO:teuthology.orchestra.run.smithi172.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T22:22:16.190 INFO:teuthology.orchestra.run.smithi172.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T22:22:16.190 INFO:teuthology.orchestra.run.smithi172.stdout: librados2-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:16.190 INFO:teuthology.orchestra.run.smithi172.stdout: librbd1-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:16.190 INFO:teuthology.orchestra.run.smithi172.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T22:22:16.190 INFO:teuthology.orchestra.run.smithi172.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T22:22:16.191 INFO:teuthology.orchestra.run.smithi172.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T22:22:16.191 INFO:teuthology.orchestra.run.smithi172.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T22:22:16.191 INFO:teuthology.orchestra.run.smithi172.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T22:22:16.191 INFO:teuthology.orchestra.run.smithi172.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T22:22:16.191 INFO:teuthology.orchestra.run.smithi172.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T22:22:16.191 INFO:teuthology.orchestra.run.smithi172.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T22:22:16.191 INFO:teuthology.orchestra.run.smithi172.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T22:22:16.191 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-28T22:22:16.191 INFO:teuthology.orchestra.run.smithi172.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T22:22:16.191 INFO:teuthology.orchestra.run.smithi172.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T22:22:16.191 INFO:teuthology.orchestra.run.smithi172.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T22:22:16.191 INFO:teuthology.orchestra.run.smithi172.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T22:22:16.191 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:16.191 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:16.192 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:16.192 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:16.192 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-grafana-dashboards-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T22:22:16.192 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-immutable-object-cache-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:16.192 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mds-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:16.192 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:16.192 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T22:22:16.192 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T22:22:16.192 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T22:22:16.192 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T22:22:16.192 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-rook-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T22:22:16.192 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mon-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:16.192 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-osd-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:16.192 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-prometheus-alerts-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T22:22:16.192 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-radosgw-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:16.193 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-selinux-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:16.193 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-test-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:16.193 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-volume-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T22:22:16.193 INFO:teuthology.orchestra.run.smithi172.stdout: cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T22:22:16.193 INFO:teuthology.orchestra.run.smithi172.stdout: cephfs-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:16.193 INFO:teuthology.orchestra.run.smithi172.stdout: cephfs-shell-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:16.193 INFO:teuthology.orchestra.run.smithi172.stdout: cephfs-top-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T22:22:16.193 INFO:teuthology.orchestra.run.smithi172.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T22:22:16.193 INFO:teuthology.orchestra.run.smithi172.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T22:22:16.193 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T22:22:16.193 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T22:22:16.193 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T22:22:16.193 INFO:teuthology.orchestra.run.smithi172.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T22:22:16.194 INFO:teuthology.orchestra.run.smithi172.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T22:22:16.194 INFO:teuthology.orchestra.run.smithi172.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T22:22:16.194 INFO:teuthology.orchestra.run.smithi172.stdout: jq-1.6-16.el9.x86_64 2024-03-28T22:22:16.194 INFO:teuthology.orchestra.run.smithi172.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T22:22:16.194 INFO:teuthology.orchestra.run.smithi172.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T22:22:16.194 INFO:teuthology.orchestra.run.smithi172.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T22:22:16.194 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-devel-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:16.194 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs2-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:16.194 INFO:teuthology.orchestra.run.smithi172.stdout: libcephsqlite-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:16.194 INFO:teuthology.orchestra.run.smithi172.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T22:22:16.194 INFO:teuthology.orchestra.run.smithi172.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T22:22:16.195 INFO:teuthology.orchestra.run.smithi172.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T22:22:16.195 INFO:teuthology.orchestra.run.smithi172.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T22:22:16.195 INFO:teuthology.orchestra.run.smithi172.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T22:22:16.195 INFO:teuthology.orchestra.run.smithi172.stdout: librados-devel-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:16.195 INFO:teuthology.orchestra.run.smithi172.stdout: libradosstriper1-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:16.195 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T22:22:16.195 INFO:teuthology.orchestra.run.smithi172.stdout: librgw2-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:16.195 INFO:teuthology.orchestra.run.smithi172.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T22:22:16.195 INFO:teuthology.orchestra.run.smithi172.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T22:22:16.195 INFO:teuthology.orchestra.run.smithi172.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T22:22:16.195 INFO:teuthology.orchestra.run.smithi172.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T22:22:16.195 INFO:teuthology.orchestra.run.smithi172.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T22:22:16.195 INFO:teuthology.orchestra.run.smithi172.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T22:22:16.195 INFO:teuthology.orchestra.run.smithi172.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T22:22:16.196 INFO:teuthology.orchestra.run.smithi172.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T22:22:16.196 INFO:teuthology.orchestra.run.smithi172.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T22:22:16.196 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T22:22:16.196 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T22:22:16.196 INFO:teuthology.orchestra.run.smithi172.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T22:22:16.196 INFO:teuthology.orchestra.run.smithi172.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T22:22:16.196 INFO:teuthology.orchestra.run.smithi172.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T22:22:16.196 INFO:teuthology.orchestra.run.smithi172.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T22:22:16.196 INFO:teuthology.orchestra.run.smithi172.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T22:22:16.196 INFO:teuthology.orchestra.run.smithi172.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T22:22:16.196 INFO:teuthology.orchestra.run.smithi172.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T22:22:16.196 INFO:teuthology.orchestra.run.smithi172.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T22:22:16.196 INFO:teuthology.orchestra.run.smithi172.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T22:22:16.197 INFO:teuthology.orchestra.run.smithi172.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T22:22:16.197 INFO:teuthology.orchestra.run.smithi172.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T22:22:16.197 INFO:teuthology.orchestra.run.smithi172.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T22:22:16.197 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T22:22:16.197 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:16.197 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:16.197 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cephfs-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:16.197 INFO:teuthology.orchestra.run.smithi172.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T22:22:16.197 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T22:22:16.197 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T22:22:16.197 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T22:22:16.197 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T22:22:16.197 INFO:teuthology.orchestra.run.smithi172.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T22:22:16.198 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T22:22:16.198 INFO:teuthology.orchestra.run.smithi172.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T22:22:16.198 INFO:teuthology.orchestra.run.smithi172.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T22:22:16.198 INFO:teuthology.orchestra.run.smithi172.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T22:22:16.198 INFO:teuthology.orchestra.run.smithi172.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T22:22:16.198 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T22:22:16.198 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T22:22:16.198 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T22:22:16.198 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T22:22:16.198 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T22:22:16.198 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T22:22:16.198 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T22:22:16.199 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T22:22:16.199 INFO:teuthology.orchestra.run.smithi172.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T22:22:16.199 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T22:22:16.199 INFO:teuthology.orchestra.run.smithi172.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T22:22:16.199 INFO:teuthology.orchestra.run.smithi172.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T22:22:16.199 INFO:teuthology.orchestra.run.smithi172.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T22:22:16.199 INFO:teuthology.orchestra.run.smithi172.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T22:22:16.199 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T22:22:16.199 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T22:22:16.199 INFO:teuthology.orchestra.run.smithi172.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T22:22:16.199 INFO:teuthology.orchestra.run.smithi172.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T22:22:16.199 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T22:22:16.199 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T22:22:16.199 INFO:teuthology.orchestra.run.smithi172.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T22:22:16.200 INFO:teuthology.orchestra.run.smithi172.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T22:22:16.200 INFO:teuthology.orchestra.run.smithi172.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T22:22:16.200 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T22:22:16.200 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T22:22:16.200 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T22:22:16.200 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T22:22:16.200 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T22:22:16.200 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T22:22:16.200 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rados-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:16.201 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rbd-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:16.201 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T22:22:16.201 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T22:22:16.201 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T22:22:16.201 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rgw-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:16.201 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T22:22:16.201 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T22:22:16.201 INFO:teuthology.orchestra.run.smithi172.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T22:22:16.201 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T22:22:16.201 INFO:teuthology.orchestra.run.smithi172.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T22:22:16.201 INFO:teuthology.orchestra.run.smithi172.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T22:22:16.201 INFO:teuthology.orchestra.run.smithi172.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T22:22:16.201 INFO:teuthology.orchestra.run.smithi172.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T22:22:16.201 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T22:22:16.202 INFO:teuthology.orchestra.run.smithi172.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T22:22:16.202 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T22:22:16.202 INFO:teuthology.orchestra.run.smithi172.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T22:22:16.202 INFO:teuthology.orchestra.run.smithi172.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T22:22:16.202 INFO:teuthology.orchestra.run.smithi172.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T22:22:16.202 INFO:teuthology.orchestra.run.smithi172.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T22:22:16.202 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:16.202 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:16.202 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-nbd-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:16.202 INFO:teuthology.orchestra.run.smithi172.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T22:22:16.202 INFO:teuthology.orchestra.run.smithi172.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T22:22:16.202 INFO:teuthology.orchestra.run.smithi172.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T22:22:16.202 INFO:teuthology.orchestra.run.smithi172.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T22:22:16.203 INFO:teuthology.orchestra.run.smithi172.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T22:22:16.203 INFO:teuthology.orchestra.run.smithi172.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T22:22:16.203 INFO:teuthology.orchestra.run.smithi172.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T22:22:16.203 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-28T22:22:16.203 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-28T22:22:16.292 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T22:22:16.356 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T22:22:16.671 DEBUG:teuthology.parallel:result is None 2024-03-28T22:22:17.359 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T22:22:17.412 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T22:22:17.446 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T22:22:17.512 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T22:22:17.628 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T22:22:17.664 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T22:22:17.908 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T22:22:17.935 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T22:22:17.961 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T22:22:17.961 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-2:19.0.0-1899.g323d23dc.el9.x86_64 1/198 2024-03-28T22:22:17.961 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-base-2:19.0.0-1899.g323d23dc.el9.x86_64 2/198 2024-03-28T22:22:17.961 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 3/198 2024-03-28T22:22:17.961 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 4/198 2024-03-28T22:22:17.961 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1899.g323d2 5/198 2024-03-28T22:22:17.962 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mds-2:19.0.0-1899.g323d23dc.el9.x86_64 6/198 2024-03-28T22:22:17.962 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-2:19.0.0-1899.g323d23dc.el9.x86_64 7/198 2024-03-28T22:22:17.962 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mon-2:19.0.0-1899.g323d23dc.el9.x86_64 8/198 2024-03-28T22:22:17.962 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-osd-2:19.0.0-1899.g323d23dc.el9.x86_64 9/198 2024-03-28T22:22:17.962 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-radosgw-2:19.0.0-1899.g323d23dc.el9.x86_64 10/198 2024-03-28T22:22:17.962 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-selinux-2:19.0.0-1899.g323d23dc.el9.x86_64 11/198 2024-03-28T22:22:17.962 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-test-2:19.0.0-1899.g323d23dc.el9.x86_64 12/198 2024-03-28T22:22:17.962 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : cephfs-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 13/198 2024-03-28T22:22:17.962 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : cephfs-shell-2:19.0.0-1899.g323d23dc.el9.x86_64 14/198 2024-03-28T22:22:17.962 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephfs-devel-2:19.0.0-1899.g323d23dc.el9.x86_ 15/198 2024-03-28T22:22:17.962 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephfs2-2:19.0.0-1899.g323d23dc.el9.x86_64 16/198 2024-03-28T22:22:17.962 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephsqlite-2:19.0.0-1899.g323d23dc.el9.x86_64 17/198 2024-03-28T22:22:17.962 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados-devel-2:19.0.0-1899.g323d23dc.el9.x86_6 18/198 2024-03-28T22:22:17.962 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libradosstriper1-2:19.0.0-1899.g323d23dc.el9.x86 19/198 2024-03-28T22:22:17.963 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librgw2-2:19.0.0-1899.g323d23dc.el9.x86_64 20/198 2024-03-28T22:22:17.963 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1899.g323d23dc.el 21/198 2024-03-28T22:22:17.963 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ceph-common-2:19.0.0-1899.g323d23dc.el9. 22/198 2024-03-28T22:22:17.963 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cephfs-2:19.0.0-1899.g323d23dc.el9.x86_6 23/198 2024-03-28T22:22:17.963 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rados-2:19.0.0-1899.g323d23dc.el9.x86_64 24/198 2024-03-28T22:22:17.963 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rbd-2:19.0.0-1899.g323d23dc.el9.x86_64 25/198 2024-03-28T22:22:17.963 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rgw-2:19.0.0-1899.g323d23dc.el9.x86_64 26/198 2024-03-28T22:22:17.963 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 27/198 2024-03-28T22:22:17.963 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 28/198 2024-03-28T22:22:17.963 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-nbd-2:19.0.0-1899.g323d23dc.el9.x86_64 29/198 2024-03-28T22:22:17.963 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1899.g323d23dc. 30/198 2024-03-28T22:22:17.963 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1899.g323d23dc.el9.noa 31/198 2024-03-28T22:22:17.963 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1899.g323d23dc.el9.n 32/198 2024-03-28T22:22:17.963 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1899.g323 33/198 2024-03-28T22:22:17.964 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1899.g323d23dc.el 34/198 2024-03-28T22:22:17.964 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1899.g323d23dc.el9.noarch 35/198 2024-03-28T22:22:17.964 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1899.g323d23dc.e 36/198 2024-03-28T22:22:17.964 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-volume-2:19.0.0-1899.g323d23dc.el9.noarch 37/198 2024-03-28T22:22:17.964 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 38/198 2024-03-28T22:22:17.964 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : cephfs-top-2:19.0.0-1899.g323d23dc.el9.noarch 39/198 2024-03-28T22:22:17.964 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T22:22:17.964 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T22:22:17.964 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T22:22:17.964 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T22:22:17.964 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T22:22:17.964 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T22:22:17.964 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T22:22:17.964 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T22:22:17.964 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T22:22:17.964 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T22:22:17.965 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T22:22:17.965 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T22:22:17.965 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T22:22:17.965 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T22:22:17.965 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T22:22:17.965 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T22:22:17.965 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T22:22:17.965 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T22:22:17.965 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-03-28T22:22:17.965 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-03-28T22:22:17.965 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-03-28T22:22:17.965 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-03-28T22:22:17.965 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-28T22:22:17.965 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-28T22:22:17.965 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-28T22:22:17.965 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-28T22:22:17.966 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-28T22:22:17.966 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-28T22:22:17.966 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-28T22:22:17.966 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-28T22:22:17.966 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-28T22:22:17.966 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-28T22:22:17.966 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-28T22:22:17.966 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-28T22:22:17.966 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-28T22:22:17.966 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-28T22:22:17.966 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-28T22:22:17.966 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-28T22:22:17.966 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-28T22:22:17.966 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-28T22:22:17.966 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-28T22:22:17.967 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-28T22:22:17.967 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-03-28T22:22:17.967 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-03-28T22:22:17.967 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-03-28T22:22:17.967 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-03-28T22:22:17.967 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-03-28T22:22:17.967 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-03-28T22:22:17.967 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-03-28T22:22:17.967 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T22:22:17.967 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-03-28T22:22:17.967 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-03-28T22:22:17.967 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-03-28T22:22:17.967 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-03-28T22:22:17.967 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-03-28T22:22:17.967 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-03-28T22:22:17.968 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-03-28T22:22:17.968 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-03-28T22:22:17.968 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-03-28T22:22:17.968 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-03-28T22:22:17.968 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-03-28T22:22:17.968 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-03-28T22:22:17.968 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-03-28T22:22:17.968 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-03-28T22:22:17.968 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T22:22:17.968 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T22:22:17.968 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T22:22:17.968 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T22:22:17.968 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T22:22:17.968 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T22:22:17.968 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T22:22:17.969 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T22:22:17.969 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T22:22:17.969 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T22:22:17.969 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T22:22:17.969 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T22:22:17.969 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T22:22:17.969 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T22:22:17.969 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T22:22:17.969 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T22:22:17.969 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T22:22:17.969 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T22:22:17.969 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T22:22:17.969 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T22:22:17.969 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T22:22:17.970 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T22:22:17.970 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T22:22:17.970 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T22:22:17.970 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T22:22:17.970 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T22:22:17.970 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T22:22:17.970 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T22:22:17.970 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T22:22:17.970 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T22:22:17.970 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T22:22:17.970 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T22:22:17.970 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T22:22:17.970 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T22:22:17.970 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T22:22:17.971 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T22:22:17.971 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T22:22:17.971 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T22:22:17.971 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T22:22:17.971 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T22:22:17.971 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T22:22:17.971 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T22:22:17.971 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T22:22:17.971 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T22:22:17.971 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T22:22:17.971 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T22:22:17.971 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T22:22:17.971 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados2-2:19.0.0-1899.g323d23dc.el9.x86_64 151/198 2024-03-28T22:22:17.971 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T22:22:17.972 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librbd1-2:19.0.0-1899.g323d23dc.el9.x86_64 153/198 2024-03-28T22:22:17.972 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T22:22:17.972 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 155/198 2024-03-28T22:22:17.972 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-03-28T22:22:17.972 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T22:22:17.972 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T22:22:17.972 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T22:22:17.972 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T22:22:17.972 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T22:22:17.972 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T22:22:17.972 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-03-28T22:22:17.972 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-03-28T22:22:17.972 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T22:22:17.972 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T22:22:17.973 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T22:22:17.973 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T22:22:17.973 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T22:22:17.973 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T22:22:17.973 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T22:22:17.973 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T22:22:17.973 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T22:22:17.973 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T22:22:17.973 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T22:22:17.973 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T22:22:17.973 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T22:22:17.973 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T22:22:17.973 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T22:22:17.973 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T22:22:17.973 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T22:22:17.974 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T22:22:17.974 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T22:22:17.974 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T22:22:17.974 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T22:22:17.974 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T22:22:17.974 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T22:22:17.974 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T22:22:17.974 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T22:22:17.974 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T22:22:17.974 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T22:22:17.974 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T22:22:17.974 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T22:22:17.974 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T22:22:17.974 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T22:22:17.975 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T22:22:17.975 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T22:22:18.023 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T22:22:18.877 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T22:22:19.019 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T22:22:20.613 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T22:22:20.706 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T22:22:20.743 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T22:22:20.765 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T22:22:20.796 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T22:22:20.968 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T22:22:20.968 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-28T22:22:20.968 INFO:teuthology.orchestra.run.smithi082.stdout:Upgraded: 2024-03-28T22:22:20.968 INFO:teuthology.orchestra.run.smithi082.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T22:22:20.968 INFO:teuthology.orchestra.run.smithi082.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T22:22:20.969 INFO:teuthology.orchestra.run.smithi082.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T22:22:20.969 INFO:teuthology.orchestra.run.smithi082.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T22:22:20.969 INFO:teuthology.orchestra.run.smithi082.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T22:22:20.969 INFO:teuthology.orchestra.run.smithi082.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T22:22:20.969 INFO:teuthology.orchestra.run.smithi082.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T22:22:20.969 INFO:teuthology.orchestra.run.smithi082.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T22:22:20.969 INFO:teuthology.orchestra.run.smithi082.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T22:22:20.969 INFO:teuthology.orchestra.run.smithi082.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T22:22:20.969 INFO:teuthology.orchestra.run.smithi082.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T22:22:20.969 INFO:teuthology.orchestra.run.smithi082.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T22:22:20.969 INFO:teuthology.orchestra.run.smithi082.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T22:22:20.969 INFO:teuthology.orchestra.run.smithi082.stdout: librados2-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:20.969 INFO:teuthology.orchestra.run.smithi082.stdout: librbd1-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:20.970 INFO:teuthology.orchestra.run.smithi082.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T22:22:20.970 INFO:teuthology.orchestra.run.smithi082.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T22:22:20.970 INFO:teuthology.orchestra.run.smithi082.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T22:22:20.970 INFO:teuthology.orchestra.run.smithi082.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T22:22:20.970 INFO:teuthology.orchestra.run.smithi082.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T22:22:20.970 INFO:teuthology.orchestra.run.smithi082.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T22:22:20.970 INFO:teuthology.orchestra.run.smithi082.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T22:22:20.970 INFO:teuthology.orchestra.run.smithi082.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T22:22:20.970 INFO:teuthology.orchestra.run.smithi082.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T22:22:20.970 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-28T22:22:20.970 INFO:teuthology.orchestra.run.smithi082.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T22:22:20.970 INFO:teuthology.orchestra.run.smithi082.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T22:22:20.970 INFO:teuthology.orchestra.run.smithi082.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T22:22:20.971 INFO:teuthology.orchestra.run.smithi082.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T22:22:20.971 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:20.971 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-base-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:20.971 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:20.971 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:20.971 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-grafana-dashboards-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T22:22:20.971 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-immutable-object-cache-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:20.971 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mds-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:20.971 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:20.971 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T22:22:20.971 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-dashboard-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T22:22:20.971 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T22:22:20.971 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-modules-core-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T22:22:20.972 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-rook-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T22:22:20.972 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mon-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:20.972 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-osd-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:20.972 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-prometheus-alerts-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T22:22:20.972 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-radosgw-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:20.972 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-selinux-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:20.972 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-test-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:20.972 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-volume-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T22:22:20.972 INFO:teuthology.orchestra.run.smithi082.stdout: cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T22:22:20.972 INFO:teuthology.orchestra.run.smithi082.stdout: cephfs-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:20.972 INFO:teuthology.orchestra.run.smithi082.stdout: cephfs-shell-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:20.972 INFO:teuthology.orchestra.run.smithi082.stdout: cephfs-top-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T22:22:20.972 INFO:teuthology.orchestra.run.smithi082.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T22:22:20.972 INFO:teuthology.orchestra.run.smithi082.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T22:22:20.972 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T22:22:20.973 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T22:22:20.973 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T22:22:20.973 INFO:teuthology.orchestra.run.smithi082.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T22:22:20.973 INFO:teuthology.orchestra.run.smithi082.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T22:22:20.973 INFO:teuthology.orchestra.run.smithi082.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T22:22:20.973 INFO:teuthology.orchestra.run.smithi082.stdout: jq-1.6-16.el9.x86_64 2024-03-28T22:22:20.973 INFO:teuthology.orchestra.run.smithi082.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T22:22:20.973 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T22:22:20.973 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T22:22:20.973 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs-devel-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:20.973 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs2-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:20.973 INFO:teuthology.orchestra.run.smithi082.stdout: libcephsqlite-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:20.973 INFO:teuthology.orchestra.run.smithi082.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T22:22:20.973 INFO:teuthology.orchestra.run.smithi082.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T22:22:20.974 INFO:teuthology.orchestra.run.smithi082.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T22:22:20.974 INFO:teuthology.orchestra.run.smithi082.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T22:22:20.974 INFO:teuthology.orchestra.run.smithi082.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T22:22:20.974 INFO:teuthology.orchestra.run.smithi082.stdout: librados-devel-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:20.974 INFO:teuthology.orchestra.run.smithi082.stdout: libradosstriper1-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:20.974 INFO:teuthology.orchestra.run.smithi082.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T22:22:20.974 INFO:teuthology.orchestra.run.smithi082.stdout: librgw2-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:20.974 INFO:teuthology.orchestra.run.smithi082.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T22:22:20.974 INFO:teuthology.orchestra.run.smithi082.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T22:22:20.974 INFO:teuthology.orchestra.run.smithi082.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T22:22:20.974 INFO:teuthology.orchestra.run.smithi082.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T22:22:20.974 INFO:teuthology.orchestra.run.smithi082.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T22:22:20.974 INFO:teuthology.orchestra.run.smithi082.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T22:22:20.975 INFO:teuthology.orchestra.run.smithi082.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T22:22:20.975 INFO:teuthology.orchestra.run.smithi082.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T22:22:20.975 INFO:teuthology.orchestra.run.smithi082.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T22:22:20.975 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T22:22:20.975 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T22:22:20.975 INFO:teuthology.orchestra.run.smithi082.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T22:22:20.975 INFO:teuthology.orchestra.run.smithi082.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T22:22:20.975 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T22:22:20.975 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T22:22:20.975 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T22:22:20.975 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T22:22:20.975 INFO:teuthology.orchestra.run.smithi082.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T22:22:20.975 INFO:teuthology.orchestra.run.smithi082.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T22:22:20.975 INFO:teuthology.orchestra.run.smithi082.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T22:22:20.975 INFO:teuthology.orchestra.run.smithi082.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T22:22:20.976 INFO:teuthology.orchestra.run.smithi082.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T22:22:20.976 INFO:teuthology.orchestra.run.smithi082.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T22:22:20.976 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T22:22:20.976 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-argparse-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:20.976 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:20.976 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cephfs-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:20.976 INFO:teuthology.orchestra.run.smithi082.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T22:22:20.976 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T22:22:20.976 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T22:22:20.976 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T22:22:20.976 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T22:22:20.976 INFO:teuthology.orchestra.run.smithi082.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T22:22:20.976 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T22:22:20.976 INFO:teuthology.orchestra.run.smithi082.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T22:22:20.977 INFO:teuthology.orchestra.run.smithi082.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T22:22:20.977 INFO:teuthology.orchestra.run.smithi082.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T22:22:20.977 INFO:teuthology.orchestra.run.smithi082.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T22:22:20.977 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T22:22:20.977 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T22:22:20.977 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T22:22:20.977 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T22:22:20.977 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T22:22:20.977 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T22:22:20.977 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T22:22:20.977 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T22:22:20.977 INFO:teuthology.orchestra.run.smithi082.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T22:22:20.977 INFO:teuthology.orchestra.run.smithi082.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T22:22:20.977 INFO:teuthology.orchestra.run.smithi082.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T22:22:20.977 INFO:teuthology.orchestra.run.smithi082.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T22:22:20.978 INFO:teuthology.orchestra.run.smithi082.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T22:22:20.978 INFO:teuthology.orchestra.run.smithi082.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T22:22:20.978 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T22:22:20.978 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T22:22:20.978 INFO:teuthology.orchestra.run.smithi082.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T22:22:20.978 INFO:teuthology.orchestra.run.smithi082.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T22:22:20.978 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T22:22:20.978 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T22:22:20.978 INFO:teuthology.orchestra.run.smithi082.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T22:22:20.978 INFO:teuthology.orchestra.run.smithi082.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T22:22:20.978 INFO:teuthology.orchestra.run.smithi082.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T22:22:20.978 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T22:22:20.978 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T22:22:20.979 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T22:22:20.979 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T22:22:20.979 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T22:22:20.979 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T22:22:20.979 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rados-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:20.979 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rbd-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:20.979 INFO:teuthology.orchestra.run.smithi082.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T22:22:20.979 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T22:22:20.979 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T22:22:20.979 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rgw-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:20.979 INFO:teuthology.orchestra.run.smithi082.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T22:22:20.979 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T22:22:20.979 INFO:teuthology.orchestra.run.smithi082.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T22:22:20.980 INFO:teuthology.orchestra.run.smithi082.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T22:22:20.980 INFO:teuthology.orchestra.run.smithi082.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T22:22:20.980 INFO:teuthology.orchestra.run.smithi082.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T22:22:20.980 INFO:teuthology.orchestra.run.smithi082.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T22:22:20.980 INFO:teuthology.orchestra.run.smithi082.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T22:22:20.980 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T22:22:20.980 INFO:teuthology.orchestra.run.smithi082.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T22:22:20.980 INFO:teuthology.orchestra.run.smithi082.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T22:22:20.980 INFO:teuthology.orchestra.run.smithi082.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T22:22:20.980 INFO:teuthology.orchestra.run.smithi082.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T22:22:20.980 INFO:teuthology.orchestra.run.smithi082.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T22:22:20.980 INFO:teuthology.orchestra.run.smithi082.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T22:22:20.980 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:20.980 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:20.980 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-nbd-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:22:20.981 INFO:teuthology.orchestra.run.smithi082.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T22:22:20.981 INFO:teuthology.orchestra.run.smithi082.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T22:22:20.981 INFO:teuthology.orchestra.run.smithi082.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T22:22:20.981 INFO:teuthology.orchestra.run.smithi082.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T22:22:20.981 INFO:teuthology.orchestra.run.smithi082.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T22:22:20.981 INFO:teuthology.orchestra.run.smithi082.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T22:22:20.981 INFO:teuthology.orchestra.run.smithi082.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T22:22:20.981 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-28T22:22:20.981 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-28T22:22:21.188 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T22:22:21.229 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : librbd1-2:19.0.0-1899.g323d23dc.el9.x86_64 124/198 2024-03-28T22:22:21.353 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librbd1-2:19.0.0-1899.g323d23dc.el9.x86_64 124/198 2024-03-28T22:22:21.386 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rbd-2:19.0.0-1899.g323d23dc.el9.x86_64 125/198 2024-03-28T22:22:21.431 DEBUG:teuthology.parallel:result is None 2024-03-28T22:22:21.442 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T22:22:21.991 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T22:22:22.057 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librgw2-2:19.0.0-1899.g323d23dc.el9.x86_64 128/198 2024-03-28T22:22:22.159 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librgw2-2:19.0.0-1899.g323d23dc.el9.x86_64 128/198 2024-03-28T22:22:22.184 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rgw-2:19.0.0-1899.g323d23dc.el9.x86_64 129/198 2024-03-28T22:22:24.912 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 130/198 2024-03-28T22:22:24.947 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 130/198 2024-03-28T22:22:25.602 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 130/198 2024-03-28T22:22:25.642 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-base-2:19.0.0-1899.g323d23dc.el9.x86_64 131/198 2024-03-28T22:22:25.798 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-base-2:19.0.0-1899.g323d23dc.el9.x86_64 131/198 2024-03-28T22:22:25.798 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T22:22:25.799 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T22:22:25.799 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:22:25.842 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-selinux-2:19.0.0-1899.g323d23dc.el9.x86_64 132/198 2024-03-28T22:22:39.082 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1899.g323d23dc.el9.x86_64 132/198 2024-03-28T22:22:39.082 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /sys 2024-03-28T22:22:39.082 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /proc 2024-03-28T22:22:39.083 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /mnt 2024-03-28T22:22:39.083 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /var/tmp 2024-03-28T22:22:39.083 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /home 2024-03-28T22:22:39.083 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /root 2024-03-28T22:22:39.083 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /tmp 2024-03-28T22:22:39.083 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:22:39.291 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-volume-2:19.0.0-1899.g323d23dc.el9.noarch 133/198 2024-03-28T22:22:39.341 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-volume-2:19.0.0-1899.g323d23dc.el9.noarch 133/198 2024-03-28T22:22:39.341 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T22:22:39.341 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T22:22:39.341 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:22:41.000 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-osd-2:19.0.0-1899.g323d23dc.el9.x86_64 134/198 2024-03-28T22:22:41.082 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-osd-2:19.0.0-1899.g323d23dc.el9.x86_64 134/198 2024-03-28T22:22:41.082 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T22:22:41.083 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T22:22:41.083 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T22:22:41.083 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T22:22:41.083 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:22:41.342 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mds-2:19.0.0-1899.g323d23dc.el9.x86_64 135/198 2024-03-28T22:22:41.419 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mds-2:19.0.0-1899.g323d23dc.el9.x86_64 135/198 2024-03-28T22:22:41.420 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T22:22:41.420 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T22:22:41.420 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T22:22:41.420 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T22:22:41.420 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:22:41.934 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mon-2:19.0.0-1899.g323d23dc.el9.x86_64 136/198 2024-03-28T22:22:42.029 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mon-2:19.0.0-1899.g323d23dc.el9.x86_64 136/198 2024-03-28T22:22:42.029 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T22:22:42.029 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T22:22:42.029 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T22:22:42.029 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T22:22:42.029 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:22:42.145 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T22:22:42.804 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 138/198 2024-03-28T22:22:42.838 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 138/198 2024-03-28T22:22:42.889 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1899.g323d23dc.e 139/198 2024-03-28T22:22:42.994 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1899.g323d23dc. 140/198 2024-03-28T22:22:43.024 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1899.g323d23dc.el9.noa 141/198 2024-03-28T22:22:44.340 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1899.g323d23dc.el9.noa 141/198 2024-03-28T22:22:44.385 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1899.g323d23dc.el9.n 142/198 2024-03-28T22:22:45.540 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1899.g323d23dc.el9.n 142/198 2024-03-28T22:22:45.576 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1899.g323 143/198 2024-03-28T22:22:45.726 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1899.g323 143/198 2024-03-28T22:22:45.923 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1899.g323d23dc.el 144/198 2024-03-28T22:22:45.953 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-2:19.0.0-1899.g323d23dc.el9.x86_64 145/198 2024-03-28T22:22:46.045 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1899.g323d23dc.el9.x86_64 145/198 2024-03-28T22:22:46.045 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T22:22:46.045 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T22:22:46.045 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T22:22:46.045 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T22:22:46.045 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:22:46.104 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-rook-2:19.0.0-1899.g323d23dc.el9.noarch 146/198 2024-03-28T22:22:46.145 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1899.g323d23dc.el9.noarch 146/198 2024-03-28T22:22:46.233 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-2:19.0.0-1899.g323d23dc.el9.x86_64 147/198 2024-03-28T22:22:47.230 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T22:22:47.285 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-radosgw-2:19.0.0-1899.g323d23dc.el9.x86_64 149/198 2024-03-28T22:22:47.385 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1899.g323d23dc.el9.x86_64 149/198 2024-03-28T22:22:47.385 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T22:22:47.385 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T22:22:47.385 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T22:22:47.386 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T22:22:47.386 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:22:47.440 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1899.g323d2 150/198 2024-03-28T22:22:47.526 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1899.g323d2 150/198 2024-03-28T22:22:47.526 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T22:22:47.526 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T22:22:47.526 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:22:47.616 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : cephfs-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 151/198 2024-03-28T22:22:47.695 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 151/198 2024-03-28T22:22:47.695 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T22:22:47.695 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T22:22:47.695 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T22:22:47.695 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T22:22:47.695 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:22:48.066 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : rbd-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 152/198 2024-03-28T22:22:48.141 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 152/198 2024-03-28T22:22:48.141 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T22:22:48.141 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T22:22:48.141 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T22:22:48.141 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T22:22:48.141 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:22:53.207 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-test-2:19.0.0-1899.g323d23dc.el9.x86_64 153/198 2024-03-28T22:22:53.262 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : rbd-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 154/198 2024-03-28T22:22:53.296 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : rbd-nbd-2:19.0.0-1899.g323d23dc.el9.x86_64 155/198 2024-03-28T22:22:53.343 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T22:22:53.644 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : cephfs-shell-2:19.0.0-1899.g323d23dc.el9.x86_64 157/198 2024-03-28T22:22:53.779 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T22:22:53.831 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 159/198 2024-03-28T22:22:53.891 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T22:22:53.963 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T22:22:54.017 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T22:22:54.051 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libcephfs-devel-2:19.0.0-1899.g323d23dc.el9.x86_ 163/198 2024-03-28T22:22:54.097 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T22:22:54.149 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T22:22:54.247 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T22:22:54.368 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T22:22:54.750 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T22:22:55.628 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T22:22:55.679 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T22:22:55.749 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T22:22:57.172 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : cephfs-top-2:19.0.0-1899.g323d23dc.el9.noarch 170/198 2024-03-28T22:22:57.276 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T22:22:57.336 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T22:22:57.479 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T22:22:57.520 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T22:22:57.561 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T22:22:57.604 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T22:22:57.648 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T22:22:57.648 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T22:22:57.800 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T22:22:57.841 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T22:22:57.842 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T22:22:57.971 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T22:22:58.016 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T22:22:58.062 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T22:22:58.111 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T22:22:58.144 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T22:22:58.213 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T22:22:58.283 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T22:22:58.332 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T22:22:58.365 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T22:22:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T22:22:58.441 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T22:22:58.474 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T22:22:58.511 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T22:22:58.511 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T22:22:58.857 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T22:22:58.858 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T22:22:58.997 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T22:22:59.027 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T22:22:59.082 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T22:22:59.122 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T22:22:59.170 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T22:22:59.296 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T22:22:59.561 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T22:22:59.561 INFO:teuthology.orchestra.run.smithi043.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-28T22:22:59.561 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:23:04.660 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-2:19.0.0-1899.g323d23dc.el9.x86_64 1/198 2024-03-28T22:23:04.660 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-base-2:19.0.0-1899.g323d23dc.el9.x86_64 2/198 2024-03-28T22:23:04.660 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 3/198 2024-03-28T22:23:04.660 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 4/198 2024-03-28T22:23:04.660 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1899.g323d2 5/198 2024-03-28T22:23:04.661 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mds-2:19.0.0-1899.g323d23dc.el9.x86_64 6/198 2024-03-28T22:23:04.661 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-2:19.0.0-1899.g323d23dc.el9.x86_64 7/198 2024-03-28T22:23:04.661 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mon-2:19.0.0-1899.g323d23dc.el9.x86_64 8/198 2024-03-28T22:23:04.661 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-osd-2:19.0.0-1899.g323d23dc.el9.x86_64 9/198 2024-03-28T22:23:04.661 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-radosgw-2:19.0.0-1899.g323d23dc.el9.x86_64 10/198 2024-03-28T22:23:04.661 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-selinux-2:19.0.0-1899.g323d23dc.el9.x86_64 11/198 2024-03-28T22:23:04.661 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-test-2:19.0.0-1899.g323d23dc.el9.x86_64 12/198 2024-03-28T22:23:04.661 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : cephfs-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 13/198 2024-03-28T22:23:04.661 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : cephfs-shell-2:19.0.0-1899.g323d23dc.el9.x86_64 14/198 2024-03-28T22:23:04.661 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libcephfs-devel-2:19.0.0-1899.g323d23dc.el9.x86_ 15/198 2024-03-28T22:23:04.661 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libcephfs2-2:19.0.0-1899.g323d23dc.el9.x86_64 16/198 2024-03-28T22:23:04.661 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libcephsqlite-2:19.0.0-1899.g323d23dc.el9.x86_64 17/198 2024-03-28T22:23:04.661 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librados-devel-2:19.0.0-1899.g323d23dc.el9.x86_6 18/198 2024-03-28T22:23:04.662 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libradosstriper1-2:19.0.0-1899.g323d23dc.el9.x86 19/198 2024-03-28T22:23:04.662 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librgw2-2:19.0.0-1899.g323d23dc.el9.x86_64 20/198 2024-03-28T22:23:04.663 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1899.g323d23dc.el 21/198 2024-03-28T22:23:04.663 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-ceph-common-2:19.0.0-1899.g323d23dc.el9. 22/198 2024-03-28T22:23:04.663 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cephfs-2:19.0.0-1899.g323d23dc.el9.x86_6 23/198 2024-03-28T22:23:04.663 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rados-2:19.0.0-1899.g323d23dc.el9.x86_64 24/198 2024-03-28T22:23:04.663 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rbd-2:19.0.0-1899.g323d23dc.el9.x86_64 25/198 2024-03-28T22:23:04.663 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rgw-2:19.0.0-1899.g323d23dc.el9.x86_64 26/198 2024-03-28T22:23:04.663 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : rbd-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 27/198 2024-03-28T22:23:04.679 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : rbd-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 28/198 2024-03-28T22:23:04.679 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : rbd-nbd-2:19.0.0-1899.g323d23dc.el9.x86_64 29/198 2024-03-28T22:23:04.679 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1899.g323d23dc. 30/198 2024-03-28T22:23:04.679 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1899.g323d23dc.el9.noa 31/198 2024-03-28T22:23:04.679 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1899.g323d23dc.el9.n 32/198 2024-03-28T22:23:04.679 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1899.g323 33/198 2024-03-28T22:23:04.679 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1899.g323d23dc.el 34/198 2024-03-28T22:23:04.679 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1899.g323d23dc.el9.noarch 35/198 2024-03-28T22:23:04.679 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1899.g323d23dc.e 36/198 2024-03-28T22:23:04.679 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-volume-2:19.0.0-1899.g323d23dc.el9.noarch 37/198 2024-03-28T22:23:04.679 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 38/198 2024-03-28T22:23:04.680 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : cephfs-top-2:19.0.0-1899.g323d23dc.el9.noarch 39/198 2024-03-28T22:23:04.680 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T22:23:04.680 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T22:23:04.680 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T22:23:04.680 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T22:23:04.680 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T22:23:04.680 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T22:23:04.680 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T22:23:04.680 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T22:23:04.680 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T22:23:04.680 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T22:23:04.680 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T22:23:04.680 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T22:23:04.680 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T22:23:04.681 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T22:23:04.681 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T22:23:04.681 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T22:23:04.681 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T22:23:04.681 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T22:23:04.681 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-03-28T22:23:04.681 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-03-28T22:23:04.681 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-03-28T22:23:04.681 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-03-28T22:23:04.681 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-28T22:23:04.681 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-28T22:23:04.681 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-28T22:23:04.681 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-28T22:23:04.682 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-28T22:23:04.682 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-28T22:23:04.682 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-28T22:23:04.682 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-28T22:23:04.682 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-28T22:23:04.682 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-28T22:23:04.682 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-28T22:23:04.682 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-28T22:23:04.682 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-28T22:23:04.682 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-28T22:23:04.682 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-28T22:23:04.682 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-28T22:23:04.682 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-28T22:23:04.682 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-28T22:23:04.683 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-28T22:23:04.683 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-28T22:23:04.683 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-03-28T22:23:04.683 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-03-28T22:23:04.683 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-03-28T22:23:04.683 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-03-28T22:23:04.683 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-03-28T22:23:04.683 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-03-28T22:23:04.683 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-03-28T22:23:04.683 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T22:23:04.683 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-03-28T22:23:04.683 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-03-28T22:23:04.683 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-03-28T22:23:04.683 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-03-28T22:23:04.684 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-03-28T22:23:04.684 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-03-28T22:23:04.684 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-03-28T22:23:04.684 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-03-28T22:23:04.684 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-03-28T22:23:04.684 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-03-28T22:23:04.684 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-03-28T22:23:04.684 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-03-28T22:23:04.684 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-03-28T22:23:04.684 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-03-28T22:23:04.684 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T22:23:04.684 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T22:23:04.684 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T22:23:04.684 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T22:23:04.685 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T22:23:04.685 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T22:23:04.685 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T22:23:04.685 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T22:23:04.685 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T22:23:04.685 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T22:23:04.685 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T22:23:04.685 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T22:23:04.685 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T22:23:04.685 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T22:23:04.685 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T22:23:04.685 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T22:23:04.685 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T22:23:04.685 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T22:23:04.686 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T22:23:04.686 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T22:23:04.686 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T22:23:04.686 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T22:23:04.686 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T22:23:04.686 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T22:23:04.686 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T22:23:04.686 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T22:23:04.686 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T22:23:04.686 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T22:23:04.687 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T22:23:04.687 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T22:23:04.687 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T22:23:04.687 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T22:23:04.687 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T22:23:04.687 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T22:23:04.687 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T22:23:04.687 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T22:23:04.687 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T22:23:04.687 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T22:23:04.687 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T22:23:04.687 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T22:23:04.687 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T22:23:04.687 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T22:23:04.688 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T22:23:04.688 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T22:23:04.688 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T22:23:04.688 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T22:23:04.688 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T22:23:04.688 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librados2-2:19.0.0-1899.g323d23dc.el9.x86_64 151/198 2024-03-28T22:23:04.688 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T22:23:04.688 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librbd1-2:19.0.0-1899.g323d23dc.el9.x86_64 153/198 2024-03-28T22:23:04.688 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T22:23:04.688 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 155/198 2024-03-28T22:23:04.688 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-03-28T22:23:04.688 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T22:23:04.688 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T22:23:04.688 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T22:23:04.689 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T22:23:04.689 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T22:23:04.689 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T22:23:04.689 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-03-28T22:23:04.689 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-03-28T22:23:04.689 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T22:23:04.689 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T22:23:04.689 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T22:23:04.689 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T22:23:04.689 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T22:23:04.689 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T22:23:04.689 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T22:23:04.689 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T22:23:04.689 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T22:23:04.689 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T22:23:04.690 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T22:23:04.690 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T22:23:04.690 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T22:23:04.690 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T22:23:04.690 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T22:23:04.690 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T22:23:04.690 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T22:23:04.690 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T22:23:04.690 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T22:23:04.690 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T22:23:04.690 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T22:23:04.690 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T22:23:04.690 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T22:23:04.690 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T22:23:04.691 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T22:23:04.691 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T22:23:04.691 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T22:23:04.691 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T22:23:04.691 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T22:23:04.691 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T22:23:04.691 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T22:23:04.691 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T22:23:04.691 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T22:23:07.400 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T22:23:07.400 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:23:07.400 INFO:teuthology.orchestra.run.smithi043.stdout:Upgraded: 2024-03-28T22:23:07.400 INFO:teuthology.orchestra.run.smithi043.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T22:23:07.400 INFO:teuthology.orchestra.run.smithi043.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T22:23:07.400 INFO:teuthology.orchestra.run.smithi043.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T22:23:07.400 INFO:teuthology.orchestra.run.smithi043.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T22:23:07.400 INFO:teuthology.orchestra.run.smithi043.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T22:23:07.401 INFO:teuthology.orchestra.run.smithi043.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T22:23:07.401 INFO:teuthology.orchestra.run.smithi043.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T22:23:07.401 INFO:teuthology.orchestra.run.smithi043.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T22:23:07.401 INFO:teuthology.orchestra.run.smithi043.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T22:23:07.401 INFO:teuthology.orchestra.run.smithi043.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T22:23:07.401 INFO:teuthology.orchestra.run.smithi043.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T22:23:07.401 INFO:teuthology.orchestra.run.smithi043.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T22:23:07.401 INFO:teuthology.orchestra.run.smithi043.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T22:23:07.401 INFO:teuthology.orchestra.run.smithi043.stdout: librados2-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:23:07.401 INFO:teuthology.orchestra.run.smithi043.stdout: librbd1-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:23:07.401 INFO:teuthology.orchestra.run.smithi043.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T22:23:07.401 INFO:teuthology.orchestra.run.smithi043.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T22:23:07.401 INFO:teuthology.orchestra.run.smithi043.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T22:23:07.402 INFO:teuthology.orchestra.run.smithi043.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T22:23:07.402 INFO:teuthology.orchestra.run.smithi043.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T22:23:07.402 INFO:teuthology.orchestra.run.smithi043.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T22:23:07.402 INFO:teuthology.orchestra.run.smithi043.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T22:23:07.402 INFO:teuthology.orchestra.run.smithi043.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T22:23:07.402 INFO:teuthology.orchestra.run.smithi043.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T22:23:07.402 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-03-28T22:23:07.402 INFO:teuthology.orchestra.run.smithi043.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T22:23:07.402 INFO:teuthology.orchestra.run.smithi043.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T22:23:07.402 INFO:teuthology.orchestra.run.smithi043.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T22:23:07.402 INFO:teuthology.orchestra.run.smithi043.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T22:23:07.403 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:23:07.403 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-base-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:23:07.403 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:23:07.403 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:23:07.403 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-grafana-dashboards-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T22:23:07.403 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-immutable-object-cache-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:23:07.403 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mds-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:23:07.403 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:23:07.403 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T22:23:07.403 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-dashboard-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T22:23:07.403 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T22:23:07.404 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-modules-core-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T22:23:07.404 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-rook-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T22:23:07.404 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mon-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:23:07.404 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-osd-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:23:07.404 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-prometheus-alerts-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T22:23:07.404 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-radosgw-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:23:07.404 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-selinux-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:23:07.404 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-test-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:23:07.404 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-volume-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T22:23:07.404 INFO:teuthology.orchestra.run.smithi043.stdout: cephadm-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T22:23:07.404 INFO:teuthology.orchestra.run.smithi043.stdout: cephfs-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:23:07.404 INFO:teuthology.orchestra.run.smithi043.stdout: cephfs-shell-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:23:07.405 INFO:teuthology.orchestra.run.smithi043.stdout: cephfs-top-2:19.0.0-1899.g323d23dc.el9.noarch 2024-03-28T22:23:07.405 INFO:teuthology.orchestra.run.smithi043.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T22:23:07.405 INFO:teuthology.orchestra.run.smithi043.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T22:23:07.405 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T22:23:07.405 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T22:23:07.405 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T22:23:07.405 INFO:teuthology.orchestra.run.smithi043.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T22:23:07.405 INFO:teuthology.orchestra.run.smithi043.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T22:23:07.405 INFO:teuthology.orchestra.run.smithi043.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T22:23:07.405 INFO:teuthology.orchestra.run.smithi043.stdout: jq-1.6-16.el9.x86_64 2024-03-28T22:23:07.405 INFO:teuthology.orchestra.run.smithi043.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T22:23:07.406 INFO:teuthology.orchestra.run.smithi043.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T22:23:07.406 INFO:teuthology.orchestra.run.smithi043.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T22:23:07.406 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs-devel-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:23:07.406 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs2-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:23:07.406 INFO:teuthology.orchestra.run.smithi043.stdout: libcephsqlite-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:23:07.406 INFO:teuthology.orchestra.run.smithi043.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T22:23:07.406 INFO:teuthology.orchestra.run.smithi043.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T22:23:07.406 INFO:teuthology.orchestra.run.smithi043.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T22:23:07.406 INFO:teuthology.orchestra.run.smithi043.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T22:23:07.406 INFO:teuthology.orchestra.run.smithi043.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T22:23:07.406 INFO:teuthology.orchestra.run.smithi043.stdout: librados-devel-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:23:07.407 INFO:teuthology.orchestra.run.smithi043.stdout: libradosstriper1-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:23:07.407 INFO:teuthology.orchestra.run.smithi043.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T22:23:07.407 INFO:teuthology.orchestra.run.smithi043.stdout: librgw2-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:23:07.407 INFO:teuthology.orchestra.run.smithi043.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T22:23:07.407 INFO:teuthology.orchestra.run.smithi043.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T22:23:07.407 INFO:teuthology.orchestra.run.smithi043.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T22:23:07.407 INFO:teuthology.orchestra.run.smithi043.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T22:23:07.407 INFO:teuthology.orchestra.run.smithi043.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T22:23:07.407 INFO:teuthology.orchestra.run.smithi043.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T22:23:07.407 INFO:teuthology.orchestra.run.smithi043.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T22:23:07.407 INFO:teuthology.orchestra.run.smithi043.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T22:23:07.408 INFO:teuthology.orchestra.run.smithi043.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T22:23:07.408 INFO:teuthology.orchestra.run.smithi043.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T22:23:07.408 INFO:teuthology.orchestra.run.smithi043.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T22:23:07.408 INFO:teuthology.orchestra.run.smithi043.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T22:23:07.408 INFO:teuthology.orchestra.run.smithi043.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T22:23:07.408 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T22:23:07.408 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T22:23:07.408 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T22:23:07.408 INFO:teuthology.orchestra.run.smithi043.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T22:23:07.408 INFO:teuthology.orchestra.run.smithi043.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T22:23:07.408 INFO:teuthology.orchestra.run.smithi043.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T22:23:07.408 INFO:teuthology.orchestra.run.smithi043.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T22:23:07.409 INFO:teuthology.orchestra.run.smithi043.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T22:23:07.409 INFO:teuthology.orchestra.run.smithi043.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T22:23:07.409 INFO:teuthology.orchestra.run.smithi043.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T22:23:07.409 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T22:23:07.409 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-argparse-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:23:07.409 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-common-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:23:07.409 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cephfs-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:23:07.409 INFO:teuthology.orchestra.run.smithi043.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T22:23:07.409 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T22:23:07.409 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T22:23:07.409 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T22:23:07.410 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T22:23:07.410 INFO:teuthology.orchestra.run.smithi043.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T22:23:07.410 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T22:23:07.410 INFO:teuthology.orchestra.run.smithi043.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T22:23:07.410 INFO:teuthology.orchestra.run.smithi043.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T22:23:07.410 INFO:teuthology.orchestra.run.smithi043.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T22:23:07.410 INFO:teuthology.orchestra.run.smithi043.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T22:23:07.410 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T22:23:07.410 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T22:23:07.410 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T22:23:07.410 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T22:23:07.411 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T22:23:07.411 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T22:23:07.411 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T22:23:07.411 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T22:23:07.411 INFO:teuthology.orchestra.run.smithi043.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T22:23:07.411 INFO:teuthology.orchestra.run.smithi043.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T22:23:07.411 INFO:teuthology.orchestra.run.smithi043.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T22:23:07.411 INFO:teuthology.orchestra.run.smithi043.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T22:23:07.411 INFO:teuthology.orchestra.run.smithi043.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T22:23:07.411 INFO:teuthology.orchestra.run.smithi043.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T22:23:07.411 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T22:23:07.412 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T22:23:07.412 INFO:teuthology.orchestra.run.smithi043.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T22:23:07.412 INFO:teuthology.orchestra.run.smithi043.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T22:23:07.412 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T22:23:07.412 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T22:23:07.412 INFO:teuthology.orchestra.run.smithi043.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T22:23:07.412 INFO:teuthology.orchestra.run.smithi043.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T22:23:07.412 INFO:teuthology.orchestra.run.smithi043.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T22:23:07.412 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T22:23:07.412 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T22:23:07.412 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T22:23:07.412 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T22:23:07.413 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T22:23:07.413 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T22:23:07.413 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rados-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:23:07.413 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rbd-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:23:07.413 INFO:teuthology.orchestra.run.smithi043.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T22:23:07.413 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T22:23:07.413 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T22:23:07.413 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rgw-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:23:07.413 INFO:teuthology.orchestra.run.smithi043.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T22:23:07.413 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T22:23:07.413 INFO:teuthology.orchestra.run.smithi043.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T22:23:07.413 INFO:teuthology.orchestra.run.smithi043.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T22:23:07.414 INFO:teuthology.orchestra.run.smithi043.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T22:23:07.414 INFO:teuthology.orchestra.run.smithi043.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T22:23:07.414 INFO:teuthology.orchestra.run.smithi043.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T22:23:07.414 INFO:teuthology.orchestra.run.smithi043.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T22:23:07.414 INFO:teuthology.orchestra.run.smithi043.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T22:23:07.414 INFO:teuthology.orchestra.run.smithi043.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T22:23:07.414 INFO:teuthology.orchestra.run.smithi043.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T22:23:07.414 INFO:teuthology.orchestra.run.smithi043.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T22:23:07.414 INFO:teuthology.orchestra.run.smithi043.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T22:23:07.414 INFO:teuthology.orchestra.run.smithi043.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T22:23:07.414 INFO:teuthology.orchestra.run.smithi043.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T22:23:07.415 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-fuse-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:23:07.415 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-mirror-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:23:07.415 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-nbd-2:19.0.0-1899.g323d23dc.el9.x86_64 2024-03-28T22:23:07.415 INFO:teuthology.orchestra.run.smithi043.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T22:23:07.415 INFO:teuthology.orchestra.run.smithi043.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T22:23:07.415 INFO:teuthology.orchestra.run.smithi043.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T22:23:07.415 INFO:teuthology.orchestra.run.smithi043.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T22:23:07.415 INFO:teuthology.orchestra.run.smithi043.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T22:23:07.415 INFO:teuthology.orchestra.run.smithi043.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T22:23:07.416 INFO:teuthology.orchestra.run.smithi043.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T22:23:07.416 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:23:07.416 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-03-28T22:23:08.028 DEBUG:teuthology.parallel:result is None 2024-03-28T22:23:08.029 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=323d23dc228bbcb7f16f188fdc9c0dcb801844be 2024-03-28T22:23:08.176 DEBUG:teuthology.orchestra.run.smithi043:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T22:23:08.236 INFO:teuthology.orchestra.run.smithi043.stdout:19.0.0-1899.g323d23dc.el9 2024-03-28T22:23:08.236 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1899.g323d23dc.el9 2024-03-28T22:23:08.236 INFO:teuthology.task.install:The correct ceph version 19.0.0-1899.g323d23dc is installed. 2024-03-28T22:23:08.238 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=323d23dc228bbcb7f16f188fdc9c0dcb801844be 2024-03-28T22:23:08.380 DEBUG:teuthology.orchestra.run.smithi082:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T22:23:08.406 INFO:teuthology.orchestra.run.smithi082.stdout:19.0.0-1899.g323d23dc.el9 2024-03-28T22:23:08.406 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1899.g323d23dc.el9 2024-03-28T22:23:08.406 INFO:teuthology.task.install:The correct ceph version 19.0.0-1899.g323d23dc is installed. 2024-03-28T22:23:08.408 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=323d23dc228bbcb7f16f188fdc9c0dcb801844be 2024-03-28T22:23:08.637 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T22:23:08.662 INFO:teuthology.orchestra.run.smithi172.stdout:19.0.0-1899.g323d23dc.el9 2024-03-28T22:23:08.663 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1899.g323d23dc.el9 2024-03-28T22:23:08.663 INFO:teuthology.task.install:The correct ceph version 19.0.0-1899.g323d23dc is installed. 2024-03-28T22:23:08.665 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-28T22:23:08.665 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-28T22:23:08.665 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T22:23:08.751 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-28T22:23:08.751 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T22:23:08.785 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-28T22:23:08.786 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T22:23:08.821 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-28T22:23:08.821 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-28T22:23:08.821 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T22:23:08.897 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T22:23:09.009 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-28T22:23:09.009 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T22:23:09.041 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T22:23:09.110 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-28T22:23:09.111 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T22:23:09.143 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T22:23:09.212 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-28T22:23:09.213 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-28T22:23:09.213 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T22:23:09.284 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T22:23:09.397 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-28T22:23:09.397 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T22:23:09.424 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T22:23:09.491 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-28T22:23:09.491 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T22:23:09.521 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T22:23:09.589 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-28T22:23:09.589 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-28T22:23:09.589 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T22:23:09.657 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T22:23:09.767 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-28T22:23:09.767 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T22:23:09.793 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T22:23:09.860 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-28T22:23:09.860 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T22:23:09.887 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T22:23:09.953 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-28T22:23:10.055 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '323d23dc228bbcb7f16f188fdc9c0dcb801844be', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-03-28T22:23:10.055 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be 2024-03-28T22:23:10.056 INFO:tasks.cephadm:Cluster fsid is c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:23:10.056 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-28T22:23:10.056 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.43', 'mon.b': '172.21.15.82', 'mon.c': '172.21.15.172'} 2024-03-28T22:23:10.056 INFO:tasks.cephadm:First mon is mon.a on smithi043 2024-03-28T22:23:10.056 INFO:tasks.cephadm:First mgr is x 2024-03-28T22:23:10.056 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-28T22:23:10.056 DEBUG:teuthology.orchestra.run.smithi043:> sudo hostname $(hostname -s) 2024-03-28T22:23:10.128 DEBUG:teuthology.orchestra.run.smithi082:> sudo hostname $(hostname -s) 2024-03-28T22:23:10.155 DEBUG:teuthology.orchestra.run.smithi172:> sudo hostname $(hostname -s) 2024-03-28T22:23:10.182 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-28T22:23:10.182 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=323d23dc228bbcb7f16f188fdc9c0dcb801844be 2024-03-28T22:23:10.321 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '323d23dc228bbcb7f16f188fdc9c0dcb801844be', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54719/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1899-g323d23dc', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1899.g323d23dc'}, 'url': 'https://4.chacra.ceph.com/r/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-27 18:23:47.768164', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-28T22:23:10.461 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref squid, sha1 323d23dc228bbcb7f16f188fdc9c0dcb801844be from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=323d23dc228bbcb7f16f188fdc9c0dcb801844be 2024-03-28T22:23:10.462 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/x86_64/flavors/default/cephadm 2024-03-28T22:23:10.463 DEBUG:teuthology.orchestra.run.smithi043:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T22:23:11.148 INFO:teuthology.orchestra.run.smithi043.stdout:-rw-r--r--. 1 ubuntu ubuntu 766659 Mar 28 22:23 /home/ubuntu/cephtest/cephadm 2024-03-28T22:23:11.148 DEBUG:teuthology.orchestra.run.smithi082:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T22:23:11.619 INFO:teuthology.orchestra.run.smithi082.stdout:-rw-r--r--. 1 ubuntu ubuntu 766659 Mar 28 22:23 /home/ubuntu/cephtest/cephadm 2024-03-28T22:23:11.619 DEBUG:teuthology.orchestra.run.smithi172:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/squid/323d23dc228bbcb7f16f188fdc9c0dcb801844be/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T22:23:12.086 INFO:teuthology.orchestra.run.smithi172.stdout:-rw-r--r--. 1 ubuntu ubuntu 766659 Mar 28 22:23 /home/ubuntu/cephtest/cephadm 2024-03-28T22:23:12.086 DEBUG:teuthology.orchestra.run.smithi043:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T22:23:12.139 DEBUG:teuthology.orchestra.run.smithi082:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T22:23:12.158 DEBUG:teuthology.orchestra.run.smithi172:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T22:23:12.192 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be on all hosts... 2024-03-28T22:23:12.193 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be pull 2024-03-28T22:23:12.199 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be pull 2024-03-28T22:23:12.202 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be pull 2024-03-28T22:23:12.978 INFO:teuthology.orchestra.run.smithi082.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be... 2024-03-28T22:23:12.987 INFO:teuthology.orchestra.run.smithi172.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be... 2024-03-28T22:23:13.745 INFO:teuthology.orchestra.run.smithi043.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be... 2024-03-28T22:23:51.613 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:23:51.613 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_version": "ceph version 19.0.0-1899-g323d23dc (323d23dc228bbcb7f16f188fdc9c0dcb801844be) squid (dev)", 2024-03-28T22:23:51.613 INFO:teuthology.orchestra.run.smithi043.stdout: "image_id": "bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52", 2024-03-28T22:23:51.613 INFO:teuthology.orchestra.run.smithi043.stdout: "repo_digests": [ 2024-03-28T22:23:51.613 INFO:teuthology.orchestra.run.smithi043.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15" 2024-03-28T22:23:51.613 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-28T22:23:51.613 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:24:07.128 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-28T22:24:07.129 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph_version": "ceph version 19.0.0-1899-g323d23dc (323d23dc228bbcb7f16f188fdc9c0dcb801844be) squid (dev)", 2024-03-28T22:24:07.129 INFO:teuthology.orchestra.run.smithi172.stdout: "image_id": "bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52", 2024-03-28T22:24:07.129 INFO:teuthology.orchestra.run.smithi172.stdout: "repo_digests": [ 2024-03-28T22:24:07.129 INFO:teuthology.orchestra.run.smithi172.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15" 2024-03-28T22:24:07.129 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-28T22:24:07.129 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-28T22:24:07.232 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-28T22:24:07.232 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_version": "ceph version 19.0.0-1899-g323d23dc (323d23dc228bbcb7f16f188fdc9c0dcb801844be) squid (dev)", 2024-03-28T22:24:07.232 INFO:teuthology.orchestra.run.smithi082.stdout: "image_id": "bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52", 2024-03-28T22:24:07.232 INFO:teuthology.orchestra.run.smithi082.stdout: "repo_digests": [ 2024-03-28T22:24:07.232 INFO:teuthology.orchestra.run.smithi082.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15" 2024-03-28T22:24:07.232 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-03-28T22:24:07.232 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-28T22:24:07.255 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /etc/ceph 2024-03-28T22:24:07.364 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /etc/ceph 2024-03-28T22:24:07.394 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /etc/ceph 2024-03-28T22:24:07.423 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 777 /etc/ceph 2024-03-28T22:24:07.523 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 777 /etc/ceph 2024-03-28T22:24:07.551 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 777 /etc/ceph 2024-03-28T22:24:07.578 INFO:tasks.cephadm:Writing seed config... 2024-03-28T22:24:07.579 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-28T22:24:07.579 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-28T22:24:07.579 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-28T22:24:07.579 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-28T22:24:07.579 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-28T22:24:07.579 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-28T22:24:07.579 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-28T22:24:07.580 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-28T22:24:07.580 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-28T22:24:07.580 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-28T22:24:07.580 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-28T22:24:07.580 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-28T22:24:07.580 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-28T22:24:07.580 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-28T22:24:07.580 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-28T22:24:07.580 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-28T22:24:07.580 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-28T22:24:07.580 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-03-28T22:24:07.580 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-03-28T22:24:07.580 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-28T22:24:07.580 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-28T22:24:07.580 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-28T22:24:07.581 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-28T22:24:07.581 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-03-28T22:24:07.581 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-03-28T22:24:07.581 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-28T22:24:07.581 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-28T22:24:07.581 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-28T22:24:07.581 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-28T22:24:07.581 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-28T22:24:07.581 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-03-28T22:24:07.581 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-28T22:24:07.581 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-28T22:24:07.581 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-28T22:24:07.581 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-28T22:24:07.581 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-28T22:24:07.581 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-28T22:24:07.581 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-28T22:24:07.582 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-28T22:24:07.582 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-28T22:24:07.582 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-28T22:24:07.582 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-28T22:24:07.582 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-28T22:24:07.582 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-28T22:24:07.582 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-28T22:24:07.582 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-28T22:24:07.582 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-03-28T22:24:07.583 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-28T22:24:07.583 DEBUG:teuthology.orchestra.run.smithi043:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-28T22:24:07.645 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = c2276428-ed51-11ee-95d0-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-28T22:24:07.645 DEBUG:teuthology.orchestra.run.smithi043:mon.a> sudo journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mon.a.service 2024-03-28T22:24:07.693 DEBUG:teuthology.orchestra.run.smithi043:mgr.x> sudo journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mgr.x.service 2024-03-28T22:24:07.739 INFO:tasks.cephadm:Bootstrapping... 2024-03-28T22:24:07.739 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be -v bootstrap --fsid c2276428-ed51-11ee-95d0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.43 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-28T22:24:08.017 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-03-28T22:24:08.017 INFO:teuthology.orchestra.run.smithi043.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be', '-v', 'bootstrap', '--fsid', 'c2276428-ed51-11ee-95d0-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.43', '--skip-admin-label'] 2024-03-28T22:24:08.077 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T22:24:08.077 INFO:teuthology.orchestra.run.smithi043.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-28T22:24:08.078 INFO:teuthology.orchestra.run.smithi043.stdout:Verifying podman|docker is present... 2024-03-28T22:24:08.131 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T22:24:08.131 INFO:teuthology.orchestra.run.smithi043.stdout:Verifying lvm2 is present... 2024-03-28T22:24:08.131 INFO:teuthology.orchestra.run.smithi043.stdout:Verifying time synchronization is in place... 2024-03-28T22:24:08.150 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-28T22:24:08.150 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-28T22:24:08.169 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-28T22:24:08.169 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout inactive 2024-03-28T22:24:08.195 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout enabled 2024-03-28T22:24:08.217 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout active 2024-03-28T22:24:08.218 INFO:teuthology.orchestra.run.smithi043.stdout:Unit chronyd.service is enabled and running 2024-03-28T22:24:08.218 INFO:teuthology.orchestra.run.smithi043.stdout:Repeating the final host check... 2024-03-28T22:24:08.277 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T22:24:08.277 INFO:teuthology.orchestra.run.smithi043.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-28T22:24:08.277 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl is present 2024-03-28T22:24:08.277 INFO:teuthology.orchestra.run.smithi043.stdout:lvcreate is present 2024-03-28T22:24:08.292 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-28T22:24:08.292 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-28T22:24:08.306 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-28T22:24:08.306 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout inactive 2024-03-28T22:24:08.326 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout enabled 2024-03-28T22:24:08.343 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout active 2024-03-28T22:24:08.343 INFO:teuthology.orchestra.run.smithi043.stdout:Unit chronyd.service is enabled and running 2024-03-28T22:24:08.343 INFO:teuthology.orchestra.run.smithi043.stdout:Host looks OK 2024-03-28T22:24:08.343 INFO:teuthology.orchestra.run.smithi043.stdout:Cluster fsid: c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:24:08.344 INFO:teuthology.orchestra.run.smithi043.stdout:Acquiring lock 139700797235152 on /run/cephadm/c2276428-ed51-11ee-95d0-87774f69a715.lock 2024-03-28T22:24:08.344 INFO:teuthology.orchestra.run.smithi043.stdout:Lock 139700797235152 acquired on /run/cephadm/c2276428-ed51-11ee-95d0-87774f69a715.lock 2024-03-28T22:24:08.345 INFO:teuthology.orchestra.run.smithi043.stdout:Verifying IP 172.21.15.43 port 3300 ... 2024-03-28T22:24:08.346 INFO:teuthology.orchestra.run.smithi043.stdout:Verifying IP 172.21.15.43 port 6789 ... 2024-03-28T22:24:08.346 INFO:teuthology.orchestra.run.smithi043.stdout:Base mon IP(s) is [172.21.15.43:3300, 172.21.15.43:6789], mon addrv is [v2:172.21.15.43:3300,v1:172.21.15.43:6789] 2024-03-28T22:24:08.360 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.43 metric 100 2024-03-28T22:24:08.361 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.43 metric 100 2024-03-28T22:24:08.372 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-28T22:24:08.372 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-28T22:24:08.372 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-28T22:24:08.382 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-28T22:24:08.383 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-28T22:24:08.383 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-28T22:24:08.383 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-28T22:24:08.383 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1358/64 scope link noprefixroute 2024-03-28T22:24:08.383 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-28T22:24:08.385 INFO:teuthology.orchestra.run.smithi043.stdout:Mon IP `172.21.15.43` is in CIDR network `172.21.0.0/20` 2024-03-28T22:24:08.385 INFO:teuthology.orchestra.run.smithi043.stdout:Mon IP `172.21.15.43` is in CIDR network `172.21.0.0/20` 2024-03-28T22:24:08.386 INFO:teuthology.orchestra.run.smithi043.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-28T22:24:08.388 INFO:teuthology.orchestra.run.smithi043.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-28T22:24:08.391 INFO:teuthology.orchestra.run.smithi043.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be... 2024-03-28T22:24:09.050 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stdout bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 2024-03-28T22:24:09.050 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be... 2024-03-28T22:24:09.050 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stderr Getting image source signatures 2024-03-28T22:24:09.050 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stderr Copying blob sha256:f2898dd983f857f34092ebd65fc601696375e5e42bb673c7cb1e443cfa2e5da5 2024-03-28T22:24:09.051 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stderr Copying blob sha256:11da15500eca1f826c328eb3265caa30b0a3730510e2a32023084d6fe9cee20c 2024-03-28T22:24:09.051 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stderr Copying config sha256:bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 2024-03-28T22:24:09.051 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-28T22:24:10.762 INFO:teuthology.orchestra.run.smithi043.stdout:ceph: stdout ceph version 19.0.0-1899-g323d23dc (323d23dc228bbcb7f16f188fdc9c0dcb801844be) squid (dev) 2024-03-28T22:24:10.762 INFO:teuthology.orchestra.run.smithi043.stdout:Ceph version: ceph version 19.0.0-1899-g323d23dc (323d23dc228bbcb7f16f188fdc9c0dcb801844be) squid (dev) 2024-03-28T22:24:10.762 INFO:teuthology.orchestra.run.smithi043.stdout:Extracting ceph user uid/gid from container image... 2024-03-28T22:24:12.125 INFO:teuthology.orchestra.run.smithi043.stdout:stat: stdout 167 167 2024-03-28T22:24:12.125 INFO:teuthology.orchestra.run.smithi043.stdout:Creating initial keys... 2024-03-28T22:24:13.200 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph-authtool: stdout AQAM7gVmihGdIRAAhviZJNDLBELeVO9V7n8Lug== 2024-03-28T22:24:14.300 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph-authtool: stdout AQAN7gVmA8T2JxAAbGqiJgu0rG8etiI8jMIcJw== 2024-03-28T22:24:15.445 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph-authtool: stdout AQAO7gVmShW0LRAAAwRAwvnycdNhY8scVui1GQ== 2024-03-28T22:24:15.447 INFO:teuthology.orchestra.run.smithi043.stdout:Creating initial monmap... 2024-03-28T22:24:16.568 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-28T22:24:16.568 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-28T22:24:16.568 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:24:16.568 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-28T22:24:16.568 INFO:teuthology.orchestra.run.smithi043.stdout:monmaptool for a [v2:172.21.15.43:3300,v1:172.21.15.43:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-28T22:24:16.568 INFO:teuthology.orchestra.run.smithi043.stdout:setting min_mon_release = quincy 2024-03-28T22:24:16.568 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/monmaptool: set fsid to c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:24:16.569 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-28T22:24:16.569 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:24:16.569 INFO:teuthology.orchestra.run.smithi043.stdout:Creating mon... 2024-03-28T22:24:17.597 INFO:teuthology.orchestra.run.smithi043.stdout:create mon.a on 2024-03-28T22:24:17.986 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-28T22:24:18.231 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-28T22:24:18.540 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-c2276428-ed51-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-c2276428-ed51-11ee-95d0-87774f69a715.target. 2024-03-28T22:24:18.540 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-c2276428-ed51-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-c2276428-ed51-11ee-95d0-87774f69a715.target. 2024-03-28T22:24:18.930 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c2276428-ed51-11ee-95d0-87774f69a715@mon.a 2024-03-28T22:24:18.930 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c2276428-ed51-11ee-95d0-87774f69a715@mon.a.service: Unit ceph-c2276428-ed51-11ee-95d0-87774f69a715@mon.a.service not loaded. 2024-03-28T22:24:19.217 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c2276428-ed51-11ee-95d0-87774f69a715.target.wants/ceph-c2276428-ed51-11ee-95d0-87774f69a715@mon.a.service → /etc/systemd/system/ceph-c2276428-ed51-11ee-95d0-87774f69a715@.service. 2024-03-28T22:24:19.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:19 smithi043 podman[31425]: 2024-03-28 22:24:19.869109145 +0000 UTC m=+0.374071596 container start 457d0611fa180b50e66d8c8e12a4b41940c4b70e54d609b726d9b1545c634bd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-323d23d, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-28T22:24:20.127 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T22:24:20.127 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout disabled 2024-03-28T22:24:20.152 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T22:24:20.152 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout inactive 2024-03-28T22:24:20.153 INFO:teuthology.orchestra.run.smithi043.stdout:firewalld.service is not enabled 2024-03-28T22:24:20.153 INFO:teuthology.orchestra.run.smithi043.stdout:Not possible to enable service . firewalld.service is not available 2024-03-28T22:24:20.155 INFO:teuthology.orchestra.run.smithi043.stdout:Waiting for mon to start... 2024-03-28T22:24:20.155 INFO:teuthology.orchestra.run.smithi043.stdout:Waiting for mon... 2024-03-28T22:24:20.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:19 smithi043 ceph-mon[31443]: mkfs c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:24:20.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:19 smithi043 ceph-mon[31443]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-28T22:24:20.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:20 smithi043 bash[31425]: 457d0611fa180b50e66d8c8e12a4b41940c4b70e54d609b726d9b1545c634bd3 2024-03-28T22:24:20.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:20 smithi043 systemd[1]: Started Ceph mon.a for c2276428-ed51-11ee-95d0-87774f69a715. 2024-03-28T22:24:21.232 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:20 smithi043 ceph-mon[31443]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-28T22:24:21.232 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:20 smithi043 ceph-mon[31443]: monmap epoch 1 2024-03-28T22:24:21.232 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:20 smithi043 ceph-mon[31443]: fsid c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:24:21.233 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:20 smithi043 ceph-mon[31443]: last_changed 2024-03-28T22:24:15.893393+0000 2024-03-28T22:24:21.233 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:20 smithi043 ceph-mon[31443]: created 2024-03-28T22:24:15.893393+0000 2024-03-28T22:24:21.233 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:20 smithi043 ceph-mon[31443]: min_mon_release 19 (squid) 2024-03-28T22:24:21.233 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:20 smithi043 ceph-mon[31443]: election_strategy: 1 2024-03-28T22:24:21.233 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:20 smithi043 ceph-mon[31443]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-03-28T22:24:21.233 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:20 smithi043 ceph-mon[31443]: fsmap 2024-03-28T22:24:21.233 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:20 smithi043 ceph-mon[31443]: osdmap e1: 0 total, 0 up, 0 in 2024-03-28T22:24:21.233 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:20 smithi043 ceph-mon[31443]: mgrmap e1: no daemons active 2024-03-28T22:24:21.233 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:20 smithi043 ceph-mon[31443]: from='client.? 172.21.15.43:0/439603923' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-28T22:24:21.526 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout cluster: 2024-03-28T22:24:21.527 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout id: c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:24:21.527 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-28T22:24:21.527 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-03-28T22:24:21.527 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout services: 2024-03-28T22:24:21.527 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.907245s) 2024-03-28T22:24:21.529 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-28T22:24:21.529 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-28T22:24:21.529 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-03-28T22:24:21.529 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout data: 2024-03-28T22:24:21.529 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-28T22:24:21.529 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-28T22:24:21.529 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-28T22:24:21.529 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout pgs: 2024-03-28T22:24:21.529 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-03-28T22:24:21.532 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.762+0000 7f74e1400700 1 Processor -- start 2024-03-28T22:24:21.533 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.762+0000 7f74e1400700 1 -- start start 2024-03-28T22:24:21.533 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.762+0000 7f74e1400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74dc0822c0 0x7f74dc110ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:21.533 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.762+0000 7f74e1400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f74dc1112b0 con 0x7f74dc0822c0 2024-03-28T22:24:21.533 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.763+0000 7f74dbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74dc0822c0 0x7f74dc110ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:21.533 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.764+0000 7f74dbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74dc0822c0 0x7f74dc110ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36808/0 (socket says 172.21.15.43:36808) 2024-03-28T22:24:21.533 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.764+0000 7f74dbe00700 1 -- 172.21.15.43:0/998178302 learned_addr learned my addr 172.21.15.43:0/998178302 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:24:21.533 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.764+0000 7f74dbe00700 1 -- 172.21.15.43:0/998178302 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74dc111430 con 0x7f74dc0822c0 2024-03-28T22:24:21.533 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.765+0000 7f74dbe00700 1 --2- 172.21.15.43:0/998178302 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74dc0822c0 0x7f74dc110ce0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f74cc0098a0 tx=0x7f74cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cd589cedb408b21 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:21.533 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.765+0000 7f74db400700 1 -- 172.21.15.43:0/998178302 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74cc004160 con 0x7f74dc0822c0 2024-03-28T22:24:21.533 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.765+0000 7f74db400700 1 -- 172.21.15.43:0/998178302 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74cc004300 con 0x7f74dc0822c0 2024-03-28T22:24:21.533 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.765+0000 7f74db400700 1 -- 172.21.15.43:0/998178302 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74cc0036a0 con 0x7f74dc0822c0 2024-03-28T22:24:21.533 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.766+0000 7f74e1400700 1 -- 172.21.15.43:0/998178302 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74dc0822c0 msgr2=0x7f74dc110ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:21.533 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.766+0000 7f74e1400700 1 --2- 172.21.15.43:0/998178302 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74dc0822c0 0x7f74dc110ce0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f74cc0098a0 tx=0x7f74cc009bb0 comp rx=0 tx=0).stop 2024-03-28T22:24:21.534 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.766+0000 7f74e1400700 1 -- 172.21.15.43:0/998178302 shutdown_connections 2024-03-28T22:24:21.534 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.766+0000 7f74e1400700 1 --2- 172.21.15.43:0/998178302 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74dc0822c0 0x7f74dc110ce0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:21.534 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.766+0000 7f74e1400700 1 -- 172.21.15.43:0/998178302 >> 172.21.15.43:0/998178302 conn(0x7f74dc07d120 msgr2=0x7f74dc07f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:24:21.534 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.766+0000 7f74e1400700 1 -- 172.21.15.43:0/998178302 shutdown_connections 2024-03-28T22:24:21.534 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.766+0000 7f74e1400700 1 -- 172.21.15.43:0/998178302 wait complete. 2024-03-28T22:24:21.534 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.767+0000 7f74e1400700 1 Processor -- start 2024-03-28T22:24:21.534 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.767+0000 7f74e1400700 1 -- start start 2024-03-28T22:24:21.534 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.768+0000 7f74e1400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74dc0822c0 0x7f74dc1a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:21.534 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.768+0000 7f74e1400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f74dc111960 con 0x7f74dc0822c0 2024-03-28T22:24:21.534 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.768+0000 7f74dbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74dc0822c0 0x7f74dc1a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:21.534 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.768+0000 7f74dbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74dc0822c0 0x7f74dc1a17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36822/0 (socket says 172.21.15.43:36822) 2024-03-28T22:24:21.534 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.768+0000 7f74dbe00700 1 -- 172.21.15.43:0/439603923 learned_addr learned my addr 172.21.15.43:0/439603923 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:24:21.535 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.768+0000 7f74dbe00700 1 -- 172.21.15.43:0/439603923 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74dc1a1cf0 con 0x7f74dc0822c0 2024-03-28T22:24:21.535 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.769+0000 7f74dbe00700 1 --2- 172.21.15.43:0/439603923 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74dc0822c0 0x7f74dc1a17b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f74cc000c00 tx=0x7f74cc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:21.537 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.769+0000 7f74d9600700 1 -- 172.21.15.43:0/439603923 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74cc003db0 con 0x7f74dc0822c0 2024-03-28T22:24:21.537 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.769+0000 7f74d9600700 1 -- 172.21.15.43:0/439603923 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74cc011630 con 0x7f74dc0822c0 2024-03-28T22:24:21.537 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.769+0000 7f74d9600700 1 -- 172.21.15.43:0/439603923 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74cc011950 con 0x7f74dc0822c0 2024-03-28T22:24:21.537 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.769+0000 7f74e1400700 1 -- 172.21.15.43:0/439603923 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74dc1a1f80 con 0x7f74dc0822c0 2024-03-28T22:24:21.537 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.769+0000 7f74e1400700 1 -- 172.21.15.43:0/439603923 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74dc1a4c70 con 0x7f74dc0822c0 2024-03-28T22:24:21.537 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.769+0000 7f74d9600700 1 -- 172.21.15.43:0/439603923 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f74cc011af0 con 0x7f74dc0822c0 2024-03-28T22:24:21.537 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.770+0000 7f74d9600700 1 -- 172.21.15.43:0/439603923 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f74cc01c450 con 0x7f74dc0822c0 2024-03-28T22:24:21.537 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.770+0000 7f74e1400700 1 -- 172.21.15.43:0/439603923 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74dc063870 con 0x7f74dc0822c0 2024-03-28T22:24:21.537 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.774+0000 7f74d9600700 1 -- 172.21.15.43:0/439603923 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f74cc032020 con 0x7f74dc0822c0 2024-03-28T22:24:21.537 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.855+0000 7f74e1400700 1 -- 172.21.15.43:0/439603923 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f74dc02dcb0 con 0x7f74dc0822c0 2024-03-28T22:24:21.537 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.855+0000 7f74d9600700 1 -- 172.21.15.43:0/439603923 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f74cc004260 con 0x7f74dc0822c0 2024-03-28T22:24:21.538 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.856+0000 7f74e1400700 1 -- 172.21.15.43:0/439603923 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74dc0822c0 msgr2=0x7f74dc1a17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:21.538 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.856+0000 7f74e1400700 1 --2- 172.21.15.43:0/439603923 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74dc0822c0 0x7f74dc1a17b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f74cc000c00 tx=0x7f74cc004750 comp rx=0 tx=0).stop 2024-03-28T22:24:21.538 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.857+0000 7f74e1400700 1 -- 172.21.15.43:0/439603923 shutdown_connections 2024-03-28T22:24:21.538 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.857+0000 7f74e1400700 1 --2- 172.21.15.43:0/439603923 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74dc0822c0 0x7f74dc1a17b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:21.538 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.857+0000 7f74e1400700 1 -- 172.21.15.43:0/439603923 >> 172.21.15.43:0/439603923 conn(0x7f74dc07d120 msgr2=0x7f74dc07f400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:24:21.538 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.857+0000 7f74e1400700 1 -- 172.21.15.43:0/439603923 shutdown_connections 2024-03-28T22:24:21.538 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:20.857+0000 7f74e1400700 1 -- 172.21.15.43:0/439603923 wait complete. 2024-03-28T22:24:21.538 INFO:teuthology.orchestra.run.smithi043.stdout:mon is available 2024-03-28T22:24:21.538 INFO:teuthology.orchestra.run.smithi043.stdout:Assimilating anything we can from ceph.conf... 2024-03-28T22:24:22.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:22 smithi043 ceph-mon[31443]: from='client.? 172.21.15.43:0/17569705' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-28T22:24:22.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:22 smithi043 ceph-mon[31443]: from='client.? 172.21.15.43:0/17569705' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-28T22:24:22.912 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-03-28T22:24:22.912 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout [global] 2024-03-28T22:24:22.912 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout fsid = c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:24:22.912 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.43:3300,v1:172.21.15.43:6789] 2024-03-28T22:24:22.913 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-28T22:24:22.913 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-28T22:24:22.913 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-28T22:24:22.913 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-28T22:24:22.914 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-03-28T22:24:22.914 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-28T22:24:22.914 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-28T22:24:22.914 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-03-28T22:24:22.915 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout [osd] 2024-03-28T22:24:22.915 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-28T22:24:22.915 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-28T22:24:22.915 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-28T22:24:22.915 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-28T22:24:22.915 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-28T22:24:22.916 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.107+0000 7ff43f400700 1 Processor -- start 2024-03-28T22:24:22.916 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.108+0000 7ff43f400700 1 -- start start 2024-03-28T22:24:22.917 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.108+0000 7ff43f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff43807dd70 0x7ff43807c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:22.917 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.108+0000 7ff43f400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff43807c820 con 0x7ff43807dd70 2024-03-28T22:24:22.917 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.109+0000 7ff43ca00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff43807dd70 0x7ff43807c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:22.917 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.109+0000 7ff43ca00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff43807dd70 0x7ff43807c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36834/0 (socket says 172.21.15.43:36834) 2024-03-28T22:24:22.918 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.109+0000 7ff43ca00700 1 -- 172.21.15.43:0/3784430017 learned_addr learned my addr 172.21.15.43:0/3784430017 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:24:22.918 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.110+0000 7ff43ca00700 1 -- 172.21.15.43:0/3784430017 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff43807c9a0 con 0x7ff43807dd70 2024-03-28T22:24:22.918 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.110+0000 7ff43ca00700 1 --2- 172.21.15.43:0/3784430017 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff43807dd70 0x7ff43807c2e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff42c009e80 tx=0x7ff42c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef910bea790a3928 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:22.918 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.111+0000 7ff437400700 1 -- 172.21.15.43:0/3784430017 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff42c017070 con 0x7ff43807dd70 2024-03-28T22:24:22.918 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.111+0000 7ff437400700 1 -- 172.21.15.43:0/3784430017 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff42c003a20 con 0x7ff43807dd70 2024-03-28T22:24:22.918 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.111+0000 7ff437400700 1 -- 172.21.15.43:0/3784430017 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff42c003d20 con 0x7ff43807dd70 2024-03-28T22:24:22.918 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.111+0000 7ff43f400700 1 -- 172.21.15.43:0/3784430017 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff43807dd70 msgr2=0x7ff43807c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:22.921 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.111+0000 7ff43f400700 1 --2- 172.21.15.43:0/3784430017 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff43807dd70 0x7ff43807c2e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff42c009e80 tx=0x7ff42c00b330 comp rx=0 tx=0).stop 2024-03-28T22:24:22.921 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.111+0000 7ff43f400700 1 -- 172.21.15.43:0/3784430017 shutdown_connections 2024-03-28T22:24:22.922 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.111+0000 7ff43f400700 1 --2- 172.21.15.43:0/3784430017 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff43807dd70 0x7ff43807c2e0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:22.922 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.112+0000 7ff43f400700 1 -- 172.21.15.43:0/3784430017 >> 172.21.15.43:0/3784430017 conn(0x7ff438103340 msgr2=0x7ff438105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:24:22.922 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.112+0000 7ff43f400700 1 -- 172.21.15.43:0/3784430017 shutdown_connections 2024-03-28T22:24:22.922 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.112+0000 7ff43f400700 1 -- 172.21.15.43:0/3784430017 wait complete. 2024-03-28T22:24:22.922 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.113+0000 7ff43f400700 1 Processor -- start 2024-03-28T22:24:22.922 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.113+0000 7ff43f400700 1 -- start start 2024-03-28T22:24:22.922 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.113+0000 7ff43f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff43807dd70 0x7ff43819d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:22.922 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.113+0000 7ff43f400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff43807ce70 con 0x7ff43807dd70 2024-03-28T22:24:22.922 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.114+0000 7ff43ca00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff43807dd70 0x7ff43819d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:22.922 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.114+0000 7ff43ca00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff43807dd70 0x7ff43819d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36840/0 (socket says 172.21.15.43:36840) 2024-03-28T22:24:22.922 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.114+0000 7ff43ca00700 1 -- 172.21.15.43:0/17569705 learned_addr learned my addr 172.21.15.43:0/17569705 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:24:22.922 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.114+0000 7ff43ca00700 1 -- 172.21.15.43:0/17569705 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff43819db30 con 0x7ff43807dd70 2024-03-28T22:24:22.922 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.114+0000 7ff43ca00700 1 --2- 172.21.15.43:0/17569705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff43807dd70 0x7ff43819d5f0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7ff43807d3b0 tx=0x7ff42c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:22.923 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.114+0000 7ff435600700 1 -- 172.21.15.43:0/17569705 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff42c022020 con 0x7ff43807dd70 2024-03-28T22:24:22.923 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.115+0000 7ff435600700 1 -- 172.21.15.43:0/17569705 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff42c018040 con 0x7ff43807dd70 2024-03-28T22:24:22.923 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.115+0000 7ff43f400700 1 -- 172.21.15.43:0/17569705 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff43819ddc0 con 0x7ff43807dd70 2024-03-28T22:24:22.923 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.115+0000 7ff435600700 1 -- 172.21.15.43:0/17569705 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff42c017070 con 0x7ff43807dd70 2024-03-28T22:24:22.923 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.115+0000 7ff43f400700 1 -- 172.21.15.43:0/17569705 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4381a0ab0 con 0x7ff43807dd70 2024-03-28T22:24:22.923 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.115+0000 7ff435600700 1 -- 172.21.15.43:0/17569705 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff42c025020 con 0x7ff43807dd70 2024-03-28T22:24:22.923 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.115+0000 7ff435600700 1 -- 172.21.15.43:0/17569705 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff42c00f050 con 0x7ff43807dd70 2024-03-28T22:24:22.923 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.116+0000 7ff43f400700 1 -- 172.21.15.43:0/17569705 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff438063870 con 0x7ff43807dd70 2024-03-28T22:24:22.923 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.119+0000 7ff435600700 1 -- 172.21.15.43:0/17569705 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff42c02c030 con 0x7ff43807dd70 2024-03-28T22:24:22.923 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.198+0000 7ff43f400700 1 -- 172.21.15.43:0/17569705 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7ff43802dcc0 con 0x7ff43807dd70 2024-03-28T22:24:22.923 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.204+0000 7ff435600700 1 -- 172.21.15.43:0/17569705 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7ff42c01e480 con 0x7ff43807dd70 2024-03-28T22:24:22.923 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.205+0000 7ff435600700 1 -- 172.21.15.43:0/17569705 <== mon.0 v2:172.21.15.43:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff42c035070 con 0x7ff43807dd70 2024-03-28T22:24:22.924 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.206+0000 7ff43f400700 1 -- 172.21.15.43:0/17569705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff43807dd70 msgr2=0x7ff43819d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:22.924 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.206+0000 7ff43f400700 1 --2- 172.21.15.43:0/17569705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff43807dd70 0x7ff43819d5f0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7ff43807d3b0 tx=0x7ff42c004ad0 comp rx=0 tx=0).stop 2024-03-28T22:24:22.924 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.206+0000 7ff43f400700 1 -- 172.21.15.43:0/17569705 shutdown_connections 2024-03-28T22:24:22.925 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.206+0000 7ff43f400700 1 --2- 172.21.15.43:0/17569705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff43807dd70 0x7ff43819d5f0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:22.925 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.206+0000 7ff43f400700 1 -- 172.21.15.43:0/17569705 >> 172.21.15.43:0/17569705 conn(0x7ff438103340 msgr2=0x7ff4381056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:24:22.925 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.206+0000 7ff43f400700 1 -- 172.21.15.43:0/17569705 shutdown_connections 2024-03-28T22:24:22.925 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:22.206+0000 7ff43f400700 1 -- 172.21.15.43:0/17569705 wait complete. 2024-03-28T22:24:22.925 INFO:teuthology.orchestra.run.smithi043.stdout:Generating new minimal ceph.conf... 2024-03-28T22:24:23.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:23 smithi043 ceph-mon[31443]: from='client.? 172.21.15.43:0/3423477357' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:24:24.210 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.462+0000 7f3125400700 1 Processor -- start 2024-03-28T22:24:24.215 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.462+0000 7f3125400700 1 -- start start 2024-03-28T22:24:24.215 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.463+0000 7f3125400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3120107ce0 0x7f3120108100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:24.215 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.463+0000 7f3125400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3120108640 con 0x7f3120107ce0 2024-03-28T22:24:24.216 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.463+0000 7f311e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3120107ce0 0x7f3120108100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:24.216 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.464+0000 7f311e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3120107ce0 0x7f3120108100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36856/0 (socket says 172.21.15.43:36856) 2024-03-28T22:24:24.216 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.464+0000 7f311e800700 1 -- 172.21.15.43:0/751986019 learned_addr learned my addr 172.21.15.43:0/751986019 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:24:24.216 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.464+0000 7f311e800700 1 -- 172.21.15.43:0/751986019 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31201087c0 con 0x7f3120107ce0 2024-03-28T22:24:24.216 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.464+0000 7f311e800700 1 --2- 172.21.15.43:0/751986019 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3120107ce0 0x7f3120108100 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f31100098a0 tx=0x7f3110009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fa74c67c277bd5f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:24.216 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.465+0000 7f311d400700 1 -- 172.21.15.43:0/751986019 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3110004160 con 0x7f3120107ce0 2024-03-28T22:24:24.216 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.465+0000 7f311d400700 1 -- 172.21.15.43:0/751986019 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3110004300 con 0x7f3120107ce0 2024-03-28T22:24:24.216 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.465+0000 7f311d400700 1 -- 172.21.15.43:0/751986019 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3110011b50 con 0x7f3120107ce0 2024-03-28T22:24:24.216 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.465+0000 7f3125400700 1 -- 172.21.15.43:0/751986019 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3120107ce0 msgr2=0x7f3120108100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:24.216 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.465+0000 7f3125400700 1 --2- 172.21.15.43:0/751986019 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3120107ce0 0x7f3120108100 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f31100098a0 tx=0x7f3110009bb0 comp rx=0 tx=0).stop 2024-03-28T22:24:24.216 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.466+0000 7f3125400700 1 -- 172.21.15.43:0/751986019 shutdown_connections 2024-03-28T22:24:24.216 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.466+0000 7f3125400700 1 --2- 172.21.15.43:0/751986019 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3120107ce0 0x7f3120108100 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:24.216 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.466+0000 7f3125400700 1 -- 172.21.15.43:0/751986019 >> 172.21.15.43:0/751986019 conn(0x7f3120103340 msgr2=0x7f3120105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:24:24.216 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.466+0000 7f3125400700 1 -- 172.21.15.43:0/751986019 shutdown_connections 2024-03-28T22:24:24.217 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.466+0000 7f3125400700 1 -- 172.21.15.43:0/751986019 wait complete. 2024-03-28T22:24:24.217 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.467+0000 7f3125400700 1 Processor -- start 2024-03-28T22:24:24.217 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.468+0000 7f3125400700 1 -- start start 2024-03-28T22:24:24.217 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.468+0000 7f3125400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3120107ce0 0x7f312007db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:24.217 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.468+0000 7f3125400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f312010c150 con 0x7f3120107ce0 2024-03-28T22:24:24.217 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.468+0000 7f311e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3120107ce0 0x7f312007db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:24.217 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.469+0000 7f311e800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3120107ce0 0x7f312007db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36862/0 (socket says 172.21.15.43:36862) 2024-03-28T22:24:24.217 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.469+0000 7f311e800700 1 -- 172.21.15.43:0/3423477357 learned_addr learned my addr 172.21.15.43:0/3423477357 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:24:24.217 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.469+0000 7f311e800700 1 -- 172.21.15.43:0/3423477357 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f312007e0b0 con 0x7f3120107ce0 2024-03-28T22:24:24.217 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.469+0000 7f311e800700 1 --2- 172.21.15.43:0/3423477357 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3120107ce0 0x7f312007db70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3110011b60 tx=0x7f3110011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:24.217 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.469+0000 7f3117400700 1 -- 172.21.15.43:0/3423477357 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3110003b20 con 0x7f3120107ce0 2024-03-28T22:24:24.217 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.470+0000 7f3117400700 1 -- 172.21.15.43:0/3423477357 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3110003cc0 con 0x7f3120107ce0 2024-03-28T22:24:24.217 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.470+0000 7f3125400700 1 -- 172.21.15.43:0/3423477357 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f312007c2a0 con 0x7f3120107ce0 2024-03-28T22:24:24.217 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.470+0000 7f3125400700 1 -- 172.21.15.43:0/3423477357 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f312007c7e0 con 0x7f3120107ce0 2024-03-28T22:24:24.217 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.470+0000 7f3117400700 1 -- 172.21.15.43:0/3423477357 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3110024840 con 0x7f3120107ce0 2024-03-28T22:24:24.218 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.471+0000 7f3117400700 1 -- 172.21.15.43:0/3423477357 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3110022020 con 0x7f3120107ce0 2024-03-28T22:24:24.218 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.471+0000 7f3117400700 1 -- 172.21.15.43:0/3423477357 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3110011e00 con 0x7f3120107ce0 2024-03-28T22:24:24.218 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.471+0000 7f3125400700 1 -- 172.21.15.43:0/3423477357 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3100005360 con 0x7f3120107ce0 2024-03-28T22:24:24.218 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.474+0000 7f3117400700 1 -- 172.21.15.43:0/3423477357 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f3110043dd0 con 0x7f3120107ce0 2024-03-28T22:24:24.218 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.550+0000 7f3125400700 1 -- 172.21.15.43:0/3423477357 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f3100005920 con 0x7f3120107ce0 2024-03-28T22:24:24.218 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.551+0000 7f3117400700 1 -- 172.21.15.43:0/3423477357 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f3110032020 con 0x7f3120107ce0 2024-03-28T22:24:24.218 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.552+0000 7f3125400700 1 -- 172.21.15.43:0/3423477357 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3120107ce0 msgr2=0x7f312007db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:24.218 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.552+0000 7f3125400700 1 --2- 172.21.15.43:0/3423477357 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3120107ce0 0x7f312007db70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3110011b60 tx=0x7f3110011c40 comp rx=0 tx=0).stop 2024-03-28T22:24:24.218 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.552+0000 7f3125400700 1 -- 172.21.15.43:0/3423477357 shutdown_connections 2024-03-28T22:24:24.218 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.552+0000 7f3125400700 1 --2- 172.21.15.43:0/3423477357 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3120107ce0 0x7f312007db70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:24.218 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.552+0000 7f3125400700 1 -- 172.21.15.43:0/3423477357 >> 172.21.15.43:0/3423477357 conn(0x7f3120103340 msgr2=0x7f3120103ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:24:24.218 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.552+0000 7f3125400700 1 -- 172.21.15.43:0/3423477357 shutdown_connections 2024-03-28T22:24:24.218 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:23.552+0000 7f3125400700 1 -- 172.21.15.43:0/3423477357 wait complete. 2024-03-28T22:24:24.219 INFO:teuthology.orchestra.run.smithi043.stdout:Restarting the monitor... 2024-03-28T22:24:24.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:24 smithi043 systemd[1]: Stopping Ceph mon.a for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:24:25.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:24 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mon-a[31439]: 2024-03-28T22:24:24.670+0000 7ff4baa00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:24:25.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:24 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mon-a[31439]: 2024-03-28T22:24:24.670+0000 7ff4baa00700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-28T22:24:25.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:24 smithi043 podman[31698]: 2024-03-28 22:24:24.845741167 +0000 UTC m=+0.292316453 container died 457d0611fa180b50e66d8c8e12a4b41940c4b70e54d609b726d9b1545c634bd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mon-a, RELEASE=squid-323d23d, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-28T22:24:25.513 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:25 smithi043 podman[31698]: 2024-03-28 22:24:25.262717504 +0000 UTC m=+0.709292783 container cleanup 457d0611fa180b50e66d8c8e12a4b41940c4b70e54d609b726d9b1545c634bd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mon-a, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-323d23d, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, ceph=True) 2024-03-28T22:24:25.513 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:25 smithi043 bash[31698]: ceph-c2276428-ed51-11ee-95d0-87774f69a715-mon-a 2024-03-28T22:24:25.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:25 smithi043 podman[31712]: 2024-03-28 22:24:25.51325122 +0000 UTC m=+0.650758987 container remove 457d0611fa180b50e66d8c8e12a4b41940c4b70e54d609b726d9b1545c634bd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-28T22:24:26.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:25 smithi043 systemd[1]: ceph-c2276428-ed51-11ee-95d0-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-28T22:24:26.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:25 smithi043 systemd[1]: Stopped Ceph mon.a for c2276428-ed51-11ee-95d0-87774f69a715. 2024-03-28T22:24:26.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:25 smithi043 systemd[1]: ceph-c2276428-ed51-11ee-95d0-87774f69a715@mon.a.service: Consumed 1.922s CPU time. 2024-03-28T22:24:26.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:25 smithi043 systemd[1]: Starting Ceph mon.a for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:24:26.661 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 podman[31805]: 2024-03-28 22:24:26.273874114 +0000 UTC m=+0.040883110 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be 2024-03-28T22:24:26.661 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 podman[31805]: 2024-03-28 22:24:26.372433643 +0000 UTC m=+0.139442659 container create f2315b91a227e5922ae97fad845c259b4c5293dd04bf8a5a1b02a4cc7d4dba39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mon-a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-323d23d, ceph=True, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-28T22:24:26.661 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 podman[31805]: 2024-03-28 22:24:26.600753977 +0000 UTC m=+0.367763000 container init f2315b91a227e5922ae97fad845c259b4c5293dd04bf8a5a1b02a4cc7d4dba39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mon-a, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-323d23d, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5) 2024-03-28T22:24:26.661 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 podman[31805]: 2024-03-28 22:24:26.611369697 +0000 UTC m=+0.378378719 container start f2315b91a227e5922ae97fad845c259b4c5293dd04bf8a5a1b02a4cc7d4dba39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mon-a, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d, org.label-schema.license=GPLv2) 2024-03-28T22:24:26.680 INFO:teuthology.orchestra.run.smithi043.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-28T22:24:26.971 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: set uid:gid to 167:167 (ceph:ceph) 2024-03-28T22:24:26.971 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: ceph version 19.0.0-1899-g323d23dc (323d23dc228bbcb7f16f188fdc9c0dcb801844be) squid (dev), process ceph-mon, pid 2 2024-03-28T22:24:26.971 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: pidfile_write: ignore empty --pid-file 2024-03-28T22:24:26.971 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 bash[31805]: f2315b91a227e5922ae97fad845c259b4c5293dd04bf8a5a1b02a4cc7d4dba39 2024-03-28T22:24:26.971 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: load: jerasure load: lrc 2024-03-28T22:24:26.972 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: RocksDB version: 7.9.2 2024-03-28T22:24:26.972 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Git sha 0 2024-03-28T22:24:26.972 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Compile date 2024-03-27 16:17:23 2024-03-28T22:24:26.972 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: DB SUMMARY 2024-03-28T22:24:26.972 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: DB Session ID: AD28QSYWIPPSGMOMJ084 2024-03-28T22:24:26.972 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: CURRENT file: CURRENT 2024-03-28T22:24:26.972 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: IDENTITY file: IDENTITY 2024-03-28T22:24:26.972 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-28T22:24:26.972 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-28T22:24:26.972 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105310 ; 2024-03-28T22:24:26.972 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.error_if_exists: 0 2024-03-28T22:24:26.973 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.create_if_missing: 0 2024-03-28T22:24:26.973 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.paranoid_checks: 1 2024-03-28T22:24:26.973 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-28T22:24:26.973 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-28T22:24:26.973 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-28T22:24:26.973 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.env: 0x55fb5fe85c60 2024-03-28T22:24:26.973 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.fs: PosixFileSystem 2024-03-28T22:24:26.973 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.info_log: 0x55fb6174bb20 2024-03-28T22:24:26.973 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.max_file_opening_threads: 16 2024-03-28T22:24:26.973 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.statistics: (nil) 2024-03-28T22:24:26.973 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.use_fsync: 0 2024-03-28T22:24:26.973 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.max_log_file_size: 0 2024-03-28T22:24:26.974 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-28T22:24:26.974 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-28T22:24:26.974 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.keep_log_file_num: 1000 2024-03-28T22:24:26.974 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.recycle_log_file_num: 0 2024-03-28T22:24:26.974 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.allow_fallocate: 1 2024-03-28T22:24:26.974 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.allow_mmap_reads: 0 2024-03-28T22:24:26.974 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.allow_mmap_writes: 0 2024-03-28T22:24:26.974 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.use_direct_reads: 0 2024-03-28T22:24:26.974 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-28T22:24:26.974 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.create_missing_column_families: 0 2024-03-28T22:24:26.974 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.db_log_dir: 2024-03-28T22:24:26.974 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.wal_dir: 2024-03-28T22:24:26.975 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-28T22:24:26.975 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-28T22:24:26.975 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-28T22:24:26.975 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-28T22:24:26.975 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-28T22:24:26.975 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-28T22:24:26.975 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.advise_random_on_open: 1 2024-03-28T22:24:26.975 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.db_write_buffer_size: 0 2024-03-28T22:24:26.976 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.write_buffer_manager: 0x55fb6171e1e0 2024-03-28T22:24:26.976 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-28T22:24:26.976 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-28T22:24:26.976 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-28T22:24:26.976 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.rate_limiter: (nil) 2024-03-28T22:24:26.976 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-28T22:24:26.976 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.wal_recovery_mode: 2 2024-03-28T22:24:26.976 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.enable_thread_tracking: 0 2024-03-28T22:24:26.976 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.enable_pipelined_write: 0 2024-03-28T22:24:26.976 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.unordered_write: 0 2024-03-28T22:24:26.976 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-28T22:24:26.976 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-28T22:24:26.976 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-28T22:24:26.977 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-28T22:24:26.977 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.row_cache: None 2024-03-28T22:24:26.977 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.wal_filter: None 2024-03-28T22:24:26.977 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-28T22:24:26.977 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.allow_ingest_behind: 0 2024-03-28T22:24:26.977 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.two_write_queues: 0 2024-03-28T22:24:26.977 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.manual_wal_flush: 0 2024-03-28T22:24:26.977 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.wal_compression: 0 2024-03-28T22:24:26.977 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.atomic_flush: 0 2024-03-28T22:24:26.977 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-28T22:24:26.977 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-28T22:24:26.977 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-28T22:24:26.977 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.log_readahead_size: 0 2024-03-28T22:24:26.977 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-28T22:24:26.978 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.best_efforts_recovery: 0 2024-03-28T22:24:26.978 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-28T22:24:26.978 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-28T22:24:26.978 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.allow_data_in_errors: 0 2024-03-28T22:24:26.978 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.db_host_id: __hostname__ 2024-03-28T22:24:26.978 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-28T22:24:26.978 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.max_background_jobs: 2 2024-03-28T22:24:26.978 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.max_background_compactions: -1 2024-03-28T22:24:26.978 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.max_subcompactions: 1 2024-03-28T22:24:26.978 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-28T22:24:26.978 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-28T22:24:26.978 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-28T22:24:26.978 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.max_total_wal_size: 0 2024-03-28T22:24:26.979 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-28T22:24:26.979 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-28T22:24:26.979 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-28T22:24:26.979 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-28T22:24:26.979 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.max_open_files: -1 2024-03-28T22:24:26.979 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.bytes_per_sync: 0 2024-03-28T22:24:26.979 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-28T22:24:26.979 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-28T22:24:26.979 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.compaction_readahead_size: 0 2024-03-28T22:24:26.979 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.max_background_flushes: -1 2024-03-28T22:24:26.979 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Compression algorithms supported: 2024-03-28T22:24:26.979 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-28T22:24:26.979 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: kZSTD supported: 0 2024-03-28T22:24:26.979 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: kXpressCompression supported: 0 2024-03-28T22:24:26.979 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: kLZ4HCCompression supported: 1 2024-03-28T22:24:26.980 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: kZlibCompression supported: 1 2024-03-28T22:24:26.980 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: kSnappyCompression supported: 1 2024-03-28T22:24:26.980 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: kLZ4Compression supported: 1 2024-03-28T22:24:26.980 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: kBZip2Compression supported: 0 2024-03-28T22:24:26.980 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-28T22:24:26.980 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-28T22:24:26.980 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-28T22:24:26.980 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-28T22:24:26.980 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-28T22:24:26.980 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.merge_operator: 2024-03-28T22:24:26.980 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.compaction_filter: None 2024-03-28T22:24:26.980 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.compaction_filter_factory: None 2024-03-28T22:24:26.980 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.sst_partitioner_factory: None 2024-03-28T22:24:26.980 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-28T22:24:26.980 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-28T22:24:26.981 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fb6174bc40) 2024-03-28T22:24:26.981 INFO:journalctl@ceph.mon.a.smithi043.stdout: cache_index_and_filter_blocks: 1 2024-03-28T22:24:26.982 INFO:journalctl@ceph.mon.a.smithi043.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-28T22:24:26.982 INFO:journalctl@ceph.mon.a.smithi043.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-28T22:24:26.982 INFO:journalctl@ceph.mon.a.smithi043.stdout: pin_top_level_index_and_filter: 1 2024-03-28T22:24:26.982 INFO:journalctl@ceph.mon.a.smithi043.stdout: index_type: 0 2024-03-28T22:24:26.982 INFO:journalctl@ceph.mon.a.smithi043.stdout: data_block_index_type: 0 2024-03-28T22:24:26.982 INFO:journalctl@ceph.mon.a.smithi043.stdout: index_shortening: 1 2024-03-28T22:24:26.982 INFO:journalctl@ceph.mon.a.smithi043.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-28T22:24:26.982 INFO:journalctl@ceph.mon.a.smithi043.stdout: checksum: 4 2024-03-28T22:24:26.982 INFO:journalctl@ceph.mon.a.smithi043.stdout: no_block_cache: 0 2024-03-28T22:24:26.983 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_cache: 0x55fb6174f090 2024-03-28T22:24:26.983 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_cache_name: BinnedLRUCache 2024-03-28T22:24:26.983 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_cache_options: 2024-03-28T22:24:26.983 INFO:journalctl@ceph.mon.a.smithi043.stdout: capacity : 536870912 2024-03-28T22:24:26.983 INFO:journalctl@ceph.mon.a.smithi043.stdout: num_shard_bits : 4 2024-03-28T22:24:26.983 INFO:journalctl@ceph.mon.a.smithi043.stdout: strict_capacity_limit : 0 2024-03-28T22:24:26.983 INFO:journalctl@ceph.mon.a.smithi043.stdout: high_pri_pool_ratio: 0.000 2024-03-28T22:24:26.983 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_cache_compressed: (nil) 2024-03-28T22:24:26.983 INFO:journalctl@ceph.mon.a.smithi043.stdout: persistent_cache: (nil) 2024-03-28T22:24:26.983 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_size: 4096 2024-03-28T22:24:26.983 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_size_deviation: 10 2024-03-28T22:24:26.983 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_restart_interval: 16 2024-03-28T22:24:26.983 INFO:journalctl@ceph.mon.a.smithi043.stdout: index_block_restart_interval: 1 2024-03-28T22:24:26.983 INFO:journalctl@ceph.mon.a.smithi043.stdout: metadata_block_size: 4096 2024-03-28T22:24:26.984 INFO:journalctl@ceph.mon.a.smithi043.stdout: partition_filters: 0 2024-03-28T22:24:26.984 INFO:journalctl@ceph.mon.a.smithi043.stdout: use_delta_encoding: 1 2024-03-28T22:24:26.984 INFO:journalctl@ceph.mon.a.smithi043.stdout: filter_policy: bloomfilter 2024-03-28T22:24:26.984 INFO:journalctl@ceph.mon.a.smithi043.stdout: whole_key_filtering: 1 2024-03-28T22:24:26.984 INFO:journalctl@ceph.mon.a.smithi043.stdout: verify_compression: 0 2024-03-28T22:24:26.984 INFO:journalctl@ceph.mon.a.smithi043.stdout: read_amp_bytes_per_bit: 0 2024-03-28T22:24:26.984 INFO:journalctl@ceph.mon.a.smithi043.stdout: format_version: 5 2024-03-28T22:24:26.984 INFO:journalctl@ceph.mon.a.smithi043.stdout: enable_index_compression: 1 2024-03-28T22:24:26.984 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_align: 0 2024-03-28T22:24:26.984 INFO:journalctl@ceph.mon.a.smithi043.stdout: max_auto_readahead_size: 262144 2024-03-28T22:24:26.984 INFO:journalctl@ceph.mon.a.smithi043.stdout: prepopulate_block_cache: 0 2024-03-28T22:24:26.984 INFO:journalctl@ceph.mon.a.smithi043.stdout: initial_auto_readahead_size: 8192 2024-03-28T22:24:26.984 INFO:journalctl@ceph.mon.a.smithi043.stdout: num_file_reads_for_auto_readahead: 2 2024-03-28T22:24:26.984 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.write_buffer_size: 33554432 2024-03-28T22:24:26.985 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.max_write_buffer_number: 2 2024-03-28T22:24:26.985 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.compression: NoCompression 2024-03-28T22:24:26.985 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.bottommost_compression: Disabled 2024-03-28T22:24:26.985 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.prefix_extractor: nullptr 2024-03-28T22:24:26.985 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-28T22:24:26.985 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.num_levels: 7 2024-03-28T22:24:26.985 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-28T22:24:26.985 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-28T22:24:26.985 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-28T22:24:26.985 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-28T22:24:26.985 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-28T22:24:26.985 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-28T22:24:26.985 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-28T22:24:26.985 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-28T22:24:26.986 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-28T22:24:26.986 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-28T22:24:26.986 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-28T22:24:26.986 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-28T22:24:26.986 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-28T22:24:26.986 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.compression_opts.level: 32767 2024-03-28T22:24:26.986 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.compression_opts.strategy: 0 2024-03-28T22:24:26.986 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-28T22:24:26.986 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-28T22:24:26.986 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-28T22:24:26.986 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-28T22:24:26.986 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.compression_opts.enabled: false 2024-03-28T22:24:26.986 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-28T22:24:26.986 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-28T22:24:26.987 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-28T22:24:26.987 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-28T22:24:26.987 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.target_file_size_base: 67108864 2024-03-28T22:24:26.987 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-28T22:24:26.987 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-28T22:24:26.987 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-28T22:24:26.987 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-28T22:24:26.987 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-28T22:24:26.987 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-28T22:24:26.987 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-28T22:24:26.987 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-28T22:24:26.987 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-28T22:24:26.987 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-28T22:24:26.987 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-28T22:24:26.987 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-28T22:24:26.988 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-28T22:24:26.988 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-28T22:24:26.988 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.arena_block_size: 1048576 2024-03-28T22:24:26.988 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-28T22:24:26.988 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-28T22:24:26.988 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.disable_auto_compactions: 0 2024-03-28T22:24:26.988 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-28T22:24:26.988 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-28T22:24:26.988 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-28T22:24:26.988 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-28T22:24:26.988 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-28T22:24:26.988 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-28T22:24:26.988 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-28T22:24:26.988 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-28T22:24:26.988 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-28T22:24:26.989 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-28T22:24:26.989 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.table_properties_collectors: 2024-03-28T22:24:26.989 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.inplace_update_support: 0 2024-03-28T22:24:26.989 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-28T22:24:26.989 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-28T22:24:26.989 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-28T22:24:26.989 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-28T22:24:26.989 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.bloom_locality: 0 2024-03-28T22:24:26.989 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.max_successive_merges: 0 2024-03-28T22:24:26.989 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-28T22:24:26.989 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.paranoid_file_checks: 0 2024-03-28T22:24:26.989 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.force_consistency_checks: 1 2024-03-28T22:24:26.989 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.report_bg_io_stats: 0 2024-03-28T22:24:26.989 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.ttl: 2592000 2024-03-28T22:24:26.990 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-28T22:24:26.990 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-28T22:24:26.990 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-28T22:24:26.990 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.enable_blob_files: false 2024-03-28T22:24:26.990 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.min_blob_size: 0 2024-03-28T22:24:26.990 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.blob_file_size: 268435456 2024-03-28T22:24:26.990 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-28T22:24:26.990 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-28T22:24:26.990 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-28T22:24:26.990 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-28T22:24:26.990 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-28T22:24:26.990 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.blob_file_starting_level: 0 2024-03-28T22:24:26.990 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-28T22:24:26.991 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-28T22:24:26.991 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-28T22:24:26.991 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 9d5f34ce-9bb3-4110-bcb8-0bebca2b4240 2024-03-28T22:24:26.991 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711664666674973, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-28T22:24:26.991 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-28T22:24:26.991 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 systemd[1]: Started Ceph mon.a for c2276428-ed51-11ee-95d0-87774f69a715. 2024-03-28T22:24:26.991 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711664666676763, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100343, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 328, "table_properties": {"data_size": 98263, "index_size": 298, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14308, "raw_average_key_size": 49, "raw_value_size": 90517, "raw_average_value_size": 315, "num_data_blocks": 12, "num_entries": 287, "num_filter_entries": 287, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711664666, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9d5f34ce-9bb3-4110-bcb8-0bebca2b4240", "db_session_id": "AD28QSYWIPPSGMOMJ084", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-28T22:24:26.991 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711664666677059, "job": 1, "event": "recovery_finished"} 2024-03-28T22:24:26.991 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-28T22:24:26.991 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-28T22:24:26.991 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55fb61840000 2024-03-28T22:24:26.991 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: DB pointer 0x55fb6182a000 2024-03-28T22:24:26.991 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-28T22:24:26.991 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-28T22:24:26.992 INFO:journalctl@ceph.mon.a.smithi043.stdout: ** DB Stats ** 2024-03-28T22:24:26.992 INFO:journalctl@ceph.mon.a.smithi043.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T22:24:26.992 INFO:journalctl@ceph.mon.a.smithi043.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-28T22:24:26.992 INFO:journalctl@ceph.mon.a.smithi043.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T22:24:26.992 INFO:journalctl@ceph.mon.a.smithi043.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T22:24:26.992 INFO:journalctl@ceph.mon.a.smithi043.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-28T22:24:26.992 INFO:journalctl@ceph.mon.a.smithi043.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T22:24:26.992 INFO:journalctl@ceph.mon.a.smithi043.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T22:24:26.992 INFO:journalctl@ceph.mon.a.smithi043.stdout: 2024-03-28T22:24:26.992 INFO:journalctl@ceph.mon.a.smithi043.stdout: ** Compaction Stats [default] ** 2024-03-28T22:24:26.992 INFO:journalctl@ceph.mon.a.smithi043.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T22:24:26.992 INFO:journalctl@ceph.mon.a.smithi043.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-28T22:24:26.992 INFO:journalctl@ceph.mon.a.smithi043.stdout: L0 2/0 99.83 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 85.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T22:24:26.992 INFO:journalctl@ceph.mon.a.smithi043.stdout: Sum 2/0 99.83 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 85.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T22:24:26.993 INFO:journalctl@ceph.mon.a.smithi043.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 85.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T22:24:26.993 INFO:journalctl@ceph.mon.a.smithi043.stdout: 2024-03-28T22:24:26.993 INFO:journalctl@ceph.mon.a.smithi043.stdout: ** Compaction Stats [default] ** 2024-03-28T22:24:26.993 INFO:journalctl@ceph.mon.a.smithi043.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T22:24:26.993 INFO:journalctl@ceph.mon.a.smithi043.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-28T22:24:26.993 INFO:journalctl@ceph.mon.a.smithi043.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 85.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T22:24:26.993 INFO:journalctl@ceph.mon.a.smithi043.stdout: 2024-03-28T22:24:26.993 INFO:journalctl@ceph.mon.a.smithi043.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-28T22:24:26.993 INFO:journalctl@ceph.mon.a.smithi043.stdout: 2024-03-28T22:24:26.993 INFO:journalctl@ceph.mon.a.smithi043.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T22:24:26.993 INFO:journalctl@ceph.mon.a.smithi043.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-28T22:24:26.993 INFO:journalctl@ceph.mon.a.smithi043.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-28T22:24:26.993 INFO:journalctl@ceph.mon.a.smithi043.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-28T22:24:26.993 INFO:journalctl@ceph.mon.a.smithi043.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-28T22:24:26.994 INFO:journalctl@ceph.mon.a.smithi043.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-28T22:24:26.994 INFO:journalctl@ceph.mon.a.smithi043.stdout: Cumulative compaction: 0.00 GB write, 11.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T22:24:26.994 INFO:journalctl@ceph.mon.a.smithi043.stdout: Interval compaction: 0.00 GB write, 11.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T22:24:26.994 INFO:journalctl@ceph.mon.a.smithi043.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-28T22:24:26.994 INFO:journalctl@ceph.mon.a.smithi043.stdout: Block cache BinnedLRUCache@0x55fb6174f090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.4e-05 secs_since: 0 2024-03-28T22:24:26.994 INFO:journalctl@ceph.mon.a.smithi043.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-28T22:24:26.994 INFO:journalctl@ceph.mon.a.smithi043.stdout: 2024-03-28T22:24:26.994 INFO:journalctl@ceph.mon.a.smithi043.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-28T22:24:26.994 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: starting mon.a rank 0 at public addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] at bind addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:24:26.994 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: mon.a@-1(???) e1 preinit fsid c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:24:26.994 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: mon.a@-1(???).mds e1 new map 2024-03-28T22:24:26.994 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: mon.a@-1(???).mds e1 print_map 2024-03-28T22:24:26.994 INFO:journalctl@ceph.mon.a.smithi043.stdout: e1 2024-03-28T22:24:26.995 INFO:journalctl@ceph.mon.a.smithi043.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T22:24:26.995 INFO:journalctl@ceph.mon.a.smithi043.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-28T22:24:26.995 INFO:journalctl@ceph.mon.a.smithi043.stdout: legacy client fscid: -1 2024-03-28T22:24:26.995 INFO:journalctl@ceph.mon.a.smithi043.stdout: 2024-03-28T22:24:26.995 INFO:journalctl@ceph.mon.a.smithi043.stdout: No filesystems configured 2024-03-28T22:24:26.995 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-28T22:24:26.995 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T22:24:26.995 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T22:24:26.995 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T22:24:26.995 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-28T22:24:26.995 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-28T22:24:26.995 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: monmap epoch 1 2024-03-28T22:24:26.995 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: fsid c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:24:26.995 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: last_changed 2024-03-28T22:24:15.893393+0000 2024-03-28T22:24:26.995 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: created 2024-03-28T22:24:15.893393+0000 2024-03-28T22:24:26.996 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: min_mon_release 19 (squid) 2024-03-28T22:24:26.996 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: election_strategy: 1 2024-03-28T22:24:26.996 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-03-28T22:24:26.996 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: fsmap 2024-03-28T22:24:26.996 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: osdmap e1: 0 total, 0 up, 0 in 2024-03-28T22:24:26.996 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:26 smithi043 ceph-mon[31823]: mgrmap e1: no daemons active 2024-03-28T22:24:28.055 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.331+0000 7f5700200700 1 Processor -- start 2024-03-28T22:24:28.056 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.331+0000 7f5700200700 1 -- start start 2024-03-28T22:24:28.057 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.331+0000 7f5700200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56f810ecd0 0x7f56f81110c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:28.057 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.331+0000 7f5700200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f56f8082500 con 0x7f56f810ecd0 2024-03-28T22:24:28.057 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.332+0000 7f56fd800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56f810ecd0 0x7f56f81110c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:28.057 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.332+0000 7f56fd800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56f810ecd0 0x7f56f81110c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41746/0 (socket says 172.21.15.43:41746) 2024-03-28T22:24:28.057 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.332+0000 7f56fd800700 1 -- 172.21.15.43:0/2464524484 learned_addr learned my addr 172.21.15.43:0/2464524484 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:24:28.060 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.333+0000 7f56fd800700 1 -- 172.21.15.43:0/2464524484 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56f8111600 con 0x7f56f810ecd0 2024-03-28T22:24:28.060 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.333+0000 7f56fd800700 1 --2- 172.21.15.43:0/2464524484 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56f810ecd0 0x7f56f81110c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f56ec0098a0 tx=0x7f56ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcf07e06ae05d89d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:28.060 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.333+0000 7f56f7e00700 1 -- 172.21.15.43:0/2464524484 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f56ec004160 con 0x7f56f810ecd0 2024-03-28T22:24:28.060 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.334+0000 7f56f7e00700 1 -- 172.21.15.43:0/2464524484 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f56ec004300 con 0x7f56f810ecd0 2024-03-28T22:24:28.061 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.334+0000 7f56f7e00700 1 -- 172.21.15.43:0/2464524484 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f56ec011b50 con 0x7f56f810ecd0 2024-03-28T22:24:28.061 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.334+0000 7f5700200700 1 -- 172.21.15.43:0/2464524484 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56f810ecd0 msgr2=0x7f56f81110c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:28.061 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.334+0000 7f5700200700 1 --2- 172.21.15.43:0/2464524484 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56f810ecd0 0x7f56f81110c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f56ec0098a0 tx=0x7f56ec009bb0 comp rx=0 tx=0).stop 2024-03-28T22:24:28.061 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.334+0000 7f5700200700 1 -- 172.21.15.43:0/2464524484 shutdown_connections 2024-03-28T22:24:28.061 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.334+0000 7f5700200700 1 --2- 172.21.15.43:0/2464524484 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56f810ecd0 0x7f56f81110c0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:28.061 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.334+0000 7f5700200700 1 -- 172.21.15.43:0/2464524484 >> 172.21.15.43:0/2464524484 conn(0x7f56f807d440 msgr2=0x7f56f807f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:24:28.061 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.335+0000 7f5700200700 1 -- 172.21.15.43:0/2464524484 shutdown_connections 2024-03-28T22:24:28.061 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.335+0000 7f5700200700 1 -- 172.21.15.43:0/2464524484 wait complete. 2024-03-28T22:24:28.061 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.336+0000 7f5700200700 1 Processor -- start 2024-03-28T22:24:28.061 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.336+0000 7f5700200700 1 -- start start 2024-03-28T22:24:28.061 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.336+0000 7f5700200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56f810ecd0 0x7f56f81a21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:28.061 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.336+0000 7f5700200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f56f8112680 con 0x7f56f810ecd0 2024-03-28T22:24:28.061 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.337+0000 7f56fd800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56f810ecd0 0x7f56f81a21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:28.061 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.337+0000 7f56fd800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56f810ecd0 0x7f56f81a21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41750/0 (socket says 172.21.15.43:41750) 2024-03-28T22:24:28.062 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.337+0000 7f56fd800700 1 -- 172.21.15.43:0/1457558188 learned_addr learned my addr 172.21.15.43:0/1457558188 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:24:28.062 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.337+0000 7f56fd800700 1 -- 172.21.15.43:0/1457558188 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56f81a2710 con 0x7f56f810ecd0 2024-03-28T22:24:28.062 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.337+0000 7f56fd800700 1 --2- 172.21.15.43:0/1457558188 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56f810ecd0 0x7f56f81a21d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f56ec011b60 tx=0x7f56ec011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:28.062 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.338+0000 7f56f6000700 1 -- 172.21.15.43:0/1457558188 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f56ec003b20 con 0x7f56f810ecd0 2024-03-28T22:24:28.062 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.338+0000 7f56f6000700 1 -- 172.21.15.43:0/1457558188 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f56ec003cc0 con 0x7f56f810ecd0 2024-03-28T22:24:28.062 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.338+0000 7f56f6000700 1 -- 172.21.15.43:0/1457558188 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f56ec024840 con 0x7f56f810ecd0 2024-03-28T22:24:28.062 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.338+0000 7f5700200700 1 -- 172.21.15.43:0/1457558188 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56f81a29a0 con 0x7f56f810ecd0 2024-03-28T22:24:28.062 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.338+0000 7f5700200700 1 -- 172.21.15.43:0/1457558188 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56f81a5690 con 0x7f56f810ecd0 2024-03-28T22:24:28.062 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.338+0000 7f56f6000700 1 -- 172.21.15.43:0/1457558188 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f56ec022020 con 0x7f56f810ecd0 2024-03-28T22:24:28.062 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.338+0000 7f56f6000700 1 -- 172.21.15.43:0/1457558188 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f56ec011e00 con 0x7f56f810ecd0 2024-03-28T22:24:28.062 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.339+0000 7f5700200700 1 -- 172.21.15.43:0/1457558188 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56f8063870 con 0x7f56f810ecd0 2024-03-28T22:24:28.062 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.342+0000 7f56f6000700 1 -- 172.21.15.43:0/1457558188 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f56ec043dd0 con 0x7f56f810ecd0 2024-03-28T22:24:28.062 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.420+0000 7f5700200700 1 -- 172.21.15.43:0/1457558188 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f56f81a5d70 con 0x7f56f810ecd0 2024-03-28T22:24:28.063 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.424+0000 7f56f6000700 1 -- 172.21.15.43:0/1457558188 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f56ec032030 con 0x7f56f810ecd0 2024-03-28T22:24:28.063 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.426+0000 7f5700200700 1 -- 172.21.15.43:0/1457558188 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56f810ecd0 msgr2=0x7f56f81a21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:28.063 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.426+0000 7f5700200700 1 --2- 172.21.15.43:0/1457558188 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56f810ecd0 0x7f56f81a21d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f56ec011b60 tx=0x7f56ec011c40 comp rx=0 tx=0).stop 2024-03-28T22:24:28.064 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.426+0000 7f5700200700 1 -- 172.21.15.43:0/1457558188 shutdown_connections 2024-03-28T22:24:28.064 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.426+0000 7f5700200700 1 --2- 172.21.15.43:0/1457558188 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56f810ecd0 0x7f56f81a21d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:28.064 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.426+0000 7f5700200700 1 -- 172.21.15.43:0/1457558188 >> 172.21.15.43:0/1457558188 conn(0x7f56f807d440 msgr2=0x7f56f807dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:24:28.064 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.426+0000 7f5700200700 1 -- 172.21.15.43:0/1457558188 shutdown_connections 2024-03-28T22:24:28.064 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:27.426+0000 7f5700200700 1 -- 172.21.15.43:0/1457558188 wait complete. 2024-03-28T22:24:28.064 INFO:teuthology.orchestra.run.smithi043.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-28T22:24:28.087 INFO:teuthology.orchestra.run.smithi043.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-28T22:24:28.088 INFO:teuthology.orchestra.run.smithi043.stdout:Creating mgr... 2024-03-28T22:24:28.089 INFO:teuthology.orchestra.run.smithi043.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-28T22:24:28.091 INFO:teuthology.orchestra.run.smithi043.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-28T22:24:28.535 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c2276428-ed51-11ee-95d0-87774f69a715@mgr.x 2024-03-28T22:24:28.535 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c2276428-ed51-11ee-95d0-87774f69a715@mgr.x.service: Unit ceph-c2276428-ed51-11ee-95d0-87774f69a715@mgr.x.service not loaded. 2024-03-28T22:24:28.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:28 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1457558188' entity='client.admin' 2024-03-28T22:24:28.800 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c2276428-ed51-11ee-95d0-87774f69a715.target.wants/ceph-c2276428-ed51-11ee-95d0-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-c2276428-ed51-11ee-95d0-87774f69a715@.service. 2024-03-28T22:24:29.174 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:28 smithi043 systemd[1]: Starting Ceph mgr.x for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:24:29.483 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:29 smithi043 podman[32036]: 2024-03-28 22:24:29.174116123 +0000 UTC m=+0.107896182 container create 9d6e46f876ef3130d16aed7408ba72f3a9bda7d46d9e94aa29d6f93599a2bcd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=squid-323d23d) 2024-03-28T22:24:29.484 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:29 smithi043 podman[32036]: 2024-03-28 22:24:29.099479712 +0000 UTC m=+0.033259763 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be 2024-03-28T22:24:29.484 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:29 smithi043 podman[32036]: 2024-03-28 22:24:29.40289499 +0000 UTC m=+0.336675081 container init 9d6e46f876ef3130d16aed7408ba72f3a9bda7d46d9e94aa29d6f93599a2bcd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x, org.label-schema.build-date=20240326, ceph=True, RELEASE=squid-323d23d, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-28T22:24:29.484 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:29 smithi043 podman[32036]: 2024-03-28 22:24:29.414030888 +0000 UTC m=+0.347810943 container start 9d6e46f876ef3130d16aed7408ba72f3a9bda7d46d9e94aa29d6f93599a2bcd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, ceph=True, RELEASE=squid-323d23d, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-28T22:24:29.484 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:29 smithi043 bash[32036]: 9d6e46f876ef3130d16aed7408ba72f3a9bda7d46d9e94aa29d6f93599a2bcd2 2024-03-28T22:24:29.514 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T22:24:29.515 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout disabled 2024-03-28T22:24:29.530 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T22:24:29.530 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout inactive 2024-03-28T22:24:29.531 INFO:teuthology.orchestra.run.smithi043.stdout:firewalld.service is not enabled 2024-03-28T22:24:29.531 INFO:teuthology.orchestra.run.smithi043.stdout:Not possible to enable service . firewalld.service is not available 2024-03-28T22:24:29.552 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T22:24:29.553 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout disabled 2024-03-28T22:24:29.570 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T22:24:29.570 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout inactive 2024-03-28T22:24:29.571 INFO:teuthology.orchestra.run.smithi043.stdout:firewalld.service is not enabled 2024-03-28T22:24:29.571 INFO:teuthology.orchestra.run.smithi043.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-28T22:24:29.571 INFO:teuthology.orchestra.run.smithi043.stdout:Waiting for mgr to start... 2024-03-28T22:24:29.571 INFO:teuthology.orchestra.run.smithi043.stdout:Waiting for mgr... 2024-03-28T22:24:29.734 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:29 smithi043 systemd[1]: Started Ceph mgr.x for c2276428-ed51-11ee-95d0-87774f69a715. 2024-03-28T22:24:30.261 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:30 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:30.234+0000 7fcf032a6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T22:24:30.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:30 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3552679030' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T22:24:30.517 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:30 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:30.382+0000 7fcf032a6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T22:24:30.892 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-03-28T22:24:30.893 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout { 2024-03-28T22:24:30.893 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "fsid": "c2276428-ed51-11ee-95d0-87774f69a715", 2024-03-28T22:24:30.894 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T22:24:30.894 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T22:24:30.894 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T22:24:30.894 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T22:24:30.894 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:24:30.894 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T22:24:30.894 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T22:24:30.903 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 0 2024-03-28T22:24:30.904 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-03-28T22:24:30.904 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T22:24:30.904 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T22:24:30.904 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-03-28T22:24:30.904 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-03-28T22:24:30.904 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T22:24:30.904 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T22:24:30.904 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T22:24:30.904 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T22:24:30.904 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:24:30.904 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T22:24:30.904 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T22:24:30.904 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T22:24:30.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T22:24:30.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T22:24:30.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T22:24:30.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T22:24:30.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T22:24:30.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:24:30.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T22:24:30.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T22:24:30.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T22:24:30.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T22:24:30.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T22:24:30.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T22:24:30.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T22:24:30.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T22:24:30.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T22:24:30.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:24:30.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T22:24:30.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T22:24:30.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T22:24:30.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T22:24:30.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:24:30.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T22:24:30.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-28T22:24:30.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T22:24:30.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T22:24:30.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T22:24:30.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T22:24:30.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T22:24:30.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-03-28T22:24:30.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T22:24:30.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:24:30.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T22:24:30.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T22:24:30.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T22:24:19.950693+0000", 2024-03-28T22:24:30.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T22:24:30.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:24:30.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T22:24:30.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout } 2024-03-28T22:24:30.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.161+0000 7f0237400700 1 Processor -- start 2024-03-28T22:24:30.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.162+0000 7f0237400700 1 -- start start 2024-03-28T22:24:30.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.162+0000 7f0237400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0230108760 0x7f023010ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:30.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.162+0000 7f0237400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f023007be50 con 0x7f0230108760 2024-03-28T22:24:30.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.163+0000 7f0234a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0230108760 0x7f023010ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:30.908 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.163+0000 7f0234a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0230108760 0x7f023010ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41776/0 (socket says 172.21.15.43:41776) 2024-03-28T22:24:30.908 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.163+0000 7f0234a00700 1 -- 172.21.15.43:0/2028292900 learned_addr learned my addr 172.21.15.43:0/2028292900 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:24:30.908 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.163+0000 7f0234a00700 1 -- 172.21.15.43:0/2028292900 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f023010b090 con 0x7f0230108760 2024-03-28T22:24:30.908 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.164+0000 7f0234a00700 1 --2- 172.21.15.43:0/2028292900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0230108760 0x7f023010ab50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f02240098a0 tx=0x7f0224009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccb4d12fb35328b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:30.908 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.164+0000 7f022f400700 1 -- 172.21.15.43:0/2028292900 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0224004160 con 0x7f0230108760 2024-03-28T22:24:30.908 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.164+0000 7f022f400700 1 -- 172.21.15.43:0/2028292900 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0224004300 con 0x7f0230108760 2024-03-28T22:24:30.908 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.164+0000 7f022f400700 1 -- 172.21.15.43:0/2028292900 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0224011b50 con 0x7f0230108760 2024-03-28T22:24:30.908 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.165+0000 7f0237400700 1 -- 172.21.15.43:0/2028292900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0230108760 msgr2=0x7f023010ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:30.908 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.165+0000 7f0237400700 1 --2- 172.21.15.43:0/2028292900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0230108760 0x7f023010ab50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f02240098a0 tx=0x7f0224009bb0 comp rx=0 tx=0).stop 2024-03-28T22:24:30.908 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.165+0000 7f0237400700 1 -- 172.21.15.43:0/2028292900 shutdown_connections 2024-03-28T22:24:30.908 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.165+0000 7f0237400700 1 --2- 172.21.15.43:0/2028292900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0230108760 0x7f023010ab50 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:30.908 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.165+0000 7f0237400700 1 -- 172.21.15.43:0/2028292900 >> 172.21.15.43:0/2028292900 conn(0x7f0230102170 msgr2=0x7f0230104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:24:30.908 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.165+0000 7f0237400700 1 -- 172.21.15.43:0/2028292900 shutdown_connections 2024-03-28T22:24:30.908 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.165+0000 7f0237400700 1 -- 172.21.15.43:0/2028292900 wait complete. 2024-03-28T22:24:30.909 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.167+0000 7f0237400700 1 Processor -- start 2024-03-28T22:24:30.909 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.167+0000 7f0237400700 1 -- start start 2024-03-28T22:24:30.909 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.168+0000 7f0237400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0230108760 0x7f023019bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:30.909 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.168+0000 7f0237400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f023010c060 con 0x7f0230108760 2024-03-28T22:24:30.909 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.168+0000 7f0234a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0230108760 0x7f023019bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:30.909 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.168+0000 7f0234a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0230108760 0x7f023019bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41784/0 (socket says 172.21.15.43:41784) 2024-03-28T22:24:30.909 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.168+0000 7f0234a00700 1 -- 172.21.15.43:0/3552679030 learned_addr learned my addr 172.21.15.43:0/3552679030 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:24:30.909 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.168+0000 7f0234a00700 1 -- 172.21.15.43:0/3552679030 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f023019c1a0 con 0x7f0230108760 2024-03-28T22:24:30.909 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.169+0000 7f0234a00700 1 --2- 172.21.15.43:0/3552679030 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0230108760 0x7f023019bc60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0224011b60 tx=0x7f0224011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:30.909 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.169+0000 7f022d600700 1 -- 172.21.15.43:0/3552679030 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0224003b20 con 0x7f0230108760 2024-03-28T22:24:30.909 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.169+0000 7f022d600700 1 -- 172.21.15.43:0/3552679030 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0224003cc0 con 0x7f0230108760 2024-03-28T22:24:30.909 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.169+0000 7f022d600700 1 -- 172.21.15.43:0/3552679030 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0224024840 con 0x7f0230108760 2024-03-28T22:24:30.909 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.169+0000 7f0237400700 1 -- 172.21.15.43:0/3552679030 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f023019c430 con 0x7f0230108760 2024-03-28T22:24:30.909 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.169+0000 7f0237400700 1 -- 172.21.15.43:0/3552679030 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f023019f120 con 0x7f0230108760 2024-03-28T22:24:30.910 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.169+0000 7f022d600700 1 -- 172.21.15.43:0/3552679030 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0224022020 con 0x7f0230108760 2024-03-28T22:24:30.910 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.169+0000 7f022d600700 1 -- 172.21.15.43:0/3552679030 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0224011e00 con 0x7f0230108760 2024-03-28T22:24:30.910 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.170+0000 7f0237400700 1 -- 172.21.15.43:0/3552679030 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0230063870 con 0x7f0230108760 2024-03-28T22:24:30.910 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.173+0000 7f022d600700 1 -- 172.21.15.43:0/3552679030 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f0224043dd0 con 0x7f0230108760 2024-03-28T22:24:30.910 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.259+0000 7f0237400700 1 -- 172.21.15.43:0/3552679030 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f023019f690 con 0x7f0230108760 2024-03-28T22:24:30.910 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.260+0000 7f022d600700 1 -- 172.21.15.43:0/3552679030 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f0224032020 con 0x7f0230108760 2024-03-28T22:24:30.910 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.263+0000 7f0237400700 1 -- 172.21.15.43:0/3552679030 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0230108760 msgr2=0x7f023019bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:30.910 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.263+0000 7f0237400700 1 --2- 172.21.15.43:0/3552679030 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0230108760 0x7f023019bc60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0224011b60 tx=0x7f0224011c40 comp rx=0 tx=0).stop 2024-03-28T22:24:30.910 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.263+0000 7f0237400700 1 -- 172.21.15.43:0/3552679030 shutdown_connections 2024-03-28T22:24:30.910 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.263+0000 7f0237400700 1 --2- 172.21.15.43:0/3552679030 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0230108760 0x7f023019bc60 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:30.910 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.263+0000 7f0237400700 1 -- 172.21.15.43:0/3552679030 >> 172.21.15.43:0/3552679030 conn(0x7f0230102170 msgr2=0x7f0230102c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:24:30.910 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.263+0000 7f0237400700 1 -- 172.21.15.43:0/3552679030 shutdown_connections 2024-03-28T22:24:30.910 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:30.263+0000 7f0237400700 1 -- 172.21.15.43:0/3552679030 wait complete. 2024-03-28T22:24:30.910 INFO:teuthology.orchestra.run.smithi043.stdout:mgr not available, waiting (1/15)... 2024-03-28T22:24:31.517 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:31 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:31.229+0000 7fcf032a6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T22:24:31.517 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:31 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:31.354+0000 7fcf032a6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T22:24:32.017 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:31 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:31.535+0000 7fcf032a6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T22:24:32.017 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:31 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:31.664+0000 7fcf032a6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T22:24:32.017 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:31 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:31.777+0000 7fcf032a6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T22:24:32.385 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:32 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:32.123+0000 7fcf032a6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T22:24:32.386 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:32 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:32.269+0000 7fcf032a6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T22:24:32.747 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:32 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:32.384+0000 7fcf032a6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T22:24:32.747 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:32 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:32.511+0000 7fcf032a6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T22:24:32.747 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:32 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:32.746+0000 7fcf032a6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T22:24:33.922 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:33 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2766429716' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T22:24:34.196 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:34 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:34.154+0000 7fcf032a6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T22:24:34.205 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-03-28T22:24:34.205 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout { 2024-03-28T22:24:34.205 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "fsid": "c2276428-ed51-11ee-95d0-87774f69a715", 2024-03-28T22:24:34.205 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T22:24:34.205 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T22:24:34.208 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T22:24:34.208 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T22:24:34.209 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:24:34.209 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T22:24:34.209 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T22:24:34.209 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 0 2024-03-28T22:24:34.209 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-03-28T22:24:34.209 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T22:24:34.209 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T22:24:34.209 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-03-28T22:24:34.209 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-03-28T22:24:34.209 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T22:24:34.209 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T22:24:34.209 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T22:24:34.209 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T22:24:34.209 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:24:34.210 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T22:24:34.210 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T22:24:34.210 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T22:24:34.210 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T22:24:34.210 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T22:24:34.210 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T22:24:34.210 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T22:24:34.210 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T22:24:34.210 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:24:34.210 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T22:24:34.210 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T22:24:34.210 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T22:24:34.210 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T22:24:34.210 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T22:24:34.210 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T22:24:34.211 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T22:24:34.211 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T22:24:34.218 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T22:24:34.218 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:24:34.218 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T22:24:34.218 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T22:24:34.218 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T22:24:34.218 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T22:24:34.218 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:24:34.218 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T22:24:34.218 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-28T22:24:34.218 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T22:24:34.218 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T22:24:34.218 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T22:24:34.218 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T22:24:34.219 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T22:24:34.219 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-03-28T22:24:34.219 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T22:24:34.219 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:24:34.219 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T22:24:34.219 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T22:24:34.219 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T22:24:19.950693+0000", 2024-03-28T22:24:34.219 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T22:24:34.219 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:24:34.219 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T22:24:34.219 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout } 2024-03-28T22:24:34.219 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.471+0000 7fe5fbc00700 1 Processor -- start 2024-03-28T22:24:34.219 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.472+0000 7fe5fbc00700 1 -- start start 2024-03-28T22:24:34.219 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.472+0000 7fe5fbc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe5f4107d40 0x7fe5f4108160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:34.219 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.472+0000 7fe5fbc00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f41086a0 con 0x7fe5f4107d40 2024-03-28T22:24:34.219 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.473+0000 7fe5f9200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe5f4107d40 0x7fe5f4108160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:34.220 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.473+0000 7fe5f9200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe5f4107d40 0x7fe5f4108160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41786/0 (socket says 172.21.15.43:41786) 2024-03-28T22:24:34.220 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.473+0000 7fe5f9200700 1 -- 172.21.15.43:0/1417605147 learned_addr learned my addr 172.21.15.43:0/1417605147 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:24:34.220 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.473+0000 7fe5f9200700 1 -- 172.21.15.43:0/1417605147 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5f4108820 con 0x7fe5f4107d40 2024-03-28T22:24:34.220 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.473+0000 7fe5f9200700 1 --2- 172.21.15.43:0/1417605147 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe5f4107d40 0x7fe5f4108160 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe5e80098a0 tx=0x7fe5e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7c5dc2a5edf3413 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:34.220 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.474+0000 7fe5f3400700 1 -- 172.21.15.43:0/1417605147 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5e8004160 con 0x7fe5f4107d40 2024-03-28T22:24:34.220 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.474+0000 7fe5f3400700 1 -- 172.21.15.43:0/1417605147 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe5e8004300 con 0x7fe5f4107d40 2024-03-28T22:24:34.220 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.474+0000 7fe5f3400700 1 -- 172.21.15.43:0/1417605147 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5e8011b50 con 0x7fe5f4107d40 2024-03-28T22:24:34.220 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.474+0000 7fe5fbc00700 1 -- 172.21.15.43:0/1417605147 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe5f4107d40 msgr2=0x7fe5f4108160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:34.220 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.474+0000 7fe5fbc00700 1 --2- 172.21.15.43:0/1417605147 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe5f4107d40 0x7fe5f4108160 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe5e80098a0 tx=0x7fe5e8009bb0 comp rx=0 tx=0).stop 2024-03-28T22:24:34.220 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.475+0000 7fe5fbc00700 1 -- 172.21.15.43:0/1417605147 shutdown_connections 2024-03-28T22:24:34.220 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.475+0000 7fe5fbc00700 1 --2- 172.21.15.43:0/1417605147 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe5f4107d40 0x7fe5f4108160 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:34.220 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.475+0000 7fe5fbc00700 1 -- 172.21.15.43:0/1417605147 >> 172.21.15.43:0/1417605147 conn(0x7fe5f41033c0 msgr2=0x7fe5f41057e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:24:34.220 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.475+0000 7fe5fbc00700 1 -- 172.21.15.43:0/1417605147 shutdown_connections 2024-03-28T22:24:34.220 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.475+0000 7fe5fbc00700 1 -- 172.21.15.43:0/1417605147 wait complete. 2024-03-28T22:24:34.220 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.476+0000 7fe5fbc00700 1 Processor -- start 2024-03-28T22:24:34.220 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.476+0000 7fe5fbc00700 1 -- start start 2024-03-28T22:24:34.221 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.476+0000 7fe5fbc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe5f4107d40 0x7fe5f419de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:34.221 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.476+0000 7fe5fbc00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f410c1b0 con 0x7fe5f4107d40 2024-03-28T22:24:34.221 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.477+0000 7fe5f9200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe5f4107d40 0x7fe5f419de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:34.221 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.477+0000 7fe5f9200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe5f4107d40 0x7fe5f419de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41788/0 (socket says 172.21.15.43:41788) 2024-03-28T22:24:34.221 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.477+0000 7fe5f9200700 1 -- 172.21.15.43:0/2766429716 learned_addr learned my addr 172.21.15.43:0/2766429716 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:24:34.221 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.477+0000 7fe5f9200700 1 -- 172.21.15.43:0/2766429716 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5f419e3c0 con 0x7fe5f4107d40 2024-03-28T22:24:34.221 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.477+0000 7fe5f9200700 1 --2- 172.21.15.43:0/2766429716 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe5f4107d40 0x7fe5f419de80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8011b60 tx=0x7fe5e8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:34.221 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.478+0000 7fe5f1600700 1 -- 172.21.15.43:0/2766429716 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5e8003b20 con 0x7fe5f4107d40 2024-03-28T22:24:34.221 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.478+0000 7fe5f1600700 1 -- 172.21.15.43:0/2766429716 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe5e8003cc0 con 0x7fe5f4107d40 2024-03-28T22:24:34.221 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.478+0000 7fe5f1600700 1 -- 172.21.15.43:0/2766429716 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5e8024840 con 0x7fe5f4107d40 2024-03-28T22:24:34.221 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.478+0000 7fe5fbc00700 1 -- 172.21.15.43:0/2766429716 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5f419e650 con 0x7fe5f4107d40 2024-03-28T22:24:34.221 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.478+0000 7fe5fbc00700 1 -- 172.21.15.43:0/2766429716 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5f41a1340 con 0x7fe5f4107d40 2024-03-28T22:24:34.221 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.478+0000 7fe5f1600700 1 -- 172.21.15.43:0/2766429716 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe5e8022020 con 0x7fe5f4107d40 2024-03-28T22:24:34.221 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.478+0000 7fe5f1600700 1 -- 172.21.15.43:0/2766429716 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe5e8011e00 con 0x7fe5f4107d40 2024-03-28T22:24:34.222 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.479+0000 7fe5fbc00700 1 -- 172.21.15.43:0/2766429716 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5f4063870 con 0x7fe5f4107d40 2024-03-28T22:24:34.222 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.482+0000 7fe5f1600700 1 -- 172.21.15.43:0/2766429716 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fe5e8043dd0 con 0x7fe5f4107d40 2024-03-28T22:24:34.222 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.562+0000 7fe5fbc00700 1 -- 172.21.15.43:0/2766429716 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fe5f41a1a20 con 0x7fe5f4107d40 2024-03-28T22:24:34.222 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.563+0000 7fe5f1600700 1 -- 172.21.15.43:0/2766429716 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fe5e8032020 con 0x7fe5f4107d40 2024-03-28T22:24:34.222 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.564+0000 7fe5fbc00700 1 -- 172.21.15.43:0/2766429716 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe5f4107d40 msgr2=0x7fe5f419de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:34.222 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.564+0000 7fe5fbc00700 1 --2- 172.21.15.43:0/2766429716 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe5f4107d40 0x7fe5f419de80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8011b60 tx=0x7fe5e8011c40 comp rx=0 tx=0).stop 2024-03-28T22:24:34.222 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.567+0000 7fe5fbc00700 1 -- 172.21.15.43:0/2766429716 shutdown_connections 2024-03-28T22:24:34.222 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.567+0000 7fe5fbc00700 1 --2- 172.21.15.43:0/2766429716 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe5f4107d40 0x7fe5f419de80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:34.222 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.567+0000 7fe5fbc00700 1 -- 172.21.15.43:0/2766429716 >> 172.21.15.43:0/2766429716 conn(0x7fe5f41033c0 msgr2=0x7fe5f4103de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:24:34.222 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.567+0000 7fe5fbc00700 1 -- 172.21.15.43:0/2766429716 shutdown_connections 2024-03-28T22:24:34.222 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:33.567+0000 7fe5fbc00700 1 -- 172.21.15.43:0/2766429716 wait complete. 2024-03-28T22:24:34.222 INFO:teuthology.orchestra.run.smithi043.stdout:mgr not available, waiting (2/15)... 2024-03-28T22:24:34.517 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:34 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:34.331+0000 7fcf032a6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T22:24:35.017 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:34 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:34.758+0000 7fcf032a6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T22:24:35.017 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:34 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:34.880+0000 7fcf032a6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T22:24:35.396 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:35 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:35.143+0000 7fcf032a6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T22:24:35.767 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:35 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:35.395+0000 7fcf032a6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T22:24:36.101 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:35 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:35.778+0000 7fcf032a6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T22:24:36.101 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:36 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:36.100+0000 7fcf032a6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T22:24:36.666 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:36 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:36.431+0000 7fcf032a6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T22:24:37.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:36 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3883645556' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T22:24:37.577 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-03-28T22:24:37.577 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout { 2024-03-28T22:24:37.577 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "fsid": "c2276428-ed51-11ee-95d0-87774f69a715", 2024-03-28T22:24:37.577 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T22:24:37.577 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T22:24:37.577 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T22:24:37.577 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T22:24:37.578 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:24:37.578 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T22:24:37.578 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T22:24:37.578 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 0 2024-03-28T22:24:37.578 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-03-28T22:24:37.578 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T22:24:37.578 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T22:24:37.578 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-03-28T22:24:37.578 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-03-28T22:24:37.578 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T22:24:37.578 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T22:24:37.578 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T22:24:37.578 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T22:24:37.578 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:24:37.579 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T22:24:37.579 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T22:24:37.579 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T22:24:37.579 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T22:24:37.579 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T22:24:37.579 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T22:24:37.579 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T22:24:37.579 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T22:24:37.579 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:24:37.579 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T22:24:37.579 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T22:24:37.579 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T22:24:37.579 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T22:24:37.579 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T22:24:37.579 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T22:24:37.580 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T22:24:37.580 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T22:24:37.580 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T22:24:37.580 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:24:37.580 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T22:24:37.580 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T22:24:37.580 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T22:24:37.580 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T22:24:37.580 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:24:37.580 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T22:24:37.580 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-28T22:24:37.580 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T22:24:37.580 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T22:24:37.580 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T22:24:37.580 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T22:24:37.581 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T22:24:37.581 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-03-28T22:24:37.581 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T22:24:37.581 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:24:37.581 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T22:24:37.581 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T22:24:37.581 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T22:24:19.950693+0000", 2024-03-28T22:24:37.581 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T22:24:37.581 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:24:37.581 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T22:24:37.581 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout } 2024-03-28T22:24:37.581 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.819+0000 7f35b3e00700 1 Processor -- start 2024-03-28T22:24:37.581 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.820+0000 7f35b3e00700 1 -- start start 2024-03-28T22:24:37.581 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.820+0000 7f35b3e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f35b407de10 0x7f35b407c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:37.581 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.820+0000 7f35b3e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f35b407c800 con 0x7f35b407de10 2024-03-28T22:24:37.582 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.820+0000 7f35b2a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f35b407de10 0x7f35b407c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:37.582 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.821+0000 7f35b2a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f35b407de10 0x7f35b407c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35178/0 (socket says 172.21.15.43:35178) 2024-03-28T22:24:37.582 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.821+0000 7f35b2a00700 1 -- 172.21.15.43:0/2561856459 learned_addr learned my addr 172.21.15.43:0/2561856459 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:24:37.582 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.821+0000 7f35b2a00700 1 -- 172.21.15.43:0/2561856459 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35b407c980 con 0x7f35b407de10 2024-03-28T22:24:37.582 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.821+0000 7f35b2a00700 1 --2- 172.21.15.43:0/2561856459 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f35b407de10 0x7f35b407c2c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f35a40098a0 tx=0x7f35a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6c00afef927698a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:37.582 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.821+0000 7f35b1600700 1 -- 172.21.15.43:0/2561856459 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35a4004160 con 0x7f35b407de10 2024-03-28T22:24:37.582 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.822+0000 7f35b1600700 1 -- 172.21.15.43:0/2561856459 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f35a4004300 con 0x7f35b407de10 2024-03-28T22:24:37.582 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.822+0000 7f35b1600700 1 -- 172.21.15.43:0/2561856459 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35a4011b50 con 0x7f35b407de10 2024-03-28T22:24:37.582 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.822+0000 7f35b3e00700 1 -- 172.21.15.43:0/2561856459 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f35b407de10 msgr2=0x7f35b407c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:37.582 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.822+0000 7f35b3e00700 1 --2- 172.21.15.43:0/2561856459 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f35b407de10 0x7f35b407c2c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f35a40098a0 tx=0x7f35a4009bb0 comp rx=0 tx=0).stop 2024-03-28T22:24:37.582 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.822+0000 7f35b3e00700 1 -- 172.21.15.43:0/2561856459 shutdown_connections 2024-03-28T22:24:37.582 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.822+0000 7f35b3e00700 1 --2- 172.21.15.43:0/2561856459 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f35b407de10 0x7f35b407c2c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:37.582 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.822+0000 7f35b3e00700 1 -- 172.21.15.43:0/2561856459 >> 172.21.15.43:0/2561856459 conn(0x7f35b41030d0 msgr2=0x7f35b4105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:24:37.583 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.823+0000 7f35b3e00700 1 -- 172.21.15.43:0/2561856459 shutdown_connections 2024-03-28T22:24:37.583 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.823+0000 7f35b3e00700 1 -- 172.21.15.43:0/2561856459 wait complete. 2024-03-28T22:24:37.583 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.824+0000 7f35b3e00700 1 Processor -- start 2024-03-28T22:24:37.583 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.824+0000 7f35b3e00700 1 -- start start 2024-03-28T22:24:37.583 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.824+0000 7f35b3e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f35b407de10 0x7f35b419dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:37.583 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.824+0000 7f35b3e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f35b410e120 con 0x7f35b407de10 2024-03-28T22:24:37.583 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.824+0000 7f35b2a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f35b407de10 0x7f35b419dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:37.583 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.824+0000 7f35b2a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f35b407de10 0x7f35b419dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35182/0 (socket says 172.21.15.43:35182) 2024-03-28T22:24:37.583 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.824+0000 7f35b2a00700 1 -- 172.21.15.43:0/3883645556 learned_addr learned my addr 172.21.15.43:0/3883645556 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:24:37.583 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.825+0000 7f35b2a00700 1 -- 172.21.15.43:0/3883645556 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35b419e210 con 0x7f35b407de10 2024-03-28T22:24:37.583 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.825+0000 7f35b2a00700 1 --2- 172.21.15.43:0/3883645556 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f35b407de10 0x7f35b419dcd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f35a4011670 tx=0x7f35a4011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:37.583 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.825+0000 7f35ab400700 1 -- 172.21.15.43:0/3883645556 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35a40039c0 con 0x7f35b407de10 2024-03-28T22:24:37.583 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.826+0000 7f35ab400700 1 -- 172.21.15.43:0/3883645556 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f35a4011e50 con 0x7f35b407de10 2024-03-28T22:24:37.583 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.826+0000 7f35ab400700 1 -- 172.21.15.43:0/3883645556 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35a401a5c0 con 0x7f35b407de10 2024-03-28T22:24:37.583 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.826+0000 7f35b3e00700 1 -- 172.21.15.43:0/3883645556 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35b419e4a0 con 0x7f35b407de10 2024-03-28T22:24:37.584 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.826+0000 7f35b3e00700 1 -- 172.21.15.43:0/3883645556 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35b41a1180 con 0x7f35b407de10 2024-03-28T22:24:37.584 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.826+0000 7f35ab400700 1 -- 172.21.15.43:0/3883645556 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f35a401ac00 con 0x7f35b407de10 2024-03-28T22:24:37.584 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.826+0000 7f35ab400700 1 -- 172.21.15.43:0/3883645556 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f35a4019700 con 0x7f35b407de10 2024-03-28T22:24:37.584 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.827+0000 7f35b3e00700 1 -- 172.21.15.43:0/3883645556 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35b4063870 con 0x7f35b407de10 2024-03-28T22:24:37.584 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.829+0000 7f35ab400700 1 -- 172.21.15.43:0/3883645556 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f35a4003b60 con 0x7f35b407de10 2024-03-28T22:24:37.584 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.908+0000 7f35b3e00700 1 -- 172.21.15.43:0/3883645556 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f35b41a1350 con 0x7f35b407de10 2024-03-28T22:24:37.584 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.909+0000 7f35ab400700 1 -- 172.21.15.43:0/3883645556 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7f35a40199a0 con 0x7f35b407de10 2024-03-28T22:24:37.584 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.910+0000 7f35b3e00700 1 -- 172.21.15.43:0/3883645556 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f35b407de10 msgr2=0x7f35b419dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:37.584 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.910+0000 7f35b3e00700 1 --2- 172.21.15.43:0/3883645556 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f35b407de10 0x7f35b419dcd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f35a4011670 tx=0x7f35a4011b60 comp rx=0 tx=0).stop 2024-03-28T22:24:37.584 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.910+0000 7f35b3e00700 1 -- 172.21.15.43:0/3883645556 shutdown_connections 2024-03-28T22:24:37.584 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.910+0000 7f35b3e00700 1 --2- 172.21.15.43:0/3883645556 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f35b407de10 0x7f35b419dcd0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:37.584 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.910+0000 7f35b3e00700 1 -- 172.21.15.43:0/3883645556 >> 172.21.15.43:0/3883645556 conn(0x7f35b41030d0 msgr2=0x7f35b4103c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:24:37.584 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.910+0000 7f35b3e00700 1 -- 172.21.15.43:0/3883645556 shutdown_connections 2024-03-28T22:24:37.584 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:36.910+0000 7f35b3e00700 1 -- 172.21.15.43:0/3883645556 wait complete. 2024-03-28T22:24:37.585 INFO:teuthology.orchestra.run.smithi043.stdout:mgr not available, waiting (3/15)... 2024-03-28T22:24:38.017 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:37 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:37.534+0000 7fcf032a6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T22:24:38.517 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:38 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:38.254+0000 7fcf032a6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T22:24:38.517 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:38 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:38.386+0000 7fcf032a6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T22:24:39.017 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:38 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:38.677+0000 7fcf032a6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T22:24:39.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:38 smithi043 ceph-mon[31823]: Activating manager daemon x 2024-03-28T22:24:39.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:38 smithi043 ceph-mon[31823]: mgrmap e2: x(active, starting, since 0.0110401s) 2024-03-28T22:24:39.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:38 smithi043 ceph-mon[31823]: from='mgr.14100 172.21.15.43:0/2039548869' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T22:24:39.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:38 smithi043 ceph-mon[31823]: from='mgr.14100 172.21.15.43:0/2039548869' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T22:24:39.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:38 smithi043 ceph-mon[31823]: from='mgr.14100 172.21.15.43:0/2039548869' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T22:24:39.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:38 smithi043 ceph-mon[31823]: from='mgr.14100 172.21.15.43:0/2039548869' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:24:39.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:38 smithi043 ceph-mon[31823]: from='mgr.14100 172.21.15.43:0/2039548869' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T22:24:39.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:38 smithi043 ceph-mon[31823]: Manager daemon x is now available 2024-03-28T22:24:39.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:38 smithi043 ceph-mon[31823]: from='mgr.14100 172.21.15.43:0/2039548869' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T22:24:39.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:38 smithi043 ceph-mon[31823]: from='mgr.14100 172.21.15.43:0/2039548869' entity='mgr.x' 2024-03-28T22:24:39.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:38 smithi043 ceph-mon[31823]: from='mgr.14100 172.21.15.43:0/2039548869' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T22:24:39.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:38 smithi043 ceph-mon[31823]: from='mgr.14100 172.21.15.43:0/2039548869' entity='mgr.x' 2024-03-28T22:24:39.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:38 smithi043 ceph-mon[31823]: from='mgr.14100 172.21.15.43:0/2039548869' entity='mgr.x' 2024-03-28T22:24:40.841 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:40 smithi043 ceph-mon[31823]: mgrmap e3: x(active, since 1.01548s) 2024-03-28T22:24:40.842 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:40 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2466009600' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T22:24:41.413 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-03-28T22:24:41.413 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout { 2024-03-28T22:24:41.413 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "fsid": "c2276428-ed51-11ee-95d0-87774f69a715", 2024-03-28T22:24:41.413 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T22:24:41.413 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T22:24:41.413 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T22:24:41.413 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T22:24:41.413 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:24:41.413 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T22:24:41.413 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T22:24:41.413 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 0 2024-03-28T22:24:41.413 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-03-28T22:24:41.413 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T22:24:41.414 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T22:24:41.414 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-03-28T22:24:41.414 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-03-28T22:24:41.414 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T22:24:41.414 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T22:24:41.414 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T22:24:41.414 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T22:24:41.414 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:24:41.414 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T22:24:41.414 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T22:24:41.414 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T22:24:41.414 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T22:24:41.414 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T22:24:41.423 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T22:24:41.423 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T22:24:41.423 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T22:24:41.423 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:24:41.424 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T22:24:41.424 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T22:24:41.424 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T22:24:41.424 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T22:24:41.424 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T22:24:41.424 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T22:24:41.424 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T22:24:41.424 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T22:24:41.424 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T22:24:41.424 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:24:41.424 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T22:24:41.424 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T22:24:41.424 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T22:24:41.424 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T22:24:41.424 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:24:41.425 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T22:24:41.425 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T22:24:41.425 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T22:24:41.425 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T22:24:41.425 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T22:24:41.425 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T22:24:41.425 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T22:24:41.425 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-03-28T22:24:41.425 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T22:24:41.425 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:24:41.425 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T22:24:41.425 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T22:24:41.425 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T22:24:19.950693+0000", 2024-03-28T22:24:41.425 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T22:24:41.425 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-03-28T22:24:41.426 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T22:24:41.426 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout } 2024-03-28T22:24:41.426 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.208+0000 7f6061c00700 1 Processor -- start 2024-03-28T22:24:41.426 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.208+0000 7f6061c00700 1 -- start start 2024-03-28T22:24:41.426 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.209+0000 7f6061c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f605c07e860 0x7f605c07ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:41.426 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.209+0000 7f6061c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f605c07f250 con 0x7f605c07e860 2024-03-28T22:24:41.426 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.209+0000 7f605be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f605c07e860 0x7f605c07ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:41.426 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.209+0000 7f605be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f605c07e860 0x7f605c07ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35264/0 (socket says 172.21.15.43:35264) 2024-03-28T22:24:41.426 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.209+0000 7f605be00700 1 -- 172.21.15.43:0/768816207 learned_addr learned my addr 172.21.15.43:0/768816207 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:24:41.426 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.210+0000 7f605be00700 1 -- 172.21.15.43:0/768816207 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f605c07fa60 con 0x7f605c07e860 2024-03-28T22:24:41.426 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.210+0000 7f605be00700 1 --2- 172.21.15.43:0/768816207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f605c07e860 0x7f605c07ec80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f604c0098a0 tx=0x7f604c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1dd2e9278c827aa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:41.426 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.210+0000 7f605aa00700 1 -- 172.21.15.43:0/768816207 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f604c004160 con 0x7f605c07e860 2024-03-28T22:24:41.426 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.211+0000 7f605aa00700 1 -- 172.21.15.43:0/768816207 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f604c004300 con 0x7f605c07e860 2024-03-28T22:24:41.427 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.211+0000 7f605aa00700 1 -- 172.21.15.43:0/768816207 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f604c011b50 con 0x7f605c07e860 2024-03-28T22:24:41.427 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.211+0000 7f6061c00700 1 -- 172.21.15.43:0/768816207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f605c07e860 msgr2=0x7f605c07ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:41.427 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.211+0000 7f6061c00700 1 --2- 172.21.15.43:0/768816207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f605c07e860 0x7f605c07ec80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f604c0098a0 tx=0x7f604c009bb0 comp rx=0 tx=0).stop 2024-03-28T22:24:41.427 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.211+0000 7f6061c00700 1 -- 172.21.15.43:0/768816207 shutdown_connections 2024-03-28T22:24:41.427 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.211+0000 7f6061c00700 1 --2- 172.21.15.43:0/768816207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f605c07e860 0x7f605c07ec80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:41.427 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.211+0000 7f6061c00700 1 -- 172.21.15.43:0/768816207 >> 172.21.15.43:0/768816207 conn(0x7f605c07d120 msgr2=0x7f605c07d530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:24:41.427 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.212+0000 7f6061c00700 1 -- 172.21.15.43:0/768816207 shutdown_connections 2024-03-28T22:24:41.427 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.212+0000 7f6061c00700 1 -- 172.21.15.43:0/768816207 wait complete. 2024-03-28T22:24:41.427 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.213+0000 7f6061c00700 1 Processor -- start 2024-03-28T22:24:41.427 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.213+0000 7f6061c00700 1 -- start start 2024-03-28T22:24:41.427 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.213+0000 7f6061c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f605c07e860 0x7f605c1a20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:41.427 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.213+0000 7f6061c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f605c080e20 con 0x7f605c07e860 2024-03-28T22:24:41.427 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.213+0000 7f605be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f605c07e860 0x7f605c1a20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:41.427 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.213+0000 7f605be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f605c07e860 0x7f605c1a20d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35276/0 (socket says 172.21.15.43:35276) 2024-03-28T22:24:41.428 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.213+0000 7f605be00700 1 -- 172.21.15.43:0/2466009600 learned_addr learned my addr 172.21.15.43:0/2466009600 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:24:41.428 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.214+0000 7f605be00700 1 -- 172.21.15.43:0/2466009600 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f605c1a2610 con 0x7f605c07e860 2024-03-28T22:24:41.428 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.214+0000 7f605be00700 1 --2- 172.21.15.43:0/2466009600 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f605c07e860 0x7f605c1a20d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f604c011b60 tx=0x7f604c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:41.428 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.214+0000 7f6058c00700 1 -- 172.21.15.43:0/2466009600 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f604c003970 con 0x7f605c07e860 2024-03-28T22:24:41.428 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.214+0000 7f6058c00700 1 -- 172.21.15.43:0/2466009600 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f604c003b10 con 0x7f605c07e860 2024-03-28T22:24:41.428 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.214+0000 7f6061c00700 1 -- 172.21.15.43:0/2466009600 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f605c1a28a0 con 0x7f605c07e860 2024-03-28T22:24:41.428 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.214+0000 7f6058c00700 1 -- 172.21.15.43:0/2466009600 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f604c024610 con 0x7f605c07e860 2024-03-28T22:24:41.428 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.214+0000 7f6061c00700 1 -- 172.21.15.43:0/2466009600 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f605c1a5590 con 0x7f605c07e860 2024-03-28T22:24:41.428 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.215+0000 7f6061c00700 1 -- 172.21.15.43:0/2466009600 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f605c063870 con 0x7f605c07e860 2024-03-28T22:24:41.428 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.215+0000 7f6058c00700 1 -- 172.21.15.43:0/2466009600 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 3) ==== 45811+0+0 (secure 0 0 0) 0x7f604c01a9f0 con 0x7f605c07e860 2024-03-28T22:24:41.428 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.216+0000 7f6058c00700 1 --2- 172.21.15.43:0/2466009600 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f6040038ad0 0x7f604003af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:41.428 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.216+0000 7f6058c00700 1 -- 172.21.15.43:0/2466009600 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f604c011e00 con 0x7f605c07e860 2024-03-28T22:24:41.428 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.216+0000 7f605b400700 1 --2- 172.21.15.43:0/2466009600 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f6040038ad0 0x7f604003af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:41.428 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.217+0000 7f605b400700 1 --2- 172.21.15.43:0/2466009600 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f6040038ad0 0x7f604003af90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6048006fd0 tx=0x7f6048006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:41.429 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.222+0000 7f6058c00700 1 -- 172.21.15.43:0/2466009600 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f604c024c40 con 0x7f605c07e860 2024-03-28T22:24:41.429 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.537+0000 7f6061c00700 1 -- 172.21.15.43:0/2466009600 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f605c02ddf0 con 0x7f605c07e860 2024-03-28T22:24:41.429 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.538+0000 7f6058c00700 1 -- 172.21.15.43:0/2466009600 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f604c024e20 con 0x7f605c07e860 2024-03-28T22:24:41.429 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.540+0000 7f6061c00700 1 -- 172.21.15.43:0/2466009600 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f6040038ad0 msgr2=0x7f604003af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:41.429 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.540+0000 7f6061c00700 1 --2- 172.21.15.43:0/2466009600 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f6040038ad0 0x7f604003af90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6048006fd0 tx=0x7f6048006e40 comp rx=0 tx=0).stop 2024-03-28T22:24:41.429 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.540+0000 7f6061c00700 1 -- 172.21.15.43:0/2466009600 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f605c07e860 msgr2=0x7f605c1a20d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:41.429 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.540+0000 7f6061c00700 1 --2- 172.21.15.43:0/2466009600 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f605c07e860 0x7f605c1a20d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f604c011b60 tx=0x7f604c011c40 comp rx=0 tx=0).stop 2024-03-28T22:24:41.429 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.541+0000 7f6061c00700 1 -- 172.21.15.43:0/2466009600 shutdown_connections 2024-03-28T22:24:41.429 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.541+0000 7f6061c00700 1 --2- 172.21.15.43:0/2466009600 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f6040038ad0 0x7f604003af90 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:41.429 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.541+0000 7f6061c00700 1 --2- 172.21.15.43:0/2466009600 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f605c07e860 0x7f605c1a20d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:41.429 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.541+0000 7f6061c00700 1 -- 172.21.15.43:0/2466009600 >> 172.21.15.43:0/2466009600 conn(0x7f605c07d120 msgr2=0x7f605c198d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:24:41.429 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.541+0000 7f6061c00700 1 -- 172.21.15.43:0/2466009600 shutdown_connections 2024-03-28T22:24:41.429 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:40.541+0000 7f6061c00700 1 -- 172.21.15.43:0/2466009600 wait complete. 2024-03-28T22:24:41.429 INFO:teuthology.orchestra.run.smithi043.stdout:mgr is available 2024-03-28T22:24:41.993 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:41 smithi043 ceph-mon[31823]: mgrmap e4: x(active, since 2s) 2024-03-28T22:24:42.886 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:42 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/179651950' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-28T22:24:42.903 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-03-28T22:24:42.904 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout [global] 2024-03-28T22:24:42.904 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout fsid = c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:24:42.904 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-28T22:24:42.904 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-28T22:24:42.904 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-28T22:24:42.904 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-28T22:24:42.904 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-03-28T22:24:42.904 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-28T22:24:42.904 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-28T22:24:42.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-03-28T22:24:42.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout [osd] 2024-03-28T22:24:42.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-28T22:24:42.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-28T22:24:42.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-28T22:24:42.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-28T22:24:42.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-28T22:24:42.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.992+0000 7f8c55800700 1 Processor -- start 2024-03-28T22:24:42.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.993+0000 7f8c55800700 1 -- start start 2024-03-28T22:24:42.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.993+0000 7f8c55800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8c5007de30 0x7f8c5007c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:42.905 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.993+0000 7f8c55800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8c5007c820 con 0x7f8c5007de30 2024-03-28T22:24:42.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.994+0000 7f8c4fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8c5007de30 0x7f8c5007c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:42.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.994+0000 7f8c4fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8c5007de30 0x7f8c5007c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35282/0 (socket says 172.21.15.43:35282) 2024-03-28T22:24:42.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.994+0000 7f8c4fe00700 1 -- 172.21.15.43:0/837370315 learned_addr learned my addr 172.21.15.43:0/837370315 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:24:42.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.994+0000 7f8c4fe00700 1 -- 172.21.15.43:0/837370315 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c5007c9a0 con 0x7f8c5007de30 2024-03-28T22:24:42.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.995+0000 7f8c4fe00700 1 --2- 172.21.15.43:0/837370315 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8c5007de30 0x7f8c5007c2e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8c400098a0 tx=0x7f8c40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e305e4c7a3dced3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:42.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.995+0000 7f8c4f400700 1 -- 172.21.15.43:0/837370315 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8c40004160 con 0x7f8c5007de30 2024-03-28T22:24:42.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.995+0000 7f8c4f400700 1 -- 172.21.15.43:0/837370315 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8c40004300 con 0x7f8c5007de30 2024-03-28T22:24:42.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.995+0000 7f8c4f400700 1 -- 172.21.15.43:0/837370315 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8c40011b50 con 0x7f8c5007de30 2024-03-28T22:24:42.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.996+0000 7f8c55800700 1 -- 172.21.15.43:0/837370315 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8c5007de30 msgr2=0x7f8c5007c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:42.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.996+0000 7f8c55800700 1 --2- 172.21.15.43:0/837370315 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8c5007de30 0x7f8c5007c2e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8c400098a0 tx=0x7f8c40009bb0 comp rx=0 tx=0).stop 2024-03-28T22:24:42.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.996+0000 7f8c55800700 1 -- 172.21.15.43:0/837370315 shutdown_connections 2024-03-28T22:24:42.906 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.996+0000 7f8c55800700 1 --2- 172.21.15.43:0/837370315 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8c5007de30 0x7f8c5007c2e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:42.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.996+0000 7f8c55800700 1 -- 172.21.15.43:0/837370315 >> 172.21.15.43:0/837370315 conn(0x7f8c501030d0 msgr2=0x7f8c50105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:24:42.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.996+0000 7f8c55800700 1 -- 172.21.15.43:0/837370315 shutdown_connections 2024-03-28T22:24:42.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.996+0000 7f8c55800700 1 -- 172.21.15.43:0/837370315 wait complete. 2024-03-28T22:24:42.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.997+0000 7f8c55800700 1 Processor -- start 2024-03-28T22:24:42.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.998+0000 7f8c55800700 1 -- start start 2024-03-28T22:24:42.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.998+0000 7f8c55800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8c5007de30 0x7f8c5019dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:42.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.998+0000 7f8c55800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8c5010e100 con 0x7f8c5007de30 2024-03-28T22:24:42.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.998+0000 7f8c4fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8c5007de30 0x7f8c5019dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:42.907 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.998+0000 7f8c4fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8c5007de30 0x7f8c5019dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35294/0 (socket says 172.21.15.43:35294) 2024-03-28T22:24:42.910 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.998+0000 7f8c4fe00700 1 -- 172.21.15.43:0/179651950 learned_addr learned my addr 172.21.15.43:0/179651950 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:24:42.910 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.999+0000 7f8c4fe00700 1 -- 172.21.15.43:0/179651950 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c5019e220 con 0x7f8c5007de30 2024-03-28T22:24:42.910 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.999+0000 7f8c4fe00700 1 --2- 172.21.15.43:0/179651950 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8c5007de30 0x7f8c5019dce0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8c40011670 tx=0x7f8c40011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:42.910 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.999+0000 7f8c4d600700 1 -- 172.21.15.43:0/179651950 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8c400039c0 con 0x7f8c5007de30 2024-03-28T22:24:42.910 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.999+0000 7f8c4d600700 1 -- 172.21.15.43:0/179651950 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8c40011e50 con 0x7f8c5007de30 2024-03-28T22:24:42.911 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.999+0000 7f8c4d600700 1 -- 172.21.15.43:0/179651950 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8c4001a5c0 con 0x7f8c5007de30 2024-03-28T22:24:42.911 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.999+0000 7f8c55800700 1 -- 172.21.15.43:0/179651950 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c5019e4b0 con 0x7f8c5007de30 2024-03-28T22:24:42.911 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:41.999+0000 7f8c55800700 1 -- 172.21.15.43:0/179651950 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c501a1190 con 0x7f8c5007de30 2024-03-28T22:24:42.911 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:42.000+0000 7f8c4d600700 1 -- 172.21.15.43:0/179651950 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 4) ==== 45917+0+0 (secure 0 0 0) 0x7f8c40024430 con 0x7f8c5007de30 2024-03-28T22:24:42.911 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:42.001+0000 7f8c55800700 1 -- 172.21.15.43:0/179651950 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c50063870 con 0x7f8c5007de30 2024-03-28T22:24:42.911 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:42.001+0000 7f8c4d600700 1 --2- 172.21.15.43:0/179651950 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f8c34038ba0 0x7f8c3403b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:42.911 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:42.001+0000 7f8c4d600700 1 -- 172.21.15.43:0/179651950 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8c4004cdd0 con 0x7f8c5007de30 2024-03-28T22:24:42.911 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:42.001+0000 7f8c47e00700 1 --2- 172.21.15.43:0/179651950 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f8c34038ba0 0x7f8c3403b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:42.911 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:42.002+0000 7f8c47e00700 1 --2- 172.21.15.43:0/179651950 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f8c34038ba0 0x7f8c3403b060 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8c38006fd0 tx=0x7f8c38006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:42.911 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:42.007+0000 7f8c4d600700 1 -- 172.21.15.43:0/179651950 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8c4002d910 con 0x7f8c5007de30 2024-03-28T22:24:42.911 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:42.220+0000 7f8c55800700 1 -- 172.21.15.43:0/179651950 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f8c501a1360 con 0x7f8c5007de30 2024-03-28T22:24:42.911 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:42.222+0000 7f8c4d600700 1 -- 172.21.15.43:0/179651950 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7f8c501a1360 con 0x7f8c5007de30 2024-03-28T22:24:42.911 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:42.224+0000 7f8c55800700 1 -- 172.21.15.43:0/179651950 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f8c34038ba0 msgr2=0x7f8c3403b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:42.911 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:42.224+0000 7f8c55800700 1 --2- 172.21.15.43:0/179651950 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f8c34038ba0 0x7f8c3403b060 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8c38006fd0 tx=0x7f8c38006e40 comp rx=0 tx=0).stop 2024-03-28T22:24:42.911 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:42.224+0000 7f8c55800700 1 -- 172.21.15.43:0/179651950 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8c5007de30 msgr2=0x7f8c5019dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:42.912 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:42.224+0000 7f8c55800700 1 --2- 172.21.15.43:0/179651950 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8c5007de30 0x7f8c5019dce0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8c40011670 tx=0x7f8c40011b60 comp rx=0 tx=0).stop 2024-03-28T22:24:42.912 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:42.225+0000 7f8c55800700 1 -- 172.21.15.43:0/179651950 shutdown_connections 2024-03-28T22:24:42.912 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:42.225+0000 7f8c55800700 1 --2- 172.21.15.43:0/179651950 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f8c34038ba0 0x7f8c3403b060 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:42.912 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:42.225+0000 7f8c55800700 1 --2- 172.21.15.43:0/179651950 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8c5007de30 0x7f8c5019dce0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:42.912 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:42.225+0000 7f8c55800700 1 -- 172.21.15.43:0/179651950 >> 172.21.15.43:0/179651950 conn(0x7f8c501030d0 msgr2=0x7f8c50103c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:24:42.912 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:42.225+0000 7f8c55800700 1 -- 172.21.15.43:0/179651950 shutdown_connections 2024-03-28T22:24:42.912 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:42.225+0000 7f8c55800700 1 -- 172.21.15.43:0/179651950 wait complete. 2024-03-28T22:24:42.912 INFO:teuthology.orchestra.run.smithi043.stdout:Enabling cephadm module... 2024-03-28T22:24:44.190 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:43 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1847064442' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-28T22:24:44.190 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:43 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: ignoring --setuser ceph since I am not root 2024-03-28T22:24:44.190 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:43 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: ignoring --setgroup ceph since I am not root 2024-03-28T22:24:44.538 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.479+0000 7f90b5c00700 1 Processor -- start 2024-03-28T22:24:44.538 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.480+0000 7f90b5c00700 1 -- start start 2024-03-28T22:24:44.538 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.480+0000 7f90b5c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90b010ecd0 0x7f90b01110c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:44.544 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.480+0000 7f90b5c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f90b0082500 con 0x7f90b010ecd0 2024-03-28T22:24:44.544 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.481+0000 7f90aea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90b010ecd0 0x7f90b01110c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:44.544 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.481+0000 7f90aea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90b010ecd0 0x7f90b01110c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35298/0 (socket says 172.21.15.43:35298) 2024-03-28T22:24:44.544 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.481+0000 7f90aea00700 1 -- 172.21.15.43:0/1126444506 learned_addr learned my addr 172.21.15.43:0/1126444506 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:24:44.544 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.481+0000 7f90aea00700 1 -- 172.21.15.43:0/1126444506 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90b0111600 con 0x7f90b010ecd0 2024-03-28T22:24:44.544 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.482+0000 7f90aea00700 1 --2- 172.21.15.43:0/1126444506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90b010ecd0 0x7f90b01110c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f90a00098a0 tx=0x7f90a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79ee85b14d5e3f20 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:44.544 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.482+0000 7f90ad600700 1 -- 172.21.15.43:0/1126444506 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f90a0004160 con 0x7f90b010ecd0 2024-03-28T22:24:44.545 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.482+0000 7f90ad600700 1 -- 172.21.15.43:0/1126444506 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f90a0004300 con 0x7f90b010ecd0 2024-03-28T22:24:44.545 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.482+0000 7f90ad600700 1 -- 172.21.15.43:0/1126444506 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f90a0011b50 con 0x7f90b010ecd0 2024-03-28T22:24:44.545 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.483+0000 7f90b5c00700 1 -- 172.21.15.43:0/1126444506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90b010ecd0 msgr2=0x7f90b01110c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:44.545 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.483+0000 7f90b5c00700 1 --2- 172.21.15.43:0/1126444506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90b010ecd0 0x7f90b01110c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f90a00098a0 tx=0x7f90a0009bb0 comp rx=0 tx=0).stop 2024-03-28T22:24:44.545 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.483+0000 7f90b5c00700 1 -- 172.21.15.43:0/1126444506 shutdown_connections 2024-03-28T22:24:44.545 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.483+0000 7f90b5c00700 1 --2- 172.21.15.43:0/1126444506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90b010ecd0 0x7f90b01110c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:44.545 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.483+0000 7f90b5c00700 1 -- 172.21.15.43:0/1126444506 >> 172.21.15.43:0/1126444506 conn(0x7f90b007d440 msgr2=0x7f90b007f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:24:44.545 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.483+0000 7f90b5c00700 1 -- 172.21.15.43:0/1126444506 shutdown_connections 2024-03-28T22:24:44.545 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.483+0000 7f90b5c00700 1 -- 172.21.15.43:0/1126444506 wait complete. 2024-03-28T22:24:44.545 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.484+0000 7f90b5c00700 1 Processor -- start 2024-03-28T22:24:44.545 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.484+0000 7f90b5c00700 1 -- start start 2024-03-28T22:24:44.545 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.485+0000 7f90b5c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90b010ecd0 0x7f90b01a21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:44.545 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.485+0000 7f90b5c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f90b0112680 con 0x7f90b010ecd0 2024-03-28T22:24:44.545 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.485+0000 7f90aea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90b010ecd0 0x7f90b01a21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:44.546 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.485+0000 7f90aea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90b010ecd0 0x7f90b01a21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35308/0 (socket says 172.21.15.43:35308) 2024-03-28T22:24:44.546 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.485+0000 7f90aea00700 1 -- 172.21.15.43:0/1847064442 learned_addr learned my addr 172.21.15.43:0/1847064442 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:24:44.546 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.485+0000 7f90aea00700 1 -- 172.21.15.43:0/1847064442 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90b01a2710 con 0x7f90b010ecd0 2024-03-28T22:24:44.546 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.486+0000 7f90aea00700 1 --2- 172.21.15.43:0/1847064442 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90b010ecd0 0x7f90b01a21d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f90a0011b60 tx=0x7f90a0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:44.546 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.486+0000 7f90a7400700 1 -- 172.21.15.43:0/1847064442 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f90a0003b20 con 0x7f90b010ecd0 2024-03-28T22:24:44.546 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.486+0000 7f90a7400700 1 -- 172.21.15.43:0/1847064442 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f90a0003cc0 con 0x7f90b010ecd0 2024-03-28T22:24:44.546 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.486+0000 7f90a7400700 1 -- 172.21.15.43:0/1847064442 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f90a0024840 con 0x7f90b010ecd0 2024-03-28T22:24:44.546 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.486+0000 7f90b5c00700 1 -- 172.21.15.43:0/1847064442 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90b01a29a0 con 0x7f90b010ecd0 2024-03-28T22:24:44.546 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.486+0000 7f90b5c00700 1 -- 172.21.15.43:0/1847064442 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90b01a5690 con 0x7f90b010ecd0 2024-03-28T22:24:44.546 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.487+0000 7f90a7400700 1 -- 172.21.15.43:0/1847064442 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 4) ==== 45917+0+0 (secure 0 0 0) 0x7f90a0022020 con 0x7f90b010ecd0 2024-03-28T22:24:44.546 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.487+0000 7f90b5c00700 1 -- 172.21.15.43:0/1847064442 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90b0063870 con 0x7f90b010ecd0 2024-03-28T22:24:44.546 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.488+0000 7f90a7400700 1 --2- 172.21.15.43:0/1847064442 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f9094038ba0 0x7f909403b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:44.546 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.488+0000 7f90a7400700 1 -- 172.21.15.43:0/1847064442 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f90a0011e00 con 0x7f90b010ecd0 2024-03-28T22:24:44.547 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.488+0000 7f90ae000700 1 --2- 172.21.15.43:0/1847064442 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f9094038ba0 0x7f909403b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:44.547 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.489+0000 7f90ae000700 1 --2- 172.21.15.43:0/1847064442 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f9094038ba0 0x7f909403b060 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f909c006fd0 tx=0x7f909c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:44.547 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.494+0000 7f90a7400700 1 -- 172.21.15.43:0/1847064442 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f90a000b3e0 con 0x7f90b010ecd0 2024-03-28T22:24:44.547 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.776+0000 7f90b5c00700 1 -- 172.21.15.43:0/1847064442 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f90b01a5860 con 0x7f90b010ecd0 2024-03-28T22:24:44.547 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.830+0000 7f90a7400700 1 -- 172.21.15.43:0/1847064442 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f90b01a5860 con 0x7f90b010ecd0 2024-03-28T22:24:44.547 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.833+0000 7f90a7400700 1 -- 172.21.15.43:0/1847064442 <== mon.0 v2:172.21.15.43:3300/0 8 ==== mgrmap(e 5) ==== 45928+0+0 (secure 0 0 0) 0x7f90a002c350 con 0x7f90b010ecd0 2024-03-28T22:24:44.547 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.835+0000 7f90b5c00700 1 -- 172.21.15.43:0/1847064442 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f9094038ba0 msgr2=0x7f909403b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:44.547 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.835+0000 7f90b5c00700 1 --2- 172.21.15.43:0/1847064442 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f9094038ba0 0x7f909403b060 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f909c006fd0 tx=0x7f909c006e40 comp rx=0 tx=0).stop 2024-03-28T22:24:44.547 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.835+0000 7f90b5c00700 1 -- 172.21.15.43:0/1847064442 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90b010ecd0 msgr2=0x7f90b01a21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:44.547 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.835+0000 7f90b5c00700 1 --2- 172.21.15.43:0/1847064442 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90b010ecd0 0x7f90b01a21d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f90a0011b60 tx=0x7f90a0011c40 comp rx=0 tx=0).stop 2024-03-28T22:24:44.547 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.835+0000 7f90b5c00700 1 -- 172.21.15.43:0/1847064442 shutdown_connections 2024-03-28T22:24:44.547 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.835+0000 7f90b5c00700 1 --2- 172.21.15.43:0/1847064442 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f9094038ba0 0x7f909403b060 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:44.547 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.835+0000 7f90b5c00700 1 --2- 172.21.15.43:0/1847064442 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90b010ecd0 0x7f90b01a21d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:44.547 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.835+0000 7f90b5c00700 1 -- 172.21.15.43:0/1847064442 >> 172.21.15.43:0/1847064442 conn(0x7f90b007d440 msgr2=0x7f90b007dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:24:44.548 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.836+0000 7f90b5c00700 1 -- 172.21.15.43:0/1847064442 shutdown_connections 2024-03-28T22:24:44.548 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:43.836+0000 7f90b5c00700 1 -- 172.21.15.43:0/1847064442 wait complete. 2024-03-28T22:24:44.791 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:44 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:44.667+0000 7fce047d2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T22:24:45.094 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:44 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1847064442' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-28T22:24:45.094 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:44 smithi043 ceph-mon[31823]: mgrmap e5: x(active, since 5s) 2024-03-28T22:24:45.095 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:44 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:44.794+0000 7fce047d2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T22:24:45.748 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:45 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:45.623+0000 7fce047d2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T22:24:46.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:45 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1742126044' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-28T22:24:46.017 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:45 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:45.753+0000 7fce047d2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T22:24:46.017 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:45 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:45.937+0000 7fce047d2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T22:24:46.083 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout { 2024-03-28T22:24:46.084 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-28T22:24:46.084 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T22:24:46.084 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-28T22:24:46.084 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-28T22:24:46.084 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout } 2024-03-28T22:24:46.084 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.093+0000 7fc912600700 1 Processor -- start 2024-03-28T22:24:46.084 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.094+0000 7fc912600700 1 -- start start 2024-03-28T22:24:46.084 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.094+0000 7fc912600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc90c108660 0x7fc90c10aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:46.084 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.094+0000 7fc912600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc90c07bdf0 con 0x7fc90c108660 2024-03-28T22:24:46.085 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.095+0000 7fc911200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc90c108660 0x7fc90c10aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:46.085 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.095+0000 7fc911200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc90c108660 0x7fc90c10aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41276/0 (socket says 172.21.15.43:41276) 2024-03-28T22:24:46.085 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.095+0000 7fc911200700 1 -- 172.21.15.43:0/1885409617 learned_addr learned my addr 172.21.15.43:0/1885409617 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:24:46.085 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.095+0000 7fc911200700 1 -- 172.21.15.43:0/1885409617 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc90c10af90 con 0x7fc90c108660 2024-03-28T22:24:46.085 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.096+0000 7fc911200700 1 --2- 172.21.15.43:0/1885409617 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc90c108660 0x7fc90c10aa50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc9000098a0 tx=0x7fc900009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcb7b040cc4da8ae server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:46.085 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.096+0000 7fc90b400700 1 -- 172.21.15.43:0/1885409617 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc900004160 con 0x7fc90c108660 2024-03-28T22:24:46.085 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.096+0000 7fc90b400700 1 -- 172.21.15.43:0/1885409617 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc900004300 con 0x7fc90c108660 2024-03-28T22:24:46.085 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.096+0000 7fc90b400700 1 -- 172.21.15.43:0/1885409617 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc900011b50 con 0x7fc90c108660 2024-03-28T22:24:46.085 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.097+0000 7fc912600700 1 -- 172.21.15.43:0/1885409617 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc90c108660 msgr2=0x7fc90c10aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:46.085 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.097+0000 7fc912600700 1 --2- 172.21.15.43:0/1885409617 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc90c108660 0x7fc90c10aa50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc9000098a0 tx=0x7fc900009bb0 comp rx=0 tx=0).stop 2024-03-28T22:24:46.085 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.097+0000 7fc912600700 1 -- 172.21.15.43:0/1885409617 shutdown_connections 2024-03-28T22:24:46.085 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.097+0000 7fc912600700 1 --2- 172.21.15.43:0/1885409617 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc90c108660 0x7fc90c10aa50 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:46.085 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.097+0000 7fc912600700 1 -- 172.21.15.43:0/1885409617 >> 172.21.15.43:0/1885409617 conn(0x7fc90c101fd0 msgr2=0x7fc90c104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:24:46.085 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.097+0000 7fc912600700 1 -- 172.21.15.43:0/1885409617 shutdown_connections 2024-03-28T22:24:46.086 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.097+0000 7fc912600700 1 -- 172.21.15.43:0/1885409617 wait complete. 2024-03-28T22:24:46.086 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.098+0000 7fc912600700 1 Processor -- start 2024-03-28T22:24:46.086 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.098+0000 7fc912600700 1 -- start start 2024-03-28T22:24:46.086 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.099+0000 7fc912600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc90c108660 0x7fc90c19baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:46.086 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.099+0000 7fc912600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc90c10bf00 con 0x7fc90c108660 2024-03-28T22:24:46.086 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.099+0000 7fc911200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc90c108660 0x7fc90c19baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:46.086 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.099+0000 7fc911200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc90c108660 0x7fc90c19baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41290/0 (socket says 172.21.15.43:41290) 2024-03-28T22:24:46.086 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.099+0000 7fc911200700 1 -- 172.21.15.43:0/1742126044 learned_addr learned my addr 172.21.15.43:0/1742126044 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:24:46.086 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.099+0000 7fc911200700 1 -- 172.21.15.43:0/1742126044 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc90c19bfe0 con 0x7fc90c108660 2024-03-28T22:24:46.086 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.100+0000 7fc911200700 1 --2- 172.21.15.43:0/1742126044 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc90c108660 0x7fc90c19baa0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc900011b60 tx=0x7fc900011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:46.086 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.100+0000 7fc909600700 1 -- 172.21.15.43:0/1742126044 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc900003b20 con 0x7fc90c108660 2024-03-28T22:24:46.086 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.100+0000 7fc909600700 1 -- 172.21.15.43:0/1742126044 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc900003cc0 con 0x7fc90c108660 2024-03-28T22:24:46.086 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.100+0000 7fc909600700 1 -- 172.21.15.43:0/1742126044 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc900024840 con 0x7fc90c108660 2024-03-28T22:24:46.086 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.100+0000 7fc912600700 1 -- 172.21.15.43:0/1742126044 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc90c19c270 con 0x7fc90c108660 2024-03-28T22:24:46.087 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.100+0000 7fc912600700 1 -- 172.21.15.43:0/1742126044 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc90c19ef60 con 0x7fc90c108660 2024-03-28T22:24:46.087 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.102+0000 7fc909600700 1 -- 172.21.15.43:0/1742126044 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 45928+0+0 (secure 0 0 0) 0x7fc900022020 con 0x7fc90c108660 2024-03-28T22:24:46.087 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.102+0000 7fc8fea00700 1 -- 172.21.15.43:0/1742126044 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8e8005330 con 0x7fc90c108660 2024-03-28T22:24:46.087 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.102+0000 7fc909600700 1 --2- 172.21.15.43:0/1742126044 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7fc8f4038c50 0x7fc8f403b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:46.087 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.102+0000 7fc909600700 1 -- 172.21.15.43:0/1742126044 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc900011e00 con 0x7fc90c108660 2024-03-28T22:24:46.087 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.102+0000 7fc90be00700 1 -- 172.21.15.43:0/1742126044 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7fc8f4038c50 msgr2=0x7fc8f403b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/1532177494 2024-03-28T22:24:46.087 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.102+0000 7fc90be00700 1 --2- 172.21.15.43:0/1742126044 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7fc8f4038c50 0x7fc8f403b110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:24:46.087 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.108+0000 7fc909600700 1 -- 172.21.15.43:0/1742126044 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc900016260 con 0x7fc90c108660 2024-03-28T22:24:46.087 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.303+0000 7fc90be00700 1 -- 172.21.15.43:0/1742126044 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7fc8f4038c50 msgr2=0x7fc8f403b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/1532177494 2024-03-28T22:24:46.087 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.303+0000 7fc90be00700 1 --2- 172.21.15.43:0/1742126044 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7fc8f4038c50 0x7fc8f403b110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:24:46.087 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.387+0000 7fc8fea00700 1 -- 172.21.15.43:0/1742126044 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fc8e8005e30 con 0x7fc90c108660 2024-03-28T22:24:46.087 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.388+0000 7fc909600700 1 -- 172.21.15.43:0/1742126044 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7fc90001a950 con 0x7fc90c108660 2024-03-28T22:24:46.087 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.392+0000 7fc8fea00700 1 -- 172.21.15.43:0/1742126044 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7fc8f4038c50 msgr2=0x7fc8f403b110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T22:24:46.087 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.392+0000 7fc8fea00700 1 --2- 172.21.15.43:0/1742126044 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7fc8f4038c50 0x7fc8f403b110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:46.088 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.392+0000 7fc8fea00700 1 -- 172.21.15.43:0/1742126044 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc90c108660 msgr2=0x7fc90c19baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:46.088 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.392+0000 7fc8fea00700 1 --2- 172.21.15.43:0/1742126044 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc90c108660 0x7fc90c19baa0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc900011b60 tx=0x7fc900011c40 comp rx=0 tx=0).stop 2024-03-28T22:24:46.088 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.392+0000 7fc8fea00700 1 -- 172.21.15.43:0/1742126044 shutdown_connections 2024-03-28T22:24:46.088 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.392+0000 7fc8fea00700 1 --2- 172.21.15.43:0/1742126044 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7fc8f4038c50 0x7fc8f403b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:46.088 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.392+0000 7fc8fea00700 1 --2- 172.21.15.43:0/1742126044 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc90c108660 0x7fc90c19baa0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:46.088 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.392+0000 7fc8fea00700 1 -- 172.21.15.43:0/1742126044 >> 172.21.15.43:0/1742126044 conn(0x7fc90c101fd0 msgr2=0x7fc90c102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:24:46.088 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.393+0000 7fc8fea00700 1 -- 172.21.15.43:0/1742126044 shutdown_connections 2024-03-28T22:24:46.088 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:45.393+0000 7fc8fea00700 1 -- 172.21.15.43:0/1742126044 wait complete. 2024-03-28T22:24:46.088 INFO:teuthology.orchestra.run.smithi043.stdout:Waiting for the mgr to restart... 2024-03-28T22:24:46.088 INFO:teuthology.orchestra.run.smithi043.stdout:Waiting for mgr epoch 5... 2024-03-28T22:24:46.368 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:46 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:46.066+0000 7fce047d2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T22:24:46.368 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:46 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:46.192+0000 7fce047d2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T22:24:46.657 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:46 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:46.542+0000 7fce047d2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T22:24:46.939 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:46 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:46.694+0000 7fce047d2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T22:24:46.939 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:46 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:46.810+0000 7fce047d2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T22:24:46.939 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:46 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:46.938+0000 7fce047d2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T22:24:47.267 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:47 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:47.163+0000 7fce047d2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T22:24:48.767 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:48 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:48.477+0000 7fce047d2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T22:24:48.767 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:48 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:48.654+0000 7fce047d2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T22:24:49.468 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:49 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:49.082+0000 7fce047d2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T22:24:49.468 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:49 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:49.204+0000 7fce047d2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T22:24:49.468 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:49 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:49.467+0000 7fce047d2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T22:24:50.017 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:49 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:49.720+0000 7fce047d2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T22:24:50.394 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:50 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:50.109+0000 7fce047d2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T22:24:50.699 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:50 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:50.437+0000 7fce047d2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T22:24:51.017 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:50 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:50.759+0000 7fce047d2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T22:24:52.017 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:51 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:51.831+0000 7fce047d2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T22:24:52.875 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:52 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:52.550+0000 7fce047d2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T22:24:52.875 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:52 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:52.684+0000 7fce047d2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T22:24:53.267 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:24:52 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:24:52.976+0000 7fce047d2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T22:24:53.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:53 smithi043 ceph-mon[31823]: Active manager daemon x restarted 2024-03-28T22:24:53.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:53 smithi043 ceph-mon[31823]: Activating manager daemon x 2024-03-28T22:24:53.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:53 smithi043 ceph-mon[31823]: osdmap e2: 0 total, 0 up, 0 in 2024-03-28T22:24:53.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:53 smithi043 ceph-mon[31823]: mgrmap e6: x(active, starting, since 0.00478457s) 2024-03-28T22:24:53.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:53 smithi043 ceph-mon[31823]: from='mgr.14120 172.21.15.43:0/1538160804' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:24:53.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:53 smithi043 ceph-mon[31823]: from='mgr.14120 172.21.15.43:0/1538160804' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T22:24:53.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:53 smithi043 ceph-mon[31823]: from='mgr.14120 172.21.15.43:0/1538160804' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T22:24:53.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:53 smithi043 ceph-mon[31823]: from='mgr.14120 172.21.15.43:0/1538160804' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T22:24:53.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:53 smithi043 ceph-mon[31823]: from='mgr.14120 172.21.15.43:0/1538160804' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T22:24:53.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:53 smithi043 ceph-mon[31823]: Manager daemon x is now available 2024-03-28T22:24:54.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:54 smithi043 ceph-mon[31823]: Found migration_current of "None". Setting to last migration. 2024-03-28T22:24:54.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:54 smithi043 ceph-mon[31823]: from='mgr.14120 172.21.15.43:0/1538160804' entity='mgr.x' 2024-03-28T22:24:54.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:54 smithi043 ceph-mon[31823]: from='mgr.14120 172.21.15.43:0/1538160804' entity='mgr.x' 2024-03-28T22:24:54.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:54 smithi043 ceph-mon[31823]: from='mgr.14120 172.21.15.43:0/1538160804' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:24:54.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:54 smithi043 ceph-mon[31823]: from='mgr.14120 172.21.15.43:0/1538160804' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:24:54.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:54 smithi043 ceph-mon[31823]: from='mgr.14120 172.21.15.43:0/1538160804' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:24:54.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:54 smithi043 ceph-mon[31823]: from='mgr.14120 172.21.15.43:0/1538160804' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T22:24:54.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:54 smithi043 ceph-mon[31823]: from='mgr.14120 172.21.15.43:0/1538160804' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T22:24:54.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:54 smithi043 ceph-mon[31823]: from='mgr.14120 172.21.15.43:0/1538160804' entity='mgr.x' 2024-03-28T22:24:54.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:54 smithi043 ceph-mon[31823]: from='mgr.14120 172.21.15.43:0/1538160804' entity='mgr.x' 2024-03-28T22:24:54.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:54 smithi043 ceph-mon[31823]: mgrmap e7: x(active, since 1.01687s) 2024-03-28T22:24:54.843 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout { 2024-03-28T22:24:54.843 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-28T22:24:54.844 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-28T22:24:54.844 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout } 2024-03-28T22:24:54.844 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.656+0000 7f8ea3a00700 1 Processor -- start 2024-03-28T22:24:54.844 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.657+0000 7f8ea3a00700 1 -- start start 2024-03-28T22:24:54.851 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.657+0000 7f8ea3a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e9c07dd70 0x7f8e9c07c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:54.851 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.657+0000 7f8ea3a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8e9c07c820 con 0x7f8e9c07dd70 2024-03-28T22:24:54.852 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.657+0000 7f8ea1000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e9c07dd70 0x7f8e9c07c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:54.852 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.658+0000 7f8ea1000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e9c07dd70 0x7f8e9c07c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41306/0 (socket says 172.21.15.43:41306) 2024-03-28T22:24:54.852 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.658+0000 7f8ea1000700 1 -- 172.21.15.43:0/4145045323 learned_addr learned my addr 172.21.15.43:0/4145045323 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:24:54.852 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.658+0000 7f8ea1000700 1 -- 172.21.15.43:0/4145045323 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e9c07c9a0 con 0x7f8e9c07dd70 2024-03-28T22:24:54.852 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.658+0000 7f8ea1000700 1 --2- 172.21.15.43:0/4145045323 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e9c07dd70 0x7f8e9c07c2e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8e900098a0 tx=0x7f8e90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e69c4b3da3b1f6f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:54.852 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.659+0000 7f8e9b400700 1 -- 172.21.15.43:0/4145045323 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e90004160 con 0x7f8e9c07dd70 2024-03-28T22:24:54.852 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.659+0000 7f8e9b400700 1 -- 172.21.15.43:0/4145045323 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8e90004300 con 0x7f8e9c07dd70 2024-03-28T22:24:54.852 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.659+0000 7f8e9b400700 1 -- 172.21.15.43:0/4145045323 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e90011b50 con 0x7f8e9c07dd70 2024-03-28T22:24:54.852 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.659+0000 7f8ea3a00700 1 -- 172.21.15.43:0/4145045323 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e9c07dd70 msgr2=0x7f8e9c07c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:54.852 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.659+0000 7f8ea3a00700 1 --2- 172.21.15.43:0/4145045323 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e9c07dd70 0x7f8e9c07c2e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8e900098a0 tx=0x7f8e90009bb0 comp rx=0 tx=0).stop 2024-03-28T22:24:54.852 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.659+0000 7f8ea3a00700 1 -- 172.21.15.43:0/4145045323 shutdown_connections 2024-03-28T22:24:54.852 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.659+0000 7f8ea3a00700 1 --2- 172.21.15.43:0/4145045323 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e9c07dd70 0x7f8e9c07c2e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:54.852 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.659+0000 7f8ea3a00700 1 -- 172.21.15.43:0/4145045323 >> 172.21.15.43:0/4145045323 conn(0x7f8e9c103340 msgr2=0x7f8e9c105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:24:54.852 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.660+0000 7f8ea3a00700 1 -- 172.21.15.43:0/4145045323 shutdown_connections 2024-03-28T22:24:54.852 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.660+0000 7f8ea3a00700 1 -- 172.21.15.43:0/4145045323 wait complete. 2024-03-28T22:24:54.852 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.661+0000 7f8ea3a00700 1 Processor -- start 2024-03-28T22:24:54.853 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.661+0000 7f8ea3a00700 1 -- start start 2024-03-28T22:24:54.853 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.662+0000 7f8ea3a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e9c07dd70 0x7f8e9c19de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:54.853 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.662+0000 7f8ea3a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8e9c10e1b0 con 0x7f8e9c07dd70 2024-03-28T22:24:54.853 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.662+0000 7f8ea1000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e9c07dd70 0x7f8e9c19de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:54.853 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.662+0000 7f8ea1000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e9c07dd70 0x7f8e9c19de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41314/0 (socket says 172.21.15.43:41314) 2024-03-28T22:24:54.853 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.662+0000 7f8ea1000700 1 -- 172.21.15.43:0/520501410 learned_addr learned my addr 172.21.15.43:0/520501410 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:24:54.853 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.662+0000 7f8ea1000700 1 -- 172.21.15.43:0/520501410 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e9c19e3b0 con 0x7f8e9c07dd70 2024-03-28T22:24:54.853 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.663+0000 7f8ea1000700 1 --2- 172.21.15.43:0/520501410 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e9c07dd70 0x7f8e9c19de70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8e90011b60 tx=0x7f8e90011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:54.853 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.663+0000 7f8e99600700 1 -- 172.21.15.43:0/520501410 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e90003b20 con 0x7f8e9c07dd70 2024-03-28T22:24:54.853 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.663+0000 7f8e99600700 1 -- 172.21.15.43:0/520501410 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8e90003cc0 con 0x7f8e9c07dd70 2024-03-28T22:24:54.853 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.663+0000 7f8e99600700 1 -- 172.21.15.43:0/520501410 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e90024840 con 0x7f8e9c07dd70 2024-03-28T22:24:54.853 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.663+0000 7f8ea3a00700 1 -- 172.21.15.43:0/520501410 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e9c19e640 con 0x7f8e9c07dd70 2024-03-28T22:24:54.853 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.663+0000 7f8ea3a00700 1 -- 172.21.15.43:0/520501410 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e9c1a1330 con 0x7f8e9c07dd70 2024-03-28T22:24:54.853 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.665+0000 7f8e99600700 1 -- 172.21.15.43:0/520501410 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 45928+0+0 (secure 0 0 0) 0x7f8e90022020 con 0x7f8e9c07dd70 2024-03-28T22:24:54.853 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.665+0000 7f8e99600700 1 --2- 172.21.15.43:0/520501410 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f8e84038bf0 0x7f8e8403b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:54.853 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.665+0000 7f8e99600700 1 -- 172.21.15.43:0/520501410 --> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8e8403b7c0 con 0x7f8e84038bf0 2024-03-28T22:24:54.854 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.665+0000 7f8e99600700 1 -- 172.21.15.43:0/520501410 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8e90011e00 con 0x7f8e9c07dd70 2024-03-28T22:24:54.854 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.665+0000 7f8e9be00700 1 -- 172.21.15.43:0/520501410 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f8e84038bf0 msgr2=0x7f8e8403b0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/1532177494 2024-03-28T22:24:54.854 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.665+0000 7f8e9be00700 1 --2- 172.21.15.43:0/520501410 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f8e84038bf0 0x7f8e8403b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:24:54.854 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.866+0000 7f8e9be00700 1 -- 172.21.15.43:0/520501410 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f8e84038bf0 msgr2=0x7f8e8403b0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/1532177494 2024-03-28T22:24:54.854 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:46.866+0000 7f8e9be00700 1 --2- 172.21.15.43:0/520501410 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f8e84038bf0 0x7f8e8403b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:24:54.854 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:47.267+0000 7f8e9be00700 1 -- 172.21.15.43:0/520501410 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f8e84038bf0 msgr2=0x7f8e8403b0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/1532177494 2024-03-28T22:24:54.854 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:47.267+0000 7f8e9be00700 1 --2- 172.21.15.43:0/520501410 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f8e84038bf0 0x7f8e8403b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:24:54.854 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:48.068+0000 7f8e9be00700 1 -- 172.21.15.43:0/520501410 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f8e84038bf0 msgr2=0x7f8e8403b0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/1532177494 2024-03-28T22:24:54.854 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:48.068+0000 7f8e9be00700 1 --2- 172.21.15.43:0/520501410 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f8e84038bf0 0x7f8e8403b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:24:54.854 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:49.670+0000 7f8e9be00700 1 -- 172.21.15.43:0/520501410 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f8e84038bf0 msgr2=0x7f8e8403b0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/1532177494 2024-03-28T22:24:54.854 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:49.670+0000 7f8e9be00700 1 --2- 172.21.15.43:0/520501410 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f8e84038bf0 0x7f8e8403b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-28T22:24:54.854 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:52.873+0000 7f8e9be00700 1 -- 172.21.15.43:0/520501410 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f8e84038bf0 msgr2=0x7f8e8403b0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/1532177494 2024-03-28T22:24:54.854 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:52.873+0000 7f8e9be00700 1 --2- 172.21.15.43:0/520501410 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f8e84038bf0 0x7f8e8403b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-28T22:24:54.854 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:52.983+0000 7f8e99600700 1 -- 172.21.15.43:0/520501410 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mgrmap(e 6) ==== 45695+0+0 (secure 0 0 0) 0x7f8e90024a40 con 0x7f8e9c07dd70 2024-03-28T22:24:54.855 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:52.984+0000 7f8e99600700 1 -- 172.21.15.43:0/520501410 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f8e84038bf0 msgr2=0x7f8e8403b0b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T22:24:54.855 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:52.984+0000 7f8e99600700 1 --2- 172.21.15.43:0/520501410 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f8e84038bf0 0x7f8e8403b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:54.855 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:53.990+0000 7f8e99600700 1 -- 172.21.15.43:0/520501410 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mgrmap(e 7) ==== 45822+0+0 (secure 0 0 0) 0x7f8e9004e1b0 con 0x7f8e9c07dd70 2024-03-28T22:24:54.855 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:53.991+0000 7f8e99600700 1 --2- 172.21.15.43:0/520501410 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f8e8403c520 0x7f8e8403e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:54.855 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:53.991+0000 7f8e99600700 1 -- 172.21.15.43:0/520501410 --> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8e90001d30 con 0x7f8e8403c520 2024-03-28T22:24:54.855 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:53.993+0000 7f8e9be00700 1 --2- 172.21.15.43:0/520501410 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f8e8403c520 0x7f8e8403e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:54.855 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:53.996+0000 7f8e9be00700 1 --2- 172.21.15.43:0/520501410 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f8e8403c520 0x7f8e8403e910 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f8e88003a10 tx=0x7f8e880092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:54.855 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:53.997+0000 7f8e99600700 1 -- 172.21.15.43:0/520501410 <== mgr.14120 v2:172.21.15.43:6800/3517712438 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f8e90001d30 con 0x7f8e8403c520 2024-03-28T22:24:54.855 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:54.007+0000 7f8ea3a00700 1 -- 172.21.15.43:0/520501410 --> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f8e9c063870 con 0x7f8e8403c520 2024-03-28T22:24:54.855 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:54.007+0000 7f8e99600700 1 -- 172.21.15.43:0/520501410 <== mgr.14120 v2:172.21.15.43:6800/3517712438 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f8e9c063870 con 0x7f8e8403c520 2024-03-28T22:24:54.855 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:54.008+0000 7f8ea3a00700 1 -- 172.21.15.43:0/520501410 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f8e8403c520 msgr2=0x7f8e8403e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:54.855 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:54.008+0000 7f8ea3a00700 1 --2- 172.21.15.43:0/520501410 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f8e8403c520 0x7f8e8403e910 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f8e88003a10 tx=0x7f8e880092b0 comp rx=0 tx=0).stop 2024-03-28T22:24:54.855 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:54.008+0000 7f8ea3a00700 1 -- 172.21.15.43:0/520501410 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e9c07dd70 msgr2=0x7f8e9c19de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:54.855 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:54.008+0000 7f8ea3a00700 1 --2- 172.21.15.43:0/520501410 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e9c07dd70 0x7f8e9c19de70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8e90011b60 tx=0x7f8e90011c40 comp rx=0 tx=0).stop 2024-03-28T22:24:54.855 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:54.008+0000 7f8ea3a00700 1 -- 172.21.15.43:0/520501410 shutdown_connections 2024-03-28T22:24:54.855 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:54.008+0000 7f8ea3a00700 1 --2- 172.21.15.43:0/520501410 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f8e8403c520 0x7f8e8403e910 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:54.856 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:54.008+0000 7f8ea3a00700 1 --2- 172.21.15.43:0/520501410 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e9c07dd70 0x7f8e9c19de70 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:54.856 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:54.008+0000 7f8ea3a00700 1 --2- 172.21.15.43:0/520501410 >> [v2:172.21.15.43:6800/1532177494,v1:172.21.15.43:6801/1532177494] conn(0x7f8e84038bf0 0x7f8e8403b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:54.856 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:54.008+0000 7f8ea3a00700 1 -- 172.21.15.43:0/520501410 >> 172.21.15.43:0/520501410 conn(0x7f8e9c103340 msgr2=0x7f8e9c103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:24:54.856 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:54.008+0000 7f8ea3a00700 1 -- 172.21.15.43:0/520501410 shutdown_connections 2024-03-28T22:24:54.856 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:54.008+0000 7f8ea3a00700 1 -- 172.21.15.43:0/520501410 wait complete. 2024-03-28T22:24:54.856 INFO:teuthology.orchestra.run.smithi043.stdout:mgr epoch 5 is available 2024-03-28T22:24:54.856 INFO:teuthology.orchestra.run.smithi043.stdout:Setting orchestrator backend to cephadm... 2024-03-28T22:24:55.180 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:55 smithi043 ceph-mon[31823]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-28T22:24:55.180 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:55 smithi043 ceph-mon[31823]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-28T22:24:55.180 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:55 smithi043 ceph-mon[31823]: [28/Mar/2024:22:24:54] ENGINE Bus STARTING 2024-03-28T22:24:55.181 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:55 smithi043 ceph-mon[31823]: [28/Mar/2024:22:24:54] ENGINE Serving on http://172.21.15.43:8765 2024-03-28T22:24:55.181 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:55 smithi043 ceph-mon[31823]: [28/Mar/2024:22:24:54] ENGINE Serving on https://172.21.15.43:7150 2024-03-28T22:24:55.181 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:55 smithi043 ceph-mon[31823]: [28/Mar/2024:22:24:54] ENGINE Bus STARTED 2024-03-28T22:24:55.181 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:55 smithi043 ceph-mon[31823]: from='mgr.14120 172.21.15.43:0/1538160804' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:24:56.398 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:56 smithi043 ceph-mon[31823]: mgrmap e8: x(active, since 2s) 2024-03-28T22:24:56.398 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:56 smithi043 ceph-mon[31823]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:24:56.398 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:56 smithi043 ceph-mon[31823]: from='mgr.14120 172.21.15.43:0/1538160804' entity='mgr.x' 2024-03-28T22:24:56.398 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:56 smithi043 ceph-mon[31823]: from='mgr.14120 172.21.15.43:0/1538160804' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:24:56.411 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.518+0000 7fac5cc00700 1 Processor -- start 2024-03-28T22:24:56.411 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.518+0000 7fac5cc00700 1 -- start start 2024-03-28T22:24:56.411 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.519+0000 7fac5cc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac5807dd90 0x7fac5807c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:56.412 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.519+0000 7fac5cc00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fac5807c840 con 0x7fac5807dd90 2024-03-28T22:24:56.412 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.520+0000 7fac55e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac5807dd90 0x7fac5807c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:56.412 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.520+0000 7fac55e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac5807dd90 0x7fac5807c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35552/0 (socket says 172.21.15.43:35552) 2024-03-28T22:24:56.412 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.520+0000 7fac55e00700 1 -- 172.21.15.43:0/2325489719 learned_addr learned my addr 172.21.15.43:0/2325489719 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:24:56.412 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.520+0000 7fac55e00700 1 -- 172.21.15.43:0/2325489719 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac5807c9c0 con 0x7fac5807dd90 2024-03-28T22:24:56.412 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.520+0000 7fac55e00700 1 --2- 172.21.15.43:0/2325489719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac5807dd90 0x7fac5807c300 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fac480098a0 tx=0x7fac48009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6286b7766ad4176a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:56.413 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.521+0000 7fac54a00700 1 -- 172.21.15.43:0/2325489719 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac48004160 con 0x7fac5807dd90 2024-03-28T22:24:56.413 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.521+0000 7fac54a00700 1 -- 172.21.15.43:0/2325489719 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fac48004300 con 0x7fac5807dd90 2024-03-28T22:24:56.413 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.521+0000 7fac54a00700 1 -- 172.21.15.43:0/2325489719 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac48011b50 con 0x7fac5807dd90 2024-03-28T22:24:56.413 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.521+0000 7fac5cc00700 1 -- 172.21.15.43:0/2325489719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac5807dd90 msgr2=0x7fac5807c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:56.413 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.521+0000 7fac5cc00700 1 --2- 172.21.15.43:0/2325489719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac5807dd90 0x7fac5807c300 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fac480098a0 tx=0x7fac48009bb0 comp rx=0 tx=0).stop 2024-03-28T22:24:56.413 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.522+0000 7fac5cc00700 1 -- 172.21.15.43:0/2325489719 shutdown_connections 2024-03-28T22:24:56.414 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.522+0000 7fac5cc00700 1 --2- 172.21.15.43:0/2325489719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac5807dd90 0x7fac5807c300 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:56.414 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.522+0000 7fac5cc00700 1 -- 172.21.15.43:0/2325489719 >> 172.21.15.43:0/2325489719 conn(0x7fac58103340 msgr2=0x7fac58105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:24:56.414 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.522+0000 7fac5cc00700 1 -- 172.21.15.43:0/2325489719 shutdown_connections 2024-03-28T22:24:56.414 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.522+0000 7fac5cc00700 1 -- 172.21.15.43:0/2325489719 wait complete. 2024-03-28T22:24:56.414 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.523+0000 7fac5cc00700 1 Processor -- start 2024-03-28T22:24:56.414 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.523+0000 7fac5cc00700 1 -- start start 2024-03-28T22:24:56.415 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.523+0000 7fac5cc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac5807dd90 0x7fac5819de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:56.415 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.523+0000 7fac5cc00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fac5810e190 con 0x7fac5807dd90 2024-03-28T22:24:56.415 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.523+0000 7fac55e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac5807dd90 0x7fac5819de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:56.415 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.523+0000 7fac55e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac5807dd90 0x7fac5819de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35554/0 (socket says 172.21.15.43:35554) 2024-03-28T22:24:56.415 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.524+0000 7fac55e00700 1 -- 172.21.15.43:0/111996542 learned_addr learned my addr 172.21.15.43:0/111996542 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:24:56.415 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.524+0000 7fac55e00700 1 -- 172.21.15.43:0/111996542 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac5819e3b0 con 0x7fac5807dd90 2024-03-28T22:24:56.415 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.524+0000 7fac55e00700 1 --2- 172.21.15.43:0/111996542 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac5807dd90 0x7fac5819de70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fac48011b60 tx=0x7fac48011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:56.416 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.524+0000 7fac4ea00700 1 -- 172.21.15.43:0/111996542 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac48003b20 con 0x7fac5807dd90 2024-03-28T22:24:56.416 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.525+0000 7fac4ea00700 1 -- 172.21.15.43:0/111996542 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fac48003cc0 con 0x7fac5807dd90 2024-03-28T22:24:56.416 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.525+0000 7fac4ea00700 1 -- 172.21.15.43:0/111996542 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac48024840 con 0x7fac5807dd90 2024-03-28T22:24:56.416 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.525+0000 7fac5cc00700 1 -- 172.21.15.43:0/111996542 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac5819e640 con 0x7fac5807dd90 2024-03-28T22:24:56.416 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.525+0000 7fac5cc00700 1 -- 172.21.15.43:0/111996542 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac581a1330 con 0x7fac5807dd90 2024-03-28T22:24:56.416 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.526+0000 7fac4ea00700 1 -- 172.21.15.43:0/111996542 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7fac48022020 con 0x7fac5807dd90 2024-03-28T22:24:56.416 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.526+0000 7fac5cc00700 1 -- 172.21.15.43:0/111996542 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac58063870 con 0x7fac5807dd90 2024-03-28T22:24:56.417 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.526+0000 7fac4ea00700 1 --2- 172.21.15.43:0/111996542 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7fac3c038bf0 0x7fac3c03b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:56.417 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.527+0000 7fac4ea00700 1 -- 172.21.15.43:0/111996542 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fac48011e00 con 0x7fac5807dd90 2024-03-28T22:24:56.417 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.527+0000 7fac55400700 1 --2- 172.21.15.43:0/111996542 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7fac3c038bf0 0x7fac3c03b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:56.417 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.528+0000 7fac55400700 1 --2- 172.21.15.43:0/111996542 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7fac3c038bf0 0x7fac3c03b0b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fac44006fd0 tx=0x7fac44006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:56.417 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.533+0000 7fac4ea00700 1 -- 172.21.15.43:0/111996542 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fac4800b3e0 con 0x7fac5807dd90 2024-03-28T22:24:56.417 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.776+0000 7fac5cc00700 1 -- 172.21.15.43:0/111996542 --> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fac581045b0 con 0x7fac3c038bf0 2024-03-28T22:24:56.417 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.786+0000 7fac4ea00700 1 -- 172.21.15.43:0/111996542 <== mgr.14120 v2:172.21.15.43:6800/3517712438 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fac581045b0 con 0x7fac3c038bf0 2024-03-28T22:24:56.418 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.789+0000 7fac5cc00700 1 -- 172.21.15.43:0/111996542 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7fac3c038bf0 msgr2=0x7fac3c03b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:56.418 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.789+0000 7fac5cc00700 1 --2- 172.21.15.43:0/111996542 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7fac3c038bf0 0x7fac3c03b0b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fac44006fd0 tx=0x7fac44006e40 comp rx=0 tx=0).stop 2024-03-28T22:24:56.418 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.789+0000 7fac5cc00700 1 -- 172.21.15.43:0/111996542 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac5807dd90 msgr2=0x7fac5819de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:56.418 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.789+0000 7fac5cc00700 1 --2- 172.21.15.43:0/111996542 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac5807dd90 0x7fac5819de70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fac48011b60 tx=0x7fac48011c40 comp rx=0 tx=0).stop 2024-03-28T22:24:56.418 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.789+0000 7fac5cc00700 1 -- 172.21.15.43:0/111996542 shutdown_connections 2024-03-28T22:24:56.418 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.789+0000 7fac5cc00700 1 --2- 172.21.15.43:0/111996542 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7fac3c038bf0 0x7fac3c03b0b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:56.418 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.789+0000 7fac5cc00700 1 --2- 172.21.15.43:0/111996542 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac5807dd90 0x7fac5819de70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:56.418 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.789+0000 7fac5cc00700 1 -- 172.21.15.43:0/111996542 >> 172.21.15.43:0/111996542 conn(0x7fac58103340 msgr2=0x7fac58103ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:24:56.419 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.789+0000 7fac5cc00700 1 -- 172.21.15.43:0/111996542 shutdown_connections 2024-03-28T22:24:56.419 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:55.790+0000 7fac5cc00700 1 -- 172.21.15.43:0/111996542 wait complete. 2024-03-28T22:24:57.911 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-28T22:24:57.911 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:56.997+0000 7f6323e00700 1 Processor -- start 2024-03-28T22:24:57.912 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:56.997+0000 7f6323e00700 1 -- start start 2024-03-28T22:24:57.912 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:56.997+0000 7f6323e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f631c07ddb0 0x7f631c07c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:57.912 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:56.997+0000 7f6323e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f631c07c860 con 0x7f631c07ddb0 2024-03-28T22:24:57.912 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:56.998+0000 7f6321400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f631c07ddb0 0x7f631c07c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:57.912 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:56.998+0000 7f6321400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f631c07ddb0 0x7f631c07c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35566/0 (socket says 172.21.15.43:35566) 2024-03-28T22:24:57.912 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:56.998+0000 7f6321400700 1 -- 172.21.15.43:0/3756948981 learned_addr learned my addr 172.21.15.43:0/3756948981 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:24:57.912 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:56.998+0000 7f6321400700 1 -- 172.21.15.43:0/3756948981 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f631c07c9e0 con 0x7f631c07ddb0 2024-03-28T22:24:57.912 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:56.999+0000 7f6321400700 1 --2- 172.21.15.43:0/3756948981 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f631c07ddb0 0x7f631c07c320 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f63100098a0 tx=0x7f6310009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9077784865183e30 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:57.912 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:56.999+0000 7f631be00700 1 -- 172.21.15.43:0/3756948981 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6310004160 con 0x7f631c07ddb0 2024-03-28T22:24:57.912 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:56.999+0000 7f631be00700 1 -- 172.21.15.43:0/3756948981 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6310004300 con 0x7f631c07ddb0 2024-03-28T22:24:57.916 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:56.999+0000 7f631be00700 1 -- 172.21.15.43:0/3756948981 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6310011b50 con 0x7f631c07ddb0 2024-03-28T22:24:57.916 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.000+0000 7f6323e00700 1 -- 172.21.15.43:0/3756948981 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f631c07ddb0 msgr2=0x7f631c07c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:57.916 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.000+0000 7f6323e00700 1 --2- 172.21.15.43:0/3756948981 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f631c07ddb0 0x7f631c07c320 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f63100098a0 tx=0x7f6310009bb0 comp rx=0 tx=0).stop 2024-03-28T22:24:57.916 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.000+0000 7f6323e00700 1 -- 172.21.15.43:0/3756948981 shutdown_connections 2024-03-28T22:24:57.916 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.000+0000 7f6323e00700 1 --2- 172.21.15.43:0/3756948981 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f631c07ddb0 0x7f631c07c320 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:57.916 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.000+0000 7f6323e00700 1 -- 172.21.15.43:0/3756948981 >> 172.21.15.43:0/3756948981 conn(0x7f631c103450 msgr2=0x7f631c105870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:24:57.917 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.000+0000 7f6323e00700 1 -- 172.21.15.43:0/3756948981 shutdown_connections 2024-03-28T22:24:57.917 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.000+0000 7f6323e00700 1 -- 172.21.15.43:0/3756948981 wait complete. 2024-03-28T22:24:57.917 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.001+0000 7f6323e00700 1 Processor -- start 2024-03-28T22:24:57.917 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.001+0000 7f6323e00700 1 -- start start 2024-03-28T22:24:57.917 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.002+0000 7f6323e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f631c07ddb0 0x7f631c19de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:57.917 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.002+0000 7f6323e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f631c10e1d0 con 0x7f631c07ddb0 2024-03-28T22:24:57.917 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.002+0000 7f6321400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f631c07ddb0 0x7f631c19de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:57.917 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.002+0000 7f6321400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f631c07ddb0 0x7f631c19de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35580/0 (socket says 172.21.15.43:35580) 2024-03-28T22:24:57.917 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.002+0000 7f6321400700 1 -- 172.21.15.43:0/1509151745 learned_addr learned my addr 172.21.15.43:0/1509151745 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:24:57.917 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.002+0000 7f6321400700 1 -- 172.21.15.43:0/1509151745 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f631c19e340 con 0x7f631c07ddb0 2024-03-28T22:24:57.917 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.003+0000 7f6321400700 1 --2- 172.21.15.43:0/1509151745 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f631c07ddb0 0x7f631c19de00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6310011b60 tx=0x7f6310011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:57.917 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.003+0000 7f631a000700 1 -- 172.21.15.43:0/1509151745 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6310003b20 con 0x7f631c07ddb0 2024-03-28T22:24:57.917 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.003+0000 7f631a000700 1 -- 172.21.15.43:0/1509151745 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6310003cc0 con 0x7f631c07ddb0 2024-03-28T22:24:57.917 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.003+0000 7f6323e00700 1 -- 172.21.15.43:0/1509151745 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f631c19e5d0 con 0x7f631c07ddb0 2024-03-28T22:24:57.917 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.003+0000 7f631a000700 1 -- 172.21.15.43:0/1509151745 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6310024840 con 0x7f631c07ddb0 2024-03-28T22:24:57.918 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.003+0000 7f6323e00700 1 -- 172.21.15.43:0/1509151745 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f631c1a12c0 con 0x7f631c07ddb0 2024-03-28T22:24:57.918 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.004+0000 7f6323e00700 1 -- 172.21.15.43:0/1509151745 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f631c063870 con 0x7f631c07ddb0 2024-03-28T22:24:57.918 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.004+0000 7f631a000700 1 -- 172.21.15.43:0/1509151745 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7f6310022020 con 0x7f631c07ddb0 2024-03-28T22:24:57.918 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.005+0000 7f631a000700 1 --2- 172.21.15.43:0/1509151745 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f6304038ba0 0x7f630403b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:24:57.918 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.005+0000 7f631a000700 1 -- 172.21.15.43:0/1509151745 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6310011e00 con 0x7f631c07ddb0 2024-03-28T22:24:57.918 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.005+0000 7f6320a00700 1 --2- 172.21.15.43:0/1509151745 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f6304038ba0 0x7f630403b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:24:57.918 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.006+0000 7f6320a00700 1 --2- 172.21.15.43:0/1509151745 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f6304038ba0 0x7f630403b060 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6308006fd0 tx=0x7f6308006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:24:57.918 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.011+0000 7f631a000700 1 -- 172.21.15.43:0/1509151745 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f631004c580 con 0x7f631c07ddb0 2024-03-28T22:24:57.918 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.239+0000 7f6323e00700 1 -- 172.21.15.43:0/1509151745 --> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f631c104620 con 0x7f6304038ba0 2024-03-28T22:24:57.918 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.241+0000 7f631a000700 1 -- 172.21.15.43:0/1509151745 <== mgr.14120 v2:172.21.15.43:6800/3517712438 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f631c104620 con 0x7f6304038ba0 2024-03-28T22:24:57.918 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.244+0000 7f6323e00700 1 -- 172.21.15.43:0/1509151745 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f6304038ba0 msgr2=0x7f630403b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:57.918 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.244+0000 7f6323e00700 1 --2- 172.21.15.43:0/1509151745 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f6304038ba0 0x7f630403b060 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6308006fd0 tx=0x7f6308006e40 comp rx=0 tx=0).stop 2024-03-28T22:24:57.918 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.244+0000 7f6323e00700 1 -- 172.21.15.43:0/1509151745 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f631c07ddb0 msgr2=0x7f631c19de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:24:57.918 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.244+0000 7f6323e00700 1 --2- 172.21.15.43:0/1509151745 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f631c07ddb0 0x7f631c19de00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6310011b60 tx=0x7f6310011c40 comp rx=0 tx=0).stop 2024-03-28T22:24:57.919 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.244+0000 7f6323e00700 1 -- 172.21.15.43:0/1509151745 shutdown_connections 2024-03-28T22:24:57.919 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.244+0000 7f6323e00700 1 --2- 172.21.15.43:0/1509151745 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f6304038ba0 0x7f630403b060 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:57.919 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.244+0000 7f6323e00700 1 --2- 172.21.15.43:0/1509151745 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f631c07ddb0 0x7f631c19de00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:24:57.919 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.244+0000 7f6323e00700 1 -- 172.21.15.43:0/1509151745 >> 172.21.15.43:0/1509151745 conn(0x7f631c103450 msgr2=0x7f631c103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:24:57.919 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.245+0000 7f6323e00700 1 -- 172.21.15.43:0/1509151745 shutdown_connections 2024-03-28T22:24:57.919 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:57.245+0000 7f6323e00700 1 -- 172.21.15.43:0/1509151745 wait complete. 2024-03-28T22:24:57.919 INFO:teuthology.orchestra.run.smithi043.stdout:Generating ssh key... 2024-03-28T22:24:58.215 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:58 smithi043 ceph-mon[31823]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:24:59.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:59 smithi043 ceph-mon[31823]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:24:59.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:24:59 smithi043 ceph-mon[31823]: Generating ssh key... 2024-03-28T22:25:00.489 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:00 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: Generating public/private rsa key pair. 2024-03-28T22:25:00.489 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:00 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: Your identification has been saved in /tmp/tmp8n5q7_oh/key. 2024-03-28T22:25:00.489 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:00 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: Your public key has been saved in /tmp/tmp8n5q7_oh/key.pub. 2024-03-28T22:25:00.489 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:00 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: The key fingerprint is: 2024-03-28T22:25:00.489 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:00 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: SHA256:FrQe0POeUCq4aKyMuryvZ+ZqRvb4Hq6LD3/dejr0DX0 ceph-c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:25:00.489 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:00 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: The key's randomart image is: 2024-03-28T22:25:00.489 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:00 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: +---[RSA 3072]----+ 2024-03-28T22:25:00.489 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:00 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: | ... | 2024-03-28T22:25:00.489 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:00 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: | oo.. | 2024-03-28T22:25:00.489 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:00 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: | . += | 2024-03-28T22:25:00.489 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:00 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: | . ..oo. | 2024-03-28T22:25:00.489 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:00 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: | . . . .S+ . | 2024-03-28T22:25:00.490 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:00 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: | o+ . ... + E | 2024-03-28T22:25:00.490 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:00 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: |*oo. o o o . | 2024-03-28T22:25:00.490 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:00 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: |=B.=o o + . | 2024-03-28T22:25:00.490 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:00 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: |BX^B o= | 2024-03-28T22:25:00.490 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:00 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: +----[SHA256]-----+ 2024-03-28T22:25:00.804 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.525+0000 7fe2aec00700 1 Processor -- start 2024-03-28T22:25:00.805 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.526+0000 7fe2aec00700 1 -- start start 2024-03-28T22:25:00.805 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.526+0000 7fe2aec00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2a8108810 0x7fe2a810ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:00.805 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.526+0000 7fe2aec00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a807be50 con 0x7fe2a8108810 2024-03-28T22:25:00.805 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.527+0000 7fe2a7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2a8108810 0x7fe2a810ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:00.806 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.527+0000 7fe2a7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2a8108810 0x7fe2a810ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35588/0 (socket says 172.21.15.43:35588) 2024-03-28T22:25:00.806 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.527+0000 7fe2a7e00700 1 -- 172.21.15.43:0/4267248847 learned_addr learned my addr 172.21.15.43:0/4267248847 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:00.806 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.527+0000 7fe2a7e00700 1 -- 172.21.15.43:0/4267248847 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2a810b140 con 0x7fe2a8108810 2024-03-28T22:25:00.807 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.528+0000 7fe2a7e00700 1 --2- 172.21.15.43:0/4267248847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2a8108810 0x7fe2a810ac00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe2980098a0 tx=0x7fe298009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f0e18c4d0c1b21b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:00.807 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.528+0000 7fe2a6a00700 1 -- 172.21.15.43:0/4267248847 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe298004160 con 0x7fe2a8108810 2024-03-28T22:25:00.807 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.528+0000 7fe2a6a00700 1 -- 172.21.15.43:0/4267248847 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe298004300 con 0x7fe2a8108810 2024-03-28T22:25:00.807 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.528+0000 7fe2a6a00700 1 -- 172.21.15.43:0/4267248847 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe298011b50 con 0x7fe2a8108810 2024-03-28T22:25:00.808 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.528+0000 7fe2aec00700 1 -- 172.21.15.43:0/4267248847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2a8108810 msgr2=0x7fe2a810ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:00.808 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.528+0000 7fe2aec00700 1 --2- 172.21.15.43:0/4267248847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2a8108810 0x7fe2a810ac00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe2980098a0 tx=0x7fe298009bb0 comp rx=0 tx=0).stop 2024-03-28T22:25:00.809 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.529+0000 7fe2aec00700 1 -- 172.21.15.43:0/4267248847 shutdown_connections 2024-03-28T22:25:00.809 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.529+0000 7fe2aec00700 1 --2- 172.21.15.43:0/4267248847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2a8108810 0x7fe2a810ac00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:00.809 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.529+0000 7fe2aec00700 1 -- 172.21.15.43:0/4267248847 >> 172.21.15.43:0/4267248847 conn(0x7fe2a8102220 msgr2=0x7fe2a8104640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:00.809 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.529+0000 7fe2aec00700 1 -- 172.21.15.43:0/4267248847 shutdown_connections 2024-03-28T22:25:00.809 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.529+0000 7fe2aec00700 1 -- 172.21.15.43:0/4267248847 wait complete. 2024-03-28T22:25:00.809 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.530+0000 7fe2aec00700 1 Processor -- start 2024-03-28T22:25:00.809 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.530+0000 7fe2aec00700 1 -- start start 2024-03-28T22:25:00.809 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.531+0000 7fe2aec00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2a8108810 0x7fe2a8108090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:00.809 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.531+0000 7fe2aec00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a810c110 con 0x7fe2a8108810 2024-03-28T22:25:00.812 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.531+0000 7fe2a7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2a8108810 0x7fe2a8108090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:00.812 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.531+0000 7fe2a7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2a8108810 0x7fe2a8108090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35604/0 (socket says 172.21.15.43:35604) 2024-03-28T22:25:00.812 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.531+0000 7fe2a7e00700 1 -- 172.21.15.43:0/3945107238 learned_addr learned my addr 172.21.15.43:0/3945107238 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:00.812 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.531+0000 7fe2a7e00700 1 -- 172.21.15.43:0/3945107238 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2a81085d0 con 0x7fe2a8108810 2024-03-28T22:25:00.812 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.532+0000 7fe2a7e00700 1 --2- 172.21.15.43:0/3945107238 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2a8108810 0x7fe2a8108090 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe298011b60 tx=0x7fe298011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:00.812 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.532+0000 7fe2a4c00700 1 -- 172.21.15.43:0/3945107238 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe298003b20 con 0x7fe2a8108810 2024-03-28T22:25:00.812 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.532+0000 7fe2a4c00700 1 -- 172.21.15.43:0/3945107238 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe298003cc0 con 0x7fe2a8108810 2024-03-28T22:25:00.812 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.532+0000 7fe2a4c00700 1 -- 172.21.15.43:0/3945107238 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe298024840 con 0x7fe2a8108810 2024-03-28T22:25:00.812 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.532+0000 7fe2aec00700 1 -- 172.21.15.43:0/3945107238 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2a81067c0 con 0x7fe2a8108810 2024-03-28T22:25:00.812 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.532+0000 7fe2aec00700 1 -- 172.21.15.43:0/3945107238 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2a8106d00 con 0x7fe2a8108810 2024-03-28T22:25:00.812 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.533+0000 7fe2a4c00700 1 -- 172.21.15.43:0/3945107238 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7fe298022020 con 0x7fe2a8108810 2024-03-28T22:25:00.812 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.533+0000 7fe2aec00700 1 -- 172.21.15.43:0/3945107238 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2a8063870 con 0x7fe2a8108810 2024-03-28T22:25:00.812 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.534+0000 7fe2a4c00700 1 --2- 172.21.15.43:0/3945107238 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7fe28c038c50 0x7fe28c03b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:00.813 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.534+0000 7fe2a4c00700 1 -- 172.21.15.43:0/3945107238 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe298011e00 con 0x7fe2a8108810 2024-03-28T22:25:00.813 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.534+0000 7fe2a7400700 1 --2- 172.21.15.43:0/3945107238 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7fe28c038c50 0x7fe28c03b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:00.813 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.535+0000 7fe2a7400700 1 --2- 172.21.15.43:0/3945107238 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7fe28c038c50 0x7fe28c03b110 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe294006fd0 tx=0x7fe294006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:00.813 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.539+0000 7fe2a4c00700 1 -- 172.21.15.43:0/3945107238 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe29800b3e0 con 0x7fe2a8108810 2024-03-28T22:25:00.813 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:24:58.764+0000 7fe2aec00700 1 -- 172.21.15.43:0/3945107238 --> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fe2a81033d0 con 0x7fe28c038c50 2024-03-28T22:25:00.813 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:00.202+0000 7fe2a4c00700 1 -- 172.21.15.43:0/3945107238 <== mgr.14120 v2:172.21.15.43:6800/3517712438 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fe2a81033d0 con 0x7fe28c038c50 2024-03-28T22:25:00.813 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:00.205+0000 7fe2aec00700 1 -- 172.21.15.43:0/3945107238 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7fe28c038c50 msgr2=0x7fe28c03b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:00.813 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:00.205+0000 7fe2aec00700 1 --2- 172.21.15.43:0/3945107238 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7fe28c038c50 0x7fe28c03b110 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe294006fd0 tx=0x7fe294006e40 comp rx=0 tx=0).stop 2024-03-28T22:25:00.813 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:00.205+0000 7fe2aec00700 1 -- 172.21.15.43:0/3945107238 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2a8108810 msgr2=0x7fe2a8108090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:00.814 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:00.205+0000 7fe2aec00700 1 --2- 172.21.15.43:0/3945107238 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2a8108810 0x7fe2a8108090 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe298011b60 tx=0x7fe298011c40 comp rx=0 tx=0).stop 2024-03-28T22:25:00.814 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:00.205+0000 7fe2aec00700 1 -- 172.21.15.43:0/3945107238 shutdown_connections 2024-03-28T22:25:00.814 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:00.205+0000 7fe2aec00700 1 --2- 172.21.15.43:0/3945107238 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7fe28c038c50 0x7fe28c03b110 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:00.814 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:00.205+0000 7fe2aec00700 1 --2- 172.21.15.43:0/3945107238 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2a8108810 0x7fe2a8108090 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:00.814 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:00.205+0000 7fe2aec00700 1 -- 172.21.15.43:0/3945107238 >> 172.21.15.43:0/3945107238 conn(0x7fe2a8102220 msgr2=0x7fe2a8102cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:00.814 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:00.205+0000 7fe2aec00700 1 -- 172.21.15.43:0/3945107238 shutdown_connections 2024-03-28T22:25:00.815 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:00.205+0000 7fe2aec00700 1 -- 172.21.15.43:0/3945107238 wait complete. 2024-03-28T22:25:01.362 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:01 smithi043 ceph-mon[31823]: from='mgr.14120 172.21.15.43:0/1538160804' entity='mgr.x' 2024-03-28T22:25:01.363 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:01 smithi043 ceph-mon[31823]: from='mgr.14120 172.21.15.43:0/1538160804' entity='mgr.x' 2024-03-28T22:25:02.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:02 smithi043 ceph-mon[31823]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:25:02.293 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSJrviJP/lLx0yXeoEueK+37ysqRX1Pj/x7QVJ2i4HjAcaGrnNEHOicYaWaZHCaXkownQjKmcexQHdPvCOR86L6kJAbkzonLDQ2JltQEZlWCimyNE5TBylHUm9YBatuBBtL3+eHrIFpRrysR314a8g2xoSEeaFKhG4LoVouYA0E82NFr9GAPjUHV9ZITZ+SiN93r9xugfsn/AyX861CCIiVdZ+RA8GEHpvQwhCRAKb2xCiXn7QHD0OPWl/iDekEgfPIoCMPet9yxER14MHXpQ+vgwCYITii7AEE3XVvkByFhWMWU8LWgHUzbCxVI7Ddj2Ha8IBCV595/MhnxyC0NelWLLoqljFXy8gitiLj30UoseNoIYLSu6Vgpz+K2Obxx3Bmfg0Y9N0NdS8tRqQIw7GMsGo5T7T4zserDYAIZ7kDsPY+VtizDJ6TiZjVWSTpFoLBaxteJO0LHrQQ7gD4+FfAfWwalB/mY/2VcRWOxbVOA7+Tdm4rSL1Bfg02SWOUzs= ceph-c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:25:02.293 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.361+0000 7f7240400700 1 Processor -- start 2024-03-28T22:25:02.293 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.361+0000 7f7240400700 1 -- start start 2024-03-28T22:25:02.294 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.362+0000 7f7240400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f723807dd70 0x7f723807c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:02.294 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.362+0000 7f7240400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f723807c820 con 0x7f723807dd70 2024-03-28T22:25:02.294 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.362+0000 7f723da00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f723807dd70 0x7f723807c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:02.294 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.362+0000 7f723da00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f723807dd70 0x7f723807c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35616/0 (socket says 172.21.15.43:35616) 2024-03-28T22:25:02.294 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.363+0000 7f723da00700 1 -- 172.21.15.43:0/2552843574 learned_addr learned my addr 172.21.15.43:0/2552843574 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:02.294 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.363+0000 7f723da00700 1 -- 172.21.15.43:0/2552843574 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f723807c9a0 con 0x7f723807dd70 2024-03-28T22:25:02.295 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.363+0000 7f723da00700 1 --2- 172.21.15.43:0/2552843574 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f723807dd70 0x7f723807c2e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f722c0098a0 tx=0x7f722c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6135258dcb95ef5b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:02.295 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.364+0000 7f7237e00700 1 -- 172.21.15.43:0/2552843574 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f722c004160 con 0x7f723807dd70 2024-03-28T22:25:02.295 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.364+0000 7f7237e00700 1 -- 172.21.15.43:0/2552843574 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f722c004300 con 0x7f723807dd70 2024-03-28T22:25:02.295 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.364+0000 7f7237e00700 1 -- 172.21.15.43:0/2552843574 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f722c011b50 con 0x7f723807dd70 2024-03-28T22:25:02.295 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.364+0000 7f7240400700 1 -- 172.21.15.43:0/2552843574 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f723807dd70 msgr2=0x7f723807c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:02.295 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.364+0000 7f7240400700 1 --2- 172.21.15.43:0/2552843574 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f723807dd70 0x7f723807c2e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f722c0098a0 tx=0x7f722c009bb0 comp rx=0 tx=0).stop 2024-03-28T22:25:02.295 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.364+0000 7f7240400700 1 -- 172.21.15.43:0/2552843574 shutdown_connections 2024-03-28T22:25:02.296 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.364+0000 7f7240400700 1 --2- 172.21.15.43:0/2552843574 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f723807dd70 0x7f723807c2e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:02.296 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.364+0000 7f7240400700 1 -- 172.21.15.43:0/2552843574 >> 172.21.15.43:0/2552843574 conn(0x7f7238103340 msgr2=0x7f7238105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:02.296 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.365+0000 7f7240400700 1 -- 172.21.15.43:0/2552843574 shutdown_connections 2024-03-28T22:25:02.296 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.365+0000 7f7240400700 1 -- 172.21.15.43:0/2552843574 wait complete. 2024-03-28T22:25:02.296 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.366+0000 7f7240400700 1 Processor -- start 2024-03-28T22:25:02.296 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.366+0000 7f7240400700 1 -- start start 2024-03-28T22:25:02.296 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.366+0000 7f7240400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f723807dd70 0x7f723819de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:02.297 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.366+0000 7f7240400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f723810e1b0 con 0x7f723807dd70 2024-03-28T22:25:02.297 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.366+0000 7f723da00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f723807dd70 0x7f723819de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:02.297 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.366+0000 7f723da00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f723807dd70 0x7f723819de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35622/0 (socket says 172.21.15.43:35622) 2024-03-28T22:25:02.297 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.366+0000 7f723da00700 1 -- 172.21.15.43:0/2054181706 learned_addr learned my addr 172.21.15.43:0/2054181706 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:02.297 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.367+0000 7f723da00700 1 -- 172.21.15.43:0/2054181706 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f723819e3b0 con 0x7f723807dd70 2024-03-28T22:25:02.297 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.367+0000 7f723da00700 1 --2- 172.21.15.43:0/2054181706 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f723807dd70 0x7f723819de70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f722c011b60 tx=0x7f722c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:02.298 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.367+0000 7f7236000700 1 -- 172.21.15.43:0/2054181706 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f722c003b20 con 0x7f723807dd70 2024-03-28T22:25:02.298 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.367+0000 7f7236000700 1 -- 172.21.15.43:0/2054181706 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f722c003cc0 con 0x7f723807dd70 2024-03-28T22:25:02.298 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.367+0000 7f7236000700 1 -- 172.21.15.43:0/2054181706 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f722c024840 con 0x7f723807dd70 2024-03-28T22:25:02.298 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.367+0000 7f7240400700 1 -- 172.21.15.43:0/2054181706 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f723819e640 con 0x7f723807dd70 2024-03-28T22:25:02.298 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.367+0000 7f7240400700 1 -- 172.21.15.43:0/2054181706 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72381a1330 con 0x7f723807dd70 2024-03-28T22:25:02.298 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.368+0000 7f7240400700 1 -- 172.21.15.43:0/2054181706 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7238063870 con 0x7f723807dd70 2024-03-28T22:25:02.298 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.369+0000 7f7236000700 1 -- 172.21.15.43:0/2054181706 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7f722c022020 con 0x7f723807dd70 2024-03-28T22:25:02.299 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.370+0000 7f7236000700 1 --2- 172.21.15.43:0/2054181706 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f7220038bf0 0x7f722003b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:02.299 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.370+0000 7f7236000700 1 -- 172.21.15.43:0/2054181706 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f722c011e00 con 0x7f723807dd70 2024-03-28T22:25:02.299 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.370+0000 7f723d000700 1 --2- 172.21.15.43:0/2054181706 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f7220038bf0 0x7f722003b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:02.299 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.371+0000 7f723d000700 1 --2- 172.21.15.43:0/2054181706 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f7220038bf0 0x7f722003b0b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7224006fd0 tx=0x7f7224006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:02.299 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.375+0000 7f7236000700 1 -- 172.21.15.43:0/2054181706 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f722c016260 con 0x7f723807dd70 2024-03-28T22:25:02.299 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.605+0000 7f7240400700 1 -- 172.21.15.43:0/2054181706 --> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f72381045e0 con 0x7f7220038bf0 2024-03-28T22:25:02.299 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.607+0000 7f7236000700 1 -- 172.21.15.43:0/2054181706 <== mgr.14120 v2:172.21.15.43:6800/3517712438 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f72381045e0 con 0x7f7220038bf0 2024-03-28T22:25:02.300 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.610+0000 7f7240400700 1 -- 172.21.15.43:0/2054181706 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f7220038bf0 msgr2=0x7f722003b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:02.300 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.610+0000 7f7240400700 1 --2- 172.21.15.43:0/2054181706 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f7220038bf0 0x7f722003b0b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7224006fd0 tx=0x7f7224006e40 comp rx=0 tx=0).stop 2024-03-28T22:25:02.300 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.610+0000 7f7240400700 1 -- 172.21.15.43:0/2054181706 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f723807dd70 msgr2=0x7f723819de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:02.300 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.610+0000 7f7240400700 1 --2- 172.21.15.43:0/2054181706 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f723807dd70 0x7f723819de70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f722c011b60 tx=0x7f722c011c40 comp rx=0 tx=0).stop 2024-03-28T22:25:02.300 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.610+0000 7f7240400700 1 -- 172.21.15.43:0/2054181706 shutdown_connections 2024-03-28T22:25:02.300 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.610+0000 7f7240400700 1 --2- 172.21.15.43:0/2054181706 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f7220038bf0 0x7f722003b0b0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:02.300 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.610+0000 7f7240400700 1 --2- 172.21.15.43:0/2054181706 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f723807dd70 0x7f723819de70 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:02.301 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.610+0000 7f7240400700 1 -- 172.21.15.43:0/2054181706 >> 172.21.15.43:0/2054181706 conn(0x7f7238103340 msgr2=0x7f7238103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:02.301 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.610+0000 7f7240400700 1 -- 172.21.15.43:0/2054181706 shutdown_connections 2024-03-28T22:25:02.301 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:01.611+0000 7f7240400700 1 -- 172.21.15.43:0/2054181706 wait complete. 2024-03-28T22:25:02.301 INFO:teuthology.orchestra.run.smithi043.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-28T22:25:02.301 INFO:teuthology.orchestra.run.smithi043.stdout:Adding key to root@localhost authorized_keys... 2024-03-28T22:25:02.301 INFO:teuthology.orchestra.run.smithi043.stdout:Adding host smithi043... 2024-03-28T22:25:04.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:04 smithi043 ceph-mon[31823]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi043", "addr": "172.21.15.43", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:25:05.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:05 smithi043 ceph-mon[31823]: Deploying cephadm binary to smithi043 2024-03-28T22:25:10.438 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout Added host 'smithi043' with addr '172.21.15.43' 2024-03-28T22:25:10.439 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.880+0000 7fc5c1600700 1 Processor -- start 2024-03-28T22:25:10.439 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.880+0000 7fc5c1600700 1 -- start start 2024-03-28T22:25:10.439 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.881+0000 7fc5c1600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5bc10ecd0 0x7fc5bc1110c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:10.439 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.881+0000 7fc5c1600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc5bc082500 con 0x7fc5bc10ecd0 2024-03-28T22:25:10.439 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.881+0000 7fc5baa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5bc10ecd0 0x7fc5bc1110c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:10.439 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.881+0000 7fc5baa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5bc10ecd0 0x7fc5bc1110c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35624/0 (socket says 172.21.15.43:35624) 2024-03-28T22:25:10.439 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.881+0000 7fc5baa00700 1 -- 172.21.15.43:0/26960921 learned_addr learned my addr 172.21.15.43:0/26960921 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:10.439 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.882+0000 7fc5baa00700 1 -- 172.21.15.43:0/26960921 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5bc111600 con 0x7fc5bc10ecd0 2024-03-28T22:25:10.439 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.882+0000 7fc5baa00700 1 --2- 172.21.15.43:0/26960921 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5bc10ecd0 0x7fc5bc1110c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac0098a0 tx=0x7fc5ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6a5f23b96ef2fd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:10.439 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.882+0000 7fc5b9600700 1 -- 172.21.15.43:0/26960921 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5ac004160 con 0x7fc5bc10ecd0 2024-03-28T22:25:10.439 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.883+0000 7fc5b9600700 1 -- 172.21.15.43:0/26960921 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc5ac004300 con 0x7fc5bc10ecd0 2024-03-28T22:25:10.439 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.883+0000 7fc5b9600700 1 -- 172.21.15.43:0/26960921 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5ac011b50 con 0x7fc5bc10ecd0 2024-03-28T22:25:10.439 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.883+0000 7fc5c1600700 1 -- 172.21.15.43:0/26960921 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5bc10ecd0 msgr2=0x7fc5bc1110c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:10.439 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.883+0000 7fc5c1600700 1 --2- 172.21.15.43:0/26960921 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5bc10ecd0 0x7fc5bc1110c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac0098a0 tx=0x7fc5ac009bb0 comp rx=0 tx=0).stop 2024-03-28T22:25:10.439 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.883+0000 7fc5c1600700 1 -- 172.21.15.43:0/26960921 shutdown_connections 2024-03-28T22:25:10.440 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.883+0000 7fc5c1600700 1 --2- 172.21.15.43:0/26960921 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5bc10ecd0 0x7fc5bc1110c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:10.440 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.883+0000 7fc5c1600700 1 -- 172.21.15.43:0/26960921 >> 172.21.15.43:0/26960921 conn(0x7fc5bc07d440 msgr2=0x7fc5bc07f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:10.440 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.883+0000 7fc5c1600700 1 -- 172.21.15.43:0/26960921 shutdown_connections 2024-03-28T22:25:10.440 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.884+0000 7fc5c1600700 1 -- 172.21.15.43:0/26960921 wait complete. 2024-03-28T22:25:10.440 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.885+0000 7fc5c1600700 1 Processor -- start 2024-03-28T22:25:10.440 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.885+0000 7fc5c1600700 1 -- start start 2024-03-28T22:25:10.440 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.885+0000 7fc5c1600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5bc10ecd0 0x7fc5bc1a21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:10.440 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.885+0000 7fc5c1600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc5bc112680 con 0x7fc5bc10ecd0 2024-03-28T22:25:10.440 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.885+0000 7fc5baa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5bc10ecd0 0x7fc5bc1a21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:10.440 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.885+0000 7fc5baa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5bc10ecd0 0x7fc5bc1a21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35634/0 (socket says 172.21.15.43:35634) 2024-03-28T22:25:10.440 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.885+0000 7fc5baa00700 1 -- 172.21.15.43:0/1906255090 learned_addr learned my addr 172.21.15.43:0/1906255090 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:10.440 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.886+0000 7fc5baa00700 1 -- 172.21.15.43:0/1906255090 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5bc1a2710 con 0x7fc5bc10ecd0 2024-03-28T22:25:10.440 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.886+0000 7fc5baa00700 1 --2- 172.21.15.43:0/1906255090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5bc10ecd0 0x7fc5bc1a21d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac011b60 tx=0x7fc5ac011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:10.440 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.886+0000 7fc5b3400700 1 -- 172.21.15.43:0/1906255090 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5ac003b20 con 0x7fc5bc10ecd0 2024-03-28T22:25:10.440 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.886+0000 7fc5b3400700 1 -- 172.21.15.43:0/1906255090 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc5ac003cc0 con 0x7fc5bc10ecd0 2024-03-28T22:25:10.441 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.887+0000 7fc5b3400700 1 -- 172.21.15.43:0/1906255090 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5ac024840 con 0x7fc5bc10ecd0 2024-03-28T22:25:10.441 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.887+0000 7fc5c1600700 1 -- 172.21.15.43:0/1906255090 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5bc1a29a0 con 0x7fc5bc10ecd0 2024-03-28T22:25:10.441 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.887+0000 7fc5c1600700 1 -- 172.21.15.43:0/1906255090 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5bc1a5690 con 0x7fc5bc10ecd0 2024-03-28T22:25:10.441 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.888+0000 7fc5b3400700 1 -- 172.21.15.43:0/1906255090 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7fc5ac022020 con 0x7fc5bc10ecd0 2024-03-28T22:25:10.441 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.888+0000 7fc5c1600700 1 -- 172.21.15.43:0/1906255090 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5bc063870 con 0x7fc5bc10ecd0 2024-03-28T22:25:10.441 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.888+0000 7fc5b3400700 1 --2- 172.21.15.43:0/1906255090 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7fc5a0038bf0 0x7fc5a003b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:10.441 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.889+0000 7fc5b3400700 1 -- 172.21.15.43:0/1906255090 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc5ac011e00 con 0x7fc5bc10ecd0 2024-03-28T22:25:10.441 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.889+0000 7fc5ba000700 1 --2- 172.21.15.43:0/1906255090 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7fc5a0038bf0 0x7fc5a003b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:10.441 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.890+0000 7fc5ba000700 1 --2- 172.21.15.43:0/1906255090 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7fc5a0038bf0 0x7fc5a003b0b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8006fd0 tx=0x7fc5a8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:10.441 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:02.895+0000 7fc5b3400700 1 -- 172.21.15.43:0/1906255090 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc5ac04c500 con 0x7fc5bc10ecd0 2024-03-28T22:25:10.441 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:03.125+0000 7fc5c1600700 1 -- 172.21.15.43:0/1906255090 --> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi043", "addr": "172.21.15.43", "target": ["mon-mgr", ""]}) -- 0x7fc5bc07e5f0 con 0x7fc5a0038bf0 2024-03-28T22:25:10.441 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:09.767+0000 7fc5b3400700 1 -- 172.21.15.43:0/1906255090 <== mgr.14120 v2:172.21.15.43:6800/3517712438 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fc5bc07e5f0 con 0x7fc5a0038bf0 2024-03-28T22:25:10.441 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:09.771+0000 7fc5c1600700 1 -- 172.21.15.43:0/1906255090 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7fc5a0038bf0 msgr2=0x7fc5a003b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:10.441 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:09.771+0000 7fc5c1600700 1 --2- 172.21.15.43:0/1906255090 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7fc5a0038bf0 0x7fc5a003b0b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8006fd0 tx=0x7fc5a8006e40 comp rx=0 tx=0).stop 2024-03-28T22:25:10.442 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:09.771+0000 7fc5c1600700 1 -- 172.21.15.43:0/1906255090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5bc10ecd0 msgr2=0x7fc5bc1a21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:10.442 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:09.771+0000 7fc5c1600700 1 --2- 172.21.15.43:0/1906255090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5bc10ecd0 0x7fc5bc1a21d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac011b60 tx=0x7fc5ac011c40 comp rx=0 tx=0).stop 2024-03-28T22:25:10.442 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:09.772+0000 7fc5c1600700 1 -- 172.21.15.43:0/1906255090 shutdown_connections 2024-03-28T22:25:10.442 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:09.772+0000 7fc5c1600700 1 --2- 172.21.15.43:0/1906255090 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7fc5a0038bf0 0x7fc5a003b0b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:10.442 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:09.772+0000 7fc5c1600700 1 --2- 172.21.15.43:0/1906255090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc5bc10ecd0 0x7fc5bc1a21d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:10.442 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:09.772+0000 7fc5c1600700 1 -- 172.21.15.43:0/1906255090 >> 172.21.15.43:0/1906255090 conn(0x7fc5bc07d440 msgr2=0x7fc5bc07dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:10.442 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:09.772+0000 7fc5c1600700 1 -- 172.21.15.43:0/1906255090 shutdown_connections 2024-03-28T22:25:10.442 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:09.772+0000 7fc5c1600700 1 -- 172.21.15.43:0/1906255090 wait complete. 2024-03-28T22:25:10.442 INFO:teuthology.orchestra.run.smithi043.stdout:Deploying unmanaged mon service... 2024-03-28T22:25:11.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:10 smithi043 ceph-mon[31823]: from='mgr.14120 172.21.15.43:0/1538160804' entity='mgr.x' 2024-03-28T22:25:11.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:10 smithi043 ceph-mon[31823]: Added host smithi043 2024-03-28T22:25:11.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:10 smithi043 ceph-mon[31823]: from='mgr.14120 172.21.15.43:0/1538160804' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:25:12.151 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-28T22:25:12.152 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.076+0000 7fac08c00700 1 Processor -- start 2024-03-28T22:25:12.152 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.076+0000 7fac08c00700 1 -- start start 2024-03-28T22:25:12.152 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.077+0000 7fac08c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac04107cc0 0x7fac041080e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:12.152 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.077+0000 7fac08c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fac04108620 con 0x7fac04107cc0 2024-03-28T22:25:12.152 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.077+0000 7fac01e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac04107cc0 0x7fac041080e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:12.155 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.077+0000 7fac01e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac04107cc0 0x7fac041080e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60996/0 (socket says 172.21.15.43:60996) 2024-03-28T22:25:12.155 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.077+0000 7fac01e00700 1 -- 172.21.15.43:0/1435601147 learned_addr learned my addr 172.21.15.43:0/1435601147 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:12.155 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.078+0000 7fac01e00700 1 -- 172.21.15.43:0/1435601147 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac041087a0 con 0x7fac04107cc0 2024-03-28T22:25:12.155 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.078+0000 7fac01e00700 1 --2- 172.21.15.43:0/1435601147 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac04107cc0 0x7fac041080e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fabf40098a0 tx=0x7fabf4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9dbd139359db2f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:12.156 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.078+0000 7fac00a00700 1 -- 172.21.15.43:0/1435601147 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fabf4004160 con 0x7fac04107cc0 2024-03-28T22:25:12.156 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.078+0000 7fac00a00700 1 -- 172.21.15.43:0/1435601147 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fabf4004300 con 0x7fac04107cc0 2024-03-28T22:25:12.156 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.078+0000 7fac00a00700 1 -- 172.21.15.43:0/1435601147 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fabf4011b50 con 0x7fac04107cc0 2024-03-28T22:25:12.156 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.079+0000 7fac08c00700 1 -- 172.21.15.43:0/1435601147 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac04107cc0 msgr2=0x7fac041080e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:12.156 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.079+0000 7fac08c00700 1 --2- 172.21.15.43:0/1435601147 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac04107cc0 0x7fac041080e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fabf40098a0 tx=0x7fabf4009bb0 comp rx=0 tx=0).stop 2024-03-28T22:25:12.156 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.079+0000 7fac08c00700 1 -- 172.21.15.43:0/1435601147 shutdown_connections 2024-03-28T22:25:12.156 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.079+0000 7fac08c00700 1 --2- 172.21.15.43:0/1435601147 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac04107cc0 0x7fac041080e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:12.156 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.079+0000 7fac08c00700 1 -- 172.21.15.43:0/1435601147 >> 172.21.15.43:0/1435601147 conn(0x7fac04103340 msgr2=0x7fac04105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:12.156 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.079+0000 7fac08c00700 1 -- 172.21.15.43:0/1435601147 shutdown_connections 2024-03-28T22:25:12.156 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.079+0000 7fac08c00700 1 -- 172.21.15.43:0/1435601147 wait complete. 2024-03-28T22:25:12.156 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.080+0000 7fac08c00700 1 Processor -- start 2024-03-28T22:25:12.156 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.080+0000 7fac08c00700 1 -- start start 2024-03-28T22:25:12.156 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.081+0000 7fac08c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac04107cc0 0x7fac0419de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:12.156 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.081+0000 7fac08c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fac0410c130 con 0x7fac04107cc0 2024-03-28T22:25:12.156 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.081+0000 7fac01e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac04107cc0 0x7fac0419de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:12.157 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.081+0000 7fac01e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac04107cc0 0x7fac0419de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32768/0 (socket says 172.21.15.43:32768) 2024-03-28T22:25:12.157 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.081+0000 7fac01e00700 1 -- 172.21.15.43:0/906337656 learned_addr learned my addr 172.21.15.43:0/906337656 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:12.157 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.081+0000 7fac01e00700 1 -- 172.21.15.43:0/906337656 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac0419e3a0 con 0x7fac04107cc0 2024-03-28T22:25:12.157 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.082+0000 7fac01e00700 1 --2- 172.21.15.43:0/906337656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac04107cc0 0x7fac0419de60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fabf4011b60 tx=0x7fabf4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:12.157 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.082+0000 7fabfaa00700 1 -- 172.21.15.43:0/906337656 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fabf4003b20 con 0x7fac04107cc0 2024-03-28T22:25:12.157 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.082+0000 7fabfaa00700 1 -- 172.21.15.43:0/906337656 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fabf4003cc0 con 0x7fac04107cc0 2024-03-28T22:25:12.157 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.082+0000 7fabfaa00700 1 -- 172.21.15.43:0/906337656 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fabf4024840 con 0x7fac04107cc0 2024-03-28T22:25:12.157 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.082+0000 7fac08c00700 1 -- 172.21.15.43:0/906337656 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac0419e630 con 0x7fac04107cc0 2024-03-28T22:25:12.157 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.082+0000 7fac08c00700 1 -- 172.21.15.43:0/906337656 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac041a1320 con 0x7fac04107cc0 2024-03-28T22:25:12.159 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.083+0000 7fabfaa00700 1 -- 172.21.15.43:0/906337656 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7fabf4022020 con 0x7fac04107cc0 2024-03-28T22:25:12.159 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.083+0000 7fac08c00700 1 -- 172.21.15.43:0/906337656 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac04063870 con 0x7fac04107cc0 2024-03-28T22:25:12.159 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.084+0000 7fabfaa00700 1 --2- 172.21.15.43:0/906337656 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7fabe8038bf0 0x7fabe803b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:12.159 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.084+0000 7fabfaa00700 1 -- 172.21.15.43:0/906337656 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fabf4011e00 con 0x7fac04107cc0 2024-03-28T22:25:12.159 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.084+0000 7fac01400700 1 --2- 172.21.15.43:0/906337656 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7fabe8038bf0 0x7fabe803b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:12.160 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.085+0000 7fac01400700 1 --2- 172.21.15.43:0/906337656 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7fabe8038bf0 0x7fabe803b0b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fabf0006fd0 tx=0x7fabf0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:12.160 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.090+0000 7fabfaa00700 1 -- 172.21.15.43:0/906337656 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fabf400b3e0 con 0x7fac04107cc0 2024-03-28T22:25:12.160 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.334+0000 7fac08c00700 1 -- 172.21.15.43:0/906337656 --> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fac04104450 con 0x7fabe8038bf0 2024-03-28T22:25:12.160 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.346+0000 7fabfaa00700 1 -- 172.21.15.43:0/906337656 <== mgr.14120 v2:172.21.15.43:6800/3517712438 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fac04104450 con 0x7fabe8038bf0 2024-03-28T22:25:12.160 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.350+0000 7fac08c00700 1 -- 172.21.15.43:0/906337656 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7fabe8038bf0 msgr2=0x7fabe803b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:12.160 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.350+0000 7fac08c00700 1 --2- 172.21.15.43:0/906337656 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7fabe8038bf0 0x7fabe803b0b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fabf0006fd0 tx=0x7fabf0006e40 comp rx=0 tx=0).stop 2024-03-28T22:25:12.160 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.350+0000 7fac08c00700 1 -- 172.21.15.43:0/906337656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac04107cc0 msgr2=0x7fac0419de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:12.160 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.350+0000 7fac08c00700 1 --2- 172.21.15.43:0/906337656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac04107cc0 0x7fac0419de60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fabf4011b60 tx=0x7fabf4011c40 comp rx=0 tx=0).stop 2024-03-28T22:25:12.160 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.350+0000 7fac08c00700 1 -- 172.21.15.43:0/906337656 shutdown_connections 2024-03-28T22:25:12.160 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.350+0000 7fac08c00700 1 --2- 172.21.15.43:0/906337656 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7fabe8038bf0 0x7fabe803b0b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:12.160 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.350+0000 7fac08c00700 1 --2- 172.21.15.43:0/906337656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac04107cc0 0x7fac0419de60 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:12.160 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.350+0000 7fac08c00700 1 -- 172.21.15.43:0/906337656 >> 172.21.15.43:0/906337656 conn(0x7fac04103340 msgr2=0x7fac04103d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:12.160 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.350+0000 7fac08c00700 1 -- 172.21.15.43:0/906337656 shutdown_connections 2024-03-28T22:25:12.160 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:11.350+0000 7fac08c00700 1 -- 172.21.15.43:0/906337656 wait complete. 2024-03-28T22:25:12.160 INFO:teuthology.orchestra.run.smithi043.stdout:Deploying unmanaged mgr service... 2024-03-28T22:25:12.576 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:12 smithi043 ceph-mon[31823]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:25:12.576 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:12 smithi043 ceph-mon[31823]: Saving service mon spec with placement count:5 2024-03-28T22:25:12.577 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:12 smithi043 ceph-mon[31823]: from='mgr.14120 172.21.15.43:0/1538160804' entity='mgr.x' 2024-03-28T22:25:14.190 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-28T22:25:14.191 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.230+0000 7f9683800700 1 Processor -- start 2024-03-28T22:25:14.191 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.231+0000 7f9683800700 1 -- start start 2024-03-28T22:25:14.191 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.231+0000 7f9683800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f967c106570 0x7f967c1089a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:14.191 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.231+0000 7f9683800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f967c07be30 con 0x7f967c106570 2024-03-28T22:25:14.191 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.232+0000 7f9680e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f967c106570 0x7f967c1089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:14.191 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.232+0000 7f9680e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f967c106570 0x7f967c1089a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32772/0 (socket says 172.21.15.43:32772) 2024-03-28T22:25:14.191 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.232+0000 7f9680e00700 1 -- 172.21.15.43:0/1273799198 learned_addr learned my addr 172.21.15.43:0/1273799198 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:14.191 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.232+0000 7f9680e00700 1 -- 172.21.15.43:0/1273799198 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f967c108ee0 con 0x7f967c106570 2024-03-28T22:25:14.191 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.233+0000 7f9680e00700 1 --2- 172.21.15.43:0/1273799198 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f967c106570 0x7f967c1089a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f96700098a0 tx=0x7f9670009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78f8b2083aff944a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:14.191 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.233+0000 7f967b400700 1 -- 172.21.15.43:0/1273799198 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9670004160 con 0x7f967c106570 2024-03-28T22:25:14.191 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.233+0000 7f967b400700 1 -- 172.21.15.43:0/1273799198 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9670004300 con 0x7f967c106570 2024-03-28T22:25:14.191 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.233+0000 7f967b400700 1 -- 172.21.15.43:0/1273799198 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9670011b50 con 0x7f967c106570 2024-03-28T22:25:14.191 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.234+0000 7f9683800700 1 -- 172.21.15.43:0/1273799198 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f967c106570 msgr2=0x7f967c1089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:14.192 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.234+0000 7f9683800700 1 --2- 172.21.15.43:0/1273799198 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f967c106570 0x7f967c1089a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f96700098a0 tx=0x7f9670009bb0 comp rx=0 tx=0).stop 2024-03-28T22:25:14.192 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.234+0000 7f9683800700 1 -- 172.21.15.43:0/1273799198 shutdown_connections 2024-03-28T22:25:14.192 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.234+0000 7f9683800700 1 --2- 172.21.15.43:0/1273799198 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f967c106570 0x7f967c1089a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:14.192 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.234+0000 7f9683800700 1 -- 172.21.15.43:0/1273799198 >> 172.21.15.43:0/1273799198 conn(0x7f967c102170 msgr2=0x7f967c104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:14.192 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.234+0000 7f9683800700 1 -- 172.21.15.43:0/1273799198 shutdown_connections 2024-03-28T22:25:14.192 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.234+0000 7f9683800700 1 -- 172.21.15.43:0/1273799198 wait complete. 2024-03-28T22:25:14.192 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.235+0000 7f9683800700 1 Processor -- start 2024-03-28T22:25:14.192 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.236+0000 7f9683800700 1 -- start start 2024-03-28T22:25:14.192 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.236+0000 7f9683800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f967c106570 0x7f967c1a2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:14.192 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.236+0000 7f9683800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f967c109f70 con 0x7f967c106570 2024-03-28T22:25:14.192 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.236+0000 7f9680e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f967c106570 0x7f967c1a2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:14.192 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.236+0000 7f9680e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f967c106570 0x7f967c1a2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32788/0 (socket says 172.21.15.43:32788) 2024-03-28T22:25:14.192 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.236+0000 7f9680e00700 1 -- 172.21.15.43:0/160884991 learned_addr learned my addr 172.21.15.43:0/160884991 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:14.192 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.237+0000 7f9680e00700 1 -- 172.21.15.43:0/160884991 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f967c1a27c0 con 0x7f967c106570 2024-03-28T22:25:14.193 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.237+0000 7f9680e00700 1 --2- 172.21.15.43:0/160884991 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f967c106570 0x7f967c1a2280 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9670011b60 tx=0x7f9670011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:14.193 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.237+0000 7f9679600700 1 -- 172.21.15.43:0/160884991 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9670003b20 con 0x7f967c106570 2024-03-28T22:25:14.193 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.237+0000 7f9679600700 1 -- 172.21.15.43:0/160884991 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9670003cc0 con 0x7f967c106570 2024-03-28T22:25:14.193 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.237+0000 7f9683800700 1 -- 172.21.15.43:0/160884991 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f967c1a2a50 con 0x7f967c106570 2024-03-28T22:25:14.193 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.237+0000 7f9683800700 1 -- 172.21.15.43:0/160884991 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f967c1a5740 con 0x7f967c106570 2024-03-28T22:25:14.193 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.238+0000 7f9679600700 1 -- 172.21.15.43:0/160884991 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9670024840 con 0x7f967c106570 2024-03-28T22:25:14.193 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.238+0000 7f9679600700 1 -- 172.21.15.43:0/160884991 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7f9670022020 con 0x7f967c106570 2024-03-28T22:25:14.193 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.239+0000 7f9683800700 1 -- 172.21.15.43:0/160884991 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f967c063870 con 0x7f967c106570 2024-03-28T22:25:14.193 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.239+0000 7f9679600700 1 --2- 172.21.15.43:0/160884991 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f9664038ba0 0x7f966403b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:14.193 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.239+0000 7f9679600700 1 -- 172.21.15.43:0/160884991 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9670011e00 con 0x7f967c106570 2024-03-28T22:25:14.193 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.239+0000 7f967be00700 1 --2- 172.21.15.43:0/160884991 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f9664038ba0 0x7f966403b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:14.193 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.240+0000 7f967be00700 1 --2- 172.21.15.43:0/160884991 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f9664038ba0 0x7f966403b060 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9668006fd0 tx=0x7f9668006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:14.193 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.245+0000 7f9679600700 1 -- 172.21.15.43:0/160884991 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9670016260 con 0x7f967c106570 2024-03-28T22:25:14.193 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.445+0000 7f9679600700 1 -- 172.21.15.43:0/160884991 <== mon.0 v2:172.21.15.43:3300/0 7 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9670052080 con 0x7f967c106570 2024-03-28T22:25:14.194 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.489+0000 7f9683800700 1 -- 172.21.15.43:0/160884991 --> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f967c0008d0 con 0x7f9664038ba0 2024-03-28T22:25:14.194 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.496+0000 7f9679600700 1 -- 172.21.15.43:0/160884991 <== mgr.14120 v2:172.21.15.43:6800/3517712438 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f967c0008d0 con 0x7f9664038ba0 2024-03-28T22:25:14.194 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.499+0000 7f9683800700 1 -- 172.21.15.43:0/160884991 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f9664038ba0 msgr2=0x7f966403b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:14.194 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.499+0000 7f9683800700 1 --2- 172.21.15.43:0/160884991 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f9664038ba0 0x7f966403b060 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9668006fd0 tx=0x7f9668006e40 comp rx=0 tx=0).stop 2024-03-28T22:25:14.194 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.499+0000 7f9683800700 1 -- 172.21.15.43:0/160884991 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f967c106570 msgr2=0x7f967c1a2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:14.194 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.499+0000 7f9683800700 1 --2- 172.21.15.43:0/160884991 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f967c106570 0x7f967c1a2280 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9670011b60 tx=0x7f9670011c40 comp rx=0 tx=0).stop 2024-03-28T22:25:14.194 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.499+0000 7f9683800700 1 -- 172.21.15.43:0/160884991 shutdown_connections 2024-03-28T22:25:14.194 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.499+0000 7f9683800700 1 --2- 172.21.15.43:0/160884991 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f9664038ba0 0x7f966403b060 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:14.194 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.499+0000 7f9683800700 1 --2- 172.21.15.43:0/160884991 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f967c106570 0x7f967c1a2280 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:14.194 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.499+0000 7f9683800700 1 -- 172.21.15.43:0/160884991 >> 172.21.15.43:0/160884991 conn(0x7f967c102170 msgr2=0x7f967c104560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:14.194 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.499+0000 7f9683800700 1 -- 172.21.15.43:0/160884991 shutdown_connections 2024-03-28T22:25:14.194 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:13.500+0000 7f9683800700 1 -- 172.21.15.43:0/160884991 wait complete. 2024-03-28T22:25:14.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:14 smithi043 ceph-mon[31823]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:25:14.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:14 smithi043 ceph-mon[31823]: from='mgr.14120 172.21.15.43:0/1538160804' entity='mgr.x' 2024-03-28T22:25:14.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:14 smithi043 ceph-mon[31823]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:25:14.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:14 smithi043 ceph-mon[31823]: Saving service mgr spec with placement count:2 2024-03-28T22:25:14.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:14 smithi043 ceph-mon[31823]: from='mgr.14120 172.21.15.43:0/1538160804' entity='mgr.x' 2024-03-28T22:25:14.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:14 smithi043 ceph-mon[31823]: from='mgr.14120 172.21.15.43:0/1538160804' entity='mgr.x' 2024-03-28T22:25:15.775 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.865+0000 7f791b000700 1 Processor -- start 2024-03-28T22:25:15.775 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.865+0000 7f791b000700 1 -- start start 2024-03-28T22:25:15.775 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.866+0000 7f791b000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7914105f80 0x7f79141063a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:15.775 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.866+0000 7f791b000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7914106970 con 0x7f7914105f80 2024-03-28T22:25:15.775 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.866+0000 7f7913e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7914105f80 0x7f79141063a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:15.775 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.866+0000 7f7913e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7914105f80 0x7f79141063a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32790/0 (socket says 172.21.15.43:32790) 2024-03-28T22:25:15.778 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.866+0000 7f7913e00700 1 -- 172.21.15.43:0/3571779005 learned_addr learned my addr 172.21.15.43:0/3571779005 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:15.778 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.867+0000 7f7913e00700 1 -- 172.21.15.43:0/3571779005 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7914108f10 con 0x7f7914105f80 2024-03-28T22:25:15.778 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.867+0000 7f7913e00700 1 --2- 172.21.15.43:0/3571779005 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7914105f80 0x7f79141063a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f79040098a0 tx=0x7f7904009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49b5f63e6b9c7bc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:15.778 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.867+0000 7f7912a00700 1 -- 172.21.15.43:0/3571779005 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7904004160 con 0x7f7914105f80 2024-03-28T22:25:15.778 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.867+0000 7f7912a00700 1 -- 172.21.15.43:0/3571779005 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7904004300 con 0x7f7914105f80 2024-03-28T22:25:15.778 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.868+0000 7f7912a00700 1 -- 172.21.15.43:0/3571779005 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7904011b20 con 0x7f7914105f80 2024-03-28T22:25:15.778 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.868+0000 7f791b000700 1 -- 172.21.15.43:0/3571779005 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7914105f80 msgr2=0x7f79141063a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:15.778 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.868+0000 7f791b000700 1 --2- 172.21.15.43:0/3571779005 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7914105f80 0x7f79141063a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f79040098a0 tx=0x7f7904009bb0 comp rx=0 tx=0).stop 2024-03-28T22:25:15.779 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.868+0000 7f791b000700 1 -- 172.21.15.43:0/3571779005 shutdown_connections 2024-03-28T22:25:15.779 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.868+0000 7f791b000700 1 --2- 172.21.15.43:0/3571779005 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7914105f80 0x7f79141063a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:15.779 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.868+0000 7f791b000700 1 -- 172.21.15.43:0/3571779005 >> 172.21.15.43:0/3571779005 conn(0x7f79141018d0 msgr2=0x7f7914103cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:15.779 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.869+0000 7f791b000700 1 -- 172.21.15.43:0/3571779005 shutdown_connections 2024-03-28T22:25:15.779 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.869+0000 7f791b000700 1 -- 172.21.15.43:0/3571779005 wait complete. 2024-03-28T22:25:15.779 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.870+0000 7f791b000700 1 Processor -- start 2024-03-28T22:25:15.779 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.870+0000 7f791b000700 1 -- start start 2024-03-28T22:25:15.779 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.870+0000 7f791b000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7914105f80 0x7f791419de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:15.779 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.870+0000 7f791b000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f791410c2c0 con 0x7f7914105f80 2024-03-28T22:25:15.779 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.870+0000 7f7913e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7914105f80 0x7f791419de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:15.779 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.870+0000 7f7913e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7914105f80 0x7f791419de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:32804/0 (socket says 172.21.15.43:32804) 2024-03-28T22:25:15.779 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.870+0000 7f7913e00700 1 -- 172.21.15.43:0/1273979528 learned_addr learned my addr 172.21.15.43:0/1273979528 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:15.779 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.871+0000 7f7913e00700 1 -- 172.21.15.43:0/1273979528 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f791419e3a0 con 0x7f7914105f80 2024-03-28T22:25:15.779 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.871+0000 7f7913e00700 1 --2- 172.21.15.43:0/1273979528 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7914105f80 0x7f791419de60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f79040117e0 tx=0x7f7904011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:15.779 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.871+0000 7f7910c00700 1 -- 172.21.15.43:0/1273979528 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7904003bf0 con 0x7f7914105f80 2024-03-28T22:25:15.782 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.871+0000 7f7910c00700 1 -- 172.21.15.43:0/1273979528 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f790401a910 con 0x7f7914105f80 2024-03-28T22:25:15.782 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.871+0000 7f791b000700 1 -- 172.21.15.43:0/1273979528 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f791419e630 con 0x7f7914105f80 2024-03-28T22:25:15.782 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.871+0000 7f7910c00700 1 -- 172.21.15.43:0/1273979528 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7904024970 con 0x7f7914105f80 2024-03-28T22:25:15.782 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.872+0000 7f791b000700 1 -- 172.21.15.43:0/1273979528 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f791419ea90 con 0x7f7914105f80 2024-03-28T22:25:15.782 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.873+0000 7f791b000700 1 -- 172.21.15.43:0/1273979528 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7914063870 con 0x7f7914105f80 2024-03-28T22:25:15.782 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.873+0000 7f7910c00700 1 -- 172.21.15.43:0/1273979528 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7f7904022020 con 0x7f7914105f80 2024-03-28T22:25:15.782 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.873+0000 7f7910c00700 1 --2- 172.21.15.43:0/1273979528 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f78f8038bf0 0x7f78f803b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:15.782 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.873+0000 7f7910c00700 1 -- 172.21.15.43:0/1273979528 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f790404cd70 con 0x7f7914105f80 2024-03-28T22:25:15.783 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.874+0000 7f7913400700 1 --2- 172.21.15.43:0/1273979528 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f78f8038bf0 0x7f78f803b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:15.783 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.874+0000 7f7913400700 1 --2- 172.21.15.43:0/1273979528 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f78f8038bf0 0x7f78f803b0b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7900006fd0 tx=0x7f7900006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:15.783 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:14.878+0000 7f7910c00700 1 -- 172.21.15.43:0/1273979528 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7904051070 con 0x7f7914105f80 2024-03-28T22:25:15.783 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:15.136+0000 7f791b000700 1 -- 172.21.15.43:0/1273979528 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f79141a1320 con 0x7f7914105f80 2024-03-28T22:25:15.783 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:15.141+0000 7f7910c00700 1 -- 172.21.15.43:0/1273979528 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f790401ab70 con 0x7f7914105f80 2024-03-28T22:25:15.783 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:15.144+0000 7f791b000700 1 -- 172.21.15.43:0/1273979528 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f78f8038bf0 msgr2=0x7f78f803b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:15.783 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:15.144+0000 7f791b000700 1 --2- 172.21.15.43:0/1273979528 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f78f8038bf0 0x7f78f803b0b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7900006fd0 tx=0x7f7900006e40 comp rx=0 tx=0).stop 2024-03-28T22:25:15.783 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:15.144+0000 7f791b000700 1 -- 172.21.15.43:0/1273979528 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7914105f80 msgr2=0x7f791419de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:15.783 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:15.144+0000 7f791b000700 1 --2- 172.21.15.43:0/1273979528 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7914105f80 0x7f791419de60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f79040117e0 tx=0x7f7904011ba0 comp rx=0 tx=0).stop 2024-03-28T22:25:15.783 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:15.144+0000 7f791b000700 1 -- 172.21.15.43:0/1273979528 shutdown_connections 2024-03-28T22:25:15.783 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:15.144+0000 7f791b000700 1 --2- 172.21.15.43:0/1273979528 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f78f8038bf0 0x7f78f803b0b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:15.783 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:15.144+0000 7f791b000700 1 --2- 172.21.15.43:0/1273979528 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7914105f80 0x7f791419de60 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:15.783 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:15.144+0000 7f791b000700 1 -- 172.21.15.43:0/1273979528 >> 172.21.15.43:0/1273979528 conn(0x7f79141018d0 msgr2=0x7f79141022d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:15.784 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:15.145+0000 7f791b000700 1 -- 172.21.15.43:0/1273979528 shutdown_connections 2024-03-28T22:25:15.784 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:15.145+0000 7f791b000700 1 -- 172.21.15.43:0/1273979528 wait complete. 2024-03-28T22:25:16.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:16 smithi043 ceph-mon[31823]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:25:16.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:16 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1273979528' entity='client.admin' 2024-03-28T22:25:17.736 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.804+0000 7f6019400700 1 Processor -- start 2024-03-28T22:25:17.738 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.804+0000 7f6019400700 1 -- start start 2024-03-28T22:25:17.738 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.804+0000 7f6019400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6014107d80 0x7f60141081a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:17.738 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.805+0000 7f6019400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6014108770 con 0x7f6014107d80 2024-03-28T22:25:17.738 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.805+0000 7f6012800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6014107d80 0x7f60141081a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:17.738 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.805+0000 7f6012800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6014107d80 0x7f60141081a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36918/0 (socket says 172.21.15.43:36918) 2024-03-28T22:25:17.738 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.805+0000 7f6012800700 1 -- 172.21.15.43:0/583394391 learned_addr learned my addr 172.21.15.43:0/583394391 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:17.738 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.806+0000 7f6012800700 1 -- 172.21.15.43:0/583394391 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6014108ee0 con 0x7f6014107d80 2024-03-28T22:25:17.738 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.806+0000 7f6012800700 1 --2- 172.21.15.43:0/583394391 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6014107d80 0x7f60141081a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f60040098a0 tx=0x7f6004009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8131c614d0c20f7d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:17.738 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.806+0000 7f6011400700 1 -- 172.21.15.43:0/583394391 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6004004160 con 0x7f6014107d80 2024-03-28T22:25:17.739 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.806+0000 7f6011400700 1 -- 172.21.15.43:0/583394391 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6004004300 con 0x7f6014107d80 2024-03-28T22:25:17.739 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.806+0000 7f6011400700 1 -- 172.21.15.43:0/583394391 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6004011b20 con 0x7f6014107d80 2024-03-28T22:25:17.739 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.808+0000 7f6019400700 1 -- 172.21.15.43:0/583394391 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6014107d80 msgr2=0x7f60141081a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:17.739 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.808+0000 7f6019400700 1 --2- 172.21.15.43:0/583394391 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6014107d80 0x7f60141081a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f60040098a0 tx=0x7f6004009bb0 comp rx=0 tx=0).stop 2024-03-28T22:25:17.742 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.808+0000 7f6019400700 1 -- 172.21.15.43:0/583394391 shutdown_connections 2024-03-28T22:25:17.743 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.808+0000 7f6019400700 1 --2- 172.21.15.43:0/583394391 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6014107d80 0x7f60141081a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:17.743 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.808+0000 7f6019400700 1 -- 172.21.15.43:0/583394391 >> 172.21.15.43:0/583394391 conn(0x7f6014103340 msgr2=0x7f6014105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:17.743 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.808+0000 7f6019400700 1 -- 172.21.15.43:0/583394391 shutdown_connections 2024-03-28T22:25:17.743 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.808+0000 7f6019400700 1 -- 172.21.15.43:0/583394391 wait complete. 2024-03-28T22:25:17.743 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.809+0000 7f6019400700 1 Processor -- start 2024-03-28T22:25:17.743 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.809+0000 7f6019400700 1 -- start start 2024-03-28T22:25:17.743 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.810+0000 7f6019400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6014107d80 0x7f601419de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:17.743 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.810+0000 7f6019400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f601410c290 con 0x7f6014107d80 2024-03-28T22:25:17.743 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.810+0000 7f6012800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6014107d80 0x7f601419de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:17.743 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.810+0000 7f6012800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6014107d80 0x7f601419de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36934/0 (socket says 172.21.15.43:36934) 2024-03-28T22:25:17.743 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.810+0000 7f6012800700 1 -- 172.21.15.43:0/477336173 learned_addr learned my addr 172.21.15.43:0/477336173 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:17.743 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.810+0000 7f6012800700 1 -- 172.21.15.43:0/477336173 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f601419e3b0 con 0x7f6014107d80 2024-03-28T22:25:17.743 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.811+0000 7f6012800700 1 --2- 172.21.15.43:0/477336173 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6014107d80 0x7f601419de70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6004000c00 tx=0x7f6004011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:17.743 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.811+0000 7f600b400700 1 -- 172.21.15.43:0/477336173 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6004003b60 con 0x7f6014107d80 2024-03-28T22:25:17.743 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.811+0000 7f6019400700 1 -- 172.21.15.43:0/477336173 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f601419e640 con 0x7f6014107d80 2024-03-28T22:25:17.744 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.811+0000 7f600b400700 1 -- 172.21.15.43:0/477336173 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f600401a910 con 0x7f6014107d80 2024-03-28T22:25:17.744 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.811+0000 7f600b400700 1 -- 172.21.15.43:0/477336173 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6004024930 con 0x7f6014107d80 2024-03-28T22:25:17.744 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.811+0000 7f6019400700 1 -- 172.21.15.43:0/477336173 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f601419eaa0 con 0x7f6014107d80 2024-03-28T22:25:17.744 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.812+0000 7f6019400700 1 -- 172.21.15.43:0/477336173 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6014063870 con 0x7f6014107d80 2024-03-28T22:25:17.744 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.813+0000 7f600b400700 1 -- 172.21.15.43:0/477336173 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7f6004022020 con 0x7f6014107d80 2024-03-28T22:25:17.744 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.813+0000 7f600b400700 1 --2- 172.21.15.43:0/477336173 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f5ff8038ba0 0x7f5ff803b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:17.744 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.813+0000 7f600b400700 1 -- 172.21.15.43:0/477336173 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f600404cc20 con 0x7f6014107d80 2024-03-28T22:25:17.744 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.814+0000 7f6011e00700 1 --2- 172.21.15.43:0/477336173 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f5ff8038ba0 0x7f5ff803b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:17.744 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.815+0000 7f6011e00700 1 --2- 172.21.15.43:0/477336173 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f5ff8038ba0 0x7f5ff803b060 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6000006fd0 tx=0x7f6000006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:17.744 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:16.819+0000 7f600b400700 1 -- 172.21.15.43:0/477336173 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f600401aac0 con 0x7f6014107d80 2024-03-28T22:25:17.744 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:17.057+0000 7f6019400700 1 -- 172.21.15.43:0/477336173 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f60141a1530 con 0x7f6014107d80 2024-03-28T22:25:17.744 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:17.060+0000 7f600b400700 1 -- 172.21.15.43:0/477336173 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f600402d3f0 con 0x7f6014107d80 2024-03-28T22:25:17.744 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:17.064+0000 7f6019400700 1 -- 172.21.15.43:0/477336173 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f5ff8038ba0 msgr2=0x7f5ff803b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:17.744 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:17.064+0000 7f6019400700 1 --2- 172.21.15.43:0/477336173 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f5ff8038ba0 0x7f5ff803b060 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6000006fd0 tx=0x7f6000006e40 comp rx=0 tx=0).stop 2024-03-28T22:25:17.744 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:17.064+0000 7f6019400700 1 -- 172.21.15.43:0/477336173 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6014107d80 msgr2=0x7f601419de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:17.745 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:17.064+0000 7f6019400700 1 --2- 172.21.15.43:0/477336173 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6014107d80 0x7f601419de70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6004000c00 tx=0x7f6004011c10 comp rx=0 tx=0).stop 2024-03-28T22:25:17.745 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:17.064+0000 7f6019400700 1 -- 172.21.15.43:0/477336173 shutdown_connections 2024-03-28T22:25:17.745 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:17.064+0000 7f6019400700 1 --2- 172.21.15.43:0/477336173 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f5ff8038ba0 0x7f5ff803b060 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:17.745 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:17.064+0000 7f6019400700 1 --2- 172.21.15.43:0/477336173 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6014107d80 0x7f601419de70 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:17.745 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:17.064+0000 7f6019400700 1 -- 172.21.15.43:0/477336173 >> 172.21.15.43:0/477336173 conn(0x7f6014103340 msgr2=0x7f6014103dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:17.745 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:17.064+0000 7f6019400700 1 -- 172.21.15.43:0/477336173 shutdown_connections 2024-03-28T22:25:17.745 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:17.065+0000 7f6019400700 1 -- 172.21.15.43:0/477336173 wait complete. 2024-03-28T22:25:17.745 INFO:teuthology.orchestra.run.smithi043.stdout:Enabling the dashboard module... 2024-03-28T22:25:18.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:17 smithi043 ceph-mon[31823]: from='mgr.14120 172.21.15.43:0/1538160804' entity='mgr.x' 2024-03-28T22:25:18.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:17 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/477336173' entity='client.admin' 2024-03-28T22:25:18.909 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:18 smithi043 ceph-mon[31823]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:25:18.909 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:18 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3623762886' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-28T22:25:19.267 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:19 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: ignoring --setuser ceph since I am not root 2024-03-28T22:25:19.267 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:19 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: ignoring --setgroup ceph since I am not root 2024-03-28T22:25:19.593 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.337+0000 7f69fb200700 1 Processor -- start 2024-03-28T22:25:19.598 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.337+0000 7f69fb200700 1 -- start start 2024-03-28T22:25:19.598 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.337+0000 7f69fb200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69f4107d80 0x7f69f41081a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:19.598 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.337+0000 7f69fb200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f69f4108770 con 0x7f69f4107d80 2024-03-28T22:25:19.598 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.338+0000 7f69f3e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69f4107d80 0x7f69f41081a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:19.598 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.338+0000 7f69f3e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69f4107d80 0x7f69f41081a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36942/0 (socket says 172.21.15.43:36942) 2024-03-28T22:25:19.598 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.338+0000 7f69f3e00700 1 -- 172.21.15.43:0/431443387 learned_addr learned my addr 172.21.15.43:0/431443387 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:19.598 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.338+0000 7f69f3e00700 1 -- 172.21.15.43:0/431443387 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69f4108ee0 con 0x7f69f4107d80 2024-03-28T22:25:19.598 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.339+0000 7f69f3e00700 1 --2- 172.21.15.43:0/431443387 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69f4107d80 0x7f69f41081a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f69e40098a0 tx=0x7f69e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c63e8105ca1a77ed server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:19.599 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.339+0000 7f69f2a00700 1 -- 172.21.15.43:0/431443387 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f69e4004160 con 0x7f69f4107d80 2024-03-28T22:25:19.599 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.339+0000 7f69f2a00700 1 -- 172.21.15.43:0/431443387 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69e4004300 con 0x7f69f4107d80 2024-03-28T22:25:19.599 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.339+0000 7f69f2a00700 1 -- 172.21.15.43:0/431443387 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f69e4011b20 con 0x7f69f4107d80 2024-03-28T22:25:19.599 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.340+0000 7f69fb200700 1 -- 172.21.15.43:0/431443387 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69f4107d80 msgr2=0x7f69f41081a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:19.599 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.340+0000 7f69fb200700 1 --2- 172.21.15.43:0/431443387 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69f4107d80 0x7f69f41081a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f69e40098a0 tx=0x7f69e4009bb0 comp rx=0 tx=0).stop 2024-03-28T22:25:19.599 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.340+0000 7f69fb200700 1 -- 172.21.15.43:0/431443387 shutdown_connections 2024-03-28T22:25:19.599 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.340+0000 7f69fb200700 1 --2- 172.21.15.43:0/431443387 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69f4107d80 0x7f69f41081a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:19.599 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.340+0000 7f69fb200700 1 -- 172.21.15.43:0/431443387 >> 172.21.15.43:0/431443387 conn(0x7f69f4103340 msgr2=0x7f69f4105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:19.599 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.341+0000 7f69fb200700 1 -- 172.21.15.43:0/431443387 shutdown_connections 2024-03-28T22:25:19.599 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.341+0000 7f69fb200700 1 -- 172.21.15.43:0/431443387 wait complete. 2024-03-28T22:25:19.599 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.342+0000 7f69fb200700 1 Processor -- start 2024-03-28T22:25:19.599 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.342+0000 7f69fb200700 1 -- start start 2024-03-28T22:25:19.599 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.342+0000 7f69fb200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69f4107d80 0x7f69f419bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:19.599 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.342+0000 7f69fb200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f69f410c290 con 0x7f69f4107d80 2024-03-28T22:25:19.599 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.343+0000 7f69f3e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69f4107d80 0x7f69f419bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:19.599 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.343+0000 7f69f3e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69f4107d80 0x7f69f419bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36946/0 (socket says 172.21.15.43:36946) 2024-03-28T22:25:19.600 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.343+0000 7f69f3e00700 1 -- 172.21.15.43:0/3623762886 learned_addr learned my addr 172.21.15.43:0/3623762886 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:19.600 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.343+0000 7f69f3e00700 1 -- 172.21.15.43:0/3623762886 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69f419c1d0 con 0x7f69f4107d80 2024-03-28T22:25:19.600 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.343+0000 7f69f3e00700 1 --2- 172.21.15.43:0/3623762886 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69f4107d80 0x7f69f419bc90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f69e4000c00 tx=0x7f69e4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:19.600 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.344+0000 7f69f0c00700 1 -- 172.21.15.43:0/3623762886 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f69e4003b60 con 0x7f69f4107d80 2024-03-28T22:25:19.600 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.344+0000 7f69f0c00700 1 -- 172.21.15.43:0/3623762886 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69e401a910 con 0x7f69f4107d80 2024-03-28T22:25:19.600 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.344+0000 7f69f0c00700 1 -- 172.21.15.43:0/3623762886 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f69e4024930 con 0x7f69f4107d80 2024-03-28T22:25:19.600 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.344+0000 7f69fb200700 1 -- 172.21.15.43:0/3623762886 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69f419c460 con 0x7f69f4107d80 2024-03-28T22:25:19.600 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.344+0000 7f69fb200700 1 -- 172.21.15.43:0/3623762886 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69f419c8c0 con 0x7f69f4107d80 2024-03-28T22:25:19.600 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.345+0000 7f69f0c00700 1 -- 172.21.15.43:0/3623762886 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7f69e4022020 con 0x7f69f4107d80 2024-03-28T22:25:19.600 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.345+0000 7f69fb200700 1 -- 172.21.15.43:0/3623762886 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69f4063870 con 0x7f69f4107d80 2024-03-28T22:25:19.600 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.346+0000 7f69f0c00700 1 --2- 172.21.15.43:0/3623762886 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f69d8038bf0 0x7f69d803b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:19.600 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.346+0000 7f69f0c00700 1 -- 172.21.15.43:0/3623762886 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f69e404cb60 con 0x7f69f4107d80 2024-03-28T22:25:19.600 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.346+0000 7f69f3400700 1 --2- 172.21.15.43:0/3623762886 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f69d8038bf0 0x7f69d803b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:19.600 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.347+0000 7f69f3400700 1 --2- 172.21.15.43:0/3623762886 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f69d8038bf0 0x7f69d803b0b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f69e0006fd0 tx=0x7f69e0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:19.600 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.352+0000 7f69f0c00700 1 -- 172.21.15.43:0/3623762886 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f69e4024ad0 con 0x7f69f4107d80 2024-03-28T22:25:19.601 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.639+0000 7f69fb200700 1 -- 172.21.15.43:0/3623762886 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f69f419f700 con 0x7f69f4107d80 2024-03-28T22:25:19.601 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.883+0000 7f69f0c00700 1 -- 172.21.15.43:0/3623762886 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f69e401a430 con 0x7f69f4107d80 2024-03-28T22:25:19.601 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.885+0000 7f69f0c00700 1 -- 172.21.15.43:0/3623762886 <== mon.0 v2:172.21.15.43:3300/0 8 ==== mgrmap(e 9) ==== 45941+0+0 (secure 0 0 0) 0x7f69e402c350 con 0x7f69f4107d80 2024-03-28T22:25:19.601 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.888+0000 7f69fb200700 1 -- 172.21.15.43:0/3623762886 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f69d8038bf0 msgr2=0x7f69d803b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:19.601 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.888+0000 7f69fb200700 1 --2- 172.21.15.43:0/3623762886 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f69d8038bf0 0x7f69d803b0b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f69e0006fd0 tx=0x7f69e0006e40 comp rx=0 tx=0).stop 2024-03-28T22:25:19.601 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.888+0000 7f69fb200700 1 -- 172.21.15.43:0/3623762886 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69f4107d80 msgr2=0x7f69f419bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:19.601 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.888+0000 7f69fb200700 1 --2- 172.21.15.43:0/3623762886 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69f4107d80 0x7f69f419bc90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f69e4000c00 tx=0x7f69e4011c10 comp rx=0 tx=0).stop 2024-03-28T22:25:19.601 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.888+0000 7f69fb200700 1 -- 172.21.15.43:0/3623762886 shutdown_connections 2024-03-28T22:25:19.601 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.888+0000 7f69fb200700 1 --2- 172.21.15.43:0/3623762886 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7f69d8038bf0 0x7f69d803b0b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:19.601 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.888+0000 7f69fb200700 1 --2- 172.21.15.43:0/3623762886 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f69f4107d80 0x7f69f419bc90 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:19.601 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.888+0000 7f69fb200700 1 -- 172.21.15.43:0/3623762886 >> 172.21.15.43:0/3623762886 conn(0x7f69f4103340 msgr2=0x7f69f4103e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:19.601 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.888+0000 7f69fb200700 1 -- 172.21.15.43:0/3623762886 shutdown_connections 2024-03-28T22:25:19.601 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:18.888+0000 7f69fb200700 1 -- 172.21.15.43:0/3623762886 wait complete. 2024-03-28T22:25:19.874 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:19 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:25:19.693+0000 7f1db27f7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T22:25:19.875 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:19 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:25:19.818+0000 7f1db27f7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T22:25:20.193 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:19 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3623762886' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-28T22:25:20.193 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:19 smithi043 ceph-mon[31823]: mgrmap e9: x(active, since 25s) 2024-03-28T22:25:20.767 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:20 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:25:20.658+0000 7f1db27f7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T22:25:21.120 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:20 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/280560355' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-28T22:25:21.121 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:20 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:25:20.792+0000 7f1db27f7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T22:25:21.121 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:20 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:25:20.987+0000 7f1db27f7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T22:25:21.121 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:21 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:25:21.120+0000 7f1db27f7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T22:25:21.177 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout { 2024-03-28T22:25:21.178 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-28T22:25:21.178 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T22:25:21.178 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-28T22:25:21.178 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-28T22:25:21.178 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout } 2024-03-28T22:25:21.178 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.193+0000 7ff1dbe00700 1 Processor -- start 2024-03-28T22:25:21.178 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.193+0000 7ff1dbe00700 1 -- start start 2024-03-28T22:25:21.178 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.193+0000 7ff1dbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1dc1063e0 0x7ff1dc106800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:21.178 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.194+0000 7ff1dbe00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff1dc07be10 con 0x7ff1dc1063e0 2024-03-28T22:25:21.178 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.194+0000 7ff1daa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1dc1063e0 0x7ff1dc106800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:21.178 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.194+0000 7ff1daa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1dc1063e0 0x7ff1dc106800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36966/0 (socket says 172.21.15.43:36966) 2024-03-28T22:25:21.178 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.194+0000 7ff1daa00700 1 -- 172.21.15.43:0/1854747110 learned_addr learned my addr 172.21.15.43:0/1854747110 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:21.179 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.195+0000 7ff1daa00700 1 -- 172.21.15.43:0/1854747110 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1dc106d40 con 0x7ff1dc1063e0 2024-03-28T22:25:21.179 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.195+0000 7ff1daa00700 1 --2- 172.21.15.43:0/1854747110 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1dc1063e0 0x7ff1dc106800 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc0098a0 tx=0x7ff1cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb7712f01f1f5881 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:21.179 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.195+0000 7ff1da000700 1 -- 172.21.15.43:0/1854747110 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1cc004160 con 0x7ff1dc1063e0 2024-03-28T22:25:21.179 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.196+0000 7ff1da000700 1 -- 172.21.15.43:0/1854747110 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1cc004300 con 0x7ff1dc1063e0 2024-03-28T22:25:21.179 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.196+0000 7ff1da000700 1 -- 172.21.15.43:0/1854747110 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1cc011b20 con 0x7ff1dc1063e0 2024-03-28T22:25:21.179 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.196+0000 7ff1dbe00700 1 -- 172.21.15.43:0/1854747110 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1dc1063e0 msgr2=0x7ff1dc106800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:21.179 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.196+0000 7ff1dbe00700 1 --2- 172.21.15.43:0/1854747110 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1dc1063e0 0x7ff1dc106800 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc0098a0 tx=0x7ff1cc009bb0 comp rx=0 tx=0).stop 2024-03-28T22:25:21.179 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.197+0000 7ff1dbe00700 1 -- 172.21.15.43:0/1854747110 shutdown_connections 2024-03-28T22:25:21.179 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.197+0000 7ff1dbe00700 1 --2- 172.21.15.43:0/1854747110 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1dc1063e0 0x7ff1dc106800 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:21.179 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.197+0000 7ff1dbe00700 1 -- 172.21.15.43:0/1854747110 >> 172.21.15.43:0/1854747110 conn(0x7ff1dc101fc0 msgr2=0x7ff1dc1043e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:21.179 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.197+0000 7ff1dbe00700 1 -- 172.21.15.43:0/1854747110 shutdown_connections 2024-03-28T22:25:21.179 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.197+0000 7ff1dbe00700 1 -- 172.21.15.43:0/1854747110 wait complete. 2024-03-28T22:25:21.179 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.198+0000 7ff1dbe00700 1 Processor -- start 2024-03-28T22:25:21.179 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.198+0000 7ff1dbe00700 1 -- start start 2024-03-28T22:25:21.179 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.198+0000 7ff1dbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1dc1063e0 0x7ff1dc19dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:21.179 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.198+0000 7ff1dbe00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff1dc107e10 con 0x7ff1dc1063e0 2024-03-28T22:25:21.180 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.199+0000 7ff1daa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1dc1063e0 0x7ff1dc19dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:21.180 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.199+0000 7ff1daa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1dc1063e0 0x7ff1dc19dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36980/0 (socket says 172.21.15.43:36980) 2024-03-28T22:25:21.180 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.199+0000 7ff1daa00700 1 -- 172.21.15.43:0/280560355 learned_addr learned my addr 172.21.15.43:0/280560355 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:21.180 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.199+0000 7ff1daa00700 1 -- 172.21.15.43:0/280560355 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1dc19e1c0 con 0x7ff1dc1063e0 2024-03-28T22:25:21.180 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.200+0000 7ff1daa00700 1 --2- 172.21.15.43:0/280560355 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1dc1063e0 0x7ff1dc19dc80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc0117e0 tx=0x7ff1cc011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:21.180 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.200+0000 7ff1d3e00700 1 -- 172.21.15.43:0/280560355 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1cc003ba0 con 0x7ff1dc1063e0 2024-03-28T22:25:21.180 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.200+0000 7ff1d3e00700 1 -- 172.21.15.43:0/280560355 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1cc01a910 con 0x7ff1dc1063e0 2024-03-28T22:25:21.180 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.200+0000 7ff1d3e00700 1 -- 172.21.15.43:0/280560355 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1cc0248f0 con 0x7ff1dc1063e0 2024-03-28T22:25:21.180 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.200+0000 7ff1dbe00700 1 -- 172.21.15.43:0/280560355 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1dc19e450 con 0x7ff1dc1063e0 2024-03-28T22:25:21.180 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.200+0000 7ff1dbe00700 1 -- 172.21.15.43:0/280560355 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1dc19e8b0 con 0x7ff1dc1063e0 2024-03-28T22:25:21.180 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.201+0000 7ff1dbe00700 1 -- 172.21.15.43:0/280560355 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1dc063870 con 0x7ff1dc1063e0 2024-03-28T22:25:21.180 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.202+0000 7ff1d3e00700 1 -- 172.21.15.43:0/280560355 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 9) ==== 45941+0+0 (secure 0 0 0) 0x7ff1cc022020 con 0x7ff1dc1063e0 2024-03-28T22:25:21.180 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.202+0000 7ff1d3e00700 1 --2- 172.21.15.43:0/280560355 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7ff1c0038c50 0x7ff1c003b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:21.180 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.202+0000 7ff1d3e00700 1 -- 172.21.15.43:0/280560355 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ff1cc04cf80 con 0x7ff1dc1063e0 2024-03-28T22:25:21.181 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.202+0000 7ff1d2000700 1 -- 172.21.15.43:0/280560355 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7ff1c0038c50 msgr2=0x7ff1c003b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/3517712438 2024-03-28T22:25:21.181 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.202+0000 7ff1d2000700 1 --2- 172.21.15.43:0/280560355 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7ff1c0038c50 0x7ff1c003b110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:25:21.181 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.212+0000 7ff1d3e00700 1 -- 172.21.15.43:0/280560355 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff1cc01ab70 con 0x7ff1dc1063e0 2024-03-28T22:25:21.181 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.403+0000 7ff1d2000700 1 -- 172.21.15.43:0/280560355 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7ff1c0038c50 msgr2=0x7ff1c003b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/3517712438 2024-03-28T22:25:21.181 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.403+0000 7ff1d2000700 1 --2- 172.21.15.43:0/280560355 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7ff1c0038c50 0x7ff1c003b110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:25:21.181 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.493+0000 7ff1dbe00700 1 -- 172.21.15.43:0/280560355 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7ff1dc062e50 con 0x7ff1dc1063e0 2024-03-28T22:25:21.181 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.494+0000 7ff1d3e00700 1 -- 172.21.15.43:0/280560355 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7ff1cc04c8d0 con 0x7ff1dc1063e0 2024-03-28T22:25:21.181 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.497+0000 7ff1dbe00700 1 -- 172.21.15.43:0/280560355 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7ff1c0038c50 msgr2=0x7ff1c003b110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T22:25:21.181 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.497+0000 7ff1dbe00700 1 --2- 172.21.15.43:0/280560355 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7ff1c0038c50 0x7ff1c003b110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:21.181 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.497+0000 7ff1dbe00700 1 -- 172.21.15.43:0/280560355 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1dc1063e0 msgr2=0x7ff1dc19dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:21.181 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.497+0000 7ff1dbe00700 1 --2- 172.21.15.43:0/280560355 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1dc1063e0 0x7ff1dc19dc80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc0117e0 tx=0x7ff1cc011ba0 comp rx=0 tx=0).stop 2024-03-28T22:25:21.181 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.497+0000 7ff1dbe00700 1 -- 172.21.15.43:0/280560355 shutdown_connections 2024-03-28T22:25:21.181 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.497+0000 7ff1dbe00700 1 --2- 172.21.15.43:0/280560355 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7ff1c0038c50 0x7ff1c003b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:21.181 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.497+0000 7ff1dbe00700 1 --2- 172.21.15.43:0/280560355 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1dc1063e0 0x7ff1dc19dc80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:21.181 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.497+0000 7ff1dbe00700 1 -- 172.21.15.43:0/280560355 >> 172.21.15.43:0/280560355 conn(0x7ff1dc101fc0 msgr2=0x7ff1dc1948a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:21.182 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.497+0000 7ff1dbe00700 1 -- 172.21.15.43:0/280560355 shutdown_connections 2024-03-28T22:25:21.182 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:20.498+0000 7ff1dbe00700 1 -- 172.21.15.43:0/280560355 wait complete. 2024-03-28T22:25:21.182 INFO:teuthology.orchestra.run.smithi043.stdout:Waiting for the mgr to restart... 2024-03-28T22:25:21.182 INFO:teuthology.orchestra.run.smithi043.stdout:Waiting for mgr epoch 9... 2024-03-28T22:25:21.477 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:21 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:25:21.246+0000 7f1db27f7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T22:25:21.755 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:21 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:25:21.604+0000 7f1db27f7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T22:25:22.017 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:21 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:25:21.756+0000 7f1db27f7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T22:25:22.017 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:21 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:25:21.876+0000 7f1db27f7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T22:25:22.017 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:22 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:25:22.004+0000 7f1db27f7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T22:25:22.393 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:22 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:25:22.231+0000 7f1db27f7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T22:25:24.017 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:23 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:25:23.541+0000 7f1db27f7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T22:25:24.017 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:23 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:25:23.717+0000 7f1db27f7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T22:25:24.517 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:24 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:25:24.144+0000 7f1db27f7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T22:25:24.517 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:24 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:25:24.265+0000 7f1db27f7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T22:25:24.780 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:24 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:25:24.530+0000 7f1db27f7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T22:25:25.163 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:24 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:25:24.780+0000 7f1db27f7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T22:25:25.163 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:25 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:25:25.163+0000 7f1db27f7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T22:25:25.767 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:25 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:25:25.489+0000 7f1db27f7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T22:25:26.267 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:25 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:25:25.801+0000 7f1db27f7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T22:25:27.267 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:26 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:25:26.858+0000 7f1db27f7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T22:25:27.999 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:27 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:25:27.578+0000 7f1db27f7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T22:25:27.999 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:27 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:25:27.712+0000 7f1db27f7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T22:25:28.267 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:25:28 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:25:28.004+0000 7f1db27f7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T22:25:28.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:28 smithi043 ceph-mon[31823]: Active manager daemon x restarted 2024-03-28T22:25:28.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:28 smithi043 ceph-mon[31823]: Activating manager daemon x 2024-03-28T22:25:28.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:28 smithi043 ceph-mon[31823]: osdmap e3: 0 total, 0 up, 0 in 2024-03-28T22:25:28.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:28 smithi043 ceph-mon[31823]: mgrmap e10: x(active, starting, since 0.00810277s) 2024-03-28T22:25:28.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:28 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:25:28.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:28 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T22:25:28.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:28 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T22:25:28.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:28 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T22:25:28.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:28 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T22:25:28.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:28 smithi043 ceph-mon[31823]: Manager daemon x is now available 2024-03-28T22:25:29.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:29 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:25:29.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:29 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:25:29.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:29 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T22:25:29.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:29 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T22:25:29.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:29 smithi043 ceph-mon[31823]: [28/Mar/2024:22:25:28] ENGINE Bus STARTING 2024-03-28T22:25:29.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:29 smithi043 ceph-mon[31823]: [28/Mar/2024:22:25:28] ENGINE Serving on http://172.21.15.43:8765 2024-03-28T22:25:29.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:29 smithi043 ceph-mon[31823]: [28/Mar/2024:22:25:28] ENGINE Serving on https://172.21.15.43:7150 2024-03-28T22:25:29.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:29 smithi043 ceph-mon[31823]: [28/Mar/2024:22:25:28] ENGINE Bus STARTED 2024-03-28T22:25:29.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:29 smithi043 ceph-mon[31823]: mgrmap e11: x(active, since 1.00808s) 2024-03-28T22:25:29.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:29 smithi043 ceph-mon[31823]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-28T22:25:29.667 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout { 2024-03-28T22:25:29.667 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-28T22:25:29.667 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-28T22:25:29.667 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout } 2024-03-28T22:25:29.667 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.782+0000 7feb75e00700 1 Processor -- start 2024-03-28T22:25:29.667 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.783+0000 7feb75e00700 1 -- start start 2024-03-28T22:25:29.668 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.783+0000 7feb75e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb7007de40 0x7feb7007c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:29.668 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.783+0000 7feb75e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7feb7007c880 con 0x7feb7007de40 2024-03-28T22:25:29.668 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.783+0000 7feb6ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb7007de40 0x7feb7007c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:29.668 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.784+0000 7feb6ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb7007de40 0x7feb7007c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36982/0 (socket says 172.21.15.43:36982) 2024-03-28T22:25:29.668 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.784+0000 7feb6ea00700 1 -- 172.21.15.43:0/2955749493 learned_addr learned my addr 172.21.15.43:0/2955749493 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:29.668 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.784+0000 7feb6ea00700 1 -- 172.21.15.43:0/2955749493 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb7007ca00 con 0x7feb7007de40 2024-03-28T22:25:29.668 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.784+0000 7feb6ea00700 1 --2- 172.21.15.43:0/2955749493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb7007de40 0x7feb7007c340 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7feb600098a0 tx=0x7feb60009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13840cfb36d0c8df server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:29.668 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.785+0000 7feb6d600700 1 -- 172.21.15.43:0/2955749493 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb60004160 con 0x7feb7007de40 2024-03-28T22:25:29.668 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.785+0000 7feb6d600700 1 -- 172.21.15.43:0/2955749493 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb60004300 con 0x7feb7007de40 2024-03-28T22:25:29.668 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.785+0000 7feb6d600700 1 -- 172.21.15.43:0/2955749493 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb60011b20 con 0x7feb7007de40 2024-03-28T22:25:29.668 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.785+0000 7feb75e00700 1 -- 172.21.15.43:0/2955749493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb7007de40 msgr2=0x7feb7007c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:29.668 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.785+0000 7feb75e00700 1 --2- 172.21.15.43:0/2955749493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb7007de40 0x7feb7007c340 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7feb600098a0 tx=0x7feb60009bb0 comp rx=0 tx=0).stop 2024-03-28T22:25:29.668 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.786+0000 7feb75e00700 1 -- 172.21.15.43:0/2955749493 shutdown_connections 2024-03-28T22:25:29.669 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.786+0000 7feb75e00700 1 --2- 172.21.15.43:0/2955749493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb7007de40 0x7feb7007c340 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:29.669 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.786+0000 7feb75e00700 1 -- 172.21.15.43:0/2955749493 >> 172.21.15.43:0/2955749493 conn(0x7feb70103340 msgr2=0x7feb70105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:29.669 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.786+0000 7feb75e00700 1 -- 172.21.15.43:0/2955749493 shutdown_connections 2024-03-28T22:25:29.669 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.786+0000 7feb75e00700 1 -- 172.21.15.43:0/2955749493 wait complete. 2024-03-28T22:25:29.669 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.787+0000 7feb75e00700 1 Processor -- start 2024-03-28T22:25:29.669 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.788+0000 7feb75e00700 1 -- start start 2024-03-28T22:25:29.669 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.788+0000 7feb75e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb7007de40 0x7feb701a2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:29.669 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.788+0000 7feb75e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7feb70109ed0 con 0x7feb7007de40 2024-03-28T22:25:29.669 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.788+0000 7feb6ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb7007de40 0x7feb701a2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:29.669 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.788+0000 7feb6ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb7007de40 0x7feb701a2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36998/0 (socket says 172.21.15.43:36998) 2024-03-28T22:25:29.669 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.788+0000 7feb6ea00700 1 -- 172.21.15.43:0/1582611397 learned_addr learned my addr 172.21.15.43:0/1582611397 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:29.669 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.789+0000 7feb6ea00700 1 -- 172.21.15.43:0/1582611397 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb701a2770 con 0x7feb7007de40 2024-03-28T22:25:29.669 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.789+0000 7feb6ea00700 1 --2- 172.21.15.43:0/1582611397 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb7007de40 0x7feb701a2230 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7feb60011b60 tx=0x7feb60011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:29.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.789+0000 7feb74a00700 1 -- 172.21.15.43:0/1582611397 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb60003b60 con 0x7feb7007de40 2024-03-28T22:25:29.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.789+0000 7feb74a00700 1 -- 172.21.15.43:0/1582611397 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb6001a910 con 0x7feb7007de40 2024-03-28T22:25:29.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.789+0000 7feb74a00700 1 -- 172.21.15.43:0/1582611397 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb60024930 con 0x7feb7007de40 2024-03-28T22:25:29.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.789+0000 7feb75e00700 1 -- 172.21.15.43:0/1582611397 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb701a2a00 con 0x7feb7007de40 2024-03-28T22:25:29.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.790+0000 7feb75e00700 1 -- 172.21.15.43:0/1582611397 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb701a2e60 con 0x7feb7007de40 2024-03-28T22:25:29.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.791+0000 7feb74a00700 1 -- 172.21.15.43:0/1582611397 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 9) ==== 45941+0+0 (secure 0 0 0) 0x7feb60022020 con 0x7feb7007de40 2024-03-28T22:25:29.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.791+0000 7feb74a00700 1 --2- 172.21.15.43:0/1582611397 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7feb54038c50 0x7feb5403b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:29.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.791+0000 7feb75e00700 1 -- 172.21.15.43:0/1582611397 --> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7feb70063870 con 0x7feb54038c50 2024-03-28T22:25:29.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.791+0000 7feb74a00700 1 -- 172.21.15.43:0/1582611397 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7feb6004ccf0 con 0x7feb7007de40 2024-03-28T22:25:29.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.791+0000 7feb6e000700 1 -- 172.21.15.43:0/1582611397 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7feb54038c50 msgr2=0x7feb5403b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/3517712438 2024-03-28T22:25:29.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.791+0000 7feb6e000700 1 --2- 172.21.15.43:0/1582611397 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7feb54038c50 0x7feb5403b110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:25:29.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.992+0000 7feb6e000700 1 -- 172.21.15.43:0/1582611397 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7feb54038c50 msgr2=0x7feb5403b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/3517712438 2024-03-28T22:25:29.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:21.992+0000 7feb6e000700 1 --2- 172.21.15.43:0/1582611397 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7feb54038c50 0x7feb5403b110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:25:29.671 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:22.393+0000 7feb6e000700 1 -- 172.21.15.43:0/1582611397 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7feb54038c50 msgr2=0x7feb5403b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/3517712438 2024-03-28T22:25:29.671 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:22.393+0000 7feb6e000700 1 --2- 172.21.15.43:0/1582611397 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7feb54038c50 0x7feb5403b110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:25:29.671 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:23.194+0000 7feb6e000700 1 -- 172.21.15.43:0/1582611397 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7feb54038c50 msgr2=0x7feb5403b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/3517712438 2024-03-28T22:25:29.671 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:23.194+0000 7feb6e000700 1 --2- 172.21.15.43:0/1582611397 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7feb54038c50 0x7feb5403b110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:25:29.671 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:24.796+0000 7feb6e000700 1 -- 172.21.15.43:0/1582611397 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7feb54038c50 msgr2=0x7feb5403b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/3517712438 2024-03-28T22:25:29.671 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:24.796+0000 7feb6e000700 1 --2- 172.21.15.43:0/1582611397 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7feb54038c50 0x7feb5403b110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-28T22:25:29.671 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:27.999+0000 7feb6e000700 1 -- 172.21.15.43:0/1582611397 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7feb54038c50 msgr2=0x7feb5403b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/3517712438 2024-03-28T22:25:29.671 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:27.999+0000 7feb6e000700 1 --2- 172.21.15.43:0/1582611397 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7feb54038c50 0x7feb5403b110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-28T22:25:29.671 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:28.013+0000 7feb74a00700 1 -- 172.21.15.43:0/1582611397 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mgrmap(e 10) ==== 45708+0+0 (secure 0 0 0) 0x7feb6002d3f0 con 0x7feb7007de40 2024-03-28T22:25:29.671 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:28.013+0000 7feb74a00700 1 -- 172.21.15.43:0/1582611397 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7feb54038c50 msgr2=0x7feb5403b110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T22:25:29.671 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:28.013+0000 7feb74a00700 1 --2- 172.21.15.43:0/1582611397 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7feb54038c50 0x7feb5403b110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:29.671 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:29.013+0000 7feb74a00700 1 -- 172.21.15.43:0/1582611397 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mgrmap(e 11) ==== 45835+0+0 (secure 0 0 0) 0x7feb60018d00 con 0x7feb7007de40 2024-03-28T22:25:29.671 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:29.014+0000 7feb74a00700 1 --2- 172.21.15.43:0/1582611397 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7feb5403c550 0x7feb5403e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:29.672 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:29.014+0000 7feb74a00700 1 -- 172.21.15.43:0/1582611397 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7feb6000af30 con 0x7feb5403c550 2024-03-28T22:25:29.672 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:29.014+0000 7feb6e000700 1 --2- 172.21.15.43:0/1582611397 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7feb5403c550 0x7feb5403e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:29.672 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:29.016+0000 7feb6e000700 1 --2- 172.21.15.43:0/1582611397 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7feb5403c550 0x7feb5403e940 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7feb5c003a10 tx=0x7feb5c009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:29.672 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:29.017+0000 7feb74a00700 1 -- 172.21.15.43:0/1582611397 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7feb6000af30 con 0x7feb5403c550 2024-03-28T22:25:29.672 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:29.025+0000 7feb75e00700 1 -- 172.21.15.43:0/1582611397 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7feb70062680 con 0x7feb5403c550 2024-03-28T22:25:29.672 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:29.025+0000 7feb74a00700 1 -- 172.21.15.43:0/1582611397 <== mgr.14152 v2:172.21.15.43:6800/369798199 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7feb70062680 con 0x7feb5403c550 2024-03-28T22:25:29.672 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:29.026+0000 7feb75e00700 1 -- 172.21.15.43:0/1582611397 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7feb5403c550 msgr2=0x7feb5403e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:29.672 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:29.026+0000 7feb75e00700 1 --2- 172.21.15.43:0/1582611397 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7feb5403c550 0x7feb5403e940 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7feb5c003a10 tx=0x7feb5c009270 comp rx=0 tx=0).stop 2024-03-28T22:25:29.672 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:29.026+0000 7feb75e00700 1 -- 172.21.15.43:0/1582611397 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb7007de40 msgr2=0x7feb701a2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:29.672 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:29.026+0000 7feb75e00700 1 --2- 172.21.15.43:0/1582611397 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb7007de40 0x7feb701a2230 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7feb60011b60 tx=0x7feb60011c40 comp rx=0 tx=0).stop 2024-03-28T22:25:29.672 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:29.026+0000 7feb75e00700 1 -- 172.21.15.43:0/1582611397 shutdown_connections 2024-03-28T22:25:29.672 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:29.026+0000 7feb75e00700 1 --2- 172.21.15.43:0/1582611397 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7feb5403c550 0x7feb5403e940 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:29.672 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:29.026+0000 7feb75e00700 1 --2- 172.21.15.43:0/1582611397 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feb7007de40 0x7feb701a2230 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:29.672 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:29.026+0000 7feb75e00700 1 --2- 172.21.15.43:0/1582611397 >> [v2:172.21.15.43:6800/3517712438,v1:172.21.15.43:6801/3517712438] conn(0x7feb54038c50 0x7feb5403b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:29.672 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:29.026+0000 7feb75e00700 1 -- 172.21.15.43:0/1582611397 >> 172.21.15.43:0/1582611397 conn(0x7feb70103340 msgr2=0x7feb70105730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:29.673 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:29.026+0000 7feb75e00700 1 -- 172.21.15.43:0/1582611397 shutdown_connections 2024-03-28T22:25:29.673 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:29.026+0000 7feb75e00700 1 -- 172.21.15.43:0/1582611397 wait complete. 2024-03-28T22:25:29.673 INFO:teuthology.orchestra.run.smithi043.stdout:mgr epoch 9 is available 2024-03-28T22:25:29.673 INFO:teuthology.orchestra.run.smithi043.stdout:Generating a dashboard self-signed certificate... 2024-03-28T22:25:30.241 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:30 smithi043 ceph-mon[31823]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-28T22:25:31.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:31 smithi043 ceph-mon[31823]: mgrmap e12: x(active, since 2s) 2024-03-28T22:25:31.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:31 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:25:31.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:31 smithi043 ceph-mon[31823]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:25:31.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:31 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:25:31.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:31 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:25:31.692 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-28T22:25:31.692 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.600+0000 7ff5b9a00700 1 Processor -- start 2024-03-28T22:25:31.692 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.600+0000 7ff5b9a00700 1 -- start start 2024-03-28T22:25:31.693 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.601+0000 7ff5b9a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5b4109ee0 0x7ff5b410a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:31.693 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.601+0000 7ff5b9a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b410a840 con 0x7ff5b4109ee0 2024-03-28T22:25:31.693 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.601+0000 7ff5b2a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5b4109ee0 0x7ff5b410a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:31.693 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.601+0000 7ff5b2a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5b4109ee0 0x7ff5b410a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55172/0 (socket says 172.21.15.43:55172) 2024-03-28T22:25:31.693 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.601+0000 7ff5b2a00700 1 -- 172.21.15.43:0/1658975172 learned_addr learned my addr 172.21.15.43:0/1658975172 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:31.693 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.602+0000 7ff5b2a00700 1 -- 172.21.15.43:0/1658975172 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5b410aff0 con 0x7ff5b4109ee0 2024-03-28T22:25:31.693 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.602+0000 7ff5b2a00700 1 --2- 172.21.15.43:0/1658975172 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5b4109ee0 0x7ff5b410a300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff5a40098a0 tx=0x7ff5a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45160d84414bc19e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:31.693 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.602+0000 7ff5b1600700 1 -- 172.21.15.43:0/1658975172 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff5a4004160 con 0x7ff5b4109ee0 2024-03-28T22:25:31.693 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.603+0000 7ff5b1600700 1 -- 172.21.15.43:0/1658975172 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5a4004300 con 0x7ff5b4109ee0 2024-03-28T22:25:31.693 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.603+0000 7ff5b1600700 1 -- 172.21.15.43:0/1658975172 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff5a4011b20 con 0x7ff5b4109ee0 2024-03-28T22:25:31.693 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.603+0000 7ff5b9a00700 1 -- 172.21.15.43:0/1658975172 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5b4109ee0 msgr2=0x7ff5b410a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:31.693 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.603+0000 7ff5b9a00700 1 --2- 172.21.15.43:0/1658975172 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5b4109ee0 0x7ff5b410a300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff5a40098a0 tx=0x7ff5a4009bb0 comp rx=0 tx=0).stop 2024-03-28T22:25:31.693 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.603+0000 7ff5b9a00700 1 -- 172.21.15.43:0/1658975172 shutdown_connections 2024-03-28T22:25:31.694 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.603+0000 7ff5b9a00700 1 --2- 172.21.15.43:0/1658975172 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5b4109ee0 0x7ff5b410a300 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:31.694 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.603+0000 7ff5b9a00700 1 -- 172.21.15.43:0/1658975172 >> 172.21.15.43:0/1658975172 conn(0x7ff5b407d440 msgr2=0x7ff5b407d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:31.694 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.603+0000 7ff5b9a00700 1 -- 172.21.15.43:0/1658975172 shutdown_connections 2024-03-28T22:25:31.694 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.604+0000 7ff5b9a00700 1 -- 172.21.15.43:0/1658975172 wait complete. 2024-03-28T22:25:31.694 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.605+0000 7ff5b9a00700 1 Processor -- start 2024-03-28T22:25:31.694 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.605+0000 7ff5b9a00700 1 -- start start 2024-03-28T22:25:31.694 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.605+0000 7ff5b9a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5b4109ee0 0x7ff5b419dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:31.694 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.605+0000 7ff5b9a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b410e430 con 0x7ff5b4109ee0 2024-03-28T22:25:31.694 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.605+0000 7ff5b2a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5b4109ee0 0x7ff5b419dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:31.694 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.605+0000 7ff5b2a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5b4109ee0 0x7ff5b419dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55188/0 (socket says 172.21.15.43:55188) 2024-03-28T22:25:31.694 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.605+0000 7ff5b2a00700 1 -- 172.21.15.43:0/2187299788 learned_addr learned my addr 172.21.15.43:0/2187299788 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:31.694 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.606+0000 7ff5b2a00700 1 -- 172.21.15.43:0/2187299788 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5b419e2d0 con 0x7ff5b4109ee0 2024-03-28T22:25:31.694 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.606+0000 7ff5b2a00700 1 --2- 172.21.15.43:0/2187299788 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5b4109ee0 0x7ff5b419dd90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff5a40117e0 tx=0x7ff5a4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:31.695 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.606+0000 7ff5ab400700 1 -- 172.21.15.43:0/2187299788 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff5a4003bf0 con 0x7ff5b4109ee0 2024-03-28T22:25:31.695 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.606+0000 7ff5ab400700 1 -- 172.21.15.43:0/2187299788 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5a401a910 con 0x7ff5b4109ee0 2024-03-28T22:25:31.695 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.606+0000 7ff5b9a00700 1 -- 172.21.15.43:0/2187299788 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5b419e560 con 0x7ff5b4109ee0 2024-03-28T22:25:31.695 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.607+0000 7ff5ab400700 1 -- 172.21.15.43:0/2187299788 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff5a4024970 con 0x7ff5b4109ee0 2024-03-28T22:25:31.695 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.607+0000 7ff5b9a00700 1 -- 172.21.15.43:0/2187299788 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5b41a1210 con 0x7ff5b4109ee0 2024-03-28T22:25:31.695 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.608+0000 7ff5ab400700 1 -- 172.21.15.43:0/2187299788 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 12) ==== 45941+0+0 (secure 0 0 0) 0x7ff5a4022020 con 0x7ff5b4109ee0 2024-03-28T22:25:31.695 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.608+0000 7ff5b9a00700 1 -- 172.21.15.43:0/2187299788 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5b4063870 con 0x7ff5b4109ee0 2024-03-28T22:25:31.695 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.608+0000 7ff5ab400700 1 --2- 172.21.15.43:0/2187299788 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff598038ba0 0x7ff59803b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:31.695 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.608+0000 7ff5ab400700 1 -- 172.21.15.43:0/2187299788 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff5a404cfa0 con 0x7ff5b4109ee0 2024-03-28T22:25:31.695 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.609+0000 7ff5b2000700 1 --2- 172.21.15.43:0/2187299788 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff598038ba0 0x7ff59803b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:31.695 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.609+0000 7ff5b2000700 1 --2- 172.21.15.43:0/2187299788 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff598038ba0 0x7ff59803b060 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0006fd0 tx=0x7ff5a0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:31.695 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.614+0000 7ff5ab400700 1 -- 172.21.15.43:0/2187299788 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff5a4011e20 con 0x7ff5b4109ee0 2024-03-28T22:25:31.695 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:30.859+0000 7ff5b9a00700 1 -- 172.21.15.43:0/2187299788 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7ff5b4107a90 con 0x7ff598038ba0 2024-03-28T22:25:31.695 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:31.035+0000 7ff5ab400700 1 -- 172.21.15.43:0/2187299788 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7ff5b4107a90 con 0x7ff598038ba0 2024-03-28T22:25:31.695 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:31.038+0000 7ff5b9a00700 1 -- 172.21.15.43:0/2187299788 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff598038ba0 msgr2=0x7ff59803b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:31.696 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:31.038+0000 7ff5b9a00700 1 --2- 172.21.15.43:0/2187299788 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff598038ba0 0x7ff59803b060 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0006fd0 tx=0x7ff5a0006e40 comp rx=0 tx=0).stop 2024-03-28T22:25:31.696 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:31.038+0000 7ff5b9a00700 1 -- 172.21.15.43:0/2187299788 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5b4109ee0 msgr2=0x7ff5b419dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:31.696 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:31.038+0000 7ff5b9a00700 1 --2- 172.21.15.43:0/2187299788 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5b4109ee0 0x7ff5b419dd90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff5a40117e0 tx=0x7ff5a4011c10 comp rx=0 tx=0).stop 2024-03-28T22:25:31.696 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:31.038+0000 7ff5b9a00700 1 -- 172.21.15.43:0/2187299788 shutdown_connections 2024-03-28T22:25:31.696 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:31.038+0000 7ff5b9a00700 1 --2- 172.21.15.43:0/2187299788 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff598038ba0 0x7ff59803b060 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:31.696 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:31.038+0000 7ff5b9a00700 1 --2- 172.21.15.43:0/2187299788 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff5b4109ee0 0x7ff5b419dd90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:31.696 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:31.038+0000 7ff5b9a00700 1 -- 172.21.15.43:0/2187299788 >> 172.21.15.43:0/2187299788 conn(0x7ff5b407d440 msgr2=0x7ff5b4107380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:31.696 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:31.039+0000 7ff5b9a00700 1 -- 172.21.15.43:0/2187299788 shutdown_connections 2024-03-28T22:25:31.696 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:31.039+0000 7ff5b9a00700 1 -- 172.21.15.43:0/2187299788 wait complete. 2024-03-28T22:25:31.696 INFO:teuthology.orchestra.run.smithi043.stdout:Creating initial admin user... 2024-03-28T22:25:33.523 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$MxzjtsPjquYf12ELRUJdP.IJ0cp52nS6KCZWqA8oJwo4xv9//S6GC", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711664732, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-28T22:25:33.523 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.327+0000 7f0fe7e00700 1 Processor -- start 2024-03-28T22:25:33.523 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.328+0000 7f0fe7e00700 1 -- start start 2024-03-28T22:25:33.524 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.328+0000 7f0fe7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0fe8108550 0x7f0fe810a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:33.524 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.328+0000 7f0fe7e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0fe807beb0 con 0x7f0fe8108550 2024-03-28T22:25:33.524 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.328+0000 7f0fe6a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0fe8108550 0x7f0fe810a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:33.524 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.329+0000 7f0fe6a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0fe8108550 0x7f0fe810a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55204/0 (socket says 172.21.15.43:55204) 2024-03-28T22:25:33.524 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.329+0000 7f0fe6a00700 1 -- 172.21.15.43:0/344400080 learned_addr learned my addr 172.21.15.43:0/344400080 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:33.524 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.329+0000 7f0fe6a00700 1 -- 172.21.15.43:0/344400080 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fe810ae80 con 0x7f0fe8108550 2024-03-28T22:25:33.524 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.329+0000 7f0fe6a00700 1 --2- 172.21.15.43:0/344400080 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0fe8108550 0x7f0fe810a940 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0fd80098a0 tx=0x7f0fd8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcce8273807642de server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:33.524 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.330+0000 7f0fe5600700 1 -- 172.21.15.43:0/344400080 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0fd8004160 con 0x7f0fe8108550 2024-03-28T22:25:33.524 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.330+0000 7f0fe5600700 1 -- 172.21.15.43:0/344400080 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fd8004300 con 0x7f0fe8108550 2024-03-28T22:25:33.524 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.330+0000 7f0fe5600700 1 -- 172.21.15.43:0/344400080 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0fd8011b20 con 0x7f0fe8108550 2024-03-28T22:25:33.524 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.331+0000 7f0fe7e00700 1 -- 172.21.15.43:0/344400080 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0fe8108550 msgr2=0x7f0fe810a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:33.524 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.331+0000 7f0fe7e00700 1 --2- 172.21.15.43:0/344400080 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0fe8108550 0x7f0fe810a940 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0fd80098a0 tx=0x7f0fd8009bb0 comp rx=0 tx=0).stop 2024-03-28T22:25:33.524 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.331+0000 7f0fe7e00700 1 -- 172.21.15.43:0/344400080 shutdown_connections 2024-03-28T22:25:33.525 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.331+0000 7f0fe7e00700 1 --2- 172.21.15.43:0/344400080 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0fe8108550 0x7f0fe810a940 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:33.525 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.331+0000 7f0fe7e00700 1 -- 172.21.15.43:0/344400080 >> 172.21.15.43:0/344400080 conn(0x7f0fe8101fc0 msgr2=0x7f0fe81043e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:33.525 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.331+0000 7f0fe7e00700 1 -- 172.21.15.43:0/344400080 shutdown_connections 2024-03-28T22:25:33.525 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.331+0000 7f0fe7e00700 1 -- 172.21.15.43:0/344400080 wait complete. 2024-03-28T22:25:33.525 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.332+0000 7f0fe7e00700 1 Processor -- start 2024-03-28T22:25:33.525 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.332+0000 7f0fe7e00700 1 -- start start 2024-03-28T22:25:33.525 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.333+0000 7f0fe7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0fe8108550 0x7f0fe8107dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:33.525 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.333+0000 7f0fe7e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0fe810bef0 con 0x7f0fe8108550 2024-03-28T22:25:33.525 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.333+0000 7f0fe6a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0fe8108550 0x7f0fe8107dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:33.525 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.333+0000 7f0fe6a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0fe8108550 0x7f0fe8107dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55210/0 (socket says 172.21.15.43:55210) 2024-03-28T22:25:33.525 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.333+0000 7f0fe6a00700 1 -- 172.21.15.43:0/1887129795 learned_addr learned my addr 172.21.15.43:0/1887129795 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:33.525 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.333+0000 7f0fe6a00700 1 -- 172.21.15.43:0/1887129795 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fe8108310 con 0x7f0fe8108550 2024-03-28T22:25:33.525 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.334+0000 7f0fe6a00700 1 --2- 172.21.15.43:0/1887129795 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0fe8108550 0x7f0fe8107dd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0fd80117e0 tx=0x7f0fd8011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:33.525 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.334+0000 7f0feca00700 1 -- 172.21.15.43:0/1887129795 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0fd8003ba0 con 0x7f0fe8108550 2024-03-28T22:25:33.525 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.334+0000 7f0feca00700 1 -- 172.21.15.43:0/1887129795 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fd801a910 con 0x7f0fe8108550 2024-03-28T22:25:33.526 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.334+0000 7f0feca00700 1 -- 172.21.15.43:0/1887129795 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0fd80248f0 con 0x7f0fe8108550 2024-03-28T22:25:33.526 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.334+0000 7f0fe7e00700 1 -- 172.21.15.43:0/1887129795 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fe8106500 con 0x7f0fe8108550 2024-03-28T22:25:33.526 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.334+0000 7f0fe7e00700 1 -- 172.21.15.43:0/1887129795 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fe81069c0 con 0x7f0fe8108550 2024-03-28T22:25:33.526 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.336+0000 7f0feca00700 1 -- 172.21.15.43:0/1887129795 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 12) ==== 45941+0+0 (secure 0 0 0) 0x7f0fd8022020 con 0x7f0fe8108550 2024-03-28T22:25:33.526 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.336+0000 7f0fe7e00700 1 -- 172.21.15.43:0/1887129795 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fe8063870 con 0x7f0fe8108550 2024-03-28T22:25:33.526 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.336+0000 7f0feca00700 1 --2- 172.21.15.43:0/1887129795 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f0fcc038c50 0x7f0fcc03b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:33.526 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.336+0000 7f0feca00700 1 -- 172.21.15.43:0/1887129795 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0fd804cf90 con 0x7f0fe8108550 2024-03-28T22:25:33.526 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.337+0000 7f0fe6000700 1 --2- 172.21.15.43:0/1887129795 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f0fcc038c50 0x7f0fcc03b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:33.526 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.337+0000 7f0fe6000700 1 --2- 172.21.15.43:0/1887129795 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f0fcc038c50 0x7f0fcc03b110 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4006fd0 tx=0x7f0fd4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:33.526 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.345+0000 7f0feca00700 1 -- 172.21.15.43:0/1887129795 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0fd800b3e0 con 0x7f0fe8108550 2024-03-28T22:25:33.526 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.592+0000 7f0fe7e00700 1 -- 172.21.15.43:0/1887129795 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f0fe81070e0 con 0x7f0fcc038c50 2024-03-28T22:25:33.526 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.861+0000 7f0feca00700 1 -- 172.21.15.43:0/1887129795 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f0fe81070e0 con 0x7f0fcc038c50 2024-03-28T22:25:33.526 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.864+0000 7f0fe7e00700 1 -- 172.21.15.43:0/1887129795 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f0fcc038c50 msgr2=0x7f0fcc03b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:33.527 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.864+0000 7f0fe7e00700 1 --2- 172.21.15.43:0/1887129795 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f0fcc038c50 0x7f0fcc03b110 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4006fd0 tx=0x7f0fd4006e40 comp rx=0 tx=0).stop 2024-03-28T22:25:33.527 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.864+0000 7f0fe7e00700 1 -- 172.21.15.43:0/1887129795 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0fe8108550 msgr2=0x7f0fe8107dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:33.527 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.864+0000 7f0fe7e00700 1 --2- 172.21.15.43:0/1887129795 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0fe8108550 0x7f0fe8107dd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0fd80117e0 tx=0x7f0fd8011ba0 comp rx=0 tx=0).stop 2024-03-28T22:25:33.527 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.864+0000 7f0fe7e00700 1 -- 172.21.15.43:0/1887129795 shutdown_connections 2024-03-28T22:25:33.527 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.864+0000 7f0fe7e00700 1 --2- 172.21.15.43:0/1887129795 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f0fcc038c50 0x7f0fcc03b110 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:33.527 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.864+0000 7f0fe7e00700 1 --2- 172.21.15.43:0/1887129795 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0fe8108550 0x7f0fe8107dd0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:33.527 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.864+0000 7f0fe7e00700 1 -- 172.21.15.43:0/1887129795 >> 172.21.15.43:0/1887129795 conn(0x7f0fe8101fc0 msgr2=0x7f0fe8102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:33.527 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.864+0000 7f0fe7e00700 1 -- 172.21.15.43:0/1887129795 shutdown_connections 2024-03-28T22:25:33.527 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:32.865+0000 7f0fe7e00700 1 -- 172.21.15.43:0/1887129795 wait complete. 2024-03-28T22:25:33.527 INFO:teuthology.orchestra.run.smithi043.stdout:Fetching dashboard port number... 2024-03-28T22:25:34.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:33 smithi043 ceph-mon[31823]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:25:34.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:33 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:25:34.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:33 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:25:35.008 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:34 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2548766026' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-28T22:25:35.664 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 8443 2024-03-28T22:25:35.665 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.146+0000 7faa61400700 1 Processor -- start 2024-03-28T22:25:35.665 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.146+0000 7faa61400700 1 -- start start 2024-03-28T22:25:35.665 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.146+0000 7faa61400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa5c108630 0x7faa5c10aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:35.665 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.147+0000 7faa61400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7faa5c07bdd0 con 0x7faa5c108630 2024-03-28T22:25:35.665 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.147+0000 7faa5be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa5c108630 0x7faa5c10aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:35.665 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.147+0000 7faa5be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa5c108630 0x7faa5c10aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55226/0 (socket says 172.21.15.43:55226) 2024-03-28T22:25:35.665 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.147+0000 7faa5be00700 1 -- 172.21.15.43:0/2605589914 learned_addr learned my addr 172.21.15.43:0/2605589914 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:35.665 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.147+0000 7faa5be00700 1 -- 172.21.15.43:0/2605589914 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa5c10af60 con 0x7faa5c108630 2024-03-28T22:25:35.665 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.148+0000 7faa5be00700 1 --2- 172.21.15.43:0/2605589914 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa5c108630 0x7faa5c10aa20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7faa4c0098a0 tx=0x7faa4c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77dd02a53e3f3789 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:35.665 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.148+0000 7faa5aa00700 1 -- 172.21.15.43:0/2605589914 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faa4c004160 con 0x7faa5c108630 2024-03-28T22:25:35.665 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.148+0000 7faa5aa00700 1 -- 172.21.15.43:0/2605589914 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa4c004300 con 0x7faa5c108630 2024-03-28T22:25:35.665 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.148+0000 7faa5aa00700 1 -- 172.21.15.43:0/2605589914 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faa4c011b20 con 0x7faa5c108630 2024-03-28T22:25:35.669 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.149+0000 7faa61400700 1 -- 172.21.15.43:0/2605589914 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa5c108630 msgr2=0x7faa5c10aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:35.669 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.149+0000 7faa61400700 1 --2- 172.21.15.43:0/2605589914 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa5c108630 0x7faa5c10aa20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7faa4c0098a0 tx=0x7faa4c009bb0 comp rx=0 tx=0).stop 2024-03-28T22:25:35.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.149+0000 7faa61400700 1 -- 172.21.15.43:0/2605589914 shutdown_connections 2024-03-28T22:25:35.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.149+0000 7faa61400700 1 --2- 172.21.15.43:0/2605589914 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa5c108630 0x7faa5c10aa20 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:35.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.149+0000 7faa61400700 1 -- 172.21.15.43:0/2605589914 >> 172.21.15.43:0/2605589914 conn(0x7faa5c101fc0 msgr2=0x7faa5c1043e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:35.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.149+0000 7faa61400700 1 -- 172.21.15.43:0/2605589914 shutdown_connections 2024-03-28T22:25:35.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.149+0000 7faa61400700 1 -- 172.21.15.43:0/2605589914 wait complete. 2024-03-28T22:25:35.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.150+0000 7faa61400700 1 Processor -- start 2024-03-28T22:25:35.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.151+0000 7faa61400700 1 -- start start 2024-03-28T22:25:35.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.151+0000 7faa61400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa5c108630 0x7faa5c19ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:35.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.151+0000 7faa61400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7faa5c10bf00 con 0x7faa5c108630 2024-03-28T22:25:35.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.151+0000 7faa5be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa5c108630 0x7faa5c19ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:35.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.151+0000 7faa5be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa5c108630 0x7faa5c19ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55230/0 (socket says 172.21.15.43:55230) 2024-03-28T22:25:35.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.151+0000 7faa5be00700 1 -- 172.21.15.43:0/2548766026 learned_addr learned my addr 172.21.15.43:0/2548766026 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:35.670 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.152+0000 7faa5be00700 1 -- 172.21.15.43:0/2548766026 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa5c19bfc0 con 0x7faa5c108630 2024-03-28T22:25:35.671 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.152+0000 7faa5be00700 1 --2- 172.21.15.43:0/2548766026 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa5c108630 0x7faa5c19ba80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7faa4c0117e0 tx=0x7faa4c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:35.671 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.152+0000 7faa58c00700 1 -- 172.21.15.43:0/2548766026 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faa4c003ba0 con 0x7faa5c108630 2024-03-28T22:25:35.671 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.152+0000 7faa58c00700 1 -- 172.21.15.43:0/2548766026 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa4c01a910 con 0x7faa5c108630 2024-03-28T22:25:35.671 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.152+0000 7faa61400700 1 -- 172.21.15.43:0/2548766026 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa5c19c250 con 0x7faa5c108630 2024-03-28T22:25:35.671 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.152+0000 7faa58c00700 1 -- 172.21.15.43:0/2548766026 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faa4c0248f0 con 0x7faa5c108630 2024-03-28T22:25:35.671 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.152+0000 7faa61400700 1 -- 172.21.15.43:0/2548766026 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa5c19c6b0 con 0x7faa5c108630 2024-03-28T22:25:35.671 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.153+0000 7faa58c00700 1 -- 172.21.15.43:0/2548766026 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 12) ==== 45941+0+0 (secure 0 0 0) 0x7faa4c022020 con 0x7faa5c108630 2024-03-28T22:25:35.671 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.154+0000 7faa58c00700 1 --2- 172.21.15.43:0/2548766026 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7faa40038c50 0x7faa4003b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:35.671 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.154+0000 7faa61400700 1 -- 172.21.15.43:0/2548766026 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa5c063870 con 0x7faa5c108630 2024-03-28T22:25:35.671 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.154+0000 7faa58c00700 1 -- 172.21.15.43:0/2548766026 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7faa4c04d060 con 0x7faa5c108630 2024-03-28T22:25:35.671 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.154+0000 7faa5b400700 1 --2- 172.21.15.43:0/2548766026 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7faa40038c50 0x7faa4003b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:35.671 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.155+0000 7faa5b400700 1 --2- 172.21.15.43:0/2548766026 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7faa40038c50 0x7faa4003b110 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7faa48006fd0 tx=0x7faa48006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:35.671 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.160+0000 7faa58c00700 1 -- 172.21.15.43:0/2548766026 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7faa4c00b3e0 con 0x7faa5c108630 2024-03-28T22:25:35.671 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.382+0000 7faa61400700 1 -- 172.21.15.43:0/2548766026 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7faa5c19f080 con 0x7faa5c108630 2024-03-28T22:25:35.672 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.382+0000 7faa58c00700 1 -- 172.21.15.43:0/2548766026 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7faa4c01e120 con 0x7faa5c108630 2024-03-28T22:25:35.672 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.385+0000 7faa61400700 1 -- 172.21.15.43:0/2548766026 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7faa40038c50 msgr2=0x7faa4003b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:35.672 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.385+0000 7faa61400700 1 --2- 172.21.15.43:0/2548766026 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7faa40038c50 0x7faa4003b110 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7faa48006fd0 tx=0x7faa48006e40 comp rx=0 tx=0).stop 2024-03-28T22:25:35.672 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.385+0000 7faa61400700 1 -- 172.21.15.43:0/2548766026 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa5c108630 msgr2=0x7faa5c19ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:35.672 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.385+0000 7faa61400700 1 --2- 172.21.15.43:0/2548766026 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa5c108630 0x7faa5c19ba80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7faa4c0117e0 tx=0x7faa4c011ba0 comp rx=0 tx=0).stop 2024-03-28T22:25:35.672 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.385+0000 7faa61400700 1 -- 172.21.15.43:0/2548766026 shutdown_connections 2024-03-28T22:25:35.672 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.385+0000 7faa61400700 1 --2- 172.21.15.43:0/2548766026 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7faa40038c50 0x7faa4003b110 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:35.672 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.385+0000 7faa61400700 1 --2- 172.21.15.43:0/2548766026 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa5c108630 0x7faa5c19ba80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:35.672 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.385+0000 7faa61400700 1 -- 172.21.15.43:0/2548766026 >> 172.21.15.43:0/2548766026 conn(0x7faa5c101fc0 msgr2=0x7faa5c102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:35.672 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.385+0000 7faa61400700 1 -- 172.21.15.43:0/2548766026 shutdown_connections 2024-03-28T22:25:35.672 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:34.385+0000 7faa61400700 1 -- 172.21.15.43:0/2548766026 wait complete. 2024-03-28T22:25:35.705 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T22:25:35.706 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout disabled 2024-03-28T22:25:35.732 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T22:25:35.732 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout inactive 2024-03-28T22:25:35.733 INFO:teuthology.orchestra.run.smithi043.stdout:firewalld.service is not enabled 2024-03-28T22:25:35.733 INFO:teuthology.orchestra.run.smithi043.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-28T22:25:35.741 INFO:teuthology.orchestra.run.smithi043.stdout:Ceph Dashboard is now available at: 2024-03-28T22:25:35.741 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:25:35.741 INFO:teuthology.orchestra.run.smithi043.stdout: URL: https://smithi043.front.sepia.ceph.com:8443/ 2024-03-28T22:25:35.741 INFO:teuthology.orchestra.run.smithi043.stdout: User: admin 2024-03-28T22:25:35.741 INFO:teuthology.orchestra.run.smithi043.stdout: Password: zky7sttg9j 2024-03-28T22:25:35.742 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:25:35.742 INFO:teuthology.orchestra.run.smithi043.stdout:Saving cluster configuration to /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config directory 2024-03-28T22:25:35.744 INFO:teuthology.orchestra.run.smithi043.stdout:Enabling autotune for osd_memory_target 2024-03-28T22:25:35.912 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:35 smithi043 ceph-mon[31823]: mgrmap e13: x(active, since 6s) 2024-03-28T22:25:37.200 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.346+0000 7ff29e400700 1 Processor -- start 2024-03-28T22:25:37.200 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.346+0000 7ff29e400700 1 -- start start 2024-03-28T22:25:37.200 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.347+0000 7ff29e400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff29807ddc0 0x7ff29807c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:37.200 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.347+0000 7ff29e400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff29807c800 con 0x7ff29807ddc0 2024-03-28T22:25:37.200 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.347+0000 7ff297400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff29807ddc0 0x7ff29807c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:37.200 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.347+0000 7ff297400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff29807ddc0 0x7ff29807c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34472/0 (socket says 172.21.15.43:34472) 2024-03-28T22:25:37.200 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.347+0000 7ff297400700 1 -- 172.21.15.43:0/1261229729 learned_addr learned my addr 172.21.15.43:0/1261229729 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:37.200 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.348+0000 7ff297400700 1 -- 172.21.15.43:0/1261229729 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff29807c980 con 0x7ff29807ddc0 2024-03-28T22:25:37.201 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.348+0000 7ff297400700 1 --2- 172.21.15.43:0/1261229729 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff29807ddc0 0x7ff29807c2c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff2880098a0 tx=0x7ff288009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bce064816c5590b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:37.201 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.348+0000 7ff296000700 1 -- 172.21.15.43:0/1261229729 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff288004160 con 0x7ff29807ddc0 2024-03-28T22:25:37.201 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.348+0000 7ff296000700 1 -- 172.21.15.43:0/1261229729 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff288004300 con 0x7ff29807ddc0 2024-03-28T22:25:37.206 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.348+0000 7ff296000700 1 -- 172.21.15.43:0/1261229729 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff288011b20 con 0x7ff29807ddc0 2024-03-28T22:25:37.206 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.349+0000 7ff29e400700 1 -- 172.21.15.43:0/1261229729 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff29807ddc0 msgr2=0x7ff29807c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:37.206 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.349+0000 7ff29e400700 1 --2- 172.21.15.43:0/1261229729 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff29807ddc0 0x7ff29807c2c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff2880098a0 tx=0x7ff288009bb0 comp rx=0 tx=0).stop 2024-03-28T22:25:37.206 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.349+0000 7ff29e400700 1 -- 172.21.15.43:0/1261229729 shutdown_connections 2024-03-28T22:25:37.206 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.349+0000 7ff29e400700 1 --2- 172.21.15.43:0/1261229729 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff29807ddc0 0x7ff29807c2c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:37.206 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.349+0000 7ff29e400700 1 -- 172.21.15.43:0/1261229729 >> 172.21.15.43:0/1261229729 conn(0x7ff298103340 msgr2=0x7ff298105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:37.206 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.349+0000 7ff29e400700 1 -- 172.21.15.43:0/1261229729 shutdown_connections 2024-03-28T22:25:37.206 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.349+0000 7ff29e400700 1 -- 172.21.15.43:0/1261229729 wait complete. 2024-03-28T22:25:37.206 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.350+0000 7ff29e400700 1 Processor -- start 2024-03-28T22:25:37.206 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.351+0000 7ff29e400700 1 -- start start 2024-03-28T22:25:37.206 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.351+0000 7ff29e400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff29807ddc0 0x7ff29819de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:37.207 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.351+0000 7ff29e400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff29810e300 con 0x7ff29807ddc0 2024-03-28T22:25:37.207 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.351+0000 7ff297400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff29807ddc0 0x7ff29819de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:37.207 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.351+0000 7ff297400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff29807ddc0 0x7ff29819de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34480/0 (socket says 172.21.15.43:34480) 2024-03-28T22:25:37.207 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.351+0000 7ff297400700 1 -- 172.21.15.43:0/401463956 learned_addr learned my addr 172.21.15.43:0/401463956 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:37.207 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.352+0000 7ff297400700 1 -- 172.21.15.43:0/401463956 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff29819e370 con 0x7ff29807ddc0 2024-03-28T22:25:37.207 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.352+0000 7ff297400700 1 --2- 172.21.15.43:0/401463956 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff29807ddc0 0x7ff29819de30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff288011b60 tx=0x7ff288011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:37.207 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.352+0000 7ff28fe00700 1 -- 172.21.15.43:0/401463956 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff288003b60 con 0x7ff29807ddc0 2024-03-28T22:25:37.207 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.352+0000 7ff28fe00700 1 -- 172.21.15.43:0/401463956 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff28801a910 con 0x7ff29807ddc0 2024-03-28T22:25:37.207 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.352+0000 7ff29e400700 1 -- 172.21.15.43:0/401463956 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff29819e600 con 0x7ff29807ddc0 2024-03-28T22:25:37.207 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.352+0000 7ff28fe00700 1 -- 172.21.15.43:0/401463956 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff288024930 con 0x7ff29807ddc0 2024-03-28T22:25:37.207 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.352+0000 7ff29e400700 1 -- 172.21.15.43:0/401463956 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff29819ea60 con 0x7ff29807ddc0 2024-03-28T22:25:37.207 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.353+0000 7ff28fe00700 1 -- 172.21.15.43:0/401463956 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7ff288022020 con 0x7ff29807ddc0 2024-03-28T22:25:37.207 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.354+0000 7ff29e400700 1 -- 172.21.15.43:0/401463956 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff298063870 con 0x7ff29807ddc0 2024-03-28T22:25:37.207 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.354+0000 7ff28fe00700 1 --2- 172.21.15.43:0/401463956 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff27c038cf0 0x7ff27c03b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:37.208 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.354+0000 7ff28fe00700 1 -- 172.21.15.43:0/401463956 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff28804cf90 con 0x7ff29807ddc0 2024-03-28T22:25:37.208 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.354+0000 7ff296a00700 1 --2- 172.21.15.43:0/401463956 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff27c038cf0 0x7ff27c03b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:37.208 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.355+0000 7ff296a00700 1 --2- 172.21.15.43:0/401463956 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff27c038cf0 0x7ff27c03b1b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff284006fd0 tx=0x7ff284006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:37.208 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.360+0000 7ff28fe00700 1 -- 172.21.15.43:0/401463956 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff28801ab30 con 0x7ff29807ddc0 2024-03-28T22:25:37.208 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.583+0000 7ff29e400700 1 -- 172.21.15.43:0/401463956 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7ff2981a14f0 con 0x7ff29807ddc0 2024-03-28T22:25:37.208 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.584+0000 7ff28fe00700 1 -- 172.21.15.43:0/401463956 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7ff28802d770 con 0x7ff29807ddc0 2024-03-28T22:25:37.208 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.586+0000 7ff29e400700 1 -- 172.21.15.43:0/401463956 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff27c038cf0 msgr2=0x7ff27c03b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:37.208 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.587+0000 7ff29e400700 1 --2- 172.21.15.43:0/401463956 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff27c038cf0 0x7ff27c03b1b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff284006fd0 tx=0x7ff284006e40 comp rx=0 tx=0).stop 2024-03-28T22:25:37.208 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.587+0000 7ff29e400700 1 -- 172.21.15.43:0/401463956 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff29807ddc0 msgr2=0x7ff29819de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:37.208 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.587+0000 7ff29e400700 1 --2- 172.21.15.43:0/401463956 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff29807ddc0 0x7ff29819de30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff288011b60 tx=0x7ff288011c40 comp rx=0 tx=0).stop 2024-03-28T22:25:37.208 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.587+0000 7ff29e400700 1 -- 172.21.15.43:0/401463956 shutdown_connections 2024-03-28T22:25:37.208 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.587+0000 7ff29e400700 1 --2- 172.21.15.43:0/401463956 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff27c038cf0 0x7ff27c03b1b0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:37.208 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.587+0000 7ff29e400700 1 --2- 172.21.15.43:0/401463956 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff29807ddc0 0x7ff29819de30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:37.208 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.587+0000 7ff29e400700 1 -- 172.21.15.43:0/401463956 >> 172.21.15.43:0/401463956 conn(0x7ff298103340 msgr2=0x7ff298103de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:37.209 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.587+0000 7ff29e400700 1 -- 172.21.15.43:0/401463956 shutdown_connections 2024-03-28T22:25:37.209 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:36.587+0000 7ff29e400700 1 -- 172.21.15.43:0/401463956 wait complete. 2024-03-28T22:25:38.793 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.802+0000 7fa88a400700 1 Processor -- start 2024-03-28T22:25:38.794 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.803+0000 7fa88a400700 1 -- start start 2024-03-28T22:25:38.794 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.803+0000 7fa88a400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8841069e0 0x7fa884106e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:38.794 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.803+0000 7fa88a400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa8841073d0 con 0x7fa8841069e0 2024-03-28T22:25:38.794 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.804+0000 7fa889000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8841069e0 0x7fa884106e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:38.794 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.804+0000 7fa889000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8841069e0 0x7fa884106e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34494/0 (socket says 172.21.15.43:34494) 2024-03-28T22:25:38.794 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.804+0000 7fa889000700 1 -- 172.21.15.43:0/3997470063 learned_addr learned my addr 172.21.15.43:0/3997470063 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:38.794 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.804+0000 7fa889000700 1 -- 172.21.15.43:0/3997470063 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa884107be0 con 0x7fa8841069e0 2024-03-28T22:25:38.794 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.805+0000 7fa889000700 1 --2- 172.21.15.43:0/3997470063 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8841069e0 0x7fa884106e00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa87800bb40 tx=0x7fa87800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed72806d3ad4ac34 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:38.794 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.805+0000 7fa883400700 1 -- 172.21.15.43:0/3997470063 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa87800dbe0 con 0x7fa8841069e0 2024-03-28T22:25:38.794 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.805+0000 7fa883400700 1 -- 172.21.15.43:0/3997470063 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa878004510 con 0x7fa8841069e0 2024-03-28T22:25:38.794 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.805+0000 7fa883400700 1 -- 172.21.15.43:0/3997470063 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa878010470 con 0x7fa8841069e0 2024-03-28T22:25:38.794 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.806+0000 7fa88a400700 1 -- 172.21.15.43:0/3997470063 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8841069e0 msgr2=0x7fa884106e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:38.794 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.806+0000 7fa88a400700 1 --2- 172.21.15.43:0/3997470063 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8841069e0 0x7fa884106e00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa87800bb40 tx=0x7fa87800be50 comp rx=0 tx=0).stop 2024-03-28T22:25:38.799 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.806+0000 7fa88a400700 1 -- 172.21.15.43:0/3997470063 shutdown_connections 2024-03-28T22:25:38.799 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.806+0000 7fa88a400700 1 --2- 172.21.15.43:0/3997470063 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8841069e0 0x7fa884106e00 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:38.799 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.806+0000 7fa88a400700 1 -- 172.21.15.43:0/3997470063 >> 172.21.15.43:0/3997470063 conn(0x7fa884101fc0 msgr2=0x7fa8841043e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:38.799 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.806+0000 7fa88a400700 1 -- 172.21.15.43:0/3997470063 shutdown_connections 2024-03-28T22:25:38.799 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.806+0000 7fa88a400700 1 -- 172.21.15.43:0/3997470063 wait complete. 2024-03-28T22:25:38.799 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.807+0000 7fa88a400700 1 Processor -- start 2024-03-28T22:25:38.799 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.808+0000 7fa88a400700 1 -- start start 2024-03-28T22:25:38.799 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.808+0000 7fa88a400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8841069e0 0x7fa88419dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:38.799 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.808+0000 7fa88a400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa88407afd0 con 0x7fa8841069e0 2024-03-28T22:25:38.799 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.808+0000 7fa889000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8841069e0 0x7fa88419dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:38.800 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.808+0000 7fa889000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8841069e0 0x7fa88419dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34508/0 (socket says 172.21.15.43:34508) 2024-03-28T22:25:38.800 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.808+0000 7fa889000700 1 -- 172.21.15.43:0/1254205488 learned_addr learned my addr 172.21.15.43:0/1254205488 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:38.800 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.809+0000 7fa889000700 1 -- 172.21.15.43:0/1254205488 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa88419e2d0 con 0x7fa8841069e0 2024-03-28T22:25:38.800 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.809+0000 7fa889000700 1 --2- 172.21.15.43:0/1254205488 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8841069e0 0x7fa88419dd90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa8780046e0 tx=0x7fa8780040b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:38.800 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.809+0000 7fa881600700 1 -- 172.21.15.43:0/1254205488 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa878010720 con 0x7fa8841069e0 2024-03-28T22:25:38.800 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.809+0000 7fa881600700 1 -- 172.21.15.43:0/1254205488 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa87801d070 con 0x7fa8841069e0 2024-03-28T22:25:38.800 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.809+0000 7fa881600700 1 -- 172.21.15.43:0/1254205488 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa878021b10 con 0x7fa8841069e0 2024-03-28T22:25:38.800 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.809+0000 7fa88a400700 1 -- 172.21.15.43:0/1254205488 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa88419e560 con 0x7fa8841069e0 2024-03-28T22:25:38.800 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.810+0000 7fa88a400700 1 -- 172.21.15.43:0/1254205488 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8841a1210 con 0x7fa8841069e0 2024-03-28T22:25:38.800 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.811+0000 7fa881600700 1 -- 172.21.15.43:0/1254205488 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7fa87801f030 con 0x7fa8841069e0 2024-03-28T22:25:38.800 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.811+0000 7fa88a400700 1 -- 172.21.15.43:0/1254205488 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa884063870 con 0x7fa8841069e0 2024-03-28T22:25:38.800 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.811+0000 7fa881600700 1 --2- 172.21.15.43:0/1254205488 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa86c038ce0 0x7fa86c03b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:38.800 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.811+0000 7fa881600700 1 -- 172.21.15.43:0/1254205488 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa87804ced0 con 0x7fa8841069e0 2024-03-28T22:25:38.800 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.811+0000 7fa883e00700 1 --2- 172.21.15.43:0/1254205488 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa86c038ce0 0x7fa86c03b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:38.800 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.812+0000 7fa883e00700 1 --2- 172.21.15.43:0/1254205488 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa86c038ce0 0x7fa86c03b1a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa870006fd0 tx=0x7fa870006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:38.801 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:37.817+0000 7fa881600700 1 -- 172.21.15.43:0/1254205488 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa878015070 con 0x7fa8841069e0 2024-03-28T22:25:38.801 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:38.157+0000 7fa88a400700 1 -- 172.21.15.43:0/1254205488 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7fa88402df00 con 0x7fa8841069e0 2024-03-28T22:25:38.801 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:38.160+0000 7fa881600700 1 -- 172.21.15.43:0/1254205488 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7fa8780183a0 con 0x7fa8841069e0 2024-03-28T22:25:38.801 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:38.163+0000 7fa88a400700 1 -- 172.21.15.43:0/1254205488 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa86c038ce0 msgr2=0x7fa86c03b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:38.801 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:38.163+0000 7fa88a400700 1 --2- 172.21.15.43:0/1254205488 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa86c038ce0 0x7fa86c03b1a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa870006fd0 tx=0x7fa870006e40 comp rx=0 tx=0).stop 2024-03-28T22:25:38.801 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:38.163+0000 7fa88a400700 1 -- 172.21.15.43:0/1254205488 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8841069e0 msgr2=0x7fa88419dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:38.801 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:38.163+0000 7fa88a400700 1 --2- 172.21.15.43:0/1254205488 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8841069e0 0x7fa88419dd90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa8780046e0 tx=0x7fa8780040b0 comp rx=0 tx=0).stop 2024-03-28T22:25:38.801 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:38.163+0000 7fa88a400700 1 -- 172.21.15.43:0/1254205488 shutdown_connections 2024-03-28T22:25:38.801 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:38.163+0000 7fa88a400700 1 --2- 172.21.15.43:0/1254205488 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa86c038ce0 0x7fa86c03b1a0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:38.801 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:38.163+0000 7fa88a400700 1 --2- 172.21.15.43:0/1254205488 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8841069e0 0x7fa88419dd90 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:38.801 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:38.163+0000 7fa88a400700 1 -- 172.21.15.43:0/1254205488 >> 172.21.15.43:0/1254205488 conn(0x7fa884101fc0 msgr2=0x7fa8841903b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:38.801 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:38.163+0000 7fa88a400700 1 -- 172.21.15.43:0/1254205488 shutdown_connections 2024-03-28T22:25:38.801 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr 2024-03-28T22:25:38.163+0000 7fa88a400700 1 -- 172.21.15.43:0/1254205488 wait complete. 2024-03-28T22:25:38.801 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-28T22:25:38.802 INFO:teuthology.orchestra.run.smithi043.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-28T22:25:38.802 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:25:38.802 INFO:teuthology.orchestra.run.smithi043.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-28T22:25:38.802 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:25:38.802 INFO:teuthology.orchestra.run.smithi043.stdout:Or, if you are only running a single cluster on this host: 2024-03-28T22:25:38.802 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:25:38.802 INFO:teuthology.orchestra.run.smithi043.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-28T22:25:38.802 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:25:38.802 INFO:teuthology.orchestra.run.smithi043.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-28T22:25:38.802 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:25:38.802 INFO:teuthology.orchestra.run.smithi043.stdout: ceph telemetry on 2024-03-28T22:25:38.802 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:25:38.802 INFO:teuthology.orchestra.run.smithi043.stdout:For more information see: 2024-03-28T22:25:38.802 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:25:38.803 INFO:teuthology.orchestra.run.smithi043.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-28T22:25:38.803 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:25:38.803 INFO:teuthology.orchestra.run.smithi043.stdout:Bootstrap complete. 2024-03-28T22:25:38.905 INFO:tasks.cephadm:Fetching config... 2024-03-28T22:25:38.905 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-28T22:25:38.905 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-28T22:25:38.951 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-28T22:25:38.951 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-28T22:25:38.951 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-28T22:25:39.040 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-28T22:25:39.040 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-28T22:25:39.040 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-28T22:25:39.172 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-28T22:25:39.172 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-28T22:25:39.172 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-28T22:25:39.270 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-28T22:25:39.271 DEBUG:teuthology.orchestra.run.smithi043:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSJrviJP/lLx0yXeoEueK+37ysqRX1Pj/x7QVJ2i4HjAcaGrnNEHOicYaWaZHCaXkownQjKmcexQHdPvCOR86L6kJAbkzonLDQ2JltQEZlWCimyNE5TBylHUm9YBatuBBtL3+eHrIFpRrysR314a8g2xoSEeaFKhG4LoVouYA0E82NFr9GAPjUHV9ZITZ+SiN93r9xugfsn/AyX861CCIiVdZ+RA8GEHpvQwhCRAKb2xCiXn7QHD0OPWl/iDekEgfPIoCMPet9yxER14MHXpQ+vgwCYITii7AEE3XVvkByFhWMWU8LWgHUzbCxVI7Ddj2Ha8IBCV595/MhnxyC0NelWLLoqljFXy8gitiLj30UoseNoIYLSu6Vgpz+K2Obxx3Bmfg0Y9N0NdS8tRqQIw7GMsGo5T7T4zserDYAIZ7kDsPY+VtizDJ6TiZjVWSTpFoLBaxteJO0LHrQQ7gD4+FfAfWwalB/mY/2VcRWOxbVOA7+Tdm4rSL1Bfg02SWOUzs= ceph-c2276428-ed51-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T22:25:39.393 INFO:teuthology.orchestra.run.smithi043.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSJrviJP/lLx0yXeoEueK+37ysqRX1Pj/x7QVJ2i4HjAcaGrnNEHOicYaWaZHCaXkownQjKmcexQHdPvCOR86L6kJAbkzonLDQ2JltQEZlWCimyNE5TBylHUm9YBatuBBtL3+eHrIFpRrysR314a8g2xoSEeaFKhG4LoVouYA0E82NFr9GAPjUHV9ZITZ+SiN93r9xugfsn/AyX861CCIiVdZ+RA8GEHpvQwhCRAKb2xCiXn7QHD0OPWl/iDekEgfPIoCMPet9yxER14MHXpQ+vgwCYITii7AEE3XVvkByFhWMWU8LWgHUzbCxVI7Ddj2Ha8IBCV595/MhnxyC0NelWLLoqljFXy8gitiLj30UoseNoIYLSu6Vgpz+K2Obxx3Bmfg0Y9N0NdS8tRqQIw7GMsGo5T7T4zserDYAIZ7kDsPY+VtizDJ6TiZjVWSTpFoLBaxteJO0LHrQQ7gD4+FfAfWwalB/mY/2VcRWOxbVOA7+Tdm4rSL1Bfg02SWOUzs= ceph-c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:25:39.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:39 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1254205488' entity='client.admin' 2024-03-28T22:25:39.461 DEBUG:teuthology.orchestra.run.smithi082:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSJrviJP/lLx0yXeoEueK+37ysqRX1Pj/x7QVJ2i4HjAcaGrnNEHOicYaWaZHCaXkownQjKmcexQHdPvCOR86L6kJAbkzonLDQ2JltQEZlWCimyNE5TBylHUm9YBatuBBtL3+eHrIFpRrysR314a8g2xoSEeaFKhG4LoVouYA0E82NFr9GAPjUHV9ZITZ+SiN93r9xugfsn/AyX861CCIiVdZ+RA8GEHpvQwhCRAKb2xCiXn7QHD0OPWl/iDekEgfPIoCMPet9yxER14MHXpQ+vgwCYITii7AEE3XVvkByFhWMWU8LWgHUzbCxVI7Ddj2Ha8IBCV595/MhnxyC0NelWLLoqljFXy8gitiLj30UoseNoIYLSu6Vgpz+K2Obxx3Bmfg0Y9N0NdS8tRqQIw7GMsGo5T7T4zserDYAIZ7kDsPY+VtizDJ6TiZjVWSTpFoLBaxteJO0LHrQQ7gD4+FfAfWwalB/mY/2VcRWOxbVOA7+Tdm4rSL1Bfg02SWOUzs= ceph-c2276428-ed51-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T22:25:39.509 INFO:teuthology.orchestra.run.smithi082.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSJrviJP/lLx0yXeoEueK+37ysqRX1Pj/x7QVJ2i4HjAcaGrnNEHOicYaWaZHCaXkownQjKmcexQHdPvCOR86L6kJAbkzonLDQ2JltQEZlWCimyNE5TBylHUm9YBatuBBtL3+eHrIFpRrysR314a8g2xoSEeaFKhG4LoVouYA0E82NFr9GAPjUHV9ZITZ+SiN93r9xugfsn/AyX861CCIiVdZ+RA8GEHpvQwhCRAKb2xCiXn7QHD0OPWl/iDekEgfPIoCMPet9yxER14MHXpQ+vgwCYITii7AEE3XVvkByFhWMWU8LWgHUzbCxVI7Ddj2Ha8IBCV595/MhnxyC0NelWLLoqljFXy8gitiLj30UoseNoIYLSu6Vgpz+K2Obxx3Bmfg0Y9N0NdS8tRqQIw7GMsGo5T7T4zserDYAIZ7kDsPY+VtizDJ6TiZjVWSTpFoLBaxteJO0LHrQQ7gD4+FfAfWwalB/mY/2VcRWOxbVOA7+Tdm4rSL1Bfg02SWOUzs= ceph-c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:25:39.524 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSJrviJP/lLx0yXeoEueK+37ysqRX1Pj/x7QVJ2i4HjAcaGrnNEHOicYaWaZHCaXkownQjKmcexQHdPvCOR86L6kJAbkzonLDQ2JltQEZlWCimyNE5TBylHUm9YBatuBBtL3+eHrIFpRrysR314a8g2xoSEeaFKhG4LoVouYA0E82NFr9GAPjUHV9ZITZ+SiN93r9xugfsn/AyX861CCIiVdZ+RA8GEHpvQwhCRAKb2xCiXn7QHD0OPWl/iDekEgfPIoCMPet9yxER14MHXpQ+vgwCYITii7AEE3XVvkByFhWMWU8LWgHUzbCxVI7Ddj2Ha8IBCV595/MhnxyC0NelWLLoqljFXy8gitiLj30UoseNoIYLSu6Vgpz+K2Obxx3Bmfg0Y9N0NdS8tRqQIw7GMsGo5T7T4zserDYAIZ7kDsPY+VtizDJ6TiZjVWSTpFoLBaxteJO0LHrQQ7gD4+FfAfWwalB/mY/2VcRWOxbVOA7+Tdm4rSL1Bfg02SWOUzs= ceph-c2276428-ed51-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T22:25:39.572 INFO:teuthology.orchestra.run.smithi172.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSJrviJP/lLx0yXeoEueK+37ysqRX1Pj/x7QVJ2i4HjAcaGrnNEHOicYaWaZHCaXkownQjKmcexQHdPvCOR86L6kJAbkzonLDQ2JltQEZlWCimyNE5TBylHUm9YBatuBBtL3+eHrIFpRrysR314a8g2xoSEeaFKhG4LoVouYA0E82NFr9GAPjUHV9ZITZ+SiN93r9xugfsn/AyX861CCIiVdZ+RA8GEHpvQwhCRAKb2xCiXn7QHD0OPWl/iDekEgfPIoCMPet9yxER14MHXpQ+vgwCYITii7AEE3XVvkByFhWMWU8LWgHUzbCxVI7Ddj2Ha8IBCV595/MhnxyC0NelWLLoqljFXy8gitiLj30UoseNoIYLSu6Vgpz+K2Obxx3Bmfg0Y9N0NdS8tRqQIw7GMsGo5T7T4zserDYAIZ7kDsPY+VtizDJ6TiZjVWSTpFoLBaxteJO0LHrQQ7gD4+FfAfWwalB/mY/2VcRWOxbVOA7+Tdm4rSL1Bfg02SWOUzs= ceph-c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:25:39.587 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-28T22:25:40.015 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:25:42.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.438+0000 7f4dea000700 1 -- 172.21.15.43:0/4116788040 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4de4103cf0 msgr2=0x7f4de4104110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:42.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.438+0000 7f4dea000700 1 --2- 172.21.15.43:0/4116788040 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4de4103cf0 0x7f4de4104110 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8009910 tx=0x7f4dd8009c20 comp rx=0 tx=0).stop 2024-03-28T22:25:42.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.438+0000 7f4dea000700 1 -- 172.21.15.43:0/4116788040 shutdown_connections 2024-03-28T22:25:42.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.438+0000 7f4dea000700 1 --2- 172.21.15.43:0/4116788040 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4de4103cf0 0x7f4de4104110 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:42.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.438+0000 7f4dea000700 1 -- 172.21.15.43:0/4116788040 >> 172.21.15.43:0/4116788040 conn(0x7f4de40ff270 msgr2=0x7f4de41016d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:42.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.439+0000 7f4dea000700 1 -- 172.21.15.43:0/4116788040 shutdown_connections 2024-03-28T22:25:42.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.439+0000 7f4dea000700 1 -- 172.21.15.43:0/4116788040 wait complete. 2024-03-28T22:25:42.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.441+0000 7f4dea000700 1 Processor -- start 2024-03-28T22:25:42.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.441+0000 7f4dea000700 1 -- start start 2024-03-28T22:25:42.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.441+0000 7f4dea000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4de4103cf0 0x7f4de4079f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:42.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.441+0000 7f4dea000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4de4109d80 con 0x7f4de4103cf0 2024-03-28T22:25:42.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.442+0000 7f4de8c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4de4103cf0 0x7f4de4079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:42.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.442+0000 7f4de8c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4de4103cf0 0x7f4de4079f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34524/0 (socket says 172.21.15.43:34524) 2024-03-28T22:25:42.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.442+0000 7f4de8c00700 1 -- 172.21.15.43:0/2327060707 learned_addr learned my addr 172.21.15.43:0/2327060707 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:42.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.442+0000 7f4de8c00700 1 -- 172.21.15.43:0/2327060707 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4de407a450 con 0x7f4de4103cf0 2024-03-28T22:25:42.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.443+0000 7f4de8c00700 1 --2- 172.21.15.43:0/2327060707 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4de4103cf0 0x7f4de4079f10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8005850 tx=0x7f4dd8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:42.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.443+0000 7f4de1600700 1 -- 172.21.15.43:0/2327060707 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4dd8026070 con 0x7f4de4103cf0 2024-03-28T22:25:42.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.443+0000 7f4de1600700 1 -- 172.21.15.43:0/2327060707 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4dd80044a0 con 0x7f4de4103cf0 2024-03-28T22:25:42.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.443+0000 7f4de1600700 1 -- 172.21.15.43:0/2327060707 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4dd8003ac0 con 0x7f4de4103cf0 2024-03-28T22:25:42.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.443+0000 7f4dea000700 1 -- 172.21.15.43:0/2327060707 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4de407a6e0 con 0x7f4de4103cf0 2024-03-28T22:25:42.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.443+0000 7f4dea000700 1 -- 172.21.15.43:0/2327060707 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4de4076ca0 con 0x7f4de4103cf0 2024-03-28T22:25:42.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.444+0000 7f4de1600700 1 -- 172.21.15.43:0/2327060707 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f4dd8029070 con 0x7f4de4103cf0 2024-03-28T22:25:42.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.444+0000 7f4dea000700 1 -- 172.21.15.43:0/2327060707 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4de407a8b0 con 0x7f4de4103cf0 2024-03-28T22:25:42.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.445+0000 7f4de1600700 1 --2- 172.21.15.43:0/2327060707 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4dcc038ca0 0x7f4dcc03b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:42.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.445+0000 7f4de1600700 1 -- 172.21.15.43:0/2327060707 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4dd801e070 con 0x7f4de4103cf0 2024-03-28T22:25:42.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.445+0000 7f4de3e00700 1 --2- 172.21.15.43:0/2327060707 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4dcc038ca0 0x7f4dcc03b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:42.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.446+0000 7f4de3e00700 1 --2- 172.21.15.43:0/2327060707 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4dcc038ca0 0x7f4dcc03b160 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4dd0006fd0 tx=0x7f4dd0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:42.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.451+0000 7f4de1600700 1 -- 172.21.15.43:0/2327060707 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4de407a8b0 con 0x7f4de4103cf0 2024-03-28T22:25:42.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.677+0000 7f4dea000700 1 -- 172.21.15.43:0/2327060707 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f4de41926c0 con 0x7f4de4103cf0 2024-03-28T22:25:42.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.683+0000 7f4de1600700 1 -- 172.21.15.43:0/2327060707 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f4dd80300c0 con 0x7f4de4103cf0 2024-03-28T22:25:42.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.686+0000 7f4dea000700 1 -- 172.21.15.43:0/2327060707 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4dcc038ca0 msgr2=0x7f4dcc03b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:42.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.686+0000 7f4dea000700 1 --2- 172.21.15.43:0/2327060707 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4dcc038ca0 0x7f4dcc03b160 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4dd0006fd0 tx=0x7f4dd0006e40 comp rx=0 tx=0).stop 2024-03-28T22:25:42.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.686+0000 7f4dea000700 1 -- 172.21.15.43:0/2327060707 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4de4103cf0 msgr2=0x7f4de4079f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:42.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.686+0000 7f4dea000700 1 --2- 172.21.15.43:0/2327060707 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4de4103cf0 0x7f4de4079f10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8005850 tx=0x7f4dd8005030 comp rx=0 tx=0).stop 2024-03-28T22:25:42.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.687+0000 7f4dea000700 1 -- 172.21.15.43:0/2327060707 shutdown_connections 2024-03-28T22:25:42.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.687+0000 7f4dea000700 1 --2- 172.21.15.43:0/2327060707 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4dcc038ca0 0x7f4dcc03b160 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:42.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.687+0000 7f4dea000700 1 --2- 172.21.15.43:0/2327060707 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4de4103cf0 0x7f4de4079f10 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:42.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.687+0000 7f4dea000700 1 -- 172.21.15.43:0/2327060707 >> 172.21.15.43:0/2327060707 conn(0x7f4de40ff270 msgr2=0x7f4de40ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:42.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.687+0000 7f4dea000700 1 -- 172.21.15.43:0/2327060707 shutdown_connections 2024-03-28T22:25:42.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:42.687+0000 7f4dea000700 1 -- 172.21.15.43:0/2327060707 wait complete. 2024-03-28T22:25:43.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:42 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:25:43.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:42 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:25:43.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:42 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi043", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:25:43.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:42 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:25:43.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:42 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:25:43.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:42 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:25:43.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:42 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2327060707' entity='client.admin' 2024-03-28T22:25:43.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:42 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:25:43.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:42 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:25:43.436 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-28T22:25:43.437 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-28T22:25:43.887 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:25:45.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:45.956+0000 7f6b6f600700 1 -- 172.21.15.43:0/3246442594 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b68102720 msgr2=0x7f6b68104b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:45.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:45.956+0000 7f6b6f600700 1 --2- 172.21.15.43:0/3246442594 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b68102720 0x7f6b68104b10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c009910 tx=0x7f6b5c009c20 comp rx=0 tx=0).stop 2024-03-28T22:25:45.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:45.957+0000 7f6b6f600700 1 -- 172.21.15.43:0/3246442594 shutdown_connections 2024-03-28T22:25:45.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:45.957+0000 7f6b6f600700 1 --2- 172.21.15.43:0/3246442594 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b68102720 0x7f6b68104b10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:45.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:45.957+0000 7f6b6f600700 1 -- 172.21.15.43:0/3246442594 >> 172.21.15.43:0/3246442594 conn(0x7f6b680fc130 msgr2=0x7f6b680fe570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:45.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:45.957+0000 7f6b6f600700 1 -- 172.21.15.43:0/3246442594 shutdown_connections 2024-03-28T22:25:45.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:45.957+0000 7f6b6f600700 1 -- 172.21.15.43:0/3246442594 wait complete. 2024-03-28T22:25:45.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:45.958+0000 7f6b6f600700 1 Processor -- start 2024-03-28T22:25:45.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:45.958+0000 7f6b6f600700 1 -- start start 2024-03-28T22:25:45.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:45.958+0000 7f6b6f600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b68102720 0x7f6b68101fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:45.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:45.958+0000 7f6b6f600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6b68107cc0 con 0x7f6b68102720 2024-03-28T22:25:45.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:45.959+0000 7f6b6cc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b68102720 0x7f6b68101fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:45.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:45.959+0000 7f6b6cc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b68102720 0x7f6b68101fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49134/0 (socket says 172.21.15.43:49134) 2024-03-28T22:25:45.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:45.959+0000 7f6b6cc00700 1 -- 172.21.15.43:0/391286280 learned_addr learned my addr 172.21.15.43:0/391286280 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:45.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:45.959+0000 7f6b6cc00700 1 -- 172.21.15.43:0/391286280 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b681024e0 con 0x7f6b68102720 2024-03-28T22:25:45.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:45.959+0000 7f6b6cc00700 1 --2- 172.21.15.43:0/391286280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b68102720 0x7f6b68101fa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c005950 tx=0x7f6b5c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:45.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:45.960+0000 7f6b65600700 1 -- 172.21.15.43:0/391286280 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b5c026070 con 0x7f6b68102720 2024-03-28T22:25:45.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:45.960+0000 7f6b65600700 1 -- 172.21.15.43:0/391286280 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b5c00bc30 con 0x7f6b68102720 2024-03-28T22:25:45.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:45.960+0000 7f6b65600700 1 -- 172.21.15.43:0/391286280 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b5c003b30 con 0x7f6b68102720 2024-03-28T22:25:45.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:45.960+0000 7f6b6f600700 1 -- 172.21.15.43:0/391286280 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b681006d0 con 0x7f6b68102720 2024-03-28T22:25:45.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:45.960+0000 7f6b6f600700 1 -- 172.21.15.43:0/391286280 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b68100c10 con 0x7f6b68102720 2024-03-28T22:25:45.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:45.961+0000 7f6b65600700 1 -- 172.21.15.43:0/391286280 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f6b5c029070 con 0x7f6b68102720 2024-03-28T22:25:45.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:45.961+0000 7f6b6f600700 1 -- 172.21.15.43:0/391286280 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b68108440 con 0x7f6b68102720 2024-03-28T22:25:45.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:45.962+0000 7f6b65600700 1 --2- 172.21.15.43:0/391286280 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f6b50038ca0 0x7f6b5003b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:45.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:45.962+0000 7f6b65600700 1 -- 172.21.15.43:0/391286280 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f6b5c01e070 con 0x7f6b68102720 2024-03-28T22:25:45.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:45.962+0000 7f6b67e00700 1 --2- 172.21.15.43:0/391286280 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f6b50038ca0 0x7f6b5003b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:45.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:45.963+0000 7f6b67e00700 1 --2- 172.21.15.43:0/391286280 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f6b50038ca0 0x7f6b5003b160 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6b54006fd0 tx=0x7f6b54006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:45.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:45.968+0000 7f6b65600700 1 -- 172.21.15.43:0/391286280 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6b5c013070 con 0x7f6b68102720 2024-03-28T22:25:46.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:46.197+0000 7f6b6f600700 1 -- 172.21.15.43:0/391286280 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f6b68190840 con 0x7f6b50038ca0 2024-03-28T22:25:46.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:46.204+0000 7f6b65600700 1 -- 172.21.15.43:0/391286280 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6b68190840 con 0x7f6b50038ca0 2024-03-28T22:25:46.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:46.207+0000 7f6b6f600700 1 -- 172.21.15.43:0/391286280 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f6b50038ca0 msgr2=0x7f6b5003b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:46.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:46.207+0000 7f6b6f600700 1 --2- 172.21.15.43:0/391286280 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f6b50038ca0 0x7f6b5003b160 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6b54006fd0 tx=0x7f6b54006e40 comp rx=0 tx=0).stop 2024-03-28T22:25:46.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:46.207+0000 7f6b6f600700 1 -- 172.21.15.43:0/391286280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b68102720 msgr2=0x7f6b68101fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:46.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:46.207+0000 7f6b6f600700 1 --2- 172.21.15.43:0/391286280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b68102720 0x7f6b68101fa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c005950 tx=0x7f6b5c005030 comp rx=0 tx=0).stop 2024-03-28T22:25:46.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:46.207+0000 7f6b6f600700 1 -- 172.21.15.43:0/391286280 shutdown_connections 2024-03-28T22:25:46.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:46.207+0000 7f6b6f600700 1 --2- 172.21.15.43:0/391286280 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f6b50038ca0 0x7f6b5003b160 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:46.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:46.207+0000 7f6b6f600700 1 --2- 172.21.15.43:0/391286280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b68102720 0x7f6b68101fa0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:46.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:46.207+0000 7f6b6f600700 1 -- 172.21.15.43:0/391286280 >> 172.21.15.43:0/391286280 conn(0x7f6b680fc130 msgr2=0x7f6b680fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:46.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:46.207+0000 7f6b6f600700 1 -- 172.21.15.43:0/391286280 shutdown_connections 2024-03-28T22:25:46.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:46.208+0000 7f6b6f600700 1 -- 172.21.15.43:0/391286280 wait complete. 2024-03-28T22:25:46.886 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi082 2024-03-28T22:25:46.887 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-28T22:25:46.887 DEBUG:teuthology.orchestra.run.smithi082:> dd of=/etc/ceph/ceph.conf 2024-03-28T22:25:46.905 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-28T22:25:46.905 DEBUG:teuthology.orchestra.run.smithi082:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:25:46.961 INFO:tasks.cephadm:Adding host smithi082 to orchestrator... 2024-03-28T22:25:46.961 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph orch host add smithi082 2024-03-28T22:25:47.380 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:25:47.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:47 smithi043 ceph-mon[31823]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:25:47.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:47 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:25:47.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:47 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:25:47.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:47 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:25:47.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:47 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:25:47.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:47 smithi043 ceph-mon[31823]: Updating smithi043:/etc/ceph/ceph.conf 2024-03-28T22:25:47.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:47 smithi043 ceph-mon[31823]: Updating smithi043:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:25:48.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:48 smithi043 ceph-mon[31823]: Updating smithi043:/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:25:49.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:49.550+0000 7f7482a00700 1 -- 172.21.15.43:0/677970249 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f747c1084a0 msgr2=0x7f747c108880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:49.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:49.550+0000 7f7482a00700 1 --2- 172.21.15.43:0/677970249 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f747c1084a0 0x7f747c108880 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f746c009910 tx=0x7f746c009c20 comp rx=0 tx=0).stop 2024-03-28T22:25:49.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:49.550+0000 7f7482a00700 1 -- 172.21.15.43:0/677970249 shutdown_connections 2024-03-28T22:25:49.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:49.550+0000 7f7482a00700 1 --2- 172.21.15.43:0/677970249 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f747c1084a0 0x7f747c108880 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:49.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:49.550+0000 7f7482a00700 1 -- 172.21.15.43:0/677970249 >> 172.21.15.43:0/677970249 conn(0x7f747c0779e0 msgr2=0x7f747c079e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:49.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:49.550+0000 7f7482a00700 1 -- 172.21.15.43:0/677970249 shutdown_connections 2024-03-28T22:25:49.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:49.550+0000 7f7482a00700 1 -- 172.21.15.43:0/677970249 wait complete. 2024-03-28T22:25:49.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:49.551+0000 7f7482a00700 1 Processor -- start 2024-03-28T22:25:49.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:49.551+0000 7f7482a00700 1 -- start start 2024-03-28T22:25:49.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:49.551+0000 7f7482a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f747c1084a0 0x7f747c19de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:49.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:49.551+0000 7f7482a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f747c112870 con 0x7f747c1084a0 2024-03-28T22:25:49.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:49.552+0000 7f747be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f747c1084a0 0x7f747c19de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:49.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:49.552+0000 7f747be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f747c1084a0 0x7f747c19de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49164/0 (socket says 172.21.15.43:49164) 2024-03-28T22:25:49.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:49.552+0000 7f747be00700 1 -- 172.21.15.43:0/1982301912 learned_addr learned my addr 172.21.15.43:0/1982301912 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:49.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:49.552+0000 7f747be00700 1 -- 172.21.15.43:0/1982301912 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f747c19e380 con 0x7f747c1084a0 2024-03-28T22:25:49.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:49.552+0000 7f747be00700 1 --2- 172.21.15.43:0/1982301912 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f747c1084a0 0x7f747c19de40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f746c006010 tx=0x7f746c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:49.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:49.553+0000 7f7478c00700 1 -- 172.21.15.43:0/1982301912 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f746c026070 con 0x7f747c1084a0 2024-03-28T22:25:49.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:49.553+0000 7f7478c00700 1 -- 172.21.15.43:0/1982301912 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f746c00bbd0 con 0x7f747c1084a0 2024-03-28T22:25:49.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:49.553+0000 7f7478c00700 1 -- 172.21.15.43:0/1982301912 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f746c003910 con 0x7f747c1084a0 2024-03-28T22:25:49.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:49.553+0000 7f7482a00700 1 -- 172.21.15.43:0/1982301912 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f747c19e670 con 0x7f747c1084a0 2024-03-28T22:25:49.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:49.553+0000 7f7482a00700 1 -- 172.21.15.43:0/1982301912 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f747c1a2670 con 0x7f747c1084a0 2024-03-28T22:25:49.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:49.554+0000 7f7478c00700 1 -- 172.21.15.43:0/1982301912 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f746c029070 con 0x7f747c1084a0 2024-03-28T22:25:49.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:49.554+0000 7f7472000700 1 -- 172.21.15.43:0/1982301912 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f747c109980 con 0x7f747c1084a0 2024-03-28T22:25:49.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:49.555+0000 7f7478c00700 1 --2- 172.21.15.43:0/1982301912 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7460038ca0 0x7f746003b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:49.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:49.555+0000 7f7478c00700 1 -- 172.21.15.43:0/1982301912 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f746c01e070 con 0x7f747c1084a0 2024-03-28T22:25:49.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:49.555+0000 7f747b400700 1 --2- 172.21.15.43:0/1982301912 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7460038ca0 0x7f746003b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:49.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:49.556+0000 7f747b400700 1 --2- 172.21.15.43:0/1982301912 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7460038ca0 0x7f746003b160 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7468006fd0 tx=0x7f7468006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:49.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:49.561+0000 7f7478c00700 1 -- 172.21.15.43:0/1982301912 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f746c013070 con 0x7f747c1084a0 2024-03-28T22:25:49.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:49.789+0000 7f7472000700 1 -- 172.21.15.43:0/1982301912 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi082", "target": ["mon-mgr", ""]}) -- 0x7f747c063870 con 0x7f7460038ca0 2024-03-28T22:25:50.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:49 smithi043 ceph-mon[31823]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:25:50.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:49 smithi043 ceph-mon[31823]: Updating smithi043:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T22:25:50.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:49 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:25:50.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:49 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:25:50.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:49 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:25:51.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:50 smithi043 ceph-mon[31823]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi082", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:25:52.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:51 smithi043 ceph-mon[31823]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:25:52.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:51 smithi043 ceph-mon[31823]: Deploying cephadm binary to smithi082 2024-03-28T22:25:54.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:53 smithi043 ceph-mon[31823]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:25:54.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:54.594+0000 7f7478c00700 1 -- 172.21.15.43:0/1982301912 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f747c063870 con 0x7f7460038ca0 2024-03-28T22:25:54.597 INFO:teuthology.orchestra.run.smithi043.stdout:Added host 'smithi082' with addr '172.21.15.82' 2024-03-28T22:25:54.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:54.598+0000 7f7472000700 1 -- 172.21.15.43:0/1982301912 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7460038ca0 msgr2=0x7f746003b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:54.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:54.598+0000 7f7472000700 1 --2- 172.21.15.43:0/1982301912 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7460038ca0 0x7f746003b160 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7468006fd0 tx=0x7f7468006e40 comp rx=0 tx=0).stop 2024-03-28T22:25:54.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:54.598+0000 7f7472000700 1 -- 172.21.15.43:0/1982301912 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f747c1084a0 msgr2=0x7f747c19de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:54.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:54.598+0000 7f7472000700 1 --2- 172.21.15.43:0/1982301912 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f747c1084a0 0x7f747c19de40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f746c006010 tx=0x7f746c005030 comp rx=0 tx=0).stop 2024-03-28T22:25:54.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:54.598+0000 7f7472000700 1 -- 172.21.15.43:0/1982301912 shutdown_connections 2024-03-28T22:25:54.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:54.598+0000 7f7472000700 1 --2- 172.21.15.43:0/1982301912 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7460038ca0 0x7f746003b160 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:54.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:54.598+0000 7f7472000700 1 --2- 172.21.15.43:0/1982301912 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f747c1084a0 0x7f747c19de40 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:54.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:54.598+0000 7f7472000700 1 -- 172.21.15.43:0/1982301912 >> 172.21.15.43:0/1982301912 conn(0x7f747c0779e0 msgr2=0x7f747c109e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:54.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:54.599+0000 7f7472000700 1 -- 172.21.15.43:0/1982301912 shutdown_connections 2024-03-28T22:25:54.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:54.599+0000 7f7472000700 1 -- 172.21.15.43:0/1982301912 wait complete. 2024-03-28T22:25:55.309 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-28T22:25:55.782 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:25:55.931 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:55 smithi043 ceph-mon[31823]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:25:55.931 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:55 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:25:55.931 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:55 smithi043 ceph-mon[31823]: Added host smithi082 2024-03-28T22:25:55.932 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:55 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:25:55.932 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:55 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:25:56.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:56 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:25:57.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:57.811+0000 7f2204c00700 1 -- 172.21.15.43:0/4211288479 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2200102e90 msgr2=0x7f2200103270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:57.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:57.811+0000 7f2204c00700 1 --2- 172.21.15.43:0/4211288479 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2200102e90 0x7f2200103270 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f21f0009910 tx=0x7f21f0009c20 comp rx=0 tx=0).stop 2024-03-28T22:25:57.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:57.811+0000 7f2204c00700 1 -- 172.21.15.43:0/4211288479 shutdown_connections 2024-03-28T22:25:57.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:57.811+0000 7f2204c00700 1 --2- 172.21.15.43:0/4211288479 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2200102e90 0x7f2200103270 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:57.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:57.811+0000 7f2204c00700 1 -- 172.21.15.43:0/4211288479 >> 172.21.15.43:0/4211288479 conn(0x7f2200077780 msgr2=0x7f22000fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:57.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:57.811+0000 7f2204c00700 1 -- 172.21.15.43:0/4211288479 shutdown_connections 2024-03-28T22:25:57.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:57.811+0000 7f2204c00700 1 -- 172.21.15.43:0/4211288479 wait complete. 2024-03-28T22:25:57.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:57.812+0000 7f2204c00700 1 Processor -- start 2024-03-28T22:25:57.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:57.813+0000 7f2204c00700 1 -- start start 2024-03-28T22:25:57.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:57.813+0000 7f2204c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2200102e90 0x7f2200100af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:57.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:57.813+0000 7f2204c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f220010c2b0 con 0x7f2200102e90 2024-03-28T22:25:57.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:57.813+0000 7f21fde00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2200102e90 0x7f2200100af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:57.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:57.813+0000 7f21fde00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2200102e90 0x7f2200100af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57236/0 (socket says 172.21.15.43:57236) 2024-03-28T22:25:57.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:57.813+0000 7f21fde00700 1 -- 172.21.15.43:0/2069664332 learned_addr learned my addr 172.21.15.43:0/2069664332 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:25:57.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:57.814+0000 7f21fde00700 1 -- 172.21.15.43:0/2069664332 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22001029e0 con 0x7f2200102e90 2024-03-28T22:25:57.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:57.814+0000 7f21fde00700 1 --2- 172.21.15.43:0/2069664332 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2200102e90 0x7f2200100af0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f21f0006010 tx=0x7f21f0005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:57.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:57.814+0000 7f21f6a00700 1 -- 172.21.15.43:0/2069664332 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21f0026070 con 0x7f2200102e90 2024-03-28T22:25:57.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:57.814+0000 7f21f6a00700 1 -- 172.21.15.43:0/2069664332 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21f000bc30 con 0x7f2200102e90 2024-03-28T22:25:57.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:57.814+0000 7f21f6a00700 1 -- 172.21.15.43:0/2069664332 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21f00039a0 con 0x7f2200102e90 2024-03-28T22:25:57.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:57.814+0000 7f2204c00700 1 -- 172.21.15.43:0/2069664332 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2200101030 con 0x7f2200102e90 2024-03-28T22:25:57.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:57.815+0000 7f2204c00700 1 -- 172.21.15.43:0/2069664332 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22001014f0 con 0x7f2200102e90 2024-03-28T22:25:57.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:57.816+0000 7f2204c00700 1 -- 172.21.15.43:0/2069664332 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2200069e50 con 0x7f2200102e90 2024-03-28T22:25:57.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:57.816+0000 7f21f6a00700 1 -- 172.21.15.43:0/2069664332 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f21f0029070 con 0x7f2200102e90 2024-03-28T22:25:57.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:57.816+0000 7f21f6a00700 1 --2- 172.21.15.43:0/2069664332 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f21e4038cf0 0x7f21e403b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:25:57.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:57.816+0000 7f21f6a00700 1 -- 172.21.15.43:0/2069664332 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f21f001e070 con 0x7f2200102e90 2024-03-28T22:25:57.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:57.817+0000 7f21fd400700 1 --2- 172.21.15.43:0/2069664332 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f21e4038cf0 0x7f21e403b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:25:57.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:57.818+0000 7f21fd400700 1 --2- 172.21.15.43:0/2069664332 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f21e4038cf0 0x7f21e403b1b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f21ec006fd0 tx=0x7f21ec006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:25:57.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:57.822+0000 7f21f6a00700 1 -- 172.21.15.43:0/2069664332 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f21f0013070 con 0x7f2200102e90 2024-03-28T22:25:58.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:57 smithi043 ceph-mon[31823]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:25:58.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:57 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:25:58.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:58.056+0000 7f2204c00700 1 -- 172.21.15.43:0/2069664332 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2200102020 con 0x7f21e4038cf0 2024-03-28T22:25:58.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:58.059+0000 7f21f6a00700 1 -- 172.21.15.43:0/2069664332 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7f2200102020 con 0x7f21e4038cf0 2024-03-28T22:25:58.059 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:25:58.059 INFO:teuthology.orchestra.run.smithi043.stdout:[{"addr": "172.21.15.43", "hostname": "smithi043", "labels": [], "status": ""}, {"addr": "172.21.15.82", "hostname": "smithi082", "labels": [], "status": ""}] 2024-03-28T22:25:58.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:58.061+0000 7f2204c00700 1 -- 172.21.15.43:0/2069664332 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f21e4038cf0 msgr2=0x7f21e403b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:58.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:58.061+0000 7f2204c00700 1 --2- 172.21.15.43:0/2069664332 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f21e4038cf0 0x7f21e403b1b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f21ec006fd0 tx=0x7f21ec006e40 comp rx=0 tx=0).stop 2024-03-28T22:25:58.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:58.061+0000 7f2204c00700 1 -- 172.21.15.43:0/2069664332 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2200102e90 msgr2=0x7f2200100af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:25:58.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:58.061+0000 7f2204c00700 1 --2- 172.21.15.43:0/2069664332 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2200102e90 0x7f2200100af0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f21f0006010 tx=0x7f21f0005030 comp rx=0 tx=0).stop 2024-03-28T22:25:58.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:58.062+0000 7f2204c00700 1 -- 172.21.15.43:0/2069664332 shutdown_connections 2024-03-28T22:25:58.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:58.062+0000 7f2204c00700 1 --2- 172.21.15.43:0/2069664332 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f21e4038cf0 0x7f21e403b1b0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:58.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:58.062+0000 7f2204c00700 1 --2- 172.21.15.43:0/2069664332 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2200102e90 0x7f2200100af0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:25:58.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:58.062+0000 7f2204c00700 1 -- 172.21.15.43:0/2069664332 >> 172.21.15.43:0/2069664332 conn(0x7f2200077780 msgr2=0x7f2200103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:25:58.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:58.062+0000 7f2204c00700 1 -- 172.21.15.43:0/2069664332 shutdown_connections 2024-03-28T22:25:58.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:25:58.062+0000 7f2204c00700 1 -- 172.21.15.43:0/2069664332 wait complete. 2024-03-28T22:25:58.776 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi172 2024-03-28T22:25:58.776 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-28T22:25:58.776 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/etc/ceph/ceph.conf 2024-03-28T22:25:58.794 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-28T22:25:58.794 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:25:58.851 INFO:tasks.cephadm:Adding host smithi172 to orchestrator... 2024-03-28T22:25:58.851 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph orch host add smithi172 2024-03-28T22:25:59.303 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:25:59.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:59 smithi043 ceph-mon[31823]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:25:59.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:25:59 smithi043 ceph-mon[31823]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:26:01.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:01.238+0000 7fc9a2000700 1 -- 172.21.15.43:0/2788800567 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc99c1040a0 msgr2=0x7fc99c104480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:01.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:01.238+0000 7fc9a2000700 1 --2- 172.21.15.43:0/2788800567 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc99c1040a0 0x7fc99c104480 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc98c009910 tx=0x7fc98c009c20 comp rx=0 tx=0).stop 2024-03-28T22:26:01.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:01.238+0000 7fc9a2000700 1 -- 172.21.15.43:0/2788800567 shutdown_connections 2024-03-28T22:26:01.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:01.238+0000 7fc9a2000700 1 --2- 172.21.15.43:0/2788800567 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc99c1040a0 0x7fc99c104480 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:01.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:01.238+0000 7fc9a2000700 1 -- 172.21.15.43:0/2788800567 >> 172.21.15.43:0/2788800567 conn(0x7fc99c0ff930 msgr2=0x7fc99c101d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:26:01.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:01.238+0000 7fc9a2000700 1 -- 172.21.15.43:0/2788800567 shutdown_connections 2024-03-28T22:26:01.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:01.238+0000 7fc9a2000700 1 -- 172.21.15.43:0/2788800567 wait complete. 2024-03-28T22:26:01.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:01.239+0000 7fc9a2000700 1 Processor -- start 2024-03-28T22:26:01.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:01.240+0000 7fc9a2000700 1 -- start start 2024-03-28T22:26:01.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:01.240+0000 7fc9a2000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc99c1040a0 0x7fc99c19f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:26:01.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:01.240+0000 7fc9a2000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc99c10e470 con 0x7fc99c1040a0 2024-03-28T22:26:01.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:01.240+0000 7fc99b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc99c1040a0 0x7fc99c19f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:26:01.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:01.240+0000 7fc99b400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc99c1040a0 0x7fc99c19f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57254/0 (socket says 172.21.15.43:57254) 2024-03-28T22:26:01.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:01.240+0000 7fc99b400700 1 -- 172.21.15.43:0/1680561819 learned_addr learned my addr 172.21.15.43:0/1680561819 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:26:01.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:01.241+0000 7fc99b400700 1 -- 172.21.15.43:0/1680561819 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc99c19fbd0 con 0x7fc99c1040a0 2024-03-28T22:26:01.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:01.241+0000 7fc99b400700 1 --2- 172.21.15.43:0/1680561819 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc99c1040a0 0x7fc99c19f690 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc98c006010 tx=0x7fc98c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:26:01.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:01.241+0000 7fc993e00700 1 -- 172.21.15.43:0/1680561819 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc98c026070 con 0x7fc99c1040a0 2024-03-28T22:26:01.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:01.241+0000 7fc993e00700 1 -- 172.21.15.43:0/1680561819 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc98c00bbd0 con 0x7fc99c1040a0 2024-03-28T22:26:01.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:01.241+0000 7fc9a2000700 1 -- 172.21.15.43:0/1680561819 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc99c19fec0 con 0x7fc99c1040a0 2024-03-28T22:26:01.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:01.241+0000 7fc993e00700 1 -- 172.21.15.43:0/1680561819 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc98c003910 con 0x7fc99c1040a0 2024-03-28T22:26:01.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:01.242+0000 7fc9a2000700 1 -- 172.21.15.43:0/1680561819 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc99c199930 con 0x7fc99c1040a0 2024-03-28T22:26:01.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:01.243+0000 7fc993e00700 1 -- 172.21.15.43:0/1680561819 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7fc98c029070 con 0x7fc99c1040a0 2024-03-28T22:26:01.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:01.243+0000 7fc9a2000700 1 -- 172.21.15.43:0/1680561819 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc99c105580 con 0x7fc99c1040a0 2024-03-28T22:26:01.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:01.243+0000 7fc993e00700 1 --2- 172.21.15.43:0/1680561819 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fc980038c40 0x7fc98003b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:26:01.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:01.243+0000 7fc993e00700 1 -- 172.21.15.43:0/1680561819 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc98c01e070 con 0x7fc99c1040a0 2024-03-28T22:26:01.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:01.243+0000 7fc99aa00700 1 --2- 172.21.15.43:0/1680561819 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fc980038c40 0x7fc98003b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:26:01.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:01.244+0000 7fc99aa00700 1 --2- 172.21.15.43:0/1680561819 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fc980038c40 0x7fc98003b100 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc988006fd0 tx=0x7fc988006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:26:01.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:01.249+0000 7fc993e00700 1 -- 172.21.15.43:0/1680561819 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc98c013070 con 0x7fc99c1040a0 2024-03-28T22:26:01.402 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:01 smithi043 ceph-mon[31823]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:01.403 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:01 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:01.403 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:01 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:01.403 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:01 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:01.403 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:01 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:01.403 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:01 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi082", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:26:01.403 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:01 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:01.403 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:01 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:26:01.403 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:01 smithi043 ceph-mon[31823]: Updating smithi082:/etc/ceph/ceph.conf 2024-03-28T22:26:01.403 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:01 smithi043 ceph-mon[31823]: Updating smithi082:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:26:01.403 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:01 smithi043 ceph-mon[31823]: Updating smithi082:/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:26:01.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:01.499+0000 7fc9a2000700 1 -- 172.21.15.43:0/1680561819 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi172", "target": ["mon-mgr", ""]}) -- 0x7fc99c19ad00 con 0x7fc980038c40 2024-03-28T22:26:02.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:02 smithi043 ceph-mon[31823]: Updating smithi082:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T22:26:02.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:02 smithi043 ceph-mon[31823]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi172", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:26:02.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:02 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:02.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:02 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:02.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:02 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:04.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:03 smithi043 ceph-mon[31823]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:04.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:03 smithi043 ceph-mon[31823]: Deploying cephadm binary to smithi172 2024-03-28T22:26:06.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:05 smithi043 ceph-mon[31823]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:06.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:06.406+0000 7fc993e00700 1 -- 172.21.15.43:0/1680561819 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fc99c19ad00 con 0x7fc980038c40 2024-03-28T22:26:06.408 INFO:teuthology.orchestra.run.smithi043.stdout:Added host 'smithi172' with addr '172.21.15.172' 2024-03-28T22:26:06.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:06.411+0000 7fc9a2000700 1 -- 172.21.15.43:0/1680561819 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fc980038c40 msgr2=0x7fc98003b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:06.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:06.411+0000 7fc9a2000700 1 --2- 172.21.15.43:0/1680561819 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fc980038c40 0x7fc98003b100 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc988006fd0 tx=0x7fc988006e40 comp rx=0 tx=0).stop 2024-03-28T22:26:06.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:06.411+0000 7fc9a2000700 1 -- 172.21.15.43:0/1680561819 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc99c1040a0 msgr2=0x7fc99c19f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:06.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:06.411+0000 7fc9a2000700 1 --2- 172.21.15.43:0/1680561819 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc99c1040a0 0x7fc99c19f690 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc98c006010 tx=0x7fc98c005030 comp rx=0 tx=0).stop 2024-03-28T22:26:06.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:06.411+0000 7fc9a2000700 1 -- 172.21.15.43:0/1680561819 shutdown_connections 2024-03-28T22:26:06.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:06.411+0000 7fc9a2000700 1 --2- 172.21.15.43:0/1680561819 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fc980038c40 0x7fc98003b100 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:06.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:06.411+0000 7fc9a2000700 1 --2- 172.21.15.43:0/1680561819 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc99c1040a0 0x7fc99c19f690 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:06.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:06.411+0000 7fc9a2000700 1 -- 172.21.15.43:0/1680561819 >> 172.21.15.43:0/1680561819 conn(0x7fc99c0ff930 msgr2=0x7fc99c105a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:26:06.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:06.411+0000 7fc9a2000700 1 -- 172.21.15.43:0/1680561819 shutdown_connections 2024-03-28T22:26:06.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:06.411+0000 7fc9a2000700 1 -- 172.21.15.43:0/1680561819 wait complete. 2024-03-28T22:26:07.150 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-28T22:26:07.625 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:26:07.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:07 smithi043 ceph-mon[31823]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:07.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:07 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:07.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:07 smithi043 ceph-mon[31823]: Added host smithi172 2024-03-28T22:26:07.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:07 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:26:07.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:07 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:08.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:08 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:09.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.588+0000 7f0626800700 1 -- 172.21.15.43:0/3270939240 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06201040a0 msgr2=0x7f0620104480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:09.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.588+0000 7f0626800700 1 --2- 172.21.15.43:0/3270939240 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06201040a0 0x7f0620104480 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0610009910 tx=0x7f0610009c20 comp rx=0 tx=0).stop 2024-03-28T22:26:09.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.589+0000 7f0626800700 1 -- 172.21.15.43:0/3270939240 shutdown_connections 2024-03-28T22:26:09.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.589+0000 7f0626800700 1 --2- 172.21.15.43:0/3270939240 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06201040a0 0x7f0620104480 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:09.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.589+0000 7f0626800700 1 -- 172.21.15.43:0/3270939240 >> 172.21.15.43:0/3270939240 conn(0x7f06200ff930 msgr2=0x7f0620101d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:26:09.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.589+0000 7f0626800700 1 -- 172.21.15.43:0/3270939240 shutdown_connections 2024-03-28T22:26:09.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.589+0000 7f0626800700 1 -- 172.21.15.43:0/3270939240 wait complete. 2024-03-28T22:26:09.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.590+0000 7f0626800700 1 Processor -- start 2024-03-28T22:26:09.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.591+0000 7f0626800700 1 -- start start 2024-03-28T22:26:09.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.591+0000 7f0626800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06201040a0 0x7f062019f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:26:09.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.591+0000 7f0626800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f062010e470 con 0x7f06201040a0 2024-03-28T22:26:09.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.591+0000 7f061f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06201040a0 0x7f062019f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:26:09.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.591+0000 7f061f400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06201040a0 0x7f062019f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55220/0 (socket says 172.21.15.43:55220) 2024-03-28T22:26:09.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.591+0000 7f061f400700 1 -- 172.21.15.43:0/2643320368 learned_addr learned my addr 172.21.15.43:0/2643320368 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:26:09.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.592+0000 7f061f400700 1 -- 172.21.15.43:0/2643320368 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f062019fbd0 con 0x7f06201040a0 2024-03-28T22:26:09.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.592+0000 7f061f400700 1 --2- 172.21.15.43:0/2643320368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06201040a0 0x7f062019f690 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0610006010 tx=0x7f0610005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:26:09.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.592+0000 7f0617e00700 1 -- 172.21.15.43:0/2643320368 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0610026070 con 0x7f06201040a0 2024-03-28T22:26:09.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.592+0000 7f0617e00700 1 -- 172.21.15.43:0/2643320368 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f061000bbd0 con 0x7f06201040a0 2024-03-28T22:26:09.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.592+0000 7f0626800700 1 -- 172.21.15.43:0/2643320368 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f062019fec0 con 0x7f06201040a0 2024-03-28T22:26:09.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.593+0000 7f0617e00700 1 -- 172.21.15.43:0/2643320368 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0610003910 con 0x7f06201040a0 2024-03-28T22:26:09.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.593+0000 7f0626800700 1 -- 172.21.15.43:0/2643320368 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0620199930 con 0x7f06201040a0 2024-03-28T22:26:09.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.594+0000 7f0617e00700 1 -- 172.21.15.43:0/2643320368 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f0610029070 con 0x7f06201040a0 2024-03-28T22:26:09.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.594+0000 7f0626800700 1 -- 172.21.15.43:0/2643320368 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0620105580 con 0x7f06201040a0 2024-03-28T22:26:09.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.594+0000 7f0617e00700 1 --2- 172.21.15.43:0/2643320368 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f0604038c40 0x7f060403b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:26:09.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.595+0000 7f0617e00700 1 -- 172.21.15.43:0/2643320368 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f061001e070 con 0x7f06201040a0 2024-03-28T22:26:09.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.595+0000 7f061ea00700 1 --2- 172.21.15.43:0/2643320368 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f0604038c40 0x7f060403b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:26:09.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.596+0000 7f061ea00700 1 --2- 172.21.15.43:0/2643320368 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f0604038c40 0x7f060403b100 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f060c006fd0 tx=0x7f060c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:26:09.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.602+0000 7f0617e00700 1 -- 172.21.15.43:0/2643320368 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0610013070 con 0x7f06201040a0 2024-03-28T22:26:09.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:09 smithi043 ceph-mon[31823]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:09.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:09 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:09.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.833+0000 7f0626800700 1 -- 172.21.15.43:0/2643320368 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f062019ad00 con 0x7f0604038c40 2024-03-28T22:26:09.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.835+0000 7f0617e00700 1 -- 172.21.15.43:0/2643320368 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7f062019ad00 con 0x7f0604038c40 2024-03-28T22:26:09.836 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:26:09.836 INFO:teuthology.orchestra.run.smithi043.stdout:[{"addr": "172.21.15.43", "hostname": "smithi043", "labels": [], "status": ""}, {"addr": "172.21.15.82", "hostname": "smithi082", "labels": [], "status": ""}, {"addr": "172.21.15.172", "hostname": "smithi172", "labels": [], "status": ""}] 2024-03-28T22:26:09.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.838+0000 7f0626800700 1 -- 172.21.15.43:0/2643320368 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f0604038c40 msgr2=0x7f060403b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:09.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.838+0000 7f0626800700 1 --2- 172.21.15.43:0/2643320368 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f0604038c40 0x7f060403b100 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f060c006fd0 tx=0x7f060c006e40 comp rx=0 tx=0).stop 2024-03-28T22:26:09.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.838+0000 7f0626800700 1 -- 172.21.15.43:0/2643320368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06201040a0 msgr2=0x7f062019f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:09.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.838+0000 7f0626800700 1 --2- 172.21.15.43:0/2643320368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06201040a0 0x7f062019f690 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0610006010 tx=0x7f0610005030 comp rx=0 tx=0).stop 2024-03-28T22:26:09.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.838+0000 7f0626800700 1 -- 172.21.15.43:0/2643320368 shutdown_connections 2024-03-28T22:26:09.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.838+0000 7f0626800700 1 --2- 172.21.15.43:0/2643320368 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f0604038c40 0x7f060403b100 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:09.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.838+0000 7f0626800700 1 --2- 172.21.15.43:0/2643320368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06201040a0 0x7f062019f690 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:09.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.838+0000 7f0626800700 1 -- 172.21.15.43:0/2643320368 >> 172.21.15.43:0/2643320368 conn(0x7f06200ff930 msgr2=0x7f0620105a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:26:09.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.839+0000 7f0626800700 1 -- 172.21.15.43:0/2643320368 shutdown_connections 2024-03-28T22:26:09.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:09.839+0000 7f0626800700 1 -- 172.21.15.43:0/2643320368 wait complete. 2024-03-28T22:26:10.530 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-28T22:26:10.531 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph osd crush tunables default 2024-03-28T22:26:10.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:10 smithi043 ceph-mon[31823]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:26:10.999 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:26:11.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:11 smithi043 ceph-mon[31823]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:13.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.057+0000 7ff936a00700 1 -- 172.21.15.43:0/2864827160 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff930102dd0 msgr2=0x7ff9301031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:13.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.057+0000 7ff936a00700 1 --2- 172.21.15.43:0/2864827160 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff930102dd0 0x7ff9301031b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff920009910 tx=0x7ff920009c20 comp rx=0 tx=0).stop 2024-03-28T22:26:13.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.057+0000 7ff936a00700 1 -- 172.21.15.43:0/2864827160 shutdown_connections 2024-03-28T22:26:13.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.057+0000 7ff936a00700 1 --2- 172.21.15.43:0/2864827160 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff930102dd0 0x7ff9301031b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:13.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.057+0000 7ff936a00700 1 -- 172.21.15.43:0/2864827160 >> 172.21.15.43:0/2864827160 conn(0x7ff930077780 msgr2=0x7ff9300fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:26:13.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.057+0000 7ff936a00700 1 -- 172.21.15.43:0/2864827160 shutdown_connections 2024-03-28T22:26:13.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.057+0000 7ff936a00700 1 -- 172.21.15.43:0/2864827160 wait complete. 2024-03-28T22:26:13.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.058+0000 7ff936a00700 1 Processor -- start 2024-03-28T22:26:13.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.059+0000 7ff936a00700 1 -- start start 2024-03-28T22:26:13.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.059+0000 7ff936a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff930102dd0 0x7ff93019d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:26:13.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.059+0000 7ff936a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff93010c2c0 con 0x7ff930102dd0 2024-03-28T22:26:13.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.059+0000 7ff92fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff930102dd0 0x7ff93019d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:26:13.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.059+0000 7ff92fe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff930102dd0 0x7ff93019d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55240/0 (socket says 172.21.15.43:55240) 2024-03-28T22:26:13.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.059+0000 7ff92fe00700 1 -- 172.21.15.43:0/4194515456 learned_addr learned my addr 172.21.15.43:0/4194515456 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:26:13.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.060+0000 7ff92fe00700 1 -- 172.21.15.43:0/4194515456 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff93019d930 con 0x7ff930102dd0 2024-03-28T22:26:13.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.060+0000 7ff92fe00700 1 --2- 172.21.15.43:0/4194515456 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff930102dd0 0x7ff93019d3f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff920006010 tx=0x7ff920005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:26:13.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.060+0000 7ff92cc00700 1 -- 172.21.15.43:0/4194515456 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff920026070 con 0x7ff930102dd0 2024-03-28T22:26:13.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.060+0000 7ff92cc00700 1 -- 172.21.15.43:0/4194515456 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff92000bc30 con 0x7ff930102dd0 2024-03-28T22:26:13.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.060+0000 7ff92cc00700 1 -- 172.21.15.43:0/4194515456 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff9200039a0 con 0x7ff930102dd0 2024-03-28T22:26:13.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.060+0000 7ff936a00700 1 -- 172.21.15.43:0/4194515456 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff93019dbc0 con 0x7ff930102dd0 2024-03-28T22:26:13.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.061+0000 7ff936a00700 1 -- 172.21.15.43:0/4194515456 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9301975b0 con 0x7ff930102dd0 2024-03-28T22:26:13.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.062+0000 7ff936a00700 1 -- 172.21.15.43:0/4194515456 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff930069f10 con 0x7ff930102dd0 2024-03-28T22:26:13.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.062+0000 7ff92cc00700 1 -- 172.21.15.43:0/4194515456 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7ff920029070 con 0x7ff930102dd0 2024-03-28T22:26:13.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.062+0000 7ff92cc00700 1 --2- 172.21.15.43:0/4194515456 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff914038c40 0x7ff91403b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:26:13.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.062+0000 7ff92cc00700 1 -- 172.21.15.43:0/4194515456 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff92001e070 con 0x7ff930102dd0 2024-03-28T22:26:13.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.063+0000 7ff92f400700 1 --2- 172.21.15.43:0/4194515456 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff914038c40 0x7ff91403b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:26:13.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.063+0000 7ff92f400700 1 --2- 172.21.15.43:0/4194515456 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff914038c40 0x7ff91403b100 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff91c006fd0 tx=0x7ff91c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:26:13.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.068+0000 7ff92cc00700 1 -- 172.21.15.43:0/4194515456 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff920013070 con 0x7ff930102dd0 2024-03-28T22:26:13.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.344+0000 7ff936a00700 1 -- 172.21.15.43:0/4194515456 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7ff930063870 con 0x7ff930102dd0 2024-03-28T22:26:13.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.463+0000 7ff92cc00700 1 -- 172.21.15.43:0/4194515456 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7ff92000bde0 con 0x7ff930102dd0 2024-03-28T22:26:13.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.467+0000 7ff936a00700 1 -- 172.21.15.43:0/4194515456 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff914038c40 msgr2=0x7ff91403b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:13.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.467+0000 7ff936a00700 1 --2- 172.21.15.43:0/4194515456 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff914038c40 0x7ff91403b100 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff91c006fd0 tx=0x7ff91c006e40 comp rx=0 tx=0).stop 2024-03-28T22:26:13.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.467+0000 7ff936a00700 1 -- 172.21.15.43:0/4194515456 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff930102dd0 msgr2=0x7ff93019d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:13.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.467+0000 7ff936a00700 1 --2- 172.21.15.43:0/4194515456 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff930102dd0 0x7ff93019d3f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff920006010 tx=0x7ff920005030 comp rx=0 tx=0).stop 2024-03-28T22:26:13.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.467+0000 7ff936a00700 1 -- 172.21.15.43:0/4194515456 shutdown_connections 2024-03-28T22:26:13.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.468+0000 7ff936a00700 1 --2- 172.21.15.43:0/4194515456 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff914038c40 0x7ff91403b100 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:13.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.468+0000 7ff936a00700 1 --2- 172.21.15.43:0/4194515456 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff930102dd0 0x7ff93019d3f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:13.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.468+0000 7ff936a00700 1 -- 172.21.15.43:0/4194515456 >> 172.21.15.43:0/4194515456 conn(0x7ff930077780 msgr2=0x7ff930103b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:26:13.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.468+0000 7ff936a00700 1 -- 172.21.15.43:0/4194515456 shutdown_connections 2024-03-28T22:26:13.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:13.468+0000 7ff936a00700 1 -- 172.21.15.43:0/4194515456 wait complete. 2024-03-28T22:26:13.470 INFO:teuthology.orchestra.run.smithi043.stderr:adjusted tunables profile to default 2024-03-28T22:26:13.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:13 smithi043 ceph-mon[31823]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:13.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:13 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:13.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:13 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:13.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:13 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:13.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:13 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:13.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:13 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi172", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:26:13.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:13 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:13.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:13 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:26:13.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:13 smithi043 ceph-mon[31823]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-28T22:26:13.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:13 smithi043 ceph-mon[31823]: Updating smithi172:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:26:13.769 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:13 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/4194515456' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-28T22:26:14.221 INFO:tasks.cephadm:Adding mon.a on smithi043 2024-03-28T22:26:14.221 INFO:tasks.cephadm:Adding mon.b on smithi082 2024-03-28T22:26:14.221 INFO:tasks.cephadm:Adding mon.c on smithi172 2024-03-28T22:26:14.221 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph orch apply mon '3;smithi043:172.21.15.43=a;smithi082:172.21.15.82=b;smithi172:172.21.15.172=c' 2024-03-28T22:26:14.451 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:26:14.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:14 smithi043 ceph-mon[31823]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:26:14.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:14 smithi043 ceph-mon[31823]: Updating smithi172:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T22:26:14.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:14 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/4194515456' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-28T22:26:14.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:14 smithi043 ceph-mon[31823]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T22:26:14.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:14 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:14.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:14 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:14.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:14 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:15.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:15.884+0000 7f9ed8c71700 1 -- 172.21.15.172:0/3356301658 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ed4102de0 msgr2=0x7f9ed41031c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:15.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:15.884+0000 7f9ed8c71700 1 --2- 172.21.15.172:0/3356301658 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ed4102de0 0x7f9ed41031c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4009910 tx=0x7f9ec4009c20 comp rx=0 tx=0).stop 2024-03-28T22:26:15.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:15.884+0000 7f9ed8c71700 1 -- 172.21.15.172:0/3356301658 shutdown_connections 2024-03-28T22:26:15.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:15.884+0000 7f9ed8c71700 1 --2- 172.21.15.172:0/3356301658 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ed4102de0 0x7f9ed41031c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:15.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:15.884+0000 7f9ed8c71700 1 -- 172.21.15.172:0/3356301658 >> 172.21.15.172:0/3356301658 conn(0x7f9ed40fc610 msgr2=0x7f9ed40fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:26:15.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:15.884+0000 7f9ed8c71700 1 -- 172.21.15.172:0/3356301658 shutdown_connections 2024-03-28T22:26:15.884 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:15.884+0000 7f9ed8c71700 1 -- 172.21.15.172:0/3356301658 wait complete. 2024-03-28T22:26:15.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:15.885+0000 7f9ed8c71700 1 Processor -- start 2024-03-28T22:26:15.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:15.885+0000 7f9ed8c71700 1 -- start start 2024-03-28T22:26:15.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:15.885+0000 7f9ed8c71700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ed4102de0 0x7f9ed419d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:26:15.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:15.885+0000 7f9ed8c71700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed410c2d0 con 0x7f9ed4102de0 2024-03-28T22:26:15.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:15.886+0000 7f9ed259c700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ed4102de0 0x7f9ed419d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:26:15.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:15.886+0000 7f9ed259c700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ed4102de0 0x7f9ed419d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.172:38134/0 (socket says 172.21.15.172:38134) 2024-03-28T22:26:15.885 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:15.886+0000 7f9ed259c700 1 -- 172.21.15.172:0/4278657775 learned_addr learned my addr 172.21.15.172:0/4278657775 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-28T22:26:15.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:15.886+0000 7f9ed259c700 1 -- 172.21.15.172:0/4278657775 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ed419d940 con 0x7f9ed4102de0 2024-03-28T22:26:15.886 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:15.887+0000 7f9ed259c700 1 --2- 172.21.15.172:0/4278657775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ed4102de0 0x7f9ed419d400 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4006010 tx=0x7f9ec4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:26:15.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:15.887+0000 7f9ec37fe700 1 -- 172.21.15.172:0/4278657775 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ec4026070 con 0x7f9ed4102de0 2024-03-28T22:26:15.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:15.887+0000 7f9ed8c71700 1 -- 172.21.15.172:0/4278657775 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ed419dbd0 con 0x7f9ed4102de0 2024-03-28T22:26:15.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:15.887+0000 7f9ed8c71700 1 -- 172.21.15.172:0/4278657775 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ed41975c0 con 0x7f9ed4102de0 2024-03-28T22:26:15.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:15.887+0000 7f9ec37fe700 1 -- 172.21.15.172:0/4278657775 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ec400bc30 con 0x7f9ed4102de0 2024-03-28T22:26:15.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:15.887+0000 7f9ec37fe700 1 -- 172.21.15.172:0/4278657775 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ec4003960 con 0x7f9ed4102de0 2024-03-28T22:26:15.887 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:15.888+0000 7f9ed8c71700 1 -- 172.21.15.172:0/4278657775 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9eb4005330 con 0x7f9ed4102de0 2024-03-28T22:26:15.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:15.889+0000 7f9ec37fe700 1 -- 172.21.15.172:0/4278657775 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f9ec4029070 con 0x7f9ed4102de0 2024-03-28T22:26:15.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:15.889+0000 7f9ec37fe700 1 --2- 172.21.15.172:0/4278657775 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9ebc038ca0 0x7f9ebc03b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:26:15.888 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:15.889+0000 7f9ec37fe700 1 -- 172.21.15.172:0/4278657775 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f9ec401e070 con 0x7f9ed4102de0 2024-03-28T22:26:15.889 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:15.889+0000 7f9ed1d9b700 1 --2- 172.21.15.172:0/4278657775 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9ebc038ca0 0x7f9ebc03b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:26:15.890 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:15.891+0000 7f9ed1d9b700 1 --2- 172.21.15.172:0/4278657775 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9ebc038ca0 0x7f9ebc03b160 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8006fd0 tx=0x7f9ec8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:26:15.904 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:15.904+0000 7f9ec37fe700 1 -- 172.21.15.172:0/4278657775 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9ec4013070 con 0x7f9ed4102de0 2024-03-28T22:26:16.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:15 smithi043 ceph-mon[31823]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:16.107 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:16.107+0000 7f9ed8c71700 1 -- 172.21.15.172:0/4278657775 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi043:172.21.15.43=a;smithi082:172.21.15.82=b;smithi172:172.21.15.172=c", "target": ["mon-mgr", ""]}) -- 0x7f9eb4000c80 con 0x7f9ebc038ca0 2024-03-28T22:26:16.122 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:16.122+0000 7f9ec37fe700 1 -- 172.21.15.172:0/4278657775 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f9eb4000c80 con 0x7f9ebc038ca0 2024-03-28T22:26:16.122 INFO:teuthology.orchestra.run.smithi172.stdout:Scheduled mon update... 2024-03-28T22:26:16.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:16.125+0000 7f9ed8c71700 1 -- 172.21.15.172:0/4278657775 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9ebc038ca0 msgr2=0x7f9ebc03b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:16.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:16.125+0000 7f9ed8c71700 1 --2- 172.21.15.172:0/4278657775 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9ebc038ca0 0x7f9ebc03b160 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8006fd0 tx=0x7f9ec8006e40 comp rx=0 tx=0).stop 2024-03-28T22:26:16.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:16.125+0000 7f9ed8c71700 1 -- 172.21.15.172:0/4278657775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ed4102de0 msgr2=0x7f9ed419d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:16.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:16.125+0000 7f9ed8c71700 1 --2- 172.21.15.172:0/4278657775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ed4102de0 0x7f9ed419d400 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4006010 tx=0x7f9ec4005030 comp rx=0 tx=0).stop 2024-03-28T22:26:16.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:16.125+0000 7f9ed8c71700 1 -- 172.21.15.172:0/4278657775 shutdown_connections 2024-03-28T22:26:16.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:16.125+0000 7f9ed8c71700 1 --2- 172.21.15.172:0/4278657775 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9ebc038ca0 0x7f9ebc03b160 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:16.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:16.125+0000 7f9ed8c71700 1 --2- 172.21.15.172:0/4278657775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ed4102de0 0x7f9ed419d400 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:16.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:16.125+0000 7f9ed8c71700 1 -- 172.21.15.172:0/4278657775 >> 172.21.15.172:0/4278657775 conn(0x7f9ed40fc610 msgr2=0x7f9ed4103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:26:16.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:16.125+0000 7f9ed8c71700 1 -- 172.21.15.172:0/4278657775 shutdown_connections 2024-03-28T22:26:16.125 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:16.125+0000 7f9ed8c71700 1 -- 172.21.15.172:0/4278657775 wait complete. 2024-03-28T22:26:16.780 DEBUG:teuthology.orchestra.run.smithi082:mon.b> sudo journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mon.b.service 2024-03-28T22:26:16.783 DEBUG:teuthology.orchestra.run.smithi172:mon.c> sudo journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mon.c.service 2024-03-28T22:26:16.785 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T22:26:16.785 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T22:26:17.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:17 smithi043 ceph-mon[31823]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:17.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:17 smithi043 ceph-mon[31823]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi043:172.21.15.43=a;smithi082:172.21.15.82=b;smithi172:172.21.15.172=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:26:17.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:17 smithi043 ceph-mon[31823]: Saving service mon spec with placement smithi043:172.21.15.43=a;smithi082:172.21.15.82=b;smithi172:172.21.15.172=c;count:3 2024-03-28T22:26:17.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:17 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:17.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:17 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:26:17.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:17 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:17.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:17 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:26:17.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:17 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:17.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:17 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T22:26:17.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:17 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:17.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:17 smithi043 ceph-mon[31823]: Deploying daemon mon.c on smithi172 2024-03-28T22:26:17.572 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:26:19.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:19 smithi043 ceph-mon[31823]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:20.412 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.410+0000 7f04f7fff700 1 -- 172.21.15.172:0/3918342710 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f04f0007b20 con 0x7f04f8101e80 2024-03-28T22:26:20.412 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.411+0000 7f04ff3b6700 1 -- 172.21.15.172:0/3918342710 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f04f8101e80 msgr2=0x7f04f8102260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:20.412 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.411+0000 7f04ff3b6700 1 --2- 172.21.15.172:0/3918342710 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f04f8101e80 0x7f04f8102260 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f04f000b0a0 tx=0x7f04f000b3b0 comp rx=0 tx=0).stop 2024-03-28T22:26:20.412 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.411+0000 7f04ff3b6700 1 -- 172.21.15.172:0/3918342710 shutdown_connections 2024-03-28T22:26:20.412 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.411+0000 7f04ff3b6700 1 --2- 172.21.15.172:0/3918342710 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f04f8101e80 0x7f04f8102260 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:20.412 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.411+0000 7f04ff3b6700 1 -- 172.21.15.172:0/3918342710 >> 172.21.15.172:0/3918342710 conn(0x7f04f80fd750 msgr2=0x7f04f80ffb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:26:20.412 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.411+0000 7f04ff3b6700 1 -- 172.21.15.172:0/3918342710 shutdown_connections 2024-03-28T22:26:20.412 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.411+0000 7f04ff3b6700 1 -- 172.21.15.172:0/3918342710 wait complete. 2024-03-28T22:26:20.412 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.411+0000 7f04ff3b6700 1 Processor -- start 2024-03-28T22:26:20.412 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.411+0000 7f04ff3b6700 1 -- start start 2024-03-28T22:26:20.412 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.412+0000 7f04ff3b6700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f04f8101e80 0x7f04f8197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:26:20.413 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.412+0000 7f04ff3b6700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f04f810c250 con 0x7f04f8101e80 2024-03-28T22:26:20.413 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.412+0000 7f04fd152700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f04f8101e80 0x7f04f8197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:26:20.413 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.412+0000 7f04fd152700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f04f8101e80 0x7f04f8197680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.172:38156/0 (socket says 172.21.15.172:38156) 2024-03-28T22:26:20.413 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.412+0000 7f04fd152700 1 -- 172.21.15.172:0/1225939129 learned_addr learned my addr 172.21.15.172:0/1225939129 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-28T22:26:20.414 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.413+0000 7f04fd152700 1 -- 172.21.15.172:0/1225939129 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04f8197bc0 con 0x7f04f8101e80 2024-03-28T22:26:20.414 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.413+0000 7f04fd152700 1 --2- 172.21.15.172:0/1225939129 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f04f8101e80 0x7f04f8197680 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f04f0015040 tx=0x7f04f0003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:26:20.414 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.413+0000 7f04f67fc700 1 -- 172.21.15.172:0/1225939129 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f04f0017040 con 0x7f04f8101e80 2024-03-28T22:26:20.414 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.414+0000 7f04ff3b6700 1 -- 172.21.15.172:0/1225939129 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04f8197e50 con 0x7f04f8101e80 2024-03-28T22:26:20.414 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.414+0000 7f04ff3b6700 1 -- 172.21.15.172:0/1225939129 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04f819bf50 con 0x7f04f8101e80 2024-03-28T22:26:20.414 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.414+0000 7f04f67fc700 1 -- 172.21.15.172:0/1225939129 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04f0022070 con 0x7f04f8101e80 2024-03-28T22:26:20.414 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.414+0000 7f04f67fc700 1 -- 172.21.15.172:0/1225939129 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f04f0009410 con 0x7f04f8101e80 2024-03-28T22:26:20.414 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.414+0000 7f04ff3b6700 1 -- 172.21.15.172:0/1225939129 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04f8103360 con 0x7f04f8101e80 2024-03-28T22:26:20.415 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.415+0000 7f04f67fc700 1 -- 172.21.15.172:0/1225939129 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f04f000bc10 con 0x7f04f8101e80 2024-03-28T22:26:20.415 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.415+0000 7f04f67fc700 1 --2- 172.21.15.172:0/1225939129 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f04e0038de0 0x7f04e003b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:26:20.415 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.415+0000 7f04f67fc700 1 -- 172.21.15.172:0/1225939129 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f04f000f050 con 0x7f04f8101e80 2024-03-28T22:26:20.416 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.416+0000 7f04fc951700 1 --2- 172.21.15.172:0/1225939129 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f04e0038de0 0x7f04e003b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:26:20.416 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.417+0000 7f04fc951700 1 --2- 172.21.15.172:0/1225939129 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f04e0038de0 0x7f04e003b2a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f04ec006fd0 tx=0x7f04ec006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:26:20.431 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.431+0000 7f04f67fc700 1 -- 172.21.15.172:0/1225939129 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f04f0007e20 con 0x7f04f8101e80 2024-03-28T22:26:20.722 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.722+0000 7f04ff3b6700 1 -- 172.21.15.172:0/1225939129 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f04f81989e0 con 0x7f04f8101e80 2024-03-28T22:26:20.723 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.724+0000 7f04f67fc700 1 -- 172.21.15.172:0/1225939129 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f04f002d250 con 0x7f04f8101e80 2024-03-28T22:26:20.723 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-28T22:26:20.723 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":1,"fsid":"c2276428-ed51-11ee-95d0-87774f69a715","modified":"2024-03-28T22:24:15.893393Z","created":"2024-03-28T22:24:15.893393Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-28T22:26:20.725 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.726+0000 7f04ff3b6700 1 -- 172.21.15.172:0/1225939129 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f04e0038de0 msgr2=0x7f04e003b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:20.726 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.726+0000 7f04ff3b6700 1 --2- 172.21.15.172:0/1225939129 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f04e0038de0 0x7f04e003b2a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f04ec006fd0 tx=0x7f04ec006e40 comp rx=0 tx=0).stop 2024-03-28T22:26:20.726 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.726+0000 7f04ff3b6700 1 -- 172.21.15.172:0/1225939129 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f04f8101e80 msgr2=0x7f04f8197680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:20.726 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.726+0000 7f04ff3b6700 1 --2- 172.21.15.172:0/1225939129 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f04f8101e80 0x7f04f8197680 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f04f0015040 tx=0x7f04f0003d50 comp rx=0 tx=0).stop 2024-03-28T22:26:20.726 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.726+0000 7f04ff3b6700 1 -- 172.21.15.172:0/1225939129 shutdown_connections 2024-03-28T22:26:20.726 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.726+0000 7f04ff3b6700 1 --2- 172.21.15.172:0/1225939129 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f04e0038de0 0x7f04e003b2a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:20.726 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.726+0000 7f04ff3b6700 1 --2- 172.21.15.172:0/1225939129 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f04f8101e80 0x7f04f8197680 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:20.726 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.726+0000 7f04ff3b6700 1 -- 172.21.15.172:0/1225939129 >> 172.21.15.172:0/1225939129 conn(0x7f04f80fd750 msgr2=0x7f04f8103850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:26:20.726 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.726+0000 7f04ff3b6700 1 -- 172.21.15.172:0/1225939129 shutdown_connections 2024-03-28T22:26:20.726 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:20.726+0000 7f04ff3b6700 1 -- 172.21.15.172:0/1225939129 wait complete. 2024-03-28T22:26:20.726 INFO:teuthology.orchestra.run.smithi172.stderr:dumped monmap epoch 1 2024-03-28T22:26:21.026 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:20 smithi172 systemd[1]: Starting Ceph mon.c for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:26:21.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:21 smithi043 ceph-mon[31823]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:21.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:21 smithi043 ceph-mon[31823]: from='client.? 172.21.15.172:0/1225939129' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T22:26:21.631 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 podman[36547]: 2024-03-28 22:26:21.412415968 +0000 UTC m=+0.018023225 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:26:21.632 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 podman[36547]: 2024-03-28 22:26:21.520262014 +0000 UTC m=+0.125869281 container create 2fa7788646ace03a9eec69aa78bcc027ad37990e426a0c386cf01878fd8c0a94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mon-c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-323d23d, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux ) 2024-03-28T22:26:21.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 podman[36547]: 2024-03-28 22:26:21.739390581 +0000 UTC m=+0.344997833 container init 2fa7788646ace03a9eec69aa78bcc027ad37990e426a0c386cf01878fd8c0a94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mon-c, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-28T22:26:21.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 podman[36547]: 2024-03-28 22:26:21.742520745 +0000 UTC m=+0.348127993 container start 2fa7788646ace03a9eec69aa78bcc027ad37990e426a0c386cf01878fd8c0a94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mon-c, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=squid-323d23d, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-28T22:26:21.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: set uid:gid to 167:167 (ceph:ceph) 2024-03-28T22:26:21.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: ceph version 19.0.0-1899-g323d23dc (323d23dc228bbcb7f16f188fdc9c0dcb801844be) squid (dev), process ceph-mon, pid 2 2024-03-28T22:26:21.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: pidfile_write: ignore empty --pid-file 2024-03-28T22:26:21.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: load: jerasure load: lrc 2024-03-28T22:26:21.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: RocksDB version: 7.9.2 2024-03-28T22:26:21.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Git sha 0 2024-03-28T22:26:21.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Compile date 2024-03-27 16:17:23 2024-03-28T22:26:21.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: DB SUMMARY 2024-03-28T22:26:21.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: DB Session ID: ZFBN4R2LU9XFYXRBWRMV 2024-03-28T22:26:21.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: CURRENT file: CURRENT 2024-03-28T22:26:21.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: IDENTITY file: IDENTITY 2024-03-28T22:26:21.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-28T22:26:21.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-28T22:26:21.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-28T22:26:21.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.error_if_exists: 0 2024-03-28T22:26:21.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.create_if_missing: 0 2024-03-28T22:26:21.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.paranoid_checks: 1 2024-03-28T22:26:21.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-28T22:26:21.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-28T22:26:21.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-28T22:26:21.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.env: 0x55e684893c60 2024-03-28T22:26:21.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.fs: PosixFileSystem 2024-03-28T22:26:21.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.info_log: 0x55e68693bae0 2024-03-28T22:26:21.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.max_file_opening_threads: 16 2024-03-28T22:26:21.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.statistics: (nil) 2024-03-28T22:26:21.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.use_fsync: 0 2024-03-28T22:26:21.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.max_log_file_size: 0 2024-03-28T22:26:21.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-28T22:26:21.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-28T22:26:21.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.keep_log_file_num: 1000 2024-03-28T22:26:21.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.recycle_log_file_num: 0 2024-03-28T22:26:21.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.allow_fallocate: 1 2024-03-28T22:26:21.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.allow_mmap_reads: 0 2024-03-28T22:26:21.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.allow_mmap_writes: 0 2024-03-28T22:26:21.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.use_direct_reads: 0 2024-03-28T22:26:21.901 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-28T22:26:21.901 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.create_missing_column_families: 0 2024-03-28T22:26:21.901 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.db_log_dir: 2024-03-28T22:26:21.901 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.wal_dir: 2024-03-28T22:26:21.901 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-28T22:26:21.901 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-28T22:26:21.901 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-28T22:26:21.901 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-28T22:26:21.901 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-28T22:26:21.901 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-28T22:26:21.901 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.advise_random_on_open: 1 2024-03-28T22:26:21.901 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.db_write_buffer_size: 0 2024-03-28T22:26:21.901 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.write_buffer_manager: 0x55e68690e1e0 2024-03-28T22:26:21.901 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-28T22:26:21.902 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-28T22:26:21.902 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-28T22:26:21.902 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.rate_limiter: (nil) 2024-03-28T22:26:21.902 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-28T22:26:21.902 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.wal_recovery_mode: 2 2024-03-28T22:26:21.902 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.enable_thread_tracking: 0 2024-03-28T22:26:21.902 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.enable_pipelined_write: 0 2024-03-28T22:26:21.902 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.unordered_write: 0 2024-03-28T22:26:21.902 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-28T22:26:21.902 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-28T22:26:21.902 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-28T22:26:21.902 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-28T22:26:21.902 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.row_cache: None 2024-03-28T22:26:21.902 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.wal_filter: None 2024-03-28T22:26:21.904 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-28T22:26:21.904 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.allow_ingest_behind: 0 2024-03-28T22:26:21.904 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.two_write_queues: 0 2024-03-28T22:26:21.904 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.manual_wal_flush: 0 2024-03-28T22:26:21.904 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.wal_compression: 0 2024-03-28T22:26:21.904 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.atomic_flush: 0 2024-03-28T22:26:21.904 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-28T22:26:21.904 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-28T22:26:21.904 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-28T22:26:21.904 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.log_readahead_size: 0 2024-03-28T22:26:21.905 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-28T22:26:21.905 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.best_efforts_recovery: 0 2024-03-28T22:26:21.905 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-28T22:26:21.905 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-28T22:26:21.905 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.allow_data_in_errors: 0 2024-03-28T22:26:21.905 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.db_host_id: __hostname__ 2024-03-28T22:26:21.905 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-28T22:26:21.905 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.max_background_jobs: 2 2024-03-28T22:26:21.905 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.max_background_compactions: -1 2024-03-28T22:26:21.905 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.max_subcompactions: 1 2024-03-28T22:26:21.905 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-28T22:26:21.905 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-28T22:26:21.905 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-28T22:26:21.905 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.max_total_wal_size: 0 2024-03-28T22:26:21.905 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-28T22:26:21.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-28T22:26:21.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-28T22:26:21.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-28T22:26:21.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.max_open_files: -1 2024-03-28T22:26:21.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.bytes_per_sync: 0 2024-03-28T22:26:21.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-28T22:26:21.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-28T22:26:21.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.compaction_readahead_size: 0 2024-03-28T22:26:21.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.max_background_flushes: -1 2024-03-28T22:26:21.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Compression algorithms supported: 2024-03-28T22:26:21.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-28T22:26:21.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: kZSTD supported: 0 2024-03-28T22:26:21.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: kXpressCompression supported: 0 2024-03-28T22:26:21.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: kLZ4HCCompression supported: 1 2024-03-28T22:26:21.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: kZlibCompression supported: 1 2024-03-28T22:26:21.907 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: kSnappyCompression supported: 1 2024-03-28T22:26:21.907 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: kLZ4Compression supported: 1 2024-03-28T22:26:21.907 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: kBZip2Compression supported: 0 2024-03-28T22:26:21.907 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-28T22:26:21.907 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-28T22:26:21.907 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-28T22:26:21.907 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-28T22:26:21.907 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-28T22:26:21.907 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.merge_operator: 2024-03-28T22:26:21.907 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.compaction_filter: None 2024-03-28T22:26:21.907 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.compaction_filter_factory: None 2024-03-28T22:26:21.907 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.sst_partitioner_factory: None 2024-03-28T22:26:21.907 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-28T22:26:21.907 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-28T22:26:21.907 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e68693bbe0) 2024-03-28T22:26:21.907 INFO:journalctl@ceph.mon.c.smithi172.stdout: cache_index_and_filter_blocks: 1 2024-03-28T22:26:21.908 INFO:journalctl@ceph.mon.c.smithi172.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-28T22:26:21.908 INFO:journalctl@ceph.mon.c.smithi172.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-28T22:26:21.908 INFO:journalctl@ceph.mon.c.smithi172.stdout: pin_top_level_index_and_filter: 1 2024-03-28T22:26:21.908 INFO:journalctl@ceph.mon.c.smithi172.stdout: index_type: 0 2024-03-28T22:26:21.908 INFO:journalctl@ceph.mon.c.smithi172.stdout: data_block_index_type: 0 2024-03-28T22:26:21.908 INFO:journalctl@ceph.mon.c.smithi172.stdout: index_shortening: 1 2024-03-28T22:26:21.908 INFO:journalctl@ceph.mon.c.smithi172.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-28T22:26:21.908 INFO:journalctl@ceph.mon.c.smithi172.stdout: checksum: 4 2024-03-28T22:26:21.908 INFO:journalctl@ceph.mon.c.smithi172.stdout: no_block_cache: 0 2024-03-28T22:26:21.908 INFO:journalctl@ceph.mon.c.smithi172.stdout: block_cache: 0x55e68693f090 2024-03-28T22:26:21.908 INFO:journalctl@ceph.mon.c.smithi172.stdout: block_cache_name: BinnedLRUCache 2024-03-28T22:26:21.908 INFO:journalctl@ceph.mon.c.smithi172.stdout: block_cache_options: 2024-03-28T22:26:21.909 INFO:journalctl@ceph.mon.c.smithi172.stdout: capacity : 536870912 2024-03-28T22:26:21.909 INFO:journalctl@ceph.mon.c.smithi172.stdout: num_shard_bits : 4 2024-03-28T22:26:21.909 INFO:journalctl@ceph.mon.c.smithi172.stdout: strict_capacity_limit : 0 2024-03-28T22:26:21.909 INFO:journalctl@ceph.mon.c.smithi172.stdout: high_pri_pool_ratio: 0.000 2024-03-28T22:26:21.909 INFO:journalctl@ceph.mon.c.smithi172.stdout: block_cache_compressed: (nil) 2024-03-28T22:26:21.909 INFO:journalctl@ceph.mon.c.smithi172.stdout: persistent_cache: (nil) 2024-03-28T22:26:21.909 INFO:journalctl@ceph.mon.c.smithi172.stdout: block_size: 4096 2024-03-28T22:26:21.909 INFO:journalctl@ceph.mon.c.smithi172.stdout: block_size_deviation: 10 2024-03-28T22:26:21.909 INFO:journalctl@ceph.mon.c.smithi172.stdout: block_restart_interval: 16 2024-03-28T22:26:21.909 INFO:journalctl@ceph.mon.c.smithi172.stdout: index_block_restart_interval: 1 2024-03-28T22:26:21.909 INFO:journalctl@ceph.mon.c.smithi172.stdout: metadata_block_size: 4096 2024-03-28T22:26:21.909 INFO:journalctl@ceph.mon.c.smithi172.stdout: partition_filters: 0 2024-03-28T22:26:21.909 INFO:journalctl@ceph.mon.c.smithi172.stdout: use_delta_encoding: 1 2024-03-28T22:26:21.909 INFO:journalctl@ceph.mon.c.smithi172.stdout: filter_policy: bloomfilter 2024-03-28T22:26:21.909 INFO:journalctl@ceph.mon.c.smithi172.stdout: whole_key_filtering: 1 2024-03-28T22:26:21.910 INFO:journalctl@ceph.mon.c.smithi172.stdout: verify_compression: 0 2024-03-28T22:26:21.910 INFO:journalctl@ceph.mon.c.smithi172.stdout: read_amp_bytes_per_bit: 0 2024-03-28T22:26:21.910 INFO:journalctl@ceph.mon.c.smithi172.stdout: format_version: 5 2024-03-28T22:26:21.910 INFO:journalctl@ceph.mon.c.smithi172.stdout: enable_index_compression: 1 2024-03-28T22:26:21.910 INFO:journalctl@ceph.mon.c.smithi172.stdout: block_align: 0 2024-03-28T22:26:21.910 INFO:journalctl@ceph.mon.c.smithi172.stdout: max_auto_readahead_size: 262144 2024-03-28T22:26:21.910 INFO:journalctl@ceph.mon.c.smithi172.stdout: prepopulate_block_cache: 0 2024-03-28T22:26:21.910 INFO:journalctl@ceph.mon.c.smithi172.stdout: initial_auto_readahead_size: 8192 2024-03-28T22:26:21.910 INFO:journalctl@ceph.mon.c.smithi172.stdout: num_file_reads_for_auto_readahead: 2 2024-03-28T22:26:21.910 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.write_buffer_size: 33554432 2024-03-28T22:26:21.910 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.max_write_buffer_number: 2 2024-03-28T22:26:21.910 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.compression: NoCompression 2024-03-28T22:26:21.910 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.bottommost_compression: Disabled 2024-03-28T22:26:21.910 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.prefix_extractor: nullptr 2024-03-28T22:26:21.910 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-28T22:26:21.911 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.num_levels: 7 2024-03-28T22:26:21.911 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-28T22:26:21.911 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-28T22:26:21.911 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-28T22:26:21.911 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-28T22:26:21.911 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-28T22:26:21.911 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-28T22:26:21.911 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-28T22:26:21.911 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-28T22:26:21.911 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-28T22:26:21.911 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-28T22:26:21.911 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-28T22:26:21.911 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-28T22:26:21.911 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-28T22:26:21.911 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.compression_opts.level: 32767 2024-03-28T22:26:21.912 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.compression_opts.strategy: 0 2024-03-28T22:26:21.912 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-28T22:26:21.912 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-28T22:26:21.912 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-28T22:26:21.912 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-28T22:26:21.912 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.compression_opts.enabled: false 2024-03-28T22:26:21.912 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-28T22:26:21.912 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-28T22:26:21.912 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-28T22:26:21.912 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-28T22:26:21.912 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.target_file_size_base: 67108864 2024-03-28T22:26:21.912 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-28T22:26:21.912 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-28T22:26:21.912 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-28T22:26:21.912 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-28T22:26:21.912 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-28T22:26:21.913 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-28T22:26:21.913 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-28T22:26:21.913 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-28T22:26:21.913 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-28T22:26:21.913 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-28T22:26:21.913 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-28T22:26:21.913 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-28T22:26:21.913 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-28T22:26:21.913 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-28T22:26:21.913 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.arena_block_size: 1048576 2024-03-28T22:26:21.913 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-28T22:26:21.913 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-28T22:26:21.913 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.disable_auto_compactions: 0 2024-03-28T22:26:21.913 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-28T22:26:21.913 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-28T22:26:21.914 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-28T22:26:21.914 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-28T22:26:21.914 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-28T22:26:21.914 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-28T22:26:21.914 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-28T22:26:21.914 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-28T22:26:21.914 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-28T22:26:21.914 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-28T22:26:21.914 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.table_properties_collectors: 2024-03-28T22:26:21.914 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.inplace_update_support: 0 2024-03-28T22:26:21.914 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-28T22:26:21.914 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-28T22:26:21.914 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-28T22:26:21.914 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-28T22:26:21.914 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.bloom_locality: 0 2024-03-28T22:26:21.915 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.max_successive_merges: 0 2024-03-28T22:26:21.915 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-28T22:26:21.915 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.paranoid_file_checks: 0 2024-03-28T22:26:21.915 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.force_consistency_checks: 1 2024-03-28T22:26:21.915 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.report_bg_io_stats: 0 2024-03-28T22:26:21.915 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.ttl: 2592000 2024-03-28T22:26:21.915 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-28T22:26:21.915 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-28T22:26:21.915 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-28T22:26:21.915 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.enable_blob_files: false 2024-03-28T22:26:21.915 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.min_blob_size: 0 2024-03-28T22:26:21.915 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.blob_file_size: 268435456 2024-03-28T22:26:21.915 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-28T22:26:21.915 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-28T22:26:21.915 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-28T22:26:21.915 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-28T22:26:21.916 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-28T22:26:21.916 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.blob_file_starting_level: 0 2024-03-28T22:26:21.916 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-28T22:26:21.916 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-28T22:26:21.916 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-28T22:26:21.916 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7d00c459-22ee-4b17-935f-9bfc4341b4fe 2024-03-28T22:26:21.916 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711664781772376, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-28T22:26:21.916 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-28T22:26:21.916 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711664781772799, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711664781, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7d00c459-22ee-4b17-935f-9bfc4341b4fe", "db_session_id": "ZFBN4R2LU9XFYXRBWRMV", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-28T22:26:21.916 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711664781772879, "job": 1, "event": "recovery_finished"} 2024-03-28T22:26:21.916 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-28T22:26:21.916 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-28T22:26:21.916 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e686a30000 2024-03-28T22:26:21.916 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: DB pointer 0x55e686a1a000 2024-03-28T22:26:21.917 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-28T22:26:21.917 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-28T22:26:21.917 INFO:journalctl@ceph.mon.c.smithi172.stdout: ** DB Stats ** 2024-03-28T22:26:21.917 INFO:journalctl@ceph.mon.c.smithi172.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T22:26:21.917 INFO:journalctl@ceph.mon.c.smithi172.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-28T22:26:21.917 INFO:journalctl@ceph.mon.c.smithi172.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T22:26:21.917 INFO:journalctl@ceph.mon.c.smithi172.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T22:26:21.917 INFO:journalctl@ceph.mon.c.smithi172.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-28T22:26:21.917 INFO:journalctl@ceph.mon.c.smithi172.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T22:26:21.917 INFO:journalctl@ceph.mon.c.smithi172.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T22:26:21.917 INFO:journalctl@ceph.mon.c.smithi172.stdout: 2024-03-28T22:26:21.917 INFO:journalctl@ceph.mon.c.smithi172.stdout: ** Compaction Stats [default] ** 2024-03-28T22:26:21.917 INFO:journalctl@ceph.mon.c.smithi172.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T22:26:21.917 INFO:journalctl@ceph.mon.c.smithi172.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-28T22:26:21.917 INFO:journalctl@ceph.mon.c.smithi172.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-28T22:26:21.917 INFO:journalctl@ceph.mon.c.smithi172.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-28T22:26:21.918 INFO:journalctl@ceph.mon.c.smithi172.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-28T22:26:21.918 INFO:journalctl@ceph.mon.c.smithi172.stdout: 2024-03-28T22:26:21.918 INFO:journalctl@ceph.mon.c.smithi172.stdout: ** Compaction Stats [default] ** 2024-03-28T22:26:21.918 INFO:journalctl@ceph.mon.c.smithi172.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T22:26:21.918 INFO:journalctl@ceph.mon.c.smithi172.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-28T22:26:21.918 INFO:journalctl@ceph.mon.c.smithi172.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-28T22:26:21.918 INFO:journalctl@ceph.mon.c.smithi172.stdout: 2024-03-28T22:26:21.918 INFO:journalctl@ceph.mon.c.smithi172.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-28T22:26:21.918 INFO:journalctl@ceph.mon.c.smithi172.stdout: 2024-03-28T22:26:21.918 INFO:journalctl@ceph.mon.c.smithi172.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T22:26:21.918 INFO:journalctl@ceph.mon.c.smithi172.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-28T22:26:21.918 INFO:journalctl@ceph.mon.c.smithi172.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-28T22:26:21.918 INFO:journalctl@ceph.mon.c.smithi172.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-28T22:26:21.918 INFO:journalctl@ceph.mon.c.smithi172.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-28T22:26:21.918 INFO:journalctl@ceph.mon.c.smithi172.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-28T22:26:21.919 INFO:journalctl@ceph.mon.c.smithi172.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T22:26:21.919 INFO:journalctl@ceph.mon.c.smithi172.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T22:26:21.919 INFO:journalctl@ceph.mon.c.smithi172.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-28T22:26:21.919 INFO:journalctl@ceph.mon.c.smithi172.stdout: Block cache BinnedLRUCache@0x55e68693f090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-28T22:26:21.919 INFO:journalctl@ceph.mon.c.smithi172.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-28T22:26:21.919 INFO:journalctl@ceph.mon.c.smithi172.stdout: 2024-03-28T22:26:21.919 INFO:journalctl@ceph.mon.c.smithi172.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-28T22:26:21.919 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-28T22:26:21.919 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: using public_addr v2:172.21.15.172:0/0 -> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] 2024-03-28T22:26:21.919 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: starting mon.c rank -1 at public addrs [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] at bind addrs [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:26:21.919 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: mon.c@-1(???) e0 preinit fsid c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:26:21.919 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: mon.c@-1(synchronizing).mds e1 new map 2024-03-28T22:26:21.919 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-28T22:26:21.919 INFO:journalctl@ceph.mon.c.smithi172.stdout: e1 2024-03-28T22:26:21.919 INFO:journalctl@ceph.mon.c.smithi172.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T22:26:21.920 INFO:journalctl@ceph.mon.c.smithi172.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-28T22:26:21.920 INFO:journalctl@ceph.mon.c.smithi172.stdout: legacy client fscid: -1 2024-03-28T22:26:21.920 INFO:journalctl@ceph.mon.c.smithi172.stdout: 2024-03-28T22:26:21.920 INFO:journalctl@ceph.mon.c.smithi172.stdout: No filesystems configured 2024-03-28T22:26:21.920 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-28T22:26:21.920 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-28T22:26:21.920 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-28T22:26:21.920 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-28T22:26:21.920 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-28T22:26:21.920 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-28T22:26:21.920 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-28T22:26:21.920 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T22:26:21.920 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T22:26:21.920 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T22:26:21.920 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:21.921 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:21.921 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:21.921 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:21.921 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:26:21.921 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:21.921 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:21.921 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:21.921 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:21.921 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:21.921 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi082", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:26:21.921 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:21.921 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:26:21.921 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: Updating smithi082:/etc/ceph/ceph.conf 2024-03-28T22:26:21.921 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: Updating smithi082:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:26:21.921 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: Updating smithi082:/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:26:21.922 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: Updating smithi082:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T22:26:21.922 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi172", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:26:21.922 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:21.922 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:21.922 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:21.922 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:21.922 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: Deploying cephadm binary to smithi172 2024-03-28T22:26:21.922 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:21.922 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:21.922 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:21.922 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: Added host smithi172 2024-03-28T22:26:21.922 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:26:21.922 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:21.922 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:21.922 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:21.923 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:21.923 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:26:21.923 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:21.923 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:21.923 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:21.923 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:21.923 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:21.923 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:21.923 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi172", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:26:21.923 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:21.923 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:26:21.923 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-28T22:26:21.923 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: Updating smithi172:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:26:21.923 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/4194515456' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-28T22:26:21.924 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:26:21.924 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: Updating smithi172:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T22:26:21.924 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/4194515456' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-28T22:26:21.924 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T22:26:21.924 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:21.924 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:21.924 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:21.924 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:21.924 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:21.924 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi043:172.21.15.43=a;smithi082:172.21.15.82=b;smithi172:172.21.15.172=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:26:21.924 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: Saving service mon spec with placement smithi043:172.21.15.43=a;smithi082:172.21.15.82=b;smithi172:172.21.15.172=c;count:3 2024-03-28T22:26:21.924 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:21.924 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:26:21.924 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:21.924 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:26:21.924 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:21.925 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T22:26:21.925 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:21.925 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: Deploying daemon mon.c on smithi172 2024-03-28T22:26:21.925 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:21.925 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:21.925 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: from='client.? 172.21.15.172:0/1225939129' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T22:26:21.925 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 ceph-mon[36566]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-28T22:26:21.925 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 bash[36547]: 2fa7788646ace03a9eec69aa78bcc027ad37990e426a0c386cf01878fd8c0a94 2024-03-28T22:26:21.925 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:21 smithi172 systemd[1]: Started Ceph mon.c for c2276428-ed51-11ee-95d0-87774f69a715. 2024-03-28T22:26:22.330 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T22:26:22.330 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T22:26:22.567 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T22:26:27.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.108+0000 7f9ea71d3700 1 -- 172.21.15.172:0/1896784884 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e900035c0 msgr2=0x7f9e90005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:27.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.108+0000 7f9ea71d3700 1 --2- 172.21.15.172:0/1896784884 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e900035c0 0x7f9e90005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0100b70 tx=0x7f9e9400feb0 comp rx=0 tx=0).stop 2024-03-28T22:26:27.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.108+0000 7f9ea71d3700 1 -- 172.21.15.172:0/1896784884 shutdown_connections 2024-03-28T22:26:27.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.108+0000 7f9ea71d3700 1 --2- 172.21.15.172:0/1896784884 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e900035c0 0x7f9e90005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:27.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.108+0000 7f9ea71d3700 1 -- 172.21.15.172:0/1896784884 >> 172.21.15.172:0/1896784884 conn(0x7f9ea00fc750 msgr2=0x7f9ea00feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:26:27.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.108+0000 7f9ea71d3700 1 -- 172.21.15.172:0/1896784884 shutdown_connections 2024-03-28T22:26:27.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.108+0000 7f9ea71d3700 1 -- 172.21.15.172:0/1896784884 wait complete. 2024-03-28T22:26:27.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.109+0000 7f9ea71d3700 1 Processor -- start 2024-03-28T22:26:27.110 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.109+0000 7f9ea71d3700 1 -- start start 2024-03-28T22:26:27.111 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:26 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:26:27.111 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:26 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:26:27.111 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:26 smithi172 ceph-mon[36566]: mon.a calling monitor election 2024-03-28T22:26:27.111 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:26 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:26:27.111 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:26 smithi172 ceph-mon[36566]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:27.111 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:26 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:26:27.111 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:26 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:26:27.111 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:26 smithi172 ceph-mon[36566]: mon.c calling monitor election 2024-03-28T22:26:27.111 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:26 smithi172 ceph-mon[36566]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:27.111 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:26 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:26:27.112 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:26 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:26:27.112 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:26 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:26:27.112 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:26 smithi172 ceph-mon[36566]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T22:26:27.112 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:26 smithi172 ceph-mon[36566]: monmap epoch 2 2024-03-28T22:26:27.112 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:26 smithi172 ceph-mon[36566]: fsid c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:26:27.112 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:26 smithi172 ceph-mon[36566]: last_changed 2024-03-28T22:26:21.805983+0000 2024-03-28T22:26:27.112 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:26 smithi172 ceph-mon[36566]: created 2024-03-28T22:24:15.893393+0000 2024-03-28T22:26:27.112 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:26 smithi172 ceph-mon[36566]: min_mon_release 19 (squid) 2024-03-28T22:26:27.112 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:26 smithi172 ceph-mon[36566]: election_strategy: 1 2024-03-28T22:26:27.112 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:26 smithi172 ceph-mon[36566]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-03-28T22:26:27.112 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:26 smithi172 ceph-mon[36566]: 1: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.c 2024-03-28T22:26:27.112 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:26 smithi172 ceph-mon[36566]: fsmap 2024-03-28T22:26:27.112 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:26 smithi172 ceph-mon[36566]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T22:26:27.112 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:26 smithi172 ceph-mon[36566]: mgrmap e13: x(active, since 58s) 2024-03-28T22:26:27.113 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:26 smithi172 ceph-mon[36566]: overall HEALTH_OK 2024-03-28T22:26:27.113 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:26 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:27.113 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:26 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:27.113 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:26 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:27.113 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:26 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T22:26:27.113 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:26 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:27.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.109+0000 7f9ea71d3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e900035c0 0x7f9ea019d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:26:27.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.109+0000 7f9ea71d3700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ea019db90 0x7f9ea0197720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:26:27.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.109+0000 7f9ea71d3700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9ea0108cd0 con 0x7f9ea019db90 2024-03-28T22:26:27.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.109+0000 7f9ea71d3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9ea0108b50 con 0x7f9e900035c0 2024-03-28T22:26:27.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.109+0000 7f9ea4f6f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e900035c0 0x7f9ea019d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:26:27.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.109+0000 7f9ea4f6f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e900035c0 0x7f9ea019d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:54234/0 (socket says 172.21.15.172:54234) 2024-03-28T22:26:27.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.109+0000 7f9ea4f6f700 1 -- 172.21.15.172:0/3324964788 learned_addr learned my addr 172.21.15.172:0/3324964788 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-28T22:26:27.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.110+0000 7f9ea4f6f700 1 -- 172.21.15.172:0/3324964788 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e900035c0 msgr2=0x7f9ea019d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-28T22:26:27.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.110+0000 7f9ea4f6f700 1 -- 172.21.15.172:0/3324964788 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e900035c0 msgr2=0x7f9ea019d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-28T22:26:27.115 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.110+0000 7f9ea4f6f700 1 --2- 172.21.15.172:0/3324964788 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e900035c0 0x7f9ea019d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T22:26:27.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.110+0000 7f9ea4f6f700 1 --2- 172.21.15.172:0/3324964788 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e900035c0 0x7f9ea019d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:26:27.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.110+0000 7f9e9ffff700 1 --2- 172.21.15.172:0/3324964788 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ea019db90 0x7f9ea0197720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:26:27.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.110+0000 7f9e9ffff700 1 -- 172.21.15.172:0/3324964788 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e900035c0 msgr2=0x7f9ea019d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T22:26:27.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.110+0000 7f9e9ffff700 1 --2- 172.21.15.172:0/3324964788 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e900035c0 0x7f9ea019d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:27.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.110+0000 7f9e9ffff700 1 -- 172.21.15.172:0/3324964788 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ea019e110 con 0x7f9ea019db90 2024-03-28T22:26:27.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.111+0000 7f9e9ffff700 1 --2- 172.21.15.172:0/3324964788 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ea019db90 0x7f9ea0197720 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c00b4d0 tx=0x7f9e8c00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:26:27.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.111+0000 7f9e9dffb700 1 -- 172.21.15.172:0/3324964788 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f9e8c011630 con 0x7f9ea019db90 2024-03-28T22:26:27.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.111+0000 7f9e9dffb700 1 -- 172.21.15.172:0/3324964788 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e8c011cb0 con 0x7f9ea019db90 2024-03-28T22:26:27.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.111+0000 7f9ea71d3700 1 -- 172.21.15.172:0/3324964788 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ea0197cc0 con 0x7f9ea019db90 2024-03-28T22:26:27.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.111+0000 7f9e9dffb700 1 -- 172.21.15.172:0/3324964788 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f9e8c010e60 con 0x7f9ea019db90 2024-03-28T22:26:27.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.112+0000 7f9ea71d3700 1 -- 172.21.15.172:0/3324964788 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ea0198250 con 0x7f9ea019db90 2024-03-28T22:26:27.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.113+0000 7f9ea71d3700 1 -- 172.21.15.172:0/3324964788 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ea0102380 con 0x7f9ea019db90 2024-03-28T22:26:27.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.113+0000 7f9e9dffb700 1 -- 172.21.15.172:0/3324964788 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f9e8c018020 con 0x7f9ea019db90 2024-03-28T22:26:27.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.113+0000 7f9e9dffb700 1 --2- 172.21.15.172:0/3324964788 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9e84038990 0x7f9e8403ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:26:27.116 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.113+0000 7f9e9dffb700 1 -- 172.21.15.172:0/3324964788 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f9e8c04b570 con 0x7f9ea019db90 2024-03-28T22:26:27.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.114+0000 7f9ea4f6f700 1 --2- 172.21.15.172:0/3324964788 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9e84038990 0x7f9e8403ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:26:27.117 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.115+0000 7f9ea4f6f700 1 --2- 172.21.15.172:0/3324964788 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9e84038990 0x7f9e8403ae50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9e9400fa60 tx=0x7f9e94005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:26:27.129 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.129+0000 7f9e9dffb700 1 -- 172.21.15.172:0/3324964788 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9e8c01f020 con 0x7f9ea019db90 2024-03-28T22:26:27.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:26 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:26:27.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:26 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:26:27.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:26 smithi043 ceph-mon[31823]: mon.a calling monitor election 2024-03-28T22:26:27.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:26 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:26:27.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:26 smithi043 ceph-mon[31823]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:27.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:26 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:26:27.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:26 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:26:27.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:26 smithi043 ceph-mon[31823]: mon.c calling monitor election 2024-03-28T22:26:27.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:26 smithi043 ceph-mon[31823]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:27.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:26 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:26:27.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:26 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:26:27.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:26 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:26:27.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:26 smithi043 ceph-mon[31823]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T22:26:27.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:26 smithi043 ceph-mon[31823]: monmap epoch 2 2024-03-28T22:26:27.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:26 smithi043 ceph-mon[31823]: fsid c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:26:27.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:26 smithi043 ceph-mon[31823]: last_changed 2024-03-28T22:26:21.805983+0000 2024-03-28T22:26:27.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:26 smithi043 ceph-mon[31823]: created 2024-03-28T22:24:15.893393+0000 2024-03-28T22:26:27.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:26 smithi043 ceph-mon[31823]: min_mon_release 19 (squid) 2024-03-28T22:26:27.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:26 smithi043 ceph-mon[31823]: election_strategy: 1 2024-03-28T22:26:27.270 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:26 smithi043 ceph-mon[31823]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-03-28T22:26:27.270 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:26 smithi043 ceph-mon[31823]: 1: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.c 2024-03-28T22:26:27.270 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:26 smithi043 ceph-mon[31823]: fsmap 2024-03-28T22:26:27.270 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:26 smithi043 ceph-mon[31823]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T22:26:27.270 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:26 smithi043 ceph-mon[31823]: mgrmap e13: x(active, since 58s) 2024-03-28T22:26:27.270 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:26 smithi043 ceph-mon[31823]: overall HEALTH_OK 2024-03-28T22:26:27.270 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:26 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:27.270 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:26 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:27.270 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:26 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:27.270 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:26 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T22:26:27.270 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:26 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:27.414 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.414+0000 7f9ea71d3700 1 -- 172.21.15.172:0/3324964788 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f9ea01985c0 con 0x7f9ea019db90 2024-03-28T22:26:27.414 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.415+0000 7f9e9dffb700 1 -- 172.21.15.172:0/3324964788 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f9e8c014170 con 0x7f9ea019db90 2024-03-28T22:26:27.415 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-28T22:26:27.415 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":2,"fsid":"c2276428-ed51-11ee-95d0-87774f69a715","modified":"2024-03-28T22:26:21.805983Z","created":"2024-03-28T22:24:15.893393Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:3300","nonce":0},{"type":"v1","addr":"172.21.15.172:6789","nonce":0}]},"addr":"172.21.15.172:6789/0","public_addr":"172.21.15.172:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-28T22:26:27.419 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.419+0000 7f9ea71d3700 1 -- 172.21.15.172:0/3324964788 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9e84038990 msgr2=0x7f9e8403ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:27.419 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.419+0000 7f9ea71d3700 1 --2- 172.21.15.172:0/3324964788 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9e84038990 0x7f9e8403ae50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9e9400fa60 tx=0x7f9e94005960 comp rx=0 tx=0).stop 2024-03-28T22:26:27.419 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.419+0000 7f9ea71d3700 1 -- 172.21.15.172:0/3324964788 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ea019db90 msgr2=0x7f9ea0197720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:27.419 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.419+0000 7f9ea71d3700 1 --2- 172.21.15.172:0/3324964788 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ea019db90 0x7f9ea0197720 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c00b4d0 tx=0x7f9e8c00b890 comp rx=0 tx=0).stop 2024-03-28T22:26:27.419 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.419+0000 7f9ea71d3700 1 -- 172.21.15.172:0/3324964788 shutdown_connections 2024-03-28T22:26:27.419 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.419+0000 7f9ea71d3700 1 --2- 172.21.15.172:0/3324964788 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9e84038990 0x7f9e8403ae50 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:27.419 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.419+0000 7f9ea71d3700 1 --2- 172.21.15.172:0/3324964788 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e900035c0 0x7f9ea019d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:27.419 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.419+0000 7f9ea71d3700 1 --2- 172.21.15.172:0/3324964788 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ea019db90 0x7f9ea0197720 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:27.419 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.419+0000 7f9ea71d3700 1 -- 172.21.15.172:0/3324964788 >> 172.21.15.172:0/3324964788 conn(0x7f9ea00fc750 msgr2=0x7f9ea006ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:26:27.419 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.419+0000 7f9ea71d3700 1 -- 172.21.15.172:0/3324964788 shutdown_connections 2024-03-28T22:26:27.419 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:27.419+0000 7f9ea71d3700 1 -- 172.21.15.172:0/3324964788 wait complete. 2024-03-28T22:26:27.420 INFO:teuthology.orchestra.run.smithi172.stderr:dumped monmap epoch 2 2024-03-28T22:26:27.999 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:27 smithi172 ceph-mon[36566]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:27.999 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:27 smithi172 ceph-mon[36566]: Deploying daemon mon.b on smithi082 2024-03-28T22:26:28.000 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:27 smithi172 ceph-mon[36566]: from='client.? 172.21.15.172:0/3324964788' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T22:26:28.000 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:27 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:26:28.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:27 smithi043 ceph-mon[31823]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:28.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:27 smithi043 ceph-mon[31823]: Deploying daemon mon.b on smithi082 2024-03-28T22:26:28.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:27 smithi043 ceph-mon[31823]: from='client.? 172.21.15.172:0/3324964788' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T22:26:28.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:27 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:26:29.029 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T22:26:29.030 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T22:26:29.267 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:26:28 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:26:28.807+0000 7f1d67a00700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-28T22:26:29.272 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T22:26:30.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:29 smithi172 ceph-mon[36566]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:30.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:29 smithi043 ceph-mon[31823]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:30.598 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:30 smithi082 ceph-mon[36471]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-28T22:26:30.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:30 smithi082 bash[36452]: 94c6712ec6de47c634a32d673f16ec7e69b4a67b01750c504bbc21bc3dd140d1 2024-03-28T22:26:30.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:30 smithi082 systemd[1]: Started Ceph mon.b for c2276428-ed51-11ee-95d0-87774f69a715. 2024-03-28T22:26:35.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:35 smithi172 ceph-mon[36566]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:35.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:35 smithi172 ceph-mon[36566]: mon.c calling monitor election 2024-03-28T22:26:35.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:35 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:26:35.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:35 smithi172 ceph-mon[36566]: mon.a calling monitor election 2024-03-28T22:26:35.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:35 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:26:35.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:35 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:26:35.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:35 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:26:35.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:35 smithi172 ceph-mon[36566]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:35.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:35 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:26:35.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:35 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:26:35.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:35 smithi172 ceph-mon[36566]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:35.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:35 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:26:35.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:35 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:26:35.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:35 smithi172 ceph-mon[36566]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T22:26:35.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:35 smithi172 ceph-mon[36566]: monmap epoch 3 2024-03-28T22:26:35.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:35 smithi172 ceph-mon[36566]: fsid c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:26:35.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:35 smithi172 ceph-mon[36566]: last_changed 2024-03-28T22:26:30.573792+0000 2024-03-28T22:26:35.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:35 smithi172 ceph-mon[36566]: created 2024-03-28T22:24:15.893393+0000 2024-03-28T22:26:35.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:35 smithi172 ceph-mon[36566]: min_mon_release 19 (squid) 2024-03-28T22:26:35.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:35 smithi172 ceph-mon[36566]: election_strategy: 1 2024-03-28T22:26:35.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:35 smithi172 ceph-mon[36566]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-03-28T22:26:35.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:35 smithi172 ceph-mon[36566]: 1: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.c 2024-03-28T22:26:35.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:35 smithi172 ceph-mon[36566]: 2: [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] mon.b 2024-03-28T22:26:35.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:35 smithi172 ceph-mon[36566]: fsmap 2024-03-28T22:26:35.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:35 smithi172 ceph-mon[36566]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T22:26:35.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:35 smithi172 ceph-mon[36566]: mgrmap e13: x(active, since 67s) 2024-03-28T22:26:35.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:35 smithi172 ceph-mon[36566]: overall HEALTH_OK 2024-03-28T22:26:35.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:35 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:35.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:35 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:35.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:35 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:35.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:35 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:36.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:35 smithi043 ceph-mon[31823]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:36.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:35 smithi043 ceph-mon[31823]: mon.c calling monitor election 2024-03-28T22:26:36.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:35 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:26:36.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:35 smithi043 ceph-mon[31823]: mon.a calling monitor election 2024-03-28T22:26:36.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:35 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:26:36.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:35 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:26:36.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:35 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:26:36.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:35 smithi043 ceph-mon[31823]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:36.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:35 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:26:36.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:35 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:26:36.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:35 smithi043 ceph-mon[31823]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:36.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:35 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:26:36.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:35 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:26:36.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:35 smithi043 ceph-mon[31823]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T22:26:36.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:35 smithi043 ceph-mon[31823]: monmap epoch 3 2024-03-28T22:26:36.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:35 smithi043 ceph-mon[31823]: fsid c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:26:36.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:35 smithi043 ceph-mon[31823]: last_changed 2024-03-28T22:26:30.573792+0000 2024-03-28T22:26:36.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:35 smithi043 ceph-mon[31823]: created 2024-03-28T22:24:15.893393+0000 2024-03-28T22:26:36.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:35 smithi043 ceph-mon[31823]: min_mon_release 19 (squid) 2024-03-28T22:26:36.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:35 smithi043 ceph-mon[31823]: election_strategy: 1 2024-03-28T22:26:36.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:35 smithi043 ceph-mon[31823]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-03-28T22:26:36.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:35 smithi043 ceph-mon[31823]: 1: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.c 2024-03-28T22:26:36.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:35 smithi043 ceph-mon[31823]: 2: [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] mon.b 2024-03-28T22:26:36.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:35 smithi043 ceph-mon[31823]: fsmap 2024-03-28T22:26:36.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:35 smithi043 ceph-mon[31823]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T22:26:36.020 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:35 smithi043 ceph-mon[31823]: mgrmap e13: x(active, since 67s) 2024-03-28T22:26:36.020 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:35 smithi043 ceph-mon[31823]: overall HEALTH_OK 2024-03-28T22:26:36.020 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:35 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:36.020 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:35 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:36.020 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:35 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:36.020 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:35 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:36.695 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.692+0000 7f03cacf7700 1 -- 172.21.15.172:0/3331971038 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03ac0062a0 msgr2=0x7f03ac008710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:36.695 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.692+0000 7f03cacf7700 1 --2- 172.21.15.172:0/3331971038 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03ac0062a0 0x7f03ac008710 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f03c4069360 tx=0x7f03c00095d0 comp rx=0 tx=0).stop 2024-03-28T22:26:36.695 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.692+0000 7f03cacf7700 1 -- 172.21.15.172:0/3331971038 shutdown_connections 2024-03-28T22:26:36.695 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.692+0000 7f03cacf7700 1 --2- 172.21.15.172:0/3331971038 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03ac0062a0 0x7f03ac008710 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:36.695 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.692+0000 7f03cacf7700 1 -- 172.21.15.172:0/3331971038 >> 172.21.15.172:0/3331971038 conn(0x7f03c40fc350 msgr2=0x7f03c40fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:26:36.695 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.692+0000 7f03cacf7700 1 -- 172.21.15.172:0/3331971038 shutdown_connections 2024-03-28T22:26:36.695 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.692+0000 7f03cacf7700 1 -- 172.21.15.172:0/3331971038 wait complete. 2024-03-28T22:26:36.695 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.693+0000 7f03cacf7700 1 Processor -- start 2024-03-28T22:26:36.695 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.693+0000 7f03cacf7700 1 -- start start 2024-03-28T22:26:36.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.693+0000 7f03cacf7700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03ac0062a0 0x7f03c41014c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:26:36.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.693+0000 7f03cacf7700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03c4101a00 0x7f03c4101e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:26:36.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.693+0000 7f03cacf7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03c41a3c10 0x7f03c41a6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:26:36.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.693+0000 7f03cacf7700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f03c4109d00 con 0x7f03c4101a00 2024-03-28T22:26:36.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.693+0000 7f03cacf7700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f03c4109b80 con 0x7f03ac0062a0 2024-03-28T22:26:36.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.693+0000 7f03cacf7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f03c4109e80 con 0x7f03c41a3c10 2024-03-28T22:26:36.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.694+0000 7f03c9294700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03c41a3c10 0x7f03c41a6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:26:36.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.694+0000 7f03c9294700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03c41a3c10 0x7f03c41a6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:58586/0 (socket says 172.21.15.172:58586) 2024-03-28T22:26:36.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.694+0000 7f03c9294700 1 -- 172.21.15.172:0/908969800 learned_addr learned my addr 172.21.15.172:0/908969800 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-28T22:26:36.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.694+0000 7f03c9294700 1 -- 172.21.15.172:0/908969800 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03c41a3c10 msgr2=0x7f03c41a6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-28T22:26:36.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.694+0000 7f03c9294700 1 -- 172.21.15.172:0/908969800 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03c41a3c10 msgr2=0x7f03c41a6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-28T22:26:36.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.694+0000 7f03c9294700 1 --2- 172.21.15.172:0/908969800 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03c41a3c10 0x7f03c41a6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T22:26:36.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.694+0000 7f03c9294700 1 --2- 172.21.15.172:0/908969800 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03c41a3c10 0x7f03c41a6000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:26:36.696 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.694+0000 7f03c8a93700 1 --2- 172.21.15.172:0/908969800 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03ac0062a0 0x7f03c41014c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:26:36.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.694+0000 7f03bbfff700 1 --2- 172.21.15.172:0/908969800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03c4101a00 0x7f03c4101e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:26:36.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.694+0000 7f03c8a93700 1 -- 172.21.15.172:0/908969800 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03ac0062a0 msgr2=0x7f03c41014c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-28T22:26:36.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.694+0000 7f03c8a93700 1 -- 172.21.15.172:0/908969800 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03ac0062a0 msgr2=0x7f03c41014c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-28T22:26:36.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.694+0000 7f03c8a93700 1 --2- 172.21.15.172:0/908969800 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03ac0062a0 0x7f03c41014c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T22:26:36.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.694+0000 7f03c8a93700 1 --2- 172.21.15.172:0/908969800 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03ac0062a0 0x7f03c41014c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:26:36.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.694+0000 7f03bbfff700 1 -- 172.21.15.172:0/908969800 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03c41a3c10 msgr2=0x7f03c41a6000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T22:26:36.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.694+0000 7f03bbfff700 1 --2- 172.21.15.172:0/908969800 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03c41a3c10 0x7f03c41a6000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:36.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.694+0000 7f03bbfff700 1 -- 172.21.15.172:0/908969800 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03ac0062a0 msgr2=0x7f03c41014c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T22:26:36.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.694+0000 7f03bbfff700 1 --2- 172.21.15.172:0/908969800 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03ac0062a0 0x7f03c41014c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:36.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.694+0000 7f03bbfff700 1 -- 172.21.15.172:0/908969800 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03c41a6730 con 0x7f03c4101a00 2024-03-28T22:26:36.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.695+0000 7f03bbfff700 1 --2- 172.21.15.172:0/908969800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03c4101a00 0x7f03c4101e80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f03b400b550 tx=0x7f03b400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:26:36.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.695+0000 7f03b9ffb700 1 -- 172.21.15.172:0/908969800 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b4004230 con 0x7f03c4101a00 2024-03-28T22:26:36.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.695+0000 7f03cacf7700 1 -- 172.21.15.172:0/908969800 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03c41a69c0 con 0x7f03c4101a00 2024-03-28T22:26:36.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.695+0000 7f03b9ffb700 1 -- 172.21.15.172:0/908969800 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03b40043d0 con 0x7f03c4101a00 2024-03-28T22:26:36.697 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.695+0000 7f03b9ffb700 1 -- 172.21.15.172:0/908969800 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b4011b90 con 0x7f03c4101a00 2024-03-28T22:26:36.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.695+0000 7f03cacf7700 1 -- 172.21.15.172:0/908969800 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03c41a6f50 con 0x7f03c4101a00 2024-03-28T22:26:36.698 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.696+0000 7f03b9ffb700 1 -- 172.21.15.172:0/908969800 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f03b4003c10 con 0x7f03c4101a00 2024-03-28T22:26:36.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.697+0000 7f03b9ffb700 1 --2- 172.21.15.172:0/908969800 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f03b00390d0 0x7f03b003b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:26:36.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.697+0000 7f03cacf7700 1 -- 172.21.15.172:0/908969800 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03c41029f0 con 0x7f03c4101a00 2024-03-28T22:26:36.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.697+0000 7f03b9ffb700 1 -- 172.21.15.172:0/908969800 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f03b404b410 con 0x7f03c4101a00 2024-03-28T22:26:36.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.697+0000 7f03c8a93700 1 --2- 172.21.15.172:0/908969800 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f03b00390d0 0x7f03b003b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:26:36.699 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.698+0000 7f03c8a93700 1 --2- 172.21.15.172:0/908969800 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f03b00390d0 0x7f03b003b590 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f03c00098a0 tx=0x7f03c0009990 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:26:36.707 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:36.707+0000 7f03b9ffb700 1 -- 172.21.15.172:0/908969800 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f03b401f070 con 0x7f03c4101a00 2024-03-28T22:26:36.968 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:36 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:26:36.968 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:36 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:26:37.008 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:37.009+0000 7f03cacf7700 1 -- 172.21.15.172:0/908969800 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f03c41a71e0 con 0x7f03c4101a00 2024-03-28T22:26:37.009 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:37.010+0000 7f03b9ffb700 1 -- 172.21.15.172:0/908969800 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1279 (secure 0 0 0) 0x7f03b400f760 con 0x7f03c4101a00 2024-03-28T22:26:37.009 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-28T22:26:37.009 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":3,"fsid":"c2276428-ed51-11ee-95d0-87774f69a715","modified":"2024-03-28T22:26:30.573792Z","created":"2024-03-28T22:24:15.893393Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:3300","nonce":0},{"type":"v1","addr":"172.21.15.172:6789","nonce":0}]},"addr":"172.21.15.172:6789/0","public_addr":"172.21.15.172:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:3300","nonce":0},{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-28T22:26:37.011 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:37.012+0000 7f03cacf7700 1 -- 172.21.15.172:0/908969800 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f03b00390d0 msgr2=0x7f03b003b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:37.011 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:37.012+0000 7f03cacf7700 1 --2- 172.21.15.172:0/908969800 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f03b00390d0 0x7f03b003b590 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f03c00098a0 tx=0x7f03c0009990 comp rx=0 tx=0).stop 2024-03-28T22:26:37.012 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:37.012+0000 7f03cacf7700 1 -- 172.21.15.172:0/908969800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03c4101a00 msgr2=0x7f03c4101e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:37.012 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:37.012+0000 7f03cacf7700 1 --2- 172.21.15.172:0/908969800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03c4101a00 0x7f03c4101e80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f03b400b550 tx=0x7f03b400b910 comp rx=0 tx=0).stop 2024-03-28T22:26:37.012 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:37.012+0000 7f03cacf7700 1 -- 172.21.15.172:0/908969800 shutdown_connections 2024-03-28T22:26:37.012 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:37.012+0000 7f03cacf7700 1 --2- 172.21.15.172:0/908969800 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f03b00390d0 0x7f03b003b590 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:37.012 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:37.012+0000 7f03cacf7700 1 --2- 172.21.15.172:0/908969800 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03c41a3c10 0x7f03c41a6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:37.012 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:37.012+0000 7f03cacf7700 1 --2- 172.21.15.172:0/908969800 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03ac0062a0 0x7f03c41014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:37.012 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:37.012+0000 7f03cacf7700 1 --2- 172.21.15.172:0/908969800 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03c4101a00 0x7f03c4101e80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:37.012 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:37.012+0000 7f03cacf7700 1 -- 172.21.15.172:0/908969800 >> 172.21.15.172:0/908969800 conn(0x7f03c40fc350 msgr2=0x7f03c4105150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:26:37.012 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:37.012+0000 7f03cacf7700 1 -- 172.21.15.172:0/908969800 shutdown_connections 2024-03-28T22:26:37.012 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:37.012+0000 7f03cacf7700 1 -- 172.21.15.172:0/908969800 wait complete. 2024-03-28T22:26:37.012 INFO:teuthology.orchestra.run.smithi172.stderr:dumped monmap epoch 3 2024-03-28T22:26:37.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:36 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:26:37.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:36 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:26:37.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:37.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: mon.c calling monitor election 2024-03-28T22:26:37.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:26:37.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: mon.a calling monitor election 2024-03-28T22:26:37.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:26:37.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:26:37.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:26:37.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:37.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:26:37.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:26:37.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:37.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:26:37.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:26:37.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T22:26:37.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: monmap epoch 3 2024-03-28T22:26:37.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: fsid c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:26:37.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: last_changed 2024-03-28T22:26:30.573792+0000 2024-03-28T22:26:37.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: created 2024-03-28T22:24:15.893393+0000 2024-03-28T22:26:37.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: min_mon_release 19 (squid) 2024-03-28T22:26:37.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: election_strategy: 1 2024-03-28T22:26:37.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-03-28T22:26:37.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: 1: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.c 2024-03-28T22:26:37.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: 2: [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] mon.b 2024-03-28T22:26:37.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: fsmap 2024-03-28T22:26:37.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T22:26:37.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: mgrmap e13: x(active, since 67s) 2024-03-28T22:26:37.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: overall HEALTH_OK 2024-03-28T22:26:37.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:37.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:37.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:37.953 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:37.953 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:26:37.953 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:37 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:26:38.008 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-28T22:26:38.009 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph config generate-minimal-conf 2024-03-28T22:26:38.428 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:26:38.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:38 smithi043 ceph-mon[31823]: mon.b calling monitor election 2024-03-28T22:26:38.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:38 smithi043 ceph-mon[31823]: Updating smithi043:/etc/ceph/ceph.conf 2024-03-28T22:26:38.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:38 smithi043 ceph-mon[31823]: Updating smithi082:/etc/ceph/ceph.conf 2024-03-28T22:26:38.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:38 smithi043 ceph-mon[31823]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-28T22:26:38.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:38 smithi043 ceph-mon[31823]: Updating smithi082:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:26:38.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:38 smithi043 ceph-mon[31823]: Updating smithi172:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:26:38.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:38 smithi043 ceph-mon[31823]: mon.b calling monitor election 2024-03-28T22:26:38.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:38 smithi043 ceph-mon[31823]: mon.c calling monitor election 2024-03-28T22:26:38.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:38 smithi043 ceph-mon[31823]: mon.a calling monitor election 2024-03-28T22:26:38.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:38 smithi043 ceph-mon[31823]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T22:26:38.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:38 smithi043 ceph-mon[31823]: monmap epoch 3 2024-03-28T22:26:38.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:38 smithi043 ceph-mon[31823]: fsid c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:26:38.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:38 smithi043 ceph-mon[31823]: last_changed 2024-03-28T22:26:30.573792+0000 2024-03-28T22:26:38.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:38 smithi043 ceph-mon[31823]: created 2024-03-28T22:24:15.893393+0000 2024-03-28T22:26:38.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:38 smithi043 ceph-mon[31823]: min_mon_release 19 (squid) 2024-03-28T22:26:38.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:38 smithi043 ceph-mon[31823]: election_strategy: 1 2024-03-28T22:26:38.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:38 smithi043 ceph-mon[31823]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-03-28T22:26:38.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:38 smithi043 ceph-mon[31823]: 1: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.c 2024-03-28T22:26:38.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:38 smithi043 ceph-mon[31823]: 2: [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] mon.b 2024-03-28T22:26:38.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:38 smithi043 ceph-mon[31823]: fsmap 2024-03-28T22:26:38.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:38 smithi043 ceph-mon[31823]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T22:26:38.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:38 smithi043 ceph-mon[31823]: mgrmap e13: x(active, since 69s) 2024-03-28T22:26:38.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:38 smithi043 ceph-mon[31823]: overall HEALTH_OK 2024-03-28T22:26:38.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:38 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:38.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:38 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:38.519 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:38 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:38.519 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:38 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:38.519 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:38 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:38.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:38 smithi172 ceph-mon[36566]: mon.b calling monitor election 2024-03-28T22:26:38.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:38 smithi172 ceph-mon[36566]: Updating smithi043:/etc/ceph/ceph.conf 2024-03-28T22:26:38.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:38 smithi172 ceph-mon[36566]: Updating smithi082:/etc/ceph/ceph.conf 2024-03-28T22:26:38.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:38 smithi172 ceph-mon[36566]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-28T22:26:38.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:38 smithi172 ceph-mon[36566]: Updating smithi082:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:26:38.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:38 smithi172 ceph-mon[36566]: Updating smithi172:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:26:38.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:38 smithi172 ceph-mon[36566]: mon.b calling monitor election 2024-03-28T22:26:38.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:38 smithi172 ceph-mon[36566]: mon.c calling monitor election 2024-03-28T22:26:38.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:38 smithi172 ceph-mon[36566]: mon.a calling monitor election 2024-03-28T22:26:38.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:38 smithi172 ceph-mon[36566]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T22:26:38.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:38 smithi172 ceph-mon[36566]: monmap epoch 3 2024-03-28T22:26:38.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:38 smithi172 ceph-mon[36566]: fsid c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:26:38.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:38 smithi172 ceph-mon[36566]: last_changed 2024-03-28T22:26:30.573792+0000 2024-03-28T22:26:38.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:38 smithi172 ceph-mon[36566]: created 2024-03-28T22:24:15.893393+0000 2024-03-28T22:26:38.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:38 smithi172 ceph-mon[36566]: min_mon_release 19 (squid) 2024-03-28T22:26:38.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:38 smithi172 ceph-mon[36566]: election_strategy: 1 2024-03-28T22:26:38.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:38 smithi172 ceph-mon[36566]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-03-28T22:26:38.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:38 smithi172 ceph-mon[36566]: 1: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.c 2024-03-28T22:26:38.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:38 smithi172 ceph-mon[36566]: 2: [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] mon.b 2024-03-28T22:26:38.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:38 smithi172 ceph-mon[36566]: fsmap 2024-03-28T22:26:38.649 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:38 smithi172 ceph-mon[36566]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T22:26:38.649 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:38 smithi172 ceph-mon[36566]: mgrmap e13: x(active, since 69s) 2024-03-28T22:26:38.649 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:38 smithi172 ceph-mon[36566]: overall HEALTH_OK 2024-03-28T22:26:38.649 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:38 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:38.649 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:38 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:38.649 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:38 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:38.649 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:38 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:38.649 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:38 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:38.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:38 smithi082 ceph-mon[36471]: mon.b calling monitor election 2024-03-28T22:26:38.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:38 smithi082 ceph-mon[36471]: Updating smithi043:/etc/ceph/ceph.conf 2024-03-28T22:26:38.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:38 smithi082 ceph-mon[36471]: Updating smithi082:/etc/ceph/ceph.conf 2024-03-28T22:26:38.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:38 smithi082 ceph-mon[36471]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-28T22:26:38.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:38 smithi082 ceph-mon[36471]: Updating smithi082:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:26:38.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:38 smithi082 ceph-mon[36471]: Updating smithi172:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:26:38.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:38 smithi082 ceph-mon[36471]: mon.b calling monitor election 2024-03-28T22:26:38.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:38 smithi082 ceph-mon[36471]: mon.c calling monitor election 2024-03-28T22:26:38.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:38 smithi082 ceph-mon[36471]: mon.a calling monitor election 2024-03-28T22:26:38.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:38 smithi082 ceph-mon[36471]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T22:26:38.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:38 smithi082 ceph-mon[36471]: monmap epoch 3 2024-03-28T22:26:38.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:38 smithi082 ceph-mon[36471]: fsid c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:26:38.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:38 smithi082 ceph-mon[36471]: last_changed 2024-03-28T22:26:30.573792+0000 2024-03-28T22:26:38.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:38 smithi082 ceph-mon[36471]: created 2024-03-28T22:24:15.893393+0000 2024-03-28T22:26:38.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:38 smithi082 ceph-mon[36471]: min_mon_release 19 (squid) 2024-03-28T22:26:38.702 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:38 smithi082 ceph-mon[36471]: election_strategy: 1 2024-03-28T22:26:38.702 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:38 smithi082 ceph-mon[36471]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-03-28T22:26:38.702 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:38 smithi082 ceph-mon[36471]: 1: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.c 2024-03-28T22:26:38.702 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:38 smithi082 ceph-mon[36471]: 2: [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] mon.b 2024-03-28T22:26:38.702 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:38 smithi082 ceph-mon[36471]: fsmap 2024-03-28T22:26:38.702 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:38 smithi082 ceph-mon[36471]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T22:26:38.702 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:38 smithi082 ceph-mon[36471]: mgrmap e13: x(active, since 69s) 2024-03-28T22:26:38.702 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:38 smithi082 ceph-mon[36471]: overall HEALTH_OK 2024-03-28T22:26:38.702 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:38 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:38.702 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:38 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:38.702 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:38 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:38.702 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:38 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:38.702 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:38 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:39.514 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:39 smithi043 ceph-mon[31823]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:39.514 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:39 smithi043 ceph-mon[31823]: Updating smithi043:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:26:39.514 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:39 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:26:39.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:39 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:39.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:39 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:39.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:39 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:39.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:39 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:39.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:39 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:39.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:39 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:39.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:39 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:39.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:39 smithi043 ceph-mon[31823]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T22:26:39.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:39 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T22:26:39.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:39 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T22:26:39.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:39 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:39.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:39 smithi043 ceph-mon[31823]: Reconfiguring daemon mon.a on smithi043 2024-03-28T22:26:39.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:39 smithi172 ceph-mon[36566]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:39.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:39 smithi172 ceph-mon[36566]: Updating smithi043:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:26:39.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:39 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:26:39.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:39 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:39.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:39 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:39.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:39 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:39.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:39 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:39.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:39 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:39.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:39 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:39.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:39 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:39.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:39 smithi172 ceph-mon[36566]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T22:26:39.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:39 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T22:26:39.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:39 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T22:26:39.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:39 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:39.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:39 smithi172 ceph-mon[36566]: Reconfiguring daemon mon.a on smithi043 2024-03-28T22:26:39.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:39 smithi082 ceph-mon[36471]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:39.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:39 smithi082 ceph-mon[36471]: Updating smithi043:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:26:39.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:39 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:26:39.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:39 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:39.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:39 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:39.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:39 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:39.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:39 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:39.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:39 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:39.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:39 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:39.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:39 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:39.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:39 smithi082 ceph-mon[36471]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T22:26:39.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:39 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T22:26:39.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:39 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T22:26:39.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:39 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:39.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:39 smithi082 ceph-mon[36471]: Reconfiguring daemon mon.a on smithi043 2024-03-28T22:26:39.767 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:26:39 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:26:39.575+0000 7f1d67a00700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-28T22:26:41.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.715+0000 7fe06be00700 1 -- 172.21.15.43:0/3012870005 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe064074670 msgr2=0x7fe064074a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:41.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.715+0000 7fe06be00700 1 --2- 172.21.15.43:0/3012870005 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe064074670 0x7fe064074a50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe058009910 tx=0x7fe058009c20 comp rx=0 tx=0).stop 2024-03-28T22:26:41.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.715+0000 7fe06be00700 1 -- 172.21.15.43:0/3012870005 shutdown_connections 2024-03-28T22:26:41.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.715+0000 7fe06be00700 1 --2- 172.21.15.43:0/3012870005 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe064074670 0x7fe064074a50 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:41.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.715+0000 7fe06be00700 1 -- 172.21.15.43:0/3012870005 >> 172.21.15.43:0/3012870005 conn(0x7fe0640fd770 msgr2=0x7fe0640ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:26:41.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.715+0000 7fe06be00700 1 -- 172.21.15.43:0/3012870005 shutdown_connections 2024-03-28T22:26:41.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.715+0000 7fe06be00700 1 -- 172.21.15.43:0/3012870005 wait complete. 2024-03-28T22:26:41.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.716+0000 7fe06be00700 1 Processor -- start 2024-03-28T22:26:41.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.716+0000 7fe06be00700 1 -- start start 2024-03-28T22:26:41.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.717+0000 7fe06be00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe064074670 0x7fe06419f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:26:41.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.717+0000 7fe06be00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe06419fe70 0x7fe064199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:26:41.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.717+0000 7fe06be00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe06419a030 0x7fe06419a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:26:41.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.717+0000 7fe06be00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe06410e510 con 0x7fe06419a030 2024-03-28T22:26:41.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.717+0000 7fe06be00700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe06410e390 con 0x7fe064074670 2024-03-28T22:26:41.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.717+0000 7fe06be00700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe06410e690 con 0x7fe06419fe70 2024-03-28T22:26:41.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.717+0000 7fe069e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe06419a030 0x7fe06419a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:26:41.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.717+0000 7fe069e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe06419a030 0x7fe06419a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53918/0 (socket says 172.21.15.43:53918) 2024-03-28T22:26:41.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.717+0000 7fe069e00700 1 -- 172.21.15.43:0/3276408375 learned_addr learned my addr 172.21.15.43:0/3276408375 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:26:41.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.717+0000 7fe069400700 1 --2- 172.21.15.43:0/3276408375 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe064074670 0x7fe06419f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:26:41.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.717+0000 7fe068a00700 1 --2- 172.21.15.43:0/3276408375 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe06419fe70 0x7fe064199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:26:41.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.718+0000 7fe069e00700 1 -- 172.21.15.43:0/3276408375 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe06419fe70 msgr2=0x7fe064199af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:41.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.718+0000 7fe069e00700 1 --2- 172.21.15.43:0/3276408375 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe06419fe70 0x7fe064199af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:41.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.718+0000 7fe069e00700 1 -- 172.21.15.43:0/3276408375 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe064074670 msgr2=0x7fe06419f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:41.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.718+0000 7fe069e00700 1 --2- 172.21.15.43:0/3276408375 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe064074670 0x7fe06419f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:41.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.718+0000 7fe069e00700 1 -- 172.21.15.43:0/3276408375 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0641a6800 con 0x7fe06419a030 2024-03-28T22:26:41.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.718+0000 7fe068a00700 1 --2- 172.21.15.43:0/3276408375 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe06419fe70 0x7fe064199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:26:41.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.718+0000 7fe069e00700 1 --2- 172.21.15.43:0/3276408375 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe06419a030 0x7fe06419a4e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe05c00ba50 tx=0x7fe05c00bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:26:41.397 INFO:teuthology.orchestra.run.smithi043.stdout:# minimal ceph.conf for c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:26:41.397 INFO:teuthology.orchestra.run.smithi043.stdout:[global] 2024-03-28T22:26:41.397 INFO:teuthology.orchestra.run.smithi043.stdout: fsid = c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:26:41.397 INFO:teuthology.orchestra.run.smithi043.stdout: mon_host = [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] 2024-03-28T22:26:41.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.718+0000 7fe062000700 1 -- 172.21.15.43:0/3276408375 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe05c019070 con 0x7fe06419a030 2024-03-28T22:26:41.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.718+0000 7fe062000700 1 -- 172.21.15.43:0/3276408375 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe05c014440 con 0x7fe06419a030 2024-03-28T22:26:41.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.718+0000 7fe062000700 1 -- 172.21.15.43:0/3276408375 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe05c0075b0 con 0x7fe06419a030 2024-03-28T22:26:41.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.718+0000 7fe06be00700 1 -- 172.21.15.43:0/3276408375 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0641a6a30 con 0x7fe06419a030 2024-03-28T22:26:41.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.718+0000 7fe06be00700 1 -- 172.21.15.43:0/3276408375 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0641a6f70 con 0x7fe06419a030 2024-03-28T22:26:41.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.720+0000 7fe062000700 1 -- 172.21.15.43:0/3276408375 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7fe05c01c070 con 0x7fe06419a030 2024-03-28T22:26:41.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.720+0000 7fe06be00700 1 -- 172.21.15.43:0/3276408375 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe064063870 con 0x7fe06419a030 2024-03-28T22:26:41.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.720+0000 7fe062000700 1 --2- 172.21.15.43:0/3276408375 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe04c0390d0 0x7fe04c03b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:26:41.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.720+0000 7fe062000700 1 -- 172.21.15.43:0/3276408375 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe05c04b100 con 0x7fe06419a030 2024-03-28T22:26:41.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.721+0000 7fe069400700 1 --2- 172.21.15.43:0/3276408375 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe04c0390d0 0x7fe04c03b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:26:41.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.721+0000 7fe069400700 1 --2- 172.21.15.43:0/3276408375 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe04c0390d0 0x7fe04c03b590 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe058006010 tx=0x7fe0580058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:26:41.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.726+0000 7fe062000700 1 -- 172.21.15.43:0/3276408375 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe05c014980 con 0x7fe06419a030 2024-03-28T22:26:41.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.958+0000 7fe06be00700 1 -- 172.21.15.43:0/3276408375 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fe0641a7200 con 0x7fe06419a030 2024-03-28T22:26:41.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.959+0000 7fe062000700 1 -- 172.21.15.43:0/3276408375 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7fe05c00c9b0 con 0x7fe06419a030 2024-03-28T22:26:41.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.962+0000 7fe06be00700 1 -- 172.21.15.43:0/3276408375 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe04c0390d0 msgr2=0x7fe04c03b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:41.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.962+0000 7fe06be00700 1 --2- 172.21.15.43:0/3276408375 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe04c0390d0 0x7fe04c03b590 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe058006010 tx=0x7fe0580058e0 comp rx=0 tx=0).stop 2024-03-28T22:26:41.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.962+0000 7fe06be00700 1 -- 172.21.15.43:0/3276408375 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe06419a030 msgr2=0x7fe06419a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:41.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.962+0000 7fe06be00700 1 --2- 172.21.15.43:0/3276408375 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe06419a030 0x7fe06419a4e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe05c00ba50 tx=0x7fe05c00bd60 comp rx=0 tx=0).stop 2024-03-28T22:26:41.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.962+0000 7fe06be00700 1 -- 172.21.15.43:0/3276408375 shutdown_connections 2024-03-28T22:26:41.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.962+0000 7fe06be00700 1 --2- 172.21.15.43:0/3276408375 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe04c0390d0 0x7fe04c03b590 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:41.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.962+0000 7fe06be00700 1 --2- 172.21.15.43:0/3276408375 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe06419a030 0x7fe06419a4e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:41.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.962+0000 7fe06be00700 1 --2- 172.21.15.43:0/3276408375 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe064074670 0x7fe06419f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:41.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.962+0000 7fe06be00700 1 --2- 172.21.15.43:0/3276408375 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe06419fe70 0x7fe064199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:41.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.963+0000 7fe06be00700 1 -- 172.21.15.43:0/3276408375 >> 172.21.15.43:0/3276408375 conn(0x7fe0640fd770 msgr2=0x7fe064106040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:26:41.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.963+0000 7fe06be00700 1 -- 172.21.15.43:0/3276408375 shutdown_connections 2024-03-28T22:26:41.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:40.963+0000 7fe06be00700 1 -- 172.21.15.43:0/3276408375 wait complete. 2024-03-28T22:26:41.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:41 smithi043 ceph-mon[31823]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:41.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:41 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3276408375' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:41.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:41 smithi172 ceph-mon[36566]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:41.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:41 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3276408375' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:41.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:41 smithi082 ceph-mon[36471]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:41.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:41 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3276408375' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:42.148 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-28T22:26:42.149 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-28T22:26:42.149 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T22:26:42.246 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-28T22:26:42.246 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:26:42.339 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-28T22:26:42.339 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T22:26:42.368 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-28T22:26:42.368 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:26:42.435 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-28T22:26:42.435 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T22:26:42.470 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-28T22:26:42.470 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:26:42.541 INFO:tasks.cephadm:Adding mgr.x on smithi043 2024-03-28T22:26:42.541 INFO:tasks.cephadm:Adding mgr.y on smithi082 2024-03-28T22:26:42.541 INFO:tasks.cephadm:Adding mgr.z on smithi172 2024-03-28T22:26:42.541 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph orch apply mgr '3;smithi043=x;smithi082=y;smithi172=z' 2024-03-28T22:26:42.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:42 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:42.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:42 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:42.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:42 smithi172 ceph-mon[36566]: Reconfiguring mon.b (monmap changed)... 2024-03-28T22:26:42.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:42 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T22:26:42.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:42 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T22:26:42.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:42 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:42.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:42 smithi172 ceph-mon[36566]: Reconfiguring daemon mon.b on smithi082 2024-03-28T22:26:42.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:42 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:42.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:42 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:42.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:42 smithi082 ceph-mon[36471]: Reconfiguring mon.b (monmap changed)... 2024-03-28T22:26:42.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:42 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T22:26:42.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:42 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T22:26:42.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:42 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:42.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:42 smithi082 ceph-mon[36471]: Reconfiguring daemon mon.b on smithi082 2024-03-28T22:26:42.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:42 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:42.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:42 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:42.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:42 smithi043 ceph-mon[31823]: Reconfiguring mon.b (monmap changed)... 2024-03-28T22:26:42.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:42 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T22:26:42.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:42 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T22:26:42.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:42 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:42.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:42 smithi043 ceph-mon[31823]: Reconfiguring daemon mon.b on smithi082 2024-03-28T22:26:42.818 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T22:26:44.041 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:43 smithi172 ceph-mon[36566]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:44.041 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:43 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:44.041 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:43 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:44.041 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:43 smithi172 ceph-mon[36566]: Reconfiguring mon.c (monmap changed)... 2024-03-28T22:26:44.041 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:43 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T22:26:44.041 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:43 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T22:26:44.042 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:43 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:44.042 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:43 smithi172 ceph-mon[36566]: Reconfiguring daemon mon.c on smithi172 2024-03-28T22:26:44.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:43 smithi082 ceph-mon[36471]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:44.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:43 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:44.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:43 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:44.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:43 smithi082 ceph-mon[36471]: Reconfiguring mon.c (monmap changed)... 2024-03-28T22:26:44.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:43 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T22:26:44.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:43 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T22:26:44.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:43 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:44.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:43 smithi082 ceph-mon[36471]: Reconfiguring daemon mon.c on smithi172 2024-03-28T22:26:44.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:43 smithi043 ceph-mon[31823]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:44.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:43 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:44.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:43 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:44.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:43 smithi043 ceph-mon[31823]: Reconfiguring mon.c (monmap changed)... 2024-03-28T22:26:44.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:43 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T22:26:44.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:43 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T22:26:44.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:43 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:44.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:43 smithi043 ceph-mon[31823]: Reconfiguring daemon mon.c on smithi172 2024-03-28T22:26:44.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.267+0000 7f3709f01700 1 -- 172.21.15.172:0/1362570389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3704100ca0 msgr2=0x7f37041010c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:44.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.267+0000 7f3709f01700 1 --2- 172.21.15.172:0/1362570389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3704100ca0 0x7f37041010c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f36f4009910 tx=0x7f36f4009c20 comp rx=0 tx=0).stop 2024-03-28T22:26:44.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.267+0000 7f3709f01700 1 -- 172.21.15.172:0/1362570389 shutdown_connections 2024-03-28T22:26:44.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.267+0000 7f3709f01700 1 --2- 172.21.15.172:0/1362570389 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3704100ca0 0x7f37041010c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:44.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.267+0000 7f3709f01700 1 -- 172.21.15.172:0/1362570389 >> 172.21.15.172:0/1362570389 conn(0x7f37040fc220 msgr2=0x7f37040fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:26:44.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.267+0000 7f3709f01700 1 -- 172.21.15.172:0/1362570389 shutdown_connections 2024-03-28T22:26:44.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.267+0000 7f3709f01700 1 -- 172.21.15.172:0/1362570389 wait complete. 2024-03-28T22:26:44.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.268+0000 7f3709f01700 1 Processor -- start 2024-03-28T22:26:44.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.268+0000 7f3709f01700 1 -- start start 2024-03-28T22:26:44.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.268+0000 7f3709f01700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3704100ca0 0x7f370406afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:26:44.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.268+0000 7f3709f01700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f370419a2a0 0x7f370419c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:26:44.270 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.269+0000 7f3709f01700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f370419cc70 0x7f370419f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:26:44.271 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.269+0000 7f3709f01700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3704106d30 con 0x7f370419cc70 2024-03-28T22:26:44.271 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.269+0000 7f3709f01700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f3704106bb0 con 0x7f3704100ca0 2024-03-28T22:26:44.271 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.269+0000 7f3709f01700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f3704106eb0 con 0x7f370419a2a0 2024-03-28T22:26:44.271 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.269+0000 7f3702ffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f370419a2a0 0x7f370419c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:26:44.271 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.269+0000 7f3702ffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f370419a2a0 0x7f370419c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:60292/0 (socket says 172.21.15.172:60292) 2024-03-28T22:26:44.271 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.269+0000 7f3702ffd700 1 -- 172.21.15.172:0/2290753361 learned_addr learned my addr 172.21.15.172:0/2290753361 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-28T22:26:44.271 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.269+0000 7f37037fe700 1 --2- 172.21.15.172:0/2290753361 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3704100ca0 0x7f370406afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:26:44.271 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.269+0000 7f3702ffd700 1 -- 172.21.15.172:0/2290753361 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3704100ca0 msgr2=0x7f370406afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:44.271 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.269+0000 7f3702ffd700 1 --2- 172.21.15.172:0/2290753361 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3704100ca0 0x7f370406afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:44.271 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.269+0000 7f3702ffd700 1 -- 172.21.15.172:0/2290753361 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f370419cc70 msgr2=0x7f370419f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:44.271 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.269+0000 7f3702ffd700 1 --2- 172.21.15.172:0/2290753361 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f370419cc70 0x7f370419f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:44.271 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.269+0000 7f3702ffd700 1 -- 172.21.15.172:0/2290753361 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f370419f9d0 con 0x7f370419a2a0 2024-03-28T22:26:44.271 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.269+0000 7f3702ffd700 1 --2- 172.21.15.172:0/2290753361 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f370419a2a0 0x7f370419c690 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f36f800b780 tx=0x7f36f800bb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:26:44.272 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.269+0000 7f3700ff9700 1 -- 172.21.15.172:0/2290753361 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f8004230 con 0x7f370419a2a0 2024-03-28T22:26:44.272 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.269+0000 7f3709f01700 1 -- 172.21.15.172:0/2290753361 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f370419fcc0 con 0x7f370419a2a0 2024-03-28T22:26:44.272 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.269+0000 7f3700ff9700 1 -- 172.21.15.172:0/2290753361 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36f80043d0 con 0x7f370419a2a0 2024-03-28T22:26:44.272 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.269+0000 7f3700ff9700 1 -- 172.21.15.172:0/2290753361 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f8011c60 con 0x7f370419a2a0 2024-03-28T22:26:44.272 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.269+0000 7f3709f01700 1 -- 172.21.15.172:0/2290753361 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37041a0270 con 0x7f370419a2a0 2024-03-28T22:26:44.272 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.270+0000 7f3700ff9700 1 -- 172.21.15.172:0/2290753361 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f36f801a400 con 0x7f370419a2a0 2024-03-28T22:26:44.272 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.270+0000 7f3700ff9700 1 --2- 172.21.15.172:0/2290753361 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f36ec0390d0 0x7f36ec03b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:26:44.272 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.270+0000 7f3709f01700 1 -- 172.21.15.172:0/2290753361 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f370402db30 con 0x7f370419a2a0 2024-03-28T22:26:44.272 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.270+0000 7f3700ff9700 1 -- 172.21.15.172:0/2290753361 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f36f804b800 con 0x7f370419a2a0 2024-03-28T22:26:44.273 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.271+0000 7f37037fe700 1 --2- 172.21.15.172:0/2290753361 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f36ec0390d0 0x7f36ec03b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:26:44.273 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.272+0000 7f37037fe700 1 --2- 172.21.15.172:0/2290753361 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f36ec0390d0 0x7f36ec03b590 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f36f400b5c0 tx=0x7f36f401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:26:44.276 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.277+0000 7f3700ff9700 1 -- 172.21.15.172:0/2290753361 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f36f8003c80 con 0x7f370419a2a0 2024-03-28T22:26:44.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.500+0000 7f3709f01700 1 -- 172.21.15.172:0/2290753361 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi043=x;smithi082=y;smithi172=z", "target": ["mon-mgr", ""]}) -- 0x7f3704062710 con 0x7f36ec0390d0 2024-03-28T22:26:44.513 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.513+0000 7f3700ff9700 1 -- 172.21.15.172:0/2290753361 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f3704062710 con 0x7f36ec0390d0 2024-03-28T22:26:44.513 INFO:teuthology.orchestra.run.smithi172.stdout:Scheduled mgr update... 2024-03-28T22:26:44.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.516+0000 7f3709f01700 1 -- 172.21.15.172:0/2290753361 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f36ec0390d0 msgr2=0x7f36ec03b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:44.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.516+0000 7f3709f01700 1 --2- 172.21.15.172:0/2290753361 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f36ec0390d0 0x7f36ec03b590 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f36f400b5c0 tx=0x7f36f401a040 comp rx=0 tx=0).stop 2024-03-28T22:26:44.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.516+0000 7f3709f01700 1 -- 172.21.15.172:0/2290753361 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f370419a2a0 msgr2=0x7f370419c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:44.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.516+0000 7f3709f01700 1 --2- 172.21.15.172:0/2290753361 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f370419a2a0 0x7f370419c690 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f36f800b780 tx=0x7f36f800bb40 comp rx=0 tx=0).stop 2024-03-28T22:26:44.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.516+0000 7f3709f01700 1 -- 172.21.15.172:0/2290753361 shutdown_connections 2024-03-28T22:26:44.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.516+0000 7f3709f01700 1 --2- 172.21.15.172:0/2290753361 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f36ec0390d0 0x7f36ec03b590 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:44.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.516+0000 7f3709f01700 1 --2- 172.21.15.172:0/2290753361 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f370419cc70 0x7f370419f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:44.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.516+0000 7f3709f01700 1 --2- 172.21.15.172:0/2290753361 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3704100ca0 0x7f370406afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:44.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.516+0000 7f3709f01700 1 --2- 172.21.15.172:0/2290753361 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f370419a2a0 0x7f370419c690 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:44.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.516+0000 7f3709f01700 1 -- 172.21.15.172:0/2290753361 >> 172.21.15.172:0/2290753361 conn(0x7f37040fc220 msgr2=0x7f37040fce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:26:44.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.516+0000 7f3709f01700 1 -- 172.21.15.172:0/2290753361 shutdown_connections 2024-03-28T22:26:44.516 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:26:44.516+0000 7f3709f01700 1 -- 172.21.15.172:0/2290753361 wait complete. 2024-03-28T22:26:45.588 DEBUG:teuthology.orchestra.run.smithi082:mgr.y> sudo journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mgr.y.service 2024-03-28T22:26:45.591 DEBUG:teuthology.orchestra.run.smithi172:mgr.z> sudo journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mgr.z.service 2024-03-28T22:26:45.593 INFO:tasks.cephadm:Deploying OSDs... 2024-03-28T22:26:45.593 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-28T22:26:45.593 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T22:26:45.654 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T22:26:45.654 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_1 2024-03-28T22:26:45.714 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T22:26:45.714 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T22:26:45.714 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 634 Links: 1 2024-03-28T22:26:45.714 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T22:26:45.714 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T22:26:45.714 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-03-28 22:25:40.521107895 +0000 2024-03-28T22:26:45.714 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-03-28 22:20:00.913036357 +0000 2024-03-28T22:26:45.714 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-03-28 22:20:00.913036357 +0000 2024-03-28T22:26:45.714 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-03-28 22:20:00.913036357 +0000 2024-03-28T22:26:45.714 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T22:26:45.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:45 smithi043 ceph-mon[31823]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:45.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:45 smithi043 ceph-mon[31823]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi043=x;smithi082=y;smithi172=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:26:45.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:45 smithi043 ceph-mon[31823]: Saving service mgr spec with placement smithi043=x;smithi082=y;smithi172=z;count:3 2024-03-28T22:26:45.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:45 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:45.812 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-03-28T22:26:45.812 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-03-28T22:26:45.812 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000291478 s, 1.8 MB/s 2024-03-28T22:26:45.816 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T22:26:45.818 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:45 smithi172 ceph-mon[36566]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:45.818 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:45 smithi172 ceph-mon[36566]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi043=x;smithi082=y;smithi172=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:26:45.818 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:45 smithi172 ceph-mon[36566]: Saving service mgr spec with placement smithi043=x;smithi082=y;smithi172=z;count:3 2024-03-28T22:26:45.818 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:45 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:45.876 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_2 2024-03-28T22:26:45.933 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T22:26:45.933 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T22:26:45.933 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 651 Links: 1 2024-03-28T22:26:45.933 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T22:26:45.934 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T22:26:45.934 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-03-28 22:25:40.589107056 +0000 2024-03-28T22:26:45.934 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-03-28 22:20:01.475029446 +0000 2024-03-28T22:26:45.934 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-03-28 22:20:01.475029446 +0000 2024-03-28T22:26:45.934 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-03-28 22:20:01.475029446 +0000 2024-03-28T22:26:45.934 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T22:26:45.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:45 smithi082 ceph-mon[36471]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:45.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:45 smithi082 ceph-mon[36471]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi043=x;smithi082=y;smithi172=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:26:45.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:45 smithi082 ceph-mon[36471]: Saving service mgr spec with placement smithi043=x;smithi082=y;smithi172=z;count:3 2024-03-28T22:26:45.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:45 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:46.021 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-03-28T22:26:46.021 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-03-28T22:26:46.021 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000240175 s, 2.1 MB/s 2024-03-28T22:26:46.025 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T22:26:46.084 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_3 2024-03-28T22:26:46.135 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T22:26:46.135 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T22:26:46.135 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 669 Links: 1 2024-03-28T22:26:46.135 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T22:26:46.135 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T22:26:46.135 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-03-28 22:25:40.642106402 +0000 2024-03-28T22:26:46.136 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-03-28 22:20:02.030022620 +0000 2024-03-28T22:26:46.136 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-03-28 22:20:02.030022620 +0000 2024-03-28T22:26:46.136 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-03-28 22:20:02.030022620 +0000 2024-03-28T22:26:46.136 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T22:26:46.224 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-03-28T22:26:46.224 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-03-28T22:26:46.224 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.00027544 s, 1.9 MB/s 2024-03-28T22:26:46.229 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T22:26:46.287 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_4 2024-03-28T22:26:46.343 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T22:26:46.343 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T22:26:46.343 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 689 Links: 1 2024-03-28T22:26:46.343 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T22:26:46.343 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T22:26:46.343 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-03-28 22:25:40.700105687 +0000 2024-03-28T22:26:46.344 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-03-28 22:20:02.597015647 +0000 2024-03-28T22:26:46.344 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-03-28 22:20:02.597015647 +0000 2024-03-28T22:26:46.344 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-03-28 22:20:02.597015647 +0000 2024-03-28T22:26:46.344 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T22:26:46.438 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-03-28T22:26:46.438 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-03-28T22:26:46.438 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000269554 s, 1.9 MB/s 2024-03-28T22:26:46.443 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T22:26:46.503 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-28T22:26:46.503 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T22:26:46.521 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T22:26:46.522 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_1 2024-03-28T22:26:46.579 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T22:26:46.579 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T22:26:46.579 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 661 Links: 1 2024-03-28T22:26:46.579 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T22:26:46.579 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T22:26:46.579 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-03-28 22:26:30.052648724 +0000 2024-03-28T22:26:46.579 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-03-28 22:19:05.959908887 +0000 2024-03-28T22:26:46.579 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-03-28 22:19:05.959908887 +0000 2024-03-28T22:26:46.579 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-03-28 22:19:05.959908887 +0000 2024-03-28T22:26:46.580 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T22:26:46.648 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-03-28T22:26:46.649 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-03-28T22:26:46.649 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000333893 s, 1.5 MB/s 2024-03-28T22:26:46.650 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T22:26:46.708 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_2 2024-03-28T22:26:46.764 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T22:26:46.764 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T22:26:46.765 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 680 Links: 1 2024-03-28T22:26:46.765 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T22:26:46.765 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T22:26:46.765 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-03-28 22:26:30.052648724 +0000 2024-03-28T22:26:46.765 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-03-28 22:19:06.310901950 +0000 2024-03-28T22:26:46.765 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-03-28 22:19:06.310901950 +0000 2024-03-28T22:26:46.765 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-03-28 22:19:06.310901950 +0000 2024-03-28T22:26:46.765 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T22:26:46.832 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-03-28T22:26:46.832 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-03-28T22:26:46.832 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000211305 s, 2.4 MB/s 2024-03-28T22:26:46.833 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T22:26:46.890 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_3 2024-03-28T22:26:46.944 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T22:26:46.944 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T22:26:46.944 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-03-28T22:26:46.944 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T22:26:46.944 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T22:26:46.945 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-03-28 22:26:30.052648724 +0000 2024-03-28T22:26:46.945 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-03-28 22:19:06.650895231 +0000 2024-03-28T22:26:46.945 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-03-28 22:19:06.650895231 +0000 2024-03-28T22:26:46.945 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-03-28 22:19:06.650895231 +0000 2024-03-28T22:26:46.945 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T22:26:46.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:46 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:46.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:46 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:46.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:46 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:26:46.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:46 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:46.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:46 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:26:46.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:46 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:46.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:46 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T22:26:46.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:46 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T22:26:46.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:46 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T22:26:46.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:46 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:46.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:46 smithi082 ceph-mon[36471]: Deploying daemon mgr.z on smithi172 2024-03-28T22:26:46.976 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-03-28T22:26:46.976 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-03-28T22:26:46.976 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000239683 s, 2.1 MB/s 2024-03-28T22:26:46.977 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T22:26:47.032 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_4 2024-03-28T22:26:47.087 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T22:26:47.087 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T22:26:47.087 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 719 Links: 1 2024-03-28T22:26:47.087 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T22:26:47.087 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T22:26:47.087 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-03-28 22:26:30.053648705 +0000 2024-03-28T22:26:47.087 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-03-28 22:19:06.992888472 +0000 2024-03-28T22:26:47.087 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-03-28 22:19:06.992888472 +0000 2024-03-28T22:26:47.087 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-03-28 22:19:06.992888472 +0000 2024-03-28T22:26:47.088 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T22:26:47.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:46 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:47.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:46 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:47.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:46 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:26:47.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:46 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:47.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:46 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:26:47.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:46 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:47.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:46 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T22:26:47.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:46 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T22:26:47.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:46 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T22:26:47.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:46 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:47.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:46 smithi172 ceph-mon[36566]: Deploying daemon mgr.z on smithi172 2024-03-28T22:26:47.152 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-03-28T22:26:47.152 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-03-28T22:26:47.153 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000215104 s, 2.4 MB/s 2024-03-28T22:26:47.154 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T22:26:47.210 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-28T22:26:47.210 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T22:26:47.225 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T22:26:47.226 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_1 2024-03-28T22:26:47.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:46 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:47.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:46 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:47.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:46 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:26:47.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:46 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:47.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:46 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:26:47.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:46 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:47.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:46 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T22:26:47.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:46 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T22:26:47.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:46 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T22:26:47.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:46 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:47.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:46 smithi043 ceph-mon[31823]: Deploying daemon mgr.z on smithi172 2024-03-28T22:26:47.281 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T22:26:47.281 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T22:26:47.281 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-03-28T22:26:47.281 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T22:26:47.281 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T22:26:47.281 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-03-28 22:26:20.785730890 +0000 2024-03-28T22:26:47.281 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-28 22:18:36.209518556 +0000 2024-03-28T22:26:47.281 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-28 22:18:36.209518556 +0000 2024-03-28T22:26:47.281 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-03-28 22:18:36.209518556 +0000 2024-03-28T22:26:47.281 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T22:26:47.347 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-03-28T22:26:47.347 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-03-28T22:26:47.347 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000201224 s, 2.5 MB/s 2024-03-28T22:26:47.348 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T22:26:47.403 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_2 2024-03-28T22:26:47.457 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T22:26:47.457 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T22:26:47.458 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 681 Links: 1 2024-03-28T22:26:47.458 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T22:26:47.458 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T22:26:47.458 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-03-28 22:26:20.786730867 +0000 2024-03-28T22:26:47.458 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-28 22:18:36.557510163 +0000 2024-03-28T22:26:47.458 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-28 22:18:36.557510163 +0000 2024-03-28T22:26:47.458 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-03-28 22:18:36.557510163 +0000 2024-03-28T22:26:47.458 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T22:26:47.524 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-03-28T22:26:47.524 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-03-28T22:26:47.524 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000292036 s, 1.8 MB/s 2024-03-28T22:26:47.525 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T22:26:47.580 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_3 2024-03-28T22:26:47.635 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T22:26:47.635 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T22:26:47.635 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 699 Links: 1 2024-03-28T22:26:47.635 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T22:26:47.635 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T22:26:47.635 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-03-28 22:26:20.786730867 +0000 2024-03-28T22:26:47.635 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-28 22:18:36.891502107 +0000 2024-03-28T22:26:47.635 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-28 22:18:36.891502107 +0000 2024-03-28T22:26:47.635 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-03-28 22:18:36.891502107 +0000 2024-03-28T22:26:47.635 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T22:26:47.827 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-03-28T22:26:47.827 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-03-28T22:26:47.827 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000240939 s, 2.1 MB/s 2024-03-28T22:26:47.829 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T22:26:47.844 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_4 2024-03-28T22:26:47.902 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T22:26:47.903 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T22:26:47.903 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-03-28T22:26:47.903 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T22:26:47.903 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T22:26:47.903 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-03-28 22:26:47.776093661 +0000 2024-03-28T22:26:47.903 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-28 22:18:37.226494028 +0000 2024-03-28T22:26:47.903 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-28 22:18:37.226494028 +0000 2024-03-28T22:26:47.903 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-03-28 22:18:37.226494028 +0000 2024-03-28T22:26:47.903 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T22:26:48.051 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-03-28T22:26:48.051 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-03-28T22:26:48.051 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000189109 s, 2.7 MB/s 2024-03-28T22:26:48.052 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T22:26:48.067 INFO:tasks.cephadm:Deploying osd.0 on smithi043 with /dev/vg_nvme/lv_4... 2024-03-28T22:26:48.068 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T22:26:48.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:47 smithi172 ceph-mon[36566]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:48.147 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:48 smithi172 systemd[1]: Starting Ceph mgr.z for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:26:48.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:47 smithi082 ceph-mon[36471]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:48.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:47 smithi043 ceph-mon[31823]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:48.537 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:26:48.563 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:48 smithi172 podman[38409]: 2024-03-28 22:26:48.270426064 +0000 UTC m=+0.111346837 container create bd4fcfee3f63124285a9a535510ca6768749c4dad4ee75bfb3cad376f820c892 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-323d23d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-28T22:26:48.564 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:48 smithi172 podman[38409]: 2024-03-28 22:26:48.176603947 +0000 UTC m=+0.017524683 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:26:48.564 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:48 smithi172 podman[38409]: 2024-03-28 22:26:48.48106444 +0000 UTC m=+0.321985187 container init bd4fcfee3f63124285a9a535510ca6768749c4dad4ee75bfb3cad376f820c892 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, RELEASE=squid-323d23d, ceph=True, org.label-schema.vendor=CentOS) 2024-03-28T22:26:48.564 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:48 smithi172 podman[38409]: 2024-03-28 22:26:48.483667536 +0000 UTC m=+0.324588277 container start bd4fcfee3f63124285a9a535510ca6768749c4dad4ee75bfb3cad376f820c892 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-323d23d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T22:26:48.564 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:48 smithi172 ceph-mgr[38428]: -- 172.21.15.172:0/2340271915 <== mon.1 v2:172.21.15.172:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555a198ca5a0 con 0x555a1989ec00 2024-03-28T22:26:48.564 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:48 smithi172 bash[38409]: bd4fcfee3f63124285a9a535510ca6768749c4dad4ee75bfb3cad376f820c892 2024-03-28T22:26:48.897 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:48 smithi172 systemd[1]: Started Ceph mgr.z for c2276428-ed51-11ee-95d0-87774f69a715. 2024-03-28T22:26:49.397 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:49 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:26:49.031+0000 7fd56b98e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T22:26:49.397 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:49 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:26:49.123+0000 7fd56b98e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T22:26:49.787 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:49 smithi043 ceph-mon[31823]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:49.859 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:49 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:49.859 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:49 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:49.859 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:49 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:49.859 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:49 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T22:26:49.859 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:49 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T22:26:49.860 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:49 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T22:26:49.860 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:49 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:49.860 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:49 smithi043 ceph-mon[31823]: Deploying daemon mgr.y on smithi082 2024-03-28T22:26:49.897 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:49 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:26:49.761+0000 7fd56b98e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T22:26:49.897 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:49 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:26:49.852+0000 7fd56b98e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T22:26:49.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:49 smithi172 ceph-mon[36566]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:49.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:49 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:49.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:49 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:49.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:49 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:49.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:49 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T22:26:49.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:49 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T22:26:49.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:49 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T22:26:49.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:49 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:49.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:49 smithi172 ceph-mon[36566]: Deploying daemon mgr.y on smithi082 2024-03-28T22:26:49.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:49 smithi082 ceph-mon[36471]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:49.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:49 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:49.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:49 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:49.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:49 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:49.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:49 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T22:26:49.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:49 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T22:26:49.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:49 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T22:26:49.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:49 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:49.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:49 smithi082 ceph-mon[36471]: Deploying daemon mgr.y on smithi082 2024-03-28T22:26:50.397 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:49 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:26:49.992+0000 7fd56b98e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T22:26:50.397 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:50 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:26:50.087+0000 7fd56b98e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T22:26:50.397 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:50 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:26:50.169+0000 7fd56b98e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T22:26:50.700 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:50 smithi082 systemd[1]: Starting Ceph mgr.y for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:26:50.720 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:50 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:26:50.430+0000 7fd56b98e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T22:26:50.720 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:50 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:26:50.537+0000 7fd56b98e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T22:26:50.720 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:50 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:26:50.622+0000 7fd56b98e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T22:26:50.720 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:50 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:26:50.720+0000 7fd56b98e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T22:26:51.058 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:50 smithi082 podman[37846]: 2024-03-28 22:26:50.781253329 +0000 UTC m=+0.099154029 container create 55e39a534d8ab5cbe4a6a6a2025794a7b4557ec5e20ffb89c1148fe1e59ce407 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-323d23d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True) 2024-03-28T22:26:51.058 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:50 smithi082 podman[37846]: 2024-03-28 22:26:50.699273201 +0000 UTC m=+0.017173919 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:26:51.058 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:50 smithi082 podman[37846]: 2024-03-28 22:26:50.99200064 +0000 UTC m=+0.309901350 container init 55e39a534d8ab5cbe4a6a6a2025794a7b4557ec5e20ffb89c1148fe1e59ce407 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y, RELEASE=squid-323d23d, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2) 2024-03-28T22:26:51.059 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:50 smithi082 podman[37846]: 2024-03-28 22:26:50.994513702 +0000 UTC m=+0.312414404 container start 55e39a534d8ab5cbe4a6a6a2025794a7b4557ec5e20ffb89c1148fe1e59ce407 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-323d23d, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T22:26:51.059 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:51 smithi082 ceph-mgr[37865]: -- 172.21.15.82:0/1933714887 <== mon.2 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b58dc7e5a0 con 0x55b58dc53000 2024-03-28T22:26:51.059 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:51 smithi082 bash[37846]: 55e39a534d8ab5cbe4a6a6a2025794a7b4557ec5e20ffb89c1148fe1e59ce407 2024-03-28T22:26:51.080 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:50 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:26:50.904+0000 7fd56b98e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T22:26:51.314 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T22:26:51.314 INFO:teuthology.orchestra.run.smithi043.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T22:26:51.314 INFO:teuthology.orchestra.run.smithi043.stderr: stderr: 10+0 records in 2024-03-28T22:26:51.314 INFO:teuthology.orchestra.run.smithi043.stderr:10+0 records out 2024-03-28T22:26:51.314 INFO:teuthology.orchestra.run.smithi043.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0332398 s, 315 MB/s 2024-03-28T22:26:51.314 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping successful for: 2024-03-28T22:26:51.450 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:51 smithi082 systemd[1]: Started Ceph mgr.y for c2276428-ed51-11ee-95d0-87774f69a715. 2024-03-28T22:26:51.917 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:51 smithi172 ceph-mon[36566]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:51.917 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:51 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:51.917 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:51 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:51.917 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:51 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:51.917 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:51 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:51.917 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:51 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:26:51.950 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:51 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:26:51.559+0000 7f5f30d11200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T22:26:51.950 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:51 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:26:51.656+0000 7f5f30d11200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T22:26:51.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:51 smithi082 ceph-mon[36471]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:51.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:51 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:51.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:51 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:51.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:51 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:51.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:51 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:51.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:51 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:26:52.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:51 smithi043 ceph-mon[31823]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:52.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:51 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:52.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:51 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:52.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:51 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:52.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:51 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:52.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:51 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:26:52.113 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi043:vg_nvme/lv_4 2024-03-28T22:26:52.257 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:51 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:26:51.939+0000 7fd56b98e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T22:26:52.257 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:52 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:26:52.081+0000 7fd56b98e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T22:26:52.510 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:52 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:26:52.422+0000 7fd56b98e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T22:26:52.510 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:52 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:26:52.510+0000 7fd56b98e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T22:26:52.564 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:26:52.572 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:52 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:26:52.334+0000 7f5f30d11200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T22:26:52.572 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:52 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:26:52.427+0000 7f5f30d11200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T22:26:52.897 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:52 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:26:52.707+0000 7fd56b98e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T22:26:52.950 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:52 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:26:52.572+0000 7f5f30d11200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T22:26:52.950 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:52 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:26:52.674+0000 7f5f30d11200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T22:26:52.950 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:52 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:26:52.756+0000 7f5f30d11200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T22:26:53.182 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:52 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:26:52.899+0000 7fd56b98e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T22:26:53.316 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:53 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:26:53.026+0000 7f5f30d11200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T22:26:53.317 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:53 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:26:53.137+0000 7f5f30d11200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T22:26:53.317 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:53 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:26:53.222+0000 7f5f30d11200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T22:26:53.317 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:53 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:26:53.317+0000 7f5f30d11200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T22:26:53.439 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:53 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:26:53.182+0000 7fd56b98e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T22:26:53.439 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:53 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:26:53.439+0000 7fd56b98e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T22:26:53.643 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:53 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:26:53.492+0000 7f5f30d11200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T22:26:53.897 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:53 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:26:53.672+0000 7fd56b98e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T22:26:53.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:53 smithi172 ceph-mon[36566]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:53.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:53 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:53.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:53 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:53.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:53 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:53.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:53 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:53.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:53 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:53.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:53 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:26:53.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:53 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:53.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:53 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:53.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:53 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T22:26:53.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:53 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T22:26:53.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:53 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:53.932 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:53 smithi043 ceph-mon[31823]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:53.932 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:53 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:53.932 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:53 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:53.932 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:53 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:53.932 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:53 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:53.932 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:53 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:53.932 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:53 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:26:53.932 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:53 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:53.932 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:53 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:53.932 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:53 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T22:26:53.933 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:53 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T22:26:53.933 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:53 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:53.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:53 smithi082 ceph-mon[36471]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:53.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:53 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:53.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:53 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:53.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:53 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:53.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:53 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:53.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:53 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:53.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:53 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:26:53.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:53 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:53.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:53 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:53.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:53 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T22:26:53.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:53 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T22:26:53.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:53 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:54.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:54 smithi043 ceph-mon[31823]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T22:26:54.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:54 smithi043 ceph-mon[31823]: Reconfiguring daemon mgr.x on smithi043 2024-03-28T22:26:54.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:54 smithi172 ceph-mon[36566]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T22:26:54.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:54 smithi172 ceph-mon[36566]: Reconfiguring daemon mgr.x on smithi043 2024-03-28T22:26:54.897 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:54 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:26:54.456+0000 7fd56b98e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T22:26:54.917 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:54 smithi082 ceph-mon[36471]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T22:26:54.917 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:54 smithi082 ceph-mon[36471]: Reconfiguring daemon mgr.x on smithi043 2024-03-28T22:26:54.918 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:54 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:26:54.460+0000 7f5f30d11200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T22:26:54.918 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:54 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:26:54.589+0000 7f5f30d11200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T22:26:54.918 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:54 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:26:54.918+0000 7f5f30d11200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T22:26:55.196 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:55 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:26:55.006+0000 7f5f30d11200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T22:26:55.197 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:55 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:26:55.198+0000 7f5f30d11200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T22:26:55.322 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:55 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:26:55.017+0000 7fd56b98e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T22:26:55.322 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:55 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:26:55.112+0000 7fd56b98e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T22:26:55.323 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:26:55 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:26:55.322+0000 7fd56b98e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T22:26:55.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.359+0000 7f8e59400700 1 -- 172.21.15.43:0/3152056609 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e54108000 msgr2=0x7f8e5410a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:55.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.359+0000 7f8e59400700 1 --2- 172.21.15.43:0/3152056609 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e54108000 0x7f8e5410a3f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c0098f0 tx=0x7f8e4c009c00 comp rx=0 tx=0).stop 2024-03-28T22:26:55.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.359+0000 7f8e59400700 1 -- 172.21.15.43:0/3152056609 shutdown_connections 2024-03-28T22:26:55.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.359+0000 7f8e59400700 1 --2- 172.21.15.43:0/3152056609 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e54108000 0x7f8e5410a3f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:55.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.359+0000 7f8e59400700 1 --2- 172.21.15.43:0/3152056609 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8e54102da0 0x7f8e54105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:55.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.359+0000 7f8e59400700 1 --2- 172.21.15.43:0/3152056609 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e541056d0 0x7f8e54107ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:55.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.359+0000 7f8e59400700 1 -- 172.21.15.43:0/3152056609 >> 172.21.15.43:0/3152056609 conn(0x7f8e540fc710 msgr2=0x7f8e540feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:26:55.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.359+0000 7f8e59400700 1 -- 172.21.15.43:0/3152056609 shutdown_connections 2024-03-28T22:26:55.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.359+0000 7f8e59400700 1 -- 172.21.15.43:0/3152056609 wait complete. 2024-03-28T22:26:55.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.360+0000 7f8e59400700 1 Processor -- start 2024-03-28T22:26:55.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.360+0000 7f8e59400700 1 -- start start 2024-03-28T22:26:55.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.361+0000 7f8e59400700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8e54102da0 0x7f8e541984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:26:55.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.361+0000 7f8e59400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e541056d0 0x7f8e54198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:26:55.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.361+0000 7f8e59400700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e54108000 0x7f8e5419fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:26:55.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.361+0000 7f8e59400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5410ded0 con 0x7f8e541056d0 2024-03-28T22:26:55.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.361+0000 7f8e59400700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5410dd50 con 0x7f8e54102da0 2024-03-28T22:26:55.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.361+0000 7f8e59400700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5410e050 con 0x7f8e54108000 2024-03-28T22:26:55.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.361+0000 7f8e52800700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8e54102da0 0x7f8e541984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:26:55.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.361+0000 7f8e51e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e541056d0 0x7f8e54198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:26:55.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.361+0000 7f8e52800700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8e54102da0 0x7f8e541984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:50296/0 (socket says 172.21.15.43:50296) 2024-03-28T22:26:55.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.361+0000 7f8e52800700 1 -- 172.21.15.43:0/2237579952 learned_addr learned my addr 172.21.15.43:0/2237579952 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:26:55.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.361+0000 7f8e53200700 1 --2- 172.21.15.43:0/2237579952 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e54108000 0x7f8e5419fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:26:55.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.361+0000 7f8e53200700 1 -- 172.21.15.43:0/2237579952 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8e54102da0 msgr2=0x7f8e541984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:55.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.361+0000 7f8e53200700 1 --2- 172.21.15.43:0/2237579952 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8e54102da0 0x7f8e541984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:55.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.362+0000 7f8e53200700 1 -- 172.21.15.43:0/2237579952 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e541056d0 msgr2=0x7f8e54198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:26:55.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.362+0000 7f8e53200700 1 --2- 172.21.15.43:0/2237579952 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e541056d0 0x7f8e54198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:26:55.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.362+0000 7f8e53200700 1 -- 172.21.15.43:0/2237579952 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e541a00e0 con 0x7f8e54108000 2024-03-28T22:26:55.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.362+0000 7f8e53200700 1 --2- 172.21.15.43:0/2237579952 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e54108000 0x7f8e5419fa40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c009230 tx=0x7f8e4c005250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:26:55.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.362+0000 7f8e4b400700 1 -- 172.21.15.43:0/2237579952 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c025070 con 0x7f8e54108000 2024-03-28T22:26:55.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.362+0000 7f8e4b400700 1 -- 172.21.15.43:0/2237579952 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e4c00f950 con 0x7f8e54108000 2024-03-28T22:26:55.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.362+0000 7f8e4b400700 1 -- 172.21.15.43:0/2237579952 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c003b90 con 0x7f8e54108000 2024-03-28T22:26:55.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.362+0000 7f8e59400700 1 -- 172.21.15.43:0/2237579952 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e541a0370 con 0x7f8e54108000 2024-03-28T22:26:55.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.362+0000 7f8e59400700 1 -- 172.21.15.43:0/2237579952 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e541a0820 con 0x7f8e54108000 2024-03-28T22:26:55.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.363+0000 7f8e59400700 1 -- 172.21.15.43:0/2237579952 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e54068f00 con 0x7f8e54108000 2024-03-28T22:26:55.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.364+0000 7f8e4b400700 1 -- 172.21.15.43:0/2237579952 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 13) ==== 45984+0+0 (secure 0 0 0) 0x7f8e4c028070 con 0x7f8e54108000 2024-03-28T22:26:55.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.364+0000 7f8e4b400700 1 --2- 172.21.15.43:0/2237579952 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f8e38038e80 0x7f8e3803b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:26:55.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.364+0000 7f8e4b400700 1 -- 172.21.15.43:0/2237579952 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f8e4c04c450 con 0x7f8e54108000 2024-03-28T22:26:55.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.364+0000 7f8e52800700 1 --2- 172.21.15.43:0/2237579952 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f8e38038e80 0x7f8e3803b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:26:55.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.365+0000 7f8e52800700 1 --2- 172.21.15.43:0/2237579952 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f8e38038e80 0x7f8e3803b340 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8e44009fd0 tx=0x7f8e4400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:26:55.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.369+0000 7f8e4b400700 1 -- 172.21.15.43:0/2237579952 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8e4c013030 con 0x7f8e54108000 2024-03-28T22:26:55.450 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:55 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:26:55.380+0000 7f5f30d11200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T22:26:55.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.607+0000 7f8e59400700 1 -- 172.21.15.43:0/2237579952 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f8e54062680 con 0x7f8e38038e80 2024-03-28T22:26:55.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:55.659+0000 7f8e4b400700 1 -- 172.21.15.43:0/2237579952 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mgrmap(e 14) ==== 91342+0+0 (secure 0 0 0) 0x7f8e4c00b560 con 0x7f8e54108000 2024-03-28T22:26:55.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:55 smithi043 ceph-mon[31823]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:55.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:55 smithi043 ceph-mon[31823]: from='mgr.? 172.21.15.172:0/1721227893' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T22:26:55.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:55 smithi043 ceph-mon[31823]: Standby manager daemon z started 2024-03-28T22:26:55.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:55 smithi043 ceph-mon[31823]: from='mgr.? 172.21.15.172:0/1721227893' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T22:26:55.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:55 smithi043 ceph-mon[31823]: from='mgr.? 172.21.15.172:0/1721227893' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T22:26:55.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:55 smithi043 ceph-mon[31823]: from='mgr.? 172.21.15.172:0/1721227893' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T22:26:55.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:55 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:26:55.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:55 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:26:55.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:55 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:55.916 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:55 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:26:55.661+0000 7f5f30d11200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T22:26:55.916 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:55 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:26:55.917+0000 7f5f30d11200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T22:26:55.916 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:55 smithi082 ceph-mon[36471]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:55.916 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:55 smithi082 ceph-mon[36471]: from='mgr.? 172.21.15.172:0/1721227893' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T22:26:55.917 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:55 smithi082 ceph-mon[36471]: Standby manager daemon z started 2024-03-28T22:26:55.917 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:55 smithi082 ceph-mon[36471]: from='mgr.? 172.21.15.172:0/1721227893' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T22:26:55.917 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:55 smithi082 ceph-mon[36471]: from='mgr.? 172.21.15.172:0/1721227893' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T22:26:55.917 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:55 smithi082 ceph-mon[36471]: from='mgr.? 172.21.15.172:0/1721227893' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T22:26:55.917 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:55 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:26:55.917 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:55 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:26:55.917 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:55 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:56.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:55 smithi172 ceph-mon[36566]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:56.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:55 smithi172 ceph-mon[36566]: from='mgr.? 172.21.15.172:0/1721227893' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T22:26:56.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:55 smithi172 ceph-mon[36566]: Standby manager daemon z started 2024-03-28T22:26:56.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:55 smithi172 ceph-mon[36566]: from='mgr.? 172.21.15.172:0/1721227893' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T22:26:56.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:55 smithi172 ceph-mon[36566]: from='mgr.? 172.21.15.172:0/1721227893' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T22:26:56.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:55 smithi172 ceph-mon[36566]: from='mgr.? 172.21.15.172:0/1721227893' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T22:26:56.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:55 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:26:56.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:55 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:26:56.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:55 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:56.200 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:56 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:26:56.149+0000 7f5f30d11200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T22:26:56.922 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:56 smithi043 ceph-mon[31823]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:26:56.922 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:56 smithi043 ceph-mon[31823]: mgrmap e14: x(active, since 87s), standbys: z 2024-03-28T22:26:56.922 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:56 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T22:26:56.922 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:56 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:56.922 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:56 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:56.922 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:56 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:26:56.922 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:56 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:56.923 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:56 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:26:56.923 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:56 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:56.945 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:56 smithi082 ceph-mon[36471]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:26:56.945 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:56 smithi082 ceph-mon[36471]: mgrmap e14: x(active, since 87s), standbys: z 2024-03-28T22:26:56.945 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:56 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T22:26:56.945 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:56 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:56.945 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:56 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:56.946 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:56 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:26:56.946 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:56 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:56.946 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:56 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:26:56.946 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:56 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:57.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:56 smithi172 ceph-mon[36566]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:26:57.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:56 smithi172 ceph-mon[36566]: mgrmap e14: x(active, since 87s), standbys: z 2024-03-28T22:26:57.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:56 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T22:26:57.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:56 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:57.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:56 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:57.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:56 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:26:57.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:56 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:26:57.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:56 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:26:57.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:56 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:26:57.200 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:56 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:26:56.946+0000 7f5f30d11200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T22:26:57.834 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:57 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:26:57.509+0000 7f5f30d11200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T22:26:57.835 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:57 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:26:57.624+0000 7f5f30d11200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T22:26:57.835 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:26:57 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:26:57.835+0000 7f5f30d11200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T22:26:58.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:26:58.013+0000 7f8e4b400700 1 -- 172.21.15.43:0/2237579952 <== mon.1 v2:172.21.15.172:3300/0 8 ==== mgrmap(e 15) ==== 136700+0+0 (secure 0 0 0) 0x7f8e4c01dc90 con 0x7f8e54108000 2024-03-28T22:26:58.105 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:58 smithi082 ceph-mon[36471]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:58.105 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:58 smithi082 ceph-mon[36471]: Standby manager daemon y started 2024-03-28T22:26:58.105 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:58 smithi082 ceph-mon[36471]: from='mgr.? 172.21.15.82:0/4205097907' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T22:26:58.105 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:58 smithi082 ceph-mon[36471]: from='mgr.? 172.21.15.82:0/4205097907' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T22:26:58.106 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:58 smithi082 ceph-mon[36471]: from='mgr.? 172.21.15.82:0/4205097907' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T22:26:58.106 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:58 smithi082 ceph-mon[36471]: from='mgr.? 172.21.15.82:0/4205097907' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T22:26:58.122 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:58 smithi043 ceph-mon[31823]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:58.122 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:58 smithi043 ceph-mon[31823]: Standby manager daemon y started 2024-03-28T22:26:58.122 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:58 smithi043 ceph-mon[31823]: from='mgr.? 172.21.15.82:0/4205097907' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T22:26:58.122 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:58 smithi043 ceph-mon[31823]: from='mgr.? 172.21.15.82:0/4205097907' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T22:26:58.122 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:58 smithi043 ceph-mon[31823]: from='mgr.? 172.21.15.82:0/4205097907' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T22:26:58.122 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:58 smithi043 ceph-mon[31823]: from='mgr.? 172.21.15.82:0/4205097907' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T22:26:58.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:58 smithi172 ceph-mon[36566]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:26:58.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:58 smithi172 ceph-mon[36566]: Standby manager daemon y started 2024-03-28T22:26:58.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:58 smithi172 ceph-mon[36566]: from='mgr.? 172.21.15.82:0/4205097907' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T22:26:58.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:58 smithi172 ceph-mon[36566]: from='mgr.? 172.21.15.82:0/4205097907' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T22:26:58.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:58 smithi172 ceph-mon[36566]: from='mgr.? 172.21.15.82:0/4205097907' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T22:26:58.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:58 smithi172 ceph-mon[36566]: from='mgr.? 172.21.15.82:0/4205097907' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T22:26:59.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:59 smithi172 ceph-mon[36566]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-03-28T22:26:59.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:26:59 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T22:26:59.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:59 smithi082 ceph-mon[36471]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-03-28T22:26:59.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:26:59 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T22:26:59.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:59 smithi043 ceph-mon[31823]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-03-28T22:26:59.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:26:59 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T22:27:00.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:00 smithi172 ceph-mon[36566]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:00.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:00 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/193307582' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b855adcd-786c-4e42-b8fe-a515a77ce979"}]: dispatch 2024-03-28T22:27:00.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:00 smithi172 ceph-mon[36566]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b855adcd-786c-4e42-b8fe-a515a77ce979"}]: dispatch 2024-03-28T22:27:00.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:00 smithi172 ceph-mon[36566]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b855adcd-786c-4e42-b8fe-a515a77ce979"}]': finished 2024-03-28T22:27:00.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:00 smithi172 ceph-mon[36566]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T22:27:00.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:00 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:27:00.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:00 smithi082 ceph-mon[36471]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:00.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:00 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/193307582' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b855adcd-786c-4e42-b8fe-a515a77ce979"}]: dispatch 2024-03-28T22:27:00.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:00 smithi082 ceph-mon[36471]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b855adcd-786c-4e42-b8fe-a515a77ce979"}]: dispatch 2024-03-28T22:27:00.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:00 smithi082 ceph-mon[36471]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b855adcd-786c-4e42-b8fe-a515a77ce979"}]': finished 2024-03-28T22:27:00.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:00 smithi082 ceph-mon[36471]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T22:27:00.451 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:00 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:27:00.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:00 smithi043 ceph-mon[31823]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:00.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:00 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/193307582' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b855adcd-786c-4e42-b8fe-a515a77ce979"}]: dispatch 2024-03-28T22:27:00.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:00 smithi043 ceph-mon[31823]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b855adcd-786c-4e42-b8fe-a515a77ce979"}]: dispatch 2024-03-28T22:27:00.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:00 smithi043 ceph-mon[31823]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b855adcd-786c-4e42-b8fe-a515a77ce979"}]': finished 2024-03-28T22:27:00.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:00 smithi043 ceph-mon[31823]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T22:27:00.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:00 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:27:01.384 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:01 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/517557135' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:27:01.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:01 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/517557135' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:27:01.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:01 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/517557135' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:27:02.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:02 smithi172 ceph-mon[36566]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:02.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:02 smithi082 ceph-mon[36471]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:02.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:02 smithi043 ceph-mon[31823]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:03.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:03 smithi172 ceph-mon[36566]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:03.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:03 smithi082 ceph-mon[36471]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:03.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:03 smithi043 ceph-mon[31823]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:05.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:05 smithi172 ceph-mon[36566]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:05.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:05 smithi082 ceph-mon[36471]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:05.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:05 smithi043 ceph-mon[31823]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:07.394 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:07 smithi172 ceph-mon[36566]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:07.398 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:07 smithi043 ceph-mon[31823]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:07.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:07 smithi082 ceph-mon[36471]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:09.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:09 smithi172 ceph-mon[36566]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:09.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:09 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T22:27:09.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:09 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:27:09.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:09 smithi172 ceph-mon[36566]: Deploying daemon osd.0 on smithi043 2024-03-28T22:27:09.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:09 smithi082 ceph-mon[36471]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:09.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:09 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T22:27:09.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:09 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:27:09.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:09 smithi082 ceph-mon[36471]: Deploying daemon osd.0 on smithi043 2024-03-28T22:27:09.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:09 smithi043 ceph-mon[31823]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:09.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:09 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T22:27:09.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:09 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:27:09.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:09 smithi043 ceph-mon[31823]: Deploying daemon osd.0 on smithi043 2024-03-28T22:27:11.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:11 smithi172 ceph-mon[36566]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:11.444 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:11 smithi043 ceph-mon[31823]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:11.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:11 smithi082 ceph-mon[36471]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:13.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:13 smithi043 ceph-mon[31823]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:13.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:13 smithi172 ceph-mon[36566]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:13.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:13 smithi082 ceph-mon[36471]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:15.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:15 smithi172 ceph-mon[36566]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:15.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:15 smithi082 ceph-mon[36471]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:15.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:15 smithi043 ceph-mon[31823]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:18.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:17 smithi043 ceph-mon[31823]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:18.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:17 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:18.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:17 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:27:18.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:17 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:18.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:17 smithi172 ceph-mon[36566]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:18.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:17 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:18.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:17 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:27:18.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:17 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:18.153 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:17 smithi082 ceph-mon[36471]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:18.153 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:17 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:18.153 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:17 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:27:18.153 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:17 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:19.920 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:19 smithi043 ceph-mon[31823]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:19.920 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:19 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:19.920 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:19 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:19.921 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:19 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:27:19.921 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:19 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:27:19.921 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:19 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:20.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:19 smithi172 ceph-mon[36566]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:20.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:19 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:20.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:19 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:20.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:19 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:27:20.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:19 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:27:20.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:19 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:20.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:19 smithi082 ceph-mon[36471]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:20.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:19 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:20.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:19 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:20.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:19 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:27:20.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:19 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:27:20.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:19 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:21.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:20 smithi043 ceph-mon[31823]: from='osd.0 [v2:172.21.15.43:6802/2826124409,v1:172.21.15.43:6803/2826124409]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T22:27:21.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:20 smithi043 ceph-mon[31823]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T22:27:21.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:20 smithi172 ceph-mon[36566]: from='osd.0 [v2:172.21.15.43:6802/2826124409,v1:172.21.15.43:6803/2826124409]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T22:27:21.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:20 smithi172 ceph-mon[36566]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T22:27:21.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:20 smithi082 ceph-mon[36471]: from='osd.0 [v2:172.21.15.43:6802/2826124409,v1:172.21.15.43:6803/2826124409]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T22:27:21.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:20 smithi082 ceph-mon[36471]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T22:27:21.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:27:21.662+0000 7f8e4b400700 1 -- 172.21.15.43:0/2237579952 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8e54062680 con 0x7f8e38038e80 2024-03-28T22:27:21.664 INFO:teuthology.orchestra.run.smithi043.stdout:Created osd(s) 0 on host 'smithi043' 2024-03-28T22:27:21.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:27:21.667+0000 7f8e59400700 1 -- 172.21.15.43:0/2237579952 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f8e38038e80 msgr2=0x7f8e3803b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:27:21.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:27:21.667+0000 7f8e59400700 1 --2- 172.21.15.43:0/2237579952 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f8e38038e80 0x7f8e3803b340 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8e44009fd0 tx=0x7f8e4400b040 comp rx=0 tx=0).stop 2024-03-28T22:27:21.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:27:21.667+0000 7f8e59400700 1 -- 172.21.15.43:0/2237579952 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e54108000 msgr2=0x7f8e5419fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:27:21.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:27:21.667+0000 7f8e59400700 1 --2- 172.21.15.43:0/2237579952 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e54108000 0x7f8e5419fa40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c009230 tx=0x7f8e4c005250 comp rx=0 tx=0).stop 2024-03-28T22:27:21.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:27:21.667+0000 7f8e59400700 1 -- 172.21.15.43:0/2237579952 shutdown_connections 2024-03-28T22:27:21.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:27:21.667+0000 7f8e59400700 1 --2- 172.21.15.43:0/2237579952 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f8e38038e80 0x7f8e3803b340 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:27:21.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:27:21.667+0000 7f8e59400700 1 --2- 172.21.15.43:0/2237579952 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e54108000 0x7f8e5419fa40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:27:21.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:27:21.667+0000 7f8e59400700 1 --2- 172.21.15.43:0/2237579952 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8e54102da0 0x7f8e541984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:27:21.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:27:21.667+0000 7f8e59400700 1 --2- 172.21.15.43:0/2237579952 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e541056d0 0x7f8e54198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:27:21.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:27:21.667+0000 7f8e59400700 1 -- 172.21.15.43:0/2237579952 >> 172.21.15.43:0/2237579952 conn(0x7f8e540fc710 msgr2=0x7f8e541037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:27:21.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:27:21.667+0000 7f8e59400700 1 -- 172.21.15.43:0/2237579952 shutdown_connections 2024-03-28T22:27:21.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:27:21.667+0000 7f8e59400700 1 -- 172.21.15.43:0/2237579952 wait complete. 2024-03-28T22:27:22.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:21 smithi043 ceph-mon[31823]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:22.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:21 smithi043 ceph-mon[31823]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T22:27:22.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:21 smithi043 ceph-mon[31823]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T22:27:22.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:21 smithi043 ceph-mon[31823]: from='osd.0 [v2:172.21.15.43:6802/2826124409,v1:172.21.15.43:6803/2826124409]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-28T22:27:22.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:21 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:27:22.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:21 smithi043 ceph-mon[31823]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-28T22:27:22.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:21 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:22.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:21 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:22.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:21 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:27:22.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:21 smithi043 ceph-mon[31823]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-03-28T22:27:22.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:21 smithi043 ceph-mon[31823]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T22:27:22.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:21 smithi172 ceph-mon[36566]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:22.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:21 smithi172 ceph-mon[36566]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T22:27:22.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:21 smithi172 ceph-mon[36566]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T22:27:22.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:21 smithi172 ceph-mon[36566]: from='osd.0 [v2:172.21.15.43:6802/2826124409,v1:172.21.15.43:6803/2826124409]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-28T22:27:22.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:27:22.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:21 smithi172 ceph-mon[36566]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-28T22:27:22.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:22.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:22.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:27:22.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:21 smithi172 ceph-mon[36566]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-03-28T22:27:22.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:21 smithi172 ceph-mon[36566]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T22:27:22.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:21 smithi082 ceph-mon[36471]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:22.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:21 smithi082 ceph-mon[36471]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T22:27:22.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:21 smithi082 ceph-mon[36471]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T22:27:22.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:21 smithi082 ceph-mon[36471]: from='osd.0 [v2:172.21.15.43:6802/2826124409,v1:172.21.15.43:6803/2826124409]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-28T22:27:22.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:21 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:27:22.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:21 smithi082 ceph-mon[36471]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-28T22:27:22.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:21 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:22.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:21 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:22.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:21 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:27:22.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:21 smithi082 ceph-mon[36471]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-03-28T22:27:22.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:21 smithi082 ceph-mon[36471]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T22:27:22.541 DEBUG:teuthology.orchestra.run.smithi043:osd.0> sudo journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.0.service 2024-03-28T22:27:22.546 INFO:tasks.cephadm:Deploying osd.1 on smithi082 with /dev/vg_nvme/lv_4... 2024-03-28T22:27:22.546 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T22:27:22.790 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T22:27:22.911 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:22 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:27:22.911 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:22 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:27:23.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:22 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:27:23.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:22 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:27:23.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:22 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:27:23.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:22 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:27:24.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:23 smithi043 ceph-mon[31823]: purged_snaps scrub starts 2024-03-28T22:27:24.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:23 smithi043 ceph-mon[31823]: purged_snaps scrub ok 2024-03-28T22:27:24.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:23 smithi043 ceph-mon[31823]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:24.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:23 smithi043 ceph-mon[31823]: osd.0 [v2:172.21.15.43:6802/2826124409,v1:172.21.15.43:6803/2826124409] boot 2024-03-28T22:27:24.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:23 smithi043 ceph-mon[31823]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T22:27:24.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:23 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:27:24.020 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:23 smithi082 ceph-mon[36471]: purged_snaps scrub starts 2024-03-28T22:27:24.020 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:23 smithi082 ceph-mon[36471]: purged_snaps scrub ok 2024-03-28T22:27:24.020 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:23 smithi082 ceph-mon[36471]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:24.020 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:23 smithi082 ceph-mon[36471]: osd.0 [v2:172.21.15.43:6802/2826124409,v1:172.21.15.43:6803/2826124409] boot 2024-03-28T22:27:24.020 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:23 smithi082 ceph-mon[36471]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T22:27:24.020 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:23 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:27:24.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:23 smithi172 ceph-mon[36566]: purged_snaps scrub starts 2024-03-28T22:27:24.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:23 smithi172 ceph-mon[36566]: purged_snaps scrub ok 2024-03-28T22:27:24.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:23 smithi172 ceph-mon[36566]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T22:27:24.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:23 smithi172 ceph-mon[36566]: osd.0 [v2:172.21.15.43:6802/2826124409,v1:172.21.15.43:6803/2826124409] boot 2024-03-28T22:27:24.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:23 smithi172 ceph-mon[36566]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T22:27:24.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:23 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:27:24.693 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T22:27:24.693 INFO:teuthology.orchestra.run.smithi082.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T22:27:24.693 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10+0 records in 2024-03-28T22:27:24.693 INFO:teuthology.orchestra.run.smithi082.stderr:10+0 records out 2024-03-28T22:27:24.693 INFO:teuthology.orchestra.run.smithi082.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112254 s, 934 MB/s 2024-03-28T22:27:24.693 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping successful for: 2024-03-28T22:27:24.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:24 smithi082 ceph-mon[36471]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T22:27:25.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:24 smithi172 ceph-mon[36566]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T22:27:25.244 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi082:vg_nvme/lv_4 2024-03-28T22:27:25.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:24 smithi043 ceph-mon[31823]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T22:27:25.478 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T22:27:25.911 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:25 smithi082 ceph-mon[36471]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:26.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:25 smithi172 ceph-mon[36566]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:26.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:25 smithi043 ceph-mon[31823]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:27.013 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.011+0000 7f1489b58700 1 -- 172.21.15.82:0/3972558107 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1484103890 msgr2=0x7f1484103cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:27:27.013 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.011+0000 7f1489b58700 1 --2- 172.21.15.82:0/3972558107 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1484103890 0x7f1484103cb0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f147400b0a0 tx=0x7f147400b3b0 comp rx=0 tx=0).stop 2024-03-28T22:27:27.013 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.011+0000 7f1489b58700 1 -- 172.21.15.82:0/3972558107 shutdown_connections 2024-03-28T22:27:27.013 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.011+0000 7f1489b58700 1 --2- 172.21.15.82:0/3972558107 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1484100cb0 0x7f148410a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:27:27.013 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.011+0000 7f1489b58700 1 --2- 172.21.15.82:0/3972558107 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1484103890 0x7f1484103cb0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:27:27.013 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.011+0000 7f1489b58700 1 --2- 172.21.15.82:0/3972558107 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1484100310 0x7f1484100770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:27:27.014 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.011+0000 7f1489b58700 1 -- 172.21.15.82:0/3972558107 >> 172.21.15.82:0/3972558107 conn(0x7f14840fbeb0 msgr2=0x7f14840fe2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:27:27.014 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.011+0000 7f1489b58700 1 -- 172.21.15.82:0/3972558107 shutdown_connections 2024-03-28T22:27:27.014 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.011+0000 7f1489b58700 1 -- 172.21.15.82:0/3972558107 wait complete. 2024-03-28T22:27:27.014 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.012+0000 7f1489b58700 1 Processor -- start 2024-03-28T22:27:27.014 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.012+0000 7f1489b58700 1 -- start start 2024-03-28T22:27:27.014 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.012+0000 7f1489b58700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1484100310 0x7f1484198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:27:27.014 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.012+0000 7f1489b58700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1484100cb0 0x7f1484198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:27:27.014 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.012+0000 7f1489b58700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1484103890 0x7f148419f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:27:27.014 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.012+0000 7f1489b58700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f148410dea0 con 0x7f1484100310 2024-03-28T22:27:27.014 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.012+0000 7f1489b58700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f148410dd20 con 0x7f1484100cb0 2024-03-28T22:27:27.014 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.012+0000 7f1489b58700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f148410e020 con 0x7f1484103890 2024-03-28T22:27:27.015 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.012+0000 7f1482ffd700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1484100cb0 0x7f1484198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:27:27.015 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.012+0000 7f1482ffd700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1484100cb0 0x7f1484198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:41858/0 (socket says 172.21.15.82:41858) 2024-03-28T22:27:27.015 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.012+0000 7f1482ffd700 1 -- 172.21.15.82:0/239251839 learned_addr learned my addr 172.21.15.82:0/239251839 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-03-28T22:27:27.015 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.012+0000 7f1483fff700 1 --2- 172.21.15.82:0/239251839 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1484103890 0x7f148419f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:27:27.015 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.012+0000 7f14837fe700 1 --2- 172.21.15.82:0/239251839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1484100310 0x7f1484198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:27:27.015 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.012+0000 7f1482ffd700 1 -- 172.21.15.82:0/239251839 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1484103890 msgr2=0x7f148419f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:27:27.015 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.012+0000 7f1482ffd700 1 --2- 172.21.15.82:0/239251839 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1484103890 0x7f148419f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:27:27.015 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.013+0000 7f1482ffd700 1 -- 172.21.15.82:0/239251839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1484100310 msgr2=0x7f1484198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:27:27.015 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.013+0000 7f1482ffd700 1 --2- 172.21.15.82:0/239251839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1484100310 0x7f1484198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:27:27.015 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.013+0000 7f1482ffd700 1 -- 172.21.15.82:0/239251839 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14841a0090 con 0x7f1484100cb0 2024-03-28T22:27:27.015 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.013+0000 7f1482ffd700 1 --2- 172.21.15.82:0/239251839 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1484100cb0 0x7f1484198970 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f147800c9b0 tx=0x7f147800cd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:27:27.015 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.013+0000 7f1480ff9700 1 -- 172.21.15.82:0/239251839 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1478007b00 con 0x7f1484100cb0 2024-03-28T22:27:27.015 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.013+0000 7f1480ff9700 1 -- 172.21.15.82:0/239251839 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1478007ca0 con 0x7f1484100cb0 2024-03-28T22:27:27.015 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.013+0000 7f1489b58700 1 -- 172.21.15.82:0/239251839 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14841a0380 con 0x7f1484100cb0 2024-03-28T22:27:27.016 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.013+0000 7f1480ff9700 1 -- 172.21.15.82:0/239251839 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1478018820 con 0x7f1484100cb0 2024-03-28T22:27:27.016 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.013+0000 7f1489b58700 1 -- 172.21.15.82:0/239251839 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14841a0910 con 0x7f1484100cb0 2024-03-28T22:27:27.017 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.014+0000 7f1489b58700 1 -- 172.21.15.82:0/239251839 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1484068f00 con 0x7f1484100cb0 2024-03-28T22:27:27.017 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.015+0000 7f1480ff9700 1 -- 172.21.15.82:0/239251839 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 15) ==== 136700+0+0 (secure 0 0 0) 0x7f147800fa30 con 0x7f1484100cb0 2024-03-28T22:27:27.017 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.016+0000 7f1480ff9700 1 --2- 172.21.15.82:0/239251839 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f146c0a1e00 0x7f146c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:27:27.017 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.016+0000 7f1480ff9700 1 -- 172.21.15.82:0/239251839 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f14780cc2c0 con 0x7f1484100cb0 2024-03-28T22:27:27.017 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.016+0000 7f14837fe700 1 --2- 172.21.15.82:0/239251839 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f146c0a1e00 0x7f146c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:27:27.017 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.017+0000 7f14837fe700 1 --2- 172.21.15.82:0/239251839 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f146c0a1e00 0x7f146c0a42c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1474006130 tx=0x7f14740060c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:27:27.020 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.020+0000 7f1480ff9700 1 -- 172.21.15.82:0/239251839 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1478014030 con 0x7f1484100cb0 2024-03-28T22:27:27.224 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:27.224+0000 7f1489b58700 1 -- 172.21.15.82:0/239251839 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f1484062680 con 0x7f146c0a1e00 2024-03-28T22:27:28.039 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:27 smithi082 ceph-mon[36471]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:28.039 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:27 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:27:28.039 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:27 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:27:28.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:27 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:27:28.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:27 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:28.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:27 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:28.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:27 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:27:28.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:27 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:28.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:27 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:27:28.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:27 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:27:28.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:27 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:28.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:27 smithi172 ceph-mon[36566]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:28.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:27 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:27:28.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:27 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:27:28.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:27 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:27:28.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:27 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:28.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:27 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:28.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:27 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:27:28.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:27 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:28.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:27 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:27:28.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:27 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:27:28.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:27 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:28.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:27 smithi043 ceph-mon[31823]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:28.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:27 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:27:28.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:27 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:27:28.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:27 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:27:28.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:27 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:28.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:27 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:28.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:27 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:27:28.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:27 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:28.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:27 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:27:28.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:27 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:27:28.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:27 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:29.021 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:28 smithi082 ceph-mon[36471]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:27:29.021 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:28 smithi082 ceph-mon[36471]: Detected new or changed devices on smithi043 2024-03-28T22:27:29.021 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:28 smithi082 ceph-mon[36471]: Adjusting osd_memory_target on smithi043 to 17095M 2024-03-28T22:27:29.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:28 smithi172 ceph-mon[36566]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:27:29.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:28 smithi172 ceph-mon[36566]: Detected new or changed devices on smithi043 2024-03-28T22:27:29.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:28 smithi172 ceph-mon[36566]: Adjusting osd_memory_target on smithi043 to 17095M 2024-03-28T22:27:29.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:28 smithi043 ceph-mon[31823]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:27:29.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:28 smithi043 ceph-mon[31823]: Detected new or changed devices on smithi043 2024-03-28T22:27:29.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:28 smithi043 ceph-mon[31823]: Adjusting osd_memory_target on smithi043 to 17095M 2024-03-28T22:27:30.140 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:29 smithi082 ceph-mon[36471]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:30.140 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:29 smithi082 ceph-mon[36471]: from='client.? 172.21.15.82:0/4093447918' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "031c143d-8ad1-4b68-8db9-3c717137e3e4"}]: dispatch 2024-03-28T22:27:30.140 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:29 smithi082 ceph-mon[36471]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "031c143d-8ad1-4b68-8db9-3c717137e3e4"}]: dispatch 2024-03-28T22:27:30.140 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:29 smithi082 ceph-mon[36471]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "031c143d-8ad1-4b68-8db9-3c717137e3e4"}]': finished 2024-03-28T22:27:30.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:29 smithi172 ceph-mon[36566]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:30.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:29 smithi172 ceph-mon[36566]: from='client.? 172.21.15.82:0/4093447918' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "031c143d-8ad1-4b68-8db9-3c717137e3e4"}]: dispatch 2024-03-28T22:27:30.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:29 smithi172 ceph-mon[36566]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "031c143d-8ad1-4b68-8db9-3c717137e3e4"}]: dispatch 2024-03-28T22:27:30.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:29 smithi172 ceph-mon[36566]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "031c143d-8ad1-4b68-8db9-3c717137e3e4"}]': finished 2024-03-28T22:27:30.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:29 smithi043 ceph-mon[31823]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:30.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:29 smithi043 ceph-mon[31823]: from='client.? 172.21.15.82:0/4093447918' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "031c143d-8ad1-4b68-8db9-3c717137e3e4"}]: dispatch 2024-03-28T22:27:30.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:29 smithi043 ceph-mon[31823]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "031c143d-8ad1-4b68-8db9-3c717137e3e4"}]: dispatch 2024-03-28T22:27:30.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:29 smithi043 ceph-mon[31823]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "031c143d-8ad1-4b68-8db9-3c717137e3e4"}]': finished 2024-03-28T22:27:31.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:30 smithi172 ceph-mon[36566]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T22:27:31.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:30 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:27:31.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:30 smithi172 ceph-mon[36566]: from='client.? 172.21.15.82:0/1462747834' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:27:31.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:30 smithi082 ceph-mon[36471]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T22:27:31.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:30 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:27:31.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:30 smithi082 ceph-mon[36471]: from='client.? 172.21.15.82:0/1462747834' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:27:31.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:30 smithi043 ceph-mon[31823]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T22:27:31.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:30 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:27:31.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:30 smithi043 ceph-mon[31823]: from='client.? 172.21.15.82:0/1462747834' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:27:32.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:31 smithi172 ceph-mon[36566]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:32.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:31 smithi082 ceph-mon[36471]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:32.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:31 smithi043 ceph-mon[31823]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:34.021 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:33 smithi082 ceph-mon[36471]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:33 smithi172 ceph-mon[36566]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:34.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:33 smithi043 ceph-mon[31823]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:35.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:35 smithi082 ceph-mon[36471]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:36.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:35 smithi172 ceph-mon[36566]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:36.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:35 smithi043 ceph-mon[31823]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:38.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:37 smithi172 ceph-mon[36566]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:38.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:37 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T22:27:38.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:37 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:27:38.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:37 smithi172 ceph-mon[36566]: Deploying daemon osd.1 on smithi082 2024-03-28T22:27:38.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:37 smithi082 ceph-mon[36471]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:38.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:37 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T22:27:38.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:37 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:27:38.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:37 smithi082 ceph-mon[36471]: Deploying daemon osd.1 on smithi082 2024-03-28T22:27:38.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:37 smithi043 ceph-mon[31823]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:38.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:37 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T22:27:38.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:37 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:27:38.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:37 smithi043 ceph-mon[31823]: Deploying daemon osd.1 on smithi082 2024-03-28T22:27:40.130 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:39 smithi082 ceph-mon[36471]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:40.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:39 smithi172 ceph-mon[36566]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:40.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:39 smithi043 ceph-mon[31823]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:41.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:41 smithi082 ceph-mon[36471]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:42.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:41 smithi172 ceph-mon[36566]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:42.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:41 smithi043 ceph-mon[31823]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:43.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:43 smithi082 ceph-mon[36471]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:43.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:43 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:43.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:43 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:43.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:43 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:27:43.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:43 smithi043 ceph-mon[31823]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:43.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:43 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:43.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:43 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:43.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:43 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:27:43.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:43 smithi172 ceph-mon[36566]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:43.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:43 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:43.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:43 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:43.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:43 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:27:45.310 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:45 smithi082 ceph-mon[36471]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:45.311 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:45.311+0000 7f1480ff9700 1 -- 172.21.15.82:0/239251839 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1484062680 con 0x7f146c0a1e00 2024-03-28T22:27:45.311 INFO:teuthology.orchestra.run.smithi082.stdout:Created osd(s) 1 on host 'smithi082' 2024-03-28T22:27:45.312 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:45.313+0000 7f1489b58700 1 -- 172.21.15.82:0/239251839 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f146c0a1e00 msgr2=0x7f146c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:27:45.312 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:45.313+0000 7f1489b58700 1 --2- 172.21.15.82:0/239251839 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f146c0a1e00 0x7f146c0a42c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1474006130 tx=0x7f14740060c0 comp rx=0 tx=0).stop 2024-03-28T22:27:45.313 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:45.313+0000 7f1489b58700 1 -- 172.21.15.82:0/239251839 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1484100cb0 msgr2=0x7f1484198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:27:45.313 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:45.313+0000 7f1489b58700 1 --2- 172.21.15.82:0/239251839 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1484100cb0 0x7f1484198970 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f147800c9b0 tx=0x7f147800cd70 comp rx=0 tx=0).stop 2024-03-28T22:27:45.313 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:45.313+0000 7f1489b58700 1 -- 172.21.15.82:0/239251839 shutdown_connections 2024-03-28T22:27:45.313 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:45.313+0000 7f1489b58700 1 --2- 172.21.15.82:0/239251839 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f146c0a1e00 0x7f146c0a42c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:27:45.313 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:45.313+0000 7f1489b58700 1 --2- 172.21.15.82:0/239251839 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1484103890 0x7f148419f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:27:45.313 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:45.313+0000 7f1489b58700 1 --2- 172.21.15.82:0/239251839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1484100310 0x7f1484198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:27:45.313 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:45.313+0000 7f1489b58700 1 --2- 172.21.15.82:0/239251839 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1484100cb0 0x7f1484198970 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:27:45.313 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:45.313+0000 7f1489b58700 1 -- 172.21.15.82:0/239251839 >> 172.21.15.82:0/239251839 conn(0x7f14840fbeb0 msgr2=0x7f14841068d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:27:45.313 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:45.313+0000 7f1489b58700 1 -- 172.21.15.82:0/239251839 shutdown_connections 2024-03-28T22:27:45.313 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:27:45.313+0000 7f1489b58700 1 -- 172.21.15.82:0/239251839 wait complete. 2024-03-28T22:27:45.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:45 smithi043 ceph-mon[31823]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:45.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:45 smithi172 ceph-mon[36566]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:46.197 DEBUG:teuthology.orchestra.run.smithi082:osd.1> sudo journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.1.service 2024-03-28T22:27:46.200 INFO:tasks.cephadm:Deploying osd.2 on smithi172 with /dev/vg_nvme/lv_4... 2024-03-28T22:27:46.200 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T22:27:46.447 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T22:27:46.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:46 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:46.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:46 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:46.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:46 smithi082 ceph-mon[36471]: from='osd.1 [v2:172.21.15.82:6800/3043263644,v1:172.21.15.82:6801/3043263644]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T22:27:46.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:46 smithi082 ceph-mon[36471]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T22:27:46.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:46 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:46.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:46 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:46.568 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:46 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:46.569 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:46 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:46.569 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:46 smithi172 ceph-mon[36566]: from='osd.1 [v2:172.21.15.82:6800/3043263644,v1:172.21.15.82:6801/3043263644]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T22:27:46.569 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:46 smithi172 ceph-mon[36566]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T22:27:46.569 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:46 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:46.569 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:46 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:46.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:46 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:46.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:46 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:46.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:46 smithi043 ceph-mon[31823]: from='osd.1 [v2:172.21.15.82:6800/3043263644,v1:172.21.15.82:6801/3043263644]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T22:27:46.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:46 smithi043 ceph-mon[31823]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T22:27:46.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:46 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:46.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:46 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:47.573 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:47 smithi172 ceph-mon[36566]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:47.573 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:47 smithi172 ceph-mon[36566]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T22:27:47.573 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:47 smithi172 ceph-mon[36566]: from='osd.1 [v2:172.21.15.82:6800/3043263644,v1:172.21.15.82:6801/3043263644]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-28T22:27:47.573 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:47 smithi172 ceph-mon[36566]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T22:27:47.573 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:47 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:27:47.573 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:47 smithi172 ceph-mon[36566]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-28T22:27:47.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:47 smithi082 ceph-mon[36471]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:47.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:47 smithi082 ceph-mon[36471]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T22:27:47.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:47 smithi082 ceph-mon[36471]: from='osd.1 [v2:172.21.15.82:6800/3043263644,v1:172.21.15.82:6801/3043263644]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-28T22:27:47.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:47 smithi082 ceph-mon[36471]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T22:27:47.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:47 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:27:47.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:47 smithi082 ceph-mon[36471]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-28T22:27:47.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:47 smithi043 ceph-mon[31823]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:47.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:47 smithi043 ceph-mon[31823]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T22:27:47.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:47 smithi043 ceph-mon[31823]: from='osd.1 [v2:172.21.15.82:6800/3043263644,v1:172.21.15.82:6801/3043263644]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-28T22:27:47.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:47 smithi043 ceph-mon[31823]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T22:27:47.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:47 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:27:47.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:47 smithi043 ceph-mon[31823]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-28T22:27:48.201 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 28 22:27:47 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-1[41055]: 2024-03-28T22:27:47.982+0000 7f750f9ea700 -1 osd.1 0 waiting for initial osdmap 2024-03-28T22:27:48.201 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 28 22:27:47 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-1[41055]: 2024-03-28T22:27:47.987+0000 7f75077d5700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T22:27:48.275 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T22:27:48.275 INFO:teuthology.orchestra.run.smithi172.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T22:27:48.275 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10+0 records in 2024-03-28T22:27:48.275 INFO:teuthology.orchestra.run.smithi172.stderr:10+0 records out 2024-03-28T22:27:48.276 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112796 s, 930 MB/s 2024-03-28T22:27:48.276 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping successful for: 2024-03-28T22:27:48.572 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:48 smithi172 ceph-mon[36566]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-03-28T22:27:48.572 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:48 smithi172 ceph-mon[36566]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T22:27:48.572 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:48 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:27:48.572 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:48 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:27:48.572 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:48 smithi172 ceph-mon[36566]: from='osd.1 ' entity='osd.1' 2024-03-28T22:27:48.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:48 smithi082 ceph-mon[36471]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-03-28T22:27:48.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:48 smithi082 ceph-mon[36471]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T22:27:48.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:48 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:27:48.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:48 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:27:48.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:48 smithi082 ceph-mon[36471]: from='osd.1 ' entity='osd.1' 2024-03-28T22:27:48.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:48 smithi043 ceph-mon[31823]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-03-28T22:27:48.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:48 smithi043 ceph-mon[31823]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T22:27:48.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:48 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:27:48.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:48 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:27:48.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:48 smithi043 ceph-mon[31823]: from='osd.1 ' entity='osd.1' 2024-03-28T22:27:48.884 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi172:vg_nvme/lv_4 2024-03-28T22:27:49.128 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T22:27:49.534 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:49 smithi172 ceph-mon[36566]: purged_snaps scrub starts 2024-03-28T22:27:49.534 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:49 smithi172 ceph-mon[36566]: purged_snaps scrub ok 2024-03-28T22:27:49.534 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:49 smithi172 ceph-mon[36566]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:49.535 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:49 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:27:49.535 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:49 smithi172 ceph-mon[36566]: osd.1 [v2:172.21.15.82:6800/3043263644,v1:172.21.15.82:6801/3043263644] boot 2024-03-28T22:27:49.535 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:49 smithi172 ceph-mon[36566]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T22:27:49.535 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:49 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:27:49.678 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:49 smithi082 ceph-mon[36471]: purged_snaps scrub starts 2024-03-28T22:27:49.678 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:49 smithi082 ceph-mon[36471]: purged_snaps scrub ok 2024-03-28T22:27:49.678 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:49 smithi082 ceph-mon[36471]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:49.678 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:49 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:27:49.678 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:49 smithi082 ceph-mon[36471]: osd.1 [v2:172.21.15.82:6800/3043263644,v1:172.21.15.82:6801/3043263644] boot 2024-03-28T22:27:49.678 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:49 smithi082 ceph-mon[36471]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T22:27:49.678 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:49 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:27:49.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:49 smithi043 ceph-mon[31823]: purged_snaps scrub starts 2024-03-28T22:27:49.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:49 smithi043 ceph-mon[31823]: purged_snaps scrub ok 2024-03-28T22:27:49.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:49 smithi043 ceph-mon[31823]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T22:27:49.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:49 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:27:49.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:49 smithi043 ceph-mon[31823]: osd.1 [v2:172.21.15.82:6800/3043263644,v1:172.21.15.82:6801/3043263644] boot 2024-03-28T22:27:49.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:49 smithi043 ceph-mon[31823]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T22:27:49.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:49 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:27:50.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.703+0000 7f438f94d700 1 -- 172.21.15.172:0/2952324094 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4388100580 msgr2=0x7f43881009a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:27:50.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.703+0000 7f438f94d700 1 --2- 172.21.15.172:0/2952324094 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4388100580 0x7f43881009a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4384009990 tx=0x7f4384009ca0 comp rx=0 tx=0).stop 2024-03-28T22:27:50.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.703+0000 7f438f94d700 1 -- 172.21.15.172:0/2952324094 shutdown_connections 2024-03-28T22:27:50.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.703+0000 7f438f94d700 1 --2- 172.21.15.172:0/2952324094 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4388103f00 0x7f438810a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:27:50.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.703+0000 7f438f94d700 1 --2- 172.21.15.172:0/2952324094 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4388100580 0x7f43881009a0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:27:50.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.703+0000 7f438f94d700 1 --2- 172.21.15.172:0/2952324094 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4388103560 0x7f43881039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:27:50.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.703+0000 7f438f94d700 1 -- 172.21.15.172:0/2952324094 >> 172.21.15.172:0/2952324094 conn(0x7f43880fbee0 msgr2=0x7f43880fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:27:50.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.703+0000 7f438f94d700 1 -- 172.21.15.172:0/2952324094 shutdown_connections 2024-03-28T22:27:50.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.703+0000 7f438f94d700 1 -- 172.21.15.172:0/2952324094 wait complete. 2024-03-28T22:27:50.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.703+0000 7f438f94d700 1 Processor -- start 2024-03-28T22:27:50.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.703+0000 7f438f94d700 1 -- start start 2024-03-28T22:27:50.704 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.703+0000 7f438f94d700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4388100580 0x7f4388102ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:27:50.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.703+0000 7f438f94d700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4388103560 0x7f4388101510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:27:50.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.703+0000 7f438f94d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4388103f00 0x7f4388101a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:27:50.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.703+0000 7f438f94d700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f438810df10 con 0x7f4388103560 2024-03-28T22:27:50.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.703+0000 7f438f94d700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f438810dd90 con 0x7f4388100580 2024-03-28T22:27:50.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.703+0000 7f438f94d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f438810e090 con 0x7f4388103f00 2024-03-28T22:27:50.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.704+0000 7f438deea700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4388103f00 0x7f4388101a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:27:50.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.704+0000 7f438deea700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4388103f00 0x7f4388101a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:53148/0 (socket says 172.21.15.172:53148) 2024-03-28T22:27:50.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.704+0000 7f438deea700 1 -- 172.21.15.172:0/2108382934 learned_addr learned my addr 172.21.15.172:0/2108382934 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-28T22:27:50.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.704+0000 7f438d6e9700 1 --2- 172.21.15.172:0/2108382934 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4388100580 0x7f4388102ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:27:50.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.704+0000 7f438deea700 1 -- 172.21.15.172:0/2108382934 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4388100580 msgr2=0x7f4388102ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:27:50.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.704+0000 7f438deea700 1 --2- 172.21.15.172:0/2108382934 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4388100580 0x7f4388102ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:27:50.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.704+0000 7f438deea700 1 -- 172.21.15.172:0/2108382934 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4388103560 msgr2=0x7f4388101510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:27:50.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.704+0000 7f438deea700 1 --2- 172.21.15.172:0/2108382934 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4388103560 0x7f4388101510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:27:50.705 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.704+0000 7f438deea700 1 -- 172.21.15.172:0/2108382934 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4388102300 con 0x7f4388103f00 2024-03-28T22:27:50.706 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.704+0000 7f438deea700 1 --2- 172.21.15.172:0/2108382934 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4388103f00 0x7f4388101a50 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f438000ef30 tx=0x7f438000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:27:50.706 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.704+0000 7f437e7fc700 1 -- 172.21.15.172:0/2108382934 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43800092e0 con 0x7f4388103f00 2024-03-28T22:27:50.706 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.704+0000 7f437e7fc700 1 -- 172.21.15.172:0/2108382934 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4380004bb0 con 0x7f4388103f00 2024-03-28T22:27:50.706 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.704+0000 7f438f94d700 1 -- 172.21.15.172:0/2108382934 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43881a2600 con 0x7f4388103f00 2024-03-28T22:27:50.706 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.704+0000 7f437e7fc700 1 -- 172.21.15.172:0/2108382934 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4380004280 con 0x7f4388103f00 2024-03-28T22:27:50.706 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.704+0000 7f438f94d700 1 -- 172.21.15.172:0/2108382934 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43881a2b10 con 0x7f4388103f00 2024-03-28T22:27:50.707 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.705+0000 7f438f94d700 1 -- 172.21.15.172:0/2108382934 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4388068f00 con 0x7f4388103f00 2024-03-28T22:27:50.707 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.707+0000 7f437e7fc700 1 -- 172.21.15.172:0/2108382934 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 15) ==== 136700+0+0 (secure 0 0 0) 0x7f438001b030 con 0x7f4388103f00 2024-03-28T22:27:50.707 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.707+0000 7f437e7fc700 1 --2- 172.21.15.172:0/2108382934 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f43740a1d80 0x7f43740a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:27:50.707 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.708+0000 7f437e7fc700 1 -- 172.21.15.172:0/2108382934 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f43800cca40 con 0x7f4388103f00 2024-03-28T22:27:50.707 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.708+0000 7f438d6e9700 1 --2- 172.21.15.172:0/2108382934 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f43740a1d80 0x7f43740a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:27:50.708 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.709+0000 7f438d6e9700 1 --2- 172.21.15.172:0/2108382934 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f43740a1d80 0x7f43740a4240 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f438400b7b0 tx=0x7f438401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:27:50.710 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.711+0000 7f437e7fc700 1 -- 172.21.15.172:0/2108382934 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4380022020 con 0x7f4388103f00 2024-03-28T22:27:50.916 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:27:50.917+0000 7f438f94d700 1 -- 172.21.15.172:0/2108382934 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f4388062740 con 0x7f43740a1d80 2024-03-28T22:27:51.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:50 smithi043 ceph-mon[31823]: Detected new or changed devices on smithi082 2024-03-28T22:27:51.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:50 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:51.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:50 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:51.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:50 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:27:51.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:50 smithi043 ceph-mon[31823]: Adjusting osd_memory_target on smithi082 to 17080M 2024-03-28T22:27:51.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:50 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:51.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:50 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:27:51.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:50 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:27:51.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:50 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:51.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:50 smithi043 ceph-mon[31823]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T22:27:51.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:50 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:27:51.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:50 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:27:51.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:50 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:27:51.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:50 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:51.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:50 smithi172 ceph-mon[36566]: Detected new or changed devices on smithi082 2024-03-28T22:27:51.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:50 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:51.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:50 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:51.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:50 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:27:51.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:50 smithi172 ceph-mon[36566]: Adjusting osd_memory_target on smithi082 to 17080M 2024-03-28T22:27:51.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:50 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:51.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:50 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:27:51.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:50 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:27:51.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:50 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:51.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:50 smithi172 ceph-mon[36566]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T22:27:51.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:50 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:27:51.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:50 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:27:51.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:50 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:27:51.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:50 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:51.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:50 smithi082 ceph-mon[36471]: Detected new or changed devices on smithi082 2024-03-28T22:27:51.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:50 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:51.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:50 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:51.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:50 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:27:51.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:50 smithi082 ceph-mon[36471]: Adjusting osd_memory_target on smithi082 to 17080M 2024-03-28T22:27:51.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:50 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:51.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:50 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:27:51.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:50 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:27:51.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:50 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:51.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:50 smithi082 ceph-mon[36471]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T22:27:51.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:50 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:27:51.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:50 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:27:51.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:50 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:27:51.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:50 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:27:51.835 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:51 smithi172 ceph-mon[36566]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:27:51.836 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:51 smithi172 ceph-mon[36566]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:27:51.836 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:51 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:27:51.836 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:51 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:27:51.836 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:51 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:27:52.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:51 smithi043 ceph-mon[31823]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:27:52.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:51 smithi043 ceph-mon[31823]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:27:52.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:51 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:27:52.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:51 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:27:52.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:51 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:27:52.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:51 smithi082 ceph-mon[36471]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:27:52.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:51 smithi082 ceph-mon[36471]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:27:52.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:51 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:27:52.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:51 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:27:52.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:51 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:27:54.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:53 smithi172 ceph-mon[36566]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:27:54.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:53 smithi172 ceph-mon[36566]: from='client.? 172.21.15.172:0/1003422846' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20632bbb-f07a-4ca1-b0c4-ff9d508e211c"}]: dispatch 2024-03-28T22:27:54.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:53 smithi172 ceph-mon[36566]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20632bbb-f07a-4ca1-b0c4-ff9d508e211c"}]: dispatch 2024-03-28T22:27:54.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:53 smithi172 ceph-mon[36566]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "20632bbb-f07a-4ca1-b0c4-ff9d508e211c"}]': finished 2024-03-28T22:27:54.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:53 smithi172 ceph-mon[36566]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T22:27:54.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:53 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:27:54.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:53 smithi082 ceph-mon[36471]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:27:54.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:53 smithi082 ceph-mon[36471]: from='client.? 172.21.15.172:0/1003422846' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20632bbb-f07a-4ca1-b0c4-ff9d508e211c"}]: dispatch 2024-03-28T22:27:54.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:53 smithi082 ceph-mon[36471]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20632bbb-f07a-4ca1-b0c4-ff9d508e211c"}]: dispatch 2024-03-28T22:27:54.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:53 smithi082 ceph-mon[36471]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "20632bbb-f07a-4ca1-b0c4-ff9d508e211c"}]': finished 2024-03-28T22:27:54.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:53 smithi082 ceph-mon[36471]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T22:27:54.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:53 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:27:54.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:53 smithi043 ceph-mon[31823]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:27:54.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:53 smithi043 ceph-mon[31823]: from='client.? 172.21.15.172:0/1003422846' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20632bbb-f07a-4ca1-b0c4-ff9d508e211c"}]: dispatch 2024-03-28T22:27:54.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:53 smithi043 ceph-mon[31823]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20632bbb-f07a-4ca1-b0c4-ff9d508e211c"}]: dispatch 2024-03-28T22:27:54.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:53 smithi043 ceph-mon[31823]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "20632bbb-f07a-4ca1-b0c4-ff9d508e211c"}]': finished 2024-03-28T22:27:54.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:53 smithi043 ceph-mon[31823]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T22:27:54.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:53 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:27:55.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:54 smithi172 ceph-mon[36566]: from='client.? 172.21.15.172:0/1419382002' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:27:55.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:54 smithi082 ceph-mon[36471]: from='client.? 172.21.15.172:0/1419382002' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:27:55.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:54 smithi043 ceph-mon[31823]: from='client.? 172.21.15.172:0/1419382002' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:27:56.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:55 smithi172 ceph-mon[36566]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:27:56.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:55 smithi082 ceph-mon[36471]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:27:56.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:55 smithi043 ceph-mon[31823]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:27:58.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:57 smithi172 ceph-mon[36566]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:27:58.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:57 smithi082 ceph-mon[36471]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:27:58.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:57 smithi043 ceph-mon[31823]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:27:59.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:27:59 smithi172 ceph-mon[36566]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:28:00.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:27:59 smithi082 ceph-mon[36471]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:28:00.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:27:59 smithi043 ceph-mon[31823]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:28:00.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:00 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T22:28:00.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:00 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:28:01.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:00 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T22:28:01.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:00 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:28:01.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:00 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T22:28:01.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:00 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:28:02.107 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:01 smithi043 ceph-mon[31823]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:28:02.107 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:01 smithi043 ceph-mon[31823]: Deploying daemon osd.2 on smithi172 2024-03-28T22:28:02.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:01 smithi172 ceph-mon[36566]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:28:02.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:01 smithi172 ceph-mon[36566]: Deploying daemon osd.2 on smithi172 2024-03-28T22:28:02.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:01 smithi082 ceph-mon[36471]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:28:02.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:01 smithi082 ceph-mon[36471]: Deploying daemon osd.2 on smithi172 2024-03-28T22:28:04.046 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:03 smithi172 ceph-mon[36566]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:28:04.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:03 smithi082 ceph-mon[36471]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:28:04.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:03 smithi043 ceph-mon[31823]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:28:05.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:05 smithi172 ceph-mon[36566]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:28:05.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:05 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:05.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:05 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:05.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:05 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:28:06.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:05 smithi082 ceph-mon[36471]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:28:06.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:05 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:06.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:05 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:06.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:05 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:28:06.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:05 smithi043 ceph-mon[31823]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:28:06.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:05 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:06.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:05 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:06.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:05 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:28:07.877 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:07 smithi172 ceph-mon[36566]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:28:08.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:07 smithi082 ceph-mon[36471]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:28:08.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:07 smithi043 ceph-mon[31823]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:28:09.204 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:28:09.205+0000 7f437e7fc700 1 -- 172.21.15.172:0/2108382934 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4388062740 con 0x7f43740a1d80 2024-03-28T22:28:09.205 INFO:teuthology.orchestra.run.smithi172.stdout:Created osd(s) 2 on host 'smithi172' 2024-03-28T22:28:09.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:28:09.207+0000 7f438f94d700 1 -- 172.21.15.172:0/2108382934 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f43740a1d80 msgr2=0x7f43740a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:28:09.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:28:09.207+0000 7f438f94d700 1 --2- 172.21.15.172:0/2108382934 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f43740a1d80 0x7f43740a4240 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f438400b7b0 tx=0x7f438401a040 comp rx=0 tx=0).stop 2024-03-28T22:28:09.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:28:09.207+0000 7f438f94d700 1 -- 172.21.15.172:0/2108382934 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4388103f00 msgr2=0x7f4388101a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:28:09.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:28:09.207+0000 7f438f94d700 1 --2- 172.21.15.172:0/2108382934 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4388103f00 0x7f4388101a50 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f438000ef30 tx=0x7f438000c3a0 comp rx=0 tx=0).stop 2024-03-28T22:28:09.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:28:09.207+0000 7f438f94d700 1 -- 172.21.15.172:0/2108382934 shutdown_connections 2024-03-28T22:28:09.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:28:09.207+0000 7f438f94d700 1 --2- 172.21.15.172:0/2108382934 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f43740a1d80 0x7f43740a4240 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:28:09.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:28:09.207+0000 7f438f94d700 1 --2- 172.21.15.172:0/2108382934 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4388103f00 0x7f4388101a50 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:28:09.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:28:09.207+0000 7f438f94d700 1 --2- 172.21.15.172:0/2108382934 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4388100580 0x7f4388102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:28:09.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:28:09.207+0000 7f438f94d700 1 --2- 172.21.15.172:0/2108382934 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4388103560 0x7f4388101510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:28:09.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:28:09.207+0000 7f438f94d700 1 -- 172.21.15.172:0/2108382934 >> 172.21.15.172:0/2108382934 conn(0x7f43880fbee0 msgr2=0x7f4388106800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:28:09.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:28:09.207+0000 7f438f94d700 1 -- 172.21.15.172:0/2108382934 shutdown_connections 2024-03-28T22:28:09.207 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:28:09.207+0000 7f438f94d700 1 -- 172.21.15.172:0/2108382934 wait complete. 2024-03-28T22:28:09.789 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:09 smithi172 ceph-mon[36566]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:28:09.789 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:09 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:09.789 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:09 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:09.789 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:09 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:28:09.789 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:09 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:28:09.789 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:09 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:09.789 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:09 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:09.789 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:09 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:09.789 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:09 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:28:09.820 DEBUG:teuthology.orchestra.run.smithi172:osd.2> sudo journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.2.service 2024-03-28T22:28:09.822 INFO:tasks.cephadm:Deploying osd.3 on smithi043 with /dev/vg_nvme/lv_3... 2024-03-28T22:28:09.822 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T22:28:09.916 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:09 smithi043 ceph-mon[31823]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:28:09.916 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:09 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:09.916 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:09 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:09.916 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:09 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:28:09.916 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:09 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:28:09.916 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:09 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:09.917 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:09 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:09.917 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:09 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:09.917 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:09 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:28:09.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:09 smithi082 ceph-mon[36471]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:28:09.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:09 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:09.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:09 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:09.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:09 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:28:09.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:09 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:28:09.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:09 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:09.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:09 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:09.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:09 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:09.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:09 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:28:10.287 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:28:10.736 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:10 smithi043 ceph-mon[31823]: from='osd.2 [v2:172.21.15.172:6800/4213953645,v1:172.21.15.172:6801/4213953645]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T22:28:10.736 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:10 smithi043 ceph-mon[31823]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T22:28:10.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:10 smithi172 ceph-mon[36566]: from='osd.2 [v2:172.21.15.172:6800/4213953645,v1:172.21.15.172:6801/4213953645]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T22:28:10.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:10 smithi172 ceph-mon[36566]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T22:28:10.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:10 smithi082 ceph-mon[36471]: from='osd.2 [v2:172.21.15.172:6800/4213953645,v1:172.21.15.172:6801/4213953645]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T22:28:10.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:10 smithi082 ceph-mon[36471]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T22:28:11.865 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:11 smithi043 ceph-mon[31823]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:28:11.865 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:11 smithi043 ceph-mon[31823]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T22:28:11.866 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:11 smithi043 ceph-mon[31823]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T22:28:11.866 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:11 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:28:11.866 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:11 smithi043 ceph-mon[31823]: from='osd.2 [v2:172.21.15.172:6800/4213953645,v1:172.21.15.172:6801/4213953645]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-28T22:28:11.866 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:11 smithi043 ceph-mon[31823]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-28T22:28:11.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:11 smithi082 ceph-mon[36471]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:28:11.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:11 smithi082 ceph-mon[36471]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T22:28:11.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:11 smithi082 ceph-mon[36471]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T22:28:11.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:11 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:28:11.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:11 smithi082 ceph-mon[36471]: from='osd.2 [v2:172.21.15.172:6800/4213953645,v1:172.21.15.172:6801/4213953645]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-28T22:28:11.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:11 smithi082 ceph-mon[36471]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-28T22:28:12.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:11 smithi172 ceph-mon[36566]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:28:12.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:11 smithi172 ceph-mon[36566]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T22:28:12.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:11 smithi172 ceph-mon[36566]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T22:28:12.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:11 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:28:12.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:11 smithi172 ceph-mon[36566]: from='osd.2 [v2:172.21.15.172:6800/4213953645,v1:172.21.15.172:6801/4213953645]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-28T22:28:12.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:11 smithi172 ceph-mon[36566]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-28T22:28:12.647 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 28 22:28:12 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-2[41722]: 2024-03-28T22:28:12.351+0000 7feed8acc700 -1 osd.2 0 waiting for initial osdmap 2024-03-28T22:28:12.647 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 28 22:28:12 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-2[41722]: 2024-03-28T22:28:12.356+0000 7feed08b7700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T22:28:12.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:12 smithi082 ceph-mon[36471]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-28T22:28:12.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:12 smithi082 ceph-mon[36471]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T22:28:12.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:12 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:28:12.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:12 smithi082 ceph-mon[36471]: from='osd.2 ' entity='osd.2' 2024-03-28T22:28:12.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:12 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:28:12.995 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:12 smithi172 ceph-mon[36566]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-28T22:28:12.995 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:12 smithi172 ceph-mon[36566]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T22:28:12.995 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:12 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:28:12.995 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:12 smithi172 ceph-mon[36566]: from='osd.2 ' entity='osd.2' 2024-03-28T22:28:12.995 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:12 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:28:13.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:12 smithi043 ceph-mon[31823]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-28T22:28:13.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:12 smithi043 ceph-mon[31823]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T22:28:13.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:12 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:28:13.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:12 smithi043 ceph-mon[31823]: from='osd.2 ' entity='osd.2' 2024-03-28T22:28:13.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:12 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:28:13.094 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T22:28:13.094 INFO:teuthology.orchestra.run.smithi043.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T22:28:13.095 INFO:teuthology.orchestra.run.smithi043.stderr: stderr: 10+0 records in 2024-03-28T22:28:13.095 INFO:teuthology.orchestra.run.smithi043.stderr:10+0 records out 2024-03-28T22:28:13.095 INFO:teuthology.orchestra.run.smithi043.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0423394 s, 248 MB/s 2024-03-28T22:28:13.095 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping successful for: 2024-03-28T22:28:13.881 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:13 smithi043 ceph-mon[31823]: purged_snaps scrub starts 2024-03-28T22:28:13.882 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:13 smithi043 ceph-mon[31823]: purged_snaps scrub ok 2024-03-28T22:28:13.882 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:13 smithi043 ceph-mon[31823]: pgmap v89: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:28:13.882 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:13 smithi043 ceph-mon[31823]: Detected new or changed devices on smithi172 2024-03-28T22:28:13.882 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:13 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:13.882 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:13 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:13.882 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:13 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:28:13.882 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:13 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:13.882 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:13 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:28:13.882 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:13 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:28:13.882 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:13 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:13.882 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:13 smithi043 ceph-mon[31823]: osd.2 [v2:172.21.15.172:6800/4213953645,v1:172.21.15.172:6801/4213953645] boot 2024-03-28T22:28:13.882 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:13 smithi043 ceph-mon[31823]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T22:28:13.882 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:13 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:28:13.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:13 smithi082 ceph-mon[36471]: purged_snaps scrub starts 2024-03-28T22:28:13.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:13 smithi082 ceph-mon[36471]: purged_snaps scrub ok 2024-03-28T22:28:13.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:13 smithi082 ceph-mon[36471]: pgmap v89: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:28:13.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:13 smithi082 ceph-mon[36471]: Detected new or changed devices on smithi172 2024-03-28T22:28:13.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:13 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:13.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:13 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:13.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:13 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:28:13.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:13 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:13.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:13 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:28:13.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:13 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:28:13.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:13 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:13.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:13 smithi082 ceph-mon[36471]: osd.2 [v2:172.21.15.172:6800/4213953645,v1:172.21.15.172:6801/4213953645] boot 2024-03-28T22:28:13.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:13 smithi082 ceph-mon[36471]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T22:28:13.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:13 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:28:13.957 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi043:vg_nvme/lv_3 2024-03-28T22:28:14.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:13 smithi172 ceph-mon[36566]: purged_snaps scrub starts 2024-03-28T22:28:14.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:13 smithi172 ceph-mon[36566]: purged_snaps scrub ok 2024-03-28T22:28:14.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:13 smithi172 ceph-mon[36566]: pgmap v89: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T22:28:14.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:13 smithi172 ceph-mon[36566]: Detected new or changed devices on smithi172 2024-03-28T22:28:14.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:13 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:14.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:13 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:14.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:13 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:28:14.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:13 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:14.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:13 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:28:14.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:13 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:28:14.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:13 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:14.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:13 smithi172 ceph-mon[36566]: osd.2 [v2:172.21.15.172:6800/4213953645,v1:172.21.15.172:6801/4213953645] boot 2024-03-28T22:28:14.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:13 smithi172 ceph-mon[36566]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T22:28:14.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:13 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:28:14.415 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:28:14.818 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:14 smithi043 ceph-mon[31823]: Adjusting osd_memory_target on smithi172 to 17080M 2024-03-28T22:28:15.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:14 smithi172 ceph-mon[36566]: Adjusting osd_memory_target on smithi172 to 17080M 2024-03-28T22:28:15.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:14 smithi082 ceph-mon[36471]: Adjusting osd_memory_target on smithi172 to 17080M 2024-03-28T22:28:16.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:15 smithi043 ceph-mon[31823]: pgmap v91: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:16.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:15 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T22:28:16.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:15 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T22:28:16.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:15 smithi043 ceph-mon[31823]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T22:28:16.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:15 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T22:28:16.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:15 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T22:28:16.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:15 smithi043 ceph-mon[31823]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T22:28:16.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:15 smithi172 ceph-mon[36566]: pgmap v91: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:16.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:15 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T22:28:16.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:15 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T22:28:16.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:15 smithi172 ceph-mon[36566]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T22:28:16.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:15 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T22:28:16.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:15 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T22:28:16.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:15 smithi172 ceph-mon[36566]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T22:28:16.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:15 smithi082 ceph-mon[36471]: pgmap v91: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:16.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:15 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T22:28:16.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:15 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T22:28:16.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:15 smithi082 ceph-mon[36471]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T22:28:16.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:15 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T22:28:16.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:15 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T22:28:16.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:15 smithi082 ceph-mon[36471]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T22:28:16.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.598+0000 7fba0e200700 1 -- 172.21.15.43:0/3530765190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba08074ca0 msgr2=0x7fba08075120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:28:16.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.598+0000 7fba0e200700 1 --2- 172.21.15.43:0/3530765190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba08074ca0 0x7fba08075120 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4009990 tx=0x7fb9f4009ca0 comp rx=0 tx=0).stop 2024-03-28T22:28:16.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.598+0000 7fba0e200700 1 -- 172.21.15.43:0/3530765190 shutdown_connections 2024-03-28T22:28:16.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.598+0000 7fba0e200700 1 --2- 172.21.15.43:0/3530765190 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba0810a080 0x7fba0810c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:28:16.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.598+0000 7fba0e200700 1 --2- 172.21.15.43:0/3530765190 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba08076270 0x7fba080746d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:28:16.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.598+0000 7fba0e200700 1 --2- 172.21.15.43:0/3530765190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba08074ca0 0x7fba08075120 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:28:16.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.598+0000 7fba0e200700 1 -- 172.21.15.43:0/3530765190 >> 172.21.15.43:0/3530765190 conn(0x7fba080fd750 msgr2=0x7fba080ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:28:16.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.598+0000 7fba0e200700 1 -- 172.21.15.43:0/3530765190 shutdown_connections 2024-03-28T22:28:16.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.598+0000 7fba0e200700 1 -- 172.21.15.43:0/3530765190 wait complete. 2024-03-28T22:28:16.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.600+0000 7fba0e200700 1 Processor -- start 2024-03-28T22:28:16.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.600+0000 7fba0e200700 1 -- start start 2024-03-28T22:28:16.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.600+0000 7fba0e200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba08074ca0 0x7fba0819a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:28:16.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.600+0000 7fba0e200700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba08076270 0x7fba0819ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:28:16.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.600+0000 7fba0e200700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba0810a080 0x7fba081a1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:28:16.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.600+0000 7fba0e200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fba0810ffe0 con 0x7fba08074ca0 2024-03-28T22:28:16.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.600+0000 7fba0e200700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fba0810fe60 con 0x7fba08076270 2024-03-28T22:28:16.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.600+0000 7fba0e200700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fba08110160 con 0x7fba0810a080 2024-03-28T22:28:16.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.600+0000 7fba0ce00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba08074ca0 0x7fba0819a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:28:16.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.600+0000 7fba07e00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba08076270 0x7fba0819ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:28:16.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.601+0000 7fba0ce00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba08074ca0 0x7fba0819a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45856/0 (socket says 172.21.15.43:45856) 2024-03-28T22:28:16.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.601+0000 7fba0ce00700 1 -- 172.21.15.43:0/3545233247 learned_addr learned my addr 172.21.15.43:0/3545233247 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:28:16.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.601+0000 7fba0d800700 1 --2- 172.21.15.43:0/3545233247 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba0810a080 0x7fba081a1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:28:16.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.601+0000 7fba0ce00700 1 -- 172.21.15.43:0/3545233247 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba0810a080 msgr2=0x7fba081a1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:28:16.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.601+0000 7fba0ce00700 1 --2- 172.21.15.43:0/3545233247 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba0810a080 0x7fba081a1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:28:16.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.601+0000 7fba0ce00700 1 -- 172.21.15.43:0/3545233247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba08076270 msgr2=0x7fba0819ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:28:16.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.601+0000 7fba0ce00700 1 --2- 172.21.15.43:0/3545233247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba08076270 0x7fba0819ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:28:16.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.601+0000 7fba0ce00700 1 -- 172.21.15.43:0/3545233247 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba081a2210 con 0x7fba08074ca0 2024-03-28T22:28:16.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.601+0000 7fba07e00700 1 --2- 172.21.15.43:0/3545233247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba08076270 0x7fba0819ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:28:16.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.601+0000 7fba0ce00700 1 --2- 172.21.15.43:0/3545233247 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba08074ca0 0x7fba0819a600 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc00ea60 tx=0x7fb9fc00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:28:16.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.602+0000 7fba05600700 1 -- 172.21.15.43:0/3545233247 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc00cbd0 con 0x7fba08074ca0 2024-03-28T22:28:16.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.602+0000 7fba0e200700 1 -- 172.21.15.43:0/3545233247 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba081a2500 con 0x7fba08074ca0 2024-03-28T22:28:16.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.602+0000 7fba05600700 1 -- 172.21.15.43:0/3545233247 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9fc004500 con 0x7fba08074ca0 2024-03-28T22:28:16.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.602+0000 7fba05600700 1 -- 172.21.15.43:0/3545233247 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc010430 con 0x7fba08074ca0 2024-03-28T22:28:16.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.602+0000 7fba0e200700 1 -- 172.21.15.43:0/3545233247 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba081a2a90 con 0x7fba08074ca0 2024-03-28T22:28:16.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.603+0000 7fba0e200700 1 -- 172.21.15.43:0/3545233247 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba08068f00 con 0x7fba08074ca0 2024-03-28T22:28:16.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.605+0000 7fba05600700 1 -- 172.21.15.43:0/3545233247 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 15) ==== 136700+0+0 (secure 0 0 0) 0x7fb9fc003ea0 con 0x7fba08074ca0 2024-03-28T22:28:16.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.607+0000 7fba05600700 1 --2- 172.21.15.43:0/3545233247 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fb9f00a1e00 0x7fb9f00a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:28:16.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.607+0000 7fba05600700 1 -- 172.21.15.43:0/3545233247 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3237+0+0 (secure 0 0 0) 0x7fb9fc0ccb80 con 0x7fba08074ca0 2024-03-28T22:28:16.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.607+0000 7fba07e00700 1 --2- 172.21.15.43:0/3545233247 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fb9f00a1e00 0x7fb9f00a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:28:16.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.608+0000 7fba07e00700 1 --2- 172.21.15.43:0/3545233247 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fb9f00a1e00 0x7fb9f00a42c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb9f400b7b0 tx=0x7fb9f4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:28:16.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.614+0000 7fba05600700 1 -- 172.21.15.43:0/3545233247 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb9fc09b450 con 0x7fba08074ca0 2024-03-28T22:28:16.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.759+0000 7fba05600700 1 -- 172.21.15.43:0/3545233247 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fb9fc0973c0 con 0x7fba08074ca0 2024-03-28T22:28:16.759 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 28 22:28:16 smithi043 sudo[48267]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T22:28:16.759 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 28 22:28:16 smithi043 sudo[48267]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T22:28:16.759 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 28 22:28:16 smithi043 sudo[48267]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T22:28:16.759 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 28 22:28:16 smithi043 sudo[48267]: pam_unix(sudo:session): session closed for user root 2024-03-28T22:28:16.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:16.855+0000 7fba0e200700 1 -- 172.21.15.43:0/3545233247 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fba08062680 con 0x7fb9f00a1e00 2024-03-28T22:28:17.517 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 28 22:28:17 smithi043 sudo[48357]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T22:28:17.517 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 28 22:28:17 smithi043 sudo[48357]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T22:28:17.517 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 28 22:28:17 smithi043 sudo[48357]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T22:28:17.517 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 28 22:28:17 smithi043 sudo[48357]: pam_unix(sudo:session): session closed for user root 2024-03-28T22:28:17.652 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 28 22:28:17 smithi082 sudo[45218]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T22:28:17.653 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 28 22:28:17 smithi082 sudo[45218]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T22:28:17.653 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 28 22:28:17 smithi082 sudo[45218]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T22:28:17.653 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 28 22:28:17 smithi082 sudo[45218]: pam_unix(sudo:session): session closed for user root 2024-03-28T22:28:17.950 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 28 22:28:17 smithi082 sudo[45221]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T22:28:17.950 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 28 22:28:17 smithi082 sudo[45221]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T22:28:17.950 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 28 22:28:17 smithi082 sudo[45221]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T22:28:17.950 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 28 22:28:17 smithi082 sudo[45221]: pam_unix(sudo:session): session closed for user root 2024-03-28T22:28:17.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:17 smithi082 ceph-mon[36471]: pgmap v94: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:17.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:17 smithi082 ceph-mon[36471]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-28T22:28:17.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:17 smithi082 ceph-mon[36471]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T22:28:17.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:17 smithi082 ceph-mon[36471]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:28:17.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:17 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:28:17.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:17 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:28:17.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:17 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:28:17.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:17 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:17.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:17 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:18.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:17 smithi043 ceph-mon[31823]: pgmap v94: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:18.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:17 smithi043 ceph-mon[31823]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-28T22:28:18.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:17 smithi043 ceph-mon[31823]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T22:28:18.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:17 smithi043 ceph-mon[31823]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:28:18.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:17 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:28:18.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:17 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:28:18.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:17 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:28:18.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:17 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:18.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:17 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:18.147 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 28 22:28:17 smithi172 sudo[45373]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T22:28:18.147 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 28 22:28:17 smithi172 sudo[45373]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T22:28:18.147 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 28 22:28:17 smithi172 sudo[45373]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T22:28:18.147 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 28 22:28:17 smithi172 sudo[45373]: pam_unix(sudo:session): session closed for user root 2024-03-28T22:28:18.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:17 smithi172 ceph-mon[36566]: pgmap v94: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:18.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:17 smithi172 ceph-mon[36566]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-28T22:28:18.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:17 smithi172 ceph-mon[36566]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T22:28:18.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:17 smithi172 ceph-mon[36566]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:28:18.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:17 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:28:18.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:17 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:28:18.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:17 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:28:18.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:17 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:18.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:17 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:18.647 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 28 22:28:18 smithi172 sudo[45376]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T22:28:18.647 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 28 22:28:18 smithi172 sudo[45376]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T22:28:18.647 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 28 22:28:18 smithi172 sudo[45376]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T22:28:18.648 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 28 22:28:18 smithi172 sudo[45376]: pam_unix(sudo:session): session closed for user root 2024-03-28T22:28:18.989 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:18 smithi043 sudo[48436]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T22:28:18.989 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:18 smithi043 sudo[48436]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T22:28:18.989 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:18 smithi043 sudo[48436]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T22:28:18.989 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:18 smithi043 sudo[48436]: pam_unix(sudo:session): session closed for user root 2024-03-28T22:28:19.594 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:19 smithi043 sudo[48460]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T22:28:19.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:19 smithi043 sudo[48460]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T22:28:19.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:19 smithi043 sudo[48460]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T22:28:19.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:19 smithi043 sudo[48460]: pam_unix(sudo:session): session closed for user root 2024-03-28T22:28:19.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:19 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:19.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:19 smithi043 ceph-mon[31823]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T22:28:19.658 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:19 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:19.658 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:19 smithi172 ceph-mon[36566]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T22:28:19.658 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:19 smithi172 sudo[45379]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T22:28:19.658 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:19 smithi172 sudo[45379]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T22:28:19.658 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:19 smithi172 sudo[45379]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T22:28:19.658 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:19 smithi172 sudo[45379]: pam_unix(sudo:session): session closed for user root 2024-03-28T22:28:19.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:19 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:19.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:19 smithi082 ceph-mon[36471]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T22:28:20.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:19 smithi172 sudo[45382]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T22:28:20.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:19 smithi172 sudo[45382]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T22:28:20.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:19 smithi172 sudo[45382]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T22:28:20.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:19 smithi172 sudo[45382]: pam_unix(sudo:session): session closed for user root 2024-03-28T22:28:20.414 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:20 smithi082 sudo[45233]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T22:28:20.414 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:20 smithi082 sudo[45233]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T22:28:20.414 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:20 smithi082 sudo[45233]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T22:28:20.414 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:20 smithi082 sudo[45233]: pam_unix(sudo:session): session closed for user root 2024-03-28T22:28:20.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:20 smithi082 sudo[45238]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T22:28:20.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:20 smithi082 sudo[45238]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T22:28:20.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:20 smithi082 sudo[45238]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T22:28:20.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:20 smithi082 sudo[45238]: pam_unix(sudo:session): session closed for user root 2024-03-28T22:28:20.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:20 smithi082 ceph-mon[36471]: pgmap v96: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:20.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:20 smithi082 ceph-mon[36471]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T22:28:20.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:20 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:28:20.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:20 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:28:20.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:20 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:28:20.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:20 smithi082 ceph-mon[36471]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T22:28:20.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:20 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:28:20.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:20 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:28:20.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:20 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:28:20.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:20 smithi082 ceph-mon[36471]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T22:28:20.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:20 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:28:20.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:20 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:28:20.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:20 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:28:20.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:20 smithi043 ceph-mon[31823]: pgmap v96: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:20.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:20 smithi043 ceph-mon[31823]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T22:28:20.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:20 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:28:20.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:20 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:28:20.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:20 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:28:20.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:20 smithi043 ceph-mon[31823]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T22:28:20.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:20 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:28:20.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:20 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:28:20.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:20 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:28:20.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:20 smithi043 ceph-mon[31823]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T22:28:20.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:20 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:28:20.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:20 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:28:20.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:20 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:28:20.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:20 smithi172 ceph-mon[36566]: pgmap v96: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:20.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:20 smithi172 ceph-mon[36566]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T22:28:20.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:20 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:28:20.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:20 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:28:20.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:20 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:28:20.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:20 smithi172 ceph-mon[36566]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T22:28:20.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:20 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:28:20.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:20 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:28:20.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:20 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:28:20.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:20 smithi172 ceph-mon[36566]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T22:28:20.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:20 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:28:20.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:20 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:28:20.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:20 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:28:21.671 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:21 smithi043 ceph-mon[31823]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T22:28:21.671 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:21 smithi043 ceph-mon[31823]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:21.671 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:21 smithi043 ceph-mon[31823]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T22:28:21.671 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:21 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3463356076' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a1f2312-4dbd-475b-8736-d5ebc8788e09"}]: dispatch 2024-03-28T22:28:21.671 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:21 smithi043 ceph-mon[31823]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a1f2312-4dbd-475b-8736-d5ebc8788e09"}]: dispatch 2024-03-28T22:28:21.671 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:21 smithi043 ceph-mon[31823]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9a1f2312-4dbd-475b-8736-d5ebc8788e09"}]': finished 2024-03-28T22:28:21.672 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:21 smithi043 ceph-mon[31823]: osdmap e22: 4 total, 3 up, 4 in 2024-03-28T22:28:21.672 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:21 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:28:21.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:21 smithi082 ceph-mon[36471]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T22:28:21.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:21 smithi082 ceph-mon[36471]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:21.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:21 smithi082 ceph-mon[36471]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T22:28:21.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:21 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3463356076' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a1f2312-4dbd-475b-8736-d5ebc8788e09"}]: dispatch 2024-03-28T22:28:21.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:21 smithi082 ceph-mon[36471]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a1f2312-4dbd-475b-8736-d5ebc8788e09"}]: dispatch 2024-03-28T22:28:21.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:21 smithi082 ceph-mon[36471]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9a1f2312-4dbd-475b-8736-d5ebc8788e09"}]': finished 2024-03-28T22:28:21.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:21 smithi082 ceph-mon[36471]: osdmap e22: 4 total, 3 up, 4 in 2024-03-28T22:28:21.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:21 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:28:21.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:21 smithi172 ceph-mon[36566]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T22:28:21.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:21 smithi172 ceph-mon[36566]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:21.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:21 smithi172 ceph-mon[36566]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T22:28:21.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:21 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3463356076' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a1f2312-4dbd-475b-8736-d5ebc8788e09"}]: dispatch 2024-03-28T22:28:21.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:21 smithi172 ceph-mon[36566]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9a1f2312-4dbd-475b-8736-d5ebc8788e09"}]: dispatch 2024-03-28T22:28:21.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:21 smithi172 ceph-mon[36566]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9a1f2312-4dbd-475b-8736-d5ebc8788e09"}]': finished 2024-03-28T22:28:21.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:21 smithi172 ceph-mon[36566]: osdmap e22: 4 total, 3 up, 4 in 2024-03-28T22:28:21.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:28:22.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:22 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2697250523' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:28:22.703 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:22 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2697250523' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:28:22.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:22 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2697250523' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:28:23.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:23 smithi082 ceph-mon[36471]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:23.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:23 smithi043 ceph-mon[31823]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:23.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:23 smithi172 ceph-mon[36566]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:25.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:25 smithi082 ceph-mon[36471]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:25.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:25 smithi043 ceph-mon[31823]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:25.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:25 smithi172 ceph-mon[36566]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:27.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:27 smithi082 ceph-mon[36471]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-03-28T22:28:27.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:27 smithi043 ceph-mon[31823]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-03-28T22:28:27.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:27 smithi172 ceph-mon[36566]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-03-28T22:28:29.674 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:29 smithi043 ceph-mon[31823]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-03-28T22:28:29.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:29 smithi082 ceph-mon[36471]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-03-28T22:28:29.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:29 smithi172 ceph-mon[36566]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-03-28T22:28:30.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:30 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T22:28:30.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:30 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:28:30.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:30 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T22:28:30.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:30 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:28:30.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:30 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T22:28:30.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:30 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:28:31.657 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:31 smithi043 ceph-mon[31823]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-28T22:28:31.657 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:31 smithi043 ceph-mon[31823]: Deploying daemon osd.3 on smithi043 2024-03-28T22:28:31.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:31 smithi172 ceph-mon[36566]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-28T22:28:31.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:31 smithi172 ceph-mon[36566]: Deploying daemon osd.3 on smithi043 2024-03-28T22:28:31.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:31 smithi082 ceph-mon[36471]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-28T22:28:31.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:31 smithi082 ceph-mon[36471]: Deploying daemon osd.3 on smithi043 2024-03-28T22:28:33.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:33 smithi043 ceph-mon[31823]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 909 B/s rd, 23 KiB/s wr, 2 op/s 2024-03-28T22:28:33.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:33 smithi172 ceph-mon[36566]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 909 B/s rd, 23 KiB/s wr, 2 op/s 2024-03-28T22:28:33.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:33 smithi082 ceph-mon[36471]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 909 B/s rd, 23 KiB/s wr, 2 op/s 2024-03-28T22:28:35.724 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:35 smithi172 ceph-mon[36566]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-28T22:28:35.744 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:35 smithi043 ceph-mon[31823]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-28T22:28:35.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:35 smithi082 ceph-mon[36471]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-28T22:28:37.751 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:37 smithi043 ceph-mon[31823]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-28T22:28:37.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:37 smithi172 ceph-mon[36566]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-28T22:28:37.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:37 smithi082 ceph-mon[36471]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-28T22:28:39.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:39 smithi082 ceph-mon[36471]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:39.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:39 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:39.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:39 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:39.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:39 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:28:39.951 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:39 smithi043 ceph-mon[31823]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:39.951 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:39 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:39.951 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:39 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:39.951 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:39 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:28:40.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:39 smithi172 ceph-mon[36566]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:40.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:39 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:40.150 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:39 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:40.150 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:39 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:28:41.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:41 smithi043 ceph-mon[31823]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:41.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:41 smithi082 ceph-mon[36471]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:42.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:41 smithi172 ceph-mon[36566]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:43.474 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:43 smithi043 ceph-mon[31823]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:43.474 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:43 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:43.474 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:43 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:43.474 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:43 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:28:43.475 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:43 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:28:43.475 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:43 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:43.475 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:43 smithi043 ceph-mon[31823]: from='osd.3 [v2:172.21.15.43:6810/1597280920,v1:172.21.15.43:6811/1597280920]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T22:28:43.600 INFO:teuthology.orchestra.run.smithi043.stdout:Created osd(s) 3 on host 'smithi043' 2024-03-28T22:28:43.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:43.600+0000 7fba05600700 1 -- 172.21.15.43:0/3545233247 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fba08062680 con 0x7fb9f00a1e00 2024-03-28T22:28:43.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:43.603+0000 7fba0e200700 1 -- 172.21.15.43:0/3545233247 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fb9f00a1e00 msgr2=0x7fb9f00a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:28:43.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:43.603+0000 7fba0e200700 1 --2- 172.21.15.43:0/3545233247 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fb9f00a1e00 0x7fb9f00a42c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb9f400b7b0 tx=0x7fb9f4005fd0 comp rx=0 tx=0).stop 2024-03-28T22:28:43.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:43.603+0000 7fba0e200700 1 -- 172.21.15.43:0/3545233247 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba08074ca0 msgr2=0x7fba0819a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:28:43.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:43.603+0000 7fba0e200700 1 --2- 172.21.15.43:0/3545233247 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba08074ca0 0x7fba0819a600 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc00ea60 tx=0x7fb9fc00ed70 comp rx=0 tx=0).stop 2024-03-28T22:28:43.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:43.603+0000 7fba0e200700 1 -- 172.21.15.43:0/3545233247 shutdown_connections 2024-03-28T22:28:43.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:43.603+0000 7fba0e200700 1 --2- 172.21.15.43:0/3545233247 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fb9f00a1e00 0x7fb9f00a42c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:28:43.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:43.603+0000 7fba0e200700 1 --2- 172.21.15.43:0/3545233247 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fba0810a080 0x7fba081a1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:28:43.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:43.603+0000 7fba0e200700 1 --2- 172.21.15.43:0/3545233247 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fba08074ca0 0x7fba0819a600 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:28:43.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:43.603+0000 7fba0e200700 1 --2- 172.21.15.43:0/3545233247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba08076270 0x7fba0819ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:28:43.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:43.603+0000 7fba0e200700 1 -- 172.21.15.43:0/3545233247 >> 172.21.15.43:0/3545233247 conn(0x7fba080fd750 msgr2=0x7fba08105f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:28:43.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:43.604+0000 7fba0e200700 1 -- 172.21.15.43:0/3545233247 shutdown_connections 2024-03-28T22:28:43.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:28:43.604+0000 7fba0e200700 1 -- 172.21.15.43:0/3545233247 wait complete. 2024-03-28T22:28:43.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:43 smithi172 ceph-mon[36566]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:43.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:43 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:43.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:43 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:43.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:43 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:28:43.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:43 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:28:43.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:43 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:43.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:43 smithi172 ceph-mon[36566]: from='osd.3 [v2:172.21.15.43:6810/1597280920,v1:172.21.15.43:6811/1597280920]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T22:28:43.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:43 smithi082 ceph-mon[36471]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:43.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:43 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:43.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:43 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:43.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:43 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:28:43.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:43 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:28:43.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:43 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:43.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:43 smithi082 ceph-mon[36471]: from='osd.3 [v2:172.21.15.43:6810/1597280920,v1:172.21.15.43:6811/1597280920]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T22:28:44.497 DEBUG:teuthology.orchestra.run.smithi043:osd.3> sudo journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.3.service 2024-03-28T22:28:44.504 INFO:tasks.cephadm:Deploying osd.4 on smithi082 with /dev/vg_nvme/lv_3... 2024-03-28T22:28:44.505 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T22:28:44.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:44 smithi082 ceph-mon[36471]: from='osd.3 [v2:172.21.15.43:6810/1597280920,v1:172.21.15.43:6811/1597280920]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T22:28:44.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:44 smithi082 ceph-mon[36471]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T22:28:44.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:44 smithi082 ceph-mon[36471]: from='osd.3 [v2:172.21.15.43:6810/1597280920,v1:172.21.15.43:6811/1597280920]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-28T22:28:44.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:44 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:28:44.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:44 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:44.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:44 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:44.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:44 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:28:44.747 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T22:28:44.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:44 smithi043 ceph-mon[31823]: from='osd.3 [v2:172.21.15.43:6810/1597280920,v1:172.21.15.43:6811/1597280920]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T22:28:44.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:44 smithi043 ceph-mon[31823]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T22:28:44.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:44 smithi043 ceph-mon[31823]: from='osd.3 [v2:172.21.15.43:6810/1597280920,v1:172.21.15.43:6811/1597280920]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-28T22:28:44.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:44 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:28:44.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:44 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:44.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:44 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:44.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:44 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:28:44.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:44 smithi172 ceph-mon[36566]: from='osd.3 [v2:172.21.15.43:6810/1597280920,v1:172.21.15.43:6811/1597280920]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T22:28:44.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:44 smithi172 ceph-mon[36566]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T22:28:44.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:44 smithi172 ceph-mon[36566]: from='osd.3 [v2:172.21.15.43:6810/1597280920,v1:172.21.15.43:6811/1597280920]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-28T22:28:44.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:44 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:28:44.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:44 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:44.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:44 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:44.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:44 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:28:45.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:45 smithi082 ceph-mon[36471]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:45.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:45 smithi082 ceph-mon[36471]: from='osd.3 [v2:172.21.15.43:6810/1597280920,v1:172.21.15.43:6811/1597280920]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-03-28T22:28:45.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:45 smithi082 ceph-mon[36471]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T22:28:45.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:45 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:28:45.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:45 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:28:45.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:45 smithi082 ceph-mon[36471]: from='osd.3 [v2:172.21.15.43:6810/1597280920,v1:172.21.15.43:6811/1597280920]' entity='osd.3' 2024-03-28T22:28:45.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:45 smithi043 ceph-mon[31823]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:45.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:45 smithi043 ceph-mon[31823]: from='osd.3 [v2:172.21.15.43:6810/1597280920,v1:172.21.15.43:6811/1597280920]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-03-28T22:28:45.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:45 smithi043 ceph-mon[31823]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T22:28:45.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:45 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:28:45.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:45 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:28:45.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:45 smithi043 ceph-mon[31823]: from='osd.3 [v2:172.21.15.43:6810/1597280920,v1:172.21.15.43:6811/1597280920]' entity='osd.3' 2024-03-28T22:28:45.517 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 28 22:28:45 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-3[50541]: 2024-03-28T22:28:45.222+0000 7ff870000700 -1 osd.3 0 waiting for initial osdmap 2024-03-28T22:28:45.518 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 28 22:28:45 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-3[50541]: 2024-03-28T22:28:45.242+0000 7ff865e00700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T22:28:45.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:45 smithi172 ceph-mon[36566]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:45.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:45 smithi172 ceph-mon[36566]: from='osd.3 [v2:172.21.15.43:6810/1597280920,v1:172.21.15.43:6811/1597280920]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-03-28T22:28:45.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:45 smithi172 ceph-mon[36566]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T22:28:45.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:45 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:28:45.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:45 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:28:45.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:45 smithi172 ceph-mon[36566]: from='osd.3 [v2:172.21.15.43:6810/1597280920,v1:172.21.15.43:6811/1597280920]' entity='osd.3' 2024-03-28T22:28:46.616 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T22:28:46.616 INFO:teuthology.orchestra.run.smithi082.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T22:28:46.616 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10+0 records in 2024-03-28T22:28:46.616 INFO:teuthology.orchestra.run.smithi082.stderr:10+0 records out 2024-03-28T22:28:46.616 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112371 s, 933 MB/s 2024-03-28T22:28:46.616 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping successful for: 2024-03-28T22:28:46.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:46 smithi082 ceph-mon[36471]: purged_snaps scrub starts 2024-03-28T22:28:46.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:46 smithi082 ceph-mon[36471]: purged_snaps scrub ok 2024-03-28T22:28:46.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:46 smithi082 ceph-mon[36471]: osd.3 [v2:172.21.15.43:6810/1597280920,v1:172.21.15.43:6811/1597280920] boot 2024-03-28T22:28:46.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:46 smithi082 ceph-mon[36471]: osdmap e25: 4 total, 4 up, 4 in 2024-03-28T22:28:46.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:46 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:28:46.741 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:46 smithi043 ceph-mon[31823]: purged_snaps scrub starts 2024-03-28T22:28:46.741 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:46 smithi043 ceph-mon[31823]: purged_snaps scrub ok 2024-03-28T22:28:46.741 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:46 smithi043 ceph-mon[31823]: osd.3 [v2:172.21.15.43:6810/1597280920,v1:172.21.15.43:6811/1597280920] boot 2024-03-28T22:28:46.741 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:46 smithi043 ceph-mon[31823]: osdmap e25: 4 total, 4 up, 4 in 2024-03-28T22:28:46.741 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:46 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:28:46.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:46 smithi172 ceph-mon[36566]: purged_snaps scrub starts 2024-03-28T22:28:46.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:46 smithi172 ceph-mon[36566]: purged_snaps scrub ok 2024-03-28T22:28:46.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:46 smithi172 ceph-mon[36566]: osd.3 [v2:172.21.15.43:6810/1597280920,v1:172.21.15.43:6811/1597280920] boot 2024-03-28T22:28:46.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:46 smithi172 ceph-mon[36566]: osdmap e25: 4 total, 4 up, 4 in 2024-03-28T22:28:46.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:46 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:28:47.260 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi082:vg_nvme/lv_3 2024-03-28T22:28:47.504 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T22:28:47.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:47 smithi082 ceph-mon[36471]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:47.533 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:47 smithi082 ceph-mon[36471]: osdmap e26: 4 total, 4 up, 4 in 2024-03-28T22:28:47.698 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:47 smithi043 ceph-mon[31823]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:47.698 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:47 smithi043 ceph-mon[31823]: osdmap e26: 4 total, 4 up, 4 in 2024-03-28T22:28:47.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:47 smithi172 ceph-mon[36566]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-28T22:28:47.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:47 smithi172 ceph-mon[36566]: osdmap e26: 4 total, 4 up, 4 in 2024-03-28T22:28:48.581 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:48 smithi082 ceph-mon[36471]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T22:28:48.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:48 smithi043 ceph-mon[31823]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T22:28:48.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:48 smithi172 ceph-mon[36566]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T22:28:49.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.095+0000 7f5a140e5700 1 -- 172.21.15.82:0/2545683499 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a0c105620 msgr2=0x7f5a0c107a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:28:49.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.095+0000 7f5a140e5700 1 --2- 172.21.15.82:0/2545683499 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a0c105620 0x7f5a0c107a10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f59fc009990 tx=0x7f59fc009ca0 comp rx=0 tx=0).stop 2024-03-28T22:28:49.097 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.095+0000 7f5a140e5700 1 -- 172.21.15.82:0/2545683499 shutdown_connections 2024-03-28T22:28:49.097 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.095+0000 7f5a140e5700 1 --2- 172.21.15.82:0/2545683499 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a0c107f50 0x7f5a0c10a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:28:49.097 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.095+0000 7f5a140e5700 1 --2- 172.21.15.82:0/2545683499 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a0c102cf0 0x7f5a0c1050e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:28:49.097 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.095+0000 7f5a140e5700 1 --2- 172.21.15.82:0/2545683499 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a0c105620 0x7f5a0c107a10 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:28:49.097 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.095+0000 7f5a140e5700 1 -- 172.21.15.82:0/2545683499 >> 172.21.15.82:0/2545683499 conn(0x7f5a0c0fc740 msgr2=0x7f5a0c0feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:28:49.097 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.095+0000 7f5a140e5700 1 -- 172.21.15.82:0/2545683499 shutdown_connections 2024-03-28T22:28:49.097 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.095+0000 7f5a140e5700 1 -- 172.21.15.82:0/2545683499 wait complete. 2024-03-28T22:28:49.097 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.096+0000 7f5a140e5700 1 Processor -- start 2024-03-28T22:28:49.097 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.096+0000 7f5a140e5700 1 -- start start 2024-03-28T22:28:49.097 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.096+0000 7f5a140e5700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a0c102cf0 0x7f5a0c198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:28:49.097 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.096+0000 7f5a140e5700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a0c105620 0x7f5a0c198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:28:49.098 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.096+0000 7f5a140e5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a0c107f50 0x7f5a0c19fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:28:49.098 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.096+0000 7f5a140e5700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0c10ded0 con 0x7f5a0c102cf0 2024-03-28T22:28:49.098 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.096+0000 7f5a140e5700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0c10dd50 con 0x7f5a0c105620 2024-03-28T22:28:49.098 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.096+0000 7f5a140e5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0c10e050 con 0x7f5a0c107f50 2024-03-28T22:28:49.098 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.096+0000 7f5a11680700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a0c105620 0x7f5a0c198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:28:49.098 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.096+0000 7f5a11680700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a0c105620 0x7f5a0c198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:49900/0 (socket says 172.21.15.82:49900) 2024-03-28T22:28:49.098 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.096+0000 7f5a11680700 1 -- 172.21.15.82:0/4203192234 learned_addr learned my addr 172.21.15.82:0/4203192234 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-03-28T22:28:49.098 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.096+0000 7f5a12682700 1 --2- 172.21.15.82:0/4203192234 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a0c107f50 0x7f5a0c19fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:28:49.098 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.096+0000 7f5a11e81700 1 --2- 172.21.15.82:0/4203192234 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a0c102cf0 0x7f5a0c198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:28:49.098 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.096+0000 7f5a11680700 1 -- 172.21.15.82:0/4203192234 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a0c107f50 msgr2=0x7f5a0c19fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:28:49.098 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.096+0000 7f5a11680700 1 --2- 172.21.15.82:0/4203192234 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a0c107f50 0x7f5a0c19fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:28:49.098 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.096+0000 7f5a11680700 1 -- 172.21.15.82:0/4203192234 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a0c102cf0 msgr2=0x7f5a0c198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:28:49.098 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.096+0000 7f5a11680700 1 --2- 172.21.15.82:0/4203192234 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a0c102cf0 0x7f5a0c198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:28:49.099 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.097+0000 7f5a11680700 1 -- 172.21.15.82:0/4203192234 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a0c1a0100 con 0x7f5a0c105620 2024-03-28T22:28:49.099 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.097+0000 7f5a12682700 1 --2- 172.21.15.82:0/4203192234 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a0c107f50 0x7f5a0c19fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:28:49.099 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.097+0000 7f5a11680700 1 --2- 172.21.15.82:0/4203192234 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a0c105620 0x7f5a0c198a50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f59fc005910 tx=0x7f59fc0047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:28:49.099 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.097+0000 7f5a02ffd700 1 -- 172.21.15.82:0/4203192234 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59fc027070 con 0x7f5a0c105620 2024-03-28T22:28:49.099 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.097+0000 7f5a140e5700 1 -- 172.21.15.82:0/4203192234 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a0c1a0390 con 0x7f5a0c105620 2024-03-28T22:28:49.099 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.097+0000 7f5a02ffd700 1 -- 172.21.15.82:0/4203192234 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59fc022470 con 0x7f5a0c105620 2024-03-28T22:28:49.099 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.097+0000 7f5a02ffd700 1 -- 172.21.15.82:0/4203192234 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59fc00f6a0 con 0x7f5a0c105620 2024-03-28T22:28:49.099 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.097+0000 7f5a140e5700 1 -- 172.21.15.82:0/4203192234 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a0c1a0860 con 0x7f5a0c105620 2024-03-28T22:28:49.100 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.098+0000 7f5a140e5700 1 -- 172.21.15.82:0/4203192234 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a0c068f00 con 0x7f5a0c105620 2024-03-28T22:28:49.100 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.099+0000 7f5a02ffd700 1 -- 172.21.15.82:0/4203192234 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f59fc02a070 con 0x7f5a0c105620 2024-03-28T22:28:49.100 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.100+0000 7f5a02ffd700 1 --2- 172.21.15.82:0/4203192234 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f59f80a1e10 0x7f59f80a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:28:49.100 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.100+0000 7f5a02ffd700 1 -- 172.21.15.82:0/4203192234 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3669+0+0 (secure 0 0 0) 0x7f59fc0cddb0 con 0x7f5a0c105620 2024-03-28T22:28:49.100 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.100+0000 7f5a11e81700 1 --2- 172.21.15.82:0/4203192234 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f59f80a1e10 0x7f59f80a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:28:49.100 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.101+0000 7f5a11e81700 1 --2- 172.21.15.82:0/4203192234 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f59f80a1e10 0x7f59f80a42d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5a08005950 tx=0x7f5a08005cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:28:49.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.103+0000 7f5a02ffd700 1 -- 172.21.15.82:0/4203192234 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f59fc013070 con 0x7f5a0c105620 2024-03-28T22:28:49.307 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:28:49.307+0000 7f5a140e5700 1 -- 172.21.15.82:0/4203192234 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f5a0c062680 con 0x7f59f80a1e10 2024-03-28T22:28:49.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:49 smithi043 ceph-mon[31823]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 130 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:28:49.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:49 smithi043 ceph-mon[31823]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T22:28:49.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:49 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:28:49.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:49 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:28:49.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:49 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:28:49.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:49 smithi172 ceph-mon[36566]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 130 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:28:49.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:49 smithi172 ceph-mon[36566]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T22:28:49.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:49 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:28:49.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:49 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:28:49.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:49 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:28:49.916 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:49 smithi082 ceph-mon[36471]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 130 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:28:49.916 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:49 smithi082 ceph-mon[36471]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T22:28:49.916 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:49 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:28:49.916 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:49 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:28:49.917 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:49 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:28:50.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:50 smithi082 ceph-mon[36471]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:28:50.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:50 smithi082 ceph-mon[36471]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T22:28:50.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:50 smithi043 ceph-mon[31823]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:28:50.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:50 smithi043 ceph-mon[31823]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T22:28:50.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:50 smithi172 ceph-mon[36566]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:28:50.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:50 smithi172 ceph-mon[36566]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T22:28:51.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:51 smithi172 ceph-mon[36566]: pgmap v120: 1 pgs: 1 peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T22:28:51.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:51 smithi172 ceph-mon[36566]: Detected new or changed devices on smithi043 2024-03-28T22:28:51.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:51 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:51.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:51 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:51.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:51 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:28:51.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:51 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:28:51.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:51 smithi172 ceph-mon[36566]: Adjusting osd_memory_target on smithi043 to 8547M 2024-03-28T22:28:51.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:51 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:51.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:51 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:28:51.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:51 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:28:51.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:51 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:51.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:51 smithi082 ceph-mon[36471]: pgmap v120: 1 pgs: 1 peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T22:28:51.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:51 smithi082 ceph-mon[36471]: Detected new or changed devices on smithi043 2024-03-28T22:28:51.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:51 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:51.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:51 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:51.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:51 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:28:51.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:51 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:28:51.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:51 smithi082 ceph-mon[36471]: Adjusting osd_memory_target on smithi043 to 8547M 2024-03-28T22:28:51.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:51 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:51.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:51 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:28:51.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:51 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:28:51.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:51 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:52.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:51 smithi043 ceph-mon[31823]: pgmap v120: 1 pgs: 1 peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T22:28:52.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:51 smithi043 ceph-mon[31823]: Detected new or changed devices on smithi043 2024-03-28T22:28:52.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:51 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:52.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:51 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:52.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:51 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:28:52.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:51 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:28:52.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:51 smithi043 ceph-mon[31823]: Adjusting osd_memory_target on smithi043 to 8547M 2024-03-28T22:28:52.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:51 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:52.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:51 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:28:52.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:51 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:28:52.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:51 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:28:52.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:52 smithi172 ceph-mon[36566]: from='client.? 172.21.15.82:0/4287159362' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "29140ead-7075-42b2-90c1-27cae0b29ec1"}]: dispatch 2024-03-28T22:28:52.918 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:52 smithi172 ceph-mon[36566]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "29140ead-7075-42b2-90c1-27cae0b29ec1"}]: dispatch 2024-03-28T22:28:52.918 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:52 smithi172 ceph-mon[36566]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "29140ead-7075-42b2-90c1-27cae0b29ec1"}]': finished 2024-03-28T22:28:52.918 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:52 smithi172 ceph-mon[36566]: osdmap e30: 5 total, 4 up, 5 in 2024-03-28T22:28:52.918 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:52 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:28:52.918 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:52 smithi172 ceph-mon[36566]: from='client.? 172.21.15.82:0/2274573231' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:28:52.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:52 smithi082 ceph-mon[36471]: from='client.? 172.21.15.82:0/4287159362' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "29140ead-7075-42b2-90c1-27cae0b29ec1"}]: dispatch 2024-03-28T22:28:52.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:52 smithi082 ceph-mon[36471]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "29140ead-7075-42b2-90c1-27cae0b29ec1"}]: dispatch 2024-03-28T22:28:52.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:52 smithi082 ceph-mon[36471]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "29140ead-7075-42b2-90c1-27cae0b29ec1"}]': finished 2024-03-28T22:28:52.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:52 smithi082 ceph-mon[36471]: osdmap e30: 5 total, 4 up, 5 in 2024-03-28T22:28:52.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:52 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:28:52.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:52 smithi082 ceph-mon[36471]: from='client.? 172.21.15.82:0/2274573231' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:28:53.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:52 smithi043 ceph-mon[31823]: from='client.? 172.21.15.82:0/4287159362' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "29140ead-7075-42b2-90c1-27cae0b29ec1"}]: dispatch 2024-03-28T22:28:53.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:52 smithi043 ceph-mon[31823]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "29140ead-7075-42b2-90c1-27cae0b29ec1"}]: dispatch 2024-03-28T22:28:53.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:52 smithi043 ceph-mon[31823]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "29140ead-7075-42b2-90c1-27cae0b29ec1"}]': finished 2024-03-28T22:28:53.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:52 smithi043 ceph-mon[31823]: osdmap e30: 5 total, 4 up, 5 in 2024-03-28T22:28:53.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:52 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:28:53.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:52 smithi043 ceph-mon[31823]: from='client.? 172.21.15.82:0/2274573231' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:28:54.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:53 smithi043 ceph-mon[31823]: pgmap v122: 1 pgs: 1 peering; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-03-28T22:28:54.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:53 smithi172 ceph-mon[36566]: pgmap v122: 1 pgs: 1 peering; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-03-28T22:28:54.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:53 smithi082 ceph-mon[36471]: pgmap v122: 1 pgs: 1 peering; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-03-28T22:28:56.189 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:55 smithi082 ceph-mon[36471]: pgmap v123: 1 pgs: 1 peering; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T22:28:56.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:55 smithi043 ceph-mon[31823]: pgmap v123: 1 pgs: 1 peering; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T22:28:56.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:55 smithi172 ceph-mon[36566]: pgmap v123: 1 pgs: 1 peering; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T22:28:58.122 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:57 smithi082 ceph-mon[36471]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-03-28T22:28:58.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:57 smithi043 ceph-mon[31823]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-03-28T22:28:58.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:57 smithi172 ceph-mon[36566]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-03-28T22:29:00.116 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:59 smithi082 ceph-mon[36471]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:29:00.116 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:59 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T22:29:00.116 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:28:59 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:29:00.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:59 smithi043 ceph-mon[31823]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:29:00.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:59 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T22:29:00.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:28:59 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:29:00.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:59 smithi172 ceph-mon[36566]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:29:00.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:59 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T22:29:00.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:28:59 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:29:01.070 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:01 smithi082 ceph-mon[36471]: Deploying daemon osd.4 on smithi082 2024-03-28T22:29:01.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:01 smithi043 ceph-mon[31823]: Deploying daemon osd.4 on smithi082 2024-03-28T22:29:01.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:01 smithi172 ceph-mon[36566]: Deploying daemon osd.4 on smithi082 2024-03-28T22:29:02.176 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:02 smithi082 ceph-mon[36471]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:29:02.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:02 smithi043 ceph-mon[31823]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:29:02.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:02 smithi172 ceph-mon[36566]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:29:04.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:04 smithi082 ceph-mon[36471]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:29:04.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:04 smithi043 ceph-mon[31823]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:29:04.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:04 smithi172 ceph-mon[36566]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:29:05.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:05 smithi082 ceph-mon[36471]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:29:05.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:05 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:05.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:05 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:05.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:05 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:29:06.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:05 smithi043 ceph-mon[31823]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:29:06.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:05 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:06.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:05 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:06.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:05 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:29:06.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:05 smithi172 ceph-mon[36566]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:29:06.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:05 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:06.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:05 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:06.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:05 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:29:07.836 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:07 smithi082 ceph-mon[36471]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:29:07.836 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:07 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:07.836 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:07 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:07.836 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:07 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:29:07.836 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:07 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:29:07.836 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:07 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:07.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:07 smithi172 ceph-mon[36566]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:29:07.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:07 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:07.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:07 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:07.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:07 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:29:07.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:07 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:29:07.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:07 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:08.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:07 smithi043 ceph-mon[31823]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:29:08.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:07 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:08.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:07 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:08.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:07 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:29:08.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:07 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:29:08.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:07 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:08.586 INFO:teuthology.orchestra.run.smithi082.stdout:Created osd(s) 4 on host 'smithi082' 2024-03-28T22:29:08.587 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:29:08.587+0000 7f5a02ffd700 1 -- 172.21.15.82:0/4203192234 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5a0c062680 con 0x7f59f80a1e10 2024-03-28T22:29:08.589 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:29:08.589+0000 7f5a140e5700 1 -- 172.21.15.82:0/4203192234 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f59f80a1e10 msgr2=0x7f59f80a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:29:08.589 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:29:08.589+0000 7f5a140e5700 1 --2- 172.21.15.82:0/4203192234 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f59f80a1e10 0x7f59f80a42d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5a08005950 tx=0x7f5a08005cf0 comp rx=0 tx=0).stop 2024-03-28T22:29:08.589 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:29:08.589+0000 7f5a140e5700 1 -- 172.21.15.82:0/4203192234 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a0c105620 msgr2=0x7f5a0c198a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:29:08.589 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:29:08.589+0000 7f5a140e5700 1 --2- 172.21.15.82:0/4203192234 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a0c105620 0x7f5a0c198a50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f59fc005910 tx=0x7f59fc0047b0 comp rx=0 tx=0).stop 2024-03-28T22:29:08.589 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:29:08.589+0000 7f5a140e5700 1 -- 172.21.15.82:0/4203192234 shutdown_connections 2024-03-28T22:29:08.589 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:29:08.589+0000 7f5a140e5700 1 --2- 172.21.15.82:0/4203192234 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f59f80a1e10 0x7f59f80a42d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:29:08.589 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:29:08.589+0000 7f5a140e5700 1 --2- 172.21.15.82:0/4203192234 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a0c107f50 0x7f5a0c19fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:29:08.589 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:29:08.589+0000 7f5a140e5700 1 --2- 172.21.15.82:0/4203192234 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a0c102cf0 0x7f5a0c198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:29:08.589 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:29:08.589+0000 7f5a140e5700 1 --2- 172.21.15.82:0/4203192234 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a0c105620 0x7f5a0c198a50 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:29:08.589 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:29:08.589+0000 7f5a140e5700 1 -- 172.21.15.82:0/4203192234 >> 172.21.15.82:0/4203192234 conn(0x7f5a0c0fc740 msgr2=0x7f5a0c103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:29:08.589 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:29:08.589+0000 7f5a140e5700 1 -- 172.21.15.82:0/4203192234 shutdown_connections 2024-03-28T22:29:08.590 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:29:08.589+0000 7f5a140e5700 1 -- 172.21.15.82:0/4203192234 wait complete. 2024-03-28T22:29:08.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:08 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:29:08.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:08 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:08.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:08 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:08.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:08 smithi082 ceph-mon[36471]: from='osd.4 [v2:172.21.15.82:6808/215451895,v1:172.21.15.82:6809/215451895]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T22:29:08.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:08 smithi082 ceph-mon[36471]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T22:29:09.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:08 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:29:09.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:08 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:09.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:08 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:09.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:08 smithi043 ceph-mon[31823]: from='osd.4 [v2:172.21.15.82:6808/215451895,v1:172.21.15.82:6809/215451895]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T22:29:09.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:08 smithi043 ceph-mon[31823]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T22:29:09.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:08 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:29:09.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:08 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:09.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:08 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:09.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:08 smithi172 ceph-mon[36566]: from='osd.4 [v2:172.21.15.82:6808/215451895,v1:172.21.15.82:6809/215451895]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T22:29:09.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:08 smithi172 ceph-mon[36566]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T22:29:09.310 DEBUG:teuthology.orchestra.run.smithi082:osd.4> sudo journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.4.service 2024-03-28T22:29:09.312 INFO:tasks.cephadm:Deploying osd.5 on smithi172 with /dev/vg_nvme/lv_3... 2024-03-28T22:29:09.312 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T22:29:09.556 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T22:29:09.943 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:09 smithi172 ceph-mon[36566]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:29:09.943 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:09 smithi172 ceph-mon[36566]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T22:29:09.943 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:09 smithi172 ceph-mon[36566]: from='osd.4 [v2:172.21.15.82:6808/215451895,v1:172.21.15.82:6809/215451895]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-28T22:29:09.943 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:09 smithi172 ceph-mon[36566]: osdmap e31: 5 total, 4 up, 5 in 2024-03-28T22:29:09.943 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:09 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:29:09.943 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:09 smithi172 ceph-mon[36566]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-28T22:29:09.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:09 smithi082 ceph-mon[36471]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:29:09.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:09 smithi082 ceph-mon[36471]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T22:29:09.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:09 smithi082 ceph-mon[36471]: from='osd.4 [v2:172.21.15.82:6808/215451895,v1:172.21.15.82:6809/215451895]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-28T22:29:09.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:09 smithi082 ceph-mon[36471]: osdmap e31: 5 total, 4 up, 5 in 2024-03-28T22:29:09.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:09 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:29:09.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:09 smithi082 ceph-mon[36471]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-28T22:29:10.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:09 smithi043 ceph-mon[31823]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:29:10.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:09 smithi043 ceph-mon[31823]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T22:29:10.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:09 smithi043 ceph-mon[31823]: from='osd.4 [v2:172.21.15.82:6808/215451895,v1:172.21.15.82:6809/215451895]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-28T22:29:10.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:09 smithi043 ceph-mon[31823]: osdmap e31: 5 total, 4 up, 5 in 2024-03-28T22:29:10.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:09 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:29:10.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:09 smithi043 ceph-mon[31823]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-28T22:29:10.804 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 28 22:29:10 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-4[48035]: 2024-03-28T22:29:10.541+0000 7f3f1ea68700 -1 osd.4 0 waiting for initial osdmap 2024-03-28T22:29:10.804 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 28 22:29:10 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-4[48035]: 2024-03-28T22:29:10.547+0000 7f3f1a85b700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T22:29:11.650 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T22:29:11.650 INFO:teuthology.orchestra.run.smithi172.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T22:29:11.650 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10+0 records in 2024-03-28T22:29:11.650 INFO:teuthology.orchestra.run.smithi172.stderr:10+0 records out 2024-03-28T22:29:11.650 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113148 s, 927 MB/s 2024-03-28T22:29:11.651 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping successful for: 2024-03-28T22:29:11.651 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:11 smithi172 ceph-mon[36566]: purged_snaps scrub starts 2024-03-28T22:29:11.651 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:11 smithi172 ceph-mon[36566]: purged_snaps scrub ok 2024-03-28T22:29:11.651 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:11 smithi172 ceph-mon[36566]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:29:11.651 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:11 smithi172 ceph-mon[36566]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-03-28T22:29:11.651 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:11 smithi172 ceph-mon[36566]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T22:29:11.651 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:11 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:29:11.651 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:11 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:29:11.931 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:11 smithi082 ceph-mon[36471]: purged_snaps scrub starts 2024-03-28T22:29:11.931 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:11 smithi082 ceph-mon[36471]: purged_snaps scrub ok 2024-03-28T22:29:11.931 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:11 smithi082 ceph-mon[36471]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:29:11.931 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:11 smithi082 ceph-mon[36471]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-03-28T22:29:11.931 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:11 smithi082 ceph-mon[36471]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T22:29:11.931 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:11 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:29:11.931 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:11 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:29:12.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:11 smithi043 ceph-mon[31823]: purged_snaps scrub starts 2024-03-28T22:29:12.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:11 smithi043 ceph-mon[31823]: purged_snaps scrub ok 2024-03-28T22:29:12.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:11 smithi043 ceph-mon[31823]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-03-28T22:29:12.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:11 smithi043 ceph-mon[31823]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-03-28T22:29:12.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:11 smithi043 ceph-mon[31823]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T22:29:12.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:11 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:29:12.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:11 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:29:12.289 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi172:vg_nvme/lv_3 2024-03-28T22:29:12.525 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T22:29:12.819 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:12 smithi172 ceph-mon[36566]: osd.4 [v2:172.21.15.82:6808/215451895,v1:172.21.15.82:6809/215451895] boot 2024-03-28T22:29:12.819 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:12 smithi172 ceph-mon[36566]: osdmap e33: 5 total, 5 up, 5 in 2024-03-28T22:29:12.819 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:12 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:29:12.819 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:12 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:12.819 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:12 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:12.819 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:12 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:29:12.819 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:12 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:29:12.820 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:12 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:12.820 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:12 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:29:12.820 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:12 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:29:12.820 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:12 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:12.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:12 smithi082 ceph-mon[36471]: osd.4 [v2:172.21.15.82:6808/215451895,v1:172.21.15.82:6809/215451895] boot 2024-03-28T22:29:12.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:12 smithi082 ceph-mon[36471]: osdmap e33: 5 total, 5 up, 5 in 2024-03-28T22:29:12.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:12 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:29:12.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:12 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:12.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:12 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:12.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:12 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:29:12.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:12 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:29:12.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:12 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:12.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:12 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:29:12.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:12 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:29:12.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:12 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:13.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:12 smithi043 ceph-mon[31823]: osd.4 [v2:172.21.15.82:6808/215451895,v1:172.21.15.82:6809/215451895] boot 2024-03-28T22:29:13.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:12 smithi043 ceph-mon[31823]: osdmap e33: 5 total, 5 up, 5 in 2024-03-28T22:29:13.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:12 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:29:13.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:12 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:13.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:12 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:13.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:12 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:29:13.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:12 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:29:13.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:12 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:13.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:12 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:29:13.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:12 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:29:13.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:12 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:13.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:13 smithi172 ceph-mon[36566]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:13.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:13 smithi172 ceph-mon[36566]: Detected new or changed devices on smithi082 2024-03-28T22:29:13.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:13 smithi172 ceph-mon[36566]: Adjusting osd_memory_target on smithi082 to 8540M 2024-03-28T22:29:13.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:13 smithi172 ceph-mon[36566]: osdmap e34: 5 total, 5 up, 5 in 2024-03-28T22:29:13.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:13 smithi082 ceph-mon[36471]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:13.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:13 smithi082 ceph-mon[36471]: Detected new or changed devices on smithi082 2024-03-28T22:29:13.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:13 smithi082 ceph-mon[36471]: Adjusting osd_memory_target on smithi082 to 8540M 2024-03-28T22:29:13.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:13 smithi082 ceph-mon[36471]: osdmap e34: 5 total, 5 up, 5 in 2024-03-28T22:29:14.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:13 smithi043 ceph-mon[31823]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:14.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:13 smithi043 ceph-mon[31823]: Detected new or changed devices on smithi082 2024-03-28T22:29:14.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:13 smithi043 ceph-mon[31823]: Adjusting osd_memory_target on smithi082 to 8540M 2024-03-28T22:29:14.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:13 smithi043 ceph-mon[31823]: osdmap e34: 5 total, 5 up, 5 in 2024-03-28T22:29:14.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.128+0000 7fac1c6c0700 1 -- 172.21.15.172:0/1084924476 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac14102d80 msgr2=0x7fac14105170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:29:14.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.128+0000 7fac1c6c0700 1 --2- 172.21.15.172:0/1084924476 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac14102d80 0x7fac14105170 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fac10009990 tx=0x7fac10009ca0 comp rx=0 tx=0).stop 2024-03-28T22:29:14.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.128+0000 7fac1c6c0700 1 -- 172.21.15.172:0/1084924476 shutdown_connections 2024-03-28T22:29:14.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.128+0000 7fac1c6c0700 1 --2- 172.21.15.172:0/1084924476 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fac14107fe0 0x7fac1410a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:29:14.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.128+0000 7fac1c6c0700 1 --2- 172.21.15.172:0/1084924476 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac14102d80 0x7fac14105170 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:29:14.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.128+0000 7fac1c6c0700 1 --2- 172.21.15.172:0/1084924476 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac141056b0 0x7fac14107aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:29:14.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.128+0000 7fac1c6c0700 1 -- 172.21.15.172:0/1084924476 >> 172.21.15.172:0/1084924476 conn(0x7fac140fc790 msgr2=0x7fac140febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:29:14.130 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.128+0000 7fac1c6c0700 1 -- 172.21.15.172:0/1084924476 shutdown_connections 2024-03-28T22:29:14.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.128+0000 7fac1c6c0700 1 -- 172.21.15.172:0/1084924476 wait complete. 2024-03-28T22:29:14.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.128+0000 7fac1c6c0700 1 Processor -- start 2024-03-28T22:29:14.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.128+0000 7fac1c6c0700 1 -- start start 2024-03-28T22:29:14.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.128+0000 7fac1c6c0700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fac14102d80 0x7fac141984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:29:14.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.128+0000 7fac1c6c0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac141056b0 0x7fac14198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:29:14.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.128+0000 7fac1c6c0700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac14107fe0 0x7fac1419faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:29:14.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.128+0000 7fac1c6c0700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fac1410ded0 con 0x7fac14107fe0 2024-03-28T22:29:14.131 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.128+0000 7fac1c6c0700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fac1410dd50 con 0x7fac14102d80 2024-03-28T22:29:14.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.128+0000 7fac1c6c0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fac1410e050 con 0x7fac141056b0 2024-03-28T22:29:14.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.129+0000 7fac19c5b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac141056b0 0x7fac14198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:29:14.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.129+0000 7fac19c5b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac141056b0 0x7fac14198a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:35994/0 (socket says 172.21.15.172:35994) 2024-03-28T22:29:14.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.129+0000 7fac19c5b700 1 -- 172.21.15.172:0/3579389426 learned_addr learned my addr 172.21.15.172:0/3579389426 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-28T22:29:14.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.129+0000 7fac1a45c700 1 --2- 172.21.15.172:0/3579389426 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fac14102d80 0x7fac141984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:29:14.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.129+0000 7fac19c5b700 1 -- 172.21.15.172:0/3579389426 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fac14102d80 msgr2=0x7fac141984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:29:14.132 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.129+0000 7fac19c5b700 1 --2- 172.21.15.172:0/3579389426 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fac14102d80 0x7fac141984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:29:14.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.129+0000 7fac19c5b700 1 -- 172.21.15.172:0/3579389426 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac14107fe0 msgr2=0x7fac1419faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:29:14.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.129+0000 7fac19c5b700 1 --2- 172.21.15.172:0/3579389426 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac14107fe0 0x7fac1419faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:29:14.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.129+0000 7fac19c5b700 1 -- 172.21.15.172:0/3579389426 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac141a0140 con 0x7fac141056b0 2024-03-28T22:29:14.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.129+0000 7fac19c5b700 1 --2- 172.21.15.172:0/3579389426 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac141056b0 0x7fac14198a20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fac0400b560 tx=0x7fac0400b920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:29:14.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.129+0000 7fac0b7fe700 1 -- 172.21.15.172:0/3579389426 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac04015020 con 0x7fac141056b0 2024-03-28T22:29:14.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.129+0000 7fac1c6c0700 1 -- 172.21.15.172:0/3579389426 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac141a0430 con 0x7fac141056b0 2024-03-28T22:29:14.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.129+0000 7fac0b7fe700 1 -- 172.21.15.172:0/3579389426 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac04011d10 con 0x7fac141056b0 2024-03-28T22:29:14.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.129+0000 7fac0b7fe700 1 -- 172.21.15.172:0/3579389426 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac0400f2e0 con 0x7fac141056b0 2024-03-28T22:29:14.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.129+0000 7fac1c6c0700 1 -- 172.21.15.172:0/3579389426 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac141a09c0 con 0x7fac141056b0 2024-03-28T22:29:14.133 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.130+0000 7fac1c6c0700 1 -- 172.21.15.172:0/3579389426 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac14068f00 con 0x7fac141056b0 2024-03-28T22:29:14.134 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.131+0000 7fac0b7fe700 1 -- 172.21.15.172:0/3579389426 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fac04018020 con 0x7fac141056b0 2024-03-28T22:29:14.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.132+0000 7fac0b7fe700 1 --2- 172.21.15.172:0/3579389426 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fac000a1e10 0x7fac000a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:29:14.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.132+0000 7fac0b7fe700 1 -- 172.21.15.172:0/3579389426 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7fac040cc550 con 0x7fac141056b0 2024-03-28T22:29:14.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.132+0000 7fac1a45c700 1 --2- 172.21.15.172:0/3579389426 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fac000a1e10 0x7fac000a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:29:14.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.133+0000 7fac1a45c700 1 --2- 172.21.15.172:0/3579389426 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fac000a1e10 0x7fac000a42d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fac1000b5c0 tx=0x7fac10005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:29:14.135 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.136+0000 7fac0b7fe700 1 -- 172.21.15.172:0/3579389426 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fac0401f020 con 0x7fac141056b0 2024-03-28T22:29:14.337 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:14.337+0000 7fac1c6c0700 1 -- 172.21.15.172:0/3579389426 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fac14062680 con 0x7fac000a1e10 2024-03-28T22:29:14.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:14 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:29:14.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:14 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:29:14.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:14 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:29:14.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:14 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:29:14.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:14 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:29:14.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:14 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:29:15.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:14 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:29:15.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:14 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:29:15.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:14 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:29:15.854 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:15 smithi172 ceph-mon[36566]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:15.854 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:15 smithi172 ceph-mon[36566]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:29:15.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:15 smithi082 ceph-mon[36471]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:15.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:15 smithi082 ceph-mon[36471]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:29:16.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:15 smithi043 ceph-mon[31823]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:16.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:15 smithi043 ceph-mon[31823]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:29:17.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:17 smithi172 ceph-mon[36566]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:17.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:17 smithi172 ceph-mon[36566]: from='client.? 172.21.15.172:0/395219844' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59644c61-93dc-4536-b2ad-0256b24e5cb6"}]: dispatch 2024-03-28T22:29:17.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:17 smithi172 ceph-mon[36566]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59644c61-93dc-4536-b2ad-0256b24e5cb6"}]: dispatch 2024-03-28T22:29:17.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:17 smithi172 ceph-mon[36566]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59644c61-93dc-4536-b2ad-0256b24e5cb6"}]': finished 2024-03-28T22:29:17.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:17 smithi172 ceph-mon[36566]: osdmap e35: 6 total, 5 up, 6 in 2024-03-28T22:29:17.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:17 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:29:17.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:17 smithi082 ceph-mon[36471]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:17.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:17 smithi082 ceph-mon[36471]: from='client.? 172.21.15.172:0/395219844' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59644c61-93dc-4536-b2ad-0256b24e5cb6"}]: dispatch 2024-03-28T22:29:17.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:17 smithi082 ceph-mon[36471]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59644c61-93dc-4536-b2ad-0256b24e5cb6"}]: dispatch 2024-03-28T22:29:17.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:17 smithi082 ceph-mon[36471]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59644c61-93dc-4536-b2ad-0256b24e5cb6"}]': finished 2024-03-28T22:29:17.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:17 smithi082 ceph-mon[36471]: osdmap e35: 6 total, 5 up, 6 in 2024-03-28T22:29:17.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:17 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:29:18.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:17 smithi043 ceph-mon[31823]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:18.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:17 smithi043 ceph-mon[31823]: from='client.? 172.21.15.172:0/395219844' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59644c61-93dc-4536-b2ad-0256b24e5cb6"}]: dispatch 2024-03-28T22:29:18.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:17 smithi043 ceph-mon[31823]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59644c61-93dc-4536-b2ad-0256b24e5cb6"}]: dispatch 2024-03-28T22:29:18.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:17 smithi043 ceph-mon[31823]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59644c61-93dc-4536-b2ad-0256b24e5cb6"}]': finished 2024-03-28T22:29:18.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:17 smithi043 ceph-mon[31823]: osdmap e35: 6 total, 5 up, 6 in 2024-03-28T22:29:18.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:17 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:29:18.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:18 smithi172 ceph-mon[36566]: from='client.? 172.21.15.172:0/3464341473' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:29:18.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:18 smithi082 ceph-mon[36471]: from='client.? 172.21.15.172:0/3464341473' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:29:19.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:18 smithi043 ceph-mon[31823]: from='client.? 172.21.15.172:0/3464341473' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:29:19.851 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:19 smithi082 ceph-mon[36471]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:19.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:19 smithi172 ceph-mon[36566]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:20.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:19 smithi043 ceph-mon[31823]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:21.847 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:21 smithi172 ceph-mon[36566]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:21.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:21 smithi082 ceph-mon[36471]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:22.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:21 smithi043 ceph-mon[31823]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:23.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:23 smithi172 ceph-mon[36566]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:23.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:23 smithi082 ceph-mon[36471]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:24.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:23 smithi043 ceph-mon[31823]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:24.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:24 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T22:29:24.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:24 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:29:24.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:24 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T22:29:24.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:24 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:29:24.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:24 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T22:29:24.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:24 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:29:25.870 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:25 smithi172 ceph-mon[36566]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:25.871 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:25 smithi172 ceph-mon[36566]: Deploying daemon osd.5 on smithi172 2024-03-28T22:29:25.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:25 smithi082 ceph-mon[36471]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:25.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:25 smithi082 ceph-mon[36471]: Deploying daemon osd.5 on smithi172 2024-03-28T22:29:26.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:25 smithi043 ceph-mon[31823]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:26.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:25 smithi043 ceph-mon[31823]: Deploying daemon osd.5 on smithi172 2024-03-28T22:29:27.808 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:27 smithi172 ceph-mon[36566]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:27.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:27 smithi082 ceph-mon[36471]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:28.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:27 smithi043 ceph-mon[31823]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:29.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:29 smithi172 ceph-mon[36566]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:29.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:29 smithi082 ceph-mon[36471]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:30.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:29 smithi043 ceph-mon[31823]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:30.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:30 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:30.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:30 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:30.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:30 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:29:31.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:30 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:31.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:30 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:31.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:30 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:29:31.140 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:30 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:31.140 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:30 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:31.140 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:30 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:29:32.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:31 smithi043 ceph-mon[31823]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:32.029 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:31 smithi172 ceph-mon[36566]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:32.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:31 smithi082 ceph-mon[36471]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:33.568 INFO:teuthology.orchestra.run.smithi172.stdout:Created osd(s) 5 on host 'smithi172' 2024-03-28T22:29:33.568 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:33.568+0000 7fac0b7fe700 1 -- 172.21.15.172:0/3579389426 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fac14062680 con 0x7fac000a1e10 2024-03-28T22:29:33.571 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:33.571+0000 7fac1c6c0700 1 -- 172.21.15.172:0/3579389426 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fac000a1e10 msgr2=0x7fac000a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:29:33.571 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:33.571+0000 7fac1c6c0700 1 --2- 172.21.15.172:0/3579389426 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fac000a1e10 0x7fac000a42d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fac1000b5c0 tx=0x7fac10005fb0 comp rx=0 tx=0).stop 2024-03-28T22:29:33.571 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:33.571+0000 7fac1c6c0700 1 -- 172.21.15.172:0/3579389426 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac141056b0 msgr2=0x7fac14198a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:29:33.571 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:33.571+0000 7fac1c6c0700 1 --2- 172.21.15.172:0/3579389426 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac141056b0 0x7fac14198a20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fac0400b560 tx=0x7fac0400b920 comp rx=0 tx=0).stop 2024-03-28T22:29:33.571 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:33.571+0000 7fac1c6c0700 1 -- 172.21.15.172:0/3579389426 shutdown_connections 2024-03-28T22:29:33.571 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:33.571+0000 7fac1c6c0700 1 --2- 172.21.15.172:0/3579389426 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fac000a1e10 0x7fac000a42d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:29:33.571 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:33.571+0000 7fac1c6c0700 1 --2- 172.21.15.172:0/3579389426 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fac14107fe0 0x7fac1419faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:29:33.571 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:33.571+0000 7fac1c6c0700 1 --2- 172.21.15.172:0/3579389426 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fac14102d80 0x7fac141984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:29:33.572 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:33.571+0000 7fac1c6c0700 1 --2- 172.21.15.172:0/3579389426 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fac141056b0 0x7fac14198a20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:29:33.572 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:33.571+0000 7fac1c6c0700 1 -- 172.21.15.172:0/3579389426 >> 172.21.15.172:0/3579389426 conn(0x7fac140fc790 msgr2=0x7fac141036c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:29:33.572 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:33.572+0000 7fac1c6c0700 1 -- 172.21.15.172:0/3579389426 shutdown_connections 2024-03-28T22:29:33.572 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:29:33.572+0000 7fac1c6c0700 1 -- 172.21.15.172:0/3579389426 wait complete. 2024-03-28T22:29:34.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:33 smithi043 ceph-mon[31823]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:34.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:33 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:34.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:33 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:34.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:33 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:29:34.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:33 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:29:34.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:33 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:34.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:33 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:34.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:33 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:34.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:33 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:29:34.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:33 smithi043 ceph-mon[31823]: from='osd.5 [v2:172.21.15.172:6808/1576148933,v1:172.21.15.172:6809/1576148933]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T22:29:34.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:33 smithi043 ceph-mon[31823]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T22:29:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:33 smithi172 ceph-mon[36566]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:33 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:33 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:33 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:29:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:33 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:29:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:33 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:33 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:33 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:33 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:29:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:33 smithi172 ceph-mon[36566]: from='osd.5 [v2:172.21.15.172:6808/1576148933,v1:172.21.15.172:6809/1576148933]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T22:29:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:33 smithi172 ceph-mon[36566]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T22:29:34.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:33 smithi082 ceph-mon[36471]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:34.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:33 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:34.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:33 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:34.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:33 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:29:34.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:33 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:29:34.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:33 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:34.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:33 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:34.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:33 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:34.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:33 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:29:34.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:33 smithi082 ceph-mon[36471]: from='osd.5 [v2:172.21.15.172:6808/1576148933,v1:172.21.15.172:6809/1576148933]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T22:29:34.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:33 smithi082 ceph-mon[36471]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T22:29:34.256 DEBUG:teuthology.orchestra.run.smithi172:osd.5> sudo journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.5.service 2024-03-28T22:29:34.258 INFO:tasks.cephadm:Deploying osd.6 on smithi043 with /dev/vg_nvme/lv_2... 2024-03-28T22:29:34.259 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T22:29:34.749 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:29:35.726 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:35 smithi172 ceph-mon[36566]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:35.726 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:35 smithi172 ceph-mon[36566]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T22:29:35.726 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:35 smithi172 ceph-mon[36566]: from='osd.5 [v2:172.21.15.172:6808/1576148933,v1:172.21.15.172:6809/1576148933]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-28T22:29:35.726 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:35 smithi172 ceph-mon[36566]: osdmap e36: 6 total, 5 up, 6 in 2024-03-28T22:29:35.726 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:35 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:29:35.726 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:35 smithi172 ceph-mon[36566]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-28T22:29:35.726 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:29:35 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:29:35.557+0000 7fd0493f4700 -1 osd.5 0 waiting for initial osdmap 2024-03-28T22:29:35.726 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:29:35 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:29:35.564+0000 7fd0451e7700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T22:29:35.937 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:35 smithi043 ceph-mon[31823]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:35.938 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:35 smithi043 ceph-mon[31823]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T22:29:35.938 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:35 smithi043 ceph-mon[31823]: from='osd.5 [v2:172.21.15.172:6808/1576148933,v1:172.21.15.172:6809/1576148933]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-28T22:29:35.938 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:35 smithi043 ceph-mon[31823]: osdmap e36: 6 total, 5 up, 6 in 2024-03-28T22:29:35.938 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:35 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:29:35.938 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:35 smithi043 ceph-mon[31823]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-28T22:29:35.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:35 smithi082 ceph-mon[36471]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:35.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:35 smithi082 ceph-mon[36471]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T22:29:35.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:35 smithi082 ceph-mon[36471]: from='osd.5 [v2:172.21.15.172:6808/1576148933,v1:172.21.15.172:6809/1576148933]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-28T22:29:35.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:35 smithi082 ceph-mon[36471]: osdmap e36: 6 total, 5 up, 6 in 2024-03-28T22:29:35.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:35 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:29:35.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:35 smithi082 ceph-mon[36471]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-28T22:29:36.804 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:36 smithi043 ceph-mon[31823]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-28T22:29:36.804 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:36 smithi043 ceph-mon[31823]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T22:29:36.804 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:36 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:29:36.804 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:36 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:29:36.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:36 smithi172 ceph-mon[36566]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-28T22:29:36.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:36 smithi172 ceph-mon[36566]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T22:29:36.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:36 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:29:36.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:36 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:29:36.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:36 smithi082 ceph-mon[36471]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-28T22:29:36.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:36 smithi082 ceph-mon[36471]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T22:29:36.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:36 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:29:36.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:36 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:29:37.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:37 smithi043 ceph-mon[31823]: purged_snaps scrub starts 2024-03-28T22:29:37.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:37 smithi043 ceph-mon[31823]: purged_snaps scrub ok 2024-03-28T22:29:37.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:37 smithi043 ceph-mon[31823]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:37.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:37 smithi043 ceph-mon[31823]: osd.5 [v2:172.21.15.172:6808/1576148933,v1:172.21.15.172:6809/1576148933] boot 2024-03-28T22:29:37.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:37 smithi043 ceph-mon[31823]: osdmap e38: 6 total, 6 up, 6 in 2024-03-28T22:29:37.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:37 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:29:37.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:37 smithi043 ceph-mon[31823]: osdmap e39: 6 total, 6 up, 6 in 2024-03-28T22:29:37.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:37 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:37.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:37 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:37.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:37 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:29:37.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:37 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:29:37.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:37 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:37.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:37 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:29:37.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:37 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:29:37.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:37 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:37.852 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T22:29:37.852 INFO:teuthology.orchestra.run.smithi043.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T22:29:37.852 INFO:teuthology.orchestra.run.smithi043.stderr: stderr: 10+0 records in 2024-03-28T22:29:37.852 INFO:teuthology.orchestra.run.smithi043.stderr:10+0 records out 2024-03-28T22:29:37.852 INFO:teuthology.orchestra.run.smithi043.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0463003 s, 226 MB/s 2024-03-28T22:29:37.852 INFO:teuthology.orchestra.run.smithi043.stderr: stderr: 2024-03-28T22:29:37.852 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping successful for: 2024-03-28T22:29:37.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:37 smithi172 ceph-mon[36566]: purged_snaps scrub starts 2024-03-28T22:29:37.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:37 smithi172 ceph-mon[36566]: purged_snaps scrub ok 2024-03-28T22:29:37.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:37 smithi172 ceph-mon[36566]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:37.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:37 smithi172 ceph-mon[36566]: osd.5 [v2:172.21.15.172:6808/1576148933,v1:172.21.15.172:6809/1576148933] boot 2024-03-28T22:29:37.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:37 smithi172 ceph-mon[36566]: osdmap e38: 6 total, 6 up, 6 in 2024-03-28T22:29:37.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:37 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:29:37.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:37 smithi172 ceph-mon[36566]: osdmap e39: 6 total, 6 up, 6 in 2024-03-28T22:29:37.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:37 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:37.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:37 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:37.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:37 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:29:37.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:37 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:29:37.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:37 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:37.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:37 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:29:37.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:37 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:29:37.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:37 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:37.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:37 smithi082 ceph-mon[36471]: purged_snaps scrub starts 2024-03-28T22:29:37.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:37 smithi082 ceph-mon[36471]: purged_snaps scrub ok 2024-03-28T22:29:37.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:37 smithi082 ceph-mon[36471]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-28T22:29:37.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:37 smithi082 ceph-mon[36471]: osd.5 [v2:172.21.15.172:6808/1576148933,v1:172.21.15.172:6809/1576148933] boot 2024-03-28T22:29:37.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:37 smithi082 ceph-mon[36471]: osdmap e38: 6 total, 6 up, 6 in 2024-03-28T22:29:37.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:37 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:29:37.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:37 smithi082 ceph-mon[36471]: osdmap e39: 6 total, 6 up, 6 in 2024-03-28T22:29:37.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:37 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:37.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:37 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:37.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:37 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:29:37.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:37 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:29:37.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:37 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:37.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:37 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:29:37.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:37 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:29:37.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:37 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:29:38.930 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:38 smithi172 ceph-mon[36566]: Detected new or changed devices on smithi172 2024-03-28T22:29:38.930 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:38 smithi172 ceph-mon[36566]: Adjusting osd_memory_target on smithi172 to 8540M 2024-03-28T22:29:38.930 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:38 smithi172 ceph-mon[36566]: osdmap e40: 6 total, 6 up, 6 in 2024-03-28T22:29:38.943 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi043:vg_nvme/lv_2 2024-03-28T22:29:39.108 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:38 smithi043 ceph-mon[31823]: Detected new or changed devices on smithi172 2024-03-28T22:29:39.108 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:38 smithi043 ceph-mon[31823]: Adjusting osd_memory_target on smithi172 to 8540M 2024-03-28T22:29:39.108 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:38 smithi043 ceph-mon[31823]: osdmap e40: 6 total, 6 up, 6 in 2024-03-28T22:29:39.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:38 smithi082 ceph-mon[36471]: Detected new or changed devices on smithi172 2024-03-28T22:29:39.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:38 smithi082 ceph-mon[36471]: Adjusting osd_memory_target on smithi172 to 8540M 2024-03-28T22:29:39.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:38 smithi082 ceph-mon[36471]: osdmap e40: 6 total, 6 up, 6 in 2024-03-28T22:29:39.369 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:29:40.071 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:39 smithi043 ceph-mon[31823]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:29:40.071 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:39 smithi043 ceph-mon[31823]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T22:29:40.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:39 smithi172 ceph-mon[36566]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:29:40.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:39 smithi172 ceph-mon[36566]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T22:29:40.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:39 smithi082 ceph-mon[36471]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:29:40.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:39 smithi082 ceph-mon[36471]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T22:29:41.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:40 smithi172 ceph-mon[36566]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T22:29:41.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:40 smithi082 ceph-mon[36471]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T22:29:41.245 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:40 smithi043 ceph-mon[31823]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T22:29:41.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.768+0000 7facdbe00700 1 -- 172.21.15.43:0/3866989586 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7facd4074ca0 msgr2=0x7facd4075120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:29:41.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.768+0000 7facdbe00700 1 --2- 172.21.15.43:0/3866989586 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7facd4074ca0 0x7facd4075120 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7facc0009990 tx=0x7facc0009ca0 comp rx=0 tx=0).stop 2024-03-28T22:29:41.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.768+0000 7facdbe00700 1 -- 172.21.15.43:0/3866989586 shutdown_connections 2024-03-28T22:29:41.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.768+0000 7facdbe00700 1 --2- 172.21.15.43:0/3866989586 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facd410a150 0x7facd410c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:29:41.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.768+0000 7facdbe00700 1 --2- 172.21.15.43:0/3866989586 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7facd4076270 0x7facd40746d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:29:41.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.768+0000 7facdbe00700 1 --2- 172.21.15.43:0/3866989586 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7facd4074ca0 0x7facd4075120 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:29:41.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.768+0000 7facdbe00700 1 -- 172.21.15.43:0/3866989586 >> 172.21.15.43:0/3866989586 conn(0x7facd40fd7e0 msgr2=0x7facd40ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:29:41.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.769+0000 7facdbe00700 1 -- 172.21.15.43:0/3866989586 shutdown_connections 2024-03-28T22:29:41.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.769+0000 7facdbe00700 1 -- 172.21.15.43:0/3866989586 wait complete. 2024-03-28T22:29:41.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.770+0000 7facdbe00700 1 Processor -- start 2024-03-28T22:29:41.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.770+0000 7facdbe00700 1 -- start start 2024-03-28T22:29:41.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.771+0000 7facdbe00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7facd4074ca0 0x7facd419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:29:41.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.771+0000 7facdbe00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7facd4076270 0x7facd419ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:29:41.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.771+0000 7facdbe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facd410a150 0x7facd41a1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:29:41.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.771+0000 7facdbe00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7facd41100e0 con 0x7facd410a150 2024-03-28T22:29:41.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.771+0000 7facdbe00700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7facd410ff60 con 0x7facd4076270 2024-03-28T22:29:41.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.771+0000 7facdbe00700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7facd4110260 con 0x7facd4074ca0 2024-03-28T22:29:41.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.771+0000 7facd9400700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7facd4074ca0 0x7facd419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:29:41.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.771+0000 7facd9e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facd410a150 0x7facd41a1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:29:41.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.771+0000 7facd8a00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7facd4076270 0x7facd419ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:29:41.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.771+0000 7facd9400700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7facd4074ca0 0x7facd419a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:47550/0 (socket says 172.21.15.43:47550) 2024-03-28T22:29:41.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.771+0000 7facd8a00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7facd4076270 0x7facd419ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:56660/0 (socket says 172.21.15.43:56660) 2024-03-28T22:29:41.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.771+0000 7facd9e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facd410a150 0x7facd41a1d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38438/0 (socket says 172.21.15.43:38438) 2024-03-28T22:29:41.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.771+0000 7facd9400700 1 -- 172.21.15.43:0/1643106096 learned_addr learned my addr 172.21.15.43:0/1643106096 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:29:41.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.772+0000 7facd8a00700 1 -- 172.21.15.43:0/1643106096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7facd4074ca0 msgr2=0x7facd419a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:29:41.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.772+0000 7facd8a00700 1 --2- 172.21.15.43:0/1643106096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7facd4074ca0 0x7facd419a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:29:41.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.772+0000 7facd8a00700 1 -- 172.21.15.43:0/1643106096 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facd410a150 msgr2=0x7facd41a1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:29:41.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.772+0000 7facd8a00700 1 --2- 172.21.15.43:0/1643106096 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facd410a150 0x7facd41a1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:29:41.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.772+0000 7facd8a00700 1 -- 172.21.15.43:0/1643106096 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facd41a23b0 con 0x7facd4076270 2024-03-28T22:29:41.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.772+0000 7facd9e00700 1 --2- 172.21.15.43:0/1643106096 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facd410a150 0x7facd41a1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:29:41.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.772+0000 7facd8a00700 1 --2- 172.21.15.43:0/1643106096 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7facd4076270 0x7facd419ac90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7facc0000c00 tx=0x7facc000bb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:29:41.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.772+0000 7facd2000700 1 -- 172.21.15.43:0/1643106096 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc0026070 con 0x7facd4076270 2024-03-28T22:29:41.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.772+0000 7facd2000700 1 -- 172.21.15.43:0/1643106096 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facc0003d70 con 0x7facd4076270 2024-03-28T22:29:41.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.773+0000 7facd2000700 1 -- 172.21.15.43:0/1643106096 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc0021410 con 0x7facd4076270 2024-03-28T22:29:41.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.773+0000 7facdbe00700 1 -- 172.21.15.43:0/1643106096 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facd41a2640 con 0x7facd4076270 2024-03-28T22:29:41.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.773+0000 7facdbe00700 1 -- 172.21.15.43:0/1643106096 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facd41a2a90 con 0x7facd4076270 2024-03-28T22:29:41.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.774+0000 7facdbe00700 1 -- 172.21.15.43:0/1643106096 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facd4068f00 con 0x7facd4076270 2024-03-28T22:29:41.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.776+0000 7facd2000700 1 -- 172.21.15.43:0/1643106096 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7facc0029070 con 0x7facd4076270 2024-03-28T22:29:41.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.777+0000 7facd2000700 1 --2- 172.21.15.43:0/1643106096 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7facbc0a1e10 0x7facbc0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:29:41.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.777+0000 7facd2000700 1 -- 172.21.15.43:0/1643106096 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 4533+0+0 (secure 0 0 0) 0x7facc00cdd30 con 0x7facd4076270 2024-03-28T22:29:41.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.777+0000 7facd9400700 1 --2- 172.21.15.43:0/1643106096 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7facbc0a1e10 0x7facbc0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:29:41.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.778+0000 7facd9400700 1 --2- 172.21.15.43:0/1643106096 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7facbc0a1e10 0x7facbc0a42d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7facc8009d00 tx=0x7facc8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:29:41.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:41.782+0000 7facd2000700 1 -- 172.21.15.43:0/1643106096 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7facc005d280 con 0x7facd4076270 2024-03-28T22:29:42.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:41 smithi043 ceph-mon[31823]: pgmap v158: 1 pgs: 1 peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail; 163 KiB/s, 0 objects/s recovering 2024-03-28T22:29:42.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:29:42.022+0000 7facdbe00700 1 -- 172.21.15.43:0/1643106096 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7facd4062680 con 0x7facbc0a1e10 2024-03-28T22:29:42.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:41 smithi172 ceph-mon[36566]: pgmap v158: 1 pgs: 1 peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail; 163 KiB/s, 0 objects/s recovering 2024-03-28T22:29:42.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:41 smithi082 ceph-mon[36471]: pgmap v158: 1 pgs: 1 peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail; 163 KiB/s, 0 objects/s recovering 2024-03-28T22:29:43.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:42 smithi172 ceph-mon[36566]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:29:43.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:42 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:29:43.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:42 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:29:43.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:42 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:29:43.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:42 smithi082 ceph-mon[36471]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:29:43.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:42 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:29:43.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:42 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:29:43.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:42 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:29:43.224 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:42 smithi043 ceph-mon[31823]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:29:43.224 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:42 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:29:43.224 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:42 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:29:43.224 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:42 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:29:44.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:43 smithi172 ceph-mon[36566]: pgmap v159: 1 pgs: 1 peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-03-28T22:29:44.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:43 smithi082 ceph-mon[36471]: pgmap v159: 1 pgs: 1 peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-03-28T22:29:44.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:43 smithi043 ceph-mon[31823]: pgmap v159: 1 pgs: 1 peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-03-28T22:29:46.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:45 smithi172 ceph-mon[36566]: pgmap v160: 1 pgs: 1 peering; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-28T22:29:46.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:45 smithi082 ceph-mon[36471]: pgmap v160: 1 pgs: 1 peering; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-28T22:29:46.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:45 smithi043 ceph-mon[31823]: pgmap v160: 1 pgs: 1 peering; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-28T22:29:47.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:46 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1290423770' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22d54e57-c3f1-4ecf-9921-8e815b48f92a"}]: dispatch 2024-03-28T22:29:47.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:46 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1290423770' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22d54e57-c3f1-4ecf-9921-8e815b48f92a"}]': finished 2024-03-28T22:29:47.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:46 smithi172 ceph-mon[36566]: osdmap e43: 7 total, 6 up, 7 in 2024-03-28T22:29:47.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:46 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:29:47.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:46 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1290423770' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22d54e57-c3f1-4ecf-9921-8e815b48f92a"}]: dispatch 2024-03-28T22:29:47.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:46 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1290423770' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22d54e57-c3f1-4ecf-9921-8e815b48f92a"}]': finished 2024-03-28T22:29:47.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:46 smithi082 ceph-mon[36471]: osdmap e43: 7 total, 6 up, 7 in 2024-03-28T22:29:47.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:46 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:29:47.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:46 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1290423770' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22d54e57-c3f1-4ecf-9921-8e815b48f92a"}]: dispatch 2024-03-28T22:29:47.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:46 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1290423770' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22d54e57-c3f1-4ecf-9921-8e815b48f92a"}]': finished 2024-03-28T22:29:47.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:46 smithi043 ceph-mon[31823]: osdmap e43: 7 total, 6 up, 7 in 2024-03-28T22:29:47.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:46 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:29:48.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:47 smithi172 ceph-mon[36566]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T22:29:48.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:47 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/793982499' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:29:48.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:47 smithi082 ceph-mon[36471]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T22:29:48.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:47 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/793982499' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:29:48.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:47 smithi043 ceph-mon[31823]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T22:29:48.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:47 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/793982499' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:29:50.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:49 smithi172 ceph-mon[36566]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:29:50.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:49 smithi082 ceph-mon[36471]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:29:50.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:49 smithi043 ceph-mon[31823]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:29:52.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:51 smithi172 ceph-mon[36566]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:29:52.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:51 smithi082 ceph-mon[36471]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:29:52.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:51 smithi043 ceph-mon[31823]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:29:53.945 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:53 smithi043 ceph-mon[31823]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:29:54.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:53 smithi172 ceph-mon[36566]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:29:54.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:53 smithi082 ceph-mon[36471]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:29:56.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:55 smithi043 ceph-mon[31823]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:29:56.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:55 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T22:29:56.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:55 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:29:56.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:55 smithi172 ceph-mon[36566]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:29:56.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:55 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T22:29:56.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:55 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:29:56.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:55 smithi082 ceph-mon[36471]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:29:56.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:55 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T22:29:56.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:55 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:29:57.049 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:56 smithi043 ceph-mon[31823]: Deploying daemon osd.6 on smithi043 2024-03-28T22:29:57.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:56 smithi172 ceph-mon[36566]: Deploying daemon osd.6 on smithi043 2024-03-28T22:29:57.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:56 smithi082 ceph-mon[36471]: Deploying daemon osd.6 on smithi043 2024-03-28T22:29:58.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:57 smithi172 ceph-mon[36566]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:29:58.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:57 smithi082 ceph-mon[36471]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:29:58.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:57 smithi043 ceph-mon[31823]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:30:00.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:29:59 smithi172 ceph-mon[36566]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:30:00.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:29:59 smithi082 ceph-mon[36471]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:30:00.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:29:59 smithi043 ceph-mon[31823]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:30:01.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:00 smithi082 ceph-mon[36471]: overall HEALTH_OK 2024-03-28T22:30:01.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:00 smithi043 ceph-mon[31823]: overall HEALTH_OK 2024-03-28T22:30:01.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:00 smithi172 ceph-mon[36566]: overall HEALTH_OK 2024-03-28T22:30:02.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:01 smithi082 ceph-mon[36471]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:30:02.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:01 smithi043 ceph-mon[31823]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:30:02.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:01 smithi172 ceph-mon[36566]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:30:04.173 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:03 smithi043 ceph-mon[31823]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:30:04.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:03 smithi082 ceph-mon[36471]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:30:04.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:03 smithi172 ceph-mon[36566]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:30:05.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:05 smithi082 ceph-mon[36471]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:30:05.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:05 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:05.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:05 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:05.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:05 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:30:05.761 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:05 smithi043 ceph-mon[31823]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:30:05.761 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:05 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:05.761 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:05 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:05.761 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:05 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:30:05.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:05 smithi172 ceph-mon[36566]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:30:05.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:05 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:05.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:05 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:05.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:05 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:30:07.593 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:07 smithi043 ceph-mon[31823]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:30:07.699 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:07 smithi172 ceph-mon[36566]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:30:07.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:07 smithi082 ceph-mon[36471]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:30:09.231 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:09 smithi043 ceph-mon[31823]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:30:09.231 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:09 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:09.231 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:09 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:09.232 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:09 smithi043 ceph-mon[31823]: from='osd.6 [v2:172.21.15.43:6818/791044279,v1:172.21.15.43:6819/791044279]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T22:30:09.232 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:09 smithi043 ceph-mon[31823]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T22:30:09.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:09 smithi172 ceph-mon[36566]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:30:09.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:09 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:09.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:09 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:09.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:09 smithi172 ceph-mon[36566]: from='osd.6 [v2:172.21.15.43:6818/791044279,v1:172.21.15.43:6819/791044279]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T22:30:09.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:09 smithi172 ceph-mon[36566]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T22:30:09.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:09 smithi082 ceph-mon[36471]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:30:09.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:09 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:09.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:09 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:09.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:09 smithi082 ceph-mon[36471]: from='osd.6 [v2:172.21.15.43:6818/791044279,v1:172.21.15.43:6819/791044279]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T22:30:09.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:09 smithi082 ceph-mon[36471]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T22:30:09.478 INFO:teuthology.orchestra.run.smithi043.stdout:Created osd(s) 6 on host 'smithi043' 2024-03-28T22:30:09.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:30:09.477+0000 7facd2000700 1 -- 172.21.15.43:0/1643106096 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7facd4062680 con 0x7facbc0a1e10 2024-03-28T22:30:09.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:30:09.480+0000 7facdbe00700 1 -- 172.21.15.43:0/1643106096 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7facbc0a1e10 msgr2=0x7facbc0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:30:09.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:30:09.480+0000 7facdbe00700 1 --2- 172.21.15.43:0/1643106096 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7facbc0a1e10 0x7facbc0a42d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7facc8009d00 tx=0x7facc8009260 comp rx=0 tx=0).stop 2024-03-28T22:30:09.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:30:09.481+0000 7facdbe00700 1 -- 172.21.15.43:0/1643106096 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7facd4076270 msgr2=0x7facd419ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:30:09.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:30:09.481+0000 7facdbe00700 1 --2- 172.21.15.43:0/1643106096 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7facd4076270 0x7facd419ac90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7facc0000c00 tx=0x7facc000bb90 comp rx=0 tx=0).stop 2024-03-28T22:30:09.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:30:09.481+0000 7facdbe00700 1 -- 172.21.15.43:0/1643106096 shutdown_connections 2024-03-28T22:30:09.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:30:09.481+0000 7facdbe00700 1 --2- 172.21.15.43:0/1643106096 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7facbc0a1e10 0x7facbc0a42d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:30:09.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:30:09.481+0000 7facdbe00700 1 --2- 172.21.15.43:0/1643106096 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7facd410a150 0x7facd41a1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:30:09.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:30:09.481+0000 7facdbe00700 1 --2- 172.21.15.43:0/1643106096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7facd4074ca0 0x7facd419a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:30:09.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:30:09.481+0000 7facdbe00700 1 --2- 172.21.15.43:0/1643106096 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7facd4076270 0x7facd419ac90 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:30:09.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:30:09.481+0000 7facdbe00700 1 -- 172.21.15.43:0/1643106096 >> 172.21.15.43:0/1643106096 conn(0x7facd40fd7e0 msgr2=0x7facd4106050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:30:09.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:30:09.481+0000 7facdbe00700 1 -- 172.21.15.43:0/1643106096 shutdown_connections 2024-03-28T22:30:09.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:30:09.481+0000 7facdbe00700 1 -- 172.21.15.43:0/1643106096 wait complete. 2024-03-28T22:30:10.288 DEBUG:teuthology.orchestra.run.smithi043:osd.6> sudo journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.6.service 2024-03-28T22:30:10.293 INFO:tasks.cephadm:Deploying osd.7 on smithi082 with /dev/vg_nvme/lv_2... 2024-03-28T22:30:10.293 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T22:30:10.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:10 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:30:10.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:10 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:30:10.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:10 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:10.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:10 smithi082 ceph-mon[36471]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T22:30:10.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:10 smithi082 ceph-mon[36471]: osdmap e44: 7 total, 6 up, 7 in 2024-03-28T22:30:10.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:10 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:30:10.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:10 smithi082 ceph-mon[36471]: from='osd.6 [v2:172.21.15.43:6818/791044279,v1:172.21.15.43:6819/791044279]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-28T22:30:10.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:10 smithi082 ceph-mon[36471]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-28T22:30:10.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:10 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:10.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:10 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:10.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:10 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:30:10.474 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:10 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:30:10.474 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:10 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:30:10.474 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:10 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:10.474 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:10 smithi043 ceph-mon[31823]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T22:30:10.474 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:10 smithi043 ceph-mon[31823]: osdmap e44: 7 total, 6 up, 7 in 2024-03-28T22:30:10.474 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:10 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:30:10.474 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:10 smithi043 ceph-mon[31823]: from='osd.6 [v2:172.21.15.43:6818/791044279,v1:172.21.15.43:6819/791044279]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-28T22:30:10.475 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:10 smithi043 ceph-mon[31823]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-28T22:30:10.475 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:10 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:10.475 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:10 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:10.475 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:10 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:30:10.549 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T22:30:10.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:10 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:30:10.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:10 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:30:10.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:10 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:10.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:10 smithi172 ceph-mon[36566]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T22:30:10.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:10 smithi172 ceph-mon[36566]: osdmap e44: 7 total, 6 up, 7 in 2024-03-28T22:30:10.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:10 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:30:10.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:10 smithi172 ceph-mon[36566]: from='osd.6 [v2:172.21.15.43:6818/791044279,v1:172.21.15.43:6819/791044279]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-28T22:30:10.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:10 smithi172 ceph-mon[36566]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-28T22:30:10.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:10 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:10.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:10 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:10.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:10 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:30:10.739 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:30:10 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:30:10.473+0000 7efe00400700 -1 osd.6 0 waiting for initial osdmap 2024-03-28T22:30:10.740 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:30:10 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:30:10.498+0000 7efdfb200700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T22:30:11.740 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:11 smithi082 ceph-mon[36471]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:30:11.740 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:11 smithi082 ceph-mon[36471]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-03-28T22:30:11.740 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:11 smithi082 ceph-mon[36471]: osdmap e45: 7 total, 6 up, 7 in 2024-03-28T22:30:11.740 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:11 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:30:11.740 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:11 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:30:11.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:11 smithi043 ceph-mon[31823]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:30:11.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:11 smithi043 ceph-mon[31823]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-03-28T22:30:11.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:11 smithi043 ceph-mon[31823]: osdmap e45: 7 total, 6 up, 7 in 2024-03-28T22:30:11.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:11 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:30:11.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:11 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:30:11.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:11 smithi172 ceph-mon[36566]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:30:11.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:11 smithi172 ceph-mon[36566]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-03-28T22:30:11.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:11 smithi172 ceph-mon[36566]: osdmap e45: 7 total, 6 up, 7 in 2024-03-28T22:30:11.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:11 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:30:11.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:11 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:30:12.472 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T22:30:12.473 INFO:teuthology.orchestra.run.smithi082.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T22:30:12.473 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10+0 records in 2024-03-28T22:30:12.473 INFO:teuthology.orchestra.run.smithi082.stderr:10+0 records out 2024-03-28T22:30:12.473 INFO:teuthology.orchestra.run.smithi082.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0134019 s, 782 MB/s 2024-03-28T22:30:12.473 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping successful for: 2024-03-28T22:30:12.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:12 smithi082 ceph-mon[36471]: purged_snaps scrub starts 2024-03-28T22:30:12.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:12 smithi082 ceph-mon[36471]: purged_snaps scrub ok 2024-03-28T22:30:12.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:12 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:30:12.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:12 smithi082 ceph-mon[36471]: osd.6 [v2:172.21.15.43:6818/791044279,v1:172.21.15.43:6819/791044279] boot 2024-03-28T22:30:12.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:12 smithi082 ceph-mon[36471]: osdmap e46: 7 total, 7 up, 7 in 2024-03-28T22:30:12.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:12 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:30:12.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:12 smithi043 ceph-mon[31823]: purged_snaps scrub starts 2024-03-28T22:30:12.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:12 smithi043 ceph-mon[31823]: purged_snaps scrub ok 2024-03-28T22:30:12.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:12 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:30:12.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:12 smithi043 ceph-mon[31823]: osd.6 [v2:172.21.15.43:6818/791044279,v1:172.21.15.43:6819/791044279] boot 2024-03-28T22:30:12.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:12 smithi043 ceph-mon[31823]: osdmap e46: 7 total, 7 up, 7 in 2024-03-28T22:30:12.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:12 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:30:12.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:12 smithi172 ceph-mon[36566]: purged_snaps scrub starts 2024-03-28T22:30:12.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:12 smithi172 ceph-mon[36566]: purged_snaps scrub ok 2024-03-28T22:30:12.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:12 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:30:12.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:12 smithi172 ceph-mon[36566]: osd.6 [v2:172.21.15.43:6818/791044279,v1:172.21.15.43:6819/791044279] boot 2024-03-28T22:30:12.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:12 smithi172 ceph-mon[36566]: osdmap e46: 7 total, 7 up, 7 in 2024-03-28T22:30:12.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:12 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:30:13.166 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi082:vg_nvme/lv_2 2024-03-28T22:30:13.411 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T22:30:13.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:13 smithi043 ceph-mon[31823]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:30:13.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:13 smithi043 ceph-mon[31823]: osdmap e47: 7 total, 7 up, 7 in 2024-03-28T22:30:13.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:13 smithi082 ceph-mon[36471]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:30:13.784 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:13 smithi082 ceph-mon[36471]: osdmap e47: 7 total, 7 up, 7 in 2024-03-28T22:30:13.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:13 smithi172 ceph-mon[36566]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-28T22:30:13.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:13 smithi172 ceph-mon[36566]: osdmap e47: 7 total, 7 up, 7 in 2024-03-28T22:30:15.020 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.020+0000 7f7fba678700 1 -- 172.21.15.82:0/2690397140 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7fb4103470 msgr2=0x7f7fb41038d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:30:15.020 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.020+0000 7f7fba678700 1 --2- 172.21.15.82:0/2690397140 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7fb4103470 0x7f7fb41038d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7fa80099c0 tx=0x7f7fa8009cd0 comp rx=0 tx=0).stop 2024-03-28T22:30:15.020 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.020+0000 7f7fba678700 1 -- 172.21.15.82:0/2690397140 shutdown_connections 2024-03-28T22:30:15.020 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.020+0000 7f7fba678700 1 --2- 172.21.15.82:0/2690397140 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7fb4103e10 0x7f7fb410a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:30:15.020 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.020+0000 7f7fba678700 1 --2- 172.21.15.82:0/2690397140 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7fb4102270 0x7f7fb4102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:30:15.020 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.020+0000 7f7fba678700 1 --2- 172.21.15.82:0/2690397140 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7fb4103470 0x7f7fb41038d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:30:15.020 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.020+0000 7f7fba678700 1 -- 172.21.15.82:0/2690397140 >> 172.21.15.82:0/2690397140 conn(0x7f7fb40fd850 msgr2=0x7f7fb40ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:30:15.021 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.020+0000 7f7fba678700 1 -- 172.21.15.82:0/2690397140 shutdown_connections 2024-03-28T22:30:15.021 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.020+0000 7f7fba678700 1 -- 172.21.15.82:0/2690397140 wait complete. 2024-03-28T22:30:15.021 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.021+0000 7f7fba678700 1 Processor -- start 2024-03-28T22:30:15.021 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.021+0000 7f7fba678700 1 -- start start 2024-03-28T22:30:15.023 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.021+0000 7f7fba678700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7fb4102270 0x7f7fb4198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:30:15.023 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.021+0000 7f7fba678700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7fb4103470 0x7f7fb4198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:30:15.024 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.021+0000 7f7fba678700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7fb4103e10 0x7f7fb419fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:30:15.024 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.021+0000 7f7fba678700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb410df50 con 0x7f7fb4102270 2024-03-28T22:30:15.024 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.021+0000 7f7fba678700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb410ddd0 con 0x7f7fb4103e10 2024-03-28T22:30:15.024 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.021+0000 7f7fba678700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb410e0d0 con 0x7f7fb4103470 2024-03-28T22:30:15.024 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.021+0000 7f7fb8c15700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7fb4103e10 0x7f7fb419fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:30:15.024 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.021+0000 7f7fb8c15700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7fb4103e10 0x7f7fb419fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:52354/0 (socket says 172.21.15.82:52354) 2024-03-28T22:30:15.024 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.021+0000 7f7fb8c15700 1 -- 172.21.15.82:0/3152970032 learned_addr learned my addr 172.21.15.82:0/3152970032 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-03-28T22:30:15.024 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.022+0000 7f7fb37fe700 1 --2- 172.21.15.82:0/3152970032 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7fb4103470 0x7f7fb4198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:30:15.024 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.022+0000 7f7fb8c15700 1 -- 172.21.15.82:0/3152970032 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7fb4103470 msgr2=0x7f7fb4198b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:30:15.024 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.022+0000 7f7fb8c15700 1 --2- 172.21.15.82:0/3152970032 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7fb4103470 0x7f7fb4198b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:30:15.024 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.022+0000 7f7fb8c15700 1 -- 172.21.15.82:0/3152970032 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7fb4102270 msgr2=0x7f7fb4198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:30:15.024 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.022+0000 7f7fb8c15700 1 --2- 172.21.15.82:0/3152970032 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7fb4102270 0x7f7fb4198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:30:15.024 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.022+0000 7f7fb8c15700 1 -- 172.21.15.82:0/3152970032 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fb41a0270 con 0x7f7fb4103e10 2024-03-28T22:30:15.025 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.022+0000 7f7fb37fe700 1 --2- 172.21.15.82:0/3152970032 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7fb4103470 0x7f7fb4198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:30:15.025 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.022+0000 7f7fb8c15700 1 --2- 172.21.15.82:0/3152970032 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7fb4103e10 0x7f7fb419fbd0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7fac00b970 tx=0x7f7fac00bd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:30:15.025 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.022+0000 7f7fb17fa700 1 -- 172.21.15.82:0/3152970032 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fac019070 con 0x7f7fb4103e10 2024-03-28T22:30:15.025 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.022+0000 7f7fba678700 1 -- 172.21.15.82:0/3152970032 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fb41a0560 con 0x7f7fb4103e10 2024-03-28T22:30:15.025 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.022+0000 7f7fb17fa700 1 -- 172.21.15.82:0/3152970032 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fac014440 con 0x7f7fb4103e10 2024-03-28T22:30:15.025 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.022+0000 7f7fb17fa700 1 -- 172.21.15.82:0/3152970032 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fac0075d0 con 0x7f7fb4103e10 2024-03-28T22:30:15.025 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.022+0000 7f7fba678700 1 -- 172.21.15.82:0/3152970032 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fb41a0b20 con 0x7f7fb4103e10 2024-03-28T22:30:15.025 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.023+0000 7f7fba678700 1 -- 172.21.15.82:0/3152970032 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fb4068f00 con 0x7f7fb4103e10 2024-03-28T22:30:15.025 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.024+0000 7f7fb17fa700 1 -- 172.21.15.82:0/3152970032 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f7fac01c070 con 0x7f7fb4103e10 2024-03-28T22:30:15.026 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.025+0000 7f7fb17fa700 1 --2- 172.21.15.82:0/3152970032 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7f9c0a1e10 0x7f7f9c0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:30:15.026 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.025+0000 7f7fb17fa700 1 -- 172.21.15.82:0/3152970032 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7f7fac0cc980 con 0x7f7fb4103e10 2024-03-28T22:30:15.026 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.025+0000 7f7fb3fff700 1 --2- 172.21.15.82:0/3152970032 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7f9c0a1e10 0x7f7f9c0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:30:15.026 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.026+0000 7f7fb3fff700 1 --2- 172.21.15.82:0/3152970032 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7f9c0a1e10 0x7f7f9c0a42d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7fb41032d0 tx=0x7f7fa4006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:30:15.028 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.029+0000 7f7fb17fa700 1 -- 172.21.15.82:0/3152970032 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7fac09aac0 con 0x7f7fb4103e10 2024-03-28T22:30:15.233 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:15.234+0000 7f7fba678700 1 -- 172.21.15.82:0/3152970032 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f7fb4062680 con 0x7f7f9c0a1e10 2024-03-28T22:30:15.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:15 smithi043 ceph-mon[31823]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:15.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:15 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:30:15.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:15 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:30:15.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:15 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:30:15.870 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:15 smithi082 ceph-mon[36471]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:15.870 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:15 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:30:15.870 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:15 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:30:15.870 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:15 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:30:15.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:15 smithi172 ceph-mon[36566]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:15.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:15 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:30:15.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:15 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:30:15.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:15 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:30:16.765 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:16 smithi082 ceph-mon[36471]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:30:16.766 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:16 smithi082 ceph-mon[36471]: Detected new or changed devices on smithi043 2024-03-28T22:30:16.766 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:16 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:16.766 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:16 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:16.766 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:16 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:30:16.766 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:16 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:30:16.766 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:16 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:30:16.766 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:16 smithi082 ceph-mon[36471]: Adjusting osd_memory_target on smithi043 to 5698M 2024-03-28T22:30:16.766 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:16 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:16.766 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:16 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:30:16.766 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:16 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:30:16.766 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:16 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:16.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:16 smithi043 ceph-mon[31823]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:30:16.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:16 smithi043 ceph-mon[31823]: Detected new or changed devices on smithi043 2024-03-28T22:30:16.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:16 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:16.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:16 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:16.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:16 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:30:16.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:16 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:30:16.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:16 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:30:16.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:16 smithi043 ceph-mon[31823]: Adjusting osd_memory_target on smithi043 to 5698M 2024-03-28T22:30:16.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:16 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:16.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:16 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:30:16.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:16 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:30:16.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:16 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:16.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:16 smithi172 ceph-mon[36566]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:30:16.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:16 smithi172 ceph-mon[36566]: Detected new or changed devices on smithi043 2024-03-28T22:30:16.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:16 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:16.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:16 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:16.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:16 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:30:16.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:16 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:30:16.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:16 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:30:16.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:16 smithi172 ceph-mon[36566]: Adjusting osd_memory_target on smithi043 to 5698M 2024-03-28T22:30:16.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:16 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:16.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:16 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:30:16.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:16 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:30:16.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:16 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:17.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:17 smithi172 ceph-mon[36566]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:17.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:17 smithi082 ceph-mon[36471]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:18.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:17 smithi043 ceph-mon[31823]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:18.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:18 smithi172 ceph-mon[36566]: from='client.? 172.21.15.82:0/1142676851' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "248cd1e2-0d6b-452f-a0a8-c51fe1b975ee"}]: dispatch 2024-03-28T22:30:18.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:18 smithi172 ceph-mon[36566]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "248cd1e2-0d6b-452f-a0a8-c51fe1b975ee"}]: dispatch 2024-03-28T22:30:18.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:18 smithi172 ceph-mon[36566]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "248cd1e2-0d6b-452f-a0a8-c51fe1b975ee"}]': finished 2024-03-28T22:30:18.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:18 smithi172 ceph-mon[36566]: osdmap e48: 8 total, 7 up, 8 in 2024-03-28T22:30:18.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:18 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:30:18.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:18 smithi172 ceph-mon[36566]: from='client.? 172.21.15.82:0/281760670' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:30:18.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:18 smithi082 ceph-mon[36471]: from='client.? 172.21.15.82:0/1142676851' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "248cd1e2-0d6b-452f-a0a8-c51fe1b975ee"}]: dispatch 2024-03-28T22:30:18.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:18 smithi082 ceph-mon[36471]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "248cd1e2-0d6b-452f-a0a8-c51fe1b975ee"}]: dispatch 2024-03-28T22:30:18.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:18 smithi082 ceph-mon[36471]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "248cd1e2-0d6b-452f-a0a8-c51fe1b975ee"}]': finished 2024-03-28T22:30:18.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:18 smithi082 ceph-mon[36471]: osdmap e48: 8 total, 7 up, 8 in 2024-03-28T22:30:18.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:18 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:30:18.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:18 smithi082 ceph-mon[36471]: from='client.? 172.21.15.82:0/281760670' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:30:19.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:18 smithi043 ceph-mon[31823]: from='client.? 172.21.15.82:0/1142676851' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "248cd1e2-0d6b-452f-a0a8-c51fe1b975ee"}]: dispatch 2024-03-28T22:30:19.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:18 smithi043 ceph-mon[31823]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "248cd1e2-0d6b-452f-a0a8-c51fe1b975ee"}]: dispatch 2024-03-28T22:30:19.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:18 smithi043 ceph-mon[31823]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "248cd1e2-0d6b-452f-a0a8-c51fe1b975ee"}]': finished 2024-03-28T22:30:19.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:18 smithi043 ceph-mon[31823]: osdmap e48: 8 total, 7 up, 8 in 2024-03-28T22:30:19.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:18 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:30:19.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:18 smithi043 ceph-mon[31823]: from='client.? 172.21.15.82:0/281760670' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:30:20.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:19 smithi172 ceph-mon[36566]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:20.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:19 smithi082 ceph-mon[36471]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:20.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:19 smithi043 ceph-mon[31823]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:22.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:21 smithi172 ceph-mon[36566]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:22.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:21 smithi082 ceph-mon[36471]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:22.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:21 smithi043 ceph-mon[31823]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:24.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:23 smithi172 ceph-mon[36566]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:24.157 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:23 smithi082 ceph-mon[36471]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:24.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:23 smithi043 ceph-mon[31823]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:26.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:25 smithi172 ceph-mon[36566]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:26.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:25 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T22:30:26.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:25 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:30:26.150 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:25 smithi082 ceph-mon[36471]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:26.150 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:25 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T22:30:26.150 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:25 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:30:26.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:25 smithi043 ceph-mon[31823]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:26.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:25 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T22:30:26.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:25 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:30:27.159 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:26 smithi082 ceph-mon[36471]: Deploying daemon osd.7 on smithi082 2024-03-28T22:30:27.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:26 smithi043 ceph-mon[31823]: Deploying daemon osd.7 on smithi082 2024-03-28T22:30:27.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:26 smithi172 ceph-mon[36566]: Deploying daemon osd.7 on smithi082 2024-03-28T22:30:28.155 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:27 smithi082 ceph-mon[36471]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:28.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:27 smithi043 ceph-mon[31823]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:28.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:27 smithi172 ceph-mon[36566]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:30.179 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:29 smithi082 ceph-mon[36471]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:30.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:29 smithi043 ceph-mon[31823]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:30.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:29 smithi172 ceph-mon[36566]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:32.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:31 smithi172 ceph-mon[36566]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:32.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:31 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:32.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:31 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:32.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:31 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:30:32.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:31 smithi082 ceph-mon[36471]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:32.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:31 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:32.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:31 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:32.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:31 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:30:32.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:31 smithi043 ceph-mon[31823]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:32.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:31 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:32.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:31 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:32.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:31 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:30:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:33 smithi172 ceph-mon[36566]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:34.163 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:33 smithi043 ceph-mon[31823]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:34.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:33 smithi082 ceph-mon[36471]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:34.380 INFO:teuthology.orchestra.run.smithi082.stdout:Created osd(s) 7 on host 'smithi082' 2024-03-28T22:30:34.380 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:34.381+0000 7f7fb17fa700 1 -- 172.21.15.82:0/3152970032 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7fb4062680 con 0x7f7f9c0a1e10 2024-03-28T22:30:34.382 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:34.383+0000 7f7fba678700 1 -- 172.21.15.82:0/3152970032 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7f9c0a1e10 msgr2=0x7f7f9c0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:30:34.382 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:34.383+0000 7f7fba678700 1 --2- 172.21.15.82:0/3152970032 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7f9c0a1e10 0x7f7f9c0a42d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7fb41032d0 tx=0x7f7fa4006d20 comp rx=0 tx=0).stop 2024-03-28T22:30:34.383 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:34.383+0000 7f7fba678700 1 -- 172.21.15.82:0/3152970032 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7fb4103e10 msgr2=0x7f7fb419fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:30:34.383 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:34.383+0000 7f7fba678700 1 --2- 172.21.15.82:0/3152970032 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7fb4103e10 0x7f7fb419fbd0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7fac00b970 tx=0x7f7fac00bd30 comp rx=0 tx=0).stop 2024-03-28T22:30:34.383 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:34.383+0000 7f7fba678700 1 -- 172.21.15.82:0/3152970032 shutdown_connections 2024-03-28T22:30:34.383 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:34.383+0000 7f7fba678700 1 --2- 172.21.15.82:0/3152970032 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7f9c0a1e10 0x7f7f9c0a42d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:30:34.383 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:34.383+0000 7f7fba678700 1 --2- 172.21.15.82:0/3152970032 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7fb4103e10 0x7f7fb419fbd0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:30:34.383 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:34.383+0000 7f7fba678700 1 --2- 172.21.15.82:0/3152970032 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7fb4102270 0x7f7fb4198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:30:34.383 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:34.383+0000 7f7fba678700 1 --2- 172.21.15.82:0/3152970032 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7fb4103470 0x7f7fb4198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:30:34.383 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:34.383+0000 7f7fba678700 1 -- 172.21.15.82:0/3152970032 >> 172.21.15.82:0/3152970032 conn(0x7f7fb40fd850 msgr2=0x7f7fb4106860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:30:34.383 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:34.383+0000 7f7fba678700 1 -- 172.21.15.82:0/3152970032 shutdown_connections 2024-03-28T22:30:34.383 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:30:34.383+0000 7f7fba678700 1 -- 172.21.15.82:0/3152970032 wait complete. 2024-03-28T22:30:35.042 DEBUG:teuthology.orchestra.run.smithi082:osd.7> sudo journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.7.service 2024-03-28T22:30:35.044 INFO:tasks.cephadm:Deploying osd.8 on smithi172 with /dev/vg_nvme/lv_2... 2024-03-28T22:30:35.044 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T22:30:35.291 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T22:30:35.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:35 smithi172 ceph-mon[36566]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:35.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:35 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:35.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:35 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:35.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:35 smithi172 ceph-mon[36566]: from='osd.7 [v2:172.21.15.82:6816/2157777323,v1:172.21.15.82:6817/2157777323]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T22:30:35.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:35 smithi172 ceph-mon[36566]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T22:30:35.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:35 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:35.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:35 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:35.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:35 smithi082 ceph-mon[36471]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:35.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:35 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:35.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:35 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:35.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:35 smithi082 ceph-mon[36471]: from='osd.7 [v2:172.21.15.82:6816/2157777323,v1:172.21.15.82:6817/2157777323]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T22:30:35.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:35 smithi082 ceph-mon[36471]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T22:30:35.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:35 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:35.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:35 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:35.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:35 smithi043 ceph-mon[31823]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:35.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:35 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:35.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:35 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:35.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:35 smithi043 ceph-mon[31823]: from='osd.7 [v2:172.21.15.82:6816/2157777323,v1:172.21.15.82:6817/2157777323]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T22:30:35.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:35 smithi043 ceph-mon[31823]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T22:30:35.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:35 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:35.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:35 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:36.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:36 smithi172 ceph-mon[36566]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T22:30:36.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:36 smithi172 ceph-mon[36566]: from='osd.7 [v2:172.21.15.82:6816/2157777323,v1:172.21.15.82:6817/2157777323]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-28T22:30:36.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:36 smithi172 ceph-mon[36566]: osdmap e49: 8 total, 7 up, 8 in 2024-03-28T22:30:36.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:36 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:30:36.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:36 smithi172 ceph-mon[36566]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-28T22:30:36.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:36 smithi082 ceph-mon[36471]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T22:30:36.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:36 smithi082 ceph-mon[36471]: from='osd.7 [v2:172.21.15.82:6816/2157777323,v1:172.21.15.82:6817/2157777323]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-28T22:30:36.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:36 smithi082 ceph-mon[36471]: osdmap e49: 8 total, 7 up, 8 in 2024-03-28T22:30:36.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:36 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:30:36.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:36 smithi082 ceph-mon[36471]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-28T22:30:36.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:36 smithi043 ceph-mon[31823]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T22:30:36.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:36 smithi043 ceph-mon[31823]: from='osd.7 [v2:172.21.15.82:6816/2157777323,v1:172.21.15.82:6817/2157777323]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-28T22:30:36.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:36 smithi043 ceph-mon[31823]: osdmap e49: 8 total, 7 up, 8 in 2024-03-28T22:30:36.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:36 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:30:36.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:36 smithi043 ceph-mon[31823]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-28T22:30:37.200 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:30:37 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:30:37.067+0000 7f3b781bf700 -1 osd.7 0 waiting for initial osdmap 2024-03-28T22:30:37.200 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:30:37 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:30:37.076+0000 7f3b6ffaa700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T22:30:37.385 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T22:30:37.385 INFO:teuthology.orchestra.run.smithi172.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T22:30:37.385 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10+0 records in 2024-03-28T22:30:37.385 INFO:teuthology.orchestra.run.smithi172.stderr:10+0 records out 2024-03-28T22:30:37.385 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130197 s, 805 MB/s 2024-03-28T22:30:37.386 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping successful for: 2024-03-28T22:30:37.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:37 smithi172 ceph-mon[36566]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:37.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:37 smithi172 ceph-mon[36566]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-03-28T22:30:37.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:37 smithi172 ceph-mon[36566]: osdmap e50: 8 total, 7 up, 8 in 2024-03-28T22:30:37.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:37 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:30:37.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:37 smithi172 ceph-mon[36566]: osdmap e51: 8 total, 7 up, 8 in 2024-03-28T22:30:37.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:37 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:30:37.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:37 smithi172 ceph-mon[36566]: from='osd.7 ' entity='osd.7' 2024-03-28T22:30:37.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:37 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:30:37.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:37 smithi082 ceph-mon[36471]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:37.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:37 smithi082 ceph-mon[36471]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-03-28T22:30:37.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:37 smithi082 ceph-mon[36471]: osdmap e50: 8 total, 7 up, 8 in 2024-03-28T22:30:37.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:37 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:30:37.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:37 smithi082 ceph-mon[36471]: osdmap e51: 8 total, 7 up, 8 in 2024-03-28T22:30:37.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:37 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:30:37.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:37 smithi082 ceph-mon[36471]: from='osd.7 ' entity='osd.7' 2024-03-28T22:30:37.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:37 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:30:37.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:37 smithi043 ceph-mon[31823]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:37.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:37 smithi043 ceph-mon[31823]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-03-28T22:30:37.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:37 smithi043 ceph-mon[31823]: osdmap e50: 8 total, 7 up, 8 in 2024-03-28T22:30:37.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:37 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:30:37.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:37 smithi043 ceph-mon[31823]: osdmap e51: 8 total, 7 up, 8 in 2024-03-28T22:30:37.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:37 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:30:37.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:37 smithi043 ceph-mon[31823]: from='osd.7 ' entity='osd.7' 2024-03-28T22:30:37.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:37 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:30:37.978 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi172:vg_nvme/lv_2 2024-03-28T22:30:38.238 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T22:30:38.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:38 smithi172 ceph-mon[36566]: purged_snaps scrub starts 2024-03-28T22:30:38.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:38 smithi172 ceph-mon[36566]: purged_snaps scrub ok 2024-03-28T22:30:38.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:38 smithi172 ceph-mon[36566]: osd.7 [v2:172.21.15.82:6816/2157777323,v1:172.21.15.82:6817/2157777323] boot 2024-03-28T22:30:38.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:38 smithi172 ceph-mon[36566]: osdmap e52: 8 total, 8 up, 8 in 2024-03-28T22:30:38.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:38 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:30:38.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:38 smithi082 ceph-mon[36471]: purged_snaps scrub starts 2024-03-28T22:30:38.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:38 smithi082 ceph-mon[36471]: purged_snaps scrub ok 2024-03-28T22:30:38.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:38 smithi082 ceph-mon[36471]: osd.7 [v2:172.21.15.82:6816/2157777323,v1:172.21.15.82:6817/2157777323] boot 2024-03-28T22:30:38.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:38 smithi082 ceph-mon[36471]: osdmap e52: 8 total, 8 up, 8 in 2024-03-28T22:30:38.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:38 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:30:38.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:38 smithi043 ceph-mon[31823]: purged_snaps scrub starts 2024-03-28T22:30:38.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:38 smithi043 ceph-mon[31823]: purged_snaps scrub ok 2024-03-28T22:30:38.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:38 smithi043 ceph-mon[31823]: osd.7 [v2:172.21.15.82:6816/2157777323,v1:172.21.15.82:6817/2157777323] boot 2024-03-28T22:30:38.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:38 smithi043 ceph-mon[31823]: osdmap e52: 8 total, 8 up, 8 in 2024-03-28T22:30:38.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:38 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:30:39.625 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:39 smithi172 ceph-mon[36566]: pgmap v197: 1 pgs: 1 unknown; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:39.626 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:39 smithi172 ceph-mon[36566]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-28T22:30:39.626 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:39 smithi172 ceph-mon[36566]: osdmap e53: 8 total, 8 up, 8 in 2024-03-28T22:30:39.626 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:39 smithi172 ceph-mon[36566]: Detected new or changed devices on smithi082 2024-03-28T22:30:39.626 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:39 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:39.626 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:39 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:39.626 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:39 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:30:39.626 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:39 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:30:39.626 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:39 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:30:39.626 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:39 smithi172 ceph-mon[36566]: Adjusting osd_memory_target on smithi082 to 5693M 2024-03-28T22:30:39.626 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:39 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:39.626 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:39 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:30:39.626 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:39 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:30:39.627 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:39 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:39.627 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:39 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:30:39.627 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:39 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:30:39.627 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:39 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:30:39.627 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:39 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:39.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:39 smithi082 ceph-mon[36471]: pgmap v197: 1 pgs: 1 unknown; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:39.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:39 smithi082 ceph-mon[36471]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-28T22:30:39.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:39 smithi082 ceph-mon[36471]: osdmap e53: 8 total, 8 up, 8 in 2024-03-28T22:30:39.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:39 smithi082 ceph-mon[36471]: Detected new or changed devices on smithi082 2024-03-28T22:30:39.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:39 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:39.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:39 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:39.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:39 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:30:39.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:39 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:30:39.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:39 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:30:39.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:39 smithi082 ceph-mon[36471]: Adjusting osd_memory_target on smithi082 to 5693M 2024-03-28T22:30:39.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:39 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:39.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:39 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:30:39.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:39 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:30:39.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:39 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:39.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:39 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:30:39.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:39 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:30:39.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:39 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:30:39.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:39 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:39.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:39 smithi043 ceph-mon[31823]: pgmap v197: 1 pgs: 1 unknown; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-28T22:30:39.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:39 smithi043 ceph-mon[31823]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-28T22:30:39.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:39 smithi043 ceph-mon[31823]: osdmap e53: 8 total, 8 up, 8 in 2024-03-28T22:30:39.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:39 smithi043 ceph-mon[31823]: Detected new or changed devices on smithi082 2024-03-28T22:30:39.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:39 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:39.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:39 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:39.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:39 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:30:39.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:39 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:30:39.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:39 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:30:39.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:39 smithi043 ceph-mon[31823]: Adjusting osd_memory_target on smithi082 to 5693M 2024-03-28T22:30:39.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:39 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:39.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:39 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:30:39.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:39 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:30:39.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:39 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:39.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:39 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:30:39.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:39 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:30:39.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:39 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:30:39.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:39 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:39.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.957+0000 7f39d2789700 1 -- 172.21.15.172:0/2900087590 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39cc105680 msgr2=0x7f39cc107a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:30:39.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.957+0000 7f39d2789700 1 --2- 172.21.15.172:0/2900087590 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39cc105680 0x7f39cc107a70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f39c0009990 tx=0x7f39c0009ca0 comp rx=0 tx=0).stop 2024-03-28T22:30:39.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.957+0000 7f39d2789700 1 -- 172.21.15.172:0/2900087590 shutdown_connections 2024-03-28T22:30:39.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.957+0000 7f39d2789700 1 --2- 172.21.15.172:0/2900087590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f39cc107fb0 0x7f39cc10a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:30:39.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.957+0000 7f39d2789700 1 --2- 172.21.15.172:0/2900087590 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f39cc102d50 0x7f39cc105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:30:39.958 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.957+0000 7f39d2789700 1 --2- 172.21.15.172:0/2900087590 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39cc105680 0x7f39cc107a70 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:30:39.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.957+0000 7f39d2789700 1 -- 172.21.15.172:0/2900087590 >> 172.21.15.172:0/2900087590 conn(0x7f39cc0fc760 msgr2=0x7f39cc0feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:30:39.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.957+0000 7f39d2789700 1 -- 172.21.15.172:0/2900087590 shutdown_connections 2024-03-28T22:30:39.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.957+0000 7f39d2789700 1 -- 172.21.15.172:0/2900087590 wait complete. 2024-03-28T22:30:39.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.957+0000 7f39d2789700 1 Processor -- start 2024-03-28T22:30:39.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.957+0000 7f39d2789700 1 -- start start 2024-03-28T22:30:39.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.958+0000 7f39d2789700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f39cc102d50 0x7f39cc102660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:30:39.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.958+0000 7f39d2789700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f39cc105680 0x7f39cc100d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:30:39.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.958+0000 7f39d2789700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39cc107fb0 0x7f39cc101240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:30:39.959 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.958+0000 7f39d2789700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f39cc10df90 con 0x7f39cc102d50 2024-03-28T22:30:39.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.958+0000 7f39d2789700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f39cc10de10 con 0x7f39cc105680 2024-03-28T22:30:39.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.958+0000 7f39d2789700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f39cc10e110 con 0x7f39cc107fb0 2024-03-28T22:30:39.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.958+0000 7f39d0d26700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39cc107fb0 0x7f39cc101240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:30:39.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.958+0000 7f39d0d26700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39cc107fb0 0x7f39cc101240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:55610/0 (socket says 172.21.15.172:55610) 2024-03-28T22:30:39.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.958+0000 7f39d0d26700 1 -- 172.21.15.172:0/3369297609 learned_addr learned my addr 172.21.15.172:0/3369297609 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-28T22:30:39.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.958+0000 7f39cb7fe700 1 --2- 172.21.15.172:0/3369297609 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f39cc105680 0x7f39cc100d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:30:39.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.958+0000 7f39d0d26700 1 -- 172.21.15.172:0/3369297609 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f39cc105680 msgr2=0x7f39cc100d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:30:39.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.958+0000 7f39d0d26700 1 --2- 172.21.15.172:0/3369297609 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f39cc105680 0x7f39cc100d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:30:39.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.958+0000 7f39d0d26700 1 -- 172.21.15.172:0/3369297609 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f39cc102d50 msgr2=0x7f39cc102660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:30:39.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.958+0000 7f39d0d26700 1 --2- 172.21.15.172:0/3369297609 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f39cc102d50 0x7f39cc102660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:30:39.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.958+0000 7f39d0d26700 1 -- 172.21.15.172:0/3369297609 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39cc101b10 con 0x7f39cc107fb0 2024-03-28T22:30:39.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.958+0000 7f39d0d26700 1 --2- 172.21.15.172:0/3369297609 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39cc107fb0 0x7f39cc101240 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f39c400b9e0 tx=0x7f39c400bcf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:30:39.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.959+0000 7f39c97fa700 1 -- 172.21.15.172:0/3369297609 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c4019070 con 0x7f39cc107fb0 2024-03-28T22:30:39.960 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.959+0000 7f39c97fa700 1 -- 172.21.15.172:0/3369297609 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39c400ce10 con 0x7f39cc107fb0 2024-03-28T22:30:39.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.959+0000 7f39c97fa700 1 -- 172.21.15.172:0/3369297609 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c4012360 con 0x7f39cc107fb0 2024-03-28T22:30:39.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.959+0000 7f39d2789700 1 -- 172.21.15.172:0/3369297609 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39cc1a2770 con 0x7f39cc107fb0 2024-03-28T22:30:39.961 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.959+0000 7f39d2789700 1 -- 172.21.15.172:0/3369297609 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39cc1a2cb0 con 0x7f39cc107fb0 2024-03-28T22:30:39.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.960+0000 7f39d2789700 1 -- 172.21.15.172:0/3369297609 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39cc068f00 con 0x7f39cc107fb0 2024-03-28T22:30:39.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.961+0000 7f39c97fa700 1 -- 172.21.15.172:0/3369297609 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f39c401c070 con 0x7f39cc107fb0 2024-03-28T22:30:39.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.961+0000 7f39c97fa700 1 --2- 172.21.15.172:0/3369297609 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f39b40a1e10 0x7f39b40a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:30:39.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.961+0000 7f39c97fa700 1 -- 172.21.15.172:0/3369297609 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5426+0+0 (secure 0 0 0) 0x7f39c40cc830 con 0x7f39cc107fb0 2024-03-28T22:30:39.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.962+0000 7f39cbfff700 1 --2- 172.21.15.172:0/3369297609 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f39b40a1e10 0x7f39b40a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:30:39.962 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.963+0000 7f39cbfff700 1 --2- 172.21.15.172:0/3369297609 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f39b40a1e10 0x7f39b40a42d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f39bc0097b0 tx=0x7f39bc006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:30:39.963 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:39.965+0000 7f39c97fa700 1 -- 172.21.15.172:0/3369297609 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f39c409a7a0 con 0x7f39cc107fb0 2024-03-28T22:30:40.169 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:40.170+0000 7f39d2789700 1 -- 172.21.15.172:0/3369297609 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f39cc062680 con 0x7f39b40a1e10 2024-03-28T22:30:41.132 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:40 smithi172 ceph-mon[36566]: osdmap e54: 8 total, 8 up, 8 in 2024-03-28T22:30:41.132 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:40 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:30:41.132 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:40 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:30:41.132 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:40 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:30:41.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:40 smithi082 ceph-mon[36471]: osdmap e54: 8 total, 8 up, 8 in 2024-03-28T22:30:41.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:40 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:30:41.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:40 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:30:41.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:40 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:30:41.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:40 smithi043 ceph-mon[31823]: osdmap e54: 8 total, 8 up, 8 in 2024-03-28T22:30:41.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:40 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:30:41.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:40 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:30:41.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:40 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:30:42.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:41 smithi172 ceph-mon[36566]: pgmap v200: 1 pgs: 1 unknown; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:30:42.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:41 smithi172 ceph-mon[36566]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:30:42.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:41 smithi172 ceph-mon[36566]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T22:30:42.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:41 smithi082 ceph-mon[36471]: pgmap v200: 1 pgs: 1 unknown; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:30:42.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:41 smithi082 ceph-mon[36471]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:30:42.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:41 smithi082 ceph-mon[36471]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T22:30:42.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:41 smithi043 ceph-mon[31823]: pgmap v200: 1 pgs: 1 unknown; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:30:42.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:41 smithi043 ceph-mon[31823]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:30:42.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:41 smithi043 ceph-mon[31823]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T22:30:43.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:42 smithi172 ceph-mon[36566]: osdmap e56: 8 total, 8 up, 8 in 2024-03-28T22:30:43.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:42 smithi082 ceph-mon[36471]: osdmap e56: 8 total, 8 up, 8 in 2024-03-28T22:30:43.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:42 smithi043 ceph-mon[31823]: osdmap e56: 8 total, 8 up, 8 in 2024-03-28T22:30:44.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:43 smithi172 ceph-mon[36566]: pgmap v203: 1 pgs: 1 unknown; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:30:44.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:43 smithi172 ceph-mon[36566]: from='client.? 172.21.15.172:0/2584574666' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d98aab6e-8bae-4c24-906c-4f492f047b8e"}]: dispatch 2024-03-28T22:30:44.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:43 smithi172 ceph-mon[36566]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d98aab6e-8bae-4c24-906c-4f492f047b8e"}]: dispatch 2024-03-28T22:30:44.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:43 smithi172 ceph-mon[36566]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d98aab6e-8bae-4c24-906c-4f492f047b8e"}]': finished 2024-03-28T22:30:44.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:43 smithi172 ceph-mon[36566]: osdmap e57: 9 total, 8 up, 9 in 2024-03-28T22:30:44.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:43 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:30:44.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:43 smithi172 ceph-mon[36566]: from='client.? 172.21.15.172:0/507703119' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:30:44.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:43 smithi082 ceph-mon[36471]: pgmap v203: 1 pgs: 1 unknown; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:30:44.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:43 smithi082 ceph-mon[36471]: from='client.? 172.21.15.172:0/2584574666' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d98aab6e-8bae-4c24-906c-4f492f047b8e"}]: dispatch 2024-03-28T22:30:44.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:43 smithi082 ceph-mon[36471]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d98aab6e-8bae-4c24-906c-4f492f047b8e"}]: dispatch 2024-03-28T22:30:44.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:43 smithi082 ceph-mon[36471]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d98aab6e-8bae-4c24-906c-4f492f047b8e"}]': finished 2024-03-28T22:30:44.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:43 smithi082 ceph-mon[36471]: osdmap e57: 9 total, 8 up, 9 in 2024-03-28T22:30:44.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:43 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:30:44.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:43 smithi082 ceph-mon[36471]: from='client.? 172.21.15.172:0/507703119' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:30:44.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:43 smithi043 ceph-mon[31823]: pgmap v203: 1 pgs: 1 unknown; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:30:44.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:43 smithi043 ceph-mon[31823]: from='client.? 172.21.15.172:0/2584574666' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d98aab6e-8bae-4c24-906c-4f492f047b8e"}]: dispatch 2024-03-28T22:30:44.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:43 smithi043 ceph-mon[31823]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d98aab6e-8bae-4c24-906c-4f492f047b8e"}]: dispatch 2024-03-28T22:30:44.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:43 smithi043 ceph-mon[31823]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d98aab6e-8bae-4c24-906c-4f492f047b8e"}]': finished 2024-03-28T22:30:44.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:43 smithi043 ceph-mon[31823]: osdmap e57: 9 total, 8 up, 9 in 2024-03-28T22:30:44.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:43 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:30:44.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:43 smithi043 ceph-mon[31823]: from='client.? 172.21.15.172:0/507703119' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:30:45.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:44 smithi082 ceph-mon[36471]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-28T22:30:45.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:44 smithi082 ceph-mon[36471]: Cluster is now healthy 2024-03-28T22:30:45.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:44 smithi043 ceph-mon[31823]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-28T22:30:45.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:44 smithi043 ceph-mon[31823]: Cluster is now healthy 2024-03-28T22:30:45.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:44 smithi172 ceph-mon[36566]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-28T22:30:45.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:44 smithi172 ceph-mon[36566]: Cluster is now healthy 2024-03-28T22:30:46.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:45 smithi082 ceph-mon[36471]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:30:46.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:45 smithi043 ceph-mon[31823]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:30:46.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:45 smithi172 ceph-mon[36566]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:30:48.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:47 smithi082 ceph-mon[36471]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:30:48.241 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:47 smithi172 ceph-mon[36566]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:30:48.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:47 smithi043 ceph-mon[31823]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:30:50.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:49 smithi172 ceph-mon[36566]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:30:50.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:49 smithi082 ceph-mon[36471]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:30:50.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:49 smithi043 ceph-mon[31823]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:30:51.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:50 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T22:30:51.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:50 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:30:51.240 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:50 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T22:30:51.240 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:50 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:30:51.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:50 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T22:30:51.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:50 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:30:52.143 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:51 smithi172 ceph-mon[36566]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:30:52.143 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:51 smithi172 ceph-mon[36566]: Deploying daemon osd.8 on smithi172 2024-03-28T22:30:52.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:51 smithi082 ceph-mon[36471]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:30:52.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:51 smithi082 ceph-mon[36471]: Deploying daemon osd.8 on smithi172 2024-03-28T22:30:52.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:51 smithi043 ceph-mon[31823]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:30:52.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:51 smithi043 ceph-mon[31823]: Deploying daemon osd.8 on smithi172 2024-03-28T22:30:54.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:53 smithi082 ceph-mon[36471]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:30:54.247 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:53 smithi172 ceph-mon[36566]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:30:54.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:53 smithi043 ceph-mon[31823]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:30:56.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:55 smithi172 ceph-mon[36566]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:30:56.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:55 smithi043 ceph-mon[31823]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:30:56.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:55 smithi082 ceph-mon[36471]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:30:57.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:57 smithi043 ceph-mon[31823]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:30:57.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:57 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:57.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:57 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:57.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:57 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:30:57.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:57 smithi172 ceph-mon[36566]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:30:57.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:57 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:57.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:57 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:57.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:57 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:30:57.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:57 smithi082 ceph-mon[36471]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:30:57.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:57 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:57.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:57 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:30:57.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:57 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:30:59.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:30:59 smithi172 ceph-mon[36566]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:30:59.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:30:59 smithi043 ceph-mon[31823]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:30:59.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:30:59 smithi082 ceph-mon[36471]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:30:59.803 INFO:teuthology.orchestra.run.smithi172.stdout:Created osd(s) 8 on host 'smithi172' 2024-03-28T22:30:59.804 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:59.804+0000 7f39c97fa700 1 -- 172.21.15.172:0/3369297609 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f39cc062680 con 0x7f39b40a1e10 2024-03-28T22:30:59.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:59.806+0000 7f39d2789700 1 -- 172.21.15.172:0/3369297609 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f39b40a1e10 msgr2=0x7f39b40a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:30:59.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:59.806+0000 7f39d2789700 1 --2- 172.21.15.172:0/3369297609 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f39b40a1e10 0x7f39b40a42d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f39bc0097b0 tx=0x7f39bc006d20 comp rx=0 tx=0).stop 2024-03-28T22:30:59.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:59.807+0000 7f39d2789700 1 -- 172.21.15.172:0/3369297609 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39cc107fb0 msgr2=0x7f39cc101240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:30:59.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:59.807+0000 7f39d2789700 1 --2- 172.21.15.172:0/3369297609 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39cc107fb0 0x7f39cc101240 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f39c400b9e0 tx=0x7f39c400bcf0 comp rx=0 tx=0).stop 2024-03-28T22:30:59.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:59.807+0000 7f39d2789700 1 -- 172.21.15.172:0/3369297609 shutdown_connections 2024-03-28T22:30:59.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:59.807+0000 7f39d2789700 1 --2- 172.21.15.172:0/3369297609 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f39b40a1e10 0x7f39b40a42d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:30:59.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:59.807+0000 7f39d2789700 1 --2- 172.21.15.172:0/3369297609 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f39cc107fb0 0x7f39cc101240 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:30:59.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:59.807+0000 7f39d2789700 1 --2- 172.21.15.172:0/3369297609 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f39cc102d50 0x7f39cc102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:30:59.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:59.807+0000 7f39d2789700 1 --2- 172.21.15.172:0/3369297609 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f39cc105680 0x7f39cc100d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:30:59.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:59.807+0000 7f39d2789700 1 -- 172.21.15.172:0/3369297609 >> 172.21.15.172:0/3369297609 conn(0x7f39cc0fc760 msgr2=0x7f39cc1036b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:30:59.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:59.807+0000 7f39d2789700 1 -- 172.21.15.172:0/3369297609 shutdown_connections 2024-03-28T22:30:59.806 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:30:59.807+0000 7f39d2789700 1 -- 172.21.15.172:0/3369297609 wait complete. 2024-03-28T22:31:00.468 DEBUG:teuthology.orchestra.run.smithi172:osd.8> sudo journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.8.service 2024-03-28T22:31:00.470 INFO:tasks.cephadm:Deploying osd.9 on smithi043 with /dev/vg_nvme/lv_1... 2024-03-28T22:31:00.470 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T22:31:00.932 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:31:01.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:00 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:01.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:00 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:01.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:00 smithi043 ceph-mon[31823]: from='osd.8 [v2:172.21.15.172:6816/1095328248,v1:172.21.15.172:6817/1095328248]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T22:31:01.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:00 smithi043 ceph-mon[31823]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T22:31:01.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:00 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:01.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:00 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:01.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:00 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:01.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:00 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:01.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:00 smithi172 ceph-mon[36566]: from='osd.8 [v2:172.21.15.172:6816/1095328248,v1:172.21.15.172:6817/1095328248]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T22:31:01.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:00 smithi172 ceph-mon[36566]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T22:31:01.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:00 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:01.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:00 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:01.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:00 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:01.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:00 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:01.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:00 smithi082 ceph-mon[36471]: from='osd.8 [v2:172.21.15.172:6816/1095328248,v1:172.21.15.172:6817/1095328248]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T22:31:01.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:00 smithi082 ceph-mon[36471]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T22:31:01.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:00 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:01.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:00 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:02.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:01 smithi172 ceph-mon[36566]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:31:02.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:01 smithi172 ceph-mon[36566]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T22:31:02.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:01 smithi172 ceph-mon[36566]: osdmap e58: 9 total, 8 up, 9 in 2024-03-28T22:31:02.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:01 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:31:02.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:01 smithi172 ceph-mon[36566]: from='osd.8 [v2:172.21.15.172:6816/1095328248,v1:172.21.15.172:6817/1095328248]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-28T22:31:02.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:01 smithi172 ceph-mon[36566]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-28T22:31:02.147 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:31:01 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:31:01.798+0000 7fc0a8e9f700 -1 osd.8 0 waiting for initial osdmap 2024-03-28T22:31:02.147 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:31:01 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:31:01.809+0000 7fc0a4c92700 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T22:31:02.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:01 smithi043 ceph-mon[31823]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:31:02.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:01 smithi043 ceph-mon[31823]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T22:31:02.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:01 smithi043 ceph-mon[31823]: osdmap e58: 9 total, 8 up, 9 in 2024-03-28T22:31:02.149 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:01 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:31:02.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:01 smithi043 ceph-mon[31823]: from='osd.8 [v2:172.21.15.172:6816/1095328248,v1:172.21.15.172:6817/1095328248]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-28T22:31:02.150 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:01 smithi043 ceph-mon[31823]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-28T22:31:02.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:01 smithi082 ceph-mon[36471]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:31:02.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:01 smithi082 ceph-mon[36471]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T22:31:02.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:01 smithi082 ceph-mon[36471]: osdmap e58: 9 total, 8 up, 9 in 2024-03-28T22:31:02.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:01 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:31:02.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:01 smithi082 ceph-mon[36471]: from='osd.8 [v2:172.21.15.172:6816/1095328248,v1:172.21.15.172:6817/1095328248]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-28T22:31:02.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:01 smithi082 ceph-mon[36471]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-28T22:31:02.988 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:02 smithi043 ceph-mon[31823]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-28T22:31:02.989 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:02 smithi043 ceph-mon[31823]: osdmap e59: 9 total, 8 up, 9 in 2024-03-28T22:31:02.989 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:02 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:31:02.989 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:02 smithi043 ceph-mon[31823]: osdmap e60: 9 total, 8 up, 9 in 2024-03-28T22:31:02.989 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:02 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:31:02.989 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:02 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:31:03.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:02 smithi172 ceph-mon[36566]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-28T22:31:03.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:02 smithi172 ceph-mon[36566]: osdmap e59: 9 total, 8 up, 9 in 2024-03-28T22:31:03.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:02 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:31:03.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:02 smithi172 ceph-mon[36566]: osdmap e60: 9 total, 8 up, 9 in 2024-03-28T22:31:03.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:02 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:31:03.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:02 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:31:03.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:02 smithi082 ceph-mon[36471]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-28T22:31:03.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:02 smithi082 ceph-mon[36471]: osdmap e59: 9 total, 8 up, 9 in 2024-03-28T22:31:03.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:02 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:31:03.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:02 smithi082 ceph-mon[36471]: osdmap e60: 9 total, 8 up, 9 in 2024-03-28T22:31:03.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:02 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:31:03.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:02 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:31:04.028 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T22:31:04.028 INFO:teuthology.orchestra.run.smithi043.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T22:31:04.028 INFO:teuthology.orchestra.run.smithi043.stderr: stderr: 10+0 records in 2024-03-28T22:31:04.028 INFO:teuthology.orchestra.run.smithi043.stderr:10+0 records out 2024-03-28T22:31:04.028 INFO:teuthology.orchestra.run.smithi043.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0400781 s, 262 MB/s 2024-03-28T22:31:04.028 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping successful for: 2024-03-28T22:31:04.073 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:03 smithi172 ceph-mon[36566]: purged_snaps scrub starts 2024-03-28T22:31:04.073 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:03 smithi172 ceph-mon[36566]: purged_snaps scrub ok 2024-03-28T22:31:04.073 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:03 smithi172 ceph-mon[36566]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:31:04.073 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:03 smithi172 ceph-mon[36566]: osd.8 [v2:172.21.15.172:6816/1095328248,v1:172.21.15.172:6817/1095328248] boot 2024-03-28T22:31:04.073 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:03 smithi172 ceph-mon[36566]: osdmap e61: 9 total, 9 up, 9 in 2024-03-28T22:31:04.073 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:03 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:31:04.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:03 smithi082 ceph-mon[36471]: purged_snaps scrub starts 2024-03-28T22:31:04.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:03 smithi082 ceph-mon[36471]: purged_snaps scrub ok 2024-03-28T22:31:04.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:03 smithi082 ceph-mon[36471]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:31:04.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:03 smithi082 ceph-mon[36471]: osd.8 [v2:172.21.15.172:6816/1095328248,v1:172.21.15.172:6817/1095328248] boot 2024-03-28T22:31:04.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:03 smithi082 ceph-mon[36471]: osdmap e61: 9 total, 9 up, 9 in 2024-03-28T22:31:04.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:03 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:31:04.254 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:03 smithi043 ceph-mon[31823]: purged_snaps scrub starts 2024-03-28T22:31:04.254 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:03 smithi043 ceph-mon[31823]: purged_snaps scrub ok 2024-03-28T22:31:04.254 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:03 smithi043 ceph-mon[31823]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-28T22:31:04.254 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:03 smithi043 ceph-mon[31823]: osd.8 [v2:172.21.15.172:6816/1095328248,v1:172.21.15.172:6817/1095328248] boot 2024-03-28T22:31:04.254 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:03 smithi043 ceph-mon[31823]: osdmap e61: 9 total, 9 up, 9 in 2024-03-28T22:31:04.254 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:03 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:31:04.892 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi043:vg_nvme/lv_1 2024-03-28T22:31:05.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:04 smithi172 ceph-mon[36566]: osdmap e62: 9 total, 9 up, 9 in 2024-03-28T22:31:05.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:04 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:05.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:04 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:05.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:04 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:31:05.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:04 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:31:05.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:04 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:31:05.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:04 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:05.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:04 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:31:05.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:04 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:31:05.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:04 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:05.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:04 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:31:05.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:04 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:31:05.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:04 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:31:05.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:04 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:05.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:04 smithi082 ceph-mon[36471]: osdmap e62: 9 total, 9 up, 9 in 2024-03-28T22:31:05.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:04 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:05.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:04 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:05.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:04 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:31:05.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:04 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:31:05.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:04 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:31:05.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:04 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:05.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:04 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:31:05.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:04 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:31:05.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:04 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:05.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:04 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:31:05.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:04 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:31:05.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:04 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:31:05.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:04 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:05.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:04 smithi043 ceph-mon[31823]: osdmap e62: 9 total, 9 up, 9 in 2024-03-28T22:31:05.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:04 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:05.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:04 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:05.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:04 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:31:05.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:04 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:31:05.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:04 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:31:05.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:04 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:05.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:04 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:31:05.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:04 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:31:05.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:04 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:05.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:04 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:31:05.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:04 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:31:05.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:04 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:31:05.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:04 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:05.347 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:31:06.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:05 smithi172 ceph-mon[36566]: pgmap v220: 1 pgs: 1 remapped+peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:06.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:05 smithi172 ceph-mon[36566]: Detected new or changed devices on smithi172 2024-03-28T22:31:06.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:05 smithi172 ceph-mon[36566]: Adjusting osd_memory_target on smithi172 to 5693M 2024-03-28T22:31:06.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:05 smithi172 ceph-mon[36566]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T22:31:06.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:05 smithi082 ceph-mon[36471]: pgmap v220: 1 pgs: 1 remapped+peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:06.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:05 smithi082 ceph-mon[36471]: Detected new or changed devices on smithi172 2024-03-28T22:31:06.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:05 smithi082 ceph-mon[36471]: Adjusting osd_memory_target on smithi172 to 5693M 2024-03-28T22:31:06.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:05 smithi082 ceph-mon[36471]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T22:31:06.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:05 smithi043 ceph-mon[31823]: pgmap v220: 1 pgs: 1 remapped+peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:06.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:05 smithi043 ceph-mon[31823]: Detected new or changed devices on smithi172 2024-03-28T22:31:06.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:05 smithi043 ceph-mon[31823]: Adjusting osd_memory_target on smithi172 to 5693M 2024-03-28T22:31:06.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:05 smithi043 ceph-mon[31823]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T22:31:07.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:06 smithi172 ceph-mon[36566]: osdmap e64: 9 total, 9 up, 9 in 2024-03-28T22:31:07.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:06 smithi082 ceph-mon[36471]: osdmap e64: 9 total, 9 up, 9 in 2024-03-28T22:31:07.205 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:06 smithi043 ceph-mon[31823]: osdmap e64: 9 total, 9 up, 9 in 2024-03-28T22:31:07.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.706+0000 7ff14b600700 1 -- 172.21.15.43:0/196738672 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff144106060 msgr2=0x7ff14410c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:31:07.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.706+0000 7ff14b600700 1 --2- 172.21.15.43:0/196738672 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff144106060 0x7ff14410c910 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff13c009fc0 tx=0x7ff13c00b330 comp rx=0 tx=0).stop 2024-03-28T22:31:07.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.706+0000 7ff14b600700 1 -- 172.21.15.43:0/196738672 shutdown_connections 2024-03-28T22:31:07.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.706+0000 7ff14b600700 1 --2- 172.21.15.43:0/196738672 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff144106060 0x7ff14410c910 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:31:07.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.706+0000 7ff14b600700 1 --2- 172.21.15.43:0/196738672 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1441044c0 0x7ff1441048e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:31:07.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.706+0000 7ff14b600700 1 --2- 172.21.15.43:0/196738672 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff1441056c0 0x7ff144105b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:31:07.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.706+0000 7ff14b600700 1 -- 172.21.15.43:0/196738672 >> 172.21.15.43:0/196738672 conn(0x7ff1440ffa60 msgr2=0x7ff144101ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:31:07.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.706+0000 7ff14b600700 1 -- 172.21.15.43:0/196738672 shutdown_connections 2024-03-28T22:31:07.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.707+0000 7ff14b600700 1 -- 172.21.15.43:0/196738672 wait complete. 2024-03-28T22:31:07.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.708+0000 7ff14b600700 1 Processor -- start 2024-03-28T22:31:07.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.708+0000 7ff14b600700 1 -- start start 2024-03-28T22:31:07.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.708+0000 7ff14b600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1441044c0 0x7ff14419a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:31:07.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.708+0000 7ff14b600700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff1441056c0 0x7ff14419ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:31:07.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.708+0000 7ff14b600700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff144106060 0x7ff1441a1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:31:07.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.708+0000 7ff14b600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff144110090 con 0x7ff1441044c0 2024-03-28T22:31:07.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.708+0000 7ff14b600700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ff14410ff10 con 0x7ff1441056c0 2024-03-28T22:31:07.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.708+0000 7ff14b600700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7ff144110210 con 0x7ff144106060 2024-03-28T22:31:07.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.709+0000 7ff148c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1441044c0 0x7ff14419a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:31:07.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.709+0000 7ff148c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1441044c0 0x7ff14419a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54804/0 (socket says 172.21.15.43:54804) 2024-03-28T22:31:07.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.709+0000 7ff148c00700 1 -- 172.21.15.43:0/4008828745 learned_addr learned my addr 172.21.15.43:0/4008828745 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:31:07.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.709+0000 7ff143e00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff1441056c0 0x7ff14419ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:31:07.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.709+0000 7ff149600700 1 --2- 172.21.15.43:0/4008828745 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff144106060 0x7ff1441a1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:31:07.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.709+0000 7ff148c00700 1 -- 172.21.15.43:0/4008828745 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff144106060 msgr2=0x7ff1441a1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:31:07.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.709+0000 7ff148c00700 1 --2- 172.21.15.43:0/4008828745 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff144106060 0x7ff1441a1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:31:07.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.709+0000 7ff148c00700 1 -- 172.21.15.43:0/4008828745 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff1441056c0 msgr2=0x7ff14419ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:31:07.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.709+0000 7ff148c00700 1 --2- 172.21.15.43:0/4008828745 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff1441056c0 0x7ff14419ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:31:07.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.709+0000 7ff148c00700 1 -- 172.21.15.43:0/4008828745 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1441a2250 con 0x7ff1441044c0 2024-03-28T22:31:07.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.709+0000 7ff149600700 1 --2- 172.21.15.43:0/4008828745 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff144106060 0x7ff1441a1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:31:07.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.709+0000 7ff148c00700 1 --2- 172.21.15.43:0/4008828745 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1441044c0 0x7ff14419a5f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff13800ec20 tx=0x7ff13800ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:31:07.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.710+0000 7ff141600700 1 -- 172.21.15.43:0/4008828745 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13800cd40 con 0x7ff1441044c0 2024-03-28T22:31:07.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.710+0000 7ff141600700 1 -- 172.21.15.43:0/4008828745 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff138010910 con 0x7ff1441044c0 2024-03-28T22:31:07.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.710+0000 7ff14b600700 1 -- 172.21.15.43:0/4008828745 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1441a2540 con 0x7ff1441044c0 2024-03-28T22:31:07.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.710+0000 7ff141600700 1 -- 172.21.15.43:0/4008828745 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff138018a00 con 0x7ff1441044c0 2024-03-28T22:31:07.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.710+0000 7ff14b600700 1 -- 172.21.15.43:0/4008828745 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1441a2a80 con 0x7ff1441044c0 2024-03-28T22:31:07.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.711+0000 7ff14b600700 1 -- 172.21.15.43:0/4008828745 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff144068f00 con 0x7ff1441044c0 2024-03-28T22:31:07.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.713+0000 7ff141600700 1 -- 172.21.15.43:0/4008828745 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7ff13801f020 con 0x7ff1441044c0 2024-03-28T22:31:07.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.714+0000 7ff141600700 1 --2- 172.21.15.43:0/4008828745 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff12c0a1e10 0x7ff12c0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:31:07.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.715+0000 7ff141600700 1 -- 172.21.15.43:0/4008828745 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5829+0+0 (secure 0 0 0) 0x7ff1380cd790 con 0x7ff1441044c0 2024-03-28T22:31:07.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.715+0000 7ff143e00700 1 --2- 172.21.15.43:0/4008828745 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff12c0a1e10 0x7ff12c0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:31:07.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.715+0000 7ff143e00700 1 --2- 172.21.15.43:0/4008828745 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff12c0a1e10 0x7ff12c0a42d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff130009c60 tx=0x7ff130009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:31:07.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.719+0000 7ff141600700 1 -- 172.21.15.43:0/4008828745 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff138053280 con 0x7ff1441044c0 2024-03-28T22:31:07.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:07.961+0000 7ff14b600700 1 -- 172.21.15.43:0/4008828745 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7ff144062680 con 0x7ff12c0a1e10 2024-03-28T22:31:08.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:07 smithi172 ceph-mon[36566]: pgmap v223: 1 pgs: 1 remapped+peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:08.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:07 smithi172 ceph-mon[36566]: osdmap e65: 9 total, 9 up, 9 in 2024-03-28T22:31:08.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:07 smithi082 ceph-mon[36471]: pgmap v223: 1 pgs: 1 remapped+peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:08.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:07 smithi082 ceph-mon[36471]: osdmap e65: 9 total, 9 up, 9 in 2024-03-28T22:31:08.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:07 smithi043 ceph-mon[31823]: pgmap v223: 1 pgs: 1 remapped+peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:08.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:07 smithi043 ceph-mon[31823]: osdmap e65: 9 total, 9 up, 9 in 2024-03-28T22:31:09.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:08 smithi172 ceph-mon[36566]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:31:09.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:08 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:31:09.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:08 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:31:09.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:08 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:31:09.155 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:08 smithi043 ceph-mon[31823]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:31:09.155 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:08 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:31:09.155 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:08 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:31:09.155 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:08 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:31:09.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:08 smithi082 ceph-mon[36471]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:31:09.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:08 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:31:09.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:08 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:31:09.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:08 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:31:10.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:09 smithi172 ceph-mon[36566]: pgmap v225: 1 pgs: 1 remapped+peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:10.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:09 smithi082 ceph-mon[36471]: pgmap v225: 1 pgs: 1 remapped+peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:10.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:09 smithi043 ceph-mon[31823]: pgmap v225: 1 pgs: 1 remapped+peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:12.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:11 smithi172 ceph-mon[36566]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T22:31:12.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:11 smithi082 ceph-mon[36471]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T22:31:12.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:11 smithi043 ceph-mon[31823]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T22:31:13.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:12 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3897060664' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "19aa383e-a0f6-45ea-96cc-e7956535caa6"}]: dispatch 2024-03-28T22:31:13.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:12 smithi043 ceph-mon[31823]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "19aa383e-a0f6-45ea-96cc-e7956535caa6"}]: dispatch 2024-03-28T22:31:13.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:12 smithi043 ceph-mon[31823]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "19aa383e-a0f6-45ea-96cc-e7956535caa6"}]': finished 2024-03-28T22:31:13.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:12 smithi043 ceph-mon[31823]: osdmap e66: 10 total, 9 up, 10 in 2024-03-28T22:31:13.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:12 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:31:13.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:12 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3897060664' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "19aa383e-a0f6-45ea-96cc-e7956535caa6"}]: dispatch 2024-03-28T22:31:13.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:12 smithi082 ceph-mon[36471]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "19aa383e-a0f6-45ea-96cc-e7956535caa6"}]: dispatch 2024-03-28T22:31:13.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:12 smithi082 ceph-mon[36471]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "19aa383e-a0f6-45ea-96cc-e7956535caa6"}]': finished 2024-03-28T22:31:13.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:12 smithi082 ceph-mon[36471]: osdmap e66: 10 total, 9 up, 10 in 2024-03-28T22:31:13.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:12 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:31:13.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:12 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3897060664' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "19aa383e-a0f6-45ea-96cc-e7956535caa6"}]: dispatch 2024-03-28T22:31:13.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:12 smithi172 ceph-mon[36566]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "19aa383e-a0f6-45ea-96cc-e7956535caa6"}]: dispatch 2024-03-28T22:31:13.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:12 smithi172 ceph-mon[36566]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "19aa383e-a0f6-45ea-96cc-e7956535caa6"}]': finished 2024-03-28T22:31:13.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:12 smithi172 ceph-mon[36566]: osdmap e66: 10 total, 9 up, 10 in 2024-03-28T22:31:13.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:12 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:31:14.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:13 smithi082 ceph-mon[36471]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-28T22:31:14.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:13 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/122089279' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:31:14.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:13 smithi043 ceph-mon[31823]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-28T22:31:14.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:13 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/122089279' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:31:14.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:13 smithi172 ceph-mon[36566]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-28T22:31:14.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:13 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/122089279' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:31:16.202 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:15 smithi082 ceph-mon[36471]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T22:31:16.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:15 smithi043 ceph-mon[31823]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T22:31:16.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:15 smithi172 ceph-mon[36566]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T22:31:18.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:17 smithi082 ceph-mon[36471]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-28T22:31:18.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:17 smithi043 ceph-mon[31823]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-28T22:31:18.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:17 smithi172 ceph-mon[36566]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-28T22:31:19.984 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:19 smithi043 ceph-mon[31823]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T22:31:20.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:19 smithi082 ceph-mon[36471]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T22:31:20.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:19 smithi172 ceph-mon[36566]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T22:31:22.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:21 smithi082 ceph-mon[36471]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:22.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:21 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T22:31:22.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:21 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:31:22.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:21 smithi043 ceph-mon[31823]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:22.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:21 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T22:31:22.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:21 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:31:22.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:21 smithi172 ceph-mon[36566]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:22.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T22:31:22.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:31:23.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:22 smithi082 ceph-mon[36471]: Deploying daemon osd.9 on smithi043 2024-03-28T22:31:23.243 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:22 smithi043 ceph-mon[31823]: Deploying daemon osd.9 on smithi043 2024-03-28T22:31:23.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:22 smithi172 ceph-mon[36566]: Deploying daemon osd.9 on smithi043 2024-03-28T22:31:24.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:23 smithi082 ceph-mon[36471]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:24.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:23 smithi043 ceph-mon[31823]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:24.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:23 smithi172 ceph-mon[36566]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:26.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:25 smithi082 ceph-mon[36471]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:26.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:25 smithi043 ceph-mon[31823]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:26.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:25 smithi172 ceph-mon[36566]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:28.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:27 smithi082 ceph-mon[36471]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:28.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:27 smithi043 ceph-mon[31823]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:28.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:27 smithi172 ceph-mon[36566]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:30.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:29 smithi082 ceph-mon[36471]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:30.215 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:29 smithi043 ceph-mon[31823]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:30.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:29 smithi172 ceph-mon[36566]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:31.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:31 smithi082 ceph-mon[36471]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:31.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:31 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:31.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:31 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:31:31.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:31 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:31.741 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:31 smithi043 ceph-mon[31823]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:31.741 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:31 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:31.741 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:31 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:31:31.741 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:31 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:31.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:31 smithi172 ceph-mon[36566]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:31.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:31 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:31.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:31 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:31:31.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:31 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:33.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:33 smithi043 ceph-mon[31823]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:33.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:33 smithi172 ceph-mon[36566]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:33.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:33 smithi082 ceph-mon[36471]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:34.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:34 smithi043 ceph-mon[31823]: from='osd.9 [v2:172.21.15.43:6826/3657791498,v1:172.21.15.43:6827/3657791498]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T22:31:34.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:34 smithi172 ceph-mon[36566]: from='osd.9 [v2:172.21.15.43:6826/3657791498,v1:172.21.15.43:6827/3657791498]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T22:31:34.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:34 smithi082 ceph-mon[36471]: from='osd.9 [v2:172.21.15.43:6826/3657791498,v1:172.21.15.43:6827/3657791498]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T22:31:35.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:35.422+0000 7ff141600700 1 -- 172.21.15.43:0/4008828745 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff144062680 con 0x7ff12c0a1e10 2024-03-28T22:31:35.422 INFO:teuthology.orchestra.run.smithi043.stdout:Created osd(s) 9 on host 'smithi043' 2024-03-28T22:31:35.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:35.425+0000 7ff14b600700 1 -- 172.21.15.43:0/4008828745 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff12c0a1e10 msgr2=0x7ff12c0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:31:35.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:35.425+0000 7ff14b600700 1 --2- 172.21.15.43:0/4008828745 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff12c0a1e10 0x7ff12c0a42d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff130009c60 tx=0x7ff130009260 comp rx=0 tx=0).stop 2024-03-28T22:31:35.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:35.425+0000 7ff14b600700 1 -- 172.21.15.43:0/4008828745 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1441044c0 msgr2=0x7ff14419a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:31:35.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:35.425+0000 7ff14b600700 1 --2- 172.21.15.43:0/4008828745 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1441044c0 0x7ff14419a5f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff13800ec20 tx=0x7ff13800ef30 comp rx=0 tx=0).stop 2024-03-28T22:31:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:35.426+0000 7ff14b600700 1 -- 172.21.15.43:0/4008828745 shutdown_connections 2024-03-28T22:31:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:35.426+0000 7ff14b600700 1 --2- 172.21.15.43:0/4008828745 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff12c0a1e10 0x7ff12c0a42d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:31:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:35.426+0000 7ff14b600700 1 --2- 172.21.15.43:0/4008828745 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff144106060 0x7ff1441a1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:31:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:35.426+0000 7ff14b600700 1 --2- 172.21.15.43:0/4008828745 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff1441044c0 0x7ff14419a5f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:31:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:35.426+0000 7ff14b600700 1 --2- 172.21.15.43:0/4008828745 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff1441056c0 0x7ff14419ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:31:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:35.426+0000 7ff14b600700 1 -- 172.21.15.43:0/4008828745 >> 172.21.15.43:0/4008828745 conn(0x7ff1440ffa60 msgr2=0x7ff144108960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:31:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:35.426+0000 7ff14b600700 1 -- 172.21.15.43:0/4008828745 shutdown_connections 2024-03-28T22:31:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:31:35.426+0000 7ff14b600700 1 -- 172.21.15.43:0/4008828745 wait complete. 2024-03-28T22:31:35.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:35 smithi043 ceph-mon[31823]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:35.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:35 smithi043 ceph-mon[31823]: from='osd.9 [v2:172.21.15.43:6826/3657791498,v1:172.21.15.43:6827/3657791498]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T22:31:35.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:35 smithi043 ceph-mon[31823]: osdmap e67: 10 total, 9 up, 10 in 2024-03-28T22:31:35.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:35 smithi043 ceph-mon[31823]: from='osd.9 [v2:172.21.15.43:6826/3657791498,v1:172.21.15.43:6827/3657791498]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-28T22:31:35.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:35 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:31:35.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:35 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:35.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:35 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:35.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:35 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:35.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:35 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:35.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:35 smithi172 ceph-mon[36566]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:35.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:35 smithi172 ceph-mon[36566]: from='osd.9 [v2:172.21.15.43:6826/3657791498,v1:172.21.15.43:6827/3657791498]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T22:31:35.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:35 smithi172 ceph-mon[36566]: osdmap e67: 10 total, 9 up, 10 in 2024-03-28T22:31:35.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:35 smithi172 ceph-mon[36566]: from='osd.9 [v2:172.21.15.43:6826/3657791498,v1:172.21.15.43:6827/3657791498]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-28T22:31:35.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:35 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:31:35.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:35 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:35.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:35 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:35.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:35 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:35.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:35 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:35.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:35 smithi082 ceph-mon[36471]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:35.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:35 smithi082 ceph-mon[36471]: from='osd.9 [v2:172.21.15.43:6826/3657791498,v1:172.21.15.43:6827/3657791498]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T22:31:35.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:35 smithi082 ceph-mon[36471]: osdmap e67: 10 total, 9 up, 10 in 2024-03-28T22:31:35.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:35 smithi082 ceph-mon[36471]: from='osd.9 [v2:172.21.15.43:6826/3657791498,v1:172.21.15.43:6827/3657791498]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-03-28T22:31:35.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:35 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:31:35.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:35 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:35.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:35 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:35.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:35 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:35.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:35 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:36.309 DEBUG:teuthology.orchestra.run.smithi043:osd.9> sudo journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.9.service 2024-03-28T22:31:36.316 INFO:tasks.cephadm:Deploying osd.10 on smithi082 with /dev/vg_nvme/lv_1... 2024-03-28T22:31:36.316 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T22:31:36.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:36 smithi043 ceph-mon[31823]: from='osd.9 [v2:172.21.15.43:6826/3657791498,v1:172.21.15.43:6827/3657791498]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-03-28T22:31:36.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:36 smithi043 ceph-mon[31823]: osdmap e68: 10 total, 9 up, 10 in 2024-03-28T22:31:36.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:36 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:31:36.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:36 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:31:36.566 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T22:31:36.687 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:36 smithi082 ceph-mon[36471]: from='osd.9 [v2:172.21.15.43:6826/3657791498,v1:172.21.15.43:6827/3657791498]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-03-28T22:31:36.687 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:36 smithi082 ceph-mon[36471]: osdmap e68: 10 total, 9 up, 10 in 2024-03-28T22:31:36.688 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:36 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:31:36.688 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:36 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:31:36.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:36 smithi172 ceph-mon[36566]: from='osd.9 [v2:172.21.15.43:6826/3657791498,v1:172.21.15.43:6827/3657791498]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-03-28T22:31:36.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:36 smithi172 ceph-mon[36566]: osdmap e68: 10 total, 9 up, 10 in 2024-03-28T22:31:36.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:36 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:31:36.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:36 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:31:37.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:37 smithi082 ceph-mon[36471]: purged_snaps scrub starts 2024-03-28T22:31:37.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:37 smithi082 ceph-mon[36471]: purged_snaps scrub ok 2024-03-28T22:31:37.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:37 smithi082 ceph-mon[36471]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:37.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:37 smithi082 ceph-mon[36471]: osd.9 [v2:172.21.15.43:6826/3657791498,v1:172.21.15.43:6827/3657791498] boot 2024-03-28T22:31:37.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:37 smithi082 ceph-mon[36471]: osdmap e69: 10 total, 10 up, 10 in 2024-03-28T22:31:37.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:37 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:31:37.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:37 smithi043 ceph-mon[31823]: purged_snaps scrub starts 2024-03-28T22:31:37.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:37 smithi043 ceph-mon[31823]: purged_snaps scrub ok 2024-03-28T22:31:37.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:37 smithi043 ceph-mon[31823]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:37.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:37 smithi043 ceph-mon[31823]: osd.9 [v2:172.21.15.43:6826/3657791498,v1:172.21.15.43:6827/3657791498] boot 2024-03-28T22:31:37.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:37 smithi043 ceph-mon[31823]: osdmap e69: 10 total, 10 up, 10 in 2024-03-28T22:31:37.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:37 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:31:37.798 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:37 smithi172 ceph-mon[36566]: purged_snaps scrub starts 2024-03-28T22:31:37.798 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:37 smithi172 ceph-mon[36566]: purged_snaps scrub ok 2024-03-28T22:31:37.798 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:37 smithi172 ceph-mon[36566]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-28T22:31:37.798 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:37 smithi172 ceph-mon[36566]: osd.9 [v2:172.21.15.43:6826/3657791498,v1:172.21.15.43:6827/3657791498] boot 2024-03-28T22:31:37.798 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:37 smithi172 ceph-mon[36566]: osdmap e69: 10 total, 10 up, 10 in 2024-03-28T22:31:37.798 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:37 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:31:38.700 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T22:31:38.700 INFO:teuthology.orchestra.run.smithi082.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T22:31:38.701 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10+0 records in 2024-03-28T22:31:38.701 INFO:teuthology.orchestra.run.smithi082.stderr:10+0 records out 2024-03-28T22:31:38.701 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129895 s, 807 MB/s 2024-03-28T22:31:38.701 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping successful for: 2024-03-28T22:31:38.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:38 smithi082 ceph-mon[36471]: osdmap e70: 10 total, 10 up, 10 in 2024-03-28T22:31:38.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:38 smithi043 ceph-mon[31823]: osdmap e70: 10 total, 10 up, 10 in 2024-03-28T22:31:38.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:38 smithi172 ceph-mon[36566]: osdmap e70: 10 total, 10 up, 10 in 2024-03-28T22:31:39.375 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi082:vg_nvme/lv_1 2024-03-28T22:31:39.618 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T22:31:39.682 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:39 smithi082 ceph-mon[36471]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:31:39.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:39 smithi043 ceph-mon[31823]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:31:39.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:39 smithi172 ceph-mon[36566]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:31:41.282 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.281+0000 7f081f40f700 1 -- 172.21.15.82:0/134925503 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0818101170 msgr2=0x7f0818101590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:31:41.282 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.281+0000 7f081f40f700 1 --2- 172.21.15.82:0/134925503 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0818101170 0x7f0818101590 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f08140099c0 tx=0x7f0814009cd0 comp rx=0 tx=0).stop 2024-03-28T22:31:41.283 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.281+0000 7f081f40f700 1 -- 172.21.15.82:0/134925503 shutdown_connections 2024-03-28T22:31:41.283 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.281+0000 7f081f40f700 1 --2- 172.21.15.82:0/134925503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0818102d10 0x7f08181095c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:31:41.283 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.281+0000 7f081f40f700 1 --2- 172.21.15.82:0/134925503 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0818101170 0x7f0818101590 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:31:41.283 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.281+0000 7f081f40f700 1 --2- 172.21.15.82:0/134925503 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0818102370 0x7f08181027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:31:41.283 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.281+0000 7f081f40f700 1 -- 172.21.15.82:0/134925503 >> 172.21.15.82:0/134925503 conn(0x7f08180fc710 msgr2=0x7f08180feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:31:41.283 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.281+0000 7f081f40f700 1 -- 172.21.15.82:0/134925503 shutdown_connections 2024-03-28T22:31:41.283 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.281+0000 7f081f40f700 1 -- 172.21.15.82:0/134925503 wait complete. 2024-03-28T22:31:41.283 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.282+0000 7f081f40f700 1 Processor -- start 2024-03-28T22:31:41.283 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.282+0000 7f081f40f700 1 -- start start 2024-03-28T22:31:41.283 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.282+0000 7f081f40f700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0818101170 0x7f081806aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:31:41.283 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.282+0000 7f081f40f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0818102370 0x7f081819d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:31:41.283 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.282+0000 7f081f40f700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0818102d10 0x7f081819f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:31:41.283 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.282+0000 7f081f40f700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f081810cdd0 con 0x7f0818102d10 2024-03-28T22:31:41.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.282+0000 7f081f40f700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f081810cc50 con 0x7f0818101170 2024-03-28T22:31:41.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.282+0000 7f081f40f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f081810cf50 con 0x7f0818102370 2024-03-28T22:31:41.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.282+0000 7f081d1ab700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0818101170 0x7f081806aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:31:41.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.282+0000 7f081d1ab700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0818101170 0x7f081806aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:54154/0 (socket says 172.21.15.82:54154) 2024-03-28T22:31:41.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.282+0000 7f081d1ab700 1 -- 172.21.15.82:0/1407527962 learned_addr learned my addr 172.21.15.82:0/1407527962 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-03-28T22:31:41.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.282+0000 7f081c9aa700 1 --2- 172.21.15.82:0/1407527962 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0818102370 0x7f081819d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:31:41.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.282+0000 7f081d1ab700 1 -- 172.21.15.82:0/1407527962 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0818102370 msgr2=0x7f081819d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:31:41.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.282+0000 7f081d1ab700 1 --2- 172.21.15.82:0/1407527962 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0818102370 0x7f081819d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:31:41.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.282+0000 7f081d1ab700 1 -- 172.21.15.82:0/1407527962 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0818102d10 msgr2=0x7f081819f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:31:41.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.282+0000 7f081d1ab700 1 --2- 172.21.15.82:0/1407527962 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0818102d10 0x7f081819f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:31:41.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.282+0000 7f081d1ab700 1 -- 172.21.15.82:0/1407527962 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08181a01d0 con 0x7f0818101170 2024-03-28T22:31:41.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.283+0000 7f081d1ab700 1 --2- 172.21.15.82:0/1407527962 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0818101170 0x7f081806aab0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0814003890 tx=0x7f0814009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:31:41.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.283+0000 7f080e7fc700 1 -- 172.21.15.82:0/1407527962 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0814027070 con 0x7f0818101170 2024-03-28T22:31:41.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.283+0000 7f080e7fc700 1 -- 172.21.15.82:0/1407527962 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08140058e0 con 0x7f0818101170 2024-03-28T22:31:41.285 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.283+0000 7f080e7fc700 1 -- 172.21.15.82:0/1407527962 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0814022ce0 con 0x7f0818101170 2024-03-28T22:31:41.285 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.283+0000 7f081f40f700 1 -- 172.21.15.82:0/1407527962 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08181a0460 con 0x7f0818101170 2024-03-28T22:31:41.285 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.283+0000 7f081f40f700 1 -- 172.21.15.82:0/1407527962 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08181a0a10 con 0x7f0818101170 2024-03-28T22:31:41.286 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.284+0000 7f081f40f700 1 -- 172.21.15.82:0/1407527962 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0818068f00 con 0x7f0818101170 2024-03-28T22:31:41.286 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.285+0000 7f080e7fc700 1 -- 172.21.15.82:0/1407527962 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f081402a070 con 0x7f0818101170 2024-03-28T22:31:41.286 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.285+0000 7f080e7fc700 1 --2- 172.21.15.82:0/1407527962 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f08040a1e90 0x7f08040a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:31:41.286 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.285+0000 7f080e7fc700 1 -- 172.21.15.82:0/1407527962 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 6261+0+0 (secure 0 0 0) 0x7f08140ce3a0 con 0x7f0818101170 2024-03-28T22:31:41.286 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.286+0000 7f081c9aa700 1 --2- 172.21.15.82:0/1407527962 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f08040a1e90 0x7f08040a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:31:41.286 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.287+0000 7f081c9aa700 1 --2- 172.21.15.82:0/1407527962 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f08040a1e90 0x7f08040a4350 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0808009a10 tx=0x7f0808009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:31:41.289 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.290+0000 7f080e7fc700 1 -- 172.21.15.82:0/1407527962 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0814013070 con 0x7f0818101170 2024-03-28T22:31:41.490 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:31:41.491+0000 7f081f40f700 1 -- 172.21.15.82:0/1407527962 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f0818062680 con 0x7f08040a1e90 2024-03-28T22:31:41.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:41 smithi043 ceph-mon[31823]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:31:41.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:41 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:31:41.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:41 smithi172 ceph-mon[36566]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:31:41.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:41 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:31:41.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:41 smithi082 ceph-mon[36471]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:31:41.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:41 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:31:42.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:42 smithi082 ceph-mon[36471]: from='client.24359 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:31:42.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:42 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:31:42.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:42 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:31:42.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:42 smithi082 ceph-mon[36471]: Detected new or changed devices on smithi043 2024-03-28T22:31:42.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:42 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:42.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:42 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:42.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:42 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:31:42.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:42 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:31:42.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:42 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:31:42.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:42 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:31:42.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:42 smithi082 ceph-mon[36471]: Adjusting osd_memory_target on smithi043 to 4273M 2024-03-28T22:31:42.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:42 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:42.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:42 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:31:42.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:42 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:31:42.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:42 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:42.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:42 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:31:42.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:42 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:31:42.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:42 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:31:42.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:42 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:42.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:42 smithi043 ceph-mon[31823]: from='client.24359 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:31:42.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:42 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:31:42.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:42 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:31:42.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:42 smithi043 ceph-mon[31823]: Detected new or changed devices on smithi043 2024-03-28T22:31:42.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:42 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:42.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:42 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:42.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:42 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:31:42.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:42 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:31:42.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:42 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:31:42.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:42 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:31:42.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:42 smithi043 ceph-mon[31823]: Adjusting osd_memory_target on smithi043 to 4273M 2024-03-28T22:31:42.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:42 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:42.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:42 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:31:42.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:42 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:31:42.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:42 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:42.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:42 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:31:42.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:42 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:31:42.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:42 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:31:42.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:42 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:42.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:42 smithi172 ceph-mon[36566]: from='client.24359 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:31:42.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:42 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:31:42.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:42 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:31:42.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:42 smithi172 ceph-mon[36566]: Detected new or changed devices on smithi043 2024-03-28T22:31:42.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:42 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:42.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:42 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:42.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:42 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:31:42.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:42 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:31:42.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:42 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:31:42.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:42 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:31:42.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:42 smithi172 ceph-mon[36566]: Adjusting osd_memory_target on smithi043 to 4273M 2024-03-28T22:31:42.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:42 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:42.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:42 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:31:42.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:42 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:31:42.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:42 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:42.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:42 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:31:42.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:42 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:31:42.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:42 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:31:42.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:42 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:44.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:43 smithi043 ceph-mon[31823]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:31:44.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:43 smithi172 ceph-mon[36566]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:31:44.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:43 smithi082 ceph-mon[36471]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:31:45.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:44 smithi043 ceph-mon[31823]: from='client.? 172.21.15.82:0/1423639442' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "666977e5-eaf7-41b9-a18f-c380e149345d"}]: dispatch 2024-03-28T22:31:45.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:44 smithi043 ceph-mon[31823]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "666977e5-eaf7-41b9-a18f-c380e149345d"}]: dispatch 2024-03-28T22:31:45.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:44 smithi043 ceph-mon[31823]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "666977e5-eaf7-41b9-a18f-c380e149345d"}]': finished 2024-03-28T22:31:45.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:44 smithi043 ceph-mon[31823]: osdmap e71: 11 total, 10 up, 11 in 2024-03-28T22:31:45.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:44 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:31:45.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:44 smithi172 ceph-mon[36566]: from='client.? 172.21.15.82:0/1423639442' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "666977e5-eaf7-41b9-a18f-c380e149345d"}]: dispatch 2024-03-28T22:31:45.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:44 smithi172 ceph-mon[36566]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "666977e5-eaf7-41b9-a18f-c380e149345d"}]: dispatch 2024-03-28T22:31:45.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:44 smithi172 ceph-mon[36566]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "666977e5-eaf7-41b9-a18f-c380e149345d"}]': finished 2024-03-28T22:31:45.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:44 smithi172 ceph-mon[36566]: osdmap e71: 11 total, 10 up, 11 in 2024-03-28T22:31:45.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:44 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:31:45.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:44 smithi082 ceph-mon[36471]: from='client.? 172.21.15.82:0/1423639442' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "666977e5-eaf7-41b9-a18f-c380e149345d"}]: dispatch 2024-03-28T22:31:45.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:44 smithi082 ceph-mon[36471]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "666977e5-eaf7-41b9-a18f-c380e149345d"}]: dispatch 2024-03-28T22:31:45.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:44 smithi082 ceph-mon[36471]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "666977e5-eaf7-41b9-a18f-c380e149345d"}]': finished 2024-03-28T22:31:45.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:44 smithi082 ceph-mon[36471]: osdmap e71: 11 total, 10 up, 11 in 2024-03-28T22:31:45.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:44 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:31:46.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:45 smithi043 ceph-mon[31823]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:31:46.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:45 smithi043 ceph-mon[31823]: from='client.? 172.21.15.82:0/2843948174' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:31:46.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:45 smithi172 ceph-mon[36566]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:31:46.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:45 smithi172 ceph-mon[36566]: from='client.? 172.21.15.82:0/2843948174' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:31:46.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:45 smithi082 ceph-mon[36471]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:31:46.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:45 smithi082 ceph-mon[36471]: from='client.? 172.21.15.82:0/2843948174' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:31:48.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:47 smithi043 ceph-mon[31823]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:31:48.334 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:47 smithi082 ceph-mon[36471]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:31:48.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:47 smithi172 ceph-mon[36566]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:31:50.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:49 smithi043 ceph-mon[31823]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:31:50.291 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:49 smithi082 ceph-mon[36471]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:31:50.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:49 smithi172 ceph-mon[36566]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:31:52.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:51 smithi043 ceph-mon[31823]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:31:52.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:51 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T22:31:52.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:51 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:31:52.382 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:51 smithi082 ceph-mon[36471]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:31:52.383 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:51 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T22:31:52.383 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:51 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:31:52.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:51 smithi172 ceph-mon[36566]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:31:52.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:51 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T22:31:52.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:51 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:31:53.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:53 smithi043 ceph-mon[31823]: Deploying daemon osd.10 on smithi082 2024-03-28T22:31:53.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:53 smithi172 ceph-mon[36566]: Deploying daemon osd.10 on smithi082 2024-03-28T22:31:53.436 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:53 smithi082 ceph-mon[36471]: Deploying daemon osd.10 on smithi082 2024-03-28T22:31:54.040 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:54 smithi082 ceph-mon[36471]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:31:54.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:54 smithi043 ceph-mon[31823]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:31:54.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:54 smithi172 ceph-mon[36566]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:31:56.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:56 smithi082 ceph-mon[36471]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:31:56.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:56 smithi043 ceph-mon[31823]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:31:56.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:56 smithi172 ceph-mon[36566]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:31:58.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:58 smithi082 ceph-mon[36471]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:31:58.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:58 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:58.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:58 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:58.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:31:58 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:31:58.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:58 smithi043 ceph-mon[31823]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:31:58.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:58 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:58.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:58 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:58.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:31:58 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:31:58.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:58 smithi172 ceph-mon[36566]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:31:58.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:58 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:58.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:58 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:31:58.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:31:58 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:32:00.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:00 smithi043 ceph-mon[31823]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:32:00.306 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:00 smithi082 ceph-mon[36471]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:32:00.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:00 smithi172 ceph-mon[36566]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:32:01.432 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:32:01.433+0000 7f080e7fc700 1 -- 172.21.15.82:0/1407527962 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f0818062680 con 0x7f08040a1e90 2024-03-28T22:32:01.433 INFO:teuthology.orchestra.run.smithi082.stdout:Created osd(s) 10 on host 'smithi082' 2024-03-28T22:32:01.434 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:32:01.435+0000 7f081f40f700 1 -- 172.21.15.82:0/1407527962 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f08040a1e90 msgr2=0x7f08040a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:01.434 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:32:01.435+0000 7f081f40f700 1 --2- 172.21.15.82:0/1407527962 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f08040a1e90 0x7f08040a4350 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0808009a10 tx=0x7f0808009190 comp rx=0 tx=0).stop 2024-03-28T22:32:01.434 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:32:01.435+0000 7f081f40f700 1 -- 172.21.15.82:0/1407527962 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0818101170 msgr2=0x7f081806aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:01.434 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:32:01.435+0000 7f081f40f700 1 --2- 172.21.15.82:0/1407527962 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0818101170 0x7f081806aab0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0814003890 tx=0x7f0814009580 comp rx=0 tx=0).stop 2024-03-28T22:32:01.434 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:32:01.435+0000 7f081f40f700 1 -- 172.21.15.82:0/1407527962 shutdown_connections 2024-03-28T22:32:01.434 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:32:01.435+0000 7f081f40f700 1 --2- 172.21.15.82:0/1407527962 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f08040a1e90 0x7f08040a4350 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:01.434 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:32:01.435+0000 7f081f40f700 1 --2- 172.21.15.82:0/1407527962 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0818102d10 0x7f081819f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:01.434 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:32:01.435+0000 7f081f40f700 1 --2- 172.21.15.82:0/1407527962 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0818101170 0x7f081806aab0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:01.434 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:32:01.435+0000 7f081f40f700 1 --2- 172.21.15.82:0/1407527962 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0818102370 0x7f081819d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:01.435 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:32:01.435+0000 7f081f40f700 1 -- 172.21.15.82:0/1407527962 >> 172.21.15.82:0/1407527962 conn(0x7f08180fc710 msgr2=0x7f08181056f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:32:01.435 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:32:01.435+0000 7f081f40f700 1 -- 172.21.15.82:0/1407527962 shutdown_connections 2024-03-28T22:32:01.435 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-28T22:32:01.435+0000 7f081f40f700 1 -- 172.21.15.82:0/1407527962 wait complete. 2024-03-28T22:32:01.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:01 smithi082 ceph-mon[36471]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:32:01.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:01 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:01.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:01 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:01.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:01 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:01.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:01 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:32:01.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:01 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:01.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:01 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:32:01.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:01 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:01.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:01 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:01.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:01 smithi082 ceph-mon[36471]: from='osd.10 [v2:172.21.15.82:6824/3454954047,v1:172.21.15.82:6825/3454954047]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T22:32:01.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:01 smithi082 ceph-mon[36471]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T22:32:01.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:01 smithi172 ceph-mon[36566]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:32:01.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:01 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:01.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:01 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:01.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:01 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:01.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:01 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:32:01.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:01 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:01.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:01 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:32:01.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:01 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:01.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:01 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:01.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:01 smithi172 ceph-mon[36566]: from='osd.10 [v2:172.21.15.82:6824/3454954047,v1:172.21.15.82:6825/3454954047]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T22:32:01.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:01 smithi172 ceph-mon[36566]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T22:32:02.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:01 smithi043 ceph-mon[31823]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:32:02.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:01 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:02.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:01 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:02.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:01 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:02.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:01 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:32:02.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:01 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:02.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:01 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:32:02.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:01 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:02.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:01 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:02.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:01 smithi043 ceph-mon[31823]: from='osd.10 [v2:172.21.15.82:6824/3454954047,v1:172.21.15.82:6825/3454954047]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T22:32:02.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:01 smithi043 ceph-mon[31823]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T22:32:02.096 DEBUG:teuthology.orchestra.run.smithi082:osd.10> sudo journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.10.service 2024-03-28T22:32:02.098 INFO:tasks.cephadm:Deploying osd.11 on smithi172 with /dev/vg_nvme/lv_1... 2024-03-28T22:32:02.098 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T22:32:02.346 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T22:32:03.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:03 smithi043 ceph-mon[31823]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:32:03.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:03 smithi043 ceph-mon[31823]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T22:32:03.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:03 smithi043 ceph-mon[31823]: from='osd.10 [v2:172.21.15.82:6824/3454954047,v1:172.21.15.82:6825/3454954047]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-28T22:32:03.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:03 smithi043 ceph-mon[31823]: osdmap e72: 11 total, 10 up, 11 in 2024-03-28T22:32:03.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:03 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:32:03.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:03 smithi043 ceph-mon[31823]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-28T22:32:03.591 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:03 smithi082 ceph-mon[36471]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:32:03.591 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:03 smithi082 ceph-mon[36471]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T22:32:03.592 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:03 smithi082 ceph-mon[36471]: from='osd.10 [v2:172.21.15.82:6824/3454954047,v1:172.21.15.82:6825/3454954047]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-28T22:32:03.592 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:03 smithi082 ceph-mon[36471]: osdmap e72: 11 total, 10 up, 11 in 2024-03-28T22:32:03.592 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:03 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:32:03.592 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:03 smithi082 ceph-mon[36471]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-28T22:32:03.592 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:32:03 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:32:03.400+0000 7f8970441700 -1 osd.10 0 waiting for initial osdmap 2024-03-28T22:32:03.592 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:32:03 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:32:03.414+0000 7f896c234700 -1 osd.10 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T22:32:03.732 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:03 smithi172 ceph-mon[36566]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:32:03.732 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:03 smithi172 ceph-mon[36566]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T22:32:03.732 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:03 smithi172 ceph-mon[36566]: from='osd.10 [v2:172.21.15.82:6824/3454954047,v1:172.21.15.82:6825/3454954047]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-28T22:32:03.732 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:03 smithi172 ceph-mon[36566]: osdmap e72: 11 total, 10 up, 11 in 2024-03-28T22:32:03.732 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:03 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:32:03.732 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:03 smithi172 ceph-mon[36566]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-28T22:32:04.451 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T22:32:04.452 INFO:teuthology.orchestra.run.smithi172.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T22:32:04.452 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10+0 records in 2024-03-28T22:32:04.452 INFO:teuthology.orchestra.run.smithi172.stderr:10+0 records out 2024-03-28T22:32:04.452 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113691 s, 922 MB/s 2024-03-28T22:32:04.452 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping successful for: 2024-03-28T22:32:04.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:04 smithi082 ceph-mon[36471]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-03-28T22:32:04.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:04 smithi082 ceph-mon[36471]: osdmap e73: 11 total, 10 up, 11 in 2024-03-28T22:32:04.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:04 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:32:04.754 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:04 smithi172 ceph-mon[36566]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-03-28T22:32:04.754 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:04 smithi172 ceph-mon[36566]: osdmap e73: 11 total, 10 up, 11 in 2024-03-28T22:32:04.754 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:04 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:32:04.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:04 smithi043 ceph-mon[31823]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-03-28T22:32:04.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:04 smithi043 ceph-mon[31823]: osdmap e73: 11 total, 10 up, 11 in 2024-03-28T22:32:04.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:04 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:32:05.138 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi172:vg_nvme/lv_1 2024-03-28T22:32:05.379 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T22:32:05.661 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:05 smithi172 ceph-mon[36566]: purged_snaps scrub starts 2024-03-28T22:32:05.662 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:05 smithi172 ceph-mon[36566]: purged_snaps scrub ok 2024-03-28T22:32:05.662 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:05 smithi172 ceph-mon[36566]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:32:05.662 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:05 smithi172 ceph-mon[36566]: osd.10 [v2:172.21.15.82:6824/3454954047,v1:172.21.15.82:6825/3454954047] boot 2024-03-28T22:32:05.662 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:05 smithi172 ceph-mon[36566]: osdmap e74: 11 total, 11 up, 11 in 2024-03-28T22:32:05.662 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:05 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:32:05.662 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:05 smithi172 ceph-mon[36566]: Detected new or changed devices on smithi082 2024-03-28T22:32:05.662 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:05 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:05.662 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:05 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:05.662 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:05 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:32:05.662 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:05 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:32:05.662 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:05 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:32:05.662 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:05 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:32:05.662 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:05 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:05.663 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:05 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:05.663 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:05 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:32:05.663 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:05 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:05.663 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:05 smithi172 ceph-mon[36566]: osdmap e75: 11 total, 11 up, 11 in 2024-03-28T22:32:05.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:05 smithi082 ceph-mon[36471]: purged_snaps scrub starts 2024-03-28T22:32:05.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:05 smithi082 ceph-mon[36471]: purged_snaps scrub ok 2024-03-28T22:32:05.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:05 smithi082 ceph-mon[36471]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:32:05.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:05 smithi082 ceph-mon[36471]: osd.10 [v2:172.21.15.82:6824/3454954047,v1:172.21.15.82:6825/3454954047] boot 2024-03-28T22:32:05.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:05 smithi082 ceph-mon[36471]: osdmap e74: 11 total, 11 up, 11 in 2024-03-28T22:32:05.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:05 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:32:05.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:05 smithi082 ceph-mon[36471]: Detected new or changed devices on smithi082 2024-03-28T22:32:05.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:05 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:05.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:05 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:05.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:05 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:32:05.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:05 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:32:05.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:05 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:32:05.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:05 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:32:05.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:05 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:05.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:05 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:05.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:05 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:32:05.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:05 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:05.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:05 smithi082 ceph-mon[36471]: osdmap e75: 11 total, 11 up, 11 in 2024-03-28T22:32:05.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:05 smithi043 ceph-mon[31823]: purged_snaps scrub starts 2024-03-28T22:32:05.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:05 smithi043 ceph-mon[31823]: purged_snaps scrub ok 2024-03-28T22:32:05.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:05 smithi043 ceph-mon[31823]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-28T22:32:05.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:05 smithi043 ceph-mon[31823]: osd.10 [v2:172.21.15.82:6824/3454954047,v1:172.21.15.82:6825/3454954047] boot 2024-03-28T22:32:05.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:05 smithi043 ceph-mon[31823]: osdmap e74: 11 total, 11 up, 11 in 2024-03-28T22:32:05.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:05 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:32:05.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:05 smithi043 ceph-mon[31823]: Detected new or changed devices on smithi082 2024-03-28T22:32:05.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:05 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:05.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:05 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:05.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:05 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:32:05.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:05 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:32:05.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:05 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:32:05.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:05 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:32:05.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:05 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:05.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:05 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:05.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:05 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:32:05.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:05 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:05.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:05 smithi043 ceph-mon[31823]: osdmap e75: 11 total, 11 up, 11 in 2024-03-28T22:32:06.568 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:06 smithi172 ceph-mon[36566]: Adjusting osd_memory_target on smithi082 to 4270M 2024-03-28T22:32:06.568 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:06 smithi172 ceph-mon[36566]: osdmap e76: 11 total, 11 up, 11 in 2024-03-28T22:32:06.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:06 smithi082 ceph-mon[36471]: Adjusting osd_memory_target on smithi082 to 4270M 2024-03-28T22:32:06.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:06 smithi082 ceph-mon[36471]: osdmap e76: 11 total, 11 up, 11 in 2024-03-28T22:32:06.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:06 smithi043 ceph-mon[31823]: Adjusting osd_memory_target on smithi082 to 4270M 2024-03-28T22:32:06.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:06 smithi043 ceph-mon[31823]: osdmap e76: 11 total, 11 up, 11 in 2024-03-28T22:32:07.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.117+0000 7fd0d87c0700 1 -- 172.21.15.172:0/2666809537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0d0100910 msgr2=0x7fd0d0100d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:07.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.117+0000 7fd0d87c0700 1 --2- 172.21.15.172:0/2666809537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0d0100910 0x7fd0d0100d30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc00b0a0 tx=0x7fd0cc00b3b0 comp rx=0 tx=0).stop 2024-03-28T22:32:07.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.117+0000 7fd0d87c0700 1 -- 172.21.15.172:0/2666809537 shutdown_connections 2024-03-28T22:32:07.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.117+0000 7fd0d87c0700 1 --2- 172.21.15.172:0/2666809537 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0d01024b0 0x7fd0d0108d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:07.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.117+0000 7fd0d87c0700 1 --2- 172.21.15.172:0/2666809537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0d0100910 0x7fd0d0100d30 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:07.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.117+0000 7fd0d87c0700 1 --2- 172.21.15.172:0/2666809537 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0d0101b10 0x7fd0d0101f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:07.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.117+0000 7fd0d87c0700 1 -- 172.21.15.172:0/2666809537 >> 172.21.15.172:0/2666809537 conn(0x7fd0d00fbed0 msgr2=0x7fd0d00fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:32:07.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.117+0000 7fd0d87c0700 1 -- 172.21.15.172:0/2666809537 shutdown_connections 2024-03-28T22:32:07.118 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.117+0000 7fd0d87c0700 1 -- 172.21.15.172:0/2666809537 wait complete. 2024-03-28T22:32:07.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.117+0000 7fd0d87c0700 1 Processor -- start 2024-03-28T22:32:07.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.117+0000 7fd0d87c0700 1 -- start start 2024-03-28T22:32:07.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.118+0000 7fd0d87c0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0d0100910 0x7fd0d0198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:07.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.118+0000 7fd0d87c0700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0d0101b10 0x7fd0d0198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:07.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.118+0000 7fd0d87c0700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0d01024b0 0x7fd0d019fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:07.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.118+0000 7fd0d87c0700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d010bcd0 con 0x7fd0d01024b0 2024-03-28T22:32:07.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.118+0000 7fd0d87c0700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d010bb50 con 0x7fd0d0101b10 2024-03-28T22:32:07.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.118+0000 7fd0d87c0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d010be50 con 0x7fd0d0100910 2024-03-28T22:32:07.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.118+0000 7fd0d655c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0d0100910 0x7fd0d0198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:07.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.118+0000 7fd0d655c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0d0100910 0x7fd0d0198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:57900/0 (socket says 172.21.15.172:57900) 2024-03-28T22:32:07.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.118+0000 7fd0d655c700 1 -- 172.21.15.172:0/747296898 learned_addr learned my addr 172.21.15.172:0/747296898 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-28T22:32:07.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.118+0000 7fd0d5d5b700 1 --2- 172.21.15.172:0/747296898 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0d0101b10 0x7fd0d0198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:07.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.118+0000 7fd0d655c700 1 -- 172.21.15.172:0/747296898 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0d0101b10 msgr2=0x7fd0d0198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:07.119 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.118+0000 7fd0d655c700 1 --2- 172.21.15.172:0/747296898 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0d0101b10 0x7fd0d0198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:07.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.118+0000 7fd0d655c700 1 -- 172.21.15.172:0/747296898 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0d01024b0 msgr2=0x7fd0d019fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:07.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.118+0000 7fd0d655c700 1 --2- 172.21.15.172:0/747296898 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0d01024b0 0x7fd0d019fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:07.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.118+0000 7fd0d655c700 1 -- 172.21.15.172:0/747296898 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0d01a0180 con 0x7fd0d0100910 2024-03-28T22:32:07.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.118+0000 7fd0d655c700 1 --2- 172.21.15.172:0/747296898 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0d0100910 0x7fd0d0198520 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc000f80 tx=0x7fd0cc00be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:32:07.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.118+0000 7fd0c77fe700 1 -- 172.21.15.172:0/747296898 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc021030 con 0x7fd0d0100910 2024-03-28T22:32:07.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.118+0000 7fd0c77fe700 1 -- 172.21.15.172:0/747296898 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0cc007b30 con 0x7fd0d0100910 2024-03-28T22:32:07.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.118+0000 7fd0d87c0700 1 -- 172.21.15.172:0/747296898 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0d01a0410 con 0x7fd0d0100910 2024-03-28T22:32:07.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.118+0000 7fd0c77fe700 1 -- 172.21.15.172:0/747296898 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc005040 con 0x7fd0d0100910 2024-03-28T22:32:07.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.118+0000 7fd0d87c0700 1 -- 172.21.15.172:0/747296898 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0d01a0990 con 0x7fd0d0100910 2024-03-28T22:32:07.120 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.119+0000 7fd0d87c0700 1 -- 172.21.15.172:0/747296898 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0d0068f00 con 0x7fd0d0100910 2024-03-28T22:32:07.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.120+0000 7fd0c77fe700 1 -- 172.21.15.172:0/747296898 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fd0cc024070 con 0x7fd0d0100910 2024-03-28T22:32:07.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.121+0000 7fd0c77fe700 1 --2- 172.21.15.172:0/747296898 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd0bc0a1e90 0x7fd0bc0a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:07.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.121+0000 7fd0c77fe700 1 -- 172.21.15.172:0/747296898 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 6694+0+0 (secure 0 0 0) 0x7fd0cc00f020 con 0x7fd0d0100910 2024-03-28T22:32:07.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.121+0000 7fd0d5d5b700 1 --2- 172.21.15.172:0/747296898 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd0bc0a1e90 0x7fd0bc0a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:07.121 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.122+0000 7fd0d5d5b700 1 --2- 172.21.15.172:0/747296898 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd0bc0a1e90 0x7fd0bc0a4350 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd0c0005950 tx=0x7fd0c00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:32:07.124 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.125+0000 7fd0c77fe700 1 -- 172.21.15.172:0/747296898 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd0cc09bc80 con 0x7fd0d0100910 2024-03-28T22:32:07.328 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:07.330+0000 7fd0d87c0700 1 -- 172.21.15.172:0/747296898 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fd0d0108740 con 0x7fd0bc0a1e90 2024-03-28T22:32:08.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:07 smithi172 ceph-mon[36566]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:32:08.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:07 smithi172 ceph-mon[36566]: osdmap e77: 11 total, 11 up, 11 in 2024-03-28T22:32:08.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:07 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:32:08.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:07 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:32:08.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:07 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:08.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:07 smithi082 ceph-mon[36471]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:32:08.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:07 smithi082 ceph-mon[36471]: osdmap e77: 11 total, 11 up, 11 in 2024-03-28T22:32:08.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:07 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:32:08.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:07 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:32:08.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:07 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:08.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:07 smithi043 ceph-mon[31823]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:32:08.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:07 smithi043 ceph-mon[31823]: osdmap e77: 11 total, 11 up, 11 in 2024-03-28T22:32:08.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:07 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T22:32:08.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:07 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T22:32:08.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:07 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:08.971 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:08 smithi172 ceph-mon[36566]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:32:08.971 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:08 smithi172 ceph-mon[36566]: osdmap e78: 11 total, 11 up, 11 in 2024-03-28T22:32:09.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:08 smithi082 ceph-mon[36471]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:32:09.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:08 smithi082 ceph-mon[36471]: osdmap e78: 11 total, 11 up, 11 in 2024-03-28T22:32:09.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:08 smithi043 ceph-mon[31823]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:32:09.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:08 smithi043 ceph-mon[31823]: osdmap e78: 11 total, 11 up, 11 in 2024-03-28T22:32:10.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:09 smithi172 ceph-mon[36566]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:32:10.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:09 smithi082 ceph-mon[36471]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:32:10.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:09 smithi043 ceph-mon[31823]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:32:11.151 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:10 smithi172 ceph-mon[36566]: from='client.? 172.21.15.172:0/4128370677' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "615cf3be-8076-470e-ad09-c880e7593717"}]: dispatch 2024-03-28T22:32:11.152 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:10 smithi172 ceph-mon[36566]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "615cf3be-8076-470e-ad09-c880e7593717"}]: dispatch 2024-03-28T22:32:11.152 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:10 smithi172 ceph-mon[36566]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "615cf3be-8076-470e-ad09-c880e7593717"}]': finished 2024-03-28T22:32:11.152 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:10 smithi172 ceph-mon[36566]: osdmap e79: 12 total, 11 up, 12 in 2024-03-28T22:32:11.152 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:10 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:32:11.152 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:10 smithi172 ceph-mon[36566]: from='client.? 172.21.15.172:0/4171572850' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:32:11.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:10 smithi082 ceph-mon[36471]: from='client.? 172.21.15.172:0/4128370677' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "615cf3be-8076-470e-ad09-c880e7593717"}]: dispatch 2024-03-28T22:32:11.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:10 smithi082 ceph-mon[36471]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "615cf3be-8076-470e-ad09-c880e7593717"}]: dispatch 2024-03-28T22:32:11.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:10 smithi082 ceph-mon[36471]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "615cf3be-8076-470e-ad09-c880e7593717"}]': finished 2024-03-28T22:32:11.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:10 smithi082 ceph-mon[36471]: osdmap e79: 12 total, 11 up, 12 in 2024-03-28T22:32:11.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:10 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:32:11.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:10 smithi082 ceph-mon[36471]: from='client.? 172.21.15.172:0/4171572850' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:32:11.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:10 smithi043 ceph-mon[31823]: from='client.? 172.21.15.172:0/4128370677' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "615cf3be-8076-470e-ad09-c880e7593717"}]: dispatch 2024-03-28T22:32:11.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:10 smithi043 ceph-mon[31823]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "615cf3be-8076-470e-ad09-c880e7593717"}]: dispatch 2024-03-28T22:32:11.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:10 smithi043 ceph-mon[31823]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "615cf3be-8076-470e-ad09-c880e7593717"}]': finished 2024-03-28T22:32:11.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:10 smithi043 ceph-mon[31823]: osdmap e79: 12 total, 11 up, 12 in 2024-03-28T22:32:11.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:10 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:32:11.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:10 smithi043 ceph-mon[31823]: from='client.? 172.21.15.172:0/4171572850' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T22:32:12.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:11 smithi172 ceph-mon[36566]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-28T22:32:12.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:11 smithi082 ceph-mon[36471]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-28T22:32:12.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:11 smithi043 ceph-mon[31823]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-28T22:32:14.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:13 smithi172 ceph-mon[36566]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-28T22:32:14.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:13 smithi082 ceph-mon[36471]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-28T22:32:14.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:13 smithi043 ceph-mon[31823]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-28T22:32:16.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:15 smithi172 ceph-mon[36566]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-28T22:32:16.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:15 smithi082 ceph-mon[36471]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-28T22:32:16.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:15 smithi043 ceph-mon[31823]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-28T22:32:18.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:17 smithi172 ceph-mon[36566]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-28T22:32:18.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:17 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T22:32:18.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:17 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:18.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:17 smithi082 ceph-mon[36471]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-28T22:32:18.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:17 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T22:32:18.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:17 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:18.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:17 smithi043 ceph-mon[31823]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-28T22:32:18.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:17 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T22:32:18.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:17 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:19.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:18 smithi172 ceph-mon[36566]: Deploying daemon osd.11 on smithi172 2024-03-28T22:32:19.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:18 smithi082 ceph-mon[36471]: Deploying daemon osd.11 on smithi172 2024-03-28T22:32:19.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:18 smithi043 ceph-mon[31823]: Deploying daemon osd.11 on smithi172 2024-03-28T22:32:20.182 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:19 smithi172 ceph-mon[36566]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T22:32:20.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:19 smithi082 ceph-mon[36471]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T22:32:20.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:19 smithi043 ceph-mon[31823]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T22:32:22.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:21 smithi172 ceph-mon[36566]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:32:22.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:21 smithi082 ceph-mon[36471]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:32:22.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:21 smithi043 ceph-mon[31823]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:32:24.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:23 smithi082 ceph-mon[36471]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:32:24.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:23 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:24.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:23 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:24.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:23 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:32:24.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:23 smithi043 ceph-mon[31823]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:32:24.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:23 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:24.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:23 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:24.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:23 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:32:24.372 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:23 smithi172 ceph-mon[36566]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:32:24.372 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:23 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:24.372 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:23 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:24.372 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:23 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:32:26.109 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:25 smithi172 ceph-mon[36566]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:32:26.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:25 smithi082 ceph-mon[36471]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:32:26.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:25 smithi043 ceph-mon[31823]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:32:27.416 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:27.417+0000 7fd0c77fe700 1 -- 172.21.15.172:0/747296898 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fd0d0108740 con 0x7fd0bc0a1e90 2024-03-28T22:32:27.416 INFO:teuthology.orchestra.run.smithi172.stdout:Created osd(s) 11 on host 'smithi172' 2024-03-28T22:32:27.419 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:27.421+0000 7fd0d87c0700 1 -- 172.21.15.172:0/747296898 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd0bc0a1e90 msgr2=0x7fd0bc0a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:27.419 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:27.421+0000 7fd0d87c0700 1 --2- 172.21.15.172:0/747296898 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd0bc0a1e90 0x7fd0bc0a4350 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd0c0005950 tx=0x7fd0c00058e0 comp rx=0 tx=0).stop 2024-03-28T22:32:27.419 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:27.421+0000 7fd0d87c0700 1 -- 172.21.15.172:0/747296898 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0d0100910 msgr2=0x7fd0d0198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:27.419 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:27.421+0000 7fd0d87c0700 1 --2- 172.21.15.172:0/747296898 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0d0100910 0x7fd0d0198520 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc000f80 tx=0x7fd0cc00be20 comp rx=0 tx=0).stop 2024-03-28T22:32:27.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:27.421+0000 7fd0d87c0700 1 -- 172.21.15.172:0/747296898 shutdown_connections 2024-03-28T22:32:27.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:27.421+0000 7fd0d87c0700 1 --2- 172.21.15.172:0/747296898 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd0bc0a1e90 0x7fd0bc0a4350 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:27.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:27.421+0000 7fd0d87c0700 1 --2- 172.21.15.172:0/747296898 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0d01024b0 0x7fd0d019fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:27.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:27.421+0000 7fd0d87c0700 1 --2- 172.21.15.172:0/747296898 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0d0100910 0x7fd0d0198520 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:27.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:27.421+0000 7fd0d87c0700 1 --2- 172.21.15.172:0/747296898 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0d0101b10 0x7fd0d0198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:27.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:27.421+0000 7fd0d87c0700 1 -- 172.21.15.172:0/747296898 >> 172.21.15.172:0/747296898 conn(0x7fd0d00fbed0 msgr2=0x7fd0d0107020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:32:27.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:27.421+0000 7fd0d87c0700 1 -- 172.21.15.172:0/747296898 shutdown_connections 2024-03-28T22:32:27.420 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:27.421+0000 7fd0d87c0700 1 -- 172.21.15.172:0/747296898 wait complete. 2024-03-28T22:32:28.046 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:27 smithi172 ceph-mon[36566]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:32:28.046 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:27 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:28.046 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:27 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:28.046 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:27 smithi172 ceph-mon[36566]: from='osd.11 [v2:172.21.15.172:6824/2563517144,v1:172.21.15.172:6825/2563517144]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T22:32:28.046 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:27 smithi172 ceph-mon[36566]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T22:32:28.072 DEBUG:teuthology.orchestra.run.smithi172:osd.11> sudo journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.11.service 2024-03-28T22:32:28.075 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-28T22:32:28.075 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-28T22:32:28.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:27 smithi082 ceph-mon[36471]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:32:28.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:27 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:28.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:27 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:28.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:27 smithi082 ceph-mon[36471]: from='osd.11 [v2:172.21.15.172:6824/2563517144,v1:172.21.15.172:6825/2563517144]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T22:32:28.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:27 smithi082 ceph-mon[36471]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T22:32:28.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:27 smithi043 ceph-mon[31823]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:32:28.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:27 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:28.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:27 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:28.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:27 smithi043 ceph-mon[31823]: from='osd.11 [v2:172.21.15.172:6824/2563517144,v1:172.21.15.172:6825/2563517144]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T22:32:28.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:27 smithi043 ceph-mon[31823]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T22:32:28.541 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:32:29.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:29 smithi043 ceph-mon[31823]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:32:29.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:29 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:29.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:29 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:29.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:29 smithi043 ceph-mon[31823]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T22:32:29.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:29 smithi043 ceph-mon[31823]: osdmap e80: 12 total, 11 up, 12 in 2024-03-28T22:32:29.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:29 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:32:29.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:29 smithi043 ceph-mon[31823]: from='osd.11 [v2:172.21.15.172:6824/2563517144,v1:172.21.15.172:6825/2563517144]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-28T22:32:29.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:29 smithi043 ceph-mon[31823]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-28T22:32:29.561 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:29 smithi172 ceph-mon[36566]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:32:29.561 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:29 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:29.562 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:29 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:29.562 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:29 smithi172 ceph-mon[36566]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T22:32:29.562 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:29 smithi172 ceph-mon[36566]: osdmap e80: 12 total, 11 up, 12 in 2024-03-28T22:32:29.562 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:29 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:32:29.562 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:29 smithi172 ceph-mon[36566]: from='osd.11 [v2:172.21.15.172:6824/2563517144,v1:172.21.15.172:6825/2563517144]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-28T22:32:29.562 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:29 smithi172 ceph-mon[36566]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-28T22:32:29.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:29 smithi082 ceph-mon[36471]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:32:29.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:29 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:29.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:29 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:29.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:29 smithi082 ceph-mon[36471]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T22:32:29.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:29 smithi082 ceph-mon[36471]: osdmap e80: 12 total, 11 up, 12 in 2024-03-28T22:32:29.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:29 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:32:29.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:29 smithi082 ceph-mon[36471]: from='osd.11 [v2:172.21.15.172:6824/2563517144,v1:172.21.15.172:6825/2563517144]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-28T22:32:29.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:29 smithi082 ceph-mon[36471]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-28T22:32:30.147 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:32:30 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:32:30.074+0000 7fdf635ae700 -1 osd.11 0 waiting for initial osdmap 2024-03-28T22:32:30.148 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:32:30 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:32:30.088+0000 7fdf5b399700 -1 osd.11 81 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T22:32:30.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:30 smithi043 ceph-mon[31823]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-28T22:32:30.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:30 smithi043 ceph-mon[31823]: osdmap e81: 12 total, 11 up, 12 in 2024-03-28T22:32:30.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:30 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:32:30.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:30 smithi043 ceph-mon[31823]: from='osd.11 ' entity='osd.11' 2024-03-28T22:32:30.601 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:30 smithi172 ceph-mon[36566]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-28T22:32:30.601 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:30 smithi172 ceph-mon[36566]: osdmap e81: 12 total, 11 up, 12 in 2024-03-28T22:32:30.601 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:30 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:32:30.601 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:30 smithi172 ceph-mon[36566]: from='osd.11 ' entity='osd.11' 2024-03-28T22:32:30.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:30 smithi082 ceph-mon[36471]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-28T22:32:30.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:30 smithi082 ceph-mon[36471]: osdmap e81: 12 total, 11 up, 12 in 2024-03-28T22:32:30.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:30 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:32:30.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:30 smithi082 ceph-mon[36471]: from='osd.11 ' entity='osd.11' 2024-03-28T22:32:30.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.874+0000 7f967c400700 1 -- 172.21.15.43:0/1755359641 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9674103480 msgr2=0x7f96741038e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:30.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.874+0000 7f967c400700 1 --2- 172.21.15.43:0/1755359641 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9674103480 0x7f96741038e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9660009990 tx=0x7f9660009ca0 comp rx=0 tx=0).stop 2024-03-28T22:32:30.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.874+0000 7f967c400700 1 -- 172.21.15.43:0/1755359641 shutdown_connections 2024-03-28T22:32:30.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.874+0000 7f967c400700 1 --2- 172.21.15.43:0/1755359641 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9674103e20 0x7f967410a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:30.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.874+0000 7f967c400700 1 --2- 172.21.15.43:0/1755359641 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9674102280 0x7f96741026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:30.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.874+0000 7f967c400700 1 --2- 172.21.15.43:0/1755359641 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9674103480 0x7f96741038e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:30.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.874+0000 7f967c400700 1 -- 172.21.15.43:0/1755359641 >> 172.21.15.43:0/1755359641 conn(0x7f96740fd7e0 msgr2=0x7f96740ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:32:30.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.874+0000 7f967c400700 1 -- 172.21.15.43:0/1755359641 shutdown_connections 2024-03-28T22:32:30.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.874+0000 7f967c400700 1 -- 172.21.15.43:0/1755359641 wait complete. 2024-03-28T22:32:30.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.875+0000 7f967c400700 1 Processor -- start 2024-03-28T22:32:30.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.876+0000 7f967c400700 1 -- start start 2024-03-28T22:32:30.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.876+0000 7f967c400700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9674102280 0x7f96741984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:30.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.876+0000 7f967c400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9674103480 0x7f9674198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:30.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.876+0000 7f967c400700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9674103e20 0x7f967419fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:30.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.876+0000 7f967c400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f967410deb0 con 0x7f9674103480 2024-03-28T22:32:30.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.876+0000 7f967c400700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f967410dd30 con 0x7f9674102280 2024-03-28T22:32:30.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.876+0000 7f967c400700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f967410e030 con 0x7f9674103e20 2024-03-28T22:32:30.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.876+0000 7f9679a00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9674102280 0x7f96741984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:30.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.877+0000 7f9679a00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9674102280 0x7f96741984c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34574/0 (socket says 172.21.15.43:34574) 2024-03-28T22:32:30.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.877+0000 7f9679a00700 1 -- 172.21.15.43:0/3949537881 learned_addr learned my addr 172.21.15.43:0/3949537881 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:32:30.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.877+0000 7f967a400700 1 --2- 172.21.15.43:0/3949537881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9674103e20 0x7f967419fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:30.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.877+0000 7f9679000700 1 --2- 172.21.15.43:0/3949537881 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9674103480 0x7f9674198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:30.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.877+0000 7f9679a00700 1 -- 172.21.15.43:0/3949537881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9674103e20 msgr2=0x7f967419fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:30.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.877+0000 7f9679a00700 1 --2- 172.21.15.43:0/3949537881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9674103e20 0x7f967419fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:30.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.877+0000 7f9679a00700 1 -- 172.21.15.43:0/3949537881 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9674103480 msgr2=0x7f9674198a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:30.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.877+0000 7f9679a00700 1 --2- 172.21.15.43:0/3949537881 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9674103480 0x7f9674198a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:30.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.877+0000 7f9679a00700 1 -- 172.21.15.43:0/3949537881 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96741a0120 con 0x7f9674102280 2024-03-28T22:32:30.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.877+0000 7f9679a00700 1 --2- 172.21.15.43:0/3949537881 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9674102280 0x7f96741984c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f966800ea50 tx=0x7f966800ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:32:30.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.878+0000 7f9672000700 1 -- 172.21.15.43:0/3949537881 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966800cbd0 con 0x7f9674102280 2024-03-28T22:32:30.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.878+0000 7f9672000700 1 -- 172.21.15.43:0/3949537881 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f966800cd70 con 0x7f9674102280 2024-03-28T22:32:30.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.878+0000 7f9672000700 1 -- 172.21.15.43:0/3949537881 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96680187a0 con 0x7f9674102280 2024-03-28T22:32:30.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.878+0000 7f967c400700 1 -- 172.21.15.43:0/3949537881 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96741a0410 con 0x7f9674102280 2024-03-28T22:32:30.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.878+0000 7f967c400700 1 -- 172.21.15.43:0/3949537881 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96741a0900 con 0x7f9674102280 2024-03-28T22:32:30.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.879+0000 7f967c400700 1 -- 172.21.15.43:0/3949537881 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9674068f00 con 0x7f9674102280 2024-03-28T22:32:30.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.881+0000 7f9672000700 1 -- 172.21.15.43:0/3949537881 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f966801f020 con 0x7f9674102280 2024-03-28T22:32:30.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.882+0000 7f9672000700 1 --2- 172.21.15.43:0/3949537881 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f965c0a1e90 0x7f965c0a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:30.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.883+0000 7f9672000700 1 -- 172.21.15.43:0/3949537881 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 6847+0+0 (secure 0 0 0) 0x7f96680cd8b0 con 0x7f9674102280 2024-03-28T22:32:30.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.883+0000 7f9679000700 1 --2- 172.21.15.43:0/3949537881 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f965c0a1e90 0x7f965c0a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:30.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.883+0000 7f9679000700 1 --2- 172.21.15.43:0/3949537881 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f965c0a1e90 0x7f965c0a4350 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f966000b2e0 tx=0x7f9660005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:32:30.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:30.887+0000 7f9672000700 1 -- 172.21.15.43:0/3949537881 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9668053280 con 0x7f9674102280 2024-03-28T22:32:31.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:31.120+0000 7f967c400700 1 -- 172.21.15.43:0/3949537881 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f9674062e80 con 0x7f9674102280 2024-03-28T22:32:31.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:31.121+0000 7f9672000700 1 -- 172.21.15.43:0/3949537881 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v82) ==== 74+0+133 (secure 0 0 0) 0x7f966809b200 con 0x7f9674102280 2024-03-28T22:32:31.119 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:32:31.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:31.124+0000 7f967c400700 1 -- 172.21.15.43:0/3949537881 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f965c0a1e90 msgr2=0x7f965c0a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:31.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:31.124+0000 7f967c400700 1 --2- 172.21.15.43:0/3949537881 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f965c0a1e90 0x7f965c0a4350 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f966000b2e0 tx=0x7f9660005fb0 comp rx=0 tx=0).stop 2024-03-28T22:32:31.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:31.124+0000 7f967c400700 1 -- 172.21.15.43:0/3949537881 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9674102280 msgr2=0x7f96741984c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:31.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:31.124+0000 7f967c400700 1 --2- 172.21.15.43:0/3949537881 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9674102280 0x7f96741984c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f966800ea50 tx=0x7f966800ed60 comp rx=0 tx=0).stop 2024-03-28T22:32:31.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:31.124+0000 7f967c400700 1 -- 172.21.15.43:0/3949537881 shutdown_connections 2024-03-28T22:32:31.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:31.124+0000 7f967c400700 1 --2- 172.21.15.43:0/3949537881 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f965c0a1e90 0x7f965c0a4350 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:31.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:31.124+0000 7f967c400700 1 --2- 172.21.15.43:0/3949537881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9674103e20 0x7f967419fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:31.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:31.124+0000 7f967c400700 1 --2- 172.21.15.43:0/3949537881 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9674102280 0x7f96741984c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:31.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:31.124+0000 7f967c400700 1 --2- 172.21.15.43:0/3949537881 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9674103480 0x7f9674198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:31.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:31.124+0000 7f967c400700 1 -- 172.21.15.43:0/3949537881 >> 172.21.15.43:0/3949537881 conn(0x7f96740fd7e0 msgr2=0x7f9674106800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:32:31.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:31.124+0000 7f967c400700 1 -- 172.21.15.43:0/3949537881 shutdown_connections 2024-03-28T22:32:31.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:31.124+0000 7f967c400700 1 -- 172.21.15.43:0/3949537881 wait complete. 2024-03-28T22:32:31.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:31 smithi172 ceph-mon[36566]: purged_snaps scrub starts 2024-03-28T22:32:31.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:31 smithi172 ceph-mon[36566]: purged_snaps scrub ok 2024-03-28T22:32:31.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:31 smithi172 ceph-mon[36566]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:32:31.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:31 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:32:31.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:31 smithi172 ceph-mon[36566]: osd.11 [v2:172.21.15.172:6824/2563517144,v1:172.21.15.172:6825/2563517144] boot 2024-03-28T22:32:31.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:31 smithi172 ceph-mon[36566]: osdmap e82: 12 total, 12 up, 12 in 2024-03-28T22:32:31.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:31 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:32:31.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:31 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3949537881' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T22:32:31.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:31 smithi082 ceph-mon[36471]: purged_snaps scrub starts 2024-03-28T22:32:31.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:31 smithi082 ceph-mon[36471]: purged_snaps scrub ok 2024-03-28T22:32:31.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:31 smithi082 ceph-mon[36471]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:32:31.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:31 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:32:31.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:31 smithi082 ceph-mon[36471]: osd.11 [v2:172.21.15.172:6824/2563517144,v1:172.21.15.172:6825/2563517144] boot 2024-03-28T22:32:31.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:31 smithi082 ceph-mon[36471]: osdmap e82: 12 total, 12 up, 12 in 2024-03-28T22:32:31.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:31 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:32:31.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:31 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3949537881' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T22:32:31.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:31 smithi043 ceph-mon[31823]: purged_snaps scrub starts 2024-03-28T22:32:31.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:31 smithi043 ceph-mon[31823]: purged_snaps scrub ok 2024-03-28T22:32:31.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:31 smithi043 ceph-mon[31823]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-28T22:32:31.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:31 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:32:31.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:31 smithi043 ceph-mon[31823]: osd.11 [v2:172.21.15.172:6824/2563517144,v1:172.21.15.172:6825/2563517144] boot 2024-03-28T22:32:31.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:31 smithi043 ceph-mon[31823]: osdmap e82: 12 total, 12 up, 12 in 2024-03-28T22:32:31.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:31 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:32:31.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:31 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3949537881' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T22:32:31.953 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":82,"num_osds":12,"num_up_osds":12,"osd_up_since":1711665151,"num_in_osds":12,"osd_in_since":1711665130,"num_remapped_pgs":0} 2024-03-28T22:32:31.954 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T22:32:32.421 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:32:32.830 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:32 smithi043 ceph-mon[31823]: Detected new or changed devices on smithi172 2024-03-28T22:32:32.830 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:32 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:32.830 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:32 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:32.830 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:32 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:32:32.830 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:32 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:32:32.830 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:32 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:32:32.831 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:32 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:32:32.831 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:32 smithi043 ceph-mon[31823]: Adjusting osd_memory_target on smithi172 to 4270M 2024-03-28T22:32:32.831 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:32 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:32.831 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:32 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:32.831 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:32 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:32:32.831 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:32 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:32.831 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:32 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:32:32.831 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:32 smithi043 ceph-mon[31823]: osdmap e83: 12 total, 12 up, 12 in 2024-03-28T22:32:33.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:32 smithi172 ceph-mon[36566]: Detected new or changed devices on smithi172 2024-03-28T22:32:33.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:32 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:33.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:32 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:33.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:32 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:32:33.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:32 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:32:33.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:32 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:32:33.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:32 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:32:33.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:32 smithi172 ceph-mon[36566]: Adjusting osd_memory_target on smithi172 to 4270M 2024-03-28T22:32:33.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:32 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:33.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:32 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:33.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:32 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:32:33.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:32 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:33.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:32 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:32:33.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:32 smithi172 ceph-mon[36566]: osdmap e83: 12 total, 12 up, 12 in 2024-03-28T22:32:33.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:32 smithi082 ceph-mon[36471]: Detected new or changed devices on smithi172 2024-03-28T22:32:33.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:32 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:33.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:32 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:33.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:32 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:32:33.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:32 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:32:33.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:32 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:32:33.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:32 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:32:33.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:32 smithi082 ceph-mon[36471]: Adjusting osd_memory_target on smithi172 to 4270M 2024-03-28T22:32:33.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:32 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:33.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:32 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:33.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:32 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:32:33.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:32 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:33.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:32 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:32:33.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:32 smithi082 ceph-mon[36471]: osdmap e83: 12 total, 12 up, 12 in 2024-03-28T22:32:34.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:33 smithi172 ceph-mon[36566]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:34.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:33 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:34.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:33 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:32:34.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:33 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:34.168 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:33 smithi043 ceph-mon[31823]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:34.168 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:33 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:34.169 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:33 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:32:34.169 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:33 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:34.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:33 smithi082 ceph-mon[36471]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:34.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:33 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:34.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:33 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:32:34.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:33 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:34.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.708+0000 7f41b9600700 1 -- 172.21.15.43:0/929289379 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41b40745e0 msgr2=0x7f41b40749c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:34.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.708+0000 7f41b9600700 1 --2- 172.21.15.43:0/929289379 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41b40745e0 0x7f41b40749c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f41a4009990 tx=0x7f41a4009ca0 comp rx=0 tx=0).stop 2024-03-28T22:32:34.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.708+0000 7f41b9600700 1 -- 172.21.15.43:0/929289379 shutdown_connections 2024-03-28T22:32:34.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.708+0000 7f41b9600700 1 --2- 172.21.15.43:0/929289379 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41b4113130 0x7f41b4115520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:34.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.708+0000 7f41b9600700 1 --2- 172.21.15.43:0/929289379 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41b40745e0 0x7f41b40749c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:34.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.708+0000 7f41b9600700 1 --2- 172.21.15.43:0/929289379 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f41b4074f00 0x7f41b4112bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:34.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.708+0000 7f41b9600700 1 -- 172.21.15.43:0/929289379 >> 172.21.15.43:0/929289379 conn(0x7f41b40fdc10 msgr2=0x7f41b4100030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:32:34.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.708+0000 7f41b9600700 1 -- 172.21.15.43:0/929289379 shutdown_connections 2024-03-28T22:32:34.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.708+0000 7f41b9600700 1 -- 172.21.15.43:0/929289379 wait complete. 2024-03-28T22:32:34.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.709+0000 7f41b9600700 1 Processor -- start 2024-03-28T22:32:34.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.709+0000 7f41b9600700 1 -- start start 2024-03-28T22:32:34.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.710+0000 7f41b9600700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f41b40745e0 0x7f41b419ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:34.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.710+0000 7f41b9600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41b4074f00 0x7f41b419f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:34.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.710+0000 7f41b9600700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41b4113130 0x7f41b41a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.710+0000 7f41b9600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f41b4118ac0 con 0x7f41b4074f00 2024-03-28T22:32:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.710+0000 7f41b9600700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f41b4118940 con 0x7f41b40745e0 2024-03-28T22:32:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.710+0000 7f41b9600700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f41b4118c40 con 0x7f41b4113130 2024-03-28T22:32:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.710+0000 7f41ab400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41b4074f00 0x7f41b419f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.710+0000 7f41ab400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41b4074f00 0x7f41b419f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60294/0 (socket says 172.21.15.43:60294) 2024-03-28T22:32:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.710+0000 7f41ab400700 1 -- 172.21.15.43:0/1785137541 learned_addr learned my addr 172.21.15.43:0/1785137541 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:32:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.710+0000 7f41b3e00700 1 --2- 172.21.15.43:0/1785137541 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f41b40745e0 0x7f41b419ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.710+0000 7f41b8c00700 1 --2- 172.21.15.43:0/1785137541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41b4113130 0x7f41b41a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.710+0000 7f41ab400700 1 -- 172.21.15.43:0/1785137541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41b4113130 msgr2=0x7f41b41a37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.710+0000 7f41ab400700 1 --2- 172.21.15.43:0/1785137541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41b4113130 0x7f41b41a37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.710+0000 7f41ab400700 1 -- 172.21.15.43:0/1785137541 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f41b40745e0 msgr2=0x7f41b419ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.710+0000 7f41ab400700 1 --2- 172.21.15.43:0/1785137541 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f41b40745e0 0x7f41b419ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.710+0000 7f41ab400700 1 -- 172.21.15.43:0/1785137541 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41b41a3f20 con 0x7f41b4074f00 2024-03-28T22:32:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.710+0000 7f41b3e00700 1 --2- 172.21.15.43:0/1785137541 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f41b40745e0 0x7f41b419ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:32:34.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.711+0000 7f41ab400700 1 --2- 172.21.15.43:0/1785137541 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41b4074f00 0x7f41b419f450 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f419c00eb30 tx=0x7f419c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:32:34.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.711+0000 7f41b1600700 1 -- 172.21.15.43:0/1785137541 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f419c00cbd0 con 0x7f41b4074f00 2024-03-28T22:32:34.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.711+0000 7f41b1600700 1 -- 172.21.15.43:0/1785137541 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f419c004500 con 0x7f41b4074f00 2024-03-28T22:32:34.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.711+0000 7f41b1600700 1 -- 172.21.15.43:0/1785137541 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f419c010430 con 0x7f41b4074f00 2024-03-28T22:32:34.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.711+0000 7f41b9600700 1 -- 172.21.15.43:0/1785137541 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41b41a4210 con 0x7f41b4074f00 2024-03-28T22:32:34.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.711+0000 7f41b9600700 1 -- 172.21.15.43:0/1785137541 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41b41aba90 con 0x7f41b4074f00 2024-03-28T22:32:34.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.713+0000 7f41b9600700 1 -- 172.21.15.43:0/1785137541 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41b4068f00 con 0x7f41b4074f00 2024-03-28T22:32:34.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.715+0000 7f41b1600700 1 -- 172.21.15.43:0/1785137541 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f419c003f00 con 0x7f41b4074f00 2024-03-28T22:32:34.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.716+0000 7f41b1600700 1 --2- 172.21.15.43:0/1785137541 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f41980a1e90 0x7f41980a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:34.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.716+0000 7f41b1600700 1 -- 172.21.15.43:0/1785137541 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f419c0cdc90 con 0x7f41b4074f00 2024-03-28T22:32:34.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.716+0000 7f41b3e00700 1 --2- 172.21.15.43:0/1785137541 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f41980a1e90 0x7f41980a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:34.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.717+0000 7f41b3e00700 1 --2- 172.21.15.43:0/1785137541 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f41980a1e90 0x7f41980a4350 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f41a400b2e0 tx=0x7f41a4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:32:34.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.721+0000 7f41b1600700 1 -- 172.21.15.43:0/1785137541 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f419c09b4c0 con 0x7f41b4074f00 2024-03-28T22:32:34.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.947+0000 7f41b9600700 1 -- 172.21.15.43:0/1785137541 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f41b402db30 con 0x7f41b4074f00 2024-03-28T22:32:34.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.952+0000 7f41b1600700 1 -- 172.21.15.43:0/1785137541 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+18421 (secure 0 0 0) 0x7f419c09adc0 con 0x7f41b4074f00 2024-03-28T22:32:34.952 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:32:34.952 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":83,"fsid":"c2276428-ed51-11ee-95d0-87774f69a715","created":"2024-03-28T22:24:19.950084+0000","modified":"2024-03-28T22:32:32.384331+0000","last_up_change":"2024-03-28T22:32:31.078737+0000","last_in_change":"2024-03-28T22:32:10.147097+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:28:14.713290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b855adcd-786c-4e42-b8fe-a515a77ce979","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6803","nonce":2826124409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6805","nonce":2826124409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6809","nonce":2826124409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6807","nonce":2826124409}]},"public_addr":"172.21.15.43:6803/2826124409","cluster_addr":"172.21.15.43:6805/2826124409","heartbeat_back_addr":"172.21.15.43:6809/2826124409","heartbeat_front_addr":"172.21.15.43:6807/2826124409","state":["exists","up"]},{"osd":1,"uuid":"031c143d-8ad1-4b68-8db9-3c717137e3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6801","nonce":3043263644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6803","nonce":3043263644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6807","nonce":3043263644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6805","nonce":3043263644}]},"public_addr":"172.21.15.82:6801/3043263644","cluster_addr":"172.21.15.82:6803/3043263644","heartbeat_back_addr":"172.21.15.82:6807/3043263644","heartbeat_front_addr":"172.21.15.82:6805/3043263644","state":["exists","up"]},{"osd":2,"uuid":"20632bbb-f07a-4ca1-b0c4-ff9d508e211c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6801","nonce":4213953645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6803","nonce":4213953645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6807","nonce":4213953645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6805","nonce":4213953645}]},"public_addr":"172.21.15.172:6801/4213953645","cluster_addr":"172.21.15.172:6803/4213953645","heartbeat_back_addr":"172.21.15.172:6807/4213953645","heartbeat_front_addr":"172.21.15.172:6805/4213953645","state":["exists","up"]},{"osd":3,"uuid":"9a1f2312-4dbd-475b-8736-d5ebc8788e09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6811","nonce":1597280920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6813","nonce":1597280920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6817","nonce":1597280920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6815","nonce":1597280920}]},"public_addr":"172.21.15.43:6811/1597280920","cluster_addr":"172.21.15.43:6813/1597280920","heartbeat_back_addr":"172.21.15.43:6817/1597280920","heartbeat_front_addr":"172.21.15.43:6815/1597280920","state":["exists","up"]},{"osd":4,"uuid":"29140ead-7075-42b2-90c1-27cae0b29ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6809","nonce":215451895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6811","nonce":215451895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6815","nonce":215451895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6813","nonce":215451895}]},"public_addr":"172.21.15.82:6809/215451895","cluster_addr":"172.21.15.82:6811/215451895","heartbeat_back_addr":"172.21.15.82:6815/215451895","heartbeat_front_addr":"172.21.15.82:6813/215451895","state":["exists","up"]},{"osd":5,"uuid":"59644c61-93dc-4536-b2ad-0256b24e5cb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6809","nonce":1576148933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6811","nonce":1576148933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6815","nonce":1576148933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6813","nonce":1576148933}]},"public_addr":"172.21.15.172:6809/1576148933","cluster_addr":"172.21.15.172:6811/1576148933","heartbeat_back_addr":"172.21.15.172:6815/1576148933","heartbeat_front_addr":"172.21.15.172:6813/1576148933","state":["exists","up"]},{"osd":6,"uuid":"22d54e57-c3f1-4ecf-9921-8e815b48f92a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6819","nonce":791044279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6821","nonce":791044279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6825","nonce":791044279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6823","nonce":791044279}]},"public_addr":"172.21.15.43:6819/791044279","cluster_addr":"172.21.15.43:6821/791044279","heartbeat_back_addr":"172.21.15.43:6825/791044279","heartbeat_front_addr":"172.21.15.43:6823/791044279","state":["exists","up"]},{"osd":7,"uuid":"248cd1e2-0d6b-452f-a0a8-c51fe1b975ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6817","nonce":2157777323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6819","nonce":2157777323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6823","nonce":2157777323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6821","nonce":2157777323}]},"public_addr":"172.21.15.82:6817/2157777323","cluster_addr":"172.21.15.82:6819/2157777323","heartbeat_back_addr":"172.21.15.82:6823/2157777323","heartbeat_front_addr":"172.21.15.82:6821/2157777323","state":["exists","up"]},{"osd":8,"uuid":"d98aab6e-8bae-4c24-906c-4f492f047b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6817","nonce":1095328248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6819","nonce":1095328248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6823","nonce":1095328248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6821","nonce":1095328248}]},"public_addr":"172.21.15.172:6817/1095328248","cluster_addr":"172.21.15.172:6819/1095328248","heartbeat_back_addr":"172.21.15.172:6823/1095328248","heartbeat_front_addr":"172.21.15.172:6821/1095328248","state":["exists","up"]},{"osd":9,"uuid":"19aa383e-a0f6-45ea-96cc-e7956535caa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6827","nonce":3657791498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6829","nonce":3657791498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6833","nonce":3657791498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6831","nonce":3657791498}]},"public_addr":"172.21.15.43:6827/3657791498","cluster_addr":"172.21.15.43:6829/3657791498","heartbeat_back_addr":"172.21.15.43:6833/3657791498","heartbeat_front_addr":"172.21.15.43:6831/3657791498","state":["exists","up"]},{"osd":10,"uuid":"666977e5-eaf7-41b9-a18f-c380e149345d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6825","nonce":3454954047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6827","nonce":3454954047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6831","nonce":3454954047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6829","nonce":3454954047}]},"public_addr":"172.21.15.82:6825/3454954047","cluster_addr":"172.21.15.82:6827/3454954047","heartbeat_back_addr":"172.21.15.82:6831/3454954047","heartbeat_front_addr":"172.21.15.82:6829/3454954047","state":["exists","up"]},{"osd":11,"uuid":"615cf3be-8076-470e-ad09-c880e7593717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6825","nonce":2563517144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6827","nonce":2563517144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6831","nonce":2563517144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6829","nonce":2563517144}]},"public_addr":"172.21.15.172:6825/2563517144","cluster_addr":"172.21.15.172:6827/2563517144","heartbeat_back_addr":"172.21.15.172:6831/2563517144","heartbeat_front_addr":"172.21.15.172:6829/2563517144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:21.727405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:47.100259+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:10.615438+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:43.632144+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:09.487940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:34.604463+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:09.594452+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:35.709795+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:01.184202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:35.455322+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:02.413166+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:28.550754+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/578791143":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/4029313215":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/1574178482":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6801/3517712438":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6800/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/1375099638":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/2710117652":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6801/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/3215634168":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6800/3517712438":"2024-03-29T22:25:28.008215+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:32:34.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.956+0000 7f41b9600700 1 -- 172.21.15.43:0/1785137541 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f41980a1e90 msgr2=0x7f41980a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:34.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.956+0000 7f41b9600700 1 --2- 172.21.15.43:0/1785137541 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f41980a1e90 0x7f41980a4350 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f41a400b2e0 tx=0x7f41a4005fd0 comp rx=0 tx=0).stop 2024-03-28T22:32:34.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.956+0000 7f41b9600700 1 -- 172.21.15.43:0/1785137541 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41b4074f00 msgr2=0x7f41b419f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:34.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.956+0000 7f41b9600700 1 --2- 172.21.15.43:0/1785137541 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41b4074f00 0x7f41b419f450 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f419c00eb30 tx=0x7f419c00ee40 comp rx=0 tx=0).stop 2024-03-28T22:32:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.956+0000 7f41b9600700 1 -- 172.21.15.43:0/1785137541 shutdown_connections 2024-03-28T22:32:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.956+0000 7f41b9600700 1 --2- 172.21.15.43:0/1785137541 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f41980a1e90 0x7f41980a4350 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.956+0000 7f41b9600700 1 --2- 172.21.15.43:0/1785137541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41b4113130 0x7f41b41a37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.956+0000 7f41b9600700 1 --2- 172.21.15.43:0/1785137541 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f41b40745e0 0x7f41b419ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.956+0000 7f41b9600700 1 --2- 172.21.15.43:0/1785137541 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41b4074f00 0x7f41b419f450 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.956+0000 7f41b9600700 1 -- 172.21.15.43:0/1785137541 >> 172.21.15.43:0/1785137541 conn(0x7f41b40fdc10 msgr2=0x7f41b40fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:32:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.957+0000 7f41b9600700 1 -- 172.21.15.43:0/1785137541 shutdown_connections 2024-03-28T22:32:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:34.957+0000 7f41b9600700 1 -- 172.21.15.43:0/1785137541 wait complete. 2024-03-28T22:32:35.775 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T22:28:14.713290+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T22:32:35.775 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-28T22:32:36.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:35 smithi043 ceph-mon[31823]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:36.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:35 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1785137541' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:32:36.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:35 smithi172 ceph-mon[36566]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:36.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:35 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1785137541' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:32:36.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:35 smithi082 ceph-mon[36471]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:36.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:35 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1785137541' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:32:36.245 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:32:38.012 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:37 smithi043 ceph-mon[31823]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:38.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:37 smithi172 ceph-mon[36566]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:38.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:37 smithi082 ceph-mon[36471]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:38.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.535+0000 7f34b0a00700 1 -- 172.21.15.43:0/2571650664 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f34ac1024f0 msgr2=0x7f34ac1028d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:38.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.535+0000 7f34b0a00700 1 --2- 172.21.15.43:0/2571650664 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f34ac1024f0 0x7f34ac1028d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f349c009f90 tx=0x7f349c00b330 comp rx=0 tx=0).stop 2024-03-28T22:32:38.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.535+0000 7f34b0a00700 1 -- 172.21.15.43:0/2571650664 shutdown_connections 2024-03-28T22:32:38.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.535+0000 7f34b0a00700 1 --2- 172.21.15.43:0/2571650664 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34ac1075c0 0x7f34ac10f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:38.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.535+0000 7f34b0a00700 1 --2- 172.21.15.43:0/2571650664 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f34ac1024f0 0x7f34ac1028d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:38.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.535+0000 7f34b0a00700 1 --2- 172.21.15.43:0/2571650664 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34ac102ea0 0x7f34ac106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:38.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.535+0000 7f34b0a00700 1 -- 172.21.15.43:0/2571650664 >> 172.21.15.43:0/2571650664 conn(0x7f34ac0793a0 msgr2=0x7f34ac0797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:32:38.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.535+0000 7f34b0a00700 1 -- 172.21.15.43:0/2571650664 shutdown_connections 2024-03-28T22:32:38.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.535+0000 7f34b0a00700 1 -- 172.21.15.43:0/2571650664 wait complete. 2024-03-28T22:32:38.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.536+0000 7f34b0a00700 1 Processor -- start 2024-03-28T22:32:38.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.537+0000 7f34b0a00700 1 -- start start 2024-03-28T22:32:38.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.537+0000 7f34b0a00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34ac1024f0 0x7f34ac19e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:38.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.537+0000 7f34b0a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34ac102ea0 0x7f34ac19ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:38.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.537+0000 7f34b0a00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f34ac1075c0 0x7f34ac1988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:38.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.537+0000 7f34b0a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f34ac1125f0 con 0x7f34ac102ea0 2024-03-28T22:32:38.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.537+0000 7f34b0a00700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f34ac112470 con 0x7f34ac1075c0 2024-03-28T22:32:38.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.537+0000 7f34b0a00700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f34ac112770 con 0x7f34ac1024f0 2024-03-28T22:32:38.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.537+0000 7f34a9e00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34ac1024f0 0x7f34ac19e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:38.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.538+0000 7f34a9e00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34ac1024f0 0x7f34ac19e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:55052/0 (socket says 172.21.15.43:55052) 2024-03-28T22:32:38.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.538+0000 7f34a9e00700 1 -- 172.21.15.43:0/2599982735 learned_addr learned my addr 172.21.15.43:0/2599982735 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:32:38.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.538+0000 7f34a1400700 1 --2- 172.21.15.43:0/2599982735 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34ac102ea0 0x7f34ac19ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:38.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.538+0000 7f34aa800700 1 --2- 172.21.15.43:0/2599982735 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f34ac1075c0 0x7f34ac1988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:38.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.538+0000 7f34a9e00700 1 -- 172.21.15.43:0/2599982735 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f34ac1075c0 msgr2=0x7f34ac1988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:38.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.538+0000 7f34a9e00700 1 --2- 172.21.15.43:0/2599982735 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f34ac1075c0 0x7f34ac1988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:38.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.538+0000 7f34a9e00700 1 -- 172.21.15.43:0/2599982735 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34ac102ea0 msgr2=0x7f34ac19ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:38.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.538+0000 7f34a9e00700 1 --2- 172.21.15.43:0/2599982735 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34ac102ea0 0x7f34ac19ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:38.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.538+0000 7f34a9e00700 1 -- 172.21.15.43:0/2599982735 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34ac199010 con 0x7f34ac1024f0 2024-03-28T22:32:38.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.538+0000 7f34a1400700 1 --2- 172.21.15.43:0/2599982735 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34ac102ea0 0x7f34ac19ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:32:38.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.538+0000 7f34a9e00700 1 --2- 172.21.15.43:0/2599982735 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34ac1024f0 0x7f34ac19e6f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f349c00d010 tx=0x7f349c00bd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:32:38.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.539+0000 7f34a3400700 1 -- 172.21.15.43:0/2599982735 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f349c021070 con 0x7f34ac1024f0 2024-03-28T22:32:38.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.539+0000 7f34a3400700 1 -- 172.21.15.43:0/2599982735 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f349c007b40 con 0x7f34ac1024f0 2024-03-28T22:32:38.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.539+0000 7f34b0a00700 1 -- 172.21.15.43:0/2599982735 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34ac1992a0 con 0x7f34ac1024f0 2024-03-28T22:32:38.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.539+0000 7f34a3400700 1 -- 172.21.15.43:0/2599982735 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f349c005280 con 0x7f34ac1024f0 2024-03-28T22:32:38.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.539+0000 7f34b0a00700 1 -- 172.21.15.43:0/2599982735 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34ac1a55c0 con 0x7f34ac1024f0 2024-03-28T22:32:38.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.540+0000 7f34b0a00700 1 -- 172.21.15.43:0/2599982735 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34ac068f00 con 0x7f34ac1024f0 2024-03-28T22:32:38.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.542+0000 7f34a3400700 1 -- 172.21.15.43:0/2599982735 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f349c024070 con 0x7f34ac1024f0 2024-03-28T22:32:38.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.543+0000 7f34a3400700 1 --2- 172.21.15.43:0/2599982735 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f34900a1e90 0x7f34900a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:38.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.544+0000 7f34a3400700 1 -- 172.21.15.43:0/2599982735 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f349c00f020 con 0x7f34ac1024f0 2024-03-28T22:32:38.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.544+0000 7f34a1400700 1 --2- 172.21.15.43:0/2599982735 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f34900a1e90 0x7f34900a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:38.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.544+0000 7f34a1400700 1 --2- 172.21.15.43:0/2599982735 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f34900a1e90 0x7f34900a4350 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f34ac101de0 tx=0x7f3494009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:32:38.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.546+0000 7f34a3400700 1 -- 172.21.15.43:0/2599982735 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f349c02b020 con 0x7f34ac1024f0 2024-03-28T22:32:38.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.773+0000 7f34b0a00700 1 -- 172.21.15.43:0/2599982735 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f34ac199de0 con 0x7f34ac1024f0 2024-03-28T22:32:38.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.774+0000 7f34a3400700 1 -- 172.21.15.43:0/2599982735 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7f349c09cb00 con 0x7f34ac1024f0 2024-03-28T22:32:38.774 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-03-28T22:32:38.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.777+0000 7f34b0a00700 1 -- 172.21.15.43:0/2599982735 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f34900a1e90 msgr2=0x7f34900a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:38.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.777+0000 7f34b0a00700 1 --2- 172.21.15.43:0/2599982735 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f34900a1e90 0x7f34900a4350 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f34ac101de0 tx=0x7f3494009260 comp rx=0 tx=0).stop 2024-03-28T22:32:38.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.777+0000 7f34b0a00700 1 -- 172.21.15.43:0/2599982735 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34ac1024f0 msgr2=0x7f34ac19e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:38.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.777+0000 7f34b0a00700 1 --2- 172.21.15.43:0/2599982735 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34ac1024f0 0x7f34ac19e6f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f349c00d010 tx=0x7f349c00bd50 comp rx=0 tx=0).stop 2024-03-28T22:32:38.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.778+0000 7f34b0a00700 1 -- 172.21.15.43:0/2599982735 shutdown_connections 2024-03-28T22:32:38.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.778+0000 7f34b0a00700 1 --2- 172.21.15.43:0/2599982735 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f34900a1e90 0x7f34900a4350 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:38.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.778+0000 7f34b0a00700 1 --2- 172.21.15.43:0/2599982735 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f34ac1075c0 0x7f34ac1988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:38.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.778+0000 7f34b0a00700 1 --2- 172.21.15.43:0/2599982735 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34ac1024f0 0x7f34ac19e6f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:38.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.778+0000 7f34b0a00700 1 --2- 172.21.15.43:0/2599982735 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34ac102ea0 0x7f34ac19ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:38.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.778+0000 7f34b0a00700 1 -- 172.21.15.43:0/2599982735 >> 172.21.15.43:0/2599982735 conn(0x7f34ac0793a0 msgr2=0x7f34ac100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:32:38.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.778+0000 7f34b0a00700 1 -- 172.21.15.43:0/2599982735 shutdown_connections 2024-03-28T22:32:38.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:38.778+0000 7f34b0a00700 1 -- 172.21.15.43:0/2599982735 wait complete. 2024-03-28T22:32:39.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:38 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2599982735' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:32:39.174 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:38 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2599982735' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:32:39.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:38 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2599982735' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:32:39.575 INFO:tasks.cephadm:Adding mds.a on smithi043 2024-03-28T22:32:39.575 INFO:tasks.cephadm:Adding mds.d on smithi043 2024-03-28T22:32:39.575 INFO:tasks.cephadm:Adding mds.g on smithi043 2024-03-28T22:32:39.575 INFO:tasks.cephadm:Adding mds.j on smithi043 2024-03-28T22:32:39.575 INFO:tasks.cephadm:Adding mds.b on smithi082 2024-03-28T22:32:39.575 INFO:tasks.cephadm:Adding mds.e on smithi082 2024-03-28T22:32:39.575 INFO:tasks.cephadm:Adding mds.h on smithi082 2024-03-28T22:32:39.575 INFO:tasks.cephadm:Adding mds.k on smithi082 2024-03-28T22:32:39.576 INFO:tasks.cephadm:Adding mds.c on smithi172 2024-03-28T22:32:39.576 INFO:tasks.cephadm:Adding mds.f on smithi172 2024-03-28T22:32:39.576 INFO:tasks.cephadm:Adding mds.i on smithi172 2024-03-28T22:32:39.576 INFO:tasks.cephadm:Adding mds.l on smithi172 2024-03-28T22:32:39.576 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph orch apply mds all '12;smithi043=a;smithi043=d;smithi043=g;smithi043=j;smithi082=b;smithi082=e;smithi082=h;smithi082=k;smithi172=c;smithi172=f;smithi172=i;smithi172=l' 2024-03-28T22:32:39.814 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T22:32:39.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:39 smithi172 ceph-mon[36566]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:40.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:39 smithi082 ceph-mon[36471]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:40.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:39 smithi043 ceph-mon[31823]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:41.498 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.497+0000 7f0b369a0700 1 -- 172.21.15.172:0/689301636 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b30103440 msgr2=0x7f0b301038a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:41.498 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.497+0000 7f0b369a0700 1 --2- 172.21.15.172:0/689301636 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b30103440 0x7f0b301038a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0b20009990 tx=0x7f0b20009ca0 comp rx=0 tx=0).stop 2024-03-28T22:32:41.498 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.497+0000 7f0b369a0700 1 -- 172.21.15.172:0/689301636 shutdown_connections 2024-03-28T22:32:41.498 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.497+0000 7f0b369a0700 1 --2- 172.21.15.172:0/689301636 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0b30103de0 0x7f0b3010a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:41.498 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.497+0000 7f0b369a0700 1 --2- 172.21.15.172:0/689301636 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b30100460 0x7f0b30100880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:41.498 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.497+0000 7f0b369a0700 1 --2- 172.21.15.172:0/689301636 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b30103440 0x7f0b301038a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:41.498 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.497+0000 7f0b369a0700 1 -- 172.21.15.172:0/689301636 >> 172.21.15.172:0/689301636 conn(0x7f0b300fbde0 msgr2=0x7f0b300fe200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:32:41.498 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.497+0000 7f0b369a0700 1 -- 172.21.15.172:0/689301636 shutdown_connections 2024-03-28T22:32:41.499 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.497+0000 7f0b369a0700 1 -- 172.21.15.172:0/689301636 wait complete. 2024-03-28T22:32:41.499 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.498+0000 7f0b369a0700 1 Processor -- start 2024-03-28T22:32:41.499 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.498+0000 7f0b369a0700 1 -- start start 2024-03-28T22:32:41.499 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.498+0000 7f0b369a0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b30100460 0x7f0b30102da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:41.499 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.498+0000 7f0b369a0700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b30103440 0x7f0b301013f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:41.499 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.498+0000 7f0b369a0700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0b30103de0 0x7f0b30101930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:41.499 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.498+0000 7f0b369a0700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3010de00 con 0x7f0b30103440 2024-03-28T22:32:41.499 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.498+0000 7f0b369a0700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3010dc80 con 0x7f0b30103de0 2024-03-28T22:32:41.499 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.498+0000 7f0b369a0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3010df80 con 0x7f0b30100460 2024-03-28T22:32:41.499 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.498+0000 7f0b3599e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b30100460 0x7f0b30102da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:41.499 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.498+0000 7f0b3599e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b30100460 0x7f0b30102da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:57414/0 (socket says 172.21.15.172:57414) 2024-03-28T22:32:41.499 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.498+0000 7f0b3599e700 1 -- 172.21.15.172:0/2107362017 learned_addr learned my addr 172.21.15.172:0/2107362017 (peer_addr_for_me v2:172.21.15.172:0/0) 2024-03-28T22:32:41.499 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.499+0000 7f0b3599e700 1 -- 172.21.15.172:0/2107362017 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0b30103de0 msgr2=0x7f0b30101930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:41.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.499+0000 7f0b3599e700 1 --2- 172.21.15.172:0/2107362017 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0b30103de0 0x7f0b30101930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:41.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.499+0000 7f0b3599e700 1 -- 172.21.15.172:0/2107362017 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b30103440 msgr2=0x7f0b301013f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:41.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.499+0000 7f0b3599e700 1 --2- 172.21.15.172:0/2107362017 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b30103440 0x7f0b301013f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:41.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.499+0000 7f0b3599e700 1 -- 172.21.15.172:0/2107362017 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b301021e0 con 0x7f0b30100460 2024-03-28T22:32:41.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.499+0000 7f0b3599e700 1 --2- 172.21.15.172:0/2107362017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b30100460 0x7f0b30102da0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c00b500 tx=0x7f0b2c00b8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:32:41.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.499+0000 7f0b26ffd700 1 -- 172.21.15.172:0/2107362017 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c015020 con 0x7f0b30100460 2024-03-28T22:32:41.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.499+0000 7f0b26ffd700 1 -- 172.21.15.172:0/2107362017 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b2c011d10 con 0x7f0b30100460 2024-03-28T22:32:41.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.499+0000 7f0b369a0700 1 -- 172.21.15.172:0/2107362017 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b301a24f0 con 0x7f0b30100460 2024-03-28T22:32:41.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.499+0000 7f0b26ffd700 1 -- 172.21.15.172:0/2107362017 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c00f2e0 con 0x7f0b30100460 2024-03-28T22:32:41.500 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.499+0000 7f0b369a0700 1 -- 172.21.15.172:0/2107362017 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b301a2ab0 con 0x7f0b30100460 2024-03-28T22:32:41.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.500+0000 7f0b369a0700 1 -- 172.21.15.172:0/2107362017 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b30068f00 con 0x7f0b30100460 2024-03-28T22:32:41.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.501+0000 7f0b26ffd700 1 -- 172.21.15.172:0/2107362017 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f0b2c018020 con 0x7f0b30100460 2024-03-28T22:32:41.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.501+0000 7f0b26ffd700 1 --2- 172.21.15.172:0/2107362017 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f0b1c0a1e90 0x7f0b1c0a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:41.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.502+0000 7f0b26ffd700 1 -- 172.21.15.172:0/2107362017 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f0b2c0cd530 con 0x7f0b30100460 2024-03-28T22:32:41.501 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.502+0000 7f0b3519d700 1 --2- 172.21.15.172:0/2107362017 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f0b1c0a1e90 0x7f0b1c0a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:41.502 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.503+0000 7f0b3519d700 1 --2- 172.21.15.172:0/2107362017 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f0b1c0a1e90 0x7f0b1c0a4350 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0b200098e0 tx=0x7f0b20005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:32:41.503 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.505+0000 7f0b26ffd700 1 -- 172.21.15.172:0/2107362017 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0b2c01f020 con 0x7f0b30100460 2024-03-28T22:32:41.711 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.712+0000 7f0b369a0700 1 -- 172.21.15.172:0/2107362017 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi043=a;smithi043=d;smithi043=g;smithi043=j;smithi082=b;smithi082=e;smithi082=h;smithi082=k;smithi172=c;smithi172=f;smithi172=i;smithi172=l", "target": ["mon-mgr", ""]}) -- 0x7f0b30062880 con 0x7f0b1c0a1e90 2024-03-28T22:32:41.727 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.729+0000 7f0b26ffd700 1 -- 172.21.15.172:0/2107362017 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f0b30062880 con 0x7f0b1c0a1e90 2024-03-28T22:32:41.728 INFO:teuthology.orchestra.run.smithi172.stdout:Scheduled mds.all update... 2024-03-28T22:32:41.730 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.731+0000 7f0b369a0700 1 -- 172.21.15.172:0/2107362017 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f0b1c0a1e90 msgr2=0x7f0b1c0a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:41.730 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.731+0000 7f0b369a0700 1 --2- 172.21.15.172:0/2107362017 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f0b1c0a1e90 0x7f0b1c0a4350 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0b200098e0 tx=0x7f0b20005fb0 comp rx=0 tx=0).stop 2024-03-28T22:32:41.730 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.731+0000 7f0b369a0700 1 -- 172.21.15.172:0/2107362017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b30100460 msgr2=0x7f0b30102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:41.730 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.731+0000 7f0b369a0700 1 --2- 172.21.15.172:0/2107362017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b30100460 0x7f0b30102da0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c00b500 tx=0x7f0b2c00b8c0 comp rx=0 tx=0).stop 2024-03-28T22:32:41.730 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.731+0000 7f0b369a0700 1 -- 172.21.15.172:0/2107362017 shutdown_connections 2024-03-28T22:32:41.730 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.731+0000 7f0b369a0700 1 --2- 172.21.15.172:0/2107362017 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f0b1c0a1e90 0x7f0b1c0a4350 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:41.730 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.731+0000 7f0b369a0700 1 --2- 172.21.15.172:0/2107362017 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0b30103de0 0x7f0b30101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:41.730 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.731+0000 7f0b369a0700 1 --2- 172.21.15.172:0/2107362017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b30100460 0x7f0b30102da0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:41.730 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.731+0000 7f0b369a0700 1 --2- 172.21.15.172:0/2107362017 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b30103440 0x7f0b301013f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:41.730 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.731+0000 7f0b369a0700 1 -- 172.21.15.172:0/2107362017 >> 172.21.15.172:0/2107362017 conn(0x7f0b300fbde0 msgr2=0x7f0b30106670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:32:41.731 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.731+0000 7f0b369a0700 1 -- 172.21.15.172:0/2107362017 shutdown_connections 2024-03-28T22:32:41.731 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-28T22:32:41.731+0000 7f0b369a0700 1 -- 172.21.15.172:0/2107362017 wait complete. 2024-03-28T22:32:42.047 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:41 smithi172 ceph-mon[36566]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:42.047 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:41 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:42.047 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:41 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:32:42.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:41 smithi082 ceph-mon[36471]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:42.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:41 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:42.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:41 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:32:42.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:41 smithi043 ceph-mon[31823]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:42.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:41 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:42.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:41 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:32:42.375 DEBUG:teuthology.orchestra.run.smithi043:mds.a> sudo journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.a.service 2024-03-28T22:32:42.383 DEBUG:teuthology.orchestra.run.smithi043:mds.d> sudo journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.d.service 2024-03-28T22:32:42.428 DEBUG:teuthology.orchestra.run.smithi043:mds.g> sudo journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.g.service 2024-03-28T22:32:42.474 DEBUG:teuthology.orchestra.run.smithi043:mds.j> sudo journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.j.service 2024-03-28T22:32:42.520 DEBUG:teuthology.orchestra.run.smithi082:mds.b> sudo journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.b.service 2024-03-28T22:32:42.523 DEBUG:teuthology.orchestra.run.smithi082:mds.e> sudo journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.e.service 2024-03-28T22:32:42.566 DEBUG:teuthology.orchestra.run.smithi082:mds.h> sudo journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.h.service 2024-03-28T22:32:42.609 DEBUG:teuthology.orchestra.run.smithi082:mds.k> sudo journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.k.service 2024-03-28T22:32:42.652 DEBUG:teuthology.orchestra.run.smithi172:mds.c> sudo journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.c.service 2024-03-28T22:32:42.654 DEBUG:teuthology.orchestra.run.smithi172:mds.f> sudo journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.f.service 2024-03-28T22:32:42.697 DEBUG:teuthology.orchestra.run.smithi172:mds.i> sudo journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.i.service 2024-03-28T22:32:42.740 DEBUG:teuthology.orchestra.run.smithi172:mds.l> sudo journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.l.service 2024-03-28T22:32:42.783 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-28T22:32:42.783 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:32:43.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.015+0000 7f2a2be00640 1 -- 172.21.15.43:0/1144227782 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a2c108da0 msgr2=0x7f2a2c109220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:43.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.015+0000 7f2a2be00640 1 --2- 172.21.15.43:0/1144227782 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a2c108da0 0x7f2a2c109220 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2a14009900 tx=0x7f2a1402bc30 comp rx=0 tx=0).stop 2024-03-28T22:32:43.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.015+0000 7f2a2be00640 1 -- 172.21.15.43:0/1144227782 shutdown_connections 2024-03-28T22:32:43.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.015+0000 7f2a2be00640 1 --2- 172.21.15.43:0/1144227782 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a2c109760 0x7f2a2c10fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:43.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.015+0000 7f2a2be00640 1 --2- 172.21.15.43:0/1144227782 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a2c108da0 0x7f2a2c109220 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:43.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.015+0000 7f2a2be00640 1 --2- 172.21.15.43:0/1144227782 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a2c107bb0 0x7f2a2c107fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:43.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.015+0000 7f2a2be00640 1 -- 172.21.15.43:0/1144227782 >> 172.21.15.43:0/1144227782 conn(0x7f2a2c076b60 msgr2=0x7f2a2c078f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:32:43.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.015+0000 7f2a2be00640 1 -- 172.21.15.43:0/1144227782 shutdown_connections 2024-03-28T22:32:43.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.015+0000 7f2a2be00640 1 -- 172.21.15.43:0/1144227782 wait complete. 2024-03-28T22:32:43.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.016+0000 7f2a2be00640 1 Processor -- start 2024-03-28T22:32:43.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.016+0000 7f2a2be00640 1 -- start start 2024-03-28T22:32:43.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.017+0000 7f2a2be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a2c107bb0 0x7f2a2c19def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:43.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.017+0000 7f2a2be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a2c108da0 0x7f2a2c19e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:43.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.017+0000 7f2a2be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a2c109760 0x7f2a2c1a54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:43.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.017+0000 7f2a2be00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2a2c113880 con 0x7f2a2c108da0 2024-03-28T22:32:43.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.017+0000 7f2a2be00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f2a2c113700 con 0x7f2a2c107bb0 2024-03-28T22:32:43.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.017+0000 7f2a2be00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f2a2c113a00 con 0x7f2a2c109760 2024-03-28T22:32:43.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.017+0000 7f2a2b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a2c109760 0x7f2a2c1a54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:43.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.017+0000 7f2a23e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a2c108da0 0x7f2a2c19e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:43.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.017+0000 7f2a2b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a2c109760 0x7f2a2c1a54a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:55066/0 (socket says 172.21.15.43:55066) 2024-03-28T22:32:43.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.017+0000 7f2a2b400640 1 -- 172.21.15.43:0/3373610262 learned_addr learned my addr 172.21.15.43:0/3373610262 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:32:43.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.017+0000 7f2a2aa00640 1 --2- 172.21.15.43:0/3373610262 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a2c107bb0 0x7f2a2c19def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:43.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.018+0000 7f2a2b400640 1 -- 172.21.15.43:0/3373610262 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a2c107bb0 msgr2=0x7f2a2c19def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:43.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.018+0000 7f2a2b400640 1 --2- 172.21.15.43:0/3373610262 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a2c107bb0 0x7f2a2c19def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:43.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.018+0000 7f2a2b400640 1 -- 172.21.15.43:0/3373610262 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a2c108da0 msgr2=0x7f2a2c19e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:43.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.018+0000 7f2a2b400640 1 --2- 172.21.15.43:0/3373610262 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a2c108da0 0x7f2a2c19e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:43.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.018+0000 7f2a2b400640 1 -- 172.21.15.43:0/3373610262 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a2c1a5b10 con 0x7f2a2c109760 2024-03-28T22:32:43.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.018+0000 7f2a2b400640 1 --2- 172.21.15.43:0/3373610262 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a2c109760 0x7f2a2c1a54a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2a2400efd0 tx=0x7f2a2400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:32:43.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.018+0000 7f2a23400640 1 -- 172.21.15.43:0/3373610262 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a24003b70 con 0x7f2a2c109760 2024-03-28T22:32:43.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.018+0000 7f2a23400640 1 -- 172.21.15.43:0/3373610262 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a24003d10 con 0x7f2a2c109760 2024-03-28T22:32:43.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.018+0000 7f2a2be00640 1 -- 172.21.15.43:0/3373610262 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a2c1a5e00 con 0x7f2a2c109760 2024-03-28T22:32:43.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.018+0000 7f2a23400640 1 -- 172.21.15.43:0/3373610262 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a240109f0 con 0x7f2a2c109760 2024-03-28T22:32:43.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.018+0000 7f2a2be00640 1 -- 172.21.15.43:0/3373610262 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a2c1a62c0 con 0x7f2a2c109760 2024-03-28T22:32:43.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.020+0000 7f2a2be00640 1 -- 172.21.15.43:0/3373610262 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a2c10eed0 con 0x7f2a2c109760 2024-03-28T22:32:43.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.022+0000 7f2a23400640 1 -- 172.21.15.43:0/3373610262 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f2a24004020 con 0x7f2a2c109760 2024-03-28T22:32:43.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.023+0000 7f2a23400640 1 --2- 172.21.15.43:0/3373610262 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f2a0c0a1e00 0x7f2a0c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:43.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.023+0000 7f2a23400640 1 -- 172.21.15.43:0/3373610262 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f2a240cdd20 con 0x7f2a2c109760 2024-03-28T22:32:43.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.024+0000 7f2a2aa00640 1 --2- 172.21.15.43:0/3373610262 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f2a0c0a1e00 0x7f2a0c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:43.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.024+0000 7f2a2aa00640 1 --2- 172.21.15.43:0/3373610262 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f2a0c0a1e00 0x7f2a0c0a42c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c108c00 tx=0x7f2a1c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:32:43.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.027+0000 7f2a23400640 1 -- 172.21.15.43:0/3373610262 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2a2409b550 con 0x7f2a2c109760 2024-03-28T22:32:43.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:42 smithi172 ceph-mon[36566]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi043=a;smithi043=d;smithi043=g;smithi043=j;smithi082=b;smithi082=e;smithi082=h;smithi082=k;smithi172=c;smithi172=f;smithi172=i;smithi172=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:32:43.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:42 smithi172 ceph-mon[36566]: Saving service mds.all spec with placement smithi043=a;smithi043=d;smithi043=g;smithi043=j;smithi082=b;smithi082=e;smithi082=h;smithi082=k;smithi172=c;smithi172=f;smithi172=i;smithi172=l;count:12 2024-03-28T22:32:43.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:42 smithi082 ceph-mon[36471]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi043=a;smithi043=d;smithi043=g;smithi043=j;smithi082=b;smithi082=e;smithi082=h;smithi082=k;smithi172=c;smithi172=f;smithi172=i;smithi172=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:32:43.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:42 smithi082 ceph-mon[36471]: Saving service mds.all spec with placement smithi043=a;smithi043=d;smithi043=g;smithi043=j;smithi082=b;smithi082=e;smithi082=h;smithi082=k;smithi172=c;smithi172=f;smithi172=i;smithi172=l;count:12 2024-03-28T22:32:43.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.229+0000 7f2a2be00640 1 -- 172.21.15.43:0/3373610262 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2a2c062350 con 0x7f2a2c109760 2024-03-28T22:32:43.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.231+0000 7f2a23400640 1 -- 172.21.15.43:0/3373610262 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+18421 (secure 0 0 0) 0x7f2a2409ae50 con 0x7f2a2c109760 2024-03-28T22:32:43.231 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:32:43.231 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":83,"fsid":"c2276428-ed51-11ee-95d0-87774f69a715","created":"2024-03-28T22:24:19.950084+0000","modified":"2024-03-28T22:32:32.384331+0000","last_up_change":"2024-03-28T22:32:31.078737+0000","last_in_change":"2024-03-28T22:32:10.147097+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:28:14.713290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b855adcd-786c-4e42-b8fe-a515a77ce979","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6803","nonce":2826124409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6805","nonce":2826124409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6809","nonce":2826124409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6807","nonce":2826124409}]},"public_addr":"172.21.15.43:6803/2826124409","cluster_addr":"172.21.15.43:6805/2826124409","heartbeat_back_addr":"172.21.15.43:6809/2826124409","heartbeat_front_addr":"172.21.15.43:6807/2826124409","state":["exists","up"]},{"osd":1,"uuid":"031c143d-8ad1-4b68-8db9-3c717137e3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6801","nonce":3043263644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6803","nonce":3043263644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6807","nonce":3043263644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6805","nonce":3043263644}]},"public_addr":"172.21.15.82:6801/3043263644","cluster_addr":"172.21.15.82:6803/3043263644","heartbeat_back_addr":"172.21.15.82:6807/3043263644","heartbeat_front_addr":"172.21.15.82:6805/3043263644","state":["exists","up"]},{"osd":2,"uuid":"20632bbb-f07a-4ca1-b0c4-ff9d508e211c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6801","nonce":4213953645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6803","nonce":4213953645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6807","nonce":4213953645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6805","nonce":4213953645}]},"public_addr":"172.21.15.172:6801/4213953645","cluster_addr":"172.21.15.172:6803/4213953645","heartbeat_back_addr":"172.21.15.172:6807/4213953645","heartbeat_front_addr":"172.21.15.172:6805/4213953645","state":["exists","up"]},{"osd":3,"uuid":"9a1f2312-4dbd-475b-8736-d5ebc8788e09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6811","nonce":1597280920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6813","nonce":1597280920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6817","nonce":1597280920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6815","nonce":1597280920}]},"public_addr":"172.21.15.43:6811/1597280920","cluster_addr":"172.21.15.43:6813/1597280920","heartbeat_back_addr":"172.21.15.43:6817/1597280920","heartbeat_front_addr":"172.21.15.43:6815/1597280920","state":["exists","up"]},{"osd":4,"uuid":"29140ead-7075-42b2-90c1-27cae0b29ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6809","nonce":215451895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6811","nonce":215451895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6815","nonce":215451895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6813","nonce":215451895}]},"public_addr":"172.21.15.82:6809/215451895","cluster_addr":"172.21.15.82:6811/215451895","heartbeat_back_addr":"172.21.15.82:6815/215451895","heartbeat_front_addr":"172.21.15.82:6813/215451895","state":["exists","up"]},{"osd":5,"uuid":"59644c61-93dc-4536-b2ad-0256b24e5cb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6809","nonce":1576148933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6811","nonce":1576148933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6815","nonce":1576148933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6813","nonce":1576148933}]},"public_addr":"172.21.15.172:6809/1576148933","cluster_addr":"172.21.15.172:6811/1576148933","heartbeat_back_addr":"172.21.15.172:6815/1576148933","heartbeat_front_addr":"172.21.15.172:6813/1576148933","state":["exists","up"]},{"osd":6,"uuid":"22d54e57-c3f1-4ecf-9921-8e815b48f92a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6819","nonce":791044279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6821","nonce":791044279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6825","nonce":791044279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6823","nonce":791044279}]},"public_addr":"172.21.15.43:6819/791044279","cluster_addr":"172.21.15.43:6821/791044279","heartbeat_back_addr":"172.21.15.43:6825/791044279","heartbeat_front_addr":"172.21.15.43:6823/791044279","state":["exists","up"]},{"osd":7,"uuid":"248cd1e2-0d6b-452f-a0a8-c51fe1b975ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6817","nonce":2157777323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6819","nonce":2157777323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6823","nonce":2157777323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6821","nonce":2157777323}]},"public_addr":"172.21.15.82:6817/2157777323","cluster_addr":"172.21.15.82:6819/2157777323","heartbeat_back_addr":"172.21.15.82:6823/2157777323","heartbeat_front_addr":"172.21.15.82:6821/2157777323","state":["exists","up"]},{"osd":8,"uuid":"d98aab6e-8bae-4c24-906c-4f492f047b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6817","nonce":1095328248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6819","nonce":1095328248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6823","nonce":1095328248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6821","nonce":1095328248}]},"public_addr":"172.21.15.172:6817/1095328248","cluster_addr":"172.21.15.172:6819/1095328248","heartbeat_back_addr":"172.21.15.172:6823/1095328248","heartbeat_front_addr":"172.21.15.172:6821/1095328248","state":["exists","up"]},{"osd":9,"uuid":"19aa383e-a0f6-45ea-96cc-e7956535caa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6827","nonce":3657791498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6829","nonce":3657791498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6833","nonce":3657791498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6831","nonce":3657791498}]},"public_addr":"172.21.15.43:6827/3657791498","cluster_addr":"172.21.15.43:6829/3657791498","heartbeat_back_addr":"172.21.15.43:6833/3657791498","heartbeat_front_addr":"172.21.15.43:6831/3657791498","state":["exists","up"]},{"osd":10,"uuid":"666977e5-eaf7-41b9-a18f-c380e149345d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6825","nonce":3454954047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6827","nonce":3454954047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6831","nonce":3454954047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6829","nonce":3454954047}]},"public_addr":"172.21.15.82:6825/3454954047","cluster_addr":"172.21.15.82:6827/3454954047","heartbeat_back_addr":"172.21.15.82:6831/3454954047","heartbeat_front_addr":"172.21.15.82:6829/3454954047","state":["exists","up"]},{"osd":11,"uuid":"615cf3be-8076-470e-ad09-c880e7593717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6825","nonce":2563517144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6827","nonce":2563517144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6831","nonce":2563517144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6829","nonce":2563517144}]},"public_addr":"172.21.15.172:6825/2563517144","cluster_addr":"172.21.15.172:6827/2563517144","heartbeat_back_addr":"172.21.15.172:6831/2563517144","heartbeat_front_addr":"172.21.15.172:6829/2563517144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:21.727405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:47.100259+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:10.615438+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:43.632144+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:09.487940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:34.604463+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:09.594452+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:35.709795+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:01.184202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:35.455322+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:02.413166+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:28.550754+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/578791143":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/4029313215":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/1574178482":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6801/3517712438":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6800/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/1375099638":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/2710117652":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6801/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/3215634168":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6800/3517712438":"2024-03-29T22:25:28.008215+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:32:43.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.233+0000 7f2a2be00640 1 -- 172.21.15.43:0/3373610262 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f2a0c0a1e00 msgr2=0x7f2a0c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:43.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.233+0000 7f2a2be00640 1 --2- 172.21.15.43:0/3373610262 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f2a0c0a1e00 0x7f2a0c0a42c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c108c00 tx=0x7f2a1c005f90 comp rx=0 tx=0).stop 2024-03-28T22:32:43.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.234+0000 7f2a2be00640 1 -- 172.21.15.43:0/3373610262 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a2c109760 msgr2=0x7f2a2c1a54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:43.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.234+0000 7f2a2be00640 1 --2- 172.21.15.43:0/3373610262 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a2c109760 0x7f2a2c1a54a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2a2400efd0 tx=0x7f2a2400c6a0 comp rx=0 tx=0).stop 2024-03-28T22:32:43.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.234+0000 7f2a2be00640 1 -- 172.21.15.43:0/3373610262 shutdown_connections 2024-03-28T22:32:43.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.234+0000 7f2a2be00640 1 --2- 172.21.15.43:0/3373610262 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a2c109760 0x7f2a2c1a54a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:43.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.234+0000 7f2a2be00640 1 --2- 172.21.15.43:0/3373610262 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a2c108da0 0x7f2a2c19e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:43.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.234+0000 7f2a2be00640 1 --2- 172.21.15.43:0/3373610262 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f2a0c0a1e00 0x7f2a0c0a42c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:43.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.234+0000 7f2a2be00640 1 --2- 172.21.15.43:0/3373610262 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a2c107bb0 0x7f2a2c19def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:43.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.234+0000 7f2a2be00640 1 -- 172.21.15.43:0/3373610262 >> 172.21.15.43:0/3373610262 conn(0x7f2a2c076b60 msgr2=0x7f2a2c10c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:32:43.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.234+0000 7f2a2be00640 1 -- 172.21.15.43:0/3373610262 shutdown_connections 2024-03-28T22:32:43.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.234+0000 7f2a2be00640 1 -- 172.21.15.43:0/3373610262 wait complete. 2024-03-28T22:32:43.250 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:42 smithi043 ceph-mon[31823]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi043=a;smithi043=d;smithi043=g;smithi043=j;smithi082=b;smithi082=e;smithi082=h;smithi082=k;smithi172=c;smithi172=f;smithi172=i;smithi172=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:32:43.250 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:42 smithi043 ceph-mon[31823]: Saving service mds.all spec with placement smithi043=a;smithi043=d;smithi043=g;smithi043=j;smithi082=b;smithi082=e;smithi082=h;smithi082=k;smithi172=c;smithi172=f;smithi172=i;smithi172=l;count:12 2024-03-28T22:32:43.254 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T22:28:14.713290+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T22:32:43.254 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T22:32:43.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.482+0000 7f8cf7e00640 1 -- 172.21.15.43:0/3873624197 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8cf81031e0 msgr2=0x7f8cf8109a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:43.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.482+0000 7f8cf7e00640 1 --2- 172.21.15.43:0/3873624197 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8cf81031e0 0x7f8cf8109a70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8009900 tx=0x7f8ce802bc30 comp rx=0 tx=0).stop 2024-03-28T22:32:43.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.482+0000 7f8cf7e00640 1 -- 172.21.15.43:0/3873624197 shutdown_connections 2024-03-28T22:32:43.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.482+0000 7f8cf7e00640 1 --2- 172.21.15.43:0/3873624197 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8cf81031e0 0x7f8cf8109a70 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:43.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.482+0000 7f8cf7e00640 1 --2- 172.21.15.43:0/3873624197 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cf8102820 0x7f8cf8102ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:43.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.482+0000 7f8cf7e00640 1 --2- 172.21.15.43:0/3873624197 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8cf8101630 0x7f8cf8101a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:43.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.482+0000 7f8cf7e00640 1 -- 172.21.15.43:0/3873624197 >> 172.21.15.43:0/3873624197 conn(0x7f8cf80fcbc0 msgr2=0x7f8cf80fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:32:43.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.482+0000 7f8cf7e00640 1 -- 172.21.15.43:0/3873624197 shutdown_connections 2024-03-28T22:32:43.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.482+0000 7f8cf7e00640 1 -- 172.21.15.43:0/3873624197 wait complete. 2024-03-28T22:32:43.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.483+0000 7f8cf7e00640 1 Processor -- start 2024-03-28T22:32:43.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.483+0000 7f8cf7e00640 1 -- start start 2024-03-28T22:32:43.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.483+0000 7f8cf7e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8cf8101630 0x7f8cf8199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:43.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.483+0000 7f8cf7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8cf8102820 0x7f8cf819a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:43.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.484+0000 7f8cf7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cf81031e0 0x7f8cf81a1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:43.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.484+0000 7f8cf7e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf810d2e0 con 0x7f8cf8102820 2024-03-28T22:32:43.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.484+0000 7f8cf7e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf810d160 con 0x7f8cf8101630 2024-03-28T22:32:43.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.484+0000 7f8cf7e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf810d460 con 0x7f8cf81031e0 2024-03-28T22:32:43.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.484+0000 7f8cf6000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8cf8102820 0x7f8cf819a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:43.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.484+0000 7f8cf6a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8cf8101630 0x7f8cf8199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:43.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.484+0000 7f8cf6000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8cf8102820 0x7f8cf819a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39546/0 (socket says 172.21.15.43:39546) 2024-03-28T22:32:43.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.484+0000 7f8cf6000640 1 -- 172.21.15.43:0/1751091475 learned_addr learned my addr 172.21.15.43:0/1751091475 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:32:43.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.484+0000 7f8cf6a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8cf8101630 0x7f8cf8199ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:43072/0 (socket says 172.21.15.43:43072) 2024-03-28T22:32:43.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.484+0000 7f8cf7400640 1 --2- 172.21.15.43:0/1751091475 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cf81031e0 0x7f8cf81a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:43.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.484+0000 7f8cf6000640 1 -- 172.21.15.43:0/1751091475 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cf81031e0 msgr2=0x7f8cf81a1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:43.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.484+0000 7f8cf6000640 1 --2- 172.21.15.43:0/1751091475 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cf81031e0 0x7f8cf81a1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:43.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.484+0000 7f8cf6000640 1 -- 172.21.15.43:0/1751091475 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8cf8101630 msgr2=0x7f8cf8199ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:43.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.484+0000 7f8cf6000640 1 --2- 172.21.15.43:0/1751091475 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8cf8101630 0x7f8cf8199ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:43.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.484+0000 7f8cf6000640 1 -- 172.21.15.43:0/1751091475 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cf81a16e0 con 0x7f8cf8102820 2024-03-28T22:32:43.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.484+0000 7f8cf7400640 1 --2- 172.21.15.43:0/1751091475 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cf81031e0 0x7f8cf81a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:32:43.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.485+0000 7f8cf6000640 1 --2- 172.21.15.43:0/1751091475 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8cf8102820 0x7f8cf819a000 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8ce000e9f0 tx=0x7f8ce000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:32:43.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.485+0000 7f8cef400640 1 -- 172.21.15.43:0/1751091475 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce000cd80 con 0x7f8cf8102820 2024-03-28T22:32:43.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.485+0000 7f8cf7e00640 1 -- 172.21.15.43:0/1751091475 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cf81a19d0 con 0x7f8cf8102820 2024-03-28T22:32:43.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.485+0000 7f8cef400640 1 -- 172.21.15.43:0/1751091475 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ce0010910 con 0x7f8cf8102820 2024-03-28T22:32:43.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.485+0000 7f8cef400640 1 -- 172.21.15.43:0/1751091475 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce0018990 con 0x7f8cf8102820 2024-03-28T22:32:43.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.485+0000 7f8cf7e00640 1 -- 172.21.15.43:0/1751091475 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cf81a1f20 con 0x7f8cf8102820 2024-03-28T22:32:43.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.486+0000 7f8cf7e00640 1 -- 172.21.15.43:0/1751091475 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cf81a21b0 con 0x7f8cf8102820 2024-03-28T22:32:43.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.488+0000 7f8cef400640 1 -- 172.21.15.43:0/1751091475 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f8ce0010ac0 con 0x7f8cf8102820 2024-03-28T22:32:43.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.490+0000 7f8cef400640 1 --2- 172.21.15.43:0/1751091475 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f8cdc0a1d80 0x7f8cdc0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:43.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.490+0000 7f8cef400640 1 -- 172.21.15.43:0/1751091475 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f8ce00cdcc0 con 0x7f8cf8102820 2024-03-28T22:32:43.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.490+0000 7f8cf6a00640 1 --2- 172.21.15.43:0/1751091475 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f8cdc0a1d80 0x7f8cdc0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:43.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.491+0000 7f8cf6a00640 1 --2- 172.21.15.43:0/1751091475 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f8cdc0a1d80 0x7f8cdc0a4240 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8cf8102680 tx=0x7f8cf0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:32:43.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.494+0000 7f8cef400640 1 -- 172.21.15.43:0/1751091475 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8ce00d4080 con 0x7f8cf8102820 2024-03-28T22:32:43.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.699+0000 7f8cf7e00640 1 -- 172.21.15.43:0/1751091475 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8cf81a27c0 con 0x7f8cf8102820 2024-03-28T22:32:43.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.700+0000 7f8cef400640 1 -- 172.21.15.43:0/1751091475 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7f8ce009aea0 con 0x7f8cf8102820 2024-03-28T22:32:43.700 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-03-28T22:32:43.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.703+0000 7f8cf7e00640 1 -- 172.21.15.43:0/1751091475 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f8cdc0a1d80 msgr2=0x7f8cdc0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:43.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.703+0000 7f8cf7e00640 1 --2- 172.21.15.43:0/1751091475 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f8cdc0a1d80 0x7f8cdc0a4240 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8cf8102680 tx=0x7f8cf0009500 comp rx=0 tx=0).stop 2024-03-28T22:32:43.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.703+0000 7f8cf7e00640 1 -- 172.21.15.43:0/1751091475 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8cf8102820 msgr2=0x7f8cf819a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:43.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.703+0000 7f8cf7e00640 1 --2- 172.21.15.43:0/1751091475 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8cf8102820 0x7f8cf819a000 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8ce000e9f0 tx=0x7f8ce000eeb0 comp rx=0 tx=0).stop 2024-03-28T22:32:43.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.703+0000 7f8cf7e00640 1 -- 172.21.15.43:0/1751091475 shutdown_connections 2024-03-28T22:32:43.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.703+0000 7f8cf7e00640 1 --2- 172.21.15.43:0/1751091475 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8cf81031e0 0x7f8cf81a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:43.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.703+0000 7f8cf7e00640 1 --2- 172.21.15.43:0/1751091475 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8cf8102820 0x7f8cf819a000 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:43.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.703+0000 7f8cf7e00640 1 --2- 172.21.15.43:0/1751091475 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f8cdc0a1d80 0x7f8cdc0a4240 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:43.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.703+0000 7f8cf7e00640 1 --2- 172.21.15.43:0/1751091475 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8cf8101630 0x7f8cf8199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:43.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.703+0000 7f8cf7e00640 1 -- 172.21.15.43:0/1751091475 >> 172.21.15.43:0/1751091475 conn(0x7f8cf80fcbc0 msgr2=0x7f8cf8073980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:32:43.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.703+0000 7f8cf7e00640 1 -- 172.21.15.43:0/1751091475 shutdown_connections 2024-03-28T22:32:43.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.703+0000 7f8cf7e00640 1 -- 172.21.15.43:0/1751091475 wait complete. 2024-03-28T22:32:43.721 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:32:43.885 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:43 smithi172 ceph-mon[36566]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:43.885 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:43 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:43.885 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:43 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:32:43.885 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:43 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:43.885 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:43 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:32:43.886 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:43 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:32:43.886 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:43 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:43.886 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:43 smithi172 ceph-mon[36566]: Deploying daemon mds.l on smithi172 2024-03-28T22:32:43.886 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:43 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3373610262' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:32:43.886 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:43 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1751091475' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:32:43.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.944+0000 7f3a6c000640 1 -- 172.21.15.43:0/2781279656 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a64105540 msgr2=0x7f3a6410bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:43.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.944+0000 7f3a6c000640 1 --2- 172.21.15.43:0/2781279656 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a64105540 0x7f3a6410bdd0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c009be0 tx=0x7f3a5c02bef0 comp rx=0 tx=0).stop 2024-03-28T22:32:43.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.944+0000 7f3a6c000640 1 -- 172.21.15.43:0/2781279656 shutdown_connections 2024-03-28T22:32:43.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.944+0000 7f3a6c000640 1 --2- 172.21.15.43:0/2781279656 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a64105540 0x7f3a6410bdd0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:43.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.944+0000 7f3a6c000640 1 --2- 172.21.15.43:0/2781279656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a64104b80 0x7f3a64105000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:43.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.944+0000 7f3a6c000640 1 --2- 172.21.15.43:0/2781279656 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a64103990 0x7f3a64103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:43.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.944+0000 7f3a6c000640 1 -- 172.21.15.43:0/2781279656 >> 172.21.15.43:0/2781279656 conn(0x7f3a640fef00 msgr2=0x7f3a64101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:32:43.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.945+0000 7f3a6c000640 1 -- 172.21.15.43:0/2781279656 shutdown_connections 2024-03-28T22:32:43.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.945+0000 7f3a6c000640 1 -- 172.21.15.43:0/2781279656 wait complete. 2024-03-28T22:32:43.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.946+0000 7f3a6c000640 1 Processor -- start 2024-03-28T22:32:43.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.946+0000 7f3a6c000640 1 -- start start 2024-03-28T22:32:43.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.946+0000 7f3a6c000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a64103990 0x7f3a64199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:43.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.946+0000 7f3a6c000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a64104b80 0x7f3a6419a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:43.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.946+0000 7f3a6c000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a64105540 0x7f3a641a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:43.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.947+0000 7f3a6c000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3a6410f640 con 0x7f3a64103990 2024-03-28T22:32:43.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.947+0000 7f3a6c000640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f3a6410f4c0 con 0x7f3a64105540 2024-03-28T22:32:43.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.947+0000 7f3a6c000640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f3a6410f7c0 con 0x7f3a64104b80 2024-03-28T22:32:43.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.947+0000 7f3a69600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a64103990 0x7f3a64199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:43.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.947+0000 7f3a69600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a64103990 0x7f3a64199c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39572/0 (socket says 172.21.15.43:39572) 2024-03-28T22:32:43.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.947+0000 7f3a69600640 1 -- 172.21.15.43:0/2785573904 learned_addr learned my addr 172.21.15.43:0/2785573904 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:32:43.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.947+0000 7f3a68c00640 1 --2- 172.21.15.43:0/2785573904 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a64104b80 0x7f3a6419a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:43.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.947+0000 7f3a6a000640 1 --2- 172.21.15.43:0/2785573904 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a64105540 0x7f3a641a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:43.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.947+0000 7f3a68c00640 1 -- 172.21.15.43:0/2785573904 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a64105540 msgr2=0x7f3a641a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:43.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.947+0000 7f3a68c00640 1 --2- 172.21.15.43:0/2785573904 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a64105540 0x7f3a641a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:43.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.947+0000 7f3a68c00640 1 -- 172.21.15.43:0/2785573904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a64103990 msgr2=0x7f3a64199c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:43.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.947+0000 7f3a68c00640 1 --2- 172.21.15.43:0/2785573904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a64103990 0x7f3a64199c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:43.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.947+0000 7f3a68c00640 1 -- 172.21.15.43:0/2785573904 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a641a1850 con 0x7f3a64104b80 2024-03-28T22:32:43.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.947+0000 7f3a69600640 1 --2- 172.21.15.43:0/2785573904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a64103990 0x7f3a64199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:32:43.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.948+0000 7f3a68c00640 1 --2- 172.21.15.43:0/2785573904 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a64104b80 0x7f3a6419a180 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3a5000e790 tx=0x7f3a5000ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:32:43.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.948+0000 7f3a62000640 1 -- 172.21.15.43:0/2785573904 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a5000eea0 con 0x7f3a64104b80 2024-03-28T22:32:43.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.948+0000 7f3a62000640 1 -- 172.21.15.43:0/2785573904 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a5000ccc0 con 0x7f3a64104b80 2024-03-28T22:32:43.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.948+0000 7f3a6c000640 1 -- 172.21.15.43:0/2785573904 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a641a1b40 con 0x7f3a64104b80 2024-03-28T22:32:43.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.948+0000 7f3a62000640 1 -- 172.21.15.43:0/2785573904 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a50018760 con 0x7f3a64104b80 2024-03-28T22:32:43.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.948+0000 7f3a6c000640 1 -- 172.21.15.43:0/2785573904 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a641a2090 con 0x7f3a64104b80 2024-03-28T22:32:43.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.949+0000 7f3a6c000640 1 -- 172.21.15.43:0/2785573904 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a6410ac50 con 0x7f3a64104b80 2024-03-28T22:32:43.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.951+0000 7f3a62000640 1 -- 172.21.15.43:0/2785573904 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f3a50010430 con 0x7f3a64104b80 2024-03-28T22:32:43.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.952+0000 7f3a62000640 1 --2- 172.21.15.43:0/2785573904 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3a4c0a1d80 0x7f3a4c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:43.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.952+0000 7f3a62000640 1 -- 172.21.15.43:0/2785573904 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f3a500cdbe0 con 0x7f3a64104b80 2024-03-28T22:32:43.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.953+0000 7f3a69600640 1 --2- 172.21.15.43:0/2785573904 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3a4c0a1d80 0x7f3a4c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:43.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.953+0000 7f3a69600640 1 --2- 172.21.15.43:0/2785573904 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3a4c0a1d80 0x7f3a4c0a4240 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3a6419b110 tx=0x7f3a58005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:32:43.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:43.956+0000 7f3a62000640 1 -- 172.21.15.43:0/2785573904 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3a5009adc0 con 0x7f3a64104b80 2024-03-28T22:32:44.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:43 smithi043 ceph-mon[31823]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:44.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:43 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:44.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:43 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:32:44.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:43 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:44.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:43 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:32:44.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:43 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:32:44.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:43 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:44.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:43 smithi043 ceph-mon[31823]: Deploying daemon mds.l on smithi172 2024-03-28T22:32:44.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:43 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3373610262' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:32:44.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:43 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1751091475' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:32:44.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:43 smithi082 ceph-mon[36471]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:44.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:43 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:44.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:43 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:32:44.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:43 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:44.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:43 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:32:44.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:43 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:32:44.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:43 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:44.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:43 smithi082 ceph-mon[36471]: Deploying daemon mds.l on smithi172 2024-03-28T22:32:44.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:43 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3373610262' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:32:44.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:43 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1751091475' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:32:44.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:44.200+0000 7f3a6c000640 1 -- 172.21.15.43:0/2785573904 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a64062350 con 0x7f3a64104b80 2024-03-28T22:32:44.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:44.201+0000 7f3a62000640 1 -- 172.21.15.43:0/2785573904 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f3a5009ee60 con 0x7f3a64104b80 2024-03-28T22:32:44.201 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:32:44.201 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-28T22:32:44.201 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 1 2024-03-28T22:32:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:44.204+0000 7f3a6c000640 1 -- 172.21.15.43:0/2785573904 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3a4c0a1d80 msgr2=0x7f3a4c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:44.204+0000 7f3a6c000640 1 --2- 172.21.15.43:0/2785573904 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3a4c0a1d80 0x7f3a4c0a4240 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3a6419b110 tx=0x7f3a58005f90 comp rx=0 tx=0).stop 2024-03-28T22:32:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:44.204+0000 7f3a6c000640 1 -- 172.21.15.43:0/2785573904 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a64104b80 msgr2=0x7f3a6419a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:44.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:44.204+0000 7f3a6c000640 1 --2- 172.21.15.43:0/2785573904 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a64104b80 0x7f3a6419a180 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3a5000e790 tx=0x7f3a5000ec50 comp rx=0 tx=0).stop 2024-03-28T22:32:44.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:44.204+0000 7f3a6c000640 1 -- 172.21.15.43:0/2785573904 shutdown_connections 2024-03-28T22:32:44.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:44.204+0000 7f3a6c000640 1 --2- 172.21.15.43:0/2785573904 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3a4c0a1d80 0x7f3a4c0a4240 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:44.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:44.204+0000 7f3a6c000640 1 --2- 172.21.15.43:0/2785573904 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a64105540 0x7f3a641a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:44.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:44.204+0000 7f3a6c000640 1 --2- 172.21.15.43:0/2785573904 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a64104b80 0x7f3a6419a180 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:44.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:44.204+0000 7f3a6c000640 1 --2- 172.21.15.43:0/2785573904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a64103990 0x7f3a64199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:44.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:44.204+0000 7f3a6c000640 1 -- 172.21.15.43:0/2785573904 >> 172.21.15.43:0/2785573904 conn(0x7f3a640fef00 msgr2=0x7f3a64107e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:32:44.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:44.204+0000 7f3a6c000640 1 -- 172.21.15.43:0/2785573904 shutdown_connections 2024-03-28T22:32:44.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:44.204+0000 7f3a6c000640 1 -- 172.21.15.43:0/2785573904 wait complete. 2024-03-28T22:32:45.144 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:44 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2785573904' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:32:45.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:44 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2785573904' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:32:45.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:44 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2785573904' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:32:45.397 INFO:journalctl@ceph.mds.l.smithi172.stdout:Mar 28 22:32:45 smithi172 podman[66345]: 2024-03-28 22:32:45.386253379 +0000 UTC m=+0.139034567 container create 83dd91753c3874b27289692f117b8e10063dc02cd0164be15a9e3414310dc742 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, RELEASE=squid-323d23d, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-28T22:32:45.834 INFO:journalctl@ceph.mds.l.smithi172.stdout:Mar 28 22:32:45 smithi172 podman[66345]: 2024-03-28 22:32:45.630763653 +0000 UTC m=+0.383544855 container init 83dd91753c3874b27289692f117b8e10063dc02cd0164be15a9e3414310dc742 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-l, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-323d23d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-28T22:32:45.834 INFO:journalctl@ceph.mds.l.smithi172.stdout:Mar 28 22:32:45 smithi172 podman[66345]: 2024-03-28 22:32:45.633422319 +0000 UTC m=+0.386203510 container start 83dd91753c3874b27289692f117b8e10063dc02cd0164be15a9e3414310dc742 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-l, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-323d23d, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T22:32:45.834 INFO:journalctl@ceph.mds.l.smithi172.stdout:Mar 28 22:32:45 smithi172 ceph-mds[66365]: -- 172.21.15.172:0/3687587674 <== mon.1 v2:172.21.15.172:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f50d753a40 con 0x55f50d726800 2024-03-28T22:32:45.835 INFO:journalctl@ceph.mds.l.smithi172.stdout:Mar 28 22:32:45 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-l[66361]: starting mds.l at 2024-03-28T22:32:45.835 INFO:journalctl@ceph.mds.l.smithi172.stdout:Mar 28 22:32:45 smithi172 bash[66345]: 83dd91753c3874b27289692f117b8e10063dc02cd0164be15a9e3414310dc742 2024-03-28T22:32:45.835 INFO:journalctl@ceph.mds.l.smithi172.stdout:Mar 28 22:32:45 smithi172 systemd[1]: Started Ceph mds.l for c2276428-ed51-11ee-95d0-87774f69a715. 2024-03-28T22:32:46.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:45 smithi172 ceph-mon[36566]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:46.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:45 smithi172 ceph-mon[36566]: mds.? [v2:172.21.15.172:6832/2579392697,v1:172.21.15.172:6833/2579392697] up:boot 2024-03-28T22:32:46.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:45 smithi172 ceph-mon[36566]: fsmap 1 up:standby 2024-03-28T22:32:46.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:45 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T22:32:46.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:45 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T22:32:46.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:45 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:46.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:45 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:46.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:45 smithi082 ceph-mon[36471]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:46.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:45 smithi082 ceph-mon[36471]: mds.? [v2:172.21.15.172:6832/2579392697,v1:172.21.15.172:6833/2579392697] up:boot 2024-03-28T22:32:46.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:45 smithi082 ceph-mon[36471]: fsmap 1 up:standby 2024-03-28T22:32:46.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:45 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T22:32:46.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:45 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T22:32:46.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:45 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:46.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:45 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:46.224 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:32:46.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:45 smithi043 ceph-mon[31823]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:46.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:45 smithi043 ceph-mon[31823]: mds.? [v2:172.21.15.172:6832/2579392697,v1:172.21.15.172:6833/2579392697] up:boot 2024-03-28T22:32:46.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:45 smithi043 ceph-mon[31823]: fsmap 1 up:standby 2024-03-28T22:32:46.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:45 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T22:32:46.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:45 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T22:32:46.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:45 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:46.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:45 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:46.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.456+0000 7f5114c00640 1 -- 172.21.15.43:0/3022438778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5110108da0 msgr2=0x7f5110109220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:46.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.456+0000 7f5114c00640 1 --2- 172.21.15.43:0/3022438778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5110108da0 0x7f5110109220 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f50f8009ca0 tx=0x7f50f802c0f0 comp rx=0 tx=0).stop 2024-03-28T22:32:46.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.456+0000 7f5114c00640 1 -- 172.21.15.43:0/3022438778 shutdown_connections 2024-03-28T22:32:46.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.456+0000 7f5114c00640 1 --2- 172.21.15.43:0/3022438778 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5110109760 0x7f511010fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:46.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.456+0000 7f5114c00640 1 --2- 172.21.15.43:0/3022438778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5110108da0 0x7f5110109220 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:46.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.456+0000 7f5114c00640 1 --2- 172.21.15.43:0/3022438778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5110107bb0 0x7f5110107fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:46.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.456+0000 7f5114c00640 1 -- 172.21.15.43:0/3022438778 >> 172.21.15.43:0/3022438778 conn(0x7f5110076b60 msgr2=0x7f5110078f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:32:46.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.456+0000 7f5114c00640 1 -- 172.21.15.43:0/3022438778 shutdown_connections 2024-03-28T22:32:46.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.456+0000 7f5114c00640 1 -- 172.21.15.43:0/3022438778 wait complete. 2024-03-28T22:32:46.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.457+0000 7f5114c00640 1 Processor -- start 2024-03-28T22:32:46.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.457+0000 7f5114c00640 1 -- start start 2024-03-28T22:32:46.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.458+0000 7f5114c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5110107bb0 0x7f511019dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:46.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.458+0000 7f5114c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5110108da0 0x7f511019e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:46.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.458+0000 7f5114c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5110109760 0x7f51101a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:46.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.458+0000 7f5114c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5110113830 con 0x7f5110109760 2024-03-28T22:32:46.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.458+0000 7f5114c00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f51101136b0 con 0x7f5110107bb0 2024-03-28T22:32:46.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.458+0000 7f5114c00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f51101139b0 con 0x7f5110108da0 2024-03-28T22:32:46.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.458+0000 7f510f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5110107bb0 0x7f511019dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:46.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.458+0000 7f510f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5110107bb0 0x7f511019dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45080/0 (socket says 172.21.15.43:45080) 2024-03-28T22:32:46.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.458+0000 7f510f400640 1 -- 172.21.15.43:0/856065449 learned_addr learned my addr 172.21.15.43:0/856065449 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:32:46.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.458+0000 7f510fe00640 1 --2- 172.21.15.43:0/856065449 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5110109760 0x7f51101a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:46.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.458+0000 7f5106a00640 1 --2- 172.21.15.43:0/856065449 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5110108da0 0x7f511019e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:46.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.458+0000 7f510f400640 1 -- 172.21.15.43:0/856065449 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5110108da0 msgr2=0x7f511019e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:46.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.458+0000 7f510f400640 1 --2- 172.21.15.43:0/856065449 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5110108da0 0x7f511019e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:46.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.458+0000 7f510f400640 1 -- 172.21.15.43:0/856065449 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5110109760 msgr2=0x7f51101a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:46.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.458+0000 7f510f400640 1 --2- 172.21.15.43:0/856065449 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5110109760 0x7f51101a5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:46.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.458+0000 7f510f400640 1 -- 172.21.15.43:0/856065449 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51101a5ac0 con 0x7f5110107bb0 2024-03-28T22:32:46.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.459+0000 7f510f400640 1 --2- 172.21.15.43:0/856065449 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5110107bb0 0x7f511019dea0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f510000e9c0 tx=0x7f510000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:32:46.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.459+0000 7f510cc00640 1 -- 172.21.15.43:0/856065449 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f510000cd80 con 0x7f5110107bb0 2024-03-28T22:32:46.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.459+0000 7f510cc00640 1 -- 172.21.15.43:0/856065449 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51000035f0 con 0x7f5110107bb0 2024-03-28T22:32:46.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.459+0000 7f510cc00640 1 -- 172.21.15.43:0/856065449 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5100010690 con 0x7f5110107bb0 2024-03-28T22:32:46.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.459+0000 7f5114c00640 1 -- 172.21.15.43:0/856065449 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51101a5db0 con 0x7f5110107bb0 2024-03-28T22:32:46.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.459+0000 7f5114c00640 1 -- 172.21.15.43:0/856065449 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51101a6210 con 0x7f5110107bb0 2024-03-28T22:32:46.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.460+0000 7f5114c00640 1 -- 172.21.15.43:0/856065449 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f511010eed0 con 0x7f5110107bb0 2024-03-28T22:32:46.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.462+0000 7f510cc00640 1 -- 172.21.15.43:0/856065449 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f51000037a0 con 0x7f5110107bb0 2024-03-28T22:32:46.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.463+0000 7f510cc00640 1 --2- 172.21.15.43:0/856065449 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f50f40a1e00 0x7f50f40a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:46.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.464+0000 7f510cc00640 1 -- 172.21.15.43:0/856065449 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f51000cdc90 con 0x7f5110107bb0 2024-03-28T22:32:46.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.464+0000 7f5106a00640 1 --2- 172.21.15.43:0/856065449 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f50f40a1e00 0x7f50f40a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:46.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.464+0000 7f5106a00640 1 --2- 172.21.15.43:0/856065449 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f50f40a1e00 0x7f50f40a42c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f50f802c700 tx=0x7f50f8009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:32:46.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.468+0000 7f510cc00640 1 -- 172.21.15.43:0/856065449 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f510009b560 con 0x7f5110107bb0 2024-03-28T22:32:46.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.711+0000 7f5114c00640 1 -- 172.21.15.43:0/856065449 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f511002d170 con 0x7f5110107bb0 2024-03-28T22:32:46.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.712+0000 7f510cc00640 1 -- 172.21.15.43:0/856065449 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f510009ae60 con 0x7f5110107bb0 2024-03-28T22:32:46.712 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:32:46.712 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-28T22:32:46.712 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 2 2024-03-28T22:32:46.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.715+0000 7f5114c00640 1 -- 172.21.15.43:0/856065449 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f50f40a1e00 msgr2=0x7f50f40a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:46.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.715+0000 7f5114c00640 1 --2- 172.21.15.43:0/856065449 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f50f40a1e00 0x7f50f40a42c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f50f802c700 tx=0x7f50f8009920 comp rx=0 tx=0).stop 2024-03-28T22:32:46.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.715+0000 7f5114c00640 1 -- 172.21.15.43:0/856065449 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5110107bb0 msgr2=0x7f511019dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:46.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.715+0000 7f5114c00640 1 --2- 172.21.15.43:0/856065449 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5110107bb0 0x7f511019dea0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f510000e9c0 tx=0x7f510000ee80 comp rx=0 tx=0).stop 2024-03-28T22:32:46.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.715+0000 7f5114c00640 1 -- 172.21.15.43:0/856065449 shutdown_connections 2024-03-28T22:32:46.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.715+0000 7f5114c00640 1 --2- 172.21.15.43:0/856065449 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5110109760 0x7f51101a5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:46.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.715+0000 7f5114c00640 1 --2- 172.21.15.43:0/856065449 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5110108da0 0x7f511019e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:46.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.715+0000 7f5114c00640 1 --2- 172.21.15.43:0/856065449 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f50f40a1e00 0x7f50f40a42c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:46.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.715+0000 7f5114c00640 1 --2- 172.21.15.43:0/856065449 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5110107bb0 0x7f511019dea0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:46.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.715+0000 7f5114c00640 1 -- 172.21.15.43:0/856065449 >> 172.21.15.43:0/856065449 conn(0x7f5110076b60 msgr2=0x7f511010c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:32:46.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.715+0000 7f5114c00640 1 -- 172.21.15.43:0/856065449 shutdown_connections 2024-03-28T22:32:46.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:46.715+0000 7f5114c00640 1 -- 172.21.15.43:0/856065449 wait complete. 2024-03-28T22:32:47.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:46 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:47.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:46 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:32:47.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:46 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:32:47.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:46 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:47.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:46 smithi043 ceph-mon[31823]: Deploying daemon mds.k on smithi082 2024-03-28T22:32:47.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:46 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/856065449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:32:47.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:46 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:47.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:46 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:32:47.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:46 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:32:47.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:46 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:47.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:46 smithi172 ceph-mon[36566]: Deploying daemon mds.k on smithi082 2024-03-28T22:32:47.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:46 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/856065449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:32:47.176 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:46 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:47.176 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:46 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:32:47.176 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:46 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:32:47.176 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:46 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:47.176 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:46 smithi082 ceph-mon[36471]: Deploying daemon mds.k on smithi082 2024-03-28T22:32:47.176 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:46 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/856065449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:32:48.129 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:47 smithi082 ceph-mon[36471]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:48.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:47 smithi172 ceph-mon[36566]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:48.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:47 smithi043 ceph-mon[31823]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:48.449 INFO:journalctl@ceph.mds.k.smithi082.stdout:Mar 28 22:32:48 smithi082 podman[66306]: 2024-03-28 22:32:48.356751629 +0000 UTC m=+0.125815115 container create 19590bac1035fa8e713ca00e77117c4639840f67a35cb7ede484211b8885d834 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-k, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-28T22:32:48.734 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:32:48.950 INFO:journalctl@ceph.mds.k.smithi082.stdout:Mar 28 22:32:48 smithi082 podman[66306]: 2024-03-28 22:32:48.584424121 +0000 UTC m=+0.353487564 container init 19590bac1035fa8e713ca00e77117c4639840f67a35cb7ede484211b8885d834 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-k, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=squid-323d23d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326) 2024-03-28T22:32:48.950 INFO:journalctl@ceph.mds.k.smithi082.stdout:Mar 28 22:32:48 smithi082 podman[66306]: 2024-03-28 22:32:48.587027608 +0000 UTC m=+0.356091054 container start 19590bac1035fa8e713ca00e77117c4639840f67a35cb7ede484211b8885d834 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-k, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-323d23d) 2024-03-28T22:32:48.950 INFO:journalctl@ceph.mds.k.smithi082.stdout:Mar 28 22:32:48 smithi082 ceph-mds[66325]: -- 172.21.15.82:0/2841996868 <== mon.2 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557112d77a40 con 0x557112d4ac00 2024-03-28T22:32:48.950 INFO:journalctl@ceph.mds.k.smithi082.stdout:Mar 28 22:32:48 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-k[66321]: starting mds.k at 2024-03-28T22:32:48.950 INFO:journalctl@ceph.mds.k.smithi082.stdout:Mar 28 22:32:48 smithi082 bash[66306]: 19590bac1035fa8e713ca00e77117c4639840f67a35cb7ede484211b8885d834 2024-03-28T22:32:48.951 INFO:journalctl@ceph.mds.k.smithi082.stdout:Mar 28 22:32:48 smithi082 systemd[1]: Started Ceph mds.k for c2276428-ed51-11ee-95d0-87774f69a715. 2024-03-28T22:32:48.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.961+0000 7f1144400640 1 -- 172.21.15.43:0/1512362989 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f113c1022d0 msgr2=0x7f113c1046c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:48.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.961+0000 7f1144400640 1 --2- 172.21.15.43:0/1512362989 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f113c1022d0 0x7f113c1046c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1130009900 tx=0x7f113002bc30 comp rx=0 tx=0).stop 2024-03-28T22:32:48.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.961+0000 7f1144400640 1 -- 172.21.15.43:0/1512362989 shutdown_connections 2024-03-28T22:32:48.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.961+0000 7f1144400640 1 --2- 172.21.15.43:0/1512362989 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f113c107530 0x7f113c109920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:48.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.961+0000 7f1144400640 1 --2- 172.21.15.43:0/1512362989 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f113c104c00 0x7f113c106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:48.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.961+0000 7f1144400640 1 --2- 172.21.15.43:0/1512362989 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f113c1022d0 0x7f113c1046c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:48.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.961+0000 7f1144400640 1 -- 172.21.15.43:0/1512362989 >> 172.21.15.43:0/1512362989 conn(0x7f113c0fbc00 msgr2=0x7f113c0fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:32:48.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.961+0000 7f1144400640 1 -- 172.21.15.43:0/1512362989 shutdown_connections 2024-03-28T22:32:48.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.961+0000 7f1144400640 1 -- 172.21.15.43:0/1512362989 wait complete. 2024-03-28T22:32:48.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.962+0000 7f1144400640 1 Processor -- start 2024-03-28T22:32:48.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.962+0000 7f1144400640 1 -- start start 2024-03-28T22:32:48.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.962+0000 7f1144400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f113c1022d0 0x7f113c199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:48.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.962+0000 7f1144400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f113c104c00 0x7f113c19a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:48.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.962+0000 7f1144400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f113c107530 0x7f113c1a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:48.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.962+0000 7f1144400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f113c10d460 con 0x7f113c107530 2024-03-28T22:32:48.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.962+0000 7f1144400640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f113c10d2e0 con 0x7f113c1022d0 2024-03-28T22:32:48.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.962+0000 7f1144400640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f113c10d5e0 con 0x7f113c104c00 2024-03-28T22:32:48.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.963+0000 7f1141a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f113c1022d0 0x7f113c199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:48.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.963+0000 7f1142400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f113c107530 0x7f113c1a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:48.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.963+0000 7f1141a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f113c1022d0 0x7f113c199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45090/0 (socket says 172.21.15.43:45090) 2024-03-28T22:32:48.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.963+0000 7f1141a00640 1 -- 172.21.15.43:0/2197166264 learned_addr learned my addr 172.21.15.43:0/2197166264 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:32:48.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.963+0000 7f1141000640 1 --2- 172.21.15.43:0/2197166264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f113c104c00 0x7f113c19a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:48.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.963+0000 7f1142400640 1 -- 172.21.15.43:0/2197166264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f113c104c00 msgr2=0x7f113c19a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:48.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.963+0000 7f1142400640 1 --2- 172.21.15.43:0/2197166264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f113c104c00 0x7f113c19a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:48.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.963+0000 7f1142400640 1 -- 172.21.15.43:0/2197166264 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f113c1022d0 msgr2=0x7f113c199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:48.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.963+0000 7f1142400640 1 --2- 172.21.15.43:0/2197166264 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f113c1022d0 0x7f113c199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:48.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.963+0000 7f1142400640 1 -- 172.21.15.43:0/2197166264 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f113c1a1940 con 0x7f113c107530 2024-03-28T22:32:48.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.964+0000 7f1142400640 1 --2- 172.21.15.43:0/2197166264 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f113c107530 0x7f113c1a12a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f113400ec50 tx=0x7f113400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:32:48.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.964+0000 7f113a000640 1 -- 172.21.15.43:0/2197166264 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1134003b70 con 0x7f113c107530 2024-03-28T22:32:48.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.964+0000 7f1144400640 1 -- 172.21.15.43:0/2197166264 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f113c1a1c30 con 0x7f113c107530 2024-03-28T22:32:48.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.964+0000 7f113a000640 1 -- 172.21.15.43:0/2197166264 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1134003d10 con 0x7f113c107530 2024-03-28T22:32:48.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.964+0000 7f113a000640 1 -- 172.21.15.43:0/2197166264 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1134010890 con 0x7f113c107530 2024-03-28T22:32:48.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.964+0000 7f1144400640 1 -- 172.21.15.43:0/2197166264 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f113c1a2180 con 0x7f113c107530 2024-03-28T22:32:48.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.965+0000 7f1144400640 1 -- 172.21.15.43:0/2197166264 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f113c109020 con 0x7f113c107530 2024-03-28T22:32:48.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.968+0000 7f113a000640 1 -- 172.21.15.43:0/2197166264 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f1134004b40 con 0x7f113c107530 2024-03-28T22:32:48.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.969+0000 7f113a000640 1 --2- 172.21.15.43:0/2197166264 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f11240a1d80 0x7f11240a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:48.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.969+0000 7f113a000640 1 -- 172.21.15.43:0/2197166264 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f11340ce0e0 con 0x7f113c107530 2024-03-28T22:32:48.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.969+0000 7f1141a00640 1 --2- 172.21.15.43:0/2197166264 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f11240a1d80 0x7f11240a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:48.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.970+0000 7f1141a00640 1 --2- 172.21.15.43:0/2197166264 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f11240a1d80 0x7f11240a4240 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1130006010 tx=0x7f113002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:32:48.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:48.972+0000 7f113a000640 1 -- 172.21.15.43:0/2197166264 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f113409b910 con 0x7f113c107530 2024-03-28T22:32:49.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:49.219+0000 7f1144400640 1 -- 172.21.15.43:0/2197166264 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f113c02d170 con 0x7f113c107530 2024-03-28T22:32:49.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:49.220+0000 7f113a000640 1 -- 172.21.15.43:0/2197166264 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f113409b210 con 0x7f113c107530 2024-03-28T22:32:49.220 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:32:49.220 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-28T22:32:49.220 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 3 2024-03-28T22:32:49.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:49.224+0000 7f1144400640 1 -- 172.21.15.43:0/2197166264 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f11240a1d80 msgr2=0x7f11240a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:49.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:49.224+0000 7f1144400640 1 --2- 172.21.15.43:0/2197166264 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f11240a1d80 0x7f11240a4240 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1130006010 tx=0x7f113002df90 comp rx=0 tx=0).stop 2024-03-28T22:32:49.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:49.224+0000 7f1144400640 1 -- 172.21.15.43:0/2197166264 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f113c107530 msgr2=0x7f113c1a12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:49.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:49.224+0000 7f1144400640 1 --2- 172.21.15.43:0/2197166264 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f113c107530 0x7f113c1a12a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f113400ec50 tx=0x7f113400c6a0 comp rx=0 tx=0).stop 2024-03-28T22:32:49.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:49.224+0000 7f1144400640 1 -- 172.21.15.43:0/2197166264 shutdown_connections 2024-03-28T22:32:49.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:49.224+0000 7f1144400640 1 --2- 172.21.15.43:0/2197166264 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f113c107530 0x7f113c1a12a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:49.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:49.224+0000 7f1144400640 1 --2- 172.21.15.43:0/2197166264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f113c104c00 0x7f113c19a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:49.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:49.224+0000 7f1144400640 1 --2- 172.21.15.43:0/2197166264 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f11240a1d80 0x7f11240a4240 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:49.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:49.224+0000 7f1144400640 1 --2- 172.21.15.43:0/2197166264 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f113c1022d0 0x7f113c199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:49.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:49.224+0000 7f1144400640 1 -- 172.21.15.43:0/2197166264 >> 172.21.15.43:0/2197166264 conn(0x7f113c0fbc00 msgr2=0x7f113c101a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:32:49.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:49.224+0000 7f1144400640 1 -- 172.21.15.43:0/2197166264 shutdown_connections 2024-03-28T22:32:49.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:49.224+0000 7f1144400640 1 -- 172.21.15.43:0/2197166264 wait complete. 2024-03-28T22:32:49.984 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:49 smithi043 ceph-mon[31823]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:49.984 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:49 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:49.984 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:49 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:49.984 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:49 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:49.984 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:49 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:32:49.984 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:49 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:32:49.984 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:49 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:49.985 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:49 smithi043 ceph-mon[31823]: Deploying daemon mds.j on smithi043 2024-03-28T22:32:49.985 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:49 smithi043 ceph-mon[31823]: mds.? [v2:172.21.15.82:6832/3363736995,v1:172.21.15.82:6833/3363736995] up:boot 2024-03-28T22:32:49.985 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:49 smithi043 ceph-mon[31823]: fsmap 2 up:standby 2024-03-28T22:32:49.985 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:49 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T22:32:49.985 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:49 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2197166264' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:32:50.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:49 smithi172 ceph-mon[36566]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:50.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:49 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:50.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:49 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:50.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:49 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:50.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:49 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:32:50.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:49 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:32:50.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:49 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:50.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:49 smithi172 ceph-mon[36566]: Deploying daemon mds.j on smithi043 2024-03-28T22:32:50.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:49 smithi172 ceph-mon[36566]: mds.? [v2:172.21.15.82:6832/3363736995,v1:172.21.15.82:6833/3363736995] up:boot 2024-03-28T22:32:50.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:49 smithi172 ceph-mon[36566]: fsmap 2 up:standby 2024-03-28T22:32:50.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:49 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T22:32:50.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:49 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2197166264' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:32:50.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:49 smithi082 ceph-mon[36471]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:50.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:49 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:50.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:49 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:50.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:49 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:50.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:49 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:32:50.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:49 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:32:50.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:49 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:50.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:49 smithi082 ceph-mon[36471]: Deploying daemon mds.j on smithi043 2024-03-28T22:32:50.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:49 smithi082 ceph-mon[36471]: mds.? [v2:172.21.15.82:6832/3363736995,v1:172.21.15.82:6833/3363736995] up:boot 2024-03-28T22:32:50.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:49 smithi082 ceph-mon[36471]: fsmap 2 up:standby 2024-03-28T22:32:50.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:49 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T22:32:50.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:49 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2197166264' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:32:51.243 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:32:51.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.645+0000 7f78b4e00640 1 -- 172.21.15.43:0/1473426041 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78b0102950 msgr2=0x7f78b0102dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:51.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.645+0000 7f78b4e00640 1 --2- 172.21.15.43:0/1473426041 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78b0102950 0x7f78b0102dd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f789c009900 tx=0x7f789c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:32:51.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.645+0000 7f78b4e00640 1 -- 172.21.15.43:0/1473426041 shutdown_connections 2024-03-28T22:32:51.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.645+0000 7f78b4e00640 1 --2- 172.21.15.43:0/1473426041 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78b0102950 0x7f78b0102dd0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:51.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.645+0000 7f78b4e00640 1 --2- 172.21.15.43:0/1473426041 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78b0103310 0x7f78b0109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:51.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.645+0000 7f78b4e00640 1 --2- 172.21.15.43:0/1473426041 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78b0101760 0x7f78b0101b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:51.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.645+0000 7f78b4e00640 1 -- 172.21.15.43:0/1473426041 >> 172.21.15.43:0/1473426041 conn(0x7f78b00fcd30 msgr2=0x7f78b00ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:32:51.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.645+0000 7f78b4e00640 1 -- 172.21.15.43:0/1473426041 shutdown_connections 2024-03-28T22:32:51.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.646+0000 7f78b4e00640 1 -- 172.21.15.43:0/1473426041 wait complete. 2024-03-28T22:32:51.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.647+0000 7f78b4e00640 1 Processor -- start 2024-03-28T22:32:51.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.647+0000 7f78b4e00640 1 -- start start 2024-03-28T22:32:51.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.648+0000 7f78b4e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78b0101760 0x7f78b0199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:51.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.648+0000 7f78b4e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78b0102950 0x7f78b019a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:51.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.648+0000 7f78b4e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78b0103310 0x7f78b01a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:51.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.648+0000 7f78b4e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f78b010d430 con 0x7f78b0101760 2024-03-28T22:32:51.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.648+0000 7f78b4e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f78b010d2b0 con 0x7f78b0103310 2024-03-28T22:32:51.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.648+0000 7f78b4e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f78b010d5b0 con 0x7f78b0102950 2024-03-28T22:32:51.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.648+0000 7f78ad400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78b0102950 0x7f78b019a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:51.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.648+0000 7f78ad400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78b0102950 0x7f78b019a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58026/0 (socket says 172.21.15.43:58026) 2024-03-28T22:32:51.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.648+0000 7f78ad400640 1 -- 172.21.15.43:0/1899028415 learned_addr learned my addr 172.21.15.43:0/1899028415 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:32:51.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.648+0000 7f78ade00640 1 --2- 172.21.15.43:0/1899028415 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78b0101760 0x7f78b0199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:51.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.648+0000 7f78ae800640 1 --2- 172.21.15.43:0/1899028415 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78b0103310 0x7f78b01a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:51.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.649+0000 7f78ad400640 1 -- 172.21.15.43:0/1899028415 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78b0103310 msgr2=0x7f78b01a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:51.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.649+0000 7f78ad400640 1 --2- 172.21.15.43:0/1899028415 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78b0103310 0x7f78b01a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:51.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.649+0000 7f78ad400640 1 -- 172.21.15.43:0/1899028415 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78b0101760 msgr2=0x7f78b0199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:51.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.649+0000 7f78ad400640 1 --2- 172.21.15.43:0/1899028415 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78b0101760 0x7f78b0199c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:51.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.649+0000 7f78ad400640 1 -- 172.21.15.43:0/1899028415 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78b01a18d0 con 0x7f78b0102950 2024-03-28T22:32:51.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.649+0000 7f78ade00640 1 --2- 172.21.15.43:0/1899028415 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78b0101760 0x7f78b0199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:32:51.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.650+0000 7f78ad400640 1 --2- 172.21.15.43:0/1899028415 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78b0102950 0x7f78b019a1d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f789c005ab0 tx=0x7f789c02c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:32:51.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.650+0000 7f78a6a00640 1 -- 172.21.15.43:0/1899028415 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c043020 con 0x7f78b0102950 2024-03-28T22:32:51.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.651+0000 7f78a6a00640 1 -- 172.21.15.43:0/1899028415 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f789c0054e0 con 0x7f78b0102950 2024-03-28T22:32:51.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.651+0000 7f78b4e00640 1 -- 172.21.15.43:0/1899028415 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78b01a1b60 con 0x7f78b0102950 2024-03-28T22:32:51.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.651+0000 7f78b4e00640 1 -- 172.21.15.43:0/1899028415 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78b01a20b0 con 0x7f78b0102950 2024-03-28T22:32:51.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.651+0000 7f78a6a00640 1 -- 172.21.15.43:0/1899028415 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c003ce0 con 0x7f78b0102950 2024-03-28T22:32:51.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.656+0000 7f78a6a00640 1 -- 172.21.15.43:0/1899028415 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f789c046020 con 0x7f78b0102950 2024-03-28T22:32:51.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.658+0000 7f78b4e00640 1 -- 172.21.15.43:0/1899028415 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7890005290 con 0x7f78b0102950 2024-03-28T22:32:51.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.659+0000 7f78a6a00640 1 --2- 172.21.15.43:0/1899028415 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f78940a1f50 0x7f78940a4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:51.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.659+0000 7f78a6a00640 1 -- 172.21.15.43:0/1899028415 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f789c0ef8b0 con 0x7f78b0102950 2024-03-28T22:32:51.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.662+0000 7f78ade00640 1 --2- 172.21.15.43:0/1899028415 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f78940a1f50 0x7f78940a4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:51.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.663+0000 7f78ade00640 1 --2- 172.21.15.43:0/1899028415 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f78940a1f50 0x7f78940a4410 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f78b019ab60 tx=0x7f78a000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:32:51.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.666+0000 7f78a6a00640 1 -- 172.21.15.43:0/1899028415 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f789c04d020 con 0x7f78b0102950 2024-03-28T22:32:51.902 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:51 smithi043 ceph-mon[31823]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:51.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.919+0000 7f78b4e00640 1 -- 172.21.15.43:0/1899028415 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7890005580 con 0x7f78b0102950 2024-03-28T22:32:51.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.920+0000 7f78a6a00640 1 -- 172.21.15.43:0/1899028415 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f789c0bd0e0 con 0x7f78b0102950 2024-03-28T22:32:51.919 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:32:51.919 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-28T22:32:51.920 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 3 2024-03-28T22:32:51.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.922+0000 7f78b4e00640 1 -- 172.21.15.43:0/1899028415 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f78940a1f50 msgr2=0x7f78940a4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:51.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.922+0000 7f78b4e00640 1 --2- 172.21.15.43:0/1899028415 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f78940a1f50 0x7f78940a4410 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f78b019ab60 tx=0x7f78a000b390 comp rx=0 tx=0).stop 2024-03-28T22:32:51.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.922+0000 7f78b4e00640 1 -- 172.21.15.43:0/1899028415 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78b0102950 msgr2=0x7f78b019a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:51.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.922+0000 7f78b4e00640 1 --2- 172.21.15.43:0/1899028415 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78b0102950 0x7f78b019a1d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f789c005ab0 tx=0x7f789c02c6c0 comp rx=0 tx=0).stop 2024-03-28T22:32:51.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.922+0000 7f78b4e00640 1 -- 172.21.15.43:0/1899028415 shutdown_connections 2024-03-28T22:32:51.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.922+0000 7f78b4e00640 1 --2- 172.21.15.43:0/1899028415 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f78940a1f50 0x7f78940a4410 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:51.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.922+0000 7f78b4e00640 1 --2- 172.21.15.43:0/1899028415 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78b0103310 0x7f78b01a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:51.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.922+0000 7f78b4e00640 1 --2- 172.21.15.43:0/1899028415 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78b0102950 0x7f78b019a1d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:51.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.922+0000 7f78b4e00640 1 --2- 172.21.15.43:0/1899028415 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78b0101760 0x7f78b0199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:51.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.922+0000 7f78b4e00640 1 -- 172.21.15.43:0/1899028415 >> 172.21.15.43:0/1899028415 conn(0x7f78b00fcd30 msgr2=0x7f78b0073980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:32:51.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.923+0000 7f78b4e00640 1 -- 172.21.15.43:0/1899028415 shutdown_connections 2024-03-28T22:32:51.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:51.923+0000 7f78b4e00640 1 -- 172.21.15.43:0/1899028415 wait complete. 2024-03-28T22:32:52.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:51 smithi172 ceph-mon[36566]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:52.152 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:51 smithi082 ceph-mon[36471]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:52.571 INFO:journalctl@ceph.mds.j.smithi043.stdout:Mar 28 22:32:52 smithi043 podman[69860]: 2024-03-28 22:32:52.292655846 +0000 UTC m=+0.150401256 container create ddb67d04777208c50055825d7934b66ac7ea8d5880fd12f288e59ca9900aa3e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-j, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-323d23d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-28T22:32:52.571 INFO:journalctl@ceph.mds.j.smithi043.stdout:Mar 28 22:32:52 smithi043 podman[69860]: 2024-03-28 22:32:52.571516854 +0000 UTC m=+0.429262271 container init ddb67d04777208c50055825d7934b66ac7ea8d5880fd12f288e59ca9900aa3e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-j, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-323d23d, org.label-schema.build-date=20240326, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T22:32:53.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:52 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1899028415' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:32:53.016 INFO:journalctl@ceph.mds.j.smithi043.stdout:Mar 28 22:32:52 smithi043 podman[69860]: 2024-03-28 22:32:52.581835414 +0000 UTC m=+0.439580831 container start ddb67d04777208c50055825d7934b66ac7ea8d5880fd12f288e59ca9900aa3e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-j, GIT_CLEAN=True, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-28T22:32:53.016 INFO:journalctl@ceph.mds.j.smithi043.stdout:Mar 28 22:32:52 smithi043 ceph-mds[69879]: -- 172.21.15.43:0/1492326302 <== mon.0 v2:172.21.15.43:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5628591b9a40 con 0x56285918d000 2024-03-28T22:32:53.016 INFO:journalctl@ceph.mds.j.smithi043.stdout:Mar 28 22:32:52 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-j[69875]: starting mds.j at 2024-03-28T22:32:53.016 INFO:journalctl@ceph.mds.j.smithi043.stdout:Mar 28 22:32:52 smithi043 bash[69860]: ddb67d04777208c50055825d7934b66ac7ea8d5880fd12f288e59ca9900aa3e3 2024-03-28T22:32:53.017 INFO:journalctl@ceph.mds.j.smithi043.stdout:Mar 28 22:32:52 smithi043 systemd[1]: Started Ceph mds.j for c2276428-ed51-11ee-95d0-87774f69a715. 2024-03-28T22:32:53.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:52 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1899028415' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:32:53.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:52 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1899028415' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:32:53.941 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:32:54.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:53 smithi043 ceph-mon[31823]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:54.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:53 smithi043 ceph-mon[31823]: mds.? [v2:172.21.15.43:6834/4195313768,v1:172.21.15.43:6835/4195313768] up:boot 2024-03-28T22:32:54.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:53 smithi043 ceph-mon[31823]: fsmap 3 up:standby 2024-03-28T22:32:54.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:53 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T22:32:54.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:53 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:54.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:53 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:54.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:53 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:54.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:53 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:32:54.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:53 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:32:54.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:53 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:54.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:53 smithi043 ceph-mon[31823]: Deploying daemon mds.d on smithi043 2024-03-28T22:32:54.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:53 smithi172 ceph-mon[36566]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:54.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:53 smithi172 ceph-mon[36566]: mds.? [v2:172.21.15.43:6834/4195313768,v1:172.21.15.43:6835/4195313768] up:boot 2024-03-28T22:32:54.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:53 smithi172 ceph-mon[36566]: fsmap 3 up:standby 2024-03-28T22:32:54.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:53 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T22:32:54.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:53 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:54.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:53 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:54.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:53 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:54.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:53 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:32:54.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:53 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:32:54.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:53 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:54.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:53 smithi172 ceph-mon[36566]: Deploying daemon mds.d on smithi043 2024-03-28T22:32:54.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.157+0000 7f08f2800640 1 -- 172.21.15.43:0/2147856623 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08ec075730 msgr2=0x7f08ec073b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:54.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.157+0000 7f08f2800640 1 --2- 172.21.15.43:0/2147856623 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08ec075730 0x7f08ec073b80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f08dc009900 tx=0x7f08dc02bcc0 comp rx=0 tx=0).stop 2024-03-28T22:32:54.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.157+0000 7f08f2800640 1 -- 172.21.15.43:0/2147856623 shutdown_connections 2024-03-28T22:32:54.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.157+0000 7f08f2800640 1 --2- 172.21.15.43:0/2147856623 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f08ec074150 0x7f08ec0745b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:54.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.157+0000 7f08f2800640 1 --2- 172.21.15.43:0/2147856623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08ec1095f0 0x7f08ec10ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:54.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.157+0000 7f08f2800640 1 --2- 172.21.15.43:0/2147856623 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08ec075730 0x7f08ec073b80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:54.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.157+0000 7f08f2800640 1 -- 172.21.15.43:0/2147856623 >> 172.21.15.43:0/2147856623 conn(0x7f08ec0fcd30 msgr2=0x7f08ec0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:32:54.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.158+0000 7f08f2800640 1 -- 172.21.15.43:0/2147856623 shutdown_connections 2024-03-28T22:32:54.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.158+0000 7f08f2800640 1 -- 172.21.15.43:0/2147856623 wait complete. 2024-03-28T22:32:54.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.159+0000 7f08f2800640 1 Processor -- start 2024-03-28T22:32:54.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.159+0000 7f08f2800640 1 -- start start 2024-03-28T22:32:54.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.159+0000 7f08f2800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08ec074150 0x7f08ec199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:54.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.159+0000 7f08f2800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f08ec075730 0x7f08ec19a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:54.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.159+0000 7f08f2800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08ec1095f0 0x7f08ec1a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:54.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.159+0000 7f08f2800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f08ec10f620 con 0x7f08ec074150 2024-03-28T22:32:54.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.159+0000 7f08f2800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f08ec10f4a0 con 0x7f08ec075730 2024-03-28T22:32:54.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.159+0000 7f08f2800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f08ec10f7a0 con 0x7f08ec1095f0 2024-03-28T22:32:54.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.159+0000 7f08eb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08ec074150 0x7f08ec199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:54.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.159+0000 7f08ebe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08ec1095f0 0x7f08ec1a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:54.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.159+0000 7f08eaa00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f08ec075730 0x7f08ec19a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:54.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.159+0000 7f08eb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08ec074150 0x7f08ec199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59632/0 (socket says 172.21.15.43:59632) 2024-03-28T22:32:54.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.160+0000 7f08eb400640 1 -- 172.21.15.43:0/2875464449 learned_addr learned my addr 172.21.15.43:0/2875464449 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:32:54.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.160+0000 7f08eb400640 1 -- 172.21.15.43:0/2875464449 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08ec1095f0 msgr2=0x7f08ec1a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:54.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.160+0000 7f08eb400640 1 --2- 172.21.15.43:0/2875464449 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08ec1095f0 0x7f08ec1a1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:54.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.160+0000 7f08eb400640 1 -- 172.21.15.43:0/2875464449 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f08ec075730 msgr2=0x7f08ec19a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:54.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.160+0000 7f08eb400640 1 --2- 172.21.15.43:0/2875464449 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f08ec075730 0x7f08ec19a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:54.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.160+0000 7f08eb400640 1 -- 172.21.15.43:0/2875464449 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08ec1a18b0 con 0x7f08ec074150 2024-03-28T22:32:54.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.160+0000 7f08eb400640 1 --2- 172.21.15.43:0/2875464449 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08ec074150 0x7f08ec199c70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f08dc000c00 tx=0x7f08dc02c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:32:54.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.161+0000 7f08e3e00640 1 -- 172.21.15.43:0/2875464449 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08dc043020 con 0x7f08ec074150 2024-03-28T22:32:54.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.161+0000 7f08f2800640 1 -- 172.21.15.43:0/2875464449 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08ec1a1b40 con 0x7f08ec074150 2024-03-28T22:32:54.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.161+0000 7f08e3e00640 1 -- 172.21.15.43:0/2875464449 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08dc0054e0 con 0x7f08ec074150 2024-03-28T22:32:54.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.161+0000 7f08e3e00640 1 -- 172.21.15.43:0/2875464449 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08dc003b20 con 0x7f08ec074150 2024-03-28T22:32:54.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.161+0000 7f08f2800640 1 -- 172.21.15.43:0/2875464449 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08ec1a20e0 con 0x7f08ec074150 2024-03-28T22:32:54.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.162+0000 7f08f2800640 1 -- 172.21.15.43:0/2875464449 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08ec1083b0 con 0x7f08ec074150 2024-03-28T22:32:54.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.164+0000 7f08e3e00640 1 -- 172.21.15.43:0/2875464449 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f08dc046020 con 0x7f08ec074150 2024-03-28T22:32:54.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.165+0000 7f08e3e00640 1 --2- 172.21.15.43:0/2875464449 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f08d00a1e00 0x7f08d00a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:54.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.165+0000 7f08e3e00640 1 -- 172.21.15.43:0/2875464449 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f08dc0ef960 con 0x7f08ec074150 2024-03-28T22:32:54.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.166+0000 7f08eaa00640 1 --2- 172.21.15.43:0/2875464449 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f08d00a1e00 0x7f08d00a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:54.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.166+0000 7f08eaa00640 1 --2- 172.21.15.43:0/2875464449 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f08d00a1e00 0x7f08d00a42c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f08ec19b170 tx=0x7f08d8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:32:54.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.174+0000 7f08e3e00640 1 -- 172.21.15.43:0/2875464449 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f08dc04d020 con 0x7f08ec074150 2024-03-28T22:32:54.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:53 smithi082 ceph-mon[36471]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:54.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:53 smithi082 ceph-mon[36471]: mds.? [v2:172.21.15.43:6834/4195313768,v1:172.21.15.43:6835/4195313768] up:boot 2024-03-28T22:32:54.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:53 smithi082 ceph-mon[36471]: fsmap 3 up:standby 2024-03-28T22:32:54.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:53 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T22:32:54.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:53 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:54.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:53 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:54.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:53 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:54.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:53 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:32:54.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:53 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:32:54.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:53 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:54.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:53 smithi082 ceph-mon[36471]: Deploying daemon mds.d on smithi043 2024-03-28T22:32:54.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.418+0000 7f08f2800640 1 -- 172.21.15.43:0/2875464449 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f08ec02d170 con 0x7f08ec074150 2024-03-28T22:32:54.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.418+0000 7f08e3e00640 1 -- 172.21.15.43:0/2875464449 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3046 (secure 0 0 0) 0x7f08dc0bd2b0 con 0x7f08ec074150 2024-03-28T22:32:54.418 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:32:54.418 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[]} 2024-03-28T22:32:54.418 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 4 2024-03-28T22:32:54.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.421+0000 7f08f2800640 1 -- 172.21.15.43:0/2875464449 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f08d00a1e00 msgr2=0x7f08d00a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:54.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.421+0000 7f08f2800640 1 --2- 172.21.15.43:0/2875464449 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f08d00a1e00 0x7f08d00a42c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f08ec19b170 tx=0x7f08d8005f90 comp rx=0 tx=0).stop 2024-03-28T22:32:54.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.421+0000 7f08f2800640 1 -- 172.21.15.43:0/2875464449 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08ec074150 msgr2=0x7f08ec199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:54.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.421+0000 7f08f2800640 1 --2- 172.21.15.43:0/2875464449 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08ec074150 0x7f08ec199c70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f08dc000c00 tx=0x7f08dc02c6e0 comp rx=0 tx=0).stop 2024-03-28T22:32:54.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.421+0000 7f08f2800640 1 -- 172.21.15.43:0/2875464449 shutdown_connections 2024-03-28T22:32:54.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.421+0000 7f08f2800640 1 --2- 172.21.15.43:0/2875464449 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f08d00a1e00 0x7f08d00a42c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:54.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.421+0000 7f08f2800640 1 --2- 172.21.15.43:0/2875464449 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f08ec075730 0x7f08ec19a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:54.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.421+0000 7f08f2800640 1 --2- 172.21.15.43:0/2875464449 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08ec1095f0 0x7f08ec1a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:54.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.421+0000 7f08f2800640 1 --2- 172.21.15.43:0/2875464449 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08ec074150 0x7f08ec199c70 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:54.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.421+0000 7f08f2800640 1 -- 172.21.15.43:0/2875464449 >> 172.21.15.43:0/2875464449 conn(0x7f08ec0fcd30 msgr2=0x7f08ec105580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:32:54.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.421+0000 7f08f2800640 1 -- 172.21.15.43:0/2875464449 shutdown_connections 2024-03-28T22:32:54.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:54.421+0000 7f08f2800640 1 -- 172.21.15.43:0/2875464449 wait complete. 2024-03-28T22:32:54.997 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:54 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2875464449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:32:55.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:54 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2875464449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:32:55.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:54 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2875464449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:32:56.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:55 smithi043 ceph-mon[31823]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:56.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:55 smithi172 ceph-mon[36566]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:56.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:55 smithi082 ceph-mon[36471]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:56.439 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:32:56.460 INFO:journalctl@ceph.mds.d.smithi043.stdout:Mar 28 22:32:56 smithi043 systemd[1]: Starting Ceph mds.d for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:32:56.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.665+0000 7f6821000640 1 -- 172.21.15.43:0/3029594772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f681c075730 msgr2=0x7f681c073b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:56.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.665+0000 7f6821000640 1 --2- 172.21.15.43:0/3029594772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f681c075730 0x7f681c073b80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f680c009900 tx=0x7f680c02bcd0 comp rx=0 tx=0).stop 2024-03-28T22:32:56.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.665+0000 7f6821000640 1 -- 172.21.15.43:0/3029594772 shutdown_connections 2024-03-28T22:32:56.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.665+0000 7f6821000640 1 --2- 172.21.15.43:0/3029594772 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f681c074150 0x7f681c0745b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:56.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.665+0000 7f6821000640 1 --2- 172.21.15.43:0/3029594772 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f681c109600 0x7f681c10baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:56.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.665+0000 7f6821000640 1 --2- 172.21.15.43:0/3029594772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f681c075730 0x7f681c073b80 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:56.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.665+0000 7f6821000640 1 -- 172.21.15.43:0/3029594772 >> 172.21.15.43:0/3029594772 conn(0x7f681c0fcd00 msgr2=0x7f681c0ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:32:56.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.665+0000 7f6821000640 1 -- 172.21.15.43:0/3029594772 shutdown_connections 2024-03-28T22:32:56.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.665+0000 7f6821000640 1 -- 172.21.15.43:0/3029594772 wait complete. 2024-03-28T22:32:56.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.666+0000 7f6821000640 1 Processor -- start 2024-03-28T22:32:56.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.666+0000 7f6821000640 1 -- start start 2024-03-28T22:32:56.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.667+0000 7f6821000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f681c074150 0x7f681c104af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:56.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.667+0000 7f6821000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f681c075730 0x7f681c105030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:56.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.667+0000 7f6821000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f681c109600 0x7f681c101660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:56.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.667+0000 7f6821000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f681c10f610 con 0x7f681c074150 2024-03-28T22:32:56.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.667+0000 7f6821000640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f681c10f490 con 0x7f681c075730 2024-03-28T22:32:56.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.667+0000 7f6821000640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f681c10f790 con 0x7f681c109600 2024-03-28T22:32:56.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.667+0000 7f6819e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f681c074150 0x7f681c104af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:56.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.667+0000 7f6819e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f681c074150 0x7f681c104af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37408/0 (socket says 172.21.15.43:37408) 2024-03-28T22:32:56.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.667+0000 7f6819e00640 1 -- 172.21.15.43:0/2660675810 learned_addr learned my addr 172.21.15.43:0/2660675810 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:32:56.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.667+0000 7f681a800640 1 --2- 172.21.15.43:0/2660675810 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f681c109600 0x7f681c101660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:56.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.667+0000 7f6819400640 1 --2- 172.21.15.43:0/2660675810 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f681c075730 0x7f681c105030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:56.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.668+0000 7f6819e00640 1 -- 172.21.15.43:0/2660675810 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f681c109600 msgr2=0x7f681c101660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:56.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.668+0000 7f6819e00640 1 --2- 172.21.15.43:0/2660675810 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f681c109600 0x7f681c101660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:56.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.668+0000 7f6819e00640 1 -- 172.21.15.43:0/2660675810 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f681c075730 msgr2=0x7f681c105030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:56.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.668+0000 7f6819e00640 1 --2- 172.21.15.43:0/2660675810 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f681c075730 0x7f681c105030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:56.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.668+0000 7f6819e00640 1 -- 172.21.15.43:0/2660675810 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f681c101f20 con 0x7f681c074150 2024-03-28T22:32:56.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.668+0000 7f6819e00640 1 --2- 172.21.15.43:0/2660675810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f681c074150 0x7f681c104af0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f680c000c00 tx=0x7f680c005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:32:56.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.668+0000 7f6812a00640 1 -- 172.21.15.43:0/2660675810 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c043020 con 0x7f681c074150 2024-03-28T22:32:56.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.668+0000 7f6812a00640 1 -- 172.21.15.43:0/2660675810 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f680c0054e0 con 0x7f681c074150 2024-03-28T22:32:56.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.668+0000 7f6812a00640 1 -- 172.21.15.43:0/2660675810 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c003cf0 con 0x7f681c074150 2024-03-28T22:32:56.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.669+0000 7f6821000640 1 -- 172.21.15.43:0/2660675810 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f681c1a60b0 con 0x7f681c074150 2024-03-28T22:32:56.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.669+0000 7f6821000640 1 -- 172.21.15.43:0/2660675810 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f681c1a64b0 con 0x7f681c074150 2024-03-28T22:32:56.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.670+0000 7f6821000640 1 -- 172.21.15.43:0/2660675810 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f681c068990 con 0x7f681c074150 2024-03-28T22:32:56.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.672+0000 7f6812a00640 1 -- 172.21.15.43:0/2660675810 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f680c046020 con 0x7f681c074150 2024-03-28T22:32:56.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.673+0000 7f6812a00640 1 --2- 172.21.15.43:0/2660675810 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f68000a1e00 0x7f68000a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:56.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.673+0000 7f6812a00640 1 -- 172.21.15.43:0/2660675810 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f680c0efae0 con 0x7f681c074150 2024-03-28T22:32:56.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.673+0000 7f6819400640 1 --2- 172.21.15.43:0/2660675810 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f68000a1e00 0x7f68000a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:56.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.674+0000 7f6819400640 1 --2- 172.21.15.43:0/2660675810 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f68000a1e00 0x7f68000a42c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f681c102a70 tx=0x7f680800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:32:56.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.679+0000 7f6812a00640 1 -- 172.21.15.43:0/2660675810 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f680c04d020 con 0x7f681c074150 2024-03-28T22:32:56.767 INFO:journalctl@ceph.mds.d.smithi043.stdout:Mar 28 22:32:56 smithi043 podman[70213]: 2024-03-28 22:32:56.366571076 +0000 UTC m=+0.038512094 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:32:56.767 INFO:journalctl@ceph.mds.d.smithi043.stdout:Mar 28 22:32:56 smithi043 podman[70213]: 2024-03-28 22:32:56.529010352 +0000 UTC m=+0.200951390 container create 04d255d931292be2e21e582213ce2659d8e3d15400e02086d582e8748d6414fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-d, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, RELEASE=squid-323d23d, org.label-schema.build-date=20240326) 2024-03-28T22:32:56.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.939+0000 7f6821000640 1 -- 172.21.15.43:0/2660675810 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f681c062350 con 0x7f681c074150 2024-03-28T22:32:56.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.941+0000 7f6812a00640 1 -- 172.21.15.43:0/2660675810 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3046 (secure 0 0 0) 0x7f680c0bd310 con 0x7f681c074150 2024-03-28T22:32:56.941 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:32:56.941 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[]} 2024-03-28T22:32:56.941 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 4 2024-03-28T22:32:56.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.944+0000 7f6821000640 1 -- 172.21.15.43:0/2660675810 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f68000a1e00 msgr2=0x7f68000a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:56.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.944+0000 7f6821000640 1 --2- 172.21.15.43:0/2660675810 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f68000a1e00 0x7f68000a42c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f681c102a70 tx=0x7f680800b410 comp rx=0 tx=0).stop 2024-03-28T22:32:56.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.944+0000 7f6821000640 1 -- 172.21.15.43:0/2660675810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f681c074150 msgr2=0x7f681c104af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:56.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.944+0000 7f6821000640 1 --2- 172.21.15.43:0/2660675810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f681c074150 0x7f681c104af0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f680c000c00 tx=0x7f680c005dd0 comp rx=0 tx=0).stop 2024-03-28T22:32:56.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.944+0000 7f6821000640 1 -- 172.21.15.43:0/2660675810 shutdown_connections 2024-03-28T22:32:56.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.944+0000 7f6821000640 1 --2- 172.21.15.43:0/2660675810 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f68000a1e00 0x7f68000a42c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:56.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.944+0000 7f6821000640 1 --2- 172.21.15.43:0/2660675810 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f681c075730 0x7f681c105030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:56.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.944+0000 7f6821000640 1 --2- 172.21.15.43:0/2660675810 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f681c109600 0x7f681c101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:56.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.944+0000 7f6821000640 1 --2- 172.21.15.43:0/2660675810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f681c074150 0x7f681c104af0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:56.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.944+0000 7f6821000640 1 -- 172.21.15.43:0/2660675810 >> 172.21.15.43:0/2660675810 conn(0x7f681c0fcd00 msgr2=0x7f681c105590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:32:56.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.944+0000 7f6821000640 1 -- 172.21.15.43:0/2660675810 shutdown_connections 2024-03-28T22:32:56.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:56.944+0000 7f6821000640 1 -- 172.21.15.43:0/2660675810 wait complete. 2024-03-28T22:32:57.267 INFO:journalctl@ceph.mds.d.smithi043.stdout:Mar 28 22:32:56 smithi043 podman[70213]: 2024-03-28 22:32:56.814130137 +0000 UTC m=+0.486071192 container init 04d255d931292be2e21e582213ce2659d8e3d15400e02086d582e8748d6414fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-d, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-323d23d, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-28T22:32:57.267 INFO:journalctl@ceph.mds.d.smithi043.stdout:Mar 28 22:32:56 smithi043 podman[70213]: 2024-03-28 22:32:56.824406245 +0000 UTC m=+0.496347323 container start 04d255d931292be2e21e582213ce2659d8e3d15400e02086d582e8748d6414fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-323d23d, CEPH_POINT_RELEASE=) 2024-03-28T22:32:57.267 INFO:journalctl@ceph.mds.d.smithi043.stdout:Mar 28 22:32:56 smithi043 ceph-mds[70271]: -- 172.21.15.43:0/1219673311 <== mon.2 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5556738aba40 con 0x55567387e800 2024-03-28T22:32:57.267 INFO:journalctl@ceph.mds.d.smithi043.stdout:Mar 28 22:32:56 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-d[70267]: starting mds.d at 2024-03-28T22:32:57.267 INFO:journalctl@ceph.mds.d.smithi043.stdout:Mar 28 22:32:56 smithi043 bash[70213]: 04d255d931292be2e21e582213ce2659d8e3d15400e02086d582e8748d6414fd 2024-03-28T22:32:57.267 INFO:journalctl@ceph.mds.d.smithi043.stdout:Mar 28 22:32:56 smithi043 systemd[1]: Started Ceph mds.d for c2276428-ed51-11ee-95d0-87774f69a715. 2024-03-28T22:32:58.044 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:57 smithi082 ceph-mon[36471]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:58.044 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:57 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2660675810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:32:58.044 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:57 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:58.044 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:57 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:58.044 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:57 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:58.044 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:57 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:32:58.044 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:57 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:32:58.044 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:57 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:58.044 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:57 smithi082 ceph-mon[36471]: Deploying daemon mds.e on smithi082 2024-03-28T22:32:58.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:57 smithi172 ceph-mon[36566]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:58.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:57 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2660675810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:32:58.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:57 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:58.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:57 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:58.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:57 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:58.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:57 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:32:58.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:57 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:32:58.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:57 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:58.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:57 smithi172 ceph-mon[36566]: Deploying daemon mds.e on smithi082 2024-03-28T22:32:58.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:57 smithi043 ceph-mon[31823]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:32:58.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:57 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2660675810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:32:58.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:57 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:58.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:57 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:58.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:57 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:32:58.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:57 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:32:58.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:57 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:32:58.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:57 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:32:58.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:57 smithi043 ceph-mon[31823]: Deploying daemon mds.e on smithi082 2024-03-28T22:32:58.965 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:32:59.048 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:58 smithi043 ceph-mon[31823]: mds.? [v2:172.21.15.43:6836/1330639541,v1:172.21.15.43:6837/1330639541] up:boot 2024-03-28T22:32:59.048 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:58 smithi043 ceph-mon[31823]: fsmap 4 up:standby 2024-03-28T22:32:59.048 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:58 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T22:32:59.069 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:58 smithi082 ceph-mon[36471]: mds.? [v2:172.21.15.43:6836/1330639541,v1:172.21.15.43:6837/1330639541] up:boot 2024-03-28T22:32:59.069 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:58 smithi082 ceph-mon[36471]: fsmap 4 up:standby 2024-03-28T22:32:59.070 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:58 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T22:32:59.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:58 smithi172 ceph-mon[36566]: mds.? [v2:172.21.15.43:6836/1330639541,v1:172.21.15.43:6837/1330639541] up:boot 2024-03-28T22:32:59.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:58 smithi172 ceph-mon[36566]: fsmap 4 up:standby 2024-03-28T22:32:59.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:58 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T22:32:59.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.188+0000 7f2022e00640 1 -- 172.21.15.43:0/1424881518 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f201c10db60 msgr2=0x7f201c10ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:59.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.188+0000 7f2022e00640 1 --2- 172.21.15.43:0/1424881518 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f201c10db60 0x7f201c10ff50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2014009cd0 tx=0x7f201402c0e0 comp rx=0 tx=0).stop 2024-03-28T22:32:59.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.188+0000 7f2022e00640 1 -- 172.21.15.43:0/1424881518 shutdown_connections 2024-03-28T22:32:59.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.188+0000 7f2022e00640 1 --2- 172.21.15.43:0/1424881518 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f201c10db60 0x7f201c10ff50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:59.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.188+0000 7f2022e00640 1 --2- 172.21.15.43:0/1424881518 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f201c0741b0 0x7f201c074630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:59.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.188+0000 7f2022e00640 1 --2- 172.21.15.43:0/1424881518 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f201c075790 0x7f201c073be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:59.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.188+0000 7f2022e00640 1 -- 172.21.15.43:0/1424881518 >> 172.21.15.43:0/1424881518 conn(0x7f201c0fcae0 msgr2=0x7f201c0fef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:32:59.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.188+0000 7f2022e00640 1 -- 172.21.15.43:0/1424881518 shutdown_connections 2024-03-28T22:32:59.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.189+0000 7f2022e00640 1 -- 172.21.15.43:0/1424881518 wait complete. 2024-03-28T22:32:59.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.190+0000 7f2022e00640 1 Processor -- start 2024-03-28T22:32:59.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.190+0000 7f2022e00640 1 -- start start 2024-03-28T22:32:59.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.190+0000 7f2022e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f201c0741b0 0x7f201c19de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:59.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.190+0000 7f2022e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f201c075790 0x7f201c19e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:59.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.190+0000 7f2022e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f201c10db60 0x7f201c1a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:59.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.190+0000 7f2022e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f201c113800 con 0x7f201c10db60 2024-03-28T22:32:59.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.190+0000 7f2022e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f201c113680 con 0x7f201c075790 2024-03-28T22:32:59.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.190+0000 7f2022e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f201c113980 con 0x7f201c0741b0 2024-03-28T22:32:59.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.191+0000 7f2021000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f201c075790 0x7f201c19e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:59.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.191+0000 7f2021000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f201c075790 0x7f201c19e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:43790/0 (socket says 172.21.15.43:43790) 2024-03-28T22:32:59.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.191+0000 7f2021000640 1 -- 172.21.15.43:0/1498278171 learned_addr learned my addr 172.21.15.43:0/1498278171 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:32:59.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.191+0000 7f2021a00640 1 --2- 172.21.15.43:0/1498278171 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f201c0741b0 0x7f201c19de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:59.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.191+0000 7f2022400640 1 --2- 172.21.15.43:0/1498278171 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f201c10db60 0x7f201c1a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:59.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.191+0000 7f2021000640 1 -- 172.21.15.43:0/1498278171 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f201c0741b0 msgr2=0x7f201c19de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:59.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.191+0000 7f2021000640 1 --2- 172.21.15.43:0/1498278171 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f201c0741b0 0x7f201c19de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:59.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.191+0000 7f2021000640 1 -- 172.21.15.43:0/1498278171 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f201c10db60 msgr2=0x7f201c1a5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:59.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.191+0000 7f2021000640 1 --2- 172.21.15.43:0/1498278171 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f201c10db60 0x7f201c1a5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:59.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.191+0000 7f2021000640 1 -- 172.21.15.43:0/1498278171 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f201c1a5aa0 con 0x7f201c075790 2024-03-28T22:32:59.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.191+0000 7f2021000640 1 --2- 172.21.15.43:0/1498278171 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f201c075790 0x7f201c19e3c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f200800e9f0 tx=0x7f200800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:32:59.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.191+0000 7f201a000640 1 -- 172.21.15.43:0/1498278171 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200800cd80 con 0x7f201c075790 2024-03-28T22:32:59.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.192+0000 7f201a000640 1 -- 172.21.15.43:0/1498278171 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2008005500 con 0x7f201c075790 2024-03-28T22:32:59.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.192+0000 7f201a000640 1 -- 172.21.15.43:0/1498278171 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2008010690 con 0x7f201c075790 2024-03-28T22:32:59.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.192+0000 7f2022e00640 1 -- 172.21.15.43:0/1498278171 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f201c1a5d90 con 0x7f201c075790 2024-03-28T22:32:59.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.192+0000 7f2022e00640 1 -- 172.21.15.43:0/1498278171 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f201c1a6230 con 0x7f201c075790 2024-03-28T22:32:59.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.193+0000 7f2022e00640 1 -- 172.21.15.43:0/1498278171 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f201c103d30 con 0x7f201c075790 2024-03-28T22:32:59.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.195+0000 7f201a000640 1 -- 172.21.15.43:0/1498278171 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f20080046c0 con 0x7f201c075790 2024-03-28T22:32:59.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.196+0000 7f201a000640 1 --2- 172.21.15.43:0/1498278171 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f20040a1e00 0x7f20040a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:32:59.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.196+0000 7f201a000640 1 -- 172.21.15.43:0/1498278171 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f20080cdd10 con 0x7f201c075790 2024-03-28T22:32:59.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.196+0000 7f2021a00640 1 --2- 172.21.15.43:0/1498278171 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f20040a1e00 0x7f20040a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:32:59.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.197+0000 7f2021a00640 1 --2- 172.21.15.43:0/1498278171 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f20040a1e00 0x7f20040a42c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2010009bd0 tx=0x7f2010005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:32:59.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.200+0000 7f201a000640 1 -- 172.21.15.43:0/1498278171 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f20080d3020 con 0x7f201c075790 2024-03-28T22:32:59.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.443+0000 7f2022e00640 1 -- 172.21.15.43:0/1498278171 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f201c062350 con 0x7f201c075790 2024-03-28T22:32:59.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.444+0000 7f201a000640 1 -- 172.21.15.43:0/1498278171 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3871 (secure 0 0 0) 0x7f200809b540 con 0x7f201c075790 2024-03-28T22:32:59.444 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:32:59.444 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[]} 2024-03-28T22:32:59.444 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 5 2024-03-28T22:32:59.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.447+0000 7f2022e00640 1 -- 172.21.15.43:0/1498278171 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f20040a1e00 msgr2=0x7f20040a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:59.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.447+0000 7f2022e00640 1 --2- 172.21.15.43:0/1498278171 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f20040a1e00 0x7f20040a42c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2010009bd0 tx=0x7f2010005f90 comp rx=0 tx=0).stop 2024-03-28T22:32:59.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.447+0000 7f2022e00640 1 -- 172.21.15.43:0/1498278171 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f201c075790 msgr2=0x7f201c19e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:32:59.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.447+0000 7f2022e00640 1 --2- 172.21.15.43:0/1498278171 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f201c075790 0x7f201c19e3c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f200800e9f0 tx=0x7f200800edf0 comp rx=0 tx=0).stop 2024-03-28T22:32:59.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.447+0000 7f2022e00640 1 -- 172.21.15.43:0/1498278171 shutdown_connections 2024-03-28T22:32:59.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.447+0000 7f2022e00640 1 --2- 172.21.15.43:0/1498278171 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f20040a1e00 0x7f20040a42c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:59.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.447+0000 7f2022e00640 1 --2- 172.21.15.43:0/1498278171 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f201c075790 0x7f201c19e3c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:59.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.447+0000 7f2022e00640 1 --2- 172.21.15.43:0/1498278171 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f201c10db60 0x7f201c1a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:59.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.447+0000 7f2022e00640 1 --2- 172.21.15.43:0/1498278171 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f201c0741b0 0x7f201c19de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:32:59.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.447+0000 7f2022e00640 1 -- 172.21.15.43:0/1498278171 >> 172.21.15.43:0/1498278171 conn(0x7f201c0fcae0 msgr2=0x7f201c100f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:32:59.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.447+0000 7f2022e00640 1 -- 172.21.15.43:0/1498278171 shutdown_connections 2024-03-28T22:32:59.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:32:59.448+0000 7f2022e00640 1 -- 172.21.15.43:0/1498278171 wait complete. 2024-03-28T22:32:59.449 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 28 22:32:59 smithi082 systemd[1]: Starting Ceph mds.e for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:32:59.728 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 28 22:32:59 smithi082 podman[66624]: 2024-03-28 22:32:59.371597474 +0000 UTC m=+0.018117185 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:32:59.728 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 28 22:32:59 smithi082 podman[66624]: 2024-03-28 22:32:59.494313176 +0000 UTC m=+0.140832903 container create c3a5469d05e7a31f57c934c8bf833e457423ec64ad918012c008f1a3571fe039 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-e, RELEASE=squid-323d23d, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-28T22:33:00.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:59 smithi172 ceph-mon[36566]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:00.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:32:59 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1498278171' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:00.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:59 smithi082 ceph-mon[36471]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:00.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:32:59 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1498278171' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:00.200 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 28 22:32:59 smithi082 podman[66624]: 2024-03-28 22:32:59.73008271 +0000 UTC m=+0.376602416 container init c3a5469d05e7a31f57c934c8bf833e457423ec64ad918012c008f1a3571fe039 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-e, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD) 2024-03-28T22:33:00.200 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 28 22:32:59 smithi082 podman[66624]: 2024-03-28 22:32:59.732733918 +0000 UTC m=+0.379253626 container start c3a5469d05e7a31f57c934c8bf833e457423ec64ad918012c008f1a3571fe039 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-e, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=squid-323d23d, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-28T22:33:00.200 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 28 22:32:59 smithi082 ceph-mds[66643]: -- 172.21.15.82:0/2887746495 <== mon.2 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55abf50a1a40 con 0x55abf5074c00 2024-03-28T22:33:00.200 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 28 22:32:59 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-e[66639]: starting mds.e at 2024-03-28T22:33:00.200 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 28 22:32:59 smithi082 bash[66624]: c3a5469d05e7a31f57c934c8bf833e457423ec64ad918012c008f1a3571fe039 2024-03-28T22:33:00.200 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 28 22:32:59 smithi082 systemd[1]: Started Ceph mds.e for c2276428-ed51-11ee-95d0-87774f69a715. 2024-03-28T22:33:00.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:59 smithi043 ceph-mon[31823]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:00.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:32:59 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1498278171' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:00.865 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:00 smithi172 ceph-mon[36566]: mds.? [v2:172.21.15.82:6834/1084988072,v1:172.21.15.82:6835/1084988072] up:boot 2024-03-28T22:33:00.865 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:00 smithi172 ceph-mon[36566]: fsmap 5 up:standby 2024-03-28T22:33:00.865 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:00 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T22:33:00.865 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:00 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:00.865 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:00 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:00.865 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:00 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:00.865 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:00 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:33:00.865 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:00 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:33:00.865 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:00 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:33:00.866 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:00 smithi172 ceph-mon[36566]: Deploying daemon mds.f on smithi172 2024-03-28T22:33:01.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:00 smithi082 ceph-mon[36471]: mds.? [v2:172.21.15.82:6834/1084988072,v1:172.21.15.82:6835/1084988072] up:boot 2024-03-28T22:33:01.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:00 smithi082 ceph-mon[36471]: fsmap 5 up:standby 2024-03-28T22:33:01.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:00 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T22:33:01.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:00 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:01.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:00 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:01.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:00 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:01.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:00 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:33:01.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:00 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:33:01.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:00 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:33:01.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:00 smithi082 ceph-mon[36471]: Deploying daemon mds.f on smithi172 2024-03-28T22:33:01.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:00 smithi043 ceph-mon[31823]: mds.? [v2:172.21.15.82:6834/1084988072,v1:172.21.15.82:6835/1084988072] up:boot 2024-03-28T22:33:01.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:00 smithi043 ceph-mon[31823]: fsmap 5 up:standby 2024-03-28T22:33:01.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:00 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T22:33:01.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:00 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:01.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:00 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:01.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:00 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:01.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:00 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:33:01.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:00 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:33:01.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:00 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:33:01.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:00 smithi043 ceph-mon[31823]: Deploying daemon mds.f on smithi172 2024-03-28T22:33:01.468 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:33:01.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.701+0000 7f53d1400640 1 -- 172.21.15.43:0/3889612296 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53cc107bb0 msgr2=0x7f53cc107fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:01.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.701+0000 7f53d1400640 1 --2- 172.21.15.43:0/3889612296 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53cc107bb0 0x7f53cc107fd0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f53bc009ca0 tx=0x7f53bc02c0f0 comp rx=0 tx=0).stop 2024-03-28T22:33:01.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.701+0000 7f53d1400640 1 -- 172.21.15.43:0/3889612296 shutdown_connections 2024-03-28T22:33:01.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.701+0000 7f53d1400640 1 --2- 172.21.15.43:0/3889612296 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f53cc109760 0x7f53cc10fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:01.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.701+0000 7f53d1400640 1 --2- 172.21.15.43:0/3889612296 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53cc108da0 0x7f53cc109220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:01.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.701+0000 7f53d1400640 1 --2- 172.21.15.43:0/3889612296 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53cc107bb0 0x7f53cc107fd0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:01.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.701+0000 7f53d1400640 1 -- 172.21.15.43:0/3889612296 >> 172.21.15.43:0/3889612296 conn(0x7f53cc076b60 msgr2=0x7f53cc078f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:01.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.701+0000 7f53d1400640 1 -- 172.21.15.43:0/3889612296 shutdown_connections 2024-03-28T22:33:01.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.701+0000 7f53d1400640 1 -- 172.21.15.43:0/3889612296 wait complete. 2024-03-28T22:33:01.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.702+0000 7f53d1400640 1 Processor -- start 2024-03-28T22:33:01.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.702+0000 7f53d1400640 1 -- start start 2024-03-28T22:33:01.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.702+0000 7f53d1400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53cc107bb0 0x7f53cc19dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:01.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.702+0000 7f53d1400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f53cc108da0 0x7f53cc19e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:01.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.702+0000 7f53d1400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53cc109760 0x7f53cc1a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:01.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.702+0000 7f53d1400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f53cc113820 con 0x7f53cc107bb0 2024-03-28T22:33:01.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.702+0000 7f53d1400640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f53cc1136a0 con 0x7f53cc108da0 2024-03-28T22:33:01.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.702+0000 7f53d1400640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f53cc1139a0 con 0x7f53cc109760 2024-03-28T22:33:01.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.703+0000 7f53cbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53cc107bb0 0x7f53cc19dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:01.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.703+0000 7f53cb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f53cc108da0 0x7f53cc19e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:01.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.703+0000 7f53cb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f53cc108da0 0x7f53cc19e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:43822/0 (socket says 172.21.15.43:43822) 2024-03-28T22:33:01.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.703+0000 7f53cb400640 1 -- 172.21.15.43:0/2288313672 learned_addr learned my addr 172.21.15.43:0/2288313672 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:01.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.703+0000 7f53cbe00640 1 --2- 172.21.15.43:0/2288313672 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53cc107bb0 0x7f53cc19dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37464/0 (socket says 172.21.15.43:37464) 2024-03-28T22:33:01.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.703+0000 7f53d0a00640 1 --2- 172.21.15.43:0/2288313672 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53cc109760 0x7f53cc1a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:01.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.703+0000 7f53cb400640 1 -- 172.21.15.43:0/2288313672 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53cc109760 msgr2=0x7f53cc1a5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:01.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.703+0000 7f53cb400640 1 --2- 172.21.15.43:0/2288313672 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53cc109760 0x7f53cc1a5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:01.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.703+0000 7f53cb400640 1 -- 172.21.15.43:0/2288313672 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53cc107bb0 msgr2=0x7f53cc19dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:01.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.703+0000 7f53cb400640 1 --2- 172.21.15.43:0/2288313672 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53cc107bb0 0x7f53cc19dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:01.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.703+0000 7f53cb400640 1 -- 172.21.15.43:0/2288313672 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53cc1a5ae0 con 0x7f53cc108da0 2024-03-28T22:33:01.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.703+0000 7f53d0a00640 1 --2- 172.21.15.43:0/2288313672 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53cc109760 0x7f53cc1a5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:33:01.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.703+0000 7f53cbe00640 1 --2- 172.21.15.43:0/2288313672 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53cc107bb0 0x7f53cc19dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:33:01.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.703+0000 7f53cb400640 1 --2- 172.21.15.43:0/2288313672 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f53cc108da0 0x7f53cc19e400 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f53b800e9f0 tx=0x7f53b800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:01.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.704+0000 7f53c8c00640 1 -- 172.21.15.43:0/2288313672 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b800cd80 con 0x7f53cc108da0 2024-03-28T22:33:01.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.704+0000 7f53c8c00640 1 -- 172.21.15.43:0/2288313672 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53b8005500 con 0x7f53cc108da0 2024-03-28T22:33:01.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.704+0000 7f53c8c00640 1 -- 172.21.15.43:0/2288313672 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b8010690 con 0x7f53cc108da0 2024-03-28T22:33:01.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.704+0000 7f53d1400640 1 -- 172.21.15.43:0/2288313672 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53cc1a5dd0 con 0x7f53cc108da0 2024-03-28T22:33:01.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.704+0000 7f53d1400640 1 -- 172.21.15.43:0/2288313672 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53cc1a6240 con 0x7f53cc108da0 2024-03-28T22:33:01.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.705+0000 7f53d1400640 1 -- 172.21.15.43:0/2288313672 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53cc10eed0 con 0x7f53cc108da0 2024-03-28T22:33:01.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.708+0000 7f53c8c00640 1 -- 172.21.15.43:0/2288313672 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f53b80046c0 con 0x7f53cc108da0 2024-03-28T22:33:01.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.709+0000 7f53c8c00640 1 --2- 172.21.15.43:0/2288313672 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f53b00a1e00 0x7f53b00a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:01.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.709+0000 7f53c8c00640 1 -- 172.21.15.43:0/2288313672 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f53b80cddd0 con 0x7f53cc108da0 2024-03-28T22:33:01.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.710+0000 7f53cbe00640 1 --2- 172.21.15.43:0/2288313672 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f53b00a1e00 0x7f53b00a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:01.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.710+0000 7f53cbe00640 1 --2- 172.21.15.43:0/2288313672 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f53b00a1e00 0x7f53b00a42c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f53bc02c630 tx=0x7f53bc009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:01.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.713+0000 7f53c8c00640 1 -- 172.21.15.43:0/2288313672 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f53b809b600 con 0x7f53cc108da0 2024-03-28T22:33:01.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.959+0000 7f53d1400640 1 -- 172.21.15.43:0/2288313672 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53cc02d170 con 0x7f53cc108da0 2024-03-28T22:33:01.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.960+0000 7f53c8c00640 1 -- 172.21.15.43:0/2288313672 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4696 (secure 0 0 0) 0x7f53b809af00 con 0x7f53cc108da0 2024-03-28T22:33:01.960 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:33:01.960 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6835/1084988072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1084988072},{"type":"v1","addr":"172.21.15.82:6835","nonce":1084988072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-28T22:33:01.960 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 6 2024-03-28T22:33:01.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.963+0000 7f53d1400640 1 -- 172.21.15.43:0/2288313672 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f53b00a1e00 msgr2=0x7f53b00a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:01.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.963+0000 7f53d1400640 1 --2- 172.21.15.43:0/2288313672 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f53b00a1e00 0x7f53b00a42c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f53bc02c630 tx=0x7f53bc009360 comp rx=0 tx=0).stop 2024-03-28T22:33:01.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.963+0000 7f53d1400640 1 -- 172.21.15.43:0/2288313672 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f53cc108da0 msgr2=0x7f53cc19e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:01.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.963+0000 7f53d1400640 1 --2- 172.21.15.43:0/2288313672 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f53cc108da0 0x7f53cc19e400 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f53b800e9f0 tx=0x7f53b800edf0 comp rx=0 tx=0).stop 2024-03-28T22:33:01.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.963+0000 7f53d1400640 1 -- 172.21.15.43:0/2288313672 shutdown_connections 2024-03-28T22:33:01.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.963+0000 7f53d1400640 1 --2- 172.21.15.43:0/2288313672 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f53b00a1e00 0x7f53b00a42c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:01.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.963+0000 7f53d1400640 1 --2- 172.21.15.43:0/2288313672 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f53cc108da0 0x7f53cc19e400 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:01.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.963+0000 7f53d1400640 1 --2- 172.21.15.43:0/2288313672 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53cc109760 0x7f53cc1a5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:01.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.963+0000 7f53d1400640 1 --2- 172.21.15.43:0/2288313672 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53cc107bb0 0x7f53cc19dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:01.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.963+0000 7f53d1400640 1 -- 172.21.15.43:0/2288313672 >> 172.21.15.43:0/2288313672 conn(0x7f53cc076b60 msgr2=0x7f53cc10c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:01.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.963+0000 7f53d1400640 1 -- 172.21.15.43:0/2288313672 shutdown_connections 2024-03-28T22:33:01.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:01.963+0000 7f53d1400640 1 -- 172.21.15.43:0/2288313672 wait complete. 2024-03-28T22:33:02.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:01 smithi082 ceph-mon[36471]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:02.230 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:01 smithi172 ceph-mon[36566]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:02.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:01 smithi043 ceph-mon[31823]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:02.896 INFO:journalctl@ceph.mds.f.smithi172.stdout:Mar 28 22:33:02 smithi172 systemd[1]: Started Ceph mds.f for c2276428-ed51-11ee-95d0-87774f69a715. 2024-03-28T22:33:03.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:02 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2288313672' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:03.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:02 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:03.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:02 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:03.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:02 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:03.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:02 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:33:03.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:02 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:33:03.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:02 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:33:03.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:02 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2288313672' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:03.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:02 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:03.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:02 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:03.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:02 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:03.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:02 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:33:03.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:02 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:33:03.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:02 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:33:03.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:02 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2288313672' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:03.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:02 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:03.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:02 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:03.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:02 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:03.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:02 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:33:03.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:02 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:33:03.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:02 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:33:03.983 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:33:04.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:03 smithi172 ceph-mon[36566]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:04.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:03 smithi172 ceph-mon[36566]: Deploying daemon mds.c on smithi172 2024-03-28T22:33:04.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:03 smithi172 ceph-mon[36566]: mds.? [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] up:boot 2024-03-28T22:33:04.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:03 smithi172 ceph-mon[36566]: fsmap 6 up:standby 2024-03-28T22:33:04.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:03 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T22:33:04.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.222+0000 7f2e09600640 1 -- 172.21.15.43:0/4286423775 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e04100ef0 msgr2=0x7f2e04101370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:04.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.222+0000 7f2e09600640 1 --2- 172.21.15.43:0/4286423775 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e04100ef0 0x7f2e04101370 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2dec009900 tx=0x7f2dec02bc30 comp rx=0 tx=0).stop 2024-03-28T22:33:04.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.222+0000 7f2e09600640 1 -- 172.21.15.43:0/4286423775 shutdown_connections 2024-03-28T22:33:04.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.222+0000 7f2e09600640 1 --2- 172.21.15.43:0/4286423775 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e04100ef0 0x7f2e04101370 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:04.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.222+0000 7f2e09600640 1 --2- 172.21.15.43:0/4286423775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2e04105bf0 0x7f2e04107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:04.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.222+0000 7f2e09600640 1 --2- 172.21.15.43:0/4286423775 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e04067bd0 0x7f2e041009b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:04.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.222+0000 7f2e09600640 1 -- 172.21.15.43:0/4286423775 >> 172.21.15.43:0/4286423775 conn(0x7f2e040fad30 msgr2=0x7f2e040fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:04.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.223+0000 7f2e09600640 1 -- 172.21.15.43:0/4286423775 shutdown_connections 2024-03-28T22:33:04.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.223+0000 7f2e09600640 1 -- 172.21.15.43:0/4286423775 wait complete. 2024-03-28T22:33:04.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.224+0000 7f2e09600640 1 Processor -- start 2024-03-28T22:33:04.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.224+0000 7f2e09600640 1 -- start start 2024-03-28T22:33:04.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.224+0000 7f2e09600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2e04067bd0 0x7f2e04199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:04.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.224+0000 7f2e09600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e04100ef0 0x7f2e0419a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:04.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.224+0000 7f2e09600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e04105bf0 0x7f2e041a1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:04.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.224+0000 7f2e09600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2e0410d250 con 0x7f2e04067bd0 2024-03-28T22:33:04.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.224+0000 7f2e09600640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f2e0410d0d0 con 0x7f2e04105bf0 2024-03-28T22:33:04.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.224+0000 7f2e09600640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f2e0410d3d0 con 0x7f2e04100ef0 2024-03-28T22:33:04.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.224+0000 7f2e03e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2e04067bd0 0x7f2e04199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:04.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.224+0000 7f2e08c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e04105bf0 0x7f2e041a1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:04.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.225+0000 7f2e03400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e04100ef0 0x7f2e0419a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:04.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.225+0000 7f2e03e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2e04067bd0 0x7f2e04199ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37486/0 (socket says 172.21.15.43:37486) 2024-03-28T22:33:04.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.225+0000 7f2e08c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e04105bf0 0x7f2e041a1090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:43832/0 (socket says 172.21.15.43:43832) 2024-03-28T22:33:04.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.225+0000 7f2e03e00640 1 -- 172.21.15.43:0/3417472790 learned_addr learned my addr 172.21.15.43:0/3417472790 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:04.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.225+0000 7f2e08c00640 1 -- 172.21.15.43:0/3417472790 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e04100ef0 msgr2=0x7f2e0419a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:04.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.225+0000 7f2e08c00640 1 --2- 172.21.15.43:0/3417472790 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e04100ef0 0x7f2e0419a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:04.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.225+0000 7f2e08c00640 1 -- 172.21.15.43:0/3417472790 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2e04067bd0 msgr2=0x7f2e04199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:04.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.225+0000 7f2e08c00640 1 --2- 172.21.15.43:0/3417472790 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2e04067bd0 0x7f2e04199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:04.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.225+0000 7f2e08c00640 1 -- 172.21.15.43:0/3417472790 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e041a1700 con 0x7f2e04105bf0 2024-03-28T22:33:04.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.225+0000 7f2e03e00640 1 --2- 172.21.15.43:0/3417472790 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2e04067bd0 0x7f2e04199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:33:04.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.225+0000 7f2e08c00640 1 --2- 172.21.15.43:0/3417472790 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e04105bf0 0x7f2e041a1090 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc00e970 tx=0x7f2dfc00ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:04.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.225+0000 7f2e00c00640 1 -- 172.21.15.43:0/3417472790 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc00ccc0 con 0x7f2e04105bf0 2024-03-28T22:33:04.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.226+0000 7f2e09600640 1 -- 172.21.15.43:0/3417472790 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e041a19f0 con 0x7f2e04105bf0 2024-03-28T22:33:04.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.226+0000 7f2e00c00640 1 -- 172.21.15.43:0/3417472790 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2dfc005020 con 0x7f2e04105bf0 2024-03-28T22:33:04.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.226+0000 7f2e00c00640 1 -- 172.21.15.43:0/3417472790 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc004a00 con 0x7f2e04105bf0 2024-03-28T22:33:04.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.226+0000 7f2e09600640 1 -- 172.21.15.43:0/3417472790 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e041a1f10 con 0x7f2e04105bf0 2024-03-28T22:33:04.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.227+0000 7f2e09600640 1 -- 172.21.15.43:0/3417472790 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e041a20e0 con 0x7f2e04105bf0 2024-03-28T22:33:04.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.229+0000 7f2e00c00640 1 -- 172.21.15.43:0/3417472790 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f2dfc0051d0 con 0x7f2e04105bf0 2024-03-28T22:33:04.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.231+0000 7f2e00c00640 1 --2- 172.21.15.43:0/3417472790 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f2de80a1d80 0x7f2de80a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:04.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.231+0000 7f2e00c00640 1 -- 172.21.15.43:0/3417472790 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f2dfc0cd910 con 0x7f2e04105bf0 2024-03-28T22:33:04.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.231+0000 7f2e03e00640 1 --2- 172.21.15.43:0/3417472790 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f2de80a1d80 0x7f2de80a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:04.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.232+0000 7f2e03e00640 1 --2- 172.21.15.43:0/3417472790 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f2de80a1d80 0x7f2de80a4240 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2df4005f40 tx=0x7f2df4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:04.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.235+0000 7f2e00c00640 1 -- 172.21.15.43:0/3417472790 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2dfc09b140 con 0x7f2e04105bf0 2024-03-28T22:33:04.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:03 smithi043 ceph-mon[31823]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:04.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:03 smithi043 ceph-mon[31823]: Deploying daemon mds.c on smithi172 2024-03-28T22:33:04.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:03 smithi043 ceph-mon[31823]: mds.? [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] up:boot 2024-03-28T22:33:04.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:03 smithi043 ceph-mon[31823]: fsmap 6 up:standby 2024-03-28T22:33:04.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:03 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T22:33:04.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:03 smithi082 ceph-mon[36471]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:04.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:03 smithi082 ceph-mon[36471]: Deploying daemon mds.c on smithi172 2024-03-28T22:33:04.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:03 smithi082 ceph-mon[36471]: mds.? [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] up:boot 2024-03-28T22:33:04.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:03 smithi082 ceph-mon[36471]: fsmap 6 up:standby 2024-03-28T22:33:04.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:03 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T22:33:04.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.479+0000 7f2e09600640 1 -- 172.21.15.43:0/3417472790 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e041a22b0 con 0x7f2e04105bf0 2024-03-28T22:33:04.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.480+0000 7f2e00c00640 1 -- 172.21.15.43:0/3417472790 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5521 (secure 0 0 0) 0x7f2dfc0ba030 con 0x7f2e04105bf0 2024-03-28T22:33:04.482 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:33:04.482 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6835/1084988072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1084988072},{"type":"v1","addr":"172.21.15.82:6835","nonce":1084988072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-03-28T22:33:04.483 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 7 2024-03-28T22:33:04.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.483+0000 7f2e09600640 1 -- 172.21.15.43:0/3417472790 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f2de80a1d80 msgr2=0x7f2de80a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:04.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.483+0000 7f2e09600640 1 --2- 172.21.15.43:0/3417472790 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f2de80a1d80 0x7f2de80a4240 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2df4005f40 tx=0x7f2df4009500 comp rx=0 tx=0).stop 2024-03-28T22:33:04.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.483+0000 7f2e09600640 1 -- 172.21.15.43:0/3417472790 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e04105bf0 msgr2=0x7f2e041a1090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:04.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.483+0000 7f2e09600640 1 --2- 172.21.15.43:0/3417472790 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e04105bf0 0x7f2e041a1090 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc00e970 tx=0x7f2dfc00ee30 comp rx=0 tx=0).stop 2024-03-28T22:33:04.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.483+0000 7f2e09600640 1 -- 172.21.15.43:0/3417472790 shutdown_connections 2024-03-28T22:33:04.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.483+0000 7f2e09600640 1 --2- 172.21.15.43:0/3417472790 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f2de80a1d80 0x7f2de80a4240 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:04.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.483+0000 7f2e09600640 1 --2- 172.21.15.43:0/3417472790 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e04105bf0 0x7f2e041a1090 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:04.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.483+0000 7f2e09600640 1 --2- 172.21.15.43:0/3417472790 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e04100ef0 0x7f2e0419a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:04.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.483+0000 7f2e09600640 1 --2- 172.21.15.43:0/3417472790 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2e04067bd0 0x7f2e04199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:04.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.483+0000 7f2e09600640 1 -- 172.21.15.43:0/3417472790 >> 172.21.15.43:0/3417472790 conn(0x7f2e040fad30 msgr2=0x7f2e040fb3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:04.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.483+0000 7f2e09600640 1 -- 172.21.15.43:0/3417472790 shutdown_connections 2024-03-28T22:33:04.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:04.483+0000 7f2e09600640 1 -- 172.21.15.43:0/3417472790 wait complete. 2024-03-28T22:33:05.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:04 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3417472790' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:05.381 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:04 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3417472790' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:05.381 INFO:journalctl@ceph.mds.c.smithi172.stdout:Mar 28 22:33:05 smithi172 systemd[1]: Starting Ceph mds.c for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:33:05.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:04 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3417472790' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:05.646 INFO:journalctl@ceph.mds.c.smithi172.stdout:Mar 28 22:33:05 smithi172 podman[66959]: 2024-03-28 22:33:05.286059168 +0000 UTC m=+0.018338211 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:33:05.647 INFO:journalctl@ceph.mds.c.smithi172.stdout:Mar 28 22:33:05 smithi172 podman[66959]: 2024-03-28 22:33:05.414016319 +0000 UTC m=+0.146295373 container create 46207b93f97a5e7f24527d83d878642ccbdf938ceaf64e5f0c8bd01594d48599 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, RELEASE=squid-323d23d, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-28T22:33:05.978 INFO:journalctl@ceph.mds.c.smithi172.stdout:Mar 28 22:33:05 smithi172 podman[66959]: 2024-03-28 22:33:05.675421982 +0000 UTC m=+0.407701019 container init 46207b93f97a5e7f24527d83d878642ccbdf938ceaf64e5f0c8bd01594d48599 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-c, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-323d23d, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0) 2024-03-28T22:33:05.978 INFO:journalctl@ceph.mds.c.smithi172.stdout:Mar 28 22:33:05 smithi172 podman[66959]: 2024-03-28 22:33:05.678431123 +0000 UTC m=+0.410710160 container start 46207b93f97a5e7f24527d83d878642ccbdf938ceaf64e5f0c8bd01594d48599 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-c, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-323d23d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-28T22:33:05.978 INFO:journalctl@ceph.mds.c.smithi172.stdout:Mar 28 22:33:05 smithi172 ceph-mds[66978]: -- 172.21.15.172:0/889550882 <== mon.1 v2:172.21.15.172:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562845a35a40 con 0x562845a08c00 2024-03-28T22:33:05.978 INFO:journalctl@ceph.mds.c.smithi172.stdout:Mar 28 22:33:05 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-c[66974]: starting mds.c at 2024-03-28T22:33:05.978 INFO:journalctl@ceph.mds.c.smithi172.stdout:Mar 28 22:33:05 smithi172 bash[66959]: 46207b93f97a5e7f24527d83d878642ccbdf938ceaf64e5f0c8bd01594d48599 2024-03-28T22:33:05.978 INFO:journalctl@ceph.mds.c.smithi172.stdout:Mar 28 22:33:05 smithi172 systemd[1]: Started Ceph mds.c for c2276428-ed51-11ee-95d0-87774f69a715. 2024-03-28T22:33:06.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:05 smithi043 ceph-mon[31823]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:06.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:05 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:06.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:05 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:06.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:05 smithi172 ceph-mon[36566]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:06.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:05 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:06.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:05 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:06.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:05 smithi082 ceph-mon[36471]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:06.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:05 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:06.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:05 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:06.503 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:33:06.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.735+0000 7f511e000640 1 -- 172.21.15.43:0/2625722754 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5118104990 msgr2=0x7f5118104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:06.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.735+0000 7f511e000640 1 --2- 172.21.15.43:0/2625722754 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5118104990 0x7f5118104e10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5104009900 tx=0x7f510402bc30 comp rx=0 tx=0).stop 2024-03-28T22:33:06.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.735+0000 7f511e000640 1 -- 172.21.15.43:0/2625722754 shutdown_connections 2024-03-28T22:33:06.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.735+0000 7f511e000640 1 --2- 172.21.15.43:0/2625722754 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5118105350 0x7f511810bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:06.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.735+0000 7f511e000640 1 --2- 172.21.15.43:0/2625722754 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5118104990 0x7f5118104e10 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:06.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.735+0000 7f511e000640 1 --2- 172.21.15.43:0/2625722754 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f51181037a0 0x7f5118103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:06.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.735+0000 7f511e000640 1 -- 172.21.15.43:0/2625722754 >> 172.21.15.43:0/2625722754 conn(0x7f51180fed30 msgr2=0x7f5118101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:06.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.735+0000 7f511e000640 1 -- 172.21.15.43:0/2625722754 shutdown_connections 2024-03-28T22:33:06.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.735+0000 7f511e000640 1 -- 172.21.15.43:0/2625722754 wait complete. 2024-03-28T22:33:06.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.736+0000 7f511e000640 1 Processor -- start 2024-03-28T22:33:06.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.736+0000 7f511e000640 1 -- start start 2024-03-28T22:33:06.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.736+0000 7f511e000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51181037a0 0x7f5118199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:06.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.736+0000 7f511e000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5118104990 0x7f5118199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:06.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.736+0000 7f511e000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5118105350 0x7f51181a1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:06.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.736+0000 7f511e000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f511810f470 con 0x7f5118104990 2024-03-28T22:33:06.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.736+0000 7f511e000640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f511810f2f0 con 0x7f5118105350 2024-03-28T22:33:06.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.736+0000 7f511e000640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f511810f5f0 con 0x7f51181037a0 2024-03-28T22:33:06.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.737+0000 7f511cc00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51181037a0 0x7f5118199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:06.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.737+0000 7f5117e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5118104990 0x7f5118199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:06.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.737+0000 7f511cc00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51181037a0 0x7f5118199a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:45196/0 (socket says 172.21.15.43:45196) 2024-03-28T22:33:06.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.737+0000 7f511cc00640 1 -- 172.21.15.43:0/4241165854 learned_addr learned my addr 172.21.15.43:0/4241165854 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:06.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.737+0000 7f511d600640 1 --2- 172.21.15.43:0/4241165854 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5118105350 0x7f51181a1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:06.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.737+0000 7f5117e00640 1 -- 172.21.15.43:0/4241165854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51181037a0 msgr2=0x7f5118199a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:06.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.737+0000 7f5117e00640 1 --2- 172.21.15.43:0/4241165854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51181037a0 0x7f5118199a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:06.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.737+0000 7f5117e00640 1 -- 172.21.15.43:0/4241165854 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5118105350 msgr2=0x7f51181a1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:06.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.737+0000 7f5117e00640 1 --2- 172.21.15.43:0/4241165854 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5118105350 0x7f51181a1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:06.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.737+0000 7f5117e00640 1 -- 172.21.15.43:0/4241165854 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51181a1680 con 0x7f5118104990 2024-03-28T22:33:06.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.737+0000 7f511d600640 1 --2- 172.21.15.43:0/4241165854 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5118105350 0x7f51181a1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:33:06.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.738+0000 7f5117e00640 1 --2- 172.21.15.43:0/4241165854 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5118104990 0x7f5118199fa0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5104036040 tx=0x7f510402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:06.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.738+0000 7f5115600640 1 -- 172.21.15.43:0/4241165854 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5104043020 con 0x7f5118104990 2024-03-28T22:33:06.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.738+0000 7f5115600640 1 -- 172.21.15.43:0/4241165854 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51040054e0 con 0x7f5118104990 2024-03-28T22:33:06.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.738+0000 7f511e000640 1 -- 172.21.15.43:0/4241165854 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51181a1910 con 0x7f5118104990 2024-03-28T22:33:06.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.738+0000 7f5115600640 1 -- 172.21.15.43:0/4241165854 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5104003c30 con 0x7f5118104990 2024-03-28T22:33:06.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.738+0000 7f511e000640 1 -- 172.21.15.43:0/4241165854 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51181a1e80 con 0x7f5118104990 2024-03-28T22:33:06.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.739+0000 7f511e000640 1 -- 172.21.15.43:0/4241165854 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f511810aa60 con 0x7f5118104990 2024-03-28T22:33:06.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.741+0000 7f5115600640 1 -- 172.21.15.43:0/4241165854 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f5104046020 con 0x7f5118104990 2024-03-28T22:33:06.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.743+0000 7f5115600640 1 --2- 172.21.15.43:0/4241165854 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f51000a1e00 0x7f51000a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:06.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.743+0000 7f5115600640 1 -- 172.21.15.43:0/4241165854 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f51040eff10 con 0x7f5118104990 2024-03-28T22:33:06.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.743+0000 7f511cc00640 1 --2- 172.21.15.43:0/4241165854 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f51000a1e00 0x7f51000a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:06.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.744+0000 7f511cc00640 1 --2- 172.21.15.43:0/4241165854 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f51000a1e00 0x7f51000a42c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f510c0058c0 tx=0x7f510c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:06.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.746+0000 7f5115600640 1 -- 172.21.15.43:0/4241165854 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f510404d020 con 0x7f5118104990 2024-03-28T22:33:06.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.992+0000 7f511e000640 1 -- 172.21.15.43:0/4241165854 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f511802d170 con 0x7f5118104990 2024-03-28T22:33:06.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.996+0000 7f5115600640 1 -- 172.21.15.43:0/4241165854 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6349 (secure 0 0 0) 0x7f51040bd740 con 0x7f5118104990 2024-03-28T22:33:06.996 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:33:06.996 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6835/1084988072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1084988072},{"type":"v1","addr":"172.21.15.82:6835","nonce":1084988072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-28T22:33:06.996 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 8 2024-03-28T22:33:06.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.999+0000 7f511e000640 1 -- 172.21.15.43:0/4241165854 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f51000a1e00 msgr2=0x7f51000a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:06.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.999+0000 7f511e000640 1 --2- 172.21.15.43:0/4241165854 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f51000a1e00 0x7f51000a42c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f510c0058c0 tx=0x7f510c005f90 comp rx=0 tx=0).stop 2024-03-28T22:33:06.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.999+0000 7f511e000640 1 -- 172.21.15.43:0/4241165854 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5118104990 msgr2=0x7f5118199fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:06.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.999+0000 7f511e000640 1 --2- 172.21.15.43:0/4241165854 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5118104990 0x7f5118199fa0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5104036040 tx=0x7f510402c650 comp rx=0 tx=0).stop 2024-03-28T22:33:06.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.999+0000 7f511e000640 1 -- 172.21.15.43:0/4241165854 shutdown_connections 2024-03-28T22:33:06.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.999+0000 7f511e000640 1 --2- 172.21.15.43:0/4241165854 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f51000a1e00 0x7f51000a42c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:06.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.999+0000 7f511e000640 1 --2- 172.21.15.43:0/4241165854 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5118105350 0x7f51181a1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:06.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.999+0000 7f511e000640 1 --2- 172.21.15.43:0/4241165854 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5118104990 0x7f5118199fa0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:06.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.999+0000 7f511e000640 1 --2- 172.21.15.43:0/4241165854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51181037a0 0x7f5118199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:06.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:06.999+0000 7f511e000640 1 -- 172.21.15.43:0/4241165854 >> 172.21.15.43:0/4241165854 conn(0x7f51180fed30 msgr2=0x7f5118107c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:06.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:07.000+0000 7f511e000640 1 -- 172.21.15.43:0/4241165854 shutdown_connections 2024-03-28T22:33:06.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:07.000+0000 7f511e000640 1 -- 172.21.15.43:0/4241165854 wait complete. 2024-03-28T22:33:07.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:06 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:07.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:06 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:33:07.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:06 smithi082 ceph-mon[36471]: mds.? [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] up:boot 2024-03-28T22:33:07.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:06 smithi082 ceph-mon[36471]: fsmap 7 up:standby 2024-03-28T22:33:07.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:06 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T22:33:07.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:06 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:33:07.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:06 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:33:07.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:06 smithi082 ceph-mon[36471]: Deploying daemon mds.h on smithi082 2024-03-28T22:33:07.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:06 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:07.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:06 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:33:07.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:06 smithi043 ceph-mon[31823]: mds.? [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] up:boot 2024-03-28T22:33:07.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:06 smithi043 ceph-mon[31823]: fsmap 7 up:standby 2024-03-28T22:33:07.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:06 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T22:33:07.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:06 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:33:07.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:06 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:33:07.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:06 smithi043 ceph-mon[31823]: Deploying daemon mds.h on smithi082 2024-03-28T22:33:07.297 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:06 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:07.297 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:06 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:33:07.297 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:06 smithi172 ceph-mon[36566]: mds.? [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] up:boot 2024-03-28T22:33:07.297 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:06 smithi172 ceph-mon[36566]: fsmap 7 up:standby 2024-03-28T22:33:07.297 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:06 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T22:33:07.297 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:06 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:33:07.297 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:06 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:33:07.297 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:06 smithi172 ceph-mon[36566]: Deploying daemon mds.h on smithi082 2024-03-28T22:33:08.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:07 smithi043 ceph-mon[31823]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:08.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:07 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/4241165854' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:08.315 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:07 smithi172 ceph-mon[36566]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:08.315 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:07 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/4241165854' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:08.388 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:07 smithi082 ceph-mon[36471]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:08.388 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:07 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/4241165854' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:08.388 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 28 22:33:08 smithi082 systemd[1]: Starting Ceph mds.h for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:33:08.658 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 28 22:33:08 smithi082 podman[66938]: 2024-03-28 22:33:08.389812947 +0000 UTC m=+0.105653966 container create 95924c8f41dd132a5a429bb5a8b27118b38ae6e1e8f63b9ede2df93600584e5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-323d23d, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True) 2024-03-28T22:33:08.658 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 28 22:33:08 smithi082 podman[66938]: 2024-03-28 22:33:08.302430994 +0000 UTC m=+0.018272019 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:33:08.658 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 28 22:33:08 smithi082 podman[66938]: 2024-03-28 22:33:08.659991511 +0000 UTC m=+0.375832544 container init 95924c8f41dd132a5a429bb5a8b27118b38ae6e1e8f63b9ede2df93600584e5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-h, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-323d23d, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240326) 2024-03-28T22:33:08.949 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 28 22:33:08 smithi082 podman[66938]: 2024-03-28 22:33:08.662606042 +0000 UTC m=+0.378447059 container start 95924c8f41dd132a5a429bb5a8b27118b38ae6e1e8f63b9ede2df93600584e5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-h, org.label-schema.license=GPLv2, RELEASE=squid-323d23d, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-28T22:33:08.950 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 28 22:33:08 smithi082 ceph-mds[66957]: -- 172.21.15.82:0/2337388593 <== mon.2 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d78851da40 con 0x55d7884f0c00 2024-03-28T22:33:08.950 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 28 22:33:08 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-h[66953]: starting mds.h at 2024-03-28T22:33:08.950 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 28 22:33:08 smithi082 bash[66938]: 95924c8f41dd132a5a429bb5a8b27118b38ae6e1e8f63b9ede2df93600584e5a 2024-03-28T22:33:08.950 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 28 22:33:08 smithi082 systemd[1]: Started Ceph mds.h for c2276428-ed51-11ee-95d0-87774f69a715. 2024-03-28T22:33:09.019 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:33:09.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.249+0000 7f3c53600640 1 -- 172.21.15.43:0/4139278357 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c4c102950 msgr2=0x7f3c4c102dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:09.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.249+0000 7f3c53600640 1 --2- 172.21.15.43:0/4139278357 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c4c102950 0x7f3c4c102dd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3c34009900 tx=0x7f3c3402bc30 comp rx=0 tx=0).stop 2024-03-28T22:33:09.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.250+0000 7f3c53600640 1 -- 172.21.15.43:0/4139278357 shutdown_connections 2024-03-28T22:33:09.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.250+0000 7f3c53600640 1 --2- 172.21.15.43:0/4139278357 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3c4c103310 0x7f3c4c109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:09.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.250+0000 7f3c53600640 1 --2- 172.21.15.43:0/4139278357 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c4c102950 0x7f3c4c102dd0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:09.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.250+0000 7f3c53600640 1 --2- 172.21.15.43:0/4139278357 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c4c101760 0x7f3c4c101b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:09.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.250+0000 7f3c53600640 1 -- 172.21.15.43:0/4139278357 >> 172.21.15.43:0/4139278357 conn(0x7f3c4c0fcd30 msgr2=0x7f3c4c0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:09.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.250+0000 7f3c53600640 1 -- 172.21.15.43:0/4139278357 shutdown_connections 2024-03-28T22:33:09.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.250+0000 7f3c53600640 1 -- 172.21.15.43:0/4139278357 wait complete. 2024-03-28T22:33:09.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.251+0000 7f3c53600640 1 Processor -- start 2024-03-28T22:33:09.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.251+0000 7f3c53600640 1 -- start start 2024-03-28T22:33:09.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.252+0000 7f3c53600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3c4c101760 0x7f3c4c197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:09.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.252+0000 7f3c53600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c4c102950 0x7f3c4c197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:09.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.252+0000 7f3c53600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c4c103310 0x7f3c4c19f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:09.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.252+0000 7f3c53600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3c4c10d410 con 0x7f3c4c103310 2024-03-28T22:33:09.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.252+0000 7f3c53600640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f3c4c10d290 con 0x7f3c4c101760 2024-03-28T22:33:09.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.252+0000 7f3c53600640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f3c4c10d590 con 0x7f3c4c102950 2024-03-28T22:33:09.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.252+0000 7f3c51600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c4c103310 0x7f3c4c19f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:09.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.252+0000 7f3c43e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c4c102950 0x7f3c4c197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:09.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.252+0000 7f3c43e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c4c102950 0x7f3c4c197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:45216/0 (socket says 172.21.15.43:45216) 2024-03-28T22:33:09.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.252+0000 7f3c43e00640 1 -- 172.21.15.43:0/2757914951 learned_addr learned my addr 172.21.15.43:0/2757914951 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:09.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.252+0000 7f3c50c00640 1 --2- 172.21.15.43:0/2757914951 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3c4c101760 0x7f3c4c197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:09.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.252+0000 7f3c43e00640 1 -- 172.21.15.43:0/2757914951 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3c4c101760 msgr2=0x7f3c4c197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:09.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.252+0000 7f3c43e00640 1 --2- 172.21.15.43:0/2757914951 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3c4c101760 0x7f3c4c197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:09.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.252+0000 7f3c43e00640 1 -- 172.21.15.43:0/2757914951 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c4c103310 msgr2=0x7f3c4c19f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:09.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.252+0000 7f3c43e00640 1 --2- 172.21.15.43:0/2757914951 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c4c103310 0x7f3c4c19f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:09.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.252+0000 7f3c43e00640 1 -- 172.21.15.43:0/2757914951 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c4c19f6a0 con 0x7f3c4c102950 2024-03-28T22:33:09.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.252+0000 7f3c50c00640 1 --2- 172.21.15.43:0/2757914951 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3c4c101760 0x7f3c4c197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:33:09.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.253+0000 7f3c43e00640 1 --2- 172.21.15.43:0/2757914951 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c4c102950 0x7f3c4c197fc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3c34005ab0 tx=0x7f3c3402c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:09.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.253+0000 7f3c4a000640 1 -- 172.21.15.43:0/2757914951 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c34043020 con 0x7f3c4c102950 2024-03-28T22:33:09.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.253+0000 7f3c4a000640 1 -- 172.21.15.43:0/2757914951 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c340054e0 con 0x7f3c4c102950 2024-03-28T22:33:09.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.253+0000 7f3c4a000640 1 -- 172.21.15.43:0/2757914951 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c34003ce0 con 0x7f3c4c102950 2024-03-28T22:33:09.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.253+0000 7f3c53600640 1 -- 172.21.15.43:0/2757914951 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c4c19f930 con 0x7f3c4c102950 2024-03-28T22:33:09.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.253+0000 7f3c53600640 1 -- 172.21.15.43:0/2757914951 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c4c19fe30 con 0x7f3c4c102950 2024-03-28T22:33:09.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.254+0000 7f3c53600640 1 -- 172.21.15.43:0/2757914951 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c4c108b00 con 0x7f3c4c102950 2024-03-28T22:33:09.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.256+0000 7f3c4a000640 1 -- 172.21.15.43:0/2757914951 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f3c34046020 con 0x7f3c4c102950 2024-03-28T22:33:09.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.257+0000 7f3c4a000640 1 --2- 172.21.15.43:0/2757914951 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3c300a1e00 0x7f3c300a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:09.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.258+0000 7f3c4a000640 1 -- 172.21.15.43:0/2757914951 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f3c340ef9a0 con 0x7f3c4c102950 2024-03-28T22:33:09.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.258+0000 7f3c50c00640 1 --2- 172.21.15.43:0/2757914951 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3c300a1e00 0x7f3c300a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:09.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.258+0000 7f3c50c00640 1 --2- 172.21.15.43:0/2757914951 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3c300a1e00 0x7f3c300a42c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c1027b0 tx=0x7f3c3c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:09.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.262+0000 7f3c4a000640 1 -- 172.21.15.43:0/2757914951 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3c3404d020 con 0x7f3c4c102950 2024-03-28T22:33:09.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.505+0000 7f3c53600640 1 -- 172.21.15.43:0/2757914951 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c4c02d170 con 0x7f3c4c102950 2024-03-28T22:33:09.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.506+0000 7f3c4a000640 1 -- 172.21.15.43:0/2757914951 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7171 (secure 0 0 0) 0x7f3c340bd1d0 con 0x7f3c4c102950 2024-03-28T22:33:09.505 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:33:09.505 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6835/1084988072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1084988072},{"type":"v1","addr":"172.21.15.82:6835","nonce":1084988072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24518,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-28T22:33:09.505 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 9 2024-03-28T22:33:09.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.509+0000 7f3c53600640 1 -- 172.21.15.43:0/2757914951 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3c300a1e00 msgr2=0x7f3c300a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:09.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.509+0000 7f3c53600640 1 --2- 172.21.15.43:0/2757914951 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3c300a1e00 0x7f3c300a42c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c1027b0 tx=0x7f3c3c005f90 comp rx=0 tx=0).stop 2024-03-28T22:33:09.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.509+0000 7f3c53600640 1 -- 172.21.15.43:0/2757914951 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c4c102950 msgr2=0x7f3c4c197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:09.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.509+0000 7f3c53600640 1 --2- 172.21.15.43:0/2757914951 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c4c102950 0x7f3c4c197fc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3c34005ab0 tx=0x7f3c3402c6c0 comp rx=0 tx=0).stop 2024-03-28T22:33:09.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.509+0000 7f3c53600640 1 -- 172.21.15.43:0/2757914951 shutdown_connections 2024-03-28T22:33:09.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.509+0000 7f3c53600640 1 --2- 172.21.15.43:0/2757914951 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3c4c103310 0x7f3c4c19f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:09.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.509+0000 7f3c53600640 1 --2- 172.21.15.43:0/2757914951 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c4c102950 0x7f3c4c197fc0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:09.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.509+0000 7f3c53600640 1 --2- 172.21.15.43:0/2757914951 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3c300a1e00 0x7f3c300a42c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:09.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.509+0000 7f3c53600640 1 --2- 172.21.15.43:0/2757914951 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3c4c101760 0x7f3c4c197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:09.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.509+0000 7f3c53600640 1 -- 172.21.15.43:0/2757914951 >> 172.21.15.43:0/2757914951 conn(0x7f3c4c0fcd30 msgr2=0x7f3c4c105cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:09.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.509+0000 7f3c53600640 1 -- 172.21.15.43:0/2757914951 shutdown_connections 2024-03-28T22:33:09.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:09.509+0000 7f3c53600640 1 -- 172.21.15.43:0/2757914951 wait complete. 2024-03-28T22:33:10.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:09 smithi172 ceph-mon[36566]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:10.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:09 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:10.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:09 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:10.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:09 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:10.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:09 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:33:10.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:09 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:33:10.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:09 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:33:10.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:09 smithi172 ceph-mon[36566]: Deploying daemon mds.i on smithi172 2024-03-28T22:33:10.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:09 smithi172 ceph-mon[36566]: mds.? [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] up:boot 2024-03-28T22:33:10.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:09 smithi172 ceph-mon[36566]: fsmap 8 up:standby 2024-03-28T22:33:10.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:09 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T22:33:10.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:09 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2757914951' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:10.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:09 smithi082 ceph-mon[36471]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:10.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:09 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:10.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:09 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:10.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:09 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:10.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:09 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:33:10.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:09 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:33:10.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:09 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:33:10.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:09 smithi082 ceph-mon[36471]: Deploying daemon mds.i on smithi172 2024-03-28T22:33:10.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:09 smithi082 ceph-mon[36471]: mds.? [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] up:boot 2024-03-28T22:33:10.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:09 smithi082 ceph-mon[36471]: fsmap 8 up:standby 2024-03-28T22:33:10.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:09 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T22:33:10.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:09 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2757914951' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:10.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:09 smithi043 ceph-mon[31823]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:10.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:09 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:10.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:09 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:10.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:09 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:10.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:09 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:33:10.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:09 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:33:10.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:09 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:33:10.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:09 smithi043 ceph-mon[31823]: Deploying daemon mds.i on smithi172 2024-03-28T22:33:10.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:09 smithi043 ceph-mon[31823]: mds.? [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] up:boot 2024-03-28T22:33:10.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:09 smithi043 ceph-mon[31823]: fsmap 8 up:standby 2024-03-28T22:33:10.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:09 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T22:33:10.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:09 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2757914951' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:11.393 INFO:journalctl@ceph.mds.i.smithi172.stdout:Mar 28 22:33:11 smithi172 systemd[1]: Starting Ceph mds.i for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:33:11.529 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:33:11.646 INFO:journalctl@ceph.mds.i.smithi172.stdout:Mar 28 22:33:11 smithi172 podman[67278]: 2024-03-28 22:33:11.297281487 +0000 UTC m=+0.017373999 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:33:11.646 INFO:journalctl@ceph.mds.i.smithi172.stdout:Mar 28 22:33:11 smithi172 podman[67278]: 2024-03-28 22:33:11.412238578 +0000 UTC m=+0.132331063 container create 59dc64b08c15d0d80dd2c407141af35c81d48dfb4acdba9bcf3c9c106a3a43e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-i, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-323d23d, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-28T22:33:11.646 INFO:journalctl@ceph.mds.i.smithi172.stdout:Mar 28 22:33:11 smithi172 podman[67278]: 2024-03-28 22:33:11.631454343 +0000 UTC m=+0.351546835 container init 59dc64b08c15d0d80dd2c407141af35c81d48dfb4acdba9bcf3c9c106a3a43e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-323d23d, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True) 2024-03-28T22:33:11.647 INFO:journalctl@ceph.mds.i.smithi172.stdout:Mar 28 22:33:11 smithi172 podman[67278]: 2024-03-28 22:33:11.634315953 +0000 UTC m=+0.354408437 container start 59dc64b08c15d0d80dd2c407141af35c81d48dfb4acdba9bcf3c9c106a3a43e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-i, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-323d23d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-28T22:33:11.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.767+0000 7f5fe7200640 1 -- 172.21.15.43:0/2721134349 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fe0104990 msgr2=0x7f5fe0104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:11.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.767+0000 7f5fe7200640 1 --2- 172.21.15.43:0/2721134349 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fe0104990 0x7f5fe0104e10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0009900 tx=0x7f5fd002bc30 comp rx=0 tx=0).stop 2024-03-28T22:33:11.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.767+0000 7f5fe7200640 1 -- 172.21.15.43:0/2721134349 shutdown_connections 2024-03-28T22:33:11.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.767+0000 7f5fe7200640 1 --2- 172.21.15.43:0/2721134349 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5fe0105350 0x7f5fe010bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:11.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.767+0000 7f5fe7200640 1 --2- 172.21.15.43:0/2721134349 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fe0104990 0x7f5fe0104e10 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:11.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.767+0000 7f5fe7200640 1 --2- 172.21.15.43:0/2721134349 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5fe01037a0 0x7f5fe0103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:11.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.767+0000 7f5fe7200640 1 -- 172.21.15.43:0/2721134349 >> 172.21.15.43:0/2721134349 conn(0x7f5fe00fed30 msgr2=0x7f5fe0101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:11.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.767+0000 7f5fe7200640 1 -- 172.21.15.43:0/2721134349 shutdown_connections 2024-03-28T22:33:11.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.767+0000 7f5fe7200640 1 -- 172.21.15.43:0/2721134349 wait complete. 2024-03-28T22:33:11.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.768+0000 7f5fe7200640 1 Processor -- start 2024-03-28T22:33:11.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.769+0000 7f5fe7200640 1 -- start start 2024-03-28T22:33:11.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.769+0000 7f5fe7200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fe01037a0 0x7f5fe0199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:11.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.769+0000 7f5fe7200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5fe0104990 0x7f5fe0199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:11.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.769+0000 7f5fe7200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5fe0105350 0x7f5fe01a0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:11.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.769+0000 7f5fe7200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe010f420 con 0x7f5fe0105350 2024-03-28T22:33:11.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.769+0000 7f5fe7200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe010f2a0 con 0x7f5fe0104990 2024-03-28T22:33:11.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.769+0000 7f5fe7200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe010f5a0 con 0x7f5fe01037a0 2024-03-28T22:33:11.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.769+0000 7f5fe5e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fe01037a0 0x7f5fe0199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:11.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.769+0000 7f5fe5e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fe01037a0 0x7f5fe0199a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:45238/0 (socket says 172.21.15.43:45238) 2024-03-28T22:33:11.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.769+0000 7f5fe5e00640 1 -- 172.21.15.43:0/478630594 learned_addr learned my addr 172.21.15.43:0/478630594 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:11.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.770+0000 7f5fe6800640 1 --2- 172.21.15.43:0/478630594 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5fe0105350 0x7f5fe01a0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:11.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.770+0000 7f5fe5400640 1 --2- 172.21.15.43:0/478630594 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5fe0104990 0x7f5fe0199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:11.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.770+0000 7f5fe5e00640 1 -- 172.21.15.43:0/478630594 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5fe0104990 msgr2=0x7f5fe0199f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:11.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.770+0000 7f5fe5e00640 1 --2- 172.21.15.43:0/478630594 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5fe0104990 0x7f5fe0199f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:11.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.770+0000 7f5fe5e00640 1 -- 172.21.15.43:0/478630594 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5fe0105350 msgr2=0x7f5fe01a0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:11.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.770+0000 7f5fe5e00640 1 --2- 172.21.15.43:0/478630594 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5fe0105350 0x7f5fe01a0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:11.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.770+0000 7f5fe5e00640 1 -- 172.21.15.43:0/478630594 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fe01a1660 con 0x7f5fe01037a0 2024-03-28T22:33:11.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.770+0000 7f5fe5e00640 1 --2- 172.21.15.43:0/478630594 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fe01037a0 0x7f5fe0199a40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5fd400e9c0 tx=0x7f5fd400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:11.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.770+0000 7f5fdaa00640 1 -- 172.21.15.43:0/478630594 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd400cde0 con 0x7f5fe01037a0 2024-03-28T22:33:11.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.770+0000 7f5fdaa00640 1 -- 172.21.15.43:0/478630594 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5fd4010910 con 0x7f5fe01037a0 2024-03-28T22:33:11.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.770+0000 7f5fdaa00640 1 -- 172.21.15.43:0/478630594 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd4018a00 con 0x7f5fe01037a0 2024-03-28T22:33:11.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.771+0000 7f5fe7200640 1 -- 172.21.15.43:0/478630594 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fe01a1950 con 0x7f5fe01037a0 2024-03-28T22:33:11.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.771+0000 7f5fe7200640 1 -- 172.21.15.43:0/478630594 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fe01a1ec0 con 0x7f5fe01037a0 2024-03-28T22:33:11.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.772+0000 7f5fe7200640 1 -- 172.21.15.43:0/478630594 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fe010aac0 con 0x7f5fe01037a0 2024-03-28T22:33:11.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.774+0000 7f5fdaa00640 1 -- 172.21.15.43:0/478630594 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f5fd401f020 con 0x7f5fe01037a0 2024-03-28T22:33:11.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.775+0000 7f5fdaa00640 1 --2- 172.21.15.43:0/478630594 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5fc80a1e00 0x7f5fc80a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:11.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.775+0000 7f5fdaa00640 1 -- 172.21.15.43:0/478630594 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f5fd40cdb00 con 0x7f5fe01037a0 2024-03-28T22:33:11.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.775+0000 7f5fe5400640 1 --2- 172.21.15.43:0/478630594 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5fc80a1e00 0x7f5fc80a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:11.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.776+0000 7f5fe5400640 1 --2- 172.21.15.43:0/478630594 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5fc80a1e00 0x7f5fc80a42c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5fd002c7b0 tx=0x7f5fd0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:11.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:11.783+0000 7f5fdaa00640 1 -- 172.21.15.43:0/478630594 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5fd409b330 con 0x7f5fe01037a0 2024-03-28T22:33:12.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:11 smithi043 ceph-mon[31823]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:12.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:11 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:12.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:12.036+0000 7f5fe7200640 1 -- 172.21.15.43:0/478630594 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5fe002d170 con 0x7f5fe01037a0 2024-03-28T22:33:12.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:12.036+0000 7f5fdaa00640 1 -- 172.21.15.43:0/478630594 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8001 (secure 0 0 0) 0x7f5fd409ac30 con 0x7f5fe01037a0 2024-03-28T22:33:12.035 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:33:12.035 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6835/1084988072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1084988072},{"type":"v1","addr":"172.21.15.82:6835","nonce":1084988072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24518,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24556,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-28T22:33:12.035 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 10 2024-03-28T22:33:12.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:12.040+0000 7f5fe7200640 1 -- 172.21.15.43:0/478630594 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5fc80a1e00 msgr2=0x7f5fc80a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:12.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:12.040+0000 7f5fe7200640 1 --2- 172.21.15.43:0/478630594 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5fc80a1e00 0x7f5fc80a42c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5fd002c7b0 tx=0x7f5fd0005fd0 comp rx=0 tx=0).stop 2024-03-28T22:33:12.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:12.040+0000 7f5fe7200640 1 -- 172.21.15.43:0/478630594 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fe01037a0 msgr2=0x7f5fe0199a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:12.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:12.040+0000 7f5fe7200640 1 --2- 172.21.15.43:0/478630594 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fe01037a0 0x7f5fe0199a40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5fd400e9c0 tx=0x7f5fd400ee80 comp rx=0 tx=0).stop 2024-03-28T22:33:12.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:12.040+0000 7f5fe7200640 1 -- 172.21.15.43:0/478630594 shutdown_connections 2024-03-28T22:33:12.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:12.040+0000 7f5fe7200640 1 --2- 172.21.15.43:0/478630594 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5fc80a1e00 0x7f5fc80a42c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:12.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:12.040+0000 7f5fe7200640 1 --2- 172.21.15.43:0/478630594 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5fe0104990 0x7f5fe0199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:12.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:12.040+0000 7f5fe7200640 1 --2- 172.21.15.43:0/478630594 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5fe0105350 0x7f5fe01a0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:12.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:12.040+0000 7f5fe7200640 1 --2- 172.21.15.43:0/478630594 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fe01037a0 0x7f5fe0199a40 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:12.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:12.040+0000 7f5fe7200640 1 -- 172.21.15.43:0/478630594 >> 172.21.15.43:0/478630594 conn(0x7f5fe00fed30 msgr2=0x7f5fe0107c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:12.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:12.040+0000 7f5fe7200640 1 -- 172.21.15.43:0/478630594 shutdown_connections 2024-03-28T22:33:12.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:12.040+0000 7f5fe7200640 1 -- 172.21.15.43:0/478630594 wait complete. 2024-03-28T22:33:12.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:11 smithi172 ceph-mon[36566]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:12.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:11 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:12.147 INFO:journalctl@ceph.mds.i.smithi172.stdout:Mar 28 22:33:11 smithi172 ceph-mds[67297]: -- 172.21.15.172:0/485687771 <== mon.1 v2:172.21.15.172:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b6fd9dfa40 con 0x55b6fd9b3000 2024-03-28T22:33:12.147 INFO:journalctl@ceph.mds.i.smithi172.stdout:Mar 28 22:33:11 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-i[67293]: starting mds.i at 2024-03-28T22:33:12.147 INFO:journalctl@ceph.mds.i.smithi172.stdout:Mar 28 22:33:11 smithi172 bash[67278]: 59dc64b08c15d0d80dd2c407141af35c81d48dfb4acdba9bcf3c9c106a3a43e9 2024-03-28T22:33:12.147 INFO:journalctl@ceph.mds.i.smithi172.stdout:Mar 28 22:33:11 smithi172 systemd[1]: Started Ceph mds.i for c2276428-ed51-11ee-95d0-87774f69a715. 2024-03-28T22:33:12.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:11 smithi082 ceph-mon[36471]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:12.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:11 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:13.094 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:12 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:13.094 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:12 smithi043 ceph-mon[31823]: mds.? [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] up:boot 2024-03-28T22:33:13.094 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:12 smithi043 ceph-mon[31823]: fsmap 9 up:standby 2024-03-28T22:33:13.094 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:12 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T22:33:13.094 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:12 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:13.094 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:12 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:33:13.094 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:12 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:33:13.094 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:12 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:33:13.094 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:12 smithi043 ceph-mon[31823]: Deploying daemon mds.g on smithi043 2024-03-28T22:33:13.094 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:12 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/478630594' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:13.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:12 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:13.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:12 smithi172 ceph-mon[36566]: mds.? [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] up:boot 2024-03-28T22:33:13.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:12 smithi172 ceph-mon[36566]: fsmap 9 up:standby 2024-03-28T22:33:13.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:12 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T22:33:13.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:12 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:13.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:12 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:33:13.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:12 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:33:13.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:12 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:33:13.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:12 smithi172 ceph-mon[36566]: Deploying daemon mds.g on smithi043 2024-03-28T22:33:13.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:12 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/478630594' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:13.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:12 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:13.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:12 smithi082 ceph-mon[36471]: mds.? [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] up:boot 2024-03-28T22:33:13.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:12 smithi082 ceph-mon[36471]: fsmap 9 up:standby 2024-03-28T22:33:13.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:12 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T22:33:13.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:12 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:13.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:12 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:33:13.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:12 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:33:13.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:12 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:33:13.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:12 smithi082 ceph-mon[36471]: Deploying daemon mds.g on smithi043 2024-03-28T22:33:13.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:12 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/478630594' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:14.058 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:33:14.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:13 smithi172 ceph-mon[36566]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:14.157 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:13 smithi043 ceph-mon[31823]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:14.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:13 smithi082 ceph-mon[36471]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:14.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.301+0000 7f10b4600640 1 -- 172.21.15.43:0/1915536441 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10ac077cf0 msgr2=0x7f10ac078590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:14.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.301+0000 7f10b4600640 1 --2- 172.21.15.43:0/1915536441 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10ac077cf0 0x7f10ac078590 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f10a4009ca0 tx=0x7f10a402c100 comp rx=0 tx=0).stop 2024-03-28T22:33:14.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.301+0000 7f10b4600640 1 -- 172.21.15.43:0/1915536441 shutdown_connections 2024-03-28T22:33:14.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.301+0000 7f10b4600640 1 --2- 172.21.15.43:0/1915536441 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10ac077cf0 0x7f10ac078590 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:14.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.301+0000 7f10b4600640 1 --2- 172.21.15.43:0/1915536441 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10ac077350 0x7f10ac0777b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:14.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.301+0000 7f10b4600640 1 --2- 172.21.15.43:0/1915536441 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f10ac076110 0x7f10ac076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:14.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.301+0000 7f10b4600640 1 -- 172.21.15.43:0/1915536441 >> 172.21.15.43:0/1915536441 conn(0x7f10ac0fef00 msgr2=0x7f10ac101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:14.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.301+0000 7f10b4600640 1 -- 172.21.15.43:0/1915536441 shutdown_connections 2024-03-28T22:33:14.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.301+0000 7f10b4600640 1 -- 172.21.15.43:0/1915536441 wait complete. 2024-03-28T22:33:14.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.302+0000 7f10b4600640 1 Processor -- start 2024-03-28T22:33:14.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.303+0000 7f10b4600640 1 -- start start 2024-03-28T22:33:14.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.303+0000 7f10b4600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10ac076110 0x7f10ac19e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:14.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.303+0000 7f10b4600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f10ac077350 0x7f10ac19e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:14.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.303+0000 7f10b4600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10ac077cf0 0x7f10ac1a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:14.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.303+0000 7f10b4600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac113a20 con 0x7f10ac076110 2024-03-28T22:33:14.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.303+0000 7f10b4600640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac1138a0 con 0x7f10ac077350 2024-03-28T22:33:14.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.303+0000 7f10b4600640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac113ba0 con 0x7f10ac077cf0 2024-03-28T22:33:14.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.303+0000 7f10b1c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10ac076110 0x7f10ac19e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:14.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.303+0000 7f10b2600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10ac077cf0 0x7f10ac1a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:14.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.303+0000 7f10b1c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10ac076110 0x7f10ac19e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42878/0 (socket says 172.21.15.43:42878) 2024-03-28T22:33:14.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.303+0000 7f10b1c00640 1 -- 172.21.15.43:0/2982629304 learned_addr learned my addr 172.21.15.43:0/2982629304 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:14.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.303+0000 7f10b1200640 1 --2- 172.21.15.43:0/2982629304 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f10ac077350 0x7f10ac19e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:14.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.304+0000 7f10b2600640 1 -- 172.21.15.43:0/2982629304 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f10ac077350 msgr2=0x7f10ac19e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:14.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.304+0000 7f10b2600640 1 --2- 172.21.15.43:0/2982629304 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f10ac077350 0x7f10ac19e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:14.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.304+0000 7f10b2600640 1 -- 172.21.15.43:0/2982629304 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10ac076110 msgr2=0x7f10ac19e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:14.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.304+0000 7f10b2600640 1 --2- 172.21.15.43:0/2982629304 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10ac076110 0x7f10ac19e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:14.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.304+0000 7f10b2600640 1 -- 172.21.15.43:0/2982629304 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10ac1a5c60 con 0x7f10ac077cf0 2024-03-28T22:33:14.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.304+0000 7f10b2600640 1 --2- 172.21.15.43:0/2982629304 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10ac077cf0 0x7f10ac1a55f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f10a4037040 tx=0x7f10a4035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:14.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.305+0000 7f10aa000640 1 -- 172.21.15.43:0/2982629304 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a4039040 con 0x7f10ac077cf0 2024-03-28T22:33:14.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.305+0000 7f10aa000640 1 -- 172.21.15.43:0/2982629304 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10a4044070 con 0x7f10ac077cf0 2024-03-28T22:33:14.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.305+0000 7f10b4600640 1 -- 172.21.15.43:0/2982629304 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10ac1a5ef0 con 0x7f10ac077cf0 2024-03-28T22:33:14.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.305+0000 7f10aa000640 1 -- 172.21.15.43:0/2982629304 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a4004450 con 0x7f10ac077cf0 2024-03-28T22:33:14.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.305+0000 7f10b4600640 1 -- 172.21.15.43:0/2982629304 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10ac1a63b0 con 0x7f10ac077cf0 2024-03-28T22:33:14.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.307+0000 7f10b4600640 1 -- 172.21.15.43:0/2982629304 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10ac110490 con 0x7f10ac077cf0 2024-03-28T22:33:14.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.310+0000 7f10aa000640 1 -- 172.21.15.43:0/2982629304 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f10a40045f0 con 0x7f10ac077cf0 2024-03-28T22:33:14.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.311+0000 7f10aa000640 1 --2- 172.21.15.43:0/2982629304 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f10940a1e00 0x7f10940a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:14.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.312+0000 7f10aa000640 1 -- 172.21.15.43:0/2982629304 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f10a40efed0 con 0x7f10ac077cf0 2024-03-28T22:33:14.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.312+0000 7f10b1c00640 1 --2- 172.21.15.43:0/2982629304 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f10940a1e00 0x7f10940a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:14.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.312+0000 7f10b1c00640 1 --2- 172.21.15.43:0/2982629304 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f10940a1e00 0x7f10940a42c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f10a0009cd0 tx=0x7f10a0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:14.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.316+0000 7f10aa000640 1 -- 172.21.15.43:0/2982629304 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f10a40bd700 con 0x7f10ac077cf0 2024-03-28T22:33:14.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.597+0000 7f10b4600640 1 -- 172.21.15.43:0/2982629304 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f10ac062350 con 0x7f10ac077cf0 2024-03-28T22:33:14.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.598+0000 7f10aa000640 1 -- 172.21.15.43:0/2982629304 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8001 (secure 0 0 0) 0x7f10a40098b0 con 0x7f10ac077cf0 2024-03-28T22:33:14.597 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:33:14.597 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6835/1084988072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1084988072},{"type":"v1","addr":"172.21.15.82:6835","nonce":1084988072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24518,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24556,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-28T22:33:14.597 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 10 2024-03-28T22:33:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.602+0000 7f10b4600640 1 -- 172.21.15.43:0/2982629304 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f10940a1e00 msgr2=0x7f10940a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.602+0000 7f10b4600640 1 --2- 172.21.15.43:0/2982629304 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f10940a1e00 0x7f10940a42c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f10a0009cd0 tx=0x7f10a0005f90 comp rx=0 tx=0).stop 2024-03-28T22:33:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.602+0000 7f10b4600640 1 -- 172.21.15.43:0/2982629304 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10ac077cf0 msgr2=0x7f10ac1a55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.602+0000 7f10b4600640 1 --2- 172.21.15.43:0/2982629304 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10ac077cf0 0x7f10ac1a55f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f10a4037040 tx=0x7f10a4035d10 comp rx=0 tx=0).stop 2024-03-28T22:33:14.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.602+0000 7f10b4600640 1 -- 172.21.15.43:0/2982629304 shutdown_connections 2024-03-28T22:33:14.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.602+0000 7f10b4600640 1 --2- 172.21.15.43:0/2982629304 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f10940a1e00 0x7f10940a42c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:14.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.602+0000 7f10b4600640 1 --2- 172.21.15.43:0/2982629304 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f10ac077350 0x7f10ac19e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:14.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.602+0000 7f10b4600640 1 --2- 172.21.15.43:0/2982629304 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10ac077cf0 0x7f10ac1a55f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:14.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.602+0000 7f10b4600640 1 --2- 172.21.15.43:0/2982629304 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10ac076110 0x7f10ac19e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:14.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.602+0000 7f10b4600640 1 -- 172.21.15.43:0/2982629304 >> 172.21.15.43:0/2982629304 conn(0x7f10ac0fef00 msgr2=0x7f10ac1005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:14.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.602+0000 7f10b4600640 1 -- 172.21.15.43:0/2982629304 shutdown_connections 2024-03-28T22:33:14.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:14.602+0000 7f10b4600640 1 -- 172.21.15.43:0/2982629304 wait complete. 2024-03-28T22:33:15.109 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:14 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2982629304' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:15.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:14 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2982629304' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:15.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:14 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2982629304' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:15.481 INFO:journalctl@ceph.mds.g.smithi043.stdout:Mar 28 22:33:15 smithi043 systemd[1]: Starting Ceph mds.g for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:33:15.767 INFO:journalctl@ceph.mds.g.smithi043.stdout:Mar 28 22:33:15 smithi043 podman[70841]: 2024-03-28 22:33:15.38774435 +0000 UTC m=+0.034784673 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:33:15.767 INFO:journalctl@ceph.mds.g.smithi043.stdout:Mar 28 22:33:15 smithi043 podman[70841]: 2024-03-28 22:33:15.524623054 +0000 UTC m=+0.171663429 container create 66fac975734c4b1da9d2f05246aaeab96c7c787f10b195fd6460f113cfa845bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-g, org.label-schema.license=GPLv2, RELEASE=squid-323d23d, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=) 2024-03-28T22:33:16.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:15 smithi082 ceph-mon[36471]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:16.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:15 smithi043 ceph-mon[31823]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:16.266 INFO:journalctl@ceph.mds.g.smithi043.stdout:Mar 28 22:33:15 smithi043 podman[70841]: 2024-03-28 22:33:15.797059337 +0000 UTC m=+0.444099691 container init 66fac975734c4b1da9d2f05246aaeab96c7c787f10b195fd6460f113cfa845bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-g, org.label-schema.build-date=20240326, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-323d23d, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T22:33:16.266 INFO:journalctl@ceph.mds.g.smithi043.stdout:Mar 28 22:33:15 smithi043 podman[70841]: 2024-03-28 22:33:15.806168577 +0000 UTC m=+0.453208888 container start 66fac975734c4b1da9d2f05246aaeab96c7c787f10b195fd6460f113cfa845bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, ceph=True, org.label-schema.license=GPLv2) 2024-03-28T22:33:16.266 INFO:journalctl@ceph.mds.g.smithi043.stdout:Mar 28 22:33:15 smithi043 ceph-mds[70866]: -- 172.21.15.43:0/3728271522 <== mon.0 v2:172.21.15.43:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564404453a40 con 0x564404426c00 2024-03-28T22:33:16.266 INFO:journalctl@ceph.mds.g.smithi043.stdout:Mar 28 22:33:15 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-g[70858]: starting mds.g at 2024-03-28T22:33:16.266 INFO:journalctl@ceph.mds.g.smithi043.stdout:Mar 28 22:33:15 smithi043 bash[70841]: 66fac975734c4b1da9d2f05246aaeab96c7c787f10b195fd6460f113cfa845bf 2024-03-28T22:33:16.267 INFO:journalctl@ceph.mds.g.smithi043.stdout:Mar 28 22:33:15 smithi043 systemd[1]: Started Ceph mds.g for c2276428-ed51-11ee-95d0-87774f69a715. 2024-03-28T22:33:16.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:15 smithi172 ceph-mon[36566]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:16.629 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:33:16.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.836+0000 7f93efe00640 1 -- 172.21.15.43:0/1123889076 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93e8105540 msgr2=0x7f93e810bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:16.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.836+0000 7f93efe00640 1 --2- 172.21.15.43:0/1123889076 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93e8105540 0x7f93e810bdd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f93e0009ca0 tx=0x7f93e002c100 comp rx=0 tx=0).stop 2024-03-28T22:33:16.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.836+0000 7f93efe00640 1 -- 172.21.15.43:0/1123889076 shutdown_connections 2024-03-28T22:33:16.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.836+0000 7f93efe00640 1 --2- 172.21.15.43:0/1123889076 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f93e8104b80 0x7f93e8105000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:16.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.836+0000 7f93efe00640 1 --2- 172.21.15.43:0/1123889076 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93e8105540 0x7f93e810bdd0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:16.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.836+0000 7f93efe00640 1 --2- 172.21.15.43:0/1123889076 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93e8103990 0x7f93e8103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:16.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.836+0000 7f93efe00640 1 -- 172.21.15.43:0/1123889076 >> 172.21.15.43:0/1123889076 conn(0x7f93e80fef00 msgr2=0x7f93e8101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:16.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.836+0000 7f93efe00640 1 -- 172.21.15.43:0/1123889076 shutdown_connections 2024-03-28T22:33:16.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.836+0000 7f93efe00640 1 -- 172.21.15.43:0/1123889076 wait complete. 2024-03-28T22:33:16.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.837+0000 7f93efe00640 1 Processor -- start 2024-03-28T22:33:16.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.837+0000 7f93efe00640 1 -- start start 2024-03-28T22:33:16.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.837+0000 7f93efe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f93e8103990 0x7f93e8199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:16.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.837+0000 7f93efe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93e8104b80 0x7f93e819a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:16.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.837+0000 7f93efe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93e8105540 0x7f93e81a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:16.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.837+0000 7f93efe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f93e810f5f0 con 0x7f93e8105540 2024-03-28T22:33:16.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.837+0000 7f93efe00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f93e810f470 con 0x7f93e8103990 2024-03-28T22:33:16.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.837+0000 7f93efe00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f93e810f770 con 0x7f93e8104b80 2024-03-28T22:33:16.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.838+0000 7f93ed400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f93e8103990 0x7f93e8199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:16.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.838+0000 7f93ed400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f93e8103990 0x7f93e8199c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59850/0 (socket says 172.21.15.43:59850) 2024-03-28T22:33:16.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.838+0000 7f93ed400640 1 -- 172.21.15.43:0/103891730 learned_addr learned my addr 172.21.15.43:0/103891730 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:16.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.838+0000 7f93eca00640 1 --2- 172.21.15.43:0/103891730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93e8104b80 0x7f93e819a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:16.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.838+0000 7f93ede00640 1 --2- 172.21.15.43:0/103891730 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93e8105540 0x7f93e81a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:16.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.838+0000 7f93ed400640 1 -- 172.21.15.43:0/103891730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93e8104b80 msgr2=0x7f93e819a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:16.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.838+0000 7f93ed400640 1 --2- 172.21.15.43:0/103891730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93e8104b80 0x7f93e819a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:16.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.838+0000 7f93ed400640 1 -- 172.21.15.43:0/103891730 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93e8105540 msgr2=0x7f93e81a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:16.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.838+0000 7f93ed400640 1 --2- 172.21.15.43:0/103891730 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93e8105540 0x7f93e81a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:16.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.838+0000 7f93ed400640 1 -- 172.21.15.43:0/103891730 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93e81a1860 con 0x7f93e8103990 2024-03-28T22:33:16.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.839+0000 7f93ed400640 1 --2- 172.21.15.43:0/103891730 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f93e8103990 0x7f93e8199c50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f93dc00e9c0 tx=0x7f93dc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:16.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.839+0000 7f93e6000640 1 -- 172.21.15.43:0/103891730 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93dc00cd80 con 0x7f93e8103990 2024-03-28T22:33:16.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.839+0000 7f93efe00640 1 -- 172.21.15.43:0/103891730 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93e81a1b50 con 0x7f93e8103990 2024-03-28T22:33:16.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.839+0000 7f93e6000640 1 -- 172.21.15.43:0/103891730 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93dc0035f0 con 0x7f93e8103990 2024-03-28T22:33:16.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.839+0000 7f93e6000640 1 -- 172.21.15.43:0/103891730 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93dc010690 con 0x7f93e8103990 2024-03-28T22:33:16.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.839+0000 7f93efe00640 1 -- 172.21.15.43:0/103891730 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93e81a20a0 con 0x7f93e8103990 2024-03-28T22:33:16.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.840+0000 7f93efe00640 1 -- 172.21.15.43:0/103891730 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93e810acb0 con 0x7f93e8103990 2024-03-28T22:33:16.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.842+0000 7f93e6000640 1 -- 172.21.15.43:0/103891730 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f93dc0037a0 con 0x7f93e8103990 2024-03-28T22:33:16.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.843+0000 7f93e6000640 1 --2- 172.21.15.43:0/103891730 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f93d00a1d80 0x7f93d00a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:16.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.844+0000 7f93e6000640 1 -- 172.21.15.43:0/103891730 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f93dc0cddc0 con 0x7f93e8103990 2024-03-28T22:33:16.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.844+0000 7f93eca00640 1 --2- 172.21.15.43:0/103891730 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f93d00a1d80 0x7f93d00a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:16.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.844+0000 7f93eca00640 1 --2- 172.21.15.43:0/103891730 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f93d00a1d80 0x7f93d00a4240 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f93d4009bd0 tx=0x7f93d4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:16.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:16.846+0000 7f93e6000640 1 -- 172.21.15.43:0/103891730 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f93dc09b5f0 con 0x7f93e8103990 2024-03-28T22:33:17.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:16 smithi043 ceph-mon[31823]: mds.? [v2:172.21.15.43:6838/2742974591,v1:172.21.15.43:6839/2742974591] up:boot 2024-03-28T22:33:17.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:16 smithi043 ceph-mon[31823]: fsmap 10 up:standby 2024-03-28T22:33:17.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:16 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T22:33:17.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:16 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:17.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:16 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:17.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:16 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:17.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:16 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:33:17.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:16 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:33:17.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:16 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:33:17.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:16 smithi043 ceph-mon[31823]: Deploying daemon mds.a on smithi043 2024-03-28T22:33:17.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:17.087+0000 7f93efe00640 1 -- 172.21.15.43:0/103891730 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f93e802d420 con 0x7f93e8103990 2024-03-28T22:33:17.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:17.088+0000 7f93e6000640 1 -- 172.21.15.43:0/103891730 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8827 (secure 0 0 0) 0x7f93dc09aef0 con 0x7f93e8103990 2024-03-28T22:33:17.086 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:33:17.087 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6835/1084988072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1084988072},{"type":"v1","addr":"172.21.15.82:6835","nonce":1084988072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24518,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24556,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-28T22:33:17.087 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 11 2024-03-28T22:33:17.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:17.091+0000 7f93efe00640 1 -- 172.21.15.43:0/103891730 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f93d00a1d80 msgr2=0x7f93d00a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:17.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:17.091+0000 7f93efe00640 1 --2- 172.21.15.43:0/103891730 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f93d00a1d80 0x7f93d00a4240 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f93d4009bd0 tx=0x7f93d4005f90 comp rx=0 tx=0).stop 2024-03-28T22:33:17.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:17.091+0000 7f93efe00640 1 -- 172.21.15.43:0/103891730 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f93e8103990 msgr2=0x7f93e8199c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:17.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:17.091+0000 7f93efe00640 1 --2- 172.21.15.43:0/103891730 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f93e8103990 0x7f93e8199c50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f93dc00e9c0 tx=0x7f93dc00ee80 comp rx=0 tx=0).stop 2024-03-28T22:33:17.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:17.091+0000 7f93efe00640 1 -- 172.21.15.43:0/103891730 shutdown_connections 2024-03-28T22:33:17.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:17.091+0000 7f93efe00640 1 --2- 172.21.15.43:0/103891730 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93e8105540 0x7f93e81a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:17.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:17.091+0000 7f93efe00640 1 --2- 172.21.15.43:0/103891730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93e8104b80 0x7f93e819a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:17.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:17.091+0000 7f93efe00640 1 --2- 172.21.15.43:0/103891730 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f93d00a1d80 0x7f93d00a4240 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:17.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:17.091+0000 7f93efe00640 1 --2- 172.21.15.43:0/103891730 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f93e8103990 0x7f93e8199c50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:17.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:17.091+0000 7f93efe00640 1 -- 172.21.15.43:0/103891730 >> 172.21.15.43:0/103891730 conn(0x7f93e80fef00 msgr2=0x7f93e8107e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:17.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:17.091+0000 7f93efe00640 1 -- 172.21.15.43:0/103891730 shutdown_connections 2024-03-28T22:33:17.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:17.091+0000 7f93efe00640 1 -- 172.21.15.43:0/103891730 wait complete. 2024-03-28T22:33:17.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:16 smithi082 ceph-mon[36471]: mds.? [v2:172.21.15.43:6838/2742974591,v1:172.21.15.43:6839/2742974591] up:boot 2024-03-28T22:33:17.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:16 smithi082 ceph-mon[36471]: fsmap 10 up:standby 2024-03-28T22:33:17.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:16 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T22:33:17.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:16 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:17.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:16 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:17.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:16 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:17.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:16 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:33:17.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:16 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:33:17.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:16 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:33:17.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:16 smithi082 ceph-mon[36471]: Deploying daemon mds.a on smithi043 2024-03-28T22:33:17.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:16 smithi172 ceph-mon[36566]: mds.? [v2:172.21.15.43:6838/2742974591,v1:172.21.15.43:6839/2742974591] up:boot 2024-03-28T22:33:17.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:16 smithi172 ceph-mon[36566]: fsmap 10 up:standby 2024-03-28T22:33:17.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:16 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T22:33:17.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:16 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:17.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:16 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:17.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:16 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:17.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:16 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:33:17.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:16 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:33:17.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:16 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:33:17.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:16 smithi172 ceph-mon[36566]: Deploying daemon mds.a on smithi043 2024-03-28T22:33:18.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:18 smithi043 ceph-mon[31823]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:18.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:18 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/103891730' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:18.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:18 smithi172 ceph-mon[36566]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:18.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:18 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/103891730' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:18.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:18 smithi082 ceph-mon[36471]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:18.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:18 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/103891730' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:19.110 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:33:19.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.405+0000 7fe214e00640 1 -- 172.21.15.43:0/3167366875 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe210103990 msgr2=0x7fe210103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:19.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.405+0000 7fe214e00640 1 --2- 172.21.15.43:0/3167366875 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe210103990 0x7fe210103db0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe200009900 tx=0x7fe20002bc30 comp rx=0 tx=0).stop 2024-03-28T22:33:19.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.405+0000 7fe214e00640 1 -- 172.21.15.43:0/3167366875 shutdown_connections 2024-03-28T22:33:19.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.405+0000 7fe214e00640 1 --2- 172.21.15.43:0/3167366875 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe210105540 0x7fe21010bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:19.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.405+0000 7fe214e00640 1 --2- 172.21.15.43:0/3167366875 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe210104b80 0x7fe210105000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:19.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.405+0000 7fe214e00640 1 --2- 172.21.15.43:0/3167366875 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe210103990 0x7fe210103db0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:19.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.405+0000 7fe214e00640 1 -- 172.21.15.43:0/3167366875 >> 172.21.15.43:0/3167366875 conn(0x7fe2100fef40 msgr2=0x7fe210101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:19.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.405+0000 7fe214e00640 1 -- 172.21.15.43:0/3167366875 shutdown_connections 2024-03-28T22:33:19.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.405+0000 7fe214e00640 1 -- 172.21.15.43:0/3167366875 wait complete. 2024-03-28T22:33:19.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.407+0000 7fe214e00640 1 Processor -- start 2024-03-28T22:33:19.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.407+0000 7fe214e00640 1 -- start start 2024-03-28T22:33:19.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.407+0000 7fe214e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe210103990 0x7fe210199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:19.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.407+0000 7fe214e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe210104b80 0x7fe21019a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:19.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.407+0000 7fe214e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe210105540 0x7fe2101a1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:19.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.407+0000 7fe214e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe21010f630 con 0x7fe210105540 2024-03-28T22:33:19.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.407+0000 7fe214e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe21010f4b0 con 0x7fe210103990 2024-03-28T22:33:19.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.407+0000 7fe214e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe21010f7b0 con 0x7fe210104b80 2024-03-28T22:33:19.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.408+0000 7fe20e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe210105540 0x7fe2101a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:19.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.408+0000 7fe20d400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe210104b80 0x7fe21019a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:19.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.408+0000 7fe20de00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe210103990 0x7fe210199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:19.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.408+0000 7fe20d400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe210104b80 0x7fe21019a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57172/0 (socket says 172.21.15.43:57172) 2024-03-28T22:33:19.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.408+0000 7fe20d400640 1 -- 172.21.15.43:0/2156343955 learned_addr learned my addr 172.21.15.43:0/2156343955 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:19.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.408+0000 7fe20de00640 1 -- 172.21.15.43:0/2156343955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe210104b80 msgr2=0x7fe21019a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:19.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.408+0000 7fe20de00640 1 --2- 172.21.15.43:0/2156343955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe210104b80 0x7fe21019a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:19.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.408+0000 7fe20de00640 1 -- 172.21.15.43:0/2156343955 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe210105540 msgr2=0x7fe2101a1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:19.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.408+0000 7fe20de00640 1 --2- 172.21.15.43:0/2156343955 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe210105540 0x7fe2101a1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:19.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.408+0000 7fe20de00640 1 -- 172.21.15.43:0/2156343955 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2101a18e0 con 0x7fe210103990 2024-03-28T22:33:19.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.409+0000 7fe20de00640 1 --2- 172.21.15.43:0/2156343955 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe210103990 0x7fe210199ca0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe200009230 tx=0x7fe20002c5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:19.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.409+0000 7fe206a00640 1 -- 172.21.15.43:0/2156343955 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe200043020 con 0x7fe210103990 2024-03-28T22:33:19.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.409+0000 7fe214e00640 1 -- 172.21.15.43:0/2156343955 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2101a1b70 con 0x7fe210103990 2024-03-28T22:33:19.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.409+0000 7fe206a00640 1 -- 172.21.15.43:0/2156343955 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2000054e0 con 0x7fe210103990 2024-03-28T22:33:19.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.409+0000 7fe206a00640 1 -- 172.21.15.43:0/2156343955 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe200003b60 con 0x7fe210103990 2024-03-28T22:33:19.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.409+0000 7fe214e00640 1 -- 172.21.15.43:0/2156343955 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2101a20e0 con 0x7fe210103990 2024-03-28T22:33:19.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.410+0000 7fe214e00640 1 -- 172.21.15.43:0/2156343955 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe21010acb0 con 0x7fe210103990 2024-03-28T22:33:19.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.412+0000 7fe206a00640 1 -- 172.21.15.43:0/2156343955 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fe200046020 con 0x7fe210103990 2024-03-28T22:33:19.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.414+0000 7fe206a00640 1 --2- 172.21.15.43:0/2156343955 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe1f40a1d80 0x7fe1f40a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:19.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.414+0000 7fe206a00640 1 -- 172.21.15.43:0/2156343955 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fe2000ef9e0 con 0x7fe210103990 2024-03-28T22:33:19.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.414+0000 7fe20d400640 1 --2- 172.21.15.43:0/2156343955 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe1f40a1d80 0x7fe1f40a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:19.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.415+0000 7fe20d400640 1 --2- 172.21.15.43:0/2156343955 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe1f40a1d80 0x7fe1f40a4240 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe21019b1a0 tx=0x7fe1fc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:19.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.418+0000 7fe206a00640 1 -- 172.21.15.43:0/2156343955 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe20004d020 con 0x7fe210103990 2024-03-28T22:33:19.675 INFO:journalctl@ceph.mds.a.smithi043.stdout:Mar 28 22:33:19 smithi043 systemd[1]: Starting Ceph mds.a for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:33:19.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.680+0000 7fe214e00640 1 -- 172.21.15.43:0/2156343955 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe21002d170 con 0x7fe210103990 2024-03-28T22:33:19.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.681+0000 7fe206a00640 1 -- 172.21.15.43:0/2156343955 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8827 (secure 0 0 0) 0x7fe2000bd210 con 0x7fe210103990 2024-03-28T22:33:19.680 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:33:19.680 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6835/1084988072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1084988072},{"type":"v1","addr":"172.21.15.82:6835","nonce":1084988072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24518,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24556,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-28T22:33:19.680 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 11 2024-03-28T22:33:19.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.684+0000 7fe214e00640 1 -- 172.21.15.43:0/2156343955 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe1f40a1d80 msgr2=0x7fe1f40a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:19.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.684+0000 7fe214e00640 1 --2- 172.21.15.43:0/2156343955 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe1f40a1d80 0x7fe1f40a4240 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe21019b1a0 tx=0x7fe1fc005f90 comp rx=0 tx=0).stop 2024-03-28T22:33:19.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.684+0000 7fe214e00640 1 -- 172.21.15.43:0/2156343955 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe210103990 msgr2=0x7fe210199ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:19.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.684+0000 7fe214e00640 1 --2- 172.21.15.43:0/2156343955 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe210103990 0x7fe210199ca0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe200009230 tx=0x7fe20002c5d0 comp rx=0 tx=0).stop 2024-03-28T22:33:19.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.684+0000 7fe214e00640 1 -- 172.21.15.43:0/2156343955 shutdown_connections 2024-03-28T22:33:19.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.684+0000 7fe214e00640 1 --2- 172.21.15.43:0/2156343955 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe210105540 0x7fe2101a1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:19.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.684+0000 7fe214e00640 1 --2- 172.21.15.43:0/2156343955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe210104b80 0x7fe21019a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:19.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.684+0000 7fe214e00640 1 --2- 172.21.15.43:0/2156343955 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe1f40a1d80 0x7fe1f40a4240 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:19.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.684+0000 7fe214e00640 1 --2- 172.21.15.43:0/2156343955 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe210103990 0x7fe210199ca0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:19.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.684+0000 7fe214e00640 1 -- 172.21.15.43:0/2156343955 >> 172.21.15.43:0/2156343955 conn(0x7fe2100fef40 msgr2=0x7fe210107e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:19.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.684+0000 7fe214e00640 1 -- 172.21.15.43:0/2156343955 shutdown_connections 2024-03-28T22:33:19.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:19.684+0000 7fe214e00640 1 -- 172.21.15.43:0/2156343955 wait complete. 2024-03-28T22:33:20.017 INFO:journalctl@ceph.mds.a.smithi043.stdout:Mar 28 22:33:19 smithi043 podman[71225]: 2024-03-28 22:33:19.583379874 +0000 UTC m=+0.036919657 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:33:20.017 INFO:journalctl@ceph.mds.a.smithi043.stdout:Mar 28 22:33:19 smithi043 podman[71225]: 2024-03-28 22:33:19.714780516 +0000 UTC m=+0.168320343 container create cfd48870702c76ea810a086f63ca8c536f9d9f8fd34c481e975b0e56e0ae21e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-a, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-28T22:33:20.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:20 smithi172 ceph-mon[36566]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:20.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:20 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2156343955' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:20.398 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:20 smithi082 ceph-mon[36471]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:20.398 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:20 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2156343955' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:20.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:20 smithi043 ceph-mon[31823]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:20.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:20 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2156343955' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:20.517 INFO:journalctl@ceph.mds.a.smithi043.stdout:Mar 28 22:33:20 smithi043 podman[71225]: 2024-03-28 22:33:20.04448481 +0000 UTC m=+0.498024664 container init cfd48870702c76ea810a086f63ca8c536f9d9f8fd34c481e975b0e56e0ae21e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-323d23d, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-28T22:33:20.517 INFO:journalctl@ceph.mds.a.smithi043.stdout:Mar 28 22:33:20 smithi043 podman[71225]: 2024-03-28 22:33:20.055945064 +0000 UTC m=+0.509484878 container start cfd48870702c76ea810a086f63ca8c536f9d9f8fd34c481e975b0e56e0ae21e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-a, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=squid-323d23d, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-28T22:33:20.517 INFO:journalctl@ceph.mds.a.smithi043.stdout:Mar 28 22:33:20 smithi043 ceph-mds[71245]: -- 172.21.15.43:0/901084097 <== mon.2 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f1c5649a40 con 0x55f1c561d000 2024-03-28T22:33:20.517 INFO:journalctl@ceph.mds.a.smithi043.stdout:Mar 28 22:33:20 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-a[71241]: starting mds.a at 2024-03-28T22:33:20.517 INFO:journalctl@ceph.mds.a.smithi043.stdout:Mar 28 22:33:20 smithi043 bash[71225]: cfd48870702c76ea810a086f63ca8c536f9d9f8fd34c481e975b0e56e0ae21e0 2024-03-28T22:33:20.517 INFO:journalctl@ceph.mds.a.smithi043.stdout:Mar 28 22:33:20 smithi043 systemd[1]: Started Ceph mds.a for c2276428-ed51-11ee-95d0-87774f69a715. 2024-03-28T22:33:21.629 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:21 smithi082 ceph-mon[36471]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:21.629 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:21 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:21.629 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:21 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:21.630 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:21 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:21.630 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:21 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:33:21.630 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:21 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:33:21.630 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:21 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:33:21.630 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:21 smithi082 ceph-mon[36471]: Deploying daemon mds.b on smithi082 2024-03-28T22:33:21.630 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:21 smithi082 ceph-mon[36471]: mds.? [v2:172.21.15.43:6840/2371736231,v1:172.21.15.43:6841/2371736231] up:boot 2024-03-28T22:33:21.630 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:21 smithi082 ceph-mon[36471]: fsmap 11 up:standby 2024-03-28T22:33:21.630 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:21 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T22:33:21.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:21 smithi172 ceph-mon[36566]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:21.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:21.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:21.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:21.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:33:21.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:33:21.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:33:21.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:21 smithi172 ceph-mon[36566]: Deploying daemon mds.b on smithi082 2024-03-28T22:33:21.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:21 smithi172 ceph-mon[36566]: mds.? [v2:172.21.15.43:6840/2371736231,v1:172.21.15.43:6841/2371736231] up:boot 2024-03-28T22:33:21.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:21 smithi172 ceph-mon[36566]: fsmap 11 up:standby 2024-03-28T22:33:21.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:21 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T22:33:21.703 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:33:21.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:21 smithi043 ceph-mon[31823]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:21.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:21 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:21.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:21 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:21.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:21 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:21.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:21 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T22:33:21.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:21 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T22:33:21.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:21 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:33:21.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:21 smithi043 ceph-mon[31823]: Deploying daemon mds.b on smithi082 2024-03-28T22:33:21.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:21 smithi043 ceph-mon[31823]: mds.? [v2:172.21.15.43:6840/2371736231,v1:172.21.15.43:6841/2371736231] up:boot 2024-03-28T22:33:21.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:21 smithi043 ceph-mon[31823]: fsmap 11 up:standby 2024-03-28T22:33:21.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:21 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T22:33:21.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.945+0000 7f7c2e400640 1 -- 172.21.15.43:0/3354806514 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c28101590 msgr2=0x7f7c281019b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:21.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.945+0000 7f7c2e400640 1 --2- 172.21.15.43:0/3354806514 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c28101590 0x7f7c281019b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c009900 tx=0x7f7c1c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:33:21.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.945+0000 7f7c2e400640 1 -- 172.21.15.43:0/3354806514 shutdown_connections 2024-03-28T22:33:21.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.945+0000 7f7c2e400640 1 --2- 172.21.15.43:0/3354806514 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c28102780 0x7f7c28102c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:21.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.945+0000 7f7c2e400640 1 --2- 172.21.15.43:0/3354806514 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7c28103140 0x7f7c281099d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:21.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.945+0000 7f7c2e400640 1 --2- 172.21.15.43:0/3354806514 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c28101590 0x7f7c281019b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:21.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.945+0000 7f7c2e400640 1 -- 172.21.15.43:0/3354806514 >> 172.21.15.43:0/3354806514 conn(0x7f7c280fcae0 msgr2=0x7f7c280fef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:21.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.945+0000 7f7c2e400640 1 -- 172.21.15.43:0/3354806514 shutdown_connections 2024-03-28T22:33:21.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.945+0000 7f7c2e400640 1 -- 172.21.15.43:0/3354806514 wait complete. 2024-03-28T22:33:21.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.946+0000 7f7c2e400640 1 Processor -- start 2024-03-28T22:33:21.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.946+0000 7f7c2e400640 1 -- start start 2024-03-28T22:33:21.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.946+0000 7f7c2e400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7c28101590 0x7f7c28199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:21.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.946+0000 7f7c2e400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c28102780 0x7f7c2819a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:21.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.946+0000 7f7c2e400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c28103140 0x7f7c281a1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:21.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.947+0000 7f7c2e400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2810d220 con 0x7f7c28103140 2024-03-28T22:33:21.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.947+0000 7f7c2e400640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2810d0a0 con 0x7f7c28102780 2024-03-28T22:33:21.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.947+0000 7f7c2e400640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2810d3a0 con 0x7f7c28101590 2024-03-28T22:33:21.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.947+0000 7f7c2da00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c28103140 0x7f7c281a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:21.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.947+0000 7f7c2da00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c28103140 0x7f7c281a1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45946/0 (socket says 172.21.15.43:45946) 2024-03-28T22:33:21.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.947+0000 7f7c2da00640 1 -- 172.21.15.43:0/1864281398 learned_addr learned my addr 172.21.15.43:0/1864281398 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:21.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.947+0000 7f7c27e00640 1 --2- 172.21.15.43:0/1864281398 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c28102780 0x7f7c2819a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:21.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.947+0000 7f7c2d000640 1 --2- 172.21.15.43:0/1864281398 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7c28101590 0x7f7c28199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:21.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.947+0000 7f7c2da00640 1 -- 172.21.15.43:0/1864281398 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7c28101590 msgr2=0x7f7c28199ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:21.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.947+0000 7f7c2da00640 1 --2- 172.21.15.43:0/1864281398 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7c28101590 0x7f7c28199ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:21.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.947+0000 7f7c2da00640 1 -- 172.21.15.43:0/1864281398 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c28102780 msgr2=0x7f7c2819a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:21.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.947+0000 7f7c2da00640 1 --2- 172.21.15.43:0/1864281398 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c28102780 0x7f7c2819a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:21.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.947+0000 7f7c2da00640 1 -- 172.21.15.43:0/1864281398 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c281a16e0 con 0x7f7c28103140 2024-03-28T22:33:21.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.947+0000 7f7c27e00640 1 --2- 172.21.15.43:0/1864281398 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c28102780 0x7f7c2819a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:33:21.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.948+0000 7f7c2da00640 1 --2- 172.21.15.43:0/1864281398 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c28103140 0x7f7c281a1070 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7c2000eb10 tx=0x7f7c2000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:21.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.948+0000 7f7c25600640 1 -- 172.21.15.43:0/1864281398 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c20004b20 con 0x7f7c28103140 2024-03-28T22:33:21.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.948+0000 7f7c2e400640 1 -- 172.21.15.43:0/1864281398 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c281a19d0 con 0x7f7c28103140 2024-03-28T22:33:21.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.948+0000 7f7c25600640 1 -- 172.21.15.43:0/1864281398 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c20004cc0 con 0x7f7c28103140 2024-03-28T22:33:21.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.948+0000 7f7c25600640 1 -- 172.21.15.43:0/1864281398 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c20010760 con 0x7f7c28103140 2024-03-28T22:33:21.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.948+0000 7f7c2e400640 1 -- 172.21.15.43:0/1864281398 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c281a1ed0 con 0x7f7c28103140 2024-03-28T22:33:21.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.949+0000 7f7c2e400640 1 -- 172.21.15.43:0/1864281398 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c281a21c0 con 0x7f7c28103140 2024-03-28T22:33:21.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.951+0000 7f7c25600640 1 -- 172.21.15.43:0/1864281398 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f7c20005020 con 0x7f7c28103140 2024-03-28T22:33:21.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.952+0000 7f7c25600640 1 --2- 172.21.15.43:0/1864281398 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7c100a1d80 0x7f7c100a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:21.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.953+0000 7f7c25600640 1 -- 172.21.15.43:0/1864281398 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f7c200cdf60 con 0x7f7c28103140 2024-03-28T22:33:21.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.953+0000 7f7c2d000640 1 --2- 172.21.15.43:0/1864281398 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7c100a1d80 0x7f7c100a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:21.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.954+0000 7f7c2d000640 1 --2- 172.21.15.43:0/1864281398 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7c100a1d80 0x7f7c100a4240 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c000c00 tx=0x7f7c1c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:21.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:21.957+0000 7f7c25600640 1 -- 172.21.15.43:0/1864281398 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7c2009b790 con 0x7f7c28103140 2024-03-28T22:33:22.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:22.199+0000 7f7c2e400640 1 -- 172.21.15.43:0/1864281398 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c281a2390 con 0x7f7c28103140 2024-03-28T22:33:22.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:22.200+0000 7f7c25600640 1 -- 172.21.15.43:0/1864281398 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9653 (secure 0 0 0) 0x7f7c2009b090 con 0x7f7c28103140 2024-03-28T22:33:22.199 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:33:22.199 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6835/1084988072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1084988072},{"type":"v1","addr":"172.21.15.82:6835","nonce":1084988072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24518,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24556,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-28T22:33:22.199 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 12 2024-03-28T22:33:22.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:22.203+0000 7f7c2e400640 1 -- 172.21.15.43:0/1864281398 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7c100a1d80 msgr2=0x7f7c100a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:22.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:22.203+0000 7f7c2e400640 1 --2- 172.21.15.43:0/1864281398 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7c100a1d80 0x7f7c100a4240 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c000c00 tx=0x7f7c1c005fd0 comp rx=0 tx=0).stop 2024-03-28T22:33:22.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:22.203+0000 7f7c2e400640 1 -- 172.21.15.43:0/1864281398 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c28103140 msgr2=0x7f7c281a1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:22.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:22.203+0000 7f7c2e400640 1 --2- 172.21.15.43:0/1864281398 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c28103140 0x7f7c281a1070 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7c2000eb10 tx=0x7f7c2000c2d0 comp rx=0 tx=0).stop 2024-03-28T22:33:22.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:22.203+0000 7f7c2e400640 1 -- 172.21.15.43:0/1864281398 shutdown_connections 2024-03-28T22:33:22.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:22.203+0000 7f7c2e400640 1 --2- 172.21.15.43:0/1864281398 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7c100a1d80 0x7f7c100a4240 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:22.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:22.203+0000 7f7c2e400640 1 --2- 172.21.15.43:0/1864281398 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c28102780 0x7f7c2819a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:22.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:22.203+0000 7f7c2e400640 1 --2- 172.21.15.43:0/1864281398 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c28103140 0x7f7c281a1070 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:22.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:22.203+0000 7f7c2e400640 1 --2- 172.21.15.43:0/1864281398 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7c28101590 0x7f7c28199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:22.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:22.203+0000 7f7c2e400640 1 -- 172.21.15.43:0/1864281398 >> 172.21.15.43:0/1864281398 conn(0x7f7c280fcae0 msgr2=0x7f7c28073980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:22.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:22.203+0000 7f7c2e400640 1 -- 172.21.15.43:0/1864281398 shutdown_connections 2024-03-28T22:33:22.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:22.203+0000 7f7c2e400640 1 -- 172.21.15.43:0/1864281398 wait complete. 2024-03-28T22:33:22.379 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:22 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1864281398' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:22.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:22 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1864281398' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:22.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:22 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1864281398' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:22.699 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 28 22:33:22 smithi082 systemd[1]: Starting Ceph mds.b for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:33:23.069 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 28 22:33:22 smithi082 podman[67255]: 2024-03-28 22:33:22.690282321 +0000 UTC m=+0.017750049 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:33:23.070 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 28 22:33:22 smithi082 podman[67255]: 2024-03-28 22:33:22.810779875 +0000 UTC m=+0.138247592 container create bfb1218e0d706d28fc0ef96213fc8b14174e8505da1c89ac83e2db5604389e5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-323d23d, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T22:33:23.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:23 smithi082 ceph-mon[36471]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:23.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:23 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:23.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:23 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:23.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:23 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:23.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:23 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:23.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:23 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:23.450 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 28 22:33:23 smithi082 podman[67255]: 2024-03-28 22:33:23.071812402 +0000 UTC m=+0.399280135 container init bfb1218e0d706d28fc0ef96213fc8b14174e8505da1c89ac83e2db5604389e5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-b, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, CEPH_POINT_RELEASE=, RELEASE=squid-323d23d, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, ceph=True) 2024-03-28T22:33:23.450 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 28 22:33:23 smithi082 podman[67255]: 2024-03-28 22:33:23.074729579 +0000 UTC m=+0.402197304 container start bfb1218e0d706d28fc0ef96213fc8b14174e8505da1c89ac83e2db5604389e5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-b, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-28T22:33:23.450 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 28 22:33:23 smithi082 ceph-mds[67274]: -- 172.21.15.82:0/1415069116 <== mon.2 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55791f5a5a40 con 0x55791f579000 2024-03-28T22:33:23.450 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 28 22:33:23 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-b[67270]: starting mds.b at 2024-03-28T22:33:23.450 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 28 22:33:23 smithi082 bash[67255]: bfb1218e0d706d28fc0ef96213fc8b14174e8505da1c89ac83e2db5604389e5f 2024-03-28T22:33:23.450 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 28 22:33:23 smithi082 systemd[1]: Started Ceph mds.b for c2276428-ed51-11ee-95d0-87774f69a715. 2024-03-28T22:33:23.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:23 smithi172 ceph-mon[36566]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:23.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:23 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:23.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:23 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:23.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:23 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:23.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:23 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:23.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:23 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:23.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:23 smithi043 ceph-mon[31823]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:23.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:23 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:23.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:23 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:23.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:23 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:23.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:23 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:23.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:23 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:24.223 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:33:24.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.436+0000 7f6770e00640 1 -- 172.21.15.43:0/3855800746 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f676c103970 msgr2=0x7f676c103d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:24.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.436+0000 7f6770e00640 1 --2- 172.21.15.43:0/3855800746 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f676c103970 0x7f676c103d90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f675c009900 tx=0x7f675c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:33:24.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.436+0000 7f6770e00640 1 -- 172.21.15.43:0/3855800746 shutdown_connections 2024-03-28T22:33:24.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.436+0000 7f6770e00640 1 --2- 172.21.15.43:0/3855800746 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f676c105520 0x7f676c10bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:24.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.436+0000 7f6770e00640 1 --2- 172.21.15.43:0/3855800746 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f676c104b60 0x7f676c104fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:24.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.436+0000 7f6770e00640 1 --2- 172.21.15.43:0/3855800746 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f676c103970 0x7f676c103d90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:24.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.436+0000 7f6770e00640 1 -- 172.21.15.43:0/3855800746 >> 172.21.15.43:0/3855800746 conn(0x7f676c0fef00 msgr2=0x7f676c101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:24.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.436+0000 7f6770e00640 1 -- 172.21.15.43:0/3855800746 shutdown_connections 2024-03-28T22:33:24.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.436+0000 7f6770e00640 1 -- 172.21.15.43:0/3855800746 wait complete. 2024-03-28T22:33:24.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.437+0000 7f6770e00640 1 Processor -- start 2024-03-28T22:33:24.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.437+0000 7f6770e00640 1 -- start start 2024-03-28T22:33:24.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.437+0000 7f6770e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f676c103970 0x7f676c199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:24.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.437+0000 7f6770e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f676c104b60 0x7f676c19a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:24.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.437+0000 7f6770e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f676c105520 0x7f676c1a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:24.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.437+0000 7f6770e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f676c10f5e0 con 0x7f676c105520 2024-03-28T22:33:24.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.437+0000 7f6770e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f676c10f460 con 0x7f676c104b60 2024-03-28T22:33:24.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.437+0000 7f6770e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f676c10f760 con 0x7f676c103970 2024-03-28T22:33:24.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.438+0000 7f6769e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f676c103970 0x7f676c199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:24.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.438+0000 7f6769e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f676c103970 0x7f676c199cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57222/0 (socket says 172.21.15.43:57222) 2024-03-28T22:33:24.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.438+0000 7f6769e00640 1 -- 172.21.15.43:0/1577033627 learned_addr learned my addr 172.21.15.43:0/1577033627 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:24.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.438+0000 7f6769400640 1 --2- 172.21.15.43:0/1577033627 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f676c104b60 0x7f676c19a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:24.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.438+0000 7f676a800640 1 --2- 172.21.15.43:0/1577033627 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f676c105520 0x7f676c1a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:24.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.438+0000 7f6769e00640 1 -- 172.21.15.43:0/1577033627 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f676c104b60 msgr2=0x7f676c19a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:24.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.438+0000 7f6769e00640 1 --2- 172.21.15.43:0/1577033627 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f676c104b60 0x7f676c19a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:24.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.438+0000 7f6769e00640 1 -- 172.21.15.43:0/1577033627 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f676c105520 msgr2=0x7f676c1a1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:24.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.438+0000 7f6769e00640 1 --2- 172.21.15.43:0/1577033627 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f676c105520 0x7f676c1a1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:24.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.438+0000 7f6769e00640 1 -- 172.21.15.43:0/1577033627 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f676c1a18f0 con 0x7f676c103970 2024-03-28T22:33:24.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.438+0000 7f6769400640 1 --2- 172.21.15.43:0/1577033627 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f676c104b60 0x7f676c19a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:33:24.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.438+0000 7f6769e00640 1 --2- 172.21.15.43:0/1577033627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f676c103970 0x7f676c199cb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f675c000c00 tx=0x7f675c02c640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:24.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.439+0000 7f6762a00640 1 -- 172.21.15.43:0/1577033627 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f675c041070 con 0x7f676c103970 2024-03-28T22:33:24.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.439+0000 7f6762a00640 1 -- 172.21.15.43:0/1577033627 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f675c004450 con 0x7f676c103970 2024-03-28T22:33:24.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.439+0000 7f6762a00640 1 -- 172.21.15.43:0/1577033627 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f675c034440 con 0x7f676c103970 2024-03-28T22:33:24.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.439+0000 7f6770e00640 1 -- 172.21.15.43:0/1577033627 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f676c1a1b80 con 0x7f676c103970 2024-03-28T22:33:24.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.439+0000 7f6770e00640 1 -- 172.21.15.43:0/1577033627 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f676c1a2020 con 0x7f676c103970 2024-03-28T22:33:24.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.441+0000 7f6770e00640 1 -- 172.21.15.43:0/1577033627 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f676c10ad60 con 0x7f676c103970 2024-03-28T22:33:24.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.443+0000 7f6762a00640 1 -- 172.21.15.43:0/1577033627 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f675c044070 con 0x7f676c103970 2024-03-28T22:33:24.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.444+0000 7f6762a00640 1 --2- 172.21.15.43:0/1577033627 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f67500a1e00 0x7f67500a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:24.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.444+0000 7f6762a00640 1 -- 172.21.15.43:0/1577033627 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f675c0ef3c0 con 0x7f676c103970 2024-03-28T22:33:24.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.444+0000 7f6769400640 1 --2- 172.21.15.43:0/1577033627 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f67500a1e00 0x7f67500a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:24.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.445+0000 7f6769400640 1 --2- 172.21.15.43:0/1577033627 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f67500a1e00 0x7f67500a42c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f676c19b1b0 tx=0x7f6758005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:24.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.448+0000 7f6762a00640 1 -- 172.21.15.43:0/1577033627 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f675c0bc630 con 0x7f676c103970 2024-03-28T22:33:24.606 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:24 smithi172 ceph-mon[36566]: mds.? [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] up:boot 2024-03-28T22:33:24.606 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:24 smithi172 ceph-mon[36566]: fsmap 12 up:standby 2024-03-28T22:33:24.606 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:24 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T22:33:24.606 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:24 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:33:24.683 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:24 smithi082 ceph-mon[36471]: mds.? [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] up:boot 2024-03-28T22:33:24.683 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:24 smithi082 ceph-mon[36471]: fsmap 12 up:standby 2024-03-28T22:33:24.683 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:24 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T22:33:24.683 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:24 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:33:24.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.729+0000 7f6770e00640 1 -- 172.21.15.43:0/1577033627 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f676c02d170 con 0x7f676c103970 2024-03-28T22:33:24.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.730+0000 7f6762a00640 1 -- 172.21.15.43:0/1577033627 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10479 (secure 0 0 0) 0x7f675c0dc0e0 con 0x7f676c103970 2024-03-28T22:33:24.728 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:33:24.729 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6835/1084988072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1084988072},{"type":"v1","addr":"172.21.15.82:6835","nonce":1084988072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24518,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24556,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24572,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-03-28T22:33:24.729 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 13 2024-03-28T22:33:24.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.733+0000 7f6770e00640 1 -- 172.21.15.43:0/1577033627 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f67500a1e00 msgr2=0x7f67500a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:24.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.733+0000 7f6770e00640 1 --2- 172.21.15.43:0/1577033627 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f67500a1e00 0x7f67500a42c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f676c19b1b0 tx=0x7f6758005f90 comp rx=0 tx=0).stop 2024-03-28T22:33:24.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.733+0000 7f6770e00640 1 -- 172.21.15.43:0/1577033627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f676c103970 msgr2=0x7f676c199cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:24.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.733+0000 7f6770e00640 1 --2- 172.21.15.43:0/1577033627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f676c103970 0x7f676c199cb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f675c000c00 tx=0x7f675c02c640 comp rx=0 tx=0).stop 2024-03-28T22:33:24.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.733+0000 7f6770e00640 1 -- 172.21.15.43:0/1577033627 shutdown_connections 2024-03-28T22:33:24.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.733+0000 7f6770e00640 1 --2- 172.21.15.43:0/1577033627 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f67500a1e00 0x7f67500a42c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:24.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.733+0000 7f6770e00640 1 --2- 172.21.15.43:0/1577033627 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f676c104b60 0x7f676c19a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:24.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.733+0000 7f6770e00640 1 --2- 172.21.15.43:0/1577033627 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f676c105520 0x7f676c1a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:24.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.733+0000 7f6770e00640 1 --2- 172.21.15.43:0/1577033627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f676c103970 0x7f676c199cb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:24.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.733+0000 7f6770e00640 1 -- 172.21.15.43:0/1577033627 >> 172.21.15.43:0/1577033627 conn(0x7f676c0fef00 msgr2=0x7f676c107f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:24.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.733+0000 7f6770e00640 1 -- 172.21.15.43:0/1577033627 shutdown_connections 2024-03-28T22:33:24.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.733+0000 7f6770e00640 1 -- 172.21.15.43:0/1577033627 wait complete. 2024-03-28T22:33:24.747 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:24 smithi043 ceph-mon[31823]: mds.? [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] up:boot 2024-03-28T22:33:24.747 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:24 smithi043 ceph-mon[31823]: fsmap 12 up:standby 2024-03-28T22:33:24.747 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:24 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T22:33:24.747 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:24 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:33:24.752 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:33:24.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.975+0000 7f544ac00640 1 -- 172.21.15.43:0/1275512838 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5444104b80 msgr2=0x7f5444105000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:24.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.975+0000 7f544ac00640 1 --2- 172.21.15.43:0/1275512838 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5444104b80 0x7f5444105000 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5430009900 tx=0x7f543002bc30 comp rx=0 tx=0).stop 2024-03-28T22:33:24.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.975+0000 7f544ac00640 1 -- 172.21.15.43:0/1275512838 shutdown_connections 2024-03-28T22:33:24.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.975+0000 7f544ac00640 1 --2- 172.21.15.43:0/1275512838 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5444105540 0x7f544410bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:24.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.975+0000 7f544ac00640 1 --2- 172.21.15.43:0/1275512838 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5444104b80 0x7f5444105000 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:24.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.975+0000 7f544ac00640 1 --2- 172.21.15.43:0/1275512838 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5444103990 0x7f5444103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:24.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.976+0000 7f544ac00640 1 -- 172.21.15.43:0/1275512838 >> 172.21.15.43:0/1275512838 conn(0x7f54440fef40 msgr2=0x7f5444101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:24.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.976+0000 7f544ac00640 1 -- 172.21.15.43:0/1275512838 shutdown_connections 2024-03-28T22:33:24.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.976+0000 7f544ac00640 1 -- 172.21.15.43:0/1275512838 wait complete. 2024-03-28T22:33:24.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.977+0000 7f544ac00640 1 Processor -- start 2024-03-28T22:33:24.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.977+0000 7f544ac00640 1 -- start start 2024-03-28T22:33:24.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.977+0000 7f544ac00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5444103990 0x7f5444199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:24.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.977+0000 7f544ac00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5444104b80 0x7f544419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:24.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.977+0000 7f544ac00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5444105540 0x7f54441a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:24.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.977+0000 7f544ac00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f544410f660 con 0x7f5444105540 2024-03-28T22:33:24.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.977+0000 7f544ac00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f544410f4e0 con 0x7f5444103990 2024-03-28T22:33:24.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.977+0000 7f544ac00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f544410f7e0 con 0x7f5444104b80 2024-03-28T22:33:24.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.977+0000 7f5443e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5444103990 0x7f5444199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:24.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.977+0000 7f5443400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5444104b80 0x7f544419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:24.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.977+0000 7f5443e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5444103990 0x7f5444199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:42820/0 (socket says 172.21.15.43:42820) 2024-03-28T22:33:24.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.977+0000 7f5443e00640 1 -- 172.21.15.43:0/1130165710 learned_addr learned my addr 172.21.15.43:0/1130165710 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:24.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.977+0000 7f5448c00640 1 --2- 172.21.15.43:0/1130165710 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5444105540 0x7f54441a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:24.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.978+0000 7f5443e00640 1 -- 172.21.15.43:0/1130165710 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5444104b80 msgr2=0x7f544419a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:24.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.978+0000 7f5443e00640 1 --2- 172.21.15.43:0/1130165710 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5444104b80 0x7f544419a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:24.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.978+0000 7f5443e00640 1 -- 172.21.15.43:0/1130165710 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5444105540 msgr2=0x7f54441a11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:24.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.978+0000 7f5443e00640 1 --2- 172.21.15.43:0/1130165710 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5444105540 0x7f54441a11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:24.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.978+0000 7f5443e00640 1 -- 172.21.15.43:0/1130165710 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54441a1880 con 0x7f5444103990 2024-03-28T22:33:24.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.978+0000 7f5443e00640 1 --2- 172.21.15.43:0/1130165710 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5444103990 0x7f5444199c70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f543400e9c0 tx=0x7f543400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:24.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.978+0000 7f5440c00640 1 -- 172.21.15.43:0/1130165710 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f543400cd80 con 0x7f5444103990 2024-03-28T22:33:24.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.978+0000 7f544ac00640 1 -- 172.21.15.43:0/1130165710 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54441a1b70 con 0x7f5444103990 2024-03-28T22:33:24.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.978+0000 7f5440c00640 1 -- 172.21.15.43:0/1130165710 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54340035f0 con 0x7f5444103990 2024-03-28T22:33:24.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.978+0000 7f5440c00640 1 -- 172.21.15.43:0/1130165710 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5434010690 con 0x7f5444103990 2024-03-28T22:33:24.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.979+0000 7f544ac00640 1 -- 172.21.15.43:0/1130165710 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54441a20c0 con 0x7f5444103990 2024-03-28T22:33:24.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.980+0000 7f544ac00640 1 -- 172.21.15.43:0/1130165710 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f544410acb0 con 0x7f5444103990 2024-03-28T22:33:24.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.981+0000 7f5440c00640 1 -- 172.21.15.43:0/1130165710 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f54340037a0 con 0x7f5444103990 2024-03-28T22:33:24.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.983+0000 7f5440c00640 1 --2- 172.21.15.43:0/1130165710 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f54280a1d80 0x7f54280a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:24.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.983+0000 7f5440c00640 1 -- 172.21.15.43:0/1130165710 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f54340cde70 con 0x7f5444103990 2024-03-28T22:33:24.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.983+0000 7f5443400640 1 --2- 172.21.15.43:0/1130165710 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f54280a1d80 0x7f54280a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:24.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.984+0000 7f5443400640 1 --2- 172.21.15.43:0/1130165710 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f54280a1d80 0x7f54280a4240 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f543002c240 tx=0x7f543002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:24.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:24.987+0000 7f5440c00640 1 -- 172.21.15.43:0/1130165710 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f543409b6a0 con 0x7f5444103990 2024-03-28T22:33:25.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.207+0000 7f544ac00640 1 -- 172.21.15.43:0/1130165710 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5444062350 con 0x7f5444103990 2024-03-28T22:33:25.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.209+0000 7f5440c00640 1 -- 172.21.15.43:0/1130165710 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+18421 (secure 0 0 0) 0x7f543409afa0 con 0x7f5444103990 2024-03-28T22:33:25.209 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:33:25.209 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":83,"fsid":"c2276428-ed51-11ee-95d0-87774f69a715","created":"2024-03-28T22:24:19.950084+0000","modified":"2024-03-28T22:32:32.384331+0000","last_up_change":"2024-03-28T22:32:31.078737+0000","last_in_change":"2024-03-28T22:32:10.147097+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:28:14.713290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b855adcd-786c-4e42-b8fe-a515a77ce979","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6803","nonce":2826124409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6805","nonce":2826124409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6809","nonce":2826124409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6807","nonce":2826124409}]},"public_addr":"172.21.15.43:6803/2826124409","cluster_addr":"172.21.15.43:6805/2826124409","heartbeat_back_addr":"172.21.15.43:6809/2826124409","heartbeat_front_addr":"172.21.15.43:6807/2826124409","state":["exists","up"]},{"osd":1,"uuid":"031c143d-8ad1-4b68-8db9-3c717137e3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6801","nonce":3043263644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6803","nonce":3043263644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6807","nonce":3043263644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6805","nonce":3043263644}]},"public_addr":"172.21.15.82:6801/3043263644","cluster_addr":"172.21.15.82:6803/3043263644","heartbeat_back_addr":"172.21.15.82:6807/3043263644","heartbeat_front_addr":"172.21.15.82:6805/3043263644","state":["exists","up"]},{"osd":2,"uuid":"20632bbb-f07a-4ca1-b0c4-ff9d508e211c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6801","nonce":4213953645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6803","nonce":4213953645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6807","nonce":4213953645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6805","nonce":4213953645}]},"public_addr":"172.21.15.172:6801/4213953645","cluster_addr":"172.21.15.172:6803/4213953645","heartbeat_back_addr":"172.21.15.172:6807/4213953645","heartbeat_front_addr":"172.21.15.172:6805/4213953645","state":["exists","up"]},{"osd":3,"uuid":"9a1f2312-4dbd-475b-8736-d5ebc8788e09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6811","nonce":1597280920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6813","nonce":1597280920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6817","nonce":1597280920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6815","nonce":1597280920}]},"public_addr":"172.21.15.43:6811/1597280920","cluster_addr":"172.21.15.43:6813/1597280920","heartbeat_back_addr":"172.21.15.43:6817/1597280920","heartbeat_front_addr":"172.21.15.43:6815/1597280920","state":["exists","up"]},{"osd":4,"uuid":"29140ead-7075-42b2-90c1-27cae0b29ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6809","nonce":215451895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6811","nonce":215451895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6815","nonce":215451895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6813","nonce":215451895}]},"public_addr":"172.21.15.82:6809/215451895","cluster_addr":"172.21.15.82:6811/215451895","heartbeat_back_addr":"172.21.15.82:6815/215451895","heartbeat_front_addr":"172.21.15.82:6813/215451895","state":["exists","up"]},{"osd":5,"uuid":"59644c61-93dc-4536-b2ad-0256b24e5cb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6809","nonce":1576148933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6811","nonce":1576148933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6815","nonce":1576148933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6813","nonce":1576148933}]},"public_addr":"172.21.15.172:6809/1576148933","cluster_addr":"172.21.15.172:6811/1576148933","heartbeat_back_addr":"172.21.15.172:6815/1576148933","heartbeat_front_addr":"172.21.15.172:6813/1576148933","state":["exists","up"]},{"osd":6,"uuid":"22d54e57-c3f1-4ecf-9921-8e815b48f92a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6819","nonce":791044279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6821","nonce":791044279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6825","nonce":791044279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6823","nonce":791044279}]},"public_addr":"172.21.15.43:6819/791044279","cluster_addr":"172.21.15.43:6821/791044279","heartbeat_back_addr":"172.21.15.43:6825/791044279","heartbeat_front_addr":"172.21.15.43:6823/791044279","state":["exists","up"]},{"osd":7,"uuid":"248cd1e2-0d6b-452f-a0a8-c51fe1b975ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6817","nonce":2157777323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6819","nonce":2157777323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6823","nonce":2157777323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6821","nonce":2157777323}]},"public_addr":"172.21.15.82:6817/2157777323","cluster_addr":"172.21.15.82:6819/2157777323","heartbeat_back_addr":"172.21.15.82:6823/2157777323","heartbeat_front_addr":"172.21.15.82:6821/2157777323","state":["exists","up"]},{"osd":8,"uuid":"d98aab6e-8bae-4c24-906c-4f492f047b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6817","nonce":1095328248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6819","nonce":1095328248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6823","nonce":1095328248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6821","nonce":1095328248}]},"public_addr":"172.21.15.172:6817/1095328248","cluster_addr":"172.21.15.172:6819/1095328248","heartbeat_back_addr":"172.21.15.172:6823/1095328248","heartbeat_front_addr":"172.21.15.172:6821/1095328248","state":["exists","up"]},{"osd":9,"uuid":"19aa383e-a0f6-45ea-96cc-e7956535caa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6827","nonce":3657791498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6829","nonce":3657791498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6833","nonce":3657791498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6831","nonce":3657791498}]},"public_addr":"172.21.15.43:6827/3657791498","cluster_addr":"172.21.15.43:6829/3657791498","heartbeat_back_addr":"172.21.15.43:6833/3657791498","heartbeat_front_addr":"172.21.15.43:6831/3657791498","state":["exists","up"]},{"osd":10,"uuid":"666977e5-eaf7-41b9-a18f-c380e149345d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6825","nonce":3454954047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6827","nonce":3454954047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6831","nonce":3454954047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6829","nonce":3454954047}]},"public_addr":"172.21.15.82:6825/3454954047","cluster_addr":"172.21.15.82:6827/3454954047","heartbeat_back_addr":"172.21.15.82:6831/3454954047","heartbeat_front_addr":"172.21.15.82:6829/3454954047","state":["exists","up"]},{"osd":11,"uuid":"615cf3be-8076-470e-ad09-c880e7593717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6825","nonce":2563517144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6827","nonce":2563517144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6831","nonce":2563517144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6829","nonce":2563517144}]},"public_addr":"172.21.15.172:6825/2563517144","cluster_addr":"172.21.15.172:6827/2563517144","heartbeat_back_addr":"172.21.15.172:6831/2563517144","heartbeat_front_addr":"172.21.15.172:6829/2563517144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:21.727405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:47.100259+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:10.615438+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:43.632144+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:09.487940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:34.604463+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:09.594452+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:35.709795+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:01.184202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:35.455322+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:02.413166+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:28.550754+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/578791143":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/4029313215":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/1574178482":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6801/3517712438":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6800/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/1375099638":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/2710117652":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6801/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/3215634168":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6800/3517712438":"2024-03-29T22:25:28.008215+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:33:25.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.212+0000 7f544ac00640 1 -- 172.21.15.43:0/1130165710 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f54280a1d80 msgr2=0x7f54280a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:25.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.212+0000 7f544ac00640 1 --2- 172.21.15.43:0/1130165710 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f54280a1d80 0x7f54280a4240 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f543002c240 tx=0x7f543002df90 comp rx=0 tx=0).stop 2024-03-28T22:33:25.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.212+0000 7f544ac00640 1 -- 172.21.15.43:0/1130165710 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5444103990 msgr2=0x7f5444199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:25.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.212+0000 7f544ac00640 1 --2- 172.21.15.43:0/1130165710 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5444103990 0x7f5444199c70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f543400e9c0 tx=0x7f543400ee80 comp rx=0 tx=0).stop 2024-03-28T22:33:25.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.212+0000 7f544ac00640 1 -- 172.21.15.43:0/1130165710 shutdown_connections 2024-03-28T22:33:25.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.212+0000 7f544ac00640 1 --2- 172.21.15.43:0/1130165710 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5444105540 0x7f54441a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:25.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.212+0000 7f544ac00640 1 --2- 172.21.15.43:0/1130165710 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5444104b80 0x7f544419a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:25.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.213+0000 7f544ac00640 1 --2- 172.21.15.43:0/1130165710 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f54280a1d80 0x7f54280a4240 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:25.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.213+0000 7f544ac00640 1 --2- 172.21.15.43:0/1130165710 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5444103990 0x7f5444199c70 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:25.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.213+0000 7f544ac00640 1 -- 172.21.15.43:0/1130165710 >> 172.21.15.43:0/1130165710 conn(0x7f54440fef40 msgr2=0x7f5444107e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:25.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.213+0000 7f544ac00640 1 -- 172.21.15.43:0/1130165710 shutdown_connections 2024-03-28T22:33:25.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.213+0000 7f544ac00640 1 -- 172.21.15.43:0/1130165710 wait complete. 2024-03-28T22:33:25.231 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T22:28:14.713290+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T22:33:25.232 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T22:33:25.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.442+0000 7f9296400640 1 -- 172.21.15.43:0/2869727966 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9290077350 msgr2=0x7f92900777b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:25.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.442+0000 7f9296400640 1 --2- 172.21.15.43:0/2869727966 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9290077350 0x7f92900777b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f927c009900 tx=0x7f927c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:33:25.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.442+0000 7f9296400640 1 -- 172.21.15.43:0/2869727966 shutdown_connections 2024-03-28T22:33:25.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.442+0000 7f9296400640 1 --2- 172.21.15.43:0/2869727966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9290077cf0 0x7f9290078590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:25.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.442+0000 7f9296400640 1 --2- 172.21.15.43:0/2869727966 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9290077350 0x7f92900777b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:25.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.442+0000 7f9296400640 1 --2- 172.21.15.43:0/2869727966 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9290076110 0x7f9290076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:25.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.442+0000 7f9296400640 1 -- 172.21.15.43:0/2869727966 >> 172.21.15.43:0/2869727966 conn(0x7f92900fef00 msgr2=0x7f9290101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:25.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.443+0000 7f9296400640 1 -- 172.21.15.43:0/2869727966 shutdown_connections 2024-03-28T22:33:25.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.443+0000 7f9296400640 1 -- 172.21.15.43:0/2869727966 wait complete. 2024-03-28T22:33:25.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.443+0000 7f9296400640 1 Processor -- start 2024-03-28T22:33:25.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.444+0000 7f9296400640 1 -- start start 2024-03-28T22:33:25.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.444+0000 7f9296400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9290076110 0x7f929019e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:25.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.444+0000 7f9296400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9290077350 0x7f929019e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:25.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.444+0000 7f9296400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9290077cf0 0x7f92901a5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:25.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.444+0000 7f9296400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9290113a20 con 0x7f9290077350 2024-03-28T22:33:25.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.444+0000 7f9296400640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f92901138a0 con 0x7f9290077cf0 2024-03-28T22:33:25.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.444+0000 7f9296400640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9290113ba0 con 0x7f9290076110 2024-03-28T22:33:25.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.444+0000 7f928f400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9290076110 0x7f929019e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:25.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.444+0000 7f928f400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9290076110 0x7f929019e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57142/0 (socket says 172.21.15.43:57142) 2024-03-28T22:33:25.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.444+0000 7f928f400640 1 -- 172.21.15.43:0/155241828 learned_addr learned my addr 172.21.15.43:0/155241828 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:25.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.444+0000 7f928ea00640 1 --2- 172.21.15.43:0/155241828 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9290077350 0x7f929019e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:25.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.445+0000 7f928fe00640 1 --2- 172.21.15.43:0/155241828 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9290077cf0 0x7f92901a5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:25.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.445+0000 7f928f400640 1 -- 172.21.15.43:0/155241828 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9290077cf0 msgr2=0x7f92901a5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:25.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.445+0000 7f928f400640 1 --2- 172.21.15.43:0/155241828 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9290077cf0 0x7f92901a5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:25.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.445+0000 7f928f400640 1 -- 172.21.15.43:0/155241828 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9290077350 msgr2=0x7f929019e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:25.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.445+0000 7f928f400640 1 --2- 172.21.15.43:0/155241828 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9290077350 0x7f929019e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:25.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.445+0000 7f928f400640 1 -- 172.21.15.43:0/155241828 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92901a5cb0 con 0x7f9290076110 2024-03-28T22:33:25.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.445+0000 7f928fe00640 1 --2- 172.21.15.43:0/155241828 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9290077cf0 0x7f92901a5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:33:25.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.445+0000 7f928ea00640 1 --2- 172.21.15.43:0/155241828 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9290077350 0x7f929019e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:33:25.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.445+0000 7f928f400640 1 --2- 172.21.15.43:0/155241828 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9290076110 0x7f929019e090 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f928000e9f0 tx=0x7f928000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:25.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.445+0000 7f9287e00640 1 -- 172.21.15.43:0/155241828 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928000cd80 con 0x7f9290076110 2024-03-28T22:33:25.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.445+0000 7f9296400640 1 -- 172.21.15.43:0/155241828 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92901a5fa0 con 0x7f9290076110 2024-03-28T22:33:25.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.446+0000 7f9287e00640 1 -- 172.21.15.43:0/155241828 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9280010910 con 0x7f9290076110 2024-03-28T22:33:25.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.446+0000 7f9287e00640 1 -- 172.21.15.43:0/155241828 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92800188f0 con 0x7f9290076110 2024-03-28T22:33:25.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.446+0000 7f9296400640 1 -- 172.21.15.43:0/155241828 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92901a6410 con 0x7f9290076110 2024-03-28T22:33:25.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.447+0000 7f9296400640 1 -- 172.21.15.43:0/155241828 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9290110300 con 0x7f9290076110 2024-03-28T22:33:25.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.449+0000 7f9287e00640 1 -- 172.21.15.43:0/155241828 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f9280010ac0 con 0x7f9290076110 2024-03-28T22:33:25.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.450+0000 7f9287e00640 1 --2- 172.21.15.43:0/155241828 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f92740a1d80 0x7f92740a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:25.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.451+0000 7f9287e00640 1 -- 172.21.15.43:0/155241828 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f92800cdff0 con 0x7f9290076110 2024-03-28T22:33:25.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.451+0000 7f928ea00640 1 --2- 172.21.15.43:0/155241828 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f92740a1d80 0x7f92740a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:25.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.451+0000 7f928ea00640 1 --2- 172.21.15.43:0/155241828 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f92740a1d80 0x7f92740a4240 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f927c02c240 tx=0x7f927c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:25.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.452+0000 7f9287e00640 1 -- 172.21.15.43:0/155241828 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f928009b880 con 0x7f9290076110 2024-03-28T22:33:25.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:25 smithi172 ceph-mon[36566]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:25.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:25 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1577033627' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:25.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:25 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1130165710' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:33:25.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:25 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:25.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:25 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:25.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.669+0000 7f9296400640 1 -- 172.21.15.43:0/155241828 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f929002d390 con 0x7f9290076110 2024-03-28T22:33:25.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.670+0000 7f9287e00640 1 -- 172.21.15.43:0/155241828 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7f928009b180 con 0x7f9290076110 2024-03-28T22:33:25.668 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-03-28T22:33:25.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.672+0000 7f9296400640 1 -- 172.21.15.43:0/155241828 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f92740a1d80 msgr2=0x7f92740a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:25.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.672+0000 7f9296400640 1 --2- 172.21.15.43:0/155241828 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f92740a1d80 0x7f92740a4240 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f927c02c240 tx=0x7f927c02df90 comp rx=0 tx=0).stop 2024-03-28T22:33:25.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.672+0000 7f9296400640 1 -- 172.21.15.43:0/155241828 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9290076110 msgr2=0x7f929019e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:25.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.672+0000 7f9296400640 1 --2- 172.21.15.43:0/155241828 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9290076110 0x7f929019e090 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f928000e9f0 tx=0x7f928000eeb0 comp rx=0 tx=0).stop 2024-03-28T22:33:25.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.672+0000 7f9296400640 1 -- 172.21.15.43:0/155241828 shutdown_connections 2024-03-28T22:33:25.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.672+0000 7f9296400640 1 --2- 172.21.15.43:0/155241828 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f92740a1d80 0x7f92740a4240 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:25.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.672+0000 7f9296400640 1 --2- 172.21.15.43:0/155241828 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9290077cf0 0x7f92901a5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:25.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.672+0000 7f9296400640 1 --2- 172.21.15.43:0/155241828 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9290077350 0x7f929019e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:25.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.672+0000 7f9296400640 1 --2- 172.21.15.43:0/155241828 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9290076110 0x7f929019e090 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:25.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.672+0000 7f9296400640 1 -- 172.21.15.43:0/155241828 >> 172.21.15.43:0/155241828 conn(0x7f92900fef00 msgr2=0x7f9290100a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:25.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.672+0000 7f9296400640 1 -- 172.21.15.43:0/155241828 shutdown_connections 2024-03-28T22:33:25.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.672+0000 7f9296400640 1 -- 172.21.15.43:0/155241828 wait complete. 2024-03-28T22:33:25.689 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:25 smithi043 ceph-mon[31823]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:25.689 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:25 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1577033627' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:25.689 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:25 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1130165710' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:33:25.689 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:25 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:25.689 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:25 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:25.691 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-28T22:33:25.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:25 smithi082 ceph-mon[36471]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:25.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:25 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1577033627' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:25.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:25 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1130165710' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:33:25.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:25 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:25.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:25 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:25.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.886+0000 7f8483e00640 1 -- 172.21.15.43:0/1999966629 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8484074130 msgr2=0x7f8484074590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:25.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.886+0000 7f8483e00640 1 --2- 172.21.15.43:0/1999966629 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8484074130 0x7f8484074590 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8470009860 tx=0x7f847002bc00 comp rx=0 tx=0).stop 2024-03-28T22:33:25.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.886+0000 7f8483e00640 1 -- 172.21.15.43:0/1999966629 shutdown_connections 2024-03-28T22:33:25.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.886+0000 7f8483e00640 1 --2- 172.21.15.43:0/1999966629 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8484074130 0x7f8484074590 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:25.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.886+0000 7f8483e00640 1 --2- 172.21.15.43:0/1999966629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8484109460 0x7f848410b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:25.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.886+0000 7f8483e00640 1 --2- 172.21.15.43:0/1999966629 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8484075710 0x7f8484073b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:25.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.886+0000 7f8483e00640 1 -- 172.21.15.43:0/1999966629 >> 172.21.15.43:0/1999966629 conn(0x7f84840fcb20 msgr2=0x7f84840fef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:25.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.886+0000 7f8483e00640 1 -- 172.21.15.43:0/1999966629 shutdown_connections 2024-03-28T22:33:25.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.886+0000 7f8483e00640 1 -- 172.21.15.43:0/1999966629 wait complete. 2024-03-28T22:33:25.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.887+0000 7f8483e00640 1 Processor -- start 2024-03-28T22:33:25.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.887+0000 7f8483e00640 1 -- start start 2024-03-28T22:33:25.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.888+0000 7f8483e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8484074130 0x7f8484199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:25.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.888+0000 7f8483e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8484075710 0x7f8484199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:25.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.888+0000 7f8483e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8484109460 0x7f84841a1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:25.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.888+0000 7f8483e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f848410f430 con 0x7f8484074130 2024-03-28T22:33:25.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.888+0000 7f8483e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f848410f2b0 con 0x7f8484109460 2024-03-28T22:33:25.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.888+0000 7f8483e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f848410f5b0 con 0x7f8484075710 2024-03-28T22:33:25.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.888+0000 7f8482a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8484074130 0x7f8484199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:25.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.888+0000 7f8482a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8484074130 0x7f8484199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44030/0 (socket says 172.21.15.43:44030) 2024-03-28T22:33:25.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.888+0000 7f8482a00640 1 -- 172.21.15.43:0/1183752165 learned_addr learned my addr 172.21.15.43:0/1183752165 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:25.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.888+0000 7f8482000640 1 --2- 172.21.15.43:0/1183752165 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8484075710 0x7f8484199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:25.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.888+0000 7f8483400640 1 --2- 172.21.15.43:0/1183752165 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8484109460 0x7f84841a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:25.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.888+0000 7f8482a00640 1 -- 172.21.15.43:0/1183752165 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8484075710 msgr2=0x7f8484199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:25.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.888+0000 7f8482a00640 1 --2- 172.21.15.43:0/1183752165 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8484075710 0x7f8484199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:25.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.888+0000 7f8482a00640 1 -- 172.21.15.43:0/1183752165 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8484109460 msgr2=0x7f84841a1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:25.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.888+0000 7f8482a00640 1 --2- 172.21.15.43:0/1183752165 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8484109460 0x7f84841a1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:25.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.888+0000 7f8482a00640 1 -- 172.21.15.43:0/1183752165 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84841a16a0 con 0x7f8484074130 2024-03-28T22:33:25.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.888+0000 7f8483400640 1 --2- 172.21.15.43:0/1183752165 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8484109460 0x7f84841a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:33:25.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.889+0000 7f8482a00640 1 --2- 172.21.15.43:0/1183752165 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8484074130 0x7f8484199a80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f847400e9f0 tx=0x7f847400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:25.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.889+0000 7f847b400640 1 -- 172.21.15.43:0/1183752165 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f847400cd80 con 0x7f8484074130 2024-03-28T22:33:25.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.889+0000 7f847b400640 1 -- 172.21.15.43:0/1183752165 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8474005500 con 0x7f8484074130 2024-03-28T22:33:25.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.889+0000 7f8483e00640 1 -- 172.21.15.43:0/1183752165 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84841a1990 con 0x7f8484074130 2024-03-28T22:33:25.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.889+0000 7f847b400640 1 -- 172.21.15.43:0/1183752165 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8474010690 con 0x7f8484074130 2024-03-28T22:33:25.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.889+0000 7f8483e00640 1 -- 172.21.15.43:0/1183752165 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84841a1e60 con 0x7f8484074130 2024-03-28T22:33:25.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.890+0000 7f8483e00640 1 -- 172.21.15.43:0/1183752165 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8484108220 con 0x7f8484074130 2024-03-28T22:33:25.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.892+0000 7f847b400640 1 -- 172.21.15.43:0/1183752165 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f84740046c0 con 0x7f8484074130 2024-03-28T22:33:25.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.894+0000 7f847b400640 1 --2- 172.21.15.43:0/1183752165 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f84680a1d80 0x7f84680a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:25.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.894+0000 7f847b400640 1 -- 172.21.15.43:0/1183752165 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f84740cde50 con 0x7f8484074130 2024-03-28T22:33:25.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.895+0000 7f8482000640 1 --2- 172.21.15.43:0/1183752165 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f84680a1d80 0x7f84680a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:25.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.895+0000 7f8482000640 1 --2- 172.21.15.43:0/1183752165 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f84680a1d80 0x7f84680a4240 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8470009830 tx=0x7f8470034b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:25.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:25.899+0000 7f847b400640 1 -- 172.21.15.43:0/1183752165 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f847409b680 con 0x7f8484074130 2024-03-28T22:33:26.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.105+0000 7f8483e00640 1 -- 172.21.15.43:0/1183752165 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f84840623f0 con 0x7f8484074130 2024-03-28T22:33:26.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.105+0000 7f847b400640 1 -- 172.21.15.43:0/1183752165 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v83) ==== 84+0+65 (secure 0 0 0) 0x7f847409af80 con 0x7f8484074130 2024-03-28T22:33:26.104 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:33:26.104 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-03-28T22:33:26.104 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:33:26.104 INFO:teuthology.orchestra.run.smithi043.stdout: "poolnum": 1, 2024-03-28T22:33:26.104 INFO:teuthology.orchestra.run.smithi043.stdout: "poolname": ".mgr" 2024-03-28T22:33:26.104 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:33:26.104 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-03-28T22:33:26.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.108+0000 7f8483e00640 1 -- 172.21.15.43:0/1183752165 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f84680a1d80 msgr2=0x7f84680a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:26.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.108+0000 7f8483e00640 1 --2- 172.21.15.43:0/1183752165 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f84680a1d80 0x7f84680a4240 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8470009830 tx=0x7f8470034b50 comp rx=0 tx=0).stop 2024-03-28T22:33:26.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.108+0000 7f8483e00640 1 -- 172.21.15.43:0/1183752165 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8484074130 msgr2=0x7f8484199a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:26.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.108+0000 7f8483e00640 1 --2- 172.21.15.43:0/1183752165 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8484074130 0x7f8484199a80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f847400e9f0 tx=0x7f847400edf0 comp rx=0 tx=0).stop 2024-03-28T22:33:26.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.108+0000 7f8483e00640 1 -- 172.21.15.43:0/1183752165 shutdown_connections 2024-03-28T22:33:26.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.108+0000 7f8483e00640 1 --2- 172.21.15.43:0/1183752165 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f84680a1d80 0x7f84680a4240 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:26.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.108+0000 7f8483e00640 1 --2- 172.21.15.43:0/1183752165 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8484109460 0x7f84841a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:26.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.108+0000 7f8483e00640 1 --2- 172.21.15.43:0/1183752165 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8484075710 0x7f8484199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:26.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.108+0000 7f8483e00640 1 --2- 172.21.15.43:0/1183752165 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8484074130 0x7f8484199a80 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:26.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.108+0000 7f8483e00640 1 -- 172.21.15.43:0/1183752165 >> 172.21.15.43:0/1183752165 conn(0x7f84840fcb20 msgr2=0x7f84841053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:26.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.108+0000 7f8483e00640 1 -- 172.21.15.43:0/1183752165 shutdown_connections 2024-03-28T22:33:26.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.108+0000 7f8483e00640 1 -- 172.21.15.43:0/1183752165 wait complete. 2024-03-28T22:33:26.124 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-28T22:33:26.124 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-28T22:33:26.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.365+0000 7fe446200640 1 -- 172.21.15.43:0/3520800296 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe440077350 msgr2=0x7fe4400777b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:26.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.365+0000 7fe446200640 1 --2- 172.21.15.43:0/3520800296 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe440077350 0x7fe4400777b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe42c009860 tx=0x7fe42c02bb70 comp rx=0 tx=0).stop 2024-03-28T22:33:26.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.365+0000 7fe446200640 1 -- 172.21.15.43:0/3520800296 shutdown_connections 2024-03-28T22:33:26.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.365+0000 7fe446200640 1 --2- 172.21.15.43:0/3520800296 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe440077cf0 0x7fe440078590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:26.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.365+0000 7fe446200640 1 --2- 172.21.15.43:0/3520800296 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe440077350 0x7fe4400777b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:26.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.365+0000 7fe446200640 1 --2- 172.21.15.43:0/3520800296 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe440076110 0x7fe440076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:26.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.365+0000 7fe446200640 1 -- 172.21.15.43:0/3520800296 >> 172.21.15.43:0/3520800296 conn(0x7fe4400fef00 msgr2=0x7fe440101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:26.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.365+0000 7fe446200640 1 -- 172.21.15.43:0/3520800296 shutdown_connections 2024-03-28T22:33:26.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.365+0000 7fe446200640 1 -- 172.21.15.43:0/3520800296 wait complete. 2024-03-28T22:33:26.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.366+0000 7fe446200640 1 Processor -- start 2024-03-28T22:33:26.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.367+0000 7fe446200640 1 -- start start 2024-03-28T22:33:26.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.367+0000 7fe446200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe440076110 0x7fe44019e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:26.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.367+0000 7fe446200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe440077350 0x7fe44019e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:26.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.367+0000 7fe446200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe440077cf0 0x7fe4401a5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:26.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.367+0000 7fe446200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe440113a00 con 0x7fe440076110 2024-03-28T22:33:26.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.367+0000 7fe446200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe440113880 con 0x7fe440077cf0 2024-03-28T22:33:26.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.367+0000 7fe446200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe440113b80 con 0x7fe440077350 2024-03-28T22:33:26.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.367+0000 7fe43f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe440076110 0x7fe44019e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:26.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.368+0000 7fe43ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe440077350 0x7fe44019e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:26.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.368+0000 7fe43f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe440076110 0x7fe44019e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44044/0 (socket says 172.21.15.43:44044) 2024-03-28T22:33:26.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.368+0000 7fe43f400640 1 -- 172.21.15.43:0/2192638944 learned_addr learned my addr 172.21.15.43:0/2192638944 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:26.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.368+0000 7fe43fe00640 1 --2- 172.21.15.43:0/2192638944 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe440077cf0 0x7fe4401a5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:26.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.368+0000 7fe43ea00640 1 -- 172.21.15.43:0/2192638944 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe440077cf0 msgr2=0x7fe4401a5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:26.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.368+0000 7fe43ea00640 1 --2- 172.21.15.43:0/2192638944 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe440077cf0 0x7fe4401a5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:26.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.368+0000 7fe43ea00640 1 -- 172.21.15.43:0/2192638944 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe440076110 msgr2=0x7fe44019e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:26.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.368+0000 7fe43ea00640 1 --2- 172.21.15.43:0/2192638944 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe440076110 0x7fe44019e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:26.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.368+0000 7fe43ea00640 1 -- 172.21.15.43:0/2192638944 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4401a5cc0 con 0x7fe440077350 2024-03-28T22:33:26.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.368+0000 7fe43f400640 1 --2- 172.21.15.43:0/2192638944 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe440076110 0x7fe44019e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:33:26.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.368+0000 7fe43ea00640 1 --2- 172.21.15.43:0/2192638944 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe440077350 0x7fe44019e5e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe42c02c0b0 tx=0x7fe42c032430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:26.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.371+0000 7fe437e00640 1 -- 172.21.15.43:0/2192638944 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42c041020 con 0x7fe440077350 2024-03-28T22:33:26.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.371+0000 7fe437e00640 1 -- 172.21.15.43:0/2192638944 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe42c02c9f0 con 0x7fe440077350 2024-03-28T22:33:26.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.371+0000 7fe437e00640 1 -- 172.21.15.43:0/2192638944 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42c039040 con 0x7fe440077350 2024-03-28T22:33:26.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.371+0000 7fe446200640 1 -- 172.21.15.43:0/2192638944 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4401a5f50 con 0x7fe440077350 2024-03-28T22:33:26.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.371+0000 7fe446200640 1 -- 172.21.15.43:0/2192638944 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4401a63c0 con 0x7fe440077350 2024-03-28T22:33:26.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.373+0000 7fe446200640 1 -- 172.21.15.43:0/2192638944 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4401102e0 con 0x7fe440077350 2024-03-28T22:33:26.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.375+0000 7fe437e00640 1 -- 172.21.15.43:0/2192638944 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fe42c044020 con 0x7fe440077350 2024-03-28T22:33:26.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.378+0000 7fe437e00640 1 --2- 172.21.15.43:0/2192638944 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe4240a1e00 0x7fe4240a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:26.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.378+0000 7fe437e00640 1 -- 172.21.15.43:0/2192638944 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fe42c0ef130 con 0x7fe440077350 2024-03-28T22:33:26.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.378+0000 7fe43f400640 1 --2- 172.21.15.43:0/2192638944 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe4240a1e00 0x7fe4240a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:26.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.378+0000 7fe43f400640 1 --2- 172.21.15.43:0/2192638944 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe4240a1e00 0x7fe4240a42c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe4300096c0 tx=0x7fe430009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:26.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.382+0000 7fe437e00640 1 -- 172.21.15.43:0/2192638944 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe42c04b020 con 0x7fe440077350 2024-03-28T22:33:26.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:26.611+0000 7fe446200640 1 -- 172.21.15.43:0/2192638944 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fe44002d170 con 0x7fe440077350 2024-03-28T22:33:26.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:26 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:26.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:26 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:26.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:26 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/155241828' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:33:26.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:26 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1183752165' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T22:33:26.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:26 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:26.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:26 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:26.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:26 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:26.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:26 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/155241828' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:33:26.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:26 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1183752165' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T22:33:26.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:26 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:26.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:26 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:26.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:26 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:26.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:26 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/155241828' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:33:26.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:26 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1183752165' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T22:33:26.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:26 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:27.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.404+0000 7fe437e00640 1 -- 172.21.15.43:0/2192638944 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v84) ==== 138+0+0 (secure 0 0 0) 0x7fe42c0bc960 con 0x7fe440077350 2024-03-28T22:33:27.404 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'cephfs_metadata' created 2024-03-28T22:33:27.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.411+0000 7fe446200640 1 -- 172.21.15.43:0/2192638944 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe4240a1e00 msgr2=0x7fe4240a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:27.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.411+0000 7fe446200640 1 --2- 172.21.15.43:0/2192638944 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe4240a1e00 0x7fe4240a42c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe4300096c0 tx=0x7fe430009260 comp rx=0 tx=0).stop 2024-03-28T22:33:27.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.412+0000 7fe446200640 1 -- 172.21.15.43:0/2192638944 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe440077350 msgr2=0x7fe44019e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:27.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.412+0000 7fe446200640 1 --2- 172.21.15.43:0/2192638944 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe440077350 0x7fe44019e5e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe42c02c0b0 tx=0x7fe42c032430 comp rx=0 tx=0).stop 2024-03-28T22:33:27.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.412+0000 7fe446200640 1 -- 172.21.15.43:0/2192638944 shutdown_connections 2024-03-28T22:33:27.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.412+0000 7fe446200640 1 --2- 172.21.15.43:0/2192638944 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe4240a1e00 0x7fe4240a42c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:27.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.412+0000 7fe446200640 1 --2- 172.21.15.43:0/2192638944 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe440077cf0 0x7fe4401a5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:27.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.412+0000 7fe446200640 1 --2- 172.21.15.43:0/2192638944 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe440077350 0x7fe44019e5e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:27.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.412+0000 7fe446200640 1 --2- 172.21.15.43:0/2192638944 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe440076110 0x7fe44019e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:27.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.412+0000 7fe446200640 1 -- 172.21.15.43:0/2192638944 >> 172.21.15.43:0/2192638944 conn(0x7fe4400fef00 msgr2=0x7fe440100a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:27.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.413+0000 7fe446200640 1 -- 172.21.15.43:0/2192638944 shutdown_connections 2024-03-28T22:33:27.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.413+0000 7fe446200640 1 -- 172.21.15.43:0/2192638944 wait complete. 2024-03-28T22:33:27.432 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-28T22:33:27.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.641+0000 7ff56d600640 1 -- 172.21.15.43:0/562389866 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff568100650 msgr2=0x7ff568100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:27.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.641+0000 7ff56d600640 1 --2- 172.21.15.43:0/562389866 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff568100650 0x7ff568100a70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff558009900 tx=0x7ff55802bc30 comp rx=0 tx=0).stop 2024-03-28T22:33:27.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.641+0000 7ff56d600640 1 -- 172.21.15.43:0/562389866 shutdown_connections 2024-03-28T22:33:27.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.641+0000 7ff56d600640 1 --2- 172.21.15.43:0/562389866 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff568102200 0x7ff568108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:27.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.641+0000 7ff56d600640 1 --2- 172.21.15.43:0/562389866 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff568101840 0x7ff568101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:27.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.641+0000 7ff56d600640 1 --2- 172.21.15.43:0/562389866 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff568100650 0x7ff568100a70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:27.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.641+0000 7ff56d600640 1 -- 172.21.15.43:0/562389866 >> 172.21.15.43:0/562389866 conn(0x7ff5680fbbe0 msgr2=0x7ff5680fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:27.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.641+0000 7ff56d600640 1 -- 172.21.15.43:0/562389866 shutdown_connections 2024-03-28T22:33:27.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.641+0000 7ff56d600640 1 -- 172.21.15.43:0/562389866 wait complete. 2024-03-28T22:33:27.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.642+0000 7ff56d600640 1 Processor -- start 2024-03-28T22:33:27.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.642+0000 7ff56d600640 1 -- start start 2024-03-28T22:33:27.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.643+0000 7ff56d600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff568100650 0x7ff568199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:27.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.643+0000 7ff56d600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff568101840 0x7ff56819a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:27.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.643+0000 7ff56d600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff568102200 0x7ff5681a1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:27.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.643+0000 7ff56d600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff568068d80 con 0x7ff568100650 2024-03-28T22:33:27.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.643+0000 7ff56d600640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ff568068c00 con 0x7ff568101840 2024-03-28T22:33:27.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.643+0000 7ff56d600640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7ff568068f00 con 0x7ff568102200 2024-03-28T22:33:27.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.643+0000 7ff566a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff568100650 0x7ff568199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:27.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.643+0000 7ff566000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff568101840 0x7ff56819a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:27.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.643+0000 7ff566a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff568100650 0x7ff568199d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44056/0 (socket says 172.21.15.43:44056) 2024-03-28T22:33:27.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.643+0000 7ff566a00640 1 -- 172.21.15.43:0/3406967827 learned_addr learned my addr 172.21.15.43:0/3406967827 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:27.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.643+0000 7ff567400640 1 --2- 172.21.15.43:0/3406967827 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff568102200 0x7ff5681a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:27.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.643+0000 7ff566000640 1 -- 172.21.15.43:0/3406967827 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff568102200 msgr2=0x7ff5681a1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:27.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.643+0000 7ff566000640 1 --2- 172.21.15.43:0/3406967827 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff568102200 0x7ff5681a1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:27.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.643+0000 7ff566000640 1 -- 172.21.15.43:0/3406967827 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff568100650 msgr2=0x7ff568199d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:27.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.643+0000 7ff566000640 1 --2- 172.21.15.43:0/3406967827 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff568100650 0x7ff568199d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:27.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.644+0000 7ff566000640 1 -- 172.21.15.43:0/3406967827 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5681a1910 con 0x7ff568101840 2024-03-28T22:33:27.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.644+0000 7ff566000640 1 --2- 172.21.15.43:0/3406967827 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff568101840 0x7ff56819a240 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff55400e9f0 tx=0x7ff55400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:27.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.644+0000 7ff55f400640 1 -- 172.21.15.43:0/3406967827 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff55400ccc0 con 0x7ff568101840 2024-03-28T22:33:27.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.644+0000 7ff55f400640 1 -- 172.21.15.43:0/3406967827 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff554010910 con 0x7ff568101840 2024-03-28T22:33:27.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.644+0000 7ff56d600640 1 -- 172.21.15.43:0/3406967827 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5681a1c00 con 0x7ff568101840 2024-03-28T22:33:27.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.644+0000 7ff55f400640 1 -- 172.21.15.43:0/3406967827 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff554018960 con 0x7ff568101840 2024-03-28T22:33:27.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.644+0000 7ff56d600640 1 -- 172.21.15.43:0/3406967827 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5681a2150 con 0x7ff568101840 2024-03-28T22:33:27.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.645+0000 7ff56d600640 1 -- 172.21.15.43:0/3406967827 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5681a2320 con 0x7ff568101840 2024-03-28T22:33:27.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.647+0000 7ff55f400640 1 -- 172.21.15.43:0/3406967827 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7ff554010ac0 con 0x7ff568101840 2024-03-28T22:33:27.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:27 smithi172 ceph-mon[36566]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:27.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:27 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:27.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:27 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:33:27.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:27 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:33:27.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:27 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:27.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:27 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2192638944' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T22:33:27.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:27 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T22:33:27.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:27 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:33:27.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.648+0000 7ff55f400640 1 --2- 172.21.15.43:0/3406967827 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff54c0a1d80 0x7ff54c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:27.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.649+0000 7ff55f400640 1 -- 172.21.15.43:0/3406967827 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7497+0+0 (secure 0 0 0) 0x7ff5540cdf30 con 0x7ff568101840 2024-03-28T22:33:27.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.649+0000 7ff566a00640 1 --2- 172.21.15.43:0/3406967827 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff54c0a1d80 0x7ff54c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:27.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.650+0000 7ff566a00640 1 --2- 172.21.15.43:0/3406967827 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff54c0a1d80 0x7ff54c0a4240 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff56819b1d0 tx=0x7ff55802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:27.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.653+0000 7ff55f400640 1 -- 172.21.15.43:0/3406967827 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff55409afa0 con 0x7ff568101840 2024-03-28T22:33:27.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:27 smithi082 ceph-mon[36471]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:27.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:27 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:27.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:27 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:33:27.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:27 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:33:27.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:27 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:27.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:27 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2192638944' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T22:33:27.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:27 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T22:33:27.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:27 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:33:27.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:27 smithi043 ceph-mon[31823]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:27.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:27 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:27.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:27 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:33:27.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:27 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:33:27.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:27 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:27.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:27 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2192638944' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T22:33:27.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:27 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T22:33:27.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:27 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:33:27.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:27.858+0000 7ff56d600640 1 -- 172.21.15.43:0/3406967827 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7ff5681a2590 con 0x7ff568101840 2024-03-28T22:33:28.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.424+0000 7ff55f400640 1 -- 172.21.15.43:0/3406967827 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v85) ==== 171+0+0 (secure 0 0 0) 0x7ff55409f040 con 0x7ff568101840 2024-03-28T22:33:28.422 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'cephfs_data' created 2024-03-28T22:33:28.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.435+0000 7ff56d600640 1 -- 172.21.15.43:0/3406967827 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff54c0a1d80 msgr2=0x7ff54c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:28.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.435+0000 7ff56d600640 1 --2- 172.21.15.43:0/3406967827 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff54c0a1d80 0x7ff54c0a4240 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff56819b1d0 tx=0x7ff55802df90 comp rx=0 tx=0).stop 2024-03-28T22:33:28.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.435+0000 7ff56d600640 1 -- 172.21.15.43:0/3406967827 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff568101840 msgr2=0x7ff56819a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:28.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.435+0000 7ff56d600640 1 --2- 172.21.15.43:0/3406967827 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff568101840 0x7ff56819a240 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff55400e9f0 tx=0x7ff55400eeb0 comp rx=0 tx=0).stop 2024-03-28T22:33:28.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.435+0000 7ff56d600640 1 -- 172.21.15.43:0/3406967827 shutdown_connections 2024-03-28T22:33:28.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.435+0000 7ff56d600640 1 --2- 172.21.15.43:0/3406967827 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff54c0a1d80 0x7ff54c0a4240 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:28.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.435+0000 7ff56d600640 1 --2- 172.21.15.43:0/3406967827 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff568101840 0x7ff56819a240 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:28.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.435+0000 7ff56d600640 1 --2- 172.21.15.43:0/3406967827 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff568102200 0x7ff5681a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:28.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.435+0000 7ff56d600640 1 --2- 172.21.15.43:0/3406967827 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff568100650 0x7ff568199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:28.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.435+0000 7ff56d600640 1 -- 172.21.15.43:0/3406967827 >> 172.21.15.43:0/3406967827 conn(0x7ff5680fbbe0 msgr2=0x7ff56810be20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:28.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.437+0000 7ff56d600640 1 -- 172.21.15.43:0/3406967827 shutdown_connections 2024-03-28T22:33:28.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.437+0000 7ff56d600640 1 -- 172.21.15.43:0/3406967827 wait complete. 2024-03-28T22:33:28.463 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-28T22:33:28.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.644+0000 7f5dd0a00640 1 -- 172.21.15.43:0/2710471505 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5dcc103990 msgr2=0x7f5dcc103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:28.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.644+0000 7f5dd0a00640 1 --2- 172.21.15.43:0/2710471505 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5dcc103990 0x7f5dcc103db0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc009ca0 tx=0x7f5dbc02c100 comp rx=0 tx=0).stop 2024-03-28T22:33:28.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.645+0000 7f5dd0a00640 1 -- 172.21.15.43:0/2710471505 shutdown_connections 2024-03-28T22:33:28.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.645+0000 7f5dd0a00640 1 --2- 172.21.15.43:0/2710471505 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dcc105540 0x7f5dcc10bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:28.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.645+0000 7f5dd0a00640 1 --2- 172.21.15.43:0/2710471505 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dcc104b80 0x7f5dcc105000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:28.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.645+0000 7f5dd0a00640 1 --2- 172.21.15.43:0/2710471505 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5dcc103990 0x7f5dcc103db0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:28.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.645+0000 7f5dd0a00640 1 -- 172.21.15.43:0/2710471505 >> 172.21.15.43:0/2710471505 conn(0x7f5dcc0fef00 msgr2=0x7f5dcc101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:28.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.645+0000 7f5dd0a00640 1 -- 172.21.15.43:0/2710471505 shutdown_connections 2024-03-28T22:33:28.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.645+0000 7f5dd0a00640 1 -- 172.21.15.43:0/2710471505 wait complete. 2024-03-28T22:33:28.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.646+0000 7f5dd0a00640 1 Processor -- start 2024-03-28T22:33:28.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.646+0000 7f5dd0a00640 1 -- start start 2024-03-28T22:33:28.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.646+0000 7f5dd0a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5dcc103990 0x7f5dcc199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:28.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.646+0000 7f5dd0a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dcc104b80 0x7f5dcc19a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:28.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.646+0000 7f5dd0a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dcc105540 0x7f5dcc1a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:28.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.646+0000 7f5dd0a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5dcc10f610 con 0x7f5dcc105540 2024-03-28T22:33:28.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.646+0000 7f5dd0a00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5dcc10f490 con 0x7f5dcc103990 2024-03-28T22:33:28.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.646+0000 7f5dd0a00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f5dcc10f790 con 0x7f5dcc104b80 2024-03-28T22:33:28.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.647+0000 7f5dc9e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5dcc103990 0x7f5dcc199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:28.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.647+0000 7f5dc9e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5dcc103990 0x7f5dcc199cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:42910/0 (socket says 172.21.15.43:42910) 2024-03-28T22:33:28.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.647+0000 7f5dc9e00640 1 -- 172.21.15.43:0/3225309527 learned_addr learned my addr 172.21.15.43:0/3225309527 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:28.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.647+0000 7f5dc1400640 1 --2- 172.21.15.43:0/3225309527 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dcc104b80 0x7f5dcc19a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:28.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.647+0000 7f5dca800640 1 --2- 172.21.15.43:0/3225309527 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dcc105540 0x7f5dcc1a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:28.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.647+0000 7f5dc9e00640 1 -- 172.21.15.43:0/3225309527 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dcc104b80 msgr2=0x7f5dcc19a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:28.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.647+0000 7f5dc9e00640 1 --2- 172.21.15.43:0/3225309527 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dcc104b80 0x7f5dcc19a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:28.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.647+0000 7f5dc9e00640 1 -- 172.21.15.43:0/3225309527 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dcc105540 msgr2=0x7f5dcc1a1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:28.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.647+0000 7f5dc9e00640 1 --2- 172.21.15.43:0/3225309527 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dcc105540 0x7f5dcc1a1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:28.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.647+0000 7f5dc9e00640 1 -- 172.21.15.43:0/3225309527 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dcc1a18f0 con 0x7f5dcc103990 2024-03-28T22:33:28.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.647+0000 7f5dc9e00640 1 --2- 172.21.15.43:0/3225309527 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5dcc103990 0x7f5dcc199cb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc0324c0 tx=0x7f5dbc035d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:28.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.648+0000 7f5dc3400640 1 -- 172.21.15.43:0/3225309527 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc039040 con 0x7f5dcc103990 2024-03-28T22:33:28.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.648+0000 7f5dc3400640 1 -- 172.21.15.43:0/3225309527 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5dbc044070 con 0x7f5dcc103990 2024-03-28T22:33:28.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.648+0000 7f5dd0a00640 1 -- 172.21.15.43:0/3225309527 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dcc1a1b80 con 0x7f5dcc103990 2024-03-28T22:33:28.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.648+0000 7f5dc3400640 1 -- 172.21.15.43:0/3225309527 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc0044a0 con 0x7f5dcc103990 2024-03-28T22:33:28.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.648+0000 7f5dd0a00640 1 -- 172.21.15.43:0/3225309527 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dcc1a20c0 con 0x7f5dcc103990 2024-03-28T22:33:28.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:28 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T22:33:28.649 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:28 smithi172 ceph-mon[36566]: osdmap e84: 12 total, 12 up, 12 in 2024-03-28T22:33:28.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.649+0000 7f5dd0a00640 1 -- 172.21.15.43:0/3225309527 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dcc10acb0 con 0x7f5dcc103990 2024-03-28T22:33:28.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.651+0000 7f5dc3400640 1 -- 172.21.15.43:0/3225309527 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f5dbc004640 con 0x7f5dcc103990 2024-03-28T22:33:28.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.652+0000 7f5dc3400640 1 --2- 172.21.15.43:0/3225309527 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5db00a1e00 0x7f5db00a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:28.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.652+0000 7f5dc3400640 1 -- 172.21.15.43:0/3225309527 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7879+0+0 (secure 0 0 0) 0x7f5dbc0f01f0 con 0x7f5dcc103990 2024-03-28T22:33:28.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.653+0000 7f5dc1400640 1 --2- 172.21.15.43:0/3225309527 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5db00a1e00 0x7f5db00a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:28.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.653+0000 7f5dc1400640 1 --2- 172.21.15.43:0/3225309527 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5db00a1e00 0x7f5db00a42c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc19b1b0 tx=0x7f5db4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:28.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.656+0000 7f5dc3400640 1 -- 172.21.15.43:0/3225309527 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5dbc0bd730 con 0x7f5dcc103990 2024-03-28T22:33:28.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:28 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T22:33:28.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:28 smithi082 ceph-mon[36471]: osdmap e84: 12 total, 12 up, 12 in 2024-03-28T22:33:28.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:28 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T22:33:28.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:28 smithi043 ceph-mon[31823]: osdmap e84: 12 total, 12 up, 12 in 2024-03-28T22:33:28.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:28.904+0000 7f5dd0a00640 1 -- 172.21.15.43:0/3225309527 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f5dcc02d170 con 0x7f5dcc103990 2024-03-28T22:33:29.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.475+0000 7f5dc3400640 1 -- 172.21.15.43:0/3225309527 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f5dbc0bd030 con 0x7f5dcc103990 2024-03-28T22:33:29.474 INFO:teuthology.orchestra.run.smithi043.stderr:new fs with metadata pool 2 and data pool 3 2024-03-28T22:33:29.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.481+0000 7f5dd0a00640 1 -- 172.21.15.43:0/3225309527 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5db00a1e00 msgr2=0x7f5db00a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:29.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.481+0000 7f5dd0a00640 1 --2- 172.21.15.43:0/3225309527 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5db00a1e00 0x7f5db00a42c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc19b1b0 tx=0x7f5db4005f90 comp rx=0 tx=0).stop 2024-03-28T22:33:29.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.481+0000 7f5dd0a00640 1 -- 172.21.15.43:0/3225309527 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5dcc103990 msgr2=0x7f5dcc199cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:29.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.481+0000 7f5dd0a00640 1 --2- 172.21.15.43:0/3225309527 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5dcc103990 0x7f5dcc199cb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc0324c0 tx=0x7f5dbc035d90 comp rx=0 tx=0).stop 2024-03-28T22:33:29.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.482+0000 7f5dd0a00640 1 -- 172.21.15.43:0/3225309527 shutdown_connections 2024-03-28T22:33:29.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.482+0000 7f5dd0a00640 1 --2- 172.21.15.43:0/3225309527 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dcc105540 0x7f5dcc1a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:29.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.482+0000 7f5dd0a00640 1 --2- 172.21.15.43:0/3225309527 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dcc104b80 0x7f5dcc19a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:29.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.482+0000 7f5dd0a00640 1 --2- 172.21.15.43:0/3225309527 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5db00a1e00 0x7f5db00a42c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:29.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.482+0000 7f5dd0a00640 1 --2- 172.21.15.43:0/3225309527 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5dcc103990 0x7f5dcc199cb0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:29.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.482+0000 7f5dd0a00640 1 -- 172.21.15.43:0/3225309527 >> 172.21.15.43:0/3225309527 conn(0x7f5dcc0fef00 msgr2=0x7f5dcc107e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:29.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.483+0000 7f5dd0a00640 1 -- 172.21.15.43:0/3225309527 shutdown_connections 2024-03-28T22:33:29.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.483+0000 7f5dd0a00640 1 -- 172.21.15.43:0/3225309527 wait complete. 2024-03-28T22:33:29.503 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-28T22:33:29.503 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-28T22:33:29.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:29 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3406967827' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-28T22:33:29.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:29 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-28T22:33:29.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:29 smithi082 ceph-mon[36471]: pgmap v314: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:29.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:29 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-28T22:33:29.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:29 smithi082 ceph-mon[36471]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T22:33:29.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:29 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:29.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:29 smithi082 ceph-mon[36471]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T22:33:29.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:29 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3225309527' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T22:33:29.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:29 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T22:33:29.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:29 smithi082 ceph-mon[36471]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-28T22:33:29.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:29 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:29.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:29 smithi082 ceph-mon[36471]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T22:33:29.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:29 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:29.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:29 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:29.701 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:29 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:29.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.720+0000 7f4a13800640 1 -- 172.21.15.43:0/1203533032 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a0c104b80 msgr2=0x7f4a0c105000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:29.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.720+0000 7f4a13800640 1 --2- 172.21.15.43:0/1203533032 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a0c104b80 0x7f4a0c105000 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f49f8009900 tx=0x7f49f802bc30 comp rx=0 tx=0).stop 2024-03-28T22:33:29.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.721+0000 7f4a13800640 1 -- 172.21.15.43:0/1203533032 shutdown_connections 2024-03-28T22:33:29.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.721+0000 7f4a13800640 1 --2- 172.21.15.43:0/1203533032 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a0c104b80 0x7f4a0c105000 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:29.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.721+0000 7f4a13800640 1 --2- 172.21.15.43:0/1203533032 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a0c105540 0x7f4a0c10bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:29.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.721+0000 7f4a13800640 1 --2- 172.21.15.43:0/1203533032 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a0c103990 0x7f4a0c103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:29.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.721+0000 7f4a13800640 1 -- 172.21.15.43:0/1203533032 >> 172.21.15.43:0/1203533032 conn(0x7f4a0c0fef00 msgr2=0x7f4a0c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:29.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.721+0000 7f4a13800640 1 -- 172.21.15.43:0/1203533032 shutdown_connections 2024-03-28T22:33:29.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.721+0000 7f4a13800640 1 -- 172.21.15.43:0/1203533032 wait complete. 2024-03-28T22:33:29.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.722+0000 7f4a13800640 1 Processor -- start 2024-03-28T22:33:29.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.722+0000 7f4a13800640 1 -- start start 2024-03-28T22:33:29.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.722+0000 7f4a13800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a0c103990 0x7f4a0c199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:29.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.722+0000 7f4a13800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a0c104b80 0x7f4a0c19a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:29.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.722+0000 7f4a13800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a0c105540 0x7f4a0c1a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:29.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.722+0000 7f4a13800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0c10f660 con 0x7f4a0c105540 2024-03-28T22:33:29.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.722+0000 7f4a13800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0c10f4e0 con 0x7f4a0c103990 2024-03-28T22:33:29.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.722+0000 7f4a13800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0c10f7e0 con 0x7f4a0c104b80 2024-03-28T22:33:29.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.722+0000 7f4a0be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a0c104b80 0x7f4a0c19a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:29.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.722+0000 7f4a0be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a0c104b80 0x7f4a0c19a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57252/0 (socket says 172.21.15.43:57252) 2024-03-28T22:33:29.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.722+0000 7f4a0be00640 1 -- 172.21.15.43:0/3422779807 learned_addr learned my addr 172.21.15.43:0/3422779807 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:29.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.722+0000 7f4a10e00640 1 --2- 172.21.15.43:0/3422779807 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a0c103990 0x7f4a0c199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:29.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.722+0000 7f4a11800640 1 --2- 172.21.15.43:0/3422779807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a0c105540 0x7f4a0c1a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:29.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.723+0000 7f4a10e00640 1 -- 172.21.15.43:0/3422779807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a0c104b80 msgr2=0x7f4a0c19a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:29.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.723+0000 7f4a10e00640 1 --2- 172.21.15.43:0/3422779807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a0c104b80 0x7f4a0c19a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:29.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.723+0000 7f4a10e00640 1 -- 172.21.15.43:0/3422779807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a0c105540 msgr2=0x7f4a0c1a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:29.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.723+0000 7f4a10e00640 1 --2- 172.21.15.43:0/3422779807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a0c105540 0x7f4a0c1a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:29.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.723+0000 7f4a10e00640 1 -- 172.21.15.43:0/3422779807 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a0c1a1900 con 0x7f4a0c103990 2024-03-28T22:33:29.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.723+0000 7f4a0be00640 1 --2- 172.21.15.43:0/3422779807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a0c104b80 0x7f4a0c19a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:33:29.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.723+0000 7f4a10e00640 1 --2- 172.21.15.43:0/3422779807 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a0c103990 0x7f4a0c199cf0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4a0000e9c0 tx=0x7f4a0000edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:29.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.723+0000 7f4a09600640 1 -- 172.21.15.43:0/3422779807 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0000ccc0 con 0x7f4a0c103990 2024-03-28T22:33:29.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.723+0000 7f4a09600640 1 -- 172.21.15.43:0/3422779807 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a000035f0 con 0x7f4a0c103990 2024-03-28T22:33:29.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.723+0000 7f4a09600640 1 -- 172.21.15.43:0/3422779807 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0000ce60 con 0x7f4a0c103990 2024-03-28T22:33:29.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.723+0000 7f4a13800640 1 -- 172.21.15.43:0/3422779807 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a0c1a1bf0 con 0x7f4a0c103990 2024-03-28T22:33:29.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.724+0000 7f4a13800640 1 -- 172.21.15.43:0/3422779807 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a0c1a20f0 con 0x7f4a0c103990 2024-03-28T22:33:29.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.725+0000 7f4a13800640 1 -- 172.21.15.43:0/3422779807 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a0c10ad30 con 0x7f4a0c103990 2024-03-28T22:33:29.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.727+0000 7f4a09600640 1 -- 172.21.15.43:0/3422779807 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f4a000037a0 con 0x7f4a0c103990 2024-03-28T22:33:29.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.728+0000 7f4a09600640 1 --2- 172.21.15.43:0/3422779807 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f49f40a1e00 0x7f49f40a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:29.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.729+0000 7f4a09600640 1 -- 172.21.15.43:0/3422779807 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f4a000ce070 con 0x7f4a0c103990 2024-03-28T22:33:29.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.729+0000 7f4a0be00640 1 --2- 172.21.15.43:0/3422779807 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f49f40a1e00 0x7f49f40a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:29.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.730+0000 7f4a0be00640 1 --2- 172.21.15.43:0/3422779807 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f49f40a1e00 0x7f49f40a42c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f49f802c8e0 tx=0x7f49f802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:29.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.730+0000 7f4a09600640 1 -- 172.21.15.43:0/3422779807 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4a0009b550 con 0x7f4a0c103990 2024-03-28T22:33:29.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:29 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3406967827' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-28T22:33:29.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:29 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-28T22:33:29.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:29 smithi043 ceph-mon[31823]: pgmap v314: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:29.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:29 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-28T22:33:29.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:29 smithi043 ceph-mon[31823]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T22:33:29.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:29 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:29.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:29 smithi043 ceph-mon[31823]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T22:33:29.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:29 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3225309527' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T22:33:29.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:29 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T22:33:29.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:29 smithi043 ceph-mon[31823]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-28T22:33:29.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:29 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:29.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:29 smithi043 ceph-mon[31823]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T22:33:29.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:29 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:29.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:29 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:29.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:29 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:29.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:29 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3406967827' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-28T22:33:29.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:29 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-28T22:33:29.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:29 smithi172 ceph-mon[36566]: pgmap v314: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:29.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:29 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-28T22:33:29.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:29 smithi172 ceph-mon[36566]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T22:33:29.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:29 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:29.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:29 smithi172 ceph-mon[36566]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T22:33:29.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:29 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3225309527' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T22:33:29.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:29 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T22:33:29.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:29 smithi172 ceph-mon[36566]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-28T22:33:29.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:29 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:29.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:29 smithi172 ceph-mon[36566]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T22:33:29.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:29 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:29.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:29 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:29.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:29 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:29.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:29.955+0000 7f4a13800640 1 -- 172.21.15.43:0/3422779807 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f4a0c02d3e0 con 0x7f4a0c103990 2024-03-28T22:33:30.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:30 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T22:33:30.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:30 smithi043 ceph-mon[31823]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-28T22:33:30.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:30 smithi043 ceph-mon[31823]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-28T22:33:30.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:30 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:30.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:30 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:30.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:30 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:33:30.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:30 smithi043 ceph-mon[31823]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T22:33:30.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:30 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:33:30.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:30 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:30.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:30 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3422779807' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-28T22:33:30.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:30 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-28T22:33:30.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:30 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T22:33:30.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:30 smithi172 ceph-mon[36566]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-28T22:33:30.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:30 smithi172 ceph-mon[36566]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-28T22:33:30.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:30 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:30.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:30 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:30.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:30 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:33:30.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:30 smithi172 ceph-mon[36566]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T22:33:30.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:30 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:33:30.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:30 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:30.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:30 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3422779807' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-28T22:33:30.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:30 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-28T22:33:30.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:30 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T22:33:30.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:30 smithi082 ceph-mon[36471]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-28T22:33:30.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:30 smithi082 ceph-mon[36471]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-28T22:33:30.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:30 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:30.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:30 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:30.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:30 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:33:30.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:30 smithi082 ceph-mon[36471]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T22:33:30.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:30 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:33:30.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:30 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:33:30.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:30 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3422779807' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-28T22:33:30.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:30 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-28T22:33:30.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:30.955+0000 7f4a09600640 1 -- 172.21.15.43:0/3422779807 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v88) ==== 157+0+0 (secure 0 0 0) 0x7f4a0009ae50 con 0x7f4a0c103990 2024-03-28T22:33:30.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:30.959+0000 7f4a13800640 1 -- 172.21.15.43:0/3422779807 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f49f40a1e00 msgr2=0x7f49f40a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:30.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:30.959+0000 7f4a13800640 1 --2- 172.21.15.43:0/3422779807 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f49f40a1e00 0x7f49f40a42c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f49f802c8e0 tx=0x7f49f802df90 comp rx=0 tx=0).stop 2024-03-28T22:33:30.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:30.959+0000 7f4a13800640 1 -- 172.21.15.43:0/3422779807 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a0c103990 msgr2=0x7f4a0c199cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:30.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:30.959+0000 7f4a13800640 1 --2- 172.21.15.43:0/3422779807 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a0c103990 0x7f4a0c199cf0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4a0000e9c0 tx=0x7f4a0000edc0 comp rx=0 tx=0).stop 2024-03-28T22:33:30.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:30.959+0000 7f4a13800640 1 -- 172.21.15.43:0/3422779807 shutdown_connections 2024-03-28T22:33:30.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:30.959+0000 7f4a13800640 1 --2- 172.21.15.43:0/3422779807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a0c105540 0x7f4a0c1a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:30.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:30.959+0000 7f4a13800640 1 --2- 172.21.15.43:0/3422779807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4a0c104b80 0x7f4a0c19a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:30.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:30.959+0000 7f4a13800640 1 --2- 172.21.15.43:0/3422779807 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f49f40a1e00 0x7f49f40a42c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:30.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:30.959+0000 7f4a13800640 1 --2- 172.21.15.43:0/3422779807 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a0c103990 0x7f4a0c199cf0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:30.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:30.959+0000 7f4a13800640 1 -- 172.21.15.43:0/3422779807 >> 172.21.15.43:0/3422779807 conn(0x7f4a0c0fef00 msgr2=0x7f4a0c107f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:30.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:30.960+0000 7f4a13800640 1 -- 172.21.15.43:0/3422779807 shutdown_connections 2024-03-28T22:33:30.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:30.960+0000 7f4a13800640 1 -- 172.21.15.43:0/3422779807 wait complete. 2024-03-28T22:33:30.982 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-28T22:33:31.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.225+0000 7f5cab000640 1 -- 172.21.15.43:0/3679995576 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ca4104b80 msgr2=0x7f5ca4105000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:31.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.225+0000 7f5cab000640 1 --2- 172.21.15.43:0/3679995576 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ca4104b80 0x7f5ca4105000 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5c94009ca0 tx=0x7f5c9402c100 comp rx=0 tx=0).stop 2024-03-28T22:33:31.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.225+0000 7f5cab000640 1 -- 172.21.15.43:0/3679995576 shutdown_connections 2024-03-28T22:33:31.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.225+0000 7f5cab000640 1 --2- 172.21.15.43:0/3679995576 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ca4104b80 0x7f5ca4105000 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:31.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.225+0000 7f5cab000640 1 --2- 172.21.15.43:0/3679995576 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ca4105540 0x7f5ca410bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:31.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.225+0000 7f5cab000640 1 --2- 172.21.15.43:0/3679995576 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ca4103990 0x7f5ca4103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:31.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.225+0000 7f5cab000640 1 -- 172.21.15.43:0/3679995576 >> 172.21.15.43:0/3679995576 conn(0x7f5ca40fef00 msgr2=0x7f5ca4101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:31.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.225+0000 7f5cab000640 1 -- 172.21.15.43:0/3679995576 shutdown_connections 2024-03-28T22:33:31.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.225+0000 7f5cab000640 1 -- 172.21.15.43:0/3679995576 wait complete. 2024-03-28T22:33:31.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.226+0000 7f5cab000640 1 Processor -- start 2024-03-28T22:33:31.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.227+0000 7f5cab000640 1 -- start start 2024-03-28T22:33:31.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.227+0000 7f5cab000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ca4103990 0x7f5ca4199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:31.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.227+0000 7f5cab000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ca4104b80 0x7f5ca419a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:31.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.227+0000 7f5cab000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ca4105540 0x7f5ca41a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:31.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.227+0000 7f5cab000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca410f610 con 0x7f5ca4105540 2024-03-28T22:33:31.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.227+0000 7f5cab000640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca410f490 con 0x7f5ca4104b80 2024-03-28T22:33:31.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.227+0000 7f5cab000640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca410f790 con 0x7f5ca4103990 2024-03-28T22:33:31.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.227+0000 7f5ca3e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ca4103990 0x7f5ca4199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:31.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.227+0000 7f5ca3400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ca4104b80 0x7f5ca419a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:31.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.227+0000 7f5ca3e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ca4103990 0x7f5ca4199cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57272/0 (socket says 172.21.15.43:57272) 2024-03-28T22:33:31.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.227+0000 7f5ca3e00640 1 -- 172.21.15.43:0/3844192721 learned_addr learned my addr 172.21.15.43:0/3844192721 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:31.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.227+0000 7f5ca9000640 1 --2- 172.21.15.43:0/3844192721 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ca4105540 0x7f5ca41a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:31.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.228+0000 7f5ca3400640 1 -- 172.21.15.43:0/3844192721 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ca4103990 msgr2=0x7f5ca4199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:31.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.228+0000 7f5ca3400640 1 --2- 172.21.15.43:0/3844192721 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ca4103990 0x7f5ca4199cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:31.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.228+0000 7f5ca3400640 1 -- 172.21.15.43:0/3844192721 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ca4105540 msgr2=0x7f5ca41a1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:31.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.228+0000 7f5ca3400640 1 --2- 172.21.15.43:0/3844192721 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ca4105540 0x7f5ca41a1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:31.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.228+0000 7f5ca3400640 1 -- 172.21.15.43:0/3844192721 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ca41a18f0 con 0x7f5ca4104b80 2024-03-28T22:33:31.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.228+0000 7f5ca9000640 1 --2- 172.21.15.43:0/3844192721 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ca4105540 0x7f5ca41a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:33:31.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.228+0000 7f5ca3400640 1 --2- 172.21.15.43:0/3844192721 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ca4104b80 0x7f5ca419a1f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5c94009bf0 tx=0x7f5c94035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:31.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.228+0000 7f5ca0c00640 1 -- 172.21.15.43:0/3844192721 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c94039040 con 0x7f5ca4104b80 2024-03-28T22:33:31.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.228+0000 7f5ca0c00640 1 -- 172.21.15.43:0/3844192721 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c94003970 con 0x7f5ca4104b80 2024-03-28T22:33:31.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.228+0000 7f5ca0c00640 1 -- 172.21.15.43:0/3844192721 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c94004360 con 0x7f5ca4104b80 2024-03-28T22:33:31.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.228+0000 7f5cab000640 1 -- 172.21.15.43:0/3844192721 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ca41a1b80 con 0x7f5ca4104b80 2024-03-28T22:33:31.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.229+0000 7f5cab000640 1 -- 172.21.15.43:0/3844192721 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ca41a2090 con 0x7f5ca4104b80 2024-03-28T22:33:31.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.230+0000 7f5cab000640 1 -- 172.21.15.43:0/3844192721 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ca410ad30 con 0x7f5ca4104b80 2024-03-28T22:33:31.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.232+0000 7f5ca0c00640 1 -- 172.21.15.43:0/3844192721 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f5c94046030 con 0x7f5ca4104b80 2024-03-28T22:33:31.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.233+0000 7f5ca0c00640 1 --2- 172.21.15.43:0/3844192721 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5c880a1e00 0x7f5c880a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:31.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.234+0000 7f5ca0c00640 1 -- 172.21.15.43:0/3844192721 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8277+0+0 (secure 0 0 0) 0x7f5c940efa80 con 0x7f5ca4104b80 2024-03-28T22:33:31.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.234+0000 7f5ca3e00640 1 --2- 172.21.15.43:0/3844192721 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5c880a1e00 0x7f5c880a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:31.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.234+0000 7f5ca3e00640 1 --2- 172.21.15.43:0/3844192721 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5c880a1e00 0x7f5c880a42c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5ca419ab80 tx=0x7f5c8c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:31.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.238+0000 7f5ca0c00640 1 -- 172.21.15.43:0/3844192721 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5c940f1e20 con 0x7f5ca4104b80 2024-03-28T22:33:31.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:31.441+0000 7f5cab000640 1 -- 172.21.15.43:0/3844192721 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f5ca4062350 con 0x7f5ca4104b80 2024-03-28T22:33:31.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:31 smithi043 ceph-mon[31823]: pgmap v318: 97 pgs: 24 active+clean, 22 creating+peering, 51 unknown; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:31.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:31 smithi043 ceph-mon[31823]: mds.? [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] up:active 2024-03-28T22:33:31.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:31 smithi043 ceph-mon[31823]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T22:33:31.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:31 smithi043 ceph-mon[31823]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T22:33:31.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:31 smithi043 ceph-mon[31823]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T22:33:31.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:31 smithi043 ceph-mon[31823]: Cluster is now healthy 2024-03-28T22:33:31.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:31 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-28T22:33:31.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:31 smithi043 ceph-mon[31823]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T22:33:31.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:31 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3844192721' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T22:33:31.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:31 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T22:33:31.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:31 smithi172 ceph-mon[36566]: pgmap v318: 97 pgs: 24 active+clean, 22 creating+peering, 51 unknown; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:31.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:31 smithi172 ceph-mon[36566]: mds.? [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] up:active 2024-03-28T22:33:31.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:31 smithi172 ceph-mon[36566]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T22:33:31.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:31 smithi172 ceph-mon[36566]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T22:33:31.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:31 smithi172 ceph-mon[36566]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T22:33:31.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:31 smithi172 ceph-mon[36566]: Cluster is now healthy 2024-03-28T22:33:31.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:31 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-28T22:33:31.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:31 smithi172 ceph-mon[36566]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T22:33:31.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:31 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3844192721' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T22:33:31.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:31 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T22:33:31.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:31 smithi082 ceph-mon[36471]: pgmap v318: 97 pgs: 24 active+clean, 22 creating+peering, 51 unknown; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:31.981 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:31 smithi082 ceph-mon[36471]: mds.? [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] up:active 2024-03-28T22:33:31.981 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:31 smithi082 ceph-mon[36471]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T22:33:31.981 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:31 smithi082 ceph-mon[36471]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T22:33:31.981 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:31 smithi082 ceph-mon[36471]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T22:33:31.981 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:31 smithi082 ceph-mon[36471]: Cluster is now healthy 2024-03-28T22:33:31.981 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:31 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-28T22:33:31.981 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:31 smithi082 ceph-mon[36471]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T22:33:31.982 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:31 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3844192721' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T22:33:31.982 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:31 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T22:33:32.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:32.983+0000 7f5ca0c00640 1 -- 172.21.15.43:0/3844192721 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v90) ==== 228+0+0 (secure 0 0 0) 0x7f5c9404d060 con 0x7f5ca4104b80 2024-03-28T22:33:32.982 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'cephfs_data_ec' created 2024-03-28T22:33:32.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:32.990+0000 7f5cab000640 1 -- 172.21.15.43:0/3844192721 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5c880a1e00 msgr2=0x7f5c880a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:32.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:32.990+0000 7f5cab000640 1 --2- 172.21.15.43:0/3844192721 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5c880a1e00 0x7f5c880a42c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5ca419ab80 tx=0x7f5c8c00b410 comp rx=0 tx=0).stop 2024-03-28T22:33:32.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:32.990+0000 7f5cab000640 1 -- 172.21.15.43:0/3844192721 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ca4104b80 msgr2=0x7f5ca419a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:32.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:32.991+0000 7f5cab000640 1 --2- 172.21.15.43:0/3844192721 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ca4104b80 0x7f5ca419a1f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5c94009bf0 tx=0x7f5c94035d10 comp rx=0 tx=0).stop 2024-03-28T22:33:32.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:32.991+0000 7f5cab000640 1 -- 172.21.15.43:0/3844192721 shutdown_connections 2024-03-28T22:33:32.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:32.991+0000 7f5cab000640 1 --2- 172.21.15.43:0/3844192721 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5c880a1e00 0x7f5c880a42c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:32.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:32.991+0000 7f5cab000640 1 --2- 172.21.15.43:0/3844192721 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ca4104b80 0x7f5ca419a1f0 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5c94009bf0 tx=0x7f5c94035d10 comp rx=0 tx=0).stop 2024-03-28T22:33:32.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:32.991+0000 7f5cab000640 1 --2- 172.21.15.43:0/3844192721 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ca4105540 0x7f5ca41a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:32.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:32.991+0000 7f5cab000640 1 --2- 172.21.15.43:0/3844192721 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ca4103990 0x7f5ca4199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:32.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:32.991+0000 7f5cab000640 1 -- 172.21.15.43:0/3844192721 >> 172.21.15.43:0/3844192721 conn(0x7f5ca40fef00 msgr2=0x7f5ca4107f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:32.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:32.992+0000 7f5cab000640 1 -- 172.21.15.43:0/3844192721 shutdown_connections 2024-03-28T22:33:32.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:32.993+0000 7f5cab000640 1 -- 172.21.15.43:0/3844192721 wait complete. 2024-03-28T22:33:33.018 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-28T22:33:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.219+0000 7fc87f000640 1 -- 172.21.15.43:0/812030966 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc878107c70 msgr2=0x7fc878108090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.219+0000 7fc87f000640 1 --2- 172.21.15.43:0/812030966 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc878107c70 0x7fc878108090 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc86c009900 tx=0x7fc86c02bc40 comp rx=0 tx=0).stop 2024-03-28T22:33:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.220+0000 7fc87f000640 1 -- 172.21.15.43:0/812030966 shutdown_connections 2024-03-28T22:33:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.220+0000 7fc87f000640 1 --2- 172.21.15.43:0/812030966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc878109820 0x7fc8781100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.220+0000 7fc87f000640 1 --2- 172.21.15.43:0/812030966 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc878108e60 0x7fc8781092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.220+0000 7fc87f000640 1 --2- 172.21.15.43:0/812030966 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc878107c70 0x7fc878108090 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:33.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.220+0000 7fc87f000640 1 -- 172.21.15.43:0/812030966 >> 172.21.15.43:0/812030966 conn(0x7fc878076c20 msgr2=0x7fc878079040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:33.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.220+0000 7fc87f000640 1 -- 172.21.15.43:0/812030966 shutdown_connections 2024-03-28T22:33:33.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.220+0000 7fc87f000640 1 -- 172.21.15.43:0/812030966 wait complete. 2024-03-28T22:33:33.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.221+0000 7fc87f000640 1 Processor -- start 2024-03-28T22:33:33.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.221+0000 7fc87f000640 1 -- start start 2024-03-28T22:33:33.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.221+0000 7fc87f000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc878107c70 0x7fc87819e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:33.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.221+0000 7fc87f000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc878108e60 0x7fc87819e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:33.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.221+0000 7fc87f000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc878109820 0x7fc8781a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:33.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.221+0000 7fc87f000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc878113930 con 0x7fc878108e60 2024-03-28T22:33:33.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.221+0000 7fc87f000640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fc8781137b0 con 0x7fc878107c70 2024-03-28T22:33:33.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.221+0000 7fc87f000640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fc878113ab0 con 0x7fc878109820 2024-03-28T22:33:33.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.221+0000 7fc87dc00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc878107c70 0x7fc87819e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:33.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.221+0000 7fc87dc00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc878107c70 0x7fc87819e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:42944/0 (socket says 172.21.15.43:42944) 2024-03-28T22:33:33.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.221+0000 7fc87dc00640 1 -- 172.21.15.43:0/1972995151 learned_addr learned my addr 172.21.15.43:0/1972995151 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:33.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.221+0000 7fc87d200640 1 --2- 172.21.15.43:0/1972995151 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc878108e60 0x7fc87819e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:33.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.221+0000 7fc87e600640 1 --2- 172.21.15.43:0/1972995151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc878109820 0x7fc8781a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:33.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.222+0000 7fc87dc00640 1 -- 172.21.15.43:0/1972995151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc878109820 msgr2=0x7fc8781a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:33.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.222+0000 7fc87dc00640 1 --2- 172.21.15.43:0/1972995151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc878109820 0x7fc8781a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:33.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.222+0000 7fc87dc00640 1 -- 172.21.15.43:0/1972995151 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc878108e60 msgr2=0x7fc87819e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:33.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.222+0000 7fc87dc00640 1 --2- 172.21.15.43:0/1972995151 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc878108e60 0x7fc87819e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:33.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.222+0000 7fc87dc00640 1 -- 172.21.15.43:0/1972995151 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8781a5bd0 con 0x7fc878107c70 2024-03-28T22:33:33.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.222+0000 7fc87dc00640 1 --2- 172.21.15.43:0/1972995151 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc878107c70 0x7fc87819e000 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc86c02efd0 tx=0x7fc86c005e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:33.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.222+0000 7fc876000640 1 -- 172.21.15.43:0/1972995151 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc86c041070 con 0x7fc878107c70 2024-03-28T22:33:33.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.222+0000 7fc876000640 1 -- 172.21.15.43:0/1972995151 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc86c0054e0 con 0x7fc878107c70 2024-03-28T22:33:33.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.222+0000 7fc876000640 1 -- 172.21.15.43:0/1972995151 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc86c034440 con 0x7fc878107c70 2024-03-28T22:33:33.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.222+0000 7fc87f000640 1 -- 172.21.15.43:0/1972995151 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8781a5e60 con 0x7fc878107c70 2024-03-28T22:33:33.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.223+0000 7fc87f000640 1 -- 172.21.15.43:0/1972995151 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8781a62c0 con 0x7fc878107c70 2024-03-28T22:33:33.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.224+0000 7fc87f000640 1 -- 172.21.15.43:0/1972995151 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc87810ef90 con 0x7fc878107c70 2024-03-28T22:33:33.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.225+0000 7fc876000640 1 -- 172.21.15.43:0/1972995151 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fc86c044070 con 0x7fc878107c70 2024-03-28T22:33:33.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.227+0000 7fc876000640 1 --2- 172.21.15.43:0/1972995151 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fc8600a1e00 0x7fc8600a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:33.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.227+0000 7fc876000640 1 -- 172.21.15.43:0/1972995151 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8770+0+0 (secure 0 0 0) 0x7fc86c0efbb0 con 0x7fc878107c70 2024-03-28T22:33:33.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.227+0000 7fc87d200640 1 --2- 172.21.15.43:0/1972995151 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fc8600a1e00 0x7fc8600a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:33.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.228+0000 7fc87d200640 1 --2- 172.21.15.43:0/1972995151 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fc8600a1e00 0x7fc8600a42c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc87819f4c0 tx=0x7fc86400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:33.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.231+0000 7fc876000640 1 -- 172.21.15.43:0/1972995151 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc86c0bc710 con 0x7fc878107c70 2024-03-28T22:33:33.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:32 smithi043 ceph-mon[31823]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T22:33:33.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:32 smithi172 ceph-mon[36566]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T22:33:33.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.432+0000 7fc87f000640 1 -- 172.21.15.43:0/1972995151 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fc8780623c0 con 0x7fc878107c70 2024-03-28T22:33:33.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:32 smithi082 ceph-mon[36471]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T22:33:33.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.985+0000 7fc876000640 1 -- 172.21.15.43:0/1972995151 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v91) ==== 169+0+0 (secure 0 0 0) 0x7fc86c0c07b0 con 0x7fc878107c70 2024-03-28T22:33:33.983 INFO:teuthology.orchestra.run.smithi043.stderr:set pool 4 allow_ec_overwrites to true 2024-03-28T22:33:33.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.990+0000 7fc87f000640 1 -- 172.21.15.43:0/1972995151 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fc8600a1e00 msgr2=0x7fc8600a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:33.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.990+0000 7fc87f000640 1 --2- 172.21.15.43:0/1972995151 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fc8600a1e00 0x7fc8600a42c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc87819f4c0 tx=0x7fc86400b390 comp rx=0 tx=0).stop 2024-03-28T22:33:33.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.991+0000 7fc87f000640 1 -- 172.21.15.43:0/1972995151 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc878107c70 msgr2=0x7fc87819e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:33.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.991+0000 7fc87f000640 1 --2- 172.21.15.43:0/1972995151 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc878107c70 0x7fc87819e000 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc86c02efd0 tx=0x7fc86c005e40 comp rx=0 tx=0).stop 2024-03-28T22:33:33.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.992+0000 7fc87f000640 1 -- 172.21.15.43:0/1972995151 shutdown_connections 2024-03-28T22:33:33.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.992+0000 7fc87f000640 1 --2- 172.21.15.43:0/1972995151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc878109820 0x7fc8781a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:33.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.992+0000 7fc87f000640 1 --2- 172.21.15.43:0/1972995151 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc878108e60 0x7fc87819e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:33.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.992+0000 7fc87f000640 1 --2- 172.21.15.43:0/1972995151 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fc8600a1e00 0x7fc8600a42c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:33.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.992+0000 7fc87f000640 1 --2- 172.21.15.43:0/1972995151 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc878107c70 0x7fc87819e000 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:33.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.992+0000 7fc87f000640 1 -- 172.21.15.43:0/1972995151 >> 172.21.15.43:0/1972995151 conn(0x7fc878076c20 msgr2=0x7fc87810c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:33.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.992+0000 7fc87f000640 1 -- 172.21.15.43:0/1972995151 shutdown_connections 2024-03-28T22:33:33.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:33.993+0000 7fc87f000640 1 -- 172.21.15.43:0/1972995151 wait complete. 2024-03-28T22:33:34.029 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-28T22:33:34.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.219+0000 7ff92e200640 1 -- 172.21.15.43:0/2238464223 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff928103990 msgr2=0x7ff928103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:34.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.219+0000 7ff92e200640 1 --2- 172.21.15.43:0/2238464223 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff928103990 0x7ff928103db0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff918009900 tx=0x7ff91802bc30 comp rx=0 tx=0).stop 2024-03-28T22:33:34.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.219+0000 7ff92e200640 1 -- 172.21.15.43:0/2238464223 shutdown_connections 2024-03-28T22:33:34.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.219+0000 7ff92e200640 1 --2- 172.21.15.43:0/2238464223 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff928105540 0x7ff92810bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:34.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.219+0000 7ff92e200640 1 --2- 172.21.15.43:0/2238464223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff928104b80 0x7ff928105000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:34.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.219+0000 7ff92e200640 1 --2- 172.21.15.43:0/2238464223 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff928103990 0x7ff928103db0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:34.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.219+0000 7ff92e200640 1 -- 172.21.15.43:0/2238464223 >> 172.21.15.43:0/2238464223 conn(0x7ff9280fef00 msgr2=0x7ff928101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:34.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.219+0000 7ff92e200640 1 -- 172.21.15.43:0/2238464223 shutdown_connections 2024-03-28T22:33:34.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.219+0000 7ff92e200640 1 -- 172.21.15.43:0/2238464223 wait complete. 2024-03-28T22:33:34.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.220+0000 7ff92e200640 1 Processor -- start 2024-03-28T22:33:34.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.220+0000 7ff92e200640 1 -- start start 2024-03-28T22:33:34.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.221+0000 7ff92e200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff928103990 0x7ff928199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:34.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.221+0000 7ff92e200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff928104b80 0x7ff92819a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:34.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.221+0000 7ff92e200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff928105540 0x7ff9281a1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:34.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.221+0000 7ff92e200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff92810f660 con 0x7ff928104b80 2024-03-28T22:33:34.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.221+0000 7ff92e200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ff92810f4e0 con 0x7ff928103990 2024-03-28T22:33:34.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.221+0000 7ff92e200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7ff92810f7e0 con 0x7ff928105540 2024-03-28T22:33:34.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.221+0000 7ff927400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff928103990 0x7ff928199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:34.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.221+0000 7ff91ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff928104b80 0x7ff92819a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:34.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.221+0000 7ff927400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff928103990 0x7ff928199ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:42952/0 (socket says 172.21.15.43:42952) 2024-03-28T22:33:34.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.221+0000 7ff927400640 1 -- 172.21.15.43:0/279841099 learned_addr learned my addr 172.21.15.43:0/279841099 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:34.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.221+0000 7ff927e00640 1 --2- 172.21.15.43:0/279841099 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff928105540 0x7ff9281a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:34.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.221+0000 7ff927400640 1 -- 172.21.15.43:0/279841099 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff928105540 msgr2=0x7ff9281a1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:34.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.221+0000 7ff927400640 1 --2- 172.21.15.43:0/279841099 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff928105540 0x7ff9281a1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:34.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.221+0000 7ff927400640 1 -- 172.21.15.43:0/279841099 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff928104b80 msgr2=0x7ff92819a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:34.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.221+0000 7ff927400640 1 --2- 172.21.15.43:0/279841099 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff928104b80 0x7ff92819a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:34.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.221+0000 7ff927400640 1 -- 172.21.15.43:0/279841099 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9281a18e0 con 0x7ff928103990 2024-03-28T22:33:34.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.222+0000 7ff927400640 1 --2- 172.21.15.43:0/279841099 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff928103990 0x7ff928199ca0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff918000c00 tx=0x7ff91802c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:34.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.222+0000 7ff924c00640 1 -- 172.21.15.43:0/279841099 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff918043020 con 0x7ff928103990 2024-03-28T22:33:34.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.222+0000 7ff924c00640 1 -- 172.21.15.43:0/279841099 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9180054e0 con 0x7ff928103990 2024-03-28T22:33:34.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.222+0000 7ff92e200640 1 -- 172.21.15.43:0/279841099 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9281a1b70 con 0x7ff928103990 2024-03-28T22:33:34.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.222+0000 7ff924c00640 1 -- 172.21.15.43:0/279841099 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff918003ce0 con 0x7ff928103990 2024-03-28T22:33:34.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.222+0000 7ff92e200640 1 -- 172.21.15.43:0/279841099 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9281a20e0 con 0x7ff928103990 2024-03-28T22:33:34.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.223+0000 7ff92e200640 1 -- 172.21.15.43:0/279841099 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff92810ac50 con 0x7ff928103990 2024-03-28T22:33:34.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.225+0000 7ff924c00640 1 -- 172.21.15.43:0/279841099 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7ff918046020 con 0x7ff928103990 2024-03-28T22:33:34.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.226+0000 7ff924c00640 1 --2- 172.21.15.43:0/279841099 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff90c0a1e00 0x7ff90c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:34.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.226+0000 7ff924c00640 1 -- 172.21.15.43:0/279841099 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8770+0+0 (secure 0 0 0) 0x7ff9180effe0 con 0x7ff928103990 2024-03-28T22:33:34.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.227+0000 7ff91ea00640 1 --2- 172.21.15.43:0/279841099 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff90c0a1e00 0x7ff90c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:34.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.227+0000 7ff91ea00640 1 --2- 172.21.15.43:0/279841099 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff90c0a1e00 0x7ff90c0a42c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff92819b1a0 tx=0x7ff910005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:34.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.230+0000 7ff924c00640 1 -- 172.21.15.43:0/279841099 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff91804d020 con 0x7ff928103990 2024-03-28T22:33:34.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:33 smithi043 ceph-mon[31823]: pgmap v321: 97 pgs: 61 active+clean, 22 creating+peering, 14 unknown; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 824 B/s wr, 3 op/s 2024-03-28T22:33:34.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:33 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T22:33:34.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:33 smithi043 ceph-mon[31823]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T22:33:34.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:33 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1972995151' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-28T22:33:34.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:33 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-28T22:33:34.320 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:34 smithi082 ceph-mon[36471]: pgmap v321: 97 pgs: 61 active+clean, 22 creating+peering, 14 unknown; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 824 B/s wr, 3 op/s 2024-03-28T22:33:34.320 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:34 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T22:33:34.320 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:34 smithi082 ceph-mon[36471]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T22:33:34.320 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:34 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1972995151' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-28T22:33:34.320 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:34 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-28T22:33:34.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:34 smithi172 ceph-mon[36566]: pgmap v321: 97 pgs: 61 active+clean, 22 creating+peering, 14 unknown; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 824 B/s wr, 3 op/s 2024-03-28T22:33:34.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:34 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T22:33:34.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:34 smithi172 ceph-mon[36566]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T22:33:34.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:34 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1972995151' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-28T22:33:34.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:34 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-28T22:33:34.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:34.471+0000 7ff92e200640 1 -- 172.21.15.43:0/279841099 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7ff92802d390 con 0x7ff928103990 2024-03-28T22:33:34.699 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 28 22:33:34 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-b[67270]: 2024-03-28T22:33:34.320+0000 7fb4be5ee700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T22:33:35.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.017+0000 7ff924c00640 1 -- 172.21.15.43:0/279841099 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7ff9180bd1f0 con 0x7ff928103990 2024-03-28T22:33:35.015 INFO:teuthology.orchestra.run.smithi043.stderr:added data pool 4 to fsmap 2024-03-28T22:33:35.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.020+0000 7ff92e200640 1 -- 172.21.15.43:0/279841099 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff90c0a1e00 msgr2=0x7ff90c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:35.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.020+0000 7ff92e200640 1 --2- 172.21.15.43:0/279841099 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff90c0a1e00 0x7ff90c0a42c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff92819b1a0 tx=0x7ff910005f90 comp rx=0 tx=0).stop 2024-03-28T22:33:35.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.020+0000 7ff92e200640 1 -- 172.21.15.43:0/279841099 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff928103990 msgr2=0x7ff928199ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:35.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.020+0000 7ff92e200640 1 --2- 172.21.15.43:0/279841099 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff928103990 0x7ff928199ca0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff918000c00 tx=0x7ff91802c6c0 comp rx=0 tx=0).stop 2024-03-28T22:33:35.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.020+0000 7ff92e200640 1 -- 172.21.15.43:0/279841099 shutdown_connections 2024-03-28T22:33:35.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.020+0000 7ff92e200640 1 --2- 172.21.15.43:0/279841099 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff928105540 0x7ff9281a1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:35.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.020+0000 7ff92e200640 1 --2- 172.21.15.43:0/279841099 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff928104b80 0x7ff92819a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:35.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.020+0000 7ff92e200640 1 --2- 172.21.15.43:0/279841099 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff90c0a1e00 0x7ff90c0a42c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:35.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.020+0000 7ff92e200640 1 --2- 172.21.15.43:0/279841099 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff928103990 0x7ff928199ca0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:35.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.021+0000 7ff92e200640 1 -- 172.21.15.43:0/279841099 >> 172.21.15.43:0/279841099 conn(0x7ff9280fef00 msgr2=0x7ff928107e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:35.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.021+0000 7ff92e200640 1 -- 172.21.15.43:0/279841099 shutdown_connections 2024-03-28T22:33:35.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.021+0000 7ff92e200640 1 -- 172.21.15.43:0/279841099 wait complete. 2024-03-28T22:33:35.042 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:33:35.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.267+0000 7fa128c00640 1 -- 172.21.15.43:0/3145662201 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa124107d90 msgr2=0x7fa1241081b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:35.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.267+0000 7fa128c00640 1 --2- 172.21.15.43:0/3145662201 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa124107d90 0x7fa1241081b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa114009900 tx=0x7fa11402bc30 comp rx=0 tx=0).stop 2024-03-28T22:33:35.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.267+0000 7fa128c00640 1 -- 172.21.15.43:0/3145662201 shutdown_connections 2024-03-28T22:33:35.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.267+0000 7fa128c00640 1 --2- 172.21.15.43:0/3145662201 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa124109940 0x7fa1241101d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:35.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.267+0000 7fa128c00640 1 --2- 172.21.15.43:0/3145662201 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa124108f80 0x7fa124109400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:35.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.267+0000 7fa128c00640 1 --2- 172.21.15.43:0/3145662201 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa124107d90 0x7fa1241081b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:35.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.267+0000 7fa128c00640 1 -- 172.21.15.43:0/3145662201 >> 172.21.15.43:0/3145662201 conn(0x7fa124076d60 msgr2=0x7fa124079180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:35.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:35 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-28T22:33:35.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:35 smithi043 ceph-mon[31823]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T22:33:35.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:35 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/279841099' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-28T22:33:35.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:35 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-28T22:33:35.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.268+0000 7fa128c00640 1 -- 172.21.15.43:0/3145662201 shutdown_connections 2024-03-28T22:33:35.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.268+0000 7fa128c00640 1 -- 172.21.15.43:0/3145662201 wait complete. 2024-03-28T22:33:35.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.269+0000 7fa128c00640 1 Processor -- start 2024-03-28T22:33:35.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.269+0000 7fa128c00640 1 -- start start 2024-03-28T22:33:35.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.269+0000 7fa128c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa124107d90 0x7fa12419e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:35.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.269+0000 7fa128c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa124108f80 0x7fa12419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:35.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.269+0000 7fa128c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa124109940 0x7fa1241a5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:35.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.269+0000 7fa128c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa124113a50 con 0x7fa124108f80 2024-03-28T22:33:35.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.269+0000 7fa128c00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa1241138d0 con 0x7fa124107d90 2024-03-28T22:33:35.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.269+0000 7fa128c00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fa124113bd0 con 0x7fa124109940 2024-03-28T22:33:35.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.269+0000 7fa119400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa124108f80 0x7fa12419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:35.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.269+0000 7fa121e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa124107d90 0x7fa12419e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:35.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.269+0000 7fa119400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa124108f80 0x7fa12419e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45512/0 (socket says 172.21.15.43:45512) 2024-03-28T22:33:35.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.269+0000 7fa119400640 1 -- 172.21.15.43:0/1366930051 learned_addr learned my addr 172.21.15.43:0/1366930051 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:35.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.269+0000 7fa122800640 1 --2- 172.21.15.43:0/1366930051 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa124109940 0x7fa1241a5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:35.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.270+0000 7fa121e00640 1 -- 172.21.15.43:0/1366930051 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa124109940 msgr2=0x7fa1241a5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:35.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.270+0000 7fa121e00640 1 --2- 172.21.15.43:0/1366930051 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa124109940 0x7fa1241a5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:35.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.270+0000 7fa121e00640 1 -- 172.21.15.43:0/1366930051 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa124108f80 msgr2=0x7fa12419e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:35.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.270+0000 7fa121e00640 1 --2- 172.21.15.43:0/1366930051 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa124108f80 0x7fa12419e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:35.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.270+0000 7fa121e00640 1 -- 172.21.15.43:0/1366930051 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1241a5cd0 con 0x7fa124107d90 2024-03-28T22:33:35.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.270+0000 7fa122800640 1 --2- 172.21.15.43:0/1366930051 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa124109940 0x7fa1241a5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:33:35.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.270+0000 7fa121e00640 1 --2- 172.21.15.43:0/1366930051 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa124107d90 0x7fa12419e100 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa114009230 tx=0x7fa11402c5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:35.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.270+0000 7fa11b400640 1 -- 172.21.15.43:0/1366930051 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa114043020 con 0x7fa124107d90 2024-03-28T22:33:35.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.270+0000 7fa11b400640 1 -- 172.21.15.43:0/1366930051 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1140054e0 con 0x7fa124107d90 2024-03-28T22:33:35.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.270+0000 7fa128c00640 1 -- 172.21.15.43:0/1366930051 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1241a5f60 con 0x7fa124107d90 2024-03-28T22:33:35.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.270+0000 7fa11b400640 1 -- 172.21.15.43:0/1366930051 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa114003b60 con 0x7fa124107d90 2024-03-28T22:33:35.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.270+0000 7fa128c00640 1 -- 172.21.15.43:0/1366930051 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1241a63d0 con 0x7fa124107d90 2024-03-28T22:33:35.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.272+0000 7fa128c00640 1 -- 172.21.15.43:0/1366930051 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa12410f130 con 0x7fa124107d90 2024-03-28T22:33:35.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.273+0000 7fa11b400640 1 -- 172.21.15.43:0/1366930051 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fa114046020 con 0x7fa124107d90 2024-03-28T22:33:35.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.274+0000 7fa11b400640 1 --2- 172.21.15.43:0/1366930051 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa1080a1e00 0x7fa1080a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:35.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.275+0000 7fa11b400640 1 -- 172.21.15.43:0/1366930051 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7fa1140f0080 con 0x7fa124107d90 2024-03-28T22:33:35.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.275+0000 7fa119400640 1 --2- 172.21.15.43:0/1366930051 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa1080a1e00 0x7fa1080a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:35.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.275+0000 7fa119400640 1 --2- 172.21.15.43:0/1366930051 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa1080a1e00 0x7fa1080a42c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa12419f5c0 tx=0x7fa10c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:35.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.278+0000 7fa11b400640 1 -- 172.21.15.43:0/1366930051 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa11404d020 con 0x7fa124107d90 2024-03-28T22:33:35.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:35 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-28T22:33:35.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:35 smithi172 ceph-mon[36566]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T22:33:35.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:35 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/279841099' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-28T22:33:35.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:35 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-28T22:33:35.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:35 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-28T22:33:35.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:35 smithi082 ceph-mon[36471]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T22:33:35.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:35 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/279841099' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-28T22:33:35.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:35 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-28T22:33:35.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.518+0000 7fa128c00640 1 -- 172.21.15.43:0/1366930051 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa124062350 con 0x7fa124107d90 2024-03-28T22:33:35.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.519+0000 7fa11b400640 1 -- 172.21.15.43:0/1366930051 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11797 (secure 0 0 0) 0x7fa1140bd210 con 0x7fa124107d90 2024-03-28T22:33:35.518 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:33:35.518 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6835/1084988072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1084988072},{"type":"v1","addr":"172.21.15.82:6835","nonce":1084988072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24518,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24556,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T22:33:28.906396+0000","modified":"2024-03-28T22:33:34.473616+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24572,"qdb_cluster":[24572]},"id":1}]} 2024-03-28T22:33:35.518 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 17 2024-03-28T22:33:35.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.522+0000 7fa128c00640 1 -- 172.21.15.43:0/1366930051 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa1080a1e00 msgr2=0x7fa1080a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:35.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.522+0000 7fa128c00640 1 --2- 172.21.15.43:0/1366930051 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa1080a1e00 0x7fa1080a42c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa12419f5c0 tx=0x7fa10c005f90 comp rx=0 tx=0).stop 2024-03-28T22:33:35.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.522+0000 7fa128c00640 1 -- 172.21.15.43:0/1366930051 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa124107d90 msgr2=0x7fa12419e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:35.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.522+0000 7fa128c00640 1 --2- 172.21.15.43:0/1366930051 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa124107d90 0x7fa12419e100 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa114009230 tx=0x7fa11402c5d0 comp rx=0 tx=0).stop 2024-03-28T22:33:35.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.522+0000 7fa128c00640 1 -- 172.21.15.43:0/1366930051 shutdown_connections 2024-03-28T22:33:35.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.522+0000 7fa128c00640 1 --2- 172.21.15.43:0/1366930051 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa124109940 0x7fa1241a5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:35.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.522+0000 7fa128c00640 1 --2- 172.21.15.43:0/1366930051 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa124108f80 0x7fa12419e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:35.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.523+0000 7fa128c00640 1 --2- 172.21.15.43:0/1366930051 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa1080a1e00 0x7fa1080a42c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:35.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.523+0000 7fa128c00640 1 --2- 172.21.15.43:0/1366930051 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa124107d90 0x7fa12419e100 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:35.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.523+0000 7fa128c00640 1 -- 172.21.15.43:0/1366930051 >> 172.21.15.43:0/1366930051 conn(0x7fa124076d60 msgr2=0x7fa12410c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:35.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.523+0000 7fa128c00640 1 -- 172.21.15.43:0/1366930051 shutdown_connections 2024-03-28T22:33:35.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.523+0000 7fa128c00640 1 -- 172.21.15.43:0/1366930051 wait complete. 2024-03-28T22:33:35.542 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:33:35.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.769+0000 7f4ce7e00640 1 -- 172.21.15.43:0/2947921316 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ce8109760 msgr2=0x7f4ce810fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:35.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.769+0000 7f4ce7e00640 1 --2- 172.21.15.43:0/2947921316 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ce8109760 0x7f4ce810fff0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0009be0 tx=0x7f4ce002bef0 comp rx=0 tx=0).stop 2024-03-28T22:33:35.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.770+0000 7f4ce7e00640 1 -- 172.21.15.43:0/2947921316 shutdown_connections 2024-03-28T22:33:35.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.770+0000 7f4ce7e00640 1 --2- 172.21.15.43:0/2947921316 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ce8109760 0x7f4ce810fff0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:35.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.770+0000 7f4ce7e00640 1 --2- 172.21.15.43:0/2947921316 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ce8108da0 0x7f4ce8109220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:35.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.770+0000 7f4ce7e00640 1 --2- 172.21.15.43:0/2947921316 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ce8107bb0 0x7f4ce8107fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:35.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.770+0000 7f4ce7e00640 1 -- 172.21.15.43:0/2947921316 >> 172.21.15.43:0/2947921316 conn(0x7f4ce8076b60 msgr2=0x7f4ce8078f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:35.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.770+0000 7f4ce7e00640 1 -- 172.21.15.43:0/2947921316 shutdown_connections 2024-03-28T22:33:35.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.770+0000 7f4ce7e00640 1 -- 172.21.15.43:0/2947921316 wait complete. 2024-03-28T22:33:35.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.771+0000 7f4ce7e00640 1 Processor -- start 2024-03-28T22:33:35.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.771+0000 7f4ce7e00640 1 -- start start 2024-03-28T22:33:35.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.771+0000 7f4ce7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ce8107bb0 0x7f4ce819dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:35.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.771+0000 7f4ce7e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ce8108da0 0x7f4ce819e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:35.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.771+0000 7f4ce7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ce8109760 0x7f4ce81a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:35.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.771+0000 7f4ce7e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce8113880 con 0x7f4ce8107bb0 2024-03-28T22:33:35.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.771+0000 7f4ce7e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce8113700 con 0x7f4ce8108da0 2024-03-28T22:33:35.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.771+0000 7f4ce7e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce8113a00 con 0x7f4ce8109760 2024-03-28T22:33:35.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.771+0000 7f4ce6000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ce8108da0 0x7f4ce819e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:35.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.771+0000 7f4ce6a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ce8107bb0 0x7f4ce819dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:35.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.771+0000 7f4ce7400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ce8109760 0x7f4ce81a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:35.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.771+0000 7f4ce6000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ce8108da0 0x7f4ce819e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:43152/0 (socket says 172.21.15.43:43152) 2024-03-28T22:33:35.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.771+0000 7f4ce6000640 1 -- 172.21.15.43:0/1660706477 learned_addr learned my addr 172.21.15.43:0/1660706477 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:35.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.772+0000 7f4ce6000640 1 -- 172.21.15.43:0/1660706477 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ce8109760 msgr2=0x7f4ce81a5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:35.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.772+0000 7f4ce6000640 1 --2- 172.21.15.43:0/1660706477 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ce8109760 0x7f4ce81a5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:35.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.772+0000 7f4ce6000640 1 -- 172.21.15.43:0/1660706477 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ce8107bb0 msgr2=0x7f4ce819dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:35.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.772+0000 7f4ce6000640 1 --2- 172.21.15.43:0/1660706477 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ce8107bb0 0x7f4ce819dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:35.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.772+0000 7f4ce6000640 1 -- 172.21.15.43:0/1660706477 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ce81a5a00 con 0x7f4ce8108da0 2024-03-28T22:33:35.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.772+0000 7f4ce7400640 1 --2- 172.21.15.43:0/1660706477 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ce8109760 0x7f4ce81a5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:33:35.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.772+0000 7f4ce6a00640 1 --2- 172.21.15.43:0/1660706477 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ce8107bb0 0x7f4ce819dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:33:35.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.772+0000 7f4ce6000640 1 --2- 172.21.15.43:0/1660706477 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ce8108da0 0x7f4ce819e320 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4cd400e9f0 tx=0x7f4cd400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:35.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.772+0000 7f4cdf400640 1 -- 172.21.15.43:0/1660706477 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd400cd80 con 0x7f4ce8108da0 2024-03-28T22:33:35.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.772+0000 7f4cdf400640 1 -- 172.21.15.43:0/1660706477 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cd4005500 con 0x7f4ce8108da0 2024-03-28T22:33:35.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.772+0000 7f4ce7e00640 1 -- 172.21.15.43:0/1660706477 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ce81a5cf0 con 0x7f4ce8108da0 2024-03-28T22:33:35.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.772+0000 7f4cdf400640 1 -- 172.21.15.43:0/1660706477 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd4010690 con 0x7f4ce8108da0 2024-03-28T22:33:35.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.773+0000 7f4ce7e00640 1 -- 172.21.15.43:0/1660706477 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ce81a61e0 con 0x7f4ce8108da0 2024-03-28T22:33:35.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.774+0000 7f4ce7e00640 1 -- 172.21.15.43:0/1660706477 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ce810ee00 con 0x7f4ce8108da0 2024-03-28T22:33:35.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.776+0000 7f4cdf400640 1 -- 172.21.15.43:0/1660706477 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f4cd40046c0 con 0x7f4ce8108da0 2024-03-28T22:33:35.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.777+0000 7f4cdf400640 1 --2- 172.21.15.43:0/1660706477 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4ccc0a1e00 0x7f4ccc0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:35.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.778+0000 7f4cdf400640 1 -- 172.21.15.43:0/1660706477 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f4cd40ce3b0 con 0x7f4ce8108da0 2024-03-28T22:33:35.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.778+0000 7f4ce6a00640 1 --2- 172.21.15.43:0/1660706477 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4ccc0a1e00 0x7f4ccc0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:35.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.778+0000 7f4ce6a00640 1 --2- 172.21.15.43:0/1660706477 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4ccc0a1e00 0x7f4ccc0a42c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4cd80096c0 tx=0x7f4cd8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:35.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.781+0000 7f4cdf400640 1 -- 172.21.15.43:0/1660706477 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4cd409b540 con 0x7f4ce8108da0 2024-03-28T22:33:35.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.984+0000 7f4ce7e00640 1 -- 172.21.15.43:0/1660706477 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4ce8062350 con 0x7f4ce8108da0 2024-03-28T22:33:35.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.986+0000 7f4cdf400640 1 -- 172.21.15.43:0/1660706477 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24093 (secure 0 0 0) 0x7f4cd409ae40 con 0x7f4ce8108da0 2024-03-28T22:33:35.984 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:33:35.984 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":92,"fsid":"c2276428-ed51-11ee-95d0-87774f69a715","created":"2024-03-28T22:24:19.950084+0000","modified":"2024-03-28T22:33:34.473170+0000","last_up_change":"2024-03-28T22:32:31.078737+0000","last_in_change":"2024-03-28T22:32:10.147097+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:28:14.713290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:33:27.167435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:33:28.411825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T22:33:32.523122+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b855adcd-786c-4e42-b8fe-a515a77ce979","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6803","nonce":2826124409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6805","nonce":2826124409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6809","nonce":2826124409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6807","nonce":2826124409}]},"public_addr":"172.21.15.43:6803/2826124409","cluster_addr":"172.21.15.43:6805/2826124409","heartbeat_back_addr":"172.21.15.43:6809/2826124409","heartbeat_front_addr":"172.21.15.43:6807/2826124409","state":["exists","up"]},{"osd":1,"uuid":"031c143d-8ad1-4b68-8db9-3c717137e3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6801","nonce":3043263644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6803","nonce":3043263644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6807","nonce":3043263644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6805","nonce":3043263644}]},"public_addr":"172.21.15.82:6801/3043263644","cluster_addr":"172.21.15.82:6803/3043263644","heartbeat_back_addr":"172.21.15.82:6807/3043263644","heartbeat_front_addr":"172.21.15.82:6805/3043263644","state":["exists","up"]},{"osd":2,"uuid":"20632bbb-f07a-4ca1-b0c4-ff9d508e211c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6801","nonce":4213953645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6803","nonce":4213953645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6807","nonce":4213953645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6805","nonce":4213953645}]},"public_addr":"172.21.15.172:6801/4213953645","cluster_addr":"172.21.15.172:6803/4213953645","heartbeat_back_addr":"172.21.15.172:6807/4213953645","heartbeat_front_addr":"172.21.15.172:6805/4213953645","state":["exists","up"]},{"osd":3,"uuid":"9a1f2312-4dbd-475b-8736-d5ebc8788e09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6811","nonce":1597280920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6813","nonce":1597280920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6817","nonce":1597280920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6815","nonce":1597280920}]},"public_addr":"172.21.15.43:6811/1597280920","cluster_addr":"172.21.15.43:6813/1597280920","heartbeat_back_addr":"172.21.15.43:6817/1597280920","heartbeat_front_addr":"172.21.15.43:6815/1597280920","state":["exists","up"]},{"osd":4,"uuid":"29140ead-7075-42b2-90c1-27cae0b29ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6809","nonce":215451895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6811","nonce":215451895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6815","nonce":215451895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6813","nonce":215451895}]},"public_addr":"172.21.15.82:6809/215451895","cluster_addr":"172.21.15.82:6811/215451895","heartbeat_back_addr":"172.21.15.82:6815/215451895","heartbeat_front_addr":"172.21.15.82:6813/215451895","state":["exists","up"]},{"osd":5,"uuid":"59644c61-93dc-4536-b2ad-0256b24e5cb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6809","nonce":1576148933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6811","nonce":1576148933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6815","nonce":1576148933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6813","nonce":1576148933}]},"public_addr":"172.21.15.172:6809/1576148933","cluster_addr":"172.21.15.172:6811/1576148933","heartbeat_back_addr":"172.21.15.172:6815/1576148933","heartbeat_front_addr":"172.21.15.172:6813/1576148933","state":["exists","up"]},{"osd":6,"uuid":"22d54e57-c3f1-4ecf-9921-8e815b48f92a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6819","nonce":791044279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6821","nonce":791044279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6825","nonce":791044279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6823","nonce":791044279}]},"public_addr":"172.21.15.43:6819/791044279","cluster_addr":"172.21.15.43:6821/791044279","heartbeat_back_addr":"172.21.15.43:6825/791044279","heartbeat_front_addr":"172.21.15.43:6823/791044279","state":["exists","up"]},{"osd":7,"uuid":"248cd1e2-0d6b-452f-a0a8-c51fe1b975ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6817","nonce":2157777323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6819","nonce":2157777323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6823","nonce":2157777323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6821","nonce":2157777323}]},"public_addr":"172.21.15.82:6817/2157777323","cluster_addr":"172.21.15.82:6819/2157777323","heartbeat_back_addr":"172.21.15.82:6823/2157777323","heartbeat_front_addr":"172.21.15.82:6821/2157777323","state":["exists","up"]},{"osd":8,"uuid":"d98aab6e-8bae-4c24-906c-4f492f047b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6817","nonce":1095328248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6819","nonce":1095328248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6823","nonce":1095328248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6821","nonce":1095328248}]},"public_addr":"172.21.15.172:6817/1095328248","cluster_addr":"172.21.15.172:6819/1095328248","heartbeat_back_addr":"172.21.15.172:6823/1095328248","heartbeat_front_addr":"172.21.15.172:6821/1095328248","state":["exists","up"]},{"osd":9,"uuid":"19aa383e-a0f6-45ea-96cc-e7956535caa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6827","nonce":3657791498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6829","nonce":3657791498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6833","nonce":3657791498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6831","nonce":3657791498}]},"public_addr":"172.21.15.43:6827/3657791498","cluster_addr":"172.21.15.43:6829/3657791498","heartbeat_back_addr":"172.21.15.43:6833/3657791498","heartbeat_front_addr":"172.21.15.43:6831/3657791498","state":["exists","up"]},{"osd":10,"uuid":"666977e5-eaf7-41b9-a18f-c380e149345d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6825","nonce":3454954047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6827","nonce":3454954047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6831","nonce":3454954047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6829","nonce":3454954047}]},"public_addr":"172.21.15.82:6825/3454954047","cluster_addr":"172.21.15.82:6827/3454954047","heartbeat_back_addr":"172.21.15.82:6831/3454954047","heartbeat_front_addr":"172.21.15.82:6829/3454954047","state":["exists","up"]},{"osd":11,"uuid":"615cf3be-8076-470e-ad09-c880e7593717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6825","nonce":2563517144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6827","nonce":2563517144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6831","nonce":2563517144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6829","nonce":2563517144}]},"public_addr":"172.21.15.172:6825/2563517144","cluster_addr":"172.21.15.172:6827/2563517144","heartbeat_back_addr":"172.21.15.172:6831/2563517144","heartbeat_front_addr":"172.21.15.172:6829/2563517144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:21.727405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:47.100259+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:10.615438+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:43.632144+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:09.487940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:34.604463+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:09.594452+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:35.709795+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:01.184202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:35.455322+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:02.413166+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:28.550754+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/578791143":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/4029313215":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/1574178482":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6801/3517712438":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6800/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/1375099638":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/2710117652":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6801/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/3215634168":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6800/3517712438":"2024-03-29T22:25:28.008215+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:33:35.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.988+0000 7f4ce7e00640 1 -- 172.21.15.43:0/1660706477 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4ccc0a1e00 msgr2=0x7f4ccc0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:35.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.988+0000 7f4ce7e00640 1 --2- 172.21.15.43:0/1660706477 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4ccc0a1e00 0x7f4ccc0a42c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4cd80096c0 tx=0x7f4cd8009260 comp rx=0 tx=0).stop 2024-03-28T22:33:35.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.988+0000 7f4ce7e00640 1 -- 172.21.15.43:0/1660706477 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ce8108da0 msgr2=0x7f4ce819e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:35.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.988+0000 7f4ce7e00640 1 --2- 172.21.15.43:0/1660706477 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ce8108da0 0x7f4ce819e320 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4cd400e9f0 tx=0x7f4cd400edf0 comp rx=0 tx=0).stop 2024-03-28T22:33:35.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.989+0000 7f4ce7e00640 1 -- 172.21.15.43:0/1660706477 shutdown_connections 2024-03-28T22:33:35.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.989+0000 7f4ce7e00640 1 --2- 172.21.15.43:0/1660706477 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4ccc0a1e00 0x7f4ccc0a42c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:35.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.989+0000 7f4ce7e00640 1 --2- 172.21.15.43:0/1660706477 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ce8108da0 0x7f4ce819e320 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:35.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.989+0000 7f4ce7e00640 1 --2- 172.21.15.43:0/1660706477 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ce8109760 0x7f4ce81a5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:35.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.989+0000 7f4ce7e00640 1 --2- 172.21.15.43:0/1660706477 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ce8107bb0 0x7f4ce819dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:35.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.989+0000 7f4ce7e00640 1 -- 172.21.15.43:0/1660706477 >> 172.21.15.43:0/1660706477 conn(0x7f4ce8076b60 msgr2=0x7f4ce810bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:35.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.989+0000 7f4ce7e00640 1 -- 172.21.15.43:0/1660706477 shutdown_connections 2024-03-28T22:33:35.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:35.989+0000 7f4ce7e00640 1 -- 172.21.15.43:0/1660706477 wait complete. 2024-03-28T22:33:36.009 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:33:36.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.222+0000 7f246f600640 1 -- 172.21.15.43:0/1134323798 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2468109630 msgr2=0x7f246810bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:36.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.222+0000 7f246f600640 1 --2- 172.21.15.43:0/1134323798 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2468109630 0x7f246810bac0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2460009be0 tx=0x7f246002bef0 comp rx=0 tx=0).stop 2024-03-28T22:33:36.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.222+0000 7f246f600640 1 -- 172.21.15.43:0/1134323798 shutdown_connections 2024-03-28T22:33:36.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.222+0000 7f246f600640 1 --2- 172.21.15.43:0/1134323798 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2468109630 0x7f246810bac0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:36.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.222+0000 7f246f600640 1 --2- 172.21.15.43:0/1134323798 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24680741b0 0x7f2468074630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:36.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.222+0000 7f246f600640 1 --2- 172.21.15.43:0/1134323798 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2468075790 0x7f2468073be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:36.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.222+0000 7f246f600640 1 -- 172.21.15.43:0/1134323798 >> 172.21.15.43:0/1134323798 conn(0x7f24680fcd00 msgr2=0x7f24680ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:36.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.222+0000 7f246f600640 1 -- 172.21.15.43:0/1134323798 shutdown_connections 2024-03-28T22:33:36.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.222+0000 7f246f600640 1 -- 172.21.15.43:0/1134323798 wait complete. 2024-03-28T22:33:36.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.223+0000 7f246f600640 1 Processor -- start 2024-03-28T22:33:36.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.223+0000 7f246f600640 1 -- start start 2024-03-28T22:33:36.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.223+0000 7f246f600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24680741b0 0x7f2468199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:36.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.224+0000 7f246f600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2468075790 0x7f246819a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:36.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.224+0000 7f246f600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2468109630 0x7f24681a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:36.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.224+0000 7f246f600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f246810f600 con 0x7f24680741b0 2024-03-28T22:33:36.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.224+0000 7f246f600640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f246810f480 con 0x7f2468109630 2024-03-28T22:33:36.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.224+0000 7f246f600640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f246810f780 con 0x7f2468075790 2024-03-28T22:33:36.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.224+0000 7f246cc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24680741b0 0x7f2468199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:36.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.224+0000 7f246d600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2468109630 0x7f24681a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:36.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.224+0000 7f246cc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24680741b0 0x7f2468199c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45550/0 (socket says 172.21.15.43:45550) 2024-03-28T22:33:36.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.224+0000 7f246d600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2468109630 0x7f24681a11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:43170/0 (socket says 172.21.15.43:43170) 2024-03-28T22:33:36.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.224+0000 7f246cc00640 1 -- 172.21.15.43:0/3424093585 learned_addr learned my addr 172.21.15.43:0/3424093585 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:36.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.224+0000 7f245fe00640 1 --2- 172.21.15.43:0/3424093585 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2468075790 0x7f246819a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:36.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.224+0000 7f246d600640 1 -- 172.21.15.43:0/3424093585 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2468075790 msgr2=0x7f246819a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:36.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.224+0000 7f246d600640 1 --2- 172.21.15.43:0/3424093585 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2468075790 0x7f246819a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:36.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.224+0000 7f246d600640 1 -- 172.21.15.43:0/3424093585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24680741b0 msgr2=0x7f2468199c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:36.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.224+0000 7f246d600640 1 --2- 172.21.15.43:0/3424093585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24680741b0 0x7f2468199c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:36.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.224+0000 7f246d600640 1 -- 172.21.15.43:0/3424093585 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24681a1840 con 0x7f2468109630 2024-03-28T22:33:36.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.224+0000 7f246d600640 1 --2- 172.21.15.43:0/3424093585 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2468109630 0x7f24681a11a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f246002c430 tx=0x7f24600337b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:36.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.225+0000 7f2466000640 1 -- 172.21.15.43:0/3424093585 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2460043020 con 0x7f2468109630 2024-03-28T22:33:36.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.225+0000 7f2466000640 1 -- 172.21.15.43:0/3424093585 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f246002ca60 con 0x7f2468109630 2024-03-28T22:33:36.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.225+0000 7f2466000640 1 -- 172.21.15.43:0/3424093585 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2460034690 con 0x7f2468109630 2024-03-28T22:33:36.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.225+0000 7f246f600640 1 -- 172.21.15.43:0/3424093585 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24681a1ad0 con 0x7f2468109630 2024-03-28T22:33:36.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.225+0000 7f246f600640 1 -- 172.21.15.43:0/3424093585 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24681a2070 con 0x7f2468109630 2024-03-28T22:33:36.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.226+0000 7f246f600640 1 -- 172.21.15.43:0/3424093585 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2468108360 con 0x7f2468109630 2024-03-28T22:33:36.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.228+0000 7f2466000640 1 -- 172.21.15.43:0/3424093585 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f2460046020 con 0x7f2468109630 2024-03-28T22:33:36.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.229+0000 7f2466000640 1 --2- 172.21.15.43:0/3424093585 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f244c0a1e00 0x7f244c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:36.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.229+0000 7f2466000640 1 -- 172.21.15.43:0/3424093585 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f24600f0550 con 0x7f2468109630 2024-03-28T22:33:36.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.229+0000 7f246cc00640 1 --2- 172.21.15.43:0/3424093585 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f244c0a1e00 0x7f244c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:36.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.230+0000 7f246cc00640 1 --2- 172.21.15.43:0/3424093585 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f244c0a1e00 0x7f244c0a42c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f246819b100 tx=0x7f2458009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:36.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.233+0000 7f2466000640 1 -- 172.21.15.43:0/3424093585 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f246004d020 con 0x7f2468109630 2024-03-28T22:33:36.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:36 smithi043 ceph-mon[31823]: pgmap v324: 129 pgs: 8 creating+peering, 97 active+clean, 24 unknown; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 9 op/s 2024-03-28T22:33:36.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:36 smithi043 ceph-mon[31823]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T22:33:36.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:36 smithi043 ceph-mon[31823]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T22:33:36.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:36 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-28T22:33:36.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:36 smithi043 ceph-mon[31823]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T22:33:36.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:36 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1366930051' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:36.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:36 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1660706477' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:33:36.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:36 smithi172 ceph-mon[36566]: pgmap v324: 129 pgs: 8 creating+peering, 97 active+clean, 24 unknown; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 9 op/s 2024-03-28T22:33:36.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:36 smithi172 ceph-mon[36566]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T22:33:36.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:36 smithi172 ceph-mon[36566]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T22:33:36.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:36 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-28T22:33:36.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:36 smithi172 ceph-mon[36566]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T22:33:36.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:36 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1366930051' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:36.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:36 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1660706477' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:33:36.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.439+0000 7f246f600640 1 -- 172.21.15.43:0/3424093585 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2468062350 con 0x7f2468109630 2024-03-28T22:33:36.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.441+0000 7f2466000640 1 -- 172.21.15.43:0/3424093585 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24093 (secure 0 0 0) 0x7f246009be20 con 0x7f2468109630 2024-03-28T22:33:36.440 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:33:36.440 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":92,"fsid":"c2276428-ed51-11ee-95d0-87774f69a715","created":"2024-03-28T22:24:19.950084+0000","modified":"2024-03-28T22:33:34.473170+0000","last_up_change":"2024-03-28T22:32:31.078737+0000","last_in_change":"2024-03-28T22:32:10.147097+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:28:14.713290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:33:27.167435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:33:28.411825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T22:33:32.523122+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b855adcd-786c-4e42-b8fe-a515a77ce979","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6803","nonce":2826124409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6805","nonce":2826124409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6809","nonce":2826124409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6807","nonce":2826124409}]},"public_addr":"172.21.15.43:6803/2826124409","cluster_addr":"172.21.15.43:6805/2826124409","heartbeat_back_addr":"172.21.15.43:6809/2826124409","heartbeat_front_addr":"172.21.15.43:6807/2826124409","state":["exists","up"]},{"osd":1,"uuid":"031c143d-8ad1-4b68-8db9-3c717137e3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6801","nonce":3043263644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6803","nonce":3043263644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6807","nonce":3043263644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6805","nonce":3043263644}]},"public_addr":"172.21.15.82:6801/3043263644","cluster_addr":"172.21.15.82:6803/3043263644","heartbeat_back_addr":"172.21.15.82:6807/3043263644","heartbeat_front_addr":"172.21.15.82:6805/3043263644","state":["exists","up"]},{"osd":2,"uuid":"20632bbb-f07a-4ca1-b0c4-ff9d508e211c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6801","nonce":4213953645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6803","nonce":4213953645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6807","nonce":4213953645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6805","nonce":4213953645}]},"public_addr":"172.21.15.172:6801/4213953645","cluster_addr":"172.21.15.172:6803/4213953645","heartbeat_back_addr":"172.21.15.172:6807/4213953645","heartbeat_front_addr":"172.21.15.172:6805/4213953645","state":["exists","up"]},{"osd":3,"uuid":"9a1f2312-4dbd-475b-8736-d5ebc8788e09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6811","nonce":1597280920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6813","nonce":1597280920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6817","nonce":1597280920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6815","nonce":1597280920}]},"public_addr":"172.21.15.43:6811/1597280920","cluster_addr":"172.21.15.43:6813/1597280920","heartbeat_back_addr":"172.21.15.43:6817/1597280920","heartbeat_front_addr":"172.21.15.43:6815/1597280920","state":["exists","up"]},{"osd":4,"uuid":"29140ead-7075-42b2-90c1-27cae0b29ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6809","nonce":215451895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6811","nonce":215451895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6815","nonce":215451895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6813","nonce":215451895}]},"public_addr":"172.21.15.82:6809/215451895","cluster_addr":"172.21.15.82:6811/215451895","heartbeat_back_addr":"172.21.15.82:6815/215451895","heartbeat_front_addr":"172.21.15.82:6813/215451895","state":["exists","up"]},{"osd":5,"uuid":"59644c61-93dc-4536-b2ad-0256b24e5cb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6809","nonce":1576148933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6811","nonce":1576148933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6815","nonce":1576148933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6813","nonce":1576148933}]},"public_addr":"172.21.15.172:6809/1576148933","cluster_addr":"172.21.15.172:6811/1576148933","heartbeat_back_addr":"172.21.15.172:6815/1576148933","heartbeat_front_addr":"172.21.15.172:6813/1576148933","state":["exists","up"]},{"osd":6,"uuid":"22d54e57-c3f1-4ecf-9921-8e815b48f92a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6819","nonce":791044279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6821","nonce":791044279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6825","nonce":791044279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6823","nonce":791044279}]},"public_addr":"172.21.15.43:6819/791044279","cluster_addr":"172.21.15.43:6821/791044279","heartbeat_back_addr":"172.21.15.43:6825/791044279","heartbeat_front_addr":"172.21.15.43:6823/791044279","state":["exists","up"]},{"osd":7,"uuid":"248cd1e2-0d6b-452f-a0a8-c51fe1b975ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6817","nonce":2157777323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6819","nonce":2157777323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6823","nonce":2157777323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6821","nonce":2157777323}]},"public_addr":"172.21.15.82:6817/2157777323","cluster_addr":"172.21.15.82:6819/2157777323","heartbeat_back_addr":"172.21.15.82:6823/2157777323","heartbeat_front_addr":"172.21.15.82:6821/2157777323","state":["exists","up"]},{"osd":8,"uuid":"d98aab6e-8bae-4c24-906c-4f492f047b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6817","nonce":1095328248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6819","nonce":1095328248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6823","nonce":1095328248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6821","nonce":1095328248}]},"public_addr":"172.21.15.172:6817/1095328248","cluster_addr":"172.21.15.172:6819/1095328248","heartbeat_back_addr":"172.21.15.172:6823/1095328248","heartbeat_front_addr":"172.21.15.172:6821/1095328248","state":["exists","up"]},{"osd":9,"uuid":"19aa383e-a0f6-45ea-96cc-e7956535caa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6827","nonce":3657791498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6829","nonce":3657791498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6833","nonce":3657791498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6831","nonce":3657791498}]},"public_addr":"172.21.15.43:6827/3657791498","cluster_addr":"172.21.15.43:6829/3657791498","heartbeat_back_addr":"172.21.15.43:6833/3657791498","heartbeat_front_addr":"172.21.15.43:6831/3657791498","state":["exists","up"]},{"osd":10,"uuid":"666977e5-eaf7-41b9-a18f-c380e149345d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6825","nonce":3454954047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6827","nonce":3454954047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6831","nonce":3454954047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6829","nonce":3454954047}]},"public_addr":"172.21.15.82:6825/3454954047","cluster_addr":"172.21.15.82:6827/3454954047","heartbeat_back_addr":"172.21.15.82:6831/3454954047","heartbeat_front_addr":"172.21.15.82:6829/3454954047","state":["exists","up"]},{"osd":11,"uuid":"615cf3be-8076-470e-ad09-c880e7593717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6825","nonce":2563517144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6827","nonce":2563517144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6831","nonce":2563517144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6829","nonce":2563517144}]},"public_addr":"172.21.15.172:6825/2563517144","cluster_addr":"172.21.15.172:6827/2563517144","heartbeat_back_addr":"172.21.15.172:6831/2563517144","heartbeat_front_addr":"172.21.15.172:6829/2563517144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:21.727405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:47.100259+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:10.615438+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:43.632144+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:09.487940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:34.604463+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:09.594452+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:35.709795+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:01.184202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:35.455322+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:02.413166+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:28.550754+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/578791143":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/4029313215":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/1574178482":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6801/3517712438":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6800/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/1375099638":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/2710117652":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6801/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/3215634168":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6800/3517712438":"2024-03-29T22:25:28.008215+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:33:36.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.445+0000 7f246f600640 1 -- 172.21.15.43:0/3424093585 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f244c0a1e00 msgr2=0x7f244c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:36.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.445+0000 7f246f600640 1 --2- 172.21.15.43:0/3424093585 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f244c0a1e00 0x7f244c0a42c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f246819b100 tx=0x7f2458009260 comp rx=0 tx=0).stop 2024-03-28T22:33:36.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.445+0000 7f246f600640 1 -- 172.21.15.43:0/3424093585 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2468109630 msgr2=0x7f24681a11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:36.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.445+0000 7f246f600640 1 --2- 172.21.15.43:0/3424093585 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2468109630 0x7f24681a11a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f246002c430 tx=0x7f24600337b0 comp rx=0 tx=0).stop 2024-03-28T22:33:36.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.445+0000 7f246f600640 1 -- 172.21.15.43:0/3424093585 shutdown_connections 2024-03-28T22:33:36.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.445+0000 7f246f600640 1 --2- 172.21.15.43:0/3424093585 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f244c0a1e00 0x7f244c0a42c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:36.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.445+0000 7f246f600640 1 --2- 172.21.15.43:0/3424093585 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2468109630 0x7f24681a11a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:36.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.445+0000 7f246f600640 1 --2- 172.21.15.43:0/3424093585 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2468075790 0x7f246819a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:36.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.445+0000 7f246f600640 1 --2- 172.21.15.43:0/3424093585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24680741b0 0x7f2468199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:36.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.445+0000 7f246f600640 1 -- 172.21.15.43:0/3424093585 >> 172.21.15.43:0/3424093585 conn(0x7f24680fcd00 msgr2=0x7f2468105530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:36.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.445+0000 7f246f600640 1 -- 172.21.15.43:0/3424093585 shutdown_connections 2024-03-28T22:33:36.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.445+0000 7f246f600640 1 -- 172.21.15.43:0/3424093585 wait complete. 2024-03-28T22:33:36.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:36 smithi082 ceph-mon[36471]: pgmap v324: 129 pgs: 8 creating+peering, 97 active+clean, 24 unknown; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 9 op/s 2024-03-28T22:33:36.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:36 smithi082 ceph-mon[36471]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T22:33:36.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:36 smithi082 ceph-mon[36471]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T22:33:36.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:36 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-28T22:33:36.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:36 smithi082 ceph-mon[36471]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T22:33:36.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:36 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1366930051' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:36.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:36 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1660706477' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:33:36.480 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:33:36.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.706+0000 7fdb5be00640 1 -- 172.21.15.43:0/1931817086 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdb54104b80 msgr2=0x7fdb54105000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:36.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.706+0000 7fdb5be00640 1 --2- 172.21.15.43:0/1931817086 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdb54104b80 0x7fdb54105000 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdb40009860 tx=0x7fdb4002bb70 comp rx=0 tx=0).stop 2024-03-28T22:33:36.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.707+0000 7fdb5be00640 1 -- 172.21.15.43:0/1931817086 shutdown_connections 2024-03-28T22:33:36.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.707+0000 7fdb5be00640 1 --2- 172.21.15.43:0/1931817086 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdb54105540 0x7fdb5410bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:36.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.707+0000 7fdb5be00640 1 --2- 172.21.15.43:0/1931817086 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdb54104b80 0x7fdb54105000 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:36.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.707+0000 7fdb5be00640 1 --2- 172.21.15.43:0/1931817086 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb54103990 0x7fdb54103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:36.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.707+0000 7fdb5be00640 1 -- 172.21.15.43:0/1931817086 >> 172.21.15.43:0/1931817086 conn(0x7fdb540fef00 msgr2=0x7fdb54101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:36.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.707+0000 7fdb5be00640 1 -- 172.21.15.43:0/1931817086 shutdown_connections 2024-03-28T22:33:36.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.707+0000 7fdb5be00640 1 -- 172.21.15.43:0/1931817086 wait complete. 2024-03-28T22:33:36.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.708+0000 7fdb5be00640 1 Processor -- start 2024-03-28T22:33:36.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.708+0000 7fdb5be00640 1 -- start start 2024-03-28T22:33:36.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.709+0000 7fdb5be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdb54103990 0x7fdb54199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:36.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.709+0000 7fdb5be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdb54104b80 0x7fdb5419a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:36.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.709+0000 7fdb5be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb54105540 0x7fdb541a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:36.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.709+0000 7fdb5be00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fdb5410f5f0 con 0x7fdb54105540 2024-03-28T22:33:36.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.709+0000 7fdb5be00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fdb5410f470 con 0x7fdb54104b80 2024-03-28T22:33:36.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.709+0000 7fdb5be00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fdb5410f770 con 0x7fdb54103990 2024-03-28T22:33:36.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.709+0000 7fdb58a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdb54104b80 0x7fdb5419a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:36.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.709+0000 7fdb58a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdb54104b80 0x7fdb5419a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:43200/0 (socket says 172.21.15.43:43200) 2024-03-28T22:33:36.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.709+0000 7fdb58a00640 1 -- 172.21.15.43:0/2480919750 learned_addr learned my addr 172.21.15.43:0/2480919750 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:36.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.709+0000 7fdb59400640 1 --2- 172.21.15.43:0/2480919750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdb54103990 0x7fdb54199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:36.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.709+0000 7fdb59e00640 1 --2- 172.21.15.43:0/2480919750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb54105540 0x7fdb541a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:36.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.709+0000 7fdb58a00640 1 -- 172.21.15.43:0/2480919750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdb54103990 msgr2=0x7fdb54199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:36.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.709+0000 7fdb58a00640 1 --2- 172.21.15.43:0/2480919750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdb54103990 0x7fdb54199c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:36.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.709+0000 7fdb58a00640 1 -- 172.21.15.43:0/2480919750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb54105540 msgr2=0x7fdb541a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:36.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.709+0000 7fdb58a00640 1 --2- 172.21.15.43:0/2480919750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb54105540 0x7fdb541a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:36.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.710+0000 7fdb58a00640 1 -- 172.21.15.43:0/2480919750 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb541a18d0 con 0x7fdb54104b80 2024-03-28T22:33:36.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.710+0000 7fdb59e00640 1 --2- 172.21.15.43:0/2480919750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb54105540 0x7fdb541a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:33:36.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.710+0000 7fdb58a00640 1 --2- 172.21.15.43:0/2480919750 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdb54104b80 0x7fdb5419a1d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdb400349e0 tx=0x7fdb40032450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:36.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.710+0000 7fdb52000640 1 -- 172.21.15.43:0/2480919750 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb40041030 con 0x7fdb54104b80 2024-03-28T22:33:36.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.710+0000 7fdb5be00640 1 -- 172.21.15.43:0/2480919750 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb541a1b60 con 0x7fdb54104b80 2024-03-28T22:33:36.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.710+0000 7fdb52000640 1 -- 172.21.15.43:0/2480919750 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb4002c510 con 0x7fdb54104b80 2024-03-28T22:33:36.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.710+0000 7fdb52000640 1 -- 172.21.15.43:0/2480919750 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb40039040 con 0x7fdb54104b80 2024-03-28T22:33:36.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.710+0000 7fdb5be00640 1 -- 172.21.15.43:0/2480919750 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb541a2050 con 0x7fdb54104b80 2024-03-28T22:33:36.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.712+0000 7fdb5be00640 1 -- 172.21.15.43:0/2480919750 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb5410ad30 con 0x7fdb54104b80 2024-03-28T22:33:36.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.713+0000 7fdb52000640 1 -- 172.21.15.43:0/2480919750 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fdb40044070 con 0x7fdb54104b80 2024-03-28T22:33:36.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.714+0000 7fdb52000640 1 --2- 172.21.15.43:0/2480919750 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fdb3c0a1d80 0x7fdb3c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:36.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.715+0000 7fdb52000640 1 -- 172.21.15.43:0/2480919750 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7fdb400ef600 con 0x7fdb54104b80 2024-03-28T22:33:36.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.715+0000 7fdb59400640 1 --2- 172.21.15.43:0/2480919750 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fdb3c0a1d80 0x7fdb3c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:36.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.715+0000 7fdb59400640 1 --2- 172.21.15.43:0/2480919750 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fdb3c0a1d80 0x7fdb3c0a4240 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fdb48009a60 tx=0x7fdb48010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:36.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.718+0000 7fdb52000640 1 -- 172.21.15.43:0/2480919750 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdb400bc790 con 0x7fdb54104b80 2024-03-28T22:33:36.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.971+0000 7fdb5be00640 1 -- 172.21.15.43:0/2480919750 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdb54062350 con 0x7fdb54104b80 2024-03-28T22:33:36.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.972+0000 7fdb52000640 1 -- 172.21.15.43:0/2480919750 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11797 (secure 0 0 0) 0x7fdb400bc090 con 0x7fdb54104b80 2024-03-28T22:33:36.971 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:33:36.971 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6835/1084988072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1084988072},{"type":"v1","addr":"172.21.15.82:6835","nonce":1084988072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24518,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24556,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T22:33:28.906396+0000","modified":"2024-03-28T22:33:34.473616+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24572,"qdb_cluster":[24572]},"id":1}]} 2024-03-28T22:33:36.971 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 17 2024-03-28T22:33:36.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.975+0000 7fdb5be00640 1 -- 172.21.15.43:0/2480919750 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fdb3c0a1d80 msgr2=0x7fdb3c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:36.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.975+0000 7fdb5be00640 1 --2- 172.21.15.43:0/2480919750 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fdb3c0a1d80 0x7fdb3c0a4240 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fdb48009a60 tx=0x7fdb48010040 comp rx=0 tx=0).stop 2024-03-28T22:33:36.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.975+0000 7fdb5be00640 1 -- 172.21.15.43:0/2480919750 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdb54104b80 msgr2=0x7fdb5419a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:36.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.975+0000 7fdb5be00640 1 --2- 172.21.15.43:0/2480919750 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdb54104b80 0x7fdb5419a1d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdb400349e0 tx=0x7fdb40032450 comp rx=0 tx=0).stop 2024-03-28T22:33:36.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.975+0000 7fdb5be00640 1 -- 172.21.15.43:0/2480919750 shutdown_connections 2024-03-28T22:33:36.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.975+0000 7fdb5be00640 1 --2- 172.21.15.43:0/2480919750 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fdb3c0a1d80 0x7fdb3c0a4240 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:36.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.975+0000 7fdb5be00640 1 --2- 172.21.15.43:0/2480919750 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdb54104b80 0x7fdb5419a1d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:36.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.975+0000 7fdb5be00640 1 --2- 172.21.15.43:0/2480919750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb54105540 0x7fdb541a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:36.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.975+0000 7fdb5be00640 1 --2- 172.21.15.43:0/2480919750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdb54103990 0x7fdb54199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:36.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.975+0000 7fdb5be00640 1 -- 172.21.15.43:0/2480919750 >> 172.21.15.43:0/2480919750 conn(0x7fdb540fef00 msgr2=0x7fdb54107f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:36.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.975+0000 7fdb5be00640 1 -- 172.21.15.43:0/2480919750 shutdown_connections 2024-03-28T22:33:36.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:36.975+0000 7fdb5be00640 1 -- 172.21.15.43:0/2480919750 wait complete. 2024-03-28T22:33:36.993 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:33:37.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.218+0000 7f3be5200640 1 -- 172.21.15.43:0/724600567 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3be0102780 msgr2=0x7f3be0102c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:37.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.218+0000 7f3be5200640 1 --2- 172.21.15.43:0/724600567 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3be0102780 0x7f3be0102c00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc009900 tx=0x7f3bcc02bc30 comp rx=0 tx=0).stop 2024-03-28T22:33:37.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.219+0000 7f3be5200640 1 -- 172.21.15.43:0/724600567 shutdown_connections 2024-03-28T22:33:37.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.219+0000 7f3be5200640 1 --2- 172.21.15.43:0/724600567 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3be0103140 0x7f3be01099d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.219+0000 7f3be5200640 1 --2- 172.21.15.43:0/724600567 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3be0102780 0x7f3be0102c00 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.219+0000 7f3be5200640 1 --2- 172.21.15.43:0/724600567 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3be0101590 0x7f3be01019b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.219+0000 7f3be5200640 1 -- 172.21.15.43:0/724600567 >> 172.21.15.43:0/724600567 conn(0x7f3be00fcae0 msgr2=0x7f3be00fef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:37.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.219+0000 7f3be5200640 1 -- 172.21.15.43:0/724600567 shutdown_connections 2024-03-28T22:33:37.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.219+0000 7f3be5200640 1 -- 172.21.15.43:0/724600567 wait complete. 2024-03-28T22:33:37.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.220+0000 7f3be5200640 1 Processor -- start 2024-03-28T22:33:37.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.220+0000 7f3be5200640 1 -- start start 2024-03-28T22:33:37.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.220+0000 7f3be5200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3be0101590 0x7f3be0199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:37.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.220+0000 7f3be5200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3be0102780 0x7f3be019a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:37.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.220+0000 7f3be5200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3be0103140 0x7f3be01a1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:37.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.220+0000 7f3be5200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3be010d240 con 0x7f3be0103140 2024-03-28T22:33:37.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.220+0000 7f3be5200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f3be010d0c0 con 0x7f3be0101590 2024-03-28T22:33:37.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.220+0000 7f3be5200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f3be010d3c0 con 0x7f3be0102780 2024-03-28T22:33:37.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.220+0000 7f3bdf400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3be0101590 0x7f3be0199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:37.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.220+0000 7f3bdfe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3be0103140 0x7f3be01a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:37.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.220+0000 7f3bdf400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3be0101590 0x7f3be0199b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:43210/0 (socket says 172.21.15.43:43210) 2024-03-28T22:33:37.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.220+0000 7f3bdfe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3be0103140 0x7f3be01a1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45586/0 (socket says 172.21.15.43:45586) 2024-03-28T22:33:37.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.220+0000 7f3bdf400640 1 -- 172.21.15.43:0/1151629821 learned_addr learned my addr 172.21.15.43:0/1151629821 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:37.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.221+0000 7f3bdea00640 1 --2- 172.21.15.43:0/1151629821 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3be0102780 0x7f3be019a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:37.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.221+0000 7f3bdf400640 1 -- 172.21.15.43:0/1151629821 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3be0102780 msgr2=0x7f3be019a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:37.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.221+0000 7f3bdf400640 1 --2- 172.21.15.43:0/1151629821 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3be0102780 0x7f3be019a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.221+0000 7f3bdf400640 1 -- 172.21.15.43:0/1151629821 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3be0103140 msgr2=0x7f3be01a1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:37.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.221+0000 7f3bdf400640 1 --2- 172.21.15.43:0/1151629821 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3be0103140 0x7f3be01a1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.221+0000 7f3bdf400640 1 -- 172.21.15.43:0/1151629821 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3be01a16e0 con 0x7f3be0101590 2024-03-28T22:33:37.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.221+0000 7f3bdf400640 1 --2- 172.21.15.43:0/1151629821 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3be0101590 0x7f3be0199b60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3bd000e9c0 tx=0x7f3bd000edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:37.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.221+0000 7f3bd7e00640 1 -- 172.21.15.43:0/1151629821 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd000ccc0 con 0x7f3be0101590 2024-03-28T22:33:37.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.221+0000 7f3bd7e00640 1 -- 172.21.15.43:0/1151629821 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bd00035f0 con 0x7f3be0101590 2024-03-28T22:33:37.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.221+0000 7f3bd7e00640 1 -- 172.21.15.43:0/1151629821 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd000ce60 con 0x7f3be0101590 2024-03-28T22:33:37.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.221+0000 7f3be5200640 1 -- 172.21.15.43:0/1151629821 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3be01a19d0 con 0x7f3be0101590 2024-03-28T22:33:37.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.222+0000 7f3be5200640 1 -- 172.21.15.43:0/1151629821 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3be01a1ef0 con 0x7f3be0101590 2024-03-28T22:33:37.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.223+0000 7f3be5200640 1 -- 172.21.15.43:0/1151629821 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3be00688b0 con 0x7f3be0101590 2024-03-28T22:33:37.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.224+0000 7f3bd7e00640 1 -- 172.21.15.43:0/1151629821 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f3bd00037a0 con 0x7f3be0101590 2024-03-28T22:33:37.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.226+0000 7f3bd7e00640 1 --2- 172.21.15.43:0/1151629821 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3bc40a1d80 0x7f3bc40a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:37.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.226+0000 7f3bd7e00640 1 -- 172.21.15.43:0/1151629821 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f3bd00ce390 con 0x7f3be0101590 2024-03-28T22:33:37.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.226+0000 7f3bdea00640 1 --2- 172.21.15.43:0/1151629821 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3bc40a1d80 0x7f3bc40a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:37.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.227+0000 7f3bdea00640 1 --2- 172.21.15.43:0/1151629821 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3bc40a1d80 0x7f3bc40a4240 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc02c240 tx=0x7f3bcc02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:37.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.234+0000 7f3bd7e00640 1 -- 172.21.15.43:0/1151629821 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3bd009b520 con 0x7f3be0101590 2024-03-28T22:33:37.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:37 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3424093585' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:33:37.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:37 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2480919750' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:37.297 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:37 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3424093585' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:33:37.297 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:37 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2480919750' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:37.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.437+0000 7f3be5200640 1 -- 172.21.15.43:0/1151629821 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3be01a2910 con 0x7f3be0101590 2024-03-28T22:33:37.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.439+0000 7f3bd7e00640 1 -- 172.21.15.43:0/1151629821 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24093 (secure 0 0 0) 0x7f3bd009ae20 con 0x7f3be0101590 2024-03-28T22:33:37.437 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:33:37.437 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":92,"fsid":"c2276428-ed51-11ee-95d0-87774f69a715","created":"2024-03-28T22:24:19.950084+0000","modified":"2024-03-28T22:33:34.473170+0000","last_up_change":"2024-03-28T22:32:31.078737+0000","last_in_change":"2024-03-28T22:32:10.147097+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:28:14.713290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:33:27.167435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:33:28.411825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T22:33:32.523122+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b855adcd-786c-4e42-b8fe-a515a77ce979","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6803","nonce":2826124409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6805","nonce":2826124409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6809","nonce":2826124409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6807","nonce":2826124409}]},"public_addr":"172.21.15.43:6803/2826124409","cluster_addr":"172.21.15.43:6805/2826124409","heartbeat_back_addr":"172.21.15.43:6809/2826124409","heartbeat_front_addr":"172.21.15.43:6807/2826124409","state":["exists","up"]},{"osd":1,"uuid":"031c143d-8ad1-4b68-8db9-3c717137e3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6801","nonce":3043263644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6803","nonce":3043263644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6807","nonce":3043263644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6805","nonce":3043263644}]},"public_addr":"172.21.15.82:6801/3043263644","cluster_addr":"172.21.15.82:6803/3043263644","heartbeat_back_addr":"172.21.15.82:6807/3043263644","heartbeat_front_addr":"172.21.15.82:6805/3043263644","state":["exists","up"]},{"osd":2,"uuid":"20632bbb-f07a-4ca1-b0c4-ff9d508e211c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6801","nonce":4213953645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6803","nonce":4213953645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6807","nonce":4213953645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6805","nonce":4213953645}]},"public_addr":"172.21.15.172:6801/4213953645","cluster_addr":"172.21.15.172:6803/4213953645","heartbeat_back_addr":"172.21.15.172:6807/4213953645","heartbeat_front_addr":"172.21.15.172:6805/4213953645","state":["exists","up"]},{"osd":3,"uuid":"9a1f2312-4dbd-475b-8736-d5ebc8788e09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6811","nonce":1597280920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6813","nonce":1597280920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6817","nonce":1597280920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6815","nonce":1597280920}]},"public_addr":"172.21.15.43:6811/1597280920","cluster_addr":"172.21.15.43:6813/1597280920","heartbeat_back_addr":"172.21.15.43:6817/1597280920","heartbeat_front_addr":"172.21.15.43:6815/1597280920","state":["exists","up"]},{"osd":4,"uuid":"29140ead-7075-42b2-90c1-27cae0b29ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6809","nonce":215451895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6811","nonce":215451895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6815","nonce":215451895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6813","nonce":215451895}]},"public_addr":"172.21.15.82:6809/215451895","cluster_addr":"172.21.15.82:6811/215451895","heartbeat_back_addr":"172.21.15.82:6815/215451895","heartbeat_front_addr":"172.21.15.82:6813/215451895","state":["exists","up"]},{"osd":5,"uuid":"59644c61-93dc-4536-b2ad-0256b24e5cb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6809","nonce":1576148933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6811","nonce":1576148933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6815","nonce":1576148933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6813","nonce":1576148933}]},"public_addr":"172.21.15.172:6809/1576148933","cluster_addr":"172.21.15.172:6811/1576148933","heartbeat_back_addr":"172.21.15.172:6815/1576148933","heartbeat_front_addr":"172.21.15.172:6813/1576148933","state":["exists","up"]},{"osd":6,"uuid":"22d54e57-c3f1-4ecf-9921-8e815b48f92a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6819","nonce":791044279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6821","nonce":791044279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6825","nonce":791044279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6823","nonce":791044279}]},"public_addr":"172.21.15.43:6819/791044279","cluster_addr":"172.21.15.43:6821/791044279","heartbeat_back_addr":"172.21.15.43:6825/791044279","heartbeat_front_addr":"172.21.15.43:6823/791044279","state":["exists","up"]},{"osd":7,"uuid":"248cd1e2-0d6b-452f-a0a8-c51fe1b975ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6817","nonce":2157777323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6819","nonce":2157777323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6823","nonce":2157777323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6821","nonce":2157777323}]},"public_addr":"172.21.15.82:6817/2157777323","cluster_addr":"172.21.15.82:6819/2157777323","heartbeat_back_addr":"172.21.15.82:6823/2157777323","heartbeat_front_addr":"172.21.15.82:6821/2157777323","state":["exists","up"]},{"osd":8,"uuid":"d98aab6e-8bae-4c24-906c-4f492f047b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6817","nonce":1095328248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6819","nonce":1095328248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6823","nonce":1095328248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6821","nonce":1095328248}]},"public_addr":"172.21.15.172:6817/1095328248","cluster_addr":"172.21.15.172:6819/1095328248","heartbeat_back_addr":"172.21.15.172:6823/1095328248","heartbeat_front_addr":"172.21.15.172:6821/1095328248","state":["exists","up"]},{"osd":9,"uuid":"19aa383e-a0f6-45ea-96cc-e7956535caa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6827","nonce":3657791498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6829","nonce":3657791498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6833","nonce":3657791498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6831","nonce":3657791498}]},"public_addr":"172.21.15.43:6827/3657791498","cluster_addr":"172.21.15.43:6829/3657791498","heartbeat_back_addr":"172.21.15.43:6833/3657791498","heartbeat_front_addr":"172.21.15.43:6831/3657791498","state":["exists","up"]},{"osd":10,"uuid":"666977e5-eaf7-41b9-a18f-c380e149345d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6825","nonce":3454954047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6827","nonce":3454954047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6831","nonce":3454954047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6829","nonce":3454954047}]},"public_addr":"172.21.15.82:6825/3454954047","cluster_addr":"172.21.15.82:6827/3454954047","heartbeat_back_addr":"172.21.15.82:6831/3454954047","heartbeat_front_addr":"172.21.15.82:6829/3454954047","state":["exists","up"]},{"osd":11,"uuid":"615cf3be-8076-470e-ad09-c880e7593717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6825","nonce":2563517144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6827","nonce":2563517144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6831","nonce":2563517144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6829","nonce":2563517144}]},"public_addr":"172.21.15.172:6825/2563517144","cluster_addr":"172.21.15.172:6827/2563517144","heartbeat_back_addr":"172.21.15.172:6831/2563517144","heartbeat_front_addr":"172.21.15.172:6829/2563517144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:21.727405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:47.100259+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:10.615438+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:43.632144+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:09.487940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:34.604463+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:09.594452+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:35.709795+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:01.184202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:35.455322+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:02.413166+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:28.550754+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/578791143":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/4029313215":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/1574178482":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6801/3517712438":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6800/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/1375099638":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/2710117652":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6801/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/3215634168":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6800/3517712438":"2024-03-29T22:25:28.008215+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:33:37.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.441+0000 7f3be5200640 1 -- 172.21.15.43:0/1151629821 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3bc40a1d80 msgr2=0x7f3bc40a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:37.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.441+0000 7f3be5200640 1 --2- 172.21.15.43:0/1151629821 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3bc40a1d80 0x7f3bc40a4240 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc02c240 tx=0x7f3bcc02df90 comp rx=0 tx=0).stop 2024-03-28T22:33:37.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.441+0000 7f3be5200640 1 -- 172.21.15.43:0/1151629821 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3be0101590 msgr2=0x7f3be0199b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:37.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.441+0000 7f3be5200640 1 --2- 172.21.15.43:0/1151629821 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3be0101590 0x7f3be0199b60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3bd000e9c0 tx=0x7f3bd000edc0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.441+0000 7f3be5200640 1 -- 172.21.15.43:0/1151629821 shutdown_connections 2024-03-28T22:33:37.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.441+0000 7f3be5200640 1 --2- 172.21.15.43:0/1151629821 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3be0103140 0x7f3be01a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.442+0000 7f3be5200640 1 --2- 172.21.15.43:0/1151629821 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3be0102780 0x7f3be019a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.442+0000 7f3be5200640 1 --2- 172.21.15.43:0/1151629821 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3bc40a1d80 0x7f3bc40a4240 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.442+0000 7f3be5200640 1 --2- 172.21.15.43:0/1151629821 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3be0101590 0x7f3be0199b60 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.442+0000 7f3be5200640 1 -- 172.21.15.43:0/1151629821 >> 172.21.15.43:0/1151629821 conn(0x7f3be00fcae0 msgr2=0x7f3be0073980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:37.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.442+0000 7f3be5200640 1 -- 172.21.15.43:0/1151629821 shutdown_connections 2024-03-28T22:33:37.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.442+0000 7f3be5200640 1 -- 172.21.15.43:0/1151629821 wait complete. 2024-03-28T22:33:37.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:37 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3424093585' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:33:37.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:37 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2480919750' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:33:37.461 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T22:33:28.906396+0000', 'modified': '2024-03-28T22:33:34.473616+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24572}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24572': {'gid': 24572, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.82:6839/2220892357', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6838', 'nonce': 2220892357}, {'type': 'v1', 'addr': '172.21.15.82:6839', 'nonce': 2220892357}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24572, 'qdb_cluster': [24572]} 2024-03-28T22:33:37.461 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T22:33:37.461 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T22:33:37.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.692+0000 7f786a400640 1 -- 172.21.15.43:0/1962872649 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7864105540 msgr2=0x7f786410bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:37.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.692+0000 7f786a400640 1 --2- 172.21.15.43:0/1962872649 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7864105540 0x7f786410bdd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f785c009ca0 tx=0x7f785c02c100 comp rx=0 tx=0).stop 2024-03-28T22:33:37.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.692+0000 7f786a400640 1 -- 172.21.15.43:0/1962872649 shutdown_connections 2024-03-28T22:33:37.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.692+0000 7f786a400640 1 --2- 172.21.15.43:0/1962872649 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7864105540 0x7f786410bdd0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.692+0000 7f786a400640 1 --2- 172.21.15.43:0/1962872649 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7864104b80 0x7f7864105000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.692+0000 7f786a400640 1 --2- 172.21.15.43:0/1962872649 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7864103990 0x7f7864103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.692+0000 7f786a400640 1 -- 172.21.15.43:0/1962872649 >> 172.21.15.43:0/1962872649 conn(0x7f78640fef00 msgr2=0x7f7864101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:37.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.693+0000 7f786a400640 1 -- 172.21.15.43:0/1962872649 shutdown_connections 2024-03-28T22:33:37.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.693+0000 7f786a400640 1 -- 172.21.15.43:0/1962872649 wait complete. 2024-03-28T22:33:37.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.694+0000 7f786a400640 1 Processor -- start 2024-03-28T22:33:37.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.694+0000 7f786a400640 1 -- start start 2024-03-28T22:33:37.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.694+0000 7f786a400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7864103990 0x7f7864199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:37.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.694+0000 7f786a400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7864104b80 0x7f786419a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:37.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.694+0000 7f786a400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7864105540 0x7f78641a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:37.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.694+0000 7f786a400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f786410f610 con 0x7f7864103990 2024-03-28T22:33:37.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.694+0000 7f786a400640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f786410f490 con 0x7f7864104b80 2024-03-28T22:33:37.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.694+0000 7f786a400640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f786410f790 con 0x7f7864105540 2024-03-28T22:33:37.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.694+0000 7f785be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7864104b80 0x7f786419a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:37.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.694+0000 7f785be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7864104b80 0x7f786419a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:43230/0 (socket says 172.21.15.43:43230) 2024-03-28T22:33:37.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.694+0000 7f785be00640 1 -- 172.21.15.43:0/2236354120 learned_addr learned my addr 172.21.15.43:0/2236354120 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:37.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.694+0000 7f7863400640 1 --2- 172.21.15.43:0/2236354120 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7864103990 0x7f7864199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:37.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.695+0000 7f7863e00640 1 --2- 172.21.15.43:0/2236354120 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7864105540 0x7f78641a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:37.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.695+0000 7f785be00640 1 -- 172.21.15.43:0/2236354120 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7864105540 msgr2=0x7f78641a1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:37.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.695+0000 7f785be00640 1 --2- 172.21.15.43:0/2236354120 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7864105540 0x7f78641a1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.695+0000 7f785be00640 1 -- 172.21.15.43:0/2236354120 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7864103990 msgr2=0x7f7864199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:37.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.695+0000 7f785be00640 1 --2- 172.21.15.43:0/2236354120 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7864103990 0x7f7864199cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.695+0000 7f785be00640 1 -- 172.21.15.43:0/2236354120 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78641a18f0 con 0x7f7864104b80 2024-03-28T22:33:37.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.695+0000 7f7863400640 1 --2- 172.21.15.43:0/2236354120 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7864103990 0x7f7864199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:33:37.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.695+0000 7f785be00640 1 --2- 172.21.15.43:0/2236354120 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7864104b80 0x7f786419a1f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f784c00d950 tx=0x7f784c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:37.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.695+0000 7f7860c00640 1 -- 172.21.15.43:0/2236354120 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f784c017030 con 0x7f7864104b80 2024-03-28T22:33:37.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.695+0000 7f7860c00640 1 -- 172.21.15.43:0/2236354120 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f784c004020 con 0x7f7864104b80 2024-03-28T22:33:37.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.695+0000 7f7860c00640 1 -- 172.21.15.43:0/2236354120 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f784c0047a0 con 0x7f7864104b80 2024-03-28T22:33:37.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.695+0000 7f786a400640 1 -- 172.21.15.43:0/2236354120 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78641a1be0 con 0x7f7864104b80 2024-03-28T22:33:37.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.695+0000 7f786a400640 1 -- 172.21.15.43:0/2236354120 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78641a20e0 con 0x7f7864104b80 2024-03-28T22:33:37.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.700+0000 7f786d071740 1 Processor -- start 2024-03-28T22:33:37.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.700+0000 7f7860c00640 1 -- 172.21.15.43:0/2236354120 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f784c0041d0 con 0x7f7864104b80 2024-03-28T22:33:37.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.700+0000 7f786d071740 1 -- start start 2024-03-28T22:33:37.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.700+0000 7f786d071740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d15f1f9570 0x55d15f1f9990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:37.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.700+0000 7f786d071740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d15f1f9f60 0x55d15f1fea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:37.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.700+0000 7f786d071740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d15f1ff0d0 0x55d15f201560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:37.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.700+0000 7f786d071740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d15f15d9c0 con 0x55d15f1f9f60 2024-03-28T22:33:37.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.700+0000 7f786d071740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55d15f106c30 con 0x55d15f1f9570 2024-03-28T22:33:37.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.700+0000 7f786d071740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55d15f159a60 con 0x55d15f1ff0d0 2024-03-28T22:33:37.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.701+0000 7f785be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d15f1ff0d0 0x55d15f201560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:37.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.701+0000 7f7863400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d15f1f9f60 0x55d15f1fea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:37.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.701+0000 7f7863e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d15f1f9570 0x55d15f1f9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:37.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.701+0000 7f7863e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d15f1f9570 0x55d15f1f9990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:43240/0 (socket says 172.21.15.43:43240) 2024-03-28T22:33:37.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.701+0000 7f7863400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d15f1f9f60 0x55d15f1fea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45604/0 (socket says 172.21.15.43:45604) 2024-03-28T22:33:37.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.701+0000 7f7863e00640 1 -- 172.21.15.43:0/2441456657 learned_addr learned my addr 172.21.15.43:0/2441456657 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:37.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.701+0000 7f7863e00640 1 -- 172.21.15.43:0/2441456657 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d15f1ff0d0 msgr2=0x55d15f201560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:33:37.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.701+0000 7f7863e00640 1 --2- 172.21.15.43:0/2441456657 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d15f1ff0d0 0x55d15f201560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.701+0000 7f7863e00640 1 -- 172.21.15.43:0/2441456657 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d15f1f9f60 msgr2=0x55d15f1fea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:33:37.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.701+0000 7f7863e00640 1 --2- 172.21.15.43:0/2441456657 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d15f1f9f60 0x55d15f1fea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.701+0000 7f7863e00640 1 -- 172.21.15.43:0/2441456657 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d15ee722d0 con 0x55d15f1f9570 2024-03-28T22:33:37.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.701+0000 7f7863e00640 1 --2- 172.21.15.43:0/2441456657 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d15f1f9570 0x55d15f1f9990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f785c02c710 tx=0x7f785c0323d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae9cfa0b4d56ea20 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:37.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.702+0000 7f7860c00640 1 --2- 172.21.15.43:0/2236354120 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f78480a1d80 0x7f78480a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:37.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.702+0000 7f7858c00640 1 -- 172.21.15.43:0/2441456657 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c043070 con 0x55d15f1f9570 2024-03-28T22:33:37.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.702+0000 7f7858c00640 1 -- 172.21.15.43:0/2441456657 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f785c032c90 con 0x55d15f1f9570 2024-03-28T22:33:37.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.702+0000 7f7858c00640 1 -- 172.21.15.43:0/2441456657 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c039040 con 0x55d15f1f9570 2024-03-28T22:33:37.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.702+0000 7f7860c00640 1 -- 172.21.15.43:0/2236354120 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f784c0cda10 con 0x7f7864104b80 2024-03-28T22:33:37.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.702+0000 7f7863400640 1 --2- 172.21.15.43:0/2236354120 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f78480a1d80 0x7f78480a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:37.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.702+0000 7f786d071740 1 -- 172.21.15.43:0/2441456657 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d15f1f9570 msgr2=0x55d15f1f9990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:37.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.702+0000 7f786d071740 1 --2- 172.21.15.43:0/2441456657 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d15f1f9570 0x55d15f1f9990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f785c02c710 tx=0x7f785c0323d0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.703+0000 7f786d071740 1 -- 172.21.15.43:0/2441456657 shutdown_connections 2024-03-28T22:33:37.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.703+0000 7f786d071740 1 --2- 172.21.15.43:0/2441456657 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d15f1ff0d0 0x55d15f201560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.703+0000 7f786d071740 1 --2- 172.21.15.43:0/2441456657 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d15f1f9f60 0x55d15f1fea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.703+0000 7f786d071740 1 --2- 172.21.15.43:0/2441456657 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d15f1f9570 0x55d15f1f9990 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.703+0000 7f786d071740 1 -- 172.21.15.43:0/2441456657 >> 172.21.15.43:0/2441456657 conn(0x55d15f1f4ae0 msgr2=0x55d15f1f4ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:37.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.703+0000 7f786d071740 1 -- 172.21.15.43:0/2441456657 shutdown_connections 2024-03-28T22:33:37.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.703+0000 7f7863400640 1 --2- 172.21.15.43:0/2236354120 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f78480a1d80 0x7f78480a4240 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x55d15f1fa360 tx=0x7f7854005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:37.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.703+0000 7f786d071740 1 -- 172.21.15.43:0/2441456657 wait complete. 2024-03-28T22:33:37.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.703+0000 7f786d071740 1 -- start start 2024-03-28T22:33:37.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.703+0000 7f786d071740 1 Processor -- start 2024-03-28T22:33:37.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.704+0000 7f786d071740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d15f1f9570 0x55d15f203ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:37.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.704+0000 7f786d071740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d15f1f9f60 0x55d15f203ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:37.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.704+0000 7f786d071740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d15f1ff0d0 0x55d15f20afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:37.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.704+0000 7f786d071740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d15f202020 con 0x55d15f1f9f60 2024-03-28T22:33:37.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.704+0000 7f786d071740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55d15f0fc180 con 0x55d15f1ff0d0 2024-03-28T22:33:37.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.704+0000 7f786d071740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55d15f2021a0 con 0x55d15f1f9570 2024-03-28T22:33:37.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.704+0000 7f786d071740 10 client.0 _ll_register_callbacks cb 0x55d15ee71ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:33:37.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.704+0000 7f7863400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d15f1f9570 0x55d15f203ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:37.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.704+0000 7f7863400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d15f1f9570 0x55d15f203ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:54796/0 (socket says 172.21.15.43:54796) 2024-03-28T22:33:37.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.704+0000 7f7863400640 1 -- 172.21.15.43:0/3063349098 learned_addr learned my addr 172.21.15.43:0/3063349098 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:37.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.704+0000 7f7863e00640 1 --2- 172.21.15.43:0/3063349098 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d15f1ff0d0 0x55d15f20afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:37.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.704+0000 7f785be00640 1 --2- 172.21.15.43:0/3063349098 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d15f1f9f60 0x55d15f203ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:37.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.705+0000 7f7863400640 1 -- 172.21.15.43:0/3063349098 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d15f1ff0d0 msgr2=0x55d15f20afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:33:37.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.705+0000 7f7863400640 1 --2- 172.21.15.43:0/3063349098 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d15f1ff0d0 0x55d15f20afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.705+0000 7f7863400640 1 -- 172.21.15.43:0/3063349098 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d15f1f9f60 msgr2=0x55d15f203ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:33:37.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.705+0000 7f7863400640 1 --2- 172.21.15.43:0/3063349098 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d15f1f9f60 0x55d15f203ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.705+0000 7f7863400640 1 -- 172.21.15.43:0/3063349098 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d15ef3c6b0 con 0x55d15f1f9570 2024-03-28T22:33:37.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.705+0000 7f7863e00640 1 --2- 172.21.15.43:0/3063349098 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d15f1ff0d0 0x55d15f20afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:33:37.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.705+0000 7f7863400640 1 --2- 172.21.15.43:0/3063349098 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d15f1f9570 0x55d15f203ab0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f785401f9f0 tx=0x7f7854004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d0451d75921b0ae server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:37.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.705+0000 7f7851600640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:33:37.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.705+0000 7f7851600640 1 -- 172.21.15.43:0/3063349098 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785401d410 con 0x55d15f1f9570 2024-03-28T22:33:37.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.705+0000 7f786d071740 1 -- 172.21.15.43:0/3063349098 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f785c032c90 con 0x55d15f1f9570 2024-03-28T22:33:37.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.705+0000 7f7851600640 1 -- 172.21.15.43:0/3063349098 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f785401d5b0 con 0x55d15f1f9570 2024-03-28T22:33:37.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.706+0000 7f7851600640 1 -- 172.21.15.43:0/3063349098 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f7854034050 con 0x55d15f1f9570 2024-03-28T22:33:37.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.706+0000 7f786d071740 10 client.24706 fetch_fsmap learned FSMap version 17 2024-03-28T22:33:37.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.706+0000 7f786d071740 1 -- 172.21.15.43:0/3063349098 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55d15f20e3f0 con 0x55d15f1f9570 2024-03-28T22:33:37.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.706+0000 7f7851600640 1 -- 172.21.15.43:0/3063349098 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f785c032c90 con 0x55d15f1f9570 2024-03-28T22:33:37.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.706+0000 7f7851600640 1 -- 172.21.15.43:0/3063349098 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f78540119d0 con 0x55d15f1f9570 2024-03-28T22:33:37.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.706+0000 7f786d071740 10 client.24706 fetch_fsmap finished waiting for FSMap version 17 2024-03-28T22:33:37.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.706+0000 7f786d071740 10 client.24706 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T22:33:37.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.706+0000 7f786d071740 20 client.24706 populate_metadata read hostname 'smithi043' 2024-03-28T22:33:37.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.706+0000 7f786d071740 1 --2- 172.21.15.43:0/3063349098 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d15f210a90 0x55d15f211f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:37.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.706+0000 7f786d071740 4 client.24706 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:33:37.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.706+0000 7f786d071740 1 -- 172.21.15.43:0/3063349098 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d15f16af00 con 0x55d15f210a90 2024-03-28T22:33:37.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.707+0000 7f785be00640 1 --2- 172.21.15.43:0/3063349098 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d15f210a90 0x55d15f211f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:37.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.707+0000 7f785be00640 1 --2- 172.21.15.43:0/3063349098 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d15f210a90 0x55d15f211f00 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2b0a5ee4bfc6b0c server_cookie=b22e2b292172c2c in_seq=0 out_seq=0 2024-03-28T22:33:37.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.707+0000 7f7851600640 10 client.24706 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:33:37.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.708+0000 7f7851600640 1 -- 172.21.15.43:0/3063349098 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d15f16af00 con 0x55d15f210a90 2024-03-28T22:33:37.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.708+0000 7f7851600640 10 client.24706 handle_command_reply: tid=0 2024-03-28T22:33:37.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.708+0000 7f786d071740 1 client.24706 shutdown 2024-03-28T22:33:37.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.708+0000 7f786d071740 1 -- 172.21.15.43:0/3063349098 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d15f1f9570 msgr2=0x55d15f203ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:37.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.708+0000 7f786d071740 1 --2- 172.21.15.43:0/3063349098 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d15f1f9570 0x55d15f203ab0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f785401f9f0 tx=0x7f7854004730 comp rx=0 tx=0).stop 2024-03-28T22:33:37.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.708+0000 7f786d071740 1 -- 172.21.15.43:0/3063349098 shutdown_connections 2024-03-28T22:33:37.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.708+0000 7f786d071740 1 --2- 172.21.15.43:0/3063349098 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d15f210a90 0x55d15f211f00 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.708+0000 7f786d071740 1 --2- 172.21.15.43:0/3063349098 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d15f1ff0d0 0x55d15f20afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.708+0000 7f786d071740 1 --2- 172.21.15.43:0/3063349098 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d15f1f9f60 0x55d15f203ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.708+0000 7f786d071740 1 --2- 172.21.15.43:0/3063349098 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d15f1f9570 0x55d15f203ab0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.708+0000 7f7851600640 0 client.24706 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:33:37.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.708+0000 7f786d071740 1 -- 172.21.15.43:0/3063349098 >> 172.21.15.43:0/3063349098 conn(0x55d15f1f4ae0 msgr2=0x55d15f1f3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:37.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.708+0000 7f786d071740 1 -- 172.21.15.43:0/3063349098 shutdown_connections 2024-03-28T22:33:37.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.708+0000 7f786d071740 1 -- 172.21.15.43:0/3063349098 wait complete. 2024-03-28T22:33:37.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.708+0000 7f786d071740 20 client.24706 trim_cache size 0 max 16384 2024-03-28T22:33:37.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.728+0000 7f786d071740 1 Processor -- start 2024-03-28T22:33:37.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.728+0000 7f786d071740 1 -- start start 2024-03-28T22:33:37.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.729+0000 7f786d071740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d15f1f9570 0x55d15f1fb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:37.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.729+0000 7f786d071740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d15f1f9f60 0x55d15f1fb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:37.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.729+0000 7f786d071740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d15f1ff0d0 0x55d15f20af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:37.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.729+0000 7f786d071740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d15f210240 con 0x55d15f1ff0d0 2024-03-28T22:33:37.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.729+0000 7f786d071740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55d15f210910 con 0x55d15f1f9570 2024-03-28T22:33:37.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.729+0000 7f786d071740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55d15f210fe0 con 0x55d15f1f9f60 2024-03-28T22:33:37.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.729+0000 7f7863e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d15f1ff0d0 0x55d15f20af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:37.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.729+0000 7f7863400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d15f1f9570 0x55d15f1fb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:37.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.729+0000 7f7863e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d15f1ff0d0 0x55d15f20af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45626/0 (socket says 172.21.15.43:45626) 2024-03-28T22:33:37.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.729+0000 7f7863e00640 1 -- 172.21.15.43:0/3509205599 learned_addr learned my addr 172.21.15.43:0/3509205599 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:37.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.729+0000 7f785be00640 1 --2- 172.21.15.43:0/3509205599 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d15f1f9f60 0x55d15f1fb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:37.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.729+0000 7f7863400640 1 -- 172.21.15.43:0/3509205599 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d15f1f9f60 msgr2=0x55d15f1fb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:33:37.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.729+0000 7f7863400640 1 --2- 172.21.15.43:0/3509205599 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d15f1f9f60 0x55d15f1fb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.729+0000 7f7863400640 1 -- 172.21.15.43:0/3509205599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d15f1ff0d0 msgr2=0x55d15f20af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:33:37.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.729+0000 7f7863400640 1 --2- 172.21.15.43:0/3509205599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d15f1ff0d0 0x55d15f20af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.729+0000 7f7863400640 1 -- 172.21.15.43:0/3509205599 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d15f20c440 con 0x55d15f1f9570 2024-03-28T22:33:37.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.730+0000 7f7863400640 1 --2- 172.21.15.43:0/3509205599 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d15f1f9570 0x55d15f1fb4a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7854011e40 tx=0x7f7854005e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf8184842242ed6e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:37.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.730+0000 7f7853e00640 1 -- 172.21.15.43:0/3509205599 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785403c070 con 0x55d15f1f9570 2024-03-28T22:33:37.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.730+0000 7f7853e00640 1 -- 172.21.15.43:0/3509205599 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f785401f7c0 con 0x55d15f1f9570 2024-03-28T22:33:37.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.730+0000 7f7853e00640 1 -- 172.21.15.43:0/3509205599 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78540117c0 con 0x55d15f1f9570 2024-03-28T22:33:37.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.731+0000 7f786d071740 1 -- 172.21.15.43:0/3509205599 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d15f1f9570 msgr2=0x55d15f1fb4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:37.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.731+0000 7f786d071740 1 --2- 172.21.15.43:0/3509205599 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d15f1f9570 0x55d15f1fb4a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7854011e40 tx=0x7f7854005e30 comp rx=0 tx=0).stop 2024-03-28T22:33:37.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.731+0000 7f786d071740 1 -- 172.21.15.43:0/3509205599 shutdown_connections 2024-03-28T22:33:37.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.731+0000 7f786d071740 1 --2- 172.21.15.43:0/3509205599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d15f1ff0d0 0x55d15f20af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.731+0000 7f786d071740 1 --2- 172.21.15.43:0/3509205599 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d15f1f9f60 0x55d15f1fb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.731+0000 7f786d071740 1 --2- 172.21.15.43:0/3509205599 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d15f1f9570 0x55d15f1fb4a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.731+0000 7f786d071740 1 -- 172.21.15.43:0/3509205599 >> 172.21.15.43:0/3509205599 conn(0x55d15f1f4ae0 msgr2=0x55d15f1f4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:37.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.731+0000 7f786d071740 1 -- 172.21.15.43:0/3509205599 shutdown_connections 2024-03-28T22:33:37.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.731+0000 7f786d071740 1 -- 172.21.15.43:0/3509205599 wait complete. 2024-03-28T22:33:37.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.731+0000 7f786d071740 1 -- start start 2024-03-28T22:33:37.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.731+0000 7f786d071740 1 Processor -- start 2024-03-28T22:33:37.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.732+0000 7f786d071740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d15f1f9570 0x55d15f206af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:37.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.732+0000 7f786d071740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d15f1f9f60 0x55d15f207030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:37.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.732+0000 7f786d071740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d15f1ff0d0 0x55d15f20a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:37.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.732+0000 7f786d071740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d15f211d80 con 0x55d15f1f9570 2024-03-28T22:33:37.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.732+0000 7f786d071740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55d15f2116b0 con 0x55d15f1f9f60 2024-03-28T22:33:37.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.732+0000 7f786d071740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55d15f212450 con 0x55d15f1ff0d0 2024-03-28T22:33:37.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.732+0000 7f786d071740 10 client.0 _ll_register_callbacks cb 0x7f78540097c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:33:37.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.732+0000 7f7863400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d15f1f9570 0x55d15f206af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:37.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.732+0000 7f785be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d15f1f9f60 0x55d15f207030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:37.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.732+0000 7f785be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d15f1f9f60 0x55d15f207030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:43268/0 (socket says 172.21.15.43:43268) 2024-03-28T22:33:37.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.732+0000 7f785be00640 1 -- 172.21.15.43:0/2778721602 learned_addr learned my addr 172.21.15.43:0/2778721602 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:37.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.732+0000 7f7863e00640 1 --2- 172.21.15.43:0/2778721602 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d15f1ff0d0 0x55d15f20a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:37.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.732+0000 7f785be00640 1 -- 172.21.15.43:0/2778721602 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d15f1ff0d0 msgr2=0x55d15f20a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:33:37.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.732+0000 7f785be00640 1 --2- 172.21.15.43:0/2778721602 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d15f1ff0d0 0x55d15f20a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.732+0000 7f785be00640 1 -- 172.21.15.43:0/2778721602 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d15f1f9570 msgr2=0x55d15f206af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:33:37.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.732+0000 7f785be00640 1 --2- 172.21.15.43:0/2778721602 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d15f1f9570 0x55d15f206af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.733+0000 7f785be00640 1 -- 172.21.15.43:0/2778721602 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d15f20a640 con 0x55d15f1f9f60 2024-03-28T22:33:37.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.733+0000 7f7863400640 1 --2- 172.21.15.43:0/2778721602 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d15f1f9570 0x55d15f206af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:33:37.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.733+0000 7f785be00640 1 --2- 172.21.15.43:0/2778721602 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d15f1f9f60 0x55d15f207030 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f784c0cb2d0 tx=0x7f784c085040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb29edea0ee050d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:37.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.733+0000 7f7851600640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:33:37.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.733+0000 7f7851600640 1 -- 172.21.15.43:0/2778721602 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f784c09a040 con 0x55d15f1f9f60 2024-03-28T22:33:37.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.733+0000 7f7851600640 1 -- 172.21.15.43:0/2778721602 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f784c098800 con 0x55d15f1f9f60 2024-03-28T22:33:37.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.733+0000 7f786d071740 1 -- 172.21.15.43:0/2778721602 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f785401f7c0 con 0x55d15f1f9f60 2024-03-28T22:33:37.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.733+0000 7f7851600640 1 -- 172.21.15.43:0/2778721602 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f784c099e60 con 0x55d15f1f9f60 2024-03-28T22:33:37.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.733+0000 7f786d071740 10 client.24686 fetch_fsmap learned FSMap version 17 2024-03-28T22:33:37.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.733+0000 7f786d071740 1 -- 172.21.15.43:0/2778721602 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55d15f276c50 con 0x55d15f1f9f60 2024-03-28T22:33:37.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.733+0000 7f7851600640 1 -- 172.21.15.43:0/2778721602 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f785401f7c0 con 0x55d15f1f9f60 2024-03-28T22:33:37.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.734+0000 7f7851600640 1 -- 172.21.15.43:0/2778721602 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f784c09e170 con 0x55d15f1f9f60 2024-03-28T22:33:37.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.734+0000 7f786d071740 10 client.24686 fetch_fsmap finished waiting for FSMap version 17 2024-03-28T22:33:37.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.734+0000 7f786d071740 10 client.24686 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T22:33:37.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.734+0000 7f786d071740 20 client.24686 populate_metadata read hostname 'smithi043' 2024-03-28T22:33:37.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.734+0000 7f786d071740 1 --2- 172.21.15.43:0/2778721602 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d15f27b800 0x55d15f27a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:37.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.734+0000 7f786d071740 4 client.24686 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T22:33:37.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.734+0000 7f786d071740 1 -- 172.21.15.43:0/2778721602 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "status"}) -- 0x55d15f165250 con 0x55d15f27b800 2024-03-28T22:33:37.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.734+0000 7f7863400640 1 --2- 172.21.15.43:0/2778721602 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d15f27b800 0x55d15f27a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:37.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.734+0000 7f7863400640 1 --2- 172.21.15.43:0/2778721602 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d15f27b800 0x55d15f27a510 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b2d5e03923b623a server_cookie=574192a84701fe4f in_seq=0 out_seq=0 2024-03-28T22:33:37.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.734+0000 7f7851600640 10 client.24686 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:33:37.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.735+0000 7f7851600640 1 -- 172.21.15.43:0/2778721602 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55d15f165250 con 0x55d15f27b800 2024-03-28T22:33:37.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.735+0000 7f7851600640 10 client.24686 handle_command_reply: tid=0 2024-03-28T22:33:37.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.735+0000 7f786d071740 1 client.24686 shutdown 2024-03-28T22:33:37.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.735+0000 7f786d071740 1 -- 172.21.15.43:0/2778721602 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d15f1f9f60 msgr2=0x55d15f207030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:37.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.735+0000 7f786d071740 1 --2- 172.21.15.43:0/2778721602 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d15f1f9f60 0x55d15f207030 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f784c0cb2d0 tx=0x7f784c085040 comp rx=0 tx=0).stop 2024-03-28T22:33:37.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.735+0000 7f786d071740 1 -- 172.21.15.43:0/2778721602 shutdown_connections 2024-03-28T22:33:37.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.735+0000 7f786d071740 1 --2- 172.21.15.43:0/2778721602 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d15f27b800 0x55d15f27a510 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.735+0000 7f786d071740 1 --2- 172.21.15.43:0/2778721602 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d15f1f9f60 0x55d15f207030 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.735+0000 7f786d071740 1 --2- 172.21.15.43:0/2778721602 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d15f1ff0d0 0x55d15f20a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.735+0000 7f786d071740 1 --2- 172.21.15.43:0/2778721602 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d15f1f9570 0x55d15f206af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.735+0000 7f786d071740 1 -- 172.21.15.43:0/2778721602 >> 172.21.15.43:0/2778721602 conn(0x55d15f1f4ae0 msgr2=0x55d15f1fb4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:37.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.735+0000 7f7851600640 0 client.24686 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:33:37.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.735+0000 7f786d071740 1 -- 172.21.15.43:0/2778721602 shutdown_connections 2024-03-28T22:33:37.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.735+0000 7f786d071740 1 -- 172.21.15.43:0/2778721602 wait complete. 2024-03-28T22:33:37.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.735+0000 7f786d071740 20 client.24686 trim_cache size 0 max 16384 2024-03-28T22:33:37.740 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:33:37.740 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_fsid": "c2276428-ed51-11ee-95d0-87774f69a715", 2024-03-28T22:33:37.740 INFO:teuthology.orchestra.run.smithi043.stdout: "whoami": 0, 2024-03-28T22:33:37.740 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 24572, 2024-03-28T22:33:37.740 INFO:teuthology.orchestra.run.smithi043.stdout: "want_state": "up:active", 2024-03-28T22:33:37.740 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "up:active", 2024-03-28T22:33:37.740 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs", 2024-03-28T22:33:37.740 INFO:teuthology.orchestra.run.smithi043.stdout: "rank_uptime": 8.2615063549999999, 2024-03-28T22:33:37.740 INFO:teuthology.orchestra.run.smithi043.stdout: "mdsmap_epoch": 17, 2024-03-28T22:33:37.740 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch": 92, 2024-03-28T22:33:37.741 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch_barrier": 92, 2024-03-28T22:33:37.741 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 14.635587691 2024-03-28T22:33:37.741 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:33:37.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.736+0000 7f786a400640 1 -- 172.21.15.43:0/2236354120 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f78480a1d80 msgr2=0x7f78480a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:37.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.736+0000 7f786a400640 1 --2- 172.21.15.43:0/2236354120 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f78480a1d80 0x7f78480a4240 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x55d15f1fa360 tx=0x7f7854005f90 comp rx=0 tx=0).stop 2024-03-28T22:33:37.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.736+0000 7f786a400640 1 -- 172.21.15.43:0/2236354120 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7864104b80 msgr2=0x7f786419a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:37.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.736+0000 7f786a400640 1 --2- 172.21.15.43:0/2236354120 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7864104b80 0x7f786419a1f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f784c00d950 tx=0x7f784c00de10 comp rx=0 tx=0).stop 2024-03-28T22:33:37.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.736+0000 7f786a400640 1 -- 172.21.15.43:0/2236354120 shutdown_connections 2024-03-28T22:33:37.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.736+0000 7f786a400640 1 --2- 172.21.15.43:0/2236354120 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f78480a1d80 0x7f78480a4240 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.736+0000 7f786a400640 1 --2- 172.21.15.43:0/2236354120 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7864104b80 0x7f786419a1f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.736+0000 7f786a400640 1 --2- 172.21.15.43:0/2236354120 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7864105540 0x7f78641a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.736+0000 7f786a400640 1 --2- 172.21.15.43:0/2236354120 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7864103990 0x7f7864199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:37.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.736+0000 7f786a400640 1 -- 172.21.15.43:0/2236354120 >> 172.21.15.43:0/2236354120 conn(0x7f78640fef00 msgr2=0x7f7864107e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:37.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.736+0000 7f786a400640 1 -- 172.21.15.43:0/2236354120 shutdown_connections 2024-03-28T22:33:37.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:37.736+0000 7f786a400640 1 -- 172.21.15.43:0/2236354120 wait complete. 2024-03-28T22:33:37.751 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-28T22:33:37.751 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-28T22:33:37.752 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph fs ls 2024-03-28T22:33:38.206 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:33:38.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:38 smithi043 ceph-mon[31823]: pgmap v326: 129 pgs: 8 creating+peering, 98 active+clean, 23 unknown; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 7 op/s 2024-03-28T22:33:38.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:38 smithi043 ceph-mon[31823]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T22:33:38.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:38 smithi043 ceph-mon[31823]: Cluster is now healthy 2024-03-28T22:33:38.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:38 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1151629821' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:33:38.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:38 smithi172 ceph-mon[36566]: pgmap v326: 129 pgs: 8 creating+peering, 98 active+clean, 23 unknown; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 7 op/s 2024-03-28T22:33:38.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:38 smithi172 ceph-mon[36566]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T22:33:38.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:38 smithi172 ceph-mon[36566]: Cluster is now healthy 2024-03-28T22:33:38.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:38 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1151629821' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:33:38.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:38 smithi082 ceph-mon[36471]: pgmap v326: 129 pgs: 8 creating+peering, 98 active+clean, 23 unknown; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 7 op/s 2024-03-28T22:33:38.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:38 smithi082 ceph-mon[36471]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T22:33:38.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:38 smithi082 ceph-mon[36471]: Cluster is now healthy 2024-03-28T22:33:38.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:38 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1151629821' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:33:40.272 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:40 smithi043 ceph-mon[31823]: pgmap v327: 129 pgs: 8 creating+peering, 121 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-03-28T22:33:40.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:40 smithi172 ceph-mon[36566]: pgmap v327: 129 pgs: 8 creating+peering, 121 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-03-28T22:33:40.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:40 smithi082 ceph-mon[36471]: pgmap v327: 129 pgs: 8 creating+peering, 121 active+clean; 579 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-03-28T22:33:40.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.546+0000 7fbd4a000700 1 -- 172.21.15.43:0/1865660969 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbd44104df0 msgr2=0x7fbd44111300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:40.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.546+0000 7fbd4a000700 1 --2- 172.21.15.43:0/1865660969 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbd44104df0 0x7fbd44111300 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c00b0a0 tx=0x7fbd3c00b3b0 comp rx=0 tx=0).stop 2024-03-28T22:33:40.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.546+0000 7fbd4a000700 1 -- 172.21.15.43:0/1865660969 shutdown_connections 2024-03-28T22:33:40.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.546+0000 7fbd4a000700 1 --2- 172.21.15.43:0/1865660969 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbd44104df0 0x7fbd44111300 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:40.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.546+0000 7fbd4a000700 1 --2- 172.21.15.43:0/1865660969 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd4410a4c0 0x7fbd4410a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:40.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.546+0000 7fbd4a000700 1 --2- 172.21.15.43:0/1865660969 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd44104430 0x7fbd441048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:40.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.546+0000 7fbd4a000700 1 -- 172.21.15.43:0/1865660969 >> 172.21.15.43:0/1865660969 conn(0x7fbd440fff20 msgr2=0x7fbd44102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:40.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.547+0000 7fbd4a000700 1 -- 172.21.15.43:0/1865660969 shutdown_connections 2024-03-28T22:33:40.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.547+0000 7fbd4a000700 1 -- 172.21.15.43:0/1865660969 wait complete. 2024-03-28T22:33:40.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.548+0000 7fbd4a000700 1 Processor -- start 2024-03-28T22:33:40.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.548+0000 7fbd4a000700 1 -- start start 2024-03-28T22:33:40.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.548+0000 7fbd4a000700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd44104430 0x7fbd441a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:40.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.548+0000 7fbd4a000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd44104df0 0x7fbd441a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:40.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.548+0000 7fbd4a000700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbd4410a4c0 0x7fbd4419a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:40.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.548+0000 7fbd4a000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbd441147a0 con 0x7fbd44104df0 2024-03-28T22:33:40.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.548+0000 7fbd4a000700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fbd44114620 con 0x7fbd4410a4c0 2024-03-28T22:33:40.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.548+0000 7fbd4a000700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fbd44114920 con 0x7fbd44104430 2024-03-28T22:33:40.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.548+0000 7fbd42a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd44104df0 0x7fbd441a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:40.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.548+0000 7fbd43e00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbd4410a4c0 0x7fbd4419a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:40.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.548+0000 7fbd43e00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbd4410a4c0 0x7fbd4419a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:43286/0 (socket says 172.21.15.43:43286) 2024-03-28T22:33:40.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.548+0000 7fbd43e00700 1 -- 172.21.15.43:0/722212079 learned_addr learned my addr 172.21.15.43:0/722212079 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:40.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.548+0000 7fbd43400700 1 --2- 172.21.15.43:0/722212079 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd44104430 0x7fbd441a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:40.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.549+0000 7fbd43e00700 1 -- 172.21.15.43:0/722212079 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd44104430 msgr2=0x7fbd441a07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:40.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.549+0000 7fbd43e00700 1 --2- 172.21.15.43:0/722212079 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd44104430 0x7fbd441a07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:40.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.549+0000 7fbd43e00700 1 -- 172.21.15.43:0/722212079 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd44104df0 msgr2=0x7fbd441a0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:40.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.549+0000 7fbd43e00700 1 --2- 172.21.15.43:0/722212079 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd44104df0 0x7fbd441a0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:40.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.549+0000 7fbd43e00700 1 -- 172.21.15.43:0/722212079 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd4419b100 con 0x7fbd4410a4c0 2024-03-28T22:33:40.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.549+0000 7fbd43e00700 1 --2- 172.21.15.43:0/722212079 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbd4410a4c0 0x7fbd4419a9a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c00b070 tx=0x7fbd3c004c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:40.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.549+0000 7fbd3be00700 1 -- 172.21.15.43:0/722212079 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c021070 con 0x7fbd4410a4c0 2024-03-28T22:33:40.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.549+0000 7fbd3be00700 1 -- 172.21.15.43:0/722212079 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd3c003c00 con 0x7fbd4410a4c0 2024-03-28T22:33:40.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.549+0000 7fbd4a000700 1 -- 172.21.15.43:0/722212079 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd4419b390 con 0x7fbd4410a4c0 2024-03-28T22:33:40.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.549+0000 7fbd3be00700 1 -- 172.21.15.43:0/722212079 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c004740 con 0x7fbd4410a4c0 2024-03-28T22:33:40.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.550+0000 7fbd4a000700 1 -- 172.21.15.43:0/722212079 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd441a76b0 con 0x7fbd4410a4c0 2024-03-28T22:33:40.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.551+0000 7fbd4a000700 1 -- 172.21.15.43:0/722212079 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd44068f00 con 0x7fbd4410a4c0 2024-03-28T22:33:40.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.553+0000 7fbd3be00700 1 -- 172.21.15.43:0/722212079 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fbd3c007980 con 0x7fbd4410a4c0 2024-03-28T22:33:40.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.554+0000 7fbd3be00700 1 --2- 172.21.15.43:0/722212079 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fbd280a1e90 0x7fbd280a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:40.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.554+0000 7fbd3be00700 1 -- 172.21.15.43:0/722212079 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7fbd3c0cee00 con 0x7fbd4410a4c0 2024-03-28T22:33:40.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.554+0000 7fbd43400700 1 --2- 172.21.15.43:0/722212079 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fbd280a1e90 0x7fbd280a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:40.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.555+0000 7fbd43400700 1 --2- 172.21.15.43:0/722212079 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fbd280a1e90 0x7fbd280a4350 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbd34009680 tx=0x7fbd34009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:40.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.562+0000 7fbd3be00700 1 -- 172.21.15.43:0/722212079 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbd3c09c020 con 0x7fbd4410a4c0 2024-03-28T22:33:40.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.842+0000 7fbd4a000700 1 -- 172.21.15.43:0/722212079 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fbd4402db30 con 0x7fbd4410a4c0 2024-03-28T22:33:40.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.843+0000 7fbd3be00700 1 -- 172.21.15.43:0/722212079 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7fbd3c09b920 con 0x7fbd4410a4c0 2024-03-28T22:33:40.841 INFO:teuthology.orchestra.run.smithi043.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-28T22:33:40.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.846+0000 7fbd4a000700 1 -- 172.21.15.43:0/722212079 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fbd280a1e90 msgr2=0x7fbd280a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:40.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.846+0000 7fbd4a000700 1 --2- 172.21.15.43:0/722212079 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fbd280a1e90 0x7fbd280a4350 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbd34009680 tx=0x7fbd34009260 comp rx=0 tx=0).stop 2024-03-28T22:33:40.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.846+0000 7fbd4a000700 1 -- 172.21.15.43:0/722212079 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbd4410a4c0 msgr2=0x7fbd4419a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:40.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.846+0000 7fbd4a000700 1 --2- 172.21.15.43:0/722212079 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbd4410a4c0 0x7fbd4419a9a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c00b070 tx=0x7fbd3c004c60 comp rx=0 tx=0).stop 2024-03-28T22:33:40.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.846+0000 7fbd4a000700 1 -- 172.21.15.43:0/722212079 shutdown_connections 2024-03-28T22:33:40.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.846+0000 7fbd4a000700 1 --2- 172.21.15.43:0/722212079 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fbd280a1e90 0x7fbd280a4350 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:40.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.846+0000 7fbd4a000700 1 --2- 172.21.15.43:0/722212079 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbd4410a4c0 0x7fbd4419a9a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:40.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.846+0000 7fbd4a000700 1 --2- 172.21.15.43:0/722212079 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd44104430 0x7fbd441a07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:40.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.846+0000 7fbd4a000700 1 --2- 172.21.15.43:0/722212079 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd44104df0 0x7fbd441a0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:40.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.846+0000 7fbd4a000700 1 -- 172.21.15.43:0/722212079 >> 172.21.15.43:0/722212079 conn(0x7fbd440fff20 msgr2=0x7fbd44100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:40.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.846+0000 7fbd4a000700 1 -- 172.21.15.43:0/722212079 shutdown_connections 2024-03-28T22:33:40.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:40.846+0000 7fbd4a000700 1 -- 172.21.15.43:0/722212079 wait complete. 2024-03-28T22:33:41.231 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:41 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/722212079' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T22:33:41.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:41 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/722212079' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T22:33:41.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:41 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/722212079' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T22:33:41.759 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-28T22:33:41.759 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-28T22:33:41.760 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi043.front.sepia.ceph.com 2024-03-28T22:33:41.760 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-28T22:33:41.760 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-28T22:33:41.760 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-28T22:33:41.760 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-28T22:33:41.760 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-28T22:33:41.760 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-28T22:33:41.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:33:41.760 DEBUG:teuthology.orchestra.run.smithi043:> ip addr 2024-03-28T22:33:41.816 INFO:teuthology.orchestra.run.smithi043.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-28T22:33:41.816 INFO:teuthology.orchestra.run.smithi043.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-28T22:33:41.816 INFO:teuthology.orchestra.run.smithi043.stdout: inet 127.0.0.1/8 scope host lo 2024-03-28T22:33:41.816 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-03-28T22:33:41.817 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 ::1/128 scope host 2024-03-28T22:33:41.817 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-03-28T22:33:41.817 INFO:teuthology.orchestra.run.smithi043.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T22:33:41.817 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:6b:fd:9c brd ff:ff:ff:ff:ff:ff 2024-03-28T22:33:41.817 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp7s0f0 2024-03-28T22:33:41.817 INFO:teuthology.orchestra.run.smithi043.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T22:33:41.817 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:6b:fd:9d brd ff:ff:ff:ff:ff:ff 2024-03-28T22:33:41.817 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp7s0f1 2024-03-28T22:33:41.817 INFO:teuthology.orchestra.run.smithi043.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-28T22:33:41.817 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:bd:13:58 brd ff:ff:ff:ff:ff:ff 2024-03-28T22:33:41.817 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp1s0f0 2024-03-28T22:33:41.817 INFO:teuthology.orchestra.run.smithi043.stdout: inet 172.21.15.43/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-28T22:33:41.817 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft 42028sec preferred_lft 42028sec 2024-03-28T22:33:41.818 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::ec4:7aff:febd:1358/64 scope link noprefixroute 2024-03-28T22:33:41.818 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-03-28T22:33:41.818 INFO:teuthology.orchestra.run.smithi043.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T22:33:41.818 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:bd:13:59 brd ff:ff:ff:ff:ff:ff 2024-03-28T22:33:41.818 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp1s0f1 2024-03-28T22:33:41.819 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-28T22:33:41.820 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T22:33:41.820 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-03-28T22:33:41.820 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link add name ceph-brx type bridge 2024-03-28T22:33:41.820 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip addr flush dev ceph-brx 2024-03-28T22:33:41.820 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link set ceph-brx up 2024-03-28T22:33:41.820 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-28T22:33:41.820 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-03-28T22:33:41.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T22:33:42.242 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:42 smithi043 ceph-mon[31823]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2024-03-28T22:33:42.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T22:33:42.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:33:42.348 DEBUG:teuthology.orchestra.run.smithi043:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-28T22:33:42.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:42 smithi172 ceph-mon[36566]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2024-03-28T22:33:42.431 INFO:teuthology.orchestra.run.smithi043.stdout:1 2024-03-28T22:33:42.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:33:42.435 DEBUG:teuthology.orchestra.run.smithi043:> route 2024-03-28T22:33:42.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:42 smithi082 ceph-mon[36471]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2024-03-28T22:33:42.495 INFO:teuthology.orchestra.run.smithi043.stdout:Kernel IP routing table 2024-03-28T22:33:42.495 INFO:teuthology.orchestra.run.smithi043.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-28T22:33:42.495 INFO:teuthology.orchestra.run.smithi043.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-28T22:33:42.495 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-28T22:33:42.495 INFO:teuthology.orchestra.run.smithi043.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-28T22:33:42.496 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T22:33:42.496 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-03-28T22:33:42.496 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-28T22:33:42.496 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-28T22:33:42.496 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-28T22:33:42.496 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-03-28T22:33:42.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T22:33:43.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T22:33:43.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:33:43.280 DEBUG:teuthology.orchestra.run.smithi043:> ip netns list 2024-03-28T22:33:43.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:33:43.341 DEBUG:teuthology.orchestra.run.smithi043:> ip netns list-id 2024-03-28T22:33:43.398 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T22:33:43.398 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-03-28T22:33:43.398 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T22:33:43.398 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-28T22:33:43.398 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-03-28T22:33:43.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T22:33:43.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T22:33:43.621 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-28T22:33:43.621 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T22:33:43.621 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-03-28T22:33:43.621 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-28T22:33:43.621 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-28T22:33:43.621 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-28T22:33:43.621 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-28T22:33:43.621 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-28T22:33:43.621 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-03-28T22:33:43.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T22:33:43.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T22:33:43.978 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T22:33:43.978 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-03-28T22:33:43.979 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link set brx.0 up 2024-03-28T22:33:43.979 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link set dev brx.0 master ceph-brx 2024-03-28T22:33:43.979 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-03-28T22:33:44.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T22:33:44.121 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:44 smithi043 ceph-mon[31823]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 376 B/s wr, 0 op/s 2024-03-28T22:33:44.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T22:33:44.219 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-28T22:33:44.219 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T22:33:44.219 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-28T22:33:44.263 INFO:teuthology.orchestra.run.smithi043.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-28T22:33:44.264 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T22:33:44.264 DEBUG:teuthology.orchestra.run.smithi043:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-28T22:33:44.350 DEBUG:teuthology.orchestra.run.smithi043:> sudo modprobe fuse 2024-03-28T22:33:44.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:44 smithi172 ceph-mon[36566]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 376 B/s wr, 0 op/s 2024-03-28T22:33:44.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:44 smithi082 ceph-mon[36471]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 376 B/s wr, 0 op/s 2024-03-28T22:33:44.500 DEBUG:teuthology.orchestra.run.smithi043:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T22:33:44.556 INFO:teuthology.orchestra.run.smithi043.stdout:/proc 2024-03-28T22:33:44.556 INFO:teuthology.orchestra.run.smithi043.stdout:/sys 2024-03-28T22:33:44.556 INFO:teuthology.orchestra.run.smithi043.stdout:/dev 2024-03-28T22:33:44.556 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/security 2024-03-28T22:33:44.556 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/shm 2024-03-28T22:33:44.556 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/pts 2024-03-28T22:33:44.556 INFO:teuthology.orchestra.run.smithi043.stdout:/run 2024-03-28T22:33:44.556 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/cgroup 2024-03-28T22:33:44.556 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/pstore 2024-03-28T22:33:44.556 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/bpf 2024-03-28T22:33:44.557 INFO:teuthology.orchestra.run.smithi043.stdout:/ 2024-03-28T22:33:44.557 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/selinux 2024-03-28T22:33:44.557 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T22:33:44.557 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/hugepages 2024-03-28T22:33:44.557 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/mqueue 2024-03-28T22:33:44.557 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/debug 2024-03-28T22:33:44.557 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/tracing 2024-03-28T22:33:44.557 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/fuse/connections 2024-03-28T22:33:44.557 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-sysctl.service 2024-03-28T22:33:44.557 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/config 2024-03-28T22:33:44.557 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-28T22:33:44.557 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-28T22:33:44.557 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/nfs/rpc_pipefs 2024-03-28T22:33:44.557 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/1000 2024-03-28T22:33:44.558 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph 2024-03-28T22:33:44.558 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay 2024-03-28T22:33:44.558 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/fe518172546cef99a5a8c86e07cb3ceb900a53f271c97a4a51c5d72882e2d6f0/merged 2024-03-28T22:33:44.558 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/4687208e912e6d527d54f8260aac2b58a6a2162e92dbc3c8ce8cda99b750b040/merged 2024-03-28T22:33:44.558 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/0 2024-03-28T22:33:44.558 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T22:33:44.558 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/22bbf09122ced5d269cff1cf1ffa641964e3219e05403855b4db1e9ea67591eb/merged 2024-03-28T22:33:44.558 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/f62092261c55900a04328b941b86965b516b5af10f2e9560e86752f9200655a1/merged 2024-03-28T22:33:44.558 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/894e112b4640db23e0e30e552fc4a3aca569b8139f45ac0f9b89d224ac08fdcf/merged 2024-03-28T22:33:44.558 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/ef5d0a506eadc7e15ad71dcbcd6726277852fbb09848d3bbf4f185a5104842d3/merged 2024-03-28T22:33:44.558 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/65b41b1bc08fadc010f9fdf317f491c611190b60b2c1c2cde81706d1ed91fd16/merged 2024-03-28T22:33:44.558 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/a92555d89a3757e9d56918cee272cadf7cf600bbf00b4ad0189819387f90aa22/merged 2024-03-28T22:33:44.558 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/d13069e2c5c6b6f7fda815bf0a9b03865dfbe7cb8329a0b1ef826f53588174dc/merged 2024-03-28T22:33:44.558 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/3c8bae2b92dd4fb04072b2be67971623901a5ba172ae2424b052427da3158989/merged 2024-03-28T22:33:44.558 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns 2024-03-28T22:33:44.559 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T22:33:44.559 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T22:33:44.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:33:44.560 DEBUG:teuthology.orchestra.run.smithi043:> ls /sys/fs/fuse/connections 2024-03-28T22:33:44.606 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-28T22:33:44.606 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-28T22:33:44.653 DEBUG:teuthology.orchestra.run.smithi043:> sudo modprobe fuse 2024-03-28T22:33:44.753 DEBUG:teuthology.orchestra.run.smithi043:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T22:33:44.799 INFO:teuthology.orchestra.run.smithi043.stdout:/proc 2024-03-28T22:33:44.799 INFO:teuthology.orchestra.run.smithi043.stdout:/sys 2024-03-28T22:33:44.800 INFO:teuthology.orchestra.run.smithi043.stdout:/dev 2024-03-28T22:33:44.800 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/security 2024-03-28T22:33:44.800 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/shm 2024-03-28T22:33:44.800 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/pts 2024-03-28T22:33:44.800 INFO:teuthology.orchestra.run.smithi043.stdout:/run 2024-03-28T22:33:44.800 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/cgroup 2024-03-28T22:33:44.800 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/pstore 2024-03-28T22:33:44.800 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/bpf 2024-03-28T22:33:44.800 INFO:teuthology.orchestra.run.smithi043.stdout:/ 2024-03-28T22:33:44.800 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/selinux 2024-03-28T22:33:44.800 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T22:33:44.800 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/hugepages 2024-03-28T22:33:44.800 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/mqueue 2024-03-28T22:33:44.800 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/debug 2024-03-28T22:33:44.801 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/tracing 2024-03-28T22:33:44.801 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/fuse/connections 2024-03-28T22:33:44.801 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-sysctl.service 2024-03-28T22:33:44.801 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/config 2024-03-28T22:33:44.801 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-28T22:33:44.801 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-28T22:33:44.801 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/nfs/rpc_pipefs 2024-03-28T22:33:44.801 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/1000 2024-03-28T22:33:44.801 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph 2024-03-28T22:33:44.801 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay 2024-03-28T22:33:44.801 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/fe518172546cef99a5a8c86e07cb3ceb900a53f271c97a4a51c5d72882e2d6f0/merged 2024-03-28T22:33:44.801 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/4687208e912e6d527d54f8260aac2b58a6a2162e92dbc3c8ce8cda99b750b040/merged 2024-03-28T22:33:44.801 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/0 2024-03-28T22:33:44.801 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T22:33:44.802 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/22bbf09122ced5d269cff1cf1ffa641964e3219e05403855b4db1e9ea67591eb/merged 2024-03-28T22:33:44.802 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/f62092261c55900a04328b941b86965b516b5af10f2e9560e86752f9200655a1/merged 2024-03-28T22:33:44.802 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/894e112b4640db23e0e30e552fc4a3aca569b8139f45ac0f9b89d224ac08fdcf/merged 2024-03-28T22:33:44.802 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/ef5d0a506eadc7e15ad71dcbcd6726277852fbb09848d3bbf4f185a5104842d3/merged 2024-03-28T22:33:44.802 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/65b41b1bc08fadc010f9fdf317f491c611190b60b2c1c2cde81706d1ed91fd16/merged 2024-03-28T22:33:44.802 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/a92555d89a3757e9d56918cee272cadf7cf600bbf00b4ad0189819387f90aa22/merged 2024-03-28T22:33:44.802 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/d13069e2c5c6b6f7fda815bf0a9b03865dfbe7cb8329a0b1ef826f53588174dc/merged 2024-03-28T22:33:44.802 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/3c8bae2b92dd4fb04072b2be67971623901a5ba172ae2424b052427da3158989/merged 2024-03-28T22:33:44.802 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns 2024-03-28T22:33:44.802 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T22:33:44.802 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T22:33:44.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:33:44.804 DEBUG:teuthology.orchestra.run.smithi043:> ls /sys/fs/fuse/connections 2024-03-28T22:33:44.879 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi043.stderr:2024-03-28T22:33:44.881+0000 7f4f1980b480 -1 init, newargv = 0x55df73eb91b0 newargc=15 2024-03-28T22:33:44.879 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi043.stderr:ceph-fuse[73365]: starting ceph client 2024-03-28T22:33:44.889 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi043.stderr:ceph-fuse[73365]: starting fuse 2024-03-28T22:33:45.945 DEBUG:teuthology.orchestra.run.smithi043:> sudo modprobe fuse 2024-03-28T22:33:46.045 DEBUG:teuthology.orchestra.run.smithi043:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T22:33:46.094 INFO:teuthology.orchestra.run.smithi043.stdout:/proc 2024-03-28T22:33:46.094 INFO:teuthology.orchestra.run.smithi043.stdout:/sys 2024-03-28T22:33:46.094 INFO:teuthology.orchestra.run.smithi043.stdout:/dev 2024-03-28T22:33:46.094 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/security 2024-03-28T22:33:46.095 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/shm 2024-03-28T22:33:46.095 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/pts 2024-03-28T22:33:46.095 INFO:teuthology.orchestra.run.smithi043.stdout:/run 2024-03-28T22:33:46.095 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/cgroup 2024-03-28T22:33:46.095 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/pstore 2024-03-28T22:33:46.095 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/bpf 2024-03-28T22:33:46.095 INFO:teuthology.orchestra.run.smithi043.stdout:/ 2024-03-28T22:33:46.095 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/selinux 2024-03-28T22:33:46.095 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T22:33:46.095 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/hugepages 2024-03-28T22:33:46.095 INFO:teuthology.orchestra.run.smithi043.stdout:/dev/mqueue 2024-03-28T22:33:46.095 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/debug 2024-03-28T22:33:46.095 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/tracing 2024-03-28T22:33:46.095 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/fs/fuse/connections 2024-03-28T22:33:46.096 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-sysctl.service 2024-03-28T22:33:46.096 INFO:teuthology.orchestra.run.smithi043.stdout:/sys/kernel/config 2024-03-28T22:33:46.096 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-28T22:33:46.096 INFO:teuthology.orchestra.run.smithi043.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-28T22:33:46.096 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/nfs/rpc_pipefs 2024-03-28T22:33:46.096 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/1000 2024-03-28T22:33:46.096 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/ceph 2024-03-28T22:33:46.096 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay 2024-03-28T22:33:46.096 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/fe518172546cef99a5a8c86e07cb3ceb900a53f271c97a4a51c5d72882e2d6f0/merged 2024-03-28T22:33:46.096 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/4687208e912e6d527d54f8260aac2b58a6a2162e92dbc3c8ce8cda99b750b040/merged 2024-03-28T22:33:46.096 INFO:teuthology.orchestra.run.smithi043.stdout:/run/user/0 2024-03-28T22:33:46.096 INFO:teuthology.orchestra.run.smithi043.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T22:33:46.096 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/22bbf09122ced5d269cff1cf1ffa641964e3219e05403855b4db1e9ea67591eb/merged 2024-03-28T22:33:46.096 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/f62092261c55900a04328b941b86965b516b5af10f2e9560e86752f9200655a1/merged 2024-03-28T22:33:46.097 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/894e112b4640db23e0e30e552fc4a3aca569b8139f45ac0f9b89d224ac08fdcf/merged 2024-03-28T22:33:46.097 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/ef5d0a506eadc7e15ad71dcbcd6726277852fbb09848d3bbf4f185a5104842d3/merged 2024-03-28T22:33:46.097 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/65b41b1bc08fadc010f9fdf317f491c611190b60b2c1c2cde81706d1ed91fd16/merged 2024-03-28T22:33:46.097 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/a92555d89a3757e9d56918cee272cadf7cf600bbf00b4ad0189819387f90aa22/merged 2024-03-28T22:33:46.097 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/d13069e2c5c6b6f7fda815bf0a9b03865dfbe7cb8329a0b1ef826f53588174dc/merged 2024-03-28T22:33:46.097 INFO:teuthology.orchestra.run.smithi043.stdout:/var/lib/containers/storage/overlay/3c8bae2b92dd4fb04072b2be67971623901a5ba172ae2424b052427da3158989/merged 2024-03-28T22:33:46.097 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns 2024-03-28T22:33:46.097 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T22:33:46.097 INFO:teuthology.orchestra.run.smithi043.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T22:33:46.097 INFO:teuthology.orchestra.run.smithi043.stdout:/home/ubuntu/cephtest/mnt.admin 2024-03-28T22:33:46.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:33:46.098 DEBUG:teuthology.orchestra.run.smithi043:> ls /sys/fs/fuse/connections 2024-03-28T22:33:46.139 INFO:teuthology.orchestra.run.smithi043.stdout:118 2024-03-28T22:33:46.140 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [118] 2024-03-28T22:33:46.140 DEBUG:teuthology.orchestra.run.smithi043:> sudo stdin-killer -- python3 -c ' 2024-03-28T22:33:46.140 DEBUG:teuthology.orchestra.run.smithi043:> import glob 2024-03-28T22:33:46.140 DEBUG:teuthology.orchestra.run.smithi043:> import re 2024-03-28T22:33:46.140 DEBUG:teuthology.orchestra.run.smithi043:> import os 2024-03-28T22:33:46.140 DEBUG:teuthology.orchestra.run.smithi043:> import subprocess 2024-03-28T22:33:46.140 DEBUG:teuthology.orchestra.run.smithi043:> 2024-03-28T22:33:46.140 DEBUG:teuthology.orchestra.run.smithi043:> def _find_admin_socket(client_name): 2024-03-28T22:33:46.140 DEBUG:teuthology.orchestra.run.smithi043:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-28T22:33:46.141 DEBUG:teuthology.orchestra.run.smithi043:> files = glob.glob(asok_path) 2024-03-28T22:33:46.141 DEBUG:teuthology.orchestra.run.smithi043:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-28T22:33:46.141 DEBUG:teuthology.orchestra.run.smithi043:> 2024-03-28T22:33:46.141 DEBUG:teuthology.orchestra.run.smithi043:> # Given a non-glob path, it better be there 2024-03-28T22:33:46.141 DEBUG:teuthology.orchestra.run.smithi043:> if "*" not in asok_path: 2024-03-28T22:33:46.141 DEBUG:teuthology.orchestra.run.smithi043:> assert(len(files) == 1) 2024-03-28T22:33:46.141 DEBUG:teuthology.orchestra.run.smithi043:> return files[0] 2024-03-28T22:33:46.141 DEBUG:teuthology.orchestra.run.smithi043:> 2024-03-28T22:33:46.141 DEBUG:teuthology.orchestra.run.smithi043:> for f in files: 2024-03-28T22:33:46.141 DEBUG:teuthology.orchestra.run.smithi043:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-28T22:33:46.141 DEBUG:teuthology.orchestra.run.smithi043:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-28T22:33:46.141 DEBUG:teuthology.orchestra.run.smithi043:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-28T22:33:46.141 DEBUG:teuthology.orchestra.run.smithi043:> contents = proc_f.read() 2024-03-28T22:33:46.141 DEBUG:teuthology.orchestra.run.smithi043:> if mountpoint in contents: 2024-03-28T22:33:46.141 DEBUG:teuthology.orchestra.run.smithi043:> return f 2024-03-28T22:33:46.141 DEBUG:teuthology.orchestra.run.smithi043:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-28T22:33:46.142 DEBUG:teuthology.orchestra.run.smithi043:> 2024-03-28T22:33:46.142 DEBUG:teuthology.orchestra.run.smithi043:> print(_find_admin_socket("client.admin")) 2024-03-28T22:33:46.142 DEBUG:teuthology.orchestra.run.smithi043:> ' 2024-03-28T22:33:46.354 INFO:teuthology.orchestra.run.smithi043.stdout:/var/run/ceph/ceph-client.admin.73365.asok 2024-03-28T22:33:46.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T22:33:46.368 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:46 smithi043 ceph-mon[31823]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-03-28T22:33:46.372 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.73365.asok 2024-03-28T22:33:46.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:33:46.372 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.73365.asok status 2024-03-28T22:33:46.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:46 smithi172 ceph-mon[36566]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-03-28T22:33:46.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:46 smithi082 ceph-mon[36471]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-03-28T22:33:46.616 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:33:46.617 INFO:teuthology.orchestra.run.smithi043.stdout: "metadata": { 2024-03-28T22:33:46.617 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_sha1": "323d23dc228bbcb7f16f188fdc9c0dcb801844be", 2024-03-28T22:33:46.617 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_version": "ceph version 19.0.0-1899-g323d23dc (323d23dc228bbcb7f16f188fdc9c0dcb801844be) squid (dev)", 2024-03-28T22:33:46.617 INFO:teuthology.orchestra.run.smithi043.stdout: "entity_id": "admin", 2024-03-28T22:33:46.617 INFO:teuthology.orchestra.run.smithi043.stdout: "hostname": "smithi043", 2024-03-28T22:33:46.617 INFO:teuthology.orchestra.run.smithi043.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-28T22:33:46.617 INFO:teuthology.orchestra.run.smithi043.stdout: "pid": "73365", 2024-03-28T22:33:46.617 INFO:teuthology.orchestra.run.smithi043.stdout: "root": "/" 2024-03-28T22:33:46.617 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:33:46.617 INFO:teuthology.orchestra.run.smithi043.stdout: "dentry_count": 0, 2024-03-28T22:33:46.617 INFO:teuthology.orchestra.run.smithi043.stdout: "dentry_pinned_count": 0, 2024-03-28T22:33:46.617 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 24698, 2024-03-28T22:33:46.617 INFO:teuthology.orchestra.run.smithi043.stdout: "inst": { 2024-03-28T22:33:46.617 INFO:teuthology.orchestra.run.smithi043.stdout: "name": { 2024-03-28T22:33:46.618 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "client", 2024-03-28T22:33:46.618 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 24698 2024-03-28T22:33:46.618 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:33:46.618 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": { 2024-03-28T22:33:46.618 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-28T22:33:46.618 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:0", 2024-03-28T22:33:46.618 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3900420419 2024-03-28T22:33:46.618 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:33:46.618 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:33:46.618 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": { 2024-03-28T22:33:46.618 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-28T22:33:46.618 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:0", 2024-03-28T22:33:46.618 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3900420419 2024-03-28T22:33:46.618 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:33:46.619 INFO:teuthology.orchestra.run.smithi043.stdout: "inst_str": "client.24698 172.21.15.43:0/3900420419", 2024-03-28T22:33:46.619 INFO:teuthology.orchestra.run.smithi043.stdout: "addr_str": "172.21.15.43:0/3900420419", 2024-03-28T22:33:46.619 INFO:teuthology.orchestra.run.smithi043.stdout: "inode_count": 1, 2024-03-28T22:33:46.619 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_epoch": 17, 2024-03-28T22:33:46.619 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_epoch": 92, 2024-03-28T22:33:46.619 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_epoch_barrier": 0, 2024-03-28T22:33:46.619 INFO:teuthology.orchestra.run.smithi043.stdout: "blocklisted": false, 2024-03-28T22:33:46.619 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs" 2024-03-28T22:33:46.619 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:33:46.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:33:46.632 DEBUG:teuthology.orchestra.run.smithi043:> stat --file-system '--printf=%T 2024-03-28T22:33:46.632 DEBUG:teuthology.orchestra.run.smithi043:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-28T22:33:46.688 INFO:teuthology.orchestra.run.smithi043.stdout:fuseblk 2024-03-28T22:33:46.689 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-28T22:33:46.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:33:46.689 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-28T22:33:46.791 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-28T22:33:46.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:46 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-28T22:33:46.946 INFO:teuthology.orchestra.run.smithi043.stdout:# file: . 2024-03-28T22:33:46.947 INFO:teuthology.orchestra.run.smithi043.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-28T22:33:46.947 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:33:46.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T22:33:46.958 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-28T22:33:46.958 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T22:33:47.013 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:33:47.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.233+0000 7f1c42000640 1 -- 172.21.15.43:0/2446832496 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c3c102780 msgr2=0x7f1c3c102c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:47.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.233+0000 7f1c42000640 1 --2- 172.21.15.43:0/2446832496 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c3c102780 0x7f1c3c102c00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1c28009860 tx=0x7f1c2802bb70 comp rx=0 tx=0).stop 2024-03-28T22:33:47.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.234+0000 7f1c42000640 1 -- 172.21.15.43:0/2446832496 shutdown_connections 2024-03-28T22:33:47.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.234+0000 7f1c42000640 1 --2- 172.21.15.43:0/2446832496 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c3c102780 0x7f1c3c102c00 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:47.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.234+0000 7f1c42000640 1 --2- 172.21.15.43:0/2446832496 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c3c103140 0x7f1c3c1099d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:47.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.234+0000 7f1c42000640 1 --2- 172.21.15.43:0/2446832496 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c3c101590 0x7f1c3c1019b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:47.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.234+0000 7f1c42000640 1 -- 172.21.15.43:0/2446832496 >> 172.21.15.43:0/2446832496 conn(0x7f1c3c0fcae0 msgr2=0x7f1c3c0fef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:47.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.234+0000 7f1c42000640 1 -- 172.21.15.43:0/2446832496 shutdown_connections 2024-03-28T22:33:47.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.234+0000 7f1c42000640 1 -- 172.21.15.43:0/2446832496 wait complete. 2024-03-28T22:33:47.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.235+0000 7f1c42000640 1 Processor -- start 2024-03-28T22:33:47.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.235+0000 7f1c42000640 1 -- start start 2024-03-28T22:33:47.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.235+0000 7f1c42000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c3c101590 0x7f1c3c199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:47.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.235+0000 7f1c42000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c3c102780 0x7f1c3c19a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:47.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.235+0000 7f1c42000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c3c103140 0x7f1c3c1a1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:47.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.235+0000 7f1c42000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3c10d220 con 0x7f1c3c102780 2024-03-28T22:33:47.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.235+0000 7f1c42000640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3c10d0a0 con 0x7f1c3c101590 2024-03-28T22:33:47.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.235+0000 7f1c42000640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3c10d3a0 con 0x7f1c3c103140 2024-03-28T22:33:47.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.236+0000 7f1c40c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c3c101590 0x7f1c3c199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:47.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.236+0000 7f1c3be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c3c102780 0x7f1c3c19a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:47.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.236+0000 7f1c40c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c3c101590 0x7f1c3c199b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48604/0 (socket says 172.21.15.43:48604) 2024-03-28T22:33:47.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.236+0000 7f1c40c00640 1 -- 172.21.15.43:0/1233341337 learned_addr learned my addr 172.21.15.43:0/1233341337 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:47.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.236+0000 7f1c41600640 1 --2- 172.21.15.43:0/1233341337 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c3c103140 0x7f1c3c1a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:47.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.236+0000 7f1c40c00640 1 -- 172.21.15.43:0/1233341337 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c3c103140 msgr2=0x7f1c3c1a1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:47.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.236+0000 7f1c40c00640 1 --2- 172.21.15.43:0/1233341337 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c3c103140 0x7f1c3c1a1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:47.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.236+0000 7f1c40c00640 1 -- 172.21.15.43:0/1233341337 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c3c102780 msgr2=0x7f1c3c19a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:47.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.236+0000 7f1c40c00640 1 --2- 172.21.15.43:0/1233341337 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c3c102780 0x7f1c3c19a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:47.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.236+0000 7f1c40c00640 1 -- 172.21.15.43:0/1233341337 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c3c1a16a0 con 0x7f1c3c101590 2024-03-28T22:33:47.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.236+0000 7f1c41600640 1 --2- 172.21.15.43:0/1233341337 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c3c103140 0x7f1c3c1a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:33:47.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.236+0000 7f1c40c00640 1 --2- 172.21.15.43:0/1233341337 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c3c101590 0x7f1c3c199b20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1c3000e9f0 tx=0x7f1c3000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:47.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.237+0000 7f1c39600640 1 -- 172.21.15.43:0/1233341337 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c3000ccc0 con 0x7f1c3c101590 2024-03-28T22:33:47.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.237+0000 7f1c39600640 1 -- 172.21.15.43:0/1233341337 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c300035f0 con 0x7f1c3c101590 2024-03-28T22:33:47.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.237+0000 7f1c39600640 1 -- 172.21.15.43:0/1233341337 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c3000ce60 con 0x7f1c3c101590 2024-03-28T22:33:47.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.237+0000 7f1c42000640 1 -- 172.21.15.43:0/1233341337 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c3c1a1990 con 0x7f1c3c101590 2024-03-28T22:33:47.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.237+0000 7f1c42000640 1 -- 172.21.15.43:0/1233341337 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c3c1a1e60 con 0x7f1c3c101590 2024-03-28T22:33:47.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.238+0000 7f1c42000640 1 -- 172.21.15.43:0/1233341337 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c3c1a20f0 con 0x7f1c3c101590 2024-03-28T22:33:47.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.240+0000 7f1c39600640 1 -- 172.21.15.43:0/1233341337 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f1c300037a0 con 0x7f1c3c101590 2024-03-28T22:33:47.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.241+0000 7f1c39600640 1 --2- 172.21.15.43:0/1233341337 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f1c240a1e00 0x7f1c240a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:47.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.241+0000 7f1c39600640 1 -- 172.21.15.43:0/1233341337 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f1c300ce2c0 con 0x7f1c3c101590 2024-03-28T22:33:47.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.241+0000 7f1c3be00640 1 --2- 172.21.15.43:0/1233341337 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f1c240a1e00 0x7f1c240a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:47.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.242+0000 7f1c3be00640 1 --2- 172.21.15.43:0/1233341337 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f1c240a1e00 0x7f1c240a42c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1c28009830 tx=0x7f1c28034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:47.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.245+0000 7f1c39600640 1 -- 172.21.15.43:0/1233341337 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1c3009b450 con 0x7f1c3c101590 2024-03-28T22:33:47.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.447+0000 7f1c42000640 1 -- 172.21.15.43:0/1233341337 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1c3c1a2930 con 0x7f1c3c101590 2024-03-28T22:33:47.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.449+0000 7f1c39600640 1 -- 172.21.15.43:0/1233341337 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24093 (secure 0 0 0) 0x7f1c3009ad50 con 0x7f1c3c101590 2024-03-28T22:33:47.447 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:33:47.447 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":92,"fsid":"c2276428-ed51-11ee-95d0-87774f69a715","created":"2024-03-28T22:24:19.950084+0000","modified":"2024-03-28T22:33:34.473170+0000","last_up_change":"2024-03-28T22:32:31.078737+0000","last_in_change":"2024-03-28T22:32:10.147097+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:28:14.713290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:33:27.167435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:33:28.411825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T22:33:32.523122+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b855adcd-786c-4e42-b8fe-a515a77ce979","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6803","nonce":2826124409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6805","nonce":2826124409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6809","nonce":2826124409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6807","nonce":2826124409}]},"public_addr":"172.21.15.43:6803/2826124409","cluster_addr":"172.21.15.43:6805/2826124409","heartbeat_back_addr":"172.21.15.43:6809/2826124409","heartbeat_front_addr":"172.21.15.43:6807/2826124409","state":["exists","up"]},{"osd":1,"uuid":"031c143d-8ad1-4b68-8db9-3c717137e3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6801","nonce":3043263644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6803","nonce":3043263644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6807","nonce":3043263644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6805","nonce":3043263644}]},"public_addr":"172.21.15.82:6801/3043263644","cluster_addr":"172.21.15.82:6803/3043263644","heartbeat_back_addr":"172.21.15.82:6807/3043263644","heartbeat_front_addr":"172.21.15.82:6805/3043263644","state":["exists","up"]},{"osd":2,"uuid":"20632bbb-f07a-4ca1-b0c4-ff9d508e211c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6801","nonce":4213953645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6803","nonce":4213953645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6807","nonce":4213953645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6805","nonce":4213953645}]},"public_addr":"172.21.15.172:6801/4213953645","cluster_addr":"172.21.15.172:6803/4213953645","heartbeat_back_addr":"172.21.15.172:6807/4213953645","heartbeat_front_addr":"172.21.15.172:6805/4213953645","state":["exists","up"]},{"osd":3,"uuid":"9a1f2312-4dbd-475b-8736-d5ebc8788e09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6811","nonce":1597280920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6813","nonce":1597280920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6817","nonce":1597280920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6815","nonce":1597280920}]},"public_addr":"172.21.15.43:6811/1597280920","cluster_addr":"172.21.15.43:6813/1597280920","heartbeat_back_addr":"172.21.15.43:6817/1597280920","heartbeat_front_addr":"172.21.15.43:6815/1597280920","state":["exists","up"]},{"osd":4,"uuid":"29140ead-7075-42b2-90c1-27cae0b29ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6809","nonce":215451895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6811","nonce":215451895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6815","nonce":215451895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6813","nonce":215451895}]},"public_addr":"172.21.15.82:6809/215451895","cluster_addr":"172.21.15.82:6811/215451895","heartbeat_back_addr":"172.21.15.82:6815/215451895","heartbeat_front_addr":"172.21.15.82:6813/215451895","state":["exists","up"]},{"osd":5,"uuid":"59644c61-93dc-4536-b2ad-0256b24e5cb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6809","nonce":1576148933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6811","nonce":1576148933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6815","nonce":1576148933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6813","nonce":1576148933}]},"public_addr":"172.21.15.172:6809/1576148933","cluster_addr":"172.21.15.172:6811/1576148933","heartbeat_back_addr":"172.21.15.172:6815/1576148933","heartbeat_front_addr":"172.21.15.172:6813/1576148933","state":["exists","up"]},{"osd":6,"uuid":"22d54e57-c3f1-4ecf-9921-8e815b48f92a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6819","nonce":791044279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6821","nonce":791044279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6825","nonce":791044279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6823","nonce":791044279}]},"public_addr":"172.21.15.43:6819/791044279","cluster_addr":"172.21.15.43:6821/791044279","heartbeat_back_addr":"172.21.15.43:6825/791044279","heartbeat_front_addr":"172.21.15.43:6823/791044279","state":["exists","up"]},{"osd":7,"uuid":"248cd1e2-0d6b-452f-a0a8-c51fe1b975ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6817","nonce":2157777323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6819","nonce":2157777323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6823","nonce":2157777323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6821","nonce":2157777323}]},"public_addr":"172.21.15.82:6817/2157777323","cluster_addr":"172.21.15.82:6819/2157777323","heartbeat_back_addr":"172.21.15.82:6823/2157777323","heartbeat_front_addr":"172.21.15.82:6821/2157777323","state":["exists","up"]},{"osd":8,"uuid":"d98aab6e-8bae-4c24-906c-4f492f047b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6817","nonce":1095328248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6819","nonce":1095328248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6823","nonce":1095328248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6821","nonce":1095328248}]},"public_addr":"172.21.15.172:6817/1095328248","cluster_addr":"172.21.15.172:6819/1095328248","heartbeat_back_addr":"172.21.15.172:6823/1095328248","heartbeat_front_addr":"172.21.15.172:6821/1095328248","state":["exists","up"]},{"osd":9,"uuid":"19aa383e-a0f6-45ea-96cc-e7956535caa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6827","nonce":3657791498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6829","nonce":3657791498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6833","nonce":3657791498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6831","nonce":3657791498}]},"public_addr":"172.21.15.43:6827/3657791498","cluster_addr":"172.21.15.43:6829/3657791498","heartbeat_back_addr":"172.21.15.43:6833/3657791498","heartbeat_front_addr":"172.21.15.43:6831/3657791498","state":["exists","up"]},{"osd":10,"uuid":"666977e5-eaf7-41b9-a18f-c380e149345d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6825","nonce":3454954047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6827","nonce":3454954047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6831","nonce":3454954047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6829","nonce":3454954047}]},"public_addr":"172.21.15.82:6825/3454954047","cluster_addr":"172.21.15.82:6827/3454954047","heartbeat_back_addr":"172.21.15.82:6831/3454954047","heartbeat_front_addr":"172.21.15.82:6829/3454954047","state":["exists","up"]},{"osd":11,"uuid":"615cf3be-8076-470e-ad09-c880e7593717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6825","nonce":2563517144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6827","nonce":2563517144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6831","nonce":2563517144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6829","nonce":2563517144}]},"public_addr":"172.21.15.172:6825/2563517144","cluster_addr":"172.21.15.172:6827/2563517144","heartbeat_back_addr":"172.21.15.172:6831/2563517144","heartbeat_front_addr":"172.21.15.172:6829/2563517144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:21.727405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:47.100259+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:10.615438+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:43.632144+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:09.487940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:34.604463+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:09.594452+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:35.709795+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:01.184202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:35.455322+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:02.413166+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:28.550754+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/578791143":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/4029313215":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/1574178482":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6801/3517712438":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6800/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/1375099638":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/2710117652":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6801/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/3215634168":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6800/3517712438":"2024-03-29T22:25:28.008215+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:33:47.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.451+0000 7f1c42000640 1 -- 172.21.15.43:0/1233341337 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f1c240a1e00 msgr2=0x7f1c240a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:47.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.451+0000 7f1c42000640 1 --2- 172.21.15.43:0/1233341337 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f1c240a1e00 0x7f1c240a42c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1c28009830 tx=0x7f1c28034ab0 comp rx=0 tx=0).stop 2024-03-28T22:33:47.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.451+0000 7f1c42000640 1 -- 172.21.15.43:0/1233341337 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c3c101590 msgr2=0x7f1c3c199b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:47.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.451+0000 7f1c42000640 1 --2- 172.21.15.43:0/1233341337 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c3c101590 0x7f1c3c199b20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1c3000e9f0 tx=0x7f1c3000eeb0 comp rx=0 tx=0).stop 2024-03-28T22:33:47.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.451+0000 7f1c42000640 1 -- 172.21.15.43:0/1233341337 shutdown_connections 2024-03-28T22:33:47.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.451+0000 7f1c42000640 1 --2- 172.21.15.43:0/1233341337 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c3c103140 0x7f1c3c1a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:47.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.451+0000 7f1c42000640 1 --2- 172.21.15.43:0/1233341337 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1c3c102780 0x7f1c3c19a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:47.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.451+0000 7f1c42000640 1 --2- 172.21.15.43:0/1233341337 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f1c240a1e00 0x7f1c240a42c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:47.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.451+0000 7f1c42000640 1 --2- 172.21.15.43:0/1233341337 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c3c101590 0x7f1c3c199b20 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:47.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.451+0000 7f1c42000640 1 -- 172.21.15.43:0/1233341337 >> 172.21.15.43:0/1233341337 conn(0x7f1c3c0fcae0 msgr2=0x7f1c3c073980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:47.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.452+0000 7f1c42000640 1 -- 172.21.15.43:0/1233341337 shutdown_connections 2024-03-28T22:33:47.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.452+0000 7f1c42000640 1 -- 172.21.15.43:0/1233341337 wait complete. 2024-03-28T22:33:47.471 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T22:28:14.713290+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T22:33:27.167435+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T22:33:28.411825+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T22:33:32.523122+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T22:33:47.471 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T22:33:47.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.694+0000 7fdf4ea00640 1 -- 172.21.15.43:0/2088925574 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf480ffea0 msgr2=0x7fdf481002c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:47.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.694+0000 7fdf4ea00640 1 --2- 172.21.15.43:0/2088925574 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf480ffea0 0x7fdf481002c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c009900 tx=0x7fdf3c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:33:47.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.694+0000 7fdf4ea00640 1 -- 172.21.15.43:0/2088925574 shutdown_connections 2024-03-28T22:33:47.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.694+0000 7fdf4ea00640 1 --2- 172.21.15.43:0/2088925574 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf481011a0 0x7fdf48101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:47.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.694+0000 7fdf4ea00640 1 --2- 172.21.15.43:0/2088925574 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf48100800 0x7fdf48100c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:47.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.694+0000 7fdf4ea00640 1 --2- 172.21.15.43:0/2088925574 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf480ffea0 0x7fdf481002c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:47.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.694+0000 7fdf4ea00640 1 -- 172.21.15.43:0/2088925574 >> 172.21.15.43:0/2088925574 conn(0x7fdf480fbac0 msgr2=0x7fdf480fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:47.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.694+0000 7fdf4ea00640 1 -- 172.21.15.43:0/2088925574 shutdown_connections 2024-03-28T22:33:47.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.694+0000 7fdf4ea00640 1 -- 172.21.15.43:0/2088925574 wait complete. 2024-03-28T22:33:47.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.695+0000 7fdf4ea00640 1 Processor -- start 2024-03-28T22:33:47.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.696+0000 7fdf4ea00640 1 -- start start 2024-03-28T22:33:47.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.696+0000 7fdf4ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf480ffea0 0x7fdf48199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:47.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.696+0000 7fdf4ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf48100800 0x7fdf4819a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:47.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.696+0000 7fdf4ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf481011a0 0x7fdf481a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:47.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.696+0000 7fdf4ea00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fdf4810f4c0 con 0x7fdf480ffea0 2024-03-28T22:33:47.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.696+0000 7fdf4ea00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fdf4810f340 con 0x7fdf48100800 2024-03-28T22:33:47.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.696+0000 7fdf4ea00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fdf4810f640 con 0x7fdf481011a0 2024-03-28T22:33:47.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.696+0000 7fdf4d600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf480ffea0 0x7fdf48199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:47.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.696+0000 7fdf4d600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf480ffea0 0x7fdf48199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59558/0 (socket says 172.21.15.43:59558) 2024-03-28T22:33:47.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.696+0000 7fdf4d600640 1 -- 172.21.15.43:0/2478358035 learned_addr learned my addr 172.21.15.43:0/2478358035 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:47.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.696+0000 7fdf4cc00640 1 --2- 172.21.15.43:0/2478358035 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf48100800 0x7fdf4819a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:47.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.697+0000 7fdf4e000640 1 --2- 172.21.15.43:0/2478358035 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf481011a0 0x7fdf481a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:47.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.697+0000 7fdf4cc00640 1 -- 172.21.15.43:0/2478358035 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf481011a0 msgr2=0x7fdf481a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:47.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.697+0000 7fdf4cc00640 1 --2- 172.21.15.43:0/2478358035 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf481011a0 0x7fdf481a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:47.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.697+0000 7fdf4cc00640 1 -- 172.21.15.43:0/2478358035 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf480ffea0 msgr2=0x7fdf48199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:47.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.697+0000 7fdf4cc00640 1 --2- 172.21.15.43:0/2478358035 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf480ffea0 0x7fdf48199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:47.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.697+0000 7fdf4cc00640 1 -- 172.21.15.43:0/2478358035 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf481a1720 con 0x7fdf48100800 2024-03-28T22:33:47.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.697+0000 7fdf4d600640 1 --2- 172.21.15.43:0/2478358035 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf480ffea0 0x7fdf48199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:33:47.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.697+0000 7fdf4cc00640 1 --2- 172.21.15.43:0/2478358035 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf48100800 0x7fdf4819a040 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdf3400d6d0 tx=0x7fdf3400db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:47.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.697+0000 7fdf46000640 1 -- 172.21.15.43:0/2478358035 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf34017070 con 0x7fdf48100800 2024-03-28T22:33:47.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.697+0000 7fdf46000640 1 -- 172.21.15.43:0/2478358035 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf34009820 con 0x7fdf48100800 2024-03-28T22:33:47.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.697+0000 7fdf46000640 1 -- 172.21.15.43:0/2478358035 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf34004d00 con 0x7fdf48100800 2024-03-28T22:33:47.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.697+0000 7fdf4ea00640 1 -- 172.21.15.43:0/2478358035 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf481a1a10 con 0x7fdf48100800 2024-03-28T22:33:47.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.698+0000 7fdf4ea00640 1 -- 172.21.15.43:0/2478358035 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf481a1f10 con 0x7fdf48100800 2024-03-28T22:33:47.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.699+0000 7fdf4ea00640 1 -- 172.21.15.43:0/2478358035 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf4810ad50 con 0x7fdf48100800 2024-03-28T22:33:47.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.701+0000 7fdf46000640 1 -- 172.21.15.43:0/2478358035 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fdf340099d0 con 0x7fdf48100800 2024-03-28T22:33:47.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.703+0000 7fdf46000640 1 --2- 172.21.15.43:0/2478358035 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fdf300a1e00 0x7fdf300a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:47.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.703+0000 7fdf46000640 1 -- 172.21.15.43:0/2478358035 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7fdf340cda90 con 0x7fdf48100800 2024-03-28T22:33:47.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.703+0000 7fdf4d600640 1 --2- 172.21.15.43:0/2478358035 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fdf300a1e00 0x7fdf300a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:47.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.704+0000 7fdf4d600640 1 --2- 172.21.15.43:0/2478358035 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fdf300a1e00 0x7fdf300a42c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c02c8e0 tx=0x7fdf3c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:47.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.707+0000 7fdf46000640 1 -- 172.21.15.43:0/2478358035 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdf3409a5d0 con 0x7fdf48100800 2024-03-28T22:33:47.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.908+0000 7fdf4ea00640 1 -- 172.21.15.43:0/2478358035 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdf48062350 con 0x7fdf48100800 2024-03-28T22:33:47.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.909+0000 7fdf46000640 1 -- 172.21.15.43:0/2478358035 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7fdf3409e670 con 0x7fdf48100800 2024-03-28T22:33:47.908 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-03-28T22:33:47.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.912+0000 7fdf4ea00640 1 -- 172.21.15.43:0/2478358035 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fdf300a1e00 msgr2=0x7fdf300a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:47.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.912+0000 7fdf4ea00640 1 --2- 172.21.15.43:0/2478358035 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fdf300a1e00 0x7fdf300a42c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c02c8e0 tx=0x7fdf3c02df90 comp rx=0 tx=0).stop 2024-03-28T22:33:47.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.912+0000 7fdf4ea00640 1 -- 172.21.15.43:0/2478358035 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf48100800 msgr2=0x7fdf4819a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:47.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.912+0000 7fdf4ea00640 1 --2- 172.21.15.43:0/2478358035 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf48100800 0x7fdf4819a040 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdf3400d6d0 tx=0x7fdf3400db90 comp rx=0 tx=0).stop 2024-03-28T22:33:47.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.912+0000 7fdf4ea00640 1 -- 172.21.15.43:0/2478358035 shutdown_connections 2024-03-28T22:33:47.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.912+0000 7fdf4ea00640 1 --2- 172.21.15.43:0/2478358035 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fdf300a1e00 0x7fdf300a42c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:47.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.912+0000 7fdf4ea00640 1 --2- 172.21.15.43:0/2478358035 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf48100800 0x7fdf4819a040 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:47.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.912+0000 7fdf4ea00640 1 --2- 172.21.15.43:0/2478358035 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf481011a0 0x7fdf481a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:47.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.912+0000 7fdf4ea00640 1 --2- 172.21.15.43:0/2478358035 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf480ffea0 0x7fdf48199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:47.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.912+0000 7fdf4ea00640 1 -- 172.21.15.43:0/2478358035 >> 172.21.15.43:0/2478358035 conn(0x7fdf480fbac0 msgr2=0x7fdf480fd190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:47.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.912+0000 7fdf4ea00640 1 -- 172.21.15.43:0/2478358035 shutdown_connections 2024-03-28T22:33:47.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:47.912+0000 7fdf4ea00640 1 -- 172.21.15.43:0/2478358035 wait complete. 2024-03-28T22:33:47.929 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T22:33:48.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.144+0000 7ffad7600640 1 -- 172.21.15.43:0/422669081 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffad0105540 msgr2=0x7ffad010bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:48.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.144+0000 7ffad7600640 1 --2- 172.21.15.43:0/422669081 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffad0105540 0x7ffad010bdd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ffac8009ca0 tx=0x7ffac802c0f0 comp rx=0 tx=0).stop 2024-03-28T22:33:48.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.144+0000 7ffad7600640 1 -- 172.21.15.43:0/422669081 shutdown_connections 2024-03-28T22:33:48.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.144+0000 7ffad7600640 1 --2- 172.21.15.43:0/422669081 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffad0105540 0x7ffad010bdd0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:48.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.144+0000 7ffad7600640 1 --2- 172.21.15.43:0/422669081 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffad0104b80 0x7ffad0105000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:48.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.144+0000 7ffad7600640 1 --2- 172.21.15.43:0/422669081 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffad0103990 0x7ffad0103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:48.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.144+0000 7ffad7600640 1 -- 172.21.15.43:0/422669081 >> 172.21.15.43:0/422669081 conn(0x7ffad00fef00 msgr2=0x7ffad0101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:48.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.144+0000 7ffad7600640 1 -- 172.21.15.43:0/422669081 shutdown_connections 2024-03-28T22:33:48.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.144+0000 7ffad7600640 1 -- 172.21.15.43:0/422669081 wait complete. 2024-03-28T22:33:48.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.145+0000 7ffad7600640 1 Processor -- start 2024-03-28T22:33:48.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.145+0000 7ffad7600640 1 -- start start 2024-03-28T22:33:48.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.146+0000 7ffad7600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffad0103990 0x7ffad0199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:48.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.146+0000 7ffad7600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffad0104b80 0x7ffad019a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:48.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.146+0000 7ffad7600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffad0105540 0x7ffad01a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:48.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.146+0000 7ffad7600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ffad010f5f0 con 0x7ffad0103990 2024-03-28T22:33:48.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.146+0000 7ffad7600640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ffad010f470 con 0x7ffad0105540 2024-03-28T22:33:48.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.146+0000 7ffad7600640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7ffad010f770 con 0x7ffad0104b80 2024-03-28T22:33:48.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.146+0000 7ffad4c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffad0103990 0x7ffad0199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:48.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.146+0000 7ffad5600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffad0105540 0x7ffad01a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:48.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.146+0000 7ffad5600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffad0105540 0x7ffad01a1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48634/0 (socket says 172.21.15.43:48634) 2024-03-28T22:33:48.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.146+0000 7ffad5600640 1 -- 172.21.15.43:0/3478221186 learned_addr learned my addr 172.21.15.43:0/3478221186 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:48.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.146+0000 7ffad4c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffad0103990 0x7ffad0199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59578/0 (socket says 172.21.15.43:59578) 2024-03-28T22:33:48.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.146+0000 7ffac7e00640 1 --2- 172.21.15.43:0/3478221186 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffad0104b80 0x7ffad019a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:48.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.146+0000 7ffad5600640 1 -- 172.21.15.43:0/3478221186 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffad0104b80 msgr2=0x7ffad019a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:48.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.146+0000 7ffad5600640 1 --2- 172.21.15.43:0/3478221186 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffad0104b80 0x7ffad019a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:48.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.146+0000 7ffad5600640 1 -- 172.21.15.43:0/3478221186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffad0103990 msgr2=0x7ffad0199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:48.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.146+0000 7ffad5600640 1 --2- 172.21.15.43:0/3478221186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffad0103990 0x7ffad0199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:48.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.146+0000 7ffad5600640 1 -- 172.21.15.43:0/3478221186 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffad01a1830 con 0x7ffad0105540 2024-03-28T22:33:48.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.147+0000 7ffad5600640 1 --2- 172.21.15.43:0/3478221186 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffad0105540 0x7ffad01a1190 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ffac802c630 tx=0x7ffac8032ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:48.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.147+0000 7fface000640 1 -- 172.21.15.43:0/3478221186 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac8041020 con 0x7ffad0105540 2024-03-28T22:33:48.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.147+0000 7fface000640 1 -- 172.21.15.43:0/3478221186 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffac8034cd0 con 0x7ffad0105540 2024-03-28T22:33:48.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.147+0000 7ffad7600640 1 -- 172.21.15.43:0/3478221186 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffad01a1ac0 con 0x7ffad0105540 2024-03-28T22:33:48.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.147+0000 7fface000640 1 -- 172.21.15.43:0/3478221186 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac8039040 con 0x7ffad0105540 2024-03-28T22:33:48.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.147+0000 7ffad7600640 1 -- 172.21.15.43:0/3478221186 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffad01a2000 con 0x7ffad0105540 2024-03-28T22:33:48.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.148+0000 7ffad7600640 1 -- 172.21.15.43:0/3478221186 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffad010ac50 con 0x7ffad0105540 2024-03-28T22:33:48.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.150+0000 7fface000640 1 -- 172.21.15.43:0/3478221186 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7ffac8044020 con 0x7ffad0105540 2024-03-28T22:33:48.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.151+0000 7fface000640 1 --2- 172.21.15.43:0/3478221186 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ffab40a1e00 0x7ffab40a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:48.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.151+0000 7fface000640 1 -- 172.21.15.43:0/3478221186 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7ffac80efe60 con 0x7ffad0105540 2024-03-28T22:33:48.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.152+0000 7ffad4c00640 1 --2- 172.21.15.43:0/3478221186 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ffab40a1e00 0x7ffab40a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:48.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.152+0000 7ffad4c00640 1 --2- 172.21.15.43:0/3478221186 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ffab40a1e00 0x7ffab40a42c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ffad019b0f0 tx=0x7ffac0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:48.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.155+0000 7fface000640 1 -- 172.21.15.43:0/3478221186 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ffac804b020 con 0x7ffad0105540 2024-03-28T22:33:48.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:48 smithi043 ceph-mon[31823]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:48.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:48 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1233341337' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:33:48.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:48 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2478358035' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:33:48.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.356+0000 7ffad7600640 1 -- 172.21.15.43:0/3478221186 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ffad002d170 con 0x7ffad0105540 2024-03-28T22:33:48.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.357+0000 7fface000640 1 -- 172.21.15.43:0/3478221186 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7ffac80bcff0 con 0x7ffad0105540 2024-03-28T22:33:48.355 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-03-28T22:33:48.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.359+0000 7ffad7600640 1 -- 172.21.15.43:0/3478221186 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ffab40a1e00 msgr2=0x7ffab40a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:48.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.359+0000 7ffad7600640 1 --2- 172.21.15.43:0/3478221186 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ffab40a1e00 0x7ffab40a42c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ffad019b0f0 tx=0x7ffac0005f90 comp rx=0 tx=0).stop 2024-03-28T22:33:48.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.359+0000 7ffad7600640 1 -- 172.21.15.43:0/3478221186 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffad0105540 msgr2=0x7ffad01a1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:48.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.359+0000 7ffad7600640 1 --2- 172.21.15.43:0/3478221186 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffad0105540 0x7ffad01a1190 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ffac802c630 tx=0x7ffac8032ea0 comp rx=0 tx=0).stop 2024-03-28T22:33:48.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.359+0000 7ffad7600640 1 -- 172.21.15.43:0/3478221186 shutdown_connections 2024-03-28T22:33:48.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.359+0000 7ffad7600640 1 --2- 172.21.15.43:0/3478221186 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ffab40a1e00 0x7ffab40a42c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:48.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.359+0000 7ffad7600640 1 --2- 172.21.15.43:0/3478221186 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffad0105540 0x7ffad01a1190 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:48.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.359+0000 7ffad7600640 1 --2- 172.21.15.43:0/3478221186 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffad0104b80 0x7ffad019a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:48.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.359+0000 7ffad7600640 1 --2- 172.21.15.43:0/3478221186 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffad0103990 0x7ffad0199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:48.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.359+0000 7ffad7600640 1 -- 172.21.15.43:0/3478221186 >> 172.21.15.43:0/3478221186 conn(0x7ffad00fef00 msgr2=0x7ffad0107e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:48.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.359+0000 7ffad7600640 1 -- 172.21.15.43:0/3478221186 shutdown_connections 2024-03-28T22:33:48.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.359+0000 7ffad7600640 1 -- 172.21.15.43:0/3478221186 wait complete. 2024-03-28T22:33:48.375 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T22:33:48.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:48 smithi172 ceph-mon[36566]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:48.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:48 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1233341337' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:33:48.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:48 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2478358035' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:33:48.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:48 smithi082 ceph-mon[36471]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:48.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:48 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1233341337' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:33:48.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:48 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2478358035' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:33:48.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.590+0000 7fd60f200640 1 -- 172.21.15.43:0/3260931360 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd610104a40 msgr2=0x7fd610104ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:48.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.590+0000 7fd60f200640 1 --2- 172.21.15.43:0/3260931360 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd610104a40 0x7fd610104ec0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd600009ca0 tx=0x7fd60002c0f0 comp rx=0 tx=0).stop 2024-03-28T22:33:48.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.590+0000 7fd60f200640 1 -- 172.21.15.43:0/3260931360 shutdown_connections 2024-03-28T22:33:48.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.590+0000 7fd60f200640 1 --2- 172.21.15.43:0/3260931360 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd610104a40 0x7fd610104ec0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:48.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.590+0000 7fd60f200640 1 --2- 172.21.15.43:0/3260931360 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd610105400 0x7fd61010bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:48.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.590+0000 7fd60f200640 1 --2- 172.21.15.43:0/3260931360 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd610103850 0x7fd610103c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:48.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.590+0000 7fd60f200640 1 -- 172.21.15.43:0/3260931360 >> 172.21.15.43:0/3260931360 conn(0x7fd6100fedc0 msgr2=0x7fd610101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:48.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.590+0000 7fd60f200640 1 -- 172.21.15.43:0/3260931360 shutdown_connections 2024-03-28T22:33:48.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.590+0000 7fd60f200640 1 -- 172.21.15.43:0/3260931360 wait complete. 2024-03-28T22:33:48.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.591+0000 7fd60f200640 1 Processor -- start 2024-03-28T22:33:48.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.592+0000 7fd60f200640 1 -- start start 2024-03-28T22:33:48.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.592+0000 7fd60f200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd610103850 0x7fd610199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:48.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.592+0000 7fd60f200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd610104a40 0x7fd61019a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:48.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.592+0000 7fd60f200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd610105400 0x7fd6101a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:48.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.592+0000 7fd60f200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd61010f4b0 con 0x7fd610104a40 2024-03-28T22:33:48.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.592+0000 7fd60f200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd61010f330 con 0x7fd610103850 2024-03-28T22:33:48.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.592+0000 7fd60f200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd61010f630 con 0x7fd610105400 2024-03-28T22:33:48.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.592+0000 7fd60de00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd610103850 0x7fd610199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:48.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.592+0000 7fd60de00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd610103850 0x7fd610199af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48660/0 (socket says 172.21.15.43:48660) 2024-03-28T22:33:48.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.592+0000 7fd60de00640 1 -- 172.21.15.43:0/3520693435 learned_addr learned my addr 172.21.15.43:0/3520693435 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:48.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.592+0000 7fd60d400640 1 --2- 172.21.15.43:0/3520693435 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd610104a40 0x7fd61019a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:48.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.592+0000 7fd60e800640 1 --2- 172.21.15.43:0/3520693435 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd610105400 0x7fd6101a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:48.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.593+0000 7fd60de00640 1 -- 172.21.15.43:0/3520693435 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd610105400 msgr2=0x7fd6101a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:48.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.593+0000 7fd60de00640 1 --2- 172.21.15.43:0/3520693435 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd610105400 0x7fd6101a10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:48.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.593+0000 7fd60de00640 1 -- 172.21.15.43:0/3520693435 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd610104a40 msgr2=0x7fd61019a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:48.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.593+0000 7fd60de00640 1 --2- 172.21.15.43:0/3520693435 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd610104a40 0x7fd61019a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:48.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.593+0000 7fd60de00640 1 -- 172.21.15.43:0/3520693435 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6101a1710 con 0x7fd610103850 2024-03-28T22:33:48.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.593+0000 7fd60de00640 1 --2- 172.21.15.43:0/3520693435 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd610103850 0x7fd610199af0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd5f800e9c0 tx=0x7fd5f800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:48.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.593+0000 7fd606a00640 1 -- 172.21.15.43:0/3520693435 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f800cd80 con 0x7fd610103850 2024-03-28T22:33:48.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.593+0000 7fd606a00640 1 -- 172.21.15.43:0/3520693435 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5f80035f0 con 0x7fd610103850 2024-03-28T22:33:48.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.593+0000 7fd606a00640 1 -- 172.21.15.43:0/3520693435 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f8010690 con 0x7fd610103850 2024-03-28T22:33:48.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.593+0000 7fd60f200640 1 -- 172.21.15.43:0/3520693435 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6101a1a00 con 0x7fd610103850 2024-03-28T22:33:48.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.593+0000 7fd60f200640 1 -- 172.21.15.43:0/3520693435 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6101a1ed0 con 0x7fd610103850 2024-03-28T22:33:48.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.595+0000 7fd60f200640 1 -- 172.21.15.43:0/3520693435 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd610068460 con 0x7fd610103850 2024-03-28T22:33:48.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.596+0000 7fd606a00640 1 -- 172.21.15.43:0/3520693435 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fd5f80037a0 con 0x7fd610103850 2024-03-28T22:33:48.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.598+0000 7fd606a00640 1 --2- 172.21.15.43:0/3520693435 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd5f40a1d80 0x7fd5f40a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:48.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.598+0000 7fd606a00640 1 -- 172.21.15.43:0/3520693435 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7fd5f80ce550 con 0x7fd610103850 2024-03-28T22:33:48.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.598+0000 7fd60d400640 1 --2- 172.21.15.43:0/3520693435 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd5f40a1d80 0x7fd5f40a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:48.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.598+0000 7fd60d400640 1 --2- 172.21.15.43:0/3520693435 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd5f40a1d80 0x7fd5f40a4240 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd600006010 tx=0x7fd600032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:48.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.602+0000 7fd606a00640 1 -- 172.21.15.43:0/3520693435 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd5f809b6e0 con 0x7fd610103850 2024-03-28T22:33:48.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.806+0000 7fd60f200640 1 -- 172.21.15.43:0/3520693435 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd61002d170 con 0x7fd610103850 2024-03-28T22:33:48.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.806+0000 7fd606a00640 1 -- 172.21.15.43:0/3520693435 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7fd5f809afe0 con 0x7fd610103850 2024-03-28T22:33:48.805 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-03-28T22:33:48.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.809+0000 7fd60f200640 1 -- 172.21.15.43:0/3520693435 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd5f40a1d80 msgr2=0x7fd5f40a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:48.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.809+0000 7fd60f200640 1 --2- 172.21.15.43:0/3520693435 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd5f40a1d80 0x7fd5f40a4240 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd600006010 tx=0x7fd600032590 comp rx=0 tx=0).stop 2024-03-28T22:33:48.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.809+0000 7fd60f200640 1 -- 172.21.15.43:0/3520693435 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd610103850 msgr2=0x7fd610199af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:48.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.809+0000 7fd60f200640 1 --2- 172.21.15.43:0/3520693435 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd610103850 0x7fd610199af0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd5f800e9c0 tx=0x7fd5f800ee80 comp rx=0 tx=0).stop 2024-03-28T22:33:48.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.810+0000 7fd60f200640 1 -- 172.21.15.43:0/3520693435 shutdown_connections 2024-03-28T22:33:48.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.810+0000 7fd60f200640 1 --2- 172.21.15.43:0/3520693435 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd610105400 0x7fd6101a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:48.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.810+0000 7fd60f200640 1 --2- 172.21.15.43:0/3520693435 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd610104a40 0x7fd61019a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:48.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.810+0000 7fd60f200640 1 --2- 172.21.15.43:0/3520693435 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd5f40a1d80 0x7fd5f40a4240 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:48.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.810+0000 7fd60f200640 1 --2- 172.21.15.43:0/3520693435 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd610103850 0x7fd610199af0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:48.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.810+0000 7fd60f200640 1 -- 172.21.15.43:0/3520693435 >> 172.21.15.43:0/3520693435 conn(0x7fd6100fedc0 msgr2=0x7fd610107d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:48.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.810+0000 7fd60f200640 1 -- 172.21.15.43:0/3520693435 shutdown_connections 2024-03-28T22:33:48.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:48.810+0000 7fd60f200640 1 -- 172.21.15.43:0/3520693435 wait complete. 2024-03-28T22:33:48.826 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T22:33:49.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.057+0000 7f9a27000640 1 -- 172.21.15.43:0/1662793245 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a20101650 msgr2=0x7f9a20101a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:49.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.057+0000 7f9a27000640 1 --2- 172.21.15.43:0/1662793245 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a20101650 0x7f9a20101a70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9a14009ca0 tx=0x7f9a1402c100 comp rx=0 tx=0).stop 2024-03-28T22:33:49.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.057+0000 7f9a27000640 1 -- 172.21.15.43:0/1662793245 shutdown_connections 2024-03-28T22:33:49.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.057+0000 7f9a27000640 1 --2- 172.21.15.43:0/1662793245 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a20103200 0x7f9a20109a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:49.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.057+0000 7f9a27000640 1 --2- 172.21.15.43:0/1662793245 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a20102840 0x7f9a20102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:49.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.057+0000 7f9a27000640 1 --2- 172.21.15.43:0/1662793245 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a20101650 0x7f9a20101a70 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:49.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.057+0000 7f9a27000640 1 -- 172.21.15.43:0/1662793245 >> 172.21.15.43:0/1662793245 conn(0x7f9a200fcbc0 msgr2=0x7f9a200ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:49.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.057+0000 7f9a27000640 1 -- 172.21.15.43:0/1662793245 shutdown_connections 2024-03-28T22:33:49.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.057+0000 7f9a27000640 1 -- 172.21.15.43:0/1662793245 wait complete. 2024-03-28T22:33:49.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.058+0000 7f9a27000640 1 Processor -- start 2024-03-28T22:33:49.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.058+0000 7f9a27000640 1 -- start start 2024-03-28T22:33:49.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.058+0000 7f9a27000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a20101650 0x7f9a20199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:49.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.058+0000 7f9a27000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a20102840 0x7f9a2019a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:49.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.058+0000 7f9a27000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a20103200 0x7f9a201a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:49.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.058+0000 7f9a27000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9a2010d2c0 con 0x7f9a20102840 2024-03-28T22:33:49.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.058+0000 7f9a27000640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9a2010d140 con 0x7f9a20101650 2024-03-28T22:33:49.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.058+0000 7f9a27000640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9a2010d440 con 0x7f9a20103200 2024-03-28T22:33:49.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.059+0000 7f9a25c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a20101650 0x7f9a20199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:49.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.059+0000 7f9a25c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a20101650 0x7f9a20199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48678/0 (socket says 172.21.15.43:48678) 2024-03-28T22:33:49.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.059+0000 7f9a25c00640 1 -- 172.21.15.43:0/687820285 learned_addr learned my addr 172.21.15.43:0/687820285 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:49.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.059+0000 7f9a25200640 1 --2- 172.21.15.43:0/687820285 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a20102840 0x7f9a2019a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:49.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.059+0000 7f9a26600640 1 --2- 172.21.15.43:0/687820285 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a20103200 0x7f9a201a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:49.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.059+0000 7f9a25c00640 1 -- 172.21.15.43:0/687820285 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a20103200 msgr2=0x7f9a201a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:49.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.059+0000 7f9a25c00640 1 --2- 172.21.15.43:0/687820285 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a20103200 0x7f9a201a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:49.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.059+0000 7f9a25c00640 1 -- 172.21.15.43:0/687820285 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a20102840 msgr2=0x7f9a2019a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:49.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.059+0000 7f9a25c00640 1 --2- 172.21.15.43:0/687820285 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a20102840 0x7f9a2019a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:49.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.059+0000 7f9a25c00640 1 -- 172.21.15.43:0/687820285 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a201a1840 con 0x7f9a20101650 2024-03-28T22:33:49.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.059+0000 7f9a25c00640 1 --2- 172.21.15.43:0/687820285 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a20101650 0x7f9a20199c80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9a14000f80 tx=0x7f9a14035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:49.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.060+0000 7f9a1e000640 1 -- 172.21.15.43:0/687820285 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a14039040 con 0x7f9a20101650 2024-03-28T22:33:49.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.060+0000 7f9a1e000640 1 -- 172.21.15.43:0/687820285 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a14044070 con 0x7f9a20101650 2024-03-28T22:33:49.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.060+0000 7f9a27000640 1 -- 172.21.15.43:0/687820285 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a201a1ad0 con 0x7f9a20101650 2024-03-28T22:33:49.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.060+0000 7f9a1e000640 1 -- 172.21.15.43:0/687820285 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a14004330 con 0x7f9a20101650 2024-03-28T22:33:49.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.060+0000 7f9a27000640 1 -- 172.21.15.43:0/687820285 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a201a1f90 con 0x7f9a20101650 2024-03-28T22:33:49.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.061+0000 7f9a27000640 1 -- 172.21.15.43:0/687820285 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a201a21c0 con 0x7f9a20101650 2024-03-28T22:33:49.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.062+0000 7f9a1e000640 1 -- 172.21.15.43:0/687820285 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f9a14003970 con 0x7f9a20101650 2024-03-28T22:33:49.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.064+0000 7f9a1e000640 1 --2- 172.21.15.43:0/687820285 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9a080a1d80 0x7f9a080a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:49.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.064+0000 7f9a1e000640 1 -- 172.21.15.43:0/687820285 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f9a140f06d0 con 0x7f9a20101650 2024-03-28T22:33:49.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.064+0000 7f9a25200640 1 --2- 172.21.15.43:0/687820285 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9a080a1d80 0x7f9a080a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:49.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.065+0000 7f9a25200640 1 --2- 172.21.15.43:0/687820285 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9a080a1d80 0x7f9a080a4240 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c009bd0 tx=0x7f9a0c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:49.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.068+0000 7f9a1e000640 1 -- 172.21.15.43:0/687820285 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9a140bd860 con 0x7f9a20101650 2024-03-28T22:33:49.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:49 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3478221186' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:33:49.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:49 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3520693435' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:33:49.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.270+0000 7f9a27000640 1 -- 172.21.15.43:0/687820285 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f9a201a2730 con 0x7f9a20101650 2024-03-28T22:33:49.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.271+0000 7f9a1e000640 1 -- 172.21.15.43:0/687820285 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) ==== 103+0+11 (secure 0 0 0) 0x7f9a140bd160 con 0x7f9a20101650 2024-03-28T22:33:49.269 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-03-28T22:33:49.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.273+0000 7f9a27000640 1 -- 172.21.15.43:0/687820285 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9a080a1d80 msgr2=0x7f9a080a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:49.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.273+0000 7f9a27000640 1 --2- 172.21.15.43:0/687820285 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9a080a1d80 0x7f9a080a4240 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c009bd0 tx=0x7f9a0c005f90 comp rx=0 tx=0).stop 2024-03-28T22:33:49.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.273+0000 7f9a27000640 1 -- 172.21.15.43:0/687820285 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a20101650 msgr2=0x7f9a20199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:49.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.273+0000 7f9a27000640 1 --2- 172.21.15.43:0/687820285 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a20101650 0x7f9a20199c80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9a14000f80 tx=0x7f9a14035d10 comp rx=0 tx=0).stop 2024-03-28T22:33:49.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.273+0000 7f9a27000640 1 -- 172.21.15.43:0/687820285 shutdown_connections 2024-03-28T22:33:49.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.273+0000 7f9a27000640 1 --2- 172.21.15.43:0/687820285 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a20103200 0x7f9a201a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:49.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.273+0000 7f9a27000640 1 --2- 172.21.15.43:0/687820285 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a20102840 0x7f9a2019a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:49.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.273+0000 7f9a27000640 1 --2- 172.21.15.43:0/687820285 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9a080a1d80 0x7f9a080a4240 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:49.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.273+0000 7f9a27000640 1 --2- 172.21.15.43:0/687820285 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a20101650 0x7f9a20199c80 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:49.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.273+0000 7f9a27000640 1 -- 172.21.15.43:0/687820285 >> 172.21.15.43:0/687820285 conn(0x7f9a200fcbc0 msgr2=0x7f9a20073980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:49.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.274+0000 7f9a27000640 1 -- 172.21.15.43:0/687820285 shutdown_connections 2024-03-28T22:33:49.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.274+0000 7f9a27000640 1 -- 172.21.15.43:0/687820285 wait complete. 2024-03-28T22:33:49.292 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T22:33:49.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:49 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3478221186' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:33:49.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:49 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3520693435' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:33:49.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:49 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3478221186' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:33:49.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:49 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3520693435' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:33:49.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.522+0000 7f5424800640 1 -- 172.21.15.43:0/503260628 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f541c105540 msgr2=0x7f541c10bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:49.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.522+0000 7f5424800640 1 --2- 172.21.15.43:0/503260628 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f541c105540 0x7f541c10bdd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5418009be0 tx=0x7f541802bef0 comp rx=0 tx=0).stop 2024-03-28T22:33:49.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.522+0000 7f5424800640 1 -- 172.21.15.43:0/503260628 shutdown_connections 2024-03-28T22:33:49.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.522+0000 7f5424800640 1 --2- 172.21.15.43:0/503260628 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f541c105540 0x7f541c10bdd0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:49.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.522+0000 7f5424800640 1 --2- 172.21.15.43:0/503260628 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f541c104b80 0x7f541c105000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:49.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.522+0000 7f5424800640 1 --2- 172.21.15.43:0/503260628 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f541c103990 0x7f541c103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:49.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.522+0000 7f5424800640 1 -- 172.21.15.43:0/503260628 >> 172.21.15.43:0/503260628 conn(0x7f541c0fef00 msgr2=0x7f541c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:49.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.522+0000 7f5424800640 1 -- 172.21.15.43:0/503260628 shutdown_connections 2024-03-28T22:33:49.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.522+0000 7f5424800640 1 -- 172.21.15.43:0/503260628 wait complete. 2024-03-28T22:33:49.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.523+0000 7f5424800640 1 Processor -- start 2024-03-28T22:33:49.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.523+0000 7f5424800640 1 -- start start 2024-03-28T22:33:49.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.523+0000 7f5424800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f541c103990 0x7f541c199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:49.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.523+0000 7f5424800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f541c104b80 0x7f541c19a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:49.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.524+0000 7f5424800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f541c105540 0x7f541c1a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:49.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.524+0000 7f5424800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f541c10f640 con 0x7f541c105540 2024-03-28T22:33:49.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.524+0000 7f5424800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f541c10f4c0 con 0x7f541c103990 2024-03-28T22:33:49.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.524+0000 7f5424800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f541c10f7c0 con 0x7f541c104b80 2024-03-28T22:33:49.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.524+0000 7f5421e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f541c103990 0x7f541c199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:49.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.524+0000 7f5421e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f541c103990 0x7f541c199c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48696/0 (socket says 172.21.15.43:48696) 2024-03-28T22:33:49.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.524+0000 7f5421e00640 1 -- 172.21.15.43:0/2459181396 learned_addr learned my addr 172.21.15.43:0/2459181396 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:49.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.524+0000 7f5421400640 1 --2- 172.21.15.43:0/2459181396 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f541c104b80 0x7f541c19a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:49.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.524+0000 7f5422800640 1 --2- 172.21.15.43:0/2459181396 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f541c105540 0x7f541c1a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:49.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.524+0000 7f5421e00640 1 -- 172.21.15.43:0/2459181396 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f541c104b80 msgr2=0x7f541c19a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:49.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.524+0000 7f5421e00640 1 --2- 172.21.15.43:0/2459181396 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f541c104b80 0x7f541c19a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:49.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.524+0000 7f5421e00640 1 -- 172.21.15.43:0/2459181396 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f541c105540 msgr2=0x7f541c1a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:49.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.524+0000 7f5421e00640 1 --2- 172.21.15.43:0/2459181396 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f541c105540 0x7f541c1a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:49.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.524+0000 7f5421e00640 1 -- 172.21.15.43:0/2459181396 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f541c1a1850 con 0x7f541c103990 2024-03-28T22:33:49.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.524+0000 7f5421400640 1 --2- 172.21.15.43:0/2459181396 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f541c104b80 0x7f541c19a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:33:49.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.525+0000 7f5421e00640 1 --2- 172.21.15.43:0/2459181396 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f541c103990 0x7f541c199c40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f541000e9c0 tx=0x7f541000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:49.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.525+0000 7f5416a00640 1 -- 172.21.15.43:0/2459181396 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541000cd80 con 0x7f541c103990 2024-03-28T22:33:49.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.525+0000 7f5416a00640 1 -- 172.21.15.43:0/2459181396 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54100035f0 con 0x7f541c103990 2024-03-28T22:33:49.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.525+0000 7f5416a00640 1 -- 172.21.15.43:0/2459181396 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5410010690 con 0x7f541c103990 2024-03-28T22:33:49.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.525+0000 7f5424800640 1 -- 172.21.15.43:0/2459181396 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f541c1a1b40 con 0x7f541c103990 2024-03-28T22:33:49.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.525+0000 7f5424800640 1 -- 172.21.15.43:0/2459181396 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f541c1a20e0 con 0x7f541c103990 2024-03-28T22:33:49.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.526+0000 7f5424800640 1 -- 172.21.15.43:0/2459181396 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f541c10ac50 con 0x7f541c103990 2024-03-28T22:33:49.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.528+0000 7f5416a00640 1 -- 172.21.15.43:0/2459181396 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f54100037a0 con 0x7f541c103990 2024-03-28T22:33:49.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.529+0000 7f5416a00640 1 --2- 172.21.15.43:0/2459181396 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f54040a1e00 0x7f54040a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:49.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.530+0000 7f5416a00640 1 -- 172.21.15.43:0/2459181396 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f54100ce3e0 con 0x7f541c103990 2024-03-28T22:33:49.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.530+0000 7f5421400640 1 --2- 172.21.15.43:0/2459181396 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f54040a1e00 0x7f54040a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:49.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.530+0000 7f5421400640 1 --2- 172.21.15.43:0/2459181396 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f54040a1e00 0x7f54040a42c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f540c0096c0 tx=0x7f540c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:49.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.534+0000 7f5416a00640 1 -- 172.21.15.43:0/2459181396 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f541009b570 con 0x7f541c103990 2024-03-28T22:33:49.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.750+0000 7f5424800640 1 -- 172.21.15.43:0/2459181396 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f541c02d170 con 0x7f541c103990 2024-03-28T22:33:49.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.750+0000 7f5416a00640 1 -- 172.21.15.43:0/2459181396 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v92) ==== 81+0+591 (secure 0 0 0) 0x7f541009ae70 con 0x7f541c103990 2024-03-28T22:33:49.749 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/578791143 2024-03-29T22:25:28.008215+0000 2024-03-28T22:33:49.749 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/4029313215 2024-03-29T22:25:28.008215+0000 2024-03-28T22:33:49.749 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/1574178482 2024-03-29T22:25:28.008215+0000 2024-03-28T22:33:49.749 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6801/3517712438 2024-03-29T22:25:28.008215+0000 2024-03-28T22:33:49.749 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6800/1532177494 2024-03-29T22:24:52.981057+0000 2024-03-28T22:33:49.749 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/1375099638 2024-03-29T22:24:52.981057+0000 2024-03-28T22:33:49.749 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/2710117652 2024-03-29T22:24:52.981057+0000 2024-03-28T22:33:49.749 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6801/1532177494 2024-03-29T22:24:52.981057+0000 2024-03-28T22:33:49.749 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/3215634168 2024-03-29T22:24:52.981057+0000 2024-03-28T22:33:49.749 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6800/3517712438 2024-03-29T22:25:28.008215+0000 2024-03-28T22:33:49.750 INFO:teuthology.orchestra.run.smithi043.stderr:listed 10 entries 2024-03-28T22:33:49.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.753+0000 7f5424800640 1 -- 172.21.15.43:0/2459181396 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f54040a1e00 msgr2=0x7f54040a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:49.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.753+0000 7f5424800640 1 --2- 172.21.15.43:0/2459181396 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f54040a1e00 0x7f54040a42c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f540c0096c0 tx=0x7f540c009260 comp rx=0 tx=0).stop 2024-03-28T22:33:49.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.753+0000 7f5424800640 1 -- 172.21.15.43:0/2459181396 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f541c103990 msgr2=0x7f541c199c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:49.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.753+0000 7f5424800640 1 --2- 172.21.15.43:0/2459181396 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f541c103990 0x7f541c199c40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f541000e9c0 tx=0x7f541000ee80 comp rx=0 tx=0).stop 2024-03-28T22:33:49.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.753+0000 7f5424800640 1 -- 172.21.15.43:0/2459181396 shutdown_connections 2024-03-28T22:33:49.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.753+0000 7f5424800640 1 --2- 172.21.15.43:0/2459181396 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f541c105540 0x7f541c1a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:49.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.753+0000 7f5424800640 1 --2- 172.21.15.43:0/2459181396 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f541c104b80 0x7f541c19a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:49.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.753+0000 7f5424800640 1 --2- 172.21.15.43:0/2459181396 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f54040a1e00 0x7f54040a42c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:49.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.753+0000 7f5424800640 1 --2- 172.21.15.43:0/2459181396 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f541c103990 0x7f541c199c40 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:49.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.753+0000 7f5424800640 1 -- 172.21.15.43:0/2459181396 >> 172.21.15.43:0/2459181396 conn(0x7f541c0fef00 msgr2=0x7f541c107e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:49.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.754+0000 7f5424800640 1 -- 172.21.15.43:0/2459181396 shutdown_connections 2024-03-28T22:33:49.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:49.754+0000 7f5424800640 1 -- 172.21.15.43:0/2459181396 wait complete. 2024-03-28T22:33:49.771 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-28T22:33:49.771 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T22:33:49.824 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:33:50.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.061+0000 7fcd6dc00640 1 -- 172.21.15.43:0/888000730 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd68101770 msgr2=0x7fcd68101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:50.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.061+0000 7fcd6dc00640 1 --2- 172.21.15.43:0/888000730 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd68101770 0x7fcd68101b90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fcd58009900 tx=0x7fcd5802bc40 comp rx=0 tx=0).stop 2024-03-28T22:33:50.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.061+0000 7fcd6dc00640 1 -- 172.21.15.43:0/888000730 shutdown_connections 2024-03-28T22:33:50.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.061+0000 7fcd6dc00640 1 --2- 172.21.15.43:0/888000730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd68103320 0x7fcd68109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:50.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.061+0000 7fcd6dc00640 1 --2- 172.21.15.43:0/888000730 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd68102960 0x7fcd68102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:50.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.061+0000 7fcd6dc00640 1 --2- 172.21.15.43:0/888000730 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd68101770 0x7fcd68101b90 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:50.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.061+0000 7fcd6dc00640 1 -- 172.21.15.43:0/888000730 >> 172.21.15.43:0/888000730 conn(0x7fcd680fcd00 msgr2=0x7fcd680ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:50.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.061+0000 7fcd6dc00640 1 -- 172.21.15.43:0/888000730 shutdown_connections 2024-03-28T22:33:50.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.061+0000 7fcd6dc00640 1 -- 172.21.15.43:0/888000730 wait complete. 2024-03-28T22:33:50.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.062+0000 7fcd6dc00640 1 Processor -- start 2024-03-28T22:33:50.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.062+0000 7fcd6dc00640 1 -- start start 2024-03-28T22:33:50.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.063+0000 7fcd6dc00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd68101770 0x7fcd68110480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:50.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.063+0000 7fcd6dc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd68102960 0x7fcd681109c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:50.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.063+0000 7fcd6dc00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd68103320 0x7fcd6810cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:50.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.063+0000 7fcd6dc00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcd68074280 con 0x7fcd68102960 2024-03-28T22:33:50.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.063+0000 7fcd6dc00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fcd68074100 con 0x7fcd68103320 2024-03-28T22:33:50.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.063+0000 7fcd6dc00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fcd68074400 con 0x7fcd68101770 2024-03-28T22:33:50.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.063+0000 7fcd66000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd68102960 0x7fcd681109c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:50.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.063+0000 7fcd66a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd68101770 0x7fcd68110480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:50.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.063+0000 7fcd67400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd68103320 0x7fcd6810cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:50.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.063+0000 7fcd66a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd68101770 0x7fcd68110480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43116/0 (socket says 172.21.15.43:43116) 2024-03-28T22:33:50.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.063+0000 7fcd66a00640 1 -- 172.21.15.43:0/1885931999 learned_addr learned my addr 172.21.15.43:0/1885931999 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:50.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.064+0000 7fcd67400640 1 -- 172.21.15.43:0/1885931999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd68101770 msgr2=0x7fcd68110480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:50.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.064+0000 7fcd67400640 1 --2- 172.21.15.43:0/1885931999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd68101770 0x7fcd68110480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:50.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.064+0000 7fcd67400640 1 -- 172.21.15.43:0/1885931999 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd68102960 msgr2=0x7fcd681109c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:50.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.064+0000 7fcd67400640 1 --2- 172.21.15.43:0/1885931999 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd68102960 0x7fcd681109c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:50.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.064+0000 7fcd67400640 1 -- 172.21.15.43:0/1885931999 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd6810d8b0 con 0x7fcd68103320 2024-03-28T22:33:50.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.064+0000 7fcd66a00640 1 --2- 172.21.15.43:0/1885931999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd68101770 0x7fcd68110480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:33:50.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.064+0000 7fcd67400640 1 --2- 172.21.15.43:0/1885931999 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd68103320 0x7fcd6810cff0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcd6000da10 tx=0x7fcd6000ded0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:50.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.064+0000 7fcd5f400640 1 -- 172.21.15.43:0/1885931999 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd6000bd70 con 0x7fcd68103320 2024-03-28T22:33:50.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.064+0000 7fcd5f400640 1 -- 172.21.15.43:0/1885931999 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd60003e50 con 0x7fcd68103320 2024-03-28T22:33:50.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.064+0000 7fcd6dc00640 1 -- 172.21.15.43:0/1885931999 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd681a60f0 con 0x7fcd68103320 2024-03-28T22:33:50.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.064+0000 7fcd5f400640 1 -- 172.21.15.43:0/1885931999 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd600106a0 con 0x7fcd68103320 2024-03-28T22:33:50.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.064+0000 7fcd6dc00640 1 -- 172.21.15.43:0/1885931999 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd681a6500 con 0x7fcd68103320 2024-03-28T22:33:50.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.065+0000 7fcd6dc00640 1 -- 172.21.15.43:0/1885931999 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd681a67c0 con 0x7fcd68103320 2024-03-28T22:33:50.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.067+0000 7fcd5f400640 1 -- 172.21.15.43:0/1885931999 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fcd60003970 con 0x7fcd68103320 2024-03-28T22:33:50.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.068+0000 7fcd5f400640 1 --2- 172.21.15.43:0/1885931999 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fcd4c0a1e00 0x7fcd4c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:50.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.068+0000 7fcd5f400640 1 -- 172.21.15.43:0/1885931999 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7fcd600ce3a0 con 0x7fcd68103320 2024-03-28T22:33:50.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.068+0000 7fcd66a00640 1 --2- 172.21.15.43:0/1885931999 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fcd4c0a1e00 0x7fcd4c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:50.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.069+0000 7fcd66a00640 1 --2- 172.21.15.43:0/1885931999 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fcd4c0a1e00 0x7fcd4c0a42c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcd58005ab0 tx=0x7fcd5802cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:50.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.072+0000 7fcd5f400640 1 -- 172.21.15.43:0/1885931999 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fcd6009b530 con 0x7fcd68103320 2024-03-28T22:33:50.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.275+0000 7fcd6dc00640 1 -- 172.21.15.43:0/1885931999 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcd681a6a30 con 0x7fcd68103320 2024-03-28T22:33:50.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.277+0000 7fcd5f400640 1 -- 172.21.15.43:0/1885931999 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24093 (secure 0 0 0) 0x7fcd6009ae30 con 0x7fcd68103320 2024-03-28T22:33:50.276 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:33:50.276 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":92,"fsid":"c2276428-ed51-11ee-95d0-87774f69a715","created":"2024-03-28T22:24:19.950084+0000","modified":"2024-03-28T22:33:34.473170+0000","last_up_change":"2024-03-28T22:32:31.078737+0000","last_in_change":"2024-03-28T22:32:10.147097+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:28:14.713290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:33:27.167435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:33:28.411825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T22:33:32.523122+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b855adcd-786c-4e42-b8fe-a515a77ce979","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6803","nonce":2826124409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6805","nonce":2826124409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6809","nonce":2826124409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6807","nonce":2826124409}]},"public_addr":"172.21.15.43:6803/2826124409","cluster_addr":"172.21.15.43:6805/2826124409","heartbeat_back_addr":"172.21.15.43:6809/2826124409","heartbeat_front_addr":"172.21.15.43:6807/2826124409","state":["exists","up"]},{"osd":1,"uuid":"031c143d-8ad1-4b68-8db9-3c717137e3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6801","nonce":3043263644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6803","nonce":3043263644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6807","nonce":3043263644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6805","nonce":3043263644}]},"public_addr":"172.21.15.82:6801/3043263644","cluster_addr":"172.21.15.82:6803/3043263644","heartbeat_back_addr":"172.21.15.82:6807/3043263644","heartbeat_front_addr":"172.21.15.82:6805/3043263644","state":["exists","up"]},{"osd":2,"uuid":"20632bbb-f07a-4ca1-b0c4-ff9d508e211c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6801","nonce":4213953645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6803","nonce":4213953645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6807","nonce":4213953645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6805","nonce":4213953645}]},"public_addr":"172.21.15.172:6801/4213953645","cluster_addr":"172.21.15.172:6803/4213953645","heartbeat_back_addr":"172.21.15.172:6807/4213953645","heartbeat_front_addr":"172.21.15.172:6805/4213953645","state":["exists","up"]},{"osd":3,"uuid":"9a1f2312-4dbd-475b-8736-d5ebc8788e09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6811","nonce":1597280920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6813","nonce":1597280920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6817","nonce":1597280920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6815","nonce":1597280920}]},"public_addr":"172.21.15.43:6811/1597280920","cluster_addr":"172.21.15.43:6813/1597280920","heartbeat_back_addr":"172.21.15.43:6817/1597280920","heartbeat_front_addr":"172.21.15.43:6815/1597280920","state":["exists","up"]},{"osd":4,"uuid":"29140ead-7075-42b2-90c1-27cae0b29ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6809","nonce":215451895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6811","nonce":215451895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6815","nonce":215451895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6813","nonce":215451895}]},"public_addr":"172.21.15.82:6809/215451895","cluster_addr":"172.21.15.82:6811/215451895","heartbeat_back_addr":"172.21.15.82:6815/215451895","heartbeat_front_addr":"172.21.15.82:6813/215451895","state":["exists","up"]},{"osd":5,"uuid":"59644c61-93dc-4536-b2ad-0256b24e5cb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6809","nonce":1576148933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6811","nonce":1576148933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6815","nonce":1576148933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6813","nonce":1576148933}]},"public_addr":"172.21.15.172:6809/1576148933","cluster_addr":"172.21.15.172:6811/1576148933","heartbeat_back_addr":"172.21.15.172:6815/1576148933","heartbeat_front_addr":"172.21.15.172:6813/1576148933","state":["exists","up"]},{"osd":6,"uuid":"22d54e57-c3f1-4ecf-9921-8e815b48f92a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6819","nonce":791044279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6821","nonce":791044279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6825","nonce":791044279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6823","nonce":791044279}]},"public_addr":"172.21.15.43:6819/791044279","cluster_addr":"172.21.15.43:6821/791044279","heartbeat_back_addr":"172.21.15.43:6825/791044279","heartbeat_front_addr":"172.21.15.43:6823/791044279","state":["exists","up"]},{"osd":7,"uuid":"248cd1e2-0d6b-452f-a0a8-c51fe1b975ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6817","nonce":2157777323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6819","nonce":2157777323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6823","nonce":2157777323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6821","nonce":2157777323}]},"public_addr":"172.21.15.82:6817/2157777323","cluster_addr":"172.21.15.82:6819/2157777323","heartbeat_back_addr":"172.21.15.82:6823/2157777323","heartbeat_front_addr":"172.21.15.82:6821/2157777323","state":["exists","up"]},{"osd":8,"uuid":"d98aab6e-8bae-4c24-906c-4f492f047b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6817","nonce":1095328248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6819","nonce":1095328248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6823","nonce":1095328248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6821","nonce":1095328248}]},"public_addr":"172.21.15.172:6817/1095328248","cluster_addr":"172.21.15.172:6819/1095328248","heartbeat_back_addr":"172.21.15.172:6823/1095328248","heartbeat_front_addr":"172.21.15.172:6821/1095328248","state":["exists","up"]},{"osd":9,"uuid":"19aa383e-a0f6-45ea-96cc-e7956535caa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6827","nonce":3657791498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6829","nonce":3657791498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6833","nonce":3657791498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6831","nonce":3657791498}]},"public_addr":"172.21.15.43:6827/3657791498","cluster_addr":"172.21.15.43:6829/3657791498","heartbeat_back_addr":"172.21.15.43:6833/3657791498","heartbeat_front_addr":"172.21.15.43:6831/3657791498","state":["exists","up"]},{"osd":10,"uuid":"666977e5-eaf7-41b9-a18f-c380e149345d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6825","nonce":3454954047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6827","nonce":3454954047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6831","nonce":3454954047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6829","nonce":3454954047}]},"public_addr":"172.21.15.82:6825/3454954047","cluster_addr":"172.21.15.82:6827/3454954047","heartbeat_back_addr":"172.21.15.82:6831/3454954047","heartbeat_front_addr":"172.21.15.82:6829/3454954047","state":["exists","up"]},{"osd":11,"uuid":"615cf3be-8076-470e-ad09-c880e7593717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6825","nonce":2563517144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6827","nonce":2563517144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6831","nonce":2563517144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6829","nonce":2563517144}]},"public_addr":"172.21.15.172:6825/2563517144","cluster_addr":"172.21.15.172:6827/2563517144","heartbeat_back_addr":"172.21.15.172:6831/2563517144","heartbeat_front_addr":"172.21.15.172:6829/2563517144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:21.727405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:47.100259+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:10.615438+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:43.632144+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:09.487940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:34.604463+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:09.594452+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:35.709795+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:01.184202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:35.455322+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:02.413166+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:28.550754+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/578791143":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/4029313215":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/1574178482":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6801/3517712438":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6800/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/1375099638":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/2710117652":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6801/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/3215634168":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6800/3517712438":"2024-03-29T22:25:28.008215+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:33:50.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.280+0000 7fcd6dc00640 1 -- 172.21.15.43:0/1885931999 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fcd4c0a1e00 msgr2=0x7fcd4c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:50.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.280+0000 7fcd6dc00640 1 --2- 172.21.15.43:0/1885931999 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fcd4c0a1e00 0x7fcd4c0a42c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcd58005ab0 tx=0x7fcd5802cf90 comp rx=0 tx=0).stop 2024-03-28T22:33:50.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.280+0000 7fcd6dc00640 1 -- 172.21.15.43:0/1885931999 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd68103320 msgr2=0x7fcd6810cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:50.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.280+0000 7fcd6dc00640 1 --2- 172.21.15.43:0/1885931999 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd68103320 0x7fcd6810cff0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcd6000da10 tx=0x7fcd6000ded0 comp rx=0 tx=0).stop 2024-03-28T22:33:50.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.281+0000 7fcd6dc00640 1 -- 172.21.15.43:0/1885931999 shutdown_connections 2024-03-28T22:33:50.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.281+0000 7fcd6dc00640 1 --2- 172.21.15.43:0/1885931999 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fcd4c0a1e00 0x7fcd4c0a42c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:50.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.281+0000 7fcd6dc00640 1 --2- 172.21.15.43:0/1885931999 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd68103320 0x7fcd6810cff0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:50.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.281+0000 7fcd6dc00640 1 --2- 172.21.15.43:0/1885931999 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd68102960 0x7fcd681109c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:50.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.281+0000 7fcd6dc00640 1 --2- 172.21.15.43:0/1885931999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd68101770 0x7fcd68110480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:50.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.281+0000 7fcd6dc00640 1 -- 172.21.15.43:0/1885931999 >> 172.21.15.43:0/1885931999 conn(0x7fcd680fcd00 msgr2=0x7fcd680fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:50.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.281+0000 7fcd6dc00640 1 -- 172.21.15.43:0/1885931999 shutdown_connections 2024-03-28T22:33:50.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.281+0000 7fcd6dc00640 1 -- 172.21.15.43:0/1885931999 wait complete. 2024-03-28T22:33:50.298 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T22:28:14.713290+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T22:33:27.167435+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T22:33:28.411825+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T22:33:32.523122+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T22:33:50.299 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T22:33:50.349 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:50 smithi082 ceph-mon[36471]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:50.350 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:50 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/687820285' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:33:50.350 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:50 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2459181396' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T22:33:50.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:50 smithi043 ceph-mon[31823]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:50.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:50 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/687820285' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:33:50.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:50 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2459181396' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T22:33:50.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:50 smithi172 ceph-mon[36566]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:33:50.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:50 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/687820285' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:33:50.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:50 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2459181396' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T22:33:50.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.527+0000 7fe4f3200640 1 -- 172.21.15.43:0/4028727394 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe4ec100930 msgr2=0x7fe4ec100d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:50.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.527+0000 7fe4f3200640 1 --2- 172.21.15.43:0/4028727394 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe4ec100930 0x7fe4ec100d90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4009900 tx=0x7fe4d402bc30 comp rx=0 tx=0).stop 2024-03-28T22:33:50.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.527+0000 7fe4f3200640 1 -- 172.21.15.43:0/4028727394 shutdown_connections 2024-03-28T22:33:50.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.527+0000 7fe4f3200640 1 --2- 172.21.15.43:0/4028727394 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe4ec100930 0x7fe4ec100d90 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:50.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.527+0000 7fe4f3200640 1 --2- 172.21.15.43:0/4028727394 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4ec1012d0 0x7fe4ec101750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:50.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.527+0000 7fe4f3200640 1 --2- 172.21.15.43:0/4028727394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4ec0fffd0 0x7fe4ec1003f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:50.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.527+0000 7fe4f3200640 1 -- 172.21.15.43:0/4028727394 >> 172.21.15.43:0/4028727394 conn(0x7fe4ec0fbbf0 msgr2=0x7fe4ec0fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:50.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.527+0000 7fe4f3200640 1 -- 172.21.15.43:0/4028727394 shutdown_connections 2024-03-28T22:33:50.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.527+0000 7fe4f3200640 1 -- 172.21.15.43:0/4028727394 wait complete. 2024-03-28T22:33:50.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.529+0000 7fe4f3200640 1 Processor -- start 2024-03-28T22:33:50.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.529+0000 7fe4f3200640 1 -- start start 2024-03-28T22:33:50.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.529+0000 7fe4f3200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4ec0fffd0 0x7fe4ec199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:50.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.529+0000 7fe4f3200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe4ec100930 0x7fe4ec19a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:50.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.530+0000 7fe4f3200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4ec1012d0 0x7fe4ec1a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:50.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.530+0000 7fe4f3200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe4ec10f5f0 con 0x7fe4ec0fffd0 2024-03-28T22:33:50.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.530+0000 7fe4f3200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe4ec10f470 con 0x7fe4ec100930 2024-03-28T22:33:50.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.530+0000 7fe4f3200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe4ec10f770 con 0x7fe4ec1012d0 2024-03-28T22:33:50.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.530+0000 7fe4ebe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4ec0fffd0 0x7fe4ec199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:50.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.530+0000 7fe4e3400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe4ec100930 0x7fe4ec19a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:50.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.530+0000 7fe4ebe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4ec0fffd0 0x7fe4ec199c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59700/0 (socket says 172.21.15.43:59700) 2024-03-28T22:33:50.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.530+0000 7fe4ebe00640 1 -- 172.21.15.43:0/1195944997 learned_addr learned my addr 172.21.15.43:0/1195944997 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:50.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.530+0000 7fe4f1200640 1 --2- 172.21.15.43:0/1195944997 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4ec1012d0 0x7fe4ec1a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:50.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.530+0000 7fe4e3400640 1 -- 172.21.15.43:0/1195944997 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4ec1012d0 msgr2=0x7fe4ec1a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:50.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.530+0000 7fe4e3400640 1 --2- 172.21.15.43:0/1195944997 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4ec1012d0 0x7fe4ec1a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:50.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.530+0000 7fe4e3400640 1 -- 172.21.15.43:0/1195944997 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4ec0fffd0 msgr2=0x7fe4ec199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:50.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.530+0000 7fe4e3400640 1 --2- 172.21.15.43:0/1195944997 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4ec0fffd0 0x7fe4ec199c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:50.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.530+0000 7fe4e3400640 1 -- 172.21.15.43:0/1195944997 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4ec1a18d0 con 0x7fe4ec100930 2024-03-28T22:33:50.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.531+0000 7fe4e3400640 1 --2- 172.21.15.43:0/1195944997 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe4ec100930 0x7fe4ec19a1d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4036040 tx=0x7fe4d402c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:50.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.531+0000 7fe4e9600640 1 -- 172.21.15.43:0/1195944997 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4043020 con 0x7fe4ec100930 2024-03-28T22:33:50.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.531+0000 7fe4e9600640 1 -- 172.21.15.43:0/1195944997 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4d40054e0 con 0x7fe4ec100930 2024-03-28T22:33:50.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.531+0000 7fe4e9600640 1 -- 172.21.15.43:0/1195944997 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4003ca0 con 0x7fe4ec100930 2024-03-28T22:33:50.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.531+0000 7fe4f3200640 1 -- 172.21.15.43:0/1195944997 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4ec1a1b60 con 0x7fe4ec100930 2024-03-28T22:33:50.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.531+0000 7fe4f3200640 1 -- 172.21.15.43:0/1195944997 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4ec1a20d0 con 0x7fe4ec100930 2024-03-28T22:33:50.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.532+0000 7fe4f3200640 1 -- 172.21.15.43:0/1195944997 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4ec10af10 con 0x7fe4ec100930 2024-03-28T22:33:50.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.534+0000 7fe4e9600640 1 -- 172.21.15.43:0/1195944997 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fe4d4046020 con 0x7fe4ec100930 2024-03-28T22:33:50.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.536+0000 7fe4e9600640 1 --2- 172.21.15.43:0/1195944997 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe4d00a1e00 0x7fe4d00a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:50.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.536+0000 7fe4e9600640 1 -- 172.21.15.43:0/1195944997 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7fe4d40efff0 con 0x7fe4ec100930 2024-03-28T22:33:50.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.536+0000 7fe4ebe00640 1 --2- 172.21.15.43:0/1195944997 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe4d00a1e00 0x7fe4d00a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:50.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.537+0000 7fe4ebe00640 1 --2- 172.21.15.43:0/1195944997 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe4d00a1e00 0x7fe4d00a42c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec19ab60 tx=0x7fe4dc009720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:50.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.540+0000 7fe4e9600640 1 -- 172.21.15.43:0/1195944997 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe4d404d020 con 0x7fe4ec100930 2024-03-28T22:33:50.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.744+0000 7fe4f3200640 1 -- 172.21.15.43:0/1195944997 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe4ec062350 con 0x7fe4ec100930 2024-03-28T22:33:50.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.745+0000 7fe4e9600640 1 -- 172.21.15.43:0/1195944997 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7fe4d40bd2c0 con 0x7fe4ec100930 2024-03-28T22:33:50.743 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-03-28T22:33:50.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.748+0000 7fe4f3200640 1 -- 172.21.15.43:0/1195944997 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe4d00a1e00 msgr2=0x7fe4d00a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:50.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.748+0000 7fe4f3200640 1 --2- 172.21.15.43:0/1195944997 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe4d00a1e00 0x7fe4d00a42c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec19ab60 tx=0x7fe4dc009720 comp rx=0 tx=0).stop 2024-03-28T22:33:50.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.748+0000 7fe4f3200640 1 -- 172.21.15.43:0/1195944997 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe4ec100930 msgr2=0x7fe4ec19a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:50.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.748+0000 7fe4f3200640 1 --2- 172.21.15.43:0/1195944997 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe4ec100930 0x7fe4ec19a1d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4036040 tx=0x7fe4d402c6c0 comp rx=0 tx=0).stop 2024-03-28T22:33:50.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.748+0000 7fe4f3200640 1 -- 172.21.15.43:0/1195944997 shutdown_connections 2024-03-28T22:33:50.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.748+0000 7fe4f3200640 1 --2- 172.21.15.43:0/1195944997 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe4d00a1e00 0x7fe4d00a42c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:50.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.748+0000 7fe4f3200640 1 --2- 172.21.15.43:0/1195944997 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe4ec100930 0x7fe4ec19a1d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:50.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.748+0000 7fe4f3200640 1 --2- 172.21.15.43:0/1195944997 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4ec1012d0 0x7fe4ec1a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:50.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.748+0000 7fe4f3200640 1 --2- 172.21.15.43:0/1195944997 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4ec0fffd0 0x7fe4ec199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:50.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.748+0000 7fe4f3200640 1 -- 172.21.15.43:0/1195944997 >> 172.21.15.43:0/1195944997 conn(0x7fe4ec0fbbf0 msgr2=0x7fe4ec0fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:50.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.748+0000 7fe4f3200640 1 -- 172.21.15.43:0/1195944997 shutdown_connections 2024-03-28T22:33:50.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:50.748+0000 7fe4f3200640 1 -- 172.21.15.43:0/1195944997 wait complete. 2024-03-28T22:33:50.767 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T22:33:51.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.002+0000 7f8b78e00640 1 -- 172.21.15.43:0/1945533307 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8b74108e60 msgr2=0x7f8b741092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:51.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.002+0000 7f8b78e00640 1 --2- 172.21.15.43:0/1945533307 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8b74108e60 0x7f8b741092e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8b60009900 tx=0x7f8b6002bc30 comp rx=0 tx=0).stop 2024-03-28T22:33:51.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.003+0000 7f8b78e00640 1 -- 172.21.15.43:0/1945533307 shutdown_connections 2024-03-28T22:33:51.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.003+0000 7f8b78e00640 1 --2- 172.21.15.43:0/1945533307 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8b74108e60 0x7f8b741092e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:51.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.003+0000 7f8b78e00640 1 --2- 172.21.15.43:0/1945533307 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b74109820 0x7f8b741100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:51.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.003+0000 7f8b78e00640 1 --2- 172.21.15.43:0/1945533307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b74107c70 0x7f8b74108090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:51.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.003+0000 7f8b78e00640 1 -- 172.21.15.43:0/1945533307 >> 172.21.15.43:0/1945533307 conn(0x7f8b74076c20 msgr2=0x7f8b74079040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:51.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.003+0000 7f8b78e00640 1 -- 172.21.15.43:0/1945533307 shutdown_connections 2024-03-28T22:33:51.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.003+0000 7f8b78e00640 1 -- 172.21.15.43:0/1945533307 wait complete. 2024-03-28T22:33:51.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.004+0000 7f8b78e00640 1 Processor -- start 2024-03-28T22:33:51.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.004+0000 7f8b78e00640 1 -- start start 2024-03-28T22:33:51.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.004+0000 7f8b78e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b74107c70 0x7f8b7419df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:51.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.004+0000 7f8b78e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b74108e60 0x7f8b7419e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:51.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.005+0000 7f8b78e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8b74109820 0x7f8b741a5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:51.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.005+0000 7f8b78e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8b74113940 con 0x7f8b74107c70 2024-03-28T22:33:51.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.005+0000 7f8b78e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8b741137c0 con 0x7f8b74109820 2024-03-28T22:33:51.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.005+0000 7f8b78e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f8b74113ac0 con 0x7f8b74108e60 2024-03-28T22:33:51.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.005+0000 7f8b73400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b74107c70 0x7f8b7419df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:51.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.005+0000 7f8b73400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b74107c70 0x7f8b7419df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59712/0 (socket says 172.21.15.43:59712) 2024-03-28T22:33:51.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.005+0000 7f8b73400640 1 -- 172.21.15.43:0/2120997106 learned_addr learned my addr 172.21.15.43:0/2120997106 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:51.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.005+0000 7f8b73e00640 1 --2- 172.21.15.43:0/2120997106 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8b74109820 0x7f8b741a5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:51.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.005+0000 7f8b72a00640 1 --2- 172.21.15.43:0/2120997106 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b74108e60 0x7f8b7419e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:51.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.005+0000 7f8b73400640 1 -- 172.21.15.43:0/2120997106 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b74108e60 msgr2=0x7f8b7419e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:51.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.005+0000 7f8b73400640 1 --2- 172.21.15.43:0/2120997106 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b74108e60 0x7f8b7419e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:51.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.005+0000 7f8b73400640 1 -- 172.21.15.43:0/2120997106 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8b74109820 msgr2=0x7f8b741a5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:51.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.005+0000 7f8b73400640 1 --2- 172.21.15.43:0/2120997106 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8b74109820 0x7f8b741a5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:51.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.005+0000 7f8b73400640 1 -- 172.21.15.43:0/2120997106 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b741a5b80 con 0x7f8b74107c70 2024-03-28T22:33:51.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.006+0000 7f8b73400640 1 --2- 172.21.15.43:0/2120997106 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b74107c70 0x7f8b7419df60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8b6400e9c0 tx=0x7f8b6400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:51.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.006+0000 7f8b6be00640 1 -- 172.21.15.43:0/2120997106 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b6400cd80 con 0x7f8b74107c70 2024-03-28T22:33:51.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.006+0000 7f8b6be00640 1 -- 172.21.15.43:0/2120997106 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b640035f0 con 0x7f8b74107c70 2024-03-28T22:33:51.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.006+0000 7f8b6be00640 1 -- 172.21.15.43:0/2120997106 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b64010690 con 0x7f8b74107c70 2024-03-28T22:33:51.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.006+0000 7f8b78e00640 1 -- 172.21.15.43:0/2120997106 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b741a5e70 con 0x7f8b74107c70 2024-03-28T22:33:51.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.006+0000 7f8b78e00640 1 -- 172.21.15.43:0/2120997106 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b741a62e0 con 0x7f8b74107c70 2024-03-28T22:33:51.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.007+0000 7f8b78e00640 1 -- 172.21.15.43:0/2120997106 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b7410ef90 con 0x7f8b74107c70 2024-03-28T22:33:51.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.009+0000 7f8b6be00640 1 -- 172.21.15.43:0/2120997106 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f8b640037a0 con 0x7f8b74107c70 2024-03-28T22:33:51.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.012+0000 7f8b6be00640 1 --2- 172.21.15.43:0/2120997106 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f8b580a1e00 0x7f8b580a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:51.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.012+0000 7f8b6be00640 1 -- 172.21.15.43:0/2120997106 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f8b640ce440 con 0x7f8b74107c70 2024-03-28T22:33:51.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.012+0000 7f8b72a00640 1 --2- 172.21.15.43:0/2120997106 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f8b580a1e00 0x7f8b580a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:51.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.013+0000 7f8b72a00640 1 --2- 172.21.15.43:0/2120997106 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f8b580a1e00 0x7f8b580a42c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8b6002c240 tx=0x7f8b6002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:51.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.015+0000 7f8b6be00640 1 -- 172.21.15.43:0/2120997106 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8b6409b660 con 0x7f8b74107c70 2024-03-28T22:33:51.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.222+0000 7f8b78e00640 1 -- 172.21.15.43:0/2120997106 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8b7402d3d0 con 0x7f8b74107c70 2024-03-28T22:33:51.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.223+0000 7f8b6be00640 1 -- 172.21.15.43:0/2120997106 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7f8b6409af60 con 0x7f8b74107c70 2024-03-28T22:33:51.221 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-03-28T22:33:51.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.226+0000 7f8b78e00640 1 -- 172.21.15.43:0/2120997106 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f8b580a1e00 msgr2=0x7f8b580a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:51.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.226+0000 7f8b78e00640 1 --2- 172.21.15.43:0/2120997106 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f8b580a1e00 0x7f8b580a42c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8b6002c240 tx=0x7f8b6002df90 comp rx=0 tx=0).stop 2024-03-28T22:33:51.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.226+0000 7f8b78e00640 1 -- 172.21.15.43:0/2120997106 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b74107c70 msgr2=0x7f8b7419df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:51.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.226+0000 7f8b78e00640 1 --2- 172.21.15.43:0/2120997106 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b74107c70 0x7f8b7419df60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8b6400e9c0 tx=0x7f8b6400ee80 comp rx=0 tx=0).stop 2024-03-28T22:33:51.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.226+0000 7f8b78e00640 1 -- 172.21.15.43:0/2120997106 shutdown_connections 2024-03-28T22:33:51.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.226+0000 7f8b78e00640 1 --2- 172.21.15.43:0/2120997106 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f8b580a1e00 0x7f8b580a42c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:51.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.226+0000 7f8b78e00640 1 --2- 172.21.15.43:0/2120997106 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8b74109820 0x7f8b741a5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:51.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.226+0000 7f8b78e00640 1 --2- 172.21.15.43:0/2120997106 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b74108e60 0x7f8b7419e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:51.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.226+0000 7f8b78e00640 1 --2- 172.21.15.43:0/2120997106 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b74107c70 0x7f8b7419df60 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:51.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.226+0000 7f8b78e00640 1 -- 172.21.15.43:0/2120997106 >> 172.21.15.43:0/2120997106 conn(0x7f8b74076c20 msgr2=0x7f8b7410c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:51.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.226+0000 7f8b78e00640 1 -- 172.21.15.43:0/2120997106 shutdown_connections 2024-03-28T22:33:51.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.226+0000 7f8b78e00640 1 -- 172.21.15.43:0/2120997106 wait complete. 2024-03-28T22:33:51.242 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T22:33:51.323 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:51 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1885931999' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:33:51.323 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:51 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1195944997' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:33:51.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:51 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1885931999' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:33:51.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:51 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1195944997' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:33:51.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:51 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1885931999' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:33:51.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:51 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1195944997' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:33:51.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.474+0000 7f5bb2c00640 1 -- 172.21.15.43:0/3385888755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bac077cf0 msgr2=0x7f5bac078590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:51.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.474+0000 7f5bb2c00640 1 --2- 172.21.15.43:0/3385888755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bac077cf0 0x7f5bac078590 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5ba4009ca0 tx=0x7f5ba402c0f0 comp rx=0 tx=0).stop 2024-03-28T22:33:51.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.474+0000 7f5bb2c00640 1 -- 172.21.15.43:0/3385888755 shutdown_connections 2024-03-28T22:33:51.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.474+0000 7f5bb2c00640 1 --2- 172.21.15.43:0/3385888755 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5bac077350 0x7f5bac0777b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:51.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.474+0000 7f5bb2c00640 1 --2- 172.21.15.43:0/3385888755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bac077cf0 0x7f5bac078590 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:51.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.474+0000 7f5bb2c00640 1 --2- 172.21.15.43:0/3385888755 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5bac076110 0x7f5bac076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:51.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.474+0000 7f5bb2c00640 1 -- 172.21.15.43:0/3385888755 >> 172.21.15.43:0/3385888755 conn(0x7f5bac0fedc0 msgr2=0x7f5bac101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:51.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.474+0000 7f5bb2c00640 1 -- 172.21.15.43:0/3385888755 shutdown_connections 2024-03-28T22:33:51.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.474+0000 7f5bb2c00640 1 -- 172.21.15.43:0/3385888755 wait complete. 2024-03-28T22:33:51.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.475+0000 7f5bb2c00640 1 Processor -- start 2024-03-28T22:33:51.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.475+0000 7f5bb2c00640 1 -- start start 2024-03-28T22:33:51.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.475+0000 7f5bb2c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bac076110 0x7f5bac19df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:51.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.475+0000 7f5bb2c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5bac077350 0x7f5bac19e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:51.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.475+0000 7f5bb2c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5bac077cf0 0x7f5bac1a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:51.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.475+0000 7f5bb2c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac1138e0 con 0x7f5bac076110 2024-03-28T22:33:51.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.475+0000 7f5bb2c00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac113760 con 0x7f5bac077350 2024-03-28T22:33:51.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.475+0000 7f5bb2c00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac113a60 con 0x7f5bac077cf0 2024-03-28T22:33:51.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.476+0000 7f5bb1800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bac076110 0x7f5bac19df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:51.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.476+0000 7f5bb1800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bac076110 0x7f5bac19df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59738/0 (socket says 172.21.15.43:59738) 2024-03-28T22:33:51.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.476+0000 7f5bb1800640 1 -- 172.21.15.43:0/1226322811 learned_addr learned my addr 172.21.15.43:0/1226322811 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:51.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.476+0000 7f5bb0e00640 1 --2- 172.21.15.43:0/1226322811 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5bac077350 0x7f5bac19e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:51.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.476+0000 7f5bb2200640 1 --2- 172.21.15.43:0/1226322811 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5bac077cf0 0x7f5bac1a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:51.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.476+0000 7f5bb1800640 1 -- 172.21.15.43:0/1226322811 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5bac077cf0 msgr2=0x7f5bac1a54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:51.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.476+0000 7f5bb1800640 1 --2- 172.21.15.43:0/1226322811 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5bac077cf0 0x7f5bac1a54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:51.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.476+0000 7f5bb1800640 1 -- 172.21.15.43:0/1226322811 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5bac077350 msgr2=0x7f5bac19e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:51.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.476+0000 7f5bb1800640 1 --2- 172.21.15.43:0/1226322811 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5bac077350 0x7f5bac19e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:51.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.476+0000 7f5bb1800640 1 -- 172.21.15.43:0/1226322811 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bac1a5b60 con 0x7f5bac076110 2024-03-28T22:33:51.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.476+0000 7f5bb2200640 1 --2- 172.21.15.43:0/1226322811 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5bac077cf0 0x7f5bac1a54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:33:51.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.477+0000 7f5bb1800640 1 --2- 172.21.15.43:0/1226322811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bac076110 0x7f5bac19df40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5ba000e9f0 tx=0x7f5ba000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:51.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.477+0000 7f5baa000640 1 -- 172.21.15.43:0/1226322811 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba000cd80 con 0x7f5bac076110 2024-03-28T22:33:51.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.477+0000 7f5baa000640 1 -- 172.21.15.43:0/1226322811 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ba0005500 con 0x7f5bac076110 2024-03-28T22:33:51.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.477+0000 7f5baa000640 1 -- 172.21.15.43:0/1226322811 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba0010690 con 0x7f5bac076110 2024-03-28T22:33:51.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.477+0000 7f5bb2c00640 1 -- 172.21.15.43:0/1226322811 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bac1a5e50 con 0x7f5bac076110 2024-03-28T22:33:51.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.477+0000 7f5bb2c00640 1 -- 172.21.15.43:0/1226322811 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bac1a6260 con 0x7f5bac076110 2024-03-28T22:33:51.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.478+0000 7f5bb2c00640 1 -- 172.21.15.43:0/1226322811 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bac0688e0 con 0x7f5bac076110 2024-03-28T22:33:51.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.480+0000 7f5baa000640 1 -- 172.21.15.43:0/1226322811 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f5ba00046c0 con 0x7f5bac076110 2024-03-28T22:33:51.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.482+0000 7f5baa000640 1 --2- 172.21.15.43:0/1226322811 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5b940a1d80 0x7f5b940a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:51.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.482+0000 7f5baa000640 1 -- 172.21.15.43:0/1226322811 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f5ba00ce4f0 con 0x7f5bac076110 2024-03-28T22:33:51.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.482+0000 7f5bb0e00640 1 --2- 172.21.15.43:0/1226322811 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5b940a1d80 0x7f5b940a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:51.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.483+0000 7f5bb0e00640 1 --2- 172.21.15.43:0/1226322811 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5b940a1d80 0x7f5b940a4240 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5b980095f0 tx=0x7f5b98005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:51.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.486+0000 7f5baa000640 1 -- 172.21.15.43:0/1226322811 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5bac0688e0 con 0x7f5bac076110 2024-03-28T22:33:51.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.693+0000 7f5bb2c00640 1 -- 172.21.15.43:0/1226322811 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5bac062350 con 0x7f5bac076110 2024-03-28T22:33:51.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.694+0000 7f5baa000640 1 -- 172.21.15.43:0/1226322811 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7f5ba009b680 con 0x7f5bac076110 2024-03-28T22:33:51.692 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-03-28T22:33:51.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.697+0000 7f5bb2c00640 1 -- 172.21.15.43:0/1226322811 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5b940a1d80 msgr2=0x7f5b940a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:51.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.697+0000 7f5bb2c00640 1 --2- 172.21.15.43:0/1226322811 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5b940a1d80 0x7f5b940a4240 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5b980095f0 tx=0x7f5b98005f90 comp rx=0 tx=0).stop 2024-03-28T22:33:51.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.697+0000 7f5bb2c00640 1 -- 172.21.15.43:0/1226322811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bac076110 msgr2=0x7f5bac19df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:51.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.697+0000 7f5bb2c00640 1 --2- 172.21.15.43:0/1226322811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bac076110 0x7f5bac19df40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5ba000e9f0 tx=0x7f5ba000edf0 comp rx=0 tx=0).stop 2024-03-28T22:33:51.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.697+0000 7f5bb2c00640 1 -- 172.21.15.43:0/1226322811 shutdown_connections 2024-03-28T22:33:51.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.697+0000 7f5bb2c00640 1 --2- 172.21.15.43:0/1226322811 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5b940a1d80 0x7f5b940a4240 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:51.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.697+0000 7f5bb2c00640 1 --2- 172.21.15.43:0/1226322811 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5bac077350 0x7f5bac19e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:51.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.697+0000 7f5bb2c00640 1 --2- 172.21.15.43:0/1226322811 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5bac077cf0 0x7f5bac1a54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:51.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.697+0000 7f5bb2c00640 1 --2- 172.21.15.43:0/1226322811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5bac076110 0x7f5bac19df40 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:51.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.697+0000 7f5bb2c00640 1 -- 172.21.15.43:0/1226322811 >> 172.21.15.43:0/1226322811 conn(0x7f5bac0fedc0 msgr2=0x7f5bac1008c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:51.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.697+0000 7f5bb2c00640 1 -- 172.21.15.43:0/1226322811 shutdown_connections 2024-03-28T22:33:51.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.697+0000 7f5bb2c00640 1 -- 172.21.15.43:0/1226322811 wait complete. 2024-03-28T22:33:51.716 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T22:33:51.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.940+0000 7f78abe00640 1 -- 172.21.15.43:0/723096008 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78ac101660 msgr2=0x7f78ac101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:51.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.940+0000 7f78abe00640 1 --2- 172.21.15.43:0/723096008 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78ac101660 0x7f78ac101a80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f789c009900 tx=0x7f789c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:33:51.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.940+0000 7f78abe00640 1 -- 172.21.15.43:0/723096008 shutdown_connections 2024-03-28T22:33:51.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.940+0000 7f78abe00640 1 --2- 172.21.15.43:0/723096008 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78ac103210 0x7f78ac109aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:51.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.940+0000 7f78abe00640 1 --2- 172.21.15.43:0/723096008 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78ac102850 0x7f78ac102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:51.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.940+0000 7f78abe00640 1 --2- 172.21.15.43:0/723096008 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78ac101660 0x7f78ac101a80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:51.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.940+0000 7f78abe00640 1 -- 172.21.15.43:0/723096008 >> 172.21.15.43:0/723096008 conn(0x7f78ac0fcbf0 msgr2=0x7f78ac0ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:51.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.940+0000 7f78abe00640 1 -- 172.21.15.43:0/723096008 shutdown_connections 2024-03-28T22:33:51.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.940+0000 7f78abe00640 1 -- 172.21.15.43:0/723096008 wait complete. 2024-03-28T22:33:51.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.941+0000 7f78abe00640 1 Processor -- start 2024-03-28T22:33:51.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.941+0000 7f78abe00640 1 -- start start 2024-03-28T22:33:51.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.942+0000 7f78abe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78ac101660 0x7f78ac199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:51.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.942+0000 7f78abe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78ac102850 0x7f78ac19a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:51.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.942+0000 7f78abe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78ac103210 0x7f78ac1a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:51.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.942+0000 7f78abe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac10d2e0 con 0x7f78ac101660 2024-03-28T22:33:51.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.942+0000 7f78abe00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac10d160 con 0x7f78ac103210 2024-03-28T22:33:51.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.942+0000 7f78abe00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac10d460 con 0x7f78ac102850 2024-03-28T22:33:51.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.942+0000 7f78ab400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78ac103210 0x7f78ac1a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:51.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.942+0000 7f78ab400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78ac103210 0x7f78ac1a1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48798/0 (socket says 172.21.15.43:48798) 2024-03-28T22:33:51.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.942+0000 7f78ab400640 1 -- 172.21.15.43:0/3369724808 learned_addr learned my addr 172.21.15.43:0/3369724808 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:51.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.942+0000 7f78aaa00640 1 --2- 172.21.15.43:0/3369724808 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78ac101660 0x7f78ac199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:51.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.942+0000 7f78aa000640 1 --2- 172.21.15.43:0/3369724808 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78ac102850 0x7f78ac19a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:51.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.942+0000 7f78ab400640 1 -- 172.21.15.43:0/3369724808 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78ac102850 msgr2=0x7f78ac19a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:51.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.942+0000 7f78ab400640 1 --2- 172.21.15.43:0/3369724808 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78ac102850 0x7f78ac19a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:51.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.942+0000 7f78ab400640 1 -- 172.21.15.43:0/3369724808 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78ac101660 msgr2=0x7f78ac199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:51.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.942+0000 7f78ab400640 1 --2- 172.21.15.43:0/3369724808 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78ac101660 0x7f78ac199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:51.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.943+0000 7f78ab400640 1 -- 172.21.15.43:0/3369724808 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78ac1a17e0 con 0x7f78ac103210 2024-03-28T22:33:51.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.943+0000 7f78aa000640 1 --2- 172.21.15.43:0/3369724808 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78ac102850 0x7f78ac19a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:33:51.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.943+0000 7f78ab400640 1 --2- 172.21.15.43:0/3369724808 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78ac103210 0x7f78ac1a1170 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f78a400eb10 tx=0x7f78a400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:51.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.943+0000 7f78a3400640 1 -- 172.21.15.43:0/3369724808 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a4004b20 con 0x7f78ac103210 2024-03-28T22:33:51.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.943+0000 7f78a3400640 1 -- 172.21.15.43:0/3369724808 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78a4004cc0 con 0x7f78ac103210 2024-03-28T22:33:51.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.943+0000 7f78a3400640 1 -- 172.21.15.43:0/3369724808 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a4010760 con 0x7f78ac103210 2024-03-28T22:33:51.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.943+0000 7f78abe00640 1 -- 172.21.15.43:0/3369724808 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78ac1a1ad0 con 0x7f78ac103210 2024-03-28T22:33:51.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.943+0000 7f78abe00640 1 -- 172.21.15.43:0/3369724808 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78ac1a1fd0 con 0x7f78ac103210 2024-03-28T22:33:51.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.944+0000 7f78abe00640 1 -- 172.21.15.43:0/3369724808 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78ac1a2260 con 0x7f78ac103210 2024-03-28T22:33:51.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.946+0000 7f78a3400640 1 -- 172.21.15.43:0/3369724808 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f78a4005020 con 0x7f78ac103210 2024-03-28T22:33:51.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.947+0000 7f78a3400640 1 --2- 172.21.15.43:0/3369724808 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f78900a1e00 0x7f78900a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:51.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.948+0000 7f78a3400640 1 -- 172.21.15.43:0/3369724808 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f78a40ce750 con 0x7f78ac103210 2024-03-28T22:33:51.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.948+0000 7f78aaa00640 1 --2- 172.21.15.43:0/3369724808 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f78900a1e00 0x7f78900a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:51.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.949+0000 7f78aaa00640 1 --2- 172.21.15.43:0/3369724808 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f78900a1e00 0x7f78900a42c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f789c000c00 tx=0x7f789c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:51.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:51.951+0000 7f78a3400640 1 -- 172.21.15.43:0/3369724808 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f78a409b8e0 con 0x7f78ac103210 2024-03-28T22:33:52.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.153+0000 7f78abe00640 1 -- 172.21.15.43:0/3369724808 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f78ac1a2430 con 0x7f78ac103210 2024-03-28T22:33:52.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.154+0000 7f78a3400640 1 -- 172.21.15.43:0/3369724808 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) ==== 103+0+11 (secure 0 0 0) 0x7f78a409b1e0 con 0x7f78ac103210 2024-03-28T22:33:52.152 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-03-28T22:33:52.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.157+0000 7f78abe00640 1 -- 172.21.15.43:0/3369724808 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f78900a1e00 msgr2=0x7f78900a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:52.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.157+0000 7f78abe00640 1 --2- 172.21.15.43:0/3369724808 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f78900a1e00 0x7f78900a42c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f789c000c00 tx=0x7f789c005fd0 comp rx=0 tx=0).stop 2024-03-28T22:33:52.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.157+0000 7f78abe00640 1 -- 172.21.15.43:0/3369724808 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78ac103210 msgr2=0x7f78ac1a1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:52.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.157+0000 7f78abe00640 1 --2- 172.21.15.43:0/3369724808 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78ac103210 0x7f78ac1a1170 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f78a400eb10 tx=0x7f78a400c2d0 comp rx=0 tx=0).stop 2024-03-28T22:33:52.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.157+0000 7f78abe00640 1 -- 172.21.15.43:0/3369724808 shutdown_connections 2024-03-28T22:33:52.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.157+0000 7f78abe00640 1 --2- 172.21.15.43:0/3369724808 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f78900a1e00 0x7f78900a42c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:52.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.157+0000 7f78abe00640 1 --2- 172.21.15.43:0/3369724808 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78ac103210 0x7f78ac1a1170 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:52.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.157+0000 7f78abe00640 1 --2- 172.21.15.43:0/3369724808 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78ac102850 0x7f78ac19a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:52.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.157+0000 7f78abe00640 1 --2- 172.21.15.43:0/3369724808 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78ac101660 0x7f78ac199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:52.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.157+0000 7f78abe00640 1 -- 172.21.15.43:0/3369724808 >> 172.21.15.43:0/3369724808 conn(0x7f78ac0fcbf0 msgr2=0x7f78ac073980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:52.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.157+0000 7f78abe00640 1 -- 172.21.15.43:0/3369724808 shutdown_connections 2024-03-28T22:33:52.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.157+0000 7f78abe00640 1 -- 172.21.15.43:0/3369724808 wait complete. 2024-03-28T22:33:52.175 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T22:33:52.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:52 smithi172 ceph-mon[36566]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-28T22:33:52.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:52 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2120997106' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:33:52.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:52 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1226322811' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:33:52.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.401+0000 7f32aca00640 1 -- 172.21.15.43:0/864239565 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f32a81049e0 msgr2=0x7f32a8106dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:52.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.401+0000 7f32aca00640 1 --2- 172.21.15.43:0/864239565 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f32a81049e0 0x7f32a8106dd0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3294009900 tx=0x7f329402bc30 comp rx=0 tx=0).stop 2024-03-28T22:33:52.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.401+0000 7f32aca00640 1 -- 172.21.15.43:0/864239565 shutdown_connections 2024-03-28T22:33:52.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.401+0000 7f32aca00640 1 --2- 172.21.15.43:0/864239565 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f32a81049e0 0x7f32a8106dd0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:52.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.401+0000 7f32aca00640 1 --2- 172.21.15.43:0/864239565 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32a8107310 0x7f32a8109760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:52.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.401+0000 7f32aca00640 1 --2- 172.21.15.43:0/864239565 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32a81020b0 0x7f32a81044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:52.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.401+0000 7f32aca00640 1 -- 172.21.15.43:0/864239565 >> 172.21.15.43:0/864239565 conn(0x7f32a80fbac0 msgr2=0x7f32a80fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:52.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.402+0000 7f32aca00640 1 -- 172.21.15.43:0/864239565 shutdown_connections 2024-03-28T22:33:52.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.402+0000 7f32aca00640 1 -- 172.21.15.43:0/864239565 wait complete. 2024-03-28T22:33:52.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.403+0000 7f32aca00640 1 Processor -- start 2024-03-28T22:33:52.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.403+0000 7f32aca00640 1 -- start start 2024-03-28T22:33:52.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.403+0000 7f32aca00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32a81020b0 0x7f32a8197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:52.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.403+0000 7f32aca00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32a81049e0 0x7f32a8197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:52.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.403+0000 7f32aca00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f32a8107310 0x7f32a819eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:52.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.403+0000 7f32aca00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f32a810d2e0 con 0x7f32a81049e0 2024-03-28T22:33:52.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.403+0000 7f32aca00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f32a810d160 con 0x7f32a8107310 2024-03-28T22:33:52.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.403+0000 7f32aca00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f32a810d460 con 0x7f32a81020b0 2024-03-28T22:33:52.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.404+0000 7f32a6a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32a81049e0 0x7f32a8197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:52.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.404+0000 7f32a6a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32a81049e0 0x7f32a8197e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59774/0 (socket says 172.21.15.43:59774) 2024-03-28T22:33:52.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.404+0000 7f32a6a00640 1 -- 172.21.15.43:0/3026807644 learned_addr learned my addr 172.21.15.43:0/3026807644 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:52.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.404+0000 7f32a7400640 1 --2- 172.21.15.43:0/3026807644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32a81020b0 0x7f32a8197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:52.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.404+0000 7f32a7e00640 1 --2- 172.21.15.43:0/3026807644 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f32a8107310 0x7f32a819eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:52.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.404+0000 7f32a6a00640 1 -- 172.21.15.43:0/3026807644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32a81020b0 msgr2=0x7f32a8197920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:52.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.404+0000 7f32a6a00640 1 --2- 172.21.15.43:0/3026807644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32a81020b0 0x7f32a8197920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:52.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.404+0000 7f32a6a00640 1 -- 172.21.15.43:0/3026807644 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f32a8107310 msgr2=0x7f32a819eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:52.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.404+0000 7f32a6a00640 1 --2- 172.21.15.43:0/3026807644 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f32a8107310 0x7f32a819eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:52.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.404+0000 7f32a6a00640 1 -- 172.21.15.43:0/3026807644 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32a819f540 con 0x7f32a81049e0 2024-03-28T22:33:52.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.404+0000 7f32a7e00640 1 --2- 172.21.15.43:0/3026807644 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f32a8107310 0x7f32a819eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:33:52.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.404+0000 7f32a7400640 1 --2- 172.21.15.43:0/3026807644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32a81020b0 0x7f32a8197920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:33:52.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.405+0000 7f32a6a00640 1 --2- 172.21.15.43:0/3026807644 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32a81049e0 0x7f32a8197e60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3294000c00 tx=0x7f329402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:52.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.405+0000 7f329fe00640 1 -- 172.21.15.43:0/3026807644 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3294043020 con 0x7f32a81049e0 2024-03-28T22:33:52.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.405+0000 7f32aca00640 1 -- 172.21.15.43:0/3026807644 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32a819f7d0 con 0x7f32a81049e0 2024-03-28T22:33:52.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.405+0000 7f329fe00640 1 -- 172.21.15.43:0/3026807644 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32940054e0 con 0x7f32a81049e0 2024-03-28T22:33:52.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.405+0000 7f329fe00640 1 -- 172.21.15.43:0/3026807644 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3294003c40 con 0x7f32a81049e0 2024-03-28T22:33:52.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.405+0000 7f32aca00640 1 -- 172.21.15.43:0/3026807644 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32a819fd20 con 0x7f32a81049e0 2024-03-28T22:33:52.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.406+0000 7f32aca00640 1 -- 172.21.15.43:0/3026807644 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32a80fd5b0 con 0x7f32a81049e0 2024-03-28T22:33:52.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.408+0000 7f329fe00640 1 -- 172.21.15.43:0/3026807644 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f3294046020 con 0x7f32a81049e0 2024-03-28T22:33:52.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.410+0000 7f329fe00640 1 --2- 172.21.15.43:0/3026807644 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f328c0a1d80 0x7f328c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:52.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.410+0000 7f329fe00640 1 -- 172.21.15.43:0/3026807644 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f32940f0410 con 0x7f32a81049e0 2024-03-28T22:33:52.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.410+0000 7f32a7400640 1 --2- 172.21.15.43:0/3026807644 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f328c0a1d80 0x7f328c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:52.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.411+0000 7f32a7400640 1 --2- 172.21.15.43:0/3026807644 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f328c0a1d80 0x7f328c0a4240 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3298009fc0 tx=0x7f3298005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:52.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.412+0000 7f329fe00640 1 -- 172.21.15.43:0/3026807644 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f329404d020 con 0x7f32a81049e0 2024-03-28T22:33:52.430 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:52 smithi043 ceph-mon[31823]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-28T22:33:52.430 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:52 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2120997106' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:33:52.430 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:52 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1226322811' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:33:52.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:52 smithi082 ceph-mon[36471]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-28T22:33:52.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:52 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2120997106' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:33:52.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:52 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1226322811' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:33:52.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.635+0000 7f32aca00640 1 -- 172.21.15.43:0/3026807644 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f32a802d170 con 0x7f32a81049e0 2024-03-28T22:33:52.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.636+0000 7f329fe00640 1 -- 172.21.15.43:0/3026807644 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v92) ==== 81+0+591 (secure 0 0 0) 0x7f32940bd5a0 con 0x7f32a81049e0 2024-03-28T22:33:52.634 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/578791143 2024-03-29T22:25:28.008215+0000 2024-03-28T22:33:52.635 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/4029313215 2024-03-29T22:25:28.008215+0000 2024-03-28T22:33:52.635 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/1574178482 2024-03-29T22:25:28.008215+0000 2024-03-28T22:33:52.635 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6801/3517712438 2024-03-29T22:25:28.008215+0000 2024-03-28T22:33:52.635 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6800/1532177494 2024-03-29T22:24:52.981057+0000 2024-03-28T22:33:52.635 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/1375099638 2024-03-29T22:24:52.981057+0000 2024-03-28T22:33:52.635 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/2710117652 2024-03-29T22:24:52.981057+0000 2024-03-28T22:33:52.635 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6801/1532177494 2024-03-29T22:24:52.981057+0000 2024-03-28T22:33:52.635 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/3215634168 2024-03-29T22:24:52.981057+0000 2024-03-28T22:33:52.635 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6800/3517712438 2024-03-29T22:25:28.008215+0000 2024-03-28T22:33:52.635 INFO:teuthology.orchestra.run.smithi043.stderr:listed 10 entries 2024-03-28T22:33:52.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.639+0000 7f32aca00640 1 -- 172.21.15.43:0/3026807644 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f328c0a1d80 msgr2=0x7f328c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:52.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.639+0000 7f32aca00640 1 --2- 172.21.15.43:0/3026807644 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f328c0a1d80 0x7f328c0a4240 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3298009fc0 tx=0x7f3298005f90 comp rx=0 tx=0).stop 2024-03-28T22:33:52.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.639+0000 7f32aca00640 1 -- 172.21.15.43:0/3026807644 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32a81049e0 msgr2=0x7f32a8197e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:52.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.639+0000 7f32aca00640 1 --2- 172.21.15.43:0/3026807644 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32a81049e0 0x7f32a8197e60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3294000c00 tx=0x7f329402c650 comp rx=0 tx=0).stop 2024-03-28T22:33:52.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.639+0000 7f32aca00640 1 -- 172.21.15.43:0/3026807644 shutdown_connections 2024-03-28T22:33:52.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.639+0000 7f32aca00640 1 --2- 172.21.15.43:0/3026807644 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f328c0a1d80 0x7f328c0a4240 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:52.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.639+0000 7f32aca00640 1 --2- 172.21.15.43:0/3026807644 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f32a8107310 0x7f32a819eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:52.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.639+0000 7f32aca00640 1 --2- 172.21.15.43:0/3026807644 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32a81049e0 0x7f32a8197e60 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:52.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.639+0000 7f32aca00640 1 --2- 172.21.15.43:0/3026807644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32a81020b0 0x7f32a8197920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:52.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.639+0000 7f32aca00640 1 -- 172.21.15.43:0/3026807644 >> 172.21.15.43:0/3026807644 conn(0x7f32a80fbac0 msgr2=0x7f32a81029f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:52.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.640+0000 7f32aca00640 1 -- 172.21.15.43:0/3026807644 shutdown_connections 2024-03-28T22:33:52.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:52.640+0000 7f32aca00640 1 -- 172.21.15.43:0/3026807644 wait complete. 2024-03-28T22:33:52.657 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi043.front.sepia.ceph.com... 2024-03-28T22:33:52.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:33:52.658 DEBUG:teuthology.orchestra.run.smithi043:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-28T22:33:52.742 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi043.stderr:ceph-fuse[73365]: fuse finished with error 0 and tester_r 0 2024-03-28T22:33:52.749 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T22:33:53.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:53 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3369724808' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:33:53.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:53 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3026807644' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T22:33:53.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:53 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3369724808' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:33:53.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:53 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3026807644' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T22:33:53.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:53 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3369724808' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:33:53.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:53 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3026807644' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T22:33:54.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:54 smithi172 ceph-mon[36566]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-28T22:33:54.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:54 smithi082 ceph-mon[36471]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-28T22:33:54.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:54 smithi043 ceph-mon[31823]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-28T22:33:56.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:56 smithi172 ceph-mon[36566]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-28T22:33:56.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:56 smithi082 ceph-mon[36471]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-28T22:33:56.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:56 smithi043 ceph-mon[31823]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-28T22:33:58.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:33:58 smithi172 ceph-mon[36566]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-28T22:33:58.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:33:58 smithi082 ceph-mon[36471]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-28T22:33:58.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:33:58 smithi043 ceph-mon[31823]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-28T22:33:58.851 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi043.front.sepia.ceph.com 2024-03-28T22:33:58.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:33:58.851 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-28T22:33:58.921 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:33:59.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.157+0000 7f9e7a000640 1 -- 172.21.15.43:0/539879348 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e74077cf0 msgr2=0x7f9e74078590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:59.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.157+0000 7f9e7a000640 1 --2- 172.21.15.43:0/539879348 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e74077cf0 0x7f9e74078590 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c009900 tx=0x7f9e6c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:33:59.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.157+0000 7f9e7a000640 1 -- 172.21.15.43:0/539879348 shutdown_connections 2024-03-28T22:33:59.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.157+0000 7f9e7a000640 1 --2- 172.21.15.43:0/539879348 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e74077cf0 0x7f9e74078590 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:59.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.157+0000 7f9e7a000640 1 --2- 172.21.15.43:0/539879348 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e74077350 0x7f9e740777b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:59.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.157+0000 7f9e7a000640 1 --2- 172.21.15.43:0/539879348 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e74076110 0x7f9e74076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:59.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.157+0000 7f9e7a000640 1 -- 172.21.15.43:0/539879348 >> 172.21.15.43:0/539879348 conn(0x7f9e740fef00 msgr2=0x7f9e74101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:59.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.157+0000 7f9e7a000640 1 -- 172.21.15.43:0/539879348 shutdown_connections 2024-03-28T22:33:59.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.157+0000 7f9e7a000640 1 -- 172.21.15.43:0/539879348 wait complete. 2024-03-28T22:33:59.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.158+0000 7f9e7a000640 1 Processor -- start 2024-03-28T22:33:59.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.158+0000 7f9e7a000640 1 -- start start 2024-03-28T22:33:59.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.158+0000 7f9e7a000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e74076110 0x7f9e7419e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:59.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.158+0000 7f9e7a000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e74077350 0x7f9e7419e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:59.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.158+0000 7f9e7a000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e74077cf0 0x7f9e741a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:59.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.158+0000 7f9e7a000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9e74113a60 con 0x7f9e74077350 2024-03-28T22:33:59.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.158+0000 7f9e7a000640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9e741138e0 con 0x7f9e74077cf0 2024-03-28T22:33:59.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.158+0000 7f9e7a000640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9e74113be0 con 0x7f9e74076110 2024-03-28T22:33:59.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.159+0000 7f9e73400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e74076110 0x7f9e7419e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:59.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.159+0000 7f9e72a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e74077350 0x7f9e7419e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:59.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.159+0000 7f9e73e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e74077cf0 0x7f9e741a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:59.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.159+0000 7f9e73400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e74076110 0x7f9e7419e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:33384/0 (socket says 172.21.15.43:33384) 2024-03-28T22:33:59.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.159+0000 7f9e73400640 1 -- 172.21.15.43:0/340059480 learned_addr learned my addr 172.21.15.43:0/340059480 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:59.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.159+0000 7f9e73e00640 1 -- 172.21.15.43:0/340059480 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e74076110 msgr2=0x7f9e7419e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:59.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.159+0000 7f9e73e00640 1 --2- 172.21.15.43:0/340059480 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e74076110 0x7f9e7419e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:59.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.159+0000 7f9e73e00640 1 -- 172.21.15.43:0/340059480 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e74077350 msgr2=0x7f9e7419e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:59.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.159+0000 7f9e73e00640 1 --2- 172.21.15.43:0/340059480 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e74077350 0x7f9e7419e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:59.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.159+0000 7f9e73e00640 1 -- 172.21.15.43:0/340059480 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e741a5c70 con 0x7f9e74077cf0 2024-03-28T22:33:59.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.160+0000 7f9e73e00640 1 --2- 172.21.15.43:0/340059480 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e74077cf0 0x7f9e741a5600 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c02c170 tx=0x7f9e6c02c640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:59.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.160+0000 7f9e6be00640 1 -- 172.21.15.43:0/340059480 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c043020 con 0x7f9e74077cf0 2024-03-28T22:33:59.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.160+0000 7f9e6be00640 1 -- 172.21.15.43:0/340059480 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e6c0054e0 con 0x7f9e74077cf0 2024-03-28T22:33:59.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.160+0000 7f9e7a000640 1 -- 172.21.15.43:0/340059480 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e741a5f00 con 0x7f9e74077cf0 2024-03-28T22:33:59.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.160+0000 7f9e6be00640 1 -- 172.21.15.43:0/340059480 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c003b40 con 0x7f9e74077cf0 2024-03-28T22:33:59.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.160+0000 7f9e7a000640 1 -- 172.21.15.43:0/340059480 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e741a63c0 con 0x7f9e74077cf0 2024-03-28T22:33:59.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.162+0000 7f9e7a000640 1 -- 172.21.15.43:0/340059480 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e74110440 con 0x7f9e74077cf0 2024-03-28T22:33:59.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.163+0000 7f9e6be00640 1 -- 172.21.15.43:0/340059480 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f9e6c046020 con 0x7f9e74077cf0 2024-03-28T22:33:59.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.164+0000 7f9e6be00640 1 --2- 172.21.15.43:0/340059480 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9e580a1d80 0x7f9e580a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:59.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.164+0000 7f9e6be00640 1 -- 172.21.15.43:0/340059480 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f9e6c0effe0 con 0x7f9e74077cf0 2024-03-28T22:33:59.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.165+0000 7f9e73400640 1 --2- 172.21.15.43:0/340059480 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9e580a1d80 0x7f9e580a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:59.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.165+0000 7f9e73400640 1 --2- 172.21.15.43:0/340059480 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9e580a1d80 0x7f9e580a4240 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9e640049d0 tx=0x7f9e64009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:59.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.172+0000 7f9e6be00640 1 -- 172.21.15.43:0/340059480 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9e6c04d020 con 0x7f9e74077cf0 2024-03-28T22:33:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.375+0000 7f9e7a000640 1 -- 172.21.15.43:0/340059480 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9e74062350 con 0x7f9e74077cf0 2024-03-28T22:33:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.377+0000 7f9e6be00640 1 -- 172.21.15.43:0/340059480 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24093 (secure 0 0 0) 0x7f9e6c0bd240 con 0x7f9e74077cf0 2024-03-28T22:33:59.376 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:33:59.377 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":92,"fsid":"c2276428-ed51-11ee-95d0-87774f69a715","created":"2024-03-28T22:24:19.950084+0000","modified":"2024-03-28T22:33:34.473170+0000","last_up_change":"2024-03-28T22:32:31.078737+0000","last_in_change":"2024-03-28T22:32:10.147097+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:28:14.713290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:33:27.167435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:33:28.411825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T22:33:32.523122+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b855adcd-786c-4e42-b8fe-a515a77ce979","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6803","nonce":2826124409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6805","nonce":2826124409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6809","nonce":2826124409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6807","nonce":2826124409}]},"public_addr":"172.21.15.43:6803/2826124409","cluster_addr":"172.21.15.43:6805/2826124409","heartbeat_back_addr":"172.21.15.43:6809/2826124409","heartbeat_front_addr":"172.21.15.43:6807/2826124409","state":["exists","up"]},{"osd":1,"uuid":"031c143d-8ad1-4b68-8db9-3c717137e3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6801","nonce":3043263644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6803","nonce":3043263644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6807","nonce":3043263644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6805","nonce":3043263644}]},"public_addr":"172.21.15.82:6801/3043263644","cluster_addr":"172.21.15.82:6803/3043263644","heartbeat_back_addr":"172.21.15.82:6807/3043263644","heartbeat_front_addr":"172.21.15.82:6805/3043263644","state":["exists","up"]},{"osd":2,"uuid":"20632bbb-f07a-4ca1-b0c4-ff9d508e211c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6801","nonce":4213953645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6803","nonce":4213953645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6807","nonce":4213953645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6805","nonce":4213953645}]},"public_addr":"172.21.15.172:6801/4213953645","cluster_addr":"172.21.15.172:6803/4213953645","heartbeat_back_addr":"172.21.15.172:6807/4213953645","heartbeat_front_addr":"172.21.15.172:6805/4213953645","state":["exists","up"]},{"osd":3,"uuid":"9a1f2312-4dbd-475b-8736-d5ebc8788e09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6811","nonce":1597280920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6813","nonce":1597280920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6817","nonce":1597280920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6815","nonce":1597280920}]},"public_addr":"172.21.15.43:6811/1597280920","cluster_addr":"172.21.15.43:6813/1597280920","heartbeat_back_addr":"172.21.15.43:6817/1597280920","heartbeat_front_addr":"172.21.15.43:6815/1597280920","state":["exists","up"]},{"osd":4,"uuid":"29140ead-7075-42b2-90c1-27cae0b29ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6809","nonce":215451895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6811","nonce":215451895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6815","nonce":215451895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6813","nonce":215451895}]},"public_addr":"172.21.15.82:6809/215451895","cluster_addr":"172.21.15.82:6811/215451895","heartbeat_back_addr":"172.21.15.82:6815/215451895","heartbeat_front_addr":"172.21.15.82:6813/215451895","state":["exists","up"]},{"osd":5,"uuid":"59644c61-93dc-4536-b2ad-0256b24e5cb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6809","nonce":1576148933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6811","nonce":1576148933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6815","nonce":1576148933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6813","nonce":1576148933}]},"public_addr":"172.21.15.172:6809/1576148933","cluster_addr":"172.21.15.172:6811/1576148933","heartbeat_back_addr":"172.21.15.172:6815/1576148933","heartbeat_front_addr":"172.21.15.172:6813/1576148933","state":["exists","up"]},{"osd":6,"uuid":"22d54e57-c3f1-4ecf-9921-8e815b48f92a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6819","nonce":791044279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6821","nonce":791044279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6825","nonce":791044279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6823","nonce":791044279}]},"public_addr":"172.21.15.43:6819/791044279","cluster_addr":"172.21.15.43:6821/791044279","heartbeat_back_addr":"172.21.15.43:6825/791044279","heartbeat_front_addr":"172.21.15.43:6823/791044279","state":["exists","up"]},{"osd":7,"uuid":"248cd1e2-0d6b-452f-a0a8-c51fe1b975ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6817","nonce":2157777323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6819","nonce":2157777323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6823","nonce":2157777323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6821","nonce":2157777323}]},"public_addr":"172.21.15.82:6817/2157777323","cluster_addr":"172.21.15.82:6819/2157777323","heartbeat_back_addr":"172.21.15.82:6823/2157777323","heartbeat_front_addr":"172.21.15.82:6821/2157777323","state":["exists","up"]},{"osd":8,"uuid":"d98aab6e-8bae-4c24-906c-4f492f047b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6817","nonce":1095328248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6819","nonce":1095328248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6823","nonce":1095328248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6821","nonce":1095328248}]},"public_addr":"172.21.15.172:6817/1095328248","cluster_addr":"172.21.15.172:6819/1095328248","heartbeat_back_addr":"172.21.15.172:6823/1095328248","heartbeat_front_addr":"172.21.15.172:6821/1095328248","state":["exists","up"]},{"osd":9,"uuid":"19aa383e-a0f6-45ea-96cc-e7956535caa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6827","nonce":3657791498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6829","nonce":3657791498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6833","nonce":3657791498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6831","nonce":3657791498}]},"public_addr":"172.21.15.43:6827/3657791498","cluster_addr":"172.21.15.43:6829/3657791498","heartbeat_back_addr":"172.21.15.43:6833/3657791498","heartbeat_front_addr":"172.21.15.43:6831/3657791498","state":["exists","up"]},{"osd":10,"uuid":"666977e5-eaf7-41b9-a18f-c380e149345d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6825","nonce":3454954047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6827","nonce":3454954047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6831","nonce":3454954047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6829","nonce":3454954047}]},"public_addr":"172.21.15.82:6825/3454954047","cluster_addr":"172.21.15.82:6827/3454954047","heartbeat_back_addr":"172.21.15.82:6831/3454954047","heartbeat_front_addr":"172.21.15.82:6829/3454954047","state":["exists","up"]},{"osd":11,"uuid":"615cf3be-8076-470e-ad09-c880e7593717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6825","nonce":2563517144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6827","nonce":2563517144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6831","nonce":2563517144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6829","nonce":2563517144}]},"public_addr":"172.21.15.172:6825/2563517144","cluster_addr":"172.21.15.172:6827/2563517144","heartbeat_back_addr":"172.21.15.172:6831/2563517144","heartbeat_front_addr":"172.21.15.172:6829/2563517144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:21.727405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:47.100259+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:10.615438+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:43.632144+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:09.487940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:34.604463+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:09.594452+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:35.709795+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:01.184202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:35.455322+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:02.413166+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:28.550754+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/578791143":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/4029313215":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/1574178482":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6801/3517712438":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6800/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/1375099638":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/2710117652":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6801/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/3215634168":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6800/3517712438":"2024-03-29T22:25:28.008215+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:33:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.379+0000 7f9e7a000640 1 -- 172.21.15.43:0/340059480 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9e580a1d80 msgr2=0x7f9e580a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.379+0000 7f9e7a000640 1 --2- 172.21.15.43:0/340059480 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9e580a1d80 0x7f9e580a4240 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9e640049d0 tx=0x7f9e64009500 comp rx=0 tx=0).stop 2024-03-28T22:33:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.379+0000 7f9e7a000640 1 -- 172.21.15.43:0/340059480 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e74077cf0 msgr2=0x7f9e741a5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.379+0000 7f9e7a000640 1 --2- 172.21.15.43:0/340059480 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e74077cf0 0x7f9e741a5600 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c02c170 tx=0x7f9e6c02c640 comp rx=0 tx=0).stop 2024-03-28T22:33:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.380+0000 7f9e7a000640 1 -- 172.21.15.43:0/340059480 shutdown_connections 2024-03-28T22:33:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.380+0000 7f9e7a000640 1 --2- 172.21.15.43:0/340059480 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9e580a1d80 0x7f9e580a4240 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.380+0000 7f9e7a000640 1 --2- 172.21.15.43:0/340059480 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e74077cf0 0x7f9e741a5600 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.380+0000 7f9e7a000640 1 --2- 172.21.15.43:0/340059480 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e74077350 0x7f9e7419e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.380+0000 7f9e7a000640 1 --2- 172.21.15.43:0/340059480 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e74076110 0x7f9e7419e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.380+0000 7f9e7a000640 1 -- 172.21.15.43:0/340059480 >> 172.21.15.43:0/340059480 conn(0x7f9e740fef00 msgr2=0x7f9e74100520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.380+0000 7f9e7a000640 1 -- 172.21.15.43:0/340059480 shutdown_connections 2024-03-28T22:33:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.380+0000 7f9e7a000640 1 -- 172.21.15.43:0/340059480 wait complete. 2024-03-28T22:33:59.401 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:33:59.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.628+0000 7f18a2800640 1 -- 172.21.15.43:0/2126006735 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f189c107bb0 msgr2=0x7f189c107fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:59.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.628+0000 7f18a2800640 1 --2- 172.21.15.43:0/2126006735 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f189c107bb0 0x7f189c107fd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1890009900 tx=0x7f189002bc40 comp rx=0 tx=0).stop 2024-03-28T22:33:59.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.628+0000 7f18a2800640 1 -- 172.21.15.43:0/2126006735 shutdown_connections 2024-03-28T22:33:59.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.628+0000 7f18a2800640 1 --2- 172.21.15.43:0/2126006735 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f189c109760 0x7f189c10fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:59.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.628+0000 7f18a2800640 1 --2- 172.21.15.43:0/2126006735 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f189c108da0 0x7f189c109220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:59.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.628+0000 7f18a2800640 1 --2- 172.21.15.43:0/2126006735 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f189c107bb0 0x7f189c107fd0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:59.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.628+0000 7f18a2800640 1 -- 172.21.15.43:0/2126006735 >> 172.21.15.43:0/2126006735 conn(0x7f189c076b60 msgr2=0x7f189c078f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:59.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.628+0000 7f18a2800640 1 -- 172.21.15.43:0/2126006735 shutdown_connections 2024-03-28T22:33:59.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.628+0000 7f18a2800640 1 -- 172.21.15.43:0/2126006735 wait complete. 2024-03-28T22:33:59.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.629+0000 7f18a2800640 1 Processor -- start 2024-03-28T22:33:59.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.629+0000 7f18a2800640 1 -- start start 2024-03-28T22:33:59.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.629+0000 7f18a2800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f189c107bb0 0x7f189c19deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:59.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.630+0000 7f18a2800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f189c108da0 0x7f189c19e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:59.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.630+0000 7f18a2800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f189c109760 0x7f189c1a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:59.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.630+0000 7f18a2800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f189c113850 con 0x7f189c108da0 2024-03-28T22:33:59.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.630+0000 7f18a2800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f189c1136d0 con 0x7f189c109760 2024-03-28T22:33:59.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.630+0000 7f18a2800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f189c1139d0 con 0x7f189c107bb0 2024-03-28T22:33:59.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.630+0000 7f18a1e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f189c109760 0x7f189c1a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:59.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.630+0000 7f18a0a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f189c108da0 0x7f189c19e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:59.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.630+0000 7f18a1e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f189c109760 0x7f189c1a5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:32848/0 (socket says 172.21.15.43:32848) 2024-03-28T22:33:59.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.630+0000 7f18a1e00640 1 -- 172.21.15.43:0/976790877 learned_addr learned my addr 172.21.15.43:0/976790877 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:33:59.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.630+0000 7f18a1400640 1 --2- 172.21.15.43:0/976790877 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f189c107bb0 0x7f189c19deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:59.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.630+0000 7f18a1e00640 1 -- 172.21.15.43:0/976790877 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f189c107bb0 msgr2=0x7f189c19deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:59.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.630+0000 7f18a1e00640 1 --2- 172.21.15.43:0/976790877 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f189c107bb0 0x7f189c19deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:59.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.630+0000 7f18a1e00640 1 -- 172.21.15.43:0/976790877 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f189c108da0 msgr2=0x7f189c19e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:59.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.630+0000 7f18a1e00640 1 --2- 172.21.15.43:0/976790877 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f189c108da0 0x7f189c19e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:59.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.630+0000 7f18a1e00640 1 -- 172.21.15.43:0/976790877 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f189c1a5ad0 con 0x7f189c109760 2024-03-28T22:33:59.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.631+0000 7f18a1e00640 1 --2- 172.21.15.43:0/976790877 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f189c109760 0x7f189c1a5460 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f189400cce0 tx=0x7f189400a620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:59.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.631+0000 7f189a000640 1 -- 172.21.15.43:0/976790877 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f189400f450 con 0x7f189c109760 2024-03-28T22:33:59.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.631+0000 7f189a000640 1 -- 172.21.15.43:0/976790877 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f189400fad0 con 0x7f189c109760 2024-03-28T22:33:59.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.631+0000 7f189a000640 1 -- 172.21.15.43:0/976790877 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1894018c10 con 0x7f189c109760 2024-03-28T22:33:59.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.631+0000 7f18a2800640 1 -- 172.21.15.43:0/976790877 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f189c1a5dc0 con 0x7f189c109760 2024-03-28T22:33:59.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.631+0000 7f18a2800640 1 -- 172.21.15.43:0/976790877 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f189c1a6230 con 0x7f189c109760 2024-03-28T22:33:59.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.632+0000 7f18a2800640 1 -- 172.21.15.43:0/976790877 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f189c10ee70 con 0x7f189c109760 2024-03-28T22:33:59.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.634+0000 7f189a000640 1 -- 172.21.15.43:0/976790877 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f189400f5f0 con 0x7f189c109760 2024-03-28T22:33:59.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.635+0000 7f189a000640 1 --2- 172.21.15.43:0/976790877 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f18840a1d80 0x7f18840a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:33:59.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.635+0000 7f189a000640 1 -- 172.21.15.43:0/976790877 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f18940ce620 con 0x7f189c109760 2024-03-28T22:33:59.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.636+0000 7f18a1400640 1 --2- 172.21.15.43:0/976790877 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f18840a1d80 0x7f18840a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:33:59.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.636+0000 7f18a1400640 1 --2- 172.21.15.43:0/976790877 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f18840a1d80 0x7f18840a4240 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f189002c250 tx=0x7f189002cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:33:59.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.639+0000 7f189a000640 1 -- 172.21.15.43:0/976790877 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1894014030 con 0x7f189c109760 2024-03-28T22:33:59.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.846+0000 7f18a2800640 1 -- 172.21.15.43:0/976790877 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f189c0623f0 con 0x7f189c109760 2024-03-28T22:33:59.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.848+0000 7f189a000640 1 -- 172.21.15.43:0/976790877 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24093 (secure 0 0 0) 0x7f189409b7b0 con 0x7f189c109760 2024-03-28T22:33:59.848 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:33:59.848 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":92,"fsid":"c2276428-ed51-11ee-95d0-87774f69a715","created":"2024-03-28T22:24:19.950084+0000","modified":"2024-03-28T22:33:34.473170+0000","last_up_change":"2024-03-28T22:32:31.078737+0000","last_in_change":"2024-03-28T22:32:10.147097+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:28:14.713290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:33:27.167435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:33:28.411825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T22:33:32.523122+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b855adcd-786c-4e42-b8fe-a515a77ce979","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6803","nonce":2826124409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6805","nonce":2826124409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6809","nonce":2826124409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6807","nonce":2826124409}]},"public_addr":"172.21.15.43:6803/2826124409","cluster_addr":"172.21.15.43:6805/2826124409","heartbeat_back_addr":"172.21.15.43:6809/2826124409","heartbeat_front_addr":"172.21.15.43:6807/2826124409","state":["exists","up"]},{"osd":1,"uuid":"031c143d-8ad1-4b68-8db9-3c717137e3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6801","nonce":3043263644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6803","nonce":3043263644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6807","nonce":3043263644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6805","nonce":3043263644}]},"public_addr":"172.21.15.82:6801/3043263644","cluster_addr":"172.21.15.82:6803/3043263644","heartbeat_back_addr":"172.21.15.82:6807/3043263644","heartbeat_front_addr":"172.21.15.82:6805/3043263644","state":["exists","up"]},{"osd":2,"uuid":"20632bbb-f07a-4ca1-b0c4-ff9d508e211c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6801","nonce":4213953645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6803","nonce":4213953645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6807","nonce":4213953645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6805","nonce":4213953645}]},"public_addr":"172.21.15.172:6801/4213953645","cluster_addr":"172.21.15.172:6803/4213953645","heartbeat_back_addr":"172.21.15.172:6807/4213953645","heartbeat_front_addr":"172.21.15.172:6805/4213953645","state":["exists","up"]},{"osd":3,"uuid":"9a1f2312-4dbd-475b-8736-d5ebc8788e09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6811","nonce":1597280920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6813","nonce":1597280920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6817","nonce":1597280920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6815","nonce":1597280920}]},"public_addr":"172.21.15.43:6811/1597280920","cluster_addr":"172.21.15.43:6813/1597280920","heartbeat_back_addr":"172.21.15.43:6817/1597280920","heartbeat_front_addr":"172.21.15.43:6815/1597280920","state":["exists","up"]},{"osd":4,"uuid":"29140ead-7075-42b2-90c1-27cae0b29ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6809","nonce":215451895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6811","nonce":215451895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6815","nonce":215451895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6813","nonce":215451895}]},"public_addr":"172.21.15.82:6809/215451895","cluster_addr":"172.21.15.82:6811/215451895","heartbeat_back_addr":"172.21.15.82:6815/215451895","heartbeat_front_addr":"172.21.15.82:6813/215451895","state":["exists","up"]},{"osd":5,"uuid":"59644c61-93dc-4536-b2ad-0256b24e5cb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6809","nonce":1576148933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6811","nonce":1576148933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6815","nonce":1576148933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6813","nonce":1576148933}]},"public_addr":"172.21.15.172:6809/1576148933","cluster_addr":"172.21.15.172:6811/1576148933","heartbeat_back_addr":"172.21.15.172:6815/1576148933","heartbeat_front_addr":"172.21.15.172:6813/1576148933","state":["exists","up"]},{"osd":6,"uuid":"22d54e57-c3f1-4ecf-9921-8e815b48f92a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6819","nonce":791044279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6821","nonce":791044279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6825","nonce":791044279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6823","nonce":791044279}]},"public_addr":"172.21.15.43:6819/791044279","cluster_addr":"172.21.15.43:6821/791044279","heartbeat_back_addr":"172.21.15.43:6825/791044279","heartbeat_front_addr":"172.21.15.43:6823/791044279","state":["exists","up"]},{"osd":7,"uuid":"248cd1e2-0d6b-452f-a0a8-c51fe1b975ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6817","nonce":2157777323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6819","nonce":2157777323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6823","nonce":2157777323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6821","nonce":2157777323}]},"public_addr":"172.21.15.82:6817/2157777323","cluster_addr":"172.21.15.82:6819/2157777323","heartbeat_back_addr":"172.21.15.82:6823/2157777323","heartbeat_front_addr":"172.21.15.82:6821/2157777323","state":["exists","up"]},{"osd":8,"uuid":"d98aab6e-8bae-4c24-906c-4f492f047b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6817","nonce":1095328248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6819","nonce":1095328248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6823","nonce":1095328248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6821","nonce":1095328248}]},"public_addr":"172.21.15.172:6817/1095328248","cluster_addr":"172.21.15.172:6819/1095328248","heartbeat_back_addr":"172.21.15.172:6823/1095328248","heartbeat_front_addr":"172.21.15.172:6821/1095328248","state":["exists","up"]},{"osd":9,"uuid":"19aa383e-a0f6-45ea-96cc-e7956535caa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6827","nonce":3657791498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6829","nonce":3657791498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6833","nonce":3657791498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6831","nonce":3657791498}]},"public_addr":"172.21.15.43:6827/3657791498","cluster_addr":"172.21.15.43:6829/3657791498","heartbeat_back_addr":"172.21.15.43:6833/3657791498","heartbeat_front_addr":"172.21.15.43:6831/3657791498","state":["exists","up"]},{"osd":10,"uuid":"666977e5-eaf7-41b9-a18f-c380e149345d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6825","nonce":3454954047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6827","nonce":3454954047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6831","nonce":3454954047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6829","nonce":3454954047}]},"public_addr":"172.21.15.82:6825/3454954047","cluster_addr":"172.21.15.82:6827/3454954047","heartbeat_back_addr":"172.21.15.82:6831/3454954047","heartbeat_front_addr":"172.21.15.82:6829/3454954047","state":["exists","up"]},{"osd":11,"uuid":"615cf3be-8076-470e-ad09-c880e7593717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6825","nonce":2563517144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6827","nonce":2563517144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6831","nonce":2563517144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6829","nonce":2563517144}]},"public_addr":"172.21.15.172:6825/2563517144","cluster_addr":"172.21.15.172:6827/2563517144","heartbeat_back_addr":"172.21.15.172:6831/2563517144","heartbeat_front_addr":"172.21.15.172:6829/2563517144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:21.727405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:47.100259+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:10.615438+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:43.632144+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:09.487940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:34.604463+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:09.594452+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:35.709795+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:01.184202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:35.455322+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:02.413166+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:28.550754+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/578791143":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/4029313215":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/1574178482":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6801/3517712438":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6800/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/1375099638":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/2710117652":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6801/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/3215634168":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6800/3517712438":"2024-03-29T22:25:28.008215+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:33:59.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.851+0000 7f18a2800640 1 -- 172.21.15.43:0/976790877 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f18840a1d80 msgr2=0x7f18840a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:59.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.851+0000 7f18a2800640 1 --2- 172.21.15.43:0/976790877 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f18840a1d80 0x7f18840a4240 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f189002c250 tx=0x7f189002cf90 comp rx=0 tx=0).stop 2024-03-28T22:33:59.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.851+0000 7f18a2800640 1 -- 172.21.15.43:0/976790877 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f189c109760 msgr2=0x7f189c1a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:33:59.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.851+0000 7f18a2800640 1 --2- 172.21.15.43:0/976790877 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f189c109760 0x7f189c1a5460 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f189400cce0 tx=0x7f189400a620 comp rx=0 tx=0).stop 2024-03-28T22:33:59.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.851+0000 7f18a2800640 1 -- 172.21.15.43:0/976790877 shutdown_connections 2024-03-28T22:33:59.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.851+0000 7f18a2800640 1 --2- 172.21.15.43:0/976790877 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f18840a1d80 0x7f18840a4240 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:59.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.851+0000 7f18a2800640 1 --2- 172.21.15.43:0/976790877 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f189c109760 0x7f189c1a5460 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:59.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.851+0000 7f18a2800640 1 --2- 172.21.15.43:0/976790877 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f189c108da0 0x7f189c19e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:59.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.851+0000 7f18a2800640 1 --2- 172.21.15.43:0/976790877 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f189c107bb0 0x7f189c19deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:33:59.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.851+0000 7f18a2800640 1 -- 172.21.15.43:0/976790877 >> 172.21.15.43:0/976790877 conn(0x7f189c076b60 msgr2=0x7f189c10c040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:33:59.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.851+0000 7f18a2800640 1 -- 172.21.15.43:0/976790877 shutdown_connections 2024-03-28T22:33:59.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:33:59.851+0000 7f18a2800640 1 -- 172.21.15.43:0/976790877 wait complete. 2024-03-28T22:33:59.871 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-28T22:34:00.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.107+0000 7f7943c00640 1 -- 172.21.15.43:0/440746423 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f793c0fffd0 msgr2=0x7f793c102400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:00.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.107+0000 7f7943c00640 1 --2- 172.21.15.43:0/440746423 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f793c0fffd0 0x7f793c102400 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f792c009900 tx=0x7f792c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:34:00.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.107+0000 7f7943c00640 1 -- 172.21.15.43:0/440746423 shutdown_connections 2024-03-28T22:34:00.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.107+0000 7f7943c00640 1 --2- 172.21.15.43:0/440746423 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f793c1052f0 0x7f793c105770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:00.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.107+0000 7f7943c00640 1 --2- 172.21.15.43:0/440746423 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f793c102940 0x7f793c104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:00.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.107+0000 7f7943c00640 1 --2- 172.21.15.43:0/440746423 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f793c0fffd0 0x7f793c102400 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:00.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.107+0000 7f7943c00640 1 -- 172.21.15.43:0/440746423 >> 172.21.15.43:0/440746423 conn(0x7f793c0fbbf0 msgr2=0x7f793c0fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:00.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.107+0000 7f7943c00640 1 -- 172.21.15.43:0/440746423 shutdown_connections 2024-03-28T22:34:00.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.107+0000 7f7943c00640 1 -- 172.21.15.43:0/440746423 wait complete. 2024-03-28T22:34:00.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.108+0000 7f7943c00640 1 Processor -- start 2024-03-28T22:34:00.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.108+0000 7f7943c00640 1 -- start start 2024-03-28T22:34:00.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.108+0000 7f7943c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f793c0fffd0 0x7f793c19e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:00.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.108+0000 7f7943c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f793c102940 0x7f793c19e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:00.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.108+0000 7f7943c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f793c1052f0 0x7f793c1a5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:00.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.108+0000 7f7943c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f793c113a20 con 0x7f793c102940 2024-03-28T22:34:00.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.108+0000 7f7943c00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f793c1138a0 con 0x7f793c0fffd0 2024-03-28T22:34:00.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.108+0000 7f7943c00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f793c113ba0 con 0x7f793c1052f0 2024-03-28T22:34:00.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.109+0000 7f7941200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f793c0fffd0 0x7f793c19e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:00.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.109+0000 7f7941200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f793c0fffd0 0x7f793c19e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:32864/0 (socket says 172.21.15.43:32864) 2024-03-28T22:34:00.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.109+0000 7f793be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f793c102940 0x7f793c19e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:00.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.109+0000 7f7941200640 1 -- 172.21.15.43:0/909655758 learned_addr learned my addr 172.21.15.43:0/909655758 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:34:00.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.109+0000 7f7941c00640 1 --2- 172.21.15.43:0/909655758 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f793c1052f0 0x7f793c1a5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:00.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.109+0000 7f7941200640 1 -- 172.21.15.43:0/909655758 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f793c1052f0 msgr2=0x7f793c1a5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:00.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.109+0000 7f7941200640 1 --2- 172.21.15.43:0/909655758 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f793c1052f0 0x7f793c1a5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:00.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.109+0000 7f7941200640 1 -- 172.21.15.43:0/909655758 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f793c102940 msgr2=0x7f793c19e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:00.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.109+0000 7f7941200640 1 --2- 172.21.15.43:0/909655758 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f793c102940 0x7f793c19e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:00.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.109+0000 7f7941200640 1 -- 172.21.15.43:0/909655758 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f793c1a5c80 con 0x7f793c0fffd0 2024-03-28T22:34:00.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.109+0000 7f7941200640 1 --2- 172.21.15.43:0/909655758 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f793c0fffd0 0x7f793c19e060 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f792c000c00 tx=0x7f792c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:00.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.110+0000 7f7939600640 1 -- 172.21.15.43:0/909655758 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792c043020 con 0x7f793c0fffd0 2024-03-28T22:34:00.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.110+0000 7f7939600640 1 -- 172.21.15.43:0/909655758 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f792c0054e0 con 0x7f793c0fffd0 2024-03-28T22:34:00.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.110+0000 7f7943c00640 1 -- 172.21.15.43:0/909655758 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f793c1a5f10 con 0x7f793c0fffd0 2024-03-28T22:34:00.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.110+0000 7f7939600640 1 -- 172.21.15.43:0/909655758 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792c003c40 con 0x7f793c0fffd0 2024-03-28T22:34:00.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.110+0000 7f7943c00640 1 -- 172.21.15.43:0/909655758 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f793c1a6380 con 0x7f793c0fffd0 2024-03-28T22:34:00.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.111+0000 7f7943c00640 1 -- 172.21.15.43:0/909655758 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f793c0fd6e0 con 0x7f793c0fffd0 2024-03-28T22:34:00.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.113+0000 7f7939600640 1 -- 172.21.15.43:0/909655758 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f792c046020 con 0x7f793c0fffd0 2024-03-28T22:34:00.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.114+0000 7f7939600640 1 --2- 172.21.15.43:0/909655758 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f79200a1e00 0x7f79200a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:00.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.114+0000 7f7939600640 1 -- 172.21.15.43:0/909655758 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f792c0effe0 con 0x7f793c0fffd0 2024-03-28T22:34:00.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.114+0000 7f793be00640 1 --2- 172.21.15.43:0/909655758 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f79200a1e00 0x7f79200a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:00.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.115+0000 7f793be00640 1 --2- 172.21.15.43:0/909655758 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f79200a1e00 0x7f79200a42c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f793c19f570 tx=0x7f7928005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:00.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.118+0000 7f7939600640 1 -- 172.21.15.43:0/909655758 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f792c04d020 con 0x7f793c0fffd0 2024-03-28T22:34:00.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:00 smithi043 ceph-mon[31823]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-28T22:34:00.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:00 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/340059480' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:34:00.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:00 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/976790877' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:34:00.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:00.360+0000 7f7943c00640 1 -- 172.21.15.43:0/909655758 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f793c02d170 con 0x7f793c0fffd0 2024-03-28T22:34:00.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:00 smithi172 ceph-mon[36566]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-28T22:34:00.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:00 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/340059480' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:34:00.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:00 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/976790877' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:34:00.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:00 smithi082 ceph-mon[36471]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-28T22:34:00.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:00 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/340059480' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:34:00.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:00 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/976790877' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:34:01.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.100+0000 7f7939600640 1 -- 172.21.15.43:0/909655758 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7f792c0bd190 con 0x7f793c0fffd0 2024-03-28T22:34:01.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.103+0000 7f7943c00640 1 -- 172.21.15.43:0/909655758 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f79200a1e00 msgr2=0x7f79200a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:01.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.103+0000 7f7943c00640 1 --2- 172.21.15.43:0/909655758 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f79200a1e00 0x7f79200a42c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f793c19f570 tx=0x7f7928005f90 comp rx=0 tx=0).stop 2024-03-28T22:34:01.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.103+0000 7f7943c00640 1 -- 172.21.15.43:0/909655758 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f793c0fffd0 msgr2=0x7f793c19e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:01.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.103+0000 7f7943c00640 1 --2- 172.21.15.43:0/909655758 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f793c0fffd0 0x7f793c19e060 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f792c000c00 tx=0x7f792c02c650 comp rx=0 tx=0).stop 2024-03-28T22:34:01.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.104+0000 7f7943c00640 1 -- 172.21.15.43:0/909655758 shutdown_connections 2024-03-28T22:34:01.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.104+0000 7f7943c00640 1 --2- 172.21.15.43:0/909655758 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f793c1052f0 0x7f793c1a5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:01.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.104+0000 7f7943c00640 1 --2- 172.21.15.43:0/909655758 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f793c102940 0x7f793c19e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:01.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.104+0000 7f7943c00640 1 --2- 172.21.15.43:0/909655758 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f79200a1e00 0x7f79200a42c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:01.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.104+0000 7f7943c00640 1 --2- 172.21.15.43:0/909655758 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f793c0fffd0 0x7f793c19e060 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:01.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.104+0000 7f7943c00640 1 -- 172.21.15.43:0/909655758 >> 172.21.15.43:0/909655758 conn(0x7f793c0fbbf0 msgr2=0x7f793c1032d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:01.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.104+0000 7f7943c00640 1 -- 172.21.15.43:0/909655758 shutdown_connections 2024-03-28T22:34:01.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.104+0000 7f7943c00640 1 -- 172.21.15.43:0/909655758 wait complete. 2024-03-28T22:34:01.125 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-03-28T22:34:01.126 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-03-28T22:34:01.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.347+0000 7f6f9ee00640 1 -- 172.21.15.43:0/536532199 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f98109820 msgr2=0x7f6f981100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:01.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.347+0000 7f6f9ee00640 1 --2- 172.21.15.43:0/536532199 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f98109820 0x7f6f981100b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6f90009ca0 tx=0x7f6f9002c100 comp rx=0 tx=0).stop 2024-03-28T22:34:01.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.347+0000 7f6f9ee00640 1 -- 172.21.15.43:0/536532199 shutdown_connections 2024-03-28T22:34:01.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.347+0000 7f6f9ee00640 1 --2- 172.21.15.43:0/536532199 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f98109820 0x7f6f981100b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:01.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.347+0000 7f6f9ee00640 1 --2- 172.21.15.43:0/536532199 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f98108e60 0x7f6f981092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:01.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.347+0000 7f6f9ee00640 1 --2- 172.21.15.43:0/536532199 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f98107c70 0x7f6f98108090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:01.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.347+0000 7f6f9ee00640 1 -- 172.21.15.43:0/536532199 >> 172.21.15.43:0/536532199 conn(0x7f6f98076c20 msgr2=0x7f6f98079040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:01.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.347+0000 7f6f9ee00640 1 -- 172.21.15.43:0/536532199 shutdown_connections 2024-03-28T22:34:01.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.347+0000 7f6f9ee00640 1 -- 172.21.15.43:0/536532199 wait complete. 2024-03-28T22:34:01.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.348+0000 7f6f9ee00640 1 Processor -- start 2024-03-28T22:34:01.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.348+0000 7f6f9ee00640 1 -- start start 2024-03-28T22:34:01.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.349+0000 7f6f9ee00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f98107c70 0x7f6f9819deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:01.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.349+0000 7f6f9ee00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f98108e60 0x7f6f9819e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:01.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.349+0000 7f6f9ee00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f98109820 0x7f6f981a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:01.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.349+0000 7f6f9ee00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6f981138d0 con 0x7f6f98108e60 2024-03-28T22:34:01.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.349+0000 7f6f9ee00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f6f98113750 con 0x7f6f98109820 2024-03-28T22:34:01.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.349+0000 7f6f9ee00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f6f98113a50 con 0x7f6f98107c70 2024-03-28T22:34:01.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.349+0000 7f6f9d000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f98108e60 0x7f6f9819e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:01.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.349+0000 7f6f9d000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f98108e60 0x7f6f9819e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34386/0 (socket says 172.21.15.43:34386) 2024-03-28T22:34:01.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.349+0000 7f6f9d000640 1 -- 172.21.15.43:0/2824752917 learned_addr learned my addr 172.21.15.43:0/2824752917 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:34:01.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.349+0000 7f6f9e400640 1 --2- 172.21.15.43:0/2824752917 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f98109820 0x7f6f981a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:01.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.350+0000 7f6f9da00640 1 --2- 172.21.15.43:0/2824752917 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f98107c70 0x7f6f9819deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:01.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.350+0000 7f6f9d000640 1 -- 172.21.15.43:0/2824752917 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f98107c70 msgr2=0x7f6f9819deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:01.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.350+0000 7f6f9d000640 1 --2- 172.21.15.43:0/2824752917 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f98107c70 0x7f6f9819deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:01.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.350+0000 7f6f9d000640 1 -- 172.21.15.43:0/2824752917 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f98109820 msgr2=0x7f6f981a5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:01.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.350+0000 7f6f9d000640 1 --2- 172.21.15.43:0/2824752917 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f98109820 0x7f6f981a5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:01.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.350+0000 7f6f9d000640 1 -- 172.21.15.43:0/2824752917 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f981a5ad0 con 0x7f6f98108e60 2024-03-28T22:34:01.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.350+0000 7f6f9da00640 1 --2- 172.21.15.43:0/2824752917 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f98107c70 0x7f6f9819deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:34:01.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.350+0000 7f6f9d000640 1 --2- 172.21.15.43:0/2824752917 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f98108e60 0x7f6f9819e3f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6f8400d950 tx=0x7f6f8400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:01.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.350+0000 7f6f96000640 1 -- 172.21.15.43:0/2824752917 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f84017070 con 0x7f6f98108e60 2024-03-28T22:34:01.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.350+0000 7f6f9ee00640 1 -- 172.21.15.43:0/2824752917 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f981a5dc0 con 0x7f6f98108e60 2024-03-28T22:34:01.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.350+0000 7f6f96000640 1 -- 172.21.15.43:0/2824752917 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f84009c30 con 0x7f6f98108e60 2024-03-28T22:34:01.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.350+0000 7f6f96000640 1 -- 172.21.15.43:0/2824752917 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f8400f6b0 con 0x7f6f98108e60 2024-03-28T22:34:01.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.351+0000 7f6f9ee00640 1 -- 172.21.15.43:0/2824752917 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f981a6290 con 0x7f6f98108e60 2024-03-28T22:34:01.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.352+0000 7f6f9ee00640 1 -- 172.21.15.43:0/2824752917 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f9810ef90 con 0x7f6f98108e60 2024-03-28T22:34:01.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.355+0000 7f6f96000640 1 -- 172.21.15.43:0/2824752917 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f6f840109d0 con 0x7f6f98108e60 2024-03-28T22:34:01.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.356+0000 7f6f96000640 1 --2- 172.21.15.43:0/2824752917 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f6f7c0a1e00 0x7f6f7c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:01.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.356+0000 7f6f96000640 1 -- 172.21.15.43:0/2824752917 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f6f840cded0 con 0x7f6f98108e60 2024-03-28T22:34:01.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.356+0000 7f6f9da00640 1 --2- 172.21.15.43:0/2824752917 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f6f7c0a1e00 0x7f6f7c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:01.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.357+0000 7f6f9da00640 1 --2- 172.21.15.43:0/2824752917 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f6f7c0a1e00 0x7f6f7c0a42c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6f88005d00 tx=0x7f6f88005ef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:01.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.361+0000 7f6f96000640 1 -- 172.21.15.43:0/2824752917 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6f84009de0 con 0x7f6f98108e60 2024-03-28T22:34:01.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:01 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/909655758' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T22:34:01.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:01 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T22:34:01.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:01 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/909655758' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T22:34:01.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:01 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T22:34:01.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:01 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/909655758' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T22:34:01.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:01 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T22:34:01.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:01.608+0000 7f6f9ee00640 1 -- 172.21.15.43:0/2824752917 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f6f98062350 con 0x7f6f98108e60 2024-03-28T22:34:02.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.109+0000 7f6f96000640 1 -- 172.21.15.43:0/2824752917 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7f6f8409b060 con 0x7f6f98108e60 2024-03-28T22:34:02.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.112+0000 7f6f9ee00640 1 -- 172.21.15.43:0/2824752917 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f6f7c0a1e00 msgr2=0x7f6f7c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:02.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.112+0000 7f6f9ee00640 1 --2- 172.21.15.43:0/2824752917 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f6f7c0a1e00 0x7f6f7c0a42c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6f88005d00 tx=0x7f6f88005ef0 comp rx=0 tx=0).stop 2024-03-28T22:34:02.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.112+0000 7f6f9ee00640 1 -- 172.21.15.43:0/2824752917 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f98108e60 msgr2=0x7f6f9819e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:02.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.112+0000 7f6f9ee00640 1 --2- 172.21.15.43:0/2824752917 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f98108e60 0x7f6f9819e3f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6f8400d950 tx=0x7f6f8400de10 comp rx=0 tx=0).stop 2024-03-28T22:34:02.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.113+0000 7f6f9ee00640 1 -- 172.21.15.43:0/2824752917 shutdown_connections 2024-03-28T22:34:02.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.113+0000 7f6f9ee00640 1 --2- 172.21.15.43:0/2824752917 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f6f7c0a1e00 0x7f6f7c0a42c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:02.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.113+0000 7f6f9ee00640 1 --2- 172.21.15.43:0/2824752917 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f98109820 0x7f6f981a5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:02.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.113+0000 7f6f9ee00640 1 --2- 172.21.15.43:0/2824752917 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f98108e60 0x7f6f9819e3f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:02.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.113+0000 7f6f9ee00640 1 --2- 172.21.15.43:0/2824752917 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f98107c70 0x7f6f9819deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:02.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.113+0000 7f6f9ee00640 1 -- 172.21.15.43:0/2824752917 >> 172.21.15.43:0/2824752917 conn(0x7f6f98076c20 msgr2=0x7f6f9810c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:02.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.113+0000 7f6f9ee00640 1 -- 172.21.15.43:0/2824752917 shutdown_connections 2024-03-28T22:34:02.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.113+0000 7f6f9ee00640 1 -- 172.21.15.43:0/2824752917 wait complete. 2024-03-28T22:34:02.140 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-28T22:34:02.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.349+0000 7f023f200640 1 -- 172.21.15.43:0/3767962995 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0240075710 msgr2=0x7f0240073b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:02.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.349+0000 7f023f200640 1 --2- 172.21.15.43:0/3767962995 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0240075710 0x7f0240073b60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0230009ca0 tx=0x7f023002c190 comp rx=0 tx=0).stop 2024-03-28T22:34:02.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.349+0000 7f023f200640 1 -- 172.21.15.43:0/3767962995 shutdown_connections 2024-03-28T22:34:02.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.349+0000 7f023f200640 1 --2- 172.21.15.43:0/3767962995 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0240109500 0x7f024010b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:02.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.349+0000 7f023f200640 1 --2- 172.21.15.43:0/3767962995 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0240074130 0x7f0240074590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:02.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.349+0000 7f023f200640 1 --2- 172.21.15.43:0/3767962995 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0240075710 0x7f0240073b60 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:02.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.349+0000 7f023f200640 1 -- 172.21.15.43:0/3767962995 >> 172.21.15.43:0/3767962995 conn(0x7f02400fcbc0 msgr2=0x7f02400ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:02.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.350+0000 7f023f200640 1 -- 172.21.15.43:0/3767962995 shutdown_connections 2024-03-28T22:34:02.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.350+0000 7f023f200640 1 -- 172.21.15.43:0/3767962995 wait complete. 2024-03-28T22:34:02.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.351+0000 7f023f200640 1 Processor -- start 2024-03-28T22:34:02.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.351+0000 7f023f200640 1 -- start start 2024-03-28T22:34:02.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.351+0000 7f023f200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0240074130 0x7f0240199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:02.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.351+0000 7f023f200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0240075710 0x7f024019a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:02.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.352+0000 7f023f200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0240109500 0x7f02401a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:02.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.352+0000 7f023f200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f024010f4b0 con 0x7f0240074130 2024-03-28T22:34:02.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.352+0000 7f023f200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f024010f330 con 0x7f0240075710 2024-03-28T22:34:02.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.352+0000 7f023f200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f024010f630 con 0x7f0240109500 2024-03-28T22:34:02.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.352+0000 7f023de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0240074130 0x7f0240199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:02.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.352+0000 7f023d400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0240075710 0x7f024019a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:02.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.352+0000 7f023de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0240074130 0x7f0240199b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34402/0 (socket says 172.21.15.43:34402) 2024-03-28T22:34:02.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.352+0000 7f023de00640 1 -- 172.21.15.43:0/3937445811 learned_addr learned my addr 172.21.15.43:0/3937445811 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:34:02.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.352+0000 7f023e800640 1 --2- 172.21.15.43:0/3937445811 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0240109500 0x7f02401a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:02.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.352+0000 7f023d400640 1 -- 172.21.15.43:0/3937445811 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0240109500 msgr2=0x7f02401a10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:02.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.352+0000 7f023d400640 1 --2- 172.21.15.43:0/3937445811 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0240109500 0x7f02401a10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:02.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.352+0000 7f023d400640 1 -- 172.21.15.43:0/3937445811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0240074130 msgr2=0x7f0240199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:02.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.352+0000 7f023d400640 1 --2- 172.21.15.43:0/3937445811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0240074130 0x7f0240199b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:02.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.352+0000 7f023d400640 1 -- 172.21.15.43:0/3937445811 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02401a1740 con 0x7f0240075710 2024-03-28T22:34:02.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.353+0000 7f023d400640 1 --2- 172.21.15.43:0/3937445811 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0240075710 0x7f024019a060 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0228005fd0 tx=0x7f022800dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:02.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.353+0000 7f0236a00640 1 -- 172.21.15.43:0/3937445811 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0228009c30 con 0x7f0240075710 2024-03-28T22:34:02.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.353+0000 7f0236a00640 1 -- 172.21.15.43:0/3937445811 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0228009dd0 con 0x7f0240075710 2024-03-28T22:34:02.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.353+0000 7f0236a00640 1 -- 172.21.15.43:0/3937445811 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0228010460 con 0x7f0240075710 2024-03-28T22:34:02.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.353+0000 7f023f200640 1 -- 172.21.15.43:0/3937445811 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02401a1a30 con 0x7f0240075710 2024-03-28T22:34:02.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.353+0000 7f023f200640 1 -- 172.21.15.43:0/3937445811 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02401a1f80 con 0x7f0240075710 2024-03-28T22:34:02.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.354+0000 7f023f200640 1 -- 172.21.15.43:0/3937445811 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02401082c0 con 0x7f0240075710 2024-03-28T22:34:02.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.356+0000 7f0236a00640 1 -- 172.21.15.43:0/3937445811 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f0228003f50 con 0x7f0240075710 2024-03-28T22:34:02.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.357+0000 7f0236a00640 1 --2- 172.21.15.43:0/3937445811 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f02240a1d80 0x7f02240a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:02.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.358+0000 7f0236a00640 1 -- 172.21.15.43:0/3937445811 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f02280cdfa0 con 0x7f0240075710 2024-03-28T22:34:02.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.358+0000 7f023de00640 1 --2- 172.21.15.43:0/3937445811 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f02240a1d80 0x7f02240a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:02.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.359+0000 7f023de00640 1 --2- 172.21.15.43:0/3937445811 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f02240a1d80 0x7f02240a4240 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0230006010 tx=0x7f0230004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:02.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.366+0000 7f0236a00640 1 -- 172.21.15.43:0/3937445811 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0228017020 con 0x7f0240075710 2024-03-28T22:34:02.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:02 smithi172 ceph-mon[36566]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-28T22:34:02.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:02 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T22:34:02.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:02 smithi172 ceph-mon[36566]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T22:34:02.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:02 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2824752917' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-28T22:34:02.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:02 smithi082 ceph-mon[36471]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-28T22:34:02.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:02 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T22:34:02.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:02 smithi082 ceph-mon[36471]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T22:34:02.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:02 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2824752917' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-28T22:34:02.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:02 smithi043 ceph-mon[31823]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-28T22:34:02.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:02 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T22:34:02.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:02 smithi043 ceph-mon[31823]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T22:34:02.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:02 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2824752917' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-28T22:34:02.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:02.615+0000 7f023f200640 1 -- 172.21.15.43:0/3937445811 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f024002d170 con 0x7f0240075710 2024-03-28T22:34:03.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.133+0000 7f0236a00640 1 -- 172.21.15.43:0/3937445811 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) ==== 121+0+0 (secure 0 0 0) 0x7f022809aae0 con 0x7f0240075710 2024-03-28T22:34:03.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.136+0000 7f023f200640 1 -- 172.21.15.43:0/3937445811 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f02240a1d80 msgr2=0x7f02240a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:03.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.137+0000 7f023f200640 1 --2- 172.21.15.43:0/3937445811 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f02240a1d80 0x7f02240a4240 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0230006010 tx=0x7f0230004a40 comp rx=0 tx=0).stop 2024-03-28T22:34:03.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.137+0000 7f023f200640 1 -- 172.21.15.43:0/3937445811 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0240075710 msgr2=0x7f024019a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:03.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.137+0000 7f023f200640 1 --2- 172.21.15.43:0/3937445811 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0240075710 0x7f024019a060 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0228005fd0 tx=0x7f022800dde0 comp rx=0 tx=0).stop 2024-03-28T22:34:03.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.137+0000 7f023f200640 1 -- 172.21.15.43:0/3937445811 shutdown_connections 2024-03-28T22:34:03.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.137+0000 7f023f200640 1 --2- 172.21.15.43:0/3937445811 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f02240a1d80 0x7f02240a4240 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:03.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.137+0000 7f023f200640 1 --2- 172.21.15.43:0/3937445811 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0240075710 0x7f024019a060 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:03.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.137+0000 7f023f200640 1 --2- 172.21.15.43:0/3937445811 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0240109500 0x7f02401a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:03.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.137+0000 7f023f200640 1 --2- 172.21.15.43:0/3937445811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0240074130 0x7f0240199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:03.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.137+0000 7f023f200640 1 -- 172.21.15.43:0/3937445811 >> 172.21.15.43:0/3937445811 conn(0x7f02400fcbc0 msgr2=0x7f0240105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:03.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.137+0000 7f023f200640 1 -- 172.21.15.43:0/3937445811 shutdown_connections 2024-03-28T22:34:03.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.137+0000 7f023f200640 1 -- 172.21.15.43:0/3937445811 wait complete. 2024-03-28T22:34:03.165 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-28T22:34:03.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.369+0000 7fa149800640 1 -- 172.21.15.43:0/2096664561 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa144104b80 msgr2=0x7fa144105000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:03.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.369+0000 7fa149800640 1 --2- 172.21.15.43:0/2096664561 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa144104b80 0x7fa144105000 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa12c009900 tx=0x7fa12c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:34:03.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.369+0000 7fa149800640 1 -- 172.21.15.43:0/2096664561 shutdown_connections 2024-03-28T22:34:03.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.369+0000 7fa149800640 1 --2- 172.21.15.43:0/2096664561 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa144105540 0x7fa14410bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:03.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.369+0000 7fa149800640 1 --2- 172.21.15.43:0/2096664561 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa144104b80 0x7fa144105000 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:03.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.369+0000 7fa149800640 1 --2- 172.21.15.43:0/2096664561 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa144103990 0x7fa144103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:03.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.369+0000 7fa149800640 1 -- 172.21.15.43:0/2096664561 >> 172.21.15.43:0/2096664561 conn(0x7fa1440fef00 msgr2=0x7fa144101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:03.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.369+0000 7fa149800640 1 -- 172.21.15.43:0/2096664561 shutdown_connections 2024-03-28T22:34:03.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.369+0000 7fa149800640 1 -- 172.21.15.43:0/2096664561 wait complete. 2024-03-28T22:34:03.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.370+0000 7fa149800640 1 Processor -- start 2024-03-28T22:34:03.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.371+0000 7fa149800640 1 -- start start 2024-03-28T22:34:03.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.371+0000 7fa149800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa144103990 0x7fa144199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:03.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.371+0000 7fa149800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa144104b80 0x7fa14419a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:03.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.371+0000 7fa149800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa144105540 0x7fa1441a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:03.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.371+0000 7fa149800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa14410f660 con 0x7fa144105540 2024-03-28T22:34:03.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.371+0000 7fa149800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa14410f4e0 con 0x7fa144103990 2024-03-28T22:34:03.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.371+0000 7fa149800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fa14410f7e0 con 0x7fa144104b80 2024-03-28T22:34:03.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.371+0000 7fa142a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa144103990 0x7fa144199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:03.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.371+0000 7fa13be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa144104b80 0x7fa14419a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:03.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.371+0000 7fa142a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa144103990 0x7fa144199cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:32924/0 (socket says 172.21.15.43:32924) 2024-03-28T22:34:03.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.371+0000 7fa142a00640 1 -- 172.21.15.43:0/3373349750 learned_addr learned my addr 172.21.15.43:0/3373349750 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:34:03.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.371+0000 7fa143400640 1 --2- 172.21.15.43:0/3373349750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa144105540 0x7fa1441a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:03.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.372+0000 7fa13be00640 1 -- 172.21.15.43:0/3373349750 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa144103990 msgr2=0x7fa144199cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:03.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.372+0000 7fa13be00640 1 --2- 172.21.15.43:0/3373349750 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa144103990 0x7fa144199cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:03.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.372+0000 7fa13be00640 1 -- 172.21.15.43:0/3373349750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa144105540 msgr2=0x7fa1441a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:03.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.372+0000 7fa13be00640 1 --2- 172.21.15.43:0/3373349750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa144105540 0x7fa1441a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:03.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.372+0000 7fa13be00640 1 -- 172.21.15.43:0/3373349750 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1441a1900 con 0x7fa144104b80 2024-03-28T22:34:03.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.372+0000 7fa143400640 1 --2- 172.21.15.43:0/3373349750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa144105540 0x7fa1441a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:34:03.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.372+0000 7fa13be00640 1 --2- 172.21.15.43:0/3373349750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa144104b80 0x7fa14419a230 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa12c036040 tx=0x7fa12c02c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:03.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.372+0000 7fa13b400640 1 -- 172.21.15.43:0/3373349750 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12c043020 con 0x7fa144104b80 2024-03-28T22:34:03.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.372+0000 7fa13b400640 1 -- 172.21.15.43:0/3373349750 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa12c0054e0 con 0x7fa144104b80 2024-03-28T22:34:03.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.372+0000 7fa13b400640 1 -- 172.21.15.43:0/3373349750 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12c003ca0 con 0x7fa144104b80 2024-03-28T22:34:03.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.372+0000 7fa149800640 1 -- 172.21.15.43:0/3373349750 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1441a1b90 con 0x7fa144104b80 2024-03-28T22:34:03.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.373+0000 7fa149800640 1 -- 172.21.15.43:0/3373349750 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1441a2100 con 0x7fa144104b80 2024-03-28T22:34:03.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.374+0000 7fa149800640 1 -- 172.21.15.43:0/3373349750 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa14410acb0 con 0x7fa144104b80 2024-03-28T22:34:03.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.376+0000 7fa13b400640 1 -- 172.21.15.43:0/3373349750 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fa12c046020 con 0x7fa144104b80 2024-03-28T22:34:03.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.377+0000 7fa13b400640 1 --2- 172.21.15.43:0/3373349750 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa1280a1e00 0x7fa1280a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:03.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.377+0000 7fa13b400640 1 -- 172.21.15.43:0/3373349750 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7fa12c0f0390 con 0x7fa144104b80 2024-03-28T22:34:03.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.377+0000 7fa142a00640 1 --2- 172.21.15.43:0/3373349750 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa1280a1e00 0x7fa1280a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:03.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.378+0000 7fa142a00640 1 --2- 172.21.15.43:0/3373349750 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa1280a1e00 0x7fa1280a42c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa14419b1c0 tx=0x7fa134005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:03.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.379+0000 7fa13b400640 1 -- 172.21.15.43:0/3373349750 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa12c04d020 con 0x7fa144104b80 2024-03-28T22:34:03.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:03 smithi172 ceph-mon[36566]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-28T22:34:03.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:03 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2824752917' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-28T22:34:03.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:03 smithi172 ceph-mon[36566]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-28T22:34:03.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:03 smithi172 ceph-mon[36566]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T22:34:03.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:03 smithi172 ceph-mon[36566]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-28T22:34:03.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:03 smithi172 ceph-mon[36566]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-28T22:34:03.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:03 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3937445811' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T22:34:03.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:03 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T22:34:03.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:03 smithi082 ceph-mon[36471]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-28T22:34:03.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:03 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2824752917' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-28T22:34:03.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:03 smithi082 ceph-mon[36471]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-28T22:34:03.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:03 smithi082 ceph-mon[36471]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T22:34:03.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:03 smithi082 ceph-mon[36471]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-28T22:34:03.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:03 smithi082 ceph-mon[36471]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-28T22:34:03.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:03 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3937445811' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T22:34:03.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:03 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T22:34:03.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:03 smithi043 ceph-mon[31823]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-28T22:34:03.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:03 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2824752917' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-28T22:34:03.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:03 smithi043 ceph-mon[31823]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-28T22:34:03.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:03 smithi043 ceph-mon[31823]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T22:34:03.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:03 smithi043 ceph-mon[31823]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-28T22:34:03.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:03 smithi043 ceph-mon[31823]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-28T22:34:03.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:03 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3937445811' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T22:34:03.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:03 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T22:34:03.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:03.626+0000 7fa149800640 1 -- 172.21.15.43:0/3373349750 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fa14402d170 con 0x7fa144104b80 2024-03-28T22:34:04.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.146+0000 7fa13b400640 1 -- 172.21.15.43:0/3373349750 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) ==== 115+0+0 (secure 0 0 0) 0x7fa12c0bd520 con 0x7fa144104b80 2024-03-28T22:34:04.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.150+0000 7fa149800640 1 -- 172.21.15.43:0/3373349750 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa1280a1e00 msgr2=0x7fa1280a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:04.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.150+0000 7fa149800640 1 --2- 172.21.15.43:0/3373349750 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa1280a1e00 0x7fa1280a42c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa14419b1c0 tx=0x7fa134005f90 comp rx=0 tx=0).stop 2024-03-28T22:34:04.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.150+0000 7fa149800640 1 -- 172.21.15.43:0/3373349750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa144104b80 msgr2=0x7fa14419a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:04.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.150+0000 7fa149800640 1 --2- 172.21.15.43:0/3373349750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa144104b80 0x7fa14419a230 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa12c036040 tx=0x7fa12c02c6c0 comp rx=0 tx=0).stop 2024-03-28T22:34:04.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.150+0000 7fa149800640 1 -- 172.21.15.43:0/3373349750 shutdown_connections 2024-03-28T22:34:04.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.150+0000 7fa149800640 1 --2- 172.21.15.43:0/3373349750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa144105540 0x7fa1441a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:04.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.150+0000 7fa149800640 1 --2- 172.21.15.43:0/3373349750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa144104b80 0x7fa14419a230 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:04.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.150+0000 7fa149800640 1 --2- 172.21.15.43:0/3373349750 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa1280a1e00 0x7fa1280a42c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:04.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.150+0000 7fa149800640 1 --2- 172.21.15.43:0/3373349750 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa144103990 0x7fa144199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:04.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.150+0000 7fa149800640 1 -- 172.21.15.43:0/3373349750 >> 172.21.15.43:0/3373349750 conn(0x7fa1440fef00 msgr2=0x7fa144107e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:04.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.151+0000 7fa149800640 1 -- 172.21.15.43:0/3373349750 shutdown_connections 2024-03-28T22:34:04.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.151+0000 7fa149800640 1 -- 172.21.15.43:0/3373349750 wait complete. 2024-03-28T22:34:04.177 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-03-28T22:34:04.177 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-28T22:34:04.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.382+0000 7ff9a9800640 1 -- 172.21.15.43:0/3101327222 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9a41049f0 msgr2=0x7ff9a4106de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:04.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.382+0000 7ff9a9800640 1 --2- 172.21.15.43:0/3101327222 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9a41049f0 0x7ff9a4106de0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff98c009900 tx=0x7ff98c02bc40 comp rx=0 tx=0).stop 2024-03-28T22:34:04.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.382+0000 7ff9a9800640 1 -- 172.21.15.43:0/3101327222 shutdown_connections 2024-03-28T22:34:04.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.382+0000 7ff9a9800640 1 --2- 172.21.15.43:0/3101327222 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9a4107320 0x7ff9a4109770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:04.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.382+0000 7ff9a9800640 1 --2- 172.21.15.43:0/3101327222 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9a41049f0 0x7ff9a4106de0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:04.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.382+0000 7ff9a9800640 1 --2- 172.21.15.43:0/3101327222 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9a41020c0 0x7ff9a41044b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:04.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.382+0000 7ff9a9800640 1 -- 172.21.15.43:0/3101327222 >> 172.21.15.43:0/3101327222 conn(0x7ff9a40fbad0 msgr2=0x7ff9a40fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:04.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.382+0000 7ff9a9800640 1 -- 172.21.15.43:0/3101327222 shutdown_connections 2024-03-28T22:34:04.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.382+0000 7ff9a9800640 1 -- 172.21.15.43:0/3101327222 wait complete. 2024-03-28T22:34:04.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.383+0000 7ff9a9800640 1 Processor -- start 2024-03-28T22:34:04.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.384+0000 7ff9a9800640 1 -- start start 2024-03-28T22:34:04.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.384+0000 7ff9a9800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9a41020c0 0x7ff9a4199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:04.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.384+0000 7ff9a9800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9a41049f0 0x7ff9a419a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:04.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.384+0000 7ff9a9800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9a4107320 0x7ff9a41a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:04.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.384+0000 7ff9a9800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a410d2b0 con 0x7ff9a41049f0 2024-03-28T22:34:04.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.384+0000 7ff9a9800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a410d130 con 0x7ff9a4107320 2024-03-28T22:34:04.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.384+0000 7ff9a9800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a410d430 con 0x7ff9a41020c0 2024-03-28T22:34:04.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.384+0000 7ff99b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9a41049f0 0x7ff9a419a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:04.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.385+0000 7ff9a8e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9a4107320 0x7ff9a41a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:04.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.385+0000 7ff99b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9a41049f0 0x7ff9a419a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34444/0 (socket says 172.21.15.43:34444) 2024-03-28T22:34:04.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.385+0000 7ff99b400640 1 -- 172.21.15.43:0/3172815852 learned_addr learned my addr 172.21.15.43:0/3172815852 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:34:04.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.385+0000 7ff9a8e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9a4107320 0x7ff9a41a1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:32934/0 (socket says 172.21.15.43:32934) 2024-03-28T22:34:04.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.385+0000 7ff9a3e00640 1 --2- 172.21.15.43:0/3172815852 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9a41020c0 0x7ff9a4199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:04.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.385+0000 7ff9a8e00640 1 -- 172.21.15.43:0/3172815852 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9a41020c0 msgr2=0x7ff9a4199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:04.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.385+0000 7ff9a8e00640 1 --2- 172.21.15.43:0/3172815852 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9a41020c0 0x7ff9a4199ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:04.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.385+0000 7ff9a8e00640 1 -- 172.21.15.43:0/3172815852 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9a41049f0 msgr2=0x7ff9a419a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:04.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.385+0000 7ff9a8e00640 1 --2- 172.21.15.43:0/3172815852 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9a41049f0 0x7ff9a419a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:04.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.385+0000 7ff9a8e00640 1 -- 172.21.15.43:0/3172815852 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a41a17c0 con 0x7ff9a4107320 2024-03-28T22:34:04.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.385+0000 7ff9a8e00640 1 --2- 172.21.15.43:0/3172815852 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9a4107320 0x7ff9a41a1150 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff99c009fd0 tx=0x7ff99c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:04.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.385+0000 7ff9a1600640 1 -- 172.21.15.43:0/3172815852 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff99c00ce20 con 0x7ff9a4107320 2024-03-28T22:34:04.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.386+0000 7ff9a1600640 1 -- 172.21.15.43:0/3172815852 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff99c005020 con 0x7ff9a4107320 2024-03-28T22:34:04.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.386+0000 7ff9a1600640 1 -- 172.21.15.43:0/3172815852 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff99c010720 con 0x7ff9a4107320 2024-03-28T22:34:04.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.386+0000 7ff9a9800640 1 -- 172.21.15.43:0/3172815852 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9a41a1ab0 con 0x7ff9a4107320 2024-03-28T22:34:04.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.386+0000 7ff9a9800640 1 -- 172.21.15.43:0/3172815852 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9a41a2020 con 0x7ff9a4107320 2024-03-28T22:34:04.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.387+0000 7ff9a9800640 1 -- 172.21.15.43:0/3172815852 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9a4108e70 con 0x7ff9a4107320 2024-03-28T22:34:04.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.389+0000 7ff9a1600640 1 -- 172.21.15.43:0/3172815852 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7ff99c005540 con 0x7ff9a4107320 2024-03-28T22:34:04.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.390+0000 7ff9a1600640 1 --2- 172.21.15.43:0/3172815852 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff9880a1e00 0x7ff9880a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:04.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.390+0000 7ff9a1600640 1 -- 172.21.15.43:0/3172815852 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7ff99c0ce610 con 0x7ff9a4107320 2024-03-28T22:34:04.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.390+0000 7ff9a3e00640 1 --2- 172.21.15.43:0/3172815852 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff9880a1e00 0x7ff9880a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:04.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.391+0000 7ff9a3e00640 1 --2- 172.21.15.43:0/3172815852 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff9880a1e00 0x7ff9880a42c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff99400bbe0 tx=0x7ff99400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:04.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:04 smithi172 ceph-mon[36566]: pgmap v339: 129 pgs: 129 active+clean; 581 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-28T22:34:04.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:04 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T22:34:04.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:04 smithi172 ceph-mon[36566]: mds.? [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] up:active 2024-03-28T22:34:04.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:04 smithi172 ceph-mon[36566]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-28T22:34:04.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:04 smithi172 ceph-mon[36566]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-28T22:34:04.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:04 smithi172 ceph-mon[36566]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-03-28T22:34:04.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:04 smithi172 ceph-mon[36566]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-28T22:34:04.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:04 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3373349750' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T22:34:04.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:04 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T22:34:04.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.398+0000 7ff9a1600640 1 -- 172.21.15.43:0/3172815852 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff99c09b7a0 con 0x7ff9a4107320 2024-03-28T22:34:04.449 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 28 22:34:04 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-b[67270]: 2024-03-28T22:34:04.321+0000 7fb4be5ee700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T22:34:04.449 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 28 22:34:04 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-b[67270]: 2024-03-28T22:34:04.321+0000 7fb4be5ee700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-28T22:34:04.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:04 smithi082 ceph-mon[36471]: pgmap v339: 129 pgs: 129 active+clean; 581 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-28T22:34:04.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:04 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T22:34:04.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:04 smithi082 ceph-mon[36471]: mds.? [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] up:active 2024-03-28T22:34:04.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:04 smithi082 ceph-mon[36471]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-28T22:34:04.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:04 smithi082 ceph-mon[36471]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-28T22:34:04.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:04 smithi082 ceph-mon[36471]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-03-28T22:34:04.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:04 smithi082 ceph-mon[36471]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-28T22:34:04.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:04 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3373349750' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T22:34:04.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:04 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T22:34:04.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:04 smithi043 ceph-mon[31823]: pgmap v339: 129 pgs: 129 active+clean; 581 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-28T22:34:04.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:04 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T22:34:04.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:04 smithi043 ceph-mon[31823]: mds.? [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] up:active 2024-03-28T22:34:04.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:04 smithi043 ceph-mon[31823]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-28T22:34:04.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:04 smithi043 ceph-mon[31823]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-28T22:34:04.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:04 smithi043 ceph-mon[31823]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-03-28T22:34:04.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:04 smithi043 ceph-mon[31823]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-28T22:34:04.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:04 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3373349750' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T22:34:04.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:04 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T22:34:04.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.617+0000 7ff9a9800640 1 -- 172.21.15.43:0/3172815852 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7ff9a4061b90 con 0x7ff9880a1e00 2024-03-28T22:34:04.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.646+0000 7ff9a1600640 1 -- 172.21.15.43:0/3172815852 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff9a4061b90 con 0x7ff9880a1e00 2024-03-28T22:34:04.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.649+0000 7ff9a9800640 1 -- 172.21.15.43:0/3172815852 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff9880a1e00 msgr2=0x7ff9880a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:04.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.649+0000 7ff9a9800640 1 --2- 172.21.15.43:0/3172815852 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff9880a1e00 0x7ff9880a42c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff99400bbe0 tx=0x7ff99400b390 comp rx=0 tx=0).stop 2024-03-28T22:34:04.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.649+0000 7ff9a9800640 1 -- 172.21.15.43:0/3172815852 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9a4107320 msgr2=0x7ff9a41a1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:04.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.649+0000 7ff9a9800640 1 --2- 172.21.15.43:0/3172815852 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9a4107320 0x7ff9a41a1150 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff99c009fd0 tx=0x7ff99c00ef10 comp rx=0 tx=0).stop 2024-03-28T22:34:04.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.649+0000 7ff9a9800640 1 -- 172.21.15.43:0/3172815852 shutdown_connections 2024-03-28T22:34:04.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.649+0000 7ff9a9800640 1 --2- 172.21.15.43:0/3172815852 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff9880a1e00 0x7ff9880a42c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:04.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.649+0000 7ff9a9800640 1 --2- 172.21.15.43:0/3172815852 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9a4107320 0x7ff9a41a1150 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:04.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.649+0000 7ff9a9800640 1 --2- 172.21.15.43:0/3172815852 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9a41049f0 0x7ff9a419a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:04.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.649+0000 7ff9a9800640 1 --2- 172.21.15.43:0/3172815852 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9a41020c0 0x7ff9a4199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:04.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.649+0000 7ff9a9800640 1 -- 172.21.15.43:0/3172815852 >> 172.21.15.43:0/3172815852 conn(0x7ff9a40fbad0 msgr2=0x7ff9a41018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:04.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.649+0000 7ff9a9800640 1 -- 172.21.15.43:0/3172815852 shutdown_connections 2024-03-28T22:34:04.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.649+0000 7ff9a9800640 1 -- 172.21.15.43:0/3172815852 wait complete. 2024-03-28T22:34:04.667 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-28T22:34:04.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.871+0000 7fb487400640 1 -- 172.21.15.43:0/53042667 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb4801089c0 msgr2=0x7fb48010adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:04.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.871+0000 7fb487400640 1 --2- 172.21.15.43:0/53042667 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb4801089c0 0x7fb48010adb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb470009900 tx=0x7fb47002bc30 comp rx=0 tx=0).stop 2024-03-28T22:34:04.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.871+0000 7fb487400640 1 -- 172.21.15.43:0/53042667 shutdown_connections 2024-03-28T22:34:04.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.871+0000 7fb487400640 1 --2- 172.21.15.43:0/53042667 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb48010de00 0x7fb4801102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:04.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.871+0000 7fb487400640 1 --2- 172.21.15.43:0/53042667 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb48010b380 0x7fb48010d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:04.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.871+0000 7fb487400640 1 --2- 172.21.15.43:0/53042667 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb4801089c0 0x7fb48010adb0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:04.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.871+0000 7fb487400640 1 -- 172.21.15.43:0/53042667 >> 172.21.15.43:0/53042667 conn(0x7fb48006e930 msgr2=0x7fb480070d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:04.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.871+0000 7fb487400640 1 -- 172.21.15.43:0/53042667 shutdown_connections 2024-03-28T22:34:04.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.871+0000 7fb487400640 1 -- 172.21.15.43:0/53042667 wait complete. 2024-03-28T22:34:04.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.872+0000 7fb487400640 1 Processor -- start 2024-03-28T22:34:04.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.872+0000 7fb487400640 1 -- start start 2024-03-28T22:34:04.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.872+0000 7fb487400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4801089c0 0x7fb4801a24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:04.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.872+0000 7fb487400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb48010b380 0x7fb4801a29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:04.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.872+0000 7fb487400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb48010de00 0x7fb4801a9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:04.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.872+0000 7fb487400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb4801139e0 con 0x7fb4801089c0 2024-03-28T22:34:04.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.872+0000 7fb487400640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb480113860 con 0x7fb48010de00 2024-03-28T22:34:04.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.872+0000 7fb487400640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fb480113b60 con 0x7fb48010b380 2024-03-28T22:34:04.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.873+0000 7fb47fe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb48010b380 0x7fb4801a29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:04.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.873+0000 7fb484a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4801089c0 0x7fb4801a24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:04.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.873+0000 7fb484a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4801089c0 0x7fb4801a24b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34502/0 (socket says 172.21.15.43:34502) 2024-03-28T22:34:04.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.873+0000 7fb484a00640 1 -- 172.21.15.43:0/4150153101 learned_addr learned my addr 172.21.15.43:0/4150153101 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:34:04.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.873+0000 7fb47fe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb48010b380 0x7fb4801a29f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:33518/0 (socket says 172.21.15.43:33518) 2024-03-28T22:34:04.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.873+0000 7fb485400640 1 --2- 172.21.15.43:0/4150153101 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb48010de00 0x7fb4801a9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:04.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.873+0000 7fb47fe00640 1 -- 172.21.15.43:0/4150153101 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb48010de00 msgr2=0x7fb4801a9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:04.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.873+0000 7fb47fe00640 1 --2- 172.21.15.43:0/4150153101 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb48010de00 0x7fb4801a9a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:04.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.873+0000 7fb47fe00640 1 -- 172.21.15.43:0/4150153101 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4801089c0 msgr2=0x7fb4801a24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:04.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.873+0000 7fb47fe00640 1 --2- 172.21.15.43:0/4150153101 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4801089c0 0x7fb4801a24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:04.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.873+0000 7fb47fe00640 1 -- 172.21.15.43:0/4150153101 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4801aa0d0 con 0x7fb48010b380 2024-03-28T22:34:04.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.873+0000 7fb484a00640 1 --2- 172.21.15.43:0/4150153101 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4801089c0 0x7fb4801a24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:34:04.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.873+0000 7fb485400640 1 --2- 172.21.15.43:0/4150153101 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb48010de00 0x7fb4801a9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:34:04.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.873+0000 7fb47fe00640 1 --2- 172.21.15.43:0/4150153101 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb48010b380 0x7fb4801a29f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb46c00afd0 tx=0x7fb46c00aac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:04.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.874+0000 7fb47d600640 1 -- 172.21.15.43:0/4150153101 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46c00c940 con 0x7fb48010b380 2024-03-28T22:34:04.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.874+0000 7fb47d600640 1 -- 172.21.15.43:0/4150153101 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb46c007450 con 0x7fb48010b380 2024-03-28T22:34:04.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.874+0000 7fb487400640 1 -- 172.21.15.43:0/4150153101 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4801aa3c0 con 0x7fb48010b380 2024-03-28T22:34:04.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.874+0000 7fb47d600640 1 -- 172.21.15.43:0/4150153101 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46c014630 con 0x7fb48010b380 2024-03-28T22:34:04.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.874+0000 7fb487400640 1 -- 172.21.15.43:0/4150153101 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4801aa960 con 0x7fb48010b380 2024-03-28T22:34:04.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.875+0000 7fb487400640 1 -- 172.21.15.43:0/4150153101 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb480070420 con 0x7fb48010b380 2024-03-28T22:34:04.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.878+0000 7fb47d600640 1 -- 172.21.15.43:0/4150153101 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fb46c007600 con 0x7fb48010b380 2024-03-28T22:34:04.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.879+0000 7fb47d600640 1 --2- 172.21.15.43:0/4150153101 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fb4640a1e00 0x7fb4640a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:04.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.879+0000 7fb47d600640 1 -- 172.21.15.43:0/4150153101 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7fb46c0cdb00 con 0x7fb48010b380 2024-03-28T22:34:04.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.879+0000 7fb484a00640 1 --2- 172.21.15.43:0/4150153101 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fb4640a1e00 0x7fb4640a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:04.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.880+0000 7fb484a00640 1 --2- 172.21.15.43:0/4150153101 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fb4640a1e00 0x7fb4640a42c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb470000c00 tx=0x7fb47002c880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:04.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:04.881+0000 7fb47d600640 1 -- 172.21.15.43:0/4150153101 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb46c011070 con 0x7fb48010b380 2024-03-28T22:34:05.017 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:34:04 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:34:04.626+0000 7f1d08400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:34:05.018 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:34:04 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:34:04.626+0000 7f1d08400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:34:05.018 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:34:04 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:34:04.626+0000 7f1d08400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:34:05.018 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:34:04 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:34:04.626+0000 7f1d08400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:34:05.018 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:34:04 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:34:04.626+0000 7f1d08400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:34:05.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.112+0000 7fb487400640 1 -- 172.21.15.43:0/4150153101 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fb480061c20 con 0x7fb4640a1e00 2024-03-28T22:34:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.150+0000 7fb47d600640 1 -- 172.21.15.43:0/4150153101 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb480061c20 con 0x7fb4640a1e00 2024-03-28T22:34:05.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.153+0000 7fb487400640 1 -- 172.21.15.43:0/4150153101 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fb4640a1e00 msgr2=0x7fb4640a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:05.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.153+0000 7fb487400640 1 --2- 172.21.15.43:0/4150153101 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fb4640a1e00 0x7fb4640a42c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb470000c00 tx=0x7fb47002c880 comp rx=0 tx=0).stop 2024-03-28T22:34:05.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.153+0000 7fb487400640 1 -- 172.21.15.43:0/4150153101 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb48010b380 msgr2=0x7fb4801a29f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:05.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.153+0000 7fb487400640 1 --2- 172.21.15.43:0/4150153101 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb48010b380 0x7fb4801a29f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb46c00afd0 tx=0x7fb46c00aac0 comp rx=0 tx=0).stop 2024-03-28T22:34:05.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.153+0000 7fb487400640 1 -- 172.21.15.43:0/4150153101 shutdown_connections 2024-03-28T22:34:05.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.153+0000 7fb487400640 1 --2- 172.21.15.43:0/4150153101 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fb4640a1e00 0x7fb4640a42c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:05.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.153+0000 7fb487400640 1 --2- 172.21.15.43:0/4150153101 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb48010de00 0x7fb4801a9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:05.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.153+0000 7fb487400640 1 --2- 172.21.15.43:0/4150153101 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb48010b380 0x7fb4801a29f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:05.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.153+0000 7fb487400640 1 --2- 172.21.15.43:0/4150153101 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4801089c0 0x7fb4801a24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:05.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.153+0000 7fb487400640 1 -- 172.21.15.43:0/4150153101 >> 172.21.15.43:0/4150153101 conn(0x7fb48006e930 msgr2=0x7fb480109280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:05.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.153+0000 7fb487400640 1 -- 172.21.15.43:0/4150153101 shutdown_connections 2024-03-28T22:34:05.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.153+0000 7fb487400640 1 -- 172.21.15.43:0/4150153101 wait complete. 2024-03-28T22:34:05.178 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-28T22:34:05.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.391+0000 7efcad400640 1 -- 172.21.15.43:0/784981994 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efca8103990 msgr2=0x7efca8103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:05.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.391+0000 7efcad400640 1 --2- 172.21.15.43:0/784981994 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efca8103990 0x7efca8103db0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7efc98009ca0 tx=0x7efc9802c0f0 comp rx=0 tx=0).stop 2024-03-28T22:34:05.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.391+0000 7efcad400640 1 -- 172.21.15.43:0/784981994 shutdown_connections 2024-03-28T22:34:05.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.391+0000 7efcad400640 1 --2- 172.21.15.43:0/784981994 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efca8105540 0x7efca810bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:05.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.391+0000 7efcad400640 1 --2- 172.21.15.43:0/784981994 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efca8104b80 0x7efca8105000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:05.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.391+0000 7efcad400640 1 --2- 172.21.15.43:0/784981994 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efca8103990 0x7efca8103db0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:05.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.391+0000 7efcad400640 1 -- 172.21.15.43:0/784981994 >> 172.21.15.43:0/784981994 conn(0x7efca80fef00 msgr2=0x7efca8101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:05.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.391+0000 7efcad400640 1 -- 172.21.15.43:0/784981994 shutdown_connections 2024-03-28T22:34:05.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.391+0000 7efcad400640 1 -- 172.21.15.43:0/784981994 wait complete. 2024-03-28T22:34:05.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.392+0000 7efcad400640 1 Processor -- start 2024-03-28T22:34:05.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.393+0000 7efcad400640 1 -- start start 2024-03-28T22:34:05.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.393+0000 7efcad400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efca8103990 0x7efca8199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:05.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.393+0000 7efcad400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efca8104b80 0x7efca819a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:05.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.393+0000 7efcad400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efca8105540 0x7efca81a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:05.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.393+0000 7efcad400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7efca810f5e0 con 0x7efca8104b80 2024-03-28T22:34:05.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.393+0000 7efcad400640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7efca810f460 con 0x7efca8103990 2024-03-28T22:34:05.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.393+0000 7efcad400640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7efca810f760 con 0x7efca8105540 2024-03-28T22:34:05.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.393+0000 7efca6800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efca8103990 0x7efca8199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:05.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.393+0000 7efca5e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efca8104b80 0x7efca819a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:05.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.393+0000 7efca6800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efca8103990 0x7efca8199cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:58368/0 (socket says 172.21.15.43:58368) 2024-03-28T22:34:05.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.393+0000 7efca6800640 1 -- 172.21.15.43:0/2490200149 learned_addr learned my addr 172.21.15.43:0/2490200149 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:34:05.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.394+0000 7efca7200640 1 --2- 172.21.15.43:0/2490200149 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efca8105540 0x7efca81a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:05.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.394+0000 7efca6800640 1 -- 172.21.15.43:0/2490200149 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efca8105540 msgr2=0x7efca81a1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:05.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.394+0000 7efca6800640 1 --2- 172.21.15.43:0/2490200149 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efca8105540 0x7efca81a1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:05.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.394+0000 7efca6800640 1 -- 172.21.15.43:0/2490200149 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efca8104b80 msgr2=0x7efca819a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:05.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.394+0000 7efca6800640 1 --2- 172.21.15.43:0/2490200149 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efca8104b80 0x7efca819a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:05.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.394+0000 7efca6800640 1 -- 172.21.15.43:0/2490200149 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efca81a18f0 con 0x7efca8103990 2024-03-28T22:34:05.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.394+0000 7efca6800640 1 --2- 172.21.15.43:0/2490200149 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efca8103990 0x7efca8199cb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7efc98000f80 tx=0x7efc98032ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:05.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.394+0000 7efc9f400640 1 -- 172.21.15.43:0/2490200149 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc98039040 con 0x7efca8103990 2024-03-28T22:34:05.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.394+0000 7efc9f400640 1 -- 172.21.15.43:0/2490200149 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc98044070 con 0x7efca8103990 2024-03-28T22:34:05.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.394+0000 7efc9f400640 1 -- 172.21.15.43:0/2490200149 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc980042a0 con 0x7efca8103990 2024-03-28T22:34:05.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.395+0000 7efcad400640 1 -- 172.21.15.43:0/2490200149 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efca81a1b80 con 0x7efca8103990 2024-03-28T22:34:05.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.395+0000 7efcad400640 1 -- 172.21.15.43:0/2490200149 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efca81a1ff0 con 0x7efca8103990 2024-03-28T22:34:05.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.396+0000 7efcad400640 1 -- 172.21.15.43:0/2490200149 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efca810ad50 con 0x7efca8103990 2024-03-28T22:34:05.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:05 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T22:34:05.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:05 smithi172 ceph-mon[36566]: mds.? [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] up:active 2024-03-28T22:34:05.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:05 smithi172 ceph-mon[36566]: daemon mds.f assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-28T22:34:05.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:05 smithi172 ceph-mon[36566]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-03-28T22:34:05.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:05 smithi172 ceph-mon[36566]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=f=up:creating} 8 up:standby 2024-03-28T22:34:05.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:05 smithi172 ceph-mon[36566]: pgmap v340: 129 pgs: 129 active+clean; 582 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s wr, 2 op/s 2024-03-28T22:34:05.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:05 smithi172 ceph-mon[36566]: daemon mds.f is now active in filesystem cephfs as rank 3 2024-03-28T22:34:05.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:05 smithi172 ceph-mon[36566]: from='client.24812 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:34:05.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:05 smithi172 ceph-mon[36566]: from='client.24841 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:34:05.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.398+0000 7efc9f400640 1 -- 172.21.15.43:0/2490200149 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7efc98003970 con 0x7efca8103990 2024-03-28T22:34:05.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.399+0000 7efc9f400640 1 --2- 172.21.15.43:0/2490200149 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7efc8c0a1d80 0x7efc8c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:05.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.399+0000 7efc9f400640 1 -- 172.21.15.43:0/2490200149 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7efc980f03b0 con 0x7efca8103990 2024-03-28T22:34:05.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.399+0000 7efca5e00640 1 --2- 172.21.15.43:0/2490200149 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7efc8c0a1d80 0x7efc8c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:05.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.400+0000 7efca5e00640 1 --2- 172.21.15.43:0/2490200149 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7efc8c0a1d80 0x7efc8c0a4240 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7efca819b1b0 tx=0x7efc9000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:05.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.403+0000 7efc9f400640 1 -- 172.21.15.43:0/2490200149 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7efc980bd540 con 0x7efca8103990 2024-03-28T22:34:05.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:05 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T22:34:05.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:05 smithi082 ceph-mon[36471]: mds.? [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] up:active 2024-03-28T22:34:05.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:05 smithi082 ceph-mon[36471]: daemon mds.f assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-28T22:34:05.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:05 smithi082 ceph-mon[36471]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-03-28T22:34:05.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:05 smithi082 ceph-mon[36471]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=f=up:creating} 8 up:standby 2024-03-28T22:34:05.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:05 smithi082 ceph-mon[36471]: pgmap v340: 129 pgs: 129 active+clean; 582 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s wr, 2 op/s 2024-03-28T22:34:05.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:05 smithi082 ceph-mon[36471]: daemon mds.f is now active in filesystem cephfs as rank 3 2024-03-28T22:34:05.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:05 smithi082 ceph-mon[36471]: from='client.24812 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:34:05.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:05 smithi082 ceph-mon[36471]: from='client.24841 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:34:05.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:05 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T22:34:05.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:05 smithi043 ceph-mon[31823]: mds.? [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] up:active 2024-03-28T22:34:05.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:05 smithi043 ceph-mon[31823]: daemon mds.f assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-28T22:34:05.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:05 smithi043 ceph-mon[31823]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-03-28T22:34:05.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:05 smithi043 ceph-mon[31823]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=f=up:creating} 8 up:standby 2024-03-28T22:34:05.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:05 smithi043 ceph-mon[31823]: pgmap v340: 129 pgs: 129 active+clean; 582 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s wr, 2 op/s 2024-03-28T22:34:05.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:05 smithi043 ceph-mon[31823]: daemon mds.f is now active in filesystem cephfs as rank 3 2024-03-28T22:34:05.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:05 smithi043 ceph-mon[31823]: from='client.24812 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:34:05.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:05 smithi043 ceph-mon[31823]: from='client.24841 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:34:05.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.637+0000 7efcad400640 1 -- 172.21.15.43:0/2490200149 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7efca8061c20 con 0x7efc8c0a1d80 2024-03-28T22:34:05.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.647+0000 7efc9f400640 1 -- 172.21.15.43:0/2490200149 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7efca8061c20 con 0x7efc8c0a1d80 2024-03-28T22:34:05.647 INFO:teuthology.orchestra.run.smithi043.stdout:/volumes/qa/sv_0/d432b182-b2c3-4c9b-90f5-5f6d8de0b49b 2024-03-28T22:34:05.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.651+0000 7efcad400640 1 -- 172.21.15.43:0/2490200149 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7efc8c0a1d80 msgr2=0x7efc8c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:05.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.651+0000 7efcad400640 1 --2- 172.21.15.43:0/2490200149 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7efc8c0a1d80 0x7efc8c0a4240 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7efca819b1b0 tx=0x7efc9000b390 comp rx=0 tx=0).stop 2024-03-28T22:34:05.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.651+0000 7efcad400640 1 -- 172.21.15.43:0/2490200149 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efca8103990 msgr2=0x7efca8199cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:05.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.651+0000 7efcad400640 1 --2- 172.21.15.43:0/2490200149 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efca8103990 0x7efca8199cb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7efc98000f80 tx=0x7efc98032ec0 comp rx=0 tx=0).stop 2024-03-28T22:34:05.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.651+0000 7efcad400640 1 -- 172.21.15.43:0/2490200149 shutdown_connections 2024-03-28T22:34:05.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.651+0000 7efcad400640 1 --2- 172.21.15.43:0/2490200149 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efca8105540 0x7efca81a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:05.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.651+0000 7efcad400640 1 --2- 172.21.15.43:0/2490200149 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efca8104b80 0x7efca819a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:05.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.651+0000 7efcad400640 1 --2- 172.21.15.43:0/2490200149 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7efc8c0a1d80 0x7efc8c0a4240 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:05.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.651+0000 7efcad400640 1 --2- 172.21.15.43:0/2490200149 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efca8103990 0x7efca8199cb0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:05.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.651+0000 7efcad400640 1 -- 172.21.15.43:0/2490200149 >> 172.21.15.43:0/2490200149 conn(0x7efca80fef00 msgr2=0x7efca8107f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:05.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.651+0000 7efcad400640 1 -- 172.21.15.43:0/2490200149 shutdown_connections 2024-03-28T22:34:05.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.651+0000 7efcad400640 1 -- 172.21.15.43:0/2490200149 wait complete. 2024-03-28T22:34:05.669 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-28T22:34:05.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.900+0000 7fe39d400640 1 -- 172.21.15.43:0/1046535716 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe398103960 msgr2=0x7fe398103d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:05.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.900+0000 7fe39d400640 1 --2- 172.21.15.43:0/1046535716 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe398103960 0x7fe398103d80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe388009900 tx=0x7fe38802bc30 comp rx=0 tx=0).stop 2024-03-28T22:34:05.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.900+0000 7fe39d400640 1 -- 172.21.15.43:0/1046535716 shutdown_connections 2024-03-28T22:34:05.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.900+0000 7fe39d400640 1 --2- 172.21.15.43:0/1046535716 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe398105510 0x7fe39810bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:05.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.900+0000 7fe39d400640 1 --2- 172.21.15.43:0/1046535716 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe398104b50 0x7fe398104fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:05.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.900+0000 7fe39d400640 1 --2- 172.21.15.43:0/1046535716 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe398103960 0x7fe398103d80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:05.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.900+0000 7fe39d400640 1 -- 172.21.15.43:0/1046535716 >> 172.21.15.43:0/1046535716 conn(0x7fe3980fef10 msgr2=0x7fe398101330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:05.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.900+0000 7fe39d400640 1 -- 172.21.15.43:0/1046535716 shutdown_connections 2024-03-28T22:34:05.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.900+0000 7fe39d400640 1 -- 172.21.15.43:0/1046535716 wait complete. 2024-03-28T22:34:05.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.901+0000 7fe39d400640 1 Processor -- start 2024-03-28T22:34:05.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.901+0000 7fe39d400640 1 -- start start 2024-03-28T22:34:05.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.902+0000 7fe39d400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe398104b50 0x7fe398199e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:05.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.902+0000 7fe39d400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe398105510 0x7fe39819a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:05.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.902+0000 7fe39d400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe39819f430 0x7fe3981a1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:05.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.902+0000 7fe39d400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe39810f600 con 0x7fe39819f430 2024-03-28T22:34:05.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.902+0000 7fe39d400640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe39810f480 con 0x7fe398105510 2024-03-28T22:34:05.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.902+0000 7fe39d400640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe39810f780 con 0x7fe398104b50 2024-03-28T22:34:05.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.902+0000 7fe396800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe398104b50 0x7fe398199e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:05.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.902+0000 7fe395e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe398105510 0x7fe39819a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:05.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.902+0000 7fe396800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe398104b50 0x7fe398199e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:49162/0 (socket says 172.21.15.43:49162) 2024-03-28T22:34:05.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.902+0000 7fe396800640 1 -- 172.21.15.43:0/2788110847 learned_addr learned my addr 172.21.15.43:0/2788110847 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:34:05.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.902+0000 7fe397200640 1 --2- 172.21.15.43:0/2788110847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe39819f430 0x7fe3981a1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:05.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.903+0000 7fe395e00640 1 -- 172.21.15.43:0/2788110847 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe398104b50 msgr2=0x7fe398199e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:05.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.903+0000 7fe395e00640 1 --2- 172.21.15.43:0/2788110847 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe398104b50 0x7fe398199e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:05.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.903+0000 7fe395e00640 1 -- 172.21.15.43:0/2788110847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe39819f430 msgr2=0x7fe3981a1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:05.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.903+0000 7fe395e00640 1 --2- 172.21.15.43:0/2788110847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe39819f430 0x7fe3981a1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:05.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.903+0000 7fe395e00640 1 -- 172.21.15.43:0/2788110847 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3981a1e90 con 0x7fe398105510 2024-03-28T22:34:05.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.903+0000 7fe395e00640 1 --2- 172.21.15.43:0/2788110847 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe398105510 0x7fe39819a3d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe38400e9f0 tx=0x7fe38400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:05.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.903+0000 7fe38f400640 1 -- 172.21.15.43:0/2788110847 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe38400ccc0 con 0x7fe398105510 2024-03-28T22:34:05.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.903+0000 7fe39d400640 1 -- 172.21.15.43:0/2788110847 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3981a2180 con 0x7fe398105510 2024-03-28T22:34:05.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.903+0000 7fe38f400640 1 -- 172.21.15.43:0/2788110847 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe384010910 con 0x7fe398105510 2024-03-28T22:34:05.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.903+0000 7fe38f400640 1 -- 172.21.15.43:0/2788110847 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe384018790 con 0x7fe398105510 2024-03-28T22:34:05.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.903+0000 7fe39d400640 1 -- 172.21.15.43:0/2788110847 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3981a2720 con 0x7fe398105510 2024-03-28T22:34:05.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.904+0000 7fe39d400640 1 -- 172.21.15.43:0/2788110847 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe39810ad50 con 0x7fe398105510 2024-03-28T22:34:05.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.907+0000 7fe38f400640 1 -- 172.21.15.43:0/2788110847 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7fe384010ac0 con 0x7fe398105510 2024-03-28T22:34:05.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.908+0000 7fe38f400640 1 --2- 172.21.15.43:0/2788110847 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe37c0a1d80 0x7fe37c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:05.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.909+0000 7fe38f400640 1 -- 172.21.15.43:0/2788110847 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7fe3840ce3e0 con 0x7fe398105510 2024-03-28T22:34:05.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.909+0000 7fe396800640 1 --2- 172.21.15.43:0/2788110847 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe37c0a1d80 0x7fe37c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:05.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.909+0000 7fe396800640 1 --2- 172.21.15.43:0/2788110847 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe37c0a1d80 0x7fe37c0a4240 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe388000c00 tx=0x7fe38802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:05.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:05.912+0000 7fe38f400640 1 -- 172.21.15.43:0/2788110847 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe38409b570 con 0x7fe398105510 2024-03-28T22:34:06.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.150+0000 7fe39d400640 1 -- 172.21.15.43:0/2788110847 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fe398061c20 con 0x7fe37c0a1d80 2024-03-28T22:34:06.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.192+0000 7fe38f400640 1 -- 172.21.15.43:0/2788110847 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fe398061c20 con 0x7fe37c0a1d80 2024-03-28T22:34:06.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.194+0000 7fe39d400640 1 -- 172.21.15.43:0/2788110847 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe37c0a1d80 msgr2=0x7fe37c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:06.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.194+0000 7fe39d400640 1 --2- 172.21.15.43:0/2788110847 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe37c0a1d80 0x7fe37c0a4240 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe388000c00 tx=0x7fe38802df90 comp rx=0 tx=0).stop 2024-03-28T22:34:06.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.194+0000 7fe39d400640 1 -- 172.21.15.43:0/2788110847 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe398105510 msgr2=0x7fe39819a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:06.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.194+0000 7fe39d400640 1 --2- 172.21.15.43:0/2788110847 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe398105510 0x7fe39819a3d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe38400e9f0 tx=0x7fe38400edf0 comp rx=0 tx=0).stop 2024-03-28T22:34:06.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.194+0000 7fe39d400640 1 -- 172.21.15.43:0/2788110847 shutdown_connections 2024-03-28T22:34:06.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.194+0000 7fe39d400640 1 --2- 172.21.15.43:0/2788110847 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe37c0a1d80 0x7fe37c0a4240 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:06.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.194+0000 7fe39d400640 1 --2- 172.21.15.43:0/2788110847 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe398105510 0x7fe39819a3d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:06.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.194+0000 7fe39d400640 1 --2- 172.21.15.43:0/2788110847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe39819f430 0x7fe3981a1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:06.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.194+0000 7fe39d400640 1 --2- 172.21.15.43:0/2788110847 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe398104b50 0x7fe398199e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:06.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.195+0000 7fe39d400640 1 -- 172.21.15.43:0/2788110847 >> 172.21.15.43:0/2788110847 conn(0x7fe3980fef10 msgr2=0x7fe398107f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:06.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.195+0000 7fe39d400640 1 -- 172.21.15.43:0/2788110847 shutdown_connections 2024-03-28T22:34:06.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.195+0000 7fe39d400640 1 -- 172.21.15.43:0/2788110847 wait complete. 2024-03-28T22:34:06.212 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-28T22:34:06.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.419+0000 7f784d600640 1 -- 172.21.15.43:0/3221687272 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78481009a0 msgr2=0x7f7848100e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:06.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.419+0000 7f784d600640 1 --2- 172.21.15.43:0/3221687272 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78481009a0 0x7f7848100e00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7834009900 tx=0x7f783402bcd0 comp rx=0 tx=0).stop 2024-03-28T22:34:06.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.419+0000 7f784d600640 1 -- 172.21.15.43:0/3221687272 shutdown_connections 2024-03-28T22:34:06.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.419+0000 7f784d600640 1 --2- 172.21.15.43:0/3221687272 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7848101340 0x7f78481017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:06.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.419+0000 7f784d600640 1 --2- 172.21.15.43:0/3221687272 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78481009a0 0x7f7848100e00 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:06.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.419+0000 7f784d600640 1 --2- 172.21.15.43:0/3221687272 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7848100040 0x7f7848100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:06.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.419+0000 7f784d600640 1 -- 172.21.15.43:0/3221687272 >> 172.21.15.43:0/3221687272 conn(0x7f78480fbbe0 msgr2=0x7f78480fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:06.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.419+0000 7f784d600640 1 -- 172.21.15.43:0/3221687272 shutdown_connections 2024-03-28T22:34:06.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.419+0000 7f784d600640 1 -- 172.21.15.43:0/3221687272 wait complete. 2024-03-28T22:34:06.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.420+0000 7f784d600640 1 Processor -- start 2024-03-28T22:34:06.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.420+0000 7f784d600640 1 -- start start 2024-03-28T22:34:06.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.421+0000 7f784d600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7848100040 0x7f7848199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:06.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.421+0000 7f784d600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78481009a0 0x7f784819a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:06.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.421+0000 7f784d600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7848101340 0x7f78481a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:06.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.421+0000 7f784d600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f784810f600 con 0x7f7848101340 2024-03-28T22:34:06.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.421+0000 7f784d600640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f784810f480 con 0x7f78481009a0 2024-03-28T22:34:06.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.421+0000 7f784d600640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f784810f780 con 0x7f7848100040 2024-03-28T22:34:06.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.421+0000 7f7846a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7848100040 0x7f7848199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:06.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.421+0000 7f7846a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7848100040 0x7f7848199cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:49176/0 (socket says 172.21.15.43:49176) 2024-03-28T22:34:06.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.421+0000 7f7846a00640 1 -- 172.21.15.43:0/4089628533 learned_addr learned my addr 172.21.15.43:0/4089628533 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:34:06.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.421+0000 7f7846000640 1 --2- 172.21.15.43:0/4089628533 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78481009a0 0x7f784819a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:06.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.421+0000 7f7847400640 1 --2- 172.21.15.43:0/4089628533 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7848101340 0x7f78481a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:06.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.421+0000 7f7846a00640 1 -- 172.21.15.43:0/4089628533 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78481009a0 msgr2=0x7f784819a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:06.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.421+0000 7f7846a00640 1 --2- 172.21.15.43:0/4089628533 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78481009a0 0x7f784819a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:06.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.421+0000 7f7846a00640 1 -- 172.21.15.43:0/4089628533 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7848101340 msgr2=0x7f78481a1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:06.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.421+0000 7f7846a00640 1 --2- 172.21.15.43:0/4089628533 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7848101340 0x7f78481a1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:06.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.421+0000 7f7846a00640 1 -- 172.21.15.43:0/4089628533 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78481a18f0 con 0x7f7848100040 2024-03-28T22:34:06.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.422+0000 7f7846a00640 1 --2- 172.21.15.43:0/4089628533 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7848100040 0x7f7848199cb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f783800e9f0 tx=0x7f783800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:06.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.422+0000 7f783f400640 1 -- 172.21.15.43:0/4089628533 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f783800cd80 con 0x7f7848100040 2024-03-28T22:34:06.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.422+0000 7f783f400640 1 -- 172.21.15.43:0/4089628533 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7838010910 con 0x7f7848100040 2024-03-28T22:34:06.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.422+0000 7f784d600640 1 -- 172.21.15.43:0/4089628533 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78481a1be0 con 0x7f7848100040 2024-03-28T22:34:06.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.422+0000 7f783f400640 1 -- 172.21.15.43:0/4089628533 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7838018990 con 0x7f7848100040 2024-03-28T22:34:06.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.422+0000 7f784d600640 1 -- 172.21.15.43:0/4089628533 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78481a2100 con 0x7f7848100040 2024-03-28T22:34:06.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.423+0000 7f784d600640 1 -- 172.21.15.43:0/4089628533 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f784810af10 con 0x7f7848100040 2024-03-28T22:34:06.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.425+0000 7f783f400640 1 -- 172.21.15.43:0/4089628533 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f7838010ac0 con 0x7f7848100040 2024-03-28T22:34:06.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.427+0000 7f783f400640 1 --2- 172.21.15.43:0/4089628533 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f782c0a1d80 0x7f782c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:06.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.427+0000 7f783f400640 1 -- 172.21.15.43:0/4089628533 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f78380ce770 con 0x7f7848100040 2024-03-28T22:34:06.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.427+0000 7f7846000640 1 --2- 172.21.15.43:0/4089628533 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f782c0a1d80 0x7f782c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:06.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.428+0000 7f7846000640 1 --2- 172.21.15.43:0/4089628533 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f782c0a1d80 0x7f782c0a4240 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f783402c2e0 tx=0x7f783402c920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:06.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.429+0000 7f783f400640 1 -- 172.21.15.43:0/4089628533 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f783809b2b0 con 0x7f7848100040 2024-03-28T22:34:06.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:06 smithi082 ceph-mon[36471]: mds.? [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] up:active 2024-03-28T22:34:06.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:06 smithi082 ceph-mon[36471]: daemon mds.h assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-28T22:34:06.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:06 smithi082 ceph-mon[36471]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-28T22:34:06.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:06 smithi082 ceph-mon[36471]: Cluster is now healthy 2024-03-28T22:34:06.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:06 smithi082 ceph-mon[36471]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=f=up:active} 8 up:standby 2024-03-28T22:34:06.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:06 smithi082 ceph-mon[36471]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-28T22:34:06.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:06 smithi082 ceph-mon[36471]: daemon mds.h is now active in filesystem cephfs as rank 4 2024-03-28T22:34:06.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:06 smithi082 ceph-mon[36471]: from='client.24830 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:34:06.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:06 smithi043 ceph-mon[31823]: mds.? [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] up:active 2024-03-28T22:34:06.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:06 smithi043 ceph-mon[31823]: daemon mds.h assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-28T22:34:06.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:06 smithi043 ceph-mon[31823]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-28T22:34:06.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:06 smithi043 ceph-mon[31823]: Cluster is now healthy 2024-03-28T22:34:06.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:06 smithi043 ceph-mon[31823]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=f=up:active} 8 up:standby 2024-03-28T22:34:06.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:06 smithi043 ceph-mon[31823]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-28T22:34:06.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:06 smithi043 ceph-mon[31823]: daemon mds.h is now active in filesystem cephfs as rank 4 2024-03-28T22:34:06.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:06 smithi043 ceph-mon[31823]: from='client.24830 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:34:06.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:06 smithi172 ceph-mon[36566]: mds.? [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] up:active 2024-03-28T22:34:06.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:06 smithi172 ceph-mon[36566]: daemon mds.h assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-28T22:34:06.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:06 smithi172 ceph-mon[36566]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-28T22:34:06.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:06 smithi172 ceph-mon[36566]: Cluster is now healthy 2024-03-28T22:34:06.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:06 smithi172 ceph-mon[36566]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=f=up:active} 8 up:standby 2024-03-28T22:34:06.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:06 smithi172 ceph-mon[36566]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-28T22:34:06.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:06 smithi172 ceph-mon[36566]: daemon mds.h is now active in filesystem cephfs as rank 4 2024-03-28T22:34:06.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:06 smithi172 ceph-mon[36566]: from='client.24830 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:34:06.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.665+0000 7f784d600640 1 -- 172.21.15.43:0/4089628533 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f7848061b90 con 0x7f782c0a1d80 2024-03-28T22:34:06.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.673+0000 7f783f400640 1 -- 172.21.15.43:0/4089628533 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f7848061b90 con 0x7f782c0a1d80 2024-03-28T22:34:06.672 INFO:teuthology.orchestra.run.smithi043.stdout:/volumes/qa/sv_1/0f44e03b-e9f6-4a20-9eab-e6caac9e0703 2024-03-28T22:34:06.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.675+0000 7f784d600640 1 -- 172.21.15.43:0/4089628533 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f782c0a1d80 msgr2=0x7f782c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:06.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.675+0000 7f784d600640 1 --2- 172.21.15.43:0/4089628533 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f782c0a1d80 0x7f782c0a4240 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f783402c2e0 tx=0x7f783402c920 comp rx=0 tx=0).stop 2024-03-28T22:34:06.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.675+0000 7f784d600640 1 -- 172.21.15.43:0/4089628533 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7848100040 msgr2=0x7f7848199cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:06.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.675+0000 7f784d600640 1 --2- 172.21.15.43:0/4089628533 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7848100040 0x7f7848199cb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f783800e9f0 tx=0x7f783800eeb0 comp rx=0 tx=0).stop 2024-03-28T22:34:06.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.676+0000 7f784d600640 1 -- 172.21.15.43:0/4089628533 shutdown_connections 2024-03-28T22:34:06.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.676+0000 7f784d600640 1 --2- 172.21.15.43:0/4089628533 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f782c0a1d80 0x7f782c0a4240 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:06.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.676+0000 7f784d600640 1 --2- 172.21.15.43:0/4089628533 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78481009a0 0x7f784819a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:06.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.676+0000 7f784d600640 1 --2- 172.21.15.43:0/4089628533 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7848101340 0x7f78481a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:06.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.676+0000 7f784d600640 1 --2- 172.21.15.43:0/4089628533 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7848100040 0x7f7848199cb0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:06.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.676+0000 7f784d600640 1 -- 172.21.15.43:0/4089628533 >> 172.21.15.43:0/4089628533 conn(0x7f78480fbbe0 msgr2=0x7f78480fd2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:06.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.676+0000 7f784d600640 1 -- 172.21.15.43:0/4089628533 shutdown_connections 2024-03-28T22:34:06.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.676+0000 7f784d600640 1 -- 172.21.15.43:0/4089628533 wait complete. 2024-03-28T22:34:06.695 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:34:06.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.922+0000 7f8496c00640 1 -- 172.21.15.43:0/2951338126 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8490105350 msgr2=0x7f849010bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:06.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.922+0000 7f8496c00640 1 --2- 172.21.15.43:0/2951338126 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8490105350 0x7f849010bbe0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8488009900 tx=0x7f848802bc30 comp rx=0 tx=0).stop 2024-03-28T22:34:06.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.922+0000 7f8496c00640 1 -- 172.21.15.43:0/2951338126 shutdown_connections 2024-03-28T22:34:06.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.922+0000 7f8496c00640 1 --2- 172.21.15.43:0/2951338126 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8490104990 0x7f8490104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:06.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.922+0000 7f8496c00640 1 --2- 172.21.15.43:0/2951338126 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8490105350 0x7f849010bbe0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:06.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.922+0000 7f8496c00640 1 --2- 172.21.15.43:0/2951338126 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f84901037a0 0x7f8490103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:06.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.922+0000 7f8496c00640 1 -- 172.21.15.43:0/2951338126 >> 172.21.15.43:0/2951338126 conn(0x7f84900fed30 msgr2=0x7f8490101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:06.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.922+0000 7f8496c00640 1 -- 172.21.15.43:0/2951338126 shutdown_connections 2024-03-28T22:34:06.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.922+0000 7f8496c00640 1 -- 172.21.15.43:0/2951338126 wait complete. 2024-03-28T22:34:06.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.923+0000 7f8496c00640 1 Processor -- start 2024-03-28T22:34:06.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.923+0000 7f8496c00640 1 -- start start 2024-03-28T22:34:06.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.924+0000 7f8496c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f84901037a0 0x7f84901999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:06.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.924+0000 7f8496c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8490104990 0x7f8490199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:06.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.924+0000 7f8496c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8490105350 0x7f84901a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:06.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.924+0000 7f8496c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f849010f400 con 0x7f8490104990 2024-03-28T22:34:06.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.924+0000 7f8496c00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f849010f280 con 0x7f84901037a0 2024-03-28T22:34:06.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.924+0000 7f8496c00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f849010f580 con 0x7f8490105350 2024-03-28T22:34:06.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.924+0000 7f8495800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f84901037a0 0x7f84901999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:06.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.924+0000 7f8495800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f84901037a0 0x7f84901999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:58420/0 (socket says 172.21.15.43:58420) 2024-03-28T22:34:06.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.924+0000 7f8495800640 1 -- 172.21.15.43:0/2868822561 learned_addr learned my addr 172.21.15.43:0/2868822561 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:34:06.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.924+0000 7f8494e00640 1 --2- 172.21.15.43:0/2868822561 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8490104990 0x7f8490199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:06.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.924+0000 7f8496200640 1 --2- 172.21.15.43:0/2868822561 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8490105350 0x7f84901a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:06.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.925+0000 7f8495800640 1 -- 172.21.15.43:0/2868822561 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8490105350 msgr2=0x7f84901a0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:06.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.925+0000 7f8495800640 1 --2- 172.21.15.43:0/2868822561 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8490105350 0x7f84901a0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:06.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.925+0000 7f8495800640 1 -- 172.21.15.43:0/2868822561 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8490104990 msgr2=0x7f8490199f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:06.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.925+0000 7f8495800640 1 --2- 172.21.15.43:0/2868822561 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8490104990 0x7f8490199f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:06.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.925+0000 7f8495800640 1 -- 172.21.15.43:0/2868822561 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84901a1600 con 0x7f84901037a0 2024-03-28T22:34:06.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.925+0000 7f8495800640 1 --2- 172.21.15.43:0/2868822561 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f84901037a0 0x7f84901999e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f848400ebc0 tx=0x7f848400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:06.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.925+0000 7f848e000640 1 -- 172.21.15.43:0/2868822561 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8484003b70 con 0x7f84901037a0 2024-03-28T22:34:06.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.925+0000 7f8496c00640 1 -- 172.21.15.43:0/2868822561 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84901a18f0 con 0x7f84901037a0 2024-03-28T22:34:06.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.925+0000 7f848e000640 1 -- 172.21.15.43:0/2868822561 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8484003d10 con 0x7f84901037a0 2024-03-28T22:34:06.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.925+0000 7f848e000640 1 -- 172.21.15.43:0/2868822561 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8484019980 con 0x7f84901037a0 2024-03-28T22:34:06.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.925+0000 7f8496c00640 1 -- 172.21.15.43:0/2868822561 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84901a1e40 con 0x7f84901037a0 2024-03-28T22:34:06.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.927+0000 7f8496c00640 1 -- 172.21.15.43:0/2868822561 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84900688b0 con 0x7f84901037a0 2024-03-28T22:34:06.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.928+0000 7f848e000640 1 -- 172.21.15.43:0/2868822561 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 136756+0+0 (secure 0 0 0) 0x7f8484019b20 con 0x7f84901037a0 2024-03-28T22:34:06.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.930+0000 7f848e000640 1 --2- 172.21.15.43:0/2868822561 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f84780a1d80 0x7f84780a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:06.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.930+0000 7f848e000640 1 -- 172.21.15.43:0/2868822561 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f84840ce850 con 0x7f84901037a0 2024-03-28T22:34:06.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.930+0000 7f8494e00640 1 --2- 172.21.15.43:0/2868822561 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f84780a1d80 0x7f84780a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:06.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.931+0000 7f8494e00640 1 --2- 172.21.15.43:0/2868822561 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f84780a1d80 0x7f84780a4240 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f847c0095f0 tx=0x7f847c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:06.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:06.936+0000 7f848e000640 1 -- 172.21.15.43:0/2868822561 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f848409b9e0 con 0x7f84901037a0 2024-03-28T22:34:07.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.181+0000 7f8496c00640 1 -- 172.21.15.43:0/2868822561 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f849010aac0 con 0x7f84901037a0 2024-03-28T22:34:07.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.181+0000 7f848e000640 1 -- 172.21.15.43:0/2868822561 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11934 (secure 0 0 0) 0x7f849010aac0 con 0x7f84901037a0 2024-03-28T22:34:07.181 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:34:07.181 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:33:28.906396+0000","modified":"2024-03-28T22:34:06.174761+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24572,"mds_1":24556,"mds_2":24535,"mds_3":24523,"mds_4":24518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/1084988072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1084988072},{"type":"v1","addr":"172.21.15.82:6835","nonce":1084988072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24535,24556,24518,24523,24572]},"id":1}]} 2024-03-28T22:34:07.181 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 28 2024-03-28T22:34:07.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.184+0000 7f8496c00640 1 -- 172.21.15.43:0/2868822561 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f84780a1d80 msgr2=0x7f84780a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:07.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.184+0000 7f8496c00640 1 --2- 172.21.15.43:0/2868822561 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f84780a1d80 0x7f84780a4240 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f847c0095f0 tx=0x7f847c005f90 comp rx=0 tx=0).stop 2024-03-28T22:34:07.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.184+0000 7f8496c00640 1 -- 172.21.15.43:0/2868822561 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f84901037a0 msgr2=0x7f84901999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:07.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.184+0000 7f8496c00640 1 --2- 172.21.15.43:0/2868822561 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f84901037a0 0x7f84901999e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f848400ebc0 tx=0x7f848400c6a0 comp rx=0 tx=0).stop 2024-03-28T22:34:07.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.184+0000 7f8496c00640 1 -- 172.21.15.43:0/2868822561 shutdown_connections 2024-03-28T22:34:07.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.184+0000 7f8496c00640 1 --2- 172.21.15.43:0/2868822561 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8490105350 0x7f84901a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:07.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.184+0000 7f8496c00640 1 --2- 172.21.15.43:0/2868822561 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8490104990 0x7f8490199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:07.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.184+0000 7f8496c00640 1 --2- 172.21.15.43:0/2868822561 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f84780a1d80 0x7f84780a4240 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:07.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.184+0000 7f8496c00640 1 --2- 172.21.15.43:0/2868822561 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f84901037a0 0x7f84901999e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:07.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.184+0000 7f8496c00640 1 -- 172.21.15.43:0/2868822561 >> 172.21.15.43:0/2868822561 conn(0x7f84900fed30 msgr2=0x7f8490107c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:07.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.185+0000 7f8496c00640 1 -- 172.21.15.43:0/2868822561 shutdown_connections 2024-03-28T22:34:07.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.185+0000 7f8496c00640 1 -- 172.21.15.43:0/2868822561 wait complete. 2024-03-28T22:34:07.209 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:34:07.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.414+0000 7fda60c00640 1 -- 172.21.15.43:0/2705184373 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda5c0ffe40 msgr2=0x7fda5c106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:07.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.414+0000 7fda60c00640 1 --2- 172.21.15.43:0/2705184373 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda5c0ffe40 0x7fda5c106e40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fda48009900 tx=0x7fda4802bc30 comp rx=0 tx=0).stop 2024-03-28T22:34:07.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.414+0000 7fda60c00640 1 -- 172.21.15.43:0/2705184373 shutdown_connections 2024-03-28T22:34:07.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.414+0000 7fda60c00640 1 --2- 172.21.15.43:0/2705184373 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda5c1073d0 0x7fda5c109890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:07.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.414+0000 7fda60c00640 1 --2- 172.21.15.43:0/2705184373 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda5c0ffe40 0x7fda5c106e40 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:07.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.414+0000 7fda60c00640 1 --2- 172.21.15.43:0/2705184373 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda5c0ff4e0 0x7fda5c0ff900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:07.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.414+0000 7fda60c00640 1 -- 172.21.15.43:0/2705184373 >> 172.21.15.43:0/2705184373 conn(0x7fda5c0faf50 msgr2=0x7fda5c0fd370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:07.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.415+0000 7fda60c00640 1 -- 172.21.15.43:0/2705184373 shutdown_connections 2024-03-28T22:34:07.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.415+0000 7fda60c00640 1 -- 172.21.15.43:0/2705184373 wait complete. 2024-03-28T22:34:07.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.416+0000 7fda60c00640 1 Processor -- start 2024-03-28T22:34:07.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.416+0000 7fda60c00640 1 -- start start 2024-03-28T22:34:07.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.416+0000 7fda60c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda5c0ff4e0 0x7fda5c102740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:07.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.416+0000 7fda60c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda5c0ffe40 0x7fda5c100dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:07.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.416+0000 7fda60c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda5c1073d0 0x7fda5c101310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:07.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.416+0000 7fda60c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fda5c10d400 con 0x7fda5c0ffe40 2024-03-28T22:34:07.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.416+0000 7fda60c00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fda5c10d280 con 0x7fda5c0ff4e0 2024-03-28T22:34:07.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.416+0000 7fda60c00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fda5c10d580 con 0x7fda5c1073d0 2024-03-28T22:34:07.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.417+0000 7fda59e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda5c0ff4e0 0x7fda5c102740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:07.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.417+0000 7fda59e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda5c0ff4e0 0x7fda5c102740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:58446/0 (socket says 172.21.15.43:58446) 2024-03-28T22:34:07.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.417+0000 7fda59e00640 1 -- 172.21.15.43:0/529190013 learned_addr learned my addr 172.21.15.43:0/529190013 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:34:07.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.417+0000 7fda5a800640 1 --2- 172.21.15.43:0/529190013 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda5c1073d0 0x7fda5c101310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:07.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.417+0000 7fda59400640 1 --2- 172.21.15.43:0/529190013 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda5c0ffe40 0x7fda5c100dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:07.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.417+0000 7fda59e00640 1 -- 172.21.15.43:0/529190013 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda5c1073d0 msgr2=0x7fda5c101310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:07.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.417+0000 7fda59e00640 1 --2- 172.21.15.43:0/529190013 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda5c1073d0 0x7fda5c101310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:07.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.417+0000 7fda59e00640 1 -- 172.21.15.43:0/529190013 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda5c0ffe40 msgr2=0x7fda5c100dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:07.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.417+0000 7fda59e00640 1 --2- 172.21.15.43:0/529190013 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda5c0ffe40 0x7fda5c100dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:07.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.417+0000 7fda59e00640 1 -- 172.21.15.43:0/529190013 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda5c101bd0 con 0x7fda5c0ff4e0 2024-03-28T22:34:07.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.417+0000 7fda59400640 1 --2- 172.21.15.43:0/529190013 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda5c0ffe40 0x7fda5c100dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:34:07.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.417+0000 7fda59e00640 1 --2- 172.21.15.43:0/529190013 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda5c0ff4e0 0x7fda5c102740 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fda4c00e9f0 tx=0x7fda4c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:07.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.418+0000 7fda52a00640 1 -- 172.21.15.43:0/529190013 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda4c00ccc0 con 0x7fda5c0ff4e0 2024-03-28T22:34:07.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.418+0000 7fda52a00640 1 -- 172.21.15.43:0/529190013 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda4c010910 con 0x7fda5c0ff4e0 2024-03-28T22:34:07.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.418+0000 7fda60c00640 1 -- 172.21.15.43:0/529190013 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda5c1a1c20 con 0x7fda5c0ff4e0 2024-03-28T22:34:07.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.418+0000 7fda52a00640 1 -- 172.21.15.43:0/529190013 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda4c0188a0 con 0x7fda5c0ff4e0 2024-03-28T22:34:07.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.418+0000 7fda60c00640 1 -- 172.21.15.43:0/529190013 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda5c1a2140 con 0x7fda5c0ff4e0 2024-03-28T22:34:07.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.419+0000 7fda60c00640 1 -- 172.21.15.43:0/529190013 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda5c105c50 con 0x7fda5c0ff4e0 2024-03-28T22:34:07.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.421+0000 7fda52a00640 1 -- 172.21.15.43:0/529190013 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fda4c010ac0 con 0x7fda5c0ff4e0 2024-03-28T22:34:07.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.422+0000 7fda52a00640 1 --2- 172.21.15.43:0/529190013 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fda400a1e10 0x7fda400a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:07.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.422+0000 7fda52a00640 1 -- 172.21.15.43:0/529190013 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7fda4c0ce670 con 0x7fda5c0ff4e0 2024-03-28T22:34:07.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.422+0000 7fda59400640 1 --2- 172.21.15.43:0/529190013 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fda400a1e10 0x7fda400a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:07.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.423+0000 7fda59400640 1 --2- 172.21.15.43:0/529190013 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fda400a1e10 0x7fda400a42d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fda4802c170 tx=0x7fda4802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:07.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.430+0000 7fda52a00640 1 -- 172.21.15.43:0/529190013 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fda4c09b6a0 con 0x7fda5c0ff4e0 2024-03-28T22:34:07.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:07 smithi082 ceph-mon[36471]: from='client.24836 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:34:07.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:07 smithi082 ceph-mon[36471]: pgmap v341: 129 pgs: 129 active+clean; 582 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 4 op/s 2024-03-28T22:34:07.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:07 smithi082 ceph-mon[36471]: mds.? [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] up:active 2024-03-28T22:34:07.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:07 smithi082 ceph-mon[36471]: fsmap cephfs:5 7 up:standby 5 up:active 2024-03-28T22:34:07.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:07 smithi082 ceph-mon[36471]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T22:34:07.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:07 smithi082 ceph-mon[36471]: from='client.24859 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:34:07.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:07 smithi043 ceph-mon[31823]: from='client.24836 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:34:07.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:07 smithi043 ceph-mon[31823]: pgmap v341: 129 pgs: 129 active+clean; 582 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 4 op/s 2024-03-28T22:34:07.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:07 smithi043 ceph-mon[31823]: mds.? [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] up:active 2024-03-28T22:34:07.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:07 smithi043 ceph-mon[31823]: fsmap cephfs:5 7 up:standby 5 up:active 2024-03-28T22:34:07.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:07 smithi043 ceph-mon[31823]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T22:34:07.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:07 smithi043 ceph-mon[31823]: from='client.24859 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:34:07.547 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:07 smithi172 ceph-mon[36566]: from='client.24836 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:34:07.547 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:07 smithi172 ceph-mon[36566]: pgmap v341: 129 pgs: 129 active+clean; 582 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 4 op/s 2024-03-28T22:34:07.547 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:07 smithi172 ceph-mon[36566]: mds.? [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] up:active 2024-03-28T22:34:07.547 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:07 smithi172 ceph-mon[36566]: fsmap cephfs:5 7 up:standby 5 up:active 2024-03-28T22:34:07.548 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:07 smithi172 ceph-mon[36566]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T22:34:07.548 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:07 smithi172 ceph-mon[36566]: from='client.24859 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:34:07.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.636+0000 7fda60c00640 1 -- 172.21.15.43:0/529190013 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fda5c02d3c0 con 0x7fda5c0ff4e0 2024-03-28T22:34:07.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.639+0000 7fda52a00640 1 -- 172.21.15.43:0/529190013 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24093 (secure 0 0 0) 0x7fda4c09afa0 con 0x7fda5c0ff4e0 2024-03-28T22:34:07.638 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:34:07.638 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":92,"fsid":"c2276428-ed51-11ee-95d0-87774f69a715","created":"2024-03-28T22:24:19.950084+0000","modified":"2024-03-28T22:33:34.473170+0000","last_up_change":"2024-03-28T22:32:31.078737+0000","last_in_change":"2024-03-28T22:32:10.147097+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:28:14.713290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:33:27.167435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:33:28.411825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T22:33:32.523122+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b855adcd-786c-4e42-b8fe-a515a77ce979","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6803","nonce":2826124409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6805","nonce":2826124409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6809","nonce":2826124409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6807","nonce":2826124409}]},"public_addr":"172.21.15.43:6803/2826124409","cluster_addr":"172.21.15.43:6805/2826124409","heartbeat_back_addr":"172.21.15.43:6809/2826124409","heartbeat_front_addr":"172.21.15.43:6807/2826124409","state":["exists","up"]},{"osd":1,"uuid":"031c143d-8ad1-4b68-8db9-3c717137e3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6801","nonce":3043263644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6803","nonce":3043263644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6807","nonce":3043263644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6805","nonce":3043263644}]},"public_addr":"172.21.15.82:6801/3043263644","cluster_addr":"172.21.15.82:6803/3043263644","heartbeat_back_addr":"172.21.15.82:6807/3043263644","heartbeat_front_addr":"172.21.15.82:6805/3043263644","state":["exists","up"]},{"osd":2,"uuid":"20632bbb-f07a-4ca1-b0c4-ff9d508e211c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6801","nonce":4213953645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6803","nonce":4213953645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6807","nonce":4213953645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6805","nonce":4213953645}]},"public_addr":"172.21.15.172:6801/4213953645","cluster_addr":"172.21.15.172:6803/4213953645","heartbeat_back_addr":"172.21.15.172:6807/4213953645","heartbeat_front_addr":"172.21.15.172:6805/4213953645","state":["exists","up"]},{"osd":3,"uuid":"9a1f2312-4dbd-475b-8736-d5ebc8788e09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6811","nonce":1597280920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6813","nonce":1597280920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6817","nonce":1597280920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6815","nonce":1597280920}]},"public_addr":"172.21.15.43:6811/1597280920","cluster_addr":"172.21.15.43:6813/1597280920","heartbeat_back_addr":"172.21.15.43:6817/1597280920","heartbeat_front_addr":"172.21.15.43:6815/1597280920","state":["exists","up"]},{"osd":4,"uuid":"29140ead-7075-42b2-90c1-27cae0b29ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6809","nonce":215451895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6811","nonce":215451895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6815","nonce":215451895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6813","nonce":215451895}]},"public_addr":"172.21.15.82:6809/215451895","cluster_addr":"172.21.15.82:6811/215451895","heartbeat_back_addr":"172.21.15.82:6815/215451895","heartbeat_front_addr":"172.21.15.82:6813/215451895","state":["exists","up"]},{"osd":5,"uuid":"59644c61-93dc-4536-b2ad-0256b24e5cb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6809","nonce":1576148933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6811","nonce":1576148933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6815","nonce":1576148933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6813","nonce":1576148933}]},"public_addr":"172.21.15.172:6809/1576148933","cluster_addr":"172.21.15.172:6811/1576148933","heartbeat_back_addr":"172.21.15.172:6815/1576148933","heartbeat_front_addr":"172.21.15.172:6813/1576148933","state":["exists","up"]},{"osd":6,"uuid":"22d54e57-c3f1-4ecf-9921-8e815b48f92a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6819","nonce":791044279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6821","nonce":791044279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6825","nonce":791044279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6823","nonce":791044279}]},"public_addr":"172.21.15.43:6819/791044279","cluster_addr":"172.21.15.43:6821/791044279","heartbeat_back_addr":"172.21.15.43:6825/791044279","heartbeat_front_addr":"172.21.15.43:6823/791044279","state":["exists","up"]},{"osd":7,"uuid":"248cd1e2-0d6b-452f-a0a8-c51fe1b975ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6817","nonce":2157777323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6819","nonce":2157777323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6823","nonce":2157777323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6821","nonce":2157777323}]},"public_addr":"172.21.15.82:6817/2157777323","cluster_addr":"172.21.15.82:6819/2157777323","heartbeat_back_addr":"172.21.15.82:6823/2157777323","heartbeat_front_addr":"172.21.15.82:6821/2157777323","state":["exists","up"]},{"osd":8,"uuid":"d98aab6e-8bae-4c24-906c-4f492f047b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6817","nonce":1095328248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6819","nonce":1095328248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6823","nonce":1095328248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6821","nonce":1095328248}]},"public_addr":"172.21.15.172:6817/1095328248","cluster_addr":"172.21.15.172:6819/1095328248","heartbeat_back_addr":"172.21.15.172:6823/1095328248","heartbeat_front_addr":"172.21.15.172:6821/1095328248","state":["exists","up"]},{"osd":9,"uuid":"19aa383e-a0f6-45ea-96cc-e7956535caa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6827","nonce":3657791498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6829","nonce":3657791498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6833","nonce":3657791498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6831","nonce":3657791498}]},"public_addr":"172.21.15.43:6827/3657791498","cluster_addr":"172.21.15.43:6829/3657791498","heartbeat_back_addr":"172.21.15.43:6833/3657791498","heartbeat_front_addr":"172.21.15.43:6831/3657791498","state":["exists","up"]},{"osd":10,"uuid":"666977e5-eaf7-41b9-a18f-c380e149345d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6825","nonce":3454954047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6827","nonce":3454954047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6831","nonce":3454954047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6829","nonce":3454954047}]},"public_addr":"172.21.15.82:6825/3454954047","cluster_addr":"172.21.15.82:6827/3454954047","heartbeat_back_addr":"172.21.15.82:6831/3454954047","heartbeat_front_addr":"172.21.15.82:6829/3454954047","state":["exists","up"]},{"osd":11,"uuid":"615cf3be-8076-470e-ad09-c880e7593717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6825","nonce":2563517144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6827","nonce":2563517144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6831","nonce":2563517144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6829","nonce":2563517144}]},"public_addr":"172.21.15.172:6825/2563517144","cluster_addr":"172.21.15.172:6827/2563517144","heartbeat_back_addr":"172.21.15.172:6831/2563517144","heartbeat_front_addr":"172.21.15.172:6829/2563517144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:21.727405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:47.100259+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:10.615438+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:43.632144+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:09.487940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:34.604463+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:09.594452+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:35.709795+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:01.184202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:35.455322+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:02.413166+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:28.550754+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/578791143":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/4029313215":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/1574178482":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6801/3517712438":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6800/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/1375099638":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/2710117652":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6801/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/3215634168":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6800/3517712438":"2024-03-29T22:25:28.008215+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:34:07.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.641+0000 7fda60c00640 1 -- 172.21.15.43:0/529190013 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fda400a1e10 msgr2=0x7fda400a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:07.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.641+0000 7fda60c00640 1 --2- 172.21.15.43:0/529190013 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fda400a1e10 0x7fda400a42d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fda4802c170 tx=0x7fda4802df90 comp rx=0 tx=0).stop 2024-03-28T22:34:07.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.641+0000 7fda60c00640 1 -- 172.21.15.43:0/529190013 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda5c0ff4e0 msgr2=0x7fda5c102740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:07.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.641+0000 7fda60c00640 1 --2- 172.21.15.43:0/529190013 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda5c0ff4e0 0x7fda5c102740 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fda4c00e9f0 tx=0x7fda4c00edf0 comp rx=0 tx=0).stop 2024-03-28T22:34:07.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.641+0000 7fda60c00640 1 -- 172.21.15.43:0/529190013 shutdown_connections 2024-03-28T22:34:07.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.641+0000 7fda60c00640 1 --2- 172.21.15.43:0/529190013 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda5c1073d0 0x7fda5c101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:07.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.641+0000 7fda60c00640 1 --2- 172.21.15.43:0/529190013 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda5c0ffe40 0x7fda5c100dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:07.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.641+0000 7fda60c00640 1 --2- 172.21.15.43:0/529190013 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fda400a1e10 0x7fda400a42d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:07.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.641+0000 7fda60c00640 1 --2- 172.21.15.43:0/529190013 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda5c0ff4e0 0x7fda5c102740 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:07.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.641+0000 7fda60c00640 1 -- 172.21.15.43:0/529190013 >> 172.21.15.43:0/529190013 conn(0x7fda5c0faf50 msgr2=0x7fda5c102e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:07.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.641+0000 7fda60c00640 1 -- 172.21.15.43:0/529190013 shutdown_connections 2024-03-28T22:34:07.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.641+0000 7fda60c00640 1 -- 172.21.15.43:0/529190013 wait complete. 2024-03-28T22:34:07.660 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-28T22:34:07.661 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T22:34:07.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.889+0000 7f25d5c00640 1 -- 172.21.15.43:0/3783954857 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f25d0104990 msgr2=0x7f25d0104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:07.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.889+0000 7f25d5c00640 1 --2- 172.21.15.43:0/3783954857 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f25d0104990 0x7f25d0104e10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f25b8009900 tx=0x7f25b802bc30 comp rx=0 tx=0).stop 2024-03-28T22:34:07.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.889+0000 7f25d5c00640 1 -- 172.21.15.43:0/3783954857 shutdown_connections 2024-03-28T22:34:07.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.889+0000 7f25d5c00640 1 --2- 172.21.15.43:0/3783954857 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f25d0104990 0x7f25d0104e10 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:07.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.889+0000 7f25d5c00640 1 --2- 172.21.15.43:0/3783954857 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25d0105350 0x7f25d010bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:07.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.889+0000 7f25d5c00640 1 --2- 172.21.15.43:0/3783954857 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25d01037a0 0x7f25d0103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:07.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.889+0000 7f25d5c00640 1 -- 172.21.15.43:0/3783954857 >> 172.21.15.43:0/3783954857 conn(0x7f25d00fed30 msgr2=0x7f25d0101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:07.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.889+0000 7f25d5c00640 1 -- 172.21.15.43:0/3783954857 shutdown_connections 2024-03-28T22:34:07.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.889+0000 7f25d5c00640 1 -- 172.21.15.43:0/3783954857 wait complete. 2024-03-28T22:34:07.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.890+0000 7f25d5c00640 1 Processor -- start 2024-03-28T22:34:07.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.890+0000 7f25d5c00640 1 -- start start 2024-03-28T22:34:07.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.891+0000 7f25d5c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25d01037a0 0x7f25d0199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:07.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.891+0000 7f25d5c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25d0104990 0x7f25d0199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:07.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.891+0000 7f25d5c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f25d0105350 0x7f25d01a1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:07.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.891+0000 7f25d5c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f25d010f450 con 0x7f25d0104990 2024-03-28T22:34:07.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.891+0000 7f25d5c00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f25d010f2d0 con 0x7f25d0105350 2024-03-28T22:34:07.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.891+0000 7f25d5c00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f25d010f5d0 con 0x7f25d01037a0 2024-03-28T22:34:07.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.891+0000 7f25cfe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25d01037a0 0x7f25d0199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:07.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.891+0000 7f25d5200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f25d0105350 0x7f25d01a1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:07.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.891+0000 7f25c7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25d0104990 0x7f25d0199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:07.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.891+0000 7f25d5200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f25d0105350 0x7f25d01a1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:58460/0 (socket says 172.21.15.43:58460) 2024-03-28T22:34:07.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.891+0000 7f25d5200640 1 -- 172.21.15.43:0/1204450669 learned_addr learned my addr 172.21.15.43:0/1204450669 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:34:07.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.891+0000 7f25cfe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25d01037a0 0x7f25d0199a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:49232/0 (socket says 172.21.15.43:49232) 2024-03-28T22:34:07.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.891+0000 7f25d5200640 1 -- 172.21.15.43:0/1204450669 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25d01037a0 msgr2=0x7f25d0199a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:07.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.891+0000 7f25d5200640 1 --2- 172.21.15.43:0/1204450669 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25d01037a0 0x7f25d0199a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:07.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.891+0000 7f25d5200640 1 -- 172.21.15.43:0/1204450669 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25d0104990 msgr2=0x7f25d0199fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:07.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.891+0000 7f25d5200640 1 --2- 172.21.15.43:0/1204450669 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25d0104990 0x7f25d0199fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:07.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.891+0000 7f25d5200640 1 -- 172.21.15.43:0/1204450669 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25d01a1680 con 0x7f25d0105350 2024-03-28T22:34:07.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.892+0000 7f25d5200640 1 --2- 172.21.15.43:0/1204450669 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f25d0105350 0x7f25d01a1010 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f25c800eb10 tx=0x7f25c800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:07.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.892+0000 7f25cd600640 1 -- 172.21.15.43:0/1204450669 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c8004b20 con 0x7f25d0105350 2024-03-28T22:34:07.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.892+0000 7f25cd600640 1 -- 172.21.15.43:0/1204450669 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25c8004cc0 con 0x7f25d0105350 2024-03-28T22:34:07.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.892+0000 7f25d5c00640 1 -- 172.21.15.43:0/1204450669 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25d01a1970 con 0x7f25d0105350 2024-03-28T22:34:07.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.892+0000 7f25cd600640 1 -- 172.21.15.43:0/1204450669 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c8010760 con 0x7f25d0105350 2024-03-28T22:34:07.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.892+0000 7f25d5c00640 1 -- 172.21.15.43:0/1204450669 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25d01a1e90 con 0x7f25d0105350 2024-03-28T22:34:07.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.893+0000 7f25d5c00640 1 -- 172.21.15.43:0/1204450669 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25d010aac0 con 0x7f25d0105350 2024-03-28T22:34:07.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.896+0000 7f25cd600640 1 -- 172.21.15.43:0/1204450669 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f25c8005020 con 0x7f25d0105350 2024-03-28T22:34:07.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.897+0000 7f25cd600640 1 --2- 172.21.15.43:0/1204450669 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f25b00a1e90 0x7f25b00a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:07.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.897+0000 7f25cd600640 1 -- 172.21.15.43:0/1204450669 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f25c80ce6d0 con 0x7f25d0105350 2024-03-28T22:34:07.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.897+0000 7f25cfe00640 1 --2- 172.21.15.43:0/1204450669 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f25b00a1e90 0x7f25b00a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:07.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.898+0000 7f25cfe00640 1 --2- 172.21.15.43:0/1204450669 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f25b00a1e90 0x7f25b00a4350 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f25d01047f0 tx=0x7f25c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:07.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:07.901+0000 7f25cd600640 1 -- 172.21.15.43:0/1204450669 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f25c809b700 con 0x7f25d0105350 2024-03-28T22:34:08.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.100+0000 7f25d5c00640 1 -- 172.21.15.43:0/1204450669 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f25d0061b90 con 0x7f25b00a1e90 2024-03-28T22:34:08.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.114+0000 7f25cd600640 1 -- 172.21.15.43:0/1204450669 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340268 (secure 0 0 0) 0x7f25d0061b90 con 0x7f25b00a1e90 2024-03-28T22:34:08.119 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:34:08.121 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-03-28T22:34:08.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.120+0000 7f25d5c00640 1 -- 172.21.15.43:0/1204450669 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f25b00a1e90 msgr2=0x7f25b00a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:08.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.120+0000 7f25d5c00640 1 --2- 172.21.15.43:0/1204450669 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f25b00a1e90 0x7f25b00a4350 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f25d01047f0 tx=0x7f25c0005f90 comp rx=0 tx=0).stop 2024-03-28T22:34:08.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.120+0000 7f25d5c00640 1 -- 172.21.15.43:0/1204450669 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f25d0105350 msgr2=0x7f25d01a1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:08.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.120+0000 7f25d5c00640 1 --2- 172.21.15.43:0/1204450669 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f25d0105350 0x7f25d01a1010 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f25c800eb10 tx=0x7f25c800c2d0 comp rx=0 tx=0).stop 2024-03-28T22:34:08.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.121+0000 7f25d5c00640 1 -- 172.21.15.43:0/1204450669 shutdown_connections 2024-03-28T22:34:08.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.121+0000 7f25d5c00640 1 --2- 172.21.15.43:0/1204450669 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f25b00a1e90 0x7f25b00a4350 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:08.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.121+0000 7f25d5c00640 1 --2- 172.21.15.43:0/1204450669 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f25d0105350 0x7f25d01a1010 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:08.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.121+0000 7f25d5c00640 1 --2- 172.21.15.43:0/1204450669 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f25d0104990 0x7f25d0199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:08.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.121+0000 7f25d5c00640 1 --2- 172.21.15.43:0/1204450669 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f25d01037a0 0x7f25d0199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:08.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.121+0000 7f25d5c00640 1 -- 172.21.15.43:0/1204450669 >> 172.21.15.43:0/1204450669 conn(0x7f25d00fed30 msgr2=0x7f25d0107c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:08.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.121+0000 7f25d5c00640 1 -- 172.21.15.43:0/1204450669 shutdown_connections 2024-03-28T22:34:08.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.121+0000 7f25d5c00640 1 -- 172.21.15.43:0/1204450669 wait complete. 2024-03-28T22:34:08.139 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":341,"stamp":"2024-03-28T22:34:06.163519+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595904,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":280,"num_write_kb":2969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":220,"ondisk_log_size":220,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":371388,"kb_used_data":13880,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124570436,"statfs":{"total":1151940427776,"available":1151560126464,"internally_reserved":0,"allocated":14213120,"data_stored":7749554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1037,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006458"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.012086+0000","last_change":"2024-03-28T22:33:28.942904+0000","last_active":"2024-03-28T22:33:36.012086+0000","last_peered":"2024-03-28T22:33:36.012086+0000","last_clean":"2024-03-28T22:33:36.012086+0000","last_became_active":"2024-03-28T22:33:28.942574+0000","last_became_peered":"2024-03-28T22:33:28.942574+0000","last_unstale":"2024-03-28T22:33:36.012086+0000","last_undegraded":"2024-03-28T22:33:36.012086+0000","last_fullsized":"2024-03-28T22:33:36.012086+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:55.323919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.012364+0000","last_change":"2024-03-28T22:33:28.951795+0000","last_active":"2024-03-28T22:33:36.012364+0000","last_peered":"2024-03-28T22:33:36.012364+0000","last_clean":"2024-03-28T22:33:36.012364+0000","last_became_active":"2024-03-28T22:33:28.951286+0000","last_became_peered":"2024-03-28T22:33:28.951286+0000","last_unstale":"2024-03-28T22:33:36.012364+0000","last_undegraded":"2024-03-28T22:33:36.012364+0000","last_fullsized":"2024-03-28T22:33:36.012364+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:56:11.718244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.011092+0000","last_change":"2024-03-28T22:33:28.949476+0000","last_active":"2024-03-28T22:33:36.011092+0000","last_peered":"2024-03-28T22:33:36.011092+0000","last_clean":"2024-03-28T22:33:36.011092+0000","last_became_active":"2024-03-28T22:33:28.949128+0000","last_became_peered":"2024-03-28T22:33:28.949128+0000","last_unstale":"2024-03-28T22:33:36.011092+0000","last_undegraded":"2024-03-28T22:33:36.011092+0000","last_fullsized":"2024-03-28T22:33:36.011092+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:49:29.052373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.004170+0000","last_change":"2024-03-28T22:33:28.930715+0000","last_active":"2024-03-28T22:33:35.004170+0000","last_peered":"2024-03-28T22:33:35.004170+0000","last_clean":"2024-03-28T22:33:35.004170+0000","last_became_active":"2024-03-28T22:33:28.930265+0000","last_became_peered":"2024-03-28T22:33:28.930265+0000","last_unstale":"2024-03-28T22:33:35.004170+0000","last_undegraded":"2024-03-28T22:33:35.004170+0000","last_fullsized":"2024-03-28T22:33:35.004170+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:29:54.695671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.004184+0000","last_change":"2024-03-28T22:33:28.944933+0000","last_active":"2024-03-28T22:33:35.004184+0000","last_peered":"2024-03-28T22:33:35.004184+0000","last_clean":"2024-03-28T22:33:35.004184+0000","last_became_active":"2024-03-28T22:33:28.944542+0000","last_became_peered":"2024-03-28T22:33:28.944542+0000","last_unstale":"2024-03-28T22:33:35.004184+0000","last_undegraded":"2024-03-28T22:33:35.004184+0000","last_fullsized":"2024-03-28T22:33:35.004184+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:35:15.719312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.141496+0000","last_change":"2024-03-28T22:33:28.927920+0000","last_active":"2024-03-28T22:33:35.141496+0000","last_peered":"2024-03-28T22:33:35.141496+0000","last_clean":"2024-03-28T22:33:35.141496+0000","last_became_active":"2024-03-28T22:33:28.927602+0000","last_became_peered":"2024-03-28T22:33:28.927602+0000","last_unstale":"2024-03-28T22:33:35.141496+0000","last_undegraded":"2024-03-28T22:33:35.141496+0000","last_fullsized":"2024-03-28T22:33:35.141496+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:24:22.072707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.375922+0000","last_change":"2024-03-28T22:33:28.917963+0000","last_active":"2024-03-28T22:33:36.375922+0000","last_peered":"2024-03-28T22:33:36.375922+0000","last_clean":"2024-03-28T22:33:36.375922+0000","last_became_active":"2024-03-28T22:33:28.917628+0000","last_became_peered":"2024-03-28T22:33:28.917628+0000","last_unstale":"2024-03-28T22:33:36.375922+0000","last_undegraded":"2024-03-28T22:33:36.375922+0000","last_fullsized":"2024-03-28T22:33:36.375922+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:37:09.111640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.377217+0000","last_change":"2024-03-28T22:33:28.942632+0000","last_active":"2024-03-28T22:33:36.377217+0000","last_peered":"2024-03-28T22:33:36.377217+0000","last_clean":"2024-03-28T22:33:36.377217+0000","last_became_active":"2024-03-28T22:33:28.941972+0000","last_became_peered":"2024-03-28T22:33:28.941972+0000","last_unstale":"2024-03-28T22:33:36.377217+0000","last_undegraded":"2024-03-28T22:33:36.377217+0000","last_fullsized":"2024-03-28T22:33:36.377217+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:49:52.642314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.375178+0000","last_change":"2024-03-28T22:33:28.924324+0000","last_active":"2024-03-28T22:33:36.375178+0000","last_peered":"2024-03-28T22:33:36.375178+0000","last_clean":"2024-03-28T22:33:36.375178+0000","last_became_active":"2024-03-28T22:33:28.924014+0000","last_became_peered":"2024-03-28T22:33:28.924014+0000","last_unstale":"2024-03-28T22:33:36.375178+0000","last_undegraded":"2024-03-28T22:33:36.375178+0000","last_fullsized":"2024-03-28T22:33:36.375178+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:17:55.172116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.141839+0000","last_change":"2024-03-28T22:33:28.927152+0000","last_active":"2024-03-28T22:33:35.141839+0000","last_peered":"2024-03-28T22:33:35.141839+0000","last_clean":"2024-03-28T22:33:35.141839+0000","last_became_active":"2024-03-28T22:33:28.926782+0000","last_became_peered":"2024-03-28T22:33:28.926782+0000","last_unstale":"2024-03-28T22:33:35.141839+0000","last_undegraded":"2024-03-28T22:33:35.141839+0000","last_fullsized":"2024-03-28T22:33:35.141839+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:33:57.048323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376438+0000","last_change":"2024-03-28T22:33:28.930802+0000","last_active":"2024-03-28T22:33:36.376438+0000","last_peered":"2024-03-28T22:33:36.376438+0000","last_clean":"2024-03-28T22:33:36.376438+0000","last_became_active":"2024-03-28T22:33:28.930475+0000","last_became_peered":"2024-03-28T22:33:28.930475+0000","last_unstale":"2024-03-28T22:33:36.376438+0000","last_undegraded":"2024-03-28T22:33:36.376438+0000","last_fullsized":"2024-03-28T22:33:36.376438+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:48.443803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.004090+0000","last_change":"2024-03-28T22:33:28.927057+0000","last_active":"2024-03-28T22:33:35.004090+0000","last_peered":"2024-03-28T22:33:35.004090+0000","last_clean":"2024-03-28T22:33:35.004090+0000","last_became_active":"2024-03-28T22:33:28.926598+0000","last_became_peered":"2024-03-28T22:33:28.926598+0000","last_unstale":"2024-03-28T22:33:35.004090+0000","last_undegraded":"2024-03-28T22:33:35.004090+0000","last_fullsized":"2024-03-28T22:33:35.004090+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:21:00.710203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.014089+0000","last_change":"2024-03-28T22:33:28.937034+0000","last_active":"2024-03-28T22:33:36.014089+0000","last_peered":"2024-03-28T22:33:36.014089+0000","last_clean":"2024-03-28T22:33:36.014089+0000","last_became_active":"2024-03-28T22:33:28.933713+0000","last_became_peered":"2024-03-28T22:33:28.933713+0000","last_unstale":"2024-03-28T22:33:36.014089+0000","last_undegraded":"2024-03-28T22:33:36.014089+0000","last_fullsized":"2024-03-28T22:33:36.014089+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:58:24.148897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.010710+0000","last_change":"2024-03-28T22:33:28.947384+0000","last_active":"2024-03-28T22:33:36.010710+0000","last_peered":"2024-03-28T22:33:36.010710+0000","last_clean":"2024-03-28T22:33:36.010710+0000","last_became_active":"2024-03-28T22:33:28.947017+0000","last_became_peered":"2024-03-28T22:33:28.947017+0000","last_unstale":"2024-03-28T22:33:36.010710+0000","last_undegraded":"2024-03-28T22:33:36.010710+0000","last_fullsized":"2024-03-28T22:33:36.010710+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:43:32.781161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.010663+0000","last_change":"2024-03-28T22:33:28.950119+0000","last_active":"2024-03-28T22:33:36.010663+0000","last_peered":"2024-03-28T22:33:36.010663+0000","last_clean":"2024-03-28T22:33:36.010663+0000","last_became_active":"2024-03-28T22:33:28.949798+0000","last_became_peered":"2024-03-28T22:33:28.949798+0000","last_unstale":"2024-03-28T22:33:36.010663+0000","last_undegraded":"2024-03-28T22:33:36.010663+0000","last_fullsized":"2024-03-28T22:33:36.010663+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:16.717678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.374939+0000","last_change":"2024-03-28T22:33:28.943255+0000","last_active":"2024-03-28T22:33:36.374939+0000","last_peered":"2024-03-28T22:33:36.374939+0000","last_clean":"2024-03-28T22:33:36.374939+0000","last_became_active":"2024-03-28T22:33:28.942923+0000","last_became_peered":"2024-03-28T22:33:28.942923+0000","last_unstale":"2024-03-28T22:33:36.374939+0000","last_undegraded":"2024-03-28T22:33:36.374939+0000","last_fullsized":"2024-03-28T22:33:36.374939+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:05:04.197635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.011835+0000","last_change":"2024-03-28T22:33:28.942906+0000","last_active":"2024-03-28T22:33:36.011835+0000","last_peered":"2024-03-28T22:33:36.011835+0000","last_clean":"2024-03-28T22:33:36.011835+0000","last_became_active":"2024-03-28T22:33:28.942573+0000","last_became_peered":"2024-03-28T22:33:28.942573+0000","last_unstale":"2024-03-28T22:33:36.011835+0000","last_undegraded":"2024-03-28T22:33:36.011835+0000","last_fullsized":"2024-03-28T22:33:36.011835+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:39:17.651101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.008726+0000","last_change":"2024-03-28T22:33:28.932075+0000","last_active":"2024-03-28T22:33:35.008726+0000","last_peered":"2024-03-28T22:33:35.008726+0000","last_clean":"2024-03-28T22:33:35.008726+0000","last_became_active":"2024-03-28T22:33:28.931689+0000","last_became_peered":"2024-03-28T22:33:28.931689+0000","last_unstale":"2024-03-28T22:33:35.008726+0000","last_undegraded":"2024-03-28T22:33:35.008726+0000","last_fullsized":"2024-03-28T22:33:35.008726+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:50:43.031082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376706+0000","last_change":"2024-03-28T22:33:28.944987+0000","last_active":"2024-03-28T22:33:36.376706+0000","last_peered":"2024-03-28T22:33:36.376706+0000","last_clean":"2024-03-28T22:33:36.376706+0000","last_became_active":"2024-03-28T22:33:28.944622+0000","last_became_peered":"2024-03-28T22:33:28.944622+0000","last_unstale":"2024-03-28T22:33:36.376706+0000","last_undegraded":"2024-03-28T22:33:36.376706+0000","last_fullsized":"2024-03-28T22:33:36.376706+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:48:58.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376507+0000","last_change":"2024-03-28T22:33:34.014395+0000","last_active":"2024-03-28T22:33:36.376507+0000","last_peered":"2024-03-28T22:33:36.376507+0000","last_clean":"2024-03-28T22:33:36.376507+0000","last_became_active":"2024-03-28T22:33:34.013976+0000","last_became_peered":"2024-03-28T22:33:34.013976+0000","last_unstale":"2024-03-28T22:33:36.376507+0000","last_undegraded":"2024-03-28T22:33:36.376507+0000","last_fullsized":"2024-03-28T22:33:36.376507+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:44:16.265629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.009018+0000","last_change":"2024-03-28T22:33:28.442524+0000","last_active":"2024-03-28T22:33:35.009018+0000","last_peered":"2024-03-28T22:33:35.009018+0000","last_clean":"2024-03-28T22:33:35.009018+0000","last_became_active":"2024-03-28T22:33:28.442169+0000","last_became_peered":"2024-03-28T22:33:28.442169+0000","last_unstale":"2024-03-28T22:33:35.009018+0000","last_undegraded":"2024-03-28T22:33:35.009018+0000","last_fullsized":"2024-03-28T22:33:35.009018+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:08:47.101119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.378959+0000","last_change":"2024-03-28T22:33:28.946697+0000","last_active":"2024-03-28T22:33:36.378959+0000","last_peered":"2024-03-28T22:33:36.378959+0000","last_clean":"2024-03-28T22:33:36.378959+0000","last_became_active":"2024-03-28T22:33:28.946303+0000","last_became_peered":"2024-03-28T22:33:28.946303+0000","last_unstale":"2024-03-28T22:33:36.378959+0000","last_undegraded":"2024-03-28T22:33:36.378959+0000","last_fullsized":"2024-03-28T22:33:36.378959+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:32:01.286073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.003794+0000","last_change":"2024-03-28T22:33:34.016868+0000","last_active":"2024-03-28T22:33:35.003794+0000","last_peered":"2024-03-28T22:33:35.003794+0000","last_clean":"2024-03-28T22:33:35.003794+0000","last_became_active":"2024-03-28T22:33:34.016472+0000","last_became_peered":"2024-03-28T22:33:34.016472+0000","last_unstale":"2024-03-28T22:33:35.003794+0000","last_undegraded":"2024-03-28T22:33:35.003794+0000","last_fullsized":"2024-03-28T22:33:35.003794+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:39:27.397699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"86'1","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.379538+0000","last_change":"2024-03-28T22:33:28.457661+0000","last_active":"2024-03-28T22:33:36.379538+0000","last_peered":"2024-03-28T22:33:36.379538+0000","last_clean":"2024-03-28T22:33:36.379538+0000","last_became_active":"2024-03-28T22:33:28.457032+0000","last_became_peered":"2024-03-28T22:33:28.457032+0000","last_unstale":"2024-03-28T22:33:36.379538+0000","last_undegraded":"2024-03-28T22:33:36.379538+0000","last_fullsized":"2024-03-28T22:33:36.379538+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:43:09.820630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.012702+0000","last_change":"2024-03-28T22:33:28.954252+0000","last_active":"2024-03-28T22:33:36.012702+0000","last_peered":"2024-03-28T22:33:36.012702+0000","last_clean":"2024-03-28T22:33:36.012702+0000","last_became_active":"2024-03-28T22:33:28.953922+0000","last_became_peered":"2024-03-28T22:33:28.953922+0000","last_unstale":"2024-03-28T22:33:36.012702+0000","last_undegraded":"2024-03-28T22:33:36.012702+0000","last_fullsized":"2024-03-28T22:33:36.012702+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:18:47.097859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.003837+0000","last_change":"2024-03-28T22:33:34.029129+0000","last_active":"2024-03-28T22:33:35.003837+0000","last_peered":"2024-03-28T22:33:35.003837+0000","last_clean":"2024-03-28T22:33:35.003837+0000","last_became_active":"2024-03-28T22:33:34.028452+0000","last_became_peered":"2024-03-28T22:33:34.028452+0000","last_unstale":"2024-03-28T22:33:35.003837+0000","last_undegraded":"2024-03-28T22:33:35.003837+0000","last_fullsized":"2024-03-28T22:33:35.003837+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:28.982773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:03.152124+0000","last_change":"2024-03-28T22:33:28.439568+0000","last_active":"2024-03-28T22:34:03.152124+0000","last_peered":"2024-03-28T22:34:03.152124+0000","last_clean":"2024-03-28T22:34:03.152124+0000","last_became_active":"2024-03-28T22:33:28.438887+0000","last_became_peered":"2024-03-28T22:33:28.438887+0000","last_unstale":"2024-03-28T22:34:03.152124+0000","last_undegraded":"2024-03-28T22:34:03.152124+0000","last_fullsized":"2024-03-28T22:34:03.152124+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:56:43.617621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.375102+0000","last_change":"2024-03-28T22:33:28.938311+0000","last_active":"2024-03-28T22:33:36.375102+0000","last_peered":"2024-03-28T22:33:36.375102+0000","last_clean":"2024-03-28T22:33:36.375102+0000","last_became_active":"2024-03-28T22:33:28.937997+0000","last_became_peered":"2024-03-28T22:33:28.937997+0000","last_unstale":"2024-03-28T22:33:36.375102+0000","last_undegraded":"2024-03-28T22:33:36.375102+0000","last_fullsized":"2024-03-28T22:33:36.375102+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:54:44.001545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.003786+0000","last_change":"2024-03-28T22:33:34.016931+0000","last_active":"2024-03-28T22:33:35.003786+0000","last_peered":"2024-03-28T22:33:35.003786+0000","last_clean":"2024-03-28T22:33:35.003786+0000","last_became_active":"2024-03-28T22:33:34.016559+0000","last_became_peered":"2024-03-28T22:33:34.016559+0000","last_unstale":"2024-03-28T22:33:35.003786+0000","last_undegraded":"2024-03-28T22:33:35.003786+0000","last_fullsized":"2024-03-28T22:33:35.003786+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:08:31.252351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.004484+0000","last_change":"2024-03-28T22:33:28.433166+0000","last_active":"2024-03-28T22:33:35.004484+0000","last_peered":"2024-03-28T22:33:35.004484+0000","last_clean":"2024-03-28T22:33:35.004484+0000","last_became_active":"2024-03-28T22:33:28.432716+0000","last_became_peered":"2024-03-28T22:33:28.432716+0000","last_unstale":"2024-03-28T22:33:35.004484+0000","last_undegraded":"2024-03-28T22:33:35.004484+0000","last_fullsized":"2024-03-28T22:33:35.004484+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:42:41.613583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.536637+0000","last_change":"2024-03-28T22:33:28.923307+0000","last_active":"2024-03-28T22:33:35.536637+0000","last_peered":"2024-03-28T22:33:35.536637+0000","last_clean":"2024-03-28T22:33:35.536637+0000","last_became_active":"2024-03-28T22:33:28.922499+0000","last_became_peered":"2024-03-28T22:33:28.922499+0000","last_unstale":"2024-03-28T22:33:35.536637+0000","last_undegraded":"2024-03-28T22:33:35.536637+0000","last_fullsized":"2024-03-28T22:33:35.536637+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:48:48.514482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376468+0000","last_change":"2024-03-28T22:33:34.002209+0000","last_active":"2024-03-28T22:33:36.376468+0000","last_peered":"2024-03-28T22:33:36.376468+0000","last_clean":"2024-03-28T22:33:36.376468+0000","last_became_active":"2024-03-28T22:33:34.001746+0000","last_became_peered":"2024-03-28T22:33:34.001746+0000","last_unstale":"2024-03-28T22:33:36.376468+0000","last_undegraded":"2024-03-28T22:33:36.376468+0000","last_fullsized":"2024-03-28T22:33:36.376468+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:02:02.333328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'1","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.536416+0000","last_change":"2024-03-28T22:33:28.435380+0000","last_active":"2024-03-28T22:33:35.536416+0000","last_peered":"2024-03-28T22:33:35.536416+0000","last_clean":"2024-03-28T22:33:35.536416+0000","last_became_active":"2024-03-28T22:33:28.434898+0000","last_became_peered":"2024-03-28T22:33:28.434898+0000","last_unstale":"2024-03-28T22:33:35.536416+0000","last_undegraded":"2024-03-28T22:33:35.536416+0000","last_fullsized":"2024-03-28T22:33:35.536416+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:58:06.810394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.004390+0000","last_change":"2024-03-28T22:33:28.926644+0000","last_active":"2024-03-28T22:33:35.004390+0000","last_peered":"2024-03-28T22:33:35.004390+0000","last_clean":"2024-03-28T22:33:35.004390+0000","last_became_active":"2024-03-28T22:33:28.926316+0000","last_became_peered":"2024-03-28T22:33:28.926316+0000","last_unstale":"2024-03-28T22:33:35.004390+0000","last_undegraded":"2024-03-28T22:33:35.004390+0000","last_fullsized":"2024-03-28T22:33:35.004390+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:59:23.190520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376589+0000","last_change":"2024-03-28T22:33:34.031350+0000","last_active":"2024-03-28T22:33:36.376589+0000","last_peered":"2024-03-28T22:33:36.376589+0000","last_clean":"2024-03-28T22:33:36.376589+0000","last_became_active":"2024-03-28T22:33:34.030946+0000","last_became_peered":"2024-03-28T22:33:34.030946+0000","last_unstale":"2024-03-28T22:33:36.376589+0000","last_undegraded":"2024-03-28T22:33:36.376589+0000","last_fullsized":"2024-03-28T22:33:36.376589+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:54:22.907363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:04.163516+0000","last_change":"2024-03-28T22:33:28.446197+0000","last_active":"2024-03-28T22:34:04.163516+0000","last_peered":"2024-03-28T22:34:04.163516+0000","last_clean":"2024-03-28T22:34:04.163516+0000","last_became_active":"2024-03-28T22:33:28.445891+0000","last_became_peered":"2024-03-28T22:33:28.445891+0000","last_unstale":"2024-03-28T22:34:04.163516+0000","last_undegraded":"2024-03-28T22:34:04.163516+0000","last_fullsized":"2024-03-28T22:34:04.163516+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:58:57.385302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.536579+0000","last_change":"2024-03-28T22:33:28.955651+0000","last_active":"2024-03-28T22:33:35.536579+0000","last_peered":"2024-03-28T22:33:35.536579+0000","last_clean":"2024-03-28T22:33:35.536579+0000","last_became_active":"2024-03-28T22:33:28.955276+0000","last_became_peered":"2024-03-28T22:33:28.955276+0000","last_unstale":"2024-03-28T22:33:35.536579+0000","last_undegraded":"2024-03-28T22:33:35.536579+0000","last_fullsized":"2024-03-28T22:33:35.536579+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:36:13.455327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.012043+0000","last_change":"2024-03-28T22:33:34.017520+0000","last_active":"2024-03-28T22:33:36.012043+0000","last_peered":"2024-03-28T22:33:36.012043+0000","last_clean":"2024-03-28T22:33:36.012043+0000","last_became_active":"2024-03-28T22:33:34.017022+0000","last_became_peered":"2024-03-28T22:33:34.017022+0000","last_unstale":"2024-03-28T22:33:36.012043+0000","last_undegraded":"2024-03-28T22:33:36.012043+0000","last_fullsized":"2024-03-28T22:33:36.012043+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:41:17.431581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.013492+0000","last_change":"2024-03-28T22:33:28.445401+0000","last_active":"2024-03-28T22:33:36.013492+0000","last_peered":"2024-03-28T22:33:36.013492+0000","last_clean":"2024-03-28T22:33:36.013492+0000","last_became_active":"2024-03-28T22:33:28.444237+0000","last_became_peered":"2024-03-28T22:33:28.444237+0000","last_unstale":"2024-03-28T22:33:36.013492+0000","last_undegraded":"2024-03-28T22:33:36.013492+0000","last_fullsized":"2024-03-28T22:33:36.013492+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:18:12.820890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.377630+0000","last_change":"2024-03-28T22:33:28.942347+0000","last_active":"2024-03-28T22:33:36.377630+0000","last_peered":"2024-03-28T22:33:36.377630+0000","last_clean":"2024-03-28T22:33:36.377630+0000","last_became_active":"2024-03-28T22:33:28.941573+0000","last_became_peered":"2024-03-28T22:33:28.941573+0000","last_unstale":"2024-03-28T22:33:36.377630+0000","last_undegraded":"2024-03-28T22:33:36.377630+0000","last_fullsized":"2024-03-28T22:33:36.377630+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:21:24.643326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.012153+0000","last_change":"2024-03-28T22:33:34.019899+0000","last_active":"2024-03-28T22:33:36.012153+0000","last_peered":"2024-03-28T22:33:36.012153+0000","last_clean":"2024-03-28T22:33:36.012153+0000","last_became_active":"2024-03-28T22:33:34.019392+0000","last_became_peered":"2024-03-28T22:33:34.019392+0000","last_unstale":"2024-03-28T22:33:36.012153+0000","last_undegraded":"2024-03-28T22:33:36.012153+0000","last_fullsized":"2024-03-28T22:33:36.012153+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:08:38.633797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.536160+0000","last_change":"2024-03-28T22:33:28.433243+0000","last_active":"2024-03-28T22:33:35.536160+0000","last_peered":"2024-03-28T22:33:35.536160+0000","last_clean":"2024-03-28T22:33:35.536160+0000","last_became_active":"2024-03-28T22:33:28.432927+0000","last_became_peered":"2024-03-28T22:33:28.432927+0000","last_unstale":"2024-03-28T22:33:35.536160+0000","last_undegraded":"2024-03-28T22:33:35.536160+0000","last_fullsized":"2024-03-28T22:33:35.536160+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:40:43.444229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.006977+0000","last_change":"2024-03-28T22:33:28.948111+0000","last_active":"2024-03-28T22:33:35.006977+0000","last_peered":"2024-03-28T22:33:35.006977+0000","last_clean":"2024-03-28T22:33:35.006977+0000","last_became_active":"2024-03-28T22:33:28.944649+0000","last_became_peered":"2024-03-28T22:33:28.944649+0000","last_unstale":"2024-03-28T22:33:35.006977+0000","last_undegraded":"2024-03-28T22:33:35.006977+0000","last_fullsized":"2024-03-28T22:33:35.006977+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:57:11.483037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.377846+0000","last_change":"2024-03-28T22:33:34.000837+0000","last_active":"2024-03-28T22:33:36.377846+0000","last_peered":"2024-03-28T22:33:36.377846+0000","last_clean":"2024-03-28T22:33:36.377846+0000","last_became_active":"2024-03-28T22:33:34.000120+0000","last_became_peered":"2024-03-28T22:33:34.000120+0000","last_unstale":"2024-03-28T22:33:36.377846+0000","last_undegraded":"2024-03-28T22:33:36.377846+0000","last_fullsized":"2024-03-28T22:33:36.377846+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:14:32.028952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:02.143560+0000","last_change":"2024-03-28T22:33:28.436985+0000","last_active":"2024-03-28T22:34:02.143560+0000","last_peered":"2024-03-28T22:34:02.143560+0000","last_clean":"2024-03-28T22:34:02.143560+0000","last_became_active":"2024-03-28T22:33:28.436625+0000","last_became_peered":"2024-03-28T22:33:28.436625+0000","last_unstale":"2024-03-28T22:34:02.143560+0000","last_undegraded":"2024-03-28T22:34:02.143560+0000","last_fullsized":"2024-03-28T22:34:02.143560+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:50:45.775191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.375265+0000","last_change":"2024-03-28T22:33:28.933265+0000","last_active":"2024-03-28T22:33:36.375265+0000","last_peered":"2024-03-28T22:33:36.375265+0000","last_clean":"2024-03-28T22:33:36.375265+0000","last_became_active":"2024-03-28T22:33:28.932954+0000","last_became_peered":"2024-03-28T22:33:28.932954+0000","last_unstale":"2024-03-28T22:33:36.375265+0000","last_undegraded":"2024-03-28T22:33:36.375265+0000","last_fullsized":"2024-03-28T22:33:36.375265+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:37.039055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.006940+0000","last_change":"2024-03-28T22:33:34.024529+0000","last_active":"2024-03-28T22:33:35.006940+0000","last_peered":"2024-03-28T22:33:35.006940+0000","last_clean":"2024-03-28T22:33:35.006940+0000","last_became_active":"2024-03-28T22:33:34.020949+0000","last_became_peered":"2024-03-28T22:33:34.020949+0000","last_unstale":"2024-03-28T22:33:35.006940+0000","last_undegraded":"2024-03-28T22:33:35.006940+0000","last_fullsized":"2024-03-28T22:33:35.006940+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:28:52.039684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"92'2","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:02.133927+0000","last_change":"2024-03-28T22:33:28.432599+0000","last_active":"2024-03-28T22:34:02.133927+0000","last_peered":"2024-03-28T22:34:02.133927+0000","last_clean":"2024-03-28T22:34:02.133927+0000","last_became_active":"2024-03-28T22:33:28.430710+0000","last_became_peered":"2024-03-28T22:33:28.430710+0000","last_unstale":"2024-03-28T22:34:02.133927+0000","last_undegraded":"2024-03-28T22:34:02.133927+0000","last_fullsized":"2024-03-28T22:34:02.133927+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:54.174747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.012898+0000","last_change":"2024-03-28T22:33:28.955441+0000","last_active":"2024-03-28T22:33:36.012898+0000","last_peered":"2024-03-28T22:33:36.012898+0000","last_clean":"2024-03-28T22:33:36.012898+0000","last_became_active":"2024-03-28T22:33:28.953269+0000","last_became_peered":"2024-03-28T22:33:28.953269+0000","last_unstale":"2024-03-28T22:33:36.012898+0000","last_undegraded":"2024-03-28T22:33:36.012898+0000","last_fullsized":"2024-03-28T22:33:36.012898+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:47:04.128605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.003994+0000","last_change":"2024-03-28T22:33:34.028791+0000","last_active":"2024-03-28T22:33:35.003994+0000","last_peered":"2024-03-28T22:33:35.003994+0000","last_clean":"2024-03-28T22:33:35.003994+0000","last_became_active":"2024-03-28T22:33:34.028361+0000","last_became_peered":"2024-03-28T22:33:34.028361+0000","last_unstale":"2024-03-28T22:33:35.003994+0000","last_undegraded":"2024-03-28T22:33:35.003994+0000","last_fullsized":"2024-03-28T22:33:35.003994+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:42:21.264459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"86'3","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.536575+0000","last_change":"2024-03-28T22:33:28.455353+0000","last_active":"2024-03-28T22:33:35.536575+0000","last_peered":"2024-03-28T22:33:35.536575+0000","last_clean":"2024-03-28T22:33:35.536575+0000","last_became_active":"2024-03-28T22:33:28.455090+0000","last_became_peered":"2024-03-28T22:33:28.455090+0000","last_unstale":"2024-03-28T22:33:35.536575+0000","last_undegraded":"2024-03-28T22:33:35.536575+0000","last_fullsized":"2024-03-28T22:33:35.536575+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:19:22.496600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.375269+0000","last_change":"2024-03-28T22:33:28.928875+0000","last_active":"2024-03-28T22:33:36.375269+0000","last_peered":"2024-03-28T22:33:36.375269+0000","last_clean":"2024-03-28T22:33:36.375269+0000","last_became_active":"2024-03-28T22:33:28.928449+0000","last_became_peered":"2024-03-28T22:33:28.928449+0000","last_unstale":"2024-03-28T22:33:36.375269+0000","last_undegraded":"2024-03-28T22:33:36.375269+0000","last_fullsized":"2024-03-28T22:33:36.375269+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:34:00.402100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.007553+0000","last_change":"2024-03-28T22:33:34.012739+0000","last_active":"2024-03-28T22:33:35.007553+0000","last_peered":"2024-03-28T22:33:35.007553+0000","last_clean":"2024-03-28T22:33:35.007553+0000","last_became_active":"2024-03-28T22:33:34.012262+0000","last_became_peered":"2024-03-28T22:33:34.012262+0000","last_unstale":"2024-03-28T22:33:35.007553+0000","last_undegraded":"2024-03-28T22:33:35.007553+0000","last_fullsized":"2024-03-28T22:33:35.007553+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:24:48.351037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"92'12","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:02.133506+0000","last_change":"2024-03-28T22:33:28.430606+0000","last_active":"2024-03-28T22:34:02.133506+0000","last_peered":"2024-03-28T22:34:02.133506+0000","last_clean":"2024-03-28T22:34:02.133506+0000","last_became_active":"2024-03-28T22:33:28.430233+0000","last_became_peered":"2024-03-28T22:33:28.430233+0000","last_unstale":"2024-03-28T22:34:02.133506+0000","last_undegraded":"2024-03-28T22:34:02.133506+0000","last_fullsized":"2024-03-28T22:34:02.133506+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:49.000569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.004329+0000","last_change":"2024-03-28T22:33:28.931167+0000","last_active":"2024-03-28T22:33:35.004329+0000","last_peered":"2024-03-28T22:33:35.004329+0000","last_clean":"2024-03-28T22:33:35.004329+0000","last_became_active":"2024-03-28T22:33:28.930824+0000","last_became_peered":"2024-03-28T22:33:28.930824+0000","last_unstale":"2024-03-28T22:33:35.004329+0000","last_undegraded":"2024-03-28T22:33:35.004329+0000","last_fullsized":"2024-03-28T22:33:35.004329+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:46:49.102791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.537046+0000","last_change":"2024-03-28T22:33:34.023056+0000","last_active":"2024-03-28T22:33:35.537046+0000","last_peered":"2024-03-28T22:33:35.537046+0000","last_clean":"2024-03-28T22:33:35.537046+0000","last_became_active":"2024-03-28T22:33:34.022612+0000","last_became_peered":"2024-03-28T22:33:34.022612+0000","last_unstale":"2024-03-28T22:33:35.537046+0000","last_undegraded":"2024-03-28T22:33:35.537046+0000","last_fullsized":"2024-03-28T22:33:35.537046+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:52:31.936008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:03.156550+0000","last_change":"2024-03-28T22:33:28.430963+0000","last_active":"2024-03-28T22:34:03.156550+0000","last_peered":"2024-03-28T22:34:03.156550+0000","last_clean":"2024-03-28T22:34:03.156550+0000","last_became_active":"2024-03-28T22:33:28.430632+0000","last_became_peered":"2024-03-28T22:33:28.430632+0000","last_unstale":"2024-03-28T22:34:03.156550+0000","last_undegraded":"2024-03-28T22:34:03.156550+0000","last_fullsized":"2024-03-28T22:34:03.156550+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:30:23.104769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.013675+0000","last_change":"2024-03-28T22:33:28.954105+0000","last_active":"2024-03-28T22:33:36.013675+0000","last_peered":"2024-03-28T22:33:36.013675+0000","last_clean":"2024-03-28T22:33:36.013675+0000","last_became_active":"2024-03-28T22:33:28.953768+0000","last_became_peered":"2024-03-28T22:33:28.953768+0000","last_unstale":"2024-03-28T22:33:36.013675+0000","last_undegraded":"2024-03-28T22:33:36.013675+0000","last_fullsized":"2024-03-28T22:33:36.013675+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:01:25.184523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.011771+0000","last_change":"2024-03-28T22:33:34.027976+0000","last_active":"2024-03-28T22:33:36.011771+0000","last_peered":"2024-03-28T22:33:36.011771+0000","last_clean":"2024-03-28T22:33:36.011771+0000","last_became_active":"2024-03-28T22:33:34.027509+0000","last_became_peered":"2024-03-28T22:33:34.027509+0000","last_unstale":"2024-03-28T22:33:36.011771+0000","last_undegraded":"2024-03-28T22:33:36.011771+0000","last_fullsized":"2024-03-28T22:33:36.011771+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:25:41.908441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.536504+0000","last_change":"2024-03-28T22:33:28.433283+0000","last_active":"2024-03-28T22:33:35.536504+0000","last_peered":"2024-03-28T22:33:35.536504+0000","last_clean":"2024-03-28T22:33:35.536504+0000","last_became_active":"2024-03-28T22:33:28.432954+0000","last_became_peered":"2024-03-28T22:33:28.432954+0000","last_unstale":"2024-03-28T22:33:35.536504+0000","last_undegraded":"2024-03-28T22:33:35.536504+0000","last_fullsized":"2024-03-28T22:33:35.536504+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:08:34.480459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376270+0000","last_change":"2024-03-28T22:33:28.939016+0000","last_active":"2024-03-28T22:33:36.376270+0000","last_peered":"2024-03-28T22:33:36.376270+0000","last_clean":"2024-03-28T22:33:36.376270+0000","last_became_active":"2024-03-28T22:33:28.938695+0000","last_became_peered":"2024-03-28T22:33:28.938695+0000","last_unstale":"2024-03-28T22:33:36.376270+0000","last_undegraded":"2024-03-28T22:33:36.376270+0000","last_fullsized":"2024-03-28T22:33:36.376270+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:26:53.264559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.008835+0000","last_change":"2024-03-28T22:33:34.030127+0000","last_active":"2024-03-28T22:33:35.008835+0000","last_peered":"2024-03-28T22:33:35.008835+0000","last_clean":"2024-03-28T22:33:35.008835+0000","last_became_active":"2024-03-28T22:33:34.028056+0000","last_became_peered":"2024-03-28T22:33:34.028056+0000","last_unstale":"2024-03-28T22:33:35.008835+0000","last_undegraded":"2024-03-28T22:33:35.008835+0000","last_fullsized":"2024-03-28T22:33:35.008835+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:29:41.785013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.003699+0000","last_change":"2024-03-28T22:33:28.435350+0000","last_active":"2024-03-28T22:33:35.003699+0000","last_peered":"2024-03-28T22:33:35.003699+0000","last_clean":"2024-03-28T22:33:35.003699+0000","last_became_active":"2024-03-28T22:33:28.434954+0000","last_became_peered":"2024-03-28T22:33:28.434954+0000","last_unstale":"2024-03-28T22:33:35.003699+0000","last_undegraded":"2024-03-28T22:33:35.003699+0000","last_fullsized":"2024-03-28T22:33:35.003699+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:04.969596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376101+0000","last_change":"2024-03-28T22:33:28.921219+0000","last_active":"2024-03-28T22:33:36.376101+0000","last_peered":"2024-03-28T22:33:36.376101+0000","last_clean":"2024-03-28T22:33:36.376101+0000","last_became_active":"2024-03-28T22:33:28.920664+0000","last_became_peered":"2024-03-28T22:33:28.920664+0000","last_unstale":"2024-03-28T22:33:36.376101+0000","last_undegraded":"2024-03-28T22:33:36.376101+0000","last_fullsized":"2024-03-28T22:33:36.376101+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:48:55.032058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.013871+0000","last_change":"2024-03-28T22:33:34.018101+0000","last_active":"2024-03-28T22:33:36.013871+0000","last_peered":"2024-03-28T22:33:36.013871+0000","last_clean":"2024-03-28T22:33:36.013871+0000","last_became_active":"2024-03-28T22:33:34.017260+0000","last_became_peered":"2024-03-28T22:33:34.017260+0000","last_unstale":"2024-03-28T22:33:36.013871+0000","last_undegraded":"2024-03-28T22:33:36.013871+0000","last_fullsized":"2024-03-28T22:33:36.013871+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:48:53.178090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"92'15","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:03.156112+0000","last_change":"2024-03-28T22:33:28.450232+0000","last_active":"2024-03-28T22:34:03.156112+0000","last_peered":"2024-03-28T22:34:03.156112+0000","last_clean":"2024-03-28T22:34:03.156112+0000","last_became_active":"2024-03-28T22:33:28.449958+0000","last_became_peered":"2024-03-28T22:33:28.449958+0000","last_unstale":"2024-03-28T22:34:03.156112+0000","last_undegraded":"2024-03-28T22:34:03.156112+0000","last_fullsized":"2024-03-28T22:34:03.156112+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:01:42.130260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.536203+0000","last_change":"2024-03-28T22:33:28.926263+0000","last_active":"2024-03-28T22:33:35.536203+0000","last_peered":"2024-03-28T22:33:35.536203+0000","last_clean":"2024-03-28T22:33:35.536203+0000","last_became_active":"2024-03-28T22:33:28.925927+0000","last_became_peered":"2024-03-28T22:33:28.925927+0000","last_unstale":"2024-03-28T22:33:35.536203+0000","last_undegraded":"2024-03-28T22:33:35.536203+0000","last_fullsized":"2024-03-28T22:33:35.536203+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:41:42.477694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.536325+0000","last_change":"2024-03-28T22:33:34.009764+0000","last_active":"2024-03-28T22:33:35.536325+0000","last_peered":"2024-03-28T22:33:35.536325+0000","last_clean":"2024-03-28T22:33:35.536325+0000","last_became_active":"2024-03-28T22:33:34.009320+0000","last_became_peered":"2024-03-28T22:33:34.009320+0000","last_unstale":"2024-03-28T22:33:35.536325+0000","last_undegraded":"2024-03-28T22:33:35.536325+0000","last_fullsized":"2024-03-28T22:33:35.536325+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:21:51.848928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'1","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.004748+0000","last_change":"2024-03-28T22:33:28.433015+0000","last_active":"2024-03-28T22:33:35.004748+0000","last_peered":"2024-03-28T22:33:35.004748+0000","last_clean":"2024-03-28T22:33:35.004748+0000","last_became_active":"2024-03-28T22:33:28.432689+0000","last_became_peered":"2024-03-28T22:33:28.432689+0000","last_unstale":"2024-03-28T22:33:35.004748+0000","last_undegraded":"2024-03-28T22:33:35.004748+0000","last_fullsized":"2024-03-28T22:33:35.004748+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:20:22.349953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.011843+0000","last_change":"2024-03-28T22:33:28.927335+0000","last_active":"2024-03-28T22:33:36.011843+0000","last_peered":"2024-03-28T22:33:36.011843+0000","last_clean":"2024-03-28T22:33:36.011843+0000","last_became_active":"2024-03-28T22:33:28.927036+0000","last_became_peered":"2024-03-28T22:33:28.927036+0000","last_unstale":"2024-03-28T22:33:36.011843+0000","last_undegraded":"2024-03-28T22:33:36.011843+0000","last_fullsized":"2024-03-28T22:33:36.011843+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:36:00.930368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.004206+0000","last_change":"2024-03-28T22:33:34.014364+0000","last_active":"2024-03-28T22:33:35.004206+0000","last_peered":"2024-03-28T22:33:35.004206+0000","last_clean":"2024-03-28T22:33:35.004206+0000","last_became_active":"2024-03-28T22:33:34.013929+0000","last_became_peered":"2024-03-28T22:33:34.013929+0000","last_unstale":"2024-03-28T22:33:35.004206+0000","last_undegraded":"2024-03-28T22:33:35.004206+0000","last_fullsized":"2024-03-28T22:33:35.004206+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:21:37.986774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.007648+0000","last_change":"2024-03-28T22:33:28.453458+0000","last_active":"2024-03-28T22:33:35.007648+0000","last_peered":"2024-03-28T22:33:35.007648+0000","last_clean":"2024-03-28T22:33:35.007648+0000","last_became_active":"2024-03-28T22:33:28.452829+0000","last_became_peered":"2024-03-28T22:33:28.452829+0000","last_unstale":"2024-03-28T22:33:35.007648+0000","last_undegraded":"2024-03-28T22:33:35.007648+0000","last_fullsized":"2024-03-28T22:33:35.007648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:33:05.885690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.141691+0000","last_change":"2024-03-28T22:33:28.942097+0000","last_active":"2024-03-28T22:33:35.141691+0000","last_peered":"2024-03-28T22:33:35.141691+0000","last_clean":"2024-03-28T22:33:35.141691+0000","last_became_active":"2024-03-28T22:33:28.941812+0000","last_became_peered":"2024-03-28T22:33:28.941812+0000","last_unstale":"2024-03-28T22:33:35.141691+0000","last_undegraded":"2024-03-28T22:33:35.141691+0000","last_fullsized":"2024-03-28T22:33:35.141691+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:36:10.204216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.013288+0000","last_change":"2024-03-28T22:33:34.023171+0000","last_active":"2024-03-28T22:33:36.013288+0000","last_peered":"2024-03-28T22:33:36.013288+0000","last_clean":"2024-03-28T22:33:36.013288+0000","last_became_active":"2024-03-28T22:33:34.021313+0000","last_became_peered":"2024-03-28T22:33:34.021313+0000","last_unstale":"2024-03-28T22:33:36.013288+0000","last_undegraded":"2024-03-28T22:33:36.013288+0000","last_fullsized":"2024-03-28T22:33:36.013288+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:29:42.592761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"92'1","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376198+0000","last_change":"2024-03-28T22:33:28.449133+0000","last_active":"2024-03-28T22:33:36.376198+0000","last_peered":"2024-03-28T22:33:36.376198+0000","last_clean":"2024-03-28T22:33:36.376198+0000","last_became_active":"2024-03-28T22:33:28.448743+0000","last_became_peered":"2024-03-28T22:33:28.448743+0000","last_unstale":"2024-03-28T22:33:36.376198+0000","last_undegraded":"2024-03-28T22:33:36.376198+0000","last_fullsized":"2024-03-28T22:33:36.376198+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:09:37.119736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.141594+0000","last_change":"2024-03-28T22:33:28.944198+0000","last_active":"2024-03-28T22:33:35.141594+0000","last_peered":"2024-03-28T22:33:35.141594+0000","last_clean":"2024-03-28T22:33:35.141594+0000","last_became_active":"2024-03-28T22:33:28.943818+0000","last_became_peered":"2024-03-28T22:33:28.943818+0000","last_unstale":"2024-03-28T22:33:35.141594+0000","last_undegraded":"2024-03-28T22:33:35.141594+0000","last_fullsized":"2024-03-28T22:33:35.141594+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:31:03.791364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.014206+0000","last_change":"2024-03-28T22:33:34.015499+0000","last_active":"2024-03-28T22:33:36.014206+0000","last_peered":"2024-03-28T22:33:36.014206+0000","last_clean":"2024-03-28T22:33:36.014206+0000","last_became_active":"2024-03-28T22:33:34.015013+0000","last_became_peered":"2024-03-28T22:33:34.015013+0000","last_unstale":"2024-03-28T22:33:36.014206+0000","last_undegraded":"2024-03-28T22:33:36.014206+0000","last_fullsized":"2024-03-28T22:33:36.014206+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:34:30.582198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.536304+0000","last_change":"2024-03-28T22:33:28.926732+0000","last_active":"2024-03-28T22:33:35.536304+0000","last_peered":"2024-03-28T22:33:35.536304+0000","last_clean":"2024-03-28T22:33:35.536304+0000","last_became_active":"2024-03-28T22:33:28.926366+0000","last_became_peered":"2024-03-28T22:33:28.926366+0000","last_unstale":"2024-03-28T22:33:35.536304+0000","last_undegraded":"2024-03-28T22:33:35.536304+0000","last_fullsized":"2024-03-28T22:33:35.536304+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:19:10.769141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"92'1","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.013963+0000","last_change":"2024-03-28T22:33:28.457400+0000","last_active":"2024-03-28T22:33:36.013963+0000","last_peered":"2024-03-28T22:33:36.013963+0000","last_clean":"2024-03-28T22:33:36.013963+0000","last_became_active":"2024-03-28T22:33:28.456994+0000","last_became_peered":"2024-03-28T22:33:28.456994+0000","last_unstale":"2024-03-28T22:33:36.013963+0000","last_undegraded":"2024-03-28T22:33:36.013963+0000","last_fullsized":"2024-03-28T22:33:36.013963+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:32:16.438540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376384+0000","last_change":"2024-03-28T22:33:34.027746+0000","last_active":"2024-03-28T22:33:36.376384+0000","last_peered":"2024-03-28T22:33:36.376384+0000","last_clean":"2024-03-28T22:33:36.376384+0000","last_became_active":"2024-03-28T22:33:34.027419+0000","last_became_peered":"2024-03-28T22:33:34.027419+0000","last_unstale":"2024-03-28T22:33:36.376384+0000","last_undegraded":"2024-03-28T22:33:36.376384+0000","last_fullsized":"2024-03-28T22:33:36.376384+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:55:43.686428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.378061+0000","last_change":"2024-03-28T22:33:28.952393+0000","last_active":"2024-03-28T22:33:36.378061+0000","last_peered":"2024-03-28T22:33:36.378061+0000","last_clean":"2024-03-28T22:33:36.378061+0000","last_became_active":"2024-03-28T22:33:28.951951+0000","last_became_peered":"2024-03-28T22:33:28.951951+0000","last_unstale":"2024-03-28T22:33:36.378061+0000","last_undegraded":"2024-03-28T22:33:36.378061+0000","last_fullsized":"2024-03-28T22:33:36.378061+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:09:25.808842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.003772+0000","last_change":"2024-03-28T22:33:28.437177+0000","last_active":"2024-03-28T22:33:35.003772+0000","last_peered":"2024-03-28T22:33:35.003772+0000","last_clean":"2024-03-28T22:33:35.003772+0000","last_became_active":"2024-03-28T22:33:28.436801+0000","last_became_peered":"2024-03-28T22:33:28.436801+0000","last_unstale":"2024-03-28T22:33:35.003772+0000","last_undegraded":"2024-03-28T22:33:35.003772+0000","last_fullsized":"2024-03-28T22:33:35.003772+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:24:52.549490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":362,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.375714+0000","last_change":"2024-03-28T22:32:07.867336+0000","last_active":"2024-03-28T22:33:36.375714+0000","last_peered":"2024-03-28T22:33:36.375714+0000","last_clean":"2024-03-28T22:33:36.375714+0000","last_became_active":"2024-03-28T22:32:07.866819+0000","last_became_peered":"2024-03-28T22:32:07.866819+0000","last_unstale":"2024-03-28T22:33:36.375714+0000","last_undegraded":"2024-03-28T22:33:36.375714+0000","last_fullsized":"2024-03-28T22:33:36.375714+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:28:14.713501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:28:14.713501+0000","last_clean_scrub_stamp":"2024-03-28T22:28:14.713501+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:00:52.438708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.536862+0000","last_change":"2024-03-28T22:33:34.020668+0000","last_active":"2024-03-28T22:33:35.536862+0000","last_peered":"2024-03-28T22:33:35.536862+0000","last_clean":"2024-03-28T22:33:35.536862+0000","last_became_active":"2024-03-28T22:33:34.020240+0000","last_became_peered":"2024-03-28T22:33:34.020240+0000","last_unstale":"2024-03-28T22:33:35.536862+0000","last_undegraded":"2024-03-28T22:33:35.536862+0000","last_fullsized":"2024-03-28T22:33:35.536862+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:18.983983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.012960+0000","last_change":"2024-03-28T22:33:28.935237+0000","last_active":"2024-03-28T22:33:36.012960+0000","last_peered":"2024-03-28T22:33:36.012960+0000","last_clean":"2024-03-28T22:33:36.012960+0000","last_became_active":"2024-03-28T22:33:28.934875+0000","last_became_peered":"2024-03-28T22:33:28.934875+0000","last_unstale":"2024-03-28T22:33:36.012960+0000","last_undegraded":"2024-03-28T22:33:36.012960+0000","last_fullsized":"2024-03-28T22:33:36.012960+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:20:50.789302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"92'3","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:03.151431+0000","last_change":"2024-03-28T22:33:28.441654+0000","last_active":"2024-03-28T22:34:03.151431+0000","last_peered":"2024-03-28T22:34:03.151431+0000","last_clean":"2024-03-28T22:34:03.151431+0000","last_became_active":"2024-03-28T22:33:28.441293+0000","last_became_peered":"2024-03-28T22:33:28.441293+0000","last_unstale":"2024-03-28T22:34:03.151431+0000","last_undegraded":"2024-03-28T22:34:03.151431+0000","last_fullsized":"2024-03-28T22:34:03.151431+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:31:17.737977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.011837+0000","last_change":"2024-03-28T22:33:34.018126+0000","last_active":"2024-03-28T22:33:36.011837+0000","last_peered":"2024-03-28T22:33:36.011837+0000","last_clean":"2024-03-28T22:33:36.011837+0000","last_became_active":"2024-03-28T22:33:34.017573+0000","last_became_peered":"2024-03-28T22:33:34.017573+0000","last_unstale":"2024-03-28T22:33:36.011837+0000","last_undegraded":"2024-03-28T22:33:36.011837+0000","last_fullsized":"2024-03-28T22:33:36.011837+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:23:31.013511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"92'1","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.004540+0000","last_change":"2024-03-28T22:33:28.454069+0000","last_active":"2024-03-28T22:33:35.004540+0000","last_peered":"2024-03-28T22:33:35.004540+0000","last_clean":"2024-03-28T22:33:35.004540+0000","last_became_active":"2024-03-28T22:33:28.453754+0000","last_became_peered":"2024-03-28T22:33:28.453754+0000","last_unstale":"2024-03-28T22:33:35.004540+0000","last_undegraded":"2024-03-28T22:33:35.004540+0000","last_fullsized":"2024-03-28T22:33:35.004540+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:23:33.981072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.013496+0000","last_change":"2024-03-28T22:33:28.934390+0000","last_active":"2024-03-28T22:33:36.013496+0000","last_peered":"2024-03-28T22:33:36.013496+0000","last_clean":"2024-03-28T22:33:36.013496+0000","last_became_active":"2024-03-28T22:33:28.933919+0000","last_became_peered":"2024-03-28T22:33:28.933919+0000","last_unstale":"2024-03-28T22:33:36.013496+0000","last_undegraded":"2024-03-28T22:33:36.013496+0000","last_fullsized":"2024-03-28T22:33:36.013496+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:27:07.595144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.377785+0000","last_change":"2024-03-28T22:33:34.019910+0000","last_active":"2024-03-28T22:33:36.377785+0000","last_peered":"2024-03-28T22:33:36.377785+0000","last_clean":"2024-03-28T22:33:36.377785+0000","last_became_active":"2024-03-28T22:33:34.019522+0000","last_became_peered":"2024-03-28T22:33:34.019522+0000","last_unstale":"2024-03-28T22:33:36.377785+0000","last_undegraded":"2024-03-28T22:33:36.377785+0000","last_fullsized":"2024-03-28T22:33:36.377785+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:01:29.298513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"87'1","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.008906+0000","last_change":"2024-03-28T22:33:28.444513+0000","last_active":"2024-03-28T22:33:35.008906+0000","last_peered":"2024-03-28T22:33:35.008906+0000","last_clean":"2024-03-28T22:33:35.008906+0000","last_became_active":"2024-03-28T22:33:28.444091+0000","last_became_peered":"2024-03-28T22:33:28.444091+0000","last_unstale":"2024-03-28T22:33:35.008906+0000","last_undegraded":"2024-03-28T22:33:35.008906+0000","last_fullsized":"2024-03-28T22:33:35.008906+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:49:58.331864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.379248+0000","last_change":"2024-03-28T22:33:28.948390+0000","last_active":"2024-03-28T22:33:36.379248+0000","last_peered":"2024-03-28T22:33:36.379248+0000","last_clean":"2024-03-28T22:33:36.379248+0000","last_became_active":"2024-03-28T22:33:28.942846+0000","last_became_peered":"2024-03-28T22:33:28.942846+0000","last_unstale":"2024-03-28T22:33:36.379248+0000","last_undegraded":"2024-03-28T22:33:36.379248+0000","last_fullsized":"2024-03-28T22:33:36.379248+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:50.596365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376293+0000","last_change":"2024-03-28T22:33:33.999429+0000","last_active":"2024-03-28T22:33:36.376293+0000","last_peered":"2024-03-28T22:33:36.376293+0000","last_clean":"2024-03-28T22:33:36.376293+0000","last_became_active":"2024-03-28T22:33:33.998979+0000","last_became_peered":"2024-03-28T22:33:33.998979+0000","last_unstale":"2024-03-28T22:33:36.376293+0000","last_undegraded":"2024-03-28T22:33:36.376293+0000","last_fullsized":"2024-03-28T22:33:36.376293+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:26.450731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"86'2","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.004341+0000","last_change":"2024-03-28T22:33:28.442861+0000","last_active":"2024-03-28T22:33:35.004341+0000","last_peered":"2024-03-28T22:33:35.004341+0000","last_clean":"2024-03-28T22:33:35.004341+0000","last_became_active":"2024-03-28T22:33:28.442468+0000","last_became_peered":"2024-03-28T22:33:28.442468+0000","last_unstale":"2024-03-28T22:33:35.004341+0000","last_undegraded":"2024-03-28T22:33:35.004341+0000","last_fullsized":"2024-03-28T22:33:35.004341+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:45:49.899031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.012371+0000","last_change":"2024-03-28T22:33:28.957197+0000","last_active":"2024-03-28T22:33:36.012371+0000","last_peered":"2024-03-28T22:33:36.012371+0000","last_clean":"2024-03-28T22:33:36.012371+0000","last_became_active":"2024-03-28T22:33:28.956836+0000","last_became_peered":"2024-03-28T22:33:28.956836+0000","last_unstale":"2024-03-28T22:33:36.012371+0000","last_undegraded":"2024-03-28T22:33:36.012371+0000","last_fullsized":"2024-03-28T22:33:36.012371+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:25:40.765500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.004336+0000","last_change":"2024-03-28T22:33:34.013826+0000","last_active":"2024-03-28T22:33:35.004336+0000","last_peered":"2024-03-28T22:33:35.004336+0000","last_clean":"2024-03-28T22:33:35.004336+0000","last_became_active":"2024-03-28T22:33:34.013415+0000","last_became_peered":"2024-03-28T22:33:34.013415+0000","last_unstale":"2024-03-28T22:33:35.004336+0000","last_undegraded":"2024-03-28T22:33:35.004336+0000","last_fullsized":"2024-03-28T22:33:35.004336+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:19:30.965940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"92'2","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:02.136787+0000","last_change":"2024-03-28T22:33:28.442225+0000","last_active":"2024-03-28T22:34:02.136787+0000","last_peered":"2024-03-28T22:34:02.136787+0000","last_clean":"2024-03-28T22:34:02.136787+0000","last_became_active":"2024-03-28T22:33:28.441844+0000","last_became_peered":"2024-03-28T22:33:28.441844+0000","last_unstale":"2024-03-28T22:34:02.136787+0000","last_undegraded":"2024-03-28T22:34:02.136787+0000","last_fullsized":"2024-03-28T22:34:02.136787+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:40:52.108663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.004042+0000","last_change":"2024-03-28T22:33:28.927571+0000","last_active":"2024-03-28T22:33:35.004042+0000","last_peered":"2024-03-28T22:33:35.004042+0000","last_clean":"2024-03-28T22:33:35.004042+0000","last_became_active":"2024-03-28T22:33:28.927275+0000","last_became_peered":"2024-03-28T22:33:28.927275+0000","last_unstale":"2024-03-28T22:33:35.004042+0000","last_undegraded":"2024-03-28T22:33:35.004042+0000","last_fullsized":"2024-03-28T22:33:35.004042+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:25:16.184533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.009016+0000","last_change":"2024-03-28T22:33:34.018429+0000","last_active":"2024-03-28T22:33:35.009016+0000","last_peered":"2024-03-28T22:33:35.009016+0000","last_clean":"2024-03-28T22:33:35.009016+0000","last_became_active":"2024-03-28T22:33:34.017066+0000","last_became_peered":"2024-03-28T22:33:34.017066+0000","last_unstale":"2024-03-28T22:33:35.009016+0000","last_undegraded":"2024-03-28T22:33:35.009016+0000","last_fullsized":"2024-03-28T22:33:35.009016+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:35:42.101072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.377872+0000","last_change":"2024-03-28T22:33:28.437760+0000","last_active":"2024-03-28T22:33:36.377872+0000","last_peered":"2024-03-28T22:33:36.377872+0000","last_clean":"2024-03-28T22:33:36.377872+0000","last_became_active":"2024-03-28T22:33:28.437096+0000","last_became_peered":"2024-03-28T22:33:28.437096+0000","last_unstale":"2024-03-28T22:33:36.377872+0000","last_undegraded":"2024-03-28T22:33:36.377872+0000","last_fullsized":"2024-03-28T22:33:36.377872+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:29:20.778551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.011450+0000","last_change":"2024-03-28T22:33:28.935805+0000","last_active":"2024-03-28T22:33:36.011450+0000","last_peered":"2024-03-28T22:33:36.011450+0000","last_clean":"2024-03-28T22:33:36.011450+0000","last_became_active":"2024-03-28T22:33:28.935520+0000","last_became_peered":"2024-03-28T22:33:28.935520+0000","last_unstale":"2024-03-28T22:33:36.011450+0000","last_undegraded":"2024-03-28T22:33:36.011450+0000","last_fullsized":"2024-03-28T22:33:36.011450+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:23.414310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.004120+0000","last_change":"2024-03-28T22:33:34.029518+0000","last_active":"2024-03-28T22:33:35.004120+0000","last_peered":"2024-03-28T22:33:35.004120+0000","last_clean":"2024-03-28T22:33:35.004120+0000","last_became_active":"2024-03-28T22:33:34.029101+0000","last_became_peered":"2024-03-28T22:33:34.029101+0000","last_unstale":"2024-03-28T22:33:35.004120+0000","last_undegraded":"2024-03-28T22:33:35.004120+0000","last_fullsized":"2024-03-28T22:33:35.004120+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:05.595543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376270+0000","last_change":"2024-03-28T22:33:28.438857+0000","last_active":"2024-03-28T22:33:36.376270+0000","last_peered":"2024-03-28T22:33:36.376270+0000","last_clean":"2024-03-28T22:33:36.376270+0000","last_became_active":"2024-03-28T22:33:28.438528+0000","last_became_peered":"2024-03-28T22:33:28.438528+0000","last_unstale":"2024-03-28T22:33:36.376270+0000","last_undegraded":"2024-03-28T22:33:36.376270+0000","last_fullsized":"2024-03-28T22:33:36.376270+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:07:54.506739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.378483+0000","last_change":"2024-03-28T22:33:28.958271+0000","last_active":"2024-03-28T22:33:36.378483+0000","last_peered":"2024-03-28T22:33:36.378483+0000","last_clean":"2024-03-28T22:33:36.378483+0000","last_became_active":"2024-03-28T22:33:28.957922+0000","last_became_peered":"2024-03-28T22:33:28.957922+0000","last_unstale":"2024-03-28T22:33:36.378483+0000","last_undegraded":"2024-03-28T22:33:36.378483+0000","last_fullsized":"2024-03-28T22:33:36.378483+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:33:19.003131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.004080+0000","last_change":"2024-03-28T22:33:34.000789+0000","last_active":"2024-03-28T22:33:35.004080+0000","last_peered":"2024-03-28T22:33:35.004080+0000","last_clean":"2024-03-28T22:33:35.004080+0000","last_became_active":"2024-03-28T22:33:34.000383+0000","last_became_peered":"2024-03-28T22:33:34.000383+0000","last_unstale":"2024-03-28T22:33:35.004080+0000","last_undegraded":"2024-03-28T22:33:35.004080+0000","last_fullsized":"2024-03-28T22:33:35.004080+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:50:29.925656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:02.132519+0000","last_change":"2024-03-28T22:33:28.439121+0000","last_active":"2024-03-28T22:34:02.132519+0000","last_peered":"2024-03-28T22:34:02.132519+0000","last_clean":"2024-03-28T22:34:02.132519+0000","last_became_active":"2024-03-28T22:33:28.438800+0000","last_became_peered":"2024-03-28T22:33:28.438800+0000","last_unstale":"2024-03-28T22:34:02.132519+0000","last_undegraded":"2024-03-28T22:34:02.132519+0000","last_fullsized":"2024-03-28T22:34:02.132519+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:57:10.013246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376130+0000","last_change":"2024-03-28T22:33:28.934499+0000","last_active":"2024-03-28T22:33:36.376130+0000","last_peered":"2024-03-28T22:33:36.376130+0000","last_clean":"2024-03-28T22:33:36.376130+0000","last_became_active":"2024-03-28T22:33:28.934147+0000","last_became_peered":"2024-03-28T22:33:28.934147+0000","last_unstale":"2024-03-28T22:33:36.376130+0000","last_undegraded":"2024-03-28T22:33:36.376130+0000","last_fullsized":"2024-03-28T22:33:36.376130+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:35:39.090052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.374902+0000","last_change":"2024-03-28T22:33:34.017946+0000","last_active":"2024-03-28T22:33:36.374902+0000","last_peered":"2024-03-28T22:33:36.374902+0000","last_clean":"2024-03-28T22:33:36.374902+0000","last_became_active":"2024-03-28T22:33:34.017546+0000","last_became_peered":"2024-03-28T22:33:34.017546+0000","last_unstale":"2024-03-28T22:33:36.374902+0000","last_undegraded":"2024-03-28T22:33:36.374902+0000","last_fullsized":"2024-03-28T22:33:36.374902+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:14:39.081677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"92'3","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:03.154378+0000","last_change":"2024-03-28T22:33:28.453387+0000","last_active":"2024-03-28T22:34:03.154378+0000","last_peered":"2024-03-28T22:34:03.154378+0000","last_clean":"2024-03-28T22:34:03.154378+0000","last_became_active":"2024-03-28T22:33:28.453029+0000","last_became_peered":"2024-03-28T22:33:28.453029+0000","last_unstale":"2024-03-28T22:34:03.154378+0000","last_undegraded":"2024-03-28T22:34:03.154378+0000","last_fullsized":"2024-03-28T22:34:03.154378+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:12:33.246863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376535+0000","last_change":"2024-03-28T22:33:28.923819+0000","last_active":"2024-03-28T22:33:36.376535+0000","last_peered":"2024-03-28T22:33:36.376535+0000","last_clean":"2024-03-28T22:33:36.376535+0000","last_became_active":"2024-03-28T22:33:28.923471+0000","last_became_peered":"2024-03-28T22:33:28.923471+0000","last_unstale":"2024-03-28T22:33:36.376535+0000","last_undegraded":"2024-03-28T22:33:36.376535+0000","last_fullsized":"2024-03-28T22:33:36.376535+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:58:03.496406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.011014+0000","last_change":"2024-03-28T22:33:34.029789+0000","last_active":"2024-03-28T22:33:36.011014+0000","last_peered":"2024-03-28T22:33:36.011014+0000","last_clean":"2024-03-28T22:33:36.011014+0000","last_became_active":"2024-03-28T22:33:34.029231+0000","last_became_peered":"2024-03-28T22:33:34.029231+0000","last_unstale":"2024-03-28T22:33:36.011014+0000","last_undegraded":"2024-03-28T22:33:36.011014+0000","last_fullsized":"2024-03-28T22:33:36.011014+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:47:50.101393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.536756+0000","last_change":"2024-03-28T22:33:28.435668+0000","last_active":"2024-03-28T22:33:35.536756+0000","last_peered":"2024-03-28T22:33:35.536756+0000","last_clean":"2024-03-28T22:33:35.536756+0000","last_became_active":"2024-03-28T22:33:28.435325+0000","last_became_peered":"2024-03-28T22:33:28.435325+0000","last_unstale":"2024-03-28T22:33:35.536756+0000","last_undegraded":"2024-03-28T22:33:35.536756+0000","last_fullsized":"2024-03-28T22:33:35.536756+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:13:06.281074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376264+0000","last_change":"2024-03-28T22:33:28.954138+0000","last_active":"2024-03-28T22:33:36.376264+0000","last_peered":"2024-03-28T22:33:36.376264+0000","last_clean":"2024-03-28T22:33:36.376264+0000","last_became_active":"2024-03-28T22:33:28.953752+0000","last_became_peered":"2024-03-28T22:33:28.953752+0000","last_unstale":"2024-03-28T22:33:36.376264+0000","last_undegraded":"2024-03-28T22:33:36.376264+0000","last_fullsized":"2024-03-28T22:33:36.376264+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:53:34.892316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.536052+0000","last_change":"2024-03-28T22:33:34.002672+0000","last_active":"2024-03-28T22:33:35.536052+0000","last_peered":"2024-03-28T22:33:35.536052+0000","last_clean":"2024-03-28T22:33:35.536052+0000","last_became_active":"2024-03-28T22:33:34.002195+0000","last_became_peered":"2024-03-28T22:33:34.002195+0000","last_unstale":"2024-03-28T22:33:35.536052+0000","last_undegraded":"2024-03-28T22:33:35.536052+0000","last_fullsized":"2024-03-28T22:33:35.536052+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:24:11.521474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.013738+0000","last_change":"2024-03-28T22:33:28.450818+0000","last_active":"2024-03-28T22:33:36.013738+0000","last_peered":"2024-03-28T22:33:36.013738+0000","last_clean":"2024-03-28T22:33:36.013738+0000","last_became_active":"2024-03-28T22:33:28.448592+0000","last_became_peered":"2024-03-28T22:33:28.448592+0000","last_unstale":"2024-03-28T22:33:36.013738+0000","last_undegraded":"2024-03-28T22:33:36.013738+0000","last_fullsized":"2024-03-28T22:33:36.013738+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:19:27.126306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376450+0000","last_change":"2024-03-28T22:33:28.925191+0000","last_active":"2024-03-28T22:33:36.376450+0000","last_peered":"2024-03-28T22:33:36.376450+0000","last_clean":"2024-03-28T22:33:36.376450+0000","last_became_active":"2024-03-28T22:33:28.924424+0000","last_became_peered":"2024-03-28T22:33:28.924424+0000","last_unstale":"2024-03-28T22:33:36.376450+0000","last_undegraded":"2024-03-28T22:33:36.376450+0000","last_fullsized":"2024-03-28T22:33:36.376450+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:50:00.415047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.011619+0000","last_change":"2024-03-28T22:33:28.925110+0000","last_active":"2024-03-28T22:33:36.011619+0000","last_peered":"2024-03-28T22:33:36.011619+0000","last_clean":"2024-03-28T22:33:36.011619+0000","last_became_active":"2024-03-28T22:33:28.924745+0000","last_became_peered":"2024-03-28T22:33:28.924745+0000","last_unstale":"2024-03-28T22:33:36.011619+0000","last_undegraded":"2024-03-28T22:33:36.011619+0000","last_fullsized":"2024-03-28T22:33:36.011619+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:23:06.331043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376687+0000","last_change":"2024-03-28T22:33:28.927068+0000","last_active":"2024-03-28T22:33:36.376687+0000","last_peered":"2024-03-28T22:33:36.376687+0000","last_clean":"2024-03-28T22:33:36.376687+0000","last_became_active":"2024-03-28T22:33:28.926734+0000","last_became_peered":"2024-03-28T22:33:28.926734+0000","last_unstale":"2024-03-28T22:33:36.376687+0000","last_undegraded":"2024-03-28T22:33:36.376687+0000","last_fullsized":"2024-03-28T22:33:36.376687+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:56.309555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376374+0000","last_change":"2024-03-28T22:33:28.955675+0000","last_active":"2024-03-28T22:33:36.376374+0000","last_peered":"2024-03-28T22:33:36.376374+0000","last_clean":"2024-03-28T22:33:36.376374+0000","last_became_active":"2024-03-28T22:33:28.955329+0000","last_became_peered":"2024-03-28T22:33:28.955329+0000","last_unstale":"2024-03-28T22:33:36.376374+0000","last_undegraded":"2024-03-28T22:33:36.376374+0000","last_fullsized":"2024-03-28T22:33:36.376374+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:34:14.166434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.011204+0000","last_change":"2024-03-28T22:33:28.955125+0000","last_active":"2024-03-28T22:33:36.011204+0000","last_peered":"2024-03-28T22:33:36.011204+0000","last_clean":"2024-03-28T22:33:36.011204+0000","last_became_active":"2024-03-28T22:33:28.954434+0000","last_became_peered":"2024-03-28T22:33:28.954434+0000","last_unstale":"2024-03-28T22:33:36.011204+0000","last_undegraded":"2024-03-28T22:33:36.011204+0000","last_fullsized":"2024-03-28T22:33:36.011204+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:56:50.705012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.013636+0000","last_change":"2024-03-28T22:33:28.945230+0000","last_active":"2024-03-28T22:33:36.013636+0000","last_peered":"2024-03-28T22:33:36.013636+0000","last_clean":"2024-03-28T22:33:36.013636+0000","last_became_active":"2024-03-28T22:33:28.944921+0000","last_became_peered":"2024-03-28T22:33:28.944921+0000","last_unstale":"2024-03-28T22:33:36.013636+0000","last_undegraded":"2024-03-28T22:33:36.013636+0000","last_fullsized":"2024-03-28T22:33:36.013636+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:26:26.505774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.011851+0000","last_change":"2024-03-28T22:33:28.928720+0000","last_active":"2024-03-28T22:33:36.011851+0000","last_peered":"2024-03-28T22:33:36.011851+0000","last_clean":"2024-03-28T22:33:36.011851+0000","last_became_active":"2024-03-28T22:33:28.928415+0000","last_became_peered":"2024-03-28T22:33:28.928415+0000","last_unstale":"2024-03-28T22:33:36.011851+0000","last_undegraded":"2024-03-28T22:33:36.011851+0000","last_fullsized":"2024-03-28T22:33:36.011851+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:00.839609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376804+0000","last_change":"2024-03-28T22:33:28.924948+0000","last_active":"2024-03-28T22:33:36.376804+0000","last_peered":"2024-03-28T22:33:36.376804+0000","last_clean":"2024-03-28T22:33:36.376804+0000","last_became_active":"2024-03-28T22:33:28.924631+0000","last_became_peered":"2024-03-28T22:33:28.924631+0000","last_unstale":"2024-03-28T22:33:36.376804+0000","last_undegraded":"2024-03-28T22:33:36.376804+0000","last_fullsized":"2024-03-28T22:33:36.376804+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:53:44.909991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.379420+0000","last_change":"2024-03-28T22:33:28.948725+0000","last_active":"2024-03-28T22:33:36.379420+0000","last_peered":"2024-03-28T22:33:36.379420+0000","last_clean":"2024-03-28T22:33:36.379420+0000","last_became_active":"2024-03-28T22:33:28.943942+0000","last_became_peered":"2024-03-28T22:33:28.943942+0000","last_unstale":"2024-03-28T22:33:36.379420+0000","last_undegraded":"2024-03-28T22:33:36.379420+0000","last_fullsized":"2024-03-28T22:33:36.379420+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:50:04.238538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.536156+0000","last_change":"2024-03-28T22:33:28.941546+0000","last_active":"2024-03-28T22:33:35.536156+0000","last_peered":"2024-03-28T22:33:35.536156+0000","last_clean":"2024-03-28T22:33:35.536156+0000","last_became_active":"2024-03-28T22:33:28.941278+0000","last_became_peered":"2024-03-28T22:33:28.941278+0000","last_unstale":"2024-03-28T22:33:35.536156+0000","last_undegraded":"2024-03-28T22:33:35.536156+0000","last_fullsized":"2024-03-28T22:33:35.536156+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:47:10.047067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.378112+0000","last_change":"2024-03-28T22:33:28.949552+0000","last_active":"2024-03-28T22:33:36.378112+0000","last_peered":"2024-03-28T22:33:36.378112+0000","last_clean":"2024-03-28T22:33:36.378112+0000","last_became_active":"2024-03-28T22:33:28.947432+0000","last_became_peered":"2024-03-28T22:33:28.947432+0000","last_unstale":"2024-03-28T22:33:36.378112+0000","last_undegraded":"2024-03-28T22:33:36.378112+0000","last_fullsized":"2024-03-28T22:33:36.378112+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:01:16.534642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.375153+0000","last_change":"2024-03-28T22:33:28.953930+0000","last_active":"2024-03-28T22:33:36.375153+0000","last_peered":"2024-03-28T22:33:36.375153+0000","last_clean":"2024-03-28T22:33:36.375153+0000","last_became_active":"2024-03-28T22:33:28.953591+0000","last_became_peered":"2024-03-28T22:33:28.953591+0000","last_unstale":"2024-03-28T22:33:36.375153+0000","last_undegraded":"2024-03-28T22:33:36.375153+0000","last_fullsized":"2024-03-28T22:33:36.375153+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:42:55.857476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.013763+0000","last_change":"2024-03-28T22:33:28.946307+0000","last_active":"2024-03-28T22:33:36.013763+0000","last_peered":"2024-03-28T22:33:36.013763+0000","last_clean":"2024-03-28T22:33:36.013763+0000","last_became_active":"2024-03-28T22:33:28.945932+0000","last_became_peered":"2024-03-28T22:33:28.945932+0000","last_unstale":"2024-03-28T22:33:36.013763+0000","last_undegraded":"2024-03-28T22:33:36.013763+0000","last_fullsized":"2024-03-28T22:33:36.013763+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:54:59.651668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.011690+0000","last_change":"2024-03-28T22:33:28.928014+0000","last_active":"2024-03-28T22:33:36.011690+0000","last_peered":"2024-03-28T22:33:36.011690+0000","last_clean":"2024-03-28T22:33:36.011690+0000","last_became_active":"2024-03-28T22:33:28.927686+0000","last_became_peered":"2024-03-28T22:33:28.927686+0000","last_unstale":"2024-03-28T22:33:36.011690+0000","last_undegraded":"2024-03-28T22:33:36.011690+0000","last_fullsized":"2024-03-28T22:33:36.011690+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:47:08.131565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5536,"num_objects":43,"num_object_clones":0,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":20058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318292,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1019904,"data_stored":496542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579929,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28644,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716508,"statfs":{"total":95995035648,"available":95965704192,"internally_reserved":0,"allocated":1667072,"data_stored":1092454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743455,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1024000,"data_stored":497124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005093,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":497168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299435,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32768,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712384,"statfs":{"total":95995035648,"available":95961481216,"internally_reserved":0,"allocated":1630208,"data_stored":1088702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495664,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1019904,"data_stored":496542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1036288,"data_stored":500966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738449,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":498020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411401,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1056768,"data_stored":501734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182464,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1028096,"data_stored":496646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920827,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":496564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757302,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28608,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716544,"statfs":{"total":95995035648,"available":95965741056,"internally_reserved":0,"allocated":1626112,"data_stored":1087092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":5544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T22:34:08.145 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T22:34:08.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.405+0000 7f3ad7e00640 1 -- 172.21.15.43:0/3859362155 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3ad8103820 msgr2=0x7f3ad8103c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:08.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.405+0000 7f3ad7e00640 1 --2- 172.21.15.43:0/3859362155 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3ad8103820 0x7f3ad8103c40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3ac8009900 tx=0x7f3ac802bc30 comp rx=0 tx=0).stop 2024-03-28T22:34:08.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.405+0000 7f3ad7e00640 1 -- 172.21.15.43:0/3859362155 shutdown_connections 2024-03-28T22:34:08.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.405+0000 7f3ad7e00640 1 --2- 172.21.15.43:0/3859362155 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ad81053d0 0x7f3ad810bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:08.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.405+0000 7f3ad7e00640 1 --2- 172.21.15.43:0/3859362155 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ad8104a10 0x7f3ad8104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:08.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.405+0000 7f3ad7e00640 1 --2- 172.21.15.43:0/3859362155 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3ad8103820 0x7f3ad8103c40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:08.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.405+0000 7f3ad7e00640 1 -- 172.21.15.43:0/3859362155 >> 172.21.15.43:0/3859362155 conn(0x7f3ad80fed90 msgr2=0x7f3ad81011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:08.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.405+0000 7f3ad7e00640 1 -- 172.21.15.43:0/3859362155 shutdown_connections 2024-03-28T22:34:08.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.405+0000 7f3ad7e00640 1 -- 172.21.15.43:0/3859362155 wait complete. 2024-03-28T22:34:08.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.406+0000 7f3ad7e00640 1 Processor -- start 2024-03-28T22:34:08.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.407+0000 7f3ad7e00640 1 -- start start 2024-03-28T22:34:08.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.407+0000 7f3ad7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ad8103820 0x7f3ad8079500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:08.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.407+0000 7f3ad7e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3ad8104a10 0x7f3ad8079a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:08.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.407+0000 7f3ad7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ad81053d0 0x7f3ad8076070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:08.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.407+0000 7f3ad7e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad810f450 con 0x7f3ad8103820 2024-03-28T22:34:08.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.407+0000 7f3ad7e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad810f2d0 con 0x7f3ad8104a10 2024-03-28T22:34:08.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.407+0000 7f3ad7e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad810f5d0 con 0x7f3ad81053d0 2024-03-28T22:34:08.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.407+0000 7f3ad6000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3ad8104a10 0x7f3ad8079a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:08.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.407+0000 7f3ad6a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ad8103820 0x7f3ad8079500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:08.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.407+0000 7f3ad6000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3ad8104a10 0x7f3ad8079a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:58470/0 (socket says 172.21.15.43:58470) 2024-03-28T22:34:08.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.407+0000 7f3ad6000640 1 -- 172.21.15.43:0/3331284428 learned_addr learned my addr 172.21.15.43:0/3331284428 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:34:08.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.407+0000 7f3ad7400640 1 --2- 172.21.15.43:0/3331284428 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ad81053d0 0x7f3ad8076070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:08.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.408+0000 7f3ad6000640 1 -- 172.21.15.43:0/3331284428 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ad81053d0 msgr2=0x7f3ad8076070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:08.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.408+0000 7f3ad6000640 1 --2- 172.21.15.43:0/3331284428 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ad81053d0 0x7f3ad8076070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:08.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.408+0000 7f3ad6000640 1 -- 172.21.15.43:0/3331284428 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ad8103820 msgr2=0x7f3ad8079500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:08.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.408+0000 7f3ad6000640 1 --2- 172.21.15.43:0/3331284428 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ad8103820 0x7f3ad8079500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:08.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.408+0000 7f3ad6000640 1 -- 172.21.15.43:0/3331284428 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ad8076930 con 0x7f3ad8104a10 2024-03-28T22:34:08.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.408+0000 7f3ad6000640 1 --2- 172.21.15.43:0/3331284428 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3ad8104a10 0x7f3ad8079a40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3ac400e9f0 tx=0x7f3ac400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:08.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.408+0000 7f3acf400640 1 -- 172.21.15.43:0/3331284428 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac400cd80 con 0x7f3ad8104a10 2024-03-28T22:34:08.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.408+0000 7f3acf400640 1 -- 172.21.15.43:0/3331284428 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ac4010910 con 0x7f3ad8104a10 2024-03-28T22:34:08.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.408+0000 7f3acf400640 1 -- 172.21.15.43:0/3331284428 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac4018990 con 0x7f3ad8104a10 2024-03-28T22:34:08.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.408+0000 7f3ad7e00640 1 -- 172.21.15.43:0/3331284428 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ad81a5f10 con 0x7f3ad8104a10 2024-03-28T22:34:08.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.409+0000 7f3ad7e00640 1 -- 172.21.15.43:0/3331284428 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ad81a6310 con 0x7f3ad8104a10 2024-03-28T22:34:08.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.410+0000 7f3ad7e00640 1 -- 172.21.15.43:0/3331284428 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ad80688e0 con 0x7f3ad8104a10 2024-03-28T22:34:08.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.412+0000 7f3acf400640 1 -- 172.21.15.43:0/3331284428 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f3ac4010ac0 con 0x7f3ad8104a10 2024-03-28T22:34:08.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.413+0000 7f3acf400640 1 --2- 172.21.15.43:0/3331284428 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3abc0a1e90 0x7f3abc0a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:08.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.414+0000 7f3acf400640 1 -- 172.21.15.43:0/3331284428 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f3ac40ce480 con 0x7f3ad8104a10 2024-03-28T22:34:08.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.414+0000 7f3ad6a00640 1 --2- 172.21.15.43:0/3331284428 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3abc0a1e90 0x7f3abc0a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:08.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.415+0000 7f3ad6a00640 1 --2- 172.21.15.43:0/3331284428 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3abc0a1e90 0x7f3abc0a4350 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3ac802c170 tx=0x7f3ac802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:08.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.417+0000 7f3acf400640 1 -- 172.21.15.43:0/3331284428 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3ad80688e0 con 0x7f3ad8104a10 2024-03-28T22:34:08.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:08 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2868822561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:34:08.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:08 smithi082 ceph-mon[36471]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-28T22:34:08.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:08 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/529190013' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:34:08.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:08 smithi082 ceph-mon[36471]: from='client.24860 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:34:08.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:08 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2868822561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:34:08.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:08 smithi043 ceph-mon[31823]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-28T22:34:08.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:08 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/529190013' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:34:08.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:08 smithi043 ceph-mon[31823]: from='client.24860 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:34:08.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.624+0000 7f3ad7e00640 1 -- 172.21.15.43:0/3331284428 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3ad8061b90 con 0x7f3abc0a1e90 2024-03-28T22:34:08.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.640+0000 7f3acf400640 1 -- 172.21.15.43:0/3331284428 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340393 (secure 0 0 0) 0x7f3ad8061b90 con 0x7f3abc0a1e90 2024-03-28T22:34:08.641 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:34:08.646 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-03-28T22:34:08.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:08 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2868822561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:34:08.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:08 smithi172 ceph-mon[36566]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-28T22:34:08.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:08 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/529190013' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:34:08.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:08 smithi172 ceph-mon[36566]: from='client.24860 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:34:08.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.648+0000 7f3ad7e00640 1 -- 172.21.15.43:0/3331284428 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3abc0a1e90 msgr2=0x7f3abc0a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:08.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.648+0000 7f3ad7e00640 1 --2- 172.21.15.43:0/3331284428 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3abc0a1e90 0x7f3abc0a4350 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3ac802c170 tx=0x7f3ac802df90 comp rx=0 tx=0).stop 2024-03-28T22:34:08.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.648+0000 7f3ad7e00640 1 -- 172.21.15.43:0/3331284428 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3ad8104a10 msgr2=0x7f3ad8079a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:08.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.648+0000 7f3ad7e00640 1 --2- 172.21.15.43:0/3331284428 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3ad8104a10 0x7f3ad8079a40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3ac400e9f0 tx=0x7f3ac400eeb0 comp rx=0 tx=0).stop 2024-03-28T22:34:08.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.649+0000 7f3ad7e00640 1 -- 172.21.15.43:0/3331284428 shutdown_connections 2024-03-28T22:34:08.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.649+0000 7f3ad7e00640 1 --2- 172.21.15.43:0/3331284428 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3abc0a1e90 0x7f3abc0a4350 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:08.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.649+0000 7f3ad7e00640 1 --2- 172.21.15.43:0/3331284428 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3ad8104a10 0x7f3ad8079a40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:08.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.649+0000 7f3ad7e00640 1 --2- 172.21.15.43:0/3331284428 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ad81053d0 0x7f3ad8076070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:08.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.649+0000 7f3ad7e00640 1 --2- 172.21.15.43:0/3331284428 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ad8103820 0x7f3ad8079500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:08.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.649+0000 7f3ad7e00640 1 -- 172.21.15.43:0/3331284428 >> 172.21.15.43:0/3331284428 conn(0x7f3ad80fed90 msgr2=0x7f3ad8107d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:08.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.649+0000 7f3ad7e00640 1 -- 172.21.15.43:0/3331284428 shutdown_connections 2024-03-28T22:34:08.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:08.649+0000 7f3ad7e00640 1 -- 172.21.15.43:0/3331284428 wait complete. 2024-03-28T22:34:08.671 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-03-28T22:34:08.164232+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":599258,"num_objects":82,"num_object_clones":0,"num_object_copies":247,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":158,"num_read_kb":139,"num_write":359,"num_write_kb":3004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":290,"ondisk_log_size":290,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":371492,"kb_used_data":14020,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124570332,"statfs":{"total":1151940427776,"available":1151560019968,"internally_reserved":0,"allocated":14356480,"data_stored":7768329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":365933882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4250,"num_objects":58,"num_object_clones":0,"num_object_copies":175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":22,"num_read_kb":20,"num_write":129,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005829"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.012086+0000","last_change":"2024-03-28T22:33:28.942904+0000","last_active":"2024-03-28T22:33:36.012086+0000","last_peered":"2024-03-28T22:33:36.012086+0000","last_clean":"2024-03-28T22:33:36.012086+0000","last_became_active":"2024-03-28T22:33:28.942574+0000","last_became_peered":"2024-03-28T22:33:28.942574+0000","last_unstale":"2024-03-28T22:33:36.012086+0000","last_undegraded":"2024-03-28T22:33:36.012086+0000","last_fullsized":"2024-03-28T22:33:36.012086+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:55.323919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.012364+0000","last_change":"2024-03-28T22:33:28.951795+0000","last_active":"2024-03-28T22:33:36.012364+0000","last_peered":"2024-03-28T22:33:36.012364+0000","last_clean":"2024-03-28T22:33:36.012364+0000","last_became_active":"2024-03-28T22:33:28.951286+0000","last_became_peered":"2024-03-28T22:33:28.951286+0000","last_unstale":"2024-03-28T22:33:36.012364+0000","last_undegraded":"2024-03-28T22:33:36.012364+0000","last_fullsized":"2024-03-28T22:33:36.012364+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:56:11.718244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.011092+0000","last_change":"2024-03-28T22:33:28.949476+0000","last_active":"2024-03-28T22:33:36.011092+0000","last_peered":"2024-03-28T22:33:36.011092+0000","last_clean":"2024-03-28T22:33:36.011092+0000","last_became_active":"2024-03-28T22:33:28.949128+0000","last_became_peered":"2024-03-28T22:33:28.949128+0000","last_unstale":"2024-03-28T22:33:36.011092+0000","last_undegraded":"2024-03-28T22:33:36.011092+0000","last_fullsized":"2024-03-28T22:33:36.011092+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:49:29.052373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.004170+0000","last_change":"2024-03-28T22:33:28.930715+0000","last_active":"2024-03-28T22:33:35.004170+0000","last_peered":"2024-03-28T22:33:35.004170+0000","last_clean":"2024-03-28T22:33:35.004170+0000","last_became_active":"2024-03-28T22:33:28.930265+0000","last_became_peered":"2024-03-28T22:33:28.930265+0000","last_unstale":"2024-03-28T22:33:35.004170+0000","last_undegraded":"2024-03-28T22:33:35.004170+0000","last_fullsized":"2024-03-28T22:33:35.004170+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:29:54.695671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.004184+0000","last_change":"2024-03-28T22:33:28.944933+0000","last_active":"2024-03-28T22:33:35.004184+0000","last_peered":"2024-03-28T22:33:35.004184+0000","last_clean":"2024-03-28T22:33:35.004184+0000","last_became_active":"2024-03-28T22:33:28.944542+0000","last_became_peered":"2024-03-28T22:33:28.944542+0000","last_unstale":"2024-03-28T22:33:35.004184+0000","last_undegraded":"2024-03-28T22:33:35.004184+0000","last_fullsized":"2024-03-28T22:33:35.004184+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:35:15.719312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.141496+0000","last_change":"2024-03-28T22:33:28.927920+0000","last_active":"2024-03-28T22:33:35.141496+0000","last_peered":"2024-03-28T22:33:35.141496+0000","last_clean":"2024-03-28T22:33:35.141496+0000","last_became_active":"2024-03-28T22:33:28.927602+0000","last_became_peered":"2024-03-28T22:33:28.927602+0000","last_unstale":"2024-03-28T22:33:35.141496+0000","last_undegraded":"2024-03-28T22:33:35.141496+0000","last_fullsized":"2024-03-28T22:33:35.141496+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:24:22.072707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.375922+0000","last_change":"2024-03-28T22:33:28.917963+0000","last_active":"2024-03-28T22:33:36.375922+0000","last_peered":"2024-03-28T22:33:36.375922+0000","last_clean":"2024-03-28T22:33:36.375922+0000","last_became_active":"2024-03-28T22:33:28.917628+0000","last_became_peered":"2024-03-28T22:33:28.917628+0000","last_unstale":"2024-03-28T22:33:36.375922+0000","last_undegraded":"2024-03-28T22:33:36.375922+0000","last_fullsized":"2024-03-28T22:33:36.375922+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:37:09.111640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.377217+0000","last_change":"2024-03-28T22:33:28.942632+0000","last_active":"2024-03-28T22:33:36.377217+0000","last_peered":"2024-03-28T22:33:36.377217+0000","last_clean":"2024-03-28T22:33:36.377217+0000","last_became_active":"2024-03-28T22:33:28.941972+0000","last_became_peered":"2024-03-28T22:33:28.941972+0000","last_unstale":"2024-03-28T22:33:36.377217+0000","last_undegraded":"2024-03-28T22:33:36.377217+0000","last_fullsized":"2024-03-28T22:33:36.377217+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:49:52.642314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.375178+0000","last_change":"2024-03-28T22:33:28.924324+0000","last_active":"2024-03-28T22:33:36.375178+0000","last_peered":"2024-03-28T22:33:36.375178+0000","last_clean":"2024-03-28T22:33:36.375178+0000","last_became_active":"2024-03-28T22:33:28.924014+0000","last_became_peered":"2024-03-28T22:33:28.924014+0000","last_unstale":"2024-03-28T22:33:36.375178+0000","last_undegraded":"2024-03-28T22:33:36.375178+0000","last_fullsized":"2024-03-28T22:33:36.375178+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:17:55.172116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.141839+0000","last_change":"2024-03-28T22:33:28.927152+0000","last_active":"2024-03-28T22:33:35.141839+0000","last_peered":"2024-03-28T22:33:35.141839+0000","last_clean":"2024-03-28T22:33:35.141839+0000","last_became_active":"2024-03-28T22:33:28.926782+0000","last_became_peered":"2024-03-28T22:33:28.926782+0000","last_unstale":"2024-03-28T22:33:35.141839+0000","last_undegraded":"2024-03-28T22:33:35.141839+0000","last_fullsized":"2024-03-28T22:33:35.141839+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:33:57.048323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376438+0000","last_change":"2024-03-28T22:33:28.930802+0000","last_active":"2024-03-28T22:33:36.376438+0000","last_peered":"2024-03-28T22:33:36.376438+0000","last_clean":"2024-03-28T22:33:36.376438+0000","last_became_active":"2024-03-28T22:33:28.930475+0000","last_became_peered":"2024-03-28T22:33:28.930475+0000","last_unstale":"2024-03-28T22:33:36.376438+0000","last_undegraded":"2024-03-28T22:33:36.376438+0000","last_fullsized":"2024-03-28T22:33:36.376438+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:48.443803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.004090+0000","last_change":"2024-03-28T22:33:28.927057+0000","last_active":"2024-03-28T22:33:35.004090+0000","last_peered":"2024-03-28T22:33:35.004090+0000","last_clean":"2024-03-28T22:33:35.004090+0000","last_became_active":"2024-03-28T22:33:28.926598+0000","last_became_peered":"2024-03-28T22:33:28.926598+0000","last_unstale":"2024-03-28T22:33:35.004090+0000","last_undegraded":"2024-03-28T22:33:35.004090+0000","last_fullsized":"2024-03-28T22:33:35.004090+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:21:00.710203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.014089+0000","last_change":"2024-03-28T22:33:28.937034+0000","last_active":"2024-03-28T22:33:36.014089+0000","last_peered":"2024-03-28T22:33:36.014089+0000","last_clean":"2024-03-28T22:33:36.014089+0000","last_became_active":"2024-03-28T22:33:28.933713+0000","last_became_peered":"2024-03-28T22:33:28.933713+0000","last_unstale":"2024-03-28T22:33:36.014089+0000","last_undegraded":"2024-03-28T22:33:36.014089+0000","last_fullsized":"2024-03-28T22:33:36.014089+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:58:24.148897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.010710+0000","last_change":"2024-03-28T22:33:28.947384+0000","last_active":"2024-03-28T22:33:36.010710+0000","last_peered":"2024-03-28T22:33:36.010710+0000","last_clean":"2024-03-28T22:33:36.010710+0000","last_became_active":"2024-03-28T22:33:28.947017+0000","last_became_peered":"2024-03-28T22:33:28.947017+0000","last_unstale":"2024-03-28T22:33:36.010710+0000","last_undegraded":"2024-03-28T22:33:36.010710+0000","last_fullsized":"2024-03-28T22:33:36.010710+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:43:32.781161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.010663+0000","last_change":"2024-03-28T22:33:28.950119+0000","last_active":"2024-03-28T22:33:36.010663+0000","last_peered":"2024-03-28T22:33:36.010663+0000","last_clean":"2024-03-28T22:33:36.010663+0000","last_became_active":"2024-03-28T22:33:28.949798+0000","last_became_peered":"2024-03-28T22:33:28.949798+0000","last_unstale":"2024-03-28T22:33:36.010663+0000","last_undegraded":"2024-03-28T22:33:36.010663+0000","last_fullsized":"2024-03-28T22:33:36.010663+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:16.717678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.374939+0000","last_change":"2024-03-28T22:33:28.943255+0000","last_active":"2024-03-28T22:33:36.374939+0000","last_peered":"2024-03-28T22:33:36.374939+0000","last_clean":"2024-03-28T22:33:36.374939+0000","last_became_active":"2024-03-28T22:33:28.942923+0000","last_became_peered":"2024-03-28T22:33:28.942923+0000","last_unstale":"2024-03-28T22:33:36.374939+0000","last_undegraded":"2024-03-28T22:33:36.374939+0000","last_fullsized":"2024-03-28T22:33:36.374939+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:05:04.197635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.011835+0000","last_change":"2024-03-28T22:33:28.942906+0000","last_active":"2024-03-28T22:33:36.011835+0000","last_peered":"2024-03-28T22:33:36.011835+0000","last_clean":"2024-03-28T22:33:36.011835+0000","last_became_active":"2024-03-28T22:33:28.942573+0000","last_became_peered":"2024-03-28T22:33:28.942573+0000","last_unstale":"2024-03-28T22:33:36.011835+0000","last_undegraded":"2024-03-28T22:33:36.011835+0000","last_fullsized":"2024-03-28T22:33:36.011835+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:39:17.651101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.008726+0000","last_change":"2024-03-28T22:33:28.932075+0000","last_active":"2024-03-28T22:33:35.008726+0000","last_peered":"2024-03-28T22:33:35.008726+0000","last_clean":"2024-03-28T22:33:35.008726+0000","last_became_active":"2024-03-28T22:33:28.931689+0000","last_became_peered":"2024-03-28T22:33:28.931689+0000","last_unstale":"2024-03-28T22:33:35.008726+0000","last_undegraded":"2024-03-28T22:33:35.008726+0000","last_fullsized":"2024-03-28T22:33:35.008726+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:50:43.031082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376706+0000","last_change":"2024-03-28T22:33:28.944987+0000","last_active":"2024-03-28T22:33:36.376706+0000","last_peered":"2024-03-28T22:33:36.376706+0000","last_clean":"2024-03-28T22:33:36.376706+0000","last_became_active":"2024-03-28T22:33:28.944622+0000","last_became_peered":"2024-03-28T22:33:28.944622+0000","last_unstale":"2024-03-28T22:33:36.376706+0000","last_undegraded":"2024-03-28T22:33:36.376706+0000","last_fullsized":"2024-03-28T22:33:36.376706+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:48:58.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376507+0000","last_change":"2024-03-28T22:33:34.014395+0000","last_active":"2024-03-28T22:33:36.376507+0000","last_peered":"2024-03-28T22:33:36.376507+0000","last_clean":"2024-03-28T22:33:36.376507+0000","last_became_active":"2024-03-28T22:33:34.013976+0000","last_became_peered":"2024-03-28T22:33:34.013976+0000","last_unstale":"2024-03-28T22:33:36.376507+0000","last_undegraded":"2024-03-28T22:33:36.376507+0000","last_fullsized":"2024-03-28T22:33:36.376507+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:44:16.265629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"92'13","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:06.184257+0000","last_change":"2024-03-28T22:33:28.442524+0000","last_active":"2024-03-28T22:34:06.184257+0000","last_peered":"2024-03-28T22:34:06.184257+0000","last_clean":"2024-03-28T22:34:06.184257+0000","last_became_active":"2024-03-28T22:33:28.442169+0000","last_became_peered":"2024-03-28T22:33:28.442169+0000","last_unstale":"2024-03-28T22:34:06.184257+0000","last_undegraded":"2024-03-28T22:34:06.184257+0000","last_fullsized":"2024-03-28T22:34:06.184257+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:08:47.101119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.378959+0000","last_change":"2024-03-28T22:33:28.946697+0000","last_active":"2024-03-28T22:33:36.378959+0000","last_peered":"2024-03-28T22:33:36.378959+0000","last_clean":"2024-03-28T22:33:36.378959+0000","last_became_active":"2024-03-28T22:33:28.946303+0000","last_became_peered":"2024-03-28T22:33:28.946303+0000","last_unstale":"2024-03-28T22:33:36.378959+0000","last_undegraded":"2024-03-28T22:33:36.378959+0000","last_fullsized":"2024-03-28T22:33:36.378959+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:32:01.286073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.003794+0000","last_change":"2024-03-28T22:33:34.016868+0000","last_active":"2024-03-28T22:33:35.003794+0000","last_peered":"2024-03-28T22:33:35.003794+0000","last_clean":"2024-03-28T22:33:35.003794+0000","last_became_active":"2024-03-28T22:33:34.016472+0000","last_became_peered":"2024-03-28T22:33:34.016472+0000","last_unstale":"2024-03-28T22:33:35.003794+0000","last_undegraded":"2024-03-28T22:33:35.003794+0000","last_fullsized":"2024-03-28T22:33:35.003794+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:39:27.397699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"92'2","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.379681+0000","last_change":"2024-03-28T22:33:28.457661+0000","last_active":"2024-03-28T22:33:36.379681+0000","last_peered":"2024-03-28T22:33:36.379681+0000","last_clean":"2024-03-28T22:33:36.379681+0000","last_became_active":"2024-03-28T22:33:28.457032+0000","last_became_peered":"2024-03-28T22:33:28.457032+0000","last_unstale":"2024-03-28T22:33:36.379681+0000","last_undegraded":"2024-03-28T22:33:36.379681+0000","last_fullsized":"2024-03-28T22:33:36.379681+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:43:09.820630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.012702+0000","last_change":"2024-03-28T22:33:28.954252+0000","last_active":"2024-03-28T22:33:36.012702+0000","last_peered":"2024-03-28T22:33:36.012702+0000","last_clean":"2024-03-28T22:33:36.012702+0000","last_became_active":"2024-03-28T22:33:28.953922+0000","last_became_peered":"2024-03-28T22:33:28.953922+0000","last_unstale":"2024-03-28T22:33:36.012702+0000","last_undegraded":"2024-03-28T22:33:36.012702+0000","last_fullsized":"2024-03-28T22:33:36.012702+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:18:47.097859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.003837+0000","last_change":"2024-03-28T22:33:34.029129+0000","last_active":"2024-03-28T22:33:35.003837+0000","last_peered":"2024-03-28T22:33:35.003837+0000","last_clean":"2024-03-28T22:33:35.003837+0000","last_became_active":"2024-03-28T22:33:34.028452+0000","last_became_peered":"2024-03-28T22:33:34.028452+0000","last_unstale":"2024-03-28T22:33:35.003837+0000","last_undegraded":"2024-03-28T22:33:35.003837+0000","last_fullsized":"2024-03-28T22:33:35.003837+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:28.982773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:03.152124+0000","last_change":"2024-03-28T22:33:28.439568+0000","last_active":"2024-03-28T22:34:03.152124+0000","last_peered":"2024-03-28T22:34:03.152124+0000","last_clean":"2024-03-28T22:34:03.152124+0000","last_became_active":"2024-03-28T22:33:28.438887+0000","last_became_peered":"2024-03-28T22:33:28.438887+0000","last_unstale":"2024-03-28T22:34:03.152124+0000","last_undegraded":"2024-03-28T22:34:03.152124+0000","last_fullsized":"2024-03-28T22:34:03.152124+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:56:43.617621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.375102+0000","last_change":"2024-03-28T22:33:28.938311+0000","last_active":"2024-03-28T22:33:36.375102+0000","last_peered":"2024-03-28T22:33:36.375102+0000","last_clean":"2024-03-28T22:33:36.375102+0000","last_became_active":"2024-03-28T22:33:28.937997+0000","last_became_peered":"2024-03-28T22:33:28.937997+0000","last_unstale":"2024-03-28T22:33:36.375102+0000","last_undegraded":"2024-03-28T22:33:36.375102+0000","last_fullsized":"2024-03-28T22:33:36.375102+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:54:44.001545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.003786+0000","last_change":"2024-03-28T22:33:34.016931+0000","last_active":"2024-03-28T22:33:35.003786+0000","last_peered":"2024-03-28T22:33:35.003786+0000","last_clean":"2024-03-28T22:33:35.003786+0000","last_became_active":"2024-03-28T22:33:34.016559+0000","last_became_peered":"2024-03-28T22:33:34.016559+0000","last_unstale":"2024-03-28T22:33:35.003786+0000","last_undegraded":"2024-03-28T22:33:35.003786+0000","last_fullsized":"2024-03-28T22:33:35.003786+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:08:31.252351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.004484+0000","last_change":"2024-03-28T22:33:28.433166+0000","last_active":"2024-03-28T22:33:35.004484+0000","last_peered":"2024-03-28T22:33:35.004484+0000","last_clean":"2024-03-28T22:33:35.004484+0000","last_became_active":"2024-03-28T22:33:28.432716+0000","last_became_peered":"2024-03-28T22:33:28.432716+0000","last_unstale":"2024-03-28T22:33:35.004484+0000","last_undegraded":"2024-03-28T22:33:35.004484+0000","last_fullsized":"2024-03-28T22:33:35.004484+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:42:41.613583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.536637+0000","last_change":"2024-03-28T22:33:28.923307+0000","last_active":"2024-03-28T22:33:35.536637+0000","last_peered":"2024-03-28T22:33:35.536637+0000","last_clean":"2024-03-28T22:33:35.536637+0000","last_became_active":"2024-03-28T22:33:28.922499+0000","last_became_peered":"2024-03-28T22:33:28.922499+0000","last_unstale":"2024-03-28T22:33:35.536637+0000","last_undegraded":"2024-03-28T22:33:35.536637+0000","last_fullsized":"2024-03-28T22:33:35.536637+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:48:48.514482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376468+0000","last_change":"2024-03-28T22:33:34.002209+0000","last_active":"2024-03-28T22:33:36.376468+0000","last_peered":"2024-03-28T22:33:36.376468+0000","last_clean":"2024-03-28T22:33:36.376468+0000","last_became_active":"2024-03-28T22:33:34.001746+0000","last_became_peered":"2024-03-28T22:33:34.001746+0000","last_unstale":"2024-03-28T22:33:36.376468+0000","last_undegraded":"2024-03-28T22:33:36.376468+0000","last_fullsized":"2024-03-28T22:33:36.376468+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:02:02.333328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"92'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:06.190245+0000","last_change":"2024-03-28T22:33:28.435380+0000","last_active":"2024-03-28T22:34:06.190245+0000","last_peered":"2024-03-28T22:34:06.190245+0000","last_clean":"2024-03-28T22:34:06.190245+0000","last_became_active":"2024-03-28T22:33:28.434898+0000","last_became_peered":"2024-03-28T22:33:28.434898+0000","last_unstale":"2024-03-28T22:34:06.190245+0000","last_undegraded":"2024-03-28T22:34:06.190245+0000","last_fullsized":"2024-03-28T22:34:06.190245+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:58:06.810394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.004390+0000","last_change":"2024-03-28T22:33:28.926644+0000","last_active":"2024-03-28T22:33:35.004390+0000","last_peered":"2024-03-28T22:33:35.004390+0000","last_clean":"2024-03-28T22:33:35.004390+0000","last_became_active":"2024-03-28T22:33:28.926316+0000","last_became_peered":"2024-03-28T22:33:28.926316+0000","last_unstale":"2024-03-28T22:33:35.004390+0000","last_undegraded":"2024-03-28T22:33:35.004390+0000","last_fullsized":"2024-03-28T22:33:35.004390+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:59:23.190520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376589+0000","last_change":"2024-03-28T22:33:34.031350+0000","last_active":"2024-03-28T22:33:36.376589+0000","last_peered":"2024-03-28T22:33:36.376589+0000","last_clean":"2024-03-28T22:33:36.376589+0000","last_became_active":"2024-03-28T22:33:34.030946+0000","last_became_peered":"2024-03-28T22:33:34.030946+0000","last_unstale":"2024-03-28T22:33:36.376589+0000","last_undegraded":"2024-03-28T22:33:36.376589+0000","last_fullsized":"2024-03-28T22:33:36.376589+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:54:22.907363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:04.163516+0000","last_change":"2024-03-28T22:33:28.446197+0000","last_active":"2024-03-28T22:34:04.163516+0000","last_peered":"2024-03-28T22:34:04.163516+0000","last_clean":"2024-03-28T22:34:04.163516+0000","last_became_active":"2024-03-28T22:33:28.445891+0000","last_became_peered":"2024-03-28T22:33:28.445891+0000","last_unstale":"2024-03-28T22:34:04.163516+0000","last_undegraded":"2024-03-28T22:34:04.163516+0000","last_fullsized":"2024-03-28T22:34:04.163516+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:58:57.385302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.536579+0000","last_change":"2024-03-28T22:33:28.955651+0000","last_active":"2024-03-28T22:33:35.536579+0000","last_peered":"2024-03-28T22:33:35.536579+0000","last_clean":"2024-03-28T22:33:35.536579+0000","last_became_active":"2024-03-28T22:33:28.955276+0000","last_became_peered":"2024-03-28T22:33:28.955276+0000","last_unstale":"2024-03-28T22:33:35.536579+0000","last_undegraded":"2024-03-28T22:33:35.536579+0000","last_fullsized":"2024-03-28T22:33:35.536579+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:36:13.455327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.012043+0000","last_change":"2024-03-28T22:33:34.017520+0000","last_active":"2024-03-28T22:33:36.012043+0000","last_peered":"2024-03-28T22:33:36.012043+0000","last_clean":"2024-03-28T22:33:36.012043+0000","last_became_active":"2024-03-28T22:33:34.017022+0000","last_became_peered":"2024-03-28T22:33:34.017022+0000","last_unstale":"2024-03-28T22:33:36.012043+0000","last_undegraded":"2024-03-28T22:33:36.012043+0000","last_fullsized":"2024-03-28T22:33:36.012043+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:41:17.431581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"92'3","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:03.159541+0000","last_change":"2024-03-28T22:33:28.445401+0000","last_active":"2024-03-28T22:34:03.159541+0000","last_peered":"2024-03-28T22:34:03.159541+0000","last_clean":"2024-03-28T22:34:03.159541+0000","last_became_active":"2024-03-28T22:33:28.444237+0000","last_became_peered":"2024-03-28T22:33:28.444237+0000","last_unstale":"2024-03-28T22:34:03.159541+0000","last_undegraded":"2024-03-28T22:34:03.159541+0000","last_fullsized":"2024-03-28T22:34:03.159541+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:18:12.820890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.377630+0000","last_change":"2024-03-28T22:33:28.942347+0000","last_active":"2024-03-28T22:33:36.377630+0000","last_peered":"2024-03-28T22:33:36.377630+0000","last_clean":"2024-03-28T22:33:36.377630+0000","last_became_active":"2024-03-28T22:33:28.941573+0000","last_became_peered":"2024-03-28T22:33:28.941573+0000","last_unstale":"2024-03-28T22:33:36.377630+0000","last_undegraded":"2024-03-28T22:33:36.377630+0000","last_fullsized":"2024-03-28T22:33:36.377630+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:21:24.643326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.012153+0000","last_change":"2024-03-28T22:33:34.019899+0000","last_active":"2024-03-28T22:33:36.012153+0000","last_peered":"2024-03-28T22:33:36.012153+0000","last_clean":"2024-03-28T22:33:36.012153+0000","last_became_active":"2024-03-28T22:33:34.019392+0000","last_became_peered":"2024-03-28T22:33:34.019392+0000","last_unstale":"2024-03-28T22:33:36.012153+0000","last_undegraded":"2024-03-28T22:33:36.012153+0000","last_fullsized":"2024-03-28T22:33:36.012153+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:08:38.633797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"92'3","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:05.176763+0000","last_change":"2024-03-28T22:33:28.433243+0000","last_active":"2024-03-28T22:34:05.176763+0000","last_peered":"2024-03-28T22:34:05.176763+0000","last_clean":"2024-03-28T22:34:05.176763+0000","last_became_active":"2024-03-28T22:33:28.432927+0000","last_became_peered":"2024-03-28T22:33:28.432927+0000","last_unstale":"2024-03-28T22:34:05.176763+0000","last_undegraded":"2024-03-28T22:34:05.176763+0000","last_fullsized":"2024-03-28T22:34:05.176763+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:40:43.444229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.006977+0000","last_change":"2024-03-28T22:33:28.948111+0000","last_active":"2024-03-28T22:33:35.006977+0000","last_peered":"2024-03-28T22:33:35.006977+0000","last_clean":"2024-03-28T22:33:35.006977+0000","last_became_active":"2024-03-28T22:33:28.944649+0000","last_became_peered":"2024-03-28T22:33:28.944649+0000","last_unstale":"2024-03-28T22:33:35.006977+0000","last_undegraded":"2024-03-28T22:33:35.006977+0000","last_fullsized":"2024-03-28T22:33:35.006977+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:57:11.483037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.377846+0000","last_change":"2024-03-28T22:33:34.000837+0000","last_active":"2024-03-28T22:33:36.377846+0000","last_peered":"2024-03-28T22:33:36.377846+0000","last_clean":"2024-03-28T22:33:36.377846+0000","last_became_active":"2024-03-28T22:33:34.000120+0000","last_became_peered":"2024-03-28T22:33:34.000120+0000","last_unstale":"2024-03-28T22:33:36.377846+0000","last_undegraded":"2024-03-28T22:33:36.377846+0000","last_fullsized":"2024-03-28T22:33:36.377846+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:14:32.028952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:02.143560+0000","last_change":"2024-03-28T22:33:28.436985+0000","last_active":"2024-03-28T22:34:02.143560+0000","last_peered":"2024-03-28T22:34:02.143560+0000","last_clean":"2024-03-28T22:34:02.143560+0000","last_became_active":"2024-03-28T22:33:28.436625+0000","last_became_peered":"2024-03-28T22:33:28.436625+0000","last_unstale":"2024-03-28T22:34:02.143560+0000","last_undegraded":"2024-03-28T22:34:02.143560+0000","last_fullsized":"2024-03-28T22:34:02.143560+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:50:45.775191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.375265+0000","last_change":"2024-03-28T22:33:28.933265+0000","last_active":"2024-03-28T22:33:36.375265+0000","last_peered":"2024-03-28T22:33:36.375265+0000","last_clean":"2024-03-28T22:33:36.375265+0000","last_became_active":"2024-03-28T22:33:28.932954+0000","last_became_peered":"2024-03-28T22:33:28.932954+0000","last_unstale":"2024-03-28T22:33:36.375265+0000","last_undegraded":"2024-03-28T22:33:36.375265+0000","last_fullsized":"2024-03-28T22:33:36.375265+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:37.039055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.006940+0000","last_change":"2024-03-28T22:33:34.024529+0000","last_active":"2024-03-28T22:33:35.006940+0000","last_peered":"2024-03-28T22:33:35.006940+0000","last_clean":"2024-03-28T22:33:35.006940+0000","last_became_active":"2024-03-28T22:33:34.020949+0000","last_became_peered":"2024-03-28T22:33:34.020949+0000","last_unstale":"2024-03-28T22:33:35.006940+0000","last_undegraded":"2024-03-28T22:33:35.006940+0000","last_fullsized":"2024-03-28T22:33:35.006940+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:28:52.039684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"92'2","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:02.133927+0000","last_change":"2024-03-28T22:33:28.432599+0000","last_active":"2024-03-28T22:34:02.133927+0000","last_peered":"2024-03-28T22:34:02.133927+0000","last_clean":"2024-03-28T22:34:02.133927+0000","last_became_active":"2024-03-28T22:33:28.430710+0000","last_became_peered":"2024-03-28T22:33:28.430710+0000","last_unstale":"2024-03-28T22:34:02.133927+0000","last_undegraded":"2024-03-28T22:34:02.133927+0000","last_fullsized":"2024-03-28T22:34:02.133927+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:54.174747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.012898+0000","last_change":"2024-03-28T22:33:28.955441+0000","last_active":"2024-03-28T22:33:36.012898+0000","last_peered":"2024-03-28T22:33:36.012898+0000","last_clean":"2024-03-28T22:33:36.012898+0000","last_became_active":"2024-03-28T22:33:28.953269+0000","last_became_peered":"2024-03-28T22:33:28.953269+0000","last_unstale":"2024-03-28T22:33:36.012898+0000","last_undegraded":"2024-03-28T22:33:36.012898+0000","last_fullsized":"2024-03-28T22:33:36.012898+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:47:04.128605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.003994+0000","last_change":"2024-03-28T22:33:34.028791+0000","last_active":"2024-03-28T22:33:35.003994+0000","last_peered":"2024-03-28T22:33:35.003994+0000","last_clean":"2024-03-28T22:33:35.003994+0000","last_became_active":"2024-03-28T22:33:34.028361+0000","last_became_peered":"2024-03-28T22:33:34.028361+0000","last_unstale":"2024-03-28T22:33:35.003994+0000","last_undegraded":"2024-03-28T22:33:35.003994+0000","last_fullsized":"2024-03-28T22:33:35.003994+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:42:21.264459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"92'5","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:04.166390+0000","last_change":"2024-03-28T22:33:28.455353+0000","last_active":"2024-03-28T22:34:04.166390+0000","last_peered":"2024-03-28T22:34:04.166390+0000","last_clean":"2024-03-28T22:34:04.166390+0000","last_became_active":"2024-03-28T22:33:28.455090+0000","last_became_peered":"2024-03-28T22:33:28.455090+0000","last_unstale":"2024-03-28T22:34:04.166390+0000","last_undegraded":"2024-03-28T22:34:04.166390+0000","last_fullsized":"2024-03-28T22:34:04.166390+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:19:22.496600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.375269+0000","last_change":"2024-03-28T22:33:28.928875+0000","last_active":"2024-03-28T22:33:36.375269+0000","last_peered":"2024-03-28T22:33:36.375269+0000","last_clean":"2024-03-28T22:33:36.375269+0000","last_became_active":"2024-03-28T22:33:28.928449+0000","last_became_peered":"2024-03-28T22:33:28.928449+0000","last_unstale":"2024-03-28T22:33:36.375269+0000","last_undegraded":"2024-03-28T22:33:36.375269+0000","last_fullsized":"2024-03-28T22:33:36.375269+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:34:00.402100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.007553+0000","last_change":"2024-03-28T22:33:34.012739+0000","last_active":"2024-03-28T22:33:35.007553+0000","last_peered":"2024-03-28T22:33:35.007553+0000","last_clean":"2024-03-28T22:33:35.007553+0000","last_became_active":"2024-03-28T22:33:34.012262+0000","last_became_peered":"2024-03-28T22:33:34.012262+0000","last_unstale":"2024-03-28T22:33:35.007553+0000","last_undegraded":"2024-03-28T22:33:35.007553+0000","last_fullsized":"2024-03-28T22:33:35.007553+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:24:48.351037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"92'12","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:02.133506+0000","last_change":"2024-03-28T22:33:28.430606+0000","last_active":"2024-03-28T22:34:02.133506+0000","last_peered":"2024-03-28T22:34:02.133506+0000","last_clean":"2024-03-28T22:34:02.133506+0000","last_became_active":"2024-03-28T22:33:28.430233+0000","last_became_peered":"2024-03-28T22:33:28.430233+0000","last_unstale":"2024-03-28T22:34:02.133506+0000","last_undegraded":"2024-03-28T22:34:02.133506+0000","last_fullsized":"2024-03-28T22:34:02.133506+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:49.000569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.004329+0000","last_change":"2024-03-28T22:33:28.931167+0000","last_active":"2024-03-28T22:33:35.004329+0000","last_peered":"2024-03-28T22:33:35.004329+0000","last_clean":"2024-03-28T22:33:35.004329+0000","last_became_active":"2024-03-28T22:33:28.930824+0000","last_became_peered":"2024-03-28T22:33:28.930824+0000","last_unstale":"2024-03-28T22:33:35.004329+0000","last_undegraded":"2024-03-28T22:33:35.004329+0000","last_fullsized":"2024-03-28T22:33:35.004329+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:46:49.102791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.537046+0000","last_change":"2024-03-28T22:33:34.023056+0000","last_active":"2024-03-28T22:33:35.537046+0000","last_peered":"2024-03-28T22:33:35.537046+0000","last_clean":"2024-03-28T22:33:35.537046+0000","last_became_active":"2024-03-28T22:33:34.022612+0000","last_became_peered":"2024-03-28T22:33:34.022612+0000","last_unstale":"2024-03-28T22:33:35.537046+0000","last_undegraded":"2024-03-28T22:33:35.537046+0000","last_fullsized":"2024-03-28T22:33:35.537046+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:52:31.936008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:03.156550+0000","last_change":"2024-03-28T22:33:28.430963+0000","last_active":"2024-03-28T22:34:03.156550+0000","last_peered":"2024-03-28T22:34:03.156550+0000","last_clean":"2024-03-28T22:34:03.156550+0000","last_became_active":"2024-03-28T22:33:28.430632+0000","last_became_peered":"2024-03-28T22:33:28.430632+0000","last_unstale":"2024-03-28T22:34:03.156550+0000","last_undegraded":"2024-03-28T22:34:03.156550+0000","last_fullsized":"2024-03-28T22:34:03.156550+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:30:23.104769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.013675+0000","last_change":"2024-03-28T22:33:28.954105+0000","last_active":"2024-03-28T22:33:36.013675+0000","last_peered":"2024-03-28T22:33:36.013675+0000","last_clean":"2024-03-28T22:33:36.013675+0000","last_became_active":"2024-03-28T22:33:28.953768+0000","last_became_peered":"2024-03-28T22:33:28.953768+0000","last_unstale":"2024-03-28T22:33:36.013675+0000","last_undegraded":"2024-03-28T22:33:36.013675+0000","last_fullsized":"2024-03-28T22:33:36.013675+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:01:25.184523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.011771+0000","last_change":"2024-03-28T22:33:34.027976+0000","last_active":"2024-03-28T22:33:36.011771+0000","last_peered":"2024-03-28T22:33:36.011771+0000","last_clean":"2024-03-28T22:33:36.011771+0000","last_became_active":"2024-03-28T22:33:34.027509+0000","last_became_peered":"2024-03-28T22:33:34.027509+0000","last_unstale":"2024-03-28T22:33:36.011771+0000","last_undegraded":"2024-03-28T22:33:36.011771+0000","last_fullsized":"2024-03-28T22:33:36.011771+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:25:41.908441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"92'5","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:05.184759+0000","last_change":"2024-03-28T22:33:28.433283+0000","last_active":"2024-03-28T22:34:05.184759+0000","last_peered":"2024-03-28T22:34:05.184759+0000","last_clean":"2024-03-28T22:34:05.184759+0000","last_became_active":"2024-03-28T22:33:28.432954+0000","last_became_peered":"2024-03-28T22:33:28.432954+0000","last_unstale":"2024-03-28T22:34:05.184759+0000","last_undegraded":"2024-03-28T22:34:05.184759+0000","last_fullsized":"2024-03-28T22:34:05.184759+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:08:34.480459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376270+0000","last_change":"2024-03-28T22:33:28.939016+0000","last_active":"2024-03-28T22:33:36.376270+0000","last_peered":"2024-03-28T22:33:36.376270+0000","last_clean":"2024-03-28T22:33:36.376270+0000","last_became_active":"2024-03-28T22:33:28.938695+0000","last_became_peered":"2024-03-28T22:33:28.938695+0000","last_unstale":"2024-03-28T22:33:36.376270+0000","last_undegraded":"2024-03-28T22:33:36.376270+0000","last_fullsized":"2024-03-28T22:33:36.376270+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:26:53.264559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.008835+0000","last_change":"2024-03-28T22:33:34.030127+0000","last_active":"2024-03-28T22:33:35.008835+0000","last_peered":"2024-03-28T22:33:35.008835+0000","last_clean":"2024-03-28T22:33:35.008835+0000","last_became_active":"2024-03-28T22:33:34.028056+0000","last_became_peered":"2024-03-28T22:33:34.028056+0000","last_unstale":"2024-03-28T22:33:35.008835+0000","last_undegraded":"2024-03-28T22:33:35.008835+0000","last_fullsized":"2024-03-28T22:33:35.008835+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:29:41.785013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"92'6","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:06.189360+0000","last_change":"2024-03-28T22:33:28.435350+0000","last_active":"2024-03-28T22:34:06.189360+0000","last_peered":"2024-03-28T22:34:06.189360+0000","last_clean":"2024-03-28T22:34:06.189360+0000","last_became_active":"2024-03-28T22:33:28.434954+0000","last_became_peered":"2024-03-28T22:33:28.434954+0000","last_unstale":"2024-03-28T22:34:06.189360+0000","last_undegraded":"2024-03-28T22:34:06.189360+0000","last_fullsized":"2024-03-28T22:34:06.189360+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:04.969596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376101+0000","last_change":"2024-03-28T22:33:28.921219+0000","last_active":"2024-03-28T22:33:36.376101+0000","last_peered":"2024-03-28T22:33:36.376101+0000","last_clean":"2024-03-28T22:33:36.376101+0000","last_became_active":"2024-03-28T22:33:28.920664+0000","last_became_peered":"2024-03-28T22:33:28.920664+0000","last_unstale":"2024-03-28T22:33:36.376101+0000","last_undegraded":"2024-03-28T22:33:36.376101+0000","last_fullsized":"2024-03-28T22:33:36.376101+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:48:55.032058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.013871+0000","last_change":"2024-03-28T22:33:34.018101+0000","last_active":"2024-03-28T22:33:36.013871+0000","last_peered":"2024-03-28T22:33:36.013871+0000","last_clean":"2024-03-28T22:33:36.013871+0000","last_became_active":"2024-03-28T22:33:34.017260+0000","last_became_peered":"2024-03-28T22:33:34.017260+0000","last_unstale":"2024-03-28T22:33:36.013871+0000","last_undegraded":"2024-03-28T22:33:36.013871+0000","last_fullsized":"2024-03-28T22:33:36.013871+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:48:53.178090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"92'15","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:03.156112+0000","last_change":"2024-03-28T22:33:28.450232+0000","last_active":"2024-03-28T22:34:03.156112+0000","last_peered":"2024-03-28T22:34:03.156112+0000","last_clean":"2024-03-28T22:34:03.156112+0000","last_became_active":"2024-03-28T22:33:28.449958+0000","last_became_peered":"2024-03-28T22:33:28.449958+0000","last_unstale":"2024-03-28T22:34:03.156112+0000","last_undegraded":"2024-03-28T22:34:03.156112+0000","last_fullsized":"2024-03-28T22:34:03.156112+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:01:42.130260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.536203+0000","last_change":"2024-03-28T22:33:28.926263+0000","last_active":"2024-03-28T22:33:35.536203+0000","last_peered":"2024-03-28T22:33:35.536203+0000","last_clean":"2024-03-28T22:33:35.536203+0000","last_became_active":"2024-03-28T22:33:28.925927+0000","last_became_peered":"2024-03-28T22:33:28.925927+0000","last_unstale":"2024-03-28T22:33:35.536203+0000","last_undegraded":"2024-03-28T22:33:35.536203+0000","last_fullsized":"2024-03-28T22:33:35.536203+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:41:42.477694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.536325+0000","last_change":"2024-03-28T22:33:34.009764+0000","last_active":"2024-03-28T22:33:35.536325+0000","last_peered":"2024-03-28T22:33:35.536325+0000","last_clean":"2024-03-28T22:33:35.536325+0000","last_became_active":"2024-03-28T22:33:34.009320+0000","last_became_peered":"2024-03-28T22:33:34.009320+0000","last_unstale":"2024-03-28T22:33:35.536325+0000","last_undegraded":"2024-03-28T22:33:35.536325+0000","last_fullsized":"2024-03-28T22:33:35.536325+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:21:51.848928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'1","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.004748+0000","last_change":"2024-03-28T22:33:28.433015+0000","last_active":"2024-03-28T22:33:35.004748+0000","last_peered":"2024-03-28T22:33:35.004748+0000","last_clean":"2024-03-28T22:33:35.004748+0000","last_became_active":"2024-03-28T22:33:28.432689+0000","last_became_peered":"2024-03-28T22:33:28.432689+0000","last_unstale":"2024-03-28T22:33:35.004748+0000","last_undegraded":"2024-03-28T22:33:35.004748+0000","last_fullsized":"2024-03-28T22:33:35.004748+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:20:22.349953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.011843+0000","last_change":"2024-03-28T22:33:28.927335+0000","last_active":"2024-03-28T22:33:36.011843+0000","last_peered":"2024-03-28T22:33:36.011843+0000","last_clean":"2024-03-28T22:33:36.011843+0000","last_became_active":"2024-03-28T22:33:28.927036+0000","last_became_peered":"2024-03-28T22:33:28.927036+0000","last_unstale":"2024-03-28T22:33:36.011843+0000","last_undegraded":"2024-03-28T22:33:36.011843+0000","last_fullsized":"2024-03-28T22:33:36.011843+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:36:00.930368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.004206+0000","last_change":"2024-03-28T22:33:34.014364+0000","last_active":"2024-03-28T22:33:35.004206+0000","last_peered":"2024-03-28T22:33:35.004206+0000","last_clean":"2024-03-28T22:33:35.004206+0000","last_became_active":"2024-03-28T22:33:34.013929+0000","last_became_peered":"2024-03-28T22:33:34.013929+0000","last_unstale":"2024-03-28T22:33:35.004206+0000","last_undegraded":"2024-03-28T22:33:35.004206+0000","last_fullsized":"2024-03-28T22:33:35.004206+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:21:37.986774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"92'1","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.007788+0000","last_change":"2024-03-28T22:33:28.453458+0000","last_active":"2024-03-28T22:33:35.007788+0000","last_peered":"2024-03-28T22:33:35.007788+0000","last_clean":"2024-03-28T22:33:35.007788+0000","last_became_active":"2024-03-28T22:33:28.452829+0000","last_became_peered":"2024-03-28T22:33:28.452829+0000","last_unstale":"2024-03-28T22:33:35.007788+0000","last_undegraded":"2024-03-28T22:33:35.007788+0000","last_fullsized":"2024-03-28T22:33:35.007788+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:33:05.885690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.141691+0000","last_change":"2024-03-28T22:33:28.942097+0000","last_active":"2024-03-28T22:33:35.141691+0000","last_peered":"2024-03-28T22:33:35.141691+0000","last_clean":"2024-03-28T22:33:35.141691+0000","last_became_active":"2024-03-28T22:33:28.941812+0000","last_became_peered":"2024-03-28T22:33:28.941812+0000","last_unstale":"2024-03-28T22:33:35.141691+0000","last_undegraded":"2024-03-28T22:33:35.141691+0000","last_fullsized":"2024-03-28T22:33:35.141691+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:36:10.204216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.013288+0000","last_change":"2024-03-28T22:33:34.023171+0000","last_active":"2024-03-28T22:33:36.013288+0000","last_peered":"2024-03-28T22:33:36.013288+0000","last_clean":"2024-03-28T22:33:36.013288+0000","last_became_active":"2024-03-28T22:33:34.021313+0000","last_became_peered":"2024-03-28T22:33:34.021313+0000","last_unstale":"2024-03-28T22:33:36.013288+0000","last_undegraded":"2024-03-28T22:33:36.013288+0000","last_fullsized":"2024-03-28T22:33:36.013288+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:29:42.592761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"92'1","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376198+0000","last_change":"2024-03-28T22:33:28.449133+0000","last_active":"2024-03-28T22:33:36.376198+0000","last_peered":"2024-03-28T22:33:36.376198+0000","last_clean":"2024-03-28T22:33:36.376198+0000","last_became_active":"2024-03-28T22:33:28.448743+0000","last_became_peered":"2024-03-28T22:33:28.448743+0000","last_unstale":"2024-03-28T22:33:36.376198+0000","last_undegraded":"2024-03-28T22:33:36.376198+0000","last_fullsized":"2024-03-28T22:33:36.376198+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:09:37.119736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.141594+0000","last_change":"2024-03-28T22:33:28.944198+0000","last_active":"2024-03-28T22:33:35.141594+0000","last_peered":"2024-03-28T22:33:35.141594+0000","last_clean":"2024-03-28T22:33:35.141594+0000","last_became_active":"2024-03-28T22:33:28.943818+0000","last_became_peered":"2024-03-28T22:33:28.943818+0000","last_unstale":"2024-03-28T22:33:35.141594+0000","last_undegraded":"2024-03-28T22:33:35.141594+0000","last_fullsized":"2024-03-28T22:33:35.141594+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:31:03.791364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.014206+0000","last_change":"2024-03-28T22:33:34.015499+0000","last_active":"2024-03-28T22:33:36.014206+0000","last_peered":"2024-03-28T22:33:36.014206+0000","last_clean":"2024-03-28T22:33:36.014206+0000","last_became_active":"2024-03-28T22:33:34.015013+0000","last_became_peered":"2024-03-28T22:33:34.015013+0000","last_unstale":"2024-03-28T22:33:36.014206+0000","last_undegraded":"2024-03-28T22:33:36.014206+0000","last_fullsized":"2024-03-28T22:33:36.014206+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:34:30.582198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.536304+0000","last_change":"2024-03-28T22:33:28.926732+0000","last_active":"2024-03-28T22:33:35.536304+0000","last_peered":"2024-03-28T22:33:35.536304+0000","last_clean":"2024-03-28T22:33:35.536304+0000","last_became_active":"2024-03-28T22:33:28.926366+0000","last_became_peered":"2024-03-28T22:33:28.926366+0000","last_unstale":"2024-03-28T22:33:35.536304+0000","last_undegraded":"2024-03-28T22:33:35.536304+0000","last_fullsized":"2024-03-28T22:33:35.536304+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:19:10.769141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"92'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:06.188814+0000","last_change":"2024-03-28T22:33:28.457400+0000","last_active":"2024-03-28T22:34:06.188814+0000","last_peered":"2024-03-28T22:34:06.188814+0000","last_clean":"2024-03-28T22:34:06.188814+0000","last_became_active":"2024-03-28T22:33:28.456994+0000","last_became_peered":"2024-03-28T22:33:28.456994+0000","last_unstale":"2024-03-28T22:34:06.188814+0000","last_undegraded":"2024-03-28T22:34:06.188814+0000","last_fullsized":"2024-03-28T22:34:06.188814+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:32:16.438540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376384+0000","last_change":"2024-03-28T22:33:34.027746+0000","last_active":"2024-03-28T22:33:36.376384+0000","last_peered":"2024-03-28T22:33:36.376384+0000","last_clean":"2024-03-28T22:33:36.376384+0000","last_became_active":"2024-03-28T22:33:34.027419+0000","last_became_peered":"2024-03-28T22:33:34.027419+0000","last_unstale":"2024-03-28T22:33:36.376384+0000","last_undegraded":"2024-03-28T22:33:36.376384+0000","last_fullsized":"2024-03-28T22:33:36.376384+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:55:43.686428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.378061+0000","last_change":"2024-03-28T22:33:28.952393+0000","last_active":"2024-03-28T22:33:36.378061+0000","last_peered":"2024-03-28T22:33:36.378061+0000","last_clean":"2024-03-28T22:33:36.378061+0000","last_became_active":"2024-03-28T22:33:28.951951+0000","last_became_peered":"2024-03-28T22:33:28.951951+0000","last_unstale":"2024-03-28T22:33:36.378061+0000","last_undegraded":"2024-03-28T22:33:36.378061+0000","last_fullsized":"2024-03-28T22:33:36.378061+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:09:25.808842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"92'6","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:06.186898+0000","last_change":"2024-03-28T22:33:28.437177+0000","last_active":"2024-03-28T22:34:06.186898+0000","last_peered":"2024-03-28T22:34:06.186898+0000","last_clean":"2024-03-28T22:34:06.186898+0000","last_became_active":"2024-03-28T22:33:28.436801+0000","last_became_peered":"2024-03-28T22:33:28.436801+0000","last_unstale":"2024-03-28T22:34:06.186898+0000","last_undegraded":"2024-03-28T22:34:06.186898+0000","last_fullsized":"2024-03-28T22:34:06.186898+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:24:52.549490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":362,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.375714+0000","last_change":"2024-03-28T22:32:07.867336+0000","last_active":"2024-03-28T22:33:36.375714+0000","last_peered":"2024-03-28T22:33:36.375714+0000","last_clean":"2024-03-28T22:33:36.375714+0000","last_became_active":"2024-03-28T22:32:07.866819+0000","last_became_peered":"2024-03-28T22:32:07.866819+0000","last_unstale":"2024-03-28T22:33:36.375714+0000","last_undegraded":"2024-03-28T22:33:36.375714+0000","last_fullsized":"2024-03-28T22:33:36.375714+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:28:14.713501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:28:14.713501+0000","last_clean_scrub_stamp":"2024-03-28T22:28:14.713501+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:00:52.438708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.536862+0000","last_change":"2024-03-28T22:33:34.020668+0000","last_active":"2024-03-28T22:33:35.536862+0000","last_peered":"2024-03-28T22:33:35.536862+0000","last_clean":"2024-03-28T22:33:35.536862+0000","last_became_active":"2024-03-28T22:33:34.020240+0000","last_became_peered":"2024-03-28T22:33:34.020240+0000","last_unstale":"2024-03-28T22:33:35.536862+0000","last_undegraded":"2024-03-28T22:33:35.536862+0000","last_fullsized":"2024-03-28T22:33:35.536862+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:18.983983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.012960+0000","last_change":"2024-03-28T22:33:28.935237+0000","last_active":"2024-03-28T22:33:36.012960+0000","last_peered":"2024-03-28T22:33:36.012960+0000","last_clean":"2024-03-28T22:33:36.012960+0000","last_became_active":"2024-03-28T22:33:28.934875+0000","last_became_peered":"2024-03-28T22:33:28.934875+0000","last_unstale":"2024-03-28T22:33:36.012960+0000","last_undegraded":"2024-03-28T22:33:36.012960+0000","last_fullsized":"2024-03-28T22:33:36.012960+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:20:50.789302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"92'3","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:03.151431+0000","last_change":"2024-03-28T22:33:28.441654+0000","last_active":"2024-03-28T22:34:03.151431+0000","last_peered":"2024-03-28T22:34:03.151431+0000","last_clean":"2024-03-28T22:34:03.151431+0000","last_became_active":"2024-03-28T22:33:28.441293+0000","last_became_peered":"2024-03-28T22:33:28.441293+0000","last_unstale":"2024-03-28T22:34:03.151431+0000","last_undegraded":"2024-03-28T22:34:03.151431+0000","last_fullsized":"2024-03-28T22:34:03.151431+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:31:17.737977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.011837+0000","last_change":"2024-03-28T22:33:34.018126+0000","last_active":"2024-03-28T22:33:36.011837+0000","last_peered":"2024-03-28T22:33:36.011837+0000","last_clean":"2024-03-28T22:33:36.011837+0000","last_became_active":"2024-03-28T22:33:34.017573+0000","last_became_peered":"2024-03-28T22:33:34.017573+0000","last_unstale":"2024-03-28T22:33:36.011837+0000","last_undegraded":"2024-03-28T22:33:36.011837+0000","last_fullsized":"2024-03-28T22:33:36.011837+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:23:31.013511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"92'1","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.004540+0000","last_change":"2024-03-28T22:33:28.454069+0000","last_active":"2024-03-28T22:33:35.004540+0000","last_peered":"2024-03-28T22:33:35.004540+0000","last_clean":"2024-03-28T22:33:35.004540+0000","last_became_active":"2024-03-28T22:33:28.453754+0000","last_became_peered":"2024-03-28T22:33:28.453754+0000","last_unstale":"2024-03-28T22:33:35.004540+0000","last_undegraded":"2024-03-28T22:33:35.004540+0000","last_fullsized":"2024-03-28T22:33:35.004540+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:23:33.981072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.013496+0000","last_change":"2024-03-28T22:33:28.934390+0000","last_active":"2024-03-28T22:33:36.013496+0000","last_peered":"2024-03-28T22:33:36.013496+0000","last_clean":"2024-03-28T22:33:36.013496+0000","last_became_active":"2024-03-28T22:33:28.933919+0000","last_became_peered":"2024-03-28T22:33:28.933919+0000","last_unstale":"2024-03-28T22:33:36.013496+0000","last_undegraded":"2024-03-28T22:33:36.013496+0000","last_fullsized":"2024-03-28T22:33:36.013496+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:27:07.595144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.377785+0000","last_change":"2024-03-28T22:33:34.019910+0000","last_active":"2024-03-28T22:33:36.377785+0000","last_peered":"2024-03-28T22:33:36.377785+0000","last_clean":"2024-03-28T22:33:36.377785+0000","last_became_active":"2024-03-28T22:33:34.019522+0000","last_became_peered":"2024-03-28T22:33:34.019522+0000","last_unstale":"2024-03-28T22:33:36.377785+0000","last_undegraded":"2024-03-28T22:33:36.377785+0000","last_fullsized":"2024-03-28T22:33:36.377785+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:01:29.298513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:05.483945+0000","last_change":"2024-03-28T22:33:28.444513+0000","last_active":"2024-03-28T22:34:05.483945+0000","last_peered":"2024-03-28T22:34:05.483945+0000","last_clean":"2024-03-28T22:34:05.483945+0000","last_became_active":"2024-03-28T22:33:28.444091+0000","last_became_peered":"2024-03-28T22:33:28.444091+0000","last_unstale":"2024-03-28T22:34:05.483945+0000","last_undegraded":"2024-03-28T22:34:05.483945+0000","last_fullsized":"2024-03-28T22:34:05.483945+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:49:58.331864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.379248+0000","last_change":"2024-03-28T22:33:28.948390+0000","last_active":"2024-03-28T22:33:36.379248+0000","last_peered":"2024-03-28T22:33:36.379248+0000","last_clean":"2024-03-28T22:33:36.379248+0000","last_became_active":"2024-03-28T22:33:28.942846+0000","last_became_peered":"2024-03-28T22:33:28.942846+0000","last_unstale":"2024-03-28T22:33:36.379248+0000","last_undegraded":"2024-03-28T22:33:36.379248+0000","last_fullsized":"2024-03-28T22:33:36.379248+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:50.596365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376293+0000","last_change":"2024-03-28T22:33:33.999429+0000","last_active":"2024-03-28T22:33:36.376293+0000","last_peered":"2024-03-28T22:33:36.376293+0000","last_clean":"2024-03-28T22:33:36.376293+0000","last_became_active":"2024-03-28T22:33:33.998979+0000","last_became_peered":"2024-03-28T22:33:33.998979+0000","last_unstale":"2024-03-28T22:33:36.376293+0000","last_undegraded":"2024-03-28T22:33:36.376293+0000","last_fullsized":"2024-03-28T22:33:36.376293+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:26.450731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:05.177871+0000","last_change":"2024-03-28T22:33:28.442861+0000","last_active":"2024-03-28T22:34:05.177871+0000","last_peered":"2024-03-28T22:34:05.177871+0000","last_clean":"2024-03-28T22:34:05.177871+0000","last_became_active":"2024-03-28T22:33:28.442468+0000","last_became_peered":"2024-03-28T22:33:28.442468+0000","last_unstale":"2024-03-28T22:34:05.177871+0000","last_undegraded":"2024-03-28T22:34:05.177871+0000","last_fullsized":"2024-03-28T22:34:05.177871+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:45:49.899031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.012371+0000","last_change":"2024-03-28T22:33:28.957197+0000","last_active":"2024-03-28T22:33:36.012371+0000","last_peered":"2024-03-28T22:33:36.012371+0000","last_clean":"2024-03-28T22:33:36.012371+0000","last_became_active":"2024-03-28T22:33:28.956836+0000","last_became_peered":"2024-03-28T22:33:28.956836+0000","last_unstale":"2024-03-28T22:33:36.012371+0000","last_undegraded":"2024-03-28T22:33:36.012371+0000","last_fullsized":"2024-03-28T22:33:36.012371+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:25:40.765500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.004336+0000","last_change":"2024-03-28T22:33:34.013826+0000","last_active":"2024-03-28T22:33:35.004336+0000","last_peered":"2024-03-28T22:33:35.004336+0000","last_clean":"2024-03-28T22:33:35.004336+0000","last_became_active":"2024-03-28T22:33:34.013415+0000","last_became_peered":"2024-03-28T22:33:34.013415+0000","last_unstale":"2024-03-28T22:33:35.004336+0000","last_undegraded":"2024-03-28T22:33:35.004336+0000","last_fullsized":"2024-03-28T22:33:35.004336+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:19:30.965940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"92'2","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:02.136787+0000","last_change":"2024-03-28T22:33:28.442225+0000","last_active":"2024-03-28T22:34:02.136787+0000","last_peered":"2024-03-28T22:34:02.136787+0000","last_clean":"2024-03-28T22:34:02.136787+0000","last_became_active":"2024-03-28T22:33:28.441844+0000","last_became_peered":"2024-03-28T22:33:28.441844+0000","last_unstale":"2024-03-28T22:34:02.136787+0000","last_undegraded":"2024-03-28T22:34:02.136787+0000","last_fullsized":"2024-03-28T22:34:02.136787+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:40:52.108663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.004042+0000","last_change":"2024-03-28T22:33:28.927571+0000","last_active":"2024-03-28T22:33:35.004042+0000","last_peered":"2024-03-28T22:33:35.004042+0000","last_clean":"2024-03-28T22:33:35.004042+0000","last_became_active":"2024-03-28T22:33:28.927275+0000","last_became_peered":"2024-03-28T22:33:28.927275+0000","last_unstale":"2024-03-28T22:33:35.004042+0000","last_undegraded":"2024-03-28T22:33:35.004042+0000","last_fullsized":"2024-03-28T22:33:35.004042+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:25:16.184533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.009016+0000","last_change":"2024-03-28T22:33:34.018429+0000","last_active":"2024-03-28T22:33:35.009016+0000","last_peered":"2024-03-28T22:33:35.009016+0000","last_clean":"2024-03-28T22:33:35.009016+0000","last_became_active":"2024-03-28T22:33:34.017066+0000","last_became_peered":"2024-03-28T22:33:34.017066+0000","last_unstale":"2024-03-28T22:33:35.009016+0000","last_undegraded":"2024-03-28T22:33:35.009016+0000","last_fullsized":"2024-03-28T22:33:35.009016+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:35:42.101072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"92'7","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:05.183474+0000","last_change":"2024-03-28T22:33:28.437760+0000","last_active":"2024-03-28T22:34:05.183474+0000","last_peered":"2024-03-28T22:34:05.183474+0000","last_clean":"2024-03-28T22:34:05.183474+0000","last_became_active":"2024-03-28T22:33:28.437096+0000","last_became_peered":"2024-03-28T22:33:28.437096+0000","last_unstale":"2024-03-28T22:34:05.183474+0000","last_undegraded":"2024-03-28T22:34:05.183474+0000","last_fullsized":"2024-03-28T22:34:05.183474+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:29:20.778551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.011450+0000","last_change":"2024-03-28T22:33:28.935805+0000","last_active":"2024-03-28T22:33:36.011450+0000","last_peered":"2024-03-28T22:33:36.011450+0000","last_clean":"2024-03-28T22:33:36.011450+0000","last_became_active":"2024-03-28T22:33:28.935520+0000","last_became_peered":"2024-03-28T22:33:28.935520+0000","last_unstale":"2024-03-28T22:33:36.011450+0000","last_undegraded":"2024-03-28T22:33:36.011450+0000","last_fullsized":"2024-03-28T22:33:36.011450+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:23.414310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"92'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.572662+0000","last_change":"2024-03-28T22:33:34.029518+0000","last_active":"2024-03-28T22:33:36.572662+0000","last_peered":"2024-03-28T22:33:36.572662+0000","last_clean":"2024-03-28T22:33:36.572662+0000","last_became_active":"2024-03-28T22:33:34.029101+0000","last_became_peered":"2024-03-28T22:33:34.029101+0000","last_unstale":"2024-03-28T22:33:36.572662+0000","last_undegraded":"2024-03-28T22:33:36.572662+0000","last_fullsized":"2024-03-28T22:33:36.572662+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:05.595543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376270+0000","last_change":"2024-03-28T22:33:28.438857+0000","last_active":"2024-03-28T22:33:36.376270+0000","last_peered":"2024-03-28T22:33:36.376270+0000","last_clean":"2024-03-28T22:33:36.376270+0000","last_became_active":"2024-03-28T22:33:28.438528+0000","last_became_peered":"2024-03-28T22:33:28.438528+0000","last_unstale":"2024-03-28T22:33:36.376270+0000","last_undegraded":"2024-03-28T22:33:36.376270+0000","last_fullsized":"2024-03-28T22:33:36.376270+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:07:54.506739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.378483+0000","last_change":"2024-03-28T22:33:28.958271+0000","last_active":"2024-03-28T22:33:36.378483+0000","last_peered":"2024-03-28T22:33:36.378483+0000","last_clean":"2024-03-28T22:33:36.378483+0000","last_became_active":"2024-03-28T22:33:28.957922+0000","last_became_peered":"2024-03-28T22:33:28.957922+0000","last_unstale":"2024-03-28T22:33:36.378483+0000","last_undegraded":"2024-03-28T22:33:36.378483+0000","last_fullsized":"2024-03-28T22:33:36.378483+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:33:19.003131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.004080+0000","last_change":"2024-03-28T22:33:34.000789+0000","last_active":"2024-03-28T22:33:35.004080+0000","last_peered":"2024-03-28T22:33:35.004080+0000","last_clean":"2024-03-28T22:33:35.004080+0000","last_became_active":"2024-03-28T22:33:34.000383+0000","last_became_peered":"2024-03-28T22:33:34.000383+0000","last_unstale":"2024-03-28T22:33:35.004080+0000","last_undegraded":"2024-03-28T22:33:35.004080+0000","last_fullsized":"2024-03-28T22:33:35.004080+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:50:29.925656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:02.132519+0000","last_change":"2024-03-28T22:33:28.439121+0000","last_active":"2024-03-28T22:34:02.132519+0000","last_peered":"2024-03-28T22:34:02.132519+0000","last_clean":"2024-03-28T22:34:02.132519+0000","last_became_active":"2024-03-28T22:33:28.438800+0000","last_became_peered":"2024-03-28T22:33:28.438800+0000","last_unstale":"2024-03-28T22:34:02.132519+0000","last_undegraded":"2024-03-28T22:34:02.132519+0000","last_fullsized":"2024-03-28T22:34:02.132519+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:57:10.013246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376130+0000","last_change":"2024-03-28T22:33:28.934499+0000","last_active":"2024-03-28T22:33:36.376130+0000","last_peered":"2024-03-28T22:33:36.376130+0000","last_clean":"2024-03-28T22:33:36.376130+0000","last_became_active":"2024-03-28T22:33:28.934147+0000","last_became_peered":"2024-03-28T22:33:28.934147+0000","last_unstale":"2024-03-28T22:33:36.376130+0000","last_undegraded":"2024-03-28T22:33:36.376130+0000","last_fullsized":"2024-03-28T22:33:36.376130+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:35:39.090052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.374902+0000","last_change":"2024-03-28T22:33:34.017946+0000","last_active":"2024-03-28T22:33:36.374902+0000","last_peered":"2024-03-28T22:33:36.374902+0000","last_clean":"2024-03-28T22:33:36.374902+0000","last_became_active":"2024-03-28T22:33:34.017546+0000","last_became_peered":"2024-03-28T22:33:34.017546+0000","last_unstale":"2024-03-28T22:33:36.374902+0000","last_undegraded":"2024-03-28T22:33:36.374902+0000","last_fullsized":"2024-03-28T22:33:36.374902+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:14:39.081677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"92'3","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:03.154378+0000","last_change":"2024-03-28T22:33:28.453387+0000","last_active":"2024-03-28T22:34:03.154378+0000","last_peered":"2024-03-28T22:34:03.154378+0000","last_clean":"2024-03-28T22:34:03.154378+0000","last_became_active":"2024-03-28T22:33:28.453029+0000","last_became_peered":"2024-03-28T22:33:28.453029+0000","last_unstale":"2024-03-28T22:34:03.154378+0000","last_undegraded":"2024-03-28T22:34:03.154378+0000","last_fullsized":"2024-03-28T22:34:03.154378+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:12:33.246863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376535+0000","last_change":"2024-03-28T22:33:28.923819+0000","last_active":"2024-03-28T22:33:36.376535+0000","last_peered":"2024-03-28T22:33:36.376535+0000","last_clean":"2024-03-28T22:33:36.376535+0000","last_became_active":"2024-03-28T22:33:28.923471+0000","last_became_peered":"2024-03-28T22:33:28.923471+0000","last_unstale":"2024-03-28T22:33:36.376535+0000","last_undegraded":"2024-03-28T22:33:36.376535+0000","last_fullsized":"2024-03-28T22:33:36.376535+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:58:03.496406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.011014+0000","last_change":"2024-03-28T22:33:34.029789+0000","last_active":"2024-03-28T22:33:36.011014+0000","last_peered":"2024-03-28T22:33:36.011014+0000","last_clean":"2024-03-28T22:33:36.011014+0000","last_became_active":"2024-03-28T22:33:34.029231+0000","last_became_peered":"2024-03-28T22:33:34.029231+0000","last_unstale":"2024-03-28T22:33:36.011014+0000","last_undegraded":"2024-03-28T22:33:36.011014+0000","last_fullsized":"2024-03-28T22:33:36.011014+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:47:50.101393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"92'3","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:05.184158+0000","last_change":"2024-03-28T22:33:28.435668+0000","last_active":"2024-03-28T22:34:05.184158+0000","last_peered":"2024-03-28T22:34:05.184158+0000","last_clean":"2024-03-28T22:34:05.184158+0000","last_became_active":"2024-03-28T22:33:28.435325+0000","last_became_peered":"2024-03-28T22:33:28.435325+0000","last_unstale":"2024-03-28T22:34:05.184158+0000","last_undegraded":"2024-03-28T22:34:05.184158+0000","last_fullsized":"2024-03-28T22:34:05.184158+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:13:06.281074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376264+0000","last_change":"2024-03-28T22:33:28.954138+0000","last_active":"2024-03-28T22:33:36.376264+0000","last_peered":"2024-03-28T22:33:36.376264+0000","last_clean":"2024-03-28T22:33:36.376264+0000","last_became_active":"2024-03-28T22:33:28.953752+0000","last_became_peered":"2024-03-28T22:33:28.953752+0000","last_unstale":"2024-03-28T22:33:36.376264+0000","last_undegraded":"2024-03-28T22:33:36.376264+0000","last_fullsized":"2024-03-28T22:33:36.376264+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:53:34.892316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.536052+0000","last_change":"2024-03-28T22:33:34.002672+0000","last_active":"2024-03-28T22:33:35.536052+0000","last_peered":"2024-03-28T22:33:35.536052+0000","last_clean":"2024-03-28T22:33:35.536052+0000","last_became_active":"2024-03-28T22:33:34.002195+0000","last_became_peered":"2024-03-28T22:33:34.002195+0000","last_unstale":"2024-03-28T22:33:35.536052+0000","last_undegraded":"2024-03-28T22:33:35.536052+0000","last_fullsized":"2024-03-28T22:33:35.536052+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:24:11.521474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:34:05.184387+0000","last_change":"2024-03-28T22:33:28.450818+0000","last_active":"2024-03-28T22:34:05.184387+0000","last_peered":"2024-03-28T22:34:05.184387+0000","last_clean":"2024-03-28T22:34:05.184387+0000","last_became_active":"2024-03-28T22:33:28.448592+0000","last_became_peered":"2024-03-28T22:33:28.448592+0000","last_unstale":"2024-03-28T22:34:05.184387+0000","last_undegraded":"2024-03-28T22:34:05.184387+0000","last_fullsized":"2024-03-28T22:34:05.184387+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:19:27.126306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376450+0000","last_change":"2024-03-28T22:33:28.925191+0000","last_active":"2024-03-28T22:33:36.376450+0000","last_peered":"2024-03-28T22:33:36.376450+0000","last_clean":"2024-03-28T22:33:36.376450+0000","last_became_active":"2024-03-28T22:33:28.924424+0000","last_became_peered":"2024-03-28T22:33:28.924424+0000","last_unstale":"2024-03-28T22:33:36.376450+0000","last_undegraded":"2024-03-28T22:33:36.376450+0000","last_fullsized":"2024-03-28T22:33:36.376450+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:50:00.415047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.011619+0000","last_change":"2024-03-28T22:33:28.925110+0000","last_active":"2024-03-28T22:33:36.011619+0000","last_peered":"2024-03-28T22:33:36.011619+0000","last_clean":"2024-03-28T22:33:36.011619+0000","last_became_active":"2024-03-28T22:33:28.924745+0000","last_became_peered":"2024-03-28T22:33:28.924745+0000","last_unstale":"2024-03-28T22:33:36.011619+0000","last_undegraded":"2024-03-28T22:33:36.011619+0000","last_fullsized":"2024-03-28T22:33:36.011619+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:23:06.331043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376687+0000","last_change":"2024-03-28T22:33:28.927068+0000","last_active":"2024-03-28T22:33:36.376687+0000","last_peered":"2024-03-28T22:33:36.376687+0000","last_clean":"2024-03-28T22:33:36.376687+0000","last_became_active":"2024-03-28T22:33:28.926734+0000","last_became_peered":"2024-03-28T22:33:28.926734+0000","last_unstale":"2024-03-28T22:33:36.376687+0000","last_undegraded":"2024-03-28T22:33:36.376687+0000","last_fullsized":"2024-03-28T22:33:36.376687+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:56.309555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376374+0000","last_change":"2024-03-28T22:33:28.955675+0000","last_active":"2024-03-28T22:33:36.376374+0000","last_peered":"2024-03-28T22:33:36.376374+0000","last_clean":"2024-03-28T22:33:36.376374+0000","last_became_active":"2024-03-28T22:33:28.955329+0000","last_became_peered":"2024-03-28T22:33:28.955329+0000","last_unstale":"2024-03-28T22:33:36.376374+0000","last_undegraded":"2024-03-28T22:33:36.376374+0000","last_fullsized":"2024-03-28T22:33:36.376374+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:34:14.166434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.011204+0000","last_change":"2024-03-28T22:33:28.955125+0000","last_active":"2024-03-28T22:33:36.011204+0000","last_peered":"2024-03-28T22:33:36.011204+0000","last_clean":"2024-03-28T22:33:36.011204+0000","last_became_active":"2024-03-28T22:33:28.954434+0000","last_became_peered":"2024-03-28T22:33:28.954434+0000","last_unstale":"2024-03-28T22:33:36.011204+0000","last_undegraded":"2024-03-28T22:33:36.011204+0000","last_fullsized":"2024-03-28T22:33:36.011204+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:56:50.705012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.013636+0000","last_change":"2024-03-28T22:33:28.945230+0000","last_active":"2024-03-28T22:33:36.013636+0000","last_peered":"2024-03-28T22:33:36.013636+0000","last_clean":"2024-03-28T22:33:36.013636+0000","last_became_active":"2024-03-28T22:33:28.944921+0000","last_became_peered":"2024-03-28T22:33:28.944921+0000","last_unstale":"2024-03-28T22:33:36.013636+0000","last_undegraded":"2024-03-28T22:33:36.013636+0000","last_fullsized":"2024-03-28T22:33:36.013636+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:26:26.505774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.011851+0000","last_change":"2024-03-28T22:33:28.928720+0000","last_active":"2024-03-28T22:33:36.011851+0000","last_peered":"2024-03-28T22:33:36.011851+0000","last_clean":"2024-03-28T22:33:36.011851+0000","last_became_active":"2024-03-28T22:33:28.928415+0000","last_became_peered":"2024-03-28T22:33:28.928415+0000","last_unstale":"2024-03-28T22:33:36.011851+0000","last_undegraded":"2024-03-28T22:33:36.011851+0000","last_fullsized":"2024-03-28T22:33:36.011851+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:00.839609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.376804+0000","last_change":"2024-03-28T22:33:28.924948+0000","last_active":"2024-03-28T22:33:36.376804+0000","last_peered":"2024-03-28T22:33:36.376804+0000","last_clean":"2024-03-28T22:33:36.376804+0000","last_became_active":"2024-03-28T22:33:28.924631+0000","last_became_peered":"2024-03-28T22:33:28.924631+0000","last_unstale":"2024-03-28T22:33:36.376804+0000","last_undegraded":"2024-03-28T22:33:36.376804+0000","last_fullsized":"2024-03-28T22:33:36.376804+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:53:44.909991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.379420+0000","last_change":"2024-03-28T22:33:28.948725+0000","last_active":"2024-03-28T22:33:36.379420+0000","last_peered":"2024-03-28T22:33:36.379420+0000","last_clean":"2024-03-28T22:33:36.379420+0000","last_became_active":"2024-03-28T22:33:28.943942+0000","last_became_peered":"2024-03-28T22:33:28.943942+0000","last_unstale":"2024-03-28T22:33:36.379420+0000","last_undegraded":"2024-03-28T22:33:36.379420+0000","last_fullsized":"2024-03-28T22:33:36.379420+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:50:04.238538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:35.536156+0000","last_change":"2024-03-28T22:33:28.941546+0000","last_active":"2024-03-28T22:33:35.536156+0000","last_peered":"2024-03-28T22:33:35.536156+0000","last_clean":"2024-03-28T22:33:35.536156+0000","last_became_active":"2024-03-28T22:33:28.941278+0000","last_became_peered":"2024-03-28T22:33:28.941278+0000","last_unstale":"2024-03-28T22:33:35.536156+0000","last_undegraded":"2024-03-28T22:33:35.536156+0000","last_fullsized":"2024-03-28T22:33:35.536156+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:47:10.047067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.378112+0000","last_change":"2024-03-28T22:33:28.949552+0000","last_active":"2024-03-28T22:33:36.378112+0000","last_peered":"2024-03-28T22:33:36.378112+0000","last_clean":"2024-03-28T22:33:36.378112+0000","last_became_active":"2024-03-28T22:33:28.947432+0000","last_became_peered":"2024-03-28T22:33:28.947432+0000","last_unstale":"2024-03-28T22:33:36.378112+0000","last_undegraded":"2024-03-28T22:33:36.378112+0000","last_fullsized":"2024-03-28T22:33:36.378112+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:01:16.534642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.375153+0000","last_change":"2024-03-28T22:33:28.953930+0000","last_active":"2024-03-28T22:33:36.375153+0000","last_peered":"2024-03-28T22:33:36.375153+0000","last_clean":"2024-03-28T22:33:36.375153+0000","last_became_active":"2024-03-28T22:33:28.953591+0000","last_became_peered":"2024-03-28T22:33:28.953591+0000","last_unstale":"2024-03-28T22:33:36.375153+0000","last_undegraded":"2024-03-28T22:33:36.375153+0000","last_fullsized":"2024-03-28T22:33:36.375153+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:42:55.857476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.013763+0000","last_change":"2024-03-28T22:33:28.946307+0000","last_active":"2024-03-28T22:33:36.013763+0000","last_peered":"2024-03-28T22:33:36.013763+0000","last_clean":"2024-03-28T22:33:36.013763+0000","last_became_active":"2024-03-28T22:33:28.945932+0000","last_became_peered":"2024-03-28T22:33:28.945932+0000","last_unstale":"2024-03-28T22:33:36.013763+0000","last_undegraded":"2024-03-28T22:33:36.013763+0000","last_fullsized":"2024-03-28T22:33:36.013763+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:54:59.651668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T22:33:36.011690+0000","last_change":"2024-03-28T22:33:28.928014+0000","last_active":"2024-03-28T22:33:36.011690+0000","last_peered":"2024-03-28T22:33:36.011690+0000","last_clean":"2024-03-28T22:33:36.011690+0000","last_became_active":"2024-03-28T22:33:28.927686+0000","last_became_peered":"2024-03-28T22:33:28.927686+0000","last_unstale":"2024-03-28T22:33:36.011690+0000","last_undegraded":"2024-03-28T22:33:36.011690+0000","last_fullsized":"2024-03-28T22:33:36.011690+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:47:08.131565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8773,"num_objects":79,"num_object_clones":0,"num_object_copies":237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79,"num_whiteouts":0,"num_read":22,"num_read_kb":20,"num_write":177,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":26545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152,"ondisk_log_size":152,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318293,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1036288,"data_stored":501830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579929,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28644,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716508,"statfs":{"total":95995035648,"available":95965704192,"internally_reserved":0,"allocated":1667072,"data_stored":1092454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743456,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":501760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005093,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":497168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299435,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32768,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712384,"statfs":{"total":95995035648,"available":95961481216,"internally_reserved":0,"allocated":1630208,"data_stored":1088702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495665,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":497264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1036288,"data_stored":500966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738449,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":498020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411401,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1056768,"data_stored":501734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182465,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1048576,"data_stored":496972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920828,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":497633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757303,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28640,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716512,"statfs":{"total":95995035648,"available":95965708288,"internally_reserved":0,"allocated":1662976,"data_stored":1093826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":5544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T22:34:08.676 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-28T22:34:08.677 INFO:tasks.cephadm:Setting up client nodes... 2024-03-28T22:34:08.677 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-28T22:34:09.190 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:34:09.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:09 smithi082 ceph-mon[36471]: pgmap v342: 129 pgs: 129 active+clean; 585 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3.9 KiB/s wr, 12 op/s 2024-03-28T22:34:09.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:09 smithi082 ceph-mon[36471]: from='client.24866 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:34:09.450 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 28 22:34:09 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-b[67270]: 2024-03-28T22:34:09.322+0000 7fb4be5ee700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-03-28T22:34:09.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:09 smithi043 ceph-mon[31823]: pgmap v342: 129 pgs: 129 active+clean; 585 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3.9 KiB/s wr, 12 op/s 2024-03-28T22:34:09.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:09 smithi043 ceph-mon[31823]: from='client.24866 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:34:09.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:09 smithi172 ceph-mon[36566]: pgmap v342: 129 pgs: 129 active+clean; 585 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3.9 KiB/s wr, 12 op/s 2024-03-28T22:34:09.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:09 smithi172 ceph-mon[36566]: from='client.24866 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:34:11.275 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:11 smithi043 ceph-mon[31823]: pgmap v343: 129 pgs: 129 active+clean; 629 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 8.4 KiB/s wr, 19 op/s 2024-03-28T22:34:11.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:11 smithi082 ceph-mon[36471]: pgmap v343: 129 pgs: 129 active+clean; 629 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 8.4 KiB/s wr, 19 op/s 2024-03-28T22:34:11.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.576+0000 7f8ecfa00700 1 -- 172.21.15.43:0/2926782009 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8ec81022c0 msgr2=0x7f8ec81026e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:11.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.576+0000 7f8ecfa00700 1 --2- 172.21.15.43:0/2926782009 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8ec81022c0 0x7f8ec81026e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc0099c0 tx=0x7f8ebc009cd0 comp rx=0 tx=0).stop 2024-03-28T22:34:11.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.576+0000 7f8ecfa00700 1 -- 172.21.15.43:0/2926782009 shutdown_connections 2024-03-28T22:34:11.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.576+0000 7f8ecfa00700 1 --2- 172.21.15.43:0/2926782009 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ec8103e60 0x7f8ec810a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:11.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.576+0000 7f8ecfa00700 1 --2- 172.21.15.43:0/2926782009 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8ec81022c0 0x7f8ec81026e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:11.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.576+0000 7f8ecfa00700 1 --2- 172.21.15.43:0/2926782009 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ec81034c0 0x7f8ec8103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:11.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.576+0000 7f8ecfa00700 1 -- 172.21.15.43:0/2926782009 >> 172.21.15.43:0/2926782009 conn(0x7f8ec80fd860 msgr2=0x7f8ec80ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:11.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.576+0000 7f8ecfa00700 1 -- 172.21.15.43:0/2926782009 shutdown_connections 2024-03-28T22:34:11.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.577+0000 7f8ecfa00700 1 -- 172.21.15.43:0/2926782009 wait complete. 2024-03-28T22:34:11.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.578+0000 7f8ecfa00700 1 Processor -- start 2024-03-28T22:34:11.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.578+0000 7f8ecfa00700 1 -- start start 2024-03-28T22:34:11.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.578+0000 7f8ecfa00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ec81022c0 0x7f8ec819a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:11.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.578+0000 7f8ecfa00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8ec81034c0 0x7f8ec819ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:11.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.578+0000 7f8ecfa00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ec8103e60 0x7f8ec81a1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:11.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.578+0000 7f8ecfa00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec810dee0 con 0x7f8ec81022c0 2024-03-28T22:34:11.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.578+0000 7f8ecfa00700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec810dd60 con 0x7f8ec81034c0 2024-03-28T22:34:11.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.578+0000 7f8ecfa00700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec810e060 con 0x7f8ec8103e60 2024-03-28T22:34:11.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.579+0000 7f8ec7e00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8ec81034c0 0x7f8ec819ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:11.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.579+0000 7f8ecd000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ec81022c0 0x7f8ec819a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:11.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.579+0000 7f8ec7e00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8ec81034c0 0x7f8ec819ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:58488/0 (socket says 172.21.15.43:58488) 2024-03-28T22:34:11.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.579+0000 7f8ec7e00700 1 -- 172.21.15.43:0/1322494281 learned_addr learned my addr 172.21.15.43:0/1322494281 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:34:11.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.579+0000 7f8ecda00700 1 --2- 172.21.15.43:0/1322494281 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ec8103e60 0x7f8ec81a1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:11.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.579+0000 7f8ecda00700 1 -- 172.21.15.43:0/1322494281 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8ec81034c0 msgr2=0x7f8ec819ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:11.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.579+0000 7f8ecda00700 1 --2- 172.21.15.43:0/1322494281 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8ec81034c0 0x7f8ec819ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:11.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.579+0000 7f8ecda00700 1 -- 172.21.15.43:0/1322494281 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ec81022c0 msgr2=0x7f8ec819a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:11.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.579+0000 7f8ecda00700 1 --2- 172.21.15.43:0/1322494281 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ec81022c0 0x7f8ec819a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:11.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.579+0000 7f8ecda00700 1 -- 172.21.15.43:0/1322494281 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ec81a22b0 con 0x7f8ec8103e60 2024-03-28T22:34:11.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.579+0000 7f8ecd000700 1 --2- 172.21.15.43:0/1322494281 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ec81022c0 0x7f8ec819a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:34:11.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.579+0000 7f8ec7e00700 1 --2- 172.21.15.43:0/1322494281 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8ec81034c0 0x7f8ec819ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:34:11.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.580+0000 7f8ecda00700 1 --2- 172.21.15.43:0/1322494281 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ec8103e60 0x7f8ec81a1c10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0007f40 tx=0x7f8ec000d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:11.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.580+0000 7f8ec5600700 1 -- 172.21.15.43:0/1322494281 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec0004db0 con 0x7f8ec8103e60 2024-03-28T22:34:11.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.580+0000 7f8ec5600700 1 -- 172.21.15.43:0/1322494281 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ec0009e20 con 0x7f8ec8103e60 2024-03-28T22:34:11.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.580+0000 7f8ec5600700 1 -- 172.21.15.43:0/1322494281 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec0015400 con 0x7f8ec8103e60 2024-03-28T22:34:11.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.580+0000 7f8ecfa00700 1 -- 172.21.15.43:0/1322494281 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ec81a25a0 con 0x7f8ec8103e60 2024-03-28T22:34:11.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.580+0000 7f8ecfa00700 1 -- 172.21.15.43:0/1322494281 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ec81a2a90 con 0x7f8ec8103e60 2024-03-28T22:34:11.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.581+0000 7f8ecfa00700 1 -- 172.21.15.43:0/1322494281 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ec8050750 con 0x7f8ec8103e60 2024-03-28T22:34:11.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.583+0000 7f8ec5600700 1 -- 172.21.15.43:0/1322494281 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f8ec00052b0 con 0x7f8ec8103e60 2024-03-28T22:34:11.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.585+0000 7f8ec5600700 1 --2- 172.21.15.43:0/1322494281 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f8eb00a1f50 0x7f8eb00a4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:11.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.585+0000 7f8ec5600700 1 -- 172.21.15.43:0/1322494281 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f8ec00cdb90 con 0x7f8ec8103e60 2024-03-28T22:34:11.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.585+0000 7f8ecd000700 1 --2- 172.21.15.43:0/1322494281 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f8eb00a1f50 0x7f8eb00a4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:11.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.586+0000 7f8ecd000700 1 --2- 172.21.15.43:0/1322494281 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f8eb00a1f50 0x7f8eb00a4410 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc00b580 tx=0x7f8ebc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:11.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.589+0000 7f8ec5600700 1 -- 172.21.15.43:0/1322494281 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8ec0010070 con 0x7f8ec8103e60 2024-03-28T22:34:11.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:11 smithi172 ceph-mon[36566]: pgmap v343: 129 pgs: 129 active+clean; 629 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 8.4 KiB/s wr, 19 op/s 2024-03-28T22:34:11.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.910+0000 7f8ecfa00700 1 -- 172.21.15.43:0/1322494281 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f8ec81a2f50 con 0x7f8ec8103e60 2024-03-28T22:34:11.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.916+0000 7f8ec5600700 1 -- 172.21.15.43:0/1322494281 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f8ec81a2f50 con 0x7f8ec8103e60 2024-03-28T22:34:11.916 INFO:teuthology.orchestra.run.smithi043.stdout:[client.0] 2024-03-28T22:34:11.916 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQBj8AVmrLZzNhAAdipXjzPLOuIG8+OKMK8DQw== 2024-03-28T22:34:11.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.920+0000 7f8ecfa00700 1 -- 172.21.15.43:0/1322494281 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f8eb00a1f50 msgr2=0x7f8eb00a4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:11.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.920+0000 7f8ecfa00700 1 --2- 172.21.15.43:0/1322494281 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f8eb00a1f50 0x7f8eb00a4410 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc00b580 tx=0x7f8ebc005fb0 comp rx=0 tx=0).stop 2024-03-28T22:34:11.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.920+0000 7f8ecfa00700 1 -- 172.21.15.43:0/1322494281 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ec8103e60 msgr2=0x7f8ec81a1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:11.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.920+0000 7f8ecfa00700 1 --2- 172.21.15.43:0/1322494281 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ec8103e60 0x7f8ec81a1c10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0007f40 tx=0x7f8ec000d3b0 comp rx=0 tx=0).stop 2024-03-28T22:34:11.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.920+0000 7f8ecfa00700 1 -- 172.21.15.43:0/1322494281 shutdown_connections 2024-03-28T22:34:11.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.920+0000 7f8ecfa00700 1 --2- 172.21.15.43:0/1322494281 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f8eb00a1f50 0x7f8eb00a4410 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:11.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.920+0000 7f8ecfa00700 1 --2- 172.21.15.43:0/1322494281 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ec8103e60 0x7f8ec81a1c10 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:11.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.920+0000 7f8ecfa00700 1 --2- 172.21.15.43:0/1322494281 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ec81022c0 0x7f8ec819a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:11.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.920+0000 7f8ecfa00700 1 --2- 172.21.15.43:0/1322494281 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8ec81034c0 0x7f8ec819ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:11.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.920+0000 7f8ecfa00700 1 -- 172.21.15.43:0/1322494281 >> 172.21.15.43:0/1322494281 conn(0x7f8ec80fd860 msgr2=0x7f8ec8074470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:11.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.920+0000 7f8ecfa00700 1 -- 172.21.15.43:0/1322494281 shutdown_connections 2024-03-28T22:34:11.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:11.920+0000 7f8ecfa00700 1 -- 172.21.15.43:0/1322494281 wait complete. 2024-03-28T22:34:12.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:12 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1322494281' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T22:34:12.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:12 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T22:34:12.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:12 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T22:34:12.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:12 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1322494281' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T22:34:12.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:12 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T22:34:12.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:12 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T22:34:12.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:12 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1322494281' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T22:34:12.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:12 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T22:34:12.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:12 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T22:34:12.747 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-28T22:34:12.748 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-28T22:34:12.748 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-28T22:34:12.915 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-28T22:34:12.915 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-28T22:34:12.915 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph mgr dump --format=json 2024-03-28T22:34:13.419 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:34:14.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:13 smithi082 ceph-mon[36471]: pgmap v344: 129 pgs: 129 active+clean; 629 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 8.5 KiB/s wr, 21 op/s 2024-03-28T22:34:14.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:13 smithi043 ceph-mon[31823]: pgmap v344: 129 pgs: 129 active+clean; 629 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 8.5 KiB/s wr, 21 op/s 2024-03-28T22:34:14.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:13 smithi172 ceph-mon[36566]: pgmap v344: 129 pgs: 129 active+clean; 629 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 8.5 KiB/s wr, 21 op/s 2024-03-28T22:34:15.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.697+0000 7fe6d1c00700 1 -- 172.21.15.43:0/2520346708 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe6cc104e10 msgr2=0x7fe6cc111320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:15.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.697+0000 7fe6d1c00700 1 --2- 172.21.15.43:0/2520346708 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe6cc104e10 0x7fe6cc111320 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe6c400b0a0 tx=0x7fe6c400b3b0 comp rx=0 tx=0).stop 2024-03-28T22:34:15.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.697+0000 7fe6d1c00700 1 -- 172.21.15.43:0/2520346708 shutdown_connections 2024-03-28T22:34:15.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.697+0000 7fe6d1c00700 1 --2- 172.21.15.43:0/2520346708 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe6cc104e10 0x7fe6cc111320 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:15.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.697+0000 7fe6d1c00700 1 --2- 172.21.15.43:0/2520346708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe6cc10a4e0 0x7fe6cc10a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:15.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.697+0000 7fe6d1c00700 1 --2- 172.21.15.43:0/2520346708 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe6cc104450 0x7fe6cc1048d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:15.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.697+0000 7fe6d1c00700 1 -- 172.21.15.43:0/2520346708 >> 172.21.15.43:0/2520346708 conn(0x7fe6cc0fff40 msgr2=0x7fe6cc102360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:15.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.697+0000 7fe6d1c00700 1 -- 172.21.15.43:0/2520346708 shutdown_connections 2024-03-28T22:34:15.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.697+0000 7fe6d1c00700 1 -- 172.21.15.43:0/2520346708 wait complete. 2024-03-28T22:34:15.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.699+0000 7fe6d1c00700 1 Processor -- start 2024-03-28T22:34:15.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.699+0000 7fe6d1c00700 1 -- start start 2024-03-28T22:34:15.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.700+0000 7fe6d1c00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe6cc104450 0x7fe6cc1a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:15.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.700+0000 7fe6d1c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe6cc104e10 0x7fe6cc1a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:15.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.700+0000 7fe6d1c00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe6cc10a4e0 0x7fe6cc19a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:15.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.700+0000 7fe6d1c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe6cc1147b0 con 0x7fe6cc104e10 2024-03-28T22:34:15.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.700+0000 7fe6d1c00700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe6cc114630 con 0x7fe6cc104450 2024-03-28T22:34:15.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.700+0000 7fe6d1c00700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe6cc114930 con 0x7fe6cc10a4e0 2024-03-28T22:34:15.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.700+0000 7fe6caa00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe6cc104450 0x7fe6cc1a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:15.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.700+0000 7fe6caa00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe6cc104450 0x7fe6cc1a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44138/0 (socket says 172.21.15.43:44138) 2024-03-28T22:34:15.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.700+0000 7fe6caa00700 1 -- 172.21.15.43:0/2211634976 learned_addr learned my addr 172.21.15.43:0/2211634976 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:34:15.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.700+0000 7fe6cb400700 1 --2- 172.21.15.43:0/2211634976 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe6cc10a4e0 0x7fe6cc19a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:15.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.700+0000 7fe6ca000700 1 --2- 172.21.15.43:0/2211634976 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe6cc104e10 0x7fe6cc1a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:15.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.700+0000 7fe6caa00700 1 -- 172.21.15.43:0/2211634976 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe6cc10a4e0 msgr2=0x7fe6cc19a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:15.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.700+0000 7fe6caa00700 1 --2- 172.21.15.43:0/2211634976 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe6cc10a4e0 0x7fe6cc19a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:15.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.700+0000 7fe6caa00700 1 -- 172.21.15.43:0/2211634976 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe6cc104e10 msgr2=0x7fe6cc1a0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:15.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.700+0000 7fe6caa00700 1 --2- 172.21.15.43:0/2211634976 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe6cc104e10 0x7fe6cc1a0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:15.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.700+0000 7fe6caa00700 1 -- 172.21.15.43:0/2211634976 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6cc19b130 con 0x7fe6cc104450 2024-03-28T22:34:15.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.701+0000 7fe6caa00700 1 --2- 172.21.15.43:0/2211634976 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe6cc104450 0x7fe6cc1a0810 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc00e9c0 tx=0x7fe6bc00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:15.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.701+0000 7fe6c3400700 1 -- 172.21.15.43:0/2211634976 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc00cbd0 con 0x7fe6cc104450 2024-03-28T22:34:15.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.701+0000 7fe6c3400700 1 -- 172.21.15.43:0/2211634976 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6bc00ee60 con 0x7fe6cc104450 2024-03-28T22:34:15.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.701+0000 7fe6c3400700 1 -- 172.21.15.43:0/2211634976 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc005630 con 0x7fe6cc104450 2024-03-28T22:34:15.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.701+0000 7fe6d1c00700 1 -- 172.21.15.43:0/2211634976 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6cc19b420 con 0x7fe6cc104450 2024-03-28T22:34:15.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.701+0000 7fe6d1c00700 1 -- 172.21.15.43:0/2211634976 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6cc1a76e0 con 0x7fe6cc104450 2024-03-28T22:34:15.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.702+0000 7fe6d1c00700 1 -- 172.21.15.43:0/2211634976 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6cc068f00 con 0x7fe6cc104450 2024-03-28T22:34:15.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.704+0000 7fe6c3400700 1 -- 172.21.15.43:0/2211634976 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fe6bc004750 con 0x7fe6cc104450 2024-03-28T22:34:15.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.706+0000 7fe6c3400700 1 --2- 172.21.15.43:0/2211634976 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe6b00a1f20 0x7fe6b00a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:15.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.706+0000 7fe6c3400700 1 -- 172.21.15.43:0/2211634976 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7fe6bc0ce570 con 0x7fe6cc104450 2024-03-28T22:34:15.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.706+0000 7fe6ca000700 1 --2- 172.21.15.43:0/2211634976 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe6b00a1f20 0x7fe6b00a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:15.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.707+0000 7fe6ca000700 1 --2- 172.21.15.43:0/2211634976 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe6b00a1f20 0x7fe6b00a43e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc1099c0 tx=0x7fe6b80073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:15.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:15.708+0000 7fe6c3400700 1 -- 172.21.15.43:0/2211634976 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe6bc053280 con 0x7fe6cc104450 2024-03-28T22:34:15.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:16.000+0000 7fe6d1c00700 1 -- 172.21.15.43:0/2211634976 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fe6cc19bf00 con 0x7fe6cc104450 2024-03-28T22:34:16.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:16.005+0000 7fe6c3400700 1 -- 172.21.15.43:0/2211634976 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+261466 (secure 0 0 0) 0x7fe6bc09b5a0 con 0x7fe6cc104450 2024-03-28T22:34:16.008 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:34:16.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:16.011+0000 7fe6d1c00700 1 -- 172.21.15.43:0/2211634976 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe6b00a1f20 msgr2=0x7fe6b00a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:16.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:16.011+0000 7fe6d1c00700 1 --2- 172.21.15.43:0/2211634976 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe6b00a1f20 0x7fe6b00a43e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc1099c0 tx=0x7fe6b80073d0 comp rx=0 tx=0).stop 2024-03-28T22:34:16.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:16.011+0000 7fe6d1c00700 1 -- 172.21.15.43:0/2211634976 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe6cc104450 msgr2=0x7fe6cc1a0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:16.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:16.011+0000 7fe6d1c00700 1 --2- 172.21.15.43:0/2211634976 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe6cc104450 0x7fe6cc1a0810 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc00e9c0 tx=0x7fe6bc00ecd0 comp rx=0 tx=0).stop 2024-03-28T22:34:16.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:16.011+0000 7fe6d1c00700 1 -- 172.21.15.43:0/2211634976 shutdown_connections 2024-03-28T22:34:16.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:16.011+0000 7fe6d1c00700 1 --2- 172.21.15.43:0/2211634976 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe6b00a1f20 0x7fe6b00a43e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:16.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:16.011+0000 7fe6d1c00700 1 --2- 172.21.15.43:0/2211634976 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe6cc10a4e0 0x7fe6cc19a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:16.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:16.011+0000 7fe6d1c00700 1 --2- 172.21.15.43:0/2211634976 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe6cc104450 0x7fe6cc1a0810 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:16.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:16.011+0000 7fe6d1c00700 1 --2- 172.21.15.43:0/2211634976 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe6cc104e10 0x7fe6cc1a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:16.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:16.011+0000 7fe6d1c00700 1 -- 172.21.15.43:0/2211634976 >> 172.21.15.43:0/2211634976 conn(0x7fe6cc0fff40 msgr2=0x7fe6cc100420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:16.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:16.012+0000 7fe6d1c00700 1 -- 172.21.15.43:0/2211634976 shutdown_connections 2024-03-28T22:34:16.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:16.012+0000 7fe6d1c00700 1 -- 172.21.15.43:0/2211634976 wait complete. 2024-03-28T22:34:16.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:15 smithi082 ceph-mon[36471]: pgmap v345: 129 pgs: 129 active+clean; 629 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 8.5 KiB/s wr, 22 op/s 2024-03-28T22:34:16.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:15 smithi043 ceph-mon[31823]: pgmap v345: 129 pgs: 129 active+clean; 629 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 8.5 KiB/s wr, 22 op/s 2024-03-28T22:34:16.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:15 smithi172 ceph-mon[36566]: pgmap v345: 129 pgs: 129 active+clean; 629 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 8.5 KiB/s wr, 22 op/s 2024-03-28T22:34:16.880 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":369798199},{"type":"v1","addr":"172.21.15.43:6801","nonce":369798199}]},"active_addr":"172.21.15.43:6801/369798199","active_change":"2024-03-28T22:25:28.008369+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.43:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":1343671266}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":4024174130}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":686050369}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":1567355374}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":2250457514}]}]} 2024-03-28T22:34:16.885 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-28T22:34:16.885 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-28T22:34:16.885 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T22:34:17.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:16 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2211634976' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T22:34:17.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:16 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2211634976' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T22:34:17.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:16 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2211634976' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T22:34:17.397 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:34:18.203 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:17 smithi082 ceph-mon[36471]: pgmap v346: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.9 KiB/s wr, 22 op/s 2024-03-28T22:34:18.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:17 smithi043 ceph-mon[31823]: pgmap v346: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.9 KiB/s wr, 22 op/s 2024-03-28T22:34:18.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:17 smithi172 ceph-mon[36566]: pgmap v346: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.9 KiB/s wr, 22 op/s 2024-03-28T22:34:19.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.697+0000 7f8b50a00700 1 -- 172.21.15.43:0/2023752038 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8b4c10ed50 msgr2=0x7f8b4c111140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:19.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.697+0000 7f8b50a00700 1 --2- 172.21.15.43:0/2023752038 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8b4c10ed50 0x7f8b4c111140 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8b44009990 tx=0x7f8b44009ca0 comp rx=0 tx=0).stop 2024-03-28T22:34:19.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.697+0000 7f8b50a00700 1 -- 172.21.15.43:0/2023752038 shutdown_connections 2024-03-28T22:34:19.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.697+0000 7f8b50a00700 1 --2- 172.21.15.43:0/2023752038 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8b4c10ed50 0x7f8b4c111140 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:19.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.697+0000 7f8b50a00700 1 --2- 172.21.15.43:0/2023752038 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b4c06a940 0x7f8b4c06ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:19.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.697+0000 7f8b50a00700 1 --2- 172.21.15.43:0/2023752038 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b4c06b260 0x7f8b4c10e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:19.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.697+0000 7f8b50a00700 1 -- 172.21.15.43:0/2023752038 >> 172.21.15.43:0/2023752038 conn(0x7f8b4c077190 msgr2=0x7f8b4c0775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:19.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.697+0000 7f8b50a00700 1 -- 172.21.15.43:0/2023752038 shutdown_connections 2024-03-28T22:34:19.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.697+0000 7f8b50a00700 1 -- 172.21.15.43:0/2023752038 wait complete. 2024-03-28T22:34:19.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.698+0000 7f8b50a00700 1 Processor -- start 2024-03-28T22:34:19.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.698+0000 7f8b50a00700 1 -- start start 2024-03-28T22:34:19.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.699+0000 7f8b50a00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8b4c06a940 0x7f8b4c1a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:19.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.699+0000 7f8b50a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b4c06b260 0x7f8b4c1a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:19.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.699+0000 7f8b50a00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b4c10ed50 0x7f8b4c19a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:19.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.699+0000 7f8b50a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8b4c114780 con 0x7f8b4c06b260 2024-03-28T22:34:19.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.699+0000 7f8b50a00700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8b4c114600 con 0x7f8b4c06a940 2024-03-28T22:34:19.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.699+0000 7f8b50a00700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f8b4c114900 con 0x7f8b4c10ed50 2024-03-28T22:34:19.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.699+0000 7f8b49e00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8b4c06a940 0x7f8b4c1a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:19.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.699+0000 7f8b49e00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8b4c06a940 0x7f8b4c1a0800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44156/0 (socket says 172.21.15.43:44156) 2024-03-28T22:34:19.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.699+0000 7f8b49e00700 1 -- 172.21.15.43:0/2490634160 learned_addr learned my addr 172.21.15.43:0/2490634160 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:34:19.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.699+0000 7f8b4a800700 1 --2- 172.21.15.43:0/2490634160 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b4c10ed50 0x7f8b4c19a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:19.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.699+0000 7f8b49400700 1 --2- 172.21.15.43:0/2490634160 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b4c06b260 0x7f8b4c1a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:19.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.700+0000 7f8b49e00700 1 -- 172.21.15.43:0/2490634160 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b4c10ed50 msgr2=0x7f8b4c19a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:19.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.700+0000 7f8b49e00700 1 --2- 172.21.15.43:0/2490634160 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b4c10ed50 0x7f8b4c19a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:19.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.700+0000 7f8b49e00700 1 -- 172.21.15.43:0/2490634160 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b4c06b260 msgr2=0x7f8b4c1a0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:19.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.700+0000 7f8b49e00700 1 --2- 172.21.15.43:0/2490634160 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b4c06b260 0x7f8b4c1a0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:19.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.700+0000 7f8b49e00700 1 -- 172.21.15.43:0/2490634160 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b4c19b120 con 0x7f8b4c06a940 2024-03-28T22:34:19.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.700+0000 7f8b49e00700 1 --2- 172.21.15.43:0/2490634160 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8b4c06a940 0x7f8b4c1a0800 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c00ef30 tx=0x7f8b3c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:19.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.700+0000 7f8b42a00700 1 -- 172.21.15.43:0/2490634160 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b3c0092e0 con 0x7f8b4c06a940 2024-03-28T22:34:19.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.700+0000 7f8b42a00700 1 -- 172.21.15.43:0/2490634160 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b3c004bb0 con 0x7f8b4c06a940 2024-03-28T22:34:19.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.700+0000 7f8b42a00700 1 -- 172.21.15.43:0/2490634160 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b3c004280 con 0x7f8b4c06a940 2024-03-28T22:34:19.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.700+0000 7f8b50a00700 1 -- 172.21.15.43:0/2490634160 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b4c19b3b0 con 0x7f8b4c06a940 2024-03-28T22:34:19.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.700+0000 7f8b50a00700 1 -- 172.21.15.43:0/2490634160 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b4c1a76d0 con 0x7f8b4c06a940 2024-03-28T22:34:19.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.701+0000 7f8b50a00700 1 -- 172.21.15.43:0/2490634160 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b4c068f00 con 0x7f8b4c06a940 2024-03-28T22:34:19.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.703+0000 7f8b42a00700 1 -- 172.21.15.43:0/2490634160 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f8b3c01b030 con 0x7f8b4c06a940 2024-03-28T22:34:19.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.704+0000 7f8b42a00700 1 --2- 172.21.15.43:0/2490634160 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f8b300a1f20 0x7f8b300a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:19.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.705+0000 7f8b42a00700 1 -- 172.21.15.43:0/2490634160 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f8b3c0cdca0 con 0x7f8b4c06a940 2024-03-28T22:34:19.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.705+0000 7f8b49400700 1 --2- 172.21.15.43:0/2490634160 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f8b300a1f20 0x7f8b300a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:19.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.705+0000 7f8b49400700 1 --2- 172.21.15.43:0/2490634160 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f8b300a1f20 0x7f8b300a43e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c1025d0 tx=0x7f8b38009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:19.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.709+0000 7f8b42a00700 1 -- 172.21.15.43:0/2490634160 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8b3c022020 con 0x7f8b4c06a940 2024-03-28T22:34:19.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.935+0000 7f8b50a00700 1 -- 172.21.15.43:0/2490634160 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8b4c02db30 con 0x7f8b4c06a940 2024-03-28T22:34:19.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.937+0000 7f8b42a00700 1 -- 172.21.15.43:0/2490634160 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24093 (secure 0 0 0) 0x7f8b3c010040 con 0x7f8b4c06a940 2024-03-28T22:34:19.937 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:34:19.937 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":92,"fsid":"c2276428-ed51-11ee-95d0-87774f69a715","created":"2024-03-28T22:24:19.950084+0000","modified":"2024-03-28T22:33:34.473170+0000","last_up_change":"2024-03-28T22:32:31.078737+0000","last_in_change":"2024-03-28T22:32:10.147097+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:28:14.713290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:33:27.167435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:33:28.411825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T22:33:32.523122+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b855adcd-786c-4e42-b8fe-a515a77ce979","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6803","nonce":2826124409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6805","nonce":2826124409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6809","nonce":2826124409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6807","nonce":2826124409}]},"public_addr":"172.21.15.43:6803/2826124409","cluster_addr":"172.21.15.43:6805/2826124409","heartbeat_back_addr":"172.21.15.43:6809/2826124409","heartbeat_front_addr":"172.21.15.43:6807/2826124409","state":["exists","up"]},{"osd":1,"uuid":"031c143d-8ad1-4b68-8db9-3c717137e3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6801","nonce":3043263644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6803","nonce":3043263644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6807","nonce":3043263644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6805","nonce":3043263644}]},"public_addr":"172.21.15.82:6801/3043263644","cluster_addr":"172.21.15.82:6803/3043263644","heartbeat_back_addr":"172.21.15.82:6807/3043263644","heartbeat_front_addr":"172.21.15.82:6805/3043263644","state":["exists","up"]},{"osd":2,"uuid":"20632bbb-f07a-4ca1-b0c4-ff9d508e211c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6801","nonce":4213953645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6803","nonce":4213953645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6807","nonce":4213953645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6805","nonce":4213953645}]},"public_addr":"172.21.15.172:6801/4213953645","cluster_addr":"172.21.15.172:6803/4213953645","heartbeat_back_addr":"172.21.15.172:6807/4213953645","heartbeat_front_addr":"172.21.15.172:6805/4213953645","state":["exists","up"]},{"osd":3,"uuid":"9a1f2312-4dbd-475b-8736-d5ebc8788e09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6811","nonce":1597280920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6813","nonce":1597280920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6817","nonce":1597280920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6815","nonce":1597280920}]},"public_addr":"172.21.15.43:6811/1597280920","cluster_addr":"172.21.15.43:6813/1597280920","heartbeat_back_addr":"172.21.15.43:6817/1597280920","heartbeat_front_addr":"172.21.15.43:6815/1597280920","state":["exists","up"]},{"osd":4,"uuid":"29140ead-7075-42b2-90c1-27cae0b29ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6809","nonce":215451895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6811","nonce":215451895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6815","nonce":215451895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6813","nonce":215451895}]},"public_addr":"172.21.15.82:6809/215451895","cluster_addr":"172.21.15.82:6811/215451895","heartbeat_back_addr":"172.21.15.82:6815/215451895","heartbeat_front_addr":"172.21.15.82:6813/215451895","state":["exists","up"]},{"osd":5,"uuid":"59644c61-93dc-4536-b2ad-0256b24e5cb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6809","nonce":1576148933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6811","nonce":1576148933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6815","nonce":1576148933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6813","nonce":1576148933}]},"public_addr":"172.21.15.172:6809/1576148933","cluster_addr":"172.21.15.172:6811/1576148933","heartbeat_back_addr":"172.21.15.172:6815/1576148933","heartbeat_front_addr":"172.21.15.172:6813/1576148933","state":["exists","up"]},{"osd":6,"uuid":"22d54e57-c3f1-4ecf-9921-8e815b48f92a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6819","nonce":791044279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6821","nonce":791044279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6825","nonce":791044279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6823","nonce":791044279}]},"public_addr":"172.21.15.43:6819/791044279","cluster_addr":"172.21.15.43:6821/791044279","heartbeat_back_addr":"172.21.15.43:6825/791044279","heartbeat_front_addr":"172.21.15.43:6823/791044279","state":["exists","up"]},{"osd":7,"uuid":"248cd1e2-0d6b-452f-a0a8-c51fe1b975ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6817","nonce":2157777323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6819","nonce":2157777323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6823","nonce":2157777323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6821","nonce":2157777323}]},"public_addr":"172.21.15.82:6817/2157777323","cluster_addr":"172.21.15.82:6819/2157777323","heartbeat_back_addr":"172.21.15.82:6823/2157777323","heartbeat_front_addr":"172.21.15.82:6821/2157777323","state":["exists","up"]},{"osd":8,"uuid":"d98aab6e-8bae-4c24-906c-4f492f047b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6817","nonce":1095328248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6819","nonce":1095328248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6823","nonce":1095328248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6821","nonce":1095328248}]},"public_addr":"172.21.15.172:6817/1095328248","cluster_addr":"172.21.15.172:6819/1095328248","heartbeat_back_addr":"172.21.15.172:6823/1095328248","heartbeat_front_addr":"172.21.15.172:6821/1095328248","state":["exists","up"]},{"osd":9,"uuid":"19aa383e-a0f6-45ea-96cc-e7956535caa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6827","nonce":3657791498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6829","nonce":3657791498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6833","nonce":3657791498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6831","nonce":3657791498}]},"public_addr":"172.21.15.43:6827/3657791498","cluster_addr":"172.21.15.43:6829/3657791498","heartbeat_back_addr":"172.21.15.43:6833/3657791498","heartbeat_front_addr":"172.21.15.43:6831/3657791498","state":["exists","up"]},{"osd":10,"uuid":"666977e5-eaf7-41b9-a18f-c380e149345d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6825","nonce":3454954047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6827","nonce":3454954047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6831","nonce":3454954047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6829","nonce":3454954047}]},"public_addr":"172.21.15.82:6825/3454954047","cluster_addr":"172.21.15.82:6827/3454954047","heartbeat_back_addr":"172.21.15.82:6831/3454954047","heartbeat_front_addr":"172.21.15.82:6829/3454954047","state":["exists","up"]},{"osd":11,"uuid":"615cf3be-8076-470e-ad09-c880e7593717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6825","nonce":2563517144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6827","nonce":2563517144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6831","nonce":2563517144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6829","nonce":2563517144}]},"public_addr":"172.21.15.172:6825/2563517144","cluster_addr":"172.21.15.172:6827/2563517144","heartbeat_back_addr":"172.21.15.172:6831/2563517144","heartbeat_front_addr":"172.21.15.172:6829/2563517144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:21.727405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:47.100259+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:10.615438+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:43.632144+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:09.487940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:34.604463+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:09.594452+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:35.709795+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:01.184202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:35.455322+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:02.413166+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:28.550754+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/578791143":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/4029313215":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/1574178482":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6801/3517712438":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6800/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/1375099638":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/2710117652":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6801/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/3215634168":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6800/3517712438":"2024-03-29T22:25:28.008215+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:34:19.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.941+0000 7f8b50a00700 1 -- 172.21.15.43:0/2490634160 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f8b300a1f20 msgr2=0x7f8b300a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:19.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.941+0000 7f8b50a00700 1 --2- 172.21.15.43:0/2490634160 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f8b300a1f20 0x7f8b300a43e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c1025d0 tx=0x7f8b38009260 comp rx=0 tx=0).stop 2024-03-28T22:34:19.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.941+0000 7f8b50a00700 1 -- 172.21.15.43:0/2490634160 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8b4c06a940 msgr2=0x7f8b4c1a0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:19.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.941+0000 7f8b50a00700 1 --2- 172.21.15.43:0/2490634160 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8b4c06a940 0x7f8b4c1a0800 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c00ef30 tx=0x7f8b3c00c3a0 comp rx=0 tx=0).stop 2024-03-28T22:34:19.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.941+0000 7f8b50a00700 1 -- 172.21.15.43:0/2490634160 shutdown_connections 2024-03-28T22:34:19.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.941+0000 7f8b50a00700 1 --2- 172.21.15.43:0/2490634160 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f8b300a1f20 0x7f8b300a43e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:19.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.941+0000 7f8b50a00700 1 --2- 172.21.15.43:0/2490634160 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8b4c10ed50 0x7f8b4c19a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:19.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.941+0000 7f8b50a00700 1 --2- 172.21.15.43:0/2490634160 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8b4c06a940 0x7f8b4c1a0800 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:19.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.941+0000 7f8b50a00700 1 --2- 172.21.15.43:0/2490634160 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8b4c06b260 0x7f8b4c1a0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:19.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.941+0000 7f8b50a00700 1 -- 172.21.15.43:0/2490634160 >> 172.21.15.43:0/2490634160 conn(0x7f8b4c077190 msgr2=0x7f8b4c110160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:19.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.941+0000 7f8b50a00700 1 -- 172.21.15.43:0/2490634160 shutdown_connections 2024-03-28T22:34:19.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:19.941+0000 7f8b50a00700 1 -- 172.21.15.43:0/2490634160 wait complete. 2024-03-28T22:34:20.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:19 smithi082 ceph-mon[36471]: pgmap v347: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.7 KiB/s wr, 22 op/s 2024-03-28T22:34:20.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:19 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2490634160' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:34:20.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:19 smithi043 ceph-mon[31823]: pgmap v347: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.7 KiB/s wr, 22 op/s 2024-03-28T22:34:20.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:19 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2490634160' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:34:20.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:19 smithi172 ceph-mon[36566]: pgmap v347: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.7 KiB/s wr, 22 op/s 2024-03-28T22:34:20.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:19 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2490634160' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:34:20.793 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-28T22:34:20.793 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T22:34:21.273 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:34:22.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:21 smithi082 ceph-mon[36471]: pgmap v348: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.8 KiB/s wr, 16 op/s 2024-03-28T22:34:22.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:21 smithi043 ceph-mon[31823]: pgmap v348: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.8 KiB/s wr, 16 op/s 2024-03-28T22:34:22.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:21 smithi172 ceph-mon[36566]: pgmap v348: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.8 KiB/s wr, 16 op/s 2024-03-28T22:34:23.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.559+0000 7fcaa6800700 1 -- 172.21.15.43:0/1374833012 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcaa010a4c0 msgr2=0x7fcaa010a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:23.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.559+0000 7fcaa6800700 1 --2- 172.21.15.43:0/1374833012 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcaa010a4c0 0x7fcaa010a8a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fca9000b0a0 tx=0x7fca9000b3b0 comp rx=0 tx=0).stop 2024-03-28T22:34:23.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.559+0000 7fcaa6800700 1 -- 172.21.15.43:0/1374833012 shutdown_connections 2024-03-28T22:34:23.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.559+0000 7fcaa6800700 1 --2- 172.21.15.43:0/1374833012 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcaa0104df0 0x7fcaa0111300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:23.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.559+0000 7fcaa6800700 1 --2- 172.21.15.43:0/1374833012 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcaa010a4c0 0x7fcaa010a8a0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:23.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.559+0000 7fcaa6800700 1 --2- 172.21.15.43:0/1374833012 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcaa0104430 0x7fcaa01048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:23.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.559+0000 7fcaa6800700 1 -- 172.21.15.43:0/1374833012 >> 172.21.15.43:0/1374833012 conn(0x7fcaa00fff40 msgr2=0x7fcaa0102360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:23.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.559+0000 7fcaa6800700 1 -- 172.21.15.43:0/1374833012 shutdown_connections 2024-03-28T22:34:23.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.559+0000 7fcaa6800700 1 -- 172.21.15.43:0/1374833012 wait complete. 2024-03-28T22:34:23.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.560+0000 7fcaa6800700 1 Processor -- start 2024-03-28T22:34:23.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.560+0000 7fcaa6800700 1 -- start start 2024-03-28T22:34:23.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.561+0000 7fcaa6800700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcaa0104430 0x7fcaa01a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:23.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.561+0000 7fcaa6800700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcaa0104df0 0x7fcaa01a0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:23.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.561+0000 7fcaa6800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcaa010a4c0 0x7fcaa019aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:23.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.561+0000 7fcaa6800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcaa0114770 con 0x7fcaa010a4c0 2024-03-28T22:34:23.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.561+0000 7fcaa6800700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fcaa01145f0 con 0x7fcaa0104430 2024-03-28T22:34:23.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.561+0000 7fcaa6800700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fcaa01148f0 con 0x7fcaa0104df0 2024-03-28T22:34:23.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.561+0000 7fca9f400700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcaa0104430 0x7fcaa01a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:23.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.561+0000 7fca9ea00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcaa0104df0 0x7fcaa01a0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:23.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.561+0000 7fca9f400700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcaa0104430 0x7fcaa01a08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44172/0 (socket says 172.21.15.43:44172) 2024-03-28T22:34:23.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.561+0000 7fca9f400700 1 -- 172.21.15.43:0/3045461397 learned_addr learned my addr 172.21.15.43:0/3045461397 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:34:23.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.561+0000 7fca9fe00700 1 --2- 172.21.15.43:0/3045461397 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcaa010a4c0 0x7fcaa019aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:23.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.562+0000 7fca9f400700 1 -- 172.21.15.43:0/3045461397 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcaa0104df0 msgr2=0x7fcaa01a0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:23.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.562+0000 7fca9f400700 1 --2- 172.21.15.43:0/3045461397 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcaa0104df0 0x7fcaa01a0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:23.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.562+0000 7fca9f400700 1 -- 172.21.15.43:0/3045461397 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcaa010a4c0 msgr2=0x7fcaa019aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:23.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.562+0000 7fca9f400700 1 --2- 172.21.15.43:0/3045461397 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcaa010a4c0 0x7fcaa019aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:23.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.562+0000 7fca9f400700 1 -- 172.21.15.43:0/3045461397 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaa019b230 con 0x7fcaa0104430 2024-03-28T22:34:23.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.562+0000 7fca9f400700 1 --2- 172.21.15.43:0/3045461397 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcaa0104430 0x7fcaa01a08c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fca9000b070 tx=0x7fca90004cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:23.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.562+0000 7fca97e00700 1 -- 172.21.15.43:0/3045461397 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca90021070 con 0x7fcaa0104430 2024-03-28T22:34:23.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.562+0000 7fca97e00700 1 -- 172.21.15.43:0/3045461397 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca90003c00 con 0x7fcaa0104430 2024-03-28T22:34:23.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.562+0000 7fcaa6800700 1 -- 172.21.15.43:0/3045461397 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcaa019b4c0 con 0x7fcaa0104430 2024-03-28T22:34:23.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.562+0000 7fca97e00700 1 -- 172.21.15.43:0/3045461397 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca900048a0 con 0x7fcaa0104430 2024-03-28T22:34:23.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.562+0000 7fcaa6800700 1 -- 172.21.15.43:0/3045461397 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcaa01a7790 con 0x7fcaa0104430 2024-03-28T22:34:23.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.564+0000 7fcaa6800700 1 -- 172.21.15.43:0/3045461397 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcaa0068f00 con 0x7fcaa0104430 2024-03-28T22:34:23.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.566+0000 7fca97e00700 1 -- 172.21.15.43:0/3045461397 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fca90003db0 con 0x7fcaa0104430 2024-03-28T22:34:23.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.567+0000 7fca97e00700 1 --2- 172.21.15.43:0/3045461397 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fca840a2040 0x7fca840a4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:23.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.567+0000 7fca97e00700 1 -- 172.21.15.43:0/3045461397 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7fca9000f020 con 0x7fcaa0104430 2024-03-28T22:34:23.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.567+0000 7fca9ea00700 1 --2- 172.21.15.43:0/3045461397 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fca840a2040 0x7fca840a4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:23.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.568+0000 7fca9ea00700 1 --2- 172.21.15.43:0/3045461397 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fca840a2040 0x7fca840a4500 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fcaa0109a10 tx=0x7fca8c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:23.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.569+0000 7fca97e00700 1 -- 172.21.15.43:0/3045461397 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fca9002a020 con 0x7fcaa0104430 2024-03-28T22:34:23.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.799+0000 7fcaa6800700 1 -- 172.21.15.43:0/3045461397 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcaa002db30 con 0x7fcaa0104430 2024-03-28T22:34:23.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.802+0000 7fca97e00700 1 -- 172.21.15.43:0/3045461397 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24093 (secure 0 0 0) 0x7fca9009cbc0 con 0x7fcaa0104430 2024-03-28T22:34:23.801 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:34:23.802 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":92,"fsid":"c2276428-ed51-11ee-95d0-87774f69a715","created":"2024-03-28T22:24:19.950084+0000","modified":"2024-03-28T22:33:34.473170+0000","last_up_change":"2024-03-28T22:32:31.078737+0000","last_in_change":"2024-03-28T22:32:10.147097+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:28:14.713290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:33:27.167435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:33:28.411825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T22:33:32.523122+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b855adcd-786c-4e42-b8fe-a515a77ce979","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6803","nonce":2826124409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6805","nonce":2826124409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6809","nonce":2826124409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6807","nonce":2826124409}]},"public_addr":"172.21.15.43:6803/2826124409","cluster_addr":"172.21.15.43:6805/2826124409","heartbeat_back_addr":"172.21.15.43:6809/2826124409","heartbeat_front_addr":"172.21.15.43:6807/2826124409","state":["exists","up"]},{"osd":1,"uuid":"031c143d-8ad1-4b68-8db9-3c717137e3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6801","nonce":3043263644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6803","nonce":3043263644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6807","nonce":3043263644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6805","nonce":3043263644}]},"public_addr":"172.21.15.82:6801/3043263644","cluster_addr":"172.21.15.82:6803/3043263644","heartbeat_back_addr":"172.21.15.82:6807/3043263644","heartbeat_front_addr":"172.21.15.82:6805/3043263644","state":["exists","up"]},{"osd":2,"uuid":"20632bbb-f07a-4ca1-b0c4-ff9d508e211c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6801","nonce":4213953645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6803","nonce":4213953645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6807","nonce":4213953645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6805","nonce":4213953645}]},"public_addr":"172.21.15.172:6801/4213953645","cluster_addr":"172.21.15.172:6803/4213953645","heartbeat_back_addr":"172.21.15.172:6807/4213953645","heartbeat_front_addr":"172.21.15.172:6805/4213953645","state":["exists","up"]},{"osd":3,"uuid":"9a1f2312-4dbd-475b-8736-d5ebc8788e09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6811","nonce":1597280920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6813","nonce":1597280920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6817","nonce":1597280920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6815","nonce":1597280920}]},"public_addr":"172.21.15.43:6811/1597280920","cluster_addr":"172.21.15.43:6813/1597280920","heartbeat_back_addr":"172.21.15.43:6817/1597280920","heartbeat_front_addr":"172.21.15.43:6815/1597280920","state":["exists","up"]},{"osd":4,"uuid":"29140ead-7075-42b2-90c1-27cae0b29ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6809","nonce":215451895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6811","nonce":215451895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6815","nonce":215451895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6813","nonce":215451895}]},"public_addr":"172.21.15.82:6809/215451895","cluster_addr":"172.21.15.82:6811/215451895","heartbeat_back_addr":"172.21.15.82:6815/215451895","heartbeat_front_addr":"172.21.15.82:6813/215451895","state":["exists","up"]},{"osd":5,"uuid":"59644c61-93dc-4536-b2ad-0256b24e5cb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6809","nonce":1576148933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6811","nonce":1576148933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6815","nonce":1576148933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6813","nonce":1576148933}]},"public_addr":"172.21.15.172:6809/1576148933","cluster_addr":"172.21.15.172:6811/1576148933","heartbeat_back_addr":"172.21.15.172:6815/1576148933","heartbeat_front_addr":"172.21.15.172:6813/1576148933","state":["exists","up"]},{"osd":6,"uuid":"22d54e57-c3f1-4ecf-9921-8e815b48f92a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6819","nonce":791044279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6821","nonce":791044279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6825","nonce":791044279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6823","nonce":791044279}]},"public_addr":"172.21.15.43:6819/791044279","cluster_addr":"172.21.15.43:6821/791044279","heartbeat_back_addr":"172.21.15.43:6825/791044279","heartbeat_front_addr":"172.21.15.43:6823/791044279","state":["exists","up"]},{"osd":7,"uuid":"248cd1e2-0d6b-452f-a0a8-c51fe1b975ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6817","nonce":2157777323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6819","nonce":2157777323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6823","nonce":2157777323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6821","nonce":2157777323}]},"public_addr":"172.21.15.82:6817/2157777323","cluster_addr":"172.21.15.82:6819/2157777323","heartbeat_back_addr":"172.21.15.82:6823/2157777323","heartbeat_front_addr":"172.21.15.82:6821/2157777323","state":["exists","up"]},{"osd":8,"uuid":"d98aab6e-8bae-4c24-906c-4f492f047b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6817","nonce":1095328248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6819","nonce":1095328248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6823","nonce":1095328248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6821","nonce":1095328248}]},"public_addr":"172.21.15.172:6817/1095328248","cluster_addr":"172.21.15.172:6819/1095328248","heartbeat_back_addr":"172.21.15.172:6823/1095328248","heartbeat_front_addr":"172.21.15.172:6821/1095328248","state":["exists","up"]},{"osd":9,"uuid":"19aa383e-a0f6-45ea-96cc-e7956535caa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6827","nonce":3657791498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6829","nonce":3657791498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6833","nonce":3657791498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6831","nonce":3657791498}]},"public_addr":"172.21.15.43:6827/3657791498","cluster_addr":"172.21.15.43:6829/3657791498","heartbeat_back_addr":"172.21.15.43:6833/3657791498","heartbeat_front_addr":"172.21.15.43:6831/3657791498","state":["exists","up"]},{"osd":10,"uuid":"666977e5-eaf7-41b9-a18f-c380e149345d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6825","nonce":3454954047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6827","nonce":3454954047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6831","nonce":3454954047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6829","nonce":3454954047}]},"public_addr":"172.21.15.82:6825/3454954047","cluster_addr":"172.21.15.82:6827/3454954047","heartbeat_back_addr":"172.21.15.82:6831/3454954047","heartbeat_front_addr":"172.21.15.82:6829/3454954047","state":["exists","up"]},{"osd":11,"uuid":"615cf3be-8076-470e-ad09-c880e7593717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6825","nonce":2563517144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6827","nonce":2563517144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6831","nonce":2563517144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6829","nonce":2563517144}]},"public_addr":"172.21.15.172:6825/2563517144","cluster_addr":"172.21.15.172:6827/2563517144","heartbeat_back_addr":"172.21.15.172:6831/2563517144","heartbeat_front_addr":"172.21.15.172:6829/2563517144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:21.727405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:47.100259+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:10.615438+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:43.632144+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:09.487940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:34.604463+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:09.594452+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:35.709795+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:01.184202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:35.455322+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:02.413166+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:28.550754+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/578791143":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/4029313215":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/1574178482":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6801/3517712438":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6800/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/1375099638":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/2710117652":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6801/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/3215634168":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6800/3517712438":"2024-03-29T22:25:28.008215+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:34:23.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.805+0000 7fcaa6800700 1 -- 172.21.15.43:0/3045461397 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fca840a2040 msgr2=0x7fca840a4500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:23.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.805+0000 7fcaa6800700 1 --2- 172.21.15.43:0/3045461397 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fca840a2040 0x7fca840a4500 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fcaa0109a10 tx=0x7fca8c009260 comp rx=0 tx=0).stop 2024-03-28T22:34:23.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.805+0000 7fcaa6800700 1 -- 172.21.15.43:0/3045461397 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcaa0104430 msgr2=0x7fcaa01a08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:23.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.805+0000 7fcaa6800700 1 --2- 172.21.15.43:0/3045461397 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcaa0104430 0x7fcaa01a08c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fca9000b070 tx=0x7fca90004cf0 comp rx=0 tx=0).stop 2024-03-28T22:34:23.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.805+0000 7fcaa6800700 1 -- 172.21.15.43:0/3045461397 shutdown_connections 2024-03-28T22:34:23.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.805+0000 7fcaa6800700 1 --2- 172.21.15.43:0/3045461397 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fca840a2040 0x7fca840a4500 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:23.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.805+0000 7fcaa6800700 1 --2- 172.21.15.43:0/3045461397 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcaa010a4c0 0x7fcaa019aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:23.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.805+0000 7fcaa6800700 1 --2- 172.21.15.43:0/3045461397 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcaa0104430 0x7fcaa01a08c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:23.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.805+0000 7fcaa6800700 1 --2- 172.21.15.43:0/3045461397 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcaa0104df0 0x7fcaa01a0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:23.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.805+0000 7fcaa6800700 1 -- 172.21.15.43:0/3045461397 >> 172.21.15.43:0/3045461397 conn(0x7fcaa00fff40 msgr2=0x7fcaa01005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:23.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.805+0000 7fcaa6800700 1 -- 172.21.15.43:0/3045461397 shutdown_connections 2024-03-28T22:34:23.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:23.805+0000 7fcaa6800700 1 -- 172.21.15.43:0/3045461397 wait complete. 2024-03-28T22:34:24.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:23 smithi082 ceph-mon[36471]: pgmap v349: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-03-28T22:34:24.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:23 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3045461397' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:34:24.245 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:23 smithi043 ceph-mon[31823]: pgmap v349: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-03-28T22:34:24.245 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:23 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3045461397' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:34:24.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:23 smithi172 ceph-mon[36566]: pgmap v349: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-03-28T22:34:24.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:23 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3045461397' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:34:24.654 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-28T22:34:24.654 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-28T22:34:24.655 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-28T22:34:24.655 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-28T22:34:24.655 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-28T22:34:24.656 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-28T22:34:24.656 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-28T22:34:24.657 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-28T22:34:24.657 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-28T22:34:24.657 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-28T22:34:24.658 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-28T22:34:24.658 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-28T22:34:25.772 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:34:25.778 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:34:25.781 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:34:25.796 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:34:25.797 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:34:25.811 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:34:25.865 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:34:26.235 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:25 smithi043 ceph-mon[31823]: pgmap v350: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-28T22:34:26.251 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:34:26.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:25 smithi172 ceph-mon[36566]: pgmap v350: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-28T22:34:26.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:25 smithi082 ceph-mon[36471]: pgmap v350: 129 pgs: 129 active+clean; 641 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-28T22:34:27.990 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:27 smithi043 ceph-mon[31823]: pgmap v351: 129 pgs: 129 active+clean; 645 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-28T22:34:28.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:27 smithi172 ceph-mon[36566]: pgmap v351: 129 pgs: 129 active+clean; 645 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-28T22:34:28.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:27 smithi082 ceph-mon[36471]: pgmap v351: 129 pgs: 129 active+clean; 645 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-28T22:34:29.249 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:28 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-03-28T22:34:29.249 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:28 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-03-28T22:34:29.249 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:28 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]: dispatch 2024-03-28T22:34:29.249 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:28 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]: dispatch 2024-03-28T22:34:29.250 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:28 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]: dispatch 2024-03-28T22:34:29.250 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:28 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]: dispatch 2024-03-28T22:34:29.250 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:28 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-03-28T22:34:29.250 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:28 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-03-28T22:34:29.250 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:28 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-03-28T22:34:29.250 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:28 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2024-03-28T22:34:29.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:28 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-03-28T22:34:29.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:28 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-03-28T22:34:29.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:28 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]: dispatch 2024-03-28T22:34:29.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:28 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]: dispatch 2024-03-28T22:34:29.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:28 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]: dispatch 2024-03-28T22:34:29.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:28 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]: dispatch 2024-03-28T22:34:29.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:28 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-03-28T22:34:29.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:28 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-03-28T22:34:29.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:28 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-03-28T22:34:29.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:28 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2024-03-28T22:34:29.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:28 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-03-28T22:34:29.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:28 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-03-28T22:34:29.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:28 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]: dispatch 2024-03-28T22:34:29.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:28 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]: dispatch 2024-03-28T22:34:29.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:28 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]: dispatch 2024-03-28T22:34:29.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:28 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]: dispatch 2024-03-28T22:34:29.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:28 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-03-28T22:34:29.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:28 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-03-28T22:34:29.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:28 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-03-28T22:34:29.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:28 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2024-03-28T22:34:30.221 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:29 smithi043 ceph-mon[31823]: pgmap v352: 129 pgs: 129 active+clean; 645 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-28T22:34:30.222 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:29 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-03-28T22:34:30.222 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:29 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-03-28T22:34:30.222 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:29 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2024-03-28T22:34:30.222 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:29 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]': finished 2024-03-28T22:34:30.222 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:29 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]': finished 2024-03-28T22:34:30.222 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:29 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]': finished 2024-03-28T22:34:30.222 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:29 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-03-28T22:34:30.222 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:29 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-03-28T22:34:30.222 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:29 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-03-28T22:34:30.222 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:29 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-03-28T22:34:30.222 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:29 smithi043 ceph-mon[31823]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T22:34:30.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:29 smithi172 ceph-mon[36566]: pgmap v352: 129 pgs: 129 active+clean; 645 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-28T22:34:30.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:29 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-03-28T22:34:30.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:29 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-03-28T22:34:30.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:29 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2024-03-28T22:34:30.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:29 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]': finished 2024-03-28T22:34:30.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:29 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]': finished 2024-03-28T22:34:30.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:29 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]': finished 2024-03-28T22:34:30.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:29 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-03-28T22:34:30.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:29 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-03-28T22:34:30.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:29 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-03-28T22:34:30.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:29 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-03-28T22:34:30.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:29 smithi172 ceph-mon[36566]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T22:34:30.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:29 smithi082 ceph-mon[36471]: pgmap v352: 129 pgs: 129 active+clean; 645 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-28T22:34:30.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:29 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-03-28T22:34:30.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:29 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-03-28T22:34:30.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:29 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2024-03-28T22:34:30.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:29 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]': finished 2024-03-28T22:34:30.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:29 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]': finished 2024-03-28T22:34:30.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:29 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]': finished 2024-03-28T22:34:30.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:29 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-03-28T22:34:30.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:29 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-03-28T22:34:30.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:29 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-03-28T22:34:30.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:29 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-03-28T22:34:30.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:29 smithi082 ceph-mon[36471]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T22:34:31.092 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:30 smithi043 ceph-mon[31823]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T22:34:31.092 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:30 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:34:31.258 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:30 smithi172 ceph-mon[36566]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T22:34:31.258 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:30 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:34:31.316 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:30 smithi082 ceph-mon[36471]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T22:34:31.316 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:30 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:34:32.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:32 smithi043 ceph-mon[31823]: pgmap v355: 129 pgs: 1 peering, 128 active+clean; 645 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 639 B/s wr, 11 op/s 2024-03-28T22:34:32.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:32 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:34:32.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:32 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:34:32.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:32 smithi172 ceph-mon[36566]: pgmap v355: 129 pgs: 1 peering, 128 active+clean; 645 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 639 B/s wr, 11 op/s 2024-03-28T22:34:32.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:32 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:34:32.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:32 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:34:32.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:32 smithi082 ceph-mon[36471]: pgmap v355: 129 pgs: 1 peering, 128 active+clean; 645 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 639 B/s wr, 11 op/s 2024-03-28T22:34:32.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:32 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:34:32.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:32 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:34:33.226 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:33 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:34:33.227 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:33 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:34:33.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.254+0000 7f7229e00700 1 -- 172.21.15.43:0/3471445932 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f722410ec40 msgr2=0x7f7224111030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:33.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.254+0000 7f7229e00700 1 --2- 172.21.15.43:0/3471445932 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f722410ec40 0x7f7224111030 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f721c009f90 tx=0x7f721c00b330 comp rx=0 tx=0).stop 2024-03-28T22:34:33.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.254+0000 7f7229e00700 1 -- 172.21.15.43:0/3471445932 shutdown_connections 2024-03-28T22:34:33.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.254+0000 7f7229e00700 1 --2- 172.21.15.43:0/3471445932 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f722410ec40 0x7f7224111030 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:33.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.254+0000 7f7229e00700 1 --2- 172.21.15.43:0/3471445932 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7224074470 0x7f7224074850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:33.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.254+0000 7f7229e00700 1 --2- 172.21.15.43:0/3471445932 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7224074d90 0x7f722410e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:33.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.254+0000 7f7229e00700 1 -- 172.21.15.43:0/3471445932 >> 172.21.15.43:0/3471445932 conn(0x7f72240793a0 msgr2=0x7f72240797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:33.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.254+0000 7f7229e00700 1 -- 172.21.15.43:0/3471445932 shutdown_connections 2024-03-28T22:34:33.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.254+0000 7f7229e00700 1 -- 172.21.15.43:0/3471445932 wait complete. 2024-03-28T22:34:33.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.255+0000 7f7229e00700 1 Processor -- start 2024-03-28T22:34:33.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.255+0000 7f7229e00700 1 -- start start 2024-03-28T22:34:33.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.256+0000 7f7229e00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7224074470 0x7f72241a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:33.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.256+0000 7f7229e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7224074d90 0x7f72241a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:33.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.256+0000 7f7229e00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f722410ec40 0x7f722419aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:33.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.256+0000 7f7229e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7224114820 con 0x7f7224074d90 2024-03-28T22:34:33.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.256+0000 7f7229e00700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f72241146a0 con 0x7f7224074470 2024-03-28T22:34:33.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.256+0000 7f7229e00700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f72241149a0 con 0x7f722410ec40 2024-03-28T22:34:33.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.256+0000 7f7222a00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7224074470 0x7f72241a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:33.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.256+0000 7f7223400700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f722410ec40 0x7f722419aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:33.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.256+0000 7f7222000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7224074d90 0x7f72241a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:33.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.256+0000 7f7222a00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7224074470 0x7f72241a0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:56226/0 (socket says 172.21.15.43:56226) 2024-03-28T22:34:33.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.256+0000 7f7222a00700 1 -- 172.21.15.43:0/4003411058 learned_addr learned my addr 172.21.15.43:0/4003411058 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:34:33.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.256+0000 7f7223400700 1 --2- 172.21.15.43:0/4003411058 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f722410ec40 0x7f722419aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:51194/0 (socket says 172.21.15.43:51194) 2024-03-28T22:34:33.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.257+0000 7f7223400700 1 -- 172.21.15.43:0/4003411058 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7224074470 msgr2=0x7f72241a0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:33.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.257+0000 7f7223400700 1 --2- 172.21.15.43:0/4003411058 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7224074470 0x7f72241a0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:33.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.257+0000 7f7223400700 1 -- 172.21.15.43:0/4003411058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7224074d90 msgr2=0x7f72241a0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:33.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.257+0000 7f7223400700 1 --2- 172.21.15.43:0/4003411058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7224074d90 0x7f72241a0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:33.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.257+0000 7f7223400700 1 -- 172.21.15.43:0/4003411058 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f722419b1a0 con 0x7f722410ec40 2024-03-28T22:34:33.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.257+0000 7f7223400700 1 --2- 172.21.15.43:0/4003411058 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f722410ec40 0x7f722419aa40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f721c009f90 tx=0x7f721c00cef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:33.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.257+0000 7f7228a00700 1 -- 172.21.15.43:0/4003411058 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f721c021070 con 0x7f722410ec40 2024-03-28T22:34:33.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.257+0000 7f7228a00700 1 -- 172.21.15.43:0/4003411058 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f721c007ac0 con 0x7f722410ec40 2024-03-28T22:34:33.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.257+0000 7f7229e00700 1 -- 172.21.15.43:0/4003411058 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f722419b430 con 0x7f722410ec40 2024-03-28T22:34:33.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.257+0000 7f7228a00700 1 -- 172.21.15.43:0/4003411058 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f721c005040 con 0x7f722410ec40 2024-03-28T22:34:33.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.257+0000 7f7229e00700 1 -- 172.21.15.43:0/4003411058 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72241a7750 con 0x7f722410ec40 2024-03-28T22:34:33.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.258+0000 7f7229e00700 1 -- 172.21.15.43:0/4003411058 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f72240ffdf0 con 0x7f722410ec40 2024-03-28T22:34:33.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.260+0000 7f7228a00700 1 -- 172.21.15.43:0/4003411058 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f721c024070 con 0x7f722410ec40 2024-03-28T22:34:33.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.261+0000 7f7228a00700 1 --2- 172.21.15.43:0/4003411058 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f72080a1ea0 0x7f72080a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:33.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.262+0000 7f7228a00700 1 -- 172.21.15.43:0/4003411058 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7f721c00f020 con 0x7f722410ec40 2024-03-28T22:34:33.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.262+0000 7f7222a00700 1 --2- 172.21.15.43:0/4003411058 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f72080a1ea0 0x7f72080a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:33.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.262+0000 7f7228a00700 1 --2- 172.21.15.43:0/4003411058 >> [v2:172.21.15.82:6816/2157777323,v1:172.21.15.82:6817/2157777323] conn(0x7f72080ad000 0x7f72080af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:33.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.262+0000 7f7228a00700 1 -- 172.21.15.43:0/4003411058 --> [v2:172.21.15.82:6816/2157777323,v1:172.21.15.82:6817/2157777323] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f721c008ea0 con 0x7f72080ad000 2024-03-28T22:34:33.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.262+0000 7f7228a00700 1 -- 172.21.15.43:0/4003411058 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_get_version_reply(handle=1 version=94) ==== 24+0+0 (secure 0 0 0) 0x7f721c0d6050 con 0x7f722410ec40 2024-03-28T22:34:33.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.262+0000 7f7222000700 1 --2- 172.21.15.43:0/4003411058 >> [v2:172.21.15.82:6816/2157777323,v1:172.21.15.82:6817/2157777323] conn(0x7f72080ad000 0x7f72080af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:33.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.263+0000 7f7222a00700 1 --2- 172.21.15.43:0/4003411058 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f72080a1ea0 0x7f72080a4360 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f722419bb80 tx=0x7f7214009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:33.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.263+0000 7f7222000700 1 --2- 172.21.15.43:0/4003411058 >> [v2:172.21.15.82:6816/2157777323,v1:172.21.15.82:6817/2157777323] conn(0x7f72080ad000 0x7f72080af4a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:33.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.264+0000 7f7228a00700 1 -- 172.21.15.43:0/4003411058 <== osd.7 v2:172.21.15.82:6816/2157777323 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f721c008ea0 con 0x7f72080ad000 2024-03-28T22:34:33.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.288+0000 7f7229e00700 1 -- 172.21.15.43:0/4003411058 --> [v2:172.21.15.82:6816/2157777323,v1:172.21.15.82:6817/2157777323] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7224075ba0 con 0x7f72080ad000 2024-03-28T22:34:33.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.289+0000 7f7228a00700 1 -- 172.21.15.43:0/4003411058 <== osd.7 v2:172.21.15.82:6816/2157777323 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7224075ba0 con 0x7f72080ad000 2024-03-28T22:34:33.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.289+0000 7f7229e00700 1 -- 172.21.15.43:0/4003411058 >> [v2:172.21.15.82:6816/2157777323,v1:172.21.15.82:6817/2157777323] conn(0x7f72080ad000 msgr2=0x7f72080af4a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:33.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.289+0000 7f7229e00700 1 --2- 172.21.15.43:0/4003411058 >> [v2:172.21.15.82:6816/2157777323,v1:172.21.15.82:6817/2157777323] conn(0x7f72080ad000 0x7f72080af4a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:33.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.289+0000 7f7229e00700 1 -- 172.21.15.43:0/4003411058 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f72080a1ea0 msgr2=0x7f72080a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:33.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.289+0000 7f7229e00700 1 --2- 172.21.15.43:0/4003411058 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f72080a1ea0 0x7f72080a4360 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f722419bb80 tx=0x7f7214009260 comp rx=0 tx=0).stop 2024-03-28T22:34:33.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.289+0000 7f7229e00700 1 -- 172.21.15.43:0/4003411058 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f722410ec40 msgr2=0x7f722419aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:33.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.289+0000 7f7229e00700 1 --2- 172.21.15.43:0/4003411058 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f722410ec40 0x7f722419aa40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f721c009f90 tx=0x7f721c00cef0 comp rx=0 tx=0).stop 2024-03-28T22:34:33.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.289+0000 7f7223400700 1 -- 172.21.15.43:0/4003411058 reap_dead start 2024-03-28T22:34:33.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.290+0000 7f7229e00700 1 -- 172.21.15.43:0/4003411058 shutdown_connections 2024-03-28T22:34:33.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.290+0000 7f7229e00700 1 -- 172.21.15.43:0/4003411058 >> 172.21.15.43:0/4003411058 conn(0x7f72240793a0 msgr2=0x7f72240ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:33.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.290+0000 7f7229e00700 1 -- 172.21.15.43:0/4003411058 shutdown_connections 2024-03-28T22:34:33.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.290+0000 7f7229e00700 1 -- 172.21.15.43:0/4003411058 wait complete. 2024-03-28T22:34:33.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:33 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:34:33.407 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:33 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:34:33.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:33 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:34:33.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:33 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:34:33.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.798+0000 7fbfc4600700 1 -- 172.21.15.43:0/3852214199 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbfbc104df0 msgr2=0x7fbfbc111300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:33.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.798+0000 7fbfc4600700 1 --2- 172.21.15.43:0/3852214199 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbfbc104df0 0x7fbfbc111300 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbfb400b0a0 tx=0x7fbfb400b3b0 comp rx=0 tx=0).stop 2024-03-28T22:34:33.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.798+0000 7fbfc4600700 1 -- 172.21.15.43:0/3852214199 shutdown_connections 2024-03-28T22:34:33.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.798+0000 7fbfc4600700 1 --2- 172.21.15.43:0/3852214199 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbfbc104df0 0x7fbfbc111300 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:33.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.798+0000 7fbfc4600700 1 --2- 172.21.15.43:0/3852214199 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfbc10a4c0 0x7fbfbc10a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:33.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.798+0000 7fbfc4600700 1 --2- 172.21.15.43:0/3852214199 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbfbc104430 0x7fbfbc1048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:33.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.798+0000 7fbfc4600700 1 -- 172.21.15.43:0/3852214199 >> 172.21.15.43:0/3852214199 conn(0x7fbfbc0fff20 msgr2=0x7fbfbc102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:33.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.798+0000 7fbfc4600700 1 -- 172.21.15.43:0/3852214199 shutdown_connections 2024-03-28T22:34:33.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.798+0000 7fbfc4600700 1 -- 172.21.15.43:0/3852214199 wait complete. 2024-03-28T22:34:33.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.799+0000 7fbfc4600700 1 Processor -- start 2024-03-28T22:34:33.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.799+0000 7fbfc4600700 1 -- start start 2024-03-28T22:34:33.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.800+0000 7fbfc4600700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbfbc104430 0x7fbfbc1a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:33.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.800+0000 7fbfc4600700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbfbc104df0 0x7fbfbc1a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:33.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.800+0000 7fbfc4600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfbc10a4c0 0x7fbfbc19a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:33.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.800+0000 7fbfc4600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbfbc1147a0 con 0x7fbfbc10a4c0 2024-03-28T22:34:33.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.800+0000 7fbfc4600700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fbfbc114620 con 0x7fbfbc104df0 2024-03-28T22:34:33.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.800+0000 7fbfc4600700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fbfbc114920 con 0x7fbfbc104430 2024-03-28T22:34:33.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.800+0000 7fbfc1c00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbfbc104430 0x7fbfbc1a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:33.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.800+0000 7fbfc1200700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbfbc104df0 0x7fbfbc1a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:33.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.800+0000 7fbfc1c00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbfbc104430 0x7fbfbc1a07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:51222/0 (socket says 172.21.15.43:51222) 2024-03-28T22:34:33.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.800+0000 7fbfc1c00700 1 -- 172.21.15.43:0/3624991699 learned_addr learned my addr 172.21.15.43:0/3624991699 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:34:33.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.800+0000 7fbfc2600700 1 --2- 172.21.15.43:0/3624991699 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfbc10a4c0 0x7fbfbc19a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:33.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.801+0000 7fbfc1c00700 1 -- 172.21.15.43:0/3624991699 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbfbc104df0 msgr2=0x7fbfbc1a0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:33.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.801+0000 7fbfc1c00700 1 --2- 172.21.15.43:0/3624991699 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbfbc104df0 0x7fbfbc1a0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:33.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.801+0000 7fbfc1c00700 1 -- 172.21.15.43:0/3624991699 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfbc10a4c0 msgr2=0x7fbfbc19a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:33.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.801+0000 7fbfc1c00700 1 --2- 172.21.15.43:0/3624991699 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfbc10a4c0 0x7fbfbc19a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:33.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.801+0000 7fbfc1c00700 1 -- 172.21.15.43:0/3624991699 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfbc19b100 con 0x7fbfbc104430 2024-03-28T22:34:33.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.801+0000 7fbfc1200700 1 --2- 172.21.15.43:0/3624991699 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbfbc104df0 0x7fbfbc1a0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:34:33.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.801+0000 7fbfc2600700 1 --2- 172.21.15.43:0/3624991699 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbfbc10a4c0 0x7fbfbc19a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:34:33.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.801+0000 7fbfc1c00700 1 --2- 172.21.15.43:0/3624991699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbfbc104430 0x7fbfbc1a07e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbfb000e9a0 tx=0x7fbfb000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:33.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.801+0000 7fbfba000700 1 -- 172.21.15.43:0/3624991699 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb000cbd0 con 0x7fbfbc104430 2024-03-28T22:34:33.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.801+0000 7fbfba000700 1 -- 172.21.15.43:0/3624991699 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbfb000cd70 con 0x7fbfbc104430 2024-03-28T22:34:33.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.801+0000 7fbfba000700 1 -- 172.21.15.43:0/3624991699 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb0018750 con 0x7fbfbc104430 2024-03-28T22:34:33.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.801+0000 7fbfc4600700 1 -- 172.21.15.43:0/3624991699 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfbc19b3f0 con 0x7fbfbc104430 2024-03-28T22:34:33.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.801+0000 7fbfc4600700 1 -- 172.21.15.43:0/3624991699 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfbc1a76b0 con 0x7fbfbc104430 2024-03-28T22:34:33.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.802+0000 7fbfc4600700 1 -- 172.21.15.43:0/3624991699 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbfbc100e20 con 0x7fbfbc104430 2024-03-28T22:34:33.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.805+0000 7fbfba000700 1 -- 172.21.15.43:0/3624991699 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fbfb0010430 con 0x7fbfbc104430 2024-03-28T22:34:33.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.806+0000 7fbfba000700 1 --2- 172.21.15.43:0/3624991699 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fbfa40a1ea0 0x7fbfa40a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:33.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.806+0000 7fbfba000700 1 -- 172.21.15.43:0/3624991699 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7fbfb00ced30 con 0x7fbfbc104430 2024-03-28T22:34:33.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.806+0000 7fbfc1200700 1 --2- 172.21.15.43:0/3624991699 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fbfa40a1ea0 0x7fbfa40a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:33.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.807+0000 7fbfba000700 1 --2- 172.21.15.43:0/3624991699 >> [v2:172.21.15.43:6818/791044279,v1:172.21.15.43:6819/791044279] conn(0x7fbfa40ad000 0x7fbfa40af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:33.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.807+0000 7fbfba000700 1 -- 172.21.15.43:0/3624991699 --> [v2:172.21.15.43:6818/791044279,v1:172.21.15.43:6819/791044279] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbfb000ae50 con 0x7fbfa40ad000 2024-03-28T22:34:33.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.807+0000 7fbfba000700 1 -- 172.21.15.43:0/3624991699 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_get_version_reply(handle=1 version=94) ==== 24+0+0 (secure 0 0 0) 0x7fbfb00cf0e0 con 0x7fbfbc104430 2024-03-28T22:34:33.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.807+0000 7fbfc2600700 1 --2- 172.21.15.43:0/3624991699 >> [v2:172.21.15.43:6818/791044279,v1:172.21.15.43:6819/791044279] conn(0x7fbfa40ad000 0x7fbfa40af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:33.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.807+0000 7fbfc1200700 1 --2- 172.21.15.43:0/3624991699 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fbfa40a1ea0 0x7fbfa40a4360 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc109930 tx=0x7fbfa8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:33.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.808+0000 7fbfc2600700 1 --2- 172.21.15.43:0/3624991699 >> [v2:172.21.15.43:6818/791044279,v1:172.21.15.43:6819/791044279] conn(0x7fbfa40ad000 0x7fbfa40af4a0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:33.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.809+0000 7fbfba000700 1 -- 172.21.15.43:0/3624991699 <== osd.6 v2:172.21.15.43:6818/791044279 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbfb000ae50 con 0x7fbfa40ad000 2024-03-28T22:34:33.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.827+0000 7fbfc4600700 1 -- 172.21.15.43:0/3624991699 --> [v2:172.21.15.43:6818/791044279,v1:172.21.15.43:6819/791044279] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbfbc105b70 con 0x7fbfa40ad000 2024-03-28T22:34:33.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.828+0000 7fbfba000700 1 -- 172.21.15.43:0/3624991699 <== osd.6 v2:172.21.15.43:6818/791044279 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fbfbc105b70 con 0x7fbfa40ad000 2024-03-28T22:34:33.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.828+0000 7fbfc4600700 1 -- 172.21.15.43:0/3624991699 >> [v2:172.21.15.43:6818/791044279,v1:172.21.15.43:6819/791044279] conn(0x7fbfa40ad000 msgr2=0x7fbfa40af4a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:33.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.828+0000 7fbfc4600700 1 --2- 172.21.15.43:0/3624991699 >> [v2:172.21.15.43:6818/791044279,v1:172.21.15.43:6819/791044279] conn(0x7fbfa40ad000 0x7fbfa40af4a0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:33.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.828+0000 7fbfc4600700 1 -- 172.21.15.43:0/3624991699 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fbfa40a1ea0 msgr2=0x7fbfa40a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:33.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.828+0000 7fbfc4600700 1 --2- 172.21.15.43:0/3624991699 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fbfa40a1ea0 0x7fbfa40a4360 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc109930 tx=0x7fbfa8009260 comp rx=0 tx=0).stop 2024-03-28T22:34:33.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.828+0000 7fbfc4600700 1 -- 172.21.15.43:0/3624991699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbfbc104430 msgr2=0x7fbfbc1a07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:33.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.828+0000 7fbfc4600700 1 --2- 172.21.15.43:0/3624991699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbfbc104430 0x7fbfbc1a07e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbfb000e9a0 tx=0x7fbfb000ecb0 comp rx=0 tx=0).stop 2024-03-28T22:34:33.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.828+0000 7fbfc2600700 1 -- 172.21.15.43:0/3624991699 reap_dead start 2024-03-28T22:34:33.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.828+0000 7fbfc4600700 1 -- 172.21.15.43:0/3624991699 shutdown_connections 2024-03-28T22:34:33.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.829+0000 7fbfc4600700 1 -- 172.21.15.43:0/3624991699 >> 172.21.15.43:0/3624991699 conn(0x7fbfbc0fff20 msgr2=0x7fbfbc100400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:33.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.829+0000 7fbfc4600700 1 -- 172.21.15.43:0/3624991699 shutdown_connections 2024-03-28T22:34:33.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:33.829+0000 7fbfc4600700 1 -- 172.21.15.43:0/3624991699 wait complete. 2024-03-28T22:34:34.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:34 smithi043 ceph-mon[31823]: pgmap v356: 129 pgs: 1 peering, 128 active+clean; 645 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 639 B/s wr, 10 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T22:34:34.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:34 smithi172 ceph-mon[36566]: pgmap v356: 129 pgs: 1 peering, 128 active+clean; 645 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 639 B/s wr, 10 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T22:34:34.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:34 smithi082 ceph-mon[36471]: pgmap v356: 129 pgs: 1 peering, 128 active+clean; 645 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 639 B/s wr, 10 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T22:34:34.489 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:34:34.490 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:34:34.578 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:34:34.916 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:34:36.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:36 smithi043 ceph-mon[31823]: pgmap v357: 129 pgs: 1 peering, 128 active+clean; 645 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T22:34:36.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:36 smithi172 ceph-mon[36566]: pgmap v357: 129 pgs: 1 peering, 128 active+clean; 645 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T22:34:36.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:36 smithi082 ceph-mon[36471]: pgmap v357: 129 pgs: 1 peering, 128 active+clean; 645 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T22:34:38.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:38 smithi043 ceph-mon[31823]: pgmap v358: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 10 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T22:34:38.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:38 smithi172 ceph-mon[36566]: pgmap v358: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 10 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T22:34:38.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:38 smithi082 ceph-mon[36471]: pgmap v358: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 10 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T22:34:38.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.574+0000 7feefcc00700 1 -- 172.21.15.43:0/1383464350 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feef8074d90 msgr2=0x7feef810e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:38.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.574+0000 7feefcc00700 1 --2- 172.21.15.43:0/1383464350 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feef8074d90 0x7feef810e730 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7feee40098f0 tx=0x7feee4009c00 comp rx=0 tx=0).stop 2024-03-28T22:34:38.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.574+0000 7feefcc00700 1 -- 172.21.15.43:0/1383464350 shutdown_connections 2024-03-28T22:34:38.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.574+0000 7feefcc00700 1 --2- 172.21.15.43:0/1383464350 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feef810ec70 0x7feef8111060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:38.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.574+0000 7feefcc00700 1 --2- 172.21.15.43:0/1383464350 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feef8074470 0x7feef8074850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:38.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.574+0000 7feefcc00700 1 --2- 172.21.15.43:0/1383464350 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feef8074d90 0x7feef810e730 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:38.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.574+0000 7feefcc00700 1 -- 172.21.15.43:0/1383464350 >> 172.21.15.43:0/1383464350 conn(0x7feef80792d0 msgr2=0x7feef80796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:38.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.574+0000 7feefcc00700 1 -- 172.21.15.43:0/1383464350 shutdown_connections 2024-03-28T22:34:38.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.574+0000 7feefcc00700 1 -- 172.21.15.43:0/1383464350 wait complete. 2024-03-28T22:34:38.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.575+0000 7feefcc00700 1 Processor -- start 2024-03-28T22:34:38.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.576+0000 7feefcc00700 1 -- start start 2024-03-28T22:34:38.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.576+0000 7feefcc00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feef8074470 0x7feef81a0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:38.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.576+0000 7feefcc00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feef8074d90 0x7feef81a0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:38.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.576+0000 7feefcc00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feef810ec70 0x7feef819ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:38.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.576+0000 7feefcc00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7feef8114750 con 0x7feef8074d90 2024-03-28T22:34:38.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.576+0000 7feefcc00700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7feef81145d0 con 0x7feef810ec70 2024-03-28T22:34:38.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.576+0000 7feefcc00700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7feef81148d0 con 0x7feef8074470 2024-03-28T22:34:38.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.576+0000 7feef6800700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feef810ec70 0x7feef819ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:38.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.576+0000 7feef5e00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feef8074470 0x7feef81a0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:38.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.576+0000 7feef6800700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feef810ec70 0x7feef819ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:40370/0 (socket says 172.21.15.43:40370) 2024-03-28T22:34:38.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.576+0000 7feef6800700 1 -- 172.21.15.43:0/796800181 learned_addr learned my addr 172.21.15.43:0/796800181 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:34:38.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.577+0000 7feef6800700 1 -- 172.21.15.43:0/796800181 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feef8074470 msgr2=0x7feef81a0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:38.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.577+0000 7feef6800700 1 --2- 172.21.15.43:0/796800181 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feef8074470 0x7feef81a0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:38.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.577+0000 7feef6800700 1 -- 172.21.15.43:0/796800181 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feef8074d90 msgr2=0x7feef81a0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:38.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.577+0000 7feef5400700 1 --2- 172.21.15.43:0/796800181 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feef8074d90 0x7feef81a0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:38.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.577+0000 7feef6800700 1 --2- 172.21.15.43:0/796800181 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feef8074d90 0x7feef81a0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:38.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.577+0000 7feef6800700 1 -- 172.21.15.43:0/796800181 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feef819b270 con 0x7feef810ec70 2024-03-28T22:34:38.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.577+0000 7feef5e00700 1 --2- 172.21.15.43:0/796800181 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feef8074470 0x7feef81a0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:34:38.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.577+0000 7feef6800700 1 --2- 172.21.15.43:0/796800181 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feef810ec70 0x7feef819ab10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7feef0007ee0 tx=0x7feef000d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:38.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.578+0000 7feeeea00700 1 -- 172.21.15.43:0/796800181 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef000ddd0 con 0x7feef810ec70 2024-03-28T22:34:38.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.578+0000 7feeeea00700 1 -- 172.21.15.43:0/796800181 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feef0004020 con 0x7feef810ec70 2024-03-28T22:34:38.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.578+0000 7feefcc00700 1 -- 172.21.15.43:0/796800181 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feef819b560 con 0x7feef810ec70 2024-03-28T22:34:38.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.578+0000 7feeeea00700 1 -- 172.21.15.43:0/796800181 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef0015660 con 0x7feef810ec70 2024-03-28T22:34:38.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.578+0000 7feefcc00700 1 -- 172.21.15.43:0/796800181 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feef81a77d0 con 0x7feef810ec70 2024-03-28T22:34:38.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.579+0000 7feefcc00700 1 -- 172.21.15.43:0/796800181 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7feef80629d0 con 0x7feef810ec70 2024-03-28T22:34:38.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.581+0000 7feeeea00700 1 -- 172.21.15.43:0/796800181 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7feef00041d0 con 0x7feef810ec70 2024-03-28T22:34:38.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.582+0000 7feeeea00700 1 --2- 172.21.15.43:0/796800181 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7feedc0a1f20 0x7feedc0a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:38.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.582+0000 7feeeea00700 1 -- 172.21.15.43:0/796800181 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7feef00ce7c0 con 0x7feef810ec70 2024-03-28T22:34:38.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.583+0000 7feef5e00700 1 --2- 172.21.15.43:0/796800181 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7feedc0a1f20 0x7feedc0a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:38.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.583+0000 7feeeea00700 1 --2- 172.21.15.43:0/796800181 >> [v2:172.21.15.43:6802/2826124409,v1:172.21.15.43:6803/2826124409] conn(0x7feedc0ad080 0x7feedc0af520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:38.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.583+0000 7feeeea00700 1 -- 172.21.15.43:0/796800181 --> [v2:172.21.15.43:6802/2826124409,v1:172.21.15.43:6803/2826124409] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feedc0afbf0 con 0x7feedc0ad080 2024-03-28T22:34:38.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.583+0000 7feeeea00700 1 -- 172.21.15.43:0/796800181 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_get_version_reply(handle=1 version=94) ==== 24+0+0 (secure 0 0 0) 0x7feef00ceb70 con 0x7feef810ec70 2024-03-28T22:34:38.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.583+0000 7feef5400700 1 --2- 172.21.15.43:0/796800181 >> [v2:172.21.15.43:6802/2826124409,v1:172.21.15.43:6803/2826124409] conn(0x7feedc0ad080 0x7feedc0af520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:38.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.584+0000 7feef5e00700 1 --2- 172.21.15.43:0/796800181 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7feedc0a1f20 0x7feedc0a43e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7feee8009750 tx=0x7feee8009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:38.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.584+0000 7feef5400700 1 --2- 172.21.15.43:0/796800181 >> [v2:172.21.15.43:6802/2826124409,v1:172.21.15.43:6803/2826124409] conn(0x7feedc0ad080 0x7feedc0af520 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:38.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.585+0000 7feeeea00700 1 -- 172.21.15.43:0/796800181 <== osd.0 v2:172.21.15.43:6802/2826124409 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feedc0afbf0 con 0x7feedc0ad080 2024-03-28T22:34:38.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.604+0000 7feefcc00700 1 -- 172.21.15.43:0/796800181 --> [v2:172.21.15.43:6802/2826124409,v1:172.21.15.43:6803/2826124409] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7feef8062dd0 con 0x7feedc0ad080 2024-03-28T22:34:38.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.604+0000 7feeeea00700 1 -- 172.21.15.43:0/796800181 <== osd.0 v2:172.21.15.43:6802/2826124409 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7feef8062dd0 con 0x7feedc0ad080 2024-03-28T22:34:38.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.605+0000 7feefcc00700 1 -- 172.21.15.43:0/796800181 >> [v2:172.21.15.43:6802/2826124409,v1:172.21.15.43:6803/2826124409] conn(0x7feedc0ad080 msgr2=0x7feedc0af520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:38.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.605+0000 7feefcc00700 1 --2- 172.21.15.43:0/796800181 >> [v2:172.21.15.43:6802/2826124409,v1:172.21.15.43:6803/2826124409] conn(0x7feedc0ad080 0x7feedc0af520 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:38.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.605+0000 7feefcc00700 1 -- 172.21.15.43:0/796800181 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7feedc0a1f20 msgr2=0x7feedc0a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:38.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.605+0000 7feefcc00700 1 --2- 172.21.15.43:0/796800181 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7feedc0a1f20 0x7feedc0a43e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7feee8009750 tx=0x7feee8009310 comp rx=0 tx=0).stop 2024-03-28T22:34:38.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.605+0000 7feefcc00700 1 -- 172.21.15.43:0/796800181 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feef810ec70 msgr2=0x7feef819ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:38.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.605+0000 7feefcc00700 1 --2- 172.21.15.43:0/796800181 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feef810ec70 0x7feef819ab10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7feef0007ee0 tx=0x7feef000d3b0 comp rx=0 tx=0).stop 2024-03-28T22:34:38.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.605+0000 7feef6800700 1 -- 172.21.15.43:0/796800181 reap_dead start 2024-03-28T22:34:38.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.605+0000 7feefcc00700 1 -- 172.21.15.43:0/796800181 shutdown_connections 2024-03-28T22:34:38.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.605+0000 7feefcc00700 1 -- 172.21.15.43:0/796800181 >> 172.21.15.43:0/796800181 conn(0x7feef80792d0 msgr2=0x7feef80ff520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:38.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.605+0000 7feefcc00700 1 -- 172.21.15.43:0/796800181 shutdown_connections 2024-03-28T22:34:38.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:38.605+0000 7feefcc00700 1 -- 172.21.15.43:0/796800181 wait complete. 2024-03-28T22:34:39.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.175+0000 7f7b96c00700 1 -- 172.21.15.43:0/2446994635 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7b901032e0 msgr2=0x7f7b901036c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:39.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.175+0000 7f7b96c00700 1 --2- 172.21.15.43:0/2446994635 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7b901032e0 0x7f7b901036c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7b84009990 tx=0x7f7b84009ca0 comp rx=0 tx=0).stop 2024-03-28T22:34:39.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.175+0000 7f7b96c00700 1 -- 172.21.15.43:0/2446994635 shutdown_connections 2024-03-28T22:34:39.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.175+0000 7f7b96c00700 1 --2- 172.21.15.43:0/2446994635 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b9010c710 0x7f7b9010ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:39.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.175+0000 7f7b96c00700 1 --2- 172.21.15.43:0/2446994635 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7b901032e0 0x7f7b901036c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:39.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.175+0000 7f7b96c00700 1 --2- 172.21.15.43:0/2446994635 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b90103c00 0x7f7b9010c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:39.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.175+0000 7f7b96c00700 1 -- 172.21.15.43:0/2446994635 >> 172.21.15.43:0/2446994635 conn(0x7f7b900fcb10 msgr2=0x7f7b900fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:39.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.175+0000 7f7b96c00700 1 -- 172.21.15.43:0/2446994635 shutdown_connections 2024-03-28T22:34:39.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.175+0000 7f7b96c00700 1 -- 172.21.15.43:0/2446994635 wait complete. 2024-03-28T22:34:39.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.176+0000 7f7b96c00700 1 Processor -- start 2024-03-28T22:34:39.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.177+0000 7f7b96c00700 1 -- start start 2024-03-28T22:34:39.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.177+0000 7f7b96c00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7b901032e0 0x7f7b9019e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:39.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.177+0000 7f7b96c00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b90103c00 0x7f7b9019eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:39.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.177+0000 7f7b96c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b9010c710 0x7f7b90198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:39.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.177+0000 7f7b96c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7b901124a0 con 0x7f7b9010c710 2024-03-28T22:34:39.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.177+0000 7f7b96c00700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7b90112320 con 0x7f7b901032e0 2024-03-28T22:34:39.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.177+0000 7f7b96c00700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f7b90112620 con 0x7f7b90103c00 2024-03-28T22:34:39.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.177+0000 7f7b94e00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b90103c00 0x7f7b9019eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:39.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.177+0000 7f7b95800700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7b901032e0 0x7f7b9019e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:39.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.177+0000 7f7b94e00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b90103c00 0x7f7b9019eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:55192/0 (socket says 172.21.15.43:55192) 2024-03-28T22:34:39.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.177+0000 7f7b94e00700 1 -- 172.21.15.43:0/643075750 learned_addr learned my addr 172.21.15.43:0/643075750 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:34:39.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.177+0000 7f7b96200700 1 --2- 172.21.15.43:0/643075750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b9010c710 0x7f7b90198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:39.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.178+0000 7f7b95800700 1 -- 172.21.15.43:0/643075750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b90103c00 msgr2=0x7f7b9019eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:39.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.178+0000 7f7b95800700 1 --2- 172.21.15.43:0/643075750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7b90103c00 0x7f7b9019eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:39.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.178+0000 7f7b95800700 1 -- 172.21.15.43:0/643075750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b9010c710 msgr2=0x7f7b90198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:39.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.178+0000 7f7b95800700 1 --2- 172.21.15.43:0/643075750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7b9010c710 0x7f7b90198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:39.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.178+0000 7f7b95800700 1 -- 172.21.15.43:0/643075750 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b90198f60 con 0x7f7b901032e0 2024-03-28T22:34:39.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.178+0000 7f7b95800700 1 --2- 172.21.15.43:0/643075750 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7b901032e0 0x7f7b9019e600 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7b84000c00 tx=0x7f7b8400bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:39.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.178+0000 7f7b8e000700 1 -- 172.21.15.43:0/643075750 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b84026070 con 0x7f7b901032e0 2024-03-28T22:34:39.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.178+0000 7f7b8e000700 1 -- 172.21.15.43:0/643075750 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b84003d70 con 0x7f7b901032e0 2024-03-28T22:34:39.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.178+0000 7f7b8e000700 1 -- 172.21.15.43:0/643075750 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b84021410 con 0x7f7b901032e0 2024-03-28T22:34:39.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.178+0000 7f7b96c00700 1 -- 172.21.15.43:0/643075750 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b901991f0 con 0x7f7b901032e0 2024-03-28T22:34:39.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.179+0000 7f7b96c00700 1 -- 172.21.15.43:0/643075750 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b901a5460 con 0x7f7b901032e0 2024-03-28T22:34:39.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.179+0000 7f7b96c00700 1 -- 172.21.15.43:0/643075750 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7b900629d0 con 0x7f7b901032e0 2024-03-28T22:34:39.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.181+0000 7f7b8e000700 1 -- 172.21.15.43:0/643075750 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f7b84029070 con 0x7f7b901032e0 2024-03-28T22:34:39.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.183+0000 7f7b8e000700 1 --2- 172.21.15.43:0/643075750 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7b780a1f20 0x7f7b780a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:39.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.183+0000 7f7b8e000700 1 -- 172.21.15.43:0/643075750 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7f7b840cf170 con 0x7f7b901032e0 2024-03-28T22:34:39.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.183+0000 7f7b94e00700 1 --2- 172.21.15.43:0/643075750 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7b780a1f20 0x7f7b780a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:39.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.183+0000 7f7b8e000700 1 --2- 172.21.15.43:0/643075750 >> [v2:172.21.15.82:6824/3454954047,v1:172.21.15.82:6825/3454954047] conn(0x7f7b780ad080 0x7f7b780af520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:39.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.183+0000 7f7b8e000700 1 -- 172.21.15.43:0/643075750 --> [v2:172.21.15.82:6824/3454954047,v1:172.21.15.82:6825/3454954047] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7b84010ea0 con 0x7f7b780ad080 2024-03-28T22:34:39.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.183+0000 7f7b8e000700 1 -- 172.21.15.43:0/643075750 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_get_version_reply(handle=1 version=94) ==== 24+0+0 (secure 0 0 0) 0x7f7b840cf520 con 0x7f7b901032e0 2024-03-28T22:34:39.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.183+0000 7f7b96200700 1 --2- 172.21.15.43:0/643075750 >> [v2:172.21.15.82:6824/3454954047,v1:172.21.15.82:6825/3454954047] conn(0x7f7b780ad080 0x7f7b780af520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:39.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.184+0000 7f7b94e00700 1 --2- 172.21.15.43:0/643075750 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7b780a1f20 0x7f7b780a43e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7b9010da90 tx=0x7f7b7c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:39.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.184+0000 7f7b96200700 1 --2- 172.21.15.43:0/643075750 >> [v2:172.21.15.82:6824/3454954047,v1:172.21.15.82:6825/3454954047] conn(0x7f7b780ad080 0x7f7b780af520 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:39.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.185+0000 7f7b8e000700 1 -- 172.21.15.43:0/643075750 <== osd.10 v2:172.21.15.82:6824/3454954047 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7b84010ea0 con 0x7f7b780ad080 2024-03-28T22:34:39.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.204+0000 7f7b96c00700 1 -- 172.21.15.43:0/643075750 --> [v2:172.21.15.82:6824/3454954047,v1:172.21.15.82:6825/3454954047] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7b90062ea0 con 0x7f7b780ad080 2024-03-28T22:34:39.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.204+0000 7f7b8e000700 1 -- 172.21.15.43:0/643075750 <== osd.10 v2:172.21.15.82:6824/3454954047 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7b90062ea0 con 0x7f7b780ad080 2024-03-28T22:34:39.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.205+0000 7f7b96c00700 1 -- 172.21.15.43:0/643075750 >> [v2:172.21.15.82:6824/3454954047,v1:172.21.15.82:6825/3454954047] conn(0x7f7b780ad080 msgr2=0x7f7b780af520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:39.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.205+0000 7f7b96c00700 1 --2- 172.21.15.43:0/643075750 >> [v2:172.21.15.82:6824/3454954047,v1:172.21.15.82:6825/3454954047] conn(0x7f7b780ad080 0x7f7b780af520 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:39.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.205+0000 7f7b96c00700 1 -- 172.21.15.43:0/643075750 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7b780a1f20 msgr2=0x7f7b780a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:39.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.205+0000 7f7b96c00700 1 --2- 172.21.15.43:0/643075750 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7b780a1f20 0x7f7b780a43e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7b9010da90 tx=0x7f7b7c009260 comp rx=0 tx=0).stop 2024-03-28T22:34:39.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.205+0000 7f7b96c00700 1 -- 172.21.15.43:0/643075750 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7b901032e0 msgr2=0x7f7b9019e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:39.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.205+0000 7f7b96c00700 1 --2- 172.21.15.43:0/643075750 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7b901032e0 0x7f7b9019e600 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7b84000c00 tx=0x7f7b8400bab0 comp rx=0 tx=0).stop 2024-03-28T22:34:39.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.205+0000 7f7b96200700 1 -- 172.21.15.43:0/643075750 reap_dead start 2024-03-28T22:34:39.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.205+0000 7f7b96c00700 1 -- 172.21.15.43:0/643075750 shutdown_connections 2024-03-28T22:34:39.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.205+0000 7f7b96c00700 1 -- 172.21.15.43:0/643075750 >> 172.21.15.43:0/643075750 conn(0x7f7b900fcb10 msgr2=0x7f7b900fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:39.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.206+0000 7f7b96c00700 1 -- 172.21.15.43:0/643075750 shutdown_connections 2024-03-28T22:34:39.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:39.206+0000 7f7b96c00700 1 -- 172.21.15.43:0/643075750 wait complete. 2024-03-28T22:34:40.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.416+0000 7fda0ea00700 1 -- 172.21.15.43:0/1954028291 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda08104430 msgr2=0x7fda081048b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:40.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.416+0000 7fda0ea00700 1 --2- 172.21.15.43:0/1954028291 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda08104430 0x7fda081048b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0009990 tx=0x7fd9f0009ca0 comp rx=0 tx=0).stop 2024-03-28T22:34:40.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.416+0000 7fda0ea00700 1 -- 172.21.15.43:0/1954028291 shutdown_connections 2024-03-28T22:34:40.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.416+0000 7fda0ea00700 1 --2- 172.21.15.43:0/1954028291 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda08104df0 0x7fda08111300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:40.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.416+0000 7fda0ea00700 1 --2- 172.21.15.43:0/1954028291 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda0810a4c0 0x7fda0810a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:40.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.416+0000 7fda0ea00700 1 --2- 172.21.15.43:0/1954028291 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda08104430 0x7fda081048b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:40.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.416+0000 7fda0ea00700 1 -- 172.21.15.43:0/1954028291 >> 172.21.15.43:0/1954028291 conn(0x7fda080fff20 msgr2=0x7fda08102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:40.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.416+0000 7fda0ea00700 1 -- 172.21.15.43:0/1954028291 shutdown_connections 2024-03-28T22:34:40.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.416+0000 7fda0ea00700 1 -- 172.21.15.43:0/1954028291 wait complete. 2024-03-28T22:34:40.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.418+0000 7fda0ea00700 1 Processor -- start 2024-03-28T22:34:40.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.418+0000 7fda0ea00700 1 -- start start 2024-03-28T22:34:40.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.418+0000 7fda0ea00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda08104430 0x7fda081a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:40.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.418+0000 7fda0ea00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda08104df0 0x7fda081a0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:40.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.418+0000 7fda0ea00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda0810a4c0 0x7fda0819aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:40.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.418+0000 7fda0ea00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fda08114790 con 0x7fda08104430 2024-03-28T22:34:40.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.418+0000 7fda0ea00700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fda08114610 con 0x7fda0810a4c0 2024-03-28T22:34:40.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.418+0000 7fda0ea00700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fda08114910 con 0x7fda08104df0 2024-03-28T22:34:40.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.419+0000 7fda07e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda08104430 0x7fda081a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:40.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.419+0000 7fd9ff400700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda08104df0 0x7fda081a0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:40.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.419+0000 7fda0ca00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda0810a4c0 0x7fda0819aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:40.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.419+0000 7fda07e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda08104430 0x7fda081a0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50324/0 (socket says 172.21.15.43:50324) 2024-03-28T22:34:40.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.419+0000 7fda07e00700 1 -- 172.21.15.43:0/2348530888 learned_addr learned my addr 172.21.15.43:0/2348530888 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:34:40.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.419+0000 7fda07e00700 1 -- 172.21.15.43:0/2348530888 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda08104df0 msgr2=0x7fda081a0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:40.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.419+0000 7fda07e00700 1 --2- 172.21.15.43:0/2348530888 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda08104df0 0x7fda081a0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:40.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.419+0000 7fda07e00700 1 -- 172.21.15.43:0/2348530888 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda0810a4c0 msgr2=0x7fda0819aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:40.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.419+0000 7fda07e00700 1 --2- 172.21.15.43:0/2348530888 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda0810a4c0 0x7fda0819aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:40.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.419+0000 7fda07e00700 1 -- 172.21.15.43:0/2348530888 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda0819b190 con 0x7fda08104430 2024-03-28T22:34:40.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.419+0000 7fda0ca00700 1 --2- 172.21.15.43:0/2348530888 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda0810a4c0 0x7fda0819aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:34:40.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.420+0000 7fda07e00700 1 --2- 172.21.15.43:0/2348530888 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda08104430 0x7fda081a0870 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd9f800e9c0 tx=0x7fd9f800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:40.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.420+0000 7fda05600700 1 -- 172.21.15.43:0/2348530888 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f800cbd0 con 0x7fda08104430 2024-03-28T22:34:40.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.420+0000 7fda05600700 1 -- 172.21.15.43:0/2348530888 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9f800ee50 con 0x7fda08104430 2024-03-28T22:34:40.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.420+0000 7fda05600700 1 -- 172.21.15.43:0/2348530888 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f8005680 con 0x7fda08104430 2024-03-28T22:34:40.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.420+0000 7fda0ea00700 1 -- 172.21.15.43:0/2348530888 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda0819b480 con 0x7fda08104430 2024-03-28T22:34:40.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.420+0000 7fda0ea00700 1 -- 172.21.15.43:0/2348530888 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda081a7740 con 0x7fda08104430 2024-03-28T22:34:40.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.421+0000 7fda0ea00700 1 -- 172.21.15.43:0/2348530888 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fda08062990 con 0x7fda08104430 2024-03-28T22:34:40.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.423+0000 7fda05600700 1 -- 172.21.15.43:0/2348530888 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fd9f8004750 con 0x7fda08104430 2024-03-28T22:34:40.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.425+0000 7fda05600700 1 --2- 172.21.15.43:0/2348530888 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd9e80a1ea0 0x7fd9e80a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:40.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.425+0000 7fda05600700 1 -- 172.21.15.43:0/2348530888 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7fd9f80ce8f0 con 0x7fda08104430 2024-03-28T22:34:40.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.425+0000 7fd9ff400700 1 --2- 172.21.15.43:0/2348530888 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd9e80a1ea0 0x7fd9e80a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:40.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.425+0000 7fda05600700 1 --2- 172.21.15.43:0/2348530888 >> [v2:172.21.15.172:6816/1095328248,v1:172.21.15.172:6817/1095328248] conn(0x7fd9e80ad000 0x7fd9e80af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:40.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.425+0000 7fda05600700 1 -- 172.21.15.43:0/2348530888 --> [v2:172.21.15.172:6816/1095328248,v1:172.21.15.172:6817/1095328248] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd9e80afb70 con 0x7fd9e80ad000 2024-03-28T22:34:40.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.425+0000 7fda05600700 1 -- 172.21.15.43:0/2348530888 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_get_version_reply(handle=1 version=94) ==== 24+0+0 (secure 0 0 0) 0x7fd9f80ceca0 con 0x7fda08104430 2024-03-28T22:34:40.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.426+0000 7fda0ca00700 1 --2- 172.21.15.43:0/2348530888 >> [v2:172.21.15.172:6816/1095328248,v1:172.21.15.172:6817/1095328248] conn(0x7fd9e80ad000 0x7fd9e80af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:40.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.426+0000 7fd9ff400700 1 --2- 172.21.15.43:0/2348530888 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd9e80a1ea0 0x7fd9e80a4360 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fda0810ac50 tx=0x7fd9f000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:40.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.426+0000 7fda0ca00700 1 --2- 172.21.15.43:0/2348530888 >> [v2:172.21.15.172:6816/1095328248,v1:172.21.15.172:6817/1095328248] conn(0x7fd9e80ad000 0x7fd9e80af4a0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:40.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.427+0000 7fda05600700 1 -- 172.21.15.43:0/2348530888 <== osd.8 v2:172.21.15.172:6816/1095328248 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd9e80afb70 con 0x7fd9e80ad000 2024-03-28T22:34:40.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.445+0000 7fda0ea00700 1 -- 172.21.15.43:0/2348530888 --> [v2:172.21.15.172:6816/1095328248,v1:172.21.15.172:6817/1095328248] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fda08062e10 con 0x7fd9e80ad000 2024-03-28T22:34:40.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.446+0000 7fda05600700 1 -- 172.21.15.43:0/2348530888 <== osd.8 v2:172.21.15.172:6816/1095328248 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fda08062e10 con 0x7fd9e80ad000 2024-03-28T22:34:40.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.446+0000 7fda0ea00700 1 -- 172.21.15.43:0/2348530888 >> [v2:172.21.15.172:6816/1095328248,v1:172.21.15.172:6817/1095328248] conn(0x7fd9e80ad000 msgr2=0x7fd9e80af4a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:40.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.446+0000 7fda0ea00700 1 --2- 172.21.15.43:0/2348530888 >> [v2:172.21.15.172:6816/1095328248,v1:172.21.15.172:6817/1095328248] conn(0x7fd9e80ad000 0x7fd9e80af4a0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:40.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.447+0000 7fda0ea00700 1 -- 172.21.15.43:0/2348530888 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd9e80a1ea0 msgr2=0x7fd9e80a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:40.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.447+0000 7fda0ea00700 1 --2- 172.21.15.43:0/2348530888 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd9e80a1ea0 0x7fd9e80a4360 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fda0810ac50 tx=0x7fd9f000b260 comp rx=0 tx=0).stop 2024-03-28T22:34:40.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.447+0000 7fda0ea00700 1 -- 172.21.15.43:0/2348530888 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda08104430 msgr2=0x7fda081a0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:40.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.447+0000 7fda0ea00700 1 --2- 172.21.15.43:0/2348530888 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda08104430 0x7fda081a0870 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd9f800e9c0 tx=0x7fd9f800ecd0 comp rx=0 tx=0).stop 2024-03-28T22:34:40.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.447+0000 7fda0ca00700 1 -- 172.21.15.43:0/2348530888 reap_dead start 2024-03-28T22:34:40.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.447+0000 7fda0ea00700 1 -- 172.21.15.43:0/2348530888 shutdown_connections 2024-03-28T22:34:40.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.447+0000 7fda0ea00700 1 -- 172.21.15.43:0/2348530888 >> 172.21.15.43:0/2348530888 conn(0x7fda080fff20 msgr2=0x7fda08100440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:40.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.447+0000 7fda0ea00700 1 -- 172.21.15.43:0/2348530888 shutdown_connections 2024-03-28T22:34:40.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:37.447+0000 7fda0ea00700 1 -- 172.21.15.43:0/2348530888 wait complete. 2024-03-28T22:34:40.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:40 smithi172 ceph-mon[36566]: pgmap v359: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 222 B/s wr, 9 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T22:34:40.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:40 smithi082 ceph-mon[36471]: pgmap v359: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 222 B/s wr, 9 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T22:34:40.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:40 smithi043 ceph-mon[31823]: pgmap v359: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 222 B/s wr, 9 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T22:34:42.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:42 smithi043 ceph-mon[31823]: pgmap v360: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 200 B/s wr, 9 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T22:34:42.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:42 smithi172 ceph-mon[36566]: pgmap v360: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 200 B/s wr, 9 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T22:34:42.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:42 smithi082 ceph-mon[36471]: pgmap v360: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 200 B/s wr, 9 op/s; 2 B/s, 0 objects/s recovering 2024-03-28T22:34:42.815 INFO:teuthology.orchestra.run.smithi043.stdout:34359738456 2024-03-28T22:34:42.815 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-28T22:34:43.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:43 smithi082 ceph-mon[36471]: pgmap v361: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s; 1 B/s, 0 objects/s recovering 2024-03-28T22:34:43.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:43 smithi043 ceph-mon[31823]: pgmap v361: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s; 1 B/s, 0 objects/s recovering 2024-03-28T22:34:43.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:43 smithi172 ceph-mon[36566]: pgmap v361: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s; 1 B/s, 0 objects/s recovering 2024-03-28T22:34:44.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.986+0000 7fe18d200700 1 -- 172.21.15.43:0/1979735138 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe188074f90 msgr2=0x7fe18810e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:44.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.986+0000 7fe18d200700 1 --2- 172.21.15.43:0/1979735138 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe188074f90 0x7fe18810e420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe174009990 tx=0x7fe174009ca0 comp rx=0 tx=0).stop 2024-03-28T22:34:44.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.986+0000 7fe18d200700 1 -- 172.21.15.43:0/1979735138 shutdown_connections 2024-03-28T22:34:44.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.986+0000 7fe18d200700 1 --2- 172.21.15.43:0/1979735138 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe18810e960 0x7fe188110dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:44.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.986+0000 7fe18d200700 1 --2- 172.21.15.43:0/1979735138 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe188074670 0x7fe188074a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:44.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.986+0000 7fe18d200700 1 --2- 172.21.15.43:0/1979735138 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe188074f90 0x7fe18810e420 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:44.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.986+0000 7fe18d200700 1 -- 172.21.15.43:0/1979735138 >> 172.21.15.43:0/1979735138 conn(0x7fe1880fdc10 msgr2=0x7fe188100030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:44.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.986+0000 7fe18d200700 1 -- 172.21.15.43:0/1979735138 shutdown_connections 2024-03-28T22:34:44.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.987+0000 7fe18d200700 1 -- 172.21.15.43:0/1979735138 wait complete. 2024-03-28T22:34:44.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.988+0000 7fe18d200700 1 Processor -- start 2024-03-28T22:34:44.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.988+0000 7fe18d200700 1 -- start start 2024-03-28T22:34:44.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.989+0000 7fe18d200700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe188074670 0x7fe1881a07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:44.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.989+0000 7fe18d200700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe188074f90 0x7fe1881a0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:44.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.989+0000 7fe18d200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe18810e960 0x7fe18819a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:44.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.989+0000 7fe18d200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe1881146b0 con 0x7fe18810e960 2024-03-28T22:34:44.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.989+0000 7fe18d200700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe188114530 con 0x7fe188074670 2024-03-28T22:34:44.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.989+0000 7fe18d200700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe188114830 con 0x7fe188074f90 2024-03-28T22:34:44.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.989+0000 7fe187400700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe188074670 0x7fe1881a07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:44.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.989+0000 7fe186a00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe188074f90 0x7fe1881a0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:44.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.989+0000 7fe187400700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe188074670 0x7fe1881a07a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:40402/0 (socket says 172.21.15.43:40402) 2024-03-28T22:34:44.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.989+0000 7fe187400700 1 -- 172.21.15.43:0/460689201 learned_addr learned my addr 172.21.15.43:0/460689201 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:34:44.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.989+0000 7fe187e00700 1 --2- 172.21.15.43:0/460689201 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe18810e960 0x7fe18819a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:44.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.990+0000 7fe187400700 1 -- 172.21.15.43:0/460689201 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe188074f90 msgr2=0x7fe1881a0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:44.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.990+0000 7fe187400700 1 --2- 172.21.15.43:0/460689201 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe188074f90 0x7fe1881a0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:44.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.990+0000 7fe187400700 1 -- 172.21.15.43:0/460689201 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe18810e960 msgr2=0x7fe18819a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:44.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.990+0000 7fe187400700 1 --2- 172.21.15.43:0/460689201 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe18810e960 0x7fe18819a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:44.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.990+0000 7fe187400700 1 -- 172.21.15.43:0/460689201 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe18819b110 con 0x7fe188074670 2024-03-28T22:34:44.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.990+0000 7fe187e00700 1 --2- 172.21.15.43:0/460689201 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe18810e960 0x7fe18819a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:34:44.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.990+0000 7fe187400700 1 --2- 172.21.15.43:0/460689201 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe188074670 0x7fe1881a07a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe17800e9a0 tx=0x7fe17800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:44.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.990+0000 7fe17fe00700 1 -- 172.21.15.43:0/460689201 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17800cbd0 con 0x7fe188074670 2024-03-28T22:34:44.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.990+0000 7fe17fe00700 1 -- 172.21.15.43:0/460689201 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe17800cd70 con 0x7fe188074670 2024-03-28T22:34:44.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.990+0000 7fe17fe00700 1 -- 172.21.15.43:0/460689201 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe178018750 con 0x7fe188074670 2024-03-28T22:34:44.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.990+0000 7fe18d200700 1 -- 172.21.15.43:0/460689201 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe18819b400 con 0x7fe188074670 2024-03-28T22:34:44.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.990+0000 7fe18d200700 1 -- 172.21.15.43:0/460689201 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1881a7670 con 0x7fe188074670 2024-03-28T22:34:44.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.991+0000 7fe18d200700 1 -- 172.21.15.43:0/460689201 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe188062950 con 0x7fe188074670 2024-03-28T22:34:44.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.993+0000 7fe17fe00700 1 -- 172.21.15.43:0/460689201 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fe178010430 con 0x7fe188074670 2024-03-28T22:34:44.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.994+0000 7fe17fe00700 1 --2- 172.21.15.43:0/460689201 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe16c0a1f20 0x7fe16c0a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:44.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.995+0000 7fe17fe00700 1 -- 172.21.15.43:0/460689201 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7fe1780ced80 con 0x7fe188074670 2024-03-28T22:34:44.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.995+0000 7fe186a00700 1 --2- 172.21.15.43:0/460689201 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe16c0a1f20 0x7fe16c0a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:44.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.995+0000 7fe17fe00700 1 --2- 172.21.15.43:0/460689201 >> [v2:172.21.15.172:6800/4213953645,v1:172.21.15.172:6801/4213953645] conn(0x7fe16c0ad080 0x7fe16c0af520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:44.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.995+0000 7fe17fe00700 1 -- 172.21.15.43:0/460689201 --> [v2:172.21.15.172:6800/4213953645,v1:172.21.15.172:6801/4213953645] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe17800ae50 con 0x7fe16c0ad080 2024-03-28T22:34:44.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.995+0000 7fe17fe00700 1 -- 172.21.15.43:0/460689201 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_get_version_reply(handle=1 version=94) ==== 24+0+0 (secure 0 0 0) 0x7fe1780cf130 con 0x7fe188074670 2024-03-28T22:34:44.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.995+0000 7fe187e00700 1 --2- 172.21.15.43:0/460689201 >> [v2:172.21.15.172:6800/4213953645,v1:172.21.15.172:6801/4213953645] conn(0x7fe16c0ad080 0x7fe16c0af520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:44.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.996+0000 7fe186a00700 1 --2- 172.21.15.43:0/460689201 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe16c0a1f20 0x7fe16c0a43e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe18810fc60 tx=0x7fe174005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:44.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.996+0000 7fe187e00700 1 --2- 172.21.15.43:0/460689201 >> [v2:172.21.15.172:6800/4213953645,v1:172.21.15.172:6801/4213953645] conn(0x7fe16c0ad080 0x7fe16c0af520 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:44.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:43.997+0000 7fe17fe00700 1 -- 172.21.15.43:0/460689201 <== osd.2 v2:172.21.15.172:6800/4213953645 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe17800ae50 con 0x7fe16c0ad080 2024-03-28T22:34:44.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.015+0000 7fe18d200700 1 -- 172.21.15.43:0/460689201 --> [v2:172.21.15.172:6800/4213953645,v1:172.21.15.172:6801/4213953645] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe188062e20 con 0x7fe16c0ad080 2024-03-28T22:34:44.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.016+0000 7fe17fe00700 1 -- 172.21.15.43:0/460689201 <== osd.2 v2:172.21.15.172:6800/4213953645 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe188062e20 con 0x7fe16c0ad080 2024-03-28T22:34:44.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.016+0000 7fe18d200700 1 -- 172.21.15.43:0/460689201 >> [v2:172.21.15.172:6800/4213953645,v1:172.21.15.172:6801/4213953645] conn(0x7fe16c0ad080 msgr2=0x7fe16c0af520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:44.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.016+0000 7fe18d200700 1 --2- 172.21.15.43:0/460689201 >> [v2:172.21.15.172:6800/4213953645,v1:172.21.15.172:6801/4213953645] conn(0x7fe16c0ad080 0x7fe16c0af520 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:44.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.016+0000 7fe18d200700 1 -- 172.21.15.43:0/460689201 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe16c0a1f20 msgr2=0x7fe16c0a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:44.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.016+0000 7fe18d200700 1 --2- 172.21.15.43:0/460689201 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe16c0a1f20 0x7fe16c0a43e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe18810fc60 tx=0x7fe174005fd0 comp rx=0 tx=0).stop 2024-03-28T22:34:44.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.016+0000 7fe18d200700 1 -- 172.21.15.43:0/460689201 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe188074670 msgr2=0x7fe1881a07a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:44.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.016+0000 7fe18d200700 1 --2- 172.21.15.43:0/460689201 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe188074670 0x7fe1881a07a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe17800e9a0 tx=0x7fe17800ecb0 comp rx=0 tx=0).stop 2024-03-28T22:34:44.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.016+0000 7fe187e00700 1 -- 172.21.15.43:0/460689201 reap_dead start 2024-03-28T22:34:44.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.016+0000 7fe18d200700 1 -- 172.21.15.43:0/460689201 shutdown_connections 2024-03-28T22:34:44.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.016+0000 7fe18d200700 1 -- 172.21.15.43:0/460689201 >> 172.21.15.43:0/460689201 conn(0x7fe1880fdc10 msgr2=0x7fe1880fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:44.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.017+0000 7fe18d200700 1 -- 172.21.15.43:0/460689201 shutdown_connections 2024-03-28T22:34:44.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.017+0000 7fe18d200700 1 -- 172.21.15.43:0/460689201 wait complete. 2024-03-28T22:34:44.496 INFO:teuthology.orchestra.run.smithi043.stdout:197568495671 2024-03-28T22:34:44.496 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-28T22:34:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.084+0000 7f167c400700 1 -- 172.21.15.43:0/3946306921 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1674074470 msgr2=0x7f1674074850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.084+0000 7f167c400700 1 --2- 172.21.15.43:0/3946306921 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1674074470 0x7f1674074850 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1668009990 tx=0x7f1668009ca0 comp rx=0 tx=0).stop 2024-03-28T22:34:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.084+0000 7f167c400700 1 -- 172.21.15.43:0/3946306921 shutdown_connections 2024-03-28T22:34:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.084+0000 7f167c400700 1 --2- 172.21.15.43:0/3946306921 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1674113410 0x7f1674115800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.084+0000 7f167c400700 1 --2- 172.21.15.43:0/3946306921 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1674074470 0x7f1674074850 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.084+0000 7f167c400700 1 --2- 172.21.15.43:0/3946306921 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1674074d90 0x7f1674112ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.084+0000 7f167c400700 1 -- 172.21.15.43:0/3946306921 >> 172.21.15.43:0/3946306921 conn(0x7f16740793a0 msgr2=0x7f16740797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.085+0000 7f167c400700 1 -- 172.21.15.43:0/3946306921 shutdown_connections 2024-03-28T22:34:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.085+0000 7f167c400700 1 -- 172.21.15.43:0/3946306921 wait complete. 2024-03-28T22:34:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.086+0000 7f167c400700 1 Processor -- start 2024-03-28T22:34:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.086+0000 7f167c400700 1 -- start start 2024-03-28T22:34:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.086+0000 7f167c400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1674074470 0x7f167419f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.086+0000 7f167c400700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1674074d90 0x7f167419f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.086+0000 7f167c400700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1674113410 0x7f16741a3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.086+0000 7f167c400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1674118c50 con 0x7f1674074470 2024-03-28T22:34:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.086+0000 7f167c400700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1674118ad0 con 0x7f1674074d90 2024-03-28T22:34:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.086+0000 7f167c400700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f1674118dd0 con 0x7f1674113410 2024-03-28T22:34:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.086+0000 7f1679a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1674074470 0x7f167419f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.086+0000 7f1679000700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1674074d90 0x7f167419f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.086+0000 7f1679a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1674074470 0x7f167419f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50384/0 (socket says 172.21.15.43:50384) 2024-03-28T22:34:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.086+0000 7f1679a00700 1 -- 172.21.15.43:0/3221812280 learned_addr learned my addr 172.21.15.43:0/3221812280 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:34:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.086+0000 7f167a400700 1 --2- 172.21.15.43:0/3221812280 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1674113410 0x7f16741a3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.087+0000 7f1679000700 1 -- 172.21.15.43:0/3221812280 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1674113410 msgr2=0x7f16741a3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.087+0000 7f1679000700 1 --2- 172.21.15.43:0/3221812280 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1674113410 0x7f16741a3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.087+0000 7f1679000700 1 -- 172.21.15.43:0/3221812280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1674074470 msgr2=0x7f167419f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.087+0000 7f1679000700 1 --2- 172.21.15.43:0/3221812280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1674074470 0x7f167419f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.087+0000 7f1679000700 1 -- 172.21.15.43:0/3221812280 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16741a4070 con 0x7f1674074d90 2024-03-28T22:34:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.087+0000 7f1679a00700 1 --2- 172.21.15.43:0/3221812280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1674074470 0x7f167419f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:34:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.087+0000 7f167a400700 1 --2- 172.21.15.43:0/3221812280 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1674113410 0x7f16741a3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:34:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.087+0000 7f1679000700 1 --2- 172.21.15.43:0/3221812280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1674074d90 0x7f167419f5a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f166000e9c0 tx=0x7f166000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.087+0000 7f1672000700 1 -- 172.21.15.43:0/3221812280 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1660004220 con 0x7f1674074d90 2024-03-28T22:34:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.087+0000 7f1672000700 1 -- 172.21.15.43:0/3221812280 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16600043c0 con 0x7f1674074d90 2024-03-28T22:34:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.087+0000 7f167c400700 1 -- 172.21.15.43:0/3221812280 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16741a4360 con 0x7f1674074d90 2024-03-28T22:34:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.087+0000 7f1672000700 1 -- 172.21.15.43:0/3221812280 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1660003d10 con 0x7f1674074d90 2024-03-28T22:34:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.088+0000 7f167c400700 1 -- 172.21.15.43:0/3221812280 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16741abbe0 con 0x7f1674074d90 2024-03-28T22:34:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.089+0000 7f167c400700 1 -- 172.21.15.43:0/3221812280 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1674112bb0 con 0x7f1674074d90 2024-03-28T22:34:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.090+0000 7f1672000700 1 -- 172.21.15.43:0/3221812280 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f166000c7c0 con 0x7f1674074d90 2024-03-28T22:34:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.092+0000 7f1672000700 1 --2- 172.21.15.43:0/3221812280 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f165c0a1ea0 0x7f165c0a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.092+0000 7f1672000700 1 -- 172.21.15.43:0/3221812280 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7f16600ce500 con 0x7f1674074d90 2024-03-28T22:34:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.092+0000 7f1679a00700 1 --2- 172.21.15.43:0/3221812280 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f165c0a1ea0 0x7f165c0a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.092+0000 7f1672000700 1 --2- 172.21.15.43:0/3221812280 >> [v2:172.21.15.82:6800/3043263644,v1:172.21.15.82:6801/3043263644] conn(0x7f165c0ad000 0x7f165c0af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.092+0000 7f1672000700 1 -- 172.21.15.43:0/3221812280 --> [v2:172.21.15.82:6800/3043263644,v1:172.21.15.82:6801/3043263644] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f166001ee50 con 0x7f165c0ad000 2024-03-28T22:34:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.092+0000 7f1672000700 1 -- 172.21.15.43:0/3221812280 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_get_version_reply(handle=1 version=94) ==== 24+0+0 (secure 0 0 0) 0x7f16600d5050 con 0x7f1674074d90 2024-03-28T22:34:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.092+0000 7f167a400700 1 --2- 172.21.15.43:0/3221812280 >> [v2:172.21.15.82:6800/3043263644,v1:172.21.15.82:6801/3043263644] conn(0x7f165c0ad000 0x7f165c0af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.093+0000 7f1679a00700 1 --2- 172.21.15.43:0/3221812280 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f165c0a1ea0 0x7f165c0a4360 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1668009750 tx=0x7f166800b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.093+0000 7f167a400700 1 --2- 172.21.15.43:0/3221812280 >> [v2:172.21.15.82:6800/3043263644,v1:172.21.15.82:6801/3043263644] conn(0x7f165c0ad000 0x7f165c0af4a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.094+0000 7f1672000700 1 -- 172.21.15.43:0/3221812280 <== osd.1 v2:172.21.15.82:6800/3043263644 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f166001ee50 con 0x7f165c0ad000 2024-03-28T22:34:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.114+0000 7f167c400700 1 -- 172.21.15.43:0/3221812280 --> [v2:172.21.15.82:6800/3043263644,v1:172.21.15.82:6801/3043263644] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1674075ba0 con 0x7f165c0ad000 2024-03-28T22:34:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.114+0000 7f1672000700 1 -- 172.21.15.43:0/3221812280 <== osd.1 v2:172.21.15.82:6800/3043263644 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1674075ba0 con 0x7f165c0ad000 2024-03-28T22:34:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.115+0000 7f167c400700 1 -- 172.21.15.43:0/3221812280 >> [v2:172.21.15.82:6800/3043263644,v1:172.21.15.82:6801/3043263644] conn(0x7f165c0ad000 msgr2=0x7f165c0af4a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.115+0000 7f167c400700 1 --2- 172.21.15.43:0/3221812280 >> [v2:172.21.15.82:6800/3043263644,v1:172.21.15.82:6801/3043263644] conn(0x7f165c0ad000 0x7f165c0af4a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.115+0000 7f167c400700 1 -- 172.21.15.43:0/3221812280 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f165c0a1ea0 msgr2=0x7f165c0a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.115+0000 7f167c400700 1 --2- 172.21.15.43:0/3221812280 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f165c0a1ea0 0x7f165c0a4360 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1668009750 tx=0x7f166800b280 comp rx=0 tx=0).stop 2024-03-28T22:34:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.115+0000 7f167c400700 1 -- 172.21.15.43:0/3221812280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1674074d90 msgr2=0x7f167419f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.115+0000 7f167c400700 1 --2- 172.21.15.43:0/3221812280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1674074d90 0x7f167419f5a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f166000e9c0 tx=0x7f166000ecd0 comp rx=0 tx=0).stop 2024-03-28T22:34:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.115+0000 7f167a400700 1 -- 172.21.15.43:0/3221812280 reap_dead start 2024-03-28T22:34:44.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.115+0000 7f167c400700 1 -- 172.21.15.43:0/3221812280 shutdown_connections 2024-03-28T22:34:44.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.115+0000 7f167c400700 1 -- 172.21.15.43:0/3221812280 >> 172.21.15.43:0/3221812280 conn(0x7f16740793a0 msgr2=0x7f16741121d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:44.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.115+0000 7f167c400700 1 -- 172.21.15.43:0/3221812280 shutdown_connections 2024-03-28T22:34:44.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:44.115+0000 7f167c400700 1 -- 172.21.15.43:0/3221812280 wait complete. 2024-03-28T22:34:45.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:45 smithi082 ceph-mon[36471]: pgmap v362: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-28T22:34:45.453 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:45 smithi043 ceph-mon[31823]: pgmap v362: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-28T22:34:45.512 INFO:teuthology.orchestra.run.smithi043.stdout:317827579936 2024-03-28T22:34:45.512 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-28T22:34:45.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:45 smithi172 ceph-mon[36566]: pgmap v362: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-28T22:34:47.366 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:47 smithi043 ceph-mon[31823]: pgmap v363: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-28T22:34:47.436 INFO:teuthology.orchestra.run.smithi043.stdout:261993005100 2024-03-28T22:34:47.436 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-28T22:34:47.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:47 smithi082 ceph-mon[36471]: pgmap v363: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-28T22:34:47.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:47 smithi172 ceph-mon[36566]: pgmap v363: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-28T22:34:48.148 INFO:teuthology.orchestra.run.smithi043.stdout:223338299441 2024-03-28T22:34:48.148 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-28T22:34:48.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.674+0000 7f7488a00700 1 -- 172.21.15.43:0/2537062612 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7484107130 msgr2=0x7f7484069ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:48.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.674+0000 7f7488a00700 1 --2- 172.21.15.43:0/2537062612 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7484107130 0x7f7484069ef0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f74740098f0 tx=0x7f7474009c00 comp rx=0 tx=0).stop 2024-03-28T22:34:48.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.674+0000 7f7488a00700 1 -- 172.21.15.43:0/2537062612 shutdown_connections 2024-03-28T22:34:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.674+0000 7f7488a00700 1 --2- 172.21.15.43:0/2537062612 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7484104c40 0x7f74841050d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.674+0000 7f7488a00700 1 --2- 172.21.15.43:0/2537062612 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7484107130 0x7f7484069ef0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.674+0000 7f7488a00700 1 --2- 172.21.15.43:0/2537062612 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f748406a430 0x7f748406a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.674+0000 7f7488a00700 1 -- 172.21.15.43:0/2537062612 >> 172.21.15.43:0/2537062612 conn(0x7f7484078070 msgr2=0x7f7484078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.674+0000 7f7488a00700 1 -- 172.21.15.43:0/2537062612 shutdown_connections 2024-03-28T22:34:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.674+0000 7f7488a00700 1 -- 172.21.15.43:0/2537062612 wait complete. 2024-03-28T22:34:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.675+0000 7f7488a00700 1 Processor -- start 2024-03-28T22:34:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.676+0000 7f7488a00700 1 -- start start 2024-03-28T22:34:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.676+0000 7f7488a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f748406a430 0x7f748419f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.676+0000 7f7488a00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7484104c40 0x7f748419f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.676+0000 7f7488a00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7484107130 0x7f74841a3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.676+0000 7f7488a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7484118b20 con 0x7f748406a430 2024-03-28T22:34:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.676+0000 7f7488a00700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f74841189a0 con 0x7f7484104c40 2024-03-28T22:34:48.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.676+0000 7f7488a00700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f7484118ca0 con 0x7f7484107130 2024-03-28T22:34:48.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.677+0000 7f7482800700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7484107130 0x7f74841a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:48.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.677+0000 7f7482800700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7484107130 0x7f74841a3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35194/0 (socket says 172.21.15.43:35194) 2024-03-28T22:34:48.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.677+0000 7f7482800700 1 -- 172.21.15.43:0/2030303211 learned_addr learned my addr 172.21.15.43:0/2030303211 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:34:48.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.677+0000 7f7481e00700 1 --2- 172.21.15.43:0/2030303211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f748406a430 0x7f748419f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:48.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.677+0000 7f7481400700 1 --2- 172.21.15.43:0/2030303211 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7484104c40 0x7f748419f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:48.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.677+0000 7f7482800700 1 -- 172.21.15.43:0/2030303211 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7484104c40 msgr2=0x7f748419f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:48.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.677+0000 7f7482800700 1 --2- 172.21.15.43:0/2030303211 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7484104c40 0x7f748419f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:48.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.677+0000 7f7482800700 1 -- 172.21.15.43:0/2030303211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f748406a430 msgr2=0x7f748419f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:48.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.677+0000 7f7482800700 1 --2- 172.21.15.43:0/2030303211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f748406a430 0x7f748419f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:48.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.677+0000 7f7482800700 1 -- 172.21.15.43:0/2030303211 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74841a4080 con 0x7f7484107130 2024-03-28T22:34:48.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.678+0000 7f7482800700 1 --2- 172.21.15.43:0/2030303211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7484107130 0x7f74841a3940 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f747c00e3c0 tx=0x7f747c00e6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:48.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.678+0000 7f747aa00700 1 -- 172.21.15.43:0/2030303211 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f747c0090d0 con 0x7f7484107130 2024-03-28T22:34:48.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.678+0000 7f7488a00700 1 -- 172.21.15.43:0/2030303211 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74841a4370 con 0x7f7484107130 2024-03-28T22:34:48.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.678+0000 7f747aa00700 1 -- 172.21.15.43:0/2030303211 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f747c009270 con 0x7f7484107130 2024-03-28T22:34:48.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.678+0000 7f747aa00700 1 -- 172.21.15.43:0/2030303211 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f747c004030 con 0x7f7484107130 2024-03-28T22:34:48.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.678+0000 7f7488a00700 1 -- 172.21.15.43:0/2030303211 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74841abbf0 con 0x7f7484107130 2024-03-28T22:34:48.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.680+0000 7f7488a00700 1 -- 172.21.15.43:0/2030303211 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7484100b90 con 0x7f7484107130 2024-03-28T22:34:48.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.681+0000 7f747aa00700 1 -- 172.21.15.43:0/2030303211 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f747c012050 con 0x7f7484107130 2024-03-28T22:34:48.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.683+0000 7f747aa00700 1 --2- 172.21.15.43:0/2030303211 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f74680a1f20 0x7f74680a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:48.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.683+0000 7f747aa00700 1 -- 172.21.15.43:0/2030303211 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7f747c0cdce0 con 0x7f7484107130 2024-03-28T22:34:48.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.683+0000 7f7481e00700 1 --2- 172.21.15.43:0/2030303211 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f74680a1f20 0x7f74680a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:48.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.683+0000 7f747aa00700 1 --2- 172.21.15.43:0/2030303211 >> [v2:172.21.15.82:6808/215451895,v1:172.21.15.82:6809/215451895] conn(0x7f74680ad080 0x7f74680af520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:48.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.683+0000 7f747aa00700 1 -- 172.21.15.43:0/2030303211 --> [v2:172.21.15.82:6808/215451895,v1:172.21.15.82:6809/215451895] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f747c016e50 con 0x7f74680ad080 2024-03-28T22:34:48.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.683+0000 7f747aa00700 1 -- 172.21.15.43:0/2030303211 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_get_version_reply(handle=1 version=94) ==== 24+0+0 (secure 0 0 0) 0x7f747c0d0300 con 0x7f7484107130 2024-03-28T22:34:48.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.684+0000 7f7481400700 1 --2- 172.21.15.43:0/2030303211 >> [v2:172.21.15.82:6808/215451895,v1:172.21.15.82:6809/215451895] conn(0x7f74680ad080 0x7f74680af520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:48.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.684+0000 7f7481e00700 1 --2- 172.21.15.43:0/2030303211 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f74680a1f20 0x7f74680a43e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7474009810 tx=0x7f7474019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:48.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.684+0000 7f7481400700 1 --2- 172.21.15.43:0/2030303211 >> [v2:172.21.15.82:6808/215451895,v1:172.21.15.82:6809/215451895] conn(0x7f74680ad080 0x7f74680af520 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:48.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.685+0000 7f747aa00700 1 -- 172.21.15.43:0/2030303211 <== osd.4 v2:172.21.15.82:6808/215451895 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f747c016e50 con 0x7f74680ad080 2024-03-28T22:34:48.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.722+0000 7f7488a00700 1 -- 172.21.15.43:0/2030303211 --> [v2:172.21.15.82:6808/215451895,v1:172.21.15.82:6809/215451895] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7484101720 con 0x7f74680ad080 2024-03-28T22:34:48.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.722+0000 7f747aa00700 1 -- 172.21.15.43:0/2030303211 <== osd.4 v2:172.21.15.82:6808/215451895 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7484101720 con 0x7f74680ad080 2024-03-28T22:34:48.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.723+0000 7f7488a00700 1 -- 172.21.15.43:0/2030303211 >> [v2:172.21.15.82:6808/215451895,v1:172.21.15.82:6809/215451895] conn(0x7f74680ad080 msgr2=0x7f74680af520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:48.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.723+0000 7f7488a00700 1 --2- 172.21.15.43:0/2030303211 >> [v2:172.21.15.82:6808/215451895,v1:172.21.15.82:6809/215451895] conn(0x7f74680ad080 0x7f74680af520 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:48.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.723+0000 7f7488a00700 1 -- 172.21.15.43:0/2030303211 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f74680a1f20 msgr2=0x7f74680a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:48.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.723+0000 7f7488a00700 1 --2- 172.21.15.43:0/2030303211 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f74680a1f20 0x7f74680a43e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7474009810 tx=0x7f7474019040 comp rx=0 tx=0).stop 2024-03-28T22:34:48.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.723+0000 7f7488a00700 1 -- 172.21.15.43:0/2030303211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7484107130 msgr2=0x7f74841a3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:48.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.723+0000 7f7488a00700 1 --2- 172.21.15.43:0/2030303211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7484107130 0x7f74841a3940 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f747c00e3c0 tx=0x7f747c00e6d0 comp rx=0 tx=0).stop 2024-03-28T22:34:48.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.724+0000 7f7482800700 1 -- 172.21.15.43:0/2030303211 reap_dead start 2024-03-28T22:34:48.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.724+0000 7f7488a00700 1 -- 172.21.15.43:0/2030303211 shutdown_connections 2024-03-28T22:34:48.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.724+0000 7f7488a00700 1 -- 172.21.15.43:0/2030303211 >> 172.21.15.43:0/2030303211 conn(0x7f7484078070 msgr2=0x7f7484100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:48.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.724+0000 7f7488a00700 1 -- 172.21.15.43:0/2030303211 shutdown_connections 2024-03-28T22:34:48.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:47.724+0000 7f7488a00700 1 -- 172.21.15.43:0/2030303211 wait complete. 2024-03-28T22:34:49.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:49 smithi172 ceph-mon[36566]: pgmap v364: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-28T22:34:49.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:49 smithi082 ceph-mon[36471]: pgmap v364: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-28T22:34:49.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:49 smithi043 ceph-mon[31823]: pgmap v364: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-28T22:34:49.622 INFO:teuthology.orchestra.run.smithi043.stdout:55834574932 2024-03-28T22:34:49.623 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-28T22:34:51.138 INFO:teuthology.orchestra.run.smithi043.stdout:141733920837 2024-03-28T22:34:51.138 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-28T22:34:51.353 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:51 smithi043 ceph-mon[31823]: pgmap v365: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-28T22:34:51.414 INFO:teuthology.orchestra.run.smithi043.stdout:77309411410 2024-03-28T22:34:51.415 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-28T22:34:51.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:51 smithi172 ceph-mon[36566]: pgmap v365: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-28T22:34:51.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:51 smithi082 ceph-mon[36471]: pgmap v365: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-28T22:34:51.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.855+0000 7f37e8e00700 1 -- 172.21.15.43:0/1909454483 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f37e410ec40 msgr2=0x7f37e4111030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:51.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.855+0000 7f37e8e00700 1 --2- 172.21.15.43:0/1909454483 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f37e410ec40 0x7f37e4111030 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f37dc00b0a0 tx=0x7f37dc00b3b0 comp rx=0 tx=0).stop 2024-03-28T22:34:51.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.855+0000 7f37e8e00700 1 -- 172.21.15.43:0/1909454483 shutdown_connections 2024-03-28T22:34:51.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.855+0000 7f37e8e00700 1 --2- 172.21.15.43:0/1909454483 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f37e410ec40 0x7f37e4111030 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:51.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.855+0000 7f37e8e00700 1 --2- 172.21.15.43:0/1909454483 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f37e4074470 0x7f37e4074850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:51.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.855+0000 7f37e8e00700 1 --2- 172.21.15.43:0/1909454483 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37e4074d90 0x7f37e410e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:51.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.855+0000 7f37e8e00700 1 -- 172.21.15.43:0/1909454483 >> 172.21.15.43:0/1909454483 conn(0x7f37e4079370 msgr2=0x7f37e4079780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:51.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.855+0000 7f37e8e00700 1 -- 172.21.15.43:0/1909454483 shutdown_connections 2024-03-28T22:34:51.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.855+0000 7f37e8e00700 1 -- 172.21.15.43:0/1909454483 wait complete. 2024-03-28T22:34:51.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.856+0000 7f37e8e00700 1 Processor -- start 2024-03-28T22:34:51.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.857+0000 7f37e8e00700 1 -- start start 2024-03-28T22:34:51.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.857+0000 7f37e8e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f37e4074470 0x7f37e41a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:51.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.857+0000 7f37e8e00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37e4074d90 0x7f37e41a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:51.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.857+0000 7f37e8e00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f37e410ec40 0x7f37e419a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:51.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.857+0000 7f37e8e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f37e41146e0 con 0x7f37e4074470 2024-03-28T22:34:51.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.857+0000 7f37e8e00700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f37e4114560 con 0x7f37e410ec40 2024-03-28T22:34:51.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.857+0000 7f37e8e00700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f37e4114860 con 0x7f37e4074d90 2024-03-28T22:34:51.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.857+0000 7f37e2800700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f37e410ec40 0x7f37e419a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:51.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.857+0000 7f37e1400700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37e4074d90 0x7f37e41a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:51.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.857+0000 7f37e2800700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f37e410ec40 0x7f37e419a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45830/0 (socket says 172.21.15.43:45830) 2024-03-28T22:34:51.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.857+0000 7f37e2800700 1 -- 172.21.15.43:0/570137779 learned_addr learned my addr 172.21.15.43:0/570137779 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:34:51.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.857+0000 7f37e1e00700 1 --2- 172.21.15.43:0/570137779 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f37e4074470 0x7f37e41a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:51.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.858+0000 7f37e2800700 1 -- 172.21.15.43:0/570137779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37e4074d90 msgr2=0x7f37e41a0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:51.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.858+0000 7f37e2800700 1 --2- 172.21.15.43:0/570137779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f37e4074d90 0x7f37e41a0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:51.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.858+0000 7f37e2800700 1 -- 172.21.15.43:0/570137779 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f37e4074470 msgr2=0x7f37e41a0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:51.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.858+0000 7f37e2800700 1 --2- 172.21.15.43:0/570137779 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f37e4074470 0x7f37e41a0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:51.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.858+0000 7f37e2800700 1 -- 172.21.15.43:0/570137779 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37e419b100 con 0x7f37e410ec40 2024-03-28T22:34:51.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.858+0000 7f37e2800700 1 --2- 172.21.15.43:0/570137779 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f37e410ec40 0x7f37e419a9a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f37dc00b780 tx=0x7f37dc004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:51.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.858+0000 7f37daa00700 1 -- 172.21.15.43:0/570137779 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc021070 con 0x7f37e410ec40 2024-03-28T22:34:51.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.858+0000 7f37daa00700 1 -- 172.21.15.43:0/570137779 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37dc007b80 con 0x7f37e410ec40 2024-03-28T22:34:51.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.858+0000 7f37daa00700 1 -- 172.21.15.43:0/570137779 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc0052a0 con 0x7f37e410ec40 2024-03-28T22:34:51.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.858+0000 7f37e8e00700 1 -- 172.21.15.43:0/570137779 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37e419b390 con 0x7f37e410ec40 2024-03-28T22:34:51.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.859+0000 7f37e8e00700 1 -- 172.21.15.43:0/570137779 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37e41a7660 con 0x7f37e410ec40 2024-03-28T22:34:51.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.859+0000 7f37e8e00700 1 -- 172.21.15.43:0/570137779 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f37e40ffeb0 con 0x7f37e410ec40 2024-03-28T22:34:51.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.863+0000 7f37daa00700 1 -- 172.21.15.43:0/570137779 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f37dc024070 con 0x7f37e410ec40 2024-03-28T22:34:51.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.864+0000 7f37daa00700 1 --2- 172.21.15.43:0/570137779 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f37c80a1f20 0x7f37c80a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:51.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.864+0000 7f37daa00700 1 -- 172.21.15.43:0/570137779 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7f37dc00f020 con 0x7f37e410ec40 2024-03-28T22:34:51.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.864+0000 7f37e1e00700 1 --2- 172.21.15.43:0/570137779 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f37c80a1f20 0x7f37c80a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:51.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.865+0000 7f37daa00700 1 --2- 172.21.15.43:0/570137779 >> [v2:172.21.15.43:6810/1597280920,v1:172.21.15.43:6811/1597280920] conn(0x7f37c80acf50 0x7f37c80af3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:51.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.865+0000 7f37daa00700 1 -- 172.21.15.43:0/570137779 --> [v2:172.21.15.43:6810/1597280920,v1:172.21.15.43:6811/1597280920] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f37dc030ea0 con 0x7f37c80acf50 2024-03-28T22:34:51.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.865+0000 7f37daa00700 1 -- 172.21.15.43:0/570137779 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_get_version_reply(handle=1 version=94) ==== 24+0+0 (secure 0 0 0) 0x7f37dc0cfd60 con 0x7f37e410ec40 2024-03-28T22:34:51.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.865+0000 7f37e1400700 1 --2- 172.21.15.43:0/570137779 >> [v2:172.21.15.43:6810/1597280920,v1:172.21.15.43:6811/1597280920] conn(0x7f37c80acf50 0x7f37c80af3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:51.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.865+0000 7f37e1e00700 1 --2- 172.21.15.43:0/570137779 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f37c80a1f20 0x7f37c80a43e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f37e4102010 tx=0x7f37d4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:51.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.866+0000 7f37e1400700 1 --2- 172.21.15.43:0/570137779 >> [v2:172.21.15.43:6810/1597280920,v1:172.21.15.43:6811/1597280920] conn(0x7f37c80acf50 0x7f37c80af3f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:51.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.867+0000 7f37daa00700 1 -- 172.21.15.43:0/570137779 <== osd.3 v2:172.21.15.43:6810/1597280920 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f37dc030ea0 con 0x7f37c80acf50 2024-03-28T22:34:51.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.886+0000 7f37e8e00700 1 -- 172.21.15.43:0/570137779 --> [v2:172.21.15.43:6810/1597280920,v1:172.21.15.43:6811/1597280920] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f37e4075b10 con 0x7f37c80acf50 2024-03-28T22:34:51.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.887+0000 7f37daa00700 1 -- 172.21.15.43:0/570137779 <== osd.3 v2:172.21.15.43:6810/1597280920 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f37e4075b10 con 0x7f37c80acf50 2024-03-28T22:34:51.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.887+0000 7f37e8e00700 1 -- 172.21.15.43:0/570137779 >> [v2:172.21.15.43:6810/1597280920,v1:172.21.15.43:6811/1597280920] conn(0x7f37c80acf50 msgr2=0x7f37c80af3f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:51.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.887+0000 7f37e8e00700 1 --2- 172.21.15.43:0/570137779 >> [v2:172.21.15.43:6810/1597280920,v1:172.21.15.43:6811/1597280920] conn(0x7f37c80acf50 0x7f37c80af3f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:51.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.887+0000 7f37e8e00700 1 -- 172.21.15.43:0/570137779 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f37c80a1f20 msgr2=0x7f37c80a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:51.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.887+0000 7f37e8e00700 1 --2- 172.21.15.43:0/570137779 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f37c80a1f20 0x7f37c80a43e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f37e4102010 tx=0x7f37d4009260 comp rx=0 tx=0).stop 2024-03-28T22:34:51.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.887+0000 7f37e8e00700 1 -- 172.21.15.43:0/570137779 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f37e410ec40 msgr2=0x7f37e419a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:51.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.887+0000 7f37e8e00700 1 --2- 172.21.15.43:0/570137779 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f37e410ec40 0x7f37e419a9a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f37dc00b780 tx=0x7f37dc004d00 comp rx=0 tx=0).stop 2024-03-28T22:34:51.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.888+0000 7f37e2800700 1 -- 172.21.15.43:0/570137779 reap_dead start 2024-03-28T22:34:51.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.888+0000 7f37e8e00700 1 -- 172.21.15.43:0/570137779 shutdown_connections 2024-03-28T22:34:51.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.888+0000 7f37e8e00700 1 -- 172.21.15.43:0/570137779 >> 172.21.15.43:0/570137779 conn(0x7f37e4079370 msgr2=0x7f37e40ff450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:51.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.888+0000 7f37e8e00700 1 -- 172.21.15.43:0/570137779 shutdown_connections 2024-03-28T22:34:51.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.888+0000 7f37e8e00700 1 -- 172.21.15.43:0/570137779 wait complete. 2024-03-28T22:34:52.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.423+0000 7ff7d7e00700 1 -- 172.21.15.43:0/1408634048 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff7d8113120 msgr2=0x7ff7d8115510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:52.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.423+0000 7ff7d7e00700 1 --2- 172.21.15.43:0/1408634048 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff7d8113120 0x7ff7d8115510 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0009f90 tx=0x7ff7d000b330 comp rx=0 tx=0).stop 2024-03-28T22:34:52.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.423+0000 7ff7d7e00700 1 -- 172.21.15.43:0/1408634048 shutdown_connections 2024-03-28T22:34:52.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.423+0000 7ff7d7e00700 1 --2- 172.21.15.43:0/1408634048 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff7d8113120 0x7ff7d8115510 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:52.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.423+0000 7ff7d7e00700 1 --2- 172.21.15.43:0/1408634048 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff7d80745e0 0x7ff7d80749c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:52.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.423+0000 7ff7d7e00700 1 --2- 172.21.15.43:0/1408634048 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7d8074f00 0x7ff7d8112be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:52.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.423+0000 7ff7d7e00700 1 -- 172.21.15.43:0/1408634048 >> 172.21.15.43:0/1408634048 conn(0x7ff7d80fdc60 msgr2=0x7ff7d8100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:52.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.423+0000 7ff7d7e00700 1 -- 172.21.15.43:0/1408634048 shutdown_connections 2024-03-28T22:34:52.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.423+0000 7ff7d7e00700 1 -- 172.21.15.43:0/1408634048 wait complete. 2024-03-28T22:34:52.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.424+0000 7ff7d7e00700 1 Processor -- start 2024-03-28T22:34:52.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.425+0000 7ff7d7e00700 1 -- start start 2024-03-28T22:34:52.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.425+0000 7ff7d7e00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7d80745e0 0x7ff7d819eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:52.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.425+0000 7ff7d7e00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff7d8074f00 0x7ff7d819f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:52.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.425+0000 7ff7d7e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff7d8113120 0x7ff7d81a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:52.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.425+0000 7ff7d7e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d8118a70 con 0x7ff7d8113120 2024-03-28T22:34:52.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.425+0000 7ff7d7e00700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d81188f0 con 0x7ff7d8074f00 2024-03-28T22:34:52.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.425+0000 7ff7d7e00700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d8118bf0 con 0x7ff7d80745e0 2024-03-28T22:34:52.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.425+0000 7ff7d6a00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7d80745e0 0x7ff7d819eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:52.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.425+0000 7ff7d6000700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff7d8074f00 0x7ff7d819f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:52.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.425+0000 7ff7d6a00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7d80745e0 0x7ff7d819eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35214/0 (socket says 172.21.15.43:35214) 2024-03-28T22:34:52.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.425+0000 7ff7d6a00700 1 -- 172.21.15.43:0/4018080879 learned_addr learned my addr 172.21.15.43:0/4018080879 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:34:52.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.425+0000 7ff7d7400700 1 --2- 172.21.15.43:0/4018080879 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff7d8113120 0x7ff7d81a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:52.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.426+0000 7ff7d6a00700 1 -- 172.21.15.43:0/4018080879 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff7d8074f00 msgr2=0x7ff7d819f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:52.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.426+0000 7ff7d6a00700 1 --2- 172.21.15.43:0/4018080879 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff7d8074f00 0x7ff7d819f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:52.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.426+0000 7ff7d6a00700 1 -- 172.21.15.43:0/4018080879 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff7d8113120 msgr2=0x7ff7d81a3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:52.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.426+0000 7ff7d6a00700 1 --2- 172.21.15.43:0/4018080879 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff7d8113120 0x7ff7d81a3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:52.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.426+0000 7ff7d6a00700 1 -- 172.21.15.43:0/4018080879 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7d81a3eb0 con 0x7ff7d80745e0 2024-03-28T22:34:52.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.426+0000 7ff7d6000700 1 --2- 172.21.15.43:0/4018080879 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff7d8074f00 0x7ff7d819f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:34:52.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.426+0000 7ff7d6a00700 1 --2- 172.21.15.43:0/4018080879 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7d80745e0 0x7ff7d819eea0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff7c800e9a0 tx=0x7ff7c800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:52.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.426+0000 7ff7cf400700 1 -- 172.21.15.43:0/4018080879 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c800ca10 con 0x7ff7d80745e0 2024-03-28T22:34:52.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.426+0000 7ff7cf400700 1 -- 172.21.15.43:0/4018080879 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7c800cbb0 con 0x7ff7d80745e0 2024-03-28T22:34:52.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.427+0000 7ff7cf400700 1 -- 172.21.15.43:0/4018080879 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c8018680 con 0x7ff7d80745e0 2024-03-28T22:34:52.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.427+0000 7ff7d7e00700 1 -- 172.21.15.43:0/4018080879 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7d81a41a0 con 0x7ff7d80745e0 2024-03-28T22:34:52.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.427+0000 7ff7d7e00700 1 -- 172.21.15.43:0/4018080879 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7d81aba20 con 0x7ff7d80745e0 2024-03-28T22:34:52.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.428+0000 7ff7d7e00700 1 -- 172.21.15.43:0/4018080879 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff7d80629d0 con 0x7ff7d80745e0 2024-03-28T22:34:52.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.430+0000 7ff7cf400700 1 -- 172.21.15.43:0/4018080879 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7ff7c8010430 con 0x7ff7d80745e0 2024-03-28T22:34:52.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.431+0000 7ff7cf400700 1 --2- 172.21.15.43:0/4018080879 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff7bc0a1fa0 0x7ff7bc0a4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:52.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.431+0000 7ff7cf400700 1 -- 172.21.15.43:0/4018080879 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7ff7c80ce4c0 con 0x7ff7d80745e0 2024-03-28T22:34:52.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.432+0000 7ff7cf400700 1 --2- 172.21.15.43:0/4018080879 >> [v2:172.21.15.172:6808/1576148933,v1:172.21.15.172:6809/1576148933] conn(0x7ff7bc0ad100 0x7ff7bc0af5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:52.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.432+0000 7ff7cf400700 1 -- 172.21.15.43:0/4018080879 --> [v2:172.21.15.172:6808/1576148933,v1:172.21.15.172:6809/1576148933] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff7c8015e80 con 0x7ff7bc0ad100 2024-03-28T22:34:52.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.432+0000 7ff7cf400700 1 -- 172.21.15.43:0/4018080879 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_get_version_reply(handle=1 version=94) ==== 24+0+0 (secure 0 0 0) 0x7ff7c80d5050 con 0x7ff7d80745e0 2024-03-28T22:34:52.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.432+0000 7ff7d7400700 1 --2- 172.21.15.43:0/4018080879 >> [v2:172.21.15.172:6808/1576148933,v1:172.21.15.172:6809/1576148933] conn(0x7ff7bc0ad100 0x7ff7bc0af5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:52.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.432+0000 7ff7d6000700 1 --2- 172.21.15.43:0/4018080879 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff7bc0a1fa0 0x7ff7bc0a4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:52.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.433+0000 7ff7d7400700 1 --2- 172.21.15.43:0/4018080879 >> [v2:172.21.15.172:6808/1576148933,v1:172.21.15.172:6809/1576148933] conn(0x7ff7bc0ad100 0x7ff7bc0af5a0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:52.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.433+0000 7ff7d6000700 1 --2- 172.21.15.43:0/4018080879 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff7bc0a1fa0 0x7ff7bc0a4460 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8105f10 tx=0x7ff7c4009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:52.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.434+0000 7ff7cf400700 1 -- 172.21.15.43:0/4018080879 <== osd.5 v2:172.21.15.172:6808/1576148933 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff7c8015e80 con 0x7ff7bc0ad100 2024-03-28T22:34:52.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.456+0000 7ff7d7e00700 1 -- 172.21.15.43:0/4018080879 --> [v2:172.21.15.172:6808/1576148933,v1:172.21.15.172:6809/1576148933] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff7d8062e00 con 0x7ff7bc0ad100 2024-03-28T22:34:52.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.456+0000 7ff7cf400700 1 -- 172.21.15.43:0/4018080879 <== osd.5 v2:172.21.15.172:6808/1576148933 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff7d8062e00 con 0x7ff7bc0ad100 2024-03-28T22:34:52.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.457+0000 7ff7d7e00700 1 -- 172.21.15.43:0/4018080879 >> [v2:172.21.15.172:6808/1576148933,v1:172.21.15.172:6809/1576148933] conn(0x7ff7bc0ad100 msgr2=0x7ff7bc0af5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:52.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.457+0000 7ff7d7e00700 1 --2- 172.21.15.43:0/4018080879 >> [v2:172.21.15.172:6808/1576148933,v1:172.21.15.172:6809/1576148933] conn(0x7ff7bc0ad100 0x7ff7bc0af5a0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:52.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.457+0000 7ff7d7e00700 1 -- 172.21.15.43:0/4018080879 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff7bc0a1fa0 msgr2=0x7ff7bc0a4460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:52.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.457+0000 7ff7d7e00700 1 --2- 172.21.15.43:0/4018080879 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff7bc0a1fa0 0x7ff7bc0a4460 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8105f10 tx=0x7ff7c4009310 comp rx=0 tx=0).stop 2024-03-28T22:34:52.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.457+0000 7ff7d7e00700 1 -- 172.21.15.43:0/4018080879 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7d80745e0 msgr2=0x7ff7d819eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:52.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.457+0000 7ff7d7e00700 1 --2- 172.21.15.43:0/4018080879 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff7d80745e0 0x7ff7d819eea0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff7c800e9a0 tx=0x7ff7c800ecb0 comp rx=0 tx=0).stop 2024-03-28T22:34:52.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.457+0000 7ff7d7400700 1 -- 172.21.15.43:0/4018080879 reap_dead start 2024-03-28T22:34:52.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.458+0000 7ff7d7e00700 1 -- 172.21.15.43:0/4018080879 shutdown_connections 2024-03-28T22:34:52.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.458+0000 7ff7d7e00700 1 -- 172.21.15.43:0/4018080879 >> 172.21.15.43:0/4018080879 conn(0x7ff7d80fdc60 msgr2=0x7ff7d80fe8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:52.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.458+0000 7ff7d7e00700 1 -- 172.21.15.43:0/4018080879 shutdown_connections 2024-03-28T22:34:52.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:51.458+0000 7ff7d7e00700 1 -- 172.21.15.43:0/4018080879 wait complete. 2024-03-28T22:34:53.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.380+0000 7fe834200700 1 -- 172.21.15.43:0/2522883862 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe82c10a4c0 msgr2=0x7fe82c10a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:53.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.380+0000 7fe834200700 1 --2- 172.21.15.43:0/2522883862 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe82c10a4c0 0x7fe82c10a8a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe8200098f0 tx=0x7fe820009c00 comp rx=0 tx=0).stop 2024-03-28T22:34:53.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.381+0000 7fe834200700 1 -- 172.21.15.43:0/2522883862 shutdown_connections 2024-03-28T22:34:53.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.381+0000 7fe834200700 1 --2- 172.21.15.43:0/2522883862 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe82c104df0 0x7fe82c111300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:53.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.381+0000 7fe834200700 1 --2- 172.21.15.43:0/2522883862 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe82c10a4c0 0x7fe82c10a8a0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:53.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.381+0000 7fe834200700 1 --2- 172.21.15.43:0/2522883862 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe82c104430 0x7fe82c1048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:53.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.381+0000 7fe834200700 1 -- 172.21.15.43:0/2522883862 >> 172.21.15.43:0/2522883862 conn(0x7fe82c0fff20 msgr2=0x7fe82c102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:53.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.381+0000 7fe834200700 1 -- 172.21.15.43:0/2522883862 shutdown_connections 2024-03-28T22:34:53.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.381+0000 7fe834200700 1 -- 172.21.15.43:0/2522883862 wait complete. 2024-03-28T22:34:53.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.382+0000 7fe834200700 1 Processor -- start 2024-03-28T22:34:53.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.382+0000 7fe834200700 1 -- start start 2024-03-28T22:34:53.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.382+0000 7fe834200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe82c104430 0x7fe82c1a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:53.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.382+0000 7fe834200700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe82c104df0 0x7fe82c1a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:53.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.382+0000 7fe834200700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe82c10a4c0 0x7fe82c19aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:53.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.382+0000 7fe834200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe82c114790 con 0x7fe82c104430 2024-03-28T22:34:53.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.382+0000 7fe834200700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe82c114610 con 0x7fe82c10a4c0 2024-03-28T22:34:53.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.382+0000 7fe834200700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe82c114910 con 0x7fe82c104df0 2024-03-28T22:34:53.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.383+0000 7fe832200700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe82c10a4c0 0x7fe82c19aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:53.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.383+0000 7fe832200700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe82c10a4c0 0x7fe82c19aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45848/0 (socket says 172.21.15.43:45848) 2024-03-28T22:34:53.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.383+0000 7fe830e00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe82c104df0 0x7fe82c1a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:53.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.383+0000 7fe832200700 1 -- 172.21.15.43:0/3702289797 learned_addr learned my addr 172.21.15.43:0/3702289797 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:34:53.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.383+0000 7fe831800700 1 --2- 172.21.15.43:0/3702289797 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe82c104430 0x7fe82c1a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:53.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.383+0000 7fe832200700 1 -- 172.21.15.43:0/3702289797 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe82c104df0 msgr2=0x7fe82c1a0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:53.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.383+0000 7fe832200700 1 --2- 172.21.15.43:0/3702289797 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe82c104df0 0x7fe82c1a0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:53.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.383+0000 7fe832200700 1 -- 172.21.15.43:0/3702289797 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe82c104430 msgr2=0x7fe82c1a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:53.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.383+0000 7fe832200700 1 --2- 172.21.15.43:0/3702289797 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe82c104430 0x7fe82c1a08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:53.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.383+0000 7fe832200700 1 -- 172.21.15.43:0/3702289797 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe82c19b1f0 con 0x7fe82c10a4c0 2024-03-28T22:34:53.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.383+0000 7fe832200700 1 --2- 172.21.15.43:0/3702289797 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe82c10a4c0 0x7fe82c19aa90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe82400c110 tx=0x7fe82400c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:53.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.384+0000 7fe82a000700 1 -- 172.21.15.43:0/3702289797 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8240092e0 con 0x7fe82c10a4c0 2024-03-28T22:34:53.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.384+0000 7fe82a000700 1 -- 172.21.15.43:0/3702289797 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe824004bb0 con 0x7fe82c10a4c0 2024-03-28T22:34:53.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.384+0000 7fe82a000700 1 -- 172.21.15.43:0/3702289797 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe824007da0 con 0x7fe82c10a4c0 2024-03-28T22:34:53.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.384+0000 7fe834200700 1 -- 172.21.15.43:0/3702289797 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe82c19b480 con 0x7fe82c10a4c0 2024-03-28T22:34:53.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.384+0000 7fe834200700 1 -- 172.21.15.43:0/3702289797 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe82c1a77a0 con 0x7fe82c10a4c0 2024-03-28T22:34:53.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.385+0000 7fe834200700 1 -- 172.21.15.43:0/3702289797 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe82c062950 con 0x7fe82c10a4c0 2024-03-28T22:34:53.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.387+0000 7fe82a000700 1 -- 172.21.15.43:0/3702289797 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fe82401a020 con 0x7fe82c10a4c0 2024-03-28T22:34:53.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.388+0000 7fe82a000700 1 --2- 172.21.15.43:0/3702289797 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe8140a1ea0 0x7fe8140a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:53.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.388+0000 7fe82a000700 1 -- 172.21.15.43:0/3702289797 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7fe8240cddd0 con 0x7fe82c10a4c0 2024-03-28T22:34:53.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.388+0000 7fe831800700 1 --2- 172.21.15.43:0/3702289797 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe8140a1ea0 0x7fe8140a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:53.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.388+0000 7fe82a000700 1 --2- 172.21.15.43:0/3702289797 >> [v2:172.21.15.172:6824/2563517144,v1:172.21.15.172:6825/2563517144] conn(0x7fe8140ad000 0x7fe8140af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:53.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.388+0000 7fe82a000700 1 -- 172.21.15.43:0/3702289797 --> [v2:172.21.15.172:6824/2563517144,v1:172.21.15.172:6825/2563517144] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe82401fe50 con 0x7fe8140ad000 2024-03-28T22:34:53.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.388+0000 7fe82a000700 1 -- 172.21.15.43:0/3702289797 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_get_version_reply(handle=1 version=94) ==== 24+0+0 (secure 0 0 0) 0x7fe8240d1020 con 0x7fe82c10a4c0 2024-03-28T22:34:53.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.389+0000 7fe830e00700 1 --2- 172.21.15.43:0/3702289797 >> [v2:172.21.15.172:6824/2563517144,v1:172.21.15.172:6825/2563517144] conn(0x7fe8140ad000 0x7fe8140af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:53.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.389+0000 7fe831800700 1 --2- 172.21.15.43:0/3702289797 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe8140a1ea0 0x7fe8140a4360 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe820009810 tx=0x7fe820019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:53.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.389+0000 7fe830e00700 1 --2- 172.21.15.43:0/3702289797 >> [v2:172.21.15.172:6824/2563517144,v1:172.21.15.172:6825/2563517144] conn(0x7fe8140ad000 0x7fe8140af4a0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:53.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.390+0000 7fe82a000700 1 -- 172.21.15.43:0/3702289797 <== osd.11 v2:172.21.15.172:6824/2563517144 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe82401fe50 con 0x7fe8140ad000 2024-03-28T22:34:53.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.409+0000 7fe834200700 1 -- 172.21.15.43:0/3702289797 --> [v2:172.21.15.172:6824/2563517144,v1:172.21.15.172:6825/2563517144] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe82c062dd0 con 0x7fe8140ad000 2024-03-28T22:34:53.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.409+0000 7fe82a000700 1 -- 172.21.15.43:0/3702289797 <== osd.11 v2:172.21.15.172:6824/2563517144 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe82c062dd0 con 0x7fe8140ad000 2024-03-28T22:34:53.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.409+0000 7fe834200700 1 -- 172.21.15.43:0/3702289797 >> [v2:172.21.15.172:6824/2563517144,v1:172.21.15.172:6825/2563517144] conn(0x7fe8140ad000 msgr2=0x7fe8140af4a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:53.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.410+0000 7fe834200700 1 --2- 172.21.15.43:0/3702289797 >> [v2:172.21.15.172:6824/2563517144,v1:172.21.15.172:6825/2563517144] conn(0x7fe8140ad000 0x7fe8140af4a0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:53.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.410+0000 7fe834200700 1 -- 172.21.15.43:0/3702289797 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe8140a1ea0 msgr2=0x7fe8140a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:53.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.410+0000 7fe834200700 1 --2- 172.21.15.43:0/3702289797 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe8140a1ea0 0x7fe8140a4360 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe820009810 tx=0x7fe820019040 comp rx=0 tx=0).stop 2024-03-28T22:34:53.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.410+0000 7fe834200700 1 -- 172.21.15.43:0/3702289797 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe82c10a4c0 msgr2=0x7fe82c19aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:53.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.410+0000 7fe834200700 1 --2- 172.21.15.43:0/3702289797 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe82c10a4c0 0x7fe82c19aa90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe82400c110 tx=0x7fe82400c420 comp rx=0 tx=0).stop 2024-03-28T22:34:53.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.410+0000 7fe832200700 1 -- 172.21.15.43:0/3702289797 reap_dead start 2024-03-28T22:34:53.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.410+0000 7fe834200700 1 -- 172.21.15.43:0/3702289797 shutdown_connections 2024-03-28T22:34:53.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.410+0000 7fe834200700 1 -- 172.21.15.43:0/3702289797 >> 172.21.15.43:0/3702289797 conn(0x7fe82c0fff20 msgr2=0x7fe82c1005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:53.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.411+0000 7fe834200700 1 -- 172.21.15.43:0/3702289797 shutdown_connections 2024-03-28T22:34:53.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:53.411+0000 7fe834200700 1 -- 172.21.15.43:0/3702289797 wait complete. 2024-03-28T22:34:53.449 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:53 smithi043 ceph-mon[31823]: pgmap v366: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:34:53.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:53 smithi172 ceph-mon[36566]: pgmap v366: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:34:53.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:53 smithi082 ceph-mon[36471]: pgmap v366: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:34:54.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.146+0000 7fb0e9e00700 1 -- 172.21.15.43:0/3535820519 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb0e4102d40 msgr2=0x7fb0e4106d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:54.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.146+0000 7fb0e9e00700 1 --2- 172.21.15.43:0/3535820519 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb0e4102d40 0x7fb0e4106d90 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc009990 tx=0x7fb0cc009ca0 comp rx=0 tx=0).stop 2024-03-28T22:34:54.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.146+0000 7fb0e9e00700 1 -- 172.21.15.43:0/3535820519 shutdown_connections 2024-03-28T22:34:54.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.146+0000 7fb0e9e00700 1 --2- 172.21.15.43:0/3535820519 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0e4107470 0x7fb0e410f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:54.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.146+0000 7fb0e9e00700 1 --2- 172.21.15.43:0/3535820519 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0e4102390 0x7fb0e4102770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:54.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.146+0000 7fb0e9e00700 1 --2- 172.21.15.43:0/3535820519 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb0e4102d40 0x7fb0e4106d90 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:54.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.146+0000 7fb0e9e00700 1 -- 172.21.15.43:0/3535820519 >> 172.21.15.43:0/3535820519 conn(0x7fb0e40fdc40 msgr2=0x7fb0e4100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:54.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.146+0000 7fb0e9e00700 1 -- 172.21.15.43:0/3535820519 shutdown_connections 2024-03-28T22:34:54.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.146+0000 7fb0e9e00700 1 -- 172.21.15.43:0/3535820519 wait complete. 2024-03-28T22:34:54.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.147+0000 7fb0e9e00700 1 Processor -- start 2024-03-28T22:34:54.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.147+0000 7fb0e9e00700 1 -- start start 2024-03-28T22:34:54.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.148+0000 7fb0e9e00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0e4102390 0x7fb0e41a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:54.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.148+0000 7fb0e9e00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb0e4102d40 0x7fb0e41a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:54.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.148+0000 7fb0e9e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0e4107470 0x7fb0e419a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:54.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.148+0000 7fb0e9e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e4112480 con 0x7fb0e4107470 2024-03-28T22:34:54.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.148+0000 7fb0e9e00700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e4112300 con 0x7fb0e4102d40 2024-03-28T22:34:54.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.148+0000 7fb0e9e00700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e4112600 con 0x7fb0e4102390 2024-03-28T22:34:54.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.148+0000 7fb0dbe00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb0e4102d40 0x7fb0e41a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:54.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.148+0000 7fb0e8a00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0e4102390 0x7fb0e41a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:54.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.148+0000 7fb0dbe00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb0e4102d40 0x7fb0e41a0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45862/0 (socket says 172.21.15.43:45862) 2024-03-28T22:34:54.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.148+0000 7fb0dbe00700 1 -- 172.21.15.43:0/892366953 learned_addr learned my addr 172.21.15.43:0/892366953 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:34:54.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.148+0000 7fb0e8a00700 1 --2- 172.21.15.43:0/892366953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0e4102390 0x7fb0e41a0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35274/0 (socket says 172.21.15.43:35274) 2024-03-28T22:34:54.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.148+0000 7fb0e9400700 1 --2- 172.21.15.43:0/892366953 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0e4107470 0x7fb0e419a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:54.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.149+0000 7fb0dbe00700 1 -- 172.21.15.43:0/892366953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0e4102390 msgr2=0x7fb0e41a0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:54.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.149+0000 7fb0dbe00700 1 --2- 172.21.15.43:0/892366953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0e4102390 0x7fb0e41a0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:54.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.149+0000 7fb0dbe00700 1 -- 172.21.15.43:0/892366953 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0e4107470 msgr2=0x7fb0e419a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:54.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.149+0000 7fb0dbe00700 1 --2- 172.21.15.43:0/892366953 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0e4107470 0x7fb0e419a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:54.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.149+0000 7fb0dbe00700 1 -- 172.21.15.43:0/892366953 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e419b100 con 0x7fb0e4102d40 2024-03-28T22:34:54.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.149+0000 7fb0dbe00700 1 --2- 172.21.15.43:0/892366953 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb0e4102d40 0x7fb0e41a0cd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc000c00 tx=0x7fb0cc005590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:54.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.149+0000 7fb0e2000700 1 -- 172.21.15.43:0/892366953 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc027070 con 0x7fb0e4102d40 2024-03-28T22:34:54.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.149+0000 7fb0e2000700 1 -- 172.21.15.43:0/892366953 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0cc0056b0 con 0x7fb0e4102d40 2024-03-28T22:34:54.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.149+0000 7fb0e9e00700 1 -- 172.21.15.43:0/892366953 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0e419b390 con 0x7fb0e4102d40 2024-03-28T22:34:54.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.149+0000 7fb0e2000700 1 -- 172.21.15.43:0/892366953 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc0173f0 con 0x7fb0e4102d40 2024-03-28T22:34:54.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.149+0000 7fb0e9e00700 1 -- 172.21.15.43:0/892366953 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0e41a7660 con 0x7fb0e4102d40 2024-03-28T22:34:54.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.150+0000 7fb0e9e00700 1 -- 172.21.15.43:0/892366953 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb0e410e760 con 0x7fb0e4102d40 2024-03-28T22:34:54.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.152+0000 7fb0e2000700 1 -- 172.21.15.43:0/892366953 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fb0cc02a070 con 0x7fb0e4102d40 2024-03-28T22:34:54.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.154+0000 7fb0e2000700 1 --2- 172.21.15.43:0/892366953 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fb0c80a1f20 0x7fb0c80a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:54.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.154+0000 7fb0e2000700 1 -- 172.21.15.43:0/892366953 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7fb0cc0ceef0 con 0x7fb0e4102d40 2024-03-28T22:34:54.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.154+0000 7fb0e8a00700 1 --2- 172.21.15.43:0/892366953 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fb0c80a1f20 0x7fb0c80a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:54.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.154+0000 7fb0e2000700 1 --2- 172.21.15.43:0/892366953 >> [v2:172.21.15.43:6826/3657791498,v1:172.21.15.43:6827/3657791498] conn(0x7fb0c80ad080 0x7fb0c80af520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:34:54.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.154+0000 7fb0e2000700 1 -- 172.21.15.43:0/892366953 --> [v2:172.21.15.43:6826/3657791498,v1:172.21.15.43:6827/3657791498] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb0cc00fe00 con 0x7fb0c80ad080 2024-03-28T22:34:54.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.154+0000 7fb0e2000700 1 -- 172.21.15.43:0/892366953 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_get_version_reply(handle=1 version=94) ==== 24+0+0 (secure 0 0 0) 0x7fb0cc0d1510 con 0x7fb0e4102d40 2024-03-28T22:34:54.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.154+0000 7fb0e9400700 1 --2- 172.21.15.43:0/892366953 >> [v2:172.21.15.43:6826/3657791498,v1:172.21.15.43:6827/3657791498] conn(0x7fb0c80ad080 0x7fb0c80af520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:34:54.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.155+0000 7fb0e8a00700 1 --2- 172.21.15.43:0/892366953 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fb0c80a1f20 0x7fb0c80a43e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb0e40fe4d0 tx=0x7fb0d4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:54.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.155+0000 7fb0e9400700 1 --2- 172.21.15.43:0/892366953 >> [v2:172.21.15.43:6826/3657791498,v1:172.21.15.43:6827/3657791498] conn(0x7fb0c80ad080 0x7fb0c80af520 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:34:54.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.156+0000 7fb0e2000700 1 -- 172.21.15.43:0/892366953 <== osd.9 v2:172.21.15.43:6826/3657791498 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb0cc00fe00 con 0x7fb0c80ad080 2024-03-28T22:34:54.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.175+0000 7fb0e9e00700 1 -- 172.21.15.43:0/892366953 --> [v2:172.21.15.43:6826/3657791498,v1:172.21.15.43:6827/3657791498] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb0e4103870 con 0x7fb0c80ad080 2024-03-28T22:34:54.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.175+0000 7fb0e2000700 1 -- 172.21.15.43:0/892366953 <== osd.9 v2:172.21.15.43:6826/3657791498 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb0e4103870 con 0x7fb0c80ad080 2024-03-28T22:34:54.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.176+0000 7fb0e9e00700 1 -- 172.21.15.43:0/892366953 >> [v2:172.21.15.43:6826/3657791498,v1:172.21.15.43:6827/3657791498] conn(0x7fb0c80ad080 msgr2=0x7fb0c80af520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:54.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.176+0000 7fb0e9e00700 1 --2- 172.21.15.43:0/892366953 >> [v2:172.21.15.43:6826/3657791498,v1:172.21.15.43:6827/3657791498] conn(0x7fb0c80ad080 0x7fb0c80af520 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:34:54.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.176+0000 7fb0e9e00700 1 -- 172.21.15.43:0/892366953 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fb0c80a1f20 msgr2=0x7fb0c80a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:54.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.176+0000 7fb0e9e00700 1 --2- 172.21.15.43:0/892366953 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fb0c80a1f20 0x7fb0c80a43e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb0e40fe4d0 tx=0x7fb0d4009260 comp rx=0 tx=0).stop 2024-03-28T22:34:54.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.176+0000 7fb0e9e00700 1 -- 172.21.15.43:0/892366953 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb0e4102d40 msgr2=0x7fb0e41a0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:34:54.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.176+0000 7fb0e9e00700 1 --2- 172.21.15.43:0/892366953 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb0e4102d40 0x7fb0e41a0cd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc000c00 tx=0x7fb0cc005590 comp rx=0 tx=0).stop 2024-03-28T22:34:54.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.176+0000 7fb0e9400700 1 -- 172.21.15.43:0/892366953 reap_dead start 2024-03-28T22:34:54.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.176+0000 7fb0e9e00700 1 -- 172.21.15.43:0/892366953 shutdown_connections 2024-03-28T22:34:54.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.176+0000 7fb0e9e00700 1 -- 172.21.15.43:0/892366953 >> 172.21.15.43:0/892366953 conn(0x7fb0e40fdc40 msgr2=0x7fb0e40fe9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:34:54.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.176+0000 7fb0e9e00700 1 -- 172.21.15.43:0/892366953 shutdown_connections 2024-03-28T22:34:54.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:34:54.176+0000 7fb0e9e00700 1 -- 172.21.15.43:0/892366953 wait complete. 2024-03-28T22:34:55.340 INFO:teuthology.orchestra.run.smithi043.stdout:107374182475 2024-03-28T22:34:55.340 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-28T22:34:55.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:55 smithi043 ceph-mon[31823]: pgmap v367: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-28T22:34:55.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:55 smithi172 ceph-mon[36566]: pgmap v367: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-28T22:34:55.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:55 smithi082 ceph-mon[36471]: pgmap v367: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-28T22:34:55.826 INFO:teuthology.orchestra.run.smithi043.stdout:296352743466 2024-03-28T22:34:55.826 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-28T22:34:56.576 INFO:teuthology.orchestra.run.smithi043.stdout:352187318303 2024-03-28T22:34:56.576 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-28T22:34:56.912 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:34:56.921 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:34:56.981 INFO:teuthology.orchestra.run.smithi043.stdout:163208757312 2024-03-28T22:34:56.981 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-28T22:34:57.020 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:34:57.483 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:57 smithi043 ceph-mon[31823]: pgmap v368: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-03-28T22:34:57.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:57 smithi172 ceph-mon[36566]: pgmap v368: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-03-28T22:34:57.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:57 smithi082 ceph-mon[36471]: pgmap v368: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-03-28T22:34:59.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:34:59 smithi043 ceph-mon[31823]: pgmap v369: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:34:59.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:34:59 smithi172 ceph-mon[36566]: pgmap v369: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:34:59.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:34:59 smithi082 ceph-mon[36471]: pgmap v369: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:35:00.878 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:35:00.884 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:35:00.885 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:35:00.891 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:35:00.892 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:35:01.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.338+0000 7ff3ee000700 1 -- 172.21.15.43:0/2157319615 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3e8104e00 msgr2=0x7ff3e8111310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:01.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.338+0000 7ff3ee000700 1 --2- 172.21.15.43:0/2157319615 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3e8104e00 0x7ff3e8111310 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff3e000b0a0 tx=0x7ff3e000b3b0 comp rx=0 tx=0).stop 2024-03-28T22:35:01.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.338+0000 7ff3ee000700 1 -- 172.21.15.43:0/2157319615 shutdown_connections 2024-03-28T22:35:01.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.338+0000 7ff3ee000700 1 --2- 172.21.15.43:0/2157319615 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3e8104e00 0x7ff3e8111310 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:01.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.338+0000 7ff3ee000700 1 --2- 172.21.15.43:0/2157319615 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3e810a4d0 0x7ff3e810a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:01.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.338+0000 7ff3ee000700 1 --2- 172.21.15.43:0/2157319615 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff3e8104440 0x7ff3e81048c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:01.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.338+0000 7ff3ee000700 1 -- 172.21.15.43:0/2157319615 >> 172.21.15.43:0/2157319615 conn(0x7ff3e80fff10 msgr2=0x7ff3e8102330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:01.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.338+0000 7ff3ee000700 1 -- 172.21.15.43:0/2157319615 shutdown_connections 2024-03-28T22:35:01.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.338+0000 7ff3ee000700 1 -- 172.21.15.43:0/2157319615 wait complete. 2024-03-28T22:35:01.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.340+0000 7ff3ee000700 1 Processor -- start 2024-03-28T22:35:01.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.340+0000 7ff3ee000700 1 -- start start 2024-03-28T22:35:01.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.340+0000 7ff3ee000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff3e8104440 0x7ff3e81a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:01.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.340+0000 7ff3ee000700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3e8104e00 0x7ff3e81a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:01.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.341+0000 7ff3ee000700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3e810a4d0 0x7ff3e819a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:01.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.341+0000 7ff3ee000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e8114780 con 0x7ff3e8104440 2024-03-28T22:35:01.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.341+0000 7ff3ee000700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e8114600 con 0x7ff3e810a4d0 2024-03-28T22:35:01.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.341+0000 7ff3ee000700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e8114900 con 0x7ff3e8104e00 2024-03-28T22:35:01.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.341+0000 7ff3e7e00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3e810a4d0 0x7ff3e819a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:01.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.341+0000 7ff3e6a00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3e8104e00 0x7ff3e81a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:01.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.341+0000 7ff3e7e00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3e810a4d0 0x7ff3e819a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:36446/0 (socket says 172.21.15.43:36446) 2024-03-28T22:35:01.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.341+0000 7ff3e7e00700 1 -- 172.21.15.43:0/2558223242 learned_addr learned my addr 172.21.15.43:0/2558223242 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:35:01.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.341+0000 7ff3e7400700 1 --2- 172.21.15.43:0/2558223242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff3e8104440 0x7ff3e81a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:01.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.341+0000 7ff3e6a00700 1 --2- 172.21.15.43:0/2558223242 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3e8104e00 0x7ff3e81a0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:51076/0 (socket says 172.21.15.43:51076) 2024-03-28T22:35:01.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.341+0000 7ff3e7e00700 1 -- 172.21.15.43:0/2558223242 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3e8104e00 msgr2=0x7ff3e81a0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:01.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.341+0000 7ff3e7e00700 1 --2- 172.21.15.43:0/2558223242 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3e8104e00 0x7ff3e81a0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:01.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.341+0000 7ff3e7e00700 1 -- 172.21.15.43:0/2558223242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff3e8104440 msgr2=0x7ff3e81a07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:01.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.341+0000 7ff3e7e00700 1 --2- 172.21.15.43:0/2558223242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff3e8104440 0x7ff3e81a07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:01.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.341+0000 7ff3e7e00700 1 -- 172.21.15.43:0/2558223242 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3e819b100 con 0x7ff3e810a4d0 2024-03-28T22:35:01.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.341+0000 7ff3e7400700 1 --2- 172.21.15.43:0/2558223242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff3e8104440 0x7ff3e81a07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:35:01.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.342+0000 7ff3e7e00700 1 --2- 172.21.15.43:0/2558223242 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3e810a4d0 0x7ff3e819a9a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff3e000b070 tx=0x7ff3e0004cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:01.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.342+0000 7ff3dfe00700 1 -- 172.21.15.43:0/2558223242 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e0021070 con 0x7ff3e810a4d0 2024-03-28T22:35:01.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.342+0000 7ff3dfe00700 1 -- 172.21.15.43:0/2558223242 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3e0003c00 con 0x7ff3e810a4d0 2024-03-28T22:35:01.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.342+0000 7ff3ee000700 1 -- 172.21.15.43:0/2558223242 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3e819b390 con 0x7ff3e810a4d0 2024-03-28T22:35:01.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.342+0000 7ff3dfe00700 1 -- 172.21.15.43:0/2558223242 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e0004850 con 0x7ff3e810a4d0 2024-03-28T22:35:01.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.342+0000 7ff3ee000700 1 -- 172.21.15.43:0/2558223242 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3e81a76b0 con 0x7ff3e810a4d0 2024-03-28T22:35:01.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.343+0000 7ff3ee000700 1 -- 172.21.15.43:0/2558223242 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3e8068f00 con 0x7ff3e810a4d0 2024-03-28T22:35:01.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.345+0000 7ff3dfe00700 1 -- 172.21.15.43:0/2558223242 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7ff3e0007980 con 0x7ff3e810a4d0 2024-03-28T22:35:01.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.346+0000 7ff3dfe00700 1 --2- 172.21.15.43:0/2558223242 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff3cc0a1f20 0x7ff3cc0a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:01.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.346+0000 7ff3dfe00700 1 -- 172.21.15.43:0/2558223242 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7ff3e00cf250 con 0x7ff3e810a4d0 2024-03-28T22:35:01.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.347+0000 7ff3e7400700 1 --2- 172.21.15.43:0/2558223242 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff3cc0a1f20 0x7ff3cc0a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:01.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.347+0000 7ff3e7400700 1 --2- 172.21.15.43:0/2558223242 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff3cc0a1f20 0x7ff3cc0a43e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff3e819bae0 tx=0x7ff3d8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:01.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.355+0000 7ff3dfe00700 1 -- 172.21.15.43:0/2558223242 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff3e009c1f0 con 0x7ff3e810a4d0 2024-03-28T22:35:01.445 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:01 smithi043 ceph-mon[31823]: pgmap v370: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-28T22:35:01.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.588+0000 7ff3ee000700 1 -- 172.21.15.43:0/2558223242 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7ff3e81a7880 con 0x7ff3e810a4d0 2024-03-28T22:35:01.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.588+0000 7ff3dfe00700 1 -- 172.21.15.43:0/2558223242 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff3e009baf0 con 0x7ff3e810a4d0 2024-03-28T22:35:01.590 INFO:teuthology.orchestra.run.smithi043.stdout:197568495676 2024-03-28T22:35:01.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.591+0000 7ff3ee000700 1 -- 172.21.15.43:0/2558223242 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff3cc0a1f20 msgr2=0x7ff3cc0a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:01.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.591+0000 7ff3ee000700 1 --2- 172.21.15.43:0/2558223242 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff3cc0a1f20 0x7ff3cc0a43e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff3e819bae0 tx=0x7ff3d8009260 comp rx=0 tx=0).stop 2024-03-28T22:35:01.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.591+0000 7ff3ee000700 1 -- 172.21.15.43:0/2558223242 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3e810a4d0 msgr2=0x7ff3e819a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:01.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.591+0000 7ff3ee000700 1 --2- 172.21.15.43:0/2558223242 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3e810a4d0 0x7ff3e819a9a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff3e000b070 tx=0x7ff3e0004cf0 comp rx=0 tx=0).stop 2024-03-28T22:35:01.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.592+0000 7ff3ee000700 1 -- 172.21.15.43:0/2558223242 shutdown_connections 2024-03-28T22:35:01.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.592+0000 7ff3ee000700 1 --2- 172.21.15.43:0/2558223242 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff3cc0a1f20 0x7ff3cc0a43e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:01.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.592+0000 7ff3ee000700 1 --2- 172.21.15.43:0/2558223242 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3e810a4d0 0x7ff3e819a9a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:01.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.592+0000 7ff3ee000700 1 --2- 172.21.15.43:0/2558223242 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff3e8104440 0x7ff3e81a07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:01.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.592+0000 7ff3ee000700 1 --2- 172.21.15.43:0/2558223242 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff3e8104e00 0x7ff3e81a0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:01.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.592+0000 7ff3ee000700 1 -- 172.21.15.43:0/2558223242 >> 172.21.15.43:0/2558223242 conn(0x7ff3e80fff10 msgr2=0x7ff3e81003b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:01.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.593+0000 7ff3ee000700 1 -- 172.21.15.43:0/2558223242 shutdown_connections 2024-03-28T22:35:01.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.594+0000 7ff3ee000700 1 -- 172.21.15.43:0/2558223242 wait complete. 2024-03-28T22:35:01.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.622+0000 7f3213a00700 1 -- 172.21.15.43:0/2506875662 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f320c074630 msgr2=0x7f320c074a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:01.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.622+0000 7f3213a00700 1 --2- 172.21.15.43:0/2506875662 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f320c074630 0x7f320c074a10 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3200009990 tx=0x7f3200009ca0 comp rx=0 tx=0).stop 2024-03-28T22:35:01.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.622+0000 7f3213a00700 1 -- 172.21.15.43:0/2506875662 shutdown_connections 2024-03-28T22:35:01.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.622+0000 7f3213a00700 1 --2- 172.21.15.43:0/2506875662 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f320c10eae0 0x7f320c110f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:01.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.622+0000 7f3213a00700 1 --2- 172.21.15.43:0/2506875662 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f320c074630 0x7f320c074a10 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:01.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.622+0000 7f3213a00700 1 --2- 172.21.15.43:0/2506875662 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f320c074f50 0x7f320c10e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:01.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.622+0000 7f3213a00700 1 -- 172.21.15.43:0/2506875662 >> 172.21.15.43:0/2506875662 conn(0x7f320c0fdd10 msgr2=0x7f320c100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:01.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.623+0000 7f3213a00700 1 -- 172.21.15.43:0/2506875662 shutdown_connections 2024-03-28T22:35:01.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.623+0000 7f3213a00700 1 -- 172.21.15.43:0/2506875662 wait complete. 2024-03-28T22:35:01.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.624+0000 7f3213a00700 1 Processor -- start 2024-03-28T22:35:01.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.624+0000 7f3213a00700 1 -- start start 2024-03-28T22:35:01.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.624+0000 7f3213a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f320c074630 0x7f320c1a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:01.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.624+0000 7f3213a00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f320c074f50 0x7f320c1a0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:01.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.624+0000 7f3213a00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f320c10eae0 0x7f320c19aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:01.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.624+0000 7f3213a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f320c1147f0 con 0x7f320c074630 2024-03-28T22:35:01.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.624+0000 7f3213a00700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f320c114670 con 0x7f320c10eae0 2024-03-28T22:35:01.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.624+0000 7f3213a00700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f320c114970 con 0x7f320c074f50 2024-03-28T22:35:01.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.625+0000 7f3211000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f320c074630 0x7f320c1a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:01.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.625+0000 7f3211a00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f320c10eae0 0x7f320c19aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:01.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.625+0000 7f320be00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f320c074f50 0x7f320c1a0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:01.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.625+0000 7f3211000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f320c074630 0x7f320c1a08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46028/0 (socket says 172.21.15.43:46028) 2024-03-28T22:35:01.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.625+0000 7f3211a00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f320c10eae0 0x7f320c19aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:36464/0 (socket says 172.21.15.43:36464) 2024-03-28T22:35:01.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.625+0000 7f3211000700 1 -- 172.21.15.43:0/2403255974 learned_addr learned my addr 172.21.15.43:0/2403255974 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:35:01.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.625+0000 7f3211a00700 1 -- 172.21.15.43:0/2403255974 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f320c074f50 msgr2=0x7f320c1a0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:01.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.625+0000 7f3211a00700 1 --2- 172.21.15.43:0/2403255974 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f320c074f50 0x7f320c1a0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:01.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.625+0000 7f3211a00700 1 -- 172.21.15.43:0/2403255974 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f320c074630 msgr2=0x7f320c1a08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:01.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.625+0000 7f3211a00700 1 --2- 172.21.15.43:0/2403255974 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f320c074630 0x7f320c1a08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:01.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.625+0000 7f3211a00700 1 -- 172.21.15.43:0/2403255974 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f320c19b210 con 0x7f320c10eae0 2024-03-28T22:35:01.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.626+0000 7f3211a00700 1 --2- 172.21.15.43:0/2403255974 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f320c10eae0 0x7f320c19aab0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3204007ee0 tx=0x7f320400d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:01.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.626+0000 7f3209600700 1 -- 172.21.15.43:0/2403255974 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3204004db0 con 0x7f320c10eae0 2024-03-28T22:35:01.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.626+0000 7f3209600700 1 -- 172.21.15.43:0/2403255974 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3204009e20 con 0x7f320c10eae0 2024-03-28T22:35:01.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.626+0000 7f3213a00700 1 -- 172.21.15.43:0/2403255974 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f320c19b500 con 0x7f320c10eae0 2024-03-28T22:35:01.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.626+0000 7f3209600700 1 -- 172.21.15.43:0/2403255974 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3204015400 con 0x7f320c10eae0 2024-03-28T22:35:01.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.626+0000 7f3213a00700 1 -- 172.21.15.43:0/2403255974 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f320c1a77c0 con 0x7f320c10eae0 2024-03-28T22:35:01.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.627+0000 7f3213a00700 1 -- 172.21.15.43:0/2403255974 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f320c068f00 con 0x7f320c10eae0 2024-03-28T22:35:01.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.629+0000 7f3209600700 1 -- 172.21.15.43:0/2403255974 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f320400db20 con 0x7f320c10eae0 2024-03-28T22:35:01.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.630+0000 7f3209600700 1 --2- 172.21.15.43:0/2403255974 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f31f40a1f20 0x7f31f40a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:01.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.631+0000 7f3209600700 1 -- 172.21.15.43:0/2403255974 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7f32040cdb90 con 0x7f320c10eae0 2024-03-28T22:35:01.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.631+0000 7f3211000700 1 --2- 172.21.15.43:0/2403255974 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f31f40a1f20 0x7f31f40a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:01.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.632+0000 7f3211000700 1 --2- 172.21.15.43:0/2403255974 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f31f40a1f20 0x7f31f40a43e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3200006010 tx=0x7f3200005ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:01.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.634+0000 7f3209600700 1 -- 172.21.15.43:0/2403255974 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3204010070 con 0x7f320c10eae0 2024-03-28T22:35:01.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:01 smithi172 ceph-mon[36566]: pgmap v370: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-28T22:35:01.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:01 smithi082 ceph-mon[36471]: pgmap v370: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-28T22:35:01.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.872+0000 7f3213a00700 1 -- 172.21.15.43:0/2403255974 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f320c02db30 con 0x7f320c10eae0 2024-03-28T22:35:01.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.873+0000 7f3209600700 1 -- 172.21.15.43:0/2403255974 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f320409aaa0 con 0x7f320c10eae0 2024-03-28T22:35:01.871 INFO:teuthology.orchestra.run.smithi043.stdout:317827579941 2024-03-28T22:35:01.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.876+0000 7f3213a00700 1 -- 172.21.15.43:0/2403255974 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f31f40a1f20 msgr2=0x7f31f40a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:01.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.876+0000 7f3213a00700 1 --2- 172.21.15.43:0/2403255974 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f31f40a1f20 0x7f31f40a43e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3200006010 tx=0x7f3200005ab0 comp rx=0 tx=0).stop 2024-03-28T22:35:01.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.876+0000 7f3213a00700 1 -- 172.21.15.43:0/2403255974 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f320c10eae0 msgr2=0x7f320c19aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:01.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.876+0000 7f3213a00700 1 --2- 172.21.15.43:0/2403255974 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f320c10eae0 0x7f320c19aab0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3204007ee0 tx=0x7f320400d3b0 comp rx=0 tx=0).stop 2024-03-28T22:35:01.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.876+0000 7f3213a00700 1 -- 172.21.15.43:0/2403255974 shutdown_connections 2024-03-28T22:35:01.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.876+0000 7f3213a00700 1 --2- 172.21.15.43:0/2403255974 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f31f40a1f20 0x7f31f40a43e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:01.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.876+0000 7f3213a00700 1 --2- 172.21.15.43:0/2403255974 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f320c10eae0 0x7f320c19aab0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:01.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.876+0000 7f3213a00700 1 --2- 172.21.15.43:0/2403255974 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f320c074630 0x7f320c1a08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:01.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.876+0000 7f3213a00700 1 --2- 172.21.15.43:0/2403255974 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f320c074f50 0x7f320c1a0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:01.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.876+0000 7f3213a00700 1 -- 172.21.15.43:0/2403255974 >> 172.21.15.43:0/2403255974 conn(0x7f320c0fdd10 msgr2=0x7f320c0fe9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:01.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.876+0000 7f3213a00700 1 -- 172.21.15.43:0/2403255974 shutdown_connections 2024-03-28T22:35:01.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:01.876+0000 7f3213a00700 1 -- 172.21.15.43:0/2403255974 wait complete. 2024-03-28T22:35:02.474 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:02 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2558223242' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T22:35:02.474 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:02 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2403255974' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T22:35:02.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:02 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2558223242' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T22:35:02.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:02 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2403255974' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T22:35:02.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:02 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2558223242' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T22:35:02.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:02 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2403255974' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T22:35:03.496 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:03 smithi043 ceph-mon[31823]: pgmap v371: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:35:03.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:03 smithi172 ceph-mon[36566]: pgmap v371: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:35:03.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:03 smithi082 ceph-mon[36471]: pgmap v371: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:35:05.344 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:05 smithi043 ceph-mon[31823]: pgmap v372: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:35:05.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:05 smithi172 ceph-mon[36566]: pgmap v372: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:35:05.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:05 smithi082 ceph-mon[36471]: pgmap v372: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:35:06.555 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:35:07.366 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:07 smithi043 ceph-mon[31823]: pgmap v373: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-28T22:35:07.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:07 smithi172 ceph-mon[36566]: pgmap v373: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-28T22:35:07.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:07 smithi082 ceph-mon[36471]: pgmap v373: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-28T22:35:07.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.866+0000 7fbffc400700 1 -- 172.21.15.43:0/4060584021 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbff4074470 msgr2=0x7fbff4074850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:07.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.866+0000 7fbffc400700 1 --2- 172.21.15.43:0/4060584021 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbff4074470 0x7fbff4074850 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8009990 tx=0x7fbfe8009ca0 comp rx=0 tx=0).stop 2024-03-28T22:35:07.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.866+0000 7fbffc400700 1 -- 172.21.15.43:0/4060584021 shutdown_connections 2024-03-28T22:35:07.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.866+0000 7fbffc400700 1 --2- 172.21.15.43:0/4060584021 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbff410ec40 0x7fbff4111030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:07.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.866+0000 7fbffc400700 1 --2- 172.21.15.43:0/4060584021 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbff4074470 0x7fbff4074850 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:07.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.866+0000 7fbffc400700 1 --2- 172.21.15.43:0/4060584021 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbff4074d90 0x7fbff410e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:07.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.866+0000 7fbffc400700 1 -- 172.21.15.43:0/4060584021 >> 172.21.15.43:0/4060584021 conn(0x7fbff4079370 msgr2=0x7fbff4079780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:07.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.866+0000 7fbffc400700 1 -- 172.21.15.43:0/4060584021 shutdown_connections 2024-03-28T22:35:07.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.866+0000 7fbffc400700 1 -- 172.21.15.43:0/4060584021 wait complete. 2024-03-28T22:35:07.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.867+0000 7fbffc400700 1 Processor -- start 2024-03-28T22:35:07.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.868+0000 7fbffc400700 1 -- start start 2024-03-28T22:35:07.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.868+0000 7fbffc400700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbff4074470 0x7fbff41a0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:07.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.868+0000 7fbffc400700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbff4074d90 0x7fbff41a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:07.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.868+0000 7fbffc400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbff410ec40 0x7fbff419ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:07.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.868+0000 7fbffc400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbff41147d0 con 0x7fbff410ec40 2024-03-28T22:35:07.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.868+0000 7fbffc400700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fbff4114650 con 0x7fbff4074d90 2024-03-28T22:35:07.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.868+0000 7fbffc400700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fbff4114950 con 0x7fbff4074470 2024-03-28T22:35:07.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.868+0000 7fbff9a00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbff4074470 0x7fbff41a0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:07.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.868+0000 7fbff9000700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbff4074d90 0x7fbff41a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:07.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.868+0000 7fbff9a00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbff4074470 0x7fbff41a0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:55070/0 (socket says 172.21.15.43:55070) 2024-03-28T22:35:07.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.868+0000 7fbff9a00700 1 -- 172.21.15.43:0/854748777 learned_addr learned my addr 172.21.15.43:0/854748777 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:35:07.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.869+0000 7fbffa400700 1 --2- 172.21.15.43:0/854748777 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbff410ec40 0x7fbff419ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:07.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.869+0000 7fbff9a00700 1 -- 172.21.15.43:0/854748777 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbff4074d90 msgr2=0x7fbff41a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:07.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.869+0000 7fbff9a00700 1 --2- 172.21.15.43:0/854748777 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbff4074d90 0x7fbff41a0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:07.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.869+0000 7fbff9a00700 1 -- 172.21.15.43:0/854748777 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbff410ec40 msgr2=0x7fbff419ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:07.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.869+0000 7fbff9a00700 1 --2- 172.21.15.43:0/854748777 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbff410ec40 0x7fbff419ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:07.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.869+0000 7fbff9a00700 1 -- 172.21.15.43:0/854748777 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff419b3f0 con 0x7fbff4074470 2024-03-28T22:35:07.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.869+0000 7fbff9a00700 1 --2- 172.21.15.43:0/854748777 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbff4074470 0x7fbff41a0a40 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8000c00 tx=0x7fbfe8003890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:07.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.870+0000 7fbff2000700 1 -- 172.21.15.43:0/854748777 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8027070 con 0x7fbff4074470 2024-03-28T22:35:07.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.870+0000 7fbff2000700 1 -- 172.21.15.43:0/854748777 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbfe8003e40 con 0x7fbff4074470 2024-03-28T22:35:07.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.870+0000 7fbffc400700 1 -- 172.21.15.43:0/854748777 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbff419b680 con 0x7fbff4074470 2024-03-28T22:35:07.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.870+0000 7fbff2000700 1 -- 172.21.15.43:0/854748777 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8022c90 con 0x7fbff4074470 2024-03-28T22:35:07.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.870+0000 7fbffc400700 1 -- 172.21.15.43:0/854748777 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbff41a77f0 con 0x7fbff4074470 2024-03-28T22:35:07.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.871+0000 7fbffc400700 1 -- 172.21.15.43:0/854748777 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbff4075b60 con 0x7fbff4074470 2024-03-28T22:35:07.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.873+0000 7fbff2000700 1 -- 172.21.15.43:0/854748777 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fbfe802a070 con 0x7fbff4074470 2024-03-28T22:35:07.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.874+0000 7fbff2000700 1 --2- 172.21.15.43:0/854748777 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fbfdc0a1ea0 0x7fbfdc0a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:07.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.875+0000 7fbff2000700 1 -- 172.21.15.43:0/854748777 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7fbfe80cf0a0 con 0x7fbff4074470 2024-03-28T22:35:07.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.875+0000 7fbff9000700 1 --2- 172.21.15.43:0/854748777 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fbfdc0a1ea0 0x7fbfdc0a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:07.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.875+0000 7fbff9000700 1 --2- 172.21.15.43:0/854748777 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fbfdc0a1ea0 0x7fbfdc0a4360 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbff419c260 tx=0x7fbfe000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:07.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:07.883+0000 7fbff2000700 1 -- 172.21.15.43:0/854748777 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbfe805e280 con 0x7fbff4074470 2024-03-28T22:35:08.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:08.111+0000 7fbffc400700 1 -- 172.21.15.43:0/854748777 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fbff402db30 con 0x7fbff4074470 2024-03-28T22:35:08.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:08.112+0000 7fbff2000700 1 -- 172.21.15.43:0/854748777 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fbfe8013070 con 0x7fbff4074470 2024-03-28T22:35:08.111 INFO:teuthology.orchestra.run.smithi043.stdout:261993005106 2024-03-28T22:35:08.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:08.115+0000 7fbffc400700 1 -- 172.21.15.43:0/854748777 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fbfdc0a1ea0 msgr2=0x7fbfdc0a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:08.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:08.116+0000 7fbffc400700 1 --2- 172.21.15.43:0/854748777 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fbfdc0a1ea0 0x7fbfdc0a4360 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbff419c260 tx=0x7fbfe000b410 comp rx=0 tx=0).stop 2024-03-28T22:35:08.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:08.116+0000 7fbffc400700 1 -- 172.21.15.43:0/854748777 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbff4074470 msgr2=0x7fbff41a0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:08.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:08.116+0000 7fbffc400700 1 --2- 172.21.15.43:0/854748777 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbff4074470 0x7fbff41a0a40 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8000c00 tx=0x7fbfe8003890 comp rx=0 tx=0).stop 2024-03-28T22:35:08.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:08.116+0000 7fbffc400700 1 -- 172.21.15.43:0/854748777 shutdown_connections 2024-03-28T22:35:08.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:08.116+0000 7fbffc400700 1 --2- 172.21.15.43:0/854748777 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fbfdc0a1ea0 0x7fbfdc0a4360 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:08.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:08.116+0000 7fbffc400700 1 --2- 172.21.15.43:0/854748777 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbff410ec40 0x7fbff419ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:08.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:08.116+0000 7fbffc400700 1 --2- 172.21.15.43:0/854748777 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbff4074470 0x7fbff41a0a40 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:08.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:08.116+0000 7fbffc400700 1 --2- 172.21.15.43:0/854748777 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbff4074d90 0x7fbff41a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:08.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:08.116+0000 7fbffc400700 1 -- 172.21.15.43:0/854748777 >> 172.21.15.43:0/854748777 conn(0x7fbff4079370 msgr2=0x7fbff40ff4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:08.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:08.116+0000 7fbffc400700 1 -- 172.21.15.43:0/854748777 shutdown_connections 2024-03-28T22:35:08.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:08.116+0000 7fbffc400700 1 -- 172.21.15.43:0/854748777 wait complete. 2024-03-28T22:35:08.197 INFO:tasks.cephadm.ceph_manager.ceph:need seq 317827579936 got 317827579941 for osd.10 2024-03-28T22:35:08.198 DEBUG:teuthology.parallel:result is None 2024-03-28T22:35:08.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:08 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/854748777' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T22:35:08.546 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:08 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/854748777' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T22:35:08.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:08 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/854748777' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T22:35:08.749 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495671 got 197568495676 for osd.6 2024-03-28T22:35:08.749 DEBUG:teuthology.parallel:result is None 2024-03-28T22:35:09.467 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:09 smithi043 ceph-mon[31823]: pgmap v374: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-03-28T22:35:09.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:09 smithi172 ceph-mon[36566]: pgmap v374: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-03-28T22:35:09.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:09 smithi082 ceph-mon[36471]: pgmap v374: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-03-28T22:35:10.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.451+0000 7ff58ba00700 1 -- 172.21.15.43:0/3491420993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff58410a4c0 msgr2=0x7ff58410a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:10.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.451+0000 7ff58ba00700 1 --2- 172.21.15.43:0/3491420993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff58410a4c0 0x7ff58410a8a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff578009990 tx=0x7ff578009ca0 comp rx=0 tx=0).stop 2024-03-28T22:35:10.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.451+0000 7ff58ba00700 1 -- 172.21.15.43:0/3491420993 shutdown_connections 2024-03-28T22:35:10.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.451+0000 7ff58ba00700 1 --2- 172.21.15.43:0/3491420993 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff584104df0 0x7ff584111300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:10.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.451+0000 7ff58ba00700 1 --2- 172.21.15.43:0/3491420993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff58410a4c0 0x7ff58410a8a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:10.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.451+0000 7ff58ba00700 1 --2- 172.21.15.43:0/3491420993 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff584104430 0x7ff5841048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:10.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.451+0000 7ff58ba00700 1 -- 172.21.15.43:0/3491420993 >> 172.21.15.43:0/3491420993 conn(0x7ff5840fff20 msgr2=0x7ff584102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:10.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.451+0000 7ff58ba00700 1 -- 172.21.15.43:0/3491420993 shutdown_connections 2024-03-28T22:35:10.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.451+0000 7ff58ba00700 1 -- 172.21.15.43:0/3491420993 wait complete. 2024-03-28T22:35:10.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.452+0000 7ff58ba00700 1 Processor -- start 2024-03-28T22:35:10.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.453+0000 7ff58ba00700 1 -- start start 2024-03-28T22:35:10.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.453+0000 7ff58ba00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff584104430 0x7ff5841a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:10.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.453+0000 7ff58ba00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff584104df0 0x7ff5841a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:10.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.453+0000 7ff58ba00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff58410a4c0 0x7ff58419aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:10.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.453+0000 7ff58ba00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff584114790 con 0x7ff584104df0 2024-03-28T22:35:10.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.453+0000 7ff58ba00700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ff584114610 con 0x7ff584104430 2024-03-28T22:35:10.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.453+0000 7ff58ba00700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7ff584114910 con 0x7ff58410a4c0 2024-03-28T22:35:10.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.453+0000 7ff589000700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff584104430 0x7ff5841a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:10.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.453+0000 7ff589000700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff584104430 0x7ff5841a0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:49704/0 (socket says 172.21.15.43:49704) 2024-03-28T22:35:10.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.453+0000 7ff589000700 1 -- 172.21.15.43:0/2672720996 learned_addr learned my addr 172.21.15.43:0/2672720996 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:35:10.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.454+0000 7ff589a00700 1 --2- 172.21.15.43:0/2672720996 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff58410a4c0 0x7ff58419aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:10.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.454+0000 7ff583e00700 1 --2- 172.21.15.43:0/2672720996 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff584104df0 0x7ff5841a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:10.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.454+0000 7ff589000700 1 -- 172.21.15.43:0/2672720996 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff58410a4c0 msgr2=0x7ff58419aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:10.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.454+0000 7ff589000700 1 --2- 172.21.15.43:0/2672720996 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff58410a4c0 0x7ff58419aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:10.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.454+0000 7ff589000700 1 -- 172.21.15.43:0/2672720996 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff584104df0 msgr2=0x7ff5841a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:10.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.454+0000 7ff589000700 1 --2- 172.21.15.43:0/2672720996 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff584104df0 0x7ff5841a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:10.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.454+0000 7ff589000700 1 -- 172.21.15.43:0/2672720996 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff58419b1b0 con 0x7ff584104430 2024-03-28T22:35:10.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.454+0000 7ff589000700 1 --2- 172.21.15.43:0/2672720996 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff584104430 0x7ff5841a0840 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff578005910 tx=0x7ff57800bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:10.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.454+0000 7ff581600700 1 -- 172.21.15.43:0/2672720996 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff578027070 con 0x7ff584104430 2024-03-28T22:35:10.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.455+0000 7ff581600700 1 -- 172.21.15.43:0/2672720996 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff578003d70 con 0x7ff584104430 2024-03-28T22:35:10.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.455+0000 7ff58ba00700 1 -- 172.21.15.43:0/2672720996 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff58419b440 con 0x7ff584104430 2024-03-28T22:35:10.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.455+0000 7ff581600700 1 -- 172.21.15.43:0/2672720996 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff57800f6c0 con 0x7ff584104430 2024-03-28T22:35:10.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.455+0000 7ff58ba00700 1 -- 172.21.15.43:0/2672720996 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5841a7710 con 0x7ff584104430 2024-03-28T22:35:10.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.456+0000 7ff58ba00700 1 -- 172.21.15.43:0/2672720996 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff58410c9c0 con 0x7ff584104430 2024-03-28T22:35:10.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.458+0000 7ff581600700 1 -- 172.21.15.43:0/2672720996 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7ff57802a070 con 0x7ff584104430 2024-03-28T22:35:10.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.459+0000 7ff581600700 1 --2- 172.21.15.43:0/2672720996 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff56c0a1f20 0x7ff56c0a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:10.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.459+0000 7ff581600700 1 -- 172.21.15.43:0/2672720996 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7ff5780cf330 con 0x7ff584104430 2024-03-28T22:35:10.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.459+0000 7ff583e00700 1 --2- 172.21.15.43:0/2672720996 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff56c0a1f20 0x7ff56c0a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:10.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.460+0000 7ff583e00700 1 --2- 172.21.15.43:0/2672720996 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff56c0a1f20 0x7ff56c0a43e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff584109af0 tx=0x7ff570009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:10.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.463+0000 7ff581600700 1 -- 172.21.15.43:0/2672720996 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff57805e280 con 0x7ff584104430 2024-03-28T22:35:10.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.701+0000 7ff58ba00700 1 -- 172.21.15.43:0/2672720996 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7ff58419bf30 con 0x7ff584104430 2024-03-28T22:35:10.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.701+0000 7ff581600700 1 -- 172.21.15.43:0/2672720996 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff578022e20 con 0x7ff584104430 2024-03-28T22:35:10.700 INFO:teuthology.orchestra.run.smithi043.stdout:141733920841 2024-03-28T22:35:10.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.704+0000 7ff58ba00700 1 -- 172.21.15.43:0/2672720996 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff56c0a1f20 msgr2=0x7ff56c0a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:10.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.704+0000 7ff58ba00700 1 --2- 172.21.15.43:0/2672720996 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff56c0a1f20 0x7ff56c0a43e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff584109af0 tx=0x7ff570009260 comp rx=0 tx=0).stop 2024-03-28T22:35:10.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.704+0000 7ff58ba00700 1 -- 172.21.15.43:0/2672720996 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff584104430 msgr2=0x7ff5841a0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:10.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.704+0000 7ff58ba00700 1 --2- 172.21.15.43:0/2672720996 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff584104430 0x7ff5841a0840 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff578005910 tx=0x7ff57800bab0 comp rx=0 tx=0).stop 2024-03-28T22:35:10.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.704+0000 7ff58ba00700 1 -- 172.21.15.43:0/2672720996 shutdown_connections 2024-03-28T22:35:10.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.704+0000 7ff58ba00700 1 --2- 172.21.15.43:0/2672720996 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff56c0a1f20 0x7ff56c0a43e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:10.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.704+0000 7ff58ba00700 1 --2- 172.21.15.43:0/2672720996 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff58410a4c0 0x7ff58419aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:10.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.704+0000 7ff58ba00700 1 --2- 172.21.15.43:0/2672720996 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff584104430 0x7ff5841a0840 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:10.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.704+0000 7ff58ba00700 1 --2- 172.21.15.43:0/2672720996 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff584104df0 0x7ff5841a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:10.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.704+0000 7ff58ba00700 1 -- 172.21.15.43:0/2672720996 >> 172.21.15.43:0/2672720996 conn(0x7ff5840fff20 msgr2=0x7ff584100560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:10.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.704+0000 7ff58ba00700 1 -- 172.21.15.43:0/2672720996 shutdown_connections 2024-03-28T22:35:10.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:10.704+0000 7ff58ba00700 1 -- 172.21.15.43:0/2672720996 wait complete. 2024-03-28T22:35:11.295 INFO:tasks.cephadm.ceph_manager.ceph:need seq 261993005100 got 261993005106 for osd.8 2024-03-28T22:35:11.295 DEBUG:teuthology.parallel:result is None 2024-03-28T22:35:11.370 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:11 smithi043 ceph-mon[31823]: pgmap v375: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-28T22:35:11.371 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:11 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2672720996' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T22:35:11.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:11 smithi172 ceph-mon[36566]: pgmap v375: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-28T22:35:11.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:11 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2672720996' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T22:35:11.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:11 smithi082 ceph-mon[36471]: pgmap v375: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-28T22:35:11.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:11 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2672720996' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T22:35:12.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.753+0000 7f4cb2a00700 1 -- 172.21.15.43:0/677697209 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4cac1075c0 msgr2=0x7f4cac10f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:12.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.753+0000 7f4cb2a00700 1 --2- 172.21.15.43:0/677697209 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4cac1075c0 0x7f4cac10f140 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4009f90 tx=0x7f4ca400b330 comp rx=0 tx=0).stop 2024-03-28T22:35:12.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.753+0000 7f4cb2a00700 1 -- 172.21.15.43:0/677697209 shutdown_connections 2024-03-28T22:35:12.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.753+0000 7f4cb2a00700 1 --2- 172.21.15.43:0/677697209 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4cac1075c0 0x7f4cac10f140 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:12.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.753+0000 7f4cb2a00700 1 --2- 172.21.15.43:0/677697209 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cac06a9e0 0x7f4cac06adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:12.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.753+0000 7f4cb2a00700 1 --2- 172.21.15.43:0/677697209 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4cac06b390 0x7f4cac106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:12.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.753+0000 7f4cb2a00700 1 -- 172.21.15.43:0/677697209 >> 172.21.15.43:0/677697209 conn(0x7f4cac077190 msgr2=0x7f4cac0775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:12.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.753+0000 7f4cb2a00700 1 -- 172.21.15.43:0/677697209 shutdown_connections 2024-03-28T22:35:12.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.753+0000 7f4cb2a00700 1 -- 172.21.15.43:0/677697209 wait complete. 2024-03-28T22:35:12.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.754+0000 7f4cb2a00700 1 Processor -- start 2024-03-28T22:35:12.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.754+0000 7f4cb2a00700 1 -- start start 2024-03-28T22:35:12.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.755+0000 7f4cb2a00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4cac06a9e0 0x7f4cac19e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:12.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.755+0000 7f4cb2a00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cac06b390 0x7f4cac19ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:12.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.755+0000 7f4cb2a00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4cac1075c0 0x7f4cac198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:12.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.755+0000 7f4cb2a00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4cac1125b0 con 0x7f4cac06b390 2024-03-28T22:35:12.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.755+0000 7f4cb2a00700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4cac112430 con 0x7f4cac06a9e0 2024-03-28T22:35:12.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.755+0000 7f4cb2a00700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4cac112730 con 0x7f4cac1075c0 2024-03-28T22:35:12.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.755+0000 7f4cabe00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4cac06a9e0 0x7f4cac19e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:12.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.755+0000 7f4cabe00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4cac06a9e0 0x7f4cac19e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:49722/0 (socket says 172.21.15.43:49722) 2024-03-28T22:35:12.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.755+0000 7f4cabe00700 1 -- 172.21.15.43:0/1337968086 learned_addr learned my addr 172.21.15.43:0/1337968086 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:35:12.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.755+0000 7f4cb0a00700 1 --2- 172.21.15.43:0/1337968086 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4cac1075c0 0x7f4cac198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:12.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.755+0000 7f4ca3400700 1 --2- 172.21.15.43:0/1337968086 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cac06b390 0x7f4cac19ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:12.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.756+0000 7f4cabe00700 1 -- 172.21.15.43:0/1337968086 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4cac1075c0 msgr2=0x7f4cac198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:12.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.756+0000 7f4cabe00700 1 --2- 172.21.15.43:0/1337968086 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4cac1075c0 0x7f4cac198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:12.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.756+0000 7f4cabe00700 1 -- 172.21.15.43:0/1337968086 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cac06b390 msgr2=0x7f4cac19ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:12.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.756+0000 7f4cabe00700 1 --2- 172.21.15.43:0/1337968086 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cac06b390 0x7f4cac19ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:12.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.756+0000 7f4cabe00700 1 -- 172.21.15.43:0/1337968086 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cac198fd0 con 0x7f4cac06a9e0 2024-03-28T22:35:12.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.756+0000 7f4cb0a00700 1 --2- 172.21.15.43:0/1337968086 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4cac1075c0 0x7f4cac198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:35:12.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.756+0000 7f4ca3400700 1 --2- 172.21.15.43:0/1337968086 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cac06b390 0x7f4cac19ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:35:12.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.756+0000 7f4cabe00700 1 --2- 172.21.15.43:0/1337968086 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4cac06a9e0 0x7f4cac19e6b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c00e9f0 tx=0x7f4c9c00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:12.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.756+0000 7f4ca9600700 1 -- 172.21.15.43:0/1337968086 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c004a30 con 0x7f4cac06a9e0 2024-03-28T22:35:12.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.756+0000 7f4ca9600700 1 -- 172.21.15.43:0/1337968086 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c9c005500 con 0x7f4cac06a9e0 2024-03-28T22:35:12.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.756+0000 7f4ca9600700 1 -- 172.21.15.43:0/1337968086 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c00cda0 con 0x7f4cac06a9e0 2024-03-28T22:35:12.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.756+0000 7f4cb2a00700 1 -- 172.21.15.43:0/1337968086 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cac1992c0 con 0x7f4cac06a9e0 2024-03-28T22:35:12.154 INFO:teuthology.orchestra.run.smithi043.stdout:77309411415 2024-03-28T22:35:12.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.757+0000 7f4cb2a00700 1 -- 172.21.15.43:0/1337968086 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cac1a5580 con 0x7f4cac06a9e0 2024-03-28T22:35:12.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.757+0000 7f4cb2a00700 1 -- 172.21.15.43:0/1337968086 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cac068f00 con 0x7f4cac06a9e0 2024-03-28T22:35:12.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.760+0000 7f4ca9600700 1 -- 172.21.15.43:0/1337968086 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f4c9c005020 con 0x7f4cac06a9e0 2024-03-28T22:35:12.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.761+0000 7f4ca9600700 1 --2- 172.21.15.43:0/1337968086 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4c900a1f20 0x7f4c900a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:12.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.761+0000 7f4ca9600700 1 -- 172.21.15.43:0/1337968086 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7f4c9c0ce5e0 con 0x7f4cac06a9e0 2024-03-28T22:35:12.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.761+0000 7f4ca3400700 1 --2- 172.21.15.43:0/1337968086 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4c900a1f20 0x7f4c900a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:12.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.762+0000 7f4ca3400700 1 --2- 172.21.15.43:0/1337968086 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4c900a1f20 0x7f4c900a43e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4cac100da0 tx=0x7f4c94009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:12.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.763+0000 7f4ca9600700 1 -- 172.21.15.43:0/1337968086 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4c9c0203a0 con 0x7f4cac06a9e0 2024-03-28T22:35:12.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.992+0000 7f4cb2a00700 1 -- 172.21.15.43:0/1337968086 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f4cac199da0 con 0x7f4cac06a9e0 2024-03-28T22:35:12.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.992+0000 7f4ca9600700 1 -- 172.21.15.43:0/1337968086 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4c9c09b4f0 con 0x7f4cac06a9e0 2024-03-28T22:35:12.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.995+0000 7f4cb2a00700 1 -- 172.21.15.43:0/1337968086 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4c900a1f20 msgr2=0x7f4c900a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:12.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.995+0000 7f4cb2a00700 1 --2- 172.21.15.43:0/1337968086 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4c900a1f20 0x7f4c900a43e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4cac100da0 tx=0x7f4c94009310 comp rx=0 tx=0).stop 2024-03-28T22:35:12.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.995+0000 7f4cb2a00700 1 -- 172.21.15.43:0/1337968086 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4cac06a9e0 msgr2=0x7f4cac19e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:12.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.995+0000 7f4cb2a00700 1 --2- 172.21.15.43:0/1337968086 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4cac06a9e0 0x7f4cac19e6b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c00e9f0 tx=0x7f4c9c00ed00 comp rx=0 tx=0).stop 2024-03-28T22:35:12.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.996+0000 7f4cb2a00700 1 -- 172.21.15.43:0/1337968086 shutdown_connections 2024-03-28T22:35:12.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.996+0000 7f4cb2a00700 1 --2- 172.21.15.43:0/1337968086 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4c900a1f20 0x7f4c900a43e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:12.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.996+0000 7f4cb2a00700 1 --2- 172.21.15.43:0/1337968086 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4cac1075c0 0x7f4cac198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:12.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.996+0000 7f4cb2a00700 1 --2- 172.21.15.43:0/1337968086 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4cac06a9e0 0x7f4cac19e6b0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:12.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.996+0000 7f4cb2a00700 1 --2- 172.21.15.43:0/1337968086 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cac06b390 0x7f4cac19ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:12.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.996+0000 7f4cb2a00700 1 -- 172.21.15.43:0/1337968086 >> 172.21.15.43:0/1337968086 conn(0x7f4cac077190 msgr2=0x7f4cac0ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:12.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.996+0000 7f4cb2a00700 1 -- 172.21.15.43:0/1337968086 shutdown_connections 2024-03-28T22:35:12.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:11.996+0000 7f4cb2a00700 1 -- 172.21.15.43:0/1337968086 wait complete. 2024-03-28T22:35:12.556 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:12 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1337968086' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T22:35:12.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:12 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1337968086' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T22:35:12.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:12 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1337968086' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T22:35:13.553 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:13 smithi043 ceph-mon[31823]: pgmap v376: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:35:13.563 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:35:13.564 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:35:13.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:13 smithi172 ceph-mon[36566]: pgmap v376: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:35:13.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:13 smithi082 ceph-mon[36471]: pgmap v376: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:35:15.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.499+0000 7f9bbd200700 1 -- 172.21.15.43:0/977545164 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9bb806a940 msgr2=0x7f9bb806adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:15.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.499+0000 7f9bbd200700 1 --2- 172.21.15.43:0/977545164 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9bb806a940 0x7f9bb806adc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9ba40099c0 tx=0x7f9ba4009cd0 comp rx=0 tx=0).stop 2024-03-28T22:35:15.499 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:15 smithi043 ceph-mon[31823]: pgmap v377: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:35:15.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.499+0000 7f9bbd200700 1 -- 172.21.15.43:0/977545164 shutdown_connections 2024-03-28T22:35:15.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.499+0000 7f9bbd200700 1 --2- 172.21.15.43:0/977545164 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9bb806b300 0x7f9bb810f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:15.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.499+0000 7f9bbd200700 1 --2- 172.21.15.43:0/977545164 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bb81041a0 0x7f9bb8104580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:15.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.499+0000 7f9bbd200700 1 --2- 172.21.15.43:0/977545164 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9bb806a940 0x7f9bb806adc0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:15.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.499+0000 7f9bbd200700 1 -- 172.21.15.43:0/977545164 >> 172.21.15.43:0/977545164 conn(0x7f9bb8077160 msgr2=0x7f9bb8077570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:15.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.499+0000 7f9bbd200700 1 -- 172.21.15.43:0/977545164 shutdown_connections 2024-03-28T22:35:15.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.499+0000 7f9bbd200700 1 -- 172.21.15.43:0/977545164 wait complete. 2024-03-28T22:35:15.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.500+0000 7f9bbd200700 1 Processor -- start 2024-03-28T22:35:15.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.500+0000 7f9bbd200700 1 -- start start 2024-03-28T22:35:15.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.501+0000 7f9bbd200700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9bb806a940 0x7f9bb81a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:15.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.501+0000 7f9bbd200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bb806b300 0x7f9bb81a0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:15.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.501+0000 7f9bbd200700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9bb81041a0 0x7f9bb819aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:15.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.501+0000 7f9bbd200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb8112670 con 0x7f9bb806b300 2024-03-28T22:35:15.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.501+0000 7f9bbd200700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb81124f0 con 0x7f9bb806a940 2024-03-28T22:35:15.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.501+0000 7f9bbd200700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb81127f0 con 0x7f9bb81041a0 2024-03-28T22:35:15.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.501+0000 7f9bb5e00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9bb806a940 0x7f9bb81a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:15.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.501+0000 7f9bb5e00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9bb806a940 0x7f9bb81a08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59270/0 (socket says 172.21.15.43:59270) 2024-03-28T22:35:15.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.501+0000 7f9bb5e00700 1 -- 172.21.15.43:0/3873993283 learned_addr learned my addr 172.21.15.43:0/3873993283 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:35:15.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.501+0000 7f9bb5400700 1 --2- 172.21.15.43:0/3873993283 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bb806b300 0x7f9bb81a0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:15.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.502+0000 7f9bb6800700 1 --2- 172.21.15.43:0/3873993283 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9bb81041a0 0x7f9bb819aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:15.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.502+0000 7f9bb5e00700 1 -- 172.21.15.43:0/3873993283 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9bb81041a0 msgr2=0x7f9bb819aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:15.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.502+0000 7f9bb5e00700 1 --2- 172.21.15.43:0/3873993283 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9bb81041a0 0x7f9bb819aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:15.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.502+0000 7f9bb5e00700 1 -- 172.21.15.43:0/3873993283 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bb806b300 msgr2=0x7f9bb81a0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:15.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.502+0000 7f9bb5e00700 1 --2- 172.21.15.43:0/3873993283 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bb806b300 0x7f9bb81a0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:15.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.502+0000 7f9bb5e00700 1 -- 172.21.15.43:0/3873993283 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bb819b210 con 0x7f9bb806a940 2024-03-28T22:35:15.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.502+0000 7f9bb6800700 1 --2- 172.21.15.43:0/3873993283 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9bb81041a0 0x7f9bb819aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:35:15.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.502+0000 7f9bb5e00700 1 --2- 172.21.15.43:0/3873993283 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9bb806a940 0x7f9bb81a08f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9ba800ef10 tx=0x7f9ba800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:15.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.502+0000 7f9baea00700 1 -- 172.21.15.43:0/3873993283 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba800ca70 con 0x7f9bb806a940 2024-03-28T22:35:15.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.503+0000 7f9baea00700 1 -- 172.21.15.43:0/3873993283 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ba8019070 con 0x7f9bb806a940 2024-03-28T22:35:15.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.503+0000 7f9bbd200700 1 -- 172.21.15.43:0/3873993283 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bb819b4a0 con 0x7f9bb806a940 2024-03-28T22:35:15.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.503+0000 7f9baea00700 1 -- 172.21.15.43:0/3873993283 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba80146e0 con 0x7f9bb806a940 2024-03-28T22:35:15.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.503+0000 7f9bbd200700 1 -- 172.21.15.43:0/3873993283 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bb81a77c0 con 0x7f9bb806a940 2024-03-28T22:35:15.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.504+0000 7f9bbd200700 1 -- 172.21.15.43:0/3873993283 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bb8068f00 con 0x7f9bb806a940 2024-03-28T22:35:15.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.506+0000 7f9baea00700 1 -- 172.21.15.43:0/3873993283 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f9ba8012050 con 0x7f9bb806a940 2024-03-28T22:35:15.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.507+0000 7f9baea00700 1 --2- 172.21.15.43:0/3873993283 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9b9c0a1ea0 0x7f9b9c0a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:15.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.508+0000 7f9baea00700 1 -- 172.21.15.43:0/3873993283 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7f9ba80ce3f0 con 0x7f9bb806a940 2024-03-28T22:35:15.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.508+0000 7f9bb5400700 1 --2- 172.21.15.43:0/3873993283 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9b9c0a1ea0 0x7f9b9c0a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:15.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.509+0000 7f9bb5400700 1 --2- 172.21.15.43:0/3873993283 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9b9c0a1ea0 0x7f9b9c0a4360 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9bb810f5b0 tx=0x7f9ba4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:15.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.516+0000 7f9baea00700 1 -- 172.21.15.43:0/3873993283 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9ba8010040 con 0x7f9bb806a940 2024-03-28T22:35:15.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:15 smithi172 ceph-mon[36566]: pgmap v377: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:35:15.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:15 smithi082 ceph-mon[36471]: pgmap v377: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:35:15.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.728+0000 7f633d200700 1 -- 172.21.15.43:0/2292037033 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6338106450 msgr2=0x7f633810e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:15.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.728+0000 7f633d200700 1 --2- 172.21.15.43:0/2292037033 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6338106450 0x7f633810e020 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f633000b0a0 tx=0x7f633000b3b0 comp rx=0 tx=0).stop 2024-03-28T22:35:15.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.728+0000 7f633d200700 1 -- 172.21.15.43:0/2292037033 shutdown_connections 2024-03-28T22:35:15.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.728+0000 7f633d200700 1 --2- 172.21.15.43:0/2292037033 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6338106450 0x7f633810e020 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:15.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.728+0000 7f633d200700 1 --2- 172.21.15.43:0/2292037033 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6338101390 0x7f6338101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:15.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.728+0000 7f633d200700 1 --2- 172.21.15.43:0/2292037033 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6338101d40 0x7f6338105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:15.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.728+0000 7f633d200700 1 -- 172.21.15.43:0/2292037033 >> 172.21.15.43:0/2292037033 conn(0x7f63380780b0 msgr2=0x7f63380784c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:15.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.729+0000 7f633d200700 1 -- 172.21.15.43:0/2292037033 shutdown_connections 2024-03-28T22:35:15.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.729+0000 7f633d200700 1 -- 172.21.15.43:0/2292037033 wait complete. 2024-03-28T22:35:15.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.730+0000 7f633d200700 1 Processor -- start 2024-03-28T22:35:15.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.730+0000 7f633d200700 1 -- start start 2024-03-28T22:35:15.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.730+0000 7f633d200700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6338101390 0x7f63381a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:15.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.730+0000 7f633d200700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6338101d40 0x7f63381a0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:15.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.730+0000 7f633d200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6338106450 0x7f633819a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:15.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.730+0000 7f633d200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6338111480 con 0x7f6338106450 2024-03-28T22:35:15.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.731+0000 7f633d200700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f6338111300 con 0x7f6338101390 2024-03-28T22:35:15.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.731+0000 7f633d200700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f6338111600 con 0x7f6338101d40 2024-03-28T22:35:15.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.731+0000 7f6335e00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6338101390 0x7f63381a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:15.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.731+0000 7f6335e00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6338101390 0x7f63381a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59284/0 (socket says 172.21.15.43:59284) 2024-03-28T22:35:15.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.731+0000 7f6335e00700 1 -- 172.21.15.43:0/1103110284 learned_addr learned my addr 172.21.15.43:0/1103110284 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:35:15.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.731+0000 7f632fe00700 1 --2- 172.21.15.43:0/1103110284 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6338101d40 0x7f63381a0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:15.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.731+0000 7f6336800700 1 --2- 172.21.15.43:0/1103110284 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6338106450 0x7f633819a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:15.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.731+0000 7f6335e00700 1 -- 172.21.15.43:0/1103110284 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6338101d40 msgr2=0x7f63381a0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:15.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.731+0000 7f6335e00700 1 --2- 172.21.15.43:0/1103110284 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6338101d40 0x7f63381a0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:15.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.731+0000 7f6335e00700 1 -- 172.21.15.43:0/1103110284 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6338106450 msgr2=0x7f633819a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:15.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.731+0000 7f6335e00700 1 --2- 172.21.15.43:0/1103110284 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6338106450 0x7f633819a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:15.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.731+0000 7f6335e00700 1 -- 172.21.15.43:0/1103110284 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f633819b140 con 0x7f6338101390 2024-03-28T22:35:15.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.732+0000 7f6335e00700 1 --2- 172.21.15.43:0/1103110284 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6338101390 0x7f63381a0820 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f632800eb30 tx=0x7f632800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:15.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.732+0000 7f632ea00700 1 -- 172.21.15.43:0/1103110284 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632800cbd0 con 0x7f6338101390 2024-03-28T22:35:15.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.732+0000 7f632ea00700 1 -- 172.21.15.43:0/1103110284 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6328004500 con 0x7f6338101390 2024-03-28T22:35:15.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.732+0000 7f632ea00700 1 -- 172.21.15.43:0/1103110284 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6328010430 con 0x7f6338101390 2024-03-28T22:35:15.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.732+0000 7f633d200700 1 -- 172.21.15.43:0/1103110284 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f633819b430 con 0x7f6338101390 2024-03-28T22:35:15.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.732+0000 7f633d200700 1 -- 172.21.15.43:0/1103110284 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63381a76f0 con 0x7f6338101390 2024-03-28T22:35:15.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.733+0000 7f633d200700 1 -- 172.21.15.43:0/1103110284 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6338102850 con 0x7f6338101390 2024-03-28T22:35:15.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.735+0000 7f632ea00700 1 -- 172.21.15.43:0/1103110284 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f6328003f00 con 0x7f6338101390 2024-03-28T22:35:15.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.737+0000 7f632ea00700 1 --2- 172.21.15.43:0/1103110284 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f631c0a1f20 0x7f631c0a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:15.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.737+0000 7f632ea00700 1 -- 172.21.15.43:0/1103110284 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7f63280ce5e0 con 0x7f6338101390 2024-03-28T22:35:15.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.737+0000 7f632fe00700 1 --2- 172.21.15.43:0/1103110284 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f631c0a1f20 0x7f631c0a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:15.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.738+0000 7f632fe00700 1 --2- 172.21.15.43:0/1103110284 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f631c0a1f20 0x7f631c0a43e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f633810e3d0 tx=0x7f6320009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:15.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.741+0000 7f632ea00700 1 -- 172.21.15.43:0/1103110284 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f632809b4f0 con 0x7f6338101390 2024-03-28T22:35:15.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.750+0000 7f9bbd200700 1 -- 172.21.15.43:0/3873993283 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f9bb81a7990 con 0x7f9bb806a940 2024-03-28T22:35:15.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.751+0000 7f9baea00700 1 -- 172.21.15.43:0/3873993283 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f9ba809b300 con 0x7f9bb806a940 2024-03-28T22:35:15.750 INFO:teuthology.orchestra.run.smithi043.stdout:107374182479 2024-03-28T22:35:15.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.754+0000 7f9bbd200700 1 -- 172.21.15.43:0/3873993283 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9b9c0a1ea0 msgr2=0x7f9b9c0a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:15.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.754+0000 7f9bbd200700 1 --2- 172.21.15.43:0/3873993283 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9b9c0a1ea0 0x7f9b9c0a4360 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9bb810f5b0 tx=0x7f9ba4005fb0 comp rx=0 tx=0).stop 2024-03-28T22:35:15.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.754+0000 7f9bbd200700 1 -- 172.21.15.43:0/3873993283 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9bb806a940 msgr2=0x7f9bb81a08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:15.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.754+0000 7f9bbd200700 1 --2- 172.21.15.43:0/3873993283 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9bb806a940 0x7f9bb81a08f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9ba800ef10 tx=0x7f9ba800c3a0 comp rx=0 tx=0).stop 2024-03-28T22:35:15.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.754+0000 7f9bbd200700 1 -- 172.21.15.43:0/3873993283 shutdown_connections 2024-03-28T22:35:15.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.754+0000 7f9bbd200700 1 --2- 172.21.15.43:0/3873993283 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9b9c0a1ea0 0x7f9b9c0a4360 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:15.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.754+0000 7f9bbd200700 1 --2- 172.21.15.43:0/3873993283 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9bb81041a0 0x7f9bb819aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:15.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.754+0000 7f9bbd200700 1 --2- 172.21.15.43:0/3873993283 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9bb806a940 0x7f9bb81a08f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:15.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.754+0000 7f9bbd200700 1 --2- 172.21.15.43:0/3873993283 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bb806b300 0x7f9bb81a0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:15.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.754+0000 7f9bbd200700 1 -- 172.21.15.43:0/3873993283 >> 172.21.15.43:0/3873993283 conn(0x7f9bb8077160 msgr2=0x7f9bb80ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:15.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.754+0000 7f9bbd200700 1 -- 172.21.15.43:0/3873993283 shutdown_connections 2024-03-28T22:35:15.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.754+0000 7f9bbd200700 1 -- 172.21.15.43:0/3873993283 wait complete. 2024-03-28T22:35:15.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.982+0000 7f633d200700 1 -- 172.21.15.43:0/1103110284 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f6338068f00 con 0x7f6338101390 2024-03-28T22:35:15.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.982+0000 7f632ea00700 1 -- 172.21.15.43:0/1103110284 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f632809adf0 con 0x7f6338101390 2024-03-28T22:35:15.982 INFO:teuthology.orchestra.run.smithi043.stdout:55834574938 2024-03-28T22:35:15.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.985+0000 7f633d200700 1 -- 172.21.15.43:0/1103110284 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f631c0a1f20 msgr2=0x7f631c0a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:15.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.985+0000 7f633d200700 1 --2- 172.21.15.43:0/1103110284 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f631c0a1f20 0x7f631c0a43e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f633810e3d0 tx=0x7f6320009260 comp rx=0 tx=0).stop 2024-03-28T22:35:15.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.985+0000 7f633d200700 1 -- 172.21.15.43:0/1103110284 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6338101390 msgr2=0x7f63381a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:15.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.985+0000 7f633d200700 1 --2- 172.21.15.43:0/1103110284 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6338101390 0x7f63381a0820 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f632800eb30 tx=0x7f632800ee40 comp rx=0 tx=0).stop 2024-03-28T22:35:15.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.986+0000 7f633d200700 1 -- 172.21.15.43:0/1103110284 shutdown_connections 2024-03-28T22:35:15.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.986+0000 7f633d200700 1 --2- 172.21.15.43:0/1103110284 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f631c0a1f20 0x7f631c0a43e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:15.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.986+0000 7f633d200700 1 --2- 172.21.15.43:0/1103110284 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6338106450 0x7f633819a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:15.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.986+0000 7f633d200700 1 --2- 172.21.15.43:0/1103110284 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6338101390 0x7f63381a0820 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:15.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.986+0000 7f633d200700 1 --2- 172.21.15.43:0/1103110284 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6338101d40 0x7f63381a0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:15.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.986+0000 7f633d200700 1 -- 172.21.15.43:0/1103110284 >> 172.21.15.43:0/1103110284 conn(0x7f63380780b0 msgr2=0x7f633806a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:15.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.986+0000 7f633d200700 1 -- 172.21.15.43:0/1103110284 shutdown_connections 2024-03-28T22:35:15.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:15.986+0000 7f633d200700 1 -- 172.21.15.43:0/1103110284 wait complete. 2024-03-28T22:35:16.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:16 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3873993283' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T22:35:16.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:16 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1103110284' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T22:35:16.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:16 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3873993283' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T22:35:16.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:16 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1103110284' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T22:35:16.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:16 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3873993283' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T22:35:16.705 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:16 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1103110284' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T22:35:17.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.527+0000 7f03ea800700 1 -- 172.21.15.43:0/2110303899 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03e4104430 msgr2=0x7f03e41048b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:17.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.527+0000 7f03ea800700 1 --2- 172.21.15.43:0/2110303899 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03e4104430 0x7f03e41048b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f03d0009990 tx=0x7f03d0009ca0 comp rx=0 tx=0).stop 2024-03-28T22:35:17.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.527+0000 7f03ea800700 1 -- 172.21.15.43:0/2110303899 shutdown_connections 2024-03-28T22:35:17.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.527+0000 7f03ea800700 1 --2- 172.21.15.43:0/2110303899 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03e4104df0 0x7f03e4111300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:17.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.527+0000 7f03ea800700 1 --2- 172.21.15.43:0/2110303899 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03e410a4c0 0x7f03e410a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:17.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.527+0000 7f03ea800700 1 --2- 172.21.15.43:0/2110303899 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03e4104430 0x7f03e41048b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:17.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.527+0000 7f03ea800700 1 -- 172.21.15.43:0/2110303899 >> 172.21.15.43:0/2110303899 conn(0x7f03e40fff20 msgr2=0x7f03e4102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:17.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.527+0000 7f03ea800700 1 -- 172.21.15.43:0/2110303899 shutdown_connections 2024-03-28T22:35:17.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.527+0000 7f03ea800700 1 -- 172.21.15.43:0/2110303899 wait complete. 2024-03-28T22:35:17.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.528+0000 7f03ea800700 1 Processor -- start 2024-03-28T22:35:17.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.528+0000 7f03ea800700 1 -- start start 2024-03-28T22:35:17.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.529+0000 7f03ea800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03e4104430 0x7f03e41a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:17.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.529+0000 7f03ea800700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03e4104df0 0x7f03e41a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:17.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.529+0000 7f03ea800700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03e410a4c0 0x7f03e419a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:17.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.529+0000 7f03ea800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f03e4114770 con 0x7f03e4104430 2024-03-28T22:35:17.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.529+0000 7f03ea800700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f03e41145f0 con 0x7f03e410a4c0 2024-03-28T22:35:17.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.529+0000 7f03ea800700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f03e41148f0 con 0x7f03e4104df0 2024-03-28T22:35:17.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.529+0000 7f03e2a00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03e4104df0 0x7f03e41a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:17.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.529+0000 7f03e3400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03e4104430 0x7f03e41a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:17.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.529+0000 7f03e3e00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03e410a4c0 0x7f03e419a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:17.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.529+0000 7f03e2a00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03e4104df0 0x7f03e41a0d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:50214/0 (socket says 172.21.15.43:50214) 2024-03-28T22:35:17.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.529+0000 7f03e2a00700 1 -- 172.21.15.43:0/4198019894 learned_addr learned my addr 172.21.15.43:0/4198019894 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:35:17.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.529+0000 7f03e2a00700 1 -- 172.21.15.43:0/4198019894 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03e410a4c0 msgr2=0x7f03e419a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:17.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.529+0000 7f03e2a00700 1 --2- 172.21.15.43:0/4198019894 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03e410a4c0 0x7f03e419a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:17.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.529+0000 7f03e2a00700 1 -- 172.21.15.43:0/4198019894 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03e4104430 msgr2=0x7f03e41a0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:17.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.529+0000 7f03e2a00700 1 --2- 172.21.15.43:0/4198019894 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03e4104430 0x7f03e41a0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:17.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.529+0000 7f03e2a00700 1 -- 172.21.15.43:0/4198019894 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03e419b150 con 0x7f03e4104df0 2024-03-28T22:35:17.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.530+0000 7f03e2a00700 1 --2- 172.21.15.43:0/4198019894 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03e4104df0 0x7f03e41a0d70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f03d0000c00 tx=0x7f03d000bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:17.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.530+0000 7f03dbe00700 1 -- 172.21.15.43:0/4198019894 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d0027070 con 0x7f03e4104df0 2024-03-28T22:35:17.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.530+0000 7f03dbe00700 1 -- 172.21.15.43:0/4198019894 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03d00056d0 con 0x7f03e4104df0 2024-03-28T22:35:17.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.530+0000 7f03ea800700 1 -- 172.21.15.43:0/4198019894 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03e419b3e0 con 0x7f03e4104df0 2024-03-28T22:35:17.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.530+0000 7f03dbe00700 1 -- 172.21.15.43:0/4198019894 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d0022e10 con 0x7f03e4104df0 2024-03-28T22:35:17.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.530+0000 7f03ea800700 1 -- 172.21.15.43:0/4198019894 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03e41a7700 con 0x7f03e4104df0 2024-03-28T22:35:17.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.531+0000 7f03ea800700 1 -- 172.21.15.43:0/4198019894 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03e4105b70 con 0x7f03e4104df0 2024-03-28T22:35:17.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.533+0000 7f03dbe00700 1 -- 172.21.15.43:0/4198019894 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f03d002a070 con 0x7f03e4104df0 2024-03-28T22:35:17.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.534+0000 7f03dbe00700 1 --2- 172.21.15.43:0/4198019894 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f03c80a1ea0 0x7f03c80a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:17.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.534+0000 7f03dbe00700 1 -- 172.21.15.43:0/4198019894 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7f03d00cefa0 con 0x7f03e4104df0 2024-03-28T22:35:17.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.535+0000 7f03e3400700 1 --2- 172.21.15.43:0/4198019894 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f03c80a1ea0 0x7f03c80a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:17.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.535+0000 7f03e3400700 1 --2- 172.21.15.43:0/4198019894 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f03c80a1ea0 0x7f03c80a4360 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f03e419bb30 tx=0x7f03d4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:17.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.538+0000 7f03dbe00700 1 -- 172.21.15.43:0/4198019894 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f03d0013070 con 0x7f03e4104df0 2024-03-28T22:35:17.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.771+0000 7f03ea800700 1 -- 172.21.15.43:0/4198019894 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f03e402db30 con 0x7f03e4104df0 2024-03-28T22:35:17.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.772+0000 7f03dbe00700 1 -- 172.21.15.43:0/4198019894 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f03d009bf40 con 0x7f03e4104df0 2024-03-28T22:35:17.322 INFO:teuthology.orchestra.run.smithi043.stdout:223338299450 2024-03-28T22:35:17.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.775+0000 7f03ea800700 1 -- 172.21.15.43:0/4198019894 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f03c80a1ea0 msgr2=0x7f03c80a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:17.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.775+0000 7f03ea800700 1 --2- 172.21.15.43:0/4198019894 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f03c80a1ea0 0x7f03c80a4360 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f03e419bb30 tx=0x7f03d4009260 comp rx=0 tx=0).stop 2024-03-28T22:35:17.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.775+0000 7f03ea800700 1 -- 172.21.15.43:0/4198019894 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03e4104df0 msgr2=0x7f03e41a0d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:17.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.775+0000 7f03ea800700 1 --2- 172.21.15.43:0/4198019894 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03e4104df0 0x7f03e41a0d70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f03d0000c00 tx=0x7f03d000bab0 comp rx=0 tx=0).stop 2024-03-28T22:35:17.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.775+0000 7f03ea800700 1 -- 172.21.15.43:0/4198019894 shutdown_connections 2024-03-28T22:35:17.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.775+0000 7f03ea800700 1 --2- 172.21.15.43:0/4198019894 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f03c80a1ea0 0x7f03c80a4360 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:17.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.775+0000 7f03ea800700 1 --2- 172.21.15.43:0/4198019894 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03e410a4c0 0x7f03e419a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:17.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.775+0000 7f03ea800700 1 --2- 172.21.15.43:0/4198019894 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03e4104430 0x7f03e41a0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:17.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.775+0000 7f03ea800700 1 --2- 172.21.15.43:0/4198019894 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03e4104df0 0x7f03e41a0d70 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:17.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.775+0000 7f03ea800700 1 -- 172.21.15.43:0/4198019894 >> 172.21.15.43:0/4198019894 conn(0x7f03e40fff20 msgr2=0x7f03e41004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:17.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.775+0000 7f03ea800700 1 -- 172.21.15.43:0/4198019894 shutdown_connections 2024-03-28T22:35:17.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:16.775+0000 7f03ea800700 1 -- 172.21.15.43:0/4198019894 wait complete. 2024-03-28T22:35:17.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:17 smithi043 ceph-mon[31823]: pgmap v378: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T22:35:17.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:17 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/4198019894' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T22:35:17.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:17 smithi172 ceph-mon[36566]: pgmap v378: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T22:35:17.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:17 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/4198019894' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T22:35:17.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:17 smithi082 ceph-mon[36471]: pgmap v378: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T22:35:17.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:17 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/4198019894' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T22:35:17.702 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920837 got 141733920841 for osd.4 2024-03-28T22:35:17.702 DEBUG:teuthology.parallel:result is None 2024-03-28T22:35:17.995 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182475 got 107374182479 for osd.3 2024-03-28T22:35:17.995 DEBUG:teuthology.parallel:result is None 2024-03-28T22:35:18.351 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574932 got 55834574938 for osd.1 2024-03-28T22:35:18.351 DEBUG:teuthology.parallel:result is None 2024-03-28T22:35:19.243 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:35:19.408 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411410 got 77309411415 for osd.2 2024-03-28T22:35:19.408 DEBUG:teuthology.parallel:result is None 2024-03-28T22:35:19.548 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:19 smithi043 ceph-mon[31823]: pgmap v379: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:35:19.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:19 smithi172 ceph-mon[36566]: pgmap v379: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:35:19.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:19 smithi082 ceph-mon[36471]: pgmap v379: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:35:20.515 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299441 got 223338299450 for osd.7 2024-03-28T22:35:20.515 DEBUG:teuthology.parallel:result is None 2024-03-28T22:35:21.621 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:21 smithi043 ceph-mon[31823]: pgmap v380: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-28T22:35:21.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:21 smithi172 ceph-mon[36566]: pgmap v380: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-28T22:35:21.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:21 smithi082 ceph-mon[36471]: pgmap v380: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-28T22:35:22.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.510+0000 7fd82f000700 1 -- 172.21.15.43:0/1205945197 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd828107550 msgr2=0x7fd82810f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:22.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.510+0000 7fd82f000700 1 --2- 172.21.15.43:0/1205945197 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd828107550 0x7fd82810f120 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd82000b0a0 tx=0x7fd82000b3b0 comp rx=0 tx=0).stop 2024-03-28T22:35:22.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.510+0000 7fd82f000700 1 -- 172.21.15.43:0/1205945197 shutdown_connections 2024-03-28T22:35:22.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.510+0000 7fd82f000700 1 --2- 172.21.15.43:0/1205945197 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd828107550 0x7fd82810f120 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:22.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.510+0000 7fd82f000700 1 --2- 172.21.15.43:0/1205945197 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd828102520 0x7fd828102900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:22.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.510+0000 7fd82f000700 1 --2- 172.21.15.43:0/1205945197 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd828102ed0 0x7fd828106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:22.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.510+0000 7fd82f000700 1 -- 172.21.15.43:0/1205945197 >> 172.21.15.43:0/1205945197 conn(0x7fd8280792d0 msgr2=0x7fd8280796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:22.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.510+0000 7fd82f000700 1 -- 172.21.15.43:0/1205945197 shutdown_connections 2024-03-28T22:35:22.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.510+0000 7fd82f000700 1 -- 172.21.15.43:0/1205945197 wait complete. 2024-03-28T22:35:22.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.512+0000 7fd82f000700 1 Processor -- start 2024-03-28T22:35:22.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.512+0000 7fd82f000700 1 -- start start 2024-03-28T22:35:22.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.512+0000 7fd82f000700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd828102520 0x7fd8281a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:22.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.512+0000 7fd82f000700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd828102ed0 0x7fd8281a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:22.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.512+0000 7fd82f000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd828107550 0x7fd82819aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:22.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.512+0000 7fd82f000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd8281125a0 con 0x7fd828107550 2024-03-28T22:35:22.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.512+0000 7fd82f000700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd828112420 con 0x7fd828102520 2024-03-28T22:35:22.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.512+0000 7fd82f000700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd828112720 con 0x7fd828102ed0 2024-03-28T22:35:22.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.513+0000 7fd827e00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd828102520 0x7fd8281a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:22.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.513+0000 7fd827e00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd828102520 0x7fd8281a0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59366/0 (socket says 172.21.15.43:59366) 2024-03-28T22:35:22.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.513+0000 7fd827e00700 1 -- 172.21.15.43:0/704997235 learned_addr learned my addr 172.21.15.43:0/704997235 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:35:22.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.513+0000 7fd827400700 1 --2- 172.21.15.43:0/704997235 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd828102ed0 0x7fd8281a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:22.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.513+0000 7fd82d000700 1 --2- 172.21.15.43:0/704997235 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd828107550 0x7fd82819aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:22.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.513+0000 7fd827e00700 1 -- 172.21.15.43:0/704997235 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd828102ed0 msgr2=0x7fd8281a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:22.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.513+0000 7fd827e00700 1 --2- 172.21.15.43:0/704997235 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd828102ed0 0x7fd8281a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:22.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.513+0000 7fd827e00700 1 -- 172.21.15.43:0/704997235 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd828107550 msgr2=0x7fd82819aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:22.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.513+0000 7fd827e00700 1 --2- 172.21.15.43:0/704997235 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd828107550 0x7fd82819aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:22.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.513+0000 7fd827e00700 1 -- 172.21.15.43:0/704997235 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd82819b160 con 0x7fd828102520 2024-03-28T22:35:22.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.513+0000 7fd82d000700 1 --2- 172.21.15.43:0/704997235 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd828107550 0x7fd82819aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:35:22.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.514+0000 7fd827e00700 1 --2- 172.21.15.43:0/704997235 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd828102520 0x7fd8281a0840 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd81800e9c0 tx=0x7fd81800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:22.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.514+0000 7fd824c00700 1 -- 172.21.15.43:0/704997235 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd81800cbd0 con 0x7fd828102520 2024-03-28T22:35:22.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.514+0000 7fd824c00700 1 -- 172.21.15.43:0/704997235 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd81800cd70 con 0x7fd828102520 2024-03-28T22:35:22.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.514+0000 7fd824c00700 1 -- 172.21.15.43:0/704997235 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8180056d0 con 0x7fd828102520 2024-03-28T22:35:22.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.514+0000 7fd82f000700 1 -- 172.21.15.43:0/704997235 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd82819b450 con 0x7fd828102520 2024-03-28T22:35:22.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.514+0000 7fd82f000700 1 -- 172.21.15.43:0/704997235 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8281a7710 con 0x7fd828102520 2024-03-28T22:35:22.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.515+0000 7fd82f000700 1 -- 172.21.15.43:0/704997235 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd828068f00 con 0x7fd828102520 2024-03-28T22:35:22.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.517+0000 7fd824c00700 1 -- 172.21.15.43:0/704997235 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7fd818004830 con 0x7fd828102520 2024-03-28T22:35:22.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.519+0000 7fd824c00700 1 --2- 172.21.15.43:0/704997235 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd80c0a20c0 0x7fd80c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:22.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.519+0000 7fd824c00700 1 -- 172.21.15.43:0/704997235 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7fd8180ce5c0 con 0x7fd828102520 2024-03-28T22:35:22.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.520+0000 7fd827400700 1 --2- 172.21.15.43:0/704997235 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd80c0a20c0 0x7fd80c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:22.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.520+0000 7fd827400700 1 --2- 172.21.15.43:0/704997235 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd80c0a20c0 0x7fd80c0a4580 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd828101ef0 tx=0x7fd81400b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:22.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.525+0000 7fd824c00700 1 -- 172.21.15.43:0/704997235 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd81800b2c0 con 0x7fd828102520 2024-03-28T22:35:22.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.776+0000 7fd82f000700 1 -- 172.21.15.43:0/704997235 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fd82819bf30 con 0x7fd828102520 2024-03-28T22:35:22.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.777+0000 7fd824c00700 1 -- 172.21.15.43:0/704997235 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd81809b4d0 con 0x7fd828102520 2024-03-28T22:35:22.776 INFO:teuthology.orchestra.run.smithi043.stdout:163208757319 2024-03-28T22:35:22.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.780+0000 7fd82f000700 1 -- 172.21.15.43:0/704997235 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd80c0a20c0 msgr2=0x7fd80c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:22.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.780+0000 7fd82f000700 1 --2- 172.21.15.43:0/704997235 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd80c0a20c0 0x7fd80c0a4580 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd828101ef0 tx=0x7fd81400b3c0 comp rx=0 tx=0).stop 2024-03-28T22:35:22.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.780+0000 7fd82f000700 1 -- 172.21.15.43:0/704997235 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd828102520 msgr2=0x7fd8281a0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:22.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.780+0000 7fd82f000700 1 --2- 172.21.15.43:0/704997235 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd828102520 0x7fd8281a0840 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd81800e9c0 tx=0x7fd81800ecd0 comp rx=0 tx=0).stop 2024-03-28T22:35:22.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.780+0000 7fd82f000700 1 -- 172.21.15.43:0/704997235 shutdown_connections 2024-03-28T22:35:22.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.780+0000 7fd82f000700 1 --2- 172.21.15.43:0/704997235 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd80c0a20c0 0x7fd80c0a4580 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:22.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.780+0000 7fd82f000700 1 --2- 172.21.15.43:0/704997235 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd828107550 0x7fd82819aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:22.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.780+0000 7fd82f000700 1 --2- 172.21.15.43:0/704997235 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd828102520 0x7fd8281a0840 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:22.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.780+0000 7fd82f000700 1 --2- 172.21.15.43:0/704997235 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd828102ed0 0x7fd8281a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:22.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.780+0000 7fd82f000700 1 -- 172.21.15.43:0/704997235 >> 172.21.15.43:0/704997235 conn(0x7fd8280792d0 msgr2=0x7fd828100430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:22.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.780+0000 7fd82f000700 1 -- 172.21.15.43:0/704997235 shutdown_connections 2024-03-28T22:35:22.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.780+0000 7fd82f000700 1 -- 172.21.15.43:0/704997235 wait complete. 2024-03-28T22:35:23.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.957+0000 7f4c17600700 1 -- 172.21.15.43:0/3125471584 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4c10104df0 msgr2=0x7f4c10111300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:23.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.957+0000 7f4c17600700 1 --2- 172.21.15.43:0/3125471584 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4c10104df0 0x7f4c10111300 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4c0800b0a0 tx=0x7f4c0800b3b0 comp rx=0 tx=0).stop 2024-03-28T22:35:23.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.957+0000 7f4c17600700 1 -- 172.21.15.43:0/3125471584 shutdown_connections 2024-03-28T22:35:23.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.957+0000 7f4c17600700 1 --2- 172.21.15.43:0/3125471584 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4c10104df0 0x7f4c10111300 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:23.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.957+0000 7f4c17600700 1 --2- 172.21.15.43:0/3125471584 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c1010a4c0 0x7f4c1010a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:23.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.957+0000 7f4c17600700 1 --2- 172.21.15.43:0/3125471584 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c10104430 0x7f4c101048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:23.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.957+0000 7f4c17600700 1 -- 172.21.15.43:0/3125471584 >> 172.21.15.43:0/3125471584 conn(0x7f4c100fff20 msgr2=0x7f4c10102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:23.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.957+0000 7f4c17600700 1 -- 172.21.15.43:0/3125471584 shutdown_connections 2024-03-28T22:35:23.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.957+0000 7f4c17600700 1 -- 172.21.15.43:0/3125471584 wait complete. 2024-03-28T22:35:23.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.958+0000 7f4c17600700 1 Processor -- start 2024-03-28T22:35:23.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.958+0000 7f4c17600700 1 -- start start 2024-03-28T22:35:23.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.959+0000 7f4c17600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c10104430 0x7f4c101a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:23.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.959+0000 7f4c17600700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4c10104df0 0x7f4c101a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:23.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.959+0000 7f4c17600700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c1010a4c0 0x7f4c1019a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:23.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.959+0000 7f4c17600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4c10114750 con 0x7f4c10104430 2024-03-28T22:35:23.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.959+0000 7f4c17600700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4c101145d0 con 0x7f4c10104df0 2024-03-28T22:35:23.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.959+0000 7f4c17600700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4c101148d0 con 0x7f4c1010a4c0 2024-03-28T22:35:23.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.959+0000 7f4c0fe00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4c10104df0 0x7f4c101a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:23.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.959+0000 7f4c0fe00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4c10104df0 0x7f4c101a0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59384/0 (socket says 172.21.15.43:59384) 2024-03-28T22:35:23.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.959+0000 7f4c0fe00700 1 -- 172.21.15.43:0/1919079003 learned_addr learned my addr 172.21.15.43:0/1919079003 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:35:23.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.959+0000 7f4c14c00700 1 --2- 172.21.15.43:0/1919079003 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c10104430 0x7f4c101a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:23.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.959+0000 7f4c15600700 1 --2- 172.21.15.43:0/1919079003 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c1010a4c0 0x7f4c1019a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:23.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.959+0000 7f4c0fe00700 1 -- 172.21.15.43:0/1919079003 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c1010a4c0 msgr2=0x7f4c1019a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:23.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.959+0000 7f4c0fe00700 1 --2- 172.21.15.43:0/1919079003 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c1010a4c0 0x7f4c1019a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:23.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.959+0000 7f4c0fe00700 1 -- 172.21.15.43:0/1919079003 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c10104430 msgr2=0x7f4c101a07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:23.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.959+0000 7f4c0fe00700 1 --2- 172.21.15.43:0/1919079003 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c10104430 0x7f4c101a07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:23.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.959+0000 7f4c0fe00700 1 -- 172.21.15.43:0/1919079003 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c1019b0d0 con 0x7f4c10104df0 2024-03-28T22:35:23.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.959+0000 7f4c14c00700 1 --2- 172.21.15.43:0/1919079003 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c10104430 0x7f4c101a07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:35:23.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.960+0000 7f4c0fe00700 1 --2- 172.21.15.43:0/1919079003 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4c10104df0 0x7f4c101a0cf0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc00e9c0 tx=0x7f4bfc00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:23.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.960+0000 7f4c0d600700 1 -- 172.21.15.43:0/1919079003 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc00cbd0 con 0x7f4c10104df0 2024-03-28T22:35:23.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.960+0000 7f4c0d600700 1 -- 172.21.15.43:0/1919079003 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bfc00cd70 con 0x7f4c10104df0 2024-03-28T22:35:23.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.960+0000 7f4c17600700 1 -- 172.21.15.43:0/1919079003 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c1019b3c0 con 0x7f4c10104df0 2024-03-28T22:35:23.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.960+0000 7f4c0d600700 1 -- 172.21.15.43:0/1919079003 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc010430 con 0x7f4c10104df0 2024-03-28T22:35:23.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.960+0000 7f4c17600700 1 -- 172.21.15.43:0/1919079003 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c101a7680 con 0x7f4c10104df0 2024-03-28T22:35:23.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.961+0000 7f4c17600700 1 -- 172.21.15.43:0/1919079003 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c10068f00 con 0x7f4c10104df0 2024-03-28T22:35:23.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.963+0000 7f4c0d600700 1 -- 172.21.15.43:0/1919079003 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f4bfc004750 con 0x7f4c10104df0 2024-03-28T22:35:23.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.964+0000 7f4c0d600700 1 --2- 172.21.15.43:0/1919079003 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4bf80a1f20 0x7f4bf80a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:23.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.964+0000 7f4c0d600700 1 -- 172.21.15.43:0/1919079003 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7f4bfc0ce4f0 con 0x7f4c10104df0 2024-03-28T22:35:23.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.965+0000 7f4c14c00700 1 --2- 172.21.15.43:0/1919079003 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4bf80a1f20 0x7f4bf80a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:23.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.965+0000 7f4c14c00700 1 --2- 172.21.15.43:0/1919079003 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4bf80a1f20 0x7f4bf80a43e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4c1019bab0 tx=0x7f4c04009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:23.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.971+0000 7f4c0d600700 1 -- 172.21.15.43:0/1919079003 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4bfc0203a0 con 0x7f4c10104df0 2024-03-28T22:35:23.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:23.190+0000 7f4c17600700 1 -- 172.21.15.43:0/1919079003 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f4c101a7940 con 0x7f4c10104df0 2024-03-28T22:35:23.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:23.190+0000 7f4c0d600700 1 -- 172.21.15.43:0/1919079003 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f4bfc0ce020 con 0x7f4c10104df0 2024-03-28T22:35:23.329 INFO:teuthology.orchestra.run.smithi043.stdout:352187318309 2024-03-28T22:35:23.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:23.193+0000 7f4c17600700 1 -- 172.21.15.43:0/1919079003 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4bf80a1f20 msgr2=0x7f4bf80a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:23.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:23.193+0000 7f4c17600700 1 --2- 172.21.15.43:0/1919079003 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4bf80a1f20 0x7f4bf80a43e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4c1019bab0 tx=0x7f4c04009260 comp rx=0 tx=0).stop 2024-03-28T22:35:23.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:23.193+0000 7f4c17600700 1 -- 172.21.15.43:0/1919079003 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4c10104df0 msgr2=0x7f4c101a0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:23.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:23.193+0000 7f4c17600700 1 --2- 172.21.15.43:0/1919079003 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4c10104df0 0x7f4c101a0cf0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc00e9c0 tx=0x7f4bfc00ecd0 comp rx=0 tx=0).stop 2024-03-28T22:35:23.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:23.194+0000 7f4c17600700 1 -- 172.21.15.43:0/1919079003 shutdown_connections 2024-03-28T22:35:23.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:23.194+0000 7f4c17600700 1 --2- 172.21.15.43:0/1919079003 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4bf80a1f20 0x7f4bf80a43e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:23.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:23.194+0000 7f4c17600700 1 --2- 172.21.15.43:0/1919079003 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c1010a4c0 0x7f4c1019a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:23.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:23.194+0000 7f4c17600700 1 --2- 172.21.15.43:0/1919079003 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c10104430 0x7f4c101a07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:23.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:23.194+0000 7f4c17600700 1 --2- 172.21.15.43:0/1919079003 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4c10104df0 0x7f4c101a0cf0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:23.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:23.194+0000 7f4c17600700 1 -- 172.21.15.43:0/1919079003 >> 172.21.15.43:0/1919079003 conn(0x7f4c100fff20 msgr2=0x7f4c101003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:23.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:23.194+0000 7f4c17600700 1 -- 172.21.15.43:0/1919079003 shutdown_connections 2024-03-28T22:35:23.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:23.194+0000 7f4c17600700 1 -- 172.21.15.43:0/1919079003 wait complete. 2024-03-28T22:35:23.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:23 smithi043 ceph-mon[31823]: pgmap v381: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:35:23.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:23 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:35:23.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:23 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:35:23.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:23 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2411119990' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T22:35:23.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:23 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/704997235' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T22:35:23.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:23 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1919079003' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T22:35:23.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:23 smithi172 ceph-mon[36566]: pgmap v381: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:35:23.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:23 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:35:23.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:23 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:35:23.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:23 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2411119990' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T22:35:23.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:23 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/704997235' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T22:35:23.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:23 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1919079003' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T22:35:23.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.386+0000 7ff446800700 1 -- 172.21.15.43:0/447938867 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff440069ef0 msgr2=0x7ff44006a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:23.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.386+0000 7ff446800700 1 --2- 172.21.15.43:0/447938867 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff440069ef0 0x7ff44006a350 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff42c009990 tx=0x7ff42c009ca0 comp rx=0 tx=0).stop 2024-03-28T22:35:23.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.387+0000 7ff446800700 1 -- 172.21.15.43:0/447938867 shutdown_connections 2024-03-28T22:35:23.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.387+0000 7ff446800700 1 --2- 172.21.15.43:0/447938867 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff44006a890 0x7ff44010f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:23.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.387+0000 7ff446800700 1 --2- 172.21.15.43:0/447938867 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff440108290 0x7ff440108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:23.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.387+0000 7ff446800700 1 --2- 172.21.15.43:0/447938867 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff440069ef0 0x7ff44006a350 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:23.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.387+0000 7ff446800700 1 -- 172.21.15.43:0/447938867 >> 172.21.15.43:0/447938867 conn(0x7ff440078070 msgr2=0x7ff440078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:23.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.387+0000 7ff446800700 1 -- 172.21.15.43:0/447938867 shutdown_connections 2024-03-28T22:35:23.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.387+0000 7ff446800700 1 -- 172.21.15.43:0/447938867 wait complete. 2024-03-28T22:35:23.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.388+0000 7ff446800700 1 Processor -- start 2024-03-28T22:35:23.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.389+0000 7ff446800700 1 -- start start 2024-03-28T22:35:23.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.389+0000 7ff446800700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff440069ef0 0x7ff44019e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:23.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.389+0000 7ff446800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff44006a890 0x7ff44019eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:23.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.389+0000 7ff446800700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff440108290 0x7ff4401987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:23.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.389+0000 7ff446800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff440112540 con 0x7ff44006a890 2024-03-28T22:35:23.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.389+0000 7ff446800700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ff4401123c0 con 0x7ff440108290 2024-03-28T22:35:23.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.389+0000 7ff446800700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7ff4401126c0 con 0x7ff440069ef0 2024-03-28T22:35:23.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.389+0000 7ff43f400700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff440069ef0 0x7ff44019e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:23.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.389+0000 7ff43f400700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff440069ef0 0x7ff44019e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:50242/0 (socket says 172.21.15.43:50242) 2024-03-28T22:35:23.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.389+0000 7ff43f400700 1 -- 172.21.15.43:0/2411119990 learned_addr learned my addr 172.21.15.43:0/2411119990 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:35:23.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.389+0000 7ff43ea00700 1 --2- 172.21.15.43:0/2411119990 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff44006a890 0x7ff44019eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:23.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.390+0000 7ff43f400700 1 -- 172.21.15.43:0/2411119990 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff440108290 msgr2=0x7ff4401987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:23.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.390+0000 7ff43f400700 1 --2- 172.21.15.43:0/2411119990 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff440108290 0x7ff4401987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:23.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.390+0000 7ff43f400700 1 -- 172.21.15.43:0/2411119990 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff44006a890 msgr2=0x7ff44019eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:23.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.390+0000 7ff43f400700 1 --2- 172.21.15.43:0/2411119990 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff44006a890 0x7ff44019eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:23.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.390+0000 7ff43f400700 1 -- 172.21.15.43:0/2411119990 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff440198fc0 con 0x7ff440069ef0 2024-03-28T22:35:23.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.390+0000 7ff43ea00700 1 --2- 172.21.15.43:0/2411119990 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff44006a890 0x7ff44019eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:35:23.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.390+0000 7ff43f400700 1 --2- 172.21.15.43:0/2411119990 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff440069ef0 0x7ff44019e660 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff43000d980 tx=0x7ff43000dd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:23.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.391+0000 7ff437e00700 1 -- 172.21.15.43:0/2411119990 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff430016070 con 0x7ff440069ef0 2024-03-28T22:35:23.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.391+0000 7ff437e00700 1 -- 172.21.15.43:0/2411119990 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff43000bdc0 con 0x7ff440069ef0 2024-03-28T22:35:23.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.391+0000 7ff446800700 1 -- 172.21.15.43:0/2411119990 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4401992b0 con 0x7ff440069ef0 2024-03-28T22:35:23.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.391+0000 7ff437e00700 1 -- 172.21.15.43:0/2411119990 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff430004cf0 con 0x7ff440069ef0 2024-03-28T22:35:23.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.391+0000 7ff446800700 1 -- 172.21.15.43:0/2411119990 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4401a54c0 con 0x7ff440069ef0 2024-03-28T22:35:23.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.392+0000 7ff446800700 1 -- 172.21.15.43:0/2411119990 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff440103940 con 0x7ff440069ef0 2024-03-28T22:35:23.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.394+0000 7ff437e00700 1 -- 172.21.15.43:0/2411119990 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7ff43000b8e0 con 0x7ff440069ef0 2024-03-28T22:35:23.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.395+0000 7ff437e00700 1 --2- 172.21.15.43:0/2411119990 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff4240a1f20 0x7ff4240a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:23.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.396+0000 7ff437e00700 1 -- 172.21.15.43:0/2411119990 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7ff4300cdfe0 con 0x7ff440069ef0 2024-03-28T22:35:23.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.396+0000 7ff43ea00700 1 --2- 172.21.15.43:0/2411119990 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff4240a1f20 0x7ff4240a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:23.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.397+0000 7ff43ea00700 1 --2- 172.21.15.43:0/2411119990 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff4240a1f20 0x7ff4240a43e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff42c00b2e0 tx=0x7ff42c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:23.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.400+0000 7ff437e00700 1 -- 172.21.15.43:0/2411119990 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff43001f020 con 0x7ff440069ef0 2024-03-28T22:35:23.655 INFO:teuthology.orchestra.run.smithi043.stdout:34359738465 2024-03-28T22:35:23.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.652+0000 7ff446800700 1 -- 172.21.15.43:0/2411119990 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7ff44002dcf0 con 0x7ff440069ef0 2024-03-28T22:35:23.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.653+0000 7ff437e00700 1 -- 172.21.15.43:0/2411119990 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff43009af10 con 0x7ff440069ef0 2024-03-28T22:35:23.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.655+0000 7ff446800700 1 -- 172.21.15.43:0/2411119990 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff4240a1f20 msgr2=0x7ff4240a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:23.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.655+0000 7ff446800700 1 --2- 172.21.15.43:0/2411119990 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff4240a1f20 0x7ff4240a43e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff42c00b2e0 tx=0x7ff42c01a040 comp rx=0 tx=0).stop 2024-03-28T22:35:23.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.655+0000 7ff446800700 1 -- 172.21.15.43:0/2411119990 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff440069ef0 msgr2=0x7ff44019e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:23.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.655+0000 7ff446800700 1 --2- 172.21.15.43:0/2411119990 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff440069ef0 0x7ff44019e660 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff43000d980 tx=0x7ff43000dd40 comp rx=0 tx=0).stop 2024-03-28T22:35:23.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.656+0000 7ff446800700 1 -- 172.21.15.43:0/2411119990 shutdown_connections 2024-03-28T22:35:23.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.656+0000 7ff446800700 1 --2- 172.21.15.43:0/2411119990 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff4240a1f20 0x7ff4240a43e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:23.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.656+0000 7ff446800700 1 --2- 172.21.15.43:0/2411119990 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff440108290 0x7ff4401987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:23.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.656+0000 7ff446800700 1 --2- 172.21.15.43:0/2411119990 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff440069ef0 0x7ff44019e660 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff43000d980 tx=0x7ff43000dd40 comp rx=0 tx=0).stop 2024-03-28T22:35:23.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.656+0000 7ff446800700 1 --2- 172.21.15.43:0/2411119990 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff44006a890 0x7ff44019eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:23.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.656+0000 7ff446800700 1 -- 172.21.15.43:0/2411119990 >> 172.21.15.43:0/2411119990 conn(0x7ff440078070 msgr2=0x7ff4400ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:23.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.656+0000 7ff446800700 1 -- 172.21.15.43:0/2411119990 shutdown_connections 2024-03-28T22:35:23.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:22.657+0000 7ff446800700 1 -- 172.21.15.43:0/2411119990 wait complete. 2024-03-28T22:35:23.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:23 smithi082 ceph-mon[36471]: pgmap v381: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:35:23.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:23 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:35:23.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:23 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:35:23.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:23 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2411119990' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T22:35:23.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:23 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/704997235' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T22:35:23.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:23 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1919079003' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T22:35:24.580 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757312 got 163208757319 for osd.5 2024-03-28T22:35:24.580 DEBUG:teuthology.parallel:result is None 2024-03-28T22:35:25.448 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:25 smithi043 ceph-mon[31823]: pgmap v382: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:35:25.448 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:25 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:35:25.448 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:25 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:35:25.449 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:25 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:35:25.527 INFO:tasks.cephadm.ceph_manager.ceph:need seq 352187318303 got 352187318309 for osd.11 2024-03-28T22:35:25.527 DEBUG:teuthology.parallel:result is None 2024-03-28T22:35:25.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:25 smithi172 ceph-mon[36566]: pgmap v382: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:35:25.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:25 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:35:25.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:25 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:35:25.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:25 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:35:25.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:25 smithi082 ceph-mon[36471]: pgmap v382: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:35:25.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:25 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:35:25.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:25 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:35:25.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:25 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:35:25.867 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738456 got 34359738465 for osd.0 2024-03-28T22:35:25.867 DEBUG:teuthology.parallel:result is None 2024-03-28T22:35:27.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.126+0000 7f0749400700 1 -- 172.21.15.43:0/1790800147 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f07441075b0 msgr2=0x7f074410f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:27.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.126+0000 7f0749400700 1 --2- 172.21.15.43:0/1790800147 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f07441075b0 0x7f074410f130 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f073c00b0a0 tx=0x7f073c00b3b0 comp rx=0 tx=0).stop 2024-03-28T22:35:27.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.126+0000 7f0749400700 1 -- 172.21.15.43:0/1790800147 shutdown_connections 2024-03-28T22:35:27.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.126+0000 7f0749400700 1 --2- 172.21.15.43:0/1790800147 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f07441075b0 0x7f074410f130 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:27.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.126+0000 7f0749400700 1 --2- 172.21.15.43:0/1790800147 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f074406ab40 0x7f074406af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:27.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.126+0000 7f0749400700 1 --2- 172.21.15.43:0/1790800147 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f074406b4f0 0x7f0744106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:27.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.126+0000 7f0749400700 1 -- 172.21.15.43:0/1790800147 >> 172.21.15.43:0/1790800147 conn(0x7f07440780b0 msgr2=0x7f07440784c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:27.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.126+0000 7f0749400700 1 -- 172.21.15.43:0/1790800147 shutdown_connections 2024-03-28T22:35:27.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.126+0000 7f0749400700 1 -- 172.21.15.43:0/1790800147 wait complete. 2024-03-28T22:35:27.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.127+0000 7f0749400700 1 Processor -- start 2024-03-28T22:35:27.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.127+0000 7f0749400700 1 -- start start 2024-03-28T22:35:27.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.128+0000 7f0749400700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f074406ab40 0x7f074419e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:27.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.128+0000 7f0749400700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f074406b4f0 0x7f074419ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:27.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.128+0000 7f0749400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f07441075b0 0x7f07441988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:27.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.128+0000 7f0749400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f07441125a0 con 0x7f07441075b0 2024-03-28T22:35:27.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.128+0000 7f0749400700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f0744112420 con 0x7f074406ab40 2024-03-28T22:35:27.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.128+0000 7f0749400700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f0744112720 con 0x7f074406b4f0 2024-03-28T22:35:27.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.128+0000 7f0742800700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f074406ab40 0x7f074419e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:27.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.128+0000 7f0742800700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f074406ab40 0x7f074419e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:56194/0 (socket says 172.21.15.43:56194) 2024-03-28T22:35:27.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.128+0000 7f073be00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f074406b4f0 0x7f074419ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:27.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.128+0000 7f0742800700 1 -- 172.21.15.43:0/1657021178 learned_addr learned my addr 172.21.15.43:0/1657021178 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:35:27.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.129+0000 7f0743200700 1 --2- 172.21.15.43:0/1657021178 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f07441075b0 0x7f07441988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:27.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.129+0000 7f0742800700 1 -- 172.21.15.43:0/1657021178 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f074406b4f0 msgr2=0x7f074419ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:27.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.129+0000 7f0742800700 1 --2- 172.21.15.43:0/1657021178 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f074406b4f0 0x7f074419ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:27.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.129+0000 7f0742800700 1 -- 172.21.15.43:0/1657021178 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f07441075b0 msgr2=0x7f07441988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:27.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.129+0000 7f0742800700 1 --2- 172.21.15.43:0/1657021178 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f07441075b0 0x7f07441988a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:27.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.129+0000 7f0742800700 1 -- 172.21.15.43:0/1657021178 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0744199000 con 0x7f074406ab40 2024-03-28T22:35:27.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.129+0000 7f0743200700 1 --2- 172.21.15.43:0/1657021178 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f07441075b0 0x7f07441988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:35:27.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.129+0000 7f0742800700 1 --2- 172.21.15.43:0/1657021178 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f074406ab40 0x7f074419e6e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f073400d950 tx=0x7f073400dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:27.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.129+0000 7f073b400700 1 -- 172.21.15.43:0/1657021178 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0734016070 con 0x7f074406ab40 2024-03-28T22:35:27.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.129+0000 7f073b400700 1 -- 172.21.15.43:0/1657021178 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0734009e20 con 0x7f074406ab40 2024-03-28T22:35:27.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.129+0000 7f0749400700 1 -- 172.21.15.43:0/1657021178 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07441992f0 con 0x7f074406ab40 2024-03-28T22:35:27.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.130+0000 7f073b400700 1 -- 172.21.15.43:0/1657021178 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f073400f620 con 0x7f074406ab40 2024-03-28T22:35:27.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.130+0000 7f0749400700 1 -- 172.21.15.43:0/1657021178 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07441a55b0 con 0x7f074406ab40 2024-03-28T22:35:27.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.131+0000 7f0749400700 1 -- 172.21.15.43:0/1657021178 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0744103960 con 0x7f074406ab40 2024-03-28T22:35:27.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.133+0000 7f073b400700 1 -- 172.21.15.43:0/1657021178 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) ==== 136807+0+0 (secure 0 0 0) 0x7f0734019070 con 0x7f074406ab40 2024-03-28T22:35:27.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.134+0000 7f073b400700 1 --2- 172.21.15.43:0/1657021178 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f07280a1ea0 0x7f07280a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:27.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.134+0000 7f073b400700 1 -- 172.21.15.43:0/1657021178 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9092+0+0 (secure 0 0 0) 0x7f07340cdeb0 con 0x7f074406ab40 2024-03-28T22:35:27.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.134+0000 7f073be00700 1 --2- 172.21.15.43:0/1657021178 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f07280a1ea0 0x7f07280a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:27.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.135+0000 7f073be00700 1 --2- 172.21.15.43:0/1657021178 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f07280a1ea0 0x7f07280a4360 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0744100d50 tx=0x7f072c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:27.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.138+0000 7f073b400700 1 -- 172.21.15.43:0/1657021178 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f073409adc0 con 0x7f074406ab40 2024-03-28T22:35:27.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.367+0000 7f0749400700 1 -- 172.21.15.43:0/1657021178 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f074402db30 con 0x7f074406ab40 2024-03-28T22:35:27.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.368+0000 7f073b400700 1 -- 172.21.15.43:0/1657021178 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f073409a6c0 con 0x7f074406ab40 2024-03-28T22:35:27.367 INFO:teuthology.orchestra.run.smithi043.stdout:296352743472 2024-03-28T22:35:27.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.370+0000 7f0749400700 1 -- 172.21.15.43:0/1657021178 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f07280a1ea0 msgr2=0x7f07280a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:27.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.370+0000 7f0749400700 1 --2- 172.21.15.43:0/1657021178 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f07280a1ea0 0x7f07280a4360 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0744100d50 tx=0x7f072c009260 comp rx=0 tx=0).stop 2024-03-28T22:35:27.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.370+0000 7f0749400700 1 -- 172.21.15.43:0/1657021178 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f074406ab40 msgr2=0x7f074419e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:27.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.370+0000 7f0749400700 1 --2- 172.21.15.43:0/1657021178 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f074406ab40 0x7f074419e6e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f073400d950 tx=0x7f073400dc60 comp rx=0 tx=0).stop 2024-03-28T22:35:27.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.371+0000 7f0749400700 1 -- 172.21.15.43:0/1657021178 shutdown_connections 2024-03-28T22:35:27.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.371+0000 7f0749400700 1 --2- 172.21.15.43:0/1657021178 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f07280a1ea0 0x7f07280a4360 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:27.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.371+0000 7f0749400700 1 --2- 172.21.15.43:0/1657021178 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f07441075b0 0x7f07441988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:27.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.371+0000 7f0749400700 1 --2- 172.21.15.43:0/1657021178 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f074406ab40 0x7f074419e6e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:27.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.371+0000 7f0749400700 1 --2- 172.21.15.43:0/1657021178 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f074406b4f0 0x7f074419ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:27.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.371+0000 7f0749400700 1 -- 172.21.15.43:0/1657021178 >> 172.21.15.43:0/1657021178 conn(0x7f07440780b0 msgr2=0x7f07440ff400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:27.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.371+0000 7f0749400700 1 -- 172.21.15.43:0/1657021178 shutdown_connections 2024-03-28T22:35:27.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:27.371+0000 7f0749400700 1 -- 172.21.15.43:0/1657021178 wait complete. 2024-03-28T22:35:27.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:27 smithi043 ceph-mon[31823]: pgmap v383: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T22:35:27.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:27 smithi172 ceph-mon[36566]: pgmap v383: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T22:35:27.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:27 smithi082 ceph-mon[36471]: pgmap v383: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T22:35:28.209 INFO:tasks.cephadm.ceph_manager.ceph:need seq 296352743466 got 296352743472 for osd.9 2024-03-28T22:35:28.209 DEBUG:teuthology.parallel:result is None 2024-03-28T22:35:28.209 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-28T22:35:28.210 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-28T22:35:28.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:28 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1657021178' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T22:35:28.641 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:35:28.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:28 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1657021178' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T22:35:28.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:28 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1657021178' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T22:35:29.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:29 smithi043 ceph-mon[31823]: pgmap v384: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:35:29.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:29 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-03-28T22:35:29.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:29 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 10]}]: dispatch 2024-03-28T22:35:29.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:29 smithi172 ceph-mon[36566]: pgmap v384: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:35:29.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:29 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-03-28T22:35:29.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:29 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 10]}]: dispatch 2024-03-28T22:35:29.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:29 smithi082 ceph-mon[36471]: pgmap v384: 129 pgs: 129 active+clean; 646 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:35:29.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:29 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-03-28T22:35:29.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:29 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 10]}]: dispatch 2024-03-28T22:35:30.450 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:30 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-03-28T22:35:30.450 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:30 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 10]}]': finished 2024-03-28T22:35:30.450 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:30 smithi043 ceph-mon[31823]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T22:35:30.450 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:30 smithi043 ceph-mon[31823]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-03-28T22:35:30.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:30 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-03-28T22:35:30.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:30 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 10]}]': finished 2024-03-28T22:35:30.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:30 smithi172 ceph-mon[36566]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T22:35:30.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:30 smithi172 ceph-mon[36566]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-03-28T22:35:30.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:30 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-03-28T22:35:30.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:30 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 10]}]': finished 2024-03-28T22:35:30.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:30 smithi082 ceph-mon[36471]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T22:35:30.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:30 smithi082 ceph-mon[36471]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-03-28T22:35:31.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.030+0000 7fd1b9400700 1 -- 172.21.15.43:0/1942545704 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1b410a4c0 msgr2=0x7fd1b410a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:31.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.030+0000 7fd1b9400700 1 --2- 172.21.15.43:0/1942545704 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1b410a4c0 0x7fd1b410a8a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4009990 tx=0x7fd1a4009ca0 comp rx=0 tx=0).stop 2024-03-28T22:35:31.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.030+0000 7fd1b9400700 1 -- 172.21.15.43:0/1942545704 shutdown_connections 2024-03-28T22:35:31.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.030+0000 7fd1b9400700 1 --2- 172.21.15.43:0/1942545704 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1b4104df0 0x7fd1b4111300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:31.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.030+0000 7fd1b9400700 1 --2- 172.21.15.43:0/1942545704 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1b410a4c0 0x7fd1b410a8a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:31.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.030+0000 7fd1b9400700 1 --2- 172.21.15.43:0/1942545704 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1b4104430 0x7fd1b41048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:31.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.030+0000 7fd1b9400700 1 -- 172.21.15.43:0/1942545704 >> 172.21.15.43:0/1942545704 conn(0x7fd1b40fff20 msgr2=0x7fd1b4102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:31.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.030+0000 7fd1b9400700 1 -- 172.21.15.43:0/1942545704 shutdown_connections 2024-03-28T22:35:31.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.030+0000 7fd1b9400700 1 -- 172.21.15.43:0/1942545704 wait complete. 2024-03-28T22:35:31.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.032+0000 7fd1b9400700 1 Processor -- start 2024-03-28T22:35:31.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.032+0000 7fd1b9400700 1 -- start start 2024-03-28T22:35:31.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.032+0000 7fd1b9400700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1b4104430 0x7fd1b419abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:31.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.032+0000 7fd1b9400700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1b4104df0 0x7fd1b419b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:31.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.032+0000 7fd1b9400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1b410a4c0 0x7fd1b419f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:31.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.032+0000 7fd1b9400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b4114720 con 0x7fd1b410a4c0 2024-03-28T22:35:31.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.032+0000 7fd1b9400700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b41145a0 con 0x7fd1b4104df0 2024-03-28T22:35:31.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.032+0000 7fd1b9400700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b41148a0 con 0x7fd1b4104430 2024-03-28T22:35:31.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.033+0000 7fd1b2800700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1b4104430 0x7fd1b419abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:31.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.033+0000 7fd1b1e00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1b4104df0 0x7fd1b419b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:31.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.033+0000 7fd1b2800700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1b4104430 0x7fd1b419abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:34420/0 (socket says 172.21.15.43:34420) 2024-03-28T22:35:31.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.033+0000 7fd1b2800700 1 -- 172.21.15.43:0/510755582 learned_addr learned my addr 172.21.15.43:0/510755582 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:35:31.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.033+0000 7fd1b1e00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1b4104df0 0x7fd1b419b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:56214/0 (socket says 172.21.15.43:56214) 2024-03-28T22:35:31.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.033+0000 7fd1b3200700 1 --2- 172.21.15.43:0/510755582 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1b410a4c0 0x7fd1b419f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:31.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.033+0000 7fd1b1e00700 1 -- 172.21.15.43:0/510755582 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1b4104430 msgr2=0x7fd1b419abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:31.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.033+0000 7fd1b1e00700 1 --2- 172.21.15.43:0/510755582 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1b4104430 0x7fd1b419abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:31.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.033+0000 7fd1b1e00700 1 -- 172.21.15.43:0/510755582 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1b410a4c0 msgr2=0x7fd1b419f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:31.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.033+0000 7fd1b1e00700 1 --2- 172.21.15.43:0/510755582 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1b410a4c0 0x7fd1b419f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:31.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.033+0000 7fd1b1e00700 1 -- 172.21.15.43:0/510755582 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1b419fbb0 con 0x7fd1b4104df0 2024-03-28T22:35:31.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.033+0000 7fd1b3200700 1 --2- 172.21.15.43:0/510755582 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1b410a4c0 0x7fd1b419f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:35:31.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.034+0000 7fd1b2800700 1 --2- 172.21.15.43:0/510755582 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1b4104430 0x7fd1b419abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:35:31.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.034+0000 7fd1b1e00700 1 --2- 172.21.15.43:0/510755582 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1b4104df0 0x7fd1b419b0f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd1a000e9f0 tx=0x7fd1a000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:31.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.034+0000 7fd1ab400700 1 -- 172.21.15.43:0/510755582 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a000cbd0 con 0x7fd1b4104df0 2024-03-28T22:35:31.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.034+0000 7fd1ab400700 1 -- 172.21.15.43:0/510755582 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1a000cd70 con 0x7fd1b4104df0 2024-03-28T22:35:31.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.034+0000 7fd1b9400700 1 -- 172.21.15.43:0/510755582 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1b419fea0 con 0x7fd1b4104df0 2024-03-28T22:35:31.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.034+0000 7fd1ab400700 1 -- 172.21.15.43:0/510755582 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a0018780 con 0x7fd1b4104df0 2024-03-28T22:35:31.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.034+0000 7fd1b9400700 1 -- 172.21.15.43:0/510755582 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1b41a7760 con 0x7fd1b4104df0 2024-03-28T22:35:31.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.035+0000 7fd1b9400700 1 -- 172.21.15.43:0/510755582 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1b410c9c0 con 0x7fd1b4104df0 2024-03-28T22:35:31.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.037+0000 7fd1ab400700 1 -- 172.21.15.43:0/510755582 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fd1a0010430 con 0x7fd1b4104df0 2024-03-28T22:35:31.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.039+0000 7fd1ab400700 1 --2- 172.21.15.43:0/510755582 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd1980a1e90 0x7fd1980a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:31.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.039+0000 7fd1ab400700 1 -- 172.21.15.43:0/510755582 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7fd1a00ce2e0 con 0x7fd1b4104df0 2024-03-28T22:35:31.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.039+0000 7fd1b2800700 1 --2- 172.21.15.43:0/510755582 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd1980a1e90 0x7fd1980a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:31.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.040+0000 7fd1b2800700 1 --2- 172.21.15.43:0/510755582 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd1980a1e90 0x7fd1980a4350 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4006010 tx=0x7fd1a4009570 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:31.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.043+0000 7fd1ab400700 1 -- 172.21.15.43:0/510755582 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd1a009b320 con 0x7fd1b4104df0 2024-03-28T22:35:31.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.266+0000 7fd1b9400700 1 -- 172.21.15.43:0/510755582 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd1b4062680 con 0x7fd1980a1e90 2024-03-28T22:35:31.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.298+0000 7fd1ab400700 1 -- 172.21.15.43:0/510755582 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340456 (secure 0 0 0) 0x7fd1b4062680 con 0x7fd1980a1e90 2024-03-28T22:35:31.299 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:35:31.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.305+0000 7fd1b9400700 1 -- 172.21.15.43:0/510755582 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd1980a1e90 msgr2=0x7fd1980a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:31.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.305+0000 7fd1b9400700 1 --2- 172.21.15.43:0/510755582 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd1980a1e90 0x7fd1980a4350 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4006010 tx=0x7fd1a4009570 comp rx=0 tx=0).stop 2024-03-28T22:35:31.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.305+0000 7fd1b9400700 1 -- 172.21.15.43:0/510755582 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1b4104df0 msgr2=0x7fd1b419b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:31.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.305+0000 7fd1b9400700 1 --2- 172.21.15.43:0/510755582 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1b4104df0 0x7fd1b419b0f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd1a000e9f0 tx=0x7fd1a000ed00 comp rx=0 tx=0).stop 2024-03-28T22:35:31.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.305+0000 7fd1b9400700 1 -- 172.21.15.43:0/510755582 shutdown_connections 2024-03-28T22:35:31.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.305+0000 7fd1b9400700 1 --2- 172.21.15.43:0/510755582 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd1980a1e90 0x7fd1980a4350 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:31.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.305+0000 7fd1b9400700 1 --2- 172.21.15.43:0/510755582 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1b410a4c0 0x7fd1b419f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:31.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.305+0000 7fd1b9400700 1 --2- 172.21.15.43:0/510755582 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1b4104430 0x7fd1b419abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:31.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.305+0000 7fd1b9400700 1 --2- 172.21.15.43:0/510755582 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd1b4104df0 0x7fd1b419b0f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:31.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.305+0000 7fd1b9400700 1 -- 172.21.15.43:0/510755582 >> 172.21.15.43:0/510755582 conn(0x7fd1b40fff20 msgr2=0x7fd1b4100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:31.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.305+0000 7fd1b9400700 1 -- 172.21.15.43:0/510755582 shutdown_connections 2024-03-28T22:35:31.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:31.305+0000 7fd1b9400700 1 -- 172.21.15.43:0/510755582 wait complete. 2024-03-28T22:35:31.308 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-03-28T22:35:31.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:31 smithi043 ceph-mon[31823]: pgmap v386: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-28T22:35:31.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:31 smithi043 ceph-mon[31823]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-28T22:35:31.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:31 smithi043 ceph-mon[31823]: osdmap e96: 12 total, 12 up, 12 in 2024-03-28T22:35:31.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:31 smithi172 ceph-mon[36566]: pgmap v386: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-28T22:35:31.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:31 smithi172 ceph-mon[36566]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-28T22:35:31.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:31 smithi172 ceph-mon[36566]: osdmap e96: 12 total, 12 up, 12 in 2024-03-28T22:35:31.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:31 smithi082 ceph-mon[36471]: pgmap v386: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-28T22:35:31.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:31 smithi082 ceph-mon[36471]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-28T22:35:31.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:31 smithi082 ceph-mon[36471]: osdmap e96: 12 total, 12 up, 12 in 2024-03-28T22:35:32.140 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":387,"stamp":"2024-03-28T22:35:30.247177+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661870,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":999,"num_read_kb":630,"num_write":429,"num_write_kb":3084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":620,"ondisk_log_size":620,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":376348,"kb_used_data":14672,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124565476,"statfs":{"total":1151940427776,"available":1151555047424,"internally_reserved":0,"allocated":15024128,"data_stored":8203622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":249,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":86,"num_read_kb":46,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.046355"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.976796+0000","last_change":"2024-03-28T22:33:28.942904+0000","last_active":"2024-03-28T22:34:29.976796+0000","last_peered":"2024-03-28T22:34:29.976796+0000","last_clean":"2024-03-28T22:34:29.976796+0000","last_became_active":"2024-03-28T22:33:28.942574+0000","last_became_peered":"2024-03-28T22:33:28.942574+0000","last_unstale":"2024-03-28T22:34:29.976796+0000","last_undegraded":"2024-03-28T22:34:29.976796+0000","last_fullsized":"2024-03-28T22:34:29.976796+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:55.323919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"94'4","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:35:28.175621+0000","last_change":"2024-03-28T22:33:28.951795+0000","last_active":"2024-03-28T22:35:28.175621+0000","last_peered":"2024-03-28T22:35:28.175621+0000","last_clean":"2024-03-28T22:35:28.175621+0000","last_became_active":"2024-03-28T22:33:28.951286+0000","last_became_peered":"2024-03-28T22:33:28.951286+0000","last_unstale":"2024-03-28T22:35:28.175621+0000","last_undegraded":"2024-03-28T22:35:28.175621+0000","last_fullsized":"2024-03-28T22:35:28.175621+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:56:11.718244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"94'7","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:35:28.176066+0000","last_change":"2024-03-28T22:33:28.949476+0000","last_active":"2024-03-28T22:35:28.176066+0000","last_peered":"2024-03-28T22:35:28.176066+0000","last_clean":"2024-03-28T22:35:28.176066+0000","last_became_active":"2024-03-28T22:33:28.949128+0000","last_became_peered":"2024-03-28T22:33:28.949128+0000","last_unstale":"2024-03-28T22:35:28.176066+0000","last_undegraded":"2024-03-28T22:35:28.176066+0000","last_fullsized":"2024-03-28T22:35:28.176066+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:49:29.052373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.978076+0000","last_change":"2024-03-28T22:33:28.930715+0000","last_active":"2024-03-28T22:34:29.978076+0000","last_peered":"2024-03-28T22:34:29.978076+0000","last_clean":"2024-03-28T22:34:29.978076+0000","last_became_active":"2024-03-28T22:33:28.930265+0000","last_became_peered":"2024-03-28T22:33:28.930265+0000","last_unstale":"2024-03-28T22:34:29.978076+0000","last_undegraded":"2024-03-28T22:34:29.978076+0000","last_fullsized":"2024-03-28T22:34:29.978076+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:29:54.695671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"94'4","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:35:28.183289+0000","last_change":"2024-03-28T22:33:28.944933+0000","last_active":"2024-03-28T22:35:28.183289+0000","last_peered":"2024-03-28T22:35:28.183289+0000","last_clean":"2024-03-28T22:35:28.183289+0000","last_became_active":"2024-03-28T22:33:28.944542+0000","last_became_peered":"2024-03-28T22:33:28.944542+0000","last_unstale":"2024-03-28T22:35:28.183289+0000","last_undegraded":"2024-03-28T22:35:28.183289+0000","last_fullsized":"2024-03-28T22:35:28.183289+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:35:15.719312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"94'6","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-28T22:35:29.244883+0000","last_change":"2024-03-28T22:33:28.927920+0000","last_active":"2024-03-28T22:35:29.244883+0000","last_peered":"2024-03-28T22:35:29.244883+0000","last_clean":"2024-03-28T22:35:29.244883+0000","last_became_active":"2024-03-28T22:33:28.927602+0000","last_became_peered":"2024-03-28T22:33:28.927602+0000","last_unstale":"2024-03-28T22:35:29.244883+0000","last_undegraded":"2024-03-28T22:35:29.244883+0000","last_fullsized":"2024-03-28T22:35:29.244883+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:24:22.072707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"94'5","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:35:28.170802+0000","last_change":"2024-03-28T22:33:28.917963+0000","last_active":"2024-03-28T22:35:28.170802+0000","last_peered":"2024-03-28T22:35:28.170802+0000","last_clean":"2024-03-28T22:35:28.170802+0000","last_became_active":"2024-03-28T22:33:28.917628+0000","last_became_peered":"2024-03-28T22:33:28.917628+0000","last_unstale":"2024-03-28T22:35:28.170802+0000","last_undegraded":"2024-03-28T22:35:28.170802+0000","last_fullsized":"2024-03-28T22:35:28.170802+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:37:09.111640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:30.427514+0000","last_change":"2024-03-28T22:33:28.942632+0000","last_active":"2024-03-28T22:34:30.427514+0000","last_peered":"2024-03-28T22:34:30.427514+0000","last_clean":"2024-03-28T22:34:30.427514+0000","last_became_active":"2024-03-28T22:33:28.941972+0000","last_became_peered":"2024-03-28T22:33:28.941972+0000","last_unstale":"2024-03-28T22:34:30.427514+0000","last_undegraded":"2024-03-28T22:34:30.427514+0000","last_fullsized":"2024-03-28T22:34:30.427514+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:49:52.642314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"94'11","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-28T22:35:29.247332+0000","last_change":"2024-03-28T22:33:28.924324+0000","last_active":"2024-03-28T22:35:29.247332+0000","last_peered":"2024-03-28T22:35:29.247332+0000","last_clean":"2024-03-28T22:35:29.247332+0000","last_became_active":"2024-03-28T22:33:28.924014+0000","last_became_peered":"2024-03-28T22:33:28.924014+0000","last_unstale":"2024-03-28T22:35:29.247332+0000","last_undegraded":"2024-03-28T22:35:29.247332+0000","last_fullsized":"2024-03-28T22:35:29.247332+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:17:55.172116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"94'16","reported_seq":51,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-28T22:35:29.245358+0000","last_change":"2024-03-28T22:34:29.987262+0000","last_active":"2024-03-28T22:35:29.245358+0000","last_peered":"2024-03-28T22:35:29.245358+0000","last_clean":"2024-03-28T22:35:29.245358+0000","last_became_active":"2024-03-28T22:34:29.986934+0000","last_became_peered":"2024-03-28T22:34:29.986934+0000","last_unstale":"2024-03-28T22:35:29.245358+0000","last_undegraded":"2024-03-28T22:35:29.245358+0000","last_fullsized":"2024-03-28T22:35:29.245358+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:00:41.574800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"94'6","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:35:28.163125+0000","last_change":"2024-03-28T22:33:28.930802+0000","last_active":"2024-03-28T22:35:28.163125+0000","last_peered":"2024-03-28T22:35:28.163125+0000","last_clean":"2024-03-28T22:35:28.163125+0000","last_became_active":"2024-03-28T22:33:28.930475+0000","last_became_peered":"2024-03-28T22:33:28.930475+0000","last_unstale":"2024-03-28T22:35:28.163125+0000","last_undegraded":"2024-03-28T22:35:28.163125+0000","last_fullsized":"2024-03-28T22:35:28.163125+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:48.443803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.978174+0000","last_change":"2024-03-28T22:33:28.927057+0000","last_active":"2024-03-28T22:34:29.978174+0000","last_peered":"2024-03-28T22:34:29.978174+0000","last_clean":"2024-03-28T22:34:29.978174+0000","last_became_active":"2024-03-28T22:33:28.926598+0000","last_became_peered":"2024-03-28T22:33:28.926598+0000","last_unstale":"2024-03-28T22:34:29.978174+0000","last_undegraded":"2024-03-28T22:34:29.978174+0000","last_fullsized":"2024-03-28T22:34:29.978174+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:21:00.710203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:30.152101+0000","last_change":"2024-03-28T22:33:28.937034+0000","last_active":"2024-03-28T22:34:30.152101+0000","last_peered":"2024-03-28T22:34:30.152101+0000","last_clean":"2024-03-28T22:34:30.152101+0000","last_became_active":"2024-03-28T22:33:28.933713+0000","last_became_peered":"2024-03-28T22:33:28.933713+0000","last_unstale":"2024-03-28T22:34:30.152101+0000","last_undegraded":"2024-03-28T22:34:30.152101+0000","last_fullsized":"2024-03-28T22:34:30.152101+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:58:24.148897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"94'7","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:35:28.167290+0000","last_change":"2024-03-28T22:33:28.947384+0000","last_active":"2024-03-28T22:35:28.167290+0000","last_peered":"2024-03-28T22:35:28.167290+0000","last_clean":"2024-03-28T22:35:28.167290+0000","last_became_active":"2024-03-28T22:33:28.947017+0000","last_became_peered":"2024-03-28T22:33:28.947017+0000","last_unstale":"2024-03-28T22:35:28.167290+0000","last_undegraded":"2024-03-28T22:35:28.167290+0000","last_fullsized":"2024-03-28T22:35:28.167290+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:43:32.781161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"94'7","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:35:28.168865+0000","last_change":"2024-03-28T22:33:28.950119+0000","last_active":"2024-03-28T22:35:28.168865+0000","last_peered":"2024-03-28T22:35:28.168865+0000","last_clean":"2024-03-28T22:35:28.168865+0000","last_became_active":"2024-03-28T22:33:28.949798+0000","last_became_peered":"2024-03-28T22:33:28.949798+0000","last_unstale":"2024-03-28T22:35:28.168865+0000","last_undegraded":"2024-03-28T22:35:28.168865+0000","last_fullsized":"2024-03-28T22:35:28.168865+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:16.717678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"94'9","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-28T22:35:29.246476+0000","last_change":"2024-03-28T22:33:28.943255+0000","last_active":"2024-03-28T22:35:29.246476+0000","last_peered":"2024-03-28T22:35:29.246476+0000","last_clean":"2024-03-28T22:35:29.246476+0000","last_became_active":"2024-03-28T22:33:28.942923+0000","last_became_peered":"2024-03-28T22:33:28.942923+0000","last_unstale":"2024-03-28T22:35:29.246476+0000","last_undegraded":"2024-03-28T22:35:29.246476+0000","last_fullsized":"2024-03-28T22:35:29.246476+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:05:04.197635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.976631+0000","last_change":"2024-03-28T22:33:28.942906+0000","last_active":"2024-03-28T22:34:29.976631+0000","last_peered":"2024-03-28T22:34:29.976631+0000","last_clean":"2024-03-28T22:34:29.976631+0000","last_became_active":"2024-03-28T22:33:28.942573+0000","last_became_peered":"2024-03-28T22:33:28.942573+0000","last_unstale":"2024-03-28T22:34:29.976631+0000","last_undegraded":"2024-03-28T22:34:29.976631+0000","last_fullsized":"2024-03-28T22:34:29.976631+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:39:17.651101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.983193+0000","last_change":"2024-03-28T22:33:28.932075+0000","last_active":"2024-03-28T22:34:29.983193+0000","last_peered":"2024-03-28T22:34:29.983193+0000","last_clean":"2024-03-28T22:34:29.983193+0000","last_became_active":"2024-03-28T22:33:28.931689+0000","last_became_peered":"2024-03-28T22:33:28.931689+0000","last_unstale":"2024-03-28T22:34:29.983193+0000","last_undegraded":"2024-03-28T22:34:29.983193+0000","last_fullsized":"2024-03-28T22:34:29.983193+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:50:43.031082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"94'7","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:35:28.182863+0000","last_change":"2024-03-28T22:33:28.944987+0000","last_active":"2024-03-28T22:35:28.182863+0000","last_peered":"2024-03-28T22:35:28.182863+0000","last_clean":"2024-03-28T22:35:28.182863+0000","last_became_active":"2024-03-28T22:33:28.944622+0000","last_became_peered":"2024-03-28T22:33:28.944622+0000","last_unstale":"2024-03-28T22:35:28.182863+0000","last_undegraded":"2024-03-28T22:35:28.182863+0000","last_fullsized":"2024-03-28T22:35:28.182863+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:48:58.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:30.429709+0000","last_change":"2024-03-28T22:34:30.429709+0000","last_active":"2024-03-28T22:34:30.429709+0000","last_peered":"2024-03-28T22:34:30.429709+0000","last_clean":"2024-03-28T22:34:30.429709+0000","last_became_active":"2024-03-28T22:34:30.429122+0000","last_became_peered":"2024-03-28T22:34:30.429122+0000","last_unstale":"2024-03-28T22:34:30.429709+0000","last_undegraded":"2024-03-28T22:34:30.429709+0000","last_fullsized":"2024-03-28T22:34:30.429709+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:58:05.313669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"92'13","reported_seq":124,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:35:25.311111+0000","last_change":"2024-03-28T22:33:28.442524+0000","last_active":"2024-03-28T22:35:25.311111+0000","last_peered":"2024-03-28T22:35:25.311111+0000","last_clean":"2024-03-28T22:35:25.311111+0000","last_became_active":"2024-03-28T22:33:28.442169+0000","last_became_peered":"2024-03-28T22:33:28.442169+0000","last_unstale":"2024-03-28T22:35:25.311111+0000","last_undegraded":"2024-03-28T22:35:25.311111+0000","last_fullsized":"2024-03-28T22:35:25.311111+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:08:47.101119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":82,"num_read_kb":82,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:30.427904+0000","last_change":"2024-03-28T22:33:28.946697+0000","last_active":"2024-03-28T22:34:30.427904+0000","last_peered":"2024-03-28T22:34:30.427904+0000","last_clean":"2024-03-28T22:34:30.427904+0000","last_became_active":"2024-03-28T22:33:28.946303+0000","last_became_peered":"2024-03-28T22:33:28.946303+0000","last_unstale":"2024-03-28T22:34:30.427904+0000","last_undegraded":"2024-03-28T22:34:30.427904+0000","last_fullsized":"2024-03-28T22:34:30.427904+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:32:01.286073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:30.429579+0000","last_change":"2024-03-28T22:34:30.429579+0000","last_active":"2024-03-28T22:34:30.429579+0000","last_peered":"2024-03-28T22:34:30.429579+0000","last_clean":"2024-03-28T22:34:30.429579+0000","last_became_active":"2024-03-28T22:34:30.429054+0000","last_became_peered":"2024-03-28T22:34:30.429054+0000","last_unstale":"2024-03-28T22:34:30.429579+0000","last_undegraded":"2024-03-28T22:34:30.429579+0000","last_fullsized":"2024-03-28T22:34:30.429579+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:03:46.492786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"92'2","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:30.428295+0000","last_change":"2024-03-28T22:33:28.457661+0000","last_active":"2024-03-28T22:34:30.428295+0000","last_peered":"2024-03-28T22:34:30.428295+0000","last_clean":"2024-03-28T22:34:30.428295+0000","last_became_active":"2024-03-28T22:33:28.457032+0000","last_became_peered":"2024-03-28T22:33:28.457032+0000","last_unstale":"2024-03-28T22:34:30.428295+0000","last_undegraded":"2024-03-28T22:34:30.428295+0000","last_fullsized":"2024-03-28T22:34:30.428295+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:43:09.820630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"94'6","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:35:28.181826+0000","last_change":"2024-03-28T22:33:28.954252+0000","last_active":"2024-03-28T22:35:28.181826+0000","last_peered":"2024-03-28T22:35:28.181826+0000","last_clean":"2024-03-28T22:35:28.181826+0000","last_became_active":"2024-03-28T22:33:28.953922+0000","last_became_peered":"2024-03-28T22:33:28.953922+0000","last_unstale":"2024-03-28T22:35:28.181826+0000","last_undegraded":"2024-03-28T22:35:28.181826+0000","last_fullsized":"2024-03-28T22:35:28.181826+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:18:47.097859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.977953+0000","last_change":"2024-03-28T22:33:34.029129+0000","last_active":"2024-03-28T22:34:29.977953+0000","last_peered":"2024-03-28T22:34:29.977953+0000","last_clean":"2024-03-28T22:34:29.977953+0000","last_became_active":"2024-03-28T22:33:34.028452+0000","last_became_peered":"2024-03-28T22:33:34.028452+0000","last_unstale":"2024-03-28T22:34:29.977953+0000","last_undegraded":"2024-03-28T22:34:29.977953+0000","last_fullsized":"2024-03-28T22:34:29.977953+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:28.982773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"92'12","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.980979+0000","last_change":"2024-03-28T22:33:28.439568+0000","last_active":"2024-03-28T22:34:29.980979+0000","last_peered":"2024-03-28T22:34:29.980979+0000","last_clean":"2024-03-28T22:34:29.980979+0000","last_became_active":"2024-03-28T22:33:28.438887+0000","last_became_peered":"2024-03-28T22:33:28.438887+0000","last_unstale":"2024-03-28T22:34:29.980979+0000","last_undegraded":"2024-03-28T22:34:29.980979+0000","last_fullsized":"2024-03-28T22:34:29.980979+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:56:43.617621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"94'9","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-28T22:35:29.247118+0000","last_change":"2024-03-28T22:33:28.938311+0000","last_active":"2024-03-28T22:35:29.247118+0000","last_peered":"2024-03-28T22:35:29.247118+0000","last_clean":"2024-03-28T22:35:29.247118+0000","last_became_active":"2024-03-28T22:33:28.937997+0000","last_became_peered":"2024-03-28T22:33:28.937997+0000","last_unstale":"2024-03-28T22:35:29.247118+0000","last_undegraded":"2024-03-28T22:35:29.247118+0000","last_fullsized":"2024-03-28T22:35:29.247118+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:54:44.001545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.977763+0000","last_change":"2024-03-28T22:33:34.016931+0000","last_active":"2024-03-28T22:34:29.977763+0000","last_peered":"2024-03-28T22:34:29.977763+0000","last_clean":"2024-03-28T22:34:29.977763+0000","last_became_active":"2024-03-28T22:33:34.016559+0000","last_became_peered":"2024-03-28T22:33:34.016559+0000","last_unstale":"2024-03-28T22:34:29.977763+0000","last_undegraded":"2024-03-28T22:34:29.977763+0000","last_fullsized":"2024-03-28T22:34:29.977763+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:08:31.252351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"92'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.980434+0000","last_change":"2024-03-28T22:33:28.433166+0000","last_active":"2024-03-28T22:34:29.980434+0000","last_peered":"2024-03-28T22:34:29.980434+0000","last_clean":"2024-03-28T22:34:29.980434+0000","last_became_active":"2024-03-28T22:33:28.432716+0000","last_became_peered":"2024-03-28T22:33:28.432716+0000","last_unstale":"2024-03-28T22:34:29.980434+0000","last_undegraded":"2024-03-28T22:34:29.980434+0000","last_fullsized":"2024-03-28T22:34:29.980434+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:42:41.613583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.978763+0000","last_change":"2024-03-28T22:33:28.923307+0000","last_active":"2024-03-28T22:34:29.978763+0000","last_peered":"2024-03-28T22:34:29.978763+0000","last_clean":"2024-03-28T22:34:29.978763+0000","last_became_active":"2024-03-28T22:33:28.922499+0000","last_became_peered":"2024-03-28T22:33:28.922499+0000","last_unstale":"2024-03-28T22:34:29.978763+0000","last_undegraded":"2024-03-28T22:34:29.978763+0000","last_fullsized":"2024-03-28T22:34:29.978763+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:48:48.514482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.974504+0000","last_change":"2024-03-28T22:33:34.002209+0000","last_active":"2024-03-28T22:34:29.974504+0000","last_peered":"2024-03-28T22:34:29.974504+0000","last_clean":"2024-03-28T22:34:29.974504+0000","last_became_active":"2024-03-28T22:33:34.001746+0000","last_became_peered":"2024-03-28T22:33:34.001746+0000","last_unstale":"2024-03-28T22:34:29.974504+0000","last_undegraded":"2024-03-28T22:34:29.974504+0000","last_fullsized":"2024-03-28T22:34:29.974504+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:02:02.333328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"92'5","reported_seq":117,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:35:25.326748+0000","last_change":"2024-03-28T22:33:28.435380+0000","last_active":"2024-03-28T22:35:25.326748+0000","last_peered":"2024-03-28T22:35:25.326748+0000","last_clean":"2024-03-28T22:35:25.326748+0000","last_became_active":"2024-03-28T22:33:28.434898+0000","last_became_peered":"2024-03-28T22:33:28.434898+0000","last_unstale":"2024-03-28T22:35:25.326748+0000","last_undegraded":"2024-03-28T22:35:25.326748+0000","last_fullsized":"2024-03-28T22:35:25.326748+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:58:06.810394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"94'7","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:35:28.178173+0000","last_change":"2024-03-28T22:34:30.121370+0000","last_active":"2024-03-28T22:35:28.178173+0000","last_peered":"2024-03-28T22:35:28.178173+0000","last_clean":"2024-03-28T22:35:28.178173+0000","last_became_active":"2024-03-28T22:34:30.121086+0000","last_became_peered":"2024-03-28T22:34:30.121086+0000","last_unstale":"2024-03-28T22:35:28.178173+0000","last_undegraded":"2024-03-28T22:35:28.178173+0000","last_fullsized":"2024-03-28T22:35:28.178173+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:26:24.244663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.975016+0000","last_change":"2024-03-28T22:33:34.031350+0000","last_active":"2024-03-28T22:34:29.975016+0000","last_peered":"2024-03-28T22:34:29.975016+0000","last_clean":"2024-03-28T22:34:29.975016+0000","last_became_active":"2024-03-28T22:33:34.030946+0000","last_became_peered":"2024-03-28T22:33:34.030946+0000","last_unstale":"2024-03-28T22:34:29.975016+0000","last_undegraded":"2024-03-28T22:34:29.975016+0000","last_fullsized":"2024-03-28T22:34:29.975016+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:54:22.907363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"92'7","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-28T22:35:29.244791+0000","last_change":"2024-03-28T22:33:28.446197+0000","last_active":"2024-03-28T22:35:29.244791+0000","last_peered":"2024-03-28T22:35:29.244791+0000","last_clean":"2024-03-28T22:35:29.244791+0000","last_became_active":"2024-03-28T22:33:28.445891+0000","last_became_peered":"2024-03-28T22:33:28.445891+0000","last_unstale":"2024-03-28T22:35:29.244791+0000","last_undegraded":"2024-03-28T22:35:29.244791+0000","last_fullsized":"2024-03-28T22:35:29.244791+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:58:57.385302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.978479+0000","last_change":"2024-03-28T22:33:28.955651+0000","last_active":"2024-03-28T22:34:29.978479+0000","last_peered":"2024-03-28T22:34:29.978479+0000","last_clean":"2024-03-28T22:34:29.978479+0000","last_became_active":"2024-03-28T22:33:28.955276+0000","last_became_peered":"2024-03-28T22:33:28.955276+0000","last_unstale":"2024-03-28T22:34:29.978479+0000","last_undegraded":"2024-03-28T22:34:29.978479+0000","last_fullsized":"2024-03-28T22:34:29.978479+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:36:13.455327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.976524+0000","last_change":"2024-03-28T22:33:34.017520+0000","last_active":"2024-03-28T22:34:29.976524+0000","last_peered":"2024-03-28T22:34:29.976524+0000","last_clean":"2024-03-28T22:34:29.976524+0000","last_became_active":"2024-03-28T22:33:34.017022+0000","last_became_peered":"2024-03-28T22:33:34.017022+0000","last_unstale":"2024-03-28T22:34:29.976524+0000","last_undegraded":"2024-03-28T22:34:29.976524+0000","last_fullsized":"2024-03-28T22:34:29.976524+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:41:17.431581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"92'3","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:30.151221+0000","last_change":"2024-03-28T22:33:28.445401+0000","last_active":"2024-03-28T22:34:30.151221+0000","last_peered":"2024-03-28T22:34:30.151221+0000","last_clean":"2024-03-28T22:34:30.151221+0000","last_became_active":"2024-03-28T22:33:28.444237+0000","last_became_peered":"2024-03-28T22:33:28.444237+0000","last_unstale":"2024-03-28T22:34:30.151221+0000","last_undegraded":"2024-03-28T22:34:30.151221+0000","last_fullsized":"2024-03-28T22:34:30.151221+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:18:12.820890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:30.157517+0000","last_change":"2024-03-28T22:34:30.156248+0000","last_active":"2024-03-28T22:34:30.157517+0000","last_peered":"2024-03-28T22:34:30.157517+0000","last_clean":"2024-03-28T22:34:30.157517+0000","last_became_active":"2024-03-28T22:34:30.155056+0000","last_became_peered":"2024-03-28T22:34:30.155056+0000","last_unstale":"2024-03-28T22:34:30.157517+0000","last_undegraded":"2024-03-28T22:34:30.157517+0000","last_fullsized":"2024-03-28T22:34:30.157517+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:28:56.469728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.989521+0000","last_change":"2024-03-28T22:33:34.019899+0000","last_active":"2024-03-28T22:34:29.989521+0000","last_peered":"2024-03-28T22:34:29.989521+0000","last_clean":"2024-03-28T22:34:29.989521+0000","last_became_active":"2024-03-28T22:33:34.019392+0000","last_became_peered":"2024-03-28T22:33:34.019392+0000","last_unstale":"2024-03-28T22:34:29.989521+0000","last_undegraded":"2024-03-28T22:34:29.989521+0000","last_fullsized":"2024-03-28T22:34:29.989521+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:08:38.633797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"92'3","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.978721+0000","last_change":"2024-03-28T22:33:28.433243+0000","last_active":"2024-03-28T22:34:29.978721+0000","last_peered":"2024-03-28T22:34:29.978721+0000","last_clean":"2024-03-28T22:34:29.978721+0000","last_became_active":"2024-03-28T22:33:28.432927+0000","last_became_peered":"2024-03-28T22:33:28.432927+0000","last_unstale":"2024-03-28T22:34:29.978721+0000","last_undegraded":"2024-03-28T22:34:29.978721+0000","last_fullsized":"2024-03-28T22:34:29.978721+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:40:43.444229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.981991+0000","last_change":"2024-03-28T22:33:28.948111+0000","last_active":"2024-03-28T22:34:29.981991+0000","last_peered":"2024-03-28T22:34:29.981991+0000","last_clean":"2024-03-28T22:34:29.981991+0000","last_became_active":"2024-03-28T22:33:28.944649+0000","last_became_peered":"2024-03-28T22:33:28.944649+0000","last_unstale":"2024-03-28T22:34:29.981991+0000","last_undegraded":"2024-03-28T22:34:29.981991+0000","last_fullsized":"2024-03-28T22:34:29.981991+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:57:11.483037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:30.426010+0000","last_change":"2024-03-28T22:33:34.000837+0000","last_active":"2024-03-28T22:34:30.426010+0000","last_peered":"2024-03-28T22:34:30.426010+0000","last_clean":"2024-03-28T22:34:30.426010+0000","last_became_active":"2024-03-28T22:33:34.000120+0000","last_became_peered":"2024-03-28T22:33:34.000120+0000","last_unstale":"2024-03-28T22:34:30.426010+0000","last_undegraded":"2024-03-28T22:34:30.426010+0000","last_fullsized":"2024-03-28T22:34:30.426010+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:14:32.028952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"92'4","reported_seq":203,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-28T22:35:29.271573+0000","last_change":"2024-03-28T22:33:28.436985+0000","last_active":"2024-03-28T22:35:29.271573+0000","last_peered":"2024-03-28T22:35:29.271573+0000","last_clean":"2024-03-28T22:35:29.271573+0000","last_became_active":"2024-03-28T22:33:28.436625+0000","last_became_peered":"2024-03-28T22:33:28.436625+0000","last_unstale":"2024-03-28T22:35:29.271573+0000","last_undegraded":"2024-03-28T22:35:29.271573+0000","last_fullsized":"2024-03-28T22:35:29.271573+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:50:45.775191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":168,"num_read_kb":168,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"94'10","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-28T22:35:29.247565+0000","last_change":"2024-03-28T22:33:28.933265+0000","last_active":"2024-03-28T22:35:29.247565+0000","last_peered":"2024-03-28T22:35:29.247565+0000","last_clean":"2024-03-28T22:35:29.247565+0000","last_became_active":"2024-03-28T22:33:28.932954+0000","last_became_peered":"2024-03-28T22:33:28.932954+0000","last_unstale":"2024-03-28T22:35:29.247565+0000","last_undegraded":"2024-03-28T22:35:29.247565+0000","last_fullsized":"2024-03-28T22:35:29.247565+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:37.039055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.981615+0000","last_change":"2024-03-28T22:33:34.024529+0000","last_active":"2024-03-28T22:34:29.981615+0000","last_peered":"2024-03-28T22:34:29.981615+0000","last_clean":"2024-03-28T22:34:29.981615+0000","last_became_active":"2024-03-28T22:33:34.020949+0000","last_became_peered":"2024-03-28T22:33:34.020949+0000","last_unstale":"2024-03-28T22:34:29.981615+0000","last_undegraded":"2024-03-28T22:34:29.981615+0000","last_fullsized":"2024-03-28T22:34:29.981615+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:28:52.039684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"92'4","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:30.120833+0000","last_change":"2024-03-28T22:33:28.432599+0000","last_active":"2024-03-28T22:34:30.120833+0000","last_peered":"2024-03-28T22:34:30.120833+0000","last_clean":"2024-03-28T22:34:30.120833+0000","last_became_active":"2024-03-28T22:33:28.430710+0000","last_became_peered":"2024-03-28T22:33:28.430710+0000","last_unstale":"2024-03-28T22:34:30.120833+0000","last_undegraded":"2024-03-28T22:34:30.120833+0000","last_fullsized":"2024-03-28T22:34:30.120833+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:54.174747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"94'9","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:35:28.176756+0000","last_change":"2024-03-28T22:33:28.955441+0000","last_active":"2024-03-28T22:35:28.176756+0000","last_peered":"2024-03-28T22:35:28.176756+0000","last_clean":"2024-03-28T22:35:28.176756+0000","last_became_active":"2024-03-28T22:33:28.953269+0000","last_became_peered":"2024-03-28T22:33:28.953269+0000","last_unstale":"2024-03-28T22:35:28.176756+0000","last_undegraded":"2024-03-28T22:35:28.176756+0000","last_fullsized":"2024-03-28T22:35:28.176756+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:47:04.128605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.978092+0000","last_change":"2024-03-28T22:33:34.028791+0000","last_active":"2024-03-28T22:34:29.978092+0000","last_peered":"2024-03-28T22:34:29.978092+0000","last_clean":"2024-03-28T22:34:29.978092+0000","last_became_active":"2024-03-28T22:33:34.028361+0000","last_became_peered":"2024-03-28T22:33:34.028361+0000","last_unstale":"2024-03-28T22:34:29.978092+0000","last_undegraded":"2024-03-28T22:34:29.978092+0000","last_fullsized":"2024-03-28T22:34:29.978092+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:42:21.264459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"92'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.978652+0000","last_change":"2024-03-28T22:33:28.455353+0000","last_active":"2024-03-28T22:34:29.978652+0000","last_peered":"2024-03-28T22:34:29.978652+0000","last_clean":"2024-03-28T22:34:29.978652+0000","last_became_active":"2024-03-28T22:33:28.455090+0000","last_became_peered":"2024-03-28T22:33:28.455090+0000","last_unstale":"2024-03-28T22:34:29.978652+0000","last_undegraded":"2024-03-28T22:34:29.978652+0000","last_fullsized":"2024-03-28T22:34:29.978652+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:19:22.496600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"94'11","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-28T22:35:29.247301+0000","last_change":"2024-03-28T22:33:28.928875+0000","last_active":"2024-03-28T22:35:29.247301+0000","last_peered":"2024-03-28T22:35:29.247301+0000","last_clean":"2024-03-28T22:35:29.247301+0000","last_became_active":"2024-03-28T22:33:28.928449+0000","last_became_peered":"2024-03-28T22:33:28.928449+0000","last_unstale":"2024-03-28T22:35:29.247301+0000","last_undegraded":"2024-03-28T22:35:29.247301+0000","last_fullsized":"2024-03-28T22:35:29.247301+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:34:00.402100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.982942+0000","last_change":"2024-03-28T22:33:34.012739+0000","last_active":"2024-03-28T22:34:29.982942+0000","last_peered":"2024-03-28T22:34:29.982942+0000","last_clean":"2024-03-28T22:34:29.982942+0000","last_became_active":"2024-03-28T22:33:34.012262+0000","last_became_peered":"2024-03-28T22:33:34.012262+0000","last_unstale":"2024-03-28T22:34:29.982942+0000","last_undegraded":"2024-03-28T22:34:29.982942+0000","last_fullsized":"2024-03-28T22:34:29.982942+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:24:48.351037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'38","reported_seq":244,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-28T22:35:29.272351+0000","last_change":"2024-03-28T22:33:28.430606+0000","last_active":"2024-03-28T22:35:29.272351+0000","last_peered":"2024-03-28T22:35:29.272351+0000","last_clean":"2024-03-28T22:35:29.272351+0000","last_became_active":"2024-03-28T22:33:28.430233+0000","last_became_peered":"2024-03-28T22:33:28.430233+0000","last_unstale":"2024-03-28T22:35:29.272351+0000","last_undegraded":"2024-03-28T22:35:29.272351+0000","last_fullsized":"2024-03-28T22:35:29.272351+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:49.000569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64455,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":175,"num_read_kb":151,"num_write":40,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"94'6","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:35:28.158474+0000","last_change":"2024-03-28T22:33:28.931167+0000","last_active":"2024-03-28T22:35:28.158474+0000","last_peered":"2024-03-28T22:35:28.158474+0000","last_clean":"2024-03-28T22:35:28.158474+0000","last_became_active":"2024-03-28T22:33:28.930824+0000","last_became_peered":"2024-03-28T22:33:28.930824+0000","last_unstale":"2024-03-28T22:35:28.158474+0000","last_undegraded":"2024-03-28T22:35:28.158474+0000","last_fullsized":"2024-03-28T22:35:28.158474+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:46:49.102791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.979363+0000","last_change":"2024-03-28T22:33:34.023056+0000","last_active":"2024-03-28T22:34:29.979363+0000","last_peered":"2024-03-28T22:34:29.979363+0000","last_clean":"2024-03-28T22:34:29.979363+0000","last_became_active":"2024-03-28T22:33:34.022612+0000","last_became_peered":"2024-03-28T22:33:34.022612+0000","last_unstale":"2024-03-28T22:34:29.979363+0000","last_undegraded":"2024-03-28T22:34:29.979363+0000","last_fullsized":"2024-03-28T22:34:29.979363+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:52:31.936008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"92'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.974846+0000","last_change":"2024-03-28T22:33:28.430963+0000","last_active":"2024-03-28T22:34:29.974846+0000","last_peered":"2024-03-28T22:34:29.974846+0000","last_clean":"2024-03-28T22:34:29.974846+0000","last_became_active":"2024-03-28T22:33:28.430632+0000","last_became_peered":"2024-03-28T22:33:28.430632+0000","last_unstale":"2024-03-28T22:34:29.974846+0000","last_undegraded":"2024-03-28T22:34:29.974846+0000","last_fullsized":"2024-03-28T22:34:29.974846+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:30:23.104769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:30.150537+0000","last_change":"2024-03-28T22:33:28.954105+0000","last_active":"2024-03-28T22:34:30.150537+0000","last_peered":"2024-03-28T22:34:30.150537+0000","last_clean":"2024-03-28T22:34:30.150537+0000","last_became_active":"2024-03-28T22:33:28.953768+0000","last_became_peered":"2024-03-28T22:33:28.953768+0000","last_unstale":"2024-03-28T22:34:30.150537+0000","last_undegraded":"2024-03-28T22:34:30.150537+0000","last_fullsized":"2024-03-28T22:34:30.150537+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:01:25.184523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.975940+0000","last_change":"2024-03-28T22:33:34.027976+0000","last_active":"2024-03-28T22:34:29.975940+0000","last_peered":"2024-03-28T22:34:29.975940+0000","last_clean":"2024-03-28T22:34:29.975940+0000","last_became_active":"2024-03-28T22:33:34.027509+0000","last_became_peered":"2024-03-28T22:33:34.027509+0000","last_unstale":"2024-03-28T22:34:29.975940+0000","last_undegraded":"2024-03-28T22:34:29.975940+0000","last_fullsized":"2024-03-28T22:34:29.975940+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:25:41.908441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"92'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.980666+0000","last_change":"2024-03-28T22:33:28.433283+0000","last_active":"2024-03-28T22:34:29.980666+0000","last_peered":"2024-03-28T22:34:29.980666+0000","last_clean":"2024-03-28T22:34:29.980666+0000","last_became_active":"2024-03-28T22:33:28.432954+0000","last_became_peered":"2024-03-28T22:33:28.432954+0000","last_unstale":"2024-03-28T22:34:29.980666+0000","last_undegraded":"2024-03-28T22:34:29.980666+0000","last_fullsized":"2024-03-28T22:34:29.980666+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:08:34.480459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"94'15","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:35:28.166414+0000","last_change":"2024-03-28T22:33:28.939016+0000","last_active":"2024-03-28T22:35:28.166414+0000","last_peered":"2024-03-28T22:35:28.166414+0000","last_clean":"2024-03-28T22:35:28.166414+0000","last_became_active":"2024-03-28T22:33:28.938695+0000","last_became_peered":"2024-03-28T22:33:28.938695+0000","last_unstale":"2024-03-28T22:35:28.166414+0000","last_undegraded":"2024-03-28T22:35:28.166414+0000","last_fullsized":"2024-03-28T22:35:28.166414+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:26:53.264559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.984498+0000","last_change":"2024-03-28T22:33:34.030127+0000","last_active":"2024-03-28T22:34:29.984498+0000","last_peered":"2024-03-28T22:34:29.984498+0000","last_clean":"2024-03-28T22:34:29.984498+0000","last_became_active":"2024-03-28T22:33:34.028056+0000","last_became_peered":"2024-03-28T22:33:34.028056+0000","last_unstale":"2024-03-28T22:34:29.984498+0000","last_undegraded":"2024-03-28T22:34:29.984498+0000","last_fullsized":"2024-03-28T22:34:29.984498+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:29:41.785013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"92'6","reported_seq":117,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:35:25.353140+0000","last_change":"2024-03-28T22:33:28.435350+0000","last_active":"2024-03-28T22:35:25.353140+0000","last_peered":"2024-03-28T22:35:25.353140+0000","last_clean":"2024-03-28T22:35:25.353140+0000","last_became_active":"2024-03-28T22:33:28.434954+0000","last_became_peered":"2024-03-28T22:33:28.434954+0000","last_unstale":"2024-03-28T22:35:25.353140+0000","last_undegraded":"2024-03-28T22:35:25.353140+0000","last_fullsized":"2024-03-28T22:35:25.353140+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:04.969596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":84,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"94'6","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:35:28.154516+0000","last_change":"2024-03-28T22:33:28.921219+0000","last_active":"2024-03-28T22:35:28.154516+0000","last_peered":"2024-03-28T22:35:28.154516+0000","last_clean":"2024-03-28T22:35:28.154516+0000","last_became_active":"2024-03-28T22:33:28.920664+0000","last_became_peered":"2024-03-28T22:33:28.920664+0000","last_unstale":"2024-03-28T22:35:28.154516+0000","last_undegraded":"2024-03-28T22:35:28.154516+0000","last_fullsized":"2024-03-28T22:35:28.154516+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:48:55.032058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:30.149675+0000","last_change":"2024-03-28T22:33:34.018101+0000","last_active":"2024-03-28T22:34:30.149675+0000","last_peered":"2024-03-28T22:34:30.149675+0000","last_clean":"2024-03-28T22:34:30.149675+0000","last_became_active":"2024-03-28T22:33:34.017260+0000","last_became_peered":"2024-03-28T22:33:34.017260+0000","last_unstale":"2024-03-28T22:34:30.149675+0000","last_undegraded":"2024-03-28T22:34:30.149675+0000","last_fullsized":"2024-03-28T22:34:30.149675+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:48:53.178090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"92'17","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.979791+0000","last_change":"2024-03-28T22:33:28.450232+0000","last_active":"2024-03-28T22:34:29.979791+0000","last_peered":"2024-03-28T22:34:29.979791+0000","last_clean":"2024-03-28T22:34:29.979791+0000","last_became_active":"2024-03-28T22:33:28.449958+0000","last_became_peered":"2024-03-28T22:33:28.449958+0000","last_unstale":"2024-03-28T22:34:29.979791+0000","last_undegraded":"2024-03-28T22:34:29.979791+0000","last_fullsized":"2024-03-28T22:34:29.979791+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:01:42.130260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.978408+0000","last_change":"2024-03-28T22:33:28.926263+0000","last_active":"2024-03-28T22:34:29.978408+0000","last_peered":"2024-03-28T22:34:29.978408+0000","last_clean":"2024-03-28T22:34:29.978408+0000","last_became_active":"2024-03-28T22:33:28.925927+0000","last_became_peered":"2024-03-28T22:33:28.925927+0000","last_unstale":"2024-03-28T22:34:29.978408+0000","last_undegraded":"2024-03-28T22:34:29.978408+0000","last_fullsized":"2024-03-28T22:34:29.978408+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:41:42.477694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":2,"reported_epoch":95,"state":"peering","last_fresh":"2024-03-28T22:33:33.992594+0000","last_change":"2024-03-28T22:35:29.421928+0000","last_active":"2024-03-28T22:33:32.966724+0000","last_peered":"2024-03-28T22:33:32.966724+0000","last_clean":"2024-03-28T22:33:32.966724+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T22:33:33.992594+0000","last_undegraded":"2024-03-28T22:33:33.992594+0000","last_fullsized":"2024-03-28T22:33:33.992594+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,3,1],"acting":[10,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.f","version":"92'2","reported_seq":17,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.987269+0000","last_change":"2024-03-28T22:34:29.986650+0000","last_active":"2024-03-28T22:34:29.987269+0000","last_peered":"2024-03-28T22:34:29.987269+0000","last_clean":"2024-03-28T22:34:29.987269+0000","last_became_active":"2024-03-28T22:34:29.981875+0000","last_became_peered":"2024-03-28T22:34:29.981875+0000","last_unstale":"2024-03-28T22:34:29.987269+0000","last_undegraded":"2024-03-28T22:34:29.987269+0000","last_fullsized":"2024-03-28T22:34:29.987269+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:45:24.166133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.982815+0000","last_change":"2024-03-28T22:34:29.982815+0000","last_active":"2024-03-28T22:34:29.982815+0000","last_peered":"2024-03-28T22:34:29.982815+0000","last_clean":"2024-03-28T22:34:29.982815+0000","last_became_active":"2024-03-28T22:34:29.982458+0000","last_became_peered":"2024-03-28T22:34:29.982458+0000","last_unstale":"2024-03-28T22:34:29.982815+0000","last_undegraded":"2024-03-28T22:34:29.982815+0000","last_fullsized":"2024-03-28T22:34:29.982815+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:11:30.617181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.980141+0000","last_change":"2024-03-28T22:33:34.014364+0000","last_active":"2024-03-28T22:34:29.980141+0000","last_peered":"2024-03-28T22:34:29.980141+0000","last_clean":"2024-03-28T22:34:29.980141+0000","last_became_active":"2024-03-28T22:33:34.013929+0000","last_became_peered":"2024-03-28T22:33:34.013929+0000","last_unstale":"2024-03-28T22:34:29.980141+0000","last_undegraded":"2024-03-28T22:34:29.980141+0000","last_fullsized":"2024-03-28T22:34:29.980141+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:21:37.986774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"92'1","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.992884+0000","last_change":"2024-03-28T22:34:29.992884+0000","last_active":"2024-03-28T22:34:29.992884+0000","last_peered":"2024-03-28T22:34:29.992884+0000","last_clean":"2024-03-28T22:34:29.992884+0000","last_became_active":"2024-03-28T22:34:29.991657+0000","last_became_peered":"2024-03-28T22:34:29.991657+0000","last_unstale":"2024-03-28T22:34:29.992884+0000","last_undegraded":"2024-03-28T22:34:29.992884+0000","last_fullsized":"2024-03-28T22:34:29.992884+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:25:14.540783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"94'4","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-28T22:35:29.245012+0000","last_change":"2024-03-28T22:33:28.942097+0000","last_active":"2024-03-28T22:35:29.245012+0000","last_peered":"2024-03-28T22:35:29.245012+0000","last_clean":"2024-03-28T22:35:29.245012+0000","last_became_active":"2024-03-28T22:33:28.941812+0000","last_became_peered":"2024-03-28T22:33:28.941812+0000","last_unstale":"2024-03-28T22:35:29.245012+0000","last_undegraded":"2024-03-28T22:35:29.245012+0000","last_fullsized":"2024-03-28T22:35:29.245012+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:36:10.204216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:30.149214+0000","last_change":"2024-03-28T22:33:34.023171+0000","last_active":"2024-03-28T22:34:30.149214+0000","last_peered":"2024-03-28T22:34:30.149214+0000","last_clean":"2024-03-28T22:34:30.149214+0000","last_became_active":"2024-03-28T22:33:34.021313+0000","last_became_peered":"2024-03-28T22:33:34.021313+0000","last_unstale":"2024-03-28T22:34:30.149214+0000","last_undegraded":"2024-03-28T22:34:30.149214+0000","last_fullsized":"2024-03-28T22:34:30.149214+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:29:42.592761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"92'5","reported_seq":118,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:35:27.314994+0000","last_change":"2024-03-28T22:33:28.449133+0000","last_active":"2024-03-28T22:35:27.314994+0000","last_peered":"2024-03-28T22:35:27.314994+0000","last_clean":"2024-03-28T22:35:27.314994+0000","last_became_active":"2024-03-28T22:33:28.448743+0000","last_became_peered":"2024-03-28T22:33:28.448743+0000","last_unstale":"2024-03-28T22:35:27.314994+0000","last_undegraded":"2024-03-28T22:35:27.314994+0000","last_fullsized":"2024-03-28T22:35:27.314994+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:09:37.119736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"94'7","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-28T22:35:29.244649+0000","last_change":"2024-03-28T22:33:28.944198+0000","last_active":"2024-03-28T22:35:29.244649+0000","last_peered":"2024-03-28T22:35:29.244649+0000","last_clean":"2024-03-28T22:35:29.244649+0000","last_became_active":"2024-03-28T22:33:28.943818+0000","last_became_peered":"2024-03-28T22:33:28.943818+0000","last_unstale":"2024-03-28T22:35:29.244649+0000","last_undegraded":"2024-03-28T22:35:29.244649+0000","last_fullsized":"2024-03-28T22:35:29.244649+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:31:03.791364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:30.149355+0000","last_change":"2024-03-28T22:33:34.015499+0000","last_active":"2024-03-28T22:34:30.149355+0000","last_peered":"2024-03-28T22:34:30.149355+0000","last_clean":"2024-03-28T22:34:30.149355+0000","last_became_active":"2024-03-28T22:33:34.015013+0000","last_became_peered":"2024-03-28T22:33:34.015013+0000","last_unstale":"2024-03-28T22:34:30.149355+0000","last_undegraded":"2024-03-28T22:34:30.149355+0000","last_fullsized":"2024-03-28T22:34:30.149355+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:34:30.582198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.978922+0000","last_change":"2024-03-28T22:33:28.926732+0000","last_active":"2024-03-28T22:34:29.978922+0000","last_peered":"2024-03-28T22:34:29.978922+0000","last_clean":"2024-03-28T22:34:29.978922+0000","last_became_active":"2024-03-28T22:33:28.926366+0000","last_became_peered":"2024-03-28T22:33:28.926366+0000","last_unstale":"2024-03-28T22:34:29.978922+0000","last_undegraded":"2024-03-28T22:34:29.978922+0000","last_fullsized":"2024-03-28T22:34:29.978922+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:19:10.769141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"92'5","reported_seq":117,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:35:26.354184+0000","last_change":"2024-03-28T22:33:28.457400+0000","last_active":"2024-03-28T22:35:26.354184+0000","last_peered":"2024-03-28T22:35:26.354184+0000","last_clean":"2024-03-28T22:35:26.354184+0000","last_became_active":"2024-03-28T22:33:28.456994+0000","last_became_peered":"2024-03-28T22:33:28.456994+0000","last_unstale":"2024-03-28T22:35:26.354184+0000","last_undegraded":"2024-03-28T22:35:26.354184+0000","last_fullsized":"2024-03-28T22:35:26.354184+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:32:16.438540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":83,"num_read_kb":83,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"92'1","reported_seq":19,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:30.119748+0000","last_change":"2024-03-28T22:33:34.027746+0000","last_active":"2024-03-28T22:34:30.119748+0000","last_peered":"2024-03-28T22:34:30.119748+0000","last_clean":"2024-03-28T22:34:30.119748+0000","last_became_active":"2024-03-28T22:33:34.027419+0000","last_became_peered":"2024-03-28T22:33:34.027419+0000","last_unstale":"2024-03-28T22:34:30.119748+0000","last_undegraded":"2024-03-28T22:34:30.119748+0000","last_fullsized":"2024-03-28T22:34:30.119748+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:55:43.686428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:30.426908+0000","last_change":"2024-03-28T22:33:28.952393+0000","last_active":"2024-03-28T22:34:30.426908+0000","last_peered":"2024-03-28T22:34:30.426908+0000","last_clean":"2024-03-28T22:34:30.426908+0000","last_became_active":"2024-03-28T22:33:28.951951+0000","last_became_peered":"2024-03-28T22:33:28.951951+0000","last_unstale":"2024-03-28T22:34:30.426908+0000","last_undegraded":"2024-03-28T22:34:30.426908+0000","last_fullsized":"2024-03-28T22:34:30.426908+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:09:25.808842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"92'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.977956+0000","last_change":"2024-03-28T22:33:28.437177+0000","last_active":"2024-03-28T22:34:29.977956+0000","last_peered":"2024-03-28T22:34:29.977956+0000","last_clean":"2024-03-28T22:34:29.977956+0000","last_became_active":"2024-03-28T22:33:28.436801+0000","last_became_peered":"2024-03-28T22:33:28.436801+0000","last_unstale":"2024-03-28T22:34:29.977956+0000","last_undegraded":"2024-03-28T22:34:29.977956+0000","last_fullsized":"2024-03-28T22:34:29.977956+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:24:52.549490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":367,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:30.120286+0000","last_change":"2024-03-28T22:32:07.867336+0000","last_active":"2024-03-28T22:34:30.120286+0000","last_peered":"2024-03-28T22:34:30.120286+0000","last_clean":"2024-03-28T22:34:30.120286+0000","last_became_active":"2024-03-28T22:32:07.866819+0000","last_became_peered":"2024-03-28T22:32:07.866819+0000","last_unstale":"2024-03-28T22:34:30.120286+0000","last_undegraded":"2024-03-28T22:34:30.120286+0000","last_fullsized":"2024-03-28T22:34:30.120286+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:28:14.713501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:28:14.713501+0000","last_clean_scrub_stamp":"2024-03-28T22:28:14.713501+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:00:52.438708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.978713+0000","last_change":"2024-03-28T22:33:34.020668+0000","last_active":"2024-03-28T22:34:29.978713+0000","last_peered":"2024-03-28T22:34:29.978713+0000","last_clean":"2024-03-28T22:34:29.978713+0000","last_became_active":"2024-03-28T22:33:34.020240+0000","last_became_peered":"2024-03-28T22:33:34.020240+0000","last_unstale":"2024-03-28T22:34:29.978713+0000","last_undegraded":"2024-03-28T22:34:29.978713+0000","last_fullsized":"2024-03-28T22:34:29.978713+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:18.983983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:30.150605+0000","last_change":"2024-03-28T22:33:28.935237+0000","last_active":"2024-03-28T22:34:30.150605+0000","last_peered":"2024-03-28T22:34:30.150605+0000","last_clean":"2024-03-28T22:34:30.150605+0000","last_became_active":"2024-03-28T22:33:28.934875+0000","last_became_peered":"2024-03-28T22:33:28.934875+0000","last_unstale":"2024-03-28T22:34:30.150605+0000","last_undegraded":"2024-03-28T22:34:30.150605+0000","last_fullsized":"2024-03-28T22:34:30.150605+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:20:50.789302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"92'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:30.120382+0000","last_change":"2024-03-28T22:33:28.441654+0000","last_active":"2024-03-28T22:34:30.120382+0000","last_peered":"2024-03-28T22:34:30.120382+0000","last_clean":"2024-03-28T22:34:30.120382+0000","last_became_active":"2024-03-28T22:33:28.441293+0000","last_became_peered":"2024-03-28T22:33:28.441293+0000","last_unstale":"2024-03-28T22:34:30.120382+0000","last_undegraded":"2024-03-28T22:34:30.120382+0000","last_fullsized":"2024-03-28T22:34:30.120382+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:31:17.737977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.976004+0000","last_change":"2024-03-28T22:33:34.018126+0000","last_active":"2024-03-28T22:34:29.976004+0000","last_peered":"2024-03-28T22:34:29.976004+0000","last_clean":"2024-03-28T22:34:29.976004+0000","last_became_active":"2024-03-28T22:33:34.017573+0000","last_became_peered":"2024-03-28T22:33:34.017573+0000","last_unstale":"2024-03-28T22:34:29.976004+0000","last_undegraded":"2024-03-28T22:34:29.976004+0000","last_fullsized":"2024-03-28T22:34:29.976004+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:23:31.013511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"92'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.980816+0000","last_change":"2024-03-28T22:33:28.454069+0000","last_active":"2024-03-28T22:34:29.980816+0000","last_peered":"2024-03-28T22:34:29.980816+0000","last_clean":"2024-03-28T22:34:29.980816+0000","last_became_active":"2024-03-28T22:33:28.453754+0000","last_became_peered":"2024-03-28T22:33:28.453754+0000","last_unstale":"2024-03-28T22:34:29.980816+0000","last_undegraded":"2024-03-28T22:34:29.980816+0000","last_fullsized":"2024-03-28T22:34:29.980816+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:23:33.981072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:30.152417+0000","last_change":"2024-03-28T22:33:28.934390+0000","last_active":"2024-03-28T22:34:30.152417+0000","last_peered":"2024-03-28T22:34:30.152417+0000","last_clean":"2024-03-28T22:34:30.152417+0000","last_became_active":"2024-03-28T22:33:28.933919+0000","last_became_peered":"2024-03-28T22:33:28.933919+0000","last_unstale":"2024-03-28T22:34:30.152417+0000","last_undegraded":"2024-03-28T22:34:30.152417+0000","last_fullsized":"2024-03-28T22:34:30.152417+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:27:07.595144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:30.426096+0000","last_change":"2024-03-28T22:33:34.019910+0000","last_active":"2024-03-28T22:34:30.426096+0000","last_peered":"2024-03-28T22:34:30.426096+0000","last_clean":"2024-03-28T22:34:30.426096+0000","last_became_active":"2024-03-28T22:33:34.019522+0000","last_became_peered":"2024-03-28T22:33:34.019522+0000","last_unstale":"2024-03-28T22:34:30.426096+0000","last_undegraded":"2024-03-28T22:34:30.426096+0000","last_fullsized":"2024-03-28T22:34:30.426096+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:01:29.298513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"92'13","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.983513+0000","last_change":"2024-03-28T22:33:28.444513+0000","last_active":"2024-03-28T22:34:29.983513+0000","last_peered":"2024-03-28T22:34:29.983513+0000","last_clean":"2024-03-28T22:34:29.983513+0000","last_became_active":"2024-03-28T22:33:28.444091+0000","last_became_peered":"2024-03-28T22:33:28.444091+0000","last_unstale":"2024-03-28T22:34:29.983513+0000","last_undegraded":"2024-03-28T22:34:29.983513+0000","last_fullsized":"2024-03-28T22:34:29.983513+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:49:58.331864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:30.427967+0000","last_change":"2024-03-28T22:33:28.948390+0000","last_active":"2024-03-28T22:34:30.427967+0000","last_peered":"2024-03-28T22:34:30.427967+0000","last_clean":"2024-03-28T22:34:30.427967+0000","last_became_active":"2024-03-28T22:33:28.942846+0000","last_became_peered":"2024-03-28T22:33:28.942846+0000","last_unstale":"2024-03-28T22:34:30.427967+0000","last_undegraded":"2024-03-28T22:34:30.427967+0000","last_fullsized":"2024-03-28T22:34:30.427967+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:50.596365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.974146+0000","last_change":"2024-03-28T22:33:33.999429+0000","last_active":"2024-03-28T22:34:29.974146+0000","last_peered":"2024-03-28T22:34:29.974146+0000","last_clean":"2024-03-28T22:34:29.974146+0000","last_became_active":"2024-03-28T22:33:33.998979+0000","last_became_peered":"2024-03-28T22:33:33.998979+0000","last_unstale":"2024-03-28T22:34:29.974146+0000","last_undegraded":"2024-03-28T22:34:29.974146+0000","last_fullsized":"2024-03-28T22:34:29.974146+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:26.450731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"92'7","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.978469+0000","last_change":"2024-03-28T22:33:28.442861+0000","last_active":"2024-03-28T22:34:29.978469+0000","last_peered":"2024-03-28T22:34:29.978469+0000","last_clean":"2024-03-28T22:34:29.978469+0000","last_became_active":"2024-03-28T22:33:28.442468+0000","last_became_peered":"2024-03-28T22:33:28.442468+0000","last_unstale":"2024-03-28T22:34:29.978469+0000","last_undegraded":"2024-03-28T22:34:29.978469+0000","last_fullsized":"2024-03-28T22:34:29.978469+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:45:49.899031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"94'7","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:35:28.188660+0000","last_change":"2024-03-28T22:34:30.429872+0000","last_active":"2024-03-28T22:35:28.188660+0000","last_peered":"2024-03-28T22:35:28.188660+0000","last_clean":"2024-03-28T22:35:28.188660+0000","last_became_active":"2024-03-28T22:34:30.429237+0000","last_became_peered":"2024-03-28T22:34:30.429237+0000","last_unstale":"2024-03-28T22:35:28.188660+0000","last_undegraded":"2024-03-28T22:35:28.188660+0000","last_fullsized":"2024-03-28T22:35:28.188660+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:14:46.608826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.978304+0000","last_change":"2024-03-28T22:33:34.013826+0000","last_active":"2024-03-28T22:34:29.978304+0000","last_peered":"2024-03-28T22:34:29.978304+0000","last_clean":"2024-03-28T22:34:29.978304+0000","last_became_active":"2024-03-28T22:33:34.013415+0000","last_became_peered":"2024-03-28T22:33:34.013415+0000","last_unstale":"2024-03-28T22:34:29.978304+0000","last_undegraded":"2024-03-28T22:34:29.978304+0000","last_fullsized":"2024-03-28T22:34:29.978304+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:19:30.965940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"92'2","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-28T22:35:29.246592+0000","last_change":"2024-03-28T22:33:28.442225+0000","last_active":"2024-03-28T22:35:29.246592+0000","last_peered":"2024-03-28T22:35:29.246592+0000","last_clean":"2024-03-28T22:35:29.246592+0000","last_became_active":"2024-03-28T22:33:28.441844+0000","last_became_peered":"2024-03-28T22:33:28.441844+0000","last_unstale":"2024-03-28T22:35:29.246592+0000","last_undegraded":"2024-03-28T22:35:29.246592+0000","last_fullsized":"2024-03-28T22:35:29.246592+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:40:52.108663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"94'2","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:35:28.117335+0000","last_change":"2024-03-28T22:33:28.927571+0000","last_active":"2024-03-28T22:35:28.117335+0000","last_peered":"2024-03-28T22:35:28.117335+0000","last_clean":"2024-03-28T22:35:28.117335+0000","last_became_active":"2024-03-28T22:33:28.927275+0000","last_became_peered":"2024-03-28T22:33:28.927275+0000","last_unstale":"2024-03-28T22:35:28.117335+0000","last_undegraded":"2024-03-28T22:35:28.117335+0000","last_fullsized":"2024-03-28T22:35:28.117335+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:25:16.184533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.985056+0000","last_change":"2024-03-28T22:33:34.018429+0000","last_active":"2024-03-28T22:34:29.985056+0000","last_peered":"2024-03-28T22:34:29.985056+0000","last_clean":"2024-03-28T22:34:29.985056+0000","last_became_active":"2024-03-28T22:33:34.017066+0000","last_became_peered":"2024-03-28T22:33:34.017066+0000","last_unstale":"2024-03-28T22:34:29.985056+0000","last_undegraded":"2024-03-28T22:34:29.985056+0000","last_fullsized":"2024-03-28T22:34:29.985056+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:35:42.101072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"92'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:30.427924+0000","last_change":"2024-03-28T22:33:28.437760+0000","last_active":"2024-03-28T22:34:30.427924+0000","last_peered":"2024-03-28T22:34:30.427924+0000","last_clean":"2024-03-28T22:34:30.427924+0000","last_became_active":"2024-03-28T22:33:28.437096+0000","last_became_peered":"2024-03-28T22:33:28.437096+0000","last_unstale":"2024-03-28T22:34:30.427924+0000","last_undegraded":"2024-03-28T22:34:30.427924+0000","last_fullsized":"2024-03-28T22:34:30.427924+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:29:20.778551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.975938+0000","last_change":"2024-03-28T22:33:28.935805+0000","last_active":"2024-03-28T22:34:29.975938+0000","last_peered":"2024-03-28T22:34:29.975938+0000","last_clean":"2024-03-28T22:34:29.975938+0000","last_became_active":"2024-03-28T22:33:28.935520+0000","last_became_peered":"2024-03-28T22:33:28.935520+0000","last_unstale":"2024-03-28T22:34:29.975938+0000","last_undegraded":"2024-03-28T22:34:29.975938+0000","last_fullsized":"2024-03-28T22:34:29.975938+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:23.414310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"92'1","reported_seq":19,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.977954+0000","last_change":"2024-03-28T22:33:34.029518+0000","last_active":"2024-03-28T22:34:29.977954+0000","last_peered":"2024-03-28T22:34:29.977954+0000","last_clean":"2024-03-28T22:34:29.977954+0000","last_became_active":"2024-03-28T22:33:34.029101+0000","last_became_peered":"2024-03-28T22:33:34.029101+0000","last_unstale":"2024-03-28T22:34:29.977954+0000","last_undegraded":"2024-03-28T22:34:29.977954+0000","last_fullsized":"2024-03-28T22:34:29.977954+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:05.595543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"92'3","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.974539+0000","last_change":"2024-03-28T22:33:28.438857+0000","last_active":"2024-03-28T22:34:29.974539+0000","last_peered":"2024-03-28T22:34:29.974539+0000","last_clean":"2024-03-28T22:34:29.974539+0000","last_became_active":"2024-03-28T22:33:28.438528+0000","last_became_peered":"2024-03-28T22:33:28.438528+0000","last_unstale":"2024-03-28T22:34:29.974539+0000","last_undegraded":"2024-03-28T22:34:29.974539+0000","last_fullsized":"2024-03-28T22:34:29.974539+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:07:54.506739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:30.427183+0000","last_change":"2024-03-28T22:33:28.958271+0000","last_active":"2024-03-28T22:34:30.427183+0000","last_peered":"2024-03-28T22:34:30.427183+0000","last_clean":"2024-03-28T22:34:30.427183+0000","last_became_active":"2024-03-28T22:33:28.957922+0000","last_became_peered":"2024-03-28T22:33:28.957922+0000","last_unstale":"2024-03-28T22:34:30.427183+0000","last_undegraded":"2024-03-28T22:34:30.427183+0000","last_fullsized":"2024-03-28T22:34:30.427183+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:33:19.003131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.977893+0000","last_change":"2024-03-28T22:33:34.000789+0000","last_active":"2024-03-28T22:34:29.977893+0000","last_peered":"2024-03-28T22:34:29.977893+0000","last_clean":"2024-03-28T22:34:29.977893+0000","last_became_active":"2024-03-28T22:33:34.000383+0000","last_became_peered":"2024-03-28T22:33:34.000383+0000","last_unstale":"2024-03-28T22:34:29.977893+0000","last_undegraded":"2024-03-28T22:34:29.977893+0000","last_fullsized":"2024-03-28T22:34:29.977893+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:50:29.925656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"92'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.980028+0000","last_change":"2024-03-28T22:33:28.439121+0000","last_active":"2024-03-28T22:34:29.980028+0000","last_peered":"2024-03-28T22:34:29.980028+0000","last_clean":"2024-03-28T22:34:29.980028+0000","last_became_active":"2024-03-28T22:33:28.438800+0000","last_became_peered":"2024-03-28T22:33:28.438800+0000","last_unstale":"2024-03-28T22:34:29.980028+0000","last_undegraded":"2024-03-28T22:34:29.980028+0000","last_fullsized":"2024-03-28T22:34:29.980028+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:57:10.013246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"94'5","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:35:28.166145+0000","last_change":"2024-03-28T22:33:28.934499+0000","last_active":"2024-03-28T22:35:28.166145+0000","last_peered":"2024-03-28T22:35:28.166145+0000","last_clean":"2024-03-28T22:35:28.166145+0000","last_became_active":"2024-03-28T22:33:28.934147+0000","last_became_peered":"2024-03-28T22:33:28.934147+0000","last_unstale":"2024-03-28T22:35:28.166145+0000","last_undegraded":"2024-03-28T22:35:28.166145+0000","last_fullsized":"2024-03-28T22:35:28.166145+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:35:39.090052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":20,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-28T22:35:29.246503+0000","last_change":"2024-03-28T22:33:34.017946+0000","last_active":"2024-03-28T22:35:29.246503+0000","last_peered":"2024-03-28T22:35:29.246503+0000","last_clean":"2024-03-28T22:35:29.246503+0000","last_became_active":"2024-03-28T22:33:34.017546+0000","last_became_peered":"2024-03-28T22:33:34.017546+0000","last_unstale":"2024-03-28T22:35:29.246503+0000","last_undegraded":"2024-03-28T22:35:29.246503+0000","last_fullsized":"2024-03-28T22:35:29.246503+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:14:39.081677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"92'5","reported_seq":119,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:35:27.270068+0000","last_change":"2024-03-28T22:33:28.453387+0000","last_active":"2024-03-28T22:35:27.270068+0000","last_peered":"2024-03-28T22:35:27.270068+0000","last_clean":"2024-03-28T22:35:27.270068+0000","last_became_active":"2024-03-28T22:33:28.453029+0000","last_became_peered":"2024-03-28T22:33:28.453029+0000","last_unstale":"2024-03-28T22:35:27.270068+0000","last_undegraded":"2024-03-28T22:35:27.270068+0000","last_fullsized":"2024-03-28T22:35:27.270068+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:12:33.246863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":85,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"94'7","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:35:28.163318+0000","last_change":"2024-03-28T22:33:28.923819+0000","last_active":"2024-03-28T22:35:28.163318+0000","last_peered":"2024-03-28T22:35:28.163318+0000","last_clean":"2024-03-28T22:35:28.163318+0000","last_became_active":"2024-03-28T22:33:28.923471+0000","last_became_peered":"2024-03-28T22:33:28.923471+0000","last_unstale":"2024-03-28T22:35:28.163318+0000","last_undegraded":"2024-03-28T22:35:28.163318+0000","last_fullsized":"2024-03-28T22:35:28.163318+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:58:03.496406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.986079+0000","last_change":"2024-03-28T22:33:34.029789+0000","last_active":"2024-03-28T22:34:29.986079+0000","last_peered":"2024-03-28T22:34:29.986079+0000","last_clean":"2024-03-28T22:34:29.986079+0000","last_became_active":"2024-03-28T22:33:34.029231+0000","last_became_peered":"2024-03-28T22:33:34.029231+0000","last_unstale":"2024-03-28T22:34:29.986079+0000","last_undegraded":"2024-03-28T22:34:29.986079+0000","last_fullsized":"2024-03-28T22:34:29.986079+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:47:50.101393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"92'3","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.978859+0000","last_change":"2024-03-28T22:33:28.435668+0000","last_active":"2024-03-28T22:34:29.978859+0000","last_peered":"2024-03-28T22:34:29.978859+0000","last_clean":"2024-03-28T22:34:29.978859+0000","last_became_active":"2024-03-28T22:33:28.435325+0000","last_became_peered":"2024-03-28T22:33:28.435325+0000","last_unstale":"2024-03-28T22:34:29.978859+0000","last_undegraded":"2024-03-28T22:34:29.978859+0000","last_fullsized":"2024-03-28T22:34:29.978859+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:13:06.281074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"94'6","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:35:28.179272+0000","last_change":"2024-03-28T22:33:28.954138+0000","last_active":"2024-03-28T22:35:28.179272+0000","last_peered":"2024-03-28T22:35:28.179272+0000","last_clean":"2024-03-28T22:35:28.179272+0000","last_became_active":"2024-03-28T22:33:28.953752+0000","last_became_peered":"2024-03-28T22:33:28.953752+0000","last_unstale":"2024-03-28T22:35:28.179272+0000","last_undegraded":"2024-03-28T22:35:28.179272+0000","last_fullsized":"2024-03-28T22:35:28.179272+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:53:34.892316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.978080+0000","last_change":"2024-03-28T22:33:34.002672+0000","last_active":"2024-03-28T22:34:29.978080+0000","last_peered":"2024-03-28T22:34:29.978080+0000","last_clean":"2024-03-28T22:34:29.978080+0000","last_became_active":"2024-03-28T22:33:34.002195+0000","last_became_peered":"2024-03-28T22:33:34.002195+0000","last_unstale":"2024-03-28T22:34:29.978080+0000","last_undegraded":"2024-03-28T22:34:29.978080+0000","last_fullsized":"2024-03-28T22:34:29.978080+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:24:11.521474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"92'5","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:30.153114+0000","last_change":"2024-03-28T22:33:28.450818+0000","last_active":"2024-03-28T22:34:30.153114+0000","last_peered":"2024-03-28T22:34:30.153114+0000","last_clean":"2024-03-28T22:34:30.153114+0000","last_became_active":"2024-03-28T22:33:28.448592+0000","last_became_peered":"2024-03-28T22:33:28.448592+0000","last_unstale":"2024-03-28T22:34:30.153114+0000","last_undegraded":"2024-03-28T22:34:30.153114+0000","last_fullsized":"2024-03-28T22:34:30.153114+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:19:27.126306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"94'9","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:35:28.159457+0000","last_change":"2024-03-28T22:33:28.925191+0000","last_active":"2024-03-28T22:35:28.159457+0000","last_peered":"2024-03-28T22:35:28.159457+0000","last_clean":"2024-03-28T22:35:28.159457+0000","last_became_active":"2024-03-28T22:33:28.924424+0000","last_became_peered":"2024-03-28T22:33:28.924424+0000","last_unstale":"2024-03-28T22:35:28.159457+0000","last_undegraded":"2024-03-28T22:35:28.159457+0000","last_fullsized":"2024-03-28T22:35:28.159457+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:50:00.415047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.987176+0000","last_change":"2024-03-28T22:34:29.987176+0000","last_active":"2024-03-28T22:34:29.987176+0000","last_peered":"2024-03-28T22:34:29.987176+0000","last_clean":"2024-03-28T22:34:29.987176+0000","last_became_active":"2024-03-28T22:34:29.986807+0000","last_became_peered":"2024-03-28T22:34:29.986807+0000","last_unstale":"2024-03-28T22:34:29.987176+0000","last_undegraded":"2024-03-28T22:34:29.987176+0000","last_fullsized":"2024-03-28T22:34:29.987176+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:00:06.774521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"94'12","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:35:28.147476+0000","last_change":"2024-03-28T22:33:28.927068+0000","last_active":"2024-03-28T22:35:28.147476+0000","last_peered":"2024-03-28T22:35:28.147476+0000","last_clean":"2024-03-28T22:35:28.147476+0000","last_became_active":"2024-03-28T22:33:28.926734+0000","last_became_peered":"2024-03-28T22:33:28.926734+0000","last_unstale":"2024-03-28T22:35:28.147476+0000","last_undegraded":"2024-03-28T22:35:28.147476+0000","last_fullsized":"2024-03-28T22:35:28.147476+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:56.309555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"94'8","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:35:28.186181+0000","last_change":"2024-03-28T22:33:28.955675+0000","last_active":"2024-03-28T22:35:28.186181+0000","last_peered":"2024-03-28T22:35:28.186181+0000","last_clean":"2024-03-28T22:35:28.186181+0000","last_became_active":"2024-03-28T22:33:28.955329+0000","last_became_peered":"2024-03-28T22:33:28.955329+0000","last_unstale":"2024-03-28T22:35:28.186181+0000","last_undegraded":"2024-03-28T22:35:28.186181+0000","last_fullsized":"2024-03-28T22:35:28.186181+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:34:14.166434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"94'7","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:35:28.176418+0000","last_change":"2024-03-28T22:33:28.955125+0000","last_active":"2024-03-28T22:35:28.176418+0000","last_peered":"2024-03-28T22:35:28.176418+0000","last_clean":"2024-03-28T22:35:28.176418+0000","last_became_active":"2024-03-28T22:33:28.954434+0000","last_became_peered":"2024-03-28T22:33:28.954434+0000","last_unstale":"2024-03-28T22:35:28.176418+0000","last_undegraded":"2024-03-28T22:35:28.176418+0000","last_fullsized":"2024-03-28T22:35:28.176418+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:56:50.705012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:30.152960+0000","last_change":"2024-03-28T22:33:28.945230+0000","last_active":"2024-03-28T22:34:30.152960+0000","last_peered":"2024-03-28T22:34:30.152960+0000","last_clean":"2024-03-28T22:34:30.152960+0000","last_became_active":"2024-03-28T22:33:28.944921+0000","last_became_peered":"2024-03-28T22:33:28.944921+0000","last_unstale":"2024-03-28T22:34:30.152960+0000","last_undegraded":"2024-03-28T22:34:30.152960+0000","last_fullsized":"2024-03-28T22:34:30.152960+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:26:26.505774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.976205+0000","last_change":"2024-03-28T22:33:28.928720+0000","last_active":"2024-03-28T22:34:29.976205+0000","last_peered":"2024-03-28T22:34:29.976205+0000","last_clean":"2024-03-28T22:34:29.976205+0000","last_became_active":"2024-03-28T22:33:28.928415+0000","last_became_peered":"2024-03-28T22:33:28.928415+0000","last_unstale":"2024-03-28T22:34:29.976205+0000","last_undegraded":"2024-03-28T22:34:29.976205+0000","last_fullsized":"2024-03-28T22:34:29.976205+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:00.839609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"94'15","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:35:28.157223+0000","last_change":"2024-03-28T22:33:28.924948+0000","last_active":"2024-03-28T22:35:28.157223+0000","last_peered":"2024-03-28T22:35:28.157223+0000","last_clean":"2024-03-28T22:35:28.157223+0000","last_became_active":"2024-03-28T22:33:28.924631+0000","last_became_peered":"2024-03-28T22:33:28.924631+0000","last_unstale":"2024-03-28T22:35:28.157223+0000","last_undegraded":"2024-03-28T22:35:28.157223+0000","last_fullsized":"2024-03-28T22:35:28.157223+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:53:44.909991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:30.428153+0000","last_change":"2024-03-28T22:33:28.948725+0000","last_active":"2024-03-28T22:34:30.428153+0000","last_peered":"2024-03-28T22:34:30.428153+0000","last_clean":"2024-03-28T22:34:30.428153+0000","last_became_active":"2024-03-28T22:33:28.943942+0000","last_became_peered":"2024-03-28T22:33:28.943942+0000","last_unstale":"2024-03-28T22:34:30.428153+0000","last_undegraded":"2024-03-28T22:34:30.428153+0000","last_fullsized":"2024-03-28T22:34:30.428153+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:50:04.238538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.978828+0000","last_change":"2024-03-28T22:33:28.941546+0000","last_active":"2024-03-28T22:34:29.978828+0000","last_peered":"2024-03-28T22:34:29.978828+0000","last_clean":"2024-03-28T22:34:29.978828+0000","last_became_active":"2024-03-28T22:33:28.941278+0000","last_became_peered":"2024-03-28T22:33:28.941278+0000","last_unstale":"2024-03-28T22:34:29.978828+0000","last_undegraded":"2024-03-28T22:34:29.978828+0000","last_fullsized":"2024-03-28T22:34:29.978828+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:47:10.047067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:30.428429+0000","last_change":"2024-03-28T22:33:28.949552+0000","last_active":"2024-03-28T22:34:30.428429+0000","last_peered":"2024-03-28T22:34:30.428429+0000","last_clean":"2024-03-28T22:34:30.428429+0000","last_became_active":"2024-03-28T22:33:28.947432+0000","last_became_peered":"2024-03-28T22:33:28.947432+0000","last_unstale":"2024-03-28T22:34:30.428429+0000","last_undegraded":"2024-03-28T22:34:30.428429+0000","last_fullsized":"2024-03-28T22:34:30.428429+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:01:16.534642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"94'10","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-28T22:35:29.246779+0000","last_change":"2024-03-28T22:33:28.953930+0000","last_active":"2024-03-28T22:35:29.246779+0000","last_peered":"2024-03-28T22:35:29.246779+0000","last_clean":"2024-03-28T22:35:29.246779+0000","last_became_active":"2024-03-28T22:33:28.953591+0000","last_became_peered":"2024-03-28T22:33:28.953591+0000","last_unstale":"2024-03-28T22:35:29.246779+0000","last_undegraded":"2024-03-28T22:35:29.246779+0000","last_fullsized":"2024-03-28T22:35:29.246779+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:42:55.857476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:30.151662+0000","last_change":"2024-03-28T22:33:28.946307+0000","last_active":"2024-03-28T22:34:30.151662+0000","last_peered":"2024-03-28T22:34:30.151662+0000","last_clean":"2024-03-28T22:34:30.151662+0000","last_became_active":"2024-03-28T22:33:28.945932+0000","last_became_peered":"2024-03-28T22:33:28.945932+0000","last_unstale":"2024-03-28T22:34:30.151662+0000","last_undegraded":"2024-03-28T22:34:30.151662+0000","last_fullsized":"2024-03-28T22:34:30.151662+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:54:59.651668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-28T22:34:29.976142+0000","last_change":"2024-03-28T22:33:28.928014+0000","last_active":"2024-03-28T22:34:29.976142+0000","last_peered":"2024-03-28T22:34:29.976142+0000","last_clean":"2024-03-28T22:34:29.976142+0000","last_became_active":"2024-03-28T22:33:28.927686+0000","last_became_peered":"2024-03-28T22:33:28.927686+0000","last_unstale":"2024-03-28T22:34:29.976142+0000","last_undegraded":"2024-03-28T22:34:29.976142+0000","last_fullsized":"2024-03-28T22:34:29.976142+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:47:08.131565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":267,"ondisk_log_size":267,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71268,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":863,"num_read_kb":511,"num_write":246,"num_write_kb":144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":214086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":214,"ondisk_log_size":214,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318310,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32228,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712924,"statfs":{"total":95995035648,"available":95962034176,"internally_reserved":0,"allocated":1073152,"data_stored":520782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579947,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28740,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716412,"statfs":{"total":95995035648,"available":95965605888,"internally_reserved":0,"allocated":1765376,"data_stored":1172563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743473,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1097728,"data_stored":524786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005111,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1085440,"data_stored":521046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299453,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36984,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708168,"statfs":{"total":95995035648,"available":95957164032,"internally_reserved":0,"allocated":1687552,"data_stored":1117706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495682,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1069056,"data_stored":516194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36448,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708704,"statfs":{"total":95995035648,"available":95957712896,"internally_reserved":0,"allocated":1138688,"data_stored":581679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1093632,"data_stored":517650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36468,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708684,"statfs":{"total":95995035648,"available":95957692416,"internally_reserved":0,"allocated":1159168,"data_stored":581899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182482,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1081344,"data_stored":515902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":520659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757320,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28680,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716472,"statfs":{"total":95995035648,"available":95965667328,"internally_reserved":0,"allocated":1695744,"data_stored":1112756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":66207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":66427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":66723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T22:35:32.145 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-28T22:35:32.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:32 smithi043 ceph-mon[31823]: from='client.25037 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:35:32.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:32 smithi172 ceph-mon[36566]: from='client.25037 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:35:32.654 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:35:32.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:32 smithi082 ceph-mon[36471]: from='client.25037 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:35:33.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:33 smithi043 ceph-mon[31823]: pgmap v388: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 511 B/s wr, 11 op/s 2024-03-28T22:35:33.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:33 smithi172 ceph-mon[36566]: pgmap v388: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 511 B/s wr, 11 op/s 2024-03-28T22:35:33.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:33 smithi082 ceph-mon[36471]: pgmap v388: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 511 B/s wr, 11 op/s 2024-03-28T22:35:34.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.829+0000 7ff9e2200700 1 -- 172.21.15.43:0/2622046529 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9dc10a4c0 msgr2=0x7ff9dc10a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:34.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.829+0000 7ff9e2200700 1 --2- 172.21.15.43:0/2622046529 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9dc10a4c0 0x7ff9dc10a8a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc009f90 tx=0x7ff9cc00b330 comp rx=0 tx=0).stop 2024-03-28T22:35:34.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.829+0000 7ff9e2200700 1 -- 172.21.15.43:0/2622046529 shutdown_connections 2024-03-28T22:35:34.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.829+0000 7ff9e2200700 1 --2- 172.21.15.43:0/2622046529 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9dc104df0 0x7ff9dc111300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:34.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.829+0000 7ff9e2200700 1 --2- 172.21.15.43:0/2622046529 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9dc10a4c0 0x7ff9dc10a8a0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:34.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.829+0000 7ff9e2200700 1 --2- 172.21.15.43:0/2622046529 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9dc104430 0x7ff9dc1048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:34.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.829+0000 7ff9e2200700 1 -- 172.21.15.43:0/2622046529 >> 172.21.15.43:0/2622046529 conn(0x7ff9dc0fff20 msgr2=0x7ff9dc102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:34.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.829+0000 7ff9e2200700 1 -- 172.21.15.43:0/2622046529 shutdown_connections 2024-03-28T22:35:34.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.829+0000 7ff9e2200700 1 -- 172.21.15.43:0/2622046529 wait complete. 2024-03-28T22:35:34.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.830+0000 7ff9e2200700 1 Processor -- start 2024-03-28T22:35:34.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.831+0000 7ff9e2200700 1 -- start start 2024-03-28T22:35:34.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.831+0000 7ff9e2200700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9dc104430 0x7ff9dc19abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:34.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.831+0000 7ff9e2200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9dc104df0 0x7ff9dc19b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:34.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.831+0000 7ff9e2200700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9dc10a4c0 0x7ff9dc19f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:34.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.831+0000 7ff9e2200700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc114740 con 0x7ff9dc104df0 2024-03-28T22:35:34.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.831+0000 7ff9e2200700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc1145c0 con 0x7ff9dc10a4c0 2024-03-28T22:35:34.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.831+0000 7ff9e2200700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc1148c0 con 0x7ff9dc104430 2024-03-28T22:35:34.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.831+0000 7ff9db400700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9dc104430 0x7ff9dc19abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:34.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.831+0000 7ff9dbe00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9dc10a4c0 0x7ff9dc19f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:34.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.831+0000 7ff9db400700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9dc104430 0x7ff9dc19abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:34434/0 (socket says 172.21.15.43:34434) 2024-03-28T22:35:34.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.831+0000 7ff9db400700 1 -- 172.21.15.43:0/1716613201 learned_addr learned my addr 172.21.15.43:0/1716613201 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:35:34.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.831+0000 7ff9d2a00700 1 --2- 172.21.15.43:0/1716613201 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9dc104df0 0x7ff9dc19b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:34.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.832+0000 7ff9dbe00700 1 -- 172.21.15.43:0/1716613201 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9dc104430 msgr2=0x7ff9dc19abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:34.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.832+0000 7ff9dbe00700 1 --2- 172.21.15.43:0/1716613201 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9dc104430 0x7ff9dc19abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:34.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.832+0000 7ff9dbe00700 1 -- 172.21.15.43:0/1716613201 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9dc104df0 msgr2=0x7ff9dc19b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:34.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.832+0000 7ff9dbe00700 1 --2- 172.21.15.43:0/1716613201 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9dc104df0 0x7ff9dc19b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:34.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.832+0000 7ff9dbe00700 1 -- 172.21.15.43:0/1716613201 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9dc19fbd0 con 0x7ff9dc10a4c0 2024-03-28T22:35:34.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.832+0000 7ff9db400700 1 --2- 172.21.15.43:0/1716613201 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9dc104430 0x7ff9dc19abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:35:34.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.832+0000 7ff9dbe00700 1 --2- 172.21.15.43:0/1716613201 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9dc10a4c0 0x7ff9dc19f4d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff9d400e940 tx=0x7ff9d400ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:34.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.832+0000 7ff9d8c00700 1 -- 172.21.15.43:0/1716613201 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d4004220 con 0x7ff9dc10a4c0 2024-03-28T22:35:34.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.832+0000 7ff9d8c00700 1 -- 172.21.15.43:0/1716613201 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9d4004d10 con 0x7ff9dc10a4c0 2024-03-28T22:35:34.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.832+0000 7ff9d8c00700 1 -- 172.21.15.43:0/1716613201 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d4010700 con 0x7ff9dc10a4c0 2024-03-28T22:35:34.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.832+0000 7ff9e2200700 1 -- 172.21.15.43:0/1716613201 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9dc19fec0 con 0x7ff9dc10a4c0 2024-03-28T22:35:34.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.833+0000 7ff9e2200700 1 -- 172.21.15.43:0/1716613201 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9dc1a7780 con 0x7ff9dc10a4c0 2024-03-28T22:35:34.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.834+0000 7ff9e2200700 1 -- 172.21.15.43:0/1716613201 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9dc10c9c0 con 0x7ff9dc10a4c0 2024-03-28T22:35:34.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.836+0000 7ff9d8c00700 1 -- 172.21.15.43:0/1716613201 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7ff9d4004750 con 0x7ff9dc10a4c0 2024-03-28T22:35:34.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.837+0000 7ff9d8c00700 1 --2- 172.21.15.43:0/1716613201 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff9c00a1e90 0x7ff9c00a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:34.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.837+0000 7ff9d8c00700 1 -- 172.21.15.43:0/1716613201 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7ff9d40cde80 con 0x7ff9dc10a4c0 2024-03-28T22:35:34.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.837+0000 7ff9db400700 1 --2- 172.21.15.43:0/1716613201 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff9c00a1e90 0x7ff9c00a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:34.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.838+0000 7ff9db400700 1 --2- 172.21.15.43:0/1716613201 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff9c00a1e90 0x7ff9c00a4350 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc00b6c0 tx=0x7ff9cc0094b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:34.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:34.843+0000 7ff9d8c00700 1 -- 172.21.15.43:0/1716613201 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff9d4052280 con 0x7ff9dc10a4c0 2024-03-28T22:35:35.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:35.069+0000 7ff9e2200700 1 -- 172.21.15.43:0/1716613201 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff9dc062680 con 0x7ff9c00a1e90 2024-03-28T22:35:35.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:35.098+0000 7ff9d8c00700 1 -- 172.21.15.43:0/1716613201 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340545 (secure 0 0 0) 0x7ff9dc062680 con 0x7ff9c00a1e90 2024-03-28T22:35:35.099 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:35:35.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:35.106+0000 7ff9e2200700 1 -- 172.21.15.43:0/1716613201 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff9c00a1e90 msgr2=0x7ff9c00a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:35.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:35.106+0000 7ff9e2200700 1 --2- 172.21.15.43:0/1716613201 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff9c00a1e90 0x7ff9c00a4350 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc00b6c0 tx=0x7ff9cc0094b0 comp rx=0 tx=0).stop 2024-03-28T22:35:35.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:35.106+0000 7ff9e2200700 1 -- 172.21.15.43:0/1716613201 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9dc10a4c0 msgr2=0x7ff9dc19f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:35.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:35.106+0000 7ff9e2200700 1 --2- 172.21.15.43:0/1716613201 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9dc10a4c0 0x7ff9dc19f4d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff9d400e940 tx=0x7ff9d400ec50 comp rx=0 tx=0).stop 2024-03-28T22:35:35.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:35.106+0000 7ff9e2200700 1 -- 172.21.15.43:0/1716613201 shutdown_connections 2024-03-28T22:35:35.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:35.106+0000 7ff9e2200700 1 --2- 172.21.15.43:0/1716613201 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff9c00a1e90 0x7ff9c00a4350 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:35.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:35.106+0000 7ff9e2200700 1 --2- 172.21.15.43:0/1716613201 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9dc10a4c0 0x7ff9dc19f4d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:35.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:35.106+0000 7ff9e2200700 1 --2- 172.21.15.43:0/1716613201 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9dc104430 0x7ff9dc19abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:35.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:35.106+0000 7ff9e2200700 1 --2- 172.21.15.43:0/1716613201 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9dc104df0 0x7ff9dc19b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:35.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:35.106+0000 7ff9e2200700 1 -- 172.21.15.43:0/1716613201 >> 172.21.15.43:0/1716613201 conn(0x7ff9dc0fff20 msgr2=0x7ff9dc1004c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:35.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:35.106+0000 7ff9e2200700 1 -- 172.21.15.43:0/1716613201 shutdown_connections 2024-03-28T22:35:35.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:35.106+0000 7ff9e2200700 1 -- 172.21.15.43:0/1716613201 wait complete. 2024-03-28T22:35:35.108 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-03-28T22:35:35.502 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:35 smithi043 ceph-mon[31823]: pgmap v389: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 639 B/s wr, 11 op/s 2024-03-28T22:35:35.502 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:35 smithi043 ceph-mon[31823]: from='client.25043 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:35:35.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:35 smithi172 ceph-mon[36566]: pgmap v389: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 639 B/s wr, 11 op/s 2024-03-28T22:35:35.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:35 smithi172 ceph-mon[36566]: from='client.25043 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:35:35.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:35 smithi082 ceph-mon[36471]: pgmap v389: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 639 B/s wr, 11 op/s 2024-03-28T22:35:35.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:35 smithi082 ceph-mon[36471]: from='client.25043 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:35:35.931 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":389,"stamp":"2024-03-28T22:35:34.206013+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662011,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1030,"num_read_kb":641,"num_write":433,"num_write_kb":3086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":856,"ondisk_log_size":856,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":376700,"kb_used_data":14992,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124565124,"statfs":{"total":1151940427776,"available":1151554686976,"internally_reserved":0,"allocated":15351808,"data_stored":8391503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":37,"num_write":7,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002835"},"pg_stats":[{"pgid":"3.3f","version":"94'8","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.450023+0000","last_change":"2024-03-28T22:33:28.942904+0000","last_active":"2024-03-28T22:35:30.450023+0000","last_peered":"2024-03-28T22:35:30.450023+0000","last_clean":"2024-03-28T22:35:30.450023+0000","last_became_active":"2024-03-28T22:33:28.942574+0000","last_became_peered":"2024-03-28T22:33:28.942574+0000","last_unstale":"2024-03-28T22:35:30.450023+0000","last_undegraded":"2024-03-28T22:35:30.450023+0000","last_fullsized":"2024-03-28T22:35:30.450023+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:55.323919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"94'4","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451519+0000","last_change":"2024-03-28T22:33:28.951795+0000","last_active":"2024-03-28T22:35:30.451519+0000","last_peered":"2024-03-28T22:35:30.451519+0000","last_clean":"2024-03-28T22:35:30.451519+0000","last_became_active":"2024-03-28T22:33:28.951286+0000","last_became_peered":"2024-03-28T22:33:28.951286+0000","last_unstale":"2024-03-28T22:35:30.451519+0000","last_undegraded":"2024-03-28T22:35:30.451519+0000","last_fullsized":"2024-03-28T22:35:30.451519+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:56:11.718244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451291+0000","last_change":"2024-03-28T22:33:28.949476+0000","last_active":"2024-03-28T22:35:30.451291+0000","last_peered":"2024-03-28T22:35:30.451291+0000","last_clean":"2024-03-28T22:35:30.451291+0000","last_became_active":"2024-03-28T22:33:28.949128+0000","last_became_peered":"2024-03-28T22:33:28.949128+0000","last_unstale":"2024-03-28T22:35:30.451291+0000","last_undegraded":"2024-03-28T22:35:30.451291+0000","last_fullsized":"2024-03-28T22:35:30.451291+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:49:29.052373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370703+0000","last_change":"2024-03-28T22:33:28.930715+0000","last_active":"2024-03-28T22:35:30.370703+0000","last_peered":"2024-03-28T22:35:30.370703+0000","last_clean":"2024-03-28T22:35:30.370703+0000","last_became_active":"2024-03-28T22:33:28.930265+0000","last_became_peered":"2024-03-28T22:33:28.930265+0000","last_unstale":"2024-03-28T22:35:30.370703+0000","last_undegraded":"2024-03-28T22:35:30.370703+0000","last_fullsized":"2024-03-28T22:35:30.370703+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:29:54.695671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"94'4","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.253628+0000","last_change":"2024-03-28T22:33:28.944933+0000","last_active":"2024-03-28T22:35:30.253628+0000","last_peered":"2024-03-28T22:35:30.253628+0000","last_clean":"2024-03-28T22:35:30.253628+0000","last_became_active":"2024-03-28T22:33:28.944542+0000","last_became_peered":"2024-03-28T22:33:28.944542+0000","last_unstale":"2024-03-28T22:35:30.253628+0000","last_undegraded":"2024-03-28T22:35:30.253628+0000","last_fullsized":"2024-03-28T22:35:30.253628+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:35:15.719312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"94'6","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-28T22:35:29.244883+0000","last_change":"2024-03-28T22:33:28.927920+0000","last_active":"2024-03-28T22:35:29.244883+0000","last_peered":"2024-03-28T22:35:29.244883+0000","last_clean":"2024-03-28T22:35:29.244883+0000","last_became_active":"2024-03-28T22:33:28.927602+0000","last_became_peered":"2024-03-28T22:33:28.927602+0000","last_unstale":"2024-03-28T22:35:29.244883+0000","last_undegraded":"2024-03-28T22:35:29.244883+0000","last_fullsized":"2024-03-28T22:35:29.244883+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:24:22.072707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"94'5","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248009+0000","last_change":"2024-03-28T22:33:28.917963+0000","last_active":"2024-03-28T22:35:30.248009+0000","last_peered":"2024-03-28T22:35:30.248009+0000","last_clean":"2024-03-28T22:35:30.248009+0000","last_became_active":"2024-03-28T22:33:28.917628+0000","last_became_peered":"2024-03-28T22:33:28.917628+0000","last_unstale":"2024-03-28T22:35:30.248009+0000","last_undegraded":"2024-03-28T22:35:30.248009+0000","last_fullsized":"2024-03-28T22:35:30.248009+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:37:09.111640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"94'9","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371107+0000","last_change":"2024-03-28T22:33:28.942632+0000","last_active":"2024-03-28T22:35:30.371107+0000","last_peered":"2024-03-28T22:35:30.371107+0000","last_clean":"2024-03-28T22:35:30.371107+0000","last_became_active":"2024-03-28T22:33:28.941972+0000","last_became_peered":"2024-03-28T22:33:28.941972+0000","last_unstale":"2024-03-28T22:35:30.371107+0000","last_undegraded":"2024-03-28T22:35:30.371107+0000","last_fullsized":"2024-03-28T22:35:30.371107+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:49:52.642314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"94'11","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-28T22:35:29.247332+0000","last_change":"2024-03-28T22:33:28.924324+0000","last_active":"2024-03-28T22:35:29.247332+0000","last_peered":"2024-03-28T22:35:29.247332+0000","last_clean":"2024-03-28T22:35:29.247332+0000","last_became_active":"2024-03-28T22:33:28.924014+0000","last_became_peered":"2024-03-28T22:33:28.924014+0000","last_unstale":"2024-03-28T22:35:29.247332+0000","last_undegraded":"2024-03-28T22:35:29.247332+0000","last_fullsized":"2024-03-28T22:35:29.247332+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:17:55.172116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"94'16","reported_seq":51,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-28T22:35:29.245358+0000","last_change":"2024-03-28T22:34:29.987262+0000","last_active":"2024-03-28T22:35:29.245358+0000","last_peered":"2024-03-28T22:35:29.245358+0000","last_clean":"2024-03-28T22:35:29.245358+0000","last_became_active":"2024-03-28T22:34:29.986934+0000","last_became_peered":"2024-03-28T22:34:29.986934+0000","last_unstale":"2024-03-28T22:35:29.245358+0000","last_undegraded":"2024-03-28T22:35:29.245358+0000","last_fullsized":"2024-03-28T22:35:29.245358+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:00:41.574800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"94'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248464+0000","last_change":"2024-03-28T22:33:28.930802+0000","last_active":"2024-03-28T22:35:30.248464+0000","last_peered":"2024-03-28T22:35:30.248464+0000","last_clean":"2024-03-28T22:35:30.248464+0000","last_became_active":"2024-03-28T22:33:28.930475+0000","last_became_peered":"2024-03-28T22:33:28.930475+0000","last_unstale":"2024-03-28T22:35:30.248464+0000","last_undegraded":"2024-03-28T22:35:30.248464+0000","last_fullsized":"2024-03-28T22:35:30.248464+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:48.443803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"94'7","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370644+0000","last_change":"2024-03-28T22:33:28.927057+0000","last_active":"2024-03-28T22:35:30.370644+0000","last_peered":"2024-03-28T22:35:30.370644+0000","last_clean":"2024-03-28T22:35:30.370644+0000","last_became_active":"2024-03-28T22:33:28.926598+0000","last_became_peered":"2024-03-28T22:33:28.926598+0000","last_unstale":"2024-03-28T22:35:30.370644+0000","last_undegraded":"2024-03-28T22:35:30.370644+0000","last_fullsized":"2024-03-28T22:35:30.370644+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:21:00.710203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"94'10","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368883+0000","last_change":"2024-03-28T22:33:28.937034+0000","last_active":"2024-03-28T22:35:30.368883+0000","last_peered":"2024-03-28T22:35:30.368883+0000","last_clean":"2024-03-28T22:35:30.368883+0000","last_became_active":"2024-03-28T22:33:28.933713+0000","last_became_peered":"2024-03-28T22:33:28.933713+0000","last_unstale":"2024-03-28T22:35:30.368883+0000","last_undegraded":"2024-03-28T22:35:30.368883+0000","last_fullsized":"2024-03-28T22:35:30.368883+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:58:24.148897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"94'7","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.450910+0000","last_change":"2024-03-28T22:33:28.947384+0000","last_active":"2024-03-28T22:35:30.450910+0000","last_peered":"2024-03-28T22:35:30.450910+0000","last_clean":"2024-03-28T22:35:30.450910+0000","last_became_active":"2024-03-28T22:33:28.947017+0000","last_became_peered":"2024-03-28T22:33:28.947017+0000","last_unstale":"2024-03-28T22:35:30.450910+0000","last_undegraded":"2024-03-28T22:35:30.450910+0000","last_fullsized":"2024-03-28T22:35:30.450910+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:43:32.781161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"94'7","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451705+0000","last_change":"2024-03-28T22:33:28.950119+0000","last_active":"2024-03-28T22:35:30.451705+0000","last_peered":"2024-03-28T22:35:30.451705+0000","last_clean":"2024-03-28T22:35:30.451705+0000","last_became_active":"2024-03-28T22:33:28.949798+0000","last_became_peered":"2024-03-28T22:33:28.949798+0000","last_unstale":"2024-03-28T22:35:30.451705+0000","last_undegraded":"2024-03-28T22:35:30.451705+0000","last_fullsized":"2024-03-28T22:35:30.451705+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:16.717678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"94'9","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-28T22:35:29.246476+0000","last_change":"2024-03-28T22:33:28.943255+0000","last_active":"2024-03-28T22:35:29.246476+0000","last_peered":"2024-03-28T22:35:29.246476+0000","last_clean":"2024-03-28T22:35:29.246476+0000","last_became_active":"2024-03-28T22:33:28.942923+0000","last_became_peered":"2024-03-28T22:33:28.942923+0000","last_unstale":"2024-03-28T22:35:29.246476+0000","last_undegraded":"2024-03-28T22:35:29.246476+0000","last_fullsized":"2024-03-28T22:35:29.246476+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:05:04.197635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"94'8","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449920+0000","last_change":"2024-03-28T22:33:28.942906+0000","last_active":"2024-03-28T22:35:30.449920+0000","last_peered":"2024-03-28T22:35:30.449920+0000","last_clean":"2024-03-28T22:35:30.449920+0000","last_became_active":"2024-03-28T22:33:28.942573+0000","last_became_peered":"2024-03-28T22:33:28.942573+0000","last_unstale":"2024-03-28T22:35:30.449920+0000","last_undegraded":"2024-03-28T22:35:30.449920+0000","last_fullsized":"2024-03-28T22:35:30.449920+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:39:17.651101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"94'10","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252793+0000","last_change":"2024-03-28T22:33:28.932075+0000","last_active":"2024-03-28T22:35:30.252793+0000","last_peered":"2024-03-28T22:35:30.252793+0000","last_clean":"2024-03-28T22:35:30.252793+0000","last_became_active":"2024-03-28T22:33:28.931689+0000","last_became_peered":"2024-03-28T22:33:28.931689+0000","last_unstale":"2024-03-28T22:35:30.252793+0000","last_undegraded":"2024-03-28T22:35:30.252793+0000","last_fullsized":"2024-03-28T22:35:30.252793+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:50:43.031082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"94'7","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252280+0000","last_change":"2024-03-28T22:33:28.944987+0000","last_active":"2024-03-28T22:35:30.252280+0000","last_peered":"2024-03-28T22:35:30.252280+0000","last_clean":"2024-03-28T22:35:30.252280+0000","last_became_active":"2024-03-28T22:33:28.944622+0000","last_became_peered":"2024-03-28T22:33:28.944622+0000","last_unstale":"2024-03-28T22:35:30.252280+0000","last_undegraded":"2024-03-28T22:35:30.252280+0000","last_fullsized":"2024-03-28T22:35:30.252280+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:48:58.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":29,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251775+0000","last_change":"2024-03-28T22:34:30.429709+0000","last_active":"2024-03-28T22:35:30.251775+0000","last_peered":"2024-03-28T22:35:30.251775+0000","last_clean":"2024-03-28T22:35:30.251775+0000","last_became_active":"2024-03-28T22:34:30.429122+0000","last_became_peered":"2024-03-28T22:34:30.429122+0000","last_unstale":"2024-03-28T22:35:30.251775+0000","last_undegraded":"2024-03-28T22:35:30.251775+0000","last_fullsized":"2024-03-28T22:35:30.251775+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:58:05.313669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"92'13","reported_seq":133,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.318893+0000","last_change":"2024-03-28T22:33:28.442524+0000","last_active":"2024-03-28T22:35:30.318893+0000","last_peered":"2024-03-28T22:35:30.318893+0000","last_clean":"2024-03-28T22:35:30.318893+0000","last_became_active":"2024-03-28T22:33:28.442169+0000","last_became_peered":"2024-03-28T22:33:28.442169+0000","last_unstale":"2024-03-28T22:35:30.318893+0000","last_undegraded":"2024-03-28T22:35:30.318893+0000","last_fullsized":"2024-03-28T22:35:30.318893+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:08:47.101119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":87,"num_read_kb":87,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"94'5","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371822+0000","last_change":"2024-03-28T22:33:28.946697+0000","last_active":"2024-03-28T22:35:30.371822+0000","last_peered":"2024-03-28T22:35:30.371822+0000","last_clean":"2024-03-28T22:35:30.371822+0000","last_became_active":"2024-03-28T22:33:28.946303+0000","last_became_peered":"2024-03-28T22:33:28.946303+0000","last_unstale":"2024-03-28T22:35:30.371822+0000","last_undegraded":"2024-03-28T22:35:30.371822+0000","last_fullsized":"2024-03-28T22:35:30.371822+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:32:01.286073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370317+0000","last_change":"2024-03-28T22:34:30.429579+0000","last_active":"2024-03-28T22:35:30.370317+0000","last_peered":"2024-03-28T22:35:30.370317+0000","last_clean":"2024-03-28T22:35:30.370317+0000","last_became_active":"2024-03-28T22:34:30.429054+0000","last_became_peered":"2024-03-28T22:34:30.429054+0000","last_unstale":"2024-03-28T22:35:30.370317+0000","last_undegraded":"2024-03-28T22:35:30.370317+0000","last_fullsized":"2024-03-28T22:35:30.370317+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:03:46.492786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"92'2","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.372174+0000","last_change":"2024-03-28T22:33:28.457661+0000","last_active":"2024-03-28T22:35:30.372174+0000","last_peered":"2024-03-28T22:35:30.372174+0000","last_clean":"2024-03-28T22:35:30.372174+0000","last_became_active":"2024-03-28T22:33:28.457032+0000","last_became_peered":"2024-03-28T22:33:28.457032+0000","last_unstale":"2024-03-28T22:35:30.372174+0000","last_undegraded":"2024-03-28T22:35:30.372174+0000","last_fullsized":"2024-03-28T22:35:30.372174+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:43:09.820630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"94'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.453641+0000","last_change":"2024-03-28T22:33:28.954252+0000","last_active":"2024-03-28T22:35:30.453641+0000","last_peered":"2024-03-28T22:35:30.453641+0000","last_clean":"2024-03-28T22:35:30.453641+0000","last_became_active":"2024-03-28T22:33:28.953922+0000","last_became_peered":"2024-03-28T22:33:28.953922+0000","last_unstale":"2024-03-28T22:35:30.453641+0000","last_undegraded":"2024-03-28T22:35:30.453641+0000","last_fullsized":"2024-03-28T22:35:30.453641+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:18:47.097859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370557+0000","last_change":"2024-03-28T22:33:34.029129+0000","last_active":"2024-03-28T22:35:30.370557+0000","last_peered":"2024-03-28T22:35:30.370557+0000","last_clean":"2024-03-28T22:35:30.370557+0000","last_became_active":"2024-03-28T22:33:34.028452+0000","last_became_peered":"2024-03-28T22:33:34.028452+0000","last_unstale":"2024-03-28T22:35:30.370557+0000","last_undegraded":"2024-03-28T22:35:30.370557+0000","last_fullsized":"2024-03-28T22:35:30.370557+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:28.982773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"92'12","reported_seq":46,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254159+0000","last_change":"2024-03-28T22:33:28.439568+0000","last_active":"2024-03-28T22:35:30.254159+0000","last_peered":"2024-03-28T22:35:30.254159+0000","last_clean":"2024-03-28T22:35:30.254159+0000","last_became_active":"2024-03-28T22:33:28.438887+0000","last_became_peered":"2024-03-28T22:33:28.438887+0000","last_unstale":"2024-03-28T22:35:30.254159+0000","last_undegraded":"2024-03-28T22:35:30.254159+0000","last_fullsized":"2024-03-28T22:35:30.254159+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:56:43.617621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"94'9","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-28T22:35:29.247118+0000","last_change":"2024-03-28T22:33:28.938311+0000","last_active":"2024-03-28T22:35:29.247118+0000","last_peered":"2024-03-28T22:35:29.247118+0000","last_clean":"2024-03-28T22:35:29.247118+0000","last_became_active":"2024-03-28T22:33:28.937997+0000","last_became_peered":"2024-03-28T22:33:28.937997+0000","last_unstale":"2024-03-28T22:35:29.247118+0000","last_undegraded":"2024-03-28T22:35:29.247118+0000","last_fullsized":"2024-03-28T22:35:29.247118+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:54:44.001545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370378+0000","last_change":"2024-03-28T22:33:34.016931+0000","last_active":"2024-03-28T22:35:30.370378+0000","last_peered":"2024-03-28T22:35:30.370378+0000","last_clean":"2024-03-28T22:35:30.370378+0000","last_became_active":"2024-03-28T22:33:34.016559+0000","last_became_peered":"2024-03-28T22:33:34.016559+0000","last_unstale":"2024-03-28T22:35:30.370378+0000","last_undegraded":"2024-03-28T22:35:30.370378+0000","last_fullsized":"2024-03-28T22:35:30.370378+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:08:31.252351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"92'5","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254044+0000","last_change":"2024-03-28T22:33:28.433166+0000","last_active":"2024-03-28T22:35:30.254044+0000","last_peered":"2024-03-28T22:35:30.254044+0000","last_clean":"2024-03-28T22:35:30.254044+0000","last_became_active":"2024-03-28T22:33:28.432716+0000","last_became_peered":"2024-03-28T22:33:28.432716+0000","last_unstale":"2024-03-28T22:35:30.254044+0000","last_undegraded":"2024-03-28T22:35:30.254044+0000","last_fullsized":"2024-03-28T22:35:30.254044+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:42:41.613583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"94'15","reported_seq":46,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275481+0000","last_change":"2024-03-28T22:33:28.923307+0000","last_active":"2024-03-28T22:35:30.275481+0000","last_peered":"2024-03-28T22:35:30.275481+0000","last_clean":"2024-03-28T22:35:30.275481+0000","last_became_active":"2024-03-28T22:33:28.922499+0000","last_became_peered":"2024-03-28T22:33:28.922499+0000","last_unstale":"2024-03-28T22:35:30.275481+0000","last_undegraded":"2024-03-28T22:35:30.275481+0000","last_fullsized":"2024-03-28T22:35:30.275481+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:48:48.514482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252136+0000","last_change":"2024-03-28T22:33:34.002209+0000","last_active":"2024-03-28T22:35:30.252136+0000","last_peered":"2024-03-28T22:35:30.252136+0000","last_clean":"2024-03-28T22:35:30.252136+0000","last_became_active":"2024-03-28T22:33:34.001746+0000","last_became_peered":"2024-03-28T22:33:34.001746+0000","last_unstale":"2024-03-28T22:35:30.252136+0000","last_undegraded":"2024-03-28T22:35:30.252136+0000","last_fullsized":"2024-03-28T22:35:30.252136+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:02:02.333328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"92'5","reported_seq":126,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.335337+0000","last_change":"2024-03-28T22:33:28.435380+0000","last_active":"2024-03-28T22:35:30.335337+0000","last_peered":"2024-03-28T22:35:30.335337+0000","last_clean":"2024-03-28T22:35:30.335337+0000","last_became_active":"2024-03-28T22:33:28.434898+0000","last_became_peered":"2024-03-28T22:33:28.434898+0000","last_unstale":"2024-03-28T22:35:30.335337+0000","last_undegraded":"2024-03-28T22:35:30.335337+0000","last_fullsized":"2024-03-28T22:35:30.335337+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:58:06.810394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"94'7","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254211+0000","last_change":"2024-03-28T22:34:30.121370+0000","last_active":"2024-03-28T22:35:30.254211+0000","last_peered":"2024-03-28T22:35:30.254211+0000","last_clean":"2024-03-28T22:35:30.254211+0000","last_became_active":"2024-03-28T22:34:30.121086+0000","last_became_peered":"2024-03-28T22:34:30.121086+0000","last_unstale":"2024-03-28T22:35:30.254211+0000","last_undegraded":"2024-03-28T22:35:30.254211+0000","last_fullsized":"2024-03-28T22:35:30.254211+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:26:24.244663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252008+0000","last_change":"2024-03-28T22:33:34.031350+0000","last_active":"2024-03-28T22:35:30.252008+0000","last_peered":"2024-03-28T22:35:30.252008+0000","last_clean":"2024-03-28T22:35:30.252008+0000","last_became_active":"2024-03-28T22:33:34.030946+0000","last_became_peered":"2024-03-28T22:33:34.030946+0000","last_unstale":"2024-03-28T22:35:30.252008+0000","last_undegraded":"2024-03-28T22:35:30.252008+0000","last_fullsized":"2024-03-28T22:35:30.252008+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:54:22.907363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"92'7","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-28T22:35:29.244791+0000","last_change":"2024-03-28T22:33:28.446197+0000","last_active":"2024-03-28T22:35:29.244791+0000","last_peered":"2024-03-28T22:35:29.244791+0000","last_clean":"2024-03-28T22:35:29.244791+0000","last_became_active":"2024-03-28T22:33:28.445891+0000","last_became_peered":"2024-03-28T22:33:28.445891+0000","last_unstale":"2024-03-28T22:35:29.244791+0000","last_undegraded":"2024-03-28T22:35:29.244791+0000","last_fullsized":"2024-03-28T22:35:29.244791+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:58:57.385302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275539+0000","last_change":"2024-03-28T22:33:28.955651+0000","last_active":"2024-03-28T22:35:30.275539+0000","last_peered":"2024-03-28T22:35:30.275539+0000","last_clean":"2024-03-28T22:35:30.275539+0000","last_became_active":"2024-03-28T22:33:28.955276+0000","last_became_peered":"2024-03-28T22:33:28.955276+0000","last_unstale":"2024-03-28T22:35:30.275539+0000","last_undegraded":"2024-03-28T22:35:30.275539+0000","last_fullsized":"2024-03-28T22:35:30.275539+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:36:13.455327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449765+0000","last_change":"2024-03-28T22:33:34.017520+0000","last_active":"2024-03-28T22:35:30.449765+0000","last_peered":"2024-03-28T22:35:30.449765+0000","last_clean":"2024-03-28T22:35:30.449765+0000","last_became_active":"2024-03-28T22:33:34.017022+0000","last_became_peered":"2024-03-28T22:33:34.017022+0000","last_unstale":"2024-03-28T22:35:30.449765+0000","last_undegraded":"2024-03-28T22:35:30.449765+0000","last_fullsized":"2024-03-28T22:35:30.449765+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:41:17.431581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"92'3","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368118+0000","last_change":"2024-03-28T22:33:28.445401+0000","last_active":"2024-03-28T22:35:30.368118+0000","last_peered":"2024-03-28T22:35:30.368118+0000","last_clean":"2024-03-28T22:35:30.368118+0000","last_became_active":"2024-03-28T22:33:28.444237+0000","last_became_peered":"2024-03-28T22:33:28.444237+0000","last_unstale":"2024-03-28T22:35:30.368118+0000","last_undegraded":"2024-03-28T22:35:30.368118+0000","last_fullsized":"2024-03-28T22:35:30.368118+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:18:12.820890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"94'8","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.250889+0000","last_change":"2024-03-28T22:34:30.156248+0000","last_active":"2024-03-28T22:35:30.250889+0000","last_peered":"2024-03-28T22:35:30.250889+0000","last_clean":"2024-03-28T22:35:30.250889+0000","last_became_active":"2024-03-28T22:34:30.155056+0000","last_became_peered":"2024-03-28T22:34:30.155056+0000","last_unstale":"2024-03-28T22:35:30.250889+0000","last_undegraded":"2024-03-28T22:35:30.250889+0000","last_fullsized":"2024-03-28T22:35:30.250889+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:28:56.469728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451480+0000","last_change":"2024-03-28T22:33:34.019899+0000","last_active":"2024-03-28T22:35:30.451480+0000","last_peered":"2024-03-28T22:35:30.451480+0000","last_clean":"2024-03-28T22:35:30.451480+0000","last_became_active":"2024-03-28T22:33:34.019392+0000","last_became_peered":"2024-03-28T22:33:34.019392+0000","last_unstale":"2024-03-28T22:35:30.451480+0000","last_undegraded":"2024-03-28T22:35:30.451480+0000","last_fullsized":"2024-03-28T22:35:30.451480+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:08:38.633797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"92'3","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275054+0000","last_change":"2024-03-28T22:33:28.433243+0000","last_active":"2024-03-28T22:35:30.275054+0000","last_peered":"2024-03-28T22:35:30.275054+0000","last_clean":"2024-03-28T22:35:30.275054+0000","last_became_active":"2024-03-28T22:33:28.432927+0000","last_became_peered":"2024-03-28T22:33:28.432927+0000","last_unstale":"2024-03-28T22:35:30.275054+0000","last_undegraded":"2024-03-28T22:35:30.275054+0000","last_fullsized":"2024-03-28T22:35:30.275054+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:40:43.444229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"94'11","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251179+0000","last_change":"2024-03-28T22:33:28.948111+0000","last_active":"2024-03-28T22:35:30.251179+0000","last_peered":"2024-03-28T22:35:30.251179+0000","last_clean":"2024-03-28T22:35:30.251179+0000","last_became_active":"2024-03-28T22:33:28.944649+0000","last_became_peered":"2024-03-28T22:33:28.944649+0000","last_unstale":"2024-03-28T22:35:30.251179+0000","last_undegraded":"2024-03-28T22:35:30.251179+0000","last_fullsized":"2024-03-28T22:35:30.251179+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:57:11.483037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371445+0000","last_change":"2024-03-28T22:33:34.000837+0000","last_active":"2024-03-28T22:35:30.371445+0000","last_peered":"2024-03-28T22:35:30.371445+0000","last_clean":"2024-03-28T22:35:30.371445+0000","last_became_active":"2024-03-28T22:33:34.000120+0000","last_became_peered":"2024-03-28T22:33:34.000120+0000","last_unstale":"2024-03-28T22:35:30.371445+0000","last_undegraded":"2024-03-28T22:35:30.371445+0000","last_fullsized":"2024-03-28T22:35:30.371445+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:14:32.028952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"92'4","reported_seq":203,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-28T22:35:29.271573+0000","last_change":"2024-03-28T22:33:28.436985+0000","last_active":"2024-03-28T22:35:29.271573+0000","last_peered":"2024-03-28T22:35:29.271573+0000","last_clean":"2024-03-28T22:35:29.271573+0000","last_became_active":"2024-03-28T22:33:28.436625+0000","last_became_peered":"2024-03-28T22:33:28.436625+0000","last_unstale":"2024-03-28T22:35:29.271573+0000","last_undegraded":"2024-03-28T22:35:29.271573+0000","last_fullsized":"2024-03-28T22:35:29.271573+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:50:45.775191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":168,"num_read_kb":168,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"94'10","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-28T22:35:29.247565+0000","last_change":"2024-03-28T22:33:28.933265+0000","last_active":"2024-03-28T22:35:29.247565+0000","last_peered":"2024-03-28T22:35:29.247565+0000","last_clean":"2024-03-28T22:35:29.247565+0000","last_became_active":"2024-03-28T22:33:28.932954+0000","last_became_peered":"2024-03-28T22:33:28.932954+0000","last_unstale":"2024-03-28T22:35:29.247565+0000","last_undegraded":"2024-03-28T22:35:29.247565+0000","last_fullsized":"2024-03-28T22:35:29.247565+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:37.039055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251115+0000","last_change":"2024-03-28T22:33:34.024529+0000","last_active":"2024-03-28T22:35:30.251115+0000","last_peered":"2024-03-28T22:35:30.251115+0000","last_clean":"2024-03-28T22:35:30.251115+0000","last_became_active":"2024-03-28T22:33:34.020949+0000","last_became_peered":"2024-03-28T22:33:34.020949+0000","last_unstale":"2024-03-28T22:35:30.251115+0000","last_undegraded":"2024-03-28T22:35:30.251115+0000","last_fullsized":"2024-03-28T22:35:30.251115+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:28:52.039684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"92'4","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248336+0000","last_change":"2024-03-28T22:33:28.432599+0000","last_active":"2024-03-28T22:35:30.248336+0000","last_peered":"2024-03-28T22:35:30.248336+0000","last_clean":"2024-03-28T22:35:30.248336+0000","last_became_active":"2024-03-28T22:33:28.430710+0000","last_became_peered":"2024-03-28T22:33:28.430710+0000","last_unstale":"2024-03-28T22:35:30.248336+0000","last_undegraded":"2024-03-28T22:35:30.248336+0000","last_fullsized":"2024-03-28T22:35:30.248336+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:54.174747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"94'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.453785+0000","last_change":"2024-03-28T22:33:28.955441+0000","last_active":"2024-03-28T22:35:30.453785+0000","last_peered":"2024-03-28T22:35:30.453785+0000","last_clean":"2024-03-28T22:35:30.453785+0000","last_became_active":"2024-03-28T22:33:28.953269+0000","last_became_peered":"2024-03-28T22:33:28.953269+0000","last_unstale":"2024-03-28T22:35:30.453785+0000","last_undegraded":"2024-03-28T22:35:30.453785+0000","last_fullsized":"2024-03-28T22:35:30.453785+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:47:04.128605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370663+0000","last_change":"2024-03-28T22:33:34.028791+0000","last_active":"2024-03-28T22:35:30.370663+0000","last_peered":"2024-03-28T22:35:30.370663+0000","last_clean":"2024-03-28T22:35:30.370663+0000","last_became_active":"2024-03-28T22:33:34.028361+0000","last_became_peered":"2024-03-28T22:33:34.028361+0000","last_unstale":"2024-03-28T22:35:30.370663+0000","last_undegraded":"2024-03-28T22:35:30.370663+0000","last_fullsized":"2024-03-28T22:35:30.370663+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:42:21.264459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"92'5","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275230+0000","last_change":"2024-03-28T22:33:28.455353+0000","last_active":"2024-03-28T22:35:30.275230+0000","last_peered":"2024-03-28T22:35:30.275230+0000","last_clean":"2024-03-28T22:35:30.275230+0000","last_became_active":"2024-03-28T22:33:28.455090+0000","last_became_peered":"2024-03-28T22:33:28.455090+0000","last_unstale":"2024-03-28T22:35:30.275230+0000","last_undegraded":"2024-03-28T22:35:30.275230+0000","last_fullsized":"2024-03-28T22:35:30.275230+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:19:22.496600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"94'11","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-28T22:35:29.247301+0000","last_change":"2024-03-28T22:33:28.928875+0000","last_active":"2024-03-28T22:35:29.247301+0000","last_peered":"2024-03-28T22:35:29.247301+0000","last_clean":"2024-03-28T22:35:29.247301+0000","last_became_active":"2024-03-28T22:33:28.928449+0000","last_became_peered":"2024-03-28T22:33:28.928449+0000","last_unstale":"2024-03-28T22:35:29.247301+0000","last_undegraded":"2024-03-28T22:35:29.247301+0000","last_fullsized":"2024-03-28T22:35:29.247301+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:34:00.402100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251201+0000","last_change":"2024-03-28T22:33:34.012739+0000","last_active":"2024-03-28T22:35:30.251201+0000","last_peered":"2024-03-28T22:35:30.251201+0000","last_clean":"2024-03-28T22:35:30.251201+0000","last_became_active":"2024-03-28T22:33:34.012262+0000","last_became_peered":"2024-03-28T22:33:34.012262+0000","last_unstale":"2024-03-28T22:35:30.251201+0000","last_undegraded":"2024-03-28T22:35:30.251201+0000","last_fullsized":"2024-03-28T22:35:30.251201+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:24:48.351037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'38","reported_seq":244,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-28T22:35:29.272351+0000","last_change":"2024-03-28T22:33:28.430606+0000","last_active":"2024-03-28T22:35:29.272351+0000","last_peered":"2024-03-28T22:35:29.272351+0000","last_clean":"2024-03-28T22:35:29.272351+0000","last_became_active":"2024-03-28T22:33:28.430233+0000","last_became_peered":"2024-03-28T22:33:28.430233+0000","last_unstale":"2024-03-28T22:35:29.272351+0000","last_undegraded":"2024-03-28T22:35:29.272351+0000","last_fullsized":"2024-03-28T22:35:29.272351+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:49.000569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64455,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":175,"num_read_kb":151,"num_write":40,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"94'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254029+0000","last_change":"2024-03-28T22:33:28.931167+0000","last_active":"2024-03-28T22:35:30.254029+0000","last_peered":"2024-03-28T22:35:30.254029+0000","last_clean":"2024-03-28T22:35:30.254029+0000","last_became_active":"2024-03-28T22:33:28.930824+0000","last_became_peered":"2024-03-28T22:33:28.930824+0000","last_unstale":"2024-03-28T22:35:30.254029+0000","last_undegraded":"2024-03-28T22:35:30.254029+0000","last_fullsized":"2024-03-28T22:35:30.254029+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:46:49.102791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275795+0000","last_change":"2024-03-28T22:33:34.023056+0000","last_active":"2024-03-28T22:35:30.275795+0000","last_peered":"2024-03-28T22:35:30.275795+0000","last_clean":"2024-03-28T22:35:30.275795+0000","last_became_active":"2024-03-28T22:33:34.022612+0000","last_became_peered":"2024-03-28T22:33:34.022612+0000","last_unstale":"2024-03-28T22:35:30.275795+0000","last_undegraded":"2024-03-28T22:35:30.275795+0000","last_fullsized":"2024-03-28T22:35:30.275795+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:52:31.936008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"92'9","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252059+0000","last_change":"2024-03-28T22:33:28.430963+0000","last_active":"2024-03-28T22:35:30.252059+0000","last_peered":"2024-03-28T22:35:30.252059+0000","last_clean":"2024-03-28T22:35:30.252059+0000","last_became_active":"2024-03-28T22:33:28.430632+0000","last_became_peered":"2024-03-28T22:33:28.430632+0000","last_unstale":"2024-03-28T22:35:30.252059+0000","last_undegraded":"2024-03-28T22:35:30.252059+0000","last_fullsized":"2024-03-28T22:35:30.252059+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:30:23.104769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"94'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368323+0000","last_change":"2024-03-28T22:33:28.954105+0000","last_active":"2024-03-28T22:35:30.368323+0000","last_peered":"2024-03-28T22:35:30.368323+0000","last_clean":"2024-03-28T22:35:30.368323+0000","last_became_active":"2024-03-28T22:33:28.953768+0000","last_became_peered":"2024-03-28T22:33:28.953768+0000","last_unstale":"2024-03-28T22:35:30.368323+0000","last_undegraded":"2024-03-28T22:35:30.368323+0000","last_fullsized":"2024-03-28T22:35:30.368323+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:01:25.184523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449514+0000","last_change":"2024-03-28T22:33:34.027976+0000","last_active":"2024-03-28T22:35:30.449514+0000","last_peered":"2024-03-28T22:35:30.449514+0000","last_clean":"2024-03-28T22:35:30.449514+0000","last_became_active":"2024-03-28T22:33:34.027509+0000","last_became_peered":"2024-03-28T22:33:34.027509+0000","last_unstale":"2024-03-28T22:35:30.449514+0000","last_undegraded":"2024-03-28T22:35:30.449514+0000","last_fullsized":"2024-03-28T22:35:30.449514+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:25:41.908441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"92'5","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275257+0000","last_change":"2024-03-28T22:33:28.433283+0000","last_active":"2024-03-28T22:35:30.275257+0000","last_peered":"2024-03-28T22:35:30.275257+0000","last_clean":"2024-03-28T22:35:30.275257+0000","last_became_active":"2024-03-28T22:33:28.432954+0000","last_became_peered":"2024-03-28T22:33:28.432954+0000","last_unstale":"2024-03-28T22:35:30.275257+0000","last_undegraded":"2024-03-28T22:35:30.275257+0000","last_fullsized":"2024-03-28T22:35:30.275257+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:08:34.480459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"94'15","reported_seq":46,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248416+0000","last_change":"2024-03-28T22:33:28.939016+0000","last_active":"2024-03-28T22:35:30.248416+0000","last_peered":"2024-03-28T22:35:30.248416+0000","last_clean":"2024-03-28T22:35:30.248416+0000","last_became_active":"2024-03-28T22:33:28.938695+0000","last_became_peered":"2024-03-28T22:33:28.938695+0000","last_unstale":"2024-03-28T22:35:30.248416+0000","last_undegraded":"2024-03-28T22:35:30.248416+0000","last_fullsized":"2024-03-28T22:35:30.248416+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:26:53.264559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252656+0000","last_change":"2024-03-28T22:33:34.030127+0000","last_active":"2024-03-28T22:35:30.252656+0000","last_peered":"2024-03-28T22:35:30.252656+0000","last_clean":"2024-03-28T22:35:30.252656+0000","last_became_active":"2024-03-28T22:33:34.028056+0000","last_became_peered":"2024-03-28T22:33:34.028056+0000","last_unstale":"2024-03-28T22:35:30.252656+0000","last_undegraded":"2024-03-28T22:35:30.252656+0000","last_fullsized":"2024-03-28T22:35:30.252656+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:29:41.785013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'7","reported_seq":128,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370846+0000","last_change":"2024-03-28T22:33:28.435350+0000","last_active":"2024-03-28T22:35:30.370846+0000","last_peered":"2024-03-28T22:35:30.370846+0000","last_clean":"2024-03-28T22:35:30.370846+0000","last_became_active":"2024-03-28T22:33:28.434954+0000","last_became_peered":"2024-03-28T22:33:28.434954+0000","last_unstale":"2024-03-28T22:35:30.370846+0000","last_undegraded":"2024-03-28T22:35:30.370846+0000","last_fullsized":"2024-03-28T22:35:30.370846+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:04.969596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":90,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"94'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248078+0000","last_change":"2024-03-28T22:33:28.921219+0000","last_active":"2024-03-28T22:35:30.248078+0000","last_peered":"2024-03-28T22:35:30.248078+0000","last_clean":"2024-03-28T22:35:30.248078+0000","last_became_active":"2024-03-28T22:33:28.920664+0000","last_became_peered":"2024-03-28T22:33:28.920664+0000","last_unstale":"2024-03-28T22:35:30.248078+0000","last_undegraded":"2024-03-28T22:35:30.248078+0000","last_fullsized":"2024-03-28T22:35:30.248078+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:48:55.032058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369304+0000","last_change":"2024-03-28T22:33:34.018101+0000","last_active":"2024-03-28T22:35:30.369304+0000","last_peered":"2024-03-28T22:35:30.369304+0000","last_clean":"2024-03-28T22:35:30.369304+0000","last_became_active":"2024-03-28T22:33:34.017260+0000","last_became_peered":"2024-03-28T22:33:34.017260+0000","last_unstale":"2024-03-28T22:35:30.369304+0000","last_undegraded":"2024-03-28T22:35:30.369304+0000","last_fullsized":"2024-03-28T22:35:30.369304+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:48:53.178090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"92'17","reported_seq":51,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.253538+0000","last_change":"2024-03-28T22:33:28.450232+0000","last_active":"2024-03-28T22:35:30.253538+0000","last_peered":"2024-03-28T22:35:30.253538+0000","last_clean":"2024-03-28T22:35:30.253538+0000","last_became_active":"2024-03-28T22:33:28.449958+0000","last_became_peered":"2024-03-28T22:33:28.449958+0000","last_unstale":"2024-03-28T22:35:30.253538+0000","last_undegraded":"2024-03-28T22:35:30.253538+0000","last_fullsized":"2024-03-28T22:35:30.253538+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:01:42.130260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"94'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275169+0000","last_change":"2024-03-28T22:33:28.926263+0000","last_active":"2024-03-28T22:35:30.275169+0000","last_peered":"2024-03-28T22:35:30.275169+0000","last_clean":"2024-03-28T22:35:30.275169+0000","last_became_active":"2024-03-28T22:33:28.925927+0000","last_became_peered":"2024-03-28T22:33:28.925927+0000","last_unstale":"2024-03-28T22:35:30.275169+0000","last_undegraded":"2024-03-28T22:35:30.275169+0000","last_fullsized":"2024-03-28T22:35:30.275169+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:41:42.477694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":2,"reported_epoch":95,"state":"peering","last_fresh":"2024-03-28T22:33:33.992594+0000","last_change":"2024-03-28T22:35:29.421928+0000","last_active":"2024-03-28T22:33:32.966724+0000","last_peered":"2024-03-28T22:33:32.966724+0000","last_clean":"2024-03-28T22:33:32.966724+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-28T22:33:33.992594+0000","last_undegraded":"2024-03-28T22:33:33.992594+0000","last_fullsized":"2024-03-28T22:33:33.992594+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,3,1],"acting":[10,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.f","version":"92'2","reported_seq":21,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449758+0000","last_change":"2024-03-28T22:34:29.986650+0000","last_active":"2024-03-28T22:35:30.449758+0000","last_peered":"2024-03-28T22:35:30.449758+0000","last_clean":"2024-03-28T22:35:30.449758+0000","last_became_active":"2024-03-28T22:34:29.981875+0000","last_became_peered":"2024-03-28T22:34:29.981875+0000","last_unstale":"2024-03-28T22:35:30.449758+0000","last_undegraded":"2024-03-28T22:35:30.449758+0000","last_fullsized":"2024-03-28T22:35:30.449758+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:45:24.166133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"94'5","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449898+0000","last_change":"2024-03-28T22:34:29.982815+0000","last_active":"2024-03-28T22:35:30.449898+0000","last_peered":"2024-03-28T22:35:30.449898+0000","last_clean":"2024-03-28T22:35:30.449898+0000","last_became_active":"2024-03-28T22:34:29.982458+0000","last_became_peered":"2024-03-28T22:34:29.982458+0000","last_unstale":"2024-03-28T22:35:30.449898+0000","last_undegraded":"2024-03-28T22:35:30.449898+0000","last_fullsized":"2024-03-28T22:35:30.449898+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:11:30.617181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":29,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451063+0000","last_change":"2024-03-28T22:35:30.451063+0000","last_active":"2024-03-28T22:35:30.451063+0000","last_peered":"2024-03-28T22:35:30.451063+0000","last_clean":"2024-03-28T22:35:30.451063+0000","last_became_active":"2024-03-28T22:35:30.450525+0000","last_became_peered":"2024-03-28T22:35:30.450525+0000","last_unstale":"2024-03-28T22:35:30.451063+0000","last_undegraded":"2024-03-28T22:35:30.451063+0000","last_fullsized":"2024-03-28T22:35:30.451063+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:35:29.004834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"92'1","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252046+0000","last_change":"2024-03-28T22:34:29.992884+0000","last_active":"2024-03-28T22:35:30.252046+0000","last_peered":"2024-03-28T22:35:30.252046+0000","last_clean":"2024-03-28T22:35:30.252046+0000","last_became_active":"2024-03-28T22:34:29.991657+0000","last_became_peered":"2024-03-28T22:34:29.991657+0000","last_unstale":"2024-03-28T22:35:30.252046+0000","last_undegraded":"2024-03-28T22:35:30.252046+0000","last_fullsized":"2024-03-28T22:35:30.252046+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:25:14.540783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"94'4","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-28T22:35:29.245012+0000","last_change":"2024-03-28T22:33:28.942097+0000","last_active":"2024-03-28T22:35:29.245012+0000","last_peered":"2024-03-28T22:35:29.245012+0000","last_clean":"2024-03-28T22:35:29.245012+0000","last_became_active":"2024-03-28T22:33:28.941812+0000","last_became_peered":"2024-03-28T22:33:28.941812+0000","last_unstale":"2024-03-28T22:35:29.245012+0000","last_undegraded":"2024-03-28T22:35:29.245012+0000","last_fullsized":"2024-03-28T22:35:29.245012+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:36:10.204216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369114+0000","last_change":"2024-03-28T22:33:34.023171+0000","last_active":"2024-03-28T22:35:30.369114+0000","last_peered":"2024-03-28T22:35:30.369114+0000","last_clean":"2024-03-28T22:35:30.369114+0000","last_became_active":"2024-03-28T22:33:34.021313+0000","last_became_peered":"2024-03-28T22:33:34.021313+0000","last_unstale":"2024-03-28T22:35:30.369114+0000","last_undegraded":"2024-03-28T22:35:30.369114+0000","last_fullsized":"2024-03-28T22:35:30.369114+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:29:42.592761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"92'5","reported_seq":128,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:33.324016+0000","last_change":"2024-03-28T22:33:28.449133+0000","last_active":"2024-03-28T22:35:33.324016+0000","last_peered":"2024-03-28T22:35:33.324016+0000","last_clean":"2024-03-28T22:35:33.324016+0000","last_became_active":"2024-03-28T22:33:28.448743+0000","last_became_peered":"2024-03-28T22:33:28.448743+0000","last_unstale":"2024-03-28T22:35:33.324016+0000","last_undegraded":"2024-03-28T22:35:33.324016+0000","last_fullsized":"2024-03-28T22:35:33.324016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:09:37.119736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":89,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"94'7","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-28T22:35:29.244649+0000","last_change":"2024-03-28T22:33:28.944198+0000","last_active":"2024-03-28T22:35:29.244649+0000","last_peered":"2024-03-28T22:35:29.244649+0000","last_clean":"2024-03-28T22:35:29.244649+0000","last_became_active":"2024-03-28T22:33:28.943818+0000","last_became_peered":"2024-03-28T22:33:28.943818+0000","last_unstale":"2024-03-28T22:35:29.244649+0000","last_undegraded":"2024-03-28T22:35:29.244649+0000","last_fullsized":"2024-03-28T22:35:29.244649+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:31:03.791364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369445+0000","last_change":"2024-03-28T22:33:34.015499+0000","last_active":"2024-03-28T22:35:30.369445+0000","last_peered":"2024-03-28T22:35:30.369445+0000","last_clean":"2024-03-28T22:35:30.369445+0000","last_became_active":"2024-03-28T22:33:34.015013+0000","last_became_peered":"2024-03-28T22:33:34.015013+0000","last_unstale":"2024-03-28T22:35:30.369445+0000","last_undegraded":"2024-03-28T22:35:30.369445+0000","last_fullsized":"2024-03-28T22:35:30.369445+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:34:30.582198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"94'8","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275296+0000","last_change":"2024-03-28T22:33:28.926732+0000","last_active":"2024-03-28T22:35:30.275296+0000","last_peered":"2024-03-28T22:35:30.275296+0000","last_clean":"2024-03-28T22:35:30.275296+0000","last_became_active":"2024-03-28T22:33:28.926366+0000","last_became_peered":"2024-03-28T22:33:28.926366+0000","last_unstale":"2024-03-28T22:35:30.275296+0000","last_undegraded":"2024-03-28T22:35:30.275296+0000","last_fullsized":"2024-03-28T22:35:30.275296+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:19:10.769141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'6","reported_seq":128,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:32.374730+0000","last_change":"2024-03-28T22:33:28.457400+0000","last_active":"2024-03-28T22:35:32.374730+0000","last_peered":"2024-03-28T22:35:32.374730+0000","last_clean":"2024-03-28T22:35:32.374730+0000","last_became_active":"2024-03-28T22:33:28.456994+0000","last_became_peered":"2024-03-28T22:33:28.456994+0000","last_unstale":"2024-03-28T22:35:32.374730+0000","last_undegraded":"2024-03-28T22:35:32.374730+0000","last_fullsized":"2024-03-28T22:35:32.374730+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:32:16.438540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":88,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"92'1","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248340+0000","last_change":"2024-03-28T22:33:34.027746+0000","last_active":"2024-03-28T22:35:30.248340+0000","last_peered":"2024-03-28T22:35:30.248340+0000","last_clean":"2024-03-28T22:35:30.248340+0000","last_became_active":"2024-03-28T22:33:34.027419+0000","last_became_peered":"2024-03-28T22:33:34.027419+0000","last_unstale":"2024-03-28T22:35:30.248340+0000","last_undegraded":"2024-03-28T22:35:30.248340+0000","last_fullsized":"2024-03-28T22:35:30.248340+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:55:43.686428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"94'8","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371473+0000","last_change":"2024-03-28T22:33:28.952393+0000","last_active":"2024-03-28T22:35:30.371473+0000","last_peered":"2024-03-28T22:35:30.371473+0000","last_clean":"2024-03-28T22:35:30.371473+0000","last_became_active":"2024-03-28T22:33:28.951951+0000","last_became_peered":"2024-03-28T22:33:28.951951+0000","last_unstale":"2024-03-28T22:35:30.371473+0000","last_undegraded":"2024-03-28T22:35:30.371473+0000","last_fullsized":"2024-03-28T22:35:30.371473+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:09:25.808842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"92'6","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370574+0000","last_change":"2024-03-28T22:33:28.437177+0000","last_active":"2024-03-28T22:35:30.370574+0000","last_peered":"2024-03-28T22:35:30.370574+0000","last_clean":"2024-03-28T22:35:30.370574+0000","last_became_active":"2024-03-28T22:33:28.436801+0000","last_became_peered":"2024-03-28T22:33:28.436801+0000","last_unstale":"2024-03-28T22:35:30.370574+0000","last_undegraded":"2024-03-28T22:35:30.370574+0000","last_fullsized":"2024-03-28T22:35:30.370574+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:24:52.549490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":371,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.247934+0000","last_change":"2024-03-28T22:32:07.867336+0000","last_active":"2024-03-28T22:35:30.247934+0000","last_peered":"2024-03-28T22:35:30.247934+0000","last_clean":"2024-03-28T22:35:30.247934+0000","last_became_active":"2024-03-28T22:32:07.866819+0000","last_became_peered":"2024-03-28T22:32:07.866819+0000","last_unstale":"2024-03-28T22:35:30.247934+0000","last_undegraded":"2024-03-28T22:35:30.247934+0000","last_fullsized":"2024-03-28T22:35:30.247934+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:28:14.713501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:28:14.713501+0000","last_clean_scrub_stamp":"2024-03-28T22:28:14.713501+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:00:52.438708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275546+0000","last_change":"2024-03-28T22:33:34.020668+0000","last_active":"2024-03-28T22:35:30.275546+0000","last_peered":"2024-03-28T22:35:30.275546+0000","last_clean":"2024-03-28T22:35:30.275546+0000","last_became_active":"2024-03-28T22:33:34.020240+0000","last_became_peered":"2024-03-28T22:33:34.020240+0000","last_unstale":"2024-03-28T22:35:30.275546+0000","last_undegraded":"2024-03-28T22:35:30.275546+0000","last_fullsized":"2024-03-28T22:35:30.275546+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:18.983983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"94'5","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368008+0000","last_change":"2024-03-28T22:33:28.935237+0000","last_active":"2024-03-28T22:35:30.368008+0000","last_peered":"2024-03-28T22:35:30.368008+0000","last_clean":"2024-03-28T22:35:30.368008+0000","last_became_active":"2024-03-28T22:33:28.934875+0000","last_became_peered":"2024-03-28T22:33:28.934875+0000","last_unstale":"2024-03-28T22:35:30.368008+0000","last_undegraded":"2024-03-28T22:35:30.368008+0000","last_fullsized":"2024-03-28T22:35:30.368008+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:20:50.789302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"92'4","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248187+0000","last_change":"2024-03-28T22:33:28.441654+0000","last_active":"2024-03-28T22:35:30.248187+0000","last_peered":"2024-03-28T22:35:30.248187+0000","last_clean":"2024-03-28T22:35:30.248187+0000","last_became_active":"2024-03-28T22:33:28.441293+0000","last_became_peered":"2024-03-28T22:33:28.441293+0000","last_unstale":"2024-03-28T22:35:30.248187+0000","last_undegraded":"2024-03-28T22:35:30.248187+0000","last_fullsized":"2024-03-28T22:35:30.248187+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:31:17.737977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449436+0000","last_change":"2024-03-28T22:33:34.018126+0000","last_active":"2024-03-28T22:35:30.449436+0000","last_peered":"2024-03-28T22:35:30.449436+0000","last_clean":"2024-03-28T22:35:30.449436+0000","last_became_active":"2024-03-28T22:33:34.017573+0000","last_became_peered":"2024-03-28T22:33:34.017573+0000","last_unstale":"2024-03-28T22:35:30.449436+0000","last_undegraded":"2024-03-28T22:35:30.449436+0000","last_fullsized":"2024-03-28T22:35:30.449436+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:23:31.013511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"92'4","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254108+0000","last_change":"2024-03-28T22:33:28.454069+0000","last_active":"2024-03-28T22:35:30.254108+0000","last_peered":"2024-03-28T22:35:30.254108+0000","last_clean":"2024-03-28T22:35:30.254108+0000","last_became_active":"2024-03-28T22:33:28.453754+0000","last_became_peered":"2024-03-28T22:33:28.453754+0000","last_unstale":"2024-03-28T22:35:30.254108+0000","last_undegraded":"2024-03-28T22:35:30.254108+0000","last_fullsized":"2024-03-28T22:35:30.254108+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:23:33.981072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"94'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369303+0000","last_change":"2024-03-28T22:33:28.934390+0000","last_active":"2024-03-28T22:35:30.369303+0000","last_peered":"2024-03-28T22:35:30.369303+0000","last_clean":"2024-03-28T22:35:30.369303+0000","last_became_active":"2024-03-28T22:33:28.933919+0000","last_became_peered":"2024-03-28T22:33:28.933919+0000","last_unstale":"2024-03-28T22:35:30.369303+0000","last_undegraded":"2024-03-28T22:35:30.369303+0000","last_fullsized":"2024-03-28T22:35:30.369303+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:27:07.595144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371256+0000","last_change":"2024-03-28T22:33:34.019910+0000","last_active":"2024-03-28T22:35:30.371256+0000","last_peered":"2024-03-28T22:35:30.371256+0000","last_clean":"2024-03-28T22:35:30.371256+0000","last_became_active":"2024-03-28T22:33:34.019522+0000","last_became_peered":"2024-03-28T22:33:34.019522+0000","last_unstale":"2024-03-28T22:35:30.371256+0000","last_undegraded":"2024-03-28T22:35:30.371256+0000","last_fullsized":"2024-03-28T22:35:30.371256+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:01:29.298513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'14","reported_seq":47,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252650+0000","last_change":"2024-03-28T22:33:28.444513+0000","last_active":"2024-03-28T22:35:30.252650+0000","last_peered":"2024-03-28T22:35:30.252650+0000","last_clean":"2024-03-28T22:35:30.252650+0000","last_became_active":"2024-03-28T22:33:28.444091+0000","last_became_peered":"2024-03-28T22:33:28.444091+0000","last_unstale":"2024-03-28T22:35:30.252650+0000","last_undegraded":"2024-03-28T22:35:30.252650+0000","last_fullsized":"2024-03-28T22:35:30.252650+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:49:58.331864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"94'5","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.372101+0000","last_change":"2024-03-28T22:33:28.948390+0000","last_active":"2024-03-28T22:35:30.372101+0000","last_peered":"2024-03-28T22:35:30.372101+0000","last_clean":"2024-03-28T22:35:30.372101+0000","last_became_active":"2024-03-28T22:33:28.942846+0000","last_became_peered":"2024-03-28T22:33:28.942846+0000","last_unstale":"2024-03-28T22:35:30.372101+0000","last_undegraded":"2024-03-28T22:35:30.372101+0000","last_fullsized":"2024-03-28T22:35:30.372101+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:50.596365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251650+0000","last_change":"2024-03-28T22:33:33.999429+0000","last_active":"2024-03-28T22:35:30.251650+0000","last_peered":"2024-03-28T22:35:30.251650+0000","last_clean":"2024-03-28T22:35:30.251650+0000","last_became_active":"2024-03-28T22:33:33.998979+0000","last_became_peered":"2024-03-28T22:33:33.998979+0000","last_unstale":"2024-03-28T22:35:30.251650+0000","last_undegraded":"2024-03-28T22:35:30.251650+0000","last_fullsized":"2024-03-28T22:35:30.251650+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:26.450731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"92'7","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370722+0000","last_change":"2024-03-28T22:33:28.442861+0000","last_active":"2024-03-28T22:35:30.370722+0000","last_peered":"2024-03-28T22:35:30.370722+0000","last_clean":"2024-03-28T22:35:30.370722+0000","last_became_active":"2024-03-28T22:33:28.442468+0000","last_became_peered":"2024-03-28T22:33:28.442468+0000","last_unstale":"2024-03-28T22:35:30.370722+0000","last_undegraded":"2024-03-28T22:35:30.370722+0000","last_fullsized":"2024-03-28T22:35:30.370722+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:45:49.899031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"94'7","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451836+0000","last_change":"2024-03-28T22:34:30.429872+0000","last_active":"2024-03-28T22:35:30.451836+0000","last_peered":"2024-03-28T22:35:30.451836+0000","last_clean":"2024-03-28T22:35:30.451836+0000","last_became_active":"2024-03-28T22:34:30.429237+0000","last_became_peered":"2024-03-28T22:34:30.429237+0000","last_unstale":"2024-03-28T22:35:30.451836+0000","last_undegraded":"2024-03-28T22:35:30.451836+0000","last_fullsized":"2024-03-28T22:35:30.451836+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:14:46.608826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370518+0000","last_change":"2024-03-28T22:33:34.013826+0000","last_active":"2024-03-28T22:35:30.370518+0000","last_peered":"2024-03-28T22:35:30.370518+0000","last_clean":"2024-03-28T22:35:30.370518+0000","last_became_active":"2024-03-28T22:33:34.013415+0000","last_became_peered":"2024-03-28T22:33:34.013415+0000","last_unstale":"2024-03-28T22:35:30.370518+0000","last_undegraded":"2024-03-28T22:35:30.370518+0000","last_fullsized":"2024-03-28T22:35:30.370518+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:19:30.965940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"92'2","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-28T22:35:29.246592+0000","last_change":"2024-03-28T22:33:28.442225+0000","last_active":"2024-03-28T22:35:29.246592+0000","last_peered":"2024-03-28T22:35:29.246592+0000","last_clean":"2024-03-28T22:35:29.246592+0000","last_became_active":"2024-03-28T22:33:28.441844+0000","last_became_peered":"2024-03-28T22:33:28.441844+0000","last_unstale":"2024-03-28T22:35:29.246592+0000","last_undegraded":"2024-03-28T22:35:29.246592+0000","last_fullsized":"2024-03-28T22:35:29.246592+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:40:52.108663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"94'2","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.253620+0000","last_change":"2024-03-28T22:33:28.927571+0000","last_active":"2024-03-28T22:35:30.253620+0000","last_peered":"2024-03-28T22:35:30.253620+0000","last_clean":"2024-03-28T22:35:30.253620+0000","last_became_active":"2024-03-28T22:33:28.927275+0000","last_became_peered":"2024-03-28T22:33:28.927275+0000","last_unstale":"2024-03-28T22:35:30.253620+0000","last_undegraded":"2024-03-28T22:35:30.253620+0000","last_fullsized":"2024-03-28T22:35:30.253620+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:25:16.184533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252877+0000","last_change":"2024-03-28T22:33:34.018429+0000","last_active":"2024-03-28T22:35:30.252877+0000","last_peered":"2024-03-28T22:35:30.252877+0000","last_clean":"2024-03-28T22:35:30.252877+0000","last_became_active":"2024-03-28T22:33:34.017066+0000","last_became_peered":"2024-03-28T22:33:34.017066+0000","last_unstale":"2024-03-28T22:35:30.252877+0000","last_undegraded":"2024-03-28T22:35:30.252877+0000","last_fullsized":"2024-03-28T22:35:30.252877+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:35:42.101072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"92'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371904+0000","last_change":"2024-03-28T22:33:28.437760+0000","last_active":"2024-03-28T22:35:30.371904+0000","last_peered":"2024-03-28T22:35:30.371904+0000","last_clean":"2024-03-28T22:35:30.371904+0000","last_became_active":"2024-03-28T22:33:28.437096+0000","last_became_peered":"2024-03-28T22:33:28.437096+0000","last_unstale":"2024-03-28T22:35:30.371904+0000","last_undegraded":"2024-03-28T22:35:30.371904+0000","last_fullsized":"2024-03-28T22:35:30.371904+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:29:20.778551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"94'9","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449293+0000","last_change":"2024-03-28T22:33:28.935805+0000","last_active":"2024-03-28T22:35:30.449293+0000","last_peered":"2024-03-28T22:35:30.449293+0000","last_clean":"2024-03-28T22:35:30.449293+0000","last_became_active":"2024-03-28T22:33:28.935520+0000","last_became_peered":"2024-03-28T22:33:28.935520+0000","last_unstale":"2024-03-28T22:35:30.449293+0000","last_undegraded":"2024-03-28T22:35:30.449293+0000","last_fullsized":"2024-03-28T22:35:30.449293+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:23.414310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"92'1","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370530+0000","last_change":"2024-03-28T22:33:34.029518+0000","last_active":"2024-03-28T22:35:30.370530+0000","last_peered":"2024-03-28T22:35:30.370530+0000","last_clean":"2024-03-28T22:35:30.370530+0000","last_became_active":"2024-03-28T22:33:34.029101+0000","last_became_peered":"2024-03-28T22:33:34.029101+0000","last_unstale":"2024-03-28T22:35:30.370530+0000","last_undegraded":"2024-03-28T22:35:30.370530+0000","last_fullsized":"2024-03-28T22:35:30.370530+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:05.595543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"92'3","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251878+0000","last_change":"2024-03-28T22:33:28.438857+0000","last_active":"2024-03-28T22:35:30.251878+0000","last_peered":"2024-03-28T22:35:30.251878+0000","last_clean":"2024-03-28T22:35:30.251878+0000","last_became_active":"2024-03-28T22:33:28.438528+0000","last_became_peered":"2024-03-28T22:33:28.438528+0000","last_unstale":"2024-03-28T22:35:30.251878+0000","last_undegraded":"2024-03-28T22:35:30.251878+0000","last_fullsized":"2024-03-28T22:35:30.251878+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:07:54.506739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"94'8","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371602+0000","last_change":"2024-03-28T22:33:28.958271+0000","last_active":"2024-03-28T22:35:30.371602+0000","last_peered":"2024-03-28T22:35:30.371602+0000","last_clean":"2024-03-28T22:35:30.371602+0000","last_became_active":"2024-03-28T22:33:28.957922+0000","last_became_peered":"2024-03-28T22:33:28.957922+0000","last_unstale":"2024-03-28T22:35:30.371602+0000","last_undegraded":"2024-03-28T22:35:30.371602+0000","last_fullsized":"2024-03-28T22:35:30.371602+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:33:19.003131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370511+0000","last_change":"2024-03-28T22:33:34.000789+0000","last_active":"2024-03-28T22:35:30.370511+0000","last_peered":"2024-03-28T22:35:30.370511+0000","last_clean":"2024-03-28T22:35:30.370511+0000","last_became_active":"2024-03-28T22:33:34.000383+0000","last_became_peered":"2024-03-28T22:33:34.000383+0000","last_unstale":"2024-03-28T22:35:30.370511+0000","last_undegraded":"2024-03-28T22:35:30.370511+0000","last_fullsized":"2024-03-28T22:35:30.370511+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:50:29.925656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"92'4","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.253761+0000","last_change":"2024-03-28T22:33:28.439121+0000","last_active":"2024-03-28T22:35:30.253761+0000","last_peered":"2024-03-28T22:35:30.253761+0000","last_clean":"2024-03-28T22:35:30.253761+0000","last_became_active":"2024-03-28T22:33:28.438800+0000","last_became_peered":"2024-03-28T22:33:28.438800+0000","last_unstale":"2024-03-28T22:35:30.253761+0000","last_undegraded":"2024-03-28T22:35:30.253761+0000","last_fullsized":"2024-03-28T22:35:30.253761+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:57:10.013246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"94'5","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248101+0000","last_change":"2024-03-28T22:33:28.934499+0000","last_active":"2024-03-28T22:35:30.248101+0000","last_peered":"2024-03-28T22:35:30.248101+0000","last_clean":"2024-03-28T22:35:30.248101+0000","last_became_active":"2024-03-28T22:33:28.934147+0000","last_became_peered":"2024-03-28T22:33:28.934147+0000","last_unstale":"2024-03-28T22:35:30.248101+0000","last_undegraded":"2024-03-28T22:35:30.248101+0000","last_fullsized":"2024-03-28T22:35:30.248101+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:35:39.090052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":20,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-28T22:35:29.246503+0000","last_change":"2024-03-28T22:33:34.017946+0000","last_active":"2024-03-28T22:35:29.246503+0000","last_peered":"2024-03-28T22:35:29.246503+0000","last_clean":"2024-03-28T22:35:29.246503+0000","last_became_active":"2024-03-28T22:33:34.017546+0000","last_became_peered":"2024-03-28T22:33:34.017546+0000","last_unstale":"2024-03-28T22:35:29.246503+0000","last_undegraded":"2024-03-28T22:35:29.246503+0000","last_fullsized":"2024-03-28T22:35:29.246503+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:14:39.081677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"92'5","reported_seq":128,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:32.275107+0000","last_change":"2024-03-28T22:33:28.453387+0000","last_active":"2024-03-28T22:35:32.275107+0000","last_peered":"2024-03-28T22:35:32.275107+0000","last_clean":"2024-03-28T22:35:32.275107+0000","last_became_active":"2024-03-28T22:33:28.453029+0000","last_became_peered":"2024-03-28T22:33:28.453029+0000","last_unstale":"2024-03-28T22:35:32.275107+0000","last_undegraded":"2024-03-28T22:35:32.275107+0000","last_fullsized":"2024-03-28T22:35:32.275107+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:12:33.246863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":90,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"94'7","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248569+0000","last_change":"2024-03-28T22:33:28.923819+0000","last_active":"2024-03-28T22:35:30.248569+0000","last_peered":"2024-03-28T22:35:30.248569+0000","last_clean":"2024-03-28T22:35:30.248569+0000","last_became_active":"2024-03-28T22:33:28.923471+0000","last_became_peered":"2024-03-28T22:33:28.923471+0000","last_unstale":"2024-03-28T22:35:30.248569+0000","last_undegraded":"2024-03-28T22:35:30.248569+0000","last_fullsized":"2024-03-28T22:35:30.248569+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:58:03.496406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.450859+0000","last_change":"2024-03-28T22:33:34.029789+0000","last_active":"2024-03-28T22:35:30.450859+0000","last_peered":"2024-03-28T22:35:30.450859+0000","last_clean":"2024-03-28T22:35:30.450859+0000","last_became_active":"2024-03-28T22:33:34.029231+0000","last_became_peered":"2024-03-28T22:33:34.029231+0000","last_unstale":"2024-03-28T22:35:30.450859+0000","last_undegraded":"2024-03-28T22:35:30.450859+0000","last_fullsized":"2024-03-28T22:35:30.450859+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:47:50.101393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"92'3","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275488+0000","last_change":"2024-03-28T22:33:28.435668+0000","last_active":"2024-03-28T22:35:30.275488+0000","last_peered":"2024-03-28T22:35:30.275488+0000","last_clean":"2024-03-28T22:35:30.275488+0000","last_became_active":"2024-03-28T22:33:28.435325+0000","last_became_peered":"2024-03-28T22:33:28.435325+0000","last_unstale":"2024-03-28T22:35:30.275488+0000","last_undegraded":"2024-03-28T22:35:30.275488+0000","last_fullsized":"2024-03-28T22:35:30.275488+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:13:06.281074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"94'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248195+0000","last_change":"2024-03-28T22:33:28.954138+0000","last_active":"2024-03-28T22:35:30.248195+0000","last_peered":"2024-03-28T22:35:30.248195+0000","last_clean":"2024-03-28T22:35:30.248195+0000","last_became_active":"2024-03-28T22:33:28.953752+0000","last_became_peered":"2024-03-28T22:33:28.953752+0000","last_unstale":"2024-03-28T22:35:30.248195+0000","last_undegraded":"2024-03-28T22:35:30.248195+0000","last_fullsized":"2024-03-28T22:35:30.248195+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:53:34.892316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.274936+0000","last_change":"2024-03-28T22:33:34.002672+0000","last_active":"2024-03-28T22:35:30.274936+0000","last_peered":"2024-03-28T22:35:30.274936+0000","last_clean":"2024-03-28T22:35:30.274936+0000","last_became_active":"2024-03-28T22:33:34.002195+0000","last_became_peered":"2024-03-28T22:33:34.002195+0000","last_unstale":"2024-03-28T22:35:30.274936+0000","last_undegraded":"2024-03-28T22:35:30.274936+0000","last_fullsized":"2024-03-28T22:35:30.274936+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:24:11.521474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"92'5","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369164+0000","last_change":"2024-03-28T22:33:28.450818+0000","last_active":"2024-03-28T22:35:30.369164+0000","last_peered":"2024-03-28T22:35:30.369164+0000","last_clean":"2024-03-28T22:35:30.369164+0000","last_became_active":"2024-03-28T22:33:28.448592+0000","last_became_peered":"2024-03-28T22:33:28.448592+0000","last_unstale":"2024-03-28T22:35:30.369164+0000","last_undegraded":"2024-03-28T22:35:30.369164+0000","last_fullsized":"2024-03-28T22:35:30.369164+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:19:27.126306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"94'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248443+0000","last_change":"2024-03-28T22:33:28.925191+0000","last_active":"2024-03-28T22:35:30.248443+0000","last_peered":"2024-03-28T22:35:30.248443+0000","last_clean":"2024-03-28T22:35:30.248443+0000","last_became_active":"2024-03-28T22:33:28.924424+0000","last_became_peered":"2024-03-28T22:33:28.924424+0000","last_unstale":"2024-03-28T22:35:30.248443+0000","last_undegraded":"2024-03-28T22:35:30.248443+0000","last_fullsized":"2024-03-28T22:35:30.248443+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:50:00.415047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"94'9","reported_seq":45,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449369+0000","last_change":"2024-03-28T22:34:29.987176+0000","last_active":"2024-03-28T22:35:30.449369+0000","last_peered":"2024-03-28T22:35:30.449369+0000","last_clean":"2024-03-28T22:35:30.449369+0000","last_became_active":"2024-03-28T22:34:29.986807+0000","last_became_peered":"2024-03-28T22:34:29.986807+0000","last_unstale":"2024-03-28T22:35:30.449369+0000","last_undegraded":"2024-03-28T22:35:30.449369+0000","last_fullsized":"2024-03-28T22:35:30.449369+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:00:06.774521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"94'12","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251938+0000","last_change":"2024-03-28T22:33:28.927068+0000","last_active":"2024-03-28T22:35:30.251938+0000","last_peered":"2024-03-28T22:35:30.251938+0000","last_clean":"2024-03-28T22:35:30.251938+0000","last_became_active":"2024-03-28T22:33:28.926734+0000","last_became_peered":"2024-03-28T22:33:28.926734+0000","last_unstale":"2024-03-28T22:35:30.251938+0000","last_undegraded":"2024-03-28T22:35:30.251938+0000","last_fullsized":"2024-03-28T22:35:30.251938+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:56.309555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"94'8","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251930+0000","last_change":"2024-03-28T22:33:28.955675+0000","last_active":"2024-03-28T22:35:30.251930+0000","last_peered":"2024-03-28T22:35:30.251930+0000","last_clean":"2024-03-28T22:35:30.251930+0000","last_became_active":"2024-03-28T22:33:28.955329+0000","last_became_peered":"2024-03-28T22:33:28.955329+0000","last_unstale":"2024-03-28T22:35:30.251930+0000","last_undegraded":"2024-03-28T22:35:30.251930+0000","last_fullsized":"2024-03-28T22:35:30.251930+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:34:14.166434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.452477+0000","last_change":"2024-03-28T22:33:28.955125+0000","last_active":"2024-03-28T22:35:30.452477+0000","last_peered":"2024-03-28T22:35:30.452477+0000","last_clean":"2024-03-28T22:35:30.452477+0000","last_became_active":"2024-03-28T22:33:28.954434+0000","last_became_peered":"2024-03-28T22:33:28.954434+0000","last_unstale":"2024-03-28T22:35:30.452477+0000","last_undegraded":"2024-03-28T22:35:30.452477+0000","last_fullsized":"2024-03-28T22:35:30.452477+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:56:50.705012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"94'4","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369377+0000","last_change":"2024-03-28T22:33:28.945230+0000","last_active":"2024-03-28T22:35:30.369377+0000","last_peered":"2024-03-28T22:35:30.369377+0000","last_clean":"2024-03-28T22:35:30.369377+0000","last_became_active":"2024-03-28T22:33:28.944921+0000","last_became_peered":"2024-03-28T22:33:28.944921+0000","last_unstale":"2024-03-28T22:35:30.369377+0000","last_undegraded":"2024-03-28T22:35:30.369377+0000","last_fullsized":"2024-03-28T22:35:30.369377+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:26:26.505774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"94'6","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449689+0000","last_change":"2024-03-28T22:33:28.928720+0000","last_active":"2024-03-28T22:35:30.449689+0000","last_peered":"2024-03-28T22:35:30.449689+0000","last_clean":"2024-03-28T22:35:30.449689+0000","last_became_active":"2024-03-28T22:33:28.928415+0000","last_became_peered":"2024-03-28T22:33:28.928415+0000","last_unstale":"2024-03-28T22:35:30.449689+0000","last_undegraded":"2024-03-28T22:35:30.449689+0000","last_fullsized":"2024-03-28T22:35:30.449689+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:00.839609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"94'15","reported_seq":46,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252371+0000","last_change":"2024-03-28T22:33:28.924948+0000","last_active":"2024-03-28T22:35:30.252371+0000","last_peered":"2024-03-28T22:35:30.252371+0000","last_clean":"2024-03-28T22:35:30.252371+0000","last_became_active":"2024-03-28T22:33:28.924631+0000","last_became_peered":"2024-03-28T22:33:28.924631+0000","last_unstale":"2024-03-28T22:35:30.252371+0000","last_undegraded":"2024-03-28T22:35:30.252371+0000","last_fullsized":"2024-03-28T22:35:30.252371+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:53:44.909991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"94'2","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.372370+0000","last_change":"2024-03-28T22:33:28.948725+0000","last_active":"2024-03-28T22:35:30.372370+0000","last_peered":"2024-03-28T22:35:30.372370+0000","last_clean":"2024-03-28T22:35:30.372370+0000","last_became_active":"2024-03-28T22:33:28.943942+0000","last_became_peered":"2024-03-28T22:33:28.943942+0000","last_unstale":"2024-03-28T22:35:30.372370+0000","last_undegraded":"2024-03-28T22:35:30.372370+0000","last_fullsized":"2024-03-28T22:35:30.372370+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:50:04.238538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"94'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275105+0000","last_change":"2024-03-28T22:33:28.941546+0000","last_active":"2024-03-28T22:35:30.275105+0000","last_peered":"2024-03-28T22:35:30.275105+0000","last_clean":"2024-03-28T22:35:30.275105+0000","last_became_active":"2024-03-28T22:33:28.941278+0000","last_became_peered":"2024-03-28T22:33:28.941278+0000","last_unstale":"2024-03-28T22:35:30.275105+0000","last_undegraded":"2024-03-28T22:35:30.275105+0000","last_fullsized":"2024-03-28T22:35:30.275105+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:47:10.047067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"94'8","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.372207+0000","last_change":"2024-03-28T22:33:28.949552+0000","last_active":"2024-03-28T22:35:30.372207+0000","last_peered":"2024-03-28T22:35:30.372207+0000","last_clean":"2024-03-28T22:35:30.372207+0000","last_became_active":"2024-03-28T22:33:28.947432+0000","last_became_peered":"2024-03-28T22:33:28.947432+0000","last_unstale":"2024-03-28T22:35:30.372207+0000","last_undegraded":"2024-03-28T22:35:30.372207+0000","last_fullsized":"2024-03-28T22:35:30.372207+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:01:16.534642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"94'10","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-28T22:35:29.246779+0000","last_change":"2024-03-28T22:33:28.953930+0000","last_active":"2024-03-28T22:35:29.246779+0000","last_peered":"2024-03-28T22:35:29.246779+0000","last_clean":"2024-03-28T22:35:29.246779+0000","last_became_active":"2024-03-28T22:33:28.953591+0000","last_became_peered":"2024-03-28T22:33:28.953591+0000","last_unstale":"2024-03-28T22:35:29.246779+0000","last_undegraded":"2024-03-28T22:35:29.246779+0000","last_fullsized":"2024-03-28T22:35:29.246779+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:42:55.857476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"94'11","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368740+0000","last_change":"2024-03-28T22:33:28.946307+0000","last_active":"2024-03-28T22:35:30.368740+0000","last_peered":"2024-03-28T22:35:30.368740+0000","last_clean":"2024-03-28T22:35:30.368740+0000","last_became_active":"2024-03-28T22:33:28.945932+0000","last_became_peered":"2024-03-28T22:33:28.945932+0000","last_unstale":"2024-03-28T22:35:30.368740+0000","last_undegraded":"2024-03-28T22:35:30.368740+0000","last_fullsized":"2024-03-28T22:35:30.368740+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:54:59.651668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"94'8","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449625+0000","last_change":"2024-03-28T22:33:28.928014+0000","last_active":"2024-03-28T22:35:30.449625+0000","last_peered":"2024-03-28T22:35:30.449625+0000","last_clean":"2024-03-28T22:35:30.449625+0000","last_became_active":"2024-03-28T22:33:28.927686+0000","last_became_peered":"2024-03-28T22:33:28.927686+0000","last_unstale":"2024-03-28T22:35:30.449625+0000","last_undegraded":"2024-03-28T22:35:30.449625+0000","last_fullsized":"2024-03-28T22:35:30.449625+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:47:08.131565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71409,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":894,"num_read_kb":522,"num_write":250,"num_write_kb":146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":214227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":217,"ondisk_log_size":217,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318311,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32260,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712892,"statfs":{"total":95995035648,"available":95962001408,"internally_reserved":0,"allocated":1105920,"data_stored":539556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579947,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28740,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716412,"statfs":{"total":95995035648,"available":95965605888,"internally_reserved":0,"allocated":1765376,"data_stored":1172563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743474,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1130496,"data_stored":543560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005112,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1118208,"data_stored":539820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299454,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37024,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708128,"statfs":{"total":95995035648,"available":95957123072,"internally_reserved":0,"allocated":1720320,"data_stored":1136480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495683,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1101824,"data_stored":534968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36448,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708704,"statfs":{"total":95995035648,"available":95957712896,"internally_reserved":0,"allocated":1138688,"data_stored":581679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738468,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1126400,"data_stored":536424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411420,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36508,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708644,"statfs":{"total":95995035648,"available":95957651456,"internally_reserved":0,"allocated":1191936,"data_stored":600673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36432,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708720,"statfs":{"total":95995035648,"available":95957729280,"internally_reserved":0,"allocated":1114112,"data_stored":534676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920846,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1110016,"data_stored":539574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28712,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716440,"statfs":{"total":95995035648,"available":95965634560,"internally_reserved":0,"allocated":1728512,"data_stored":1131530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":66207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":66427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":66723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T22:35:35.936 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-28T22:35:36.454 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:35:36.606 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:36 smithi043 ceph-mon[31823]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-28T22:35:36.607 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:36 smithi043 ceph-mon[31823]: Cluster is now healthy 2024-03-28T22:35:36.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:36 smithi172 ceph-mon[36566]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-28T22:35:36.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:36 smithi172 ceph-mon[36566]: Cluster is now healthy 2024-03-28T22:35:36.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:36 smithi082 ceph-mon[36471]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-28T22:35:36.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:36 smithi082 ceph-mon[36471]: Cluster is now healthy 2024-03-28T22:35:37.634 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:37 smithi043 ceph-mon[31823]: pgmap v390: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 639 B/s wr, 11 op/s 2024-03-28T22:35:37.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:37 smithi172 ceph-mon[36566]: pgmap v390: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 639 B/s wr, 11 op/s 2024-03-28T22:35:37.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:37 smithi082 ceph-mon[36471]: pgmap v390: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 639 B/s wr, 11 op/s 2024-03-28T22:35:38.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.741+0000 7fa883000700 1 -- 172.21.15.43:0/2363440065 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa87c108310 msgr2=0x7fa87c1086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:38.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.741+0000 7fa883000700 1 --2- 172.21.15.43:0/2363440065 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa87c108310 0x7fa87c1086f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa86c0099c0 tx=0x7fa86c009cd0 comp rx=0 tx=0).stop 2024-03-28T22:35:38.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.741+0000 7fa883000700 1 -- 172.21.15.43:0/2363440065 shutdown_connections 2024-03-28T22:35:38.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.741+0000 7fa883000700 1 --2- 172.21.15.43:0/2363440065 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa87c102c80 0x7fa87c10f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:38.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.741+0000 7fa883000700 1 --2- 172.21.15.43:0/2363440065 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa87c108310 0x7fa87c1086f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:38.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.741+0000 7fa883000700 1 --2- 172.21.15.43:0/2363440065 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa87c1022c0 0x7fa87c102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:38.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.741+0000 7fa883000700 1 -- 172.21.15.43:0/2363440065 >> 172.21.15.43:0/2363440065 conn(0x7fa87c079390 msgr2=0x7fa87c0797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:38.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.741+0000 7fa883000700 1 -- 172.21.15.43:0/2363440065 shutdown_connections 2024-03-28T22:35:38.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.742+0000 7fa883000700 1 -- 172.21.15.43:0/2363440065 wait complete. 2024-03-28T22:35:38.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.743+0000 7fa883000700 1 Processor -- start 2024-03-28T22:35:38.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.743+0000 7fa883000700 1 -- start start 2024-03-28T22:35:38.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.743+0000 7fa883000700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa87c1022c0 0x7fa87c074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:38.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.743+0000 7fa883000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa87c102c80 0x7fa87c074a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:38.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.743+0000 7fa883000700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa87c108310 0x7fa87c074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:38.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.743+0000 7fa883000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa87c112580 con 0x7fa87c102c80 2024-03-28T22:35:38.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.743+0000 7fa883000700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa87c112400 con 0x7fa87c108310 2024-03-28T22:35:38.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.743+0000 7fa883000700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fa87c112700 con 0x7fa87c1022c0 2024-03-28T22:35:38.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.744+0000 7fa87be00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa87c1022c0 0x7fa87c074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:38.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.744+0000 7fa87be00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa87c1022c0 0x7fa87c074540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:33730/0 (socket says 172.21.15.43:33730) 2024-03-28T22:35:38.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.744+0000 7fa87be00700 1 -- 172.21.15.43:0/530261069 learned_addr learned my addr 172.21.15.43:0/530261069 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:35:38.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.744+0000 7fa881000700 1 --2- 172.21.15.43:0/530261069 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa87c108310 0x7fa87c074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:38.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.744+0000 7fa873400700 1 --2- 172.21.15.43:0/530261069 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa87c102c80 0x7fa87c074a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:38.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.744+0000 7fa87be00700 1 -- 172.21.15.43:0/530261069 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa87c108310 msgr2=0x7fa87c074fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:38.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.744+0000 7fa87be00700 1 --2- 172.21.15.43:0/530261069 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa87c108310 0x7fa87c074fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:38.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.744+0000 7fa87be00700 1 -- 172.21.15.43:0/530261069 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa87c102c80 msgr2=0x7fa87c074a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:38.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.744+0000 7fa87be00700 1 --2- 172.21.15.43:0/530261069 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa87c102c80 0x7fa87c074a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:38.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.744+0000 7fa87be00700 1 -- 172.21.15.43:0/530261069 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa87c1a6fc0 con 0x7fa87c1022c0 2024-03-28T22:35:38.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.744+0000 7fa881000700 1 --2- 172.21.15.43:0/530261069 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa87c108310 0x7fa87c074fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:35:38.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.744+0000 7fa873400700 1 --2- 172.21.15.43:0/530261069 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa87c102c80 0x7fa87c074a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:35:38.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.744+0000 7fa87be00700 1 --2- 172.21.15.43:0/530261069 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa87c1022c0 0x7fa87c074540 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa86c006010 tx=0x7fa86c003e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:38.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.745+0000 7fa879600700 1 -- 172.21.15.43:0/530261069 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa86c009e60 con 0x7fa87c1022c0 2024-03-28T22:35:38.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.745+0000 7fa879600700 1 -- 172.21.15.43:0/530261069 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa86c003680 con 0x7fa87c1022c0 2024-03-28T22:35:38.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.745+0000 7fa883000700 1 -- 172.21.15.43:0/530261069 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa87c1a7250 con 0x7fa87c1022c0 2024-03-28T22:35:38.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.745+0000 7fa879600700 1 -- 172.21.15.43:0/530261069 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa86c0173f0 con 0x7fa87c1022c0 2024-03-28T22:35:38.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.745+0000 7fa883000700 1 -- 172.21.15.43:0/530261069 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa87c1a7700 con 0x7fa87c1022c0 2024-03-28T22:35:38.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.746+0000 7fa883000700 1 -- 172.21.15.43:0/530261069 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa87c10a7d0 con 0x7fa87c1022c0 2024-03-28T22:35:38.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.748+0000 7fa879600700 1 -- 172.21.15.43:0/530261069 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fa86c00f680 con 0x7fa87c1022c0 2024-03-28T22:35:38.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.749+0000 7fa879600700 1 --2- 172.21.15.43:0/530261069 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa8600a1e90 0x7fa8600a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:38.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.750+0000 7fa879600700 1 -- 172.21.15.43:0/530261069 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7fa86c0cef30 con 0x7fa87c1022c0 2024-03-28T22:35:38.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.750+0000 7fa873400700 1 --2- 172.21.15.43:0/530261069 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa8600a1e90 0x7fa8600a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:38.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.751+0000 7fa873400700 1 --2- 172.21.15.43:0/530261069 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa8600a1e90 0x7fa8600a4350 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa87c079a70 tx=0x7fa864009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:38.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.754+0000 7fa879600700 1 -- 172.21.15.43:0/530261069 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa86c013070 con 0x7fa87c1022c0 2024-03-28T22:35:38.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:38.975+0000 7fa883000700 1 -- 172.21.15.43:0/530261069 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa87c075a70 con 0x7fa8600a1e90 2024-03-28T22:35:39.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:39.006+0000 7fa879600700 1 -- 172.21.15.43:0/530261069 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340538 (secure 0 0 0) 0x7fa87c075a70 con 0x7fa8600a1e90 2024-03-28T22:35:39.013 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:35:39.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:39.014+0000 7fa883000700 1 -- 172.21.15.43:0/530261069 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa8600a1e90 msgr2=0x7fa8600a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:39.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:39.014+0000 7fa883000700 1 --2- 172.21.15.43:0/530261069 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa8600a1e90 0x7fa8600a4350 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa87c079a70 tx=0x7fa864009260 comp rx=0 tx=0).stop 2024-03-28T22:35:39.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:39.014+0000 7fa883000700 1 -- 172.21.15.43:0/530261069 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa87c1022c0 msgr2=0x7fa87c074540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:39.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:39.014+0000 7fa883000700 1 --2- 172.21.15.43:0/530261069 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa87c1022c0 0x7fa87c074540 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa86c006010 tx=0x7fa86c003e00 comp rx=0 tx=0).stop 2024-03-28T22:35:39.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:39.015+0000 7fa883000700 1 -- 172.21.15.43:0/530261069 shutdown_connections 2024-03-28T22:35:39.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:39.015+0000 7fa883000700 1 --2- 172.21.15.43:0/530261069 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa8600a1e90 0x7fa8600a4350 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:39.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:39.015+0000 7fa883000700 1 --2- 172.21.15.43:0/530261069 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa87c108310 0x7fa87c074fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:39.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:39.015+0000 7fa883000700 1 --2- 172.21.15.43:0/530261069 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa87c1022c0 0x7fa87c074540 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:39.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:39.015+0000 7fa883000700 1 --2- 172.21.15.43:0/530261069 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa87c102c80 0x7fa87c074a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:39.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:39.015+0000 7fa883000700 1 -- 172.21.15.43:0/530261069 >> 172.21.15.43:0/530261069 conn(0x7fa87c079390 msgr2=0x7fa87c101080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:39.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:39.015+0000 7fa883000700 1 -- 172.21.15.43:0/530261069 shutdown_connections 2024-03-28T22:35:39.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:39.015+0000 7fa883000700 1 -- 172.21.15.43:0/530261069 wait complete. 2024-03-28T22:35:39.018 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-03-28T22:35:39.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:39 smithi172 ceph-mon[36566]: pgmap v391: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 570 B/s wr, 11 op/s 2024-03-28T22:35:39.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:39 smithi172 ceph-mon[36566]: from='client.25069 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:35:39.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:39 smithi082 ceph-mon[36471]: pgmap v391: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 570 B/s wr, 11 op/s 2024-03-28T22:35:39.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:39 smithi082 ceph-mon[36471]: from='client.25069 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:35:39.749 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:39 smithi043 ceph-mon[31823]: pgmap v391: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 570 B/s wr, 11 op/s 2024-03-28T22:35:39.749 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:39 smithi043 ceph-mon[31823]: from='client.25069 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:35:39.796 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":391,"stamp":"2024-03-28T22:35:38.207880+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662011,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1070,"num_read_kb":666,"num_write":433,"num_write_kb":3086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":856,"ondisk_log_size":856,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":376780,"kb_used_data":15056,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124565044,"statfs":{"total":1151940427776,"available":1151554605056,"internally_reserved":0,"allocated":15417344,"data_stored":8429051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":92,"num_read_kb":52,"num_write":7,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.969599"},"pg_stats":[{"pgid":"3.3f","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.450073+0000","last_change":"2024-03-28T22:33:28.942904+0000","last_active":"2024-03-28T22:35:30.450073+0000","last_peered":"2024-03-28T22:35:30.450073+0000","last_clean":"2024-03-28T22:35:30.450073+0000","last_became_active":"2024-03-28T22:33:28.942574+0000","last_became_peered":"2024-03-28T22:33:28.942574+0000","last_unstale":"2024-03-28T22:35:30.450073+0000","last_undegraded":"2024-03-28T22:35:30.450073+0000","last_fullsized":"2024-03-28T22:35:30.450073+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:55.323919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"94'4","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451519+0000","last_change":"2024-03-28T22:33:28.951795+0000","last_active":"2024-03-28T22:35:30.451519+0000","last_peered":"2024-03-28T22:35:30.451519+0000","last_clean":"2024-03-28T22:35:30.451519+0000","last_became_active":"2024-03-28T22:33:28.951286+0000","last_became_peered":"2024-03-28T22:33:28.951286+0000","last_unstale":"2024-03-28T22:35:30.451519+0000","last_undegraded":"2024-03-28T22:35:30.451519+0000","last_fullsized":"2024-03-28T22:35:30.451519+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:56:11.718244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451291+0000","last_change":"2024-03-28T22:33:28.949476+0000","last_active":"2024-03-28T22:35:30.451291+0000","last_peered":"2024-03-28T22:35:30.451291+0000","last_clean":"2024-03-28T22:35:30.451291+0000","last_became_active":"2024-03-28T22:33:28.949128+0000","last_became_peered":"2024-03-28T22:33:28.949128+0000","last_unstale":"2024-03-28T22:35:30.451291+0000","last_undegraded":"2024-03-28T22:35:30.451291+0000","last_fullsized":"2024-03-28T22:35:30.451291+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:49:29.052373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370703+0000","last_change":"2024-03-28T22:33:28.930715+0000","last_active":"2024-03-28T22:35:30.370703+0000","last_peered":"2024-03-28T22:35:30.370703+0000","last_clean":"2024-03-28T22:35:30.370703+0000","last_became_active":"2024-03-28T22:33:28.930265+0000","last_became_peered":"2024-03-28T22:33:28.930265+0000","last_unstale":"2024-03-28T22:35:30.370703+0000","last_undegraded":"2024-03-28T22:35:30.370703+0000","last_fullsized":"2024-03-28T22:35:30.370703+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:29:54.695671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"94'4","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.253628+0000","last_change":"2024-03-28T22:33:28.944933+0000","last_active":"2024-03-28T22:35:30.253628+0000","last_peered":"2024-03-28T22:35:30.253628+0000","last_clean":"2024-03-28T22:35:30.253628+0000","last_became_active":"2024-03-28T22:33:28.944542+0000","last_became_peered":"2024-03-28T22:33:28.944542+0000","last_unstale":"2024-03-28T22:35:30.253628+0000","last_undegraded":"2024-03-28T22:35:30.253628+0000","last_fullsized":"2024-03-28T22:35:30.253628+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:35:15.719312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"94'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.259239+0000","last_change":"2024-03-28T22:33:28.927920+0000","last_active":"2024-03-28T22:35:30.259239+0000","last_peered":"2024-03-28T22:35:30.259239+0000","last_clean":"2024-03-28T22:35:30.259239+0000","last_became_active":"2024-03-28T22:33:28.927602+0000","last_became_peered":"2024-03-28T22:33:28.927602+0000","last_unstale":"2024-03-28T22:35:30.259239+0000","last_undegraded":"2024-03-28T22:35:30.259239+0000","last_fullsized":"2024-03-28T22:35:30.259239+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:24:22.072707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"94'5","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248009+0000","last_change":"2024-03-28T22:33:28.917963+0000","last_active":"2024-03-28T22:35:30.248009+0000","last_peered":"2024-03-28T22:35:30.248009+0000","last_clean":"2024-03-28T22:35:30.248009+0000","last_became_active":"2024-03-28T22:33:28.917628+0000","last_became_peered":"2024-03-28T22:33:28.917628+0000","last_unstale":"2024-03-28T22:35:30.248009+0000","last_undegraded":"2024-03-28T22:35:30.248009+0000","last_fullsized":"2024-03-28T22:35:30.248009+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:37:09.111640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"94'9","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371107+0000","last_change":"2024-03-28T22:33:28.942632+0000","last_active":"2024-03-28T22:35:30.371107+0000","last_peered":"2024-03-28T22:35:30.371107+0000","last_clean":"2024-03-28T22:35:30.371107+0000","last_became_active":"2024-03-28T22:33:28.941972+0000","last_became_peered":"2024-03-28T22:33:28.941972+0000","last_unstale":"2024-03-28T22:35:30.371107+0000","last_undegraded":"2024-03-28T22:35:30.371107+0000","last_fullsized":"2024-03-28T22:35:30.371107+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:49:52.642314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"94'11","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448719+0000","last_change":"2024-03-28T22:33:28.924324+0000","last_active":"2024-03-28T22:35:30.448719+0000","last_peered":"2024-03-28T22:35:30.448719+0000","last_clean":"2024-03-28T22:35:30.448719+0000","last_became_active":"2024-03-28T22:33:28.924014+0000","last_became_peered":"2024-03-28T22:33:28.924014+0000","last_unstale":"2024-03-28T22:35:30.448719+0000","last_undegraded":"2024-03-28T22:35:30.448719+0000","last_fullsized":"2024-03-28T22:35:30.448719+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:17:55.172116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"94'16","reported_seq":53,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.259531+0000","last_change":"2024-03-28T22:34:29.987262+0000","last_active":"2024-03-28T22:35:30.259531+0000","last_peered":"2024-03-28T22:35:30.259531+0000","last_clean":"2024-03-28T22:35:30.259531+0000","last_became_active":"2024-03-28T22:34:29.986934+0000","last_became_peered":"2024-03-28T22:34:29.986934+0000","last_unstale":"2024-03-28T22:35:30.259531+0000","last_undegraded":"2024-03-28T22:35:30.259531+0000","last_fullsized":"2024-03-28T22:35:30.259531+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:00:41.574800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"94'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248464+0000","last_change":"2024-03-28T22:33:28.930802+0000","last_active":"2024-03-28T22:35:30.248464+0000","last_peered":"2024-03-28T22:35:30.248464+0000","last_clean":"2024-03-28T22:35:30.248464+0000","last_became_active":"2024-03-28T22:33:28.930475+0000","last_became_peered":"2024-03-28T22:33:28.930475+0000","last_unstale":"2024-03-28T22:35:30.248464+0000","last_undegraded":"2024-03-28T22:35:30.248464+0000","last_fullsized":"2024-03-28T22:35:30.248464+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:48.443803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"94'7","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370644+0000","last_change":"2024-03-28T22:33:28.927057+0000","last_active":"2024-03-28T22:35:30.370644+0000","last_peered":"2024-03-28T22:35:30.370644+0000","last_clean":"2024-03-28T22:35:30.370644+0000","last_became_active":"2024-03-28T22:33:28.926598+0000","last_became_peered":"2024-03-28T22:33:28.926598+0000","last_unstale":"2024-03-28T22:35:30.370644+0000","last_undegraded":"2024-03-28T22:35:30.370644+0000","last_fullsized":"2024-03-28T22:35:30.370644+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:21:00.710203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"94'10","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368883+0000","last_change":"2024-03-28T22:33:28.937034+0000","last_active":"2024-03-28T22:35:30.368883+0000","last_peered":"2024-03-28T22:35:30.368883+0000","last_clean":"2024-03-28T22:35:30.368883+0000","last_became_active":"2024-03-28T22:33:28.933713+0000","last_became_peered":"2024-03-28T22:33:28.933713+0000","last_unstale":"2024-03-28T22:35:30.368883+0000","last_undegraded":"2024-03-28T22:35:30.368883+0000","last_fullsized":"2024-03-28T22:35:30.368883+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:58:24.148897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"94'7","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.450910+0000","last_change":"2024-03-28T22:33:28.947384+0000","last_active":"2024-03-28T22:35:30.450910+0000","last_peered":"2024-03-28T22:35:30.450910+0000","last_clean":"2024-03-28T22:35:30.450910+0000","last_became_active":"2024-03-28T22:33:28.947017+0000","last_became_peered":"2024-03-28T22:33:28.947017+0000","last_unstale":"2024-03-28T22:35:30.450910+0000","last_undegraded":"2024-03-28T22:35:30.450910+0000","last_fullsized":"2024-03-28T22:35:30.450910+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:43:32.781161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"94'7","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451705+0000","last_change":"2024-03-28T22:33:28.950119+0000","last_active":"2024-03-28T22:35:30.451705+0000","last_peered":"2024-03-28T22:35:30.451705+0000","last_clean":"2024-03-28T22:35:30.451705+0000","last_became_active":"2024-03-28T22:33:28.949798+0000","last_became_peered":"2024-03-28T22:33:28.949798+0000","last_unstale":"2024-03-28T22:35:30.451705+0000","last_undegraded":"2024-03-28T22:35:30.451705+0000","last_fullsized":"2024-03-28T22:35:30.451705+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:16.717678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"94'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448256+0000","last_change":"2024-03-28T22:33:28.943255+0000","last_active":"2024-03-28T22:35:30.448256+0000","last_peered":"2024-03-28T22:35:30.448256+0000","last_clean":"2024-03-28T22:35:30.448256+0000","last_became_active":"2024-03-28T22:33:28.942923+0000","last_became_peered":"2024-03-28T22:33:28.942923+0000","last_unstale":"2024-03-28T22:35:30.448256+0000","last_undegraded":"2024-03-28T22:35:30.448256+0000","last_fullsized":"2024-03-28T22:35:30.448256+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:05:04.197635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"94'8","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449920+0000","last_change":"2024-03-28T22:33:28.942906+0000","last_active":"2024-03-28T22:35:30.449920+0000","last_peered":"2024-03-28T22:35:30.449920+0000","last_clean":"2024-03-28T22:35:30.449920+0000","last_became_active":"2024-03-28T22:33:28.942573+0000","last_became_peered":"2024-03-28T22:33:28.942573+0000","last_unstale":"2024-03-28T22:35:30.449920+0000","last_undegraded":"2024-03-28T22:35:30.449920+0000","last_fullsized":"2024-03-28T22:35:30.449920+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:39:17.651101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"94'10","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252793+0000","last_change":"2024-03-28T22:33:28.932075+0000","last_active":"2024-03-28T22:35:30.252793+0000","last_peered":"2024-03-28T22:35:30.252793+0000","last_clean":"2024-03-28T22:35:30.252793+0000","last_became_active":"2024-03-28T22:33:28.931689+0000","last_became_peered":"2024-03-28T22:33:28.931689+0000","last_unstale":"2024-03-28T22:35:30.252793+0000","last_undegraded":"2024-03-28T22:35:30.252793+0000","last_fullsized":"2024-03-28T22:35:30.252793+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:50:43.031082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"94'7","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252280+0000","last_change":"2024-03-28T22:33:28.944987+0000","last_active":"2024-03-28T22:35:30.252280+0000","last_peered":"2024-03-28T22:35:30.252280+0000","last_clean":"2024-03-28T22:35:30.252280+0000","last_became_active":"2024-03-28T22:33:28.944622+0000","last_became_peered":"2024-03-28T22:33:28.944622+0000","last_unstale":"2024-03-28T22:35:30.252280+0000","last_undegraded":"2024-03-28T22:35:30.252280+0000","last_fullsized":"2024-03-28T22:35:30.252280+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:48:58.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":29,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251775+0000","last_change":"2024-03-28T22:34:30.429709+0000","last_active":"2024-03-28T22:35:30.251775+0000","last_peered":"2024-03-28T22:35:30.251775+0000","last_clean":"2024-03-28T22:35:30.251775+0000","last_became_active":"2024-03-28T22:34:30.429122+0000","last_became_peered":"2024-03-28T22:34:30.429122+0000","last_unstale":"2024-03-28T22:35:30.251775+0000","last_undegraded":"2024-03-28T22:35:30.251775+0000","last_fullsized":"2024-03-28T22:35:30.251775+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:58:05.313669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"92'13","reported_seq":139,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:36.328203+0000","last_change":"2024-03-28T22:33:28.442524+0000","last_active":"2024-03-28T22:35:36.328203+0000","last_peered":"2024-03-28T22:35:36.328203+0000","last_clean":"2024-03-28T22:35:36.328203+0000","last_became_active":"2024-03-28T22:33:28.442169+0000","last_became_peered":"2024-03-28T22:33:28.442169+0000","last_unstale":"2024-03-28T22:35:36.328203+0000","last_undegraded":"2024-03-28T22:35:36.328203+0000","last_fullsized":"2024-03-28T22:35:36.328203+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:08:47.101119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":92,"num_read_kb":92,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"94'5","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371822+0000","last_change":"2024-03-28T22:33:28.946697+0000","last_active":"2024-03-28T22:35:30.371822+0000","last_peered":"2024-03-28T22:35:30.371822+0000","last_clean":"2024-03-28T22:35:30.371822+0000","last_became_active":"2024-03-28T22:33:28.946303+0000","last_became_peered":"2024-03-28T22:33:28.946303+0000","last_unstale":"2024-03-28T22:35:30.371822+0000","last_undegraded":"2024-03-28T22:35:30.371822+0000","last_fullsized":"2024-03-28T22:35:30.371822+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:32:01.286073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370317+0000","last_change":"2024-03-28T22:34:30.429579+0000","last_active":"2024-03-28T22:35:30.370317+0000","last_peered":"2024-03-28T22:35:30.370317+0000","last_clean":"2024-03-28T22:35:30.370317+0000","last_became_active":"2024-03-28T22:34:30.429054+0000","last_became_peered":"2024-03-28T22:34:30.429054+0000","last_unstale":"2024-03-28T22:35:30.370317+0000","last_undegraded":"2024-03-28T22:35:30.370317+0000","last_fullsized":"2024-03-28T22:35:30.370317+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:03:46.492786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"92'2","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.372174+0000","last_change":"2024-03-28T22:33:28.457661+0000","last_active":"2024-03-28T22:35:30.372174+0000","last_peered":"2024-03-28T22:35:30.372174+0000","last_clean":"2024-03-28T22:35:30.372174+0000","last_became_active":"2024-03-28T22:33:28.457032+0000","last_became_peered":"2024-03-28T22:33:28.457032+0000","last_unstale":"2024-03-28T22:35:30.372174+0000","last_undegraded":"2024-03-28T22:35:30.372174+0000","last_fullsized":"2024-03-28T22:35:30.372174+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:43:09.820630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"94'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.453641+0000","last_change":"2024-03-28T22:33:28.954252+0000","last_active":"2024-03-28T22:35:30.453641+0000","last_peered":"2024-03-28T22:35:30.453641+0000","last_clean":"2024-03-28T22:35:30.453641+0000","last_became_active":"2024-03-28T22:33:28.953922+0000","last_became_peered":"2024-03-28T22:33:28.953922+0000","last_unstale":"2024-03-28T22:35:30.453641+0000","last_undegraded":"2024-03-28T22:35:30.453641+0000","last_fullsized":"2024-03-28T22:35:30.453641+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:18:47.097859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370557+0000","last_change":"2024-03-28T22:33:34.029129+0000","last_active":"2024-03-28T22:35:30.370557+0000","last_peered":"2024-03-28T22:35:30.370557+0000","last_clean":"2024-03-28T22:35:30.370557+0000","last_became_active":"2024-03-28T22:33:34.028452+0000","last_became_peered":"2024-03-28T22:33:34.028452+0000","last_unstale":"2024-03-28T22:35:30.370557+0000","last_undegraded":"2024-03-28T22:35:30.370557+0000","last_fullsized":"2024-03-28T22:35:30.370557+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:28.982773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"92'12","reported_seq":46,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254159+0000","last_change":"2024-03-28T22:33:28.439568+0000","last_active":"2024-03-28T22:35:30.254159+0000","last_peered":"2024-03-28T22:35:30.254159+0000","last_clean":"2024-03-28T22:35:30.254159+0000","last_became_active":"2024-03-28T22:33:28.438887+0000","last_became_peered":"2024-03-28T22:33:28.438887+0000","last_unstale":"2024-03-28T22:35:30.254159+0000","last_undegraded":"2024-03-28T22:35:30.254159+0000","last_fullsized":"2024-03-28T22:35:30.254159+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:56:43.617621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"94'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448533+0000","last_change":"2024-03-28T22:33:28.938311+0000","last_active":"2024-03-28T22:35:30.448533+0000","last_peered":"2024-03-28T22:35:30.448533+0000","last_clean":"2024-03-28T22:35:30.448533+0000","last_became_active":"2024-03-28T22:33:28.937997+0000","last_became_peered":"2024-03-28T22:33:28.937997+0000","last_unstale":"2024-03-28T22:35:30.448533+0000","last_undegraded":"2024-03-28T22:35:30.448533+0000","last_fullsized":"2024-03-28T22:35:30.448533+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:54:44.001545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370378+0000","last_change":"2024-03-28T22:33:34.016931+0000","last_active":"2024-03-28T22:35:30.370378+0000","last_peered":"2024-03-28T22:35:30.370378+0000","last_clean":"2024-03-28T22:35:30.370378+0000","last_became_active":"2024-03-28T22:33:34.016559+0000","last_became_peered":"2024-03-28T22:33:34.016559+0000","last_unstale":"2024-03-28T22:35:30.370378+0000","last_undegraded":"2024-03-28T22:35:30.370378+0000","last_fullsized":"2024-03-28T22:35:30.370378+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:08:31.252351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"92'5","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254044+0000","last_change":"2024-03-28T22:33:28.433166+0000","last_active":"2024-03-28T22:35:30.254044+0000","last_peered":"2024-03-28T22:35:30.254044+0000","last_clean":"2024-03-28T22:35:30.254044+0000","last_became_active":"2024-03-28T22:33:28.432716+0000","last_became_peered":"2024-03-28T22:33:28.432716+0000","last_unstale":"2024-03-28T22:35:30.254044+0000","last_undegraded":"2024-03-28T22:35:30.254044+0000","last_fullsized":"2024-03-28T22:35:30.254044+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:42:41.613583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"94'15","reported_seq":46,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275481+0000","last_change":"2024-03-28T22:33:28.923307+0000","last_active":"2024-03-28T22:35:30.275481+0000","last_peered":"2024-03-28T22:35:30.275481+0000","last_clean":"2024-03-28T22:35:30.275481+0000","last_became_active":"2024-03-28T22:33:28.922499+0000","last_became_peered":"2024-03-28T22:33:28.922499+0000","last_unstale":"2024-03-28T22:35:30.275481+0000","last_undegraded":"2024-03-28T22:35:30.275481+0000","last_fullsized":"2024-03-28T22:35:30.275481+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:48:48.514482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252136+0000","last_change":"2024-03-28T22:33:34.002209+0000","last_active":"2024-03-28T22:35:30.252136+0000","last_peered":"2024-03-28T22:35:30.252136+0000","last_clean":"2024-03-28T22:35:30.252136+0000","last_became_active":"2024-03-28T22:33:34.001746+0000","last_became_peered":"2024-03-28T22:33:34.001746+0000","last_unstale":"2024-03-28T22:35:30.252136+0000","last_undegraded":"2024-03-28T22:35:30.252136+0000","last_fullsized":"2024-03-28T22:35:30.252136+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:02:02.333328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"92'5","reported_seq":131,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:35.343976+0000","last_change":"2024-03-28T22:33:28.435380+0000","last_active":"2024-03-28T22:35:35.343976+0000","last_peered":"2024-03-28T22:35:35.343976+0000","last_clean":"2024-03-28T22:35:35.343976+0000","last_became_active":"2024-03-28T22:33:28.434898+0000","last_became_peered":"2024-03-28T22:33:28.434898+0000","last_unstale":"2024-03-28T22:35:35.343976+0000","last_undegraded":"2024-03-28T22:35:35.343976+0000","last_fullsized":"2024-03-28T22:35:35.343976+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:58:06.810394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":93,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"94'7","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254211+0000","last_change":"2024-03-28T22:34:30.121370+0000","last_active":"2024-03-28T22:35:30.254211+0000","last_peered":"2024-03-28T22:35:30.254211+0000","last_clean":"2024-03-28T22:35:30.254211+0000","last_became_active":"2024-03-28T22:34:30.121086+0000","last_became_peered":"2024-03-28T22:34:30.121086+0000","last_unstale":"2024-03-28T22:35:30.254211+0000","last_undegraded":"2024-03-28T22:35:30.254211+0000","last_fullsized":"2024-03-28T22:35:30.254211+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:26:24.244663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252008+0000","last_change":"2024-03-28T22:33:34.031350+0000","last_active":"2024-03-28T22:35:30.252008+0000","last_peered":"2024-03-28T22:35:30.252008+0000","last_clean":"2024-03-28T22:35:30.252008+0000","last_became_active":"2024-03-28T22:33:34.030946+0000","last_became_peered":"2024-03-28T22:33:34.030946+0000","last_unstale":"2024-03-28T22:35:30.252008+0000","last_undegraded":"2024-03-28T22:35:30.252008+0000","last_fullsized":"2024-03-28T22:35:30.252008+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:54:22.907363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"92'7","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.259165+0000","last_change":"2024-03-28T22:33:28.446197+0000","last_active":"2024-03-28T22:35:30.259165+0000","last_peered":"2024-03-28T22:35:30.259165+0000","last_clean":"2024-03-28T22:35:30.259165+0000","last_became_active":"2024-03-28T22:33:28.445891+0000","last_became_peered":"2024-03-28T22:33:28.445891+0000","last_unstale":"2024-03-28T22:35:30.259165+0000","last_undegraded":"2024-03-28T22:35:30.259165+0000","last_fullsized":"2024-03-28T22:35:30.259165+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:58:57.385302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275539+0000","last_change":"2024-03-28T22:33:28.955651+0000","last_active":"2024-03-28T22:35:30.275539+0000","last_peered":"2024-03-28T22:35:30.275539+0000","last_clean":"2024-03-28T22:35:30.275539+0000","last_became_active":"2024-03-28T22:33:28.955276+0000","last_became_peered":"2024-03-28T22:33:28.955276+0000","last_unstale":"2024-03-28T22:35:30.275539+0000","last_undegraded":"2024-03-28T22:35:30.275539+0000","last_fullsized":"2024-03-28T22:35:30.275539+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:36:13.455327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449765+0000","last_change":"2024-03-28T22:33:34.017520+0000","last_active":"2024-03-28T22:35:30.449765+0000","last_peered":"2024-03-28T22:35:30.449765+0000","last_clean":"2024-03-28T22:35:30.449765+0000","last_became_active":"2024-03-28T22:33:34.017022+0000","last_became_peered":"2024-03-28T22:33:34.017022+0000","last_unstale":"2024-03-28T22:35:30.449765+0000","last_undegraded":"2024-03-28T22:35:30.449765+0000","last_fullsized":"2024-03-28T22:35:30.449765+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:41:17.431581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"92'3","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368118+0000","last_change":"2024-03-28T22:33:28.445401+0000","last_active":"2024-03-28T22:35:30.368118+0000","last_peered":"2024-03-28T22:35:30.368118+0000","last_clean":"2024-03-28T22:35:30.368118+0000","last_became_active":"2024-03-28T22:33:28.444237+0000","last_became_peered":"2024-03-28T22:33:28.444237+0000","last_unstale":"2024-03-28T22:35:30.368118+0000","last_undegraded":"2024-03-28T22:35:30.368118+0000","last_fullsized":"2024-03-28T22:35:30.368118+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:18:12.820890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"94'8","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.250889+0000","last_change":"2024-03-28T22:34:30.156248+0000","last_active":"2024-03-28T22:35:30.250889+0000","last_peered":"2024-03-28T22:35:30.250889+0000","last_clean":"2024-03-28T22:35:30.250889+0000","last_became_active":"2024-03-28T22:34:30.155056+0000","last_became_peered":"2024-03-28T22:34:30.155056+0000","last_unstale":"2024-03-28T22:35:30.250889+0000","last_undegraded":"2024-03-28T22:35:30.250889+0000","last_fullsized":"2024-03-28T22:35:30.250889+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:28:56.469728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451480+0000","last_change":"2024-03-28T22:33:34.019899+0000","last_active":"2024-03-28T22:35:30.451480+0000","last_peered":"2024-03-28T22:35:30.451480+0000","last_clean":"2024-03-28T22:35:30.451480+0000","last_became_active":"2024-03-28T22:33:34.019392+0000","last_became_peered":"2024-03-28T22:33:34.019392+0000","last_unstale":"2024-03-28T22:35:30.451480+0000","last_undegraded":"2024-03-28T22:35:30.451480+0000","last_fullsized":"2024-03-28T22:35:30.451480+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:08:38.633797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"92'3","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275054+0000","last_change":"2024-03-28T22:33:28.433243+0000","last_active":"2024-03-28T22:35:30.275054+0000","last_peered":"2024-03-28T22:35:30.275054+0000","last_clean":"2024-03-28T22:35:30.275054+0000","last_became_active":"2024-03-28T22:33:28.432927+0000","last_became_peered":"2024-03-28T22:33:28.432927+0000","last_unstale":"2024-03-28T22:35:30.275054+0000","last_undegraded":"2024-03-28T22:35:30.275054+0000","last_fullsized":"2024-03-28T22:35:30.275054+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:40:43.444229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"94'11","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251179+0000","last_change":"2024-03-28T22:33:28.948111+0000","last_active":"2024-03-28T22:35:30.251179+0000","last_peered":"2024-03-28T22:35:30.251179+0000","last_clean":"2024-03-28T22:35:30.251179+0000","last_became_active":"2024-03-28T22:33:28.944649+0000","last_became_peered":"2024-03-28T22:33:28.944649+0000","last_unstale":"2024-03-28T22:35:30.251179+0000","last_undegraded":"2024-03-28T22:35:30.251179+0000","last_fullsized":"2024-03-28T22:35:30.251179+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:57:11.483037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371445+0000","last_change":"2024-03-28T22:33:34.000837+0000","last_active":"2024-03-28T22:35:30.371445+0000","last_peered":"2024-03-28T22:35:30.371445+0000","last_clean":"2024-03-28T22:35:30.371445+0000","last_became_active":"2024-03-28T22:33:34.000120+0000","last_became_peered":"2024-03-28T22:33:34.000120+0000","last_unstale":"2024-03-28T22:35:30.371445+0000","last_undegraded":"2024-03-28T22:35:30.371445+0000","last_fullsized":"2024-03-28T22:35:30.371445+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:14:32.028952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"92'4","reported_seq":215,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:34.276599+0000","last_change":"2024-03-28T22:33:28.436985+0000","last_active":"2024-03-28T22:35:34.276599+0000","last_peered":"2024-03-28T22:35:34.276599+0000","last_clean":"2024-03-28T22:35:34.276599+0000","last_became_active":"2024-03-28T22:33:28.436625+0000","last_became_peered":"2024-03-28T22:33:28.436625+0000","last_unstale":"2024-03-28T22:35:34.276599+0000","last_undegraded":"2024-03-28T22:35:34.276599+0000","last_fullsized":"2024-03-28T22:35:34.276599+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:50:45.775191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":178,"num_read_kb":178,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"94'10","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448868+0000","last_change":"2024-03-28T22:33:28.933265+0000","last_active":"2024-03-28T22:35:30.448868+0000","last_peered":"2024-03-28T22:35:30.448868+0000","last_clean":"2024-03-28T22:35:30.448868+0000","last_became_active":"2024-03-28T22:33:28.932954+0000","last_became_peered":"2024-03-28T22:33:28.932954+0000","last_unstale":"2024-03-28T22:35:30.448868+0000","last_undegraded":"2024-03-28T22:35:30.448868+0000","last_fullsized":"2024-03-28T22:35:30.448868+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:37.039055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251115+0000","last_change":"2024-03-28T22:33:34.024529+0000","last_active":"2024-03-28T22:35:30.251115+0000","last_peered":"2024-03-28T22:35:30.251115+0000","last_clean":"2024-03-28T22:35:30.251115+0000","last_became_active":"2024-03-28T22:33:34.020949+0000","last_became_peered":"2024-03-28T22:33:34.020949+0000","last_unstale":"2024-03-28T22:35:30.251115+0000","last_undegraded":"2024-03-28T22:35:30.251115+0000","last_fullsized":"2024-03-28T22:35:30.251115+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:28:52.039684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"92'4","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248336+0000","last_change":"2024-03-28T22:33:28.432599+0000","last_active":"2024-03-28T22:35:30.248336+0000","last_peered":"2024-03-28T22:35:30.248336+0000","last_clean":"2024-03-28T22:35:30.248336+0000","last_became_active":"2024-03-28T22:33:28.430710+0000","last_became_peered":"2024-03-28T22:33:28.430710+0000","last_unstale":"2024-03-28T22:35:30.248336+0000","last_undegraded":"2024-03-28T22:35:30.248336+0000","last_fullsized":"2024-03-28T22:35:30.248336+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:54.174747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"94'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.453785+0000","last_change":"2024-03-28T22:33:28.955441+0000","last_active":"2024-03-28T22:35:30.453785+0000","last_peered":"2024-03-28T22:35:30.453785+0000","last_clean":"2024-03-28T22:35:30.453785+0000","last_became_active":"2024-03-28T22:33:28.953269+0000","last_became_peered":"2024-03-28T22:33:28.953269+0000","last_unstale":"2024-03-28T22:35:30.453785+0000","last_undegraded":"2024-03-28T22:35:30.453785+0000","last_fullsized":"2024-03-28T22:35:30.453785+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:47:04.128605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370663+0000","last_change":"2024-03-28T22:33:34.028791+0000","last_active":"2024-03-28T22:35:30.370663+0000","last_peered":"2024-03-28T22:35:30.370663+0000","last_clean":"2024-03-28T22:35:30.370663+0000","last_became_active":"2024-03-28T22:33:34.028361+0000","last_became_peered":"2024-03-28T22:33:34.028361+0000","last_unstale":"2024-03-28T22:35:30.370663+0000","last_undegraded":"2024-03-28T22:35:30.370663+0000","last_fullsized":"2024-03-28T22:35:30.370663+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:42:21.264459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"92'5","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275230+0000","last_change":"2024-03-28T22:33:28.455353+0000","last_active":"2024-03-28T22:35:30.275230+0000","last_peered":"2024-03-28T22:35:30.275230+0000","last_clean":"2024-03-28T22:35:30.275230+0000","last_became_active":"2024-03-28T22:33:28.455090+0000","last_became_peered":"2024-03-28T22:33:28.455090+0000","last_unstale":"2024-03-28T22:35:30.275230+0000","last_undegraded":"2024-03-28T22:35:30.275230+0000","last_fullsized":"2024-03-28T22:35:30.275230+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:19:22.496600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"94'11","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448676+0000","last_change":"2024-03-28T22:33:28.928875+0000","last_active":"2024-03-28T22:35:30.448676+0000","last_peered":"2024-03-28T22:35:30.448676+0000","last_clean":"2024-03-28T22:35:30.448676+0000","last_became_active":"2024-03-28T22:33:28.928449+0000","last_became_peered":"2024-03-28T22:33:28.928449+0000","last_unstale":"2024-03-28T22:35:30.448676+0000","last_undegraded":"2024-03-28T22:35:30.448676+0000","last_fullsized":"2024-03-28T22:35:30.448676+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:34:00.402100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251201+0000","last_change":"2024-03-28T22:33:34.012739+0000","last_active":"2024-03-28T22:35:30.251201+0000","last_peered":"2024-03-28T22:35:30.251201+0000","last_clean":"2024-03-28T22:35:30.251201+0000","last_became_active":"2024-03-28T22:33:34.012262+0000","last_became_peered":"2024-03-28T22:33:34.012262+0000","last_unstale":"2024-03-28T22:35:30.251201+0000","last_undegraded":"2024-03-28T22:35:30.251201+0000","last_fullsized":"2024-03-28T22:35:30.251201+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:24:48.351037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'38","reported_seq":256,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:34.279031+0000","last_change":"2024-03-28T22:33:28.430606+0000","last_active":"2024-03-28T22:35:34.279031+0000","last_peered":"2024-03-28T22:35:34.279031+0000","last_clean":"2024-03-28T22:35:34.279031+0000","last_became_active":"2024-03-28T22:33:28.430233+0000","last_became_peered":"2024-03-28T22:33:28.430233+0000","last_unstale":"2024-03-28T22:35:34.279031+0000","last_undegraded":"2024-03-28T22:35:34.279031+0000","last_fullsized":"2024-03-28T22:35:34.279031+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:49.000569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64455,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":185,"num_read_kb":156,"num_write":40,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"94'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254029+0000","last_change":"2024-03-28T22:33:28.931167+0000","last_active":"2024-03-28T22:35:30.254029+0000","last_peered":"2024-03-28T22:35:30.254029+0000","last_clean":"2024-03-28T22:35:30.254029+0000","last_became_active":"2024-03-28T22:33:28.930824+0000","last_became_peered":"2024-03-28T22:33:28.930824+0000","last_unstale":"2024-03-28T22:35:30.254029+0000","last_undegraded":"2024-03-28T22:35:30.254029+0000","last_fullsized":"2024-03-28T22:35:30.254029+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:46:49.102791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275795+0000","last_change":"2024-03-28T22:33:34.023056+0000","last_active":"2024-03-28T22:35:30.275795+0000","last_peered":"2024-03-28T22:35:30.275795+0000","last_clean":"2024-03-28T22:35:30.275795+0000","last_became_active":"2024-03-28T22:33:34.022612+0000","last_became_peered":"2024-03-28T22:33:34.022612+0000","last_unstale":"2024-03-28T22:35:30.275795+0000","last_undegraded":"2024-03-28T22:35:30.275795+0000","last_fullsized":"2024-03-28T22:35:30.275795+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:52:31.936008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"92'9","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252059+0000","last_change":"2024-03-28T22:33:28.430963+0000","last_active":"2024-03-28T22:35:30.252059+0000","last_peered":"2024-03-28T22:35:30.252059+0000","last_clean":"2024-03-28T22:35:30.252059+0000","last_became_active":"2024-03-28T22:33:28.430632+0000","last_became_peered":"2024-03-28T22:33:28.430632+0000","last_unstale":"2024-03-28T22:35:30.252059+0000","last_undegraded":"2024-03-28T22:35:30.252059+0000","last_fullsized":"2024-03-28T22:35:30.252059+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:30:23.104769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"94'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368323+0000","last_change":"2024-03-28T22:33:28.954105+0000","last_active":"2024-03-28T22:35:30.368323+0000","last_peered":"2024-03-28T22:35:30.368323+0000","last_clean":"2024-03-28T22:35:30.368323+0000","last_became_active":"2024-03-28T22:33:28.953768+0000","last_became_peered":"2024-03-28T22:33:28.953768+0000","last_unstale":"2024-03-28T22:35:30.368323+0000","last_undegraded":"2024-03-28T22:35:30.368323+0000","last_fullsized":"2024-03-28T22:35:30.368323+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:01:25.184523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449514+0000","last_change":"2024-03-28T22:33:34.027976+0000","last_active":"2024-03-28T22:35:30.449514+0000","last_peered":"2024-03-28T22:35:30.449514+0000","last_clean":"2024-03-28T22:35:30.449514+0000","last_became_active":"2024-03-28T22:33:34.027509+0000","last_became_peered":"2024-03-28T22:33:34.027509+0000","last_unstale":"2024-03-28T22:35:30.449514+0000","last_undegraded":"2024-03-28T22:35:30.449514+0000","last_fullsized":"2024-03-28T22:35:30.449514+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:25:41.908441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"92'5","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275257+0000","last_change":"2024-03-28T22:33:28.433283+0000","last_active":"2024-03-28T22:35:30.275257+0000","last_peered":"2024-03-28T22:35:30.275257+0000","last_clean":"2024-03-28T22:35:30.275257+0000","last_became_active":"2024-03-28T22:33:28.432954+0000","last_became_peered":"2024-03-28T22:33:28.432954+0000","last_unstale":"2024-03-28T22:35:30.275257+0000","last_undegraded":"2024-03-28T22:35:30.275257+0000","last_fullsized":"2024-03-28T22:35:30.275257+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:08:34.480459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"94'15","reported_seq":46,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248416+0000","last_change":"2024-03-28T22:33:28.939016+0000","last_active":"2024-03-28T22:35:30.248416+0000","last_peered":"2024-03-28T22:35:30.248416+0000","last_clean":"2024-03-28T22:35:30.248416+0000","last_became_active":"2024-03-28T22:33:28.938695+0000","last_became_peered":"2024-03-28T22:33:28.938695+0000","last_unstale":"2024-03-28T22:35:30.248416+0000","last_undegraded":"2024-03-28T22:35:30.248416+0000","last_fullsized":"2024-03-28T22:35:30.248416+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:26:53.264559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252656+0000","last_change":"2024-03-28T22:33:34.030127+0000","last_active":"2024-03-28T22:35:30.252656+0000","last_peered":"2024-03-28T22:35:30.252656+0000","last_clean":"2024-03-28T22:35:30.252656+0000","last_became_active":"2024-03-28T22:33:34.028056+0000","last_became_peered":"2024-03-28T22:33:34.028056+0000","last_unstale":"2024-03-28T22:35:30.252656+0000","last_undegraded":"2024-03-28T22:35:30.252656+0000","last_fullsized":"2024-03-28T22:35:30.252656+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:29:41.785013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'7","reported_seq":133,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:35.383574+0000","last_change":"2024-03-28T22:33:28.435350+0000","last_active":"2024-03-28T22:35:35.383574+0000","last_peered":"2024-03-28T22:35:35.383574+0000","last_clean":"2024-03-28T22:35:35.383574+0000","last_became_active":"2024-03-28T22:33:28.434954+0000","last_became_peered":"2024-03-28T22:33:28.434954+0000","last_unstale":"2024-03-28T22:35:35.383574+0000","last_undegraded":"2024-03-28T22:35:35.383574+0000","last_fullsized":"2024-03-28T22:35:35.383574+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:04.969596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":95,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"94'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248078+0000","last_change":"2024-03-28T22:33:28.921219+0000","last_active":"2024-03-28T22:35:30.248078+0000","last_peered":"2024-03-28T22:35:30.248078+0000","last_clean":"2024-03-28T22:35:30.248078+0000","last_became_active":"2024-03-28T22:33:28.920664+0000","last_became_peered":"2024-03-28T22:33:28.920664+0000","last_unstale":"2024-03-28T22:35:30.248078+0000","last_undegraded":"2024-03-28T22:35:30.248078+0000","last_fullsized":"2024-03-28T22:35:30.248078+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:48:55.032058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369304+0000","last_change":"2024-03-28T22:33:34.018101+0000","last_active":"2024-03-28T22:35:30.369304+0000","last_peered":"2024-03-28T22:35:30.369304+0000","last_clean":"2024-03-28T22:35:30.369304+0000","last_became_active":"2024-03-28T22:33:34.017260+0000","last_became_peered":"2024-03-28T22:33:34.017260+0000","last_unstale":"2024-03-28T22:35:30.369304+0000","last_undegraded":"2024-03-28T22:35:30.369304+0000","last_fullsized":"2024-03-28T22:35:30.369304+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:48:53.178090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"92'17","reported_seq":51,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.253538+0000","last_change":"2024-03-28T22:33:28.450232+0000","last_active":"2024-03-28T22:35:30.253538+0000","last_peered":"2024-03-28T22:35:30.253538+0000","last_clean":"2024-03-28T22:35:30.253538+0000","last_became_active":"2024-03-28T22:33:28.449958+0000","last_became_peered":"2024-03-28T22:33:28.449958+0000","last_unstale":"2024-03-28T22:35:30.253538+0000","last_undegraded":"2024-03-28T22:35:30.253538+0000","last_fullsized":"2024-03-28T22:35:30.253538+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:01:42.130260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"94'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275169+0000","last_change":"2024-03-28T22:33:28.926263+0000","last_active":"2024-03-28T22:35:30.275169+0000","last_peered":"2024-03-28T22:35:30.275169+0000","last_clean":"2024-03-28T22:35:30.275169+0000","last_became_active":"2024-03-28T22:33:28.925927+0000","last_became_peered":"2024-03-28T22:33:28.925927+0000","last_unstale":"2024-03-28T22:35:30.275169+0000","last_undegraded":"2024-03-28T22:35:30.275169+0000","last_fullsized":"2024-03-28T22:35:30.275169+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:41:42.477694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.455002+0000","last_change":"2024-03-28T22:35:30.455002+0000","last_active":"2024-03-28T22:35:30.455002+0000","last_peered":"2024-03-28T22:35:30.455002+0000","last_clean":"2024-03-28T22:35:30.455002+0000","last_became_active":"2024-03-28T22:35:30.454479+0000","last_became_peered":"2024-03-28T22:35:30.454479+0000","last_unstale":"2024-03-28T22:35:30.455002+0000","last_undegraded":"2024-03-28T22:35:30.455002+0000","last_fullsized":"2024-03-28T22:35:30.455002+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:57:06.300031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,3,1],"acting":[10,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.f","version":"92'2","reported_seq":21,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449758+0000","last_change":"2024-03-28T22:34:29.986650+0000","last_active":"2024-03-28T22:35:30.449758+0000","last_peered":"2024-03-28T22:35:30.449758+0000","last_clean":"2024-03-28T22:35:30.449758+0000","last_became_active":"2024-03-28T22:34:29.981875+0000","last_became_peered":"2024-03-28T22:34:29.981875+0000","last_unstale":"2024-03-28T22:35:30.449758+0000","last_undegraded":"2024-03-28T22:35:30.449758+0000","last_fullsized":"2024-03-28T22:35:30.449758+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:45:24.166133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"94'5","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449898+0000","last_change":"2024-03-28T22:34:29.982815+0000","last_active":"2024-03-28T22:35:30.449898+0000","last_peered":"2024-03-28T22:35:30.449898+0000","last_clean":"2024-03-28T22:35:30.449898+0000","last_became_active":"2024-03-28T22:34:29.982458+0000","last_became_peered":"2024-03-28T22:34:29.982458+0000","last_unstale":"2024-03-28T22:35:30.449898+0000","last_undegraded":"2024-03-28T22:35:30.449898+0000","last_fullsized":"2024-03-28T22:35:30.449898+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:11:30.617181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":29,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451063+0000","last_change":"2024-03-28T22:35:30.451063+0000","last_active":"2024-03-28T22:35:30.451063+0000","last_peered":"2024-03-28T22:35:30.451063+0000","last_clean":"2024-03-28T22:35:30.451063+0000","last_became_active":"2024-03-28T22:35:30.450525+0000","last_became_peered":"2024-03-28T22:35:30.450525+0000","last_unstale":"2024-03-28T22:35:30.451063+0000","last_undegraded":"2024-03-28T22:35:30.451063+0000","last_fullsized":"2024-03-28T22:35:30.451063+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:35:29.004834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"92'1","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252202+0000","last_change":"2024-03-28T22:34:29.992884+0000","last_active":"2024-03-28T22:35:30.252202+0000","last_peered":"2024-03-28T22:35:30.252202+0000","last_clean":"2024-03-28T22:35:30.252202+0000","last_became_active":"2024-03-28T22:34:29.991657+0000","last_became_peered":"2024-03-28T22:34:29.991657+0000","last_unstale":"2024-03-28T22:35:30.252202+0000","last_undegraded":"2024-03-28T22:35:30.252202+0000","last_fullsized":"2024-03-28T22:35:30.252202+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:25:14.540783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"94'4","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.259438+0000","last_change":"2024-03-28T22:33:28.942097+0000","last_active":"2024-03-28T22:35:30.259438+0000","last_peered":"2024-03-28T22:35:30.259438+0000","last_clean":"2024-03-28T22:35:30.259438+0000","last_became_active":"2024-03-28T22:33:28.941812+0000","last_became_peered":"2024-03-28T22:33:28.941812+0000","last_unstale":"2024-03-28T22:35:30.259438+0000","last_undegraded":"2024-03-28T22:35:30.259438+0000","last_fullsized":"2024-03-28T22:35:30.259438+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:36:10.204216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369114+0000","last_change":"2024-03-28T22:33:34.023171+0000","last_active":"2024-03-28T22:35:30.369114+0000","last_peered":"2024-03-28T22:35:30.369114+0000","last_clean":"2024-03-28T22:35:30.369114+0000","last_became_active":"2024-03-28T22:33:34.021313+0000","last_became_peered":"2024-03-28T22:33:34.021313+0000","last_unstale":"2024-03-28T22:35:30.369114+0000","last_undegraded":"2024-03-28T22:35:30.369114+0000","last_fullsized":"2024-03-28T22:35:30.369114+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:29:42.592761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"92'5","reported_seq":128,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:33.324016+0000","last_change":"2024-03-28T22:33:28.449133+0000","last_active":"2024-03-28T22:35:33.324016+0000","last_peered":"2024-03-28T22:35:33.324016+0000","last_clean":"2024-03-28T22:35:33.324016+0000","last_became_active":"2024-03-28T22:33:28.448743+0000","last_became_peered":"2024-03-28T22:33:28.448743+0000","last_unstale":"2024-03-28T22:35:33.324016+0000","last_undegraded":"2024-03-28T22:35:33.324016+0000","last_fullsized":"2024-03-28T22:35:33.324016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:09:37.119736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":89,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.259332+0000","last_change":"2024-03-28T22:33:28.944198+0000","last_active":"2024-03-28T22:35:30.259332+0000","last_peered":"2024-03-28T22:35:30.259332+0000","last_clean":"2024-03-28T22:35:30.259332+0000","last_became_active":"2024-03-28T22:33:28.943818+0000","last_became_peered":"2024-03-28T22:33:28.943818+0000","last_unstale":"2024-03-28T22:35:30.259332+0000","last_undegraded":"2024-03-28T22:35:30.259332+0000","last_fullsized":"2024-03-28T22:35:30.259332+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:31:03.791364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369445+0000","last_change":"2024-03-28T22:33:34.015499+0000","last_active":"2024-03-28T22:35:30.369445+0000","last_peered":"2024-03-28T22:35:30.369445+0000","last_clean":"2024-03-28T22:35:30.369445+0000","last_became_active":"2024-03-28T22:33:34.015013+0000","last_became_peered":"2024-03-28T22:33:34.015013+0000","last_unstale":"2024-03-28T22:35:30.369445+0000","last_undegraded":"2024-03-28T22:35:30.369445+0000","last_fullsized":"2024-03-28T22:35:30.369445+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:34:30.582198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"94'8","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275296+0000","last_change":"2024-03-28T22:33:28.926732+0000","last_active":"2024-03-28T22:35:30.275296+0000","last_peered":"2024-03-28T22:35:30.275296+0000","last_clean":"2024-03-28T22:35:30.275296+0000","last_became_active":"2024-03-28T22:33:28.926366+0000","last_became_peered":"2024-03-28T22:33:28.926366+0000","last_unstale":"2024-03-28T22:35:30.275296+0000","last_undegraded":"2024-03-28T22:35:30.275296+0000","last_fullsized":"2024-03-28T22:35:30.275296+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:19:10.769141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'6","reported_seq":133,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:36.384921+0000","last_change":"2024-03-28T22:33:28.457400+0000","last_active":"2024-03-28T22:35:36.384921+0000","last_peered":"2024-03-28T22:35:36.384921+0000","last_clean":"2024-03-28T22:35:36.384921+0000","last_became_active":"2024-03-28T22:33:28.456994+0000","last_became_peered":"2024-03-28T22:33:28.456994+0000","last_unstale":"2024-03-28T22:35:36.384921+0000","last_undegraded":"2024-03-28T22:35:36.384921+0000","last_fullsized":"2024-03-28T22:35:36.384921+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:32:16.438540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":93,"num_read_kb":93,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"92'1","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248340+0000","last_change":"2024-03-28T22:33:34.027746+0000","last_active":"2024-03-28T22:35:30.248340+0000","last_peered":"2024-03-28T22:35:30.248340+0000","last_clean":"2024-03-28T22:35:30.248340+0000","last_became_active":"2024-03-28T22:33:34.027419+0000","last_became_peered":"2024-03-28T22:33:34.027419+0000","last_unstale":"2024-03-28T22:35:30.248340+0000","last_undegraded":"2024-03-28T22:35:30.248340+0000","last_fullsized":"2024-03-28T22:35:30.248340+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:55:43.686428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371544+0000","last_change":"2024-03-28T22:33:28.952393+0000","last_active":"2024-03-28T22:35:30.371544+0000","last_peered":"2024-03-28T22:35:30.371544+0000","last_clean":"2024-03-28T22:35:30.371544+0000","last_became_active":"2024-03-28T22:33:28.951951+0000","last_became_peered":"2024-03-28T22:33:28.951951+0000","last_unstale":"2024-03-28T22:35:30.371544+0000","last_undegraded":"2024-03-28T22:35:30.371544+0000","last_fullsized":"2024-03-28T22:35:30.371544+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:09:25.808842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"92'6","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370674+0000","last_change":"2024-03-28T22:33:28.437177+0000","last_active":"2024-03-28T22:35:30.370674+0000","last_peered":"2024-03-28T22:35:30.370674+0000","last_clean":"2024-03-28T22:35:30.370674+0000","last_became_active":"2024-03-28T22:33:28.436801+0000","last_became_peered":"2024-03-28T22:33:28.436801+0000","last_unstale":"2024-03-28T22:35:30.370674+0000","last_undegraded":"2024-03-28T22:35:30.370674+0000","last_fullsized":"2024-03-28T22:35:30.370674+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:24:52.549490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":371,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.247934+0000","last_change":"2024-03-28T22:32:07.867336+0000","last_active":"2024-03-28T22:35:30.247934+0000","last_peered":"2024-03-28T22:35:30.247934+0000","last_clean":"2024-03-28T22:35:30.247934+0000","last_became_active":"2024-03-28T22:32:07.866819+0000","last_became_peered":"2024-03-28T22:32:07.866819+0000","last_unstale":"2024-03-28T22:35:30.247934+0000","last_undegraded":"2024-03-28T22:35:30.247934+0000","last_fullsized":"2024-03-28T22:35:30.247934+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:28:14.713501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:28:14.713501+0000","last_clean_scrub_stamp":"2024-03-28T22:28:14.713501+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:00:52.438708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275546+0000","last_change":"2024-03-28T22:33:34.020668+0000","last_active":"2024-03-28T22:35:30.275546+0000","last_peered":"2024-03-28T22:35:30.275546+0000","last_clean":"2024-03-28T22:35:30.275546+0000","last_became_active":"2024-03-28T22:33:34.020240+0000","last_became_peered":"2024-03-28T22:33:34.020240+0000","last_unstale":"2024-03-28T22:35:30.275546+0000","last_undegraded":"2024-03-28T22:35:30.275546+0000","last_fullsized":"2024-03-28T22:35:30.275546+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:18.983983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"94'5","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368061+0000","last_change":"2024-03-28T22:33:28.935237+0000","last_active":"2024-03-28T22:35:30.368061+0000","last_peered":"2024-03-28T22:35:30.368061+0000","last_clean":"2024-03-28T22:35:30.368061+0000","last_became_active":"2024-03-28T22:33:28.934875+0000","last_became_peered":"2024-03-28T22:33:28.934875+0000","last_unstale":"2024-03-28T22:35:30.368061+0000","last_undegraded":"2024-03-28T22:35:30.368061+0000","last_fullsized":"2024-03-28T22:35:30.368061+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:20:50.789302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"92'4","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248187+0000","last_change":"2024-03-28T22:33:28.441654+0000","last_active":"2024-03-28T22:35:30.248187+0000","last_peered":"2024-03-28T22:35:30.248187+0000","last_clean":"2024-03-28T22:35:30.248187+0000","last_became_active":"2024-03-28T22:33:28.441293+0000","last_became_peered":"2024-03-28T22:33:28.441293+0000","last_unstale":"2024-03-28T22:35:30.248187+0000","last_undegraded":"2024-03-28T22:35:30.248187+0000","last_fullsized":"2024-03-28T22:35:30.248187+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:31:17.737977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449436+0000","last_change":"2024-03-28T22:33:34.018126+0000","last_active":"2024-03-28T22:35:30.449436+0000","last_peered":"2024-03-28T22:35:30.449436+0000","last_clean":"2024-03-28T22:35:30.449436+0000","last_became_active":"2024-03-28T22:33:34.017573+0000","last_became_peered":"2024-03-28T22:33:34.017573+0000","last_unstale":"2024-03-28T22:35:30.449436+0000","last_undegraded":"2024-03-28T22:35:30.449436+0000","last_fullsized":"2024-03-28T22:35:30.449436+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:23:31.013511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"92'4","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254108+0000","last_change":"2024-03-28T22:33:28.454069+0000","last_active":"2024-03-28T22:35:30.254108+0000","last_peered":"2024-03-28T22:35:30.254108+0000","last_clean":"2024-03-28T22:35:30.254108+0000","last_became_active":"2024-03-28T22:33:28.453754+0000","last_became_peered":"2024-03-28T22:33:28.453754+0000","last_unstale":"2024-03-28T22:35:30.254108+0000","last_undegraded":"2024-03-28T22:35:30.254108+0000","last_fullsized":"2024-03-28T22:35:30.254108+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:23:33.981072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"94'9","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369387+0000","last_change":"2024-03-28T22:33:28.934390+0000","last_active":"2024-03-28T22:35:30.369387+0000","last_peered":"2024-03-28T22:35:30.369387+0000","last_clean":"2024-03-28T22:35:30.369387+0000","last_became_active":"2024-03-28T22:33:28.933919+0000","last_became_peered":"2024-03-28T22:33:28.933919+0000","last_unstale":"2024-03-28T22:35:30.369387+0000","last_undegraded":"2024-03-28T22:35:30.369387+0000","last_fullsized":"2024-03-28T22:35:30.369387+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:27:07.595144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371312+0000","last_change":"2024-03-28T22:33:34.019910+0000","last_active":"2024-03-28T22:35:30.371312+0000","last_peered":"2024-03-28T22:35:30.371312+0000","last_clean":"2024-03-28T22:35:30.371312+0000","last_became_active":"2024-03-28T22:33:34.019522+0000","last_became_peered":"2024-03-28T22:33:34.019522+0000","last_unstale":"2024-03-28T22:35:30.371312+0000","last_undegraded":"2024-03-28T22:35:30.371312+0000","last_fullsized":"2024-03-28T22:35:30.371312+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:01:29.298513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'14","reported_seq":47,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252650+0000","last_change":"2024-03-28T22:33:28.444513+0000","last_active":"2024-03-28T22:35:30.252650+0000","last_peered":"2024-03-28T22:35:30.252650+0000","last_clean":"2024-03-28T22:35:30.252650+0000","last_became_active":"2024-03-28T22:33:28.444091+0000","last_became_peered":"2024-03-28T22:33:28.444091+0000","last_unstale":"2024-03-28T22:35:30.252650+0000","last_undegraded":"2024-03-28T22:35:30.252650+0000","last_fullsized":"2024-03-28T22:35:30.252650+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:49:58.331864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"94'5","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.372101+0000","last_change":"2024-03-28T22:33:28.948390+0000","last_active":"2024-03-28T22:35:30.372101+0000","last_peered":"2024-03-28T22:35:30.372101+0000","last_clean":"2024-03-28T22:35:30.372101+0000","last_became_active":"2024-03-28T22:33:28.942846+0000","last_became_peered":"2024-03-28T22:33:28.942846+0000","last_unstale":"2024-03-28T22:35:30.372101+0000","last_undegraded":"2024-03-28T22:35:30.372101+0000","last_fullsized":"2024-03-28T22:35:30.372101+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:50.596365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251650+0000","last_change":"2024-03-28T22:33:33.999429+0000","last_active":"2024-03-28T22:35:30.251650+0000","last_peered":"2024-03-28T22:35:30.251650+0000","last_clean":"2024-03-28T22:35:30.251650+0000","last_became_active":"2024-03-28T22:33:33.998979+0000","last_became_peered":"2024-03-28T22:33:33.998979+0000","last_unstale":"2024-03-28T22:35:30.251650+0000","last_undegraded":"2024-03-28T22:35:30.251650+0000","last_fullsized":"2024-03-28T22:35:30.251650+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:26.450731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"92'7","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370722+0000","last_change":"2024-03-28T22:33:28.442861+0000","last_active":"2024-03-28T22:35:30.370722+0000","last_peered":"2024-03-28T22:35:30.370722+0000","last_clean":"2024-03-28T22:35:30.370722+0000","last_became_active":"2024-03-28T22:33:28.442468+0000","last_became_peered":"2024-03-28T22:33:28.442468+0000","last_unstale":"2024-03-28T22:35:30.370722+0000","last_undegraded":"2024-03-28T22:35:30.370722+0000","last_fullsized":"2024-03-28T22:35:30.370722+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:45:49.899031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"94'7","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451836+0000","last_change":"2024-03-28T22:34:30.429872+0000","last_active":"2024-03-28T22:35:30.451836+0000","last_peered":"2024-03-28T22:35:30.451836+0000","last_clean":"2024-03-28T22:35:30.451836+0000","last_became_active":"2024-03-28T22:34:30.429237+0000","last_became_peered":"2024-03-28T22:34:30.429237+0000","last_unstale":"2024-03-28T22:35:30.451836+0000","last_undegraded":"2024-03-28T22:35:30.451836+0000","last_fullsized":"2024-03-28T22:35:30.451836+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:14:46.608826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370518+0000","last_change":"2024-03-28T22:33:34.013826+0000","last_active":"2024-03-28T22:35:30.370518+0000","last_peered":"2024-03-28T22:35:30.370518+0000","last_clean":"2024-03-28T22:35:30.370518+0000","last_became_active":"2024-03-28T22:33:34.013415+0000","last_became_peered":"2024-03-28T22:33:34.013415+0000","last_unstale":"2024-03-28T22:35:30.370518+0000","last_undegraded":"2024-03-28T22:35:30.370518+0000","last_fullsized":"2024-03-28T22:35:30.370518+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:19:30.965940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"92'2","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448334+0000","last_change":"2024-03-28T22:33:28.442225+0000","last_active":"2024-03-28T22:35:30.448334+0000","last_peered":"2024-03-28T22:35:30.448334+0000","last_clean":"2024-03-28T22:35:30.448334+0000","last_became_active":"2024-03-28T22:33:28.441844+0000","last_became_peered":"2024-03-28T22:33:28.441844+0000","last_unstale":"2024-03-28T22:35:30.448334+0000","last_undegraded":"2024-03-28T22:35:30.448334+0000","last_fullsized":"2024-03-28T22:35:30.448334+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:40:52.108663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"94'2","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.253620+0000","last_change":"2024-03-28T22:33:28.927571+0000","last_active":"2024-03-28T22:35:30.253620+0000","last_peered":"2024-03-28T22:35:30.253620+0000","last_clean":"2024-03-28T22:35:30.253620+0000","last_became_active":"2024-03-28T22:33:28.927275+0000","last_became_peered":"2024-03-28T22:33:28.927275+0000","last_unstale":"2024-03-28T22:35:30.253620+0000","last_undegraded":"2024-03-28T22:35:30.253620+0000","last_fullsized":"2024-03-28T22:35:30.253620+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:25:16.184533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252877+0000","last_change":"2024-03-28T22:33:34.018429+0000","last_active":"2024-03-28T22:35:30.252877+0000","last_peered":"2024-03-28T22:35:30.252877+0000","last_clean":"2024-03-28T22:35:30.252877+0000","last_became_active":"2024-03-28T22:33:34.017066+0000","last_became_peered":"2024-03-28T22:33:34.017066+0000","last_unstale":"2024-03-28T22:35:30.252877+0000","last_undegraded":"2024-03-28T22:35:30.252877+0000","last_fullsized":"2024-03-28T22:35:30.252877+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:35:42.101072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"92'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371904+0000","last_change":"2024-03-28T22:33:28.437760+0000","last_active":"2024-03-28T22:35:30.371904+0000","last_peered":"2024-03-28T22:35:30.371904+0000","last_clean":"2024-03-28T22:35:30.371904+0000","last_became_active":"2024-03-28T22:33:28.437096+0000","last_became_peered":"2024-03-28T22:33:28.437096+0000","last_unstale":"2024-03-28T22:35:30.371904+0000","last_undegraded":"2024-03-28T22:35:30.371904+0000","last_fullsized":"2024-03-28T22:35:30.371904+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:29:20.778551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"94'9","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449293+0000","last_change":"2024-03-28T22:33:28.935805+0000","last_active":"2024-03-28T22:35:30.449293+0000","last_peered":"2024-03-28T22:35:30.449293+0000","last_clean":"2024-03-28T22:35:30.449293+0000","last_became_active":"2024-03-28T22:33:28.935520+0000","last_became_peered":"2024-03-28T22:33:28.935520+0000","last_unstale":"2024-03-28T22:35:30.449293+0000","last_undegraded":"2024-03-28T22:35:30.449293+0000","last_fullsized":"2024-03-28T22:35:30.449293+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:23.414310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"92'1","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370530+0000","last_change":"2024-03-28T22:33:34.029518+0000","last_active":"2024-03-28T22:35:30.370530+0000","last_peered":"2024-03-28T22:35:30.370530+0000","last_clean":"2024-03-28T22:35:30.370530+0000","last_became_active":"2024-03-28T22:33:34.029101+0000","last_became_peered":"2024-03-28T22:33:34.029101+0000","last_unstale":"2024-03-28T22:35:30.370530+0000","last_undegraded":"2024-03-28T22:35:30.370530+0000","last_fullsized":"2024-03-28T22:35:30.370530+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:05.595543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"92'3","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251878+0000","last_change":"2024-03-28T22:33:28.438857+0000","last_active":"2024-03-28T22:35:30.251878+0000","last_peered":"2024-03-28T22:35:30.251878+0000","last_clean":"2024-03-28T22:35:30.251878+0000","last_became_active":"2024-03-28T22:33:28.438528+0000","last_became_peered":"2024-03-28T22:33:28.438528+0000","last_unstale":"2024-03-28T22:35:30.251878+0000","last_undegraded":"2024-03-28T22:35:30.251878+0000","last_fullsized":"2024-03-28T22:35:30.251878+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:07:54.506739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"94'8","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371602+0000","last_change":"2024-03-28T22:33:28.958271+0000","last_active":"2024-03-28T22:35:30.371602+0000","last_peered":"2024-03-28T22:35:30.371602+0000","last_clean":"2024-03-28T22:35:30.371602+0000","last_became_active":"2024-03-28T22:33:28.957922+0000","last_became_peered":"2024-03-28T22:33:28.957922+0000","last_unstale":"2024-03-28T22:35:30.371602+0000","last_undegraded":"2024-03-28T22:35:30.371602+0000","last_fullsized":"2024-03-28T22:35:30.371602+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:33:19.003131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370511+0000","last_change":"2024-03-28T22:33:34.000789+0000","last_active":"2024-03-28T22:35:30.370511+0000","last_peered":"2024-03-28T22:35:30.370511+0000","last_clean":"2024-03-28T22:35:30.370511+0000","last_became_active":"2024-03-28T22:33:34.000383+0000","last_became_peered":"2024-03-28T22:33:34.000383+0000","last_unstale":"2024-03-28T22:35:30.370511+0000","last_undegraded":"2024-03-28T22:35:30.370511+0000","last_fullsized":"2024-03-28T22:35:30.370511+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:50:29.925656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"92'4","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.253761+0000","last_change":"2024-03-28T22:33:28.439121+0000","last_active":"2024-03-28T22:35:30.253761+0000","last_peered":"2024-03-28T22:35:30.253761+0000","last_clean":"2024-03-28T22:35:30.253761+0000","last_became_active":"2024-03-28T22:33:28.438800+0000","last_became_peered":"2024-03-28T22:33:28.438800+0000","last_unstale":"2024-03-28T22:35:30.253761+0000","last_undegraded":"2024-03-28T22:35:30.253761+0000","last_fullsized":"2024-03-28T22:35:30.253761+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:57:10.013246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"94'5","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248101+0000","last_change":"2024-03-28T22:33:28.934499+0000","last_active":"2024-03-28T22:35:30.248101+0000","last_peered":"2024-03-28T22:35:30.248101+0000","last_clean":"2024-03-28T22:35:30.248101+0000","last_became_active":"2024-03-28T22:33:28.934147+0000","last_became_peered":"2024-03-28T22:33:28.934147+0000","last_unstale":"2024-03-28T22:35:30.248101+0000","last_undegraded":"2024-03-28T22:35:30.248101+0000","last_fullsized":"2024-03-28T22:35:30.248101+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:35:39.090052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448349+0000","last_change":"2024-03-28T22:33:34.017946+0000","last_active":"2024-03-28T22:35:30.448349+0000","last_peered":"2024-03-28T22:35:30.448349+0000","last_clean":"2024-03-28T22:35:30.448349+0000","last_became_active":"2024-03-28T22:33:34.017546+0000","last_became_peered":"2024-03-28T22:33:34.017546+0000","last_unstale":"2024-03-28T22:35:30.448349+0000","last_undegraded":"2024-03-28T22:35:30.448349+0000","last_fullsized":"2024-03-28T22:35:30.448349+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:14:39.081677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"92'5","reported_seq":128,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:32.275107+0000","last_change":"2024-03-28T22:33:28.453387+0000","last_active":"2024-03-28T22:35:32.275107+0000","last_peered":"2024-03-28T22:35:32.275107+0000","last_clean":"2024-03-28T22:35:32.275107+0000","last_became_active":"2024-03-28T22:33:28.453029+0000","last_became_peered":"2024-03-28T22:33:28.453029+0000","last_unstale":"2024-03-28T22:35:32.275107+0000","last_undegraded":"2024-03-28T22:35:32.275107+0000","last_fullsized":"2024-03-28T22:35:32.275107+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:12:33.246863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":90,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"94'7","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248569+0000","last_change":"2024-03-28T22:33:28.923819+0000","last_active":"2024-03-28T22:35:30.248569+0000","last_peered":"2024-03-28T22:35:30.248569+0000","last_clean":"2024-03-28T22:35:30.248569+0000","last_became_active":"2024-03-28T22:33:28.923471+0000","last_became_peered":"2024-03-28T22:33:28.923471+0000","last_unstale":"2024-03-28T22:35:30.248569+0000","last_undegraded":"2024-03-28T22:35:30.248569+0000","last_fullsized":"2024-03-28T22:35:30.248569+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:58:03.496406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.450859+0000","last_change":"2024-03-28T22:33:34.029789+0000","last_active":"2024-03-28T22:35:30.450859+0000","last_peered":"2024-03-28T22:35:30.450859+0000","last_clean":"2024-03-28T22:35:30.450859+0000","last_became_active":"2024-03-28T22:33:34.029231+0000","last_became_peered":"2024-03-28T22:33:34.029231+0000","last_unstale":"2024-03-28T22:35:30.450859+0000","last_undegraded":"2024-03-28T22:35:30.450859+0000","last_fullsized":"2024-03-28T22:35:30.450859+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:47:50.101393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"92'3","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275488+0000","last_change":"2024-03-28T22:33:28.435668+0000","last_active":"2024-03-28T22:35:30.275488+0000","last_peered":"2024-03-28T22:35:30.275488+0000","last_clean":"2024-03-28T22:35:30.275488+0000","last_became_active":"2024-03-28T22:33:28.435325+0000","last_became_peered":"2024-03-28T22:33:28.435325+0000","last_unstale":"2024-03-28T22:35:30.275488+0000","last_undegraded":"2024-03-28T22:35:30.275488+0000","last_fullsized":"2024-03-28T22:35:30.275488+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:13:06.281074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"94'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248195+0000","last_change":"2024-03-28T22:33:28.954138+0000","last_active":"2024-03-28T22:35:30.248195+0000","last_peered":"2024-03-28T22:35:30.248195+0000","last_clean":"2024-03-28T22:35:30.248195+0000","last_became_active":"2024-03-28T22:33:28.953752+0000","last_became_peered":"2024-03-28T22:33:28.953752+0000","last_unstale":"2024-03-28T22:35:30.248195+0000","last_undegraded":"2024-03-28T22:35:30.248195+0000","last_fullsized":"2024-03-28T22:35:30.248195+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:53:34.892316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.274936+0000","last_change":"2024-03-28T22:33:34.002672+0000","last_active":"2024-03-28T22:35:30.274936+0000","last_peered":"2024-03-28T22:35:30.274936+0000","last_clean":"2024-03-28T22:35:30.274936+0000","last_became_active":"2024-03-28T22:33:34.002195+0000","last_became_peered":"2024-03-28T22:33:34.002195+0000","last_unstale":"2024-03-28T22:35:30.274936+0000","last_undegraded":"2024-03-28T22:35:30.274936+0000","last_fullsized":"2024-03-28T22:35:30.274936+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:24:11.521474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"92'5","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369164+0000","last_change":"2024-03-28T22:33:28.450818+0000","last_active":"2024-03-28T22:35:30.369164+0000","last_peered":"2024-03-28T22:35:30.369164+0000","last_clean":"2024-03-28T22:35:30.369164+0000","last_became_active":"2024-03-28T22:33:28.448592+0000","last_became_peered":"2024-03-28T22:33:28.448592+0000","last_unstale":"2024-03-28T22:35:30.369164+0000","last_undegraded":"2024-03-28T22:35:30.369164+0000","last_fullsized":"2024-03-28T22:35:30.369164+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:19:27.126306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"94'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248443+0000","last_change":"2024-03-28T22:33:28.925191+0000","last_active":"2024-03-28T22:35:30.248443+0000","last_peered":"2024-03-28T22:35:30.248443+0000","last_clean":"2024-03-28T22:35:30.248443+0000","last_became_active":"2024-03-28T22:33:28.924424+0000","last_became_peered":"2024-03-28T22:33:28.924424+0000","last_unstale":"2024-03-28T22:35:30.248443+0000","last_undegraded":"2024-03-28T22:35:30.248443+0000","last_fullsized":"2024-03-28T22:35:30.248443+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:50:00.415047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"94'9","reported_seq":46,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449424+0000","last_change":"2024-03-28T22:34:29.987176+0000","last_active":"2024-03-28T22:35:30.449424+0000","last_peered":"2024-03-28T22:35:30.449424+0000","last_clean":"2024-03-28T22:35:30.449424+0000","last_became_active":"2024-03-28T22:34:29.986807+0000","last_became_peered":"2024-03-28T22:34:29.986807+0000","last_unstale":"2024-03-28T22:35:30.449424+0000","last_undegraded":"2024-03-28T22:35:30.449424+0000","last_fullsized":"2024-03-28T22:35:30.449424+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:00:06.774521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"94'12","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251938+0000","last_change":"2024-03-28T22:33:28.927068+0000","last_active":"2024-03-28T22:35:30.251938+0000","last_peered":"2024-03-28T22:35:30.251938+0000","last_clean":"2024-03-28T22:35:30.251938+0000","last_became_active":"2024-03-28T22:33:28.926734+0000","last_became_peered":"2024-03-28T22:33:28.926734+0000","last_unstale":"2024-03-28T22:35:30.251938+0000","last_undegraded":"2024-03-28T22:35:30.251938+0000","last_fullsized":"2024-03-28T22:35:30.251938+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:56.309555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"94'8","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251930+0000","last_change":"2024-03-28T22:33:28.955675+0000","last_active":"2024-03-28T22:35:30.251930+0000","last_peered":"2024-03-28T22:35:30.251930+0000","last_clean":"2024-03-28T22:35:30.251930+0000","last_became_active":"2024-03-28T22:33:28.955329+0000","last_became_peered":"2024-03-28T22:33:28.955329+0000","last_unstale":"2024-03-28T22:35:30.251930+0000","last_undegraded":"2024-03-28T22:35:30.251930+0000","last_fullsized":"2024-03-28T22:35:30.251930+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:34:14.166434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.452477+0000","last_change":"2024-03-28T22:33:28.955125+0000","last_active":"2024-03-28T22:35:30.452477+0000","last_peered":"2024-03-28T22:35:30.452477+0000","last_clean":"2024-03-28T22:35:30.452477+0000","last_became_active":"2024-03-28T22:33:28.954434+0000","last_became_peered":"2024-03-28T22:33:28.954434+0000","last_unstale":"2024-03-28T22:35:30.452477+0000","last_undegraded":"2024-03-28T22:35:30.452477+0000","last_fullsized":"2024-03-28T22:35:30.452477+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:56:50.705012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"94'4","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369377+0000","last_change":"2024-03-28T22:33:28.945230+0000","last_active":"2024-03-28T22:35:30.369377+0000","last_peered":"2024-03-28T22:35:30.369377+0000","last_clean":"2024-03-28T22:35:30.369377+0000","last_became_active":"2024-03-28T22:33:28.944921+0000","last_became_peered":"2024-03-28T22:33:28.944921+0000","last_unstale":"2024-03-28T22:35:30.369377+0000","last_undegraded":"2024-03-28T22:35:30.369377+0000","last_fullsized":"2024-03-28T22:35:30.369377+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:26:26.505774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"94'6","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449689+0000","last_change":"2024-03-28T22:33:28.928720+0000","last_active":"2024-03-28T22:35:30.449689+0000","last_peered":"2024-03-28T22:35:30.449689+0000","last_clean":"2024-03-28T22:35:30.449689+0000","last_became_active":"2024-03-28T22:33:28.928415+0000","last_became_peered":"2024-03-28T22:33:28.928415+0000","last_unstale":"2024-03-28T22:35:30.449689+0000","last_undegraded":"2024-03-28T22:35:30.449689+0000","last_fullsized":"2024-03-28T22:35:30.449689+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:00.839609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"94'15","reported_seq":46,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252371+0000","last_change":"2024-03-28T22:33:28.924948+0000","last_active":"2024-03-28T22:35:30.252371+0000","last_peered":"2024-03-28T22:35:30.252371+0000","last_clean":"2024-03-28T22:35:30.252371+0000","last_became_active":"2024-03-28T22:33:28.924631+0000","last_became_peered":"2024-03-28T22:33:28.924631+0000","last_unstale":"2024-03-28T22:35:30.252371+0000","last_undegraded":"2024-03-28T22:35:30.252371+0000","last_fullsized":"2024-03-28T22:35:30.252371+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:53:44.909991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"94'2","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.372370+0000","last_change":"2024-03-28T22:33:28.948725+0000","last_active":"2024-03-28T22:35:30.372370+0000","last_peered":"2024-03-28T22:35:30.372370+0000","last_clean":"2024-03-28T22:35:30.372370+0000","last_became_active":"2024-03-28T22:33:28.943942+0000","last_became_peered":"2024-03-28T22:33:28.943942+0000","last_unstale":"2024-03-28T22:35:30.372370+0000","last_undegraded":"2024-03-28T22:35:30.372370+0000","last_fullsized":"2024-03-28T22:35:30.372370+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:50:04.238538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"94'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275105+0000","last_change":"2024-03-28T22:33:28.941546+0000","last_active":"2024-03-28T22:35:30.275105+0000","last_peered":"2024-03-28T22:35:30.275105+0000","last_clean":"2024-03-28T22:35:30.275105+0000","last_became_active":"2024-03-28T22:33:28.941278+0000","last_became_peered":"2024-03-28T22:33:28.941278+0000","last_unstale":"2024-03-28T22:35:30.275105+0000","last_undegraded":"2024-03-28T22:35:30.275105+0000","last_fullsized":"2024-03-28T22:35:30.275105+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:47:10.047067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"94'8","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.372207+0000","last_change":"2024-03-28T22:33:28.949552+0000","last_active":"2024-03-28T22:35:30.372207+0000","last_peered":"2024-03-28T22:35:30.372207+0000","last_clean":"2024-03-28T22:35:30.372207+0000","last_became_active":"2024-03-28T22:33:28.947432+0000","last_became_peered":"2024-03-28T22:33:28.947432+0000","last_unstale":"2024-03-28T22:35:30.372207+0000","last_undegraded":"2024-03-28T22:35:30.372207+0000","last_fullsized":"2024-03-28T22:35:30.372207+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:01:16.534642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"94'10","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448461+0000","last_change":"2024-03-28T22:33:28.953930+0000","last_active":"2024-03-28T22:35:30.448461+0000","last_peered":"2024-03-28T22:35:30.448461+0000","last_clean":"2024-03-28T22:35:30.448461+0000","last_became_active":"2024-03-28T22:33:28.953591+0000","last_became_peered":"2024-03-28T22:33:28.953591+0000","last_unstale":"2024-03-28T22:35:30.448461+0000","last_undegraded":"2024-03-28T22:35:30.448461+0000","last_fullsized":"2024-03-28T22:35:30.448461+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:42:55.857476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"94'11","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368740+0000","last_change":"2024-03-28T22:33:28.946307+0000","last_active":"2024-03-28T22:35:30.368740+0000","last_peered":"2024-03-28T22:35:30.368740+0000","last_clean":"2024-03-28T22:35:30.368740+0000","last_became_active":"2024-03-28T22:33:28.945932+0000","last_became_peered":"2024-03-28T22:33:28.945932+0000","last_unstale":"2024-03-28T22:35:30.368740+0000","last_undegraded":"2024-03-28T22:35:30.368740+0000","last_fullsized":"2024-03-28T22:35:30.368740+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:54:59.651668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"94'8","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449625+0000","last_change":"2024-03-28T22:33:28.928014+0000","last_active":"2024-03-28T22:35:30.449625+0000","last_peered":"2024-03-28T22:35:30.449625+0000","last_clean":"2024-03-28T22:35:30.449625+0000","last_became_active":"2024-03-28T22:33:28.927686+0000","last_became_peered":"2024-03-28T22:33:28.927686+0000","last_unstale":"2024-03-28T22:35:30.449625+0000","last_undegraded":"2024-03-28T22:35:30.449625+0000","last_fullsized":"2024-03-28T22:35:30.449625+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:47:08.131565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71409,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":934,"num_read_kb":547,"num_write":250,"num_write_kb":146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":214227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":217,"ondisk_log_size":217,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318312,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32260,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712892,"statfs":{"total":95995035648,"available":95962001408,"internally_reserved":0,"allocated":1105920,"data_stored":539556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579948,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28780,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716372,"statfs":{"total":95995035648,"available":95965564928,"internally_reserved":0,"allocated":1798144,"data_stored":1191337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743475,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1130496,"data_stored":543560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005112,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1118208,"data_stored":539820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299454,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37024,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708128,"statfs":{"total":95995035648,"available":95957123072,"internally_reserved":0,"allocated":1720320,"data_stored":1136480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495684,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1101824,"data_stored":534968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36488,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708664,"statfs":{"total":95995035648,"available":95957671936,"internally_reserved":0,"allocated":1171456,"data_stored":600453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738468,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1126400,"data_stored":536424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411420,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36508,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708644,"statfs":{"total":95995035648,"available":95957651456,"internally_reserved":0,"allocated":1191936,"data_stored":600673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182484,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36432,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708720,"statfs":{"total":95995035648,"available":95957729280,"internally_reserved":0,"allocated":1114112,"data_stored":534676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1110016,"data_stored":539574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757322,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28712,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716440,"statfs":{"total":95995035648,"available":95965634560,"internally_reserved":0,"allocated":1728512,"data_stored":1131530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":66207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":66427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":66723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T22:35:39.802 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph status --format=json 2024-03-28T22:35:40.328 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:35:41.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:41 smithi172 ceph-mon[36566]: pgmap v392: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:35:41.682 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:41 smithi043 ceph-mon[31823]: pgmap v392: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:35:41.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:41 smithi082 ceph-mon[36471]: pgmap v392: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:35:42.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.568+0000 7fdfbee00700 1 -- 172.21.15.43:0/1512882619 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdfb810a4c0 msgr2=0x7fdfb810a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:42.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.568+0000 7fdfbee00700 1 --2- 172.21.15.43:0/1512882619 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdfb810a4c0 0x7fdfb810a8a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8009990 tx=0x7fdfa8009ca0 comp rx=0 tx=0).stop 2024-03-28T22:35:42.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.568+0000 7fdfbee00700 1 -- 172.21.15.43:0/1512882619 shutdown_connections 2024-03-28T22:35:42.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.568+0000 7fdfbee00700 1 --2- 172.21.15.43:0/1512882619 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdfb8104df0 0x7fdfb8111300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:42.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.568+0000 7fdfbee00700 1 --2- 172.21.15.43:0/1512882619 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdfb810a4c0 0x7fdfb810a8a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:42.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.568+0000 7fdfbee00700 1 --2- 172.21.15.43:0/1512882619 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfb8104430 0x7fdfb81048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:42.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.568+0000 7fdfbee00700 1 -- 172.21.15.43:0/1512882619 >> 172.21.15.43:0/1512882619 conn(0x7fdfb80fff20 msgr2=0x7fdfb8102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:42.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.568+0000 7fdfbee00700 1 -- 172.21.15.43:0/1512882619 shutdown_connections 2024-03-28T22:35:42.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.569+0000 7fdfbee00700 1 -- 172.21.15.43:0/1512882619 wait complete. 2024-03-28T22:35:42.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.570+0000 7fdfbee00700 1 Processor -- start 2024-03-28T22:35:42.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.570+0000 7fdfbee00700 1 -- start start 2024-03-28T22:35:42.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.570+0000 7fdfbee00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfb8104430 0x7fdfb819ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:42.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.570+0000 7fdfbee00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdfb8104df0 0x7fdfb819b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:42.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.570+0000 7fdfbee00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdfb810a4c0 0x7fdfb819f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:42.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.570+0000 7fdfbee00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb8114720 con 0x7fdfb810a4c0 2024-03-28T22:35:42.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.570+0000 7fdfbee00700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb81145a0 con 0x7fdfb8104df0 2024-03-28T22:35:42.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.570+0000 7fdfbee00700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb81148a0 con 0x7fdfb8104430 2024-03-28T22:35:42.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.571+0000 7fdfb7e00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfb8104430 0x7fdfb819ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:42.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.571+0000 7fdfb7400700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdfb8104df0 0x7fdfb819b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:42.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.571+0000 7fdfb7e00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfb8104430 0x7fdfb819ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:33758/0 (socket says 172.21.15.43:33758) 2024-03-28T22:35:42.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.571+0000 7fdfb7e00700 1 -- 172.21.15.43:0/1555093190 learned_addr learned my addr 172.21.15.43:0/1555093190 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:35:42.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.571+0000 7fdfbce00700 1 --2- 172.21.15.43:0/1555093190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdfb810a4c0 0x7fdfb819f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:42.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.571+0000 7fdfb7e00700 1 -- 172.21.15.43:0/1555093190 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdfb8104df0 msgr2=0x7fdfb819b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:42.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.571+0000 7fdfb7e00700 1 --2- 172.21.15.43:0/1555093190 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdfb8104df0 0x7fdfb819b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:42.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.571+0000 7fdfb7e00700 1 -- 172.21.15.43:0/1555093190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdfb810a4c0 msgr2=0x7fdfb819f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:42.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.571+0000 7fdfb7e00700 1 --2- 172.21.15.43:0/1555093190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdfb810a4c0 0x7fdfb819f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:42.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.571+0000 7fdfb7e00700 1 -- 172.21.15.43:0/1555093190 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfb819fc10 con 0x7fdfb8104430 2024-03-28T22:35:42.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.571+0000 7fdfbce00700 1 --2- 172.21.15.43:0/1555093190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdfb810a4c0 0x7fdfb819f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:35:42.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.571+0000 7fdfb7400700 1 --2- 172.21.15.43:0/1555093190 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdfb8104df0 0x7fdfb819b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:35:42.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.571+0000 7fdfb7e00700 1 --2- 172.21.15.43:0/1555093190 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfb8104430 0x7fdfb819ac10 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdfa800bae0 tx=0x7fdfa800bb10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:42.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.572+0000 7fdfb4c00700 1 -- 172.21.15.43:0/1555093190 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa8026070 con 0x7fdfb8104430 2024-03-28T22:35:42.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.572+0000 7fdfb4c00700 1 -- 172.21.15.43:0/1555093190 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfa8003d70 con 0x7fdfb8104430 2024-03-28T22:35:42.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.572+0000 7fdfb4c00700 1 -- 172.21.15.43:0/1555093190 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa8021410 con 0x7fdfb8104430 2024-03-28T22:35:42.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.572+0000 7fdfbee00700 1 -- 172.21.15.43:0/1555093190 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfb819fea0 con 0x7fdfb8104430 2024-03-28T22:35:42.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.572+0000 7fdfbee00700 1 -- 172.21.15.43:0/1555093190 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfb81a0330 con 0x7fdfb8104430 2024-03-28T22:35:42.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.573+0000 7fdfbee00700 1 -- 172.21.15.43:0/1555093190 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfb8068f00 con 0x7fdfb8104430 2024-03-28T22:35:42.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.575+0000 7fdfb4c00700 1 -- 172.21.15.43:0/1555093190 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fdfa8029070 con 0x7fdfb8104430 2024-03-28T22:35:42.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.576+0000 7fdfb4c00700 1 --2- 172.21.15.43:0/1555093190 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fdf9c0a1e10 0x7fdf9c0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:42.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.577+0000 7fdfb4c00700 1 -- 172.21.15.43:0/1555093190 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7fdfa80ceef0 con 0x7fdfb8104430 2024-03-28T22:35:42.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.577+0000 7fdfb7400700 1 --2- 172.21.15.43:0/1555093190 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fdf9c0a1e10 0x7fdf9c0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:42.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.577+0000 7fdfb7400700 1 --2- 172.21.15.43:0/1555093190 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fdf9c0a1e10 0x7fdf9c0a42d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fdfb81097c0 tx=0x7fdfa4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:42.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.581+0000 7fdfb4c00700 1 -- 172.21.15.43:0/1555093190 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdfa805d280 con 0x7fdfb8104430 2024-03-28T22:35:42.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.898+0000 7fdfbee00700 1 -- 172.21.15.43:0/1555093190 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7fdfb802db30 con 0x7fdfb8104430 2024-03-28T22:35:42.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.898+0000 7fdfb4c00700 1 -- 172.21.15.43:0/1555093190 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1804 (secure 0 0 0) 0x7fdfa8013070 con 0x7fdfb8104430 2024-03-28T22:35:42.898 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:35:42.898 INFO:teuthology.orchestra.run.smithi043.stdout:{"fsid":"c2276428-ed51-11ee-95d0-87774f69a715","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":545,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":96,"num_osds":12,"num_up_osds":12,"osd_up_since":1711665151,"num_in_osds":12,"osd_in_since":1711665130,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":129}],"num_pgs":129,"num_pools":4,"num_objects":102,"data_bytes":662011,"bytes_used":385822720,"bytes_avail":1151554605056,"bytes_total":1151940427776,"read_bytes_sec":4793,"write_bytes_sec":171,"read_op_per_sec":9,"write_op_per_sec":0},"fsmap":{"epoch":28,"id":1,"up":5,"in":5,"max":5,"by_rank":[{"filesystem_id":1,"rank":4,"name":"k","status":"up:standby-replay","gid":24443},{"filesystem_id":1,"rank":3,"name":"l","status":"up:standby-replay","gid":24457},{"filesystem_id":1,"rank":2,"name":"j","status":"up:standby-replay","gid":24461},{"filesystem_id":1,"rank":1,"name":"d","status":"up:standby-replay","gid":24479},{"filesystem_id":1,"rank":0,"name":"e","status":"up:standby-replay","gid":24491},{"filesystem_id":1,"rank":4,"name":"h","status":"up:active","gid":24518},{"filesystem_id":1,"rank":3,"name":"f","status":"up:active","gid":24523},{"filesystem_id":1,"rank":2,"name":"c","status":"up:active","gid":24535},{"filesystem_id":1,"rank":1,"name":"i","status":"up:active","gid":24556},{"filesystem_id":1,"rank":0,"name":"b","status":"up:active","gid":24572}],"up:standby":2},"mgrmap":{"available":true,"num_standbys":2,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.43:8443/"}},"servicemap":{"epoch":31,"modified":"2024-03-28T22:35:14.197415+0000","services":{}},"progress_events":{}} 2024-03-28T22:35:42.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.901+0000 7fdfbee00700 1 -- 172.21.15.43:0/1555093190 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fdf9c0a1e10 msgr2=0x7fdf9c0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:42.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.901+0000 7fdfbee00700 1 --2- 172.21.15.43:0/1555093190 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fdf9c0a1e10 0x7fdf9c0a42d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fdfb81097c0 tx=0x7fdfa4009260 comp rx=0 tx=0).stop 2024-03-28T22:35:42.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.901+0000 7fdfbee00700 1 -- 172.21.15.43:0/1555093190 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfb8104430 msgr2=0x7fdfb819ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:42.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.901+0000 7fdfbee00700 1 --2- 172.21.15.43:0/1555093190 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfb8104430 0x7fdfb819ac10 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdfa800bae0 tx=0x7fdfa800bb10 comp rx=0 tx=0).stop 2024-03-28T22:35:42.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.902+0000 7fdfbee00700 1 -- 172.21.15.43:0/1555093190 shutdown_connections 2024-03-28T22:35:42.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.902+0000 7fdfbee00700 1 --2- 172.21.15.43:0/1555093190 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fdf9c0a1e10 0x7fdf9c0a42d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:42.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.902+0000 7fdfbee00700 1 --2- 172.21.15.43:0/1555093190 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdfb810a4c0 0x7fdfb819f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:42.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.902+0000 7fdfbee00700 1 --2- 172.21.15.43:0/1555093190 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfb8104430 0x7fdfb819ac10 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:42.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.902+0000 7fdfbee00700 1 --2- 172.21.15.43:0/1555093190 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdfb8104df0 0x7fdfb819b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:42.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.902+0000 7fdfbee00700 1 -- 172.21.15.43:0/1555093190 >> 172.21.15.43:0/1555093190 conn(0x7fdfb80fff20 msgr2=0x7fdfb8100440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:42.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.902+0000 7fdfbee00700 1 -- 172.21.15.43:0/1555093190 shutdown_connections 2024-03-28T22:35:42.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:42.902+0000 7fdfbee00700 1 -- 172.21.15.43:0/1555093190 wait complete. 2024-03-28T22:35:43.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:43 smithi172 ceph-mon[36566]: pgmap v393: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 171 B/s wr, 10 op/s 2024-03-28T22:35:43.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:43 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1555093190' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-28T22:35:43.676 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:43 smithi043 ceph-mon[31823]: pgmap v393: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 171 B/s wr, 10 op/s 2024-03-28T22:35:43.676 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:43 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1555093190' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-28T22:35:43.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:43 smithi082 ceph-mon[36471]: pgmap v393: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 171 B/s wr, 10 op/s 2024-03-28T22:35:43.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:43 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1555093190' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-28T22:35:43.748 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-28T22:35:43.748 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-28T22:35:44.209 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:35:45.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:45 smithi172 ceph-mon[36566]: pgmap v394: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-28T22:35:45.659 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:45 smithi043 ceph-mon[31823]: pgmap v394: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-28T22:35:45.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:45 smithi082 ceph-mon[36471]: pgmap v394: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-28T22:35:46.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.528+0000 7ffa71e00700 1 -- 172.21.15.43:0/70372213 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffa6c074d90 msgr2=0x7ffa6c10e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:46.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.528+0000 7ffa71e00700 1 --2- 172.21.15.43:0/70372213 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffa6c074d90 0x7ffa6c10e700 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ffa58009990 tx=0x7ffa58009ca0 comp rx=0 tx=0).stop 2024-03-28T22:35:46.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.528+0000 7ffa71e00700 1 -- 172.21.15.43:0/70372213 shutdown_connections 2024-03-28T22:35:46.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.528+0000 7ffa71e00700 1 --2- 172.21.15.43:0/70372213 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa6c10ec40 0x7ffa6c111030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:46.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.528+0000 7ffa71e00700 1 --2- 172.21.15.43:0/70372213 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa6c074470 0x7ffa6c074850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:46.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.528+0000 7ffa71e00700 1 --2- 172.21.15.43:0/70372213 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffa6c074d90 0x7ffa6c10e700 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:46.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.528+0000 7ffa71e00700 1 -- 172.21.15.43:0/70372213 >> 172.21.15.43:0/70372213 conn(0x7ffa6c079370 msgr2=0x7ffa6c079780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:46.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.528+0000 7ffa71e00700 1 -- 172.21.15.43:0/70372213 shutdown_connections 2024-03-28T22:35:46.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.528+0000 7ffa71e00700 1 -- 172.21.15.43:0/70372213 wait complete. 2024-03-28T22:35:46.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.529+0000 7ffa71e00700 1 Processor -- start 2024-03-28T22:35:46.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.530+0000 7ffa71e00700 1 -- start start 2024-03-28T22:35:46.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.530+0000 7ffa71e00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa6c074470 0x7ffa6c1a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:46.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.530+0000 7ffa71e00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffa6c074d90 0x7ffa6c1a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:46.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.530+0000 7ffa71e00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa6c10ec40 0x7ffa6c19a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:46.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.530+0000 7ffa71e00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ffa6c114710 con 0x7ffa6c10ec40 2024-03-28T22:35:46.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.530+0000 7ffa71e00700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ffa6c114590 con 0x7ffa6c074d90 2024-03-28T22:35:46.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.530+0000 7ffa71e00700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7ffa6c114890 con 0x7ffa6c074470 2024-03-28T22:35:46.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.530+0000 7ffa6aa00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa6c074470 0x7ffa6c1a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:46.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.530+0000 7ffa6a000700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffa6c074d90 0x7ffa6c1a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:46.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.530+0000 7ffa6aa00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa6c074470 0x7ffa6c1a0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:50480/0 (socket says 172.21.15.43:50480) 2024-03-28T22:35:46.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.530+0000 7ffa6aa00700 1 -- 172.21.15.43:0/1979630291 learned_addr learned my addr 172.21.15.43:0/1979630291 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:35:46.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.531+0000 7ffa6b400700 1 --2- 172.21.15.43:0/1979630291 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa6c10ec40 0x7ffa6c19a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:46.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.531+0000 7ffa6a000700 1 -- 172.21.15.43:0/1979630291 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa6c074470 msgr2=0x7ffa6c1a0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:46.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.531+0000 7ffa6a000700 1 --2- 172.21.15.43:0/1979630291 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa6c074470 0x7ffa6c1a0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:46.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.531+0000 7ffa6a000700 1 -- 172.21.15.43:0/1979630291 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa6c10ec40 msgr2=0x7ffa6c19a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:46.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.531+0000 7ffa6a000700 1 --2- 172.21.15.43:0/1979630291 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa6c10ec40 0x7ffa6c19a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:46.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.531+0000 7ffa6a000700 1 -- 172.21.15.43:0/1979630291 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa6c19b150 con 0x7ffa6c074d90 2024-03-28T22:35:46.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.531+0000 7ffa6a000700 1 --2- 172.21.15.43:0/1979630291 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffa6c074d90 0x7ffa6c1a0d70 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ffa58006010 tx=0x7ffa5800ba90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:46.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.531+0000 7ffa70a00700 1 -- 172.21.15.43:0/1979630291 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa58027070 con 0x7ffa6c074d90 2024-03-28T22:35:46.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.531+0000 7ffa70a00700 1 -- 172.21.15.43:0/1979630291 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa58022470 con 0x7ffa6c074d90 2024-03-28T22:35:46.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.531+0000 7ffa71e00700 1 -- 172.21.15.43:0/1979630291 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa6c19b3e0 con 0x7ffa6c074d90 2024-03-28T22:35:46.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.531+0000 7ffa70a00700 1 -- 172.21.15.43:0/1979630291 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa5800f6a0 con 0x7ffa6c074d90 2024-03-28T22:35:46.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.532+0000 7ffa71e00700 1 -- 172.21.15.43:0/1979630291 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa6c1a7700 con 0x7ffa6c074d90 2024-03-28T22:35:46.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.533+0000 7ffa71e00700 1 -- 172.21.15.43:0/1979630291 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa6c068f00 con 0x7ffa6c074d90 2024-03-28T22:35:46.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.534+0000 7ffa70a00700 1 -- 172.21.15.43:0/1979630291 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7ffa5802a070 con 0x7ffa6c074d90 2024-03-28T22:35:46.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.536+0000 7ffa70a00700 1 --2- 172.21.15.43:0/1979630291 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ffa500a1e90 0x7ffa500a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:46.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.536+0000 7ffa70a00700 1 -- 172.21.15.43:0/1979630291 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7ffa580cef10 con 0x7ffa6c074d90 2024-03-28T22:35:46.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.536+0000 7ffa6aa00700 1 --2- 172.21.15.43:0/1979630291 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ffa500a1e90 0x7ffa500a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:46.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.537+0000 7ffa6aa00700 1 --2- 172.21.15.43:0/1979630291 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ffa500a1e90 0x7ffa500a4350 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c19bb30 tx=0x7ffa5c00b480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:46.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.541+0000 7ffa70a00700 1 -- 172.21.15.43:0/1979630291 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ffa58013070 con 0x7ffa6c074d90 2024-03-28T22:35:46.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.763+0000 7ffa71e00700 1 -- 172.21.15.43:0/1979630291 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ffa6c19bf20 con 0x7ffa500a1e90 2024-03-28T22:35:46.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.793+0000 7ffa70a00700 1 -- 172.21.15.43:0/1979630291 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340517 (secure 0 0 0) 0x7ffa6c19bf20 con 0x7ffa500a1e90 2024-03-28T22:35:46.800 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:35:46.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.801+0000 7ffa71e00700 1 -- 172.21.15.43:0/1979630291 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ffa500a1e90 msgr2=0x7ffa500a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:46.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.801+0000 7ffa71e00700 1 --2- 172.21.15.43:0/1979630291 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ffa500a1e90 0x7ffa500a4350 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c19bb30 tx=0x7ffa5c00b480 comp rx=0 tx=0).stop 2024-03-28T22:35:46.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.801+0000 7ffa71e00700 1 -- 172.21.15.43:0/1979630291 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffa6c074d90 msgr2=0x7ffa6c1a0d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:46.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.801+0000 7ffa71e00700 1 --2- 172.21.15.43:0/1979630291 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffa6c074d90 0x7ffa6c1a0d70 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ffa58006010 tx=0x7ffa5800ba90 comp rx=0 tx=0).stop 2024-03-28T22:35:46.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.802+0000 7ffa71e00700 1 -- 172.21.15.43:0/1979630291 shutdown_connections 2024-03-28T22:35:46.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.802+0000 7ffa71e00700 1 --2- 172.21.15.43:0/1979630291 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ffa500a1e90 0x7ffa500a4350 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:46.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.802+0000 7ffa71e00700 1 --2- 172.21.15.43:0/1979630291 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa6c10ec40 0x7ffa6c19a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:46.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.802+0000 7ffa71e00700 1 --2- 172.21.15.43:0/1979630291 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa6c074470 0x7ffa6c1a0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:46.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.802+0000 7ffa71e00700 1 --2- 172.21.15.43:0/1979630291 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffa6c074d90 0x7ffa6c1a0d70 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:46.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.802+0000 7ffa71e00700 1 -- 172.21.15.43:0/1979630291 >> 172.21.15.43:0/1979630291 conn(0x7ffa6c079370 msgr2=0x7ffa6c0ff4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:46.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.802+0000 7ffa71e00700 1 -- 172.21.15.43:0/1979630291 shutdown_connections 2024-03-28T22:35:46.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:46.802+0000 7ffa71e00700 1 -- 172.21.15.43:0/1979630291 wait complete. 2024-03-28T22:35:46.804 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-03-28T22:35:47.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:47 smithi043 ceph-mon[31823]: pgmap v395: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T22:35:47.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:47 smithi043 ceph-mon[31823]: from='client.25061 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:35:47.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:47 smithi172 ceph-mon[36566]: pgmap v395: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T22:35:47.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:47 smithi172 ceph-mon[36566]: from='client.25061 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:35:47.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:47 smithi082 ceph-mon[36471]: pgmap v395: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T22:35:47.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:47 smithi082 ceph-mon[36471]: from='client.25061 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:35:47.708 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":395,"stamp":"2024-03-28T22:35:46.211028+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662011,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1150,"num_read_kb":706,"num_write":433,"num_write_kb":3086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":856,"ondisk_log_size":856,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":376780,"kb_used_data":15056,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124565044,"statfs":{"total":1151940427776,"available":1151554605056,"internally_reserved":0,"allocated":15417344,"data_stored":8429051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":120,"num_read_kb":65,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005014"},"pg_stats":[{"pgid":"3.3f","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.450073+0000","last_change":"2024-03-28T22:33:28.942904+0000","last_active":"2024-03-28T22:35:30.450073+0000","last_peered":"2024-03-28T22:35:30.450073+0000","last_clean":"2024-03-28T22:35:30.450073+0000","last_became_active":"2024-03-28T22:33:28.942574+0000","last_became_peered":"2024-03-28T22:33:28.942574+0000","last_unstale":"2024-03-28T22:35:30.450073+0000","last_undegraded":"2024-03-28T22:35:30.450073+0000","last_fullsized":"2024-03-28T22:35:30.450073+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:55.323919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"94'4","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451519+0000","last_change":"2024-03-28T22:33:28.951795+0000","last_active":"2024-03-28T22:35:30.451519+0000","last_peered":"2024-03-28T22:35:30.451519+0000","last_clean":"2024-03-28T22:35:30.451519+0000","last_became_active":"2024-03-28T22:33:28.951286+0000","last_became_peered":"2024-03-28T22:33:28.951286+0000","last_unstale":"2024-03-28T22:35:30.451519+0000","last_undegraded":"2024-03-28T22:35:30.451519+0000","last_fullsized":"2024-03-28T22:35:30.451519+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:56:11.718244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451291+0000","last_change":"2024-03-28T22:33:28.949476+0000","last_active":"2024-03-28T22:35:30.451291+0000","last_peered":"2024-03-28T22:35:30.451291+0000","last_clean":"2024-03-28T22:35:30.451291+0000","last_became_active":"2024-03-28T22:33:28.949128+0000","last_became_peered":"2024-03-28T22:33:28.949128+0000","last_unstale":"2024-03-28T22:35:30.451291+0000","last_undegraded":"2024-03-28T22:35:30.451291+0000","last_fullsized":"2024-03-28T22:35:30.451291+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:49:29.052373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370703+0000","last_change":"2024-03-28T22:33:28.930715+0000","last_active":"2024-03-28T22:35:30.370703+0000","last_peered":"2024-03-28T22:35:30.370703+0000","last_clean":"2024-03-28T22:35:30.370703+0000","last_became_active":"2024-03-28T22:33:28.930265+0000","last_became_peered":"2024-03-28T22:33:28.930265+0000","last_unstale":"2024-03-28T22:35:30.370703+0000","last_undegraded":"2024-03-28T22:35:30.370703+0000","last_fullsized":"2024-03-28T22:35:30.370703+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:29:54.695671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"94'4","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.253628+0000","last_change":"2024-03-28T22:33:28.944933+0000","last_active":"2024-03-28T22:35:30.253628+0000","last_peered":"2024-03-28T22:35:30.253628+0000","last_clean":"2024-03-28T22:35:30.253628+0000","last_became_active":"2024-03-28T22:33:28.944542+0000","last_became_peered":"2024-03-28T22:33:28.944542+0000","last_unstale":"2024-03-28T22:35:30.253628+0000","last_undegraded":"2024-03-28T22:35:30.253628+0000","last_fullsized":"2024-03-28T22:35:30.253628+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:35:15.719312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"94'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.259239+0000","last_change":"2024-03-28T22:33:28.927920+0000","last_active":"2024-03-28T22:35:30.259239+0000","last_peered":"2024-03-28T22:35:30.259239+0000","last_clean":"2024-03-28T22:35:30.259239+0000","last_became_active":"2024-03-28T22:33:28.927602+0000","last_became_peered":"2024-03-28T22:33:28.927602+0000","last_unstale":"2024-03-28T22:35:30.259239+0000","last_undegraded":"2024-03-28T22:35:30.259239+0000","last_fullsized":"2024-03-28T22:35:30.259239+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:24:22.072707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"94'5","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248009+0000","last_change":"2024-03-28T22:33:28.917963+0000","last_active":"2024-03-28T22:35:30.248009+0000","last_peered":"2024-03-28T22:35:30.248009+0000","last_clean":"2024-03-28T22:35:30.248009+0000","last_became_active":"2024-03-28T22:33:28.917628+0000","last_became_peered":"2024-03-28T22:33:28.917628+0000","last_unstale":"2024-03-28T22:35:30.248009+0000","last_undegraded":"2024-03-28T22:35:30.248009+0000","last_fullsized":"2024-03-28T22:35:30.248009+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:37:09.111640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"94'9","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371107+0000","last_change":"2024-03-28T22:33:28.942632+0000","last_active":"2024-03-28T22:35:30.371107+0000","last_peered":"2024-03-28T22:35:30.371107+0000","last_clean":"2024-03-28T22:35:30.371107+0000","last_became_active":"2024-03-28T22:33:28.941972+0000","last_became_peered":"2024-03-28T22:33:28.941972+0000","last_unstale":"2024-03-28T22:35:30.371107+0000","last_undegraded":"2024-03-28T22:35:30.371107+0000","last_fullsized":"2024-03-28T22:35:30.371107+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:49:52.642314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"94'11","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448719+0000","last_change":"2024-03-28T22:33:28.924324+0000","last_active":"2024-03-28T22:35:30.448719+0000","last_peered":"2024-03-28T22:35:30.448719+0000","last_clean":"2024-03-28T22:35:30.448719+0000","last_became_active":"2024-03-28T22:33:28.924014+0000","last_became_peered":"2024-03-28T22:33:28.924014+0000","last_unstale":"2024-03-28T22:35:30.448719+0000","last_undegraded":"2024-03-28T22:35:30.448719+0000","last_fullsized":"2024-03-28T22:35:30.448719+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:17:55.172116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"94'16","reported_seq":53,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.259531+0000","last_change":"2024-03-28T22:34:29.987262+0000","last_active":"2024-03-28T22:35:30.259531+0000","last_peered":"2024-03-28T22:35:30.259531+0000","last_clean":"2024-03-28T22:35:30.259531+0000","last_became_active":"2024-03-28T22:34:29.986934+0000","last_became_peered":"2024-03-28T22:34:29.986934+0000","last_unstale":"2024-03-28T22:35:30.259531+0000","last_undegraded":"2024-03-28T22:35:30.259531+0000","last_fullsized":"2024-03-28T22:35:30.259531+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:00:41.574800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"94'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248464+0000","last_change":"2024-03-28T22:33:28.930802+0000","last_active":"2024-03-28T22:35:30.248464+0000","last_peered":"2024-03-28T22:35:30.248464+0000","last_clean":"2024-03-28T22:35:30.248464+0000","last_became_active":"2024-03-28T22:33:28.930475+0000","last_became_peered":"2024-03-28T22:33:28.930475+0000","last_unstale":"2024-03-28T22:35:30.248464+0000","last_undegraded":"2024-03-28T22:35:30.248464+0000","last_fullsized":"2024-03-28T22:35:30.248464+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:48.443803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"94'7","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370644+0000","last_change":"2024-03-28T22:33:28.927057+0000","last_active":"2024-03-28T22:35:30.370644+0000","last_peered":"2024-03-28T22:35:30.370644+0000","last_clean":"2024-03-28T22:35:30.370644+0000","last_became_active":"2024-03-28T22:33:28.926598+0000","last_became_peered":"2024-03-28T22:33:28.926598+0000","last_unstale":"2024-03-28T22:35:30.370644+0000","last_undegraded":"2024-03-28T22:35:30.370644+0000","last_fullsized":"2024-03-28T22:35:30.370644+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:21:00.710203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"94'10","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368883+0000","last_change":"2024-03-28T22:33:28.937034+0000","last_active":"2024-03-28T22:35:30.368883+0000","last_peered":"2024-03-28T22:35:30.368883+0000","last_clean":"2024-03-28T22:35:30.368883+0000","last_became_active":"2024-03-28T22:33:28.933713+0000","last_became_peered":"2024-03-28T22:33:28.933713+0000","last_unstale":"2024-03-28T22:35:30.368883+0000","last_undegraded":"2024-03-28T22:35:30.368883+0000","last_fullsized":"2024-03-28T22:35:30.368883+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:58:24.148897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"94'7","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.450910+0000","last_change":"2024-03-28T22:33:28.947384+0000","last_active":"2024-03-28T22:35:30.450910+0000","last_peered":"2024-03-28T22:35:30.450910+0000","last_clean":"2024-03-28T22:35:30.450910+0000","last_became_active":"2024-03-28T22:33:28.947017+0000","last_became_peered":"2024-03-28T22:33:28.947017+0000","last_unstale":"2024-03-28T22:35:30.450910+0000","last_undegraded":"2024-03-28T22:35:30.450910+0000","last_fullsized":"2024-03-28T22:35:30.450910+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:43:32.781161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"94'7","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451705+0000","last_change":"2024-03-28T22:33:28.950119+0000","last_active":"2024-03-28T22:35:30.451705+0000","last_peered":"2024-03-28T22:35:30.451705+0000","last_clean":"2024-03-28T22:35:30.451705+0000","last_became_active":"2024-03-28T22:33:28.949798+0000","last_became_peered":"2024-03-28T22:33:28.949798+0000","last_unstale":"2024-03-28T22:35:30.451705+0000","last_undegraded":"2024-03-28T22:35:30.451705+0000","last_fullsized":"2024-03-28T22:35:30.451705+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:16.717678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"94'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448256+0000","last_change":"2024-03-28T22:33:28.943255+0000","last_active":"2024-03-28T22:35:30.448256+0000","last_peered":"2024-03-28T22:35:30.448256+0000","last_clean":"2024-03-28T22:35:30.448256+0000","last_became_active":"2024-03-28T22:33:28.942923+0000","last_became_peered":"2024-03-28T22:33:28.942923+0000","last_unstale":"2024-03-28T22:35:30.448256+0000","last_undegraded":"2024-03-28T22:35:30.448256+0000","last_fullsized":"2024-03-28T22:35:30.448256+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:05:04.197635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"94'8","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449920+0000","last_change":"2024-03-28T22:33:28.942906+0000","last_active":"2024-03-28T22:35:30.449920+0000","last_peered":"2024-03-28T22:35:30.449920+0000","last_clean":"2024-03-28T22:35:30.449920+0000","last_became_active":"2024-03-28T22:33:28.942573+0000","last_became_peered":"2024-03-28T22:33:28.942573+0000","last_unstale":"2024-03-28T22:35:30.449920+0000","last_undegraded":"2024-03-28T22:35:30.449920+0000","last_fullsized":"2024-03-28T22:35:30.449920+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:39:17.651101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"94'10","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252793+0000","last_change":"2024-03-28T22:33:28.932075+0000","last_active":"2024-03-28T22:35:30.252793+0000","last_peered":"2024-03-28T22:35:30.252793+0000","last_clean":"2024-03-28T22:35:30.252793+0000","last_became_active":"2024-03-28T22:33:28.931689+0000","last_became_peered":"2024-03-28T22:33:28.931689+0000","last_unstale":"2024-03-28T22:35:30.252793+0000","last_undegraded":"2024-03-28T22:35:30.252793+0000","last_fullsized":"2024-03-28T22:35:30.252793+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:50:43.031082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252343+0000","last_change":"2024-03-28T22:33:28.944987+0000","last_active":"2024-03-28T22:35:30.252343+0000","last_peered":"2024-03-28T22:35:30.252343+0000","last_clean":"2024-03-28T22:35:30.252343+0000","last_became_active":"2024-03-28T22:33:28.944622+0000","last_became_peered":"2024-03-28T22:33:28.944622+0000","last_unstale":"2024-03-28T22:35:30.252343+0000","last_undegraded":"2024-03-28T22:35:30.252343+0000","last_fullsized":"2024-03-28T22:35:30.252343+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:48:58.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":29,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251775+0000","last_change":"2024-03-28T22:34:30.429709+0000","last_active":"2024-03-28T22:35:30.251775+0000","last_peered":"2024-03-28T22:35:30.251775+0000","last_clean":"2024-03-28T22:35:30.251775+0000","last_became_active":"2024-03-28T22:34:30.429122+0000","last_became_peered":"2024-03-28T22:34:30.429122+0000","last_unstale":"2024-03-28T22:35:30.251775+0000","last_undegraded":"2024-03-28T22:35:30.251775+0000","last_fullsized":"2024-03-28T22:35:30.251775+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:58:05.313669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"92'13","reported_seq":144,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:40.334742+0000","last_change":"2024-03-28T22:33:28.442524+0000","last_active":"2024-03-28T22:35:40.334742+0000","last_peered":"2024-03-28T22:35:40.334742+0000","last_clean":"2024-03-28T22:35:40.334742+0000","last_became_active":"2024-03-28T22:33:28.442169+0000","last_became_peered":"2024-03-28T22:33:28.442169+0000","last_unstale":"2024-03-28T22:35:40.334742+0000","last_undegraded":"2024-03-28T22:35:40.334742+0000","last_fullsized":"2024-03-28T22:35:40.334742+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:08:47.101119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":97,"num_read_kb":97,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"94'5","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371822+0000","last_change":"2024-03-28T22:33:28.946697+0000","last_active":"2024-03-28T22:35:30.371822+0000","last_peered":"2024-03-28T22:35:30.371822+0000","last_clean":"2024-03-28T22:35:30.371822+0000","last_became_active":"2024-03-28T22:33:28.946303+0000","last_became_peered":"2024-03-28T22:33:28.946303+0000","last_unstale":"2024-03-28T22:35:30.371822+0000","last_undegraded":"2024-03-28T22:35:30.371822+0000","last_fullsized":"2024-03-28T22:35:30.371822+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:32:01.286073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370317+0000","last_change":"2024-03-28T22:34:30.429579+0000","last_active":"2024-03-28T22:35:30.370317+0000","last_peered":"2024-03-28T22:35:30.370317+0000","last_clean":"2024-03-28T22:35:30.370317+0000","last_became_active":"2024-03-28T22:34:30.429054+0000","last_became_peered":"2024-03-28T22:34:30.429054+0000","last_unstale":"2024-03-28T22:35:30.370317+0000","last_undegraded":"2024-03-28T22:35:30.370317+0000","last_fullsized":"2024-03-28T22:35:30.370317+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:03:46.492786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"92'2","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.372174+0000","last_change":"2024-03-28T22:33:28.457661+0000","last_active":"2024-03-28T22:35:30.372174+0000","last_peered":"2024-03-28T22:35:30.372174+0000","last_clean":"2024-03-28T22:35:30.372174+0000","last_became_active":"2024-03-28T22:33:28.457032+0000","last_became_peered":"2024-03-28T22:33:28.457032+0000","last_unstale":"2024-03-28T22:35:30.372174+0000","last_undegraded":"2024-03-28T22:35:30.372174+0000","last_fullsized":"2024-03-28T22:35:30.372174+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:43:09.820630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"94'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.453641+0000","last_change":"2024-03-28T22:33:28.954252+0000","last_active":"2024-03-28T22:35:30.453641+0000","last_peered":"2024-03-28T22:35:30.453641+0000","last_clean":"2024-03-28T22:35:30.453641+0000","last_became_active":"2024-03-28T22:33:28.953922+0000","last_became_peered":"2024-03-28T22:33:28.953922+0000","last_unstale":"2024-03-28T22:35:30.453641+0000","last_undegraded":"2024-03-28T22:35:30.453641+0000","last_fullsized":"2024-03-28T22:35:30.453641+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:18:47.097859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370557+0000","last_change":"2024-03-28T22:33:34.029129+0000","last_active":"2024-03-28T22:35:30.370557+0000","last_peered":"2024-03-28T22:35:30.370557+0000","last_clean":"2024-03-28T22:35:30.370557+0000","last_became_active":"2024-03-28T22:33:34.028452+0000","last_became_peered":"2024-03-28T22:33:34.028452+0000","last_unstale":"2024-03-28T22:35:30.370557+0000","last_undegraded":"2024-03-28T22:35:30.370557+0000","last_fullsized":"2024-03-28T22:35:30.370557+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:28.982773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"92'12","reported_seq":46,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254159+0000","last_change":"2024-03-28T22:33:28.439568+0000","last_active":"2024-03-28T22:35:30.254159+0000","last_peered":"2024-03-28T22:35:30.254159+0000","last_clean":"2024-03-28T22:35:30.254159+0000","last_became_active":"2024-03-28T22:33:28.438887+0000","last_became_peered":"2024-03-28T22:33:28.438887+0000","last_unstale":"2024-03-28T22:35:30.254159+0000","last_undegraded":"2024-03-28T22:35:30.254159+0000","last_fullsized":"2024-03-28T22:35:30.254159+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:56:43.617621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"94'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448533+0000","last_change":"2024-03-28T22:33:28.938311+0000","last_active":"2024-03-28T22:35:30.448533+0000","last_peered":"2024-03-28T22:35:30.448533+0000","last_clean":"2024-03-28T22:35:30.448533+0000","last_became_active":"2024-03-28T22:33:28.937997+0000","last_became_peered":"2024-03-28T22:33:28.937997+0000","last_unstale":"2024-03-28T22:35:30.448533+0000","last_undegraded":"2024-03-28T22:35:30.448533+0000","last_fullsized":"2024-03-28T22:35:30.448533+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:54:44.001545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370378+0000","last_change":"2024-03-28T22:33:34.016931+0000","last_active":"2024-03-28T22:35:30.370378+0000","last_peered":"2024-03-28T22:35:30.370378+0000","last_clean":"2024-03-28T22:35:30.370378+0000","last_became_active":"2024-03-28T22:33:34.016559+0000","last_became_peered":"2024-03-28T22:33:34.016559+0000","last_unstale":"2024-03-28T22:35:30.370378+0000","last_undegraded":"2024-03-28T22:35:30.370378+0000","last_fullsized":"2024-03-28T22:35:30.370378+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:08:31.252351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"92'5","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254044+0000","last_change":"2024-03-28T22:33:28.433166+0000","last_active":"2024-03-28T22:35:30.254044+0000","last_peered":"2024-03-28T22:35:30.254044+0000","last_clean":"2024-03-28T22:35:30.254044+0000","last_became_active":"2024-03-28T22:33:28.432716+0000","last_became_peered":"2024-03-28T22:33:28.432716+0000","last_unstale":"2024-03-28T22:35:30.254044+0000","last_undegraded":"2024-03-28T22:35:30.254044+0000","last_fullsized":"2024-03-28T22:35:30.254044+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:42:41.613583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"94'15","reported_seq":46,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275481+0000","last_change":"2024-03-28T22:33:28.923307+0000","last_active":"2024-03-28T22:35:30.275481+0000","last_peered":"2024-03-28T22:35:30.275481+0000","last_clean":"2024-03-28T22:35:30.275481+0000","last_became_active":"2024-03-28T22:33:28.922499+0000","last_became_peered":"2024-03-28T22:33:28.922499+0000","last_unstale":"2024-03-28T22:35:30.275481+0000","last_undegraded":"2024-03-28T22:35:30.275481+0000","last_fullsized":"2024-03-28T22:35:30.275481+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:48:48.514482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252136+0000","last_change":"2024-03-28T22:33:34.002209+0000","last_active":"2024-03-28T22:35:30.252136+0000","last_peered":"2024-03-28T22:35:30.252136+0000","last_clean":"2024-03-28T22:35:30.252136+0000","last_became_active":"2024-03-28T22:33:34.001746+0000","last_became_peered":"2024-03-28T22:33:34.001746+0000","last_unstale":"2024-03-28T22:35:30.252136+0000","last_undegraded":"2024-03-28T22:35:30.252136+0000","last_fullsized":"2024-03-28T22:35:30.252136+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:02:02.333328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"92'5","reported_seq":136,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:40.353784+0000","last_change":"2024-03-28T22:33:28.435380+0000","last_active":"2024-03-28T22:35:40.353784+0000","last_peered":"2024-03-28T22:35:40.353784+0000","last_clean":"2024-03-28T22:35:40.353784+0000","last_became_active":"2024-03-28T22:33:28.434898+0000","last_became_peered":"2024-03-28T22:33:28.434898+0000","last_unstale":"2024-03-28T22:35:40.353784+0000","last_undegraded":"2024-03-28T22:35:40.353784+0000","last_fullsized":"2024-03-28T22:35:40.353784+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:58:06.810394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":98,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"94'7","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254211+0000","last_change":"2024-03-28T22:34:30.121370+0000","last_active":"2024-03-28T22:35:30.254211+0000","last_peered":"2024-03-28T22:35:30.254211+0000","last_clean":"2024-03-28T22:35:30.254211+0000","last_became_active":"2024-03-28T22:34:30.121086+0000","last_became_peered":"2024-03-28T22:34:30.121086+0000","last_unstale":"2024-03-28T22:35:30.254211+0000","last_undegraded":"2024-03-28T22:35:30.254211+0000","last_fullsized":"2024-03-28T22:35:30.254211+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:26:24.244663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252008+0000","last_change":"2024-03-28T22:33:34.031350+0000","last_active":"2024-03-28T22:35:30.252008+0000","last_peered":"2024-03-28T22:35:30.252008+0000","last_clean":"2024-03-28T22:35:30.252008+0000","last_became_active":"2024-03-28T22:33:34.030946+0000","last_became_peered":"2024-03-28T22:33:34.030946+0000","last_unstale":"2024-03-28T22:35:30.252008+0000","last_undegraded":"2024-03-28T22:35:30.252008+0000","last_fullsized":"2024-03-28T22:35:30.252008+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:54:22.907363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"92'7","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.259165+0000","last_change":"2024-03-28T22:33:28.446197+0000","last_active":"2024-03-28T22:35:30.259165+0000","last_peered":"2024-03-28T22:35:30.259165+0000","last_clean":"2024-03-28T22:35:30.259165+0000","last_became_active":"2024-03-28T22:33:28.445891+0000","last_became_peered":"2024-03-28T22:33:28.445891+0000","last_unstale":"2024-03-28T22:35:30.259165+0000","last_undegraded":"2024-03-28T22:35:30.259165+0000","last_fullsized":"2024-03-28T22:35:30.259165+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:58:57.385302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275539+0000","last_change":"2024-03-28T22:33:28.955651+0000","last_active":"2024-03-28T22:35:30.275539+0000","last_peered":"2024-03-28T22:35:30.275539+0000","last_clean":"2024-03-28T22:35:30.275539+0000","last_became_active":"2024-03-28T22:33:28.955276+0000","last_became_peered":"2024-03-28T22:33:28.955276+0000","last_unstale":"2024-03-28T22:35:30.275539+0000","last_undegraded":"2024-03-28T22:35:30.275539+0000","last_fullsized":"2024-03-28T22:35:30.275539+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:36:13.455327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449809+0000","last_change":"2024-03-28T22:33:34.017520+0000","last_active":"2024-03-28T22:35:30.449809+0000","last_peered":"2024-03-28T22:35:30.449809+0000","last_clean":"2024-03-28T22:35:30.449809+0000","last_became_active":"2024-03-28T22:33:34.017022+0000","last_became_peered":"2024-03-28T22:33:34.017022+0000","last_unstale":"2024-03-28T22:35:30.449809+0000","last_undegraded":"2024-03-28T22:35:30.449809+0000","last_fullsized":"2024-03-28T22:35:30.449809+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:41:17.431581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"92'3","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368118+0000","last_change":"2024-03-28T22:33:28.445401+0000","last_active":"2024-03-28T22:35:30.368118+0000","last_peered":"2024-03-28T22:35:30.368118+0000","last_clean":"2024-03-28T22:35:30.368118+0000","last_became_active":"2024-03-28T22:33:28.444237+0000","last_became_peered":"2024-03-28T22:33:28.444237+0000","last_unstale":"2024-03-28T22:35:30.368118+0000","last_undegraded":"2024-03-28T22:35:30.368118+0000","last_fullsized":"2024-03-28T22:35:30.368118+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:18:12.820890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"94'8","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.250889+0000","last_change":"2024-03-28T22:34:30.156248+0000","last_active":"2024-03-28T22:35:30.250889+0000","last_peered":"2024-03-28T22:35:30.250889+0000","last_clean":"2024-03-28T22:35:30.250889+0000","last_became_active":"2024-03-28T22:34:30.155056+0000","last_became_peered":"2024-03-28T22:34:30.155056+0000","last_unstale":"2024-03-28T22:35:30.250889+0000","last_undegraded":"2024-03-28T22:35:30.250889+0000","last_fullsized":"2024-03-28T22:35:30.250889+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:28:56.469728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451480+0000","last_change":"2024-03-28T22:33:34.019899+0000","last_active":"2024-03-28T22:35:30.451480+0000","last_peered":"2024-03-28T22:35:30.451480+0000","last_clean":"2024-03-28T22:35:30.451480+0000","last_became_active":"2024-03-28T22:33:34.019392+0000","last_became_peered":"2024-03-28T22:33:34.019392+0000","last_unstale":"2024-03-28T22:35:30.451480+0000","last_undegraded":"2024-03-28T22:35:30.451480+0000","last_fullsized":"2024-03-28T22:35:30.451480+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:08:38.633797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"92'3","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275054+0000","last_change":"2024-03-28T22:33:28.433243+0000","last_active":"2024-03-28T22:35:30.275054+0000","last_peered":"2024-03-28T22:35:30.275054+0000","last_clean":"2024-03-28T22:35:30.275054+0000","last_became_active":"2024-03-28T22:33:28.432927+0000","last_became_peered":"2024-03-28T22:33:28.432927+0000","last_unstale":"2024-03-28T22:35:30.275054+0000","last_undegraded":"2024-03-28T22:35:30.275054+0000","last_fullsized":"2024-03-28T22:35:30.275054+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:40:43.444229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"94'11","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251179+0000","last_change":"2024-03-28T22:33:28.948111+0000","last_active":"2024-03-28T22:35:30.251179+0000","last_peered":"2024-03-28T22:35:30.251179+0000","last_clean":"2024-03-28T22:35:30.251179+0000","last_became_active":"2024-03-28T22:33:28.944649+0000","last_became_peered":"2024-03-28T22:33:28.944649+0000","last_unstale":"2024-03-28T22:35:30.251179+0000","last_undegraded":"2024-03-28T22:35:30.251179+0000","last_fullsized":"2024-03-28T22:35:30.251179+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:57:11.483037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371445+0000","last_change":"2024-03-28T22:33:34.000837+0000","last_active":"2024-03-28T22:35:30.371445+0000","last_peered":"2024-03-28T22:35:30.371445+0000","last_clean":"2024-03-28T22:35:30.371445+0000","last_became_active":"2024-03-28T22:33:34.000120+0000","last_became_peered":"2024-03-28T22:33:34.000120+0000","last_unstale":"2024-03-28T22:35:30.371445+0000","last_undegraded":"2024-03-28T22:35:30.371445+0000","last_fullsized":"2024-03-28T22:35:30.371445+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:14:32.028952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"92'4","reported_seq":235,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:44.286771+0000","last_change":"2024-03-28T22:33:28.436985+0000","last_active":"2024-03-28T22:35:44.286771+0000","last_peered":"2024-03-28T22:35:44.286771+0000","last_clean":"2024-03-28T22:35:44.286771+0000","last_became_active":"2024-03-28T22:33:28.436625+0000","last_became_peered":"2024-03-28T22:33:28.436625+0000","last_unstale":"2024-03-28T22:35:44.286771+0000","last_undegraded":"2024-03-28T22:35:44.286771+0000","last_fullsized":"2024-03-28T22:35:44.286771+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:50:45.775191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":198,"num_read_kb":198,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"94'10","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448868+0000","last_change":"2024-03-28T22:33:28.933265+0000","last_active":"2024-03-28T22:35:30.448868+0000","last_peered":"2024-03-28T22:35:30.448868+0000","last_clean":"2024-03-28T22:35:30.448868+0000","last_became_active":"2024-03-28T22:33:28.932954+0000","last_became_peered":"2024-03-28T22:33:28.932954+0000","last_unstale":"2024-03-28T22:35:30.448868+0000","last_undegraded":"2024-03-28T22:35:30.448868+0000","last_fullsized":"2024-03-28T22:35:30.448868+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:37.039055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251115+0000","last_change":"2024-03-28T22:33:34.024529+0000","last_active":"2024-03-28T22:35:30.251115+0000","last_peered":"2024-03-28T22:35:30.251115+0000","last_clean":"2024-03-28T22:35:30.251115+0000","last_became_active":"2024-03-28T22:33:34.020949+0000","last_became_peered":"2024-03-28T22:33:34.020949+0000","last_unstale":"2024-03-28T22:35:30.251115+0000","last_undegraded":"2024-03-28T22:35:30.251115+0000","last_fullsized":"2024-03-28T22:35:30.251115+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:28:52.039684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"92'4","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248336+0000","last_change":"2024-03-28T22:33:28.432599+0000","last_active":"2024-03-28T22:35:30.248336+0000","last_peered":"2024-03-28T22:35:30.248336+0000","last_clean":"2024-03-28T22:35:30.248336+0000","last_became_active":"2024-03-28T22:33:28.430710+0000","last_became_peered":"2024-03-28T22:33:28.430710+0000","last_unstale":"2024-03-28T22:35:30.248336+0000","last_undegraded":"2024-03-28T22:35:30.248336+0000","last_fullsized":"2024-03-28T22:35:30.248336+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:54.174747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"94'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.453785+0000","last_change":"2024-03-28T22:33:28.955441+0000","last_active":"2024-03-28T22:35:30.453785+0000","last_peered":"2024-03-28T22:35:30.453785+0000","last_clean":"2024-03-28T22:35:30.453785+0000","last_became_active":"2024-03-28T22:33:28.953269+0000","last_became_peered":"2024-03-28T22:33:28.953269+0000","last_unstale":"2024-03-28T22:35:30.453785+0000","last_undegraded":"2024-03-28T22:35:30.453785+0000","last_fullsized":"2024-03-28T22:35:30.453785+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:47:04.128605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370663+0000","last_change":"2024-03-28T22:33:34.028791+0000","last_active":"2024-03-28T22:35:30.370663+0000","last_peered":"2024-03-28T22:35:30.370663+0000","last_clean":"2024-03-28T22:35:30.370663+0000","last_became_active":"2024-03-28T22:33:34.028361+0000","last_became_peered":"2024-03-28T22:33:34.028361+0000","last_unstale":"2024-03-28T22:35:30.370663+0000","last_undegraded":"2024-03-28T22:35:30.370663+0000","last_fullsized":"2024-03-28T22:35:30.370663+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:42:21.264459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"92'5","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275230+0000","last_change":"2024-03-28T22:33:28.455353+0000","last_active":"2024-03-28T22:35:30.275230+0000","last_peered":"2024-03-28T22:35:30.275230+0000","last_clean":"2024-03-28T22:35:30.275230+0000","last_became_active":"2024-03-28T22:33:28.455090+0000","last_became_peered":"2024-03-28T22:33:28.455090+0000","last_unstale":"2024-03-28T22:35:30.275230+0000","last_undegraded":"2024-03-28T22:35:30.275230+0000","last_fullsized":"2024-03-28T22:35:30.275230+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:19:22.496600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"94'11","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448676+0000","last_change":"2024-03-28T22:33:28.928875+0000","last_active":"2024-03-28T22:35:30.448676+0000","last_peered":"2024-03-28T22:35:30.448676+0000","last_clean":"2024-03-28T22:35:30.448676+0000","last_became_active":"2024-03-28T22:33:28.928449+0000","last_became_peered":"2024-03-28T22:33:28.928449+0000","last_unstale":"2024-03-28T22:35:30.448676+0000","last_undegraded":"2024-03-28T22:35:30.448676+0000","last_fullsized":"2024-03-28T22:35:30.448676+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:34:00.402100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251201+0000","last_change":"2024-03-28T22:33:34.012739+0000","last_active":"2024-03-28T22:35:30.251201+0000","last_peered":"2024-03-28T22:35:30.251201+0000","last_clean":"2024-03-28T22:35:30.251201+0000","last_became_active":"2024-03-28T22:33:34.012262+0000","last_became_peered":"2024-03-28T22:33:34.012262+0000","last_unstale":"2024-03-28T22:35:30.251201+0000","last_undegraded":"2024-03-28T22:35:30.251201+0000","last_fullsized":"2024-03-28T22:35:30.251201+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:24:48.351037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'38","reported_seq":276,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:44.287347+0000","last_change":"2024-03-28T22:33:28.430606+0000","last_active":"2024-03-28T22:35:44.287347+0000","last_peered":"2024-03-28T22:35:44.287347+0000","last_clean":"2024-03-28T22:35:44.287347+0000","last_became_active":"2024-03-28T22:33:28.430233+0000","last_became_peered":"2024-03-28T22:33:28.430233+0000","last_unstale":"2024-03-28T22:35:44.287347+0000","last_undegraded":"2024-03-28T22:35:44.287347+0000","last_fullsized":"2024-03-28T22:35:44.287347+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:49.000569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64455,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":205,"num_read_kb":166,"num_write":40,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"94'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254029+0000","last_change":"2024-03-28T22:33:28.931167+0000","last_active":"2024-03-28T22:35:30.254029+0000","last_peered":"2024-03-28T22:35:30.254029+0000","last_clean":"2024-03-28T22:35:30.254029+0000","last_became_active":"2024-03-28T22:33:28.930824+0000","last_became_peered":"2024-03-28T22:33:28.930824+0000","last_unstale":"2024-03-28T22:35:30.254029+0000","last_undegraded":"2024-03-28T22:35:30.254029+0000","last_fullsized":"2024-03-28T22:35:30.254029+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:46:49.102791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275795+0000","last_change":"2024-03-28T22:33:34.023056+0000","last_active":"2024-03-28T22:35:30.275795+0000","last_peered":"2024-03-28T22:35:30.275795+0000","last_clean":"2024-03-28T22:35:30.275795+0000","last_became_active":"2024-03-28T22:33:34.022612+0000","last_became_peered":"2024-03-28T22:33:34.022612+0000","last_unstale":"2024-03-28T22:35:30.275795+0000","last_undegraded":"2024-03-28T22:35:30.275795+0000","last_fullsized":"2024-03-28T22:35:30.275795+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:52:31.936008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"92'9","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252059+0000","last_change":"2024-03-28T22:33:28.430963+0000","last_active":"2024-03-28T22:35:30.252059+0000","last_peered":"2024-03-28T22:35:30.252059+0000","last_clean":"2024-03-28T22:35:30.252059+0000","last_became_active":"2024-03-28T22:33:28.430632+0000","last_became_peered":"2024-03-28T22:33:28.430632+0000","last_unstale":"2024-03-28T22:35:30.252059+0000","last_undegraded":"2024-03-28T22:35:30.252059+0000","last_fullsized":"2024-03-28T22:35:30.252059+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:30:23.104769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"94'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368323+0000","last_change":"2024-03-28T22:33:28.954105+0000","last_active":"2024-03-28T22:35:30.368323+0000","last_peered":"2024-03-28T22:35:30.368323+0000","last_clean":"2024-03-28T22:35:30.368323+0000","last_became_active":"2024-03-28T22:33:28.953768+0000","last_became_peered":"2024-03-28T22:33:28.953768+0000","last_unstale":"2024-03-28T22:35:30.368323+0000","last_undegraded":"2024-03-28T22:35:30.368323+0000","last_fullsized":"2024-03-28T22:35:30.368323+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:01:25.184523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449514+0000","last_change":"2024-03-28T22:33:34.027976+0000","last_active":"2024-03-28T22:35:30.449514+0000","last_peered":"2024-03-28T22:35:30.449514+0000","last_clean":"2024-03-28T22:35:30.449514+0000","last_became_active":"2024-03-28T22:33:34.027509+0000","last_became_peered":"2024-03-28T22:33:34.027509+0000","last_unstale":"2024-03-28T22:35:30.449514+0000","last_undegraded":"2024-03-28T22:35:30.449514+0000","last_fullsized":"2024-03-28T22:35:30.449514+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:25:41.908441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"92'5","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275257+0000","last_change":"2024-03-28T22:33:28.433283+0000","last_active":"2024-03-28T22:35:30.275257+0000","last_peered":"2024-03-28T22:35:30.275257+0000","last_clean":"2024-03-28T22:35:30.275257+0000","last_became_active":"2024-03-28T22:33:28.432954+0000","last_became_peered":"2024-03-28T22:33:28.432954+0000","last_unstale":"2024-03-28T22:35:30.275257+0000","last_undegraded":"2024-03-28T22:35:30.275257+0000","last_fullsized":"2024-03-28T22:35:30.275257+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:08:34.480459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"94'15","reported_seq":46,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248416+0000","last_change":"2024-03-28T22:33:28.939016+0000","last_active":"2024-03-28T22:35:30.248416+0000","last_peered":"2024-03-28T22:35:30.248416+0000","last_clean":"2024-03-28T22:35:30.248416+0000","last_became_active":"2024-03-28T22:33:28.938695+0000","last_became_peered":"2024-03-28T22:33:28.938695+0000","last_unstale":"2024-03-28T22:35:30.248416+0000","last_undegraded":"2024-03-28T22:35:30.248416+0000","last_fullsized":"2024-03-28T22:35:30.248416+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:26:53.264559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252656+0000","last_change":"2024-03-28T22:33:34.030127+0000","last_active":"2024-03-28T22:35:30.252656+0000","last_peered":"2024-03-28T22:35:30.252656+0000","last_clean":"2024-03-28T22:35:30.252656+0000","last_became_active":"2024-03-28T22:33:34.028056+0000","last_became_peered":"2024-03-28T22:33:34.028056+0000","last_unstale":"2024-03-28T22:35:30.252656+0000","last_undegraded":"2024-03-28T22:35:30.252656+0000","last_fullsized":"2024-03-28T22:35:30.252656+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:29:41.785013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'7","reported_seq":138,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:40.395795+0000","last_change":"2024-03-28T22:33:28.435350+0000","last_active":"2024-03-28T22:35:40.395795+0000","last_peered":"2024-03-28T22:35:40.395795+0000","last_clean":"2024-03-28T22:35:40.395795+0000","last_became_active":"2024-03-28T22:33:28.434954+0000","last_became_peered":"2024-03-28T22:33:28.434954+0000","last_unstale":"2024-03-28T22:35:40.395795+0000","last_undegraded":"2024-03-28T22:35:40.395795+0000","last_fullsized":"2024-03-28T22:35:40.395795+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:04.969596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":100,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"94'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248078+0000","last_change":"2024-03-28T22:33:28.921219+0000","last_active":"2024-03-28T22:35:30.248078+0000","last_peered":"2024-03-28T22:35:30.248078+0000","last_clean":"2024-03-28T22:35:30.248078+0000","last_became_active":"2024-03-28T22:33:28.920664+0000","last_became_peered":"2024-03-28T22:33:28.920664+0000","last_unstale":"2024-03-28T22:35:30.248078+0000","last_undegraded":"2024-03-28T22:35:30.248078+0000","last_fullsized":"2024-03-28T22:35:30.248078+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:48:55.032058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369304+0000","last_change":"2024-03-28T22:33:34.018101+0000","last_active":"2024-03-28T22:35:30.369304+0000","last_peered":"2024-03-28T22:35:30.369304+0000","last_clean":"2024-03-28T22:35:30.369304+0000","last_became_active":"2024-03-28T22:33:34.017260+0000","last_became_peered":"2024-03-28T22:33:34.017260+0000","last_unstale":"2024-03-28T22:35:30.369304+0000","last_undegraded":"2024-03-28T22:35:30.369304+0000","last_fullsized":"2024-03-28T22:35:30.369304+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:48:53.178090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"92'17","reported_seq":51,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.253538+0000","last_change":"2024-03-28T22:33:28.450232+0000","last_active":"2024-03-28T22:35:30.253538+0000","last_peered":"2024-03-28T22:35:30.253538+0000","last_clean":"2024-03-28T22:35:30.253538+0000","last_became_active":"2024-03-28T22:33:28.449958+0000","last_became_peered":"2024-03-28T22:33:28.449958+0000","last_unstale":"2024-03-28T22:35:30.253538+0000","last_undegraded":"2024-03-28T22:35:30.253538+0000","last_fullsized":"2024-03-28T22:35:30.253538+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:01:42.130260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"94'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275169+0000","last_change":"2024-03-28T22:33:28.926263+0000","last_active":"2024-03-28T22:35:30.275169+0000","last_peered":"2024-03-28T22:35:30.275169+0000","last_clean":"2024-03-28T22:35:30.275169+0000","last_became_active":"2024-03-28T22:33:28.925927+0000","last_became_peered":"2024-03-28T22:33:28.925927+0000","last_unstale":"2024-03-28T22:35:30.275169+0000","last_undegraded":"2024-03-28T22:35:30.275169+0000","last_fullsized":"2024-03-28T22:35:30.275169+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:41:42.477694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.455002+0000","last_change":"2024-03-28T22:35:30.455002+0000","last_active":"2024-03-28T22:35:30.455002+0000","last_peered":"2024-03-28T22:35:30.455002+0000","last_clean":"2024-03-28T22:35:30.455002+0000","last_became_active":"2024-03-28T22:35:30.454479+0000","last_became_peered":"2024-03-28T22:35:30.454479+0000","last_unstale":"2024-03-28T22:35:30.455002+0000","last_undegraded":"2024-03-28T22:35:30.455002+0000","last_fullsized":"2024-03-28T22:35:30.455002+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:57:06.300031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,3,1],"acting":[10,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.f","version":"92'2","reported_seq":21,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449758+0000","last_change":"2024-03-28T22:34:29.986650+0000","last_active":"2024-03-28T22:35:30.449758+0000","last_peered":"2024-03-28T22:35:30.449758+0000","last_clean":"2024-03-28T22:35:30.449758+0000","last_became_active":"2024-03-28T22:34:29.981875+0000","last_became_peered":"2024-03-28T22:34:29.981875+0000","last_unstale":"2024-03-28T22:35:30.449758+0000","last_undegraded":"2024-03-28T22:35:30.449758+0000","last_fullsized":"2024-03-28T22:35:30.449758+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:45:24.166133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"94'5","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449898+0000","last_change":"2024-03-28T22:34:29.982815+0000","last_active":"2024-03-28T22:35:30.449898+0000","last_peered":"2024-03-28T22:35:30.449898+0000","last_clean":"2024-03-28T22:35:30.449898+0000","last_became_active":"2024-03-28T22:34:29.982458+0000","last_became_peered":"2024-03-28T22:34:29.982458+0000","last_unstale":"2024-03-28T22:35:30.449898+0000","last_undegraded":"2024-03-28T22:35:30.449898+0000","last_fullsized":"2024-03-28T22:35:30.449898+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:11:30.617181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":30,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451505+0000","last_change":"2024-03-28T22:35:30.451063+0000","last_active":"2024-03-28T22:35:30.451505+0000","last_peered":"2024-03-28T22:35:30.451505+0000","last_clean":"2024-03-28T22:35:30.451505+0000","last_became_active":"2024-03-28T22:35:30.450525+0000","last_became_peered":"2024-03-28T22:35:30.450525+0000","last_unstale":"2024-03-28T22:35:30.451505+0000","last_undegraded":"2024-03-28T22:35:30.451505+0000","last_fullsized":"2024-03-28T22:35:30.451505+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:35:29.004834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"92'1","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252202+0000","last_change":"2024-03-28T22:34:29.992884+0000","last_active":"2024-03-28T22:35:30.252202+0000","last_peered":"2024-03-28T22:35:30.252202+0000","last_clean":"2024-03-28T22:35:30.252202+0000","last_became_active":"2024-03-28T22:34:29.991657+0000","last_became_peered":"2024-03-28T22:34:29.991657+0000","last_unstale":"2024-03-28T22:35:30.252202+0000","last_undegraded":"2024-03-28T22:35:30.252202+0000","last_fullsized":"2024-03-28T22:35:30.252202+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:25:14.540783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"94'4","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.259438+0000","last_change":"2024-03-28T22:33:28.942097+0000","last_active":"2024-03-28T22:35:30.259438+0000","last_peered":"2024-03-28T22:35:30.259438+0000","last_clean":"2024-03-28T22:35:30.259438+0000","last_became_active":"2024-03-28T22:33:28.941812+0000","last_became_peered":"2024-03-28T22:33:28.941812+0000","last_unstale":"2024-03-28T22:35:30.259438+0000","last_undegraded":"2024-03-28T22:35:30.259438+0000","last_fullsized":"2024-03-28T22:35:30.259438+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:36:10.204216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369177+0000","last_change":"2024-03-28T22:33:34.023171+0000","last_active":"2024-03-28T22:35:30.369177+0000","last_peered":"2024-03-28T22:35:30.369177+0000","last_clean":"2024-03-28T22:35:30.369177+0000","last_became_active":"2024-03-28T22:33:34.021313+0000","last_became_peered":"2024-03-28T22:33:34.021313+0000","last_unstale":"2024-03-28T22:35:30.369177+0000","last_undegraded":"2024-03-28T22:35:30.369177+0000","last_fullsized":"2024-03-28T22:35:30.369177+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:29:42.592761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"92'5","reported_seq":138,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:42.338520+0000","last_change":"2024-03-28T22:33:28.449133+0000","last_active":"2024-03-28T22:35:42.338520+0000","last_peered":"2024-03-28T22:35:42.338520+0000","last_clean":"2024-03-28T22:35:42.338520+0000","last_became_active":"2024-03-28T22:33:28.448743+0000","last_became_peered":"2024-03-28T22:33:28.448743+0000","last_unstale":"2024-03-28T22:35:42.338520+0000","last_undegraded":"2024-03-28T22:35:42.338520+0000","last_fullsized":"2024-03-28T22:35:42.338520+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:09:37.119736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":99,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.259332+0000","last_change":"2024-03-28T22:33:28.944198+0000","last_active":"2024-03-28T22:35:30.259332+0000","last_peered":"2024-03-28T22:35:30.259332+0000","last_clean":"2024-03-28T22:35:30.259332+0000","last_became_active":"2024-03-28T22:33:28.943818+0000","last_became_peered":"2024-03-28T22:33:28.943818+0000","last_unstale":"2024-03-28T22:35:30.259332+0000","last_undegraded":"2024-03-28T22:35:30.259332+0000","last_fullsized":"2024-03-28T22:35:30.259332+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:31:03.791364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369445+0000","last_change":"2024-03-28T22:33:34.015499+0000","last_active":"2024-03-28T22:35:30.369445+0000","last_peered":"2024-03-28T22:35:30.369445+0000","last_clean":"2024-03-28T22:35:30.369445+0000","last_became_active":"2024-03-28T22:33:34.015013+0000","last_became_peered":"2024-03-28T22:33:34.015013+0000","last_unstale":"2024-03-28T22:35:30.369445+0000","last_undegraded":"2024-03-28T22:35:30.369445+0000","last_fullsized":"2024-03-28T22:35:30.369445+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:34:30.582198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"94'8","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275296+0000","last_change":"2024-03-28T22:33:28.926732+0000","last_active":"2024-03-28T22:35:30.275296+0000","last_peered":"2024-03-28T22:35:30.275296+0000","last_clean":"2024-03-28T22:35:30.275296+0000","last_became_active":"2024-03-28T22:33:28.926366+0000","last_became_peered":"2024-03-28T22:33:28.926366+0000","last_unstale":"2024-03-28T22:35:30.275296+0000","last_undegraded":"2024-03-28T22:35:30.275296+0000","last_fullsized":"2024-03-28T22:35:30.275296+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:19:10.769141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'6","reported_seq":138,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:41.397722+0000","last_change":"2024-03-28T22:33:28.457400+0000","last_active":"2024-03-28T22:35:41.397722+0000","last_peered":"2024-03-28T22:35:41.397722+0000","last_clean":"2024-03-28T22:35:41.397722+0000","last_became_active":"2024-03-28T22:33:28.456994+0000","last_became_peered":"2024-03-28T22:33:28.456994+0000","last_unstale":"2024-03-28T22:35:41.397722+0000","last_undegraded":"2024-03-28T22:35:41.397722+0000","last_fullsized":"2024-03-28T22:35:41.397722+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:32:16.438540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":98,"num_read_kb":98,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"92'1","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248405+0000","last_change":"2024-03-28T22:33:34.027746+0000","last_active":"2024-03-28T22:35:30.248405+0000","last_peered":"2024-03-28T22:35:30.248405+0000","last_clean":"2024-03-28T22:35:30.248405+0000","last_became_active":"2024-03-28T22:33:34.027419+0000","last_became_peered":"2024-03-28T22:33:34.027419+0000","last_unstale":"2024-03-28T22:35:30.248405+0000","last_undegraded":"2024-03-28T22:35:30.248405+0000","last_fullsized":"2024-03-28T22:35:30.248405+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:55:43.686428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371544+0000","last_change":"2024-03-28T22:33:28.952393+0000","last_active":"2024-03-28T22:35:30.371544+0000","last_peered":"2024-03-28T22:35:30.371544+0000","last_clean":"2024-03-28T22:35:30.371544+0000","last_became_active":"2024-03-28T22:33:28.951951+0000","last_became_peered":"2024-03-28T22:33:28.951951+0000","last_unstale":"2024-03-28T22:35:30.371544+0000","last_undegraded":"2024-03-28T22:35:30.371544+0000","last_fullsized":"2024-03-28T22:35:30.371544+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:09:25.808842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"92'6","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370674+0000","last_change":"2024-03-28T22:33:28.437177+0000","last_active":"2024-03-28T22:35:30.370674+0000","last_peered":"2024-03-28T22:35:30.370674+0000","last_clean":"2024-03-28T22:35:30.370674+0000","last_became_active":"2024-03-28T22:33:28.436801+0000","last_became_peered":"2024-03-28T22:33:28.436801+0000","last_unstale":"2024-03-28T22:35:30.370674+0000","last_undegraded":"2024-03-28T22:35:30.370674+0000","last_fullsized":"2024-03-28T22:35:30.370674+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:24:52.549490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":371,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.247934+0000","last_change":"2024-03-28T22:32:07.867336+0000","last_active":"2024-03-28T22:35:30.247934+0000","last_peered":"2024-03-28T22:35:30.247934+0000","last_clean":"2024-03-28T22:35:30.247934+0000","last_became_active":"2024-03-28T22:32:07.866819+0000","last_became_peered":"2024-03-28T22:32:07.866819+0000","last_unstale":"2024-03-28T22:35:30.247934+0000","last_undegraded":"2024-03-28T22:35:30.247934+0000","last_fullsized":"2024-03-28T22:35:30.247934+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:28:14.713501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:28:14.713501+0000","last_clean_scrub_stamp":"2024-03-28T22:28:14.713501+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:00:52.438708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275546+0000","last_change":"2024-03-28T22:33:34.020668+0000","last_active":"2024-03-28T22:35:30.275546+0000","last_peered":"2024-03-28T22:35:30.275546+0000","last_clean":"2024-03-28T22:35:30.275546+0000","last_became_active":"2024-03-28T22:33:34.020240+0000","last_became_peered":"2024-03-28T22:33:34.020240+0000","last_unstale":"2024-03-28T22:35:30.275546+0000","last_undegraded":"2024-03-28T22:35:30.275546+0000","last_fullsized":"2024-03-28T22:35:30.275546+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:18.983983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"94'5","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368061+0000","last_change":"2024-03-28T22:33:28.935237+0000","last_active":"2024-03-28T22:35:30.368061+0000","last_peered":"2024-03-28T22:35:30.368061+0000","last_clean":"2024-03-28T22:35:30.368061+0000","last_became_active":"2024-03-28T22:33:28.934875+0000","last_became_peered":"2024-03-28T22:33:28.934875+0000","last_unstale":"2024-03-28T22:35:30.368061+0000","last_undegraded":"2024-03-28T22:35:30.368061+0000","last_fullsized":"2024-03-28T22:35:30.368061+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:20:50.789302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"92'4","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248187+0000","last_change":"2024-03-28T22:33:28.441654+0000","last_active":"2024-03-28T22:35:30.248187+0000","last_peered":"2024-03-28T22:35:30.248187+0000","last_clean":"2024-03-28T22:35:30.248187+0000","last_became_active":"2024-03-28T22:33:28.441293+0000","last_became_peered":"2024-03-28T22:33:28.441293+0000","last_unstale":"2024-03-28T22:35:30.248187+0000","last_undegraded":"2024-03-28T22:35:30.248187+0000","last_fullsized":"2024-03-28T22:35:30.248187+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:31:17.737977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449486+0000","last_change":"2024-03-28T22:33:34.018126+0000","last_active":"2024-03-28T22:35:30.449486+0000","last_peered":"2024-03-28T22:35:30.449486+0000","last_clean":"2024-03-28T22:35:30.449486+0000","last_became_active":"2024-03-28T22:33:34.017573+0000","last_became_peered":"2024-03-28T22:33:34.017573+0000","last_unstale":"2024-03-28T22:35:30.449486+0000","last_undegraded":"2024-03-28T22:35:30.449486+0000","last_fullsized":"2024-03-28T22:35:30.449486+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:23:31.013511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"92'4","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254108+0000","last_change":"2024-03-28T22:33:28.454069+0000","last_active":"2024-03-28T22:35:30.254108+0000","last_peered":"2024-03-28T22:35:30.254108+0000","last_clean":"2024-03-28T22:35:30.254108+0000","last_became_active":"2024-03-28T22:33:28.453754+0000","last_became_peered":"2024-03-28T22:33:28.453754+0000","last_unstale":"2024-03-28T22:35:30.254108+0000","last_undegraded":"2024-03-28T22:35:30.254108+0000","last_fullsized":"2024-03-28T22:35:30.254108+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:23:33.981072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"94'9","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369387+0000","last_change":"2024-03-28T22:33:28.934390+0000","last_active":"2024-03-28T22:35:30.369387+0000","last_peered":"2024-03-28T22:35:30.369387+0000","last_clean":"2024-03-28T22:35:30.369387+0000","last_became_active":"2024-03-28T22:33:28.933919+0000","last_became_peered":"2024-03-28T22:33:28.933919+0000","last_unstale":"2024-03-28T22:35:30.369387+0000","last_undegraded":"2024-03-28T22:35:30.369387+0000","last_fullsized":"2024-03-28T22:35:30.369387+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:27:07.595144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371312+0000","last_change":"2024-03-28T22:33:34.019910+0000","last_active":"2024-03-28T22:35:30.371312+0000","last_peered":"2024-03-28T22:35:30.371312+0000","last_clean":"2024-03-28T22:35:30.371312+0000","last_became_active":"2024-03-28T22:33:34.019522+0000","last_became_peered":"2024-03-28T22:33:34.019522+0000","last_unstale":"2024-03-28T22:35:30.371312+0000","last_undegraded":"2024-03-28T22:35:30.371312+0000","last_fullsized":"2024-03-28T22:35:30.371312+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:01:29.298513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'14","reported_seq":48,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252817+0000","last_change":"2024-03-28T22:33:28.444513+0000","last_active":"2024-03-28T22:35:30.252817+0000","last_peered":"2024-03-28T22:35:30.252817+0000","last_clean":"2024-03-28T22:35:30.252817+0000","last_became_active":"2024-03-28T22:33:28.444091+0000","last_became_peered":"2024-03-28T22:33:28.444091+0000","last_unstale":"2024-03-28T22:35:30.252817+0000","last_undegraded":"2024-03-28T22:35:30.252817+0000","last_fullsized":"2024-03-28T22:35:30.252817+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:49:58.331864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"94'5","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.372223+0000","last_change":"2024-03-28T22:33:28.948390+0000","last_active":"2024-03-28T22:35:30.372223+0000","last_peered":"2024-03-28T22:35:30.372223+0000","last_clean":"2024-03-28T22:35:30.372223+0000","last_became_active":"2024-03-28T22:33:28.942846+0000","last_became_peered":"2024-03-28T22:33:28.942846+0000","last_unstale":"2024-03-28T22:35:30.372223+0000","last_undegraded":"2024-03-28T22:35:30.372223+0000","last_fullsized":"2024-03-28T22:35:30.372223+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:50.596365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251712+0000","last_change":"2024-03-28T22:33:33.999429+0000","last_active":"2024-03-28T22:35:30.251712+0000","last_peered":"2024-03-28T22:35:30.251712+0000","last_clean":"2024-03-28T22:35:30.251712+0000","last_became_active":"2024-03-28T22:33:33.998979+0000","last_became_peered":"2024-03-28T22:33:33.998979+0000","last_unstale":"2024-03-28T22:35:30.251712+0000","last_undegraded":"2024-03-28T22:35:30.251712+0000","last_fullsized":"2024-03-28T22:35:30.251712+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:26.450731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"92'7","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370791+0000","last_change":"2024-03-28T22:33:28.442861+0000","last_active":"2024-03-28T22:35:30.370791+0000","last_peered":"2024-03-28T22:35:30.370791+0000","last_clean":"2024-03-28T22:35:30.370791+0000","last_became_active":"2024-03-28T22:33:28.442468+0000","last_became_peered":"2024-03-28T22:33:28.442468+0000","last_unstale":"2024-03-28T22:35:30.370791+0000","last_undegraded":"2024-03-28T22:35:30.370791+0000","last_fullsized":"2024-03-28T22:35:30.370791+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:45:49.899031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"94'7","reported_seq":45,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451985+0000","last_change":"2024-03-28T22:34:30.429872+0000","last_active":"2024-03-28T22:35:30.451985+0000","last_peered":"2024-03-28T22:35:30.451985+0000","last_clean":"2024-03-28T22:35:30.451985+0000","last_became_active":"2024-03-28T22:34:30.429237+0000","last_became_peered":"2024-03-28T22:34:30.429237+0000","last_unstale":"2024-03-28T22:35:30.451985+0000","last_undegraded":"2024-03-28T22:35:30.451985+0000","last_fullsized":"2024-03-28T22:35:30.451985+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:14:46.608826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370518+0000","last_change":"2024-03-28T22:33:34.013826+0000","last_active":"2024-03-28T22:35:30.370518+0000","last_peered":"2024-03-28T22:35:30.370518+0000","last_clean":"2024-03-28T22:35:30.370518+0000","last_became_active":"2024-03-28T22:33:34.013415+0000","last_became_peered":"2024-03-28T22:33:34.013415+0000","last_unstale":"2024-03-28T22:35:30.370518+0000","last_undegraded":"2024-03-28T22:35:30.370518+0000","last_fullsized":"2024-03-28T22:35:30.370518+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:19:30.965940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"92'2","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448399+0000","last_change":"2024-03-28T22:33:28.442225+0000","last_active":"2024-03-28T22:35:30.448399+0000","last_peered":"2024-03-28T22:35:30.448399+0000","last_clean":"2024-03-28T22:35:30.448399+0000","last_became_active":"2024-03-28T22:33:28.441844+0000","last_became_peered":"2024-03-28T22:33:28.441844+0000","last_unstale":"2024-03-28T22:35:30.448399+0000","last_undegraded":"2024-03-28T22:35:30.448399+0000","last_fullsized":"2024-03-28T22:35:30.448399+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:40:52.108663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"94'2","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.253675+0000","last_change":"2024-03-28T22:33:28.927571+0000","last_active":"2024-03-28T22:35:30.253675+0000","last_peered":"2024-03-28T22:35:30.253675+0000","last_clean":"2024-03-28T22:35:30.253675+0000","last_became_active":"2024-03-28T22:33:28.927275+0000","last_became_peered":"2024-03-28T22:33:28.927275+0000","last_unstale":"2024-03-28T22:35:30.253675+0000","last_undegraded":"2024-03-28T22:35:30.253675+0000","last_fullsized":"2024-03-28T22:35:30.253675+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:25:16.184533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252877+0000","last_change":"2024-03-28T22:33:34.018429+0000","last_active":"2024-03-28T22:35:30.252877+0000","last_peered":"2024-03-28T22:35:30.252877+0000","last_clean":"2024-03-28T22:35:30.252877+0000","last_became_active":"2024-03-28T22:33:34.017066+0000","last_became_peered":"2024-03-28T22:33:34.017066+0000","last_unstale":"2024-03-28T22:35:30.252877+0000","last_undegraded":"2024-03-28T22:35:30.252877+0000","last_fullsized":"2024-03-28T22:35:30.252877+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:35:42.101072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"92'7","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371970+0000","last_change":"2024-03-28T22:33:28.437760+0000","last_active":"2024-03-28T22:35:30.371970+0000","last_peered":"2024-03-28T22:35:30.371970+0000","last_clean":"2024-03-28T22:35:30.371970+0000","last_became_active":"2024-03-28T22:33:28.437096+0000","last_became_peered":"2024-03-28T22:33:28.437096+0000","last_unstale":"2024-03-28T22:35:30.371970+0000","last_undegraded":"2024-03-28T22:35:30.371970+0000","last_fullsized":"2024-03-28T22:35:30.371970+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:29:20.778551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"94'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449358+0000","last_change":"2024-03-28T22:33:28.935805+0000","last_active":"2024-03-28T22:35:30.449358+0000","last_peered":"2024-03-28T22:35:30.449358+0000","last_clean":"2024-03-28T22:35:30.449358+0000","last_became_active":"2024-03-28T22:33:28.935520+0000","last_became_peered":"2024-03-28T22:33:28.935520+0000","last_unstale":"2024-03-28T22:35:30.449358+0000","last_undegraded":"2024-03-28T22:35:30.449358+0000","last_fullsized":"2024-03-28T22:35:30.449358+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:23.414310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"92'1","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370530+0000","last_change":"2024-03-28T22:33:34.029518+0000","last_active":"2024-03-28T22:35:30.370530+0000","last_peered":"2024-03-28T22:35:30.370530+0000","last_clean":"2024-03-28T22:35:30.370530+0000","last_became_active":"2024-03-28T22:33:34.029101+0000","last_became_peered":"2024-03-28T22:33:34.029101+0000","last_unstale":"2024-03-28T22:35:30.370530+0000","last_undegraded":"2024-03-28T22:35:30.370530+0000","last_fullsized":"2024-03-28T22:35:30.370530+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:05.595543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"92'3","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251929+0000","last_change":"2024-03-28T22:33:28.438857+0000","last_active":"2024-03-28T22:35:30.251929+0000","last_peered":"2024-03-28T22:35:30.251929+0000","last_clean":"2024-03-28T22:35:30.251929+0000","last_became_active":"2024-03-28T22:33:28.438528+0000","last_became_peered":"2024-03-28T22:33:28.438528+0000","last_unstale":"2024-03-28T22:35:30.251929+0000","last_undegraded":"2024-03-28T22:35:30.251929+0000","last_fullsized":"2024-03-28T22:35:30.251929+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:07:54.506739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"94'8","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371602+0000","last_change":"2024-03-28T22:33:28.958271+0000","last_active":"2024-03-28T22:35:30.371602+0000","last_peered":"2024-03-28T22:35:30.371602+0000","last_clean":"2024-03-28T22:35:30.371602+0000","last_became_active":"2024-03-28T22:33:28.957922+0000","last_became_peered":"2024-03-28T22:33:28.957922+0000","last_unstale":"2024-03-28T22:35:30.371602+0000","last_undegraded":"2024-03-28T22:35:30.371602+0000","last_fullsized":"2024-03-28T22:35:30.371602+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:33:19.003131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370511+0000","last_change":"2024-03-28T22:33:34.000789+0000","last_active":"2024-03-28T22:35:30.370511+0000","last_peered":"2024-03-28T22:35:30.370511+0000","last_clean":"2024-03-28T22:35:30.370511+0000","last_became_active":"2024-03-28T22:33:34.000383+0000","last_became_peered":"2024-03-28T22:33:34.000383+0000","last_unstale":"2024-03-28T22:35:30.370511+0000","last_undegraded":"2024-03-28T22:35:30.370511+0000","last_fullsized":"2024-03-28T22:35:30.370511+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:50:29.925656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"92'4","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.253813+0000","last_change":"2024-03-28T22:33:28.439121+0000","last_active":"2024-03-28T22:35:30.253813+0000","last_peered":"2024-03-28T22:35:30.253813+0000","last_clean":"2024-03-28T22:35:30.253813+0000","last_became_active":"2024-03-28T22:33:28.438800+0000","last_became_peered":"2024-03-28T22:33:28.438800+0000","last_unstale":"2024-03-28T22:35:30.253813+0000","last_undegraded":"2024-03-28T22:35:30.253813+0000","last_fullsized":"2024-03-28T22:35:30.253813+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:57:10.013246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"94'5","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248157+0000","last_change":"2024-03-28T22:33:28.934499+0000","last_active":"2024-03-28T22:35:30.248157+0000","last_peered":"2024-03-28T22:35:30.248157+0000","last_clean":"2024-03-28T22:35:30.248157+0000","last_became_active":"2024-03-28T22:33:28.934147+0000","last_became_peered":"2024-03-28T22:33:28.934147+0000","last_unstale":"2024-03-28T22:35:30.248157+0000","last_undegraded":"2024-03-28T22:35:30.248157+0000","last_fullsized":"2024-03-28T22:35:30.248157+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:35:39.090052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448402+0000","last_change":"2024-03-28T22:33:34.017946+0000","last_active":"2024-03-28T22:35:30.448402+0000","last_peered":"2024-03-28T22:35:30.448402+0000","last_clean":"2024-03-28T22:35:30.448402+0000","last_became_active":"2024-03-28T22:33:34.017546+0000","last_became_peered":"2024-03-28T22:33:34.017546+0000","last_unstale":"2024-03-28T22:35:30.448402+0000","last_undegraded":"2024-03-28T22:35:30.448402+0000","last_fullsized":"2024-03-28T22:35:30.448402+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:14:39.081677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"92'5","reported_seq":139,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:43.286205+0000","last_change":"2024-03-28T22:33:28.453387+0000","last_active":"2024-03-28T22:35:43.286205+0000","last_peered":"2024-03-28T22:35:43.286205+0000","last_clean":"2024-03-28T22:35:43.286205+0000","last_became_active":"2024-03-28T22:33:28.453029+0000","last_became_peered":"2024-03-28T22:33:28.453029+0000","last_unstale":"2024-03-28T22:35:43.286205+0000","last_undegraded":"2024-03-28T22:35:43.286205+0000","last_fullsized":"2024-03-28T22:35:43.286205+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:12:33.246863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":100,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"94'7","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248569+0000","last_change":"2024-03-28T22:33:28.923819+0000","last_active":"2024-03-28T22:35:30.248569+0000","last_peered":"2024-03-28T22:35:30.248569+0000","last_clean":"2024-03-28T22:35:30.248569+0000","last_became_active":"2024-03-28T22:33:28.923471+0000","last_became_peered":"2024-03-28T22:33:28.923471+0000","last_unstale":"2024-03-28T22:35:30.248569+0000","last_undegraded":"2024-03-28T22:35:30.248569+0000","last_fullsized":"2024-03-28T22:35:30.248569+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:58:03.496406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.450859+0000","last_change":"2024-03-28T22:33:34.029789+0000","last_active":"2024-03-28T22:35:30.450859+0000","last_peered":"2024-03-28T22:35:30.450859+0000","last_clean":"2024-03-28T22:35:30.450859+0000","last_became_active":"2024-03-28T22:33:34.029231+0000","last_became_peered":"2024-03-28T22:33:34.029231+0000","last_unstale":"2024-03-28T22:35:30.450859+0000","last_undegraded":"2024-03-28T22:35:30.450859+0000","last_fullsized":"2024-03-28T22:35:30.450859+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:47:50.101393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"92'3","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275488+0000","last_change":"2024-03-28T22:33:28.435668+0000","last_active":"2024-03-28T22:35:30.275488+0000","last_peered":"2024-03-28T22:35:30.275488+0000","last_clean":"2024-03-28T22:35:30.275488+0000","last_became_active":"2024-03-28T22:33:28.435325+0000","last_became_peered":"2024-03-28T22:33:28.435325+0000","last_unstale":"2024-03-28T22:35:30.275488+0000","last_undegraded":"2024-03-28T22:35:30.275488+0000","last_fullsized":"2024-03-28T22:35:30.275488+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:13:06.281074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"94'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248195+0000","last_change":"2024-03-28T22:33:28.954138+0000","last_active":"2024-03-28T22:35:30.248195+0000","last_peered":"2024-03-28T22:35:30.248195+0000","last_clean":"2024-03-28T22:35:30.248195+0000","last_became_active":"2024-03-28T22:33:28.953752+0000","last_became_peered":"2024-03-28T22:33:28.953752+0000","last_unstale":"2024-03-28T22:35:30.248195+0000","last_undegraded":"2024-03-28T22:35:30.248195+0000","last_fullsized":"2024-03-28T22:35:30.248195+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:53:34.892316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.274936+0000","last_change":"2024-03-28T22:33:34.002672+0000","last_active":"2024-03-28T22:35:30.274936+0000","last_peered":"2024-03-28T22:35:30.274936+0000","last_clean":"2024-03-28T22:35:30.274936+0000","last_became_active":"2024-03-28T22:33:34.002195+0000","last_became_peered":"2024-03-28T22:33:34.002195+0000","last_unstale":"2024-03-28T22:35:30.274936+0000","last_undegraded":"2024-03-28T22:35:30.274936+0000","last_fullsized":"2024-03-28T22:35:30.274936+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:24:11.521474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"92'5","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369212+0000","last_change":"2024-03-28T22:33:28.450818+0000","last_active":"2024-03-28T22:35:30.369212+0000","last_peered":"2024-03-28T22:35:30.369212+0000","last_clean":"2024-03-28T22:35:30.369212+0000","last_became_active":"2024-03-28T22:33:28.448592+0000","last_became_peered":"2024-03-28T22:33:28.448592+0000","last_unstale":"2024-03-28T22:35:30.369212+0000","last_undegraded":"2024-03-28T22:35:30.369212+0000","last_fullsized":"2024-03-28T22:35:30.369212+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:19:27.126306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"94'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248443+0000","last_change":"2024-03-28T22:33:28.925191+0000","last_active":"2024-03-28T22:35:30.248443+0000","last_peered":"2024-03-28T22:35:30.248443+0000","last_clean":"2024-03-28T22:35:30.248443+0000","last_became_active":"2024-03-28T22:33:28.924424+0000","last_became_peered":"2024-03-28T22:33:28.924424+0000","last_unstale":"2024-03-28T22:35:30.248443+0000","last_undegraded":"2024-03-28T22:35:30.248443+0000","last_fullsized":"2024-03-28T22:35:30.248443+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:50:00.415047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"94'9","reported_seq":46,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449424+0000","last_change":"2024-03-28T22:34:29.987176+0000","last_active":"2024-03-28T22:35:30.449424+0000","last_peered":"2024-03-28T22:35:30.449424+0000","last_clean":"2024-03-28T22:35:30.449424+0000","last_became_active":"2024-03-28T22:34:29.986807+0000","last_became_peered":"2024-03-28T22:34:29.986807+0000","last_unstale":"2024-03-28T22:35:30.449424+0000","last_undegraded":"2024-03-28T22:35:30.449424+0000","last_fullsized":"2024-03-28T22:35:30.449424+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:00:06.774521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"94'12","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251938+0000","last_change":"2024-03-28T22:33:28.927068+0000","last_active":"2024-03-28T22:35:30.251938+0000","last_peered":"2024-03-28T22:35:30.251938+0000","last_clean":"2024-03-28T22:35:30.251938+0000","last_became_active":"2024-03-28T22:33:28.926734+0000","last_became_peered":"2024-03-28T22:33:28.926734+0000","last_unstale":"2024-03-28T22:35:30.251938+0000","last_undegraded":"2024-03-28T22:35:30.251938+0000","last_fullsized":"2024-03-28T22:35:30.251938+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:56.309555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"94'8","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251930+0000","last_change":"2024-03-28T22:33:28.955675+0000","last_active":"2024-03-28T22:35:30.251930+0000","last_peered":"2024-03-28T22:35:30.251930+0000","last_clean":"2024-03-28T22:35:30.251930+0000","last_became_active":"2024-03-28T22:33:28.955329+0000","last_became_peered":"2024-03-28T22:33:28.955329+0000","last_unstale":"2024-03-28T22:35:30.251930+0000","last_undegraded":"2024-03-28T22:35:30.251930+0000","last_fullsized":"2024-03-28T22:35:30.251930+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:34:14.166434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.452477+0000","last_change":"2024-03-28T22:33:28.955125+0000","last_active":"2024-03-28T22:35:30.452477+0000","last_peered":"2024-03-28T22:35:30.452477+0000","last_clean":"2024-03-28T22:35:30.452477+0000","last_became_active":"2024-03-28T22:33:28.954434+0000","last_became_peered":"2024-03-28T22:33:28.954434+0000","last_unstale":"2024-03-28T22:35:30.452477+0000","last_undegraded":"2024-03-28T22:35:30.452477+0000","last_fullsized":"2024-03-28T22:35:30.452477+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:56:50.705012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"94'4","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369433+0000","last_change":"2024-03-28T22:33:28.945230+0000","last_active":"2024-03-28T22:35:30.369433+0000","last_peered":"2024-03-28T22:35:30.369433+0000","last_clean":"2024-03-28T22:35:30.369433+0000","last_became_active":"2024-03-28T22:33:28.944921+0000","last_became_peered":"2024-03-28T22:33:28.944921+0000","last_unstale":"2024-03-28T22:35:30.369433+0000","last_undegraded":"2024-03-28T22:35:30.369433+0000","last_fullsized":"2024-03-28T22:35:30.369433+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:26:26.505774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"94'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449741+0000","last_change":"2024-03-28T22:33:28.928720+0000","last_active":"2024-03-28T22:35:30.449741+0000","last_peered":"2024-03-28T22:35:30.449741+0000","last_clean":"2024-03-28T22:35:30.449741+0000","last_became_active":"2024-03-28T22:33:28.928415+0000","last_became_peered":"2024-03-28T22:33:28.928415+0000","last_unstale":"2024-03-28T22:35:30.449741+0000","last_undegraded":"2024-03-28T22:35:30.449741+0000","last_fullsized":"2024-03-28T22:35:30.449741+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:00.839609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"94'15","reported_seq":47,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252418+0000","last_change":"2024-03-28T22:33:28.924948+0000","last_active":"2024-03-28T22:35:30.252418+0000","last_peered":"2024-03-28T22:35:30.252418+0000","last_clean":"2024-03-28T22:35:30.252418+0000","last_became_active":"2024-03-28T22:33:28.924631+0000","last_became_peered":"2024-03-28T22:33:28.924631+0000","last_unstale":"2024-03-28T22:35:30.252418+0000","last_undegraded":"2024-03-28T22:35:30.252418+0000","last_fullsized":"2024-03-28T22:35:30.252418+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:53:44.909991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"94'2","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.372408+0000","last_change":"2024-03-28T22:33:28.948725+0000","last_active":"2024-03-28T22:35:30.372408+0000","last_peered":"2024-03-28T22:35:30.372408+0000","last_clean":"2024-03-28T22:35:30.372408+0000","last_became_active":"2024-03-28T22:33:28.943942+0000","last_became_peered":"2024-03-28T22:33:28.943942+0000","last_unstale":"2024-03-28T22:35:30.372408+0000","last_undegraded":"2024-03-28T22:35:30.372408+0000","last_fullsized":"2024-03-28T22:35:30.372408+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:50:04.238538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"94'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275158+0000","last_change":"2024-03-28T22:33:28.941546+0000","last_active":"2024-03-28T22:35:30.275158+0000","last_peered":"2024-03-28T22:35:30.275158+0000","last_clean":"2024-03-28T22:35:30.275158+0000","last_became_active":"2024-03-28T22:33:28.941278+0000","last_became_peered":"2024-03-28T22:33:28.941278+0000","last_unstale":"2024-03-28T22:35:30.275158+0000","last_undegraded":"2024-03-28T22:35:30.275158+0000","last_fullsized":"2024-03-28T22:35:30.275158+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:47:10.047067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"94'8","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.372207+0000","last_change":"2024-03-28T22:33:28.949552+0000","last_active":"2024-03-28T22:35:30.372207+0000","last_peered":"2024-03-28T22:35:30.372207+0000","last_clean":"2024-03-28T22:35:30.372207+0000","last_became_active":"2024-03-28T22:33:28.947432+0000","last_became_peered":"2024-03-28T22:33:28.947432+0000","last_unstale":"2024-03-28T22:35:30.372207+0000","last_undegraded":"2024-03-28T22:35:30.372207+0000","last_fullsized":"2024-03-28T22:35:30.372207+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:01:16.534642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"94'10","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448510+0000","last_change":"2024-03-28T22:33:28.953930+0000","last_active":"2024-03-28T22:35:30.448510+0000","last_peered":"2024-03-28T22:35:30.448510+0000","last_clean":"2024-03-28T22:35:30.448510+0000","last_became_active":"2024-03-28T22:33:28.953591+0000","last_became_peered":"2024-03-28T22:33:28.953591+0000","last_unstale":"2024-03-28T22:35:30.448510+0000","last_undegraded":"2024-03-28T22:35:30.448510+0000","last_fullsized":"2024-03-28T22:35:30.448510+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:42:55.857476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"94'11","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368740+0000","last_change":"2024-03-28T22:33:28.946307+0000","last_active":"2024-03-28T22:35:30.368740+0000","last_peered":"2024-03-28T22:35:30.368740+0000","last_clean":"2024-03-28T22:35:30.368740+0000","last_became_active":"2024-03-28T22:33:28.945932+0000","last_became_peered":"2024-03-28T22:33:28.945932+0000","last_unstale":"2024-03-28T22:35:30.368740+0000","last_undegraded":"2024-03-28T22:35:30.368740+0000","last_fullsized":"2024-03-28T22:35:30.368740+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:54:59.651668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"94'8","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449625+0000","last_change":"2024-03-28T22:33:28.928014+0000","last_active":"2024-03-28T22:35:30.449625+0000","last_peered":"2024-03-28T22:35:30.449625+0000","last_clean":"2024-03-28T22:35:30.449625+0000","last_became_active":"2024-03-28T22:33:28.927686+0000","last_became_peered":"2024-03-28T22:33:28.927686+0000","last_unstale":"2024-03-28T22:35:30.449625+0000","last_undegraded":"2024-03-28T22:35:30.449625+0000","last_fullsized":"2024-03-28T22:35:30.449625+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:47:08.131565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71409,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1014,"num_read_kb":587,"num_write":250,"num_write_kb":146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":214227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":217,"ondisk_log_size":217,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318313,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32260,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712892,"statfs":{"total":95995035648,"available":95962001408,"internally_reserved":0,"allocated":1105920,"data_stored":539556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579950,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28780,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716372,"statfs":{"total":95995035648,"available":95965564928,"internally_reserved":0,"allocated":1798144,"data_stored":1191337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743476,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1130496,"data_stored":543560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005114,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1118208,"data_stored":539820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299456,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37024,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708128,"statfs":{"total":95995035648,"available":95957123072,"internally_reserved":0,"allocated":1720320,"data_stored":1136480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495685,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1101824,"data_stored":534968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574945,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36488,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708664,"statfs":{"total":95995035648,"available":95957671936,"internally_reserved":0,"allocated":1171456,"data_stored":600453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738470,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1126400,"data_stored":536424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411422,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36508,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708644,"statfs":{"total":95995035648,"available":95957651456,"internally_reserved":0,"allocated":1191936,"data_stored":600673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182485,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36432,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708720,"statfs":{"total":95995035648,"available":95957729280,"internally_reserved":0,"allocated":1114112,"data_stored":534676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920848,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1110016,"data_stored":539574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757323,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28712,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716440,"statfs":{"total":95995035648,"available":95965634560,"internally_reserved":0,"allocated":1728512,"data_stored":1131530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":66207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":66427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":66723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T22:35:49.434 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:49 smithi043 ceph-mon[31823]: pgmap v396: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:35:49.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:49 smithi172 ceph-mon[36566]: pgmap v396: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:35:49.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:49 smithi082 ceph-mon[36471]: pgmap v396: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:35:50.714 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-28T22:35:51.212 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:35:51.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:51 smithi172 ceph-mon[36566]: pgmap v397: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-28T22:35:51.699 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:51 smithi043 ceph-mon[31823]: pgmap v397: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-28T22:35:51.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:51 smithi082 ceph-mon[36471]: pgmap v397: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-28T22:35:53.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.434+0000 7f6933600700 1 -- 172.21.15.43:0/3389090033 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f692c10a4b0 msgr2=0x7f692c10a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:53.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.434+0000 7f6933600700 1 --2- 172.21.15.43:0/3389090033 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f692c10a4b0 0x7f692c10a890 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6920009990 tx=0x7f6920009ca0 comp rx=0 tx=0).stop 2024-03-28T22:35:53.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.434+0000 7f6933600700 1 -- 172.21.15.43:0/3389090033 shutdown_connections 2024-03-28T22:35:53.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.434+0000 7f6933600700 1 --2- 172.21.15.43:0/3389090033 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f692c104de0 0x7f692c1112f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:53.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.434+0000 7f6933600700 1 --2- 172.21.15.43:0/3389090033 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f692c10a4b0 0x7f692c10a890 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:53.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.434+0000 7f6933600700 1 --2- 172.21.15.43:0/3389090033 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f692c104420 0x7f692c1048a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:53.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.434+0000 7f6933600700 1 -- 172.21.15.43:0/3389090033 >> 172.21.15.43:0/3389090033 conn(0x7f692c0fff50 msgr2=0x7f692c102370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:53.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.434+0000 7f6933600700 1 -- 172.21.15.43:0/3389090033 shutdown_connections 2024-03-28T22:35:53.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.434+0000 7f6933600700 1 -- 172.21.15.43:0/3389090033 wait complete. 2024-03-28T22:35:53.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.435+0000 7f6933600700 1 Processor -- start 2024-03-28T22:35:53.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.436+0000 7f6933600700 1 -- start start 2024-03-28T22:35:53.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.436+0000 7f6933600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f692c104420 0x7f692c1a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:53.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.436+0000 7f6933600700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f692c104de0 0x7f692c1a0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:53.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.436+0000 7f6933600700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f692c10a4b0 0x7f692c19a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:53.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.436+0000 7f6933600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f692c114760 con 0x7f692c104420 2024-03-28T22:35:53.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.436+0000 7f6933600700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f692c1145e0 con 0x7f692c104de0 2024-03-28T22:35:53.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.436+0000 7f6933600700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f692c1148e0 con 0x7f692c10a4b0 2024-03-28T22:35:53.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.436+0000 7f6930c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f692c104420 0x7f692c1a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:53.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.436+0000 7f692be00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f692c104de0 0x7f692c1a0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:53.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.436+0000 7f6930c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f692c104420 0x7f692c1a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41024/0 (socket says 172.21.15.43:41024) 2024-03-28T22:35:53.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.436+0000 7f692be00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f692c104de0 0x7f692c1a0d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:54764/0 (socket says 172.21.15.43:54764) 2024-03-28T22:35:53.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.436+0000 7f6930c00700 1 -- 172.21.15.43:0/2766169863 learned_addr learned my addr 172.21.15.43:0/2766169863 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:35:53.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.436+0000 7f6931600700 1 --2- 172.21.15.43:0/2766169863 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f692c10a4b0 0x7f692c19a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:53.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.437+0000 7f692be00700 1 -- 172.21.15.43:0/2766169863 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f692c10a4b0 msgr2=0x7f692c19a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:53.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.437+0000 7f692be00700 1 --2- 172.21.15.43:0/2766169863 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f692c10a4b0 0x7f692c19a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:53.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.437+0000 7f692be00700 1 -- 172.21.15.43:0/2766169863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f692c104420 msgr2=0x7f692c1a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:53.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.437+0000 7f692be00700 1 --2- 172.21.15.43:0/2766169863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f692c104420 0x7f692c1a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:53.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.437+0000 7f692be00700 1 -- 172.21.15.43:0/2766169863 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f692c19b140 con 0x7f692c104de0 2024-03-28T22:35:53.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.437+0000 7f692be00700 1 --2- 172.21.15.43:0/2766169863 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f692c104de0 0x7f692c1a0d60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f691800e9f0 tx=0x7f691800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:53.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.437+0000 7f6929600700 1 -- 172.21.15.43:0/2766169863 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f691800c9c0 con 0x7f692c104de0 2024-03-28T22:35:53.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.437+0000 7f6929600700 1 -- 172.21.15.43:0/2766169863 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f691800cb60 con 0x7f692c104de0 2024-03-28T22:35:53.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.437+0000 7f6929600700 1 -- 172.21.15.43:0/2766169863 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6918018680 con 0x7f692c104de0 2024-03-28T22:35:53.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.438+0000 7f6933600700 1 -- 172.21.15.43:0/2766169863 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f692c19b430 con 0x7f692c104de0 2024-03-28T22:35:53.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.438+0000 7f6933600700 1 -- 172.21.15.43:0/2766169863 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f692c1a76f0 con 0x7f692c104de0 2024-03-28T22:35:53.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.439+0000 7f6933600700 1 -- 172.21.15.43:0/2766169863 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f692c068f00 con 0x7f692c104de0 2024-03-28T22:35:53.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.441+0000 7f6929600700 1 -- 172.21.15.43:0/2766169863 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f6918010430 con 0x7f692c104de0 2024-03-28T22:35:53.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.442+0000 7f6929600700 1 --2- 172.21.15.43:0/2766169863 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f69140a1e90 0x7f69140a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:53.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.442+0000 7f6929600700 1 -- 172.21.15.43:0/2766169863 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f69180ce240 con 0x7f692c104de0 2024-03-28T22:35:53.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.442+0000 7f6930c00700 1 --2- 172.21.15.43:0/2766169863 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f69140a1e90 0x7f69140a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:53.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.443+0000 7f6930c00700 1 --2- 172.21.15.43:0/2766169863 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f69140a1e90 0x7f69140a4350 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6920009750 tx=0x7f692001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:53.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.446+0000 7f6929600700 1 -- 172.21.15.43:0/2766169863 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f691809b280 con 0x7f692c104de0 2024-03-28T22:35:53.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:53 smithi043 ceph-mon[31823]: pgmap v398: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:35:53.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:53 smithi172 ceph-mon[36566]: pgmap v398: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:35:53.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.667+0000 7f6933600700 1 -- 172.21.15.43:0/2766169863 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f692c062680 con 0x7f69140a1e90 2024-03-28T22:35:53.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.684+0000 7f6929600700 1 -- 172.21.15.43:0/2766169863 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340523 (secure 0 0 0) 0x7f692007b240 con 0x7f69140a1e90 2024-03-28T22:35:53.684 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:35:53.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.690+0000 7f6933600700 1 -- 172.21.15.43:0/2766169863 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f69140a1e90 msgr2=0x7f69140a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:53.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.690+0000 7f6933600700 1 --2- 172.21.15.43:0/2766169863 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f69140a1e90 0x7f69140a4350 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6920009750 tx=0x7f692001a040 comp rx=0 tx=0).stop 2024-03-28T22:35:53.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.690+0000 7f6933600700 1 -- 172.21.15.43:0/2766169863 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f692c104de0 msgr2=0x7f692c1a0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:53.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.691+0000 7f6933600700 1 --2- 172.21.15.43:0/2766169863 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f692c104de0 0x7f692c1a0d60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f691800e9f0 tx=0x7f691800ed00 comp rx=0 tx=0).stop 2024-03-28T22:35:53.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.691+0000 7f6933600700 1 -- 172.21.15.43:0/2766169863 shutdown_connections 2024-03-28T22:35:53.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.691+0000 7f6933600700 1 --2- 172.21.15.43:0/2766169863 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f69140a1e90 0x7f69140a4350 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:53.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.691+0000 7f6933600700 1 --2- 172.21.15.43:0/2766169863 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f692c10a4b0 0x7f692c19a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:53.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.691+0000 7f6933600700 1 --2- 172.21.15.43:0/2766169863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f692c104420 0x7f692c1a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:53.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.691+0000 7f6933600700 1 --2- 172.21.15.43:0/2766169863 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f692c104de0 0x7f692c1a0d60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:53.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.691+0000 7f6933600700 1 -- 172.21.15.43:0/2766169863 >> 172.21.15.43:0/2766169863 conn(0x7f692c0fff50 msgr2=0x7f692c1004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:53.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.691+0000 7f6933600700 1 -- 172.21.15.43:0/2766169863 shutdown_connections 2024-03-28T22:35:53.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:53.691+0000 7f6933600700 1 -- 172.21.15.43:0/2766169863 wait complete. 2024-03-28T22:35:53.692 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-03-28T22:35:53.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:53 smithi082 ceph-mon[36471]: pgmap v398: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:35:54.577 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":398,"stamp":"2024-03-28T22:35:52.214005+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662011,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1215,"num_read_kb":736,"num_write":433,"num_write_kb":3086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":856,"ondisk_log_size":856,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":376780,"kb_used_data":15056,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124565044,"statfs":{"total":1151940427776,"available":1151554605056,"internally_reserved":0,"allocated":15417344,"data_stored":8429051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":115,"num_read_kb":55,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005031"},"pg_stats":[{"pgid":"3.3f","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.450073+0000","last_change":"2024-03-28T22:33:28.942904+0000","last_active":"2024-03-28T22:35:30.450073+0000","last_peered":"2024-03-28T22:35:30.450073+0000","last_clean":"2024-03-28T22:35:30.450073+0000","last_became_active":"2024-03-28T22:33:28.942574+0000","last_became_peered":"2024-03-28T22:33:28.942574+0000","last_unstale":"2024-03-28T22:35:30.450073+0000","last_undegraded":"2024-03-28T22:35:30.450073+0000","last_fullsized":"2024-03-28T22:35:30.450073+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:55.323919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"94'4","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451519+0000","last_change":"2024-03-28T22:33:28.951795+0000","last_active":"2024-03-28T22:35:30.451519+0000","last_peered":"2024-03-28T22:35:30.451519+0000","last_clean":"2024-03-28T22:35:30.451519+0000","last_became_active":"2024-03-28T22:33:28.951286+0000","last_became_peered":"2024-03-28T22:33:28.951286+0000","last_unstale":"2024-03-28T22:35:30.451519+0000","last_undegraded":"2024-03-28T22:35:30.451519+0000","last_fullsized":"2024-03-28T22:35:30.451519+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:56:11.718244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451291+0000","last_change":"2024-03-28T22:33:28.949476+0000","last_active":"2024-03-28T22:35:30.451291+0000","last_peered":"2024-03-28T22:35:30.451291+0000","last_clean":"2024-03-28T22:35:30.451291+0000","last_became_active":"2024-03-28T22:33:28.949128+0000","last_became_peered":"2024-03-28T22:33:28.949128+0000","last_unstale":"2024-03-28T22:35:30.451291+0000","last_undegraded":"2024-03-28T22:35:30.451291+0000","last_fullsized":"2024-03-28T22:35:30.451291+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:49:29.052373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370703+0000","last_change":"2024-03-28T22:33:28.930715+0000","last_active":"2024-03-28T22:35:30.370703+0000","last_peered":"2024-03-28T22:35:30.370703+0000","last_clean":"2024-03-28T22:35:30.370703+0000","last_became_active":"2024-03-28T22:33:28.930265+0000","last_became_peered":"2024-03-28T22:33:28.930265+0000","last_unstale":"2024-03-28T22:35:30.370703+0000","last_undegraded":"2024-03-28T22:35:30.370703+0000","last_fullsized":"2024-03-28T22:35:30.370703+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:29:54.695671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"94'4","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.253628+0000","last_change":"2024-03-28T22:33:28.944933+0000","last_active":"2024-03-28T22:35:30.253628+0000","last_peered":"2024-03-28T22:35:30.253628+0000","last_clean":"2024-03-28T22:35:30.253628+0000","last_became_active":"2024-03-28T22:33:28.944542+0000","last_became_peered":"2024-03-28T22:33:28.944542+0000","last_unstale":"2024-03-28T22:35:30.253628+0000","last_undegraded":"2024-03-28T22:35:30.253628+0000","last_fullsized":"2024-03-28T22:35:30.253628+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:35:15.719312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"94'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.259239+0000","last_change":"2024-03-28T22:33:28.927920+0000","last_active":"2024-03-28T22:35:30.259239+0000","last_peered":"2024-03-28T22:35:30.259239+0000","last_clean":"2024-03-28T22:35:30.259239+0000","last_became_active":"2024-03-28T22:33:28.927602+0000","last_became_peered":"2024-03-28T22:33:28.927602+0000","last_unstale":"2024-03-28T22:35:30.259239+0000","last_undegraded":"2024-03-28T22:35:30.259239+0000","last_fullsized":"2024-03-28T22:35:30.259239+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:24:22.072707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"94'5","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248009+0000","last_change":"2024-03-28T22:33:28.917963+0000","last_active":"2024-03-28T22:35:30.248009+0000","last_peered":"2024-03-28T22:35:30.248009+0000","last_clean":"2024-03-28T22:35:30.248009+0000","last_became_active":"2024-03-28T22:33:28.917628+0000","last_became_peered":"2024-03-28T22:33:28.917628+0000","last_unstale":"2024-03-28T22:35:30.248009+0000","last_undegraded":"2024-03-28T22:35:30.248009+0000","last_fullsized":"2024-03-28T22:35:30.248009+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:37:09.111640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"94'9","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371107+0000","last_change":"2024-03-28T22:33:28.942632+0000","last_active":"2024-03-28T22:35:30.371107+0000","last_peered":"2024-03-28T22:35:30.371107+0000","last_clean":"2024-03-28T22:35:30.371107+0000","last_became_active":"2024-03-28T22:33:28.941972+0000","last_became_peered":"2024-03-28T22:33:28.941972+0000","last_unstale":"2024-03-28T22:35:30.371107+0000","last_undegraded":"2024-03-28T22:35:30.371107+0000","last_fullsized":"2024-03-28T22:35:30.371107+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:49:52.642314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"94'11","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448719+0000","last_change":"2024-03-28T22:33:28.924324+0000","last_active":"2024-03-28T22:35:30.448719+0000","last_peered":"2024-03-28T22:35:30.448719+0000","last_clean":"2024-03-28T22:35:30.448719+0000","last_became_active":"2024-03-28T22:33:28.924014+0000","last_became_peered":"2024-03-28T22:33:28.924014+0000","last_unstale":"2024-03-28T22:35:30.448719+0000","last_undegraded":"2024-03-28T22:35:30.448719+0000","last_fullsized":"2024-03-28T22:35:30.448719+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:17:55.172116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"94'16","reported_seq":53,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.259531+0000","last_change":"2024-03-28T22:34:29.987262+0000","last_active":"2024-03-28T22:35:30.259531+0000","last_peered":"2024-03-28T22:35:30.259531+0000","last_clean":"2024-03-28T22:35:30.259531+0000","last_became_active":"2024-03-28T22:34:29.986934+0000","last_became_peered":"2024-03-28T22:34:29.986934+0000","last_unstale":"2024-03-28T22:35:30.259531+0000","last_undegraded":"2024-03-28T22:35:30.259531+0000","last_fullsized":"2024-03-28T22:35:30.259531+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:00:41.574800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"94'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248464+0000","last_change":"2024-03-28T22:33:28.930802+0000","last_active":"2024-03-28T22:35:30.248464+0000","last_peered":"2024-03-28T22:35:30.248464+0000","last_clean":"2024-03-28T22:35:30.248464+0000","last_became_active":"2024-03-28T22:33:28.930475+0000","last_became_peered":"2024-03-28T22:33:28.930475+0000","last_unstale":"2024-03-28T22:35:30.248464+0000","last_undegraded":"2024-03-28T22:35:30.248464+0000","last_fullsized":"2024-03-28T22:35:30.248464+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:48.443803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"94'7","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370644+0000","last_change":"2024-03-28T22:33:28.927057+0000","last_active":"2024-03-28T22:35:30.370644+0000","last_peered":"2024-03-28T22:35:30.370644+0000","last_clean":"2024-03-28T22:35:30.370644+0000","last_became_active":"2024-03-28T22:33:28.926598+0000","last_became_peered":"2024-03-28T22:33:28.926598+0000","last_unstale":"2024-03-28T22:35:30.370644+0000","last_undegraded":"2024-03-28T22:35:30.370644+0000","last_fullsized":"2024-03-28T22:35:30.370644+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:21:00.710203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"94'10","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368883+0000","last_change":"2024-03-28T22:33:28.937034+0000","last_active":"2024-03-28T22:35:30.368883+0000","last_peered":"2024-03-28T22:35:30.368883+0000","last_clean":"2024-03-28T22:35:30.368883+0000","last_became_active":"2024-03-28T22:33:28.933713+0000","last_became_peered":"2024-03-28T22:33:28.933713+0000","last_unstale":"2024-03-28T22:35:30.368883+0000","last_undegraded":"2024-03-28T22:35:30.368883+0000","last_fullsized":"2024-03-28T22:35:30.368883+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:58:24.148897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.453212+0000","last_change":"2024-03-28T22:33:28.947384+0000","last_active":"2024-03-28T22:35:30.453212+0000","last_peered":"2024-03-28T22:35:30.453212+0000","last_clean":"2024-03-28T22:35:30.453212+0000","last_became_active":"2024-03-28T22:33:28.947017+0000","last_became_peered":"2024-03-28T22:33:28.947017+0000","last_unstale":"2024-03-28T22:35:30.453212+0000","last_undegraded":"2024-03-28T22:35:30.453212+0000","last_fullsized":"2024-03-28T22:35:30.453212+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:43:32.781161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451880+0000","last_change":"2024-03-28T22:33:28.950119+0000","last_active":"2024-03-28T22:35:30.451880+0000","last_peered":"2024-03-28T22:35:30.451880+0000","last_clean":"2024-03-28T22:35:30.451880+0000","last_became_active":"2024-03-28T22:33:28.949798+0000","last_became_peered":"2024-03-28T22:33:28.949798+0000","last_unstale":"2024-03-28T22:35:30.451880+0000","last_undegraded":"2024-03-28T22:35:30.451880+0000","last_fullsized":"2024-03-28T22:35:30.451880+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:16.717678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"94'9","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448318+0000","last_change":"2024-03-28T22:33:28.943255+0000","last_active":"2024-03-28T22:35:30.448318+0000","last_peered":"2024-03-28T22:35:30.448318+0000","last_clean":"2024-03-28T22:35:30.448318+0000","last_became_active":"2024-03-28T22:33:28.942923+0000","last_became_peered":"2024-03-28T22:33:28.942923+0000","last_unstale":"2024-03-28T22:35:30.448318+0000","last_undegraded":"2024-03-28T22:35:30.448318+0000","last_fullsized":"2024-03-28T22:35:30.448318+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:05:04.197635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449964+0000","last_change":"2024-03-28T22:33:28.942906+0000","last_active":"2024-03-28T22:35:30.449964+0000","last_peered":"2024-03-28T22:35:30.449964+0000","last_clean":"2024-03-28T22:35:30.449964+0000","last_became_active":"2024-03-28T22:33:28.942573+0000","last_became_peered":"2024-03-28T22:33:28.942573+0000","last_unstale":"2024-03-28T22:35:30.449964+0000","last_undegraded":"2024-03-28T22:35:30.449964+0000","last_fullsized":"2024-03-28T22:35:30.449964+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:39:17.651101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"94'10","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252948+0000","last_change":"2024-03-28T22:33:28.932075+0000","last_active":"2024-03-28T22:35:30.252948+0000","last_peered":"2024-03-28T22:35:30.252948+0000","last_clean":"2024-03-28T22:35:30.252948+0000","last_became_active":"2024-03-28T22:33:28.931689+0000","last_became_peered":"2024-03-28T22:33:28.931689+0000","last_unstale":"2024-03-28T22:35:30.252948+0000","last_undegraded":"2024-03-28T22:35:30.252948+0000","last_fullsized":"2024-03-28T22:35:30.252948+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:50:43.031082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252343+0000","last_change":"2024-03-28T22:33:28.944987+0000","last_active":"2024-03-28T22:35:30.252343+0000","last_peered":"2024-03-28T22:35:30.252343+0000","last_clean":"2024-03-28T22:35:30.252343+0000","last_became_active":"2024-03-28T22:33:28.944622+0000","last_became_peered":"2024-03-28T22:33:28.944622+0000","last_unstale":"2024-03-28T22:35:30.252343+0000","last_undegraded":"2024-03-28T22:35:30.252343+0000","last_fullsized":"2024-03-28T22:35:30.252343+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:48:58.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":29,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251775+0000","last_change":"2024-03-28T22:34:30.429709+0000","last_active":"2024-03-28T22:35:30.251775+0000","last_peered":"2024-03-28T22:35:30.251775+0000","last_clean":"2024-03-28T22:35:30.251775+0000","last_became_active":"2024-03-28T22:34:30.429122+0000","last_became_peered":"2024-03-28T22:34:30.429122+0000","last_unstale":"2024-03-28T22:35:30.251775+0000","last_undegraded":"2024-03-28T22:35:30.251775+0000","last_fullsized":"2024-03-28T22:35:30.251775+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:58:05.313669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"92'13","reported_seq":154,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:50.350706+0000","last_change":"2024-03-28T22:33:28.442524+0000","last_active":"2024-03-28T22:35:50.350706+0000","last_peered":"2024-03-28T22:35:50.350706+0000","last_clean":"2024-03-28T22:35:50.350706+0000","last_became_active":"2024-03-28T22:33:28.442169+0000","last_became_peered":"2024-03-28T22:33:28.442169+0000","last_unstale":"2024-03-28T22:35:50.350706+0000","last_undegraded":"2024-03-28T22:35:50.350706+0000","last_fullsized":"2024-03-28T22:35:50.350706+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:08:47.101119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":107,"num_read_kb":107,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"94'5","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371822+0000","last_change":"2024-03-28T22:33:28.946697+0000","last_active":"2024-03-28T22:35:30.371822+0000","last_peered":"2024-03-28T22:35:30.371822+0000","last_clean":"2024-03-28T22:35:30.371822+0000","last_became_active":"2024-03-28T22:33:28.946303+0000","last_became_peered":"2024-03-28T22:33:28.946303+0000","last_unstale":"2024-03-28T22:35:30.371822+0000","last_undegraded":"2024-03-28T22:35:30.371822+0000","last_fullsized":"2024-03-28T22:35:30.371822+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:32:01.286073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370317+0000","last_change":"2024-03-28T22:34:30.429579+0000","last_active":"2024-03-28T22:35:30.370317+0000","last_peered":"2024-03-28T22:35:30.370317+0000","last_clean":"2024-03-28T22:35:30.370317+0000","last_became_active":"2024-03-28T22:34:30.429054+0000","last_became_peered":"2024-03-28T22:34:30.429054+0000","last_unstale":"2024-03-28T22:35:30.370317+0000","last_undegraded":"2024-03-28T22:35:30.370317+0000","last_fullsized":"2024-03-28T22:35:30.370317+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:03:46.492786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"92'2","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.372174+0000","last_change":"2024-03-28T22:33:28.457661+0000","last_active":"2024-03-28T22:35:30.372174+0000","last_peered":"2024-03-28T22:35:30.372174+0000","last_clean":"2024-03-28T22:35:30.372174+0000","last_became_active":"2024-03-28T22:33:28.457032+0000","last_became_peered":"2024-03-28T22:33:28.457032+0000","last_unstale":"2024-03-28T22:35:30.372174+0000","last_undegraded":"2024-03-28T22:35:30.372174+0000","last_fullsized":"2024-03-28T22:35:30.372174+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:43:09.820630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"94'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.453641+0000","last_change":"2024-03-28T22:33:28.954252+0000","last_active":"2024-03-28T22:35:30.453641+0000","last_peered":"2024-03-28T22:35:30.453641+0000","last_clean":"2024-03-28T22:35:30.453641+0000","last_became_active":"2024-03-28T22:33:28.953922+0000","last_became_peered":"2024-03-28T22:33:28.953922+0000","last_unstale":"2024-03-28T22:35:30.453641+0000","last_undegraded":"2024-03-28T22:35:30.453641+0000","last_fullsized":"2024-03-28T22:35:30.453641+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:18:47.097859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370557+0000","last_change":"2024-03-28T22:33:34.029129+0000","last_active":"2024-03-28T22:35:30.370557+0000","last_peered":"2024-03-28T22:35:30.370557+0000","last_clean":"2024-03-28T22:35:30.370557+0000","last_became_active":"2024-03-28T22:33:34.028452+0000","last_became_peered":"2024-03-28T22:33:34.028452+0000","last_unstale":"2024-03-28T22:35:30.370557+0000","last_undegraded":"2024-03-28T22:35:30.370557+0000","last_fullsized":"2024-03-28T22:35:30.370557+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:28.982773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"92'12","reported_seq":46,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254159+0000","last_change":"2024-03-28T22:33:28.439568+0000","last_active":"2024-03-28T22:35:30.254159+0000","last_peered":"2024-03-28T22:35:30.254159+0000","last_clean":"2024-03-28T22:35:30.254159+0000","last_became_active":"2024-03-28T22:33:28.438887+0000","last_became_peered":"2024-03-28T22:33:28.438887+0000","last_unstale":"2024-03-28T22:35:30.254159+0000","last_undegraded":"2024-03-28T22:35:30.254159+0000","last_fullsized":"2024-03-28T22:35:30.254159+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:56:43.617621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"94'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448533+0000","last_change":"2024-03-28T22:33:28.938311+0000","last_active":"2024-03-28T22:35:30.448533+0000","last_peered":"2024-03-28T22:35:30.448533+0000","last_clean":"2024-03-28T22:35:30.448533+0000","last_became_active":"2024-03-28T22:33:28.937997+0000","last_became_peered":"2024-03-28T22:33:28.937997+0000","last_unstale":"2024-03-28T22:35:30.448533+0000","last_undegraded":"2024-03-28T22:35:30.448533+0000","last_fullsized":"2024-03-28T22:35:30.448533+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:54:44.001545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370378+0000","last_change":"2024-03-28T22:33:34.016931+0000","last_active":"2024-03-28T22:35:30.370378+0000","last_peered":"2024-03-28T22:35:30.370378+0000","last_clean":"2024-03-28T22:35:30.370378+0000","last_became_active":"2024-03-28T22:33:34.016559+0000","last_became_peered":"2024-03-28T22:33:34.016559+0000","last_unstale":"2024-03-28T22:35:30.370378+0000","last_undegraded":"2024-03-28T22:35:30.370378+0000","last_fullsized":"2024-03-28T22:35:30.370378+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:08:31.252351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"92'5","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254044+0000","last_change":"2024-03-28T22:33:28.433166+0000","last_active":"2024-03-28T22:35:30.254044+0000","last_peered":"2024-03-28T22:35:30.254044+0000","last_clean":"2024-03-28T22:35:30.254044+0000","last_became_active":"2024-03-28T22:33:28.432716+0000","last_became_peered":"2024-03-28T22:33:28.432716+0000","last_unstale":"2024-03-28T22:35:30.254044+0000","last_undegraded":"2024-03-28T22:35:30.254044+0000","last_fullsized":"2024-03-28T22:35:30.254044+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:42:41.613583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"94'15","reported_seq":46,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275481+0000","last_change":"2024-03-28T22:33:28.923307+0000","last_active":"2024-03-28T22:35:30.275481+0000","last_peered":"2024-03-28T22:35:30.275481+0000","last_clean":"2024-03-28T22:35:30.275481+0000","last_became_active":"2024-03-28T22:33:28.922499+0000","last_became_peered":"2024-03-28T22:33:28.922499+0000","last_unstale":"2024-03-28T22:35:30.275481+0000","last_undegraded":"2024-03-28T22:35:30.275481+0000","last_fullsized":"2024-03-28T22:35:30.275481+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:48:48.514482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252136+0000","last_change":"2024-03-28T22:33:34.002209+0000","last_active":"2024-03-28T22:35:30.252136+0000","last_peered":"2024-03-28T22:35:30.252136+0000","last_clean":"2024-03-28T22:35:30.252136+0000","last_became_active":"2024-03-28T22:33:34.001746+0000","last_became_peered":"2024-03-28T22:33:34.001746+0000","last_unstale":"2024-03-28T22:35:30.252136+0000","last_undegraded":"2024-03-28T22:35:30.252136+0000","last_fullsized":"2024-03-28T22:35:30.252136+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:02:02.333328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"92'5","reported_seq":146,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:50.371524+0000","last_change":"2024-03-28T22:33:28.435380+0000","last_active":"2024-03-28T22:35:50.371524+0000","last_peered":"2024-03-28T22:35:50.371524+0000","last_clean":"2024-03-28T22:35:50.371524+0000","last_became_active":"2024-03-28T22:33:28.434898+0000","last_became_peered":"2024-03-28T22:33:28.434898+0000","last_unstale":"2024-03-28T22:35:50.371524+0000","last_undegraded":"2024-03-28T22:35:50.371524+0000","last_fullsized":"2024-03-28T22:35:50.371524+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:58:06.810394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":108,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"94'7","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254211+0000","last_change":"2024-03-28T22:34:30.121370+0000","last_active":"2024-03-28T22:35:30.254211+0000","last_peered":"2024-03-28T22:35:30.254211+0000","last_clean":"2024-03-28T22:35:30.254211+0000","last_became_active":"2024-03-28T22:34:30.121086+0000","last_became_peered":"2024-03-28T22:34:30.121086+0000","last_unstale":"2024-03-28T22:35:30.254211+0000","last_undegraded":"2024-03-28T22:35:30.254211+0000","last_fullsized":"2024-03-28T22:35:30.254211+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:26:24.244663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252008+0000","last_change":"2024-03-28T22:33:34.031350+0000","last_active":"2024-03-28T22:35:30.252008+0000","last_peered":"2024-03-28T22:35:30.252008+0000","last_clean":"2024-03-28T22:35:30.252008+0000","last_became_active":"2024-03-28T22:33:34.030946+0000","last_became_peered":"2024-03-28T22:33:34.030946+0000","last_unstale":"2024-03-28T22:35:30.252008+0000","last_undegraded":"2024-03-28T22:35:30.252008+0000","last_fullsized":"2024-03-28T22:35:30.252008+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:54:22.907363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"92'7","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.259165+0000","last_change":"2024-03-28T22:33:28.446197+0000","last_active":"2024-03-28T22:35:30.259165+0000","last_peered":"2024-03-28T22:35:30.259165+0000","last_clean":"2024-03-28T22:35:30.259165+0000","last_became_active":"2024-03-28T22:33:28.445891+0000","last_became_peered":"2024-03-28T22:33:28.445891+0000","last_unstale":"2024-03-28T22:35:30.259165+0000","last_undegraded":"2024-03-28T22:35:30.259165+0000","last_fullsized":"2024-03-28T22:35:30.259165+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:58:57.385302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275539+0000","last_change":"2024-03-28T22:33:28.955651+0000","last_active":"2024-03-28T22:35:30.275539+0000","last_peered":"2024-03-28T22:35:30.275539+0000","last_clean":"2024-03-28T22:35:30.275539+0000","last_became_active":"2024-03-28T22:33:28.955276+0000","last_became_peered":"2024-03-28T22:33:28.955276+0000","last_unstale":"2024-03-28T22:35:30.275539+0000","last_undegraded":"2024-03-28T22:35:30.275539+0000","last_fullsized":"2024-03-28T22:35:30.275539+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:36:13.455327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449809+0000","last_change":"2024-03-28T22:33:34.017520+0000","last_active":"2024-03-28T22:35:30.449809+0000","last_peered":"2024-03-28T22:35:30.449809+0000","last_clean":"2024-03-28T22:35:30.449809+0000","last_became_active":"2024-03-28T22:33:34.017022+0000","last_became_peered":"2024-03-28T22:33:34.017022+0000","last_unstale":"2024-03-28T22:35:30.449809+0000","last_undegraded":"2024-03-28T22:35:30.449809+0000","last_fullsized":"2024-03-28T22:35:30.449809+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:41:17.431581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"92'3","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368118+0000","last_change":"2024-03-28T22:33:28.445401+0000","last_active":"2024-03-28T22:35:30.368118+0000","last_peered":"2024-03-28T22:35:30.368118+0000","last_clean":"2024-03-28T22:35:30.368118+0000","last_became_active":"2024-03-28T22:33:28.444237+0000","last_became_peered":"2024-03-28T22:33:28.444237+0000","last_unstale":"2024-03-28T22:35:30.368118+0000","last_undegraded":"2024-03-28T22:35:30.368118+0000","last_fullsized":"2024-03-28T22:35:30.368118+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:18:12.820890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"94'8","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.250889+0000","last_change":"2024-03-28T22:34:30.156248+0000","last_active":"2024-03-28T22:35:30.250889+0000","last_peered":"2024-03-28T22:35:30.250889+0000","last_clean":"2024-03-28T22:35:30.250889+0000","last_became_active":"2024-03-28T22:34:30.155056+0000","last_became_peered":"2024-03-28T22:34:30.155056+0000","last_unstale":"2024-03-28T22:35:30.250889+0000","last_undegraded":"2024-03-28T22:35:30.250889+0000","last_fullsized":"2024-03-28T22:35:30.250889+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:28:56.469728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451480+0000","last_change":"2024-03-28T22:33:34.019899+0000","last_active":"2024-03-28T22:35:30.451480+0000","last_peered":"2024-03-28T22:35:30.451480+0000","last_clean":"2024-03-28T22:35:30.451480+0000","last_became_active":"2024-03-28T22:33:34.019392+0000","last_became_peered":"2024-03-28T22:33:34.019392+0000","last_unstale":"2024-03-28T22:35:30.451480+0000","last_undegraded":"2024-03-28T22:35:30.451480+0000","last_fullsized":"2024-03-28T22:35:30.451480+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:08:38.633797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"92'3","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275054+0000","last_change":"2024-03-28T22:33:28.433243+0000","last_active":"2024-03-28T22:35:30.275054+0000","last_peered":"2024-03-28T22:35:30.275054+0000","last_clean":"2024-03-28T22:35:30.275054+0000","last_became_active":"2024-03-28T22:33:28.432927+0000","last_became_peered":"2024-03-28T22:33:28.432927+0000","last_unstale":"2024-03-28T22:35:30.275054+0000","last_undegraded":"2024-03-28T22:35:30.275054+0000","last_fullsized":"2024-03-28T22:35:30.275054+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:40:43.444229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"94'11","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251179+0000","last_change":"2024-03-28T22:33:28.948111+0000","last_active":"2024-03-28T22:35:30.251179+0000","last_peered":"2024-03-28T22:35:30.251179+0000","last_clean":"2024-03-28T22:35:30.251179+0000","last_became_active":"2024-03-28T22:33:28.944649+0000","last_became_peered":"2024-03-28T22:33:28.944649+0000","last_unstale":"2024-03-28T22:35:30.251179+0000","last_undegraded":"2024-03-28T22:35:30.251179+0000","last_fullsized":"2024-03-28T22:35:30.251179+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:57:11.483037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371445+0000","last_change":"2024-03-28T22:33:34.000837+0000","last_active":"2024-03-28T22:35:30.371445+0000","last_peered":"2024-03-28T22:35:30.371445+0000","last_clean":"2024-03-28T22:35:30.371445+0000","last_became_active":"2024-03-28T22:33:34.000120+0000","last_became_peered":"2024-03-28T22:33:34.000120+0000","last_unstale":"2024-03-28T22:35:30.371445+0000","last_undegraded":"2024-03-28T22:35:30.371445+0000","last_fullsized":"2024-03-28T22:35:30.371445+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:14:32.028952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"92'4","reported_seq":245,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:49.291864+0000","last_change":"2024-03-28T22:33:28.436985+0000","last_active":"2024-03-28T22:35:49.291864+0000","last_peered":"2024-03-28T22:35:49.291864+0000","last_clean":"2024-03-28T22:35:49.291864+0000","last_became_active":"2024-03-28T22:33:28.436625+0000","last_became_peered":"2024-03-28T22:33:28.436625+0000","last_unstale":"2024-03-28T22:35:49.291864+0000","last_undegraded":"2024-03-28T22:35:49.291864+0000","last_fullsized":"2024-03-28T22:35:49.291864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:50:45.775191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":208,"num_read_kb":208,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"94'10","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448868+0000","last_change":"2024-03-28T22:33:28.933265+0000","last_active":"2024-03-28T22:35:30.448868+0000","last_peered":"2024-03-28T22:35:30.448868+0000","last_clean":"2024-03-28T22:35:30.448868+0000","last_became_active":"2024-03-28T22:33:28.932954+0000","last_became_peered":"2024-03-28T22:33:28.932954+0000","last_unstale":"2024-03-28T22:35:30.448868+0000","last_undegraded":"2024-03-28T22:35:30.448868+0000","last_fullsized":"2024-03-28T22:35:30.448868+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:37.039055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251115+0000","last_change":"2024-03-28T22:33:34.024529+0000","last_active":"2024-03-28T22:35:30.251115+0000","last_peered":"2024-03-28T22:35:30.251115+0000","last_clean":"2024-03-28T22:35:30.251115+0000","last_became_active":"2024-03-28T22:33:34.020949+0000","last_became_peered":"2024-03-28T22:33:34.020949+0000","last_unstale":"2024-03-28T22:35:30.251115+0000","last_undegraded":"2024-03-28T22:35:30.251115+0000","last_fullsized":"2024-03-28T22:35:30.251115+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:28:52.039684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"92'4","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248336+0000","last_change":"2024-03-28T22:33:28.432599+0000","last_active":"2024-03-28T22:35:30.248336+0000","last_peered":"2024-03-28T22:35:30.248336+0000","last_clean":"2024-03-28T22:35:30.248336+0000","last_became_active":"2024-03-28T22:33:28.430710+0000","last_became_peered":"2024-03-28T22:33:28.430710+0000","last_unstale":"2024-03-28T22:35:30.248336+0000","last_undegraded":"2024-03-28T22:35:30.248336+0000","last_fullsized":"2024-03-28T22:35:30.248336+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:54.174747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"94'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.453785+0000","last_change":"2024-03-28T22:33:28.955441+0000","last_active":"2024-03-28T22:35:30.453785+0000","last_peered":"2024-03-28T22:35:30.453785+0000","last_clean":"2024-03-28T22:35:30.453785+0000","last_became_active":"2024-03-28T22:33:28.953269+0000","last_became_peered":"2024-03-28T22:33:28.953269+0000","last_unstale":"2024-03-28T22:35:30.453785+0000","last_undegraded":"2024-03-28T22:35:30.453785+0000","last_fullsized":"2024-03-28T22:35:30.453785+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:47:04.128605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370663+0000","last_change":"2024-03-28T22:33:34.028791+0000","last_active":"2024-03-28T22:35:30.370663+0000","last_peered":"2024-03-28T22:35:30.370663+0000","last_clean":"2024-03-28T22:35:30.370663+0000","last_became_active":"2024-03-28T22:33:34.028361+0000","last_became_peered":"2024-03-28T22:33:34.028361+0000","last_unstale":"2024-03-28T22:35:30.370663+0000","last_undegraded":"2024-03-28T22:35:30.370663+0000","last_fullsized":"2024-03-28T22:35:30.370663+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:42:21.264459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"92'5","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275230+0000","last_change":"2024-03-28T22:33:28.455353+0000","last_active":"2024-03-28T22:35:30.275230+0000","last_peered":"2024-03-28T22:35:30.275230+0000","last_clean":"2024-03-28T22:35:30.275230+0000","last_became_active":"2024-03-28T22:33:28.455090+0000","last_became_peered":"2024-03-28T22:33:28.455090+0000","last_unstale":"2024-03-28T22:35:30.275230+0000","last_undegraded":"2024-03-28T22:35:30.275230+0000","last_fullsized":"2024-03-28T22:35:30.275230+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:19:22.496600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"94'11","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448743+0000","last_change":"2024-03-28T22:33:28.928875+0000","last_active":"2024-03-28T22:35:30.448743+0000","last_peered":"2024-03-28T22:35:30.448743+0000","last_clean":"2024-03-28T22:35:30.448743+0000","last_became_active":"2024-03-28T22:33:28.928449+0000","last_became_peered":"2024-03-28T22:33:28.928449+0000","last_unstale":"2024-03-28T22:35:30.448743+0000","last_undegraded":"2024-03-28T22:35:30.448743+0000","last_fullsized":"2024-03-28T22:35:30.448743+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:34:00.402100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251201+0000","last_change":"2024-03-28T22:33:34.012739+0000","last_active":"2024-03-28T22:35:30.251201+0000","last_peered":"2024-03-28T22:35:30.251201+0000","last_clean":"2024-03-28T22:35:30.251201+0000","last_became_active":"2024-03-28T22:33:34.012262+0000","last_became_peered":"2024-03-28T22:33:34.012262+0000","last_unstale":"2024-03-28T22:35:30.251201+0000","last_undegraded":"2024-03-28T22:35:30.251201+0000","last_fullsized":"2024-03-28T22:35:30.251201+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:24:48.351037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'38","reported_seq":287,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:49.292138+0000","last_change":"2024-03-28T22:33:28.430606+0000","last_active":"2024-03-28T22:35:49.292138+0000","last_peered":"2024-03-28T22:35:49.292138+0000","last_clean":"2024-03-28T22:35:49.292138+0000","last_became_active":"2024-03-28T22:33:28.430233+0000","last_became_peered":"2024-03-28T22:33:28.430233+0000","last_unstale":"2024-03-28T22:35:49.292138+0000","last_undegraded":"2024-03-28T22:35:49.292138+0000","last_fullsized":"2024-03-28T22:35:49.292138+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:49.000569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64455,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":215,"num_read_kb":171,"num_write":40,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"94'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254029+0000","last_change":"2024-03-28T22:33:28.931167+0000","last_active":"2024-03-28T22:35:30.254029+0000","last_peered":"2024-03-28T22:35:30.254029+0000","last_clean":"2024-03-28T22:35:30.254029+0000","last_became_active":"2024-03-28T22:33:28.930824+0000","last_became_peered":"2024-03-28T22:33:28.930824+0000","last_unstale":"2024-03-28T22:35:30.254029+0000","last_undegraded":"2024-03-28T22:35:30.254029+0000","last_fullsized":"2024-03-28T22:35:30.254029+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:46:49.102791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275795+0000","last_change":"2024-03-28T22:33:34.023056+0000","last_active":"2024-03-28T22:35:30.275795+0000","last_peered":"2024-03-28T22:35:30.275795+0000","last_clean":"2024-03-28T22:35:30.275795+0000","last_became_active":"2024-03-28T22:33:34.022612+0000","last_became_peered":"2024-03-28T22:33:34.022612+0000","last_unstale":"2024-03-28T22:35:30.275795+0000","last_undegraded":"2024-03-28T22:35:30.275795+0000","last_fullsized":"2024-03-28T22:35:30.275795+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:52:31.936008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"92'9","reported_seq":45,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252111+0000","last_change":"2024-03-28T22:33:28.430963+0000","last_active":"2024-03-28T22:35:30.252111+0000","last_peered":"2024-03-28T22:35:30.252111+0000","last_clean":"2024-03-28T22:35:30.252111+0000","last_became_active":"2024-03-28T22:33:28.430632+0000","last_became_peered":"2024-03-28T22:33:28.430632+0000","last_unstale":"2024-03-28T22:35:30.252111+0000","last_undegraded":"2024-03-28T22:35:30.252111+0000","last_fullsized":"2024-03-28T22:35:30.252111+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:30:23.104769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"94'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368323+0000","last_change":"2024-03-28T22:33:28.954105+0000","last_active":"2024-03-28T22:35:30.368323+0000","last_peered":"2024-03-28T22:35:30.368323+0000","last_clean":"2024-03-28T22:35:30.368323+0000","last_became_active":"2024-03-28T22:33:28.953768+0000","last_became_peered":"2024-03-28T22:33:28.953768+0000","last_unstale":"2024-03-28T22:35:30.368323+0000","last_undegraded":"2024-03-28T22:35:30.368323+0000","last_fullsized":"2024-03-28T22:35:30.368323+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:01:25.184523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449514+0000","last_change":"2024-03-28T22:33:34.027976+0000","last_active":"2024-03-28T22:35:30.449514+0000","last_peered":"2024-03-28T22:35:30.449514+0000","last_clean":"2024-03-28T22:35:30.449514+0000","last_became_active":"2024-03-28T22:33:34.027509+0000","last_became_peered":"2024-03-28T22:33:34.027509+0000","last_unstale":"2024-03-28T22:35:30.449514+0000","last_undegraded":"2024-03-28T22:35:30.449514+0000","last_fullsized":"2024-03-28T22:35:30.449514+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:25:41.908441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"92'5","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275307+0000","last_change":"2024-03-28T22:33:28.433283+0000","last_active":"2024-03-28T22:35:30.275307+0000","last_peered":"2024-03-28T22:35:30.275307+0000","last_clean":"2024-03-28T22:35:30.275307+0000","last_became_active":"2024-03-28T22:33:28.432954+0000","last_became_peered":"2024-03-28T22:33:28.432954+0000","last_unstale":"2024-03-28T22:35:30.275307+0000","last_undegraded":"2024-03-28T22:35:30.275307+0000","last_fullsized":"2024-03-28T22:35:30.275307+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:08:34.480459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"94'15","reported_seq":46,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248416+0000","last_change":"2024-03-28T22:33:28.939016+0000","last_active":"2024-03-28T22:35:30.248416+0000","last_peered":"2024-03-28T22:35:30.248416+0000","last_clean":"2024-03-28T22:35:30.248416+0000","last_became_active":"2024-03-28T22:33:28.938695+0000","last_became_peered":"2024-03-28T22:33:28.938695+0000","last_unstale":"2024-03-28T22:35:30.248416+0000","last_undegraded":"2024-03-28T22:35:30.248416+0000","last_fullsized":"2024-03-28T22:35:30.248416+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:26:53.264559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252656+0000","last_change":"2024-03-28T22:33:34.030127+0000","last_active":"2024-03-28T22:35:30.252656+0000","last_peered":"2024-03-28T22:35:30.252656+0000","last_clean":"2024-03-28T22:35:30.252656+0000","last_became_active":"2024-03-28T22:33:34.028056+0000","last_became_peered":"2024-03-28T22:33:34.028056+0000","last_unstale":"2024-03-28T22:35:30.252656+0000","last_undegraded":"2024-03-28T22:35:30.252656+0000","last_fullsized":"2024-03-28T22:35:30.252656+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:29:41.785013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'7","reported_seq":149,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:51.424497+0000","last_change":"2024-03-28T22:33:28.435350+0000","last_active":"2024-03-28T22:35:51.424497+0000","last_peered":"2024-03-28T22:35:51.424497+0000","last_clean":"2024-03-28T22:35:51.424497+0000","last_became_active":"2024-03-28T22:33:28.434954+0000","last_became_peered":"2024-03-28T22:33:28.434954+0000","last_unstale":"2024-03-28T22:35:51.424497+0000","last_undegraded":"2024-03-28T22:35:51.424497+0000","last_fullsized":"2024-03-28T22:35:51.424497+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:04.969596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":110,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"94'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248078+0000","last_change":"2024-03-28T22:33:28.921219+0000","last_active":"2024-03-28T22:35:30.248078+0000","last_peered":"2024-03-28T22:35:30.248078+0000","last_clean":"2024-03-28T22:35:30.248078+0000","last_became_active":"2024-03-28T22:33:28.920664+0000","last_became_peered":"2024-03-28T22:33:28.920664+0000","last_unstale":"2024-03-28T22:35:30.248078+0000","last_undegraded":"2024-03-28T22:35:30.248078+0000","last_fullsized":"2024-03-28T22:35:30.248078+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:48:55.032058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369304+0000","last_change":"2024-03-28T22:33:34.018101+0000","last_active":"2024-03-28T22:35:30.369304+0000","last_peered":"2024-03-28T22:35:30.369304+0000","last_clean":"2024-03-28T22:35:30.369304+0000","last_became_active":"2024-03-28T22:33:34.017260+0000","last_became_peered":"2024-03-28T22:33:34.017260+0000","last_unstale":"2024-03-28T22:35:30.369304+0000","last_undegraded":"2024-03-28T22:35:30.369304+0000","last_fullsized":"2024-03-28T22:35:30.369304+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:48:53.178090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"92'17","reported_seq":52,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.253595+0000","last_change":"2024-03-28T22:33:28.450232+0000","last_active":"2024-03-28T22:35:30.253595+0000","last_peered":"2024-03-28T22:35:30.253595+0000","last_clean":"2024-03-28T22:35:30.253595+0000","last_became_active":"2024-03-28T22:33:28.449958+0000","last_became_peered":"2024-03-28T22:33:28.449958+0000","last_unstale":"2024-03-28T22:35:30.253595+0000","last_undegraded":"2024-03-28T22:35:30.253595+0000","last_fullsized":"2024-03-28T22:35:30.253595+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:01:42.130260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"94'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275235+0000","last_change":"2024-03-28T22:33:28.926263+0000","last_active":"2024-03-28T22:35:30.275235+0000","last_peered":"2024-03-28T22:35:30.275235+0000","last_clean":"2024-03-28T22:35:30.275235+0000","last_became_active":"2024-03-28T22:33:28.925927+0000","last_became_peered":"2024-03-28T22:33:28.925927+0000","last_unstale":"2024-03-28T22:35:30.275235+0000","last_undegraded":"2024-03-28T22:35:30.275235+0000","last_fullsized":"2024-03-28T22:35:30.275235+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:41:42.477694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.455002+0000","last_change":"2024-03-28T22:35:30.455002+0000","last_active":"2024-03-28T22:35:30.455002+0000","last_peered":"2024-03-28T22:35:30.455002+0000","last_clean":"2024-03-28T22:35:30.455002+0000","last_became_active":"2024-03-28T22:35:30.454479+0000","last_became_peered":"2024-03-28T22:35:30.454479+0000","last_unstale":"2024-03-28T22:35:30.455002+0000","last_undegraded":"2024-03-28T22:35:30.455002+0000","last_fullsized":"2024-03-28T22:35:30.455002+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:57:06.300031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,3,1],"acting":[10,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.f","version":"92'2","reported_seq":21,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449758+0000","last_change":"2024-03-28T22:34:29.986650+0000","last_active":"2024-03-28T22:35:30.449758+0000","last_peered":"2024-03-28T22:35:30.449758+0000","last_clean":"2024-03-28T22:35:30.449758+0000","last_became_active":"2024-03-28T22:34:29.981875+0000","last_became_peered":"2024-03-28T22:34:29.981875+0000","last_unstale":"2024-03-28T22:35:30.449758+0000","last_undegraded":"2024-03-28T22:35:30.449758+0000","last_fullsized":"2024-03-28T22:35:30.449758+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:45:24.166133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"94'5","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449947+0000","last_change":"2024-03-28T22:34:29.982815+0000","last_active":"2024-03-28T22:35:30.449947+0000","last_peered":"2024-03-28T22:35:30.449947+0000","last_clean":"2024-03-28T22:35:30.449947+0000","last_became_active":"2024-03-28T22:34:29.982458+0000","last_became_peered":"2024-03-28T22:34:29.982458+0000","last_unstale":"2024-03-28T22:35:30.449947+0000","last_undegraded":"2024-03-28T22:35:30.449947+0000","last_fullsized":"2024-03-28T22:35:30.449947+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:11:30.617181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":30,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451505+0000","last_change":"2024-03-28T22:35:30.451063+0000","last_active":"2024-03-28T22:35:30.451505+0000","last_peered":"2024-03-28T22:35:30.451505+0000","last_clean":"2024-03-28T22:35:30.451505+0000","last_became_active":"2024-03-28T22:35:30.450525+0000","last_became_peered":"2024-03-28T22:35:30.450525+0000","last_unstale":"2024-03-28T22:35:30.451505+0000","last_undegraded":"2024-03-28T22:35:30.451505+0000","last_fullsized":"2024-03-28T22:35:30.451505+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:35:29.004834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"92'1","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252202+0000","last_change":"2024-03-28T22:34:29.992884+0000","last_active":"2024-03-28T22:35:30.252202+0000","last_peered":"2024-03-28T22:35:30.252202+0000","last_clean":"2024-03-28T22:35:30.252202+0000","last_became_active":"2024-03-28T22:34:29.991657+0000","last_became_peered":"2024-03-28T22:34:29.991657+0000","last_unstale":"2024-03-28T22:35:30.252202+0000","last_undegraded":"2024-03-28T22:35:30.252202+0000","last_fullsized":"2024-03-28T22:35:30.252202+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:25:14.540783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"94'4","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.259438+0000","last_change":"2024-03-28T22:33:28.942097+0000","last_active":"2024-03-28T22:35:30.259438+0000","last_peered":"2024-03-28T22:35:30.259438+0000","last_clean":"2024-03-28T22:35:30.259438+0000","last_became_active":"2024-03-28T22:33:28.941812+0000","last_became_peered":"2024-03-28T22:33:28.941812+0000","last_unstale":"2024-03-28T22:35:30.259438+0000","last_undegraded":"2024-03-28T22:35:30.259438+0000","last_fullsized":"2024-03-28T22:35:30.259438+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:36:10.204216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369177+0000","last_change":"2024-03-28T22:33:34.023171+0000","last_active":"2024-03-28T22:35:30.369177+0000","last_peered":"2024-03-28T22:35:30.369177+0000","last_clean":"2024-03-28T22:35:30.369177+0000","last_became_active":"2024-03-28T22:33:34.021313+0000","last_became_peered":"2024-03-28T22:33:34.021313+0000","last_unstale":"2024-03-28T22:35:30.369177+0000","last_undegraded":"2024-03-28T22:35:30.369177+0000","last_fullsized":"2024-03-28T22:35:30.369177+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:29:42.592761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"92'5","reported_seq":143,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:47.346395+0000","last_change":"2024-03-28T22:33:28.449133+0000","last_active":"2024-03-28T22:35:47.346395+0000","last_peered":"2024-03-28T22:35:47.346395+0000","last_clean":"2024-03-28T22:35:47.346395+0000","last_became_active":"2024-03-28T22:33:28.448743+0000","last_became_peered":"2024-03-28T22:33:28.448743+0000","last_unstale":"2024-03-28T22:35:47.346395+0000","last_undegraded":"2024-03-28T22:35:47.346395+0000","last_fullsized":"2024-03-28T22:35:47.346395+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:09:37.119736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":104,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.259332+0000","last_change":"2024-03-28T22:33:28.944198+0000","last_active":"2024-03-28T22:35:30.259332+0000","last_peered":"2024-03-28T22:35:30.259332+0000","last_clean":"2024-03-28T22:35:30.259332+0000","last_became_active":"2024-03-28T22:33:28.943818+0000","last_became_peered":"2024-03-28T22:33:28.943818+0000","last_unstale":"2024-03-28T22:35:30.259332+0000","last_undegraded":"2024-03-28T22:35:30.259332+0000","last_fullsized":"2024-03-28T22:35:30.259332+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:31:03.791364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369583+0000","last_change":"2024-03-28T22:33:34.015499+0000","last_active":"2024-03-28T22:35:30.369583+0000","last_peered":"2024-03-28T22:35:30.369583+0000","last_clean":"2024-03-28T22:35:30.369583+0000","last_became_active":"2024-03-28T22:33:34.015013+0000","last_became_peered":"2024-03-28T22:33:34.015013+0000","last_unstale":"2024-03-28T22:35:30.369583+0000","last_undegraded":"2024-03-28T22:35:30.369583+0000","last_fullsized":"2024-03-28T22:35:30.369583+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:34:30.582198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"94'8","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275296+0000","last_change":"2024-03-28T22:33:28.926732+0000","last_active":"2024-03-28T22:35:30.275296+0000","last_peered":"2024-03-28T22:35:30.275296+0000","last_clean":"2024-03-28T22:35:30.275296+0000","last_became_active":"2024-03-28T22:33:28.926366+0000","last_became_peered":"2024-03-28T22:33:28.926366+0000","last_unstale":"2024-03-28T22:35:30.275296+0000","last_undegraded":"2024-03-28T22:35:30.275296+0000","last_fullsized":"2024-03-28T22:35:30.275296+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:19:10.769141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'6","reported_seq":143,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:46.411409+0000","last_change":"2024-03-28T22:33:28.457400+0000","last_active":"2024-03-28T22:35:46.411409+0000","last_peered":"2024-03-28T22:35:46.411409+0000","last_clean":"2024-03-28T22:35:46.411409+0000","last_became_active":"2024-03-28T22:33:28.456994+0000","last_became_peered":"2024-03-28T22:33:28.456994+0000","last_unstale":"2024-03-28T22:35:46.411409+0000","last_undegraded":"2024-03-28T22:35:46.411409+0000","last_fullsized":"2024-03-28T22:35:46.411409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:32:16.438540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":103,"num_read_kb":103,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"92'1","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248405+0000","last_change":"2024-03-28T22:33:34.027746+0000","last_active":"2024-03-28T22:35:30.248405+0000","last_peered":"2024-03-28T22:35:30.248405+0000","last_clean":"2024-03-28T22:35:30.248405+0000","last_became_active":"2024-03-28T22:33:34.027419+0000","last_became_peered":"2024-03-28T22:33:34.027419+0000","last_unstale":"2024-03-28T22:35:30.248405+0000","last_undegraded":"2024-03-28T22:35:30.248405+0000","last_fullsized":"2024-03-28T22:35:30.248405+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:55:43.686428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371544+0000","last_change":"2024-03-28T22:33:28.952393+0000","last_active":"2024-03-28T22:35:30.371544+0000","last_peered":"2024-03-28T22:35:30.371544+0000","last_clean":"2024-03-28T22:35:30.371544+0000","last_became_active":"2024-03-28T22:33:28.951951+0000","last_became_peered":"2024-03-28T22:33:28.951951+0000","last_unstale":"2024-03-28T22:35:30.371544+0000","last_undegraded":"2024-03-28T22:35:30.371544+0000","last_fullsized":"2024-03-28T22:35:30.371544+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:09:25.808842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"92'6","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370674+0000","last_change":"2024-03-28T22:33:28.437177+0000","last_active":"2024-03-28T22:35:30.370674+0000","last_peered":"2024-03-28T22:35:30.370674+0000","last_clean":"2024-03-28T22:35:30.370674+0000","last_became_active":"2024-03-28T22:33:28.436801+0000","last_became_peered":"2024-03-28T22:33:28.436801+0000","last_unstale":"2024-03-28T22:35:30.370674+0000","last_undegraded":"2024-03-28T22:35:30.370674+0000","last_fullsized":"2024-03-28T22:35:30.370674+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:24:52.549490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":371,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.247934+0000","last_change":"2024-03-28T22:32:07.867336+0000","last_active":"2024-03-28T22:35:30.247934+0000","last_peered":"2024-03-28T22:35:30.247934+0000","last_clean":"2024-03-28T22:35:30.247934+0000","last_became_active":"2024-03-28T22:32:07.866819+0000","last_became_peered":"2024-03-28T22:32:07.866819+0000","last_unstale":"2024-03-28T22:35:30.247934+0000","last_undegraded":"2024-03-28T22:35:30.247934+0000","last_fullsized":"2024-03-28T22:35:30.247934+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:28:14.713501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:28:14.713501+0000","last_clean_scrub_stamp":"2024-03-28T22:28:14.713501+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:00:52.438708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275597+0000","last_change":"2024-03-28T22:33:34.020668+0000","last_active":"2024-03-28T22:35:30.275597+0000","last_peered":"2024-03-28T22:35:30.275597+0000","last_clean":"2024-03-28T22:35:30.275597+0000","last_became_active":"2024-03-28T22:33:34.020240+0000","last_became_peered":"2024-03-28T22:33:34.020240+0000","last_unstale":"2024-03-28T22:35:30.275597+0000","last_undegraded":"2024-03-28T22:35:30.275597+0000","last_fullsized":"2024-03-28T22:35:30.275597+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:18.983983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"94'5","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368061+0000","last_change":"2024-03-28T22:33:28.935237+0000","last_active":"2024-03-28T22:35:30.368061+0000","last_peered":"2024-03-28T22:35:30.368061+0000","last_clean":"2024-03-28T22:35:30.368061+0000","last_became_active":"2024-03-28T22:33:28.934875+0000","last_became_peered":"2024-03-28T22:33:28.934875+0000","last_unstale":"2024-03-28T22:35:30.368061+0000","last_undegraded":"2024-03-28T22:35:30.368061+0000","last_fullsized":"2024-03-28T22:35:30.368061+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:20:50.789302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"92'4","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248187+0000","last_change":"2024-03-28T22:33:28.441654+0000","last_active":"2024-03-28T22:35:30.248187+0000","last_peered":"2024-03-28T22:35:30.248187+0000","last_clean":"2024-03-28T22:35:30.248187+0000","last_became_active":"2024-03-28T22:33:28.441293+0000","last_became_peered":"2024-03-28T22:33:28.441293+0000","last_unstale":"2024-03-28T22:35:30.248187+0000","last_undegraded":"2024-03-28T22:35:30.248187+0000","last_fullsized":"2024-03-28T22:35:30.248187+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:31:17.737977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449486+0000","last_change":"2024-03-28T22:33:34.018126+0000","last_active":"2024-03-28T22:35:30.449486+0000","last_peered":"2024-03-28T22:35:30.449486+0000","last_clean":"2024-03-28T22:35:30.449486+0000","last_became_active":"2024-03-28T22:33:34.017573+0000","last_became_peered":"2024-03-28T22:33:34.017573+0000","last_unstale":"2024-03-28T22:35:30.449486+0000","last_undegraded":"2024-03-28T22:35:30.449486+0000","last_fullsized":"2024-03-28T22:35:30.449486+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:23:31.013511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"92'4","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254108+0000","last_change":"2024-03-28T22:33:28.454069+0000","last_active":"2024-03-28T22:35:30.254108+0000","last_peered":"2024-03-28T22:35:30.254108+0000","last_clean":"2024-03-28T22:35:30.254108+0000","last_became_active":"2024-03-28T22:33:28.453754+0000","last_became_peered":"2024-03-28T22:33:28.453754+0000","last_unstale":"2024-03-28T22:35:30.254108+0000","last_undegraded":"2024-03-28T22:35:30.254108+0000","last_fullsized":"2024-03-28T22:35:30.254108+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:23:33.981072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"94'9","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369387+0000","last_change":"2024-03-28T22:33:28.934390+0000","last_active":"2024-03-28T22:35:30.369387+0000","last_peered":"2024-03-28T22:35:30.369387+0000","last_clean":"2024-03-28T22:35:30.369387+0000","last_became_active":"2024-03-28T22:33:28.933919+0000","last_became_peered":"2024-03-28T22:33:28.933919+0000","last_unstale":"2024-03-28T22:35:30.369387+0000","last_undegraded":"2024-03-28T22:35:30.369387+0000","last_fullsized":"2024-03-28T22:35:30.369387+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:27:07.595144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371312+0000","last_change":"2024-03-28T22:33:34.019910+0000","last_active":"2024-03-28T22:35:30.371312+0000","last_peered":"2024-03-28T22:35:30.371312+0000","last_clean":"2024-03-28T22:35:30.371312+0000","last_became_active":"2024-03-28T22:33:34.019522+0000","last_became_peered":"2024-03-28T22:33:34.019522+0000","last_unstale":"2024-03-28T22:35:30.371312+0000","last_undegraded":"2024-03-28T22:35:30.371312+0000","last_fullsized":"2024-03-28T22:35:30.371312+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:01:29.298513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'14","reported_seq":48,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252817+0000","last_change":"2024-03-28T22:33:28.444513+0000","last_active":"2024-03-28T22:35:30.252817+0000","last_peered":"2024-03-28T22:35:30.252817+0000","last_clean":"2024-03-28T22:35:30.252817+0000","last_became_active":"2024-03-28T22:33:28.444091+0000","last_became_peered":"2024-03-28T22:33:28.444091+0000","last_unstale":"2024-03-28T22:35:30.252817+0000","last_undegraded":"2024-03-28T22:35:30.252817+0000","last_fullsized":"2024-03-28T22:35:30.252817+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:49:58.331864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"94'5","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.372223+0000","last_change":"2024-03-28T22:33:28.948390+0000","last_active":"2024-03-28T22:35:30.372223+0000","last_peered":"2024-03-28T22:35:30.372223+0000","last_clean":"2024-03-28T22:35:30.372223+0000","last_became_active":"2024-03-28T22:33:28.942846+0000","last_became_peered":"2024-03-28T22:33:28.942846+0000","last_unstale":"2024-03-28T22:35:30.372223+0000","last_undegraded":"2024-03-28T22:35:30.372223+0000","last_fullsized":"2024-03-28T22:35:30.372223+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:50.596365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251712+0000","last_change":"2024-03-28T22:33:33.999429+0000","last_active":"2024-03-28T22:35:30.251712+0000","last_peered":"2024-03-28T22:35:30.251712+0000","last_clean":"2024-03-28T22:35:30.251712+0000","last_became_active":"2024-03-28T22:33:33.998979+0000","last_became_peered":"2024-03-28T22:33:33.998979+0000","last_unstale":"2024-03-28T22:35:30.251712+0000","last_undegraded":"2024-03-28T22:35:30.251712+0000","last_fullsized":"2024-03-28T22:35:30.251712+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:26.450731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"92'7","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370791+0000","last_change":"2024-03-28T22:33:28.442861+0000","last_active":"2024-03-28T22:35:30.370791+0000","last_peered":"2024-03-28T22:35:30.370791+0000","last_clean":"2024-03-28T22:35:30.370791+0000","last_became_active":"2024-03-28T22:33:28.442468+0000","last_became_peered":"2024-03-28T22:33:28.442468+0000","last_unstale":"2024-03-28T22:35:30.370791+0000","last_undegraded":"2024-03-28T22:35:30.370791+0000","last_fullsized":"2024-03-28T22:35:30.370791+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:45:49.899031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"94'7","reported_seq":45,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451985+0000","last_change":"2024-03-28T22:34:30.429872+0000","last_active":"2024-03-28T22:35:30.451985+0000","last_peered":"2024-03-28T22:35:30.451985+0000","last_clean":"2024-03-28T22:35:30.451985+0000","last_became_active":"2024-03-28T22:34:30.429237+0000","last_became_peered":"2024-03-28T22:34:30.429237+0000","last_unstale":"2024-03-28T22:35:30.451985+0000","last_undegraded":"2024-03-28T22:35:30.451985+0000","last_fullsized":"2024-03-28T22:35:30.451985+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:14:46.608826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370565+0000","last_change":"2024-03-28T22:33:34.013826+0000","last_active":"2024-03-28T22:35:30.370565+0000","last_peered":"2024-03-28T22:35:30.370565+0000","last_clean":"2024-03-28T22:35:30.370565+0000","last_became_active":"2024-03-28T22:33:34.013415+0000","last_became_peered":"2024-03-28T22:33:34.013415+0000","last_unstale":"2024-03-28T22:35:30.370565+0000","last_undegraded":"2024-03-28T22:35:30.370565+0000","last_fullsized":"2024-03-28T22:35:30.370565+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:19:30.965940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"92'2","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448399+0000","last_change":"2024-03-28T22:33:28.442225+0000","last_active":"2024-03-28T22:35:30.448399+0000","last_peered":"2024-03-28T22:35:30.448399+0000","last_clean":"2024-03-28T22:35:30.448399+0000","last_became_active":"2024-03-28T22:33:28.441844+0000","last_became_peered":"2024-03-28T22:33:28.441844+0000","last_unstale":"2024-03-28T22:35:30.448399+0000","last_undegraded":"2024-03-28T22:35:30.448399+0000","last_fullsized":"2024-03-28T22:35:30.448399+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:40:52.108663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"94'2","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.253675+0000","last_change":"2024-03-28T22:33:28.927571+0000","last_active":"2024-03-28T22:35:30.253675+0000","last_peered":"2024-03-28T22:35:30.253675+0000","last_clean":"2024-03-28T22:35:30.253675+0000","last_became_active":"2024-03-28T22:33:28.927275+0000","last_became_peered":"2024-03-28T22:33:28.927275+0000","last_unstale":"2024-03-28T22:35:30.253675+0000","last_undegraded":"2024-03-28T22:35:30.253675+0000","last_fullsized":"2024-03-28T22:35:30.253675+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:25:16.184533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.253011+0000","last_change":"2024-03-28T22:33:34.018429+0000","last_active":"2024-03-28T22:35:30.253011+0000","last_peered":"2024-03-28T22:35:30.253011+0000","last_clean":"2024-03-28T22:35:30.253011+0000","last_became_active":"2024-03-28T22:33:34.017066+0000","last_became_peered":"2024-03-28T22:33:34.017066+0000","last_unstale":"2024-03-28T22:35:30.253011+0000","last_undegraded":"2024-03-28T22:35:30.253011+0000","last_fullsized":"2024-03-28T22:35:30.253011+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:35:42.101072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"92'7","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371970+0000","last_change":"2024-03-28T22:33:28.437760+0000","last_active":"2024-03-28T22:35:30.371970+0000","last_peered":"2024-03-28T22:35:30.371970+0000","last_clean":"2024-03-28T22:35:30.371970+0000","last_became_active":"2024-03-28T22:33:28.437096+0000","last_became_peered":"2024-03-28T22:33:28.437096+0000","last_unstale":"2024-03-28T22:35:30.371970+0000","last_undegraded":"2024-03-28T22:35:30.371970+0000","last_fullsized":"2024-03-28T22:35:30.371970+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:29:20.778551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"94'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449358+0000","last_change":"2024-03-28T22:33:28.935805+0000","last_active":"2024-03-28T22:35:30.449358+0000","last_peered":"2024-03-28T22:35:30.449358+0000","last_clean":"2024-03-28T22:35:30.449358+0000","last_became_active":"2024-03-28T22:33:28.935520+0000","last_became_peered":"2024-03-28T22:33:28.935520+0000","last_unstale":"2024-03-28T22:35:30.449358+0000","last_undegraded":"2024-03-28T22:35:30.449358+0000","last_fullsized":"2024-03-28T22:35:30.449358+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:23.414310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"92'1","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370585+0000","last_change":"2024-03-28T22:33:34.029518+0000","last_active":"2024-03-28T22:35:30.370585+0000","last_peered":"2024-03-28T22:35:30.370585+0000","last_clean":"2024-03-28T22:35:30.370585+0000","last_became_active":"2024-03-28T22:33:34.029101+0000","last_became_peered":"2024-03-28T22:33:34.029101+0000","last_unstale":"2024-03-28T22:35:30.370585+0000","last_undegraded":"2024-03-28T22:35:30.370585+0000","last_fullsized":"2024-03-28T22:35:30.370585+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:05.595543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"92'3","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251929+0000","last_change":"2024-03-28T22:33:28.438857+0000","last_active":"2024-03-28T22:35:30.251929+0000","last_peered":"2024-03-28T22:35:30.251929+0000","last_clean":"2024-03-28T22:35:30.251929+0000","last_became_active":"2024-03-28T22:33:28.438528+0000","last_became_peered":"2024-03-28T22:33:28.438528+0000","last_unstale":"2024-03-28T22:35:30.251929+0000","last_undegraded":"2024-03-28T22:35:30.251929+0000","last_fullsized":"2024-03-28T22:35:30.251929+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:07:54.506739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371660+0000","last_change":"2024-03-28T22:33:28.958271+0000","last_active":"2024-03-28T22:35:30.371660+0000","last_peered":"2024-03-28T22:35:30.371660+0000","last_clean":"2024-03-28T22:35:30.371660+0000","last_became_active":"2024-03-28T22:33:28.957922+0000","last_became_peered":"2024-03-28T22:33:28.957922+0000","last_unstale":"2024-03-28T22:35:30.371660+0000","last_undegraded":"2024-03-28T22:35:30.371660+0000","last_fullsized":"2024-03-28T22:35:30.371660+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:33:19.003131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370592+0000","last_change":"2024-03-28T22:33:34.000789+0000","last_active":"2024-03-28T22:35:30.370592+0000","last_peered":"2024-03-28T22:35:30.370592+0000","last_clean":"2024-03-28T22:35:30.370592+0000","last_became_active":"2024-03-28T22:33:34.000383+0000","last_became_peered":"2024-03-28T22:33:34.000383+0000","last_unstale":"2024-03-28T22:35:30.370592+0000","last_undegraded":"2024-03-28T22:35:30.370592+0000","last_fullsized":"2024-03-28T22:35:30.370592+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:50:29.925656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"92'4","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.253813+0000","last_change":"2024-03-28T22:33:28.439121+0000","last_active":"2024-03-28T22:35:30.253813+0000","last_peered":"2024-03-28T22:35:30.253813+0000","last_clean":"2024-03-28T22:35:30.253813+0000","last_became_active":"2024-03-28T22:33:28.438800+0000","last_became_peered":"2024-03-28T22:33:28.438800+0000","last_unstale":"2024-03-28T22:35:30.253813+0000","last_undegraded":"2024-03-28T22:35:30.253813+0000","last_fullsized":"2024-03-28T22:35:30.253813+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:57:10.013246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"94'5","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248157+0000","last_change":"2024-03-28T22:33:28.934499+0000","last_active":"2024-03-28T22:35:30.248157+0000","last_peered":"2024-03-28T22:35:30.248157+0000","last_clean":"2024-03-28T22:35:30.248157+0000","last_became_active":"2024-03-28T22:33:28.934147+0000","last_became_peered":"2024-03-28T22:33:28.934147+0000","last_unstale":"2024-03-28T22:35:30.248157+0000","last_undegraded":"2024-03-28T22:35:30.248157+0000","last_fullsized":"2024-03-28T22:35:30.248157+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:35:39.090052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448402+0000","last_change":"2024-03-28T22:33:34.017946+0000","last_active":"2024-03-28T22:35:30.448402+0000","last_peered":"2024-03-28T22:35:30.448402+0000","last_clean":"2024-03-28T22:35:30.448402+0000","last_became_active":"2024-03-28T22:33:34.017546+0000","last_became_peered":"2024-03-28T22:33:34.017546+0000","last_unstale":"2024-03-28T22:35:30.448402+0000","last_undegraded":"2024-03-28T22:35:30.448402+0000","last_fullsized":"2024-03-28T22:35:30.448402+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:14:39.081677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"92'5","reported_seq":144,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:47.290463+0000","last_change":"2024-03-28T22:33:28.453387+0000","last_active":"2024-03-28T22:35:47.290463+0000","last_peered":"2024-03-28T22:35:47.290463+0000","last_clean":"2024-03-28T22:35:47.290463+0000","last_became_active":"2024-03-28T22:33:28.453029+0000","last_became_peered":"2024-03-28T22:33:28.453029+0000","last_unstale":"2024-03-28T22:35:47.290463+0000","last_undegraded":"2024-03-28T22:35:47.290463+0000","last_fullsized":"2024-03-28T22:35:47.290463+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:12:33.246863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":105,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248615+0000","last_change":"2024-03-28T22:33:28.923819+0000","last_active":"2024-03-28T22:35:30.248615+0000","last_peered":"2024-03-28T22:35:30.248615+0000","last_clean":"2024-03-28T22:35:30.248615+0000","last_became_active":"2024-03-28T22:33:28.923471+0000","last_became_peered":"2024-03-28T22:33:28.923471+0000","last_unstale":"2024-03-28T22:35:30.248615+0000","last_undegraded":"2024-03-28T22:35:30.248615+0000","last_fullsized":"2024-03-28T22:35:30.248615+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:58:03.496406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451029+0000","last_change":"2024-03-28T22:33:34.029789+0000","last_active":"2024-03-28T22:35:30.451029+0000","last_peered":"2024-03-28T22:35:30.451029+0000","last_clean":"2024-03-28T22:35:30.451029+0000","last_became_active":"2024-03-28T22:33:34.029231+0000","last_became_peered":"2024-03-28T22:33:34.029231+0000","last_unstale":"2024-03-28T22:35:30.451029+0000","last_undegraded":"2024-03-28T22:35:30.451029+0000","last_fullsized":"2024-03-28T22:35:30.451029+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:47:50.101393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"92'3","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275541+0000","last_change":"2024-03-28T22:33:28.435668+0000","last_active":"2024-03-28T22:35:30.275541+0000","last_peered":"2024-03-28T22:35:30.275541+0000","last_clean":"2024-03-28T22:35:30.275541+0000","last_became_active":"2024-03-28T22:33:28.435325+0000","last_became_peered":"2024-03-28T22:33:28.435325+0000","last_unstale":"2024-03-28T22:35:30.275541+0000","last_undegraded":"2024-03-28T22:35:30.275541+0000","last_fullsized":"2024-03-28T22:35:30.275541+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:13:06.281074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"94'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248244+0000","last_change":"2024-03-28T22:33:28.954138+0000","last_active":"2024-03-28T22:35:30.248244+0000","last_peered":"2024-03-28T22:35:30.248244+0000","last_clean":"2024-03-28T22:35:30.248244+0000","last_became_active":"2024-03-28T22:33:28.953752+0000","last_became_peered":"2024-03-28T22:33:28.953752+0000","last_unstale":"2024-03-28T22:35:30.248244+0000","last_undegraded":"2024-03-28T22:35:30.248244+0000","last_fullsized":"2024-03-28T22:35:30.248244+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:53:34.892316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.274983+0000","last_change":"2024-03-28T22:33:34.002672+0000","last_active":"2024-03-28T22:35:30.274983+0000","last_peered":"2024-03-28T22:35:30.274983+0000","last_clean":"2024-03-28T22:35:30.274983+0000","last_became_active":"2024-03-28T22:33:34.002195+0000","last_became_peered":"2024-03-28T22:33:34.002195+0000","last_unstale":"2024-03-28T22:35:30.274983+0000","last_undegraded":"2024-03-28T22:35:30.274983+0000","last_fullsized":"2024-03-28T22:35:30.274983+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:24:11.521474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"92'5","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369212+0000","last_change":"2024-03-28T22:33:28.450818+0000","last_active":"2024-03-28T22:35:30.369212+0000","last_peered":"2024-03-28T22:35:30.369212+0000","last_clean":"2024-03-28T22:35:30.369212+0000","last_became_active":"2024-03-28T22:33:28.448592+0000","last_became_peered":"2024-03-28T22:33:28.448592+0000","last_unstale":"2024-03-28T22:35:30.369212+0000","last_undegraded":"2024-03-28T22:35:30.369212+0000","last_fullsized":"2024-03-28T22:35:30.369212+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:19:27.126306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"94'9","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248488+0000","last_change":"2024-03-28T22:33:28.925191+0000","last_active":"2024-03-28T22:35:30.248488+0000","last_peered":"2024-03-28T22:35:30.248488+0000","last_clean":"2024-03-28T22:35:30.248488+0000","last_became_active":"2024-03-28T22:33:28.924424+0000","last_became_peered":"2024-03-28T22:33:28.924424+0000","last_unstale":"2024-03-28T22:35:30.248488+0000","last_undegraded":"2024-03-28T22:35:30.248488+0000","last_fullsized":"2024-03-28T22:35:30.248488+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:50:00.415047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"94'9","reported_seq":46,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449424+0000","last_change":"2024-03-28T22:34:29.987176+0000","last_active":"2024-03-28T22:35:30.449424+0000","last_peered":"2024-03-28T22:35:30.449424+0000","last_clean":"2024-03-28T22:35:30.449424+0000","last_became_active":"2024-03-28T22:34:29.986807+0000","last_became_peered":"2024-03-28T22:34:29.986807+0000","last_unstale":"2024-03-28T22:35:30.449424+0000","last_undegraded":"2024-03-28T22:35:30.449424+0000","last_fullsized":"2024-03-28T22:35:30.449424+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:00:06.774521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"94'12","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251938+0000","last_change":"2024-03-28T22:33:28.927068+0000","last_active":"2024-03-28T22:35:30.251938+0000","last_peered":"2024-03-28T22:35:30.251938+0000","last_clean":"2024-03-28T22:35:30.251938+0000","last_became_active":"2024-03-28T22:33:28.926734+0000","last_became_peered":"2024-03-28T22:33:28.926734+0000","last_unstale":"2024-03-28T22:35:30.251938+0000","last_undegraded":"2024-03-28T22:35:30.251938+0000","last_fullsized":"2024-03-28T22:35:30.251938+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:56.309555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"94'8","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251930+0000","last_change":"2024-03-28T22:33:28.955675+0000","last_active":"2024-03-28T22:35:30.251930+0000","last_peered":"2024-03-28T22:35:30.251930+0000","last_clean":"2024-03-28T22:35:30.251930+0000","last_became_active":"2024-03-28T22:33:28.955329+0000","last_became_peered":"2024-03-28T22:33:28.955329+0000","last_unstale":"2024-03-28T22:35:30.251930+0000","last_undegraded":"2024-03-28T22:35:30.251930+0000","last_fullsized":"2024-03-28T22:35:30.251930+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:34:14.166434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.452477+0000","last_change":"2024-03-28T22:33:28.955125+0000","last_active":"2024-03-28T22:35:30.452477+0000","last_peered":"2024-03-28T22:35:30.452477+0000","last_clean":"2024-03-28T22:35:30.452477+0000","last_became_active":"2024-03-28T22:33:28.954434+0000","last_became_peered":"2024-03-28T22:33:28.954434+0000","last_unstale":"2024-03-28T22:35:30.452477+0000","last_undegraded":"2024-03-28T22:35:30.452477+0000","last_fullsized":"2024-03-28T22:35:30.452477+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:56:50.705012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"94'4","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369433+0000","last_change":"2024-03-28T22:33:28.945230+0000","last_active":"2024-03-28T22:35:30.369433+0000","last_peered":"2024-03-28T22:35:30.369433+0000","last_clean":"2024-03-28T22:35:30.369433+0000","last_became_active":"2024-03-28T22:33:28.944921+0000","last_became_peered":"2024-03-28T22:33:28.944921+0000","last_unstale":"2024-03-28T22:35:30.369433+0000","last_undegraded":"2024-03-28T22:35:30.369433+0000","last_fullsized":"2024-03-28T22:35:30.369433+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:26:26.505774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"94'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449741+0000","last_change":"2024-03-28T22:33:28.928720+0000","last_active":"2024-03-28T22:35:30.449741+0000","last_peered":"2024-03-28T22:35:30.449741+0000","last_clean":"2024-03-28T22:35:30.449741+0000","last_became_active":"2024-03-28T22:33:28.928415+0000","last_became_peered":"2024-03-28T22:33:28.928415+0000","last_unstale":"2024-03-28T22:35:30.449741+0000","last_undegraded":"2024-03-28T22:35:30.449741+0000","last_fullsized":"2024-03-28T22:35:30.449741+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:00.839609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"94'15","reported_seq":47,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252418+0000","last_change":"2024-03-28T22:33:28.924948+0000","last_active":"2024-03-28T22:35:30.252418+0000","last_peered":"2024-03-28T22:35:30.252418+0000","last_clean":"2024-03-28T22:35:30.252418+0000","last_became_active":"2024-03-28T22:33:28.924631+0000","last_became_peered":"2024-03-28T22:33:28.924631+0000","last_unstale":"2024-03-28T22:35:30.252418+0000","last_undegraded":"2024-03-28T22:35:30.252418+0000","last_fullsized":"2024-03-28T22:35:30.252418+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:53:44.909991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"94'2","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.372408+0000","last_change":"2024-03-28T22:33:28.948725+0000","last_active":"2024-03-28T22:35:30.372408+0000","last_peered":"2024-03-28T22:35:30.372408+0000","last_clean":"2024-03-28T22:35:30.372408+0000","last_became_active":"2024-03-28T22:33:28.943942+0000","last_became_peered":"2024-03-28T22:33:28.943942+0000","last_unstale":"2024-03-28T22:35:30.372408+0000","last_undegraded":"2024-03-28T22:35:30.372408+0000","last_fullsized":"2024-03-28T22:35:30.372408+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:50:04.238538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"94'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275158+0000","last_change":"2024-03-28T22:33:28.941546+0000","last_active":"2024-03-28T22:35:30.275158+0000","last_peered":"2024-03-28T22:35:30.275158+0000","last_clean":"2024-03-28T22:35:30.275158+0000","last_became_active":"2024-03-28T22:33:28.941278+0000","last_became_peered":"2024-03-28T22:33:28.941278+0000","last_unstale":"2024-03-28T22:35:30.275158+0000","last_undegraded":"2024-03-28T22:35:30.275158+0000","last_fullsized":"2024-03-28T22:35:30.275158+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:47:10.047067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.372264+0000","last_change":"2024-03-28T22:33:28.949552+0000","last_active":"2024-03-28T22:35:30.372264+0000","last_peered":"2024-03-28T22:35:30.372264+0000","last_clean":"2024-03-28T22:35:30.372264+0000","last_became_active":"2024-03-28T22:33:28.947432+0000","last_became_peered":"2024-03-28T22:33:28.947432+0000","last_unstale":"2024-03-28T22:35:30.372264+0000","last_undegraded":"2024-03-28T22:35:30.372264+0000","last_fullsized":"2024-03-28T22:35:30.372264+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:01:16.534642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"94'10","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448510+0000","last_change":"2024-03-28T22:33:28.953930+0000","last_active":"2024-03-28T22:35:30.448510+0000","last_peered":"2024-03-28T22:35:30.448510+0000","last_clean":"2024-03-28T22:35:30.448510+0000","last_became_active":"2024-03-28T22:33:28.953591+0000","last_became_peered":"2024-03-28T22:33:28.953591+0000","last_unstale":"2024-03-28T22:35:30.448510+0000","last_undegraded":"2024-03-28T22:35:30.448510+0000","last_fullsized":"2024-03-28T22:35:30.448510+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:42:55.857476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"94'11","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368801+0000","last_change":"2024-03-28T22:33:28.946307+0000","last_active":"2024-03-28T22:35:30.368801+0000","last_peered":"2024-03-28T22:35:30.368801+0000","last_clean":"2024-03-28T22:35:30.368801+0000","last_became_active":"2024-03-28T22:33:28.945932+0000","last_became_peered":"2024-03-28T22:33:28.945932+0000","last_unstale":"2024-03-28T22:35:30.368801+0000","last_undegraded":"2024-03-28T22:35:30.368801+0000","last_fullsized":"2024-03-28T22:35:30.368801+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:54:59.651668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449676+0000","last_change":"2024-03-28T22:33:28.928014+0000","last_active":"2024-03-28T22:35:30.449676+0000","last_peered":"2024-03-28T22:35:30.449676+0000","last_clean":"2024-03-28T22:35:30.449676+0000","last_became_active":"2024-03-28T22:33:28.927686+0000","last_became_peered":"2024-03-28T22:33:28.927686+0000","last_unstale":"2024-03-28T22:35:30.449676+0000","last_undegraded":"2024-03-28T22:35:30.449676+0000","last_fullsized":"2024-03-28T22:35:30.449676+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:47:08.131565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71409,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1079,"num_read_kb":617,"num_write":250,"num_write_kb":146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":214227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":217,"ondisk_log_size":217,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318315,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32260,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712892,"statfs":{"total":95995035648,"available":95962001408,"internally_reserved":0,"allocated":1105920,"data_stored":539556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579951,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28780,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716372,"statfs":{"total":95995035648,"available":95965564928,"internally_reserved":0,"allocated":1798144,"data_stored":1191337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743478,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1130496,"data_stored":543560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005115,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1118208,"data_stored":539820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299457,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37024,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708128,"statfs":{"total":95995035648,"available":95957123072,"internally_reserved":0,"allocated":1720320,"data_stored":1136480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495686,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1101824,"data_stored":534968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574946,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36488,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708664,"statfs":{"total":95995035648,"available":95957671936,"internally_reserved":0,"allocated":1171456,"data_stored":600453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738471,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1126400,"data_stored":536424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411423,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36508,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708644,"statfs":{"total":95995035648,"available":95957651456,"internally_reserved":0,"allocated":1191936,"data_stored":600673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182487,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36432,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708720,"statfs":{"total":95995035648,"available":95957729280,"internally_reserved":0,"allocated":1114112,"data_stored":534676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920850,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1110016,"data_stored":539574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757325,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28712,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716440,"statfs":{"total":95995035648,"available":95965634560,"internally_reserved":0,"allocated":1728512,"data_stored":1131530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":66207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":66427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":66723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T22:35:54.583 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-28T22:35:54.583 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-28T22:35:54.583 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-28T22:35:54.583 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph health --format=json 2024-03-28T22:35:54.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:54 smithi172 ceph-mon[36566]: from='client.25067 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:35:54.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:54 smithi082 ceph-mon[36471]: from='client.25067 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:35:54.710 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:54 smithi043 ceph-mon[31823]: from='client.25067 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:35:55.096 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:35:55.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:55 smithi043 ceph-mon[31823]: pgmap v399: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:35:55.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:55 smithi172 ceph-mon[36566]: pgmap v399: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:35:55.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:55 smithi082 ceph-mon[36471]: pgmap v399: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:35:57.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.318+0000 7f5c38600700 1 -- 172.21.15.43:0/2829801690 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c301088b0 msgr2=0x7f5c30108d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:57.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.318+0000 7f5c38600700 1 --2- 172.21.15.43:0/2829801690 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c301088b0 0x7f5c30108d30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c009990 tx=0x7f5c1c009ca0 comp rx=0 tx=0).stop 2024-03-28T22:35:57.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.318+0000 7f5c38600700 1 -- 172.21.15.43:0/2829801690 shutdown_connections 2024-03-28T22:35:57.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.318+0000 7f5c38600700 1 --2- 172.21.15.43:0/2829801690 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5c30109270 0x7f5c30115780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:57.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.318+0000 7f5c38600700 1 --2- 172.21.15.43:0/2829801690 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c3010e8b0 0x7f5c3010ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:57.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.318+0000 7f5c38600700 1 --2- 172.21.15.43:0/2829801690 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c301088b0 0x7f5c30108d30 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:57.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.318+0000 7f5c38600700 1 -- 172.21.15.43:0/2829801690 >> 172.21.15.43:0/2829801690 conn(0x7f5c30076890 msgr2=0x7f5c30078cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:57.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.318+0000 7f5c38600700 1 -- 172.21.15.43:0/2829801690 shutdown_connections 2024-03-28T22:35:57.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.318+0000 7f5c38600700 1 -- 172.21.15.43:0/2829801690 wait complete. 2024-03-28T22:35:57.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.320+0000 7f5c38600700 1 Processor -- start 2024-03-28T22:35:57.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.320+0000 7f5c38600700 1 -- start start 2024-03-28T22:35:57.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.320+0000 7f5c38600700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c301088b0 0x7f5c3019f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:57.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.320+0000 7f5c38600700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5c30109270 0x7f5c3019f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:57.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.320+0000 7f5c38600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c3010e8b0 0x7f5c301a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:57.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.320+0000 7f5c38600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5c30118c00 con 0x7f5c3010e8b0 2024-03-28T22:35:57.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.320+0000 7f5c38600700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5c30118a80 con 0x7f5c30109270 2024-03-28T22:35:57.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.320+0000 7f5c38600700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f5c30118d80 con 0x7f5c301088b0 2024-03-28T22:35:57.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.321+0000 7f5c35c00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c301088b0 0x7f5c3019f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:57.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.321+0000 7f5c35c00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c301088b0 0x7f5c3019f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35820/0 (socket says 172.21.15.43:35820) 2024-03-28T22:35:57.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.321+0000 7f5c35c00700 1 -- 172.21.15.43:0/3502147221 learned_addr learned my addr 172.21.15.43:0/3502147221 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:35:57.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.321+0000 7f5c35200700 1 --2- 172.21.15.43:0/3502147221 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5c30109270 0x7f5c3019f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:57.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.321+0000 7f5c36600700 1 --2- 172.21.15.43:0/3502147221 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c3010e8b0 0x7f5c301a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:57.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.321+0000 7f5c35c00700 1 -- 172.21.15.43:0/3502147221 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5c30109270 msgr2=0x7f5c3019f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:57.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.321+0000 7f5c35c00700 1 --2- 172.21.15.43:0/3502147221 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5c30109270 0x7f5c3019f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:57.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.321+0000 7f5c35c00700 1 -- 172.21.15.43:0/3502147221 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c3010e8b0 msgr2=0x7f5c301a3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:57.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.321+0000 7f5c35c00700 1 --2- 172.21.15.43:0/3502147221 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c3010e8b0 0x7f5c301a3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:57.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.321+0000 7f5c35c00700 1 -- 172.21.15.43:0/3502147221 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c301a4050 con 0x7f5c301088b0 2024-03-28T22:35:57.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.321+0000 7f5c35c00700 1 --2- 172.21.15.43:0/3502147221 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c301088b0 0x7f5c3019f090 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5c2400e9a0 tx=0x7f5c2400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:57.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.322+0000 7f5c2e000700 1 -- 172.21.15.43:0/3502147221 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c2400cbd0 con 0x7f5c301088b0 2024-03-28T22:35:57.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.322+0000 7f5c38600700 1 -- 172.21.15.43:0/3502147221 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c301a4340 con 0x7f5c301088b0 2024-03-28T22:35:57.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.322+0000 7f5c2e000700 1 -- 172.21.15.43:0/3502147221 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c2400cd70 con 0x7f5c301088b0 2024-03-28T22:35:57.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.322+0000 7f5c2e000700 1 -- 172.21.15.43:0/3502147221 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c24018750 con 0x7f5c301088b0 2024-03-28T22:35:57.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.322+0000 7f5c38600700 1 -- 172.21.15.43:0/3502147221 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c301abbc0 con 0x7f5c301088b0 2024-03-28T22:35:57.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.323+0000 7f5c38600700 1 -- 172.21.15.43:0/3502147221 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c30068f00 con 0x7f5c301088b0 2024-03-28T22:35:57.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.325+0000 7f5c2e000700 1 -- 172.21.15.43:0/3502147221 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f5c24010430 con 0x7f5c301088b0 2024-03-28T22:35:57.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.326+0000 7f5c2e000700 1 --2- 172.21.15.43:0/3502147221 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5c180a1e10 0x7f5c180a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:57.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.326+0000 7f5c2e000700 1 -- 172.21.15.43:0/3502147221 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f5c240ce3e0 con 0x7f5c301088b0 2024-03-28T22:35:57.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.327+0000 7f5c35200700 1 --2- 172.21.15.43:0/3502147221 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5c180a1e10 0x7f5c180a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:57.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.327+0000 7f5c35200700 1 --2- 172.21.15.43:0/3502147221 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5c180a1e10 0x7f5c180a42d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c00b2e0 tx=0x7f5c1c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:57.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.335+0000 7f5c2e000700 1 -- 172.21.15.43:0/3502147221 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5c2409b420 con 0x7f5c301088b0 2024-03-28T22:35:57.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:57 smithi043 ceph-mon[31823]: pgmap v400: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T22:35:57.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.619+0000 7f5c38600700 1 -- 172.21.15.43:0/3502147221 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f5c3002db30 con 0x7f5c301088b0 2024-03-28T22:35:57.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.619+0000 7f5c2e000700 1 -- 172.21.15.43:0/3502147221 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f5c2409ad20 con 0x7f5c301088b0 2024-03-28T22:35:57.618 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:35:57.618 INFO:teuthology.orchestra.run.smithi043.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-28T22:35:57.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.622+0000 7f5c38600700 1 -- 172.21.15.43:0/3502147221 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5c180a1e10 msgr2=0x7f5c180a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:57.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.622+0000 7f5c38600700 1 --2- 172.21.15.43:0/3502147221 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5c180a1e10 0x7f5c180a42d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c00b2e0 tx=0x7f5c1c005960 comp rx=0 tx=0).stop 2024-03-28T22:35:57.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.623+0000 7f5c38600700 1 -- 172.21.15.43:0/3502147221 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c301088b0 msgr2=0x7f5c3019f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:57.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.623+0000 7f5c38600700 1 --2- 172.21.15.43:0/3502147221 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c301088b0 0x7f5c3019f090 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5c2400e9a0 tx=0x7f5c2400ecb0 comp rx=0 tx=0).stop 2024-03-28T22:35:57.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.623+0000 7f5c38600700 1 -- 172.21.15.43:0/3502147221 shutdown_connections 2024-03-28T22:35:57.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.623+0000 7f5c38600700 1 --2- 172.21.15.43:0/3502147221 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5c180a1e10 0x7f5c180a42d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:57.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.623+0000 7f5c38600700 1 --2- 172.21.15.43:0/3502147221 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c3010e8b0 0x7f5c301a3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:57.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.623+0000 7f5c38600700 1 --2- 172.21.15.43:0/3502147221 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c301088b0 0x7f5c3019f090 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:57.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.623+0000 7f5c38600700 1 --2- 172.21.15.43:0/3502147221 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5c30109270 0x7f5c3019f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:57.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.623+0000 7f5c38600700 1 -- 172.21.15.43:0/3502147221 >> 172.21.15.43:0/3502147221 conn(0x7f5c30076890 msgr2=0x7f5c30076f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:57.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.623+0000 7f5c38600700 1 -- 172.21.15.43:0/3502147221 shutdown_connections 2024-03-28T22:35:57.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:57.623+0000 7f5c38600700 1 -- 172.21.15.43:0/3502147221 wait complete. 2024-03-28T22:35:57.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:57 smithi172 ceph-mon[36566]: pgmap v400: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T22:35:57.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:57 smithi082 ceph-mon[36471]: pgmap v400: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T22:35:58.535 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-28T22:35:58.536 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:35:58.629 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:58 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3502147221' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T22:35:58.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:58 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3502147221' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T22:35:58.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:58 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3502147221' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T22:35:58.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.775+0000 7fd29b200640 1 -- 172.21.15.43:0/757393211 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd2941037a0 msgr2=0x7fd294103bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:58.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.775+0000 7fd29b200640 1 --2- 172.21.15.43:0/757393211 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd2941037a0 0x7fd294103bc0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd288009900 tx=0x7fd28802bc30 comp rx=0 tx=0).stop 2024-03-28T22:35:58.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.775+0000 7fd29b200640 1 -- 172.21.15.43:0/757393211 shutdown_connections 2024-03-28T22:35:58.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.775+0000 7fd29b200640 1 --2- 172.21.15.43:0/757393211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd294105350 0x7fd29410bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:58.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.775+0000 7fd29b200640 1 --2- 172.21.15.43:0/757393211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd294104990 0x7fd294104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:58.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.775+0000 7fd29b200640 1 --2- 172.21.15.43:0/757393211 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd2941037a0 0x7fd294103bc0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:58.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.775+0000 7fd29b200640 1 -- 172.21.15.43:0/757393211 >> 172.21.15.43:0/757393211 conn(0x7fd2940fed30 msgr2=0x7fd294101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:58.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.776+0000 7fd29b200640 1 -- 172.21.15.43:0/757393211 shutdown_connections 2024-03-28T22:35:58.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.776+0000 7fd29b200640 1 -- 172.21.15.43:0/757393211 wait complete. 2024-03-28T22:35:58.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.777+0000 7fd29b200640 1 Processor -- start 2024-03-28T22:35:58.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.777+0000 7fd29b200640 1 -- start start 2024-03-28T22:35:58.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.777+0000 7fd29b200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2941037a0 0x7fd294199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:58.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.777+0000 7fd29b200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd294104990 0x7fd294199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:58.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.777+0000 7fd29b200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd294105350 0x7fd2941a1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:58.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.777+0000 7fd29b200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd29410f470 con 0x7fd294104990 2024-03-28T22:35:58.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.777+0000 7fd29b200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd29410f2f0 con 0x7fd294105350 2024-03-28T22:35:58.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.777+0000 7fd29b200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd29410f5f0 con 0x7fd2941037a0 2024-03-28T22:35:58.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.778+0000 7fd299400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd294104990 0x7fd294199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:58.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.778+0000 7fd29a800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd294105350 0x7fd2941a1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:58.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.778+0000 7fd299400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd294104990 0x7fd294199fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59764/0 (socket says 172.21.15.43:59764) 2024-03-28T22:35:58.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.778+0000 7fd299400640 1 -- 172.21.15.43:0/3061319306 learned_addr learned my addr 172.21.15.43:0/3061319306 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:35:58.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.778+0000 7fd29a800640 1 -- 172.21.15.43:0/3061319306 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2941037a0 msgr2=0x7fd294199a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:58.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.778+0000 7fd29a800640 1 --2- 172.21.15.43:0/3061319306 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2941037a0 0x7fd294199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:58.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.778+0000 7fd29a800640 1 -- 172.21.15.43:0/3061319306 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd294104990 msgr2=0x7fd294199fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:58.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.778+0000 7fd29a800640 1 --2- 172.21.15.43:0/3061319306 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd294104990 0x7fd294199fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:58.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.778+0000 7fd29a800640 1 -- 172.21.15.43:0/3061319306 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2941a16b0 con 0x7fd294105350 2024-03-28T22:35:58.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.778+0000 7fd299400640 1 --2- 172.21.15.43:0/3061319306 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd294104990 0x7fd294199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:35:58.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.778+0000 7fd29a800640 1 --2- 172.21.15.43:0/3061319306 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd294105350 0x7fd2941a1040 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd290009fd0 tx=0x7fd29000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:58.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.779+0000 7fd28ea00640 1 -- 172.21.15.43:0/3061319306 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd29000ce20 con 0x7fd294105350 2024-03-28T22:35:58.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.779+0000 7fd28ea00640 1 -- 172.21.15.43:0/3061319306 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd290005020 con 0x7fd294105350 2024-03-28T22:35:58.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.779+0000 7fd28ea00640 1 -- 172.21.15.43:0/3061319306 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd290010720 con 0x7fd294105350 2024-03-28T22:35:58.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.779+0000 7fd29b200640 1 -- 172.21.15.43:0/3061319306 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2941a19a0 con 0x7fd294105350 2024-03-28T22:35:58.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.779+0000 7fd29b200640 1 -- 172.21.15.43:0/3061319306 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2941a1ef0 con 0x7fd294105350 2024-03-28T22:35:58.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.780+0000 7fd29b200640 1 -- 172.21.15.43:0/3061319306 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd29410aa60 con 0x7fd294105350 2024-03-28T22:35:58.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.782+0000 7fd28ea00640 1 -- 172.21.15.43:0/3061319306 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fd290005540 con 0x7fd294105350 2024-03-28T22:35:58.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.783+0000 7fd28ea00640 1 --2- 172.21.15.43:0/3061319306 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd27c0a1d80 0x7fd27c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:58.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.784+0000 7fd28ea00640 1 -- 172.21.15.43:0/3061319306 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7fd2900ce860 con 0x7fd294105350 2024-03-28T22:35:58.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.784+0000 7fd299e00640 1 --2- 172.21.15.43:0/3061319306 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd27c0a1d80 0x7fd27c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:58.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.785+0000 7fd299e00640 1 --2- 172.21.15.43:0/3061319306 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd27c0a1d80 0x7fd27c0a4240 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd288036040 tx=0x7fd288000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:58.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:58.789+0000 7fd28ea00640 1 -- 172.21.15.43:0/3061319306 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd29009b8a0 con 0x7fd294105350 2024-03-28T22:35:58.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.000+0000 7fd29b200640 1 -- 172.21.15.43:0/3061319306 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd294062350 con 0x7fd294105350 2024-03-28T22:35:59.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.003+0000 7fd28ea00640 1 -- 172.21.15.43:0/3061319306 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+24712 (secure 0 0 0) 0x7fd29009b1a0 con 0x7fd294105350 2024-03-28T22:35:59.002 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:35:59.002 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":96,"fsid":"c2276428-ed51-11ee-95d0-87774f69a715","created":"2024-03-28T22:24:19.950084+0000","modified":"2024-03-28T22:35:30.237654+0000","last_up_change":"2024-03-28T22:32:31.078737+0000","last_in_change":"2024-03-28T22:32:10.147097+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:28:14.713290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:33:27.167435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:33:28.411825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T22:33:32.523122+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b855adcd-786c-4e42-b8fe-a515a77ce979","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6803","nonce":2826124409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6805","nonce":2826124409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6809","nonce":2826124409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6807","nonce":2826124409}]},"public_addr":"172.21.15.43:6803/2826124409","cluster_addr":"172.21.15.43:6805/2826124409","heartbeat_back_addr":"172.21.15.43:6809/2826124409","heartbeat_front_addr":"172.21.15.43:6807/2826124409","state":["exists","up"]},{"osd":1,"uuid":"031c143d-8ad1-4b68-8db9-3c717137e3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6801","nonce":3043263644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6803","nonce":3043263644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6807","nonce":3043263644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6805","nonce":3043263644}]},"public_addr":"172.21.15.82:6801/3043263644","cluster_addr":"172.21.15.82:6803/3043263644","heartbeat_back_addr":"172.21.15.82:6807/3043263644","heartbeat_front_addr":"172.21.15.82:6805/3043263644","state":["exists","up"]},{"osd":2,"uuid":"20632bbb-f07a-4ca1-b0c4-ff9d508e211c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6801","nonce":4213953645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6803","nonce":4213953645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6807","nonce":4213953645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6805","nonce":4213953645}]},"public_addr":"172.21.15.172:6801/4213953645","cluster_addr":"172.21.15.172:6803/4213953645","heartbeat_back_addr":"172.21.15.172:6807/4213953645","heartbeat_front_addr":"172.21.15.172:6805/4213953645","state":["exists","up"]},{"osd":3,"uuid":"9a1f2312-4dbd-475b-8736-d5ebc8788e09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6811","nonce":1597280920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6813","nonce":1597280920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6817","nonce":1597280920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6815","nonce":1597280920}]},"public_addr":"172.21.15.43:6811/1597280920","cluster_addr":"172.21.15.43:6813/1597280920","heartbeat_back_addr":"172.21.15.43:6817/1597280920","heartbeat_front_addr":"172.21.15.43:6815/1597280920","state":["exists","up"]},{"osd":4,"uuid":"29140ead-7075-42b2-90c1-27cae0b29ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6809","nonce":215451895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6811","nonce":215451895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6815","nonce":215451895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6813","nonce":215451895}]},"public_addr":"172.21.15.82:6809/215451895","cluster_addr":"172.21.15.82:6811/215451895","heartbeat_back_addr":"172.21.15.82:6815/215451895","heartbeat_front_addr":"172.21.15.82:6813/215451895","state":["exists","up"]},{"osd":5,"uuid":"59644c61-93dc-4536-b2ad-0256b24e5cb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6809","nonce":1576148933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6811","nonce":1576148933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6815","nonce":1576148933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6813","nonce":1576148933}]},"public_addr":"172.21.15.172:6809/1576148933","cluster_addr":"172.21.15.172:6811/1576148933","heartbeat_back_addr":"172.21.15.172:6815/1576148933","heartbeat_front_addr":"172.21.15.172:6813/1576148933","state":["exists","up"]},{"osd":6,"uuid":"22d54e57-c3f1-4ecf-9921-8e815b48f92a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6819","nonce":791044279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6821","nonce":791044279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6825","nonce":791044279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6823","nonce":791044279}]},"public_addr":"172.21.15.43:6819/791044279","cluster_addr":"172.21.15.43:6821/791044279","heartbeat_back_addr":"172.21.15.43:6825/791044279","heartbeat_front_addr":"172.21.15.43:6823/791044279","state":["exists","up"]},{"osd":7,"uuid":"248cd1e2-0d6b-452f-a0a8-c51fe1b975ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6817","nonce":2157777323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6819","nonce":2157777323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6823","nonce":2157777323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6821","nonce":2157777323}]},"public_addr":"172.21.15.82:6817/2157777323","cluster_addr":"172.21.15.82:6819/2157777323","heartbeat_back_addr":"172.21.15.82:6823/2157777323","heartbeat_front_addr":"172.21.15.82:6821/2157777323","state":["exists","up"]},{"osd":8,"uuid":"d98aab6e-8bae-4c24-906c-4f492f047b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6817","nonce":1095328248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6819","nonce":1095328248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6823","nonce":1095328248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6821","nonce":1095328248}]},"public_addr":"172.21.15.172:6817/1095328248","cluster_addr":"172.21.15.172:6819/1095328248","heartbeat_back_addr":"172.21.15.172:6823/1095328248","heartbeat_front_addr":"172.21.15.172:6821/1095328248","state":["exists","up"]},{"osd":9,"uuid":"19aa383e-a0f6-45ea-96cc-e7956535caa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6827","nonce":3657791498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6829","nonce":3657791498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6833","nonce":3657791498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6831","nonce":3657791498}]},"public_addr":"172.21.15.43:6827/3657791498","cluster_addr":"172.21.15.43:6829/3657791498","heartbeat_back_addr":"172.21.15.43:6833/3657791498","heartbeat_front_addr":"172.21.15.43:6831/3657791498","state":["exists","up"]},{"osd":10,"uuid":"666977e5-eaf7-41b9-a18f-c380e149345d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6825","nonce":3454954047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6827","nonce":3454954047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6831","nonce":3454954047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6829","nonce":3454954047}]},"public_addr":"172.21.15.82:6825/3454954047","cluster_addr":"172.21.15.82:6827/3454954047","heartbeat_back_addr":"172.21.15.82:6831/3454954047","heartbeat_front_addr":"172.21.15.82:6829/3454954047","state":["exists","up"]},{"osd":11,"uuid":"615cf3be-8076-470e-ad09-c880e7593717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6825","nonce":2563517144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6827","nonce":2563517144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6831","nonce":2563517144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6829","nonce":2563517144}]},"public_addr":"172.21.15.172:6825/2563517144","cluster_addr":"172.21.15.172:6827/2563517144","heartbeat_back_addr":"172.21.15.172:6831/2563517144","heartbeat_front_addr":"172.21.15.172:6829/2563517144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:21.727405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:47.100259+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:10.615438+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:43.632144+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:09.487940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:34.604463+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:09.594452+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:35.709795+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:01.184202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:35.455322+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:02.413166+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:28.550754+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.9","mappings":[{"from":5,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/578791143":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/4029313215":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/1574178482":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6801/3517712438":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6800/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/1375099638":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/2710117652":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6801/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/3215634168":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6800/3517712438":"2024-03-29T22:25:28.008215+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:35:59.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.006+0000 7fd29b200640 1 -- 172.21.15.43:0/3061319306 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd27c0a1d80 msgr2=0x7fd27c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:59.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.006+0000 7fd29b200640 1 --2- 172.21.15.43:0/3061319306 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd27c0a1d80 0x7fd27c0a4240 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd288036040 tx=0x7fd288000bc0 comp rx=0 tx=0).stop 2024-03-28T22:35:59.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.006+0000 7fd29b200640 1 -- 172.21.15.43:0/3061319306 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd294105350 msgr2=0x7fd2941a1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:59.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.006+0000 7fd29b200640 1 --2- 172.21.15.43:0/3061319306 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd294105350 0x7fd2941a1040 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd290009fd0 tx=0x7fd29000ef10 comp rx=0 tx=0).stop 2024-03-28T22:35:59.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.006+0000 7fd29b200640 1 -- 172.21.15.43:0/3061319306 shutdown_connections 2024-03-28T22:35:59.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.006+0000 7fd29b200640 1 --2- 172.21.15.43:0/3061319306 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd27c0a1d80 0x7fd27c0a4240 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:59.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.006+0000 7fd29b200640 1 --2- 172.21.15.43:0/3061319306 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd294105350 0x7fd2941a1040 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:59.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.006+0000 7fd29b200640 1 --2- 172.21.15.43:0/3061319306 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd294104990 0x7fd294199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:59.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.006+0000 7fd29b200640 1 --2- 172.21.15.43:0/3061319306 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2941037a0 0x7fd294199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:59.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.006+0000 7fd29b200640 1 -- 172.21.15.43:0/3061319306 >> 172.21.15.43:0/3061319306 conn(0x7fd2940fed30 msgr2=0x7fd294107c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:59.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.006+0000 7fd29b200640 1 -- 172.21.15.43:0/3061319306 shutdown_connections 2024-03-28T22:35:59.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.006+0000 7fd29b200640 1 -- 172.21.15.43:0/3061319306 wait complete. 2024-03-28T22:35:59.023 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T22:28:14.713290+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T22:33:27.167435+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T22:33:28.411825+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T22:33:32.523122+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T22:35:59.024 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T22:35:59.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.259+0000 7f174f200640 1 -- 172.21.15.43:0/3347018617 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1748104b80 msgr2=0x7f1748105000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:59.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.259+0000 7f174f200640 1 --2- 172.21.15.43:0/3347018617 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1748104b80 0x7f1748105000 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1730009ca0 tx=0x7f173002c100 comp rx=0 tx=0).stop 2024-03-28T22:35:59.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.259+0000 7f174f200640 1 -- 172.21.15.43:0/3347018617 shutdown_connections 2024-03-28T22:35:59.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.259+0000 7f174f200640 1 --2- 172.21.15.43:0/3347018617 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1748105540 0x7f174810bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:59.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.259+0000 7f174f200640 1 --2- 172.21.15.43:0/3347018617 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1748104b80 0x7f1748105000 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:59.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.259+0000 7f174f200640 1 --2- 172.21.15.43:0/3347018617 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1748103990 0x7f1748103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:59.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.259+0000 7f174f200640 1 -- 172.21.15.43:0/3347018617 >> 172.21.15.43:0/3347018617 conn(0x7f17480fef00 msgr2=0x7f1748101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:59.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.259+0000 7f174f200640 1 -- 172.21.15.43:0/3347018617 shutdown_connections 2024-03-28T22:35:59.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.259+0000 7f174f200640 1 -- 172.21.15.43:0/3347018617 wait complete. 2024-03-28T22:35:59.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.260+0000 7f174f200640 1 Processor -- start 2024-03-28T22:35:59.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.260+0000 7f174f200640 1 -- start start 2024-03-28T22:35:59.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.261+0000 7f174f200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1748103990 0x7f1748199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:59.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.261+0000 7f174f200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1748104b80 0x7f174819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:59.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.261+0000 7f174f200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1748105540 0x7f17481a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:59.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.261+0000 7f174f200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f174810f5f0 con 0x7f1748105540 2024-03-28T22:35:59.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.261+0000 7f174f200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f174810f470 con 0x7f1748104b80 2024-03-28T22:35:59.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.261+0000 7f174f200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f174810f770 con 0x7f1748103990 2024-03-28T22:35:59.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.261+0000 7f1747e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1748103990 0x7f1748199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:59.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.261+0000 7f1747400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1748104b80 0x7f174819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:59.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.261+0000 7f1747e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1748103990 0x7f1748199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35860/0 (socket says 172.21.15.43:35860) 2024-03-28T22:35:59.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.261+0000 7f1747e00640 1 -- 172.21.15.43:0/2580893659 learned_addr learned my addr 172.21.15.43:0/2580893659 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:35:59.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.261+0000 7f174d200640 1 --2- 172.21.15.43:0/2580893659 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1748105540 0x7f17481a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:59.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.261+0000 7f1747400640 1 -- 172.21.15.43:0/2580893659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1748103990 msgr2=0x7f1748199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:59.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.261+0000 7f1747400640 1 --2- 172.21.15.43:0/2580893659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1748103990 0x7f1748199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:59.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.261+0000 7f1747400640 1 -- 172.21.15.43:0/2580893659 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1748105540 msgr2=0x7f17481a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:59.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.261+0000 7f1747400640 1 --2- 172.21.15.43:0/2580893659 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1748105540 0x7f17481a11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:59.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.261+0000 7f1747400640 1 -- 172.21.15.43:0/2580893659 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17481a1890 con 0x7f1748104b80 2024-03-28T22:35:59.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.261+0000 7f174d200640 1 --2- 172.21.15.43:0/2580893659 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1748105540 0x7f17481a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:35:59.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.262+0000 7f1747e00640 1 --2- 172.21.15.43:0/2580893659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1748103990 0x7f1748199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:35:59.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.262+0000 7f1747400640 1 --2- 172.21.15.43:0/2580893659 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1748104b80 0x7f174819a1c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f173002c640 tx=0x7f1730035d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:59.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.262+0000 7f1744c00640 1 -- 172.21.15.43:0/2580893659 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1730039040 con 0x7f1748104b80 2024-03-28T22:35:59.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.262+0000 7f1744c00640 1 -- 172.21.15.43:0/2580893659 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1730003970 con 0x7f1748104b80 2024-03-28T22:35:59.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.262+0000 7f1744c00640 1 -- 172.21.15.43:0/2580893659 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1730004330 con 0x7f1748104b80 2024-03-28T22:35:59.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.262+0000 7f174f200640 1 -- 172.21.15.43:0/2580893659 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17481a1b20 con 0x7f1748104b80 2024-03-28T22:35:59.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.262+0000 7f174f200640 1 -- 172.21.15.43:0/2580893659 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17481a2060 con 0x7f1748104b80 2024-03-28T22:35:59.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.263+0000 7f174f200640 1 -- 172.21.15.43:0/2580893659 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f174810acb0 con 0x7f1748104b80 2024-03-28T22:35:59.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.265+0000 7f1744c00640 1 -- 172.21.15.43:0/2580893659 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f1730046030 con 0x7f1748104b80 2024-03-28T22:35:59.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.267+0000 7f1744c00640 1 --2- 172.21.15.43:0/2580893659 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f17280a1e00 0x7f17280a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:59.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.267+0000 7f1744c00640 1 -- 172.21.15.43:0/2580893659 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f17300efe40 con 0x7f1748104b80 2024-03-28T22:35:59.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.267+0000 7f1747e00640 1 --2- 172.21.15.43:0/2580893659 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f17280a1e00 0x7f17280a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:59.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.268+0000 7f1747e00640 1 --2- 172.21.15.43:0/2580893659 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f17280a1e00 0x7f17280a42c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f174819b150 tx=0x7f1738005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:59.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.269+0000 7f1744c00640 1 -- 172.21.15.43:0/2580893659 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f173004d060 con 0x7f1748104b80 2024-03-28T22:35:59.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.473+0000 7f174f200640 1 -- 172.21.15.43:0/2580893659 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f174802d400 con 0x7f1748104b80 2024-03-28T22:35:59.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.473+0000 7f1744c00640 1 -- 172.21.15.43:0/2580893659 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) ==== 93+0+10 (secure 0 0 0) 0x7f17300bce80 con 0x7f1748104b80 2024-03-28T22:35:59.471 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-03-28T22:35:59.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.476+0000 7f174f200640 1 -- 172.21.15.43:0/2580893659 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f17280a1e00 msgr2=0x7f17280a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:59.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.476+0000 7f174f200640 1 --2- 172.21.15.43:0/2580893659 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f17280a1e00 0x7f17280a42c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f174819b150 tx=0x7f1738005f90 comp rx=0 tx=0).stop 2024-03-28T22:35:59.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.476+0000 7f174f200640 1 -- 172.21.15.43:0/2580893659 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1748104b80 msgr2=0x7f174819a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:59.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.476+0000 7f174f200640 1 --2- 172.21.15.43:0/2580893659 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1748104b80 0x7f174819a1c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f173002c640 tx=0x7f1730035d90 comp rx=0 tx=0).stop 2024-03-28T22:35:59.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.476+0000 7f174f200640 1 -- 172.21.15.43:0/2580893659 shutdown_connections 2024-03-28T22:35:59.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.476+0000 7f174f200640 1 --2- 172.21.15.43:0/2580893659 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f17280a1e00 0x7f17280a42c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:59.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.477+0000 7f174f200640 1 --2- 172.21.15.43:0/2580893659 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1748104b80 0x7f174819a1c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:59.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.477+0000 7f174f200640 1 --2- 172.21.15.43:0/2580893659 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1748105540 0x7f17481a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:59.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.477+0000 7f174f200640 1 --2- 172.21.15.43:0/2580893659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1748103990 0x7f1748199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:59.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.477+0000 7f174f200640 1 -- 172.21.15.43:0/2580893659 >> 172.21.15.43:0/2580893659 conn(0x7f17480fef00 msgr2=0x7f1748107e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:59.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.477+0000 7f174f200640 1 -- 172.21.15.43:0/2580893659 shutdown_connections 2024-03-28T22:35:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.477+0000 7f174f200640 1 -- 172.21.15.43:0/2580893659 wait complete. 2024-03-28T22:35:59.493 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T22:35:59.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:59 smithi172 ceph-mon[36566]: pgmap v401: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:35:59.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:35:59 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3061319306' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:35:59.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:59 smithi082 ceph-mon[36471]: pgmap v401: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:35:59.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:35:59 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3061319306' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:35:59.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.717+0000 7f530b200640 1 -- 172.21.15.43:0/67160528 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f530c108e60 msgr2=0x7f530c1092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:59.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.717+0000 7f530b200640 1 --2- 172.21.15.43:0/67160528 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f530c108e60 0x7f530c1092e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f52f4009900 tx=0x7f52f402bc30 comp rx=0 tx=0).stop 2024-03-28T22:35:59.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.717+0000 7f530b200640 1 -- 172.21.15.43:0/67160528 shutdown_connections 2024-03-28T22:35:59.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.717+0000 7f530b200640 1 --2- 172.21.15.43:0/67160528 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f530c108e60 0x7f530c1092e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:59.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.717+0000 7f530b200640 1 --2- 172.21.15.43:0/67160528 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f530c109820 0x7f530c1100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:59.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.717+0000 7f530b200640 1 --2- 172.21.15.43:0/67160528 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f530c107c70 0x7f530c108090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:59.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.717+0000 7f530b200640 1 -- 172.21.15.43:0/67160528 >> 172.21.15.43:0/67160528 conn(0x7f530c076c20 msgr2=0x7f530c079040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:59.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.718+0000 7f530b200640 1 -- 172.21.15.43:0/67160528 shutdown_connections 2024-03-28T22:35:59.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.718+0000 7f530b200640 1 -- 172.21.15.43:0/67160528 wait complete. 2024-03-28T22:35:59.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.719+0000 7f530b200640 1 Processor -- start 2024-03-28T22:35:59.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.719+0000 7f530b200640 1 -- start start 2024-03-28T22:35:59.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.719+0000 7f530b200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f530c107c70 0x7f530c19df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:59.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.720+0000 7f530b200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f530c108e60 0x7f530c19e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:59.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.720+0000 7f530b200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f530c109820 0x7f530c1a5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:59.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.720+0000 7f530b200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f530c113940 con 0x7f530c108e60 2024-03-28T22:35:59.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.720+0000 7f530b200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f530c1137c0 con 0x7f530c109820 2024-03-28T22:35:59.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.720+0000 7f530b200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f530c113ac0 con 0x7f530c107c70 2024-03-28T22:35:59.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.720+0000 7f530a800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f530c109820 0x7f530c1a5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:59.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.720+0000 7f530a800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f530c109820 0x7f530c1a5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47944/0 (socket says 172.21.15.43:47944) 2024-03-28T22:35:59.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.720+0000 7f530a800640 1 -- 172.21.15.43:0/2232800074 learned_addr learned my addr 172.21.15.43:0/2232800074 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:35:59.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.720+0000 7f5309e00640 1 --2- 172.21.15.43:0/2232800074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f530c107c70 0x7f530c19df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:59.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.720+0000 7f5303e00640 1 --2- 172.21.15.43:0/2232800074 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f530c108e60 0x7f530c19e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:59.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.720+0000 7f530a800640 1 -- 172.21.15.43:0/2232800074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f530c107c70 msgr2=0x7f530c19df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:59.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.720+0000 7f530a800640 1 --2- 172.21.15.43:0/2232800074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f530c107c70 0x7f530c19df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:59.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.720+0000 7f530a800640 1 -- 172.21.15.43:0/2232800074 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f530c108e60 msgr2=0x7f530c19e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:59.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.720+0000 7f530a800640 1 --2- 172.21.15.43:0/2232800074 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f530c108e60 0x7f530c19e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:59.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.720+0000 7f530a800640 1 -- 172.21.15.43:0/2232800074 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f530c1a5b80 con 0x7f530c109820 2024-03-28T22:35:59.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.720+0000 7f5303e00640 1 --2- 172.21.15.43:0/2232800074 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f530c108e60 0x7f530c19e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:35:59.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.720+0000 7f5309e00640 1 --2- 172.21.15.43:0/2232800074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f530c107c70 0x7f530c19df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:35:59.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.721+0000 7f530a800640 1 --2- 172.21.15.43:0/2232800074 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f530c109820 0x7f530c1a5510 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f530400eb10 tx=0x7f530400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:59.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.721+0000 7f5302a00640 1 -- 172.21.15.43:0/2232800074 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5304004b20 con 0x7f530c109820 2024-03-28T22:35:59.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.721+0000 7f5302a00640 1 -- 172.21.15.43:0/2232800074 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5304004cc0 con 0x7f530c109820 2024-03-28T22:35:59.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.721+0000 7f5302a00640 1 -- 172.21.15.43:0/2232800074 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5304010760 con 0x7f530c109820 2024-03-28T22:35:59.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.721+0000 7f530b200640 1 -- 172.21.15.43:0/2232800074 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f530c1a5e70 con 0x7f530c109820 2024-03-28T22:35:59.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.721+0000 7f530b200640 1 -- 172.21.15.43:0/2232800074 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f530c1a62e0 con 0x7f530c109820 2024-03-28T22:35:59.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.722+0000 7f530b200640 1 -- 172.21.15.43:0/2232800074 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f530c10ef90 con 0x7f530c109820 2024-03-28T22:35:59.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.724+0000 7f5302a00640 1 -- 172.21.15.43:0/2232800074 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f5304005020 con 0x7f530c109820 2024-03-28T22:35:59.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.725+0000 7f5302a00640 1 --2- 172.21.15.43:0/2232800074 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f52f00a1e00 0x7f52f00a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:35:59.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.726+0000 7f5302a00640 1 -- 172.21.15.43:0/2232800074 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f53040ce8a0 con 0x7f530c109820 2024-03-28T22:35:59.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.726+0000 7f5309e00640 1 --2- 172.21.15.43:0/2232800074 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f52f00a1e00 0x7f52f00a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:35:59.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.727+0000 7f5309e00640 1 --2- 172.21.15.43:0/2232800074 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f52f00a1e00 0x7f52f00a42c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f530c108cc0 tx=0x7f52fc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:35:59.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.729+0000 7f5302a00640 1 -- 172.21.15.43:0/2232800074 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f530409b8e0 con 0x7f530c109820 2024-03-28T22:35:59.765 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:59 smithi043 ceph-mon[31823]: pgmap v401: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:35:59.765 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:35:59 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3061319306' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:35:59.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.935+0000 7f530b200640 1 -- 172.21.15.43:0/2232800074 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f530c02d390 con 0x7f530c109820 2024-03-28T22:35:59.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.935+0000 7f5302a00640 1 -- 172.21.15.43:0/2232800074 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) ==== 104+0+11 (secure 0 0 0) 0x7f530409b1e0 con 0x7f530c109820 2024-03-28T22:35:59.933 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-03-28T22:35:59.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.938+0000 7f530b200640 1 -- 172.21.15.43:0/2232800074 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f52f00a1e00 msgr2=0x7f52f00a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:59.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.938+0000 7f530b200640 1 --2- 172.21.15.43:0/2232800074 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f52f00a1e00 0x7f52f00a42c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f530c108cc0 tx=0x7f52fc005f90 comp rx=0 tx=0).stop 2024-03-28T22:35:59.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.938+0000 7f530b200640 1 -- 172.21.15.43:0/2232800074 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f530c109820 msgr2=0x7f530c1a5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:35:59.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.938+0000 7f530b200640 1 --2- 172.21.15.43:0/2232800074 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f530c109820 0x7f530c1a5510 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f530400eb10 tx=0x7f530400c2d0 comp rx=0 tx=0).stop 2024-03-28T22:35:59.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.938+0000 7f530b200640 1 -- 172.21.15.43:0/2232800074 shutdown_connections 2024-03-28T22:35:59.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.938+0000 7f530b200640 1 --2- 172.21.15.43:0/2232800074 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f52f00a1e00 0x7f52f00a42c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:59.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.938+0000 7f530b200640 1 --2- 172.21.15.43:0/2232800074 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f530c109820 0x7f530c1a5510 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:59.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.938+0000 7f530b200640 1 --2- 172.21.15.43:0/2232800074 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f530c108e60 0x7f530c19e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:59.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.938+0000 7f530b200640 1 --2- 172.21.15.43:0/2232800074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f530c107c70 0x7f530c19df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:35:59.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.938+0000 7f530b200640 1 -- 172.21.15.43:0/2232800074 >> 172.21.15.43:0/2232800074 conn(0x7f530c076c20 msgr2=0x7f530c10c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:35:59.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.939+0000 7f530b200640 1 -- 172.21.15.43:0/2232800074 shutdown_connections 2024-03-28T22:35:59.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:35:59.939+0000 7f530b200640 1 -- 172.21.15.43:0/2232800074 wait complete. 2024-03-28T22:35:59.955 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T22:36:00.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.186+0000 7f6a66e00640 1 -- 172.21.15.43:0/1752345402 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a60075790 msgr2=0x7f6a60073be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:00.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.186+0000 7f6a66e00640 1 --2- 172.21.15.43:0/1752345402 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a60075790 0x7f6a60073be0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6a54009900 tx=0x7f6a5402bc30 comp rx=0 tx=0).stop 2024-03-28T22:36:00.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.187+0000 7f6a66e00640 1 -- 172.21.15.43:0/1752345402 shutdown_connections 2024-03-28T22:36:00.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.187+0000 7f6a66e00640 1 --2- 172.21.15.43:0/1752345402 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a601094f0 0x7f6a6010b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:00.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.187+0000 7f6a66e00640 1 --2- 172.21.15.43:0/1752345402 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a600741b0 0x7f6a60074630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:00.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.187+0000 7f6a66e00640 1 --2- 172.21.15.43:0/1752345402 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a60075790 0x7f6a60073be0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:00.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.187+0000 7f6a66e00640 1 -- 172.21.15.43:0/1752345402 >> 172.21.15.43:0/1752345402 conn(0x7f6a600fcbc0 msgr2=0x7f6a600fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:00.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.187+0000 7f6a66e00640 1 -- 172.21.15.43:0/1752345402 shutdown_connections 2024-03-28T22:36:00.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.187+0000 7f6a66e00640 1 -- 172.21.15.43:0/1752345402 wait complete. 2024-03-28T22:36:00.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.188+0000 7f6a66e00640 1 Processor -- start 2024-03-28T22:36:00.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.188+0000 7f6a66e00640 1 -- start start 2024-03-28T22:36:00.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.189+0000 7f6a66e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a600741b0 0x7f6a60199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:00.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.189+0000 7f6a66e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a60075790 0x7f6a6019a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:00.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.189+0000 7f6a66e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a601094f0 0x7f6a601a1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:00.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.189+0000 7f6a66e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6010f470 con 0x7f6a60075790 2024-03-28T22:36:00.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.189+0000 7f6a66e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6010f2f0 con 0x7f6a600741b0 2024-03-28T22:36:00.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.189+0000 7f6a66e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6010f5f0 con 0x7f6a601094f0 2024-03-28T22:36:00.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.189+0000 7f6a65a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a600741b0 0x7f6a60199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:00.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.189+0000 7f6a65a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a600741b0 0x7f6a60199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47962/0 (socket says 172.21.15.43:47962) 2024-03-28T22:36:00.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.189+0000 7f6a65a00640 1 -- 172.21.15.43:0/3787165761 learned_addr learned my addr 172.21.15.43:0/3787165761 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:00.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.189+0000 7f6a65000640 1 --2- 172.21.15.43:0/3787165761 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a60075790 0x7f6a6019a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:00.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.189+0000 7f6a66400640 1 --2- 172.21.15.43:0/3787165761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a601094f0 0x7f6a601a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:00.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.189+0000 7f6a65a00640 1 -- 172.21.15.43:0/3787165761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a601094f0 msgr2=0x7f6a601a1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:00.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.190+0000 7f6a65a00640 1 --2- 172.21.15.43:0/3787165761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a601094f0 0x7f6a601a1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:00.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.190+0000 7f6a65a00640 1 -- 172.21.15.43:0/3787165761 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a60075790 msgr2=0x7f6a6019a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:00.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.190+0000 7f6a65a00640 1 --2- 172.21.15.43:0/3787165761 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a60075790 0x7f6a6019a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:00.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.190+0000 7f6a65a00640 1 -- 172.21.15.43:0/3787165761 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a601a16f0 con 0x7f6a600741b0 2024-03-28T22:36:00.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.190+0000 7f6a66400640 1 --2- 172.21.15.43:0/3787165761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a601094f0 0x7f6a601a1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:00.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.190+0000 7f6a65a00640 1 --2- 172.21.15.43:0/3787165761 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a600741b0 0x7f6a60199ad0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6a54009230 tx=0x7f6a5402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:00.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.190+0000 7f6a5e000640 1 -- 172.21.15.43:0/3787165761 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a54041070 con 0x7f6a600741b0 2024-03-28T22:36:00.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.190+0000 7f6a5e000640 1 -- 172.21.15.43:0/3787165761 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a54004450 con 0x7f6a600741b0 2024-03-28T22:36:00.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.190+0000 7f6a5e000640 1 -- 172.21.15.43:0/3787165761 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a54003e20 con 0x7f6a600741b0 2024-03-28T22:36:00.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.190+0000 7f6a66e00640 1 -- 172.21.15.43:0/3787165761 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a601a1980 con 0x7f6a600741b0 2024-03-28T22:36:00.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.190+0000 7f6a66e00640 1 -- 172.21.15.43:0/3787165761 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a601a1ec0 con 0x7f6a600741b0 2024-03-28T22:36:00.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.192+0000 7f6a66e00640 1 -- 172.21.15.43:0/3787165761 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a60108220 con 0x7f6a600741b0 2024-03-28T22:36:00.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.193+0000 7f6a5e000640 1 -- 172.21.15.43:0/3787165761 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f6a54044070 con 0x7f6a600741b0 2024-03-28T22:36:00.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.194+0000 7f6a5e000640 1 --2- 172.21.15.43:0/3787165761 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f6a480a1e00 0x7f6a480a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:00.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.195+0000 7f6a5e000640 1 -- 172.21.15.43:0/3787165761 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f6a540efcd0 con 0x7f6a600741b0 2024-03-28T22:36:00.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.195+0000 7f6a65000640 1 --2- 172.21.15.43:0/3787165761 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f6a480a1e00 0x7f6a480a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:00.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.196+0000 7f6a65000640 1 --2- 172.21.15.43:0/3787165761 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f6a480a1e00 0x7f6a480a42c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6a6019afe0 tx=0x7f6a4c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:00.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.198+0000 7f6a5e000640 1 -- 172.21.15.43:0/3787165761 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6a540bc6c0 con 0x7f6a600741b0 2024-03-28T22:36:00.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.403+0000 7f6a66e00640 1 -- 172.21.15.43:0/3787165761 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6a60062350 con 0x7f6a600741b0 2024-03-28T22:36:00.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.403+0000 7f6a5e000640 1 -- 172.21.15.43:0/3787165761 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) ==== 100+0+11 (secure 0 0 0) 0x7f6a540c0760 con 0x7f6a600741b0 2024-03-28T22:36:00.401 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-03-28T22:36:00.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.406+0000 7f6a66e00640 1 -- 172.21.15.43:0/3787165761 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f6a480a1e00 msgr2=0x7f6a480a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:00.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.406+0000 7f6a66e00640 1 --2- 172.21.15.43:0/3787165761 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f6a480a1e00 0x7f6a480a42c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6a6019afe0 tx=0x7f6a4c005f90 comp rx=0 tx=0).stop 2024-03-28T22:36:00.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.406+0000 7f6a66e00640 1 -- 172.21.15.43:0/3787165761 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a600741b0 msgr2=0x7f6a60199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:00.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.406+0000 7f6a66e00640 1 --2- 172.21.15.43:0/3787165761 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a600741b0 0x7f6a60199ad0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6a54009230 tx=0x7f6a5402c650 comp rx=0 tx=0).stop 2024-03-28T22:36:00.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.406+0000 7f6a66e00640 1 -- 172.21.15.43:0/3787165761 shutdown_connections 2024-03-28T22:36:00.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.406+0000 7f6a66e00640 1 --2- 172.21.15.43:0/3787165761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a601094f0 0x7f6a601a1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:00.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.406+0000 7f6a66e00640 1 --2- 172.21.15.43:0/3787165761 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a60075790 0x7f6a6019a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:00.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.406+0000 7f6a66e00640 1 --2- 172.21.15.43:0/3787165761 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f6a480a1e00 0x7f6a480a42c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:00.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.406+0000 7f6a66e00640 1 --2- 172.21.15.43:0/3787165761 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a600741b0 0x7f6a60199ad0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:00.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.406+0000 7f6a66e00640 1 -- 172.21.15.43:0/3787165761 >> 172.21.15.43:0/3787165761 conn(0x7f6a600fcbc0 msgr2=0x7f6a601053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:00.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.406+0000 7f6a66e00640 1 -- 172.21.15.43:0/3787165761 shutdown_connections 2024-03-28T22:36:00.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.406+0000 7f6a66e00640 1 -- 172.21.15.43:0/3787165761 wait complete. 2024-03-28T22:36:00.425 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T22:36:00.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:00 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2580893659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:36:00.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:00 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2232800074' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:36:00.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.653+0000 7fd742e00640 1 -- 172.21.15.43:0/2585313215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd73c100800 msgr2=0x7fd73c100c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:00.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.653+0000 7fd742e00640 1 --2- 172.21.15.43:0/2585313215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd73c100800 0x7fd73c100c60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd728009900 tx=0x7fd72802bc40 comp rx=0 tx=0).stop 2024-03-28T22:36:00.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.653+0000 7fd742e00640 1 -- 172.21.15.43:0/2585313215 shutdown_connections 2024-03-28T22:36:00.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.653+0000 7fd742e00640 1 --2- 172.21.15.43:0/2585313215 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd73c1011a0 0x7fd73c101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:00.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.653+0000 7fd742e00640 1 --2- 172.21.15.43:0/2585313215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd73c100800 0x7fd73c100c60 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:00.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.653+0000 7fd742e00640 1 --2- 172.21.15.43:0/2585313215 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd73c0ffea0 0x7fd73c1002c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:00.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.653+0000 7fd742e00640 1 -- 172.21.15.43:0/2585313215 >> 172.21.15.43:0/2585313215 conn(0x7fd73c0fbac0 msgr2=0x7fd73c0fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:00.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.653+0000 7fd742e00640 1 -- 172.21.15.43:0/2585313215 shutdown_connections 2024-03-28T22:36:00.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.653+0000 7fd742e00640 1 -- 172.21.15.43:0/2585313215 wait complete. 2024-03-28T22:36:00.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.654+0000 7fd742e00640 1 Processor -- start 2024-03-28T22:36:00.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.654+0000 7fd742e00640 1 -- start start 2024-03-28T22:36:00.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.655+0000 7fd742e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd73c0ffea0 0x7fd73c199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:00.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.655+0000 7fd742e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd73c100800 0x7fd73c19a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:00.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.655+0000 7fd742e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd73c1011a0 0x7fd73c1a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:00.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.655+0000 7fd742e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd73c10f4e0 con 0x7fd73c100800 2024-03-28T22:36:00.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.655+0000 7fd742e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd73c10f360 con 0x7fd73c0ffea0 2024-03-28T22:36:00.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.655+0000 7fd742e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd73c10f660 con 0x7fd73c1011a0 2024-03-28T22:36:00.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.655+0000 7fd741a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd73c0ffea0 0x7fd73c199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:00.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.655+0000 7fd742400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd73c1011a0 0x7fd73c1a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:00.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.655+0000 7fd742400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd73c1011a0 0x7fd73c1a1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35908/0 (socket says 172.21.15.43:35908) 2024-03-28T22:36:00.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.655+0000 7fd742400640 1 -- 172.21.15.43:0/1134593448 learned_addr learned my addr 172.21.15.43:0/1134593448 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:00.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.655+0000 7fd741000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd73c100800 0x7fd73c19a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:00.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.656+0000 7fd741a00640 1 -- 172.21.15.43:0/1134593448 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd73c1011a0 msgr2=0x7fd73c1a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:00.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.656+0000 7fd741a00640 1 --2- 172.21.15.43:0/1134593448 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd73c1011a0 0x7fd73c1a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:00.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.656+0000 7fd741a00640 1 -- 172.21.15.43:0/1134593448 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd73c100800 msgr2=0x7fd73c19a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:00.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.656+0000 7fd741a00640 1 --2- 172.21.15.43:0/1134593448 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd73c100800 0x7fd73c19a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:00.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.656+0000 7fd741a00640 1 -- 172.21.15.43:0/1134593448 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd73c1a1770 con 0x7fd73c0ffea0 2024-03-28T22:36:00.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.656+0000 7fd741a00640 1 --2- 172.21.15.43:0/1134593448 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd73c0ffea0 0x7fd73c199b50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd73000e9f0 tx=0x7fd73000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:00.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.656+0000 7fd73a000640 1 -- 172.21.15.43:0/1134593448 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd73000ccc0 con 0x7fd73c0ffea0 2024-03-28T22:36:00.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.656+0000 7fd73a000640 1 -- 172.21.15.43:0/1134593448 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd730005020 con 0x7fd73c0ffea0 2024-03-28T22:36:00.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.656+0000 7fd742e00640 1 -- 172.21.15.43:0/1134593448 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd73c1a1a60 con 0x7fd73c0ffea0 2024-03-28T22:36:00.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.656+0000 7fd73a000640 1 -- 172.21.15.43:0/1134593448 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7300049e0 con 0x7fd73c0ffea0 2024-03-28T22:36:00.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.656+0000 7fd742e00640 1 -- 172.21.15.43:0/1134593448 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd73c1a1f30 con 0x7fd73c0ffea0 2024-03-28T22:36:00.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.658+0000 7fd742e00640 1 -- 172.21.15.43:0/1134593448 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd73c10ad90 con 0x7fd73c0ffea0 2024-03-28T22:36:00.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.660+0000 7fd73a000640 1 -- 172.21.15.43:0/1134593448 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fd730005560 con 0x7fd73c0ffea0 2024-03-28T22:36:00.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.661+0000 7fd73a000640 1 --2- 172.21.15.43:0/1134593448 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd7240a1d80 0x7fd7240a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:00.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.661+0000 7fd73a000640 1 -- 172.21.15.43:0/1134593448 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7fd7300ce120 con 0x7fd73c0ffea0 2024-03-28T22:36:00.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.661+0000 7fd741000640 1 --2- 172.21.15.43:0/1134593448 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd7240a1d80 0x7fd7240a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:00.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.662+0000 7fd741000640 1 --2- 172.21.15.43:0/1134593448 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd7240a1d80 0x7fd7240a4240 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd72802c250 tx=0x7fd72802cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:00.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.665+0000 7fd73a000640 1 -- 172.21.15.43:0/1134593448 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd7300051d0 con 0x7fd73c0ffea0 2024-03-28T22:36:00.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:00 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2580893659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:36:00.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:00 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2232800074' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:36:00.765 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:00 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2580893659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:36:00.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:00 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2232800074' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:36:00.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.867+0000 7fd742e00640 1 -- 172.21.15.43:0/1134593448 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd73c062350 con 0x7fd73c0ffea0 2024-03-28T22:36:00.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.868+0000 7fd73a000640 1 -- 172.21.15.43:0/1134593448 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v96) ==== 103+0+11 (secure 0 0 0) 0x7fd73009b160 con 0x7fd73c0ffea0 2024-03-28T22:36:00.866 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-03-28T22:36:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.871+0000 7fd742e00640 1 -- 172.21.15.43:0/1134593448 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd7240a1d80 msgr2=0x7fd7240a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.871+0000 7fd742e00640 1 --2- 172.21.15.43:0/1134593448 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd7240a1d80 0x7fd7240a4240 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd72802c250 tx=0x7fd72802cf90 comp rx=0 tx=0).stop 2024-03-28T22:36:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.871+0000 7fd742e00640 1 -- 172.21.15.43:0/1134593448 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd73c0ffea0 msgr2=0x7fd73c199b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.871+0000 7fd742e00640 1 --2- 172.21.15.43:0/1134593448 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd73c0ffea0 0x7fd73c199b50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd73000e9f0 tx=0x7fd73000eeb0 comp rx=0 tx=0).stop 2024-03-28T22:36:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.871+0000 7fd742e00640 1 -- 172.21.15.43:0/1134593448 shutdown_connections 2024-03-28T22:36:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.871+0000 7fd742e00640 1 --2- 172.21.15.43:0/1134593448 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd73c1011a0 0x7fd73c1a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.871+0000 7fd742e00640 1 --2- 172.21.15.43:0/1134593448 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd73c100800 0x7fd73c19a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.871+0000 7fd742e00640 1 --2- 172.21.15.43:0/1134593448 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd7240a1d80 0x7fd7240a4240 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.871+0000 7fd742e00640 1 --2- 172.21.15.43:0/1134593448 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd73c0ffea0 0x7fd73c199b50 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.871+0000 7fd742e00640 1 -- 172.21.15.43:0/1134593448 >> 172.21.15.43:0/1134593448 conn(0x7fd73c0fbac0 msgr2=0x7fd73c0fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:00.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.871+0000 7fd742e00640 1 -- 172.21.15.43:0/1134593448 shutdown_connections 2024-03-28T22:36:00.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:00.871+0000 7fd742e00640 1 -- 172.21.15.43:0/1134593448 wait complete. 2024-03-28T22:36:00.888 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:36:01.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.114+0000 7f1d25e00640 1 -- 172.21.15.43:0/1284507104 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1d200741b0 msgr2=0x7f1d20074630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:01.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.114+0000 7f1d25e00640 1 --2- 172.21.15.43:0/1284507104 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1d200741b0 0x7f1d20074630 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c009890 tx=0x7f1d0c02bb70 comp rx=0 tx=0).stop 2024-03-28T22:36:01.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.114+0000 7f1d25e00640 1 -- 172.21.15.43:0/1284507104 shutdown_connections 2024-03-28T22:36:01.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.114+0000 7f1d25e00640 1 --2- 172.21.15.43:0/1284507104 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1d200741b0 0x7f1d20074630 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:01.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.114+0000 7f1d25e00640 1 --2- 172.21.15.43:0/1284507104 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d2010db60 0x7f1d2010ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:01.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.114+0000 7f1d25e00640 1 --2- 172.21.15.43:0/1284507104 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d20075790 0x7f1d20073be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:01.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.114+0000 7f1d25e00640 1 -- 172.21.15.43:0/1284507104 >> 172.21.15.43:0/1284507104 conn(0x7f1d200fcae0 msgr2=0x7f1d200fef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:01.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.114+0000 7f1d25e00640 1 -- 172.21.15.43:0/1284507104 shutdown_connections 2024-03-28T22:36:01.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.114+0000 7f1d25e00640 1 -- 172.21.15.43:0/1284507104 wait complete. 2024-03-28T22:36:01.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.116+0000 7f1d25e00640 1 Processor -- start 2024-03-28T22:36:01.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.116+0000 7f1d25e00640 1 -- start start 2024-03-28T22:36:01.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.116+0000 7f1d25e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d200741b0 0x7f1d2019def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:01.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.116+0000 7f1d25e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1d20075790 0x7f1d2019e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:01.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.116+0000 7f1d25e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d2010db60 0x7f1d201a54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:01.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.117+0000 7f1d25e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1d201137e0 con 0x7f1d2010db60 2024-03-28T22:36:01.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.117+0000 7f1d25e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1d20113660 con 0x7f1d20075790 2024-03-28T22:36:01.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.117+0000 7f1d25e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f1d20113960 con 0x7f1d200741b0 2024-03-28T22:36:01.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.117+0000 7f1d1fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1d20075790 0x7f1d2019e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:01.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.117+0000 7f1d1fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1d20075790 0x7f1d2019e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48022/0 (socket says 172.21.15.43:48022) 2024-03-28T22:36:01.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.117+0000 7f1d1fe00640 1 -- 172.21.15.43:0/362605773 learned_addr learned my addr 172.21.15.43:0/362605773 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:01.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.117+0000 7f1d24a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d200741b0 0x7f1d2019def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:01.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.117+0000 7f1d25400640 1 --2- 172.21.15.43:0/362605773 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d2010db60 0x7f1d201a54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:01.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.117+0000 7f1d1fe00640 1 -- 172.21.15.43:0/362605773 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d200741b0 msgr2=0x7f1d2019def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:01.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.117+0000 7f1d1fe00640 1 --2- 172.21.15.43:0/362605773 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d200741b0 0x7f1d2019def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:01.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.117+0000 7f1d1fe00640 1 -- 172.21.15.43:0/362605773 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d2010db60 msgr2=0x7f1d201a54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:01.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.117+0000 7f1d1fe00640 1 --2- 172.21.15.43:0/362605773 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d2010db60 0x7f1d201a54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:01.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.117+0000 7f1d1fe00640 1 -- 172.21.15.43:0/362605773 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d201a5b10 con 0x7f1d20075790 2024-03-28T22:36:01.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.117+0000 7f1d1fe00640 1 --2- 172.21.15.43:0/362605773 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1d20075790 0x7f1d2019e430 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c000c00 tx=0x7f1d0c032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:01.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.118+0000 7f1d1d600640 1 -- 172.21.15.43:0/362605773 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c041020 con 0x7f1d20075790 2024-03-28T22:36:01.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.118+0000 7f1d1d600640 1 -- 172.21.15.43:0/362605773 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d0c02c9f0 con 0x7f1d20075790 2024-03-28T22:36:01.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.118+0000 7f1d25e00640 1 -- 172.21.15.43:0/362605773 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d201a5da0 con 0x7f1d20075790 2024-03-28T22:36:01.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.118+0000 7f1d1d600640 1 -- 172.21.15.43:0/362605773 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c039040 con 0x7f1d20075790 2024-03-28T22:36:01.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.118+0000 7f1d25e00640 1 -- 172.21.15.43:0/362605773 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d201a6280 con 0x7f1d20075790 2024-03-28T22:36:01.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.119+0000 7f1d25e00640 1 -- 172.21.15.43:0/362605773 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d20103d30 con 0x7f1d20075790 2024-03-28T22:36:01.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.121+0000 7f1d1d600640 1 -- 172.21.15.43:0/362605773 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f1d0c044020 con 0x7f1d20075790 2024-03-28T22:36:01.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.122+0000 7f1d1d600640 1 --2- 172.21.15.43:0/362605773 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f1d080a1e00 0x7f1d080a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:01.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.123+0000 7f1d1d600640 1 -- 172.21.15.43:0/362605773 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f1d0c0efae0 con 0x7f1d20075790 2024-03-28T22:36:01.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.123+0000 7f1d24a00640 1 --2- 172.21.15.43:0/362605773 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f1d080a1e00 0x7f1d080a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:01.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.123+0000 7f1d24a00640 1 --2- 172.21.15.43:0/362605773 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f1d080a1e00 0x7f1d080a42c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1d14009bd0 tx=0x7f1d14005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:01.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.126+0000 7f1d1d600640 1 -- 172.21.15.43:0/362605773 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1d0c04b020 con 0x7f1d20075790 2024-03-28T22:36:01.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.368+0000 7f1d25e00640 1 -- 172.21.15.43:0/362605773 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1d200623f0 con 0x7f1d20075790 2024-03-28T22:36:01.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.369+0000 7f1d1d600640 1 -- 172.21.15.43:0/362605773 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11934 (secure 0 0 0) 0x7f1d0c0bcb20 con 0x7f1d20075790 2024-03-28T22:36:01.368 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:36:01.368 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:33:28.906396+0000","modified":"2024-03-28T22:34:06.174761+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24572,"mds_1":24556,"mds_2":24535,"mds_3":24523,"mds_4":24518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/1084988072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1084988072},{"type":"v1","addr":"172.21.15.82:6835","nonce":1084988072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24535,24556,24518,24523,24572]},"id":1}]} 2024-03-28T22:36:01.368 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 28 2024-03-28T22:36:01.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.372+0000 7f1d25e00640 1 -- 172.21.15.43:0/362605773 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f1d080a1e00 msgr2=0x7f1d080a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:01.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.372+0000 7f1d25e00640 1 --2- 172.21.15.43:0/362605773 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f1d080a1e00 0x7f1d080a42c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1d14009bd0 tx=0x7f1d14005f90 comp rx=0 tx=0).stop 2024-03-28T22:36:01.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.372+0000 7f1d25e00640 1 -- 172.21.15.43:0/362605773 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1d20075790 msgr2=0x7f1d2019e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:01.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.372+0000 7f1d25e00640 1 --2- 172.21.15.43:0/362605773 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1d20075790 0x7f1d2019e430 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c000c00 tx=0x7f1d0c032430 comp rx=0 tx=0).stop 2024-03-28T22:36:01.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.372+0000 7f1d25e00640 1 -- 172.21.15.43:0/362605773 shutdown_connections 2024-03-28T22:36:01.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.372+0000 7f1d25e00640 1 --2- 172.21.15.43:0/362605773 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f1d080a1e00 0x7f1d080a42c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:01.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.372+0000 7f1d25e00640 1 --2- 172.21.15.43:0/362605773 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1d20075790 0x7f1d2019e430 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:01.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.372+0000 7f1d25e00640 1 --2- 172.21.15.43:0/362605773 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d2010db60 0x7f1d201a54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:01.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.372+0000 7f1d25e00640 1 --2- 172.21.15.43:0/362605773 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d200741b0 0x7f1d2019def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:01.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.372+0000 7f1d25e00640 1 -- 172.21.15.43:0/362605773 >> 172.21.15.43:0/362605773 conn(0x7f1d200fcae0 msgr2=0x7f1d20100f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:01.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.372+0000 7f1d25e00640 1 -- 172.21.15.43:0/362605773 shutdown_connections 2024-03-28T22:36:01.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.372+0000 7f1d25e00640 1 -- 172.21.15.43:0/362605773 wait complete. 2024-03-28T22:36:01.390 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:36:01.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.618+0000 7f41ff000640 1 -- 172.21.15.43:0/2818832865 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41f8104990 msgr2=0x7f41f8104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:01.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.618+0000 7f41ff000640 1 --2- 172.21.15.43:0/2818832865 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41f8104990 0x7f41f8104e10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f41e4009900 tx=0x7f41e402bc30 comp rx=0 tx=0).stop 2024-03-28T22:36:01.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.618+0000 7f41ff000640 1 -- 172.21.15.43:0/2818832865 shutdown_connections 2024-03-28T22:36:01.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.618+0000 7f41ff000640 1 --2- 172.21.15.43:0/2818832865 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41f8105350 0x7f41f810bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:01.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.618+0000 7f41ff000640 1 --2- 172.21.15.43:0/2818832865 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41f8104990 0x7f41f8104e10 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:01.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.618+0000 7f41ff000640 1 --2- 172.21.15.43:0/2818832865 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f41f81037a0 0x7f41f8103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:01.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.618+0000 7f41ff000640 1 -- 172.21.15.43:0/2818832865 >> 172.21.15.43:0/2818832865 conn(0x7f41f80fed30 msgr2=0x7f41f8101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:01.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.618+0000 7f41ff000640 1 -- 172.21.15.43:0/2818832865 shutdown_connections 2024-03-28T22:36:01.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.618+0000 7f41ff000640 1 -- 172.21.15.43:0/2818832865 wait complete. 2024-03-28T22:36:01.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.619+0000 7f41ff000640 1 Processor -- start 2024-03-28T22:36:01.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.619+0000 7f41ff000640 1 -- start start 2024-03-28T22:36:01.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.620+0000 7f41ff000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41f81037a0 0x7f41f8199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:01.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.620+0000 7f41ff000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41f8104990 0x7f41f819a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:01.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.620+0000 7f41ff000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f41f8105350 0x7f41f81a0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:01.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.620+0000 7f41ff000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f41f810f400 con 0x7f41f8104990 2024-03-28T22:36:01.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.620+0000 7f41ff000640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f41f810f280 con 0x7f41f8105350 2024-03-28T22:36:01.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.620+0000 7f41ff000640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f41f810f580 con 0x7f41f81037a0 2024-03-28T22:36:01.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.620+0000 7f41fdc00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41f81037a0 0x7f41f8199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:01.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.620+0000 7f41fdc00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41f81037a0 0x7f41f8199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35940/0 (socket says 172.21.15.43:35940) 2024-03-28T22:36:01.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.620+0000 7f41fdc00640 1 -- 172.21.15.43:0/3422657485 learned_addr learned my addr 172.21.15.43:0/3422657485 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:01.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.620+0000 7f41fd200640 1 --2- 172.21.15.43:0/3422657485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41f8104990 0x7f41f819a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:01.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.620+0000 7f41fe600640 1 --2- 172.21.15.43:0/3422657485 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f41f8105350 0x7f41f81a0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:01.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.621+0000 7f41fdc00640 1 -- 172.21.15.43:0/3422657485 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f41f8105350 msgr2=0x7f41f81a0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:01.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.621+0000 7f41fdc00640 1 --2- 172.21.15.43:0/3422657485 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f41f8105350 0x7f41f81a0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:01.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.621+0000 7f41fdc00640 1 -- 172.21.15.43:0/3422657485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41f8104990 msgr2=0x7f41f819a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:01.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.621+0000 7f41fdc00640 1 --2- 172.21.15.43:0/3422657485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41f8104990 0x7f41f819a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:01.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.621+0000 7f41fdc00640 1 -- 172.21.15.43:0/3422657485 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41f81a1650 con 0x7f41f81037a0 2024-03-28T22:36:01.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.621+0000 7f41fe600640 1 --2- 172.21.15.43:0/3422657485 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f41f8105350 0x7f41f81a0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:01.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.621+0000 7f41fdc00640 1 --2- 172.21.15.43:0/3422657485 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41f81037a0 0x7f41f8199ad0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f41ec00e9c0 tx=0x7f41ec00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:01.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.621+0000 7f41f6000640 1 -- 172.21.15.43:0/3422657485 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41ec00cd80 con 0x7f41f81037a0 2024-03-28T22:36:01.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.621+0000 7f41ff000640 1 -- 172.21.15.43:0/3422657485 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41f81a1940 con 0x7f41f81037a0 2024-03-28T22:36:01.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.621+0000 7f41f6000640 1 -- 172.21.15.43:0/3422657485 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41ec0035f0 con 0x7f41f81037a0 2024-03-28T22:36:01.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.621+0000 7f41f6000640 1 -- 172.21.15.43:0/3422657485 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41ec010690 con 0x7f41f81037a0 2024-03-28T22:36:01.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.621+0000 7f41ff000640 1 -- 172.21.15.43:0/3422657485 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41f81a1e60 con 0x7f41f81037a0 2024-03-28T22:36:01.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.622+0000 7f41ff000640 1 -- 172.21.15.43:0/3422657485 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41f80688b0 con 0x7f41f81037a0 2024-03-28T22:36:01.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.626+0000 7f41f6000640 1 -- 172.21.15.43:0/3422657485 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f41ec0037a0 con 0x7f41f81037a0 2024-03-28T22:36:01.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.628+0000 7f41f6000640 1 --2- 172.21.15.43:0/3422657485 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f41e00a1d80 0x7f41e00a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:01.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.628+0000 7f41f6000640 1 -- 172.21.15.43:0/3422657485 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f41ec0ce500 con 0x7f41f81037a0 2024-03-28T22:36:01.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.628+0000 7f41fd200640 1 --2- 172.21.15.43:0/3422657485 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f41e00a1d80 0x7f41e00a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:01.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.629+0000 7f41fd200640 1 --2- 172.21.15.43:0/3422657485 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f41e00a1d80 0x7f41e00a4240 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f41e402c7b0 tx=0x7f41e4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:01.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.633+0000 7f41f6000640 1 -- 172.21.15.43:0/3422657485 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f41ec09b540 con 0x7f41f81037a0 2024-03-28T22:36:01.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:01 smithi172 ceph-mon[36566]: pgmap v402: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-28T22:36:01.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:01 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3787165761' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:36:01.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:01 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1134593448' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:36:01.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:01 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/362605773' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:36:01.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:01 smithi082 ceph-mon[36471]: pgmap v402: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-28T22:36:01.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:01 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3787165761' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:36:01.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:01 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1134593448' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:36:01.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:01 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/362605773' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:36:01.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:01 smithi043 ceph-mon[31823]: pgmap v402: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-28T22:36:01.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:01 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3787165761' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:36:01.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:01 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1134593448' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:36:01.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:01 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/362605773' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:36:01.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.837+0000 7f41ff000640 1 -- 172.21.15.43:0/3422657485 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f41f802d1a0 con 0x7f41f81037a0 2024-03-28T22:36:01.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.839+0000 7f41f6000640 1 -- 172.21.15.43:0/3422657485 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+24712 (secure 0 0 0) 0x7f41ec09ae40 con 0x7f41f81037a0 2024-03-28T22:36:01.837 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:36:01.837 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":96,"fsid":"c2276428-ed51-11ee-95d0-87774f69a715","created":"2024-03-28T22:24:19.950084+0000","modified":"2024-03-28T22:35:30.237654+0000","last_up_change":"2024-03-28T22:32:31.078737+0000","last_in_change":"2024-03-28T22:32:10.147097+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:28:14.713290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:33:27.167435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:33:28.411825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T22:33:32.523122+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b855adcd-786c-4e42-b8fe-a515a77ce979","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6803","nonce":2826124409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6805","nonce":2826124409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6809","nonce":2826124409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6807","nonce":2826124409}]},"public_addr":"172.21.15.43:6803/2826124409","cluster_addr":"172.21.15.43:6805/2826124409","heartbeat_back_addr":"172.21.15.43:6809/2826124409","heartbeat_front_addr":"172.21.15.43:6807/2826124409","state":["exists","up"]},{"osd":1,"uuid":"031c143d-8ad1-4b68-8db9-3c717137e3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6801","nonce":3043263644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6803","nonce":3043263644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6807","nonce":3043263644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6805","nonce":3043263644}]},"public_addr":"172.21.15.82:6801/3043263644","cluster_addr":"172.21.15.82:6803/3043263644","heartbeat_back_addr":"172.21.15.82:6807/3043263644","heartbeat_front_addr":"172.21.15.82:6805/3043263644","state":["exists","up"]},{"osd":2,"uuid":"20632bbb-f07a-4ca1-b0c4-ff9d508e211c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6801","nonce":4213953645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6803","nonce":4213953645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6807","nonce":4213953645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6805","nonce":4213953645}]},"public_addr":"172.21.15.172:6801/4213953645","cluster_addr":"172.21.15.172:6803/4213953645","heartbeat_back_addr":"172.21.15.172:6807/4213953645","heartbeat_front_addr":"172.21.15.172:6805/4213953645","state":["exists","up"]},{"osd":3,"uuid":"9a1f2312-4dbd-475b-8736-d5ebc8788e09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6811","nonce":1597280920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6813","nonce":1597280920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6817","nonce":1597280920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6815","nonce":1597280920}]},"public_addr":"172.21.15.43:6811/1597280920","cluster_addr":"172.21.15.43:6813/1597280920","heartbeat_back_addr":"172.21.15.43:6817/1597280920","heartbeat_front_addr":"172.21.15.43:6815/1597280920","state":["exists","up"]},{"osd":4,"uuid":"29140ead-7075-42b2-90c1-27cae0b29ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6809","nonce":215451895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6811","nonce":215451895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6815","nonce":215451895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6813","nonce":215451895}]},"public_addr":"172.21.15.82:6809/215451895","cluster_addr":"172.21.15.82:6811/215451895","heartbeat_back_addr":"172.21.15.82:6815/215451895","heartbeat_front_addr":"172.21.15.82:6813/215451895","state":["exists","up"]},{"osd":5,"uuid":"59644c61-93dc-4536-b2ad-0256b24e5cb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6809","nonce":1576148933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6811","nonce":1576148933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6815","nonce":1576148933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6813","nonce":1576148933}]},"public_addr":"172.21.15.172:6809/1576148933","cluster_addr":"172.21.15.172:6811/1576148933","heartbeat_back_addr":"172.21.15.172:6815/1576148933","heartbeat_front_addr":"172.21.15.172:6813/1576148933","state":["exists","up"]},{"osd":6,"uuid":"22d54e57-c3f1-4ecf-9921-8e815b48f92a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6819","nonce":791044279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6821","nonce":791044279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6825","nonce":791044279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6823","nonce":791044279}]},"public_addr":"172.21.15.43:6819/791044279","cluster_addr":"172.21.15.43:6821/791044279","heartbeat_back_addr":"172.21.15.43:6825/791044279","heartbeat_front_addr":"172.21.15.43:6823/791044279","state":["exists","up"]},{"osd":7,"uuid":"248cd1e2-0d6b-452f-a0a8-c51fe1b975ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6817","nonce":2157777323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6819","nonce":2157777323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6823","nonce":2157777323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6821","nonce":2157777323}]},"public_addr":"172.21.15.82:6817/2157777323","cluster_addr":"172.21.15.82:6819/2157777323","heartbeat_back_addr":"172.21.15.82:6823/2157777323","heartbeat_front_addr":"172.21.15.82:6821/2157777323","state":["exists","up"]},{"osd":8,"uuid":"d98aab6e-8bae-4c24-906c-4f492f047b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6817","nonce":1095328248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6819","nonce":1095328248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6823","nonce":1095328248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6821","nonce":1095328248}]},"public_addr":"172.21.15.172:6817/1095328248","cluster_addr":"172.21.15.172:6819/1095328248","heartbeat_back_addr":"172.21.15.172:6823/1095328248","heartbeat_front_addr":"172.21.15.172:6821/1095328248","state":["exists","up"]},{"osd":9,"uuid":"19aa383e-a0f6-45ea-96cc-e7956535caa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6827","nonce":3657791498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6829","nonce":3657791498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6833","nonce":3657791498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6831","nonce":3657791498}]},"public_addr":"172.21.15.43:6827/3657791498","cluster_addr":"172.21.15.43:6829/3657791498","heartbeat_back_addr":"172.21.15.43:6833/3657791498","heartbeat_front_addr":"172.21.15.43:6831/3657791498","state":["exists","up"]},{"osd":10,"uuid":"666977e5-eaf7-41b9-a18f-c380e149345d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6825","nonce":3454954047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6827","nonce":3454954047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6831","nonce":3454954047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6829","nonce":3454954047}]},"public_addr":"172.21.15.82:6825/3454954047","cluster_addr":"172.21.15.82:6827/3454954047","heartbeat_back_addr":"172.21.15.82:6831/3454954047","heartbeat_front_addr":"172.21.15.82:6829/3454954047","state":["exists","up"]},{"osd":11,"uuid":"615cf3be-8076-470e-ad09-c880e7593717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6825","nonce":2563517144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6827","nonce":2563517144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6831","nonce":2563517144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6829","nonce":2563517144}]},"public_addr":"172.21.15.172:6825/2563517144","cluster_addr":"172.21.15.172:6827/2563517144","heartbeat_back_addr":"172.21.15.172:6831/2563517144","heartbeat_front_addr":"172.21.15.172:6829/2563517144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:21.727405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:47.100259+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:10.615438+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:43.632144+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:09.487940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:34.604463+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:09.594452+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:35.709795+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:01.184202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:35.455322+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:02.413166+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:28.550754+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.9","mappings":[{"from":5,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/578791143":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/4029313215":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/1574178482":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6801/3517712438":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6800/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/1375099638":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/2710117652":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6801/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/3215634168":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6800/3517712438":"2024-03-29T22:25:28.008215+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:36:01.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.841+0000 7f41ff000640 1 -- 172.21.15.43:0/3422657485 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f41e00a1d80 msgr2=0x7f41e00a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:01.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.842+0000 7f41ff000640 1 --2- 172.21.15.43:0/3422657485 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f41e00a1d80 0x7f41e00a4240 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f41e402c7b0 tx=0x7f41e4005fd0 comp rx=0 tx=0).stop 2024-03-28T22:36:01.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.842+0000 7f41ff000640 1 -- 172.21.15.43:0/3422657485 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41f81037a0 msgr2=0x7f41f8199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:01.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.842+0000 7f41ff000640 1 --2- 172.21.15.43:0/3422657485 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41f81037a0 0x7f41f8199ad0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f41ec00e9c0 tx=0x7f41ec00ee80 comp rx=0 tx=0).stop 2024-03-28T22:36:01.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.842+0000 7f41ff000640 1 -- 172.21.15.43:0/3422657485 shutdown_connections 2024-03-28T22:36:01.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.842+0000 7f41ff000640 1 --2- 172.21.15.43:0/3422657485 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f41e00a1d80 0x7f41e00a4240 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:01.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.842+0000 7f41ff000640 1 --2- 172.21.15.43:0/3422657485 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f41f8105350 0x7f41f81a0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:01.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.842+0000 7f41ff000640 1 --2- 172.21.15.43:0/3422657485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41f8104990 0x7f41f819a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:01.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.842+0000 7f41ff000640 1 --2- 172.21.15.43:0/3422657485 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f41f81037a0 0x7f41f8199ad0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:01.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.842+0000 7f41ff000640 1 -- 172.21.15.43:0/3422657485 >> 172.21.15.43:0/3422657485 conn(0x7f41f80fed30 msgr2=0x7f41f8107c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:01.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.842+0000 7f41ff000640 1 -- 172.21.15.43:0/3422657485 shutdown_connections 2024-03-28T22:36:01.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:01.842+0000 7f41ff000640 1 -- 172.21.15.43:0/3422657485 wait complete. 2024-03-28T22:36:01.860 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T22:28:14.713290+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T22:33:27.167435+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T22:33:28.411825+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T22:33:32.523122+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T22:36:01.860 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T22:36:02.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.092+0000 7f0617e00640 1 -- 172.21.15.43:0/3057097897 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0618103200 msgr2=0x7f0618109a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:02.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.092+0000 7f0617e00640 1 --2- 172.21.15.43:0/3057097897 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0618103200 0x7f0618109a90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0610009860 tx=0x7f061002bb70 comp rx=0 tx=0).stop 2024-03-28T22:36:02.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.092+0000 7f0617e00640 1 -- 172.21.15.43:0/3057097897 shutdown_connections 2024-03-28T22:36:02.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.092+0000 7f0617e00640 1 --2- 172.21.15.43:0/3057097897 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0618103200 0x7f0618109a90 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:02.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.092+0000 7f0617e00640 1 --2- 172.21.15.43:0/3057097897 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0618102840 0x7f0618102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:02.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.092+0000 7f0617e00640 1 --2- 172.21.15.43:0/3057097897 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0618101650 0x7f0618101a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:02.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.092+0000 7f0617e00640 1 -- 172.21.15.43:0/3057097897 >> 172.21.15.43:0/3057097897 conn(0x7f06180fcbc0 msgr2=0x7f06180ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:02.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.092+0000 7f0617e00640 1 -- 172.21.15.43:0/3057097897 shutdown_connections 2024-03-28T22:36:02.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.092+0000 7f0617e00640 1 -- 172.21.15.43:0/3057097897 wait complete. 2024-03-28T22:36:02.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.093+0000 7f0617e00640 1 Processor -- start 2024-03-28T22:36:02.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.093+0000 7f0617e00640 1 -- start start 2024-03-28T22:36:02.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.093+0000 7f0617e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0618101650 0x7f06181978f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:02.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.093+0000 7f0617e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0618102840 0x7f0618197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:02.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.093+0000 7f0617e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0618103200 0x7f061819eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:02.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.094+0000 7f0617e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f061810d2b0 con 0x7f0618102840 2024-03-28T22:36:02.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.094+0000 7f0617e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f061810d130 con 0x7f0618101650 2024-03-28T22:36:02.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.094+0000 7f0617e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f061810d430 con 0x7f0618103200 2024-03-28T22:36:02.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.094+0000 7f0616a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0618101650 0x7f06181978f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:02.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.094+0000 7f0616a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0618101650 0x7f06181978f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48054/0 (socket says 172.21.15.43:48054) 2024-03-28T22:36:02.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.094+0000 7f0617400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0618103200 0x7f061819eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:02.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.094+0000 7f0616a00640 1 -- 172.21.15.43:0/3053769452 learned_addr learned my addr 172.21.15.43:0/3053769452 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:02.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.094+0000 7f060e000640 1 --2- 172.21.15.43:0/3053769452 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0618102840 0x7f0618197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:02.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.094+0000 7f0616a00640 1 -- 172.21.15.43:0/3053769452 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0618103200 msgr2=0x7f061819eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:02.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.094+0000 7f0616a00640 1 --2- 172.21.15.43:0/3053769452 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0618103200 0x7f061819eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:02.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.094+0000 7f0616a00640 1 -- 172.21.15.43:0/3053769452 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0618102840 msgr2=0x7f0618197e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:02.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.094+0000 7f0616a00640 1 --2- 172.21.15.43:0/3053769452 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0618102840 0x7f0618197e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:02.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.094+0000 7f0616a00640 1 -- 172.21.15.43:0/3053769452 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f061819f510 con 0x7f0618101650 2024-03-28T22:36:02.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.094+0000 7f060e000640 1 --2- 172.21.15.43:0/3053769452 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0618102840 0x7f0618197e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:36:02.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.094+0000 7f0616a00640 1 --2- 172.21.15.43:0/3053769452 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0618101650 0x7f06181978f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f060800ebc0 tx=0x7f060800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:02.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.095+0000 7f060fe00640 1 -- 172.21.15.43:0/3053769452 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0608003b70 con 0x7f0618101650 2024-03-28T22:36:02.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.095+0000 7f060fe00640 1 -- 172.21.15.43:0/3053769452 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0608003d10 con 0x7f0618101650 2024-03-28T22:36:02.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.095+0000 7f060fe00640 1 -- 172.21.15.43:0/3053769452 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0608019980 con 0x7f0618101650 2024-03-28T22:36:02.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.095+0000 7f0617e00640 1 -- 172.21.15.43:0/3053769452 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f061819f800 con 0x7f0618101650 2024-03-28T22:36:02.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.095+0000 7f0617e00640 1 -- 172.21.15.43:0/3053769452 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f061819fd00 con 0x7f0618101650 2024-03-28T22:36:02.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.096+0000 7f0617e00640 1 -- 172.21.15.43:0/3053769452 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0618108970 con 0x7f0618101650 2024-03-28T22:36:02.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.098+0000 7f060fe00640 1 -- 172.21.15.43:0/3053769452 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f0608004b60 con 0x7f0618101650 2024-03-28T22:36:02.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.099+0000 7f060fe00640 1 --2- 172.21.15.43:0/3053769452 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f05fc0a1d80 0x7f05fc0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:02.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.099+0000 7f060fe00640 1 -- 172.21.15.43:0/3053769452 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f06080cea40 con 0x7f0618101650 2024-03-28T22:36:02.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.100+0000 7f060e000640 1 --2- 172.21.15.43:0/3053769452 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f05fc0a1d80 0x7f05fc0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:02.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.100+0000 7f060e000640 1 --2- 172.21.15.43:0/3053769452 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f05fc0a1d80 0x7f05fc0a4240 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0618198e00 tx=0x7f0600009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:02.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.106+0000 7f060fe00640 1 -- 172.21.15.43:0/3053769452 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f060809ba80 con 0x7f0618101650 2024-03-28T22:36:02.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.309+0000 7f0617e00640 1 -- 172.21.15.43:0/3053769452 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0618062350 con 0x7f0618101650 2024-03-28T22:36:02.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.310+0000 7f060fe00640 1 -- 172.21.15.43:0/3053769452 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) ==== 93+0+10 (secure 0 0 0) 0x7f060809b380 con 0x7f0618101650 2024-03-28T22:36:02.308 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-03-28T22:36:02.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.313+0000 7f0617e00640 1 -- 172.21.15.43:0/3053769452 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f05fc0a1d80 msgr2=0x7f05fc0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:02.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.313+0000 7f0617e00640 1 --2- 172.21.15.43:0/3053769452 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f05fc0a1d80 0x7f05fc0a4240 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0618198e00 tx=0x7f0600009260 comp rx=0 tx=0).stop 2024-03-28T22:36:02.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.313+0000 7f0617e00640 1 -- 172.21.15.43:0/3053769452 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0618101650 msgr2=0x7f06181978f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:02.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.313+0000 7f0617e00640 1 --2- 172.21.15.43:0/3053769452 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0618101650 0x7f06181978f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f060800ebc0 tx=0x7f060800c6a0 comp rx=0 tx=0).stop 2024-03-28T22:36:02.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.313+0000 7f0617e00640 1 -- 172.21.15.43:0/3053769452 shutdown_connections 2024-03-28T22:36:02.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.313+0000 7f0617e00640 1 --2- 172.21.15.43:0/3053769452 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0618103200 0x7f061819eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:02.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.313+0000 7f0617e00640 1 --2- 172.21.15.43:0/3053769452 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0618102840 0x7f0618197e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:02.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.313+0000 7f0617e00640 1 --2- 172.21.15.43:0/3053769452 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f05fc0a1d80 0x7f05fc0a4240 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:02.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.313+0000 7f0617e00640 1 --2- 172.21.15.43:0/3053769452 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0618101650 0x7f06181978f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:02.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.313+0000 7f0617e00640 1 -- 172.21.15.43:0/3053769452 >> 172.21.15.43:0/3053769452 conn(0x7f06180fcbc0 msgr2=0x7f0618105b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:02.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.313+0000 7f0617e00640 1 -- 172.21.15.43:0/3053769452 shutdown_connections 2024-03-28T22:36:02.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.313+0000 7f0617e00640 1 -- 172.21.15.43:0/3053769452 wait complete. 2024-03-28T22:36:02.333 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T22:36:02.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.564+0000 7f5f07e00640 1 -- 172.21.15.43:0/449372300 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5f08108e60 msgr2=0x7f5f081092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:02.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.564+0000 7f5f07e00640 1 --2- 172.21.15.43:0/449372300 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5f08108e60 0x7f5f081092e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0009900 tx=0x7f5ef002bc30 comp rx=0 tx=0).stop 2024-03-28T22:36:02.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.565+0000 7f5f07e00640 1 -- 172.21.15.43:0/449372300 shutdown_connections 2024-03-28T22:36:02.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.565+0000 7f5f07e00640 1 --2- 172.21.15.43:0/449372300 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5f08108e60 0x7f5f081092e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:02.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.565+0000 7f5f07e00640 1 --2- 172.21.15.43:0/449372300 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5f08109820 0x7f5f081100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:02.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.565+0000 7f5f07e00640 1 --2- 172.21.15.43:0/449372300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f08107c70 0x7f5f08108090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:02.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.565+0000 7f5f07e00640 1 -- 172.21.15.43:0/449372300 >> 172.21.15.43:0/449372300 conn(0x7f5f08076c20 msgr2=0x7f5f08079040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:02.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.565+0000 7f5f07e00640 1 -- 172.21.15.43:0/449372300 shutdown_connections 2024-03-28T22:36:02.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.565+0000 7f5f07e00640 1 -- 172.21.15.43:0/449372300 wait complete. 2024-03-28T22:36:02.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.566+0000 7f5f07e00640 1 Processor -- start 2024-03-28T22:36:02.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.566+0000 7f5f07e00640 1 -- start start 2024-03-28T22:36:02.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.567+0000 7f5f07e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f08107c70 0x7f5f0819df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:02.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.567+0000 7f5f07e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5f08108e60 0x7f5f0819e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:02.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.567+0000 7f5f07e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5f08109820 0x7f5f081a54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:02.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.567+0000 7f5f07e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5f08113920 con 0x7f5f08108e60 2024-03-28T22:36:02.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.567+0000 7f5f07e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5f081137a0 con 0x7f5f08109820 2024-03-28T22:36:02.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.567+0000 7f5f07e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f5f08113aa0 con 0x7f5f08107c70 2024-03-28T22:36:02.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.567+0000 7f5f06a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f08107c70 0x7f5f0819df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:02.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.567+0000 7f5efe000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5f08108e60 0x7f5f0819e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:02.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.567+0000 7f5f06a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f08107c70 0x7f5f0819df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35976/0 (socket says 172.21.15.43:35976) 2024-03-28T22:36:02.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.567+0000 7f5f06a00640 1 -- 172.21.15.43:0/4184476306 learned_addr learned my addr 172.21.15.43:0/4184476306 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:02.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.567+0000 7f5f07400640 1 --2- 172.21.15.43:0/4184476306 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5f08109820 0x7f5f081a54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:02.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.567+0000 7f5f06a00640 1 -- 172.21.15.43:0/4184476306 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5f08109820 msgr2=0x7f5f081a54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:02.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.567+0000 7f5f06a00640 1 --2- 172.21.15.43:0/4184476306 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5f08109820 0x7f5f081a54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:02.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.567+0000 7f5f06a00640 1 -- 172.21.15.43:0/4184476306 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5f08108e60 msgr2=0x7f5f0819e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:02.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.567+0000 7f5f06a00640 1 --2- 172.21.15.43:0/4184476306 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5f08108e60 0x7f5f0819e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:02.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.567+0000 7f5f06a00640 1 -- 172.21.15.43:0/4184476306 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f081a5b20 con 0x7f5f08107c70 2024-03-28T22:36:02.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.568+0000 7f5f06a00640 1 --2- 172.21.15.43:0/4184476306 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f08107c70 0x7f5f0819df00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5ef800e9c0 tx=0x7f5ef800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:02.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.568+0000 7f5effe00640 1 -- 172.21.15.43:0/4184476306 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef800cd80 con 0x7f5f08107c70 2024-03-28T22:36:02.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.568+0000 7f5effe00640 1 -- 172.21.15.43:0/4184476306 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ef80035f0 con 0x7f5f08107c70 2024-03-28T22:36:02.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.568+0000 7f5f07e00640 1 -- 172.21.15.43:0/4184476306 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f081a5e10 con 0x7f5f08107c70 2024-03-28T22:36:02.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.568+0000 7f5effe00640 1 -- 172.21.15.43:0/4184476306 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef8010690 con 0x7f5f08107c70 2024-03-28T22:36:02.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.569+0000 7f5f07e00640 1 -- 172.21.15.43:0/4184476306 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f081a6270 con 0x7f5f08107c70 2024-03-28T22:36:02.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.570+0000 7f5f07e00640 1 -- 172.21.15.43:0/4184476306 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f0810ef90 con 0x7f5f08107c70 2024-03-28T22:36:02.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.571+0000 7f5effe00640 1 -- 172.21.15.43:0/4184476306 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f5ef80037a0 con 0x7f5f08107c70 2024-03-28T22:36:02.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.573+0000 7f5effe00640 1 --2- 172.21.15.43:0/4184476306 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5eec0a1d80 0x7f5eec0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:02.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.573+0000 7f5effe00640 1 -- 172.21.15.43:0/4184476306 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f5ef80ce4c0 con 0x7f5f08107c70 2024-03-28T22:36:02.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.573+0000 7f5efe000640 1 --2- 172.21.15.43:0/4184476306 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5eec0a1d80 0x7f5eec0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:02.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.574+0000 7f5efe000640 1 --2- 172.21.15.43:0/4184476306 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5eec0a1d80 0x7f5eec0a4240 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5ef002c240 tx=0x7f5ef002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:02.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.581+0000 7f5effe00640 1 -- 172.21.15.43:0/4184476306 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5ef809b560 con 0x7f5f08107c70 2024-03-28T22:36:02.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:02 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3422657485' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:36:02.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:02 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3053769452' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:36:02.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:02 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3422657485' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:36:02.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:02 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3053769452' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:36:02.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.785+0000 7f5f07e00640 1 -- 172.21.15.43:0/4184476306 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5f0802d170 con 0x7f5f08107c70 2024-03-28T22:36:02.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.786+0000 7f5effe00640 1 -- 172.21.15.43:0/4184476306 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) ==== 104+0+11 (secure 0 0 0) 0x7f5f0802d170 con 0x7f5f08107c70 2024-03-28T22:36:02.784 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-03-28T22:36:02.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.789+0000 7f5f07e00640 1 -- 172.21.15.43:0/4184476306 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5eec0a1d80 msgr2=0x7f5eec0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:02.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.789+0000 7f5f07e00640 1 --2- 172.21.15.43:0/4184476306 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5eec0a1d80 0x7f5eec0a4240 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5ef002c240 tx=0x7f5ef002df90 comp rx=0 tx=0).stop 2024-03-28T22:36:02.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.789+0000 7f5f07e00640 1 -- 172.21.15.43:0/4184476306 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f08107c70 msgr2=0x7f5f0819df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:02.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.789+0000 7f5f07e00640 1 --2- 172.21.15.43:0/4184476306 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f08107c70 0x7f5f0819df00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5ef800e9c0 tx=0x7f5ef800ee80 comp rx=0 tx=0).stop 2024-03-28T22:36:02.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.789+0000 7f5f07e00640 1 -- 172.21.15.43:0/4184476306 shutdown_connections 2024-03-28T22:36:02.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.789+0000 7f5f07e00640 1 --2- 172.21.15.43:0/4184476306 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5eec0a1d80 0x7f5eec0a4240 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:02.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.789+0000 7f5f07e00640 1 --2- 172.21.15.43:0/4184476306 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5f08109820 0x7f5f081a54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:02.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.789+0000 7f5f07e00640 1 --2- 172.21.15.43:0/4184476306 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5f08108e60 0x7f5f0819e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:02.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.789+0000 7f5f07e00640 1 --2- 172.21.15.43:0/4184476306 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5f08107c70 0x7f5f0819df00 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:02.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.789+0000 7f5f07e00640 1 -- 172.21.15.43:0/4184476306 >> 172.21.15.43:0/4184476306 conn(0x7f5f08076c20 msgr2=0x7f5f0810c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:02.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.789+0000 7f5f07e00640 1 -- 172.21.15.43:0/4184476306 shutdown_connections 2024-03-28T22:36:02.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:02.789+0000 7f5f07e00640 1 -- 172.21.15.43:0/4184476306 wait complete. 2024-03-28T22:36:02.806 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T22:36:02.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:02 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3422657485' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:36:02.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:02 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3053769452' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:36:03.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.040+0000 7f0a0be00640 1 -- 172.21.15.43:0/1634762690 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a0c104a60 msgr2=0x7f0a0c104ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:03.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.040+0000 7f0a0be00640 1 --2- 172.21.15.43:0/1634762690 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a0c104a60 0x7f0a0c104ee0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f09f8009900 tx=0x7f09f802bc30 comp rx=0 tx=0).stop 2024-03-28T22:36:03.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.040+0000 7f0a0be00640 1 -- 172.21.15.43:0/1634762690 shutdown_connections 2024-03-28T22:36:03.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.040+0000 7f0a0be00640 1 --2- 172.21.15.43:0/1634762690 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a0c104a60 0x7f0a0c104ee0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:03.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.040+0000 7f0a0be00640 1 --2- 172.21.15.43:0/1634762690 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a0c105420 0x7f0a0c10bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:03.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.040+0000 7f0a0be00640 1 --2- 172.21.15.43:0/1634762690 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0a0c103870 0x7f0a0c103c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:03.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.040+0000 7f0a0be00640 1 -- 172.21.15.43:0/1634762690 >> 172.21.15.43:0/1634762690 conn(0x7f0a0c0fee00 msgr2=0x7f0a0c101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:03.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.040+0000 7f0a0be00640 1 -- 172.21.15.43:0/1634762690 shutdown_connections 2024-03-28T22:36:03.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.040+0000 7f0a0be00640 1 -- 172.21.15.43:0/1634762690 wait complete. 2024-03-28T22:36:03.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.041+0000 7f0a0be00640 1 Processor -- start 2024-03-28T22:36:03.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.042+0000 7f0a0be00640 1 -- start start 2024-03-28T22:36:03.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.042+0000 7f0a0be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0a0c103870 0x7f0a0c199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:03.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.042+0000 7f0a0be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a0c104a60 0x7f0a0c19a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:03.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.042+0000 7f0a0be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a0c105420 0x7f0a0c1a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:03.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.042+0000 7f0a0be00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0c10f540 con 0x7f0a0c103870 2024-03-28T22:36:03.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.042+0000 7f0a0be00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0c10f3c0 con 0x7f0a0c105420 2024-03-28T22:36:03.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.042+0000 7f0a0be00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0c10f6c0 con 0x7f0a0c104a60 2024-03-28T22:36:03.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.042+0000 7f0a0b400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a0c105420 0x7f0a0c1a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:03.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.042+0000 7f0a0a000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a0c104a60 0x7f0a0c19a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:03.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.042+0000 7f0a0b400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a0c105420 0x7f0a0c1a1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48086/0 (socket says 172.21.15.43:48086) 2024-03-28T22:36:03.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.042+0000 7f0a0b400640 1 -- 172.21.15.43:0/3390143484 learned_addr learned my addr 172.21.15.43:0/3390143484 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:03.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.042+0000 7f0a0a000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a0c104a60 0x7f0a0c19a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35994/0 (socket says 172.21.15.43:35994) 2024-03-28T22:36:03.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.042+0000 7f0a0aa00640 1 --2- 172.21.15.43:0/3390143484 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0a0c103870 0x7f0a0c199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:03.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.043+0000 7f0a0b400640 1 -- 172.21.15.43:0/3390143484 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a0c104a60 msgr2=0x7f0a0c19a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:03.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.043+0000 7f0a0b400640 1 --2- 172.21.15.43:0/3390143484 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a0c104a60 0x7f0a0c19a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:03.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.043+0000 7f0a0b400640 1 -- 172.21.15.43:0/3390143484 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0a0c103870 msgr2=0x7f0a0c199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:03.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.043+0000 7f0a0b400640 1 --2- 172.21.15.43:0/3390143484 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0a0c103870 0x7f0a0c199ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:03.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.043+0000 7f0a0b400640 1 -- 172.21.15.43:0/3390143484 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a0c1a17c0 con 0x7f0a0c105420 2024-03-28T22:36:03.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.043+0000 7f0a0b400640 1 --2- 172.21.15.43:0/3390143484 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a0c105420 0x7f0a0c1a1150 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0a0400e970 tx=0x7f0a0400ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:03.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.043+0000 7f0a03400640 1 -- 172.21.15.43:0/3390143484 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a0400ccc0 con 0x7f0a0c105420 2024-03-28T22:36:03.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.043+0000 7f0a03400640 1 -- 172.21.15.43:0/3390143484 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a0400ce60 con 0x7f0a0c105420 2024-03-28T22:36:03.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.043+0000 7f0a0be00640 1 -- 172.21.15.43:0/3390143484 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a0c1a1ab0 con 0x7f0a0c105420 2024-03-28T22:36:03.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.043+0000 7f0a03400640 1 -- 172.21.15.43:0/3390143484 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a04010490 con 0x7f0a0c105420 2024-03-28T22:36:03.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.044+0000 7f0a0be00640 1 -- 172.21.15.43:0/3390143484 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a0c1a2020 con 0x7f0a0c105420 2024-03-28T22:36:03.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.045+0000 7f0a0be00640 1 -- 172.21.15.43:0/3390143484 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a0c10ab30 con 0x7f0a0c105420 2024-03-28T22:36:03.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.047+0000 7f0a03400640 1 -- 172.21.15.43:0/3390143484 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f0a04005020 con 0x7f0a0c105420 2024-03-28T22:36:03.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.048+0000 7f0a03400640 1 --2- 172.21.15.43:0/3390143484 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f09f00a1e00 0x7f09f00a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:03.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.048+0000 7f0a03400640 1 -- 172.21.15.43:0/3390143484 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f0a040ce510 con 0x7f0a0c105420 2024-03-28T22:36:03.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.048+0000 7f0a0aa00640 1 --2- 172.21.15.43:0/3390143484 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f09f00a1e00 0x7f09f00a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:03.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.049+0000 7f0a0aa00640 1 --2- 172.21.15.43:0/3390143484 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f09f00a1e00 0x7f09f00a42c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c1048c0 tx=0x7f09fc009670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:03.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.052+0000 7f0a03400640 1 -- 172.21.15.43:0/3390143484 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0a0409b550 con 0x7f0a0c105420 2024-03-28T22:36:03.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.259+0000 7f0a0be00640 1 -- 172.21.15.43:0/3390143484 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0a0c062350 con 0x7f0a0c105420 2024-03-28T22:36:03.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.260+0000 7f0a03400640 1 -- 172.21.15.43:0/3390143484 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) ==== 100+0+11 (secure 0 0 0) 0x7f0a0409ae50 con 0x7f0a0c105420 2024-03-28T22:36:03.258 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-03-28T22:36:03.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.263+0000 7f0a0be00640 1 -- 172.21.15.43:0/3390143484 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f09f00a1e00 msgr2=0x7f09f00a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:03.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.263+0000 7f0a0be00640 1 --2- 172.21.15.43:0/3390143484 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f09f00a1e00 0x7f09f00a42c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c1048c0 tx=0x7f09fc009670 comp rx=0 tx=0).stop 2024-03-28T22:36:03.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.264+0000 7f0a0be00640 1 -- 172.21.15.43:0/3390143484 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a0c105420 msgr2=0x7f0a0c1a1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:03.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.264+0000 7f0a0be00640 1 --2- 172.21.15.43:0/3390143484 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a0c105420 0x7f0a0c1a1150 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0a0400e970 tx=0x7f0a0400ed70 comp rx=0 tx=0).stop 2024-03-28T22:36:03.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.264+0000 7f0a0be00640 1 -- 172.21.15.43:0/3390143484 shutdown_connections 2024-03-28T22:36:03.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.264+0000 7f0a0be00640 1 --2- 172.21.15.43:0/3390143484 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f09f00a1e00 0x7f09f00a42c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:03.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.264+0000 7f0a0be00640 1 --2- 172.21.15.43:0/3390143484 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a0c105420 0x7f0a0c1a1150 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:03.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.264+0000 7f0a0be00640 1 --2- 172.21.15.43:0/3390143484 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a0c104a60 0x7f0a0c19a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:03.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.264+0000 7f0a0be00640 1 --2- 172.21.15.43:0/3390143484 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0a0c103870 0x7f0a0c199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:03.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.264+0000 7f0a0be00640 1 -- 172.21.15.43:0/3390143484 >> 172.21.15.43:0/3390143484 conn(0x7f0a0c0fee00 msgr2=0x7f0a0c107d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:03.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.264+0000 7f0a0be00640 1 -- 172.21.15.43:0/3390143484 shutdown_connections 2024-03-28T22:36:03.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.264+0000 7f0a0be00640 1 -- 172.21.15.43:0/3390143484 wait complete. 2024-03-28T22:36:03.283 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T22:36:03.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.519+0000 7f5458c00640 1 -- 172.21.15.43:0/907019782 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5454101650 msgr2=0x7f5454101a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:03.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.519+0000 7f5458c00640 1 --2- 172.21.15.43:0/907019782 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5454101650 0x7f5454101a70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5444009900 tx=0x7f544402bc50 comp rx=0 tx=0).stop 2024-03-28T22:36:03.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.519+0000 7f5458c00640 1 -- 172.21.15.43:0/907019782 shutdown_connections 2024-03-28T22:36:03.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.519+0000 7f5458c00640 1 --2- 172.21.15.43:0/907019782 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5454103200 0x7f5454109a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:03.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.519+0000 7f5458c00640 1 --2- 172.21.15.43:0/907019782 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5454102840 0x7f5454102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:03.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.519+0000 7f5458c00640 1 --2- 172.21.15.43:0/907019782 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5454101650 0x7f5454101a70 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:03.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.519+0000 7f5458c00640 1 -- 172.21.15.43:0/907019782 >> 172.21.15.43:0/907019782 conn(0x7f54540fcbc0 msgr2=0x7f54540ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:03.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.519+0000 7f5458c00640 1 -- 172.21.15.43:0/907019782 shutdown_connections 2024-03-28T22:36:03.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.519+0000 7f5458c00640 1 -- 172.21.15.43:0/907019782 wait complete. 2024-03-28T22:36:03.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.521+0000 7f5458c00640 1 Processor -- start 2024-03-28T22:36:03.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.521+0000 7f5458c00640 1 -- start start 2024-03-28T22:36:03.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.521+0000 7f5458c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5454101650 0x7f5454199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:03.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.521+0000 7f5458c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5454102840 0x7f545419a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:03.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.521+0000 7f5458c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5454103200 0x7f54541a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:03.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.521+0000 7f5458c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f545410d320 con 0x7f5454103200 2024-03-28T22:36:03.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.521+0000 7f5458c00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f545410d1a0 con 0x7f5454101650 2024-03-28T22:36:03.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.521+0000 7f5458c00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f545410d4a0 con 0x7f5454102840 2024-03-28T22:36:03.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.521+0000 7f5453400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5454101650 0x7f5454199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:03.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.521+0000 7f5453400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5454101650 0x7f5454199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48108/0 (socket says 172.21.15.43:48108) 2024-03-28T22:36:03.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.521+0000 7f5453400640 1 -- 172.21.15.43:0/2274884368 learned_addr learned my addr 172.21.15.43:0/2274884368 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:03.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.521+0000 7f544aa00640 1 --2- 172.21.15.43:0/2274884368 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5454102840 0x7f545419a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:03.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.521+0000 7f5453e00640 1 --2- 172.21.15.43:0/2274884368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5454103200 0x7f54541a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:03.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.522+0000 7f5453400640 1 -- 172.21.15.43:0/2274884368 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5454102840 msgr2=0x7f545419a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:03.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.522+0000 7f5453400640 1 --2- 172.21.15.43:0/2274884368 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5454102840 0x7f545419a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:03.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.522+0000 7f5453400640 1 -- 172.21.15.43:0/2274884368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5454103200 msgr2=0x7f54541a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:03.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.522+0000 7f5453400640 1 --2- 172.21.15.43:0/2274884368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5454103200 0x7f54541a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:03.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.522+0000 7f5453400640 1 -- 172.21.15.43:0/2274884368 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54541a1830 con 0x7f5454101650 2024-03-28T22:36:03.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.522+0000 7f5453e00640 1 --2- 172.21.15.43:0/2274884368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5454103200 0x7f54541a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:36:03.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.522+0000 7f5453400640 1 --2- 172.21.15.43:0/2274884368 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5454101650 0x7f5454199c20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5444005ab0 tx=0x7f544402c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:03.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.522+0000 7f5450c00640 1 -- 172.21.15.43:0/2274884368 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5444041070 con 0x7f5454101650 2024-03-28T22:36:03.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.522+0000 7f5450c00640 1 -- 172.21.15.43:0/2274884368 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5444004450 con 0x7f5454101650 2024-03-28T22:36:03.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.522+0000 7f5450c00640 1 -- 172.21.15.43:0/2274884368 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5444034440 con 0x7f5454101650 2024-03-28T22:36:03.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.523+0000 7f5458c00640 1 -- 172.21.15.43:0/2274884368 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54541a1ac0 con 0x7f5454101650 2024-03-28T22:36:03.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.523+0000 7f5458c00640 1 -- 172.21.15.43:0/2274884368 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54541a1fc0 con 0x7f5454101650 2024-03-28T22:36:03.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.524+0000 7f5458c00640 1 -- 172.21.15.43:0/2274884368 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54541a21f0 con 0x7f5454101650 2024-03-28T22:36:03.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.526+0000 7f5450c00640 1 -- 172.21.15.43:0/2274884368 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f5444044070 con 0x7f5454101650 2024-03-28T22:36:03.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.527+0000 7f5450c00640 1 --2- 172.21.15.43:0/2274884368 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f54380a1e00 0x7f54380a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:03.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.527+0000 7f5450c00640 1 -- 172.21.15.43:0/2274884368 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f54440efdd0 con 0x7f5454101650 2024-03-28T22:36:03.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.527+0000 7f544aa00640 1 --2- 172.21.15.43:0/2274884368 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f54380a1e00 0x7f54380a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:03.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.528+0000 7f544aa00640 1 --2- 172.21.15.43:0/2274884368 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f54380a1e00 0x7f54380a42c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f543c009bd0 tx=0x7f543c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:03.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.531+0000 7f5450c00640 1 -- 172.21.15.43:0/2274884368 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f54440bce10 con 0x7f5454101650 2024-03-28T22:36:03.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:03 smithi082 ceph-mon[36471]: pgmap v403: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:36:03.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:03 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/4184476306' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:36:03.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:03 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3390143484' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:36:03.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.733+0000 7f5458c00640 1 -- 172.21.15.43:0/2274884368 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f54541a23c0 con 0x7f5454101650 2024-03-28T22:36:03.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.733+0000 7f5450c00640 1 -- 172.21.15.43:0/2274884368 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v96) ==== 103+0+11 (secure 0 0 0) 0x7f54440bc710 con 0x7f5454101650 2024-03-28T22:36:03.732 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-03-28T22:36:03.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.736+0000 7f5458c00640 1 -- 172.21.15.43:0/2274884368 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f54380a1e00 msgr2=0x7f54380a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:03.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.736+0000 7f5458c00640 1 --2- 172.21.15.43:0/2274884368 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f54380a1e00 0x7f54380a42c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f543c009bd0 tx=0x7f543c005f90 comp rx=0 tx=0).stop 2024-03-28T22:36:03.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.736+0000 7f5458c00640 1 -- 172.21.15.43:0/2274884368 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5454101650 msgr2=0x7f5454199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:03.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.736+0000 7f5458c00640 1 --2- 172.21.15.43:0/2274884368 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5454101650 0x7f5454199c20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5444005ab0 tx=0x7f544402c6e0 comp rx=0 tx=0).stop 2024-03-28T22:36:03.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.737+0000 7f5458c00640 1 -- 172.21.15.43:0/2274884368 shutdown_connections 2024-03-28T22:36:03.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.737+0000 7f5458c00640 1 --2- 172.21.15.43:0/2274884368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5454103200 0x7f54541a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:03.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.737+0000 7f5458c00640 1 --2- 172.21.15.43:0/2274884368 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5454102840 0x7f545419a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:03.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.737+0000 7f5458c00640 1 --2- 172.21.15.43:0/2274884368 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f54380a1e00 0x7f54380a42c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:03.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.737+0000 7f5458c00640 1 --2- 172.21.15.43:0/2274884368 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5454101650 0x7f5454199c20 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:03.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.737+0000 7f5458c00640 1 -- 172.21.15.43:0/2274884368 >> 172.21.15.43:0/2274884368 conn(0x7f54540fcbc0 msgr2=0x7f5454073980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:03.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.737+0000 7f5458c00640 1 -- 172.21.15.43:0/2274884368 shutdown_connections 2024-03-28T22:36:03.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.737+0000 7f5458c00640 1 -- 172.21.15.43:0/2274884368 wait complete. 2024-03-28T22:36:03.752 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:03 smithi043 ceph-mon[31823]: pgmap v403: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:36:03.752 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:03 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/4184476306' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:36:03.753 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:03 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3390143484' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:36:03.757 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:36:03.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:03 smithi172 ceph-mon[36566]: pgmap v403: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:36:03.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:03 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/4184476306' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:36:03.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:03 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3390143484' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:36:03.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.984+0000 7f89ca000640 1 -- 172.21.15.43:0/3629696497 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89c4103990 msgr2=0x7f89c4103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:03.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.984+0000 7f89ca000640 1 --2- 172.21.15.43:0/3629696497 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89c4103990 0x7f89c4103db0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f89ac009900 tx=0x7f89ac02bc40 comp rx=0 tx=0).stop 2024-03-28T22:36:03.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.984+0000 7f89ca000640 1 -- 172.21.15.43:0/3629696497 shutdown_connections 2024-03-28T22:36:03.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.984+0000 7f89ca000640 1 --2- 172.21.15.43:0/3629696497 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89c4105540 0x7f89c410bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:03.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.984+0000 7f89ca000640 1 --2- 172.21.15.43:0/3629696497 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f89c4104b80 0x7f89c4105000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:03.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.984+0000 7f89ca000640 1 --2- 172.21.15.43:0/3629696497 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89c4103990 0x7f89c4103db0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:03.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.984+0000 7f89ca000640 1 -- 172.21.15.43:0/3629696497 >> 172.21.15.43:0/3629696497 conn(0x7f89c40fef00 msgr2=0x7f89c4101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:03.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.984+0000 7f89ca000640 1 -- 172.21.15.43:0/3629696497 shutdown_connections 2024-03-28T22:36:03.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.984+0000 7f89ca000640 1 -- 172.21.15.43:0/3629696497 wait complete. 2024-03-28T22:36:03.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.985+0000 7f89ca000640 1 Processor -- start 2024-03-28T22:36:03.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.986+0000 7f89ca000640 1 -- start start 2024-03-28T22:36:03.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.986+0000 7f89ca000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f89c4103990 0x7f89c4199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:03.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.986+0000 7f89ca000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89c4104b80 0x7f89c419a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:03.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.986+0000 7f89ca000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89c4105540 0x7f89c41a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:03.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.986+0000 7f89ca000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f89c410f630 con 0x7f89c4103990 2024-03-28T22:36:03.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.986+0000 7f89ca000640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f89c410f4b0 con 0x7f89c4104b80 2024-03-28T22:36:03.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.986+0000 7f89ca000640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f89c410f7b0 con 0x7f89c4105540 2024-03-28T22:36:03.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.986+0000 7f89c3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f89c4103990 0x7f89c4199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:03.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.986+0000 7f89c3e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89c4105540 0x7f89c41a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:03.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.986+0000 7f89c2a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89c4104b80 0x7f89c419a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:03.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.986+0000 7f89c3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f89c4103990 0x7f89c4199d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:59968/0 (socket says 172.21.15.43:59968) 2024-03-28T22:36:03.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.986+0000 7f89c3400640 1 -- 172.21.15.43:0/332246955 learned_addr learned my addr 172.21.15.43:0/332246955 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:03.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.987+0000 7f89c3e00640 1 -- 172.21.15.43:0/332246955 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89c4104b80 msgr2=0x7f89c419a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:03.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.987+0000 7f89c3e00640 1 --2- 172.21.15.43:0/332246955 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89c4104b80 0x7f89c419a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:03.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.987+0000 7f89c3e00640 1 -- 172.21.15.43:0/332246955 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f89c4103990 msgr2=0x7f89c4199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:03.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.987+0000 7f89c3e00640 1 --2- 172.21.15.43:0/332246955 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f89c4103990 0x7f89c4199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:03.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.987+0000 7f89c3e00640 1 -- 172.21.15.43:0/332246955 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89c41a1920 con 0x7f89c4105540 2024-03-28T22:36:03.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.987+0000 7f89c3e00640 1 --2- 172.21.15.43:0/332246955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89c4105540 0x7f89c41a1280 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f89bc00ec50 tx=0x7f89bc00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:03.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.987+0000 7f89bbe00640 1 -- 172.21.15.43:0/332246955 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89bc003b70 con 0x7f89c4105540 2024-03-28T22:36:03.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.987+0000 7f89bbe00640 1 -- 172.21.15.43:0/332246955 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89bc003d10 con 0x7f89c4105540 2024-03-28T22:36:03.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.987+0000 7f89ca000640 1 -- 172.21.15.43:0/332246955 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89c41a1c10 con 0x7f89c4105540 2024-03-28T22:36:03.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.987+0000 7f89bbe00640 1 -- 172.21.15.43:0/332246955 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89bc010890 con 0x7f89c4105540 2024-03-28T22:36:03.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.988+0000 7f89ca000640 1 -- 172.21.15.43:0/332246955 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89c41a20e0 con 0x7f89c4105540 2024-03-28T22:36:03.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.989+0000 7f89ca000640 1 -- 172.21.15.43:0/332246955 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89c410acb0 con 0x7f89c4105540 2024-03-28T22:36:03.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.991+0000 7f89bbe00640 1 -- 172.21.15.43:0/332246955 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f89bc004b40 con 0x7f89c4105540 2024-03-28T22:36:03.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.992+0000 7f89bbe00640 1 --2- 172.21.15.43:0/332246955 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f89a80a1d80 0x7f89a80a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:03.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.992+0000 7f89bbe00640 1 -- 172.21.15.43:0/332246955 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f89bc0ce720 con 0x7f89c4105540 2024-03-28T22:36:03.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.992+0000 7f89c3400640 1 --2- 172.21.15.43:0/332246955 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f89a80a1d80 0x7f89a80a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:03.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.993+0000 7f89c3400640 1 --2- 172.21.15.43:0/332246955 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f89a80a1d80 0x7f89a80a4240 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f89ac005ab0 tx=0x7f89ac02cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:03.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:03.994+0000 7f89bbe00640 1 -- 172.21.15.43:0/332246955 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f89bc09b760 con 0x7f89c4105540 2024-03-28T22:36:04.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.240+0000 7f89ca000640 1 -- 172.21.15.43:0/332246955 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89c402d3c0 con 0x7f89c4105540 2024-03-28T22:36:04.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.241+0000 7f89bbe00640 1 -- 172.21.15.43:0/332246955 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11934 (secure 0 0 0) 0x7f89bc09b060 con 0x7f89c4105540 2024-03-28T22:36:04.239 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:36:04.240 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:33:28.906396+0000","modified":"2024-03-28T22:34:06.174761+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24572,"mds_1":24556,"mds_2":24535,"mds_3":24523,"mds_4":24518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/1084988072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1084988072},{"type":"v1","addr":"172.21.15.82:6835","nonce":1084988072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24535,24556,24518,24523,24572]},"id":1}]} 2024-03-28T22:36:04.240 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 28 2024-03-28T22:36:04.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.245+0000 7f89ca000640 1 -- 172.21.15.43:0/332246955 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f89a80a1d80 msgr2=0x7f89a80a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:04.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.245+0000 7f89ca000640 1 --2- 172.21.15.43:0/332246955 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f89a80a1d80 0x7f89a80a4240 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f89ac005ab0 tx=0x7f89ac02cf90 comp rx=0 tx=0).stop 2024-03-28T22:36:04.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.245+0000 7f89ca000640 1 -- 172.21.15.43:0/332246955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89c4105540 msgr2=0x7f89c41a1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:04.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.245+0000 7f89ca000640 1 --2- 172.21.15.43:0/332246955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89c4105540 0x7f89c41a1280 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f89bc00ec50 tx=0x7f89bc00c6a0 comp rx=0 tx=0).stop 2024-03-28T22:36:04.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.245+0000 7f89ca000640 1 -- 172.21.15.43:0/332246955 shutdown_connections 2024-03-28T22:36:04.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.245+0000 7f89ca000640 1 --2- 172.21.15.43:0/332246955 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f89a80a1d80 0x7f89a80a4240 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:04.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.245+0000 7f89ca000640 1 --2- 172.21.15.43:0/332246955 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89c4104b80 0x7f89c419a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:04.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.245+0000 7f89ca000640 1 --2- 172.21.15.43:0/332246955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89c4105540 0x7f89c41a1280 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:04.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.245+0000 7f89ca000640 1 --2- 172.21.15.43:0/332246955 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f89c4103990 0x7f89c4199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:04.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.245+0000 7f89ca000640 1 -- 172.21.15.43:0/332246955 >> 172.21.15.43:0/332246955 conn(0x7f89c40fef00 msgr2=0x7f89c4107e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:04.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.245+0000 7f89ca000640 1 -- 172.21.15.43:0/332246955 shutdown_connections 2024-03-28T22:36:04.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.245+0000 7f89ca000640 1 -- 172.21.15.43:0/332246955 wait complete. 2024-03-28T22:36:04.265 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:36:04.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.487+0000 7fbf7e000640 1 -- 172.21.15.43:0/557372684 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbf781037a0 msgr2=0x7fbf78103bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:04.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.487+0000 7fbf7e000640 1 --2- 172.21.15.43:0/557372684 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbf781037a0 0x7fbf78103bc0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c009900 tx=0x7fbf6c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:36:04.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.488+0000 7fbf7e000640 1 -- 172.21.15.43:0/557372684 shutdown_connections 2024-03-28T22:36:04.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.488+0000 7fbf7e000640 1 --2- 172.21.15.43:0/557372684 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf78105350 0x7fbf7810bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:04.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.488+0000 7fbf7e000640 1 --2- 172.21.15.43:0/557372684 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf78104990 0x7fbf78104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:04.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.488+0000 7fbf7e000640 1 --2- 172.21.15.43:0/557372684 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbf781037a0 0x7fbf78103bc0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:04.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.488+0000 7fbf7e000640 1 -- 172.21.15.43:0/557372684 >> 172.21.15.43:0/557372684 conn(0x7fbf780fed30 msgr2=0x7fbf78101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:04.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.488+0000 7fbf7e000640 1 -- 172.21.15.43:0/557372684 shutdown_connections 2024-03-28T22:36:04.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.488+0000 7fbf7e000640 1 -- 172.21.15.43:0/557372684 wait complete. 2024-03-28T22:36:04.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.489+0000 7fbf7e000640 1 Processor -- start 2024-03-28T22:36:04.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.489+0000 7fbf7e000640 1 -- start start 2024-03-28T22:36:04.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.489+0000 7fbf7e000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbf781037a0 0x7fbf78199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:04.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.489+0000 7fbf7e000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf78104990 0x7fbf7819a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:04.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.489+0000 7fbf7e000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf78105350 0x7fbf781a1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:04.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.489+0000 7fbf7e000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbf7810f440 con 0x7fbf78104990 2024-03-28T22:36:04.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.489+0000 7fbf7e000640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fbf7810f2c0 con 0x7fbf781037a0 2024-03-28T22:36:04.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.489+0000 7fbf7e000640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fbf7810f5c0 con 0x7fbf78105350 2024-03-28T22:36:04.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.489+0000 7fbf7cc00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbf781037a0 0x7fbf78199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:04.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.489+0000 7fbf7cc00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbf781037a0 0x7fbf78199ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48146/0 (socket says 172.21.15.43:48146) 2024-03-28T22:36:04.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.490+0000 7fbf7cc00640 1 -- 172.21.15.43:0/1548174905 learned_addr learned my addr 172.21.15.43:0/1548174905 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:04.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.490+0000 7fbf7d600640 1 --2- 172.21.15.43:0/1548174905 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf78105350 0x7fbf781a1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:04.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.490+0000 7fbf77e00640 1 --2- 172.21.15.43:0/1548174905 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf78104990 0x7fbf7819a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:04.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.490+0000 7fbf7cc00640 1 -- 172.21.15.43:0/1548174905 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf78105350 msgr2=0x7fbf781a1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:04.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.490+0000 7fbf7cc00640 1 --2- 172.21.15.43:0/1548174905 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf78105350 0x7fbf781a1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:04.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.490+0000 7fbf7cc00640 1 -- 172.21.15.43:0/1548174905 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf78104990 msgr2=0x7fbf7819a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:04.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.490+0000 7fbf7cc00640 1 --2- 172.21.15.43:0/1548174905 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf78104990 0x7fbf7819a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:04.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.490+0000 7fbf7cc00640 1 -- 172.21.15.43:0/1548174905 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf781a1700 con 0x7fbf781037a0 2024-03-28T22:36:04.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.490+0000 7fbf7d600640 1 --2- 172.21.15.43:0/1548174905 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf78105350 0x7fbf781a1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:04.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.490+0000 7fbf7cc00640 1 --2- 172.21.15.43:0/1548174905 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbf781037a0 0x7fbf78199ae0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c000c00 tx=0x7fbf6c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:04.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.490+0000 7fbf75600640 1 -- 172.21.15.43:0/1548174905 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c043020 con 0x7fbf781037a0 2024-03-28T22:36:04.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.491+0000 7fbf75600640 1 -- 172.21.15.43:0/1548174905 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf6c0054e0 con 0x7fbf781037a0 2024-03-28T22:36:04.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.491+0000 7fbf7e000640 1 -- 172.21.15.43:0/1548174905 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf781a1990 con 0x7fbf781037a0 2024-03-28T22:36:04.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.491+0000 7fbf75600640 1 -- 172.21.15.43:0/1548174905 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c003bd0 con 0x7fbf781037a0 2024-03-28T22:36:04.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.491+0000 7fbf7e000640 1 -- 172.21.15.43:0/1548174905 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf781a1f30 con 0x7fbf781037a0 2024-03-28T22:36:04.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.492+0000 7fbf7e000640 1 -- 172.21.15.43:0/1548174905 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf7810ab40 con 0x7fbf781037a0 2024-03-28T22:36:04.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.494+0000 7fbf75600640 1 -- 172.21.15.43:0/1548174905 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fbf6c046020 con 0x7fbf781037a0 2024-03-28T22:36:04.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.495+0000 7fbf75600640 1 --2- 172.21.15.43:0/1548174905 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fbf600a1e00 0x7fbf600a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:04.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.495+0000 7fbf75600640 1 -- 172.21.15.43:0/1548174905 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7fbf6c0f0100 con 0x7fbf781037a0 2024-03-28T22:36:04.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.495+0000 7fbf77e00640 1 --2- 172.21.15.43:0/1548174905 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fbf600a1e00 0x7fbf600a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:04.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.496+0000 7fbf77e00640 1 --2- 172.21.15.43:0/1548174905 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fbf600a1e00 0x7fbf600a42c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fbf680095f0 tx=0x7fbf68005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:04.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.499+0000 7fbf75600640 1 -- 172.21.15.43:0/1548174905 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbf6c04d020 con 0x7fbf781037a0 2024-03-28T22:36:04.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:04 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2274884368' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:36:04.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:04 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/332246955' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:36:04.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:04 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2274884368' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:36:04.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:04 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/332246955' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:36:04.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.705+0000 7fbf7e000640 1 -- 172.21.15.43:0/1548174905 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbf78062350 con 0x7fbf781037a0 2024-03-28T22:36:04.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.707+0000 7fbf75600640 1 -- 172.21.15.43:0/1548174905 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+24712 (secure 0 0 0) 0x7fbf6c0dc030 con 0x7fbf781037a0 2024-03-28T22:36:04.705 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:36:04.706 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":96,"fsid":"c2276428-ed51-11ee-95d0-87774f69a715","created":"2024-03-28T22:24:19.950084+0000","modified":"2024-03-28T22:35:30.237654+0000","last_up_change":"2024-03-28T22:32:31.078737+0000","last_in_change":"2024-03-28T22:32:10.147097+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:28:14.713290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:33:27.167435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:33:28.411825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T22:33:32.523122+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b855adcd-786c-4e42-b8fe-a515a77ce979","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6803","nonce":2826124409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6805","nonce":2826124409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6809","nonce":2826124409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6807","nonce":2826124409}]},"public_addr":"172.21.15.43:6803/2826124409","cluster_addr":"172.21.15.43:6805/2826124409","heartbeat_back_addr":"172.21.15.43:6809/2826124409","heartbeat_front_addr":"172.21.15.43:6807/2826124409","state":["exists","up"]},{"osd":1,"uuid":"031c143d-8ad1-4b68-8db9-3c717137e3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6801","nonce":3043263644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6803","nonce":3043263644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6807","nonce":3043263644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6805","nonce":3043263644}]},"public_addr":"172.21.15.82:6801/3043263644","cluster_addr":"172.21.15.82:6803/3043263644","heartbeat_back_addr":"172.21.15.82:6807/3043263644","heartbeat_front_addr":"172.21.15.82:6805/3043263644","state":["exists","up"]},{"osd":2,"uuid":"20632bbb-f07a-4ca1-b0c4-ff9d508e211c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6801","nonce":4213953645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6803","nonce":4213953645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6807","nonce":4213953645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6805","nonce":4213953645}]},"public_addr":"172.21.15.172:6801/4213953645","cluster_addr":"172.21.15.172:6803/4213953645","heartbeat_back_addr":"172.21.15.172:6807/4213953645","heartbeat_front_addr":"172.21.15.172:6805/4213953645","state":["exists","up"]},{"osd":3,"uuid":"9a1f2312-4dbd-475b-8736-d5ebc8788e09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6811","nonce":1597280920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6813","nonce":1597280920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6817","nonce":1597280920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6815","nonce":1597280920}]},"public_addr":"172.21.15.43:6811/1597280920","cluster_addr":"172.21.15.43:6813/1597280920","heartbeat_back_addr":"172.21.15.43:6817/1597280920","heartbeat_front_addr":"172.21.15.43:6815/1597280920","state":["exists","up"]},{"osd":4,"uuid":"29140ead-7075-42b2-90c1-27cae0b29ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6809","nonce":215451895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6811","nonce":215451895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6815","nonce":215451895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6813","nonce":215451895}]},"public_addr":"172.21.15.82:6809/215451895","cluster_addr":"172.21.15.82:6811/215451895","heartbeat_back_addr":"172.21.15.82:6815/215451895","heartbeat_front_addr":"172.21.15.82:6813/215451895","state":["exists","up"]},{"osd":5,"uuid":"59644c61-93dc-4536-b2ad-0256b24e5cb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6809","nonce":1576148933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6811","nonce":1576148933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6815","nonce":1576148933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6813","nonce":1576148933}]},"public_addr":"172.21.15.172:6809/1576148933","cluster_addr":"172.21.15.172:6811/1576148933","heartbeat_back_addr":"172.21.15.172:6815/1576148933","heartbeat_front_addr":"172.21.15.172:6813/1576148933","state":["exists","up"]},{"osd":6,"uuid":"22d54e57-c3f1-4ecf-9921-8e815b48f92a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6819","nonce":791044279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6821","nonce":791044279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6825","nonce":791044279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6823","nonce":791044279}]},"public_addr":"172.21.15.43:6819/791044279","cluster_addr":"172.21.15.43:6821/791044279","heartbeat_back_addr":"172.21.15.43:6825/791044279","heartbeat_front_addr":"172.21.15.43:6823/791044279","state":["exists","up"]},{"osd":7,"uuid":"248cd1e2-0d6b-452f-a0a8-c51fe1b975ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6817","nonce":2157777323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6819","nonce":2157777323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6823","nonce":2157777323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6821","nonce":2157777323}]},"public_addr":"172.21.15.82:6817/2157777323","cluster_addr":"172.21.15.82:6819/2157777323","heartbeat_back_addr":"172.21.15.82:6823/2157777323","heartbeat_front_addr":"172.21.15.82:6821/2157777323","state":["exists","up"]},{"osd":8,"uuid":"d98aab6e-8bae-4c24-906c-4f492f047b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6817","nonce":1095328248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6819","nonce":1095328248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6823","nonce":1095328248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6821","nonce":1095328248}]},"public_addr":"172.21.15.172:6817/1095328248","cluster_addr":"172.21.15.172:6819/1095328248","heartbeat_back_addr":"172.21.15.172:6823/1095328248","heartbeat_front_addr":"172.21.15.172:6821/1095328248","state":["exists","up"]},{"osd":9,"uuid":"19aa383e-a0f6-45ea-96cc-e7956535caa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6827","nonce":3657791498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6829","nonce":3657791498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6833","nonce":3657791498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6831","nonce":3657791498}]},"public_addr":"172.21.15.43:6827/3657791498","cluster_addr":"172.21.15.43:6829/3657791498","heartbeat_back_addr":"172.21.15.43:6833/3657791498","heartbeat_front_addr":"172.21.15.43:6831/3657791498","state":["exists","up"]},{"osd":10,"uuid":"666977e5-eaf7-41b9-a18f-c380e149345d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6825","nonce":3454954047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6827","nonce":3454954047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6831","nonce":3454954047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6829","nonce":3454954047}]},"public_addr":"172.21.15.82:6825/3454954047","cluster_addr":"172.21.15.82:6827/3454954047","heartbeat_back_addr":"172.21.15.82:6831/3454954047","heartbeat_front_addr":"172.21.15.82:6829/3454954047","state":["exists","up"]},{"osd":11,"uuid":"615cf3be-8076-470e-ad09-c880e7593717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6825","nonce":2563517144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6827","nonce":2563517144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6831","nonce":2563517144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6829","nonce":2563517144}]},"public_addr":"172.21.15.172:6825/2563517144","cluster_addr":"172.21.15.172:6827/2563517144","heartbeat_back_addr":"172.21.15.172:6831/2563517144","heartbeat_front_addr":"172.21.15.172:6829/2563517144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:21.727405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:47.100259+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:10.615438+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:43.632144+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:09.487940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:34.604463+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:09.594452+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:35.709795+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:01.184202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:35.455322+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:02.413166+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:28.550754+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.9","mappings":[{"from":5,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/578791143":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/4029313215":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/1574178482":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6801/3517712438":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6800/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/1375099638":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/2710117652":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6801/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/3215634168":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6800/3517712438":"2024-03-29T22:25:28.008215+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:36:04.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.709+0000 7fbf7e000640 1 -- 172.21.15.43:0/1548174905 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fbf600a1e00 msgr2=0x7fbf600a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:04.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.709+0000 7fbf7e000640 1 --2- 172.21.15.43:0/1548174905 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fbf600a1e00 0x7fbf600a42c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fbf680095f0 tx=0x7fbf68005f90 comp rx=0 tx=0).stop 2024-03-28T22:36:04.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.709+0000 7fbf7e000640 1 -- 172.21.15.43:0/1548174905 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbf781037a0 msgr2=0x7fbf78199ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:04.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.709+0000 7fbf7e000640 1 --2- 172.21.15.43:0/1548174905 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbf781037a0 0x7fbf78199ae0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c000c00 tx=0x7fbf6c02c650 comp rx=0 tx=0).stop 2024-03-28T22:36:04.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.709+0000 7fbf7e000640 1 -- 172.21.15.43:0/1548174905 shutdown_connections 2024-03-28T22:36:04.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.710+0000 7fbf7e000640 1 --2- 172.21.15.43:0/1548174905 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf78105350 0x7fbf781a1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:04.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.710+0000 7fbf7e000640 1 --2- 172.21.15.43:0/1548174905 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf78104990 0x7fbf7819a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:04.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.710+0000 7fbf7e000640 1 --2- 172.21.15.43:0/1548174905 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fbf600a1e00 0x7fbf600a42c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:04.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.710+0000 7fbf7e000640 1 --2- 172.21.15.43:0/1548174905 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbf781037a0 0x7fbf78199ae0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:04.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.710+0000 7fbf7e000640 1 -- 172.21.15.43:0/1548174905 >> 172.21.15.43:0/1548174905 conn(0x7fbf780fed30 msgr2=0x7fbf78107d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:04.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.710+0000 7fbf7e000640 1 -- 172.21.15.43:0/1548174905 shutdown_connections 2024-03-28T22:36:04.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.710+0000 7fbf7e000640 1 -- 172.21.15.43:0/1548174905 wait complete. 2024-03-28T22:36:04.729 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:36:04.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:04 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2274884368' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:36:04.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:04 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/332246955' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:36:04.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.955+0000 7fe3af000640 1 -- 172.21.15.43:0/2694290570 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe3a8076110 msgr2=0x7fe3a8076530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:04.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.955+0000 7fe3af000640 1 --2- 172.21.15.43:0/2694290570 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe3a8076110 0x7fe3a8076530 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe39c009900 tx=0x7fe39c02bc50 comp rx=0 tx=0).stop 2024-03-28T22:36:04.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.955+0000 7fe3af000640 1 -- 172.21.15.43:0/2694290570 shutdown_connections 2024-03-28T22:36:04.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.955+0000 7fe3af000640 1 --2- 172.21.15.43:0/2694290570 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3a8077cf0 0x7fe3a8078590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:04.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.955+0000 7fe3af000640 1 --2- 172.21.15.43:0/2694290570 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe3a8077350 0x7fe3a80777b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:04.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.955+0000 7fe3af000640 1 --2- 172.21.15.43:0/2694290570 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe3a8076110 0x7fe3a8076530 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:04.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.955+0000 7fe3af000640 1 -- 172.21.15.43:0/2694290570 >> 172.21.15.43:0/2694290570 conn(0x7fe3a80fed20 msgr2=0x7fe3a8101160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:04.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.955+0000 7fe3af000640 1 -- 172.21.15.43:0/2694290570 shutdown_connections 2024-03-28T22:36:04.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.955+0000 7fe3af000640 1 -- 172.21.15.43:0/2694290570 wait complete. 2024-03-28T22:36:04.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.956+0000 7fe3af000640 1 Processor -- start 2024-03-28T22:36:04.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.956+0000 7fe3af000640 1 -- start start 2024-03-28T22:36:04.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.957+0000 7fe3af000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe3a8076110 0x7fe3a819dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:04.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.957+0000 7fe3af000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3a8077350 0x7fe3a819e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:04.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.957+0000 7fe3af000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe3a8077cf0 0x7fe3a81a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:04.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.957+0000 7fe3af000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a8113880 con 0x7fe3a8077cf0 2024-03-28T22:36:04.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.957+0000 7fe3af000640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a8113700 con 0x7fe3a8076110 2024-03-28T22:36:04.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.957+0000 7fe3af000640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a8113a00 con 0x7fe3a8077350 2024-03-28T22:36:04.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.957+0000 7fe3ad200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3a8077350 0x7fe3a819e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:04.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.957+0000 7fe3adc00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe3a8076110 0x7fe3a819dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:04.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.957+0000 7fe3ae600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe3a8077cf0 0x7fe3a81a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:04.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.957+0000 7fe3adc00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe3a8076110 0x7fe3a819dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59480/0 (socket says 172.21.15.43:59480) 2024-03-28T22:36:04.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.957+0000 7fe3adc00640 1 -- 172.21.15.43:0/1403198489 learned_addr learned my addr 172.21.15.43:0/1403198489 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:04.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.957+0000 7fe3ad200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3a8077350 0x7fe3a819e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35434/0 (socket says 172.21.15.43:35434) 2024-03-28T22:36:04.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.958+0000 7fe3ae600640 1 -- 172.21.15.43:0/1403198489 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3a8077350 msgr2=0x7fe3a819e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:04.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.958+0000 7fe3ae600640 1 --2- 172.21.15.43:0/1403198489 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3a8077350 0x7fe3a819e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:04.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.958+0000 7fe3ae600640 1 -- 172.21.15.43:0/1403198489 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe3a8076110 msgr2=0x7fe3a819dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:04.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.958+0000 7fe3ae600640 1 --2- 172.21.15.43:0/1403198489 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe3a8076110 0x7fe3a819dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:04.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.958+0000 7fe3ae600640 1 -- 172.21.15.43:0/1403198489 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3a81a5ae0 con 0x7fe3a8077cf0 2024-03-28T22:36:04.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.958+0000 7fe3adc00640 1 --2- 172.21.15.43:0/1403198489 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe3a8076110 0x7fe3a819dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:04.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.958+0000 7fe3ae600640 1 --2- 172.21.15.43:0/1403198489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe3a8077cf0 0x7fe3a81a5470 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0009fd0 tx=0x7fe3a000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:04.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.958+0000 7fe3a6000640 1 -- 172.21.15.43:0/1403198489 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a000ce20 con 0x7fe3a8077cf0 2024-03-28T22:36:04.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.958+0000 7fe3a6000640 1 -- 172.21.15.43:0/1403198489 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3a0005020 con 0x7fe3a8077cf0 2024-03-28T22:36:04.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.958+0000 7fe3af000640 1 -- 172.21.15.43:0/1403198489 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3a81a5dd0 con 0x7fe3a8077cf0 2024-03-28T22:36:04.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.958+0000 7fe3a6000640 1 -- 172.21.15.43:0/1403198489 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a0010720 con 0x7fe3a8077cf0 2024-03-28T22:36:04.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.959+0000 7fe3af000640 1 -- 172.21.15.43:0/1403198489 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3a81a6290 con 0x7fe3a8077cf0 2024-03-28T22:36:04.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.960+0000 7fe3af000640 1 -- 172.21.15.43:0/1403198489 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3a8110120 con 0x7fe3a8077cf0 2024-03-28T22:36:04.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.962+0000 7fe3a6000640 1 -- 172.21.15.43:0/1403198489 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fe3a0005540 con 0x7fe3a8077cf0 2024-03-28T22:36:04.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.963+0000 7fe3a6000640 1 --2- 172.21.15.43:0/1403198489 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe3900a1d80 0x7fe3900a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:04.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.963+0000 7fe3a6000640 1 -- 172.21.15.43:0/1403198489 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7fe3a00ce7f0 con 0x7fe3a8077cf0 2024-03-28T22:36:04.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.963+0000 7fe3adc00640 1 --2- 172.21.15.43:0/1403198489 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe3900a1d80 0x7fe3900a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:04.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.964+0000 7fe3adc00640 1 --2- 172.21.15.43:0/1403198489 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe3900a1d80 0x7fe3900a4240 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe39c0096c0 tx=0x7fe39c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:04.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:04.969+0000 7fe3a6000640 1 -- 172.21.15.43:0/1403198489 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe3a009b830 con 0x7fe3a8077cf0 2024-03-28T22:36:05.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.212+0000 7fe3af000640 1 -- 172.21.15.43:0/1403198489 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3a802d170 con 0x7fe3a8077cf0 2024-03-28T22:36:05.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.214+0000 7fe3a6000640 1 -- 172.21.15.43:0/1403198489 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11934 (secure 0 0 0) 0x7fe3a009b130 con 0x7fe3a8077cf0 2024-03-28T22:36:05.212 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:36:05.212 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:33:28.906396+0000","modified":"2024-03-28T22:34:06.174761+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24572,"mds_1":24556,"mds_2":24535,"mds_3":24523,"mds_4":24518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/1084988072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1084988072},{"type":"v1","addr":"172.21.15.82:6835","nonce":1084988072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24535,24556,24518,24523,24572]},"id":1}]} 2024-03-28T22:36:05.212 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 28 2024-03-28T22:36:05.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.216+0000 7fe3af000640 1 -- 172.21.15.43:0/1403198489 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe3900a1d80 msgr2=0x7fe3900a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:05.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.216+0000 7fe3af000640 1 --2- 172.21.15.43:0/1403198489 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe3900a1d80 0x7fe3900a4240 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe39c0096c0 tx=0x7fe39c000bc0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.216+0000 7fe3af000640 1 -- 172.21.15.43:0/1403198489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe3a8077cf0 msgr2=0x7fe3a81a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:05.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.216+0000 7fe3af000640 1 --2- 172.21.15.43:0/1403198489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe3a8077cf0 0x7fe3a81a5470 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0009fd0 tx=0x7fe3a000ef10 comp rx=0 tx=0).stop 2024-03-28T22:36:05.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.217+0000 7fe3af000640 1 -- 172.21.15.43:0/1403198489 shutdown_connections 2024-03-28T22:36:05.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.217+0000 7fe3af000640 1 --2- 172.21.15.43:0/1403198489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe3a8077cf0 0x7fe3a81a5470 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.217+0000 7fe3af000640 1 --2- 172.21.15.43:0/1403198489 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3a8077350 0x7fe3a819e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.217+0000 7fe3af000640 1 --2- 172.21.15.43:0/1403198489 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe3900a1d80 0x7fe3900a4240 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.217+0000 7fe3af000640 1 --2- 172.21.15.43:0/1403198489 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe3a8076110 0x7fe3a819dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.217+0000 7fe3af000640 1 -- 172.21.15.43:0/1403198489 >> 172.21.15.43:0/1403198489 conn(0x7fe3a80fed20 msgr2=0x7fe3a8100820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:05.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.217+0000 7fe3af000640 1 -- 172.21.15.43:0/1403198489 shutdown_connections 2024-03-28T22:36:05.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.217+0000 7fe3af000640 1 -- 172.21.15.43:0/1403198489 wait complete. 2024-03-28T22:36:05.236 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-28T22:33:28.906396+0000', 'modified': '2024-03-28T22:34:06.174761+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24572, 'mds_1': 24556, 'mds_2': 24535, 'mds_3': 24523, 'mds_4': 24518}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24443': {'gid': 24443, 'name': 'k', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.82:6833/3363736995', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6832', 'nonce': 3363736995}, {'type': 'v1', 'addr': '172.21.15.82:6833', 'nonce': 3363736995}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24457': {'gid': 24457, 'name': 'l', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.172:6833/2579392697', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6832', 'nonce': 2579392697}, {'type': 'v1', 'addr': '172.21.15.172:6833', 'nonce': 2579392697}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24461': {'gid': 24461, 'name': 'j', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.43:6835/4195313768', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6834', 'nonce': 4195313768}, {'type': 'v1', 'addr': '172.21.15.43:6835', 'nonce': 4195313768}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24479': {'gid': 24479, 'name': 'd', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.43:6837/1330639541', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6836', 'nonce': 1330639541}, {'type': 'v1', 'addr': '172.21.15.43:6837', 'nonce': 1330639541}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24491': {'gid': 24491, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.82:6835/1084988072', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6834', 'nonce': 1084988072}, {'type': 'v1', 'addr': '172.21.15.82:6835', 'nonce': 1084988072}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24518': {'gid': 24518, 'name': 'h', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.82:6837/757432882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6836', 'nonce': 757432882}, {'type': 'v1', 'addr': '172.21.15.82:6837', 'nonce': 757432882}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24523': {'gid': 24523, 'name': 'f', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.172:6835/318854103', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6834', 'nonce': 318854103}, {'type': 'v1', 'addr': '172.21.15.172:6835', 'nonce': 318854103}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24535': {'gid': 24535, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.172:6837/3745415724', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6836', 'nonce': 3745415724}, {'type': 'v1', 'addr': '172.21.15.172:6837', 'nonce': 3745415724}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24556': {'gid': 24556, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.172:6839/3592465067', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6838', 'nonce': 3592465067}, {'type': 'v1', 'addr': '172.21.15.172:6839', 'nonce': 3592465067}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24572': {'gid': 24572, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.82:6839/2220892357', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6838', 'nonce': 2220892357}, {'type': 'v1', 'addr': '172.21.15.82:6839', 'nonce': 2220892357}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24572, 'qdb_cluster': [24535, 24556, 24518, 24523, 24572]} 2024-03-28T22:36:05.236 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-28T22:36:05.236 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-28T22:36:05.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.461+0000 7f86f1200640 1 -- 172.21.15.43:0/2019014917 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86ec1049e0 msgr2=0x7f86ec106dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:05.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.461+0000 7f86f1200640 1 --2- 172.21.15.43:0/2019014917 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86ec1049e0 0x7f86ec106dd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f86d4009900 tx=0x7f86d402bc30 comp rx=0 tx=0).stop 2024-03-28T22:36:05.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.461+0000 7f86f1200640 1 -- 172.21.15.43:0/2019014917 shutdown_connections 2024-03-28T22:36:05.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.461+0000 7f86f1200640 1 --2- 172.21.15.43:0/2019014917 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86ec1049e0 0x7f86ec106dd0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.461+0000 7f86f1200640 1 --2- 172.21.15.43:0/2019014917 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86ec107310 0x7f86ec109760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.461+0000 7f86f1200640 1 --2- 172.21.15.43:0/2019014917 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86ec1020b0 0x7f86ec1044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.461+0000 7f86f1200640 1 -- 172.21.15.43:0/2019014917 >> 172.21.15.43:0/2019014917 conn(0x7f86ec0fbac0 msgr2=0x7f86ec0fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:05.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.461+0000 7f86f1200640 1 -- 172.21.15.43:0/2019014917 shutdown_connections 2024-03-28T22:36:05.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.461+0000 7f86f1200640 1 -- 172.21.15.43:0/2019014917 wait complete. 2024-03-28T22:36:05.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.462+0000 7f86f1200640 1 Processor -- start 2024-03-28T22:36:05.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.462+0000 7f86f1200640 1 -- start start 2024-03-28T22:36:05.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.463+0000 7f86f1200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86ec1020b0 0x7f86ec197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.463+0000 7f86f1200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86ec1049e0 0x7f86ec197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.463+0000 7f86f1200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86ec107310 0x7f86ec19eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.463+0000 7f86f1200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f86ec10d2a0 con 0x7f86ec1049e0 2024-03-28T22:36:05.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.463+0000 7f86f1200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f86ec10d120 con 0x7f86ec107310 2024-03-28T22:36:05.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.463+0000 7f86f1200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f86ec10d420 con 0x7f86ec1020b0 2024-03-28T22:36:05.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.463+0000 7f86eb400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86ec1020b0 0x7f86ec197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.463+0000 7f86eb400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86ec1020b0 0x7f86ec197940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35458/0 (socket says 172.21.15.43:35458) 2024-03-28T22:36:05.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.463+0000 7f86eb400640 1 -- 172.21.15.43:0/1199403255 learned_addr learned my addr 172.21.15.43:0/1199403255 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:05.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.463+0000 7f86e2a00640 1 --2- 172.21.15.43:0/1199403255 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86ec1049e0 0x7f86ec197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.463+0000 7f86ebe00640 1 --2- 172.21.15.43:0/1199403255 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86ec107310 0x7f86ec19eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.463+0000 7f86eb400640 1 -- 172.21.15.43:0/1199403255 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86ec107310 msgr2=0x7f86ec19eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:05.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.463+0000 7f86eb400640 1 --2- 172.21.15.43:0/1199403255 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86ec107310 0x7f86ec19eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.463+0000 7f86eb400640 1 -- 172.21.15.43:0/1199403255 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86ec1049e0 msgr2=0x7f86ec197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:05.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.463+0000 7f86eb400640 1 --2- 172.21.15.43:0/1199403255 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86ec1049e0 0x7f86ec197e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.463+0000 7f86eb400640 1 -- 172.21.15.43:0/1199403255 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86ec19f560 con 0x7f86ec1020b0 2024-03-28T22:36:05.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.463+0000 7f86ebe00640 1 --2- 172.21.15.43:0/1199403255 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86ec107310 0x7f86ec19eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:05.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.464+0000 7f86eb400640 1 --2- 172.21.15.43:0/1199403255 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86ec1020b0 0x7f86ec197940 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f86dc00e9f0 tx=0x7f86dc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:05.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.464+0000 7f86e8c00640 1 -- 172.21.15.43:0/1199403255 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86dc00cd80 con 0x7f86ec1020b0 2024-03-28T22:36:05.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.464+0000 7f86e8c00640 1 -- 172.21.15.43:0/1199403255 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86dc010910 con 0x7f86ec1020b0 2024-03-28T22:36:05.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.464+0000 7f86e8c00640 1 -- 172.21.15.43:0/1199403255 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86dc018990 con 0x7f86ec1020b0 2024-03-28T22:36:05.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.464+0000 7f86f1200640 1 -- 172.21.15.43:0/1199403255 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86ec19f850 con 0x7f86ec1020b0 2024-03-28T22:36:05.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.464+0000 7f86f1200640 1 -- 172.21.15.43:0/1199403255 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86ec19fd20 con 0x7f86ec1020b0 2024-03-28T22:36:05.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.469+0000 7f86f55c1740 1 Processor -- start 2024-03-28T22:36:05.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.469+0000 7f86e8c00640 1 -- 172.21.15.43:0/1199403255 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f86dc010ac0 con 0x7f86ec1020b0 2024-03-28T22:36:05.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.469+0000 7f86f55c1740 1 -- start start 2024-03-28T22:36:05.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.469+0000 7f86f55c1740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556faee71b50 0x556faee71f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.469+0000 7f86f55c1740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556faee72540 0x556faee77040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.469+0000 7f86f55c1740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556faee776b0 0x556faee79b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.469+0000 7f86f55c1740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x556faedd19c0 con 0x556faee776b0 2024-03-28T22:36:05.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.469+0000 7f86f55c1740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x556faed7ac30 con 0x556faee72540 2024-03-28T22:36:05.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.469+0000 7f86f55c1740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x556faedcda60 con 0x556faee71b50 2024-03-28T22:36:05.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.470+0000 7f86eb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556faee72540 0x556faee77040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.470+0000 7f86ebe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556faee71b50 0x556faee71f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.470+0000 7f86eb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556faee72540 0x556faee77040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59508/0 (socket says 172.21.15.43:59508) 2024-03-28T22:36:05.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.470+0000 7f86e2a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556faee776b0 0x556faee79b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.470+0000 7f86eb400640 1 -- 172.21.15.43:0/1052576905 learned_addr learned my addr 172.21.15.43:0/1052576905 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:05.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.470+0000 7f86eb400640 1 -- 172.21.15.43:0/1052576905 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556faee71b50 msgr2=0x556faee71f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:05.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.470+0000 7f86eb400640 1 --2- 172.21.15.43:0/1052576905 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556faee71b50 0x556faee71f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.470+0000 7f86eb400640 1 -- 172.21.15.43:0/1052576905 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556faee776b0 msgr2=0x556faee79b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:05.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.470+0000 7f86eb400640 1 --2- 172.21.15.43:0/1052576905 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556faee776b0 0x556faee79b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.470+0000 7f86eb400640 1 -- 172.21.15.43:0/1052576905 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556faeae62d0 con 0x556faee72540 2024-03-28T22:36:05.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.470+0000 7f86eb400640 1 --2- 172.21.15.43:0/1052576905 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556faee72540 0x556faee77040 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f86dc0d6560 tx=0x7f86dc0d6a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=caaafded95614f87 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:05.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.470+0000 7f86e8c00640 1 --2- 172.21.15.43:0/1199403255 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f86d00a1e00 0x7f86d00a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.471+0000 7f86e0c00640 1 -- 172.21.15.43:0/1052576905 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86dc0e2070 con 0x556faee72540 2024-03-28T22:36:05.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.471+0000 7f86ebe00640 1 --2- 172.21.15.43:0/1199403255 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f86d00a1e00 0x7f86d00a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.471+0000 7f86e0c00640 1 -- 172.21.15.43:0/1052576905 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86dc0231f0 con 0x556faee72540 2024-03-28T22:36:05.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.471+0000 7f86e0c00640 1 -- 172.21.15.43:0/1052576905 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86dc09a970 con 0x556faee72540 2024-03-28T22:36:05.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.471+0000 7f86e8c00640 1 -- 172.21.15.43:0/1199403255 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f86dc0ce340 con 0x7f86ec1020b0 2024-03-28T22:36:05.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.471+0000 7f86f55c1740 1 -- 172.21.15.43:0/1052576905 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556faee72540 msgr2=0x556faee77040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:05.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.471+0000 7f86f55c1740 1 --2- 172.21.15.43:0/1052576905 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556faee72540 0x556faee77040 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f86dc0d6560 tx=0x7f86dc0d6a20 comp rx=0 tx=0).stop 2024-03-28T22:36:05.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.472+0000 7f86ebe00640 1 --2- 172.21.15.43:0/1199403255 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f86d00a1e00 0x7f86d00a42c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x556faee72da0 tx=0x7f86e4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:05.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.472+0000 7f86f55c1740 1 -- 172.21.15.43:0/1052576905 shutdown_connections 2024-03-28T22:36:05.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.472+0000 7f86f55c1740 1 --2- 172.21.15.43:0/1052576905 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556faee72540 0x556faee77040 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.472+0000 7f86f55c1740 1 --2- 172.21.15.43:0/1052576905 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556faee776b0 0x556faee79b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.472+0000 7f86f55c1740 1 --2- 172.21.15.43:0/1052576905 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556faee71b50 0x556faee71f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.472+0000 7f86f55c1740 1 -- 172.21.15.43:0/1052576905 >> 172.21.15.43:0/1052576905 conn(0x556faee68cb0 msgr2=0x556faee690b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:05.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.472+0000 7f86f55c1740 1 -- 172.21.15.43:0/1052576905 shutdown_connections 2024-03-28T22:36:05.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.472+0000 7f86f55c1740 1 -- 172.21.15.43:0/1052576905 wait complete. 2024-03-28T22:36:05.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.472+0000 7f86f55c1740 1 -- start start 2024-03-28T22:36:05.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.472+0000 7f86f55c1740 1 Processor -- start 2024-03-28T22:36:05.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.473+0000 7f86f55c1740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556faee71b50 0x556faee7c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.473+0000 7f86f55c1740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556faee72540 0x556faee7c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.473+0000 7f86f55c1740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556faee776b0 0x556faee835e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.473+0000 7f86f55c1740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x556faee7a660 con 0x556faee71b50 2024-03-28T22:36:05.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.473+0000 7f86f55c1740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x556faed70180 con 0x556faee776b0 2024-03-28T22:36:05.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.473+0000 7f86f55c1740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x556faee7a7e0 con 0x556faee72540 2024-03-28T22:36:05.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.473+0000 7f86f55c1740 10 client.0 _ll_register_callbacks cb 0x556faeae5ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:36:05.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.473+0000 7f86eb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556faee71b50 0x556faee7c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.473+0000 7f86ebe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556faee776b0 0x556faee835e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.473+0000 7f86e2a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556faee72540 0x556faee7c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.473+0000 7f86ebe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556faee776b0 0x556faee835e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59524/0 (socket says 172.21.15.43:59524) 2024-03-28T22:36:05.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.473+0000 7f86e2a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556faee72540 0x556faee7c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35490/0 (socket says 172.21.15.43:35490) 2024-03-28T22:36:05.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.473+0000 7f86eb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556faee71b50 0x556faee7c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49804/0 (socket says 172.21.15.43:49804) 2024-03-28T22:36:05.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.473+0000 7f86ebe00640 1 -- 172.21.15.43:0/1464520628 learned_addr learned my addr 172.21.15.43:0/1464520628 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:05.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.473+0000 7f86ebe00640 1 -- 172.21.15.43:0/1464520628 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556faee72540 msgr2=0x556faee7c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:05.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.474+0000 7f86ebe00640 1 --2- 172.21.15.43:0/1464520628 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556faee72540 0x556faee7c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.474+0000 7f86ebe00640 1 -- 172.21.15.43:0/1464520628 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556faee71b50 msgr2=0x556faee7c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:05.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.474+0000 7f86ebe00640 1 --2- 172.21.15.43:0/1464520628 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556faee71b50 0x556faee7c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.474+0000 7f86ebe00640 1 -- 172.21.15.43:0/1464520628 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556faebb06b0 con 0x556faee776b0 2024-03-28T22:36:05.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.474+0000 7f86ebe00640 1 --2- 172.21.15.43:0/1464520628 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556faee776b0 0x556faee835e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f86e40127c0 tx=0x7f86e4004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b7f348d8aaafdda server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:05.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.474+0000 7f86d9600640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:36:05.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.474+0000 7f86d9600640 1 -- 172.21.15.43:0/1464520628 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e4011c60 con 0x556faee776b0 2024-03-28T22:36:05.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.474+0000 7f86f55c1740 1 -- 172.21.15.43:0/1464520628 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86dc0231f0 con 0x556faee776b0 2024-03-28T22:36:05.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.474+0000 7f86d9600640 1 -- 172.21.15.43:0/1464520628 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f86e4011e00 con 0x556faee776b0 2024-03-28T22:36:05.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.474+0000 7f86d9600640 1 -- 172.21.15.43:0/1464520628 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f86e4029440 con 0x556faee776b0 2024-03-28T22:36:05.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.474+0000 7f86f55c1740 10 client.25169 fetch_fsmap learned FSMap version 28 2024-03-28T22:36:05.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.474+0000 7f86f55c1740 1 -- 172.21.15.43:0/1464520628 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556faee86a10 con 0x556faee776b0 2024-03-28T22:36:05.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.475+0000 7f86d9600640 1 -- 172.21.15.43:0/1464520628 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86dc0231f0 con 0x556faee776b0 2024-03-28T22:36:05.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.475+0000 7f86d9600640 1 -- 172.21.15.43:0/1464520628 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f86e4027070 con 0x556faee776b0 2024-03-28T22:36:05.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.475+0000 7f86f55c1740 10 client.25169 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T22:36:05.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.475+0000 7f86f55c1740 10 client.25169 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-28T22:36:05.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.475+0000 7f86f55c1740 20 client.25169 populate_metadata read hostname 'smithi043' 2024-03-28T22:36:05.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.475+0000 7f86f55c1740 1 --2- 172.21.15.43:0/1464520628 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x556faee890c0 0x556faee8a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.475+0000 7f86f55c1740 4 client.25169 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:36:05.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.475+0000 7f86f55c1740 1 -- 172.21.15.43:0/1464520628 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556faeddef00 con 0x556faee890c0 2024-03-28T22:36:05.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.475+0000 7f86eb400640 1 --2- 172.21.15.43:0/1464520628 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x556faee890c0 0x556faee8a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.476+0000 7f86eb400640 1 --2- 172.21.15.43:0/1464520628 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x556faee890c0 0x556faee8a530 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=57e6286a2f378f0c server_cookie=d2275b6ee32fb06 in_seq=0 out_seq=0 2024-03-28T22:36:05.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.476+0000 7f86d9600640 10 client.25169 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:36:05.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.477+0000 7f86d9600640 1 -- 172.21.15.43:0/1464520628 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556faeddef00 con 0x556faee890c0 2024-03-28T22:36:05.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.477+0000 7f86d9600640 10 client.25169 handle_command_reply: tid=0 2024-03-28T22:36:05.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.477+0000 7f86f55c1740 1 client.25169 shutdown 2024-03-28T22:36:05.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.477+0000 7f86f55c1740 1 -- 172.21.15.43:0/1464520628 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556faee776b0 msgr2=0x556faee835e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:05.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.477+0000 7f86f55c1740 1 --2- 172.21.15.43:0/1464520628 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556faee776b0 0x556faee835e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f86e40127c0 tx=0x7f86e4004ab0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.477+0000 7f86f55c1740 1 -- 172.21.15.43:0/1464520628 shutdown_connections 2024-03-28T22:36:05.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.477+0000 7f86f55c1740 1 --2- 172.21.15.43:0/1464520628 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x556faee890c0 0x556faee8a530 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.477+0000 7f86f55c1740 1 --2- 172.21.15.43:0/1464520628 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556faee776b0 0x556faee835e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.477+0000 7f86f55c1740 1 --2- 172.21.15.43:0/1464520628 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556faee72540 0x556faee7c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.477+0000 7f86f55c1740 1 --2- 172.21.15.43:0/1464520628 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556faee71b50 0x556faee7c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.477+0000 7f86f55c1740 1 -- 172.21.15.43:0/1464520628 >> 172.21.15.43:0/1464520628 conn(0x556faee68cb0 msgr2=0x556faee67df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:05.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.477+0000 7f86d9600640 0 client.25169 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:36:05.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.477+0000 7f86f55c1740 1 -- 172.21.15.43:0/1464520628 shutdown_connections 2024-03-28T22:36:05.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.477+0000 7f86f55c1740 1 -- 172.21.15.43:0/1464520628 wait complete. 2024-03-28T22:36:05.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.477+0000 7f86f55c1740 20 client.25169 trim_cache size 0 max 16384 2024-03-28T22:36:05.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.497+0000 7f86f55c1740 1 Processor -- start 2024-03-28T22:36:05.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.497+0000 7f86f55c1740 1 -- start start 2024-03-28T22:36:05.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.497+0000 7f86f55c1740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556faee71b50 0x556faee7bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.497+0000 7f86f55c1740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556faee72540 0x556faee83500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.498+0000 7f86f55c1740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556faee776b0 0x556faee7c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.498+0000 7f86f55c1740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x556faee88870 con 0x556faee71b50 2024-03-28T22:36:05.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.498+0000 7f86f55c1740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x556faee88f40 con 0x556faee776b0 2024-03-28T22:36:05.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.498+0000 7f86f55c1740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x556faee89610 con 0x556faee72540 2024-03-28T22:36:05.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.498+0000 7f86eb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556faee71b50 0x556faee7bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.498+0000 7f86ebe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556faee776b0 0x556faee7c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.498+0000 7f86ebe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556faee776b0 0x556faee7c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59536/0 (socket says 172.21.15.43:59536) 2024-03-28T22:36:05.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.498+0000 7f86ebe00640 1 -- 172.21.15.43:0/3108655384 learned_addr learned my addr 172.21.15.43:0/3108655384 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:05.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.498+0000 7f86eb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556faee71b50 0x556faee7bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49810/0 (socket says 172.21.15.43:49810) 2024-03-28T22:36:05.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.498+0000 7f86e2a00640 1 --2- 172.21.15.43:0/3108655384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556faee72540 0x556faee83500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.498+0000 7f86ebe00640 1 -- 172.21.15.43:0/3108655384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556faee72540 msgr2=0x556faee83500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:05.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.498+0000 7f86ebe00640 1 --2- 172.21.15.43:0/3108655384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556faee72540 0x556faee83500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.498+0000 7f86ebe00640 1 -- 172.21.15.43:0/3108655384 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556faee71b50 msgr2=0x556faee7bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:05.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.498+0000 7f86ebe00640 1 --2- 172.21.15.43:0/3108655384 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556faee71b50 0x556faee7bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.498+0000 7f86ebe00640 1 -- 172.21.15.43:0/3108655384 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556faee76d10 con 0x556faee776b0 2024-03-28T22:36:05.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.498+0000 7f86eb400640 1 --2- 172.21.15.43:0/3108655384 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556faee71b50 0x556faee7bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:05.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.499+0000 7f86ebe00640 1 --2- 172.21.15.43:0/3108655384 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556faee776b0 0x556faee7c1b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f86e400f800 tx=0x7f86e4005c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f88fcf5170acf68 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:05.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.499+0000 7f86dbe00640 1 -- 172.21.15.43:0/3108655384 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e403c070 con 0x556faee776b0 2024-03-28T22:36:05.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.499+0000 7f86dbe00640 1 -- 172.21.15.43:0/3108655384 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86e401fdf0 con 0x556faee776b0 2024-03-28T22:36:05.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.499+0000 7f86dbe00640 1 -- 172.21.15.43:0/3108655384 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e4029e50 con 0x556faee776b0 2024-03-28T22:36:05.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.500+0000 7f86f55c1740 1 -- 172.21.15.43:0/3108655384 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556faee776b0 msgr2=0x556faee7c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:05.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.500+0000 7f86f55c1740 1 --2- 172.21.15.43:0/3108655384 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556faee776b0 0x556faee7c1b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f86e400f800 tx=0x7f86e4005c50 comp rx=0 tx=0).stop 2024-03-28T22:36:05.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.500+0000 7f86f55c1740 1 -- 172.21.15.43:0/3108655384 shutdown_connections 2024-03-28T22:36:05.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.500+0000 7f86f55c1740 1 --2- 172.21.15.43:0/3108655384 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556faee776b0 0x556faee7c1b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.500+0000 7f86f55c1740 1 --2- 172.21.15.43:0/3108655384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556faee72540 0x556faee83500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.500+0000 7f86f55c1740 1 --2- 172.21.15.43:0/3108655384 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556faee71b50 0x556faee7bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.500+0000 7f86f55c1740 1 -- 172.21.15.43:0/3108655384 >> 172.21.15.43:0/3108655384 conn(0x556faee68cb0 msgr2=0x556faee73a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:05.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.500+0000 7f86f55c1740 1 -- 172.21.15.43:0/3108655384 shutdown_connections 2024-03-28T22:36:05.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.500+0000 7f86f55c1740 1 -- 172.21.15.43:0/3108655384 wait complete. 2024-03-28T22:36:05.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.500+0000 7f86f55c1740 1 -- start start 2024-03-28T22:36:05.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.501+0000 7f86f55c1740 1 Processor -- start 2024-03-28T22:36:05.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.501+0000 7f86f55c1740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556faee71b50 0x556faee82f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.501+0000 7f86f55c1740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556faee72540 0x556faee83470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.501+0000 7f86f55c1740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556faee776b0 0x556faee7fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.501+0000 7f86f55c1740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x556faee8a3b0 con 0x556faee776b0 2024-03-28T22:36:05.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.501+0000 7f86f55c1740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x556faee89ce0 con 0x556faee71b50 2024-03-28T22:36:05.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.501+0000 7f86f55c1740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x556faee8aa80 con 0x556faee72540 2024-03-28T22:36:05.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.501+0000 7f86f55c1740 10 client.0 _ll_register_callbacks cb 0x7f86e4012710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:36:05.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.501+0000 7f86eb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556faee71b50 0x556faee82f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.501+0000 7f86ebe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556faee776b0 0x556faee7fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.501+0000 7f86e2a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556faee72540 0x556faee83470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.501+0000 7f86eb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556faee71b50 0x556faee82f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59544/0 (socket says 172.21.15.43:59544) 2024-03-28T22:36:05.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.501+0000 7f86eb400640 1 -- 172.21.15.43:0/1618834965 learned_addr learned my addr 172.21.15.43:0/1618834965 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:05.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.502+0000 7f86eb400640 1 -- 172.21.15.43:0/1618834965 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556faee72540 msgr2=0x556faee83470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:05.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.502+0000 7f86eb400640 1 --2- 172.21.15.43:0/1618834965 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556faee72540 0x556faee83470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.502+0000 7f86eb400640 1 -- 172.21.15.43:0/1618834965 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556faee776b0 msgr2=0x556faee7fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:05.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.502+0000 7f86eb400640 1 --2- 172.21.15.43:0/1618834965 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556faee776b0 0x556faee7fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.502+0000 7f86eb400640 1 -- 172.21.15.43:0/1618834965 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556faee80180 con 0x556faee71b50 2024-03-28T22:36:05.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.502+0000 7f86eb400640 1 --2- 172.21.15.43:0/1618834965 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556faee71b50 0x556faee82f30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f86dc00c6a0 tx=0x7f86dc018420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=905ba26f9e9e6259 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:05.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.502+0000 7f86d9600640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:36:05.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.502+0000 7f86d9600640 1 -- 172.21.15.43:0/1618834965 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86dc058760 con 0x556faee71b50 2024-03-28T22:36:05.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.502+0000 7f86d9600640 1 -- 172.21.15.43:0/1618834965 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f86dc099130 con 0x556faee71b50 2024-03-28T22:36:05.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.502+0000 7f86f55c1740 1 -- 172.21.15.43:0/1618834965 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86e401fdf0 con 0x556faee71b50 2024-03-28T22:36:05.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.502+0000 7f86d9600640 1 -- 172.21.15.43:0/1618834965 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f86dc056040 con 0x556faee71b50 2024-03-28T22:36:05.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.502+0000 7f86f55c1740 10 client.25175 fetch_fsmap learned FSMap version 28 2024-03-28T22:36:05.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.502+0000 7f86f55c1740 1 -- 172.21.15.43:0/1618834965 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556faeeefa70 con 0x556faee71b50 2024-03-28T22:36:05.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.503+0000 7f86d9600640 1 -- 172.21.15.43:0/1618834965 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86e401fdf0 con 0x556faee71b50 2024-03-28T22:36:05.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.503+0000 7f86d9600640 1 -- 172.21.15.43:0/1618834965 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f86dc0d1b00 con 0x556faee71b50 2024-03-28T22:36:05.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.503+0000 7f86f55c1740 10 client.25175 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T22:36:05.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.503+0000 7f86f55c1740 10 client.25175 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-28T22:36:05.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.503+0000 7f86f55c1740 20 client.25175 populate_metadata read hostname 'smithi043' 2024-03-28T22:36:05.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.503+0000 7f86f55c1740 1 --2- 172.21.15.43:0/1618834965 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x556faeef4100 0x556faeef2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.503+0000 7f86f55c1740 4 client.25175 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T22:36:05.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.503+0000 7f86f55c1740 1 -- 172.21.15.43:0/1618834965 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "status"}) -- 0x556faedd9250 con 0x556faeef4100 2024-03-28T22:36:05.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.503+0000 7f86e2a00640 1 --2- 172.21.15.43:0/1618834965 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x556faeef4100 0x556faeef2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.504+0000 7f86e2a00640 1 --2- 172.21.15.43:0/1618834965 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x556faeef4100 0x556faeef2e10 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=89bf821750efb87c server_cookie=496643d38c462786 in_seq=0 out_seq=0 2024-03-28T22:36:05.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.504+0000 7f86d9600640 10 client.25175 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:36:05.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.504+0000 7f86d9600640 1 -- 172.21.15.43:0/1618834965 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x556faedd9250 con 0x556faeef4100 2024-03-28T22:36:05.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.504+0000 7f86d9600640 10 client.25175 handle_command_reply: tid=0 2024-03-28T22:36:05.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.504+0000 7f86f55c1740 1 client.25175 shutdown 2024-03-28T22:36:05.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.504+0000 7f86f55c1740 1 -- 172.21.15.43:0/1618834965 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556faee71b50 msgr2=0x556faee82f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:05.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.504+0000 7f86f55c1740 1 --2- 172.21.15.43:0/1618834965 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556faee71b50 0x556faee82f30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f86dc00c6a0 tx=0x7f86dc018420 comp rx=0 tx=0).stop 2024-03-28T22:36:05.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.504+0000 7f86f55c1740 1 -- 172.21.15.43:0/1618834965 shutdown_connections 2024-03-28T22:36:05.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.504+0000 7f86f55c1740 1 --2- 172.21.15.43:0/1618834965 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x556faeef4100 0x556faeef2e10 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.504+0000 7f86f55c1740 1 --2- 172.21.15.43:0/1618834965 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556faee776b0 0x556faee7fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.504+0000 7f86f55c1740 1 --2- 172.21.15.43:0/1618834965 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556faee72540 0x556faee83470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.504+0000 7f86f55c1740 1 --2- 172.21.15.43:0/1618834965 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556faee71b50 0x556faee82f30 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.504+0000 7f86d9600640 0 client.25175 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:36:05.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.504+0000 7f86f55c1740 1 -- 172.21.15.43:0/1618834965 >> 172.21.15.43:0/1618834965 conn(0x556faee68cb0 msgr2=0x556faee7bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:05.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.504+0000 7f86f55c1740 1 -- 172.21.15.43:0/1618834965 shutdown_connections 2024-03-28T22:36:05.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.504+0000 7f86f55c1740 1 -- 172.21.15.43:0/1618834965 wait complete. 2024-03-28T22:36:05.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.505+0000 7f86f55c1740 20 client.25175 trim_cache size 0 max 16384 2024-03-28T22:36:05.507 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:36:05.508 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_fsid": "c2276428-ed51-11ee-95d0-87774f69a715", 2024-03-28T22:36:05.508 INFO:teuthology.orchestra.run.smithi043.stdout: "whoami": 4, 2024-03-28T22:36:05.508 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 24518, 2024-03-28T22:36:05.508 INFO:teuthology.orchestra.run.smithi043.stdout: "want_state": "up:active", 2024-03-28T22:36:05.508 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "up:active", 2024-03-28T22:36:05.508 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs", 2024-03-28T22:36:05.508 INFO:teuthology.orchestra.run.smithi043.stdout: "rank_uptime": 120.335918081, 2024-03-28T22:36:05.508 INFO:teuthology.orchestra.run.smithi043.stdout: "mdsmap_epoch": 28, 2024-03-28T22:36:05.508 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch": 96, 2024-03-28T22:36:05.508 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch_barrier": 92, 2024-03-28T22:36:05.508 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 176.81688699599999 2024-03-28T22:36:05.508 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:36:05.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.505+0000 7f86f1200640 1 -- 172.21.15.43:0/1199403255 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f86d00a1e00 msgr2=0x7f86d00a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:05.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.505+0000 7f86f1200640 1 --2- 172.21.15.43:0/1199403255 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f86d00a1e00 0x7f86d00a42c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x556faee72da0 tx=0x7f86e4009500 comp rx=0 tx=0).stop 2024-03-28T22:36:05.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.505+0000 7f86f1200640 1 -- 172.21.15.43:0/1199403255 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86ec1020b0 msgr2=0x7f86ec197940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:05.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.505+0000 7f86f1200640 1 --2- 172.21.15.43:0/1199403255 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86ec1020b0 0x7f86ec197940 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f86dc00e9f0 tx=0x7f86dc00eeb0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.505+0000 7f86f1200640 1 -- 172.21.15.43:0/1199403255 shutdown_connections 2024-03-28T22:36:05.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.505+0000 7f86f1200640 1 --2- 172.21.15.43:0/1199403255 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f86d00a1e00 0x7f86d00a42c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.505+0000 7f86f1200640 1 --2- 172.21.15.43:0/1199403255 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86ec107310 0x7f86ec19eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.505+0000 7f86f1200640 1 --2- 172.21.15.43:0/1199403255 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86ec1049e0 0x7f86ec197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.505+0000 7f86f1200640 1 --2- 172.21.15.43:0/1199403255 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86ec1020b0 0x7f86ec197940 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.505+0000 7f86f1200640 1 -- 172.21.15.43:0/1199403255 >> 172.21.15.43:0/1199403255 conn(0x7f86ec0fbac0 msgr2=0x7f86ec1029f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:05.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.505+0000 7f86f1200640 1 -- 172.21.15.43:0/1199403255 shutdown_connections 2024-03-28T22:36:05.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.506+0000 7f86f1200640 1 -- 172.21.15.43:0/1199403255 wait complete. 2024-03-28T22:36:05.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:05 smithi043 ceph-mon[31823]: pgmap v404: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:36:05.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:05 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1548174905' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:36:05.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:05 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1403198489' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:36:05.520 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-03-28T22:36:05.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:05 smithi082 ceph-mon[36471]: pgmap v404: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:36:05.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:05 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1548174905' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:36:05.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:05 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1403198489' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:36:05.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.731+0000 7fa234200640 1 -- 172.21.15.43:0/2816583266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa22c105540 msgr2=0x7fa22c10bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:05.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.731+0000 7fa234200640 1 --2- 172.21.15.43:0/2816583266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa22c105540 0x7fa22c10bdd0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa224009ca0 tx=0x7fa22402c100 comp rx=0 tx=0).stop 2024-03-28T22:36:05.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.731+0000 7fa234200640 1 -- 172.21.15.43:0/2816583266 shutdown_connections 2024-03-28T22:36:05.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.731+0000 7fa234200640 1 --2- 172.21.15.43:0/2816583266 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa22c104b80 0x7fa22c105000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.731+0000 7fa234200640 1 --2- 172.21.15.43:0/2816583266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa22c105540 0x7fa22c10bdd0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.731+0000 7fa234200640 1 --2- 172.21.15.43:0/2816583266 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa22c103990 0x7fa22c103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.731+0000 7fa234200640 1 -- 172.21.15.43:0/2816583266 >> 172.21.15.43:0/2816583266 conn(0x7fa22c0fef00 msgr2=0x7fa22c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:05.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.731+0000 7fa234200640 1 -- 172.21.15.43:0/2816583266 shutdown_connections 2024-03-28T22:36:05.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.731+0000 7fa234200640 1 -- 172.21.15.43:0/2816583266 wait complete. 2024-03-28T22:36:05.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.732+0000 7fa234200640 1 Processor -- start 2024-03-28T22:36:05.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.732+0000 7fa234200640 1 -- start start 2024-03-28T22:36:05.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.733+0000 7fa234200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa22c103990 0x7fa22c199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.733+0000 7fa234200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa22c104b80 0x7fa22c19a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.733+0000 7fa234200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa22c105540 0x7fa22c1a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.733+0000 7fa234200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa22c10f5f0 con 0x7fa22c103990 2024-03-28T22:36:05.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.733+0000 7fa234200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa22c10f470 con 0x7fa22c104b80 2024-03-28T22:36:05.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.733+0000 7fa234200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fa22c10f770 con 0x7fa22c105540 2024-03-28T22:36:05.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.733+0000 7fa230e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa22c104b80 0x7fa22c19a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.733+0000 7fa232200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa22c105540 0x7fa22c1a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.733+0000 7fa231800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa22c103990 0x7fa22c199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.733+0000 7fa230e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa22c104b80 0x7fa22c19a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59564/0 (socket says 172.21.15.43:59564) 2024-03-28T22:36:05.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.733+0000 7fa230e00640 1 -- 172.21.15.43:0/4044593107 learned_addr learned my addr 172.21.15.43:0/4044593107 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:05.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.734+0000 7fa230e00640 1 -- 172.21.15.43:0/4044593107 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa22c105540 msgr2=0x7fa22c1a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:05.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.734+0000 7fa230e00640 1 --2- 172.21.15.43:0/4044593107 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa22c105540 0x7fa22c1a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.734+0000 7fa230e00640 1 -- 172.21.15.43:0/4044593107 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa22c103990 msgr2=0x7fa22c199c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:05.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.734+0000 7fa230e00640 1 --2- 172.21.15.43:0/4044593107 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa22c103990 0x7fa22c199c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.734+0000 7fa230e00640 1 -- 172.21.15.43:0/4044593107 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa22c1a1860 con 0x7fa22c104b80 2024-03-28T22:36:05.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.734+0000 7fa232200640 1 --2- 172.21.15.43:0/4044593107 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa22c105540 0x7fa22c1a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:05.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.734+0000 7fa230e00640 1 --2- 172.21.15.43:0/4044593107 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa22c104b80 0x7fa22c19a190 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa21800e9f0 tx=0x7fa21800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:05.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.734+0000 7fa22a000640 1 -- 172.21.15.43:0/4044593107 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa21800cd80 con 0x7fa22c104b80 2024-03-28T22:36:05.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.734+0000 7fa22a000640 1 -- 172.21.15.43:0/4044593107 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa218010910 con 0x7fa22c104b80 2024-03-28T22:36:05.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.734+0000 7fa22a000640 1 -- 172.21.15.43:0/4044593107 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa218018990 con 0x7fa22c104b80 2024-03-28T22:36:05.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.734+0000 7fa234200640 1 -- 172.21.15.43:0/4044593107 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa22c1a1b50 con 0x7fa22c104b80 2024-03-28T22:36:05.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.734+0000 7fa234200640 1 -- 172.21.15.43:0/4044593107 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa22c1a2050 con 0x7fa22c104b80 2024-03-28T22:36:05.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.739+0000 7fa236fc7740 1 Processor -- start 2024-03-28T22:36:05.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.740+0000 7fa236fc7740 1 -- start start 2024-03-28T22:36:05.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.740+0000 7fa236fc7740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557974a75570 0x557974a75990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.740+0000 7fa236fc7740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557974a75f60 0x557974a7aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.740+0000 7fa236fc7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557974a7b0d0 0x557974a7d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.740+0000 7fa236fc7740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5579749d99c0 con 0x557974a75f60 2024-03-28T22:36:05.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.740+0000 7fa236fc7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x557974982c30 con 0x557974a7b0d0 2024-03-28T22:36:05.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.740+0000 7fa236fc7740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5579749d5a60 con 0x557974a75570 2024-03-28T22:36:05.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.740+0000 7fa22a000640 1 -- 172.21.15.43:0/4044593107 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fa218010ac0 con 0x7fa22c104b80 2024-03-28T22:36:05.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.740+0000 7fa231800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557974a75f60 0x557974a7aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.740+0000 7fa232200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557974a75570 0x557974a75990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.740+0000 7fa231800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557974a75f60 0x557974a7aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49848/0 (socket says 172.21.15.43:49848) 2024-03-28T22:36:05.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.740+0000 7fa231800640 1 -- 172.21.15.43:0/3194943727 learned_addr learned my addr 172.21.15.43:0/3194943727 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:05.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.740+0000 7fa232200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557974a75570 0x557974a75990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35518/0 (socket says 172.21.15.43:35518) 2024-03-28T22:36:05.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.741+0000 7fa230e00640 1 --2- 172.21.15.43:0/3194943727 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557974a7b0d0 0x557974a7d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.741+0000 7fa231800640 1 -- 172.21.15.43:0/3194943727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557974a75570 msgr2=0x557974a75990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:05.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.741+0000 7fa231800640 1 --2- 172.21.15.43:0/3194943727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557974a75570 0x557974a75990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.741+0000 7fa231800640 1 -- 172.21.15.43:0/3194943727 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557974a7b0d0 msgr2=0x557974a7d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:05.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.741+0000 7fa231800640 1 --2- 172.21.15.43:0/3194943727 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557974a7b0d0 0x557974a7d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.741+0000 7fa231800640 1 -- 172.21.15.43:0/3194943727 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579746ee2d0 con 0x557974a75f60 2024-03-28T22:36:05.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.741+0000 7fa230e00640 1 --2- 172.21.15.43:0/3194943727 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557974a7b0d0 0x557974a7d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:36:05.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.741+0000 7fa232200640 1 --2- 172.21.15.43:0/3194943727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557974a75570 0x557974a75990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:05.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.741+0000 7fa231800640 1 --2- 172.21.15.43:0/3194943727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557974a75f60 0x557974a7aa60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa22000bc70 tx=0x7fa2200107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce7dfcb328be090a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:05.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.742+0000 7fa21ea00640 1 -- 172.21.15.43:0/3194943727 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa220004020 con 0x557974a75f60 2024-03-28T22:36:05.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.742+0000 7fa21ea00640 1 -- 172.21.15.43:0/3194943727 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2200041c0 con 0x557974a75f60 2024-03-28T22:36:05.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.742+0000 7fa21ea00640 1 -- 172.21.15.43:0/3194943727 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa220011730 con 0x557974a75f60 2024-03-28T22:36:05.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.742+0000 7fa22a000640 1 --2- 172.21.15.43:0/4044593107 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa2140a1e00 0x7fa2140a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.742+0000 7fa22a000640 1 -- 172.21.15.43:0/4044593107 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7fa2180ce470 con 0x7fa22c104b80 2024-03-28T22:36:05.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.742+0000 7fa232200640 1 --2- 172.21.15.43:0/4044593107 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa2140a1e00 0x7fa2140a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.742+0000 7fa236fc7740 1 -- 172.21.15.43:0/3194943727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557974a75f60 msgr2=0x557974a7aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:05.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.742+0000 7fa236fc7740 1 --2- 172.21.15.43:0/3194943727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557974a75f60 0x557974a7aa60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa22000bc70 tx=0x7fa2200107a0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.742+0000 7fa236fc7740 1 -- 172.21.15.43:0/3194943727 shutdown_connections 2024-03-28T22:36:05.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.742+0000 7fa236fc7740 1 --2- 172.21.15.43:0/3194943727 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557974a7b0d0 0x557974a7d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.742+0000 7fa236fc7740 1 --2- 172.21.15.43:0/3194943727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557974a75f60 0x557974a7aa60 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.742+0000 7fa236fc7740 1 --2- 172.21.15.43:0/3194943727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557974a75570 0x557974a75990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.742+0000 7fa236fc7740 1 -- 172.21.15.43:0/3194943727 >> 172.21.15.43:0/3194943727 conn(0x557974a70ae0 msgr2=0x557974a70ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:05.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.743+0000 7fa236fc7740 1 -- 172.21.15.43:0/3194943727 shutdown_connections 2024-03-28T22:36:05.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.743+0000 7fa236fc7740 1 -- 172.21.15.43:0/3194943727 wait complete. 2024-03-28T22:36:05.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.743+0000 7fa232200640 1 --2- 172.21.15.43:0/4044593107 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa2140a1e00 0x7fa2140a42c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa22402c640 tx=0x7fa224032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:05.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.743+0000 7fa236fc7740 1 -- start start 2024-03-28T22:36:05.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.743+0000 7fa236fc7740 1 Processor -- start 2024-03-28T22:36:05.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.744+0000 7fa236fc7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557974a75570 0x557974a7fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.744+0000 7fa236fc7740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557974a75f60 0x557974a7fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.744+0000 7fa236fc7740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557974a7b0d0 0x557974a87030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.744+0000 7fa236fc7740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x557974a7e080 con 0x557974a7b0d0 2024-03-28T22:36:05.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.744+0000 7fa236fc7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x557974978180 con 0x557974a75570 2024-03-28T22:36:05.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.744+0000 7fa236fc7740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x557974a7e200 con 0x557974a75f60 2024-03-28T22:36:05.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.744+0000 7fa236fc7740 10 client.0 _ll_register_callbacks cb 0x5579746edac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:36:05.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.744+0000 7fa230e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557974a75f60 0x557974a7fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.744+0000 7fa231800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557974a75570 0x557974a7fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.744+0000 7fa230e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557974a75f60 0x557974a7fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35526/0 (socket says 172.21.15.43:35526) 2024-03-28T22:36:05.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.744+0000 7fa230e00640 1 -- 172.21.15.43:0/2603639222 learned_addr learned my addr 172.21.15.43:0/2603639222 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:05.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.744+0000 7fa231800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557974a75570 0x557974a7fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59588/0 (socket says 172.21.15.43:59588) 2024-03-28T22:36:05.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.744+0000 7fa232200640 1 --2- 172.21.15.43:0/2603639222 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557974a7b0d0 0x557974a87030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.745+0000 7fa230e00640 1 -- 172.21.15.43:0/2603639222 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557974a75570 msgr2=0x557974a7fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:05.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.745+0000 7fa230e00640 1 --2- 172.21.15.43:0/2603639222 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557974a75570 0x557974a7fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.745+0000 7fa230e00640 1 -- 172.21.15.43:0/2603639222 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557974a7b0d0 msgr2=0x557974a87030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:05.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.745+0000 7fa230e00640 1 --2- 172.21.15.43:0/2603639222 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557974a7b0d0 0x557974a87030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.745+0000 7fa230e00640 1 -- 172.21.15.43:0/2603639222 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579747b86b0 con 0x557974a75f60 2024-03-28T22:36:05.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.745+0000 7fa231800640 1 --2- 172.21.15.43:0/2603639222 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557974a75570 0x557974a7fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:05.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.745+0000 7fa232200640 1 --2- 172.21.15.43:0/2603639222 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557974a7b0d0 0x557974a87030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:36:05.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.745+0000 7fa230e00640 1 --2- 172.21.15.43:0/2603639222 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557974a75f60 0x557974a7fff0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa218097800 tx=0x7fa218097150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e4f1ae75da2f038 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:05.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.745+0000 7fa213400640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:36:05.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.745+0000 7fa213400640 1 -- 172.21.15.43:0/2603639222 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2180aee60 con 0x557974a75f60 2024-03-28T22:36:05.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.745+0000 7fa213400640 1 -- 172.21.15.43:0/2603639222 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7fa218018d30 con 0x557974a75f60 2024-03-28T22:36:05.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.745+0000 7fa236fc7740 1 -- 172.21.15.43:0/2603639222 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2200041c0 con 0x557974a75f60 2024-03-28T22:36:05.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.745+0000 7fa213400640 1 -- 172.21.15.43:0/2603639222 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa218099000 con 0x557974a75f60 2024-03-28T22:36:05.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.745+0000 7fa236fc7740 10 client.25204 fetch_fsmap learned FSMap version 28 2024-03-28T22:36:05.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.745+0000 7fa236fc7740 1 -- 172.21.15.43:0/2603639222 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557974a8a470 con 0x557974a75f60 2024-03-28T22:36:05.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.746+0000 7fa213400640 1 -- 172.21.15.43:0/2603639222 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2200041c0 con 0x557974a75f60 2024-03-28T22:36:05.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.746+0000 7fa213400640 1 -- 172.21.15.43:0/2603639222 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa2180b3b40 con 0x557974a75f60 2024-03-28T22:36:05.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.746+0000 7fa236fc7740 10 client.25204 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T22:36:05.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.746+0000 7fa236fc7740 10 client.25204 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-28T22:36:05.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.746+0000 7fa236fc7740 20 client.25204 populate_metadata read hostname 'smithi043' 2024-03-28T22:36:05.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.746+0000 7fa236fc7740 1 --2- 172.21.15.43:0/2603639222 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x557974a8cb20 0x557974a8df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.746+0000 7fa236fc7740 4 client.25204 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:36:05.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.746+0000 7fa236fc7740 1 -- 172.21.15.43:0/2603639222 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5579749e6f00 con 0x557974a8cb20 2024-03-28T22:36:05.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.746+0000 7fa231800640 1 --2- 172.21.15.43:0/2603639222 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x557974a8cb20 0x557974a8df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.747+0000 7fa231800640 1 --2- 172.21.15.43:0/2603639222 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x557974a8cb20 0x557974a8df90 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5528bddef6f15d69 server_cookie=4509f7ff8ecabbc5 in_seq=0 out_seq=0 2024-03-28T22:36:05.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.747+0000 7fa213400640 10 client.25204 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:36:05.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.748+0000 7fa213400640 1 -- 172.21.15.43:0/2603639222 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5579749e6f00 con 0x557974a8cb20 2024-03-28T22:36:05.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.748+0000 7fa213400640 10 client.25204 handle_command_reply: tid=0 2024-03-28T22:36:05.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.748+0000 7fa236fc7740 1 client.25204 shutdown 2024-03-28T22:36:05.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.749+0000 7fa236fc7740 1 -- 172.21.15.43:0/2603639222 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557974a75f60 msgr2=0x557974a7fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:05.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.749+0000 7fa236fc7740 1 --2- 172.21.15.43:0/2603639222 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557974a75f60 0x557974a7fff0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa218097800 tx=0x7fa218097150 comp rx=0 tx=0).stop 2024-03-28T22:36:05.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.749+0000 7fa236fc7740 1 -- 172.21.15.43:0/2603639222 shutdown_connections 2024-03-28T22:36:05.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.749+0000 7fa236fc7740 1 --2- 172.21.15.43:0/2603639222 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x557974a8cb20 0x557974a8df90 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.749+0000 7fa236fc7740 1 --2- 172.21.15.43:0/2603639222 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557974a7b0d0 0x557974a87030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.749+0000 7fa236fc7740 1 --2- 172.21.15.43:0/2603639222 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557974a75f60 0x557974a7fff0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.749+0000 7fa236fc7740 1 --2- 172.21.15.43:0/2603639222 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557974a75570 0x557974a7fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.749+0000 7fa236fc7740 1 -- 172.21.15.43:0/2603639222 >> 172.21.15.43:0/2603639222 conn(0x557974a70ae0 msgr2=0x557974a6fce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:05.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.749+0000 7fa213400640 0 client.25204 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:36:05.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.749+0000 7fa236fc7740 1 -- 172.21.15.43:0/2603639222 shutdown_connections 2024-03-28T22:36:05.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.749+0000 7fa236fc7740 1 -- 172.21.15.43:0/2603639222 wait complete. 2024-03-28T22:36:05.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.749+0000 7fa236fc7740 20 client.25204 trim_cache size 0 max 16384 2024-03-28T22:36:05.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.769+0000 7fa236fc7740 1 Processor -- start 2024-03-28T22:36:05.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.769+0000 7fa236fc7740 1 -- start start 2024-03-28T22:36:05.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.770+0000 7fa236fc7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557974a75570 0x557974a7d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.770+0000 7fa236fc7740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557974a75f60 0x557974a86ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.770+0000 7fa236fc7740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557974a7b0d0 0x557974a6f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.770+0000 7fa236fc7740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x557974a8c2d0 con 0x557974a75f60 2024-03-28T22:36:05.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.770+0000 7fa236fc7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x557974a8c9a0 con 0x557974a75570 2024-03-28T22:36:05.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.770+0000 7fa236fc7740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x557974a8d070 con 0x557974a7b0d0 2024-03-28T22:36:05.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.770+0000 7fa230e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557974a75f60 0x557974a86ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.770+0000 7fa230e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557974a75f60 0x557974a86ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:49864/0 (socket says 172.21.15.43:49864) 2024-03-28T22:36:05.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.770+0000 7fa231800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557974a75570 0x557974a7d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.770+0000 7fa230e00640 1 -- 172.21.15.43:0/2044661448 learned_addr learned my addr 172.21.15.43:0/2044661448 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:05.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.770+0000 7fa232200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557974a7b0d0 0x557974a6f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.770+0000 7fa230e00640 1 -- 172.21.15.43:0/2044661448 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557974a7b0d0 msgr2=0x557974a6f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:05.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.770+0000 7fa230e00640 1 --2- 172.21.15.43:0/2044661448 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557974a7b0d0 0x557974a6f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.770+0000 7fa230e00640 1 -- 172.21.15.43:0/2044661448 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557974a75570 msgr2=0x557974a7d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:05.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.770+0000 7fa230e00640 1 --2- 172.21.15.43:0/2044661448 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557974a75570 0x557974a7d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.770+0000 7fa230e00640 1 -- 172.21.15.43:0/2044661448 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557974a71850 con 0x557974a75f60 2024-03-28T22:36:05.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.771+0000 7fa230e00640 1 --2- 172.21.15.43:0/2044661448 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557974a75f60 0x557974a86ff0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa218097290 tx=0x7fa21809a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=644dcb604d9098e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:05.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.771+0000 7fa21e000640 1 -- 172.21.15.43:0/2044661448 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2180a35b0 con 0x557974a75f60 2024-03-28T22:36:05.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.771+0000 7fa21e000640 1 -- 172.21.15.43:0/2044661448 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa21809aec0 con 0x557974a75f60 2024-03-28T22:36:05.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.771+0000 7fa21e000640 1 -- 172.21.15.43:0/2044661448 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa21809a310 con 0x557974a75f60 2024-03-28T22:36:05.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.772+0000 7fa236fc7740 1 -- 172.21.15.43:0/2044661448 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557974a75f60 msgr2=0x557974a86ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:05.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.772+0000 7fa236fc7740 1 --2- 172.21.15.43:0/2044661448 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557974a75f60 0x557974a86ff0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa218097290 tx=0x7fa21809a8e0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.772+0000 7fa236fc7740 1 -- 172.21.15.43:0/2044661448 shutdown_connections 2024-03-28T22:36:05.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.772+0000 7fa236fc7740 1 --2- 172.21.15.43:0/2044661448 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557974a7b0d0 0x557974a6f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.772+0000 7fa236fc7740 1 --2- 172.21.15.43:0/2044661448 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557974a75f60 0x557974a86ff0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.772+0000 7fa236fc7740 1 --2- 172.21.15.43:0/2044661448 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557974a75570 0x557974a7d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.772+0000 7fa236fc7740 1 -- 172.21.15.43:0/2044661448 >> 172.21.15.43:0/2044661448 conn(0x557974a70ae0 msgr2=0x557974a77460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:05.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.772+0000 7fa236fc7740 1 -- 172.21.15.43:0/2044661448 shutdown_connections 2024-03-28T22:36:05.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.772+0000 7fa236fc7740 1 -- 172.21.15.43:0/2044661448 wait complete. 2024-03-28T22:36:05.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.772+0000 7fa236fc7740 1 -- start start 2024-03-28T22:36:05.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.772+0000 7fa236fc7740 1 Processor -- start 2024-03-28T22:36:05.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.773+0000 7fa236fc7740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557974a75570 0x557974a82b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.773+0000 7fa236fc7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557974a75f60 0x557974a830c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.773+0000 7fa236fc7740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557974a7b0d0 0x557974a85fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.773+0000 7fa236fc7740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x557974a8de10 con 0x557974a7b0d0 2024-03-28T22:36:05.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.773+0000 7fa236fc7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x557974a8d740 con 0x557974a75f60 2024-03-28T22:36:05.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.773+0000 7fa236fc7740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x557974a8e4e0 con 0x557974a75570 2024-03-28T22:36:05.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.773+0000 7fa236fc7740 10 client.0 _ll_register_callbacks cb 0x7fa21800e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:36:05.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.773+0000 7fa231800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557974a75570 0x557974a82b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.773+0000 7fa231800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557974a75570 0x557974a82b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35544/0 (socket says 172.21.15.43:35544) 2024-03-28T22:36:05.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.773+0000 7fa231800640 1 -- 172.21.15.43:0/501837038 learned_addr learned my addr 172.21.15.43:0/501837038 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:05.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.773+0000 7fa230e00640 1 --2- 172.21.15.43:0/501837038 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557974a75f60 0x557974a830c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.773+0000 7fa232200640 1 --2- 172.21.15.43:0/501837038 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557974a7b0d0 0x557974a85fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.773+0000 7fa231800640 1 -- 172.21.15.43:0/501837038 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557974a75f60 msgr2=0x557974a830c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:05.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.773+0000 7fa231800640 1 --2- 172.21.15.43:0/501837038 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557974a75f60 0x557974a830c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.773+0000 7fa231800640 1 -- 172.21.15.43:0/501837038 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557974a7b0d0 msgr2=0x557974a85fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:05.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.773+0000 7fa231800640 1 --2- 172.21.15.43:0/501837038 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557974a7b0d0 0x557974a85fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.774+0000 7fa231800640 1 -- 172.21.15.43:0/501837038 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557974a865c0 con 0x557974a75570 2024-03-28T22:36:05.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.774+0000 7fa230e00640 1 --2- 172.21.15.43:0/501837038 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557974a75f60 0x557974a830c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:36:05.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.774+0000 7fa231800640 1 --2- 172.21.15.43:0/501837038 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557974a75570 0x557974a82b80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa22000bc70 tx=0x7fa220005c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ecf6ce64cb0739 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:05.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.774+0000 7fa213400640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:36:05.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.774+0000 7fa213400640 1 -- 172.21.15.43:0/501837038 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa22002e060 con 0x557974a75570 2024-03-28T22:36:05.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.774+0000 7fa236fc7740 1 -- 172.21.15.43:0/501837038 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa21809aec0 con 0x557974a75570 2024-03-28T22:36:05.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.774+0000 7fa213400640 1 -- 172.21.15.43:0/501837038 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7fa220029630 con 0x557974a75570 2024-03-28T22:36:05.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.774+0000 7fa213400640 1 -- 172.21.15.43:0/501837038 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa220035050 con 0x557974a75570 2024-03-28T22:36:05.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.774+0000 7fa236fc7740 10 client.25210 fetch_fsmap learned FSMap version 28 2024-03-28T22:36:05.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.774+0000 7fa236fc7740 1 -- 172.21.15.43:0/501837038 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557974af2c30 con 0x557974a75570 2024-03-28T22:36:05.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.775+0000 7fa213400640 1 -- 172.21.15.43:0/501837038 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa21809aec0 con 0x557974a75570 2024-03-28T22:36:05.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.775+0000 7fa213400640 1 -- 172.21.15.43:0/501837038 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa220003a80 con 0x557974a75570 2024-03-28T22:36:05.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.775+0000 7fa236fc7740 10 client.25210 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T22:36:05.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.775+0000 7fa236fc7740 10 client.25210 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-28T22:36:05.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.775+0000 7fa236fc7740 20 client.25210 populate_metadata read hostname 'smithi043' 2024-03-28T22:36:05.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.775+0000 7fa236fc7740 1 --2- 172.21.15.43:0/501837038 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x557974af7500 0x557974af6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:05.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.775+0000 7fa236fc7740 4 client.25210 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T22:36:05.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.775+0000 7fa236fc7740 1 -- 172.21.15.43:0/501837038 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "status"}) -- 0x5579749e1250 con 0x557974af7500 2024-03-28T22:36:05.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.775+0000 7fa230e00640 1 --2- 172.21.15.43:0/501837038 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x557974af7500 0x557974af6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:05.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.776+0000 7fa230e00640 1 --2- 172.21.15.43:0/501837038 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x557974af7500 0x557974af6210 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=29cb5bb324b5c8e9 server_cookie=c55146bba0b3bf96 in_seq=0 out_seq=0 2024-03-28T22:36:05.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.776+0000 7fa213400640 10 client.25210 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:36:05.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.776+0000 7fa213400640 1 -- 172.21.15.43:0/501837038 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5579749e1250 con 0x557974af7500 2024-03-28T22:36:05.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.776+0000 7fa213400640 10 client.25210 handle_command_reply: tid=0 2024-03-28T22:36:05.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.776+0000 7fa236fc7740 1 client.25210 shutdown 2024-03-28T22:36:05.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.776+0000 7fa236fc7740 1 -- 172.21.15.43:0/501837038 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557974a75570 msgr2=0x557974a82b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:05.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.776+0000 7fa236fc7740 1 --2- 172.21.15.43:0/501837038 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557974a75570 0x557974a82b80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa22000bc70 tx=0x7fa220005c30 comp rx=0 tx=0).stop 2024-03-28T22:36:05.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.776+0000 7fa236fc7740 1 -- 172.21.15.43:0/501837038 shutdown_connections 2024-03-28T22:36:05.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.776+0000 7fa236fc7740 1 --2- 172.21.15.43:0/501837038 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x557974af7500 0x557974af6210 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.776+0000 7fa236fc7740 1 --2- 172.21.15.43:0/501837038 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557974a75f60 0x557974a830c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.776+0000 7fa236fc7740 1 --2- 172.21.15.43:0/501837038 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557974a7b0d0 0x557974a85fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.776+0000 7fa213400640 0 client.25210 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:36:05.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.776+0000 7fa236fc7740 1 --2- 172.21.15.43:0/501837038 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557974a75570 0x557974a82b80 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.776+0000 7fa236fc7740 1 -- 172.21.15.43:0/501837038 >> 172.21.15.43:0/501837038 conn(0x557974a70ae0 msgr2=0x557974a745d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:05.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.776+0000 7fa236fc7740 1 -- 172.21.15.43:0/501837038 shutdown_connections 2024-03-28T22:36:05.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.776+0000 7fa236fc7740 1 -- 172.21.15.43:0/501837038 wait complete. 2024-03-28T22:36:05.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.776+0000 7fa236fc7740 20 client.25210 trim_cache size 0 max 16384 2024-03-28T22:36:05.780 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:36:05.780 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_fsid": "c2276428-ed51-11ee-95d0-87774f69a715", 2024-03-28T22:36:05.780 INFO:teuthology.orchestra.run.smithi043.stdout: "whoami": 3, 2024-03-28T22:36:05.780 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 24523, 2024-03-28T22:36:05.780 INFO:teuthology.orchestra.run.smithi043.stdout: "want_state": "up:active", 2024-03-28T22:36:05.780 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "up:active", 2024-03-28T22:36:05.780 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs", 2024-03-28T22:36:05.780 INFO:teuthology.orchestra.run.smithi043.stdout: "rank_uptime": 121.62220260399999, 2024-03-28T22:36:05.780 INFO:teuthology.orchestra.run.smithi043.stdout: "mdsmap_epoch": 28, 2024-03-28T22:36:05.780 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch": 96, 2024-03-28T22:36:05.780 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch_barrier": 92, 2024-03-28T22:36:05.780 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 183.045921002 2024-03-28T22:36:05.780 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:36:05.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.777+0000 7fa234200640 1 -- 172.21.15.43:0/4044593107 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa2140a1e00 msgr2=0x7fa2140a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:05.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.777+0000 7fa234200640 1 --2- 172.21.15.43:0/4044593107 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa2140a1e00 0x7fa2140a42c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa22402c640 tx=0x7fa224032590 comp rx=0 tx=0).stop 2024-03-28T22:36:05.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.777+0000 7fa234200640 1 -- 172.21.15.43:0/4044593107 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa22c104b80 msgr2=0x7fa22c19a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:05.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.777+0000 7fa234200640 1 --2- 172.21.15.43:0/4044593107 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa22c104b80 0x7fa22c19a190 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa21800e9f0 tx=0x7fa21800eeb0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.777+0000 7fa234200640 1 -- 172.21.15.43:0/4044593107 shutdown_connections 2024-03-28T22:36:05.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.777+0000 7fa234200640 1 --2- 172.21.15.43:0/4044593107 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa2140a1e00 0x7fa2140a42c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.777+0000 7fa234200640 1 --2- 172.21.15.43:0/4044593107 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa22c104b80 0x7fa22c19a190 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.777+0000 7fa234200640 1 --2- 172.21.15.43:0/4044593107 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa22c105540 0x7fa22c1a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.777+0000 7fa234200640 1 --2- 172.21.15.43:0/4044593107 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa22c103990 0x7fa22c199c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:05.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.777+0000 7fa234200640 1 -- 172.21.15.43:0/4044593107 >> 172.21.15.43:0/4044593107 conn(0x7fa22c0fef00 msgr2=0x7fa22c107e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:05.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.777+0000 7fa234200640 1 -- 172.21.15.43:0/4044593107 shutdown_connections 2024-03-28T22:36:05.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:05.778+0000 7fa234200640 1 -- 172.21.15.43:0/4044593107 wait complete. 2024-03-28T22:36:05.795 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-28T22:36:05.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:05 smithi172 ceph-mon[36566]: pgmap v404: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:36:05.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:05 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1548174905' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:36:05.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:05 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1403198489' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:36:06.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.014+0000 7ff056200640 1 -- 172.21.15.43:0/2473104308 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff050105420 msgr2=0x7ff05010bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:06.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.014+0000 7ff056200640 1 --2- 172.21.15.43:0/2473104308 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff050105420 0x7ff05010bcb0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff048009be0 tx=0x7ff04802bef0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.014+0000 7ff056200640 1 -- 172.21.15.43:0/2473104308 shutdown_connections 2024-03-28T22:36:06.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.014+0000 7ff056200640 1 --2- 172.21.15.43:0/2473104308 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff050105420 0x7ff05010bcb0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.014+0000 7ff056200640 1 --2- 172.21.15.43:0/2473104308 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff050104a60 0x7ff050104ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.014+0000 7ff056200640 1 --2- 172.21.15.43:0/2473104308 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff050103870 0x7ff050103c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.014+0000 7ff056200640 1 -- 172.21.15.43:0/2473104308 >> 172.21.15.43:0/2473104308 conn(0x7ff0500fee00 msgr2=0x7ff050101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:06.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.014+0000 7ff056200640 1 -- 172.21.15.43:0/2473104308 shutdown_connections 2024-03-28T22:36:06.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.015+0000 7ff056200640 1 -- 172.21.15.43:0/2473104308 wait complete. 2024-03-28T22:36:06.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.015+0000 7ff056200640 1 Processor -- start 2024-03-28T22:36:06.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.016+0000 7ff056200640 1 -- start start 2024-03-28T22:36:06.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.016+0000 7ff056200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff050103870 0x7ff050199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.016+0000 7ff056200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff050104a60 0x7ff05019a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.016+0000 7ff056200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff050105420 0x7ff0501a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.016+0000 7ff056200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff05010f4f0 con 0x7ff050103870 2024-03-28T22:36:06.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.016+0000 7ff056200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ff05010f370 con 0x7ff050105420 2024-03-28T22:36:06.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.016+0000 7ff056200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7ff05010f670 con 0x7ff050104a60 2024-03-28T22:36:06.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.016+0000 7ff04fe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff050104a60 0x7ff05019a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.016+0000 7ff054e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff050103870 0x7ff050199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.016+0000 7ff055800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff050105420 0x7ff0501a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.016+0000 7ff04fe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff050104a60 0x7ff05019a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35564/0 (socket says 172.21.15.43:35564) 2024-03-28T22:36:06.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.016+0000 7ff04fe00640 1 -- 172.21.15.43:0/2435643444 learned_addr learned my addr 172.21.15.43:0/2435643444 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:06.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.017+0000 7ff04fe00640 1 -- 172.21.15.43:0/2435643444 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff050105420 msgr2=0x7ff0501a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:06.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.017+0000 7ff04fe00640 1 --2- 172.21.15.43:0/2435643444 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff050105420 0x7ff0501a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.017+0000 7ff04fe00640 1 -- 172.21.15.43:0/2435643444 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff050103870 msgr2=0x7ff050199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:06.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.017+0000 7ff04fe00640 1 --2- 172.21.15.43:0/2435643444 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff050103870 0x7ff050199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.017+0000 7ff04fe00640 1 -- 172.21.15.43:0/2435643444 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0501a1720 con 0x7ff050104a60 2024-03-28T22:36:06.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.017+0000 7ff055800640 1 --2- 172.21.15.43:0/2435643444 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff050105420 0x7ff0501a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:06.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.017+0000 7ff04fe00640 1 --2- 172.21.15.43:0/2435643444 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff050104a60 0x7ff05019a040 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff03c00d950 tx=0x7ff03c00dd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:06.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.017+0000 7ff04d600640 1 -- 172.21.15.43:0/2435643444 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03c017070 con 0x7ff050104a60 2024-03-28T22:36:06.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.017+0000 7ff04d600640 1 -- 172.21.15.43:0/2435643444 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff03c00de80 con 0x7ff050104a60 2024-03-28T22:36:06.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.017+0000 7ff056200640 1 -- 172.21.15.43:0/2435643444 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0501a1a10 con 0x7ff050104a60 2024-03-28T22:36:06.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.017+0000 7ff04d600640 1 -- 172.21.15.43:0/2435643444 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03c00f620 con 0x7ff050104a60 2024-03-28T22:36:06.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.018+0000 7ff056200640 1 -- 172.21.15.43:0/2435643444 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0501a1fb0 con 0x7ff050104a60 2024-03-28T22:36:06.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.022+0000 7ff05a3e6740 1 Processor -- start 2024-03-28T22:36:06.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.022+0000 7ff04d600640 1 -- 172.21.15.43:0/2435643444 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7ff03c009c30 con 0x7ff050104a60 2024-03-28T22:36:06.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.022+0000 7ff05a3e6740 1 -- start start 2024-03-28T22:36:06.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.023+0000 7ff05a3e6740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a53db22740 0x55a53db22b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.023+0000 7ff05a3e6740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a53db23130 0x55a53db27c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.023+0000 7ff05a3e6740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a53db282a0 0x55a53db2a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.023+0000 7ff05a3e6740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55a53da869c0 con 0x55a53db22740 2024-03-28T22:36:06.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.023+0000 7ff05a3e6740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55a53da2fc30 con 0x55a53db282a0 2024-03-28T22:36:06.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.023+0000 7ff05a3e6740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55a53da82a60 con 0x55a53db23130 2024-03-28T22:36:06.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.023+0000 7ff04fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a53db282a0 0x55a53db2a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.023+0000 7ff055800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a53db22740 0x55a53db22b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.023+0000 7ff04fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a53db282a0 0x55a53db2a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59640/0 (socket says 172.21.15.43:59640) 2024-03-28T22:36:06.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.023+0000 7ff04fe00640 1 -- 172.21.15.43:0/292835630 learned_addr learned my addr 172.21.15.43:0/292835630 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:06.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.024+0000 7ff04fe00640 1 -- 172.21.15.43:0/292835630 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a53db23130 msgr2=0x55a53db27c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:06.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.024+0000 7ff04fe00640 1 --2- 172.21.15.43:0/292835630 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a53db23130 0x55a53db27c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.024+0000 7ff04fe00640 1 -- 172.21.15.43:0/292835630 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a53db22740 msgr2=0x55a53db22b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:06.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.024+0000 7ff04fe00640 1 --2- 172.21.15.43:0/292835630 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a53db22740 0x55a53db22b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.024+0000 7ff04fe00640 1 -- 172.21.15.43:0/292835630 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a53d79b2d0 con 0x55a53db282a0 2024-03-28T22:36:06.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.024+0000 7ff04fe00640 1 --2- 172.21.15.43:0/292835630 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a53db282a0 0x55a53db2a730 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff03c0d1e30 tx=0x7ff03c0d6760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de9844674308313c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:06.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.024+0000 7ff04d600640 1 --2- 172.21.15.43:0/2435643444 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff0380a1e00 0x7ff0380a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.024+0000 7ff042000640 1 -- 172.21.15.43:0/292835630 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03c0968d0 con 0x55a53db282a0 2024-03-28T22:36:06.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.024+0000 7ff042000640 1 -- 172.21.15.43:0/292835630 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff03c0db070 con 0x55a53db282a0 2024-03-28T22:36:06.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.024+0000 7ff042000640 1 -- 172.21.15.43:0/292835630 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03c099a80 con 0x55a53db282a0 2024-03-28T22:36:06.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.024+0000 7ff055800640 1 --2- 172.21.15.43:0/2435643444 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff0380a1e00 0x7ff0380a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.025+0000 7ff04d600640 1 -- 172.21.15.43:0/2435643444 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7ff03c0cdc90 con 0x7ff050104a60 2024-03-28T22:36:06.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.025+0000 7ff05a3e6740 1 -- 172.21.15.43:0/292835630 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a53db282a0 msgr2=0x55a53db2a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:06.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.025+0000 7ff05a3e6740 1 --2- 172.21.15.43:0/292835630 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a53db282a0 0x55a53db2a730 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff03c0d1e30 tx=0x7ff03c0d6760 comp rx=0 tx=0).stop 2024-03-28T22:36:06.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.025+0000 7ff05a3e6740 1 -- 172.21.15.43:0/292835630 shutdown_connections 2024-03-28T22:36:06.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.025+0000 7ff05a3e6740 1 --2- 172.21.15.43:0/292835630 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a53db282a0 0x55a53db2a730 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.025+0000 7ff05a3e6740 1 --2- 172.21.15.43:0/292835630 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a53db23130 0x55a53db27c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.025+0000 7ff05a3e6740 1 --2- 172.21.15.43:0/292835630 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a53db22740 0x55a53db22b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.025+0000 7ff05a3e6740 1 -- 172.21.15.43:0/292835630 >> 172.21.15.43:0/292835630 conn(0x55a53db1dcb0 msgr2=0x55a53db1e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:06.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.025+0000 7ff05a3e6740 1 -- 172.21.15.43:0/292835630 shutdown_connections 2024-03-28T22:36:06.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.026+0000 7ff055800640 1 --2- 172.21.15.43:0/2435643444 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff0380a1e00 0x7ff0380a42c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff048004ab0 tx=0x7ff048004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:06.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.026+0000 7ff05a3e6740 1 -- 172.21.15.43:0/292835630 wait complete. 2024-03-28T22:36:06.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.026+0000 7ff05a3e6740 1 -- start start 2024-03-28T22:36:06.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.026+0000 7ff05a3e6740 1 Processor -- start 2024-03-28T22:36:06.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.027+0000 7ff05a3e6740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a53db22740 0x55a53db2cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.027+0000 7ff05a3e6740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a53db23130 0x55a53db2d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.027+0000 7ff05a3e6740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a53db282a0 0x55a53db341c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.027+0000 7ff05a3e6740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55a53db2b250 con 0x55a53db22740 2024-03-28T22:36:06.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.027+0000 7ff05a3e6740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55a53da25180 con 0x55a53db23130 2024-03-28T22:36:06.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.027+0000 7ff05a3e6740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55a53db2b3d0 con 0x55a53db282a0 2024-03-28T22:36:06.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.027+0000 7ff05a3e6740 10 client.0 _ll_register_callbacks cb 0x55a53d79aac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:36:06.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.027+0000 7ff04fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a53db23130 0x55a53db2d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.027+0000 7ff054e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a53db22740 0x55a53db2cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.027+0000 7ff04fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a53db23130 0x55a53db2d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59644/0 (socket says 172.21.15.43:59644) 2024-03-28T22:36:06.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.027+0000 7ff04fe00640 1 -- 172.21.15.43:0/2196239282 learned_addr learned my addr 172.21.15.43:0/2196239282 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:06.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.027+0000 7ff055800640 1 --2- 172.21.15.43:0/2196239282 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a53db282a0 0x55a53db341c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.028+0000 7ff04fe00640 1 -- 172.21.15.43:0/2196239282 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a53db282a0 msgr2=0x55a53db341c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:06.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.028+0000 7ff04fe00640 1 --2- 172.21.15.43:0/2196239282 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a53db282a0 0x55a53db341c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.028+0000 7ff04fe00640 1 -- 172.21.15.43:0/2196239282 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a53db22740 msgr2=0x55a53db2cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:06.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.028+0000 7ff04fe00640 1 --2- 172.21.15.43:0/2196239282 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a53db22740 0x55a53db2cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.028+0000 7ff04fe00640 1 -- 172.21.15.43:0/2196239282 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a53d8656b0 con 0x55a53db23130 2024-03-28T22:36:06.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.028+0000 7ff04fe00640 1 --2- 172.21.15.43:0/2196239282 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a53db23130 0x55a53db2d180 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff03c0a28a0 tx=0x7ff03c009740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a821454fa021ee2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:06.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.028+0000 7ff036a00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:36:06.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.028+0000 7ff036a00640 1 -- 172.21.15.43:0/2196239282 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03c020070 con 0x55a53db23130 2024-03-28T22:36:06.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.028+0000 7ff036a00640 1 -- 172.21.15.43:0/2196239282 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7ff03c098930 con 0x55a53db23130 2024-03-28T22:36:06.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.028+0000 7ff05a3e6740 1 -- 172.21.15.43:0/2196239282 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff03c0db070 con 0x55a53db23130 2024-03-28T22:36:06.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.028+0000 7ff036a00640 1 -- 172.21.15.43:0/2196239282 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff03c099db0 con 0x55a53db23130 2024-03-28T22:36:06.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.029+0000 7ff05a3e6740 10 client.25205 fetch_fsmap learned FSMap version 28 2024-03-28T22:36:06.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.029+0000 7ff05a3e6740 1 -- 172.21.15.43:0/2196239282 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a53db37530 con 0x55a53db23130 2024-03-28T22:36:06.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.029+0000 7ff036a00640 1 -- 172.21.15.43:0/2196239282 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff03c0db070 con 0x55a53db23130 2024-03-28T22:36:06.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.029+0000 7ff036a00640 1 -- 172.21.15.43:0/2196239282 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff03c0d3050 con 0x55a53db23130 2024-03-28T22:36:06.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.029+0000 7ff05a3e6740 10 client.25205 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T22:36:06.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.029+0000 7ff05a3e6740 10 client.25205 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-28T22:36:06.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.029+0000 7ff05a3e6740 20 client.25205 populate_metadata read hostname 'smithi043' 2024-03-28T22:36:06.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.029+0000 7ff05a3e6740 1 --2- 172.21.15.43:0/2196239282 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55a53db386b0 0x55a53db3a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.029+0000 7ff05a3e6740 4 client.25205 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:36:06.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.029+0000 7ff05a3e6740 1 -- 172.21.15.43:0/2196239282 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a53da93f00 con 0x55a53db386b0 2024-03-28T22:36:06.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.030+0000 7ff054e00640 1 --2- 172.21.15.43:0/2196239282 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55a53db386b0 0x55a53db3a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.031+0000 7ff054e00640 1 --2- 172.21.15.43:0/2196239282 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55a53db386b0 0x55a53db3a8a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e3f319e7ca8aaf5f server_cookie=1aac4d00e136224d in_seq=0 out_seq=0 2024-03-28T22:36:06.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.031+0000 7ff036a00640 10 client.25205 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:36:06.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.032+0000 7ff036a00640 1 -- 172.21.15.43:0/2196239282 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a53da93f00 con 0x55a53db386b0 2024-03-28T22:36:06.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.032+0000 7ff036a00640 10 client.25205 handle_command_reply: tid=0 2024-03-28T22:36:06.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.032+0000 7ff05a3e6740 1 client.25205 shutdown 2024-03-28T22:36:06.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.032+0000 7ff05a3e6740 1 -- 172.21.15.43:0/2196239282 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a53db23130 msgr2=0x55a53db2d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:06.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.032+0000 7ff05a3e6740 1 --2- 172.21.15.43:0/2196239282 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a53db23130 0x55a53db2d180 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff03c0a28a0 tx=0x7ff03c009740 comp rx=0 tx=0).stop 2024-03-28T22:36:06.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.032+0000 7ff05a3e6740 1 -- 172.21.15.43:0/2196239282 shutdown_connections 2024-03-28T22:36:06.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.032+0000 7ff05a3e6740 1 --2- 172.21.15.43:0/2196239282 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55a53db386b0 0x55a53db3a8a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.032+0000 7ff05a3e6740 1 --2- 172.21.15.43:0/2196239282 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a53db23130 0x55a53db2d180 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.032+0000 7ff05a3e6740 1 --2- 172.21.15.43:0/2196239282 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a53db282a0 0x55a53db341c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.032+0000 7ff036a00640 0 client.25205 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:36:06.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.032+0000 7ff05a3e6740 1 --2- 172.21.15.43:0/2196239282 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a53db22740 0x55a53db2cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.032+0000 7ff05a3e6740 1 -- 172.21.15.43:0/2196239282 >> 172.21.15.43:0/2196239282 conn(0x55a53db1dcb0 msgr2=0x55a53db1cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:06.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.032+0000 7ff05a3e6740 1 -- 172.21.15.43:0/2196239282 shutdown_connections 2024-03-28T22:36:06.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.032+0000 7ff05a3e6740 1 -- 172.21.15.43:0/2196239282 wait complete. 2024-03-28T22:36:06.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.032+0000 7ff05a3e6740 20 client.25205 trim_cache size 0 max 16384 2024-03-28T22:36:06.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.053+0000 7ff05a3e6740 1 Processor -- start 2024-03-28T22:36:06.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.053+0000 7ff05a3e6740 1 -- start start 2024-03-28T22:36:06.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.053+0000 7ff05a3e6740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a53db22740 0x55a53db1e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.053+0000 7ff05a3e6740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a53db23130 0x55a53db34180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.053+0000 7ff05a3e6740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a53db282a0 0x55a53db2c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.053+0000 7ff05a3e6740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55a53db38be0 con 0x55a53db23130 2024-03-28T22:36:06.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.053+0000 7ff05a3e6740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55a53db392b0 con 0x55a53db282a0 2024-03-28T22:36:06.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.053+0000 7ff05a3e6740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55a53db39980 con 0x55a53db22740 2024-03-28T22:36:06.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.054+0000 7ff054e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a53db22740 0x55a53db1e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.054+0000 7ff054e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a53db22740 0x55a53db1e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35582/0 (socket says 172.21.15.43:35582) 2024-03-28T22:36:06.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.054+0000 7ff054e00640 1 -- 172.21.15.43:0/2095337394 learned_addr learned my addr 172.21.15.43:0/2095337394 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:06.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.054+0000 7ff04fe00640 1 --2- 172.21.15.43:0/2095337394 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a53db23130 0x55a53db34180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.054+0000 7ff055800640 1 --2- 172.21.15.43:0/2095337394 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a53db282a0 0x55a53db2c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.054+0000 7ff054e00640 1 -- 172.21.15.43:0/2095337394 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a53db282a0 msgr2=0x55a53db2c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:06.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.054+0000 7ff054e00640 1 --2- 172.21.15.43:0/2095337394 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a53db282a0 0x55a53db2c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.054+0000 7ff054e00640 1 -- 172.21.15.43:0/2095337394 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a53db23130 msgr2=0x55a53db34180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:06.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.054+0000 7ff054e00640 1 --2- 172.21.15.43:0/2095337394 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a53db23130 0x55a53db34180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.054+0000 7ff054e00640 1 -- 172.21.15.43:0/2095337394 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a53db23950 con 0x55a53db22740 2024-03-28T22:36:06.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.054+0000 7ff054e00640 1 --2- 172.21.15.43:0/2095337394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a53db22740 0x55a53db1e090 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff044009c00 tx=0x7ff044009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d37775d0c1d56ad1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:06.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.054+0000 7ff041600640 1 -- 172.21.15.43:0/2095337394 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff044023070 con 0x55a53db22740 2024-03-28T22:36:06.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.055+0000 7ff041600640 1 -- 172.21.15.43:0/2095337394 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff04401d630 con 0x55a53db22740 2024-03-28T22:36:06.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.055+0000 7ff041600640 1 -- 172.21.15.43:0/2095337394 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff044003850 con 0x55a53db22740 2024-03-28T22:36:06.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.055+0000 7ff05a3e6740 1 -- 172.21.15.43:0/2095337394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a53db22740 msgr2=0x55a53db1e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:06.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.055+0000 7ff05a3e6740 1 --2- 172.21.15.43:0/2095337394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a53db22740 0x55a53db1e090 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff044009c00 tx=0x7ff044009f90 comp rx=0 tx=0).stop 2024-03-28T22:36:06.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.055+0000 7ff05a3e6740 1 -- 172.21.15.43:0/2095337394 shutdown_connections 2024-03-28T22:36:06.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.055+0000 7ff05a3e6740 1 --2- 172.21.15.43:0/2095337394 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a53db282a0 0x55a53db2c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.055+0000 7ff05a3e6740 1 --2- 172.21.15.43:0/2095337394 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a53db23130 0x55a53db34180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.055+0000 7ff05a3e6740 1 --2- 172.21.15.43:0/2095337394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a53db22740 0x55a53db1e090 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.055+0000 7ff05a3e6740 1 -- 172.21.15.43:0/2095337394 >> 172.21.15.43:0/2095337394 conn(0x55a53db1dcb0 msgr2=0x55a53db23bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:06.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.055+0000 7ff05a3e6740 1 -- 172.21.15.43:0/2095337394 shutdown_connections 2024-03-28T22:36:06.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.055+0000 7ff05a3e6740 1 -- 172.21.15.43:0/2095337394 wait complete. 2024-03-28T22:36:06.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.056+0000 7ff05a3e6740 1 -- start start 2024-03-28T22:36:06.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.056+0000 7ff05a3e6740 1 Processor -- start 2024-03-28T22:36:06.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.056+0000 7ff05a3e6740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a53db22740 0x55a53db303d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.056+0000 7ff05a3e6740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a53db23130 0x55a53db30910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.056+0000 7ff05a3e6740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a53db282a0 0x55a53db33430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.056+0000 7ff05a3e6740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55a53db3a720 con 0x55a53db282a0 2024-03-28T22:36:06.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.056+0000 7ff05a3e6740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55a53db3a050 con 0x55a53db23130 2024-03-28T22:36:06.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.056+0000 7ff05a3e6740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55a53db3adf0 con 0x55a53db22740 2024-03-28T22:36:06.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.056+0000 7ff05a3e6740 10 client.0 _ll_register_callbacks cb 0x7ff048009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:36:06.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.057+0000 7ff054e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a53db22740 0x55a53db303d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.057+0000 7ff054e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a53db22740 0x55a53db303d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35596/0 (socket says 172.21.15.43:35596) 2024-03-28T22:36:06.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.057+0000 7ff054e00640 1 -- 172.21.15.43:0/1467751207 learned_addr learned my addr 172.21.15.43:0/1467751207 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:06.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.057+0000 7ff04fe00640 1 --2- 172.21.15.43:0/1467751207 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a53db23130 0x55a53db30910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.057+0000 7ff055800640 1 --2- 172.21.15.43:0/1467751207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a53db282a0 0x55a53db33430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.057+0000 7ff04fe00640 1 -- 172.21.15.43:0/1467751207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a53db22740 msgr2=0x55a53db303d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:06.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.057+0000 7ff04fe00640 1 --2- 172.21.15.43:0/1467751207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a53db22740 0x55a53db303d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.057+0000 7ff04fe00640 1 -- 172.21.15.43:0/1467751207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a53db282a0 msgr2=0x55a53db33430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:06.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.057+0000 7ff04fe00640 1 --2- 172.21.15.43:0/1467751207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a53db282a0 0x55a53db33430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.057+0000 7ff04fe00640 1 -- 172.21.15.43:0/1467751207 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a53db33a70 con 0x55a53db23130 2024-03-28T22:36:06.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.057+0000 7ff054e00640 1 --2- 172.21.15.43:0/1467751207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a53db22740 0x55a53db303d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:06.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.057+0000 7ff04fe00640 1 --2- 172.21.15.43:0/1467751207 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a53db23130 0x55a53db30910 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff03c096a90 tx=0x7ff03c005bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc6d4a40a49a2c62 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:06.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.057+0000 7ff036a00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:36:06.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.058+0000 7ff036a00640 1 -- 172.21.15.43:0/1467751207 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03c020070 con 0x55a53db23130 2024-03-28T22:36:06.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.058+0000 7ff036a00640 1 -- 172.21.15.43:0/1467751207 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7ff03c098930 con 0x55a53db23130 2024-03-28T22:36:06.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.058+0000 7ff05a3e6740 1 -- 172.21.15.43:0/1467751207 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff04401d630 con 0x55a53db23130 2024-03-28T22:36:06.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.058+0000 7ff036a00640 1 -- 172.21.15.43:0/1467751207 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff03c099f90 con 0x55a53db23130 2024-03-28T22:36:06.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.058+0000 7ff05a3e6740 10 client.25211 fetch_fsmap learned FSMap version 28 2024-03-28T22:36:06.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.058+0000 7ff05a3e6740 1 -- 172.21.15.43:0/1467751207 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a53db33a70 con 0x55a53db23130 2024-03-28T22:36:06.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.058+0000 7ff036a00640 1 -- 172.21.15.43:0/1467751207 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff04401d630 con 0x55a53db23130 2024-03-28T22:36:06.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.058+0000 7ff036a00640 1 -- 172.21.15.43:0/1467751207 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff03c0d3070 con 0x55a53db23130 2024-03-28T22:36:06.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.058+0000 7ff05a3e6740 10 client.25211 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T22:36:06.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.058+0000 7ff05a3e6740 10 client.25211 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-28T22:36:06.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.058+0000 7ff05a3e6740 20 client.25211 populate_metadata read hostname 'smithi043' 2024-03-28T22:36:06.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.058+0000 7ff05a3e6740 1 --2- 172.21.15.43:0/1467751207 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55a53dba3fa0 0x55a53dba2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.058+0000 7ff05a3e6740 4 client.25211 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T22:36:06.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.058+0000 7ff05a3e6740 1 -- 172.21.15.43:0/1467751207 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "status"}) -- 0x55a53da8e250 con 0x55a53dba3fa0 2024-03-28T22:36:06.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.059+0000 7ff054e00640 1 --2- 172.21.15.43:0/1467751207 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55a53dba3fa0 0x55a53dba2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.061+0000 7ff054e00640 1 --2- 172.21.15.43:0/1467751207 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55a53dba3fa0 0x55a53dba2cb0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=88fdbffdf29ad17c server_cookie=9004485cd246e127 in_seq=0 out_seq=0 2024-03-28T22:36:06.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.061+0000 7ff036a00640 10 client.25211 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:36:06.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.061+0000 7ff036a00640 1 -- 172.21.15.43:0/1467751207 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55a53da8e250 con 0x55a53dba3fa0 2024-03-28T22:36:06.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.061+0000 7ff036a00640 10 client.25211 handle_command_reply: tid=0 2024-03-28T22:36:06.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.061+0000 7ff05a3e6740 1 client.25211 shutdown 2024-03-28T22:36:06.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.061+0000 7ff05a3e6740 1 -- 172.21.15.43:0/1467751207 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a53db23130 msgr2=0x55a53db30910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:06.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.061+0000 7ff05a3e6740 1 --2- 172.21.15.43:0/1467751207 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a53db23130 0x55a53db30910 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff03c096a90 tx=0x7ff03c005bb0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.061+0000 7ff05a3e6740 1 -- 172.21.15.43:0/1467751207 shutdown_connections 2024-03-28T22:36:06.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.061+0000 7ff05a3e6740 1 --2- 172.21.15.43:0/1467751207 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55a53dba3fa0 0x55a53dba2cb0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.061+0000 7ff05a3e6740 1 --2- 172.21.15.43:0/1467751207 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a53db23130 0x55a53db30910 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.061+0000 7ff05a3e6740 1 --2- 172.21.15.43:0/1467751207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a53db282a0 0x55a53db33430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.061+0000 7ff036a00640 0 client.25211 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:36:06.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.061+0000 7ff05a3e6740 1 --2- 172.21.15.43:0/1467751207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a53db22740 0x55a53db303d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.061+0000 7ff05a3e6740 1 -- 172.21.15.43:0/1467751207 >> 172.21.15.43:0/1467751207 conn(0x55a53db1dcb0 msgr2=0x55a53db1e090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:06.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.061+0000 7ff05a3e6740 1 -- 172.21.15.43:0/1467751207 shutdown_connections 2024-03-28T22:36:06.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.061+0000 7ff05a3e6740 1 -- 172.21.15.43:0/1467751207 wait complete. 2024-03-28T22:36:06.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.061+0000 7ff05a3e6740 20 client.25211 trim_cache size 0 max 16384 2024-03-28T22:36:06.069 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:36:06.069 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_fsid": "c2276428-ed51-11ee-95d0-87774f69a715", 2024-03-28T22:36:06.069 INFO:teuthology.orchestra.run.smithi043.stdout: "whoami": 2, 2024-03-28T22:36:06.069 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 24535, 2024-03-28T22:36:06.069 INFO:teuthology.orchestra.run.smithi043.stdout: "want_state": "up:active", 2024-03-28T22:36:06.069 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "up:active", 2024-03-28T22:36:06.069 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs", 2024-03-28T22:36:06.070 INFO:teuthology.orchestra.run.smithi043.stdout: "rank_uptime": 122.918471563, 2024-03-28T22:36:06.070 INFO:teuthology.orchestra.run.smithi043.stdout: "mdsmap_epoch": 28, 2024-03-28T22:36:06.070 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch": 96, 2024-03-28T22:36:06.070 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch_barrier": 92, 2024-03-28T22:36:06.070 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 180.357058679 2024-03-28T22:36:06.070 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:36:06.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.062+0000 7ff056200640 1 -- 172.21.15.43:0/2435643444 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff0380a1e00 msgr2=0x7ff0380a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:06.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.062+0000 7ff056200640 1 --2- 172.21.15.43:0/2435643444 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff0380a1e00 0x7ff0380a42c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff048004ab0 tx=0x7ff048004a40 comp rx=0 tx=0).stop 2024-03-28T22:36:06.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.062+0000 7ff056200640 1 -- 172.21.15.43:0/2435643444 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff050104a60 msgr2=0x7ff05019a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:06.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.062+0000 7ff056200640 1 --2- 172.21.15.43:0/2435643444 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff050104a60 0x7ff05019a040 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff03c00d950 tx=0x7ff03c00dd50 comp rx=0 tx=0).stop 2024-03-28T22:36:06.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.062+0000 7ff056200640 1 -- 172.21.15.43:0/2435643444 shutdown_connections 2024-03-28T22:36:06.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.062+0000 7ff056200640 1 --2- 172.21.15.43:0/2435643444 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7ff0380a1e00 0x7ff0380a42c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.062+0000 7ff056200640 1 --2- 172.21.15.43:0/2435643444 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff050105420 0x7ff0501a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.062+0000 7ff056200640 1 --2- 172.21.15.43:0/2435643444 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff050104a60 0x7ff05019a040 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.062+0000 7ff056200640 1 --2- 172.21.15.43:0/2435643444 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff050103870 0x7ff050199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.062+0000 7ff056200640 1 -- 172.21.15.43:0/2435643444 >> 172.21.15.43:0/2435643444 conn(0x7ff0500fee00 msgr2=0x7ff050107d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:06.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.062+0000 7ff056200640 1 -- 172.21.15.43:0/2435643444 shutdown_connections 2024-03-28T22:36:06.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.062+0000 7ff056200640 1 -- 172.21.15.43:0/2435643444 wait complete. 2024-03-28T22:36:06.080 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-28T22:36:06.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.300+0000 7f9763e00640 1 -- 172.21.15.43:0/1363982789 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f97641020b0 msgr2=0x7f97641044a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:06.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.300+0000 7f9763e00640 1 --2- 172.21.15.43:0/1363982789 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f97641020b0 0x7f97641044a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9754009ca0 tx=0x7f975402c100 comp rx=0 tx=0).stop 2024-03-28T22:36:06.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.300+0000 7f9763e00640 1 -- 172.21.15.43:0/1363982789 shutdown_connections 2024-03-28T22:36:06.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.300+0000 7f9763e00640 1 --2- 172.21.15.43:0/1363982789 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9764107310 0x7f9764109760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.300+0000 7f9763e00640 1 --2- 172.21.15.43:0/1363982789 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f97641049e0 0x7f9764106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.300+0000 7f9763e00640 1 --2- 172.21.15.43:0/1363982789 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f97641020b0 0x7f97641044a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.300+0000 7f9763e00640 1 -- 172.21.15.43:0/1363982789 >> 172.21.15.43:0/1363982789 conn(0x7f97640fbac0 msgr2=0x7f97640fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:06.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.301+0000 7f9763e00640 1 -- 172.21.15.43:0/1363982789 shutdown_connections 2024-03-28T22:36:06.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.301+0000 7f9763e00640 1 -- 172.21.15.43:0/1363982789 wait complete. 2024-03-28T22:36:06.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.302+0000 7f9763e00640 1 Processor -- start 2024-03-28T22:36:06.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.302+0000 7f9763e00640 1 -- start start 2024-03-28T22:36:06.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.302+0000 7f9763e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f97641020b0 0x7f9764199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.302+0000 7f9763e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97641049e0 0x7f976419a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.302+0000 7f9763e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9764107310 0x7f97641a1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.302+0000 7f9763e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f976410d2a0 con 0x7f9764107310 2024-03-28T22:36:06.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.302+0000 7f9763e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f976410d120 con 0x7f97641020b0 2024-03-28T22:36:06.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.302+0000 7f9763e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f976410d420 con 0x7f97641049e0 2024-03-28T22:36:06.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.302+0000 7f9762a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f97641020b0 0x7f9764199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.302+0000 7f9762000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97641049e0 0x7f976419a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.302+0000 7f9762a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f97641020b0 0x7f9764199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59674/0 (socket says 172.21.15.43:59674) 2024-03-28T22:36:06.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.302+0000 7f9762a00640 1 -- 172.21.15.43:0/514567966 learned_addr learned my addr 172.21.15.43:0/514567966 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:06.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.302+0000 7f9763400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9764107310 0x7f97641a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.303+0000 7f9762a00640 1 -- 172.21.15.43:0/514567966 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97641049e0 msgr2=0x7f976419a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:06.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.303+0000 7f9762a00640 1 --2- 172.21.15.43:0/514567966 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97641049e0 0x7f976419a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.303+0000 7f9762a00640 1 -- 172.21.15.43:0/514567966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9764107310 msgr2=0x7f97641a1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:06.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.303+0000 7f9762a00640 1 --2- 172.21.15.43:0/514567966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9764107310 0x7f97641a1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.303+0000 7f9762a00640 1 -- 172.21.15.43:0/514567966 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97641a16f0 con 0x7f97641020b0 2024-03-28T22:36:06.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.303+0000 7f9762a00640 1 --2- 172.21.15.43:0/514567966 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f97641020b0 0x7f9764199ad0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f97540325f0 tx=0x7f9754035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:06.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.303+0000 7f975b400640 1 -- 172.21.15.43:0/514567966 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9754039040 con 0x7f97641020b0 2024-03-28T22:36:06.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.303+0000 7f9763e00640 1 -- 172.21.15.43:0/514567966 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97641a1980 con 0x7f97641020b0 2024-03-28T22:36:06.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.303+0000 7f975b400640 1 -- 172.21.15.43:0/514567966 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9754044070 con 0x7f97641020b0 2024-03-28T22:36:06.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.303+0000 7f975b400640 1 -- 172.21.15.43:0/514567966 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9754004610 con 0x7f97641020b0 2024-03-28T22:36:06.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.303+0000 7f9763e00640 1 -- 172.21.15.43:0/514567966 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97641a1e90 con 0x7f97641020b0 2024-03-28T22:36:06.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.306+0000 7f976c7b3740 1 Processor -- start 2024-03-28T22:36:06.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.307+0000 7f975b400640 1 -- 172.21.15.43:0/514567966 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f9754003970 con 0x7f97641020b0 2024-03-28T22:36:06.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.307+0000 7f976c7b3740 1 -- start start 2024-03-28T22:36:06.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.307+0000 7f976c7b3740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56535391e740 0x56535391eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.307+0000 7f976c7b3740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56535391f130 0x565353923c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.307+0000 7f976c7b3740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5653539242a0 0x565353926730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.307+0000 7f976c7b3740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5653538829c0 con 0x56535391e740 2024-03-28T22:36:06.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.307+0000 7f976c7b3740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x56535382bc30 con 0x56535391f130 2024-03-28T22:36:06.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.307+0000 7f976c7b3740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x56535387ea60 con 0x5653539242a0 2024-03-28T22:36:06.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.307+0000 7f9762000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5653539242a0 0x565353926730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.307+0000 7f9763400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56535391e740 0x56535391eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.307+0000 7f9762a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56535391f130 0x565353923c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.307+0000 7f9762000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5653539242a0 0x565353926730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35620/0 (socket says 172.21.15.43:35620) 2024-03-28T22:36:06.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.307+0000 7f9762000640 1 -- 172.21.15.43:0/241802406 learned_addr learned my addr 172.21.15.43:0/241802406 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:06.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.308+0000 7f9762000640 1 -- 172.21.15.43:0/241802406 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56535391f130 msgr2=0x565353923c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:06.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.308+0000 7f9762000640 1 --2- 172.21.15.43:0/241802406 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56535391f130 0x565353923c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.308+0000 7f9762000640 1 -- 172.21.15.43:0/241802406 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56535391e740 msgr2=0x56535391eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:06.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.308+0000 7f9762000640 1 --2- 172.21.15.43:0/241802406 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56535391e740 0x56535391eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.308+0000 7f9762000640 1 -- 172.21.15.43:0/241802406 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653535972d0 con 0x5653539242a0 2024-03-28T22:36:06.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.308+0000 7f9762000640 1 --2- 172.21.15.43:0/241802406 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5653539242a0 0x565353926730 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f974c00bc70 tx=0x7f974c0107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef6f0c70897af1c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:06.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.308+0000 7f9753e00640 1 -- 172.21.15.43:0/241802406 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974c005220 con 0x5653539242a0 2024-03-28T22:36:06.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.308+0000 7f9753e00640 1 -- 172.21.15.43:0/241802406 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f974c00be20 con 0x5653539242a0 2024-03-28T22:36:06.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.308+0000 7f9753e00640 1 -- 172.21.15.43:0/241802406 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974c011750 con 0x5653539242a0 2024-03-28T22:36:06.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.308+0000 7f975b400640 1 --2- 172.21.15.43:0/514567966 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f97480a1e00 0x7f97480a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.308+0000 7f975b400640 1 -- 172.21.15.43:0/514567966 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f97540f07f0 con 0x7f97641020b0 2024-03-28T22:36:06.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.309+0000 7f9763400640 1 --2- 172.21.15.43:0/514567966 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f97480a1e00 0x7f97480a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.309+0000 7f976c7b3740 1 -- 172.21.15.43:0/241802406 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5653539242a0 msgr2=0x565353926730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:06.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.309+0000 7f976c7b3740 1 --2- 172.21.15.43:0/241802406 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5653539242a0 0x565353926730 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f974c00bc70 tx=0x7f974c0107a0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.309+0000 7f976c7b3740 1 -- 172.21.15.43:0/241802406 shutdown_connections 2024-03-28T22:36:06.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.309+0000 7f976c7b3740 1 --2- 172.21.15.43:0/241802406 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56535391f130 0x565353923c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.309+0000 7f976c7b3740 1 --2- 172.21.15.43:0/241802406 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5653539242a0 0x565353926730 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.309+0000 7f976c7b3740 1 --2- 172.21.15.43:0/241802406 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56535391e740 0x56535391eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.309+0000 7f976c7b3740 1 -- 172.21.15.43:0/241802406 >> 172.21.15.43:0/241802406 conn(0x565353919cb0 msgr2=0x56535391a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:06.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.309+0000 7f976c7b3740 1 -- 172.21.15.43:0/241802406 shutdown_connections 2024-03-28T22:36:06.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.309+0000 7f976c7b3740 1 -- 172.21.15.43:0/241802406 wait complete. 2024-03-28T22:36:06.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.309+0000 7f9763400640 1 --2- 172.21.15.43:0/514567966 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f97480a1e00 0x7f97480a42c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x56535391f990 tx=0x7f975c00a250 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:06.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.310+0000 7f976c7b3740 1 -- start start 2024-03-28T22:36:06.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.310+0000 7f976c7b3740 1 Processor -- start 2024-03-28T22:36:06.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.310+0000 7f976c7b3740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56535391e740 0x565353928d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.311+0000 7f976c7b3740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56535391f130 0x565353929290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.311+0000 7f976c7b3740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653539242a0 0x5653539302d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.311+0000 7f976c7b3740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x565353927250 con 0x5653539242a0 2024-03-28T22:36:06.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.311+0000 7f976c7b3740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x565353821180 con 0x56535391e740 2024-03-28T22:36:06.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.311+0000 7f976c7b3740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5653539273d0 con 0x56535391f130 2024-03-28T22:36:06.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.311+0000 7f976c7b3740 10 client.0 _ll_register_callbacks cb 0x565353596ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:36:06.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.311+0000 7f9762a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56535391e740 0x565353928d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.311+0000 7f9762a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56535391e740 0x565353928d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59698/0 (socket says 172.21.15.43:59698) 2024-03-28T22:36:06.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.311+0000 7f9762a00640 1 -- 172.21.15.43:0/1971951872 learned_addr learned my addr 172.21.15.43:0/1971951872 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:06.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.311+0000 7f9762000640 1 --2- 172.21.15.43:0/1971951872 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56535391f130 0x565353929290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.311+0000 7f9763400640 1 --2- 172.21.15.43:0/1971951872 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653539242a0 0x5653539302d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.311+0000 7f9762a00640 1 -- 172.21.15.43:0/1971951872 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56535391f130 msgr2=0x565353929290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:06.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.311+0000 7f9762a00640 1 --2- 172.21.15.43:0/1971951872 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56535391f130 0x565353929290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.311+0000 7f9762a00640 1 -- 172.21.15.43:0/1971951872 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653539242a0 msgr2=0x5653539302d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:06.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.311+0000 7f9762a00640 1 --2- 172.21.15.43:0/1971951872 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653539242a0 0x5653539302d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.311+0000 7f9762a00640 1 -- 172.21.15.43:0/1971951872 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653536616b0 con 0x56535391e740 2024-03-28T22:36:06.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.311+0000 7f9762a00640 1 --2- 172.21.15.43:0/1971951872 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56535391e740 0x565353928d50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9754000f80 tx=0x7f9754004140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe346553cb031d2b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:06.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.311+0000 7f9750c00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:36:06.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.312+0000 7f9750c00640 1 -- 172.21.15.43:0/1971951872 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97540bcf00 con 0x56535391e740 2024-03-28T22:36:06.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.312+0000 7f9750c00640 1 -- 172.21.15.43:0/1971951872 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f9754032920 con 0x56535391e740 2024-03-28T22:36:06.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.312+0000 7f976c7b3740 1 -- 172.21.15.43:0/1971951872 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f974c00be20 con 0x56535391e740 2024-03-28T22:36:06.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.312+0000 7f9750c00640 1 -- 172.21.15.43:0/1971951872 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f97540bca90 con 0x56535391e740 2024-03-28T22:36:06.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.312+0000 7f976c7b3740 10 client.25223 fetch_fsmap learned FSMap version 28 2024-03-28T22:36:06.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.312+0000 7f976c7b3740 1 -- 172.21.15.43:0/1971951872 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x565353933640 con 0x56535391e740 2024-03-28T22:36:06.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.312+0000 7f9750c00640 1 -- 172.21.15.43:0/1971951872 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f974c00be20 con 0x56535391e740 2024-03-28T22:36:06.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.312+0000 7f9750c00640 1 -- 172.21.15.43:0/1971951872 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f975403c030 con 0x56535391e740 2024-03-28T22:36:06.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.312+0000 7f976c7b3740 10 client.25223 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T22:36:06.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.312+0000 7f976c7b3740 10 client.25223 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T22:36:06.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.312+0000 7f976c7b3740 20 client.25223 populate_metadata read hostname 'smithi043' 2024-03-28T22:36:06.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.312+0000 7f976c7b3740 1 --2- 172.21.15.43:0/1971951872 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x565353935ca0 0x565353937110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.313+0000 7f976c7b3740 4 client.25223 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:36:06.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.313+0000 7f976c7b3740 1 -- 172.21.15.43:0/1971951872 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56535388ff00 con 0x565353935ca0 2024-03-28T22:36:06.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.313+0000 7f9762000640 1 --2- 172.21.15.43:0/1971951872 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x565353935ca0 0x565353937110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.313+0000 7f9762000640 1 --2- 172.21.15.43:0/1971951872 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x565353935ca0 0x565353937110 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f5fca36162a5ed24 server_cookie=a0c7201f5f01d811 in_seq=0 out_seq=0 2024-03-28T22:36:06.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.313+0000 7f9750c00640 10 client.25223 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:36:06.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.315+0000 7f9750c00640 1 -- 172.21.15.43:0/1971951872 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56535388ff00 con 0x565353935ca0 2024-03-28T22:36:06.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.315+0000 7f9750c00640 10 client.25223 handle_command_reply: tid=0 2024-03-28T22:36:06.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.315+0000 7f976c7b3740 1 client.25223 shutdown 2024-03-28T22:36:06.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.315+0000 7f976c7b3740 1 -- 172.21.15.43:0/1971951872 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56535391e740 msgr2=0x565353928d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:06.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.315+0000 7f976c7b3740 1 --2- 172.21.15.43:0/1971951872 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56535391e740 0x565353928d50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9754000f80 tx=0x7f9754004140 comp rx=0 tx=0).stop 2024-03-28T22:36:06.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.315+0000 7f976c7b3740 1 -- 172.21.15.43:0/1971951872 shutdown_connections 2024-03-28T22:36:06.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.315+0000 7f976c7b3740 1 --2- 172.21.15.43:0/1971951872 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x565353935ca0 0x565353937110 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.315+0000 7f976c7b3740 1 --2- 172.21.15.43:0/1971951872 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653539242a0 0x5653539302d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.315+0000 7f976c7b3740 1 --2- 172.21.15.43:0/1971951872 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56535391f130 0x565353929290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.315+0000 7f976c7b3740 1 --2- 172.21.15.43:0/1971951872 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56535391e740 0x565353928d50 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.315+0000 7f976c7b3740 1 -- 172.21.15.43:0/1971951872 >> 172.21.15.43:0/1971951872 conn(0x565353919cb0 msgr2=0x565353918df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:06.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.315+0000 7f9750c00640 0 client.25223 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:36:06.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.315+0000 7f976c7b3740 1 -- 172.21.15.43:0/1971951872 shutdown_connections 2024-03-28T22:36:06.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.315+0000 7f976c7b3740 1 -- 172.21.15.43:0/1971951872 wait complete. 2024-03-28T22:36:06.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.316+0000 7f976c7b3740 20 client.25223 trim_cache size 0 max 16384 2024-03-28T22:36:06.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.335+0000 7f976c7b3740 1 Processor -- start 2024-03-28T22:36:06.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.336+0000 7f976c7b3740 1 -- start start 2024-03-28T22:36:06.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.336+0000 7f976c7b3740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56535391e740 0x56535391a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.336+0000 7f976c7b3740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56535391f130 0x565353930260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.336+0000 7f976c7b3740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653539242a0 0x5653539277a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.336+0000 7f976c7b3740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x565353935450 con 0x56535391f130 2024-03-28T22:36:06.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.336+0000 7f976c7b3740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x565353935b20 con 0x5653539242a0 2024-03-28T22:36:06.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.336+0000 7f976c7b3740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5653539361f0 con 0x56535391e740 2024-03-28T22:36:06.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.336+0000 7f9763400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653539242a0 0x5653539277a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.336+0000 7f9762a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56535391e740 0x56535391a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.336+0000 7f9763400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653539242a0 0x5653539277a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59706/0 (socket says 172.21.15.43:59706) 2024-03-28T22:36:06.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.336+0000 7f9763400640 1 -- 172.21.15.43:0/2058560878 learned_addr learned my addr 172.21.15.43:0/2058560878 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:06.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.336+0000 7f9762000640 1 --2- 172.21.15.43:0/2058560878 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56535391f130 0x565353930260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.337+0000 7f9762a00640 1 -- 172.21.15.43:0/2058560878 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653539242a0 msgr2=0x5653539277a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:06.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.337+0000 7f9762a00640 1 --2- 172.21.15.43:0/2058560878 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653539242a0 0x5653539277a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.337+0000 7f9762a00640 1 -- 172.21.15.43:0/2058560878 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56535391f130 msgr2=0x565353930260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:06.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.337+0000 7f9762a00640 1 --2- 172.21.15.43:0/2058560878 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56535391f130 0x565353930260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.337+0000 7f9762a00640 1 -- 172.21.15.43:0/2058560878 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56535391f510 con 0x56535391e740 2024-03-28T22:36:06.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.337+0000 7f9762000640 1 --2- 172.21.15.43:0/2058560878 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56535391f130 0x565353930260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:06.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.337+0000 7f9762a00640 1 --2- 172.21.15.43:0/2058560878 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56535391e740 0x56535391a090 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f97540b9590 tx=0x7f97540bb640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f62e45f4573024d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:06.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.337+0000 7f9753400640 1 -- 172.21.15.43:0/2058560878 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97540bcc70 con 0x56535391e740 2024-03-28T22:36:06.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.337+0000 7f9753400640 1 -- 172.21.15.43:0/2058560878 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97540d2020 con 0x56535391e740 2024-03-28T22:36:06.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.337+0000 7f9753400640 1 -- 172.21.15.43:0/2058560878 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97540f6250 con 0x56535391e740 2024-03-28T22:36:06.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.338+0000 7f976c7b3740 1 -- 172.21.15.43:0/2058560878 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56535391e740 msgr2=0x56535391a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:06.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.338+0000 7f976c7b3740 1 --2- 172.21.15.43:0/2058560878 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56535391e740 0x56535391a090 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f97540b9590 tx=0x7f97540bb640 comp rx=0 tx=0).stop 2024-03-28T22:36:06.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.338+0000 7f976c7b3740 1 -- 172.21.15.43:0/2058560878 shutdown_connections 2024-03-28T22:36:06.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.338+0000 7f976c7b3740 1 --2- 172.21.15.43:0/2058560878 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653539242a0 0x5653539277a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.338+0000 7f976c7b3740 1 --2- 172.21.15.43:0/2058560878 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56535391f130 0x565353930260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.338+0000 7f976c7b3740 1 --2- 172.21.15.43:0/2058560878 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56535391e740 0x56535391a090 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.338+0000 7f976c7b3740 1 -- 172.21.15.43:0/2058560878 >> 172.21.15.43:0/2058560878 conn(0x565353919cb0 msgr2=0x565353928960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:06.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.338+0000 7f976c7b3740 1 -- 172.21.15.43:0/2058560878 shutdown_connections 2024-03-28T22:36:06.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.338+0000 7f976c7b3740 1 -- 172.21.15.43:0/2058560878 wait complete. 2024-03-28T22:36:06.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.338+0000 7f976c7b3740 1 -- start start 2024-03-28T22:36:06.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.339+0000 7f976c7b3740 1 Processor -- start 2024-03-28T22:36:06.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.339+0000 7f976c7b3740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56535391e740 0x56535392bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.339+0000 7f976c7b3740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56535391f130 0x56535392c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.339+0000 7f976c7b3740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653539242a0 0x56535392f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.339+0000 7f976c7b3740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x565353936f90 con 0x5653539242a0 2024-03-28T22:36:06.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.339+0000 7f976c7b3740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5653539368c0 con 0x56535391f130 2024-03-28T22:36:06.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.339+0000 7f976c7b3740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x565353937660 con 0x56535391e740 2024-03-28T22:36:06.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.339+0000 7f976c7b3740 10 client.0 _ll_register_callbacks cb 0x7f9754004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:36:06.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.339+0000 7f9762000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56535391f130 0x56535392c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.339+0000 7f9762000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56535391f130 0x56535392c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59716/0 (socket says 172.21.15.43:59716) 2024-03-28T22:36:06.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.339+0000 7f9762000640 1 -- 172.21.15.43:0/2764796848 learned_addr learned my addr 172.21.15.43:0/2764796848 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:06.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.339+0000 7f9762a00640 1 --2- 172.21.15.43:0/2764796848 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56535391e740 0x56535392bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.339+0000 7f9763400640 1 --2- 172.21.15.43:0/2764796848 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653539242a0 0x56535392f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.340+0000 7f9762000640 1 -- 172.21.15.43:0/2764796848 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56535391e740 msgr2=0x56535392bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:06.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.340+0000 7f9762000640 1 --2- 172.21.15.43:0/2764796848 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56535391e740 0x56535392bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.340+0000 7f9762000640 1 -- 172.21.15.43:0/2764796848 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653539242a0 msgr2=0x56535392f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:06.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.340+0000 7f9762000640 1 --2- 172.21.15.43:0/2764796848 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653539242a0 0x56535392f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.340+0000 7f9762000640 1 -- 172.21.15.43:0/2764796848 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56535392fab0 con 0x56535391f130 2024-03-28T22:36:06.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.340+0000 7f9763400640 1 --2- 172.21.15.43:0/2764796848 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653539242a0 0x56535392f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:36:06.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.340+0000 7f9762000640 1 --2- 172.21.15.43:0/2764796848 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56535391f130 0x56535392c510 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x56535391ac20 tx=0x7f974c005cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af36505cf0d592cc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:06.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.340+0000 7f9750c00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:36:06.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.340+0000 7f9750c00640 1 -- 172.21.15.43:0/2764796848 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974c0053a0 con 0x56535391f130 2024-03-28T22:36:06.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.340+0000 7f976c7b3740 1 -- 172.21.15.43:0/2764796848 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97540d2020 con 0x56535391f130 2024-03-28T22:36:06.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.340+0000 7f9750c00640 1 -- 172.21.15.43:0/2764796848 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f974c005540 con 0x56535391f130 2024-03-28T22:36:06.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.341+0000 7f9750c00640 1 -- 172.21.15.43:0/2764796848 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f974c037050 con 0x56535391f130 2024-03-28T22:36:06.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.341+0000 7f976c7b3740 10 client.25229 fetch_fsmap learned FSMap version 28 2024-03-28T22:36:06.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.341+0000 7f976c7b3740 1 -- 172.21.15.43:0/2764796848 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56535399c050 con 0x56535391f130 2024-03-28T22:36:06.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.341+0000 7f9750c00640 1 -- 172.21.15.43:0/2764796848 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97540d2020 con 0x56535391f130 2024-03-28T22:36:06.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.341+0000 7f9750c00640 1 -- 172.21.15.43:0/2764796848 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f974c003870 con 0x56535391f130 2024-03-28T22:36:06.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.341+0000 7f976c7b3740 10 client.25229 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T22:36:06.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.341+0000 7f976c7b3740 10 client.25229 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T22:36:06.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.341+0000 7f976c7b3740 20 client.25229 populate_metadata read hostname 'smithi043' 2024-03-28T22:36:06.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.341+0000 7f976c7b3740 1 --2- 172.21.15.43:0/2764796848 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x5653539a0ad0 0x56535399f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.341+0000 7f976c7b3740 4 client.25229 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T22:36:06.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.341+0000 7f976c7b3740 1 -- 172.21.15.43:0/2764796848 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "status"}) -- 0x56535388a250 con 0x5653539a0ad0 2024-03-28T22:36:06.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.342+0000 7f9762a00640 1 --2- 172.21.15.43:0/2764796848 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x5653539a0ad0 0x56535399f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.342+0000 7f9762a00640 1 --2- 172.21.15.43:0/2764796848 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x5653539a0ad0 0x56535399f7e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=de20d10b8078c429 server_cookie=a4e2cf8d39c7dc14 in_seq=0 out_seq=0 2024-03-28T22:36:06.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.342+0000 7f9750c00640 10 client.25229 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:36:06.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.342+0000 7f9750c00640 1 -- 172.21.15.43:0/2764796848 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x56535388a250 con 0x5653539a0ad0 2024-03-28T22:36:06.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.342+0000 7f9750c00640 10 client.25229 handle_command_reply: tid=0 2024-03-28T22:36:06.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.342+0000 7f976c7b3740 1 client.25229 shutdown 2024-03-28T22:36:06.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.343+0000 7f976c7b3740 1 -- 172.21.15.43:0/2764796848 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56535391f130 msgr2=0x56535392c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:06.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.343+0000 7f976c7b3740 1 --2- 172.21.15.43:0/2764796848 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56535391f130 0x56535392c510 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x56535391ac20 tx=0x7f974c005cc0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.343+0000 7f976c7b3740 1 -- 172.21.15.43:0/2764796848 shutdown_connections 2024-03-28T22:36:06.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.343+0000 7f976c7b3740 1 --2- 172.21.15.43:0/2764796848 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x5653539a0ad0 0x56535399f7e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.343+0000 7f976c7b3740 1 --2- 172.21.15.43:0/2764796848 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56535391f130 0x56535392c510 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.343+0000 7f976c7b3740 1 --2- 172.21.15.43:0/2764796848 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5653539242a0 0x56535392f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.343+0000 7f976c7b3740 1 --2- 172.21.15.43:0/2764796848 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56535391e740 0x56535392bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.343+0000 7f9750c00640 0 client.25229 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:36:06.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.343+0000 7f976c7b3740 1 -- 172.21.15.43:0/2764796848 >> 172.21.15.43:0/2764796848 conn(0x565353919cb0 msgr2=0x56535391cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:06.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.343+0000 7f976c7b3740 1 -- 172.21.15.43:0/2764796848 shutdown_connections 2024-03-28T22:36:06.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.343+0000 7f976c7b3740 1 -- 172.21.15.43:0/2764796848 wait complete. 2024-03-28T22:36:06.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.343+0000 7f976c7b3740 20 client.25229 trim_cache size 0 max 16384 2024-03-28T22:36:06.345 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:36:06.345 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_fsid": "c2276428-ed51-11ee-95d0-87774f69a715", 2024-03-28T22:36:06.345 INFO:teuthology.orchestra.run.smithi043.stdout: "whoami": 1, 2024-03-28T22:36:06.345 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 24556, 2024-03-28T22:36:06.345 INFO:teuthology.orchestra.run.smithi043.stdout: "want_state": "up:active", 2024-03-28T22:36:06.345 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "up:active", 2024-03-28T22:36:06.346 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs", 2024-03-28T22:36:06.346 INFO:teuthology.orchestra.run.smithi043.stdout: "rank_uptime": 124.219761164, 2024-03-28T22:36:06.346 INFO:teuthology.orchestra.run.smithi043.stdout: "mdsmap_epoch": 28, 2024-03-28T22:36:06.346 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch": 96, 2024-03-28T22:36:06.346 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch_barrier": 92, 2024-03-28T22:36:06.346 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 174.68274505900001 2024-03-28T22:36:06.346 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:36:06.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.344+0000 7f9763e00640 1 -- 172.21.15.43:0/514567966 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f97480a1e00 msgr2=0x7f97480a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:06.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.344+0000 7f9763e00640 1 --2- 172.21.15.43:0/514567966 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f97480a1e00 0x7f97480a42c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x56535391f990 tx=0x7f975c00a250 comp rx=0 tx=0).stop 2024-03-28T22:36:06.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.344+0000 7f9763e00640 1 -- 172.21.15.43:0/514567966 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f97641020b0 msgr2=0x7f9764199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:06.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.344+0000 7f9763e00640 1 --2- 172.21.15.43:0/514567966 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f97641020b0 0x7f9764199ad0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f97540325f0 tx=0x7f9754035d10 comp rx=0 tx=0).stop 2024-03-28T22:36:06.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.344+0000 7f9763e00640 1 -- 172.21.15.43:0/514567966 shutdown_connections 2024-03-28T22:36:06.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.344+0000 7f9763e00640 1 --2- 172.21.15.43:0/514567966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9764107310 0x7f97641a1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.344+0000 7f9763e00640 1 --2- 172.21.15.43:0/514567966 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f97641049e0 0x7f976419a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.344+0000 7f9763e00640 1 --2- 172.21.15.43:0/514567966 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f97480a1e00 0x7f97480a42c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.344+0000 7f9763e00640 1 --2- 172.21.15.43:0/514567966 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f97641020b0 0x7f9764199ad0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.344+0000 7f9763e00640 1 -- 172.21.15.43:0/514567966 >> 172.21.15.43:0/514567966 conn(0x7f97640fbac0 msgr2=0x7f9764101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:06.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.344+0000 7f9763e00640 1 -- 172.21.15.43:0/514567966 shutdown_connections 2024-03-28T22:36:06.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.344+0000 7f9763e00640 1 -- 172.21.15.43:0/514567966 wait complete. 2024-03-28T22:36:06.359 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T22:36:06.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.566+0000 7f2abfa00640 1 -- 172.21.15.43:0/3962978522 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2ab8104b80 msgr2=0x7f2ab8105000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:06.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.566+0000 7f2abfa00640 1 --2- 172.21.15.43:0/3962978522 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2ab8104b80 0x7f2ab8105000 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2aa0009be0 tx=0x7f2aa002bef0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.566+0000 7f2abfa00640 1 -- 172.21.15.43:0/3962978522 shutdown_connections 2024-03-28T22:36:06.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.566+0000 7f2abfa00640 1 --2- 172.21.15.43:0/3962978522 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2ab8104b80 0x7f2ab8105000 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.566+0000 7f2abfa00640 1 --2- 172.21.15.43:0/3962978522 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2ab8105540 0x7f2ab810bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.566+0000 7f2abfa00640 1 --2- 172.21.15.43:0/3962978522 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ab8103990 0x7f2ab8103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.566+0000 7f2abfa00640 1 -- 172.21.15.43:0/3962978522 >> 172.21.15.43:0/3962978522 conn(0x7f2ab80fef00 msgr2=0x7f2ab8101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:06.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.566+0000 7f2abfa00640 1 -- 172.21.15.43:0/3962978522 shutdown_connections 2024-03-28T22:36:06.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.567+0000 7f2abfa00640 1 -- 172.21.15.43:0/3962978522 wait complete. 2024-03-28T22:36:06.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.568+0000 7f2abfa00640 1 Processor -- start 2024-03-28T22:36:06.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.568+0000 7f2abfa00640 1 -- start start 2024-03-28T22:36:06.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.568+0000 7f2abfa00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2ab8103990 0x7f2ab8199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.568+0000 7f2abfa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ab8104b80 0x7f2ab819a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.568+0000 7f2abfa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2ab8105540 0x7f2ab81a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.568+0000 7f2abfa00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab810f660 con 0x7f2ab8104b80 2024-03-28T22:36:06.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.568+0000 7f2abfa00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab810f4e0 con 0x7f2ab8103990 2024-03-28T22:36:06.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.568+0000 7f2abfa00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab810f7e0 con 0x7f2ab8105540 2024-03-28T22:36:06.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.568+0000 7f2abd000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2ab8103990 0x7f2ab8199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.568+0000 7f2abd000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2ab8103990 0x7f2ab8199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59730/0 (socket says 172.21.15.43:59730) 2024-03-28T22:36:06.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.568+0000 7f2abd000640 1 -- 172.21.15.43:0/1505365873 learned_addr learned my addr 172.21.15.43:0/1505365873 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:06.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.568+0000 7f2ab7e00640 1 --2- 172.21.15.43:0/1505365873 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ab8104b80 0x7f2ab819a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.569+0000 7f2abd000640 1 -- 172.21.15.43:0/1505365873 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2ab8105540 msgr2=0x7f2ab81a1210 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T22:36:06.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.569+0000 7f2abd000640 1 --2- 172.21.15.43:0/1505365873 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2ab8105540 0x7f2ab81a1210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.569+0000 7f2abd000640 1 -- 172.21.15.43:0/1505365873 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ab8104b80 msgr2=0x7f2ab819a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:06.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.569+0000 7f2abd000640 1 --2- 172.21.15.43:0/1505365873 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ab8104b80 0x7f2ab819a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.569+0000 7f2abd000640 1 -- 172.21.15.43:0/1505365873 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ab81a18b0 con 0x7f2ab8103990 2024-03-28T22:36:06.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.569+0000 7f2ab7e00640 1 --2- 172.21.15.43:0/1505365873 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ab8104b80 0x7f2ab819a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:36:06.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.569+0000 7f2abd000640 1 --2- 172.21.15.43:0/1505365873 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2ab8103990 0x7f2ab8199c70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2ab000e9f0 tx=0x7f2ab000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:06.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.569+0000 7f2ab5600640 1 -- 172.21.15.43:0/1505365873 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab000ccc0 con 0x7f2ab8103990 2024-03-28T22:36:06.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.569+0000 7f2ab5600640 1 -- 172.21.15.43:0/1505365873 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ab00035f0 con 0x7f2ab8103990 2024-03-28T22:36:06.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.569+0000 7f2ab5600640 1 -- 172.21.15.43:0/1505365873 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab000ce60 con 0x7f2ab8103990 2024-03-28T22:36:06.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.569+0000 7f2abfa00640 1 -- 172.21.15.43:0/1505365873 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ab81a1ba0 con 0x7f2ab8103990 2024-03-28T22:36:06.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.569+0000 7f2abfa00640 1 -- 172.21.15.43:0/1505365873 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ab81a2140 con 0x7f2ab8103990 2024-03-28T22:36:06.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.572+0000 7f2ab5600640 1 -- 172.21.15.43:0/1505365873 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f2ab00037a0 con 0x7f2ab8103990 2024-03-28T22:36:06.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.573+0000 7f2ac25ed740 1 Processor -- start 2024-03-28T22:36:06.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.573+0000 7f2ac25ed740 1 -- start start 2024-03-28T22:36:06.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.573+0000 7f2ac25ed740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56240ce4b570 0x56240ce4b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.573+0000 7f2ac25ed740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56240ce4bf60 0x56240ce50a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.573+0000 7f2ac25ed740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56240ce510d0 0x56240ce53560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.573+0000 7f2ac25ed740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56240cdaf9c0 con 0x56240ce510d0 2024-03-28T22:36:06.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.573+0000 7f2ac25ed740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x56240cd58c30 con 0x56240ce4bf60 2024-03-28T22:36:06.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.573+0000 7f2ac25ed740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x56240cdaba60 con 0x56240ce4b570 2024-03-28T22:36:06.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.573+0000 7f2abda00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56240ce4b570 0x56240ce4b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.573+0000 7f2abd000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56240ce4bf60 0x56240ce50a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.573+0000 7f2abda00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56240ce4b570 0x56240ce4b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35676/0 (socket says 172.21.15.43:35676) 2024-03-28T22:36:06.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.573+0000 7f2abda00640 1 -- 172.21.15.43:0/1202837729 learned_addr learned my addr 172.21.15.43:0/1202837729 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:06.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.574+0000 7f2ab7e00640 1 --2- 172.21.15.43:0/1202837729 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56240ce510d0 0x56240ce53560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.574+0000 7f2abda00640 1 -- 172.21.15.43:0/1202837729 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56240ce4bf60 msgr2=0x56240ce50a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:06.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.574+0000 7f2abda00640 1 --2- 172.21.15.43:0/1202837729 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56240ce4bf60 0x56240ce50a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.574+0000 7f2abda00640 1 -- 172.21.15.43:0/1202837729 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56240ce510d0 msgr2=0x56240ce53560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:06.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.574+0000 7f2abda00640 1 --2- 172.21.15.43:0/1202837729 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56240ce510d0 0x56240ce53560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.574+0000 7f2abda00640 1 -- 172.21.15.43:0/1202837729 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56240cac42d0 con 0x56240ce4b570 2024-03-28T22:36:06.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.574+0000 7f2abd000640 1 --2- 172.21.15.43:0/1202837729 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56240ce4bf60 0x56240ce50a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:06.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.574+0000 7f2abda00640 1 --2- 172.21.15.43:0/1202837729 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56240ce4b570 0x56240ce4b990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8009bd0 tx=0x7f2aa800e720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58b185a17ab3a6c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:06.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.574+0000 7f2ab5600640 1 --2- 172.21.15.43:0/1505365873 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f2a9c0a1e00 0x7f2a9c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.575+0000 7f2aae000640 1 -- 172.21.15.43:0/1202837729 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa8005020 con 0x56240ce4b570 2024-03-28T22:36:06.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.575+0000 7f2aae000640 1 -- 172.21.15.43:0/1202837729 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2aa80047a0 con 0x56240ce4b570 2024-03-28T22:36:06.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.575+0000 7f2aae000640 1 -- 172.21.15.43:0/1202837729 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa8010690 con 0x56240ce4b570 2024-03-28T22:36:06.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.575+0000 7f2ab5600640 1 -- 172.21.15.43:0/1505365873 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f2ab00ce4c0 con 0x7f2ab8103990 2024-03-28T22:36:06.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.575+0000 7f2abd000640 1 --2- 172.21.15.43:0/1505365873 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f2a9c0a1e00 0x7f2a9c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.575+0000 7f2ac25ed740 1 -- 172.21.15.43:0/1202837729 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56240ce4b570 msgr2=0x56240ce4b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:06.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.575+0000 7f2ac25ed740 1 --2- 172.21.15.43:0/1202837729 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56240ce4b570 0x56240ce4b990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8009bd0 tx=0x7f2aa800e720 comp rx=0 tx=0).stop 2024-03-28T22:36:06.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.575+0000 7f2ac25ed740 1 -- 172.21.15.43:0/1202837729 shutdown_connections 2024-03-28T22:36:06.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.575+0000 7f2ac25ed740 1 --2- 172.21.15.43:0/1202837729 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56240ce4bf60 0x56240ce50a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.575+0000 7f2ac25ed740 1 --2- 172.21.15.43:0/1202837729 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56240ce510d0 0x56240ce53560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.575+0000 7f2abd000640 1 --2- 172.21.15.43:0/1505365873 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f2a9c0a1e00 0x7f2a9c0a42c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2ab00cbbd0 tx=0x7f2ab00d1f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:06.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.575+0000 7f2ac25ed740 1 --2- 172.21.15.43:0/1202837729 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56240ce4b570 0x56240ce4b990 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.575+0000 7f2ac25ed740 1 -- 172.21.15.43:0/1202837729 >> 172.21.15.43:0/1202837729 conn(0x56240ce46ae0 msgr2=0x56240ce46ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:06.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.575+0000 7f2ac25ed740 1 -- 172.21.15.43:0/1202837729 shutdown_connections 2024-03-28T22:36:06.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.576+0000 7f2ac25ed740 1 -- 172.21.15.43:0/1202837729 wait complete. 2024-03-28T22:36:06.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.576+0000 7f2ac25ed740 1 -- start start 2024-03-28T22:36:06.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.576+0000 7f2ac25ed740 1 Processor -- start 2024-03-28T22:36:06.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.577+0000 7f2ac25ed740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56240ce4b570 0x56240ce55a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.577+0000 7f2ac25ed740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56240ce4bf60 0x56240ce55fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.577+0000 7f2ac25ed740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56240ce510d0 0x56240ce5d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.577+0000 7f2ac25ed740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56240ce54080 con 0x56240ce4bf60 2024-03-28T22:36:06.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.577+0000 7f2ac25ed740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x56240cd4e180 con 0x56240ce510d0 2024-03-28T22:36:06.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.577+0000 7f2ac25ed740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x56240ce54200 con 0x56240ce4b570 2024-03-28T22:36:06.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.577+0000 7f2ac25ed740 10 client.0 _ll_register_callbacks cb 0x56240cac3ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:36:06.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.577+0000 7f2abda00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56240ce4b570 0x56240ce55a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.577+0000 7f2abda00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56240ce4b570 0x56240ce55a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35678/0 (socket says 172.21.15.43:35678) 2024-03-28T22:36:06.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.577+0000 7f2abda00640 1 -- 172.21.15.43:0/2557465232 learned_addr learned my addr 172.21.15.43:0/2557465232 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:06.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.577+0000 7f2abda00640 1 --2- 172.21.15.43:0/2557465232 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56240ce510d0 0x56240ce5d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.577+0000 7f2ab7e00640 1 --2- 172.21.15.43:0/2557465232 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56240ce4bf60 0x56240ce55fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.577+0000 7f2abda00640 1 -- 172.21.15.43:0/2557465232 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56240ce510d0 msgr2=0x56240ce5d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:06.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.577+0000 7f2abda00640 1 --2- 172.21.15.43:0/2557465232 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56240ce510d0 0x56240ce5d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.577+0000 7f2abda00640 1 -- 172.21.15.43:0/2557465232 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56240ce4bf60 msgr2=0x56240ce55fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:06.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.577+0000 7f2abda00640 1 --2- 172.21.15.43:0/2557465232 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56240ce4bf60 0x56240ce55fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.577+0000 7f2abda00640 1 -- 172.21.15.43:0/2557465232 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56240cb8e6b0 con 0x56240ce4b570 2024-03-28T22:36:06.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.578+0000 7f2abda00640 1 --2- 172.21.15.43:0/2557465232 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56240ce4b570 0x56240ce55a80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8004650 tx=0x7f2aa8004700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d17145d1817bca62 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:06.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.578+0000 7f2aa6a00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:36:06.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.578+0000 7f2aa6a00640 1 -- 172.21.15.43:0/2557465232 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa802b060 con 0x56240ce4b570 2024-03-28T22:36:06.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.578+0000 7f2aa6a00640 1 -- 172.21.15.43:0/2557465232 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f2aa8005460 con 0x56240ce4b570 2024-03-28T22:36:06.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.578+0000 7f2ac25ed740 1 -- 172.21.15.43:0/2557465232 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2aa80047a0 con 0x56240ce4b570 2024-03-28T22:36:06.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.578+0000 7f2aa6a00640 1 -- 172.21.15.43:0/2557465232 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2aa8010430 con 0x56240ce4b570 2024-03-28T22:36:06.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.578+0000 7f2ac25ed740 10 client.25249 fetch_fsmap learned FSMap version 28 2024-03-28T22:36:06.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.578+0000 7f2ac25ed740 1 -- 172.21.15.43:0/2557465232 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56240ce60370 con 0x56240ce4b570 2024-03-28T22:36:06.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.579+0000 7f2aa6a00640 1 -- 172.21.15.43:0/2557465232 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2aa80047a0 con 0x56240ce4b570 2024-03-28T22:36:06.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.579+0000 7f2aa6a00640 1 -- 172.21.15.43:0/2557465232 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f2aa80184c0 con 0x56240ce4b570 2024-03-28T22:36:06.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.579+0000 7f2ac25ed740 10 client.25249 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T22:36:06.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.579+0000 7f2ac25ed740 10 client.25249 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T22:36:06.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.579+0000 7f2ac25ed740 20 client.25249 populate_metadata read hostname 'smithi043' 2024-03-28T22:36:06.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.579+0000 7f2ac25ed740 1 --2- 172.21.15.43:0/2557465232 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56240ce62650 0x56240ce63ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.579+0000 7f2ac25ed740 4 client.25249 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:36:06.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.579+0000 7f2ac25ed740 1 -- 172.21.15.43:0/2557465232 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56240cdbcf00 con 0x56240ce62650 2024-03-28T22:36:06.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.579+0000 7f2ab7e00640 1 --2- 172.21.15.43:0/2557465232 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56240ce62650 0x56240ce63ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.580+0000 7f2ab7e00640 1 --2- 172.21.15.43:0/2557465232 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56240ce62650 0x56240ce63ac0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7dd2325aee88e89c server_cookie=cd4046ffb8eca761 in_seq=0 out_seq=0 2024-03-28T22:36:06.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.580+0000 7f2aa6a00640 10 client.25249 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:36:06.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.581+0000 7f2aa6a00640 1 -- 172.21.15.43:0/2557465232 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56240cdbcf00 con 0x56240ce62650 2024-03-28T22:36:06.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.581+0000 7f2aa6a00640 10 client.25249 handle_command_reply: tid=0 2024-03-28T22:36:06.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.581+0000 7f2ac25ed740 1 client.25249 shutdown 2024-03-28T22:36:06.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.581+0000 7f2ac25ed740 1 -- 172.21.15.43:0/2557465232 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56240ce4b570 msgr2=0x56240ce55a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:06.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.581+0000 7f2ac25ed740 1 --2- 172.21.15.43:0/2557465232 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56240ce4b570 0x56240ce55a80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8004650 tx=0x7f2aa8004700 comp rx=0 tx=0).stop 2024-03-28T22:36:06.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.582+0000 7f2ac25ed740 1 -- 172.21.15.43:0/2557465232 shutdown_connections 2024-03-28T22:36:06.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.582+0000 7f2ac25ed740 1 --2- 172.21.15.43:0/2557465232 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56240ce62650 0x56240ce63ac0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.582+0000 7f2ac25ed740 1 --2- 172.21.15.43:0/2557465232 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56240ce510d0 0x56240ce5d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.582+0000 7f2ac25ed740 1 --2- 172.21.15.43:0/2557465232 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56240ce4bf60 0x56240ce55fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.582+0000 7f2ac25ed740 1 --2- 172.21.15.43:0/2557465232 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56240ce4b570 0x56240ce55a80 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.582+0000 7f2aa6a00640 0 client.25249 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:36:06.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.582+0000 7f2ac25ed740 1 -- 172.21.15.43:0/2557465232 >> 172.21.15.43:0/2557465232 conn(0x56240ce46ae0 msgr2=0x56240ce45ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:06.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.582+0000 7f2ac25ed740 1 -- 172.21.15.43:0/2557465232 shutdown_connections 2024-03-28T22:36:06.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.582+0000 7f2ac25ed740 1 -- 172.21.15.43:0/2557465232 wait complete. 2024-03-28T22:36:06.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.582+0000 7f2ac25ed740 20 client.25249 trim_cache size 0 max 16384 2024-03-28T22:36:06.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.602+0000 7f2ac25ed740 1 Processor -- start 2024-03-28T22:36:06.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.602+0000 7f2ac25ed740 1 -- start start 2024-03-28T22:36:06.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.602+0000 7f2ac25ed740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56240ce4b570 0x56240ce4ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.603+0000 7f2ac25ed740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56240ce4bf60 0x56240ce5cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.603+0000 7f2ac25ed740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56240ce510d0 0x56240ce55690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.603+0000 7f2ac25ed740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56240ce61e00 con 0x56240ce510d0 2024-03-28T22:36:06.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.603+0000 7f2ac25ed740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x56240ce624d0 con 0x56240ce4b570 2024-03-28T22:36:06.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.603+0000 7f2ac25ed740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x56240ce62ba0 con 0x56240ce4bf60 2024-03-28T22:36:06.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.603+0000 7f2abda00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56240ce4b570 0x56240ce4ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.603+0000 7f2abda00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56240ce4b570 0x56240ce4ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59744/0 (socket says 172.21.15.43:59744) 2024-03-28T22:36:06.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.603+0000 7f2abda00640 1 -- 172.21.15.43:0/775541046 learned_addr learned my addr 172.21.15.43:0/775541046 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:06.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.603+0000 7f2abda00640 1 --2- 172.21.15.43:0/775541046 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56240ce510d0 0x56240ce55690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.604+0000 7f2ab7e00640 1 --2- 172.21.15.43:0/775541046 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56240ce4bf60 0x56240ce5cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.604+0000 7f2abda00640 1 -- 172.21.15.43:0/775541046 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56240ce4bf60 msgr2=0x56240ce5cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:06.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.604+0000 7f2abda00640 1 --2- 172.21.15.43:0/775541046 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56240ce4bf60 0x56240ce5cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.604+0000 7f2abda00640 1 -- 172.21.15.43:0/775541046 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56240ce510d0 msgr2=0x56240ce55690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:06.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.604+0000 7f2abda00640 1 --2- 172.21.15.43:0/775541046 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56240ce510d0 0x56240ce55690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.604+0000 7f2abda00640 1 -- 172.21.15.43:0/775541046 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56240ce543c0 con 0x56240ce4b570 2024-03-28T22:36:06.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.604+0000 7f2abda00640 1 --2- 172.21.15.43:0/775541046 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56240ce4b570 0x56240ce4ca00 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2aa800eca0 tx=0x7f2aa800e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ed1dec5cdac25b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:06.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.604+0000 7f2aad600640 1 -- 172.21.15.43:0/775541046 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa802b040 con 0x56240ce4b570 2024-03-28T22:36:06.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.604+0000 7f2aad600640 1 -- 172.21.15.43:0/775541046 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2aa8004770 con 0x56240ce4b570 2024-03-28T22:36:06.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.604+0000 7f2aad600640 1 -- 172.21.15.43:0/775541046 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa8018b00 con 0x56240ce4b570 2024-03-28T22:36:06.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.605+0000 7f2ac25ed740 1 -- 172.21.15.43:0/775541046 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56240ce4b570 msgr2=0x56240ce4ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:06.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.605+0000 7f2ac25ed740 1 --2- 172.21.15.43:0/775541046 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56240ce4b570 0x56240ce4ca00 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2aa800eca0 tx=0x7f2aa800e720 comp rx=0 tx=0).stop 2024-03-28T22:36:06.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.605+0000 7f2ac25ed740 1 -- 172.21.15.43:0/775541046 shutdown_connections 2024-03-28T22:36:06.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.605+0000 7f2ac25ed740 1 --2- 172.21.15.43:0/775541046 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56240ce510d0 0x56240ce55690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.605+0000 7f2ac25ed740 1 --2- 172.21.15.43:0/775541046 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56240ce4bf60 0x56240ce5cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.605+0000 7f2ac25ed740 1 --2- 172.21.15.43:0/775541046 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56240ce4b570 0x56240ce4ca00 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.605+0000 7f2ac25ed740 1 -- 172.21.15.43:0/775541046 >> 172.21.15.43:0/775541046 conn(0x56240ce46ae0 msgr2=0x56240ce46ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:06.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.605+0000 7f2ac25ed740 1 -- 172.21.15.43:0/775541046 shutdown_connections 2024-03-28T22:36:06.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.605+0000 7f2ac25ed740 1 -- 172.21.15.43:0/775541046 wait complete. 2024-03-28T22:36:06.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.606+0000 7f2ac25ed740 1 -- start start 2024-03-28T22:36:06.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.606+0000 7f2ac25ed740 1 Processor -- start 2024-03-28T22:36:06.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.606+0000 7f2ac25ed740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56240ce4b570 0x56240ce5afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.606+0000 7f2ac25ed740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56240ce4bf60 0x56240cec43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.606+0000 7f2ac25ed740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56240ce510d0 0x56240cec68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.606+0000 7f2ac25ed740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56240ce63940 con 0x56240ce4bf60 2024-03-28T22:36:06.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.606+0000 7f2ac25ed740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x56240ce63270 con 0x56240ce510d0 2024-03-28T22:36:06.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.606+0000 7f2ac25ed740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x56240ce64010 con 0x56240ce4b570 2024-03-28T22:36:06.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.606+0000 7f2ac25ed740 10 client.0 _ll_register_callbacks cb 0x7f2ab0005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:36:06.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.607+0000 7f2abda00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56240ce4b570 0x56240ce5afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.607+0000 7f2abda00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56240ce4b570 0x56240ce5afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35688/0 (socket says 172.21.15.43:35688) 2024-03-28T22:36:06.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.607+0000 7f2abda00640 1 -- 172.21.15.43:0/3445043207 learned_addr learned my addr 172.21.15.43:0/3445043207 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:06.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.607+0000 7f2abda00640 1 --2- 172.21.15.43:0/3445043207 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56240ce510d0 0x56240cec68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.607+0000 7f2ab7e00640 1 --2- 172.21.15.43:0/3445043207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56240ce4bf60 0x56240cec43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.607+0000 7f2abda00640 1 -- 172.21.15.43:0/3445043207 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56240ce510d0 msgr2=0x56240cec68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:06.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.607+0000 7f2abda00640 1 --2- 172.21.15.43:0/3445043207 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56240ce510d0 0x56240cec68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.607+0000 7f2abda00640 1 -- 172.21.15.43:0/3445043207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56240ce4bf60 msgr2=0x56240cec43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:06.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.607+0000 7f2abda00640 1 --2- 172.21.15.43:0/3445043207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56240ce4bf60 0x56240cec43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.607+0000 7f2abda00640 1 -- 172.21.15.43:0/3445043207 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56240cec7000 con 0x56240ce4b570 2024-03-28T22:36:06.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.607+0000 7f2abda00640 1 --2- 172.21.15.43:0/3445043207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56240ce4b570 0x56240ce5afb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2aa80189b0 tx=0x7f2aa8000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c9d4c54c387ee93 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:06.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.607+0000 7f2aa6a00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:36:06.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.608+0000 7f2aa6a00640 1 -- 172.21.15.43:0/3445043207 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa802b070 con 0x56240ce4b570 2024-03-28T22:36:06.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.608+0000 7f2ac25ed740 1 -- 172.21.15.43:0/3445043207 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2aa8004770 con 0x56240ce4b570 2024-03-28T22:36:06.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.608+0000 7f2aa6a00640 1 -- 172.21.15.43:0/3445043207 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f2aa803f040 con 0x56240ce4b570 2024-03-28T22:36:06.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.608+0000 7f2aa6a00640 1 -- 172.21.15.43:0/3445043207 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2aa8018b00 con 0x56240ce4b570 2024-03-28T22:36:06.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.608+0000 7f2ac25ed740 10 client.25255 fetch_fsmap learned FSMap version 28 2024-03-28T22:36:06.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.608+0000 7f2ac25ed740 1 -- 172.21.15.43:0/3445043207 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56240cec8300 con 0x56240ce4b570 2024-03-28T22:36:06.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.608+0000 7f2aa6a00640 1 -- 172.21.15.43:0/3445043207 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2aa8004770 con 0x56240ce4b570 2024-03-28T22:36:06.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.608+0000 7f2aa6a00640 1 -- 172.21.15.43:0/3445043207 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f2aa80035f0 con 0x56240ce4b570 2024-03-28T22:36:06.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.608+0000 7f2ac25ed740 10 client.25255 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T22:36:06.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.609+0000 7f2ac25ed740 10 client.25255 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T22:36:06.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.609+0000 7f2ac25ed740 20 client.25255 populate_metadata read hostname 'smithi043' 2024-03-28T22:36:06.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.609+0000 7f2ac25ed740 1 --2- 172.21.15.43:0/3445043207 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56240cecd2c0 0x56240cecbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:06.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.609+0000 7f2ac25ed740 4 client.25255 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T22:36:06.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.609+0000 7f2ac25ed740 1 -- 172.21.15.43:0/3445043207 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "status"}) -- 0x56240cdb7250 con 0x56240cecd2c0 2024-03-28T22:36:06.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.609+0000 7f2ab7e00640 1 --2- 172.21.15.43:0/3445043207 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56240cecd2c0 0x56240cecbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:06.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.609+0000 7f2ab7e00640 1 --2- 172.21.15.43:0/3445043207 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56240cecd2c0 0x56240cecbfd0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ed8aff52caf99e3 server_cookie=3a817523285dc613 in_seq=0 out_seq=0 2024-03-28T22:36:06.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.609+0000 7f2aa6a00640 10 client.25255 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:36:06.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.610+0000 7f2aa6a00640 1 -- 172.21.15.43:0/3445043207 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x56240cdb7250 con 0x56240cecd2c0 2024-03-28T22:36:06.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.610+0000 7f2aa6a00640 10 client.25255 handle_command_reply: tid=0 2024-03-28T22:36:06.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.610+0000 7f2ac25ed740 1 client.25255 shutdown 2024-03-28T22:36:06.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.610+0000 7f2ac25ed740 1 -- 172.21.15.43:0/3445043207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56240ce4b570 msgr2=0x56240ce5afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:06.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.610+0000 7f2ac25ed740 1 --2- 172.21.15.43:0/3445043207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56240ce4b570 0x56240ce5afb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2aa80189b0 tx=0x7f2aa8000bc0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.610+0000 7f2ac25ed740 1 -- 172.21.15.43:0/3445043207 shutdown_connections 2024-03-28T22:36:06.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.610+0000 7f2ac25ed740 1 --2- 172.21.15.43:0/3445043207 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56240cecd2c0 0x56240cecbfd0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.610+0000 7f2ac25ed740 1 --2- 172.21.15.43:0/3445043207 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56240ce510d0 0x56240cec68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.610+0000 7f2ac25ed740 1 --2- 172.21.15.43:0/3445043207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56240ce4bf60 0x56240cec43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.610+0000 7f2ac25ed740 1 --2- 172.21.15.43:0/3445043207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56240ce4b570 0x56240ce5afb0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.610+0000 7f2aa6a00640 0 client.25255 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:36:06.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.610+0000 7f2ac25ed740 1 -- 172.21.15.43:0/3445043207 >> 172.21.15.43:0/3445043207 conn(0x56240ce46ae0 msgr2=0x56240ce55690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:06.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.610+0000 7f2ac25ed740 1 -- 172.21.15.43:0/3445043207 shutdown_connections 2024-03-28T22:36:06.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.610+0000 7f2ac25ed740 1 -- 172.21.15.43:0/3445043207 wait complete. 2024-03-28T22:36:06.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.610+0000 7f2ac25ed740 20 client.25255 trim_cache size 0 max 16384 2024-03-28T22:36:06.613 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:36:06.613 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_fsid": "c2276428-ed51-11ee-95d0-87774f69a715", 2024-03-28T22:36:06.613 INFO:teuthology.orchestra.run.smithi043.stdout: "whoami": 0, 2024-03-28T22:36:06.613 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 24572, 2024-03-28T22:36:06.613 INFO:teuthology.orchestra.run.smithi043.stdout: "want_state": "up:active", 2024-03-28T22:36:06.613 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "up:active", 2024-03-28T22:36:06.613 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs", 2024-03-28T22:36:06.613 INFO:teuthology.orchestra.run.smithi043.stdout: "rank_uptime": 157.136373349, 2024-03-28T22:36:06.613 INFO:teuthology.orchestra.run.smithi043.stdout: "mdsmap_epoch": 28, 2024-03-28T22:36:06.613 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch": 96, 2024-03-28T22:36:06.613 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch_barrier": 92, 2024-03-28T22:36:06.613 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 163.510451872 2024-03-28T22:36:06.613 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:36:06.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.611+0000 7f2abfa00640 1 -- 172.21.15.43:0/1505365873 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f2a9c0a1e00 msgr2=0x7f2a9c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:06.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.611+0000 7f2abfa00640 1 --2- 172.21.15.43:0/1505365873 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f2a9c0a1e00 0x7f2a9c0a42c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2ab00cbbd0 tx=0x7f2ab00d1f00 comp rx=0 tx=0).stop 2024-03-28T22:36:06.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.611+0000 7f2abfa00640 1 -- 172.21.15.43:0/1505365873 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2ab8103990 msgr2=0x7f2ab8199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:06.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.611+0000 7f2abfa00640 1 --2- 172.21.15.43:0/1505365873 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2ab8103990 0x7f2ab8199c70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2ab000e9f0 tx=0x7f2ab000eeb0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.611+0000 7f2abfa00640 1 -- 172.21.15.43:0/1505365873 shutdown_connections 2024-03-28T22:36:06.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.611+0000 7f2abfa00640 1 --2- 172.21.15.43:0/1505365873 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2ab8105540 0x7f2ab81a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.611+0000 7f2abfa00640 1 --2- 172.21.15.43:0/1505365873 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2ab8104b80 0x7f2ab819a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.611+0000 7f2abfa00640 1 --2- 172.21.15.43:0/1505365873 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f2a9c0a1e00 0x7f2a9c0a42c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.611+0000 7f2abfa00640 1 --2- 172.21.15.43:0/1505365873 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2ab8103990 0x7f2ab8199c70 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:06.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.611+0000 7f2abfa00640 1 -- 172.21.15.43:0/1505365873 >> 172.21.15.43:0/1505365873 conn(0x7f2ab80fef00 msgr2=0x7f2ab8107e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:06.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.611+0000 7f2abfa00640 1 -- 172.21.15.43:0/1505365873 shutdown_connections 2024-03-28T22:36:06.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:06.611+0000 7f2abfa00640 1 -- 172.21.15.43:0/1505365873 wait complete. 2024-03-28T22:36:06.626 INFO:tasks.cephadm:Setup complete, yielding 2024-03-28T22:36:06.626 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-28T22:36:06.636 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi043.front.sepia.ceph.com 2024-03-28T22:36:06.637 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- bash -c 'ceph orch status' 2024-03-28T22:36:07.109 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:36:07.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:07 smithi043 ceph-mon[31823]: pgmap v405: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T22:36:07.702 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:07 smithi082 ceph-mon[36471]: pgmap v405: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T22:36:07.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:07 smithi172 ceph-mon[36566]: pgmap v405: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T22:36:09.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.350+0000 7f3580800700 1 -- 172.21.15.43:0/846953258 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3578102260 msgr2=0x7f3578102680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:09.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.350+0000 7f3580800700 1 --2- 172.21.15.43:0/846953258 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3578102260 0x7f3578102680 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f356c009990 tx=0x7f356c009ca0 comp rx=0 tx=0).stop 2024-03-28T22:36:09.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.350+0000 7f3580800700 1 -- 172.21.15.43:0/846953258 shutdown_connections 2024-03-28T22:36:09.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.350+0000 7f3580800700 1 --2- 172.21.15.43:0/846953258 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3578103e00 0x7f357810a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:09.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.350+0000 7f3580800700 1 --2- 172.21.15.43:0/846953258 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3578102260 0x7f3578102680 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:09.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.350+0000 7f3580800700 1 --2- 172.21.15.43:0/846953258 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3578103460 0x7f35781038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:09.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.350+0000 7f3580800700 1 -- 172.21.15.43:0/846953258 >> 172.21.15.43:0/846953258 conn(0x7f35780fd820 msgr2=0x7f35780ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:09.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.351+0000 7f3580800700 1 -- 172.21.15.43:0/846953258 shutdown_connections 2024-03-28T22:36:09.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.351+0000 7f3580800700 1 -- 172.21.15.43:0/846953258 wait complete. 2024-03-28T22:36:09.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.352+0000 7f3580800700 1 Processor -- start 2024-03-28T22:36:09.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.352+0000 7f3580800700 1 -- start start 2024-03-28T22:36:09.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.353+0000 7f3580800700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3578102260 0x7f357819a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:09.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.353+0000 7f3580800700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3578103460 0x7f357819ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:09.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.353+0000 7f3580800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3578103e00 0x7f35781a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:09.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.353+0000 7f3580800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f357810de70 con 0x7f3578103e00 2024-03-28T22:36:09.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.353+0000 7f3580800700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f357810dcf0 con 0x7f3578102260 2024-03-28T22:36:09.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.353+0000 7f3580800700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f357810dff0 con 0x7f3578103460 2024-03-28T22:36:09.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.353+0000 7f357d400700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3578103460 0x7f357819ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:09.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.353+0000 7f357de00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3578102260 0x7f357819a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:09.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.353+0000 7f357d400700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3578103460 0x7f357819ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35710/0 (socket says 172.21.15.43:35710) 2024-03-28T22:36:09.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.353+0000 7f357d400700 1 -- 172.21.15.43:0/2926244207 learned_addr learned my addr 172.21.15.43:0/2926244207 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:09.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.354+0000 7f357e800700 1 --2- 172.21.15.43:0/2926244207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3578103e00 0x7f35781a1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:09.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.354+0000 7f357de00700 1 -- 172.21.15.43:0/2926244207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3578103460 msgr2=0x7f357819ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:09.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.354+0000 7f357de00700 1 --2- 172.21.15.43:0/2926244207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3578103460 0x7f357819ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:09.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.354+0000 7f357de00700 1 -- 172.21.15.43:0/2926244207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3578103e00 msgr2=0x7f35781a1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:09.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.354+0000 7f357de00700 1 --2- 172.21.15.43:0/2926244207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3578103e00 0x7f35781a1c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:09.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.354+0000 7f357de00700 1 -- 172.21.15.43:0/2926244207 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35781a2330 con 0x7f3578102260 2024-03-28T22:36:09.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.354+0000 7f357d400700 1 --2- 172.21.15.43:0/2926244207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3578103460 0x7f357819ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:09.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.354+0000 7f357de00700 1 --2- 172.21.15.43:0/2926244207 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3578102260 0x7f357819a7c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f356c000c00 tx=0x7f356c009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:09.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.354+0000 7f3572a00700 1 -- 172.21.15.43:0/2926244207 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f356c026070 con 0x7f3578102260 2024-03-28T22:36:09.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.354+0000 7f3572a00700 1 -- 172.21.15.43:0/2926244207 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f356c00fdb0 con 0x7f3578102260 2024-03-28T22:36:09.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.354+0000 7f3580800700 1 -- 172.21.15.43:0/2926244207 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35781a25c0 con 0x7f3578102260 2024-03-28T22:36:09.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.354+0000 7f3572a00700 1 -- 172.21.15.43:0/2926244207 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f356c017650 con 0x7f3578102260 2024-03-28T22:36:09.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.355+0000 7f3580800700 1 -- 172.21.15.43:0/2926244207 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35781a2a10 con 0x7f3578102260 2024-03-28T22:36:09.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.356+0000 7f3580800700 1 -- 172.21.15.43:0/2926244207 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3578068f30 con 0x7f3578102260 2024-03-28T22:36:09.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.357+0000 7f3572a00700 1 -- 172.21.15.43:0/2926244207 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f356c029070 con 0x7f3578102260 2024-03-28T22:36:09.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.359+0000 7f3572a00700 1 --2- 172.21.15.43:0/2926244207 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f35600a1e10 0x7f35600a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:09.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.359+0000 7f3572a00700 1 -- 172.21.15.43:0/2926244207 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f356c0cf1b0 con 0x7f3578102260 2024-03-28T22:36:09.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.359+0000 7f357d400700 1 --2- 172.21.15.43:0/2926244207 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f35600a1e10 0x7f35600a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:09.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.360+0000 7f357d400700 1 --2- 172.21.15.43:0/2926244207 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f35600a1e10 0x7f35600a42d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3568011550 tx=0x7f3568009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:09.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.363+0000 7f3572a00700 1 -- 172.21.15.43:0/2926244207 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f356c05d280 con 0x7f3578102260 2024-03-28T22:36:09.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:09 smithi043 ceph-mon[31823]: pgmap v406: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:36:09.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.615+0000 7f3580800700 1 -- 172.21.15.43:0/2926244207 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f3578075b90 con 0x7f35600a1e10 2024-03-28T22:36:09.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.617+0000 7f3572a00700 1 -- 172.21.15.43:0/2926244207 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f3578075b90 con 0x7f35600a1e10 2024-03-28T22:36:09.616 INFO:teuthology.orchestra.run.smithi043.stdout:Backend: cephadm 2024-03-28T22:36:09.616 INFO:teuthology.orchestra.run.smithi043.stdout:Available: Yes 2024-03-28T22:36:09.616 INFO:teuthology.orchestra.run.smithi043.stdout:Paused: No 2024-03-28T22:36:09.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.621+0000 7f3580800700 1 -- 172.21.15.43:0/2926244207 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f35600a1e10 msgr2=0x7f35600a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:09.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.621+0000 7f3580800700 1 --2- 172.21.15.43:0/2926244207 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f35600a1e10 0x7f35600a42d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3568011550 tx=0x7f3568009260 comp rx=0 tx=0).stop 2024-03-28T22:36:09.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.621+0000 7f3580800700 1 -- 172.21.15.43:0/2926244207 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3578102260 msgr2=0x7f357819a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:09.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.621+0000 7f3580800700 1 --2- 172.21.15.43:0/2926244207 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3578102260 0x7f357819a7c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f356c000c00 tx=0x7f356c009580 comp rx=0 tx=0).stop 2024-03-28T22:36:09.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.621+0000 7f3580800700 1 -- 172.21.15.43:0/2926244207 shutdown_connections 2024-03-28T22:36:09.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.621+0000 7f3580800700 1 --2- 172.21.15.43:0/2926244207 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f35600a1e10 0x7f35600a42d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:09.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.621+0000 7f3580800700 1 --2- 172.21.15.43:0/2926244207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3578103e00 0x7f35781a1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:09.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.621+0000 7f3580800700 1 --2- 172.21.15.43:0/2926244207 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3578102260 0x7f357819a7c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:09.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.621+0000 7f3580800700 1 --2- 172.21.15.43:0/2926244207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3578103460 0x7f357819ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:09.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.621+0000 7f3580800700 1 -- 172.21.15.43:0/2926244207 >> 172.21.15.43:0/2926244207 conn(0x7f35780fd820 msgr2=0x7f3578074470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:09.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.621+0000 7f3580800700 1 -- 172.21.15.43:0/2926244207 shutdown_connections 2024-03-28T22:36:09.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:09.622+0000 7f3580800700 1 -- 172.21.15.43:0/2926244207 wait complete. 2024-03-28T22:36:09.697 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:09 smithi172 ceph-mon[36566]: pgmap v406: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:36:09.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:09 smithi082 ceph-mon[36471]: pgmap v406: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:36:10.546 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-28T22:36:10.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:10 smithi082 ceph-mon[36471]: from='client.25253 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:36:10.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:10 smithi043 ceph-mon[31823]: from='client.25253 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:36:10.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:10 smithi172 ceph-mon[36566]: from='client.25253 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:36:11.034 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:36:11.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:11 smithi043 ceph-mon[31823]: pgmap v407: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-28T22:36:11.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:11 smithi082 ceph-mon[36471]: pgmap v407: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-28T22:36:11.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:11 smithi172 ceph-mon[36566]: pgmap v407: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-28T22:36:13.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.320+0000 7f28e8600700 1 -- 172.21.15.43:0/355887574 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28e0105630 msgr2=0x7f28e0107a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:13.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.320+0000 7f28e8600700 1 --2- 172.21.15.43:0/355887574 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28e0105630 0x7f28e0107a20 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f28cc009990 tx=0x7f28cc009ca0 comp rx=0 tx=0).stop 2024-03-28T22:36:13.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.320+0000 7f28e8600700 1 -- 172.21.15.43:0/355887574 shutdown_connections 2024-03-28T22:36:13.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.320+0000 7f28e8600700 1 --2- 172.21.15.43:0/355887574 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28e0107f60 0x7f28e010a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:13.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.320+0000 7f28e8600700 1 --2- 172.21.15.43:0/355887574 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f28e0102d00 0x7f28e01050f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:13.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.320+0000 7f28e8600700 1 --2- 172.21.15.43:0/355887574 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28e0105630 0x7f28e0107a20 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:13.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.320+0000 7f28e8600700 1 -- 172.21.15.43:0/355887574 >> 172.21.15.43:0/355887574 conn(0x7f28e00fc710 msgr2=0x7f28e00feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:13.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.320+0000 7f28e8600700 1 -- 172.21.15.43:0/355887574 shutdown_connections 2024-03-28T22:36:13.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.320+0000 7f28e8600700 1 -- 172.21.15.43:0/355887574 wait complete. 2024-03-28T22:36:13.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.321+0000 7f28e8600700 1 Processor -- start 2024-03-28T22:36:13.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.322+0000 7f28e8600700 1 -- start start 2024-03-28T22:36:13.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.322+0000 7f28e8600700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f28e0102d00 0x7f28e01984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:13.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.322+0000 7f28e8600700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28e0105630 0x7f28e0198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:13.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.322+0000 7f28e8600700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28e0107f60 0x7f28e019fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:13.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.323+0000 7f28e8600700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f28e010de50 con 0x7f28e0105630 2024-03-28T22:36:13.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.323+0000 7f28e8600700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f28e010dcd0 con 0x7f28e0102d00 2024-03-28T22:36:13.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.323+0000 7f28e8600700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f28e010dfd0 con 0x7f28e0107f60 2024-03-28T22:36:13.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.323+0000 7f28e5c00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f28e0102d00 0x7f28e01984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:13.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.323+0000 7f28e5200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28e0105630 0x7f28e0198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:13.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.323+0000 7f28e6600700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28e0107f60 0x7f28e019fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:13.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.323+0000 7f28e5200700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28e0105630 0x7f28e0198a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50088/0 (socket says 172.21.15.43:50088) 2024-03-28T22:36:13.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.323+0000 7f28e5200700 1 -- 172.21.15.43:0/2565404458 learned_addr learned my addr 172.21.15.43:0/2565404458 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:13.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.323+0000 7f28e5c00700 1 -- 172.21.15.43:0/2565404458 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28e0107f60 msgr2=0x7f28e019fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:13.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.323+0000 7f28e5c00700 1 --2- 172.21.15.43:0/2565404458 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28e0107f60 0x7f28e019fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:13.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.323+0000 7f28e5c00700 1 -- 172.21.15.43:0/2565404458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28e0105630 msgr2=0x7f28e0198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:13.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.323+0000 7f28e5c00700 1 --2- 172.21.15.43:0/2565404458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28e0105630 0x7f28e0198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:13.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.323+0000 7f28e5c00700 1 -- 172.21.15.43:0/2565404458 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28e01a0130 con 0x7f28e0102d00 2024-03-28T22:36:13.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.324+0000 7f28e5c00700 1 --2- 172.21.15.43:0/2565404458 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f28e0102d00 0x7f28e01984d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f28d400e9f0 tx=0x7f28d400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:13.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.324+0000 7f28de000700 1 -- 172.21.15.43:0/2565404458 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d400c9c0 con 0x7f28e0102d00 2024-03-28T22:36:13.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.324+0000 7f28de000700 1 -- 172.21.15.43:0/2565404458 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28d400cb60 con 0x7f28e0102d00 2024-03-28T22:36:13.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.324+0000 7f28e8600700 1 -- 172.21.15.43:0/2565404458 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28e01a0420 con 0x7f28e0102d00 2024-03-28T22:36:13.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.324+0000 7f28de000700 1 -- 172.21.15.43:0/2565404458 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d4018680 con 0x7f28e0102d00 2024-03-28T22:36:13.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.324+0000 7f28e8600700 1 -- 172.21.15.43:0/2565404458 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28e01a0960 con 0x7f28e0102d00 2024-03-28T22:36:13.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.325+0000 7f28e8600700 1 -- 172.21.15.43:0/2565404458 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28e0068f00 con 0x7f28e0102d00 2024-03-28T22:36:13.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.327+0000 7f28de000700 1 -- 172.21.15.43:0/2565404458 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f28d4010430 con 0x7f28e0102d00 2024-03-28T22:36:13.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.329+0000 7f28de000700 1 --2- 172.21.15.43:0/2565404458 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f28c80a1e10 0x7f28c80a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:13.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.329+0000 7f28de000700 1 -- 172.21.15.43:0/2565404458 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f28d40ce730 con 0x7f28e0102d00 2024-03-28T22:36:13.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.329+0000 7f28e5200700 1 --2- 172.21.15.43:0/2565404458 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f28c80a1e10 0x7f28c80a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:13.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.330+0000 7f28e5200700 1 --2- 172.21.15.43:0/2565404458 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f28c80a1e10 0x7f28c80a42d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f28cc009750 tx=0x7f28cc005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:13.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.335+0000 7f28de000700 1 -- 172.21.15.43:0/2565404458 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f28d409b770 con 0x7f28e0102d00 2024-03-28T22:36:13.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.577+0000 7f28e8600700 1 -- 172.21.15.43:0/2565404458 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f28e0062680 con 0x7f28c80a1e10 2024-03-28T22:36:13.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.600+0000 7f28de000700 1 -- 172.21.15.43:0/2565404458 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f28e0062680 con 0x7f28c80a1e10 2024-03-28T22:36:13.599 INFO:teuthology.orchestra.run.smithi043.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-28T22:36:13.599 INFO:teuthology.orchestra.run.smithi043.stdout:mds.a smithi043 running (2m) 51s ago 2m 12.3M - 19.0.0-1899-g323d23dc bed8e21b47c6 cfd48870702c 2024-03-28T22:36:13.599 INFO:teuthology.orchestra.run.smithi043.stdout:mds.b smithi082 running (2m) 101s ago 2m 26.3M - 19.0.0-1899-g323d23dc bed8e21b47c6 bfb1218e0d70 2024-03-28T22:36:13.599 INFO:teuthology.orchestra.run.smithi043.stdout:mds.c smithi172 running (3m) 101s ago 3m 17.3M - 19.0.0-1899-g323d23dc bed8e21b47c6 46207b93f97a 2024-03-28T22:36:13.599 INFO:teuthology.orchestra.run.smithi043.stdout:mds.d smithi043 running (3m) 51s ago 3m 14.8M - 19.0.0-1899-g323d23dc bed8e21b47c6 04d255d93129 2024-03-28T22:36:13.599 INFO:teuthology.orchestra.run.smithi043.stdout:mds.e smithi082 running (3m) 101s ago 3m 18.1M - 19.0.0-1899-g323d23dc bed8e21b47c6 c3a5469d05e7 2024-03-28T22:36:13.599 INFO:teuthology.orchestra.run.smithi043.stdout:mds.f smithi172 running (3m) 101s ago 3m 17.0M - 19.0.0-1899-g323d23dc bed8e21b47c6 18384718f7a0 2024-03-28T22:36:13.599 INFO:teuthology.orchestra.run.smithi043.stdout:mds.g smithi043 running (2m) 51s ago 2m 12.3M - 19.0.0-1899-g323d23dc bed8e21b47c6 66fac975734c 2024-03-28T22:36:13.599 INFO:teuthology.orchestra.run.smithi043.stdout:mds.h smithi082 running (3m) 101s ago 3m 15.6M - 19.0.0-1899-g323d23dc bed8e21b47c6 95924c8f41dd 2024-03-28T22:36:13.599 INFO:teuthology.orchestra.run.smithi043.stdout:mds.i smithi172 running (3m) 101s ago 3m 21.7M - 19.0.0-1899-g323d23dc bed8e21b47c6 59dc64b08c15 2024-03-28T22:36:13.599 INFO:teuthology.orchestra.run.smithi043.stdout:mds.j smithi043 running (3m) 51s ago 3m 14.8M - 19.0.0-1899-g323d23dc bed8e21b47c6 ddb67d047772 2024-03-28T22:36:13.600 INFO:teuthology.orchestra.run.smithi043.stdout:mds.k smithi082 running (3m) 101s ago 3m 15.8M - 19.0.0-1899-g323d23dc bed8e21b47c6 19590bac1035 2024-03-28T22:36:13.600 INFO:teuthology.orchestra.run.smithi043.stdout:mds.l smithi172 running (3m) 101s ago 3m 15.9M - 19.0.0-1899-g323d23dc bed8e21b47c6 83dd91753c38 2024-03-28T22:36:13.600 INFO:teuthology.orchestra.run.smithi043.stdout:mgr.x smithi043 *:9283,8765 running (11m) 51s ago 11m 514M - 19.0.0-1899-g323d23dc bed8e21b47c6 9d6e46f876ef 2024-03-28T22:36:13.600 INFO:teuthology.orchestra.run.smithi043.stdout:mgr.y smithi082 *:8443,8765 running (9m) 101s ago 9m 453M - 19.0.0-1899-g323d23dc bed8e21b47c6 55e39a534d8a 2024-03-28T22:36:13.600 INFO:teuthology.orchestra.run.smithi043.stdout:mgr.z smithi172 *:8443,8765 running (9m) 101s ago 9m 453M - 19.0.0-1899-g323d23dc bed8e21b47c6 bd4fcfee3f63 2024-03-28T22:36:13.600 INFO:teuthology.orchestra.run.smithi043.stdout:mon.a smithi043 running (11m) 51s ago 11m 48.1M 2048M 19.0.0-1899-g323d23dc bed8e21b47c6 f2315b91a227 2024-03-28T22:36:13.600 INFO:teuthology.orchestra.run.smithi043.stdout:mon.b smithi082 running (9m) 101s ago 9m 41.4M 2048M 19.0.0-1899-g323d23dc bed8e21b47c6 94c6712ec6de 2024-03-28T22:36:13.600 INFO:teuthology.orchestra.run.smithi043.stdout:mon.c smithi172 running (9m) 101s ago 9m 40.5M 2048M 19.0.0-1899-g323d23dc bed8e21b47c6 2fa7788646ac 2024-03-28T22:36:13.600 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0 smithi043 running (8m) 51s ago 8m 46.2M 4273M 19.0.0-1899-g323d23dc bed8e21b47c6 98cb6dd46f33 2024-03-28T22:36:13.600 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1 smithi082 running (8m) 101s ago 8m 53.5M 4270M 19.0.0-1899-g323d23dc bed8e21b47c6 d4e635bc71ba 2024-03-28T22:36:13.600 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2 smithi172 running (8m) 101s ago 8m 55.4M 4270M 19.0.0-1899-g323d23dc bed8e21b47c6 6ff01883a429 2024-03-28T22:36:13.600 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3 smithi043 running (7m) 51s ago 7m 51.3M 4273M 19.0.0-1899-g323d23dc bed8e21b47c6 6ed163cc2e4a 2024-03-28T22:36:13.600 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4 smithi082 running (7m) 101s ago 7m 42.4M 4270M 19.0.0-1899-g323d23dc bed8e21b47c6 1dc40d4323ea 2024-03-28T22:36:13.600 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5 smithi172 running (6m) 101s ago 6m 45.4M 4270M 19.0.0-1899-g323d23dc bed8e21b47c6 bfefd97bf3eb 2024-03-28T22:36:13.600 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6 smithi043 running (6m) 51s ago 6m 42.4M 4273M 19.0.0-1899-g323d23dc bed8e21b47c6 a42963d43b0d 2024-03-28T22:36:13.601 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7 smithi082 running (5m) 101s ago 5m 52.1M 4270M 19.0.0-1899-g323d23dc bed8e21b47c6 8fdba4227919 2024-03-28T22:36:13.601 INFO:teuthology.orchestra.run.smithi043.stdout:osd.8 smithi172 running (5m) 101s ago 5m 43.2M 4270M 19.0.0-1899-g323d23dc bed8e21b47c6 0c117c9970f8 2024-03-28T22:36:13.601 INFO:teuthology.orchestra.run.smithi043.stdout:osd.9 smithi043 running (4m) 51s ago 4m 41.1M 4273M 19.0.0-1899-g323d23dc bed8e21b47c6 54750ead066a 2024-03-28T22:36:13.601 INFO:teuthology.orchestra.run.smithi043.stdout:osd.10 smithi082 running (4m) 101s ago 4m 44.4M 4270M 19.0.0-1899-g323d23dc bed8e21b47c6 c810979396e6 2024-03-28T22:36:13.601 INFO:teuthology.orchestra.run.smithi043.stdout:osd.11 smithi172 running (3m) 101s ago 3m 48.0M 4270M 19.0.0-1899-g323d23dc bed8e21b47c6 db0f58b43a9d 2024-03-28T22:36:13.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.603+0000 7f28e8600700 1 -- 172.21.15.43:0/2565404458 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f28c80a1e10 msgr2=0x7f28c80a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:13.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.603+0000 7f28e8600700 1 --2- 172.21.15.43:0/2565404458 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f28c80a1e10 0x7f28c80a42d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f28cc009750 tx=0x7f28cc005960 comp rx=0 tx=0).stop 2024-03-28T22:36:13.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.603+0000 7f28e8600700 1 -- 172.21.15.43:0/2565404458 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f28e0102d00 msgr2=0x7f28e01984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:13.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.603+0000 7f28e8600700 1 --2- 172.21.15.43:0/2565404458 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f28e0102d00 0x7f28e01984d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f28d400e9f0 tx=0x7f28d400ed00 comp rx=0 tx=0).stop 2024-03-28T22:36:13.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.603+0000 7f28e8600700 1 -- 172.21.15.43:0/2565404458 shutdown_connections 2024-03-28T22:36:13.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.603+0000 7f28e8600700 1 --2- 172.21.15.43:0/2565404458 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f28c80a1e10 0x7f28c80a42d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:13.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.603+0000 7f28e8600700 1 --2- 172.21.15.43:0/2565404458 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28e0107f60 0x7f28e019fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:13.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.603+0000 7f28e8600700 1 --2- 172.21.15.43:0/2565404458 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f28e0102d00 0x7f28e01984d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:13.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.603+0000 7f28e8600700 1 --2- 172.21.15.43:0/2565404458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28e0105630 0x7f28e0198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:13.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.603+0000 7f28e8600700 1 -- 172.21.15.43:0/2565404458 >> 172.21.15.43:0/2565404458 conn(0x7f28e00fc710 msgr2=0x7f28e01037e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:13.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.603+0000 7f28e8600700 1 -- 172.21.15.43:0/2565404458 shutdown_connections 2024-03-28T22:36:13.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:13.604+0000 7f28e8600700 1 -- 172.21.15.43:0/2565404458 wait complete. 2024-03-28T22:36:13.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:13 smithi082 ceph-mon[36471]: pgmap v408: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:36:13.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:13 smithi043 ceph-mon[31823]: pgmap v408: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:36:13.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:13 smithi172 ceph-mon[36566]: pgmap v408: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:36:14.399 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-28T22:36:14.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:14 smithi082 ceph-mon[36471]: from='client.25259 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:36:14.765 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:14 smithi043 ceph-mon[31823]: from='client.25259 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:36:14.874 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:36:14.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:14 smithi172 ceph-mon[36566]: from='client.25259 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:36:15.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:15 smithi082 ceph-mon[36471]: pgmap v409: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:36:15.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:15 smithi043 ceph-mon[31823]: pgmap v409: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:36:15.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:15 smithi172 ceph-mon[36566]: pgmap v409: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:36:17.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.253+0000 7f5fc1c00700 1 -- 172.21.15.43:0/3465333661 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5fbc1033b0 msgr2=0x7f5fbc103810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:17.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.253+0000 7f5fc1c00700 1 --2- 172.21.15.43:0/3465333661 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5fbc1033b0 0x7f5fbc103810 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4009920 tx=0x7f5fa4009c30 comp rx=0 tx=0).stop 2024-03-28T22:36:17.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.253+0000 7f5fc1c00700 1 -- 172.21.15.43:0/3465333661 shutdown_connections 2024-03-28T22:36:17.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.253+0000 7f5fc1c00700 1 --2- 172.21.15.43:0/3465333661 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fbc103d50 0x7f5fbc10a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:17.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.253+0000 7f5fc1c00700 1 --2- 172.21.15.43:0/3465333661 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5fbc1021b0 0x7f5fbc1025d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:17.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.253+0000 7f5fc1c00700 1 --2- 172.21.15.43:0/3465333661 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5fbc1033b0 0x7f5fbc103810 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:17.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.253+0000 7f5fc1c00700 1 -- 172.21.15.43:0/3465333661 >> 172.21.15.43:0/3465333661 conn(0x7f5fbc0fd750 msgr2=0x7f5fbc0ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:17.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.253+0000 7f5fc1c00700 1 -- 172.21.15.43:0/3465333661 shutdown_connections 2024-03-28T22:36:17.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.253+0000 7f5fc1c00700 1 -- 172.21.15.43:0/3465333661 wait complete. 2024-03-28T22:36:17.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.254+0000 7f5fc1c00700 1 Processor -- start 2024-03-28T22:36:17.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.255+0000 7f5fc1c00700 1 -- start start 2024-03-28T22:36:17.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.255+0000 7f5fc1c00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fbc1021b0 0x7f5fbc19a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:17.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.255+0000 7f5fc1c00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5fbc1033b0 0x7f5fbc19ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:17.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.255+0000 7f5fc1c00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5fbc103d50 0x7f5fbc1a1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:17.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.255+0000 7f5fc1c00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5fbc10dd80 con 0x7f5fbc1033b0 2024-03-28T22:36:17.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.255+0000 7f5fc1c00700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5fbc10dc00 con 0x7f5fbc103d50 2024-03-28T22:36:17.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.255+0000 7f5fc1c00700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f5fbc10df00 con 0x7f5fbc1021b0 2024-03-28T22:36:17.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.255+0000 7f5fbbe00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fbc1021b0 0x7f5fbc19a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:17.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.255+0000 7f5fbbe00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fbc1021b0 0x7f5fbc19a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40572/0 (socket says 172.21.15.43:40572) 2024-03-28T22:36:17.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.255+0000 7f5fbbe00700 1 -- 172.21.15.43:0/2360340550 learned_addr learned my addr 172.21.15.43:0/2360340550 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:17.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.255+0000 7f5fc1200700 1 --2- 172.21.15.43:0/2360340550 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5fbc103d50 0x7f5fbc1a1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:17.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.256+0000 7f5fb3400700 1 --2- 172.21.15.43:0/2360340550 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5fbc1033b0 0x7f5fbc19ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:17.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.256+0000 7f5fbbe00700 1 -- 172.21.15.43:0/2360340550 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5fbc103d50 msgr2=0x7f5fbc1a1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:17.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.256+0000 7f5fbbe00700 1 --2- 172.21.15.43:0/2360340550 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5fbc103d50 0x7f5fbc1a1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:17.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.256+0000 7f5fbbe00700 1 -- 172.21.15.43:0/2360340550 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5fbc1033b0 msgr2=0x7f5fbc19ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:17.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.256+0000 7f5fbbe00700 1 --2- 172.21.15.43:0/2360340550 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5fbc1033b0 0x7f5fbc19ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:17.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.256+0000 7f5fbbe00700 1 -- 172.21.15.43:0/2360340550 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fbc1a2240 con 0x7f5fbc1021b0 2024-03-28T22:36:17.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.256+0000 7f5fc1200700 1 --2- 172.21.15.43:0/2360340550 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5fbc103d50 0x7f5fbc1a1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:17.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.256+0000 7f5fbbe00700 1 --2- 172.21.15.43:0/2360340550 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fbc1021b0 0x7f5fbc19a5e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5fac009f40 tx=0x7f5fac00c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:17.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.256+0000 7f5fb9600700 1 -- 172.21.15.43:0/2360340550 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fac0092e0 con 0x7f5fbc1021b0 2024-03-28T22:36:17.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.256+0000 7f5fc1c00700 1 -- 172.21.15.43:0/2360340550 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fbc1a24d0 con 0x7f5fbc1021b0 2024-03-28T22:36:17.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.256+0000 7f5fb9600700 1 -- 172.21.15.43:0/2360340550 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5fac003a00 con 0x7f5fbc1021b0 2024-03-28T22:36:17.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.256+0000 7f5fb9600700 1 -- 172.21.15.43:0/2360340550 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fac004680 con 0x7f5fbc1021b0 2024-03-28T22:36:17.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.257+0000 7f5fc1c00700 1 -- 172.21.15.43:0/2360340550 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fbc1a2a10 con 0x7f5fbc1021b0 2024-03-28T22:36:17.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.258+0000 7f5fc1c00700 1 -- 172.21.15.43:0/2360340550 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fbc068f00 con 0x7f5fbc1021b0 2024-03-28T22:36:17.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.260+0000 7f5fb9600700 1 -- 172.21.15.43:0/2360340550 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f5fac01b030 con 0x7f5fbc1021b0 2024-03-28T22:36:17.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.261+0000 7f5fb9600700 1 --2- 172.21.15.43:0/2360340550 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5fa00a1e10 0x7f5fa00a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:17.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.261+0000 7f5fb9600700 1 -- 172.21.15.43:0/2360340550 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f5fac0ce0d0 con 0x7f5fbc1021b0 2024-03-28T22:36:17.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.261+0000 7f5fb3400700 1 --2- 172.21.15.43:0/2360340550 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5fa00a1e10 0x7f5fa00a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:17.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.262+0000 7f5fb3400700 1 --2- 172.21.15.43:0/2360340550 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5fa00a1e10 0x7f5fa00a42d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5fa400fca0 tx=0x7f5fa4019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:17.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.270+0000 7f5fb9600700 1 -- 172.21.15.43:0/2360340550 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5fac022020 con 0x7f5fbc1021b0 2024-03-28T22:36:17.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.514+0000 7f5fc1c00700 1 -- 172.21.15.43:0/2360340550 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f5fbc075b90 con 0x7f5fa00a1e10 2024-03-28T22:36:17.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.531+0000 7f5fb9600700 1 -- 172.21.15.43:0/2360340550 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f5fbc075b90 con 0x7f5fa00a1e10 2024-03-28T22:36:17.530 INFO:teuthology.orchestra.run.smithi043.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-28T22:36:17.530 INFO:teuthology.orchestra.run.smithi043.stdout:mds.all 12/12 105s ago 3m smithi043=a;smithi043=d;smithi043=g;smithi043=j;smithi082=b;smithi082=e;smithi082=h;smithi082=k;smithi172=c;smithi172=f;smithi172=i;smithi172=l;count:12 2024-03-28T22:36:17.530 INFO:teuthology.orchestra.run.smithi043.stdout:mgr 3/3 105s ago 9m smithi043=x;smithi082=y;smithi172=z;count:3 2024-03-28T22:36:17.531 INFO:teuthology.orchestra.run.smithi043.stdout:mon 3/3 105s ago 10m smithi043:172.21.15.43=a;smithi082:172.21.15.82=b;smithi172:172.21.15.172=c;count:3 2024-03-28T22:36:17.531 INFO:teuthology.orchestra.run.smithi043.stdout:osd 12 105s ago - 2024-03-28T22:36:17.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.534+0000 7f5fc1c00700 1 -- 172.21.15.43:0/2360340550 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5fa00a1e10 msgr2=0x7f5fa00a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:17.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.534+0000 7f5fc1c00700 1 --2- 172.21.15.43:0/2360340550 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5fa00a1e10 0x7f5fa00a42d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5fa400fca0 tx=0x7f5fa4019040 comp rx=0 tx=0).stop 2024-03-28T22:36:17.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.534+0000 7f5fc1c00700 1 -- 172.21.15.43:0/2360340550 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fbc1021b0 msgr2=0x7f5fbc19a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:17.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.534+0000 7f5fc1c00700 1 --2- 172.21.15.43:0/2360340550 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fbc1021b0 0x7f5fbc19a5e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5fac009f40 tx=0x7f5fac00c450 comp rx=0 tx=0).stop 2024-03-28T22:36:17.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.534+0000 7f5fc1c00700 1 -- 172.21.15.43:0/2360340550 shutdown_connections 2024-03-28T22:36:17.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.534+0000 7f5fc1c00700 1 --2- 172.21.15.43:0/2360340550 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5fa00a1e10 0x7f5fa00a42d0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:17.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.534+0000 7f5fc1c00700 1 --2- 172.21.15.43:0/2360340550 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5fbc103d50 0x7f5fbc1a1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:17.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.534+0000 7f5fc1c00700 1 --2- 172.21.15.43:0/2360340550 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5fbc1021b0 0x7f5fbc19a5e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:17.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.534+0000 7f5fc1c00700 1 --2- 172.21.15.43:0/2360340550 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5fbc1033b0 0x7f5fbc19ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:17.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.534+0000 7f5fc1c00700 1 -- 172.21.15.43:0/2360340550 >> 172.21.15.43:0/2360340550 conn(0x7f5fbc0fd750 msgr2=0x7f5fbc074470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:17.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.535+0000 7f5fc1c00700 1 -- 172.21.15.43:0/2360340550 shutdown_connections 2024-03-28T22:36:17.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:17.535+0000 7f5fc1c00700 1 -- 172.21.15.43:0/2360340550 wait complete. 2024-03-28T22:36:17.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:17 smithi082 ceph-mon[36471]: pgmap v410: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T22:36:17.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:17 smithi043 ceph-mon[31823]: pgmap v410: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T22:36:17.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:17 smithi172 ceph-mon[36566]: pgmap v410: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T22:36:18.447 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-28T22:36:18.526 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:18 smithi043 ceph-mon[31823]: from='client.25273 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:36:18.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:18 smithi172 ceph-mon[36566]: from='client.25273 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:36:18.927 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:36:18.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:18 smithi082 ceph-mon[36471]: from='client.25273 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:36:19.683 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:19 smithi043 ceph-mon[31823]: pgmap v411: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:36:19.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:19 smithi172 ceph-mon[36566]: pgmap v411: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:36:19.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:19 smithi082 ceph-mon[36471]: pgmap v411: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:36:21.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.232+0000 7fecca000700 1 -- 172.21.15.43:0/3923606674 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fecc41044c0 msgr2=0x7fecc41048e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:21.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.232+0000 7fecca000700 1 --2- 172.21.15.43:0/3923606674 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fecc41044c0 0x7fecc41048e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fecb4009990 tx=0x7fecb4009ca0 comp rx=0 tx=0).stop 2024-03-28T22:36:21.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.232+0000 7fecca000700 1 -- 172.21.15.43:0/3923606674 shutdown_connections 2024-03-28T22:36:21.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.232+0000 7fecca000700 1 --2- 172.21.15.43:0/3923606674 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fecc4106060 0x7fecc410c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:21.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.232+0000 7fecca000700 1 --2- 172.21.15.43:0/3923606674 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fecc41044c0 0x7fecc41048e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:21.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.232+0000 7fecca000700 1 --2- 172.21.15.43:0/3923606674 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fecc41056c0 0x7fecc4105b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:21.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.232+0000 7fecca000700 1 -- 172.21.15.43:0/3923606674 >> 172.21.15.43:0/3923606674 conn(0x7fecc40ffa60 msgr2=0x7fecc4101ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:21.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.232+0000 7fecca000700 1 -- 172.21.15.43:0/3923606674 shutdown_connections 2024-03-28T22:36:21.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.233+0000 7fecca000700 1 -- 172.21.15.43:0/3923606674 wait complete. 2024-03-28T22:36:21.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.234+0000 7fecca000700 1 Processor -- start 2024-03-28T22:36:21.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.234+0000 7fecca000700 1 -- start start 2024-03-28T22:36:21.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.234+0000 7fecca000700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fecc41044c0 0x7fecc419a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:21.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.234+0000 7fecca000700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fecc41056c0 0x7fecc419ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:21.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.234+0000 7fecca000700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fecc4106060 0x7fecc41a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:21.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.234+0000 7fecca000700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fecc41100f0 con 0x7fecc41044c0 2024-03-28T22:36:21.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.234+0000 7fecca000700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fecc410ff70 con 0x7fecc4106060 2024-03-28T22:36:21.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.234+0000 7fecca000700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fecc4110270 con 0x7fecc41056c0 2024-03-28T22:36:21.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.234+0000 7fecc2a00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fecc41056c0 0x7fecc419ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:21.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.234+0000 7fecc3e00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fecc4106060 0x7fecc41a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:21.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.235+0000 7fecc2a00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fecc41056c0 0x7fecc419ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40600/0 (socket says 172.21.15.43:40600) 2024-03-28T22:36:21.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.235+0000 7fecc2a00700 1 -- 172.21.15.43:0/1589723878 learned_addr learned my addr 172.21.15.43:0/1589723878 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:21.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.235+0000 7fecc3400700 1 --2- 172.21.15.43:0/1589723878 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fecc41044c0 0x7fecc419a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:21.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.235+0000 7fecc3e00700 1 -- 172.21.15.43:0/1589723878 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fecc41056c0 msgr2=0x7fecc419ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:21.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.235+0000 7fecc3e00700 1 --2- 172.21.15.43:0/1589723878 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fecc41056c0 0x7fecc419ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:21.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.235+0000 7fecc3e00700 1 -- 172.21.15.43:0/1589723878 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fecc41044c0 msgr2=0x7fecc419a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:21.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.235+0000 7fecc3e00700 1 --2- 172.21.15.43:0/1589723878 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fecc41044c0 0x7fecc419a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:21.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.235+0000 7fecc3e00700 1 -- 172.21.15.43:0/1589723878 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecc41a2390 con 0x7fecc4106060 2024-03-28T22:36:21.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.235+0000 7fecc3e00700 1 --2- 172.21.15.43:0/1589723878 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fecc4106060 0x7fecc41a1cf0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fecbc00efc0 tx=0x7fecbc00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:21.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.235+0000 7fecbbe00700 1 -- 172.21.15.43:0/1589723878 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecbc0092e0 con 0x7fecc4106060 2024-03-28T22:36:21.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.235+0000 7fecbbe00700 1 -- 172.21.15.43:0/1589723878 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecbc004bb0 con 0x7fecc4106060 2024-03-28T22:36:21.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.235+0000 7fecca000700 1 -- 172.21.15.43:0/1589723878 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecc41a2620 con 0x7fecc4106060 2024-03-28T22:36:21.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.235+0000 7fecbbe00700 1 -- 172.21.15.43:0/1589723878 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecbc004280 con 0x7fecc4106060 2024-03-28T22:36:21.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.236+0000 7fecca000700 1 -- 172.21.15.43:0/1589723878 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecc41a2b60 con 0x7fecc4106060 2024-03-28T22:36:21.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.237+0000 7fecca000700 1 -- 172.21.15.43:0/1589723878 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecc4068f00 con 0x7fecc4106060 2024-03-28T22:36:21.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.239+0000 7fecbbe00700 1 -- 172.21.15.43:0/1589723878 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fecbc01b030 con 0x7fecc4106060 2024-03-28T22:36:21.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.240+0000 7fecbbe00700 1 --2- 172.21.15.43:0/1589723878 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7feca80a1e90 0x7feca80a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:21.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.240+0000 7fecbbe00700 1 -- 172.21.15.43:0/1589723878 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7fecbc0cdd40 con 0x7fecc4106060 2024-03-28T22:36:21.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.240+0000 7fecc3400700 1 --2- 172.21.15.43:0/1589723878 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7feca80a1e90 0x7feca80a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:21.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.241+0000 7fecc3400700 1 --2- 172.21.15.43:0/1589723878 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7feca80a1e90 0x7feca80a4350 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fecb400b2e0 tx=0x7fecb401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:21.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.244+0000 7fecbbe00700 1 -- 172.21.15.43:0/1589723878 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fecbc022020 con 0x7fecc4106060 2024-03-28T22:36:21.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.478+0000 7fecca000700 1 -- 172.21.15.43:0/1589723878 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fecc4062680 con 0x7feca80a1e90 2024-03-28T22:36:21.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.481+0000 7fecbbe00700 1 -- 172.21.15.43:0/1589723878 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fecc4062680 con 0x7feca80a1e90 2024-03-28T22:36:21.479 INFO:teuthology.orchestra.run.smithi043.stdout:HOST ADDR LABELS STATUS 2024-03-28T22:36:21.479 INFO:teuthology.orchestra.run.smithi043.stdout:smithi043 172.21.15.43 2024-03-28T22:36:21.479 INFO:teuthology.orchestra.run.smithi043.stdout:smithi082 172.21.15.82 2024-03-28T22:36:21.479 INFO:teuthology.orchestra.run.smithi043.stdout:smithi172 172.21.15.172 2024-03-28T22:36:21.479 INFO:teuthology.orchestra.run.smithi043.stdout:3 hosts in cluster 2024-03-28T22:36:21.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.484+0000 7fecca000700 1 -- 172.21.15.43:0/1589723878 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7feca80a1e90 msgr2=0x7feca80a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:21.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.484+0000 7fecca000700 1 --2- 172.21.15.43:0/1589723878 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7feca80a1e90 0x7feca80a4350 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fecb400b2e0 tx=0x7fecb401a040 comp rx=0 tx=0).stop 2024-03-28T22:36:21.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.484+0000 7fecca000700 1 -- 172.21.15.43:0/1589723878 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fecc4106060 msgr2=0x7fecc41a1cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:21.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.484+0000 7fecca000700 1 --2- 172.21.15.43:0/1589723878 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fecc4106060 0x7fecc41a1cf0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fecbc00efc0 tx=0x7fecbc00c3a0 comp rx=0 tx=0).stop 2024-03-28T22:36:21.482 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:21 smithi043 ceph-mon[31823]: pgmap v412: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-28T22:36:21.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.484+0000 7fecca000700 1 -- 172.21.15.43:0/1589723878 shutdown_connections 2024-03-28T22:36:21.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.484+0000 7fecca000700 1 --2- 172.21.15.43:0/1589723878 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7feca80a1e90 0x7feca80a4350 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:21.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.484+0000 7fecca000700 1 --2- 172.21.15.43:0/1589723878 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fecc4106060 0x7fecc41a1cf0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:21.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.484+0000 7fecca000700 1 --2- 172.21.15.43:0/1589723878 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fecc41044c0 0x7fecc419a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:21.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.484+0000 7fecca000700 1 --2- 172.21.15.43:0/1589723878 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fecc41056c0 0x7fecc419ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:21.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.484+0000 7fecca000700 1 -- 172.21.15.43:0/1589723878 >> 172.21.15.43:0/1589723878 conn(0x7fecc40ffa60 msgr2=0x7fecc41089c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:21.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.484+0000 7fecca000700 1 -- 172.21.15.43:0/1589723878 shutdown_connections 2024-03-28T22:36:21.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:21.484+0000 7fecca000700 1 -- 172.21.15.43:0/1589723878 wait complete. 2024-03-28T22:36:21.749 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:21 smithi082 ceph-mon[36471]: pgmap v412: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-28T22:36:21.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:21 smithi172 ceph-mon[36566]: pgmap v412: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-28T22:36:22.272 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-28T22:36:22.766 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:36:22.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:22 smithi043 ceph-mon[31823]: from='client.25271 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:36:22.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:22 smithi172 ceph-mon[36566]: from='client.25271 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:36:22.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:22 smithi082 ceph-mon[36471]: from='client.25271 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:36:23.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:23 smithi172 ceph-mon[36566]: pgmap v413: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:36:23.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:23 smithi082 ceph-mon[36471]: pgmap v413: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:36:23.998 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:23 smithi043 ceph-mon[31823]: pgmap v413: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:36:25.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.078+0000 7f76bba00700 1 -- 172.21.15.43:0/3861377650 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76b41056c0 msgr2=0x7f76b4105b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:25.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.078+0000 7f76bba00700 1 --2- 172.21.15.43:0/3861377650 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76b41056c0 0x7f76b4105b20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f76a0009990 tx=0x7f76a0009ca0 comp rx=0 tx=0).stop 2024-03-28T22:36:25.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.078+0000 7f76bba00700 1 -- 172.21.15.43:0/3861377650 shutdown_connections 2024-03-28T22:36:25.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.078+0000 7f76bba00700 1 --2- 172.21.15.43:0/3861377650 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76b4106060 0x7f76b410c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:25.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.078+0000 7f76bba00700 1 --2- 172.21.15.43:0/3861377650 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f76b41044c0 0x7f76b41048e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:25.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.078+0000 7f76bba00700 1 --2- 172.21.15.43:0/3861377650 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76b41056c0 0x7f76b4105b20 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:25.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.078+0000 7f76bba00700 1 -- 172.21.15.43:0/3861377650 >> 172.21.15.43:0/3861377650 conn(0x7f76b40ffa60 msgr2=0x7f76b4101ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:25.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.078+0000 7f76bba00700 1 -- 172.21.15.43:0/3861377650 shutdown_connections 2024-03-28T22:36:25.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.078+0000 7f76bba00700 1 -- 172.21.15.43:0/3861377650 wait complete. 2024-03-28T22:36:25.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.079+0000 7f76bba00700 1 Processor -- start 2024-03-28T22:36:25.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.080+0000 7f76bba00700 1 -- start start 2024-03-28T22:36:25.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.080+0000 7f76bba00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f76b41044c0 0x7f76b419a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:25.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.080+0000 7f76bba00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76b41056c0 0x7f76b419ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:25.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.080+0000 7f76bba00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76b4106060 0x7f76b41a1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:25.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.080+0000 7f76bba00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f76b41100e0 con 0x7f76b41056c0 2024-03-28T22:36:25.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.080+0000 7f76bba00700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f76b410ff60 con 0x7f76b41044c0 2024-03-28T22:36:25.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.080+0000 7f76bba00700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f76b4110260 con 0x7f76b4106060 2024-03-28T22:36:25.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.080+0000 7f76b9000700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f76b41044c0 0x7f76b419a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:25.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.080+0000 7f76b9000700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f76b41044c0 0x7f76b419a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33826/0 (socket says 172.21.15.43:33826) 2024-03-28T22:36:25.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.080+0000 7f76b9a00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76b4106060 0x7f76b41a1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:25.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.080+0000 7f76b9000700 1 -- 172.21.15.43:0/2267126387 learned_addr learned my addr 172.21.15.43:0/2267126387 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:25.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.080+0000 7f76b3e00700 1 --2- 172.21.15.43:0/2267126387 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76b41056c0 0x7f76b419ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:25.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.081+0000 7f76b9000700 1 -- 172.21.15.43:0/2267126387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76b4106060 msgr2=0x7f76b41a1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:25.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.081+0000 7f76b9000700 1 --2- 172.21.15.43:0/2267126387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76b4106060 0x7f76b41a1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:25.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.081+0000 7f76b9000700 1 -- 172.21.15.43:0/2267126387 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76b41056c0 msgr2=0x7f76b419ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:25.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.081+0000 7f76b9000700 1 --2- 172.21.15.43:0/2267126387 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76b41056c0 0x7f76b419ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:25.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.081+0000 7f76b9000700 1 -- 172.21.15.43:0/2267126387 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76b41a2370 con 0x7f76b41044c0 2024-03-28T22:36:25.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.081+0000 7f76b9a00700 1 --2- 172.21.15.43:0/2267126387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76b4106060 0x7f76b41a1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:25.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.081+0000 7f76b9000700 1 --2- 172.21.15.43:0/2267126387 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f76b41044c0 0x7f76b419a710 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f76a800e970 tx=0x7f76a800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:25.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.081+0000 7f76b1600700 1 -- 172.21.15.43:0/2267126387 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76a8004220 con 0x7f76b41044c0 2024-03-28T22:36:25.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.082+0000 7f76b1600700 1 -- 172.21.15.43:0/2267126387 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76a80043c0 con 0x7f76b41044c0 2024-03-28T22:36:25.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.082+0000 7f76bba00700 1 -- 172.21.15.43:0/2267126387 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76b41a2660 con 0x7f76b41044c0 2024-03-28T22:36:25.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.082+0000 7f76b1600700 1 -- 172.21.15.43:0/2267126387 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76a8003df0 con 0x7f76b41044c0 2024-03-28T22:36:25.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.082+0000 7f76bba00700 1 -- 172.21.15.43:0/2267126387 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76b41a2bf0 con 0x7f76b41044c0 2024-03-28T22:36:25.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.083+0000 7f76bba00700 1 -- 172.21.15.43:0/2267126387 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76b4068f00 con 0x7f76b41044c0 2024-03-28T22:36:25.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.085+0000 7f76b1600700 1 -- 172.21.15.43:0/2267126387 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f76a800c7e0 con 0x7f76b41044c0 2024-03-28T22:36:25.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.086+0000 7f76b1600700 1 --2- 172.21.15.43:0/2267126387 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f769c0a1e90 0x7f769c0a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:25.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.087+0000 7f76b1600700 1 -- 172.21.15.43:0/2267126387 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f76a80ce2b0 con 0x7f76b41044c0 2024-03-28T22:36:25.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.087+0000 7f76b3e00700 1 --2- 172.21.15.43:0/2267126387 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f769c0a1e90 0x7f769c0a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:25.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.088+0000 7f76b3e00700 1 --2- 172.21.15.43:0/2267126387 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f769c0a1e90 0x7f769c0a4350 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f76a000b2e0 tx=0x7f76a000b8f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:25.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.095+0000 7f76b1600700 1 -- 172.21.15.43:0/2267126387 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f76a809b2f0 con 0x7f76b41044c0 2024-03-28T22:36:25.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.338+0000 7f76bba00700 1 -- 172.21.15.43:0/2267126387 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f76b4062680 con 0x7f769c0a1e90 2024-03-28T22:36:25.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.341+0000 7f76b1600700 1 -- 172.21.15.43:0/2267126387 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f76b4062680 con 0x7f769c0a1e90 2024-03-28T22:36:25.339 INFO:teuthology.orchestra.run.smithi043.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-28T22:36:25.339 INFO:teuthology.orchestra.run.smithi043.stdout:smithi043 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB751300GH480DGN 447G No 4m ago Has a FileSystem, LVM detected 2024-03-28T22:36:25.339 INFO:teuthology.orchestra.run.smithi043.stdout:smithi082 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400SD400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-03-28T22:36:25.339 INFO:teuthology.orchestra.run.smithi043.stdout:smithi172 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6233005M400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-28T22:36:25.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.344+0000 7f76bba00700 1 -- 172.21.15.43:0/2267126387 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f769c0a1e90 msgr2=0x7f769c0a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:25.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.344+0000 7f76bba00700 1 --2- 172.21.15.43:0/2267126387 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f769c0a1e90 0x7f769c0a4350 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f76a000b2e0 tx=0x7f76a000b8f0 comp rx=0 tx=0).stop 2024-03-28T22:36:25.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.344+0000 7f76bba00700 1 -- 172.21.15.43:0/2267126387 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f76b41044c0 msgr2=0x7f76b419a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:25.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.344+0000 7f76bba00700 1 --2- 172.21.15.43:0/2267126387 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f76b41044c0 0x7f76b419a710 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f76a800e970 tx=0x7f76a800ec80 comp rx=0 tx=0).stop 2024-03-28T22:36:25.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.344+0000 7f76bba00700 1 -- 172.21.15.43:0/2267126387 shutdown_connections 2024-03-28T22:36:25.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.344+0000 7f76bba00700 1 --2- 172.21.15.43:0/2267126387 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f769c0a1e90 0x7f769c0a4350 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:25.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.344+0000 7f76bba00700 1 --2- 172.21.15.43:0/2267126387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76b4106060 0x7f76b41a1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:25.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.344+0000 7f76bba00700 1 --2- 172.21.15.43:0/2267126387 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f76b41044c0 0x7f76b419a710 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:25.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.344+0000 7f76bba00700 1 --2- 172.21.15.43:0/2267126387 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f76b41056c0 0x7f76b419ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:25.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.344+0000 7f76bba00700 1 -- 172.21.15.43:0/2267126387 >> 172.21.15.43:0/2267126387 conn(0x7f76b40ffa60 msgr2=0x7f76b4108960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:25.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.344+0000 7f76bba00700 1 -- 172.21.15.43:0/2267126387 shutdown_connections 2024-03-28T22:36:25.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:25.344+0000 7f76bba00700 1 -- 172.21.15.43:0/2267126387 wait complete. 2024-03-28T22:36:25.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:25 smithi172 ceph-mon[36566]: pgmap v414: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:36:25.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:25 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:36:25.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:25 smithi082 ceph-mon[36471]: pgmap v414: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:36:25.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:25 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:36:26.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:25 smithi043 ceph-mon[31823]: pgmap v414: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T22:36:26.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:25 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:36:26.250 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-28T22:36:26.260 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi043.front.sepia.ceph.com 2024-03-28T22:36:26.260 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-28T22:36:26.729 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:36:26.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:26 smithi082 ceph-mon[36471]: from='client.25277 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:36:26.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:26 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:26.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:26 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:26.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:26 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:26.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:26 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:26.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:26 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:26.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:26 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:27.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:26 smithi043 ceph-mon[31823]: from='client.25277 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:36:27.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:26 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:27.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:26 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:27.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:26 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:27.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:26 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:27.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:26 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:27.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:26 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:27.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:26 smithi172 ceph-mon[36566]: from='client.25277 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:36:27.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:26 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:27.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:26 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:27.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:26 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:27.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:26 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:27.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:26 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:27.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:26 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:28.139 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:27 smithi043 ceph-mon[31823]: pgmap v415: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-03-28T22:36:28.139 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:27 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:28.139 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:27 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:28.139 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:27 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:28.139 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:27 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:28.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:27 smithi172 ceph-mon[36566]: pgmap v415: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-03-28T22:36:28.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:27 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:28.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:27 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:28.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:27 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:28.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:27 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:28.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:27 smithi082 ceph-mon[36471]: pgmap v415: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-03-28T22:36:28.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:27 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:28.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:27 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:28.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:27 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:28.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:27 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:29.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.108+0000 7f397b800700 1 -- 172.21.15.43:0/4167426858 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f397410a150 msgr2=0x7f397410c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:29.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.108+0000 7f397b800700 1 --2- 172.21.15.43:0/4167426858 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f397410a150 0x7f397410c5e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f396c009f90 tx=0x7f396c00b330 comp rx=0 tx=0).stop 2024-03-28T22:36:29.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.108+0000 7f397b800700 1 -- 172.21.15.43:0/4167426858 shutdown_connections 2024-03-28T22:36:29.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.108+0000 7f397b800700 1 --2- 172.21.15.43:0/4167426858 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f397410a150 0x7f397410c5e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:29.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.108+0000 7f397b800700 1 --2- 172.21.15.43:0/4167426858 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3974076270 0x7f39740746d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:29.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.108+0000 7f397b800700 1 --2- 172.21.15.43:0/4167426858 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3974074ca0 0x7f3974075120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:29.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.108+0000 7f397b800700 1 -- 172.21.15.43:0/4167426858 >> 172.21.15.43:0/4167426858 conn(0x7f39740fd7e0 msgr2=0x7f39740ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:29.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.109+0000 7f397b800700 1 -- 172.21.15.43:0/4167426858 shutdown_connections 2024-03-28T22:36:29.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.109+0000 7f397b800700 1 -- 172.21.15.43:0/4167426858 wait complete. 2024-03-28T22:36:29.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.110+0000 7f397b800700 1 Processor -- start 2024-03-28T22:36:29.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.110+0000 7f397b800700 1 -- start start 2024-03-28T22:36:29.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.110+0000 7f397b800700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3974074ca0 0x7f397419a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:29.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.110+0000 7f397b800700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3974076270 0x7f397419abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:29.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.110+0000 7f397b800700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f397410a150 0x7f39741a1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:29.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.110+0000 7f397b800700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f39741100c0 con 0x7f397410a150 2024-03-28T22:36:29.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.110+0000 7f397b800700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f397410ff40 con 0x7f3974074ca0 2024-03-28T22:36:29.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.110+0000 7f397b800700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f3974110240 con 0x7f3974076270 2024-03-28T22:36:29.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.111+0000 7f3973e00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3974076270 0x7f397419abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:29.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.111+0000 7f3978e00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3974074ca0 0x7f397419a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:29.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.111+0000 7f3973e00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3974076270 0x7f397419abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:34820/0 (socket says 172.21.15.43:34820) 2024-03-28T22:36:29.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.111+0000 7f3978e00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3974074ca0 0x7f397419a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33852/0 (socket says 172.21.15.43:33852) 2024-03-28T22:36:29.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.111+0000 7f3973e00700 1 -- 172.21.15.43:0/1066245001 learned_addr learned my addr 172.21.15.43:0/1066245001 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:29.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.111+0000 7f3979800700 1 --2- 172.21.15.43:0/1066245001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f397410a150 0x7f39741a1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:29.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.111+0000 7f3973e00700 1 -- 172.21.15.43:0/1066245001 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3974074ca0 msgr2=0x7f397419a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:29.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.111+0000 7f3973e00700 1 --2- 172.21.15.43:0/1066245001 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3974074ca0 0x7f397419a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:29.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.111+0000 7f3973e00700 1 -- 172.21.15.43:0/1066245001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f397410a150 msgr2=0x7f39741a1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:29.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.111+0000 7f3973e00700 1 --2- 172.21.15.43:0/1066245001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f397410a150 0x7f39741a1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:29.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.111+0000 7f3973e00700 1 -- 172.21.15.43:0/1066245001 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39741a22d0 con 0x7f3974076270 2024-03-28T22:36:29.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.111+0000 7f3978e00700 1 --2- 172.21.15.43:0/1066245001 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3974074ca0 0x7f397419a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:29.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.111+0000 7f3979800700 1 --2- 172.21.15.43:0/1066245001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f397410a150 0x7f39741a1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:36:29.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.112+0000 7f3973e00700 1 --2- 172.21.15.43:0/1066245001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3974076270 0x7f397419abb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f396000d9b0 tx=0x7f396000dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:29.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.112+0000 7f3971600700 1 -- 172.21.15.43:0/1066245001 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3960017030 con 0x7f3974076270 2024-03-28T22:36:29.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.112+0000 7f3971600700 1 -- 172.21.15.43:0/1066245001 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3960004500 con 0x7f3974076270 2024-03-28T22:36:29.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.112+0000 7f3971600700 1 -- 172.21.15.43:0/1066245001 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3960003aa0 con 0x7f3974076270 2024-03-28T22:36:29.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.112+0000 7f397b800700 1 -- 172.21.15.43:0/1066245001 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39741a25c0 con 0x7f3974076270 2024-03-28T22:36:29.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.112+0000 7f397b800700 1 -- 172.21.15.43:0/1066245001 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39741a2ab0 con 0x7f3974076270 2024-03-28T22:36:29.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.113+0000 7f397b800700 1 -- 172.21.15.43:0/1066245001 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3974068f00 con 0x7f3974076270 2024-03-28T22:36:29.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.116+0000 7f3971600700 1 -- 172.21.15.43:0/1066245001 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f396001a070 con 0x7f3974076270 2024-03-28T22:36:29.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.117+0000 7f3971600700 1 --2- 172.21.15.43:0/1066245001 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f395c0a1e10 0x7f395c0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:29.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.117+0000 7f3971600700 1 -- 172.21.15.43:0/1066245001 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f39600cde40 con 0x7f3974076270 2024-03-28T22:36:29.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.117+0000 7f3978e00700 1 --2- 172.21.15.43:0/1066245001 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f395c0a1e10 0x7f395c0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:29.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.118+0000 7f3978e00700 1 --2- 172.21.15.43:0/1066245001 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f395c0a1e10 0x7f395c0a42d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3968009750 tx=0x7f3968009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:29.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.119+0000 7f3971600700 1 -- 172.21.15.43:0/1066245001 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f396009ae80 con 0x7f3974076270 2024-03-28T22:36:29.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.400+0000 7f397b800700 1 -- 172.21.15.43:0/1066245001 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f397402db60 con 0x7f3974076270 2024-03-28T22:36:29.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.401+0000 7f3971600700 1 -- 172.21.15.43:0/1066245001 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) ==== 76+0+3092 (secure 0 0 0) 0x7f396009a780 con 0x7f3974076270 2024-03-28T22:36:29.400 INFO:teuthology.orchestra.run.smithi043.stdout:e28 2024-03-28T22:36:29.400 INFO:teuthology.orchestra.run.smithi043.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T22:36:29.400 INFO:teuthology.orchestra.run.smithi043.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-28T22:36:29.401 INFO:teuthology.orchestra.run.smithi043.stdout:legacy client fscid: 1 2024-03-28T22:36:29.401 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:36:29.401 INFO:teuthology.orchestra.run.smithi043.stdout:Filesystem 'cephfs' (1) 2024-03-28T22:36:29.401 INFO:teuthology.orchestra.run.smithi043.stdout:fs_name cephfs 2024-03-28T22:36:29.401 INFO:teuthology.orchestra.run.smithi043.stdout:epoch 28 2024-03-28T22:36:29.401 INFO:teuthology.orchestra.run.smithi043.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-28T22:36:29.401 INFO:teuthology.orchestra.run.smithi043.stdout:created 2024-03-28T22:33:28.906396+0000 2024-03-28T22:36:29.401 INFO:teuthology.orchestra.run.smithi043.stdout:modified 2024-03-28T22:34:06.174761+0000 2024-03-28T22:36:29.401 INFO:teuthology.orchestra.run.smithi043.stdout:tableserver 0 2024-03-28T22:36:29.401 INFO:teuthology.orchestra.run.smithi043.stdout:root 0 2024-03-28T22:36:29.401 INFO:teuthology.orchestra.run.smithi043.stdout:session_timeout 300 2024-03-28T22:36:29.401 INFO:teuthology.orchestra.run.smithi043.stdout:session_autoclose 300 2024-03-28T22:36:29.401 INFO:teuthology.orchestra.run.smithi043.stdout:max_file_size 1099511627776 2024-03-28T22:36:29.402 INFO:teuthology.orchestra.run.smithi043.stdout:max_xattr_size 65536 2024-03-28T22:36:29.402 INFO:teuthology.orchestra.run.smithi043.stdout:required_client_features {} 2024-03-28T22:36:29.402 INFO:teuthology.orchestra.run.smithi043.stdout:last_failure 0 2024-03-28T22:36:29.402 INFO:teuthology.orchestra.run.smithi043.stdout:last_failure_osd_epoch 0 2024-03-28T22:36:29.402 INFO:teuthology.orchestra.run.smithi043.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-28T22:36:29.402 INFO:teuthology.orchestra.run.smithi043.stdout:max_mds 5 2024-03-28T22:36:29.402 INFO:teuthology.orchestra.run.smithi043.stdout:in 0,1,2,3,4 2024-03-28T22:36:29.402 INFO:teuthology.orchestra.run.smithi043.stdout:up {0=24572,1=24556,2=24535,3=24523,4=24518} 2024-03-28T22:36:29.402 INFO:teuthology.orchestra.run.smithi043.stdout:failed 2024-03-28T22:36:29.402 INFO:teuthology.orchestra.run.smithi043.stdout:damaged 2024-03-28T22:36:29.402 INFO:teuthology.orchestra.run.smithi043.stdout:stopped 2024-03-28T22:36:29.402 INFO:teuthology.orchestra.run.smithi043.stdout:data_pools [3,4] 2024-03-28T22:36:29.402 INFO:teuthology.orchestra.run.smithi043.stdout:metadata_pool 2 2024-03-28T22:36:29.402 INFO:teuthology.orchestra.run.smithi043.stdout:inline_data disabled 2024-03-28T22:36:29.403 INFO:teuthology.orchestra.run.smithi043.stdout:balancer 2024-03-28T22:36:29.403 INFO:teuthology.orchestra.run.smithi043.stdout:bal_rank_mask -1 2024-03-28T22:36:29.403 INFO:teuthology.orchestra.run.smithi043.stdout:standby_count_wanted 0 2024-03-28T22:36:29.403 INFO:teuthology.orchestra.run.smithi043.stdout:qdb_cluster leader: 24572 members: 24535,24556,24518,24523,24572 2024-03-28T22:36:29.403 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.b{0:24572} state up:active seq 3 addr [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] compat {c=[1],r=[1],i=[1fff]}] 2024-03-28T22:36:29.403 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.e{0:24491} state up:standby-replay seq 1 addr [v2:172.21.15.82:6834/1084988072,v1:172.21.15.82:6835/1084988072] compat {c=[1],r=[1],i=[1fff]}] 2024-03-28T22:36:29.403 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.i{1:24556} state up:active seq 14 addr [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] compat {c=[1],r=[1],i=[1fff]}] 2024-03-28T22:36:29.403 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.d{1:24479} state up:standby-replay seq 1 addr [v2:172.21.15.43:6836/1330639541,v1:172.21.15.43:6837/1330639541] compat {c=[1],r=[1],i=[1fff]}] 2024-03-28T22:36:29.403 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.c{2:24535} state up:active seq 16 addr [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] compat {c=[1],r=[1],i=[1fff]}] 2024-03-28T22:36:29.403 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.j{2:24461} state up:standby-replay seq 1 addr [v2:172.21.15.43:6834/4195313768,v1:172.21.15.43:6835/4195313768] compat {c=[1],r=[1],i=[1fff]}] 2024-03-28T22:36:29.403 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.f{3:24523} state up:active seq 17 addr [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] compat {c=[1],r=[1],i=[1fff]}] 2024-03-28T22:36:29.403 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.l{3:24457} state up:standby-replay seq 1 addr [v2:172.21.15.172:6832/2579392697,v1:172.21.15.172:6833/2579392697] compat {c=[1],r=[1],i=[1fff]}] 2024-03-28T22:36:29.403 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.h{4:24518} state up:active seq 16 addr [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] compat {c=[1],r=[1],i=[1fff]}] 2024-03-28T22:36:29.403 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.k{4:24443} state up:standby-replay seq 1 addr [v2:172.21.15.82:6832/3363736995,v1:172.21.15.82:6833/3363736995] compat {c=[1],r=[1],i=[1fff]}] 2024-03-28T22:36:29.404 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:36:29.404 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:36:29.404 INFO:teuthology.orchestra.run.smithi043.stdout:Standby daemons: 2024-03-28T22:36:29.404 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:36:29.404 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.g{-1:14451} state up:standby seq 1 addr [v2:172.21.15.43:6838/2742974591,v1:172.21.15.43:6839/2742974591] compat {c=[1],r=[1],i=[1fff]}] 2024-03-28T22:36:29.404 INFO:teuthology.orchestra.run.smithi043.stdout:[mds.a{-1:14469} state up:standby seq 1 addr [v2:172.21.15.43:6840/2371736231,v1:172.21.15.43:6841/2371736231] compat {c=[1],r=[1],i=[1fff]}] 2024-03-28T22:36:29.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.404+0000 7f397b800700 1 -- 172.21.15.43:0/1066245001 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f395c0a1e10 msgr2=0x7f395c0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:29.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.404+0000 7f397b800700 1 --2- 172.21.15.43:0/1066245001 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f395c0a1e10 0x7f395c0a42d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3968009750 tx=0x7f3968009310 comp rx=0 tx=0).stop 2024-03-28T22:36:29.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.404+0000 7f397b800700 1 -- 172.21.15.43:0/1066245001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3974076270 msgr2=0x7f397419abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:29.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.404+0000 7f397b800700 1 --2- 172.21.15.43:0/1066245001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3974076270 0x7f397419abb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f396000d9b0 tx=0x7f396000dcc0 comp rx=0 tx=0).stop 2024-03-28T22:36:29.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.404+0000 7f397b800700 1 -- 172.21.15.43:0/1066245001 shutdown_connections 2024-03-28T22:36:29.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.404+0000 7f397b800700 1 --2- 172.21.15.43:0/1066245001 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f395c0a1e10 0x7f395c0a42d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:29.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.404+0000 7f397b800700 1 --2- 172.21.15.43:0/1066245001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f397410a150 0x7f39741a1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:29.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.404+0000 7f397b800700 1 --2- 172.21.15.43:0/1066245001 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3974074ca0 0x7f397419a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:29.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.404+0000 7f397b800700 1 --2- 172.21.15.43:0/1066245001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3974076270 0x7f397419abb0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:29.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.404+0000 7f397b800700 1 -- 172.21.15.43:0/1066245001 >> 172.21.15.43:0/1066245001 conn(0x7f39740fd7e0 msgr2=0x7f3974106050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:29.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.404+0000 7f397b800700 1 -- 172.21.15.43:0/1066245001 shutdown_connections 2024-03-28T22:36:29.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:29.404+0000 7f397b800700 1 -- 172.21.15.43:0/1066245001 wait complete. 2024-03-28T22:36:29.406 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 28 2024-03-28T22:36:30.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:29 smithi172 ceph-mon[36566]: pgmap v416: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-28T22:36:30.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:29 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1066245001' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T22:36:30.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:29 smithi082 ceph-mon[36471]: pgmap v416: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-28T22:36:30.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:29 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1066245001' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T22:36:30.219 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:29 smithi043 ceph-mon[31823]: pgmap v416: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-28T22:36:30.219 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:29 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1066245001' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T22:36:30.274 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-28T22:36:30.755 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:36:32.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:31 smithi172 ceph-mon[36566]: pgmap v417: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-28T22:36:32.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:31 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:32.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:31 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:32.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:31 smithi082 ceph-mon[36471]: pgmap v417: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-28T22:36:32.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:31 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:32.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:31 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:32.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:31 smithi043 ceph-mon[31823]: pgmap v417: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-28T22:36:32.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:31 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:32.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:31 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:33.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.181+0000 7f9a60400700 1 -- 172.21.15.43:0/2379667926 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a581056c0 msgr2=0x7f9a58105b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:33.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.181+0000 7f9a60400700 1 --2- 172.21.15.43:0/2379667926 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a581056c0 0x7f9a58105b20 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9a44009920 tx=0x7f9a44009c30 comp rx=0 tx=0).stop 2024-03-28T22:36:33.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.182+0000 7f9a60400700 1 -- 172.21.15.43:0/2379667926 shutdown_connections 2024-03-28T22:36:33.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.182+0000 7f9a60400700 1 --2- 172.21.15.43:0/2379667926 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a58106060 0x7f9a5810c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:33.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.182+0000 7f9a60400700 1 --2- 172.21.15.43:0/2379667926 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a581044c0 0x7f9a581048e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:33.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.182+0000 7f9a60400700 1 --2- 172.21.15.43:0/2379667926 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a581056c0 0x7f9a58105b20 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:33.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.182+0000 7f9a60400700 1 -- 172.21.15.43:0/2379667926 >> 172.21.15.43:0/2379667926 conn(0x7f9a580ffa60 msgr2=0x7f9a58101ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:33.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.182+0000 7f9a60400700 1 -- 172.21.15.43:0/2379667926 shutdown_connections 2024-03-28T22:36:33.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.182+0000 7f9a60400700 1 -- 172.21.15.43:0/2379667926 wait complete. 2024-03-28T22:36:33.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.183+0000 7f9a60400700 1 Processor -- start 2024-03-28T22:36:33.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.183+0000 7f9a60400700 1 -- start start 2024-03-28T22:36:33.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.183+0000 7f9a60400700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a581044c0 0x7f9a5819a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:33.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.183+0000 7f9a60400700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a581056c0 0x7f9a5819ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:33.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.183+0000 7f9a60400700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a58106060 0x7f9a581a1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:33.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.183+0000 7f9a60400700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9a58110100 con 0x7f9a581044c0 2024-03-28T22:36:33.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.183+0000 7f9a60400700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5810ff80 con 0x7f9a581056c0 2024-03-28T22:36:33.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.184+0000 7f9a60400700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9a58110280 con 0x7f9a58106060 2024-03-28T22:36:33.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.184+0000 7f9a5d000700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a581056c0 0x7f9a5819ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:33.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.184+0000 7f9a5e400700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a58106060 0x7f9a581a1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:33.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.184+0000 7f9a5d000700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a581056c0 0x7f9a5819ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33876/0 (socket says 172.21.15.43:33876) 2024-03-28T22:36:33.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.184+0000 7f9a5d000700 1 -- 172.21.15.43:0/1402691722 learned_addr learned my addr 172.21.15.43:0/1402691722 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:33.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.184+0000 7f9a5da00700 1 --2- 172.21.15.43:0/1402691722 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a581044c0 0x7f9a5819a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:33.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.184+0000 7f9a5d000700 1 -- 172.21.15.43:0/1402691722 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a58106060 msgr2=0x7f9a581a1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:33.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.184+0000 7f9a5d000700 1 --2- 172.21.15.43:0/1402691722 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a58106060 0x7f9a581a1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:33.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.184+0000 7f9a5d000700 1 -- 172.21.15.43:0/1402691722 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a581044c0 msgr2=0x7f9a5819a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:33.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.184+0000 7f9a5d000700 1 --2- 172.21.15.43:0/1402691722 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a581044c0 0x7f9a5819a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:33.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.184+0000 7f9a5d000700 1 -- 172.21.15.43:0/1402691722 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a581a23a0 con 0x7f9a581056c0 2024-03-28T22:36:33.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.185+0000 7f9a5d000700 1 --2- 172.21.15.43:0/1402691722 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a581056c0 0x7f9a5819ac80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9a4400fca0 tx=0x7f9a44005250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:33.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.185+0000 7f9a56000700 1 -- 172.21.15.43:0/1402691722 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4400f950 con 0x7f9a581056c0 2024-03-28T22:36:33.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.185+0000 7f9a56000700 1 -- 172.21.15.43:0/1402691722 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a4400ba40 con 0x7f9a581056c0 2024-03-28T22:36:33.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.185+0000 7f9a56000700 1 -- 172.21.15.43:0/1402691722 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a44003bc0 con 0x7f9a581056c0 2024-03-28T22:36:33.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.185+0000 7f9a60400700 1 -- 172.21.15.43:0/1402691722 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a581a2630 con 0x7f9a581056c0 2024-03-28T22:36:33.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.185+0000 7f9a60400700 1 -- 172.21.15.43:0/1402691722 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a581a2a30 con 0x7f9a581056c0 2024-03-28T22:36:33.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.186+0000 7f9a60400700 1 -- 172.21.15.43:0/1402691722 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a58068f00 con 0x7f9a581056c0 2024-03-28T22:36:33.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.188+0000 7f9a56000700 1 -- 172.21.15.43:0/1402691722 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f9a44005440 con 0x7f9a581056c0 2024-03-28T22:36:33.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.189+0000 7f9a56000700 1 --2- 172.21.15.43:0/1402691722 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9a400a1fb0 0x7f9a400a4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:33.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.189+0000 7f9a56000700 1 -- 172.21.15.43:0/1402691722 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f9a440cf1f0 con 0x7f9a581056c0 2024-03-28T22:36:33.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.189+0000 7f9a5da00700 1 --2- 172.21.15.43:0/1402691722 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9a400a1fb0 0x7f9a400a4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:33.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.190+0000 7f9a5da00700 1 --2- 172.21.15.43:0/1402691722 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9a400a1fb0 0x7f9a400a4470 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9a58105520 tx=0x7f9a4c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:33.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.193+0000 7f9a56000700 1 -- 172.21.15.43:0/1402691722 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9a44013030 con 0x7f9a581056c0 2024-03-28T22:36:33.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.428+0000 7f9a60400700 1 -- 172.21.15.43:0/1402691722 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f9a58062e50 con 0x7f9a581056c0 2024-03-28T22:36:33.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.429+0000 7f9a56000700 1 -- 172.21.15.43:0/1402691722 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v96) ==== 56+0+5641 (secure 0 0 0) 0x7f9a4409c2c0 con 0x7f9a581056c0 2024-03-28T22:36:33.428 INFO:teuthology.orchestra.run.smithi043.stdout:epoch 96 2024-03-28T22:36:33.428 INFO:teuthology.orchestra.run.smithi043.stdout:fsid c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:36:33.428 INFO:teuthology.orchestra.run.smithi043.stdout:created 2024-03-28T22:24:19.950084+0000 2024-03-28T22:36:33.428 INFO:teuthology.orchestra.run.smithi043.stdout:modified 2024-03-28T22:35:30.237654+0000 2024-03-28T22:36:33.428 INFO:teuthology.orchestra.run.smithi043.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-28T22:36:33.428 INFO:teuthology.orchestra.run.smithi043.stdout:crush_version 27 2024-03-28T22:36:33.428 INFO:teuthology.orchestra.run.smithi043.stdout:full_ratio 0.95 2024-03-28T22:36:33.428 INFO:teuthology.orchestra.run.smithi043.stdout:backfillfull_ratio 0.9 2024-03-28T22:36:33.428 INFO:teuthology.orchestra.run.smithi043.stdout:nearfull_ratio 0.85 2024-03-28T22:36:33.428 INFO:teuthology.orchestra.run.smithi043.stdout:require_min_compat_client luminous 2024-03-28T22:36:33.428 INFO:teuthology.orchestra.run.smithi043.stdout:min_compat_client luminous 2024-03-28T22:36:33.428 INFO:teuthology.orchestra.run.smithi043.stdout:require_osd_release squid 2024-03-28T22:36:33.428 INFO:teuthology.orchestra.run.smithi043.stdout:stretch_mode_enabled false 2024-03-28T22:36:33.429 INFO:teuthology.orchestra.run.smithi043.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-28T22:36:33.429 INFO:teuthology.orchestra.run.smithi043.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-28T22:36:33.429 INFO:teuthology.orchestra.run.smithi043.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 87 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-03-28T22:36:33.429 INFO:teuthology.orchestra.run.smithi043.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 92 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-03-28T22:36:33.429 INFO:teuthology.orchestra.run.smithi043.stdout:max_osd 12 2024-03-28T22:36:33.429 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0 up in weight 1 up_from 8 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.43:6802/2826124409,v1:172.21.15.43:6803/2826124409] [v2:172.21.15.43:6804/2826124409,v1:172.21.15.43:6805/2826124409] exists,up b855adcd-786c-4e42-b8fe-a515a77ce979 2024-03-28T22:36:33.429 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1 up in weight 1 up_from 13 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6800/3043263644,v1:172.21.15.82:6801/3043263644] [v2:172.21.15.82:6802/3043263644,v1:172.21.15.82:6803/3043263644] exists,up 031c143d-8ad1-4b68-8db9-3c717137e3e4 2024-03-28T22:36:33.429 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2 up in weight 1 up_from 18 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6800/4213953645,v1:172.21.15.172:6801/4213953645] [v2:172.21.15.172:6802/4213953645,v1:172.21.15.172:6803/4213953645] exists,up 20632bbb-f07a-4ca1-b0c4-ff9d508e211c 2024-03-28T22:36:33.429 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3 up in weight 1 up_from 25 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.43:6810/1597280920,v1:172.21.15.43:6811/1597280920] [v2:172.21.15.43:6812/1597280920,v1:172.21.15.43:6813/1597280920] exists,up 9a1f2312-4dbd-475b-8736-d5ebc8788e09 2024-03-28T22:36:33.429 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4 up in weight 1 up_from 33 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6808/215451895,v1:172.21.15.82:6809/215451895] [v2:172.21.15.82:6810/215451895,v1:172.21.15.82:6811/215451895] exists,up 29140ead-7075-42b2-90c1-27cae0b29ec1 2024-03-28T22:36:33.429 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5 up in weight 1 up_from 38 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6808/1576148933,v1:172.21.15.172:6809/1576148933] [v2:172.21.15.172:6810/1576148933,v1:172.21.15.172:6811/1576148933] exists,up 59644c61-93dc-4536-b2ad-0256b24e5cb6 2024-03-28T22:36:33.429 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6 up in weight 1 up_from 46 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.43:6818/791044279,v1:172.21.15.43:6819/791044279] [v2:172.21.15.43:6820/791044279,v1:172.21.15.43:6821/791044279] exists,up 22d54e57-c3f1-4ecf-9921-8e815b48f92a 2024-03-28T22:36:33.429 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7 up in weight 1 up_from 52 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6816/2157777323,v1:172.21.15.82:6817/2157777323] [v2:172.21.15.82:6818/2157777323,v1:172.21.15.82:6819/2157777323] exists,up 248cd1e2-0d6b-452f-a0a8-c51fe1b975ee 2024-03-28T22:36:33.430 INFO:teuthology.orchestra.run.smithi043.stdout:osd.8 up in weight 1 up_from 61 up_thru 95 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6816/1095328248,v1:172.21.15.172:6817/1095328248] [v2:172.21.15.172:6818/1095328248,v1:172.21.15.172:6819/1095328248] exists,up d98aab6e-8bae-4c24-906c-4f492f047b8e 2024-03-28T22:36:33.430 INFO:teuthology.orchestra.run.smithi043.stdout:osd.9 up in weight 1 up_from 69 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.43:6826/3657791498,v1:172.21.15.43:6827/3657791498] [v2:172.21.15.43:6828/3657791498,v1:172.21.15.43:6829/3657791498] exists,up 19aa383e-a0f6-45ea-96cc-e7956535caa6 2024-03-28T22:36:33.430 INFO:teuthology.orchestra.run.smithi043.stdout:osd.10 up in weight 1 up_from 74 up_thru 95 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6824/3454954047,v1:172.21.15.82:6825/3454954047] [v2:172.21.15.82:6826/3454954047,v1:172.21.15.82:6827/3454954047] exists,up 666977e5-eaf7-41b9-a18f-c380e149345d 2024-03-28T22:36:33.430 INFO:teuthology.orchestra.run.smithi043.stdout:osd.11 up in weight 1 up_from 82 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6824/2563517144,v1:172.21.15.172:6825/2563517144] [v2:172.21.15.172:6826/2563517144,v1:172.21.15.172:6827/2563517144] exists,up 615cf3be-8076-470e-ad09-c880e7593717 2024-03-28T22:36:33.430 INFO:teuthology.orchestra.run.smithi043.stdout:pg_upmap_items 2.4 [2,11] 2024-03-28T22:36:33.430 INFO:teuthology.orchestra.run.smithi043.stdout:pg_upmap_items 2.f [8,11] 2024-03-28T22:36:33.430 INFO:teuthology.orchestra.run.smithi043.stdout:pg_upmap_items 3.6 [4,6] 2024-03-28T22:36:33.430 INFO:teuthology.orchestra.run.smithi043.stdout:pg_upmap_items 3.e [4,8] 2024-03-28T22:36:33.430 INFO:teuthology.orchestra.run.smithi043.stdout:pg_upmap_items 3.18 [3,9] 2024-03-28T22:36:33.430 INFO:teuthology.orchestra.run.smithi043.stdout:pg_upmap_items 3.1a [5,7] 2024-03-28T22:36:33.430 INFO:teuthology.orchestra.run.smithi043.stdout:pg_upmap_items 3.20 [5,9] 2024-03-28T22:36:33.430 INFO:teuthology.orchestra.run.smithi043.stdout:pg_upmap_items 3.36 [4,9] 2024-03-28T22:36:33.430 INFO:teuthology.orchestra.run.smithi043.stdout:pg_upmap_items 4.2 [9,1] 2024-03-28T22:36:33.431 INFO:teuthology.orchestra.run.smithi043.stdout:pg_upmap_items 4.9 [5,10] 2024-03-28T22:36:33.431 INFO:teuthology.orchestra.run.smithi043.stdout:pg_upmap_items 4.18 [9,8] 2024-03-28T22:36:33.431 INFO:teuthology.orchestra.run.smithi043.stdout:pg_upmap_items 4.19 [9,8] 2024-03-28T22:36:33.431 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/578791143 expires 2024-03-29T22:25:28.008215+0000 2024-03-28T22:36:33.431 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/4029313215 expires 2024-03-29T22:25:28.008215+0000 2024-03-28T22:36:33.431 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/1574178482 expires 2024-03-29T22:25:28.008215+0000 2024-03-28T22:36:33.431 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:6801/3517712438 expires 2024-03-29T22:25:28.008215+0000 2024-03-28T22:36:33.431 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:6800/1532177494 expires 2024-03-29T22:24:52.981057+0000 2024-03-28T22:36:33.431 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/1375099638 expires 2024-03-29T22:24:52.981057+0000 2024-03-28T22:36:33.431 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/2710117652 expires 2024-03-29T22:24:52.981057+0000 2024-03-28T22:36:33.431 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:6801/1532177494 expires 2024-03-29T22:24:52.981057+0000 2024-03-28T22:36:33.431 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/3215634168 expires 2024-03-29T22:24:52.981057+0000 2024-03-28T22:36:33.431 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:6800/3517712438 expires 2024-03-29T22:25:28.008215+0000 2024-03-28T22:36:33.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.432+0000 7f9a60400700 1 -- 172.21.15.43:0/1402691722 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9a400a1fb0 msgr2=0x7f9a400a4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:33.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.432+0000 7f9a60400700 1 --2- 172.21.15.43:0/1402691722 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9a400a1fb0 0x7f9a400a4470 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9a58105520 tx=0x7f9a4c009310 comp rx=0 tx=0).stop 2024-03-28T22:36:33.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.432+0000 7f9a60400700 1 -- 172.21.15.43:0/1402691722 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a581056c0 msgr2=0x7f9a5819ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:33.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.432+0000 7f9a60400700 1 --2- 172.21.15.43:0/1402691722 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a581056c0 0x7f9a5819ac80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9a4400fca0 tx=0x7f9a44005250 comp rx=0 tx=0).stop 2024-03-28T22:36:33.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.432+0000 7f9a60400700 1 -- 172.21.15.43:0/1402691722 shutdown_connections 2024-03-28T22:36:33.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.432+0000 7f9a60400700 1 --2- 172.21.15.43:0/1402691722 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9a400a1fb0 0x7f9a400a4470 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:33.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.432+0000 7f9a60400700 1 --2- 172.21.15.43:0/1402691722 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a58106060 0x7f9a581a1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:33.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.432+0000 7f9a60400700 1 --2- 172.21.15.43:0/1402691722 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a581044c0 0x7f9a5819a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:33.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.432+0000 7f9a60400700 1 --2- 172.21.15.43:0/1402691722 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a581056c0 0x7f9a5819ac80 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:33.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.432+0000 7f9a60400700 1 -- 172.21.15.43:0/1402691722 >> 172.21.15.43:0/1402691722 conn(0x7f9a580ffa60 msgr2=0x7f9a581089c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:33.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.432+0000 7f9a60400700 1 -- 172.21.15.43:0/1402691722 shutdown_connections 2024-03-28T22:36:33.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:33.432+0000 7f9a60400700 1 -- 172.21.15.43:0/1402691722 wait complete. 2024-03-28T22:36:34.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:33 smithi172 ceph-mon[36566]: pgmap v418: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-28T22:36:34.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:33 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:36:34.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:33 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:36:34.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:33 smithi172 ceph-mon[36566]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:34.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:33 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1402691722' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T22:36:34.156 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:33 smithi043 ceph-mon[31823]: pgmap v418: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-28T22:36:34.156 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:33 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:36:34.156 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:33 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:36:34.156 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:33 smithi043 ceph-mon[31823]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:34.156 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:33 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1402691722' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T22:36:34.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:33 smithi082 ceph-mon[36471]: pgmap v418: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-28T22:36:34.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:33 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:36:34.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:33 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:36:34.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:33 smithi082 ceph-mon[36471]: from='mgr.14152 172.21.15.43:0/96476537' entity='mgr.x' 2024-03-28T22:36:34.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:33 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1402691722' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T22:36:34.225 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-28T22:36:34.237 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:36:34.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.475+0000 7f9a97200640 1 -- 172.21.15.43:0/1751507990 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a90107bb0 msgr2=0x7f9a90107fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:34.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.475+0000 7f9a97200640 1 --2- 172.21.15.43:0/1751507990 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a90107bb0 0x7f9a90107fd0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9a84009900 tx=0x7f9a8402bc30 comp rx=0 tx=0).stop 2024-03-28T22:36:34.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.475+0000 7f9a97200640 1 -- 172.21.15.43:0/1751507990 shutdown_connections 2024-03-28T22:36:34.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.475+0000 7f9a97200640 1 --2- 172.21.15.43:0/1751507990 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a90109760 0x7f9a9010fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:34.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.475+0000 7f9a97200640 1 --2- 172.21.15.43:0/1751507990 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a90108da0 0x7f9a90109220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:34.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.475+0000 7f9a97200640 1 --2- 172.21.15.43:0/1751507990 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a90107bb0 0x7f9a90107fd0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:34.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.475+0000 7f9a97200640 1 -- 172.21.15.43:0/1751507990 >> 172.21.15.43:0/1751507990 conn(0x7f9a90076b60 msgr2=0x7f9a90078f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:34.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.475+0000 7f9a97200640 1 -- 172.21.15.43:0/1751507990 shutdown_connections 2024-03-28T22:36:34.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.475+0000 7f9a97200640 1 -- 172.21.15.43:0/1751507990 wait complete. 2024-03-28T22:36:34.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.476+0000 7f9a97200640 1 Processor -- start 2024-03-28T22:36:34.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.476+0000 7f9a97200640 1 -- start start 2024-03-28T22:36:34.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.477+0000 7f9a97200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a90107bb0 0x7f9a9019dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:34.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.477+0000 7f9a97200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a90108da0 0x7f9a9019e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:34.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.477+0000 7f9a97200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a90109760 0x7f9a901a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:34.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.477+0000 7f9a97200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9a90113850 con 0x7f9a90108da0 2024-03-28T22:36:34.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.477+0000 7f9a97200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9a901136d0 con 0x7f9a90109760 2024-03-28T22:36:34.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.477+0000 7f9a97200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9a901139d0 con 0x7f9a90107bb0 2024-03-28T22:36:34.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.477+0000 7f9a96800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a90109760 0x7f9a901a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:34.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.477+0000 7f9a95e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a90107bb0 0x7f9a9019dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:34.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.477+0000 7f9a96800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a90109760 0x7f9a901a5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33902/0 (socket says 172.21.15.43:33902) 2024-03-28T22:36:34.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.477+0000 7f9a96800640 1 -- 172.21.15.43:0/313817708 learned_addr learned my addr 172.21.15.43:0/313817708 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:34.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.477+0000 7f9a95e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a90107bb0 0x7f9a9019dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:34858/0 (socket says 172.21.15.43:34858) 2024-03-28T22:36:34.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.477+0000 7f9a95400640 1 --2- 172.21.15.43:0/313817708 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a90108da0 0x7f9a9019e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:34.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.478+0000 7f9a95e00640 1 -- 172.21.15.43:0/313817708 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a90109760 msgr2=0x7f9a901a5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:34.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.478+0000 7f9a95e00640 1 --2- 172.21.15.43:0/313817708 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a90109760 0x7f9a901a5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:34.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.478+0000 7f9a95e00640 1 -- 172.21.15.43:0/313817708 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a90108da0 msgr2=0x7f9a9019e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:34.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.478+0000 7f9a95e00640 1 --2- 172.21.15.43:0/313817708 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a90108da0 0x7f9a9019e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:34.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.478+0000 7f9a95e00640 1 -- 172.21.15.43:0/313817708 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a901a5ae0 con 0x7f9a90107bb0 2024-03-28T22:36:34.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.478+0000 7f9a95400640 1 --2- 172.21.15.43:0/313817708 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a90108da0 0x7f9a9019e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:36:34.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.478+0000 7f9a96800640 1 --2- 172.21.15.43:0/313817708 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a90109760 0x7f9a901a5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:34.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.478+0000 7f9a95e00640 1 --2- 172.21.15.43:0/313817708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a90107bb0 0x7f9a9019dec0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9a84009230 tx=0x7f9a8402c5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:34.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.478+0000 7f9a8aa00640 1 -- 172.21.15.43:0/313817708 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a84043020 con 0x7f9a90107bb0 2024-03-28T22:36:34.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.478+0000 7f9a97200640 1 -- 172.21.15.43:0/313817708 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a901a5d70 con 0x7f9a90107bb0 2024-03-28T22:36:34.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.478+0000 7f9a8aa00640 1 -- 172.21.15.43:0/313817708 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a840054e0 con 0x7f9a90107bb0 2024-03-28T22:36:34.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.478+0000 7f9a8aa00640 1 -- 172.21.15.43:0/313817708 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a84003b60 con 0x7f9a90107bb0 2024-03-28T22:36:34.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.478+0000 7f9a97200640 1 -- 172.21.15.43:0/313817708 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a901a6220 con 0x7f9a90107bb0 2024-03-28T22:36:34.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.480+0000 7f9a97200640 1 -- 172.21.15.43:0/313817708 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a9010eed0 con 0x7f9a90107bb0 2024-03-28T22:36:34.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.481+0000 7f9a8aa00640 1 -- 172.21.15.43:0/313817708 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f9a84046020 con 0x7f9a90107bb0 2024-03-28T22:36:34.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.483+0000 7f9a8aa00640 1 --2- 172.21.15.43:0/313817708 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9a780a1e00 0x7f9a780a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:34.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.483+0000 7f9a8aa00640 1 -- 172.21.15.43:0/313817708 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f9a840f0200 con 0x7f9a90107bb0 2024-03-28T22:36:34.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.483+0000 7f9a95400640 1 --2- 172.21.15.43:0/313817708 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9a780a1e00 0x7f9a780a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:34.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.484+0000 7f9a95400640 1 --2- 172.21.15.43:0/313817708 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9a780a1e00 0x7f9a780a42c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9a9019f3d0 tx=0x7f9a80005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:34.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.491+0000 7f9a8aa00640 1 -- 172.21.15.43:0/313817708 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9a8404d020 con 0x7f9a90107bb0 2024-03-28T22:36:34.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.698+0000 7f9a97200640 1 -- 172.21.15.43:0/313817708 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9a90062350 con 0x7f9a90107bb0 2024-03-28T22:36:34.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.700+0000 7f9a8aa00640 1 -- 172.21.15.43:0/313817708 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+24712 (secure 0 0 0) 0x7f9a840bd240 con 0x7f9a90107bb0 2024-03-28T22:36:34.699 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:36:34.699 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":96,"fsid":"c2276428-ed51-11ee-95d0-87774f69a715","created":"2024-03-28T22:24:19.950084+0000","modified":"2024-03-28T22:35:30.237654+0000","last_up_change":"2024-03-28T22:32:31.078737+0000","last_in_change":"2024-03-28T22:32:10.147097+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:28:14.713290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:33:27.167435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:33:28.411825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T22:33:32.523122+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b855adcd-786c-4e42-b8fe-a515a77ce979","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6803","nonce":2826124409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6805","nonce":2826124409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6809","nonce":2826124409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6807","nonce":2826124409}]},"public_addr":"172.21.15.43:6803/2826124409","cluster_addr":"172.21.15.43:6805/2826124409","heartbeat_back_addr":"172.21.15.43:6809/2826124409","heartbeat_front_addr":"172.21.15.43:6807/2826124409","state":["exists","up"]},{"osd":1,"uuid":"031c143d-8ad1-4b68-8db9-3c717137e3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6801","nonce":3043263644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6803","nonce":3043263644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6807","nonce":3043263644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6805","nonce":3043263644}]},"public_addr":"172.21.15.82:6801/3043263644","cluster_addr":"172.21.15.82:6803/3043263644","heartbeat_back_addr":"172.21.15.82:6807/3043263644","heartbeat_front_addr":"172.21.15.82:6805/3043263644","state":["exists","up"]},{"osd":2,"uuid":"20632bbb-f07a-4ca1-b0c4-ff9d508e211c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6801","nonce":4213953645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6803","nonce":4213953645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6807","nonce":4213953645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6805","nonce":4213953645}]},"public_addr":"172.21.15.172:6801/4213953645","cluster_addr":"172.21.15.172:6803/4213953645","heartbeat_back_addr":"172.21.15.172:6807/4213953645","heartbeat_front_addr":"172.21.15.172:6805/4213953645","state":["exists","up"]},{"osd":3,"uuid":"9a1f2312-4dbd-475b-8736-d5ebc8788e09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6811","nonce":1597280920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6813","nonce":1597280920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6817","nonce":1597280920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6815","nonce":1597280920}]},"public_addr":"172.21.15.43:6811/1597280920","cluster_addr":"172.21.15.43:6813/1597280920","heartbeat_back_addr":"172.21.15.43:6817/1597280920","heartbeat_front_addr":"172.21.15.43:6815/1597280920","state":["exists","up"]},{"osd":4,"uuid":"29140ead-7075-42b2-90c1-27cae0b29ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6809","nonce":215451895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6811","nonce":215451895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6815","nonce":215451895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6813","nonce":215451895}]},"public_addr":"172.21.15.82:6809/215451895","cluster_addr":"172.21.15.82:6811/215451895","heartbeat_back_addr":"172.21.15.82:6815/215451895","heartbeat_front_addr":"172.21.15.82:6813/215451895","state":["exists","up"]},{"osd":5,"uuid":"59644c61-93dc-4536-b2ad-0256b24e5cb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6809","nonce":1576148933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6811","nonce":1576148933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6815","nonce":1576148933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6813","nonce":1576148933}]},"public_addr":"172.21.15.172:6809/1576148933","cluster_addr":"172.21.15.172:6811/1576148933","heartbeat_back_addr":"172.21.15.172:6815/1576148933","heartbeat_front_addr":"172.21.15.172:6813/1576148933","state":["exists","up"]},{"osd":6,"uuid":"22d54e57-c3f1-4ecf-9921-8e815b48f92a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6819","nonce":791044279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6821","nonce":791044279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6825","nonce":791044279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6823","nonce":791044279}]},"public_addr":"172.21.15.43:6819/791044279","cluster_addr":"172.21.15.43:6821/791044279","heartbeat_back_addr":"172.21.15.43:6825/791044279","heartbeat_front_addr":"172.21.15.43:6823/791044279","state":["exists","up"]},{"osd":7,"uuid":"248cd1e2-0d6b-452f-a0a8-c51fe1b975ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6817","nonce":2157777323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6819","nonce":2157777323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6823","nonce":2157777323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6821","nonce":2157777323}]},"public_addr":"172.21.15.82:6817/2157777323","cluster_addr":"172.21.15.82:6819/2157777323","heartbeat_back_addr":"172.21.15.82:6823/2157777323","heartbeat_front_addr":"172.21.15.82:6821/2157777323","state":["exists","up"]},{"osd":8,"uuid":"d98aab6e-8bae-4c24-906c-4f492f047b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6817","nonce":1095328248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6819","nonce":1095328248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6823","nonce":1095328248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6821","nonce":1095328248}]},"public_addr":"172.21.15.172:6817/1095328248","cluster_addr":"172.21.15.172:6819/1095328248","heartbeat_back_addr":"172.21.15.172:6823/1095328248","heartbeat_front_addr":"172.21.15.172:6821/1095328248","state":["exists","up"]},{"osd":9,"uuid":"19aa383e-a0f6-45ea-96cc-e7956535caa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6827","nonce":3657791498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6829","nonce":3657791498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6833","nonce":3657791498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6831","nonce":3657791498}]},"public_addr":"172.21.15.43:6827/3657791498","cluster_addr":"172.21.15.43:6829/3657791498","heartbeat_back_addr":"172.21.15.43:6833/3657791498","heartbeat_front_addr":"172.21.15.43:6831/3657791498","state":["exists","up"]},{"osd":10,"uuid":"666977e5-eaf7-41b9-a18f-c380e149345d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6825","nonce":3454954047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6827","nonce":3454954047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6831","nonce":3454954047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6829","nonce":3454954047}]},"public_addr":"172.21.15.82:6825/3454954047","cluster_addr":"172.21.15.82:6827/3454954047","heartbeat_back_addr":"172.21.15.82:6831/3454954047","heartbeat_front_addr":"172.21.15.82:6829/3454954047","state":["exists","up"]},{"osd":11,"uuid":"615cf3be-8076-470e-ad09-c880e7593717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6825","nonce":2563517144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6827","nonce":2563517144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6831","nonce":2563517144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6829","nonce":2563517144}]},"public_addr":"172.21.15.172:6825/2563517144","cluster_addr":"172.21.15.172:6827/2563517144","heartbeat_back_addr":"172.21.15.172:6831/2563517144","heartbeat_front_addr":"172.21.15.172:6829/2563517144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:21.727405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:47.100259+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:10.615438+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:43.632144+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:09.487940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:34.604463+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:09.594452+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:35.709795+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:01.184202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:35.455322+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:02.413166+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:28.550754+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.9","mappings":[{"from":5,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/578791143":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/4029313215":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/1574178482":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6801/3517712438":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6800/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/1375099638":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/2710117652":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6801/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/3215634168":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6800/3517712438":"2024-03-29T22:25:28.008215+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:36:34.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.703+0000 7f9a97200640 1 -- 172.21.15.43:0/313817708 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9a780a1e00 msgr2=0x7f9a780a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:34.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.703+0000 7f9a97200640 1 --2- 172.21.15.43:0/313817708 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9a780a1e00 0x7f9a780a42c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9a9019f3d0 tx=0x7f9a80005f90 comp rx=0 tx=0).stop 2024-03-28T22:36:34.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.703+0000 7f9a97200640 1 -- 172.21.15.43:0/313817708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a90107bb0 msgr2=0x7f9a9019dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:34.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.703+0000 7f9a97200640 1 --2- 172.21.15.43:0/313817708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a90107bb0 0x7f9a9019dec0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9a84009230 tx=0x7f9a8402c5d0 comp rx=0 tx=0).stop 2024-03-28T22:36:34.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.703+0000 7f9a97200640 1 -- 172.21.15.43:0/313817708 shutdown_connections 2024-03-28T22:36:34.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.703+0000 7f9a97200640 1 --2- 172.21.15.43:0/313817708 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9a780a1e00 0x7f9a780a42c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:34.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.703+0000 7f9a97200640 1 --2- 172.21.15.43:0/313817708 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a90109760 0x7f9a901a5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:34.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.703+0000 7f9a97200640 1 --2- 172.21.15.43:0/313817708 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a90108da0 0x7f9a9019e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:34.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.703+0000 7f9a97200640 1 --2- 172.21.15.43:0/313817708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a90107bb0 0x7f9a9019dec0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:34.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.703+0000 7f9a97200640 1 -- 172.21.15.43:0/313817708 >> 172.21.15.43:0/313817708 conn(0x7f9a90076b60 msgr2=0x7f9a9010c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:34.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.703+0000 7f9a97200640 1 -- 172.21.15.43:0/313817708 shutdown_connections 2024-03-28T22:36:34.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.703+0000 7f9a97200640 1 -- 172.21.15.43:0/313817708 wait complete. 2024-03-28T22:36:34.722 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T22:28:14.713290+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T22:33:27.167435+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T22:33:28.411825+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T22:33:32.523122+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T22:36:34.723 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T22:36:34.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.944+0000 7fe62b200640 1 -- 172.21.15.43:0/2834615903 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe62c076110 msgr2=0x7fe62c076530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:34.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.944+0000 7fe62b200640 1 --2- 172.21.15.43:0/2834615903 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe62c076110 0x7fe62c076530 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe61c009900 tx=0x7fe61c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:36:34.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.944+0000 7fe62b200640 1 -- 172.21.15.43:0/2834615903 shutdown_connections 2024-03-28T22:36:34.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.944+0000 7fe62b200640 1 --2- 172.21.15.43:0/2834615903 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe62c077350 0x7fe62c0777b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:34.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.944+0000 7fe62b200640 1 --2- 172.21.15.43:0/2834615903 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe62c077cf0 0x7fe62c078590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:34.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.944+0000 7fe62b200640 1 --2- 172.21.15.43:0/2834615903 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe62c076110 0x7fe62c076530 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:34.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.944+0000 7fe62b200640 1 -- 172.21.15.43:0/2834615903 >> 172.21.15.43:0/2834615903 conn(0x7fe62c0fedc0 msgr2=0x7fe62c101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:34.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.944+0000 7fe62b200640 1 -- 172.21.15.43:0/2834615903 shutdown_connections 2024-03-28T22:36:34.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.944+0000 7fe62b200640 1 -- 172.21.15.43:0/2834615903 wait complete. 2024-03-28T22:36:34.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.945+0000 7fe62b200640 1 Processor -- start 2024-03-28T22:36:34.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.945+0000 7fe62b200640 1 -- start start 2024-03-28T22:36:34.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.946+0000 7fe62b200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe62c076110 0x7fe62c19df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:34.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.946+0000 7fe62b200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe62c077350 0x7fe62c19e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:34.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.946+0000 7fe62b200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe62c077cf0 0x7fe62c1a54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:34.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.946+0000 7fe62b200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe62c113920 con 0x7fe62c077cf0 2024-03-28T22:36:34.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.946+0000 7fe62b200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe62c1137a0 con 0x7fe62c076110 2024-03-28T22:36:34.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.946+0000 7fe62b200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe62c113aa0 con 0x7fe62c077350 2024-03-28T22:36:34.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.946+0000 7fe629e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe62c076110 0x7fe62c19df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:34.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.946+0000 7fe629e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe62c076110 0x7fe62c19df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47032/0 (socket says 172.21.15.43:47032) 2024-03-28T22:36:34.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.946+0000 7fe629e00640 1 -- 172.21.15.43:0/2377276303 learned_addr learned my addr 172.21.15.43:0/2377276303 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:34.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.946+0000 7fe621400640 1 --2- 172.21.15.43:0/2377276303 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe62c077350 0x7fe62c19e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:34.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.946+0000 7fe62a800640 1 --2- 172.21.15.43:0/2377276303 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe62c077cf0 0x7fe62c1a54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:34.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.946+0000 7fe629e00640 1 -- 172.21.15.43:0/2377276303 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe62c077350 msgr2=0x7fe62c19e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:34.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.946+0000 7fe629e00640 1 --2- 172.21.15.43:0/2377276303 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe62c077350 0x7fe62c19e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:34.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.947+0000 7fe629e00640 1 -- 172.21.15.43:0/2377276303 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe62c077cf0 msgr2=0x7fe62c1a54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:34.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.947+0000 7fe629e00640 1 --2- 172.21.15.43:0/2377276303 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe62c077cf0 0x7fe62c1a54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:34.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.947+0000 7fe629e00640 1 -- 172.21.15.43:0/2377276303 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe62c1a5b40 con 0x7fe62c076110 2024-03-28T22:36:34.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.947+0000 7fe62a800640 1 --2- 172.21.15.43:0/2377276303 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe62c077cf0 0x7fe62c1a54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:36:34.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.947+0000 7fe629e00640 1 --2- 172.21.15.43:0/2377276303 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe62c076110 0x7fe62c19df20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe61c000c00 tx=0x7fe61c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:34.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.947+0000 7fe623400640 1 -- 172.21.15.43:0/2377276303 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe61c043020 con 0x7fe62c076110 2024-03-28T22:36:34.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.947+0000 7fe623400640 1 -- 172.21.15.43:0/2377276303 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe61c0054e0 con 0x7fe62c076110 2024-03-28T22:36:34.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.947+0000 7fe62b200640 1 -- 172.21.15.43:0/2377276303 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe62c1a5dd0 con 0x7fe62c076110 2024-03-28T22:36:34.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.947+0000 7fe623400640 1 -- 172.21.15.43:0/2377276303 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe61c003c40 con 0x7fe62c076110 2024-03-28T22:36:34.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.947+0000 7fe62b200640 1 -- 172.21.15.43:0/2377276303 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe62c1a6230 con 0x7fe62c076110 2024-03-28T22:36:34.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.949+0000 7fe62b200640 1 -- 172.21.15.43:0/2377276303 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe62c1101c0 con 0x7fe62c076110 2024-03-28T22:36:34.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.950+0000 7fe623400640 1 -- 172.21.15.43:0/2377276303 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fe61c046020 con 0x7fe62c076110 2024-03-28T22:36:34.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.951+0000 7fe623400640 1 --2- 172.21.15.43:0/2377276303 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe6100a1d80 0x7fe6100a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:34.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.951+0000 7fe623400640 1 -- 172.21.15.43:0/2377276303 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7fe61c0f0280 con 0x7fe62c076110 2024-03-28T22:36:34.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.952+0000 7fe621400640 1 --2- 172.21.15.43:0/2377276303 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe6100a1d80 0x7fe6100a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:34.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.952+0000 7fe621400640 1 --2- 172.21.15.43:0/2377276303 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe6100a1d80 0x7fe6100a4240 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe62c19f430 tx=0x7fe614009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:34.960+0000 7fe623400640 1 -- 172.21.15.43:0/2377276303 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe61c04d020 con 0x7fe62c076110 2024-03-28T22:36:35.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:34 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/313817708' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:36:35.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:34 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/313817708' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:36:35.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.163+0000 7fe62b200640 1 -- 172.21.15.43:0/2377276303 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe62c062350 con 0x7fe62c076110 2024-03-28T22:36:35.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.164+0000 7fe623400640 1 -- 172.21.15.43:0/2377276303 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) ==== 93+0+10 (secure 0 0 0) 0x7fe61c0bd2c0 con 0x7fe62c076110 2024-03-28T22:36:35.162 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-03-28T22:36:35.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.167+0000 7fe62b200640 1 -- 172.21.15.43:0/2377276303 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe6100a1d80 msgr2=0x7fe6100a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:35.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.167+0000 7fe62b200640 1 --2- 172.21.15.43:0/2377276303 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe6100a1d80 0x7fe6100a4240 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe62c19f430 tx=0x7fe614009500 comp rx=0 tx=0).stop 2024-03-28T22:36:35.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.167+0000 7fe62b200640 1 -- 172.21.15.43:0/2377276303 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe62c076110 msgr2=0x7fe62c19df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:35.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.167+0000 7fe62b200640 1 --2- 172.21.15.43:0/2377276303 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe62c076110 0x7fe62c19df20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe61c000c00 tx=0x7fe61c02c650 comp rx=0 tx=0).stop 2024-03-28T22:36:35.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.167+0000 7fe62b200640 1 -- 172.21.15.43:0/2377276303 shutdown_connections 2024-03-28T22:36:35.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.167+0000 7fe62b200640 1 --2- 172.21.15.43:0/2377276303 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe62c077cf0 0x7fe62c1a54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:35.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.167+0000 7fe62b200640 1 --2- 172.21.15.43:0/2377276303 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe62c077350 0x7fe62c19e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:35.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.167+0000 7fe62b200640 1 --2- 172.21.15.43:0/2377276303 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe6100a1d80 0x7fe6100a4240 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:35.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.167+0000 7fe62b200640 1 --2- 172.21.15.43:0/2377276303 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe62c076110 0x7fe62c19df20 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:35.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.167+0000 7fe62b200640 1 -- 172.21.15.43:0/2377276303 >> 172.21.15.43:0/2377276303 conn(0x7fe62c0fedc0 msgr2=0x7fe62c102120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:35.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.167+0000 7fe62b200640 1 -- 172.21.15.43:0/2377276303 shutdown_connections 2024-03-28T22:36:35.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.167+0000 7fe62b200640 1 -- 172.21.15.43:0/2377276303 wait complete. 2024-03-28T22:36:35.186 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T22:36:35.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:34 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/313817708' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:36:35.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.418+0000 7f9ba6a00640 1 -- 172.21.15.43:0/3818425158 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ba0076110 msgr2=0x7f9ba0076530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:35.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.418+0000 7f9ba6a00640 1 --2- 172.21.15.43:0/3818425158 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ba0076110 0x7f9ba0076530 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9b90009900 tx=0x7f9b9002bc30 comp rx=0 tx=0).stop 2024-03-28T22:36:35.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.418+0000 7f9ba6a00640 1 -- 172.21.15.43:0/3818425158 shutdown_connections 2024-03-28T22:36:35.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.418+0000 7f9ba6a00640 1 --2- 172.21.15.43:0/3818425158 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ba0077cf0 0x7f9ba0078590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:35.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.418+0000 7f9ba6a00640 1 --2- 172.21.15.43:0/3818425158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ba0077350 0x7f9ba00777b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:35.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.418+0000 7f9ba6a00640 1 --2- 172.21.15.43:0/3818425158 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ba0076110 0x7f9ba0076530 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:35.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.418+0000 7f9ba6a00640 1 -- 172.21.15.43:0/3818425158 >> 172.21.15.43:0/3818425158 conn(0x7f9ba00fef00 msgr2=0x7f9ba0101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:35.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.418+0000 7f9ba6a00640 1 -- 172.21.15.43:0/3818425158 shutdown_connections 2024-03-28T22:36:35.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.418+0000 7f9ba6a00640 1 -- 172.21.15.43:0/3818425158 wait complete. 2024-03-28T22:36:35.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.419+0000 7f9ba6a00640 1 Processor -- start 2024-03-28T22:36:35.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.419+0000 7f9ba6a00640 1 -- start start 2024-03-28T22:36:35.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.420+0000 7f9ba6a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ba0076110 0x7f9ba019e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:35.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.420+0000 7f9ba6a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ba0077350 0x7f9ba019e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:35.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.420+0000 7f9ba6a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ba0077cf0 0x7f9ba01a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:35.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.420+0000 7f9ba6a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba0113a60 con 0x7f9ba0077350 2024-03-28T22:36:35.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.420+0000 7f9ba6a00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba01138e0 con 0x7f9ba0077cf0 2024-03-28T22:36:35.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.420+0000 7f9ba6a00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba0113be0 con 0x7f9ba0076110 2024-03-28T22:36:35.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.420+0000 7f9b9fe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ba0076110 0x7f9ba019e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:35.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.420+0000 7f9b9f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ba0077350 0x7f9ba019e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:35.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.420+0000 7f9ba4a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ba0077cf0 0x7f9ba01a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:35.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.420+0000 7f9b9fe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ba0076110 0x7f9ba019e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:53508/0 (socket says 172.21.15.43:53508) 2024-03-28T22:36:35.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.420+0000 7f9ba4a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ba0077cf0 0x7f9ba01a5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47062/0 (socket says 172.21.15.43:47062) 2024-03-28T22:36:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.420+0000 7f9b9fe00640 1 -- 172.21.15.43:0/1500072100 learned_addr learned my addr 172.21.15.43:0/1500072100 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.420+0000 7f9b9fe00640 1 -- 172.21.15.43:0/1500072100 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ba0077cf0 msgr2=0x7f9ba01a5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.420+0000 7f9b9fe00640 1 --2- 172.21.15.43:0/1500072100 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ba0077cf0 0x7f9ba01a5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.420+0000 7f9b9fe00640 1 -- 172.21.15.43:0/1500072100 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ba0077350 msgr2=0x7f9ba019e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.420+0000 7f9b9fe00640 1 --2- 172.21.15.43:0/1500072100 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ba0077350 0x7f9ba019e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.420+0000 7f9b9fe00640 1 -- 172.21.15.43:0/1500072100 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ba01a5ce0 con 0x7f9ba0076110 2024-03-28T22:36:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.421+0000 7f9b9fe00640 1 --2- 172.21.15.43:0/1500072100 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ba0076110 0x7f9ba019e0c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9b90005ab0 tx=0x7f9b9002c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.421+0000 7f9b9cc00640 1 -- 172.21.15.43:0/1500072100 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b90043020 con 0x7f9ba0076110 2024-03-28T22:36:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.421+0000 7f9b9cc00640 1 -- 172.21.15.43:0/1500072100 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b900054e0 con 0x7f9ba0076110 2024-03-28T22:36:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.421+0000 7f9b9cc00640 1 -- 172.21.15.43:0/1500072100 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b90003ce0 con 0x7f9ba0076110 2024-03-28T22:36:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.421+0000 7f9ba6a00640 1 -- 172.21.15.43:0/1500072100 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ba01a5f70 con 0x7f9ba0076110 2024-03-28T22:36:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.421+0000 7f9ba6a00640 1 -- 172.21.15.43:0/1500072100 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ba01a6430 con 0x7f9ba0076110 2024-03-28T22:36:35.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.422+0000 7f9ba6a00640 1 -- 172.21.15.43:0/1500072100 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ba0110300 con 0x7f9ba0076110 2024-03-28T22:36:35.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.424+0000 7f9b9cc00640 1 -- 172.21.15.43:0/1500072100 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f9b90046020 con 0x7f9ba0076110 2024-03-28T22:36:35.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.425+0000 7f9b9cc00640 1 --2- 172.21.15.43:0/1500072100 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9b840a1e00 0x7f9b840a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:35.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.426+0000 7f9b9cc00640 1 -- 172.21.15.43:0/1500072100 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f9b900f0200 con 0x7f9ba0076110 2024-03-28T22:36:35.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.426+0000 7f9b9f400640 1 --2- 172.21.15.43:0/1500072100 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9b840a1e00 0x7f9b840a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:35.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.427+0000 7f9b9f400640 1 --2- 172.21.15.43:0/1500072100 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9b840a1e00 0x7f9b840a42c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9ba019f5d0 tx=0x7f9b88005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:35.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.429+0000 7f9b9cc00640 1 -- 172.21.15.43:0/1500072100 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9b9004d020 con 0x7f9ba0076110 2024-03-28T22:36:35.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.630+0000 7f9ba6a00640 1 -- 172.21.15.43:0/1500072100 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9ba00623f0 con 0x7f9ba0076110 2024-03-28T22:36:35.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.631+0000 7f9b9cc00640 1 -- 172.21.15.43:0/1500072100 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) ==== 104+0+11 (secure 0 0 0) 0x7f9b900bd240 con 0x7f9ba0076110 2024-03-28T22:36:35.629 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-03-28T22:36:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.634+0000 7f9ba6a00640 1 -- 172.21.15.43:0/1500072100 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9b840a1e00 msgr2=0x7f9b840a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.634+0000 7f9ba6a00640 1 --2- 172.21.15.43:0/1500072100 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9b840a1e00 0x7f9b840a42c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9ba019f5d0 tx=0x7f9b88005f90 comp rx=0 tx=0).stop 2024-03-28T22:36:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.634+0000 7f9ba6a00640 1 -- 172.21.15.43:0/1500072100 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ba0076110 msgr2=0x7f9ba019e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.634+0000 7f9ba6a00640 1 --2- 172.21.15.43:0/1500072100 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ba0076110 0x7f9ba019e0c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9b90005ab0 tx=0x7f9b9002c6c0 comp rx=0 tx=0).stop 2024-03-28T22:36:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.634+0000 7f9ba6a00640 1 -- 172.21.15.43:0/1500072100 shutdown_connections 2024-03-28T22:36:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.634+0000 7f9ba6a00640 1 --2- 172.21.15.43:0/1500072100 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9b840a1e00 0x7f9b840a42c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.634+0000 7f9ba6a00640 1 --2- 172.21.15.43:0/1500072100 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ba0077cf0 0x7f9ba01a5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.634+0000 7f9ba6a00640 1 --2- 172.21.15.43:0/1500072100 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ba0077350 0x7f9ba019e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.634+0000 7f9ba6a00640 1 --2- 172.21.15.43:0/1500072100 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ba0076110 0x7f9ba019e0c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.634+0000 7f9ba6a00640 1 -- 172.21.15.43:0/1500072100 >> 172.21.15.43:0/1500072100 conn(0x7f9ba00fef00 msgr2=0x7f9ba0100a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:35.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.634+0000 7f9ba6a00640 1 -- 172.21.15.43:0/1500072100 shutdown_connections 2024-03-28T22:36:35.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.634+0000 7f9ba6a00640 1 -- 172.21.15.43:0/1500072100 wait complete. 2024-03-28T22:36:35.652 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T22:36:35.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.881+0000 7f2e67e00640 1 -- 172.21.15.43:0/2828244727 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e68075790 msgr2=0x7f2e68073be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:35.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.881+0000 7f2e67e00640 1 --2- 172.21.15.43:0/2828244727 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e68075790 0x7f2e68073be0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2e58009cd0 tx=0x7f2e5802c0f0 comp rx=0 tx=0).stop 2024-03-28T22:36:35.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.881+0000 7f2e67e00640 1 -- 172.21.15.43:0/2828244727 shutdown_connections 2024-03-28T22:36:35.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.881+0000 7f2e67e00640 1 --2- 172.21.15.43:0/2828244727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2e6810dc20 0x7f2e68110010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:35.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.881+0000 7f2e67e00640 1 --2- 172.21.15.43:0/2828244727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e680741b0 0x7f2e68074630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:35.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.881+0000 7f2e67e00640 1 --2- 172.21.15.43:0/2828244727 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e68075790 0x7f2e68073be0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:35.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.881+0000 7f2e67e00640 1 -- 172.21.15.43:0/2828244727 >> 172.21.15.43:0/2828244727 conn(0x7f2e680fcbc0 msgr2=0x7f2e680ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:35.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.881+0000 7f2e67e00640 1 -- 172.21.15.43:0/2828244727 shutdown_connections 2024-03-28T22:36:35.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.881+0000 7f2e67e00640 1 -- 172.21.15.43:0/2828244727 wait complete. 2024-03-28T22:36:35.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.882+0000 7f2e67e00640 1 Processor -- start 2024-03-28T22:36:35.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.882+0000 7f2e67e00640 1 -- start start 2024-03-28T22:36:35.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.882+0000 7f2e67e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e680741b0 0x7f2e6819dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:35.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.883+0000 7f2e67e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e68075790 0x7f2e6819e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:35.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.883+0000 7f2e67e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2e6810dc20 0x7f2e681a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:35.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.883+0000 7f2e67e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2e681138c0 con 0x7f2e6810dc20 2024-03-28T22:36:35.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.883+0000 7f2e67e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f2e68113740 con 0x7f2e68075790 2024-03-28T22:36:35.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.883+0000 7f2e67e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f2e68113a40 con 0x7f2e680741b0 2024-03-28T22:36:35.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.883+0000 7f2e66a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e680741b0 0x7f2e6819dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:35.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.883+0000 7f2e5e000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e68075790 0x7f2e6819e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:35.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.883+0000 7f2e66a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e680741b0 0x7f2e6819dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:53520/0 (socket says 172.21.15.43:53520) 2024-03-28T22:36:35.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.883+0000 7f2e66a00640 1 -- 172.21.15.43:0/3559683634 learned_addr learned my addr 172.21.15.43:0/3559683634 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:35.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.883+0000 7f2e67400640 1 --2- 172.21.15.43:0/3559683634 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2e6810dc20 0x7f2e681a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:35.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.883+0000 7f2e66a00640 1 -- 172.21.15.43:0/3559683634 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e68075790 msgr2=0x7f2e6819e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:35.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.883+0000 7f2e66a00640 1 --2- 172.21.15.43:0/3559683634 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e68075790 0x7f2e6819e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:35.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.883+0000 7f2e66a00640 1 -- 172.21.15.43:0/3559683634 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2e6810dc20 msgr2=0x7f2e681a5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:35.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.883+0000 7f2e66a00640 1 --2- 172.21.15.43:0/3559683634 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2e6810dc20 0x7f2e681a5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:35.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.883+0000 7f2e66a00640 1 -- 172.21.15.43:0/3559683634 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e681a5bc0 con 0x7f2e680741b0 2024-03-28T22:36:35.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.884+0000 7f2e66a00640 1 --2- 172.21.15.43:0/3559683634 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e680741b0 0x7f2e6819dfa0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2e58000f80 tx=0x7f2e58032ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:35.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.884+0000 7f2e5fe00640 1 -- 172.21.15.43:0/3559683634 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e58041020 con 0x7f2e680741b0 2024-03-28T22:36:35.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.884+0000 7f2e5fe00640 1 -- 172.21.15.43:0/3559683634 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e58034cd0 con 0x7f2e680741b0 2024-03-28T22:36:35.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.884+0000 7f2e67e00640 1 -- 172.21.15.43:0/3559683634 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e681a5e50 con 0x7f2e680741b0 2024-03-28T22:36:35.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.884+0000 7f2e5fe00640 1 -- 172.21.15.43:0/3559683634 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e58039040 con 0x7f2e680741b0 2024-03-28T22:36:35.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.884+0000 7f2e67e00640 1 -- 172.21.15.43:0/3559683634 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e681a62b0 con 0x7f2e680741b0 2024-03-28T22:36:35.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.885+0000 7f2e67e00640 1 -- 172.21.15.43:0/3559683634 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e68103df0 con 0x7f2e680741b0 2024-03-28T22:36:35.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.887+0000 7f2e5fe00640 1 -- 172.21.15.43:0/3559683634 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f2e58044020 con 0x7f2e680741b0 2024-03-28T22:36:35.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.888+0000 7f2e5fe00640 1 --2- 172.21.15.43:0/3559683634 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f2e4c0a1e00 0x7f2e4c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:35.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.888+0000 7f2e5fe00640 1 -- 172.21.15.43:0/3559683634 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f2e580effe0 con 0x7f2e680741b0 2024-03-28T22:36:35.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.888+0000 7f2e5e000640 1 --2- 172.21.15.43:0/3559683634 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f2e4c0a1e00 0x7f2e4c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:35.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.889+0000 7f2e5e000640 1 --2- 172.21.15.43:0/3559683634 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f2e4c0a1e00 0x7f2e4c0a42c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2e6819f4b0 tx=0x7f2e50005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:35.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:35.892+0000 7f2e5fe00640 1 -- 172.21.15.43:0/3559683634 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2e5804b020 con 0x7f2e680741b0 2024-03-28T22:36:36.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:35 smithi043 ceph-mon[31823]: pgmap v419: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-03-28T22:36:36.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:35 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2377276303' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:36:36.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:35 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1500072100' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:36:36.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.094+0000 7f2e67e00640 1 -- 172.21.15.43:0/3559683634 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2e68062350 con 0x7f2e680741b0 2024-03-28T22:36:36.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.095+0000 7f2e5fe00640 1 -- 172.21.15.43:0/3559683634 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) ==== 100+0+11 (secure 0 0 0) 0x7f2e580bd020 con 0x7f2e680741b0 2024-03-28T22:36:36.094 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-03-28T22:36:36.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.097+0000 7f2e67e00640 1 -- 172.21.15.43:0/3559683634 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f2e4c0a1e00 msgr2=0x7f2e4c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:36.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.097+0000 7f2e67e00640 1 --2- 172.21.15.43:0/3559683634 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f2e4c0a1e00 0x7f2e4c0a42c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2e6819f4b0 tx=0x7f2e50005f90 comp rx=0 tx=0).stop 2024-03-28T22:36:36.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.097+0000 7f2e67e00640 1 -- 172.21.15.43:0/3559683634 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e680741b0 msgr2=0x7f2e6819dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:36.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.097+0000 7f2e67e00640 1 --2- 172.21.15.43:0/3559683634 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e680741b0 0x7f2e6819dfa0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2e58000f80 tx=0x7f2e58032ec0 comp rx=0 tx=0).stop 2024-03-28T22:36:36.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.097+0000 7f2e67e00640 1 -- 172.21.15.43:0/3559683634 shutdown_connections 2024-03-28T22:36:36.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.097+0000 7f2e67e00640 1 --2- 172.21.15.43:0/3559683634 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f2e4c0a1e00 0x7f2e4c0a42c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:36.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.097+0000 7f2e67e00640 1 --2- 172.21.15.43:0/3559683634 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e68075790 0x7f2e6819e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:36.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.097+0000 7f2e67e00640 1 --2- 172.21.15.43:0/3559683634 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2e6810dc20 0x7f2e681a5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:36.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.098+0000 7f2e67e00640 1 --2- 172.21.15.43:0/3559683634 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e680741b0 0x7f2e6819dfa0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:36.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.098+0000 7f2e67e00640 1 -- 172.21.15.43:0/3559683634 >> 172.21.15.43:0/3559683634 conn(0x7f2e680fcbc0 msgr2=0x7f2e68100fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:36.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.098+0000 7f2e67e00640 1 -- 172.21.15.43:0/3559683634 shutdown_connections 2024-03-28T22:36:36.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.098+0000 7f2e67e00640 1 -- 172.21.15.43:0/3559683634 wait complete. 2024-03-28T22:36:36.115 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T22:36:36.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:35 smithi172 ceph-mon[36566]: pgmap v419: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-03-28T22:36:36.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:35 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2377276303' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:36:36.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:35 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1500072100' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:36:36.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:35 smithi082 ceph-mon[36471]: pgmap v419: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-03-28T22:36:36.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:35 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2377276303' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:36:36.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:35 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1500072100' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:36:36.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.344+0000 7f561b200640 1 -- 172.21.15.43:0/4285495840 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f561c103870 msgr2=0x7f561c103c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:36.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.344+0000 7f561b200640 1 --2- 172.21.15.43:0/4285495840 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f561c103870 0x7f561c103c90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f560c009900 tx=0x7f560c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:36:36.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.344+0000 7f561b200640 1 -- 172.21.15.43:0/4285495840 shutdown_connections 2024-03-28T22:36:36.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.344+0000 7f561b200640 1 --2- 172.21.15.43:0/4285495840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f561c105420 0x7f561c10bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:36.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.344+0000 7f561b200640 1 --2- 172.21.15.43:0/4285495840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f561c104a60 0x7f561c104ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:36.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.344+0000 7f561b200640 1 --2- 172.21.15.43:0/4285495840 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f561c103870 0x7f561c103c90 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:36.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.344+0000 7f561b200640 1 -- 172.21.15.43:0/4285495840 >> 172.21.15.43:0/4285495840 conn(0x7f561c0fee00 msgr2=0x7f561c101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:36.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.344+0000 7f561b200640 1 -- 172.21.15.43:0/4285495840 shutdown_connections 2024-03-28T22:36:36.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.344+0000 7f561b200640 1 -- 172.21.15.43:0/4285495840 wait complete. 2024-03-28T22:36:36.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.345+0000 7f561b200640 1 Processor -- start 2024-03-28T22:36:36.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.346+0000 7f561b200640 1 -- start start 2024-03-28T22:36:36.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.346+0000 7f561b200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f561c103870 0x7f561c199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:36.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.346+0000 7f561b200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f561c104a60 0x7f561c19a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:36.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.346+0000 7f561b200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f561c105420 0x7f561c1a1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:36.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.346+0000 7f561b200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f561c10f540 con 0x7f561c105420 2024-03-28T22:36:36.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.346+0000 7f561b200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f561c10f3c0 con 0x7f561c104a60 2024-03-28T22:36:36.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.346+0000 7f561b200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f561c10f6c0 con 0x7f561c103870 2024-03-28T22:36:36.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.346+0000 7f5619e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f561c103870 0x7f561c199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:36.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.346+0000 7f5619400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f561c104a60 0x7f561c19a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:36.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.346+0000 7f5619e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f561c103870 0x7f561c199b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:53534/0 (socket says 172.21.15.43:53534) 2024-03-28T22:36:36.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.346+0000 7f5619400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f561c104a60 0x7f561c19a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47096/0 (socket says 172.21.15.43:47096) 2024-03-28T22:36:36.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.346+0000 7f5619e00640 1 -- 172.21.15.43:0/3515465138 learned_addr learned my addr 172.21.15.43:0/3515465138 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:36.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.346+0000 7f561a800640 1 --2- 172.21.15.43:0/3515465138 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f561c105420 0x7f561c1a1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:36.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.347+0000 7f5619e00640 1 -- 172.21.15.43:0/3515465138 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f561c104a60 msgr2=0x7f561c19a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:36.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.347+0000 7f5619e00640 1 --2- 172.21.15.43:0/3515465138 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f561c104a60 0x7f561c19a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:36.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.347+0000 7f5619e00640 1 -- 172.21.15.43:0/3515465138 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f561c105420 msgr2=0x7f561c1a1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:36.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.347+0000 7f5619e00640 1 --2- 172.21.15.43:0/3515465138 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f561c105420 0x7f561c1a1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:36.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.347+0000 7f5619e00640 1 -- 172.21.15.43:0/3515465138 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f561c1a1780 con 0x7f561c103870 2024-03-28T22:36:36.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.347+0000 7f5619e00640 1 --2- 172.21.15.43:0/3515465138 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f561c103870 0x7f561c199b60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f560c000c00 tx=0x7f560c02c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:36.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.348+0000 7f5612a00640 1 -- 172.21.15.43:0/3515465138 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f560c043020 con 0x7f561c103870 2024-03-28T22:36:36.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.348+0000 7f5612a00640 1 -- 172.21.15.43:0/3515465138 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f560c0054e0 con 0x7f561c103870 2024-03-28T22:36:36.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.348+0000 7f5612a00640 1 -- 172.21.15.43:0/3515465138 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f560c003c40 con 0x7f561c103870 2024-03-28T22:36:36.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.348+0000 7f561b200640 1 -- 172.21.15.43:0/3515465138 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f561c1a1a10 con 0x7f561c103870 2024-03-28T22:36:36.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.348+0000 7f561b200640 1 -- 172.21.15.43:0/3515465138 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f561c1a1f10 con 0x7f561c103870 2024-03-28T22:36:36.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.349+0000 7f561b200640 1 -- 172.21.15.43:0/3515465138 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f561c10ab30 con 0x7f561c103870 2024-03-28T22:36:36.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.351+0000 7f5612a00640 1 -- 172.21.15.43:0/3515465138 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f560c046020 con 0x7f561c103870 2024-03-28T22:36:36.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.352+0000 7f5612a00640 1 --2- 172.21.15.43:0/3515465138 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f56000a1d80 0x7f56000a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:36.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.352+0000 7f5612a00640 1 -- 172.21.15.43:0/3515465138 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f560c0f0150 con 0x7f561c103870 2024-03-28T22:36:36.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.353+0000 7f5619400640 1 --2- 172.21.15.43:0/3515465138 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f56000a1d80 0x7f56000a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:36.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.353+0000 7f5619400640 1 --2- 172.21.15.43:0/3515465138 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f56000a1d80 0x7f56000a4240 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f561c19b070 tx=0x7f5608005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:36.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.356+0000 7f5612a00640 1 -- 172.21.15.43:0/3515465138 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f560c04d020 con 0x7f561c103870 2024-03-28T22:36:36.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.562+0000 7f561b200640 1 -- 172.21.15.43:0/3515465138 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f561c02d390 con 0x7f561c103870 2024-03-28T22:36:36.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.563+0000 7f5612a00640 1 -- 172.21.15.43:0/3515465138 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v96) ==== 103+0+11 (secure 0 0 0) 0x7f560c0bd190 con 0x7f561c103870 2024-03-28T22:36:36.562 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-03-28T22:36:36.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.566+0000 7f561b200640 1 -- 172.21.15.43:0/3515465138 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f56000a1d80 msgr2=0x7f56000a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:36.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.566+0000 7f561b200640 1 --2- 172.21.15.43:0/3515465138 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f56000a1d80 0x7f56000a4240 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f561c19b070 tx=0x7f5608005f90 comp rx=0 tx=0).stop 2024-03-28T22:36:36.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.566+0000 7f561b200640 1 -- 172.21.15.43:0/3515465138 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f561c103870 msgr2=0x7f561c199b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:36.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.566+0000 7f561b200640 1 --2- 172.21.15.43:0/3515465138 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f561c103870 0x7f561c199b60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f560c000c00 tx=0x7f560c02c650 comp rx=0 tx=0).stop 2024-03-28T22:36:36.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.566+0000 7f561b200640 1 -- 172.21.15.43:0/3515465138 shutdown_connections 2024-03-28T22:36:36.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.566+0000 7f561b200640 1 --2- 172.21.15.43:0/3515465138 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f56000a1d80 0x7f56000a4240 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:36.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.566+0000 7f561b200640 1 --2- 172.21.15.43:0/3515465138 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f561c104a60 0x7f561c19a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:36.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.566+0000 7f561b200640 1 --2- 172.21.15.43:0/3515465138 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f561c105420 0x7f561c1a1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:36.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.566+0000 7f561b200640 1 --2- 172.21.15.43:0/3515465138 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f561c103870 0x7f561c199b60 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:36.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.566+0000 7f561b200640 1 -- 172.21.15.43:0/3515465138 >> 172.21.15.43:0/3515465138 conn(0x7f561c0fee00 msgr2=0x7f561c107d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:36.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.566+0000 7f561b200640 1 -- 172.21.15.43:0/3515465138 shutdown_connections 2024-03-28T22:36:36.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.566+0000 7f561b200640 1 -- 172.21.15.43:0/3515465138 wait complete. 2024-03-28T22:36:36.586 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:36:36.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.809+0000 7f5010c00640 1 -- 172.21.15.43:0/491792192 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f500c075790 msgr2=0x7f500c073be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:36.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.809+0000 7f5010c00640 1 --2- 172.21.15.43:0/491792192 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f500c075790 0x7f500c073be0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc009ca0 tx=0x7f4ffc02c100 comp rx=0 tx=0).stop 2024-03-28T22:36:36.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.809+0000 7f5010c00640 1 -- 172.21.15.43:0/491792192 shutdown_connections 2024-03-28T22:36:36.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.809+0000 7f5010c00640 1 --2- 172.21.15.43:0/491792192 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f500c10db80 0x7f500c10ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:36.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.809+0000 7f5010c00640 1 --2- 172.21.15.43:0/491792192 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f500c0741b0 0x7f500c074630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:36.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.809+0000 7f5010c00640 1 --2- 172.21.15.43:0/491792192 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f500c075790 0x7f500c073be0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:36.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.809+0000 7f5010c00640 1 -- 172.21.15.43:0/491792192 >> 172.21.15.43:0/491792192 conn(0x7f500c0fcb20 msgr2=0x7f500c0fef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:36.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.810+0000 7f5010c00640 1 -- 172.21.15.43:0/491792192 shutdown_connections 2024-03-28T22:36:36.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.810+0000 7f5010c00640 1 -- 172.21.15.43:0/491792192 wait complete. 2024-03-28T22:36:36.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.811+0000 7f5010c00640 1 Processor -- start 2024-03-28T22:36:36.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.811+0000 7f5010c00640 1 -- start start 2024-03-28T22:36:36.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.811+0000 7f5010c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f500c0741b0 0x7f500c19de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:36.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.812+0000 7f5010c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f500c075790 0x7f500c19e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:36.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.812+0000 7f5010c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f500c10db80 0x7f500c1a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:36.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.812+0000 7f5010c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f500c113800 con 0x7f500c10db80 2024-03-28T22:36:36.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.812+0000 7f5010c00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f500c113680 con 0x7f500c0741b0 2024-03-28T22:36:36.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.812+0000 7f5010c00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f500c113980 con 0x7f500c075790 2024-03-28T22:36:36.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.812+0000 7f500b400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f500c0741b0 0x7f500c19de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:36.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.812+0000 7f500aa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f500c075790 0x7f500c19e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:36.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.812+0000 7f500b400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f500c0741b0 0x7f500c19de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47114/0 (socket says 172.21.15.43:47114) 2024-03-28T22:36:36.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.812+0000 7f500b400640 1 -- 172.21.15.43:0/2385182462 learned_addr learned my addr 172.21.15.43:0/2385182462 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:36.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.812+0000 7f500be00640 1 --2- 172.21.15.43:0/2385182462 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f500c10db80 0x7f500c1a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:36.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.812+0000 7f500b400640 1 -- 172.21.15.43:0/2385182462 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f500c075790 msgr2=0x7f500c19e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:36.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.812+0000 7f500b400640 1 --2- 172.21.15.43:0/2385182462 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f500c075790 0x7f500c19e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:36.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.812+0000 7f500b400640 1 -- 172.21.15.43:0/2385182462 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f500c10db80 msgr2=0x7f500c1a5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:36.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.812+0000 7f500b400640 1 --2- 172.21.15.43:0/2385182462 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f500c10db80 0x7f500c1a5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:36.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.812+0000 7f500b400640 1 -- 172.21.15.43:0/2385182462 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f500c1a5a90 con 0x7f500c0741b0 2024-03-28T22:36:36.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.812+0000 7f500be00640 1 --2- 172.21.15.43:0/2385182462 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f500c10db80 0x7f500c1a5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:36:36.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.813+0000 7f500b400640 1 --2- 172.21.15.43:0/2385182462 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f500c0741b0 0x7f500c19de70 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc0325f0 tx=0x7f4ffc034670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:36.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.813+0000 7f5003e00640 1 -- 172.21.15.43:0/2385182462 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ffc039050 con 0x7f500c0741b0 2024-03-28T22:36:36.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.813+0000 7f5003e00640 1 -- 172.21.15.43:0/2385182462 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ffc045070 con 0x7f500c0741b0 2024-03-28T22:36:36.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.813+0000 7f5003e00640 1 -- 172.21.15.43:0/2385182462 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ffc004790 con 0x7f500c0741b0 2024-03-28T22:36:36.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.813+0000 7f5010c00640 1 -- 172.21.15.43:0/2385182462 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f500c1a5d20 con 0x7f500c0741b0 2024-03-28T22:36:36.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.813+0000 7f5010c00640 1 -- 172.21.15.43:0/2385182462 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f500c1a61c0 con 0x7f500c0741b0 2024-03-28T22:36:36.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.814+0000 7f5010c00640 1 -- 172.21.15.43:0/2385182462 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f500c103d50 con 0x7f500c0741b0 2024-03-28T22:36:36.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.816+0000 7f5003e00640 1 -- 172.21.15.43:0/2385182462 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f4ffc003970 con 0x7f500c0741b0 2024-03-28T22:36:36.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.817+0000 7f5003e00640 1 --2- 172.21.15.43:0/2385182462 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4ff00a1d80 0x7f4ff00a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:36.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.817+0000 7f5003e00640 1 -- 172.21.15.43:0/2385182462 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f4ffc0f09b0 con 0x7f500c0741b0 2024-03-28T22:36:36.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.817+0000 7f500aa00640 1 --2- 172.21.15.43:0/2385182462 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4ff00a1d80 0x7f4ff00a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:36.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.818+0000 7f500aa00640 1 --2- 172.21.15.43:0/2385182462 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4ff00a1d80 0x7f4ff00a4240 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f500c19f380 tx=0x7f4ff400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:36.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:36.821+0000 7f5003e00640 1 -- 172.21.15.43:0/2385182462 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4ffc0bd9f0 con 0x7f500c0741b0 2024-03-28T22:36:37.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.064+0000 7f5010c00640 1 -- 172.21.15.43:0/2385182462 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f500c062350 con 0x7f500c0741b0 2024-03-28T22:36:37.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.065+0000 7f5003e00640 1 -- 172.21.15.43:0/2385182462 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11934 (secure 0 0 0) 0x7f4ffc0bd2f0 con 0x7f500c0741b0 2024-03-28T22:36:37.065 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:36:37.065 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:33:28.906396+0000","modified":"2024-03-28T22:34:06.174761+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24572,"mds_1":24556,"mds_2":24535,"mds_3":24523,"mds_4":24518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/1084988072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1084988072},{"type":"v1","addr":"172.21.15.82:6835","nonce":1084988072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24535,24556,24518,24523,24572]},"id":1}]} 2024-03-28T22:36:37.065 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 28 2024-03-28T22:36:37.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.067+0000 7f5010c00640 1 -- 172.21.15.43:0/2385182462 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4ff00a1d80 msgr2=0x7f4ff00a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:37.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.067+0000 7f5010c00640 1 --2- 172.21.15.43:0/2385182462 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4ff00a1d80 0x7f4ff00a4240 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f500c19f380 tx=0x7f4ff400b410 comp rx=0 tx=0).stop 2024-03-28T22:36:37.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.067+0000 7f5010c00640 1 -- 172.21.15.43:0/2385182462 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f500c0741b0 msgr2=0x7f500c19de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:37.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.067+0000 7f5010c00640 1 --2- 172.21.15.43:0/2385182462 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f500c0741b0 0x7f500c19de70 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc0325f0 tx=0x7f4ffc034670 comp rx=0 tx=0).stop 2024-03-28T22:36:37.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.067+0000 7f5010c00640 1 -- 172.21.15.43:0/2385182462 shutdown_connections 2024-03-28T22:36:37.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.067+0000 7f5010c00640 1 --2- 172.21.15.43:0/2385182462 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f500c10db80 0x7f500c1a5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:37.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.067+0000 7f5010c00640 1 --2- 172.21.15.43:0/2385182462 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f500c075790 0x7f500c19e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:37.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.067+0000 7f5010c00640 1 --2- 172.21.15.43:0/2385182462 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4ff00a1d80 0x7f4ff00a4240 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:37.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.067+0000 7f5010c00640 1 --2- 172.21.15.43:0/2385182462 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f500c0741b0 0x7f500c19de70 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:37.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.067+0000 7f5010c00640 1 -- 172.21.15.43:0/2385182462 >> 172.21.15.43:0/2385182462 conn(0x7f500c0fcb20 msgr2=0x7f500c100f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:37.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.068+0000 7f5010c00640 1 -- 172.21.15.43:0/2385182462 shutdown_connections 2024-03-28T22:36:37.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.068+0000 7f5010c00640 1 -- 172.21.15.43:0/2385182462 wait complete. 2024-03-28T22:36:37.084 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:36:37.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:36 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3559683634' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:36:37.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:36 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3515465138' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:36:37.169 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:36 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3559683634' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:36:37.169 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:36 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3515465138' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:36:37.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:36 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3559683634' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:36:37.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:36 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3515465138' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:36:37.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.303+0000 7fe127200640 1 -- 172.21.15.43:0/2527541080 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe128108da0 msgr2=0x7fe128109220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:37.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.303+0000 7fe127200640 1 --2- 172.21.15.43:0/2527541080 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe128108da0 0x7fe128109220 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe110009900 tx=0x7fe11002bc40 comp rx=0 tx=0).stop 2024-03-28T22:36:37.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.303+0000 7fe127200640 1 -- 172.21.15.43:0/2527541080 shutdown_connections 2024-03-28T22:36:37.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.303+0000 7fe127200640 1 --2- 172.21.15.43:0/2527541080 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe128109760 0x7fe12810fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:37.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.303+0000 7fe127200640 1 --2- 172.21.15.43:0/2527541080 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe128108da0 0x7fe128109220 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:37.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.303+0000 7fe127200640 1 --2- 172.21.15.43:0/2527541080 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe128107bb0 0x7fe128107fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:37.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.303+0000 7fe127200640 1 -- 172.21.15.43:0/2527541080 >> 172.21.15.43:0/2527541080 conn(0x7fe128076b60 msgr2=0x7fe128078f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:37.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.304+0000 7fe127200640 1 -- 172.21.15.43:0/2527541080 shutdown_connections 2024-03-28T22:36:37.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.304+0000 7fe127200640 1 -- 172.21.15.43:0/2527541080 wait complete. 2024-03-28T22:36:37.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.304+0000 7fe127200640 1 Processor -- start 2024-03-28T22:36:37.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.305+0000 7fe127200640 1 -- start start 2024-03-28T22:36:37.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.305+0000 7fe127200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe128107bb0 0x7fe12819deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:37.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.305+0000 7fe127200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe128108da0 0x7fe12819e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:37.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.305+0000 7fe127200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe128109760 0x7fe1281a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:37.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.305+0000 7fe127200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe128113860 con 0x7fe128109760 2024-03-28T22:36:37.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.305+0000 7fe127200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe1281136e0 con 0x7fe128107bb0 2024-03-28T22:36:37.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.305+0000 7fe127200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe1281139e0 con 0x7fe128108da0 2024-03-28T22:36:37.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.305+0000 7fe125e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe128107bb0 0x7fe12819deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:37.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.305+0000 7fe11fe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe128108da0 0x7fe12819e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:37.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.305+0000 7fe125e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe128107bb0 0x7fe12819deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47122/0 (socket says 172.21.15.43:47122) 2024-03-28T22:36:37.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.305+0000 7fe11fe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe128108da0 0x7fe12819e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:53584/0 (socket says 172.21.15.43:53584) 2024-03-28T22:36:37.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.305+0000 7fe125e00640 1 -- 172.21.15.43:0/3849751157 learned_addr learned my addr 172.21.15.43:0/3849751157 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:37.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.305+0000 7fe126800640 1 --2- 172.21.15.43:0/3849751157 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe128109760 0x7fe1281a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:37.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.306+0000 7fe11fe00640 1 -- 172.21.15.43:0/3849751157 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe128107bb0 msgr2=0x7fe12819deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:37.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.306+0000 7fe11fe00640 1 --2- 172.21.15.43:0/3849751157 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe128107bb0 0x7fe12819deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:37.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.306+0000 7fe11fe00640 1 -- 172.21.15.43:0/3849751157 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe128109760 msgr2=0x7fe1281a5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:37.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.306+0000 7fe11fe00640 1 --2- 172.21.15.43:0/3849751157 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe128109760 0x7fe1281a5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:37.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.306+0000 7fe11fe00640 1 -- 172.21.15.43:0/3849751157 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1281a5ad0 con 0x7fe128108da0 2024-03-28T22:36:37.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.306+0000 7fe126800640 1 --2- 172.21.15.43:0/3849751157 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe128109760 0x7fe1281a5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:36:37.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.306+0000 7fe11fe00640 1 --2- 172.21.15.43:0/3849751157 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe128108da0 0x7fe12819e3f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe110036040 tx=0x7fe11002c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:37.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.306+0000 7fe11ea00640 1 -- 172.21.15.43:0/3849751157 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe110041070 con 0x7fe128108da0 2024-03-28T22:36:37.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.306+0000 7fe11ea00640 1 -- 172.21.15.43:0/3849751157 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe110004450 con 0x7fe128108da0 2024-03-28T22:36:37.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.306+0000 7fe11ea00640 1 -- 172.21.15.43:0/3849751157 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe110003e60 con 0x7fe128108da0 2024-03-28T22:36:37.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.306+0000 7fe127200640 1 -- 172.21.15.43:0/3849751157 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1281a5d60 con 0x7fe128108da0 2024-03-28T22:36:37.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.306+0000 7fe127200640 1 -- 172.21.15.43:0/3849751157 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1281a6210 con 0x7fe128108da0 2024-03-28T22:36:37.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.307+0000 7fe127200640 1 -- 172.21.15.43:0/3849751157 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe12810eed0 con 0x7fe128108da0 2024-03-28T22:36:37.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.309+0000 7fe11ea00640 1 -- 172.21.15.43:0/3849751157 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fe110044070 con 0x7fe128108da0 2024-03-28T22:36:37.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.311+0000 7fe11ea00640 1 --2- 172.21.15.43:0/3849751157 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe10c0a1e00 0x7fe10c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:37.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.311+0000 7fe11ea00640 1 -- 172.21.15.43:0/3849751157 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7fe1100efe20 con 0x7fe128108da0 2024-03-28T22:36:37.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.311+0000 7fe125e00640 1 --2- 172.21.15.43:0/3849751157 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe10c0a1e00 0x7fe10c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:37.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.312+0000 7fe125e00640 1 --2- 172.21.15.43:0/3849751157 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe10c0a1e00 0x7fe10c0a42c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe1180058c0 tx=0x7fe118005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:37.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.313+0000 7fe11ea00640 1 -- 172.21.15.43:0/3849751157 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe1100bce60 con 0x7fe128108da0 2024-03-28T22:36:37.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.507+0000 7fe127200640 1 -- 172.21.15.43:0/3849751157 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe12802d3c0 con 0x7fe128108da0 2024-03-28T22:36:37.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.509+0000 7fe11ea00640 1 -- 172.21.15.43:0/3849751157 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+24712 (secure 0 0 0) 0x7fe1100bc760 con 0x7fe128108da0 2024-03-28T22:36:37.508 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:36:37.508 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":96,"fsid":"c2276428-ed51-11ee-95d0-87774f69a715","created":"2024-03-28T22:24:19.950084+0000","modified":"2024-03-28T22:35:30.237654+0000","last_up_change":"2024-03-28T22:32:31.078737+0000","last_in_change":"2024-03-28T22:32:10.147097+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:28:14.713290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:33:27.167435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:33:28.411825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T22:33:32.523122+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b855adcd-786c-4e42-b8fe-a515a77ce979","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6803","nonce":2826124409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6805","nonce":2826124409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6809","nonce":2826124409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6807","nonce":2826124409}]},"public_addr":"172.21.15.43:6803/2826124409","cluster_addr":"172.21.15.43:6805/2826124409","heartbeat_back_addr":"172.21.15.43:6809/2826124409","heartbeat_front_addr":"172.21.15.43:6807/2826124409","state":["exists","up"]},{"osd":1,"uuid":"031c143d-8ad1-4b68-8db9-3c717137e3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6801","nonce":3043263644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6803","nonce":3043263644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6807","nonce":3043263644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6805","nonce":3043263644}]},"public_addr":"172.21.15.82:6801/3043263644","cluster_addr":"172.21.15.82:6803/3043263644","heartbeat_back_addr":"172.21.15.82:6807/3043263644","heartbeat_front_addr":"172.21.15.82:6805/3043263644","state":["exists","up"]},{"osd":2,"uuid":"20632bbb-f07a-4ca1-b0c4-ff9d508e211c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6801","nonce":4213953645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6803","nonce":4213953645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6807","nonce":4213953645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6805","nonce":4213953645}]},"public_addr":"172.21.15.172:6801/4213953645","cluster_addr":"172.21.15.172:6803/4213953645","heartbeat_back_addr":"172.21.15.172:6807/4213953645","heartbeat_front_addr":"172.21.15.172:6805/4213953645","state":["exists","up"]},{"osd":3,"uuid":"9a1f2312-4dbd-475b-8736-d5ebc8788e09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6811","nonce":1597280920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6813","nonce":1597280920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6817","nonce":1597280920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6815","nonce":1597280920}]},"public_addr":"172.21.15.43:6811/1597280920","cluster_addr":"172.21.15.43:6813/1597280920","heartbeat_back_addr":"172.21.15.43:6817/1597280920","heartbeat_front_addr":"172.21.15.43:6815/1597280920","state":["exists","up"]},{"osd":4,"uuid":"29140ead-7075-42b2-90c1-27cae0b29ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6809","nonce":215451895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6811","nonce":215451895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6815","nonce":215451895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6813","nonce":215451895}]},"public_addr":"172.21.15.82:6809/215451895","cluster_addr":"172.21.15.82:6811/215451895","heartbeat_back_addr":"172.21.15.82:6815/215451895","heartbeat_front_addr":"172.21.15.82:6813/215451895","state":["exists","up"]},{"osd":5,"uuid":"59644c61-93dc-4536-b2ad-0256b24e5cb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6809","nonce":1576148933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6811","nonce":1576148933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6815","nonce":1576148933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6813","nonce":1576148933}]},"public_addr":"172.21.15.172:6809/1576148933","cluster_addr":"172.21.15.172:6811/1576148933","heartbeat_back_addr":"172.21.15.172:6815/1576148933","heartbeat_front_addr":"172.21.15.172:6813/1576148933","state":["exists","up"]},{"osd":6,"uuid":"22d54e57-c3f1-4ecf-9921-8e815b48f92a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6819","nonce":791044279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6821","nonce":791044279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6825","nonce":791044279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6823","nonce":791044279}]},"public_addr":"172.21.15.43:6819/791044279","cluster_addr":"172.21.15.43:6821/791044279","heartbeat_back_addr":"172.21.15.43:6825/791044279","heartbeat_front_addr":"172.21.15.43:6823/791044279","state":["exists","up"]},{"osd":7,"uuid":"248cd1e2-0d6b-452f-a0a8-c51fe1b975ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6817","nonce":2157777323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6819","nonce":2157777323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6823","nonce":2157777323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6821","nonce":2157777323}]},"public_addr":"172.21.15.82:6817/2157777323","cluster_addr":"172.21.15.82:6819/2157777323","heartbeat_back_addr":"172.21.15.82:6823/2157777323","heartbeat_front_addr":"172.21.15.82:6821/2157777323","state":["exists","up"]},{"osd":8,"uuid":"d98aab6e-8bae-4c24-906c-4f492f047b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6817","nonce":1095328248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6819","nonce":1095328248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6823","nonce":1095328248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6821","nonce":1095328248}]},"public_addr":"172.21.15.172:6817/1095328248","cluster_addr":"172.21.15.172:6819/1095328248","heartbeat_back_addr":"172.21.15.172:6823/1095328248","heartbeat_front_addr":"172.21.15.172:6821/1095328248","state":["exists","up"]},{"osd":9,"uuid":"19aa383e-a0f6-45ea-96cc-e7956535caa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6827","nonce":3657791498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6829","nonce":3657791498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6833","nonce":3657791498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6831","nonce":3657791498}]},"public_addr":"172.21.15.43:6827/3657791498","cluster_addr":"172.21.15.43:6829/3657791498","heartbeat_back_addr":"172.21.15.43:6833/3657791498","heartbeat_front_addr":"172.21.15.43:6831/3657791498","state":["exists","up"]},{"osd":10,"uuid":"666977e5-eaf7-41b9-a18f-c380e149345d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6825","nonce":3454954047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6827","nonce":3454954047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6831","nonce":3454954047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6829","nonce":3454954047}]},"public_addr":"172.21.15.82:6825/3454954047","cluster_addr":"172.21.15.82:6827/3454954047","heartbeat_back_addr":"172.21.15.82:6831/3454954047","heartbeat_front_addr":"172.21.15.82:6829/3454954047","state":["exists","up"]},{"osd":11,"uuid":"615cf3be-8076-470e-ad09-c880e7593717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6825","nonce":2563517144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6827","nonce":2563517144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6831","nonce":2563517144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6829","nonce":2563517144}]},"public_addr":"172.21.15.172:6825/2563517144","cluster_addr":"172.21.15.172:6827/2563517144","heartbeat_back_addr":"172.21.15.172:6831/2563517144","heartbeat_front_addr":"172.21.15.172:6829/2563517144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:21.727405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:47.100259+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:10.615438+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:43.632144+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:09.487940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:34.604463+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:09.594452+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:35.709795+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:01.184202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:35.455322+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:02.413166+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:28.550754+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.9","mappings":[{"from":5,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/578791143":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/4029313215":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/1574178482":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6801/3517712438":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6800/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/1375099638":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/2710117652":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6801/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/3215634168":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6800/3517712438":"2024-03-29T22:25:28.008215+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:36:37.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.511+0000 7fe127200640 1 -- 172.21.15.43:0/3849751157 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe10c0a1e00 msgr2=0x7fe10c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:37.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.511+0000 7fe127200640 1 --2- 172.21.15.43:0/3849751157 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe10c0a1e00 0x7fe10c0a42c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe1180058c0 tx=0x7fe118005f90 comp rx=0 tx=0).stop 2024-03-28T22:36:37.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.511+0000 7fe127200640 1 -- 172.21.15.43:0/3849751157 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe128108da0 msgr2=0x7fe12819e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:37.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.511+0000 7fe127200640 1 --2- 172.21.15.43:0/3849751157 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe128108da0 0x7fe12819e3f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe110036040 tx=0x7fe11002c660 comp rx=0 tx=0).stop 2024-03-28T22:36:37.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.511+0000 7fe127200640 1 -- 172.21.15.43:0/3849751157 shutdown_connections 2024-03-28T22:36:37.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.511+0000 7fe127200640 1 --2- 172.21.15.43:0/3849751157 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe128109760 0x7fe1281a5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:37.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.511+0000 7fe127200640 1 --2- 172.21.15.43:0/3849751157 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe128108da0 0x7fe12819e3f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:37.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.511+0000 7fe127200640 1 --2- 172.21.15.43:0/3849751157 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe10c0a1e00 0x7fe10c0a42c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:37.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.511+0000 7fe127200640 1 --2- 172.21.15.43:0/3849751157 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe128107bb0 0x7fe12819deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:37.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.511+0000 7fe127200640 1 -- 172.21.15.43:0/3849751157 >> 172.21.15.43:0/3849751157 conn(0x7fe128076b60 msgr2=0x7fe12810c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:37.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.511+0000 7fe127200640 1 -- 172.21.15.43:0/3849751157 shutdown_connections 2024-03-28T22:36:37.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.512+0000 7fe127200640 1 -- 172.21.15.43:0/3849751157 wait complete. 2024-03-28T22:36:37.531 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T22:28:14.713290+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T22:33:27.167435+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T22:33:28.411825+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T22:33:32.523122+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T22:36:37.531 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T22:36:37.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.758+0000 7f9c8be00640 1 -- 172.21.15.43:0/969550005 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9c8c108e60 msgr2=0x7f9c8c1092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:37.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.758+0000 7f9c8be00640 1 --2- 172.21.15.43:0/969550005 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9c8c108e60 0x7f9c8c1092e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f9c78009900 tx=0x7f9c7802bc40 comp rx=0 tx=0).stop 2024-03-28T22:36:37.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.758+0000 7f9c8be00640 1 -- 172.21.15.43:0/969550005 shutdown_connections 2024-03-28T22:36:37.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.758+0000 7f9c8be00640 1 --2- 172.21.15.43:0/969550005 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9c8c108e60 0x7f9c8c1092e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:37.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.758+0000 7f9c8be00640 1 --2- 172.21.15.43:0/969550005 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c8c109820 0x7f9c8c1100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:37.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.758+0000 7f9c8be00640 1 --2- 172.21.15.43:0/969550005 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c8c107c70 0x7f9c8c108090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:37.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.758+0000 7f9c8be00640 1 -- 172.21.15.43:0/969550005 >> 172.21.15.43:0/969550005 conn(0x7f9c8c076c20 msgr2=0x7f9c8c079040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:37.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.758+0000 7f9c8be00640 1 -- 172.21.15.43:0/969550005 shutdown_connections 2024-03-28T22:36:37.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.758+0000 7f9c8be00640 1 -- 172.21.15.43:0/969550005 wait complete. 2024-03-28T22:36:37.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.759+0000 7f9c8be00640 1 Processor -- start 2024-03-28T22:36:37.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.760+0000 7f9c8be00640 1 -- start start 2024-03-28T22:36:37.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.760+0000 7f9c8be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c8c107c70 0x7f9c8c19dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:37.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.760+0000 7f9c8be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c8c108e60 0x7f9c8c19e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:37.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.760+0000 7f9c8be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9c8c109820 0x7f9c8c1a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:37.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.760+0000 7f9c8be00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9c8c1138d0 con 0x7f9c8c108e60 2024-03-28T22:36:37.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.760+0000 7f9c8be00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9c8c113750 con 0x7f9c8c109820 2024-03-28T22:36:37.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.760+0000 7f9c8be00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9c8c113a50 con 0x7f9c8c107c70 2024-03-28T22:36:37.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.760+0000 7f9c8aa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c8c107c70 0x7f9c8c19dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:37.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.760+0000 7f9c8aa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c8c107c70 0x7f9c8c19dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:53602/0 (socket says 172.21.15.43:53602) 2024-03-28T22:36:37.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.760+0000 7f9c8aa00640 1 -- 172.21.15.43:0/3503126479 learned_addr learned my addr 172.21.15.43:0/3503126479 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:37.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.760+0000 7f9c8b400640 1 --2- 172.21.15.43:0/3503126479 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9c8c109820 0x7f9c8c1a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:37.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.761+0000 7f9c8a000640 1 --2- 172.21.15.43:0/3503126479 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c8c108e60 0x7f9c8c19e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:37.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.761+0000 7f9c8aa00640 1 -- 172.21.15.43:0/3503126479 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9c8c109820 msgr2=0x7f9c8c1a5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:37.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.761+0000 7f9c8aa00640 1 --2- 172.21.15.43:0/3503126479 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9c8c109820 0x7f9c8c1a5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:37.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.761+0000 7f9c8aa00640 1 -- 172.21.15.43:0/3503126479 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c8c108e60 msgr2=0x7f9c8c19e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:37.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.761+0000 7f9c8aa00640 1 --2- 172.21.15.43:0/3503126479 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c8c108e60 0x7f9c8c19e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:37.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.761+0000 7f9c8aa00640 1 -- 172.21.15.43:0/3503126479 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c8c1a5b70 con 0x7f9c8c107c70 2024-03-28T22:36:37.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.761+0000 7f9c8b400640 1 --2- 172.21.15.43:0/3503126479 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9c8c109820 0x7f9c8c1a5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:37.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.761+0000 7f9c8a000640 1 --2- 172.21.15.43:0/3503126479 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c8c108e60 0x7f9c8c19e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:36:37.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.761+0000 7f9c8aa00640 1 --2- 172.21.15.43:0/3503126479 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c8c107c70 0x7f9c8c19dfa0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c00e9c0 tx=0x7f9c7c00edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:37.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.761+0000 7f9c83400640 1 -- 172.21.15.43:0/3503126479 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c00ccc0 con 0x7f9c8c107c70 2024-03-28T22:36:37.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.761+0000 7f9c83400640 1 -- 172.21.15.43:0/3503126479 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c7c0035f0 con 0x7f9c8c107c70 2024-03-28T22:36:37.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.761+0000 7f9c83400640 1 -- 172.21.15.43:0/3503126479 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c00ce60 con 0x7f9c8c107c70 2024-03-28T22:36:37.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.762+0000 7f9c8be00640 1 -- 172.21.15.43:0/3503126479 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c8c1a5e60 con 0x7f9c8c107c70 2024-03-28T22:36:37.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.762+0000 7f9c8be00640 1 -- 172.21.15.43:0/3503126479 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c8c1a62d0 con 0x7f9c8c107c70 2024-03-28T22:36:37.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.763+0000 7f9c8be00640 1 -- 172.21.15.43:0/3503126479 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c8c10ef90 con 0x7f9c8c107c70 2024-03-28T22:36:37.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.765+0000 7f9c83400640 1 -- 172.21.15.43:0/3503126479 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f9c7c0037a0 con 0x7f9c8c107c70 2024-03-28T22:36:37.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.766+0000 7f9c83400640 1 --2- 172.21.15.43:0/3503126479 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9c700a1e00 0x7f9c700a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:37.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.766+0000 7f9c83400640 1 -- 172.21.15.43:0/3503126479 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f9c7c0ce520 con 0x7f9c8c107c70 2024-03-28T22:36:37.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.766+0000 7f9c8a000640 1 --2- 172.21.15.43:0/3503126479 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9c700a1e00 0x7f9c700a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:37.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.767+0000 7f9c8a000640 1 --2- 172.21.15.43:0/3503126479 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9c700a1e00 0x7f9c700a42c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9c78006010 tx=0x7f9c7802c8b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:37.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.768+0000 7f9c83400640 1 -- 172.21.15.43:0/3503126479 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9c7c09b560 con 0x7f9c8c107c70 2024-03-28T22:36:37.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.970+0000 7f9c8be00640 1 -- 172.21.15.43:0/3503126479 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9c8c02d170 con 0x7f9c8c107c70 2024-03-28T22:36:37.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.971+0000 7f9c83400640 1 -- 172.21.15.43:0/3503126479 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) ==== 93+0+10 (secure 0 0 0) 0x7f9c7c09ae60 con 0x7f9c8c107c70 2024-03-28T22:36:37.970 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-03-28T22:36:37.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.974+0000 7f9c8be00640 1 -- 172.21.15.43:0/3503126479 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9c700a1e00 msgr2=0x7f9c700a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:37.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.974+0000 7f9c8be00640 1 --2- 172.21.15.43:0/3503126479 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9c700a1e00 0x7f9c700a42c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9c78006010 tx=0x7f9c7802c8b0 comp rx=0 tx=0).stop 2024-03-28T22:36:37.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.974+0000 7f9c8be00640 1 -- 172.21.15.43:0/3503126479 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c8c107c70 msgr2=0x7f9c8c19dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:37.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.974+0000 7f9c8be00640 1 --2- 172.21.15.43:0/3503126479 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c8c107c70 0x7f9c8c19dfa0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c00e9c0 tx=0x7f9c7c00edc0 comp rx=0 tx=0).stop 2024-03-28T22:36:37.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.974+0000 7f9c8be00640 1 -- 172.21.15.43:0/3503126479 shutdown_connections 2024-03-28T22:36:37.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.974+0000 7f9c8be00640 1 --2- 172.21.15.43:0/3503126479 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9c700a1e00 0x7f9c700a42c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:37.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.974+0000 7f9c8be00640 1 --2- 172.21.15.43:0/3503126479 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9c8c109820 0x7f9c8c1a5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:37.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.974+0000 7f9c8be00640 1 --2- 172.21.15.43:0/3503126479 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9c8c108e60 0x7f9c8c19e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:37.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.974+0000 7f9c8be00640 1 --2- 172.21.15.43:0/3503126479 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9c8c107c70 0x7f9c8c19dfa0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:37.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.974+0000 7f9c8be00640 1 -- 172.21.15.43:0/3503126479 >> 172.21.15.43:0/3503126479 conn(0x7f9c8c076c20 msgr2=0x7f9c8c10c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:37.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.974+0000 7f9c8be00640 1 -- 172.21.15.43:0/3503126479 shutdown_connections 2024-03-28T22:36:37.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:37.974+0000 7f9c8be00640 1 -- 172.21.15.43:0/3503126479 wait complete. 2024-03-28T22:36:37.994 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T22:36:38.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:37 smithi172 ceph-mon[36566]: pgmap v420: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-28T22:36:38.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:37 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2385182462' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:36:38.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:37 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3849751157' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:36:38.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:37 smithi082 ceph-mon[36471]: pgmap v420: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-28T22:36:38.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:37 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2385182462' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:36:38.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:37 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3849751157' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:36:38.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.212+0000 7fcad8a00640 1 -- 172.21.15.43:0/2405093443 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcad4076110 msgr2=0x7fcad4076530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:38.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.212+0000 7fcad8a00640 1 --2- 172.21.15.43:0/2405093443 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcad4076110 0x7fcad4076530 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fcac4009ca0 tx=0x7fcac402c100 comp rx=0 tx=0).stop 2024-03-28T22:36:38.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.212+0000 7fcad8a00640 1 -- 172.21.15.43:0/2405093443 shutdown_connections 2024-03-28T22:36:38.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.212+0000 7fcad8a00640 1 --2- 172.21.15.43:0/2405093443 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcad4077cf0 0x7fcad4078590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:38.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.212+0000 7fcad8a00640 1 --2- 172.21.15.43:0/2405093443 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcad4077350 0x7fcad40777b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:38.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.212+0000 7fcad8a00640 1 --2- 172.21.15.43:0/2405093443 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcad4076110 0x7fcad4076530 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:38.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.212+0000 7fcad8a00640 1 -- 172.21.15.43:0/2405093443 >> 172.21.15.43:0/2405093443 conn(0x7fcad40fef00 msgr2=0x7fcad4101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:38.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.212+0000 7fcad8a00640 1 -- 172.21.15.43:0/2405093443 shutdown_connections 2024-03-28T22:36:38.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.212+0000 7fcad8a00640 1 -- 172.21.15.43:0/2405093443 wait complete. 2024-03-28T22:36:38.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.213+0000 7fcad8a00640 1 Processor -- start 2024-03-28T22:36:38.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.213+0000 7fcad8a00640 1 -- start start 2024-03-28T22:36:38.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.213+0000 7fcad8a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcad4076110 0x7fcad419e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:38.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.213+0000 7fcad8a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcad4077350 0x7fcad419e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:38.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.213+0000 7fcad8a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcad4077cf0 0x7fcad41a5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:38.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.213+0000 7fcad8a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcad4113a20 con 0x7fcad4077350 2024-03-28T22:36:38.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.213+0000 7fcad8a00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fcad41138a0 con 0x7fcad4077cf0 2024-03-28T22:36:38.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.213+0000 7fcad8a00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fcad4113ba0 con 0x7fcad4076110 2024-03-28T22:36:38.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.214+0000 7fcad1e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcad4076110 0x7fcad419e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:38.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.214+0000 7fcad1e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcad4076110 0x7fcad419e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:53622/0 (socket says 172.21.15.43:53622) 2024-03-28T22:36:38.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.214+0000 7fcad1e00640 1 -- 172.21.15.43:0/1604021624 learned_addr learned my addr 172.21.15.43:0/1604021624 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:38.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.214+0000 7fcac9400640 1 --2- 172.21.15.43:0/1604021624 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcad4077350 0x7fcad419e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:38.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.214+0000 7fcad2800640 1 --2- 172.21.15.43:0/1604021624 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcad4077cf0 0x7fcad41a5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:38.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.214+0000 7fcad1e00640 1 -- 172.21.15.43:0/1604021624 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcad4077cf0 msgr2=0x7fcad41a5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:38.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.214+0000 7fcad1e00640 1 --2- 172.21.15.43:0/1604021624 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcad4077cf0 0x7fcad41a5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:38.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.214+0000 7fcad1e00640 1 -- 172.21.15.43:0/1604021624 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcad4077350 msgr2=0x7fcad419e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:38.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.214+0000 7fcad1e00640 1 --2- 172.21.15.43:0/1604021624 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcad4077350 0x7fcad419e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:38.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.214+0000 7fcad1e00640 1 -- 172.21.15.43:0/1604021624 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcad41a5cb0 con 0x7fcad4076110 2024-03-28T22:36:38.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.214+0000 7fcac9400640 1 --2- 172.21.15.43:0/1604021624 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcad4077350 0x7fcad419e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:36:38.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.214+0000 7fcad1e00640 1 --2- 172.21.15.43:0/1604021624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcad4076110 0x7fcad419e090 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcac4009dc0 tx=0x7fcac4035d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:38.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.215+0000 7fcacb400640 1 -- 172.21.15.43:0/1604021624 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac4039040 con 0x7fcad4076110 2024-03-28T22:36:38.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.215+0000 7fcacb400640 1 -- 172.21.15.43:0/1604021624 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcac4044070 con 0x7fcad4076110 2024-03-28T22:36:38.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.215+0000 7fcacb400640 1 -- 172.21.15.43:0/1604021624 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac4004680 con 0x7fcad4076110 2024-03-28T22:36:38.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.215+0000 7fcad8a00640 1 -- 172.21.15.43:0/1604021624 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcad41a5f40 con 0x7fcad4076110 2024-03-28T22:36:38.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.215+0000 7fcad8a00640 1 -- 172.21.15.43:0/1604021624 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcad41a6350 con 0x7fcad4076110 2024-03-28T22:36:38.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.216+0000 7fcad8a00640 1 -- 172.21.15.43:0/1604021624 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcad4110300 con 0x7fcad4076110 2024-03-28T22:36:38.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.218+0000 7fcacb400640 1 -- 172.21.15.43:0/1604021624 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fcac4003970 con 0x7fcad4076110 2024-03-28T22:36:38.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.219+0000 7fcacb400640 1 --2- 172.21.15.43:0/1604021624 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fcab80a1e00 0x7fcab80a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:38.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.219+0000 7fcacb400640 1 -- 172.21.15.43:0/1604021624 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7fcac40f09a0 con 0x7fcad4076110 2024-03-28T22:36:38.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.220+0000 7fcac9400640 1 --2- 172.21.15.43:0/1604021624 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fcab80a1e00 0x7fcab80a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:38.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.220+0000 7fcac9400640 1 --2- 172.21.15.43:0/1604021624 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fcab80a1e00 0x7fcab80a42c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcabc0058c0 tx=0x7fcabc005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:38.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.228+0000 7fcacb400640 1 -- 172.21.15.43:0/1604021624 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fcac4004820 con 0x7fcad4076110 2024-03-28T22:36:38.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:37 smithi043 ceph-mon[31823]: pgmap v420: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-28T22:36:38.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:37 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2385182462' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:36:38.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:37 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3849751157' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:36:38.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.434+0000 7fcad8a00640 1 -- 172.21.15.43:0/1604021624 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fcad4061b90 con 0x7fcad4076110 2024-03-28T22:36:38.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.435+0000 7fcacb400640 1 -- 172.21.15.43:0/1604021624 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) ==== 104+0+11 (secure 0 0 0) 0x7fcac40bd9e0 con 0x7fcad4076110 2024-03-28T22:36:38.434 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-03-28T22:36:38.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.438+0000 7fcad8a00640 1 -- 172.21.15.43:0/1604021624 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fcab80a1e00 msgr2=0x7fcab80a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:38.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.438+0000 7fcad8a00640 1 --2- 172.21.15.43:0/1604021624 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fcab80a1e00 0x7fcab80a42c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcabc0058c0 tx=0x7fcabc005850 comp rx=0 tx=0).stop 2024-03-28T22:36:38.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.438+0000 7fcad8a00640 1 -- 172.21.15.43:0/1604021624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcad4076110 msgr2=0x7fcad419e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:38.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.438+0000 7fcad8a00640 1 --2- 172.21.15.43:0/1604021624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcad4076110 0x7fcad419e090 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcac4009dc0 tx=0x7fcac4035d90 comp rx=0 tx=0).stop 2024-03-28T22:36:38.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.438+0000 7fcad8a00640 1 -- 172.21.15.43:0/1604021624 shutdown_connections 2024-03-28T22:36:38.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.438+0000 7fcad8a00640 1 --2- 172.21.15.43:0/1604021624 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fcab80a1e00 0x7fcab80a42c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:38.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.438+0000 7fcad8a00640 1 --2- 172.21.15.43:0/1604021624 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcad4077cf0 0x7fcad41a5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:38.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.438+0000 7fcad8a00640 1 --2- 172.21.15.43:0/1604021624 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcad4077350 0x7fcad419e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:38.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.438+0000 7fcad8a00640 1 --2- 172.21.15.43:0/1604021624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcad4076110 0x7fcad419e090 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:38.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.438+0000 7fcad8a00640 1 -- 172.21.15.43:0/1604021624 >> 172.21.15.43:0/1604021624 conn(0x7fcad40fef00 msgr2=0x7fcad4100a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:38.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.438+0000 7fcad8a00640 1 -- 172.21.15.43:0/1604021624 shutdown_connections 2024-03-28T22:36:38.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.438+0000 7fcad8a00640 1 -- 172.21.15.43:0/1604021624 wait complete. 2024-03-28T22:36:38.458 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T22:36:38.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.691+0000 7fb70b800640 1 -- 172.21.15.43:0/3431482379 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb704103990 msgr2=0x7fb704103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:38.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.691+0000 7fb70b800640 1 --2- 172.21.15.43:0/3431482379 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb704103990 0x7fb704103db0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8009860 tx=0x7fb6f802bb70 comp rx=0 tx=0).stop 2024-03-28T22:36:38.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.691+0000 7fb70b800640 1 -- 172.21.15.43:0/3431482379 shutdown_connections 2024-03-28T22:36:38.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.691+0000 7fb70b800640 1 --2- 172.21.15.43:0/3431482379 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb704105540 0x7fb70410bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:38.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.691+0000 7fb70b800640 1 --2- 172.21.15.43:0/3431482379 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb704104b80 0x7fb704105000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:38.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.691+0000 7fb70b800640 1 --2- 172.21.15.43:0/3431482379 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb704103990 0x7fb704103db0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:38.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.691+0000 7fb70b800640 1 -- 172.21.15.43:0/3431482379 >> 172.21.15.43:0/3431482379 conn(0x7fb7040fef00 msgr2=0x7fb704101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:38.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.691+0000 7fb70b800640 1 -- 172.21.15.43:0/3431482379 shutdown_connections 2024-03-28T22:36:38.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.691+0000 7fb70b800640 1 -- 172.21.15.43:0/3431482379 wait complete. 2024-03-28T22:36:38.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.692+0000 7fb70b800640 1 Processor -- start 2024-03-28T22:36:38.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.692+0000 7fb70b800640 1 -- start start 2024-03-28T22:36:38.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.693+0000 7fb70b800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb704103990 0x7fb704199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:38.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.693+0000 7fb70b800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb704104b80 0x7fb70419a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:38.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.693+0000 7fb70b800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb704105540 0x7fb7041a1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:38.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.693+0000 7fb70b800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb70410f5e0 con 0x7fb704105540 2024-03-28T22:36:38.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.693+0000 7fb70b800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb70410f460 con 0x7fb704103990 2024-03-28T22:36:38.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.693+0000 7fb70b800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fb70410f760 con 0x7fb704104b80 2024-03-28T22:36:38.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.693+0000 7fb703e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb704104b80 0x7fb70419a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:38.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.693+0000 7fb708e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb704103990 0x7fb704199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:38.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.693+0000 7fb703e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb704104b80 0x7fb70419a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:53642/0 (socket says 172.21.15.43:53642) 2024-03-28T22:36:38.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.693+0000 7fb708e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb704103990 0x7fb704199d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47176/0 (socket says 172.21.15.43:47176) 2024-03-28T22:36:38.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.693+0000 7fb703e00640 1 -- 172.21.15.43:0/2843393235 learned_addr learned my addr 172.21.15.43:0/2843393235 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:38.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.693+0000 7fb709800640 1 --2- 172.21.15.43:0/2843393235 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb704105540 0x7fb7041a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:38.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.694+0000 7fb708e00640 1 -- 172.21.15.43:0/2843393235 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb704104b80 msgr2=0x7fb70419a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:38.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.694+0000 7fb708e00640 1 --2- 172.21.15.43:0/2843393235 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb704104b80 0x7fb70419a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:38.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.694+0000 7fb708e00640 1 -- 172.21.15.43:0/2843393235 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb704105540 msgr2=0x7fb7041a1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:38.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.694+0000 7fb708e00640 1 --2- 172.21.15.43:0/2843393235 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb704105540 0x7fb7041a1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:38.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.694+0000 7fb708e00640 1 -- 172.21.15.43:0/2843393235 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7041a1910 con 0x7fb704103990 2024-03-28T22:36:38.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.694+0000 7fb709800640 1 --2- 172.21.15.43:0/2843393235 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb704105540 0x7fb7041a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:36:38.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.694+0000 7fb708e00640 1 --2- 172.21.15.43:0/2843393235 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb704103990 0x7fb704199d00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8009780 tx=0x7fb6f8032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:38.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.694+0000 7fb701600640 1 -- 172.21.15.43:0/2843393235 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f8041020 con 0x7fb704103990 2024-03-28T22:36:38.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.694+0000 7fb701600640 1 -- 172.21.15.43:0/2843393235 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6f802c9f0 con 0x7fb704103990 2024-03-28T22:36:38.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.694+0000 7fb70b800640 1 -- 172.21.15.43:0/2843393235 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7041a1ba0 con 0x7fb704103990 2024-03-28T22:36:38.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.694+0000 7fb701600640 1 -- 172.21.15.43:0/2843393235 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f8039040 con 0x7fb704103990 2024-03-28T22:36:38.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.694+0000 7fb70b800640 1 -- 172.21.15.43:0/2843393235 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7041a20b0 con 0x7fb704103990 2024-03-28T22:36:38.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.695+0000 7fb70b800640 1 -- 172.21.15.43:0/2843393235 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb70410ad50 con 0x7fb704103990 2024-03-28T22:36:38.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.697+0000 7fb701600640 1 -- 172.21.15.43:0/2843393235 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fb6f8044020 con 0x7fb704103990 2024-03-28T22:36:38.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.699+0000 7fb701600640 1 --2- 172.21.15.43:0/2843393235 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fb6ec0a1e00 0x7fb6ec0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:38.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.699+0000 7fb701600640 1 -- 172.21.15.43:0/2843393235 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7fb6f80ef9e0 con 0x7fb704103990 2024-03-28T22:36:38.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.699+0000 7fb703e00640 1 --2- 172.21.15.43:0/2843393235 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fb6ec0a1e00 0x7fb6ec0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:38.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.700+0000 7fb703e00640 1 --2- 172.21.15.43:0/2843393235 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fb6ec0a1e00 0x7fb6ec0a42c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0005c80 tx=0x7fb6f0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:38.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.703+0000 7fb701600640 1 -- 172.21.15.43:0/2843393235 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb6f804b020 con 0x7fb704103990 2024-03-28T22:36:38.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.908+0000 7fb70b800640 1 -- 172.21.15.43:0/2843393235 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb704062380 con 0x7fb704103990 2024-03-28T22:36:38.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.909+0000 7fb701600640 1 -- 172.21.15.43:0/2843393235 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) ==== 100+0+11 (secure 0 0 0) 0x7fb6f80bca20 con 0x7fb704103990 2024-03-28T22:36:38.908 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-03-28T22:36:38.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.912+0000 7fb70b800640 1 -- 172.21.15.43:0/2843393235 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fb6ec0a1e00 msgr2=0x7fb6ec0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:38.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.912+0000 7fb70b800640 1 --2- 172.21.15.43:0/2843393235 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fb6ec0a1e00 0x7fb6ec0a42c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0005c80 tx=0x7fb6f0009260 comp rx=0 tx=0).stop 2024-03-28T22:36:38.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.912+0000 7fb70b800640 1 -- 172.21.15.43:0/2843393235 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb704103990 msgr2=0x7fb704199d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:38.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.912+0000 7fb70b800640 1 --2- 172.21.15.43:0/2843393235 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb704103990 0x7fb704199d00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8009780 tx=0x7fb6f8032430 comp rx=0 tx=0).stop 2024-03-28T22:36:38.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.912+0000 7fb70b800640 1 -- 172.21.15.43:0/2843393235 shutdown_connections 2024-03-28T22:36:38.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.912+0000 7fb70b800640 1 --2- 172.21.15.43:0/2843393235 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb704105540 0x7fb7041a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:38.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.912+0000 7fb70b800640 1 --2- 172.21.15.43:0/2843393235 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb704104b80 0x7fb70419a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:38.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.912+0000 7fb70b800640 1 --2- 172.21.15.43:0/2843393235 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fb6ec0a1e00 0x7fb6ec0a42c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:38.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.912+0000 7fb70b800640 1 --2- 172.21.15.43:0/2843393235 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb704103990 0x7fb704199d00 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:38.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.912+0000 7fb70b800640 1 -- 172.21.15.43:0/2843393235 >> 172.21.15.43:0/2843393235 conn(0x7fb7040fef00 msgr2=0x7fb704107f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:38.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.912+0000 7fb70b800640 1 -- 172.21.15.43:0/2843393235 shutdown_connections 2024-03-28T22:36:38.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:38.912+0000 7fb70b800640 1 -- 172.21.15.43:0/2843393235 wait complete. 2024-03-28T22:36:38.932 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T22:36:39.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:38 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3503126479' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:36:39.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:38 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1604021624' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:36:39.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.163+0000 7fe451200640 1 -- 172.21.15.43:0/2951739771 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe44c109820 msgr2=0x7fe44c1100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:39.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.163+0000 7fe451200640 1 --2- 172.21.15.43:0/2951739771 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe44c109820 0x7fe44c1100b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe444009ca0 tx=0x7fe44402c100 comp rx=0 tx=0).stop 2024-03-28T22:36:39.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.163+0000 7fe451200640 1 -- 172.21.15.43:0/2951739771 shutdown_connections 2024-03-28T22:36:39.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.163+0000 7fe451200640 1 --2- 172.21.15.43:0/2951739771 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe44c109820 0x7fe44c1100b0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:39.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.163+0000 7fe451200640 1 --2- 172.21.15.43:0/2951739771 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe44c108e60 0x7fe44c1092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:39.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.163+0000 7fe451200640 1 --2- 172.21.15.43:0/2951739771 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe44c107c70 0x7fe44c108090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:39.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.163+0000 7fe451200640 1 -- 172.21.15.43:0/2951739771 >> 172.21.15.43:0/2951739771 conn(0x7fe44c076c20 msgr2=0x7fe44c079040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:39.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.164+0000 7fe451200640 1 -- 172.21.15.43:0/2951739771 shutdown_connections 2024-03-28T22:36:39.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.164+0000 7fe451200640 1 -- 172.21.15.43:0/2951739771 wait complete. 2024-03-28T22:36:39.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.165+0000 7fe451200640 1 Processor -- start 2024-03-28T22:36:39.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.165+0000 7fe451200640 1 -- start start 2024-03-28T22:36:39.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.165+0000 7fe451200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe44c107c70 0x7fe44c19df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:39.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.165+0000 7fe451200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe44c108e60 0x7fe44c19e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:39.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.165+0000 7fe451200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe44c109820 0x7fe44c1a54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:39.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.165+0000 7fe451200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe44c1138f0 con 0x7fe44c107c70 2024-03-28T22:36:39.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.165+0000 7fe451200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe44c113770 con 0x7fe44c109820 2024-03-28T22:36:39.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.165+0000 7fe451200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe44c113a70 con 0x7fe44c108e60 2024-03-28T22:36:39.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.165+0000 7fe44b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe44c107c70 0x7fe44c19df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:39.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.166+0000 7fe443e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe44c108e60 0x7fe44c19e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:39.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.166+0000 7fe44be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe44c109820 0x7fe44c1a54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:39.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.166+0000 7fe44b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe44c107c70 0x7fe44c19df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48312/0 (socket says 172.21.15.43:48312) 2024-03-28T22:36:39.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.166+0000 7fe44b400640 1 -- 172.21.15.43:0/2925770062 learned_addr learned my addr 172.21.15.43:0/2925770062 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:39.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.166+0000 7fe443e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe44c108e60 0x7fe44c19e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:53658/0 (socket says 172.21.15.43:53658) 2024-03-28T22:36:39.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.166+0000 7fe44be00640 1 -- 172.21.15.43:0/2925770062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe44c108e60 msgr2=0x7fe44c19e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:39.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.166+0000 7fe44be00640 1 --2- 172.21.15.43:0/2925770062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe44c108e60 0x7fe44c19e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:39.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.166+0000 7fe44be00640 1 -- 172.21.15.43:0/2925770062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe44c107c70 msgr2=0x7fe44c19df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:39.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.166+0000 7fe44be00640 1 --2- 172.21.15.43:0/2925770062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe44c107c70 0x7fe44c19df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:39.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.166+0000 7fe44be00640 1 -- 172.21.15.43:0/2925770062 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe44c1a5b20 con 0x7fe44c109820 2024-03-28T22:36:39.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.166+0000 7fe44be00640 1 --2- 172.21.15.43:0/2925770062 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe44c109820 0x7fe44c1a54b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe444037040 tx=0x7fe444035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:39.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.167+0000 7fe448c00640 1 -- 172.21.15.43:0/2925770062 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe444039040 con 0x7fe44c109820 2024-03-28T22:36:39.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.167+0000 7fe448c00640 1 -- 172.21.15.43:0/2925770062 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe444044070 con 0x7fe44c109820 2024-03-28T22:36:39.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.167+0000 7fe451200640 1 -- 172.21.15.43:0/2925770062 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe44c1a5db0 con 0x7fe44c109820 2024-03-28T22:36:39.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.167+0000 7fe448c00640 1 -- 172.21.15.43:0/2925770062 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe444004450 con 0x7fe44c109820 2024-03-28T22:36:39.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.167+0000 7fe451200640 1 -- 172.21.15.43:0/2925770062 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe44c1a6210 con 0x7fe44c109820 2024-03-28T22:36:39.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.168+0000 7fe451200640 1 -- 172.21.15.43:0/2925770062 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe44c10eec0 con 0x7fe44c109820 2024-03-28T22:36:39.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.170+0000 7fe448c00640 1 -- 172.21.15.43:0/2925770062 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fe4440045f0 con 0x7fe44c109820 2024-03-28T22:36:39.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.171+0000 7fe448c00640 1 --2- 172.21.15.43:0/2925770062 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe4300a1d80 0x7fe4300a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:39.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.171+0000 7fe448c00640 1 -- 172.21.15.43:0/2925770062 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7fe4440f07a0 con 0x7fe44c109820 2024-03-28T22:36:39.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.171+0000 7fe44b400640 1 --2- 172.21.15.43:0/2925770062 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe4300a1d80 0x7fe4300a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:39.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.172+0000 7fe44b400640 1 --2- 172.21.15.43:0/2925770062 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe4300a1d80 0x7fe4300a4240 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe43c0058c0 tx=0x7fe43c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:39.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.175+0000 7fe448c00640 1 -- 172.21.15.43:0/2925770062 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe4440bd7e0 con 0x7fe44c109820 2024-03-28T22:36:39.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:38 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3503126479' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:36:39.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:38 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1604021624' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:36:39.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:38 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3503126479' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:36:39.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:38 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1604021624' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:36:39.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.374+0000 7fe451200640 1 -- 172.21.15.43:0/2925770062 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fe44c062350 con 0x7fe44c109820 2024-03-28T22:36:39.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.375+0000 7fe448c00640 1 -- 172.21.15.43:0/2925770062 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v96) ==== 103+0+11 (secure 0 0 0) 0x7fe4440098b0 con 0x7fe44c109820 2024-03-28T22:36:39.374 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-03-28T22:36:39.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.378+0000 7fe451200640 1 -- 172.21.15.43:0/2925770062 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe4300a1d80 msgr2=0x7fe4300a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:39.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.378+0000 7fe451200640 1 --2- 172.21.15.43:0/2925770062 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe4300a1d80 0x7fe4300a4240 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe43c0058c0 tx=0x7fe43c005f90 comp rx=0 tx=0).stop 2024-03-28T22:36:39.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.378+0000 7fe451200640 1 -- 172.21.15.43:0/2925770062 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe44c109820 msgr2=0x7fe44c1a54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:39.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.378+0000 7fe451200640 1 --2- 172.21.15.43:0/2925770062 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe44c109820 0x7fe44c1a54b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe444037040 tx=0x7fe444035d10 comp rx=0 tx=0).stop 2024-03-28T22:36:39.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.378+0000 7fe451200640 1 -- 172.21.15.43:0/2925770062 shutdown_connections 2024-03-28T22:36:39.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.378+0000 7fe451200640 1 --2- 172.21.15.43:0/2925770062 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fe4300a1d80 0x7fe4300a4240 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:39.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.378+0000 7fe451200640 1 --2- 172.21.15.43:0/2925770062 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe44c109820 0x7fe44c1a54b0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:39.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.378+0000 7fe451200640 1 --2- 172.21.15.43:0/2925770062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe44c108e60 0x7fe44c19e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:39.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.378+0000 7fe451200640 1 --2- 172.21.15.43:0/2925770062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe44c107c70 0x7fe44c19df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:39.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.378+0000 7fe451200640 1 -- 172.21.15.43:0/2925770062 >> 172.21.15.43:0/2925770062 conn(0x7fe44c076c20 msgr2=0x7fe44c10c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:39.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.378+0000 7fe451200640 1 -- 172.21.15.43:0/2925770062 shutdown_connections 2024-03-28T22:36:39.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.378+0000 7fe451200640 1 -- 172.21.15.43:0/2925770062 wait complete. 2024-03-28T22:36:39.399 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:36:39.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.624+0000 7f42c4a00640 1 -- 172.21.15.43:0/3466434755 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f42c0104a10 msgr2=0x7f42c0106e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:39.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.624+0000 7f42c4a00640 1 --2- 172.21.15.43:0/3466434755 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f42c0104a10 0x7f42c0106e00 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f42a8009ca0 tx=0x7f42a802c100 comp rx=0 tx=0).stop 2024-03-28T22:36:39.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.624+0000 7f42c4a00640 1 -- 172.21.15.43:0/3466434755 shutdown_connections 2024-03-28T22:36:39.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.624+0000 7f42c4a00640 1 --2- 172.21.15.43:0/3466434755 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f42c0104a10 0x7f42c0106e00 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:39.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.624+0000 7f42c4a00640 1 --2- 172.21.15.43:0/3466434755 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42c0107340 0x7f42c0109730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:39.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.624+0000 7f42c4a00640 1 --2- 172.21.15.43:0/3466434755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42c01020e0 0x7f42c01044d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:39.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.624+0000 7f42c4a00640 1 -- 172.21.15.43:0/3466434755 >> 172.21.15.43:0/3466434755 conn(0x7f42c00fba30 msgr2=0x7f42c00fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:39.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.624+0000 7f42c4a00640 1 -- 172.21.15.43:0/3466434755 shutdown_connections 2024-03-28T22:36:39.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.624+0000 7f42c4a00640 1 -- 172.21.15.43:0/3466434755 wait complete. 2024-03-28T22:36:39.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.626+0000 7f42c4a00640 1 Processor -- start 2024-03-28T22:36:39.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.626+0000 7f42c4a00640 1 -- start start 2024-03-28T22:36:39.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.626+0000 7f42c4a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42c01020e0 0x7f42c0199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:39.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.626+0000 7f42c4a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f42c0104a10 0x7f42c0199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:39.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.626+0000 7f42c4a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42c0107340 0x7f42c01a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:39.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.626+0000 7f42c4a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f42c010d240 con 0x7f42c01020e0 2024-03-28T22:36:39.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.626+0000 7f42c4a00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f42c010d0c0 con 0x7f42c0104a10 2024-03-28T22:36:39.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.626+0000 7f42c4a00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f42c010d3c0 con 0x7f42c0107340 2024-03-28T22:36:39.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.627+0000 7f42bde00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f42c0104a10 0x7f42c0199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:39.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.627+0000 7f42bde00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f42c0104a10 0x7f42c0199ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47218/0 (socket says 172.21.15.43:47218) 2024-03-28T22:36:39.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.627+0000 7f42bde00640 1 -- 172.21.15.43:0/3930219158 learned_addr learned my addr 172.21.15.43:0/3930219158 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:39.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.627+0000 7f42bf200640 1 --2- 172.21.15.43:0/3930219158 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42c0107340 0x7f42c01a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:39.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.627+0000 7f42be800640 1 --2- 172.21.15.43:0/3930219158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42c01020e0 0x7f42c0199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:39.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.627+0000 7f42bde00640 1 -- 172.21.15.43:0/3930219158 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42c0107340 msgr2=0x7f42c01a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:39.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.627+0000 7f42bde00640 1 --2- 172.21.15.43:0/3930219158 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42c0107340 0x7f42c01a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:39.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.627+0000 7f42bde00640 1 -- 172.21.15.43:0/3930219158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42c01020e0 msgr2=0x7f42c0199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:39.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.627+0000 7f42bde00640 1 --2- 172.21.15.43:0/3930219158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42c01020e0 0x7f42c0199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:39.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.627+0000 7f42bde00640 1 -- 172.21.15.43:0/3930219158 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42c01a16d0 con 0x7f42c0104a10 2024-03-28T22:36:39.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.627+0000 7f42bde00640 1 --2- 172.21.15.43:0/3930219158 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f42c0104a10 0x7f42c0199ff0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f42a80324c0 tx=0x7f42a8035d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:39.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.628+0000 7f42b7400640 1 -- 172.21.15.43:0/3930219158 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a8039040 con 0x7f42c0104a10 2024-03-28T22:36:39.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.628+0000 7f42b7400640 1 -- 172.21.15.43:0/3930219158 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42a8044070 con 0x7f42c0104a10 2024-03-28T22:36:39.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.628+0000 7f42c4a00640 1 -- 172.21.15.43:0/3930219158 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42c01a1960 con 0x7f42c0104a10 2024-03-28T22:36:39.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.628+0000 7f42b7400640 1 -- 172.21.15.43:0/3930219158 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a80044a0 con 0x7f42c0104a10 2024-03-28T22:36:39.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.628+0000 7f42c4a00640 1 -- 172.21.15.43:0/3930219158 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42c01a1ea0 con 0x7f42c0104a10 2024-03-28T22:36:39.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.629+0000 7f42c4a00640 1 -- 172.21.15.43:0/3930219158 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42c0108e30 con 0x7f42c0104a10 2024-03-28T22:36:39.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.631+0000 7f42b7400640 1 -- 172.21.15.43:0/3930219158 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f42a8004640 con 0x7f42c0104a10 2024-03-28T22:36:39.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.632+0000 7f42b7400640 1 --2- 172.21.15.43:0/3930219158 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f42a00a1e00 0x7f42a00a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:39.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.632+0000 7f42b7400640 1 -- 172.21.15.43:0/3930219158 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f42a80f0790 con 0x7f42c0104a10 2024-03-28T22:36:39.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.632+0000 7f42be800640 1 --2- 172.21.15.43:0/3930219158 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f42a00a1e00 0x7f42a00a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:39.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.633+0000 7f42be800640 1 --2- 172.21.15.43:0/3930219158 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f42a00a1e00 0x7f42a00a42c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f42b0005d00 tx=0x7f42b000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:39.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.640+0000 7f42b7400640 1 -- 172.21.15.43:0/3930219158 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f42a80bd7d0 con 0x7f42c0104a10 2024-03-28T22:36:39.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.891+0000 7f42c4a00640 1 -- 172.21.15.43:0/3930219158 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42c0062350 con 0x7f42c0104a10 2024-03-28T22:36:39.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.892+0000 7f42b7400640 1 -- 172.21.15.43:0/3930219158 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11934 (secure 0 0 0) 0x7f42a80bd0d0 con 0x7f42c0104a10 2024-03-28T22:36:39.892 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:36:39.892 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:33:28.906396+0000","modified":"2024-03-28T22:34:06.174761+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24572,"mds_1":24556,"mds_2":24535,"mds_3":24523,"mds_4":24518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/1084988072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1084988072},{"type":"v1","addr":"172.21.15.82:6835","nonce":1084988072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24535,24556,24518,24523,24572]},"id":1}]} 2024-03-28T22:36:39.892 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 28 2024-03-28T22:36:39.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.896+0000 7f42c4a00640 1 -- 172.21.15.43:0/3930219158 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f42a00a1e00 msgr2=0x7f42a00a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:39.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.896+0000 7f42c4a00640 1 --2- 172.21.15.43:0/3930219158 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f42a00a1e00 0x7f42a00a42c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f42b0005d00 tx=0x7f42b000b410 comp rx=0 tx=0).stop 2024-03-28T22:36:39.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.896+0000 7f42c4a00640 1 -- 172.21.15.43:0/3930219158 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f42c0104a10 msgr2=0x7f42c0199ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:39.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.896+0000 7f42c4a00640 1 --2- 172.21.15.43:0/3930219158 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f42c0104a10 0x7f42c0199ff0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f42a80324c0 tx=0x7f42a8035d90 comp rx=0 tx=0).stop 2024-03-28T22:36:39.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.896+0000 7f42c4a00640 1 -- 172.21.15.43:0/3930219158 shutdown_connections 2024-03-28T22:36:39.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.896+0000 7f42c4a00640 1 --2- 172.21.15.43:0/3930219158 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f42a00a1e00 0x7f42a00a42c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:39.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.896+0000 7f42c4a00640 1 --2- 172.21.15.43:0/3930219158 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f42c0104a10 0x7f42c0199ff0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:39.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.896+0000 7f42c4a00640 1 --2- 172.21.15.43:0/3930219158 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f42c0107340 0x7f42c01a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:39.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.896+0000 7f42c4a00640 1 --2- 172.21.15.43:0/3930219158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42c01020e0 0x7f42c0199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:39.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.896+0000 7f42c4a00640 1 -- 172.21.15.43:0/3930219158 >> 172.21.15.43:0/3930219158 conn(0x7f42c00fba30 msgr2=0x7f42c0101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:39.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.896+0000 7f42c4a00640 1 -- 172.21.15.43:0/3930219158 shutdown_connections 2024-03-28T22:36:39.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:39.896+0000 7f42c4a00640 1 -- 172.21.15.43:0/3930219158 wait complete. 2024-03-28T22:36:39.915 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:36:40.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.129+0000 7f5173000640 1 -- 172.21.15.43:0/2393334875 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f516c077cf0 msgr2=0x7f516c078590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:40.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.129+0000 7f5173000640 1 --2- 172.21.15.43:0/2393334875 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f516c077cf0 0x7f516c078590 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5164009be0 tx=0x7f516402bef0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.130+0000 7f5173000640 1 -- 172.21.15.43:0/2393334875 shutdown_connections 2024-03-28T22:36:40.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.130+0000 7f5173000640 1 --2- 172.21.15.43:0/2393334875 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f516c077cf0 0x7f516c078590 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.130+0000 7f5173000640 1 --2- 172.21.15.43:0/2393334875 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f516c077350 0x7f516c0777b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.130+0000 7f5173000640 1 --2- 172.21.15.43:0/2393334875 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f516c076110 0x7f516c076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.130+0000 7f5173000640 1 -- 172.21.15.43:0/2393334875 >> 172.21.15.43:0/2393334875 conn(0x7f516c0fef00 msgr2=0x7f516c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:40.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.130+0000 7f5173000640 1 -- 172.21.15.43:0/2393334875 shutdown_connections 2024-03-28T22:36:40.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.130+0000 7f5173000640 1 -- 172.21.15.43:0/2393334875 wait complete. 2024-03-28T22:36:40.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.131+0000 7f5173000640 1 Processor -- start 2024-03-28T22:36:40.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.131+0000 7f5173000640 1 -- start start 2024-03-28T22:36:40.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.131+0000 7f5173000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f516c076110 0x7f516c19dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.132+0000 7f5173000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f516c077350 0x7f516c19e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.132+0000 7f5173000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f516c077cf0 0x7f516c1a5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.132+0000 7f5173000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f516c113a60 con 0x7f516c076110 2024-03-28T22:36:40.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.132+0000 7f5173000640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f516c1138e0 con 0x7f516c077350 2024-03-28T22:36:40.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.132+0000 7f5173000640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f516c113be0 con 0x7f516c077cf0 2024-03-28T22:36:40.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.132+0000 7f5163e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f516c077350 0x7f516c19e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.132+0000 7f516be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f516c076110 0x7f516c19dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.132+0000 7f5171000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f516c077cf0 0x7f516c1a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.132+0000 7f5163e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f516c077350 0x7f516c19e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47232/0 (socket says 172.21.15.43:47232) 2024-03-28T22:36:40.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.132+0000 7f5163e00640 1 -- 172.21.15.43:0/71567114 learned_addr learned my addr 172.21.15.43:0/71567114 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:40.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.132+0000 7f5163e00640 1 -- 172.21.15.43:0/71567114 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f516c077cf0 msgr2=0x7f516c1a5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:40.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.132+0000 7f5163e00640 1 --2- 172.21.15.43:0/71567114 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f516c077cf0 0x7f516c1a5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.132+0000 7f5163e00640 1 -- 172.21.15.43:0/71567114 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f516c076110 msgr2=0x7f516c19dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:40.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.132+0000 7f5163e00640 1 --2- 172.21.15.43:0/71567114 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f516c076110 0x7f516c19dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.132+0000 7f5163e00640 1 -- 172.21.15.43:0/71567114 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f516c1a5c00 con 0x7f516c077350 2024-03-28T22:36:40.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.133+0000 7f5163e00640 1 --2- 172.21.15.43:0/71567114 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f516c077350 0x7f516c19e520 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f515400d950 tx=0x7f515400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:40.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.133+0000 7f5169600640 1 -- 172.21.15.43:0/71567114 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5154017070 con 0x7f516c077350 2024-03-28T22:36:40.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.133+0000 7f5169600640 1 -- 172.21.15.43:0/71567114 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5154009c30 con 0x7f516c077350 2024-03-28T22:36:40.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.133+0000 7f5173000640 1 -- 172.21.15.43:0/71567114 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f516c1a5ef0 con 0x7f516c077350 2024-03-28T22:36:40.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.133+0000 7f5169600640 1 -- 172.21.15.43:0/71567114 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f515400f6b0 con 0x7f516c077350 2024-03-28T22:36:40.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.133+0000 7f5173000640 1 -- 172.21.15.43:0/71567114 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f516c1a6400 con 0x7f516c077350 2024-03-28T22:36:40.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.134+0000 7f5173000640 1 -- 172.21.15.43:0/71567114 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f516c110410 con 0x7f516c077350 2024-03-28T22:36:40.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.136+0000 7f5169600640 1 -- 172.21.15.43:0/71567114 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f5154010460 con 0x7f516c077350 2024-03-28T22:36:40.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.137+0000 7f5169600640 1 --2- 172.21.15.43:0/71567114 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f51500a1e00 0x7f51500a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.137+0000 7f5169600640 1 -- 172.21.15.43:0/71567114 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f51540cdcb0 con 0x7f516c077350 2024-03-28T22:36:40.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.137+0000 7f516be00640 1 --2- 172.21.15.43:0/71567114 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f51500a1e00 0x7f51500a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.138+0000 7f516be00640 1 --2- 172.21.15.43:0/71567114 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f51500a1e00 0x7f51500a42c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f515c009530 tx=0x7f515c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:40.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.141+0000 7f5169600640 1 -- 172.21.15.43:0/71567114 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5154009de0 con 0x7f516c077350 2024-03-28T22:36:40.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:39 smithi172 ceph-mon[36566]: pgmap v421: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-28T22:36:40.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:39 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2843393235' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:36:40.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:39 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2925770062' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:36:40.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:39 smithi082 ceph-mon[36471]: pgmap v421: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-28T22:36:40.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:39 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2843393235' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:36:40.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:39 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2925770062' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:36:40.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:39 smithi043 ceph-mon[31823]: pgmap v421: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-28T22:36:40.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:39 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2843393235' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:36:40.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:39 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2925770062' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:36:40.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.346+0000 7f5173000640 1 -- 172.21.15.43:0/71567114 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f516c0623f0 con 0x7f516c077350 2024-03-28T22:36:40.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.348+0000 7f5169600640 1 -- 172.21.15.43:0/71567114 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+24712 (secure 0 0 0) 0x7f515409a6a0 con 0x7f516c077350 2024-03-28T22:36:40.348 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:36:40.348 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":96,"fsid":"c2276428-ed51-11ee-95d0-87774f69a715","created":"2024-03-28T22:24:19.950084+0000","modified":"2024-03-28T22:35:30.237654+0000","last_up_change":"2024-03-28T22:32:31.078737+0000","last_in_change":"2024-03-28T22:32:10.147097+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:28:14.713290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:33:27.167435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:33:28.411825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T22:33:32.523122+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b855adcd-786c-4e42-b8fe-a515a77ce979","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6803","nonce":2826124409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6805","nonce":2826124409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6809","nonce":2826124409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6807","nonce":2826124409}]},"public_addr":"172.21.15.43:6803/2826124409","cluster_addr":"172.21.15.43:6805/2826124409","heartbeat_back_addr":"172.21.15.43:6809/2826124409","heartbeat_front_addr":"172.21.15.43:6807/2826124409","state":["exists","up"]},{"osd":1,"uuid":"031c143d-8ad1-4b68-8db9-3c717137e3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6801","nonce":3043263644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6803","nonce":3043263644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6807","nonce":3043263644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6805","nonce":3043263644}]},"public_addr":"172.21.15.82:6801/3043263644","cluster_addr":"172.21.15.82:6803/3043263644","heartbeat_back_addr":"172.21.15.82:6807/3043263644","heartbeat_front_addr":"172.21.15.82:6805/3043263644","state":["exists","up"]},{"osd":2,"uuid":"20632bbb-f07a-4ca1-b0c4-ff9d508e211c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6801","nonce":4213953645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6803","nonce":4213953645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6807","nonce":4213953645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6805","nonce":4213953645}]},"public_addr":"172.21.15.172:6801/4213953645","cluster_addr":"172.21.15.172:6803/4213953645","heartbeat_back_addr":"172.21.15.172:6807/4213953645","heartbeat_front_addr":"172.21.15.172:6805/4213953645","state":["exists","up"]},{"osd":3,"uuid":"9a1f2312-4dbd-475b-8736-d5ebc8788e09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6811","nonce":1597280920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6813","nonce":1597280920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6817","nonce":1597280920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6815","nonce":1597280920}]},"public_addr":"172.21.15.43:6811/1597280920","cluster_addr":"172.21.15.43:6813/1597280920","heartbeat_back_addr":"172.21.15.43:6817/1597280920","heartbeat_front_addr":"172.21.15.43:6815/1597280920","state":["exists","up"]},{"osd":4,"uuid":"29140ead-7075-42b2-90c1-27cae0b29ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6809","nonce":215451895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6811","nonce":215451895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6815","nonce":215451895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6813","nonce":215451895}]},"public_addr":"172.21.15.82:6809/215451895","cluster_addr":"172.21.15.82:6811/215451895","heartbeat_back_addr":"172.21.15.82:6815/215451895","heartbeat_front_addr":"172.21.15.82:6813/215451895","state":["exists","up"]},{"osd":5,"uuid":"59644c61-93dc-4536-b2ad-0256b24e5cb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6809","nonce":1576148933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6811","nonce":1576148933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6815","nonce":1576148933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6813","nonce":1576148933}]},"public_addr":"172.21.15.172:6809/1576148933","cluster_addr":"172.21.15.172:6811/1576148933","heartbeat_back_addr":"172.21.15.172:6815/1576148933","heartbeat_front_addr":"172.21.15.172:6813/1576148933","state":["exists","up"]},{"osd":6,"uuid":"22d54e57-c3f1-4ecf-9921-8e815b48f92a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6819","nonce":791044279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6821","nonce":791044279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6825","nonce":791044279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6823","nonce":791044279}]},"public_addr":"172.21.15.43:6819/791044279","cluster_addr":"172.21.15.43:6821/791044279","heartbeat_back_addr":"172.21.15.43:6825/791044279","heartbeat_front_addr":"172.21.15.43:6823/791044279","state":["exists","up"]},{"osd":7,"uuid":"248cd1e2-0d6b-452f-a0a8-c51fe1b975ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6817","nonce":2157777323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6819","nonce":2157777323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6823","nonce":2157777323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6821","nonce":2157777323}]},"public_addr":"172.21.15.82:6817/2157777323","cluster_addr":"172.21.15.82:6819/2157777323","heartbeat_back_addr":"172.21.15.82:6823/2157777323","heartbeat_front_addr":"172.21.15.82:6821/2157777323","state":["exists","up"]},{"osd":8,"uuid":"d98aab6e-8bae-4c24-906c-4f492f047b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6817","nonce":1095328248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6819","nonce":1095328248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6823","nonce":1095328248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6821","nonce":1095328248}]},"public_addr":"172.21.15.172:6817/1095328248","cluster_addr":"172.21.15.172:6819/1095328248","heartbeat_back_addr":"172.21.15.172:6823/1095328248","heartbeat_front_addr":"172.21.15.172:6821/1095328248","state":["exists","up"]},{"osd":9,"uuid":"19aa383e-a0f6-45ea-96cc-e7956535caa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6827","nonce":3657791498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6829","nonce":3657791498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6833","nonce":3657791498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6831","nonce":3657791498}]},"public_addr":"172.21.15.43:6827/3657791498","cluster_addr":"172.21.15.43:6829/3657791498","heartbeat_back_addr":"172.21.15.43:6833/3657791498","heartbeat_front_addr":"172.21.15.43:6831/3657791498","state":["exists","up"]},{"osd":10,"uuid":"666977e5-eaf7-41b9-a18f-c380e149345d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6825","nonce":3454954047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6827","nonce":3454954047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6831","nonce":3454954047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6829","nonce":3454954047}]},"public_addr":"172.21.15.82:6825/3454954047","cluster_addr":"172.21.15.82:6827/3454954047","heartbeat_back_addr":"172.21.15.82:6831/3454954047","heartbeat_front_addr":"172.21.15.82:6829/3454954047","state":["exists","up"]},{"osd":11,"uuid":"615cf3be-8076-470e-ad09-c880e7593717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6825","nonce":2563517144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6827","nonce":2563517144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6831","nonce":2563517144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6829","nonce":2563517144}]},"public_addr":"172.21.15.172:6825/2563517144","cluster_addr":"172.21.15.172:6827/2563517144","heartbeat_back_addr":"172.21.15.172:6831/2563517144","heartbeat_front_addr":"172.21.15.172:6829/2563517144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:21.727405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:47.100259+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:10.615438+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:43.632144+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:09.487940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:34.604463+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:09.594452+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:35.709795+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:01.184202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:35.455322+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:02.413166+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:28.550754+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.9","mappings":[{"from":5,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/578791143":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/4029313215":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/1574178482":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6801/3517712438":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6800/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/1375099638":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/2710117652":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6801/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/3215634168":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6800/3517712438":"2024-03-29T22:25:28.008215+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:36:40.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.351+0000 7f5173000640 1 -- 172.21.15.43:0/71567114 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f51500a1e00 msgr2=0x7f51500a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:40.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.351+0000 7f5173000640 1 --2- 172.21.15.43:0/71567114 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f51500a1e00 0x7f51500a42c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f515c009530 tx=0x7f515c009260 comp rx=0 tx=0).stop 2024-03-28T22:36:40.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.351+0000 7f5173000640 1 -- 172.21.15.43:0/71567114 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f516c077350 msgr2=0x7f516c19e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:40.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.351+0000 7f5173000640 1 --2- 172.21.15.43:0/71567114 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f516c077350 0x7f516c19e520 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f515400d950 tx=0x7f515400de10 comp rx=0 tx=0).stop 2024-03-28T22:36:40.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.351+0000 7f5173000640 1 -- 172.21.15.43:0/71567114 shutdown_connections 2024-03-28T22:36:40.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.351+0000 7f5173000640 1 --2- 172.21.15.43:0/71567114 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f51500a1e00 0x7f51500a42c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.351+0000 7f5173000640 1 --2- 172.21.15.43:0/71567114 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f516c077350 0x7f516c19e520 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.351+0000 7f5173000640 1 --2- 172.21.15.43:0/71567114 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f516c077cf0 0x7f516c1a5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.351+0000 7f5173000640 1 --2- 172.21.15.43:0/71567114 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f516c076110 0x7f516c19dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.351+0000 7f5173000640 1 -- 172.21.15.43:0/71567114 >> 172.21.15.43:0/71567114 conn(0x7f516c0fef00 msgr2=0x7f516c100580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:40.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.352+0000 7f5173000640 1 -- 172.21.15.43:0/71567114 shutdown_connections 2024-03-28T22:36:40.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.352+0000 7f5173000640 1 -- 172.21.15.43:0/71567114 wait complete. 2024-03-28T22:36:40.371 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-28T22:33:28.906396+0000', 'modified': '2024-03-28T22:34:06.174761+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24572, 'mds_1': 24556, 'mds_2': 24535, 'mds_3': 24523, 'mds_4': 24518}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24443': {'gid': 24443, 'name': 'k', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.82:6833/3363736995', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6832', 'nonce': 3363736995}, {'type': 'v1', 'addr': '172.21.15.82:6833', 'nonce': 3363736995}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24457': {'gid': 24457, 'name': 'l', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.172:6833/2579392697', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6832', 'nonce': 2579392697}, {'type': 'v1', 'addr': '172.21.15.172:6833', 'nonce': 2579392697}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24461': {'gid': 24461, 'name': 'j', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.43:6835/4195313768', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6834', 'nonce': 4195313768}, {'type': 'v1', 'addr': '172.21.15.43:6835', 'nonce': 4195313768}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24479': {'gid': 24479, 'name': 'd', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.43:6837/1330639541', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6836', 'nonce': 1330639541}, {'type': 'v1', 'addr': '172.21.15.43:6837', 'nonce': 1330639541}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24491': {'gid': 24491, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.82:6835/1084988072', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6834', 'nonce': 1084988072}, {'type': 'v1', 'addr': '172.21.15.82:6835', 'nonce': 1084988072}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24518': {'gid': 24518, 'name': 'h', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.82:6837/757432882', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6836', 'nonce': 757432882}, {'type': 'v1', 'addr': '172.21.15.82:6837', 'nonce': 757432882}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24523': {'gid': 24523, 'name': 'f', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.172:6835/318854103', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6834', 'nonce': 318854103}, {'type': 'v1', 'addr': '172.21.15.172:6835', 'nonce': 318854103}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24535': {'gid': 24535, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.172:6837/3745415724', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6836', 'nonce': 3745415724}, {'type': 'v1', 'addr': '172.21.15.172:6837', 'nonce': 3745415724}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24556': {'gid': 24556, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.172:6839/3592465067', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6838', 'nonce': 3592465067}, {'type': 'v1', 'addr': '172.21.15.172:6839', 'nonce': 3592465067}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24572': {'gid': 24572, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.82:6839/2220892357', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6838', 'nonce': 2220892357}, {'type': 'v1', 'addr': '172.21.15.82:6839', 'nonce': 2220892357}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24572, 'qdb_cluster': [24535, 24556, 24518, 24523, 24572]} 2024-03-28T22:36:40.372 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-28T22:36:40.372 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-28T22:36:40.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.605+0000 7f6b39400640 1 -- 172.21.15.43:0/4063768645 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b34105400 msgr2=0x7f6b3410bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:40.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.605+0000 7f6b39400640 1 --2- 172.21.15.43:0/4063768645 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b34105400 0x7f6b3410bc90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c009900 tx=0x7f6b2c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:36:40.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.605+0000 7f6b39400640 1 -- 172.21.15.43:0/4063768645 shutdown_connections 2024-03-28T22:36:40.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.605+0000 7f6b39400640 1 --2- 172.21.15.43:0/4063768645 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b34104a40 0x7f6b34104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.605+0000 7f6b39400640 1 --2- 172.21.15.43:0/4063768645 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b34105400 0x7f6b3410bc90 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.605+0000 7f6b39400640 1 --2- 172.21.15.43:0/4063768645 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b34103850 0x7f6b34103c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.605+0000 7f6b39400640 1 -- 172.21.15.43:0/4063768645 >> 172.21.15.43:0/4063768645 conn(0x7f6b340fedc0 msgr2=0x7f6b34101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:40.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.605+0000 7f6b39400640 1 -- 172.21.15.43:0/4063768645 shutdown_connections 2024-03-28T22:36:40.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.605+0000 7f6b39400640 1 -- 172.21.15.43:0/4063768645 wait complete. 2024-03-28T22:36:40.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.606+0000 7f6b39400640 1 Processor -- start 2024-03-28T22:36:40.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.606+0000 7f6b39400640 1 -- start start 2024-03-28T22:36:40.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.607+0000 7f6b39400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b34103850 0x7f6b34199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.607+0000 7f6b39400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b34104a40 0x7f6b3419a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.607+0000 7f6b39400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b34105400 0x7f6b341a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.607+0000 7f6b39400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3410f4d0 con 0x7f6b34104a40 2024-03-28T22:36:40.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.607+0000 7f6b39400640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3410f350 con 0x7f6b34103850 2024-03-28T22:36:40.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.607+0000 7f6b39400640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3410f650 con 0x7f6b34105400 2024-03-28T22:36:40.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.607+0000 7f6b38a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b34105400 0x7f6b341a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.607+0000 7f6b33e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b34103850 0x7f6b34199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.607+0000 7f6b2b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b34104a40 0x7f6b3419a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.607+0000 7f6b38a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b34105400 0x7f6b341a10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:53700/0 (socket says 172.21.15.43:53700) 2024-03-28T22:36:40.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.607+0000 7f6b2b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b34104a40 0x7f6b3419a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48362/0 (socket says 172.21.15.43:48362) 2024-03-28T22:36:40.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.607+0000 7f6b38a00640 1 -- 172.21.15.43:0/2189660356 learned_addr learned my addr 172.21.15.43:0/2189660356 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:40.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.607+0000 7f6b33e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b34103850 0x7f6b34199b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47240/0 (socket says 172.21.15.43:47240) 2024-03-28T22:36:40.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.607+0000 7f6b33e00640 1 -- 172.21.15.43:0/2189660356 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b34105400 msgr2=0x7f6b341a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:40.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.607+0000 7f6b33e00640 1 --2- 172.21.15.43:0/2189660356 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b34105400 0x7f6b341a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.607+0000 7f6b33e00640 1 -- 172.21.15.43:0/2189660356 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b34104a40 msgr2=0x7f6b3419a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:40.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.607+0000 7f6b33e00640 1 --2- 172.21.15.43:0/2189660356 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b34104a40 0x7f6b3419a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.607+0000 7f6b33e00640 1 -- 172.21.15.43:0/2189660356 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b341a1730 con 0x7f6b34103850 2024-03-28T22:36:40.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.607+0000 7f6b38a00640 1 --2- 172.21.15.43:0/2189660356 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b34105400 0x7f6b341a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:40.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.608+0000 7f6b33e00640 1 --2- 172.21.15.43:0/2189660356 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b34103850 0x7f6b34199b10 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6b24005fd0 tx=0x7f6b2400ef30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:40.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.608+0000 7f6b31600640 1 -- 172.21.15.43:0/2189660356 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b2400cde0 con 0x7f6b34103850 2024-03-28T22:36:40.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.608+0000 7f6b31600640 1 -- 172.21.15.43:0/2189660356 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b24011950 con 0x7f6b34103850 2024-03-28T22:36:40.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.608+0000 7f6b31600640 1 -- 172.21.15.43:0/2189660356 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b2401aa80 con 0x7f6b34103850 2024-03-28T22:36:40.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.608+0000 7f6b39400640 1 -- 172.21.15.43:0/2189660356 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b341a1a20 con 0x7f6b34103850 2024-03-28T22:36:40.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.608+0000 7f6b39400640 1 -- 172.21.15.43:0/2189660356 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b341a1f20 con 0x7f6b34103850 2024-03-28T22:36:40.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.611+0000 7f6b31600640 1 -- 172.21.15.43:0/2189660356 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f6b24011b00 con 0x7f6b34103850 2024-03-28T22:36:40.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.611+0000 7f6b3d791740 1 Processor -- start 2024-03-28T22:36:40.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.612+0000 7f6b3d791740 1 -- start start 2024-03-28T22:36:40.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.612+0000 7f6b3d791740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565082579740 0x565082579b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.612+0000 7f6b3d791740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56508257a130 0x56508257ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.612+0000 7f6b3d791740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56508257f2a0 0x565082581730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.612+0000 7f6b3d791740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5650824dd9c0 con 0x565082579740 2024-03-28T22:36:40.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.612+0000 7f6b3d791740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x565082486c30 con 0x56508257a130 2024-03-28T22:36:40.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.612+0000 7f6b3d791740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5650824d9a60 con 0x56508257f2a0 2024-03-28T22:36:40.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.612+0000 7f6b33e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56508257a130 0x56508257ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.612+0000 7f6b2b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56508257f2a0 0x565082581730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.612+0000 7f6b38a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565082579740 0x565082579b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.612+0000 7f6b2b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56508257f2a0 0x565082581730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:53712/0 (socket says 172.21.15.43:53712) 2024-03-28T22:36:40.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.612+0000 7f6b2b400640 1 -- 172.21.15.43:0/1321941524 learned_addr learned my addr 172.21.15.43:0/1321941524 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:40.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.613+0000 7f6b2b400640 1 -- 172.21.15.43:0/1321941524 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56508257a130 msgr2=0x56508257ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:40.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.613+0000 7f6b2b400640 1 --2- 172.21.15.43:0/1321941524 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56508257a130 0x56508257ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.613+0000 7f6b2b400640 1 -- 172.21.15.43:0/1321941524 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565082579740 msgr2=0x565082579b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:40.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.613+0000 7f6b2b400640 1 --2- 172.21.15.43:0/1321941524 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565082579740 0x565082579b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.613+0000 7f6b2b400640 1 -- 172.21.15.43:0/1321941524 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650821f22d0 con 0x56508257f2a0 2024-03-28T22:36:40.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.613+0000 7f6b33e00640 1 --2- 172.21.15.43:0/1321941524 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56508257a130 0x56508257ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:40.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.613+0000 7f6b31600640 1 --2- 172.21.15.43:0/2189660356 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f6b180a1d80 0x7f6b180a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.613+0000 7f6b31600640 1 -- 172.21.15.43:0/2189660356 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f6b240ce6f0 con 0x7f6b34103850 2024-03-28T22:36:40.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.613+0000 7f6b2b400640 1 --2- 172.21.15.43:0/1321941524 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56508257f2a0 0x565082581730 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c009c20 tx=0x7f6b1c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7f8f3f8d2d25f00 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:40.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.613+0000 7f6b38a00640 1 --2- 172.21.15.43:0/2189660356 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f6b180a1d80 0x7f6b180a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.613+0000 7f6b29200640 1 -- 172.21.15.43:0/1321941524 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c004230 con 0x56508257f2a0 2024-03-28T22:36:40.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.613+0000 7f6b29200640 1 -- 172.21.15.43:0/1321941524 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b1c0043d0 con 0x56508257f2a0 2024-03-28T22:36:40.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.613+0000 7f6b29200640 1 -- 172.21.15.43:0/1321941524 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c004db0 con 0x56508257f2a0 2024-03-28T22:36:40.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.614+0000 7f6b3d791740 1 -- 172.21.15.43:0/1321941524 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56508257f2a0 msgr2=0x565082581730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:40.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.614+0000 7f6b3d791740 1 --2- 172.21.15.43:0/1321941524 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56508257f2a0 0x565082581730 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c009c20 tx=0x7f6b1c011810 comp rx=0 tx=0).stop 2024-03-28T22:36:40.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.614+0000 7f6b38a00640 1 --2- 172.21.15.43:0/2189660356 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f6b180a1d80 0x7f6b180a4240 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c02ce70 tx=0x7f6b2c02e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:40.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.614+0000 7f6b3d791740 1 -- 172.21.15.43:0/1321941524 shutdown_connections 2024-03-28T22:36:40.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.614+0000 7f6b3d791740 1 --2- 172.21.15.43:0/1321941524 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56508257a130 0x56508257ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.614+0000 7f6b3d791740 1 --2- 172.21.15.43:0/1321941524 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56508257f2a0 0x565082581730 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.614+0000 7f6b3d791740 1 --2- 172.21.15.43:0/1321941524 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565082579740 0x565082579b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.614+0000 7f6b3d791740 1 -- 172.21.15.43:0/1321941524 >> 172.21.15.43:0/1321941524 conn(0x565082574cb0 msgr2=0x5650825750b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:40.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.614+0000 7f6b3d791740 1 -- 172.21.15.43:0/1321941524 shutdown_connections 2024-03-28T22:36:40.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.614+0000 7f6b3d791740 1 -- 172.21.15.43:0/1321941524 wait complete. 2024-03-28T22:36:40.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.615+0000 7f6b3d791740 1 -- start start 2024-03-28T22:36:40.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.615+0000 7f6b3d791740 1 Processor -- start 2024-03-28T22:36:40.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.615+0000 7f6b3d791740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x565082579740 0x565082583c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.615+0000 7f6b3d791740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56508257a130 0x5650825841a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.615+0000 7f6b3d791740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56508257f2a0 0x56508258b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.615+0000 7f6b3d791740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x565082582250 con 0x56508257f2a0 2024-03-28T22:36:40.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.615+0000 7f6b3d791740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x56508247c180 con 0x56508257a130 2024-03-28T22:36:40.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.615+0000 7f6b3d791740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5650825823d0 con 0x565082579740 2024-03-28T22:36:40.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.616+0000 7f6b33e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x565082579740 0x565082583c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.616+0000 7f6b33e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x565082579740 0x565082583c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:53716/0 (socket says 172.21.15.43:53716) 2024-03-28T22:36:40.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.616+0000 7f6b33e00640 1 -- 172.21.15.43:0/1928863988 learned_addr learned my addr 172.21.15.43:0/1928863988 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:40.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.616+0000 7f6b3d791740 10 client.0 _ll_register_callbacks cb 0x5650821f1ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:36:40.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.616+0000 7f6b2b400640 1 --2- 172.21.15.43:0/1928863988 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56508257a130 0x5650825841a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.616+0000 7f6b38a00640 1 --2- 172.21.15.43:0/1928863988 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56508257f2a0 0x56508258b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.616+0000 7f6b33e00640 1 -- 172.21.15.43:0/1928863988 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56508257a130 msgr2=0x5650825841a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:40.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.616+0000 7f6b33e00640 1 --2- 172.21.15.43:0/1928863988 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56508257a130 0x5650825841a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.616+0000 7f6b33e00640 1 -- 172.21.15.43:0/1928863988 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56508257f2a0 msgr2=0x56508258b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:40.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.616+0000 7f6b33e00640 1 --2- 172.21.15.43:0/1928863988 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56508257f2a0 0x56508258b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.616+0000 7f6b33e00640 1 -- 172.21.15.43:0/1928863988 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650822bc6b0 con 0x565082579740 2024-03-28T22:36:40.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.616+0000 7f6b38a00640 1 --2- 172.21.15.43:0/1928863988 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56508257f2a0 0x56508258b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:36:40.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.616+0000 7f6b33e00640 1 --2- 172.21.15.43:0/1928863988 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x565082579740 0x565082583c60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6b2400c700 tx=0x7f6b24097390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97d6615a97cbdaf3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:40.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.616+0000 7f6b21600640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:36:40.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.616+0000 7f6b21600640 1 -- 172.21.15.43:0/1928863988 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b2409b730 con 0x565082579740 2024-03-28T22:36:40.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.617+0000 7f6b21600640 1 -- 172.21.15.43:0/1928863988 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f6b240a3340 con 0x565082579740 2024-03-28T22:36:40.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.617+0000 7f6b3d791740 1 -- 172.21.15.43:0/1928863988 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b1c0043d0 con 0x565082579740 2024-03-28T22:36:40.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.617+0000 7f6b21600640 1 -- 172.21.15.43:0/1928863988 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6b2409b030 con 0x565082579740 2024-03-28T22:36:40.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.617+0000 7f6b3d791740 10 client.25387 fetch_fsmap learned FSMap version 28 2024-03-28T22:36:40.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.617+0000 7f6b3d791740 1 -- 172.21.15.43:0/1928863988 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56508258e610 con 0x565082579740 2024-03-28T22:36:40.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.617+0000 7f6b21600640 1 -- 172.21.15.43:0/1928863988 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b1c0043d0 con 0x565082579740 2024-03-28T22:36:40.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.617+0000 7f6b21600640 1 -- 172.21.15.43:0/1928863988 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6b24022030 con 0x565082579740 2024-03-28T22:36:40.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.617+0000 7f6b3d791740 10 client.25387 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T22:36:40.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.617+0000 7f6b3d791740 10 client.25387 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-28T22:36:40.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.617+0000 7f6b3d791740 20 client.25387 populate_metadata read hostname 'smithi043' 2024-03-28T22:36:40.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.617+0000 7f6b3d791740 1 --2- 172.21.15.43:0/1928863988 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x565082590c00 0x565082592070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.617+0000 7f6b3d791740 4 client.25387 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:36:40.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.617+0000 7f6b3d791740 1 -- 172.21.15.43:0/1928863988 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5650824eaf00 con 0x565082590c00 2024-03-28T22:36:40.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.618+0000 7f6b2b400640 1 --2- 172.21.15.43:0/1928863988 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x565082590c00 0x565082592070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.618+0000 7f6b2b400640 1 --2- 172.21.15.43:0/1928863988 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x565082590c00 0x565082592070 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2982f09f80fbda7c server_cookie=b25778274fc7b60b in_seq=0 out_seq=0 2024-03-28T22:36:40.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.618+0000 7f6b21600640 10 client.25387 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:36:40.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.619+0000 7f6b21600640 1 -- 172.21.15.43:0/1928863988 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5650824eaf00 con 0x565082590c00 2024-03-28T22:36:40.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.619+0000 7f6b21600640 10 client.25387 handle_command_reply: tid=0 2024-03-28T22:36:40.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.619+0000 7f6b3d791740 1 client.25387 shutdown 2024-03-28T22:36:40.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.620+0000 7f6b3d791740 1 -- 172.21.15.43:0/1928863988 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x565082579740 msgr2=0x565082583c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:40.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.620+0000 7f6b3d791740 1 --2- 172.21.15.43:0/1928863988 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x565082579740 0x565082583c60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6b2400c700 tx=0x7f6b24097390 comp rx=0 tx=0).stop 2024-03-28T22:36:40.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.620+0000 7f6b3d791740 1 -- 172.21.15.43:0/1928863988 shutdown_connections 2024-03-28T22:36:40.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.620+0000 7f6b3d791740 1 --2- 172.21.15.43:0/1928863988 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x565082590c00 0x565082592070 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.620+0000 7f6b3d791740 1 --2- 172.21.15.43:0/1928863988 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56508257a130 0x5650825841a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.620+0000 7f6b3d791740 1 --2- 172.21.15.43:0/1928863988 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56508257f2a0 0x56508258b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.620+0000 7f6b21600640 0 client.25387 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:36:40.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.620+0000 7f6b3d791740 1 --2- 172.21.15.43:0/1928863988 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x565082579740 0x565082583c60 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.620+0000 7f6b3d791740 1 -- 172.21.15.43:0/1928863988 >> 172.21.15.43:0/1928863988 conn(0x565082574cb0 msgr2=0x565082573df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:40.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.620+0000 7f6b3d791740 1 -- 172.21.15.43:0/1928863988 shutdown_connections 2024-03-28T22:36:40.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.620+0000 7f6b3d791740 1 -- 172.21.15.43:0/1928863988 wait complete. 2024-03-28T22:36:40.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.620+0000 7f6b3d791740 20 client.25387 trim_cache size 0 max 16384 2024-03-28T22:36:40.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.640+0000 7f6b3d791740 1 Processor -- start 2024-03-28T22:36:40.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.641+0000 7f6b3d791740 1 -- start start 2024-03-28T22:36:40.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.641+0000 7f6b3d791740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565082579740 0x56508257b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.641+0000 7f6b3d791740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56508257a130 0x56508258b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.641+0000 7f6b3d791740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56508257f2a0 0x56508257bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.641+0000 7f6b3d791740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5650825903b0 con 0x56508257a130 2024-03-28T22:36:40.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.641+0000 7f6b3d791740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x565082590a80 con 0x565082579740 2024-03-28T22:36:40.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.641+0000 7f6b3d791740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x565082591150 con 0x56508257f2a0 2024-03-28T22:36:40.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.641+0000 7f6b33e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565082579740 0x56508257b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.641+0000 7f6b33e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565082579740 0x56508257b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47262/0 (socket says 172.21.15.43:47262) 2024-03-28T22:36:40.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.641+0000 7f6b33e00640 1 -- 172.21.15.43:0/3890119737 learned_addr learned my addr 172.21.15.43:0/3890119737 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:40.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.641+0000 7f6b38a00640 1 --2- 172.21.15.43:0/3890119737 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56508257f2a0 0x56508257bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.641+0000 7f6b2b400640 1 --2- 172.21.15.43:0/3890119737 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56508257a130 0x56508258b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.642+0000 7f6b33e00640 1 -- 172.21.15.43:0/3890119737 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56508257f2a0 msgr2=0x56508257bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:40.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.642+0000 7f6b33e00640 1 --2- 172.21.15.43:0/3890119737 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56508257f2a0 0x56508257bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.642+0000 7f6b33e00640 1 -- 172.21.15.43:0/3890119737 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56508257a130 msgr2=0x56508258b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:40.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.642+0000 7f6b33e00640 1 --2- 172.21.15.43:0/3890119737 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56508257a130 0x56508258b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.642+0000 7f6b33e00640 1 -- 172.21.15.43:0/3890119737 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56508258e410 con 0x565082579740 2024-03-28T22:36:40.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.642+0000 7f6b33e00640 1 --2- 172.21.15.43:0/3890119737 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565082579740 0x56508257b5f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6b24097490 tx=0x7f6b24005e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fd9244ed2594e28 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:40.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.642+0000 7f6b23e00640 1 -- 172.21.15.43:0/3890119737 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b2409b730 con 0x565082579740 2024-03-28T22:36:40.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.642+0000 7f6b23e00640 1 -- 172.21.15.43:0/3890119737 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b240a37d0 con 0x565082579740 2024-03-28T22:36:40.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.642+0000 7f6b23e00640 1 -- 172.21.15.43:0/3890119737 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b2409a6e0 con 0x565082579740 2024-03-28T22:36:40.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.643+0000 7f6b3d791740 1 -- 172.21.15.43:0/3890119737 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565082579740 msgr2=0x56508257b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:40.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.643+0000 7f6b3d791740 1 --2- 172.21.15.43:0/3890119737 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565082579740 0x56508257b5f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6b24097490 tx=0x7f6b24005e30 comp rx=0 tx=0).stop 2024-03-28T22:36:40.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.643+0000 7f6b3d791740 1 -- 172.21.15.43:0/3890119737 shutdown_connections 2024-03-28T22:36:40.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.643+0000 7f6b3d791740 1 --2- 172.21.15.43:0/3890119737 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56508257f2a0 0x56508257bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.643+0000 7f6b3d791740 1 --2- 172.21.15.43:0/3890119737 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56508257a130 0x56508258b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.643+0000 7f6b3d791740 1 --2- 172.21.15.43:0/3890119737 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565082579740 0x56508257b5f0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.643+0000 7f6b3d791740 1 -- 172.21.15.43:0/3890119737 >> 172.21.15.43:0/3890119737 conn(0x565082574cb0 msgr2=0x565082575090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:40.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.643+0000 7f6b3d791740 1 -- 172.21.15.43:0/3890119737 shutdown_connections 2024-03-28T22:36:40.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.643+0000 7f6b3d791740 1 -- 172.21.15.43:0/3890119737 wait complete. 2024-03-28T22:36:40.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.643+0000 7f6b3d791740 1 -- start start 2024-03-28T22:36:40.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.644+0000 7f6b3d791740 1 Processor -- start 2024-03-28T22:36:40.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.644+0000 7f6b3d791740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565082579740 0x565082587040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.644+0000 7f6b3d791740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56508257a130 0x565082587580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.644+0000 7f6b3d791740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56508257f2a0 0x56508258a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.644+0000 7f6b3d791740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x565082591ef0 con 0x56508257f2a0 2024-03-28T22:36:40.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.644+0000 7f6b3d791740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x565082591820 con 0x565082579740 2024-03-28T22:36:40.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.644+0000 7f6b3d791740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5650825925c0 con 0x56508257a130 2024-03-28T22:36:40.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.644+0000 7f6b3d791740 10 client.0 _ll_register_callbacks cb 0x7f6b240d4220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:36:40.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.644+0000 7f6b2b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56508257a130 0x565082587580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.644+0000 7f6b33e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565082579740 0x565082587040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.644+0000 7f6b2b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56508257a130 0x565082587580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:53746/0 (socket says 172.21.15.43:53746) 2024-03-28T22:36:40.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.644+0000 7f6b2b400640 1 -- 172.21.15.43:0/4169305960 learned_addr learned my addr 172.21.15.43:0/4169305960 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:40.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.644+0000 7f6b38a00640 1 --2- 172.21.15.43:0/4169305960 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56508257f2a0 0x56508258a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.645+0000 7f6b2b400640 1 -- 172.21.15.43:0/4169305960 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565082579740 msgr2=0x565082587040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:40.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.645+0000 7f6b2b400640 1 --2- 172.21.15.43:0/4169305960 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565082579740 0x565082587040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.645+0000 7f6b2b400640 1 -- 172.21.15.43:0/4169305960 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56508257f2a0 msgr2=0x56508258a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:40.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.645+0000 7f6b2b400640 1 --2- 172.21.15.43:0/4169305960 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56508257f2a0 0x56508258a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.645+0000 7f6b2b400640 1 -- 172.21.15.43:0/4169305960 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56508258ab90 con 0x56508257a130 2024-03-28T22:36:40.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.645+0000 7f6b33e00640 1 --2- 172.21.15.43:0/4169305960 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565082579740 0x565082587040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:40.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.645+0000 7f6b2b400640 1 --2- 172.21.15.43:0/4169305960 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56508257a130 0x565082587580 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x565082575c20 tx=0x7f6b1c0096d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=912124e66978fe8a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:40.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.645+0000 7f6b21600640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:36:40.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.645+0000 7f6b21600640 1 -- 172.21.15.43:0/4169305960 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c01e040 con 0x56508257a130 2024-03-28T22:36:40.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.645+0000 7f6b21600640 1 -- 172.21.15.43:0/4169305960 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f6b1c005230 con 0x56508257a130 2024-03-28T22:36:40.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.645+0000 7f6b3d791740 1 -- 172.21.15.43:0/4169305960 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b240a37d0 con 0x56508257a130 2024-03-28T22:36:40.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.646+0000 7f6b21600640 1 -- 172.21.15.43:0/4169305960 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6b1c038050 con 0x56508257a130 2024-03-28T22:36:40.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.646+0000 7f6b3d791740 10 client.25393 fetch_fsmap learned FSMap version 28 2024-03-28T22:36:40.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.646+0000 7f6b3d791740 1 -- 172.21.15.43:0/4169305960 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5650825f7120 con 0x56508257a130 2024-03-28T22:36:40.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.646+0000 7f6b21600640 1 -- 172.21.15.43:0/4169305960 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b240a37d0 con 0x56508257a130 2024-03-28T22:36:40.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.646+0000 7f6b21600640 1 -- 172.21.15.43:0/4169305960 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6b1c024040 con 0x56508257a130 2024-03-28T22:36:40.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.646+0000 7f6b3d791740 10 client.25393 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T22:36:40.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.646+0000 7f6b3d791740 10 client.25393 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-28T22:36:40.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.646+0000 7f6b3d791740 20 client.25393 populate_metadata read hostname 'smithi043' 2024-03-28T22:36:40.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.646+0000 7f6b3d791740 1 --2- 172.21.15.43:0/4169305960 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x5650825fbbe0 0x5650825fa8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.646+0000 7f6b3d791740 4 client.25393 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T22:36:40.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.646+0000 7f6b3d791740 1 -- 172.21.15.43:0/4169305960 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "status"}) -- 0x5650824e5250 con 0x5650825fbbe0 2024-03-28T22:36:40.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.647+0000 7f6b33e00640 1 --2- 172.21.15.43:0/4169305960 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x5650825fbbe0 0x5650825fa8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.647+0000 7f6b33e00640 1 --2- 172.21.15.43:0/4169305960 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x5650825fbbe0 0x5650825fa8f0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1d6645a9d53680f3 server_cookie=24b2ddc5a330d488 in_seq=0 out_seq=0 2024-03-28T22:36:40.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.647+0000 7f6b21600640 10 client.25393 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:36:40.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.647+0000 7f6b21600640 1 -- 172.21.15.43:0/4169305960 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+321 (crc 0 0 0) 0x5650824e5250 con 0x5650825fbbe0 2024-03-28T22:36:40.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.647+0000 7f6b21600640 10 client.25393 handle_command_reply: tid=0 2024-03-28T22:36:40.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.647+0000 7f6b3d791740 1 client.25393 shutdown 2024-03-28T22:36:40.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.647+0000 7f6b3d791740 1 -- 172.21.15.43:0/4169305960 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56508257a130 msgr2=0x565082587580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:40.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.647+0000 7f6b3d791740 1 --2- 172.21.15.43:0/4169305960 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56508257a130 0x565082587580 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x565082575c20 tx=0x7f6b1c0096d0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.648+0000 7f6b3d791740 1 -- 172.21.15.43:0/4169305960 shutdown_connections 2024-03-28T22:36:40.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.648+0000 7f6b3d791740 1 --2- 172.21.15.43:0/4169305960 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x5650825fbbe0 0x5650825fa8f0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.648+0000 7f6b3d791740 1 --2- 172.21.15.43:0/4169305960 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56508257f2a0 0x56508258a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.648+0000 7f6b3d791740 1 --2- 172.21.15.43:0/4169305960 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56508257a130 0x565082587580 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.648+0000 7f6b3d791740 1 --2- 172.21.15.43:0/4169305960 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565082579740 0x565082587040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.648+0000 7f6b21600640 0 client.25393 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:36:40.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.648+0000 7f6b3d791740 1 -- 172.21.15.43:0/4169305960 >> 172.21.15.43:0/4169305960 conn(0x565082574cb0 msgr2=0x56508257b5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:40.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.648+0000 7f6b3d791740 1 -- 172.21.15.43:0/4169305960 shutdown_connections 2024-03-28T22:36:40.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.648+0000 7f6b3d791740 1 -- 172.21.15.43:0/4169305960 wait complete. 2024-03-28T22:36:40.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.648+0000 7f6b3d791740 20 client.25393 trim_cache size 0 max 16384 2024-03-28T22:36:40.651 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:36:40.651 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_fsid": "c2276428-ed51-11ee-95d0-87774f69a715", 2024-03-28T22:36:40.651 INFO:teuthology.orchestra.run.smithi043.stdout: "whoami": 4, 2024-03-28T22:36:40.651 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 24518, 2024-03-28T22:36:40.651 INFO:teuthology.orchestra.run.smithi043.stdout: "want_state": "up:active", 2024-03-28T22:36:40.652 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "up:active", 2024-03-28T22:36:40.652 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs", 2024-03-28T22:36:40.652 INFO:teuthology.orchestra.run.smithi043.stdout: "rank_uptime": 155.480282759, 2024-03-28T22:36:40.652 INFO:teuthology.orchestra.run.smithi043.stdout: "mdsmap_epoch": 28, 2024-03-28T22:36:40.652 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch": 96, 2024-03-28T22:36:40.652 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch_barrier": 92, 2024-03-28T22:36:40.652 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 211.96125182 2024-03-28T22:36:40.652 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:36:40.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.648+0000 7f6b39400640 1 -- 172.21.15.43:0/2189660356 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f6b180a1d80 msgr2=0x7f6b180a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:40.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.648+0000 7f6b39400640 1 --2- 172.21.15.43:0/2189660356 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f6b180a1d80 0x7f6b180a4240 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c02ce70 tx=0x7f6b2c02e040 comp rx=0 tx=0).stop 2024-03-28T22:36:40.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.648+0000 7f6b39400640 1 -- 172.21.15.43:0/2189660356 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b34103850 msgr2=0x7f6b34199b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:40.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.648+0000 7f6b39400640 1 --2- 172.21.15.43:0/2189660356 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b34103850 0x7f6b34199b10 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6b24005fd0 tx=0x7f6b2400ef30 comp rx=0 tx=0).stop 2024-03-28T22:36:40.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.649+0000 7f6b39400640 1 -- 172.21.15.43:0/2189660356 shutdown_connections 2024-03-28T22:36:40.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.649+0000 7f6b39400640 1 --2- 172.21.15.43:0/2189660356 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b34105400 0x7f6b341a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.649+0000 7f6b39400640 1 --2- 172.21.15.43:0/2189660356 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b34104a40 0x7f6b3419a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.649+0000 7f6b39400640 1 --2- 172.21.15.43:0/2189660356 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f6b180a1d80 0x7f6b180a4240 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.649+0000 7f6b39400640 1 --2- 172.21.15.43:0/2189660356 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b34103850 0x7f6b34199b10 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.649+0000 7f6b39400640 1 -- 172.21.15.43:0/2189660356 >> 172.21.15.43:0/2189660356 conn(0x7f6b340fedc0 msgr2=0x7f6b34107dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:40.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.649+0000 7f6b39400640 1 -- 172.21.15.43:0/2189660356 shutdown_connections 2024-03-28T22:36:40.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.649+0000 7f6b39400640 1 -- 172.21.15.43:0/2189660356 wait complete. 2024-03-28T22:36:40.665 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-03-28T22:36:40.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.890+0000 7f7ceb200640 1 -- 172.21.15.43:0/1283951487 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ce4104b80 msgr2=0x7f7ce4105000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:40.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.890+0000 7f7ceb200640 1 --2- 172.21.15.43:0/1283951487 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ce4104b80 0x7f7ce4105000 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc009900 tx=0x7f7ccc02bc50 comp rx=0 tx=0).stop 2024-03-28T22:36:40.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.890+0000 7f7ceb200640 1 -- 172.21.15.43:0/1283951487 shutdown_connections 2024-03-28T22:36:40.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.890+0000 7f7ceb200640 1 --2- 172.21.15.43:0/1283951487 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ce4105540 0x7f7ce410bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.890+0000 7f7ceb200640 1 --2- 172.21.15.43:0/1283951487 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ce4104b80 0x7f7ce4105000 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.890+0000 7f7ceb200640 1 --2- 172.21.15.43:0/1283951487 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ce4103990 0x7f7ce4103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.890+0000 7f7ceb200640 1 -- 172.21.15.43:0/1283951487 >> 172.21.15.43:0/1283951487 conn(0x7f7ce40fef00 msgr2=0x7f7ce4101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:40.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.890+0000 7f7ceb200640 1 -- 172.21.15.43:0/1283951487 shutdown_connections 2024-03-28T22:36:40.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.890+0000 7f7ceb200640 1 -- 172.21.15.43:0/1283951487 wait complete. 2024-03-28T22:36:40.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.892+0000 7f7ceb200640 1 Processor -- start 2024-03-28T22:36:40.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.892+0000 7f7ceb200640 1 -- start start 2024-03-28T22:36:40.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.892+0000 7f7ceb200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ce4103990 0x7f7ce4199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.892+0000 7f7ceb200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ce4104b80 0x7f7ce419a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.892+0000 7f7ceb200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ce4105540 0x7f7ce41a1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.892+0000 7f7ceb200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce410f610 con 0x7f7ce4104b80 2024-03-28T22:36:40.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.892+0000 7f7ceb200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce410f490 con 0x7f7ce4103990 2024-03-28T22:36:40.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.892+0000 7f7ceb200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce410f790 con 0x7f7ce4105540 2024-03-28T22:36:40.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.892+0000 7f7ce3e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ce4103990 0x7f7ce4199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.893+0000 7f7ce3e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ce4103990 0x7f7ce4199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47280/0 (socket says 172.21.15.43:47280) 2024-03-28T22:36:40.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.893+0000 7f7ce3e00640 1 -- 172.21.15.43:0/3090126976 learned_addr learned my addr 172.21.15.43:0/3090126976 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:40.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.892+0000 7f7ce9200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ce4105540 0x7f7ce41a1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.893+0000 7f7cdbe00640 1 --2- 172.21.15.43:0/3090126976 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ce4104b80 0x7f7ce419a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.893+0000 7f7ce3e00640 1 -- 172.21.15.43:0/3090126976 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ce4105540 msgr2=0x7f7ce41a1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:40.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.893+0000 7f7ce3e00640 1 --2- 172.21.15.43:0/3090126976 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ce4105540 0x7f7ce41a1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.893+0000 7f7ce3e00640 1 -- 172.21.15.43:0/3090126976 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ce4104b80 msgr2=0x7f7ce419a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:40.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.893+0000 7f7ce3e00640 1 --2- 172.21.15.43:0/3090126976 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ce4104b80 0x7f7ce419a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.893+0000 7f7ce3e00640 1 -- 172.21.15.43:0/3090126976 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ce41a1970 con 0x7f7ce4103990 2024-03-28T22:36:40.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.893+0000 7f7ce9200640 1 --2- 172.21.15.43:0/3090126976 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ce4105540 0x7f7ce41a1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:40.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.893+0000 7f7ce3e00640 1 --2- 172.21.15.43:0/3090126976 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ce4103990 0x7f7ce4199d50 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7cd400e9c0 tx=0x7f7cd400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:40.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.894+0000 7f7ce1600640 1 -- 172.21.15.43:0/3090126976 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd400cd80 con 0x7f7ce4103990 2024-03-28T22:36:40.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.894+0000 7f7ce1600640 1 -- 172.21.15.43:0/3090126976 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cd40035f0 con 0x7f7ce4103990 2024-03-28T22:36:40.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.894+0000 7f7ce1600640 1 -- 172.21.15.43:0/3090126976 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd4010690 con 0x7f7ce4103990 2024-03-28T22:36:40.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.894+0000 7f7ceb200640 1 -- 172.21.15.43:0/3090126976 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ce41a1c60 con 0x7f7ce4103990 2024-03-28T22:36:40.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.894+0000 7f7ceb200640 1 -- 172.21.15.43:0/3090126976 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ce41a21d0 con 0x7f7ce4103990 2024-03-28T22:36:40.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.897+0000 7f7cede3b740 1 Processor -- start 2024-03-28T22:36:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.897+0000 7f7cede3b740 1 -- start start 2024-03-28T22:36:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.897+0000 7f7ce1600640 1 -- 172.21.15.43:0/3090126976 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f7cd40037a0 con 0x7f7ce4103990 2024-03-28T22:36:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.898+0000 7f7cede3b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557c25e90570 0x557c25e90990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.898+0000 7f7cede3b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557c25e90f60 0x557c25e95a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.898+0000 7f7cede3b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557c25e960d0 0x557c25e98560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.898+0000 7f7cede3b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x557c25df49c0 con 0x557c25e960d0 2024-03-28T22:36:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.898+0000 7f7cede3b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x557c25d9dc30 con 0x557c25e90570 2024-03-28T22:36:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.898+0000 7f7cede3b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x557c25df0a60 con 0x557c25e90f60 2024-03-28T22:36:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.898+0000 7f7ce9200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557c25e90570 0x557c25e90990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.898+0000 7f7ce9200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557c25e90570 0x557c25e90990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47292/0 (socket says 172.21.15.43:47292) 2024-03-28T22:36:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.898+0000 7f7ce9200640 1 -- 172.21.15.43:0/3609125480 learned_addr learned my addr 172.21.15.43:0/3609125480 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.898+0000 7f7ce3e00640 1 --2- 172.21.15.43:0/3609125480 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557c25e90f60 0x557c25e95a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.898+0000 7f7cdbe00640 1 --2- 172.21.15.43:0/3609125480 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557c25e960d0 0x557c25e98560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.899+0000 7f7ce9200640 1 -- 172.21.15.43:0/3609125480 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557c25e90f60 msgr2=0x557c25e95a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:40.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.899+0000 7f7ce9200640 1 --2- 172.21.15.43:0/3609125480 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557c25e90f60 0x557c25e95a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.899+0000 7f7ce9200640 1 -- 172.21.15.43:0/3609125480 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557c25e960d0 msgr2=0x557c25e98560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:40.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.899+0000 7f7ce9200640 1 --2- 172.21.15.43:0/3609125480 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557c25e960d0 0x557c25e98560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.899+0000 7f7ce9200640 1 -- 172.21.15.43:0/3609125480 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c25b092d0 con 0x557c25e90570 2024-03-28T22:36:40.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.899+0000 7f7ce9200640 1 --2- 172.21.15.43:0/3609125480 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557c25e90570 0x557c25e90990 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc0117e0 tx=0x7f7cdc011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efd112e6cf1f73e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:40.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.899+0000 7f7cd9600640 1 -- 172.21.15.43:0/3609125480 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc011310 con 0x557c25e90570 2024-03-28T22:36:40.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.899+0000 7f7cd9600640 1 -- 172.21.15.43:0/3609125480 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cdc003e50 con 0x557c25e90570 2024-03-28T22:36:40.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.899+0000 7f7cd9600640 1 -- 172.21.15.43:0/3609125480 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc004620 con 0x557c25e90570 2024-03-28T22:36:40.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.899+0000 7f7ce1600640 1 --2- 172.21.15.43:0/3090126976 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7cc80a1e00 0x7f7cc80a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.900+0000 7f7ce1600640 1 -- 172.21.15.43:0/3090126976 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f7cd40ce4d0 con 0x7f7ce4103990 2024-03-28T22:36:40.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.900+0000 7f7ce3e00640 1 --2- 172.21.15.43:0/3090126976 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7cc80a1e00 0x7f7cc80a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.900+0000 7f7cede3b740 1 -- 172.21.15.43:0/3609125480 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557c25e90570 msgr2=0x557c25e90990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:40.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.900+0000 7f7cede3b740 1 --2- 172.21.15.43:0/3609125480 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557c25e90570 0x557c25e90990 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc0117e0 tx=0x7f7cdc011ca0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.900+0000 7f7cede3b740 1 -- 172.21.15.43:0/3609125480 shutdown_connections 2024-03-28T22:36:40.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.900+0000 7f7cede3b740 1 --2- 172.21.15.43:0/3609125480 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557c25e960d0 0x557c25e98560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.900+0000 7f7cede3b740 1 --2- 172.21.15.43:0/3609125480 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557c25e90f60 0x557c25e95a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.900+0000 7f7cede3b740 1 --2- 172.21.15.43:0/3609125480 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557c25e90570 0x557c25e90990 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.900+0000 7f7cede3b740 1 -- 172.21.15.43:0/3609125480 >> 172.21.15.43:0/3609125480 conn(0x557c25e8bae0 msgr2=0x557c25e8bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:40.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.900+0000 7f7cede3b740 1 -- 172.21.15.43:0/3609125480 shutdown_connections 2024-03-28T22:36:40.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.900+0000 7f7ce3e00640 1 --2- 172.21.15.43:0/3090126976 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7cc80a1e00 0x7f7cc80a42c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4097270 tx=0x7f7cd40d1f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:40.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.901+0000 7f7cede3b740 1 -- 172.21.15.43:0/3609125480 wait complete. 2024-03-28T22:36:40.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.901+0000 7f7cede3b740 1 -- start start 2024-03-28T22:36:40.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.901+0000 7f7cede3b740 1 Processor -- start 2024-03-28T22:36:40.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.902+0000 7f7cede3b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557c25e90570 0x557c25e9aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.902+0000 7f7cede3b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557c25e90f60 0x557c25e9afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.902+0000 7f7cede3b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557c25e960d0 0x557c25ea2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.902+0000 7f7cede3b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x557c25e99080 con 0x557c25e960d0 2024-03-28T22:36:40.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.902+0000 7f7cede3b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x557c25d93180 con 0x557c25e90f60 2024-03-28T22:36:40.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.902+0000 7f7cede3b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x557c25e99200 con 0x557c25e90570 2024-03-28T22:36:40.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.902+0000 7f7cede3b740 10 client.0 _ll_register_callbacks cb 0x557c25b08ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:36:40.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.902+0000 7f7ce9200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557c25e90570 0x557c25e9aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.902+0000 7f7cdbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557c25e90f60 0x557c25e9afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.902+0000 7f7cdbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557c25e90f60 0x557c25e9afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47308/0 (socket says 172.21.15.43:47308) 2024-03-28T22:36:40.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.902+0000 7f7cdbe00640 1 -- 172.21.15.43:0/194218614 learned_addr learned my addr 172.21.15.43:0/194218614 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:40.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.902+0000 7f7ce9200640 1 --2- 172.21.15.43:0/194218614 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557c25e960d0 0x557c25ea2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.902+0000 7f7cdbe00640 1 -- 172.21.15.43:0/194218614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557c25e90570 msgr2=0x557c25e9aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:40.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.902+0000 7f7cdbe00640 1 --2- 172.21.15.43:0/194218614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557c25e90570 0x557c25e9aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.902+0000 7f7cdbe00640 1 -- 172.21.15.43:0/194218614 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557c25e960d0 msgr2=0x557c25ea2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.902+0000 7f7cdbe00640 1 --2- 172.21.15.43:0/194218614 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557c25e960d0 0x557c25ea2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.902+0000 7f7cdbe00640 1 -- 172.21.15.43:0/194218614 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c25bd36b0 con 0x557c25e90f60 2024-03-28T22:36:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.902+0000 7f7ce9200640 1 --2- 172.21.15.43:0/194218614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557c25e90570 0x557c25e9aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:36:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.903+0000 7f7cdbe00640 1 --2- 172.21.15.43:0/194218614 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557c25e90f60 0x557c25e9afc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc009900 tx=0x7f7ccc02c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bf6c6ca9d111f15 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.903+0000 7f7cd2000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:36:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.903+0000 7f7cd2000640 1 -- 172.21.15.43:0/194218614 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc043070 con 0x557c25e90f60 2024-03-28T22:36:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.903+0000 7f7cd2000640 1 -- 172.21.15.43:0/194218614 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f7ccc005030 con 0x557c25e90f60 2024-03-28T22:36:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.903+0000 7f7cede3b740 1 -- 172.21.15.43:0/194218614 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7cdc003e50 con 0x557c25e90f60 2024-03-28T22:36:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.903+0000 7f7cd2000640 1 -- 172.21.15.43:0/194218614 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7ccc050050 con 0x557c25e90f60 2024-03-28T22:36:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.903+0000 7f7cede3b740 10 client.25397 fetch_fsmap learned FSMap version 28 2024-03-28T22:36:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.903+0000 7f7cede3b740 1 -- 172.21.15.43:0/194218614 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557c25ea53e0 con 0x557c25e90f60 2024-03-28T22:36:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.904+0000 7f7cd2000640 1 -- 172.21.15.43:0/194218614 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cdc003e50 con 0x557c25e90f60 2024-03-28T22:36:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.904+0000 7f7cd2000640 1 -- 172.21.15.43:0/194218614 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7ccc0035f0 con 0x557c25e90f60 2024-03-28T22:36:40.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.904+0000 7f7cede3b740 10 client.25397 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T22:36:40.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.904+0000 7f7cede3b740 10 client.25397 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-28T22:36:40.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.904+0000 7f7cede3b740 20 client.25397 populate_metadata read hostname 'smithi043' 2024-03-28T22:36:40.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.904+0000 7f7cede3b740 1 --2- 172.21.15.43:0/194218614 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x557c25ea7a00 0x557c25ea8e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.904+0000 7f7cede3b740 4 client.25397 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:36:40.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.904+0000 7f7cede3b740 1 -- 172.21.15.43:0/194218614 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557c25e01f00 con 0x557c25ea7a00 2024-03-28T22:36:40.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.904+0000 7f7ce9200640 1 --2- 172.21.15.43:0/194218614 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x557c25ea7a00 0x557c25ea8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.905+0000 7f7ce9200640 1 --2- 172.21.15.43:0/194218614 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x557c25ea7a00 0x557c25ea8e70 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e981c1946594b415 server_cookie=f621e8dd7ffa7722 in_seq=0 out_seq=0 2024-03-28T22:36:40.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.905+0000 7f7cd2000640 10 client.25397 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:36:40.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.905+0000 7f7cd2000640 1 -- 172.21.15.43:0/194218614 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557c25e01f00 con 0x557c25ea7a00 2024-03-28T22:36:40.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.905+0000 7f7cd2000640 10 client.25397 handle_command_reply: tid=0 2024-03-28T22:36:40.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.906+0000 7f7cede3b740 1 client.25397 shutdown 2024-03-28T22:36:40.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.906+0000 7f7cede3b740 1 -- 172.21.15.43:0/194218614 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557c25e90f60 msgr2=0x557c25e9afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:40.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.906+0000 7f7cede3b740 1 --2- 172.21.15.43:0/194218614 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557c25e90f60 0x557c25e9afc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc009900 tx=0x7f7ccc02c680 comp rx=0 tx=0).stop 2024-03-28T22:36:40.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.906+0000 7f7cede3b740 1 -- 172.21.15.43:0/194218614 shutdown_connections 2024-03-28T22:36:40.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.906+0000 7f7cede3b740 1 --2- 172.21.15.43:0/194218614 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x557c25ea7a00 0x557c25ea8e70 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.906+0000 7f7cede3b740 1 --2- 172.21.15.43:0/194218614 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557c25e90f60 0x557c25e9afc0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.906+0000 7f7cede3b740 1 --2- 172.21.15.43:0/194218614 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557c25e960d0 0x557c25ea2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.906+0000 7f7cede3b740 1 --2- 172.21.15.43:0/194218614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557c25e90570 0x557c25e9aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.906+0000 7f7cede3b740 1 -- 172.21.15.43:0/194218614 >> 172.21.15.43:0/194218614 conn(0x557c25e8bae0 msgr2=0x557c25e8ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:40.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.906+0000 7f7cd2000640 0 client.25397 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:36:40.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.906+0000 7f7cede3b740 1 -- 172.21.15.43:0/194218614 shutdown_connections 2024-03-28T22:36:40.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.906+0000 7f7cede3b740 1 -- 172.21.15.43:0/194218614 wait complete. 2024-03-28T22:36:40.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.906+0000 7f7cede3b740 20 client.25397 trim_cache size 0 max 16384 2024-03-28T22:36:40.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.926+0000 7f7cede3b740 1 Processor -- start 2024-03-28T22:36:40.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.926+0000 7f7cede3b740 1 -- start start 2024-03-28T22:36:40.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.927+0000 7f7cede3b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557c25e90570 0x557c25e98f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.927+0000 7f7cede3b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557c25e90f60 0x557c25ea1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.927+0000 7f7cede3b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557c25e960d0 0x557c25e8bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.927+0000 7f7cede3b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x557c25ea71b0 con 0x557c25e90f60 2024-03-28T22:36:40.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.927+0000 7f7cede3b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x557c25ea7880 con 0x557c25e90570 2024-03-28T22:36:40.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.927+0000 7f7cede3b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x557c25ea7f50 con 0x557c25e960d0 2024-03-28T22:36:40.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.927+0000 7f7cdbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557c25e90f60 0x557c25ea1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.927+0000 7f7ce9200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557c25e90570 0x557c25e98f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.927+0000 7f7cdbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557c25e90f60 0x557c25ea1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48430/0 (socket says 172.21.15.43:48430) 2024-03-28T22:36:40.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.927+0000 7f7cdbe00640 1 -- 172.21.15.43:0/3928323725 learned_addr learned my addr 172.21.15.43:0/3928323725 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:40.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.927+0000 7f7ce9200640 1 --2- 172.21.15.43:0/3928323725 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557c25e960d0 0x557c25e8bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.927+0000 7f7cdbe00640 1 -- 172.21.15.43:0/3928323725 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557c25e960d0 msgr2=0x557c25e8bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:40.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.927+0000 7f7cdbe00640 1 --2- 172.21.15.43:0/3928323725 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557c25e960d0 0x557c25e8bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.928+0000 7f7cdbe00640 1 -- 172.21.15.43:0/3928323725 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557c25e90570 msgr2=0x557c25e98f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:40.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.928+0000 7f7cdbe00640 1 --2- 172.21.15.43:0/3928323725 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557c25e90570 0x557c25e98f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.928+0000 7f7cdbe00640 1 -- 172.21.15.43:0/3928323725 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c25ea91a0 con 0x557c25e90f60 2024-03-28T22:36:40.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.928+0000 7f7ce9200640 1 --2- 172.21.15.43:0/3928323725 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557c25e90570 0x557c25e98f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:40.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.928+0000 7f7cdbe00640 1 --2- 172.21.15.43:0/3928323725 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557c25e90f60 0x557c25ea1fc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc000c00 tx=0x7f7ccc005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4e871471d2ac9de server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:40.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.928+0000 7f7cd8c00640 1 -- 172.21.15.43:0/3928323725 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc043070 con 0x557c25e90f60 2024-03-28T22:36:40.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.928+0000 7f7cd8c00640 1 -- 172.21.15.43:0/3928323725 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ccc059070 con 0x557c25e90f60 2024-03-28T22:36:40.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.928+0000 7f7cd8c00640 1 -- 172.21.15.43:0/3928323725 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc003c30 con 0x557c25e90f60 2024-03-28T22:36:40.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.929+0000 7f7cede3b740 1 -- 172.21.15.43:0/3928323725 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557c25e90f60 msgr2=0x557c25ea1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:40.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.929+0000 7f7cede3b740 1 --2- 172.21.15.43:0/3928323725 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557c25e90f60 0x557c25ea1fc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc000c00 tx=0x7f7ccc005850 comp rx=0 tx=0).stop 2024-03-28T22:36:40.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.929+0000 7f7cede3b740 1 -- 172.21.15.43:0/3928323725 shutdown_connections 2024-03-28T22:36:40.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.929+0000 7f7cede3b740 1 --2- 172.21.15.43:0/3928323725 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557c25e960d0 0x557c25e8bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.929+0000 7f7cede3b740 1 --2- 172.21.15.43:0/3928323725 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557c25e90f60 0x557c25ea1fc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.929+0000 7f7cede3b740 1 --2- 172.21.15.43:0/3928323725 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557c25e90570 0x557c25e98f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.929+0000 7f7cede3b740 1 -- 172.21.15.43:0/3928323725 >> 172.21.15.43:0/3928323725 conn(0x557c25e8bae0 msgr2=0x557c25e868b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:40.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.929+0000 7f7cede3b740 1 -- 172.21.15.43:0/3928323725 shutdown_connections 2024-03-28T22:36:40.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.929+0000 7f7cede3b740 1 -- 172.21.15.43:0/3928323725 wait complete. 2024-03-28T22:36:40.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.929+0000 7f7cede3b740 1 -- start start 2024-03-28T22:36:40.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.930+0000 7f7cede3b740 1 Processor -- start 2024-03-28T22:36:40.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.930+0000 7f7cede3b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557c25e90570 0x557c25e9e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.930+0000 7f7cede3b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557c25e90f60 0x557c25e9e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.930+0000 7f7cede3b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557c25e960d0 0x557c25ea11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.930+0000 7f7cede3b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x557c25ea8cf0 con 0x557c25e960d0 2024-03-28T22:36:40.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.930+0000 7f7cede3b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x557c25ea8620 con 0x557c25e90570 2024-03-28T22:36:40.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.930+0000 7f7cede3b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x557c25ea93c0 con 0x557c25e90f60 2024-03-28T22:36:40.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.930+0000 7f7cede3b740 10 client.0 _ll_register_callbacks cb 0x7f7ccc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:36:40.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.930+0000 7f7ce9200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557c25e90570 0x557c25e9e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.930+0000 7f7ce9200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557c25e960d0 0x557c25ea11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.931+0000 7f7cdbe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557c25e90f60 0x557c25e9e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.931+0000 7f7ce9200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557c25e90570 0x557c25e9e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47312/0 (socket says 172.21.15.43:47312) 2024-03-28T22:36:40.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.931+0000 7f7ce9200640 1 -- 172.21.15.43:0/1343854139 learned_addr learned my addr 172.21.15.43:0/1343854139 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:40.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.931+0000 7f7ce9200640 1 -- 172.21.15.43:0/1343854139 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557c25e90f60 msgr2=0x557c25e9e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:40.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.931+0000 7f7ce9200640 1 --2- 172.21.15.43:0/1343854139 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557c25e90f60 0x557c25e9e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.931+0000 7f7ce9200640 1 -- 172.21.15.43:0/1343854139 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557c25e960d0 msgr2=0x557c25ea11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:40.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.931+0000 7f7ce9200640 1 --2- 172.21.15.43:0/1343854139 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557c25e960d0 0x557c25ea11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.931+0000 7f7ce9200640 1 -- 172.21.15.43:0/1343854139 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c25ea17e0 con 0x557c25e90570 2024-03-28T22:36:40.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.931+0000 7f7cdbe00640 1 --2- 172.21.15.43:0/1343854139 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557c25e90f60 0x557c25e9e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:40.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.931+0000 7f7ce9200640 1 --2- 172.21.15.43:0/1343854139 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557c25e90570 0x557c25e9e150 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc0117b0 tx=0x7f7cdc005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e556ebb2c5c0367a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:40.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.931+0000 7f7cd2000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:36:40.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.931+0000 7f7cd2000640 1 -- 172.21.15.43:0/1343854139 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc011480 con 0x557c25e90570 2024-03-28T22:36:40.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.931+0000 7f7cd2000640 1 -- 172.21.15.43:0/1343854139 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f7cdc00f9a0 con 0x557c25e90570 2024-03-28T22:36:40.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.931+0000 7f7cede3b740 1 -- 172.21.15.43:0/1343854139 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ccc059070 con 0x557c25e90570 2024-03-28T22:36:40.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.932+0000 7f7cd2000640 1 -- 172.21.15.43:0/1343854139 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7cdc03e050 con 0x557c25e90570 2024-03-28T22:36:40.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.932+0000 7f7cede3b740 10 client.25403 fetch_fsmap learned FSMap version 28 2024-03-28T22:36:40.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.932+0000 7f7cede3b740 1 -- 172.21.15.43:0/1343854139 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557c25f0dc60 con 0x557c25e90570 2024-03-28T22:36:40.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.932+0000 7f7cd2000640 1 -- 172.21.15.43:0/1343854139 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ccc059070 con 0x557c25e90570 2024-03-28T22:36:40.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.932+0000 7f7cd2000640 1 -- 172.21.15.43:0/1343854139 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7cdc028040 con 0x557c25e90570 2024-03-28T22:36:40.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.932+0000 7f7cede3b740 10 client.25403 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T22:36:40.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.932+0000 7f7cede3b740 10 client.25403 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-28T22:36:40.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.932+0000 7f7cede3b740 20 client.25403 populate_metadata read hostname 'smithi043' 2024-03-28T22:36:40.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.932+0000 7f7cede3b740 1 --2- 172.21.15.43:0/1343854139 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x557c25f12620 0x557c25f11330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:40.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.932+0000 7f7cede3b740 4 client.25403 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T22:36:40.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.932+0000 7f7cede3b740 1 -- 172.21.15.43:0/1343854139 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "status"}) -- 0x557c25dfc250 con 0x557c25f12620 2024-03-28T22:36:40.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.933+0000 7f7cdbe00640 1 --2- 172.21.15.43:0/1343854139 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x557c25f12620 0x557c25f11330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:40.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.933+0000 7f7cdbe00640 1 --2- 172.21.15.43:0/1343854139 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x557c25f12620 0x557c25f11330 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=85f00e1f0ea046e4 server_cookie=bf768beb01e08483 in_seq=0 out_seq=0 2024-03-28T22:36:40.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.933+0000 7f7cd2000640 10 client.25403 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:36:40.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.933+0000 7f7cd2000640 1 -- 172.21.15.43:0/1343854139 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x557c25dfc250 con 0x557c25f12620 2024-03-28T22:36:40.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.933+0000 7f7cd2000640 10 client.25403 handle_command_reply: tid=0 2024-03-28T22:36:40.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.933+0000 7f7cede3b740 1 client.25403 shutdown 2024-03-28T22:36:40.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.934+0000 7f7cede3b740 1 -- 172.21.15.43:0/1343854139 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557c25e90570 msgr2=0x557c25e9e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:40.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.934+0000 7f7cede3b740 1 --2- 172.21.15.43:0/1343854139 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557c25e90570 0x557c25e9e150 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc0117b0 tx=0x7f7cdc005fd0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.934+0000 7f7cede3b740 1 -- 172.21.15.43:0/1343854139 shutdown_connections 2024-03-28T22:36:40.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.934+0000 7f7cede3b740 1 --2- 172.21.15.43:0/1343854139 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x557c25f12620 0x557c25f11330 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.934+0000 7f7cede3b740 1 --2- 172.21.15.43:0/1343854139 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557c25e960d0 0x557c25ea11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.934+0000 7f7cede3b740 1 --2- 172.21.15.43:0/1343854139 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557c25e90f60 0x557c25e9e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.934+0000 7f7cede3b740 1 --2- 172.21.15.43:0/1343854139 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557c25e90570 0x557c25e9e150 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.934+0000 7f7cd2000640 0 client.25403 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:36:40.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.934+0000 7f7cede3b740 1 -- 172.21.15.43:0/1343854139 >> 172.21.15.43:0/1343854139 conn(0x557c25e8bae0 msgr2=0x557c25e8bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:40.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.934+0000 7f7cede3b740 1 -- 172.21.15.43:0/1343854139 shutdown_connections 2024-03-28T22:36:40.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.934+0000 7f7cede3b740 1 -- 172.21.15.43:0/1343854139 wait complete. 2024-03-28T22:36:40.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.934+0000 7f7cede3b740 20 client.25403 trim_cache size 0 max 16384 2024-03-28T22:36:40.940 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:36:40.940 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_fsid": "c2276428-ed51-11ee-95d0-87774f69a715", 2024-03-28T22:36:40.940 INFO:teuthology.orchestra.run.smithi043.stdout: "whoami": 3, 2024-03-28T22:36:40.940 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 24523, 2024-03-28T22:36:40.940 INFO:teuthology.orchestra.run.smithi043.stdout: "want_state": "up:active", 2024-03-28T22:36:40.940 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "up:active", 2024-03-28T22:36:40.940 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs", 2024-03-28T22:36:40.940 INFO:teuthology.orchestra.run.smithi043.stdout: "rank_uptime": 156.78069936, 2024-03-28T22:36:40.940 INFO:teuthology.orchestra.run.smithi043.stdout: "mdsmap_epoch": 28, 2024-03-28T22:36:40.940 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch": 96, 2024-03-28T22:36:40.941 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch_barrier": 92, 2024-03-28T22:36:40.941 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 218.20441823600001 2024-03-28T22:36:40.941 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:36:40.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.935+0000 7f7ceb200640 1 -- 172.21.15.43:0/3090126976 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7cc80a1e00 msgr2=0x7f7cc80a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:40.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.935+0000 7f7ceb200640 1 --2- 172.21.15.43:0/3090126976 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7cc80a1e00 0x7f7cc80a42c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4097270 tx=0x7f7cd40d1f10 comp rx=0 tx=0).stop 2024-03-28T22:36:40.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.935+0000 7f7ceb200640 1 -- 172.21.15.43:0/3090126976 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ce4103990 msgr2=0x7f7ce4199d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:40.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.935+0000 7f7ceb200640 1 --2- 172.21.15.43:0/3090126976 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ce4103990 0x7f7ce4199d50 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7cd400e9c0 tx=0x7f7cd400ee80 comp rx=0 tx=0).stop 2024-03-28T22:36:40.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.935+0000 7f7ceb200640 1 -- 172.21.15.43:0/3090126976 shutdown_connections 2024-03-28T22:36:40.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.935+0000 7f7ceb200640 1 --2- 172.21.15.43:0/3090126976 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ce4105540 0x7f7ce41a1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.935+0000 7f7ceb200640 1 --2- 172.21.15.43:0/3090126976 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ce4104b80 0x7f7ce419a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.935+0000 7f7ceb200640 1 --2- 172.21.15.43:0/3090126976 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7cc80a1e00 0x7f7cc80a42c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.935+0000 7f7ceb200640 1 --2- 172.21.15.43:0/3090126976 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ce4103990 0x7f7ce4199d50 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:40.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.935+0000 7f7ceb200640 1 -- 172.21.15.43:0/3090126976 >> 172.21.15.43:0/3090126976 conn(0x7f7ce40fef00 msgr2=0x7f7ce4107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:40.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.935+0000 7f7ceb200640 1 -- 172.21.15.43:0/3090126976 shutdown_connections 2024-03-28T22:36:40.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:40.935+0000 7f7ceb200640 1 -- 172.21.15.43:0/3090126976 wait complete. 2024-03-28T22:36:40.950 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:40 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3930219158' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:36:40.950 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:40 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/71567114' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:36:40.953 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-28T22:36:41.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:40 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3930219158' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:36:41.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:40 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/71567114' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:36:41.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.162+0000 7f3d0fa00640 1 -- 172.21.15.43:0/2454939950 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d08105540 msgr2=0x7f3d0810bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:41.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.162+0000 7f3d0fa00640 1 --2- 172.21.15.43:0/2454939950 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d08105540 0x7f3d0810bdd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3d00009ca0 tx=0x7f3d0002c100 comp rx=0 tx=0).stop 2024-03-28T22:36:41.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.162+0000 7f3d0fa00640 1 -- 172.21.15.43:0/2454939950 shutdown_connections 2024-03-28T22:36:41.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.162+0000 7f3d0fa00640 1 --2- 172.21.15.43:0/2454939950 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d08104b80 0x7f3d08105000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.162+0000 7f3d0fa00640 1 --2- 172.21.15.43:0/2454939950 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d08105540 0x7f3d0810bdd0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.162+0000 7f3d0fa00640 1 --2- 172.21.15.43:0/2454939950 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3d08103990 0x7f3d08103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.162+0000 7f3d0fa00640 1 -- 172.21.15.43:0/2454939950 >> 172.21.15.43:0/2454939950 conn(0x7f3d080fef00 msgr2=0x7f3d08101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:41.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.162+0000 7f3d0fa00640 1 -- 172.21.15.43:0/2454939950 shutdown_connections 2024-03-28T22:36:41.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.162+0000 7f3d0fa00640 1 -- 172.21.15.43:0/2454939950 wait complete. 2024-03-28T22:36:41.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.163+0000 7f3d0fa00640 1 Processor -- start 2024-03-28T22:36:41.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.163+0000 7f3d0fa00640 1 -- start start 2024-03-28T22:36:41.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.163+0000 7f3d0fa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3d08103990 0x7f3d08199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.163+0000 7f3d0fa00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d08104b80 0x7f3d0819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.164+0000 7f3d0fa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d08105540 0x7f3d081a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.164+0000 7f3d0fa00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3d0810f610 con 0x7f3d08103990 2024-03-28T22:36:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.164+0000 7f3d0fa00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f3d0810f490 con 0x7f3d08104b80 2024-03-28T22:36:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.164+0000 7f3d0fa00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f3d0810f790 con 0x7f3d08105540 2024-03-28T22:36:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.164+0000 7f3d0d000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3d08103990 0x7f3d08199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.164+0000 7f3d07e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d08104b80 0x7f3d0819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.164+0000 7f3d0da00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d08105540 0x7f3d081a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.164+0000 7f3d07e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d08104b80 0x7f3d0819a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47340/0 (socket says 172.21.15.43:47340) 2024-03-28T22:36:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.164+0000 7f3d07e00640 1 -- 172.21.15.43:0/2380988916 learned_addr learned my addr 172.21.15.43:0/2380988916 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.164+0000 7f3d0d000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3d08103990 0x7f3d08199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48464/0 (socket says 172.21.15.43:48464) 2024-03-28T22:36:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.164+0000 7f3d07e00640 1 -- 172.21.15.43:0/2380988916 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d08105540 msgr2=0x7f3d081a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.164+0000 7f3d07e00640 1 --2- 172.21.15.43:0/2380988916 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d08105540 0x7f3d081a11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.164+0000 7f3d07e00640 1 -- 172.21.15.43:0/2380988916 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3d08103990 msgr2=0x7f3d08199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.164+0000 7f3d07e00640 1 --2- 172.21.15.43:0/2380988916 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3d08103990 0x7f3d08199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.164+0000 7f3d07e00640 1 -- 172.21.15.43:0/2380988916 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d081a1890 con 0x7f3d08104b80 2024-03-28T22:36:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.164+0000 7f3d0da00640 1 --2- 172.21.15.43:0/2380988916 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d08105540 0x7f3d081a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:41.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.165+0000 7f3d07e00640 1 --2- 172.21.15.43:0/2380988916 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d08104b80 0x7f3d0819a1c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3cf400e9f0 tx=0x7f3cf400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:41.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.165+0000 7f3d05600640 1 -- 172.21.15.43:0/2380988916 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf400cd80 con 0x7f3d08104b80 2024-03-28T22:36:41.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.165+0000 7f3d05600640 1 -- 172.21.15.43:0/2380988916 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3cf4005500 con 0x7f3d08104b80 2024-03-28T22:36:41.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.165+0000 7f3d05600640 1 -- 172.21.15.43:0/2380988916 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf4010690 con 0x7f3d08104b80 2024-03-28T22:36:41.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.165+0000 7f3d0fa00640 1 -- 172.21.15.43:0/2380988916 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d081a1b80 con 0x7f3d08104b80 2024-03-28T22:36:41.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.165+0000 7f3d0fa00640 1 -- 172.21.15.43:0/2380988916 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d081a2080 con 0x7f3d08104b80 2024-03-28T22:36:41.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.168+0000 7f3d12817740 1 Processor -- start 2024-03-28T22:36:41.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.168+0000 7f3d12817740 1 -- start start 2024-03-28T22:36:41.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.169+0000 7f3d12817740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5609d1659570 0x5609d1659990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.169+0000 7f3d12817740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5609d1659f60 0x5609d165ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.169+0000 7f3d12817740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609d165f0d0 0x5609d1661560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.169+0000 7f3d12817740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5609d15bd9c0 con 0x5609d165f0d0 2024-03-28T22:36:41.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.169+0000 7f3d12817740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5609d1566c30 con 0x5609d1659f60 2024-03-28T22:36:41.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.169+0000 7f3d12817740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5609d15b9a60 con 0x5609d1659570 2024-03-28T22:36:41.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.169+0000 7f3d0da00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5609d1659570 0x5609d1659990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.169+0000 7f3d05600640 1 -- 172.21.15.43:0/2380988916 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f3cf40046c0 con 0x7f3d08104b80 2024-03-28T22:36:41.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.169+0000 7f3d0da00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5609d1659570 0x5609d1659990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:53824/0 (socket says 172.21.15.43:53824) 2024-03-28T22:36:41.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.169+0000 7f3d0da00640 1 -- 172.21.15.43:0/3833214274 learned_addr learned my addr 172.21.15.43:0/3833214274 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:41.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.169+0000 7f3d0d000640 1 --2- 172.21.15.43:0/3833214274 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5609d1659f60 0x5609d165ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.169+0000 7f3d07e00640 1 --2- 172.21.15.43:0/3833214274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609d165f0d0 0x5609d1661560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.169+0000 7f3d0da00640 1 -- 172.21.15.43:0/3833214274 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5609d1659f60 msgr2=0x5609d165ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:41.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.169+0000 7f3d0da00640 1 --2- 172.21.15.43:0/3833214274 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5609d1659f60 0x5609d165ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.169+0000 7f3d0da00640 1 -- 172.21.15.43:0/3833214274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609d165f0d0 msgr2=0x5609d1661560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:41.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.169+0000 7f3d0da00640 1 --2- 172.21.15.43:0/3833214274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609d165f0d0 0x5609d1661560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.169+0000 7f3d0da00640 1 -- 172.21.15.43:0/3833214274 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609d12d22d0 con 0x5609d1659570 2024-03-28T22:36:41.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.169+0000 7f3d0d000640 1 --2- 172.21.15.43:0/3833214274 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5609d1659f60 0x5609d165ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:36:41.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.169+0000 7f3d07e00640 1 --2- 172.21.15.43:0/3833214274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609d165f0d0 0x5609d1661560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:36:41.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.170+0000 7f3d0da00640 1 --2- 172.21.15.43:0/3833214274 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5609d1659570 0x5609d1659990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3d0002c710 tx=0x7f3d000323d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=663161334f04e2cf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:41.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.170+0000 7f3cfa000640 1 -- 172.21.15.43:0/3833214274 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d00043070 con 0x5609d1659570 2024-03-28T22:36:41.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.170+0000 7f3cfa000640 1 -- 172.21.15.43:0/3833214274 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d00032c90 con 0x5609d1659570 2024-03-28T22:36:41.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.170+0000 7f3cfa000640 1 -- 172.21.15.43:0/3833214274 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d00039040 con 0x5609d1659570 2024-03-28T22:36:41.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.171+0000 7f3d05600640 1 --2- 172.21.15.43:0/2380988916 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3cf00a1d80 0x7f3cf00a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.171+0000 7f3d12817740 1 -- 172.21.15.43:0/3833214274 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5609d1659570 msgr2=0x5609d1659990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:41.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.171+0000 7f3d12817740 1 --2- 172.21.15.43:0/3833214274 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5609d1659570 0x5609d1659990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3d0002c710 tx=0x7f3d000323d0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.171+0000 7f3d05600640 1 -- 172.21.15.43:0/2380988916 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f3cf40ce620 con 0x7f3d08104b80 2024-03-28T22:36:41.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.171+0000 7f3d12817740 1 -- 172.21.15.43:0/3833214274 shutdown_connections 2024-03-28T22:36:41.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.171+0000 7f3d12817740 1 --2- 172.21.15.43:0/3833214274 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5609d1659f60 0x5609d165ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.171+0000 7f3d12817740 1 --2- 172.21.15.43:0/3833214274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609d165f0d0 0x5609d1661560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.171+0000 7f3d12817740 1 --2- 172.21.15.43:0/3833214274 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5609d1659570 0x5609d1659990 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.171+0000 7f3d0d000640 1 --2- 172.21.15.43:0/2380988916 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3cf00a1d80 0x7f3cf00a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.171+0000 7f3d12817740 1 -- 172.21.15.43:0/3833214274 >> 172.21.15.43:0/3833214274 conn(0x5609d1654ae0 msgr2=0x5609d1654ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:41.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.171+0000 7f3d12817740 1 -- 172.21.15.43:0/3833214274 shutdown_connections 2024-03-28T22:36:41.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.171+0000 7f3d12817740 1 -- 172.21.15.43:0/3833214274 wait complete. 2024-03-28T22:36:41.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.171+0000 7f3d12817740 1 -- start start 2024-03-28T22:36:41.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.172+0000 7f3d0d000640 1 --2- 172.21.15.43:0/2380988916 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3cf00a1d80 0x7f3cf00a4240 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x5609d1655a50 tx=0x7f3cfc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:41.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.172+0000 7f3d12817740 1 Processor -- start 2024-03-28T22:36:41.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.172+0000 7f3d12817740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609d1659570 0x5609d1663ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.172+0000 7f3d12817740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5609d1659f60 0x5609d1663ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.172+0000 7f3d12817740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5609d165f0d0 0x5609d166b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.172+0000 7f3d12817740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5609d1662020 con 0x5609d1659570 2024-03-28T22:36:41.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.172+0000 7f3d12817740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5609d155c180 con 0x5609d1659f60 2024-03-28T22:36:41.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.172+0000 7f3d12817740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5609d16621a0 con 0x5609d165f0d0 2024-03-28T22:36:41.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.172+0000 7f3d12817740 10 client.0 _ll_register_callbacks cb 0x5609d12d1ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:36:41.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.172+0000 7f3d0d000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609d1659570 0x5609d1663ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.173+0000 7f3d07e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5609d1659f60 0x5609d1663ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.173+0000 7f3d0d000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609d1659570 0x5609d1663ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48492/0 (socket says 172.21.15.43:48492) 2024-03-28T22:36:41.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.173+0000 7f3d0d000640 1 -- 172.21.15.43:0/2179134219 learned_addr learned my addr 172.21.15.43:0/2179134219 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:41.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.173+0000 7f3d0da00640 1 --2- 172.21.15.43:0/2179134219 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5609d165f0d0 0x5609d166b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.173+0000 7f3d07e00640 1 -- 172.21.15.43:0/2179134219 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5609d165f0d0 msgr2=0x5609d166b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:41.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.173+0000 7f3d07e00640 1 --2- 172.21.15.43:0/2179134219 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5609d165f0d0 0x5609d166b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.173+0000 7f3d07e00640 1 -- 172.21.15.43:0/2179134219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609d1659570 msgr2=0x5609d1663ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:41.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.173+0000 7f3d07e00640 1 --2- 172.21.15.43:0/2179134219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609d1659570 0x5609d1663ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.173+0000 7f3d07e00640 1 -- 172.21.15.43:0/2179134219 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609d139c6b0 con 0x5609d1659f60 2024-03-28T22:36:41.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.173+0000 7f3d0da00640 1 --2- 172.21.15.43:0/2179134219 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5609d165f0d0 0x5609d166b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:41.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.173+0000 7f3d07e00640 1 --2- 172.21.15.43:0/2179134219 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5609d1659f60 0x5609d1663ff0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3cf40cbd30 tx=0x7f3cf409ab60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93b4790a0c1d7df1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:41.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.173+0000 7f3ceea00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:36:41.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.173+0000 7f3ceea00640 1 -- 172.21.15.43:0/2179134219 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf40a3470 con 0x5609d1659f60 2024-03-28T22:36:41.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.174+0000 7f3ceea00640 1 -- 172.21.15.43:0/2179134219 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f3cf409a3b0 con 0x5609d1659f60 2024-03-28T22:36:41.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.174+0000 7f3d12817740 1 -- 172.21.15.43:0/2179134219 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d00032c90 con 0x5609d1659f60 2024-03-28T22:36:41.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.174+0000 7f3ceea00640 1 -- 172.21.15.43:0/2179134219 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3cf409af60 con 0x5609d1659f60 2024-03-28T22:36:41.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.174+0000 7f3d12817740 10 client.25415 fetch_fsmap learned FSMap version 28 2024-03-28T22:36:41.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.174+0000 7f3d12817740 1 -- 172.21.15.43:0/2179134219 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5609d166e380 con 0x5609d1659f60 2024-03-28T22:36:41.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.174+0000 7f3ceea00640 1 -- 172.21.15.43:0/2179134219 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d00032c90 con 0x5609d1659f60 2024-03-28T22:36:41.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.174+0000 7f3ceea00640 1 -- 172.21.15.43:0/2179134219 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3cf40b9240 con 0x5609d1659f60 2024-03-28T22:36:41.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.174+0000 7f3d12817740 10 client.25415 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T22:36:41.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.174+0000 7f3d12817740 10 client.25415 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-28T22:36:41.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.174+0000 7f3d12817740 20 client.25415 populate_metadata read hostname 'smithi043' 2024-03-28T22:36:41.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.174+0000 7f3d12817740 1 --2- 172.21.15.43:0/2179134219 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5609d1670960 0x5609d1671dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.174+0000 7f3d12817740 4 client.25415 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:36:41.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.174+0000 7f3d12817740 1 -- 172.21.15.43:0/2179134219 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5609d15caf00 con 0x5609d1670960 2024-03-28T22:36:41.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.175+0000 7f3d0d000640 1 --2- 172.21.15.43:0/2179134219 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5609d1670960 0x5609d1671dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.175+0000 7f3d0d000640 1 --2- 172.21.15.43:0/2179134219 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5609d1670960 0x5609d1671dd0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a6760d2586dcb547 server_cookie=ce0f8ea88682da10 in_seq=0 out_seq=0 2024-03-28T22:36:41.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.175+0000 7f3ceea00640 10 client.25415 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:36:41.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.176+0000 7f3ceea00640 1 -- 172.21.15.43:0/2179134219 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5609d15caf00 con 0x5609d1670960 2024-03-28T22:36:41.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.176+0000 7f3ceea00640 10 client.25415 handle_command_reply: tid=0 2024-03-28T22:36:41.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.176+0000 7f3d12817740 1 client.25415 shutdown 2024-03-28T22:36:41.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.176+0000 7f3d12817740 1 -- 172.21.15.43:0/2179134219 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5609d1659f60 msgr2=0x5609d1663ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:41.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.176+0000 7f3d12817740 1 --2- 172.21.15.43:0/2179134219 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5609d1659f60 0x5609d1663ff0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3cf40cbd30 tx=0x7f3cf409ab60 comp rx=0 tx=0).stop 2024-03-28T22:36:41.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.176+0000 7f3d12817740 1 -- 172.21.15.43:0/2179134219 shutdown_connections 2024-03-28T22:36:41.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.176+0000 7f3d12817740 1 --2- 172.21.15.43:0/2179134219 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5609d1670960 0x5609d1671dd0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.176+0000 7f3d12817740 1 --2- 172.21.15.43:0/2179134219 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5609d1659f60 0x5609d1663ff0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.176+0000 7f3d12817740 1 --2- 172.21.15.43:0/2179134219 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5609d165f0d0 0x5609d166b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.176+0000 7f3d12817740 1 --2- 172.21.15.43:0/2179134219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609d1659570 0x5609d1663ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.176+0000 7f3d12817740 1 -- 172.21.15.43:0/2179134219 >> 172.21.15.43:0/2179134219 conn(0x5609d1654ae0 msgr2=0x5609d1653ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:41.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.176+0000 7f3ceea00640 0 client.25415 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:36:41.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.176+0000 7f3d12817740 1 -- 172.21.15.43:0/2179134219 shutdown_connections 2024-03-28T22:36:41.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.176+0000 7f3d12817740 1 -- 172.21.15.43:0/2179134219 wait complete. 2024-03-28T22:36:41.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.177+0000 7f3d12817740 20 client.25415 trim_cache size 0 max 16384 2024-03-28T22:36:41.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.197+0000 7f3d12817740 1 Processor -- start 2024-03-28T22:36:41.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.197+0000 7f3d12817740 1 -- start start 2024-03-28T22:36:41.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.197+0000 7f3d12817740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609d1659570 0x5609d16636c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.197+0000 7f3d12817740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5609d1659f60 0x5609d166aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.197+0000 7f3d12817740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5609d165f0d0 0x5609d1663c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.197+0000 7f3d12817740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5609d1670110 con 0x5609d1659570 2024-03-28T22:36:41.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.197+0000 7f3d12817740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5609d16707e0 con 0x5609d165f0d0 2024-03-28T22:36:41.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.197+0000 7f3d12817740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5609d1670eb0 con 0x5609d1659f60 2024-03-28T22:36:41.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.197+0000 7f3d0da00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5609d165f0d0 0x5609d1663c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.197+0000 7f3d07e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5609d1659f60 0x5609d166aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.197+0000 7f3d0da00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5609d165f0d0 0x5609d1663c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47358/0 (socket says 172.21.15.43:47358) 2024-03-28T22:36:41.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.197+0000 7f3d0da00640 1 -- 172.21.15.43:0/1592188360 learned_addr learned my addr 172.21.15.43:0/1592188360 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:41.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.197+0000 7f3d0d000640 1 --2- 172.21.15.43:0/1592188360 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609d1659570 0x5609d16636c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.198+0000 7f3d0da00640 1 -- 172.21.15.43:0/1592188360 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5609d1659f60 msgr2=0x5609d166aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:41.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.198+0000 7f3d0da00640 1 --2- 172.21.15.43:0/1592188360 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5609d1659f60 0x5609d166aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.198+0000 7f3d0da00640 1 -- 172.21.15.43:0/1592188360 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609d1659570 msgr2=0x5609d16636c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:41.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.198+0000 7f3d0da00640 1 --2- 172.21.15.43:0/1592188360 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609d1659570 0x5609d16636c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.198+0000 7f3d0da00640 1 -- 172.21.15.43:0/1592188360 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609d165a340 con 0x5609d165f0d0 2024-03-28T22:36:41.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.198+0000 7f3d07e00640 1 --2- 172.21.15.43:0/1592188360 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5609d1659f60 0x5609d166aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:41.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.198+0000 7f3d0da00640 1 --2- 172.21.15.43:0/1592188360 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5609d165f0d0 0x5609d1663c00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f3d0002c710 tx=0x7f3d00005bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e536d49b50b389f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:41.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.198+0000 7f3cf9600640 1 -- 172.21.15.43:0/1592188360 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d00039030 con 0x5609d165f0d0 2024-03-28T22:36:41.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.198+0000 7f3cf9600640 1 -- 172.21.15.43:0/1592188360 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d00003e50 con 0x5609d165f0d0 2024-03-28T22:36:41.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.198+0000 7f3cf9600640 1 -- 172.21.15.43:0/1592188360 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d00043040 con 0x5609d165f0d0 2024-03-28T22:36:41.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.199+0000 7f3d12817740 1 -- 172.21.15.43:0/1592188360 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5609d165f0d0 msgr2=0x5609d1663c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:41.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.199+0000 7f3d12817740 1 --2- 172.21.15.43:0/1592188360 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5609d165f0d0 0x5609d1663c00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f3d0002c710 tx=0x7f3d00005bf0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.199+0000 7f3d12817740 1 -- 172.21.15.43:0/1592188360 shutdown_connections 2024-03-28T22:36:41.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.199+0000 7f3d12817740 1 --2- 172.21.15.43:0/1592188360 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5609d165f0d0 0x5609d1663c00 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.199+0000 7f3d12817740 1 --2- 172.21.15.43:0/1592188360 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5609d1659f60 0x5609d166aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.199+0000 7f3d12817740 1 --2- 172.21.15.43:0/1592188360 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609d1659570 0x5609d16636c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.199+0000 7f3d12817740 1 -- 172.21.15.43:0/1592188360 >> 172.21.15.43:0/1592188360 conn(0x5609d1654ae0 msgr2=0x5609d1654ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:41.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.199+0000 7f3d12817740 1 -- 172.21.15.43:0/1592188360 shutdown_connections 2024-03-28T22:36:41.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.199+0000 7f3d12817740 1 -- 172.21.15.43:0/1592188360 wait complete. 2024-03-28T22:36:41.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.200+0000 7f3d12817740 1 -- start start 2024-03-28T22:36:41.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.200+0000 7f3d12817740 1 Processor -- start 2024-03-28T22:36:41.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:40 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3930219158' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:36:41.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:40 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/71567114' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:36:41.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.200+0000 7f3d12817740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5609d1659570 0x5609d1668fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.200+0000 7f3d12817740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5609d1659f60 0x5609d16d26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.200+0000 7f3d12817740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609d165f0d0 0x5609d16d4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.200+0000 7f3d12817740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5609d1671c50 con 0x5609d165f0d0 2024-03-28T22:36:41.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.200+0000 7f3d12817740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5609d1671580 con 0x5609d1659f60 2024-03-28T22:36:41.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.200+0000 7f3d12817740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5609d1672320 con 0x5609d1659570 2024-03-28T22:36:41.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.200+0000 7f3d12817740 10 client.0 _ll_register_callbacks cb 0x7f3cf400cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:36:41.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.200+0000 7f3d07e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5609d1659f60 0x5609d16d26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.200+0000 7f3d07e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5609d1659f60 0x5609d16d26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47362/0 (socket says 172.21.15.43:47362) 2024-03-28T22:36:41.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.200+0000 7f3d0da00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609d165f0d0 0x5609d16d4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.200+0000 7f3d07e00640 1 -- 172.21.15.43:0/519199076 learned_addr learned my addr 172.21.15.43:0/519199076 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:41.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.200+0000 7f3d0d000640 1 --2- 172.21.15.43:0/519199076 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5609d1659570 0x5609d1668fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.201+0000 7f3d07e00640 1 -- 172.21.15.43:0/519199076 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5609d1659570 msgr2=0x5609d1668fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:41.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.201+0000 7f3d07e00640 1 --2- 172.21.15.43:0/519199076 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5609d1659570 0x5609d1668fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.201+0000 7f3d07e00640 1 -- 172.21.15.43:0/519199076 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609d165f0d0 msgr2=0x5609d16d4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:41.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.201+0000 7f3d07e00640 1 --2- 172.21.15.43:0/519199076 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609d165f0d0 0x5609d16d4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.201+0000 7f3d07e00640 1 -- 172.21.15.43:0/519199076 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609d16d5140 con 0x5609d1659f60 2024-03-28T22:36:41.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.201+0000 7f3d07e00640 1 --2- 172.21.15.43:0/519199076 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5609d1659f60 0x5609d16d26b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4004c00 tx=0x7f3cf40ce910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f9b8980ba55318d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:41.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.201+0000 7f3ceea00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:36:41.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.201+0000 7f3ceea00640 1 -- 172.21.15.43:0/519199076 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf40990a0 con 0x5609d1659f60 2024-03-28T22:36:41.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.201+0000 7f3ceea00640 1 -- 172.21.15.43:0/519199076 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f3cf4096c20 con 0x5609d1659f60 2024-03-28T22:36:41.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.201+0000 7f3d12817740 1 -- 172.21.15.43:0/519199076 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d00003e50 con 0x5609d1659f60 2024-03-28T22:36:41.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.201+0000 7f3ceea00640 1 -- 172.21.15.43:0/519199076 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3cf4097170 con 0x5609d1659f60 2024-03-28T22:36:41.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.202+0000 7f3d12817740 10 client.25421 fetch_fsmap learned FSMap version 28 2024-03-28T22:36:41.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.202+0000 7f3d12817740 1 -- 172.21.15.43:0/519199076 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5609d16d6410 con 0x5609d1659f60 2024-03-28T22:36:41.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.202+0000 7f3ceea00640 1 -- 172.21.15.43:0/519199076 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d00003e50 con 0x5609d1659f60 2024-03-28T22:36:41.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.202+0000 7f3ceea00640 1 -- 172.21.15.43:0/519199076 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3cf4096e20 con 0x5609d1659f60 2024-03-28T22:36:41.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.202+0000 7f3d12817740 10 client.25421 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T22:36:41.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.202+0000 7f3d12817740 10 client.25421 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-28T22:36:41.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.202+0000 7f3d12817740 20 client.25421 populate_metadata read hostname 'smithi043' 2024-03-28T22:36:41.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.202+0000 7f3d12817740 1 --2- 172.21.15.43:0/519199076 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5609d16db020 0x5609d16d9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.202+0000 7f3d12817740 4 client.25421 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T22:36:41.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.202+0000 7f3d12817740 1 -- 172.21.15.43:0/519199076 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "status"}) -- 0x5609d15c5250 con 0x5609d16db020 2024-03-28T22:36:41.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.202+0000 7f3d0d000640 1 --2- 172.21.15.43:0/519199076 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5609d16db020 0x5609d16d9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.203+0000 7f3d0d000640 1 --2- 172.21.15.43:0/519199076 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5609d16db020 0x5609d16d9d30 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=50db5c589fab549e server_cookie=55e247c59e242e48 in_seq=0 out_seq=0 2024-03-28T22:36:41.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.203+0000 7f3ceea00640 10 client.25421 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:36:41.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.203+0000 7f3ceea00640 1 -- 172.21.15.43:0/519199076 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5609d15c5250 con 0x5609d16db020 2024-03-28T22:36:41.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.203+0000 7f3ceea00640 10 client.25421 handle_command_reply: tid=0 2024-03-28T22:36:41.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.203+0000 7f3d12817740 1 client.25421 shutdown 2024-03-28T22:36:41.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.203+0000 7f3d12817740 1 -- 172.21.15.43:0/519199076 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5609d1659f60 msgr2=0x5609d16d26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:41.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.203+0000 7f3d12817740 1 --2- 172.21.15.43:0/519199076 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5609d1659f60 0x5609d16d26b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4004c00 tx=0x7f3cf40ce910 comp rx=0 tx=0).stop 2024-03-28T22:36:41.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.203+0000 7f3d12817740 1 -- 172.21.15.43:0/519199076 shutdown_connections 2024-03-28T22:36:41.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.203+0000 7f3d12817740 1 --2- 172.21.15.43:0/519199076 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5609d16db020 0x5609d16d9d30 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.203+0000 7f3d12817740 1 --2- 172.21.15.43:0/519199076 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5609d1659f60 0x5609d16d26b0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.203+0000 7f3d12817740 1 --2- 172.21.15.43:0/519199076 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5609d165f0d0 0x5609d16d4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.203+0000 7f3d12817740 1 --2- 172.21.15.43:0/519199076 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5609d1659570 0x5609d1668fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.203+0000 7f3d12817740 1 -- 172.21.15.43:0/519199076 >> 172.21.15.43:0/519199076 conn(0x5609d1654ae0 msgr2=0x5609d16636c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:41.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.203+0000 7f3ceea00640 0 client.25421 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:36:41.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.203+0000 7f3d12817740 1 -- 172.21.15.43:0/519199076 shutdown_connections 2024-03-28T22:36:41.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.203+0000 7f3d12817740 1 -- 172.21.15.43:0/519199076 wait complete. 2024-03-28T22:36:41.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.204+0000 7f3d12817740 20 client.25421 trim_cache size 0 max 16384 2024-03-28T22:36:41.207 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:36:41.207 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_fsid": "c2276428-ed51-11ee-95d0-87774f69a715", 2024-03-28T22:36:41.208 INFO:teuthology.orchestra.run.smithi043.stdout: "whoami": 2, 2024-03-28T22:36:41.208 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 24535, 2024-03-28T22:36:41.208 INFO:teuthology.orchestra.run.smithi043.stdout: "want_state": "up:active", 2024-03-28T22:36:41.208 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "up:active", 2024-03-28T22:36:41.208 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs", 2024-03-28T22:36:41.208 INFO:teuthology.orchestra.run.smithi043.stdout: "rank_uptime": 158.06158787000001, 2024-03-28T22:36:41.208 INFO:teuthology.orchestra.run.smithi043.stdout: "mdsmap_epoch": 28, 2024-03-28T22:36:41.208 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch": 96, 2024-03-28T22:36:41.208 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch_barrier": 92, 2024-03-28T22:36:41.208 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 215.500173725 2024-03-28T22:36:41.208 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:36:41.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.204+0000 7f3d0fa00640 1 -- 172.21.15.43:0/2380988916 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3cf00a1d80 msgr2=0x7f3cf00a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:41.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.204+0000 7f3d0fa00640 1 --2- 172.21.15.43:0/2380988916 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3cf00a1d80 0x7f3cf00a4240 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x5609d1655a50 tx=0x7f3cfc005f90 comp rx=0 tx=0).stop 2024-03-28T22:36:41.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.204+0000 7f3d0fa00640 1 -- 172.21.15.43:0/2380988916 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d08104b80 msgr2=0x7f3d0819a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:41.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.204+0000 7f3d0fa00640 1 --2- 172.21.15.43:0/2380988916 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d08104b80 0x7f3d0819a1c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3cf400e9f0 tx=0x7f3cf400edf0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.204+0000 7f3d0fa00640 1 -- 172.21.15.43:0/2380988916 shutdown_connections 2024-03-28T22:36:41.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.204+0000 7f3d0fa00640 1 --2- 172.21.15.43:0/2380988916 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3cf00a1d80 0x7f3cf00a4240 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.204+0000 7f3d0fa00640 1 --2- 172.21.15.43:0/2380988916 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d08104b80 0x7f3d0819a1c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.204+0000 7f3d0fa00640 1 --2- 172.21.15.43:0/2380988916 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3d08105540 0x7f3d081a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.204+0000 7f3d0fa00640 1 --2- 172.21.15.43:0/2380988916 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3d08103990 0x7f3d08199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.204+0000 7f3d0fa00640 1 -- 172.21.15.43:0/2380988916 >> 172.21.15.43:0/2380988916 conn(0x7f3d080fef00 msgr2=0x7f3d08107e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:41.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.204+0000 7f3d0fa00640 1 -- 172.21.15.43:0/2380988916 shutdown_connections 2024-03-28T22:36:41.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.204+0000 7f3d0fa00640 1 -- 172.21.15.43:0/2380988916 wait complete. 2024-03-28T22:36:41.220 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-28T22:36:41.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.447+0000 7f8182600640 1 -- 172.21.15.43:0/2019422758 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f817c107c70 msgr2=0x7f817c108090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:41.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.447+0000 7f8182600640 1 --2- 172.21.15.43:0/2019422758 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f817c107c70 0x7f817c108090 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8170009900 tx=0x7f817002bc30 comp rx=0 tx=0).stop 2024-03-28T22:36:41.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.447+0000 7f8182600640 1 -- 172.21.15.43:0/2019422758 shutdown_connections 2024-03-28T22:36:41.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.447+0000 7f8182600640 1 --2- 172.21.15.43:0/2019422758 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f817c109820 0x7f817c1100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.447+0000 7f8182600640 1 --2- 172.21.15.43:0/2019422758 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f817c108e60 0x7f817c1092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.447+0000 7f8182600640 1 --2- 172.21.15.43:0/2019422758 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f817c107c70 0x7f817c108090 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.447+0000 7f8182600640 1 -- 172.21.15.43:0/2019422758 >> 172.21.15.43:0/2019422758 conn(0x7f817c076c20 msgr2=0x7f817c079040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:41.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.447+0000 7f8182600640 1 -- 172.21.15.43:0/2019422758 shutdown_connections 2024-03-28T22:36:41.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.447+0000 7f8182600640 1 -- 172.21.15.43:0/2019422758 wait complete. 2024-03-28T22:36:41.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.448+0000 7f8182600640 1 Processor -- start 2024-03-28T22:36:41.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.448+0000 7f8182600640 1 -- start start 2024-03-28T22:36:41.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.448+0000 7f8182600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f817c107c70 0x7f817c19df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.448+0000 7f8182600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f817c108e60 0x7f817c19e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.449+0000 7f8182600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f817c109820 0x7f817c1a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.449+0000 7f8182600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f817c113940 con 0x7f817c107c70 2024-03-28T22:36:41.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.449+0000 7f8182600640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f817c1137c0 con 0x7f817c109820 2024-03-28T22:36:41.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.449+0000 7f8182600640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f817c113ac0 con 0x7f817c108e60 2024-03-28T22:36:41.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.449+0000 7f817be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f817c108e60 0x7f817c19e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.449+0000 7f8181c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f817c109820 0x7f817c1a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.449+0000 7f817be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f817c108e60 0x7f817c19e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:53856/0 (socket says 172.21.15.43:53856) 2024-03-28T22:36:41.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.449+0000 7f817be00640 1 -- 172.21.15.43:0/3043704511 learned_addr learned my addr 172.21.15.43:0/3043704511 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:41.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.449+0000 7f8181200640 1 --2- 172.21.15.43:0/3043704511 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f817c107c70 0x7f817c19df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.449+0000 7f817be00640 1 -- 172.21.15.43:0/3043704511 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f817c109820 msgr2=0x7f817c1a5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:41.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.449+0000 7f817be00640 1 --2- 172.21.15.43:0/3043704511 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f817c109820 0x7f817c1a5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.449+0000 7f817be00640 1 -- 172.21.15.43:0/3043704511 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f817c107c70 msgr2=0x7f817c19df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:41.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.449+0000 7f817be00640 1 --2- 172.21.15.43:0/3043704511 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f817c107c70 0x7f817c19df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.449+0000 7f817be00640 1 -- 172.21.15.43:0/3043704511 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f817c1a5b70 con 0x7f817c108e60 2024-03-28T22:36:41.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.449+0000 7f8181c00640 1 --2- 172.21.15.43:0/3043704511 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f817c109820 0x7f817c1a5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:41.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.449+0000 7f817be00640 1 --2- 172.21.15.43:0/3043704511 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f817c108e60 0x7f817c19e490 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f816800e9f0 tx=0x7f816800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:41.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.450+0000 7f8179600640 1 -- 172.21.15.43:0/3043704511 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816800cd80 con 0x7f817c108e60 2024-03-28T22:36:41.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.450+0000 7f8179600640 1 -- 172.21.15.43:0/3043704511 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8168010910 con 0x7f817c108e60 2024-03-28T22:36:41.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.450+0000 7f8179600640 1 -- 172.21.15.43:0/3043704511 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8168018990 con 0x7f817c108e60 2024-03-28T22:36:41.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.450+0000 7f8182600640 1 -- 172.21.15.43:0/3043704511 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f817c1a5e60 con 0x7f817c108e60 2024-03-28T22:36:41.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.450+0000 7f8182600640 1 -- 172.21.15.43:0/3043704511 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f817c1a62d0 con 0x7f817c108e60 2024-03-28T22:36:41.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.453+0000 7f8179600640 1 -- 172.21.15.43:0/3043704511 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f8168010ac0 con 0x7f817c108e60 2024-03-28T22:36:41.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.453+0000 7f81867e9740 1 Processor -- start 2024-03-28T22:36:41.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.453+0000 7f81867e9740 1 -- start start 2024-03-28T22:36:41.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.454+0000 7f81867e9740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559cb9a40740 0x559cb9a40b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.454+0000 7f81867e9740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559cb9a41130 0x559cb9a45c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.454+0000 7f81867e9740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cb9a462a0 0x559cb9a48730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.454+0000 7f81867e9740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x559cb99a49c0 con 0x559cb9a40740 2024-03-28T22:36:41.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.454+0000 7f81867e9740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x559cb994dc30 con 0x559cb9a462a0 2024-03-28T22:36:41.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.454+0000 7f81867e9740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x559cb99a0a60 con 0x559cb9a41130 2024-03-28T22:36:41.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.454+0000 7f8181200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559cb9a41130 0x559cb9a45c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.454+0000 7f817be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cb9a462a0 0x559cb9a48730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.454+0000 7f8181200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559cb9a41130 0x559cb9a45c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:53868/0 (socket says 172.21.15.43:53868) 2024-03-28T22:36:41.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.454+0000 7f8181200640 1 -- 172.21.15.43:0/4029885522 learned_addr learned my addr 172.21.15.43:0/4029885522 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:41.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.454+0000 7f8181c00640 1 --2- 172.21.15.43:0/4029885522 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559cb9a40740 0x559cb9a40b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.454+0000 7f8181200640 1 -- 172.21.15.43:0/4029885522 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cb9a462a0 msgr2=0x559cb9a48730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:41.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.454+0000 7f8181200640 1 --2- 172.21.15.43:0/4029885522 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cb9a462a0 0x559cb9a48730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.454+0000 7f8181200640 1 -- 172.21.15.43:0/4029885522 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559cb9a40740 msgr2=0x559cb9a40b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:41.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.454+0000 7f8181200640 1 --2- 172.21.15.43:0/4029885522 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559cb9a40740 0x559cb9a40b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.454+0000 7f8181200640 1 -- 172.21.15.43:0/4029885522 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559cb96b92d0 con 0x559cb9a41130 2024-03-28T22:36:41.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.455+0000 7f8179600640 1 --2- 172.21.15.43:0/3043704511 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f81640a1e00 0x7f81640a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.455+0000 7f8181200640 1 --2- 172.21.15.43:0/4029885522 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559cb9a41130 0x559cb9a45c30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8170000c00 tx=0x7f8170009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce8e5b308dd391e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:41.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.455+0000 7f8179600640 1 -- 172.21.15.43:0/3043704511 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f81680ce3e0 con 0x7f817c108e60 2024-03-28T22:36:41.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.455+0000 7f8181c00640 1 --2- 172.21.15.43:0/3043704511 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f81640a1e00 0x7f81640a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.455+0000 7f816e000640 1 -- 172.21.15.43:0/4029885522 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f817002c980 con 0x559cb9a41130 2024-03-28T22:36:41.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.455+0000 7f816e000640 1 -- 172.21.15.43:0/4029885522 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f817002cb20 con 0x559cb9a41130 2024-03-28T22:36:41.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.455+0000 7f816e000640 1 -- 172.21.15.43:0/4029885522 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81700046c0 con 0x559cb9a41130 2024-03-28T22:36:41.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.456+0000 7f81867e9740 1 -- 172.21.15.43:0/4029885522 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559cb9a41130 msgr2=0x559cb9a45c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:41.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.456+0000 7f8181c00640 1 --2- 172.21.15.43:0/3043704511 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f81640a1e00 0x7f81640a42c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x559cb9a41990 tx=0x7f8174009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:41.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.456+0000 7f81867e9740 1 --2- 172.21.15.43:0/4029885522 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559cb9a41130 0x559cb9a45c30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8170000c00 tx=0x7f8170009560 comp rx=0 tx=0).stop 2024-03-28T22:36:41.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.456+0000 7f81867e9740 1 -- 172.21.15.43:0/4029885522 shutdown_connections 2024-03-28T22:36:41.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.456+0000 7f81867e9740 1 --2- 172.21.15.43:0/4029885522 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cb9a462a0 0x559cb9a48730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.456+0000 7f81867e9740 1 --2- 172.21.15.43:0/4029885522 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559cb9a41130 0x559cb9a45c30 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.456+0000 7f81867e9740 1 --2- 172.21.15.43:0/4029885522 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559cb9a40740 0x559cb9a40b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.456+0000 7f81867e9740 1 -- 172.21.15.43:0/4029885522 >> 172.21.15.43:0/4029885522 conn(0x559cb9a3bcb0 msgr2=0x559cb9a3c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:41.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.456+0000 7f81867e9740 1 -- 172.21.15.43:0/4029885522 shutdown_connections 2024-03-28T22:36:41.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.456+0000 7f81867e9740 1 -- 172.21.15.43:0/4029885522 wait complete. 2024-03-28T22:36:41.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.457+0000 7f81867e9740 1 -- start start 2024-03-28T22:36:41.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.457+0000 7f81867e9740 1 Processor -- start 2024-03-28T22:36:41.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.457+0000 7f81867e9740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cb9a40740 0x559cb9a4abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.457+0000 7f81867e9740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559cb9a41130 0x559cb9a4b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.457+0000 7f81867e9740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559cb9a462a0 0x559cb9a52140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.457+0000 7f81867e9740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x559cb9a491f0 con 0x559cb9a462a0 2024-03-28T22:36:41.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.457+0000 7f81867e9740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x559cb9943180 con 0x559cb9a40740 2024-03-28T22:36:41.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.457+0000 7f81867e9740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x559cb9a49370 con 0x559cb9a41130 2024-03-28T22:36:41.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.457+0000 7f81867e9740 10 client.0 _ll_register_callbacks cb 0x559cb96b8ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:36:41.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.457+0000 7f8181200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cb9a40740 0x559cb9a4abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.458+0000 7f8181c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559cb9a462a0 0x559cb9a52140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.458+0000 7f8181200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cb9a40740 0x559cb9a4abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47416/0 (socket says 172.21.15.43:47416) 2024-03-28T22:36:41.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.458+0000 7f8181200640 1 -- 172.21.15.43:0/1607818904 learned_addr learned my addr 172.21.15.43:0/1607818904 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:41.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.458+0000 7f817be00640 1 --2- 172.21.15.43:0/1607818904 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559cb9a41130 0x559cb9a4b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.458+0000 7f8181200640 1 -- 172.21.15.43:0/1607818904 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559cb9a41130 msgr2=0x559cb9a4b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:41.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.458+0000 7f8181200640 1 --2- 172.21.15.43:0/1607818904 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559cb9a41130 0x559cb9a4b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.458+0000 7f8181200640 1 -- 172.21.15.43:0/1607818904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559cb9a462a0 msgr2=0x559cb9a52140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:41.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.458+0000 7f8181200640 1 --2- 172.21.15.43:0/1607818904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559cb9a462a0 0x559cb9a52140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.458+0000 7f8181200640 1 -- 172.21.15.43:0/1607818904 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559cb97836b0 con 0x559cb9a40740 2024-03-28T22:36:41.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.458+0000 7f8181200640 1 --2- 172.21.15.43:0/1607818904 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cb9a40740 0x559cb9a4abc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f817002c240 tx=0x7f817003f9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35e047915777b15 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:41.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.458+0000 7f8162a00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:36:41.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.458+0000 7f8162a00640 1 -- 172.21.15.43:0/1607818904 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f817002c980 con 0x559cb9a40740 2024-03-28T22:36:41.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.458+0000 7f8162a00640 1 -- 172.21.15.43:0/1607818904 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f817003fdd0 con 0x559cb9a40740 2024-03-28T22:36:41.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.458+0000 7f81867e9740 1 -- 172.21.15.43:0/1607818904 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f817002cb20 con 0x559cb9a40740 2024-03-28T22:36:41.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.459+0000 7f8162a00640 1 -- 172.21.15.43:0/1607818904 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8170059050 con 0x559cb9a40740 2024-03-28T22:36:41.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.459+0000 7f81867e9740 10 client.25433 fetch_fsmap learned FSMap version 28 2024-03-28T22:36:41.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.459+0000 7f81867e9740 1 -- 172.21.15.43:0/1607818904 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559cb9a554b0 con 0x559cb9a40740 2024-03-28T22:36:41.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.459+0000 7f8162a00640 1 -- 172.21.15.43:0/1607818904 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f817002cb20 con 0x559cb9a40740 2024-03-28T22:36:41.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.459+0000 7f8162a00640 1 -- 172.21.15.43:0/1607818904 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8170054740 con 0x559cb9a40740 2024-03-28T22:36:41.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.459+0000 7f81867e9740 10 client.25433 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T22:36:41.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.459+0000 7f81867e9740 10 client.25433 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T22:36:41.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.459+0000 7f81867e9740 20 client.25433 populate_metadata read hostname 'smithi043' 2024-03-28T22:36:41.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.459+0000 7f81867e9740 1 --2- 172.21.15.43:0/1607818904 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x559cb9a576e0 0x559cb9a58b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.459+0000 7f81867e9740 4 client.25433 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:36:41.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.459+0000 7f81867e9740 1 -- 172.21.15.43:0/1607818904 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559cb99b1f00 con 0x559cb9a576e0 2024-03-28T22:36:41.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.460+0000 7f817be00640 1 --2- 172.21.15.43:0/1607818904 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x559cb9a576e0 0x559cb9a58b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.460+0000 7f817be00640 1 --2- 172.21.15.43:0/1607818904 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x559cb9a576e0 0x559cb9a58b50 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=86949b88728e248c server_cookie=24f5148298575e6e in_seq=0 out_seq=0 2024-03-28T22:36:41.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.460+0000 7f8162a00640 10 client.25433 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:36:41.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.461+0000 7f8162a00640 1 -- 172.21.15.43:0/1607818904 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559cb99b1f00 con 0x559cb9a576e0 2024-03-28T22:36:41.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.461+0000 7f8162a00640 10 client.25433 handle_command_reply: tid=0 2024-03-28T22:36:41.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.461+0000 7f81867e9740 1 client.25433 shutdown 2024-03-28T22:36:41.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.461+0000 7f81867e9740 1 -- 172.21.15.43:0/1607818904 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cb9a40740 msgr2=0x559cb9a4abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:41.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.461+0000 7f81867e9740 1 --2- 172.21.15.43:0/1607818904 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cb9a40740 0x559cb9a4abc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f817002c240 tx=0x7f817003f9e0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.461+0000 7f81867e9740 1 -- 172.21.15.43:0/1607818904 shutdown_connections 2024-03-28T22:36:41.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.461+0000 7f81867e9740 1 --2- 172.21.15.43:0/1607818904 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x559cb9a576e0 0x559cb9a58b50 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.461+0000 7f81867e9740 1 --2- 172.21.15.43:0/1607818904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559cb9a462a0 0x559cb9a52140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.461+0000 7f81867e9740 1 --2- 172.21.15.43:0/1607818904 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559cb9a41130 0x559cb9a4b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.461+0000 7f81867e9740 1 --2- 172.21.15.43:0/1607818904 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cb9a40740 0x559cb9a4abc0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.461+0000 7f81867e9740 1 -- 172.21.15.43:0/1607818904 >> 172.21.15.43:0/1607818904 conn(0x559cb9a3bcb0 msgr2=0x559cb9a3adf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:41.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.461+0000 7f8162a00640 0 client.25433 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:36:41.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.461+0000 7f81867e9740 1 -- 172.21.15.43:0/1607818904 shutdown_connections 2024-03-28T22:36:41.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.462+0000 7f81867e9740 1 -- 172.21.15.43:0/1607818904 wait complete. 2024-03-28T22:36:41.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.462+0000 7f81867e9740 20 client.25433 trim_cache size 0 max 16384 2024-03-28T22:36:41.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.482+0000 7f81867e9740 1 Processor -- start 2024-03-28T22:36:41.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.482+0000 7f81867e9740 1 -- start start 2024-03-28T22:36:41.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.482+0000 7f81867e9740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559cb9a40740 0x559cb9a48690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.482+0000 7f81867e9740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cb9a41130 0x559cb9a52100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.482+0000 7f81867e9740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559cb9a462a0 0x559cb9a3c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.482+0000 7f81867e9740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x559cb9a56e90 con 0x559cb9a462a0 2024-03-28T22:36:41.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.482+0000 7f81867e9740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x559cb9a57560 con 0x559cb9a41130 2024-03-28T22:36:41.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.482+0000 7f81867e9740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x559cb9a57c30 con 0x559cb9a40740 2024-03-28T22:36:41.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.482+0000 7f8181200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559cb9a40740 0x559cb9a48690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.482+0000 7f817be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cb9a41130 0x559cb9a52100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.482+0000 7f817be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cb9a41130 0x559cb9a52100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47428/0 (socket says 172.21.15.43:47428) 2024-03-28T22:36:41.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.482+0000 7f8181c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559cb9a462a0 0x559cb9a3c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.482+0000 7f8181200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559cb9a40740 0x559cb9a48690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:53884/0 (socket says 172.21.15.43:53884) 2024-03-28T22:36:41.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.482+0000 7f817be00640 1 -- 172.21.15.43:0/976145712 learned_addr learned my addr 172.21.15.43:0/976145712 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:41.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.483+0000 7f817be00640 1 -- 172.21.15.43:0/976145712 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559cb9a40740 msgr2=0x559cb9a48690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:41.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.483+0000 7f817be00640 1 --2- 172.21.15.43:0/976145712 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559cb9a40740 0x559cb9a48690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.483+0000 7f817be00640 1 -- 172.21.15.43:0/976145712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559cb9a462a0 msgr2=0x559cb9a3c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:41.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.483+0000 7f817be00640 1 --2- 172.21.15.43:0/976145712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559cb9a462a0 0x559cb9a3c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.483+0000 7f817be00640 1 -- 172.21.15.43:0/976145712 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559cb9a45f70 con 0x559cb9a41130 2024-03-28T22:36:41.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.483+0000 7f8181200640 1 --2- 172.21.15.43:0/976145712 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559cb9a40740 0x559cb9a48690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:41.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.483+0000 7f817be00640 1 --2- 172.21.15.43:0/976145712 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cb9a41130 0x559cb9a52100 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f816800c6a0 tx=0x7f81680d1ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49b94a57209eeb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:41.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.483+0000 7f816d600640 1 -- 172.21.15.43:0/976145712 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816809a790 con 0x559cb9a41130 2024-03-28T22:36:41.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.483+0000 7f816d600640 1 -- 172.21.15.43:0/976145712 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f816809ae30 con 0x559cb9a41130 2024-03-28T22:36:41.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.483+0000 7f816d600640 1 -- 172.21.15.43:0/976145712 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81680974a0 con 0x559cb9a41130 2024-03-28T22:36:41.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.484+0000 7f81867e9740 1 -- 172.21.15.43:0/976145712 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cb9a41130 msgr2=0x559cb9a52100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:41.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.484+0000 7f81867e9740 1 --2- 172.21.15.43:0/976145712 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cb9a41130 0x559cb9a52100 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f816800c6a0 tx=0x7f81680d1ec0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.484+0000 7f81867e9740 1 -- 172.21.15.43:0/976145712 shutdown_connections 2024-03-28T22:36:41.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.484+0000 7f81867e9740 1 --2- 172.21.15.43:0/976145712 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cb9a41130 0x559cb9a52100 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.484+0000 7f81867e9740 1 --2- 172.21.15.43:0/976145712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559cb9a462a0 0x559cb9a3c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.484+0000 7f81867e9740 1 --2- 172.21.15.43:0/976145712 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559cb9a40740 0x559cb9a48690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.484+0000 7f81867e9740 1 -- 172.21.15.43:0/976145712 >> 172.21.15.43:0/976145712 conn(0x559cb9a3bcb0 msgr2=0x559cb9a36b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:41.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.484+0000 7f81867e9740 1 -- 172.21.15.43:0/976145712 shutdown_connections 2024-03-28T22:36:41.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.484+0000 7f81867e9740 1 -- 172.21.15.43:0/976145712 wait complete. 2024-03-28T22:36:41.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.484+0000 7f81867e9740 1 -- start start 2024-03-28T22:36:41.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.484+0000 7f81867e9740 1 Processor -- start 2024-03-28T22:36:41.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.485+0000 7f81867e9740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559cb9a40740 0x559cb9a4d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.485+0000 7f81867e9740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cb9a41130 0x559cb9a4d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.485+0000 7f81867e9740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559cb9a462a0 0x559cb9abb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.485+0000 7f81867e9740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x559cb9a589d0 con 0x559cb9a462a0 2024-03-28T22:36:41.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.485+0000 7f81867e9740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x559cb9a58300 con 0x559cb9a41130 2024-03-28T22:36:41.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.485+0000 7f81867e9740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x559cb9a590a0 con 0x559cb9a40740 2024-03-28T22:36:41.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.485+0000 7f81867e9740 10 client.0 _ll_register_callbacks cb 0x7f81680d1d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:36:41.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.485+0000 7f8181200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559cb9a40740 0x559cb9a4d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.485+0000 7f817be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cb9a41130 0x559cb9a4d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.485+0000 7f8181200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559cb9a40740 0x559cb9a4d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:53892/0 (socket says 172.21.15.43:53892) 2024-03-28T22:36:41.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.485+0000 7f8181c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559cb9a462a0 0x559cb9abb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.485+0000 7f8181200640 1 -- 172.21.15.43:0/1436681362 learned_addr learned my addr 172.21.15.43:0/1436681362 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:41.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.485+0000 7f8181200640 1 -- 172.21.15.43:0/1436681362 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cb9a41130 msgr2=0x559cb9a4d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:41.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.485+0000 7f8181200640 1 --2- 172.21.15.43:0/1436681362 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cb9a41130 0x559cb9a4d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.485+0000 7f8181200640 1 -- 172.21.15.43:0/1436681362 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559cb9a462a0 msgr2=0x559cb9abb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:41.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.485+0000 7f8181200640 1 --2- 172.21.15.43:0/1436681362 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559cb9a462a0 0x559cb9abb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.485+0000 7f8181200640 1 -- 172.21.15.43:0/1436681362 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559cb9abb9d0 con 0x559cb9a40740 2024-03-28T22:36:41.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.486+0000 7f817be00640 1 --2- 172.21.15.43:0/1436681362 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cb9a41130 0x559cb9a4d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:41.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.486+0000 7f8181200640 1 --2- 172.21.15.43:0/1436681362 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559cb9a40740 0x559cb9a4d0c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8170006010 tx=0x7f81700058e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77b662036ee1e28d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:41.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.486+0000 7f8162a00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:36:41.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.486+0000 7f8162a00640 1 -- 172.21.15.43:0/1436681362 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f817003fc50 con 0x559cb9a40740 2024-03-28T22:36:41.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.486+0000 7f8162a00640 1 -- 172.21.15.43:0/1436681362 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f817003fdf0 con 0x559cb9a40740 2024-03-28T22:36:41.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.486+0000 7f81867e9740 1 -- 172.21.15.43:0/1436681362 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f816809ae30 con 0x559cb9a40740 2024-03-28T22:36:41.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.486+0000 7f8162a00640 1 -- 172.21.15.43:0/1436681362 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8170066050 con 0x559cb9a40740 2024-03-28T22:36:41.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.486+0000 7f81867e9740 10 client.25447 fetch_fsmap learned FSMap version 28 2024-03-28T22:36:41.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.486+0000 7f81867e9740 1 -- 172.21.15.43:0/1436681362 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559cb9abccd0 con 0x559cb9a40740 2024-03-28T22:36:41.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.487+0000 7f8162a00640 1 -- 172.21.15.43:0/1436681362 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f816809ae30 con 0x559cb9a40740 2024-03-28T22:36:41.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.487+0000 7f8162a00640 1 -- 172.21.15.43:0/1436681362 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f81700349c0 con 0x559cb9a40740 2024-03-28T22:36:41.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.487+0000 7f81867e9740 10 client.25447 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T22:36:41.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.487+0000 7f81867e9740 10 client.25447 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T22:36:41.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.487+0000 7f81867e9740 20 client.25447 populate_metadata read hostname 'smithi043' 2024-03-28T22:36:41.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.487+0000 7f81867e9740 1 --2- 172.21.15.43:0/1436681362 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x559cb9ac1850 0x559cb9ac0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.487+0000 7f81867e9740 4 client.25447 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T22:36:41.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.487+0000 7f81867e9740 1 -- 172.21.15.43:0/1436681362 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "status"}) -- 0x559cb99ac250 con 0x559cb9ac1850 2024-03-28T22:36:41.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.487+0000 7f817be00640 1 --2- 172.21.15.43:0/1436681362 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x559cb9ac1850 0x559cb9ac0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.488+0000 7f817be00640 1 --2- 172.21.15.43:0/1436681362 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x559cb9ac1850 0x559cb9ac0560 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b9c4d998ee42fcf8 server_cookie=e4fa78f3dc4f4de5 in_seq=0 out_seq=0 2024-03-28T22:36:41.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.488+0000 7f8162a00640 10 client.25447 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:36:41.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.488+0000 7f8162a00640 1 -- 172.21.15.43:0/1436681362 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x559cb99ac250 con 0x559cb9ac1850 2024-03-28T22:36:41.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.488+0000 7f8162a00640 10 client.25447 handle_command_reply: tid=0 2024-03-28T22:36:41.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.488+0000 7f81867e9740 1 client.25447 shutdown 2024-03-28T22:36:41.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.488+0000 7f81867e9740 1 -- 172.21.15.43:0/1436681362 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559cb9a40740 msgr2=0x559cb9a4d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:41.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.488+0000 7f81867e9740 1 --2- 172.21.15.43:0/1436681362 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559cb9a40740 0x559cb9a4d0c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8170006010 tx=0x7f81700058e0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.488+0000 7f81867e9740 1 -- 172.21.15.43:0/1436681362 shutdown_connections 2024-03-28T22:36:41.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.488+0000 7f81867e9740 1 --2- 172.21.15.43:0/1436681362 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x559cb9ac1850 0x559cb9ac0560 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.488+0000 7f81867e9740 1 --2- 172.21.15.43:0/1436681362 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cb9a41130 0x559cb9a4d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.488+0000 7f81867e9740 1 --2- 172.21.15.43:0/1436681362 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559cb9a462a0 0x559cb9abb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.488+0000 7f81867e9740 1 --2- 172.21.15.43:0/1436681362 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559cb9a40740 0x559cb9a4d0c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.488+0000 7f8162a00640 0 client.25447 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:36:41.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.488+0000 7f81867e9740 1 -- 172.21.15.43:0/1436681362 >> 172.21.15.43:0/1436681362 conn(0x559cb9a3bcb0 msgr2=0x559cb9a3c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:41.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.488+0000 7f81867e9740 1 -- 172.21.15.43:0/1436681362 shutdown_connections 2024-03-28T22:36:41.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.488+0000 7f81867e9740 1 -- 172.21.15.43:0/1436681362 wait complete. 2024-03-28T22:36:41.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.489+0000 7f81867e9740 20 client.25447 trim_cache size 0 max 16384 2024-03-28T22:36:41.496 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:36:41.496 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_fsid": "c2276428-ed51-11ee-95d0-87774f69a715", 2024-03-28T22:36:41.496 INFO:teuthology.orchestra.run.smithi043.stdout: "whoami": 1, 2024-03-28T22:36:41.496 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 24556, 2024-03-28T22:36:41.496 INFO:teuthology.orchestra.run.smithi043.stdout: "want_state": "up:active", 2024-03-28T22:36:41.497 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "up:active", 2024-03-28T22:36:41.497 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs", 2024-03-28T22:36:41.497 INFO:teuthology.orchestra.run.smithi043.stdout: "rank_uptime": 159.36627870699999, 2024-03-28T22:36:41.497 INFO:teuthology.orchestra.run.smithi043.stdout: "mdsmap_epoch": 28, 2024-03-28T22:36:41.497 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch": 96, 2024-03-28T22:36:41.497 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch_barrier": 92, 2024-03-28T22:36:41.498 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 209.82926734700001 2024-03-28T22:36:41.498 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:36:41.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.489+0000 7f8182600640 1 -- 172.21.15.43:0/3043704511 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f81640a1e00 msgr2=0x7f81640a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:41.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.489+0000 7f8182600640 1 --2- 172.21.15.43:0/3043704511 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f81640a1e00 0x7f81640a42c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x559cb9a41990 tx=0x7f8174009500 comp rx=0 tx=0).stop 2024-03-28T22:36:41.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.489+0000 7f8182600640 1 -- 172.21.15.43:0/3043704511 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f817c108e60 msgr2=0x7f817c19e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:41.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.489+0000 7f8182600640 1 --2- 172.21.15.43:0/3043704511 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f817c108e60 0x7f817c19e490 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f816800e9f0 tx=0x7f816800eeb0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.489+0000 7f8182600640 1 -- 172.21.15.43:0/3043704511 shutdown_connections 2024-03-28T22:36:41.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.489+0000 7f8182600640 1 --2- 172.21.15.43:0/3043704511 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f81640a1e00 0x7f81640a42c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.489+0000 7f8182600640 1 --2- 172.21.15.43:0/3043704511 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f817c109820 0x7f817c1a5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.489+0000 7f8182600640 1 --2- 172.21.15.43:0/3043704511 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f817c108e60 0x7f817c19e490 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.489+0000 7f8182600640 1 --2- 172.21.15.43:0/3043704511 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f817c107c70 0x7f817c19df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.489+0000 7f8182600640 1 -- 172.21.15.43:0/3043704511 >> 172.21.15.43:0/3043704511 conn(0x7f817c076c20 msgr2=0x7f817c10c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:41.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.489+0000 7f8182600640 1 -- 172.21.15.43:0/3043704511 shutdown_connections 2024-03-28T22:36:41.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.489+0000 7f8182600640 1 -- 172.21.15.43:0/3043704511 wait complete. 2024-03-28T22:36:41.506 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T22:36:41.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.734+0000 7fd7ee800640 1 -- 172.21.15.43:0/1974375315 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7e8104a60 msgr2=0x7fd7e8104ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:41.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.734+0000 7fd7ee800640 1 --2- 172.21.15.43:0/1974375315 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7e8104a60 0x7fd7e8104ee0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4009900 tx=0x7fd7d402bc30 comp rx=0 tx=0).stop 2024-03-28T22:36:41.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.734+0000 7fd7ee800640 1 -- 172.21.15.43:0/1974375315 shutdown_connections 2024-03-28T22:36:41.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.734+0000 7fd7ee800640 1 --2- 172.21.15.43:0/1974375315 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7e8104a60 0x7fd7e8104ee0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.734+0000 7fd7ee800640 1 --2- 172.21.15.43:0/1974375315 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7e8105420 0x7fd7e810bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.734+0000 7fd7ee800640 1 --2- 172.21.15.43:0/1974375315 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7e8103870 0x7fd7e8103c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.734+0000 7fd7ee800640 1 -- 172.21.15.43:0/1974375315 >> 172.21.15.43:0/1974375315 conn(0x7fd7e80fee00 msgr2=0x7fd7e8101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:41.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.735+0000 7fd7ee800640 1 -- 172.21.15.43:0/1974375315 shutdown_connections 2024-03-28T22:36:41.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.735+0000 7fd7ee800640 1 -- 172.21.15.43:0/1974375315 wait complete. 2024-03-28T22:36:41.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.735+0000 7fd7ee800640 1 Processor -- start 2024-03-28T22:36:41.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.736+0000 7fd7ee800640 1 -- start start 2024-03-28T22:36:41.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.736+0000 7fd7ee800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7e8103870 0x7fd7e8199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.736+0000 7fd7ee800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7e8104a60 0x7fd7e819a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.736+0000 7fd7ee800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7e8105420 0x7fd7e81a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.736+0000 7fd7ee800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e810f540 con 0x7fd7e8103870 2024-03-28T22:36:41.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.736+0000 7fd7ee800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e810f3c0 con 0x7fd7e8104a60 2024-03-28T22:36:41.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.736+0000 7fd7ee800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e810f6c0 con 0x7fd7e8105420 2024-03-28T22:36:41.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.736+0000 7fd7eca00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7e8104a60 0x7fd7e819a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.736+0000 7fd7ede00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7e8105420 0x7fd7e81a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.736+0000 7fd7ed400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7e8103870 0x7fd7e8199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.736+0000 7fd7eca00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7e8104a60 0x7fd7e819a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47446/0 (socket says 172.21.15.43:47446) 2024-03-28T22:36:41.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.736+0000 7fd7eca00640 1 -- 172.21.15.43:0/1530805118 learned_addr learned my addr 172.21.15.43:0/1530805118 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:41.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.737+0000 7fd7eca00640 1 -- 172.21.15.43:0/1530805118 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7e8105420 msgr2=0x7fd7e81a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:41.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.737+0000 7fd7eca00640 1 --2- 172.21.15.43:0/1530805118 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7e8105420 0x7fd7e81a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.737+0000 7fd7eca00640 1 -- 172.21.15.43:0/1530805118 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7e8103870 msgr2=0x7fd7e8199b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:41.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.737+0000 7fd7eca00640 1 --2- 172.21.15.43:0/1530805118 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7e8103870 0x7fd7e8199b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.737+0000 7fd7eca00640 1 -- 172.21.15.43:0/1530805118 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7e81a1770 con 0x7fd7e8104a60 2024-03-28T22:36:41.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.737+0000 7fd7eca00640 1 --2- 172.21.15.43:0/1530805118 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7e8104a60 0x7fd7e819a090 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4005ab0 tx=0x7fd7d402c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:41.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.737+0000 7fd7e6000640 1 -- 172.21.15.43:0/1530805118 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d4043020 con 0x7fd7e8104a60 2024-03-28T22:36:41.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.737+0000 7fd7e6000640 1 -- 172.21.15.43:0/1530805118 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7d40054e0 con 0x7fd7e8104a60 2024-03-28T22:36:41.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.737+0000 7fd7e6000640 1 -- 172.21.15.43:0/1530805118 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d4003ce0 con 0x7fd7e8104a60 2024-03-28T22:36:41.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.737+0000 7fd7ee800640 1 -- 172.21.15.43:0/1530805118 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7e81a1a00 con 0x7fd7e8104a60 2024-03-28T22:36:41.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.737+0000 7fd7ee800640 1 -- 172.21.15.43:0/1530805118 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7e81a1f00 con 0x7fd7e8104a60 2024-03-28T22:36:41.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.740+0000 7fd7e6000640 1 -- 172.21.15.43:0/1530805118 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7fd7d4046020 con 0x7fd7e8104a60 2024-03-28T22:36:41.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.740+0000 7fd7f2b6a740 1 Processor -- start 2024-03-28T22:36:41.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.741+0000 7fd7f2b6a740 1 -- start start 2024-03-28T22:36:41.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.741+0000 7fd7f2b6a740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fa2286a740 0x55fa2286ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.741+0000 7fd7f2b6a740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fa2286b130 0x55fa2286fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.741+0000 7fd7f2b6a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fa228702a0 0x55fa22872730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.741+0000 7fd7f2b6a740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55fa227ce9c0 con 0x55fa2286b130 2024-03-28T22:36:41.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.741+0000 7fd7f2b6a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55fa22777c30 con 0x55fa228702a0 2024-03-28T22:36:41.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.741+0000 7fd7f2b6a740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55fa227caa60 con 0x55fa2286a740 2024-03-28T22:36:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.741+0000 7fd7ede00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fa2286a740 0x55fa2286ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.741+0000 7fd7ed400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fa2286b130 0x55fa2286fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.741+0000 7fd7ede00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fa2286a740 0x55fa2286ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:53916/0 (socket says 172.21.15.43:53916) 2024-03-28T22:36:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.741+0000 7fd7ede00640 1 -- 172.21.15.43:0/2852644503 learned_addr learned my addr 172.21.15.43:0/2852644503 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.741+0000 7fd7eca00640 1 --2- 172.21.15.43:0/2852644503 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fa228702a0 0x55fa22872730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.741+0000 7fd7ede00640 1 -- 172.21.15.43:0/2852644503 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fa228702a0 msgr2=0x55fa22872730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.741+0000 7fd7ede00640 1 --2- 172.21.15.43:0/2852644503 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fa228702a0 0x55fa22872730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.742+0000 7fd7ede00640 1 -- 172.21.15.43:0/2852644503 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fa2286b130 msgr2=0x55fa2286fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.742+0000 7fd7ede00640 1 --2- 172.21.15.43:0/2852644503 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fa2286b130 0x55fa2286fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.742+0000 7fd7ede00640 1 -- 172.21.15.43:0/2852644503 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa224e32d0 con 0x55fa2286a740 2024-03-28T22:36:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.742+0000 7fd7ede00640 1 --2- 172.21.15.43:0/2852644503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fa2286a740 0x55fa2286ab60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0010370 tx=0x7fd7e0010830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44dc08b60010f3d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.742+0000 7fd7da600640 1 -- 172.21.15.43:0/2852644503 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e0016070 con 0x55fa2286a740 2024-03-28T22:36:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.742+0000 7fd7e6000640 1 --2- 172.21.15.43:0/1530805118 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd7d00a1e00 0x7fd7d00a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.742+0000 7fd7da600640 1 -- 172.21.15.43:0/2852644503 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7e0004d20 con 0x55fa2286a740 2024-03-28T22:36:41.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.742+0000 7fd7da600640 1 -- 172.21.15.43:0/2852644503 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e0005480 con 0x55fa2286a740 2024-03-28T22:36:41.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.742+0000 7fd7e6000640 1 -- 172.21.15.43:0/1530805118 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7fd7d40f0190 con 0x7fd7e8104a60 2024-03-28T22:36:41.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.742+0000 7fd7ed400640 1 --2- 172.21.15.43:0/1530805118 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd7d00a1e00 0x7fd7d00a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.743+0000 7fd7f2b6a740 1 -- 172.21.15.43:0/2852644503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fa2286a740 msgr2=0x55fa2286ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:41.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.743+0000 7fd7f2b6a740 1 --2- 172.21.15.43:0/2852644503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fa2286a740 0x55fa2286ab60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0010370 tx=0x7fd7e0010830 comp rx=0 tx=0).stop 2024-03-28T22:36:41.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.743+0000 7fd7f2b6a740 1 -- 172.21.15.43:0/2852644503 shutdown_connections 2024-03-28T22:36:41.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.743+0000 7fd7f2b6a740 1 --2- 172.21.15.43:0/2852644503 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fa228702a0 0x55fa22872730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.743+0000 7fd7f2b6a740 1 --2- 172.21.15.43:0/2852644503 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fa2286b130 0x55fa2286fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.743+0000 7fd7f2b6a740 1 --2- 172.21.15.43:0/2852644503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fa2286a740 0x55fa2286ab60 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.743+0000 7fd7f2b6a740 1 -- 172.21.15.43:0/2852644503 >> 172.21.15.43:0/2852644503 conn(0x55fa22865cb0 msgr2=0x55fa228660b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:41.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.743+0000 7fd7f2b6a740 1 -- 172.21.15.43:0/2852644503 shutdown_connections 2024-03-28T22:36:41.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.743+0000 7fd7f2b6a740 1 -- 172.21.15.43:0/2852644503 wait complete. 2024-03-28T22:36:41.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.743+0000 7fd7ed400640 1 --2- 172.21.15.43:0/1530805118 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd7d00a1e00 0x7fd7d00a42c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x55fa2286b530 tx=0x7fd7dc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:41.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.744+0000 7fd7f2b6a740 1 -- start start 2024-03-28T22:36:41.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.744+0000 7fd7f2b6a740 1 Processor -- start 2024-03-28T22:36:41.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.744+0000 7fd7f2b6a740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fa2286a740 0x55fa22874cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.744+0000 7fd7f2b6a740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fa2286b130 0x55fa228751f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.744+0000 7fd7f2b6a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fa228702a0 0x55fa2287c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.744+0000 7fd7f2b6a740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55fa22873250 con 0x55fa2286b130 2024-03-28T22:36:41.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.744+0000 7fd7f2b6a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55fa2276d180 con 0x55fa228702a0 2024-03-28T22:36:41.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.745+0000 7fd7f2b6a740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55fa228733d0 con 0x55fa2286a740 2024-03-28T22:36:41.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.745+0000 7fd7f2b6a740 10 client.0 _ll_register_callbacks cb 0x55fa224e2ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:36:41.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.745+0000 7fd7ed400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fa2286a740 0x55fa22874cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.745+0000 7fd7ed400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fa2286a740 0x55fa22874cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:53922/0 (socket says 172.21.15.43:53922) 2024-03-28T22:36:41.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.745+0000 7fd7ed400640 1 -- 172.21.15.43:0/1042703997 learned_addr learned my addr 172.21.15.43:0/1042703997 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:41.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.745+0000 7fd7ede00640 1 --2- 172.21.15.43:0/1042703997 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fa228702a0 0x55fa2287c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.745+0000 7fd7eca00640 1 --2- 172.21.15.43:0/1042703997 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fa2286b130 0x55fa228751f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.745+0000 7fd7ede00640 1 -- 172.21.15.43:0/1042703997 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fa2286a740 msgr2=0x55fa22874cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:41.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.745+0000 7fd7ede00640 1 --2- 172.21.15.43:0/1042703997 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fa2286a740 0x55fa22874cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.745+0000 7fd7ede00640 1 -- 172.21.15.43:0/1042703997 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fa2286b130 msgr2=0x55fa228751f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:41.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.745+0000 7fd7ede00640 1 --2- 172.21.15.43:0/1042703997 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fa2286b130 0x55fa228751f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.745+0000 7fd7ede00640 1 -- 172.21.15.43:0/1042703997 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa225ad6b0 con 0x55fa228702a0 2024-03-28T22:36:41.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.745+0000 7fd7ed400640 1 --2- 172.21.15.43:0/1042703997 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fa2286a740 0x55fa22874cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:41.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.745+0000 7fd7eca00640 1 --2- 172.21.15.43:0/1042703997 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fa2286b130 0x55fa228751f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:41.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.746+0000 7fd7ede00640 1 --2- 172.21.15.43:0/1042703997 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fa228702a0 0x55fa2287c230 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0010fe0 tx=0x7fd7e0004f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99f62b6ef964c48e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:41.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.746+0000 7fd7cea00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:36:41.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.746+0000 7fd7cea00640 1 -- 172.21.15.43:0/1042703997 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e0029060 con 0x55fa228702a0 2024-03-28T22:36:41.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.746+0000 7fd7f2b6a740 1 -- 172.21.15.43:0/1042703997 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd7e0004d20 con 0x55fa228702a0 2024-03-28T22:36:41.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.746+0000 7fd7cea00640 1 -- 172.21.15.43:0/1042703997 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7fd7e0003e00 con 0x55fa228702a0 2024-03-28T22:36:41.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.746+0000 7fd7cea00640 1 -- 172.21.15.43:0/1042703997 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd7e0030050 con 0x55fa228702a0 2024-03-28T22:36:41.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.746+0000 7fd7f2b6a740 10 client.25451 fetch_fsmap learned FSMap version 28 2024-03-28T22:36:41.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.746+0000 7fd7f2b6a740 1 -- 172.21.15.43:0/1042703997 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fa2287f5f0 con 0x55fa228702a0 2024-03-28T22:36:41.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.746+0000 7fd7cea00640 1 -- 172.21.15.43:0/1042703997 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7e0004d20 con 0x55fa228702a0 2024-03-28T22:36:41.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.747+0000 7fd7cea00640 1 -- 172.21.15.43:0/1042703997 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd7e000f040 con 0x55fa228702a0 2024-03-28T22:36:41.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.747+0000 7fd7f2b6a740 10 client.25451 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T22:36:41.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.747+0000 7fd7f2b6a740 10 client.25451 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T22:36:41.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.747+0000 7fd7f2b6a740 20 client.25451 populate_metadata read hostname 'smithi043' 2024-03-28T22:36:41.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.747+0000 7fd7f2b6a740 1 --2- 172.21.15.43:0/1042703997 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55fa228817a0 0x55fa22882c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.747+0000 7fd7f2b6a740 4 client.25451 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:36:41.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.747+0000 7fd7f2b6a740 1 -- 172.21.15.43:0/1042703997 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa227dbf00 con 0x55fa228817a0 2024-03-28T22:36:41.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.747+0000 7fd7ed400640 1 --2- 172.21.15.43:0/1042703997 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55fa228817a0 0x55fa22882c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.748+0000 7fd7ed400640 1 --2- 172.21.15.43:0/1042703997 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55fa228817a0 0x55fa22882c10 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=159d26442a027520 server_cookie=abbd5ddd8bc837e1 in_seq=0 out_seq=0 2024-03-28T22:36:41.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.748+0000 7fd7cea00640 10 client.25451 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:36:41.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.748+0000 7fd7cea00640 1 -- 172.21.15.43:0/1042703997 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fa227dbf00 con 0x55fa228817a0 2024-03-28T22:36:41.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.748+0000 7fd7cea00640 10 client.25451 handle_command_reply: tid=0 2024-03-28T22:36:41.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.749+0000 7fd7f2b6a740 1 client.25451 shutdown 2024-03-28T22:36:41.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.749+0000 7fd7f2b6a740 1 -- 172.21.15.43:0/1042703997 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fa228702a0 msgr2=0x55fa2287c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:41.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.749+0000 7fd7f2b6a740 1 --2- 172.21.15.43:0/1042703997 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fa228702a0 0x55fa2287c230 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0010fe0 tx=0x7fd7e0004f40 comp rx=0 tx=0).stop 2024-03-28T22:36:41.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.749+0000 7fd7f2b6a740 1 -- 172.21.15.43:0/1042703997 shutdown_connections 2024-03-28T22:36:41.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.749+0000 7fd7f2b6a740 1 --2- 172.21.15.43:0/1042703997 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55fa228817a0 0x55fa22882c10 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.749+0000 7fd7f2b6a740 1 --2- 172.21.15.43:0/1042703997 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fa228702a0 0x55fa2287c230 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.749+0000 7fd7f2b6a740 1 --2- 172.21.15.43:0/1042703997 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fa2286b130 0x55fa228751f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.749+0000 7fd7f2b6a740 1 --2- 172.21.15.43:0/1042703997 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fa2286a740 0x55fa22874cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.749+0000 7fd7f2b6a740 1 -- 172.21.15.43:0/1042703997 >> 172.21.15.43:0/1042703997 conn(0x55fa22865cb0 msgr2=0x55fa22864df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:41.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.749+0000 7fd7cea00640 0 client.25451 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:36:41.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.749+0000 7fd7f2b6a740 1 -- 172.21.15.43:0/1042703997 shutdown_connections 2024-03-28T22:36:41.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.749+0000 7fd7f2b6a740 1 -- 172.21.15.43:0/1042703997 wait complete. 2024-03-28T22:36:41.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.749+0000 7fd7f2b6a740 20 client.25451 trim_cache size 0 max 16384 2024-03-28T22:36:41.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.770+0000 7fd7f2b6a740 1 Processor -- start 2024-03-28T22:36:41.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.770+0000 7fd7f2b6a740 1 -- start start 2024-03-28T22:36:41.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.770+0000 7fd7f2b6a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fa2286a740 0x55fa22872690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.770+0000 7fd7f2b6a740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fa2286b130 0x55fa2287c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.770+0000 7fd7f2b6a740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fa228702a0 0x55fa22866090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.770+0000 7fd7f2b6a740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55fa22880f50 con 0x55fa228702a0 2024-03-28T22:36:41.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.770+0000 7fd7f2b6a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55fa22881620 con 0x55fa2286a740 2024-03-28T22:36:41.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.770+0000 7fd7f2b6a740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55fa22881cf0 con 0x55fa2286b130 2024-03-28T22:36:41.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.770+0000 7fd7eca00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fa2286b130 0x55fa2287c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.770+0000 7fd7ed400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fa2286a740 0x55fa22872690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.771+0000 7fd7eca00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fa2286b130 0x55fa2287c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:53934/0 (socket says 172.21.15.43:53934) 2024-03-28T22:36:41.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.771+0000 7fd7eca00640 1 -- 172.21.15.43:0/1828858579 learned_addr learned my addr 172.21.15.43:0/1828858579 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:41.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.771+0000 7fd7ed400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fa2286a740 0x55fa22872690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47486/0 (socket says 172.21.15.43:47486) 2024-03-28T22:36:41.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.771+0000 7fd7ede00640 1 --2- 172.21.15.43:0/1828858579 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fa228702a0 0x55fa22866090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.771+0000 7fd7eca00640 1 -- 172.21.15.43:0/1828858579 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fa2286a740 msgr2=0x55fa22872690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:41.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.771+0000 7fd7eca00640 1 --2- 172.21.15.43:0/1828858579 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fa2286a740 0x55fa22872690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.771+0000 7fd7eca00640 1 -- 172.21.15.43:0/1828858579 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fa228702a0 msgr2=0x55fa22866090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:41.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.771+0000 7fd7eca00640 1 --2- 172.21.15.43:0/1828858579 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fa228702a0 0x55fa22866090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.771+0000 7fd7eca00640 1 -- 172.21.15.43:0/1828858579 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa22866c20 con 0x55fa2286b130 2024-03-28T22:36:41.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.771+0000 7fd7ed400640 1 --2- 172.21.15.43:0/1828858579 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fa2286a740 0x55fa22872690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:41.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.771+0000 7fd7eca00640 1 --2- 172.21.15.43:0/1828858579 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fa2286b130 0x55fa2287c150 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd7d40bca60 tx=0x7fd7d40f3bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bcc9536417e15764 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:41.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.771+0000 7fd7d9c00640 1 -- 172.21.15.43:0/1828858579 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d40b8d70 con 0x55fa2286b130 2024-03-28T22:36:41.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.771+0000 7fd7d9c00640 1 -- 172.21.15.43:0/1828858579 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7d4004c80 con 0x55fa2286b130 2024-03-28T22:36:41.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.771+0000 7fd7d9c00640 1 -- 172.21.15.43:0/1828858579 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d40bc430 con 0x55fa2286b130 2024-03-28T22:36:41.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.772+0000 7fd7f2b6a740 1 -- 172.21.15.43:0/1828858579 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fa2286b130 msgr2=0x55fa2287c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:41.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.772+0000 7fd7f2b6a740 1 --2- 172.21.15.43:0/1828858579 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fa2286b130 0x55fa2287c150 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd7d40bca60 tx=0x7fd7d40f3bd0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.772+0000 7fd7f2b6a740 1 -- 172.21.15.43:0/1828858579 shutdown_connections 2024-03-28T22:36:41.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.772+0000 7fd7f2b6a740 1 --2- 172.21.15.43:0/1828858579 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fa228702a0 0x55fa22866090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.772+0000 7fd7f2b6a740 1 --2- 172.21.15.43:0/1828858579 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fa2286b130 0x55fa2287c150 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.772+0000 7fd7f2b6a740 1 --2- 172.21.15.43:0/1828858579 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fa2286a740 0x55fa22872690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.772+0000 7fd7f2b6a740 1 -- 172.21.15.43:0/1828858579 >> 172.21.15.43:0/1828858579 conn(0x55fa22865cb0 msgr2=0x55fa2286bb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:41.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.772+0000 7fd7f2b6a740 1 -- 172.21.15.43:0/1828858579 shutdown_connections 2024-03-28T22:36:41.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.772+0000 7fd7f2b6a740 1 -- 172.21.15.43:0/1828858579 wait complete. 2024-03-28T22:36:41.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.773+0000 7fd7f2b6a740 1 -- start start 2024-03-28T22:36:41.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.773+0000 7fd7f2b6a740 1 Processor -- start 2024-03-28T22:36:41.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.773+0000 7fd7f2b6a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fa2286a740 0x55fa228771d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.773+0000 7fd7f2b6a740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fa2286b130 0x55fa22877710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.773+0000 7fd7f2b6a740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fa228702a0 0x55fa2287a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.773+0000 7fd7f2b6a740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55fa22882a90 con 0x55fa2286b130 2024-03-28T22:36:41.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.773+0000 7fd7f2b6a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55fa228823c0 con 0x55fa2286a740 2024-03-28T22:36:41.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.773+0000 7fd7f2b6a740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55fa22883160 con 0x55fa228702a0 2024-03-28T22:36:41.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.773+0000 7fd7f2b6a740 10 client.0 _ll_register_callbacks cb 0x7fd7dc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:36:41.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.773+0000 7fd7ed400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fa2286a740 0x55fa228771d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.773+0000 7fd7ed400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fa2286a740 0x55fa228771d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47492/0 (socket says 172.21.15.43:47492) 2024-03-28T22:36:41.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.773+0000 7fd7ed400640 1 -- 172.21.15.43:0/1170826073 learned_addr learned my addr 172.21.15.43:0/1170826073 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:41.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.773+0000 7fd7eca00640 1 --2- 172.21.15.43:0/1170826073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fa2286b130 0x55fa22877710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.773+0000 7fd7ede00640 1 --2- 172.21.15.43:0/1170826073 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fa228702a0 0x55fa2287a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.774+0000 7fd7ed400640 1 -- 172.21.15.43:0/1170826073 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fa228702a0 msgr2=0x55fa2287a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:41.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.774+0000 7fd7ed400640 1 --2- 172.21.15.43:0/1170826073 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fa228702a0 0x55fa2287a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.774+0000 7fd7ed400640 1 -- 172.21.15.43:0/1170826073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fa2286b130 msgr2=0x55fa22877710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:36:41.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.774+0000 7fd7ed400640 1 --2- 172.21.15.43:0/1170826073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fa2286b130 0x55fa22877710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.774+0000 7fd7ed400640 1 -- 172.21.15.43:0/1170826073 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa2287ad20 con 0x55fa2286a740 2024-03-28T22:36:41.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.774+0000 7fd7ed400640 1 --2- 172.21.15.43:0/1170826073 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fa2286a740 0x55fa228771d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x55fa22883350 tx=0x7fd7dc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d255551077357082 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:41.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.774+0000 7fd7cea00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:36:41.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.774+0000 7fd7cea00640 1 -- 172.21.15.43:0/1170826073 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc02bc50 con 0x55fa2286a740 2024-03-28T22:36:41.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.774+0000 7fd7cea00640 1 -- 172.21.15.43:0/1170826073 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7fd7dc02bdf0 con 0x55fa2286a740 2024-03-28T22:36:41.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.774+0000 7fd7f2b6a740 1 -- 172.21.15.43:0/1170826073 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd7d4004c80 con 0x55fa2286a740 2024-03-28T22:36:41.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.775+0000 7fd7cea00640 1 -- 172.21.15.43:0/1170826073 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd7dc03c050 con 0x55fa2286a740 2024-03-28T22:36:41.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.775+0000 7fd7f2b6a740 10 client.25457 fetch_fsmap learned FSMap version 28 2024-03-28T22:36:41.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.775+0000 7fd7f2b6a740 1 -- 172.21.15.43:0/1170826073 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fa2287c090 con 0x55fa2286a740 2024-03-28T22:36:41.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.775+0000 7fd7cea00640 1 -- 172.21.15.43:0/1170826073 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7d4004c80 con 0x55fa2286a740 2024-03-28T22:36:41.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.775+0000 7fd7cea00640 1 -- 172.21.15.43:0/1170826073 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd7dc01f720 con 0x55fa2286a740 2024-03-28T22:36:41.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.775+0000 7fd7f2b6a740 10 client.25457 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T22:36:41.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.775+0000 7fd7f2b6a740 10 client.25457 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T22:36:41.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.775+0000 7fd7f2b6a740 20 client.25457 populate_metadata read hostname 'smithi043' 2024-03-28T22:36:41.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.775+0000 7fd7f2b6a740 1 --2- 172.21.15.43:0/1170826073 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55fa228ebab0 0x55fa228ea7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:41.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.775+0000 7fd7f2b6a740 4 client.25457 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T22:36:41.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.775+0000 7fd7f2b6a740 1 -- 172.21.15.43:0/1170826073 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "status"}) -- 0x55fa227d6250 con 0x55fa228ebab0 2024-03-28T22:36:41.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.776+0000 7fd7eca00640 1 --2- 172.21.15.43:0/1170826073 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55fa228ebab0 0x55fa228ea7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:41.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.776+0000 7fd7eca00640 1 --2- 172.21.15.43:0/1170826073 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55fa228ebab0 0x55fa228ea7c0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa194cf93db7bf4c server_cookie=4deead9629d345e5 in_seq=0 out_seq=0 2024-03-28T22:36:41.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.776+0000 7fd7cea00640 10 client.25457 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:36:41.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.776+0000 7fd7cea00640 1 -- 172.21.15.43:0/1170826073 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55fa227d6250 con 0x55fa228ebab0 2024-03-28T22:36:41.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.776+0000 7fd7cea00640 10 client.25457 handle_command_reply: tid=0 2024-03-28T22:36:41.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.776+0000 7fd7f2b6a740 1 client.25457 shutdown 2024-03-28T22:36:41.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.777+0000 7fd7f2b6a740 1 -- 172.21.15.43:0/1170826073 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fa2286a740 msgr2=0x55fa228771d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:41.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.777+0000 7fd7f2b6a740 1 --2- 172.21.15.43:0/1170826073 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fa2286a740 0x55fa228771d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x55fa22883350 tx=0x7fd7dc0046c0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.777+0000 7fd7f2b6a740 1 -- 172.21.15.43:0/1170826073 shutdown_connections 2024-03-28T22:36:41.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.777+0000 7fd7f2b6a740 1 --2- 172.21.15.43:0/1170826073 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fa228702a0 0x55fa2287a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.777+0000 7fd7f2b6a740 1 --2- 172.21.15.43:0/1170826073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fa2286b130 0x55fa22877710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.777+0000 7fd7f2b6a740 1 --2- 172.21.15.43:0/1170826073 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55fa228ebab0 0x55fa228ea7c0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.777+0000 7fd7f2b6a740 1 --2- 172.21.15.43:0/1170826073 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fa2286a740 0x55fa228771d0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.777+0000 7fd7f2b6a740 1 -- 172.21.15.43:0/1170826073 >> 172.21.15.43:0/1170826073 conn(0x55fa22865cb0 msgr2=0x55fa22866090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:41.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.777+0000 7fd7cea00640 0 client.25457 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:36:41.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.777+0000 7fd7f2b6a740 1 -- 172.21.15.43:0/1170826073 shutdown_connections 2024-03-28T22:36:41.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.777+0000 7fd7f2b6a740 1 -- 172.21.15.43:0/1170826073 wait complete. 2024-03-28T22:36:41.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.777+0000 7fd7f2b6a740 20 client.25457 trim_cache size 0 max 16384 2024-03-28T22:36:41.781 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:36:41.782 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_fsid": "c2276428-ed51-11ee-95d0-87774f69a715", 2024-03-28T22:36:41.782 INFO:teuthology.orchestra.run.smithi043.stdout: "whoami": 0, 2024-03-28T22:36:41.782 INFO:teuthology.orchestra.run.smithi043.stdout: "id": 24572, 2024-03-28T22:36:41.782 INFO:teuthology.orchestra.run.smithi043.stdout: "want_state": "up:active", 2024-03-28T22:36:41.782 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "up:active", 2024-03-28T22:36:41.782 INFO:teuthology.orchestra.run.smithi043.stdout: "fs_name": "cephfs", 2024-03-28T22:36:41.782 INFO:teuthology.orchestra.run.smithi043.stdout: "rank_uptime": 192.30416747300001, 2024-03-28T22:36:41.782 INFO:teuthology.orchestra.run.smithi043.stdout: "mdsmap_epoch": 28, 2024-03-28T22:36:41.782 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch": 96, 2024-03-28T22:36:41.782 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_epoch_barrier": 92, 2024-03-28T22:36:41.782 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 198.67824778900001 2024-03-28T22:36:41.782 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:36:41.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.778+0000 7fd7ee800640 1 -- 172.21.15.43:0/1530805118 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd7d00a1e00 msgr2=0x7fd7d00a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:41.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.778+0000 7fd7ee800640 1 --2- 172.21.15.43:0/1530805118 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd7d00a1e00 0x7fd7d00a42c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x55fa2286b530 tx=0x7fd7dc005f90 comp rx=0 tx=0).stop 2024-03-28T22:36:41.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.778+0000 7fd7ee800640 1 -- 172.21.15.43:0/1530805118 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7e8104a60 msgr2=0x7fd7e819a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:41.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.778+0000 7fd7ee800640 1 --2- 172.21.15.43:0/1530805118 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7e8104a60 0x7fd7e819a090 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4005ab0 tx=0x7fd7d402c6c0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.778+0000 7fd7ee800640 1 -- 172.21.15.43:0/1530805118 shutdown_connections 2024-03-28T22:36:41.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.778+0000 7fd7ee800640 1 --2- 172.21.15.43:0/1530805118 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd7d00a1e00 0x7fd7d00a42c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.778+0000 7fd7ee800640 1 --2- 172.21.15.43:0/1530805118 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7e8104a60 0x7fd7e819a090 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.778+0000 7fd7ee800640 1 --2- 172.21.15.43:0/1530805118 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7e8105420 0x7fd7e81a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.778+0000 7fd7ee800640 1 --2- 172.21.15.43:0/1530805118 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7e8103870 0x7fd7e8199b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:41.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.778+0000 7fd7ee800640 1 -- 172.21.15.43:0/1530805118 >> 172.21.15.43:0/1530805118 conn(0x7fd7e80fee00 msgr2=0x7fd7e8107d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:41.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.778+0000 7fd7ee800640 1 -- 172.21.15.43:0/1530805118 shutdown_connections 2024-03-28T22:36:41.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:41.778+0000 7fd7ee800640 1 -- 172.21.15.43:0/1530805118 wait complete. 2024-03-28T22:36:41.795 INFO:teuthology.run_tasks:Running task kclient... 2024-03-28T22:36:41.811 INFO:tasks.kclient:Mounting kernel clients... 2024-03-28T22:36:41.811 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-28T22:36:41.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:36:41.811 DEBUG:teuthology.orchestra.run.smithi043:> ip netns list 2024-03-28T22:36:41.865 INFO:teuthology.orchestra.run.smithi043.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-28T22:36:41.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:36:41.866 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T22:36:41.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:36:41.963 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link delete ceph-brx 2024-03-28T22:36:42.045 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:41 smithi043 ceph-mon[31823]: pgmap v422: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-28T22:36:42.088 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'nowsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-03-28T22:36:42.088 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-28T22:36:42.088 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-28T22:36:42.088 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph fs ls 2024-03-28T22:36:42.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:41 smithi172 ceph-mon[36566]: pgmap v422: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-28T22:36:42.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:41 smithi082 ceph-mon[36471]: pgmap v422: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-28T22:36:42.559 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:36:44.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:43 smithi172 ceph-mon[36566]: pgmap v423: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-28T22:36:44.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:43 smithi082 ceph-mon[36471]: pgmap v423: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-28T22:36:44.233 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:43 smithi043 ceph-mon[31823]: pgmap v423: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-28T22:36:44.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.750+0000 7f3bdfe00700 1 -- 172.21.15.43:0/1905895865 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3bd8104df0 msgr2=0x7f3bd8111300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:44.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.750+0000 7f3bdfe00700 1 --2- 172.21.15.43:0/1905895865 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3bd8104df0 0x7f3bd8111300 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3bd000b0a0 tx=0x7f3bd000b3b0 comp rx=0 tx=0).stop 2024-03-28T22:36:44.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.750+0000 7f3bdfe00700 1 -- 172.21.15.43:0/1905895865 shutdown_connections 2024-03-28T22:36:44.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.750+0000 7f3bdfe00700 1 --2- 172.21.15.43:0/1905895865 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3bd8104df0 0x7f3bd8111300 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:44.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.750+0000 7f3bdfe00700 1 --2- 172.21.15.43:0/1905895865 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3bd810a4c0 0x7f3bd810a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:44.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.750+0000 7f3bdfe00700 1 --2- 172.21.15.43:0/1905895865 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3bd8104430 0x7f3bd81048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:44.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.750+0000 7f3bdfe00700 1 -- 172.21.15.43:0/1905895865 >> 172.21.15.43:0/1905895865 conn(0x7f3bd80fff20 msgr2=0x7f3bd8102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:44.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.750+0000 7f3bdfe00700 1 -- 172.21.15.43:0/1905895865 shutdown_connections 2024-03-28T22:36:44.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.750+0000 7f3bdfe00700 1 -- 172.21.15.43:0/1905895865 wait complete. 2024-03-28T22:36:44.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.751+0000 7f3bdfe00700 1 Processor -- start 2024-03-28T22:36:44.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.751+0000 7f3bdfe00700 1 -- start start 2024-03-28T22:36:44.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.752+0000 7f3bdfe00700 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3bd8104430 0x7f3bd81a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:44.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.752+0000 7f3bdfe00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3bd8104df0 0x7f3bd81a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:44.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.752+0000 7f3bdfe00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3bd810a4c0 0x7f3bd819a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:44.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.752+0000 7f3bdfe00700 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd81147a0 con 0x7f3bd8104430 2024-03-28T22:36:44.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.752+0000 7f3bdfe00700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd8114620 con 0x7f3bd8104df0 2024-03-28T22:36:44.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.752+0000 7f3bdfe00700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd8114920 con 0x7f3bd810a4c0 2024-03-28T22:36:44.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.752+0000 7f3bdde00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3bd810a4c0 0x7f3bd819a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:44.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.752+0000 7f3bdca00700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3bd8104df0 0x7f3bd81a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:44.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.752+0000 7f3bdde00700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3bd810a4c0 0x7f3bd819a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:53966/0 (socket says 172.21.15.43:53966) 2024-03-28T22:36:44.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.752+0000 7f3bdde00700 1 -- 172.21.15.43:0/1309695780 learned_addr learned my addr 172.21.15.43:0/1309695780 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:44.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.752+0000 7f3bdd400700 1 --2- 172.21.15.43:0/1309695780 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3bd8104430 0x7f3bd81a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:44.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.753+0000 7f3bdde00700 1 -- 172.21.15.43:0/1309695780 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3bd8104df0 msgr2=0x7f3bd81a0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:44.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.753+0000 7f3bdde00700 1 --2- 172.21.15.43:0/1309695780 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3bd8104df0 0x7f3bd81a0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:44.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.753+0000 7f3bdde00700 1 -- 172.21.15.43:0/1309695780 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3bd8104430 msgr2=0x7f3bd81a0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:44.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.753+0000 7f3bdde00700 1 --2- 172.21.15.43:0/1309695780 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3bd8104430 0x7f3bd81a0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:44.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.753+0000 7f3bdde00700 1 -- 172.21.15.43:0/1309695780 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bd819b120 con 0x7f3bd810a4c0 2024-03-28T22:36:44.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.753+0000 7f3bdd400700 1 --2- 172.21.15.43:0/1309695780 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3bd8104430 0x7f3bd81a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:36:44.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.753+0000 7f3bdca00700 1 --2- 172.21.15.43:0/1309695780 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3bd8104df0 0x7f3bd81a0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:44.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.753+0000 7f3bdde00700 1 --2- 172.21.15.43:0/1309695780 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3bd810a4c0 0x7f3bd819a9c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3bd000b070 tx=0x7f3bd0004cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:44.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.753+0000 7f3bd6000700 1 -- 172.21.15.43:0/1309695780 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd0021070 con 0x7f3bd810a4c0 2024-03-28T22:36:44.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.753+0000 7f3bd6000700 1 -- 172.21.15.43:0/1309695780 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bd0003c00 con 0x7f3bd810a4c0 2024-03-28T22:36:44.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.753+0000 7f3bd6000700 1 -- 172.21.15.43:0/1309695780 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd0004850 con 0x7f3bd810a4c0 2024-03-28T22:36:44.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.753+0000 7f3bdfe00700 1 -- 172.21.15.43:0/1309695780 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bd819b3b0 con 0x7f3bd810a4c0 2024-03-28T22:36:44.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.753+0000 7f3bdfe00700 1 -- 172.21.15.43:0/1309695780 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bd81a76d0 con 0x7f3bd810a4c0 2024-03-28T22:36:44.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.755+0000 7f3bdfe00700 1 -- 172.21.15.43:0/1309695780 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bd8068f00 con 0x7f3bd810a4c0 2024-03-28T22:36:44.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.756+0000 7f3bd6000700 1 -- 172.21.15.43:0/1309695780 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f3bd0007980 con 0x7f3bd810a4c0 2024-03-28T22:36:44.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.758+0000 7f3bd6000700 1 --2- 172.21.15.43:0/1309695780 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3bc00a1e90 0x7f3bc00a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:44.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.758+0000 7f3bd6000700 1 -- 172.21.15.43:0/1309695780 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f3bd000f020 con 0x7f3bd810a4c0 2024-03-28T22:36:44.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.758+0000 7f3bdd400700 1 --2- 172.21.15.43:0/1309695780 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3bc00a1e90 0x7f3bc00a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:44.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.759+0000 7f3bdd400700 1 --2- 172.21.15.43:0/1309695780 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3bc00a1e90 0x7f3bc00a4350 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc005e10 tx=0x7f3bcc005d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:44.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:44.762+0000 7f3bd6000700 1 -- 172.21.15.43:0/1309695780 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3bd002a020 con 0x7f3bd810a4c0 2024-03-28T22:36:45.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:45.043+0000 7f3bdfe00700 1 -- 172.21.15.43:0/1309695780 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f3bd81a78a0 con 0x7f3bd810a4c0 2024-03-28T22:36:45.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:45.044+0000 7f3bd6000700 1 -- 172.21.15.43:0/1309695780 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) ==== 53+0+88 (secure 0 0 0) 0x7f3bd009c6a0 con 0x7f3bd810a4c0 2024-03-28T22:36:45.043 INFO:teuthology.orchestra.run.smithi043.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-28T22:36:45.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:45.047+0000 7f3bdfe00700 1 -- 172.21.15.43:0/1309695780 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3bc00a1e90 msgr2=0x7f3bc00a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:45.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:45.047+0000 7f3bdfe00700 1 --2- 172.21.15.43:0/1309695780 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3bc00a1e90 0x7f3bc00a4350 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc005e10 tx=0x7f3bcc005d20 comp rx=0 tx=0).stop 2024-03-28T22:36:45.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:45.047+0000 7f3bdfe00700 1 -- 172.21.15.43:0/1309695780 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3bd810a4c0 msgr2=0x7f3bd819a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:45.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:45.047+0000 7f3bdfe00700 1 --2- 172.21.15.43:0/1309695780 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3bd810a4c0 0x7f3bd819a9c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3bd000b070 tx=0x7f3bd0004cf0 comp rx=0 tx=0).stop 2024-03-28T22:36:45.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:45.048+0000 7f3bdfe00700 1 -- 172.21.15.43:0/1309695780 shutdown_connections 2024-03-28T22:36:45.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:45.048+0000 7f3bdfe00700 1 --2- 172.21.15.43:0/1309695780 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3bc00a1e90 0x7f3bc00a4350 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:45.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:45.048+0000 7f3bdfe00700 1 --2- 172.21.15.43:0/1309695780 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3bd810a4c0 0x7f3bd819a9c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:45.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:45.048+0000 7f3bdfe00700 1 --2- 172.21.15.43:0/1309695780 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3bd8104430 0x7f3bd81a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:45.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:45.048+0000 7f3bdfe00700 1 --2- 172.21.15.43:0/1309695780 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3bd8104df0 0x7f3bd81a0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:45.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:45.048+0000 7f3bdfe00700 1 -- 172.21.15.43:0/1309695780 >> 172.21.15.43:0/1309695780 conn(0x7f3bd80fff20 msgr2=0x7f3bd81003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:45.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:45.048+0000 7f3bdfe00700 1 -- 172.21.15.43:0/1309695780 shutdown_connections 2024-03-28T22:36:45.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:45.048+0000 7f3bdfe00700 1 -- 172.21.15.43:0/1309695780 wait complete. 2024-03-28T22:36:45.838 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-28T22:36:45.839 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-28T22:36:45.839 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi043.front.sepia.ceph.com 2024-03-28T22:36:45.839 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-28T22:36:45.839 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-28T22:36:45.839 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-28T22:36:45.839 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-28T22:36:45.839 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-28T22:36:45.839 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-28T22:36:45.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:36:45.839 DEBUG:teuthology.orchestra.run.smithi043:> ip addr 2024-03-28T22:36:45.898 INFO:teuthology.orchestra.run.smithi043.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-28T22:36:45.899 INFO:teuthology.orchestra.run.smithi043.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-28T22:36:45.899 INFO:teuthology.orchestra.run.smithi043.stdout: inet 127.0.0.1/8 scope host lo 2024-03-28T22:36:45.899 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-03-28T22:36:45.899 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 ::1/128 scope host 2024-03-28T22:36:45.899 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-03-28T22:36:45.899 INFO:teuthology.orchestra.run.smithi043.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T22:36:45.899 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:6b:fd:9c brd ff:ff:ff:ff:ff:ff 2024-03-28T22:36:45.899 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp7s0f0 2024-03-28T22:36:45.899 INFO:teuthology.orchestra.run.smithi043.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T22:36:45.899 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:6b:fd:9d brd ff:ff:ff:ff:ff:ff 2024-03-28T22:36:45.899 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp7s0f1 2024-03-28T22:36:45.899 INFO:teuthology.orchestra.run.smithi043.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-28T22:36:45.899 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:bd:13:58 brd ff:ff:ff:ff:ff:ff 2024-03-28T22:36:45.899 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp1s0f0 2024-03-28T22:36:45.900 INFO:teuthology.orchestra.run.smithi043.stdout: inet 172.21.15.43/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-28T22:36:45.900 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft 41844sec preferred_lft 41844sec 2024-03-28T22:36:45.900 INFO:teuthology.orchestra.run.smithi043.stdout: inet6 fe80::ec4:7aff:febd:1358/64 scope link noprefixroute 2024-03-28T22:36:45.900 INFO:teuthology.orchestra.run.smithi043.stdout: valid_lft forever preferred_lft forever 2024-03-28T22:36:45.900 INFO:teuthology.orchestra.run.smithi043.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T22:36:45.900 INFO:teuthology.orchestra.run.smithi043.stdout: link/ether 0c:c4:7a:bd:13:59 brd ff:ff:ff:ff:ff:ff 2024-03-28T22:36:45.900 INFO:teuthology.orchestra.run.smithi043.stdout: altname enp1s0f1 2024-03-28T22:36:45.901 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-28T22:36:45.901 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T22:36:45.901 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-03-28T22:36:45.901 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link add name ceph-brx type bridge 2024-03-28T22:36:45.901 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip addr flush dev ceph-brx 2024-03-28T22:36:45.901 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link set ceph-brx up 2024-03-28T22:36:45.901 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-28T22:36:45.902 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-03-28T22:36:46.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T22:36:46.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:45 smithi043 ceph-mon[31823]: pgmap v424: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-28T22:36:46.044 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:45 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1309695780' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T22:36:46.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:45 smithi172 ceph-mon[36566]: pgmap v424: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-28T22:36:46.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:45 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1309695780' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T22:36:46.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T22:36:46.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:36:46.193 DEBUG:teuthology.orchestra.run.smithi043:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-28T22:36:46.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:45 smithi082 ceph-mon[36471]: pgmap v424: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-28T22:36:46.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:45 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1309695780' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T22:36:46.269 INFO:teuthology.orchestra.run.smithi043.stdout:1 2024-03-28T22:36:46.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:36:46.272 DEBUG:teuthology.orchestra.run.smithi043:> route 2024-03-28T22:36:46.324 INFO:teuthology.orchestra.run.smithi043.stdout:Kernel IP routing table 2024-03-28T22:36:46.324 INFO:teuthology.orchestra.run.smithi043.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-28T22:36:46.324 INFO:teuthology.orchestra.run.smithi043.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-28T22:36:46.324 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-28T22:36:46.324 INFO:teuthology.orchestra.run.smithi043.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-28T22:36:46.325 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T22:36:46.325 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-03-28T22:36:46.325 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-28T22:36:46.325 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-28T22:36:46.325 DEBUG:teuthology.orchestra.run.smithi043:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-28T22:36:46.325 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-03-28T22:36:46.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T22:36:46.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T22:36:46.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:36:46.593 DEBUG:teuthology.orchestra.run.smithi043:> ip netns list 2024-03-28T22:36:46.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:36:46.645 DEBUG:teuthology.orchestra.run.smithi043:> ip netns list-id 2024-03-28T22:36:46.697 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T22:36:46.697 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-03-28T22:36:46.697 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T22:36:46.697 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-28T22:36:46.697 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-03-28T22:36:46.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T22:36:46.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T22:36:46.924 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-28T22:36:46.924 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T22:36:46.924 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-03-28T22:36:46.924 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-28T22:36:46.924 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-28T22:36:46.924 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-28T22:36:46.925 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-28T22:36:46.925 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-28T22:36:46.925 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-03-28T22:36:47.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T22:36:47.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T22:36:47.257 DEBUG:teuthology.orchestra.run.smithi043:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-28T22:36:47.257 DEBUG:teuthology.orchestra.run.smithi043:> set -e 2024-03-28T22:36:47.257 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link set brx.0 up 2024-03-28T22:36:47.258 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link set dev brx.0 master ceph-brx 2024-03-28T22:36:47.258 DEBUG:teuthology.orchestra.run.smithi043:> ') 2024-03-28T22:36:47.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T22:36:47.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T22:36:47.496 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T22:36:47.496 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-28T22:36:47.547 INFO:teuthology.orchestra.run.smithi043.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-28T22:36:47.548 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T22:36:47.548 DEBUG:teuthology.orchestra.run.smithi043:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-28T22:36:47.596 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-28T22:36:47.596 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-28T22:36:47.596 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/0f44e03b-e9f6-4a20-9eab-e6caac9e0703 2024-03-28T22:36:47.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:36:47.597 DEBUG:teuthology.orchestra.run.smithi043:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/0f44e03b-e9f6-4a20-9eab-e6caac9e0703 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2024-03-28T22:36:47.736 INFO:teuthology.orchestra.run.smithi043.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2024-03-28T22:36:47.737 INFO:teuthology.orchestra.run.smithi043.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync". 2024-03-28T22:36:47.737 INFO:teuthology.orchestra.run.smithi043.stdout:invalid new device string format 2024-03-28T22:36:48.012 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:47 smithi043 ceph-mon[31823]: pgmap v425: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T22:36:48.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:47 smithi172 ceph-mon[36566]: pgmap v425: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T22:36:48.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:47 smithi082 ceph-mon[36471]: pgmap v425: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T22:36:48.417 INFO:teuthology.orchestra.run.smithi043.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2024-03-28T22:36:48.418 INFO:teuthology.orchestra.run.smithi043.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync". 2024-03-28T22:36:48.418 INFO:teuthology.orchestra.run.smithi043.stdout:invalid new device string format 2024-03-28T22:36:48.418 INFO:teuthology.orchestra.run.smithi043.stdout:mount.ceph: resolved to: "172.21.15.43:3300,172.21.15.82:3300,172.21.15.172:3300" 2024-03-28T22:36:48.418 INFO:teuthology.orchestra.run.smithi043.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.43:3300,172.21.15.82:3300,172.21.15.172:3300:/volumes/qa/sv_1/0f44e03b-e9f6-4a20-9eab-e6caac9e0703 2024-03-28T22:36:48.418 INFO:teuthology.orchestra.run.smithi043.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync,key=0,fsid=c2276428-ed51-11ee-95d0-87774f69a715" will pass to kernel 2024-03-28T22:36:48.421 INFO:teuthology.orchestra.run.smithi043.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-28T22:36:48.421 INFO:teuthology.orchestra.run.smithi043.stdout: You just mounted a file system that supports labels which does not 2024-03-28T22:36:48.421 INFO:teuthology.orchestra.run.smithi043.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-28T22:36:48.421 INFO:teuthology.orchestra.run.smithi043.stdout: applications will generate AVC messages and not be allowed access to 2024-03-28T22:36:48.421 INFO:teuthology.orchestra.run.smithi043.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-28T22:36:48.427 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-28T22:36:48.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:36:48.428 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-28T22:36:48.517 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-28T22:36:48.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-28T22:36:48.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T22:36:48.633 INFO:teuthology.orchestra.run.smithi043.stdout:client25474 2024-03-28T22:36:48.633 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-28T22:36:48.633 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T22:36:48.679 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-28T22:36:48.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-28T22:36:48.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T22:36:48.833 INFO:teuthology.orchestra.run.smithi043.stdout:client25474 2024-03-28T22:36:48.833 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-28T22:36:48.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-28T22:36:48.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T22:36:48.951 INFO:teuthology.orchestra.run.smithi043.stdout:client25474 2024-03-28T22:36:48.952 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/c2276428-ed51-11ee-95d0-87774f69a715.client25474/status') 2024-03-28T22:36:49.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-28T22:36:49.087 INFO:teuthology.orchestra.run.smithi043.stdout:instance: client.25474 (3)172.21.15.43:0/2823821172 2024-03-28T22:36:49.087 INFO:teuthology.orchestra.run.smithi043.stdout:blocklisted: false 2024-03-28T22:36:49.087 INFO:teuthology.orchestra.run.smithi043.stderr:0+1 records in 2024-03-28T22:36:49.087 INFO:teuthology.orchestra.run.smithi043.stderr:0+1 records out 2024-03-28T22:36:49.087 INFO:teuthology.orchestra.run.smithi043.stderr:71 bytes copied, 5.6319e-05 s, 1.3 MB/s 2024-03-28T22:36:49.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T22:36:49.099 INFO:teuthology.run_tasks:Running task exec... 2024-03-28T22:36:49.108 INFO:teuthology.task.exec:Executing custom commands... 2024-03-28T22:36:49.108 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi043.front.sepia.ceph.com 2024-03-28T22:36:49.109 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-03-28T22:36:49.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.304+0000 7efdbc200640 1 -- 172.21.15.43:0/4077231734 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdb4103340 msgr2=0x7efdb4109bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:49.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.304+0000 7efdbc200640 1 --2- 172.21.15.43:0/4077231734 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdb4103340 0x7efdb4109bd0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7efdac009900 tx=0x7efdac02bc50 comp rx=0 tx=0).stop 2024-03-28T22:36:49.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.304+0000 7efdbc200640 1 -- 172.21.15.43:0/4077231734 shutdown_connections 2024-03-28T22:36:49.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.304+0000 7efdbc200640 1 --2- 172.21.15.43:0/4077231734 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdb4103340 0x7efdb4109bd0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:49.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.304+0000 7efdbc200640 1 --2- 172.21.15.43:0/4077231734 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efdb4102980 0x7efdb4102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:49.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.304+0000 7efdbc200640 1 --2- 172.21.15.43:0/4077231734 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efdb4101790 0x7efdb4101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:49.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.304+0000 7efdbc200640 1 -- 172.21.15.43:0/4077231734 >> 172.21.15.43:0/4077231734 conn(0x7efdb40fcce0 msgr2=0x7efdb40ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:49.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.304+0000 7efdbc200640 1 -- 172.21.15.43:0/4077231734 shutdown_connections 2024-03-28T22:36:49.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.304+0000 7efdbc200640 1 -- 172.21.15.43:0/4077231734 wait complete. 2024-03-28T22:36:49.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.306+0000 7efdbc200640 1 Processor -- start 2024-03-28T22:36:49.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.306+0000 7efdbc200640 1 -- start start 2024-03-28T22:36:49.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.306+0000 7efdbc200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efdb4101790 0x7efdb4199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:49.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.306+0000 7efdbc200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdb4102980 0x7efdb419a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:49.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.306+0000 7efdbc200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efdb4103340 0x7efdb41a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:49.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.306+0000 7efdbc200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7efdb410d440 con 0x7efdb4103340 2024-03-28T22:36:49.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.306+0000 7efdbc200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7efdb410d2c0 con 0x7efdb4102980 2024-03-28T22:36:49.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.306+0000 7efdbc200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7efdb410d5c0 con 0x7efdb4101790 2024-03-28T22:36:49.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.306+0000 7efdb8e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdb4102980 0x7efdb419a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:49.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.306+0000 7efdb8e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdb4102980 0x7efdb419a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:43910/0 (socket says 172.21.15.43:43910) 2024-03-28T22:36:49.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.306+0000 7efdb8e00640 1 -- 172.21.15.43:0/1500670552 learned_addr learned my addr 172.21.15.43:0/1500670552 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:49.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.306+0000 7efdb9800640 1 --2- 172.21.15.43:0/1500670552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efdb4101790 0x7efdb4199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:49.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.307+0000 7efdba200640 1 --2- 172.21.15.43:0/1500670552 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efdb4103340 0x7efdb41a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:49.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.307+0000 7efdb8e00640 1 -- 172.21.15.43:0/1500670552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efdb4101790 msgr2=0x7efdb4199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:49.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.307+0000 7efdb8e00640 1 --2- 172.21.15.43:0/1500670552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efdb4101790 0x7efdb4199ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:49.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.307+0000 7efdb8e00640 1 -- 172.21.15.43:0/1500670552 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efdb4103340 msgr2=0x7efdb41a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:49.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.307+0000 7efdb8e00640 1 --2- 172.21.15.43:0/1500670552 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efdb4103340 0x7efdb41a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:49.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.307+0000 7efdb8e00640 1 -- 172.21.15.43:0/1500670552 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdb41a17c0 con 0x7efdb4102980 2024-03-28T22:36:49.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.307+0000 7efdb9800640 1 --2- 172.21.15.43:0/1500670552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efdb4101790 0x7efdb4199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:49.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.307+0000 7efdb8e00640 1 --2- 172.21.15.43:0/1500670552 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdb4102980 0x7efdb419a0e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7efda000d950 tx=0x7efda000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:49.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.307+0000 7efdb2000640 1 -- 172.21.15.43:0/1500670552 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda0017030 con 0x7efdb4102980 2024-03-28T22:36:49.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.307+0000 7efdb2000640 1 -- 172.21.15.43:0/1500670552 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efda0004020 con 0x7efdb4102980 2024-03-28T22:36:49.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.307+0000 7efdb2000640 1 -- 172.21.15.43:0/1500670552 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda00047a0 con 0x7efdb4102980 2024-03-28T22:36:49.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.308+0000 7efdbc200640 1 -- 172.21.15.43:0/1500670552 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdb41a1ab0 con 0x7efdb4102980 2024-03-28T22:36:49.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.308+0000 7efdbc200640 1 -- 172.21.15.43:0/1500670552 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdb41a2050 con 0x7efdb4102980 2024-03-28T22:36:49.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.309+0000 7efdbc200640 1 -- 172.21.15.43:0/1500670552 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdb41a2280 con 0x7efdb4102980 2024-03-28T22:36:49.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.311+0000 7efdb2000640 1 -- 172.21.15.43:0/1500670552 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7efda00041d0 con 0x7efdb4102980 2024-03-28T22:36:49.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.313+0000 7efdb2000640 1 --2- 172.21.15.43:0/1500670552 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7efd9c0a1ea0 0x7efd9c0a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:49.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.313+0000 7efdb2000640 1 -- 172.21.15.43:0/1500670552 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7efda00cdb10 con 0x7efdb4102980 2024-03-28T22:36:49.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.313+0000 7efdb9800640 1 --2- 172.21.15.43:0/1500670552 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7efd9c0a1ea0 0x7efd9c0a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:49.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.314+0000 7efdb9800640 1 --2- 172.21.15.43:0/1500670552 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7efd9c0a1ea0 0x7efd9c0a4360 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7efda80099a0 tx=0x7efda8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:49.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.322+0000 7efdb2000640 1 -- 172.21.15.43:0/1500670552 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7efda009ab50 con 0x7efdb4102980 2024-03-28T22:36:49.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.594+0000 7efdbc200640 1 -- 172.21.15.43:0/1500670552 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7efdb41a2670 con 0x7efdb4102980 2024-03-28T22:36:49.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.894+0000 7efdb2000640 1 -- 172.21.15.43:0/1500670552 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v29) ==== 118+0+0 (secure 0 0 0) 0x7efda009a450 con 0x7efdb4102980 2024-03-28T22:36:49.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.898+0000 7efdbc200640 1 -- 172.21.15.43:0/1500670552 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7efd9c0a1ea0 msgr2=0x7efd9c0a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:49.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.898+0000 7efdbc200640 1 --2- 172.21.15.43:0/1500670552 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7efd9c0a1ea0 0x7efd9c0a4360 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7efda80099a0 tx=0x7efda8009500 comp rx=0 tx=0).stop 2024-03-28T22:36:49.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.898+0000 7efdbc200640 1 -- 172.21.15.43:0/1500670552 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdb4102980 msgr2=0x7efdb419a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:49.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.898+0000 7efdbc200640 1 --2- 172.21.15.43:0/1500670552 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdb4102980 0x7efdb419a0e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7efda000d950 tx=0x7efda000de10 comp rx=0 tx=0).stop 2024-03-28T22:36:49.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.898+0000 7efdbc200640 1 -- 172.21.15.43:0/1500670552 shutdown_connections 2024-03-28T22:36:49.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.898+0000 7efdbc200640 1 --2- 172.21.15.43:0/1500670552 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7efd9c0a1ea0 0x7efd9c0a4360 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:49.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.898+0000 7efdbc200640 1 --2- 172.21.15.43:0/1500670552 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efdb4102980 0x7efdb419a0e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:49.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.898+0000 7efdbc200640 1 --2- 172.21.15.43:0/1500670552 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efdb4103340 0x7efdb41a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:49.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.898+0000 7efdbc200640 1 --2- 172.21.15.43:0/1500670552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efdb4101790 0x7efdb4199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:49.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.898+0000 7efdbc200640 1 -- 172.21.15.43:0/1500670552 >> 172.21.15.43:0/1500670552 conn(0x7efdb40fcce0 msgr2=0x7efdb4073980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:49.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.898+0000 7efdbc200640 1 -- 172.21.15.43:0/1500670552 shutdown_connections 2024-03-28T22:36:49.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:49.898+0000 7efdbc200640 1 -- 172.21.15.43:0/1500670552 wait complete. 2024-03-28T22:36:49.920 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-03-28T22:36:50.002 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:49 smithi043 ceph-mon[31823]: pgmap v426: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:36:50.002 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:49 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1500670552' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-28T22:36:50.002 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:49 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-28T22:36:50.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.140+0000 7f9369800640 1 -- 172.21.15.43:0/3541285612 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93641039a0 msgr2=0x7f9364103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:50.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.140+0000 7f9369800640 1 --2- 172.21.15.43:0/3541285612 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93641039a0 0x7f9364103dc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9354009900 tx=0x7f935402bc30 comp rx=0 tx=0).stop 2024-03-28T22:36:50.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.140+0000 7f9369800640 1 -- 172.21.15.43:0/3541285612 shutdown_connections 2024-03-28T22:36:50.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.140+0000 7f9369800640 1 --2- 172.21.15.43:0/3541285612 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9364105550 0x7f936410bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:50.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.140+0000 7f9369800640 1 --2- 172.21.15.43:0/3541285612 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9364104b90 0x7f9364105010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:50.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.140+0000 7f9369800640 1 --2- 172.21.15.43:0/3541285612 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93641039a0 0x7f9364103dc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:50.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.140+0000 7f9369800640 1 -- 172.21.15.43:0/3541285612 >> 172.21.15.43:0/3541285612 conn(0x7f93640fef10 msgr2=0x7f9364101350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:50.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.140+0000 7f9369800640 1 -- 172.21.15.43:0/3541285612 shutdown_connections 2024-03-28T22:36:50.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.140+0000 7f9369800640 1 -- 172.21.15.43:0/3541285612 wait complete. 2024-03-28T22:36:50.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.141+0000 7f9369800640 1 Processor -- start 2024-03-28T22:36:50.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.142+0000 7f9369800640 1 -- start start 2024-03-28T22:36:50.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.142+0000 7f9369800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f93641039a0 0x7f9364199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:50.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.142+0000 7f9369800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9364104b90 0x7f936419a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:50.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.142+0000 7f9369800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9364105550 0x7f93641a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:50.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.142+0000 7f9369800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f936410f670 con 0x7f9364104b90 2024-03-28T22:36:50.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.142+0000 7f9369800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f936410f4f0 con 0x7f93641039a0 2024-03-28T22:36:50.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.142+0000 7f9369800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f936410f7f0 con 0x7f9364105550 2024-03-28T22:36:50.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.142+0000 7f9362a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f93641039a0 0x7f9364199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:50.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.142+0000 7f9362a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f93641039a0 0x7f9364199bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:43932/0 (socket says 172.21.15.43:43932) 2024-03-28T22:36:50.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.142+0000 7f9362a00640 1 -- 172.21.15.43:0/1086230719 learned_addr learned my addr 172.21.15.43:0/1086230719 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:50.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.142+0000 7f935a000640 1 --2- 172.21.15.43:0/1086230719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9364104b90 0x7f936419a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:50.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.142+0000 7f9363400640 1 --2- 172.21.15.43:0/1086230719 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9364105550 0x7f93641a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:50.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.143+0000 7f9362a00640 1 -- 172.21.15.43:0/1086230719 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9364105550 msgr2=0x7f93641a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:50.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.143+0000 7f9362a00640 1 --2- 172.21.15.43:0/1086230719 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9364105550 0x7f93641a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:50.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.143+0000 7f9362a00640 1 -- 172.21.15.43:0/1086230719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9364104b90 msgr2=0x7f936419a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:50.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.143+0000 7f9362a00640 1 --2- 172.21.15.43:0/1086230719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9364104b90 0x7f936419a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:50.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.143+0000 7f9362a00640 1 -- 172.21.15.43:0/1086230719 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93641a17e0 con 0x7f93641039a0 2024-03-28T22:36:50.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.143+0000 7f935a000640 1 --2- 172.21.15.43:0/1086230719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9364104b90 0x7f936419a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:36:50.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.143+0000 7f9362a00640 1 --2- 172.21.15.43:0/1086230719 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f93641039a0 0x7f9364199bc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9354000c00 tx=0x7f935402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:50.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.143+0000 7f935be00640 1 -- 172.21.15.43:0/1086230719 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9354043020 con 0x7f93641039a0 2024-03-28T22:36:50.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.143+0000 7f935be00640 1 -- 172.21.15.43:0/1086230719 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93540054e0 con 0x7f93641039a0 2024-03-28T22:36:50.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.143+0000 7f935be00640 1 -- 172.21.15.43:0/1086230719 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9354003c40 con 0x7f93641039a0 2024-03-28T22:36:50.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.143+0000 7f9369800640 1 -- 172.21.15.43:0/1086230719 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93641a1a70 con 0x7f93641039a0 2024-03-28T22:36:50.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.143+0000 7f9369800640 1 -- 172.21.15.43:0/1086230719 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93641a1f70 con 0x7f93641039a0 2024-03-28T22:36:50.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.144+0000 7f9369800640 1 -- 172.21.15.43:0/1086230719 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f936410abf0 con 0x7f93641039a0 2024-03-28T22:36:50.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.146+0000 7f935be00640 1 -- 172.21.15.43:0/1086230719 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f9354046020 con 0x7f93641039a0 2024-03-28T22:36:50.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:49 smithi172 ceph-mon[36566]: pgmap v426: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:36:50.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:49 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1500670552' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-28T22:36:50.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:49 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-28T22:36:50.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.147+0000 7f935be00640 1 --2- 172.21.15.43:0/1086230719 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f93440a1ea0 0x7f93440a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:50.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.147+0000 7f935be00640 1 -- 172.21.15.43:0/1086230719 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f93540f0180 con 0x7f93641039a0 2024-03-28T22:36:50.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.147+0000 7f935a000640 1 --2- 172.21.15.43:0/1086230719 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f93440a1ea0 0x7f93440a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:50.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.148+0000 7f935a000640 1 --2- 172.21.15.43:0/1086230719 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f93440a1ea0 0x7f93440a4360 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f936419b0d0 tx=0x7f934c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:50.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.151+0000 7f935be00640 1 -- 172.21.15.43:0/1086230719 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f935404d020 con 0x7f93641039a0 2024-03-28T22:36:50.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:49 smithi082 ceph-mon[36471]: pgmap v426: 129 pgs: 129 active+clean; 646 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:36:50.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:49 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1500670552' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-28T22:36:50.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:49 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-28T22:36:50.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.379+0000 7f9369800640 1 -- 172.21.15.43:0/1086230719 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f9364061c30 con 0x7f93440a1ea0 2024-03-28T22:36:50.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.406+0000 7f935be00640 1 -- 172.21.15.43:0/1086230719 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7f9364061c30 con 0x7f93440a1ea0 2024-03-28T22:36:50.406 INFO:teuthology.orchestra.run.smithi043.stdout:{} 2024-03-28T22:36:50.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.409+0000 7f9369800640 1 -- 172.21.15.43:0/1086230719 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f93440a1ea0 msgr2=0x7f93440a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:50.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.409+0000 7f9369800640 1 --2- 172.21.15.43:0/1086230719 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f93440a1ea0 0x7f93440a4360 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f936419b0d0 tx=0x7f934c005f90 comp rx=0 tx=0).stop 2024-03-28T22:36:50.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.409+0000 7f9369800640 1 -- 172.21.15.43:0/1086230719 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f93641039a0 msgr2=0x7f9364199bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:50.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.409+0000 7f9369800640 1 --2- 172.21.15.43:0/1086230719 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f93641039a0 0x7f9364199bc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9354000c00 tx=0x7f935402c650 comp rx=0 tx=0).stop 2024-03-28T22:36:50.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.409+0000 7f9369800640 1 -- 172.21.15.43:0/1086230719 shutdown_connections 2024-03-28T22:36:50.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.409+0000 7f9369800640 1 --2- 172.21.15.43:0/1086230719 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9364105550 0x7f93641a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:50.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.409+0000 7f9369800640 1 --2- 172.21.15.43:0/1086230719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9364104b90 0x7f936419a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:50.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.409+0000 7f9369800640 1 --2- 172.21.15.43:0/1086230719 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f93440a1ea0 0x7f93440a4360 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:50.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.409+0000 7f9369800640 1 --2- 172.21.15.43:0/1086230719 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f93641039a0 0x7f9364199bc0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:50.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.409+0000 7f9369800640 1 -- 172.21.15.43:0/1086230719 >> 172.21.15.43:0/1086230719 conn(0x7f93640fef10 msgr2=0x7f9364107dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:50.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.410+0000 7f9369800640 1 -- 172.21.15.43:0/1086230719 shutdown_connections 2024-03-28T22:36:50.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.410+0000 7f9369800640 1 -- 172.21.15.43:0/1086230719 wait complete. 2024-03-28T22:36:50.428 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-28T22:36:50.439 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-28T22:36:50.492 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:36:50.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.720+0000 7f16db200640 1 -- 172.21.15.43:0/973210109 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f16dc1037a0 msgr2=0x7f16dc103bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:50.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.720+0000 7f16db200640 1 --2- 172.21.15.43:0/973210109 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f16dc1037a0 0x7f16dc103bc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f16cc009900 tx=0x7f16cc02bc30 comp rx=0 tx=0).stop 2024-03-28T22:36:50.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.720+0000 7f16db200640 1 -- 172.21.15.43:0/973210109 shutdown_connections 2024-03-28T22:36:50.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.720+0000 7f16db200640 1 --2- 172.21.15.43:0/973210109 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16dc105350 0x7f16dc10bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:50.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.720+0000 7f16db200640 1 --2- 172.21.15.43:0/973210109 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16dc104990 0x7f16dc104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:50.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.720+0000 7f16db200640 1 --2- 172.21.15.43:0/973210109 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f16dc1037a0 0x7f16dc103bc0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:50.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.720+0000 7f16db200640 1 -- 172.21.15.43:0/973210109 >> 172.21.15.43:0/973210109 conn(0x7f16dc0fed30 msgr2=0x7f16dc101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:50.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.720+0000 7f16db200640 1 -- 172.21.15.43:0/973210109 shutdown_connections 2024-03-28T22:36:50.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.720+0000 7f16db200640 1 -- 172.21.15.43:0/973210109 wait complete. 2024-03-28T22:36:50.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.721+0000 7f16db200640 1 Processor -- start 2024-03-28T22:36:50.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.721+0000 7f16db200640 1 -- start start 2024-03-28T22:36:50.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.722+0000 7f16db200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16dc1037a0 0x7f16dc199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:50.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.722+0000 7f16db200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f16dc104990 0x7f16dc199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:50.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.722+0000 7f16db200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16dc105350 0x7f16dc1a1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:50.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.722+0000 7f16db200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f16dc10f410 con 0x7f16dc105350 2024-03-28T22:36:50.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.722+0000 7f16db200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f16dc10f290 con 0x7f16dc104990 2024-03-28T22:36:50.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.722+0000 7f16db200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f16dc10f590 con 0x7f16dc1037a0 2024-03-28T22:36:50.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.722+0000 7f16d9e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16dc1037a0 0x7f16dc199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:50.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.722+0000 7f16d9400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f16dc104990 0x7f16dc199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:50.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.722+0000 7f16d9400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f16dc104990 0x7f16dc199fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:43980/0 (socket says 172.21.15.43:43980) 2024-03-28T22:36:50.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.722+0000 7f16d9400640 1 -- 172.21.15.43:0/1577938270 learned_addr learned my addr 172.21.15.43:0/1577938270 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:50.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.722+0000 7f16d9e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16dc1037a0 0x7f16dc199a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:55152/0 (socket says 172.21.15.43:55152) 2024-03-28T22:36:50.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.722+0000 7f16da800640 1 --2- 172.21.15.43:0/1577938270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16dc105350 0x7f16dc1a1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:50.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.722+0000 7f16d9400640 1 -- 172.21.15.43:0/1577938270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16dc1037a0 msgr2=0x7f16dc199a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:50.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.722+0000 7f16d9400640 1 --2- 172.21.15.43:0/1577938270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16dc1037a0 0x7f16dc199a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:50.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.722+0000 7f16d9400640 1 -- 172.21.15.43:0/1577938270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16dc105350 msgr2=0x7f16dc1a1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:50.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.722+0000 7f16d9400640 1 --2- 172.21.15.43:0/1577938270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16dc105350 0x7f16dc1a1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:50.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.722+0000 7f16d9400640 1 -- 172.21.15.43:0/1577938270 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16dc1a1680 con 0x7f16dc104990 2024-03-28T22:36:50.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.722+0000 7f16da800640 1 --2- 172.21.15.43:0/1577938270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16dc105350 0x7f16dc1a1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:36:50.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.723+0000 7f16d9400640 1 --2- 172.21.15.43:0/1577938270 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f16dc104990 0x7f16dc199fa0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f16c800ea80 tx=0x7f16c800ef40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:50.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.723+0000 7f16d2a00640 1 -- 172.21.15.43:0/1577938270 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c800ce40 con 0x7f16dc104990 2024-03-28T22:36:50.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.723+0000 7f16d2a00640 1 -- 172.21.15.43:0/1577938270 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16c8005020 con 0x7f16dc104990 2024-03-28T22:36:50.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.723+0000 7f16d2a00640 1 -- 172.21.15.43:0/1577938270 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c8011760 con 0x7f16dc104990 2024-03-28T22:36:50.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.723+0000 7f16db200640 1 -- 172.21.15.43:0/1577938270 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16dc1a1970 con 0x7f16dc104990 2024-03-28T22:36:50.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.723+0000 7f16db200640 1 -- 172.21.15.43:0/1577938270 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16dc1a1e70 con 0x7f16dc104990 2024-03-28T22:36:50.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.724+0000 7f16db200640 1 -- 172.21.15.43:0/1577938270 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16dc10ab40 con 0x7f16dc104990 2024-03-28T22:36:50.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.727+0000 7f16d2a00640 1 -- 172.21.15.43:0/1577938270 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 18) ==== 136756+0+0 (secure 0 0 0) 0x7f16c80051d0 con 0x7f16dc104990 2024-03-28T22:36:50.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.728+0000 7f16d2a00640 1 --2- 172.21.15.43:0/1577938270 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f16c00a1e00 0x7f16c00a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:50.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.728+0000 7f16d2a00640 1 -- 172.21.15.43:0/1577938270 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f16c80ce760 con 0x7f16dc104990 2024-03-28T22:36:50.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.728+0000 7f16d9e00640 1 --2- 172.21.15.43:0/1577938270 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f16c00a1e00 0x7f16c00a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:50.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.729+0000 7f16d9e00640 1 --2- 172.21.15.43:0/1577938270 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f16c00a1e00 0x7f16c00a42c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f16cc000c00 tx=0x7f16cc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:50.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.737+0000 7f16d2a00640 1 -- 172.21.15.43:0/1577938270 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f16c8016030 con 0x7f16dc104990 2024-03-28T22:36:50.766 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:36:50 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:36:50.388+0000 7f1d08400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:36:50.766 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:36:50 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:36:50.388+0000 7f1d08400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:36:50.766 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:36:50 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:36:50.388+0000 7f1d08400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:36:50.766 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:36:50 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:36:50.388+0000 7f1d08400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:36:50.766 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:36:50 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:36:50.388+0000 7f1d08400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:36:50.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.923+0000 7f16d2a00640 1 -- 172.21.15.43:0/1577938270 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7f16c8097710 con 0x7f16dc104990 2024-03-28T22:36:50.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.957+0000 7f16db200640 1 -- 172.21.15.43:0/1577938270 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f16dc062350 con 0x7f16dc104990 2024-03-28T22:36:50.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.959+0000 7f16d2a00640 1 -- 172.21.15.43:0/1577938270 <== mon.2 v2:172.21.15.82:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+24712 (secure 0 0 0) 0x7f16c80c4fa0 con 0x7f16dc104990 2024-03-28T22:36:50.958 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:36:50.958 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":96,"fsid":"c2276428-ed51-11ee-95d0-87774f69a715","created":"2024-03-28T22:24:19.950084+0000","modified":"2024-03-28T22:35:30.237654+0000","last_up_change":"2024-03-28T22:32:31.078737+0000","last_in_change":"2024-03-28T22:32:10.147097+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:28:14.713290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:33:27.167435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:33:28.411825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T22:33:32.523122+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b855adcd-786c-4e42-b8fe-a515a77ce979","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6803","nonce":2826124409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6805","nonce":2826124409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6809","nonce":2826124409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6807","nonce":2826124409}]},"public_addr":"172.21.15.43:6803/2826124409","cluster_addr":"172.21.15.43:6805/2826124409","heartbeat_back_addr":"172.21.15.43:6809/2826124409","heartbeat_front_addr":"172.21.15.43:6807/2826124409","state":["exists","up"]},{"osd":1,"uuid":"031c143d-8ad1-4b68-8db9-3c717137e3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6801","nonce":3043263644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6803","nonce":3043263644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6807","nonce":3043263644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6805","nonce":3043263644}]},"public_addr":"172.21.15.82:6801/3043263644","cluster_addr":"172.21.15.82:6803/3043263644","heartbeat_back_addr":"172.21.15.82:6807/3043263644","heartbeat_front_addr":"172.21.15.82:6805/3043263644","state":["exists","up"]},{"osd":2,"uuid":"20632bbb-f07a-4ca1-b0c4-ff9d508e211c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6801","nonce":4213953645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6803","nonce":4213953645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6807","nonce":4213953645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6805","nonce":4213953645}]},"public_addr":"172.21.15.172:6801/4213953645","cluster_addr":"172.21.15.172:6803/4213953645","heartbeat_back_addr":"172.21.15.172:6807/4213953645","heartbeat_front_addr":"172.21.15.172:6805/4213953645","state":["exists","up"]},{"osd":3,"uuid":"9a1f2312-4dbd-475b-8736-d5ebc8788e09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6811","nonce":1597280920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6813","nonce":1597280920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6817","nonce":1597280920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6815","nonce":1597280920}]},"public_addr":"172.21.15.43:6811/1597280920","cluster_addr":"172.21.15.43:6813/1597280920","heartbeat_back_addr":"172.21.15.43:6817/1597280920","heartbeat_front_addr":"172.21.15.43:6815/1597280920","state":["exists","up"]},{"osd":4,"uuid":"29140ead-7075-42b2-90c1-27cae0b29ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6809","nonce":215451895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6811","nonce":215451895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6815","nonce":215451895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6813","nonce":215451895}]},"public_addr":"172.21.15.82:6809/215451895","cluster_addr":"172.21.15.82:6811/215451895","heartbeat_back_addr":"172.21.15.82:6815/215451895","heartbeat_front_addr":"172.21.15.82:6813/215451895","state":["exists","up"]},{"osd":5,"uuid":"59644c61-93dc-4536-b2ad-0256b24e5cb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6809","nonce":1576148933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6811","nonce":1576148933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6815","nonce":1576148933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6813","nonce":1576148933}]},"public_addr":"172.21.15.172:6809/1576148933","cluster_addr":"172.21.15.172:6811/1576148933","heartbeat_back_addr":"172.21.15.172:6815/1576148933","heartbeat_front_addr":"172.21.15.172:6813/1576148933","state":["exists","up"]},{"osd":6,"uuid":"22d54e57-c3f1-4ecf-9921-8e815b48f92a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6819","nonce":791044279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6821","nonce":791044279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6825","nonce":791044279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6823","nonce":791044279}]},"public_addr":"172.21.15.43:6819/791044279","cluster_addr":"172.21.15.43:6821/791044279","heartbeat_back_addr":"172.21.15.43:6825/791044279","heartbeat_front_addr":"172.21.15.43:6823/791044279","state":["exists","up"]},{"osd":7,"uuid":"248cd1e2-0d6b-452f-a0a8-c51fe1b975ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6817","nonce":2157777323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6819","nonce":2157777323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6823","nonce":2157777323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6821","nonce":2157777323}]},"public_addr":"172.21.15.82:6817/2157777323","cluster_addr":"172.21.15.82:6819/2157777323","heartbeat_back_addr":"172.21.15.82:6823/2157777323","heartbeat_front_addr":"172.21.15.82:6821/2157777323","state":["exists","up"]},{"osd":8,"uuid":"d98aab6e-8bae-4c24-906c-4f492f047b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6817","nonce":1095328248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6819","nonce":1095328248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6823","nonce":1095328248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6821","nonce":1095328248}]},"public_addr":"172.21.15.172:6817/1095328248","cluster_addr":"172.21.15.172:6819/1095328248","heartbeat_back_addr":"172.21.15.172:6823/1095328248","heartbeat_front_addr":"172.21.15.172:6821/1095328248","state":["exists","up"]},{"osd":9,"uuid":"19aa383e-a0f6-45ea-96cc-e7956535caa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6827","nonce":3657791498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6829","nonce":3657791498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6833","nonce":3657791498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6831","nonce":3657791498}]},"public_addr":"172.21.15.43:6827/3657791498","cluster_addr":"172.21.15.43:6829/3657791498","heartbeat_back_addr":"172.21.15.43:6833/3657791498","heartbeat_front_addr":"172.21.15.43:6831/3657791498","state":["exists","up"]},{"osd":10,"uuid":"666977e5-eaf7-41b9-a18f-c380e149345d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6825","nonce":3454954047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6827","nonce":3454954047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6831","nonce":3454954047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6829","nonce":3454954047}]},"public_addr":"172.21.15.82:6825/3454954047","cluster_addr":"172.21.15.82:6827/3454954047","heartbeat_back_addr":"172.21.15.82:6831/3454954047","heartbeat_front_addr":"172.21.15.82:6829/3454954047","state":["exists","up"]},{"osd":11,"uuid":"615cf3be-8076-470e-ad09-c880e7593717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6825","nonce":2563517144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6827","nonce":2563517144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6831","nonce":2563517144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6829","nonce":2563517144}]},"public_addr":"172.21.15.172:6825/2563517144","cluster_addr":"172.21.15.172:6827/2563517144","heartbeat_back_addr":"172.21.15.172:6831/2563517144","heartbeat_front_addr":"172.21.15.172:6829/2563517144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:21.727405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:47.100259+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:10.615438+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:43.632144+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:09.487940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:34.604463+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:09.594452+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:35.709795+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:01.184202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:35.455322+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:02.413166+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:28.550754+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.9","mappings":[{"from":5,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/578791143":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/4029313215":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/1574178482":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6801/3517712438":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6800/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/1375099638":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/2710117652":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6801/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/3215634168":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6800/3517712438":"2024-03-29T22:25:28.008215+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:36:50.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.962+0000 7f16db200640 1 -- 172.21.15.43:0/1577938270 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f16c00a1e00 msgr2=0x7f16c00a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:50.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.962+0000 7f16db200640 1 --2- 172.21.15.43:0/1577938270 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f16c00a1e00 0x7f16c00a42c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f16cc000c00 tx=0x7f16cc005fd0 comp rx=0 tx=0).stop 2024-03-28T22:36:50.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.962+0000 7f16db200640 1 -- 172.21.15.43:0/1577938270 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f16dc104990 msgr2=0x7f16dc199fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:50.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.962+0000 7f16db200640 1 --2- 172.21.15.43:0/1577938270 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f16dc104990 0x7f16dc199fa0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f16c800ea80 tx=0x7f16c800ef40 comp rx=0 tx=0).stop 2024-03-28T22:36:50.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.962+0000 7f16db200640 1 -- 172.21.15.43:0/1577938270 shutdown_connections 2024-03-28T22:36:50.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.962+0000 7f16db200640 1 --2- 172.21.15.43:0/1577938270 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f16c00a1e00 0x7f16c00a42c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:50.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.962+0000 7f16db200640 1 --2- 172.21.15.43:0/1577938270 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f16dc104990 0x7f16dc199fa0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:50.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.962+0000 7f16db200640 1 --2- 172.21.15.43:0/1577938270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16dc105350 0x7f16dc1a1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:50.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.962+0000 7f16db200640 1 --2- 172.21.15.43:0/1577938270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16dc1037a0 0x7f16dc199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:50.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.962+0000 7f16db200640 1 -- 172.21.15.43:0/1577938270 >> 172.21.15.43:0/1577938270 conn(0x7f16dc0fed30 msgr2=0x7f16dc107d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:50.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.962+0000 7f16db200640 1 -- 172.21.15.43:0/1577938270 shutdown_connections 2024-03-28T22:36:50.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:50.962+0000 7f16db200640 1 -- 172.21.15.43:0/1577938270 wait complete. 2024-03-28T22:36:50.980 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T22:28:14.713290+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T22:33:27.167435+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T22:33:28.411825+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T22:33:32.523122+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T22:36:50.981 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T22:36:51.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:50 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-28T22:36:51.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:50 smithi082 ceph-mon[36471]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T22:36:51.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.206+0000 7fd214a00640 1 -- 172.21.15.43:0/3594191042 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd210103870 msgr2=0x7fd210103c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:51.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.206+0000 7fd214a00640 1 --2- 172.21.15.43:0/3594191042 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd210103870 0x7fd210103c90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd200009ca0 tx=0x7fd20002c100 comp rx=0 tx=0).stop 2024-03-28T22:36:51.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.206+0000 7fd214a00640 1 -- 172.21.15.43:0/3594191042 shutdown_connections 2024-03-28T22:36:51.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.206+0000 7fd214a00640 1 --2- 172.21.15.43:0/3594191042 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd210104a60 0x7fd210104ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:51.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.206+0000 7fd214a00640 1 --2- 172.21.15.43:0/3594191042 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd210105420 0x7fd21010bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:51.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.206+0000 7fd214a00640 1 --2- 172.21.15.43:0/3594191042 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd210103870 0x7fd210103c90 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:51.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.206+0000 7fd214a00640 1 -- 172.21.15.43:0/3594191042 >> 172.21.15.43:0/3594191042 conn(0x7fd2100fee00 msgr2=0x7fd210101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:51.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.206+0000 7fd214a00640 1 -- 172.21.15.43:0/3594191042 shutdown_connections 2024-03-28T22:36:51.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.206+0000 7fd214a00640 1 -- 172.21.15.43:0/3594191042 wait complete. 2024-03-28T22:36:51.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.208+0000 7fd214a00640 1 Processor -- start 2024-03-28T22:36:51.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.208+0000 7fd214a00640 1 -- start start 2024-03-28T22:36:51.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.208+0000 7fd214a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd210103870 0x7fd210199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:51.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.208+0000 7fd214a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd210104a60 0x7fd21019a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:51.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.208+0000 7fd214a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd210105420 0x7fd2101a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:51.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.208+0000 7fd214a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd21010f4c0 con 0x7fd210105420 2024-03-28T22:36:51.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.208+0000 7fd214a00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd21010f340 con 0x7fd210103870 2024-03-28T22:36:51.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.208+0000 7fd214a00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd21010f640 con 0x7fd210104a60 2024-03-28T22:36:51.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.208+0000 7fd20e800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd210103870 0x7fd210199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:51.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.209+0000 7fd20f200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd210105420 0x7fd2101a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:51.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.209+0000 7fd20e800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd210103870 0x7fd210199b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44002/0 (socket says 172.21.15.43:44002) 2024-03-28T22:36:51.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.209+0000 7fd20e800640 1 -- 172.21.15.43:0/2107330978 learned_addr learned my addr 172.21.15.43:0/2107330978 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:51.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.209+0000 7fd20de00640 1 --2- 172.21.15.43:0/2107330978 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd210104a60 0x7fd21019a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:51.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.209+0000 7fd20e800640 1 -- 172.21.15.43:0/2107330978 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd210104a60 msgr2=0x7fd21019a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:51.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.209+0000 7fd20e800640 1 --2- 172.21.15.43:0/2107330978 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd210104a60 0x7fd21019a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:51.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.209+0000 7fd20e800640 1 -- 172.21.15.43:0/2107330978 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd210105420 msgr2=0x7fd2101a1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:51.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.209+0000 7fd20e800640 1 --2- 172.21.15.43:0/2107330978 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd210105420 0x7fd2101a1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:51.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.209+0000 7fd20e800640 1 -- 172.21.15.43:0/2107330978 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2101a17b0 con 0x7fd210103870 2024-03-28T22:36:51.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.209+0000 7fd20f200640 1 --2- 172.21.15.43:0/2107330978 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd210105420 0x7fd2101a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:36:51.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.209+0000 7fd20e800640 1 --2- 172.21.15.43:0/2107330978 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd210103870 0x7fd210199b90 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd200000f80 tx=0x7fd200035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:51.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.209+0000 7fd207400640 1 -- 172.21.15.43:0/2107330978 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd200039040 con 0x7fd210103870 2024-03-28T22:36:51.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.210+0000 7fd207400640 1 -- 172.21.15.43:0/2107330978 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd200044070 con 0x7fd210103870 2024-03-28T22:36:51.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.210+0000 7fd214a00640 1 -- 172.21.15.43:0/2107330978 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2101a1a40 con 0x7fd210103870 2024-03-28T22:36:51.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.210+0000 7fd207400640 1 -- 172.21.15.43:0/2107330978 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2000044a0 con 0x7fd210103870 2024-03-28T22:36:51.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.210+0000 7fd214a00640 1 -- 172.21.15.43:0/2107330978 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2101a1f50 con 0x7fd210103870 2024-03-28T22:36:51.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.211+0000 7fd214a00640 1 -- 172.21.15.43:0/2107330978 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2100688b0 con 0x7fd210103870 2024-03-28T22:36:51.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.212+0000 7fd207400640 1 -- 172.21.15.43:0/2107330978 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7fd200004640 con 0x7fd210103870 2024-03-28T22:36:51.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.214+0000 7fd207400640 1 --2- 172.21.15.43:0/2107330978 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd1f40a1e90 0x7fd1f40a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:51.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.214+0000 7fd207400640 1 -- 172.21.15.43:0/2107330978 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7fd2000f0770 con 0x7fd210103870 2024-03-28T22:36:51.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.214+0000 7fd20de00640 1 --2- 172.21.15.43:0/2107330978 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd1f40a1e90 0x7fd1f40a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:51.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.215+0000 7fd20de00640 1 --2- 172.21.15.43:0/2107330978 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd1f40a1e90 0x7fd1f40a4350 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fd21019b0a0 tx=0x7fd1f8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:51.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.218+0000 7fd207400640 1 -- 172.21.15.43:0/2107330978 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd2000bd7c0 con 0x7fd210103870 2024-03-28T22:36:51.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:50 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-28T22:36:51.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:50 smithi043 ceph-mon[31823]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T22:36:51.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:50 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-28T22:36:51.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:50 smithi172 ceph-mon[36566]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T22:36:51.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.420+0000 7fd214a00640 1 -- 172.21.15.43:0/2107330978 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd21002d530 con 0x7fd210103870 2024-03-28T22:36:51.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.421+0000 7fd207400640 1 -- 172.21.15.43:0/2107330978 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) ==== 93+0+10 (secure 0 0 0) 0x7fd2000bd0c0 con 0x7fd210103870 2024-03-28T22:36:51.420 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-03-28T22:36:51.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.423+0000 7fd214a00640 1 -- 172.21.15.43:0/2107330978 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd1f40a1e90 msgr2=0x7fd1f40a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:51.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.423+0000 7fd214a00640 1 --2- 172.21.15.43:0/2107330978 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd1f40a1e90 0x7fd1f40a4350 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fd21019b0a0 tx=0x7fd1f8005f90 comp rx=0 tx=0).stop 2024-03-28T22:36:51.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.423+0000 7fd214a00640 1 -- 172.21.15.43:0/2107330978 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd210103870 msgr2=0x7fd210199b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:51.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.423+0000 7fd214a00640 1 --2- 172.21.15.43:0/2107330978 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd210103870 0x7fd210199b90 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd200000f80 tx=0x7fd200035d10 comp rx=0 tx=0).stop 2024-03-28T22:36:51.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.423+0000 7fd214a00640 1 -- 172.21.15.43:0/2107330978 shutdown_connections 2024-03-28T22:36:51.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.423+0000 7fd214a00640 1 --2- 172.21.15.43:0/2107330978 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd210105420 0x7fd2101a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:51.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.423+0000 7fd214a00640 1 --2- 172.21.15.43:0/2107330978 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd210104a60 0x7fd21019a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:51.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.423+0000 7fd214a00640 1 --2- 172.21.15.43:0/2107330978 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fd1f40a1e90 0x7fd1f40a4350 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:51.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.423+0000 7fd214a00640 1 --2- 172.21.15.43:0/2107330978 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd210103870 0x7fd210199b90 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:51.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.423+0000 7fd214a00640 1 -- 172.21.15.43:0/2107330978 >> 172.21.15.43:0/2107330978 conn(0x7fd2100fee00 msgr2=0x7fd210107de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:51.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.424+0000 7fd214a00640 1 -- 172.21.15.43:0/2107330978 shutdown_connections 2024-03-28T22:36:51.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.424+0000 7fd214a00640 1 -- 172.21.15.43:0/2107330978 wait complete. 2024-03-28T22:36:51.441 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T22:36:51.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.673+0000 7fc7cfe00640 1 -- 172.21.15.43:0/3583125162 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7d01020b0 msgr2=0x7fc7d01044a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:51.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.673+0000 7fc7cfe00640 1 --2- 172.21.15.43:0/3583125162 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7d01020b0 0x7fc7d01044a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc7c0009ca0 tx=0x7fc7c002c0f0 comp rx=0 tx=0).stop 2024-03-28T22:36:51.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.673+0000 7fc7cfe00640 1 -- 172.21.15.43:0/3583125162 shutdown_connections 2024-03-28T22:36:51.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.673+0000 7fc7cfe00640 1 --2- 172.21.15.43:0/3583125162 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7d0107310 0x7fc7d0109760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:51.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.673+0000 7fc7cfe00640 1 --2- 172.21.15.43:0/3583125162 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7d01049e0 0x7fc7d0106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:51.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.673+0000 7fc7cfe00640 1 --2- 172.21.15.43:0/3583125162 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7d01020b0 0x7fc7d01044a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:51.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.673+0000 7fc7cfe00640 1 -- 172.21.15.43:0/3583125162 >> 172.21.15.43:0/3583125162 conn(0x7fc7d00fbac0 msgr2=0x7fc7d00fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:51.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.673+0000 7fc7cfe00640 1 -- 172.21.15.43:0/3583125162 shutdown_connections 2024-03-28T22:36:51.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.673+0000 7fc7cfe00640 1 -- 172.21.15.43:0/3583125162 wait complete. 2024-03-28T22:36:51.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.674+0000 7fc7cfe00640 1 Processor -- start 2024-03-28T22:36:51.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.674+0000 7fc7cfe00640 1 -- start start 2024-03-28T22:36:51.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.675+0000 7fc7cfe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7d01020b0 0x7fc7d0199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:51.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.675+0000 7fc7cfe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7d01049e0 0x7fc7d019a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:51.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.675+0000 7fc7cfe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7d0107310 0x7fc7d01a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:51.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.675+0000 7fc7cfe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d010d2a0 con 0x7fc7d0107310 2024-03-28T22:36:51.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.675+0000 7fc7cfe00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d010d120 con 0x7fc7d01020b0 2024-03-28T22:36:51.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.675+0000 7fc7cfe00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d010d420 con 0x7fc7d01049e0 2024-03-28T22:36:51.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.675+0000 7fc7cea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7d01020b0 0x7fc7d0199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:51.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.675+0000 7fc7c6000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7d01049e0 0x7fc7d019a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:51.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.675+0000 7fc7cea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7d01020b0 0x7fc7d0199b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44008/0 (socket says 172.21.15.43:44008) 2024-03-28T22:36:51.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.675+0000 7fc7c6000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7d01049e0 0x7fc7d019a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:55170/0 (socket says 172.21.15.43:55170) 2024-03-28T22:36:51.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.675+0000 7fc7cea00640 1 -- 172.21.15.43:0/3169083163 learned_addr learned my addr 172.21.15.43:0/3169083163 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:51.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.675+0000 7fc7cf400640 1 --2- 172.21.15.43:0/3169083163 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7d0107310 0x7fc7d01a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:51.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.675+0000 7fc7cea00640 1 -- 172.21.15.43:0/3169083163 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7d01049e0 msgr2=0x7fc7d019a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:51.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.675+0000 7fc7cea00640 1 --2- 172.21.15.43:0/3169083163 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7d01049e0 0x7fc7d019a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:51.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.675+0000 7fc7cea00640 1 -- 172.21.15.43:0/3169083163 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7d0107310 msgr2=0x7fc7d01a1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:51.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.675+0000 7fc7cea00640 1 --2- 172.21.15.43:0/3169083163 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7d0107310 0x7fc7d01a1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:51.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.675+0000 7fc7cea00640 1 -- 172.21.15.43:0/3169083163 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7d01a17b0 con 0x7fc7d01020b0 2024-03-28T22:36:51.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.675+0000 7fc7cf400640 1 --2- 172.21.15.43:0/3169083163 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7d0107310 0x7fc7d01a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:36:51.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.676+0000 7fc7cea00640 1 --2- 172.21.15.43:0/3169083163 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7d01020b0 0x7fc7d0199b90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc7c00325f0 tx=0x7fc7c0032ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:51.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.676+0000 7fc7d4a00640 1 -- 172.21.15.43:0/3169083163 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c0041020 con 0x7fc7d01020b0 2024-03-28T22:36:51.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.676+0000 7fc7d4a00640 1 -- 172.21.15.43:0/3169083163 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7c0034cd0 con 0x7fc7d01020b0 2024-03-28T22:36:51.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.676+0000 7fc7d4a00640 1 -- 172.21.15.43:0/3169083163 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c0039040 con 0x7fc7d01020b0 2024-03-28T22:36:51.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.676+0000 7fc7cfe00640 1 -- 172.21.15.43:0/3169083163 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7d01a1a40 con 0x7fc7d01020b0 2024-03-28T22:36:51.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.676+0000 7fc7cfe00640 1 -- 172.21.15.43:0/3169083163 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7d01a1eb0 con 0x7fc7d01020b0 2024-03-28T22:36:51.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.677+0000 7fc7cfe00640 1 -- 172.21.15.43:0/3169083163 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7d0108e60 con 0x7fc7d01020b0 2024-03-28T22:36:51.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.679+0000 7fc7d4a00640 1 -- 172.21.15.43:0/3169083163 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7fc7c0044020 con 0x7fc7d01020b0 2024-03-28T22:36:51.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.680+0000 7fc7d4a00640 1 --2- 172.21.15.43:0/3169083163 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fc7b40a1e90 0x7fc7b40a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:51.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.681+0000 7fc7d4a00640 1 -- 172.21.15.43:0/3169083163 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7fc7c00effe0 con 0x7fc7d01020b0 2024-03-28T22:36:51.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.681+0000 7fc7c6000640 1 --2- 172.21.15.43:0/3169083163 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fc7b40a1e90 0x7fc7b40a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:51.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.682+0000 7fc7c6000640 1 --2- 172.21.15.43:0/3169083163 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fc7b40a1e90 0x7fc7b40a4350 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc7d019b0a0 tx=0x7fc7b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:51.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.685+0000 7fc7d4a00640 1 -- 172.21.15.43:0/3169083163 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc7c004b020 con 0x7fc7d01020b0 2024-03-28T22:36:51.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.887+0000 7fc7cfe00640 1 -- 172.21.15.43:0/3169083163 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc7d002d3e0 con 0x7fc7d01020b0 2024-03-28T22:36:51.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.888+0000 7fc7d4a00640 1 -- 172.21.15.43:0/3169083163 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) ==== 104+0+11 (secure 0 0 0) 0x7fc7c00bd0a0 con 0x7fc7d01020b0 2024-03-28T22:36:51.887 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-03-28T22:36:51.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.891+0000 7fc7cfe00640 1 -- 172.21.15.43:0/3169083163 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fc7b40a1e90 msgr2=0x7fc7b40a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:51.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.891+0000 7fc7cfe00640 1 --2- 172.21.15.43:0/3169083163 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fc7b40a1e90 0x7fc7b40a4350 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc7d019b0a0 tx=0x7fc7b8005f90 comp rx=0 tx=0).stop 2024-03-28T22:36:51.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.891+0000 7fc7cfe00640 1 -- 172.21.15.43:0/3169083163 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7d01020b0 msgr2=0x7fc7d0199b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:51.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.891+0000 7fc7cfe00640 1 --2- 172.21.15.43:0/3169083163 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7d01020b0 0x7fc7d0199b90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc7c00325f0 tx=0x7fc7c0032ec0 comp rx=0 tx=0).stop 2024-03-28T22:36:51.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.891+0000 7fc7cfe00640 1 -- 172.21.15.43:0/3169083163 shutdown_connections 2024-03-28T22:36:51.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.892+0000 7fc7cfe00640 1 --2- 172.21.15.43:0/3169083163 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7d0107310 0x7fc7d01a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:51.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.892+0000 7fc7cfe00640 1 --2- 172.21.15.43:0/3169083163 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7d01049e0 0x7fc7d019a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:51.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.892+0000 7fc7cfe00640 1 --2- 172.21.15.43:0/3169083163 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fc7b40a1e90 0x7fc7b40a4350 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:51.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.892+0000 7fc7cfe00640 1 --2- 172.21.15.43:0/3169083163 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7d01020b0 0x7fc7d0199b90 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:51.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.892+0000 7fc7cfe00640 1 -- 172.21.15.43:0/3169083163 >> 172.21.15.43:0/3169083163 conn(0x7fc7d00fbac0 msgr2=0x7fc7d01018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:51.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.892+0000 7fc7cfe00640 1 -- 172.21.15.43:0/3169083163 shutdown_connections 2024-03-28T22:36:51.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:51.892+0000 7fc7cfe00640 1 -- 172.21.15.43:0/3169083163 wait complete. 2024-03-28T22:36:51.911 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T22:36:52.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.146+0000 7f82c6c00640 1 -- 172.21.15.43:0/3260886975 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f82c0109f00 msgr2=0x7f82c010a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:52.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.146+0000 7f82c6c00640 1 --2- 172.21.15.43:0/3260886975 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f82c0109f00 0x7f82c010a340 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f82b0009900 tx=0x7f82b002bc50 comp rx=0 tx=0).stop 2024-03-28T22:36:52.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.146+0000 7f82c6c00640 1 -- 172.21.15.43:0/3260886975 shutdown_connections 2024-03-28T22:36:52.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.146+0000 7f82c6c00640 1 --2- 172.21.15.43:0/3260886975 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82c010ba90 0x7f82c01123b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:52.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.146+0000 7f82c6c00640 1 --2- 172.21.15.43:0/3260886975 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f82c010b0d0 0x7f82c010b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:52.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.146+0000 7f82c6c00640 1 --2- 172.21.15.43:0/3260886975 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f82c0109f00 0x7f82c010a340 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:52.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.146+0000 7f82c6c00640 1 -- 172.21.15.43:0/3260886975 >> 172.21.15.43:0/3260886975 conn(0x7f82c0070710 msgr2=0x7f82c0072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:52.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.146+0000 7f82c6c00640 1 -- 172.21.15.43:0/3260886975 shutdown_connections 2024-03-28T22:36:52.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.146+0000 7f82c6c00640 1 -- 172.21.15.43:0/3260886975 wait complete. 2024-03-28T22:36:52.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.148+0000 7f82c6c00640 1 Processor -- start 2024-03-28T22:36:52.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.148+0000 7f82c6c00640 1 -- start start 2024-03-28T22:36:52.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.148+0000 7f82c6c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82c0109f00 0x7f82c01a0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:52.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.148+0000 7f82c6c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f82c010b0d0 0x7f82c01a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:52.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.148+0000 7f82c6c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f82c010ba90 0x7f82c01a78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:52.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.148+0000 7f82c6c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f82c0115c00 con 0x7f82c0109f00 2024-03-28T22:36:52.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.148+0000 7f82c6c00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f82c0115a80 con 0x7f82c010b0d0 2024-03-28T22:36:52.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.148+0000 7f82c6c00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f82c0115d80 con 0x7f82c010ba90 2024-03-28T22:36:52.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.149+0000 7f82b7e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f82c010b0d0 0x7f82c01a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:52.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.149+0000 7f82c4c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f82c010ba90 0x7f82c01a78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:52.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.149+0000 7f82b7e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f82c010b0d0 0x7f82c01a08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44018/0 (socket says 172.21.15.43:44018) 2024-03-28T22:36:52.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.149+0000 7f82c4c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f82c010ba90 0x7f82c01a78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:55184/0 (socket says 172.21.15.43:55184) 2024-03-28T22:36:52.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.149+0000 7f82b7e00640 1 -- 172.21.15.43:0/1879588533 learned_addr learned my addr 172.21.15.43:0/1879588533 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:52.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.149+0000 7f82bfe00640 1 --2- 172.21.15.43:0/1879588533 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82c0109f00 0x7f82c01a0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:52.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.149+0000 7f82b7e00640 1 -- 172.21.15.43:0/1879588533 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f82c010ba90 msgr2=0x7f82c01a78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:52.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.149+0000 7f82b7e00640 1 --2- 172.21.15.43:0/1879588533 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f82c010ba90 0x7f82c01a78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:52.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.149+0000 7f82b7e00640 1 -- 172.21.15.43:0/1879588533 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82c0109f00 msgr2=0x7f82c01a0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:52.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.149+0000 7f82b7e00640 1 --2- 172.21.15.43:0/1879588533 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82c0109f00 0x7f82c01a0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:52.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.149+0000 7f82b7e00640 1 -- 172.21.15.43:0/1879588533 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82c01a7f60 con 0x7f82c010b0d0 2024-03-28T22:36:52.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.149+0000 7f82b7e00640 1 --2- 172.21.15.43:0/1879588533 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f82c010b0d0 0x7f82c01a08d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f82a800ea60 tx=0x7f82a800ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:52.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.150+0000 7f82bd600640 1 -- 172.21.15.43:0/1879588533 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a800cd20 con 0x7f82c010b0d0 2024-03-28T22:36:52.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.150+0000 7f82bd600640 1 -- 172.21.15.43:0/1879588533 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82a80035f0 con 0x7f82c010b0d0 2024-03-28T22:36:52.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.150+0000 7f82bd600640 1 -- 172.21.15.43:0/1879588533 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a8010690 con 0x7f82c010b0d0 2024-03-28T22:36:52.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.150+0000 7f82c6c00640 1 -- 172.21.15.43:0/1879588533 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82c01a8250 con 0x7f82c010b0d0 2024-03-28T22:36:52.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.150+0000 7f82c6c00640 1 -- 172.21.15.43:0/1879588533 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82c01a8750 con 0x7f82c010b0d0 2024-03-28T22:36:52.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.151+0000 7f82c6c00640 1 -- 172.21.15.43:0/1879588533 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82c0111400 con 0x7f82c010b0d0 2024-03-28T22:36:52.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.153+0000 7f82bd600640 1 -- 172.21.15.43:0/1879588533 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7f82a80037a0 con 0x7f82c010b0d0 2024-03-28T22:36:52.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.154+0000 7f82bd600640 1 --2- 172.21.15.43:0/1879588533 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f82a40a1e90 0x7f82a40a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:52.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.154+0000 7f82bd600640 1 -- 172.21.15.43:0/1879588533 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f82a80ce560 con 0x7f82c010b0d0 2024-03-28T22:36:52.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.155+0000 7f82bfe00640 1 --2- 172.21.15.43:0/1879588533 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f82a40a1e90 0x7f82a40a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:52.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.155+0000 7f82bfe00640 1 --2- 172.21.15.43:0/1879588533 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f82a40a1e90 0x7f82a40a4350 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f82b002c190 tx=0x7f82b0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:52.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.158+0000 7f82bd600640 1 -- 172.21.15.43:0/1879588533 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f82a809b440 con 0x7f82c010b0d0 2024-03-28T22:36:52.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:51 smithi082 ceph-mon[36471]: pgmap v427: 129 pgs: 129 active+clean; 647 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-28T22:36:52.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:51 smithi082 ceph-mon[36471]: from='client.25475 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:36:52.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:51 smithi082 ceph-mon[36471]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-03-28T22:36:52.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:51 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1577938270' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:36:52.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:51 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2107330978' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:36:52.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:51 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3169083163' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:36:52.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:51 smithi043 ceph-mon[31823]: pgmap v427: 129 pgs: 129 active+clean; 647 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-28T22:36:52.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:51 smithi043 ceph-mon[31823]: from='client.25475 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:36:52.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:51 smithi043 ceph-mon[31823]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-03-28T22:36:52.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:51 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1577938270' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:36:52.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:51 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2107330978' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:36:52.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:51 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3169083163' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:36:52.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.369+0000 7f82c6c00640 1 -- 172.21.15.43:0/1879588533 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f82c002d3c0 con 0x7f82c010b0d0 2024-03-28T22:36:52.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.369+0000 7f82bd600640 1 -- 172.21.15.43:0/1879588533 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) ==== 100+0+11 (secure 0 0 0) 0x7f82a809ad40 con 0x7f82c010b0d0 2024-03-28T22:36:52.368 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-03-28T22:36:52.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.372+0000 7f82c6c00640 1 -- 172.21.15.43:0/1879588533 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f82a40a1e90 msgr2=0x7f82a40a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:52.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.372+0000 7f82c6c00640 1 --2- 172.21.15.43:0/1879588533 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f82a40a1e90 0x7f82a40a4350 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f82b002c190 tx=0x7f82b0000bc0 comp rx=0 tx=0).stop 2024-03-28T22:36:52.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.372+0000 7f82c6c00640 1 -- 172.21.15.43:0/1879588533 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f82c010b0d0 msgr2=0x7f82c01a08d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:52.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.372+0000 7f82c6c00640 1 --2- 172.21.15.43:0/1879588533 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f82c010b0d0 0x7f82c01a08d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f82a800ea60 tx=0x7f82a800ee60 comp rx=0 tx=0).stop 2024-03-28T22:36:52.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.372+0000 7f82c6c00640 1 -- 172.21.15.43:0/1879588533 shutdown_connections 2024-03-28T22:36:52.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.372+0000 7f82c6c00640 1 --2- 172.21.15.43:0/1879588533 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f82a40a1e90 0x7f82a40a4350 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:52.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.372+0000 7f82c6c00640 1 --2- 172.21.15.43:0/1879588533 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f82c010b0d0 0x7f82c01a08d0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:52.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.372+0000 7f82c6c00640 1 --2- 172.21.15.43:0/1879588533 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f82c010ba90 0x7f82c01a78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:52.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.372+0000 7f82c6c00640 1 --2- 172.21.15.43:0/1879588533 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82c0109f00 0x7f82c01a0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:52.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.372+0000 7f82c6c00640 1 -- 172.21.15.43:0/1879588533 >> 172.21.15.43:0/1879588533 conn(0x7f82c0070710 msgr2=0x7f82c010e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:52.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.372+0000 7f82c6c00640 1 -- 172.21.15.43:0/1879588533 shutdown_connections 2024-03-28T22:36:52.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.372+0000 7f82c6c00640 1 -- 172.21.15.43:0/1879588533 wait complete. 2024-03-28T22:36:52.389 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T22:36:52.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:51 smithi172 ceph-mon[36566]: pgmap v427: 129 pgs: 129 active+clean; 647 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-28T22:36:52.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:51 smithi172 ceph-mon[36566]: from='client.25475 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:36:52.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:51 smithi172 ceph-mon[36566]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-03-28T22:36:52.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:51 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1577938270' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:36:52.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:51 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2107330978' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:36:52.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:51 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3169083163' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:36:52.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.611+0000 7f939d600640 1 -- 172.21.15.43:0/2821380322 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9398103870 msgr2=0x7f9398103c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:52.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.611+0000 7f939d600640 1 --2- 172.21.15.43:0/2821380322 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9398103870 0x7f9398103c90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9388009900 tx=0x7f938802bc30 comp rx=0 tx=0).stop 2024-03-28T22:36:52.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.611+0000 7f939d600640 1 -- 172.21.15.43:0/2821380322 shutdown_connections 2024-03-28T22:36:52.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.611+0000 7f939d600640 1 --2- 172.21.15.43:0/2821380322 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9398105420 0x7f939810bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:52.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.611+0000 7f939d600640 1 --2- 172.21.15.43:0/2821380322 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9398104a60 0x7f9398104ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:52.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.611+0000 7f939d600640 1 --2- 172.21.15.43:0/2821380322 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9398103870 0x7f9398103c90 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:52.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.611+0000 7f939d600640 1 -- 172.21.15.43:0/2821380322 >> 172.21.15.43:0/2821380322 conn(0x7f93980fee00 msgr2=0x7f9398101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:52.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.611+0000 7f939d600640 1 -- 172.21.15.43:0/2821380322 shutdown_connections 2024-03-28T22:36:52.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.611+0000 7f939d600640 1 -- 172.21.15.43:0/2821380322 wait complete. 2024-03-28T22:36:52.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.612+0000 7f939d600640 1 Processor -- start 2024-03-28T22:36:52.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.613+0000 7f939d600640 1 -- start start 2024-03-28T22:36:52.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.613+0000 7f939d600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9398103870 0x7f9398199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:52.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.613+0000 7f939d600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9398104a60 0x7f939819a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:52.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.613+0000 7f939d600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9398105420 0x7f93981a1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:52.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.613+0000 7f939d600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f939810f540 con 0x7f9398103870 2024-03-28T22:36:52.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.613+0000 7f939d600640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f939810f3c0 con 0x7f9398104a60 2024-03-28T22:36:52.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.613+0000 7f939d600640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f939810f6c0 con 0x7f9398105420 2024-03-28T22:36:52.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.613+0000 7f9397400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9398104a60 0x7f939819a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:52.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.613+0000 7f9397400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9398104a60 0x7f939819a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44034/0 (socket says 172.21.15.43:44034) 2024-03-28T22:36:52.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.613+0000 7f9397400640 1 -- 172.21.15.43:0/1992602948 learned_addr learned my addr 172.21.15.43:0/1992602948 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:52.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.613+0000 7f939cc00640 1 --2- 172.21.15.43:0/1992602948 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9398105420 0x7f93981a1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:52.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.614+0000 7f9397e00640 1 --2- 172.21.15.43:0/1992602948 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9398103870 0x7f9398199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:52.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.614+0000 7f9397400640 1 -- 172.21.15.43:0/1992602948 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9398105420 msgr2=0x7f93981a1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:52.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.614+0000 7f9397400640 1 --2- 172.21.15.43:0/1992602948 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9398105420 0x7f93981a1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:52.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.614+0000 7f9397400640 1 -- 172.21.15.43:0/1992602948 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9398103870 msgr2=0x7f9398199b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:52.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.614+0000 7f9397400640 1 --2- 172.21.15.43:0/1992602948 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9398103870 0x7f9398199b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:52.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.614+0000 7f9397400640 1 -- 172.21.15.43:0/1992602948 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93981a1780 con 0x7f9398104a60 2024-03-28T22:36:52.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.614+0000 7f9397e00640 1 --2- 172.21.15.43:0/1992602948 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9398103870 0x7f9398199b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:36:52.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.614+0000 7f939cc00640 1 --2- 172.21.15.43:0/1992602948 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9398105420 0x7f93981a1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:52.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.614+0000 7f9397400640 1 --2- 172.21.15.43:0/1992602948 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9398104a60 0x7f939819a0a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f938400e9f0 tx=0x7f938400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:52.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.614+0000 7f9394c00640 1 -- 172.21.15.43:0/1992602948 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f938400cd80 con 0x7f9398104a60 2024-03-28T22:36:52.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.614+0000 7f9394c00640 1 -- 172.21.15.43:0/1992602948 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9384010910 con 0x7f9398104a60 2024-03-28T22:36:52.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.614+0000 7f939d600640 1 -- 172.21.15.43:0/1992602948 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93981a1a70 con 0x7f9398104a60 2024-03-28T22:36:52.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.614+0000 7f9394c00640 1 -- 172.21.15.43:0/1992602948 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9384018990 con 0x7f9398104a60 2024-03-28T22:36:52.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.615+0000 7f939d600640 1 -- 172.21.15.43:0/1992602948 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93981a1fc0 con 0x7f9398104a60 2024-03-28T22:36:52.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.616+0000 7f939d600640 1 -- 172.21.15.43:0/1992602948 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f939810ab30 con 0x7f9398104a60 2024-03-28T22:36:52.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.618+0000 7f9394c00640 1 -- 172.21.15.43:0/1992602948 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7f9384010ac0 con 0x7f9398104a60 2024-03-28T22:36:52.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.620+0000 7f9394c00640 1 --2- 172.21.15.43:0/1992602948 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f937c0a1e10 0x7f937c0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:52.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.620+0000 7f9394c00640 1 -- 172.21.15.43:0/1992602948 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f93840ce6b0 con 0x7f9398104a60 2024-03-28T22:36:52.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.620+0000 7f9397e00640 1 --2- 172.21.15.43:0/1992602948 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f937c0a1e10 0x7f937c0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:52.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.621+0000 7f9397e00640 1 --2- 172.21.15.43:0/1992602948 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f937c0a1e10 0x7f937c0a42d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f938802c170 tx=0x7f938802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:52.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.628+0000 7f9394c00640 1 -- 172.21.15.43:0/1992602948 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f938409b590 con 0x7f9398104a60 2024-03-28T22:36:52.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.835+0000 7f939d600640 1 -- 172.21.15.43:0/1992602948 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f939802d3c0 con 0x7f9398104a60 2024-03-28T22:36:52.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.835+0000 7f9394c00640 1 -- 172.21.15.43:0/1992602948 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v96) ==== 103+0+11 (secure 0 0 0) 0x7f938409ae90 con 0x7f9398104a60 2024-03-28T22:36:52.835 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-03-28T22:36:52.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.838+0000 7f939d600640 1 -- 172.21.15.43:0/1992602948 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f937c0a1e10 msgr2=0x7f937c0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:52.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.838+0000 7f939d600640 1 --2- 172.21.15.43:0/1992602948 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f937c0a1e10 0x7f937c0a42d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f938802c170 tx=0x7f938802df90 comp rx=0 tx=0).stop 2024-03-28T22:36:52.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.839+0000 7f939d600640 1 -- 172.21.15.43:0/1992602948 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9398104a60 msgr2=0x7f939819a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:52.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.839+0000 7f939d600640 1 --2- 172.21.15.43:0/1992602948 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9398104a60 0x7f939819a0a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f938400e9f0 tx=0x7f938400eeb0 comp rx=0 tx=0).stop 2024-03-28T22:36:52.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.839+0000 7f939d600640 1 -- 172.21.15.43:0/1992602948 shutdown_connections 2024-03-28T22:36:52.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.839+0000 7f939d600640 1 --2- 172.21.15.43:0/1992602948 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f937c0a1e10 0x7f937c0a42d0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:52.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.839+0000 7f939d600640 1 --2- 172.21.15.43:0/1992602948 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9398104a60 0x7f939819a0a0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:52.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.839+0000 7f939d600640 1 --2- 172.21.15.43:0/1992602948 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9398105420 0x7f93981a1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:52.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.839+0000 7f939d600640 1 --2- 172.21.15.43:0/1992602948 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9398103870 0x7f9398199b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:52.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.839+0000 7f939d600640 1 -- 172.21.15.43:0/1992602948 >> 172.21.15.43:0/1992602948 conn(0x7f93980fee00 msgr2=0x7f9398107d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:52.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.839+0000 7f939d600640 1 -- 172.21.15.43:0/1992602948 shutdown_connections 2024-03-28T22:36:52.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:52.839+0000 7f939d600640 1 -- 172.21.15.43:0/1992602948 wait complete. 2024-03-28T22:36:52.857 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:36:53.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.087+0000 7f89d5c00640 1 -- 172.21.15.43:0/3218720230 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f89d01037a0 msgr2=0x7f89d0103bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:53.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.087+0000 7f89d5c00640 1 --2- 172.21.15.43:0/3218720230 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f89d01037a0 0x7f89d0103bc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f89c0009900 tx=0x7f89c002bc50 comp rx=0 tx=0).stop 2024-03-28T22:36:53.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.087+0000 7f89d5c00640 1 -- 172.21.15.43:0/3218720230 shutdown_connections 2024-03-28T22:36:53.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.087+0000 7f89d5c00640 1 --2- 172.21.15.43:0/3218720230 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89d0105350 0x7f89d010bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:53.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.087+0000 7f89d5c00640 1 --2- 172.21.15.43:0/3218720230 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89d0104990 0x7f89d0104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:53.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.087+0000 7f89d5c00640 1 --2- 172.21.15.43:0/3218720230 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f89d01037a0 0x7f89d0103bc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:53.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.087+0000 7f89d5c00640 1 -- 172.21.15.43:0/3218720230 >> 172.21.15.43:0/3218720230 conn(0x7f89d00fed30 msgr2=0x7f89d0101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:53.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.087+0000 7f89d5c00640 1 -- 172.21.15.43:0/3218720230 shutdown_connections 2024-03-28T22:36:53.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.087+0000 7f89d5c00640 1 -- 172.21.15.43:0/3218720230 wait complete. 2024-03-28T22:36:53.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.088+0000 7f89d5c00640 1 Processor -- start 2024-03-28T22:36:53.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.088+0000 7f89d5c00640 1 -- start start 2024-03-28T22:36:53.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.088+0000 7f89d5c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89d01037a0 0x7f89d0199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:53.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.088+0000 7f89d5c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89d0104990 0x7f89d019a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:53.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.089+0000 7f89d5c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f89d0105350 0x7f89d01a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:53.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.089+0000 7f89d5c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f89d010f440 con 0x7f89d0105350 2024-03-28T22:36:53.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.089+0000 7f89d5c00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f89d010f2c0 con 0x7f89d01037a0 2024-03-28T22:36:53.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.089+0000 7f89d5c00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f89d010f5c0 con 0x7f89d0104990 2024-03-28T22:36:53.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.089+0000 7f89c7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89d0104990 0x7f89d019a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:53.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.089+0000 7f89cfe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89d01037a0 0x7f89d0199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:53.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.089+0000 7f89c7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89d0104990 0x7f89d019a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:55210/0 (socket says 172.21.15.43:55210) 2024-03-28T22:36:53.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.089+0000 7f89cfe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89d01037a0 0x7f89d0199af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44052/0 (socket says 172.21.15.43:44052) 2024-03-28T22:36:53.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.089+0000 7f89c7e00640 1 -- 172.21.15.43:0/2004913956 learned_addr learned my addr 172.21.15.43:0/2004913956 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:53.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.089+0000 7f89d5200640 1 --2- 172.21.15.43:0/2004913956 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f89d0105350 0x7f89d01a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:53.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.089+0000 7f89cfe00640 1 -- 172.21.15.43:0/2004913956 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89d0104990 msgr2=0x7f89d019a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:53.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.089+0000 7f89cfe00640 1 --2- 172.21.15.43:0/2004913956 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89d0104990 0x7f89d019a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:53.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.089+0000 7f89cfe00640 1 -- 172.21.15.43:0/2004913956 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f89d0105350 msgr2=0x7f89d01a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:53.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.089+0000 7f89cfe00640 1 --2- 172.21.15.43:0/2004913956 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f89d0105350 0x7f89d01a10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:53.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.089+0000 7f89cfe00640 1 -- 172.21.15.43:0/2004913956 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89d01a1710 con 0x7f89d01037a0 2024-03-28T22:36:53.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.090+0000 7f89cfe00640 1 --2- 172.21.15.43:0/2004913956 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89d01037a0 0x7f89d0199af0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f89c0036040 tx=0x7f89c002c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:53.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.090+0000 7f89cd600640 1 -- 172.21.15.43:0/2004913956 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c0041070 con 0x7f89d01037a0 2024-03-28T22:36:53.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.090+0000 7f89cd600640 1 -- 172.21.15.43:0/2004913956 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89c0004450 con 0x7f89d01037a0 2024-03-28T22:36:53.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.090+0000 7f89cd600640 1 -- 172.21.15.43:0/2004913956 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c0003e60 con 0x7f89d01037a0 2024-03-28T22:36:53.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.090+0000 7f89d5c00640 1 -- 172.21.15.43:0/2004913956 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89d01a19a0 con 0x7f89d01037a0 2024-03-28T22:36:53.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.090+0000 7f89d5c00640 1 -- 172.21.15.43:0/2004913956 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89d01a1e70 con 0x7f89d01037a0 2024-03-28T22:36:53.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.091+0000 7f89d5c00640 1 -- 172.21.15.43:0/2004913956 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89d010aa60 con 0x7f89d01037a0 2024-03-28T22:36:53.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.093+0000 7f89cd600640 1 -- 172.21.15.43:0/2004913956 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7f89c0044070 con 0x7f89d01037a0 2024-03-28T22:36:53.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.094+0000 7f89cd600640 1 --2- 172.21.15.43:0/2004913956 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f89b40a1e90 0x7f89b40a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:53.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.094+0000 7f89cd600640 1 -- 172.21.15.43:0/2004913956 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f89c00efc30 con 0x7f89d01037a0 2024-03-28T22:36:53.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.094+0000 7f89c7e00640 1 --2- 172.21.15.43:0/2004913956 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f89b40a1e90 0x7f89b40a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:53.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.095+0000 7f89c7e00640 1 --2- 172.21.15.43:0/2004913956 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f89b40a1e90 0x7f89b40a4350 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f89d019b000 tx=0x7f89b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:53.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.098+0000 7f89cd600640 1 -- 172.21.15.43:0/2004913956 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f89c00bc5d0 con 0x7f89d01037a0 2024-03-28T22:36:53.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:52 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1879588533' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:36:53.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:52 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1992602948' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:36:53.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:52 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1879588533' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:36:53.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:52 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1992602948' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:36:53.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.306+0000 7f89d5c00640 1 -- 172.21.15.43:0/2004913956 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f89d0062350 con 0x7f89d01037a0 2024-03-28T22:36:53.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.308+0000 7f89cd600640 1 -- 172.21.15.43:0/2004913956 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+24712 (secure 0 0 0) 0x7f89c00c0670 con 0x7f89d01037a0 2024-03-28T22:36:53.308 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:36:53.308 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":96,"fsid":"c2276428-ed51-11ee-95d0-87774f69a715","created":"2024-03-28T22:24:19.950084+0000","modified":"2024-03-28T22:35:30.237654+0000","last_up_change":"2024-03-28T22:32:31.078737+0000","last_in_change":"2024-03-28T22:32:10.147097+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:28:14.713290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:33:27.167435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:33:28.411825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T22:33:32.523122+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b855adcd-786c-4e42-b8fe-a515a77ce979","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6803","nonce":2826124409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6805","nonce":2826124409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6809","nonce":2826124409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6807","nonce":2826124409}]},"public_addr":"172.21.15.43:6803/2826124409","cluster_addr":"172.21.15.43:6805/2826124409","heartbeat_back_addr":"172.21.15.43:6809/2826124409","heartbeat_front_addr":"172.21.15.43:6807/2826124409","state":["exists","up"]},{"osd":1,"uuid":"031c143d-8ad1-4b68-8db9-3c717137e3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6801","nonce":3043263644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6803","nonce":3043263644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6807","nonce":3043263644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6805","nonce":3043263644}]},"public_addr":"172.21.15.82:6801/3043263644","cluster_addr":"172.21.15.82:6803/3043263644","heartbeat_back_addr":"172.21.15.82:6807/3043263644","heartbeat_front_addr":"172.21.15.82:6805/3043263644","state":["exists","up"]},{"osd":2,"uuid":"20632bbb-f07a-4ca1-b0c4-ff9d508e211c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6801","nonce":4213953645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6803","nonce":4213953645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6807","nonce":4213953645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6805","nonce":4213953645}]},"public_addr":"172.21.15.172:6801/4213953645","cluster_addr":"172.21.15.172:6803/4213953645","heartbeat_back_addr":"172.21.15.172:6807/4213953645","heartbeat_front_addr":"172.21.15.172:6805/4213953645","state":["exists","up"]},{"osd":3,"uuid":"9a1f2312-4dbd-475b-8736-d5ebc8788e09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6811","nonce":1597280920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6813","nonce":1597280920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6817","nonce":1597280920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6815","nonce":1597280920}]},"public_addr":"172.21.15.43:6811/1597280920","cluster_addr":"172.21.15.43:6813/1597280920","heartbeat_back_addr":"172.21.15.43:6817/1597280920","heartbeat_front_addr":"172.21.15.43:6815/1597280920","state":["exists","up"]},{"osd":4,"uuid":"29140ead-7075-42b2-90c1-27cae0b29ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6809","nonce":215451895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6811","nonce":215451895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6815","nonce":215451895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6813","nonce":215451895}]},"public_addr":"172.21.15.82:6809/215451895","cluster_addr":"172.21.15.82:6811/215451895","heartbeat_back_addr":"172.21.15.82:6815/215451895","heartbeat_front_addr":"172.21.15.82:6813/215451895","state":["exists","up"]},{"osd":5,"uuid":"59644c61-93dc-4536-b2ad-0256b24e5cb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6809","nonce":1576148933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6811","nonce":1576148933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6815","nonce":1576148933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6813","nonce":1576148933}]},"public_addr":"172.21.15.172:6809/1576148933","cluster_addr":"172.21.15.172:6811/1576148933","heartbeat_back_addr":"172.21.15.172:6815/1576148933","heartbeat_front_addr":"172.21.15.172:6813/1576148933","state":["exists","up"]},{"osd":6,"uuid":"22d54e57-c3f1-4ecf-9921-8e815b48f92a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6819","nonce":791044279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6821","nonce":791044279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6825","nonce":791044279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6823","nonce":791044279}]},"public_addr":"172.21.15.43:6819/791044279","cluster_addr":"172.21.15.43:6821/791044279","heartbeat_back_addr":"172.21.15.43:6825/791044279","heartbeat_front_addr":"172.21.15.43:6823/791044279","state":["exists","up"]},{"osd":7,"uuid":"248cd1e2-0d6b-452f-a0a8-c51fe1b975ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6817","nonce":2157777323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6819","nonce":2157777323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6823","nonce":2157777323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6821","nonce":2157777323}]},"public_addr":"172.21.15.82:6817/2157777323","cluster_addr":"172.21.15.82:6819/2157777323","heartbeat_back_addr":"172.21.15.82:6823/2157777323","heartbeat_front_addr":"172.21.15.82:6821/2157777323","state":["exists","up"]},{"osd":8,"uuid":"d98aab6e-8bae-4c24-906c-4f492f047b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6817","nonce":1095328248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6819","nonce":1095328248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6823","nonce":1095328248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6821","nonce":1095328248}]},"public_addr":"172.21.15.172:6817/1095328248","cluster_addr":"172.21.15.172:6819/1095328248","heartbeat_back_addr":"172.21.15.172:6823/1095328248","heartbeat_front_addr":"172.21.15.172:6821/1095328248","state":["exists","up"]},{"osd":9,"uuid":"19aa383e-a0f6-45ea-96cc-e7956535caa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6827","nonce":3657791498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6829","nonce":3657791498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6833","nonce":3657791498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6831","nonce":3657791498}]},"public_addr":"172.21.15.43:6827/3657791498","cluster_addr":"172.21.15.43:6829/3657791498","heartbeat_back_addr":"172.21.15.43:6833/3657791498","heartbeat_front_addr":"172.21.15.43:6831/3657791498","state":["exists","up"]},{"osd":10,"uuid":"666977e5-eaf7-41b9-a18f-c380e149345d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6825","nonce":3454954047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6827","nonce":3454954047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6831","nonce":3454954047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6829","nonce":3454954047}]},"public_addr":"172.21.15.82:6825/3454954047","cluster_addr":"172.21.15.82:6827/3454954047","heartbeat_back_addr":"172.21.15.82:6831/3454954047","heartbeat_front_addr":"172.21.15.82:6829/3454954047","state":["exists","up"]},{"osd":11,"uuid":"615cf3be-8076-470e-ad09-c880e7593717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6825","nonce":2563517144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6827","nonce":2563517144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6831","nonce":2563517144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6829","nonce":2563517144}]},"public_addr":"172.21.15.172:6825/2563517144","cluster_addr":"172.21.15.172:6827/2563517144","heartbeat_back_addr":"172.21.15.172:6831/2563517144","heartbeat_front_addr":"172.21.15.172:6829/2563517144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:21.727405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:47.100259+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:10.615438+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:43.632144+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:09.487940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:34.604463+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:09.594452+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:35.709795+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:01.184202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:35.455322+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:02.413166+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:28.550754+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.9","mappings":[{"from":5,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/578791143":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/4029313215":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/1574178482":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6801/3517712438":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6800/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/1375099638":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/2710117652":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6801/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/3215634168":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6800/3517712438":"2024-03-29T22:25:28.008215+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:36:53.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.311+0000 7f89d5c00640 1 -- 172.21.15.43:0/2004913956 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f89b40a1e90 msgr2=0x7f89b40a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:53.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.311+0000 7f89d5c00640 1 --2- 172.21.15.43:0/2004913956 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f89b40a1e90 0x7f89b40a4350 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f89d019b000 tx=0x7f89b8005f90 comp rx=0 tx=0).stop 2024-03-28T22:36:53.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.311+0000 7f89d5c00640 1 -- 172.21.15.43:0/2004913956 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89d01037a0 msgr2=0x7f89d0199af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:53.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.311+0000 7f89d5c00640 1 --2- 172.21.15.43:0/2004913956 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89d01037a0 0x7f89d0199af0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f89c0036040 tx=0x7f89c002c670 comp rx=0 tx=0).stop 2024-03-28T22:36:53.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.312+0000 7f89d5c00640 1 -- 172.21.15.43:0/2004913956 shutdown_connections 2024-03-28T22:36:53.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.312+0000 7f89d5c00640 1 --2- 172.21.15.43:0/2004913956 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f89d0105350 0x7f89d01a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:53.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.312+0000 7f89d5c00640 1 --2- 172.21.15.43:0/2004913956 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f89d0104990 0x7f89d019a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:53.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.312+0000 7f89d5c00640 1 --2- 172.21.15.43:0/2004913956 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f89b40a1e90 0x7f89b40a4350 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:53.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.312+0000 7f89d5c00640 1 --2- 172.21.15.43:0/2004913956 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89d01037a0 0x7f89d0199af0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:53.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.312+0000 7f89d5c00640 1 -- 172.21.15.43:0/2004913956 >> 172.21.15.43:0/2004913956 conn(0x7f89d00fed30 msgr2=0x7f89d0107c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:53.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.312+0000 7f89d5c00640 1 -- 172.21.15.43:0/2004913956 shutdown_connections 2024-03-28T22:36:53.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.312+0000 7f89d5c00640 1 -- 172.21.15.43:0/2004913956 wait complete. 2024-03-28T22:36:53.333 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T22:28:14.713290+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T22:33:27.167435+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T22:33:28.411825+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T22:33:32.523122+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T22:36:53.333 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T22:36:53.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:52 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1879588533' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:36:53.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:52 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1992602948' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:36:53.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.568+0000 7f5ce7e00640 1 -- 172.21.15.43:0/3456376037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ce8101650 msgr2=0x7f5ce8101a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:53.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.568+0000 7f5ce7e00640 1 --2- 172.21.15.43:0/3456376037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ce8101650 0x7f5ce8101a70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8009900 tx=0x7f5cd802bc30 comp rx=0 tx=0).stop 2024-03-28T22:36:53.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.569+0000 7f5ce7e00640 1 -- 172.21.15.43:0/3456376037 shutdown_connections 2024-03-28T22:36:53.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.569+0000 7f5ce7e00640 1 --2- 172.21.15.43:0/3456376037 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ce8103200 0x7f5ce8109a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:53.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.569+0000 7f5ce7e00640 1 --2- 172.21.15.43:0/3456376037 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ce8102840 0x7f5ce8102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:53.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.569+0000 7f5ce7e00640 1 --2- 172.21.15.43:0/3456376037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ce8101650 0x7f5ce8101a70 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:53.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.569+0000 7f5ce7e00640 1 -- 172.21.15.43:0/3456376037 >> 172.21.15.43:0/3456376037 conn(0x7f5ce80fcbc0 msgr2=0x7f5ce80ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:53.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.569+0000 7f5ce7e00640 1 -- 172.21.15.43:0/3456376037 shutdown_connections 2024-03-28T22:36:53.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.569+0000 7f5ce7e00640 1 -- 172.21.15.43:0/3456376037 wait complete. 2024-03-28T22:36:53.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.570+0000 7f5ce7e00640 1 Processor -- start 2024-03-28T22:36:53.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.570+0000 7f5ce7e00640 1 -- start start 2024-03-28T22:36:53.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.570+0000 7f5ce7e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ce8101650 0x7f5ce8075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:53.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.570+0000 7f5ce7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ce8102840 0x7f5ce8073b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:53.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.570+0000 7f5ce7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ce8103200 0x7f5ce8074080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:53.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.570+0000 7f5ce7e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce810d320 con 0x7f5ce8102840 2024-03-28T22:36:53.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.570+0000 7f5ce7e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce810d1a0 con 0x7f5ce8101650 2024-03-28T22:36:53.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.570+0000 7f5ce7e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce810d4a0 con 0x7f5ce8103200 2024-03-28T22:36:53.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.570+0000 7f5ce6a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ce8101650 0x7f5ce8075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:53.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.571+0000 7f5ce6a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ce8101650 0x7f5ce8075500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44082/0 (socket says 172.21.15.43:44082) 2024-03-28T22:36:53.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.571+0000 7f5ce6a00640 1 -- 172.21.15.43:0/3860662621 learned_addr learned my addr 172.21.15.43:0/3860662621 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:53.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.571+0000 7f5ce7400640 1 --2- 172.21.15.43:0/3860662621 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ce8103200 0x7f5ce8074080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:53.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.571+0000 7f5cde000640 1 --2- 172.21.15.43:0/3860662621 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ce8102840 0x7f5ce8073b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:53.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.571+0000 7f5ce6a00640 1 -- 172.21.15.43:0/3860662621 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ce8103200 msgr2=0x7f5ce8074080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:53.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.571+0000 7f5ce6a00640 1 --2- 172.21.15.43:0/3860662621 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ce8103200 0x7f5ce8074080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:53.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.571+0000 7f5ce6a00640 1 -- 172.21.15.43:0/3860662621 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ce8102840 msgr2=0x7f5ce8073b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:53.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.571+0000 7f5ce6a00640 1 --2- 172.21.15.43:0/3860662621 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ce8102840 0x7f5ce8073b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:53.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.571+0000 7f5ce6a00640 1 -- 172.21.15.43:0/3860662621 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ce8074910 con 0x7f5ce8101650 2024-03-28T22:36:53.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.571+0000 7f5ce7400640 1 --2- 172.21.15.43:0/3860662621 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ce8103200 0x7f5ce8074080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:53.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.571+0000 7f5ce6a00640 1 --2- 172.21.15.43:0/3860662621 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ce8101650 0x7f5ce8075500 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8000c00 tx=0x7f5cd802c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:53.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.572+0000 7f5cdfe00640 1 -- 172.21.15.43:0/3860662621 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd8043020 con 0x7f5ce8101650 2024-03-28T22:36:53.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.572+0000 7f5cdfe00640 1 -- 172.21.15.43:0/3860662621 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cd80054e0 con 0x7f5ce8101650 2024-03-28T22:36:53.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.572+0000 7f5cdfe00640 1 -- 172.21.15.43:0/3860662621 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd8003ce0 con 0x7f5ce8101650 2024-03-28T22:36:53.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.572+0000 7f5ce7e00640 1 -- 172.21.15.43:0/3860662621 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ce81a1a80 con 0x7f5ce8101650 2024-03-28T22:36:53.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.572+0000 7f5ce7e00640 1 -- 172.21.15.43:0/3860662621 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ce81a1f50 con 0x7f5ce8101650 2024-03-28T22:36:53.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.573+0000 7f5ce7e00640 1 -- 172.21.15.43:0/3860662621 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ce8108910 con 0x7f5ce8101650 2024-03-28T22:36:53.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.575+0000 7f5cdfe00640 1 -- 172.21.15.43:0/3860662621 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7f5cd8046020 con 0x7f5ce8101650 2024-03-28T22:36:53.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.576+0000 7f5cdfe00640 1 --2- 172.21.15.43:0/3860662621 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5ccc0a1e10 0x7f5ccc0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:53.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.577+0000 7f5cdfe00640 1 -- 172.21.15.43:0/3860662621 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f5cd80f02a0 con 0x7f5ce8101650 2024-03-28T22:36:53.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.577+0000 7f5cde000640 1 --2- 172.21.15.43:0/3860662621 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5ccc0a1e10 0x7f5ccc0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:53.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.577+0000 7f5cde000640 1 --2- 172.21.15.43:0/3860662621 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5ccc0a1e10 0x7f5ccc0a42d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f5ce8075270 tx=0x7f5cd000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:53.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.580+0000 7f5cdfe00640 1 -- 172.21.15.43:0/3860662621 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5cd804d020 con 0x7f5ce8101650 2024-03-28T22:36:53.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.782+0000 7f5ce7e00640 1 -- 172.21.15.43:0/3860662621 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5ce802d3b0 con 0x7f5ce8101650 2024-03-28T22:36:53.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.782+0000 7f5cdfe00640 1 -- 172.21.15.43:0/3860662621 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) ==== 93+0+10 (secure 0 0 0) 0x7f5cd80bd290 con 0x7f5ce8101650 2024-03-28T22:36:53.782 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-03-28T22:36:53.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.785+0000 7f5ce7e00640 1 -- 172.21.15.43:0/3860662621 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5ccc0a1e10 msgr2=0x7f5ccc0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:53.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.785+0000 7f5ce7e00640 1 --2- 172.21.15.43:0/3860662621 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5ccc0a1e10 0x7f5ccc0a42d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f5ce8075270 tx=0x7f5cd000b410 comp rx=0 tx=0).stop 2024-03-28T22:36:53.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.786+0000 7f5ce7e00640 1 -- 172.21.15.43:0/3860662621 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ce8101650 msgr2=0x7f5ce8075500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:53.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.786+0000 7f5ce7e00640 1 --2- 172.21.15.43:0/3860662621 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ce8101650 0x7f5ce8075500 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8000c00 tx=0x7f5cd802c6c0 comp rx=0 tx=0).stop 2024-03-28T22:36:53.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.786+0000 7f5ce7e00640 1 -- 172.21.15.43:0/3860662621 shutdown_connections 2024-03-28T22:36:53.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.786+0000 7f5ce7e00640 1 --2- 172.21.15.43:0/3860662621 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ce8103200 0x7f5ce8074080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:53.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.786+0000 7f5ce7e00640 1 --2- 172.21.15.43:0/3860662621 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ce8102840 0x7f5ce8073b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:53.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.786+0000 7f5ce7e00640 1 --2- 172.21.15.43:0/3860662621 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5ccc0a1e10 0x7f5ccc0a42d0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:53.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.786+0000 7f5ce7e00640 1 --2- 172.21.15.43:0/3860662621 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ce8101650 0x7f5ce8075500 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:53.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.786+0000 7f5ce7e00640 1 -- 172.21.15.43:0/3860662621 >> 172.21.15.43:0/3860662621 conn(0x7f5ce80fcbc0 msgr2=0x7f5ce8105ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:53.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.786+0000 7f5ce7e00640 1 -- 172.21.15.43:0/3860662621 shutdown_connections 2024-03-28T22:36:53.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:53.786+0000 7f5ce7e00640 1 -- 172.21.15.43:0/3860662621 wait complete. 2024-03-28T22:36:53.805 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T22:36:54.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.030+0000 7f7afac00640 1 -- 172.21.15.43:0/855809208 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7af4103990 msgr2=0x7f7af4103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:54.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.030+0000 7f7afac00640 1 --2- 172.21.15.43:0/855809208 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7af4103990 0x7f7af4103db0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4009900 tx=0x7f7ae402bc30 comp rx=0 tx=0).stop 2024-03-28T22:36:54.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.030+0000 7f7afac00640 1 -- 172.21.15.43:0/855809208 shutdown_connections 2024-03-28T22:36:54.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.030+0000 7f7afac00640 1 --2- 172.21.15.43:0/855809208 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7af4104b80 0x7f7af4105000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:54.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.030+0000 7f7afac00640 1 --2- 172.21.15.43:0/855809208 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7af4105540 0x7f7af410bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:54.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.030+0000 7f7afac00640 1 --2- 172.21.15.43:0/855809208 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7af4103990 0x7f7af4103db0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:54.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.030+0000 7f7afac00640 1 -- 172.21.15.43:0/855809208 >> 172.21.15.43:0/855809208 conn(0x7f7af40fef00 msgr2=0x7f7af4101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:54.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.030+0000 7f7afac00640 1 -- 172.21.15.43:0/855809208 shutdown_connections 2024-03-28T22:36:54.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.030+0000 7f7afac00640 1 -- 172.21.15.43:0/855809208 wait complete. 2024-03-28T22:36:54.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.032+0000 7f7afac00640 1 Processor -- start 2024-03-28T22:36:54.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.032+0000 7f7afac00640 1 -- start start 2024-03-28T22:36:54.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.032+0000 7f7afac00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7af4103990 0x7f7af4199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:54.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.032+0000 7f7afac00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7af4104b80 0x7f7af419a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:54.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.032+0000 7f7afac00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7af4105540 0x7f7af41a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:54.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.032+0000 7f7afac00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7af410f630 con 0x7f7af4104b80 2024-03-28T22:36:54.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.032+0000 7f7afac00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7af410f4b0 con 0x7f7af4105540 2024-03-28T22:36:54.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.032+0000 7f7afac00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f7af410f7b0 con 0x7f7af4103990 2024-03-28T22:36:54.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.032+0000 7f7af3e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7af4103990 0x7f7af4199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:54.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.033+0000 7f7af8c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7af4105540 0x7f7af41a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:54.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.033+0000 7f7af3e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7af4103990 0x7f7af4199cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:55252/0 (socket says 172.21.15.43:55252) 2024-03-28T22:36:54.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.033+0000 7f7af3e00640 1 -- 172.21.15.43:0/175020436 learned_addr learned my addr 172.21.15.43:0/175020436 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:54.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.033+0000 7f7aebe00640 1 --2- 172.21.15.43:0/175020436 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7af4104b80 0x7f7af419a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:54.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.033+0000 7f7af8c00640 1 -- 172.21.15.43:0/175020436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7af4103990 msgr2=0x7f7af4199cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:54.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.033+0000 7f7af8c00640 1 --2- 172.21.15.43:0/175020436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7af4103990 0x7f7af4199cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:54.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.033+0000 7f7af8c00640 1 -- 172.21.15.43:0/175020436 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7af4104b80 msgr2=0x7f7af419a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:54.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.033+0000 7f7af8c00640 1 --2- 172.21.15.43:0/175020436 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7af4104b80 0x7f7af419a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:54.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.033+0000 7f7af8c00640 1 -- 172.21.15.43:0/175020436 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7af41a1900 con 0x7f7af4105540 2024-03-28T22:36:54.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.033+0000 7f7af3e00640 1 --2- 172.21.15.43:0/175020436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7af4103990 0x7f7af4199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:54.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.033+0000 7f7aebe00640 1 --2- 172.21.15.43:0/175020436 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7af4104b80 0x7f7af419a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:36:54.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.034+0000 7f7af8c00640 1 --2- 172.21.15.43:0/175020436 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7af4105540 0x7f7af41a1260 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7aec00e940 tx=0x7f7aec00ee00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:54.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.034+0000 7f7af1600640 1 -- 172.21.15.43:0/175020436 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aec004d20 con 0x7f7af4105540 2024-03-28T22:36:54.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.034+0000 7f7af1600640 1 -- 172.21.15.43:0/175020436 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7aec005020 con 0x7f7af4105540 2024-03-28T22:36:54.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.034+0000 7f7af1600640 1 -- 172.21.15.43:0/175020436 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aec0116b0 con 0x7f7af4105540 2024-03-28T22:36:54.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.034+0000 7f7afac00640 1 -- 172.21.15.43:0/175020436 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7af41a1bf0 con 0x7f7af4105540 2024-03-28T22:36:54.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.034+0000 7f7afac00640 1 -- 172.21.15.43:0/175020436 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7af41a2140 con 0x7f7af4105540 2024-03-28T22:36:54.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.035+0000 7f7afac00640 1 -- 172.21.15.43:0/175020436 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7af410acb0 con 0x7f7af4105540 2024-03-28T22:36:54.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.037+0000 7f7af1600640 1 -- 172.21.15.43:0/175020436 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7f7aec0054d0 con 0x7f7af4105540 2024-03-28T22:36:54.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.039+0000 7f7af1600640 1 --2- 172.21.15.43:0/175020436 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7ad80a1e90 0x7f7ad80a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:54.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.039+0000 7f7af1600640 1 -- 172.21.15.43:0/175020436 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f7aec0ce3a0 con 0x7f7af4105540 2024-03-28T22:36:54.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.039+0000 7f7af3e00640 1 --2- 172.21.15.43:0/175020436 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7ad80a1e90 0x7f7ad80a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:54.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.040+0000 7f7af3e00640 1 --2- 172.21.15.43:0/175020436 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7ad80a1e90 0x7f7ad80a4350 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f7af419b1c0 tx=0x7f7ae402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:54.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.041+0000 7f7af1600640 1 -- 172.21.15.43:0/175020436 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7aec016030 con 0x7f7af4105540 2024-03-28T22:36:54.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:53 smithi082 ceph-mon[36471]: pgmap v428: 129 pgs: 129 active+clean; 647 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-28T22:36:54.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:53 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2004913956' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:36:54.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:53 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3860662621' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:36:54.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.240+0000 7f7afac00640 1 -- 172.21.15.43:0/175020436 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7af402d3e0 con 0x7f7af4105540 2024-03-28T22:36:54.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.240+0000 7f7af1600640 1 -- 172.21.15.43:0/175020436 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) ==== 104+0+11 (secure 0 0 0) 0x7f7aec09b300 con 0x7f7af4105540 2024-03-28T22:36:54.239 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-03-28T22:36:54.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.243+0000 7f7afac00640 1 -- 172.21.15.43:0/175020436 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7ad80a1e90 msgr2=0x7f7ad80a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:54.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.243+0000 7f7afac00640 1 --2- 172.21.15.43:0/175020436 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7ad80a1e90 0x7f7ad80a4350 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f7af419b1c0 tx=0x7f7ae402df90 comp rx=0 tx=0).stop 2024-03-28T22:36:54.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.243+0000 7f7afac00640 1 -- 172.21.15.43:0/175020436 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7af4105540 msgr2=0x7f7af41a1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:54.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.243+0000 7f7afac00640 1 --2- 172.21.15.43:0/175020436 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7af4105540 0x7f7af41a1260 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7aec00e940 tx=0x7f7aec00ee00 comp rx=0 tx=0).stop 2024-03-28T22:36:54.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.243+0000 7f7afac00640 1 -- 172.21.15.43:0/175020436 shutdown_connections 2024-03-28T22:36:54.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.243+0000 7f7afac00640 1 --2- 172.21.15.43:0/175020436 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7ad80a1e90 0x7f7ad80a4350 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:54.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.243+0000 7f7afac00640 1 --2- 172.21.15.43:0/175020436 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7af4105540 0x7f7af41a1260 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:54.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.243+0000 7f7afac00640 1 --2- 172.21.15.43:0/175020436 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7af4104b80 0x7f7af419a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:54.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.243+0000 7f7afac00640 1 --2- 172.21.15.43:0/175020436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7af4103990 0x7f7af4199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:54.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.243+0000 7f7afac00640 1 -- 172.21.15.43:0/175020436 >> 172.21.15.43:0/175020436 conn(0x7f7af40fef00 msgr2=0x7f7af4107e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:54.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.243+0000 7f7afac00640 1 -- 172.21.15.43:0/175020436 shutdown_connections 2024-03-28T22:36:54.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.243+0000 7f7afac00640 1 -- 172.21.15.43:0/175020436 wait complete. 2024-03-28T22:36:54.257 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:53 smithi043 ceph-mon[31823]: pgmap v428: 129 pgs: 129 active+clean; 647 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-28T22:36:54.257 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:53 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2004913956' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:36:54.257 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:53 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3860662621' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:36:54.261 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T22:36:54.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:53 smithi172 ceph-mon[36566]: pgmap v428: 129 pgs: 129 active+clean; 647 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-28T22:36:54.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:53 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2004913956' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:36:54.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:53 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3860662621' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:36:54.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.489+0000 7f4ae5c00640 1 -- 172.21.15.43:0/3697287188 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ae0104a60 msgr2=0x7f4ae0104ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:54.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.489+0000 7f4ae5c00640 1 --2- 172.21.15.43:0/3697287188 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ae0104a60 0x7f4ae0104ee0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4acc009900 tx=0x7f4acc02bc30 comp rx=0 tx=0).stop 2024-03-28T22:36:54.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.489+0000 7f4ae5c00640 1 -- 172.21.15.43:0/3697287188 shutdown_connections 2024-03-28T22:36:54.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.489+0000 7f4ae5c00640 1 --2- 172.21.15.43:0/3697287188 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ae0104a60 0x7f4ae0104ee0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:54.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.489+0000 7f4ae5c00640 1 --2- 172.21.15.43:0/3697287188 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ae0105420 0x7f4ae010bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:54.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.489+0000 7f4ae5c00640 1 --2- 172.21.15.43:0/3697287188 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ae0103870 0x7f4ae0103c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:54.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.489+0000 7f4ae5c00640 1 -- 172.21.15.43:0/3697287188 >> 172.21.15.43:0/3697287188 conn(0x7f4ae00fee00 msgr2=0x7f4ae0101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:54.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.489+0000 7f4ae5c00640 1 -- 172.21.15.43:0/3697287188 shutdown_connections 2024-03-28T22:36:54.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.489+0000 7f4ae5c00640 1 -- 172.21.15.43:0/3697287188 wait complete. 2024-03-28T22:36:54.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.490+0000 7f4ae5c00640 1 Processor -- start 2024-03-28T22:36:54.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.490+0000 7f4ae5c00640 1 -- start start 2024-03-28T22:36:54.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.490+0000 7f4ae5c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ae0103870 0x7f4ae0199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:54.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.490+0000 7f4ae5c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ae0104a60 0x7f4ae019a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:54.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.490+0000 7f4ae5c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ae0105420 0x7f4ae01a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:54.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.490+0000 7f4ae5c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae010f520 con 0x7f4ae0105420 2024-03-28T22:36:54.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.490+0000 7f4ae5c00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae010f3a0 con 0x7f4ae0103870 2024-03-28T22:36:54.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.490+0000 7f4ae5c00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae010f6a0 con 0x7f4ae0104a60 2024-03-28T22:36:54.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.491+0000 7f4adfe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ae0103870 0x7f4ae0199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:54.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.491+0000 7f4adf400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ae0104a60 0x7f4ae019a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:54.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.491+0000 7f4ae5200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ae0105420 0x7f4ae01a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:54.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.491+0000 7f4adfe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ae0103870 0x7f4ae0199b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44110/0 (socket says 172.21.15.43:44110) 2024-03-28T22:36:54.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.491+0000 7f4adfe00640 1 -- 172.21.15.43:0/1293275977 learned_addr learned my addr 172.21.15.43:0/1293275977 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:54.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.491+0000 7f4ae5200640 1 -- 172.21.15.43:0/1293275977 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ae0104a60 msgr2=0x7f4ae019a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:54.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.491+0000 7f4ae5200640 1 --2- 172.21.15.43:0/1293275977 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ae0104a60 0x7f4ae019a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:54.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.491+0000 7f4ae5200640 1 -- 172.21.15.43:0/1293275977 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ae0103870 msgr2=0x7f4ae0199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:54.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.491+0000 7f4ae5200640 1 --2- 172.21.15.43:0/1293275977 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ae0103870 0x7f4ae0199b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:54.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.491+0000 7f4ae5200640 1 -- 172.21.15.43:0/1293275977 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ae01a1790 con 0x7f4ae0105420 2024-03-28T22:36:54.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.491+0000 7f4adfe00640 1 --2- 172.21.15.43:0/1293275977 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ae0103870 0x7f4ae0199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:54.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.491+0000 7f4ae5200640 1 --2- 172.21.15.43:0/1293275977 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ae0105420 0x7f4ae01a1120 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4ad800ec50 tx=0x7f4ad800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:54.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.492+0000 7f4adcc00640 1 -- 172.21.15.43:0/1293275977 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad8003b70 con 0x7f4ae0105420 2024-03-28T22:36:54.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.492+0000 7f4adcc00640 1 -- 172.21.15.43:0/1293275977 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ad8003d10 con 0x7f4ae0105420 2024-03-28T22:36:54.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.492+0000 7f4ae5c00640 1 -- 172.21.15.43:0/1293275977 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ae01a1a80 con 0x7f4ae0105420 2024-03-28T22:36:54.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.492+0000 7f4adcc00640 1 -- 172.21.15.43:0/1293275977 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad8010890 con 0x7f4ae0105420 2024-03-28T22:36:54.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.492+0000 7f4ae5c00640 1 -- 172.21.15.43:0/1293275977 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ae01a1f50 con 0x7f4ae0105420 2024-03-28T22:36:54.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.493+0000 7f4ae5c00640 1 -- 172.21.15.43:0/1293275977 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ae00688b0 con 0x7f4ae0105420 2024-03-28T22:36:54.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.495+0000 7f4adcc00640 1 -- 172.21.15.43:0/1293275977 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7f4ad8003fe0 con 0x7f4ae0105420 2024-03-28T22:36:54.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.496+0000 7f4adcc00640 1 --2- 172.21.15.43:0/1293275977 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4ac40a1e10 0x7f4ac40a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:54.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.496+0000 7f4adcc00640 1 -- 172.21.15.43:0/1293275977 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f4ad80ce890 con 0x7f4ae0105420 2024-03-28T22:36:54.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.496+0000 7f4adfe00640 1 --2- 172.21.15.43:0/1293275977 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4ac40a1e10 0x7f4ac40a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:54.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.497+0000 7f4adfe00640 1 --2- 172.21.15.43:0/1293275977 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4ac40a1e10 0x7f4ac40a42d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4ad00058c0 tx=0x7f4ad0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:54.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.501+0000 7f4adcc00640 1 -- 172.21.15.43:0/1293275977 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4ad809b770 con 0x7f4ae0105420 2024-03-28T22:36:54.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.702+0000 7f4ae5c00640 1 -- 172.21.15.43:0/1293275977 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4ae002d170 con 0x7f4ae0105420 2024-03-28T22:36:54.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.703+0000 7f4adcc00640 1 -- 172.21.15.43:0/1293275977 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) ==== 100+0+11 (secure 0 0 0) 0x7f4ad809b070 con 0x7f4ae0105420 2024-03-28T22:36:54.702 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-03-28T22:36:54.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.706+0000 7f4ae5c00640 1 -- 172.21.15.43:0/1293275977 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4ac40a1e10 msgr2=0x7f4ac40a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:54.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.706+0000 7f4ae5c00640 1 --2- 172.21.15.43:0/1293275977 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4ac40a1e10 0x7f4ac40a42d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4ad00058c0 tx=0x7f4ad0005f90 comp rx=0 tx=0).stop 2024-03-28T22:36:54.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.706+0000 7f4ae5c00640 1 -- 172.21.15.43:0/1293275977 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ae0105420 msgr2=0x7f4ae01a1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:54.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.706+0000 7f4ae5c00640 1 --2- 172.21.15.43:0/1293275977 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ae0105420 0x7f4ae01a1120 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4ad800ec50 tx=0x7f4ad800c6a0 comp rx=0 tx=0).stop 2024-03-28T22:36:54.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.706+0000 7f4ae5c00640 1 -- 172.21.15.43:0/1293275977 shutdown_connections 2024-03-28T22:36:54.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.706+0000 7f4ae5c00640 1 --2- 172.21.15.43:0/1293275977 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ae0105420 0x7f4ae01a1120 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:54.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.706+0000 7f4ae5c00640 1 --2- 172.21.15.43:0/1293275977 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ae0104a60 0x7f4ae019a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:54.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.706+0000 7f4ae5c00640 1 --2- 172.21.15.43:0/1293275977 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f4ac40a1e10 0x7f4ac40a42d0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:54.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.706+0000 7f4ae5c00640 1 --2- 172.21.15.43:0/1293275977 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ae0103870 0x7f4ae0199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:54.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.706+0000 7f4ae5c00640 1 -- 172.21.15.43:0/1293275977 >> 172.21.15.43:0/1293275977 conn(0x7f4ae00fee00 msgr2=0x7f4ae0107d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:54.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.706+0000 7f4ae5c00640 1 -- 172.21.15.43:0/1293275977 shutdown_connections 2024-03-28T22:36:54.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.706+0000 7f4ae5c00640 1 -- 172.21.15.43:0/1293275977 wait complete. 2024-03-28T22:36:54.724 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T22:36:54.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.946+0000 7fadb9e00640 1 -- 172.21.15.43:0/3710961017 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fadb4109820 msgr2=0x7fadb41100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:54.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.946+0000 7fadb9e00640 1 --2- 172.21.15.43:0/3710961017 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fadb4109820 0x7fadb41100b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fadac009ca0 tx=0x7fadac02c100 comp rx=0 tx=0).stop 2024-03-28T22:36:54.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.946+0000 7fadb9e00640 1 -- 172.21.15.43:0/3710961017 shutdown_connections 2024-03-28T22:36:54.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.946+0000 7fadb9e00640 1 --2- 172.21.15.43:0/3710961017 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fadb4109820 0x7fadb41100b0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:54.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.946+0000 7fadb9e00640 1 --2- 172.21.15.43:0/3710961017 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fadb4108e60 0x7fadb41092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:54.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.946+0000 7fadb9e00640 1 --2- 172.21.15.43:0/3710961017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fadb4107c70 0x7fadb4108090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:54.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.946+0000 7fadb9e00640 1 -- 172.21.15.43:0/3710961017 >> 172.21.15.43:0/3710961017 conn(0x7fadb4076c20 msgr2=0x7fadb4079040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:54.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.946+0000 7fadb9e00640 1 -- 172.21.15.43:0/3710961017 shutdown_connections 2024-03-28T22:36:54.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.946+0000 7fadb9e00640 1 -- 172.21.15.43:0/3710961017 wait complete. 2024-03-28T22:36:54.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.947+0000 7fadb9e00640 1 Processor -- start 2024-03-28T22:36:54.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.948+0000 7fadb9e00640 1 -- start start 2024-03-28T22:36:54.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.948+0000 7fadb9e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fadb4107c70 0x7fadb419deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:54.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.948+0000 7fadb9e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fadb4108e60 0x7fadb419e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:54.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.948+0000 7fadb9e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fadb4109820 0x7fadb41a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:54.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.948+0000 7fadb9e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fadb41138d0 con 0x7fadb4109820 2024-03-28T22:36:54.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.948+0000 7fadb9e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fadb4113750 con 0x7fadb4108e60 2024-03-28T22:36:54.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.948+0000 7fadb9e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fadb4113a50 con 0x7fadb4107c70 2024-03-28T22:36:54.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.948+0000 7fadb8a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fadb4107c70 0x7fadb419deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:54.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.948+0000 7fadb8a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fadb4107c70 0x7fadb419deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:46662/0 (socket says 172.21.15.43:46662) 2024-03-28T22:36:54.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.948+0000 7fadb8a00640 1 -- 172.21.15.43:0/1671036691 learned_addr learned my addr 172.21.15.43:0/1671036691 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:54.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.948+0000 7fadb3e00640 1 --2- 172.21.15.43:0/1671036691 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fadb4108e60 0x7fadb419e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:54.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.948+0000 7fadb9400640 1 --2- 172.21.15.43:0/1671036691 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fadb4109820 0x7fadb41a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:54.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.949+0000 7fadb3e00640 1 -- 172.21.15.43:0/1671036691 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fadb4107c70 msgr2=0x7fadb419deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:54.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.949+0000 7fadb3e00640 1 --2- 172.21.15.43:0/1671036691 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fadb4107c70 0x7fadb419deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:54.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.949+0000 7fadb3e00640 1 -- 172.21.15.43:0/1671036691 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fadb4109820 msgr2=0x7fadb41a5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:54.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.949+0000 7fadb3e00640 1 --2- 172.21.15.43:0/1671036691 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fadb4109820 0x7fadb41a5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:54.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.949+0000 7fadb3e00640 1 -- 172.21.15.43:0/1671036691 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadb41a5ad0 con 0x7fadb4108e60 2024-03-28T22:36:54.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.949+0000 7fadb8a00640 1 --2- 172.21.15.43:0/1671036691 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fadb4107c70 0x7fadb419deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:36:54.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.949+0000 7fadb3e00640 1 --2- 172.21.15.43:0/1671036691 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fadb4108e60 0x7fadb419e3f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fada400d6d0 tx=0x7fada400db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:54.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.949+0000 7fadb1600640 1 -- 172.21.15.43:0/1671036691 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada4017070 con 0x7fadb4108e60 2024-03-28T22:36:54.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.949+0000 7fadb1600640 1 -- 172.21.15.43:0/1671036691 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fada4009820 con 0x7fadb4108e60 2024-03-28T22:36:54.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.949+0000 7fadb1600640 1 -- 172.21.15.43:0/1671036691 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada4004d00 con 0x7fadb4108e60 2024-03-28T22:36:54.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.949+0000 7fadb9e00640 1 -- 172.21.15.43:0/1671036691 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadb41a5dc0 con 0x7fadb4108e60 2024-03-28T22:36:54.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.949+0000 7fadb9e00640 1 -- 172.21.15.43:0/1671036691 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadb41a6290 con 0x7fadb4108e60 2024-03-28T22:36:54.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.951+0000 7fadb9e00640 1 -- 172.21.15.43:0/1671036691 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadb410ef90 con 0x7fadb4108e60 2024-03-28T22:36:54.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.952+0000 7fadb1600640 1 -- 172.21.15.43:0/1671036691 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7fada40099d0 con 0x7fadb4108e60 2024-03-28T22:36:54.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.954+0000 7fadb1600640 1 --2- 172.21.15.43:0/1671036691 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fad9c0a1e90 0x7fad9c0a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:54.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.955+0000 7fadb1600640 1 -- 172.21.15.43:0/1671036691 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7fada40cdd70 con 0x7fadb4108e60 2024-03-28T22:36:54.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.955+0000 7fadb8a00640 1 --2- 172.21.15.43:0/1671036691 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fad9c0a1e90 0x7fad9c0a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:54.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.955+0000 7fadb8a00640 1 --2- 172.21.15.43:0/1671036691 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fad9c0a1e90 0x7fad9c0a4350 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fada80058c0 tx=0x7fada8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:54.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:54.959+0000 7fadb1600640 1 -- 172.21.15.43:0/1671036691 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fada409a600 con 0x7fadb4108e60 2024-03-28T22:36:55.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:54 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/175020436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:36:55.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:54 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1293275977' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:36:55.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.161+0000 7fadb9e00640 1 -- 172.21.15.43:0/1671036691 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fadb402d210 con 0x7fadb4108e60 2024-03-28T22:36:55.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.162+0000 7fadb1600640 1 -- 172.21.15.43:0/1671036691 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v96) ==== 103+0+11 (secure 0 0 0) 0x7fada409e6a0 con 0x7fadb4108e60 2024-03-28T22:36:55.161 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-03-28T22:36:55.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.164+0000 7fadb9e00640 1 -- 172.21.15.43:0/1671036691 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fad9c0a1e90 msgr2=0x7fad9c0a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:55.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.164+0000 7fadb9e00640 1 --2- 172.21.15.43:0/1671036691 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fad9c0a1e90 0x7fad9c0a4350 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fada80058c0 tx=0x7fada8005f90 comp rx=0 tx=0).stop 2024-03-28T22:36:55.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.164+0000 7fadb9e00640 1 -- 172.21.15.43:0/1671036691 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fadb4108e60 msgr2=0x7fadb419e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:55.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.164+0000 7fadb9e00640 1 --2- 172.21.15.43:0/1671036691 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fadb4108e60 0x7fadb419e3f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fada400d6d0 tx=0x7fada400db90 comp rx=0 tx=0).stop 2024-03-28T22:36:55.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.165+0000 7fadb9e00640 1 -- 172.21.15.43:0/1671036691 shutdown_connections 2024-03-28T22:36:55.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.165+0000 7fadb9e00640 1 --2- 172.21.15.43:0/1671036691 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fad9c0a1e90 0x7fad9c0a4350 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:55.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.165+0000 7fadb9e00640 1 --2- 172.21.15.43:0/1671036691 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fadb4108e60 0x7fadb419e3f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:55.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.165+0000 7fadb9e00640 1 --2- 172.21.15.43:0/1671036691 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fadb4109820 0x7fadb41a5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:55.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.165+0000 7fadb9e00640 1 --2- 172.21.15.43:0/1671036691 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fadb4107c70 0x7fadb419deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:55.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.165+0000 7fadb9e00640 1 -- 172.21.15.43:0/1671036691 >> 172.21.15.43:0/1671036691 conn(0x7fadb4076c20 msgr2=0x7fadb410c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:55.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.165+0000 7fadb9e00640 1 -- 172.21.15.43:0/1671036691 shutdown_connections 2024-03-28T22:36:55.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.165+0000 7fadb9e00640 1 -- 172.21.15.43:0/1671036691 wait complete. 2024-03-28T22:36:55.181 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-28T22:36:55.181 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:36:55.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:54 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/175020436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:36:55.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:54 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1293275977' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:36:55.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.400+0000 7f8273e00640 1 -- 172.21.15.43:0/3752751881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8274103160 msgr2=0x7f82741099f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:55.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.400+0000 7f8273e00640 1 --2- 172.21.15.43:0/3752751881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8274103160 0x7f82741099f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f826c009ca0 tx=0x7f826c02c100 comp rx=0 tx=0).stop 2024-03-28T22:36:55.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.400+0000 7f8273e00640 1 -- 172.21.15.43:0/3752751881 shutdown_connections 2024-03-28T22:36:55.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.400+0000 7f8273e00640 1 --2- 172.21.15.43:0/3752751881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8274103160 0x7f82741099f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:55.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.400+0000 7f8273e00640 1 --2- 172.21.15.43:0/3752751881 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82741027a0 0x7f8274102c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:55.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.400+0000 7f8273e00640 1 --2- 172.21.15.43:0/3752751881 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f82741015b0 0x7f82741019d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:55.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.400+0000 7f8273e00640 1 -- 172.21.15.43:0/3752751881 >> 172.21.15.43:0/3752751881 conn(0x7f82740fcb20 msgr2=0x7f82740fef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:55.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.400+0000 7f8273e00640 1 -- 172.21.15.43:0/3752751881 shutdown_connections 2024-03-28T22:36:55.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.400+0000 7f8273e00640 1 -- 172.21.15.43:0/3752751881 wait complete. 2024-03-28T22:36:55.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.401+0000 7f8273e00640 1 Processor -- start 2024-03-28T22:36:55.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.401+0000 7f8273e00640 1 -- start start 2024-03-28T22:36:55.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.402+0000 7f8273e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f82741015b0 0x7f8274199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:55.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.402+0000 7f8273e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82741027a0 0x7f8274199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:55.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.402+0000 7f8273e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8274103160 0x7f82741a0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:55.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.402+0000 7f8273e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f827410d210 con 0x7f82741027a0 2024-03-28T22:36:55.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.402+0000 7f8273e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f827410d090 con 0x7f82741015b0 2024-03-28T22:36:55.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.402+0000 7f8273e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f827410d390 con 0x7f8274103160 2024-03-28T22:36:55.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.402+0000 7f826a000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82741027a0 0x7f8274199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:55.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.402+0000 7f8272a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f82741015b0 0x7f8274199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:55.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.402+0000 7f8272a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f82741015b0 0x7f8274199a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:37334/0 (socket says 172.21.15.43:37334) 2024-03-28T22:36:55.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.402+0000 7f8272a00640 1 -- 172.21.15.43:0/3446644194 learned_addr learned my addr 172.21.15.43:0/3446644194 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:55.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.402+0000 7f8272a00640 1 -- 172.21.15.43:0/3446644194 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8274103160 msgr2=0x7f82741a0ff0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T22:36:55.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.403+0000 7f8272a00640 1 --2- 172.21.15.43:0/3446644194 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8274103160 0x7f82741a0ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:55.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.403+0000 7f8272a00640 1 -- 172.21.15.43:0/3446644194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82741027a0 msgr2=0x7f8274199f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:55.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.403+0000 7f8272a00640 1 --2- 172.21.15.43:0/3446644194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82741027a0 0x7f8274199f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:55.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.403+0000 7f8272a00640 1 -- 172.21.15.43:0/3446644194 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82741a1660 con 0x7f82741015b0 2024-03-28T22:36:55.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.403+0000 7f8272a00640 1 --2- 172.21.15.43:0/3446644194 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f82741015b0 0x7f8274199a40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f826400e9c0 tx=0x7f826400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:55.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.403+0000 7f8278a00640 1 -- 172.21.15.43:0/3446644194 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826400cd80 con 0x7f82741015b0 2024-03-28T22:36:55.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.403+0000 7f8278a00640 1 -- 172.21.15.43:0/3446644194 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82640035f0 con 0x7f82741015b0 2024-03-28T22:36:55.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.403+0000 7f8273e00640 1 -- 172.21.15.43:0/3446644194 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82741a1950 con 0x7f82741015b0 2024-03-28T22:36:55.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.403+0000 7f8278a00640 1 -- 172.21.15.43:0/3446644194 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8264010690 con 0x7f82741015b0 2024-03-28T22:36:55.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.403+0000 7f8273e00640 1 -- 172.21.15.43:0/3446644194 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82741a1ef0 con 0x7f82741015b0 2024-03-28T22:36:55.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.404+0000 7f8273e00640 1 -- 172.21.15.43:0/3446644194 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82741a2180 con 0x7f82741015b0 2024-03-28T22:36:55.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.406+0000 7f8278a00640 1 -- 172.21.15.43:0/3446644194 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7f82640037a0 con 0x7f82741015b0 2024-03-28T22:36:55.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.408+0000 7f8278a00640 1 --2- 172.21.15.43:0/3446644194 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f82580a1e90 0x7f82580a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:55.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.408+0000 7f8278a00640 1 -- 172.21.15.43:0/3446644194 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f82640ce710 con 0x7f82741015b0 2024-03-28T22:36:55.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.408+0000 7f826a000640 1 --2- 172.21.15.43:0/3446644194 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f82580a1e90 0x7f82580a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:55.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.409+0000 7f826a000640 1 --2- 172.21.15.43:0/3446644194 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f82580a1e90 0x7f82580a4350 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f827419af50 tx=0x7f825c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:55.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.412+0000 7f8278a00640 1 -- 172.21.15.43:0/3446644194 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f826409b5f0 con 0x7f82741015b0 2024-03-28T22:36:55.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:54 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/175020436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:36:55.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:54 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1293275977' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:36:55.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.653+0000 7f8273e00640 1 -- 172.21.15.43:0/3446644194 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82741a2350 con 0x7f82741015b0 2024-03-28T22:36:55.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.654+0000 7f8278a00640 1 -- 172.21.15.43:0/3446644194 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11934 (secure 0 0 0) 0x7f826409aef0 con 0x7f82741015b0 2024-03-28T22:36:55.653 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:36:55.653 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:33:28.906396+0000","modified":"2024-03-28T22:36:49.597670+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24572,"mds_1":24556,"mds_2":24535,"mds_3":24523,"mds_4":24518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/1084988072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1084988072},{"type":"v1","addr":"172.21.15.82:6835","nonce":1084988072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572,24523,24518,24535,24556]},"id":1}]} 2024-03-28T22:36:55.654 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 29 2024-03-28T22:36:55.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.656+0000 7f8273e00640 1 -- 172.21.15.43:0/3446644194 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f82580a1e90 msgr2=0x7f82580a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:55.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.656+0000 7f8273e00640 1 --2- 172.21.15.43:0/3446644194 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f82580a1e90 0x7f82580a4350 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f827419af50 tx=0x7f825c005f90 comp rx=0 tx=0).stop 2024-03-28T22:36:55.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.656+0000 7f8273e00640 1 -- 172.21.15.43:0/3446644194 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f82741015b0 msgr2=0x7f8274199a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:55.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.656+0000 7f8273e00640 1 --2- 172.21.15.43:0/3446644194 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f82741015b0 0x7f8274199a40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f826400e9c0 tx=0x7f826400ee80 comp rx=0 tx=0).stop 2024-03-28T22:36:55.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.657+0000 7f8273e00640 1 -- 172.21.15.43:0/3446644194 shutdown_connections 2024-03-28T22:36:55.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.657+0000 7f8273e00640 1 --2- 172.21.15.43:0/3446644194 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8274103160 0x7f82741a0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:55.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.657+0000 7f8273e00640 1 --2- 172.21.15.43:0/3446644194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82741027a0 0x7f8274199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:55.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.657+0000 7f8273e00640 1 --2- 172.21.15.43:0/3446644194 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f82580a1e90 0x7f82580a4350 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:55.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.657+0000 7f8273e00640 1 --2- 172.21.15.43:0/3446644194 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f82741015b0 0x7f8274199a40 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:55.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.657+0000 7f8273e00640 1 -- 172.21.15.43:0/3446644194 >> 172.21.15.43:0/3446644194 conn(0x7f82740fcb20 msgr2=0x7f8274073980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:55.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.657+0000 7f8273e00640 1 -- 172.21.15.43:0/3446644194 shutdown_connections 2024-03-28T22:36:55.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.657+0000 7f8273e00640 1 -- 172.21.15.43:0/3446644194 wait complete. 2024-03-28T22:36:55.676 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-28T22:36:55.676 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-28T22:36:55.676 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T22:36:55.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.902+0000 7f5a0ea00640 1 -- 172.21.15.43:0/1083597709 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a08104b20 msgr2=0x7f5a08106f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:55.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.902+0000 7f5a0ea00640 1 --2- 172.21.15.43:0/1083597709 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a08104b20 0x7f5a08106f10 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f59f4009900 tx=0x7f59f402bc50 comp rx=0 tx=0).stop 2024-03-28T22:36:55.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.902+0000 7f5a0ea00640 1 -- 172.21.15.43:0/1083597709 shutdown_connections 2024-03-28T22:36:55.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.902+0000 7f5a0ea00640 1 --2- 172.21.15.43:0/1083597709 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a08104b20 0x7f5a08106f10 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:55.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.902+0000 7f5a0ea00640 1 --2- 172.21.15.43:0/1083597709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a08107450 0x7f5a081098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:55.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.902+0000 7f5a0ea00640 1 --2- 172.21.15.43:0/1083597709 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a081021f0 0x7f5a081045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:55.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.902+0000 7f5a0ea00640 1 -- 172.21.15.43:0/1083597709 >> 172.21.15.43:0/1083597709 conn(0x7f5a080fbbe0 msgr2=0x7f5a080fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:55.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.903+0000 7f5a0ea00640 1 -- 172.21.15.43:0/1083597709 shutdown_connections 2024-03-28T22:36:55.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.903+0000 7f5a0ea00640 1 -- 172.21.15.43:0/1083597709 wait complete. 2024-03-28T22:36:55.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.904+0000 7f5a0ea00640 1 Processor -- start 2024-03-28T22:36:55.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.904+0000 7f5a0ea00640 1 -- start start 2024-03-28T22:36:55.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.904+0000 7f5a0ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a081021f0 0x7f5a08199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:55.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.904+0000 7f5a0ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a08104b20 0x7f5a0819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:55.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.904+0000 7f5a0ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a08107450 0x7f5a081a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:55.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.904+0000 7f5a0ea00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0810d430 con 0x7f5a08107450 2024-03-28T22:36:55.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.904+0000 7f5a0ea00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0810d2b0 con 0x7f5a081021f0 2024-03-28T22:36:55.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.904+0000 7f5a0ea00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0810d5b0 con 0x7f5a08104b20 2024-03-28T22:36:55.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.904+0000 7f5a07e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a081021f0 0x7f5a08199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:55.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.904+0000 7f5a07e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a081021f0 0x7f5a08199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:37348/0 (socket says 172.21.15.43:37348) 2024-03-28T22:36:55.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.904+0000 7f5a07e00640 1 -- 172.21.15.43:0/2707371818 learned_addr learned my addr 172.21.15.43:0/2707371818 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:55.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.904+0000 7f5a0ca00640 1 --2- 172.21.15.43:0/2707371818 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a08107450 0x7f5a081a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:55.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.905+0000 7f5a07e00640 1 -- 172.21.15.43:0/2707371818 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a08104b20 msgr2=0x7f5a0819a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:55.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.905+0000 7f5a07e00640 1 --2- 172.21.15.43:0/2707371818 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a08104b20 0x7f5a0819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:55.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.905+0000 7f5a07e00640 1 -- 172.21.15.43:0/2707371818 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a08107450 msgr2=0x7f5a081a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:55.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.905+0000 7f5a07e00640 1 --2- 172.21.15.43:0/2707371818 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a08107450 0x7f5a081a11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:55.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.905+0000 7f5a07e00640 1 -- 172.21.15.43:0/2707371818 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a081a1890 con 0x7f5a081021f0 2024-03-28T22:36:55.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.905+0000 7f5a07e00640 1 --2- 172.21.15.43:0/2707371818 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a081021f0 0x7f5a08199c80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f59f800ea60 tx=0x7f59f800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:55.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.905+0000 7f5a04c00640 1 -- 172.21.15.43:0/2707371818 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f800cde0 con 0x7f5a081021f0 2024-03-28T22:36:55.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.905+0000 7f5a04c00640 1 -- 172.21.15.43:0/2707371818 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59f80035f0 con 0x7f5a081021f0 2024-03-28T22:36:55.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.905+0000 7f5a04c00640 1 -- 172.21.15.43:0/2707371818 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f8010670 con 0x7f5a081021f0 2024-03-28T22:36:55.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.905+0000 7f5a0ea00640 1 -- 172.21.15.43:0/2707371818 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a081a1b80 con 0x7f5a081021f0 2024-03-28T22:36:55.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.905+0000 7f5a0ea00640 1 -- 172.21.15.43:0/2707371818 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a081a20d0 con 0x7f5a081021f0 2024-03-28T22:36:55.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.906+0000 7f5a0ea00640 1 -- 172.21.15.43:0/2707371818 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a08108fa0 con 0x7f5a081021f0 2024-03-28T22:36:55.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.908+0000 7f5a04c00640 1 -- 172.21.15.43:0/2707371818 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7f59f80037a0 con 0x7f5a081021f0 2024-03-28T22:36:55.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.909+0000 7f5a04c00640 1 --2- 172.21.15.43:0/2707371818 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f59ec0a1e10 0x7f59ec0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:55.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.910+0000 7f5a04c00640 1 -- 172.21.15.43:0/2707371818 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f59f80ce700 con 0x7f5a081021f0 2024-03-28T22:36:55.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.910+0000 7f5a07400640 1 --2- 172.21.15.43:0/2707371818 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f59ec0a1e10 0x7f59ec0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:55.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.910+0000 7f5a07400640 1 --2- 172.21.15.43:0/2707371818 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f59ec0a1e10 0x7f59ec0a42d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f59f4036040 tx=0x7f59f4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:55.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:55.913+0000 7f5a04c00640 1 -- 172.21.15.43:0/2707371818 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f59f809b5e0 con 0x7f5a081021f0 2024-03-28T22:36:56.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.111+0000 7f5a0ea00640 1 -- 172.21.15.43:0/2707371818 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5a08061b90 con 0x7f59ec0a1e10 2024-03-28T22:36:56.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.133+0000 7f5a04c00640 1 -- 172.21.15.43:0/2707371818 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340529 (secure 0 0 0) 0x7f5a08061b90 con 0x7f59ec0a1e10 2024-03-28T22:36:56.136 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:36:56.140 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-03-28T22:36:56.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.142+0000 7f5a0ea00640 1 -- 172.21.15.43:0/2707371818 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f59ec0a1e10 msgr2=0x7f59ec0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:56.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.142+0000 7f5a0ea00640 1 --2- 172.21.15.43:0/2707371818 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f59ec0a1e10 0x7f59ec0a42d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f59f4036040 tx=0x7f59f4000bc0 comp rx=0 tx=0).stop 2024-03-28T22:36:56.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.142+0000 7f5a0ea00640 1 -- 172.21.15.43:0/2707371818 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a081021f0 msgr2=0x7f5a08199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:56.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.142+0000 7f5a0ea00640 1 --2- 172.21.15.43:0/2707371818 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a081021f0 0x7f5a08199c80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f59f800ea60 tx=0x7f59f800ef20 comp rx=0 tx=0).stop 2024-03-28T22:36:56.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.142+0000 7f5a0ea00640 1 -- 172.21.15.43:0/2707371818 shutdown_connections 2024-03-28T22:36:56.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.142+0000 7f5a0ea00640 1 --2- 172.21.15.43:0/2707371818 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a08107450 0x7f5a081a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:56.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.142+0000 7f5a0ea00640 1 --2- 172.21.15.43:0/2707371818 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a08104b20 0x7f5a0819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:56.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.142+0000 7f5a0ea00640 1 --2- 172.21.15.43:0/2707371818 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f59ec0a1e10 0x7f59ec0a42d0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:56.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.142+0000 7f5a0ea00640 1 --2- 172.21.15.43:0/2707371818 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a081021f0 0x7f5a08199c80 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:56.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.142+0000 7f5a0ea00640 1 -- 172.21.15.43:0/2707371818 >> 172.21.15.43:0/2707371818 conn(0x7f5a080fbbe0 msgr2=0x7f5a08101910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:56.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.142+0000 7f5a0ea00640 1 -- 172.21.15.43:0/2707371818 shutdown_connections 2024-03-28T22:36:56.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.143+0000 7f5a0ea00640 1 -- 172.21.15.43:0/2707371818 wait complete. 2024-03-28T22:36:56.160 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2024-03-28T22:36:54.244860+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":663405,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1831,"num_read_kb":1043,"num_write":445,"num_write_kb":3095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":865,"ondisk_log_size":865,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":376724,"kb_used_data":15064,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124565100,"statfs":{"total":1151940427776,"available":1151554662400,"internally_reserved":0,"allocated":15425536,"data_stored":8441517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1394,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":119,"num_read_kb":59,"num_write":12,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005232"},"pg_stats":[{"pgid":"3.3f","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.450073+0000","last_change":"2024-03-28T22:33:28.942904+0000","last_active":"2024-03-28T22:35:30.450073+0000","last_peered":"2024-03-28T22:35:30.450073+0000","last_clean":"2024-03-28T22:35:30.450073+0000","last_became_active":"2024-03-28T22:33:28.942574+0000","last_became_peered":"2024-03-28T22:33:28.942574+0000","last_unstale":"2024-03-28T22:35:30.450073+0000","last_undegraded":"2024-03-28T22:35:30.450073+0000","last_fullsized":"2024-03-28T22:35:30.450073+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:55.323919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"94'4","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451519+0000","last_change":"2024-03-28T22:33:28.951795+0000","last_active":"2024-03-28T22:35:30.451519+0000","last_peered":"2024-03-28T22:35:30.451519+0000","last_clean":"2024-03-28T22:35:30.451519+0000","last_became_active":"2024-03-28T22:33:28.951286+0000","last_became_peered":"2024-03-28T22:33:28.951286+0000","last_unstale":"2024-03-28T22:35:30.451519+0000","last_undegraded":"2024-03-28T22:35:30.451519+0000","last_fullsized":"2024-03-28T22:35:30.451519+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:56:11.718244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451291+0000","last_change":"2024-03-28T22:33:28.949476+0000","last_active":"2024-03-28T22:35:30.451291+0000","last_peered":"2024-03-28T22:35:30.451291+0000","last_clean":"2024-03-28T22:35:30.451291+0000","last_became_active":"2024-03-28T22:33:28.949128+0000","last_became_peered":"2024-03-28T22:33:28.949128+0000","last_unstale":"2024-03-28T22:35:30.451291+0000","last_undegraded":"2024-03-28T22:35:30.451291+0000","last_fullsized":"2024-03-28T22:35:30.451291+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:49:29.052373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370703+0000","last_change":"2024-03-28T22:33:28.930715+0000","last_active":"2024-03-28T22:35:30.370703+0000","last_peered":"2024-03-28T22:35:30.370703+0000","last_clean":"2024-03-28T22:35:30.370703+0000","last_became_active":"2024-03-28T22:33:28.930265+0000","last_became_peered":"2024-03-28T22:33:28.930265+0000","last_unstale":"2024-03-28T22:35:30.370703+0000","last_undegraded":"2024-03-28T22:35:30.370703+0000","last_fullsized":"2024-03-28T22:35:30.370703+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:29:54.695671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"94'4","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.253685+0000","last_change":"2024-03-28T22:33:28.944933+0000","last_active":"2024-03-28T22:35:30.253685+0000","last_peered":"2024-03-28T22:35:30.253685+0000","last_clean":"2024-03-28T22:35:30.253685+0000","last_became_active":"2024-03-28T22:33:28.944542+0000","last_became_peered":"2024-03-28T22:33:28.944542+0000","last_unstale":"2024-03-28T22:35:30.253685+0000","last_undegraded":"2024-03-28T22:35:30.253685+0000","last_fullsized":"2024-03-28T22:35:30.253685+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:35:15.719312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"94'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.259295+0000","last_change":"2024-03-28T22:33:28.927920+0000","last_active":"2024-03-28T22:35:30.259295+0000","last_peered":"2024-03-28T22:35:30.259295+0000","last_clean":"2024-03-28T22:35:30.259295+0000","last_became_active":"2024-03-28T22:33:28.927602+0000","last_became_peered":"2024-03-28T22:33:28.927602+0000","last_unstale":"2024-03-28T22:35:30.259295+0000","last_undegraded":"2024-03-28T22:35:30.259295+0000","last_fullsized":"2024-03-28T22:35:30.259295+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:24:22.072707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"94'5","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248059+0000","last_change":"2024-03-28T22:33:28.917963+0000","last_active":"2024-03-28T22:35:30.248059+0000","last_peered":"2024-03-28T22:35:30.248059+0000","last_clean":"2024-03-28T22:35:30.248059+0000","last_became_active":"2024-03-28T22:33:28.917628+0000","last_became_peered":"2024-03-28T22:33:28.917628+0000","last_unstale":"2024-03-28T22:35:30.248059+0000","last_undegraded":"2024-03-28T22:35:30.248059+0000","last_fullsized":"2024-03-28T22:35:30.248059+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:37:09.111640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"94'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371177+0000","last_change":"2024-03-28T22:33:28.942632+0000","last_active":"2024-03-28T22:35:30.371177+0000","last_peered":"2024-03-28T22:35:30.371177+0000","last_clean":"2024-03-28T22:35:30.371177+0000","last_became_active":"2024-03-28T22:33:28.941972+0000","last_became_peered":"2024-03-28T22:33:28.941972+0000","last_unstale":"2024-03-28T22:35:30.371177+0000","last_undegraded":"2024-03-28T22:35:30.371177+0000","last_fullsized":"2024-03-28T22:35:30.371177+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:49:52.642314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"94'11","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448774+0000","last_change":"2024-03-28T22:33:28.924324+0000","last_active":"2024-03-28T22:35:30.448774+0000","last_peered":"2024-03-28T22:35:30.448774+0000","last_clean":"2024-03-28T22:35:30.448774+0000","last_became_active":"2024-03-28T22:33:28.924014+0000","last_became_peered":"2024-03-28T22:33:28.924014+0000","last_unstale":"2024-03-28T22:35:30.448774+0000","last_undegraded":"2024-03-28T22:35:30.448774+0000","last_fullsized":"2024-03-28T22:35:30.448774+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:17:55.172116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"94'16","reported_seq":54,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.259584+0000","last_change":"2024-03-28T22:34:29.987262+0000","last_active":"2024-03-28T22:35:30.259584+0000","last_peered":"2024-03-28T22:35:30.259584+0000","last_clean":"2024-03-28T22:35:30.259584+0000","last_became_active":"2024-03-28T22:34:29.986934+0000","last_became_peered":"2024-03-28T22:34:29.986934+0000","last_unstale":"2024-03-28T22:35:30.259584+0000","last_undegraded":"2024-03-28T22:35:30.259584+0000","last_fullsized":"2024-03-28T22:35:30.259584+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:00:41.574800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"94'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248516+0000","last_change":"2024-03-28T22:33:28.930802+0000","last_active":"2024-03-28T22:35:30.248516+0000","last_peered":"2024-03-28T22:35:30.248516+0000","last_clean":"2024-03-28T22:35:30.248516+0000","last_became_active":"2024-03-28T22:33:28.930475+0000","last_became_peered":"2024-03-28T22:33:28.930475+0000","last_unstale":"2024-03-28T22:35:30.248516+0000","last_undegraded":"2024-03-28T22:35:30.248516+0000","last_fullsized":"2024-03-28T22:35:30.248516+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:48.443803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"94'7","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370695+0000","last_change":"2024-03-28T22:33:28.927057+0000","last_active":"2024-03-28T22:35:30.370695+0000","last_peered":"2024-03-28T22:35:30.370695+0000","last_clean":"2024-03-28T22:35:30.370695+0000","last_became_active":"2024-03-28T22:33:28.926598+0000","last_became_peered":"2024-03-28T22:33:28.926598+0000","last_unstale":"2024-03-28T22:35:30.370695+0000","last_undegraded":"2024-03-28T22:35:30.370695+0000","last_fullsized":"2024-03-28T22:35:30.370695+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:21:00.710203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"94'10","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368918+0000","last_change":"2024-03-28T22:33:28.937034+0000","last_active":"2024-03-28T22:35:30.368918+0000","last_peered":"2024-03-28T22:35:30.368918+0000","last_clean":"2024-03-28T22:35:30.368918+0000","last_became_active":"2024-03-28T22:33:28.933713+0000","last_became_peered":"2024-03-28T22:33:28.933713+0000","last_unstale":"2024-03-28T22:35:30.368918+0000","last_undegraded":"2024-03-28T22:35:30.368918+0000","last_fullsized":"2024-03-28T22:35:30.368918+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:58:24.148897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.453212+0000","last_change":"2024-03-28T22:33:28.947384+0000","last_active":"2024-03-28T22:35:30.453212+0000","last_peered":"2024-03-28T22:35:30.453212+0000","last_clean":"2024-03-28T22:35:30.453212+0000","last_became_active":"2024-03-28T22:33:28.947017+0000","last_became_peered":"2024-03-28T22:33:28.947017+0000","last_unstale":"2024-03-28T22:35:30.453212+0000","last_undegraded":"2024-03-28T22:35:30.453212+0000","last_fullsized":"2024-03-28T22:35:30.453212+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:43:32.781161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451880+0000","last_change":"2024-03-28T22:33:28.950119+0000","last_active":"2024-03-28T22:35:30.451880+0000","last_peered":"2024-03-28T22:35:30.451880+0000","last_clean":"2024-03-28T22:35:30.451880+0000","last_became_active":"2024-03-28T22:33:28.949798+0000","last_became_peered":"2024-03-28T22:33:28.949798+0000","last_unstale":"2024-03-28T22:35:30.451880+0000","last_undegraded":"2024-03-28T22:35:30.451880+0000","last_fullsized":"2024-03-28T22:35:30.451880+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:16.717678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"94'9","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448318+0000","last_change":"2024-03-28T22:33:28.943255+0000","last_active":"2024-03-28T22:35:30.448318+0000","last_peered":"2024-03-28T22:35:30.448318+0000","last_clean":"2024-03-28T22:35:30.448318+0000","last_became_active":"2024-03-28T22:33:28.942923+0000","last_became_peered":"2024-03-28T22:33:28.942923+0000","last_unstale":"2024-03-28T22:35:30.448318+0000","last_undegraded":"2024-03-28T22:35:30.448318+0000","last_fullsized":"2024-03-28T22:35:30.448318+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:05:04.197635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449964+0000","last_change":"2024-03-28T22:33:28.942906+0000","last_active":"2024-03-28T22:35:30.449964+0000","last_peered":"2024-03-28T22:35:30.449964+0000","last_clean":"2024-03-28T22:35:30.449964+0000","last_became_active":"2024-03-28T22:33:28.942573+0000","last_became_peered":"2024-03-28T22:33:28.942573+0000","last_unstale":"2024-03-28T22:35:30.449964+0000","last_undegraded":"2024-03-28T22:35:30.449964+0000","last_fullsized":"2024-03-28T22:35:30.449964+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:39:17.651101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"94'10","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252948+0000","last_change":"2024-03-28T22:33:28.932075+0000","last_active":"2024-03-28T22:35:30.252948+0000","last_peered":"2024-03-28T22:35:30.252948+0000","last_clean":"2024-03-28T22:35:30.252948+0000","last_became_active":"2024-03-28T22:33:28.931689+0000","last_became_peered":"2024-03-28T22:33:28.931689+0000","last_unstale":"2024-03-28T22:35:30.252948+0000","last_undegraded":"2024-03-28T22:35:30.252948+0000","last_fullsized":"2024-03-28T22:35:30.252948+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:50:43.031082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252343+0000","last_change":"2024-03-28T22:33:28.944987+0000","last_active":"2024-03-28T22:35:30.252343+0000","last_peered":"2024-03-28T22:35:30.252343+0000","last_clean":"2024-03-28T22:35:30.252343+0000","last_became_active":"2024-03-28T22:33:28.944622+0000","last_became_peered":"2024-03-28T22:33:28.944622+0000","last_unstale":"2024-03-28T22:35:30.252343+0000","last_undegraded":"2024-03-28T22:35:30.252343+0000","last_fullsized":"2024-03-28T22:35:30.252343+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:48:58.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":30,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251826+0000","last_change":"2024-03-28T22:34:30.429709+0000","last_active":"2024-03-28T22:35:30.251826+0000","last_peered":"2024-03-28T22:35:30.251826+0000","last_clean":"2024-03-28T22:35:30.251826+0000","last_became_active":"2024-03-28T22:34:30.429122+0000","last_became_peered":"2024-03-28T22:34:30.429122+0000","last_unstale":"2024-03-28T22:35:30.251826+0000","last_undegraded":"2024-03-28T22:35:30.251826+0000","last_fullsized":"2024-03-28T22:35:30.251826+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:58:05.313669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"92'13","reported_seq":214,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:36:50.437836+0000","last_change":"2024-03-28T22:33:28.442524+0000","last_active":"2024-03-28T22:36:50.437836+0000","last_peered":"2024-03-28T22:36:50.437836+0000","last_clean":"2024-03-28T22:36:50.437836+0000","last_became_active":"2024-03-28T22:33:28.442169+0000","last_became_peered":"2024-03-28T22:33:28.442169+0000","last_unstale":"2024-03-28T22:36:50.437836+0000","last_undegraded":"2024-03-28T22:36:50.437836+0000","last_fullsized":"2024-03-28T22:36:50.437836+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:08:47.101119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":167,"num_read_kb":167,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"94'5","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371822+0000","last_change":"2024-03-28T22:33:28.946697+0000","last_active":"2024-03-28T22:35:30.371822+0000","last_peered":"2024-03-28T22:35:30.371822+0000","last_clean":"2024-03-28T22:35:30.371822+0000","last_became_active":"2024-03-28T22:33:28.946303+0000","last_became_peered":"2024-03-28T22:33:28.946303+0000","last_unstale":"2024-03-28T22:35:30.371822+0000","last_undegraded":"2024-03-28T22:35:30.371822+0000","last_fullsized":"2024-03-28T22:35:30.371822+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:32:01.286073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":30,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370372+0000","last_change":"2024-03-28T22:34:30.429579+0000","last_active":"2024-03-28T22:35:30.370372+0000","last_peered":"2024-03-28T22:35:30.370372+0000","last_clean":"2024-03-28T22:35:30.370372+0000","last_became_active":"2024-03-28T22:34:30.429054+0000","last_became_peered":"2024-03-28T22:34:30.429054+0000","last_unstale":"2024-03-28T22:35:30.370372+0000","last_undegraded":"2024-03-28T22:35:30.370372+0000","last_fullsized":"2024-03-28T22:35:30.370372+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:03:46.492786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"92'2","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.372174+0000","last_change":"2024-03-28T22:33:28.457661+0000","last_active":"2024-03-28T22:35:30.372174+0000","last_peered":"2024-03-28T22:35:30.372174+0000","last_clean":"2024-03-28T22:35:30.372174+0000","last_became_active":"2024-03-28T22:33:28.457032+0000","last_became_peered":"2024-03-28T22:33:28.457032+0000","last_unstale":"2024-03-28T22:35:30.372174+0000","last_undegraded":"2024-03-28T22:35:30.372174+0000","last_fullsized":"2024-03-28T22:35:30.372174+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:43:09.820630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"94'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.453641+0000","last_change":"2024-03-28T22:33:28.954252+0000","last_active":"2024-03-28T22:35:30.453641+0000","last_peered":"2024-03-28T22:35:30.453641+0000","last_clean":"2024-03-28T22:35:30.453641+0000","last_became_active":"2024-03-28T22:33:28.953922+0000","last_became_peered":"2024-03-28T22:33:28.953922+0000","last_unstale":"2024-03-28T22:35:30.453641+0000","last_undegraded":"2024-03-28T22:35:30.453641+0000","last_fullsized":"2024-03-28T22:35:30.453641+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:18:47.097859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370607+0000","last_change":"2024-03-28T22:33:34.029129+0000","last_active":"2024-03-28T22:35:30.370607+0000","last_peered":"2024-03-28T22:35:30.370607+0000","last_clean":"2024-03-28T22:35:30.370607+0000","last_became_active":"2024-03-28T22:33:34.028452+0000","last_became_peered":"2024-03-28T22:33:34.028452+0000","last_unstale":"2024-03-28T22:35:30.370607+0000","last_undegraded":"2024-03-28T22:35:30.370607+0000","last_fullsized":"2024-03-28T22:35:30.370607+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:28.982773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"92'12","reported_seq":47,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254205+0000","last_change":"2024-03-28T22:33:28.439568+0000","last_active":"2024-03-28T22:35:30.254205+0000","last_peered":"2024-03-28T22:35:30.254205+0000","last_clean":"2024-03-28T22:35:30.254205+0000","last_became_active":"2024-03-28T22:33:28.438887+0000","last_became_peered":"2024-03-28T22:33:28.438887+0000","last_unstale":"2024-03-28T22:35:30.254205+0000","last_undegraded":"2024-03-28T22:35:30.254205+0000","last_fullsized":"2024-03-28T22:35:30.254205+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:56:43.617621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"94'9","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448797+0000","last_change":"2024-03-28T22:33:28.938311+0000","last_active":"2024-03-28T22:35:30.448797+0000","last_peered":"2024-03-28T22:35:30.448797+0000","last_clean":"2024-03-28T22:35:30.448797+0000","last_became_active":"2024-03-28T22:33:28.937997+0000","last_became_peered":"2024-03-28T22:33:28.937997+0000","last_unstale":"2024-03-28T22:35:30.448797+0000","last_undegraded":"2024-03-28T22:35:30.448797+0000","last_fullsized":"2024-03-28T22:35:30.448797+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:54:44.001545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370428+0000","last_change":"2024-03-28T22:33:34.016931+0000","last_active":"2024-03-28T22:35:30.370428+0000","last_peered":"2024-03-28T22:35:30.370428+0000","last_clean":"2024-03-28T22:35:30.370428+0000","last_became_active":"2024-03-28T22:33:34.016559+0000","last_became_peered":"2024-03-28T22:33:34.016559+0000","last_unstale":"2024-03-28T22:35:30.370428+0000","last_undegraded":"2024-03-28T22:35:30.370428+0000","last_fullsized":"2024-03-28T22:35:30.370428+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:08:31.252351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"92'5","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254097+0000","last_change":"2024-03-28T22:33:28.433166+0000","last_active":"2024-03-28T22:35:30.254097+0000","last_peered":"2024-03-28T22:35:30.254097+0000","last_clean":"2024-03-28T22:35:30.254097+0000","last_became_active":"2024-03-28T22:33:28.432716+0000","last_became_peered":"2024-03-28T22:33:28.432716+0000","last_unstale":"2024-03-28T22:35:30.254097+0000","last_undegraded":"2024-03-28T22:35:30.254097+0000","last_fullsized":"2024-03-28T22:35:30.254097+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:42:41.613583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"94'15","reported_seq":47,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275541+0000","last_change":"2024-03-28T22:33:28.923307+0000","last_active":"2024-03-28T22:35:30.275541+0000","last_peered":"2024-03-28T22:35:30.275541+0000","last_clean":"2024-03-28T22:35:30.275541+0000","last_became_active":"2024-03-28T22:33:28.922499+0000","last_became_peered":"2024-03-28T22:33:28.922499+0000","last_unstale":"2024-03-28T22:35:30.275541+0000","last_undegraded":"2024-03-28T22:35:30.275541+0000","last_fullsized":"2024-03-28T22:35:30.275541+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:48:48.514482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252183+0000","last_change":"2024-03-28T22:33:34.002209+0000","last_active":"2024-03-28T22:35:30.252183+0000","last_peered":"2024-03-28T22:35:30.252183+0000","last_clean":"2024-03-28T22:35:30.252183+0000","last_became_active":"2024-03-28T22:33:34.001746+0000","last_became_peered":"2024-03-28T22:33:34.001746+0000","last_unstale":"2024-03-28T22:35:30.252183+0000","last_undegraded":"2024-03-28T22:35:30.252183+0000","last_fullsized":"2024-03-28T22:35:30.252183+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:02:02.333328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"96'7","reported_seq":211,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:36:50.482307+0000","last_change":"2024-03-28T22:33:28.435380+0000","last_active":"2024-03-28T22:36:50.482307+0000","last_peered":"2024-03-28T22:36:50.482307+0000","last_clean":"2024-03-28T22:36:50.482307+0000","last_became_active":"2024-03-28T22:33:28.434898+0000","last_became_peered":"2024-03-28T22:33:28.434898+0000","last_unstale":"2024-03-28T22:36:50.482307+0000","last_undegraded":"2024-03-28T22:36:50.482307+0000","last_fullsized":"2024-03-28T22:36:50.482307+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:58:06.810394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":170,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"94'7","reported_seq":45,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254282+0000","last_change":"2024-03-28T22:34:30.121370+0000","last_active":"2024-03-28T22:35:30.254282+0000","last_peered":"2024-03-28T22:35:30.254282+0000","last_clean":"2024-03-28T22:35:30.254282+0000","last_became_active":"2024-03-28T22:34:30.121086+0000","last_became_peered":"2024-03-28T22:34:30.121086+0000","last_unstale":"2024-03-28T22:35:30.254282+0000","last_undegraded":"2024-03-28T22:35:30.254282+0000","last_fullsized":"2024-03-28T22:35:30.254282+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:26:24.244663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252057+0000","last_change":"2024-03-28T22:33:34.031350+0000","last_active":"2024-03-28T22:35:30.252057+0000","last_peered":"2024-03-28T22:35:30.252057+0000","last_clean":"2024-03-28T22:35:30.252057+0000","last_became_active":"2024-03-28T22:33:34.030946+0000","last_became_peered":"2024-03-28T22:33:34.030946+0000","last_unstale":"2024-03-28T22:35:30.252057+0000","last_undegraded":"2024-03-28T22:35:30.252057+0000","last_fullsized":"2024-03-28T22:35:30.252057+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:54:22.907363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"92'7","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.259226+0000","last_change":"2024-03-28T22:33:28.446197+0000","last_active":"2024-03-28T22:35:30.259226+0000","last_peered":"2024-03-28T22:35:30.259226+0000","last_clean":"2024-03-28T22:35:30.259226+0000","last_became_active":"2024-03-28T22:33:28.445891+0000","last_became_peered":"2024-03-28T22:33:28.445891+0000","last_unstale":"2024-03-28T22:35:30.259226+0000","last_undegraded":"2024-03-28T22:35:30.259226+0000","last_fullsized":"2024-03-28T22:35:30.259226+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:58:57.385302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"94'8","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275591+0000","last_change":"2024-03-28T22:33:28.955651+0000","last_active":"2024-03-28T22:35:30.275591+0000","last_peered":"2024-03-28T22:35:30.275591+0000","last_clean":"2024-03-28T22:35:30.275591+0000","last_became_active":"2024-03-28T22:33:28.955276+0000","last_became_peered":"2024-03-28T22:33:28.955276+0000","last_unstale":"2024-03-28T22:35:30.275591+0000","last_undegraded":"2024-03-28T22:35:30.275591+0000","last_fullsized":"2024-03-28T22:35:30.275591+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:36:13.455327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449809+0000","last_change":"2024-03-28T22:33:34.017520+0000","last_active":"2024-03-28T22:35:30.449809+0000","last_peered":"2024-03-28T22:35:30.449809+0000","last_clean":"2024-03-28T22:35:30.449809+0000","last_became_active":"2024-03-28T22:33:34.017022+0000","last_became_peered":"2024-03-28T22:33:34.017022+0000","last_unstale":"2024-03-28T22:35:30.449809+0000","last_undegraded":"2024-03-28T22:35:30.449809+0000","last_fullsized":"2024-03-28T22:35:30.449809+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:41:17.431581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"92'3","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368171+0000","last_change":"2024-03-28T22:33:28.445401+0000","last_active":"2024-03-28T22:35:30.368171+0000","last_peered":"2024-03-28T22:35:30.368171+0000","last_clean":"2024-03-28T22:35:30.368171+0000","last_became_active":"2024-03-28T22:33:28.444237+0000","last_became_peered":"2024-03-28T22:33:28.444237+0000","last_unstale":"2024-03-28T22:35:30.368171+0000","last_undegraded":"2024-03-28T22:35:30.368171+0000","last_fullsized":"2024-03-28T22:35:30.368171+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:18:12.820890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"94'8","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.250889+0000","last_change":"2024-03-28T22:34:30.156248+0000","last_active":"2024-03-28T22:35:30.250889+0000","last_peered":"2024-03-28T22:35:30.250889+0000","last_clean":"2024-03-28T22:35:30.250889+0000","last_became_active":"2024-03-28T22:34:30.155056+0000","last_became_peered":"2024-03-28T22:34:30.155056+0000","last_unstale":"2024-03-28T22:35:30.250889+0000","last_undegraded":"2024-03-28T22:35:30.250889+0000","last_fullsized":"2024-03-28T22:35:30.250889+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:28:56.469728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451480+0000","last_change":"2024-03-28T22:33:34.019899+0000","last_active":"2024-03-28T22:35:30.451480+0000","last_peered":"2024-03-28T22:35:30.451480+0000","last_clean":"2024-03-28T22:35:30.451480+0000","last_became_active":"2024-03-28T22:33:34.019392+0000","last_became_peered":"2024-03-28T22:33:34.019392+0000","last_unstale":"2024-03-28T22:35:30.451480+0000","last_undegraded":"2024-03-28T22:35:30.451480+0000","last_fullsized":"2024-03-28T22:35:30.451480+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:08:38.633797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"92'3","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275105+0000","last_change":"2024-03-28T22:33:28.433243+0000","last_active":"2024-03-28T22:35:30.275105+0000","last_peered":"2024-03-28T22:35:30.275105+0000","last_clean":"2024-03-28T22:35:30.275105+0000","last_became_active":"2024-03-28T22:33:28.432927+0000","last_became_peered":"2024-03-28T22:33:28.432927+0000","last_unstale":"2024-03-28T22:35:30.275105+0000","last_undegraded":"2024-03-28T22:35:30.275105+0000","last_fullsized":"2024-03-28T22:35:30.275105+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:40:43.444229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"94'11","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251345+0000","last_change":"2024-03-28T22:33:28.948111+0000","last_active":"2024-03-28T22:35:30.251345+0000","last_peered":"2024-03-28T22:35:30.251345+0000","last_clean":"2024-03-28T22:35:30.251345+0000","last_became_active":"2024-03-28T22:33:28.944649+0000","last_became_peered":"2024-03-28T22:33:28.944649+0000","last_unstale":"2024-03-28T22:35:30.251345+0000","last_undegraded":"2024-03-28T22:35:30.251345+0000","last_fullsized":"2024-03-28T22:35:30.251345+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:57:11.483037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371506+0000","last_change":"2024-03-28T22:33:34.000837+0000","last_active":"2024-03-28T22:35:30.371506+0000","last_peered":"2024-03-28T22:35:30.371506+0000","last_clean":"2024-03-28T22:35:30.371506+0000","last_became_active":"2024-03-28T22:33:34.000120+0000","last_became_peered":"2024-03-28T22:33:34.000120+0000","last_unstale":"2024-03-28T22:35:30.371506+0000","last_undegraded":"2024-03-28T22:35:30.371506+0000","last_fullsized":"2024-03-28T22:35:30.371506+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:14:32.028952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"96'5","reported_seq":366,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:36:48.476091+0000","last_change":"2024-03-28T22:33:28.436985+0000","last_active":"2024-03-28T22:36:48.476091+0000","last_peered":"2024-03-28T22:36:48.476091+0000","last_clean":"2024-03-28T22:36:48.476091+0000","last_became_active":"2024-03-28T22:33:28.436625+0000","last_became_peered":"2024-03-28T22:33:28.436625+0000","last_unstale":"2024-03-28T22:36:48.476091+0000","last_undegraded":"2024-03-28T22:36:48.476091+0000","last_fullsized":"2024-03-28T22:36:48.476091+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:50:45.775191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":327,"num_read_kb":327,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"94'10","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448895+0000","last_change":"2024-03-28T22:33:28.933265+0000","last_active":"2024-03-28T22:35:30.448895+0000","last_peered":"2024-03-28T22:35:30.448895+0000","last_clean":"2024-03-28T22:35:30.448895+0000","last_became_active":"2024-03-28T22:33:28.932954+0000","last_became_peered":"2024-03-28T22:33:28.932954+0000","last_unstale":"2024-03-28T22:35:30.448895+0000","last_undegraded":"2024-03-28T22:35:30.448895+0000","last_fullsized":"2024-03-28T22:35:30.448895+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:37.039055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251275+0000","last_change":"2024-03-28T22:33:34.024529+0000","last_active":"2024-03-28T22:35:30.251275+0000","last_peered":"2024-03-28T22:35:30.251275+0000","last_clean":"2024-03-28T22:35:30.251275+0000","last_became_active":"2024-03-28T22:33:34.020949+0000","last_became_peered":"2024-03-28T22:33:34.020949+0000","last_unstale":"2024-03-28T22:35:30.251275+0000","last_undegraded":"2024-03-28T22:35:30.251275+0000","last_fullsized":"2024-03-28T22:35:30.251275+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:28:52.039684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"92'4","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248388+0000","last_change":"2024-03-28T22:33:28.432599+0000","last_active":"2024-03-28T22:35:30.248388+0000","last_peered":"2024-03-28T22:35:30.248388+0000","last_clean":"2024-03-28T22:35:30.248388+0000","last_became_active":"2024-03-28T22:33:28.430710+0000","last_became_peered":"2024-03-28T22:33:28.430710+0000","last_unstale":"2024-03-28T22:35:30.248388+0000","last_undegraded":"2024-03-28T22:35:30.248388+0000","last_fullsized":"2024-03-28T22:35:30.248388+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:54.174747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"94'9","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.453918+0000","last_change":"2024-03-28T22:33:28.955441+0000","last_active":"2024-03-28T22:35:30.453918+0000","last_peered":"2024-03-28T22:35:30.453918+0000","last_clean":"2024-03-28T22:35:30.453918+0000","last_became_active":"2024-03-28T22:33:28.953269+0000","last_became_peered":"2024-03-28T22:33:28.953269+0000","last_unstale":"2024-03-28T22:35:30.453918+0000","last_undegraded":"2024-03-28T22:35:30.453918+0000","last_fullsized":"2024-03-28T22:35:30.453918+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:47:04.128605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370719+0000","last_change":"2024-03-28T22:33:34.028791+0000","last_active":"2024-03-28T22:35:30.370719+0000","last_peered":"2024-03-28T22:35:30.370719+0000","last_clean":"2024-03-28T22:35:30.370719+0000","last_became_active":"2024-03-28T22:33:34.028361+0000","last_became_peered":"2024-03-28T22:33:34.028361+0000","last_unstale":"2024-03-28T22:35:30.370719+0000","last_undegraded":"2024-03-28T22:35:30.370719+0000","last_fullsized":"2024-03-28T22:35:30.370719+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:42:21.264459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"92'5","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275283+0000","last_change":"2024-03-28T22:33:28.455353+0000","last_active":"2024-03-28T22:35:30.275283+0000","last_peered":"2024-03-28T22:35:30.275283+0000","last_clean":"2024-03-28T22:35:30.275283+0000","last_became_active":"2024-03-28T22:33:28.455090+0000","last_became_peered":"2024-03-28T22:33:28.455090+0000","last_unstale":"2024-03-28T22:35:30.275283+0000","last_undegraded":"2024-03-28T22:35:30.275283+0000","last_fullsized":"2024-03-28T22:35:30.275283+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:19:22.496600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"94'11","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448743+0000","last_change":"2024-03-28T22:33:28.928875+0000","last_active":"2024-03-28T22:35:30.448743+0000","last_peered":"2024-03-28T22:35:30.448743+0000","last_clean":"2024-03-28T22:35:30.448743+0000","last_became_active":"2024-03-28T22:33:28.928449+0000","last_became_peered":"2024-03-28T22:33:28.928449+0000","last_unstale":"2024-03-28T22:35:30.448743+0000","last_undegraded":"2024-03-28T22:35:30.448743+0000","last_fullsized":"2024-03-28T22:35:30.448743+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:34:00.402100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251348+0000","last_change":"2024-03-28T22:33:34.012739+0000","last_active":"2024-03-28T22:35:30.251348+0000","last_peered":"2024-03-28T22:35:30.251348+0000","last_clean":"2024-03-28T22:35:30.251348+0000","last_became_active":"2024-03-28T22:33:34.012262+0000","last_became_peered":"2024-03-28T22:33:34.012262+0000","last_unstale":"2024-03-28T22:35:30.251348+0000","last_undegraded":"2024-03-28T22:35:30.251348+0000","last_fullsized":"2024-03-28T22:35:30.251348+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:24:48.351037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"96'40","reported_seq":410,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:36:49.352812+0000","last_change":"2024-03-28T22:33:28.430606+0000","last_active":"2024-03-28T22:36:49.352812+0000","last_peered":"2024-03-28T22:36:49.352812+0000","last_clean":"2024-03-28T22:36:49.352812+0000","last_became_active":"2024-03-28T22:33:28.430233+0000","last_became_peered":"2024-03-28T22:33:28.430233+0000","last_unstale":"2024-03-28T22:36:49.352812+0000","last_undegraded":"2024-03-28T22:36:49.352812+0000","last_fullsized":"2024-03-28T22:36:49.352812+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:49.000569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64761,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":336,"num_read_kb":232,"num_write":42,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"94'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254080+0000","last_change":"2024-03-28T22:33:28.931167+0000","last_active":"2024-03-28T22:35:30.254080+0000","last_peered":"2024-03-28T22:35:30.254080+0000","last_clean":"2024-03-28T22:35:30.254080+0000","last_became_active":"2024-03-28T22:33:28.930824+0000","last_became_peered":"2024-03-28T22:33:28.930824+0000","last_unstale":"2024-03-28T22:35:30.254080+0000","last_undegraded":"2024-03-28T22:35:30.254080+0000","last_fullsized":"2024-03-28T22:35:30.254080+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:46:49.102791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275836+0000","last_change":"2024-03-28T22:33:34.023056+0000","last_active":"2024-03-28T22:35:30.275836+0000","last_peered":"2024-03-28T22:35:30.275836+0000","last_clean":"2024-03-28T22:35:30.275836+0000","last_became_active":"2024-03-28T22:33:34.022612+0000","last_became_peered":"2024-03-28T22:33:34.022612+0000","last_unstale":"2024-03-28T22:35:30.275836+0000","last_undegraded":"2024-03-28T22:35:30.275836+0000","last_fullsized":"2024-03-28T22:35:30.275836+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:52:31.936008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"92'9","reported_seq":45,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252111+0000","last_change":"2024-03-28T22:33:28.430963+0000","last_active":"2024-03-28T22:35:30.252111+0000","last_peered":"2024-03-28T22:35:30.252111+0000","last_clean":"2024-03-28T22:35:30.252111+0000","last_became_active":"2024-03-28T22:33:28.430632+0000","last_became_peered":"2024-03-28T22:33:28.430632+0000","last_unstale":"2024-03-28T22:35:30.252111+0000","last_undegraded":"2024-03-28T22:35:30.252111+0000","last_fullsized":"2024-03-28T22:35:30.252111+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:30:23.104769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"94'9","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368385+0000","last_change":"2024-03-28T22:33:28.954105+0000","last_active":"2024-03-28T22:35:30.368385+0000","last_peered":"2024-03-28T22:35:30.368385+0000","last_clean":"2024-03-28T22:35:30.368385+0000","last_became_active":"2024-03-28T22:33:28.953768+0000","last_became_peered":"2024-03-28T22:33:28.953768+0000","last_unstale":"2024-03-28T22:35:30.368385+0000","last_undegraded":"2024-03-28T22:35:30.368385+0000","last_fullsized":"2024-03-28T22:35:30.368385+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:01:25.184523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449565+0000","last_change":"2024-03-28T22:33:34.027976+0000","last_active":"2024-03-28T22:35:30.449565+0000","last_peered":"2024-03-28T22:35:30.449565+0000","last_clean":"2024-03-28T22:35:30.449565+0000","last_became_active":"2024-03-28T22:33:34.027509+0000","last_became_peered":"2024-03-28T22:33:34.027509+0000","last_unstale":"2024-03-28T22:35:30.449565+0000","last_undegraded":"2024-03-28T22:35:30.449565+0000","last_fullsized":"2024-03-28T22:35:30.449565+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:25:41.908441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"92'5","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275307+0000","last_change":"2024-03-28T22:33:28.433283+0000","last_active":"2024-03-28T22:35:30.275307+0000","last_peered":"2024-03-28T22:35:30.275307+0000","last_clean":"2024-03-28T22:35:30.275307+0000","last_became_active":"2024-03-28T22:33:28.432954+0000","last_became_peered":"2024-03-28T22:33:28.432954+0000","last_unstale":"2024-03-28T22:35:30.275307+0000","last_undegraded":"2024-03-28T22:35:30.275307+0000","last_fullsized":"2024-03-28T22:35:30.275307+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:08:34.480459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"94'15","reported_seq":47,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248465+0000","last_change":"2024-03-28T22:33:28.939016+0000","last_active":"2024-03-28T22:35:30.248465+0000","last_peered":"2024-03-28T22:35:30.248465+0000","last_clean":"2024-03-28T22:35:30.248465+0000","last_became_active":"2024-03-28T22:33:28.938695+0000","last_became_peered":"2024-03-28T22:33:28.938695+0000","last_unstale":"2024-03-28T22:35:30.248465+0000","last_undegraded":"2024-03-28T22:35:30.248465+0000","last_fullsized":"2024-03-28T22:35:30.248465+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:26:53.264559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252805+0000","last_change":"2024-03-28T22:33:34.030127+0000","last_active":"2024-03-28T22:35:30.252805+0000","last_peered":"2024-03-28T22:35:30.252805+0000","last_clean":"2024-03-28T22:35:30.252805+0000","last_became_active":"2024-03-28T22:33:34.028056+0000","last_became_peered":"2024-03-28T22:33:34.028056+0000","last_unstale":"2024-03-28T22:35:30.252805+0000","last_undegraded":"2024-03-28T22:35:30.252805+0000","last_fullsized":"2024-03-28T22:35:30.252805+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:29:41.785013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'7","reported_seq":209,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:36:50.560497+0000","last_change":"2024-03-28T22:33:28.435350+0000","last_active":"2024-03-28T22:36:50.560497+0000","last_peered":"2024-03-28T22:36:50.560497+0000","last_clean":"2024-03-28T22:36:50.560497+0000","last_became_active":"2024-03-28T22:33:28.434954+0000","last_became_peered":"2024-03-28T22:33:28.434954+0000","last_unstale":"2024-03-28T22:36:50.560497+0000","last_undegraded":"2024-03-28T22:36:50.560497+0000","last_fullsized":"2024-03-28T22:36:50.560497+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:04.969596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":170,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"94'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248129+0000","last_change":"2024-03-28T22:33:28.921219+0000","last_active":"2024-03-28T22:35:30.248129+0000","last_peered":"2024-03-28T22:35:30.248129+0000","last_clean":"2024-03-28T22:35:30.248129+0000","last_became_active":"2024-03-28T22:33:28.920664+0000","last_became_peered":"2024-03-28T22:33:28.920664+0000","last_unstale":"2024-03-28T22:35:30.248129+0000","last_undegraded":"2024-03-28T22:35:30.248129+0000","last_fullsized":"2024-03-28T22:35:30.248129+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:48:55.032058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369383+0000","last_change":"2024-03-28T22:33:34.018101+0000","last_active":"2024-03-28T22:35:30.369383+0000","last_peered":"2024-03-28T22:35:30.369383+0000","last_clean":"2024-03-28T22:35:30.369383+0000","last_became_active":"2024-03-28T22:33:34.017260+0000","last_became_peered":"2024-03-28T22:33:34.017260+0000","last_unstale":"2024-03-28T22:35:30.369383+0000","last_undegraded":"2024-03-28T22:35:30.369383+0000","last_fullsized":"2024-03-28T22:35:30.369383+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:48:53.178090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"92'17","reported_seq":52,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.253595+0000","last_change":"2024-03-28T22:33:28.450232+0000","last_active":"2024-03-28T22:35:30.253595+0000","last_peered":"2024-03-28T22:35:30.253595+0000","last_clean":"2024-03-28T22:35:30.253595+0000","last_became_active":"2024-03-28T22:33:28.449958+0000","last_became_peered":"2024-03-28T22:33:28.449958+0000","last_unstale":"2024-03-28T22:35:30.253595+0000","last_undegraded":"2024-03-28T22:35:30.253595+0000","last_fullsized":"2024-03-28T22:35:30.253595+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:01:42.130260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"94'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275235+0000","last_change":"2024-03-28T22:33:28.926263+0000","last_active":"2024-03-28T22:35:30.275235+0000","last_peered":"2024-03-28T22:35:30.275235+0000","last_clean":"2024-03-28T22:35:30.275235+0000","last_became_active":"2024-03-28T22:33:28.925927+0000","last_became_peered":"2024-03-28T22:33:28.925927+0000","last_unstale":"2024-03-28T22:35:30.275235+0000","last_undegraded":"2024-03-28T22:35:30.275235+0000","last_fullsized":"2024-03-28T22:35:30.275235+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:41:42.477694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.455002+0000","last_change":"2024-03-28T22:35:30.455002+0000","last_active":"2024-03-28T22:35:30.455002+0000","last_peered":"2024-03-28T22:35:30.455002+0000","last_clean":"2024-03-28T22:35:30.455002+0000","last_became_active":"2024-03-28T22:35:30.454479+0000","last_became_peered":"2024-03-28T22:35:30.454479+0000","last_unstale":"2024-03-28T22:35:30.455002+0000","last_undegraded":"2024-03-28T22:35:30.455002+0000","last_fullsized":"2024-03-28T22:35:30.455002+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:57:06.300031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,3,1],"acting":[10,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.f","version":"92'2","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449832+0000","last_change":"2024-03-28T22:34:29.986650+0000","last_active":"2024-03-28T22:35:30.449832+0000","last_peered":"2024-03-28T22:35:30.449832+0000","last_clean":"2024-03-28T22:35:30.449832+0000","last_became_active":"2024-03-28T22:34:29.981875+0000","last_became_peered":"2024-03-28T22:34:29.981875+0000","last_unstale":"2024-03-28T22:35:30.449832+0000","last_undegraded":"2024-03-28T22:35:30.449832+0000","last_fullsized":"2024-03-28T22:35:30.449832+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:45:24.166133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"94'5","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449947+0000","last_change":"2024-03-28T22:34:29.982815+0000","last_active":"2024-03-28T22:35:30.449947+0000","last_peered":"2024-03-28T22:35:30.449947+0000","last_clean":"2024-03-28T22:35:30.449947+0000","last_became_active":"2024-03-28T22:34:29.982458+0000","last_became_peered":"2024-03-28T22:34:29.982458+0000","last_unstale":"2024-03-28T22:35:30.449947+0000","last_undegraded":"2024-03-28T22:35:30.449947+0000","last_fullsized":"2024-03-28T22:35:30.449947+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:11:30.617181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":30,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451505+0000","last_change":"2024-03-28T22:35:30.451063+0000","last_active":"2024-03-28T22:35:30.451505+0000","last_peered":"2024-03-28T22:35:30.451505+0000","last_clean":"2024-03-28T22:35:30.451505+0000","last_became_active":"2024-03-28T22:35:30.450525+0000","last_became_peered":"2024-03-28T22:35:30.450525+0000","last_unstale":"2024-03-28T22:35:30.451505+0000","last_undegraded":"2024-03-28T22:35:30.451505+0000","last_fullsized":"2024-03-28T22:35:30.451505+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:35:29.004834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"92'1","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252202+0000","last_change":"2024-03-28T22:34:29.992884+0000","last_active":"2024-03-28T22:35:30.252202+0000","last_peered":"2024-03-28T22:35:30.252202+0000","last_clean":"2024-03-28T22:35:30.252202+0000","last_became_active":"2024-03-28T22:34:29.991657+0000","last_became_peered":"2024-03-28T22:34:29.991657+0000","last_unstale":"2024-03-28T22:35:30.252202+0000","last_undegraded":"2024-03-28T22:35:30.252202+0000","last_fullsized":"2024-03-28T22:35:30.252202+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:25:14.540783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"94'4","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.259438+0000","last_change":"2024-03-28T22:33:28.942097+0000","last_active":"2024-03-28T22:35:30.259438+0000","last_peered":"2024-03-28T22:35:30.259438+0000","last_clean":"2024-03-28T22:35:30.259438+0000","last_became_active":"2024-03-28T22:33:28.941812+0000","last_became_peered":"2024-03-28T22:33:28.941812+0000","last_unstale":"2024-03-28T22:35:30.259438+0000","last_undegraded":"2024-03-28T22:35:30.259438+0000","last_fullsized":"2024-03-28T22:35:30.259438+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:36:10.204216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369177+0000","last_change":"2024-03-28T22:33:34.023171+0000","last_active":"2024-03-28T22:35:30.369177+0000","last_peered":"2024-03-28T22:35:30.369177+0000","last_clean":"2024-03-28T22:35:30.369177+0000","last_became_active":"2024-03-28T22:33:34.021313+0000","last_became_peered":"2024-03-28T22:33:34.021313+0000","last_unstale":"2024-03-28T22:35:30.369177+0000","last_undegraded":"2024-03-28T22:35:30.369177+0000","last_fullsized":"2024-03-28T22:35:30.369177+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:29:42.592761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"92'5","reported_seq":207,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:36:51.439644+0000","last_change":"2024-03-28T22:33:28.449133+0000","last_active":"2024-03-28T22:36:51.439644+0000","last_peered":"2024-03-28T22:36:51.439644+0000","last_clean":"2024-03-28T22:36:51.439644+0000","last_became_active":"2024-03-28T22:33:28.448743+0000","last_became_peered":"2024-03-28T22:33:28.448743+0000","last_unstale":"2024-03-28T22:36:51.439644+0000","last_undegraded":"2024-03-28T22:36:51.439644+0000","last_fullsized":"2024-03-28T22:36:51.439644+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:09:37.119736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":168,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.259332+0000","last_change":"2024-03-28T22:33:28.944198+0000","last_active":"2024-03-28T22:35:30.259332+0000","last_peered":"2024-03-28T22:35:30.259332+0000","last_clean":"2024-03-28T22:35:30.259332+0000","last_became_active":"2024-03-28T22:33:28.943818+0000","last_became_peered":"2024-03-28T22:33:28.943818+0000","last_unstale":"2024-03-28T22:35:30.259332+0000","last_undegraded":"2024-03-28T22:35:30.259332+0000","last_fullsized":"2024-03-28T22:35:30.259332+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:31:03.791364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369583+0000","last_change":"2024-03-28T22:33:34.015499+0000","last_active":"2024-03-28T22:35:30.369583+0000","last_peered":"2024-03-28T22:35:30.369583+0000","last_clean":"2024-03-28T22:35:30.369583+0000","last_became_active":"2024-03-28T22:33:34.015013+0000","last_became_peered":"2024-03-28T22:33:34.015013+0000","last_unstale":"2024-03-28T22:35:30.369583+0000","last_undegraded":"2024-03-28T22:35:30.369583+0000","last_fullsized":"2024-03-28T22:35:30.369583+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:34:30.582198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"94'8","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275296+0000","last_change":"2024-03-28T22:33:28.926732+0000","last_active":"2024-03-28T22:35:30.275296+0000","last_peered":"2024-03-28T22:35:30.275296+0000","last_clean":"2024-03-28T22:35:30.275296+0000","last_became_active":"2024-03-28T22:33:28.926366+0000","last_became_peered":"2024-03-28T22:33:28.926366+0000","last_unstale":"2024-03-28T22:35:30.275296+0000","last_undegraded":"2024-03-28T22:35:30.275296+0000","last_fullsized":"2024-03-28T22:35:30.275296+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:19:10.769141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'6","reported_seq":207,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:36:50.559858+0000","last_change":"2024-03-28T22:33:28.457400+0000","last_active":"2024-03-28T22:36:50.559858+0000","last_peered":"2024-03-28T22:36:50.559858+0000","last_clean":"2024-03-28T22:36:50.559858+0000","last_became_active":"2024-03-28T22:33:28.456994+0000","last_became_peered":"2024-03-28T22:33:28.456994+0000","last_unstale":"2024-03-28T22:36:50.559858+0000","last_undegraded":"2024-03-28T22:36:50.559858+0000","last_fullsized":"2024-03-28T22:36:50.559858+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:32:16.438540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":167,"num_read_kb":167,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"92'1","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248405+0000","last_change":"2024-03-28T22:33:34.027746+0000","last_active":"2024-03-28T22:35:30.248405+0000","last_peered":"2024-03-28T22:35:30.248405+0000","last_clean":"2024-03-28T22:35:30.248405+0000","last_became_active":"2024-03-28T22:33:34.027419+0000","last_became_peered":"2024-03-28T22:33:34.027419+0000","last_unstale":"2024-03-28T22:35:30.248405+0000","last_undegraded":"2024-03-28T22:35:30.248405+0000","last_fullsized":"2024-03-28T22:35:30.248405+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:55:43.686428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371544+0000","last_change":"2024-03-28T22:33:28.952393+0000","last_active":"2024-03-28T22:35:30.371544+0000","last_peered":"2024-03-28T22:35:30.371544+0000","last_clean":"2024-03-28T22:35:30.371544+0000","last_became_active":"2024-03-28T22:33:28.951951+0000","last_became_peered":"2024-03-28T22:33:28.951951+0000","last_unstale":"2024-03-28T22:35:30.371544+0000","last_undegraded":"2024-03-28T22:35:30.371544+0000","last_fullsized":"2024-03-28T22:35:30.371544+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:09:25.808842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"92'6","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370674+0000","last_change":"2024-03-28T22:33:28.437177+0000","last_active":"2024-03-28T22:35:30.370674+0000","last_peered":"2024-03-28T22:35:30.370674+0000","last_clean":"2024-03-28T22:35:30.370674+0000","last_became_active":"2024-03-28T22:33:28.436801+0000","last_became_peered":"2024-03-28T22:33:28.436801+0000","last_unstale":"2024-03-28T22:35:30.370674+0000","last_undegraded":"2024-03-28T22:35:30.370674+0000","last_fullsized":"2024-03-28T22:35:30.370674+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:24:52.549490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":372,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.247989+0000","last_change":"2024-03-28T22:32:07.867336+0000","last_active":"2024-03-28T22:35:30.247989+0000","last_peered":"2024-03-28T22:35:30.247989+0000","last_clean":"2024-03-28T22:35:30.247989+0000","last_became_active":"2024-03-28T22:32:07.866819+0000","last_became_peered":"2024-03-28T22:32:07.866819+0000","last_unstale":"2024-03-28T22:35:30.247989+0000","last_undegraded":"2024-03-28T22:35:30.247989+0000","last_fullsized":"2024-03-28T22:35:30.247989+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:28:14.713501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:28:14.713501+0000","last_clean_scrub_stamp":"2024-03-28T22:28:14.713501+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:00:52.438708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275597+0000","last_change":"2024-03-28T22:33:34.020668+0000","last_active":"2024-03-28T22:35:30.275597+0000","last_peered":"2024-03-28T22:35:30.275597+0000","last_clean":"2024-03-28T22:35:30.275597+0000","last_became_active":"2024-03-28T22:33:34.020240+0000","last_became_peered":"2024-03-28T22:33:34.020240+0000","last_unstale":"2024-03-28T22:35:30.275597+0000","last_undegraded":"2024-03-28T22:35:30.275597+0000","last_fullsized":"2024-03-28T22:35:30.275597+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:18.983983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"94'5","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368061+0000","last_change":"2024-03-28T22:33:28.935237+0000","last_active":"2024-03-28T22:35:30.368061+0000","last_peered":"2024-03-28T22:35:30.368061+0000","last_clean":"2024-03-28T22:35:30.368061+0000","last_became_active":"2024-03-28T22:33:28.934875+0000","last_became_peered":"2024-03-28T22:33:28.934875+0000","last_unstale":"2024-03-28T22:35:30.368061+0000","last_undegraded":"2024-03-28T22:35:30.368061+0000","last_fullsized":"2024-03-28T22:35:30.368061+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:20:50.789302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"92'4","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248187+0000","last_change":"2024-03-28T22:33:28.441654+0000","last_active":"2024-03-28T22:35:30.248187+0000","last_peered":"2024-03-28T22:35:30.248187+0000","last_clean":"2024-03-28T22:35:30.248187+0000","last_became_active":"2024-03-28T22:33:28.441293+0000","last_became_peered":"2024-03-28T22:33:28.441293+0000","last_unstale":"2024-03-28T22:35:30.248187+0000","last_undegraded":"2024-03-28T22:35:30.248187+0000","last_fullsized":"2024-03-28T22:35:30.248187+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:31:17.737977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449486+0000","last_change":"2024-03-28T22:33:34.018126+0000","last_active":"2024-03-28T22:35:30.449486+0000","last_peered":"2024-03-28T22:35:30.449486+0000","last_clean":"2024-03-28T22:35:30.449486+0000","last_became_active":"2024-03-28T22:33:34.017573+0000","last_became_peered":"2024-03-28T22:33:34.017573+0000","last_unstale":"2024-03-28T22:35:30.449486+0000","last_undegraded":"2024-03-28T22:35:30.449486+0000","last_fullsized":"2024-03-28T22:35:30.449486+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:23:31.013511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"92'4","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254108+0000","last_change":"2024-03-28T22:33:28.454069+0000","last_active":"2024-03-28T22:35:30.254108+0000","last_peered":"2024-03-28T22:35:30.254108+0000","last_clean":"2024-03-28T22:35:30.254108+0000","last_became_active":"2024-03-28T22:33:28.453754+0000","last_became_peered":"2024-03-28T22:33:28.453754+0000","last_unstale":"2024-03-28T22:35:30.254108+0000","last_undegraded":"2024-03-28T22:35:30.254108+0000","last_fullsized":"2024-03-28T22:35:30.254108+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:23:33.981072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"94'9","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369387+0000","last_change":"2024-03-28T22:33:28.934390+0000","last_active":"2024-03-28T22:35:30.369387+0000","last_peered":"2024-03-28T22:35:30.369387+0000","last_clean":"2024-03-28T22:35:30.369387+0000","last_became_active":"2024-03-28T22:33:28.933919+0000","last_became_peered":"2024-03-28T22:33:28.933919+0000","last_unstale":"2024-03-28T22:35:30.369387+0000","last_undegraded":"2024-03-28T22:35:30.369387+0000","last_fullsized":"2024-03-28T22:35:30.369387+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:27:07.595144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371312+0000","last_change":"2024-03-28T22:33:34.019910+0000","last_active":"2024-03-28T22:35:30.371312+0000","last_peered":"2024-03-28T22:35:30.371312+0000","last_clean":"2024-03-28T22:35:30.371312+0000","last_became_active":"2024-03-28T22:33:34.019522+0000","last_became_peered":"2024-03-28T22:33:34.019522+0000","last_unstale":"2024-03-28T22:35:30.371312+0000","last_undegraded":"2024-03-28T22:35:30.371312+0000","last_fullsized":"2024-03-28T22:35:30.371312+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:01:29.298513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"96'16","reported_seq":50,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:36:48.496850+0000","last_change":"2024-03-28T22:33:28.444513+0000","last_active":"2024-03-28T22:36:48.496850+0000","last_peered":"2024-03-28T22:36:48.496850+0000","last_clean":"2024-03-28T22:36:48.496850+0000","last_became_active":"2024-03-28T22:33:28.444091+0000","last_became_peered":"2024-03-28T22:33:28.444091+0000","last_unstale":"2024-03-28T22:36:48.496850+0000","last_undegraded":"2024-03-28T22:36:48.496850+0000","last_fullsized":"2024-03-28T22:36:48.496850+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:49:58.331864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"94'5","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.372223+0000","last_change":"2024-03-28T22:33:28.948390+0000","last_active":"2024-03-28T22:35:30.372223+0000","last_peered":"2024-03-28T22:35:30.372223+0000","last_clean":"2024-03-28T22:35:30.372223+0000","last_became_active":"2024-03-28T22:33:28.942846+0000","last_became_peered":"2024-03-28T22:33:28.942846+0000","last_unstale":"2024-03-28T22:35:30.372223+0000","last_undegraded":"2024-03-28T22:35:30.372223+0000","last_fullsized":"2024-03-28T22:35:30.372223+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:50.596365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251712+0000","last_change":"2024-03-28T22:33:33.999429+0000","last_active":"2024-03-28T22:35:30.251712+0000","last_peered":"2024-03-28T22:35:30.251712+0000","last_clean":"2024-03-28T22:35:30.251712+0000","last_became_active":"2024-03-28T22:33:33.998979+0000","last_became_peered":"2024-03-28T22:33:33.998979+0000","last_unstale":"2024-03-28T22:35:30.251712+0000","last_undegraded":"2024-03-28T22:35:30.251712+0000","last_fullsized":"2024-03-28T22:35:30.251712+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:26.450731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"92'7","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370791+0000","last_change":"2024-03-28T22:33:28.442861+0000","last_active":"2024-03-28T22:35:30.370791+0000","last_peered":"2024-03-28T22:35:30.370791+0000","last_clean":"2024-03-28T22:35:30.370791+0000","last_became_active":"2024-03-28T22:33:28.442468+0000","last_became_peered":"2024-03-28T22:33:28.442468+0000","last_unstale":"2024-03-28T22:35:30.370791+0000","last_undegraded":"2024-03-28T22:35:30.370791+0000","last_fullsized":"2024-03-28T22:35:30.370791+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:45:49.899031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"94'7","reported_seq":45,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451985+0000","last_change":"2024-03-28T22:34:30.429872+0000","last_active":"2024-03-28T22:35:30.451985+0000","last_peered":"2024-03-28T22:35:30.451985+0000","last_clean":"2024-03-28T22:35:30.451985+0000","last_became_active":"2024-03-28T22:34:30.429237+0000","last_became_peered":"2024-03-28T22:34:30.429237+0000","last_unstale":"2024-03-28T22:35:30.451985+0000","last_undegraded":"2024-03-28T22:35:30.451985+0000","last_fullsized":"2024-03-28T22:35:30.451985+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:14:46.608826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370565+0000","last_change":"2024-03-28T22:33:34.013826+0000","last_active":"2024-03-28T22:35:30.370565+0000","last_peered":"2024-03-28T22:35:30.370565+0000","last_clean":"2024-03-28T22:35:30.370565+0000","last_became_active":"2024-03-28T22:33:34.013415+0000","last_became_peered":"2024-03-28T22:33:34.013415+0000","last_unstale":"2024-03-28T22:35:30.370565+0000","last_undegraded":"2024-03-28T22:35:30.370565+0000","last_fullsized":"2024-03-28T22:35:30.370565+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:19:30.965940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"92'2","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448399+0000","last_change":"2024-03-28T22:33:28.442225+0000","last_active":"2024-03-28T22:35:30.448399+0000","last_peered":"2024-03-28T22:35:30.448399+0000","last_clean":"2024-03-28T22:35:30.448399+0000","last_became_active":"2024-03-28T22:33:28.441844+0000","last_became_peered":"2024-03-28T22:33:28.441844+0000","last_unstale":"2024-03-28T22:35:30.448399+0000","last_undegraded":"2024-03-28T22:35:30.448399+0000","last_fullsized":"2024-03-28T22:35:30.448399+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:40:52.108663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"94'2","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.253675+0000","last_change":"2024-03-28T22:33:28.927571+0000","last_active":"2024-03-28T22:35:30.253675+0000","last_peered":"2024-03-28T22:35:30.253675+0000","last_clean":"2024-03-28T22:35:30.253675+0000","last_became_active":"2024-03-28T22:33:28.927275+0000","last_became_peered":"2024-03-28T22:33:28.927275+0000","last_unstale":"2024-03-28T22:35:30.253675+0000","last_undegraded":"2024-03-28T22:35:30.253675+0000","last_fullsized":"2024-03-28T22:35:30.253675+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:25:16.184533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.253011+0000","last_change":"2024-03-28T22:33:34.018429+0000","last_active":"2024-03-28T22:35:30.253011+0000","last_peered":"2024-03-28T22:35:30.253011+0000","last_clean":"2024-03-28T22:35:30.253011+0000","last_became_active":"2024-03-28T22:33:34.017066+0000","last_became_peered":"2024-03-28T22:33:34.017066+0000","last_unstale":"2024-03-28T22:35:30.253011+0000","last_undegraded":"2024-03-28T22:35:30.253011+0000","last_fullsized":"2024-03-28T22:35:30.253011+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:35:42.101072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"92'7","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371970+0000","last_change":"2024-03-28T22:33:28.437760+0000","last_active":"2024-03-28T22:35:30.371970+0000","last_peered":"2024-03-28T22:35:30.371970+0000","last_clean":"2024-03-28T22:35:30.371970+0000","last_became_active":"2024-03-28T22:33:28.437096+0000","last_became_peered":"2024-03-28T22:33:28.437096+0000","last_unstale":"2024-03-28T22:35:30.371970+0000","last_undegraded":"2024-03-28T22:35:30.371970+0000","last_fullsized":"2024-03-28T22:35:30.371970+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:29:20.778551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"94'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449358+0000","last_change":"2024-03-28T22:33:28.935805+0000","last_active":"2024-03-28T22:35:30.449358+0000","last_peered":"2024-03-28T22:35:30.449358+0000","last_clean":"2024-03-28T22:35:30.449358+0000","last_became_active":"2024-03-28T22:33:28.935520+0000","last_became_peered":"2024-03-28T22:33:28.935520+0000","last_unstale":"2024-03-28T22:35:30.449358+0000","last_undegraded":"2024-03-28T22:35:30.449358+0000","last_fullsized":"2024-03-28T22:35:30.449358+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:23.414310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"92'1","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370585+0000","last_change":"2024-03-28T22:33:34.029518+0000","last_active":"2024-03-28T22:35:30.370585+0000","last_peered":"2024-03-28T22:35:30.370585+0000","last_clean":"2024-03-28T22:35:30.370585+0000","last_became_active":"2024-03-28T22:33:34.029101+0000","last_became_peered":"2024-03-28T22:33:34.029101+0000","last_unstale":"2024-03-28T22:35:30.370585+0000","last_undegraded":"2024-03-28T22:35:30.370585+0000","last_fullsized":"2024-03-28T22:35:30.370585+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:05.595543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"96'4","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:38.679235+0000","last_change":"2024-03-28T22:33:28.438857+0000","last_active":"2024-03-28T22:35:38.679235+0000","last_peered":"2024-03-28T22:35:38.679235+0000","last_clean":"2024-03-28T22:35:38.679235+0000","last_became_active":"2024-03-28T22:33:28.438528+0000","last_became_peered":"2024-03-28T22:33:28.438528+0000","last_unstale":"2024-03-28T22:35:38.679235+0000","last_undegraded":"2024-03-28T22:35:38.679235+0000","last_fullsized":"2024-03-28T22:35:38.679235+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:07:54.506739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371660+0000","last_change":"2024-03-28T22:33:28.958271+0000","last_active":"2024-03-28T22:35:30.371660+0000","last_peered":"2024-03-28T22:35:30.371660+0000","last_clean":"2024-03-28T22:35:30.371660+0000","last_became_active":"2024-03-28T22:33:28.957922+0000","last_became_peered":"2024-03-28T22:33:28.957922+0000","last_unstale":"2024-03-28T22:35:30.371660+0000","last_undegraded":"2024-03-28T22:35:30.371660+0000","last_fullsized":"2024-03-28T22:35:30.371660+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:33:19.003131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370592+0000","last_change":"2024-03-28T22:33:34.000789+0000","last_active":"2024-03-28T22:35:30.370592+0000","last_peered":"2024-03-28T22:35:30.370592+0000","last_clean":"2024-03-28T22:35:30.370592+0000","last_became_active":"2024-03-28T22:33:34.000383+0000","last_became_peered":"2024-03-28T22:33:34.000383+0000","last_unstale":"2024-03-28T22:35:30.370592+0000","last_undegraded":"2024-03-28T22:35:30.370592+0000","last_fullsized":"2024-03-28T22:35:30.370592+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:50:29.925656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"92'4","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.253813+0000","last_change":"2024-03-28T22:33:28.439121+0000","last_active":"2024-03-28T22:35:30.253813+0000","last_peered":"2024-03-28T22:35:30.253813+0000","last_clean":"2024-03-28T22:35:30.253813+0000","last_became_active":"2024-03-28T22:33:28.438800+0000","last_became_peered":"2024-03-28T22:33:28.438800+0000","last_unstale":"2024-03-28T22:35:30.253813+0000","last_undegraded":"2024-03-28T22:35:30.253813+0000","last_fullsized":"2024-03-28T22:35:30.253813+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:57:10.013246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"94'5","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248157+0000","last_change":"2024-03-28T22:33:28.934499+0000","last_active":"2024-03-28T22:35:30.248157+0000","last_peered":"2024-03-28T22:35:30.248157+0000","last_clean":"2024-03-28T22:35:30.248157+0000","last_became_active":"2024-03-28T22:33:28.934147+0000","last_became_peered":"2024-03-28T22:33:28.934147+0000","last_unstale":"2024-03-28T22:35:30.248157+0000","last_undegraded":"2024-03-28T22:35:30.248157+0000","last_fullsized":"2024-03-28T22:35:30.248157+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:35:39.090052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448402+0000","last_change":"2024-03-28T22:33:34.017946+0000","last_active":"2024-03-28T22:35:30.448402+0000","last_peered":"2024-03-28T22:35:30.448402+0000","last_clean":"2024-03-28T22:35:30.448402+0000","last_became_active":"2024-03-28T22:33:34.017546+0000","last_became_peered":"2024-03-28T22:33:34.017546+0000","last_unstale":"2024-03-28T22:35:30.448402+0000","last_undegraded":"2024-03-28T22:35:30.448402+0000","last_fullsized":"2024-03-28T22:35:30.448402+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:14:39.081677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"96'6","reported_seq":211,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:36:52.356927+0000","last_change":"2024-03-28T22:33:28.453387+0000","last_active":"2024-03-28T22:36:52.356927+0000","last_peered":"2024-03-28T22:36:52.356927+0000","last_clean":"2024-03-28T22:36:52.356927+0000","last_became_active":"2024-03-28T22:33:28.453029+0000","last_became_peered":"2024-03-28T22:33:28.453029+0000","last_unstale":"2024-03-28T22:36:52.356927+0000","last_undegraded":"2024-03-28T22:36:52.356927+0000","last_fullsized":"2024-03-28T22:36:52.356927+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:12:33.246863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":171,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248615+0000","last_change":"2024-03-28T22:33:28.923819+0000","last_active":"2024-03-28T22:35:30.248615+0000","last_peered":"2024-03-28T22:35:30.248615+0000","last_clean":"2024-03-28T22:35:30.248615+0000","last_became_active":"2024-03-28T22:33:28.923471+0000","last_became_peered":"2024-03-28T22:33:28.923471+0000","last_unstale":"2024-03-28T22:35:30.248615+0000","last_undegraded":"2024-03-28T22:35:30.248615+0000","last_fullsized":"2024-03-28T22:35:30.248615+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:58:03.496406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451029+0000","last_change":"2024-03-28T22:33:34.029789+0000","last_active":"2024-03-28T22:35:30.451029+0000","last_peered":"2024-03-28T22:35:30.451029+0000","last_clean":"2024-03-28T22:35:30.451029+0000","last_became_active":"2024-03-28T22:33:34.029231+0000","last_became_peered":"2024-03-28T22:33:34.029231+0000","last_unstale":"2024-03-28T22:35:30.451029+0000","last_undegraded":"2024-03-28T22:35:30.451029+0000","last_fullsized":"2024-03-28T22:35:30.451029+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:47:50.101393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"92'3","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275541+0000","last_change":"2024-03-28T22:33:28.435668+0000","last_active":"2024-03-28T22:35:30.275541+0000","last_peered":"2024-03-28T22:35:30.275541+0000","last_clean":"2024-03-28T22:35:30.275541+0000","last_became_active":"2024-03-28T22:33:28.435325+0000","last_became_peered":"2024-03-28T22:33:28.435325+0000","last_unstale":"2024-03-28T22:35:30.275541+0000","last_undegraded":"2024-03-28T22:35:30.275541+0000","last_fullsized":"2024-03-28T22:35:30.275541+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:13:06.281074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"94'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248244+0000","last_change":"2024-03-28T22:33:28.954138+0000","last_active":"2024-03-28T22:35:30.248244+0000","last_peered":"2024-03-28T22:35:30.248244+0000","last_clean":"2024-03-28T22:35:30.248244+0000","last_became_active":"2024-03-28T22:33:28.953752+0000","last_became_peered":"2024-03-28T22:33:28.953752+0000","last_unstale":"2024-03-28T22:35:30.248244+0000","last_undegraded":"2024-03-28T22:35:30.248244+0000","last_fullsized":"2024-03-28T22:35:30.248244+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:53:34.892316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.274983+0000","last_change":"2024-03-28T22:33:34.002672+0000","last_active":"2024-03-28T22:35:30.274983+0000","last_peered":"2024-03-28T22:35:30.274983+0000","last_clean":"2024-03-28T22:35:30.274983+0000","last_became_active":"2024-03-28T22:33:34.002195+0000","last_became_peered":"2024-03-28T22:33:34.002195+0000","last_unstale":"2024-03-28T22:35:30.274983+0000","last_undegraded":"2024-03-28T22:35:30.274983+0000","last_fullsized":"2024-03-28T22:35:30.274983+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:24:11.521474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"92'5","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369212+0000","last_change":"2024-03-28T22:33:28.450818+0000","last_active":"2024-03-28T22:35:30.369212+0000","last_peered":"2024-03-28T22:35:30.369212+0000","last_clean":"2024-03-28T22:35:30.369212+0000","last_became_active":"2024-03-28T22:33:28.448592+0000","last_became_peered":"2024-03-28T22:33:28.448592+0000","last_unstale":"2024-03-28T22:35:30.369212+0000","last_undegraded":"2024-03-28T22:35:30.369212+0000","last_fullsized":"2024-03-28T22:35:30.369212+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:19:27.126306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"94'9","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248488+0000","last_change":"2024-03-28T22:33:28.925191+0000","last_active":"2024-03-28T22:35:30.248488+0000","last_peered":"2024-03-28T22:35:30.248488+0000","last_clean":"2024-03-28T22:35:30.248488+0000","last_became_active":"2024-03-28T22:33:28.924424+0000","last_became_peered":"2024-03-28T22:33:28.924424+0000","last_unstale":"2024-03-28T22:35:30.248488+0000","last_undegraded":"2024-03-28T22:35:30.248488+0000","last_fullsized":"2024-03-28T22:35:30.248488+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:50:00.415047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"94'9","reported_seq":46,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449424+0000","last_change":"2024-03-28T22:34:29.987176+0000","last_active":"2024-03-28T22:35:30.449424+0000","last_peered":"2024-03-28T22:35:30.449424+0000","last_clean":"2024-03-28T22:35:30.449424+0000","last_became_active":"2024-03-28T22:34:29.986807+0000","last_became_peered":"2024-03-28T22:34:29.986807+0000","last_unstale":"2024-03-28T22:35:30.449424+0000","last_undegraded":"2024-03-28T22:35:30.449424+0000","last_fullsized":"2024-03-28T22:35:30.449424+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:00:06.774521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"94'12","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251938+0000","last_change":"2024-03-28T22:33:28.927068+0000","last_active":"2024-03-28T22:35:30.251938+0000","last_peered":"2024-03-28T22:35:30.251938+0000","last_clean":"2024-03-28T22:35:30.251938+0000","last_became_active":"2024-03-28T22:33:28.926734+0000","last_became_peered":"2024-03-28T22:33:28.926734+0000","last_unstale":"2024-03-28T22:35:30.251938+0000","last_undegraded":"2024-03-28T22:35:30.251938+0000","last_fullsized":"2024-03-28T22:35:30.251938+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:56.309555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"94'8","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251930+0000","last_change":"2024-03-28T22:33:28.955675+0000","last_active":"2024-03-28T22:35:30.251930+0000","last_peered":"2024-03-28T22:35:30.251930+0000","last_clean":"2024-03-28T22:35:30.251930+0000","last_became_active":"2024-03-28T22:33:28.955329+0000","last_became_peered":"2024-03-28T22:33:28.955329+0000","last_unstale":"2024-03-28T22:35:30.251930+0000","last_undegraded":"2024-03-28T22:35:30.251930+0000","last_fullsized":"2024-03-28T22:35:30.251930+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:34:14.166434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.452477+0000","last_change":"2024-03-28T22:33:28.955125+0000","last_active":"2024-03-28T22:35:30.452477+0000","last_peered":"2024-03-28T22:35:30.452477+0000","last_clean":"2024-03-28T22:35:30.452477+0000","last_became_active":"2024-03-28T22:33:28.954434+0000","last_became_peered":"2024-03-28T22:33:28.954434+0000","last_unstale":"2024-03-28T22:35:30.452477+0000","last_undegraded":"2024-03-28T22:35:30.452477+0000","last_fullsized":"2024-03-28T22:35:30.452477+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:56:50.705012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"94'4","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369433+0000","last_change":"2024-03-28T22:33:28.945230+0000","last_active":"2024-03-28T22:35:30.369433+0000","last_peered":"2024-03-28T22:35:30.369433+0000","last_clean":"2024-03-28T22:35:30.369433+0000","last_became_active":"2024-03-28T22:33:28.944921+0000","last_became_peered":"2024-03-28T22:33:28.944921+0000","last_unstale":"2024-03-28T22:35:30.369433+0000","last_undegraded":"2024-03-28T22:35:30.369433+0000","last_fullsized":"2024-03-28T22:35:30.369433+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:26:26.505774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"94'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449741+0000","last_change":"2024-03-28T22:33:28.928720+0000","last_active":"2024-03-28T22:35:30.449741+0000","last_peered":"2024-03-28T22:35:30.449741+0000","last_clean":"2024-03-28T22:35:30.449741+0000","last_became_active":"2024-03-28T22:33:28.928415+0000","last_became_peered":"2024-03-28T22:33:28.928415+0000","last_unstale":"2024-03-28T22:35:30.449741+0000","last_undegraded":"2024-03-28T22:35:30.449741+0000","last_fullsized":"2024-03-28T22:35:30.449741+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:00.839609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"94'15","reported_seq":47,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252418+0000","last_change":"2024-03-28T22:33:28.924948+0000","last_active":"2024-03-28T22:35:30.252418+0000","last_peered":"2024-03-28T22:35:30.252418+0000","last_clean":"2024-03-28T22:35:30.252418+0000","last_became_active":"2024-03-28T22:33:28.924631+0000","last_became_peered":"2024-03-28T22:33:28.924631+0000","last_unstale":"2024-03-28T22:35:30.252418+0000","last_undegraded":"2024-03-28T22:35:30.252418+0000","last_fullsized":"2024-03-28T22:35:30.252418+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:53:44.909991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"94'2","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.372408+0000","last_change":"2024-03-28T22:33:28.948725+0000","last_active":"2024-03-28T22:35:30.372408+0000","last_peered":"2024-03-28T22:35:30.372408+0000","last_clean":"2024-03-28T22:35:30.372408+0000","last_became_active":"2024-03-28T22:33:28.943942+0000","last_became_peered":"2024-03-28T22:33:28.943942+0000","last_unstale":"2024-03-28T22:35:30.372408+0000","last_undegraded":"2024-03-28T22:35:30.372408+0000","last_fullsized":"2024-03-28T22:35:30.372408+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:50:04.238538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"94'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275158+0000","last_change":"2024-03-28T22:33:28.941546+0000","last_active":"2024-03-28T22:35:30.275158+0000","last_peered":"2024-03-28T22:35:30.275158+0000","last_clean":"2024-03-28T22:35:30.275158+0000","last_became_active":"2024-03-28T22:33:28.941278+0000","last_became_peered":"2024-03-28T22:33:28.941278+0000","last_unstale":"2024-03-28T22:35:30.275158+0000","last_undegraded":"2024-03-28T22:35:30.275158+0000","last_fullsized":"2024-03-28T22:35:30.275158+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:47:10.047067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.372264+0000","last_change":"2024-03-28T22:33:28.949552+0000","last_active":"2024-03-28T22:35:30.372264+0000","last_peered":"2024-03-28T22:35:30.372264+0000","last_clean":"2024-03-28T22:35:30.372264+0000","last_became_active":"2024-03-28T22:33:28.947432+0000","last_became_peered":"2024-03-28T22:33:28.947432+0000","last_unstale":"2024-03-28T22:35:30.372264+0000","last_undegraded":"2024-03-28T22:35:30.372264+0000","last_fullsized":"2024-03-28T22:35:30.372264+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:01:16.534642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"94'10","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448510+0000","last_change":"2024-03-28T22:33:28.953930+0000","last_active":"2024-03-28T22:35:30.448510+0000","last_peered":"2024-03-28T22:35:30.448510+0000","last_clean":"2024-03-28T22:35:30.448510+0000","last_became_active":"2024-03-28T22:33:28.953591+0000","last_became_peered":"2024-03-28T22:33:28.953591+0000","last_unstale":"2024-03-28T22:35:30.448510+0000","last_undegraded":"2024-03-28T22:35:30.448510+0000","last_fullsized":"2024-03-28T22:35:30.448510+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:42:55.857476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"94'11","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368801+0000","last_change":"2024-03-28T22:33:28.946307+0000","last_active":"2024-03-28T22:35:30.368801+0000","last_peered":"2024-03-28T22:35:30.368801+0000","last_clean":"2024-03-28T22:35:30.368801+0000","last_became_active":"2024-03-28T22:33:28.945932+0000","last_became_peered":"2024-03-28T22:33:28.945932+0000","last_unstale":"2024-03-28T22:35:30.368801+0000","last_undegraded":"2024-03-28T22:35:30.368801+0000","last_fullsized":"2024-03-28T22:35:30.368801+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:54:59.651668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449676+0000","last_change":"2024-03-28T22:33:28.928014+0000","last_active":"2024-03-28T22:35:30.449676+0000","last_peered":"2024-03-28T22:35:30.449676+0000","last_clean":"2024-03-28T22:35:30.449676+0000","last_became_active":"2024-03-28T22:33:28.927686+0000","last_became_peered":"2024-03-28T22:33:28.927686+0000","last_unstale":"2024-03-28T22:35:30.449676+0000","last_undegraded":"2024-03-28T22:35:30.449676+0000","last_fullsized":"2024-03-28T22:35:30.449676+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:47:08.131565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":72803,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1695,"num_read_kb":924,"num_write":262,"num_write_kb":155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":585728,"data_stored":226693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":226,"ondisk_log_size":226,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318327,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32260,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712892,"statfs":{"total":95995035648,"available":95962001408,"internally_reserved":0,"allocated":1105920,"data_stored":539947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579963,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28772,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716380,"statfs":{"total":95995035648,"available":95965573120,"internally_reserved":0,"allocated":1798144,"data_stored":1191643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743490,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1130496,"data_stored":543560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005128,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1118208,"data_stored":539820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299470,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37016,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708136,"statfs":{"total":95995035648,"available":95957131264,"internally_reserved":0,"allocated":1720320,"data_stored":1137177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495699,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1101824,"data_stored":534968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574958,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36480,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708672,"statfs":{"total":95995035648,"available":95957680128,"internally_reserved":0,"allocated":1171456,"data_stored":600759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738484,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1126400,"data_stored":536815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411436,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36508,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708644,"statfs":{"total":95995035648,"available":95957651456,"internally_reserved":0,"allocated":1200128,"data_stored":610351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182499,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36424,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708728,"statfs":{"total":95995035648,"available":95957737472,"internally_reserved":0,"allocated":1114112,"data_stored":534676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920862,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1110016,"data_stored":539574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757337,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28704,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716448,"statfs":{"total":95995035648,"available":95965642752,"internally_reserved":0,"allocated":1728512,"data_stored":1132227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":66513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":76105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":67029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T22:36:56.166 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T22:36:56.228 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:55 smithi043 ceph-mon[31823]: pgmap v429: 129 pgs: 129 active+clean; 648 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-28T22:36:56.228 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:55 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1671036691' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:36:56.228 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:55 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3446644194' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:36:56.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:55 smithi172 ceph-mon[36566]: pgmap v429: 129 pgs: 129 active+clean; 648 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-28T22:36:56.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:55 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1671036691' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:36:56.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:55 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3446644194' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:36:56.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.435+0000 7f3bf3200640 1 -- 172.21.15.43:0/2717948452 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3bf4108da0 msgr2=0x7f3bf4109220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:56.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.435+0000 7f3bf3200640 1 --2- 172.21.15.43:0/2717948452 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3bf4108da0 0x7f3bf4109220 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc009900 tx=0x7f3bdc02bc30 comp rx=0 tx=0).stop 2024-03-28T22:36:56.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.435+0000 7f3bf3200640 1 -- 172.21.15.43:0/2717948452 shutdown_connections 2024-03-28T22:36:56.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.435+0000 7f3bf3200640 1 --2- 172.21.15.43:0/2717948452 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3bf4108da0 0x7f3bf4109220 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:56.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.435+0000 7f3bf3200640 1 --2- 172.21.15.43:0/2717948452 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3bf4109760 0x7f3bf410fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:56.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.435+0000 7f3bf3200640 1 --2- 172.21.15.43:0/2717948452 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3bf4107bb0 0x7f3bf4107fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:56.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.435+0000 7f3bf3200640 1 -- 172.21.15.43:0/2717948452 >> 172.21.15.43:0/2717948452 conn(0x7f3bf4076b60 msgr2=0x7f3bf4078f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:56.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.435+0000 7f3bf3200640 1 -- 172.21.15.43:0/2717948452 shutdown_connections 2024-03-28T22:36:56.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.435+0000 7f3bf3200640 1 -- 172.21.15.43:0/2717948452 wait complete. 2024-03-28T22:36:56.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.437+0000 7f3bf3200640 1 Processor -- start 2024-03-28T22:36:56.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.437+0000 7f3bf3200640 1 -- start start 2024-03-28T22:36:56.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.437+0000 7f3bf3200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3bf4107bb0 0x7f3bf419de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:56.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.437+0000 7f3bf3200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3bf4108da0 0x7f3bf419e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:56.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.437+0000 7f3bf3200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3bf4109760 0x7f3bf41a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:56.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.437+0000 7f3bf3200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf4113810 con 0x7f3bf4108da0 2024-03-28T22:36:56.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.437+0000 7f3bf3200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf4113690 con 0x7f3bf4109760 2024-03-28T22:36:56.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.437+0000 7f3bf3200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf4113990 con 0x7f3bf4107bb0 2024-03-28T22:36:56.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.438+0000 7f3bf1e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3bf4107bb0 0x7f3bf419de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:56.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.438+0000 7f3bf1e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3bf4107bb0 0x7f3bf419de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:46730/0 (socket says 172.21.15.43:46730) 2024-03-28T22:36:56.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.438+0000 7f3bf1e00640 1 -- 172.21.15.43:0/2586595298 learned_addr learned my addr 172.21.15.43:0/2586595298 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:56.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.438+0000 7f3bf2800640 1 --2- 172.21.15.43:0/2586595298 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3bf4109760 0x7f3bf41a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:56.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.438+0000 7f3bebe00640 1 --2- 172.21.15.43:0/2586595298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3bf4108da0 0x7f3bf419e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:56.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.438+0000 7f3bf1e00640 1 -- 172.21.15.43:0/2586595298 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3bf4109760 msgr2=0x7f3bf41a5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:56.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.438+0000 7f3bf1e00640 1 --2- 172.21.15.43:0/2586595298 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3bf4109760 0x7f3bf41a5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:56.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.438+0000 7f3bf1e00640 1 -- 172.21.15.43:0/2586595298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3bf4108da0 msgr2=0x7f3bf419e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:56.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.438+0000 7f3bf1e00640 1 --2- 172.21.15.43:0/2586595298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3bf4108da0 0x7f3bf419e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:56.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.438+0000 7f3bf1e00640 1 -- 172.21.15.43:0/2586595298 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bf41a5a70 con 0x7f3bf4107bb0 2024-03-28T22:36:56.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.438+0000 7f3bebe00640 1 --2- 172.21.15.43:0/2586595298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3bf4108da0 0x7f3bf419e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:36:56.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.438+0000 7f3bf1e00640 1 --2- 172.21.15.43:0/2586595298 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3bf4107bb0 0x7f3bf419de50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3be400e9c0 tx=0x7f3be400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:56.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.439+0000 7f3beaa00640 1 -- 172.21.15.43:0/2586595298 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be400cd80 con 0x7f3bf4107bb0 2024-03-28T22:36:56.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.439+0000 7f3bf3200640 1 -- 172.21.15.43:0/2586595298 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bf41a5d60 con 0x7f3bf4107bb0 2024-03-28T22:36:56.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.439+0000 7f3beaa00640 1 -- 172.21.15.43:0/2586595298 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3be40035f0 con 0x7f3bf4107bb0 2024-03-28T22:36:56.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.439+0000 7f3beaa00640 1 -- 172.21.15.43:0/2586595298 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be4010690 con 0x7f3bf4107bb0 2024-03-28T22:36:56.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.439+0000 7f3bf3200640 1 -- 172.21.15.43:0/2586595298 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bf41a6210 con 0x7f3bf4107bb0 2024-03-28T22:36:56.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.440+0000 7f3bf3200640 1 -- 172.21.15.43:0/2586595298 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bf410eed0 con 0x7f3bf4107bb0 2024-03-28T22:36:56.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.442+0000 7f3beaa00640 1 -- 172.21.15.43:0/2586595298 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7f3be40037a0 con 0x7f3bf4107bb0 2024-03-28T22:36:56.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.444+0000 7f3beaa00640 1 --2- 172.21.15.43:0/2586595298 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3bd80a1e90 0x7f3bd80a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:56.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.444+0000 7f3beaa00640 1 -- 172.21.15.43:0/2586595298 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f3be40ce700 con 0x7f3bf4107bb0 2024-03-28T22:36:56.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.444+0000 7f3bebe00640 1 --2- 172.21.15.43:0/2586595298 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3bd80a1e90 0x7f3bd80a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:56.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.445+0000 7f3bebe00640 1 --2- 172.21.15.43:0/2586595298 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3bd80a1e90 0x7f3bd80a4350 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc006010 tx=0x7f3bdc02c8a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:56.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:55 smithi082 ceph-mon[36471]: pgmap v429: 129 pgs: 129 active+clean; 648 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-28T22:36:56.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:55 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1671036691' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:36:56.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:55 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3446644194' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:36:56.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.450+0000 7f3beaa00640 1 -- 172.21.15.43:0/2586595298 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3be409b5e0 con 0x7f3bf4107bb0 2024-03-28T22:36:56.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.652+0000 7f3bf3200640 1 -- 172.21.15.43:0/2586595298 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3bf4061b90 con 0x7f3bd80a1e90 2024-03-28T22:36:56.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.665+0000 7f3beaa00640 1 -- 172.21.15.43:0/2586595298 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340530 (secure 0 0 0) 0x7f3bf4061b90 con 0x7f3bd80a1e90 2024-03-28T22:36:56.669 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:36:56.671 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-03-28T22:36:56.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.672+0000 7f3bf3200640 1 -- 172.21.15.43:0/2586595298 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3bd80a1e90 msgr2=0x7f3bd80a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:56.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.672+0000 7f3bf3200640 1 --2- 172.21.15.43:0/2586595298 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3bd80a1e90 0x7f3bd80a4350 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc006010 tx=0x7f3bdc02c8a0 comp rx=0 tx=0).stop 2024-03-28T22:36:56.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.672+0000 7f3bf3200640 1 -- 172.21.15.43:0/2586595298 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3bf4107bb0 msgr2=0x7f3bf419de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:56.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.672+0000 7f3bf3200640 1 --2- 172.21.15.43:0/2586595298 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3bf4107bb0 0x7f3bf419de50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3be400e9c0 tx=0x7f3be400ee80 comp rx=0 tx=0).stop 2024-03-28T22:36:56.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.672+0000 7f3bf3200640 1 -- 172.21.15.43:0/2586595298 shutdown_connections 2024-03-28T22:36:56.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.672+0000 7f3bf3200640 1 --2- 172.21.15.43:0/2586595298 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3bd80a1e90 0x7f3bd80a4350 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:56.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.672+0000 7f3bf3200640 1 --2- 172.21.15.43:0/2586595298 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3bf4109760 0x7f3bf41a5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:56.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.672+0000 7f3bf3200640 1 --2- 172.21.15.43:0/2586595298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3bf4108da0 0x7f3bf419e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:56.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.672+0000 7f3bf3200640 1 --2- 172.21.15.43:0/2586595298 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3bf4107bb0 0x7f3bf419de50 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:56.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.672+0000 7f3bf3200640 1 -- 172.21.15.43:0/2586595298 >> 172.21.15.43:0/2586595298 conn(0x7f3bf4076b60 msgr2=0x7f3bf410c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:56.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.673+0000 7f3bf3200640 1 -- 172.21.15.43:0/2586595298 shutdown_connections 2024-03-28T22:36:56.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.673+0000 7f3bf3200640 1 -- 172.21.15.43:0/2586595298 wait complete. 2024-03-28T22:36:56.693 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":430,"stamp":"2024-03-28T22:36:56.245667+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":671689,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1853,"num_read_kb":1067,"num_write":448,"num_write_kb":3105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":868,"ondisk_log_size":868,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":376740,"kb_used_data":15080,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124565084,"statfs":{"total":1151940427776,"available":1151554646016,"internally_reserved":0,"allocated":15441920,"data_stored":8458085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9678,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":126,"num_read_kb":78,"num_write":15,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005234"},"pg_stats":[{"pgid":"3.3f","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.450073+0000","last_change":"2024-03-28T22:33:28.942904+0000","last_active":"2024-03-28T22:35:30.450073+0000","last_peered":"2024-03-28T22:35:30.450073+0000","last_clean":"2024-03-28T22:35:30.450073+0000","last_became_active":"2024-03-28T22:33:28.942574+0000","last_became_peered":"2024-03-28T22:33:28.942574+0000","last_unstale":"2024-03-28T22:35:30.450073+0000","last_undegraded":"2024-03-28T22:35:30.450073+0000","last_fullsized":"2024-03-28T22:35:30.450073+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:55.323919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"94'4","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451519+0000","last_change":"2024-03-28T22:33:28.951795+0000","last_active":"2024-03-28T22:35:30.451519+0000","last_peered":"2024-03-28T22:35:30.451519+0000","last_clean":"2024-03-28T22:35:30.451519+0000","last_became_active":"2024-03-28T22:33:28.951286+0000","last_became_peered":"2024-03-28T22:33:28.951286+0000","last_unstale":"2024-03-28T22:35:30.451519+0000","last_undegraded":"2024-03-28T22:35:30.451519+0000","last_fullsized":"2024-03-28T22:35:30.451519+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:56:11.718244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451291+0000","last_change":"2024-03-28T22:33:28.949476+0000","last_active":"2024-03-28T22:35:30.451291+0000","last_peered":"2024-03-28T22:35:30.451291+0000","last_clean":"2024-03-28T22:35:30.451291+0000","last_became_active":"2024-03-28T22:33:28.949128+0000","last_became_peered":"2024-03-28T22:33:28.949128+0000","last_unstale":"2024-03-28T22:35:30.451291+0000","last_undegraded":"2024-03-28T22:35:30.451291+0000","last_fullsized":"2024-03-28T22:35:30.451291+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:49:29.052373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370703+0000","last_change":"2024-03-28T22:33:28.930715+0000","last_active":"2024-03-28T22:35:30.370703+0000","last_peered":"2024-03-28T22:35:30.370703+0000","last_clean":"2024-03-28T22:35:30.370703+0000","last_became_active":"2024-03-28T22:33:28.930265+0000","last_became_peered":"2024-03-28T22:33:28.930265+0000","last_unstale":"2024-03-28T22:35:30.370703+0000","last_undegraded":"2024-03-28T22:35:30.370703+0000","last_fullsized":"2024-03-28T22:35:30.370703+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:29:54.695671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"94'4","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.253685+0000","last_change":"2024-03-28T22:33:28.944933+0000","last_active":"2024-03-28T22:35:30.253685+0000","last_peered":"2024-03-28T22:35:30.253685+0000","last_clean":"2024-03-28T22:35:30.253685+0000","last_became_active":"2024-03-28T22:33:28.944542+0000","last_became_peered":"2024-03-28T22:33:28.944542+0000","last_unstale":"2024-03-28T22:35:30.253685+0000","last_undegraded":"2024-03-28T22:35:30.253685+0000","last_fullsized":"2024-03-28T22:35:30.253685+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:35:15.719312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"94'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.259295+0000","last_change":"2024-03-28T22:33:28.927920+0000","last_active":"2024-03-28T22:35:30.259295+0000","last_peered":"2024-03-28T22:35:30.259295+0000","last_clean":"2024-03-28T22:35:30.259295+0000","last_became_active":"2024-03-28T22:33:28.927602+0000","last_became_peered":"2024-03-28T22:33:28.927602+0000","last_unstale":"2024-03-28T22:35:30.259295+0000","last_undegraded":"2024-03-28T22:35:30.259295+0000","last_fullsized":"2024-03-28T22:35:30.259295+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:24:22.072707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"94'5","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248059+0000","last_change":"2024-03-28T22:33:28.917963+0000","last_active":"2024-03-28T22:35:30.248059+0000","last_peered":"2024-03-28T22:35:30.248059+0000","last_clean":"2024-03-28T22:35:30.248059+0000","last_became_active":"2024-03-28T22:33:28.917628+0000","last_became_peered":"2024-03-28T22:33:28.917628+0000","last_unstale":"2024-03-28T22:35:30.248059+0000","last_undegraded":"2024-03-28T22:35:30.248059+0000","last_fullsized":"2024-03-28T22:35:30.248059+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:37:09.111640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"94'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371177+0000","last_change":"2024-03-28T22:33:28.942632+0000","last_active":"2024-03-28T22:35:30.371177+0000","last_peered":"2024-03-28T22:35:30.371177+0000","last_clean":"2024-03-28T22:35:30.371177+0000","last_became_active":"2024-03-28T22:33:28.941972+0000","last_became_peered":"2024-03-28T22:33:28.941972+0000","last_unstale":"2024-03-28T22:35:30.371177+0000","last_undegraded":"2024-03-28T22:35:30.371177+0000","last_fullsized":"2024-03-28T22:35:30.371177+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:49:52.642314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"94'11","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448774+0000","last_change":"2024-03-28T22:33:28.924324+0000","last_active":"2024-03-28T22:35:30.448774+0000","last_peered":"2024-03-28T22:35:30.448774+0000","last_clean":"2024-03-28T22:35:30.448774+0000","last_became_active":"2024-03-28T22:33:28.924014+0000","last_became_peered":"2024-03-28T22:33:28.924014+0000","last_unstale":"2024-03-28T22:35:30.448774+0000","last_undegraded":"2024-03-28T22:35:30.448774+0000","last_fullsized":"2024-03-28T22:35:30.448774+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:17:55.172116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"94'16","reported_seq":54,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.259584+0000","last_change":"2024-03-28T22:34:29.987262+0000","last_active":"2024-03-28T22:35:30.259584+0000","last_peered":"2024-03-28T22:35:30.259584+0000","last_clean":"2024-03-28T22:35:30.259584+0000","last_became_active":"2024-03-28T22:34:29.986934+0000","last_became_peered":"2024-03-28T22:34:29.986934+0000","last_unstale":"2024-03-28T22:35:30.259584+0000","last_undegraded":"2024-03-28T22:35:30.259584+0000","last_fullsized":"2024-03-28T22:35:30.259584+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:00:41.574800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"94'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248516+0000","last_change":"2024-03-28T22:33:28.930802+0000","last_active":"2024-03-28T22:35:30.248516+0000","last_peered":"2024-03-28T22:35:30.248516+0000","last_clean":"2024-03-28T22:35:30.248516+0000","last_became_active":"2024-03-28T22:33:28.930475+0000","last_became_peered":"2024-03-28T22:33:28.930475+0000","last_unstale":"2024-03-28T22:35:30.248516+0000","last_undegraded":"2024-03-28T22:35:30.248516+0000","last_fullsized":"2024-03-28T22:35:30.248516+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:48.443803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"94'7","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370695+0000","last_change":"2024-03-28T22:33:28.927057+0000","last_active":"2024-03-28T22:35:30.370695+0000","last_peered":"2024-03-28T22:35:30.370695+0000","last_clean":"2024-03-28T22:35:30.370695+0000","last_became_active":"2024-03-28T22:33:28.926598+0000","last_became_peered":"2024-03-28T22:33:28.926598+0000","last_unstale":"2024-03-28T22:35:30.370695+0000","last_undegraded":"2024-03-28T22:35:30.370695+0000","last_fullsized":"2024-03-28T22:35:30.370695+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:21:00.710203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"94'10","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368918+0000","last_change":"2024-03-28T22:33:28.937034+0000","last_active":"2024-03-28T22:35:30.368918+0000","last_peered":"2024-03-28T22:35:30.368918+0000","last_clean":"2024-03-28T22:35:30.368918+0000","last_became_active":"2024-03-28T22:33:28.933713+0000","last_became_peered":"2024-03-28T22:33:28.933713+0000","last_unstale":"2024-03-28T22:35:30.368918+0000","last_undegraded":"2024-03-28T22:35:30.368918+0000","last_fullsized":"2024-03-28T22:35:30.368918+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:58:24.148897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.453212+0000","last_change":"2024-03-28T22:33:28.947384+0000","last_active":"2024-03-28T22:35:30.453212+0000","last_peered":"2024-03-28T22:35:30.453212+0000","last_clean":"2024-03-28T22:35:30.453212+0000","last_became_active":"2024-03-28T22:33:28.947017+0000","last_became_peered":"2024-03-28T22:33:28.947017+0000","last_unstale":"2024-03-28T22:35:30.453212+0000","last_undegraded":"2024-03-28T22:35:30.453212+0000","last_fullsized":"2024-03-28T22:35:30.453212+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:43:32.781161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451880+0000","last_change":"2024-03-28T22:33:28.950119+0000","last_active":"2024-03-28T22:35:30.451880+0000","last_peered":"2024-03-28T22:35:30.451880+0000","last_clean":"2024-03-28T22:35:30.451880+0000","last_became_active":"2024-03-28T22:33:28.949798+0000","last_became_peered":"2024-03-28T22:33:28.949798+0000","last_unstale":"2024-03-28T22:35:30.451880+0000","last_undegraded":"2024-03-28T22:35:30.451880+0000","last_fullsized":"2024-03-28T22:35:30.451880+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:16.717678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"94'9","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448318+0000","last_change":"2024-03-28T22:33:28.943255+0000","last_active":"2024-03-28T22:35:30.448318+0000","last_peered":"2024-03-28T22:35:30.448318+0000","last_clean":"2024-03-28T22:35:30.448318+0000","last_became_active":"2024-03-28T22:33:28.942923+0000","last_became_peered":"2024-03-28T22:33:28.942923+0000","last_unstale":"2024-03-28T22:35:30.448318+0000","last_undegraded":"2024-03-28T22:35:30.448318+0000","last_fullsized":"2024-03-28T22:35:30.448318+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:05:04.197635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449964+0000","last_change":"2024-03-28T22:33:28.942906+0000","last_active":"2024-03-28T22:35:30.449964+0000","last_peered":"2024-03-28T22:35:30.449964+0000","last_clean":"2024-03-28T22:35:30.449964+0000","last_became_active":"2024-03-28T22:33:28.942573+0000","last_became_peered":"2024-03-28T22:33:28.942573+0000","last_unstale":"2024-03-28T22:35:30.449964+0000","last_undegraded":"2024-03-28T22:35:30.449964+0000","last_fullsized":"2024-03-28T22:35:30.449964+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:39:17.651101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"94'10","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252948+0000","last_change":"2024-03-28T22:33:28.932075+0000","last_active":"2024-03-28T22:35:30.252948+0000","last_peered":"2024-03-28T22:35:30.252948+0000","last_clean":"2024-03-28T22:35:30.252948+0000","last_became_active":"2024-03-28T22:33:28.931689+0000","last_became_peered":"2024-03-28T22:33:28.931689+0000","last_unstale":"2024-03-28T22:35:30.252948+0000","last_undegraded":"2024-03-28T22:35:30.252948+0000","last_fullsized":"2024-03-28T22:35:30.252948+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:50:43.031082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252343+0000","last_change":"2024-03-28T22:33:28.944987+0000","last_active":"2024-03-28T22:35:30.252343+0000","last_peered":"2024-03-28T22:35:30.252343+0000","last_clean":"2024-03-28T22:35:30.252343+0000","last_became_active":"2024-03-28T22:33:28.944622+0000","last_became_peered":"2024-03-28T22:33:28.944622+0000","last_unstale":"2024-03-28T22:35:30.252343+0000","last_undegraded":"2024-03-28T22:35:30.252343+0000","last_fullsized":"2024-03-28T22:35:30.252343+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:48:58.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":30,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251826+0000","last_change":"2024-03-28T22:34:30.429709+0000","last_active":"2024-03-28T22:35:30.251826+0000","last_peered":"2024-03-28T22:35:30.251826+0000","last_clean":"2024-03-28T22:35:30.251826+0000","last_became_active":"2024-03-28T22:34:30.429122+0000","last_became_peered":"2024-03-28T22:34:30.429122+0000","last_unstale":"2024-03-28T22:35:30.251826+0000","last_undegraded":"2024-03-28T22:35:30.251826+0000","last_fullsized":"2024-03-28T22:35:30.251826+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:58:05.313669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"92'13","reported_seq":214,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:36:50.437836+0000","last_change":"2024-03-28T22:33:28.442524+0000","last_active":"2024-03-28T22:36:50.437836+0000","last_peered":"2024-03-28T22:36:50.437836+0000","last_clean":"2024-03-28T22:36:50.437836+0000","last_became_active":"2024-03-28T22:33:28.442169+0000","last_became_peered":"2024-03-28T22:33:28.442169+0000","last_unstale":"2024-03-28T22:36:50.437836+0000","last_undegraded":"2024-03-28T22:36:50.437836+0000","last_fullsized":"2024-03-28T22:36:50.437836+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:08:47.101119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":167,"num_read_kb":167,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"94'5","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371822+0000","last_change":"2024-03-28T22:33:28.946697+0000","last_active":"2024-03-28T22:35:30.371822+0000","last_peered":"2024-03-28T22:35:30.371822+0000","last_clean":"2024-03-28T22:35:30.371822+0000","last_became_active":"2024-03-28T22:33:28.946303+0000","last_became_peered":"2024-03-28T22:33:28.946303+0000","last_unstale":"2024-03-28T22:35:30.371822+0000","last_undegraded":"2024-03-28T22:35:30.371822+0000","last_fullsized":"2024-03-28T22:35:30.371822+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:32:01.286073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":30,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370372+0000","last_change":"2024-03-28T22:34:30.429579+0000","last_active":"2024-03-28T22:35:30.370372+0000","last_peered":"2024-03-28T22:35:30.370372+0000","last_clean":"2024-03-28T22:35:30.370372+0000","last_became_active":"2024-03-28T22:34:30.429054+0000","last_became_peered":"2024-03-28T22:34:30.429054+0000","last_unstale":"2024-03-28T22:35:30.370372+0000","last_undegraded":"2024-03-28T22:35:30.370372+0000","last_fullsized":"2024-03-28T22:35:30.370372+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:03:46.492786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"92'2","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.372174+0000","last_change":"2024-03-28T22:33:28.457661+0000","last_active":"2024-03-28T22:35:30.372174+0000","last_peered":"2024-03-28T22:35:30.372174+0000","last_clean":"2024-03-28T22:35:30.372174+0000","last_became_active":"2024-03-28T22:33:28.457032+0000","last_became_peered":"2024-03-28T22:33:28.457032+0000","last_unstale":"2024-03-28T22:35:30.372174+0000","last_undegraded":"2024-03-28T22:35:30.372174+0000","last_fullsized":"2024-03-28T22:35:30.372174+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:43:09.820630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"94'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.453641+0000","last_change":"2024-03-28T22:33:28.954252+0000","last_active":"2024-03-28T22:35:30.453641+0000","last_peered":"2024-03-28T22:35:30.453641+0000","last_clean":"2024-03-28T22:35:30.453641+0000","last_became_active":"2024-03-28T22:33:28.953922+0000","last_became_peered":"2024-03-28T22:33:28.953922+0000","last_unstale":"2024-03-28T22:35:30.453641+0000","last_undegraded":"2024-03-28T22:35:30.453641+0000","last_fullsized":"2024-03-28T22:35:30.453641+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:18:47.097859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370607+0000","last_change":"2024-03-28T22:33:34.029129+0000","last_active":"2024-03-28T22:35:30.370607+0000","last_peered":"2024-03-28T22:35:30.370607+0000","last_clean":"2024-03-28T22:35:30.370607+0000","last_became_active":"2024-03-28T22:33:34.028452+0000","last_became_peered":"2024-03-28T22:33:34.028452+0000","last_unstale":"2024-03-28T22:35:30.370607+0000","last_undegraded":"2024-03-28T22:35:30.370607+0000","last_fullsized":"2024-03-28T22:35:30.370607+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:28.982773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"92'12","reported_seq":47,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254205+0000","last_change":"2024-03-28T22:33:28.439568+0000","last_active":"2024-03-28T22:35:30.254205+0000","last_peered":"2024-03-28T22:35:30.254205+0000","last_clean":"2024-03-28T22:35:30.254205+0000","last_became_active":"2024-03-28T22:33:28.438887+0000","last_became_peered":"2024-03-28T22:33:28.438887+0000","last_unstale":"2024-03-28T22:35:30.254205+0000","last_undegraded":"2024-03-28T22:35:30.254205+0000","last_fullsized":"2024-03-28T22:35:30.254205+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:56:43.617621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"94'9","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448797+0000","last_change":"2024-03-28T22:33:28.938311+0000","last_active":"2024-03-28T22:35:30.448797+0000","last_peered":"2024-03-28T22:35:30.448797+0000","last_clean":"2024-03-28T22:35:30.448797+0000","last_became_active":"2024-03-28T22:33:28.937997+0000","last_became_peered":"2024-03-28T22:33:28.937997+0000","last_unstale":"2024-03-28T22:35:30.448797+0000","last_undegraded":"2024-03-28T22:35:30.448797+0000","last_fullsized":"2024-03-28T22:35:30.448797+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:54:44.001545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370428+0000","last_change":"2024-03-28T22:33:34.016931+0000","last_active":"2024-03-28T22:35:30.370428+0000","last_peered":"2024-03-28T22:35:30.370428+0000","last_clean":"2024-03-28T22:35:30.370428+0000","last_became_active":"2024-03-28T22:33:34.016559+0000","last_became_peered":"2024-03-28T22:33:34.016559+0000","last_unstale":"2024-03-28T22:35:30.370428+0000","last_undegraded":"2024-03-28T22:35:30.370428+0000","last_fullsized":"2024-03-28T22:35:30.370428+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:08:31.252351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"92'5","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254097+0000","last_change":"2024-03-28T22:33:28.433166+0000","last_active":"2024-03-28T22:35:30.254097+0000","last_peered":"2024-03-28T22:35:30.254097+0000","last_clean":"2024-03-28T22:35:30.254097+0000","last_became_active":"2024-03-28T22:33:28.432716+0000","last_became_peered":"2024-03-28T22:33:28.432716+0000","last_unstale":"2024-03-28T22:35:30.254097+0000","last_undegraded":"2024-03-28T22:35:30.254097+0000","last_fullsized":"2024-03-28T22:35:30.254097+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:42:41.613583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"94'15","reported_seq":47,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275541+0000","last_change":"2024-03-28T22:33:28.923307+0000","last_active":"2024-03-28T22:35:30.275541+0000","last_peered":"2024-03-28T22:35:30.275541+0000","last_clean":"2024-03-28T22:35:30.275541+0000","last_became_active":"2024-03-28T22:33:28.922499+0000","last_became_peered":"2024-03-28T22:33:28.922499+0000","last_unstale":"2024-03-28T22:35:30.275541+0000","last_undegraded":"2024-03-28T22:35:30.275541+0000","last_fullsized":"2024-03-28T22:35:30.275541+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:48:48.514482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252183+0000","last_change":"2024-03-28T22:33:34.002209+0000","last_active":"2024-03-28T22:35:30.252183+0000","last_peered":"2024-03-28T22:35:30.252183+0000","last_clean":"2024-03-28T22:35:30.252183+0000","last_became_active":"2024-03-28T22:33:34.001746+0000","last_became_peered":"2024-03-28T22:33:34.001746+0000","last_unstale":"2024-03-28T22:35:30.252183+0000","last_undegraded":"2024-03-28T22:35:30.252183+0000","last_fullsized":"2024-03-28T22:35:30.252183+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:02:02.333328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"96'7","reported_seq":211,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:36:50.482307+0000","last_change":"2024-03-28T22:33:28.435380+0000","last_active":"2024-03-28T22:36:50.482307+0000","last_peered":"2024-03-28T22:36:50.482307+0000","last_clean":"2024-03-28T22:36:50.482307+0000","last_became_active":"2024-03-28T22:33:28.434898+0000","last_became_peered":"2024-03-28T22:33:28.434898+0000","last_unstale":"2024-03-28T22:36:50.482307+0000","last_undegraded":"2024-03-28T22:36:50.482307+0000","last_fullsized":"2024-03-28T22:36:50.482307+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:58:06.810394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":170,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"94'7","reported_seq":45,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254282+0000","last_change":"2024-03-28T22:34:30.121370+0000","last_active":"2024-03-28T22:35:30.254282+0000","last_peered":"2024-03-28T22:35:30.254282+0000","last_clean":"2024-03-28T22:35:30.254282+0000","last_became_active":"2024-03-28T22:34:30.121086+0000","last_became_peered":"2024-03-28T22:34:30.121086+0000","last_unstale":"2024-03-28T22:35:30.254282+0000","last_undegraded":"2024-03-28T22:35:30.254282+0000","last_fullsized":"2024-03-28T22:35:30.254282+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:26:24.244663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252057+0000","last_change":"2024-03-28T22:33:34.031350+0000","last_active":"2024-03-28T22:35:30.252057+0000","last_peered":"2024-03-28T22:35:30.252057+0000","last_clean":"2024-03-28T22:35:30.252057+0000","last_became_active":"2024-03-28T22:33:34.030946+0000","last_became_peered":"2024-03-28T22:33:34.030946+0000","last_unstale":"2024-03-28T22:35:30.252057+0000","last_undegraded":"2024-03-28T22:35:30.252057+0000","last_fullsized":"2024-03-28T22:35:30.252057+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:54:22.907363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"92'7","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.259226+0000","last_change":"2024-03-28T22:33:28.446197+0000","last_active":"2024-03-28T22:35:30.259226+0000","last_peered":"2024-03-28T22:35:30.259226+0000","last_clean":"2024-03-28T22:35:30.259226+0000","last_became_active":"2024-03-28T22:33:28.445891+0000","last_became_peered":"2024-03-28T22:33:28.445891+0000","last_unstale":"2024-03-28T22:35:30.259226+0000","last_undegraded":"2024-03-28T22:35:30.259226+0000","last_fullsized":"2024-03-28T22:35:30.259226+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:58:57.385302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"94'8","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275591+0000","last_change":"2024-03-28T22:33:28.955651+0000","last_active":"2024-03-28T22:35:30.275591+0000","last_peered":"2024-03-28T22:35:30.275591+0000","last_clean":"2024-03-28T22:35:30.275591+0000","last_became_active":"2024-03-28T22:33:28.955276+0000","last_became_peered":"2024-03-28T22:33:28.955276+0000","last_unstale":"2024-03-28T22:35:30.275591+0000","last_undegraded":"2024-03-28T22:35:30.275591+0000","last_fullsized":"2024-03-28T22:35:30.275591+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:36:13.455327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449809+0000","last_change":"2024-03-28T22:33:34.017520+0000","last_active":"2024-03-28T22:35:30.449809+0000","last_peered":"2024-03-28T22:35:30.449809+0000","last_clean":"2024-03-28T22:35:30.449809+0000","last_became_active":"2024-03-28T22:33:34.017022+0000","last_became_peered":"2024-03-28T22:33:34.017022+0000","last_unstale":"2024-03-28T22:35:30.449809+0000","last_undegraded":"2024-03-28T22:35:30.449809+0000","last_fullsized":"2024-03-28T22:35:30.449809+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:41:17.431581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"92'3","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368171+0000","last_change":"2024-03-28T22:33:28.445401+0000","last_active":"2024-03-28T22:35:30.368171+0000","last_peered":"2024-03-28T22:35:30.368171+0000","last_clean":"2024-03-28T22:35:30.368171+0000","last_became_active":"2024-03-28T22:33:28.444237+0000","last_became_peered":"2024-03-28T22:33:28.444237+0000","last_unstale":"2024-03-28T22:35:30.368171+0000","last_undegraded":"2024-03-28T22:35:30.368171+0000","last_fullsized":"2024-03-28T22:35:30.368171+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:18:12.820890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"94'8","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.250889+0000","last_change":"2024-03-28T22:34:30.156248+0000","last_active":"2024-03-28T22:35:30.250889+0000","last_peered":"2024-03-28T22:35:30.250889+0000","last_clean":"2024-03-28T22:35:30.250889+0000","last_became_active":"2024-03-28T22:34:30.155056+0000","last_became_peered":"2024-03-28T22:34:30.155056+0000","last_unstale":"2024-03-28T22:35:30.250889+0000","last_undegraded":"2024-03-28T22:35:30.250889+0000","last_fullsized":"2024-03-28T22:35:30.250889+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:28:56.469728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451480+0000","last_change":"2024-03-28T22:33:34.019899+0000","last_active":"2024-03-28T22:35:30.451480+0000","last_peered":"2024-03-28T22:35:30.451480+0000","last_clean":"2024-03-28T22:35:30.451480+0000","last_became_active":"2024-03-28T22:33:34.019392+0000","last_became_peered":"2024-03-28T22:33:34.019392+0000","last_unstale":"2024-03-28T22:35:30.451480+0000","last_undegraded":"2024-03-28T22:35:30.451480+0000","last_fullsized":"2024-03-28T22:35:30.451480+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:08:38.633797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"92'3","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275105+0000","last_change":"2024-03-28T22:33:28.433243+0000","last_active":"2024-03-28T22:35:30.275105+0000","last_peered":"2024-03-28T22:35:30.275105+0000","last_clean":"2024-03-28T22:35:30.275105+0000","last_became_active":"2024-03-28T22:33:28.432927+0000","last_became_peered":"2024-03-28T22:33:28.432927+0000","last_unstale":"2024-03-28T22:35:30.275105+0000","last_undegraded":"2024-03-28T22:35:30.275105+0000","last_fullsized":"2024-03-28T22:35:30.275105+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:40:43.444229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"94'11","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251345+0000","last_change":"2024-03-28T22:33:28.948111+0000","last_active":"2024-03-28T22:35:30.251345+0000","last_peered":"2024-03-28T22:35:30.251345+0000","last_clean":"2024-03-28T22:35:30.251345+0000","last_became_active":"2024-03-28T22:33:28.944649+0000","last_became_peered":"2024-03-28T22:33:28.944649+0000","last_unstale":"2024-03-28T22:35:30.251345+0000","last_undegraded":"2024-03-28T22:35:30.251345+0000","last_fullsized":"2024-03-28T22:35:30.251345+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:57:11.483037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371506+0000","last_change":"2024-03-28T22:33:34.000837+0000","last_active":"2024-03-28T22:35:30.371506+0000","last_peered":"2024-03-28T22:35:30.371506+0000","last_clean":"2024-03-28T22:35:30.371506+0000","last_became_active":"2024-03-28T22:33:34.000120+0000","last_became_peered":"2024-03-28T22:33:34.000120+0000","last_unstale":"2024-03-28T22:35:30.371506+0000","last_undegraded":"2024-03-28T22:35:30.371506+0000","last_fullsized":"2024-03-28T22:35:30.371506+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:14:32.028952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"96'6","reported_seq":377,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:36:53.486772+0000","last_change":"2024-03-28T22:33:28.436985+0000","last_active":"2024-03-28T22:36:53.486772+0000","last_peered":"2024-03-28T22:36:53.486772+0000","last_clean":"2024-03-28T22:36:53.486772+0000","last_became_active":"2024-03-28T22:33:28.436625+0000","last_became_peered":"2024-03-28T22:33:28.436625+0000","last_unstale":"2024-03-28T22:36:53.486772+0000","last_undegraded":"2024-03-28T22:36:53.486772+0000","last_fullsized":"2024-03-28T22:36:53.486772+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:50:45.775191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":337,"num_read_kb":337,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"94'10","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448895+0000","last_change":"2024-03-28T22:33:28.933265+0000","last_active":"2024-03-28T22:35:30.448895+0000","last_peered":"2024-03-28T22:35:30.448895+0000","last_clean":"2024-03-28T22:35:30.448895+0000","last_became_active":"2024-03-28T22:33:28.932954+0000","last_became_peered":"2024-03-28T22:33:28.932954+0000","last_unstale":"2024-03-28T22:35:30.448895+0000","last_undegraded":"2024-03-28T22:35:30.448895+0000","last_fullsized":"2024-03-28T22:35:30.448895+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:37.039055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251275+0000","last_change":"2024-03-28T22:33:34.024529+0000","last_active":"2024-03-28T22:35:30.251275+0000","last_peered":"2024-03-28T22:35:30.251275+0000","last_clean":"2024-03-28T22:35:30.251275+0000","last_became_active":"2024-03-28T22:33:34.020949+0000","last_became_peered":"2024-03-28T22:33:34.020949+0000","last_unstale":"2024-03-28T22:35:30.251275+0000","last_undegraded":"2024-03-28T22:35:30.251275+0000","last_fullsized":"2024-03-28T22:35:30.251275+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:28:52.039684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"92'4","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248388+0000","last_change":"2024-03-28T22:33:28.432599+0000","last_active":"2024-03-28T22:35:30.248388+0000","last_peered":"2024-03-28T22:35:30.248388+0000","last_clean":"2024-03-28T22:35:30.248388+0000","last_became_active":"2024-03-28T22:33:28.430710+0000","last_became_peered":"2024-03-28T22:33:28.430710+0000","last_unstale":"2024-03-28T22:35:30.248388+0000","last_undegraded":"2024-03-28T22:35:30.248388+0000","last_fullsized":"2024-03-28T22:35:30.248388+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:54.174747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"94'9","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.453918+0000","last_change":"2024-03-28T22:33:28.955441+0000","last_active":"2024-03-28T22:35:30.453918+0000","last_peered":"2024-03-28T22:35:30.453918+0000","last_clean":"2024-03-28T22:35:30.453918+0000","last_became_active":"2024-03-28T22:33:28.953269+0000","last_became_peered":"2024-03-28T22:33:28.953269+0000","last_unstale":"2024-03-28T22:35:30.453918+0000","last_undegraded":"2024-03-28T22:35:30.453918+0000","last_fullsized":"2024-03-28T22:35:30.453918+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:47:04.128605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370719+0000","last_change":"2024-03-28T22:33:34.028791+0000","last_active":"2024-03-28T22:35:30.370719+0000","last_peered":"2024-03-28T22:35:30.370719+0000","last_clean":"2024-03-28T22:35:30.370719+0000","last_became_active":"2024-03-28T22:33:34.028361+0000","last_became_peered":"2024-03-28T22:33:34.028361+0000","last_unstale":"2024-03-28T22:35:30.370719+0000","last_undegraded":"2024-03-28T22:35:30.370719+0000","last_fullsized":"2024-03-28T22:35:30.370719+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:42:21.264459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"92'5","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275283+0000","last_change":"2024-03-28T22:33:28.455353+0000","last_active":"2024-03-28T22:35:30.275283+0000","last_peered":"2024-03-28T22:35:30.275283+0000","last_clean":"2024-03-28T22:35:30.275283+0000","last_became_active":"2024-03-28T22:33:28.455090+0000","last_became_peered":"2024-03-28T22:33:28.455090+0000","last_unstale":"2024-03-28T22:35:30.275283+0000","last_undegraded":"2024-03-28T22:35:30.275283+0000","last_fullsized":"2024-03-28T22:35:30.275283+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:19:22.496600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"94'11","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448743+0000","last_change":"2024-03-28T22:33:28.928875+0000","last_active":"2024-03-28T22:35:30.448743+0000","last_peered":"2024-03-28T22:35:30.448743+0000","last_clean":"2024-03-28T22:35:30.448743+0000","last_became_active":"2024-03-28T22:33:28.928449+0000","last_became_peered":"2024-03-28T22:33:28.928449+0000","last_unstale":"2024-03-28T22:35:30.448743+0000","last_undegraded":"2024-03-28T22:35:30.448743+0000","last_fullsized":"2024-03-28T22:35:30.448743+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:34:00.402100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251348+0000","last_change":"2024-03-28T22:33:34.012739+0000","last_active":"2024-03-28T22:35:30.251348+0000","last_peered":"2024-03-28T22:35:30.251348+0000","last_clean":"2024-03-28T22:35:30.251348+0000","last_became_active":"2024-03-28T22:33:34.012262+0000","last_became_peered":"2024-03-28T22:33:34.012262+0000","last_unstale":"2024-03-28T22:35:30.251348+0000","last_undegraded":"2024-03-28T22:35:30.251348+0000","last_fullsized":"2024-03-28T22:35:30.251348+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:24:48.351037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"96'42","reported_seq":424,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:36:54.358887+0000","last_change":"2024-03-28T22:33:28.430606+0000","last_active":"2024-03-28T22:36:54.358887+0000","last_peered":"2024-03-28T22:36:54.358887+0000","last_clean":"2024-03-28T22:36:54.358887+0000","last_became_active":"2024-03-28T22:33:28.430233+0000","last_became_peered":"2024-03-28T22:33:28.430233+0000","last_unstale":"2024-03-28T22:36:54.358887+0000","last_undegraded":"2024-03-28T22:36:54.358887+0000","last_fullsized":"2024-03-28T22:36:54.358887+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:49.000569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73045,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":348,"num_read_kb":246,"num_write":44,"num_write_kb":92,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"94'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254080+0000","last_change":"2024-03-28T22:33:28.931167+0000","last_active":"2024-03-28T22:35:30.254080+0000","last_peered":"2024-03-28T22:35:30.254080+0000","last_clean":"2024-03-28T22:35:30.254080+0000","last_became_active":"2024-03-28T22:33:28.930824+0000","last_became_peered":"2024-03-28T22:33:28.930824+0000","last_unstale":"2024-03-28T22:35:30.254080+0000","last_undegraded":"2024-03-28T22:35:30.254080+0000","last_fullsized":"2024-03-28T22:35:30.254080+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:46:49.102791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275836+0000","last_change":"2024-03-28T22:33:34.023056+0000","last_active":"2024-03-28T22:35:30.275836+0000","last_peered":"2024-03-28T22:35:30.275836+0000","last_clean":"2024-03-28T22:35:30.275836+0000","last_became_active":"2024-03-28T22:33:34.022612+0000","last_became_peered":"2024-03-28T22:33:34.022612+0000","last_unstale":"2024-03-28T22:35:30.275836+0000","last_undegraded":"2024-03-28T22:35:30.275836+0000","last_fullsized":"2024-03-28T22:35:30.275836+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:52:31.936008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"92'9","reported_seq":45,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252111+0000","last_change":"2024-03-28T22:33:28.430963+0000","last_active":"2024-03-28T22:35:30.252111+0000","last_peered":"2024-03-28T22:35:30.252111+0000","last_clean":"2024-03-28T22:35:30.252111+0000","last_became_active":"2024-03-28T22:33:28.430632+0000","last_became_peered":"2024-03-28T22:33:28.430632+0000","last_unstale":"2024-03-28T22:35:30.252111+0000","last_undegraded":"2024-03-28T22:35:30.252111+0000","last_fullsized":"2024-03-28T22:35:30.252111+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:30:23.104769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"94'9","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368385+0000","last_change":"2024-03-28T22:33:28.954105+0000","last_active":"2024-03-28T22:35:30.368385+0000","last_peered":"2024-03-28T22:35:30.368385+0000","last_clean":"2024-03-28T22:35:30.368385+0000","last_became_active":"2024-03-28T22:33:28.953768+0000","last_became_peered":"2024-03-28T22:33:28.953768+0000","last_unstale":"2024-03-28T22:35:30.368385+0000","last_undegraded":"2024-03-28T22:35:30.368385+0000","last_fullsized":"2024-03-28T22:35:30.368385+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:01:25.184523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449565+0000","last_change":"2024-03-28T22:33:34.027976+0000","last_active":"2024-03-28T22:35:30.449565+0000","last_peered":"2024-03-28T22:35:30.449565+0000","last_clean":"2024-03-28T22:35:30.449565+0000","last_became_active":"2024-03-28T22:33:34.027509+0000","last_became_peered":"2024-03-28T22:33:34.027509+0000","last_unstale":"2024-03-28T22:35:30.449565+0000","last_undegraded":"2024-03-28T22:35:30.449565+0000","last_fullsized":"2024-03-28T22:35:30.449565+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:25:41.908441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"92'5","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275307+0000","last_change":"2024-03-28T22:33:28.433283+0000","last_active":"2024-03-28T22:35:30.275307+0000","last_peered":"2024-03-28T22:35:30.275307+0000","last_clean":"2024-03-28T22:35:30.275307+0000","last_became_active":"2024-03-28T22:33:28.432954+0000","last_became_peered":"2024-03-28T22:33:28.432954+0000","last_unstale":"2024-03-28T22:35:30.275307+0000","last_undegraded":"2024-03-28T22:35:30.275307+0000","last_fullsized":"2024-03-28T22:35:30.275307+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:08:34.480459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"94'15","reported_seq":47,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248465+0000","last_change":"2024-03-28T22:33:28.939016+0000","last_active":"2024-03-28T22:35:30.248465+0000","last_peered":"2024-03-28T22:35:30.248465+0000","last_clean":"2024-03-28T22:35:30.248465+0000","last_became_active":"2024-03-28T22:33:28.938695+0000","last_became_peered":"2024-03-28T22:33:28.938695+0000","last_unstale":"2024-03-28T22:35:30.248465+0000","last_undegraded":"2024-03-28T22:35:30.248465+0000","last_fullsized":"2024-03-28T22:35:30.248465+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:26:53.264559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252805+0000","last_change":"2024-03-28T22:33:34.030127+0000","last_active":"2024-03-28T22:35:30.252805+0000","last_peered":"2024-03-28T22:35:30.252805+0000","last_clean":"2024-03-28T22:35:30.252805+0000","last_became_active":"2024-03-28T22:33:34.028056+0000","last_became_peered":"2024-03-28T22:33:34.028056+0000","last_unstale":"2024-03-28T22:35:30.252805+0000","last_undegraded":"2024-03-28T22:35:30.252805+0000","last_fullsized":"2024-03-28T22:35:30.252805+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:29:41.785013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'7","reported_seq":209,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:36:50.560497+0000","last_change":"2024-03-28T22:33:28.435350+0000","last_active":"2024-03-28T22:36:50.560497+0000","last_peered":"2024-03-28T22:36:50.560497+0000","last_clean":"2024-03-28T22:36:50.560497+0000","last_became_active":"2024-03-28T22:33:28.434954+0000","last_became_peered":"2024-03-28T22:33:28.434954+0000","last_unstale":"2024-03-28T22:36:50.560497+0000","last_undegraded":"2024-03-28T22:36:50.560497+0000","last_fullsized":"2024-03-28T22:36:50.560497+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:04.969596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":170,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"94'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248129+0000","last_change":"2024-03-28T22:33:28.921219+0000","last_active":"2024-03-28T22:35:30.248129+0000","last_peered":"2024-03-28T22:35:30.248129+0000","last_clean":"2024-03-28T22:35:30.248129+0000","last_became_active":"2024-03-28T22:33:28.920664+0000","last_became_peered":"2024-03-28T22:33:28.920664+0000","last_unstale":"2024-03-28T22:35:30.248129+0000","last_undegraded":"2024-03-28T22:35:30.248129+0000","last_fullsized":"2024-03-28T22:35:30.248129+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:48:55.032058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369383+0000","last_change":"2024-03-28T22:33:34.018101+0000","last_active":"2024-03-28T22:35:30.369383+0000","last_peered":"2024-03-28T22:35:30.369383+0000","last_clean":"2024-03-28T22:35:30.369383+0000","last_became_active":"2024-03-28T22:33:34.017260+0000","last_became_peered":"2024-03-28T22:33:34.017260+0000","last_unstale":"2024-03-28T22:35:30.369383+0000","last_undegraded":"2024-03-28T22:35:30.369383+0000","last_fullsized":"2024-03-28T22:35:30.369383+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:48:53.178090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"92'17","reported_seq":52,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.253595+0000","last_change":"2024-03-28T22:33:28.450232+0000","last_active":"2024-03-28T22:35:30.253595+0000","last_peered":"2024-03-28T22:35:30.253595+0000","last_clean":"2024-03-28T22:35:30.253595+0000","last_became_active":"2024-03-28T22:33:28.449958+0000","last_became_peered":"2024-03-28T22:33:28.449958+0000","last_unstale":"2024-03-28T22:35:30.253595+0000","last_undegraded":"2024-03-28T22:35:30.253595+0000","last_fullsized":"2024-03-28T22:35:30.253595+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:01:42.130260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"94'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275235+0000","last_change":"2024-03-28T22:33:28.926263+0000","last_active":"2024-03-28T22:35:30.275235+0000","last_peered":"2024-03-28T22:35:30.275235+0000","last_clean":"2024-03-28T22:35:30.275235+0000","last_became_active":"2024-03-28T22:33:28.925927+0000","last_became_peered":"2024-03-28T22:33:28.925927+0000","last_unstale":"2024-03-28T22:35:30.275235+0000","last_undegraded":"2024-03-28T22:35:30.275235+0000","last_fullsized":"2024-03-28T22:35:30.275235+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:41:42.477694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.455002+0000","last_change":"2024-03-28T22:35:30.455002+0000","last_active":"2024-03-28T22:35:30.455002+0000","last_peered":"2024-03-28T22:35:30.455002+0000","last_clean":"2024-03-28T22:35:30.455002+0000","last_became_active":"2024-03-28T22:35:30.454479+0000","last_became_peered":"2024-03-28T22:35:30.454479+0000","last_unstale":"2024-03-28T22:35:30.455002+0000","last_undegraded":"2024-03-28T22:35:30.455002+0000","last_fullsized":"2024-03-28T22:35:30.455002+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:57:06.300031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,3,1],"acting":[10,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.f","version":"92'2","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449832+0000","last_change":"2024-03-28T22:34:29.986650+0000","last_active":"2024-03-28T22:35:30.449832+0000","last_peered":"2024-03-28T22:35:30.449832+0000","last_clean":"2024-03-28T22:35:30.449832+0000","last_became_active":"2024-03-28T22:34:29.981875+0000","last_became_peered":"2024-03-28T22:34:29.981875+0000","last_unstale":"2024-03-28T22:35:30.449832+0000","last_undegraded":"2024-03-28T22:35:30.449832+0000","last_fullsized":"2024-03-28T22:35:30.449832+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:45:24.166133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"94'5","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449947+0000","last_change":"2024-03-28T22:34:29.982815+0000","last_active":"2024-03-28T22:35:30.449947+0000","last_peered":"2024-03-28T22:35:30.449947+0000","last_clean":"2024-03-28T22:35:30.449947+0000","last_became_active":"2024-03-28T22:34:29.982458+0000","last_became_peered":"2024-03-28T22:34:29.982458+0000","last_unstale":"2024-03-28T22:35:30.449947+0000","last_undegraded":"2024-03-28T22:35:30.449947+0000","last_fullsized":"2024-03-28T22:35:30.449947+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:11:30.617181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":30,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451505+0000","last_change":"2024-03-28T22:35:30.451063+0000","last_active":"2024-03-28T22:35:30.451505+0000","last_peered":"2024-03-28T22:35:30.451505+0000","last_clean":"2024-03-28T22:35:30.451505+0000","last_became_active":"2024-03-28T22:35:30.450525+0000","last_became_peered":"2024-03-28T22:35:30.450525+0000","last_unstale":"2024-03-28T22:35:30.451505+0000","last_undegraded":"2024-03-28T22:35:30.451505+0000","last_fullsized":"2024-03-28T22:35:30.451505+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:35:29.004834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"92'1","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252202+0000","last_change":"2024-03-28T22:34:29.992884+0000","last_active":"2024-03-28T22:35:30.252202+0000","last_peered":"2024-03-28T22:35:30.252202+0000","last_clean":"2024-03-28T22:35:30.252202+0000","last_became_active":"2024-03-28T22:34:29.991657+0000","last_became_peered":"2024-03-28T22:34:29.991657+0000","last_unstale":"2024-03-28T22:35:30.252202+0000","last_undegraded":"2024-03-28T22:35:30.252202+0000","last_fullsized":"2024-03-28T22:35:30.252202+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:25:14.540783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"94'4","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.259438+0000","last_change":"2024-03-28T22:33:28.942097+0000","last_active":"2024-03-28T22:35:30.259438+0000","last_peered":"2024-03-28T22:35:30.259438+0000","last_clean":"2024-03-28T22:35:30.259438+0000","last_became_active":"2024-03-28T22:33:28.941812+0000","last_became_peered":"2024-03-28T22:33:28.941812+0000","last_unstale":"2024-03-28T22:35:30.259438+0000","last_undegraded":"2024-03-28T22:35:30.259438+0000","last_fullsized":"2024-03-28T22:35:30.259438+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:36:10.204216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369177+0000","last_change":"2024-03-28T22:33:34.023171+0000","last_active":"2024-03-28T22:35:30.369177+0000","last_peered":"2024-03-28T22:35:30.369177+0000","last_clean":"2024-03-28T22:35:30.369177+0000","last_became_active":"2024-03-28T22:33:34.021313+0000","last_became_peered":"2024-03-28T22:33:34.021313+0000","last_unstale":"2024-03-28T22:35:30.369177+0000","last_undegraded":"2024-03-28T22:35:30.369177+0000","last_fullsized":"2024-03-28T22:35:30.369177+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:29:42.592761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"92'5","reported_seq":207,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:36:51.439644+0000","last_change":"2024-03-28T22:33:28.449133+0000","last_active":"2024-03-28T22:36:51.439644+0000","last_peered":"2024-03-28T22:36:51.439644+0000","last_clean":"2024-03-28T22:36:51.439644+0000","last_became_active":"2024-03-28T22:33:28.448743+0000","last_became_peered":"2024-03-28T22:33:28.448743+0000","last_unstale":"2024-03-28T22:36:51.439644+0000","last_undegraded":"2024-03-28T22:36:51.439644+0000","last_fullsized":"2024-03-28T22:36:51.439644+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:09:37.119736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":168,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.259332+0000","last_change":"2024-03-28T22:33:28.944198+0000","last_active":"2024-03-28T22:35:30.259332+0000","last_peered":"2024-03-28T22:35:30.259332+0000","last_clean":"2024-03-28T22:35:30.259332+0000","last_became_active":"2024-03-28T22:33:28.943818+0000","last_became_peered":"2024-03-28T22:33:28.943818+0000","last_unstale":"2024-03-28T22:35:30.259332+0000","last_undegraded":"2024-03-28T22:35:30.259332+0000","last_fullsized":"2024-03-28T22:35:30.259332+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:31:03.791364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369583+0000","last_change":"2024-03-28T22:33:34.015499+0000","last_active":"2024-03-28T22:35:30.369583+0000","last_peered":"2024-03-28T22:35:30.369583+0000","last_clean":"2024-03-28T22:35:30.369583+0000","last_became_active":"2024-03-28T22:33:34.015013+0000","last_became_peered":"2024-03-28T22:33:34.015013+0000","last_unstale":"2024-03-28T22:35:30.369583+0000","last_undegraded":"2024-03-28T22:35:30.369583+0000","last_fullsized":"2024-03-28T22:35:30.369583+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:34:30.582198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"94'8","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275296+0000","last_change":"2024-03-28T22:33:28.926732+0000","last_active":"2024-03-28T22:35:30.275296+0000","last_peered":"2024-03-28T22:35:30.275296+0000","last_clean":"2024-03-28T22:35:30.275296+0000","last_became_active":"2024-03-28T22:33:28.926366+0000","last_became_peered":"2024-03-28T22:33:28.926366+0000","last_unstale":"2024-03-28T22:35:30.275296+0000","last_undegraded":"2024-03-28T22:35:30.275296+0000","last_fullsized":"2024-03-28T22:35:30.275296+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:19:10.769141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'6","reported_seq":207,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:36:50.559858+0000","last_change":"2024-03-28T22:33:28.457400+0000","last_active":"2024-03-28T22:36:50.559858+0000","last_peered":"2024-03-28T22:36:50.559858+0000","last_clean":"2024-03-28T22:36:50.559858+0000","last_became_active":"2024-03-28T22:33:28.456994+0000","last_became_peered":"2024-03-28T22:33:28.456994+0000","last_unstale":"2024-03-28T22:36:50.559858+0000","last_undegraded":"2024-03-28T22:36:50.559858+0000","last_fullsized":"2024-03-28T22:36:50.559858+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:32:16.438540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":167,"num_read_kb":167,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"92'1","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248405+0000","last_change":"2024-03-28T22:33:34.027746+0000","last_active":"2024-03-28T22:35:30.248405+0000","last_peered":"2024-03-28T22:35:30.248405+0000","last_clean":"2024-03-28T22:35:30.248405+0000","last_became_active":"2024-03-28T22:33:34.027419+0000","last_became_peered":"2024-03-28T22:33:34.027419+0000","last_unstale":"2024-03-28T22:35:30.248405+0000","last_undegraded":"2024-03-28T22:35:30.248405+0000","last_fullsized":"2024-03-28T22:35:30.248405+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:55:43.686428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371544+0000","last_change":"2024-03-28T22:33:28.952393+0000","last_active":"2024-03-28T22:35:30.371544+0000","last_peered":"2024-03-28T22:35:30.371544+0000","last_clean":"2024-03-28T22:35:30.371544+0000","last_became_active":"2024-03-28T22:33:28.951951+0000","last_became_peered":"2024-03-28T22:33:28.951951+0000","last_unstale":"2024-03-28T22:35:30.371544+0000","last_undegraded":"2024-03-28T22:35:30.371544+0000","last_fullsized":"2024-03-28T22:35:30.371544+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:09:25.808842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"92'6","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370674+0000","last_change":"2024-03-28T22:33:28.437177+0000","last_active":"2024-03-28T22:35:30.370674+0000","last_peered":"2024-03-28T22:35:30.370674+0000","last_clean":"2024-03-28T22:35:30.370674+0000","last_became_active":"2024-03-28T22:33:28.436801+0000","last_became_peered":"2024-03-28T22:33:28.436801+0000","last_unstale":"2024-03-28T22:35:30.370674+0000","last_undegraded":"2024-03-28T22:35:30.370674+0000","last_fullsized":"2024-03-28T22:35:30.370674+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:24:52.549490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":372,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.247989+0000","last_change":"2024-03-28T22:32:07.867336+0000","last_active":"2024-03-28T22:35:30.247989+0000","last_peered":"2024-03-28T22:35:30.247989+0000","last_clean":"2024-03-28T22:35:30.247989+0000","last_became_active":"2024-03-28T22:32:07.866819+0000","last_became_peered":"2024-03-28T22:32:07.866819+0000","last_unstale":"2024-03-28T22:35:30.247989+0000","last_undegraded":"2024-03-28T22:35:30.247989+0000","last_fullsized":"2024-03-28T22:35:30.247989+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:28:14.713501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:28:14.713501+0000","last_clean_scrub_stamp":"2024-03-28T22:28:14.713501+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:00:52.438708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275597+0000","last_change":"2024-03-28T22:33:34.020668+0000","last_active":"2024-03-28T22:35:30.275597+0000","last_peered":"2024-03-28T22:35:30.275597+0000","last_clean":"2024-03-28T22:35:30.275597+0000","last_became_active":"2024-03-28T22:33:34.020240+0000","last_became_peered":"2024-03-28T22:33:34.020240+0000","last_unstale":"2024-03-28T22:35:30.275597+0000","last_undegraded":"2024-03-28T22:35:30.275597+0000","last_fullsized":"2024-03-28T22:35:30.275597+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:18.983983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"94'5","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368061+0000","last_change":"2024-03-28T22:33:28.935237+0000","last_active":"2024-03-28T22:35:30.368061+0000","last_peered":"2024-03-28T22:35:30.368061+0000","last_clean":"2024-03-28T22:35:30.368061+0000","last_became_active":"2024-03-28T22:33:28.934875+0000","last_became_peered":"2024-03-28T22:33:28.934875+0000","last_unstale":"2024-03-28T22:35:30.368061+0000","last_undegraded":"2024-03-28T22:35:30.368061+0000","last_fullsized":"2024-03-28T22:35:30.368061+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:20:50.789302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"92'4","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248187+0000","last_change":"2024-03-28T22:33:28.441654+0000","last_active":"2024-03-28T22:35:30.248187+0000","last_peered":"2024-03-28T22:35:30.248187+0000","last_clean":"2024-03-28T22:35:30.248187+0000","last_became_active":"2024-03-28T22:33:28.441293+0000","last_became_peered":"2024-03-28T22:33:28.441293+0000","last_unstale":"2024-03-28T22:35:30.248187+0000","last_undegraded":"2024-03-28T22:35:30.248187+0000","last_fullsized":"2024-03-28T22:35:30.248187+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:31:17.737977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449486+0000","last_change":"2024-03-28T22:33:34.018126+0000","last_active":"2024-03-28T22:35:30.449486+0000","last_peered":"2024-03-28T22:35:30.449486+0000","last_clean":"2024-03-28T22:35:30.449486+0000","last_became_active":"2024-03-28T22:33:34.017573+0000","last_became_peered":"2024-03-28T22:33:34.017573+0000","last_unstale":"2024-03-28T22:35:30.449486+0000","last_undegraded":"2024-03-28T22:35:30.449486+0000","last_fullsized":"2024-03-28T22:35:30.449486+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:23:31.013511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"92'4","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254108+0000","last_change":"2024-03-28T22:33:28.454069+0000","last_active":"2024-03-28T22:35:30.254108+0000","last_peered":"2024-03-28T22:35:30.254108+0000","last_clean":"2024-03-28T22:35:30.254108+0000","last_became_active":"2024-03-28T22:33:28.453754+0000","last_became_peered":"2024-03-28T22:33:28.453754+0000","last_unstale":"2024-03-28T22:35:30.254108+0000","last_undegraded":"2024-03-28T22:35:30.254108+0000","last_fullsized":"2024-03-28T22:35:30.254108+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:23:33.981072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"94'9","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369387+0000","last_change":"2024-03-28T22:33:28.934390+0000","last_active":"2024-03-28T22:35:30.369387+0000","last_peered":"2024-03-28T22:35:30.369387+0000","last_clean":"2024-03-28T22:35:30.369387+0000","last_became_active":"2024-03-28T22:33:28.933919+0000","last_became_peered":"2024-03-28T22:33:28.933919+0000","last_unstale":"2024-03-28T22:35:30.369387+0000","last_undegraded":"2024-03-28T22:35:30.369387+0000","last_fullsized":"2024-03-28T22:35:30.369387+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:27:07.595144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371312+0000","last_change":"2024-03-28T22:33:34.019910+0000","last_active":"2024-03-28T22:35:30.371312+0000","last_peered":"2024-03-28T22:35:30.371312+0000","last_clean":"2024-03-28T22:35:30.371312+0000","last_became_active":"2024-03-28T22:33:34.019522+0000","last_became_peered":"2024-03-28T22:33:34.019522+0000","last_unstale":"2024-03-28T22:35:30.371312+0000","last_undegraded":"2024-03-28T22:35:30.371312+0000","last_fullsized":"2024-03-28T22:35:30.371312+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:01:29.298513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"96'16","reported_seq":50,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:36:48.496850+0000","last_change":"2024-03-28T22:33:28.444513+0000","last_active":"2024-03-28T22:36:48.496850+0000","last_peered":"2024-03-28T22:36:48.496850+0000","last_clean":"2024-03-28T22:36:48.496850+0000","last_became_active":"2024-03-28T22:33:28.444091+0000","last_became_peered":"2024-03-28T22:33:28.444091+0000","last_unstale":"2024-03-28T22:36:48.496850+0000","last_undegraded":"2024-03-28T22:36:48.496850+0000","last_fullsized":"2024-03-28T22:36:48.496850+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:49:58.331864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"94'5","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.372223+0000","last_change":"2024-03-28T22:33:28.948390+0000","last_active":"2024-03-28T22:35:30.372223+0000","last_peered":"2024-03-28T22:35:30.372223+0000","last_clean":"2024-03-28T22:35:30.372223+0000","last_became_active":"2024-03-28T22:33:28.942846+0000","last_became_peered":"2024-03-28T22:33:28.942846+0000","last_unstale":"2024-03-28T22:35:30.372223+0000","last_undegraded":"2024-03-28T22:35:30.372223+0000","last_fullsized":"2024-03-28T22:35:30.372223+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:50.596365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251712+0000","last_change":"2024-03-28T22:33:33.999429+0000","last_active":"2024-03-28T22:35:30.251712+0000","last_peered":"2024-03-28T22:35:30.251712+0000","last_clean":"2024-03-28T22:35:30.251712+0000","last_became_active":"2024-03-28T22:33:33.998979+0000","last_became_peered":"2024-03-28T22:33:33.998979+0000","last_unstale":"2024-03-28T22:35:30.251712+0000","last_undegraded":"2024-03-28T22:35:30.251712+0000","last_fullsized":"2024-03-28T22:35:30.251712+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:26.450731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"92'7","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370791+0000","last_change":"2024-03-28T22:33:28.442861+0000","last_active":"2024-03-28T22:35:30.370791+0000","last_peered":"2024-03-28T22:35:30.370791+0000","last_clean":"2024-03-28T22:35:30.370791+0000","last_became_active":"2024-03-28T22:33:28.442468+0000","last_became_peered":"2024-03-28T22:33:28.442468+0000","last_unstale":"2024-03-28T22:35:30.370791+0000","last_undegraded":"2024-03-28T22:35:30.370791+0000","last_fullsized":"2024-03-28T22:35:30.370791+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:45:49.899031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"94'7","reported_seq":45,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451985+0000","last_change":"2024-03-28T22:34:30.429872+0000","last_active":"2024-03-28T22:35:30.451985+0000","last_peered":"2024-03-28T22:35:30.451985+0000","last_clean":"2024-03-28T22:35:30.451985+0000","last_became_active":"2024-03-28T22:34:30.429237+0000","last_became_peered":"2024-03-28T22:34:30.429237+0000","last_unstale":"2024-03-28T22:35:30.451985+0000","last_undegraded":"2024-03-28T22:35:30.451985+0000","last_fullsized":"2024-03-28T22:35:30.451985+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:14:46.608826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370565+0000","last_change":"2024-03-28T22:33:34.013826+0000","last_active":"2024-03-28T22:35:30.370565+0000","last_peered":"2024-03-28T22:35:30.370565+0000","last_clean":"2024-03-28T22:35:30.370565+0000","last_became_active":"2024-03-28T22:33:34.013415+0000","last_became_peered":"2024-03-28T22:33:34.013415+0000","last_unstale":"2024-03-28T22:35:30.370565+0000","last_undegraded":"2024-03-28T22:35:30.370565+0000","last_fullsized":"2024-03-28T22:35:30.370565+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:19:30.965940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"92'2","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448399+0000","last_change":"2024-03-28T22:33:28.442225+0000","last_active":"2024-03-28T22:35:30.448399+0000","last_peered":"2024-03-28T22:35:30.448399+0000","last_clean":"2024-03-28T22:35:30.448399+0000","last_became_active":"2024-03-28T22:33:28.441844+0000","last_became_peered":"2024-03-28T22:33:28.441844+0000","last_unstale":"2024-03-28T22:35:30.448399+0000","last_undegraded":"2024-03-28T22:35:30.448399+0000","last_fullsized":"2024-03-28T22:35:30.448399+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:40:52.108663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"94'2","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.253675+0000","last_change":"2024-03-28T22:33:28.927571+0000","last_active":"2024-03-28T22:35:30.253675+0000","last_peered":"2024-03-28T22:35:30.253675+0000","last_clean":"2024-03-28T22:35:30.253675+0000","last_became_active":"2024-03-28T22:33:28.927275+0000","last_became_peered":"2024-03-28T22:33:28.927275+0000","last_unstale":"2024-03-28T22:35:30.253675+0000","last_undegraded":"2024-03-28T22:35:30.253675+0000","last_fullsized":"2024-03-28T22:35:30.253675+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:25:16.184533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.253011+0000","last_change":"2024-03-28T22:33:34.018429+0000","last_active":"2024-03-28T22:35:30.253011+0000","last_peered":"2024-03-28T22:35:30.253011+0000","last_clean":"2024-03-28T22:35:30.253011+0000","last_became_active":"2024-03-28T22:33:34.017066+0000","last_became_peered":"2024-03-28T22:33:34.017066+0000","last_unstale":"2024-03-28T22:35:30.253011+0000","last_undegraded":"2024-03-28T22:35:30.253011+0000","last_fullsized":"2024-03-28T22:35:30.253011+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:35:42.101072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"92'7","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371970+0000","last_change":"2024-03-28T22:33:28.437760+0000","last_active":"2024-03-28T22:35:30.371970+0000","last_peered":"2024-03-28T22:35:30.371970+0000","last_clean":"2024-03-28T22:35:30.371970+0000","last_became_active":"2024-03-28T22:33:28.437096+0000","last_became_peered":"2024-03-28T22:33:28.437096+0000","last_unstale":"2024-03-28T22:35:30.371970+0000","last_undegraded":"2024-03-28T22:35:30.371970+0000","last_fullsized":"2024-03-28T22:35:30.371970+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:29:20.778551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"94'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449358+0000","last_change":"2024-03-28T22:33:28.935805+0000","last_active":"2024-03-28T22:35:30.449358+0000","last_peered":"2024-03-28T22:35:30.449358+0000","last_clean":"2024-03-28T22:35:30.449358+0000","last_became_active":"2024-03-28T22:33:28.935520+0000","last_became_peered":"2024-03-28T22:33:28.935520+0000","last_unstale":"2024-03-28T22:35:30.449358+0000","last_undegraded":"2024-03-28T22:35:30.449358+0000","last_fullsized":"2024-03-28T22:35:30.449358+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:23.414310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"92'1","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370585+0000","last_change":"2024-03-28T22:33:34.029518+0000","last_active":"2024-03-28T22:35:30.370585+0000","last_peered":"2024-03-28T22:35:30.370585+0000","last_clean":"2024-03-28T22:35:30.370585+0000","last_became_active":"2024-03-28T22:33:34.029101+0000","last_became_peered":"2024-03-28T22:33:34.029101+0000","last_unstale":"2024-03-28T22:35:30.370585+0000","last_undegraded":"2024-03-28T22:35:30.370585+0000","last_fullsized":"2024-03-28T22:35:30.370585+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:05.595543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"96'4","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:38.679235+0000","last_change":"2024-03-28T22:33:28.438857+0000","last_active":"2024-03-28T22:35:38.679235+0000","last_peered":"2024-03-28T22:35:38.679235+0000","last_clean":"2024-03-28T22:35:38.679235+0000","last_became_active":"2024-03-28T22:33:28.438528+0000","last_became_peered":"2024-03-28T22:33:28.438528+0000","last_unstale":"2024-03-28T22:35:38.679235+0000","last_undegraded":"2024-03-28T22:35:38.679235+0000","last_fullsized":"2024-03-28T22:35:38.679235+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:07:54.506739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371660+0000","last_change":"2024-03-28T22:33:28.958271+0000","last_active":"2024-03-28T22:35:30.371660+0000","last_peered":"2024-03-28T22:35:30.371660+0000","last_clean":"2024-03-28T22:35:30.371660+0000","last_became_active":"2024-03-28T22:33:28.957922+0000","last_became_peered":"2024-03-28T22:33:28.957922+0000","last_unstale":"2024-03-28T22:35:30.371660+0000","last_undegraded":"2024-03-28T22:35:30.371660+0000","last_fullsized":"2024-03-28T22:35:30.371660+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:33:19.003131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370592+0000","last_change":"2024-03-28T22:33:34.000789+0000","last_active":"2024-03-28T22:35:30.370592+0000","last_peered":"2024-03-28T22:35:30.370592+0000","last_clean":"2024-03-28T22:35:30.370592+0000","last_became_active":"2024-03-28T22:33:34.000383+0000","last_became_peered":"2024-03-28T22:33:34.000383+0000","last_unstale":"2024-03-28T22:35:30.370592+0000","last_undegraded":"2024-03-28T22:35:30.370592+0000","last_fullsized":"2024-03-28T22:35:30.370592+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:50:29.925656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"92'4","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.253813+0000","last_change":"2024-03-28T22:33:28.439121+0000","last_active":"2024-03-28T22:35:30.253813+0000","last_peered":"2024-03-28T22:35:30.253813+0000","last_clean":"2024-03-28T22:35:30.253813+0000","last_became_active":"2024-03-28T22:33:28.438800+0000","last_became_peered":"2024-03-28T22:33:28.438800+0000","last_unstale":"2024-03-28T22:35:30.253813+0000","last_undegraded":"2024-03-28T22:35:30.253813+0000","last_fullsized":"2024-03-28T22:35:30.253813+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:57:10.013246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"94'5","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248157+0000","last_change":"2024-03-28T22:33:28.934499+0000","last_active":"2024-03-28T22:35:30.248157+0000","last_peered":"2024-03-28T22:35:30.248157+0000","last_clean":"2024-03-28T22:35:30.248157+0000","last_became_active":"2024-03-28T22:33:28.934147+0000","last_became_peered":"2024-03-28T22:33:28.934147+0000","last_unstale":"2024-03-28T22:35:30.248157+0000","last_undegraded":"2024-03-28T22:35:30.248157+0000","last_fullsized":"2024-03-28T22:35:30.248157+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:35:39.090052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448402+0000","last_change":"2024-03-28T22:33:34.017946+0000","last_active":"2024-03-28T22:35:30.448402+0000","last_peered":"2024-03-28T22:35:30.448402+0000","last_clean":"2024-03-28T22:35:30.448402+0000","last_became_active":"2024-03-28T22:33:34.017546+0000","last_became_peered":"2024-03-28T22:33:34.017546+0000","last_unstale":"2024-03-28T22:35:30.448402+0000","last_undegraded":"2024-03-28T22:35:30.448402+0000","last_fullsized":"2024-03-28T22:35:30.448402+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:14:39.081677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"96'6","reported_seq":211,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:36:52.356927+0000","last_change":"2024-03-28T22:33:28.453387+0000","last_active":"2024-03-28T22:36:52.356927+0000","last_peered":"2024-03-28T22:36:52.356927+0000","last_clean":"2024-03-28T22:36:52.356927+0000","last_became_active":"2024-03-28T22:33:28.453029+0000","last_became_peered":"2024-03-28T22:33:28.453029+0000","last_unstale":"2024-03-28T22:36:52.356927+0000","last_undegraded":"2024-03-28T22:36:52.356927+0000","last_fullsized":"2024-03-28T22:36:52.356927+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:12:33.246863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":171,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248615+0000","last_change":"2024-03-28T22:33:28.923819+0000","last_active":"2024-03-28T22:35:30.248615+0000","last_peered":"2024-03-28T22:35:30.248615+0000","last_clean":"2024-03-28T22:35:30.248615+0000","last_became_active":"2024-03-28T22:33:28.923471+0000","last_became_peered":"2024-03-28T22:33:28.923471+0000","last_unstale":"2024-03-28T22:35:30.248615+0000","last_undegraded":"2024-03-28T22:35:30.248615+0000","last_fullsized":"2024-03-28T22:35:30.248615+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:58:03.496406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451029+0000","last_change":"2024-03-28T22:33:34.029789+0000","last_active":"2024-03-28T22:35:30.451029+0000","last_peered":"2024-03-28T22:35:30.451029+0000","last_clean":"2024-03-28T22:35:30.451029+0000","last_became_active":"2024-03-28T22:33:34.029231+0000","last_became_peered":"2024-03-28T22:33:34.029231+0000","last_unstale":"2024-03-28T22:35:30.451029+0000","last_undegraded":"2024-03-28T22:35:30.451029+0000","last_fullsized":"2024-03-28T22:35:30.451029+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:47:50.101393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"92'3","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275541+0000","last_change":"2024-03-28T22:33:28.435668+0000","last_active":"2024-03-28T22:35:30.275541+0000","last_peered":"2024-03-28T22:35:30.275541+0000","last_clean":"2024-03-28T22:35:30.275541+0000","last_became_active":"2024-03-28T22:33:28.435325+0000","last_became_peered":"2024-03-28T22:33:28.435325+0000","last_unstale":"2024-03-28T22:35:30.275541+0000","last_undegraded":"2024-03-28T22:35:30.275541+0000","last_fullsized":"2024-03-28T22:35:30.275541+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:13:06.281074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"94'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248244+0000","last_change":"2024-03-28T22:33:28.954138+0000","last_active":"2024-03-28T22:35:30.248244+0000","last_peered":"2024-03-28T22:35:30.248244+0000","last_clean":"2024-03-28T22:35:30.248244+0000","last_became_active":"2024-03-28T22:33:28.953752+0000","last_became_peered":"2024-03-28T22:33:28.953752+0000","last_unstale":"2024-03-28T22:35:30.248244+0000","last_undegraded":"2024-03-28T22:35:30.248244+0000","last_fullsized":"2024-03-28T22:35:30.248244+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:53:34.892316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.274983+0000","last_change":"2024-03-28T22:33:34.002672+0000","last_active":"2024-03-28T22:35:30.274983+0000","last_peered":"2024-03-28T22:35:30.274983+0000","last_clean":"2024-03-28T22:35:30.274983+0000","last_became_active":"2024-03-28T22:33:34.002195+0000","last_became_peered":"2024-03-28T22:33:34.002195+0000","last_unstale":"2024-03-28T22:35:30.274983+0000","last_undegraded":"2024-03-28T22:35:30.274983+0000","last_fullsized":"2024-03-28T22:35:30.274983+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:24:11.521474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"92'5","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369212+0000","last_change":"2024-03-28T22:33:28.450818+0000","last_active":"2024-03-28T22:35:30.369212+0000","last_peered":"2024-03-28T22:35:30.369212+0000","last_clean":"2024-03-28T22:35:30.369212+0000","last_became_active":"2024-03-28T22:33:28.448592+0000","last_became_peered":"2024-03-28T22:33:28.448592+0000","last_unstale":"2024-03-28T22:35:30.369212+0000","last_undegraded":"2024-03-28T22:35:30.369212+0000","last_fullsized":"2024-03-28T22:35:30.369212+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:19:27.126306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"94'9","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248488+0000","last_change":"2024-03-28T22:33:28.925191+0000","last_active":"2024-03-28T22:35:30.248488+0000","last_peered":"2024-03-28T22:35:30.248488+0000","last_clean":"2024-03-28T22:35:30.248488+0000","last_became_active":"2024-03-28T22:33:28.924424+0000","last_became_peered":"2024-03-28T22:33:28.924424+0000","last_unstale":"2024-03-28T22:35:30.248488+0000","last_undegraded":"2024-03-28T22:35:30.248488+0000","last_fullsized":"2024-03-28T22:35:30.248488+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:50:00.415047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"94'9","reported_seq":46,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449424+0000","last_change":"2024-03-28T22:34:29.987176+0000","last_active":"2024-03-28T22:35:30.449424+0000","last_peered":"2024-03-28T22:35:30.449424+0000","last_clean":"2024-03-28T22:35:30.449424+0000","last_became_active":"2024-03-28T22:34:29.986807+0000","last_became_peered":"2024-03-28T22:34:29.986807+0000","last_unstale":"2024-03-28T22:35:30.449424+0000","last_undegraded":"2024-03-28T22:35:30.449424+0000","last_fullsized":"2024-03-28T22:35:30.449424+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:00:06.774521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"94'12","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251938+0000","last_change":"2024-03-28T22:33:28.927068+0000","last_active":"2024-03-28T22:35:30.251938+0000","last_peered":"2024-03-28T22:35:30.251938+0000","last_clean":"2024-03-28T22:35:30.251938+0000","last_became_active":"2024-03-28T22:33:28.926734+0000","last_became_peered":"2024-03-28T22:33:28.926734+0000","last_unstale":"2024-03-28T22:35:30.251938+0000","last_undegraded":"2024-03-28T22:35:30.251938+0000","last_fullsized":"2024-03-28T22:35:30.251938+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:56.309555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"94'8","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251930+0000","last_change":"2024-03-28T22:33:28.955675+0000","last_active":"2024-03-28T22:35:30.251930+0000","last_peered":"2024-03-28T22:35:30.251930+0000","last_clean":"2024-03-28T22:35:30.251930+0000","last_became_active":"2024-03-28T22:33:28.955329+0000","last_became_peered":"2024-03-28T22:33:28.955329+0000","last_unstale":"2024-03-28T22:35:30.251930+0000","last_undegraded":"2024-03-28T22:35:30.251930+0000","last_fullsized":"2024-03-28T22:35:30.251930+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:34:14.166434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.452477+0000","last_change":"2024-03-28T22:33:28.955125+0000","last_active":"2024-03-28T22:35:30.452477+0000","last_peered":"2024-03-28T22:35:30.452477+0000","last_clean":"2024-03-28T22:35:30.452477+0000","last_became_active":"2024-03-28T22:33:28.954434+0000","last_became_peered":"2024-03-28T22:33:28.954434+0000","last_unstale":"2024-03-28T22:35:30.452477+0000","last_undegraded":"2024-03-28T22:35:30.452477+0000","last_fullsized":"2024-03-28T22:35:30.452477+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:56:50.705012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"94'4","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369433+0000","last_change":"2024-03-28T22:33:28.945230+0000","last_active":"2024-03-28T22:35:30.369433+0000","last_peered":"2024-03-28T22:35:30.369433+0000","last_clean":"2024-03-28T22:35:30.369433+0000","last_became_active":"2024-03-28T22:33:28.944921+0000","last_became_peered":"2024-03-28T22:33:28.944921+0000","last_unstale":"2024-03-28T22:35:30.369433+0000","last_undegraded":"2024-03-28T22:35:30.369433+0000","last_fullsized":"2024-03-28T22:35:30.369433+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:26:26.505774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"94'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449741+0000","last_change":"2024-03-28T22:33:28.928720+0000","last_active":"2024-03-28T22:35:30.449741+0000","last_peered":"2024-03-28T22:35:30.449741+0000","last_clean":"2024-03-28T22:35:30.449741+0000","last_became_active":"2024-03-28T22:33:28.928415+0000","last_became_peered":"2024-03-28T22:33:28.928415+0000","last_unstale":"2024-03-28T22:35:30.449741+0000","last_undegraded":"2024-03-28T22:35:30.449741+0000","last_fullsized":"2024-03-28T22:35:30.449741+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:00.839609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"94'15","reported_seq":47,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252418+0000","last_change":"2024-03-28T22:33:28.924948+0000","last_active":"2024-03-28T22:35:30.252418+0000","last_peered":"2024-03-28T22:35:30.252418+0000","last_clean":"2024-03-28T22:35:30.252418+0000","last_became_active":"2024-03-28T22:33:28.924631+0000","last_became_peered":"2024-03-28T22:33:28.924631+0000","last_unstale":"2024-03-28T22:35:30.252418+0000","last_undegraded":"2024-03-28T22:35:30.252418+0000","last_fullsized":"2024-03-28T22:35:30.252418+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:53:44.909991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"94'2","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.372408+0000","last_change":"2024-03-28T22:33:28.948725+0000","last_active":"2024-03-28T22:35:30.372408+0000","last_peered":"2024-03-28T22:35:30.372408+0000","last_clean":"2024-03-28T22:35:30.372408+0000","last_became_active":"2024-03-28T22:33:28.943942+0000","last_became_peered":"2024-03-28T22:33:28.943942+0000","last_unstale":"2024-03-28T22:35:30.372408+0000","last_undegraded":"2024-03-28T22:35:30.372408+0000","last_fullsized":"2024-03-28T22:35:30.372408+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:50:04.238538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"94'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275158+0000","last_change":"2024-03-28T22:33:28.941546+0000","last_active":"2024-03-28T22:35:30.275158+0000","last_peered":"2024-03-28T22:35:30.275158+0000","last_clean":"2024-03-28T22:35:30.275158+0000","last_became_active":"2024-03-28T22:33:28.941278+0000","last_became_peered":"2024-03-28T22:33:28.941278+0000","last_unstale":"2024-03-28T22:35:30.275158+0000","last_undegraded":"2024-03-28T22:35:30.275158+0000","last_fullsized":"2024-03-28T22:35:30.275158+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:47:10.047067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.372264+0000","last_change":"2024-03-28T22:33:28.949552+0000","last_active":"2024-03-28T22:35:30.372264+0000","last_peered":"2024-03-28T22:35:30.372264+0000","last_clean":"2024-03-28T22:35:30.372264+0000","last_became_active":"2024-03-28T22:33:28.947432+0000","last_became_peered":"2024-03-28T22:33:28.947432+0000","last_unstale":"2024-03-28T22:35:30.372264+0000","last_undegraded":"2024-03-28T22:35:30.372264+0000","last_fullsized":"2024-03-28T22:35:30.372264+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:01:16.534642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"94'10","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448510+0000","last_change":"2024-03-28T22:33:28.953930+0000","last_active":"2024-03-28T22:35:30.448510+0000","last_peered":"2024-03-28T22:35:30.448510+0000","last_clean":"2024-03-28T22:35:30.448510+0000","last_became_active":"2024-03-28T22:33:28.953591+0000","last_became_peered":"2024-03-28T22:33:28.953591+0000","last_unstale":"2024-03-28T22:35:30.448510+0000","last_undegraded":"2024-03-28T22:35:30.448510+0000","last_fullsized":"2024-03-28T22:35:30.448510+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:42:55.857476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"94'11","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368801+0000","last_change":"2024-03-28T22:33:28.946307+0000","last_active":"2024-03-28T22:35:30.368801+0000","last_peered":"2024-03-28T22:35:30.368801+0000","last_clean":"2024-03-28T22:35:30.368801+0000","last_became_active":"2024-03-28T22:33:28.945932+0000","last_became_peered":"2024-03-28T22:33:28.945932+0000","last_unstale":"2024-03-28T22:35:30.368801+0000","last_undegraded":"2024-03-28T22:35:30.368801+0000","last_fullsized":"2024-03-28T22:35:30.368801+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:54:59.651668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449676+0000","last_change":"2024-03-28T22:33:28.928014+0000","last_active":"2024-03-28T22:35:30.449676+0000","last_peered":"2024-03-28T22:35:30.449676+0000","last_clean":"2024-03-28T22:35:30.449676+0000","last_became_active":"2024-03-28T22:33:28.927686+0000","last_became_peered":"2024-03-28T22:33:28.927686+0000","last_unstale":"2024-03-28T22:35:30.449676+0000","last_undegraded":"2024-03-28T22:35:30.449676+0000","last_fullsized":"2024-03-28T22:35:30.449676+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:47:08.131565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":81087,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1717,"num_read_kb":948,"num_write":265,"num_write_kb":165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":602112,"data_stored":243261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":229,"ondisk_log_size":229,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318327,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32260,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712892,"statfs":{"total":95995035648,"available":95962001408,"internally_reserved":0,"allocated":1105920,"data_stored":539947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579964,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28780,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716372,"statfs":{"total":95995035648,"available":95965564928,"internally_reserved":0,"allocated":1806336,"data_stored":1199927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743490,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1130496,"data_stored":543560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005128,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1118208,"data_stored":539820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299470,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37016,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708136,"statfs":{"total":95995035648,"available":95957131264,"internally_reserved":0,"allocated":1720320,"data_stored":1137177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495699,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1101824,"data_stored":534968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574959,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36488,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708664,"statfs":{"total":95995035648,"available":95957671936,"internally_reserved":0,"allocated":1179648,"data_stored":609043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738484,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1126400,"data_stored":536815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411436,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36508,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708644,"statfs":{"total":95995035648,"available":95957651456,"internally_reserved":0,"allocated":1200128,"data_stored":610351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182499,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36424,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708728,"statfs":{"total":95995035648,"available":95957737472,"internally_reserved":0,"allocated":1114112,"data_stored":534676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920862,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1110016,"data_stored":539574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757337,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28704,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716448,"statfs":{"total":95995035648,"available":95965642752,"internally_reserved":0,"allocated":1728512,"data_stored":1132227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":74797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":76105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":75313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T22:36:56.699 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-28T22:36:56.699 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T22:36:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.967+0000 7fa295a00640 1 -- 172.21.15.43:0/3101574912 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa2901049a0 msgr2=0x7fa290104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.967+0000 7fa295a00640 1 --2- 172.21.15.43:0/3101574912 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa2901049a0 0x7fa290104e20 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa27c009900 tx=0x7fa27c02bc60 comp rx=0 tx=0).stop 2024-03-28T22:36:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.967+0000 7fa295a00640 1 -- 172.21.15.43:0/3101574912 shutdown_connections 2024-03-28T22:36:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.967+0000 7fa295a00640 1 --2- 172.21.15.43:0/3101574912 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa2901049a0 0x7fa290104e20 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.967+0000 7fa295a00640 1 --2- 172.21.15.43:0/3101574912 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa290105360 0x7fa29010bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.967+0000 7fa295a00640 1 --2- 172.21.15.43:0/3101574912 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2901037b0 0x7fa290103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.967+0000 7fa295a00640 1 -- 172.21.15.43:0/3101574912 >> 172.21.15.43:0/3101574912 conn(0x7fa2900fed20 msgr2=0x7fa290101160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.967+0000 7fa295a00640 1 -- 172.21.15.43:0/3101574912 shutdown_connections 2024-03-28T22:36:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.967+0000 7fa295a00640 1 -- 172.21.15.43:0/3101574912 wait complete. 2024-03-28T22:36:56.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.968+0000 7fa295a00640 1 Processor -- start 2024-03-28T22:36:56.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.969+0000 7fa295a00640 1 -- start start 2024-03-28T22:36:56.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.969+0000 7fa295a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa2901037b0 0x7fa290199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:56.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.969+0000 7fa295a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2901049a0 0x7fa290199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:56.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.969+0000 7fa295a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa290105360 0x7fa2901a1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:56.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.969+0000 7fa295a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa29010f460 con 0x7fa290105360 2024-03-28T22:36:56.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.969+0000 7fa295a00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa29010f2e0 con 0x7fa2901037b0 2024-03-28T22:36:56.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.969+0000 7fa295a00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fa29010f5e0 con 0x7fa2901049a0 2024-03-28T22:36:56.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.969+0000 7fa28fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa2901037b0 0x7fa290199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:56.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.969+0000 7fa28f400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2901049a0 0x7fa290199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:56.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.969+0000 7fa28fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa2901037b0 0x7fa290199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:37394/0 (socket says 172.21.15.43:37394) 2024-03-28T22:36:56.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.969+0000 7fa28fe00640 1 -- 172.21.15.43:0/2539573764 learned_addr learned my addr 172.21.15.43:0/2539573764 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:56.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.969+0000 7fa295000640 1 --2- 172.21.15.43:0/2539573764 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa290105360 0x7fa2901a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:56.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.970+0000 7fa28fe00640 1 -- 172.21.15.43:0/2539573764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2901049a0 msgr2=0x7fa290199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:56.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.970+0000 7fa28fe00640 1 --2- 172.21.15.43:0/2539573764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2901049a0 0x7fa290199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:56.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.970+0000 7fa28fe00640 1 -- 172.21.15.43:0/2539573764 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa290105360 msgr2=0x7fa2901a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:56.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.970+0000 7fa28fe00640 1 --2- 172.21.15.43:0/2539573764 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa290105360 0x7fa2901a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:56.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.970+0000 7fa28fe00640 1 -- 172.21.15.43:0/2539573764 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2901a16c0 con 0x7fa2901037b0 2024-03-28T22:36:56.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.970+0000 7fa28fe00640 1 --2- 172.21.15.43:0/2539573764 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa2901037b0 0x7fa290199aa0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa28000e9c0 tx=0x7fa28000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:56.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.970+0000 7fa28cc00640 1 -- 172.21.15.43:0/2539573764 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa28000cd80 con 0x7fa2901037b0 2024-03-28T22:36:56.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.970+0000 7fa28cc00640 1 -- 172.21.15.43:0/2539573764 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2800035f0 con 0x7fa2901037b0 2024-03-28T22:36:56.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.970+0000 7fa295a00640 1 -- 172.21.15.43:0/2539573764 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2901a19b0 con 0x7fa2901037b0 2024-03-28T22:36:56.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.970+0000 7fa28cc00640 1 -- 172.21.15.43:0/2539573764 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa280010690 con 0x7fa2901037b0 2024-03-28T22:36:56.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.970+0000 7fa295a00640 1 -- 172.21.15.43:0/2539573764 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2901a1ed0 con 0x7fa2901037b0 2024-03-28T22:36:56.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.972+0000 7fa295a00640 1 -- 172.21.15.43:0/2539573764 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa29010aad0 con 0x7fa2901037b0 2024-03-28T22:36:56.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.973+0000 7fa28cc00640 1 -- 172.21.15.43:0/2539573764 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7fa2800037a0 con 0x7fa2901037b0 2024-03-28T22:36:56.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.975+0000 7fa28cc00640 1 --2- 172.21.15.43:0/2539573764 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa2740a1e10 0x7fa2740a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:56.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.975+0000 7fa28cc00640 1 -- 172.21.15.43:0/2539573764 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7fa2800ce760 con 0x7fa2901037b0 2024-03-28T22:36:56.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.975+0000 7fa28f400640 1 --2- 172.21.15.43:0/2539573764 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa2740a1e10 0x7fa2740a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:56.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.976+0000 7fa28f400640 1 --2- 172.21.15.43:0/2539573764 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa2740a1e10 0x7fa2740a42d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa27c02c270 tx=0x7fa27c02c890 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:56.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:56.983+0000 7fa28cc00640 1 -- 172.21.15.43:0/2539573764 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa28009b640 con 0x7fa2901037b0 2024-03-28T22:36:57.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.183+0000 7fa295a00640 1 -- 172.21.15.43:0/2539573764 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa290061b90 con 0x7fa2740a1e10 2024-03-28T22:36:57.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.193+0000 7fa28cc00640 1 -- 172.21.15.43:0/2539573764 <== mgr.14152 v2:172.21.15.43:6800/369798199 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340530 (secure 0 0 0) 0x7fa290061b90 con 0x7fa2740a1e10 2024-03-28T22:36:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:36:57.200 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-03-28T22:36:57.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.201+0000 7fa295a00640 1 -- 172.21.15.43:0/2539573764 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa2740a1e10 msgr2=0x7fa2740a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:57.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.201+0000 7fa295a00640 1 --2- 172.21.15.43:0/2539573764 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa2740a1e10 0x7fa2740a42d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa27c02c270 tx=0x7fa27c02c890 comp rx=0 tx=0).stop 2024-03-28T22:36:57.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.202+0000 7fa295a00640 1 -- 172.21.15.43:0/2539573764 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa2901037b0 msgr2=0x7fa290199aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:57.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.202+0000 7fa295a00640 1 --2- 172.21.15.43:0/2539573764 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa2901037b0 0x7fa290199aa0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa28000e9c0 tx=0x7fa28000ee80 comp rx=0 tx=0).stop 2024-03-28T22:36:57.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.202+0000 7fa295a00640 1 -- 172.21.15.43:0/2539573764 shutdown_connections 2024-03-28T22:36:57.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.202+0000 7fa295a00640 1 --2- 172.21.15.43:0/2539573764 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa290105360 0x7fa2901a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:57.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.202+0000 7fa295a00640 1 --2- 172.21.15.43:0/2539573764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2901049a0 0x7fa290199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:57.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.202+0000 7fa295a00640 1 --2- 172.21.15.43:0/2539573764 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fa2740a1e10 0x7fa2740a42d0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:57.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.202+0000 7fa295a00640 1 --2- 172.21.15.43:0/2539573764 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa2901037b0 0x7fa290199aa0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:57.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.202+0000 7fa295a00640 1 -- 172.21.15.43:0/2539573764 >> 172.21.15.43:0/2539573764 conn(0x7fa2900fed20 msgr2=0x7fa290107ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:57.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.202+0000 7fa295a00640 1 -- 172.21.15.43:0/2539573764 shutdown_connections 2024-03-28T22:36:57.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.202+0000 7fa295a00640 1 -- 172.21.15.43:0/2539573764 wait complete. 2024-03-28T22:36:57.217 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:56 smithi043 ceph-mon[31823]: from='client.25553 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:36:57.221 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":430,"stamp":"2024-03-28T22:36:56.245667+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":671689,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1853,"num_read_kb":1067,"num_write":448,"num_write_kb":3105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":868,"ondisk_log_size":868,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":376740,"kb_used_data":15080,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124565084,"statfs":{"total":1151940427776,"available":1151554646016,"internally_reserved":0,"allocated":15441920,"data_stored":8458085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9678,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":126,"num_read_kb":78,"num_write":15,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005234"},"pg_stats":[{"pgid":"3.3f","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.450073+0000","last_change":"2024-03-28T22:33:28.942904+0000","last_active":"2024-03-28T22:35:30.450073+0000","last_peered":"2024-03-28T22:35:30.450073+0000","last_clean":"2024-03-28T22:35:30.450073+0000","last_became_active":"2024-03-28T22:33:28.942574+0000","last_became_peered":"2024-03-28T22:33:28.942574+0000","last_unstale":"2024-03-28T22:35:30.450073+0000","last_undegraded":"2024-03-28T22:35:30.450073+0000","last_fullsized":"2024-03-28T22:35:30.450073+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:55.323919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"94'4","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451519+0000","last_change":"2024-03-28T22:33:28.951795+0000","last_active":"2024-03-28T22:35:30.451519+0000","last_peered":"2024-03-28T22:35:30.451519+0000","last_clean":"2024-03-28T22:35:30.451519+0000","last_became_active":"2024-03-28T22:33:28.951286+0000","last_became_peered":"2024-03-28T22:33:28.951286+0000","last_unstale":"2024-03-28T22:35:30.451519+0000","last_undegraded":"2024-03-28T22:35:30.451519+0000","last_fullsized":"2024-03-28T22:35:30.451519+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:56:11.718244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451291+0000","last_change":"2024-03-28T22:33:28.949476+0000","last_active":"2024-03-28T22:35:30.451291+0000","last_peered":"2024-03-28T22:35:30.451291+0000","last_clean":"2024-03-28T22:35:30.451291+0000","last_became_active":"2024-03-28T22:33:28.949128+0000","last_became_peered":"2024-03-28T22:33:28.949128+0000","last_unstale":"2024-03-28T22:35:30.451291+0000","last_undegraded":"2024-03-28T22:35:30.451291+0000","last_fullsized":"2024-03-28T22:35:30.451291+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:49:29.052373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370703+0000","last_change":"2024-03-28T22:33:28.930715+0000","last_active":"2024-03-28T22:35:30.370703+0000","last_peered":"2024-03-28T22:35:30.370703+0000","last_clean":"2024-03-28T22:35:30.370703+0000","last_became_active":"2024-03-28T22:33:28.930265+0000","last_became_peered":"2024-03-28T22:33:28.930265+0000","last_unstale":"2024-03-28T22:35:30.370703+0000","last_undegraded":"2024-03-28T22:35:30.370703+0000","last_fullsized":"2024-03-28T22:35:30.370703+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:29:54.695671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"94'4","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.253685+0000","last_change":"2024-03-28T22:33:28.944933+0000","last_active":"2024-03-28T22:35:30.253685+0000","last_peered":"2024-03-28T22:35:30.253685+0000","last_clean":"2024-03-28T22:35:30.253685+0000","last_became_active":"2024-03-28T22:33:28.944542+0000","last_became_peered":"2024-03-28T22:33:28.944542+0000","last_unstale":"2024-03-28T22:35:30.253685+0000","last_undegraded":"2024-03-28T22:35:30.253685+0000","last_fullsized":"2024-03-28T22:35:30.253685+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:35:15.719312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"94'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.259295+0000","last_change":"2024-03-28T22:33:28.927920+0000","last_active":"2024-03-28T22:35:30.259295+0000","last_peered":"2024-03-28T22:35:30.259295+0000","last_clean":"2024-03-28T22:35:30.259295+0000","last_became_active":"2024-03-28T22:33:28.927602+0000","last_became_peered":"2024-03-28T22:33:28.927602+0000","last_unstale":"2024-03-28T22:35:30.259295+0000","last_undegraded":"2024-03-28T22:35:30.259295+0000","last_fullsized":"2024-03-28T22:35:30.259295+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:24:22.072707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"94'5","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248059+0000","last_change":"2024-03-28T22:33:28.917963+0000","last_active":"2024-03-28T22:35:30.248059+0000","last_peered":"2024-03-28T22:35:30.248059+0000","last_clean":"2024-03-28T22:35:30.248059+0000","last_became_active":"2024-03-28T22:33:28.917628+0000","last_became_peered":"2024-03-28T22:33:28.917628+0000","last_unstale":"2024-03-28T22:35:30.248059+0000","last_undegraded":"2024-03-28T22:35:30.248059+0000","last_fullsized":"2024-03-28T22:35:30.248059+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:37:09.111640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"94'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371177+0000","last_change":"2024-03-28T22:33:28.942632+0000","last_active":"2024-03-28T22:35:30.371177+0000","last_peered":"2024-03-28T22:35:30.371177+0000","last_clean":"2024-03-28T22:35:30.371177+0000","last_became_active":"2024-03-28T22:33:28.941972+0000","last_became_peered":"2024-03-28T22:33:28.941972+0000","last_unstale":"2024-03-28T22:35:30.371177+0000","last_undegraded":"2024-03-28T22:35:30.371177+0000","last_fullsized":"2024-03-28T22:35:30.371177+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:49:52.642314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"94'11","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448774+0000","last_change":"2024-03-28T22:33:28.924324+0000","last_active":"2024-03-28T22:35:30.448774+0000","last_peered":"2024-03-28T22:35:30.448774+0000","last_clean":"2024-03-28T22:35:30.448774+0000","last_became_active":"2024-03-28T22:33:28.924014+0000","last_became_peered":"2024-03-28T22:33:28.924014+0000","last_unstale":"2024-03-28T22:35:30.448774+0000","last_undegraded":"2024-03-28T22:35:30.448774+0000","last_fullsized":"2024-03-28T22:35:30.448774+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:17:55.172116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"94'16","reported_seq":54,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.259584+0000","last_change":"2024-03-28T22:34:29.987262+0000","last_active":"2024-03-28T22:35:30.259584+0000","last_peered":"2024-03-28T22:35:30.259584+0000","last_clean":"2024-03-28T22:35:30.259584+0000","last_became_active":"2024-03-28T22:34:29.986934+0000","last_became_peered":"2024-03-28T22:34:29.986934+0000","last_unstale":"2024-03-28T22:35:30.259584+0000","last_undegraded":"2024-03-28T22:35:30.259584+0000","last_fullsized":"2024-03-28T22:35:30.259584+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:00:41.574800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"94'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248516+0000","last_change":"2024-03-28T22:33:28.930802+0000","last_active":"2024-03-28T22:35:30.248516+0000","last_peered":"2024-03-28T22:35:30.248516+0000","last_clean":"2024-03-28T22:35:30.248516+0000","last_became_active":"2024-03-28T22:33:28.930475+0000","last_became_peered":"2024-03-28T22:33:28.930475+0000","last_unstale":"2024-03-28T22:35:30.248516+0000","last_undegraded":"2024-03-28T22:35:30.248516+0000","last_fullsized":"2024-03-28T22:35:30.248516+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:48.443803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"94'7","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370695+0000","last_change":"2024-03-28T22:33:28.927057+0000","last_active":"2024-03-28T22:35:30.370695+0000","last_peered":"2024-03-28T22:35:30.370695+0000","last_clean":"2024-03-28T22:35:30.370695+0000","last_became_active":"2024-03-28T22:33:28.926598+0000","last_became_peered":"2024-03-28T22:33:28.926598+0000","last_unstale":"2024-03-28T22:35:30.370695+0000","last_undegraded":"2024-03-28T22:35:30.370695+0000","last_fullsized":"2024-03-28T22:35:30.370695+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:21:00.710203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"94'10","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368918+0000","last_change":"2024-03-28T22:33:28.937034+0000","last_active":"2024-03-28T22:35:30.368918+0000","last_peered":"2024-03-28T22:35:30.368918+0000","last_clean":"2024-03-28T22:35:30.368918+0000","last_became_active":"2024-03-28T22:33:28.933713+0000","last_became_peered":"2024-03-28T22:33:28.933713+0000","last_unstale":"2024-03-28T22:35:30.368918+0000","last_undegraded":"2024-03-28T22:35:30.368918+0000","last_fullsized":"2024-03-28T22:35:30.368918+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:58:24.148897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.453212+0000","last_change":"2024-03-28T22:33:28.947384+0000","last_active":"2024-03-28T22:35:30.453212+0000","last_peered":"2024-03-28T22:35:30.453212+0000","last_clean":"2024-03-28T22:35:30.453212+0000","last_became_active":"2024-03-28T22:33:28.947017+0000","last_became_peered":"2024-03-28T22:33:28.947017+0000","last_unstale":"2024-03-28T22:35:30.453212+0000","last_undegraded":"2024-03-28T22:35:30.453212+0000","last_fullsized":"2024-03-28T22:35:30.453212+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:43:32.781161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451880+0000","last_change":"2024-03-28T22:33:28.950119+0000","last_active":"2024-03-28T22:35:30.451880+0000","last_peered":"2024-03-28T22:35:30.451880+0000","last_clean":"2024-03-28T22:35:30.451880+0000","last_became_active":"2024-03-28T22:33:28.949798+0000","last_became_peered":"2024-03-28T22:33:28.949798+0000","last_unstale":"2024-03-28T22:35:30.451880+0000","last_undegraded":"2024-03-28T22:35:30.451880+0000","last_fullsized":"2024-03-28T22:35:30.451880+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:16.717678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"94'9","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448318+0000","last_change":"2024-03-28T22:33:28.943255+0000","last_active":"2024-03-28T22:35:30.448318+0000","last_peered":"2024-03-28T22:35:30.448318+0000","last_clean":"2024-03-28T22:35:30.448318+0000","last_became_active":"2024-03-28T22:33:28.942923+0000","last_became_peered":"2024-03-28T22:33:28.942923+0000","last_unstale":"2024-03-28T22:35:30.448318+0000","last_undegraded":"2024-03-28T22:35:30.448318+0000","last_fullsized":"2024-03-28T22:35:30.448318+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:05:04.197635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449964+0000","last_change":"2024-03-28T22:33:28.942906+0000","last_active":"2024-03-28T22:35:30.449964+0000","last_peered":"2024-03-28T22:35:30.449964+0000","last_clean":"2024-03-28T22:35:30.449964+0000","last_became_active":"2024-03-28T22:33:28.942573+0000","last_became_peered":"2024-03-28T22:33:28.942573+0000","last_unstale":"2024-03-28T22:35:30.449964+0000","last_undegraded":"2024-03-28T22:35:30.449964+0000","last_fullsized":"2024-03-28T22:35:30.449964+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:39:17.651101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"94'10","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252948+0000","last_change":"2024-03-28T22:33:28.932075+0000","last_active":"2024-03-28T22:35:30.252948+0000","last_peered":"2024-03-28T22:35:30.252948+0000","last_clean":"2024-03-28T22:35:30.252948+0000","last_became_active":"2024-03-28T22:33:28.931689+0000","last_became_peered":"2024-03-28T22:33:28.931689+0000","last_unstale":"2024-03-28T22:35:30.252948+0000","last_undegraded":"2024-03-28T22:35:30.252948+0000","last_fullsized":"2024-03-28T22:35:30.252948+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:50:43.031082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252343+0000","last_change":"2024-03-28T22:33:28.944987+0000","last_active":"2024-03-28T22:35:30.252343+0000","last_peered":"2024-03-28T22:35:30.252343+0000","last_clean":"2024-03-28T22:35:30.252343+0000","last_became_active":"2024-03-28T22:33:28.944622+0000","last_became_peered":"2024-03-28T22:33:28.944622+0000","last_unstale":"2024-03-28T22:35:30.252343+0000","last_undegraded":"2024-03-28T22:35:30.252343+0000","last_fullsized":"2024-03-28T22:35:30.252343+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:48:58.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":30,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251826+0000","last_change":"2024-03-28T22:34:30.429709+0000","last_active":"2024-03-28T22:35:30.251826+0000","last_peered":"2024-03-28T22:35:30.251826+0000","last_clean":"2024-03-28T22:35:30.251826+0000","last_became_active":"2024-03-28T22:34:30.429122+0000","last_became_peered":"2024-03-28T22:34:30.429122+0000","last_unstale":"2024-03-28T22:35:30.251826+0000","last_undegraded":"2024-03-28T22:35:30.251826+0000","last_fullsized":"2024-03-28T22:35:30.251826+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:58:05.313669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"92'13","reported_seq":214,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:36:50.437836+0000","last_change":"2024-03-28T22:33:28.442524+0000","last_active":"2024-03-28T22:36:50.437836+0000","last_peered":"2024-03-28T22:36:50.437836+0000","last_clean":"2024-03-28T22:36:50.437836+0000","last_became_active":"2024-03-28T22:33:28.442169+0000","last_became_peered":"2024-03-28T22:33:28.442169+0000","last_unstale":"2024-03-28T22:36:50.437836+0000","last_undegraded":"2024-03-28T22:36:50.437836+0000","last_fullsized":"2024-03-28T22:36:50.437836+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:08:47.101119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":167,"num_read_kb":167,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"94'5","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371822+0000","last_change":"2024-03-28T22:33:28.946697+0000","last_active":"2024-03-28T22:35:30.371822+0000","last_peered":"2024-03-28T22:35:30.371822+0000","last_clean":"2024-03-28T22:35:30.371822+0000","last_became_active":"2024-03-28T22:33:28.946303+0000","last_became_peered":"2024-03-28T22:33:28.946303+0000","last_unstale":"2024-03-28T22:35:30.371822+0000","last_undegraded":"2024-03-28T22:35:30.371822+0000","last_fullsized":"2024-03-28T22:35:30.371822+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:32:01.286073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":30,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370372+0000","last_change":"2024-03-28T22:34:30.429579+0000","last_active":"2024-03-28T22:35:30.370372+0000","last_peered":"2024-03-28T22:35:30.370372+0000","last_clean":"2024-03-28T22:35:30.370372+0000","last_became_active":"2024-03-28T22:34:30.429054+0000","last_became_peered":"2024-03-28T22:34:30.429054+0000","last_unstale":"2024-03-28T22:35:30.370372+0000","last_undegraded":"2024-03-28T22:35:30.370372+0000","last_fullsized":"2024-03-28T22:35:30.370372+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:03:46.492786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"92'2","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.372174+0000","last_change":"2024-03-28T22:33:28.457661+0000","last_active":"2024-03-28T22:35:30.372174+0000","last_peered":"2024-03-28T22:35:30.372174+0000","last_clean":"2024-03-28T22:35:30.372174+0000","last_became_active":"2024-03-28T22:33:28.457032+0000","last_became_peered":"2024-03-28T22:33:28.457032+0000","last_unstale":"2024-03-28T22:35:30.372174+0000","last_undegraded":"2024-03-28T22:35:30.372174+0000","last_fullsized":"2024-03-28T22:35:30.372174+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:43:09.820630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"94'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.453641+0000","last_change":"2024-03-28T22:33:28.954252+0000","last_active":"2024-03-28T22:35:30.453641+0000","last_peered":"2024-03-28T22:35:30.453641+0000","last_clean":"2024-03-28T22:35:30.453641+0000","last_became_active":"2024-03-28T22:33:28.953922+0000","last_became_peered":"2024-03-28T22:33:28.953922+0000","last_unstale":"2024-03-28T22:35:30.453641+0000","last_undegraded":"2024-03-28T22:35:30.453641+0000","last_fullsized":"2024-03-28T22:35:30.453641+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:18:47.097859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370607+0000","last_change":"2024-03-28T22:33:34.029129+0000","last_active":"2024-03-28T22:35:30.370607+0000","last_peered":"2024-03-28T22:35:30.370607+0000","last_clean":"2024-03-28T22:35:30.370607+0000","last_became_active":"2024-03-28T22:33:34.028452+0000","last_became_peered":"2024-03-28T22:33:34.028452+0000","last_unstale":"2024-03-28T22:35:30.370607+0000","last_undegraded":"2024-03-28T22:35:30.370607+0000","last_fullsized":"2024-03-28T22:35:30.370607+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:52:28.982773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"92'12","reported_seq":47,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254205+0000","last_change":"2024-03-28T22:33:28.439568+0000","last_active":"2024-03-28T22:35:30.254205+0000","last_peered":"2024-03-28T22:35:30.254205+0000","last_clean":"2024-03-28T22:35:30.254205+0000","last_became_active":"2024-03-28T22:33:28.438887+0000","last_became_peered":"2024-03-28T22:33:28.438887+0000","last_unstale":"2024-03-28T22:35:30.254205+0000","last_undegraded":"2024-03-28T22:35:30.254205+0000","last_fullsized":"2024-03-28T22:35:30.254205+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:56:43.617621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"94'9","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448797+0000","last_change":"2024-03-28T22:33:28.938311+0000","last_active":"2024-03-28T22:35:30.448797+0000","last_peered":"2024-03-28T22:35:30.448797+0000","last_clean":"2024-03-28T22:35:30.448797+0000","last_became_active":"2024-03-28T22:33:28.937997+0000","last_became_peered":"2024-03-28T22:33:28.937997+0000","last_unstale":"2024-03-28T22:35:30.448797+0000","last_undegraded":"2024-03-28T22:35:30.448797+0000","last_fullsized":"2024-03-28T22:35:30.448797+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:54:44.001545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370428+0000","last_change":"2024-03-28T22:33:34.016931+0000","last_active":"2024-03-28T22:35:30.370428+0000","last_peered":"2024-03-28T22:35:30.370428+0000","last_clean":"2024-03-28T22:35:30.370428+0000","last_became_active":"2024-03-28T22:33:34.016559+0000","last_became_peered":"2024-03-28T22:33:34.016559+0000","last_unstale":"2024-03-28T22:35:30.370428+0000","last_undegraded":"2024-03-28T22:35:30.370428+0000","last_fullsized":"2024-03-28T22:35:30.370428+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:08:31.252351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"92'5","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254097+0000","last_change":"2024-03-28T22:33:28.433166+0000","last_active":"2024-03-28T22:35:30.254097+0000","last_peered":"2024-03-28T22:35:30.254097+0000","last_clean":"2024-03-28T22:35:30.254097+0000","last_became_active":"2024-03-28T22:33:28.432716+0000","last_became_peered":"2024-03-28T22:33:28.432716+0000","last_unstale":"2024-03-28T22:35:30.254097+0000","last_undegraded":"2024-03-28T22:35:30.254097+0000","last_fullsized":"2024-03-28T22:35:30.254097+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:42:41.613583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"94'15","reported_seq":47,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275541+0000","last_change":"2024-03-28T22:33:28.923307+0000","last_active":"2024-03-28T22:35:30.275541+0000","last_peered":"2024-03-28T22:35:30.275541+0000","last_clean":"2024-03-28T22:35:30.275541+0000","last_became_active":"2024-03-28T22:33:28.922499+0000","last_became_peered":"2024-03-28T22:33:28.922499+0000","last_unstale":"2024-03-28T22:35:30.275541+0000","last_undegraded":"2024-03-28T22:35:30.275541+0000","last_fullsized":"2024-03-28T22:35:30.275541+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:48:48.514482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252183+0000","last_change":"2024-03-28T22:33:34.002209+0000","last_active":"2024-03-28T22:35:30.252183+0000","last_peered":"2024-03-28T22:35:30.252183+0000","last_clean":"2024-03-28T22:35:30.252183+0000","last_became_active":"2024-03-28T22:33:34.001746+0000","last_became_peered":"2024-03-28T22:33:34.001746+0000","last_unstale":"2024-03-28T22:35:30.252183+0000","last_undegraded":"2024-03-28T22:35:30.252183+0000","last_fullsized":"2024-03-28T22:35:30.252183+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:02:02.333328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"96'7","reported_seq":211,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:36:50.482307+0000","last_change":"2024-03-28T22:33:28.435380+0000","last_active":"2024-03-28T22:36:50.482307+0000","last_peered":"2024-03-28T22:36:50.482307+0000","last_clean":"2024-03-28T22:36:50.482307+0000","last_became_active":"2024-03-28T22:33:28.434898+0000","last_became_peered":"2024-03-28T22:33:28.434898+0000","last_unstale":"2024-03-28T22:36:50.482307+0000","last_undegraded":"2024-03-28T22:36:50.482307+0000","last_fullsized":"2024-03-28T22:36:50.482307+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:58:06.810394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":170,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"94'7","reported_seq":45,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254282+0000","last_change":"2024-03-28T22:34:30.121370+0000","last_active":"2024-03-28T22:35:30.254282+0000","last_peered":"2024-03-28T22:35:30.254282+0000","last_clean":"2024-03-28T22:35:30.254282+0000","last_became_active":"2024-03-28T22:34:30.121086+0000","last_became_peered":"2024-03-28T22:34:30.121086+0000","last_unstale":"2024-03-28T22:35:30.254282+0000","last_undegraded":"2024-03-28T22:35:30.254282+0000","last_fullsized":"2024-03-28T22:35:30.254282+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:26:24.244663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252057+0000","last_change":"2024-03-28T22:33:34.031350+0000","last_active":"2024-03-28T22:35:30.252057+0000","last_peered":"2024-03-28T22:35:30.252057+0000","last_clean":"2024-03-28T22:35:30.252057+0000","last_became_active":"2024-03-28T22:33:34.030946+0000","last_became_peered":"2024-03-28T22:33:34.030946+0000","last_unstale":"2024-03-28T22:35:30.252057+0000","last_undegraded":"2024-03-28T22:35:30.252057+0000","last_fullsized":"2024-03-28T22:35:30.252057+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:54:22.907363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"92'7","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.259226+0000","last_change":"2024-03-28T22:33:28.446197+0000","last_active":"2024-03-28T22:35:30.259226+0000","last_peered":"2024-03-28T22:35:30.259226+0000","last_clean":"2024-03-28T22:35:30.259226+0000","last_became_active":"2024-03-28T22:33:28.445891+0000","last_became_peered":"2024-03-28T22:33:28.445891+0000","last_unstale":"2024-03-28T22:35:30.259226+0000","last_undegraded":"2024-03-28T22:35:30.259226+0000","last_fullsized":"2024-03-28T22:35:30.259226+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:58:57.385302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"94'8","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275591+0000","last_change":"2024-03-28T22:33:28.955651+0000","last_active":"2024-03-28T22:35:30.275591+0000","last_peered":"2024-03-28T22:35:30.275591+0000","last_clean":"2024-03-28T22:35:30.275591+0000","last_became_active":"2024-03-28T22:33:28.955276+0000","last_became_peered":"2024-03-28T22:33:28.955276+0000","last_unstale":"2024-03-28T22:35:30.275591+0000","last_undegraded":"2024-03-28T22:35:30.275591+0000","last_fullsized":"2024-03-28T22:35:30.275591+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:36:13.455327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449809+0000","last_change":"2024-03-28T22:33:34.017520+0000","last_active":"2024-03-28T22:35:30.449809+0000","last_peered":"2024-03-28T22:35:30.449809+0000","last_clean":"2024-03-28T22:35:30.449809+0000","last_became_active":"2024-03-28T22:33:34.017022+0000","last_became_peered":"2024-03-28T22:33:34.017022+0000","last_unstale":"2024-03-28T22:35:30.449809+0000","last_undegraded":"2024-03-28T22:35:30.449809+0000","last_fullsized":"2024-03-28T22:35:30.449809+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:41:17.431581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"92'3","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368171+0000","last_change":"2024-03-28T22:33:28.445401+0000","last_active":"2024-03-28T22:35:30.368171+0000","last_peered":"2024-03-28T22:35:30.368171+0000","last_clean":"2024-03-28T22:35:30.368171+0000","last_became_active":"2024-03-28T22:33:28.444237+0000","last_became_peered":"2024-03-28T22:33:28.444237+0000","last_unstale":"2024-03-28T22:35:30.368171+0000","last_undegraded":"2024-03-28T22:35:30.368171+0000","last_fullsized":"2024-03-28T22:35:30.368171+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:18:12.820890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"94'8","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.250889+0000","last_change":"2024-03-28T22:34:30.156248+0000","last_active":"2024-03-28T22:35:30.250889+0000","last_peered":"2024-03-28T22:35:30.250889+0000","last_clean":"2024-03-28T22:35:30.250889+0000","last_became_active":"2024-03-28T22:34:30.155056+0000","last_became_peered":"2024-03-28T22:34:30.155056+0000","last_unstale":"2024-03-28T22:35:30.250889+0000","last_undegraded":"2024-03-28T22:35:30.250889+0000","last_fullsized":"2024-03-28T22:35:30.250889+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:28:56.469728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451480+0000","last_change":"2024-03-28T22:33:34.019899+0000","last_active":"2024-03-28T22:35:30.451480+0000","last_peered":"2024-03-28T22:35:30.451480+0000","last_clean":"2024-03-28T22:35:30.451480+0000","last_became_active":"2024-03-28T22:33:34.019392+0000","last_became_peered":"2024-03-28T22:33:34.019392+0000","last_unstale":"2024-03-28T22:35:30.451480+0000","last_undegraded":"2024-03-28T22:35:30.451480+0000","last_fullsized":"2024-03-28T22:35:30.451480+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:08:38.633797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"92'3","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275105+0000","last_change":"2024-03-28T22:33:28.433243+0000","last_active":"2024-03-28T22:35:30.275105+0000","last_peered":"2024-03-28T22:35:30.275105+0000","last_clean":"2024-03-28T22:35:30.275105+0000","last_became_active":"2024-03-28T22:33:28.432927+0000","last_became_peered":"2024-03-28T22:33:28.432927+0000","last_unstale":"2024-03-28T22:35:30.275105+0000","last_undegraded":"2024-03-28T22:35:30.275105+0000","last_fullsized":"2024-03-28T22:35:30.275105+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:40:43.444229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"94'11","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251345+0000","last_change":"2024-03-28T22:33:28.948111+0000","last_active":"2024-03-28T22:35:30.251345+0000","last_peered":"2024-03-28T22:35:30.251345+0000","last_clean":"2024-03-28T22:35:30.251345+0000","last_became_active":"2024-03-28T22:33:28.944649+0000","last_became_peered":"2024-03-28T22:33:28.944649+0000","last_unstale":"2024-03-28T22:35:30.251345+0000","last_undegraded":"2024-03-28T22:35:30.251345+0000","last_fullsized":"2024-03-28T22:35:30.251345+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:57:11.483037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371506+0000","last_change":"2024-03-28T22:33:34.000837+0000","last_active":"2024-03-28T22:35:30.371506+0000","last_peered":"2024-03-28T22:35:30.371506+0000","last_clean":"2024-03-28T22:35:30.371506+0000","last_became_active":"2024-03-28T22:33:34.000120+0000","last_became_peered":"2024-03-28T22:33:34.000120+0000","last_unstale":"2024-03-28T22:35:30.371506+0000","last_undegraded":"2024-03-28T22:35:30.371506+0000","last_fullsized":"2024-03-28T22:35:30.371506+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:14:32.028952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"96'6","reported_seq":377,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:36:53.486772+0000","last_change":"2024-03-28T22:33:28.436985+0000","last_active":"2024-03-28T22:36:53.486772+0000","last_peered":"2024-03-28T22:36:53.486772+0000","last_clean":"2024-03-28T22:36:53.486772+0000","last_became_active":"2024-03-28T22:33:28.436625+0000","last_became_peered":"2024-03-28T22:33:28.436625+0000","last_unstale":"2024-03-28T22:36:53.486772+0000","last_undegraded":"2024-03-28T22:36:53.486772+0000","last_fullsized":"2024-03-28T22:36:53.486772+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:50:45.775191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":337,"num_read_kb":337,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"94'10","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448895+0000","last_change":"2024-03-28T22:33:28.933265+0000","last_active":"2024-03-28T22:35:30.448895+0000","last_peered":"2024-03-28T22:35:30.448895+0000","last_clean":"2024-03-28T22:35:30.448895+0000","last_became_active":"2024-03-28T22:33:28.932954+0000","last_became_peered":"2024-03-28T22:33:28.932954+0000","last_unstale":"2024-03-28T22:35:30.448895+0000","last_undegraded":"2024-03-28T22:35:30.448895+0000","last_fullsized":"2024-03-28T22:35:30.448895+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:37.039055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251275+0000","last_change":"2024-03-28T22:33:34.024529+0000","last_active":"2024-03-28T22:35:30.251275+0000","last_peered":"2024-03-28T22:35:30.251275+0000","last_clean":"2024-03-28T22:35:30.251275+0000","last_became_active":"2024-03-28T22:33:34.020949+0000","last_became_peered":"2024-03-28T22:33:34.020949+0000","last_unstale":"2024-03-28T22:35:30.251275+0000","last_undegraded":"2024-03-28T22:35:30.251275+0000","last_fullsized":"2024-03-28T22:35:30.251275+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:28:52.039684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"92'4","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248388+0000","last_change":"2024-03-28T22:33:28.432599+0000","last_active":"2024-03-28T22:35:30.248388+0000","last_peered":"2024-03-28T22:35:30.248388+0000","last_clean":"2024-03-28T22:35:30.248388+0000","last_became_active":"2024-03-28T22:33:28.430710+0000","last_became_peered":"2024-03-28T22:33:28.430710+0000","last_unstale":"2024-03-28T22:35:30.248388+0000","last_undegraded":"2024-03-28T22:35:30.248388+0000","last_fullsized":"2024-03-28T22:35:30.248388+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:54.174747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"94'9","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.453918+0000","last_change":"2024-03-28T22:33:28.955441+0000","last_active":"2024-03-28T22:35:30.453918+0000","last_peered":"2024-03-28T22:35:30.453918+0000","last_clean":"2024-03-28T22:35:30.453918+0000","last_became_active":"2024-03-28T22:33:28.953269+0000","last_became_peered":"2024-03-28T22:33:28.953269+0000","last_unstale":"2024-03-28T22:35:30.453918+0000","last_undegraded":"2024-03-28T22:35:30.453918+0000","last_fullsized":"2024-03-28T22:35:30.453918+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:47:04.128605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370719+0000","last_change":"2024-03-28T22:33:34.028791+0000","last_active":"2024-03-28T22:35:30.370719+0000","last_peered":"2024-03-28T22:35:30.370719+0000","last_clean":"2024-03-28T22:35:30.370719+0000","last_became_active":"2024-03-28T22:33:34.028361+0000","last_became_peered":"2024-03-28T22:33:34.028361+0000","last_unstale":"2024-03-28T22:35:30.370719+0000","last_undegraded":"2024-03-28T22:35:30.370719+0000","last_fullsized":"2024-03-28T22:35:30.370719+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:42:21.264459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"92'5","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275283+0000","last_change":"2024-03-28T22:33:28.455353+0000","last_active":"2024-03-28T22:35:30.275283+0000","last_peered":"2024-03-28T22:35:30.275283+0000","last_clean":"2024-03-28T22:35:30.275283+0000","last_became_active":"2024-03-28T22:33:28.455090+0000","last_became_peered":"2024-03-28T22:33:28.455090+0000","last_unstale":"2024-03-28T22:35:30.275283+0000","last_undegraded":"2024-03-28T22:35:30.275283+0000","last_fullsized":"2024-03-28T22:35:30.275283+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:19:22.496600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"94'11","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448743+0000","last_change":"2024-03-28T22:33:28.928875+0000","last_active":"2024-03-28T22:35:30.448743+0000","last_peered":"2024-03-28T22:35:30.448743+0000","last_clean":"2024-03-28T22:35:30.448743+0000","last_became_active":"2024-03-28T22:33:28.928449+0000","last_became_peered":"2024-03-28T22:33:28.928449+0000","last_unstale":"2024-03-28T22:35:30.448743+0000","last_undegraded":"2024-03-28T22:35:30.448743+0000","last_fullsized":"2024-03-28T22:35:30.448743+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:34:00.402100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251348+0000","last_change":"2024-03-28T22:33:34.012739+0000","last_active":"2024-03-28T22:35:30.251348+0000","last_peered":"2024-03-28T22:35:30.251348+0000","last_clean":"2024-03-28T22:35:30.251348+0000","last_became_active":"2024-03-28T22:33:34.012262+0000","last_became_peered":"2024-03-28T22:33:34.012262+0000","last_unstale":"2024-03-28T22:35:30.251348+0000","last_undegraded":"2024-03-28T22:35:30.251348+0000","last_fullsized":"2024-03-28T22:35:30.251348+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:24:48.351037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"96'42","reported_seq":424,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:36:54.358887+0000","last_change":"2024-03-28T22:33:28.430606+0000","last_active":"2024-03-28T22:36:54.358887+0000","last_peered":"2024-03-28T22:36:54.358887+0000","last_clean":"2024-03-28T22:36:54.358887+0000","last_became_active":"2024-03-28T22:33:28.430233+0000","last_became_peered":"2024-03-28T22:33:28.430233+0000","last_unstale":"2024-03-28T22:36:54.358887+0000","last_undegraded":"2024-03-28T22:36:54.358887+0000","last_fullsized":"2024-03-28T22:36:54.358887+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:59:49.000569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73045,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":348,"num_read_kb":246,"num_write":44,"num_write_kb":92,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"94'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254080+0000","last_change":"2024-03-28T22:33:28.931167+0000","last_active":"2024-03-28T22:35:30.254080+0000","last_peered":"2024-03-28T22:35:30.254080+0000","last_clean":"2024-03-28T22:35:30.254080+0000","last_became_active":"2024-03-28T22:33:28.930824+0000","last_became_peered":"2024-03-28T22:33:28.930824+0000","last_unstale":"2024-03-28T22:35:30.254080+0000","last_undegraded":"2024-03-28T22:35:30.254080+0000","last_fullsized":"2024-03-28T22:35:30.254080+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:46:49.102791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275836+0000","last_change":"2024-03-28T22:33:34.023056+0000","last_active":"2024-03-28T22:35:30.275836+0000","last_peered":"2024-03-28T22:35:30.275836+0000","last_clean":"2024-03-28T22:35:30.275836+0000","last_became_active":"2024-03-28T22:33:34.022612+0000","last_became_peered":"2024-03-28T22:33:34.022612+0000","last_unstale":"2024-03-28T22:35:30.275836+0000","last_undegraded":"2024-03-28T22:35:30.275836+0000","last_fullsized":"2024-03-28T22:35:30.275836+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:52:31.936008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"92'9","reported_seq":45,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252111+0000","last_change":"2024-03-28T22:33:28.430963+0000","last_active":"2024-03-28T22:35:30.252111+0000","last_peered":"2024-03-28T22:35:30.252111+0000","last_clean":"2024-03-28T22:35:30.252111+0000","last_became_active":"2024-03-28T22:33:28.430632+0000","last_became_peered":"2024-03-28T22:33:28.430632+0000","last_unstale":"2024-03-28T22:35:30.252111+0000","last_undegraded":"2024-03-28T22:35:30.252111+0000","last_fullsized":"2024-03-28T22:35:30.252111+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:30:23.104769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"94'9","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368385+0000","last_change":"2024-03-28T22:33:28.954105+0000","last_active":"2024-03-28T22:35:30.368385+0000","last_peered":"2024-03-28T22:35:30.368385+0000","last_clean":"2024-03-28T22:35:30.368385+0000","last_became_active":"2024-03-28T22:33:28.953768+0000","last_became_peered":"2024-03-28T22:33:28.953768+0000","last_unstale":"2024-03-28T22:35:30.368385+0000","last_undegraded":"2024-03-28T22:35:30.368385+0000","last_fullsized":"2024-03-28T22:35:30.368385+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:01:25.184523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449565+0000","last_change":"2024-03-28T22:33:34.027976+0000","last_active":"2024-03-28T22:35:30.449565+0000","last_peered":"2024-03-28T22:35:30.449565+0000","last_clean":"2024-03-28T22:35:30.449565+0000","last_became_active":"2024-03-28T22:33:34.027509+0000","last_became_peered":"2024-03-28T22:33:34.027509+0000","last_unstale":"2024-03-28T22:35:30.449565+0000","last_undegraded":"2024-03-28T22:35:30.449565+0000","last_fullsized":"2024-03-28T22:35:30.449565+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:25:41.908441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"92'5","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275307+0000","last_change":"2024-03-28T22:33:28.433283+0000","last_active":"2024-03-28T22:35:30.275307+0000","last_peered":"2024-03-28T22:35:30.275307+0000","last_clean":"2024-03-28T22:35:30.275307+0000","last_became_active":"2024-03-28T22:33:28.432954+0000","last_became_peered":"2024-03-28T22:33:28.432954+0000","last_unstale":"2024-03-28T22:35:30.275307+0000","last_undegraded":"2024-03-28T22:35:30.275307+0000","last_fullsized":"2024-03-28T22:35:30.275307+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:08:34.480459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"94'15","reported_seq":47,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248465+0000","last_change":"2024-03-28T22:33:28.939016+0000","last_active":"2024-03-28T22:35:30.248465+0000","last_peered":"2024-03-28T22:35:30.248465+0000","last_clean":"2024-03-28T22:35:30.248465+0000","last_became_active":"2024-03-28T22:33:28.938695+0000","last_became_peered":"2024-03-28T22:33:28.938695+0000","last_unstale":"2024-03-28T22:35:30.248465+0000","last_undegraded":"2024-03-28T22:35:30.248465+0000","last_fullsized":"2024-03-28T22:35:30.248465+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:26:53.264559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252805+0000","last_change":"2024-03-28T22:33:34.030127+0000","last_active":"2024-03-28T22:35:30.252805+0000","last_peered":"2024-03-28T22:35:30.252805+0000","last_clean":"2024-03-28T22:35:30.252805+0000","last_became_active":"2024-03-28T22:33:34.028056+0000","last_became_peered":"2024-03-28T22:33:34.028056+0000","last_unstale":"2024-03-28T22:35:30.252805+0000","last_undegraded":"2024-03-28T22:35:30.252805+0000","last_fullsized":"2024-03-28T22:35:30.252805+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:29:41.785013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'7","reported_seq":209,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:36:50.560497+0000","last_change":"2024-03-28T22:33:28.435350+0000","last_active":"2024-03-28T22:36:50.560497+0000","last_peered":"2024-03-28T22:36:50.560497+0000","last_clean":"2024-03-28T22:36:50.560497+0000","last_became_active":"2024-03-28T22:33:28.434954+0000","last_became_peered":"2024-03-28T22:33:28.434954+0000","last_unstale":"2024-03-28T22:36:50.560497+0000","last_undegraded":"2024-03-28T22:36:50.560497+0000","last_fullsized":"2024-03-28T22:36:50.560497+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:44:04.969596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":170,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"94'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248129+0000","last_change":"2024-03-28T22:33:28.921219+0000","last_active":"2024-03-28T22:35:30.248129+0000","last_peered":"2024-03-28T22:35:30.248129+0000","last_clean":"2024-03-28T22:35:30.248129+0000","last_became_active":"2024-03-28T22:33:28.920664+0000","last_became_peered":"2024-03-28T22:33:28.920664+0000","last_unstale":"2024-03-28T22:35:30.248129+0000","last_undegraded":"2024-03-28T22:35:30.248129+0000","last_fullsized":"2024-03-28T22:35:30.248129+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:48:55.032058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369383+0000","last_change":"2024-03-28T22:33:34.018101+0000","last_active":"2024-03-28T22:35:30.369383+0000","last_peered":"2024-03-28T22:35:30.369383+0000","last_clean":"2024-03-28T22:35:30.369383+0000","last_became_active":"2024-03-28T22:33:34.017260+0000","last_became_peered":"2024-03-28T22:33:34.017260+0000","last_unstale":"2024-03-28T22:35:30.369383+0000","last_undegraded":"2024-03-28T22:35:30.369383+0000","last_fullsized":"2024-03-28T22:35:30.369383+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:48:53.178090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"92'17","reported_seq":52,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.253595+0000","last_change":"2024-03-28T22:33:28.450232+0000","last_active":"2024-03-28T22:35:30.253595+0000","last_peered":"2024-03-28T22:35:30.253595+0000","last_clean":"2024-03-28T22:35:30.253595+0000","last_became_active":"2024-03-28T22:33:28.449958+0000","last_became_peered":"2024-03-28T22:33:28.449958+0000","last_unstale":"2024-03-28T22:35:30.253595+0000","last_undegraded":"2024-03-28T22:35:30.253595+0000","last_fullsized":"2024-03-28T22:35:30.253595+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:01:42.130260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"94'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275235+0000","last_change":"2024-03-28T22:33:28.926263+0000","last_active":"2024-03-28T22:35:30.275235+0000","last_peered":"2024-03-28T22:35:30.275235+0000","last_clean":"2024-03-28T22:35:30.275235+0000","last_became_active":"2024-03-28T22:33:28.925927+0000","last_became_peered":"2024-03-28T22:33:28.925927+0000","last_unstale":"2024-03-28T22:35:30.275235+0000","last_undegraded":"2024-03-28T22:35:30.275235+0000","last_fullsized":"2024-03-28T22:35:30.275235+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:41:42.477694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.455002+0000","last_change":"2024-03-28T22:35:30.455002+0000","last_active":"2024-03-28T22:35:30.455002+0000","last_peered":"2024-03-28T22:35:30.455002+0000","last_clean":"2024-03-28T22:35:30.455002+0000","last_became_active":"2024-03-28T22:35:30.454479+0000","last_became_peered":"2024-03-28T22:35:30.454479+0000","last_unstale":"2024-03-28T22:35:30.455002+0000","last_undegraded":"2024-03-28T22:35:30.455002+0000","last_fullsized":"2024-03-28T22:35:30.455002+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:57:06.300031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,3,1],"acting":[10,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.f","version":"92'2","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449832+0000","last_change":"2024-03-28T22:34:29.986650+0000","last_active":"2024-03-28T22:35:30.449832+0000","last_peered":"2024-03-28T22:35:30.449832+0000","last_clean":"2024-03-28T22:35:30.449832+0000","last_became_active":"2024-03-28T22:34:29.981875+0000","last_became_peered":"2024-03-28T22:34:29.981875+0000","last_unstale":"2024-03-28T22:35:30.449832+0000","last_undegraded":"2024-03-28T22:35:30.449832+0000","last_fullsized":"2024-03-28T22:35:30.449832+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:45:24.166133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"94'5","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449947+0000","last_change":"2024-03-28T22:34:29.982815+0000","last_active":"2024-03-28T22:35:30.449947+0000","last_peered":"2024-03-28T22:35:30.449947+0000","last_clean":"2024-03-28T22:35:30.449947+0000","last_became_active":"2024-03-28T22:34:29.982458+0000","last_became_peered":"2024-03-28T22:34:29.982458+0000","last_unstale":"2024-03-28T22:35:30.449947+0000","last_undegraded":"2024-03-28T22:35:30.449947+0000","last_fullsized":"2024-03-28T22:35:30.449947+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:11:30.617181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":30,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451505+0000","last_change":"2024-03-28T22:35:30.451063+0000","last_active":"2024-03-28T22:35:30.451505+0000","last_peered":"2024-03-28T22:35:30.451505+0000","last_clean":"2024-03-28T22:35:30.451505+0000","last_became_active":"2024-03-28T22:35:30.450525+0000","last_became_peered":"2024-03-28T22:35:30.450525+0000","last_unstale":"2024-03-28T22:35:30.451505+0000","last_undegraded":"2024-03-28T22:35:30.451505+0000","last_fullsized":"2024-03-28T22:35:30.451505+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:35:29.004834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"92'1","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252202+0000","last_change":"2024-03-28T22:34:29.992884+0000","last_active":"2024-03-28T22:35:30.252202+0000","last_peered":"2024-03-28T22:35:30.252202+0000","last_clean":"2024-03-28T22:35:30.252202+0000","last_became_active":"2024-03-28T22:34:29.991657+0000","last_became_peered":"2024-03-28T22:34:29.991657+0000","last_unstale":"2024-03-28T22:35:30.252202+0000","last_undegraded":"2024-03-28T22:35:30.252202+0000","last_fullsized":"2024-03-28T22:35:30.252202+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:25:14.540783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"94'4","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.259438+0000","last_change":"2024-03-28T22:33:28.942097+0000","last_active":"2024-03-28T22:35:30.259438+0000","last_peered":"2024-03-28T22:35:30.259438+0000","last_clean":"2024-03-28T22:35:30.259438+0000","last_became_active":"2024-03-28T22:33:28.941812+0000","last_became_peered":"2024-03-28T22:33:28.941812+0000","last_unstale":"2024-03-28T22:35:30.259438+0000","last_undegraded":"2024-03-28T22:35:30.259438+0000","last_fullsized":"2024-03-28T22:35:30.259438+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:36:10.204216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369177+0000","last_change":"2024-03-28T22:33:34.023171+0000","last_active":"2024-03-28T22:35:30.369177+0000","last_peered":"2024-03-28T22:35:30.369177+0000","last_clean":"2024-03-28T22:35:30.369177+0000","last_became_active":"2024-03-28T22:33:34.021313+0000","last_became_peered":"2024-03-28T22:33:34.021313+0000","last_unstale":"2024-03-28T22:35:30.369177+0000","last_undegraded":"2024-03-28T22:35:30.369177+0000","last_fullsized":"2024-03-28T22:35:30.369177+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:29:42.592761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"92'5","reported_seq":207,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:36:51.439644+0000","last_change":"2024-03-28T22:33:28.449133+0000","last_active":"2024-03-28T22:36:51.439644+0000","last_peered":"2024-03-28T22:36:51.439644+0000","last_clean":"2024-03-28T22:36:51.439644+0000","last_became_active":"2024-03-28T22:33:28.448743+0000","last_became_peered":"2024-03-28T22:33:28.448743+0000","last_unstale":"2024-03-28T22:36:51.439644+0000","last_undegraded":"2024-03-28T22:36:51.439644+0000","last_fullsized":"2024-03-28T22:36:51.439644+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:09:37.119736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":168,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.259332+0000","last_change":"2024-03-28T22:33:28.944198+0000","last_active":"2024-03-28T22:35:30.259332+0000","last_peered":"2024-03-28T22:35:30.259332+0000","last_clean":"2024-03-28T22:35:30.259332+0000","last_became_active":"2024-03-28T22:33:28.943818+0000","last_became_peered":"2024-03-28T22:33:28.943818+0000","last_unstale":"2024-03-28T22:35:30.259332+0000","last_undegraded":"2024-03-28T22:35:30.259332+0000","last_fullsized":"2024-03-28T22:35:30.259332+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:31:03.791364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369583+0000","last_change":"2024-03-28T22:33:34.015499+0000","last_active":"2024-03-28T22:35:30.369583+0000","last_peered":"2024-03-28T22:35:30.369583+0000","last_clean":"2024-03-28T22:35:30.369583+0000","last_became_active":"2024-03-28T22:33:34.015013+0000","last_became_peered":"2024-03-28T22:33:34.015013+0000","last_unstale":"2024-03-28T22:35:30.369583+0000","last_undegraded":"2024-03-28T22:35:30.369583+0000","last_fullsized":"2024-03-28T22:35:30.369583+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:34:30.582198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"94'8","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275296+0000","last_change":"2024-03-28T22:33:28.926732+0000","last_active":"2024-03-28T22:35:30.275296+0000","last_peered":"2024-03-28T22:35:30.275296+0000","last_clean":"2024-03-28T22:35:30.275296+0000","last_became_active":"2024-03-28T22:33:28.926366+0000","last_became_peered":"2024-03-28T22:33:28.926366+0000","last_unstale":"2024-03-28T22:35:30.275296+0000","last_undegraded":"2024-03-28T22:35:30.275296+0000","last_fullsized":"2024-03-28T22:35:30.275296+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:19:10.769141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'6","reported_seq":207,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:36:50.559858+0000","last_change":"2024-03-28T22:33:28.457400+0000","last_active":"2024-03-28T22:36:50.559858+0000","last_peered":"2024-03-28T22:36:50.559858+0000","last_clean":"2024-03-28T22:36:50.559858+0000","last_became_active":"2024-03-28T22:33:28.456994+0000","last_became_peered":"2024-03-28T22:33:28.456994+0000","last_unstale":"2024-03-28T22:36:50.559858+0000","last_undegraded":"2024-03-28T22:36:50.559858+0000","last_fullsized":"2024-03-28T22:36:50.559858+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:32:16.438540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":167,"num_read_kb":167,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"92'1","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248405+0000","last_change":"2024-03-28T22:33:34.027746+0000","last_active":"2024-03-28T22:35:30.248405+0000","last_peered":"2024-03-28T22:35:30.248405+0000","last_clean":"2024-03-28T22:35:30.248405+0000","last_became_active":"2024-03-28T22:33:34.027419+0000","last_became_peered":"2024-03-28T22:33:34.027419+0000","last_unstale":"2024-03-28T22:35:30.248405+0000","last_undegraded":"2024-03-28T22:35:30.248405+0000","last_fullsized":"2024-03-28T22:35:30.248405+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:55:43.686428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371544+0000","last_change":"2024-03-28T22:33:28.952393+0000","last_active":"2024-03-28T22:35:30.371544+0000","last_peered":"2024-03-28T22:35:30.371544+0000","last_clean":"2024-03-28T22:35:30.371544+0000","last_became_active":"2024-03-28T22:33:28.951951+0000","last_became_peered":"2024-03-28T22:33:28.951951+0000","last_unstale":"2024-03-28T22:35:30.371544+0000","last_undegraded":"2024-03-28T22:35:30.371544+0000","last_fullsized":"2024-03-28T22:35:30.371544+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:09:25.808842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"92'6","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370674+0000","last_change":"2024-03-28T22:33:28.437177+0000","last_active":"2024-03-28T22:35:30.370674+0000","last_peered":"2024-03-28T22:35:30.370674+0000","last_clean":"2024-03-28T22:35:30.370674+0000","last_became_active":"2024-03-28T22:33:28.436801+0000","last_became_peered":"2024-03-28T22:33:28.436801+0000","last_unstale":"2024-03-28T22:35:30.370674+0000","last_undegraded":"2024-03-28T22:35:30.370674+0000","last_fullsized":"2024-03-28T22:35:30.370674+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:24:52.549490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":372,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.247989+0000","last_change":"2024-03-28T22:32:07.867336+0000","last_active":"2024-03-28T22:35:30.247989+0000","last_peered":"2024-03-28T22:35:30.247989+0000","last_clean":"2024-03-28T22:35:30.247989+0000","last_became_active":"2024-03-28T22:32:07.866819+0000","last_became_peered":"2024-03-28T22:32:07.866819+0000","last_unstale":"2024-03-28T22:35:30.247989+0000","last_undegraded":"2024-03-28T22:35:30.247989+0000","last_fullsized":"2024-03-28T22:35:30.247989+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:28:14.713501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:28:14.713501+0000","last_clean_scrub_stamp":"2024-03-28T22:28:14.713501+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:00:52.438708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275597+0000","last_change":"2024-03-28T22:33:34.020668+0000","last_active":"2024-03-28T22:35:30.275597+0000","last_peered":"2024-03-28T22:35:30.275597+0000","last_clean":"2024-03-28T22:35:30.275597+0000","last_became_active":"2024-03-28T22:33:34.020240+0000","last_became_peered":"2024-03-28T22:33:34.020240+0000","last_unstale":"2024-03-28T22:35:30.275597+0000","last_undegraded":"2024-03-28T22:35:30.275597+0000","last_fullsized":"2024-03-28T22:35:30.275597+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:26:18.983983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"94'5","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368061+0000","last_change":"2024-03-28T22:33:28.935237+0000","last_active":"2024-03-28T22:35:30.368061+0000","last_peered":"2024-03-28T22:35:30.368061+0000","last_clean":"2024-03-28T22:35:30.368061+0000","last_became_active":"2024-03-28T22:33:28.934875+0000","last_became_peered":"2024-03-28T22:33:28.934875+0000","last_unstale":"2024-03-28T22:35:30.368061+0000","last_undegraded":"2024-03-28T22:35:30.368061+0000","last_fullsized":"2024-03-28T22:35:30.368061+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:20:50.789302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"92'4","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248187+0000","last_change":"2024-03-28T22:33:28.441654+0000","last_active":"2024-03-28T22:35:30.248187+0000","last_peered":"2024-03-28T22:35:30.248187+0000","last_clean":"2024-03-28T22:35:30.248187+0000","last_became_active":"2024-03-28T22:33:28.441293+0000","last_became_peered":"2024-03-28T22:33:28.441293+0000","last_unstale":"2024-03-28T22:35:30.248187+0000","last_undegraded":"2024-03-28T22:35:30.248187+0000","last_fullsized":"2024-03-28T22:35:30.248187+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:31:17.737977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449486+0000","last_change":"2024-03-28T22:33:34.018126+0000","last_active":"2024-03-28T22:35:30.449486+0000","last_peered":"2024-03-28T22:35:30.449486+0000","last_clean":"2024-03-28T22:35:30.449486+0000","last_became_active":"2024-03-28T22:33:34.017573+0000","last_became_peered":"2024-03-28T22:33:34.017573+0000","last_unstale":"2024-03-28T22:35:30.449486+0000","last_undegraded":"2024-03-28T22:35:30.449486+0000","last_fullsized":"2024-03-28T22:35:30.449486+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:23:31.013511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"92'4","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.254108+0000","last_change":"2024-03-28T22:33:28.454069+0000","last_active":"2024-03-28T22:35:30.254108+0000","last_peered":"2024-03-28T22:35:30.254108+0000","last_clean":"2024-03-28T22:35:30.254108+0000","last_became_active":"2024-03-28T22:33:28.453754+0000","last_became_peered":"2024-03-28T22:33:28.453754+0000","last_unstale":"2024-03-28T22:35:30.254108+0000","last_undegraded":"2024-03-28T22:35:30.254108+0000","last_fullsized":"2024-03-28T22:35:30.254108+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:23:33.981072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"94'9","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369387+0000","last_change":"2024-03-28T22:33:28.934390+0000","last_active":"2024-03-28T22:35:30.369387+0000","last_peered":"2024-03-28T22:35:30.369387+0000","last_clean":"2024-03-28T22:35:30.369387+0000","last_became_active":"2024-03-28T22:33:28.933919+0000","last_became_peered":"2024-03-28T22:33:28.933919+0000","last_unstale":"2024-03-28T22:35:30.369387+0000","last_undegraded":"2024-03-28T22:35:30.369387+0000","last_fullsized":"2024-03-28T22:35:30.369387+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:27:07.595144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371312+0000","last_change":"2024-03-28T22:33:34.019910+0000","last_active":"2024-03-28T22:35:30.371312+0000","last_peered":"2024-03-28T22:35:30.371312+0000","last_clean":"2024-03-28T22:35:30.371312+0000","last_became_active":"2024-03-28T22:33:34.019522+0000","last_became_peered":"2024-03-28T22:33:34.019522+0000","last_unstale":"2024-03-28T22:35:30.371312+0000","last_undegraded":"2024-03-28T22:35:30.371312+0000","last_fullsized":"2024-03-28T22:35:30.371312+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:01:29.298513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"96'16","reported_seq":50,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:36:48.496850+0000","last_change":"2024-03-28T22:33:28.444513+0000","last_active":"2024-03-28T22:36:48.496850+0000","last_peered":"2024-03-28T22:36:48.496850+0000","last_clean":"2024-03-28T22:36:48.496850+0000","last_became_active":"2024-03-28T22:33:28.444091+0000","last_became_peered":"2024-03-28T22:33:28.444091+0000","last_unstale":"2024-03-28T22:36:48.496850+0000","last_undegraded":"2024-03-28T22:36:48.496850+0000","last_fullsized":"2024-03-28T22:36:48.496850+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:49:58.331864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"94'5","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.372223+0000","last_change":"2024-03-28T22:33:28.948390+0000","last_active":"2024-03-28T22:35:30.372223+0000","last_peered":"2024-03-28T22:35:30.372223+0000","last_clean":"2024-03-28T22:35:30.372223+0000","last_became_active":"2024-03-28T22:33:28.942846+0000","last_became_peered":"2024-03-28T22:33:28.942846+0000","last_unstale":"2024-03-28T22:35:30.372223+0000","last_undegraded":"2024-03-28T22:35:30.372223+0000","last_fullsized":"2024-03-28T22:35:30.372223+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:50.596365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251712+0000","last_change":"2024-03-28T22:33:33.999429+0000","last_active":"2024-03-28T22:35:30.251712+0000","last_peered":"2024-03-28T22:35:30.251712+0000","last_clean":"2024-03-28T22:35:30.251712+0000","last_became_active":"2024-03-28T22:33:33.998979+0000","last_became_peered":"2024-03-28T22:33:33.998979+0000","last_unstale":"2024-03-28T22:35:30.251712+0000","last_undegraded":"2024-03-28T22:35:30.251712+0000","last_fullsized":"2024-03-28T22:35:30.251712+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:34:26.450731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"92'7","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370791+0000","last_change":"2024-03-28T22:33:28.442861+0000","last_active":"2024-03-28T22:35:30.370791+0000","last_peered":"2024-03-28T22:35:30.370791+0000","last_clean":"2024-03-28T22:35:30.370791+0000","last_became_active":"2024-03-28T22:33:28.442468+0000","last_became_peered":"2024-03-28T22:33:28.442468+0000","last_unstale":"2024-03-28T22:35:30.370791+0000","last_undegraded":"2024-03-28T22:35:30.370791+0000","last_fullsized":"2024-03-28T22:35:30.370791+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:45:49.899031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"94'7","reported_seq":45,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451985+0000","last_change":"2024-03-28T22:34:30.429872+0000","last_active":"2024-03-28T22:35:30.451985+0000","last_peered":"2024-03-28T22:35:30.451985+0000","last_clean":"2024-03-28T22:35:30.451985+0000","last_became_active":"2024-03-28T22:34:30.429237+0000","last_became_peered":"2024-03-28T22:34:30.429237+0000","last_unstale":"2024-03-28T22:35:30.451985+0000","last_undegraded":"2024-03-28T22:35:30.451985+0000","last_fullsized":"2024-03-28T22:35:30.451985+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:14:46.608826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370565+0000","last_change":"2024-03-28T22:33:34.013826+0000","last_active":"2024-03-28T22:35:30.370565+0000","last_peered":"2024-03-28T22:35:30.370565+0000","last_clean":"2024-03-28T22:35:30.370565+0000","last_became_active":"2024-03-28T22:33:34.013415+0000","last_became_peered":"2024-03-28T22:33:34.013415+0000","last_unstale":"2024-03-28T22:35:30.370565+0000","last_undegraded":"2024-03-28T22:35:30.370565+0000","last_fullsized":"2024-03-28T22:35:30.370565+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T10:19:30.965940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"92'2","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448399+0000","last_change":"2024-03-28T22:33:28.442225+0000","last_active":"2024-03-28T22:35:30.448399+0000","last_peered":"2024-03-28T22:35:30.448399+0000","last_clean":"2024-03-28T22:35:30.448399+0000","last_became_active":"2024-03-28T22:33:28.441844+0000","last_became_peered":"2024-03-28T22:33:28.441844+0000","last_unstale":"2024-03-28T22:35:30.448399+0000","last_undegraded":"2024-03-28T22:35:30.448399+0000","last_fullsized":"2024-03-28T22:35:30.448399+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:40:52.108663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"94'2","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.253675+0000","last_change":"2024-03-28T22:33:28.927571+0000","last_active":"2024-03-28T22:35:30.253675+0000","last_peered":"2024-03-28T22:35:30.253675+0000","last_clean":"2024-03-28T22:35:30.253675+0000","last_became_active":"2024-03-28T22:33:28.927275+0000","last_became_peered":"2024-03-28T22:33:28.927275+0000","last_unstale":"2024-03-28T22:35:30.253675+0000","last_undegraded":"2024-03-28T22:35:30.253675+0000","last_fullsized":"2024-03-28T22:35:30.253675+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:25:16.184533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.253011+0000","last_change":"2024-03-28T22:33:34.018429+0000","last_active":"2024-03-28T22:35:30.253011+0000","last_peered":"2024-03-28T22:35:30.253011+0000","last_clean":"2024-03-28T22:35:30.253011+0000","last_became_active":"2024-03-28T22:33:34.017066+0000","last_became_peered":"2024-03-28T22:33:34.017066+0000","last_unstale":"2024-03-28T22:35:30.253011+0000","last_undegraded":"2024-03-28T22:35:30.253011+0000","last_fullsized":"2024-03-28T22:35:30.253011+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:35:42.101072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"92'7","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371970+0000","last_change":"2024-03-28T22:33:28.437760+0000","last_active":"2024-03-28T22:35:30.371970+0000","last_peered":"2024-03-28T22:35:30.371970+0000","last_clean":"2024-03-28T22:35:30.371970+0000","last_became_active":"2024-03-28T22:33:28.437096+0000","last_became_peered":"2024-03-28T22:33:28.437096+0000","last_unstale":"2024-03-28T22:35:30.371970+0000","last_undegraded":"2024-03-28T22:35:30.371970+0000","last_fullsized":"2024-03-28T22:35:30.371970+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:29:20.778551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"94'9","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449358+0000","last_change":"2024-03-28T22:33:28.935805+0000","last_active":"2024-03-28T22:35:30.449358+0000","last_peered":"2024-03-28T22:35:30.449358+0000","last_clean":"2024-03-28T22:35:30.449358+0000","last_became_active":"2024-03-28T22:33:28.935520+0000","last_became_peered":"2024-03-28T22:33:28.935520+0000","last_unstale":"2024-03-28T22:35:30.449358+0000","last_undegraded":"2024-03-28T22:35:30.449358+0000","last_fullsized":"2024-03-28T22:35:30.449358+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:23.414310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"92'1","reported_seq":24,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370585+0000","last_change":"2024-03-28T22:33:34.029518+0000","last_active":"2024-03-28T22:35:30.370585+0000","last_peered":"2024-03-28T22:35:30.370585+0000","last_clean":"2024-03-28T22:35:30.370585+0000","last_became_active":"2024-03-28T22:33:34.029101+0000","last_became_peered":"2024-03-28T22:33:34.029101+0000","last_unstale":"2024-03-28T22:35:30.370585+0000","last_undegraded":"2024-03-28T22:35:30.370585+0000","last_fullsized":"2024-03-28T22:35:30.370585+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:14:05.595543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"96'4","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:38.679235+0000","last_change":"2024-03-28T22:33:28.438857+0000","last_active":"2024-03-28T22:35:38.679235+0000","last_peered":"2024-03-28T22:35:38.679235+0000","last_clean":"2024-03-28T22:35:38.679235+0000","last_became_active":"2024-03-28T22:33:28.438528+0000","last_became_peered":"2024-03-28T22:33:28.438528+0000","last_unstale":"2024-03-28T22:35:38.679235+0000","last_undegraded":"2024-03-28T22:35:38.679235+0000","last_fullsized":"2024-03-28T22:35:38.679235+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:07:54.506739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.371660+0000","last_change":"2024-03-28T22:33:28.958271+0000","last_active":"2024-03-28T22:35:30.371660+0000","last_peered":"2024-03-28T22:35:30.371660+0000","last_clean":"2024-03-28T22:35:30.371660+0000","last_became_active":"2024-03-28T22:33:28.957922+0000","last_became_peered":"2024-03-28T22:33:28.957922+0000","last_unstale":"2024-03-28T22:35:30.371660+0000","last_undegraded":"2024-03-28T22:35:30.371660+0000","last_fullsized":"2024-03-28T22:35:30.371660+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:33:19.003131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.370592+0000","last_change":"2024-03-28T22:33:34.000789+0000","last_active":"2024-03-28T22:35:30.370592+0000","last_peered":"2024-03-28T22:35:30.370592+0000","last_clean":"2024-03-28T22:35:30.370592+0000","last_became_active":"2024-03-28T22:33:34.000383+0000","last_became_peered":"2024-03-28T22:33:34.000383+0000","last_unstale":"2024-03-28T22:35:30.370592+0000","last_undegraded":"2024-03-28T22:35:30.370592+0000","last_fullsized":"2024-03-28T22:35:30.370592+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:50:29.925656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"92'4","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.253813+0000","last_change":"2024-03-28T22:33:28.439121+0000","last_active":"2024-03-28T22:35:30.253813+0000","last_peered":"2024-03-28T22:35:30.253813+0000","last_clean":"2024-03-28T22:35:30.253813+0000","last_became_active":"2024-03-28T22:33:28.438800+0000","last_became_peered":"2024-03-28T22:33:28.438800+0000","last_unstale":"2024-03-28T22:35:30.253813+0000","last_undegraded":"2024-03-28T22:35:30.253813+0000","last_fullsized":"2024-03-28T22:35:30.253813+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:57:10.013246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"94'5","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248157+0000","last_change":"2024-03-28T22:33:28.934499+0000","last_active":"2024-03-28T22:35:30.248157+0000","last_peered":"2024-03-28T22:35:30.248157+0000","last_clean":"2024-03-28T22:35:30.248157+0000","last_became_active":"2024-03-28T22:33:28.934147+0000","last_became_peered":"2024-03-28T22:33:28.934147+0000","last_unstale":"2024-03-28T22:35:30.248157+0000","last_undegraded":"2024-03-28T22:35:30.248157+0000","last_fullsized":"2024-03-28T22:35:30.248157+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:35:39.090052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448402+0000","last_change":"2024-03-28T22:33:34.017946+0000","last_active":"2024-03-28T22:35:30.448402+0000","last_peered":"2024-03-28T22:35:30.448402+0000","last_clean":"2024-03-28T22:35:30.448402+0000","last_became_active":"2024-03-28T22:33:34.017546+0000","last_became_peered":"2024-03-28T22:33:34.017546+0000","last_unstale":"2024-03-28T22:35:30.448402+0000","last_undegraded":"2024-03-28T22:35:30.448402+0000","last_fullsized":"2024-03-28T22:35:30.448402+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:14:39.081677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"96'6","reported_seq":211,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:36:52.356927+0000","last_change":"2024-03-28T22:33:28.453387+0000","last_active":"2024-03-28T22:36:52.356927+0000","last_peered":"2024-03-28T22:36:52.356927+0000","last_clean":"2024-03-28T22:36:52.356927+0000","last_became_active":"2024-03-28T22:33:28.453029+0000","last_became_peered":"2024-03-28T22:33:28.453029+0000","last_unstale":"2024-03-28T22:36:52.356927+0000","last_undegraded":"2024-03-28T22:36:52.356927+0000","last_fullsized":"2024-03-28T22:36:52.356927+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:12:33.246863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":171,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248615+0000","last_change":"2024-03-28T22:33:28.923819+0000","last_active":"2024-03-28T22:35:30.248615+0000","last_peered":"2024-03-28T22:35:30.248615+0000","last_clean":"2024-03-28T22:35:30.248615+0000","last_became_active":"2024-03-28T22:33:28.923471+0000","last_became_peered":"2024-03-28T22:33:28.923471+0000","last_unstale":"2024-03-28T22:35:30.248615+0000","last_undegraded":"2024-03-28T22:35:30.248615+0000","last_fullsized":"2024-03-28T22:35:30.248615+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:58:03.496406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.451029+0000","last_change":"2024-03-28T22:33:34.029789+0000","last_active":"2024-03-28T22:35:30.451029+0000","last_peered":"2024-03-28T22:35:30.451029+0000","last_clean":"2024-03-28T22:35:30.451029+0000","last_became_active":"2024-03-28T22:33:34.029231+0000","last_became_peered":"2024-03-28T22:33:34.029231+0000","last_unstale":"2024-03-28T22:35:30.451029+0000","last_undegraded":"2024-03-28T22:35:30.451029+0000","last_fullsized":"2024-03-28T22:35:30.451029+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:47:50.101393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"92'3","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275541+0000","last_change":"2024-03-28T22:33:28.435668+0000","last_active":"2024-03-28T22:35:30.275541+0000","last_peered":"2024-03-28T22:35:30.275541+0000","last_clean":"2024-03-28T22:35:30.275541+0000","last_became_active":"2024-03-28T22:33:28.435325+0000","last_became_peered":"2024-03-28T22:33:28.435325+0000","last_unstale":"2024-03-28T22:35:30.275541+0000","last_undegraded":"2024-03-28T22:35:30.275541+0000","last_fullsized":"2024-03-28T22:35:30.275541+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:13:06.281074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"94'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248244+0000","last_change":"2024-03-28T22:33:28.954138+0000","last_active":"2024-03-28T22:35:30.248244+0000","last_peered":"2024-03-28T22:35:30.248244+0000","last_clean":"2024-03-28T22:35:30.248244+0000","last_became_active":"2024-03-28T22:33:28.953752+0000","last_became_peered":"2024-03-28T22:33:28.953752+0000","last_unstale":"2024-03-28T22:35:30.248244+0000","last_undegraded":"2024-03-28T22:35:30.248244+0000","last_fullsized":"2024-03-28T22:35:30.248244+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:53:34.892316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.274983+0000","last_change":"2024-03-28T22:33:34.002672+0000","last_active":"2024-03-28T22:35:30.274983+0000","last_peered":"2024-03-28T22:35:30.274983+0000","last_clean":"2024-03-28T22:35:30.274983+0000","last_became_active":"2024-03-28T22:33:34.002195+0000","last_became_peered":"2024-03-28T22:33:34.002195+0000","last_unstale":"2024-03-28T22:35:30.274983+0000","last_undegraded":"2024-03-28T22:35:30.274983+0000","last_fullsized":"2024-03-28T22:35:30.274983+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:32.966724+0000","last_clean_scrub_stamp":"2024-03-28T22:33:32.966724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:24:11.521474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"92'5","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369212+0000","last_change":"2024-03-28T22:33:28.450818+0000","last_active":"2024-03-28T22:35:30.369212+0000","last_peered":"2024-03-28T22:35:30.369212+0000","last_clean":"2024-03-28T22:35:30.369212+0000","last_became_active":"2024-03-28T22:33:28.448592+0000","last_became_peered":"2024-03-28T22:33:28.448592+0000","last_unstale":"2024-03-28T22:35:30.369212+0000","last_undegraded":"2024-03-28T22:35:30.369212+0000","last_fullsized":"2024-03-28T22:35:30.369212+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:27.397805+0000","last_clean_scrub_stamp":"2024-03-28T22:33:27.397805+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:19:27.126306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"94'9","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.248488+0000","last_change":"2024-03-28T22:33:28.925191+0000","last_active":"2024-03-28T22:35:30.248488+0000","last_peered":"2024-03-28T22:35:30.248488+0000","last_clean":"2024-03-28T22:35:30.248488+0000","last_became_active":"2024-03-28T22:33:28.924424+0000","last_became_peered":"2024-03-28T22:33:28.924424+0000","last_unstale":"2024-03-28T22:35:30.248488+0000","last_undegraded":"2024-03-28T22:35:30.248488+0000","last_fullsized":"2024-03-28T22:35:30.248488+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:50:00.415047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"94'9","reported_seq":46,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449424+0000","last_change":"2024-03-28T22:34:29.987176+0000","last_active":"2024-03-28T22:35:30.449424+0000","last_peered":"2024-03-28T22:35:30.449424+0000","last_clean":"2024-03-28T22:35:30.449424+0000","last_became_active":"2024-03-28T22:34:29.986807+0000","last_became_peered":"2024-03-28T22:34:29.986807+0000","last_unstale":"2024-03-28T22:35:30.449424+0000","last_undegraded":"2024-03-28T22:35:30.449424+0000","last_fullsized":"2024-03-28T22:35:30.449424+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:00:06.774521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"94'12","reported_seq":44,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251938+0000","last_change":"2024-03-28T22:33:28.927068+0000","last_active":"2024-03-28T22:35:30.251938+0000","last_peered":"2024-03-28T22:35:30.251938+0000","last_clean":"2024-03-28T22:35:30.251938+0000","last_became_active":"2024-03-28T22:33:28.926734+0000","last_became_peered":"2024-03-28T22:33:28.926734+0000","last_unstale":"2024-03-28T22:35:30.251938+0000","last_undegraded":"2024-03-28T22:35:30.251938+0000","last_fullsized":"2024-03-28T22:35:30.251938+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:56.309555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"94'8","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.251930+0000","last_change":"2024-03-28T22:33:28.955675+0000","last_active":"2024-03-28T22:35:30.251930+0000","last_peered":"2024-03-28T22:35:30.251930+0000","last_clean":"2024-03-28T22:35:30.251930+0000","last_became_active":"2024-03-28T22:33:28.955329+0000","last_became_peered":"2024-03-28T22:33:28.955329+0000","last_unstale":"2024-03-28T22:35:30.251930+0000","last_undegraded":"2024-03-28T22:35:30.251930+0000","last_fullsized":"2024-03-28T22:35:30.251930+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T07:34:14.166434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"94'7","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.452477+0000","last_change":"2024-03-28T22:33:28.955125+0000","last_active":"2024-03-28T22:35:30.452477+0000","last_peered":"2024-03-28T22:35:30.452477+0000","last_clean":"2024-03-28T22:35:30.452477+0000","last_became_active":"2024-03-28T22:33:28.954434+0000","last_became_peered":"2024-03-28T22:33:28.954434+0000","last_unstale":"2024-03-28T22:35:30.452477+0000","last_undegraded":"2024-03-28T22:35:30.452477+0000","last_fullsized":"2024-03-28T22:35:30.452477+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:56:50.705012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"94'4","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.369433+0000","last_change":"2024-03-28T22:33:28.945230+0000","last_active":"2024-03-28T22:35:30.369433+0000","last_peered":"2024-03-28T22:35:30.369433+0000","last_clean":"2024-03-28T22:35:30.369433+0000","last_became_active":"2024-03-28T22:33:28.944921+0000","last_became_peered":"2024-03-28T22:33:28.944921+0000","last_unstale":"2024-03-28T22:35:30.369433+0000","last_undegraded":"2024-03-28T22:35:30.369433+0000","last_fullsized":"2024-03-28T22:35:30.369433+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:26:26.505774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"94'6","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449741+0000","last_change":"2024-03-28T22:33:28.928720+0000","last_active":"2024-03-28T22:35:30.449741+0000","last_peered":"2024-03-28T22:35:30.449741+0000","last_clean":"2024-03-28T22:35:30.449741+0000","last_became_active":"2024-03-28T22:33:28.928415+0000","last_became_peered":"2024-03-28T22:33:28.928415+0000","last_unstale":"2024-03-28T22:35:30.449741+0000","last_undegraded":"2024-03-28T22:35:30.449741+0000","last_fullsized":"2024-03-28T22:35:30.449741+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:00.839609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"94'15","reported_seq":47,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.252418+0000","last_change":"2024-03-28T22:33:28.924948+0000","last_active":"2024-03-28T22:35:30.252418+0000","last_peered":"2024-03-28T22:35:30.252418+0000","last_clean":"2024-03-28T22:35:30.252418+0000","last_became_active":"2024-03-28T22:33:28.924631+0000","last_became_peered":"2024-03-28T22:33:28.924631+0000","last_unstale":"2024-03-28T22:35:30.252418+0000","last_undegraded":"2024-03-28T22:35:30.252418+0000","last_fullsized":"2024-03-28T22:35:30.252418+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T09:53:44.909991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"94'2","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.372408+0000","last_change":"2024-03-28T22:33:28.948725+0000","last_active":"2024-03-28T22:35:30.372408+0000","last_peered":"2024-03-28T22:35:30.372408+0000","last_clean":"2024-03-28T22:35:30.372408+0000","last_became_active":"2024-03-28T22:33:28.943942+0000","last_became_peered":"2024-03-28T22:33:28.943942+0000","last_unstale":"2024-03-28T22:35:30.372408+0000","last_undegraded":"2024-03-28T22:35:30.372408+0000","last_fullsized":"2024-03-28T22:35:30.372408+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:50:04.238538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"94'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.275158+0000","last_change":"2024-03-28T22:33:28.941546+0000","last_active":"2024-03-28T22:35:30.275158+0000","last_peered":"2024-03-28T22:35:30.275158+0000","last_clean":"2024-03-28T22:35:30.275158+0000","last_became_active":"2024-03-28T22:33:28.941278+0000","last_became_peered":"2024-03-28T22:33:28.941278+0000","last_unstale":"2024-03-28T22:35:30.275158+0000","last_undegraded":"2024-03-28T22:35:30.275158+0000","last_fullsized":"2024-03-28T22:35:30.275158+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:47:10.047067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.372264+0000","last_change":"2024-03-28T22:33:28.949552+0000","last_active":"2024-03-28T22:35:30.372264+0000","last_peered":"2024-03-28T22:35:30.372264+0000","last_clean":"2024-03-28T22:35:30.372264+0000","last_became_active":"2024-03-28T22:33:28.947432+0000","last_became_peered":"2024-03-28T22:33:28.947432+0000","last_unstale":"2024-03-28T22:35:30.372264+0000","last_undegraded":"2024-03-28T22:35:30.372264+0000","last_fullsized":"2024-03-28T22:35:30.372264+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T08:01:16.534642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"94'10","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.448510+0000","last_change":"2024-03-28T22:33:28.953930+0000","last_active":"2024-03-28T22:35:30.448510+0000","last_peered":"2024-03-28T22:35:30.448510+0000","last_clean":"2024-03-28T22:35:30.448510+0000","last_became_active":"2024-03-28T22:33:28.953591+0000","last_became_peered":"2024-03-28T22:33:28.953591+0000","last_unstale":"2024-03-28T22:35:30.448510+0000","last_undegraded":"2024-03-28T22:35:30.448510+0000","last_fullsized":"2024-03-28T22:35:30.448510+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:42:55.857476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"94'11","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.368801+0000","last_change":"2024-03-28T22:33:28.946307+0000","last_active":"2024-03-28T22:35:30.368801+0000","last_peered":"2024-03-28T22:35:30.368801+0000","last_clean":"2024-03-28T22:35:30.368801+0000","last_became_active":"2024-03-28T22:33:28.945932+0000","last_became_peered":"2024-03-28T22:33:28.945932+0000","last_unstale":"2024-03-28T22:35:30.368801+0000","last_undegraded":"2024-03-28T22:35:30.368801+0000","last_fullsized":"2024-03-28T22:35:30.368801+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:54:59.651668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"94'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-28T22:35:30.449676+0000","last_change":"2024-03-28T22:33:28.928014+0000","last_active":"2024-03-28T22:35:30.449676+0000","last_peered":"2024-03-28T22:35:30.449676+0000","last_clean":"2024-03-28T22:35:30.449676+0000","last_became_active":"2024-03-28T22:33:28.927686+0000","last_became_peered":"2024-03-28T22:33:28.927686+0000","last_unstale":"2024-03-28T22:35:30.449676+0000","last_undegraded":"2024-03-28T22:35:30.449676+0000","last_fullsized":"2024-03-28T22:35:30.449676+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T22:33:28.412030+0000","last_clean_scrub_stamp":"2024-03-28T22:33:28.412030+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:47:08.131565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":81087,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1717,"num_read_kb":948,"num_write":265,"num_write_kb":165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":602112,"data_stored":243261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":229,"ondisk_log_size":229,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318327,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32260,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712892,"statfs":{"total":95995035648,"available":95962001408,"internally_reserved":0,"allocated":1105920,"data_stored":539947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579964,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28780,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716372,"statfs":{"total":95995035648,"available":95965564928,"internally_reserved":0,"allocated":1806336,"data_stored":1199927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743490,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1130496,"data_stored":543560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005128,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1118208,"data_stored":539820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299470,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37016,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708136,"statfs":{"total":95995035648,"available":95957131264,"internally_reserved":0,"allocated":1720320,"data_stored":1137177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495699,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1101824,"data_stored":534968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574959,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36488,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708664,"statfs":{"total":95995035648,"available":95957671936,"internally_reserved":0,"allocated":1179648,"data_stored":609043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738484,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1126400,"data_stored":536815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411436,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36508,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708644,"statfs":{"total":95995035648,"available":95957651456,"internally_reserved":0,"allocated":1200128,"data_stored":610351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182499,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36424,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708728,"statfs":{"total":95995035648,"available":95957737472,"internally_reserved":0,"allocated":1114112,"data_stored":534676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920862,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1110016,"data_stored":539574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757337,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28704,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716448,"statfs":{"total":95995035648,"available":95965642752,"internally_reserved":0,"allocated":1728512,"data_stored":1132227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":74797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":76105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":75313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T22:36:57.227 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:36:57.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:56 smithi172 ceph-mon[36566]: from='client.25553 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:36:57.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:56 smithi082 ceph-mon[36471]: from='client.25553 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:36:57.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.482+0000 7f3defe00640 1 -- 172.21.15.43:0/1897247246 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3df0104990 msgr2=0x7f3df0104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:57.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.482+0000 7f3defe00640 1 --2- 172.21.15.43:0/1897247246 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3df0104990 0x7f3df0104e10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8009900 tx=0x7f3dd802bc30 comp rx=0 tx=0).stop 2024-03-28T22:36:57.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.482+0000 7f3defe00640 1 -- 172.21.15.43:0/1897247246 shutdown_connections 2024-03-28T22:36:57.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.482+0000 7f3defe00640 1 --2- 172.21.15.43:0/1897247246 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3df0105350 0x7f3df010bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:57.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.482+0000 7f3defe00640 1 --2- 172.21.15.43:0/1897247246 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3df0104990 0x7f3df0104e10 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:57.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.482+0000 7f3defe00640 1 --2- 172.21.15.43:0/1897247246 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3df01037a0 0x7f3df0103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:57.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.482+0000 7f3defe00640 1 -- 172.21.15.43:0/1897247246 >> 172.21.15.43:0/1897247246 conn(0x7f3df00fed30 msgr2=0x7f3df0101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:57.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.482+0000 7f3defe00640 1 -- 172.21.15.43:0/1897247246 shutdown_connections 2024-03-28T22:36:57.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.482+0000 7f3defe00640 1 -- 172.21.15.43:0/1897247246 wait complete. 2024-03-28T22:36:57.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.483+0000 7f3defe00640 1 Processor -- start 2024-03-28T22:36:57.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.483+0000 7f3defe00640 1 -- start start 2024-03-28T22:36:57.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.483+0000 7f3defe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3df01037a0 0x7f3df0199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:57.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.483+0000 7f3defe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3df0104990 0x7f3df0199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:57.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.483+0000 7f3defe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3df0105350 0x7f3df01a1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:57.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.483+0000 7f3defe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3df010f400 con 0x7f3df0104990 2024-03-28T22:36:57.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.483+0000 7f3defe00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f3df010f280 con 0x7f3df01037a0 2024-03-28T22:36:57.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.483+0000 7f3defe00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f3df010f580 con 0x7f3df0105350 2024-03-28T22:36:57.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.484+0000 7f3deea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3df01037a0 0x7f3df0199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:57.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.484+0000 7f3de6000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3df0104990 0x7f3df0199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:57.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.484+0000 7f3deea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3df01037a0 0x7f3df0199a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:37422/0 (socket says 172.21.15.43:37422) 2024-03-28T22:36:57.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.484+0000 7f3deea00640 1 -- 172.21.15.43:0/2236424359 learned_addr learned my addr 172.21.15.43:0/2236424359 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:57.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.484+0000 7f3def400640 1 --2- 172.21.15.43:0/2236424359 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3df0105350 0x7f3df01a1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:57.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.484+0000 7f3deea00640 1 -- 172.21.15.43:0/2236424359 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3df0105350 msgr2=0x7f3df01a1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:57.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.484+0000 7f3deea00640 1 --2- 172.21.15.43:0/2236424359 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3df0105350 0x7f3df01a1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:57.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.484+0000 7f3deea00640 1 -- 172.21.15.43:0/2236424359 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3df0104990 msgr2=0x7f3df0199fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:57.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.484+0000 7f3deea00640 1 --2- 172.21.15.43:0/2236424359 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3df0104990 0x7f3df0199fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:57.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.484+0000 7f3deea00640 1 -- 172.21.15.43:0/2236424359 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3df01a1690 con 0x7f3df01037a0 2024-03-28T22:36:57.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.484+0000 7f3deea00640 1 --2- 172.21.15.43:0/2236424359 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3df01037a0 0x7f3df0199a70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f3de000d770 tx=0x7f3de000dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:57.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.484+0000 7f3de7e00640 1 -- 172.21.15.43:0/2236424359 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de0017070 con 0x7f3df01037a0 2024-03-28T22:36:57.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.484+0000 7f3de7e00640 1 -- 172.21.15.43:0/2236424359 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3de0009c30 con 0x7f3df01037a0 2024-03-28T22:36:57.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.484+0000 7f3defe00640 1 -- 172.21.15.43:0/2236424359 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3df01a1980 con 0x7f3df01037a0 2024-03-28T22:36:57.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.485+0000 7f3de7e00640 1 -- 172.21.15.43:0/2236424359 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de0003dd0 con 0x7f3df01037a0 2024-03-28T22:36:57.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.485+0000 7f3defe00640 1 -- 172.21.15.43:0/2236424359 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3df01a1ef0 con 0x7f3df01037a0 2024-03-28T22:36:57.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.486+0000 7f3defe00640 1 -- 172.21.15.43:0/2236424359 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3df010ab40 con 0x7f3df01037a0 2024-03-28T22:36:57.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.487+0000 7f3de7e00640 1 -- 172.21.15.43:0/2236424359 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7f3de0003fb0 con 0x7f3df01037a0 2024-03-28T22:36:57.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.489+0000 7f3de7e00640 1 --2- 172.21.15.43:0/2236424359 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3dd40a1e90 0x7f3dd40a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:57.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.489+0000 7f3de7e00640 1 -- 172.21.15.43:0/2236424359 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f3de00cdb60 con 0x7f3df01037a0 2024-03-28T22:36:57.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.489+0000 7f3de6000640 1 --2- 172.21.15.43:0/2236424359 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3dd40a1e90 0x7f3dd40a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:57.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.490+0000 7f3de6000640 1 --2- 172.21.15.43:0/2236424359 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3dd40a1e90 0x7f3dd40a4350 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3dd802c7b0 tx=0x7f3dd8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:57.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.493+0000 7f3de7e00640 1 -- 172.21.15.43:0/2236424359 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3de0009de0 con 0x7f3df01037a0 2024-03-28T22:36:57.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.698+0000 7f3defe00640 1 -- 172.21.15.43:0/2236424359 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3df002d380 con 0x7f3df01037a0 2024-03-28T22:36:57.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.701+0000 7f3de7e00640 1 -- 172.21.15.43:0/2236424359 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+24712 (secure 0 0 0) 0x7f3de009a430 con 0x7f3df01037a0 2024-03-28T22:36:57.699 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:36:57.700 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":96,"fsid":"c2276428-ed51-11ee-95d0-87774f69a715","created":"2024-03-28T22:24:19.950084+0000","modified":"2024-03-28T22:35:30.237654+0000","last_up_change":"2024-03-28T22:32:31.078737+0000","last_in_change":"2024-03-28T22:32:10.147097+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:28:14.713290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:33:27.167435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:33:28.411825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T22:33:32.523122+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b855adcd-786c-4e42-b8fe-a515a77ce979","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6803","nonce":2826124409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6805","nonce":2826124409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6809","nonce":2826124409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6807","nonce":2826124409}]},"public_addr":"172.21.15.43:6803/2826124409","cluster_addr":"172.21.15.43:6805/2826124409","heartbeat_back_addr":"172.21.15.43:6809/2826124409","heartbeat_front_addr":"172.21.15.43:6807/2826124409","state":["exists","up"]},{"osd":1,"uuid":"031c143d-8ad1-4b68-8db9-3c717137e3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6801","nonce":3043263644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6803","nonce":3043263644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6807","nonce":3043263644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6805","nonce":3043263644}]},"public_addr":"172.21.15.82:6801/3043263644","cluster_addr":"172.21.15.82:6803/3043263644","heartbeat_back_addr":"172.21.15.82:6807/3043263644","heartbeat_front_addr":"172.21.15.82:6805/3043263644","state":["exists","up"]},{"osd":2,"uuid":"20632bbb-f07a-4ca1-b0c4-ff9d508e211c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6801","nonce":4213953645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6803","nonce":4213953645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6807","nonce":4213953645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6805","nonce":4213953645}]},"public_addr":"172.21.15.172:6801/4213953645","cluster_addr":"172.21.15.172:6803/4213953645","heartbeat_back_addr":"172.21.15.172:6807/4213953645","heartbeat_front_addr":"172.21.15.172:6805/4213953645","state":["exists","up"]},{"osd":3,"uuid":"9a1f2312-4dbd-475b-8736-d5ebc8788e09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6811","nonce":1597280920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6813","nonce":1597280920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6817","nonce":1597280920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6815","nonce":1597280920}]},"public_addr":"172.21.15.43:6811/1597280920","cluster_addr":"172.21.15.43:6813/1597280920","heartbeat_back_addr":"172.21.15.43:6817/1597280920","heartbeat_front_addr":"172.21.15.43:6815/1597280920","state":["exists","up"]},{"osd":4,"uuid":"29140ead-7075-42b2-90c1-27cae0b29ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6809","nonce":215451895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6811","nonce":215451895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6815","nonce":215451895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6813","nonce":215451895}]},"public_addr":"172.21.15.82:6809/215451895","cluster_addr":"172.21.15.82:6811/215451895","heartbeat_back_addr":"172.21.15.82:6815/215451895","heartbeat_front_addr":"172.21.15.82:6813/215451895","state":["exists","up"]},{"osd":5,"uuid":"59644c61-93dc-4536-b2ad-0256b24e5cb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6809","nonce":1576148933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6811","nonce":1576148933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6815","nonce":1576148933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6813","nonce":1576148933}]},"public_addr":"172.21.15.172:6809/1576148933","cluster_addr":"172.21.15.172:6811/1576148933","heartbeat_back_addr":"172.21.15.172:6815/1576148933","heartbeat_front_addr":"172.21.15.172:6813/1576148933","state":["exists","up"]},{"osd":6,"uuid":"22d54e57-c3f1-4ecf-9921-8e815b48f92a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6819","nonce":791044279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6821","nonce":791044279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6825","nonce":791044279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6823","nonce":791044279}]},"public_addr":"172.21.15.43:6819/791044279","cluster_addr":"172.21.15.43:6821/791044279","heartbeat_back_addr":"172.21.15.43:6825/791044279","heartbeat_front_addr":"172.21.15.43:6823/791044279","state":["exists","up"]},{"osd":7,"uuid":"248cd1e2-0d6b-452f-a0a8-c51fe1b975ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6817","nonce":2157777323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6819","nonce":2157777323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6823","nonce":2157777323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6821","nonce":2157777323}]},"public_addr":"172.21.15.82:6817/2157777323","cluster_addr":"172.21.15.82:6819/2157777323","heartbeat_back_addr":"172.21.15.82:6823/2157777323","heartbeat_front_addr":"172.21.15.82:6821/2157777323","state":["exists","up"]},{"osd":8,"uuid":"d98aab6e-8bae-4c24-906c-4f492f047b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6817","nonce":1095328248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6819","nonce":1095328248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6823","nonce":1095328248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6821","nonce":1095328248}]},"public_addr":"172.21.15.172:6817/1095328248","cluster_addr":"172.21.15.172:6819/1095328248","heartbeat_back_addr":"172.21.15.172:6823/1095328248","heartbeat_front_addr":"172.21.15.172:6821/1095328248","state":["exists","up"]},{"osd":9,"uuid":"19aa383e-a0f6-45ea-96cc-e7956535caa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6827","nonce":3657791498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6829","nonce":3657791498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6833","nonce":3657791498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6831","nonce":3657791498}]},"public_addr":"172.21.15.43:6827/3657791498","cluster_addr":"172.21.15.43:6829/3657791498","heartbeat_back_addr":"172.21.15.43:6833/3657791498","heartbeat_front_addr":"172.21.15.43:6831/3657791498","state":["exists","up"]},{"osd":10,"uuid":"666977e5-eaf7-41b9-a18f-c380e149345d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6825","nonce":3454954047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6827","nonce":3454954047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6831","nonce":3454954047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6829","nonce":3454954047}]},"public_addr":"172.21.15.82:6825/3454954047","cluster_addr":"172.21.15.82:6827/3454954047","heartbeat_back_addr":"172.21.15.82:6831/3454954047","heartbeat_front_addr":"172.21.15.82:6829/3454954047","state":["exists","up"]},{"osd":11,"uuid":"615cf3be-8076-470e-ad09-c880e7593717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6825","nonce":2563517144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6827","nonce":2563517144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6831","nonce":2563517144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6829","nonce":2563517144}]},"public_addr":"172.21.15.172:6825/2563517144","cluster_addr":"172.21.15.172:6827/2563517144","heartbeat_back_addr":"172.21.15.172:6831/2563517144","heartbeat_front_addr":"172.21.15.172:6829/2563517144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:21.727405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:47.100259+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:10.615438+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:43.632144+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:09.487940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:34.604463+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:09.594452+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:35.709795+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:01.184202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:35.455322+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:02.413166+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:28.550754+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.9","mappings":[{"from":5,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/578791143":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/4029313215":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/1574178482":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6801/3517712438":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6800/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/1375099638":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/2710117652":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6801/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/3215634168":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6800/3517712438":"2024-03-29T22:25:28.008215+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:36:57.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.703+0000 7f3defe00640 1 -- 172.21.15.43:0/2236424359 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3dd40a1e90 msgr2=0x7f3dd40a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:57.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.703+0000 7f3defe00640 1 --2- 172.21.15.43:0/2236424359 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3dd40a1e90 0x7f3dd40a4350 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3dd802c7b0 tx=0x7f3dd8005fd0 comp rx=0 tx=0).stop 2024-03-28T22:36:57.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.703+0000 7f3defe00640 1 -- 172.21.15.43:0/2236424359 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3df01037a0 msgr2=0x7f3df0199a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:57.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.703+0000 7f3defe00640 1 --2- 172.21.15.43:0/2236424359 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3df01037a0 0x7f3df0199a70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f3de000d770 tx=0x7f3de000dc30 comp rx=0 tx=0).stop 2024-03-28T22:36:57.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.703+0000 7f3defe00640 1 -- 172.21.15.43:0/2236424359 shutdown_connections 2024-03-28T22:36:57.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.703+0000 7f3defe00640 1 --2- 172.21.15.43:0/2236424359 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3df0105350 0x7f3df01a1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:57.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.703+0000 7f3defe00640 1 --2- 172.21.15.43:0/2236424359 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3df0104990 0x7f3df0199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:57.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.703+0000 7f3defe00640 1 --2- 172.21.15.43:0/2236424359 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f3dd40a1e90 0x7f3dd40a4350 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:57.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.703+0000 7f3defe00640 1 --2- 172.21.15.43:0/2236424359 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3df01037a0 0x7f3df0199a70 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:57.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.703+0000 7f3defe00640 1 -- 172.21.15.43:0/2236424359 >> 172.21.15.43:0/2236424359 conn(0x7f3df00fed30 msgr2=0x7f3df0107d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:57.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.703+0000 7f3defe00640 1 -- 172.21.15.43:0/2236424359 shutdown_connections 2024-03-28T22:36:57.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.703+0000 7f3defe00640 1 -- 172.21.15.43:0/2236424359 wait complete. 2024-03-28T22:36:57.721 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T22:28:14.713290+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T22:33:27.167435+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T22:33:28.411825+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T22:33:32.523122+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T22:36:57.721 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T22:36:57.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.941+0000 7f7433200640 1 -- 172.21.15.43:0/3357683851 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7434104a60 msgr2=0x7f7434104ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:57.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.941+0000 7f7433200640 1 --2- 172.21.15.43:0/3357683851 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7434104a60 0x7f7434104ee0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f741c009900 tx=0x7f741c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:36:57.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.941+0000 7f7433200640 1 -- 172.21.15.43:0/3357683851 shutdown_connections 2024-03-28T22:36:57.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.941+0000 7f7433200640 1 --2- 172.21.15.43:0/3357683851 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7434105420 0x7f743410bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:57.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.941+0000 7f7433200640 1 --2- 172.21.15.43:0/3357683851 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7434104a60 0x7f7434104ee0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:57.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.941+0000 7f7433200640 1 --2- 172.21.15.43:0/3357683851 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7434103870 0x7f7434103c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:57.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.941+0000 7f7433200640 1 -- 172.21.15.43:0/3357683851 >> 172.21.15.43:0/3357683851 conn(0x7f74340fee00 msgr2=0x7f7434101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:57.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.942+0000 7f7433200640 1 -- 172.21.15.43:0/3357683851 shutdown_connections 2024-03-28T22:36:57.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.942+0000 7f7433200640 1 -- 172.21.15.43:0/3357683851 wait complete. 2024-03-28T22:36:57.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.943+0000 7f7433200640 1 Processor -- start 2024-03-28T22:36:57.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.943+0000 7f7433200640 1 -- start start 2024-03-28T22:36:57.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.943+0000 7f7433200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7434103870 0x7f7434199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:57.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.944+0000 7f7433200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7434104a60 0x7f743419a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:57.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.944+0000 7f7433200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7434105420 0x7f74341a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:57.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.944+0000 7f7433200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f743410f520 con 0x7f7434105420 2024-03-28T22:36:57.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.944+0000 7f7433200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f743410f3a0 con 0x7f7434103870 2024-03-28T22:36:57.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.944+0000 7f7433200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f743410f6a0 con 0x7f7434104a60 2024-03-28T22:36:57.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.944+0000 7f7431400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7434104a60 0x7f743419a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:57.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.944+0000 7f7431400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7434104a60 0x7f743419a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:46794/0 (socket says 172.21.15.43:46794) 2024-03-28T22:36:57.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.944+0000 7f7432800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7434105420 0x7f74341a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:57.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.944+0000 7f7431400640 1 -- 172.21.15.43:0/4029641025 learned_addr learned my addr 172.21.15.43:0/4029641025 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:57.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.944+0000 7f7431e00640 1 --2- 172.21.15.43:0/4029641025 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7434103870 0x7f7434199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:57.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.944+0000 7f7431400640 1 -- 172.21.15.43:0/4029641025 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7434103870 msgr2=0x7f7434199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:57.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.944+0000 7f7431400640 1 --2- 172.21.15.43:0/4029641025 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7434103870 0x7f7434199b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:57.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.944+0000 7f7431400640 1 -- 172.21.15.43:0/4029641025 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7434105420 msgr2=0x7f74341a1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:57.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.944+0000 7f7431400640 1 --2- 172.21.15.43:0/4029641025 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7434105420 0x7f74341a1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:57.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.944+0000 7f7431400640 1 -- 172.21.15.43:0/4029641025 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74341a1790 con 0x7f7434104a60 2024-03-28T22:36:57.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.945+0000 7f7431400640 1 --2- 172.21.15.43:0/4029641025 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7434104a60 0x7f743419a0b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f741c000c00 tx=0x7f741c02c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:57.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.945+0000 7f742aa00640 1 -- 172.21.15.43:0/4029641025 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741c043020 con 0x7f7434104a60 2024-03-28T22:36:57.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.945+0000 7f742aa00640 1 -- 172.21.15.43:0/4029641025 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f741c0054e0 con 0x7f7434104a60 2024-03-28T22:36:57.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.945+0000 7f742aa00640 1 -- 172.21.15.43:0/4029641025 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741c003bd0 con 0x7f7434104a60 2024-03-28T22:36:57.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.945+0000 7f7433200640 1 -- 172.21.15.43:0/4029641025 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74341a1a20 con 0x7f7434104a60 2024-03-28T22:36:57.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.945+0000 7f7433200640 1 -- 172.21.15.43:0/4029641025 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74341a1fc0 con 0x7f7434104a60 2024-03-28T22:36:57.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.946+0000 7f7433200640 1 -- 172.21.15.43:0/4029641025 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f743410ab90 con 0x7f7434104a60 2024-03-28T22:36:57.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.948+0000 7f742aa00640 1 -- 172.21.15.43:0/4029641025 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7f741c046020 con 0x7f7434104a60 2024-03-28T22:36:57.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.949+0000 7f742aa00640 1 --2- 172.21.15.43:0/4029641025 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f74180a1e90 0x7f74180a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:57.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.950+0000 7f742aa00640 1 -- 172.21.15.43:0/4029641025 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f741c0f02c0 con 0x7f7434104a60 2024-03-28T22:36:57.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.950+0000 7f7431e00640 1 --2- 172.21.15.43:0/4029641025 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f74180a1e90 0x7f74180a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:57.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.950+0000 7f7431e00640 1 --2- 172.21.15.43:0/4029641025 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f74180a1e90 0x7f74180a4350 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f74341048c0 tx=0x7f7424005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:57.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:57.954+0000 7f742aa00640 1 -- 172.21.15.43:0/4029641025 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f741c04d020 con 0x7f7434104a60 2024-03-28T22:36:58.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:57 smithi043 ceph-mon[31823]: pgmap v430: 129 pgs: 129 active+clean; 656 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-03-28T22:36:58.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:57 smithi043 ceph-mon[31823]: from='client.25564 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:36:58.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:57 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2236424359' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:36:58.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.156+0000 7f7433200640 1 -- 172.21.15.43:0/4029641025 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f743402d390 con 0x7f7434104a60 2024-03-28T22:36:58.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.157+0000 7f742aa00640 1 -- 172.21.15.43:0/4029641025 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) ==== 93+0+10 (secure 0 0 0) 0x7f741c0bd2b0 con 0x7f7434104a60 2024-03-28T22:36:58.157 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-03-28T22:36:58.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.160+0000 7f7433200640 1 -- 172.21.15.43:0/4029641025 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f74180a1e90 msgr2=0x7f74180a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:58.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.160+0000 7f7433200640 1 --2- 172.21.15.43:0/4029641025 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f74180a1e90 0x7f74180a4350 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f74341048c0 tx=0x7f7424005f90 comp rx=0 tx=0).stop 2024-03-28T22:36:58.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.160+0000 7f7433200640 1 -- 172.21.15.43:0/4029641025 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7434104a60 msgr2=0x7f743419a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:58.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.160+0000 7f7433200640 1 --2- 172.21.15.43:0/4029641025 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7434104a60 0x7f743419a0b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f741c000c00 tx=0x7f741c02c650 comp rx=0 tx=0).stop 2024-03-28T22:36:58.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.160+0000 7f7433200640 1 -- 172.21.15.43:0/4029641025 shutdown_connections 2024-03-28T22:36:58.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.160+0000 7f7433200640 1 --2- 172.21.15.43:0/4029641025 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7434105420 0x7f74341a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:58.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.160+0000 7f7433200640 1 --2- 172.21.15.43:0/4029641025 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7434104a60 0x7f743419a0b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:58.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.160+0000 7f7433200640 1 --2- 172.21.15.43:0/4029641025 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f74180a1e90 0x7f74180a4350 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:58.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.160+0000 7f7433200640 1 --2- 172.21.15.43:0/4029641025 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7434103870 0x7f7434199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:58.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.160+0000 7f7433200640 1 -- 172.21.15.43:0/4029641025 >> 172.21.15.43:0/4029641025 conn(0x7f74340fee00 msgr2=0x7f7434107d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:58.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.160+0000 7f7433200640 1 -- 172.21.15.43:0/4029641025 shutdown_connections 2024-03-28T22:36:58.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.160+0000 7f7433200640 1 -- 172.21.15.43:0/4029641025 wait complete. 2024-03-28T22:36:58.179 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T22:36:58.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:57 smithi172 ceph-mon[36566]: pgmap v430: 129 pgs: 129 active+clean; 656 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-03-28T22:36:58.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:57 smithi172 ceph-mon[36566]: from='client.25564 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:36:58.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:57 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2236424359' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:36:58.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.410+0000 7f32a3e00640 1 -- 172.21.15.43:0/546958694 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f329c105520 msgr2=0x7f329c10bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:58.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.410+0000 7f32a3e00640 1 --2- 172.21.15.43:0/546958694 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f329c105520 0x7f329c10bdb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3294009ca0 tx=0x7f329402c100 comp rx=0 tx=0).stop 2024-03-28T22:36:58.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.410+0000 7f32a3e00640 1 -- 172.21.15.43:0/546958694 shutdown_connections 2024-03-28T22:36:58.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.410+0000 7f32a3e00640 1 --2- 172.21.15.43:0/546958694 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f329c104b60 0x7f329c104fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:58.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.410+0000 7f32a3e00640 1 --2- 172.21.15.43:0/546958694 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f329c105520 0x7f329c10bdb0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:58.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.410+0000 7f32a3e00640 1 --2- 172.21.15.43:0/546958694 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f329c103970 0x7f329c103d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:58.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.410+0000 7f32a3e00640 1 -- 172.21.15.43:0/546958694 >> 172.21.15.43:0/546958694 conn(0x7f329c0fef00 msgr2=0x7f329c101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:58.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.411+0000 7f32a3e00640 1 -- 172.21.15.43:0/546958694 shutdown_connections 2024-03-28T22:36:58.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.411+0000 7f32a3e00640 1 -- 172.21.15.43:0/546958694 wait complete. 2024-03-28T22:36:58.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.412+0000 7f32a3e00640 1 Processor -- start 2024-03-28T22:36:58.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.412+0000 7f32a3e00640 1 -- start start 2024-03-28T22:36:58.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.412+0000 7f32a3e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f329c103970 0x7f329c199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:58.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.412+0000 7f32a3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f329c104b60 0x7f329c19a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:58.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.412+0000 7f32a3e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f329c105520 0x7f329c1a1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:58.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.412+0000 7f32a3e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f329c10f5d0 con 0x7f329c104b60 2024-03-28T22:36:58.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.412+0000 7f32a3e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f329c10f450 con 0x7f329c105520 2024-03-28T22:36:58.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.412+0000 7f32a3e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f329c10f750 con 0x7f329c103970 2024-03-28T22:36:58.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.412+0000 7f32a1400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f329c103970 0x7f329c199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:58.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.412+0000 7f32a1400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f329c103970 0x7f329c199bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:46814/0 (socket says 172.21.15.43:46814) 2024-03-28T22:36:58.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.412+0000 7f32a1400640 1 -- 172.21.15.43:0/4070740037 learned_addr learned my addr 172.21.15.43:0/4070740037 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:58.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.412+0000 7f32a1e00640 1 --2- 172.21.15.43:0/4070740037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f329c105520 0x7f329c1a1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:58.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.413+0000 7f32a0a00640 1 --2- 172.21.15.43:0/4070740037 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f329c104b60 0x7f329c19a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:58.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.413+0000 7f32a1400640 1 -- 172.21.15.43:0/4070740037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f329c105520 msgr2=0x7f329c1a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:58.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.413+0000 7f32a1400640 1 --2- 172.21.15.43:0/4070740037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f329c105520 0x7f329c1a1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:58.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.413+0000 7f32a1400640 1 -- 172.21.15.43:0/4070740037 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f329c104b60 msgr2=0x7f329c19a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:58.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.413+0000 7f32a1400640 1 --2- 172.21.15.43:0/4070740037 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f329c104b60 0x7f329c19a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:58.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.413+0000 7f32a1400640 1 -- 172.21.15.43:0/4070740037 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f329c1a17d0 con 0x7f329c103970 2024-03-28T22:36:58.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.413+0000 7f32a1e00640 1 --2- 172.21.15.43:0/4070740037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f329c105520 0x7f329c1a1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:36:58.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.413+0000 7f32a1400640 1 --2- 172.21.15.43:0/4070740037 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f329c103970 0x7f329c199bb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f329000d770 tx=0x7f329000dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:58.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.413+0000 7f329a000640 1 -- 172.21.15.43:0/4070740037 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3290017070 con 0x7f329c103970 2024-03-28T22:36:58.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.413+0000 7f329a000640 1 -- 172.21.15.43:0/4070740037 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3290009c30 con 0x7f329c103970 2024-03-28T22:36:58.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.413+0000 7f32a3e00640 1 -- 172.21.15.43:0/4070740037 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f329c1a1ac0 con 0x7f329c103970 2024-03-28T22:36:58.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.414+0000 7f329a000640 1 -- 172.21.15.43:0/4070740037 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3290003dd0 con 0x7f329c103970 2024-03-28T22:36:58.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.414+0000 7f32a3e00640 1 -- 172.21.15.43:0/4070740037 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f329c1a2010 con 0x7f329c103970 2024-03-28T22:36:58.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.415+0000 7f32a3e00640 1 -- 172.21.15.43:0/4070740037 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f329c10ac90 con 0x7f329c103970 2024-03-28T22:36:58.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.417+0000 7f329a000640 1 -- 172.21.15.43:0/4070740037 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7f3290003f70 con 0x7f329c103970 2024-03-28T22:36:58.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.418+0000 7f329a000640 1 --2- 172.21.15.43:0/4070740037 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f32840a1e10 0x7f32840a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:58.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.418+0000 7f329a000640 1 -- 172.21.15.43:0/4070740037 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f32900cdca0 con 0x7f329c103970 2024-03-28T22:36:58.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.418+0000 7f32a0a00640 1 --2- 172.21.15.43:0/4070740037 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f32840a1e10 0x7f32840a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:58.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.419+0000 7f32a0a00640 1 --2- 172.21.15.43:0/4070740037 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f32840a1e10 0x7f32840a42d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f329c19b0c0 tx=0x7f3288005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:58.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.427+0000 7f329a000640 1 -- 172.21.15.43:0/4070740037 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f329009ab80 con 0x7f329c103970 2024-03-28T22:36:58.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:57 smithi082 ceph-mon[36471]: pgmap v430: 129 pgs: 129 active+clean; 656 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-03-28T22:36:58.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:57 smithi082 ceph-mon[36471]: from='client.25564 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:36:58.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:57 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2236424359' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:36:58.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.628+0000 7f32a3e00640 1 -- 172.21.15.43:0/4070740037 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f329c02d3d0 con 0x7f329c103970 2024-03-28T22:36:58.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.628+0000 7f329a000640 1 -- 172.21.15.43:0/4070740037 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) ==== 104+0+11 (secure 0 0 0) 0x7f3290009de0 con 0x7f329c103970 2024-03-28T22:36:58.627 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-03-28T22:36:58.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.631+0000 7f32a3e00640 1 -- 172.21.15.43:0/4070740037 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f32840a1e10 msgr2=0x7f32840a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:58.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.631+0000 7f32a3e00640 1 --2- 172.21.15.43:0/4070740037 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f32840a1e10 0x7f32840a42d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f329c19b0c0 tx=0x7f3288005f90 comp rx=0 tx=0).stop 2024-03-28T22:36:58.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.631+0000 7f32a3e00640 1 -- 172.21.15.43:0/4070740037 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f329c103970 msgr2=0x7f329c199bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:58.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.631+0000 7f32a3e00640 1 --2- 172.21.15.43:0/4070740037 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f329c103970 0x7f329c199bb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f329000d770 tx=0x7f329000dc30 comp rx=0 tx=0).stop 2024-03-28T22:36:58.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.631+0000 7f32a3e00640 1 -- 172.21.15.43:0/4070740037 shutdown_connections 2024-03-28T22:36:58.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.631+0000 7f32a3e00640 1 --2- 172.21.15.43:0/4070740037 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f32840a1e10 0x7f32840a42d0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:58.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.631+0000 7f32a3e00640 1 --2- 172.21.15.43:0/4070740037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f329c105520 0x7f329c1a1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:58.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.631+0000 7f32a3e00640 1 --2- 172.21.15.43:0/4070740037 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f329c104b60 0x7f329c19a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:58.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.631+0000 7f32a3e00640 1 --2- 172.21.15.43:0/4070740037 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f329c103970 0x7f329c199bb0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:58.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.631+0000 7f32a3e00640 1 -- 172.21.15.43:0/4070740037 >> 172.21.15.43:0/4070740037 conn(0x7f329c0fef00 msgr2=0x7f329c107e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:58.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.632+0000 7f32a3e00640 1 -- 172.21.15.43:0/4070740037 shutdown_connections 2024-03-28T22:36:58.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.632+0000 7f32a3e00640 1 -- 172.21.15.43:0/4070740037 wait complete. 2024-03-28T22:36:58.653 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T22:36:58.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.884+0000 7f7ef9a00640 1 -- 172.21.15.43:0/2805967658 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ef4108e60 msgr2=0x7f7ef41092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:58.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.884+0000 7f7ef9a00640 1 --2- 172.21.15.43:0/2805967658 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ef4108e60 0x7f7ef41092e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7edc009900 tx=0x7f7edc02bc30 comp rx=0 tx=0).stop 2024-03-28T22:36:58.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.884+0000 7f7ef9a00640 1 -- 172.21.15.43:0/2805967658 shutdown_connections 2024-03-28T22:36:58.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.884+0000 7f7ef9a00640 1 --2- 172.21.15.43:0/2805967658 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ef4108e60 0x7f7ef41092e0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:58.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.884+0000 7f7ef9a00640 1 --2- 172.21.15.43:0/2805967658 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ef4109820 0x7f7ef41100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:58.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.884+0000 7f7ef9a00640 1 --2- 172.21.15.43:0/2805967658 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ef4107c70 0x7f7ef4108090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:58.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.884+0000 7f7ef9a00640 1 -- 172.21.15.43:0/2805967658 >> 172.21.15.43:0/2805967658 conn(0x7f7ef4076c20 msgr2=0x7f7ef4079040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:58.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.884+0000 7f7ef9a00640 1 -- 172.21.15.43:0/2805967658 shutdown_connections 2024-03-28T22:36:58.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.884+0000 7f7ef9a00640 1 -- 172.21.15.43:0/2805967658 wait complete. 2024-03-28T22:36:58.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.885+0000 7f7ef9a00640 1 Processor -- start 2024-03-28T22:36:58.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.885+0000 7f7ef9a00640 1 -- start start 2024-03-28T22:36:58.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.886+0000 7f7ef9a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ef4107c70 0x7f7ef419dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:58.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.886+0000 7f7ef9a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ef4108e60 0x7f7ef419e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:58.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.886+0000 7f7ef9a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ef4109820 0x7f7ef41a54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:58.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.886+0000 7f7ef9a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef41138d0 con 0x7f7ef4108e60 2024-03-28T22:36:58.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.886+0000 7f7ef9a00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef4113750 con 0x7f7ef4107c70 2024-03-28T22:36:58.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.886+0000 7f7ef9a00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef4113a50 con 0x7f7ef4109820 2024-03-28T22:36:58.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.886+0000 7f7ef3e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ef4107c70 0x7f7ef419dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:58.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.886+0000 7f7ef3e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ef4107c70 0x7f7ef419dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:37454/0 (socket says 172.21.15.43:37454) 2024-03-28T22:36:58.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.886+0000 7f7ef3e00640 1 -- 172.21.15.43:0/3679623676 learned_addr learned my addr 172.21.15.43:0/3679623676 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:58.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.886+0000 7f7ef9000640 1 --2- 172.21.15.43:0/3679623676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ef4109820 0x7f7ef41a54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:58.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.886+0000 7f7eebe00640 1 --2- 172.21.15.43:0/3679623676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ef4108e60 0x7f7ef419e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:58.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.886+0000 7f7ef3e00640 1 -- 172.21.15.43:0/3679623676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ef4109820 msgr2=0x7f7ef41a54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:58.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.886+0000 7f7ef3e00640 1 --2- 172.21.15.43:0/3679623676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ef4109820 0x7f7ef41a54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:58.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.886+0000 7f7ef3e00640 1 -- 172.21.15.43:0/3679623676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ef4108e60 msgr2=0x7f7ef419e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:58.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.886+0000 7f7ef3e00640 1 --2- 172.21.15.43:0/3679623676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ef4108e60 0x7f7ef419e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:58.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.886+0000 7f7ef3e00640 1 -- 172.21.15.43:0/3679623676 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ef41a5b20 con 0x7f7ef4107c70 2024-03-28T22:36:58.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.887+0000 7f7ef3e00640 1 --2- 172.21.15.43:0/3679623676 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ef4107c70 0x7f7ef419dfa0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7ee400e9c0 tx=0x7f7ee400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:58.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.887+0000 7f7ef1600640 1 -- 172.21.15.43:0/3679623676 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee400cd80 con 0x7f7ef4107c70 2024-03-28T22:36:58.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.887+0000 7f7ef1600640 1 -- 172.21.15.43:0/3679623676 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ee40035f0 con 0x7f7ef4107c70 2024-03-28T22:36:58.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.887+0000 7f7ef1600640 1 -- 172.21.15.43:0/3679623676 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4010690 con 0x7f7ef4107c70 2024-03-28T22:36:58.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.887+0000 7f7ef9a00640 1 -- 172.21.15.43:0/3679623676 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ef41a5e10 con 0x7f7ef4107c70 2024-03-28T22:36:58.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.887+0000 7f7ef9a00640 1 -- 172.21.15.43:0/3679623676 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ef41a6270 con 0x7f7ef4107c70 2024-03-28T22:36:58.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.888+0000 7f7ef9a00640 1 -- 172.21.15.43:0/3679623676 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ef410ef90 con 0x7f7ef4107c70 2024-03-28T22:36:58.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.890+0000 7f7ef1600640 1 -- 172.21.15.43:0/3679623676 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7f7ee40037a0 con 0x7f7ef4107c70 2024-03-28T22:36:58.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.892+0000 7f7ef1600640 1 --2- 172.21.15.43:0/3679623676 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7ed80a1e10 0x7f7ed80a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:58.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.892+0000 7f7ef1600640 1 -- 172.21.15.43:0/3679623676 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f7ee40ce760 con 0x7f7ef4107c70 2024-03-28T22:36:58.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.892+0000 7f7eebe00640 1 --2- 172.21.15.43:0/3679623676 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7ed80a1e10 0x7f7ed80a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:58.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.893+0000 7f7eebe00640 1 --2- 172.21.15.43:0/3679623676 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7ed80a1e10 0x7f7ed80a42d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7edc006010 tx=0x7f7edc000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:58.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:58.896+0000 7f7ef1600640 1 -- 172.21.15.43:0/3679623676 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7ee409b640 con 0x7f7ef4107c70 2024-03-28T22:36:59.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.099+0000 7f7ef9a00640 1 -- 172.21.15.43:0/3679623676 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7ef402d170 con 0x7f7ef4107c70 2024-03-28T22:36:59.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.099+0000 7f7ef1600640 1 -- 172.21.15.43:0/3679623676 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) ==== 100+0+11 (secure 0 0 0) 0x7f7ef402d170 con 0x7f7ef4107c70 2024-03-28T22:36:59.098 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-03-28T22:36:59.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.102+0000 7f7ef9a00640 1 -- 172.21.15.43:0/3679623676 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7ed80a1e10 msgr2=0x7f7ed80a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:59.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.102+0000 7f7ef9a00640 1 --2- 172.21.15.43:0/3679623676 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7ed80a1e10 0x7f7ed80a42d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7edc006010 tx=0x7f7edc000bc0 comp rx=0 tx=0).stop 2024-03-28T22:36:59.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.102+0000 7f7ef9a00640 1 -- 172.21.15.43:0/3679623676 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ef4107c70 msgr2=0x7f7ef419dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:59.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.102+0000 7f7ef9a00640 1 --2- 172.21.15.43:0/3679623676 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ef4107c70 0x7f7ef419dfa0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7ee400e9c0 tx=0x7f7ee400ee80 comp rx=0 tx=0).stop 2024-03-28T22:36:59.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.102+0000 7f7ef9a00640 1 -- 172.21.15.43:0/3679623676 shutdown_connections 2024-03-28T22:36:59.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.102+0000 7f7ef9a00640 1 --2- 172.21.15.43:0/3679623676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ef4109820 0x7f7ef41a54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:59.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.102+0000 7f7ef9a00640 1 --2- 172.21.15.43:0/3679623676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ef4108e60 0x7f7ef419e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:59.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.102+0000 7f7ef9a00640 1 --2- 172.21.15.43:0/3679623676 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f7ed80a1e10 0x7f7ed80a42d0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:59.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.102+0000 7f7ef9a00640 1 --2- 172.21.15.43:0/3679623676 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ef4107c70 0x7f7ef419dfa0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:59.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.102+0000 7f7ef9a00640 1 -- 172.21.15.43:0/3679623676 >> 172.21.15.43:0/3679623676 conn(0x7f7ef4076c20 msgr2=0x7f7ef410c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:59.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.103+0000 7f7ef9a00640 1 -- 172.21.15.43:0/3679623676 shutdown_connections 2024-03-28T22:36:59.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.103+0000 7f7ef9a00640 1 -- 172.21.15.43:0/3679623676 wait complete. 2024-03-28T22:36:59.122 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T22:36:59.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:58 smithi043 ceph-mon[31823]: from='client.25565 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:36:59.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:58 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/4029641025' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:36:59.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:58 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/4070740037' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:36:59.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.353+0000 7f1b71400640 1 -- 172.21.15.43:0/698297743 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b6c103850 msgr2=0x7f1b6c103c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:59.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.353+0000 7f1b71400640 1 --2- 172.21.15.43:0/698297743 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b6c103850 0x7f1b6c103c70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c009900 tx=0x7f1b5c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:36:59.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.353+0000 7f1b71400640 1 -- 172.21.15.43:0/698297743 shutdown_connections 2024-03-28T22:36:59.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.353+0000 7f1b71400640 1 --2- 172.21.15.43:0/698297743 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1b6c105400 0x7f1b6c10bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:59.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.353+0000 7f1b71400640 1 --2- 172.21.15.43:0/698297743 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b6c104a40 0x7f1b6c104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:59.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.353+0000 7f1b71400640 1 --2- 172.21.15.43:0/698297743 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b6c103850 0x7f1b6c103c70 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:59.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.353+0000 7f1b71400640 1 -- 172.21.15.43:0/698297743 >> 172.21.15.43:0/698297743 conn(0x7f1b6c0fedc0 msgr2=0x7f1b6c101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:59.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.353+0000 7f1b71400640 1 -- 172.21.15.43:0/698297743 shutdown_connections 2024-03-28T22:36:59.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.353+0000 7f1b71400640 1 -- 172.21.15.43:0/698297743 wait complete. 2024-03-28T22:36:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.355+0000 7f1b71400640 1 Processor -- start 2024-03-28T22:36:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.355+0000 7f1b71400640 1 -- start start 2024-03-28T22:36:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.355+0000 7f1b71400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b6c103850 0x7f1b6c199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.355+0000 7f1b71400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1b6c104a40 0x7f1b6c19a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.355+0000 7f1b71400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b6c105400 0x7f1b6c1a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.356+0000 7f1b71400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1b6c10f4a0 con 0x7f1b6c105400 2024-03-28T22:36:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.356+0000 7f1b71400640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1b6c10f320 con 0x7f1b6c104a40 2024-03-28T22:36:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.356+0000 7f1b71400640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f1b6c10f620 con 0x7f1b6c103850 2024-03-28T22:36:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.356+0000 7f1b6b400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1b6c104a40 0x7f1b6c19a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.356+0000 7f1b6be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b6c103850 0x7f1b6c199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.356+0000 7f1b6b400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1b6c104a40 0x7f1b6c19a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:37474/0 (socket says 172.21.15.43:37474) 2024-03-28T22:36:59.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.356+0000 7f1b6b400640 1 -- 172.21.15.43:0/1618128236 learned_addr learned my addr 172.21.15.43:0/1618128236 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:59.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.356+0000 7f1b70a00640 1 --2- 172.21.15.43:0/1618128236 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b6c105400 0x7f1b6c1a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:59.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.356+0000 7f1b6b400640 1 -- 172.21.15.43:0/1618128236 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b6c103850 msgr2=0x7f1b6c199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:59.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.356+0000 7f1b6b400640 1 --2- 172.21.15.43:0/1618128236 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b6c103850 0x7f1b6c199b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:59.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.356+0000 7f1b6b400640 1 -- 172.21.15.43:0/1618128236 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b6c105400 msgr2=0x7f1b6c1a1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:59.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.356+0000 7f1b6b400640 1 --2- 172.21.15.43:0/1618128236 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b6c105400 0x7f1b6c1a1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:59.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.356+0000 7f1b6b400640 1 -- 172.21.15.43:0/1618128236 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b6c1a1790 con 0x7f1b6c104a40 2024-03-28T22:36:59.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.356+0000 7f1b6b400640 1 --2- 172.21.15.43:0/1618128236 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1b6c104a40 0x7f1b6c19a0b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1b5800e9f0 tx=0x7f1b5800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:59.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.357+0000 7f1b68c00640 1 -- 172.21.15.43:0/1618128236 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b5800ccc0 con 0x7f1b6c104a40 2024-03-28T22:36:59.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.357+0000 7f1b68c00640 1 -- 172.21.15.43:0/1618128236 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b58010910 con 0x7f1b6c104a40 2024-03-28T22:36:59.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.357+0000 7f1b68c00640 1 -- 172.21.15.43:0/1618128236 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b580188a0 con 0x7f1b6c104a40 2024-03-28T22:36:59.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.357+0000 7f1b71400640 1 -- 172.21.15.43:0/1618128236 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b6c1a1a80 con 0x7f1b6c104a40 2024-03-28T22:36:59.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.357+0000 7f1b71400640 1 -- 172.21.15.43:0/1618128236 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b6c1a1f80 con 0x7f1b6c104a40 2024-03-28T22:36:59.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.358+0000 7f1b71400640 1 -- 172.21.15.43:0/1618128236 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b6c10ac40 con 0x7f1b6c104a40 2024-03-28T22:36:59.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.360+0000 7f1b68c00640 1 -- 172.21.15.43:0/1618128236 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7f1b58010ac0 con 0x7f1b6c104a40 2024-03-28T22:36:59.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.362+0000 7f1b68c00640 1 --2- 172.21.15.43:0/1618128236 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f1b500a1e90 0x7f1b500a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:59.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.362+0000 7f1b68c00640 1 -- 172.21.15.43:0/1618128236 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f1b580ce610 con 0x7f1b6c104a40 2024-03-28T22:36:59.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.362+0000 7f1b6be00640 1 --2- 172.21.15.43:0/1618128236 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f1b500a1e90 0x7f1b500a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:59.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.363+0000 7f1b6be00640 1 --2- 172.21.15.43:0/1618128236 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f1b500a1e90 0x7f1b500a4350 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c000c00 tx=0x7f1b5c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:59.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.366+0000 7f1b68c00640 1 -- 172.21.15.43:0/1618128236 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1b5809b550 con 0x7f1b6c104a40 2024-03-28T22:36:59.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:58 smithi172 ceph-mon[36566]: from='client.25565 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:36:59.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:58 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/4029641025' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:36:59.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:58 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/4070740037' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:36:59.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:58 smithi082 ceph-mon[36471]: from='client.25565 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T22:36:59.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:58 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/4029641025' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:36:59.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:58 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/4070740037' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:36:59.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.566+0000 7f1b71400640 1 -- 172.21.15.43:0/1618128236 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f1b6c02d320 con 0x7f1b6c104a40 2024-03-28T22:36:59.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.567+0000 7f1b68c00640 1 -- 172.21.15.43:0/1618128236 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v96) ==== 103+0+11 (secure 0 0 0) 0x7f1b5809ae50 con 0x7f1b6c104a40 2024-03-28T22:36:59.566 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-03-28T22:36:59.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.570+0000 7f1b71400640 1 -- 172.21.15.43:0/1618128236 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f1b500a1e90 msgr2=0x7f1b500a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:59.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.570+0000 7f1b71400640 1 --2- 172.21.15.43:0/1618128236 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f1b500a1e90 0x7f1b500a4350 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c000c00 tx=0x7f1b5c005fd0 comp rx=0 tx=0).stop 2024-03-28T22:36:59.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.570+0000 7f1b71400640 1 -- 172.21.15.43:0/1618128236 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1b6c104a40 msgr2=0x7f1b6c19a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:59.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.570+0000 7f1b71400640 1 --2- 172.21.15.43:0/1618128236 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1b6c104a40 0x7f1b6c19a0b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1b5800e9f0 tx=0x7f1b5800edf0 comp rx=0 tx=0).stop 2024-03-28T22:36:59.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.570+0000 7f1b71400640 1 -- 172.21.15.43:0/1618128236 shutdown_connections 2024-03-28T22:36:59.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.570+0000 7f1b71400640 1 --2- 172.21.15.43:0/1618128236 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f1b500a1e90 0x7f1b500a4350 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:59.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.570+0000 7f1b71400640 1 --2- 172.21.15.43:0/1618128236 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1b6c104a40 0x7f1b6c19a0b0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:59.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.570+0000 7f1b71400640 1 --2- 172.21.15.43:0/1618128236 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1b6c105400 0x7f1b6c1a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:59.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.570+0000 7f1b71400640 1 --2- 172.21.15.43:0/1618128236 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1b6c103850 0x7f1b6c199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:59.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.570+0000 7f1b71400640 1 -- 172.21.15.43:0/1618128236 >> 172.21.15.43:0/1618128236 conn(0x7f1b6c0fedc0 msgr2=0x7f1b6c107e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:59.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.570+0000 7f1b71400640 1 -- 172.21.15.43:0/1618128236 shutdown_connections 2024-03-28T22:36:59.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.570+0000 7f1b71400640 1 -- 172.21.15.43:0/1618128236 wait complete. 2024-03-28T22:36:59.590 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:36:59.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.806+0000 7f5a58e00640 1 -- 172.21.15.43:0/4171974662 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a54102780 msgr2=0x7f5a54104bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:59.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.806+0000 7f5a58e00640 1 --2- 172.21.15.43:0/4171974662 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a54102780 0x7f5a54104bf0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c009900 tx=0x7f5a3c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:36:59.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.806+0000 7f5a58e00640 1 -- 172.21.15.43:0/4171974662 shutdown_connections 2024-03-28T22:36:59.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.806+0000 7f5a58e00640 1 --2- 172.21.15.43:0/4171974662 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a54105130 0x7f5a541055b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:59.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.806+0000 7f5a58e00640 1 --2- 172.21.15.43:0/4171974662 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a54102780 0x7f5a54104bf0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:59.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.806+0000 7f5a58e00640 1 --2- 172.21.15.43:0/4171974662 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a540ffe10 0x7f5a54102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:59.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.806+0000 7f5a58e00640 1 -- 172.21.15.43:0/4171974662 >> 172.21.15.43:0/4171974662 conn(0x7f5a540fba30 msgr2=0x7f5a540fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:36:59.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.806+0000 7f5a58e00640 1 -- 172.21.15.43:0/4171974662 shutdown_connections 2024-03-28T22:36:59.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.806+0000 7f5a58e00640 1 -- 172.21.15.43:0/4171974662 wait complete. 2024-03-28T22:36:59.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.807+0000 7f5a58e00640 1 Processor -- start 2024-03-28T22:36:59.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.808+0000 7f5a58e00640 1 -- start start 2024-03-28T22:36:59.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.808+0000 7f5a58e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a540ffe10 0x7f5a5419df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:59.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.808+0000 7f5a58e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a54102780 0x7f5a5419e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:59.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.808+0000 7f5a58e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a54105130 0x7f5a541a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:59.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.808+0000 7f5a58e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5a54113860 con 0x7f5a54102780 2024-03-28T22:36:59.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.808+0000 7f5a58e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5a541136e0 con 0x7f5a540ffe10 2024-03-28T22:36:59.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.808+0000 7f5a58e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f5a541139e0 con 0x7f5a54105130 2024-03-28T22:36:59.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.808+0000 7f5a53400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a540ffe10 0x7f5a5419df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:59.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.808+0000 7f5a4aa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a54102780 0x7f5a5419e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:59.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.808+0000 7f5a53400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a540ffe10 0x7f5a5419df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:37496/0 (socket says 172.21.15.43:37496) 2024-03-28T22:36:59.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.808+0000 7f5a53400640 1 -- 172.21.15.43:0/1069313246 learned_addr learned my addr 172.21.15.43:0/1069313246 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:36:59.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.808+0000 7f5a53e00640 1 --2- 172.21.15.43:0/1069313246 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a54105130 0x7f5a541a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:59.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.809+0000 7f5a53400640 1 -- 172.21.15.43:0/1069313246 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a54105130 msgr2=0x7f5a541a5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:59.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.809+0000 7f5a53400640 1 --2- 172.21.15.43:0/1069313246 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a54105130 0x7f5a541a5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:59.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.809+0000 7f5a53400640 1 -- 172.21.15.43:0/1069313246 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a54102780 msgr2=0x7f5a5419e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:36:59.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.809+0000 7f5a53400640 1 --2- 172.21.15.43:0/1069313246 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a54102780 0x7f5a5419e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:36:59.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.809+0000 7f5a53400640 1 -- 172.21.15.43:0/1069313246 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a541a5aa0 con 0x7f5a540ffe10 2024-03-28T22:36:59.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.809+0000 7f5a53e00640 1 --2- 172.21.15.43:0/1069313246 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a54105130 0x7f5a541a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:36:59.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.809+0000 7f5a53400640 1 --2- 172.21.15.43:0/1069313246 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a540ffe10 0x7f5a5419df40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f5a4400db80 tx=0x7f5a4400b630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:59.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.809+0000 7f5a50c00640 1 -- 172.21.15.43:0/1069313246 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a44017070 con 0x7f5a540ffe10 2024-03-28T22:36:59.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.809+0000 7f5a58e00640 1 -- 172.21.15.43:0/1069313246 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a541a5d90 con 0x7f5a540ffe10 2024-03-28T22:36:59.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.809+0000 7f5a50c00640 1 -- 172.21.15.43:0/1069313246 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a44003b70 con 0x7f5a540ffe10 2024-03-28T22:36:59.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.809+0000 7f5a50c00640 1 -- 172.21.15.43:0/1069313246 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a440106c0 con 0x7f5a540ffe10 2024-03-28T22:36:59.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.809+0000 7f5a58e00640 1 -- 172.21.15.43:0/1069313246 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a541a6250 con 0x7f5a540ffe10 2024-03-28T22:36:59.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.811+0000 7f5a58e00640 1 -- 172.21.15.43:0/1069313246 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a540fd520 con 0x7f5a540ffe10 2024-03-28T22:36:59.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.812+0000 7f5a50c00640 1 -- 172.21.15.43:0/1069313246 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7f5a44004020 con 0x7f5a540ffe10 2024-03-28T22:36:59.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.814+0000 7f5a50c00640 1 --2- 172.21.15.43:0/1069313246 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5a380a1e10 0x7f5a380a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:36:59.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.814+0000 7f5a50c00640 1 -- 172.21.15.43:0/1069313246 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f5a440ce1d0 con 0x7f5a540ffe10 2024-03-28T22:36:59.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.814+0000 7f5a4aa00640 1 --2- 172.21.15.43:0/1069313246 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5a380a1e10 0x7f5a380a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:36:59.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.815+0000 7f5a4aa00640 1 --2- 172.21.15.43:0/1069313246 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5a380a1e10 0x7f5a380a42d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c02c240 tx=0x7f5a3c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:36:59.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:36:59.817+0000 7f5a50c00640 1 -- 172.21.15.43:0/1069313246 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5a44020070 con 0x7f5a540ffe10 2024-03-28T22:37:00.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.064+0000 7f5a58e00640 1 -- 172.21.15.43:0/1069313246 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a5402d3d0 con 0x7f5a540ffe10 2024-03-28T22:37:00.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.065+0000 7f5a50c00640 1 -- 172.21.15.43:0/1069313246 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11934 (secure 0 0 0) 0x7f5a4409b0b0 con 0x7f5a540ffe10 2024-03-28T22:37:00.064 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:37:00.064 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:33:28.906396+0000","modified":"2024-03-28T22:36:49.597670+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24572,"mds_1":24556,"mds_2":24535,"mds_3":24523,"mds_4":24518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/1084988072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1084988072},{"type":"v1","addr":"172.21.15.82:6835","nonce":1084988072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572,24523,24518,24535,24556]},"id":1}]} 2024-03-28T22:37:00.064 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 29 2024-03-28T22:37:00.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.067+0000 7f5a58e00640 1 -- 172.21.15.43:0/1069313246 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5a380a1e10 msgr2=0x7f5a380a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:00.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.067+0000 7f5a58e00640 1 --2- 172.21.15.43:0/1069313246 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5a380a1e10 0x7f5a380a42d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c02c240 tx=0x7f5a3c02df90 comp rx=0 tx=0).stop 2024-03-28T22:37:00.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.067+0000 7f5a58e00640 1 -- 172.21.15.43:0/1069313246 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a540ffe10 msgr2=0x7f5a5419df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:00.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.067+0000 7f5a58e00640 1 --2- 172.21.15.43:0/1069313246 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a540ffe10 0x7f5a5419df40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f5a4400db80 tx=0x7f5a4400b630 comp rx=0 tx=0).stop 2024-03-28T22:37:00.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.068+0000 7f5a58e00640 1 -- 172.21.15.43:0/1069313246 shutdown_connections 2024-03-28T22:37:00.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.068+0000 7f5a58e00640 1 --2- 172.21.15.43:0/1069313246 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a54105130 0x7f5a541a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.068+0000 7f5a58e00640 1 --2- 172.21.15.43:0/1069313246 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a54102780 0x7f5a5419e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.068+0000 7f5a58e00640 1 --2- 172.21.15.43:0/1069313246 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f5a380a1e10 0x7f5a380a42d0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.068+0000 7f5a58e00640 1 --2- 172.21.15.43:0/1069313246 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a540ffe10 0x7f5a5419df40 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.068+0000 7f5a58e00640 1 -- 172.21.15.43:0/1069313246 >> 172.21.15.43:0/1069313246 conn(0x7f5a540fba30 msgr2=0x7f5a54100b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:00.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.068+0000 7f5a58e00640 1 -- 172.21.15.43:0/1069313246 shutdown_connections 2024-03-28T22:37:00.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.068+0000 7f5a58e00640 1 -- 172.21.15.43:0/1069313246 wait complete. 2024-03-28T22:37:00.087 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:37:00.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:59 smithi043 ceph-mon[31823]: pgmap v431: 129 pgs: 129 active+clean; 656 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-03-28T22:37:00.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:59 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3679623676' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:37:00.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:36:59 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1618128236' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:37:00.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.315+0000 7f53a0a00640 1 -- 172.21.15.43:0/1065987259 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f539c102950 msgr2=0x7f539c104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:00.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.315+0000 7f53a0a00640 1 --2- 172.21.15.43:0/1065987259 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f539c102950 0x7f539c104dc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5384009900 tx=0x7f538402bc40 comp rx=0 tx=0).stop 2024-03-28T22:37:00.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.316+0000 7f53a0a00640 1 -- 172.21.15.43:0/1065987259 shutdown_connections 2024-03-28T22:37:00.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.316+0000 7f53a0a00640 1 --2- 172.21.15.43:0/1065987259 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f539c105300 0x7f539c105780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.316+0000 7f53a0a00640 1 --2- 172.21.15.43:0/1065987259 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f539c102950 0x7f539c104dc0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.316+0000 7f53a0a00640 1 --2- 172.21.15.43:0/1065987259 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f539c0fffe0 0x7f539c102410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.316+0000 7f53a0a00640 1 -- 172.21.15.43:0/1065987259 >> 172.21.15.43:0/1065987259 conn(0x7f539c0fbc00 msgr2=0x7f539c0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:00.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.316+0000 7f53a0a00640 1 -- 172.21.15.43:0/1065987259 shutdown_connections 2024-03-28T22:37:00.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.316+0000 7f53a0a00640 1 -- 172.21.15.43:0/1065987259 wait complete. 2024-03-28T22:37:00.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.317+0000 7f53a0a00640 1 Processor -- start 2024-03-28T22:37:00.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.317+0000 7f53a0a00640 1 -- start start 2024-03-28T22:37:00.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.318+0000 7f53a0a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f539c0fffe0 0x7f539c19e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:00.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.318+0000 7f53a0a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f539c102950 0x7f539c19e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:00.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.318+0000 7f53a0a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f539c105300 0x7f539c1a5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:00.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.318+0000 7f53a0a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f539c113a10 con 0x7f539c102950 2024-03-28T22:37:00.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.318+0000 7f53a0a00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f539c113890 con 0x7f539c0fffe0 2024-03-28T22:37:00.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.318+0000 7f53a0a00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f539c113b90 con 0x7f539c105300 2024-03-28T22:37:00.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.318+0000 7f5399e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f539c0fffe0 0x7f539c19e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:00.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.318+0000 7f5399400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f539c102950 0x7f539c19e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:00.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.318+0000 7f539a800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f539c105300 0x7f539c1a5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:00.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.318+0000 7f5399e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f539c0fffe0 0x7f539c19e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:37524/0 (socket says 172.21.15.43:37524) 2024-03-28T22:37:00.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.318+0000 7f5399e00640 1 -- 172.21.15.43:0/2202629670 learned_addr learned my addr 172.21.15.43:0/2202629670 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:00.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.318+0000 7f539a800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f539c105300 0x7f539c1a5630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:46888/0 (socket says 172.21.15.43:46888) 2024-03-28T22:37:00.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.318+0000 7f539a800640 1 -- 172.21.15.43:0/2202629670 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f539c0fffe0 msgr2=0x7f539c19e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:00.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.318+0000 7f539a800640 1 --2- 172.21.15.43:0/2202629670 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f539c0fffe0 0x7f539c19e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.318+0000 7f539a800640 1 -- 172.21.15.43:0/2202629670 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f539c102950 msgr2=0x7f539c19e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:00.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.319+0000 7f539a800640 1 --2- 172.21.15.43:0/2202629670 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f539c102950 0x7f539c19e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.319+0000 7f539a800640 1 -- 172.21.15.43:0/2202629670 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f539c1a5ca0 con 0x7f539c105300 2024-03-28T22:37:00.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.319+0000 7f539a800640 1 --2- 172.21.15.43:0/2202629670 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f539c105300 0x7f539c1a5630 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f539400d6b0 tx=0x7f539400db70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:00.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.319+0000 7f5392a00640 1 -- 172.21.15.43:0/2202629670 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5394004230 con 0x7f539c105300 2024-03-28T22:37:00.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.319+0000 7f5392a00640 1 -- 172.21.15.43:0/2202629670 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53940043d0 con 0x7f539c105300 2024-03-28T22:37:00.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.319+0000 7f5392a00640 1 -- 172.21.15.43:0/2202629670 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5394010460 con 0x7f539c105300 2024-03-28T22:37:00.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.319+0000 7f53a0a00640 1 -- 172.21.15.43:0/2202629670 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f539c1a5f90 con 0x7f539c105300 2024-03-28T22:37:00.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.319+0000 7f53a0a00640 1 -- 172.21.15.43:0/2202629670 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f539c1a6400 con 0x7f539c105300 2024-03-28T22:37:00.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.321+0000 7f53a0a00640 1 -- 172.21.15.43:0/2202629670 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f539c0fd6f0 con 0x7f539c105300 2024-03-28T22:37:00.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.323+0000 7f5392a00640 1 -- 172.21.15.43:0/2202629670 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7f5394045a60 con 0x7f539c105300 2024-03-28T22:37:00.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.324+0000 7f5392a00640 1 --2- 172.21.15.43:0/2202629670 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f53800a1e90 0x7f53800a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:00.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.324+0000 7f5392a00640 1 -- 172.21.15.43:0/2202629670 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f53940cdbe0 con 0x7f539c105300 2024-03-28T22:37:00.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.324+0000 7f5399e00640 1 --2- 172.21.15.43:0/2202629670 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f53800a1e90 0x7f53800a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:00.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.325+0000 7f5399e00640 1 --2- 172.21.15.43:0/2202629670 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f53800a1e90 0x7f53800a4350 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f538c009fd0 tx=0x7f538c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:00.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.326+0000 7f5392a00640 1 -- 172.21.15.43:0/2202629670 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5394017020 con 0x7f539c105300 2024-03-28T22:37:00.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:59 smithi172 ceph-mon[36566]: pgmap v431: 129 pgs: 129 active+clean; 656 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-03-28T22:37:00.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:59 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3679623676' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:37:00.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:36:59 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1618128236' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:37:00.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:59 smithi082 ceph-mon[36471]: pgmap v431: 129 pgs: 129 active+clean; 656 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-03-28T22:37:00.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:59 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3679623676' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:37:00.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:36:59 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1618128236' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:37:00.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.530+0000 7f53a0a00640 1 -- 172.21.15.43:0/2202629670 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f539c02d3c0 con 0x7f539c105300 2024-03-28T22:37:00.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.533+0000 7f5392a00640 1 -- 172.21.15.43:0/2202629670 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+24712 (secure 0 0 0) 0x7f539409a4f0 con 0x7f539c105300 2024-03-28T22:37:00.533 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:37:00.533 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":96,"fsid":"c2276428-ed51-11ee-95d0-87774f69a715","created":"2024-03-28T22:24:19.950084+0000","modified":"2024-03-28T22:35:30.237654+0000","last_up_change":"2024-03-28T22:32:31.078737+0000","last_in_change":"2024-03-28T22:32:10.147097+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:28:14.713290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:33:27.167435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:33:28.411825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T22:33:32.523122+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b855adcd-786c-4e42-b8fe-a515a77ce979","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6803","nonce":2826124409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6805","nonce":2826124409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6809","nonce":2826124409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6807","nonce":2826124409}]},"public_addr":"172.21.15.43:6803/2826124409","cluster_addr":"172.21.15.43:6805/2826124409","heartbeat_back_addr":"172.21.15.43:6809/2826124409","heartbeat_front_addr":"172.21.15.43:6807/2826124409","state":["exists","up"]},{"osd":1,"uuid":"031c143d-8ad1-4b68-8db9-3c717137e3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6801","nonce":3043263644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6803","nonce":3043263644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6807","nonce":3043263644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6805","nonce":3043263644}]},"public_addr":"172.21.15.82:6801/3043263644","cluster_addr":"172.21.15.82:6803/3043263644","heartbeat_back_addr":"172.21.15.82:6807/3043263644","heartbeat_front_addr":"172.21.15.82:6805/3043263644","state":["exists","up"]},{"osd":2,"uuid":"20632bbb-f07a-4ca1-b0c4-ff9d508e211c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6801","nonce":4213953645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6803","nonce":4213953645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6807","nonce":4213953645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6805","nonce":4213953645}]},"public_addr":"172.21.15.172:6801/4213953645","cluster_addr":"172.21.15.172:6803/4213953645","heartbeat_back_addr":"172.21.15.172:6807/4213953645","heartbeat_front_addr":"172.21.15.172:6805/4213953645","state":["exists","up"]},{"osd":3,"uuid":"9a1f2312-4dbd-475b-8736-d5ebc8788e09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6811","nonce":1597280920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6813","nonce":1597280920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6817","nonce":1597280920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6815","nonce":1597280920}]},"public_addr":"172.21.15.43:6811/1597280920","cluster_addr":"172.21.15.43:6813/1597280920","heartbeat_back_addr":"172.21.15.43:6817/1597280920","heartbeat_front_addr":"172.21.15.43:6815/1597280920","state":["exists","up"]},{"osd":4,"uuid":"29140ead-7075-42b2-90c1-27cae0b29ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6809","nonce":215451895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6811","nonce":215451895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6815","nonce":215451895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6813","nonce":215451895}]},"public_addr":"172.21.15.82:6809/215451895","cluster_addr":"172.21.15.82:6811/215451895","heartbeat_back_addr":"172.21.15.82:6815/215451895","heartbeat_front_addr":"172.21.15.82:6813/215451895","state":["exists","up"]},{"osd":5,"uuid":"59644c61-93dc-4536-b2ad-0256b24e5cb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6809","nonce":1576148933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6811","nonce":1576148933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6815","nonce":1576148933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6813","nonce":1576148933}]},"public_addr":"172.21.15.172:6809/1576148933","cluster_addr":"172.21.15.172:6811/1576148933","heartbeat_back_addr":"172.21.15.172:6815/1576148933","heartbeat_front_addr":"172.21.15.172:6813/1576148933","state":["exists","up"]},{"osd":6,"uuid":"22d54e57-c3f1-4ecf-9921-8e815b48f92a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6819","nonce":791044279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6821","nonce":791044279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6825","nonce":791044279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6823","nonce":791044279}]},"public_addr":"172.21.15.43:6819/791044279","cluster_addr":"172.21.15.43:6821/791044279","heartbeat_back_addr":"172.21.15.43:6825/791044279","heartbeat_front_addr":"172.21.15.43:6823/791044279","state":["exists","up"]},{"osd":7,"uuid":"248cd1e2-0d6b-452f-a0a8-c51fe1b975ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6817","nonce":2157777323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6819","nonce":2157777323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6823","nonce":2157777323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6821","nonce":2157777323}]},"public_addr":"172.21.15.82:6817/2157777323","cluster_addr":"172.21.15.82:6819/2157777323","heartbeat_back_addr":"172.21.15.82:6823/2157777323","heartbeat_front_addr":"172.21.15.82:6821/2157777323","state":["exists","up"]},{"osd":8,"uuid":"d98aab6e-8bae-4c24-906c-4f492f047b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6817","nonce":1095328248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6819","nonce":1095328248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6823","nonce":1095328248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6821","nonce":1095328248}]},"public_addr":"172.21.15.172:6817/1095328248","cluster_addr":"172.21.15.172:6819/1095328248","heartbeat_back_addr":"172.21.15.172:6823/1095328248","heartbeat_front_addr":"172.21.15.172:6821/1095328248","state":["exists","up"]},{"osd":9,"uuid":"19aa383e-a0f6-45ea-96cc-e7956535caa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6827","nonce":3657791498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6829","nonce":3657791498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6833","nonce":3657791498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6831","nonce":3657791498}]},"public_addr":"172.21.15.43:6827/3657791498","cluster_addr":"172.21.15.43:6829/3657791498","heartbeat_back_addr":"172.21.15.43:6833/3657791498","heartbeat_front_addr":"172.21.15.43:6831/3657791498","state":["exists","up"]},{"osd":10,"uuid":"666977e5-eaf7-41b9-a18f-c380e149345d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6825","nonce":3454954047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6827","nonce":3454954047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6831","nonce":3454954047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6829","nonce":3454954047}]},"public_addr":"172.21.15.82:6825/3454954047","cluster_addr":"172.21.15.82:6827/3454954047","heartbeat_back_addr":"172.21.15.82:6831/3454954047","heartbeat_front_addr":"172.21.15.82:6829/3454954047","state":["exists","up"]},{"osd":11,"uuid":"615cf3be-8076-470e-ad09-c880e7593717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6825","nonce":2563517144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6827","nonce":2563517144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6831","nonce":2563517144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6829","nonce":2563517144}]},"public_addr":"172.21.15.172:6825/2563517144","cluster_addr":"172.21.15.172:6827/2563517144","heartbeat_back_addr":"172.21.15.172:6831/2563517144","heartbeat_front_addr":"172.21.15.172:6829/2563517144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:21.727405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:47.100259+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:10.615438+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:43.632144+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:09.487940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:34.604463+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:09.594452+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:35.709795+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:01.184202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:35.455322+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:02.413166+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:28.550754+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.9","mappings":[{"from":5,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/578791143":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/4029313215":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/1574178482":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6801/3517712438":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6800/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/1375099638":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/2710117652":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6801/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/3215634168":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6800/3517712438":"2024-03-29T22:25:28.008215+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:37:00.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.536+0000 7f53a0a00640 1 -- 172.21.15.43:0/2202629670 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f53800a1e90 msgr2=0x7f53800a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:00.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.536+0000 7f53a0a00640 1 --2- 172.21.15.43:0/2202629670 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f53800a1e90 0x7f53800a4350 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f538c009fd0 tx=0x7f538c009500 comp rx=0 tx=0).stop 2024-03-28T22:37:00.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.536+0000 7f53a0a00640 1 -- 172.21.15.43:0/2202629670 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f539c105300 msgr2=0x7f539c1a5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:00.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.536+0000 7f53a0a00640 1 --2- 172.21.15.43:0/2202629670 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f539c105300 0x7f539c1a5630 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f539400d6b0 tx=0x7f539400db70 comp rx=0 tx=0).stop 2024-03-28T22:37:00.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.536+0000 7f53a0a00640 1 -- 172.21.15.43:0/2202629670 shutdown_connections 2024-03-28T22:37:00.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.536+0000 7f53a0a00640 1 --2- 172.21.15.43:0/2202629670 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f539c105300 0x7f539c1a5630 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.536+0000 7f53a0a00640 1 --2- 172.21.15.43:0/2202629670 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f539c102950 0x7f539c19e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.536+0000 7f53a0a00640 1 --2- 172.21.15.43:0/2202629670 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f53800a1e90 0x7f53800a4350 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.536+0000 7f53a0a00640 1 --2- 172.21.15.43:0/2202629670 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f539c0fffe0 0x7f539c19e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.536+0000 7f53a0a00640 1 -- 172.21.15.43:0/2202629670 >> 172.21.15.43:0/2202629670 conn(0x7f539c0fbc00 msgr2=0x7f539c100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:00.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.536+0000 7f53a0a00640 1 -- 172.21.15.43:0/2202629670 shutdown_connections 2024-03-28T22:37:00.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.536+0000 7f53a0a00640 1 -- 172.21.15.43:0/2202629670 wait complete. 2024-03-28T22:37:00.555 DEBUG:tasks.fwd_scrub:Yielding 2024-03-28T22:37:00.556 INFO:teuthology.run_tasks:Running task exec... 2024-03-28T22:37:00.566 INFO:teuthology.task.exec:Executing custom commands... 2024-03-28T22:37:00.566 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi043.front.sepia.ceph.com 2024-03-28T22:37:00.566 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-03-28T22:37:00.567 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-28T22:37:00.567 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:37:00.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:37:00.567 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:37:00.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.761+0000 7fcf73200640 1 -- 172.21.15.43:0/840826412 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcf6c104950 msgr2=0x7fcf6c106d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:00.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.761+0000 7fcf73200640 1 --2- 172.21.15.43:0/840826412 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcf6c104950 0x7fcf6c106d40 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c009900 tx=0x7fcf5c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:37:00.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.761+0000 7fcf73200640 1 -- 172.21.15.43:0/840826412 shutdown_connections 2024-03-28T22:37:00.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.761+0000 7fcf73200640 1 --2- 172.21.15.43:0/840826412 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcf6c104950 0x7fcf6c106d40 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.761+0000 7fcf73200640 1 --2- 172.21.15.43:0/840826412 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf6c107280 0x7fcf6c1096d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.761+0000 7fcf73200640 1 --2- 172.21.15.43:0/840826412 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf6c102020 0x7fcf6c104410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.761+0000 7fcf73200640 1 -- 172.21.15.43:0/840826412 >> 172.21.15.43:0/840826412 conn(0x7fcf6c0fba30 msgr2=0x7fcf6c0fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:00.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.761+0000 7fcf73200640 1 -- 172.21.15.43:0/840826412 shutdown_connections 2024-03-28T22:37:00.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.761+0000 7fcf73200640 1 -- 172.21.15.43:0/840826412 wait complete. 2024-03-28T22:37:00.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.762+0000 7fcf73200640 1 Processor -- start 2024-03-28T22:37:00.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.763+0000 7fcf73200640 1 -- start start 2024-03-28T22:37:00.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.763+0000 7fcf73200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcf6c102020 0x7fcf6c199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:00.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.763+0000 7fcf73200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf6c104950 0x7fcf6c199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:00.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.763+0000 7fcf73200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf6c107280 0x7fcf6c1a1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:00.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.763+0000 7fcf73200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6c10d230 con 0x7fcf6c104950 2024-03-28T22:37:00.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.763+0000 7fcf73200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6c10d0b0 con 0x7fcf6c102020 2024-03-28T22:37:00.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.763+0000 7fcf73200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6c10d3b0 con 0x7fcf6c107280 2024-03-28T22:37:00.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.763+0000 7fcf71e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcf6c102020 0x7fcf6c199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:00.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.763+0000 7fcf71e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcf6c102020 0x7fcf6c199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:37548/0 (socket says 172.21.15.43:37548) 2024-03-28T22:37:00.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.764+0000 7fcf71e00640 1 -- 172.21.15.43:0/271623546 learned_addr learned my addr 172.21.15.43:0/271623546 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:00.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.764+0000 7fcf71400640 1 --2- 172.21.15.43:0/271623546 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf6c104950 0x7fcf6c199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:00.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.764+0000 7fcf71e00640 1 -- 172.21.15.43:0/271623546 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf6c107280 msgr2=0x7fcf6c1a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:00.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.764+0000 7fcf71e00640 1 --2- 172.21.15.43:0/271623546 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf6c107280 0x7fcf6c1a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.764+0000 7fcf71e00640 1 -- 172.21.15.43:0/271623546 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf6c104950 msgr2=0x7fcf6c199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:00.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.764+0000 7fcf71e00640 1 --2- 172.21.15.43:0/271623546 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf6c104950 0x7fcf6c199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.764+0000 7fcf71e00640 1 -- 172.21.15.43:0/271623546 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf6c1a16c0 con 0x7fcf6c102020 2024-03-28T22:37:00.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.764+0000 7fcf71e00640 1 --2- 172.21.15.43:0/271623546 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcf6c102020 0x7fcf6c199aa0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fcf6000e9f0 tx=0x7fcf6000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:00.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.765+0000 7fcf66a00640 1 -- 172.21.15.43:0/271623546 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf6000cd80 con 0x7fcf6c102020 2024-03-28T22:37:00.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.765+0000 7fcf66a00640 1 -- 172.21.15.43:0/271623546 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf60010910 con 0x7fcf6c102020 2024-03-28T22:37:00.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.765+0000 7fcf73200640 1 -- 172.21.15.43:0/271623546 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf6c1a19b0 con 0x7fcf6c102020 2024-03-28T22:37:00.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.765+0000 7fcf66a00640 1 -- 172.21.15.43:0/271623546 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf60018990 con 0x7fcf6c102020 2024-03-28T22:37:00.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.765+0000 7fcf73200640 1 -- 172.21.15.43:0/271623546 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf6c1a1f20 con 0x7fcf6c102020 2024-03-28T22:37:00.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.766+0000 7fcf73200640 1 -- 172.21.15.43:0/271623546 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf6c108dd0 con 0x7fcf6c102020 2024-03-28T22:37:00.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.769+0000 7fcf66a00640 1 -- 172.21.15.43:0/271623546 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7fcf60010ac0 con 0x7fcf6c102020 2024-03-28T22:37:00.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.771+0000 7fcf66a00640 1 --2- 172.21.15.43:0/271623546 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fcf540a1e80 0x7fcf540a4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:00.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.771+0000 7fcf66a00640 1 -- 172.21.15.43:0/271623546 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7fcf600ce820 con 0x7fcf6c102020 2024-03-28T22:37:00.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.771+0000 7fcf71400640 1 --2- 172.21.15.43:0/271623546 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fcf540a1e80 0x7fcf540a4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:00.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.771+0000 7fcf66a00640 1 -- 172.21.15.43:0/271623546 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fcf6009b700 con 0x7fcf6c102020 2024-03-28T22:37:00.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.772+0000 7fcf71400640 1 --2- 172.21.15.43:0/271623546 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fcf540a1e80 0x7fcf540a4340 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c000c00 tx=0x7fcf5c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:00.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.782+0000 7f1e81400640 1 -- 172.21.15.43:0/2955683696 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1e7c104b00 msgr2=0x7f1e7c106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:00.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.782+0000 7f1e81400640 1 --2- 172.21.15.43:0/2955683696 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1e7c104b00 0x7f1e7c106ef0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1e68009900 tx=0x7f1e6802bc30 comp rx=0 tx=0).stop 2024-03-28T22:37:00.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.783+0000 7f1e81400640 1 -- 172.21.15.43:0/2955683696 shutdown_connections 2024-03-28T22:37:00.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.783+0000 7f1e81400640 1 --2- 172.21.15.43:0/2955683696 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1e7c104b00 0x7f1e7c106ef0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.783+0000 7f1e81400640 1 --2- 172.21.15.43:0/2955683696 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e7c107430 0x7f1e7c109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.783+0000 7f1e81400640 1 --2- 172.21.15.43:0/2955683696 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1e7c1021d0 0x7f1e7c1045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.783+0000 7f1e81400640 1 -- 172.21.15.43:0/2955683696 >> 172.21.15.43:0/2955683696 conn(0x7f1e7c0fbbe0 msgr2=0x7f1e7c0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:00.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.783+0000 7f1e81400640 1 -- 172.21.15.43:0/2955683696 shutdown_connections 2024-03-28T22:37:00.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.783+0000 7f1e81400640 1 -- 172.21.15.43:0/2955683696 wait complete. 2024-03-28T22:37:00.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.784+0000 7f1e81400640 1 Processor -- start 2024-03-28T22:37:00.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.784+0000 7f1e81400640 1 -- start start 2024-03-28T22:37:00.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.784+0000 7f1e81400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1e7c1021d0 0x7f1e7c101b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:00.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.784+0000 7f1e81400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1e7c104b00 0x7f1e7c100180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:00.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.784+0000 7f1e81400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e7c107430 0x7f1e7c1006c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:00.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.784+0000 7f1e81400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7c10d410 con 0x7f1e7c107430 2024-03-28T22:37:00.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.784+0000 7f1e81400640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7c10d290 con 0x7f1e7c1021d0 2024-03-28T22:37:00.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.784+0000 7f1e81400640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7c10d590 con 0x7f1e7c104b00 2024-03-28T22:37:00.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.785+0000 7f1e7a800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1e7c1021d0 0x7f1e7c101b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:00.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.785+0000 7f1e7a800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1e7c1021d0 0x7f1e7c101b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:37566/0 (socket says 172.21.15.43:37566) 2024-03-28T22:37:00.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.785+0000 7f1e7a800640 1 -- 172.21.15.43:0/2827323983 learned_addr learned my addr 172.21.15.43:0/2827323983 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:00.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.785+0000 7f1e79e00640 1 --2- 172.21.15.43:0/2827323983 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1e7c104b00 0x7f1e7c100180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:00.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.785+0000 7f1e7b200640 1 --2- 172.21.15.43:0/2827323983 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e7c107430 0x7f1e7c1006c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:00.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.785+0000 7f1e7a800640 1 -- 172.21.15.43:0/2827323983 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1e7c104b00 msgr2=0x7f1e7c100180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:00.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.785+0000 7f1e7a800640 1 --2- 172.21.15.43:0/2827323983 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1e7c104b00 0x7f1e7c100180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.785+0000 7f1e7a800640 1 -- 172.21.15.43:0/2827323983 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e7c107430 msgr2=0x7f1e7c1006c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:00.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.785+0000 7f1e7a800640 1 --2- 172.21.15.43:0/2827323983 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e7c107430 0x7f1e7c1006c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.785+0000 7f1e7a800640 1 -- 172.21.15.43:0/2827323983 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e7c100f50 con 0x7f1e7c1021d0 2024-03-28T22:37:00.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.785+0000 7f1e7a800640 1 --2- 172.21.15.43:0/2827323983 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1e7c1021d0 0x7f1e7c101b40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c00e9f0 tx=0x7f1e6c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:00.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.786+0000 7f1e73400640 1 -- 172.21.15.43:0/2827323983 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c00cd80 con 0x7f1e7c1021d0 2024-03-28T22:37:00.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.786+0000 7f1e73400640 1 -- 172.21.15.43:0/2827323983 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e6c010910 con 0x7f1e7c1021d0 2024-03-28T22:37:00.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.786+0000 7f1e73400640 1 -- 172.21.15.43:0/2827323983 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c018990 con 0x7f1e7c1021d0 2024-03-28T22:37:00.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.786+0000 7f1e81400640 1 -- 172.21.15.43:0/2827323983 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e7c1a1be0 con 0x7f1e7c1021d0 2024-03-28T22:37:00.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.786+0000 7f1e81400640 1 -- 172.21.15.43:0/2827323983 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e7c1a20b0 con 0x7f1e7c1021d0 2024-03-28T22:37:00.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.789+0000 7f1e73400640 1 -- 172.21.15.43:0/2827323983 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7f1e6c010ac0 con 0x7f1e7c1021d0 2024-03-28T22:37:00.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.789+0000 7f1e83ff8740 1 Processor -- start 2024-03-28T22:37:00.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.789+0000 7f1e83ff8740 1 -- start start 2024-03-28T22:37:00.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.790+0000 7f1e83ff8740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55987042af90 0x55987042b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:00.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.790+0000 7f1e83ff8740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55987042b980 0x559870430480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:00.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.790+0000 7f1e83ff8740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559870430af0 0x559870432f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:00.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.790+0000 7f1e83ff8740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55987038fc40 con 0x55987042b980 2024-03-28T22:37:00.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.790+0000 7f1e83ff8740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x559870338c40 con 0x55987042af90 2024-03-28T22:37:00.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.790+0000 7f1e83ff8740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55987038bce0 con 0x559870430af0 2024-03-28T22:37:00.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.790+0000 7f1e7b200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55987042af90 0x55987042b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:00.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.790+0000 7f1e7b200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55987042af90 0x55987042b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:37578/0 (socket says 172.21.15.43:37578) 2024-03-28T22:37:00.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.790+0000 7f1e7b200640 1 -- 172.21.15.43:0/3185868222 learned_addr learned my addr 172.21.15.43:0/3185868222 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:00.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.790+0000 7f1e79e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559870430af0 0x559870432f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:00.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.790+0000 7f1e7a800640 1 --2- 172.21.15.43:0/3185868222 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55987042b980 0x559870430480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:00.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.790+0000 7f1e7b200640 1 -- 172.21.15.43:0/3185868222 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559870430af0 msgr2=0x559870432f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:00.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.790+0000 7f1e7b200640 1 --2- 172.21.15.43:0/3185868222 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559870430af0 0x559870432f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.790+0000 7f1e7b200640 1 -- 172.21.15.43:0/3185868222 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55987042b980 msgr2=0x559870430480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:00.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.790+0000 7f1e7b200640 1 --2- 172.21.15.43:0/3185868222 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55987042b980 0x559870430480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.790+0000 7f1e7b200640 1 -- 172.21.15.43:0/3185868222 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55987016c7e0 con 0x55987042af90 2024-03-28T22:37:00.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.790+0000 7f1e7a800640 1 --2- 172.21.15.43:0/3185868222 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55987042b980 0x559870430480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:37:00.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.791+0000 7f1e7b200640 1 --2- 172.21.15.43:0/3185868222 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55987042af90 0x55987042b3b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1e74009df0 tx=0x7f1e7400f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e09fd3e837a6b19b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:00.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.791+0000 7f1e73400640 1 --2- 172.21.15.43:0/2827323983 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f1e600a1e90 0x7f1e600a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:00.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.791+0000 7f1e67400640 1 -- 172.21.15.43:0/3185868222 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e74003970 con 0x55987042af90 2024-03-28T22:37:00.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.791+0000 7f1e67400640 1 -- 172.21.15.43:0/3185868222 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e74004030 con 0x55987042af90 2024-03-28T22:37:00.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.791+0000 7f1e67400640 1 -- 172.21.15.43:0/3185868222 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e740118c0 con 0x55987042af90 2024-03-28T22:37:00.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.791+0000 7f1e73400640 1 -- 172.21.15.43:0/2827323983 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f1e6c0ce670 con 0x7f1e7c1021d0 2024-03-28T22:37:00.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.791+0000 7f1e7a800640 1 --2- 172.21.15.43:0/2827323983 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f1e600a1e90 0x7f1e600a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:00.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.792+0000 7f1e83ff8740 1 -- 172.21.15.43:0/3185868222 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55987042af90 msgr2=0x55987042b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:00.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.792+0000 7f1e83ff8740 1 --2- 172.21.15.43:0/3185868222 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55987042af90 0x55987042b3b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1e74009df0 tx=0x7f1e7400f730 comp rx=0 tx=0).stop 2024-03-28T22:37:00.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.792+0000 7f1e7a800640 1 --2- 172.21.15.43:0/2827323983 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f1e600a1e90 0x7f1e600a4350 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x55987042bd80 tx=0x7f1e6c018420 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:00.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.792+0000 7f1e83ff8740 1 -- 172.21.15.43:0/3185868222 shutdown_connections 2024-03-28T22:37:00.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.792+0000 7f1e83ff8740 1 --2- 172.21.15.43:0/3185868222 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559870430af0 0x559870432f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.792+0000 7f1e83ff8740 1 --2- 172.21.15.43:0/3185868222 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55987042b980 0x559870430480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.792+0000 7f1e83ff8740 1 --2- 172.21.15.43:0/3185868222 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55987042af90 0x55987042b3b0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.792+0000 7f1e83ff8740 1 -- 172.21.15.43:0/3185868222 >> 172.21.15.43:0/3185868222 conn(0x559870426500 msgr2=0x559870426900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:00.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.792+0000 7f1e83ff8740 1 -- 172.21.15.43:0/3185868222 shutdown_connections 2024-03-28T22:37:00.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.792+0000 7f1e83ff8740 1 -- 172.21.15.43:0/3185868222 wait complete. 2024-03-28T22:37:00.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.793+0000 7f1e83ff8740 1 -- start start 2024-03-28T22:37:00.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.793+0000 7f1e83ff8740 1 Processor -- start 2024-03-28T22:37:00.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.794+0000 7f1e83ff8740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55987042af90 0x559870435430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:00.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.794+0000 7f1e83ff8740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55987042b980 0x559870435970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:00.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.794+0000 7f1e83ff8740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559870430af0 0x55987043c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:00.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.794+0000 7f1e83ff8740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x559870433aa0 con 0x55987042b980 2024-03-28T22:37:00.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.794+0000 7f1e83ff8740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55987032e250 con 0x55987042af90 2024-03-28T22:37:00.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.794+0000 7f1e83ff8740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x559870433c20 con 0x559870430af0 2024-03-28T22:37:00.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.794+0000 7f1e83ff8740 10 client.0 _ll_register_callbacks cb 0x55987034fe70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:37:00.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.794+0000 7f1e7b200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55987042af90 0x559870435430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:00.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.794+0000 7f1e7b200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55987042af90 0x559870435430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:37586/0 (socket says 172.21.15.43:37586) 2024-03-28T22:37:00.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.794+0000 7f1e79e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55987042b980 0x559870435970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:00.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.794+0000 7f1e7b200640 1 -- 172.21.15.43:0/3911847010 learned_addr learned my addr 172.21.15.43:0/3911847010 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:00.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.794+0000 7f1e7b200640 1 --2- 172.21.15.43:0/3911847010 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559870430af0 0x55987043c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:00.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.794+0000 7f1e79e00640 1 -- 172.21.15.43:0/3911847010 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559870430af0 msgr2=0x55987043c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:00.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.795+0000 7f1e79e00640 1 --2- 172.21.15.43:0/3911847010 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559870430af0 0x55987043c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.795+0000 7f1e79e00640 1 -- 172.21.15.43:0/3911847010 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55987042af90 msgr2=0x559870435430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:00.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.795+0000 7f1e79e00640 1 --2- 172.21.15.43:0/3911847010 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55987042af90 0x559870435430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.795+0000 7f1e79e00640 1 -- 172.21.15.43:0/3911847010 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598701ccf70 con 0x55987042b980 2024-03-28T22:37:00.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.795+0000 7f1e79e00640 1 --2- 172.21.15.43:0/3911847010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55987042b980 0x559870435970 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1e68009230 tx=0x7f1e68004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92768632fcf14585 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:00.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.795+0000 7f1e5be00640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:37:00.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.795+0000 7f1e5be00640 1 -- 172.21.15.43:0/3911847010 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e68005040 con 0x55987042b980 2024-03-28T22:37:00.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.795+0000 7f1e5be00640 1 -- 172.21.15.43:0/3911847010 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f1e680051e0 con 0x55987042b980 2024-03-28T22:37:00.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.795+0000 7f1e83ff8740 1 -- 172.21.15.43:0/3911847010 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e74004030 con 0x55987042b980 2024-03-28T22:37:00.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.795+0000 7f1e5be00640 1 -- 172.21.15.43:0/3911847010 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1e68053050 con 0x55987042b980 2024-03-28T22:37:00.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.796+0000 7f1e83ff8740 10 client.15510 fetch_fsmap learned FSMap version 29 2024-03-28T22:37:00.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.796+0000 7f1e83ff8740 1 -- 172.21.15.43:0/3911847010 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559870064170 con 0x55987042b980 2024-03-28T22:37:00.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.796+0000 7f1e5be00640 1 -- 172.21.15.43:0/3911847010 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e74004030 con 0x55987042b980 2024-03-28T22:37:00.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.796+0000 7f1e5be00640 1 -- 172.21.15.43:0/3911847010 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f1e68003680 con 0x55987042b980 2024-03-28T22:37:00.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.796+0000 7f1e83ff8740 10 client.15510 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:37:00.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.796+0000 7f1e83ff8740 10 client.15510 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:37:00.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.796+0000 7f1e83ff8740 20 client.15510 populate_metadata read hostname 'smithi043' 2024-03-28T22:37:00.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.796+0000 7f1e83ff8740 1 --2- 172.21.15.43:0/3911847010 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5598704411e0 0x559870443660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:00.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.796+0000 7f1e83ff8740 4 client.15510 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:37:00.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.796+0000 7f1e83ff8740 1 -- 172.21.15.43:0/3911847010 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55987039d180 con 0x5598704411e0 2024-03-28T22:37:00.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.797+0000 7f1e7b200640 1 --2- 172.21.15.43:0/3911847010 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5598704411e0 0x559870443660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:00.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.797+0000 7f1e7b200640 1 --2- 172.21.15.43:0/3911847010 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5598704411e0 0x559870443660 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=edc3b66260eae5ff server_cookie=aeded4b19343105f in_seq=0 out_seq=0 2024-03-28T22:37:00.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.797+0000 7f1e5be00640 10 client.15510 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:00.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.798+0000 7f1e5be00640 1 -- 172.21.15.43:0/3911847010 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55987039d180 con 0x5598704411e0 2024-03-28T22:37:00.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.798+0000 7f1e5be00640 10 client.15510 handle_command_reply: tid=0 2024-03-28T22:37:00.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.798+0000 7f1e83ff8740 1 client.15510 shutdown 2024-03-28T22:37:00.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.798+0000 7f1e83ff8740 1 -- 172.21.15.43:0/3911847010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55987042b980 msgr2=0x559870435970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:00.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.798+0000 7f1e83ff8740 1 --2- 172.21.15.43:0/3911847010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55987042b980 0x559870435970 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1e68009230 tx=0x7f1e68004270 comp rx=0 tx=0).stop 2024-03-28T22:37:00.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.798+0000 7f1e83ff8740 1 -- 172.21.15.43:0/3911847010 shutdown_connections 2024-03-28T22:37:00.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.798+0000 7f1e83ff8740 1 --2- 172.21.15.43:0/3911847010 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559870430af0 0x55987043c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.798+0000 7f1e83ff8740 1 --2- 172.21.15.43:0/3911847010 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55987042b980 0x559870435970 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.798+0000 7f1e83ff8740 1 --2- 172.21.15.43:0/3911847010 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5598704411e0 0x559870443660 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.798+0000 7f1e83ff8740 1 --2- 172.21.15.43:0/3911847010 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55987042af90 0x559870435430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.798+0000 7f1e83ff8740 1 -- 172.21.15.43:0/3911847010 >> 172.21.15.43:0/3911847010 conn(0x559870426500 msgr2=0x559870425700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:00.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.798+0000 7f1e5be00640 0 client.15510 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:00.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.798+0000 7f1e83ff8740 1 -- 172.21.15.43:0/3911847010 shutdown_connections 2024-03-28T22:37:00.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.798+0000 7f1e83ff8740 1 -- 172.21.15.43:0/3911847010 wait complete. 2024-03-28T22:37:00.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.798+0000 7f1e83ff8740 20 client.15510 trim_cache size 0 max 16384 2024-03-28T22:37:00.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.818+0000 7f1e83ff8740 1 Processor -- start 2024-03-28T22:37:00.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.818+0000 7f1e83ff8740 1 -- start start 2024-03-28T22:37:00.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.818+0000 7f1e83ff8740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559870426500 0x559870429e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:00.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.818+0000 7f1e83ff8740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55987042af90 0x55987043c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:00.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.818+0000 7f1e83ff8740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55987042b980 0x559870447f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:00.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.818+0000 7f1e83ff8740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7c1ac660 con 0x559870426500 2024-03-28T22:37:00.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.818+0000 7f1e83ff8740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7c1ae1a0 con 0x55987042b980 2024-03-28T22:37:00.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.818+0000 7f1e83ff8740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7c1acd30 con 0x55987042af90 2024-03-28T22:37:00.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.818+0000 7f1e7b200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559870426500 0x559870429e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:00.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.819+0000 7f1e79e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55987042af90 0x55987043c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:00.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.819+0000 7f1e7b200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559870426500 0x559870429e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41024/0 (socket says 172.21.15.43:41024) 2024-03-28T22:37:00.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.819+0000 7f1e7b200640 1 -- 172.21.15.43:0/695537290 learned_addr learned my addr 172.21.15.43:0/695537290 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:00.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.819+0000 7f1e7b200640 1 --2- 172.21.15.43:0/695537290 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55987042b980 0x559870447f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:00.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.819+0000 7f1e79e00640 1 -- 172.21.15.43:0/695537290 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55987042b980 msgr2=0x559870447f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:00.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.819+0000 7f1e79e00640 1 --2- 172.21.15.43:0/695537290 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55987042b980 0x559870447f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.819+0000 7f1e79e00640 1 -- 172.21.15.43:0/695537290 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559870426500 msgr2=0x559870429e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:00.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.819+0000 7f1e79e00640 1 --2- 172.21.15.43:0/695537290 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559870426500 0x559870429e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.819+0000 7f1e79e00640 1 -- 172.21.15.43:0/695537290 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55987042ad80 con 0x55987042af90 2024-03-28T22:37:00.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.819+0000 7f1e7b200640 1 --2- 172.21.15.43:0/695537290 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55987042b980 0x559870447f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:37:00.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.819+0000 7f1e79e00640 1 --2- 172.21.15.43:0/695537290 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55987042af90 0x55987043c970 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1e68003cc0 tx=0x7f1e68000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=563c08d2a5dfa366 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:00.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.820+0000 7f1e66a00640 1 -- 172.21.15.43:0/695537290 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e68003d40 con 0x55987042af90 2024-03-28T22:37:00.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.820+0000 7f1e66a00640 1 -- 172.21.15.43:0/695537290 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e68005610 con 0x55987042af90 2024-03-28T22:37:00.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.820+0000 7f1e66a00640 1 -- 172.21.15.43:0/695537290 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e68034b80 con 0x55987042af90 2024-03-28T22:37:00.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.821+0000 7f1e83ff8740 1 -- 172.21.15.43:0/695537290 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55987042af90 msgr2=0x55987043c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:00.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.821+0000 7f1e83ff8740 1 --2- 172.21.15.43:0/695537290 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55987042af90 0x55987043c970 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1e68003cc0 tx=0x7f1e68000bc0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.821+0000 7f1e83ff8740 1 -- 172.21.15.43:0/695537290 shutdown_connections 2024-03-28T22:37:00.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.821+0000 7f1e83ff8740 1 --2- 172.21.15.43:0/695537290 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55987042b980 0x559870447f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.821+0000 7f1e83ff8740 1 --2- 172.21.15.43:0/695537290 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55987042af90 0x55987043c970 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.821+0000 7f1e83ff8740 1 --2- 172.21.15.43:0/695537290 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559870426500 0x559870429e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.821+0000 7f1e83ff8740 1 -- 172.21.15.43:0/695537290 >> 172.21.15.43:0/695537290 conn(0x559870421580 msgr2=0x559870429760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:00.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.821+0000 7f1e83ff8740 1 -- 172.21.15.43:0/695537290 shutdown_connections 2024-03-28T22:37:00.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.821+0000 7f1e83ff8740 1 -- 172.21.15.43:0/695537290 wait complete. 2024-03-28T22:37:00.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.821+0000 7f1e83ff8740 1 -- start start 2024-03-28T22:37:00.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.822+0000 7f1e83ff8740 1 Processor -- start 2024-03-28T22:37:00.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.822+0000 7f1e83ff8740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559870426500 0x55987043c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:00.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.822+0000 7f1e83ff8740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55987042af90 0x55987043ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:00.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.822+0000 7f1e83ff8740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55987042b980 0x55987043b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:00.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.822+0000 7f1e83ff8740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7c1adad0 con 0x55987042b980 2024-03-28T22:37:00.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.822+0000 7f1e83ff8740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7c1ad400 con 0x55987042af90 2024-03-28T22:37:00.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.822+0000 7f1e83ff8740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7c1aa640 con 0x559870426500 2024-03-28T22:37:00.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.823+0000 7f1e83ff8740 10 client.0 _ll_register_callbacks cb 0x7f1e68009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:37:00.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.823+0000 7f1e79e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55987042af90 0x55987043ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:00.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.823+0000 7f1e79e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55987042af90 0x55987043ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:37608/0 (socket says 172.21.15.43:37608) 2024-03-28T22:37:00.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.823+0000 7f1e7b200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55987042b980 0x55987043b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:00.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.823+0000 7f1e79e00640 1 -- 172.21.15.43:0/1283979667 learned_addr learned my addr 172.21.15.43:0/1283979667 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:00.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.823+0000 7f1e7b200640 1 --2- 172.21.15.43:0/1283979667 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559870426500 0x55987043c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:00.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.823+0000 7f1e79e00640 1 -- 172.21.15.43:0/1283979667 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559870426500 msgr2=0x55987043c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:00.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.823+0000 7f1e79e00640 1 --2- 172.21.15.43:0/1283979667 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559870426500 0x55987043c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.823+0000 7f1e79e00640 1 -- 172.21.15.43:0/1283979667 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55987042b980 msgr2=0x55987043b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:00.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.823+0000 7f1e79e00640 1 --2- 172.21.15.43:0/1283979667 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55987042b980 0x55987043b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.823+0000 7f1e79e00640 1 -- 172.21.15.43:0/1283979667 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55987043b700 con 0x55987042af90 2024-03-28T22:37:00.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.823+0000 7f1e7b200640 1 --2- 172.21.15.43:0/1283979667 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559870426500 0x55987043c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:37:00.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.823+0000 7f1e79e00640 1 --2- 172.21.15.43:0/1283979667 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55987042af90 0x55987043ac80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1e6802c7b0 tx=0x7f1e68009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=606e49eecfb81599 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:00.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.823+0000 7f1e5be00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:37:00.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.824+0000 7f1e5be00640 1 -- 172.21.15.43:0/1283979667 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e68004250 con 0x55987042af90 2024-03-28T22:37:00.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.824+0000 7f1e5be00640 1 -- 172.21.15.43:0/1283979667 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f1e68034650 con 0x55987042af90 2024-03-28T22:37:00.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.824+0000 7f1e83ff8740 1 -- 172.21.15.43:0/1283979667 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e68005610 con 0x55987042af90 2024-03-28T22:37:00.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.824+0000 7f1e5be00640 1 -- 172.21.15.43:0/1283979667 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1e6806a050 con 0x55987042af90 2024-03-28T22:37:00.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.824+0000 7f1e83ff8740 10 client.25631 fetch_fsmap learned FSMap version 29 2024-03-28T22:37:00.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.824+0000 7f1e83ff8740 1 -- 172.21.15.43:0/1283979667 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5598704a8310 con 0x55987042af90 2024-03-28T22:37:00.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.824+0000 7f1e5be00640 1 -- 172.21.15.43:0/1283979667 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e68005610 con 0x55987042af90 2024-03-28T22:37:00.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.825+0000 7f1e5be00640 1 -- 172.21.15.43:0/1283979667 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f1e680586e0 con 0x55987042af90 2024-03-28T22:37:00.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.825+0000 7f1e83ff8740 10 client.25631 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:37:00.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.825+0000 7f1e83ff8740 10 client.25631 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:37:00.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.825+0000 7f1e83ff8740 20 client.25631 populate_metadata read hostname 'smithi043' 2024-03-28T22:37:00.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.825+0000 7f1e83ff8740 1 --2- 172.21.15.43:0/1283979667 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559870430af0 0x5598704aacc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:00.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.825+0000 7f1e83ff8740 4 client.25631 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:37:00.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.825+0000 7f1e83ff8740 1 -- 172.21.15.43:0/1283979667 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5598703974d0 con 0x559870430af0 2024-03-28T22:37:00.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.825+0000 7f1e7b200640 1 --2- 172.21.15.43:0/1283979667 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559870430af0 0x5598704aacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:00.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.825+0000 7f1e7b200640 1 --2- 172.21.15.43:0/1283979667 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559870430af0 0x5598704aacc0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ef10d1ccb6402c6 server_cookie=93acce161d6b3761 in_seq=0 out_seq=0 2024-03-28T22:37:00.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.825+0000 7f1e5be00640 10 client.25631 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:00.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.827+0000 7f1e5be00640 1 -- 172.21.15.43:0/1283979667 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5598703974d0 con 0x559870430af0 2024-03-28T22:37:00.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.827+0000 7f1e5be00640 10 client.25631 handle_command_reply: tid=0 2024-03-28T22:37:00.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.827+0000 7f1e83ff8740 1 client.25631 shutdown 2024-03-28T22:37:00.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.827+0000 7f1e83ff8740 1 -- 172.21.15.43:0/1283979667 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55987042af90 msgr2=0x55987043ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:00.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.827+0000 7f1e83ff8740 1 --2- 172.21.15.43:0/1283979667 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55987042af90 0x55987043ac80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1e6802c7b0 tx=0x7f1e68009260 comp rx=0 tx=0).stop 2024-03-28T22:37:00.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.827+0000 7f1e83ff8740 1 -- 172.21.15.43:0/1283979667 shutdown_connections 2024-03-28T22:37:00.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.827+0000 7f1e83ff8740 1 --2- 172.21.15.43:0/1283979667 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559870430af0 0x5598704aacc0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.827+0000 7f1e83ff8740 1 --2- 172.21.15.43:0/1283979667 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55987042af90 0x55987043ac80 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.827+0000 7f1e83ff8740 1 --2- 172.21.15.43:0/1283979667 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55987042b980 0x55987043b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.827+0000 7f1e83ff8740 1 --2- 172.21.15.43:0/1283979667 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559870426500 0x55987043c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.827+0000 7f1e5be00640 0 client.25631 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:00.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.827+0000 7f1e83ff8740 1 -- 172.21.15.43:0/1283979667 >> 172.21.15.43:0/1283979667 conn(0x559870421580 msgr2=0x559870429ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:00.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.827+0000 7f1e83ff8740 1 -- 172.21.15.43:0/1283979667 shutdown_connections 2024-03-28T22:37:00.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.827+0000 7f1e83ff8740 1 -- 172.21.15.43:0/1283979667 wait complete. 2024-03-28T22:37:00.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.828+0000 7f1e83ff8740 20 client.25631 trim_cache size 0 max 16384 2024-03-28T22:37:00.828 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:37:00.828 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-03-28T22:37:00.829 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "18052a97-6246-4652-91f5-2f6eb6c8a975", 2024-03-28T22:37:00.829 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-03-28T22:37:00.829 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:37:00.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.828+0000 7f1e81400640 1 -- 172.21.15.43:0/2827323983 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f1e600a1e90 msgr2=0x7f1e600a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:00.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.828+0000 7f1e81400640 1 --2- 172.21.15.43:0/2827323983 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f1e600a1e90 0x7f1e600a4350 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x55987042bd80 tx=0x7f1e6c018420 comp rx=0 tx=0).stop 2024-03-28T22:37:00.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.828+0000 7f1e81400640 1 -- 172.21.15.43:0/2827323983 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1e7c1021d0 msgr2=0x7f1e7c101b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:00.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.828+0000 7f1e81400640 1 --2- 172.21.15.43:0/2827323983 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1e7c1021d0 0x7f1e7c101b40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c00e9f0 tx=0x7f1e6c00eeb0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.828+0000 7f1e81400640 1 -- 172.21.15.43:0/2827323983 shutdown_connections 2024-03-28T22:37:00.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.828+0000 7f1e81400640 1 --2- 172.21.15.43:0/2827323983 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1e7c107430 0x7f1e7c1006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.828+0000 7f1e81400640 1 --2- 172.21.15.43:0/2827323983 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1e7c104b00 0x7f1e7c100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.828+0000 7f1e81400640 1 --2- 172.21.15.43:0/2827323983 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f1e600a1e90 0x7f1e600a4350 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.828+0000 7f1e81400640 1 --2- 172.21.15.43:0/2827323983 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1e7c1021d0 0x7f1e7c101b40 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:00.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.828+0000 7f1e81400640 1 -- 172.21.15.43:0/2827323983 >> 172.21.15.43:0/2827323983 conn(0x7f1e7c0fbbe0 msgr2=0x7f1e7c102a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:00.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.829+0000 7f1e81400640 1 -- 172.21.15.43:0/2827323983 shutdown_connections 2024-03-28T22:37:00.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:00.829+0000 7f1e81400640 1 -- 172.21.15.43:0/2827323983 wait complete. 2024-03-28T22:37:00.846 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:37:00.846 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:37:01.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.041+0000 7fcf73200640 1 -- 172.21.15.43:0/271623546 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fcf6c02d170 con 0x7fcf6c102020 2024-03-28T22:37:01.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.066+0000 7f9d2ac00640 1 -- 172.21.15.43:0/844185972 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d24103200 msgr2=0x7f9d24109a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:01.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.066+0000 7f9d2ac00640 1 --2- 172.21.15.43:0/844185972 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d24103200 0x7f9d24109a90 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c009be0 tx=0x7f9d1c02bef0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.066+0000 7f9d2ac00640 1 -- 172.21.15.43:0/844185972 shutdown_connections 2024-03-28T22:37:01.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.066+0000 7f9d2ac00640 1 --2- 172.21.15.43:0/844185972 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9d24102840 0x7f9d24102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.066+0000 7f9d2ac00640 1 --2- 172.21.15.43:0/844185972 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d24103200 0x7f9d24109a90 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.066+0000 7f9d2ac00640 1 --2- 172.21.15.43:0/844185972 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9d24101650 0x7f9d24101a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.066+0000 7f9d2ac00640 1 -- 172.21.15.43:0/844185972 >> 172.21.15.43:0/844185972 conn(0x7f9d240fcbc0 msgr2=0x7f9d240ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:01.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.066+0000 7f9d2ac00640 1 -- 172.21.15.43:0/844185972 shutdown_connections 2024-03-28T22:37:01.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.066+0000 7f9d2ac00640 1 -- 172.21.15.43:0/844185972 wait complete. 2024-03-28T22:37:01.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.068+0000 7f9d2ac00640 1 Processor -- start 2024-03-28T22:37:01.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.068+0000 7f9d2ac00640 1 -- start start 2024-03-28T22:37:01.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.068+0000 7f9d2ac00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9d24101650 0x7f9d241978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:01.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.068+0000 7f9d2ac00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d24102840 0x7f9d24197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:01.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.068+0000 7f9d2ac00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9d24103200 0x7f9d2419ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:01.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.068+0000 7f9d2ac00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9d2410d300 con 0x7f9d24101650 2024-03-28T22:37:01.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.068+0000 7f9d2ac00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9d2410d180 con 0x7f9d24103200 2024-03-28T22:37:01.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.068+0000 7f9d2ac00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9d2410d480 con 0x7f9d24102840 2024-03-28T22:37:01.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.068+0000 7f9d28e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d24102840 0x7f9d24197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:01.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.069+0000 7f9d29800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9d24101650 0x7f9d241978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:01.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.069+0000 7f9d2a200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9d24103200 0x7f9d2419ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:01.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.069+0000 7f9d28e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d24102840 0x7f9d24197e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:46978/0 (socket says 172.21.15.43:46978) 2024-03-28T22:37:01.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.069+0000 7f9d28e00640 1 -- 172.21.15.43:0/512331096 learned_addr learned my addr 172.21.15.43:0/512331096 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:01.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.069+0000 7f9d2a200640 1 -- 172.21.15.43:0/512331096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d24102840 msgr2=0x7f9d24197e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:01.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.069+0000 7f9d2a200640 1 --2- 172.21.15.43:0/512331096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d24102840 0x7f9d24197e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.069+0000 7f9d2a200640 1 -- 172.21.15.43:0/512331096 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9d24101650 msgr2=0x7f9d241978c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:01.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.069+0000 7f9d2a200640 1 --2- 172.21.15.43:0/512331096 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9d24101650 0x7f9d241978c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.069+0000 7f9d2a200640 1 -- 172.21.15.43:0/512331096 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d2419f4e0 con 0x7f9d24103200 2024-03-28T22:37:01.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.069+0000 7f9d2a200640 1 --2- 172.21.15.43:0/512331096 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9d24103200 0x7f9d2419ee70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c02c430 tx=0x7f9d1c0337b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:01.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.069+0000 7f9d22000640 1 -- 172.21.15.43:0/512331096 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d1c043020 con 0x7f9d24103200 2024-03-28T22:37:01.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.069+0000 7f9d22000640 1 -- 172.21.15.43:0/512331096 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d1c02ca60 con 0x7f9d24103200 2024-03-28T22:37:01.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.070+0000 7f9d2ac00640 1 -- 172.21.15.43:0/512331096 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d2419f770 con 0x7f9d24103200 2024-03-28T22:37:01.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.070+0000 7f9d22000640 1 -- 172.21.15.43:0/512331096 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d1c034690 con 0x7f9d24103200 2024-03-28T22:37:01.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.070+0000 7f9d2ac00640 1 -- 172.21.15.43:0/512331096 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d2419fd10 con 0x7f9d24103200 2024-03-28T22:37:01.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.074+0000 7f9d2efd3740 1 Processor -- start 2024-03-28T22:37:01.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.074+0000 7f9d22000640 1 -- 172.21.15.43:0/512331096 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 19) ==== 136807+0+0 (secure 0 0 0) 0x7f9d1c046020 con 0x7f9d24103200 2024-03-28T22:37:01.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.074+0000 7f9d2efd3740 1 -- start start 2024-03-28T22:37:01.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.075+0000 7f9d2efd3740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cf421b6440 0x55cf421b6860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:01.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.075+0000 7f9d2efd3740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cf421b6e30 0x55cf421bb930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:01.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.075+0000 7f9d2efd3740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf421bbfa0 0x55cf421be430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:01.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.075+0000 7f9d2efd3740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55cf42117100 con 0x55cf421bbfa0 2024-03-28T22:37:01.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.075+0000 7f9d2efd3740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55cf420bfba0 con 0x55cf421b6440 2024-03-28T22:37:01.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.075+0000 7f9d2efd3740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55cf421131a0 con 0x55cf421b6e30 2024-03-28T22:37:01.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.075+0000 7f9d29800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cf421b6e30 0x55cf421bb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:01.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.075+0000 7f9d29800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cf421b6e30 0x55cf421bb930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:46984/0 (socket says 172.21.15.43:46984) 2024-03-28T22:37:01.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.075+0000 7f9d29800640 1 -- 172.21.15.43:0/78078601 learned_addr learned my addr 172.21.15.43:0/78078601 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:01.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.075+0000 7f9d28e00640 1 --2- 172.21.15.43:0/78078601 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cf421b6440 0x55cf421b6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:01.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.075+0000 7f9d28e00640 1 --2- 172.21.15.43:0/78078601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf421bbfa0 0x55cf421be430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:01.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.075+0000 7f9d29800640 1 -- 172.21.15.43:0/78078601 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cf421b6440 msgr2=0x55cf421b6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:01.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.075+0000 7f9d29800640 1 --2- 172.21.15.43:0/78078601 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cf421b6440 0x55cf421b6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.075+0000 7f9d29800640 1 -- 172.21.15.43:0/78078601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf421bbfa0 msgr2=0x55cf421be430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:01.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.075+0000 7f9d29800640 1 --2- 172.21.15.43:0/78078601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf421bbfa0 0x55cf421be430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.075+0000 7f9d29800640 1 -- 172.21.15.43:0/78078601 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf420f01a0 con 0x55cf421b6e30 2024-03-28T22:37:01.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.076+0000 7f9d29800640 1 --2- 172.21.15.43:0/78078601 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cf421b6e30 0x55cf421bb930 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9d18009ca0 tx=0x7f9d18011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=605354767c50359 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:01.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.076+0000 7f9d16a00640 1 -- 172.21.15.43:0/78078601 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d18004230 con 0x55cf421b6e30 2024-03-28T22:37:01.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.076+0000 7f9d16a00640 1 -- 172.21.15.43:0/78078601 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d180043d0 con 0x55cf421b6e30 2024-03-28T22:37:01.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.076+0000 7f9d16a00640 1 -- 172.21.15.43:0/78078601 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d18004e50 con 0x55cf421b6e30 2024-03-28T22:37:01.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.076+0000 7f9d22000640 1 --2- 172.21.15.43:0/512331096 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9d0c0a1e90 0x7f9d0c0a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:01.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.076+0000 7f9d22000640 1 -- 172.21.15.43:0/512331096 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f9d1c0f07a0 con 0x7f9d24103200 2024-03-28T22:37:01.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.076+0000 7f9d28e00640 1 --2- 172.21.15.43:0/512331096 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9d0c0a1e90 0x7f9d0c0a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:01.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.077+0000 7f9d2efd3740 1 -- 172.21.15.43:0/78078601 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cf421b6e30 msgr2=0x55cf421bb930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:01.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.077+0000 7f9d2efd3740 1 --2- 172.21.15.43:0/78078601 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cf421b6e30 0x55cf421bb930 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9d18009ca0 tx=0x7f9d18011810 comp rx=0 tx=0).stop 2024-03-28T22:37:01.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.077+0000 7f9d2efd3740 1 -- 172.21.15.43:0/78078601 shutdown_connections 2024-03-28T22:37:01.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.077+0000 7f9d2efd3740 1 --2- 172.21.15.43:0/78078601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf421bbfa0 0x55cf421be430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.077+0000 7f9d2efd3740 1 --2- 172.21.15.43:0/78078601 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cf421b6e30 0x55cf421bb930 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.077+0000 7f9d2efd3740 1 --2- 172.21.15.43:0/78078601 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cf421b6440 0x55cf421b6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.077+0000 7f9d2efd3740 1 -- 172.21.15.43:0/78078601 >> 172.21.15.43:0/78078601 conn(0x55cf42179a10 msgr2=0x55cf421ad9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:01.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.077+0000 7f9d2efd3740 1 -- 172.21.15.43:0/78078601 shutdown_connections 2024-03-28T22:37:01.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.077+0000 7f9d2efd3740 1 -- 172.21.15.43:0/78078601 wait complete. 2024-03-28T22:37:01.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.077+0000 7f9d28e00640 1 --2- 172.21.15.43:0/512331096 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9d0c0a1e90 0x7f9d0c0a4350 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x55cf421b7690 tx=0x7f9d10013040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:01.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.077+0000 7f9d2efd3740 1 -- start start 2024-03-28T22:37:01.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.078+0000 7f9d2efd3740 1 Processor -- start 2024-03-28T22:37:01.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.078+0000 7f9d2efd3740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cf421b6440 0x55cf421c09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:01.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.078+0000 7f9d2efd3740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cf421b6e30 0x55cf421c0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:01.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.078+0000 7f9d2efd3740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf421bbfa0 0x55cf421c7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:01.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.078+0000 7f9d2efd3740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55cf421bef50 con 0x55cf421bbfa0 2024-03-28T22:37:01.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.078+0000 7f9d2efd3740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55cf420b5250 con 0x55cf421b6e30 2024-03-28T22:37:01.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.078+0000 7f9d2efd3740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55cf421bf0d0 con 0x55cf421b6440 2024-03-28T22:37:01.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.078+0000 7f9d2efd3740 10 client.0 _ll_register_callbacks cb 0x55cf41e2abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:37:01.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.078+0000 7f9d29800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cf421b6440 0x55cf421c09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:01.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.078+0000 7f9d29800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cf421b6440 0x55cf421c09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:46998/0 (socket says 172.21.15.43:46998) 2024-03-28T22:37:01.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.078+0000 7f9d29800640 1 -- 172.21.15.43:0/2000403603 learned_addr learned my addr 172.21.15.43:0/2000403603 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:01.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.078+0000 7f9d28e00640 1 --2- 172.21.15.43:0/2000403603 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cf421b6e30 0x55cf421c0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:01.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.078+0000 7f9d2a200640 1 --2- 172.21.15.43:0/2000403603 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf421bbfa0 0x55cf421c7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:01.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.079+0000 7f9d29800640 1 -- 172.21.15.43:0/2000403603 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cf421b6e30 msgr2=0x55cf421c0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:01.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.079+0000 7f9d29800640 1 --2- 172.21.15.43:0/2000403603 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cf421b6e30 0x55cf421c0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.079+0000 7f9d29800640 1 -- 172.21.15.43:0/2000403603 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf421bbfa0 msgr2=0x55cf421c7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:01.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.079+0000 7f9d29800640 1 --2- 172.21.15.43:0/2000403603 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf421bbfa0 0x55cf421c7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.079+0000 7f9d29800640 1 -- 172.21.15.43:0/2000403603 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf420d1960 con 0x55cf421b6440 2024-03-28T22:37:01.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.079+0000 7f9d2a200640 1 --2- 172.21.15.43:0/2000403603 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf421bbfa0 0x55cf421c7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:37:01.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.079+0000 7f9d29800640 1 --2- 172.21.15.43:0/2000403603 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cf421b6440 0x55cf421c09b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9d18011d10 tx=0x7f9d18019b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da1625ffdfdc6c31 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:01.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.079+0000 7f9d0b400640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:37:01.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.080+0000 7f9d0b400640 1 -- 172.21.15.43:0/2000403603 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d18028020 con 0x55cf421b6440 2024-03-28T22:37:01.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.080+0000 7f9d0b400640 1 -- 172.21.15.43:0/2000403603 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f9d18024420 con 0x55cf421b6440 2024-03-28T22:37:01.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.080+0000 7f9d2efd3740 1 -- 172.21.15.43:0/2000403603 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d180043d0 con 0x55cf421b6440 2024-03-28T22:37:01.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.080+0000 7f9d0b400640 1 -- 172.21.15.43:0/2000403603 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9d18033050 con 0x55cf421b6440 2024-03-28T22:37:01.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.080+0000 7f9d2efd3740 10 client.25642 fetch_fsmap learned FSMap version 29 2024-03-28T22:37:01.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.080+0000 7f9d2efd3740 1 -- 172.21.15.43:0/2000403603 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cf41ef57d0 con 0x55cf421b6440 2024-03-28T22:37:01.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.081+0000 7f9d0b400640 1 -- 172.21.15.43:0/2000403603 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d180043d0 con 0x55cf421b6440 2024-03-28T22:37:01.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.081+0000 7f9d0b400640 1 -- 172.21.15.43:0/2000403603 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f9d18004c20 con 0x55cf421b6440 2024-03-28T22:37:01.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.081+0000 7f9d2efd3740 10 client.25642 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:37:01.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.081+0000 7f9d2efd3740 10 client.25642 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:37:01.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.081+0000 7f9d2efd3740 20 client.25642 populate_metadata read hostname 'smithi043' 2024-03-28T22:37:01.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.081+0000 7f9d2efd3740 1 --2- 172.21.15.43:0/2000403603 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55cf421cc590 0x55cf421cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:01.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.081+0000 7f9d2efd3740 4 client.25642 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:37:01.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.081+0000 7f9d2efd3740 1 -- 172.21.15.43:0/2000403603 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf42124640 con 0x55cf421cc590 2024-03-28T22:37:01.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.081+0000 7f9d28e00640 1 --2- 172.21.15.43:0/2000403603 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55cf421cc590 0x55cf421cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:01.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.082+0000 7f9d28e00640 1 --2- 172.21.15.43:0/2000403603 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55cf421cc590 0x55cf421cea10 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=946d0786b263780 server_cookie=9bc3d8a5be7bfd4e in_seq=0 out_seq=0 2024-03-28T22:37:01.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.082+0000 7f9d0b400640 10 client.25642 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:01.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.082+0000 7f9d0b400640 1 -- 172.21.15.43:0/2000403603 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cf42124640 con 0x55cf421cc590 2024-03-28T22:37:01.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.082+0000 7f9d0b400640 10 client.25642 handle_command_reply: tid=0 2024-03-28T22:37:01.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.082+0000 7f9d2efd3740 1 client.25642 shutdown 2024-03-28T22:37:01.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.083+0000 7f9d2efd3740 1 -- 172.21.15.43:0/2000403603 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cf421b6440 msgr2=0x55cf421c09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:01.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.083+0000 7f9d2efd3740 1 --2- 172.21.15.43:0/2000403603 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cf421b6440 0x55cf421c09b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9d18011d10 tx=0x7f9d18019b60 comp rx=0 tx=0).stop 2024-03-28T22:37:01.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.083+0000 7f9d2efd3740 1 -- 172.21.15.43:0/2000403603 shutdown_connections 2024-03-28T22:37:01.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.083+0000 7f9d2efd3740 1 --2- 172.21.15.43:0/2000403603 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55cf421cc590 0x55cf421cea10 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.083+0000 7f9d2efd3740 1 --2- 172.21.15.43:0/2000403603 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cf421b6e30 0x55cf421c0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.083+0000 7f9d0b400640 0 client.25642 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:01.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.083+0000 7f9d2efd3740 1 --2- 172.21.15.43:0/2000403603 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf421bbfa0 0x55cf421c7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.083+0000 7f9d2efd3740 1 --2- 172.21.15.43:0/2000403603 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cf421b6440 0x55cf421c09b0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.083+0000 7f9d2efd3740 1 -- 172.21.15.43:0/2000403603 >> 172.21.15.43:0/2000403603 conn(0x55cf42179a10 msgr2=0x55cf421acd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:01.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.083+0000 7f9d2efd3740 1 -- 172.21.15.43:0/2000403603 shutdown_connections 2024-03-28T22:37:01.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.083+0000 7f9d2efd3740 1 -- 172.21.15.43:0/2000403603 wait complete. 2024-03-28T22:37:01.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.083+0000 7f9d2efd3740 20 client.25642 trim_cache size 0 max 16384 2024-03-28T22:37:01.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.102+0000 7f9d2efd3740 1 Processor -- start 2024-03-28T22:37:01.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.102+0000 7f9d2efd3740 1 -- start start 2024-03-28T22:37:01.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.102+0000 7f9d2efd3740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cf421b6440 0x55cf421b12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:01.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.103+0000 7f9d2efd3740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cf421b6e30 0x55cf421c7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:01.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.103+0000 7f9d2efd3740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf421bbfa0 0x55cf421ac470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:01.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.103+0000 7f9d2efd3740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9d241a7f70 con 0x55cf421bbfa0 2024-03-28T22:37:01.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.103+0000 7f9d2efd3740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9d241a9ab0 con 0x55cf421b6e30 2024-03-28T22:37:01.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.103+0000 7f9d2efd3740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9d241a8640 con 0x55cf421b6440 2024-03-28T22:37:01.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.103+0000 7f9d29800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cf421b6440 0x55cf421b12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:01.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.103+0000 7f9d28e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cf421b6e30 0x55cf421c7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:01.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.103+0000 7f9d29800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cf421b6440 0x55cf421b12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:47004/0 (socket says 172.21.15.43:47004) 2024-03-28T22:37:01.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.103+0000 7f9d29800640 1 -- 172.21.15.43:0/1354719675 learned_addr learned my addr 172.21.15.43:0/1354719675 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:01.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.103+0000 7f9d2a200640 1 --2- 172.21.15.43:0/1354719675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf421bbfa0 0x55cf421ac470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:01.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.103+0000 7f9d29800640 1 -- 172.21.15.43:0/1354719675 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cf421b6e30 msgr2=0x55cf421c7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:01.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.103+0000 7f9d29800640 1 --2- 172.21.15.43:0/1354719675 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cf421b6e30 0x55cf421c7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.103+0000 7f9d29800640 1 -- 172.21.15.43:0/1354719675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf421bbfa0 msgr2=0x55cf421ac470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:01.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.103+0000 7f9d29800640 1 --2- 172.21.15.43:0/1354719675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf421bbfa0 0x55cf421ac470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.103+0000 7f9d29800640 1 -- 172.21.15.43:0/1354719675 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf41e2b3f0 con 0x55cf421b6440 2024-03-28T22:37:01.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.103+0000 7f9d29800640 1 --2- 172.21.15.43:0/1354719675 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cf421b6440 0x55cf421b12f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9d18012fd0 tx=0x7f9d18009600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d8c7c00ee12c870 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:01.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.104+0000 7f9d16000640 1 -- 172.21.15.43:0/1354719675 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d18028070 con 0x55cf421b6440 2024-03-28T22:37:01.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.104+0000 7f9d16000640 1 -- 172.21.15.43:0/1354719675 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d1801f040 con 0x55cf421b6440 2024-03-28T22:37:01.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.104+0000 7f9d16000640 1 -- 172.21.15.43:0/1354719675 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d18019e30 con 0x55cf421b6440 2024-03-28T22:37:01.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.104+0000 7f9d2efd3740 1 -- 172.21.15.43:0/1354719675 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cf421b6440 msgr2=0x55cf421b12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:01.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.104+0000 7f9d2efd3740 1 --2- 172.21.15.43:0/1354719675 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cf421b6440 0x55cf421b12f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9d18012fd0 tx=0x7f9d18009600 comp rx=0 tx=0).stop 2024-03-28T22:37:01.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.104+0000 7f9d2efd3740 1 -- 172.21.15.43:0/1354719675 shutdown_connections 2024-03-28T22:37:01.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.104+0000 7f9d2efd3740 1 --2- 172.21.15.43:0/1354719675 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cf421b6e30 0x55cf421c7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.104+0000 7f9d2efd3740 1 --2- 172.21.15.43:0/1354719675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf421bbfa0 0x55cf421ac470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.104+0000 7f9d2efd3740 1 --2- 172.21.15.43:0/1354719675 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cf421b6440 0x55cf421b12f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.104+0000 7f9d2efd3740 1 -- 172.21.15.43:0/1354719675 >> 172.21.15.43:0/1354719675 conn(0x55cf42179a10 msgr2=0x55cf421b0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:01.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.104+0000 7f9d2efd3740 1 -- 172.21.15.43:0/1354719675 shutdown_connections 2024-03-28T22:37:01.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.105+0000 7f9d2efd3740 1 -- 172.21.15.43:0/1354719675 wait complete. 2024-03-28T22:37:01.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.105+0000 7f9d2efd3740 1 -- start start 2024-03-28T22:37:01.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.105+0000 7f9d2efd3740 1 Processor -- start 2024-03-28T22:37:01.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.105+0000 7f9d2efd3740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cf421b6440 0x55cf421c7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:01.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.105+0000 7f9d2efd3740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf421b6e30 0x55cf421c4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:01.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.105+0000 7f9d2efd3740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cf421bbfa0 0x55cf421c4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:01.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.105+0000 7f9d2efd3740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9d241a93e0 con 0x55cf421b6e30 2024-03-28T22:37:01.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.105+0000 7f9d2efd3740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9d241a8d10 con 0x55cf421bbfa0 2024-03-28T22:37:01.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.105+0000 7f9d2efd3740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9d241a5f50 con 0x55cf421b6440 2024-03-28T22:37:01.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.106+0000 7f9d2efd3740 10 client.0 _ll_register_callbacks cb 0x7f9d10009af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:37:01.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.106+0000 7f9d29800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cf421b6440 0x55cf421c7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:01.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.106+0000 7f9d2a200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cf421bbfa0 0x55cf421c4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:01.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.106+0000 7f9d28e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf421b6e30 0x55cf421c4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:01.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.106+0000 7f9d29800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cf421b6440 0x55cf421c7f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:47016/0 (socket says 172.21.15.43:47016) 2024-03-28T22:37:01.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.106+0000 7f9d29800640 1 -- 172.21.15.43:0/837495770 learned_addr learned my addr 172.21.15.43:0/837495770 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:01.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.106+0000 7f9d2a200640 1 --2- 172.21.15.43:0/837495770 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cf421bbfa0 0x55cf421c4fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:37670/0 (socket says 172.21.15.43:37670) 2024-03-28T22:37:01.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.106+0000 7f9d29800640 1 -- 172.21.15.43:0/837495770 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cf421bbfa0 msgr2=0x55cf421c4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:01.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.106+0000 7f9d29800640 1 --2- 172.21.15.43:0/837495770 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cf421bbfa0 0x55cf421c4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.106+0000 7f9d29800640 1 -- 172.21.15.43:0/837495770 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf421b6e30 msgr2=0x55cf421c4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:01.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.106+0000 7f9d29800640 1 --2- 172.21.15.43:0/837495770 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf421b6e30 0x55cf421c4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.106+0000 7f9d29800640 1 -- 172.21.15.43:0/837495770 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf41deb0f0 con 0x55cf421b6440 2024-03-28T22:37:01.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.106+0000 7f9d29800640 1 --2- 172.21.15.43:0/837495770 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cf421b6440 0x55cf421c7f40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9d18005230 tx=0x7f9d18019d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fff6e37cd3ca91d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:01.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.106+0000 7f9d0b400640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:37:01.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.107+0000 7f9d0b400640 1 -- 172.21.15.43:0/837495770 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d18028070 con 0x55cf421b6440 2024-03-28T22:37:01.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.107+0000 7f9d2efd3740 1 -- 172.21.15.43:0/837495770 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d1801f040 con 0x55cf421b6440 2024-03-28T22:37:01.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.107+0000 7f9d0b400640 1 -- 172.21.15.43:0/837495770 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f9d18004260 con 0x55cf421b6440 2024-03-28T22:37:01.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.107+0000 7f9d0b400640 1 -- 172.21.15.43:0/837495770 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9d18049050 con 0x55cf421b6440 2024-03-28T22:37:01.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.107+0000 7f9d2efd3740 10 client.25648 fetch_fsmap learned FSMap version 29 2024-03-28T22:37:01.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.107+0000 7f9d2efd3740 1 -- 172.21.15.43:0/837495770 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cf42231a50 con 0x55cf421b6440 2024-03-28T22:37:01.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.107+0000 7f9d0b400640 1 -- 172.21.15.43:0/837495770 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d1801f040 con 0x55cf421b6440 2024-03-28T22:37:01.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.107+0000 7f9d0b400640 1 -- 172.21.15.43:0/837495770 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f9d180035f0 con 0x55cf421b6440 2024-03-28T22:37:01.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.107+0000 7f9d2efd3740 10 client.25648 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:37:01.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.107+0000 7f9d2efd3740 10 client.25648 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:37:01.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.107+0000 7f9d2efd3740 20 client.25648 populate_metadata read hostname 'smithi043' 2024-03-28T22:37:01.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.107+0000 7f9d2efd3740 1 --2- 172.21.15.43:0/837495770 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55cf42232350 0x55cf42234770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:01.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.107+0000 7f9d2efd3740 4 client.25648 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:37:01.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.107+0000 7f9d2efd3740 1 -- 172.21.15.43:0/837495770 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cf4211e990 con 0x55cf42232350 2024-03-28T22:37:01.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.108+0000 7f9d28e00640 1 --2- 172.21.15.43:0/837495770 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55cf42232350 0x55cf42234770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:01.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.108+0000 7f9d28e00640 1 --2- 172.21.15.43:0/837495770 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55cf42232350 0x55cf42234770 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcf6fa9381a0b828 server_cookie=57170841f5e5964 in_seq=0 out_seq=0 2024-03-28T22:37:01.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.108+0000 7f9d0b400640 10 client.25648 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:01.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.108+0000 7f9d0b400640 1 -- 172.21.15.43:0/837495770 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55cf4211e990 con 0x55cf42232350 2024-03-28T22:37:01.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.108+0000 7f9d0b400640 10 client.25648 handle_command_reply: tid=0 2024-03-28T22:37:01.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.108+0000 7f9d2efd3740 1 client.25648 shutdown 2024-03-28T22:37:01.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.108+0000 7f9d2efd3740 1 -- 172.21.15.43:0/837495770 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cf421b6440 msgr2=0x55cf421c7f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:01.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.108+0000 7f9d2efd3740 1 --2- 172.21.15.43:0/837495770 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cf421b6440 0x55cf421c7f40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9d18005230 tx=0x7f9d18019d70 comp rx=0 tx=0).stop 2024-03-28T22:37:01.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.108+0000 7f9d2efd3740 1 -- 172.21.15.43:0/837495770 shutdown_connections 2024-03-28T22:37:01.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.108+0000 7f9d2efd3740 1 --2- 172.21.15.43:0/837495770 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55cf42232350 0x55cf42234770 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.108+0000 7f9d2efd3740 1 --2- 172.21.15.43:0/837495770 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cf421bbfa0 0x55cf421c4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.108+0000 7f9d2efd3740 1 --2- 172.21.15.43:0/837495770 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cf421b6e30 0x55cf421c4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.108+0000 7f9d2efd3740 1 --2- 172.21.15.43:0/837495770 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cf421b6440 0x55cf421c7f40 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.109+0000 7f9d2efd3740 1 -- 172.21.15.43:0/837495770 >> 172.21.15.43:0/837495770 conn(0x55cf42179a10 msgr2=0x55cf421ac470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:01.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.108+0000 7f9d0b400640 0 client.25648 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:01.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.109+0000 7f9d2efd3740 1 -- 172.21.15.43:0/837495770 shutdown_connections 2024-03-28T22:37:01.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.109+0000 7f9d2efd3740 1 -- 172.21.15.43:0/837495770 wait complete. 2024-03-28T22:37:01.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.109+0000 7f9d2efd3740 20 client.25648 trim_cache size 0 max 16384 2024-03-28T22:37:01.113 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:37:01.113 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:37:01.114 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-03-28T22:37:01.114 INFO:teuthology.orchestra.run.smithi043.stdout: "18052a97-6246-4652-91f5-2f6eb6c8a975": { 2024-03-28T22:37:01.114 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-03-28T22:37:01.114 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "18052a97-6246-4652-91f5-2f6eb6c8a975", 2024-03-28T22:37:01.114 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-03-28T22:37:01.114 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:37:01.114 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:37:01.114 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:37:01.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.109+0000 7f9d2ac00640 1 -- 172.21.15.43:0/512331096 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9d0c0a1e90 msgr2=0x7f9d0c0a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:01.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.109+0000 7f9d2ac00640 1 --2- 172.21.15.43:0/512331096 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9d0c0a1e90 0x7f9d0c0a4350 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x55cf421b7690 tx=0x7f9d10013040 comp rx=0 tx=0).stop 2024-03-28T22:37:01.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.109+0000 7f9d2ac00640 1 -- 172.21.15.43:0/512331096 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9d24103200 msgr2=0x7f9d2419ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:01.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.109+0000 7f9d2ac00640 1 --2- 172.21.15.43:0/512331096 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9d24103200 0x7f9d2419ee70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c02c430 tx=0x7f9d1c0337b0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.109+0000 7f9d2ac00640 1 -- 172.21.15.43:0/512331096 shutdown_connections 2024-03-28T22:37:01.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.109+0000 7f9d2ac00640 1 --2- 172.21.15.43:0/512331096 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9d0c0a1e90 0x7f9d0c0a4350 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.109+0000 7f9d2ac00640 1 --2- 172.21.15.43:0/512331096 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9d24103200 0x7f9d2419ee70 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.109+0000 7f9d2ac00640 1 --2- 172.21.15.43:0/512331096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d24102840 0x7f9d24197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.109+0000 7f9d2ac00640 1 --2- 172.21.15.43:0/512331096 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9d24101650 0x7f9d241978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:01.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.110+0000 7f9d2ac00640 1 -- 172.21.15.43:0/512331096 >> 172.21.15.43:0/512331096 conn(0x7f9d240fcbc0 msgr2=0x7f9d24105ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:01.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.110+0000 7f9d2ac00640 1 -- 172.21.15.43:0/512331096 shutdown_connections 2024-03-28T22:37:01.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:01.110+0000 7f9d2ac00640 1 -- 172.21.15.43:0/512331096 wait complete. 2024-03-28T22:37:01.124 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:00 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1069313246' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:37:01.124 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:00 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2202629670' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:37:01.126 INFO:tasks.cephfs.filesystem:scrub status for tag:18052a97-6246-4652-91f5-2f6eb6c8a975 - {'path': '/', 'tag': '18052a97-6246-4652-91f5-2f6eb6c8a975', 'options': 'recursive,force'} 2024-03-28T22:37:01.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:00 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1069313246' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:37:01.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:00 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2202629670' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:37:01.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:00 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1069313246' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:37:01.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:00 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2202629670' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:37:02.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.016+0000 7fcf66a00640 1 -- 172.21.15.43:0/271623546 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mgrmap(e 20) ==== 136824+0+0 (secure 0 0 0) 0x7fcf60097670 con 0x7fcf6c102020 2024-03-28T22:37:02.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.018+0000 7fcf66a00640 1 -- 172.21.15.43:0/271623546 <== mon.2 v2:172.21.15.82:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v20) ==== 92+0+0 (secure 0 0 0) 0x7fcf600cf560 con 0x7fcf6c102020 2024-03-28T22:37:02.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.023+0000 7fcf73200640 1 -- 172.21.15.43:0/271623546 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fcf540a1e80 msgr2=0x7fcf540a4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:02.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.023+0000 7fcf73200640 1 --2- 172.21.15.43:0/271623546 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fcf540a1e80 0x7fcf540a4340 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c000c00 tx=0x7fcf5c005fd0 comp rx=0 tx=0).stop 2024-03-28T22:37:02.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.023+0000 7fcf73200640 1 -- 172.21.15.43:0/271623546 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcf6c102020 msgr2=0x7fcf6c199aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:02.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.023+0000 7fcf73200640 1 --2- 172.21.15.43:0/271623546 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcf6c102020 0x7fcf6c199aa0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fcf6000e9f0 tx=0x7fcf6000eeb0 comp rx=0 tx=0).stop 2024-03-28T22:37:02.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.024+0000 7fcf73200640 1 -- 172.21.15.43:0/271623546 shutdown_connections 2024-03-28T22:37:02.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.024+0000 7fcf73200640 1 --2- 172.21.15.43:0/271623546 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf6c107280 0x7fcf6c1a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:02.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.024+0000 7fcf73200640 1 --2- 172.21.15.43:0/271623546 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf6c104950 0x7fcf6c199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:02.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.024+0000 7fcf73200640 1 --2- 172.21.15.43:0/271623546 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7fcf540a1e80 0x7fcf540a4340 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:02.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.024+0000 7fcf73200640 1 --2- 172.21.15.43:0/271623546 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcf6c102020 0x7fcf6c199aa0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:02.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.024+0000 7fcf73200640 1 -- 172.21.15.43:0/271623546 >> 172.21.15.43:0/271623546 conn(0x7fcf6c0fba30 msgr2=0x7fcf6c1017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:02.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.024+0000 7fcf73200640 1 -- 172.21.15.43:0/271623546 shutdown_connections 2024-03-28T22:37:02.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.024+0000 7fcf73200640 1 -- 172.21.15.43:0/271623546 wait complete. 2024-03-28T22:37:02.045 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-03-28T22:37:02.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.234+0000 7f4282e00640 1 -- 172.21.15.43:0/3930028565 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f427c101d30 msgr2=0x7f427c102150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:02.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.234+0000 7f4282e00640 1 --2- 172.21.15.43:0/3930028565 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f427c101d30 0x7f427c102150 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f426c009900 tx=0x7f426c02bc40 comp rx=0 tx=0).stop 2024-03-28T22:37:02.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.234+0000 7f4282e00640 1 -- 172.21.15.43:0/3930028565 shutdown_connections 2024-03-28T22:37:02.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.234+0000 7f4282e00640 1 --2- 172.21.15.43:0/3930028565 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f427c1001e0 0x7f427c100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:02.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.234+0000 7f4282e00640 1 --2- 172.21.15.43:0/3930028565 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f427c0ff820 0x7f427c0ffca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:02.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.234+0000 7f4282e00640 1 --2- 172.21.15.43:0/3930028565 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f427c101d30 0x7f427c102150 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:02.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.234+0000 7f4282e00640 1 -- 172.21.15.43:0/3930028565 >> 172.21.15.43:0/3930028565 conn(0x7f427c0fb3a0 msgr2=0x7f427c0fd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:02.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.234+0000 7f4282e00640 1 -- 172.21.15.43:0/3930028565 shutdown_connections 2024-03-28T22:37:02.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.234+0000 7f4282e00640 1 -- 172.21.15.43:0/3930028565 wait complete. 2024-03-28T22:37:02.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.235+0000 7f4282e00640 1 Processor -- start 2024-03-28T22:37:02.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.235+0000 7f4282e00640 1 -- start start 2024-03-28T22:37:02.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.235+0000 7f4282e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f427c0ff820 0x7f427c199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:02.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.235+0000 7f4282e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f427c1001e0 0x7f427c19a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:02.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.235+0000 7f4282e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f427c101d30 0x7f427c1a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:02.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.235+0000 7f4282e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f427c10f610 con 0x7f427c1001e0 2024-03-28T22:37:02.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.235+0000 7f4282e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f427c10f490 con 0x7f427c0ff820 2024-03-28T22:37:02.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.235+0000 7f4282e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f427c10f790 con 0x7f427c101d30 2024-03-28T22:37:02.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.236+0000 7f427be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f427c0ff820 0x7f427c199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:02.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.236+0000 7f427be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f427c0ff820 0x7f427c199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:37694/0 (socket says 172.21.15.43:37694) 2024-03-28T22:37:02.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.236+0000 7f427be00640 1 -- 172.21.15.43:0/479061367 learned_addr learned my addr 172.21.15.43:0/479061367 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:02.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.236+0000 7f4280e00640 1 --2- 172.21.15.43:0/479061367 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f427c101d30 0x7f427c1a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:02.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.236+0000 7f427b400640 1 --2- 172.21.15.43:0/479061367 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f427c1001e0 0x7f427c19a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:02.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.236+0000 7f427be00640 1 -- 172.21.15.43:0/479061367 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f427c101d30 msgr2=0x7f427c1a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:02.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.236+0000 7f427be00640 1 --2- 172.21.15.43:0/479061367 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f427c101d30 0x7f427c1a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:02.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.236+0000 7f427be00640 1 -- 172.21.15.43:0/479061367 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f427c1001e0 msgr2=0x7f427c19a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:02.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.236+0000 7f427be00640 1 --2- 172.21.15.43:0/479061367 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f427c1001e0 0x7f427c19a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:02.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.236+0000 7f427be00640 1 -- 172.21.15.43:0/479061367 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f427c1a1810 con 0x7f427c0ff820 2024-03-28T22:37:02.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.236+0000 7f427be00640 1 --2- 172.21.15.43:0/479061367 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f427c0ff820 0x7f427c199bf0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f426c009230 tx=0x7f426c02c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:02.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.237+0000 7f4278c00640 1 -- 172.21.15.43:0/479061367 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426c041070 con 0x7f427c0ff820 2024-03-28T22:37:02.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.237+0000 7f4278c00640 1 -- 172.21.15.43:0/479061367 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f426c004450 con 0x7f427c0ff820 2024-03-28T22:37:02.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.237+0000 7f4282e00640 1 -- 172.21.15.43:0/479061367 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f427c1a1aa0 con 0x7f427c0ff820 2024-03-28T22:37:02.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.237+0000 7f4278c00640 1 -- 172.21.15.43:0/479061367 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426c003d90 con 0x7f427c0ff820 2024-03-28T22:37:02.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.237+0000 7f4282e00640 1 -- 172.21.15.43:0/479061367 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f427c1a1f90 con 0x7f427c0ff820 2024-03-28T22:37:02.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.238+0000 7f4282e00640 1 -- 172.21.15.43:0/479061367 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f427c10b750 con 0x7f427c0ff820 2024-03-28T22:37:02.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.240+0000 7f4278c00640 1 -- 172.21.15.43:0/479061367 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 20) ==== 136824+0+0 (secure 0 0 0) 0x7f426c044070 con 0x7f427c0ff820 2024-03-28T22:37:02.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.241+0000 7f4278c00640 1 --2- 172.21.15.43:0/479061367 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f42600a1f10 0x7f42600a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:02.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.241+0000 7f427b400640 1 -- 172.21.15.43:0/479061367 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f42600a1f10 msgr2=0x7f42600a43d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/369798199 2024-03-28T22:37:02.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.241+0000 7f4278c00640 1 -- 172.21.15.43:0/479061367 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f426c0efe50 con 0x7f427c0ff820 2024-03-28T22:37:02.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.241+0000 7f427b400640 1 --2- 172.21.15.43:0/479061367 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f42600a1f10 0x7f42600a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:37:02.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.245+0000 7f4278c00640 1 -- 172.21.15.43:0/479061367 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f426c0bce40 con 0x7f427c0ff820 2024-03-28T22:37:02.265 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:02 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: ignoring --setuser ceph since I am not root 2024-03-28T22:37:02.265 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:02 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: ignoring --setgroup ceph since I am not root 2024-03-28T22:37:02.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:01 smithi043 ceph-mon[31823]: pgmap v432: 129 pgs: 129 active+clean; 656 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-03-28T22:37:02.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:01 smithi043 ceph-mon[31823]: scrub queued for path: / 2024-03-28T22:37:02.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:01 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:37:02.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:01 smithi043 ceph-mon[31823]: scrub summary: active paths [/] 2024-03-28T22:37:02.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:01 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:37:02.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:01 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/271623546' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T22:37:02.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:01 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T22:37:02.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:01 smithi172 ceph-mon[36566]: pgmap v432: 129 pgs: 129 active+clean; 656 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-03-28T22:37:02.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:01 smithi172 ceph-mon[36566]: scrub queued for path: / 2024-03-28T22:37:02.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:01 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:37:02.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:01 smithi172 ceph-mon[36566]: scrub summary: active paths [/] 2024-03-28T22:37:02.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:01 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:37:02.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:01 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/271623546' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T22:37:02.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:01 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T22:37:02.397 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:37:02 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: ignoring --setuser ceph since I am not root 2024-03-28T22:37:02.397 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:37:02 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: ignoring --setgroup ceph since I am not root 2024-03-28T22:37:02.397 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:37:02 smithi172 ceph-mgr[38428]: -- 172.21.15.172:0/693220772 <== mon.1 v2:172.21.15.172:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b33c9065a0 con 0x55b33c8db000 2024-03-28T22:37:02.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.442+0000 7f427b400640 1 -- 172.21.15.43:0/479061367 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f42600a1f10 msgr2=0x7f42600a43d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/369798199 2024-03-28T22:37:02.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.442+0000 7f427b400640 1 --2- 172.21.15.43:0/479061367 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f42600a1f10 0x7f42600a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:37:02.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:01 smithi082 ceph-mon[36471]: pgmap v432: 129 pgs: 129 active+clean; 656 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-03-28T22:37:02.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:01 smithi082 ceph-mon[36471]: scrub queued for path: / 2024-03-28T22:37:02.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:01 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:37:02.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:01 smithi082 ceph-mon[36471]: scrub summary: active paths [/] 2024-03-28T22:37:02.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:01 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:37:02.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:01 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/271623546' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T22:37:02.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:01 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T22:37:02.450 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:37:02 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: ignoring --setuser ceph since I am not root 2024-03-28T22:37:02.450 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:37:02 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: ignoring --setgroup ceph since I am not root 2024-03-28T22:37:02.450 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:37:02 smithi082 ceph-mgr[37865]: -- 172.21.15.82:0/3987525460 <== mon.2 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559b06f3c5a0 con 0x559b06f10c00 2024-03-28T22:37:02.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.480+0000 7f4282e00640 1 -- 172.21.15.43:0/479061367 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f427c02d170 con 0x7f427c0ff820 2024-03-28T22:37:02.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.487+0000 7f4278c00640 1 -- 172.21.15.43:0/479061367 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v28)=0 v28) ==== 138+0+0 (secure 0 0 0) 0x7f426c0bc740 con 0x7f427c0ff820 2024-03-28T22:37:02.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.490+0000 7f4282e00640 1 -- 172.21.15.43:0/479061367 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f42600a1f10 msgr2=0x7f42600a43d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T22:37:02.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.490+0000 7f4282e00640 1 --2- 172.21.15.43:0/479061367 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f42600a1f10 0x7f42600a43d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:02.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.490+0000 7f4282e00640 1 -- 172.21.15.43:0/479061367 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f427c0ff820 msgr2=0x7f427c199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:02.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.490+0000 7f4282e00640 1 --2- 172.21.15.43:0/479061367 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f427c0ff820 0x7f427c199bf0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f426c009230 tx=0x7f426c02c660 comp rx=0 tx=0).stop 2024-03-28T22:37:02.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.490+0000 7f4282e00640 1 -- 172.21.15.43:0/479061367 shutdown_connections 2024-03-28T22:37:02.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.490+0000 7f4282e00640 1 --2- 172.21.15.43:0/479061367 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f427c101d30 0x7f427c1a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:02.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.490+0000 7f4282e00640 1 --2- 172.21.15.43:0/479061367 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f427c1001e0 0x7f427c19a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:02.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.490+0000 7f4282e00640 1 --2- 172.21.15.43:0/479061367 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f42600a1f10 0x7f42600a43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:02.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.490+0000 7f4282e00640 1 --2- 172.21.15.43:0/479061367 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f427c0ff820 0x7f427c199bf0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:02.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.490+0000 7f4282e00640 1 -- 172.21.15.43:0/479061367 >> 172.21.15.43:0/479061367 conn(0x7f427c0fb3a0 msgr2=0x7f427c0fca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:02.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.490+0000 7f4282e00640 1 -- 172.21.15.43:0/479061367 shutdown_connections 2024-03-28T22:37:02.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.490+0000 7f4282e00640 1 -- 172.21.15.43:0/479061367 wait complete. 2024-03-28T22:37:02.507 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-03-28T22:37:02.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.697+0000 7f9bfd800640 1 -- 172.21.15.43:0/698513913 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bf80741b0 msgr2=0x7f9bf8074630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:02.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.697+0000 7f9bfd800640 1 --2- 172.21.15.43:0/698513913 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bf80741b0 0x7f9bf8074630 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9be4009900 tx=0x7f9be402bc30 comp rx=0 tx=0).stop 2024-03-28T22:37:02.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.698+0000 7f9bfd800640 1 -- 172.21.15.43:0/698513913 shutdown_connections 2024-03-28T22:37:02.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.698+0000 7f9bfd800640 1 --2- 172.21.15.43:0/698513913 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9bf8109480 0x7f9bf810b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:02.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.698+0000 7f9bfd800640 1 --2- 172.21.15.43:0/698513913 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bf80741b0 0x7f9bf8074630 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:02.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.698+0000 7f9bfd800640 1 --2- 172.21.15.43:0/698513913 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9bf8075790 0x7f9bf8073be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:02.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.698+0000 7f9bfd800640 1 -- 172.21.15.43:0/698513913 >> 172.21.15.43:0/698513913 conn(0x7f9bf80fcb30 msgr2=0x7f9bf80fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:02.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.698+0000 7f9bfd800640 1 -- 172.21.15.43:0/698513913 shutdown_connections 2024-03-28T22:37:02.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.698+0000 7f9bfd800640 1 -- 172.21.15.43:0/698513913 wait complete. 2024-03-28T22:37:02.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.699+0000 7f9bfd800640 1 Processor -- start 2024-03-28T22:37:02.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.699+0000 7f9bfd800640 1 -- start start 2024-03-28T22:37:02.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.700+0000 7f9bfd800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9bf80741b0 0x7f9bf8199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:02.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.700+0000 7f9bfd800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9bf8075790 0x7f9bf8199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:02.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.700+0000 7f9bfd800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bf8109480 0x7f9bf81a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:02.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.700+0000 7f9bfd800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf810f470 con 0x7f9bf8109480 2024-03-28T22:37:02.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.700+0000 7f9bfd800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf810f2f0 con 0x7f9bf8075790 2024-03-28T22:37:02.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.700+0000 7f9bfd800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf810f5f0 con 0x7f9bf80741b0 2024-03-28T22:37:02.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.700+0000 7f9bf7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9bf80741b0 0x7f9bf8199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:02.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.700+0000 7f9bf7400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9bf8075790 0x7f9bf8199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:02.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.700+0000 7f9bf7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9bf80741b0 0x7f9bf8199a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:47064/0 (socket says 172.21.15.43:47064) 2024-03-28T22:37:02.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.700+0000 7f9bf7e00640 1 -- 172.21.15.43:0/1806369 learned_addr learned my addr 172.21.15.43:0/1806369 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:02.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.700+0000 7f9bfce00640 1 --2- 172.21.15.43:0/1806369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bf8109480 0x7f9bf81a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:02.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.701+0000 7f9bf7e00640 1 -- 172.21.15.43:0/1806369 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9bf8075790 msgr2=0x7f9bf8199f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:02.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.701+0000 7f9bf7e00640 1 --2- 172.21.15.43:0/1806369 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9bf8075790 0x7f9bf8199f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:02.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.701+0000 7f9bf7e00640 1 -- 172.21.15.43:0/1806369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bf8109480 msgr2=0x7f9bf81a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:02.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.701+0000 7f9bf7e00640 1 --2- 172.21.15.43:0/1806369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bf8109480 0x7f9bf81a1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:02.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.701+0000 7f9bf7e00640 1 -- 172.21.15.43:0/1806369 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bf81a1670 con 0x7f9bf80741b0 2024-03-28T22:37:02.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.701+0000 7f9bf7400640 1 --2- 172.21.15.43:0/1806369 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9bf8075790 0x7f9bf8199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:37:02.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.701+0000 7f9bf7e00640 1 --2- 172.21.15.43:0/1806369 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9bf80741b0 0x7f9bf8199a50 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9be800ea60 tx=0x7f9be800ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:02.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.701+0000 7f9bf4c00640 1 -- 172.21.15.43:0/1806369 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be800cde0 con 0x7f9bf80741b0 2024-03-28T22:37:02.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.701+0000 7f9bfd800640 1 -- 172.21.15.43:0/1806369 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bf81a1960 con 0x7f9bf80741b0 2024-03-28T22:37:02.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.701+0000 7f9bf4c00640 1 -- 172.21.15.43:0/1806369 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9be80035f0 con 0x7f9bf80741b0 2024-03-28T22:37:02.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.701+0000 7f9bf4c00640 1 -- 172.21.15.43:0/1806369 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be8010670 con 0x7f9bf80741b0 2024-03-28T22:37:02.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.701+0000 7f9bfd800640 1 -- 172.21.15.43:0/1806369 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bf81a1ed0 con 0x7f9bf80741b0 2024-03-28T22:37:02.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.703+0000 7f9bfd800640 1 -- 172.21.15.43:0/1806369 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bf81081b0 con 0x7f9bf80741b0 2024-03-28T22:37:02.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.704+0000 7f9bf4c00640 1 -- 172.21.15.43:0/1806369 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 20) ==== 136824+0+0 (secure 0 0 0) 0x7f9be80037a0 con 0x7f9bf80741b0 2024-03-28T22:37:02.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.706+0000 7f9bf4c00640 1 --2- 172.21.15.43:0/1806369 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9bdc0a1ed0 0x7f9bdc0a4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:02.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.706+0000 7f9bf7400640 1 -- 172.21.15.43:0/1806369 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9bdc0a1ed0 msgr2=0x7f9bdc0a4390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/369798199 2024-03-28T22:37:02.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.706+0000 7f9bf4c00640 1 -- 172.21.15.43:0/1806369 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f9be80ce600 con 0x7f9bf80741b0 2024-03-28T22:37:02.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.706+0000 7f9bf7400640 1 --2- 172.21.15.43:0/1806369 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9bdc0a1ed0 0x7f9bdc0a4390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:37:02.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.714+0000 7f9bf4c00640 1 -- 172.21.15.43:0/1806369 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9be809b4e0 con 0x7f9bf80741b0 2024-03-28T22:37:02.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.907+0000 7f9bf7400640 1 -- 172.21.15.43:0/1806369 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9bdc0a1ed0 msgr2=0x7f9bdc0a4390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/369798199 2024-03-28T22:37:02.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.907+0000 7f9bf7400640 1 --2- 172.21.15.43:0/1806369 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9bdc0a1ed0 0x7f9bdc0a4390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:37:02.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.915+0000 7f9bfd800640 1 -- 172.21.15.43:0/1806369 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f9bf802d170 con 0x7f9bf80741b0 2024-03-28T22:37:02.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.925+0000 7f9bf4c00640 1 -- 172.21.15.43:0/1806369 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v29)=0 v29) ==== 133+0+0 (secure 0 0 0) 0x7f9be809ade0 con 0x7f9bf80741b0 2024-03-28T22:37:02.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.928+0000 7f9bfd800640 1 -- 172.21.15.43:0/1806369 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9bdc0a1ed0 msgr2=0x7f9bdc0a4390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T22:37:02.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.928+0000 7f9bfd800640 1 --2- 172.21.15.43:0/1806369 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9bdc0a1ed0 0x7f9bdc0a4390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:02.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.928+0000 7f9bfd800640 1 -- 172.21.15.43:0/1806369 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9bf80741b0 msgr2=0x7f9bf8199a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:02.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.928+0000 7f9bfd800640 1 --2- 172.21.15.43:0/1806369 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9bf80741b0 0x7f9bf8199a50 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9be800ea60 tx=0x7f9be800ef20 comp rx=0 tx=0).stop 2024-03-28T22:37:02.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.928+0000 7f9bfd800640 1 -- 172.21.15.43:0/1806369 shutdown_connections 2024-03-28T22:37:02.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.928+0000 7f9bfd800640 1 --2- 172.21.15.43:0/1806369 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f9bdc0a1ed0 0x7f9bdc0a4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:02.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.928+0000 7f9bfd800640 1 --2- 172.21.15.43:0/1806369 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9bf8075790 0x7f9bf8199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:02.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.928+0000 7f9bfd800640 1 --2- 172.21.15.43:0/1806369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9bf8109480 0x7f9bf81a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:02.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.928+0000 7f9bfd800640 1 --2- 172.21.15.43:0/1806369 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9bf80741b0 0x7f9bf8199a50 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:02.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.928+0000 7f9bfd800640 1 -- 172.21.15.43:0/1806369 >> 172.21.15.43:0/1806369 conn(0x7f9bf80fcb30 msgr2=0x7f9bf8105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:02.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.928+0000 7f9bfd800640 1 -- 172.21.15.43:0/1806369 shutdown_connections 2024-03-28T22:37:02.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:02.928+0000 7f9bfd800640 1 -- 172.21.15.43:0/1806369 wait complete. 2024-03-28T22:37:02.945 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-03-28T22:37:02.949 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:37:02 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:37:02.658+0000 7fbf03246200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T22:37:02.949 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:37:02 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:37:02.754+0000 7fbf03246200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T22:37:03.001 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:37:02 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:37:02.655+0000 7f8ace4d5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T22:37:03.001 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:37:02 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:37:02.771+0000 7f8ace4d5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T22:37:03.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.153+0000 7f1930600640 1 -- 172.21.15.43:0/1228102712 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1928077350 msgr2=0x7f19280777b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:03.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.153+0000 7f1930600640 1 --2- 172.21.15.43:0/1228102712 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1928077350 0x7f19280777b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1914009900 tx=0x7f191402bc40 comp rx=0 tx=0).stop 2024-03-28T22:37:03.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.153+0000 7f1930600640 1 -- 172.21.15.43:0/1228102712 shutdown_connections 2024-03-28T22:37:03.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.153+0000 7f1930600640 1 --2- 172.21.15.43:0/1228102712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1928077cf0 0x7f1928078590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:03.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.153+0000 7f1930600640 1 --2- 172.21.15.43:0/1228102712 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1928077350 0x7f19280777b0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:03.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.153+0000 7f1930600640 1 --2- 172.21.15.43:0/1228102712 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1928076110 0x7f1928076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:03.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.153+0000 7f1930600640 1 -- 172.21.15.43:0/1228102712 >> 172.21.15.43:0/1228102712 conn(0x7f19280fef40 msgr2=0x7f1928101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:03.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.153+0000 7f1930600640 1 -- 172.21.15.43:0/1228102712 shutdown_connections 2024-03-28T22:37:03.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.153+0000 7f1930600640 1 -- 172.21.15.43:0/1228102712 wait complete. 2024-03-28T22:37:03.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.154+0000 7f1930600640 1 Processor -- start 2024-03-28T22:37:03.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.154+0000 7f1930600640 1 -- start start 2024-03-28T22:37:03.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.155+0000 7f1930600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1928076110 0x7f192819dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:03.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.155+0000 7f1930600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1928077350 0x7f192819e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:03.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.155+0000 7f1930600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1928077cf0 0x7f19281a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:03.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.155+0000 7f1930600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1928113a80 con 0x7f1928077350 2024-03-28T22:37:03.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.155+0000 7f1930600640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1928113900 con 0x7f1928077cf0 2024-03-28T22:37:03.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.155+0000 7f1930600640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f1928113c00 con 0x7f1928076110 2024-03-28T22:37:03.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.155+0000 7f192dc00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1928076110 0x7f192819dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:03.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.155+0000 7f192dc00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1928076110 0x7f192819dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:47076/0 (socket says 172.21.15.43:47076) 2024-03-28T22:37:03.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.155+0000 7f192e600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1928077cf0 0x7f19281a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:03.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.155+0000 7f192dc00640 1 -- 172.21.15.43:0/3888834314 learned_addr learned my addr 172.21.15.43:0/3888834314 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:03.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.155+0000 7f192d200640 1 --2- 172.21.15.43:0/3888834314 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1928077350 0x7f192819e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:03.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.155+0000 7f192dc00640 1 -- 172.21.15.43:0/3888834314 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1928077cf0 msgr2=0x7f19281a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:03.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.155+0000 7f192dc00640 1 --2- 172.21.15.43:0/3888834314 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1928077cf0 0x7f19281a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:03.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.155+0000 7f192dc00640 1 -- 172.21.15.43:0/3888834314 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1928077350 msgr2=0x7f192819e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:03.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.155+0000 7f192dc00640 1 --2- 172.21.15.43:0/3888834314 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1928077350 0x7f192819e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:03.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.155+0000 7f192dc00640 1 -- 172.21.15.43:0/3888834314 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19281a5bd0 con 0x7f1928076110 2024-03-28T22:37:03.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.156+0000 7f192e600640 1 --2- 172.21.15.43:0/3888834314 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1928077cf0 0x7f19281a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:37:03.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.156+0000 7f192dc00640 1 --2- 172.21.15.43:0/3888834314 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1928076110 0x7f192819dfb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f191c00e9f0 tx=0x7f191c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:03.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.156+0000 7f1926000640 1 -- 172.21.15.43:0/3888834314 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191c00cd80 con 0x7f1928076110 2024-03-28T22:37:03.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.156+0000 7f1926000640 1 -- 172.21.15.43:0/3888834314 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f191c005500 con 0x7f1928076110 2024-03-28T22:37:03.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.156+0000 7f1930600640 1 -- 172.21.15.43:0/3888834314 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19281a5ec0 con 0x7f1928076110 2024-03-28T22:37:03.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.156+0000 7f1926000640 1 -- 172.21.15.43:0/3888834314 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191c010690 con 0x7f1928076110 2024-03-28T22:37:03.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.156+0000 7f1930600640 1 -- 172.21.15.43:0/3888834314 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19281a6320 con 0x7f1928076110 2024-03-28T22:37:03.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.158+0000 7f1930600640 1 -- 172.21.15.43:0/3888834314 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1928110320 con 0x7f1928076110 2024-03-28T22:37:03.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.159+0000 7f1926000640 1 -- 172.21.15.43:0/3888834314 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 20) ==== 136824+0+0 (secure 0 0 0) 0x7f191c0046c0 con 0x7f1928076110 2024-03-28T22:37:03.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.160+0000 7f1926000640 1 --2- 172.21.15.43:0/3888834314 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f19100a1f90 0x7f19100a4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:03.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.161+0000 7f192d200640 1 -- 172.21.15.43:0/3888834314 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f19100a1f90 msgr2=0x7f19100a4450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/369798199 2024-03-28T22:37:03.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.161+0000 7f1926000640 1 -- 172.21.15.43:0/3888834314 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f191c0ce6f0 con 0x7f1928076110 2024-03-28T22:37:03.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.161+0000 7f192d200640 1 --2- 172.21.15.43:0/3888834314 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f19100a1f90 0x7f19100a4450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:37:03.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.168+0000 7f1926000640 1 -- 172.21.15.43:0/3888834314 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f191c09b5d0 con 0x7f1928076110 2024-03-28T22:37:03.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:03 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-28T22:37:03.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:03 smithi043 ceph-mon[31823]: mgrmap e20: x(active, since 11m), standbys: z, y 2024-03-28T22:37:03.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:03 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' 2024-03-28T22:37:03.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:03 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' 2024-03-28T22:37:03.266 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:02 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:02.889+0000 7f362cfd2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T22:37:03.267 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:03 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:03.028+0000 7f362cfd2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T22:37:03.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.362+0000 7f192d200640 1 -- 172.21.15.43:0/3888834314 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f19100a1f90 msgr2=0x7f19100a4450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/369798199 2024-03-28T22:37:03.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.362+0000 7f192d200640 1 --2- 172.21.15.43:0/3888834314 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f19100a1f90 0x7f19100a4450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:37:03.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.389+0000 7f1930600640 1 -- 172.21.15.43:0/3888834314 --> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f1928061b90 con 0x7f19100a1f90 2024-03-28T22:37:03.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:03 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-28T22:37:03.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:03 smithi172 ceph-mon[36566]: mgrmap e20: x(active, since 11m), standbys: z, y 2024-03-28T22:37:03.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:03 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' 2024-03-28T22:37:03.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:03 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' 2024-03-28T22:37:03.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:03 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-28T22:37:03.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:03 smithi082 ceph-mon[36471]: mgrmap e20: x(active, since 11m), standbys: z, y 2024-03-28T22:37:03.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:03 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' 2024-03-28T22:37:03.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:03 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' 2024-03-28T22:37:03.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.763+0000 7f192d200640 1 -- 172.21.15.43:0/3888834314 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f19100a1f90 msgr2=0x7f19100a4450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/369798199 2024-03-28T22:37:03.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:03.763+0000 7f192d200640 1 --2- 172.21.15.43:0/3888834314 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f19100a1f90 0x7f19100a4450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:37:03.812 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:37:03 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:37:03.479+0000 7fbf03246200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T22:37:03.812 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:37:03 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:37:03.573+0000 7fbf03246200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T22:37:03.812 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:37:03 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:37:03.716+0000 7fbf03246200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T22:37:03.812 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:37:03 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:37:03.813+0000 7fbf03246200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T22:37:03.831 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:37:03 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:37:03.543+0000 7f8ace4d5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T22:37:03.831 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:37:03 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:37:03.681+0000 7f8ace4d5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T22:37:04.146 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:37:03 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:37:03.833+0000 7f8ace4d5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T22:37:04.146 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:37:03 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:37:03.977+0000 7f8ace4d5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T22:37:04.146 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:37:04 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:37:04.081+0000 7f8ace4d5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T22:37:04.181 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:03 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:03.866+0000 7f362cfd2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T22:37:04.181 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:03 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:03.992+0000 7f362cfd2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T22:37:04.181 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:04 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:04.181+0000 7f362cfd2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T22:37:04.182 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:37:03 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:37:03.898+0000 7fbf03246200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T22:37:04.449 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:37:04 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:37:04.184+0000 7fbf03246200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T22:37:04.449 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:37:04 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:37:04.296+0000 7fbf03246200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T22:37:04.449 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:37:04 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:37:04.383+0000 7fbf03246200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T22:37:04.516 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:04 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:04.313+0000 7f362cfd2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T22:37:04.516 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:04 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:04.427+0000 7f362cfd2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T22:37:04.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:04.564+0000 7f192d200640 1 -- 172.21.15.43:0/3888834314 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f19100a1f90 msgr2=0x7f19100a4450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/369798199 2024-03-28T22:37:04.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:04.564+0000 7f192d200640 1 --2- 172.21.15.43:0/3888834314 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f19100a1f90 0x7f19100a4450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:37:04.703 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:37:04 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:37:04.452+0000 7f8ace4d5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T22:37:04.703 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:37:04 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:37:04.583+0000 7f8ace4d5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T22:37:04.949 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:37:04 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:37:04.489+0000 7fbf03246200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T22:37:04.949 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:37:04 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:37:04.667+0000 7fbf03246200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T22:37:04.989 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:37:04 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:37:04.704+0000 7f8ace4d5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T22:37:04.989 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:37:04 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:37:04.802+0000 7f8ace4d5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T22:37:05.045 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:04 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:04.775+0000 7f362cfd2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T22:37:05.045 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:04 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:04.922+0000 7f362cfd2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T22:37:05.045 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:05 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:05.045+0000 7f362cfd2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T22:37:05.396 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:37:04 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:37:04.991+0000 7f8ace4d5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T22:37:05.398 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:05 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:05.174+0000 7f362cfd2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T22:37:05.398 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:05 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:05.399+0000 7f362cfd2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T22:37:06.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:06.166+0000 7f192d200640 1 -- 172.21.15.43:0/3888834314 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f19100a1f90 msgr2=0x7f19100a4450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/369798199 2024-03-28T22:37:06.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:06.166+0000 7f192d200640 1 --2- 172.21.15.43:0/3888834314 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f19100a1f90 0x7f19100a4450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-28T22:37:06.199 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:37:05 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:37:05.762+0000 7fbf03246200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T22:37:06.199 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:37:05 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:37:05.925+0000 7fbf03246200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T22:37:06.589 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:37:06 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:37:06.296+0000 7fbf03246200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T22:37:06.589 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:37:06 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:37:06.386+0000 7fbf03246200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T22:37:06.589 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:37:06 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:37:06.590+0000 7fbf03246200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T22:37:06.646 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:37:06 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:37:06.254+0000 7f8ace4d5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T22:37:06.646 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:37:06 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:37:06.411+0000 7f8ace4d5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T22:37:06.949 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:37:06 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:37:06.778+0000 7fbf03246200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T22:37:07.016 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:06 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:06.729+0000 7f362cfd2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T22:37:07.016 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:06 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:06.908+0000 7f362cfd2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T22:37:07.111 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:37:06 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:37:06.813+0000 7f8ace4d5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T22:37:07.112 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:37:06 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:37:06.905+0000 7f8ace4d5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T22:37:07.396 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:37:07 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:37:07.113+0000 7f8ace4d5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T22:37:07.396 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:37:07 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:37:07.314+0000 7f8ace4d5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T22:37:07.408 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:37:07 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:37:07.098+0000 7fbf03246200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T22:37:07.408 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:37:07 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:37:07.409+0000 7fbf03246200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T22:37:07.699 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:37:07 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:37:07.648+0000 7fbf03246200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T22:37:07.736 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:07 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:07.342+0000 7f362cfd2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T22:37:07.736 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:07 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:07.466+0000 7f362cfd2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T22:37:07.884 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:37:07 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:37:07.607+0000 7f8ace4d5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T22:37:07.995 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:07 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:07.736+0000 7f362cfd2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T22:37:07.995 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:07 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:07.995+0000 7f362cfd2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T22:37:08.146 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:37:07 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:37:07.886+0000 7f8ace4d5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T22:37:08.146 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:37:08 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:37:08.135+0000 7f8ace4d5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T22:37:08.716 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:08 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:08.388+0000 7f362cfd2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T22:37:08.716 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:08 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:08.716+0000 7f362cfd2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T22:37:08.949 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:37:08 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:37:08.474+0000 7fbf03246200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T22:37:09.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:09.370+0000 7f192d200640 1 -- 172.21.15.43:0/3888834314 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f19100a1f90 msgr2=0x7f19100a4450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:6800/369798199 2024-03-28T22:37:09.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:09.370+0000 7f192d200640 1 --2- 172.21.15.43:0/3888834314 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f19100a1f90 0x7f19100a4450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-28T22:37:09.398 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:37:09 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:37:09.018+0000 7f8ace4d5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T22:37:09.449 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:37:09 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:37:09.182+0000 7fbf03246200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T22:37:09.449 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:37:09 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:37:09.281+0000 7fbf03246200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T22:37:09.515 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:09 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:09.035+0000 7f362cfd2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T22:37:09.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:09.597+0000 7f1926000640 1 -- 172.21.15.43:0/3888834314 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mgrmap(e 21) ==== 136824+0+0 (secure 0 0 0) 0x7f191c097540 con 0x7f1928076110 2024-03-28T22:37:09.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:09 smithi172 ceph-mon[36566]: Standby manager daemon y restarted 2024-03-28T22:37:09.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:09 smithi172 ceph-mon[36566]: from='mgr.? 172.21.15.82:0/3072117419' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T22:37:09.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:09 smithi172 ceph-mon[36566]: Standby manager daemon y started 2024-03-28T22:37:09.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:09 smithi172 ceph-mon[36566]: from='mgr.? 172.21.15.82:0/3072117419' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T22:37:09.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:09 smithi172 ceph-mon[36566]: from='mgr.? 172.21.15.82:0/3072117419' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T22:37:09.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:09 smithi172 ceph-mon[36566]: from='mgr.? 172.21.15.82:0/3072117419' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T22:37:09.896 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:37:09 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:37:09.652+0000 7f8ace4d5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T22:37:09.897 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:37:09 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:37:09.783+0000 7f8ace4d5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T22:37:09.949 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:37:09 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y[37861]: 2024-03-28T22:37:09.508+0000 7fbf03246200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T22:37:09.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:09 smithi082 ceph-mon[36471]: Standby manager daemon y restarted 2024-03-28T22:37:09.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:09 smithi082 ceph-mon[36471]: from='mgr.? 172.21.15.82:0/3072117419' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T22:37:09.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:09 smithi082 ceph-mon[36471]: Standby manager daemon y started 2024-03-28T22:37:09.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:09 smithi082 ceph-mon[36471]: from='mgr.? 172.21.15.82:0/3072117419' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T22:37:09.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:09 smithi082 ceph-mon[36471]: from='mgr.? 172.21.15.82:0/3072117419' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T22:37:09.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:09 smithi082 ceph-mon[36471]: from='mgr.? 172.21.15.82:0/3072117419' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T22:37:10.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:09 smithi043 ceph-mon[31823]: Standby manager daemon y restarted 2024-03-28T22:37:10.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:09 smithi043 ceph-mon[31823]: from='mgr.? 172.21.15.82:0/3072117419' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T22:37:10.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:09 smithi043 ceph-mon[31823]: Standby manager daemon y started 2024-03-28T22:37:10.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:09 smithi043 ceph-mon[31823]: from='mgr.? 172.21.15.82:0/3072117419' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T22:37:10.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:09 smithi043 ceph-mon[31823]: from='mgr.? 172.21.15.82:0/3072117419' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T22:37:10.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:09 smithi043 ceph-mon[31823]: from='mgr.? 172.21.15.82:0/3072117419' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T22:37:10.396 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:37:10 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z[38424]: 2024-03-28T22:37:10.105+0000 7f8ace4d5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T22:37:10.516 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:10 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:10.131+0000 7f362cfd2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T22:37:10.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:10.627+0000 7f1926000640 1 -- 172.21.15.43:0/3888834314 <== mon.1 v2:172.21.15.172:3300/0 8 ==== mgrmap(e 22) ==== 136824+0+0 (secure 0 0 0) 0x7f191c0cbf00 con 0x7f1928076110 2024-03-28T22:37:10.866 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:10 smithi043 ceph-mon[31823]: mgrmap e21: x(active, since 11m), standbys: z, y 2024-03-28T22:37:10.866 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:10 smithi043 ceph-mon[31823]: Standby manager daemon z restarted 2024-03-28T22:37:10.867 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:10 smithi043 ceph-mon[31823]: Standby manager daemon z started 2024-03-28T22:37:10.867 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:10 smithi043 ceph-mon[31823]: from='mgr.? 172.21.15.172:0/335930670' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T22:37:10.867 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:10 smithi043 ceph-mon[31823]: from='mgr.? 172.21.15.172:0/335930670' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T22:37:10.867 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:10 smithi043 ceph-mon[31823]: from='mgr.? 172.21.15.172:0/335930670' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T22:37:10.867 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:10 smithi043 ceph-mon[31823]: from='mgr.? 172.21.15.172:0/335930670' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T22:37:10.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:10 smithi172 ceph-mon[36566]: mgrmap e21: x(active, since 11m), standbys: z, y 2024-03-28T22:37:10.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:10 smithi172 ceph-mon[36566]: Standby manager daemon z restarted 2024-03-28T22:37:10.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:10 smithi172 ceph-mon[36566]: Standby manager daemon z started 2024-03-28T22:37:10.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:10 smithi172 ceph-mon[36566]: from='mgr.? 172.21.15.172:0/335930670' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T22:37:10.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:10 smithi172 ceph-mon[36566]: from='mgr.? 172.21.15.172:0/335930670' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T22:37:10.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:10 smithi172 ceph-mon[36566]: from='mgr.? 172.21.15.172:0/335930670' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T22:37:10.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:10 smithi172 ceph-mon[36566]: from='mgr.? 172.21.15.172:0/335930670' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T22:37:10.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:10 smithi082 ceph-mon[36471]: mgrmap e21: x(active, since 11m), standbys: z, y 2024-03-28T22:37:10.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:10 smithi082 ceph-mon[36471]: Standby manager daemon z restarted 2024-03-28T22:37:10.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:10 smithi082 ceph-mon[36471]: Standby manager daemon z started 2024-03-28T22:37:10.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:10 smithi082 ceph-mon[36471]: from='mgr.? 172.21.15.172:0/335930670' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T22:37:10.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:10 smithi082 ceph-mon[36471]: from='mgr.? 172.21.15.172:0/335930670' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T22:37:10.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:10 smithi082 ceph-mon[36471]: from='mgr.? 172.21.15.172:0/335930670' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T22:37:10.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:10 smithi082 ceph-mon[36471]: from='mgr.? 172.21.15.172:0/335930670' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T22:37:11.266 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:10 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:10.867+0000 7f362cfd2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T22:37:11.266 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:11.005+0000 7f362cfd2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T22:37:11.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:11.329+0000 7f1926000640 1 -- 172.21.15.43:0/3888834314 <== mon.1 v2:172.21.15.172:3300/0 9 ==== mgrmap(e 23) ==== 136441+0+0 (secure 0 0 0) 0x7f191c098be0 con 0x7f1928076110 2024-03-28T22:37:11.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:11.332+0000 7f1926000640 1 -- 172.21.15.43:0/3888834314 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f19100a1f90 msgr2=0x7f19100a4450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T22:37:11.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:11.332+0000 7f1926000640 1 --2- 172.21.15.43:0/3888834314 >> [v2:172.21.15.43:6800/369798199,v1:172.21.15.43:6801/369798199] conn(0x7f19100a1f90 0x7f19100a4450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:11.617 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:11.303+0000 7f362cfd2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T22:37:11.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: mgrmap e22: x(active, since 11m), standbys: y, z 2024-03-28T22:37:11.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: Active manager daemon x restarted 2024-03-28T22:37:11.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: Activating manager daemon x 2024-03-28T22:37:11.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: osdmap e97: 12 total, 12 up, 12 in 2024-03-28T22:37:11.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: mgrmap e23: x(active, starting, since 0.0251415s), standbys: y, z 2024-03-28T22:37:11.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:37:11.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:37:11.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:37:11.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T22:37:11.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T22:37:11.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T22:37:11.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T22:37:11.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T22:37:11.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T22:37:11.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T22:37:11.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T22:37:11.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T22:37:11.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T22:37:11.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T22:37:11.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T22:37:11.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T22:37:11.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T22:37:11.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T22:37:11.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:37:11.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:37:11.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:37:11.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:37:11.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:37:11.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:37:11.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:37:11.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:37:11.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:37:11.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:37:11.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:37:11.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:37:11.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T22:37:11.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T22:37:11.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T22:37:11.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: Manager daemon x is now available 2024-03-28T22:37:11.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:37:11.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:37:11.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T22:37:11.901 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:11 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T22:37:11.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: mgrmap e22: x(active, since 11m), standbys: y, z 2024-03-28T22:37:11.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: Active manager daemon x restarted 2024-03-28T22:37:11.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: Activating manager daemon x 2024-03-28T22:37:11.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: osdmap e97: 12 total, 12 up, 12 in 2024-03-28T22:37:11.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: mgrmap e23: x(active, starting, since 0.0251415s), standbys: y, z 2024-03-28T22:37:11.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:37:11.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:37:11.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:37:11.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T22:37:11.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T22:37:11.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T22:37:11.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T22:37:11.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T22:37:11.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T22:37:11.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T22:37:11.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T22:37:11.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T22:37:11.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T22:37:11.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T22:37:11.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T22:37:11.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T22:37:11.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T22:37:11.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T22:37:11.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:37:11.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:37:11.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:37:11.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:37:11.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:37:11.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:37:11.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:37:11.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:37:11.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:37:11.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:37:11.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:37:11.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:37:11.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T22:37:11.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T22:37:11.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T22:37:11.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: Manager daemon x is now available 2024-03-28T22:37:11.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:37:11.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:37:11.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T22:37:11.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:11 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T22:37:12.016 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:11.645+0000 7f3583400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:37:12.016 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:11.645+0000 7f3583400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:37:12.016 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:11.645+0000 7f3583400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:37:12.016 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:11.645+0000 7f3583400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:37:12.017 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:11.645+0000 7f3583400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:37:12.017 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:11.646+0000 7f357e400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:37:12.017 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:11.646+0000 7f357e400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:37:12.017 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:11.646+0000 7f357e400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:37:12.017 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:11.646+0000 7f357e400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:37:12.017 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:37:11.646+0000 7f357e400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:37:12.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: mgrmap e22: x(active, since 11m), standbys: y, z 2024-03-28T22:37:12.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: Active manager daemon x restarted 2024-03-28T22:37:12.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: Activating manager daemon x 2024-03-28T22:37:12.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: osdmap e97: 12 total, 12 up, 12 in 2024-03-28T22:37:12.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: mgrmap e23: x(active, starting, since 0.0251415s), standbys: y, z 2024-03-28T22:37:12.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T22:37:12.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T22:37:12.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T22:37:12.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T22:37:12.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T22:37:12.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T22:37:12.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T22:37:12.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T22:37:12.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T22:37:12.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T22:37:12.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T22:37:12.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T22:37:12.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T22:37:12.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T22:37:12.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T22:37:12.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T22:37:12.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T22:37:12.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T22:37:12.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T22:37:12.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T22:37:12.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T22:37:12.020 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T22:37:12.020 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T22:37:12.020 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T22:37:12.020 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T22:37:12.020 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T22:37:12.020 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T22:37:12.020 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T22:37:12.020 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T22:37:12.020 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T22:37:12.020 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T22:37:12.020 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T22:37:12.020 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T22:37:12.020 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: Manager daemon x is now available 2024-03-28T22:37:12.020 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:37:12.020 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:37:12.021 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T22:37:12.021 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:11 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T22:37:12.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.337+0000 7f1926000640 1 -- 172.21.15.43:0/3888834314 <== mon.1 v2:172.21.15.172:3300/0 10 ==== mgrmap(e 24) ==== 136568+0+0 (secure 0 0 0) 0x7f191c0429d0 con 0x7f1928076110 2024-03-28T22:37:12.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.338+0000 7f1926000640 1 --2- 172.21.15.43:0/3888834314 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f191004db10 0x7f191004def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:12.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.338+0000 7f1926000640 1 -- 172.21.15.43:0/3888834314 --> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f191c063720 con 0x7f191004db10 2024-03-28T22:37:12.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.349+0000 7f192d200640 1 --2- 172.21.15.43:0/3888834314 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f191004db10 0x7f191004def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:12.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.353+0000 7f192d200640 1 --2- 172.21.15.43:0/3888834314 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f191004db10 0x7f191004def0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f191402c250 tx=0x7f191402cf90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:12.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.450+0000 7f1926000640 1 -- 172.21.15.43:0/3888834314 <== mgr.25663 v2:172.21.15.43:6800/1685312925 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f191c063720 con 0x7f191004db10 2024-03-28T22:37:12.448 INFO:teuthology.orchestra.run.smithi043.stdout:Schedule set for path / 2024-03-28T22:37:12.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.452+0000 7f1930600640 1 -- 172.21.15.43:0/3888834314 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f191004db10 msgr2=0x7f191004def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:12.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.452+0000 7f1930600640 1 --2- 172.21.15.43:0/3888834314 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f191004db10 0x7f191004def0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f191402c250 tx=0x7f191402cf90 comp rx=0 tx=0).stop 2024-03-28T22:37:12.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.452+0000 7f1930600640 1 -- 172.21.15.43:0/3888834314 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1928076110 msgr2=0x7f192819dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:12.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.452+0000 7f1930600640 1 --2- 172.21.15.43:0/3888834314 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1928076110 0x7f192819dfb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f191c00e9f0 tx=0x7f191c00edf0 comp rx=0 tx=0).stop 2024-03-28T22:37:12.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.452+0000 7f192e600640 1 -- 172.21.15.43:0/3888834314 reap_dead start 2024-03-28T22:37:12.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.452+0000 7f1930600640 1 -- 172.21.15.43:0/3888834314 shutdown_connections 2024-03-28T22:37:12.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.452+0000 7f1930600640 1 -- 172.21.15.43:0/3888834314 >> 172.21.15.43:0/3888834314 conn(0x7f19280fef40 msgr2=0x7f1928100a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:12.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.453+0000 7f1930600640 1 -- 172.21.15.43:0/3888834314 shutdown_connections 2024-03-28T22:37:12.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.453+0000 7f1930600640 1 -- 172.21.15.43:0/3888834314 wait complete. 2024-03-28T22:37:12.470 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-03-28T22:37:12.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.656+0000 7f6c3ea00640 1 -- 172.21.15.43:0/3038746286 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6c38076110 msgr2=0x7f6c38076530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:12.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.656+0000 7f6c3ea00640 1 --2- 172.21.15.43:0/3038746286 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6c38076110 0x7f6c38076530 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6c28009860 tx=0x7f6c2802bb70 comp rx=0 tx=0).stop 2024-03-28T22:37:12.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.656+0000 7f6c3ea00640 1 -- 172.21.15.43:0/3038746286 shutdown_connections 2024-03-28T22:37:12.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.656+0000 7f6c3ea00640 1 --2- 172.21.15.43:0/3038746286 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c38077cf0 0x7f6c38078590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:12.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.656+0000 7f6c3ea00640 1 --2- 172.21.15.43:0/3038746286 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c38077350 0x7f6c380777b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:12.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.656+0000 7f6c3ea00640 1 --2- 172.21.15.43:0/3038746286 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6c38076110 0x7f6c38076530 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:12.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.656+0000 7f6c3ea00640 1 -- 172.21.15.43:0/3038746286 >> 172.21.15.43:0/3038746286 conn(0x7f6c380fef40 msgr2=0x7f6c38101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:12.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.656+0000 7f6c3ea00640 1 -- 172.21.15.43:0/3038746286 shutdown_connections 2024-03-28T22:37:12.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.656+0000 7f6c3ea00640 1 -- 172.21.15.43:0/3038746286 wait complete. 2024-03-28T22:37:12.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.657+0000 7f6c3ea00640 1 Processor -- start 2024-03-28T22:37:12.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.657+0000 7f6c3ea00640 1 -- start start 2024-03-28T22:37:12.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.657+0000 7f6c3ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6c38076110 0x7f6c3819dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:12.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.657+0000 7f6c3ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c38077350 0x7f6c3819e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:12.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.657+0000 7f6c3ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c38077cf0 0x7f6c381a55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:12.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.657+0000 7f6c3ea00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6c38113a40 con 0x7f6c38077cf0 2024-03-28T22:37:12.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.657+0000 7f6c3ea00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f6c381138c0 con 0x7f6c38076110 2024-03-28T22:37:12.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.657+0000 7f6c3ea00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f6c38113bc0 con 0x7f6c38077350 2024-03-28T22:37:12.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.657+0000 7f6c37e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6c38076110 0x7f6c3819dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:12.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.657+0000 7f6c37e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6c38076110 0x7f6c3819dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:35192/0 (socket says 172.21.15.43:35192) 2024-03-28T22:37:12.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.658+0000 7f6c37e00640 1 -- 172.21.15.43:0/4130719334 learned_addr learned my addr 172.21.15.43:0/4130719334 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:12.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.658+0000 7f6c3ca00640 1 --2- 172.21.15.43:0/4130719334 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c38077cf0 0x7f6c381a55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:12.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.658+0000 7f6c37400640 1 --2- 172.21.15.43:0/4130719334 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c38077350 0x7f6c3819e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:12.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.658+0000 7f6c37e00640 1 -- 172.21.15.43:0/4130719334 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c38077350 msgr2=0x7f6c3819e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:12.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.658+0000 7f6c37e00640 1 --2- 172.21.15.43:0/4130719334 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c38077350 0x7f6c3819e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:12.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.658+0000 7f6c37e00640 1 -- 172.21.15.43:0/4130719334 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c38077cf0 msgr2=0x7f6c381a55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:12.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.658+0000 7f6c37e00640 1 --2- 172.21.15.43:0/4130719334 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c38077cf0 0x7f6c381a55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:12.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.658+0000 7f6c37e00640 1 -- 172.21.15.43:0/4130719334 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c381a5c10 con 0x7f6c38076110 2024-03-28T22:37:12.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.658+0000 7f6c3ca00640 1 --2- 172.21.15.43:0/4130719334 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c38077cf0 0x7f6c381a55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:37:12.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.658+0000 7f6c37e00640 1 --2- 172.21.15.43:0/4130719334 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6c38076110 0x7f6c3819dff0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6c280349e0 tx=0x7f6c280042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:12.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.658+0000 7f6c34c00640 1 -- 172.21.15.43:0/4130719334 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c28041030 con 0x7f6c38076110 2024-03-28T22:37:12.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.658+0000 7f6c34c00640 1 -- 172.21.15.43:0/4130719334 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c280347b0 con 0x7f6c38076110 2024-03-28T22:37:12.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.658+0000 7f6c3ea00640 1 -- 172.21.15.43:0/4130719334 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c381a5ea0 con 0x7f6c38076110 2024-03-28T22:37:12.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.658+0000 7f6c34c00640 1 -- 172.21.15.43:0/4130719334 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c28039040 con 0x7f6c38076110 2024-03-28T22:37:12.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.659+0000 7f6c3ea00640 1 -- 172.21.15.43:0/4130719334 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c381a6300 con 0x7f6c38076110 2024-03-28T22:37:12.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.660+0000 7f6c3ea00640 1 -- 172.21.15.43:0/4130719334 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c38110320 con 0x7f6c38076110 2024-03-28T22:37:12.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.661+0000 7f6c34c00640 1 -- 172.21.15.43:0/4130719334 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 24) ==== 136568+0+0 (secure 0 0 0) 0x7f6c28044070 con 0x7f6c38076110 2024-03-28T22:37:12.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.663+0000 7f6c34c00640 1 --2- 172.21.15.43:0/4130719334 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6c1c0a1cb0 0x7f6c1c0a4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:12.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.663+0000 7f6c34c00640 1 -- 172.21.15.43:0/4130719334 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f6c280ef4c0 con 0x7f6c38076110 2024-03-28T22:37:12.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.663+0000 7f6c37400640 1 --2- 172.21.15.43:0/4130719334 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6c1c0a1cb0 0x7f6c1c0a4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:12.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.664+0000 7f6c37400640 1 --2- 172.21.15.43:0/4130719334 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6c1c0a1cb0 0x7f6c1c0a4170 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6c3819f500 tx=0x7f6c24009260 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:12.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.667+0000 7f6c34c00640 1 -- 172.21.15.43:0/4130719334 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6c280bc6e0 con 0x7f6c38076110 2024-03-28T22:37:12.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:12.962+0000 7f6c3ea00640 1 -- 172.21.15.43:0/4130719334 --> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f6c38061cf0 con 0x7f6c1c0a1cb0 2024-03-28T22:37:13.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.045+0000 7f6c34c00640 1 -- 172.21.15.43:0/4130719334 <== mgr.25663 v2:172.21.15.43:6800/1685312925 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f6c38061cf0 con 0x7f6c1c0a1cb0 2024-03-28T22:37:13.043 INFO:teuthology.orchestra.run.smithi043.stdout:Retention added to path / 2024-03-28T22:37:13.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.049+0000 7f6c3ea00640 1 -- 172.21.15.43:0/4130719334 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6c1c0a1cb0 msgr2=0x7f6c1c0a4170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:13.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.049+0000 7f6c3ea00640 1 --2- 172.21.15.43:0/4130719334 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6c1c0a1cb0 0x7f6c1c0a4170 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6c3819f500 tx=0x7f6c24009260 comp rx=0 tx=0).stop 2024-03-28T22:37:13.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.049+0000 7f6c3ea00640 1 -- 172.21.15.43:0/4130719334 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6c38076110 msgr2=0x7f6c3819dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:13.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.049+0000 7f6c3ea00640 1 --2- 172.21.15.43:0/4130719334 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6c38076110 0x7f6c3819dff0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6c280349e0 tx=0x7f6c280042f0 comp rx=0 tx=0).stop 2024-03-28T22:37:13.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.049+0000 7f6c3ea00640 1 -- 172.21.15.43:0/4130719334 shutdown_connections 2024-03-28T22:37:13.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.049+0000 7f6c3ea00640 1 --2- 172.21.15.43:0/4130719334 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6c1c0a1cb0 0x7f6c1c0a4170 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:13.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.049+0000 7f6c3ea00640 1 --2- 172.21.15.43:0/4130719334 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6c38077cf0 0x7f6c381a55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:13.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.049+0000 7f6c3ea00640 1 --2- 172.21.15.43:0/4130719334 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6c38077350 0x7f6c3819e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:13.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.049+0000 7f6c3ea00640 1 --2- 172.21.15.43:0/4130719334 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6c38076110 0x7f6c3819dff0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:13.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.049+0000 7f6c3ea00640 1 -- 172.21.15.43:0/4130719334 >> 172.21.15.43:0/4130719334 conn(0x7f6c380fef40 msgr2=0x7f6c38100a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:13.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.049+0000 7f6c3ea00640 1 -- 172.21.15.43:0/4130719334 shutdown_connections 2024-03-28T22:37:13.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.049+0000 7f6c3ea00640 1 -- 172.21.15.43:0/4130719334 wait complete. 2024-03-28T22:37:13.063 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-28T22:37:13.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.246+0000 7f78f9c00640 1 -- 172.21.15.43:0/753291696 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78f41006c0 msgr2=0x7f78f4100ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:13.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.246+0000 7f78f9c00640 1 --2- 172.21.15.43:0/753291696 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78f41006c0 0x7f78f4100ae0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f78e4009900 tx=0x7f78e402bc40 comp rx=0 tx=0).stop 2024-03-28T22:37:13.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.247+0000 7f78f9c00640 1 -- 172.21.15.43:0/753291696 shutdown_connections 2024-03-28T22:37:13.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.247+0000 7f78f9c00640 1 --2- 172.21.15.43:0/753291696 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78f4102270 0x7f78f4108b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:13.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.247+0000 7f78f9c00640 1 --2- 172.21.15.43:0/753291696 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78f41018b0 0x7f78f4101d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:13.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.247+0000 7f78f9c00640 1 --2- 172.21.15.43:0/753291696 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78f41006c0 0x7f78f4100ae0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:13.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.247+0000 7f78f9c00640 1 -- 172.21.15.43:0/753291696 >> 172.21.15.43:0/753291696 conn(0x7f78f40fbc30 msgr2=0x7f78f40fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:13.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.247+0000 7f78f9c00640 1 -- 172.21.15.43:0/753291696 shutdown_connections 2024-03-28T22:37:13.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.247+0000 7f78f9c00640 1 -- 172.21.15.43:0/753291696 wait complete. 2024-03-28T22:37:13.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.248+0000 7f78f9c00640 1 Processor -- start 2024-03-28T22:37:13.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.248+0000 7f78f9c00640 1 -- start start 2024-03-28T22:37:13.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.248+0000 7f78f9c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78f41006c0 0x7f78f4199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:13.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.248+0000 7f78f9c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78f41018b0 0x7f78f419a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:13.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.248+0000 7f78f9c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78f4102270 0x7f78f41a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:13.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.248+0000 7f78f9c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f78f410c310 con 0x7f78f4102270 2024-03-28T22:37:13.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.248+0000 7f78f9c00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f78f410c190 con 0x7f78f41006c0 2024-03-28T22:37:13.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.248+0000 7f78f9c00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f78f410c490 con 0x7f78f41018b0 2024-03-28T22:37:13.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.249+0000 7f78f2a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78f41006c0 0x7f78f4199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:13.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.249+0000 7f78f2a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78f41006c0 0x7f78f4199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:35214/0 (socket says 172.21.15.43:35214) 2024-03-28T22:37:13.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.249+0000 7f78f2a00640 1 -- 172.21.15.43:0/754334737 learned_addr learned my addr 172.21.15.43:0/754334737 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:13.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.249+0000 7f78f3400640 1 --2- 172.21.15.43:0/754334737 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78f4102270 0x7f78f41a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:13.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.249+0000 7f78ea000640 1 --2- 172.21.15.43:0/754334737 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78f41018b0 0x7f78f419a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:13.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.249+0000 7f78f2a00640 1 -- 172.21.15.43:0/754334737 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78f41018b0 msgr2=0x7f78f419a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:13.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.249+0000 7f78f2a00640 1 --2- 172.21.15.43:0/754334737 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78f41018b0 0x7f78f419a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:13.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.249+0000 7f78f2a00640 1 -- 172.21.15.43:0/754334737 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78f4102270 msgr2=0x7f78f41a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:13.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.249+0000 7f78f2a00640 1 --2- 172.21.15.43:0/754334737 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78f4102270 0x7f78f41a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:13.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.249+0000 7f78f2a00640 1 -- 172.21.15.43:0/754334737 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f41a1810 con 0x7f78f41006c0 2024-03-28T22:37:13.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.249+0000 7f78f2a00640 1 --2- 172.21.15.43:0/754334737 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78f41006c0 0x7f78f4199bf0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f78e4000c00 tx=0x7f78e402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:13.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.250+0000 7f78ebe00640 1 -- 172.21.15.43:0/754334737 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e4041070 con 0x7f78f41006c0 2024-03-28T22:37:13.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.250+0000 7f78ebe00640 1 -- 172.21.15.43:0/754334737 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78e4004450 con 0x7f78f41006c0 2024-03-28T22:37:13.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.250+0000 7f78f9c00640 1 -- 172.21.15.43:0/754334737 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78f41a1aa0 con 0x7f78f41006c0 2024-03-28T22:37:13.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.250+0000 7f78ebe00640 1 -- 172.21.15.43:0/754334737 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e4034440 con 0x7f78f41006c0 2024-03-28T22:37:13.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.250+0000 7f78f9c00640 1 -- 172.21.15.43:0/754334737 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78f41a1fe0 con 0x7f78f41006c0 2024-03-28T22:37:13.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.251+0000 7f78f9c00640 1 -- 172.21.15.43:0/754334737 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78f41a2210 con 0x7f78f41006c0 2024-03-28T22:37:13.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.253+0000 7f78ebe00640 1 -- 172.21.15.43:0/754334737 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 24) ==== 136568+0+0 (secure 0 0 0) 0x7f78e4044070 con 0x7f78f41006c0 2024-03-28T22:37:13.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.254+0000 7f78ebe00640 1 --2- 172.21.15.43:0/754334737 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f78d80a1cb0 0x7f78d80a4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:13.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.254+0000 7f78ebe00640 1 -- 172.21.15.43:0/754334737 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f78e40efc50 con 0x7f78f41006c0 2024-03-28T22:37:13.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.255+0000 7f78ea000640 1 --2- 172.21.15.43:0/754334737 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f78d80a1cb0 0x7f78d80a4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:13.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.255+0000 7f78ea000640 1 --2- 172.21.15.43:0/754334737 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f78d80a1cb0 0x7f78d80a4170 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f78f419b100 tx=0x7f78dc005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:13.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.260+0000 7f78ebe00640 1 -- 172.21.15.43:0/754334737 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f78e40bcf00 con 0x7f78f41006c0 2024-03-28T22:37:13.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.355+0000 7f78ebe00640 1 -- 172.21.15.43:0/754334737 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mgrmap(e 25) ==== 136876+0+0 (secure 0 0 0) 0x7f78e40b8e70 con 0x7f78f41006c0 2024-03-28T22:37:13.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.489+0000 7f78f9c00640 1 -- 172.21.15.43:0/754334737 --> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f78f40fdd60 con 0x7f78d80a1cb0 2024-03-28T22:37:13.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.498+0000 7f78ebe00640 1 -- 172.21.15.43:0/754334737 <== mgr.25663 v2:172.21.15.43:6800/1685312925 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f78f40fdd60 con 0x7f78d80a1cb0 2024-03-28T22:37:13.496 INFO:teuthology.orchestra.run.smithi043.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-28T00:00:00", "created": "2024-03-28T22:37:12", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-03-28T22:37:13.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.501+0000 7f78f9c00640 1 -- 172.21.15.43:0/754334737 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f78d80a1cb0 msgr2=0x7f78d80a4170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:13.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.501+0000 7f78f9c00640 1 --2- 172.21.15.43:0/754334737 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f78d80a1cb0 0x7f78d80a4170 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f78f419b100 tx=0x7f78dc005f90 comp rx=0 tx=0).stop 2024-03-28T22:37:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.501+0000 7f78f9c00640 1 -- 172.21.15.43:0/754334737 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78f41006c0 msgr2=0x7f78f4199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.501+0000 7f78f9c00640 1 --2- 172.21.15.43:0/754334737 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78f41006c0 0x7f78f4199bf0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f78e4000c00 tx=0x7f78e402c650 comp rx=0 tx=0).stop 2024-03-28T22:37:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.501+0000 7f78f9c00640 1 -- 172.21.15.43:0/754334737 shutdown_connections 2024-03-28T22:37:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.501+0000 7f78f9c00640 1 --2- 172.21.15.43:0/754334737 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f78d80a1cb0 0x7f78d80a4170 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.501+0000 7f78f9c00640 1 --2- 172.21.15.43:0/754334737 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f78f4102270 0x7f78f41a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.501+0000 7f78f9c00640 1 --2- 172.21.15.43:0/754334737 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f78f41018b0 0x7f78f419a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.501+0000 7f78f9c00640 1 --2- 172.21.15.43:0/754334737 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78f41006c0 0x7f78f4199bf0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.502+0000 7f78f9c00640 1 -- 172.21.15.43:0/754334737 >> 172.21.15.43:0/754334737 conn(0x7f78f40fbc30 msgr2=0x7f78f40fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.502+0000 7f78f9c00640 1 -- 172.21.15.43:0/754334737 shutdown_connections 2024-03-28T22:37:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.502+0000 7f78f9c00640 1 -- 172.21.15.43:0/754334737 wait complete. 2024-03-28T22:37:13.516 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-28T22:37:13.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:13 smithi172 ceph-mon[36566]: mgrmap e24: x(active, since 1.02438s), standbys: y, z 2024-03-28T22:37:13.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:13 smithi172 ceph-mon[36566]: [28/Mar/2024:22:37:13] ENGINE Bus STARTING 2024-03-28T22:37:13.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:13 smithi172 ceph-mon[36566]: scrub summary: idle 2024-03-28T22:37:13.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:13 smithi172 ceph-mon[36566]: [28/Mar/2024:22:37:13] ENGINE Serving on http://172.21.15.43:8765 2024-03-28T22:37:13.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:13 smithi082 ceph-mon[36471]: mgrmap e24: x(active, since 1.02438s), standbys: y, z 2024-03-28T22:37:13.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:13 smithi082 ceph-mon[36471]: [28/Mar/2024:22:37:13] ENGINE Bus STARTING 2024-03-28T22:37:13.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:13 smithi082 ceph-mon[36471]: scrub summary: idle 2024-03-28T22:37:13.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:13 smithi082 ceph-mon[36471]: [28/Mar/2024:22:37:13] ENGINE Serving on http://172.21.15.43:8765 2024-03-28T22:37:13.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.718+0000 7fbd27200640 1 -- 172.21.15.43:0/1873902150 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbd281027c0 msgr2=0x7fbd28102c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:13.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.718+0000 7fbd27200640 1 --2- 172.21.15.43:0/1873902150 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbd281027c0 0x7fbd28102c40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fbd14009900 tx=0x7fbd1402bc30 comp rx=0 tx=0).stop 2024-03-28T22:37:13.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.718+0000 7fbd27200640 1 -- 172.21.15.43:0/1873902150 shutdown_connections 2024-03-28T22:37:13.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.718+0000 7fbd27200640 1 --2- 172.21.15.43:0/1873902150 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbd281027c0 0x7fbd28102c40 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:13.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.718+0000 7fbd27200640 1 --2- 172.21.15.43:0/1873902150 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd28103180 0x7fbd28109a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:13.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.718+0000 7fbd27200640 1 --2- 172.21.15.43:0/1873902150 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd281015d0 0x7fbd281019f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:13.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.718+0000 7fbd27200640 1 -- 172.21.15.43:0/1873902150 >> 172.21.15.43:0/1873902150 conn(0x7fbd280fcb60 msgr2=0x7fbd280fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:13.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.719+0000 7fbd27200640 1 -- 172.21.15.43:0/1873902150 shutdown_connections 2024-03-28T22:37:13.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.719+0000 7fbd27200640 1 -- 172.21.15.43:0/1873902150 wait complete. 2024-03-28T22:37:13.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.720+0000 7fbd27200640 1 Processor -- start 2024-03-28T22:37:13.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.720+0000 7fbd27200640 1 -- start start 2024-03-28T22:37:13.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.721+0000 7fbd27200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd281015d0 0x7fbd28199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:13.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.721+0000 7fbd27200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbd281027c0 0x7fbd28199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:13.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.721+0000 7fbd27200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd28103180 0x7fbd281a1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:13.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.721+0000 7fbd27200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2810d250 con 0x7fbd281015d0 2024-03-28T22:37:13.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.721+0000 7fbd27200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2810d0d0 con 0x7fbd281027c0 2024-03-28T22:37:13.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.721+0000 7fbd27200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2810d3d0 con 0x7fbd28103180 2024-03-28T22:37:13.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.721+0000 7fbd25e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd281015d0 0x7fbd28199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:13.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.721+0000 7fbd25400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbd281027c0 0x7fbd28199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:13.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.721+0000 7fbd25400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbd281027c0 0x7fbd28199fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:35248/0 (socket says 172.21.15.43:35248) 2024-03-28T22:37:13.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.721+0000 7fbd25400640 1 -- 172.21.15.43:0/1225547157 learned_addr learned my addr 172.21.15.43:0/1225547157 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:13.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.721+0000 7fbd25e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd281015d0 0x7fbd28199a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39144/0 (socket says 172.21.15.43:39144) 2024-03-28T22:37:13.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.721+0000 7fbd26800640 1 --2- 172.21.15.43:0/1225547157 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd28103180 0x7fbd281a1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:13.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.722+0000 7fbd25400640 1 -- 172.21.15.43:0/1225547157 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd28103180 msgr2=0x7fbd281a1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:13.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.722+0000 7fbd25400640 1 --2- 172.21.15.43:0/1225547157 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd28103180 0x7fbd281a1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:13.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.722+0000 7fbd25400640 1 -- 172.21.15.43:0/1225547157 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd281015d0 msgr2=0x7fbd28199a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:13.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.722+0000 7fbd25400640 1 --2- 172.21.15.43:0/1225547157 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd281015d0 0x7fbd28199a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:13.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.722+0000 7fbd25400640 1 -- 172.21.15.43:0/1225547157 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd281a1690 con 0x7fbd281027c0 2024-03-28T22:37:13.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.722+0000 7fbd25400640 1 --2- 172.21.15.43:0/1225547157 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbd281027c0 0x7fbd28199fb0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fbd14009230 tx=0x7fbd14005dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:13.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.722+0000 7fbd1ea00640 1 -- 172.21.15.43:0/1225547157 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd14041070 con 0x7fbd281027c0 2024-03-28T22:37:13.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.722+0000 7fbd1ea00640 1 -- 172.21.15.43:0/1225547157 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd140054e0 con 0x7fbd281027c0 2024-03-28T22:37:13.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.722+0000 7fbd1ea00640 1 -- 172.21.15.43:0/1225547157 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd14034440 con 0x7fbd281027c0 2024-03-28T22:37:13.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.723+0000 7fbd27200640 1 -- 172.21.15.43:0/1225547157 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd281a1920 con 0x7fbd281027c0 2024-03-28T22:37:13.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.723+0000 7fbd27200640 1 -- 172.21.15.43:0/1225547157 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd281a1dc0 con 0x7fbd281027c0 2024-03-28T22:37:13.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.724+0000 7fbd27200640 1 -- 172.21.15.43:0/1225547157 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd281a1fc0 con 0x7fbd281027c0 2024-03-28T22:37:13.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.726+0000 7fbd1ea00640 1 -- 172.21.15.43:0/1225547157 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 25) ==== 136876+0+0 (secure 0 0 0) 0x7fbd14044070 con 0x7fbd281027c0 2024-03-28T22:37:13.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.727+0000 7fbd1ea00640 1 --2- 172.21.15.43:0/1225547157 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fbd0c0a1ee0 0x7fbd0c0a43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:13.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.727+0000 7fbd1ea00640 1 -- 172.21.15.43:0/1225547157 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fbd140f0180 con 0x7fbd281027c0 2024-03-28T22:37:13.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.728+0000 7fbd25e00640 1 --2- 172.21.15.43:0/1225547157 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fbd0c0a1ee0 0x7fbd0c0a43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:13.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.728+0000 7fbd25e00640 1 --2- 172.21.15.43:0/1225547157 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fbd0c0a1ee0 0x7fbd0c0a43a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbd28102620 tx=0x7fbd1800b390 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:13.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.731+0000 7fbd1ea00640 1 -- 172.21.15.43:0/1225547157 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbd140bcf80 con 0x7fbd281027c0 2024-03-28T22:37:13.765 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:13 smithi043 ceph-mon[31823]: mgrmap e24: x(active, since 1.02438s), standbys: y, z 2024-03-28T22:37:13.765 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:13 smithi043 ceph-mon[31823]: [28/Mar/2024:22:37:13] ENGINE Bus STARTING 2024-03-28T22:37:13.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:13 smithi043 ceph-mon[31823]: scrub summary: idle 2024-03-28T22:37:13.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:13 smithi043 ceph-mon[31823]: [28/Mar/2024:22:37:13] ENGINE Serving on http://172.21.15.43:8765 2024-03-28T22:37:13.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.965+0000 7fbd27200640 1 -- 172.21.15.43:0/1225547157 --> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fbd281a2190 con 0x7fbd0c0a1ee0 2024-03-28T22:37:13.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.974+0000 7fbd1ea00640 1 -- 172.21.15.43:0/1225547157 <== mgr.25663 v2:172.21.15.43:6800/1685312925 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fbd281a2190 con 0x7fbd0c0a1ee0 2024-03-28T22:37:13.971 INFO:teuthology.orchestra.run.smithi043.stdout:/ 1m 6m3h 2024-03-28T22:37:13.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.976+0000 7fbd27200640 1 -- 172.21.15.43:0/1225547157 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fbd0c0a1ee0 msgr2=0x7fbd0c0a43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:13.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.976+0000 7fbd27200640 1 --2- 172.21.15.43:0/1225547157 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fbd0c0a1ee0 0x7fbd0c0a43a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbd28102620 tx=0x7fbd1800b390 comp rx=0 tx=0).stop 2024-03-28T22:37:13.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.976+0000 7fbd27200640 1 -- 172.21.15.43:0/1225547157 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbd281027c0 msgr2=0x7fbd28199fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:13.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.976+0000 7fbd27200640 1 --2- 172.21.15.43:0/1225547157 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbd281027c0 0x7fbd28199fb0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fbd14009230 tx=0x7fbd14005dd0 comp rx=0 tx=0).stop 2024-03-28T22:37:13.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.976+0000 7fbd27200640 1 -- 172.21.15.43:0/1225547157 shutdown_connections 2024-03-28T22:37:13.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.976+0000 7fbd27200640 1 --2- 172.21.15.43:0/1225547157 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fbd0c0a1ee0 0x7fbd0c0a43a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:13.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.976+0000 7fbd27200640 1 --2- 172.21.15.43:0/1225547157 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbd281027c0 0x7fbd28199fb0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:13.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.976+0000 7fbd27200640 1 --2- 172.21.15.43:0/1225547157 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbd28103180 0x7fbd281a1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:13.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.976+0000 7fbd27200640 1 --2- 172.21.15.43:0/1225547157 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbd281015d0 0x7fbd28199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:13.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.976+0000 7fbd27200640 1 -- 172.21.15.43:0/1225547157 >> 172.21.15.43:0/1225547157 conn(0x7fbd280fcb60 msgr2=0x7fbd28073980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:13.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.976+0000 7fbd27200640 1 -- 172.21.15.43:0/1225547157 shutdown_connections 2024-03-28T22:37:13.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:13.976+0000 7fbd27200640 1 -- 172.21.15.43:0/1225547157 wait complete. 2024-03-28T22:37:13.991 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-03-28T22:37:14.072 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-03-28T22:37:14.083 INFO:teuthology.run_tasks:Running task background_exec... 2024-03-28T22:37:14.124 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi043.front.sepia.ceph.com 2024-03-28T22:37:14.124 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-03-28T22:37:14.134 INFO:teuthology.run_tasks:Running task workunit... 2024-03-28T22:37:14.148 INFO:tasks.workunit:Pulling workunits from ref 55f5138942aa4866e49329c9103468889cec3f73 2024-03-28T22:37:14.149 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-28T22:37:14.149 INFO:tasks.workunit:timeout=3h 2024-03-28T22:37:14.149 INFO:tasks.workunit:cleanup=True 2024-03-28T22:37:14.149 DEBUG:teuthology.orchestra.run.smithi043:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-28T22:37:14.192 INFO:teuthology.orchestra.run.smithi043.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-28T22:37:14.192 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-28T22:37:14.192 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 76h/118d Inode: 1099511628283 Links: 2 2024-03-28T22:37:14.192 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T22:37:14.192 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-28T22:37:14.192 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-03-28 22:34:06.161839752 +0000 2024-03-28T22:37:14.193 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-03-28 22:34:06.161839752 +0000 2024-03-28T22:37:14.193 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-03-28 22:36:48.510845803 +0000 2024-03-28T22:37:14.193 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-03-28 22:34:06.161839752 +0000 2024-03-28T22:37:14.193 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-28T22:37:14.193 DEBUG:teuthology.orchestra.run.smithi043:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-28T22:37:14.323 DEBUG:teuthology.orchestra.run.smithi043:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 55f5138942aa4866e49329c9103468889cec3f73 2024-03-28T22:37:14.366 INFO:tasks.workunit.client.0.smithi043.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-28T22:37:14.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:14 smithi172 ceph-mon[36566]: [28/Mar/2024:22:37:13] ENGINE Serving on https://172.21.15.43:7150 2024-03-28T22:37:14.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:14 smithi172 ceph-mon[36566]: [28/Mar/2024:22:37:13] ENGINE Bus STARTED 2024-03-28T22:37:14.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:14 smithi172 ceph-mon[36566]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:37:14.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:14 smithi172 ceph-mon[36566]: mgrmap e25: x(active, since 2s), standbys: y, z 2024-03-28T22:37:14.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:14 smithi172 ceph-mon[36566]: from='client.25727 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:37:14.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:14 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:14.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:14 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:14.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:14 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:14.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:14 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:14.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:14 smithi172 ceph-mon[36566]: from='client.25739 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:37:14.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:14 smithi082 ceph-mon[36471]: [28/Mar/2024:22:37:13] ENGINE Serving on https://172.21.15.43:7150 2024-03-28T22:37:14.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:14 smithi082 ceph-mon[36471]: [28/Mar/2024:22:37:13] ENGINE Bus STARTED 2024-03-28T22:37:14.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:14 smithi082 ceph-mon[36471]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:37:14.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:14 smithi082 ceph-mon[36471]: mgrmap e25: x(active, since 2s), standbys: y, z 2024-03-28T22:37:14.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:14 smithi082 ceph-mon[36471]: from='client.25727 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:37:14.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:14 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:14.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:14 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:14.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:14 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:14.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:14 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:14.700 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:14 smithi082 ceph-mon[36471]: from='client.25739 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:37:14.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:14 smithi043 ceph-mon[31823]: [28/Mar/2024:22:37:13] ENGINE Serving on https://172.21.15.43:7150 2024-03-28T22:37:14.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:14 smithi043 ceph-mon[31823]: [28/Mar/2024:22:37:13] ENGINE Bus STARTED 2024-03-28T22:37:14.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:14 smithi043 ceph-mon[31823]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:37:14.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:14 smithi043 ceph-mon[31823]: mgrmap e25: x(active, since 2s), standbys: y, z 2024-03-28T22:37:14.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:14 smithi043 ceph-mon[31823]: from='client.25727 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:37:14.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:14 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:14.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:14 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:14.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:14 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:14.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:14 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:14.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:14 smithi043 ceph-mon[31823]: from='client.25739 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:37:15.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:15 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:15.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:15 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:15.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:15 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:15.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:15 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:15.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:15 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:15.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:15 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:15.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:15 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:15.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:15 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:15.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:15 smithi082 ceph-mon[36471]: Adjusting osd_memory_target on smithi082 to 174.1M 2024-03-28T22:37:15.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:15 smithi082 ceph-mon[36471]: Unable to set osd_memory_target on smithi082 to 182611251: error parsing value: Value '182611251' is below minimum 939524096 2024-03-28T22:37:15.951 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:15 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:15.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:15 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:15.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:15 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:15.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:15 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:15.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:15 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:15.952 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:15 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:16.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:15 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:16.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:15 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:16.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:15 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:16.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:15 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:16.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:15 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:16.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:15 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:16.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:15 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:16.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:15 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:16.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:15 smithi043 ceph-mon[31823]: Adjusting osd_memory_target on smithi082 to 174.1M 2024-03-28T22:37:16.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:15 smithi043 ceph-mon[31823]: Unable to set osd_memory_target on smithi082 to 182611251: error parsing value: Value '182611251' is below minimum 939524096 2024-03-28T22:37:16.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:15 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:16.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:15 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:16.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:15 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:16.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:15 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:16.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:15 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:16.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:15 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:16.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:15 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:16.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:15 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:16.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:15 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:16.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:15 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:16.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:15 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:16.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:15 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:16.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:15 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:16.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:15 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:16.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:15 smithi172 ceph-mon[36566]: Adjusting osd_memory_target on smithi082 to 174.1M 2024-03-28T22:37:16.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:15 smithi172 ceph-mon[36566]: Unable to set osd_memory_target on smithi082 to 182611251: error parsing value: Value '182611251' is below minimum 939524096 2024-03-28T22:37:16.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:15 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:16.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:15 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:16.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:15 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:16.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:15 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:16.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:15 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:16.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:15 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:17.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:17 smithi082 ceph-mon[36471]: Adjusting osd_memory_target on smithi172 to 174.1M 2024-03-28T22:37:17.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:17 smithi082 ceph-mon[36471]: Unable to set osd_memory_target on smithi172 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T22:37:17.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:17 smithi082 ceph-mon[36471]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:37:17.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:17 smithi082 ceph-mon[36471]: mgrmap e26: x(active, since 4s), standbys: y, z 2024-03-28T22:37:17.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:17 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:17.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:17 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:17.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:17 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:17.451 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:17 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:17.451 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:17 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:17.451 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:17 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:17.451 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:17 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:37:17.451 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:17 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:37:17.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:17 smithi043 ceph-mon[31823]: Adjusting osd_memory_target on smithi172 to 174.1M 2024-03-28T22:37:17.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:17 smithi043 ceph-mon[31823]: Unable to set osd_memory_target on smithi172 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T22:37:17.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:17 smithi043 ceph-mon[31823]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:37:17.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:17 smithi043 ceph-mon[31823]: mgrmap e26: x(active, since 4s), standbys: y, z 2024-03-28T22:37:17.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:17 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:17.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:17 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:17.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:17 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:17.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:17 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:17.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:17 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:17.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:17 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:17.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:17 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:37:17.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:17 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:37:17.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:17 smithi172 ceph-mon[36566]: Adjusting osd_memory_target on smithi172 to 174.1M 2024-03-28T22:37:17.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:17 smithi172 ceph-mon[36566]: Unable to set osd_memory_target on smithi172 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T22:37:17.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:17 smithi172 ceph-mon[36566]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:37:17.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:17 smithi172 ceph-mon[36566]: mgrmap e26: x(active, since 4s), standbys: y, z 2024-03-28T22:37:17.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:17 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:17.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:17 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:17.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:17 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:17.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:17 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:17.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:17 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:17.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:17 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:37:17.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:17 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:37:17.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:17 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:37:18.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:18 smithi082 ceph-mon[36471]: Adjusting osd_memory_target on smithi043 to 177.8M 2024-03-28T22:37:18.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:18 smithi082 ceph-mon[36471]: Unable to set osd_memory_target on smithi043 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T22:37:18.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:18 smithi082 ceph-mon[36471]: Updating smithi043:/etc/ceph/ceph.conf 2024-03-28T22:37:18.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:18 smithi082 ceph-mon[36471]: Updating smithi082:/etc/ceph/ceph.conf 2024-03-28T22:37:18.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:18 smithi082 ceph-mon[36471]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-28T22:37:18.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:18 smithi082 ceph-mon[36471]: Updating smithi082:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:37:18.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:18 smithi082 ceph-mon[36471]: Updating smithi172:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:37:18.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:18 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:18.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:18 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:18.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:18 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:18.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:18 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:18.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:18 smithi043 ceph-mon[31823]: Adjusting osd_memory_target on smithi043 to 177.8M 2024-03-28T22:37:18.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:18 smithi043 ceph-mon[31823]: Unable to set osd_memory_target on smithi043 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T22:37:18.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:18 smithi043 ceph-mon[31823]: Updating smithi043:/etc/ceph/ceph.conf 2024-03-28T22:37:18.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:18 smithi043 ceph-mon[31823]: Updating smithi082:/etc/ceph/ceph.conf 2024-03-28T22:37:18.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:18 smithi043 ceph-mon[31823]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-28T22:37:18.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:18 smithi043 ceph-mon[31823]: Updating smithi082:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:37:18.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:18 smithi043 ceph-mon[31823]: Updating smithi172:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:37:18.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:18 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:18.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:18 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:18.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:18 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:18.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:18 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:18.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:18 smithi172 ceph-mon[36566]: Adjusting osd_memory_target on smithi043 to 177.8M 2024-03-28T22:37:18.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:18 smithi172 ceph-mon[36566]: Unable to set osd_memory_target on smithi043 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T22:37:18.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:18 smithi172 ceph-mon[36566]: Updating smithi043:/etc/ceph/ceph.conf 2024-03-28T22:37:18.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:18 smithi172 ceph-mon[36566]: Updating smithi082:/etc/ceph/ceph.conf 2024-03-28T22:37:18.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:18 smithi172 ceph-mon[36566]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-28T22:37:18.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:18 smithi172 ceph-mon[36566]: Updating smithi082:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:37:18.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:18 smithi172 ceph-mon[36566]: Updating smithi172:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:37:18.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:18 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:18.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:18 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:18.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:18 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:18.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:18 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:19.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:19 smithi082 ceph-mon[36471]: pgmap v6: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 94 KiB/s wr, 11 op/s 2024-03-28T22:37:19.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:19 smithi082 ceph-mon[36471]: Updating smithi082:/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:37:19.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:19 smithi082 ceph-mon[36471]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:37:19.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:19 smithi082 ceph-mon[36471]: Updating smithi043:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:37:19.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:19 smithi082 ceph-mon[36471]: Updating smithi082:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T22:37:19.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:19 smithi082 ceph-mon[36471]: Updating smithi172:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T22:37:19.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:19 smithi082 ceph-mon[36471]: Updating smithi043:/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:37:19.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:19 smithi082 ceph-mon[36471]: Updating smithi043:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T22:37:19.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:19 smithi043 ceph-mon[31823]: pgmap v6: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 94 KiB/s wr, 11 op/s 2024-03-28T22:37:19.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:19 smithi043 ceph-mon[31823]: Updating smithi082:/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:37:19.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:19 smithi043 ceph-mon[31823]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:37:19.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:19 smithi043 ceph-mon[31823]: Updating smithi043:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:37:19.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:19 smithi043 ceph-mon[31823]: Updating smithi082:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T22:37:19.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:19 smithi043 ceph-mon[31823]: Updating smithi172:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T22:37:19.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:19 smithi043 ceph-mon[31823]: Updating smithi043:/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:37:19.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:19 smithi043 ceph-mon[31823]: Updating smithi043:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T22:37:19.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:19 smithi172 ceph-mon[36566]: pgmap v6: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 94 KiB/s wr, 11 op/s 2024-03-28T22:37:19.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:19 smithi172 ceph-mon[36566]: Updating smithi082:/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:37:19.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:19 smithi172 ceph-mon[36566]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:37:19.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:19 smithi172 ceph-mon[36566]: Updating smithi043:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T22:37:19.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:19 smithi172 ceph-mon[36566]: Updating smithi082:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T22:37:19.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:19 smithi172 ceph-mon[36566]: Updating smithi172:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T22:37:19.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:19 smithi172 ceph-mon[36566]: Updating smithi043:/etc/ceph/ceph.client.admin.keyring 2024-03-28T22:37:19.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:19 smithi172 ceph-mon[36566]: Updating smithi043:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T22:37:21.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:20 smithi082 ceph-mon[36471]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 132 KiB/s wr, 80 op/s 2024-03-28T22:37:21.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:20 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:21.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:20 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:21.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:20 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:21.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:20 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:37:21.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:20 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:37:21.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:20 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:37:21.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:20 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:21.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:20 smithi043 ceph-mon[31823]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 132 KiB/s wr, 80 op/s 2024-03-28T22:37:21.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:20 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:21.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:20 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:21.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:20 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:21.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:20 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:37:21.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:20 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:37:21.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:20 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:37:21.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:20 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:21.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:20 smithi172 ceph-mon[36566]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 132 KiB/s wr, 80 op/s 2024-03-28T22:37:21.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:20 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:21.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:20 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:21.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:20 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:21.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:20 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:37:21.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:20 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:37:21.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:20 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:37:21.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:20 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:37:23.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:22 smithi082 ceph-mon[36471]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 103 KiB/s wr, 62 op/s 2024-03-28T22:37:23.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:22 smithi043 ceph-mon[31823]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 103 KiB/s wr, 62 op/s 2024-03-28T22:37:23.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:22 smithi172 ceph-mon[36566]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 103 KiB/s wr, 62 op/s 2024-03-28T22:37:25.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:24 smithi082 ceph-mon[36471]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 KiB/s wr, 56 op/s 2024-03-28T22:37:25.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:24 smithi043 ceph-mon[31823]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 KiB/s wr, 56 op/s 2024-03-28T22:37:25.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:24 smithi172 ceph-mon[36566]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 KiB/s wr, 56 op/s 2024-03-28T22:37:27.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:26 smithi082 ceph-mon[36471]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-03-28T22:37:27.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:26 smithi043 ceph-mon[31823]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-03-28T22:37:27.387 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.389+0000 7f3b57c00640 1 -- 172.21.15.43:0/352701764 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3b50112700 msgr2=0x7f3b50114af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:27.388 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.389+0000 7f3b57c00640 1 --2- 172.21.15.43:0/352701764 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3b50112700 0x7f3b50114af0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f3b48009ca0 tx=0x7f3b4802c180 comp rx=0 tx=0).stop 2024-03-28T22:37:27.388 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.389+0000 7f3b57c00640 1 -- 172.21.15.43:0/352701764 shutdown_connections 2024-03-28T22:37:27.388 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.389+0000 7f3b57c00640 1 --2- 172.21.15.43:0/352701764 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3b50112700 0x7f3b50114af0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:27.388 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.389+0000 7f3b57c00640 1 --2- 172.21.15.43:0/352701764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b50074410 0x7f3b501121c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:27.388 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.389+0000 7f3b57c00640 1 --2- 172.21.15.43:0/352701764 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b50073af0 0x7f3b50073ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:27.388 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.389+0000 7f3b57c00640 1 -- 172.21.15.43:0/352701764 >> 172.21.15.43:0/352701764 conn(0x7f3b500fd1e0 msgr2=0x7f3b500ff600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:27.388 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.389+0000 7f3b57c00640 1 -- 172.21.15.43:0/352701764 shutdown_connections 2024-03-28T22:37:27.388 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.389+0000 7f3b57c00640 1 -- 172.21.15.43:0/352701764 wait complete. 2024-03-28T22:37:27.389 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.391+0000 7f3b57c00640 1 Processor -- start 2024-03-28T22:37:27.389 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.391+0000 7f3b57c00640 1 -- start start 2024-03-28T22:37:27.389 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.391+0000 7f3b57c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b50073af0 0x7f3b5019e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:27.389 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.391+0000 7f3b57c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3b50074410 0x7f3b5019ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:27.390 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.391+0000 7f3b57c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b50112700 0x7f3b501a2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:27.390 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.391+0000 7f3b57c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3b501180a0 con 0x7f3b50073af0 2024-03-28T22:37:27.390 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.391+0000 7f3b57c00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f3b50117f20 con 0x7f3b50074410 2024-03-28T22:37:27.390 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.391+0000 7f3b57c00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f3b50118220 con 0x7f3b50112700 2024-03-28T22:37:27.390 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.392+0000 7f3b4fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3b50074410 0x7f3b5019ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:27.390 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.392+0000 7f3b4fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3b50074410 0x7f3b5019ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:36376/0 (socket says 172.21.15.43:36376) 2024-03-28T22:37:27.390 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.392+0000 7f3b4fe00640 1 -- 172.21.15.43:0/532468487 learned_addr learned my addr 172.21.15.43:0/532468487 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:27.390 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.392+0000 7f3b55200640 1 --2- 172.21.15.43:0/532468487 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b50073af0 0x7f3b5019e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:27.390 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.392+0000 7f3b55c00640 1 --2- 172.21.15.43:0/532468487 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b50112700 0x7f3b501a2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:27.391 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.392+0000 7f3b4fe00640 1 -- 172.21.15.43:0/532468487 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b50112700 msgr2=0x7f3b501a2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:27.391 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.392+0000 7f3b4fe00640 1 --2- 172.21.15.43:0/532468487 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b50112700 0x7f3b501a2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:27.391 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.392+0000 7f3b4fe00640 1 -- 172.21.15.43:0/532468487 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b50073af0 msgr2=0x7f3b5019e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:27.391 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.392+0000 7f3b4fe00640 1 --2- 172.21.15.43:0/532468487 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b50073af0 0x7f3b5019e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:27.391 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.392+0000 7f3b4fe00640 1 -- 172.21.15.43:0/532468487 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b501a3510 con 0x7f3b50074410 2024-03-28T22:37:27.392 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.392+0000 7f3b55200640 1 --2- 172.21.15.43:0/532468487 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b50073af0 0x7f3b5019e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:37:27.392 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.392+0000 7f3b4fe00640 1 --2- 172.21.15.43:0/532468487 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3b50074410 0x7f3b5019ea60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c00ea60 tx=0x7f3b3c00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:27.392 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.393+0000 7f3b4d600640 1 -- 172.21.15.43:0/532468487 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b3c00cde0 con 0x7f3b50074410 2024-03-28T22:37:27.392 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.393+0000 7f3b4d600640 1 -- 172.21.15.43:0/532468487 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b3c0035f0 con 0x7f3b50074410 2024-03-28T22:37:27.392 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.393+0000 7f3b4d600640 1 -- 172.21.15.43:0/532468487 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b3c010670 con 0x7f3b50074410 2024-03-28T22:37:27.392 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.393+0000 7f3b57c00640 1 -- 172.21.15.43:0/532468487 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b501a3800 con 0x7f3b50074410 2024-03-28T22:37:27.392 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.393+0000 7f3b57c00640 1 -- 172.21.15.43:0/532468487 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b501ab090 con 0x7f3b50074410 2024-03-28T22:37:27.394 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.396+0000 7f3b5a884740 1 Processor -- start 2024-03-28T22:37:27.394 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.396+0000 7f3b5a884740 1 -- start start 2024-03-28T22:37:27.394 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.397+0000 7f3b4d600640 1 -- 172.21.15.43:0/532468487 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 26) ==== 136932+0+0 (secure 0 0 0) 0x7f3b3c0037a0 con 0x7f3b50074410 2024-03-28T22:37:27.397 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.397+0000 7f3b5a884740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5614d75623a0 0x5614d7562780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:27.397 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.397+0000 7f3b5a884740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5614d7562cc0 0x5614d756b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:27.397 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.397+0000 7f3b5a884740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5614d756ba90 0x5614d756de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:27.397 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.397+0000 7f3b5a884740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5614d74c5640 con 0x5614d756ba90 2024-03-28T22:37:27.397 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.397+0000 7f3b5a884740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5614d746e810 con 0x5614d7562cc0 2024-03-28T22:37:27.397 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.397+0000 7f3b5a884740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5614d74c16e0 con 0x5614d75623a0 2024-03-28T22:37:27.397 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.397+0000 7f3b55200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5614d7562cc0 0x5614d756b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:27.398 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.397+0000 7f3b55200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5614d7562cc0 0x5614d756b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:36392/0 (socket says 172.21.15.43:36392) 2024-03-28T22:37:27.398 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.397+0000 7f3b55200640 1 -- 172.21.15.43:0/2537783413 learned_addr learned my addr 172.21.15.43:0/2537783413 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:27.398 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.397+0000 7f3b4fe00640 1 --2- 172.21.15.43:0/2537783413 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5614d756ba90 0x5614d756de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:27.398 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.397+0000 7f3b55200640 1 -- 172.21.15.43:0/2537783413 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5614d75623a0 msgr2=0x5614d7562780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:27.398 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.397+0000 7f3b55200640 1 --2- 172.21.15.43:0/2537783413 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5614d75623a0 0x5614d7562780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:27.398 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.397+0000 7f3b55200640 1 -- 172.21.15.43:0/2537783413 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5614d756ba90 msgr2=0x5614d756de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:27.398 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.398+0000 7f3b55200640 1 --2- 172.21.15.43:0/2537783413 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5614d756ba90 0x5614d756de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:27.398 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.398+0000 7f3b55200640 1 -- 172.21.15.43:0/2537783413 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614d71da210 con 0x5614d7562cc0 2024-03-28T22:37:27.398 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.398+0000 7f3b55200640 1 --2- 172.21.15.43:0/2537783413 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5614d7562cc0 0x5614d756b550 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3b44009d50 tx=0x7f3b440117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d596d520ecacea1e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:27.398 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.398+0000 7f3b42000640 1 -- 172.21.15.43:0/2537783413 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b44012650 con 0x5614d7562cc0 2024-03-28T22:37:27.398 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.398+0000 7f3b42000640 1 -- 172.21.15.43:0/2537783413 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b44009e40 con 0x5614d7562cc0 2024-03-28T22:37:27.398 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.398+0000 7f3b42000640 1 -- 172.21.15.43:0/2537783413 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b4400fc60 con 0x5614d7562cc0 2024-03-28T22:37:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.398+0000 7f3b4d600640 1 --2- 172.21.15.43:0/532468487 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f3b380a20b0 0x7f3b380a4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.399+0000 7f3b4d600640 1 -- 172.21.15.43:0/532468487 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f3b3c0ce9f0 con 0x7f3b50074410 2024-03-28T22:37:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.399+0000 7f3b55c00640 1 --2- 172.21.15.43:0/532468487 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f3b380a20b0 0x7f3b380a4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.399+0000 7f3b5a884740 1 -- 172.21.15.43:0/2537783413 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5614d7562cc0 msgr2=0x5614d756b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.399+0000 7f3b5a884740 1 --2- 172.21.15.43:0/2537783413 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5614d7562cc0 0x5614d756b550 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3b44009d50 tx=0x7f3b440117b0 comp rx=0 tx=0).stop 2024-03-28T22:37:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.399+0000 7f3b5a884740 1 -- 172.21.15.43:0/2537783413 shutdown_connections 2024-03-28T22:37:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.399+0000 7f3b5a884740 1 --2- 172.21.15.43:0/2537783413 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5614d7562cc0 0x5614d756b550 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.399+0000 7f3b5a884740 1 --2- 172.21.15.43:0/2537783413 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5614d756ba90 0x5614d756de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.399+0000 7f3b5a884740 1 --2- 172.21.15.43:0/2537783413 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5614d75623a0 0x5614d7562780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.399+0000 7f3b5a884740 1 -- 172.21.15.43:0/2537783413 >> 172.21.15.43:0/2537783413 conn(0x5614d755ccc0 msgr2=0x5614d755d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.399+0000 7f3b5a884740 1 -- 172.21.15.43:0/2537783413 shutdown_connections 2024-03-28T22:37:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.399+0000 7f3b5a884740 1 -- 172.21.15.43:0/2537783413 wait complete. 2024-03-28T22:37:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.400+0000 7f3b55c00640 1 --2- 172.21.15.43:0/532468487 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f3b380a20b0 0x7f3b380a4570 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3b4802c6c0 tx=0x7f3b48009360 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:27.400 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:26 smithi172 ceph-mon[36566]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-03-28T22:37:27.404 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.400+0000 7f3b5a884740 1 -- start start 2024-03-28T22:37:27.404 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.400+0000 7f3b5a884740 1 Processor -- start 2024-03-28T22:37:27.404 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.400+0000 7f3b5a884740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5614d75623a0 0x5614d7568b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:27.404 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.401+0000 7f3b5a884740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5614d7562cc0 0x5614d7569050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:27.404 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.401+0000 7f3b5a884740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5614d756ba90 0x5614d7569590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:27.404 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.401+0000 7f3b5a884740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5614d756e6c0 con 0x5614d7562cc0 2024-03-28T22:37:27.404 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.401+0000 7f3b5a884740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5614d7463d00 con 0x5614d75623a0 2024-03-28T22:37:27.404 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.401+0000 7f3b5a884740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5614d756e840 con 0x5614d756ba90 2024-03-28T22:37:27.404 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.401+0000 7f3b55200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5614d75623a0 0x5614d7568b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:27.404 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.401+0000 7f3b5a884740 10 client.0 _ll_register_callbacks cb 0x5614d744e490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:37:27.404 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.401+0000 7f3b55200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5614d75623a0 0x5614d7568b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:36394/0 (socket says 172.21.15.43:36394) 2024-03-28T22:37:27.404 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.401+0000 7f3b55200640 1 -- 172.21.15.43:0/374220735 learned_addr learned my addr 172.21.15.43:0/374220735 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:27.405 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.401+0000 7f3b4fe00640 1 --2- 172.21.15.43:0/374220735 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5614d7562cc0 0x5614d7569050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:27.405 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.401+0000 7f3b55200640 1 -- 172.21.15.43:0/374220735 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5614d756ba90 msgr2=0x5614d7569590 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T22:37:27.405 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.401+0000 7f3b55200640 1 --2- 172.21.15.43:0/374220735 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5614d756ba90 0x5614d7569590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:27.405 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.401+0000 7f3b55200640 1 -- 172.21.15.43:0/374220735 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5614d7562cc0 msgr2=0x5614d7569050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:27.405 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.401+0000 7f3b55200640 1 --2- 172.21.15.43:0/374220735 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5614d7562cc0 0x5614d7569050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:27.405 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.401+0000 7f3b55200640 1 -- 172.21.15.43:0/374220735 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614d72a6620 con 0x5614d75623a0 2024-03-28T22:37:27.405 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.402+0000 7f3b55200640 1 --2- 172.21.15.43:0/374220735 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5614d75623a0 0x5614d7568b10 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3b44011cb0 tx=0x7f3b44004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a50150705e8d46d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:27.405 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.402+0000 7f3b36a00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:37:27.405 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.402+0000 7f3b36a00640 1 -- 172.21.15.43:0/374220735 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b44028020 con 0x5614d75623a0 2024-03-28T22:37:27.405 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.402+0000 7f3b36a00640 1 -- 172.21.15.43:0/374220735 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f3b4400fbf0 con 0x5614d75623a0 2024-03-28T22:37:27.405 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.402+0000 7f3b5a884740 1 -- 172.21.15.43:0/374220735 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5614d74d9030 con 0x5614d75623a0 2024-03-28T22:37:27.405 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.402+0000 7f3b36a00640 1 -- 172.21.15.43:0/374220735 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3b44033050 con 0x5614d75623a0 2024-03-28T22:37:27.405 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.402+0000 7f3b5a884740 10 client.25751 fetch_fsmap learned FSMap version 29 2024-03-28T22:37:27.406 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.402+0000 7f3b5a884740 1 -- 172.21.15.43:0/374220735 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5614d757a5e0 con 0x5614d75623a0 2024-03-28T22:37:27.406 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.402+0000 7f3b36a00640 1 -- 172.21.15.43:0/374220735 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5614d74d9030 con 0x5614d75623a0 2024-03-28T22:37:27.406 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.403+0000 7f3b36a00640 1 -- 172.21.15.43:0/374220735 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f3b44012570 con 0x5614d75623a0 2024-03-28T22:37:27.406 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.403+0000 7f3b5a884740 10 client.25751 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:37:27.406 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.403+0000 7f3b5a884740 10 client.25751 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:37:27.406 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.403+0000 7f3b5a884740 20 client.25751 populate_metadata read hostname 'smithi043' 2024-03-28T22:37:27.406 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.403+0000 7f3b5a884740 1 --2- 172.21.15.43:0/374220735 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5614d757ba40 0x5614d757de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:27.406 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.403+0000 7f3b5a884740 4 client.25751 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:37:27.406 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.403+0000 7f3b5a884740 1 -- 172.21.15.43:0/374220735 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5614d74d2b80 con 0x5614d757ba40 2024-03-28T22:37:27.406 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.403+0000 7f3b4fe00640 1 --2- 172.21.15.43:0/374220735 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5614d757ba40 0x5614d757de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:27.406 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.404+0000 7f3b4fe00640 1 --2- 172.21.15.43:0/374220735 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5614d757ba40 0x5614d757de30 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5085f1b6d169dd18 server_cookie=5dbc729f8722275a in_seq=0 out_seq=0 2024-03-28T22:37:27.406 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.404+0000 7f3b36a00640 10 client.25751 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:27.406 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.404+0000 7f3b36a00640 1 -- 172.21.15.43:0/374220735 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5614d74d2b80 con 0x5614d757ba40 2024-03-28T22:37:27.406 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.404+0000 7f3b36a00640 10 client.25751 handle_command_reply: tid=0 2024-03-28T22:37:27.407 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.404+0000 7f3b5a884740 1 client.25751 shutdown 2024-03-28T22:37:27.407 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.404+0000 7f3b5a884740 1 -- 172.21.15.43:0/374220735 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5614d75623a0 msgr2=0x5614d7568b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:27.407 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.404+0000 7f3b5a884740 1 --2- 172.21.15.43:0/374220735 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5614d75623a0 0x5614d7568b10 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3b44011cb0 tx=0x7f3b44004270 comp rx=0 tx=0).stop 2024-03-28T22:37:27.407 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.405+0000 7f3b5a884740 1 -- 172.21.15.43:0/374220735 shutdown_connections 2024-03-28T22:37:27.407 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.405+0000 7f3b5a884740 1 --2- 172.21.15.43:0/374220735 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5614d756ba90 0x5614d7569590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:27.407 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.405+0000 7f3b5a884740 1 --2- 172.21.15.43:0/374220735 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5614d7562cc0 0x5614d7569050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:27.407 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.405+0000 7f3b5a884740 1 --2- 172.21.15.43:0/374220735 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5614d757ba40 0x5614d757de30 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:27.407 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.405+0000 7f3b5a884740 1 --2- 172.21.15.43:0/374220735 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5614d75623a0 0x5614d7568b10 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:27.407 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.405+0000 7f3b5a884740 1 -- 172.21.15.43:0/374220735 >> 172.21.15.43:0/374220735 conn(0x5614d755ccc0 msgr2=0x5614d755ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:27.407 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.405+0000 7f3b36a00640 0 client.25751 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:27.407 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.405+0000 7f3b5a884740 1 -- 172.21.15.43:0/374220735 shutdown_connections 2024-03-28T22:37:27.407 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.405+0000 7f3b5a884740 1 -- 172.21.15.43:0/374220735 wait complete. 2024-03-28T22:37:27.407 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.405+0000 7f3b5a884740 20 client.25751 trim_cache size 0 max 16384 2024-03-28T22:37:27.419 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.421+0000 7f3b5a884740 1 Processor -- start 2024-03-28T22:37:27.419 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.421+0000 7f3b5a884740 1 -- start start 2024-03-28T22:37:27.420 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.421+0000 7f3b5a884740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5614d75623a0 0x5614d7577460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:27.420 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.421+0000 7f3b5a884740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5614d7562cc0 0x5614d756a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:27.420 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.421+0000 7f3b5a884740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5614d756ba90 0x5614d7583460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:27.420 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.421+0000 7f3b5a884740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3b501b51f0 con 0x5614d7562cc0 2024-03-28T22:37:27.421 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.421+0000 7f3b5a884740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f3b501b6d30 con 0x5614d756ba90 2024-03-28T22:37:27.421 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.421+0000 7f3b5a884740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f3b501b58c0 con 0x5614d75623a0 2024-03-28T22:37:27.421 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.422+0000 7f3b55c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5614d756ba90 0x5614d7583460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:27.421 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.422+0000 7f3b55c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5614d756ba90 0x5614d7583460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:36398/0 (socket says 172.21.15.43:36398) 2024-03-28T22:37:27.421 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.422+0000 7f3b55c00640 1 -- 172.21.15.43:0/3286780648 learned_addr learned my addr 172.21.15.43:0/3286780648 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:27.421 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.422+0000 7f3b55200640 1 --2- 172.21.15.43:0/3286780648 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5614d75623a0 0x5614d7577460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:27.421 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.422+0000 7f3b4fe00640 1 --2- 172.21.15.43:0/3286780648 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5614d7562cc0 0x5614d756a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:27.421 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.422+0000 7f3b55c00640 1 -- 172.21.15.43:0/3286780648 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5614d75623a0 msgr2=0x5614d7577460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:27.421 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.422+0000 7f3b55c00640 1 --2- 172.21.15.43:0/3286780648 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5614d75623a0 0x5614d7577460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:27.421 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.422+0000 7f3b55c00640 1 -- 172.21.15.43:0/3286780648 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5614d7562cc0 msgr2=0x5614d756a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:27.421 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.422+0000 7f3b55c00640 1 --2- 172.21.15.43:0/3286780648 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5614d7562cc0 0x5614d756a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:27.421 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.422+0000 7f3b55c00640 1 -- 172.21.15.43:0/3286780648 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614d75779a0 con 0x5614d756ba90 2024-03-28T22:37:27.421 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.422+0000 7f3b4fe00640 1 --2- 172.21.15.43:0/3286780648 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5614d7562cc0 0x5614d756a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:37:27.421 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.422+0000 7f3b55c00640 1 --2- 172.21.15.43:0/3286780648 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5614d756ba90 0x5614d7583460 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3b48009ca0 tx=0x7f3b4803e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=289161c31f0db847 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:27.421 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.423+0000 7f3b41600640 1 -- 172.21.15.43:0/3286780648 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b48044070 con 0x5614d756ba90 2024-03-28T22:37:27.422 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.423+0000 7f3b41600640 1 -- 172.21.15.43:0/3286780648 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b48050070 con 0x5614d756ba90 2024-03-28T22:37:27.422 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.423+0000 7f3b41600640 1 -- 172.21.15.43:0/3286780648 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b48039040 con 0x5614d756ba90 2024-03-28T22:37:27.425 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.423+0000 7f3b5a884740 1 -- 172.21.15.43:0/3286780648 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5614d756ba90 msgr2=0x5614d7583460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:27.425 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.423+0000 7f3b5a884740 1 --2- 172.21.15.43:0/3286780648 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5614d756ba90 0x5614d7583460 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3b48009ca0 tx=0x7f3b4803e000 comp rx=0 tx=0).stop 2024-03-28T22:37:27.425 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.423+0000 7f3b5a884740 1 -- 172.21.15.43:0/3286780648 shutdown_connections 2024-03-28T22:37:27.425 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.423+0000 7f3b5a884740 1 --2- 172.21.15.43:0/3286780648 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5614d756ba90 0x5614d7583460 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:27.425 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.423+0000 7f3b5a884740 1 --2- 172.21.15.43:0/3286780648 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5614d7562cc0 0x5614d756a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:27.425 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.423+0000 7f3b5a884740 1 --2- 172.21.15.43:0/3286780648 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5614d75623a0 0x5614d7577460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:27.425 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.424+0000 7f3b5a884740 1 -- 172.21.15.43:0/3286780648 >> 172.21.15.43:0/3286780648 conn(0x5614d755ccc0 msgr2=0x5614d756a230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:27.425 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.424+0000 7f3b5a884740 1 -- 172.21.15.43:0/3286780648 shutdown_connections 2024-03-28T22:37:27.425 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.424+0000 7f3b5a884740 1 -- 172.21.15.43:0/3286780648 wait complete. 2024-03-28T22:37:27.425 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.424+0000 7f3b5a884740 1 -- start start 2024-03-28T22:37:27.425 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.424+0000 7f3b5a884740 1 Processor -- start 2024-03-28T22:37:27.425 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.425+0000 7f3b5a884740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5614d75623a0 0x5614d7565f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:27.426 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.425+0000 7f3b5a884740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5614d7562cc0 0x5614d75664c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:27.426 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.425+0000 7f3b5a884740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5614d756ba90 0x5614d75daad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:27.426 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.425+0000 7f3b5a884740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3b501b6660 con 0x5614d75623a0 2024-03-28T22:37:27.426 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.425+0000 7f3b5a884740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f3b501b5f90 con 0x5614d7562cc0 2024-03-28T22:37:27.426 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.425+0000 7f3b5a884740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f3b501b31d0 con 0x5614d756ba90 2024-03-28T22:37:27.426 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.425+0000 7f3b5a884740 10 client.0 _ll_register_callbacks cb 0x7f3b44009ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:37:27.426 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.425+0000 7f3b55200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5614d75623a0 0x5614d7565f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:27.426 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.425+0000 7f3b55c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5614d756ba90 0x5614d75daad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:27.426 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.425+0000 7f3b4fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5614d7562cc0 0x5614d75664c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:27.426 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.425+0000 7f3b55c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5614d756ba90 0x5614d75daad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:52288/0 (socket says 172.21.15.43:52288) 2024-03-28T22:37:27.426 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.425+0000 7f3b55c00640 1 -- 172.21.15.43:0/4196826831 learned_addr learned my addr 172.21.15.43:0/4196826831 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:27.426 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.425+0000 7f3b55c00640 1 -- 172.21.15.43:0/4196826831 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5614d7562cc0 msgr2=0x5614d75664c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:27.427 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.425+0000 7f3b55c00640 1 --2- 172.21.15.43:0/4196826831 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5614d7562cc0 0x5614d75664c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:27.427 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.425+0000 7f3b55c00640 1 -- 172.21.15.43:0/4196826831 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5614d75623a0 msgr2=0x5614d7565f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:27.427 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.425+0000 7f3b55c00640 1 --2- 172.21.15.43:0/4196826831 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5614d75623a0 0x5614d7565f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:27.427 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.425+0000 7f3b55c00640 1 -- 172.21.15.43:0/4196826831 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614d75db110 con 0x5614d756ba90 2024-03-28T22:37:27.427 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.426+0000 7f3b55c00640 1 --2- 172.21.15.43:0/4196826831 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5614d756ba90 0x5614d75daad0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3b4802c790 tx=0x7f3b48003f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=824f42a63a068731 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:27.427 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.426+0000 7f3b36a00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:37:27.427 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.426+0000 7f3b36a00640 1 -- 172.21.15.43:0/4196826831 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b48058020 con 0x5614d756ba90 2024-03-28T22:37:27.427 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.426+0000 7f3b5a884740 1 -- 172.21.15.43:0/4196826831 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b48050070 con 0x5614d756ba90 2024-03-28T22:37:27.427 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.426+0000 7f3b36a00640 1 -- 172.21.15.43:0/4196826831 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f3b48004200 con 0x5614d756ba90 2024-03-28T22:37:27.427 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.426+0000 7f3b36a00640 1 -- 172.21.15.43:0/4196826831 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3b48063050 con 0x5614d756ba90 2024-03-28T22:37:27.427 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.426+0000 7f3b5a884740 10 client.25747 fetch_fsmap learned FSMap version 29 2024-03-28T22:37:27.427 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.426+0000 7f3b5a884740 1 -- 172.21.15.43:0/4196826831 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5614d75e2440 con 0x5614d756ba90 2024-03-28T22:37:27.427 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.427+0000 7f3b36a00640 1 -- 172.21.15.43:0/4196826831 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b48050070 con 0x5614d756ba90 2024-03-28T22:37:27.427 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.427+0000 7f3b36a00640 1 -- 172.21.15.43:0/4196826831 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f3b4804bc00 con 0x5614d756ba90 2024-03-28T22:37:27.427 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.427+0000 7f3b5a884740 10 client.25747 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:37:27.427 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.427+0000 7f3b5a884740 10 client.25747 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:37:27.428 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.427+0000 7f3b5a884740 20 client.25747 populate_metadata read hostname 'smithi043' 2024-03-28T22:37:27.428 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.427+0000 7f3b5a884740 1 --2- 172.21.15.43:0/4196826831 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5614d75e2b00 0x5614d75e4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:27.428 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.427+0000 7f3b5a884740 4 client.25747 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T22:37:27.428 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.427+0000 7f3b5a884740 1 -- 172.21.15.43:0/4196826831 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5614d74cced0 con 0x5614d75e2b00 2024-03-28T22:37:27.428 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.427+0000 7f3b55200640 1 --2- 172.21.15.43:0/4196826831 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5614d75e2b00 0x5614d75e4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:27.428 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.427+0000 7f3b36a00640 10 client.25747 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:27.428 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.427+0000 7f3b55200640 1 --2- 172.21.15.43:0/4196826831 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5614d75e2b00 0x5614d75e4f20 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fa138eb24bc5f44 server_cookie=da0804d820cfa038 in_seq=0 out_seq=0 2024-03-28T22:37:27.439 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.441+0000 7f3b36a00640 1 -- 172.21.15.43:0/4196826831 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5614d74cced0 con 0x5614d75e2b00 2024-03-28T22:37:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.441+0000 7f3b36a00640 10 client.25747 handle_command_reply: tid=0 2024-03-28T22:37:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.441+0000 7f3b5a884740 1 client.25747 shutdown 2024-03-28T22:37:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.441+0000 7f3b5a884740 1 -- 172.21.15.43:0/4196826831 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5614d756ba90 msgr2=0x5614d75daad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.441+0000 7f3b5a884740 1 --2- 172.21.15.43:0/4196826831 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5614d756ba90 0x5614d75daad0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3b4802c790 tx=0x7f3b48003f30 comp rx=0 tx=0).stop 2024-03-28T22:37:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.441+0000 7f3b5a884740 1 -- 172.21.15.43:0/4196826831 shutdown_connections 2024-03-28T22:37:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.441+0000 7f3b5a884740 1 --2- 172.21.15.43:0/4196826831 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5614d75e2b00 0x5614d75e4f20 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.441+0000 7f3b5a884740 1 --2- 172.21.15.43:0/4196826831 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5614d7562cc0 0x5614d75664c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.441+0000 7f3b36a00640 0 client.25747 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.441+0000 7f3b5a884740 1 --2- 172.21.15.43:0/4196826831 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5614d756ba90 0x5614d75daad0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.441+0000 7f3b5a884740 1 --2- 172.21.15.43:0/4196826831 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5614d75623a0 0x5614d7565f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.441+0000 7f3b5a884740 1 -- 172.21.15.43:0/4196826831 >> 172.21.15.43:0/4196826831 conn(0x5614d755ccc0 msgr2=0x5614d755bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.441+0000 7f3b5a884740 1 -- 172.21.15.43:0/4196826831 shutdown_connections 2024-03-28T22:37:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.441+0000 7f3b5a884740 1 -- 172.21.15.43:0/4196826831 wait complete. 2024-03-28T22:37:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.441+0000 7f3b5a884740 20 client.25747 trim_cache size 0 max 16384 2024-03-28T22:37:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:{ 2024-03-28T22:37:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "message": "", 2024-03-28T22:37:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "return_code": 0 2024-03-28T22:37:27.441 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:} 2024-03-28T22:37:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.442+0000 7f3b57c00640 1 -- 172.21.15.43:0/532468487 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f3b380a20b0 msgr2=0x7f3b380a4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.442+0000 7f3b57c00640 1 --2- 172.21.15.43:0/532468487 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f3b380a20b0 0x7f3b380a4570 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3b4802c6c0 tx=0x7f3b48009360 comp rx=0 tx=0).stop 2024-03-28T22:37:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.442+0000 7f3b57c00640 1 -- 172.21.15.43:0/532468487 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3b50074410 msgr2=0x7f3b5019ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.442+0000 7f3b57c00640 1 --2- 172.21.15.43:0/532468487 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3b50074410 0x7f3b5019ea60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c00ea60 tx=0x7f3b3c00ef20 comp rx=0 tx=0).stop 2024-03-28T22:37:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.442+0000 7f3b57c00640 1 -- 172.21.15.43:0/532468487 shutdown_connections 2024-03-28T22:37:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.442+0000 7f3b57c00640 1 --2- 172.21.15.43:0/532468487 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f3b380a20b0 0x7f3b380a4570 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.442+0000 7f3b57c00640 1 --2- 172.21.15.43:0/532468487 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3b50074410 0x7f3b5019ea60 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.442+0000 7f3b57c00640 1 --2- 172.21.15.43:0/532468487 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3b50112700 0x7f3b501a2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.442+0000 7f3b57c00640 1 --2- 172.21.15.43:0/532468487 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3b50073af0 0x7f3b5019e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.442+0000 7f3b57c00640 1 -- 172.21.15.43:0/532468487 >> 172.21.15.43:0/532468487 conn(0x7f3b500fd1e0 msgr2=0x7f3b500fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.443+0000 7f3b57c00640 1 -- 172.21.15.43:0/532468487 shutdown_connections 2024-03-28T22:37:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:27.443+0000 7f3b57c00640 1 -- 172.21.15.43:0/532468487 wait complete. 2024-03-28T22:37:29.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:28 smithi082 ceph-mon[36471]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 77 KiB/s wr, 53 op/s 2024-03-28T22:37:29.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:28 smithi043 ceph-mon[31823]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 77 KiB/s wr, 53 op/s 2024-03-28T22:37:29.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:28 smithi172 ceph-mon[36566]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 77 KiB/s wr, 53 op/s 2024-03-28T22:37:31.127 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:37:31.127 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:37:31.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:30 smithi082 ceph-mon[36471]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 38 KiB/s wr, 50 op/s 2024-03-28T22:37:31.222 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:30 smithi043 ceph-mon[31823]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 38 KiB/s wr, 50 op/s 2024-03-28T22:37:31.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.376+0000 7f2a62800640 1 -- 172.21.15.43:0/625787963 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a5c106590 msgr2=0x7f2a5c106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:31.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.376+0000 7f2a62800640 1 --2- 172.21.15.43:0/625787963 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a5c106590 0x7f2a5c106970 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2a50009930 tx=0x7f2a5002bc30 comp rx=0 tx=0).stop 2024-03-28T22:37:31.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.376+0000 7f2a62800640 1 -- 172.21.15.43:0/625787963 shutdown_connections 2024-03-28T22:37:31.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.376+0000 7f2a62800640 1 --2- 172.21.15.43:0/625787963 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a5c1004d0 0x7f2a5c100950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.376+0000 7f2a62800640 1 --2- 172.21.15.43:0/625787963 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a5c100e90 0x7f2a5c10d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.376+0000 7f2a62800640 1 --2- 172.21.15.43:0/625787963 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a5c106590 0x7f2a5c106970 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.376+0000 7f2a62800640 1 -- 172.21.15.43:0/625787963 >> 172.21.15.43:0/625787963 conn(0x7f2a5c0fbfa0 msgr2=0x7f2a5c0fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:31.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.376+0000 7f2a62800640 1 -- 172.21.15.43:0/625787963 shutdown_connections 2024-03-28T22:37:31.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.376+0000 7f2a62800640 1 -- 172.21.15.43:0/625787963 wait complete. 2024-03-28T22:37:31.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.377+0000 7f2a62800640 1 Processor -- start 2024-03-28T22:37:31.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.378+0000 7f2a62800640 1 -- start start 2024-03-28T22:37:31.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.378+0000 7f2a62800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a5c1004d0 0x7f2a5c102140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:31.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.378+0000 7f2a62800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a5c100e90 0x7f2a5c102680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:31.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.378+0000 7f2a62800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a5c106590 0x7f2a5c102bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:31.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.378+0000 7f2a62800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5c1108a0 con 0x7f2a5c1004d0 2024-03-28T22:37:31.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.378+0000 7f2a62800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5c110720 con 0x7f2a5c100e90 2024-03-28T22:37:31.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.378+0000 7f2a62800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5c110a20 con 0x7f2a5c106590 2024-03-28T22:37:31.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.378+0000 7f2a61400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a5c1004d0 0x7f2a5c102140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:31.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.378+0000 7f2a60a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a5c100e90 0x7f2a5c102680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:31.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.379+0000 7f2a61400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a5c1004d0 0x7f2a5c102140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46820/0 (socket says 172.21.15.43:46820) 2024-03-28T22:37:31.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.379+0000 7f2a61e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a5c106590 0x7f2a5c102bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:31.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.379+0000 7f2a61400640 1 -- 172.21.15.43:0/4026009145 learned_addr learned my addr 172.21.15.43:0/4026009145 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:31.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.379+0000 7f2a60a00640 1 -- 172.21.15.43:0/4026009145 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a5c106590 msgr2=0x7f2a5c102bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:31.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.379+0000 7f2a60a00640 1 --2- 172.21.15.43:0/4026009145 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a5c106590 0x7f2a5c102bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.379+0000 7f2a60a00640 1 -- 172.21.15.43:0/4026009145 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a5c1004d0 msgr2=0x7f2a5c102140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:31.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.379+0000 7f2a60a00640 1 --2- 172.21.15.43:0/4026009145 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a5c1004d0 0x7f2a5c102140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.379+0000 7f2a60a00640 1 -- 172.21.15.43:0/4026009145 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a5c1a6330 con 0x7f2a5c100e90 2024-03-28T22:37:31.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.379+0000 7f2a61400640 1 --2- 172.21.15.43:0/4026009145 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a5c1004d0 0x7f2a5c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:37:31.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.379+0000 7f2a60a00640 1 --2- 172.21.15.43:0/4026009145 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a5c100e90 0x7f2a5c102680 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2a4800e9f0 tx=0x7f2a4800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:31.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.380+0000 7f2a5a000640 1 -- 172.21.15.43:0/4026009145 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a4800cd80 con 0x7f2a5c100e90 2024-03-28T22:37:31.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.380+0000 7f2a5a000640 1 -- 172.21.15.43:0/4026009145 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a480035f0 con 0x7f2a5c100e90 2024-03-28T22:37:31.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.380+0000 7f2a5a000640 1 -- 172.21.15.43:0/4026009145 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a48010690 con 0x7f2a5c100e90 2024-03-28T22:37:31.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.380+0000 7f2a62800640 1 -- 172.21.15.43:0/4026009145 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a5c1a6620 con 0x7f2a5c100e90 2024-03-28T22:37:31.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.380+0000 7f2a62800640 1 -- 172.21.15.43:0/4026009145 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a5c1a6b20 con 0x7f2a5c100e90 2024-03-28T22:37:31.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.383+0000 7f2a66a84740 1 Processor -- start 2024-03-28T22:37:31.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.383+0000 7f2a66a84740 1 -- start start 2024-03-28T22:37:31.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.383+0000 7f2a5a000640 1 -- 172.21.15.43:0/4026009145 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 26) ==== 136932+0+0 (secure 0 0 0) 0x7f2a48003b30 con 0x7f2a5c100e90 2024-03-28T22:37:31.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.383+0000 7f2a66a84740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fcb84b2db0 0x55fcb84b1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:31.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.384+0000 7f2a66a84740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fcb84b20d0 0x55fcb84b2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:31.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.384+0000 7f2a66a84740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fcb84bb440 0x55fcb84bd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:31.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.384+0000 7f2a66a84740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55fcb8416100 con 0x55fcb84b20d0 2024-03-28T22:37:31.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.384+0000 7f2a66a84740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55fcb83beba0 con 0x55fcb84b2db0 2024-03-28T22:37:31.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.384+0000 7f2a66a84740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55fcb84121a0 con 0x55fcb84bb440 2024-03-28T22:37:31.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.384+0000 7f2a61e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fcb84b2db0 0x55fcb84b1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:31.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.384+0000 7f2a61e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fcb84b2db0 0x55fcb84b1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:36446/0 (socket says 172.21.15.43:36446) 2024-03-28T22:37:31.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.384+0000 7f2a61e00640 1 -- 172.21.15.43:0/780891435 learned_addr learned my addr 172.21.15.43:0/780891435 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:31.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.384+0000 7f2a61400640 1 --2- 172.21.15.43:0/780891435 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fcb84b20d0 0x55fcb84b2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:31.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.384+0000 7f2a60a00640 1 --2- 172.21.15.43:0/780891435 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fcb84bb440 0x55fcb84bd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:31.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.384+0000 7f2a61e00640 1 -- 172.21.15.43:0/780891435 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fcb84bb440 msgr2=0x55fcb84bd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:31.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.384+0000 7f2a61e00640 1 --2- 172.21.15.43:0/780891435 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fcb84bb440 0x55fcb84bd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.384+0000 7f2a61e00640 1 -- 172.21.15.43:0/780891435 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fcb84b20d0 msgr2=0x55fcb84b2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:31.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.384+0000 7f2a61e00640 1 --2- 172.21.15.43:0/780891435 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fcb84b20d0 0x55fcb84b2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.384+0000 7f2a61e00640 1 -- 172.21.15.43:0/780891435 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcb83ef1a0 con 0x55fcb84b2db0 2024-03-28T22:37:31.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.384+0000 7f2a61400640 1 --2- 172.21.15.43:0/780891435 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fcb84b20d0 0x55fcb84b2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:37:31.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.384+0000 7f2a61e00640 1 --2- 172.21.15.43:0/780891435 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fcb84b2db0 0x55fcb84b1b00 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2a54009f70 tx=0x7f2a54010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68fe684808e8b0c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:31.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.385+0000 7f2a4e600640 1 -- 172.21.15.43:0/780891435 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a54016070 con 0x55fcb84b2db0 2024-03-28T22:37:31.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.385+0000 7f2a4e600640 1 -- 172.21.15.43:0/780891435 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a54004d20 con 0x55fcb84b2db0 2024-03-28T22:37:31.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.385+0000 7f2a4e600640 1 -- 172.21.15.43:0/780891435 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a54012950 con 0x55fcb84b2db0 2024-03-28T22:37:31.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.385+0000 7f2a5a000640 1 --2- 172.21.15.43:0/4026009145 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f2a440a2000 0x7f2a440a44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:31.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.385+0000 7f2a66a84740 1 -- 172.21.15.43:0/780891435 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fcb84b2db0 msgr2=0x55fcb84b1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:31.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.385+0000 7f2a66a84740 1 --2- 172.21.15.43:0/780891435 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fcb84b2db0 0x55fcb84b1b00 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2a54009f70 tx=0x7f2a54010800 comp rx=0 tx=0).stop 2024-03-28T22:37:31.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.385+0000 7f2a5a000640 1 -- 172.21.15.43:0/4026009145 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f2a480ce980 con 0x7f2a5c100e90 2024-03-28T22:37:31.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.385+0000 7f2a66a84740 1 -- 172.21.15.43:0/780891435 shutdown_connections 2024-03-28T22:37:31.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.385+0000 7f2a66a84740 1 --2- 172.21.15.43:0/780891435 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fcb84bb440 0x55fcb84bd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.385+0000 7f2a66a84740 1 --2- 172.21.15.43:0/780891435 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fcb84b20d0 0x55fcb84b2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.385+0000 7f2a66a84740 1 --2- 172.21.15.43:0/780891435 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fcb84b2db0 0x55fcb84b1b00 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.385+0000 7f2a66a84740 1 -- 172.21.15.43:0/780891435 >> 172.21.15.43:0/780891435 conn(0x55fcb8478a10 msgr2=0x55fcb84acb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:31.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.386+0000 7f2a61400640 1 --2- 172.21.15.43:0/4026009145 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f2a440a2000 0x7f2a440a44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:31.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.386+0000 7f2a66a84740 1 -- 172.21.15.43:0/780891435 shutdown_connections 2024-03-28T22:37:31.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.386+0000 7f2a66a84740 1 -- 172.21.15.43:0/780891435 wait complete. 2024-03-28T22:37:31.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.386+0000 7f2a66a84740 1 -- start start 2024-03-28T22:37:31.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.386+0000 7f2a61400640 1 --2- 172.21.15.43:0/4026009145 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f2a440a2000 0x7f2a440a44c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x55fcb84b3300 tx=0x7f2a5002df90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:31.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.387+0000 7f2a66a84740 1 Processor -- start 2024-03-28T22:37:31.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.387+0000 7f2a66a84740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fcb84b20d0 0x55fcb84b8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:31.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.387+0000 7f2a66a84740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fcb84b2db0 0x55fcb84b8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:31.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.387+0000 7f2a66a84740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fcb84bb440 0x55fcb84b8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:31.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.387+0000 7f2a66a84740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55fcb84be2b0 con 0x55fcb84bb440 2024-03-28T22:37:31.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.387+0000 7f2a66a84740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55fcb83b4250 con 0x55fcb84b20d0 2024-03-28T22:37:31.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.387+0000 7f2a66a84740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55fcb84be430 con 0x55fcb84b2db0 2024-03-28T22:37:31.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.387+0000 7f2a66a84740 10 client.0 _ll_register_callbacks cb 0x55fcb8129be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:37:31.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.387+0000 7f2a61400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fcb84b20d0 0x55fcb84b8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:31.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.387+0000 7f2a61e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fcb84bb440 0x55fcb84b8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:31.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.387+0000 7f2a61400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fcb84b20d0 0x55fcb84b8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:36460/0 (socket says 172.21.15.43:36460) 2024-03-28T22:37:31.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.388+0000 7f2a61400640 1 -- 172.21.15.43:0/3359717200 learned_addr learned my addr 172.21.15.43:0/3359717200 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:31.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.388+0000 7f2a61400640 1 -- 172.21.15.43:0/3359717200 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fcb84b2db0 msgr2=0x55fcb84b8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:31.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.388+0000 7f2a61400640 1 --2- 172.21.15.43:0/3359717200 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fcb84b2db0 0x55fcb84b8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.388+0000 7f2a61400640 1 -- 172.21.15.43:0/3359717200 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fcb84bb440 msgr2=0x55fcb84b8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:31.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.388+0000 7f2a61400640 1 --2- 172.21.15.43:0/3359717200 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fcb84bb440 0x55fcb84b8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.388+0000 7f2a61400640 1 -- 172.21.15.43:0/3359717200 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcb83d0960 con 0x55fcb84b20d0 2024-03-28T22:37:31.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.388+0000 7f2a61e00640 1 --2- 172.21.15.43:0/3359717200 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fcb84bb440 0x55fcb84b8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:37:31.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.388+0000 7f2a61400640 1 --2- 172.21.15.43:0/3359717200 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fcb84b20d0 0x55fcb84b8430 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2a50004470 tx=0x7f2a50009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51f550b6b9385491 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:31.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.388+0000 7f2a42a00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:37:31.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.389+0000 7f2a42a00640 1 -- 172.21.15.43:0/3359717200 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a50045070 con 0x55fcb84b20d0 2024-03-28T22:37:31.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.389+0000 7f2a42a00640 1 -- 172.21.15.43:0/3359717200 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f2a5002cb00 con 0x55fcb84b20d0 2024-03-28T22:37:31.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.389+0000 7f2a66a84740 1 -- 172.21.15.43:0/3359717200 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a54004d20 con 0x55fcb84b20d0 2024-03-28T22:37:31.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.389+0000 7f2a42a00640 1 -- 172.21.15.43:0/3359717200 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2a5002cd00 con 0x55fcb84b20d0 2024-03-28T22:37:31.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.389+0000 7f2a66a84740 10 client.25769 fetch_fsmap learned FSMap version 29 2024-03-28T22:37:31.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.389+0000 7f2a66a84740 1 -- 172.21.15.43:0/3359717200 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fcb81f47d0 con 0x55fcb84b20d0 2024-03-28T22:37:31.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.389+0000 7f2a42a00640 1 -- 172.21.15.43:0/3359717200 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a54004d20 con 0x55fcb84b20d0 2024-03-28T22:37:31.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.389+0000 7f2a42a00640 1 -- 172.21.15.43:0/3359717200 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f2a500039c0 con 0x55fcb84b20d0 2024-03-28T22:37:31.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.389+0000 7f2a66a84740 10 client.25769 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:37:31.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.389+0000 7f2a66a84740 10 client.25769 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:37:31.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.389+0000 7f2a66a84740 20 client.25769 populate_metadata read hostname 'smithi043' 2024-03-28T22:37:31.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.389+0000 7f2a66a84740 1 --2- 172.21.15.43:0/3359717200 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55fcb84cb950 0x55fcb84cde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:31.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.390+0000 7f2a66a84740 4 client.25769 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:37:31.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.390+0000 7f2a66a84740 1 -- 172.21.15.43:0/3359717200 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fcb8423640 con 0x55fcb84cb950 2024-03-28T22:37:31.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.390+0000 7f2a60a00640 1 --2- 172.21.15.43:0/3359717200 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55fcb84cb950 0x55fcb84cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:31.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.390+0000 7f2a60a00640 1 --2- 172.21.15.43:0/3359717200 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55fcb84cb950 0x55fcb84cde10 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c950ee0c9b38941a server_cookie=6326534945b41c0c in_seq=0 out_seq=0 2024-03-28T22:37:31.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.390+0000 7f2a42a00640 10 client.25769 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:31.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.391+0000 7f2a42a00640 1 -- 172.21.15.43:0/3359717200 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fcb8423640 con 0x55fcb84cb950 2024-03-28T22:37:31.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.391+0000 7f2a42a00640 10 client.25769 handle_command_reply: tid=0 2024-03-28T22:37:31.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.391+0000 7f2a66a84740 1 client.25769 shutdown 2024-03-28T22:37:31.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.391+0000 7f2a66a84740 1 -- 172.21.15.43:0/3359717200 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fcb84b20d0 msgr2=0x55fcb84b8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:31.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.392+0000 7f2a66a84740 1 --2- 172.21.15.43:0/3359717200 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fcb84b20d0 0x55fcb84b8430 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2a50004470 tx=0x7f2a50009260 comp rx=0 tx=0).stop 2024-03-28T22:37:31.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.392+0000 7f2a66a84740 1 -- 172.21.15.43:0/3359717200 shutdown_connections 2024-03-28T22:37:31.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.392+0000 7f2a66a84740 1 --2- 172.21.15.43:0/3359717200 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fcb84bb440 0x55fcb84b8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.392+0000 7f2a66a84740 1 --2- 172.21.15.43:0/3359717200 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fcb84b2db0 0x55fcb84b8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.392+0000 7f2a66a84740 1 --2- 172.21.15.43:0/3359717200 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55fcb84cb950 0x55fcb84cde10 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.392+0000 7f2a66a84740 1 --2- 172.21.15.43:0/3359717200 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fcb84b20d0 0x55fcb84b8430 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.392+0000 7f2a42a00640 0 client.25769 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:31.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.392+0000 7f2a66a84740 1 -- 172.21.15.43:0/3359717200 >> 172.21.15.43:0/3359717200 conn(0x55fcb8478a10 msgr2=0x55fcb84ae870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:31.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.392+0000 7f2a66a84740 1 -- 172.21.15.43:0/3359717200 shutdown_connections 2024-03-28T22:37:31.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.392+0000 7f2a66a84740 1 -- 172.21.15.43:0/3359717200 wait complete. 2024-03-28T22:37:31.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.392+0000 7f2a66a84740 20 client.25769 trim_cache size 0 max 16384 2024-03-28T22:37:31.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:30 smithi172 ceph-mon[36566]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 38 KiB/s wr, 50 op/s 2024-03-28T22:37:31.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.411+0000 7f2a66a84740 1 Processor -- start 2024-03-28T22:37:31.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.411+0000 7f2a66a84740 1 -- start start 2024-03-28T22:37:31.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.411+0000 7f2a66a84740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fcb84b20d0 0x55fcb84abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:31.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.411+0000 7f2a66a84740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fcb84b2db0 0x55fcb84b7d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:31.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.411+0000 7f2a66a84740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fcb84bb440 0x55fcb84c70c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:31.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.411+0000 7f2a66a84740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5c1b0c70 con 0x55fcb84bb440 2024-03-28T22:37:31.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.411+0000 7f2a66a84740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5c1b27b0 con 0x55fcb84b2db0 2024-03-28T22:37:31.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.411+0000 7f2a66a84740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5c1b1340 con 0x55fcb84b20d0 2024-03-28T22:37:31.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.412+0000 7f2a61e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fcb84bb440 0x55fcb84c70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:31.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.412+0000 7f2a61e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fcb84bb440 0x55fcb84c70c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46846/0 (socket says 172.21.15.43:46846) 2024-03-28T22:37:31.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.412+0000 7f2a61e00640 1 -- 172.21.15.43:0/264954222 learned_addr learned my addr 172.21.15.43:0/264954222 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:31.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.412+0000 7f2a60a00640 1 --2- 172.21.15.43:0/264954222 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fcb84b2db0 0x55fcb84b7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:31.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.412+0000 7f2a61400640 1 --2- 172.21.15.43:0/264954222 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fcb84b20d0 0x55fcb84abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:31.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.412+0000 7f2a60a00640 1 -- 172.21.15.43:0/264954222 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fcb84b20d0 msgr2=0x55fcb84abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:31.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.412+0000 7f2a60a00640 1 --2- 172.21.15.43:0/264954222 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fcb84b20d0 0x55fcb84abd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.412+0000 7f2a60a00640 1 -- 172.21.15.43:0/264954222 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fcb84bb440 msgr2=0x55fcb84c70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:31.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.412+0000 7f2a60a00640 1 --2- 172.21.15.43:0/264954222 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fcb84bb440 0x55fcb84c70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.412+0000 7f2a60a00640 1 -- 172.21.15.43:0/264954222 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcb812a3f0 con 0x55fcb84b2db0 2024-03-28T22:37:31.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.412+0000 7f2a61e00640 1 --2- 172.21.15.43:0/264954222 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fcb84bb440 0x55fcb84c70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:37:31.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.413+0000 7f2a60a00640 1 --2- 172.21.15.43:0/264954222 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fcb84b2db0 0x55fcb84b7d10 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2a480cbc10 tx=0x7f2a480d0dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75fced4f888e08c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:31.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.413+0000 7f2a4dc00640 1 -- 172.21.15.43:0/264954222 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a4809ef50 con 0x55fcb84b2db0 2024-03-28T22:37:31.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.413+0000 7f2a4dc00640 1 -- 172.21.15.43:0/264954222 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a480aee50 con 0x55fcb84b2db0 2024-03-28T22:37:31.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.413+0000 7f2a4dc00640 1 -- 172.21.15.43:0/264954222 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a48097470 con 0x55fcb84b2db0 2024-03-28T22:37:31.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.413+0000 7f2a66a84740 1 -- 172.21.15.43:0/264954222 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fcb84b2db0 msgr2=0x55fcb84b7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:31.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.413+0000 7f2a66a84740 1 --2- 172.21.15.43:0/264954222 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fcb84b2db0 0x55fcb84b7d10 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2a480cbc10 tx=0x7f2a480d0dd0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.413+0000 7f2a66a84740 1 -- 172.21.15.43:0/264954222 shutdown_connections 2024-03-28T22:37:31.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.413+0000 7f2a66a84740 1 --2- 172.21.15.43:0/264954222 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fcb84b2db0 0x55fcb84b7d10 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.413+0000 7f2a66a84740 1 --2- 172.21.15.43:0/264954222 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fcb84bb440 0x55fcb84c70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.413+0000 7f2a66a84740 1 --2- 172.21.15.43:0/264954222 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fcb84b20d0 0x55fcb84abd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.413+0000 7f2a66a84740 1 -- 172.21.15.43:0/264954222 >> 172.21.15.43:0/264954222 conn(0x55fcb8478a10 msgr2=0x55fcb84adf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:31.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.414+0000 7f2a66a84740 1 -- 172.21.15.43:0/264954222 shutdown_connections 2024-03-28T22:37:31.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.414+0000 7f2a66a84740 1 -- 172.21.15.43:0/264954222 wait complete. 2024-03-28T22:37:31.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.414+0000 7f2a66a84740 1 -- start start 2024-03-28T22:37:31.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.414+0000 7f2a66a84740 1 Processor -- start 2024-03-28T22:37:31.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.415+0000 7f2a66a84740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fcb84b20d0 0x55fcb84c3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:31.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.415+0000 7f2a66a84740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fcb84b2db0 0x55fcb84b56b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:31.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.415+0000 7f2a66a84740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fcb84bb440 0x55fcb84b5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:31.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.415+0000 7f2a66a84740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5c1b20e0 con 0x55fcb84b2db0 2024-03-28T22:37:31.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.415+0000 7f2a66a84740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5c1b1a10 con 0x55fcb84b20d0 2024-03-28T22:37:31.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.415+0000 7f2a66a84740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5c1aec50 con 0x55fcb84bb440 2024-03-28T22:37:31.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.415+0000 7f2a66a84740 10 client.0 _ll_register_callbacks cb 0x7f2a50009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:37:31.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.415+0000 7f2a60a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fcb84b2db0 0x55fcb84b56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:31.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.415+0000 7f2a61e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fcb84bb440 0x55fcb84b5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:31.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.415+0000 7f2a61400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fcb84b20d0 0x55fcb84c3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:31.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.415+0000 7f2a60a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fcb84b2db0 0x55fcb84b56b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46848/0 (socket says 172.21.15.43:46848) 2024-03-28T22:37:31.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.415+0000 7f2a60a00640 1 -- 172.21.15.43:0/1501149414 learned_addr learned my addr 172.21.15.43:0/1501149414 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:31.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.415+0000 7f2a61400640 1 -- 172.21.15.43:0/1501149414 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fcb84bb440 msgr2=0x55fcb84b5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:31.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.415+0000 7f2a61400640 1 --2- 172.21.15.43:0/1501149414 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fcb84bb440 0x55fcb84b5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.415+0000 7f2a61400640 1 -- 172.21.15.43:0/1501149414 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fcb84b2db0 msgr2=0x55fcb84b56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:31.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.415+0000 7f2a61400640 1 --2- 172.21.15.43:0/1501149414 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fcb84b2db0 0x55fcb84b56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.415+0000 7f2a61400640 1 -- 172.21.15.43:0/1501149414 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcb80ea0f0 con 0x55fcb84b20d0 2024-03-28T22:37:31.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.415+0000 7f2a60a00640 1 --2- 172.21.15.43:0/1501149414 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fcb84b2db0 0x55fcb84b56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:37:31.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.416+0000 7f2a61400640 1 --2- 172.21.15.43:0/1501149414 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fcb84b20d0 0x55fcb84c3a00 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f2a50004440 tx=0x7f2a5002c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c45d120ce326f705 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:31.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.416+0000 7f2a42a00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:37:31.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.416+0000 7f2a42a00640 1 -- 172.21.15.43:0/1501149414 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a50003c30 con 0x55fcb84b20d0 2024-03-28T22:37:31.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.416+0000 7f2a42a00640 1 -- 172.21.15.43:0/1501149414 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f2a50004f20 con 0x55fcb84b20d0 2024-03-28T22:37:31.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.416+0000 7f2a66a84740 1 -- 172.21.15.43:0/1501149414 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a480aee50 con 0x55fcb84b20d0 2024-03-28T22:37:31.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.416+0000 7f2a42a00640 1 -- 172.21.15.43:0/1501149414 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2a50005120 con 0x55fcb84b20d0 2024-03-28T22:37:31.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.416+0000 7f2a66a84740 10 client.25775 fetch_fsmap learned FSMap version 29 2024-03-28T22:37:31.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.416+0000 7f2a66a84740 1 -- 172.21.15.43:0/1501149414 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fcb8531d10 con 0x55fcb84b20d0 2024-03-28T22:37:31.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.416+0000 7f2a42a00640 1 -- 172.21.15.43:0/1501149414 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a480aee50 con 0x55fcb84b20d0 2024-03-28T22:37:31.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.417+0000 7f2a42a00640 1 -- 172.21.15.43:0/1501149414 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f2a50034aa0 con 0x55fcb84b20d0 2024-03-28T22:37:31.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.417+0000 7f2a66a84740 10 client.25775 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:37:31.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.417+0000 7f2a66a84740 10 client.25775 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:37:31.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.417+0000 7f2a66a84740 20 client.25775 populate_metadata read hostname 'smithi043' 2024-03-28T22:37:31.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.417+0000 7f2a66a84740 1 --2- 172.21.15.43:0/1501149414 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55fcb8532050 0x55fcb8534470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:31.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.417+0000 7f2a66a84740 4 client.25775 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:37:31.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.417+0000 7f2a66a84740 1 -- 172.21.15.43:0/1501149414 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fcb841d990 con 0x55fcb8532050 2024-03-28T22:37:31.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.417+0000 7f2a60a00640 1 --2- 172.21.15.43:0/1501149414 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55fcb8532050 0x55fcb8534470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:31.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.417+0000 7f2a60a00640 1 --2- 172.21.15.43:0/1501149414 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55fcb8532050 0x55fcb8534470 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f172e1791499e1c7 server_cookie=f156da2034bd36f0 in_seq=0 out_seq=0 2024-03-28T22:37:31.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.417+0000 7f2a42a00640 10 client.25775 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:31.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.418+0000 7f2a42a00640 1 -- 172.21.15.43:0/1501149414 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55fcb841d990 con 0x55fcb8532050 2024-03-28T22:37:31.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.418+0000 7f2a42a00640 10 client.25775 handle_command_reply: tid=0 2024-03-28T22:37:31.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.418+0000 7f2a66a84740 1 client.25775 shutdown 2024-03-28T22:37:31.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.418+0000 7f2a66a84740 1 -- 172.21.15.43:0/1501149414 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fcb84b20d0 msgr2=0x55fcb84c3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:31.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.418+0000 7f2a66a84740 1 --2- 172.21.15.43:0/1501149414 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fcb84b20d0 0x55fcb84c3a00 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f2a50004440 tx=0x7f2a5002c7e0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.418+0000 7f2a66a84740 1 -- 172.21.15.43:0/1501149414 shutdown_connections 2024-03-28T22:37:31.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.418+0000 7f2a66a84740 1 --2- 172.21.15.43:0/1501149414 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fcb84bb440 0x55fcb84b5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.418+0000 7f2a66a84740 1 --2- 172.21.15.43:0/1501149414 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fcb84b2db0 0x55fcb84b56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.418+0000 7f2a66a84740 1 --2- 172.21.15.43:0/1501149414 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55fcb8532050 0x55fcb8534470 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.418+0000 7f2a66a84740 1 --2- 172.21.15.43:0/1501149414 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fcb84b20d0 0x55fcb84c3a00 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.418+0000 7f2a66a84740 1 -- 172.21.15.43:0/1501149414 >> 172.21.15.43:0/1501149414 conn(0x55fcb8478a10 msgr2=0x55fcb84abb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:31.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.418+0000 7f2a42a00640 0 client.25775 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:31.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.418+0000 7f2a66a84740 1 -- 172.21.15.43:0/1501149414 shutdown_connections 2024-03-28T22:37:31.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.418+0000 7f2a66a84740 1 -- 172.21.15.43:0/1501149414 wait complete. 2024-03-28T22:37:31.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.418+0000 7f2a66a84740 20 client.25775 trim_cache size 0 max 16384 2024-03-28T22:37:31.421 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:37:31.421 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-03-28T22:37:31.421 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-03-28T22:37:31.421 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:37:31.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.419+0000 7f2a62800640 1 -- 172.21.15.43:0/4026009145 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f2a440a2000 msgr2=0x7f2a440a44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:31.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.419+0000 7f2a62800640 1 --2- 172.21.15.43:0/4026009145 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f2a440a2000 0x7f2a440a44c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x55fcb84b3300 tx=0x7f2a5002df90 comp rx=0 tx=0).stop 2024-03-28T22:37:31.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.419+0000 7f2a62800640 1 -- 172.21.15.43:0/4026009145 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a5c100e90 msgr2=0x7f2a5c102680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:31.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.419+0000 7f2a62800640 1 --2- 172.21.15.43:0/4026009145 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a5c100e90 0x7f2a5c102680 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2a4800e9f0 tx=0x7f2a4800eeb0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.419+0000 7f2a62800640 1 -- 172.21.15.43:0/4026009145 shutdown_connections 2024-03-28T22:37:31.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.419+0000 7f2a62800640 1 --2- 172.21.15.43:0/4026009145 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f2a440a2000 0x7f2a440a44c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.419+0000 7f2a62800640 1 --2- 172.21.15.43:0/4026009145 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a5c100e90 0x7f2a5c102680 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.419+0000 7f2a62800640 1 --2- 172.21.15.43:0/4026009145 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2a5c106590 0x7f2a5c102bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.419+0000 7f2a62800640 1 --2- 172.21.15.43:0/4026009145 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2a5c1004d0 0x7f2a5c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.419+0000 7f2a62800640 1 -- 172.21.15.43:0/4026009145 >> 172.21.15.43:0/4026009145 conn(0x7f2a5c0fbfa0 msgr2=0x7f2a5c0fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:31.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.419+0000 7f2a62800640 1 -- 172.21.15.43:0/4026009145 shutdown_connections 2024-03-28T22:37:31.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.419+0000 7f2a62800640 1 -- 172.21.15.43:0/4026009145 wait complete. 2024-03-28T22:37:31.434 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:37:31.434 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:37:31.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.654+0000 7f90ffa00640 1 -- 172.21.15.43:0/248435131 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90f81076f0 msgr2=0x7f90f8107ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:31.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.654+0000 7f90ffa00640 1 --2- 172.21.15.43:0/248435131 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90f81076f0 0x7f90f8107ad0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f90e8009900 tx=0x7f90e802bc30 comp rx=0 tx=0).stop 2024-03-28T22:37:31.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.654+0000 7f90ffa00640 1 -- 172.21.15.43:0/248435131 shutdown_connections 2024-03-28T22:37:31.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.654+0000 7f90ffa00640 1 --2- 172.21.15.43:0/248435131 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90f81020a0 0x7f90f810e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.654+0000 7f90ffa00640 1 --2- 172.21.15.43:0/248435131 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90f81016e0 0x7f90f8101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.654+0000 7f90ffa00640 1 --2- 172.21.15.43:0/248435131 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90f81076f0 0x7f90f8107ad0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.654+0000 7f90ffa00640 1 -- 172.21.15.43:0/248435131 >> 172.21.15.43:0/248435131 conn(0x7f90f80fd1f0 msgr2=0x7f90f80ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:31.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.654+0000 7f90ffa00640 1 -- 172.21.15.43:0/248435131 shutdown_connections 2024-03-28T22:37:31.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.654+0000 7f90ffa00640 1 -- 172.21.15.43:0/248435131 wait complete. 2024-03-28T22:37:31.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.655+0000 7f90ffa00640 1 Processor -- start 2024-03-28T22:37:31.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.656+0000 7f90ffa00640 1 -- start start 2024-03-28T22:37:31.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.656+0000 7f90ffa00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90f81016e0 0x7f90f8073a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:31.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.656+0000 7f90ffa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90f81020a0 0x7f90f8073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:31.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.656+0000 7f90ffa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90f81076f0 0x7f90f80744d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:31.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.656+0000 7f90ffa00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f90f8111b50 con 0x7f90f81020a0 2024-03-28T22:37:31.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.656+0000 7f90ffa00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f90f81119d0 con 0x7f90f81016e0 2024-03-28T22:37:31.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.656+0000 7f90ffa00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f90f8111cd0 con 0x7f90f81076f0 2024-03-28T22:37:31.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.656+0000 7f90fd000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90f81016e0 0x7f90f8073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:31.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.656+0000 7f90fd000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90f81016e0 0x7f90f8073a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:36478/0 (socket says 172.21.15.43:36478) 2024-03-28T22:37:31.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.656+0000 7f90fd000640 1 -- 172.21.15.43:0/3490403483 learned_addr learned my addr 172.21.15.43:0/3490403483 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:31.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.657+0000 7f90fd000640 1 -- 172.21.15.43:0/3490403483 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90f81076f0 msgr2=0x7f90f80744d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T22:37:31.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.657+0000 7f90fd000640 1 --2- 172.21.15.43:0/3490403483 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90f81076f0 0x7f90f80744d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.657+0000 7f90fd000640 1 -- 172.21.15.43:0/3490403483 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90f81020a0 msgr2=0x7f90f8073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:31.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.657+0000 7f90fd000640 1 --2- 172.21.15.43:0/3490403483 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90f81020a0 0x7f90f8073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.657+0000 7f90fd000640 1 -- 172.21.15.43:0/3490403483 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90f81a64f0 con 0x7f90f81016e0 2024-03-28T22:37:31.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.657+0000 7f90fd000640 1 --2- 172.21.15.43:0/3490403483 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90f81016e0 0x7f90f8073a50 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f90e802efd0 tx=0x7f90e8004430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:31.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.657+0000 7f90f5600640 1 -- 172.21.15.43:0/3490403483 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e8041070 con 0x7f90f81016e0 2024-03-28T22:37:31.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.657+0000 7f90f5600640 1 -- 172.21.15.43:0/3490403483 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90e80054e0 con 0x7f90f81016e0 2024-03-28T22:37:31.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.657+0000 7f90f5600640 1 -- 172.21.15.43:0/3490403483 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e8003d40 con 0x7f90f81016e0 2024-03-28T22:37:31.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.658+0000 7f90ffa00640 1 -- 172.21.15.43:0/3490403483 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90f81a6780 con 0x7f90f81016e0 2024-03-28T22:37:31.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.658+0000 7f90ffa00640 1 -- 172.21.15.43:0/3490403483 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90f81a6c20 con 0x7f90f81016e0 2024-03-28T22:37:31.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.659+0000 7f90ffa00640 1 -- 172.21.15.43:0/3490403483 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90f8068990 con 0x7f90f81016e0 2024-03-28T22:37:31.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.661+0000 7f90f5600640 1 -- 172.21.15.43:0/3490403483 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 26) ==== 136932+0+0 (secure 0 0 0) 0x7f90e8044070 con 0x7f90f81016e0 2024-03-28T22:37:31.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.662+0000 7f90f5600640 1 --2- 172.21.15.43:0/3490403483 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f90dc0a2000 0x7f90dc0a44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:31.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.662+0000 7f90f5600640 1 -- 172.21.15.43:0/3490403483 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f90e80f01a0 con 0x7f90f81016e0 2024-03-28T22:37:31.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.662+0000 7f90f7e00640 1 --2- 172.21.15.43:0/3490403483 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f90dc0a2000 0x7f90dc0a44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:31.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.663+0000 7f90f7e00640 1 --2- 172.21.15.43:0/3490403483 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f90dc0a2000 0x7f90dc0a44c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f90e40058c0 tx=0x7f90e4005850 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:31.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.671+0000 7f90f5600640 1 -- 172.21.15.43:0/3490403483 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f90e80bce30 con 0x7f90f81016e0 2024-03-28T22:37:31.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.918+0000 7f90ffa00640 1 -- 172.21.15.43:0/3490403483 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f90f8074f80 con 0x7f90f81016e0 2024-03-28T22:37:31.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.920+0000 7f90f5600640 1 -- 172.21.15.43:0/3490403483 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11934 (secure 0 0 0) 0x7f90e80bc730 con 0x7f90f81016e0 2024-03-28T22:37:31.918 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:37:31.918 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:33:28.906396+0000","modified":"2024-03-28T22:36:49.597670+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24572,"mds_1":24556,"mds_2":24535,"mds_3":24523,"mds_4":24518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/1084988072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1084988072},{"type":"v1","addr":"172.21.15.82:6835","nonce":1084988072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572,24523,24518,24535,24556]},"id":1}]} 2024-03-28T22:37:31.918 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 29 2024-03-28T22:37:31.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.923+0000 7f90ffa00640 1 -- 172.21.15.43:0/3490403483 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f90dc0a2000 msgr2=0x7f90dc0a44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:31.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.923+0000 7f90ffa00640 1 --2- 172.21.15.43:0/3490403483 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f90dc0a2000 0x7f90dc0a44c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f90e40058c0 tx=0x7f90e4005850 comp rx=0 tx=0).stop 2024-03-28T22:37:31.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.923+0000 7f90ffa00640 1 -- 172.21.15.43:0/3490403483 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90f81016e0 msgr2=0x7f90f8073a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:31.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.923+0000 7f90ffa00640 1 --2- 172.21.15.43:0/3490403483 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90f81016e0 0x7f90f8073a50 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f90e802efd0 tx=0x7f90e8004430 comp rx=0 tx=0).stop 2024-03-28T22:37:31.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.923+0000 7f90ffa00640 1 -- 172.21.15.43:0/3490403483 shutdown_connections 2024-03-28T22:37:31.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.923+0000 7f90ffa00640 1 --2- 172.21.15.43:0/3490403483 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f90dc0a2000 0x7f90dc0a44c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.923+0000 7f90ffa00640 1 --2- 172.21.15.43:0/3490403483 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90f81076f0 0x7f90f80744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.923+0000 7f90ffa00640 1 --2- 172.21.15.43:0/3490403483 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90f81020a0 0x7f90f8073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.923+0000 7f90ffa00640 1 --2- 172.21.15.43:0/3490403483 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f90f81016e0 0x7f90f8073a50 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:31.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.923+0000 7f90ffa00640 1 -- 172.21.15.43:0/3490403483 >> 172.21.15.43:0/3490403483 conn(0x7f90f80fd1f0 msgr2=0x7f90f80fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:31.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.923+0000 7f90ffa00640 1 -- 172.21.15.43:0/3490403483 shutdown_connections 2024-03-28T22:37:31.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:31.923+0000 7f90ffa00640 1 -- 172.21.15.43:0/3490403483 wait complete. 2024-03-28T22:37:31.943 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T22:37:32.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.167+0000 7fde1b200640 1 -- 172.21.15.43:0/3616084173 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fde141038f0 msgr2=0x7fde14103d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:32.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.167+0000 7fde1b200640 1 --2- 172.21.15.43:0/3616084173 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fde141038f0 0x7fde14103d70 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fddfc009930 tx=0x7fddfc02bc30 comp rx=0 tx=0).stop 2024-03-28T22:37:32.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.167+0000 7fde1b200640 1 -- 172.21.15.43:0/3616084173 shutdown_connections 2024-03-28T22:37:32.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.167+0000 7fde1b200640 1 --2- 172.21.15.43:0/3616084173 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde141042b0 0x7fde141107c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.167+0000 7fde1b200640 1 --2- 172.21.15.43:0/3616084173 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fde141038f0 0x7fde14103d70 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.167+0000 7fde1b200640 1 --2- 172.21.15.43:0/3616084173 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fde14109900 0x7fde14109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.167+0000 7fde1b200640 1 -- 172.21.15.43:0/3616084173 >> 172.21.15.43:0/3616084173 conn(0x7fde140ff3c0 msgr2=0x7fde141017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:32.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.168+0000 7fde1b200640 1 -- 172.21.15.43:0/3616084173 shutdown_connections 2024-03-28T22:37:32.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.168+0000 7fde1b200640 1 -- 172.21.15.43:0/3616084173 wait complete. 2024-03-28T22:37:32.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.169+0000 7fde1b200640 1 Processor -- start 2024-03-28T22:37:32.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.169+0000 7fde1b200640 1 -- start start 2024-03-28T22:37:32.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.169+0000 7fde1b200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fde141038f0 0x7fde1419a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.169+0000 7fde1b200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fde141042b0 0x7fde1419a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.169+0000 7fde1b200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde14109900 0x7fde1419eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.169+0000 7fde1b200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fde14113d10 con 0x7fde14109900 2024-03-28T22:37:32.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.169+0000 7fde1b200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fde14113b90 con 0x7fde141042b0 2024-03-28T22:37:32.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.169+0000 7fde1b200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fde14113e90 con 0x7fde141038f0 2024-03-28T22:37:32.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.170+0000 7fde0be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fde141042b0 0x7fde1419a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.170+0000 7fde0be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fde141042b0 0x7fde1419a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:36496/0 (socket says 172.21.15.43:36496) 2024-03-28T22:37:32.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.170+0000 7fde0be00640 1 -- 172.21.15.43:0/145968880 learned_addr learned my addr 172.21.15.43:0/145968880 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:32.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.170+0000 7fde13e00640 1 --2- 172.21.15.43:0/145968880 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fde141038f0 0x7fde1419a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.170+0000 7fde19200640 1 --2- 172.21.15.43:0/145968880 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde14109900 0x7fde1419eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.170+0000 7fde13e00640 1 -- 172.21.15.43:0/145968880 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fde141042b0 msgr2=0x7fde1419a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:32.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.170+0000 7fde13e00640 1 --2- 172.21.15.43:0/145968880 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fde141042b0 0x7fde1419a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.170+0000 7fde13e00640 1 -- 172.21.15.43:0/145968880 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde14109900 msgr2=0x7fde1419eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:32.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.170+0000 7fde13e00640 1 --2- 172.21.15.43:0/145968880 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde14109900 0x7fde1419eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.170+0000 7fde13e00640 1 -- 172.21.15.43:0/145968880 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde1419f260 con 0x7fde141038f0 2024-03-28T22:37:32.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.170+0000 7fde0be00640 1 --2- 172.21.15.43:0/145968880 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fde141042b0 0x7fde1419a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:37:32.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.170+0000 7fde13e00640 1 --2- 172.21.15.43:0/145968880 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fde141038f0 0x7fde1419a270 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fde04005fd0 tx=0x7fde0400dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:32.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.171+0000 7fde11600640 1 -- 172.21.15.43:0/145968880 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde04009c30 con 0x7fde141038f0 2024-03-28T22:37:32.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.171+0000 7fde1b200640 1 -- 172.21.15.43:0/145968880 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde1419f550 con 0x7fde141038f0 2024-03-28T22:37:32.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.171+0000 7fde11600640 1 -- 172.21.15.43:0/145968880 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde04009dd0 con 0x7fde141038f0 2024-03-28T22:37:32.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.171+0000 7fde11600640 1 -- 172.21.15.43:0/145968880 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde04010460 con 0x7fde141038f0 2024-03-28T22:37:32.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.171+0000 7fde1b200640 1 -- 172.21.15.43:0/145968880 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde141a6de0 con 0x7fde141038f0 2024-03-28T22:37:32.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.175+0000 7fde1dfbb740 1 Processor -- start 2024-03-28T22:37:32.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.175+0000 7fde11600640 1 -- 172.21.15.43:0/145968880 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 26) ==== 136932+0+0 (secure 0 0 0) 0x7fde04003f50 con 0x7fde141038f0 2024-03-28T22:37:32.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.176+0000 7fde1dfbb740 1 -- start start 2024-03-28T22:37:32.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.176+0000 7fde1dfbb740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562428f57380 0x562428f57760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.176+0000 7fde1dfbb740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562428f57ca0 0x562428f60340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.176+0000 7fde1dfbb740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562428f60880 0x562428f62c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.176+0000 7fde1dfbb740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562428ebb050 con 0x562428f57ca0 2024-03-28T22:37:32.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.176+0000 7fde1dfbb740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x562428e63ce0 con 0x562428f57380 2024-03-28T22:37:32.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.176+0000 7fde1dfbb740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x562428eb70f0 con 0x562428f60880 2024-03-28T22:37:32.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.176+0000 7fde19200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562428f57380 0x562428f57760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.176+0000 7fde13e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562428f57ca0 0x562428f60340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.176+0000 7fde0be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562428f60880 0x562428f62c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.176+0000 7fde19200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562428f57380 0x562428f57760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:36510/0 (socket says 172.21.15.43:36510) 2024-03-28T22:37:32.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.176+0000 7fde19200640 1 -- 172.21.15.43:0/4168160606 learned_addr learned my addr 172.21.15.43:0/4168160606 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:32.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.177+0000 7fde19200640 1 -- 172.21.15.43:0/4168160606 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562428f60880 msgr2=0x562428f62c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:32.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.177+0000 7fde19200640 1 --2- 172.21.15.43:0/4168160606 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562428f60880 0x562428f62c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.177+0000 7fde19200640 1 -- 172.21.15.43:0/4168160606 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562428f57ca0 msgr2=0x562428f60340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:32.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.177+0000 7fde19200640 1 --2- 172.21.15.43:0/4168160606 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562428f57ca0 0x562428f60340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.177+0000 7fde19200640 1 -- 172.21.15.43:0/4168160606 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562428bcf3f0 con 0x562428f57380 2024-03-28T22:37:32.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.177+0000 7fde19200640 1 --2- 172.21.15.43:0/4168160606 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562428f57380 0x562428f57760 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fde0c010370 tx=0x7fde0c010830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcc994b67961d337 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:32.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.177+0000 7fde09600640 1 -- 172.21.15.43:0/4168160606 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde0c016070 con 0x562428f57380 2024-03-28T22:37:32.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.177+0000 7fde09600640 1 -- 172.21.15.43:0/4168160606 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde0c004d20 con 0x562428f57380 2024-03-28T22:37:32.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.177+0000 7fde09600640 1 -- 172.21.15.43:0/4168160606 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde0c0054c0 con 0x562428f57380 2024-03-28T22:37:32.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.177+0000 7fde11600640 1 --2- 172.21.15.43:0/145968880 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fddf80a1f80 0x7fddf80a4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.178+0000 7fde11600640 1 -- 172.21.15.43:0/145968880 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fde040ce560 con 0x7fde141038f0 2024-03-28T22:37:32.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.178+0000 7fde13e00640 1 --2- 172.21.15.43:0/145968880 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fddf80a1f80 0x7fddf80a4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.178+0000 7fde1dfbb740 1 -- 172.21.15.43:0/4168160606 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562428f57380 msgr2=0x562428f57760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:32.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.178+0000 7fde1dfbb740 1 --2- 172.21.15.43:0/4168160606 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562428f57380 0x562428f57760 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fde0c010370 tx=0x7fde0c010830 comp rx=0 tx=0).stop 2024-03-28T22:37:32.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.178+0000 7fde1dfbb740 1 -- 172.21.15.43:0/4168160606 shutdown_connections 2024-03-28T22:37:32.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.178+0000 7fde1dfbb740 1 --2- 172.21.15.43:0/4168160606 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562428f60880 0x562428f62c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.178+0000 7fde1dfbb740 1 --2- 172.21.15.43:0/4168160606 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562428f57ca0 0x562428f60340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.178+0000 7fde1dfbb740 1 --2- 172.21.15.43:0/4168160606 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562428f57380 0x562428f57760 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.178+0000 7fde1dfbb740 1 -- 172.21.15.43:0/4168160606 >> 172.21.15.43:0/4168160606 conn(0x562428f51ab0 msgr2=0x562428f51e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:32.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.178+0000 7fde1dfbb740 1 -- 172.21.15.43:0/4168160606 shutdown_connections 2024-03-28T22:37:32.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.178+0000 7fde1dfbb740 1 -- 172.21.15.43:0/4168160606 wait complete. 2024-03-28T22:37:32.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.178+0000 7fde13e00640 1 --2- 172.21.15.43:0/145968880 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fddf80a1f80 0x7fddf80a4440 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fde040cb8b0 tx=0x7fde040d4060 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:32.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.179+0000 7fde1dfbb740 1 -- start start 2024-03-28T22:37:32.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.179+0000 7fde1dfbb740 1 Processor -- start 2024-03-28T22:37:32.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.179+0000 7fde1dfbb740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562428f57380 0x562428f5d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.179+0000 7fde1dfbb740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562428f57ca0 0x562428f5de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.179+0000 7fde1dfbb740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562428f60880 0x562428f5e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.179+0000 7fde1dfbb740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562428f63670 con 0x562428f57ca0 2024-03-28T22:37:32.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.179+0000 7fde1dfbb740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x562428e592f0 con 0x562428f57380 2024-03-28T22:37:32.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.179+0000 7fde1dfbb740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x562428f637f0 con 0x562428f60880 2024-03-28T22:37:32.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.179+0000 7fde1dfbb740 10 client.0 _ll_register_callbacks cb 0x562428bcebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:37:32.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.180+0000 7fde0be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562428f57ca0 0x562428f5de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.180+0000 7fde19200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562428f57380 0x562428f5d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.180+0000 7fde0be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562428f57ca0 0x562428f5de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46892/0 (socket says 172.21.15.43:46892) 2024-03-28T22:37:32.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.180+0000 7fde0be00640 1 -- 172.21.15.43:0/2937838584 learned_addr learned my addr 172.21.15.43:0/2937838584 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:32.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.180+0000 7fde19200640 1 --2- 172.21.15.43:0/2937838584 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562428f60880 0x562428f5e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.180+0000 7fde0be00640 1 -- 172.21.15.43:0/2937838584 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562428f60880 msgr2=0x562428f5e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:32.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.180+0000 7fde0be00640 1 --2- 172.21.15.43:0/2937838584 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562428f60880 0x562428f5e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.180+0000 7fde0be00640 1 -- 172.21.15.43:0/2937838584 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562428f57380 msgr2=0x562428f5d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:32.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.180+0000 7fde0be00640 1 --2- 172.21.15.43:0/2937838584 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562428f57380 0x562428f5d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.180+0000 7fde0be00640 1 -- 172.21.15.43:0/2937838584 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562428c997d0 con 0x562428f57ca0 2024-03-28T22:37:32.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.180+0000 7fde19200640 1 --2- 172.21.15.43:0/2937838584 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562428f60880 0x562428f5e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:37:32.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.181+0000 7fde0be00640 1 --2- 172.21.15.43:0/2937838584 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562428f57ca0 0x562428f5de80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fddfc005c40 tx=0x7fddfc02cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25d2c28e487b9124 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:32.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.181+0000 7fde02000640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:37:32.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.181+0000 7fde02000640 1 -- 172.21.15.43:0/2937838584 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddfc004040 con 0x562428f57ca0 2024-03-28T22:37:32.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.181+0000 7fde02000640 1 -- 172.21.15.43:0/2937838584 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fddfc0041e0 con 0x562428f57ca0 2024-03-28T22:37:32.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.181+0000 7fde1dfbb740 1 -- 172.21.15.43:0/2937838584 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde0c004d20 con 0x562428f57ca0 2024-03-28T22:37:32.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.181+0000 7fde02000640 1 -- 172.21.15.43:0/2937838584 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fddfc0043e0 con 0x562428f57ca0 2024-03-28T22:37:32.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.181+0000 7fde1dfbb740 10 client.15669 fetch_fsmap learned FSMap version 29 2024-03-28T22:37:32.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.181+0000 7fde1dfbb740 1 -- 172.21.15.43:0/2937838584 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562428f6f9a0 con 0x562428f57ca0 2024-03-28T22:37:32.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.181+0000 7fde02000640 1 -- 172.21.15.43:0/2937838584 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde0c004d20 con 0x562428f57ca0 2024-03-28T22:37:32.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.182+0000 7fde02000640 1 -- 172.21.15.43:0/2937838584 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fddfc003a30 con 0x562428f57ca0 2024-03-28T22:37:32.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.182+0000 7fde1dfbb740 10 client.15669 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:37:32.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.182+0000 7fde1dfbb740 10 client.15669 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:37:32.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.182+0000 7fde1dfbb740 20 client.15669 populate_metadata read hostname 'smithi043' 2024-03-28T22:37:32.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.182+0000 7fde1dfbb740 1 --2- 172.21.15.43:0/2937838584 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x562428f70e60 0x562428f73290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.182+0000 7fde1dfbb740 4 client.15669 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:37:32.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.182+0000 7fde1dfbb740 1 -- 172.21.15.43:0/2937838584 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562428e64680 con 0x562428f70e60 2024-03-28T22:37:32.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.182+0000 7fde19200640 1 --2- 172.21.15.43:0/2937838584 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x562428f70e60 0x562428f73290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.183+0000 7fde19200640 1 --2- 172.21.15.43:0/2937838584 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x562428f70e60 0x562428f73290 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c4d82b5204aad61b server_cookie=fe158aa8f832669d in_seq=0 out_seq=0 2024-03-28T22:37:32.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.183+0000 7fde02000640 10 client.15669 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:37:32.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.183+0000 7fde02000640 1 -- 172.21.15.43:0/2937838584 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562428e64680 con 0x562428f70e60 2024-03-28T22:37:32.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.183+0000 7fde02000640 10 client.15669 handle_command_reply: tid=0 2024-03-28T22:37:32.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.183+0000 7fde1dfbb740 1 client.15669 shutdown 2024-03-28T22:37:32.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.184+0000 7fde1dfbb740 1 -- 172.21.15.43:0/2937838584 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562428f57ca0 msgr2=0x562428f5de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:32.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.184+0000 7fde1dfbb740 1 --2- 172.21.15.43:0/2937838584 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562428f57ca0 0x562428f5de80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fddfc005c40 tx=0x7fddfc02cab0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.184+0000 7fde1dfbb740 1 -- 172.21.15.43:0/2937838584 shutdown_connections 2024-03-28T22:37:32.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.184+0000 7fde1dfbb740 1 --2- 172.21.15.43:0/2937838584 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x562428f70e60 0x562428f73290 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.184+0000 7fde1dfbb740 1 --2- 172.21.15.43:0/2937838584 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562428f60880 0x562428f5e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.184+0000 7fde1dfbb740 1 --2- 172.21.15.43:0/2937838584 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562428f57ca0 0x562428f5de80 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.184+0000 7fde1dfbb740 1 --2- 172.21.15.43:0/2937838584 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562428f57380 0x562428f5d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.184+0000 7fde1dfbb740 1 -- 172.21.15.43:0/2937838584 >> 172.21.15.43:0/2937838584 conn(0x562428f51ab0 msgr2=0x562428f53800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:32.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.184+0000 7fde02000640 0 client.15669 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:37:32.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.184+0000 7fde1dfbb740 1 -- 172.21.15.43:0/2937838584 shutdown_connections 2024-03-28T22:37:32.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.184+0000 7fde1dfbb740 1 -- 172.21.15.43:0/2937838584 wait complete. 2024-03-28T22:37:32.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.184+0000 7fde1dfbb740 20 client.15669 trim_cache size 0 max 16384 2024-03-28T22:37:32.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.199+0000 7fde1dfbb740 1 Processor -- start 2024-03-28T22:37:32.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.200+0000 7fde1dfbb740 1 -- start start 2024-03-28T22:37:32.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.200+0000 7fde1dfbb740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562428f57380 0x562428f60070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.200+0000 7fde1dfbb740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562428f57ca0 0x562428f6c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.200+0000 7fde1dfbb740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562428f60880 0x562428f78b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.200+0000 7fde1dfbb740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fde141b0f50 con 0x562428f57ca0 2024-03-28T22:37:32.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.200+0000 7fde1dfbb740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fde141b2a90 con 0x562428f57380 2024-03-28T22:37:32.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.200+0000 7fde1dfbb740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fde141b1620 con 0x562428f60880 2024-03-28T22:37:32.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.200+0000 7fde0be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562428f57ca0 0x562428f6c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.200+0000 7fde19200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562428f57380 0x562428f60070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.200+0000 7fde0be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562428f57ca0 0x562428f6c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46898/0 (socket says 172.21.15.43:46898) 2024-03-28T22:37:32.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.200+0000 7fde0be00640 1 -- 172.21.15.43:0/519359651 learned_addr learned my addr 172.21.15.43:0/519359651 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:32.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.200+0000 7fde19200640 1 --2- 172.21.15.43:0/519359651 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562428f60880 0x562428f78b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:31 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3490403483' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:37:32.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.201+0000 7fde19200640 1 -- 172.21.15.43:0/519359651 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562428f60880 msgr2=0x562428f78b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:32.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.201+0000 7fde19200640 1 --2- 172.21.15.43:0/519359651 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562428f60880 0x562428f78b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.201+0000 7fde19200640 1 -- 172.21.15.43:0/519359651 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562428f57ca0 msgr2=0x562428f6c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:32.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.201+0000 7fde19200640 1 --2- 172.21.15.43:0/519359651 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562428f57ca0 0x562428f6c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.201+0000 7fde19200640 1 -- 172.21.15.43:0/519359651 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562428f5c8f0 con 0x562428f57380 2024-03-28T22:37:32.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.201+0000 7fde0be00640 1 --2- 172.21.15.43:0/519359651 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562428f57ca0 0x562428f6c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:37:32.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.201+0000 7fde19200640 1 --2- 172.21.15.43:0/519359651 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562428f57380 0x562428f60070 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fde0c005120 tx=0x7fde0c000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50edc6069111eafd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:32.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.201+0000 7fde08c00640 1 -- 172.21.15.43:0/519359651 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde0c033070 con 0x562428f57380 2024-03-28T22:37:32.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.201+0000 7fde08c00640 1 -- 172.21.15.43:0/519359651 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde0c02c520 con 0x562428f57380 2024-03-28T22:37:32.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.201+0000 7fde08c00640 1 -- 172.21.15.43:0/519359651 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde0c016040 con 0x562428f57380 2024-03-28T22:37:32.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.202+0000 7fde1dfbb740 1 -- 172.21.15.43:0/519359651 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562428f57380 msgr2=0x562428f60070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:32.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.202+0000 7fde1dfbb740 1 --2- 172.21.15.43:0/519359651 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562428f57380 0x562428f60070 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fde0c005120 tx=0x7fde0c000f40 comp rx=0 tx=0).stop 2024-03-28T22:37:32.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.202+0000 7fde1dfbb740 1 -- 172.21.15.43:0/519359651 shutdown_connections 2024-03-28T22:37:32.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.202+0000 7fde1dfbb740 1 --2- 172.21.15.43:0/519359651 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562428f60880 0x562428f78b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.202+0000 7fde1dfbb740 1 --2- 172.21.15.43:0/519359651 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562428f57ca0 0x562428f6c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.202+0000 7fde1dfbb740 1 --2- 172.21.15.43:0/519359651 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562428f57380 0x562428f60070 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.202+0000 7fde1dfbb740 1 -- 172.21.15.43:0/519359651 >> 172.21.15.43:0/519359651 conn(0x562428f51ab0 msgr2=0x562428f5e190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:32.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.202+0000 7fde1dfbb740 1 -- 172.21.15.43:0/519359651 shutdown_connections 2024-03-28T22:37:32.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.202+0000 7fde1dfbb740 1 -- 172.21.15.43:0/519359651 wait complete. 2024-03-28T22:37:32.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.203+0000 7fde1dfbb740 1 -- start start 2024-03-28T22:37:32.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.203+0000 7fde1dfbb740 1 Processor -- start 2024-03-28T22:37:32.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.203+0000 7fde1dfbb740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562428f57380 0x562428f689d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.203+0000 7fde1dfbb740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562428f57ca0 0x562428f64d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.203+0000 7fde1dfbb740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562428f60880 0x562428f65290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.203+0000 7fde1dfbb740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fde141b23c0 con 0x562428f60880 2024-03-28T22:37:32.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.203+0000 7fde1dfbb740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fde141b1cf0 con 0x562428f57380 2024-03-28T22:37:32.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.203+0000 7fde1dfbb740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fde141aef30 con 0x562428f57ca0 2024-03-28T22:37:32.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.203+0000 7fde1dfbb740 10 client.0 _ll_register_callbacks cb 0x7fddfc005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:37:32.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.204+0000 7fde0be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562428f57ca0 0x562428f64d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.204+0000 7fde0be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562428f57ca0 0x562428f64d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:52418/0 (socket says 172.21.15.43:52418) 2024-03-28T22:37:32.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.204+0000 7fde0be00640 1 -- 172.21.15.43:0/1826392973 learned_addr learned my addr 172.21.15.43:0/1826392973 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:32.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.204+0000 7fde19200640 1 --2- 172.21.15.43:0/1826392973 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562428f60880 0x562428f65290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.204+0000 7fde19200640 1 --2- 172.21.15.43:0/1826392973 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562428f57380 0x562428f689d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.204+0000 7fde0be00640 1 -- 172.21.15.43:0/1826392973 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562428f57380 msgr2=0x562428f689d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:32.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.204+0000 7fde0be00640 1 --2- 172.21.15.43:0/1826392973 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562428f57380 0x562428f689d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.204+0000 7fde0be00640 1 -- 172.21.15.43:0/1826392973 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562428f60880 msgr2=0x562428f65290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:32.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.204+0000 7fde0be00640 1 --2- 172.21.15.43:0/1826392973 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562428f60880 0x562428f65290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.204+0000 7fde0be00640 1 -- 172.21.15.43:0/1826392973 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562428f657d0 con 0x562428f57ca0 2024-03-28T22:37:32.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.204+0000 7fde19200640 1 --2- 172.21.15.43:0/1826392973 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562428f60880 0x562428f65290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:37:32.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.204+0000 7fde0be00640 1 --2- 172.21.15.43:0/1826392973 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562428f57ca0 0x562428f64d50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fddfc003e50 tx=0x7fddfc02cd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84517dea72ffb88e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:32.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.204+0000 7fde02000640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:37:32.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.204+0000 7fde02000640 1 -- 172.21.15.43:0/1826392973 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddfc0346a0 con 0x562428f57ca0 2024-03-28T22:37:32.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.205+0000 7fde1dfbb740 1 -- 172.21.15.43:0/1826392973 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde0c02c520 con 0x562428f57ca0 2024-03-28T22:37:32.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.205+0000 7fde02000640 1 -- 172.21.15.43:0/1826392973 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fddfc034910 con 0x562428f57ca0 2024-03-28T22:37:32.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.205+0000 7fde02000640 1 -- 172.21.15.43:0/1826392973 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fddfc064050 con 0x562428f57ca0 2024-03-28T22:37:32.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.205+0000 7fde1dfbb740 10 client.25780 fetch_fsmap learned FSMap version 29 2024-03-28T22:37:32.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.205+0000 7fde1dfbb740 1 -- 172.21.15.43:0/1826392973 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562428fd7ec0 con 0x562428f57ca0 2024-03-28T22:37:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.205+0000 7fde02000640 1 -- 172.21.15.43:0/1826392973 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde0c02c520 con 0x562428f57ca0 2024-03-28T22:37:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.205+0000 7fde02000640 1 -- 172.21.15.43:0/1826392973 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fddfc003850 con 0x562428f57ca0 2024-03-28T22:37:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.205+0000 7fde1dfbb740 10 client.25780 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:37:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.205+0000 7fde1dfbb740 10 client.25780 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:37:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.205+0000 7fde1dfbb740 20 client.25780 populate_metadata read hostname 'smithi043' 2024-03-28T22:37:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.205+0000 7fde1dfbb740 1 --2- 172.21.15.43:0/1826392973 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x562428fd86c0 0x562428fdaae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.205+0000 7fde1dfbb740 4 client.25780 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:37:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.205+0000 7fde1dfbb740 1 -- 172.21.15.43:0/1826392973 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562428ec8590 con 0x562428fd86c0 2024-03-28T22:37:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.206+0000 7fde19200640 1 --2- 172.21.15.43:0/1826392973 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x562428fd86c0 0x562428fdaae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.206+0000 7fde19200640 1 --2- 172.21.15.43:0/1826392973 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x562428fd86c0 0x562428fdaae0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d7d488ee25850e52 server_cookie=2c2a0dd5c93cf0d0 in_seq=0 out_seq=0 2024-03-28T22:37:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.206+0000 7fde02000640 10 client.25780 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:37:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.206+0000 7fde02000640 1 -- 172.21.15.43:0/1826392973 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562428ec8590 con 0x562428fd86c0 2024-03-28T22:37:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.206+0000 7fde02000640 10 client.25780 handle_command_reply: tid=0 2024-03-28T22:37:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.206+0000 7fde1dfbb740 1 client.25780 shutdown 2024-03-28T22:37:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.207+0000 7fde1dfbb740 1 -- 172.21.15.43:0/1826392973 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562428f57ca0 msgr2=0x562428f64d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.207+0000 7fde1dfbb740 1 --2- 172.21.15.43:0/1826392973 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562428f57ca0 0x562428f64d50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fddfc003e50 tx=0x7fddfc02cd50 comp rx=0 tx=0).stop 2024-03-28T22:37:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.207+0000 7fde1dfbb740 1 -- 172.21.15.43:0/1826392973 shutdown_connections 2024-03-28T22:37:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.207+0000 7fde1dfbb740 1 --2- 172.21.15.43:0/1826392973 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x562428fd86c0 0x562428fdaae0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.207+0000 7fde1dfbb740 1 --2- 172.21.15.43:0/1826392973 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562428f60880 0x562428f65290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.207+0000 7fde1dfbb740 1 --2- 172.21.15.43:0/1826392973 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562428f57ca0 0x562428f64d50 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.207+0000 7fde1dfbb740 1 --2- 172.21.15.43:0/1826392973 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562428f57380 0x562428f689d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.207+0000 7fde1dfbb740 1 -- 172.21.15.43:0/1826392973 >> 172.21.15.43:0/1826392973 conn(0x562428f51ab0 msgr2=0x562428f5d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.207+0000 7fde02000640 0 client.25780 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:37:32.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.207+0000 7fde1dfbb740 1 -- 172.21.15.43:0/1826392973 shutdown_connections 2024-03-28T22:37:32.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.207+0000 7fde1dfbb740 1 -- 172.21.15.43:0/1826392973 wait complete. 2024-03-28T22:37:32.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.207+0000 7fde1dfbb740 20 client.25780 trim_cache size 0 max 16384 2024-03-28T22:37:32.210 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:37:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.208+0000 7fde1b200640 1 -- 172.21.15.43:0/145968880 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fddf80a1f80 msgr2=0x7fddf80a4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.208+0000 7fde1b200640 1 --2- 172.21.15.43:0/145968880 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fddf80a1f80 0x7fddf80a4440 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fde040cb8b0 tx=0x7fde040d4060 comp rx=0 tx=0).stop 2024-03-28T22:37:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.208+0000 7fde1b200640 1 -- 172.21.15.43:0/145968880 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fde141038f0 msgr2=0x7fde1419a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.208+0000 7fde1b200640 1 --2- 172.21.15.43:0/145968880 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fde141038f0 0x7fde1419a270 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fde04005fd0 tx=0x7fde0400dde0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.208+0000 7fde1b200640 1 -- 172.21.15.43:0/145968880 shutdown_connections 2024-03-28T22:37:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.208+0000 7fde1b200640 1 --2- 172.21.15.43:0/145968880 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fddf80a1f80 0x7fddf80a4440 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.208+0000 7fde1b200640 1 --2- 172.21.15.43:0/145968880 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fde141042b0 0x7fde1419a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.208+0000 7fde1b200640 1 --2- 172.21.15.43:0/145968880 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fde14109900 0x7fde1419eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.208+0000 7fde1b200640 1 --2- 172.21.15.43:0/145968880 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fde141038f0 0x7fde1419a270 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.208+0000 7fde1b200640 1 -- 172.21.15.43:0/145968880 >> 172.21.15.43:0/145968880 conn(0x7fde140ff3c0 msgr2=0x7fde140ffc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.208+0000 7fde1b200640 1 -- 172.21.15.43:0/145968880 shutdown_connections 2024-03-28T22:37:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.208+0000 7fde1b200640 1 -- 172.21.15.43:0/145968880 wait complete. 2024-03-28T22:37:32.220 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:31 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3490403483' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:37:32.223 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T22:37:32.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:31 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3490403483' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:37:32.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.433+0000 7f24b1800640 1 -- 172.21.15.43:0/4029388342 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24ac075c80 msgr2=0x7f24ac076060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:32.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.433+0000 7f24b1800640 1 --2- 172.21.15.43:0/4029388342 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24ac075c80 0x7f24ac076060 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f249c009930 tx=0x7f249c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:37:32.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.433+0000 7f24b1800640 1 -- 172.21.15.43:0/4029388342 shutdown_connections 2024-03-28T22:37:32.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.433+0000 7f24b1800640 1 --2- 172.21.15.43:0/4029388342 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24ac112590 0x7f24ac114980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.433+0000 7f24b1800640 1 --2- 172.21.15.43:0/4029388342 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24ac0765a0 0x7f24ac112050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.433+0000 7f24b1800640 1 --2- 172.21.15.43:0/4029388342 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24ac075c80 0x7f24ac076060 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.433+0000 7f24b1800640 1 -- 172.21.15.43:0/4029388342 >> 172.21.15.43:0/4029388342 conn(0x7f24ac0ff3a0 msgr2=0x7f24ac1017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:32.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.433+0000 7f24b1800640 1 -- 172.21.15.43:0/4029388342 shutdown_connections 2024-03-28T22:37:32.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.433+0000 7f24b1800640 1 -- 172.21.15.43:0/4029388342 wait complete. 2024-03-28T22:37:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.434+0000 7f24b1800640 1 Processor -- start 2024-03-28T22:37:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.434+0000 7f24b1800640 1 -- start start 2024-03-28T22:37:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.435+0000 7f24b1800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24ac075c80 0x7f24ac19e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.435+0000 7f24b1800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24ac0765a0 0x7f24ac19ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.435+0000 7f24b1800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24ac112590 0x7f24ac1a2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.435+0000 7f24b1800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f24ac118130 con 0x7f24ac0765a0 2024-03-28T22:37:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.435+0000 7f24b1800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f24ac117fb0 con 0x7f24ac112590 2024-03-28T22:37:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.435+0000 7f24b1800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f24ac1182b0 con 0x7f24ac075c80 2024-03-28T22:37:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.435+0000 7f24aaa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24ac075c80 0x7f24ac19e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.435+0000 7f24aaa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24ac075c80 0x7f24ac19e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:52446/0 (socket says 172.21.15.43:52446) 2024-03-28T22:37:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.435+0000 7f24aaa00640 1 -- 172.21.15.43:0/1716273778 learned_addr learned my addr 172.21.15.43:0/1716273778 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.435+0000 7f24ab400640 1 --2- 172.21.15.43:0/1716273778 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24ac112590 0x7f24ac1a2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.435+0000 7f24aa000640 1 --2- 172.21.15.43:0/1716273778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24ac0765a0 0x7f24ac19ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.436+0000 7f24aaa00640 1 -- 172.21.15.43:0/1716273778 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24ac112590 msgr2=0x7f24ac1a2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.436+0000 7f24aaa00640 1 --2- 172.21.15.43:0/1716273778 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24ac112590 0x7f24ac1a2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.436+0000 7f24aaa00640 1 -- 172.21.15.43:0/1716273778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24ac0765a0 msgr2=0x7f24ac19ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.436+0000 7f24aaa00640 1 --2- 172.21.15.43:0/1716273778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24ac0765a0 0x7f24ac19ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.436+0000 7f24aaa00640 1 -- 172.21.15.43:0/1716273778 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24ac1a3690 con 0x7f24ac075c80 2024-03-28T22:37:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.436+0000 7f24aa000640 1 --2- 172.21.15.43:0/1716273778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24ac0765a0 0x7f24ac19ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:37:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.436+0000 7f24aaa00640 1 --2- 172.21.15.43:0/1716273778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24ac075c80 0x7f24ac19e6f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f249c009900 tx=0x7f249c004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.436+0000 7f24a3400640 1 -- 172.21.15.43:0/1716273778 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f249c043020 con 0x7f24ac075c80 2024-03-28T22:37:32.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.436+0000 7f24b1800640 1 -- 172.21.15.43:0/1716273778 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24ac1a3920 con 0x7f24ac075c80 2024-03-28T22:37:32.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.436+0000 7f24a3400640 1 -- 172.21.15.43:0/1716273778 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f249c0054e0 con 0x7f24ac075c80 2024-03-28T22:37:32.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.436+0000 7f24a3400640 1 -- 172.21.15.43:0/1716273778 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f249c003bb0 con 0x7f24ac075c80 2024-03-28T22:37:32.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.436+0000 7f24b1800640 1 -- 172.21.15.43:0/1716273778 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24ac1ab210 con 0x7f24ac075c80 2024-03-28T22:37:32.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.440+0000 7f24b43e4740 1 Processor -- start 2024-03-28T22:37:32.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.441+0000 7f24b43e4740 1 -- start start 2024-03-28T22:37:32.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.441+0000 7f24b43e4740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a2b872380 0x560a2b872760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.441+0000 7f24b43e4740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a2b872ca0 0x560a2b87b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.441+0000 7f24b43e4740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a2b87b880 0x560a2b87dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.441+0000 7f24a3400640 1 -- 172.21.15.43:0/1716273778 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 26) ==== 136932+0+0 (secure 0 0 0) 0x7f249c046020 con 0x7f24ac075c80 2024-03-28T22:37:32.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.441+0000 7f24b43e4740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x560a2b7d6050 con 0x560a2b87b880 2024-03-28T22:37:32.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.441+0000 7f24b43e4740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x560a2b77ece0 con 0x560a2b872380 2024-03-28T22:37:32.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.441+0000 7f24b43e4740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x560a2b7d20f0 con 0x560a2b872ca0 2024-03-28T22:37:32.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.441+0000 7f24ab400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a2b872380 0x560a2b872760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.441+0000 7f24ab400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a2b872380 0x560a2b872760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:36576/0 (socket says 172.21.15.43:36576) 2024-03-28T22:37:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.442+0000 7f24ab400640 1 -- 172.21.15.43:0/3632000491 learned_addr learned my addr 172.21.15.43:0/3632000491 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.442+0000 7f24aaa00640 1 --2- 172.21.15.43:0/3632000491 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a2b872ca0 0x560a2b87b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.442+0000 7f24aa000640 1 --2- 172.21.15.43:0/3632000491 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a2b87b880 0x560a2b87dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.442+0000 7f24ab400640 1 -- 172.21.15.43:0/3632000491 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a2b872ca0 msgr2=0x560a2b87b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.442+0000 7f24ab400640 1 --2- 172.21.15.43:0/3632000491 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a2b872ca0 0x560a2b87b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.442+0000 7f24ab400640 1 -- 172.21.15.43:0/3632000491 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a2b87b880 msgr2=0x560a2b87dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.442+0000 7f24ab400640 1 --2- 172.21.15.43:0/3632000491 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a2b87b880 0x560a2b87dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.442+0000 7f24ab400640 1 -- 172.21.15.43:0/3632000491 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a2b4ea3f0 con 0x560a2b872380 2024-03-28T22:37:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.442+0000 7f24aa000640 1 --2- 172.21.15.43:0/3632000491 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a2b87b880 0x560a2b87dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:37:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.442+0000 7f24ab400640 1 --2- 172.21.15.43:0/3632000491 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a2b872380 0x560a2b872760 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f24a4009f30 tx=0x7f24a40118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a84b60adb8457478 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:32.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.443+0000 7f2497e00640 1 -- 172.21.15.43:0/3632000491 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a4004d20 con 0x560a2b872380 2024-03-28T22:37:32.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.443+0000 7f2497e00640 1 -- 172.21.15.43:0/3632000491 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24a4004050 con 0x560a2b872380 2024-03-28T22:37:32.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.443+0000 7f2497e00640 1 -- 172.21.15.43:0/3632000491 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a4005260 con 0x560a2b872380 2024-03-28T22:37:32.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.443+0000 7f24a3400640 1 --2- 172.21.15.43:0/1716273778 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f24900a1f80 0x7f24900a4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.443+0000 7f24a3400640 1 -- 172.21.15.43:0/1716273778 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f249c0f06a0 con 0x7f24ac075c80 2024-03-28T22:37:32.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.443+0000 7f24aaa00640 1 --2- 172.21.15.43:0/1716273778 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f24900a1f80 0x7f24900a4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.443+0000 7f24b43e4740 1 -- 172.21.15.43:0/3632000491 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a2b872380 msgr2=0x560a2b872760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:32.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.443+0000 7f24b43e4740 1 --2- 172.21.15.43:0/3632000491 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a2b872380 0x560a2b872760 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f24a4009f30 tx=0x7f24a40118d0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.444+0000 7f24b43e4740 1 -- 172.21.15.43:0/3632000491 shutdown_connections 2024-03-28T22:37:32.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.444+0000 7f24b43e4740 1 --2- 172.21.15.43:0/3632000491 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a2b87b880 0x560a2b87dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.444+0000 7f24b43e4740 1 --2- 172.21.15.43:0/3632000491 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a2b872ca0 0x560a2b87b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.444+0000 7f24b43e4740 1 --2- 172.21.15.43:0/3632000491 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a2b872380 0x560a2b872760 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.444+0000 7f24b43e4740 1 -- 172.21.15.43:0/3632000491 >> 172.21.15.43:0/3632000491 conn(0x560a2b86cab0 msgr2=0x560a2b86ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:32.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.444+0000 7f24b43e4740 1 -- 172.21.15.43:0/3632000491 shutdown_connections 2024-03-28T22:37:32.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.444+0000 7f24b43e4740 1 -- 172.21.15.43:0/3632000491 wait complete. 2024-03-28T22:37:32.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.444+0000 7f24aaa00640 1 --2- 172.21.15.43:0/1716273778 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f24900a1f80 0x7f24900a4440 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f249c0edb00 tx=0x7f249c0f6480 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:32.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.444+0000 7f24b43e4740 1 -- start start 2024-03-28T22:37:32.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.445+0000 7f24b43e4740 1 Processor -- start 2024-03-28T22:37:32.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.445+0000 7f24b43e4740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a2b872380 0x560a2b878850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.445+0000 7f24b43e4740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a2b872ca0 0x560a2b878d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.445+0000 7f24b43e4740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a2b87b880 0x560a2b8792d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.445+0000 7f24b43e4740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x560a2b87e670 con 0x560a2b872ca0 2024-03-28T22:37:32.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.445+0000 7f24b43e4740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x560a2b7742f0 con 0x560a2b872380 2024-03-28T22:37:32.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.445+0000 7f24b43e4740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x560a2b87e7f0 con 0x560a2b87b880 2024-03-28T22:37:32.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.445+0000 7f24b43e4740 10 client.0 _ll_register_callbacks cb 0x560a2b4e9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:37:32.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.445+0000 7f24ab400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a2b872380 0x560a2b878850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.445+0000 7f24aa000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a2b872ca0 0x560a2b878d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.445+0000 7f24ab400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a2b872380 0x560a2b878850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:36580/0 (socket says 172.21.15.43:36580) 2024-03-28T22:37:32.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.445+0000 7f24ab400640 1 -- 172.21.15.43:0/4134628925 learned_addr learned my addr 172.21.15.43:0/4134628925 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:32.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.445+0000 7f24ab400640 1 --2- 172.21.15.43:0/4134628925 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a2b87b880 0x560a2b8792d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.446+0000 7f24aa000640 1 -- 172.21.15.43:0/4134628925 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a2b87b880 msgr2=0x560a2b8792d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:32.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.446+0000 7f24aa000640 1 --2- 172.21.15.43:0/4134628925 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a2b87b880 0x560a2b8792d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.446+0000 7f24aa000640 1 -- 172.21.15.43:0/4134628925 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a2b872380 msgr2=0x560a2b878850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:32.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.446+0000 7f24aa000640 1 --2- 172.21.15.43:0/4134628925 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a2b872380 0x560a2b878850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.446+0000 7f24aa000640 1 -- 172.21.15.43:0/4134628925 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a2b5b47d0 con 0x560a2b872ca0 2024-03-28T22:37:32.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.446+0000 7f24ab400640 1 --2- 172.21.15.43:0/4134628925 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a2b872380 0x560a2b878850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:37:32.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.446+0000 7f24aa000640 1 --2- 172.21.15.43:0/4134628925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a2b872ca0 0x560a2b878d90 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2498013760 tx=0x7f2498013c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59ab3f568ceaf514 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:32.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.446+0000 7f2494c00640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:37:32.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.447+0000 7f2494c00640 1 -- 172.21.15.43:0/4134628925 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2498020060 con 0x560a2b872ca0 2024-03-28T22:37:32.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.447+0000 7f2494c00640 1 -- 172.21.15.43:0/4134628925 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f2498011dd0 con 0x560a2b872ca0 2024-03-28T22:37:32.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.447+0000 7f24b43e4740 1 -- 172.21.15.43:0/4134628925 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24a4004050 con 0x560a2b872ca0 2024-03-28T22:37:32.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.447+0000 7f2494c00640 1 -- 172.21.15.43:0/4134628925 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2498027050 con 0x560a2b872ca0 2024-03-28T22:37:32.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.447+0000 7f24b43e4740 10 client.15684 fetch_fsmap learned FSMap version 29 2024-03-28T22:37:32.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.447+0000 7f24b43e4740 1 -- 172.21.15.43:0/4134628925 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560a2b88a940 con 0x560a2b872ca0 2024-03-28T22:37:32.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.447+0000 7f2494c00640 1 -- 172.21.15.43:0/4134628925 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24a4004050 con 0x560a2b872ca0 2024-03-28T22:37:32.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.448+0000 7f2494c00640 1 -- 172.21.15.43:0/4134628925 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f249801a030 con 0x560a2b872ca0 2024-03-28T22:37:32.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.448+0000 7f24b43e4740 10 client.15684 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:37:32.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.448+0000 7f24b43e4740 10 client.15684 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:37:32.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.448+0000 7f24b43e4740 20 client.15684 populate_metadata read hostname 'smithi043' 2024-03-28T22:37:32.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.448+0000 7f24b43e4740 1 --2- 172.21.15.43:0/4134628925 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x560a2b88be00 0x560a2b88e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.448+0000 7f24b43e4740 4 client.15684 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:37:32.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.448+0000 7f24b43e4740 1 -- 172.21.15.43:0/4134628925 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560a2b77f680 con 0x560a2b88be00 2024-03-28T22:37:32.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.448+0000 7f24ab400640 1 --2- 172.21.15.43:0/4134628925 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x560a2b88be00 0x560a2b88e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.448+0000 7f24ab400640 1 --2- 172.21.15.43:0/4134628925 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x560a2b88be00 0x560a2b88e1f0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b7765df91a505e5a server_cookie=2408cbaaeac3c70b in_seq=0 out_seq=0 2024-03-28T22:37:32.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.448+0000 7f2494c00640 10 client.15684 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:37:32.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.449+0000 7f2494c00640 1 -- 172.21.15.43:0/4134628925 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560a2b77f680 con 0x560a2b88be00 2024-03-28T22:37:32.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.449+0000 7f2494c00640 10 client.15684 handle_command_reply: tid=0 2024-03-28T22:37:32.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.449+0000 7f24b43e4740 1 client.15684 shutdown 2024-03-28T22:37:32.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.449+0000 7f24b43e4740 1 -- 172.21.15.43:0/4134628925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a2b872ca0 msgr2=0x560a2b878d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:32.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.449+0000 7f24b43e4740 1 --2- 172.21.15.43:0/4134628925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a2b872ca0 0x560a2b878d90 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2498013760 tx=0x7f2498013c20 comp rx=0 tx=0).stop 2024-03-28T22:37:32.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.449+0000 7f24b43e4740 1 -- 172.21.15.43:0/4134628925 shutdown_connections 2024-03-28T22:37:32.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.449+0000 7f24b43e4740 1 --2- 172.21.15.43:0/4134628925 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x560a2b88be00 0x560a2b88e1f0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.450+0000 7f24b43e4740 1 --2- 172.21.15.43:0/4134628925 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a2b87b880 0x560a2b8792d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.450+0000 7f24b43e4740 1 --2- 172.21.15.43:0/4134628925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a2b872ca0 0x560a2b878d90 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.450+0000 7f24b43e4740 1 --2- 172.21.15.43:0/4134628925 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a2b872380 0x560a2b878850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.450+0000 7f2494c00640 0 client.15684 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:37:32.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.450+0000 7f24b43e4740 1 -- 172.21.15.43:0/4134628925 >> 172.21.15.43:0/4134628925 conn(0x560a2b86cab0 msgr2=0x560a2b86e800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:32.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.450+0000 7f24b43e4740 1 -- 172.21.15.43:0/4134628925 shutdown_connections 2024-03-28T22:37:32.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.450+0000 7f24b43e4740 1 -- 172.21.15.43:0/4134628925 wait complete. 2024-03-28T22:37:32.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.450+0000 7f24b43e4740 20 client.15684 trim_cache size 0 max 16384 2024-03-28T22:37:32.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.465+0000 7f24b43e4740 1 Processor -- start 2024-03-28T22:37:32.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.465+0000 7f24b43e4740 1 -- start start 2024-03-28T22:37:32.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.465+0000 7f24b43e4740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a2b872380 0x560a2b87af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.465+0000 7f24b43e4740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a2b872ca0 0x560a2b887440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.465+0000 7f24b43e4740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a2b87b880 0x560a2b893b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.465+0000 7f24b43e4740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f24ac1b5230 con 0x560a2b87b880 2024-03-28T22:37:32.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.465+0000 7f24b43e4740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f24ac1b6d70 con 0x560a2b872ca0 2024-03-28T22:37:32.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.465+0000 7f24b43e4740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f24ac1b5900 con 0x560a2b872380 2024-03-28T22:37:32.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.465+0000 7f24aa000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a2b872ca0 0x560a2b887440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.465+0000 7f24aa000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a2b872ca0 0x560a2b887440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:36592/0 (socket says 172.21.15.43:36592) 2024-03-28T22:37:32.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.465+0000 7f24aa000640 1 -- 172.21.15.43:0/1545909773 learned_addr learned my addr 172.21.15.43:0/1545909773 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:32.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.466+0000 7f24ab400640 1 --2- 172.21.15.43:0/1545909773 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a2b872380 0x560a2b87af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.466+0000 7f24aa000640 1 -- 172.21.15.43:0/1545909773 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a2b872380 msgr2=0x560a2b87af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:32.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.466+0000 7f24aa000640 1 --2- 172.21.15.43:0/1545909773 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a2b872380 0x560a2b87af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.466+0000 7f24ab400640 1 --2- 172.21.15.43:0/1545909773 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a2b87b880 0x560a2b893b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.466+0000 7f24aa000640 1 -- 172.21.15.43:0/1545909773 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a2b87b880 msgr2=0x560a2b893b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:32.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.466+0000 7f24aa000640 1 --2- 172.21.15.43:0/1545909773 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a2b87b880 0x560a2b893b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.466+0000 7f24aa000640 1 -- 172.21.15.43:0/1545909773 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a2b872ad0 con 0x560a2b872ca0 2024-03-28T22:37:32.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.466+0000 7f24ab400640 1 --2- 172.21.15.43:0/1545909773 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a2b87b880 0x560a2b893b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:37:32.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.466+0000 7f24aa000640 1 --2- 172.21.15.43:0/1545909773 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a2b872ca0 0x560a2b887440 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f2498005860 tx=0x7f2498009de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31818aee998aea85 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:32.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.466+0000 7f2497400640 1 -- 172.21.15.43:0/1545909773 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2498030070 con 0x560a2b872ca0 2024-03-28T22:37:32.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.466+0000 7f2497400640 1 -- 172.21.15.43:0/1545909773 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f249800e430 con 0x560a2b872ca0 2024-03-28T22:37:32.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.466+0000 7f2497400640 1 -- 172.21.15.43:0/1545909773 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24980035f0 con 0x560a2b872ca0 2024-03-28T22:37:32.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.467+0000 7f24b43e4740 1 -- 172.21.15.43:0/1545909773 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a2b872ca0 msgr2=0x560a2b887440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:32.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.467+0000 7f24b43e4740 1 --2- 172.21.15.43:0/1545909773 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a2b872ca0 0x560a2b887440 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f2498005860 tx=0x7f2498009de0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.467+0000 7f24b43e4740 1 -- 172.21.15.43:0/1545909773 shutdown_connections 2024-03-28T22:37:32.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.467+0000 7f24b43e4740 1 --2- 172.21.15.43:0/1545909773 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a2b872ca0 0x560a2b887440 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.467+0000 7f24b43e4740 1 --2- 172.21.15.43:0/1545909773 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a2b87b880 0x560a2b893b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.467+0000 7f24b43e4740 1 --2- 172.21.15.43:0/1545909773 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a2b872380 0x560a2b87af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.467+0000 7f24b43e4740 1 -- 172.21.15.43:0/1545909773 >> 172.21.15.43:0/1545909773 conn(0x560a2b86cab0 msgr2=0x560a2b873c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:32.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.467+0000 7f24b43e4740 1 -- 172.21.15.43:0/1545909773 shutdown_connections 2024-03-28T22:37:32.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.467+0000 7f24b43e4740 1 -- 172.21.15.43:0/1545909773 wait complete. 2024-03-28T22:37:32.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.468+0000 7f24b43e4740 1 -- start start 2024-03-28T22:37:32.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.468+0000 7f24b43e4740 1 Processor -- start 2024-03-28T22:37:32.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.468+0000 7f24b43e4740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a2b872380 0x560a2b883d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.468+0000 7f24b43e4740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a2b872ca0 0x560a2b87f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.468+0000 7f24b43e4740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a2b87b880 0x560a2b87f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.468+0000 7f24b43e4740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f24ac1b66a0 con 0x560a2b872380 2024-03-28T22:37:32.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.468+0000 7f24b43e4740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f24ac1b5fd0 con 0x560a2b87b880 2024-03-28T22:37:32.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.468+0000 7f24b43e4740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f24ac1b3210 con 0x560a2b872ca0 2024-03-28T22:37:32.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.468+0000 7f24b43e4740 10 client.0 _ll_register_callbacks cb 0x7f2498009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:37:32.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.469+0000 7f24ab400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a2b872380 0x560a2b883d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.469+0000 7f24aa000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a2b872ca0 0x560a2b87f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.469+0000 7f24ab400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a2b872380 0x560a2b883d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46950/0 (socket says 172.21.15.43:46950) 2024-03-28T22:37:32.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.469+0000 7f24ab400640 1 -- 172.21.15.43:0/574106396 learned_addr learned my addr 172.21.15.43:0/574106396 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:32.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.469+0000 7f24ab400640 1 --2- 172.21.15.43:0/574106396 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a2b87b880 0x560a2b87f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.469+0000 7f24aa000640 1 -- 172.21.15.43:0/574106396 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a2b87b880 msgr2=0x560a2b87f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:32.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.469+0000 7f24aa000640 1 --2- 172.21.15.43:0/574106396 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a2b87b880 0x560a2b87f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.469+0000 7f24aa000640 1 -- 172.21.15.43:0/574106396 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a2b872380 msgr2=0x560a2b883d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:32.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.469+0000 7f24aa000640 1 --2- 172.21.15.43:0/574106396 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a2b872380 0x560a2b883d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.469+0000 7f24aa000640 1 -- 172.21.15.43:0/574106396 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a2b87fe80 con 0x560a2b872ca0 2024-03-28T22:37:32.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.469+0000 7f24ab400640 1 --2- 172.21.15.43:0/574106396 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a2b872380 0x560a2b883d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:37:32.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.469+0000 7f24aa000640 1 --2- 172.21.15.43:0/574106396 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a2b872ca0 0x560a2b87f0f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x560a2b86d8e0 tx=0x7f249800ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bf87b2d7cddab0f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:32.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.469+0000 7f2494c00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:37:32.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.470+0000 7f2494c00640 1 -- 172.21.15.43:0/574106396 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2498030070 con 0x560a2b872ca0 2024-03-28T22:37:32.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.470+0000 7f24b43e4740 1 -- 172.21.15.43:0/574106396 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f249800e430 con 0x560a2b872ca0 2024-03-28T22:37:32.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.470+0000 7f2494c00640 1 -- 172.21.15.43:0/574106396 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f2498005110 con 0x560a2b872ca0 2024-03-28T22:37:32.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.470+0000 7f2494c00640 1 -- 172.21.15.43:0/574106396 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2498020070 con 0x560a2b872ca0 2024-03-28T22:37:32.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.470+0000 7f24b43e4740 10 client.25798 fetch_fsmap learned FSMap version 29 2024-03-28T22:37:32.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.470+0000 7f24b43e4740 1 -- 172.21.15.43:0/574106396 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560a2b8f2130 con 0x560a2b872ca0 2024-03-28T22:37:32.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.470+0000 7f2494c00640 1 -- 172.21.15.43:0/574106396 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f249800e430 con 0x560a2b872ca0 2024-03-28T22:37:32.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.470+0000 7f2494c00640 1 -- 172.21.15.43:0/574106396 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f249801a050 con 0x560a2b872ca0 2024-03-28T22:37:32.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.470+0000 7f24b43e4740 10 client.25798 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:37:32.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.470+0000 7f24b43e4740 10 client.25798 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:37:32.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.470+0000 7f24b43e4740 20 client.25798 populate_metadata read hostname 'smithi043' 2024-03-28T22:37:32.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.471+0000 7f24b43e4740 1 --2- 172.21.15.43:0/574106396 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x560a2b8f2790 0x560a2b8f4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.471+0000 7f24b43e4740 4 client.25798 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:37:32.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.471+0000 7f24b43e4740 1 -- 172.21.15.43:0/574106396 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560a2b7e3590 con 0x560a2b8f2790 2024-03-28T22:37:32.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.471+0000 7f24ab400640 1 --2- 172.21.15.43:0/574106396 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x560a2b8f2790 0x560a2b8f4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.471+0000 7f24ab400640 1 --2- 172.21.15.43:0/574106396 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x560a2b8f2790 0x560a2b8f4b80 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3d2fb9401b985035 server_cookie=f71eac5521111f30 in_seq=0 out_seq=0 2024-03-28T22:37:32.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.471+0000 7f2494c00640 10 client.25798 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:37:32.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.471+0000 7f2494c00640 1 -- 172.21.15.43:0/574106396 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560a2b7e3590 con 0x560a2b8f2790 2024-03-28T22:37:32.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.471+0000 7f2494c00640 10 client.25798 handle_command_reply: tid=0 2024-03-28T22:37:32.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.472+0000 7f24b43e4740 1 client.25798 shutdown 2024-03-28T22:37:32.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.472+0000 7f24b43e4740 1 -- 172.21.15.43:0/574106396 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a2b872ca0 msgr2=0x560a2b87f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:32.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.472+0000 7f24b43e4740 1 --2- 172.21.15.43:0/574106396 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a2b872ca0 0x560a2b87f0f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x560a2b86d8e0 tx=0x7f249800ee60 comp rx=0 tx=0).stop 2024-03-28T22:37:32.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.472+0000 7f24b43e4740 1 -- 172.21.15.43:0/574106396 shutdown_connections 2024-03-28T22:37:32.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.472+0000 7f24b43e4740 1 --2- 172.21.15.43:0/574106396 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x560a2b8f2790 0x560a2b8f4b80 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.472+0000 7f24b43e4740 1 --2- 172.21.15.43:0/574106396 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a2b87b880 0x560a2b87f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.472+0000 7f24b43e4740 1 --2- 172.21.15.43:0/574106396 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a2b872ca0 0x560a2b87f0f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.472+0000 7f24b43e4740 1 --2- 172.21.15.43:0/574106396 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a2b872380 0x560a2b883d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.472+0000 7f2494c00640 0 client.25798 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:37:32.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.472+0000 7f24b43e4740 1 -- 172.21.15.43:0/574106396 >> 172.21.15.43:0/574106396 conn(0x560a2b86cab0 msgr2=0x560a2b86b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:32.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.472+0000 7f24b43e4740 1 -- 172.21.15.43:0/574106396 shutdown_connections 2024-03-28T22:37:32.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.472+0000 7f24b43e4740 1 -- 172.21.15.43:0/574106396 wait complete. 2024-03-28T22:37:32.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.472+0000 7f24b43e4740 20 client.25798 trim_cache size 0 max 16384 2024-03-28T22:37:32.474 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:37:32.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.473+0000 7f24b1800640 1 -- 172.21.15.43:0/1716273778 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f24900a1f80 msgr2=0x7f24900a4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:32.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.473+0000 7f24b1800640 1 --2- 172.21.15.43:0/1716273778 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f24900a1f80 0x7f24900a4440 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f249c0edb00 tx=0x7f249c0f6480 comp rx=0 tx=0).stop 2024-03-28T22:37:32.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.473+0000 7f24b1800640 1 -- 172.21.15.43:0/1716273778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24ac075c80 msgr2=0x7f24ac19e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:32.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.473+0000 7f24b1800640 1 --2- 172.21.15.43:0/1716273778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24ac075c80 0x7f24ac19e6f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f249c009900 tx=0x7f249c004360 comp rx=0 tx=0).stop 2024-03-28T22:37:32.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.473+0000 7f24b1800640 1 -- 172.21.15.43:0/1716273778 shutdown_connections 2024-03-28T22:37:32.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.473+0000 7f24b1800640 1 --2- 172.21.15.43:0/1716273778 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f24900a1f80 0x7f24900a4440 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.473+0000 7f24b1800640 1 --2- 172.21.15.43:0/1716273778 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24ac112590 0x7f24ac1a2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.473+0000 7f24b1800640 1 --2- 172.21.15.43:0/1716273778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24ac0765a0 0x7f24ac19ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.473+0000 7f24b1800640 1 --2- 172.21.15.43:0/1716273778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24ac075c80 0x7f24ac19e6f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.473+0000 7f24b1800640 1 -- 172.21.15.43:0/1716273778 >> 172.21.15.43:0/1716273778 conn(0x7f24ac0ff3a0 msgr2=0x7f24ac101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:32.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.473+0000 7f24b1800640 1 -- 172.21.15.43:0/1716273778 shutdown_connections 2024-03-28T22:37:32.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.473+0000 7f24b1800640 1 -- 172.21.15.43:0/1716273778 wait complete. 2024-03-28T22:37:32.489 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:37:32.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.710+0000 7febe1000640 1 -- 172.21.15.43:0/3332474315 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7febdc074410 msgr2=0x7febdc112080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:32.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.710+0000 7febe1000640 1 --2- 172.21.15.43:0/3332474315 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7febdc074410 0x7febdc112080 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7febc8009860 tx=0x7febc802bc00 comp rx=0 tx=0).stop 2024-03-28T22:37:32.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.710+0000 7febe1000640 1 -- 172.21.15.43:0/3332474315 shutdown_connections 2024-03-28T22:37:32.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.710+0000 7febe1000640 1 --2- 172.21.15.43:0/3332474315 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7febdc074410 0x7febdc112080 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.710+0000 7febe1000640 1 --2- 172.21.15.43:0/3332474315 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febdc1125c0 0x7febdc1149b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.710+0000 7febe1000640 1 --2- 172.21.15.43:0/3332474315 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7febdc073af0 0x7febdc073ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.710+0000 7febe1000640 1 -- 172.21.15.43:0/3332474315 >> 172.21.15.43:0/3332474315 conn(0x7febdc0fd0c0 msgr2=0x7febdc0ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:32.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.710+0000 7febe1000640 1 -- 172.21.15.43:0/3332474315 shutdown_connections 2024-03-28T22:37:32.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.710+0000 7febe1000640 1 -- 172.21.15.43:0/3332474315 wait complete. 2024-03-28T22:37:32.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.711+0000 7febe1000640 1 Processor -- start 2024-03-28T22:37:32.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.712+0000 7febe1000640 1 -- start start 2024-03-28T22:37:32.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.712+0000 7febe1000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7febdc073af0 0x7febdc19e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.712+0000 7febe1000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7febdc074410 0x7febdc19ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.712+0000 7febe1000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febdc1125c0 0x7febdc1a2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.712+0000 7febe1000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7febdc117fc0 con 0x7febdc074410 2024-03-28T22:37:32.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.712+0000 7febe1000640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7febdc117e40 con 0x7febdc073af0 2024-03-28T22:37:32.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.712+0000 7febe1000640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7febdc118140 con 0x7febdc1125c0 2024-03-28T22:37:32.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.712+0000 7febdaa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7febdc074410 0x7febdc19ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.712+0000 7febdbe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febdc1125c0 0x7febdc1a2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.712+0000 7febdb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7febdc073af0 0x7febdc19e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.712+0000 7febdaa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7febdc074410 0x7febdc19ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46968/0 (socket says 172.21.15.43:46968) 2024-03-28T22:37:32.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.712+0000 7febdaa00640 1 -- 172.21.15.43:0/2535295806 learned_addr learned my addr 172.21.15.43:0/2535295806 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:32.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.713+0000 7febdaa00640 1 -- 172.21.15.43:0/2535295806 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febdc1125c0 msgr2=0x7febdc1a2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:32.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.713+0000 7febdaa00640 1 --2- 172.21.15.43:0/2535295806 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febdc1125c0 0x7febdc1a2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.713+0000 7febdaa00640 1 -- 172.21.15.43:0/2535295806 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7febdc073af0 msgr2=0x7febdc19e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:32.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.713+0000 7febdaa00640 1 --2- 172.21.15.43:0/2535295806 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7febdc073af0 0x7febdc19e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.713+0000 7febdaa00640 1 -- 172.21.15.43:0/2535295806 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febdc1a34f0 con 0x7febdc074410 2024-03-28T22:37:32.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.713+0000 7febdb400640 1 --2- 172.21.15.43:0/2535295806 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7febdc073af0 0x7febdc19e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:37:32.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.713+0000 7febdaa00640 1 --2- 172.21.15.43:0/2535295806 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7febdc074410 0x7febdc19ea40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7febc8037040 tx=0x7febc80344c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:32.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.713+0000 7febd3e00640 1 -- 172.21.15.43:0/2535295806 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc8041020 con 0x7febdc074410 2024-03-28T22:37:32.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.713+0000 7febe1000640 1 -- 172.21.15.43:0/2535295806 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febdc1a3780 con 0x7febdc074410 2024-03-28T22:37:32.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.713+0000 7febd3e00640 1 -- 172.21.15.43:0/2535295806 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febc802ca80 con 0x7febdc074410 2024-03-28T22:37:32.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.714+0000 7febe1000640 1 -- 172.21.15.43:0/2535295806 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febdc1ab070 con 0x7febdc074410 2024-03-28T22:37:32.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.714+0000 7febd3e00640 1 -- 172.21.15.43:0/2535295806 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc8039040 con 0x7febdc074410 2024-03-28T22:37:32.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.717+0000 7febe5251740 1 Processor -- start 2024-03-28T22:37:32.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.717+0000 7febe5251740 1 -- start start 2024-03-28T22:37:32.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.717+0000 7febd3e00640 1 -- 172.21.15.43:0/2535295806 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 26) ==== 136932+0+0 (secure 0 0 0) 0x7febc8044020 con 0x7febdc074410 2024-03-28T22:37:32.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.717+0000 7febe5251740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56167c1ce550 0x56167c1ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.717+0000 7febe5251740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56167c1cee70 0x56167c1d7510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.718+0000 7febe5251740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56167c1d7a50 0x56167c1d9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.718+0000 7febe5251740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56167c132050 con 0x56167c1cee70 2024-03-28T22:37:32.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.718+0000 7febe5251740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x56167c0dace0 con 0x56167c1ce550 2024-03-28T22:37:32.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.718+0000 7febe5251740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x56167c12e0f0 con 0x56167c1d7a50 2024-03-28T22:37:32.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.718+0000 7febdbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56167c1ce550 0x56167c1ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.718+0000 7febdbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56167c1ce550 0x56167c1ce930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:36642/0 (socket says 172.21.15.43:36642) 2024-03-28T22:37:32.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.718+0000 7febdbe00640 1 -- 172.21.15.43:0/2030272260 learned_addr learned my addr 172.21.15.43:0/2030272260 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:32.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.718+0000 7febdb400640 1 --2- 172.21.15.43:0/2030272260 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56167c1cee70 0x56167c1d7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.718+0000 7febdaa00640 1 --2- 172.21.15.43:0/2030272260 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56167c1d7a50 0x56167c1d9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.718+0000 7febdbe00640 1 -- 172.21.15.43:0/2030272260 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56167c1d7a50 msgr2=0x56167c1d9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:32.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.718+0000 7febdbe00640 1 --2- 172.21.15.43:0/2030272260 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56167c1d7a50 0x56167c1d9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.718+0000 7febdbe00640 1 -- 172.21.15.43:0/2030272260 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56167c1cee70 msgr2=0x56167c1d7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:32.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.718+0000 7febdbe00640 1 --2- 172.21.15.43:0/2030272260 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56167c1cee70 0x56167c1d7510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.718+0000 7febdbe00640 1 -- 172.21.15.43:0/2030272260 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56167be463f0 con 0x56167c1ce550 2024-03-28T22:37:32.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.719+0000 7febdbe00640 1 --2- 172.21.15.43:0/2030272260 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56167c1ce550 0x56167c1ce930 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7febd4009be0 tx=0x7febd4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8cd7e89932ac197 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:32.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.719+0000 7febd0c00640 1 -- 172.21.15.43:0/2030272260 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd401e070 con 0x56167c1ce550 2024-03-28T22:37:32.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.719+0000 7febd0c00640 1 -- 172.21.15.43:0/2030272260 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febd4009d90 con 0x56167c1ce550 2024-03-28T22:37:32.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.719+0000 7febd0c00640 1 -- 172.21.15.43:0/2030272260 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd40196b0 con 0x56167c1ce550 2024-03-28T22:37:32.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.719+0000 7febd3e00640 1 --2- 172.21.15.43:0/2535295806 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7febc00a1f80 0x7febc00a4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.719+0000 7febd3e00640 1 -- 172.21.15.43:0/2535295806 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7febc80efcf0 con 0x7febdc074410 2024-03-28T22:37:32.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.719+0000 7febdb400640 1 --2- 172.21.15.43:0/2535295806 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7febc00a1f80 0x7febc00a4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.720+0000 7febe5251740 1 -- 172.21.15.43:0/2030272260 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56167c1ce550 msgr2=0x56167c1ce930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:32.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.720+0000 7febe5251740 1 --2- 172.21.15.43:0/2030272260 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56167c1ce550 0x56167c1ce930 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7febd4009be0 tx=0x7febd4011810 comp rx=0 tx=0).stop 2024-03-28T22:37:32.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.720+0000 7febe5251740 1 -- 172.21.15.43:0/2030272260 shutdown_connections 2024-03-28T22:37:32.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.720+0000 7febe5251740 1 --2- 172.21.15.43:0/2030272260 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56167c1d7a50 0x56167c1d9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.720+0000 7febe5251740 1 --2- 172.21.15.43:0/2030272260 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56167c1cee70 0x56167c1d7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.720+0000 7febe5251740 1 --2- 172.21.15.43:0/2030272260 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56167c1ce550 0x56167c1ce930 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.720+0000 7febe5251740 1 -- 172.21.15.43:0/2030272260 >> 172.21.15.43:0/2030272260 conn(0x56167c1c8c80 msgr2=0x56167c1c9060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:32.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.720+0000 7febdb400640 1 --2- 172.21.15.43:0/2535295806 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7febc00a1f80 0x7febc00a4440 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x56167c1ce020 tx=0x7febcc009500 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:32.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.720+0000 7febe5251740 1 -- 172.21.15.43:0/2030272260 shutdown_connections 2024-03-28T22:37:32.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.720+0000 7febe5251740 1 -- 172.21.15.43:0/2030272260 wait complete. 2024-03-28T22:37:32.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.720+0000 7febe5251740 1 -- start start 2024-03-28T22:37:32.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.721+0000 7febe5251740 1 Processor -- start 2024-03-28T22:37:32.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.721+0000 7febe5251740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56167c1ce550 0x56167c1d4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.721+0000 7febe5251740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56167c1cee70 0x56167c1d5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.721+0000 7febe5251740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56167c1d7a50 0x56167c1d5590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.721+0000 7febe5251740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56167c1da840 con 0x56167c1cee70 2024-03-28T22:37:32.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.721+0000 7febe5251740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x56167c0d02f0 con 0x56167c1d7a50 2024-03-28T22:37:32.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.721+0000 7febe5251740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x56167c1da9c0 con 0x56167c1ce550 2024-03-28T22:37:32.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.721+0000 7febe5251740 10 client.0 _ll_register_callbacks cb 0x56167be45be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:37:32.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.721+0000 7febdbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56167c1d7a50 0x56167c1d5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.721+0000 7febdbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56167c1d7a50 0x56167c1d5590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:36650/0 (socket says 172.21.15.43:36650) 2024-03-28T22:37:32.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.721+0000 7febdaa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56167c1cee70 0x56167c1d5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.721+0000 7febdbe00640 1 -- 172.21.15.43:0/1920113034 learned_addr learned my addr 172.21.15.43:0/1920113034 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:32.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.722+0000 7febdbe00640 1 -- 172.21.15.43:0/1920113034 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56167c1ce550 msgr2=0x56167c1d4b10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T22:37:32.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.722+0000 7febdbe00640 1 --2- 172.21.15.43:0/1920113034 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56167c1ce550 0x56167c1d4b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.722+0000 7febdbe00640 1 -- 172.21.15.43:0/1920113034 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56167c1cee70 msgr2=0x56167c1d5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:32.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.722+0000 7febdbe00640 1 --2- 172.21.15.43:0/1920113034 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56167c1cee70 0x56167c1d5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.722+0000 7febdbe00640 1 -- 172.21.15.43:0/1920113034 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56167bf107d0 con 0x56167c1d7a50 2024-03-28T22:37:32.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.722+0000 7febdbe00640 1 --2- 172.21.15.43:0/1920113034 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56167c1d7a50 0x56167c1d5590 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7febd4009230 tx=0x7febd4005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e28dcfa60e4d2f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:32.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.722+0000 7febc5600640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:37:32.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.722+0000 7febc5600640 1 -- 172.21.15.43:0/1920113034 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd401e040 con 0x56167c1d7a50 2024-03-28T22:37:32.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.722+0000 7febc5600640 1 -- 172.21.15.43:0/1920113034 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7febd4016040 con 0x56167c1d7a50 2024-03-28T22:37:32.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.722+0000 7febe5251740 1 -- 172.21.15.43:0/1920113034 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7febd4009d90 con 0x56167c1d7a50 2024-03-28T22:37:32.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.722+0000 7febc5600640 1 -- 172.21.15.43:0/1920113034 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7febd4031050 con 0x56167c1d7a50 2024-03-28T22:37:32.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.723+0000 7febe5251740 10 client.25826 fetch_fsmap learned FSMap version 29 2024-03-28T22:37:32.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.723+0000 7febe5251740 1 -- 172.21.15.43:0/1920113034 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56167c1e6bc0 con 0x56167c1d7a50 2024-03-28T22:37:32.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.723+0000 7febc5600640 1 -- 172.21.15.43:0/1920113034 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febd4009d90 con 0x56167c1d7a50 2024-03-28T22:37:32.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.723+0000 7febc5600640 1 -- 172.21.15.43:0/1920113034 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7febd40046c0 con 0x56167c1d7a50 2024-03-28T22:37:32.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.723+0000 7febe5251740 10 client.25826 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:37:32.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.723+0000 7febe5251740 10 client.25826 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:37:32.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.723+0000 7febe5251740 20 client.25826 populate_metadata read hostname 'smithi043' 2024-03-28T22:37:32.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.723+0000 7febe5251740 1 --2- 172.21.15.43:0/1920113034 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x56167c1e8020 0x56167c1ea410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.723+0000 7febe5251740 4 client.25826 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:37:32.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.723+0000 7febe5251740 1 -- 172.21.15.43:0/1920113034 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56167c0db680 con 0x56167c1e8020 2024-03-28T22:37:32.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.726+0000 7febdb400640 1 --2- 172.21.15.43:0/1920113034 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x56167c1e8020 0x56167c1ea410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.731+0000 7febdb400640 1 --2- 172.21.15.43:0/1920113034 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x56167c1e8020 0x56167c1ea410 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=faf6381009e0382f server_cookie=b424cdb667eaa03a in_seq=0 out_seq=0 2024-03-28T22:37:32.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.731+0000 7febc5600640 10 client.25826 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:37:32.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.736+0000 7febc5600640 1 -- 172.21.15.43:0/1920113034 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56167c0db680 con 0x56167c1e8020 2024-03-28T22:37:32.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.736+0000 7febc5600640 10 client.25826 handle_command_reply: tid=0 2024-03-28T22:37:32.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.736+0000 7febe5251740 1 client.25826 shutdown 2024-03-28T22:37:32.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.736+0000 7febe5251740 1 -- 172.21.15.43:0/1920113034 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56167c1d7a50 msgr2=0x56167c1d5590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:32.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.736+0000 7febe5251740 1 --2- 172.21.15.43:0/1920113034 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56167c1d7a50 0x56167c1d5590 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7febd4009230 tx=0x7febd4005270 comp rx=0 tx=0).stop 2024-03-28T22:37:32.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.736+0000 7febe5251740 1 -- 172.21.15.43:0/1920113034 shutdown_connections 2024-03-28T22:37:32.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.736+0000 7febe5251740 1 --2- 172.21.15.43:0/1920113034 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x56167c1e8020 0x56167c1ea410 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.736+0000 7febe5251740 1 --2- 172.21.15.43:0/1920113034 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56167c1d7a50 0x56167c1d5590 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.736+0000 7febc5600640 0 client.25826 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:37:32.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.736+0000 7febe5251740 1 --2- 172.21.15.43:0/1920113034 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56167c1cee70 0x56167c1d5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.736+0000 7febe5251740 1 --2- 172.21.15.43:0/1920113034 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56167c1ce550 0x56167c1d4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.737+0000 7febe5251740 1 -- 172.21.15.43:0/1920113034 >> 172.21.15.43:0/1920113034 conn(0x56167c1c8c80 msgr2=0x56167c1ca9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:32.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.737+0000 7febe5251740 1 -- 172.21.15.43:0/1920113034 shutdown_connections 2024-03-28T22:37:32.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.737+0000 7febe5251740 1 -- 172.21.15.43:0/1920113034 wait complete. 2024-03-28T22:37:32.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.737+0000 7febe5251740 20 client.25826 trim_cache size 0 max 16384 2024-03-28T22:37:32.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.752+0000 7febe5251740 1 Processor -- start 2024-03-28T22:37:32.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.752+0000 7febe5251740 1 -- start start 2024-03-28T22:37:32.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.752+0000 7febe5251740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56167c1ce550 0x56167c1e35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.752+0000 7febe5251740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56167c1cee70 0x56167c1d53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.752+0000 7febe5251740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56167c1d7a50 0x56167c1efb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.752+0000 7febe5251740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7febdc1b4fd0 con 0x56167c1d7a50 2024-03-28T22:37:32.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.752+0000 7febe5251740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7febdc1b6b10 con 0x56167c1cee70 2024-03-28T22:37:32.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.752+0000 7febe5251740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7febdc1b56a0 con 0x56167c1ce550 2024-03-28T22:37:32.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.752+0000 7febdaa00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56167c1cee70 0x56167c1d53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.752+0000 7febdaa00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56167c1cee70 0x56167c1d53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:36654/0 (socket says 172.21.15.43:36654) 2024-03-28T22:37:32.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.752+0000 7febdaa00640 1 -- 172.21.15.43:0/895963547 learned_addr learned my addr 172.21.15.43:0/895963547 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:32.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.752+0000 7febdb400640 1 --2- 172.21.15.43:0/895963547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56167c1ce550 0x56167c1e35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.753+0000 7febdbe00640 1 --2- 172.21.15.43:0/895963547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56167c1d7a50 0x56167c1efb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.753+0000 7febdaa00640 1 -- 172.21.15.43:0/895963547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56167c1ce550 msgr2=0x56167c1e35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:32.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.753+0000 7febdaa00640 1 --2- 172.21.15.43:0/895963547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56167c1ce550 0x56167c1e35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.753+0000 7febdaa00640 1 -- 172.21.15.43:0/895963547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56167c1d7a50 msgr2=0x56167c1efb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:32.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.753+0000 7febdaa00640 1 --2- 172.21.15.43:0/895963547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56167c1d7a50 0x56167c1efb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.753+0000 7febdaa00640 1 -- 172.21.15.43:0/895963547 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56167c1c9850 con 0x56167c1cee70 2024-03-28T22:37:32.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.753+0000 7febdaa00640 1 --2- 172.21.15.43:0/895963547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56167c1cee70 0x56167c1d53b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7febc802c4b0 tx=0x7febc8005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=280036be2c13b940 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:32.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.753+0000 7febc7e00640 1 -- 172.21.15.43:0/895963547 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc80bbbe0 con 0x56167c1cee70 2024-03-28T22:37:32.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.753+0000 7febc7e00640 1 -- 172.21.15.43:0/895963547 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febc80bc980 con 0x56167c1cee70 2024-03-28T22:37:32.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.753+0000 7febc7e00640 1 -- 172.21.15.43:0/895963547 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc80ba580 con 0x56167c1cee70 2024-03-28T22:37:32.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.754+0000 7febe5251740 1 -- 172.21.15.43:0/895963547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56167c1cee70 msgr2=0x56167c1d53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:32.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.754+0000 7febe5251740 1 --2- 172.21.15.43:0/895963547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56167c1cee70 0x56167c1d53b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7febc802c4b0 tx=0x7febc8005740 comp rx=0 tx=0).stop 2024-03-28T22:37:32.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.754+0000 7febe5251740 1 -- 172.21.15.43:0/895963547 shutdown_connections 2024-03-28T22:37:32.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.754+0000 7febe5251740 1 --2- 172.21.15.43:0/895963547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56167c1cee70 0x56167c1d53b0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.754+0000 7febe5251740 1 --2- 172.21.15.43:0/895963547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56167c1d7a50 0x56167c1efb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.754+0000 7febe5251740 1 --2- 172.21.15.43:0/895963547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56167c1ce550 0x56167c1e35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.754+0000 7febe5251740 1 -- 172.21.15.43:0/895963547 >> 172.21.15.43:0/895963547 conn(0x56167c1c8c80 msgr2=0x56167c1c7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:32.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.754+0000 7febe5251740 1 -- 172.21.15.43:0/895963547 shutdown_connections 2024-03-28T22:37:32.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.754+0000 7febe5251740 1 -- 172.21.15.43:0/895963547 wait complete. 2024-03-28T22:37:32.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.754+0000 7febe5251740 1 -- start start 2024-03-28T22:37:32.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.755+0000 7febe5251740 1 Processor -- start 2024-03-28T22:37:32.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.755+0000 7febe5251740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56167c1ce550 0x56167c1df800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.755+0000 7febe5251740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56167c1cee70 0x56167c1dfd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.755+0000 7febe5251740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56167c1d7a50 0x56167c1dac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.755+0000 7febe5251740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7febdc1b6440 con 0x56167c1ce550 2024-03-28T22:37:32.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.755+0000 7febe5251740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7febdc1b5d70 con 0x56167c1d7a50 2024-03-28T22:37:32.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.755+0000 7febe5251740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7febdc1b2fb0 con 0x56167c1cee70 2024-03-28T22:37:32.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.755+0000 7febe5251740 10 client.0 _ll_register_callbacks cb 0x7febd401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:37:32.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.755+0000 7febdb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56167c1ce550 0x56167c1df800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.755+0000 7febdbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56167c1d7a50 0x56167c1dac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.755+0000 7febdbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56167c1d7a50 0x56167c1dac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:36656/0 (socket says 172.21.15.43:36656) 2024-03-28T22:37:32.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.756+0000 7febdbe00640 1 -- 172.21.15.43:0/1589864922 learned_addr learned my addr 172.21.15.43:0/1589864922 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:32.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.756+0000 7febdaa00640 1 --2- 172.21.15.43:0/1589864922 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56167c1cee70 0x56167c1dfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.756+0000 7febdbe00640 1 -- 172.21.15.43:0/1589864922 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56167c1cee70 msgr2=0x56167c1dfd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:32.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.756+0000 7febdbe00640 1 --2- 172.21.15.43:0/1589864922 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56167c1cee70 0x56167c1dfd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.756+0000 7febdbe00640 1 -- 172.21.15.43:0/1589864922 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56167c1ce550 msgr2=0x56167c1df800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:32.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.756+0000 7febdbe00640 1 --2- 172.21.15.43:0/1589864922 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56167c1ce550 0x56167c1df800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.756+0000 7febdbe00640 1 -- 172.21.15.43:0/1589864922 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56167c1db400 con 0x56167c1d7a50 2024-03-28T22:37:32.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.756+0000 7febdb400640 1 --2- 172.21.15.43:0/1589864922 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56167c1ce550 0x56167c1df800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:37:32.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.756+0000 7febdaa00640 1 --2- 172.21.15.43:0/1589864922 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56167c1cee70 0x56167c1dfd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:37:32.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.756+0000 7febdbe00640 1 --2- 172.21.15.43:0/1589864922 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56167c1d7a50 0x56167c1dac70 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7febd4011850 tx=0x7febd40058e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=199c13fb4557d51f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:32.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.756+0000 7febc5600640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:37:32.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.756+0000 7febc5600640 1 -- 172.21.15.43:0/1589864922 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd401e040 con 0x56167c1d7a50 2024-03-28T22:37:32.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.756+0000 7febe5251740 1 -- 172.21.15.43:0/1589864922 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7febc80bc980 con 0x56167c1d7a50 2024-03-28T22:37:32.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.756+0000 7febc5600640 1 -- 172.21.15.43:0/1589864922 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7febd40052e0 con 0x56167c1d7a50 2024-03-28T22:37:32.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.757+0000 7febc5600640 1 -- 172.21.15.43:0/1589864922 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7febd4011d10 con 0x56167c1d7a50 2024-03-28T22:37:32.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.757+0000 7febe5251740 10 client.25832 fetch_fsmap learned FSMap version 29 2024-03-28T22:37:32.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.757+0000 7febe5251740 1 -- 172.21.15.43:0/1589864922 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56167c24dbf0 con 0x56167c1d7a50 2024-03-28T22:37:32.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.757+0000 7febc5600640 1 -- 172.21.15.43:0/1589864922 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febc80bc980 con 0x56167c1d7a50 2024-03-28T22:37:32.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.757+0000 7febc5600640 1 -- 172.21.15.43:0/1589864922 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7febd4019bc0 con 0x56167c1d7a50 2024-03-28T22:37:32.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.757+0000 7febe5251740 10 client.25832 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:37:32.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.757+0000 7febe5251740 10 client.25832 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:37:32.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.757+0000 7febe5251740 20 client.25832 populate_metadata read hostname 'smithi043' 2024-03-28T22:37:32.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.757+0000 7febe5251740 1 --2- 172.21.15.43:0/1589864922 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x56167c24e310 0x56167c250730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:32.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.757+0000 7febe5251740 4 client.25832 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:37:32.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.757+0000 7febe5251740 1 -- 172.21.15.43:0/1589864922 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56167c13f590 con 0x56167c24e310 2024-03-28T22:37:32.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.758+0000 7febdb400640 1 --2- 172.21.15.43:0/1589864922 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x56167c24e310 0x56167c250730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:32.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.758+0000 7febdb400640 1 --2- 172.21.15.43:0/1589864922 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x56167c24e310 0x56167c250730 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dd926c8879e0acb9 server_cookie=f5db4b3cc8e7c4f5 in_seq=0 out_seq=0 2024-03-28T22:37:32.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.758+0000 7febc5600640 10 client.25832 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:37:32.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.758+0000 7febc5600640 1 -- 172.21.15.43:0/1589864922 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56167c13f590 con 0x56167c24e310 2024-03-28T22:37:32.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.758+0000 7febc5600640 10 client.25832 handle_command_reply: tid=0 2024-03-28T22:37:32.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.758+0000 7febe5251740 1 client.25832 shutdown 2024-03-28T22:37:32.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.758+0000 7febe5251740 1 -- 172.21.15.43:0/1589864922 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56167c1d7a50 msgr2=0x56167c1dac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:32.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.758+0000 7febe5251740 1 --2- 172.21.15.43:0/1589864922 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56167c1d7a50 0x56167c1dac70 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7febd4011850 tx=0x7febd40058e0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.758+0000 7febe5251740 1 -- 172.21.15.43:0/1589864922 shutdown_connections 2024-03-28T22:37:32.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.758+0000 7febe5251740 1 --2- 172.21.15.43:0/1589864922 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x56167c24e310 0x56167c250730 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.758+0000 7febe5251740 1 --2- 172.21.15.43:0/1589864922 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56167c1d7a50 0x56167c1dac70 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.758+0000 7febe5251740 1 --2- 172.21.15.43:0/1589864922 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56167c1cee70 0x56167c1dfd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.758+0000 7febc5600640 0 client.25832 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:37:32.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.758+0000 7febe5251740 1 --2- 172.21.15.43:0/1589864922 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56167c1ce550 0x56167c1df800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.759+0000 7febe5251740 1 -- 172.21.15.43:0/1589864922 >> 172.21.15.43:0/1589864922 conn(0x56167c1c8c80 msgr2=0x56167c1c9f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:32.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.759+0000 7febe5251740 1 -- 172.21.15.43:0/1589864922 shutdown_connections 2024-03-28T22:37:32.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.759+0000 7febe5251740 1 -- 172.21.15.43:0/1589864922 wait complete. 2024-03-28T22:37:32.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.759+0000 7febe5251740 20 client.25832 trim_cache size 0 max 16384 2024-03-28T22:37:32.761 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:37:32.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.759+0000 7febe1000640 1 -- 172.21.15.43:0/2535295806 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7febc00a1f80 msgr2=0x7febc00a4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:32.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.759+0000 7febe1000640 1 --2- 172.21.15.43:0/2535295806 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7febc00a1f80 0x7febc00a4440 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x56167c1ce020 tx=0x7febcc009500 comp rx=0 tx=0).stop 2024-03-28T22:37:32.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.759+0000 7febe1000640 1 -- 172.21.15.43:0/2535295806 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7febdc074410 msgr2=0x7febdc19ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:32.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.759+0000 7febe1000640 1 --2- 172.21.15.43:0/2535295806 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7febdc074410 0x7febdc19ea40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7febc8037040 tx=0x7febc80344c0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.760+0000 7febe1000640 1 -- 172.21.15.43:0/2535295806 shutdown_connections 2024-03-28T22:37:32.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.760+0000 7febe1000640 1 --2- 172.21.15.43:0/2535295806 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7febc00a1f80 0x7febc00a4440 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.760+0000 7febe1000640 1 --2- 172.21.15.43:0/2535295806 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7febdc1125c0 0x7febdc1a2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.760+0000 7febe1000640 1 --2- 172.21.15.43:0/2535295806 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7febdc074410 0x7febdc19ea40 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.760+0000 7febe1000640 1 --2- 172.21.15.43:0/2535295806 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7febdc073af0 0x7febdc19e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:32.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.760+0000 7febe1000640 1 -- 172.21.15.43:0/2535295806 >> 172.21.15.43:0/2535295806 conn(0x7febdc0fd0c0 msgr2=0x7febdc0fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:32.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.760+0000 7febe1000640 1 -- 172.21.15.43:0/2535295806 shutdown_connections 2024-03-28T22:37:32.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:32.760+0000 7febe1000640 1 -- 172.21.15.43:0/2535295806 wait complete. 2024-03-28T22:37:32.775 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:37:33.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.002+0000 7f3fedc00640 1 -- 172.21.15.43:0/92036734 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3fe81015d0 msgr2=0x7f3fe8101a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:33.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.002+0000 7f3fedc00640 1 --2- 172.21.15.43:0/92036734 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3fe81015d0 0x7f3fe8101a50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4009930 tx=0x7f3fd402bc20 comp rx=0 tx=0).stop 2024-03-28T22:37:33.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.002+0000 7f3fedc00640 1 -- 172.21.15.43:0/92036734 shutdown_connections 2024-03-28T22:37:33.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.002+0000 7f3fedc00640 1 --2- 172.21.15.43:0/92036734 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3fe8101f90 0x7f3fe810e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.002+0000 7f3fedc00640 1 --2- 172.21.15.43:0/92036734 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3fe81015d0 0x7f3fe8101a50 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.002+0000 7f3fedc00640 1 --2- 172.21.15.43:0/92036734 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3fe8107690 0x7f3fe8107a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.002+0000 7f3fedc00640 1 -- 172.21.15.43:0/92036734 >> 172.21.15.43:0/92036734 conn(0x7f3fe80fd0c0 msgr2=0x7f3fe80ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:33.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.002+0000 7f3fedc00640 1 -- 172.21.15.43:0/92036734 shutdown_connections 2024-03-28T22:37:33.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.002+0000 7f3fedc00640 1 -- 172.21.15.43:0/92036734 wait complete. 2024-03-28T22:37:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.003+0000 7f3fedc00640 1 Processor -- start 2024-03-28T22:37:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.003+0000 7f3fedc00640 1 -- start start 2024-03-28T22:37:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.004+0000 7f3fedc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3fe81015d0 0x7f3fe8073a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.004+0000 7f3fedc00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3fe8101f90 0x7f3fe8073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.004+0000 7f3fedc00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3fe8107690 0x7f3fe80744d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.004+0000 7f3fedc00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe8111a20 con 0x7f3fe81015d0 2024-03-28T22:37:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.004+0000 7f3fedc00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe81118a0 con 0x7f3fe8101f90 2024-03-28T22:37:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.004+0000 7f3fedc00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe8111ba0 con 0x7f3fe8107690 2024-03-28T22:37:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.004+0000 7f3fe7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3fe81015d0 0x7f3fe8073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.004+0000 7f3fe7400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3fe8101f90 0x7f3fe8073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.004+0000 7f3fe7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3fe81015d0 0x7f3fe8073a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47016/0 (socket says 172.21.15.43:47016) 2024-03-28T22:37:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.004+0000 7f3fe7e00640 1 -- 172.21.15.43:0/2630623200 learned_addr learned my addr 172.21.15.43:0/2630623200 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.004+0000 7f3fed200640 1 --2- 172.21.15.43:0/2630623200 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3fe8107690 0x7f3fe80744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.004+0000 7f3fe7400640 1 -- 172.21.15.43:0/2630623200 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3fe8107690 msgr2=0x7f3fe80744d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.004+0000 7f3fe7400640 1 --2- 172.21.15.43:0/2630623200 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3fe8107690 0x7f3fe80744d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.004+0000 7f3fe7400640 1 -- 172.21.15.43:0/2630623200 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3fe81015d0 msgr2=0x7f3fe8073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.004+0000 7f3fe7400640 1 --2- 172.21.15.43:0/2630623200 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3fe81015d0 0x7f3fe8073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.004+0000 7f3fe7400640 1 -- 172.21.15.43:0/2630623200 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fe81a6340 con 0x7f3fe8101f90 2024-03-28T22:37:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.005+0000 7f3fe7400640 1 --2- 172.21.15.43:0/2630623200 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3fe8101f90 0x7f3fe8073f90 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4036040 tx=0x7f3fd4004250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.005+0000 7f3fe4c00640 1 -- 172.21.15.43:0/2630623200 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd4043020 con 0x7f3fe8101f90 2024-03-28T22:37:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.005+0000 7f3fe4c00640 1 -- 172.21.15.43:0/2630623200 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fd4005000 con 0x7f3fe8101f90 2024-03-28T22:37:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.005+0000 7f3fe4c00640 1 -- 172.21.15.43:0/2630623200 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd4003a80 con 0x7f3fe8101f90 2024-03-28T22:37:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.005+0000 7f3fedc00640 1 -- 172.21.15.43:0/2630623200 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fe81a65d0 con 0x7f3fe8101f90 2024-03-28T22:37:33.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.005+0000 7f3fedc00640 1 -- 172.21.15.43:0/2630623200 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fe81a6aa0 con 0x7f3fe8101f90 2024-03-28T22:37:33.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.008+0000 7f3fe4c00640 1 -- 172.21.15.43:0/2630623200 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 26) ==== 136932+0+0 (secure 0 0 0) 0x7f3fd4046020 con 0x7f3fe8101f90 2024-03-28T22:37:33.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.009+0000 7f3ff1e82740 1 Processor -- start 2024-03-28T22:37:33.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.009+0000 7f3ff1e82740 1 -- start start 2024-03-28T22:37:33.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.009+0000 7f3ff1e82740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557a9a5c1550 0x557a9a5c1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.009+0000 7f3ff1e82740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557a9a5c1e70 0x557a9a5ca510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.009+0000 7f3ff1e82740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557a9a5caa50 0x557a9a5cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.009+0000 7f3ff1e82740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x557a9a525050 con 0x557a9a5c1550 2024-03-28T22:37:33.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.009+0000 7f3ff1e82740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x557a9a4cdce0 con 0x557a9a5c1e70 2024-03-28T22:37:33.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.009+0000 7f3ff1e82740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x557a9a5210f0 con 0x557a9a5caa50 2024-03-28T22:37:33.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.009+0000 7f3fe7e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557a9a5c1e70 0x557a9a5ca510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.009+0000 7f3fed200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557a9a5c1550 0x557a9a5c1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.009+0000 7f3fe7e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557a9a5c1e70 0x557a9a5ca510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:36686/0 (socket says 172.21.15.43:36686) 2024-03-28T22:37:33.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.009+0000 7f3fe7e00640 1 -- 172.21.15.43:0/3796974840 learned_addr learned my addr 172.21.15.43:0/3796974840 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:33.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.010+0000 7f3fe7e00640 1 -- 172.21.15.43:0/3796974840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557a9a5caa50 msgr2=0x557a9a5cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:33.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.010+0000 7f3fe7e00640 1 --2- 172.21.15.43:0/3796974840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557a9a5caa50 0x557a9a5cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.010+0000 7f3fe7e00640 1 -- 172.21.15.43:0/3796974840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557a9a5c1550 msgr2=0x557a9a5c1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:33.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.010+0000 7f3fe7e00640 1 --2- 172.21.15.43:0/3796974840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557a9a5c1550 0x557a9a5c1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.010+0000 7f3fe7e00640 1 -- 172.21.15.43:0/3796974840 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a9a2393f0 con 0x557a9a5c1e70 2024-03-28T22:37:33.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.010+0000 7f3fe7e00640 1 --2- 172.21.15.43:0/3796974840 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557a9a5c1e70 0x557a9a5ca510 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3fd8009da0 tx=0x7f3fd8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69b8a7b20d324424 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:33.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.010+0000 7f3fe4c00640 1 --2- 172.21.15.43:0/2630623200 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f3fcc0a2000 0x7f3fcc0a44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.010+0000 7f3fdd600640 1 -- 172.21.15.43:0/3796974840 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd801e070 con 0x557a9a5c1e70 2024-03-28T22:37:33.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.010+0000 7f3fdd600640 1 -- 172.21.15.43:0/3796974840 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fd8019b10 con 0x557a9a5c1e70 2024-03-28T22:37:33.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.010+0000 7f3fdd600640 1 -- 172.21.15.43:0/3796974840 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd800fb80 con 0x557a9a5c1e70 2024-03-28T22:37:33.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.010+0000 7f3fe4c00640 1 -- 172.21.15.43:0/2630623200 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f3fd40f0490 con 0x7f3fe8101f90 2024-03-28T22:37:33.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.011+0000 7f3fed200640 1 --2- 172.21.15.43:0/2630623200 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f3fcc0a2000 0x7f3fcc0a44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.011+0000 7f3ff1e82740 1 -- 172.21.15.43:0/3796974840 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557a9a5c1e70 msgr2=0x557a9a5ca510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:33.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.011+0000 7f3ff1e82740 1 --2- 172.21.15.43:0/3796974840 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557a9a5c1e70 0x557a9a5ca510 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3fd8009da0 tx=0x7f3fd8011810 comp rx=0 tx=0).stop 2024-03-28T22:37:33.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.011+0000 7f3ff1e82740 1 -- 172.21.15.43:0/3796974840 shutdown_connections 2024-03-28T22:37:33.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.011+0000 7f3ff1e82740 1 --2- 172.21.15.43:0/3796974840 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557a9a5c1e70 0x557a9a5ca510 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.011+0000 7f3ff1e82740 1 --2- 172.21.15.43:0/3796974840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557a9a5caa50 0x557a9a5cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.011+0000 7f3ff1e82740 1 --2- 172.21.15.43:0/3796974840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557a9a5c1550 0x557a9a5c1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.011+0000 7f3ff1e82740 1 -- 172.21.15.43:0/3796974840 >> 172.21.15.43:0/3796974840 conn(0x557a9a5bbc80 msgr2=0x557a9a5bc060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:33.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.011+0000 7f3ff1e82740 1 -- 172.21.15.43:0/3796974840 shutdown_connections 2024-03-28T22:37:33.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.011+0000 7f3ff1e82740 1 -- 172.21.15.43:0/3796974840 wait complete. 2024-03-28T22:37:33.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.011+0000 7f3fed200640 1 --2- 172.21.15.43:0/2630623200 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f3fcc0a2000 0x7f3fcc0a44c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3fe0005fd0 tx=0x7f3fe0009500 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:33.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.012+0000 7f3ff1e82740 1 -- start start 2024-03-28T22:37:33.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.012+0000 7f3ff1e82740 1 Processor -- start 2024-03-28T22:37:33.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.013+0000 7f3ff1e82740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557a9a5c1550 0x557a9a5c7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.013+0000 7f3ff1e82740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557a9a5c1e70 0x557a9a5c7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.013+0000 7f3ff1e82740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557a9a5caa50 0x557a9a5c8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.013+0000 7f3ff1e82740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x557a9a5cd840 con 0x557a9a5caa50 2024-03-28T22:37:33.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.013+0000 7f3ff1e82740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x557a9a4c32f0 con 0x557a9a5c1e70 2024-03-28T22:37:33.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.013+0000 7f3ff1e82740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x557a9a5cd9c0 con 0x557a9a5c1550 2024-03-28T22:37:33.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.013+0000 7f3ff1e82740 10 client.0 _ll_register_callbacks cb 0x557a9a238be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:37:33.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.013+0000 7f3fe7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557a9a5c1550 0x557a9a5c7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.013+0000 7f3fe7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557a9a5c1550 0x557a9a5c7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:52538/0 (socket says 172.21.15.43:52538) 2024-03-28T22:37:33.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.013+0000 7f3fe7e00640 1 -- 172.21.15.43:0/1693465908 learned_addr learned my addr 172.21.15.43:0/1693465908 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:33.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.013+0000 7f3fe7400640 1 --2- 172.21.15.43:0/1693465908 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557a9a5c1e70 0x557a9a5c7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.013+0000 7f3fed200640 1 --2- 172.21.15.43:0/1693465908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557a9a5caa50 0x557a9a5c8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.013+0000 7f3fe7e00640 1 -- 172.21.15.43:0/1693465908 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557a9a5c1e70 msgr2=0x557a9a5c7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:33.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.013+0000 7f3fe7e00640 1 --2- 172.21.15.43:0/1693465908 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557a9a5c1e70 0x557a9a5c7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.013+0000 7f3fe7e00640 1 -- 172.21.15.43:0/1693465908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557a9a5caa50 msgr2=0x557a9a5c8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:33.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.013+0000 7f3fe7e00640 1 --2- 172.21.15.43:0/1693465908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557a9a5caa50 0x557a9a5c8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.013+0000 7f3fe7e00640 1 -- 172.21.15.43:0/1693465908 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a9a3037d0 con 0x557a9a5c1550 2024-03-28T22:37:33.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.013+0000 7f3fed200640 1 --2- 172.21.15.43:0/1693465908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557a9a5caa50 0x557a9a5c8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:37:33.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.014+0000 7f3fe7e00640 1 --2- 172.21.15.43:0/1693465908 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557a9a5c1550 0x557a9a5c7a10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3fd800f4e0 tx=0x7f3fd800f540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2577adb112afc8d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:33.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.014+0000 7f3fd2000640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:37:33.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.014+0000 7f3fd2000640 1 -- 172.21.15.43:0/1693465908 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd8026070 con 0x557a9a5c1550 2024-03-28T22:37:33.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.014+0000 7f3fd2000640 1 -- 172.21.15.43:0/1693465908 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f3fd8019840 con 0x557a9a5c1550 2024-03-28T22:37:33.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.014+0000 7f3ff1e82740 1 -- 172.21.15.43:0/1693465908 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3fd8019b10 con 0x557a9a5c1550 2024-03-28T22:37:33.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.014+0000 7f3fd2000640 1 -- 172.21.15.43:0/1693465908 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3fd8032050 con 0x557a9a5c1550 2024-03-28T22:37:33.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.014+0000 7f3ff1e82740 10 client.25819 fetch_fsmap learned FSMap version 29 2024-03-28T22:37:33.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.014+0000 7f3ff1e82740 1 -- 172.21.15.43:0/1693465908 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557a9a5d9ab0 con 0x557a9a5c1550 2024-03-28T22:37:33.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.014+0000 7f3fd2000640 1 -- 172.21.15.43:0/1693465908 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fd8019b10 con 0x557a9a5c1550 2024-03-28T22:37:33.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.015+0000 7f3fd2000640 1 -- 172.21.15.43:0/1693465908 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f3fd80046c0 con 0x557a9a5c1550 2024-03-28T22:37:33.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.015+0000 7f3ff1e82740 10 client.25819 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:37:33.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.015+0000 7f3ff1e82740 10 client.25819 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:37:33.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.015+0000 7f3ff1e82740 20 client.25819 populate_metadata read hostname 'smithi043' 2024-03-28T22:37:33.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.015+0000 7f3ff1e82740 1 --2- 172.21.15.43:0/1693465908 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x557a9a5dae00 0x557a9a5dd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.015+0000 7f3ff1e82740 4 client.25819 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:37:33.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.015+0000 7f3ff1e82740 1 -- 172.21.15.43:0/1693465908 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a9a4ce680 con 0x557a9a5dae00 2024-03-28T22:37:33.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.015+0000 7f3fe7400640 1 --2- 172.21.15.43:0/1693465908 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x557a9a5dae00 0x557a9a5dd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.017+0000 7f3fe7400640 1 --2- 172.21.15.43:0/1693465908 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x557a9a5dae00 0x557a9a5dd1f0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fff6699515486ef server_cookie=5f94d34049207ac9 in_seq=0 out_seq=0 2024-03-28T22:37:33.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.017+0000 7f3fd2000640 10 client.25819 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:37:33.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.019+0000 7f3fd2000640 1 -- 172.21.15.43:0/1693465908 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557a9a4ce680 con 0x557a9a5dae00 2024-03-28T22:37:33.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.019+0000 7f3fd2000640 10 client.25819 handle_command_reply: tid=0 2024-03-28T22:37:33.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.019+0000 7f3ff1e82740 1 client.25819 shutdown 2024-03-28T22:37:33.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.019+0000 7f3ff1e82740 1 -- 172.21.15.43:0/1693465908 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557a9a5c1550 msgr2=0x557a9a5c7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:33.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.019+0000 7f3ff1e82740 1 --2- 172.21.15.43:0/1693465908 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557a9a5c1550 0x557a9a5c7a10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3fd800f4e0 tx=0x7f3fd800f540 comp rx=0 tx=0).stop 2024-03-28T22:37:33.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.019+0000 7f3ff1e82740 1 -- 172.21.15.43:0/1693465908 shutdown_connections 2024-03-28T22:37:33.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.019+0000 7f3ff1e82740 1 --2- 172.21.15.43:0/1693465908 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x557a9a5dae00 0x557a9a5dd1f0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.019+0000 7f3ff1e82740 1 --2- 172.21.15.43:0/1693465908 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557a9a5c1e70 0x557a9a5c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.019+0000 7f3ff1e82740 1 --2- 172.21.15.43:0/1693465908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557a9a5caa50 0x557a9a5c8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.019+0000 7f3ff1e82740 1 --2- 172.21.15.43:0/1693465908 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557a9a5c1550 0x557a9a5c7a10 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.019+0000 7f3fd2000640 0 client.25819 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:37:33.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.019+0000 7f3ff1e82740 1 -- 172.21.15.43:0/1693465908 >> 172.21.15.43:0/1693465908 conn(0x557a9a5bbc80 msgr2=0x557a9a5bd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:33.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.019+0000 7f3ff1e82740 1 -- 172.21.15.43:0/1693465908 shutdown_connections 2024-03-28T22:37:33.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.019+0000 7f3ff1e82740 1 -- 172.21.15.43:0/1693465908 wait complete. 2024-03-28T22:37:33.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.019+0000 7f3ff1e82740 20 client.25819 trim_cache size 0 max 16384 2024-03-28T22:37:33.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.034+0000 7f3ff1e82740 1 Processor -- start 2024-03-28T22:37:33.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.034+0000 7f3ff1e82740 1 -- start start 2024-03-28T22:37:33.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.034+0000 7f3ff1e82740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557a9a5c1550 0x557a9a5bdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.034+0000 7f3ff1e82740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557a9a5c1e70 0x557a9a5d65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.034+0000 7f3ff1e82740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557a9a5caa50 0x557a9a5c82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.034+0000 7f3ff1e82740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe81b0b40 con 0x557a9a5c1e70 2024-03-28T22:37:33.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.034+0000 7f3ff1e82740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe81b2680 con 0x557a9a5c1550 2024-03-28T22:37:33.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.034+0000 7f3ff1e82740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe81b1210 con 0x557a9a5caa50 2024-03-28T22:37:33.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.035+0000 7f3fed200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557a9a5caa50 0x557a9a5c82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.035+0000 7f3fed200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557a9a5caa50 0x557a9a5c82b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:52540/0 (socket says 172.21.15.43:52540) 2024-03-28T22:37:33.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.035+0000 7f3fed200640 1 -- 172.21.15.43:0/2759600625 learned_addr learned my addr 172.21.15.43:0/2759600625 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:33.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.035+0000 7f3fe7e00640 1 --2- 172.21.15.43:0/2759600625 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557a9a5c1550 0x557a9a5bdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.035+0000 7f3fe7400640 1 --2- 172.21.15.43:0/2759600625 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557a9a5c1e70 0x557a9a5d65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.035+0000 7f3fed200640 1 -- 172.21.15.43:0/2759600625 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557a9a5c1550 msgr2=0x557a9a5bdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:33.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.035+0000 7f3fed200640 1 --2- 172.21.15.43:0/2759600625 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557a9a5c1550 0x557a9a5bdb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.035+0000 7f3fed200640 1 -- 172.21.15.43:0/2759600625 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557a9a5c1e70 msgr2=0x557a9a5d65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:33.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.035+0000 7f3fed200640 1 --2- 172.21.15.43:0/2759600625 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557a9a5c1e70 0x557a9a5d65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.035+0000 7f3fed200640 1 -- 172.21.15.43:0/2759600625 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a9a5dd780 con 0x557a9a5caa50 2024-03-28T22:37:33.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.035+0000 7f3fe7e00640 1 --2- 172.21.15.43:0/2759600625 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557a9a5c1550 0x557a9a5bdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:37:33.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.035+0000 7f3fed200640 1 --2- 172.21.15.43:0/2759600625 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557a9a5caa50 0x557a9a5c82b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3fe0003de0 tx=0x7f3fe0016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f6a7b2f51d1ae94 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:33.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.036+0000 7f3fdcc00640 1 -- 172.21.15.43:0/2759600625 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe0004b20 con 0x557a9a5caa50 2024-03-28T22:37:33.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.036+0000 7f3fdcc00640 1 -- 172.21.15.43:0/2759600625 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fe0004cc0 con 0x557a9a5caa50 2024-03-28T22:37:33.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.036+0000 7f3fdcc00640 1 -- 172.21.15.43:0/2759600625 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe00051a0 con 0x557a9a5caa50 2024-03-28T22:37:33.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.036+0000 7f3ff1e82740 1 -- 172.21.15.43:0/2759600625 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557a9a5caa50 msgr2=0x557a9a5c82b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:33.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.036+0000 7f3ff1e82740 1 --2- 172.21.15.43:0/2759600625 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557a9a5caa50 0x557a9a5c82b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3fe0003de0 tx=0x7f3fe0016000 comp rx=0 tx=0).stop 2024-03-28T22:37:33.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.036+0000 7f3ff1e82740 1 -- 172.21.15.43:0/2759600625 shutdown_connections 2024-03-28T22:37:33.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.036+0000 7f3ff1e82740 1 --2- 172.21.15.43:0/2759600625 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557a9a5caa50 0x557a9a5c82b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.036+0000 7f3ff1e82740 1 --2- 172.21.15.43:0/2759600625 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557a9a5c1e70 0x557a9a5d65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.036+0000 7f3ff1e82740 1 --2- 172.21.15.43:0/2759600625 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557a9a5c1550 0x557a9a5bdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.036+0000 7f3ff1e82740 1 -- 172.21.15.43:0/2759600625 >> 172.21.15.43:0/2759600625 conn(0x557a9a5bbc80 msgr2=0x557a9a5badc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:33.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.036+0000 7f3ff1e82740 1 -- 172.21.15.43:0/2759600625 shutdown_connections 2024-03-28T22:37:33.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.037+0000 7f3ff1e82740 1 -- 172.21.15.43:0/2759600625 wait complete. 2024-03-28T22:37:33.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.037+0000 7f3ff1e82740 1 -- start start 2024-03-28T22:37:33.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.037+0000 7f3ff1e82740 1 Processor -- start 2024-03-28T22:37:33.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.037+0000 7f3ff1e82740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557a9a5c1550 0x557a9a639350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.038+0000 7f3ff1e82740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557a9a5c1e70 0x557a9a639890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.038+0000 7f3ff1e82740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557a9a5caa50 0x557a9a5cdd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.038+0000 7f3ff1e82740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe81b1fb0 con 0x557a9a5caa50 2024-03-28T22:37:33.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.038+0000 7f3ff1e82740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe81b18e0 con 0x557a9a5c1e70 2024-03-28T22:37:33.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.038+0000 7f3ff1e82740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe81aeb20 con 0x557a9a5c1550 2024-03-28T22:37:33.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.038+0000 7f3ff1e82740 10 client.0 _ll_register_callbacks cb 0x7f3fd8009c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:37:33.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.038+0000 7f3fe7400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557a9a5c1e70 0x557a9a639890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.038+0000 7f3fe7400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557a9a5c1e70 0x557a9a639890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:36718/0 (socket says 172.21.15.43:36718) 2024-03-28T22:37:33.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.038+0000 7f3fe7400640 1 -- 172.21.15.43:0/2300130063 learned_addr learned my addr 172.21.15.43:0/2300130063 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:33.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.038+0000 7f3fed200640 1 --2- 172.21.15.43:0/2300130063 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557a9a5caa50 0x557a9a5cdd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.038+0000 7f3fe7e00640 1 --2- 172.21.15.43:0/2300130063 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557a9a5c1550 0x557a9a639350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.038+0000 7f3fe7400640 1 -- 172.21.15.43:0/2300130063 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557a9a5c1550 msgr2=0x557a9a639350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:33.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.038+0000 7f3fe7400640 1 --2- 172.21.15.43:0/2300130063 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557a9a5c1550 0x557a9a639350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.038+0000 7f3fe7400640 1 -- 172.21.15.43:0/2300130063 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557a9a5caa50 msgr2=0x557a9a5cdd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:33.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.038+0000 7f3fe7400640 1 --2- 172.21.15.43:0/2300130063 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557a9a5caa50 0x557a9a5cdd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.038+0000 7f3fe7400640 1 -- 172.21.15.43:0/2300130063 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a9a5ce420 con 0x557a9a5c1e70 2024-03-28T22:37:33.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.038+0000 7f3fe7e00640 1 --2- 172.21.15.43:0/2300130063 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557a9a5c1550 0x557a9a639350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:37:33.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.039+0000 7f3fe7400640 1 --2- 172.21.15.43:0/2300130063 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557a9a5c1e70 0x557a9a639890 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4009930 tx=0x7f3fd40f2040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=feba1dfadf27f2fa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:33.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.039+0000 7f3fd2000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:37:33.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.039+0000 7f3fd2000640 1 -- 172.21.15.43:0/2300130063 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd40d4070 con 0x557a9a5c1e70 2024-03-28T22:37:33.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.039+0000 7f3fd2000640 1 -- 172.21.15.43:0/2300130063 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f3fd40f0120 con 0x557a9a5c1e70 2024-03-28T22:37:33.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.039+0000 7f3ff1e82740 1 -- 172.21.15.43:0/2300130063 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3fe0004cc0 con 0x557a9a5c1e70 2024-03-28T22:37:33.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.039+0000 7f3fd2000640 1 -- 172.21.15.43:0/2300130063 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3fd40c51c0 con 0x557a9a5c1e70 2024-03-28T22:37:33.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.039+0000 7f3ff1e82740 10 client.25850 fetch_fsmap learned FSMap version 29 2024-03-28T22:37:33.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.039+0000 7f3ff1e82740 1 -- 172.21.15.43:0/2300130063 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557a9a640b50 con 0x557a9a5c1e70 2024-03-28T22:37:33.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.039+0000 7f3fd2000640 1 -- 172.21.15.43:0/2300130063 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fe0004cc0 con 0x557a9a5c1e70 2024-03-28T22:37:33.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.039+0000 7f3fd2000640 1 -- 172.21.15.43:0/2300130063 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f3fd40bcc80 con 0x557a9a5c1e70 2024-03-28T22:37:33.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.039+0000 7f3ff1e82740 10 client.25850 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:37:33.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.039+0000 7f3ff1e82740 10 client.25850 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:37:33.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.039+0000 7f3ff1e82740 20 client.25850 populate_metadata read hostname 'smithi043' 2024-03-28T22:37:33.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.039+0000 7f3ff1e82740 1 --2- 172.21.15.43:0/2300130063 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x557a9a6412f0 0x557a9a643710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.039+0000 7f3ff1e82740 4 client.25850 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:37:33.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.039+0000 7f3ff1e82740 1 -- 172.21.15.43:0/2300130063 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557a9a532590 con 0x557a9a6412f0 2024-03-28T22:37:33.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.040+0000 7f3fe7e00640 1 --2- 172.21.15.43:0/2300130063 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x557a9a6412f0 0x557a9a643710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.040+0000 7f3fe7e00640 1 --2- 172.21.15.43:0/2300130063 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x557a9a6412f0 0x557a9a643710 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6249257dd755bbd6 server_cookie=3fda07a706f1b1b7 in_seq=0 out_seq=0 2024-03-28T22:37:33.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.040+0000 7f3fd2000640 10 client.25850 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:37:33.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.040+0000 7f3fd2000640 1 -- 172.21.15.43:0/2300130063 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557a9a532590 con 0x557a9a6412f0 2024-03-28T22:37:33.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.040+0000 7f3fd2000640 10 client.25850 handle_command_reply: tid=0 2024-03-28T22:37:33.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.040+0000 7f3ff1e82740 1 client.25850 shutdown 2024-03-28T22:37:33.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.041+0000 7f3ff1e82740 1 -- 172.21.15.43:0/2300130063 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557a9a5c1e70 msgr2=0x557a9a639890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:33.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.041+0000 7f3ff1e82740 1 --2- 172.21.15.43:0/2300130063 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557a9a5c1e70 0x557a9a639890 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4009930 tx=0x7f3fd40f2040 comp rx=0 tx=0).stop 2024-03-28T22:37:33.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.041+0000 7f3ff1e82740 1 -- 172.21.15.43:0/2300130063 shutdown_connections 2024-03-28T22:37:33.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.041+0000 7f3ff1e82740 1 --2- 172.21.15.43:0/2300130063 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x557a9a6412f0 0x557a9a643710 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.041+0000 7f3ff1e82740 1 --2- 172.21.15.43:0/2300130063 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557a9a5c1e70 0x557a9a639890 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.041+0000 7f3ff1e82740 1 --2- 172.21.15.43:0/2300130063 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557a9a5caa50 0x557a9a5cdd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.041+0000 7f3ff1e82740 1 --2- 172.21.15.43:0/2300130063 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557a9a5c1550 0x557a9a639350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.041+0000 7f3ff1e82740 1 -- 172.21.15.43:0/2300130063 >> 172.21.15.43:0/2300130063 conn(0x557a9a5bbc80 msgr2=0x557a9a5c7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:33.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.041+0000 7f3fd2000640 0 client.25850 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:37:33.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.041+0000 7f3ff1e82740 1 -- 172.21.15.43:0/2300130063 shutdown_connections 2024-03-28T22:37:33.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.041+0000 7f3ff1e82740 1 -- 172.21.15.43:0/2300130063 wait complete. 2024-03-28T22:37:33.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.042+0000 7f3ff1e82740 20 client.25850 trim_cache size 0 max 16384 2024-03-28T22:37:33.043 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:37:33.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.042+0000 7f3fedc00640 1 -- 172.21.15.43:0/2630623200 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f3fcc0a2000 msgr2=0x7f3fcc0a44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:33.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.042+0000 7f3fedc00640 1 --2- 172.21.15.43:0/2630623200 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f3fcc0a2000 0x7f3fcc0a44c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3fe0005fd0 tx=0x7f3fe0009500 comp rx=0 tx=0).stop 2024-03-28T22:37:33.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.042+0000 7f3fedc00640 1 -- 172.21.15.43:0/2630623200 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3fe8101f90 msgr2=0x7f3fe8073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:33.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.042+0000 7f3fedc00640 1 --2- 172.21.15.43:0/2630623200 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3fe8101f90 0x7f3fe8073f90 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4036040 tx=0x7f3fd4004250 comp rx=0 tx=0).stop 2024-03-28T22:37:33.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.042+0000 7f3fedc00640 1 -- 172.21.15.43:0/2630623200 shutdown_connections 2024-03-28T22:37:33.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.042+0000 7f3fedc00640 1 --2- 172.21.15.43:0/2630623200 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f3fcc0a2000 0x7f3fcc0a44c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.042+0000 7f3fedc00640 1 --2- 172.21.15.43:0/2630623200 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3fe8101f90 0x7f3fe8073f90 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.042+0000 7f3fedc00640 1 --2- 172.21.15.43:0/2630623200 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3fe8107690 0x7f3fe80744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.042+0000 7f3fedc00640 1 --2- 172.21.15.43:0/2630623200 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3fe81015d0 0x7f3fe8073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.042+0000 7f3fedc00640 1 -- 172.21.15.43:0/2630623200 >> 172.21.15.43:0/2630623200 conn(0x7f3fe80fd0c0 msgr2=0x7f3fe80fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:33.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.042+0000 7f3fedc00640 1 -- 172.21.15.43:0/2630623200 shutdown_connections 2024-03-28T22:37:33.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.043+0000 7f3fedc00640 1 -- 172.21.15.43:0/2630623200 wait complete. 2024-03-28T22:37:33.055 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:32 smithi043 ceph-mon[31823]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:37:33.057 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:37:33.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:32 smithi082 ceph-mon[36471]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:37:33.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.280+0000 7f4ef3200640 1 -- 172.21.15.43:0/142162103 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ef410cf50 msgr2=0x7f4ef4114b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:33.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.280+0000 7f4ef3200640 1 --2- 172.21.15.43:0/142162103 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ef410cf50 0x7f4ef4114b20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4eec009900 tx=0x7f4eec02bc40 comp rx=0 tx=0).stop 2024-03-28T22:37:33.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.281+0000 7f4ef3200640 1 -- 172.21.15.43:0/142162103 shutdown_connections 2024-03-28T22:37:33.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.281+0000 7f4ef3200640 1 --2- 172.21.15.43:0/142162103 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ef410cf50 0x7f4ef4114b20 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.281+0000 7f4ef3200640 1 --2- 172.21.15.43:0/142162103 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ef41088b0 0x7f4ef410c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.281+0000 7f4ef3200640 1 --2- 172.21.15.43:0/142162103 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ef4107f00 0x7f4ef41082e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.281+0000 7f4ef3200640 1 -- 172.21.15.43:0/142162103 >> 172.21.15.43:0/142162103 conn(0x7f4ef4077570 msgr2=0x7f4ef4106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:33.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.281+0000 7f4ef3200640 1 -- 172.21.15.43:0/142162103 shutdown_connections 2024-03-28T22:37:33.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.281+0000 7f4ef3200640 1 -- 172.21.15.43:0/142162103 wait complete. 2024-03-28T22:37:33.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.282+0000 7f4ef3200640 1 Processor -- start 2024-03-28T22:37:33.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.282+0000 7f4ef3200640 1 -- start start 2024-03-28T22:37:33.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.282+0000 7f4ef3200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ef4107f00 0x7f4ef419e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.282+0000 7f4ef3200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ef410cf50 0x7f4ef419ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.282+0000 7f4ef3200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ef41a2d40 0x7f4ef41a31f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.282+0000 7f4ef3200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef4118070 con 0x7f4ef410cf50 2024-03-28T22:37:33.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.282+0000 7f4ef3200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef4117ef0 con 0x7f4ef41a2d40 2024-03-28T22:37:33.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.282+0000 7f4ef3200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef41181f0 con 0x7f4ef4107f00 2024-03-28T22:37:33.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.282+0000 7f4ef1e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ef4107f00 0x7f4ef419e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.282+0000 7f4ef1e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ef4107f00 0x7f4ef419e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:52556/0 (socket says 172.21.15.43:52556) 2024-03-28T22:37:33.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.283+0000 7f4ef1e00640 1 -- 172.21.15.43:0/1901882781 learned_addr learned my addr 172.21.15.43:0/1901882781 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:33.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.283+0000 7f4ee9400640 1 --2- 172.21.15.43:0/1901882781 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ef410cf50 0x7f4ef419ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.283+0000 7f4ef2800640 1 --2- 172.21.15.43:0/1901882781 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ef41a2d40 0x7f4ef41a31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.283+0000 7f4ef1e00640 1 -- 172.21.15.43:0/1901882781 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ef41a2d40 msgr2=0x7f4ef41a31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:33.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.283+0000 7f4ef1e00640 1 --2- 172.21.15.43:0/1901882781 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ef41a2d40 0x7f4ef41a31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.283+0000 7f4ef1e00640 1 -- 172.21.15.43:0/1901882781 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ef410cf50 msgr2=0x7f4ef419ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:33.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.283+0000 7f4ef1e00640 1 --2- 172.21.15.43:0/1901882781 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ef410cf50 0x7f4ef419ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.283+0000 7f4ef1e00640 1 -- 172.21.15.43:0/1901882781 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ef41a3830 con 0x7f4ef4107f00 2024-03-28T22:37:33.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.283+0000 7f4ef1e00640 1 --2- 172.21.15.43:0/1901882781 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ef4107f00 0x7f4ef419e4d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4ee400eb90 tx=0x7f4ee400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:33.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.283+0000 7f4eeb400640 1 -- 172.21.15.43:0/1901882781 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee4004d20 con 0x7f4ef4107f00 2024-03-28T22:37:33.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.283+0000 7f4ef3200640 1 -- 172.21.15.43:0/1901882781 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ef41a3b20 con 0x7f4ef4107f00 2024-03-28T22:37:33.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.283+0000 7f4eeb400640 1 -- 172.21.15.43:0/1901882781 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ee400ee90 con 0x7f4ef4107f00 2024-03-28T22:37:33.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.284+0000 7f4eeb400640 1 -- 172.21.15.43:0/1901882781 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee4010830 con 0x7f4ef4107f00 2024-03-28T22:37:33.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.284+0000 7f4ef3200640 1 -- 172.21.15.43:0/1901882781 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ef41ab230 con 0x7f4ef4107f00 2024-03-28T22:37:33.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.287+0000 7f4efbb63740 1 Processor -- start 2024-03-28T22:37:33.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.287+0000 7f4eeb400640 1 -- 172.21.15.43:0/1901882781 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 26) ==== 136932+0+0 (secure 0 0 0) 0x7f4ee4005020 con 0x7f4ef4107f00 2024-03-28T22:37:33.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.287+0000 7f4efbb63740 1 -- start start 2024-03-28T22:37:33.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.287+0000 7f4efbb63740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56394c1ee550 0x56394c1ee930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.287+0000 7f4efbb63740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56394c1eee70 0x56394c1f7510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.287+0000 7f4efbb63740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56394c1f7a50 0x56394c1f9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.287+0000 7f4efbb63740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56394c152050 con 0x56394c1f7a50 2024-03-28T22:37:33.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.287+0000 7f4efbb63740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x56394c0face0 con 0x56394c1ee550 2024-03-28T22:37:33.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.287+0000 7f4efbb63740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x56394c14e0f0 con 0x56394c1eee70 2024-03-28T22:37:33.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.288+0000 7f4ef2800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56394c1ee550 0x56394c1ee930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.288+0000 7f4ef1e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56394c1eee70 0x56394c1f7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.288+0000 7f4ef2800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56394c1ee550 0x56394c1ee930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:36750/0 (socket says 172.21.15.43:36750) 2024-03-28T22:37:33.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.288+0000 7f4ef2800640 1 -- 172.21.15.43:0/2369033810 learned_addr learned my addr 172.21.15.43:0/2369033810 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:33.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.288+0000 7f4ee9400640 1 --2- 172.21.15.43:0/2369033810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56394c1f7a50 0x56394c1f9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.288+0000 7f4ef2800640 1 -- 172.21.15.43:0/2369033810 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56394c1eee70 msgr2=0x56394c1f7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:33.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.288+0000 7f4ef2800640 1 --2- 172.21.15.43:0/2369033810 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56394c1eee70 0x56394c1f7510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.288+0000 7f4ef2800640 1 -- 172.21.15.43:0/2369033810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56394c1f7a50 msgr2=0x56394c1f9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:33.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.288+0000 7f4ef2800640 1 --2- 172.21.15.43:0/2369033810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56394c1f7a50 0x56394c1f9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.288+0000 7f4ef2800640 1 -- 172.21.15.43:0/2369033810 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56394be663f0 con 0x56394c1ee550 2024-03-28T22:37:33.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.288+0000 7f4ef2800640 1 --2- 172.21.15.43:0/2369033810 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56394c1ee550 0x56394c1ee930 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4eec02c250 tx=0x7f4eec005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c07b7b5ed3e914 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:33.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.288+0000 7f4eeb400640 1 --2- 172.21.15.43:0/1901882781 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4ed80a1f80 0x7f4ed80a4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.289+0000 7f4ee3400640 1 -- 172.21.15.43:0/2369033810 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eec042070 con 0x56394c1ee550 2024-03-28T22:37:33.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.289+0000 7f4ee3400640 1 -- 172.21.15.43:0/2369033810 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4eec004240 con 0x56394c1ee550 2024-03-28T22:37:33.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.289+0000 7f4ee3400640 1 -- 172.21.15.43:0/2369033810 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eec004f00 con 0x56394c1ee550 2024-03-28T22:37:33.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.289+0000 7f4eeb400640 1 -- 172.21.15.43:0/1901882781 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f4ee40ced90 con 0x7f4ef4107f00 2024-03-28T22:37:33.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.289+0000 7f4ef1e00640 1 --2- 172.21.15.43:0/1901882781 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4ed80a1f80 0x7f4ed80a4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.289+0000 7f4efbb63740 1 -- 172.21.15.43:0/2369033810 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56394c1ee550 msgr2=0x56394c1ee930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:33.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.289+0000 7f4efbb63740 1 --2- 172.21.15.43:0/2369033810 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56394c1ee550 0x56394c1ee930 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4eec02c250 tx=0x7f4eec005a20 comp rx=0 tx=0).stop 2024-03-28T22:37:33.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.290+0000 7f4ef1e00640 1 --2- 172.21.15.43:0/1901882781 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4ed80a1f80 0x7f4ed80a4440 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4ee40cc080 tx=0x7f4ee4005960 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:33.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.290+0000 7f4efbb63740 1 -- 172.21.15.43:0/2369033810 shutdown_connections 2024-03-28T22:37:33.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.290+0000 7f4efbb63740 1 --2- 172.21.15.43:0/2369033810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56394c1f7a50 0x56394c1f9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.290+0000 7f4efbb63740 1 --2- 172.21.15.43:0/2369033810 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56394c1eee70 0x56394c1f7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.290+0000 7f4efbb63740 1 --2- 172.21.15.43:0/2369033810 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56394c1ee550 0x56394c1ee930 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.290+0000 7f4efbb63740 1 -- 172.21.15.43:0/2369033810 >> 172.21.15.43:0/2369033810 conn(0x56394c1e8c80 msgr2=0x56394c1e9060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:33.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.290+0000 7f4efbb63740 1 -- 172.21.15.43:0/2369033810 shutdown_connections 2024-03-28T22:37:33.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.290+0000 7f4efbb63740 1 -- 172.21.15.43:0/2369033810 wait complete. 2024-03-28T22:37:33.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.290+0000 7f4efbb63740 1 -- start start 2024-03-28T22:37:33.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.290+0000 7f4efbb63740 1 Processor -- start 2024-03-28T22:37:33.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.291+0000 7f4efbb63740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56394c1ee550 0x56394c1f4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.291+0000 7f4efbb63740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56394c1eee70 0x56394c1f4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.291+0000 7f4efbb63740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56394c1f7a50 0x56394c1f5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.291+0000 7f4efbb63740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56394c1fa7e0 con 0x56394c1ee550 2024-03-28T22:37:33.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.291+0000 7f4efbb63740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x56394c0f02f0 con 0x56394c1eee70 2024-03-28T22:37:33.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.291+0000 7f4efbb63740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x56394c1fa960 con 0x56394c1f7a50 2024-03-28T22:37:33.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.291+0000 7f4efbb63740 10 client.0 _ll_register_callbacks cb 0x56394be65be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:37:33.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.291+0000 7f4ee9400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56394c1eee70 0x56394c1f4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.291+0000 7f4ef2800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56394c1ee550 0x56394c1f4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.291+0000 7f4ee9400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56394c1eee70 0x56394c1f4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:36754/0 (socket says 172.21.15.43:36754) 2024-03-28T22:37:33.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.291+0000 7f4ee9400640 1 -- 172.21.15.43:0/3784607493 learned_addr learned my addr 172.21.15.43:0/3784607493 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:33.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.291+0000 7f4ef2800640 1 --2- 172.21.15.43:0/3784607493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56394c1f7a50 0x56394c1f5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.291+0000 7f4ee9400640 1 -- 172.21.15.43:0/3784607493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56394c1f7a50 msgr2=0x56394c1f5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:33.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.291+0000 7f4ee9400640 1 --2- 172.21.15.43:0/3784607493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56394c1f7a50 0x56394c1f5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.291+0000 7f4ee9400640 1 -- 172.21.15.43:0/3784607493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56394c1ee550 msgr2=0x56394c1f4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:33.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.291+0000 7f4ee9400640 1 --2- 172.21.15.43:0/3784607493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56394c1ee550 0x56394c1f4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.292+0000 7f4ee9400640 1 -- 172.21.15.43:0/3784607493 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56394bf307d0 con 0x56394c1eee70 2024-03-28T22:37:33.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.292+0000 7f4ef2800640 1 --2- 172.21.15.43:0/3784607493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56394c1f7a50 0x56394c1f5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:37:33.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.292+0000 7f4ee9400640 1 --2- 172.21.15.43:0/3784607493 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56394c1eee70 0x56394c1f4fe0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4108e70 tx=0x7f4edc012820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f6a93ef1e8b2b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:33.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.292+0000 7f4ed7e00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:37:33.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.292+0000 7f4ed7e00640 1 -- 172.21.15.43:0/3784607493 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4edc004220 con 0x56394c1eee70 2024-03-28T22:37:33.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.292+0000 7f4ed7e00640 1 -- 172.21.15.43:0/3784607493 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f4edc0043c0 con 0x56394c1eee70 2024-03-28T22:37:33.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.292+0000 7f4efbb63740 1 -- 172.21.15.43:0/3784607493 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4eec004240 con 0x56394c1eee70 2024-03-28T22:37:33.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.292+0000 7f4ed7e00640 1 -- 172.21.15.43:0/3784607493 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4edc024050 con 0x56394c1eee70 2024-03-28T22:37:33.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.292+0000 7f4efbb63740 10 client.25862 fetch_fsmap learned FSMap version 29 2024-03-28T22:37:33.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.292+0000 7f4efbb63740 1 -- 172.21.15.43:0/3784607493 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56394c206b00 con 0x56394c1eee70 2024-03-28T22:37:33.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.293+0000 7f4ed7e00640 1 -- 172.21.15.43:0/3784607493 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4eec004240 con 0x56394c1eee70 2024-03-28T22:37:33.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.293+0000 7f4ed7e00640 1 -- 172.21.15.43:0/3784607493 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f4edc003a30 con 0x56394c1eee70 2024-03-28T22:37:33.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.293+0000 7f4efbb63740 10 client.25862 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:37:33.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.293+0000 7f4efbb63740 10 client.25862 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:37:33.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.293+0000 7f4efbb63740 20 client.25862 populate_metadata read hostname 'smithi043' 2024-03-28T22:37:33.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.293+0000 7f4efbb63740 1 --2- 172.21.15.43:0/3784607493 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56394c207fc0 0x56394c20a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.293+0000 7f4efbb63740 4 client.25862 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:37:33.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.293+0000 7f4efbb63740 1 -- 172.21.15.43:0/3784607493 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56394c0fb680 con 0x56394c207fc0 2024-03-28T22:37:33.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.293+0000 7f4ef2800640 1 --2- 172.21.15.43:0/3784607493 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56394c207fc0 0x56394c20a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.294+0000 7f4ef2800640 1 --2- 172.21.15.43:0/3784607493 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56394c207fc0 0x56394c20a3f0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71052fb852ecf102 server_cookie=5bccf6abeb7ab67b in_seq=0 out_seq=0 2024-03-28T22:37:33.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.294+0000 7f4ed7e00640 10 client.25862 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:33.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.295+0000 7f4ed7e00640 1 -- 172.21.15.43:0/3784607493 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56394c0fb680 con 0x56394c207fc0 2024-03-28T22:37:33.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.295+0000 7f4ed7e00640 10 client.25862 handle_command_reply: tid=0 2024-03-28T22:37:33.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.295+0000 7f4efbb63740 1 client.25862 shutdown 2024-03-28T22:37:33.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.295+0000 7f4efbb63740 1 -- 172.21.15.43:0/3784607493 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56394c1eee70 msgr2=0x56394c1f4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:33.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.295+0000 7f4efbb63740 1 --2- 172.21.15.43:0/3784607493 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56394c1eee70 0x56394c1f4fe0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4108e70 tx=0x7f4edc012820 comp rx=0 tx=0).stop 2024-03-28T22:37:33.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.295+0000 7f4efbb63740 1 -- 172.21.15.43:0/3784607493 shutdown_connections 2024-03-28T22:37:33.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.295+0000 7f4efbb63740 1 --2- 172.21.15.43:0/3784607493 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56394c207fc0 0x56394c20a3f0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.295+0000 7f4efbb63740 1 --2- 172.21.15.43:0/3784607493 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56394c1eee70 0x56394c1f4fe0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.295+0000 7f4efbb63740 1 --2- 172.21.15.43:0/3784607493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56394c1f7a50 0x56394c1f5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.295+0000 7f4efbb63740 1 --2- 172.21.15.43:0/3784607493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56394c1ee550 0x56394c1f4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.295+0000 7f4efbb63740 1 -- 172.21.15.43:0/3784607493 >> 172.21.15.43:0/3784607493 conn(0x56394c1e8c80 msgr2=0x56394c1ea9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:33.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.295+0000 7f4ed7e00640 0 client.25862 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:33.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.295+0000 7f4efbb63740 1 -- 172.21.15.43:0/3784607493 shutdown_connections 2024-03-28T22:37:33.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.295+0000 7f4efbb63740 1 -- 172.21.15.43:0/3784607493 wait complete. 2024-03-28T22:37:33.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.295+0000 7f4efbb63740 20 client.25862 trim_cache size 0 max 16384 2024-03-28T22:37:33.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.310+0000 7f4efbb63740 1 Processor -- start 2024-03-28T22:37:33.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.310+0000 7f4efbb63740 1 -- start start 2024-03-28T22:37:33.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.310+0000 7f4efbb63740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56394c1ee550 0x56394c1f71d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.310+0000 7f4efbb63740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56394c1eee70 0x56394c203580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.310+0000 7f4efbb63740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56394c1f7a50 0x56394c20fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.310+0000 7f4efbb63740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef41b4f50 con 0x56394c1f7a50 2024-03-28T22:37:33.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.310+0000 7f4efbb63740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef41b6a90 con 0x56394c1ee550 2024-03-28T22:37:33.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.310+0000 7f4efbb63740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef41b5620 con 0x56394c1eee70 2024-03-28T22:37:33.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.311+0000 7f4ee9400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56394c1eee70 0x56394c203580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.311+0000 7f4ee9400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56394c1eee70 0x56394c203580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:52582/0 (socket says 172.21.15.43:52582) 2024-03-28T22:37:33.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.311+0000 7f4ee9400640 1 -- 172.21.15.43:0/231027731 learned_addr learned my addr 172.21.15.43:0/231027731 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:33.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.311+0000 7f4ef2800640 1 --2- 172.21.15.43:0/231027731 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56394c1ee550 0x56394c1f71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.311+0000 7f4ef2800640 1 --2- 172.21.15.43:0/231027731 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56394c1f7a50 0x56394c20fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.311+0000 7f4ee9400640 1 -- 172.21.15.43:0/231027731 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56394c1ee550 msgr2=0x56394c1f71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:33.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.311+0000 7f4ee9400640 1 --2- 172.21.15.43:0/231027731 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56394c1ee550 0x56394c1f71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.311+0000 7f4ee9400640 1 -- 172.21.15.43:0/231027731 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56394c1f7a50 msgr2=0x56394c20fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:33.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.311+0000 7f4ee9400640 1 --2- 172.21.15.43:0/231027731 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56394c1f7a50 0x56394c20fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.311+0000 7f4ee9400640 1 -- 172.21.15.43:0/231027731 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56394c1f2ff0 con 0x56394c1eee70 2024-03-28T22:37:33.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.311+0000 7f4ef2800640 1 --2- 172.21.15.43:0/231027731 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56394c1f7a50 0x56394c20fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:37:33.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.311+0000 7f4ee9400640 1 --2- 172.21.15.43:0/231027731 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56394c1eee70 0x56394c203580 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4edc01c450 tx=0x7f4edc005c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bada3006a9a843e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:33.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.312+0000 7f4ee2a00640 1 -- 172.21.15.43:0/231027731 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4edc000e60 con 0x56394c1eee70 2024-03-28T22:37:33.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.312+0000 7f4ee2a00640 1 -- 172.21.15.43:0/231027731 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4edc021040 con 0x56394c1eee70 2024-03-28T22:37:33.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.312+0000 7f4ee2a00640 1 -- 172.21.15.43:0/231027731 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4edc02c070 con 0x56394c1eee70 2024-03-28T22:37:33.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.312+0000 7f4efbb63740 1 -- 172.21.15.43:0/231027731 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56394c1eee70 msgr2=0x56394c203580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:33.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.312+0000 7f4efbb63740 1 --2- 172.21.15.43:0/231027731 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56394c1eee70 0x56394c203580 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4edc01c450 tx=0x7f4edc005c10 comp rx=0 tx=0).stop 2024-03-28T22:37:33.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.312+0000 7f4efbb63740 1 -- 172.21.15.43:0/231027731 shutdown_connections 2024-03-28T22:37:33.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.312+0000 7f4efbb63740 1 --2- 172.21.15.43:0/231027731 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56394c1f7a50 0x56394c20fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.312+0000 7f4efbb63740 1 --2- 172.21.15.43:0/231027731 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56394c1eee70 0x56394c203580 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.312+0000 7f4efbb63740 1 --2- 172.21.15.43:0/231027731 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56394c1ee550 0x56394c1f71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.312+0000 7f4efbb63740 1 -- 172.21.15.43:0/231027731 >> 172.21.15.43:0/231027731 conn(0x56394c1e8c80 msgr2=0x56394c1f52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:33.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.312+0000 7f4efbb63740 1 -- 172.21.15.43:0/231027731 shutdown_connections 2024-03-28T22:37:33.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.312+0000 7f4efbb63740 1 -- 172.21.15.43:0/231027731 wait complete. 2024-03-28T22:37:33.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.313+0000 7f4efbb63740 1 -- start start 2024-03-28T22:37:33.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.313+0000 7f4efbb63740 1 Processor -- start 2024-03-28T22:37:33.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.313+0000 7f4efbb63740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56394c1ee550 0x56394c1f1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.313+0000 7f4efbb63740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56394c1eee70 0x56394c1f23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.313+0000 7f4efbb63740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56394c1f7a50 0x56394c267300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.313+0000 7f4efbb63740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef41b63c0 con 0x56394c1f7a50 2024-03-28T22:37:33.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.313+0000 7f4efbb63740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef41b5cf0 con 0x56394c1ee550 2024-03-28T22:37:33.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.313+0000 7f4efbb63740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef41b2f30 con 0x56394c1eee70 2024-03-28T22:37:33.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.313+0000 7f4efbb63740 10 client.0 _ll_register_callbacks cb 0x7f4edc00ba60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:37:33.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.314+0000 7f4ee9400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56394c1eee70 0x56394c1f23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.314+0000 7f4ee9400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56394c1eee70 0x56394c1f23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:52588/0 (socket says 172.21.15.43:52588) 2024-03-28T22:37:33.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.314+0000 7f4ee9400640 1 -- 172.21.15.43:0/778757341 learned_addr learned my addr 172.21.15.43:0/778757341 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:33.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.314+0000 7f4ef2800640 1 --2- 172.21.15.43:0/778757341 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56394c1ee550 0x56394c1f1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.314+0000 7f4ef2800640 1 --2- 172.21.15.43:0/778757341 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56394c1f7a50 0x56394c267300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.314+0000 7f4ee9400640 1 -- 172.21.15.43:0/778757341 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56394c1ee550 msgr2=0x56394c1f1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:33.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.314+0000 7f4ee9400640 1 --2- 172.21.15.43:0/778757341 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56394c1ee550 0x56394c1f1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.314+0000 7f4ee9400640 1 -- 172.21.15.43:0/778757341 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56394c1f7a50 msgr2=0x56394c267300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:33.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.314+0000 7f4ee9400640 1 --2- 172.21.15.43:0/778757341 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56394c1f7a50 0x56394c267300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.314+0000 7f4ee9400640 1 -- 172.21.15.43:0/778757341 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56394c2678f0 con 0x56394c1eee70 2024-03-28T22:37:33.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.314+0000 7f4ee9400640 1 --2- 172.21.15.43:0/778757341 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56394c1eee70 0x56394c1f23c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4edc005ca0 tx=0x7f4edc004230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=848aa2b041556a5f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:33.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.314+0000 7f4ed7e00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:37:33.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.315+0000 7f4ed7e00640 1 -- 172.21.15.43:0/778757341 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4edc034070 con 0x56394c1eee70 2024-03-28T22:37:33.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.315+0000 7f4ed7e00640 1 -- 172.21.15.43:0/778757341 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f4edc003a00 con 0x56394c1eee70 2024-03-28T22:37:33.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.315+0000 7f4efbb63740 1 -- 172.21.15.43:0/778757341 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4edc021040 con 0x56394c1eee70 2024-03-28T22:37:33.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.315+0000 7f4ed7e00640 1 -- 172.21.15.43:0/778757341 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4edc042050 con 0x56394c1eee70 2024-03-28T22:37:33.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.315+0000 7f4efbb63740 10 client.25837 fetch_fsmap learned FSMap version 29 2024-03-28T22:37:33.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.315+0000 7f4efbb63740 1 -- 172.21.15.43:0/778757341 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56394c26ec10 con 0x56394c1eee70 2024-03-28T22:37:33.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.315+0000 7f4ed7e00640 1 -- 172.21.15.43:0/778757341 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4edc021040 con 0x56394c1eee70 2024-03-28T22:37:33.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.315+0000 7f4ed7e00640 1 -- 172.21.15.43:0/778757341 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f4edc010a80 con 0x56394c1eee70 2024-03-28T22:37:33.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.315+0000 7f4efbb63740 10 client.25837 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:37:33.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.315+0000 7f4efbb63740 10 client.25837 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:37:33.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.315+0000 7f4efbb63740 20 client.25837 populate_metadata read hostname 'smithi043' 2024-03-28T22:37:33.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.315+0000 7f4efbb63740 1 --2- 172.21.15.43:0/778757341 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56394c26f360 0x56394c271780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:33.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.316+0000 7f4efbb63740 4 client.25837 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:37:33.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.316+0000 7f4efbb63740 1 -- 172.21.15.43:0/778757341 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56394c15f590 con 0x56394c26f360 2024-03-28T22:37:33.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.316+0000 7f4ef2800640 1 --2- 172.21.15.43:0/778757341 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56394c26f360 0x56394c271780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:33.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.316+0000 7f4ef2800640 1 --2- 172.21.15.43:0/778757341 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56394c26f360 0x56394c271780 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4c29151309d2a9b server_cookie=c2a23175423482f4 in_seq=0 out_seq=0 2024-03-28T22:37:33.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.316+0000 7f4ed7e00640 10 client.25837 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:33.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.316+0000 7f4ed7e00640 1 -- 172.21.15.43:0/778757341 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56394c15f590 con 0x56394c26f360 2024-03-28T22:37:33.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.316+0000 7f4ed7e00640 10 client.25837 handle_command_reply: tid=0 2024-03-28T22:37:33.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.316+0000 7f4efbb63740 1 client.25837 shutdown 2024-03-28T22:37:33.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.317+0000 7f4efbb63740 1 -- 172.21.15.43:0/778757341 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56394c1eee70 msgr2=0x56394c1f23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:33.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.317+0000 7f4efbb63740 1 --2- 172.21.15.43:0/778757341 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56394c1eee70 0x56394c1f23c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4edc005ca0 tx=0x7f4edc004230 comp rx=0 tx=0).stop 2024-03-28T22:37:33.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.317+0000 7f4efbb63740 1 -- 172.21.15.43:0/778757341 shutdown_connections 2024-03-28T22:37:33.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.317+0000 7f4efbb63740 1 --2- 172.21.15.43:0/778757341 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56394c1f7a50 0x56394c267300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.317+0000 7f4efbb63740 1 --2- 172.21.15.43:0/778757341 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56394c1eee70 0x56394c1f23c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.317+0000 7f4efbb63740 1 --2- 172.21.15.43:0/778757341 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56394c26f360 0x56394c271780 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.317+0000 7f4efbb63740 1 --2- 172.21.15.43:0/778757341 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56394c1ee550 0x56394c1f1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.317+0000 7f4efbb63740 1 -- 172.21.15.43:0/778757341 >> 172.21.15.43:0/778757341 conn(0x56394c1e8c80 msgr2=0x56394c1e7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:33.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.317+0000 7f4ed7e00640 0 client.25837 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:33.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.317+0000 7f4efbb63740 1 -- 172.21.15.43:0/778757341 shutdown_connections 2024-03-28T22:37:33.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.317+0000 7f4efbb63740 1 -- 172.21.15.43:0/778757341 wait complete. 2024-03-28T22:37:33.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.317+0000 7f4efbb63740 20 client.25837 trim_cache size 0 max 16384 2024-03-28T22:37:33.323 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:37:33.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.318+0000 7f4ef3200640 1 -- 172.21.15.43:0/1901882781 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4ed80a1f80 msgr2=0x7f4ed80a4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:33.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.318+0000 7f4ef3200640 1 --2- 172.21.15.43:0/1901882781 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4ed80a1f80 0x7f4ed80a4440 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4ee40cc080 tx=0x7f4ee4005960 comp rx=0 tx=0).stop 2024-03-28T22:37:33.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.318+0000 7f4ef3200640 1 -- 172.21.15.43:0/1901882781 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ef4107f00 msgr2=0x7f4ef419e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:33.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.318+0000 7f4ef3200640 1 --2- 172.21.15.43:0/1901882781 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ef4107f00 0x7f4ef419e4d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4ee400eb90 tx=0x7f4ee400c6a0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.318+0000 7f4ef3200640 1 -- 172.21.15.43:0/1901882781 shutdown_connections 2024-03-28T22:37:33.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.318+0000 7f4ef3200640 1 --2- 172.21.15.43:0/1901882781 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4ed80a1f80 0x7f4ed80a4440 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.318+0000 7f4ef3200640 1 --2- 172.21.15.43:0/1901882781 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ef41a2d40 0x7f4ef41a31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.318+0000 7f4ef3200640 1 --2- 172.21.15.43:0/1901882781 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ef410cf50 0x7f4ef419ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.318+0000 7f4ef3200640 1 --2- 172.21.15.43:0/1901882781 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ef4107f00 0x7f4ef419e4d0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:33.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.318+0000 7f4ef3200640 1 -- 172.21.15.43:0/1901882781 >> 172.21.15.43:0/1901882781 conn(0x7f4ef4077570 msgr2=0x7f4ef4077b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:33.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.318+0000 7f4ef3200640 1 -- 172.21.15.43:0/1901882781 shutdown_connections 2024-03-28T22:37:33.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:33.318+0000 7f4ef3200640 1 -- 172.21.15.43:0/1901882781 wait complete. 2024-03-28T22:37:33.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:32 smithi172 ceph-mon[36566]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:37:34.336 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:37:34.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:37:34.337 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:37:34.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.582+0000 7f617bc00640 1 -- 172.21.15.43:0/2883576663 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f61741038f0 msgr2=0x7f6174103d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:34.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.582+0000 7f617bc00640 1 --2- 172.21.15.43:0/2883576663 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f61741038f0 0x7f6174103d70 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6160009900 tx=0x7f616002bc30 comp rx=0 tx=0).stop 2024-03-28T22:37:34.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.582+0000 7f617bc00640 1 -- 172.21.15.43:0/2883576663 shutdown_connections 2024-03-28T22:37:34.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.582+0000 7f617bc00640 1 --2- 172.21.15.43:0/2883576663 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f61741038f0 0x7f6174103d70 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.582+0000 7f617bc00640 1 --2- 172.21.15.43:0/2883576663 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61741042b0 0x7f61741107c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.582+0000 7f617bc00640 1 --2- 172.21.15.43:0/2883576663 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6174109900 0x7f6174109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.582+0000 7f617bc00640 1 -- 172.21.15.43:0/2883576663 >> 172.21.15.43:0/2883576663 conn(0x7f61740ff400 msgr2=0x7f6174101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:34.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.582+0000 7f617bc00640 1 -- 172.21.15.43:0/2883576663 shutdown_connections 2024-03-28T22:37:34.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.582+0000 7f617bc00640 1 -- 172.21.15.43:0/2883576663 wait complete. 2024-03-28T22:37:34.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.583+0000 7f617bc00640 1 Processor -- start 2024-03-28T22:37:34.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.583+0000 7f617bc00640 1 -- start start 2024-03-28T22:37:34.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.584+0000 7f617bc00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f61741038f0 0x7f617419a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.584+0000 7f617bc00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61741042b0 0x7f617419a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.584+0000 7f617bc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6174109900 0x7f617419eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.584+0000 7f617bc00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6174113d10 con 0x7f6174109900 2024-03-28T22:37:34.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.584+0000 7f617bc00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f6174113b90 con 0x7f61741038f0 2024-03-28T22:37:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.584+0000 7f617bc00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f6174113e90 con 0x7f61741042b0 2024-03-28T22:37:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.584+0000 7f6179200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f61741038f0 0x7f617419a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.584+0000 7f6179200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f61741038f0 0x7f617419a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:36784/0 (socket says 172.21.15.43:36784) 2024-03-28T22:37:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.584+0000 7f6179200640 1 -- 172.21.15.43:0/2070476016 learned_addr learned my addr 172.21.15.43:0/2070476016 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.584+0000 7f6179c00640 1 --2- 172.21.15.43:0/2070476016 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6174109900 0x7f617419eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.584+0000 7f6173e00640 1 --2- 172.21.15.43:0/2070476016 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61741042b0 0x7f617419a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.585+0000 7f6179200640 1 -- 172.21.15.43:0/2070476016 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61741042b0 msgr2=0x7f617419a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.585+0000 7f6179200640 1 --2- 172.21.15.43:0/2070476016 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61741042b0 0x7f617419a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.585+0000 7f6179200640 1 -- 172.21.15.43:0/2070476016 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6174109900 msgr2=0x7f617419eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.585+0000 7f6179200640 1 --2- 172.21.15.43:0/2070476016 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6174109900 0x7f617419eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.585+0000 7f6179200640 1 -- 172.21.15.43:0/2070476016 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f617419f240 con 0x7f61741038f0 2024-03-28T22:37:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.585+0000 7f6179200640 1 --2- 172.21.15.43:0/2070476016 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f61741038f0 0x7f617419a250 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f616400e9f0 tx=0x7f616400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:34.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.585+0000 7f6171600640 1 -- 172.21.15.43:0/2070476016 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f616400cd80 con 0x7f61741038f0 2024-03-28T22:37:34.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.585+0000 7f6171600640 1 -- 172.21.15.43:0/2070476016 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6164010910 con 0x7f61741038f0 2024-03-28T22:37:34.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.585+0000 7f617bc00640 1 -- 172.21.15.43:0/2070476016 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f617419f530 con 0x7f61741038f0 2024-03-28T22:37:34.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.585+0000 7f6171600640 1 -- 172.21.15.43:0/2070476016 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6164018990 con 0x7f61741038f0 2024-03-28T22:37:34.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.585+0000 7f617bc00640 1 -- 172.21.15.43:0/2070476016 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61741a6dc0 con 0x7f61741038f0 2024-03-28T22:37:34.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.588+0000 7f6171600640 1 -- 172.21.15.43:0/2070476016 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 26) ==== 136932+0+0 (secure 0 0 0) 0x7f6164010ac0 con 0x7f61741038f0 2024-03-28T22:37:34.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.589+0000 7f617e9b9740 1 Processor -- start 2024-03-28T22:37:34.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.589+0000 7f617e9b9740 1 -- start start 2024-03-28T22:37:34.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.589+0000 7f617e9b9740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6a92d2f70 0x55c6a92d3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.589+0000 7f617e9b9740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6a92d3890 0x55c6a92dbf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.589+0000 7f617e9b9740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6a92dc470 0x55c6a92de860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.589+0000 7f617e9b9740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c6a9236c40 con 0x55c6a92d3890 2024-03-28T22:37:34.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.589+0000 7f617e9b9740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55c6a91dfc40 con 0x55c6a92d2f70 2024-03-28T22:37:34.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.589+0000 7f617e9b9740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55c6a9232ce0 con 0x55c6a92dc470 2024-03-28T22:37:34.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.589+0000 7f6179c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6a92d2f70 0x55c6a92d3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.589+0000 7f6179c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6a92d2f70 0x55c6a92d3350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:36796/0 (socket says 172.21.15.43:36796) 2024-03-28T22:37:34.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.589+0000 7f6179c00640 1 -- 172.21.15.43:0/2348563174 learned_addr learned my addr 172.21.15.43:0/2348563174 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:34.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.590+0000 7f6179200640 1 --2- 172.21.15.43:0/2348563174 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6a92d3890 0x55c6a92dbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.590+0000 7f6173e00640 1 --2- 172.21.15.43:0/2348563174 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6a92dc470 0x55c6a92de860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.590+0000 7f6179c00640 1 -- 172.21.15.43:0/2348563174 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6a92dc470 msgr2=0x55c6a92de860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:34.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.590+0000 7f6179c00640 1 --2- 172.21.15.43:0/2348563174 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6a92dc470 0x55c6a92de860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.590+0000 7f6179c00640 1 -- 172.21.15.43:0/2348563174 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6a92d3890 msgr2=0x55c6a92dbf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:34.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.590+0000 7f6179c00640 1 --2- 172.21.15.43:0/2348563174 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6a92d3890 0x55c6a92dbf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.590+0000 7f6179c00640 1 -- 172.21.15.43:0/2348563174 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6a90137e0 con 0x55c6a92d2f70 2024-03-28T22:37:34.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.590+0000 7f6173e00640 1 --2- 172.21.15.43:0/2348563174 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6a92dc470 0x55c6a92de860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:37:34.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.590+0000 7f6179c00640 1 --2- 172.21.15.43:0/2348563174 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6a92d2f70 0x55c6a92d3350 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f616c009fd0 tx=0x7f616c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81b5da988d6f8a2d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:34.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.590+0000 7f6171600640 1 --2- 172.21.15.43:0/2070476016 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f61580a2000 0x7f61580a44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.590+0000 7f616a000640 1 -- 172.21.15.43:0/2348563174 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f616c004d20 con 0x55c6a92d2f70 2024-03-28T22:37:34.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.590+0000 7f616a000640 1 -- 172.21.15.43:0/2348563174 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f616c004050 con 0x55c6a92d2f70 2024-03-28T22:37:34.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.590+0000 7f616a000640 1 -- 172.21.15.43:0/2348563174 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f616c005260 con 0x55c6a92d2f70 2024-03-28T22:37:34.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.591+0000 7f6171600640 1 -- 172.21.15.43:0/2070476016 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f61640ce940 con 0x7f61741038f0 2024-03-28T22:37:34.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.591+0000 7f6179200640 1 --2- 172.21.15.43:0/2070476016 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f61580a2000 0x7f61580a44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.591+0000 7f617e9b9740 1 -- 172.21.15.43:0/2348563174 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6a92d2f70 msgr2=0x55c6a92d3350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:34.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.591+0000 7f617e9b9740 1 --2- 172.21.15.43:0/2348563174 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6a92d2f70 0x55c6a92d3350 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f616c009fd0 tx=0x7f616c0118d0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.591+0000 7f617e9b9740 1 -- 172.21.15.43:0/2348563174 shutdown_connections 2024-03-28T22:37:34.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.591+0000 7f617e9b9740 1 --2- 172.21.15.43:0/2348563174 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6a92dc470 0x55c6a92de860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.591+0000 7f617e9b9740 1 --2- 172.21.15.43:0/2348563174 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6a92d3890 0x55c6a92dbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.591+0000 7f617e9b9740 1 --2- 172.21.15.43:0/2348563174 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6a92d2f70 0x55c6a92d3350 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.591+0000 7f617e9b9740 1 -- 172.21.15.43:0/2348563174 >> 172.21.15.43:0/2348563174 conn(0x55c6a92cd6a0 msgr2=0x55c6a92cda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:34.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.591+0000 7f6179200640 1 --2- 172.21.15.43:0/2070476016 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f61580a2000 0x7f61580a44c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f61640cbc30 tx=0x7f6164000bc0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:34.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.591+0000 7f617e9b9740 1 -- 172.21.15.43:0/2348563174 shutdown_connections 2024-03-28T22:37:34.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.592+0000 7f617e9b9740 1 -- 172.21.15.43:0/2348563174 wait complete. 2024-03-28T22:37:34.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.592+0000 7f617e9b9740 1 -- start start 2024-03-28T22:37:34.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.592+0000 7f617e9b9740 1 Processor -- start 2024-03-28T22:37:34.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.593+0000 7f617e9b9740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6a92d2f70 0x55c6a92d94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.593+0000 7f617e9b9740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6a92d3890 0x55c6a92d99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.593+0000 7f617e9b9740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6a92dc470 0x55c6a92d9f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.593+0000 7f617e9b9740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c6a92df260 con 0x55c6a92d2f70 2024-03-28T22:37:34.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.593+0000 7f617e9b9740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55c6a91d5250 con 0x55c6a92dc470 2024-03-28T22:37:34.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.593+0000 7f617e9b9740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55c6a92df3e0 con 0x55c6a92d3890 2024-03-28T22:37:34.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.593+0000 7f617e9b9740 10 client.0 _ll_register_callbacks cb 0x55c6a91f6e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:37:34.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.593+0000 7f6179c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6a92d2f70 0x55c6a92d94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.593+0000 7f6173e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6a92d3890 0x55c6a92d99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.593+0000 7f6173e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6a92d3890 0x55c6a92d99e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:52630/0 (socket says 172.21.15.43:52630) 2024-03-28T22:37:34.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.593+0000 7f6179c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6a92d2f70 0x55c6a92d94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47144/0 (socket says 172.21.15.43:47144) 2024-03-28T22:37:34.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.593+0000 7f6173e00640 1 -- 172.21.15.43:0/3213543128 learned_addr learned my addr 172.21.15.43:0/3213543128 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:34.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.593+0000 7f6179c00640 1 --2- 172.21.15.43:0/3213543128 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6a92dc470 0x55c6a92d9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.593+0000 7f6173e00640 1 -- 172.21.15.43:0/3213543128 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6a92dc470 msgr2=0x55c6a92d9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:34.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.593+0000 7f6173e00640 1 --2- 172.21.15.43:0/3213543128 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6a92dc470 0x55c6a92d9f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.593+0000 7f6173e00640 1 -- 172.21.15.43:0/3213543128 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6a92d2f70 msgr2=0x55c6a92d94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:34.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.593+0000 7f6173e00640 1 --2- 172.21.15.43:0/3213543128 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6a92d2f70 0x55c6a92d94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.593+0000 7f6173e00640 1 -- 172.21.15.43:0/3213543128 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6a9073f70 con 0x55c6a92d3890 2024-03-28T22:37:34.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.594+0000 7f6173e00640 1 --2- 172.21.15.43:0/3213543128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6a92d3890 0x55c6a92d99e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6160035000 tx=0x7f616002c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fafe8ecfd0ad8bb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:34.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.594+0000 7f615ea00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:37:34.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.594+0000 7f615ea00640 1 -- 172.21.15.43:0/3213543128 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f616002cab0 con 0x55c6a92d3890 2024-03-28T22:37:34.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.594+0000 7f615ea00640 1 -- 172.21.15.43:0/3213543128 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f616002cc50 con 0x55c6a92d3890 2024-03-28T22:37:34.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.594+0000 7f617e9b9740 1 -- 172.21.15.43:0/3213543128 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f616c004050 con 0x55c6a92d3890 2024-03-28T22:37:34.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.595+0000 7f615ea00640 1 -- 172.21.15.43:0/3213543128 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f61600046c0 con 0x55c6a92d3890 2024-03-28T22:37:34.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.595+0000 7f617e9b9740 10 client.25849 fetch_fsmap learned FSMap version 29 2024-03-28T22:37:34.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.595+0000 7f617e9b9740 1 -- 172.21.15.43:0/3213543128 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c6a8f0b170 con 0x55c6a92d3890 2024-03-28T22:37:34.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.595+0000 7f615ea00640 1 -- 172.21.15.43:0/3213543128 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f616c004050 con 0x55c6a92d3890 2024-03-28T22:37:34.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.595+0000 7f615ea00640 1 -- 172.21.15.43:0/3213543128 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f61600035f0 con 0x55c6a92d3890 2024-03-28T22:37:34.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.595+0000 7f617e9b9740 10 client.25849 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:37:34.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.595+0000 7f617e9b9740 10 client.25849 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:37:34.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.595+0000 7f617e9b9740 20 client.25849 populate_metadata read hostname 'smithi043' 2024-03-28T22:37:34.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.595+0000 7f617e9b9740 1 --2- 172.21.15.43:0/3213543128 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c6a92ec820 0x55c6a92eec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.595+0000 7f617e9b9740 4 client.25849 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:37:34.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.595+0000 7f617e9b9740 1 -- 172.21.15.43:0/3213543128 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c6a9244180 con 0x55c6a92ec820 2024-03-28T22:37:34.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.595+0000 7f6179c00640 1 --2- 172.21.15.43:0/3213543128 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c6a92ec820 0x55c6a92eec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.596+0000 7f6179c00640 1 --2- 172.21.15.43:0/3213543128 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c6a92ec820 0x55c6a92eec50 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97eb78d268e70bb9 server_cookie=6600d839eae5dfa4 in_seq=0 out_seq=0 2024-03-28T22:37:34.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.596+0000 7f615ea00640 10 client.25849 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:34.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.597+0000 7f615ea00640 1 -- 172.21.15.43:0/3213543128 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c6a9244180 con 0x55c6a92ec820 2024-03-28T22:37:34.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.597+0000 7f615ea00640 10 client.25849 handle_command_reply: tid=0 2024-03-28T22:37:34.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.597+0000 7f617e9b9740 1 client.25849 shutdown 2024-03-28T22:37:34.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.597+0000 7f617e9b9740 1 -- 172.21.15.43:0/3213543128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6a92d3890 msgr2=0x55c6a92d99e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:34.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.597+0000 7f617e9b9740 1 --2- 172.21.15.43:0/3213543128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6a92d3890 0x55c6a92d99e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6160035000 tx=0x7f616002c800 comp rx=0 tx=0).stop 2024-03-28T22:37:34.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.597+0000 7f617e9b9740 1 -- 172.21.15.43:0/3213543128 shutdown_connections 2024-03-28T22:37:34.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.597+0000 7f617e9b9740 1 --2- 172.21.15.43:0/3213543128 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c6a92ec820 0x55c6a92eec50 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.597+0000 7f617e9b9740 1 --2- 172.21.15.43:0/3213543128 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6a92dc470 0x55c6a92d9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.597+0000 7f617e9b9740 1 --2- 172.21.15.43:0/3213543128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6a92d3890 0x55c6a92d99e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.597+0000 7f615ea00640 0 client.25849 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:34.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.597+0000 7f617e9b9740 1 --2- 172.21.15.43:0/3213543128 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6a92d2f70 0x55c6a92d94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.597+0000 7f617e9b9740 1 -- 172.21.15.43:0/3213543128 >> 172.21.15.43:0/3213543128 conn(0x55c6a92cd6a0 msgr2=0x55c6a92cf3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:34.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.597+0000 7f617e9b9740 1 -- 172.21.15.43:0/3213543128 shutdown_connections 2024-03-28T22:37:34.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.597+0000 7f617e9b9740 1 -- 172.21.15.43:0/3213543128 wait complete. 2024-03-28T22:37:34.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.598+0000 7f617e9b9740 20 client.25849 trim_cache size 0 max 16384 2024-03-28T22:37:34.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.618+0000 7f617e9b9740 1 Processor -- start 2024-03-28T22:37:34.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.618+0000 7f617e9b9740 1 -- start start 2024-03-28T22:37:34.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.619+0000 7f617e9b9740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6a92d2f70 0x55c6a92dbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.619+0000 7f617e9b9740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6a92d3890 0x55c6a92e8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.619+0000 7f617e9b9740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6a92dc470 0x55c6a92f4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.619+0000 7f617e9b9740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f61741b0de0 con 0x55c6a92d3890 2024-03-28T22:37:34.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.619+0000 7f617e9b9740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f61741b2920 con 0x55c6a92d2f70 2024-03-28T22:37:34.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.619+0000 7f617e9b9740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f61741b14b0 con 0x55c6a92dc470 2024-03-28T22:37:34.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.619+0000 7f6179c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6a92d2f70 0x55c6a92dbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.619+0000 7f6179c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6a92d2f70 0x55c6a92dbbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:36820/0 (socket says 172.21.15.43:36820) 2024-03-28T22:37:34.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.619+0000 7f6179c00640 1 -- 172.21.15.43:0/2333118526 learned_addr learned my addr 172.21.15.43:0/2333118526 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:34.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.619+0000 7f6173e00640 1 --2- 172.21.15.43:0/2333118526 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6a92d3890 0x55c6a92e8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.619+0000 7f6179c00640 1 --2- 172.21.15.43:0/2333118526 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6a92dc470 0x55c6a92f4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.619+0000 7f6179c00640 1 -- 172.21.15.43:0/2333118526 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6a92dc470 msgr2=0x55c6a92f4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:34.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.619+0000 7f6179c00640 1 --2- 172.21.15.43:0/2333118526 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6a92dc470 0x55c6a92f4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.619+0000 7f6179c00640 1 -- 172.21.15.43:0/2333118526 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6a92d3890 msgr2=0x55c6a92e8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:34.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.619+0000 7f6179c00640 1 --2- 172.21.15.43:0/2333118526 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6a92d3890 0x55c6a92e8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.619+0000 7f6179c00640 1 -- 172.21.15.43:0/2333118526 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6a92ef2d0 con 0x55c6a92d2f70 2024-03-28T22:37:34.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.620+0000 7f6179c00640 1 --2- 172.21.15.43:0/2333118526 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6a92d2f70 0x55c6a92dbbd0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f616c009730 tx=0x7f616c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc24c04c310989cb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:34.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.620+0000 7f6169600640 1 -- 172.21.15.43:0/2333118526 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f616c004000 con 0x55c6a92d2f70 2024-03-28T22:37:34.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.620+0000 7f6169600640 1 -- 172.21.15.43:0/2333118526 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f616c004680 con 0x55c6a92d2f70 2024-03-28T22:37:34.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.620+0000 7f6169600640 1 -- 172.21.15.43:0/2333118526 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f616c00fd20 con 0x55c6a92d2f70 2024-03-28T22:37:34.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.621+0000 7f617e9b9740 1 -- 172.21.15.43:0/2333118526 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6a92d2f70 msgr2=0x55c6a92dbbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:34.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.621+0000 7f617e9b9740 1 --2- 172.21.15.43:0/2333118526 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6a92d2f70 0x55c6a92dbbd0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f616c009730 tx=0x7f616c0118d0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.621+0000 7f617e9b9740 1 -- 172.21.15.43:0/2333118526 shutdown_connections 2024-03-28T22:37:34.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.621+0000 7f617e9b9740 1 --2- 172.21.15.43:0/2333118526 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6a92dc470 0x55c6a92f4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.621+0000 7f617e9b9740 1 --2- 172.21.15.43:0/2333118526 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6a92d3890 0x55c6a92e8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.621+0000 7f617e9b9740 1 --2- 172.21.15.43:0/2333118526 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6a92d2f70 0x55c6a92dbbd0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.621+0000 7f617e9b9740 1 -- 172.21.15.43:0/2333118526 >> 172.21.15.43:0/2333118526 conn(0x55c6a92cd6a0 msgr2=0x55c6a92cf360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:34.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.621+0000 7f617e9b9740 1 -- 172.21.15.43:0/2333118526 shutdown_connections 2024-03-28T22:37:34.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.621+0000 7f617e9b9740 1 -- 172.21.15.43:0/2333118526 wait complete. 2024-03-28T22:37:34.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.621+0000 7f617e9b9740 1 -- start start 2024-03-28T22:37:34.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.622+0000 7f617e9b9740 1 Processor -- start 2024-03-28T22:37:34.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.622+0000 7f617e9b9740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6a92d2f70 0x55c6a92e1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.622+0000 7f617e9b9740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6a92d3890 0x55c6a92e1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.622+0000 7f617e9b9740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6a92dc470 0x55c6a92e20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.622+0000 7f617e9b9740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f61741b2250 con 0x55c6a92d2f70 2024-03-28T22:37:34.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.622+0000 7f617e9b9740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f61741b1b80 con 0x55c6a92dc470 2024-03-28T22:37:34.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.622+0000 7f617e9b9740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f61741aedc0 con 0x55c6a92d3890 2024-03-28T22:37:34.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.622+0000 7f617e9b9740 10 client.0 _ll_register_callbacks cb 0x7f616c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:37:34.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.622+0000 7f6173e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6a92d3890 0x55c6a92e1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.622+0000 7f6173e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6a92d3890 0x55c6a92e1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:52640/0 (socket says 172.21.15.43:52640) 2024-03-28T22:37:34.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.622+0000 7f6173e00640 1 -- 172.21.15.43:0/479643005 learned_addr learned my addr 172.21.15.43:0/479643005 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:34.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.622+0000 7f6179c00640 1 --2- 172.21.15.43:0/479643005 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6a92d2f70 0x55c6a92e1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.623+0000 7f6179c00640 1 --2- 172.21.15.43:0/479643005 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6a92dc470 0x55c6a92e20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.623+0000 7f6173e00640 1 -- 172.21.15.43:0/479643005 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6a92dc470 msgr2=0x55c6a92e20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:34.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.623+0000 7f6173e00640 1 --2- 172.21.15.43:0/479643005 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6a92dc470 0x55c6a92e20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.623+0000 7f6173e00640 1 -- 172.21.15.43:0/479643005 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6a92d2f70 msgr2=0x55c6a92e1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:34.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.623+0000 7f6173e00640 1 --2- 172.21.15.43:0/479643005 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6a92d2f70 0x55c6a92e1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.623+0000 7f6173e00640 1 -- 172.21.15.43:0/479643005 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6a9352cc0 con 0x55c6a92d3890 2024-03-28T22:37:34.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.623+0000 7f6179c00640 1 --2- 172.21.15.43:0/479643005 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6a92d2f70 0x55c6a92e1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:37:34.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.623+0000 7f6173e00640 1 --2- 172.21.15.43:0/479643005 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6a92d3890 0x55c6a92e1b70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6160009800 tx=0x7f6160004610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b00e546639575943 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:34.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.623+0000 7f615ea00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:37:34.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.623+0000 7f615ea00640 1 -- 172.21.15.43:0/479643005 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6160003e20 con 0x55c6a92d3890 2024-03-28T22:37:34.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.624+0000 7f617e9b9740 1 -- 172.21.15.43:0/479643005 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f616c004680 con 0x55c6a92d3890 2024-03-28T22:37:34.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.624+0000 7f615ea00640 1 -- 172.21.15.43:0/479643005 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f616002c8b0 con 0x55c6a92d3890 2024-03-28T22:37:34.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.624+0000 7f615ea00640 1 -- 172.21.15.43:0/479643005 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6160065050 con 0x55c6a92d3890 2024-03-28T22:37:34.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.624+0000 7f617e9b9740 10 client.25855 fetch_fsmap learned FSMap version 29 2024-03-28T22:37:34.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.624+0000 7f617e9b9740 1 -- 172.21.15.43:0/479643005 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c6a9353f80 con 0x55c6a92d3890 2024-03-28T22:37:34.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.624+0000 7f615ea00640 1 -- 172.21.15.43:0/479643005 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f616c004680 con 0x55c6a92d3890 2024-03-28T22:37:34.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.624+0000 7f615ea00640 1 -- 172.21.15.43:0/479643005 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6160034c50 con 0x55c6a92d3890 2024-03-28T22:37:34.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.624+0000 7f617e9b9740 10 client.25855 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:37:34.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.624+0000 7f617e9b9740 10 client.25855 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:37:34.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.624+0000 7f617e9b9740 20 client.25855 populate_metadata read hostname 'smithi043' 2024-03-28T22:37:34.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.624+0000 7f617e9b9740 1 --2- 172.21.15.43:0/479643005 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c6a9354780 0x55c6a9356ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.624+0000 7f617e9b9740 4 client.25855 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:37:34.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.625+0000 7f617e9b9740 1 -- 172.21.15.43:0/479643005 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c6a923e4d0 con 0x55c6a9354780 2024-03-28T22:37:34.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.625+0000 7f6179c00640 1 --2- 172.21.15.43:0/479643005 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c6a9354780 0x55c6a9356ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.625+0000 7f6179c00640 1 --2- 172.21.15.43:0/479643005 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c6a9354780 0x55c6a9356ba0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6412251236389c18 server_cookie=58527266b86127f8 in_seq=0 out_seq=0 2024-03-28T22:37:34.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.625+0000 7f615ea00640 10 client.25855 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:34.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.626+0000 7f615ea00640 1 -- 172.21.15.43:0/479643005 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c6a923e4d0 con 0x55c6a9354780 2024-03-28T22:37:34.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.626+0000 7f615ea00640 10 client.25855 handle_command_reply: tid=0 2024-03-28T22:37:34.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.626+0000 7f617e9b9740 1 client.25855 shutdown 2024-03-28T22:37:34.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.626+0000 7f617e9b9740 1 -- 172.21.15.43:0/479643005 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6a92d3890 msgr2=0x55c6a92e1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:34.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.626+0000 7f617e9b9740 1 --2- 172.21.15.43:0/479643005 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6a92d3890 0x55c6a92e1b70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6160009800 tx=0x7f6160004610 comp rx=0 tx=0).stop 2024-03-28T22:37:34.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.627+0000 7f617e9b9740 1 -- 172.21.15.43:0/479643005 shutdown_connections 2024-03-28T22:37:34.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.627+0000 7f617e9b9740 1 --2- 172.21.15.43:0/479643005 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c6a9354780 0x55c6a9356ba0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.627+0000 7f617e9b9740 1 --2- 172.21.15.43:0/479643005 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6a92dc470 0x55c6a92e20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.627+0000 7f617e9b9740 1 --2- 172.21.15.43:0/479643005 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6a92d3890 0x55c6a92e1b70 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.627+0000 7f615ea00640 0 client.25855 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:34.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.627+0000 7f617e9b9740 1 --2- 172.21.15.43:0/479643005 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6a92d2f70 0x55c6a92e1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.627+0000 7f617e9b9740 1 -- 172.21.15.43:0/479643005 >> 172.21.15.43:0/479643005 conn(0x55c6a92cd6a0 msgr2=0x55c6a92d9440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:34.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.627+0000 7f617e9b9740 1 -- 172.21.15.43:0/479643005 shutdown_connections 2024-03-28T22:37:34.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.627+0000 7f617e9b9740 1 -- 172.21.15.43:0/479643005 wait complete. 2024-03-28T22:37:34.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.627+0000 7f617e9b9740 20 client.25855 trim_cache size 0 max 16384 2024-03-28T22:37:34.640 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:37:34.640 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-03-28T22:37:34.640 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "f064f169-cd47-4260-ab24-677b50c14929", 2024-03-28T22:37:34.640 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-03-28T22:37:34.641 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:37:34.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.628+0000 7f617bc00640 1 -- 172.21.15.43:0/2070476016 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f61580a2000 msgr2=0x7f61580a44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:34.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.628+0000 7f617bc00640 1 --2- 172.21.15.43:0/2070476016 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f61580a2000 0x7f61580a44c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f61640cbc30 tx=0x7f6164000bc0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.628+0000 7f617bc00640 1 -- 172.21.15.43:0/2070476016 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f61741038f0 msgr2=0x7f617419a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:34.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.628+0000 7f617bc00640 1 --2- 172.21.15.43:0/2070476016 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f61741038f0 0x7f617419a250 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f616400e9f0 tx=0x7f616400eeb0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.628+0000 7f617bc00640 1 -- 172.21.15.43:0/2070476016 shutdown_connections 2024-03-28T22:37:34.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.628+0000 7f617bc00640 1 --2- 172.21.15.43:0/2070476016 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f61580a2000 0x7f61580a44c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.628+0000 7f617bc00640 1 --2- 172.21.15.43:0/2070476016 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6174109900 0x7f617419eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.628+0000 7f617bc00640 1 --2- 172.21.15.43:0/2070476016 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f61741042b0 0x7f617419a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.628+0000 7f617bc00640 1 --2- 172.21.15.43:0/2070476016 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f61741038f0 0x7f617419a250 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.628+0000 7f617bc00640 1 -- 172.21.15.43:0/2070476016 >> 172.21.15.43:0/2070476016 conn(0x7f61740ff400 msgr2=0x7f61740ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:34.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.628+0000 7f617bc00640 1 -- 172.21.15.43:0/2070476016 shutdown_connections 2024-03-28T22:37:34.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.628+0000 7f617bc00640 1 -- 172.21.15.43:0/2070476016 wait complete. 2024-03-28T22:37:34.646 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:37:34.646 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:37:34.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.868+0000 7ffbb5e00640 1 -- 172.21.15.43:0/807059564 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbb0075c80 msgr2=0x7ffbb0076060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:34.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.868+0000 7ffbb5e00640 1 --2- 172.21.15.43:0/807059564 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbb0075c80 0x7ffbb0076060 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ffba4009930 tx=0x7ffba402bc20 comp rx=0 tx=0).stop 2024-03-28T22:37:34.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.868+0000 7ffbb5e00640 1 -- 172.21.15.43:0/807059564 shutdown_connections 2024-03-28T22:37:34.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.868+0000 7ffbb5e00640 1 --2- 172.21.15.43:0/807059564 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffbb00765a0 0x7ffbb0111f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.868+0000 7ffbb5e00640 1 --2- 172.21.15.43:0/807059564 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffbb0112470 0x7ffbb0114860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.868+0000 7ffbb5e00640 1 --2- 172.21.15.43:0/807059564 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbb0075c80 0x7ffbb0076060 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.868+0000 7ffbb5e00640 1 -- 172.21.15.43:0/807059564 >> 172.21.15.43:0/807059564 conn(0x7ffbb00ff260 msgr2=0x7ffbb0101680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:34.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.868+0000 7ffbb5e00640 1 -- 172.21.15.43:0/807059564 shutdown_connections 2024-03-28T22:37:34.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.868+0000 7ffbb5e00640 1 -- 172.21.15.43:0/807059564 wait complete. 2024-03-28T22:37:34.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.869+0000 7ffbb5e00640 1 Processor -- start 2024-03-28T22:37:34.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.870+0000 7ffbb5e00640 1 -- start start 2024-03-28T22:37:34.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.870+0000 7ffbb5e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbb0075c80 0x7ffbb01a4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.870+0000 7ffbb5e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffbb00765a0 0x7ffbb01a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.870+0000 7ffbb5e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffbb0112470 0x7ffbb019e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.870+0000 7ffbb5e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb0118020 con 0x7ffbb00765a0 2024-03-28T22:37:34.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.870+0000 7ffbb5e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb0117ea0 con 0x7ffbb0112470 2024-03-28T22:37:34.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.870+0000 7ffbb5e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb01181a0 con 0x7ffbb0075c80 2024-03-28T22:37:34.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.870+0000 7ffbb4a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbb0075c80 0x7ffbb01a4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.870+0000 7ffbb4a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbb0075c80 0x7ffbb01a4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:52666/0 (socket says 172.21.15.43:52666) 2024-03-28T22:37:34.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.870+0000 7ffbafe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffbb00765a0 0x7ffbb01a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.870+0000 7ffbb4a00640 1 -- 172.21.15.43:0/25077039 learned_addr learned my addr 172.21.15.43:0/25077039 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:34.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.870+0000 7ffbb5400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffbb0112470 0x7ffbb019e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.871+0000 7ffbb5400640 1 -- 172.21.15.43:0/25077039 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbb0075c80 msgr2=0x7ffbb01a4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:34.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.871+0000 7ffbb5400640 1 --2- 172.21.15.43:0/25077039 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbb0075c80 0x7ffbb01a4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.871+0000 7ffbb5400640 1 -- 172.21.15.43:0/25077039 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffbb00765a0 msgr2=0x7ffbb01a4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:34.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.871+0000 7ffbb5400640 1 --2- 172.21.15.43:0/25077039 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffbb00765a0 0x7ffbb01a4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.871+0000 7ffbb5400640 1 -- 172.21.15.43:0/25077039 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbb019efa0 con 0x7ffbb0112470 2024-03-28T22:37:34.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.871+0000 7ffbb4a00640 1 --2- 172.21.15.43:0/25077039 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbb0075c80 0x7ffbb01a4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:37:34.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.871+0000 7ffbb5400640 1 --2- 172.21.15.43:0/25077039 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffbb0112470 0x7ffbb019e6e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ffba8005fd0 tx=0x7ffba800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:34.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.871+0000 7ffbad600640 1 -- 172.21.15.43:0/25077039 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba800cdf0 con 0x7ffbb0112470 2024-03-28T22:37:34.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.871+0000 7ffbad600640 1 -- 172.21.15.43:0/25077039 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffba8005020 con 0x7ffbb0112470 2024-03-28T22:37:34.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.871+0000 7ffbb5e00640 1 -- 172.21.15.43:0/25077039 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbb019f290 con 0x7ffbb0112470 2024-03-28T22:37:34.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.871+0000 7ffbad600640 1 -- 172.21.15.43:0/25077039 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba80106d0 con 0x7ffbb0112470 2024-03-28T22:37:34.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.871+0000 7ffbb5e00640 1 -- 172.21.15.43:0/25077039 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbb01ab5a0 con 0x7ffbb0112470 2024-03-28T22:37:34.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.875+0000 7ffbba00e740 1 Processor -- start 2024-03-28T22:37:34.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.875+0000 7ffbad600640 1 -- 172.21.15.43:0/25077039 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 26) ==== 136932+0+0 (secure 0 0 0) 0x7ffba80054d0 con 0x7ffbb0112470 2024-03-28T22:37:34.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.875+0000 7ffbba00e740 1 -- start start 2024-03-28T22:37:34.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.876+0000 7ffbba00e740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597c4387db0 0x5597c4386b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.876+0000 7ffbba00e740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5597c43870d0 0x5597c4387550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.876+0000 7ffbba00e740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5597c4390440 0x5597c4392830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.876+0000 7ffbba00e740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5597c42eb100 con 0x5597c43870d0 2024-03-28T22:37:34.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.876+0000 7ffbba00e740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5597c4293ba0 con 0x5597c4387db0 2024-03-28T22:37:34.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.876+0000 7ffbba00e740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5597c42e71a0 con 0x5597c4390440 2024-03-28T22:37:34.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.876+0000 7ffbafe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597c4387db0 0x5597c4386b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.876+0000 7ffbb4a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5597c43870d0 0x5597c4387550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.876+0000 7ffbafe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597c4387db0 0x5597c4386b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:36852/0 (socket says 172.21.15.43:36852) 2024-03-28T22:37:34.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.876+0000 7ffbafe00640 1 -- 172.21.15.43:0/955618017 learned_addr learned my addr 172.21.15.43:0/955618017 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:34.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.876+0000 7ffbb4a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5597c43870d0 0x5597c4387550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47180/0 (socket says 172.21.15.43:47180) 2024-03-28T22:37:34.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.876+0000 7ffbafe00640 1 --2- 172.21.15.43:0/955618017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5597c4390440 0x5597c4392830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.877+0000 7ffbb4a00640 1 -- 172.21.15.43:0/955618017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5597c4390440 msgr2=0x5597c4392830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:34.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.877+0000 7ffbb4a00640 1 --2- 172.21.15.43:0/955618017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5597c4390440 0x5597c4392830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.877+0000 7ffbb4a00640 1 -- 172.21.15.43:0/955618017 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597c4387db0 msgr2=0x5597c4386b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:34.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.877+0000 7ffbb4a00640 1 --2- 172.21.15.43:0/955618017 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597c4387db0 0x5597c4386b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.877+0000 7ffbb4a00640 1 -- 172.21.15.43:0/955618017 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597c42c41a0 con 0x5597c43870d0 2024-03-28T22:37:34.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.877+0000 7ffbafe00640 1 --2- 172.21.15.43:0/955618017 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597c4387db0 0x5597c4386b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:37:34.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.877+0000 7ffbb4a00640 1 --2- 172.21.15.43:0/955618017 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5597c43870d0 0x5597c4387550 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ffba4000c00 tx=0x7ffba4005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ecf3d94f46d7a83 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:34.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.877+0000 7ffbad600640 1 --2- 172.21.15.43:0/25077039 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ffb980a2000 0x7ffb980a44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.877+0000 7ffbad600640 1 -- 172.21.15.43:0/25077039 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7ffba80ced10 con 0x7ffbb0112470 2024-03-28T22:37:34.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.877+0000 7ffbafe00640 1 --2- 172.21.15.43:0/25077039 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ffb980a2000 0x7ffb980a44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.877+0000 7ffb9dc00640 1 -- 172.21.15.43:0/955618017 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba4045070 con 0x5597c43870d0 2024-03-28T22:37:34.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.877+0000 7ffb9dc00640 1 -- 172.21.15.43:0/955618017 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffba402c640 con 0x5597c43870d0 2024-03-28T22:37:34.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.877+0000 7ffb9dc00640 1 -- 172.21.15.43:0/955618017 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba4004ee0 con 0x5597c43870d0 2024-03-28T22:37:34.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.878+0000 7ffbba00e740 1 -- 172.21.15.43:0/955618017 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5597c43870d0 msgr2=0x5597c4387550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:34.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.878+0000 7ffbba00e740 1 --2- 172.21.15.43:0/955618017 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5597c43870d0 0x5597c4387550 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ffba4000c00 tx=0x7ffba4005e80 comp rx=0 tx=0).stop 2024-03-28T22:37:34.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.878+0000 7ffbafe00640 1 --2- 172.21.15.43:0/25077039 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ffb980a2000 0x7ffb980a44c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ffba0009fd0 tx=0x7ffba0009370 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:34.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.878+0000 7ffbba00e740 1 -- 172.21.15.43:0/955618017 shutdown_connections 2024-03-28T22:37:34.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.878+0000 7ffbba00e740 1 --2- 172.21.15.43:0/955618017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5597c4390440 0x5597c4392830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.878+0000 7ffbba00e740 1 --2- 172.21.15.43:0/955618017 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5597c43870d0 0x5597c4387550 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.878+0000 7ffbba00e740 1 --2- 172.21.15.43:0/955618017 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597c4387db0 0x5597c4386b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.878+0000 7ffbba00e740 1 -- 172.21.15.43:0/955618017 >> 172.21.15.43:0/955618017 conn(0x5597c434da10 msgr2=0x5597c4381b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:34.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.878+0000 7ffbba00e740 1 -- 172.21.15.43:0/955618017 shutdown_connections 2024-03-28T22:37:34.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.878+0000 7ffbba00e740 1 -- 172.21.15.43:0/955618017 wait complete. 2024-03-28T22:37:34.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.879+0000 7ffbba00e740 1 -- start start 2024-03-28T22:37:34.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.879+0000 7ffbba00e740 1 Processor -- start 2024-03-28T22:37:34.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.879+0000 7ffbba00e740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5597c43870d0 0x5597c438d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.879+0000 7ffbba00e740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5597c4387db0 0x5597c438d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.879+0000 7ffbba00e740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597c4390440 0x5597c438de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.880+0000 7ffbba00e740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5597c4393250 con 0x5597c4387db0 2024-03-28T22:37:34.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.880+0000 7ffbba00e740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5597c4289250 con 0x5597c4390440 2024-03-28T22:37:34.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.880+0000 7ffbba00e740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5597c43933d0 con 0x5597c43870d0 2024-03-28T22:37:34.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.880+0000 7ffbba00e740 10 client.0 _ll_register_callbacks cb 0x5597c3ffebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:37:34.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.880+0000 7ffbafe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5597c4387db0 0x5597c438d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.880+0000 7ffbb5400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597c4390440 0x5597c438de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.880+0000 7ffbafe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5597c4387db0 0x5597c438d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47196/0 (socket says 172.21.15.43:47196) 2024-03-28T22:37:34.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.880+0000 7ffbafe00640 1 -- 172.21.15.43:0/167387230 learned_addr learned my addr 172.21.15.43:0/167387230 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:34.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.880+0000 7ffbb5400640 1 -- 172.21.15.43:0/167387230 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5597c43870d0 msgr2=0x5597c438d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:34.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.880+0000 7ffbb5400640 1 --2- 172.21.15.43:0/167387230 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5597c43870d0 0x5597c438d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.880+0000 7ffbb5400640 1 -- 172.21.15.43:0/167387230 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5597c4387db0 msgr2=0x5597c438d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:34.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.880+0000 7ffbb5400640 1 --2- 172.21.15.43:0/167387230 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5597c4387db0 0x5597c438d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.880+0000 7ffbb5400640 1 -- 172.21.15.43:0/167387230 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597c42a5960 con 0x5597c4390440 2024-03-28T22:37:34.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.880+0000 7ffbb5400640 1 --2- 172.21.15.43:0/167387230 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597c4390440 0x5597c438de40 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ffba80cbf70 tx=0x7ffba809b890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77c59b04bd74218c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:34.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.881+0000 7ffb92000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:37:34.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.881+0000 7ffb92000640 1 -- 172.21.15.43:0/167387230 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba80cc450 con 0x5597c4390440 2024-03-28T22:37:34.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.881+0000 7ffb92000640 1 -- 172.21.15.43:0/167387230 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7ffba809b3f0 con 0x5597c4390440 2024-03-28T22:37:34.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.881+0000 7ffbba00e740 1 -- 172.21.15.43:0/167387230 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffba402c640 con 0x5597c4390440 2024-03-28T22:37:34.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.881+0000 7ffb92000640 1 -- 172.21.15.43:0/167387230 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ffba80a3930 con 0x5597c4390440 2024-03-28T22:37:34.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.881+0000 7ffbba00e740 10 client.25898 fetch_fsmap learned FSMap version 29 2024-03-28T22:37:34.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.881+0000 7ffbba00e740 1 -- 172.21.15.43:0/167387230 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5597c40c97d0 con 0x5597c4390440 2024-03-28T22:37:34.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.881+0000 7ffb92000640 1 -- 172.21.15.43:0/167387230 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffba402c640 con 0x5597c4390440 2024-03-28T22:37:34.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.881+0000 7ffb92000640 1 -- 172.21.15.43:0/167387230 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ffba809a5e0 con 0x5597c4390440 2024-03-28T22:37:34.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.881+0000 7ffbba00e740 10 client.25898 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:37:34.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.881+0000 7ffbba00e740 10 client.25898 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:37:34.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.881+0000 7ffbba00e740 20 client.25898 populate_metadata read hostname 'smithi043' 2024-03-28T22:37:34.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.881+0000 7ffbba00e740 1 --2- 172.21.15.43:0/167387230 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5597c43a08b0 0x5597c43a2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.881+0000 7ffbba00e740 4 client.25898 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:37:34.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.881+0000 7ffbba00e740 1 -- 172.21.15.43:0/167387230 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5597c42f8640 con 0x5597c43a08b0 2024-03-28T22:37:34.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.882+0000 7ffbb4a00640 1 --2- 172.21.15.43:0/167387230 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5597c43a08b0 0x5597c43a2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.882+0000 7ffbb4a00640 1 --2- 172.21.15.43:0/167387230 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5597c43a08b0 0x5597c43a2d70 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b107d3322d69d56 server_cookie=6e8d19f129118c4e in_seq=0 out_seq=0 2024-03-28T22:37:34.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.882+0000 7ffb92000640 10 client.25898 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:34.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.883+0000 7ffb92000640 1 -- 172.21.15.43:0/167387230 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5597c42f8640 con 0x5597c43a08b0 2024-03-28T22:37:34.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.883+0000 7ffb92000640 10 client.25898 handle_command_reply: tid=0 2024-03-28T22:37:34.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.883+0000 7ffbba00e740 1 client.25898 shutdown 2024-03-28T22:37:34.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.883+0000 7ffbba00e740 1 -- 172.21.15.43:0/167387230 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597c4390440 msgr2=0x5597c438de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:34.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.883+0000 7ffbba00e740 1 --2- 172.21.15.43:0/167387230 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597c4390440 0x5597c438de40 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ffba80cbf70 tx=0x7ffba809b890 comp rx=0 tx=0).stop 2024-03-28T22:37:34.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.883+0000 7ffbba00e740 1 -- 172.21.15.43:0/167387230 shutdown_connections 2024-03-28T22:37:34.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.883+0000 7ffbba00e740 1 --2- 172.21.15.43:0/167387230 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5597c43a08b0 0x5597c43a2d70 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.883+0000 7ffbba00e740 1 --2- 172.21.15.43:0/167387230 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597c4390440 0x5597c438de40 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.883+0000 7ffbba00e740 1 --2- 172.21.15.43:0/167387230 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5597c4387db0 0x5597c438d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.883+0000 7ffbba00e740 1 --2- 172.21.15.43:0/167387230 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5597c43870d0 0x5597c438d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.883+0000 7ffb92000640 0 client.25898 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:34.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.883+0000 7ffbba00e740 1 -- 172.21.15.43:0/167387230 >> 172.21.15.43:0/167387230 conn(0x5597c434da10 msgr2=0x5597c4383870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:34.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.883+0000 7ffbba00e740 1 -- 172.21.15.43:0/167387230 shutdown_connections 2024-03-28T22:37:34.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.883+0000 7ffbba00e740 1 -- 172.21.15.43:0/167387230 wait complete. 2024-03-28T22:37:34.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.884+0000 7ffbba00e740 20 client.25898 trim_cache size 0 max 16384 2024-03-28T22:37:34.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.903+0000 7ffbba00e740 1 Processor -- start 2024-03-28T22:37:34.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.903+0000 7ffbba00e740 1 -- start start 2024-03-28T22:37:34.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.903+0000 7ffbba00e740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5597c43870d0 0x5597c438faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.903+0000 7ffbba00e740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5597c4387db0 0x5597c438c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.903+0000 7ffbba00e740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597c4390440 0x5597c439c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.903+0000 7ffbba00e740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb01b4f60 con 0x5597c4387db0 2024-03-28T22:37:34.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.903+0000 7ffbba00e740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb01b6aa0 con 0x5597c4390440 2024-03-28T22:37:34.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.903+0000 7ffbba00e740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb01b5630 con 0x5597c43870d0 2024-03-28T22:37:34.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.903+0000 7ffbb5400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597c4390440 0x5597c439c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.904+0000 7ffbafe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5597c4387db0 0x5597c438c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.904+0000 7ffbb5400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597c4390440 0x5597c439c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:43126/0 (socket says 172.21.15.43:43126) 2024-03-28T22:37:34.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.904+0000 7ffbb5400640 1 -- 172.21.15.43:0/166338768 learned_addr learned my addr 172.21.15.43:0/166338768 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:34.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.904+0000 7ffbb5400640 1 -- 172.21.15.43:0/166338768 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5597c43870d0 msgr2=0x5597c438faf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T22:37:34.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.904+0000 7ffbb5400640 1 --2- 172.21.15.43:0/166338768 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5597c43870d0 0x5597c438faf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.904+0000 7ffbb5400640 1 -- 172.21.15.43:0/166338768 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5597c4387db0 msgr2=0x5597c438c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:34.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.904+0000 7ffbb5400640 1 --2- 172.21.15.43:0/166338768 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5597c4387db0 0x5597c438c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.904+0000 7ffbb5400640 1 -- 172.21.15.43:0/166338768 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597c3fff3f0 con 0x5597c4390440 2024-03-28T22:37:34.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.904+0000 7ffbb5400640 1 --2- 172.21.15.43:0/166338768 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597c4390440 0x5597c439c030 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ffba809ae90 tx=0x7ffba80996a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c87a9f70b32e217 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:34.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.904+0000 7ffb9d200640 1 -- 172.21.15.43:0/166338768 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba80a34a0 con 0x5597c4390440 2024-03-28T22:37:34.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.904+0000 7ffb9d200640 1 -- 172.21.15.43:0/166338768 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffba80a3640 con 0x5597c4390440 2024-03-28T22:37:34.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.904+0000 7ffb9d200640 1 -- 172.21.15.43:0/166338768 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba80cc4b0 con 0x5597c4390440 2024-03-28T22:37:34.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.905+0000 7ffbba00e740 1 -- 172.21.15.43:0/166338768 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597c4390440 msgr2=0x5597c439c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:34.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.905+0000 7ffbba00e740 1 --2- 172.21.15.43:0/166338768 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597c4390440 0x5597c439c030 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ffba809ae90 tx=0x7ffba80996a0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.905+0000 7ffbba00e740 1 -- 172.21.15.43:0/166338768 shutdown_connections 2024-03-28T22:37:34.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.905+0000 7ffbba00e740 1 --2- 172.21.15.43:0/166338768 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597c4390440 0x5597c439c030 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.905+0000 7ffbba00e740 1 --2- 172.21.15.43:0/166338768 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5597c4387db0 0x5597c438c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.905+0000 7ffbba00e740 1 --2- 172.21.15.43:0/166338768 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5597c43870d0 0x5597c438faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.905+0000 7ffbba00e740 1 -- 172.21.15.43:0/166338768 >> 172.21.15.43:0/166338768 conn(0x5597c434da10 msgr2=0x5597c4380d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:34.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.905+0000 7ffbba00e740 1 -- 172.21.15.43:0/166338768 shutdown_connections 2024-03-28T22:37:34.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.905+0000 7ffbba00e740 1 -- 172.21.15.43:0/166338768 wait complete. 2024-03-28T22:37:34.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.906+0000 7ffbba00e740 1 -- start start 2024-03-28T22:37:34.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.906+0000 7ffbba00e740 1 Processor -- start 2024-03-28T22:37:34.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.906+0000 7ffbba00e740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597c43870d0 0x5597c438a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.906+0000 7ffbba00e740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5597c4387db0 0x5597c438ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.906+0000 7ffbba00e740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5597c4390440 0x5597c4405c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.906+0000 7ffbba00e740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb01b63d0 con 0x5597c4390440 2024-03-28T22:37:34.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.906+0000 7ffbba00e740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb01b5d00 con 0x5597c43870d0 2024-03-28T22:37:34.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.906+0000 7ffbba00e740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb01b2f40 con 0x5597c4387db0 2024-03-28T22:37:34.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.906+0000 7ffbba00e740 10 client.0 _ll_register_callbacks cb 0x7ffba80cbe30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:37:34.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.907+0000 7ffbb4a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597c43870d0 0x5597c438a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.907+0000 7ffbb4a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597c43870d0 0x5597c438a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:43134/0 (socket says 172.21.15.43:43134) 2024-03-28T22:37:34.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.907+0000 7ffbb4a00640 1 -- 172.21.15.43:0/1262112624 learned_addr learned my addr 172.21.15.43:0/1262112624 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:34.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.907+0000 7ffbb5400640 1 --2- 172.21.15.43:0/1262112624 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5597c4390440 0x5597c4405c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.907+0000 7ffbb4a00640 1 -- 172.21.15.43:0/1262112624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5597c4387db0 msgr2=0x5597c438ab40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T22:37:34.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.907+0000 7ffbb4a00640 1 --2- 172.21.15.43:0/1262112624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5597c4387db0 0x5597c438ab40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.907+0000 7ffbb4a00640 1 -- 172.21.15.43:0/1262112624 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5597c4390440 msgr2=0x5597c4405c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:34.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.907+0000 7ffbb4a00640 1 --2- 172.21.15.43:0/1262112624 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5597c4390440 0x5597c4405c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.907+0000 7ffbb4a00640 1 -- 172.21.15.43:0/1262112624 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597c3fbf0f0 con 0x5597c43870d0 2024-03-28T22:37:34.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.907+0000 7ffbb4a00640 1 --2- 172.21.15.43:0/1262112624 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597c43870d0 0x5597c438a600 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ffba4036040 tx=0x7ffba402cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb2e93d97121d690 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:34.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.907+0000 7ffb92000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:37:34.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.907+0000 7ffb92000640 1 -- 172.21.15.43:0/1262112624 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba4050070 con 0x5597c43870d0 2024-03-28T22:37:34.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.907+0000 7ffbba00e740 1 -- 172.21.15.43:0/1262112624 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffba80a3640 con 0x5597c43870d0 2024-03-28T22:37:34.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.907+0000 7ffb92000640 1 -- 172.21.15.43:0/1262112624 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7ffba4004c70 con 0x5597c43870d0 2024-03-28T22:37:34.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.908+0000 7ffb92000640 1 -- 172.21.15.43:0/1262112624 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ffba4045040 con 0x5597c43870d0 2024-03-28T22:37:34.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.908+0000 7ffbba00e740 10 client.25904 fetch_fsmap learned FSMap version 29 2024-03-28T22:37:34.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.908+0000 7ffbba00e740 1 -- 172.21.15.43:0/1262112624 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5597c4407280 con 0x5597c43870d0 2024-03-28T22:37:34.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.908+0000 7ffb92000640 1 -- 172.21.15.43:0/1262112624 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffba80a3640 con 0x5597c43870d0 2024-03-28T22:37:34.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.908+0000 7ffb92000640 1 -- 172.21.15.43:0/1262112624 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ffba404a570 con 0x5597c43870d0 2024-03-28T22:37:34.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.908+0000 7ffbba00e740 10 client.25904 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:37:34.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.908+0000 7ffbba00e740 10 client.25904 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:37:34.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.908+0000 7ffbba00e740 20 client.25904 populate_metadata read hostname 'smithi043' 2024-03-28T22:37:34.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.908+0000 7ffbba00e740 1 --2- 172.21.15.43:0/1262112624 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5597c44075c0 0x5597c44099b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:34.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.908+0000 7ffbba00e740 4 client.25904 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:37:34.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.908+0000 7ffbba00e740 1 -- 172.21.15.43:0/1262112624 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5597c42f2990 con 0x5597c44075c0 2024-03-28T22:37:34.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.908+0000 7ffbafe00640 1 --2- 172.21.15.43:0/1262112624 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5597c44075c0 0x5597c44099b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:34.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.909+0000 7ffbafe00640 1 --2- 172.21.15.43:0/1262112624 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5597c44075c0 0x5597c44099b0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6b6f503f047ae5c server_cookie=33a38fe19fe8b3f3 in_seq=0 out_seq=0 2024-03-28T22:37:34.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.909+0000 7ffb92000640 10 client.25904 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:34.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.909+0000 7ffb92000640 1 -- 172.21.15.43:0/1262112624 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5597c42f2990 con 0x5597c44075c0 2024-03-28T22:37:34.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.909+0000 7ffb92000640 10 client.25904 handle_command_reply: tid=0 2024-03-28T22:37:34.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.909+0000 7ffbba00e740 1 client.25904 shutdown 2024-03-28T22:37:34.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.909+0000 7ffbba00e740 1 -- 172.21.15.43:0/1262112624 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597c43870d0 msgr2=0x5597c438a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:34.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.909+0000 7ffbba00e740 1 --2- 172.21.15.43:0/1262112624 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597c43870d0 0x5597c438a600 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ffba4036040 tx=0x7ffba402cbf0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.909+0000 7ffbba00e740 1 -- 172.21.15.43:0/1262112624 shutdown_connections 2024-03-28T22:37:34.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.909+0000 7ffbba00e740 1 --2- 172.21.15.43:0/1262112624 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5597c4390440 0x5597c4405c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.909+0000 7ffbba00e740 1 --2- 172.21.15.43:0/1262112624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5597c4387db0 0x5597c438ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.909+0000 7ffbba00e740 1 --2- 172.21.15.43:0/1262112624 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5597c44075c0 0x5597c44099b0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.909+0000 7ffbba00e740 1 --2- 172.21.15.43:0/1262112624 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5597c43870d0 0x5597c438a600 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.909+0000 7ffbba00e740 1 -- 172.21.15.43:0/1262112624 >> 172.21.15.43:0/1262112624 conn(0x5597c434da10 msgr2=0x5597c438d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:34.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.909+0000 7ffb92000640 0 client.25904 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:34.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.910+0000 7ffbba00e740 1 -- 172.21.15.43:0/1262112624 shutdown_connections 2024-03-28T22:37:34.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.910+0000 7ffbba00e740 1 -- 172.21.15.43:0/1262112624 wait complete. 2024-03-28T22:37:34.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.910+0000 7ffbba00e740 20 client.25904 trim_cache size 0 max 16384 2024-03-28T22:37:34.912 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:37:34.912 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:37:34.912 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-03-28T22:37:34.912 INFO:teuthology.orchestra.run.smithi043.stdout: "f064f169-cd47-4260-ab24-677b50c14929": { 2024-03-28T22:37:34.912 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-03-28T22:37:34.912 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "f064f169-cd47-4260-ab24-677b50c14929", 2024-03-28T22:37:34.912 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-03-28T22:37:34.912 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:37:34.912 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:37:34.913 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:37:34.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.910+0000 7ffbb5e00640 1 -- 172.21.15.43:0/25077039 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ffb980a2000 msgr2=0x7ffb980a44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:34.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.910+0000 7ffbb5e00640 1 --2- 172.21.15.43:0/25077039 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ffb980a2000 0x7ffb980a44c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ffba0009fd0 tx=0x7ffba0009370 comp rx=0 tx=0).stop 2024-03-28T22:37:34.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.910+0000 7ffbb5e00640 1 -- 172.21.15.43:0/25077039 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffbb0112470 msgr2=0x7ffbb019e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:34.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.910+0000 7ffbb5e00640 1 --2- 172.21.15.43:0/25077039 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffbb0112470 0x7ffbb019e6e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ffba8005fd0 tx=0x7ffba800ef10 comp rx=0 tx=0).stop 2024-03-28T22:37:34.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.910+0000 7ffbb5e00640 1 -- 172.21.15.43:0/25077039 shutdown_connections 2024-03-28T22:37:34.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.910+0000 7ffbb5e00640 1 --2- 172.21.15.43:0/25077039 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ffb980a2000 0x7ffb980a44c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.910+0000 7ffbb5e00640 1 --2- 172.21.15.43:0/25077039 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffbb0112470 0x7ffbb019e6e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.910+0000 7ffbb5e00640 1 --2- 172.21.15.43:0/25077039 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffbb00765a0 0x7ffbb01a4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.910+0000 7ffbb5e00640 1 --2- 172.21.15.43:0/25077039 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffbb0075c80 0x7ffbb01a4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:34.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.911+0000 7ffbb5e00640 1 -- 172.21.15.43:0/25077039 >> 172.21.15.43:0/25077039 conn(0x7ffbb00ff260 msgr2=0x7ffbb0101650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:34.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.911+0000 7ffbb5e00640 1 -- 172.21.15.43:0/25077039 shutdown_connections 2024-03-28T22:37:34.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:37:34.911+0000 7ffbb5e00640 1 -- 172.21.15.43:0/25077039 wait complete. 2024-03-28T22:37:34.926 INFO:tasks.cephfs.filesystem:scrub status for tag:f064f169-cd47-4260-ab24-677b50c14929 - {'path': '/', 'tag': 'f064f169-cd47-4260-ab24-677b50c14929', 'options': 'recursive,force'} 2024-03-28T22:37:35.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:34 smithi043 ceph-mon[31823]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.6 KiB/s wr, 17 op/s 2024-03-28T22:37:35.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:34 smithi082 ceph-mon[36471]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.6 KiB/s wr, 17 op/s 2024-03-28T22:37:35.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:34 smithi172 ceph-mon[36566]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.6 KiB/s wr, 17 op/s 2024-03-28T22:37:36.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:35 smithi043 ceph-mon[31823]: scrub queued for path: / 2024-03-28T22:37:36.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:35 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:37:36.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:35 smithi043 ceph-mon[31823]: scrub summary: active paths [/] 2024-03-28T22:37:36.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:35 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:37:36.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:35 smithi172 ceph-mon[36566]: scrub queued for path: / 2024-03-28T22:37:36.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:35 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:37:36.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:35 smithi172 ceph-mon[36566]: scrub summary: active paths [/] 2024-03-28T22:37:36.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:35 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:37:36.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:35 smithi082 ceph-mon[36471]: scrub queued for path: / 2024-03-28T22:37:36.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:35 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:37:36.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:35 smithi082 ceph-mon[36471]: scrub summary: active paths [/] 2024-03-28T22:37:36.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:35 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:37:37.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:36 smithi043 ceph-mon[31823]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-03-28T22:37:37.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:36 smithi172 ceph-mon[36566]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-03-28T22:37:37.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:36 smithi082 ceph-mon[36471]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-03-28T22:37:39.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:38 smithi043 ceph-mon[31823]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.6 KiB/s wr, 15 op/s 2024-03-28T22:37:39.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:38 smithi043 ceph-mon[31823]: scrub summary: idle 2024-03-28T22:37:39.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:38 smithi172 ceph-mon[36566]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.6 KiB/s wr, 15 op/s 2024-03-28T22:37:39.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:38 smithi172 ceph-mon[36566]: scrub summary: idle 2024-03-28T22:37:39.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:38 smithi082 ceph-mon[36471]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.6 KiB/s wr, 15 op/s 2024-03-28T22:37:39.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:38 smithi082 ceph-mon[36471]: scrub summary: idle 2024-03-28T22:37:40.584 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.586+0000 7f0f3b200640 1 -- 172.21.15.43:0/2009005531 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f3c1030c0 msgr2=0x7f0f3c10b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:40.584 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.586+0000 7f0f3b200640 1 --2- 172.21.15.43:0/2009005531 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f3c1030c0 0x7f0f3c10b5a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f0f28009900 tx=0x7f0f2802bcd0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.584 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.586+0000 7f0f3b200640 1 -- 172.21.15.43:0/2009005531 shutdown_connections 2024-03-28T22:37:40.584 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.586+0000 7f0f3b200640 1 --2- 172.21.15.43:0/2009005531 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f3c1030c0 0x7f0f3c10b5a0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.584 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.586+0000 7f0f3b200640 1 --2- 172.21.15.43:0/2009005531 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f3c10bb70 0x7f0f3c10dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.584 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.586+0000 7f0f3b200640 1 --2- 172.21.15.43:0/2009005531 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0f3c1027a0 0x7f0f3c102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.584 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.586+0000 7f0f3b200640 1 -- 172.21.15.43:0/2009005531 >> 172.21.15.43:0/2009005531 conn(0x7f0f3c0fbf60 msgr2=0x7f0f3c0fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:40.585 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.586+0000 7f0f3b200640 1 -- 172.21.15.43:0/2009005531 shutdown_connections 2024-03-28T22:37:40.585 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.586+0000 7f0f3b200640 1 -- 172.21.15.43:0/2009005531 wait complete. 2024-03-28T22:37:40.586 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.587+0000 7f0f3b200640 1 Processor -- start 2024-03-28T22:37:40.586 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.587+0000 7f0f3b200640 1 -- start start 2024-03-28T22:37:40.586 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.588+0000 7f0f3b200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f3c1027a0 0x7f0f3c197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:40.586 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.588+0000 7f0f3b200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0f3c1030c0 0x7f0f3c1983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:40.586 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.588+0000 7f0f3b200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f3c10bb70 0x7f0f3c19c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:40.586 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.588+0000 7f0f3b200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0f3c111950 con 0x7f0f3c1030c0 2024-03-28T22:37:40.586 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.588+0000 7f0f3b200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f0f3c1117d0 con 0x7f0f3c1027a0 2024-03-28T22:37:40.586 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.588+0000 7f0f3b200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f0f3c111ad0 con 0x7f0f3c10bb70 2024-03-28T22:37:40.586 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.588+0000 7f0f39e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f3c1027a0 0x7f0f3c197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:40.586 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.588+0000 7f0f39400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0f3c1030c0 0x7f0f3c1983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:40.586 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.588+0000 7f0f39e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f3c1027a0 0x7f0f3c197e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:43150/0 (socket says 172.21.15.43:43150) 2024-03-28T22:37:40.586 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.588+0000 7f0f39e00640 1 -- 172.21.15.43:0/510879431 learned_addr learned my addr 172.21.15.43:0/510879431 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:40.587 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.588+0000 7f0f39e00640 1 -- 172.21.15.43:0/510879431 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f3c10bb70 msgr2=0x7f0f3c19c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:40.587 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.588+0000 7f0f39e00640 1 --2- 172.21.15.43:0/510879431 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f3c10bb70 0x7f0f3c19c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.588 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.588+0000 7f0f39e00640 1 -- 172.21.15.43:0/510879431 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0f3c1030c0 msgr2=0x7f0f3c1983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:40.588 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.588+0000 7f0f39e00640 1 --2- 172.21.15.43:0/510879431 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0f3c1030c0 0x7f0f3c1983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.588 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.588+0000 7f0f39e00640 1 -- 172.21.15.43:0/510879431 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f3c19ce10 con 0x7f0f3c1027a0 2024-03-28T22:37:40.588 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.589+0000 7f0f39e00640 1 --2- 172.21.15.43:0/510879431 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f3c1027a0 0x7f0f3c197e70 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c00e9f0 tx=0x7f0f2c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:40.588 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.589+0000 7f0f32a00640 1 -- 172.21.15.43:0/510879431 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c00cd80 con 0x7f0f3c1027a0 2024-03-28T22:37:40.588 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.589+0000 7f0f32a00640 1 -- 172.21.15.43:0/510879431 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f2c010910 con 0x7f0f3c1027a0 2024-03-28T22:37:40.588 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.589+0000 7f0f32a00640 1 -- 172.21.15.43:0/510879431 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c018990 con 0x7f0f3c1027a0 2024-03-28T22:37:40.588 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.589+0000 7f0f3b200640 1 -- 172.21.15.43:0/510879431 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f3c19d100 con 0x7f0f3c1027a0 2024-03-28T22:37:40.588 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.589+0000 7f0f3b200640 1 -- 172.21.15.43:0/510879431 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f3c1a4990 con 0x7f0f3c1027a0 2024-03-28T22:37:40.590 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.592+0000 7f0f32a00640 1 -- 172.21.15.43:0/510879431 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 26) ==== 136932+0+0 (secure 0 0 0) 0x7f0f2c010ac0 con 0x7f0f3c1027a0 2024-03-28T22:37:40.592 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.594+0000 7f0f32a00640 1 --2- 172.21.15.43:0/510879431 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0f200a1ff0 0x7f0f200a44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:40.592 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.594+0000 7f0f32a00640 1 -- 172.21.15.43:0/510879431 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f0f2c0cea50 con 0x7f0f3c1027a0 2024-03-28T22:37:40.592 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.594+0000 7f0f43777740 1 Processor -- start 2024-03-28T22:37:40.592 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.594+0000 7f0f39400640 1 --2- 172.21.15.43:0/510879431 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0f200a1ff0 0x7f0f200a44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:40.592 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.594+0000 7f0f43777740 1 -- start start 2024-03-28T22:37:40.594 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.595+0000 7f0f43777740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56349dbeaf30 0x56349dbeb310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:40.594 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.595+0000 7f0f43777740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56349dbeb850 0x56349dbf8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:40.594 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.595+0000 7f0f43777740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56349dbf9070 0x56349dbfb460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:40.594 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.595+0000 7f0f43777740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56349db4e640 con 0x56349dbeaf30 2024-03-28T22:37:40.594 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.595+0000 7f0f43777740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x56349daf7810 con 0x56349dbeb850 2024-03-28T22:37:40.594 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.595+0000 7f0f43777740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x56349db4a6e0 con 0x56349dbf9070 2024-03-28T22:37:40.595 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.595+0000 7f0f39400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56349dbeb850 0x56349dbf8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:40.595 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.595+0000 7f0f39e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56349dbeaf30 0x56349dbeb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:40.595 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.595+0000 7f0f39400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56349dbeb850 0x56349dbf8b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:43156/0 (socket says 172.21.15.43:43156) 2024-03-28T22:37:40.595 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.595+0000 7f0f39400640 1 -- 172.21.15.43:0/2077984195 learned_addr learned my addr 172.21.15.43:0/2077984195 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:40.595 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.595+0000 7f0f39400640 1 --2- 172.21.15.43:0/510879431 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0f200a1ff0 0x7f0f200a44b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0f28005ab0 tx=0x7f0f28005a20 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:40.595 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.595+0000 7f0f3a800640 1 --2- 172.21.15.43:0/2077984195 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56349dbf9070 0x56349dbfb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:40.596 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.595+0000 7f0f39e00640 1 -- 172.21.15.43:0/2077984195 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56349dbf9070 msgr2=0x56349dbfb460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:40.596 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.595+0000 7f0f39e00640 1 --2- 172.21.15.43:0/2077984195 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56349dbf9070 0x56349dbfb460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.596 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.595+0000 7f0f39e00640 1 -- 172.21.15.43:0/2077984195 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56349dbeb850 msgr2=0x56349dbf8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:40.596 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.595+0000 7f0f39e00640 1 --2- 172.21.15.43:0/2077984195 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56349dbeb850 0x56349dbf8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.596 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.595+0000 7f0f39e00640 1 -- 172.21.15.43:0/2077984195 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56349d863210 con 0x56349dbeaf30 2024-03-28T22:37:40.597 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.596+0000 7f0f39400640 1 --2- 172.21.15.43:0/2077984195 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56349dbeb850 0x56349dbf8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:37:40.597 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.596+0000 7f0f39e00640 1 --2- 172.21.15.43:0/2077984195 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56349dbeaf30 0x56349dbeb310 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c0cbd40 tx=0x7f0f2c09b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a76f96c5806b5e87 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:40.597 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.596+0000 7f0f27000640 1 -- 172.21.15.43:0/2077984195 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c09f080 con 0x56349dbeaf30 2024-03-28T22:37:40.597 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.596+0000 7f0f27000640 1 -- 172.21.15.43:0/2077984195 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f2c0a36d0 con 0x56349dbeaf30 2024-03-28T22:37:40.597 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.596+0000 7f0f27000640 1 -- 172.21.15.43:0/2077984195 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c09a380 con 0x56349dbeaf30 2024-03-28T22:37:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.597+0000 7f0f43777740 1 -- 172.21.15.43:0/2077984195 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56349dbeaf30 msgr2=0x56349dbeb310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.597+0000 7f0f43777740 1 --2- 172.21.15.43:0/2077984195 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56349dbeaf30 0x56349dbeb310 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c0cbd40 tx=0x7f0f2c09b630 comp rx=0 tx=0).stop 2024-03-28T22:37:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.597+0000 7f0f43777740 1 -- 172.21.15.43:0/2077984195 shutdown_connections 2024-03-28T22:37:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.597+0000 7f0f43777740 1 --2- 172.21.15.43:0/2077984195 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56349dbeb850 0x56349dbf8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.597+0000 7f0f43777740 1 --2- 172.21.15.43:0/2077984195 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56349dbf9070 0x56349dbfb460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.597+0000 7f0f43777740 1 --2- 172.21.15.43:0/2077984195 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56349dbeaf30 0x56349dbeb310 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.597+0000 7f0f43777740 1 -- 172.21.15.43:0/2077984195 >> 172.21.15.43:0/2077984195 conn(0x56349dbe5e90 msgr2=0x56349dbe6270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.597+0000 7f0f43777740 1 -- 172.21.15.43:0/2077984195 shutdown_connections 2024-03-28T22:37:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.597+0000 7f0f43777740 1 -- 172.21.15.43:0/2077984195 wait complete. 2024-03-28T22:37:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.598+0000 7f0f43777740 1 -- start start 2024-03-28T22:37:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.598+0000 7f0f43777740 1 Processor -- start 2024-03-28T22:37:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.598+0000 7f0f43777740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56349dbeaf30 0x56349dbf6100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.598+0000 7f0f43777740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56349dbeb850 0x56349dbf6640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.598+0000 7f0f43777740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56349dbf9070 0x56349dbf6b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.598+0000 7f0f43777740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56349dbfbbe0 con 0x56349dbf9070 2024-03-28T22:37:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.598+0000 7f0f43777740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x56349daecd00 con 0x56349dbeaf30 2024-03-28T22:37:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.598+0000 7f0f43777740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x56349dbfbd60 con 0x56349dbeb850 2024-03-28T22:37:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.598+0000 7f0f43777740 10 client.0 _ll_register_callbacks cb 0x56349dad7490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:37:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.599+0000 7f0f39e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56349dbeaf30 0x56349dbf6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.599+0000 7f0f39400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56349dbeb850 0x56349dbf6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.599+0000 7f0f3a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56349dbf9070 0x56349dbf6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.599+0000 7f0f39400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56349dbeb850 0x56349dbf6640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43274/0 (socket says 172.21.15.43:43274) 2024-03-28T22:37:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.599+0000 7f0f39400640 1 -- 172.21.15.43:0/1911098313 learned_addr learned my addr 172.21.15.43:0/1911098313 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.599+0000 7f0f39e00640 1 -- 172.21.15.43:0/1911098313 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56349dbeb850 msgr2=0x56349dbf6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.599+0000 7f0f39e00640 1 --2- 172.21.15.43:0/1911098313 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56349dbeb850 0x56349dbf6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.599+0000 7f0f39e00640 1 -- 172.21.15.43:0/1911098313 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56349dbf9070 msgr2=0x56349dbf6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.599+0000 7f0f39e00640 1 --2- 172.21.15.43:0/1911098313 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56349dbf9070 0x56349dbf6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.599+0000 7f0f39e00640 1 -- 172.21.15.43:0/1911098313 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56349d92f620 con 0x56349dbeaf30 2024-03-28T22:37:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.599+0000 7f0f39e00640 1 --2- 172.21.15.43:0/1911098313 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56349dbeaf30 0x56349dbf6100 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c0991e0 tx=0x7f0f2c0cc2a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf7c4d28956eb4d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.599+0000 7f0f1b400640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:37:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.600+0000 7f0f1b400640 1 -- 172.21.15.43:0/1911098313 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c0b7020 con 0x56349dbeaf30 2024-03-28T22:37:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.600+0000 7f0f1b400640 1 -- 172.21.15.43:0/1911098313 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f0f2c09a590 con 0x56349dbeaf30 2024-03-28T22:37:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.600+0000 7f0f43777740 1 -- 172.21.15.43:0/1911098313 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f2c0a36d0 con 0x56349dbeaf30 2024-03-28T22:37:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.600+0000 7f0f1b400640 1 -- 172.21.15.43:0/1911098313 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0f2c09f080 con 0x56349dbeaf30 2024-03-28T22:37:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.600+0000 7f0f43777740 10 client.25916 fetch_fsmap learned FSMap version 29 2024-03-28T22:37:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.600+0000 7f0f43777740 1 -- 172.21.15.43:0/1911098313 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56349dc07b10 con 0x56349dbeaf30 2024-03-28T22:37:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.600+0000 7f0f1b400640 1 -- 172.21.15.43:0/1911098313 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f2c0a36d0 con 0x56349dbeaf30 2024-03-28T22:37:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.600+0000 7f0f1b400640 1 -- 172.21.15.43:0/1911098313 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f0f2c099df0 con 0x56349dbeaf30 2024-03-28T22:37:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.600+0000 7f0f43777740 10 client.25916 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:37:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.600+0000 7f0f43777740 10 client.25916 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:37:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.600+0000 7f0f43777740 20 client.25916 populate_metadata read hostname 'smithi043' 2024-03-28T22:37:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.600+0000 7f0f43777740 1 --2- 172.21.15.43:0/1911098313 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56349dc08e60 0x56349dc0b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.600+0000 7f0f43777740 4 client.25916 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:37:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.600+0000 7f0f43777740 1 -- 172.21.15.43:0/1911098313 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56349db62030 con 0x56349dc08e60 2024-03-28T22:37:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.601+0000 7f0f39400640 1 --2- 172.21.15.43:0/1911098313 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56349dc08e60 0x56349dc0b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.601+0000 7f0f39400640 1 --2- 172.21.15.43:0/1911098313 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56349dc08e60 0x56349dc0b250 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60b12e269dda4d3 server_cookie=9c3f6a769dd21737 in_seq=0 out_seq=0 2024-03-28T22:37:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.601+0000 7f0f1b400640 10 client.25916 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.602+0000 7f0f1b400640 1 -- 172.21.15.43:0/1911098313 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56349db62030 con 0x56349dc08e60 2024-03-28T22:37:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.602+0000 7f0f1b400640 10 client.25916 handle_command_reply: tid=0 2024-03-28T22:37:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.602+0000 7f0f43777740 1 client.25916 shutdown 2024-03-28T22:37:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.602+0000 7f0f43777740 1 -- 172.21.15.43:0/1911098313 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56349dbeaf30 msgr2=0x56349dbf6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.602+0000 7f0f43777740 1 --2- 172.21.15.43:0/1911098313 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56349dbeaf30 0x56349dbf6100 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c0991e0 tx=0x7f0f2c0cc2a0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.602+0000 7f0f43777740 1 -- 172.21.15.43:0/1911098313 shutdown_connections 2024-03-28T22:37:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.602+0000 7f0f43777740 1 --2- 172.21.15.43:0/1911098313 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56349dbf9070 0x56349dbf6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.602+0000 7f0f43777740 1 --2- 172.21.15.43:0/1911098313 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56349dbeb850 0x56349dbf6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.602+0000 7f0f43777740 1 --2- 172.21.15.43:0/1911098313 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56349dc08e60 0x56349dc0b250 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.607 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.602+0000 7f0f43777740 1 --2- 172.21.15.43:0/1911098313 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56349dbeaf30 0x56349dbf6100 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.607 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.602+0000 7f0f1b400640 0 client.25916 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:40.607 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.602+0000 7f0f43777740 1 -- 172.21.15.43:0/1911098313 >> 172.21.15.43:0/1911098313 conn(0x56349dbe5e90 msgr2=0x56349dbe7be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:40.607 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.602+0000 7f0f43777740 1 -- 172.21.15.43:0/1911098313 shutdown_connections 2024-03-28T22:37:40.607 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.603+0000 7f0f43777740 1 -- 172.21.15.43:0/1911098313 wait complete. 2024-03-28T22:37:40.607 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.603+0000 7f0f43777740 20 client.25916 trim_cache size 0 max 16384 2024-03-28T22:37:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.633+0000 7f0f43777740 1 Processor -- start 2024-03-28T22:37:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.633+0000 7f0f43777740 1 -- start start 2024-03-28T22:37:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.634+0000 7f0f43777740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56349dbeaf30 0x56349dbf8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.634+0000 7f0f43777740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56349dbeb850 0x56349dbf5260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.634+0000 7f0f43777740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56349dbf9070 0x56349dc04980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.634+0000 7f0f43777740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0f3c1ac8d0 con 0x56349dbeaf30 2024-03-28T22:37:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.634+0000 7f0f43777740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f0f3c1ae410 con 0x56349dbf9070 2024-03-28T22:37:40.634 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.634+0000 7f0f43777740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f0f3c1acfa0 con 0x56349dbeb850 2024-03-28T22:37:40.634 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.634+0000 7f0f39e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56349dbeaf30 0x56349dbf8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:40.634 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.634+0000 7f0f3a800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56349dbf9070 0x56349dc04980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:40.634 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.634+0000 7f0f39e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56349dbeaf30 0x56349dbf8830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42442/0 (socket says 172.21.15.43:42442) 2024-03-28T22:37:40.634 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.634+0000 7f0f39e00640 1 -- 172.21.15.43:0/2162768420 learned_addr learned my addr 172.21.15.43:0/2162768420 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:40.634 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.634+0000 7f0f3a800640 1 -- 172.21.15.43:0/2162768420 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56349dbeb850 msgr2=0x56349dbf5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:40.634 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.634+0000 7f0f3a800640 1 --2- 172.21.15.43:0/2162768420 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56349dbeb850 0x56349dbf5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.634 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.634+0000 7f0f3a800640 1 -- 172.21.15.43:0/2162768420 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56349dbeaf30 msgr2=0x56349dbf8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:40.634 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.634+0000 7f0f3a800640 1 --2- 172.21.15.43:0/2162768420 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56349dbeaf30 0x56349dbf8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.634 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.634+0000 7f0f3a800640 1 -- 172.21.15.43:0/2162768420 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56349dbf4650 con 0x56349dbf9070 2024-03-28T22:37:40.634 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.634+0000 7f0f39e00640 1 --2- 172.21.15.43:0/2162768420 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56349dbeaf30 0x56349dbf8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:37:40.634 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.635+0000 7f0f3a800640 1 --2- 172.21.15.43:0/2162768420 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56349dbf9070 0x56349dc04980 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x56349dbe7020 tx=0x7f0f34003e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a67862a7b96e14c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:40.634 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.635+0000 7f0f26600640 1 -- 172.21.15.43:0/2162768420 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f34004550 con 0x56349dbf9070 2024-03-28T22:37:40.634 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.635+0000 7f0f26600640 1 -- 172.21.15.43:0/2162768420 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f340046f0 con 0x56349dbf9070 2024-03-28T22:37:40.635 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.635+0000 7f0f26600640 1 -- 172.21.15.43:0/2162768420 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f340117b0 con 0x56349dbf9070 2024-03-28T22:37:40.637 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.636+0000 7f0f43777740 1 -- 172.21.15.43:0/2162768420 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56349dbf9070 msgr2=0x56349dc04980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:40.638 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.636+0000 7f0f43777740 1 --2- 172.21.15.43:0/2162768420 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56349dbf9070 0x56349dc04980 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x56349dbe7020 tx=0x7f0f34003e00 comp rx=0 tx=0).stop 2024-03-28T22:37:40.638 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.636+0000 7f0f43777740 1 -- 172.21.15.43:0/2162768420 shutdown_connections 2024-03-28T22:37:40.638 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.636+0000 7f0f43777740 1 --2- 172.21.15.43:0/2162768420 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56349dbf9070 0x56349dc04980 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.638 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.636+0000 7f0f43777740 1 --2- 172.21.15.43:0/2162768420 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56349dbeb850 0x56349dbf5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.638 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.636+0000 7f0f43777740 1 --2- 172.21.15.43:0/2162768420 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56349dbeaf30 0x56349dbf8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.638 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.636+0000 7f0f43777740 1 -- 172.21.15.43:0/2162768420 >> 172.21.15.43:0/2162768420 conn(0x56349dbe5e90 msgr2=0x56349dbe6270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:40.638 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.636+0000 7f0f43777740 1 -- 172.21.15.43:0/2162768420 shutdown_connections 2024-03-28T22:37:40.638 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.636+0000 7f0f43777740 1 -- 172.21.15.43:0/2162768420 wait complete. 2024-03-28T22:37:40.638 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.636+0000 7f0f43777740 1 -- start start 2024-03-28T22:37:40.638 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.637+0000 7f0f43777740 1 Processor -- start 2024-03-28T22:37:40.638 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.637+0000 7f0f43777740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56349dbeaf30 0x56349dbfcf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:40.638 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.637+0000 7f0f43777740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56349dbeb850 0x56349dbfd490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:40.638 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.637+0000 7f0f43777740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56349dbf9070 0x56349dc6de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.637+0000 7f0f43777740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0f3c1add40 con 0x56349dbf9070 2024-03-28T22:37:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.637+0000 7f0f43777740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f0f3c1ad670 con 0x56349dbeaf30 2024-03-28T22:37:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.637+0000 7f0f43777740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f0f3c1aa8b0 con 0x56349dbeb850 2024-03-28T22:37:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.637+0000 7f0f43777740 10 client.0 _ll_register_callbacks cb 0x7f0f340093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:37:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.637+0000 7f0f39e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56349dbeaf30 0x56349dbfcf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.637+0000 7f0f39e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56349dbeaf30 0x56349dbfcf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:43166/0 (socket says 172.21.15.43:43166) 2024-03-28T22:37:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.637+0000 7f0f3a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56349dbf9070 0x56349dc6de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.637+0000 7f0f39e00640 1 -- 172.21.15.43:0/2778223257 learned_addr learned my addr 172.21.15.43:0/2778223257 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.638+0000 7f0f39400640 1 --2- 172.21.15.43:0/2778223257 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56349dbeb850 0x56349dbfd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.638+0000 7f0f39e00640 1 -- 172.21.15.43:0/2778223257 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56349dbeb850 msgr2=0x56349dbfd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.638+0000 7f0f39e00640 1 --2- 172.21.15.43:0/2778223257 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56349dbeb850 0x56349dbfd490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.638+0000 7f0f39e00640 1 -- 172.21.15.43:0/2778223257 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56349dbf9070 msgr2=0x56349dc6de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.638+0000 7f0f39e00640 1 --2- 172.21.15.43:0/2778223257 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56349dbf9070 0x56349dc6de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.638+0000 7f0f39e00640 1 -- 172.21.15.43:0/2778223257 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56349dc6e360 con 0x56349dbeaf30 2024-03-28T22:37:40.640 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.638+0000 7f0f39400640 1 --2- 172.21.15.43:0/2778223257 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56349dbeb850 0x56349dbfd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:37:40.640 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.638+0000 7f0f39e00640 1 --2- 172.21.15.43:0/2778223257 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56349dbeaf30 0x56349dbfcf50 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c09aec0 tx=0x7f0f2c097300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61135b9d777b7e33 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:40.640 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.638+0000 7f0f1b400640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:37:40.640 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.638+0000 7f0f1b400640 1 -- 172.21.15.43:0/2778223257 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c0b7040 con 0x56349dbeaf30 2024-03-28T22:37:40.640 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.638+0000 7f0f1b400640 1 -- 172.21.15.43:0/2778223257 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f0f2c0b92c0 con 0x56349dbeaf30 2024-03-28T22:37:40.640 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.638+0000 7f0f43777740 1 -- 172.21.15.43:0/2778223257 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f340046f0 con 0x56349dbeaf30 2024-03-28T22:37:40.640 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.639+0000 7f0f1b400640 1 -- 172.21.15.43:0/2778223257 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0f2c09af30 con 0x56349dbeaf30 2024-03-28T22:37:40.640 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.639+0000 7f0f43777740 10 client.25922 fetch_fsmap learned FSMap version 29 2024-03-28T22:37:40.640 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.639+0000 7f0f43777740 1 -- 172.21.15.43:0/2778223257 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56349dc6e360 con 0x56349dbeaf30 2024-03-28T22:37:40.640 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.639+0000 7f0f1b400640 1 -- 172.21.15.43:0/2778223257 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f340046f0 con 0x56349dbeaf30 2024-03-28T22:37:40.640 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.639+0000 7f0f1b400640 1 -- 172.21.15.43:0/2778223257 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f0f2c09ffa0 con 0x56349dbeaf30 2024-03-28T22:37:40.640 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.639+0000 7f0f43777740 10 client.25922 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:37:40.640 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.639+0000 7f0f43777740 10 client.25922 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:37:40.640 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.639+0000 7f0f43777740 20 client.25922 populate_metadata read hostname 'smithi043' 2024-03-28T22:37:40.640 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.639+0000 7f0f43777740 1 --2- 172.21.15.43:0/2778223257 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56349dc6f940 0x56349dc71d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:40.641 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.639+0000 7f0f43777740 4 client.25922 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T22:37:40.641 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.639+0000 7f0f43777740 1 -- 172.21.15.43:0/2778223257 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56349db5bb80 con 0x56349dc6f940 2024-03-28T22:37:40.641 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.639+0000 7f0f39400640 1 --2- 172.21.15.43:0/2778223257 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56349dc6f940 0x56349dc71d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:40.641 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.640+0000 7f0f39400640 1 --2- 172.21.15.43:0/2778223257 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56349dc6f940 0x56349dc71d30 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4e9808bc84a99d3 server_cookie=ec028487828f69e4 in_seq=0 out_seq=0 2024-03-28T22:37:40.641 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.640+0000 7f0f1b400640 10 client.25922 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.643+0000 7f0f1b400640 1 -- 172.21.15.43:0/2778223257 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56349db5bb80 con 0x56349dc6f940 2024-03-28T22:37:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.643+0000 7f0f1b400640 10 client.25922 handle_command_reply: tid=0 2024-03-28T22:37:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.643+0000 7f0f43777740 1 client.25922 shutdown 2024-03-28T22:37:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.643+0000 7f0f43777740 1 -- 172.21.15.43:0/2778223257 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56349dbeaf30 msgr2=0x56349dbfcf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.643+0000 7f0f43777740 1 --2- 172.21.15.43:0/2778223257 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56349dbeaf30 0x56349dbfcf50 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c09aec0 tx=0x7f0f2c097300 comp rx=0 tx=0).stop 2024-03-28T22:37:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.643+0000 7f0f43777740 1 -- 172.21.15.43:0/2778223257 shutdown_connections 2024-03-28T22:37:40.643 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.643+0000 7f0f43777740 1 --2- 172.21.15.43:0/2778223257 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56349dbf9070 0x56349dc6de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.643 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.643+0000 7f0f43777740 1 --2- 172.21.15.43:0/2778223257 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56349dbeb850 0x56349dbfd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.643 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.643+0000 7f0f43777740 1 --2- 172.21.15.43:0/2778223257 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56349dc6f940 0x56349dc71d30 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.643 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.643+0000 7f0f43777740 1 --2- 172.21.15.43:0/2778223257 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56349dbeaf30 0x56349dbfcf50 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.643 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.643+0000 7f0f43777740 1 -- 172.21.15.43:0/2778223257 >> 172.21.15.43:0/2778223257 conn(0x56349dbe5e90 msgr2=0x56349dbe4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:40.643 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.643+0000 7f0f1b400640 0 client.25922 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:40.643 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.643+0000 7f0f43777740 1 -- 172.21.15.43:0/2778223257 shutdown_connections 2024-03-28T22:37:40.643 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.643+0000 7f0f43777740 1 -- 172.21.15.43:0/2778223257 wait complete. 2024-03-28T22:37:40.643 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.644+0000 7f0f43777740 20 client.25922 trim_cache size 0 max 16384 2024-03-28T22:37:40.643 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:{ 2024-03-28T22:37:40.643 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "message": "", 2024-03-28T22:37:40.644 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "return_code": 0 2024-03-28T22:37:40.644 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:} 2024-03-28T22:37:40.644 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.644+0000 7f0f3b200640 1 -- 172.21.15.43:0/510879431 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0f200a1ff0 msgr2=0x7f0f200a44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:40.644 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.644+0000 7f0f3b200640 1 --2- 172.21.15.43:0/510879431 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0f200a1ff0 0x7f0f200a44b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0f28005ab0 tx=0x7f0f28005a20 comp rx=0 tx=0).stop 2024-03-28T22:37:40.644 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.644+0000 7f0f3b200640 1 -- 172.21.15.43:0/510879431 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f3c1027a0 msgr2=0x7f0f3c197e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:40.644 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.644+0000 7f0f3b200640 1 --2- 172.21.15.43:0/510879431 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f3c1027a0 0x7f0f3c197e70 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c00e9f0 tx=0x7f0f2c00eeb0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.645 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.644+0000 7f0f3b200640 1 -- 172.21.15.43:0/510879431 shutdown_connections 2024-03-28T22:37:40.645 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.644+0000 7f0f3b200640 1 --2- 172.21.15.43:0/510879431 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0f200a1ff0 0x7f0f200a44b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.645 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.644+0000 7f0f3b200640 1 --2- 172.21.15.43:0/510879431 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0f3c10bb70 0x7f0f3c19c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.645 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.644+0000 7f0f3b200640 1 --2- 172.21.15.43:0/510879431 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0f3c1030c0 0x7f0f3c1983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.645 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.644+0000 7f0f3b200640 1 --2- 172.21.15.43:0/510879431 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f3c1027a0 0x7f0f3c197e70 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:40.645 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.644+0000 7f0f3b200640 1 -- 172.21.15.43:0/510879431 >> 172.21.15.43:0/510879431 conn(0x7f0f3c0fbf60 msgr2=0x7f0f3c0fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:40.645 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.645+0000 7f0f3b200640 1 -- 172.21.15.43:0/510879431 shutdown_connections 2024-03-28T22:37:40.645 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:40.645+0000 7f0f3b200640 1 -- 172.21.15.43:0/510879431 wait complete. 2024-03-28T22:37:41.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:40 smithi043 ceph-mon[31823]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.6 KiB/s wr, 16 op/s 2024-03-28T22:37:41.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:40 smithi172 ceph-mon[36566]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.6 KiB/s wr, 16 op/s 2024-03-28T22:37:41.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:40 smithi082 ceph-mon[36471]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.6 KiB/s wr, 16 op/s 2024-03-28T22:37:43.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:42 smithi043 ceph-mon[31823]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.6 KiB/s wr, 14 op/s 2024-03-28T22:37:43.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:42 smithi172 ceph-mon[36566]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.6 KiB/s wr, 14 op/s 2024-03-28T22:37:43.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:42 smithi082 ceph-mon[36471]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.6 KiB/s wr, 14 op/s 2024-03-28T22:37:45.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:44 smithi043 ceph-mon[31823]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.8 KiB/s wr, 18 op/s 2024-03-28T22:37:45.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:44 smithi172 ceph-mon[36566]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.8 KiB/s wr, 18 op/s 2024-03-28T22:37:45.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:44 smithi082 ceph-mon[36471]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.8 KiB/s wr, 18 op/s 2024-03-28T22:37:47.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:46 smithi043 ceph-mon[31823]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-28T22:37:47.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:46 smithi172 ceph-mon[36566]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-28T22:37:47.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:46 smithi082 ceph-mon[36471]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-28T22:37:49.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:48 smithi043 ceph-mon[31823]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T22:37:49.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:48 smithi172 ceph-mon[36566]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T22:37:49.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:48 smithi082 ceph-mon[36471]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T22:37:51.250 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:50 smithi082 ceph-mon[36471]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-28T22:37:51.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:50 smithi043 ceph-mon[31823]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-28T22:37:51.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:50 smithi172 ceph-mon[36566]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-28T22:37:53.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:52 smithi043 ceph-mon[31823]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T22:37:53.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:52 smithi172 ceph-mon[36566]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T22:37:53.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:52 smithi082 ceph-mon[36471]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T22:37:53.789 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.791+0000 7fb283200640 1 -- 172.21.15.43:0/1362787847 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb27c107630 msgr2=0x7fb27c107a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:53.789 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.791+0000 7fb283200640 1 --2- 172.21.15.43:0/1362787847 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb27c107630 0x7fb27c107a10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb270009930 tx=0x7fb27002bc30 comp rx=0 tx=0).stop 2024-03-28T22:37:53.790 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.791+0000 7fb283200640 1 -- 172.21.15.43:0/1362787847 shutdown_connections 2024-03-28T22:37:53.790 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.791+0000 7fb283200640 1 --2- 172.21.15.43:0/1362787847 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb27c101f30 0x7fb27c10e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.790 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.791+0000 7fb283200640 1 --2- 172.21.15.43:0/1362787847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb27c101570 0x7fb27c1019f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.790 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.791+0000 7fb283200640 1 --2- 172.21.15.43:0/1362787847 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb27c107630 0x7fb27c107a10 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.790 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.791+0000 7fb283200640 1 -- 172.21.15.43:0/1362787847 >> 172.21.15.43:0/1362787847 conn(0x7fb27c0fd060 msgr2=0x7fb27c0ff480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:53.790 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.792+0000 7fb283200640 1 -- 172.21.15.43:0/1362787847 shutdown_connections 2024-03-28T22:37:53.790 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.792+0000 7fb283200640 1 -- 172.21.15.43:0/1362787847 wait complete. 2024-03-28T22:37:53.791 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.793+0000 7fb283200640 1 Processor -- start 2024-03-28T22:37:53.791 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.793+0000 7fb283200640 1 -- start start 2024-03-28T22:37:53.793 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.794+0000 7fb283200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb27c101570 0x7fb27c197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:53.793 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.794+0000 7fb283200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb27c101f30 0x7fb27c198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:53.793 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.794+0000 7fb283200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb27c107630 0x7fb27c19c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:53.793 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.794+0000 7fb283200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c111960 con 0x7fb27c107630 2024-03-28T22:37:53.793 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.794+0000 7fb283200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c1117e0 con 0x7fb27c101570 2024-03-28T22:37:53.793 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.794+0000 7fb283200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c111ae0 con 0x7fb27c101f30 2024-03-28T22:37:53.793 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.794+0000 7fb281e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb27c101570 0x7fb27c197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:53.793 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.794+0000 7fb281400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb27c101f30 0x7fb27c198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:53.793 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.794+0000 7fb281e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb27c101570 0x7fb27c197e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:41820/0 (socket says 172.21.15.43:41820) 2024-03-28T22:37:53.793 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.794+0000 7fb281e00640 1 -- 172.21.15.43:0/4041200900 learned_addr learned my addr 172.21.15.43:0/4041200900 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.794+0000 7fb282800640 1 --2- 172.21.15.43:0/4041200900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb27c107630 0x7fb27c19c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.794+0000 7fb281e00640 1 -- 172.21.15.43:0/4041200900 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb27c101f30 msgr2=0x7fb27c198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.794+0000 7fb281e00640 1 --2- 172.21.15.43:0/4041200900 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb27c101f30 0x7fb27c198380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.794+0000 7fb281e00640 1 -- 172.21.15.43:0/4041200900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb27c107630 msgr2=0x7fb27c19c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.794+0000 7fb281e00640 1 --2- 172.21.15.43:0/4041200900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb27c107630 0x7fb27c19c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.794+0000 7fb281e00640 1 -- 172.21.15.43:0/4041200900 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb27c19cde0 con 0x7fb27c101570 2024-03-28T22:37:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.795+0000 7fb281e00640 1 --2- 172.21.15.43:0/4041200900 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb27c101570 0x7fb27c197e40 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb270009900 tx=0x7fb270004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.795+0000 7fb276a00640 1 -- 172.21.15.43:0/4041200900 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb270043020 con 0x7fb27c101570 2024-03-28T22:37:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.795+0000 7fb276a00640 1 -- 172.21.15.43:0/4041200900 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2700054e0 con 0x7fb27c101570 2024-03-28T22:37:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.795+0000 7fb283200640 1 -- 172.21.15.43:0/4041200900 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb27c19d070 con 0x7fb27c101570 2024-03-28T22:37:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.795+0000 7fb276a00640 1 -- 172.21.15.43:0/4041200900 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb270003b80 con 0x7fb27c101570 2024-03-28T22:37:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.795+0000 7fb283200640 1 -- 172.21.15.43:0/4041200900 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb27c1a4960 con 0x7fb27c101570 2024-03-28T22:37:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.799+0000 7fb276a00640 1 -- 172.21.15.43:0/4041200900 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 26) ==== 136932+0+0 (secure 0 0 0) 0x7fb270046020 con 0x7fb27c101570 2024-03-28T22:37:53.797 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.800+0000 7fb2875cb740 1 Processor -- start 2024-03-28T22:37:53.797 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.800+0000 7fb2875cb740 1 -- start start 2024-03-28T22:37:53.800 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.800+0000 7fb2875cb740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccf5e86570 0x55ccf5e86950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:53.800 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.800+0000 7fb2875cb740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccf5e86e90 0x55ccf5e93b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:53.800 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.800+0000 7fb2875cb740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccf5e94070 0x55ccf5e96460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:53.800 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.800+0000 7fb2875cb740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ccf5de9640 con 0x55ccf5e86e90 2024-03-28T22:37:53.800 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.800+0000 7fb2875cb740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55ccf5d92810 con 0x55ccf5e94070 2024-03-28T22:37:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.800+0000 7fb2875cb740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55ccf5de56e0 con 0x55ccf5e86570 2024-03-28T22:37:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.800+0000 7fb276a00640 1 --2- 172.21.15.43:0/4041200900 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb2640a1ff0 0x7fb2640a44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.801+0000 7fb282800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccf5e86570 0x55ccf5e86950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.801+0000 7fb281400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccf5e94070 0x55ccf5e96460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.801+0000 7fb282800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccf5e86570 0x55ccf5e86950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:32886/0 (socket says 172.21.15.43:32886) 2024-03-28T22:37:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.801+0000 7fb282800640 1 -- 172.21.15.43:0/1244454210 learned_addr learned my addr 172.21.15.43:0/1244454210 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.801+0000 7fb276a00640 1 -- 172.21.15.43:0/4041200900 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fb2700f07c0 con 0x7fb27c101570 2024-03-28T22:37:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.801+0000 7fb281e00640 1 --2- 172.21.15.43:0/1244454210 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccf5e86e90 0x55ccf5e93b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.801+0000 7fb282800640 1 --2- 172.21.15.43:0/4041200900 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb2640a1ff0 0x7fb2640a44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.801+0000 7fb281400640 1 -- 172.21.15.43:0/1244454210 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccf5e86570 msgr2=0x55ccf5e86950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.801+0000 7fb281400640 1 --2- 172.21.15.43:0/1244454210 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccf5e86570 0x55ccf5e86950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.801+0000 7fb281400640 1 -- 172.21.15.43:0/1244454210 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccf5e86e90 msgr2=0x55ccf5e93b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.801+0000 7fb281400640 1 --2- 172.21.15.43:0/1244454210 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccf5e86e90 0x55ccf5e93b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.801+0000 7fb281400640 1 -- 172.21.15.43:0/1244454210 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ccf5afe210 con 0x55ccf5e94070 2024-03-28T22:37:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.802+0000 7fb281400640 1 --2- 172.21.15.43:0/1244454210 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccf5e94070 0x55ccf5e96460 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb26c009bf0 tx=0x7fb26c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc7668ea91ed6370 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.802+0000 7fb282800640 1 --2- 172.21.15.43:0/4041200900 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb2640a1ff0 0x7fb2640a44b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb278005c40 tx=0x7fb278004ab0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.802+0000 7fb26b000640 1 -- 172.21.15.43:0/1244454210 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb26c004050 con 0x55ccf5e94070 2024-03-28T22:37:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.802+0000 7fb26b000640 1 -- 172.21.15.43:0/1244454210 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb26c005500 con 0x55ccf5e94070 2024-03-28T22:37:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.802+0000 7fb26b000640 1 -- 172.21.15.43:0/1244454210 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb26c004e10 con 0x55ccf5e94070 2024-03-28T22:37:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.803+0000 7fb2875cb740 1 -- 172.21.15.43:0/1244454210 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccf5e94070 msgr2=0x55ccf5e96460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.803+0000 7fb2875cb740 1 --2- 172.21.15.43:0/1244454210 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccf5e94070 0x55ccf5e96460 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb26c009bf0 tx=0x7fb26c011810 comp rx=0 tx=0).stop 2024-03-28T22:37:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.803+0000 7fb2875cb740 1 -- 172.21.15.43:0/1244454210 shutdown_connections 2024-03-28T22:37:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.803+0000 7fb2875cb740 1 --2- 172.21.15.43:0/1244454210 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccf5e94070 0x55ccf5e96460 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.803+0000 7fb2875cb740 1 --2- 172.21.15.43:0/1244454210 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccf5e86e90 0x55ccf5e93b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.803+0000 7fb2875cb740 1 --2- 172.21.15.43:0/1244454210 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccf5e86570 0x55ccf5e86950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.803+0000 7fb2875cb740 1 -- 172.21.15.43:0/1244454210 >> 172.21.15.43:0/1244454210 conn(0x55ccf5e80e90 msgr2=0x55ccf5e81270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.803+0000 7fb2875cb740 1 -- 172.21.15.43:0/1244454210 shutdown_connections 2024-03-28T22:37:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.803+0000 7fb2875cb740 1 -- 172.21.15.43:0/1244454210 wait complete. 2024-03-28T22:37:53.807 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.803+0000 7fb2875cb740 1 -- start start 2024-03-28T22:37:53.807 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.804+0000 7fb2875cb740 1 Processor -- start 2024-03-28T22:37:53.807 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.804+0000 7fb2875cb740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccf5e86570 0x55ccf5e910b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:53.807 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.804+0000 7fb2875cb740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccf5e86e90 0x55ccf5e915f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:53.807 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.804+0000 7fb2875cb740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccf5e94070 0x55ccf5e91b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:53.807 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.804+0000 7fb2875cb740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ccf5e96ca0 con 0x55ccf5e86570 2024-03-28T22:37:53.807 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.804+0000 7fb2875cb740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55ccf5d87d00 con 0x55ccf5e86e90 2024-03-28T22:37:53.807 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.804+0000 7fb2875cb740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55ccf5e96e20 con 0x55ccf5e94070 2024-03-28T22:37:53.807 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.804+0000 7fb2875cb740 10 client.0 _ll_register_callbacks cb 0x55ccf5d72490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:37:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.805+0000 7fb281400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccf5e86e90 0x55ccf5e915f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.805+0000 7fb281400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccf5e86e90 0x55ccf5e915f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:41834/0 (socket says 172.21.15.43:41834) 2024-03-28T22:37:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.805+0000 7fb282800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccf5e94070 0x55ccf5e91b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.805+0000 7fb281400640 1 -- 172.21.15.43:0/3623242749 learned_addr learned my addr 172.21.15.43:0/3623242749 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.805+0000 7fb281e00640 1 --2- 172.21.15.43:0/3623242749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccf5e86570 0x55ccf5e910b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.805+0000 7fb281400640 1 -- 172.21.15.43:0/3623242749 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccf5e94070 msgr2=0x55ccf5e91b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.805+0000 7fb281400640 1 --2- 172.21.15.43:0/3623242749 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccf5e94070 0x55ccf5e91b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.805+0000 7fb281400640 1 -- 172.21.15.43:0/3623242749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccf5e86570 msgr2=0x55ccf5e910b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.805+0000 7fb281400640 1 --2- 172.21.15.43:0/3623242749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccf5e86570 0x55ccf5e910b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.805+0000 7fb281400640 1 -- 172.21.15.43:0/3623242749 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ccf5bca620 con 0x55ccf5e86e90 2024-03-28T22:37:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.805+0000 7fb281e00640 1 --2- 172.21.15.43:0/3623242749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccf5e86570 0x55ccf5e910b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:37:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.805+0000 7fb282800640 1 --2- 172.21.15.43:0/3623242749 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccf5e94070 0x55ccf5e91b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:37:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.806+0000 7fb281400640 1 --2- 172.21.15.43:0/3623242749 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccf5e86e90 0x55ccf5e915f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb26c000c00 tx=0x7fb26c009fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ee84d2d2ea6b423 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.806+0000 7fb25f400640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:37:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.806+0000 7fb25f400640 1 -- 172.21.15.43:0/3623242749 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb26c029070 con 0x55ccf5e86e90 2024-03-28T22:37:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.806+0000 7fb2875cb740 1 -- 172.21.15.43:0/3623242749 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb26c005500 con 0x55ccf5e86e90 2024-03-28T22:37:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.806+0000 7fb25f400640 1 -- 172.21.15.43:0/3623242749 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fb26c004d50 con 0x55ccf5e86e90 2024-03-28T22:37:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.806+0000 7fb25f400640 1 -- 172.21.15.43:0/3623242749 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb26c035050 con 0x55ccf5e86e90 2024-03-28T22:37:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.806+0000 7fb2875cb740 10 client.25934 fetch_fsmap learned FSMap version 29 2024-03-28T22:37:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.806+0000 7fb2875cb740 1 -- 172.21.15.43:0/3623242749 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ccf5ea2b90 con 0x55ccf5e86e90 2024-03-28T22:37:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.806+0000 7fb25f400640 1 -- 172.21.15.43:0/3623242749 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb26c005500 con 0x55ccf5e86e90 2024-03-28T22:37:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.807+0000 7fb25f400640 1 -- 172.21.15.43:0/3623242749 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb26c003c60 con 0x55ccf5e86e90 2024-03-28T22:37:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.807+0000 7fb2875cb740 10 client.25934 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:37:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.807+0000 7fb2875cb740 10 client.25934 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:37:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.807+0000 7fb2875cb740 20 client.25934 populate_metadata read hostname 'smithi043' 2024-03-28T22:37:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.807+0000 7fb2875cb740 1 --2- 172.21.15.43:0/3623242749 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55ccf5ea3fa0 0x55ccf5ea6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.807+0000 7fb2875cb740 4 client.25934 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:37:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.807+0000 7fb2875cb740 1 -- 172.21.15.43:0/3623242749 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ccf5dfd030 con 0x55ccf5ea3fa0 2024-03-28T22:37:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.807+0000 7fb281e00640 1 --2- 172.21.15.43:0/3623242749 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55ccf5ea3fa0 0x55ccf5ea6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.808+0000 7fb281e00640 1 --2- 172.21.15.43:0/3623242749 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55ccf5ea3fa0 0x55ccf5ea6390 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73aadefc5f816961 server_cookie=f6c21c50289d3005 in_seq=0 out_seq=0 2024-03-28T22:37:53.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.808+0000 7fb25f400640 10 client.25934 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.808+0000 7fb25f400640 1 -- 172.21.15.43:0/3623242749 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ccf5dfd030 con 0x55ccf5ea3fa0 2024-03-28T22:37:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.808+0000 7fb25f400640 10 client.25934 handle_command_reply: tid=0 2024-03-28T22:37:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.808+0000 7fb2875cb740 1 client.25934 shutdown 2024-03-28T22:37:53.811 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.809+0000 7fb2875cb740 1 -- 172.21.15.43:0/3623242749 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccf5e86e90 msgr2=0x55ccf5e915f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.809+0000 7fb2875cb740 1 --2- 172.21.15.43:0/3623242749 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccf5e86e90 0x55ccf5e915f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb26c000c00 tx=0x7fb26c009fd0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.809+0000 7fb2875cb740 1 -- 172.21.15.43:0/3623242749 shutdown_connections 2024-03-28T22:37:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.809+0000 7fb2875cb740 1 --2- 172.21.15.43:0/3623242749 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55ccf5ea3fa0 0x55ccf5ea6390 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.809+0000 7fb2875cb740 1 --2- 172.21.15.43:0/3623242749 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccf5e86e90 0x55ccf5e915f0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.809+0000 7fb2875cb740 1 --2- 172.21.15.43:0/3623242749 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccf5e94070 0x55ccf5e91b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.809+0000 7fb2875cb740 1 --2- 172.21.15.43:0/3623242749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccf5e86570 0x55ccf5e910b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.809+0000 7fb25f400640 0 client.25934 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.809+0000 7fb2875cb740 1 -- 172.21.15.43:0/3623242749 >> 172.21.15.43:0/3623242749 conn(0x55ccf5e80e90 msgr2=0x55ccf5e82be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.809+0000 7fb2875cb740 1 -- 172.21.15.43:0/3623242749 shutdown_connections 2024-03-28T22:37:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.809+0000 7fb2875cb740 1 -- 172.21.15.43:0/3623242749 wait complete. 2024-03-28T22:37:53.812 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.809+0000 7fb2875cb740 20 client.25934 trim_cache size 0 max 16384 2024-03-28T22:37:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.829+0000 7fb2875cb740 1 Processor -- start 2024-03-28T22:37:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.829+0000 7fb2875cb740 1 -- start start 2024-03-28T22:37:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.830+0000 7fb2875cb740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccf5e86570 0x55ccf5e937e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.830+0000 7fb2875cb740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccf5e86e90 0x55ccf5e9fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.830+0000 7fb2875cb740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccf5e94070 0x55ccf5e7ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.830+0000 7fb2875cb740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c1ac800 con 0x55ccf5e86e90 2024-03-28T22:37:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.830+0000 7fb2875cb740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c1ae340 con 0x55ccf5e86570 2024-03-28T22:37:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.830+0000 7fb2875cb740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c1aced0 con 0x55ccf5e94070 2024-03-28T22:37:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.830+0000 7fb281e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccf5e86570 0x55ccf5e937e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.830+0000 7fb281e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccf5e86570 0x55ccf5e937e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:41850/0 (socket says 172.21.15.43:41850) 2024-03-28T22:37:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.830+0000 7fb281e00640 1 -- 172.21.15.43:0/987253315 learned_addr learned my addr 172.21.15.43:0/987253315 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.830+0000 7fb282800640 1 --2- 172.21.15.43:0/987253315 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccf5e94070 0x55ccf5e7ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.830+0000 7fb281400640 1 --2- 172.21.15.43:0/987253315 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccf5e86e90 0x55ccf5e9fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.830+0000 7fb281e00640 1 -- 172.21.15.43:0/987253315 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccf5e94070 msgr2=0x55ccf5e7ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.830+0000 7fb281e00640 1 --2- 172.21.15.43:0/987253315 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccf5e94070 0x55ccf5e7ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.830+0000 7fb281e00640 1 -- 172.21.15.43:0/987253315 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccf5e86e90 msgr2=0x55ccf5e9fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.830+0000 7fb281e00640 1 --2- 172.21.15.43:0/987253315 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccf5e86e90 0x55ccf5e9fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.830+0000 7fb281e00640 1 -- 172.21.15.43:0/987253315 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ccf5e86cc0 con 0x55ccf5e86570 2024-03-28T22:37:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.831+0000 7fb281e00640 1 --2- 172.21.15.43:0/987253315 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccf5e86570 0x55ccf5e937e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fb2700096c0 tx=0x7fb270004ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16b0efb599cf83e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.831+0000 7fb26a600640 1 -- 172.21.15.43:0/987253315 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb270005690 con 0x55ccf5e86570 2024-03-28T22:37:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.831+0000 7fb26a600640 1 -- 172.21.15.43:0/987253315 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2700d4070 con 0x55ccf5e86570 2024-03-28T22:37:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.831+0000 7fb26a600640 1 -- 172.21.15.43:0/987253315 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb270003d20 con 0x55ccf5e86570 2024-03-28T22:37:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.832+0000 7fb2875cb740 1 -- 172.21.15.43:0/987253315 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccf5e86570 msgr2=0x55ccf5e937e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.832+0000 7fb2875cb740 1 --2- 172.21.15.43:0/987253315 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccf5e86570 0x55ccf5e937e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fb2700096c0 tx=0x7fb270004ee0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.832+0000 7fb2875cb740 1 -- 172.21.15.43:0/987253315 shutdown_connections 2024-03-28T22:37:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.832+0000 7fb2875cb740 1 --2- 172.21.15.43:0/987253315 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccf5e94070 0x55ccf5e7ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.832+0000 7fb2875cb740 1 --2- 172.21.15.43:0/987253315 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccf5e86e90 0x55ccf5e9fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.832+0000 7fb2875cb740 1 --2- 172.21.15.43:0/987253315 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccf5e86570 0x55ccf5e937e0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.832+0000 7fb2875cb740 1 -- 172.21.15.43:0/987253315 >> 172.21.15.43:0/987253315 conn(0x55ccf5e80e90 msgr2=0x55ccf5e81270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:53.833 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.832+0000 7fb2875cb740 1 -- 172.21.15.43:0/987253315 shutdown_connections 2024-03-28T22:37:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.832+0000 7fb2875cb740 1 -- 172.21.15.43:0/987253315 wait complete. 2024-03-28T22:37:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.832+0000 7fb2875cb740 1 -- start start 2024-03-28T22:37:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.832+0000 7fb2875cb740 1 Processor -- start 2024-03-28T22:37:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.833+0000 7fb2875cb740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccf5e86570 0x55ccf5e9c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.833+0000 7fb2875cb740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccf5e86e90 0x55ccf5e8d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.833+0000 7fb2875cb740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccf5e94070 0x55ccf5e8dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.833+0000 7fb2875cb740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c1adc70 con 0x55ccf5e94070 2024-03-28T22:37:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.833+0000 7fb2875cb740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c1ad5a0 con 0x55ccf5e86570 2024-03-28T22:37:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.833+0000 7fb2875cb740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c1aa7e0 con 0x55ccf5e86e90 2024-03-28T22:37:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.833+0000 7fb2875cb740 10 client.0 _ll_register_callbacks cb 0x7fb2700ed9f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:37:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.833+0000 7fb281e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccf5e86570 0x55ccf5e9c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.833+0000 7fb281e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccf5e86570 0x55ccf5e9c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:41858/0 (socket says 172.21.15.43:41858) 2024-03-28T22:37:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.833+0000 7fb281e00640 1 -- 172.21.15.43:0/3559409032 learned_addr learned my addr 172.21.15.43:0/3559409032 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:37:53.834 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.833+0000 7fb281400640 1 --2- 172.21.15.43:0/3559409032 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccf5e86e90 0x55ccf5e8d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.834+0000 7fb281400640 1 -- 172.21.15.43:0/3559409032 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccf5e86570 msgr2=0x55ccf5e9c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.834+0000 7fb281400640 1 --2- 172.21.15.43:0/3559409032 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccf5e86570 0x55ccf5e9c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.834+0000 7fb281400640 1 -- 172.21.15.43:0/3559409032 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccf5e94070 msgr2=0x55ccf5e8dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:37:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.834+0000 7fb281400640 1 --2- 172.21.15.43:0/3559409032 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccf5e94070 0x55ccf5e8dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.834+0000 7fb281400640 1 -- 172.21.15.43:0/3559409032 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ccf5e8e1f0 con 0x55ccf5e86e90 2024-03-28T22:37:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.834+0000 7fb281e00640 1 --2- 172.21.15.43:0/3559409032 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccf5e86570 0x55ccf5e9c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:37:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.834+0000 7fb281400640 1 --2- 172.21.15.43:0/3559409032 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccf5e86e90 0x55ccf5e8d770 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb26c011850 tx=0x7fb26c005b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73a512e9e666f482 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:37:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.834+0000 7fb25f400640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:37:53.835 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.835+0000 7fb25f400640 1 -- 172.21.15.43:0/3559409032 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb26c029040 con 0x55ccf5e86e90 2024-03-28T22:37:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.835+0000 7fb25f400640 1 -- 172.21.15.43:0/3559409032 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fb26c023c20 con 0x55ccf5e86e90 2024-03-28T22:37:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.835+0000 7fb2875cb740 1 -- 172.21.15.43:0/3559409032 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2700d4070 con 0x55ccf5e86e90 2024-03-28T22:37:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.835+0000 7fb25f400640 1 -- 172.21.15.43:0/3559409032 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb26c023e20 con 0x55ccf5e86e90 2024-03-28T22:37:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.835+0000 7fb2875cb740 10 client.25894 fetch_fsmap learned FSMap version 29 2024-03-28T22:37:53.836 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.835+0000 7fb2875cb740 1 -- 172.21.15.43:0/3559409032 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ccf5f09d30 con 0x55ccf5e86e90 2024-03-28T22:37:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.836+0000 7fb25f400640 1 -- 172.21.15.43:0/3559409032 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2700d4070 con 0x55ccf5e86e90 2024-03-28T22:37:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.836+0000 7fb25f400640 1 -- 172.21.15.43:0/3559409032 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb26c02e440 con 0x55ccf5e86e90 2024-03-28T22:37:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.836+0000 7fb2875cb740 10 client.25894 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:37:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.836+0000 7fb2875cb740 10 client.25894 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:37:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.836+0000 7fb2875cb740 20 client.25894 populate_metadata read hostname 'smithi043' 2024-03-28T22:37:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.836+0000 7fb2875cb740 1 --2- 172.21.15.43:0/3559409032 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55ccf5f0a420 0x55ccf5f0c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:37:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.836+0000 7fb2875cb740 4 client.25894 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T22:37:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.836+0000 7fb2875cb740 1 -- 172.21.15.43:0/3559409032 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ccf5df6b80 con 0x55ccf5f0a420 2024-03-28T22:37:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.836+0000 7fb281e00640 1 --2- 172.21.15.43:0/3559409032 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55ccf5f0a420 0x55ccf5f0c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:37:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.837+0000 7fb25f400640 10 client.25894 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.837+0000 7fb281e00640 1 --2- 172.21.15.43:0/3559409032 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55ccf5f0a420 0x55ccf5f0c840 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6c4840924623263 server_cookie=18c0d4ccd4a2c54d in_seq=0 out_seq=0 2024-03-28T22:37:53.842 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.844+0000 7fb25f400640 1 -- 172.21.15.43:0/3559409032 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ccf5df6b80 con 0x55ccf5f0a420 2024-03-28T22:37:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.844+0000 7fb25f400640 10 client.25894 handle_command_reply: tid=0 2024-03-28T22:37:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.844+0000 7fb2875cb740 1 client.25894 shutdown 2024-03-28T22:37:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.844+0000 7fb2875cb740 1 -- 172.21.15.43:0/3559409032 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccf5e86e90 msgr2=0x55ccf5e8d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.844+0000 7fb2875cb740 1 --2- 172.21.15.43:0/3559409032 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccf5e86e90 0x55ccf5e8d770 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb26c011850 tx=0x7fb26c005b90 comp rx=0 tx=0).stop 2024-03-28T22:37:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.845+0000 7fb2875cb740 1 -- 172.21.15.43:0/3559409032 shutdown_connections 2024-03-28T22:37:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.845+0000 7fb2875cb740 1 --2- 172.21.15.43:0/3559409032 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccf5e94070 0x55ccf5e8dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.845+0000 7fb2875cb740 1 --2- 172.21.15.43:0/3559409032 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccf5e86e90 0x55ccf5e8d770 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.845+0000 7fb2875cb740 1 --2- 172.21.15.43:0/3559409032 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55ccf5f0a420 0x55ccf5f0c840 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.845+0000 7fb2875cb740 1 --2- 172.21.15.43:0/3559409032 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccf5e86570 0x55ccf5e9c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.845+0000 7fb25f400640 0 client.25894 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:37:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.845+0000 7fb2875cb740 1 -- 172.21.15.43:0/3559409032 >> 172.21.15.43:0/3559409032 conn(0x55ccf5e80e90 msgr2=0x55ccf5e7ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.845+0000 7fb2875cb740 1 -- 172.21.15.43:0/3559409032 shutdown_connections 2024-03-28T22:37:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.845+0000 7fb2875cb740 1 -- 172.21.15.43:0/3559409032 wait complete. 2024-03-28T22:37:53.843 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.845+0000 7fb2875cb740 20 client.25894 trim_cache size 0 max 16384 2024-03-28T22:37:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:{ 2024-03-28T22:37:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "message": "", 2024-03-28T22:37:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "return_code": 0 2024-03-28T22:37:53.844 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:} 2024-03-28T22:37:53.845 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.846+0000 7fb283200640 1 -- 172.21.15.43:0/4041200900 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb2640a1ff0 msgr2=0x7fb2640a44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:53.845 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.846+0000 7fb283200640 1 --2- 172.21.15.43:0/4041200900 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb2640a1ff0 0x7fb2640a44b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb278005c40 tx=0x7fb278004ab0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.845 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.846+0000 7fb283200640 1 -- 172.21.15.43:0/4041200900 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb27c101570 msgr2=0x7fb27c197e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:37:53.845 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.846+0000 7fb283200640 1 --2- 172.21.15.43:0/4041200900 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb27c101570 0x7fb27c197e40 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb270009900 tx=0x7fb270004360 comp rx=0 tx=0).stop 2024-03-28T22:37:53.845 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.846+0000 7fb283200640 1 -- 172.21.15.43:0/4041200900 shutdown_connections 2024-03-28T22:37:53.845 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.846+0000 7fb283200640 1 --2- 172.21.15.43:0/4041200900 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb2640a1ff0 0x7fb2640a44b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.845 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.846+0000 7fb283200640 1 --2- 172.21.15.43:0/4041200900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb27c107630 0x7fb27c19c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.845 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.846+0000 7fb283200640 1 --2- 172.21.15.43:0/4041200900 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb27c101f30 0x7fb27c198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.845 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.846+0000 7fb283200640 1 --2- 172.21.15.43:0/4041200900 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb27c101570 0x7fb27c197e40 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:37:53.845 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.846+0000 7fb283200640 1 -- 172.21.15.43:0/4041200900 >> 172.21.15.43:0/4041200900 conn(0x7fb27c0fd060 msgr2=0x7fb27c0fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:37:53.845 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.846+0000 7fb283200640 1 -- 172.21.15.43:0/4041200900 shutdown_connections 2024-03-28T22:37:53.845 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:37:53.846+0000 7fb283200640 1 -- 172.21.15.43:0/4041200900 wait complete. 2024-03-28T22:37:55.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:54 smithi043 ceph-mon[31823]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-03-28T22:37:55.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:54 smithi172 ceph-mon[36566]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-03-28T22:37:55.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:54 smithi082 ceph-mon[36471]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-03-28T22:37:57.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:57 smithi043 ceph-mon[31823]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 938 B/s wr, 8 op/s 2024-03-28T22:37:57.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:57 smithi172 ceph-mon[36566]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 938 B/s wr, 8 op/s 2024-03-28T22:37:57.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:57 smithi082 ceph-mon[36471]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 938 B/s wr, 8 op/s 2024-03-28T22:37:59.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:37:59 smithi043 ceph-mon[31823]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-03-28T22:37:59.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:37:59 smithi172 ceph-mon[36566]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-03-28T22:37:59.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:37:59 smithi082 ceph-mon[36471]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-03-28T22:38:00.516 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:38:00 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:38:00.059+0000 7f3566800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:38:00.516 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:38:00 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:38:00.059+0000 7f3566800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:38:00.516 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:38:00 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:38:00.059+0000 7f3566800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:38:00.516 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:38:00 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:38:00.059+0000 7f3566800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:38:00.516 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:38:00 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:38:00.059+0000 7f3566800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T22:38:01.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:01 smithi043 ceph-mon[31823]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-28T22:38:01.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:01 smithi172 ceph-mon[36566]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-28T22:38:01.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:01 smithi082 ceph-mon[36471]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-28T22:38:03.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:03 smithi043 ceph-mon[31823]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-28T22:38:03.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:03 smithi043 ceph-mon[31823]: mgrmap e27: x(active, since 50s), standbys: y, z 2024-03-28T22:38:03.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:03 smithi172 ceph-mon[36566]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-28T22:38:03.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:03 smithi172 ceph-mon[36566]: mgrmap e27: x(active, since 50s), standbys: y, z 2024-03-28T22:38:03.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:03 smithi082 ceph-mon[36471]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-28T22:38:03.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:03 smithi082 ceph-mon[36471]: mgrmap e27: x(active, since 50s), standbys: y, z 2024-03-28T22:38:04.927 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:38:04.927 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:38:05.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.111+0000 7f53fe400640 1 -- 172.21.15.43:0/2068310998 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53f8101f80 msgr2=0x7f53f810e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:05.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.111+0000 7f53fe400640 1 --2- 172.21.15.43:0/2068310998 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53f8101f80 0x7f53f810e490 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f53f0009c10 tx=0x7f53f002bee0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.111+0000 7f53fe400640 1 -- 172.21.15.43:0/2068310998 shutdown_connections 2024-03-28T22:38:05.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.111+0000 7f53fe400640 1 --2- 172.21.15.43:0/2068310998 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53f8101f80 0x7f53f810e490 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.111+0000 7f53fe400640 1 --2- 172.21.15.43:0/2068310998 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53f81015c0 0x7f53f8101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.111+0000 7f53fe400640 1 --2- 172.21.15.43:0/2068310998 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f53f8107660 0x7f53f8107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.111+0000 7f53fe400640 1 -- 172.21.15.43:0/2068310998 >> 172.21.15.43:0/2068310998 conn(0x7f53f80fd0f0 msgr2=0x7f53f80ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:05.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.111+0000 7f53fe400640 1 -- 172.21.15.43:0/2068310998 shutdown_connections 2024-03-28T22:38:05.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.111+0000 7f53fe400640 1 -- 172.21.15.43:0/2068310998 wait complete. 2024-03-28T22:38:05.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.112+0000 7f53fe400640 1 Processor -- start 2024-03-28T22:38:05.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.113+0000 7f53fe400640 1 -- start start 2024-03-28T22:38:05.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.113+0000 7f53fe400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53f81015c0 0x7f53f8197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.113+0000 7f53fe400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f53f8101f80 0x7f53f8198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.113+0000 7f53fe400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53f8107660 0x7f53f819c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.113+0000 7f53fe400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f53f81119f0 con 0x7f53f8107660 2024-03-28T22:38:05.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.113+0000 7f53fe400640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f53f8111870 con 0x7f53f8101f80 2024-03-28T22:38:05.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.113+0000 7f53fe400640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f53f8111b70 con 0x7f53f81015c0 2024-03-28T22:38:05.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.114+0000 7f53fd000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53f81015c0 0x7f53f8197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.114+0000 7f53fd000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53f81015c0 0x7f53f8197e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57168/0 (socket says 172.21.15.43:57168) 2024-03-28T22:38:05.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.114+0000 7f53fd000640 1 -- 172.21.15.43:0/1523083228 learned_addr learned my addr 172.21.15.43:0/1523083228 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:05.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.114+0000 7f53f7e00640 1 --2- 172.21.15.43:0/1523083228 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f53f8101f80 0x7f53f8198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.114+0000 7f53fda00640 1 --2- 172.21.15.43:0/1523083228 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53f8107660 0x7f53f819c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.114+0000 7f53fd000640 1 -- 172.21.15.43:0/1523083228 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f53f8101f80 msgr2=0x7f53f8198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:05.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.114+0000 7f53fd000640 1 --2- 172.21.15.43:0/1523083228 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f53f8101f80 0x7f53f8198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.114+0000 7f53fd000640 1 -- 172.21.15.43:0/1523083228 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53f8107660 msgr2=0x7f53f819c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:05.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.114+0000 7f53fd000640 1 --2- 172.21.15.43:0/1523083228 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53f8107660 0x7f53f819c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.114+0000 7f53fd000640 1 -- 172.21.15.43:0/1523083228 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53f819ce00 con 0x7f53f81015c0 2024-03-28T22:38:05.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.114+0000 7f53fd000640 1 --2- 172.21.15.43:0/1523083228 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53f81015c0 0x7f53f8197e10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f53ec00e9f0 tx=0x7f53ec00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:05.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.115+0000 7f53f5600640 1 -- 172.21.15.43:0/1523083228 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ec00ccc0 con 0x7f53f81015c0 2024-03-28T22:38:05.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.115+0000 7f53f5600640 1 -- 172.21.15.43:0/1523083228 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53ec0035f0 con 0x7f53f81015c0 2024-03-28T22:38:05.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.115+0000 7f53f5600640 1 -- 172.21.15.43:0/1523083228 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ec00ce60 con 0x7f53f81015c0 2024-03-28T22:38:05.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.115+0000 7f53fe400640 1 -- 172.21.15.43:0/1523083228 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53f819d0f0 con 0x7f53f81015c0 2024-03-28T22:38:05.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.115+0000 7f53fe400640 1 -- 172.21.15.43:0/1523083228 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53f8073980 con 0x7f53f81015c0 2024-03-28T22:38:05.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.118+0000 7f54027bc740 1 Processor -- start 2024-03-28T22:38:05.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.118+0000 7f54027bc740 1 -- start start 2024-03-28T22:38:05.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.119+0000 7f54027bc740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556c54b49db0 0x556c54b48b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.119+0000 7f54027bc740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556c54b490d0 0x556c54b49550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.119+0000 7f54027bc740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c54b52440 0x556c54b54830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.119+0000 7f54027bc740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x556c54aad100 con 0x556c54b52440 2024-03-28T22:38:05.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.119+0000 7f54027bc740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x556c54a55ba0 con 0x556c54b490d0 2024-03-28T22:38:05.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.119+0000 7f54027bc740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x556c54aa91a0 con 0x556c54b49db0 2024-03-28T22:38:05.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.119+0000 7f53fda00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556c54b49db0 0x556c54b48b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.119+0000 7f53fda00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556c54b49db0 0x556c54b48b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57170/0 (socket says 172.21.15.43:57170) 2024-03-28T22:38:05.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.119+0000 7f53fda00640 1 -- 172.21.15.43:0/1305108651 learned_addr learned my addr 172.21.15.43:0/1305108651 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:05.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.119+0000 7f53f7e00640 1 --2- 172.21.15.43:0/1305108651 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c54b52440 0x556c54b54830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.119+0000 7f53fda00640 1 -- 172.21.15.43:0/1305108651 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556c54b490d0 msgr2=0x556c54b49550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:38:05.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.119+0000 7f53fda00640 1 --2- 172.21.15.43:0/1305108651 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556c54b490d0 0x556c54b49550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.119+0000 7f53fda00640 1 -- 172.21.15.43:0/1305108651 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c54b52440 msgr2=0x556c54b54830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:05.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.119+0000 7f53fda00640 1 --2- 172.21.15.43:0/1305108651 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c54b52440 0x556c54b54830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.119+0000 7f53f5600640 1 -- 172.21.15.43:0/1523083228 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 27) ==== 136989+0+0 (secure 0 0 0) 0x7f53ec0037a0 con 0x7f53f81015c0 2024-03-28T22:38:05.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.120+0000 7f53fda00640 1 -- 172.21.15.43:0/1305108651 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c54a861a0 con 0x556c54b49db0 2024-03-28T22:38:05.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.120+0000 7f53f7e00640 1 --2- 172.21.15.43:0/1305108651 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c54b52440 0x556c54b54830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:38:05.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.120+0000 7f53fda00640 1 --2- 172.21.15.43:0/1305108651 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556c54b49db0 0x556c54b48b00 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f53f0004af0 tx=0x7f53f0009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84d2c44b3f617277 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:05.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.120+0000 7f53ea000640 1 -- 172.21.15.43:0/1305108651 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f0043070 con 0x556c54b49db0 2024-03-28T22:38:05.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.120+0000 7f53ea000640 1 -- 172.21.15.43:0/1305108651 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53f0033cc0 con 0x556c54b49db0 2024-03-28T22:38:05.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.120+0000 7f53ea000640 1 -- 172.21.15.43:0/1305108651 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f002cae0 con 0x556c54b49db0 2024-03-28T22:38:05.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.121+0000 7f54027bc740 1 -- 172.21.15.43:0/1305108651 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556c54b49db0 msgr2=0x556c54b48b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:05.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.121+0000 7f54027bc740 1 --2- 172.21.15.43:0/1305108651 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556c54b49db0 0x556c54b48b00 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f53f0004af0 tx=0x7f53f0009840 comp rx=0 tx=0).stop 2024-03-28T22:38:05.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.121+0000 7f54027bc740 1 -- 172.21.15.43:0/1305108651 shutdown_connections 2024-03-28T22:38:05.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.121+0000 7f54027bc740 1 --2- 172.21.15.43:0/1305108651 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556c54b490d0 0x556c54b49550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.121+0000 7f54027bc740 1 --2- 172.21.15.43:0/1305108651 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c54b52440 0x556c54b54830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.121+0000 7f54027bc740 1 --2- 172.21.15.43:0/1305108651 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556c54b49db0 0x556c54b48b00 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.121+0000 7f54027bc740 1 -- 172.21.15.43:0/1305108651 >> 172.21.15.43:0/1305108651 conn(0x556c54b0fa10 msgr2=0x556c54b43b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:05.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.121+0000 7f53f5600640 1 --2- 172.21.15.43:0/1523083228 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f53e00a2090 0x7f53e00a4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.121+0000 7f54027bc740 1 -- 172.21.15.43:0/1305108651 shutdown_connections 2024-03-28T22:38:05.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.121+0000 7f53f5600640 1 -- 172.21.15.43:0/1523083228 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f53ec0ceac0 con 0x7f53f81015c0 2024-03-28T22:38:05.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.122+0000 7f54027bc740 1 -- 172.21.15.43:0/1305108651 wait complete. 2024-03-28T22:38:05.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.122+0000 7f54027bc740 1 -- start start 2024-03-28T22:38:05.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.122+0000 7f54027bc740 1 Processor -- start 2024-03-28T22:38:05.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.122+0000 7f53fda00640 1 --2- 172.21.15.43:0/1523083228 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f53e00a2090 0x7f53e00a4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.123+0000 7f54027bc740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c54b490d0 0x556c54b4f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.123+0000 7f54027bc740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556c54b49db0 0x556c54b4f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.123+0000 7f54027bc740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556c54b52440 0x556c54b4fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.123+0000 7f54027bc740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x556c54b55250 con 0x556c54b490d0 2024-03-28T22:38:05.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.123+0000 7f54027bc740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x556c54a4b250 con 0x556c54b49db0 2024-03-28T22:38:05.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.123+0000 7f54027bc740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x556c54b553d0 con 0x556c54b52440 2024-03-28T22:38:05.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.123+0000 7f53fda00640 1 --2- 172.21.15.43:0/1523083228 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f53e00a2090 0x7f53e00a4550 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f53f0033ff0 tx=0x7f53f00333e0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:05.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.123+0000 7f53f7e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556c54b49db0 0x556c54b4f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.123+0000 7f54027bc740 10 client.0 _ll_register_callbacks cb 0x556c547c0be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:05.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.123+0000 7f53fda00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556c54b52440 0x556c54b4fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.123+0000 7f53f7e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556c54b49db0 0x556c54b4f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33984/0 (socket says 172.21.15.43:33984) 2024-03-28T22:38:05.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.123+0000 7f53f7e00640 1 -- 172.21.15.43:0/964527088 learned_addr learned my addr 172.21.15.43:0/964527088 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:05.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.123+0000 7f53f7e00640 1 -- 172.21.15.43:0/964527088 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556c54b52440 msgr2=0x556c54b4fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:05.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.124+0000 7f53f7e00640 1 --2- 172.21.15.43:0/964527088 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556c54b52440 0x556c54b4fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.124+0000 7f53f7e00640 1 -- 172.21.15.43:0/964527088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c54b490d0 msgr2=0x556c54b4f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:05.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.124+0000 7f53f7e00640 1 --2- 172.21.15.43:0/964527088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c54b490d0 0x556c54b4f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.124+0000 7f53f7e00640 1 -- 172.21.15.43:0/964527088 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c54a67960 con 0x556c54b49db0 2024-03-28T22:38:05.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.124+0000 7f53fda00640 1 --2- 172.21.15.43:0/964527088 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556c54b52440 0x556c54b4fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:05.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.124+0000 7f53f7e00640 1 --2- 172.21.15.43:0/964527088 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556c54b49db0 0x556c54b4f990 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f53e4013710 tx=0x7f53e4013bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43f48a7d478f4b6f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:05.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.124+0000 7f53dea00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:38:05.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.124+0000 7f53dea00640 1 -- 172.21.15.43:0/964527088 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e4020060 con 0x556c54b49db0 2024-03-28T22:38:05.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.124+0000 7f54027bc740 1 -- 172.21.15.43:0/964527088 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53f0033cc0 con 0x556c54b49db0 2024-03-28T22:38:05.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.124+0000 7f53dea00640 1 -- 172.21.15.43:0/964527088 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f53e4011dd0 con 0x556c54b49db0 2024-03-28T22:38:05.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.125+0000 7f53dea00640 1 -- 172.21.15.43:0/964527088 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f53e401b430 con 0x556c54b49db0 2024-03-28T22:38:05.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.125+0000 7f54027bc740 10 client.25955 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:05.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.125+0000 7f54027bc740 1 -- 172.21.15.43:0/964527088 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556c5488b7d0 con 0x556c54b49db0 2024-03-28T22:38:05.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.125+0000 7f53dea00640 1 -- 172.21.15.43:0/964527088 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53f0033cc0 con 0x556c54b49db0 2024-03-28T22:38:05.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.126+0000 7f53dea00640 1 -- 172.21.15.43:0/964527088 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f53e400f3c0 con 0x556c54b49db0 2024-03-28T22:38:05.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.126+0000 7f54027bc740 10 client.25955 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:05.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.126+0000 7f54027bc740 10 client.25955 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:38:05.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.126+0000 7f54027bc740 20 client.25955 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:05.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.126+0000 7f54027bc740 1 --2- 172.21.15.43:0/964527088 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x556c54b60780 0x556c54b62c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.126+0000 7f54027bc740 4 client.25955 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:38:05.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.126+0000 7f54027bc740 1 -- 172.21.15.43:0/964527088 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556c54aba640 con 0x556c54b60780 2024-03-28T22:38:05.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.126+0000 7f53fd000640 1 --2- 172.21.15.43:0/964527088 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x556c54b60780 0x556c54b62c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.127+0000 7f53fd000640 1 --2- 172.21.15.43:0/964527088 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x556c54b60780 0x556c54b62c40 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc41041fefeb03aa server_cookie=e6ee36b126dc212d in_seq=0 out_seq=0 2024-03-28T22:38:05.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.127+0000 7f53dea00640 10 client.25955 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:05.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.128+0000 7f53dea00640 1 -- 172.21.15.43:0/964527088 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556c54aba640 con 0x556c54b60780 2024-03-28T22:38:05.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.128+0000 7f53dea00640 10 client.25955 handle_command_reply: tid=0 2024-03-28T22:38:05.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.128+0000 7f54027bc740 1 client.25955 shutdown 2024-03-28T22:38:05.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.128+0000 7f54027bc740 1 -- 172.21.15.43:0/964527088 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556c54b49db0 msgr2=0x556c54b4f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:05.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.128+0000 7f54027bc740 1 --2- 172.21.15.43:0/964527088 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556c54b49db0 0x556c54b4f990 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f53e4013710 tx=0x7f53e4013bd0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.128+0000 7f54027bc740 1 -- 172.21.15.43:0/964527088 shutdown_connections 2024-03-28T22:38:05.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.128+0000 7f54027bc740 1 --2- 172.21.15.43:0/964527088 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x556c54b60780 0x556c54b62c40 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.128+0000 7f54027bc740 1 --2- 172.21.15.43:0/964527088 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556c54b49db0 0x556c54b4f990 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.128+0000 7f54027bc740 1 --2- 172.21.15.43:0/964527088 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556c54b52440 0x556c54b4fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.128+0000 7f54027bc740 1 --2- 172.21.15.43:0/964527088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c54b490d0 0x556c54b4f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.128+0000 7f54027bc740 1 -- 172.21.15.43:0/964527088 >> 172.21.15.43:0/964527088 conn(0x556c54b0fa10 msgr2=0x556c54b45870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:05.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.128+0000 7f53dea00640 0 client.25955 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:05.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.128+0000 7f54027bc740 1 -- 172.21.15.43:0/964527088 shutdown_connections 2024-03-28T22:38:05.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.128+0000 7f54027bc740 1 -- 172.21.15.43:0/964527088 wait complete. 2024-03-28T22:38:05.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.129+0000 7f54027bc740 20 client.25955 trim_cache size 0 max 16384 2024-03-28T22:38:05.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.147+0000 7f54027bc740 1 Processor -- start 2024-03-28T22:38:05.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.147+0000 7f54027bc740 1 -- start start 2024-03-28T22:38:05.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.147+0000 7f54027bc740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556c54b490d0 0x556c54b51b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.147+0000 7f54027bc740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c54b49db0 0x556c54b68450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.148+0000 7f54027bc740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556c54b52440 0x556c54b53830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.148+0000 7f54027bc740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f53f81ac840 con 0x556c54b49db0 2024-03-28T22:38:05.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.148+0000 7f54027bc740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f53f81ae380 con 0x556c54b490d0 2024-03-28T22:38:05.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.148+0000 7f54027bc740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f53f81acf10 con 0x556c54b52440 2024-03-28T22:38:05.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.148+0000 7f53fd000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556c54b490d0 0x556c54b51b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.148+0000 7f53fd000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556c54b490d0 0x556c54b51b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33996/0 (socket says 172.21.15.43:33996) 2024-03-28T22:38:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.148+0000 7f53fd000640 1 -- 172.21.15.43:0/2343883212 learned_addr learned my addr 172.21.15.43:0/2343883212 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.148+0000 7f53fda00640 1 --2- 172.21.15.43:0/2343883212 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556c54b52440 0x556c54b53830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.148+0000 7f53f7e00640 1 --2- 172.21.15.43:0/2343883212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c54b49db0 0x556c54b68450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.148+0000 7f53fd000640 1 -- 172.21.15.43:0/2343883212 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556c54b52440 msgr2=0x556c54b53830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.148+0000 7f53fd000640 1 --2- 172.21.15.43:0/2343883212 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556c54b52440 0x556c54b53830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.148+0000 7f53fd000640 1 -- 172.21.15.43:0/2343883212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c54b49db0 msgr2=0x556c54b68450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.148+0000 7f53fd000640 1 --2- 172.21.15.43:0/2343883212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c54b49db0 0x556c54b68450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.148+0000 7f53fd000640 1 -- 172.21.15.43:0/2343883212 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c547c13f0 con 0x556c54b490d0 2024-03-28T22:38:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.148+0000 7f53f7e00640 1 --2- 172.21.15.43:0/2343883212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c54b49db0 0x556c54b68450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:38:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.149+0000 7f53fd000640 1 --2- 172.21.15.43:0/2343883212 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556c54b490d0 0x556c54b51b80 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f53ec09ae50 tx=0x7f53ec09b5c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a4635fe65ef3072 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.149+0000 7f53e9600640 1 -- 172.21.15.43:0/2343883212 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ec0b8070 con 0x556c54b490d0 2024-03-28T22:38:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.149+0000 7f53e9600640 1 -- 172.21.15.43:0/2343883212 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53ec09f010 con 0x556c54b490d0 2024-03-28T22:38:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.149+0000 7f53e9600640 1 -- 172.21.15.43:0/2343883212 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ec0b2030 con 0x556c54b490d0 2024-03-28T22:38:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.149+0000 7f54027bc740 1 -- 172.21.15.43:0/2343883212 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556c54b490d0 msgr2=0x556c54b51b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:05.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.149+0000 7f54027bc740 1 --2- 172.21.15.43:0/2343883212 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556c54b490d0 0x556c54b51b80 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f53ec09ae50 tx=0x7f53ec09b5c0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.150+0000 7f54027bc740 1 -- 172.21.15.43:0/2343883212 shutdown_connections 2024-03-28T22:38:05.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.150+0000 7f54027bc740 1 --2- 172.21.15.43:0/2343883212 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556c54b52440 0x556c54b53830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.150+0000 7f54027bc740 1 --2- 172.21.15.43:0/2343883212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c54b49db0 0x556c54b68450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.150+0000 7f54027bc740 1 --2- 172.21.15.43:0/2343883212 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556c54b490d0 0x556c54b51b80 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.150+0000 7f54027bc740 1 -- 172.21.15.43:0/2343883212 >> 172.21.15.43:0/2343883212 conn(0x556c54b0fa10 msgr2=0x556c54b43650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:05.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.150+0000 7f54027bc740 1 -- 172.21.15.43:0/2343883212 shutdown_connections 2024-03-28T22:38:05.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.150+0000 7f54027bc740 1 -- 172.21.15.43:0/2343883212 wait complete. 2024-03-28T22:38:05.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.151+0000 7f54027bc740 1 -- start start 2024-03-28T22:38:05.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.151+0000 7f54027bc740 1 Processor -- start 2024-03-28T22:38:05.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.151+0000 7f54027bc740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c54b490d0 0x556c54b55620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.151+0000 7f54027bc740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556c54b49db0 0x556c54b55b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.151+0000 7f54027bc740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556c54b52440 0x556c54b560a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.151+0000 7f54027bc740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f53f81adcb0 con 0x556c54b490d0 2024-03-28T22:38:05.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.151+0000 7f54027bc740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f53f81ad5e0 con 0x556c54b52440 2024-03-28T22:38:05.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.151+0000 7f54027bc740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f53f81aa820 con 0x556c54b49db0 2024-03-28T22:38:05.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.151+0000 7f54027bc740 10 client.0 _ll_register_callbacks cb 0x7f53e4009490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:05.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.152+0000 7f53fda00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556c54b52440 0x556c54b560a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.152+0000 7f53f7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556c54b49db0 0x556c54b55b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.152+0000 7f53fd000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c54b490d0 0x556c54b55620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.152+0000 7f53fda00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556c54b52440 0x556c54b560a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34010/0 (socket says 172.21.15.43:34010) 2024-03-28T22:38:05.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.152+0000 7f53fda00640 1 -- 172.21.15.43:0/162564617 learned_addr learned my addr 172.21.15.43:0/162564617 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:05.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.152+0000 7f53f7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556c54b49db0 0x556c54b55b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57202/0 (socket says 172.21.15.43:57202) 2024-03-28T22:38:05.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.152+0000 7f53fd000640 1 -- 172.21.15.43:0/162564617 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556c54b49db0 msgr2=0x556c54b55b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:05.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.152+0000 7f53fd000640 1 --2- 172.21.15.43:0/162564617 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556c54b49db0 0x556c54b55b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.152+0000 7f53fd000640 1 -- 172.21.15.43:0/162564617 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556c54b52440 msgr2=0x556c54b560a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:05.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.152+0000 7f53fd000640 1 --2- 172.21.15.43:0/162564617 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556c54b52440 0x556c54b560a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.152+0000 7f53fd000640 1 -- 172.21.15.43:0/162564617 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c547810f0 con 0x556c54b490d0 2024-03-28T22:38:05.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.152+0000 7f53fda00640 1 --2- 172.21.15.43:0/162564617 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556c54b52440 0x556c54b560a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:05.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.152+0000 7f53f7e00640 1 --2- 172.21.15.43:0/162564617 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556c54b49db0 0x556c54b55b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:38:05.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.152+0000 7f53fd000640 1 --2- 172.21.15.43:0/162564617 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c54b490d0 0x556c54b55620 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f53ec09ae50 tx=0x7f53ec0993d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b2aa8ffced14e86 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:05.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.152+0000 7f53dea00640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:38:05.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.153+0000 7f53dea00640 1 -- 172.21.15.43:0/162564617 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ec0b2020 con 0x556c54b490d0 2024-03-28T22:38:05.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.153+0000 7f53dea00640 1 -- 172.21.15.43:0/162564617 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f53ec0b9270 con 0x556c54b490d0 2024-03-28T22:38:05.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.153+0000 7f54027bc740 1 -- 172.21.15.43:0/162564617 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53ec09f010 con 0x556c54b490d0 2024-03-28T22:38:05.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.153+0000 7f53dea00640 1 -- 172.21.15.43:0/162564617 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f53ec0b8030 con 0x556c54b490d0 2024-03-28T22:38:05.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.153+0000 7f54027bc740 10 client.15822 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:05.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.153+0000 7f54027bc740 1 -- 172.21.15.43:0/162564617 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556c54beb1a0 con 0x556c54b490d0 2024-03-28T22:38:05.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.154+0000 7f53dea00640 1 -- 172.21.15.43:0/162564617 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53ec09f010 con 0x556c54b490d0 2024-03-28T22:38:05.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.154+0000 7f53dea00640 1 -- 172.21.15.43:0/162564617 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f53ec079ce0 con 0x556c54b490d0 2024-03-28T22:38:05.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.154+0000 7f54027bc740 10 client.15822 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:05.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.154+0000 7f54027bc740 10 client.15822 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:38:05.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.154+0000 7f54027bc740 20 client.15822 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:05.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.154+0000 7f54027bc740 1 --2- 172.21.15.43:0/162564617 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x556c54beb760 0x556c54bedb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.154+0000 7f54027bc740 4 client.15822 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:38:05.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.154+0000 7f54027bc740 1 -- 172.21.15.43:0/162564617 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556c54ab4990 con 0x556c54beb760 2024-03-28T22:38:05.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.154+0000 7f53f7e00640 1 --2- 172.21.15.43:0/162564617 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x556c54beb760 0x556c54bedb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.156+0000 7f53dea00640 10 client.15822 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:05.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.157+0000 7f53f7e00640 1 --2- 172.21.15.43:0/162564617 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x556c54beb760 0x556c54bedb80 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb78a4e2ed42a01b server_cookie=87a2882260a7167f in_seq=0 out_seq=0 2024-03-28T22:38:05.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.157+0000 7f53dea00640 1 -- 172.21.15.43:0/162564617 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556c54ab4990 con 0x556c54beb760 2024-03-28T22:38:05.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.157+0000 7f53dea00640 10 client.15822 handle_command_reply: tid=0 2024-03-28T22:38:05.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.157+0000 7f54027bc740 1 client.15822 shutdown 2024-03-28T22:38:05.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.157+0000 7f54027bc740 1 -- 172.21.15.43:0/162564617 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c54b490d0 msgr2=0x556c54b55620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:05.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.157+0000 7f54027bc740 1 --2- 172.21.15.43:0/162564617 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c54b490d0 0x556c54b55620 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f53ec09ae50 tx=0x7f53ec0993d0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.157+0000 7f54027bc740 1 -- 172.21.15.43:0/162564617 shutdown_connections 2024-03-28T22:38:05.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.157+0000 7f54027bc740 1 --2- 172.21.15.43:0/162564617 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x556c54beb760 0x556c54bedb80 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.157+0000 7f54027bc740 1 --2- 172.21.15.43:0/162564617 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556c54b52440 0x556c54b560a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.157+0000 7f54027bc740 1 --2- 172.21.15.43:0/162564617 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556c54b49db0 0x556c54b55b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.157+0000 7f54027bc740 1 --2- 172.21.15.43:0/162564617 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556c54b490d0 0x556c54b55620 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.157+0000 7f53dea00640 0 client.15822 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:05.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.157+0000 7f54027bc740 1 -- 172.21.15.43:0/162564617 >> 172.21.15.43:0/162564617 conn(0x556c54b0fa10 msgr2=0x556c54b4f410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:05.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.158+0000 7f54027bc740 1 -- 172.21.15.43:0/162564617 shutdown_connections 2024-03-28T22:38:05.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.158+0000 7f54027bc740 1 -- 172.21.15.43:0/162564617 wait complete. 2024-03-28T22:38:05.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.158+0000 7f54027bc740 20 client.15822 trim_cache size 0 max 16384 2024-03-28T22:38:05.158 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:38:05.158 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-03-28T22:38:05.158 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-03-28T22:38:05.158 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:38:05.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.158+0000 7f53fe400640 1 -- 172.21.15.43:0/1523083228 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f53e00a2090 msgr2=0x7f53e00a4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:05.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.158+0000 7f53fe400640 1 --2- 172.21.15.43:0/1523083228 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f53e00a2090 0x7f53e00a4550 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f53f0033ff0 tx=0x7f53f00333e0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.158+0000 7f53fe400640 1 -- 172.21.15.43:0/1523083228 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53f81015c0 msgr2=0x7f53f8197e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:05.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.158+0000 7f53fe400640 1 --2- 172.21.15.43:0/1523083228 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53f81015c0 0x7f53f8197e10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f53ec00e9f0 tx=0x7f53ec00eeb0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.159+0000 7f53fe400640 1 -- 172.21.15.43:0/1523083228 shutdown_connections 2024-03-28T22:38:05.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.159+0000 7f53fe400640 1 --2- 172.21.15.43:0/1523083228 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f53e00a2090 0x7f53e00a4550 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.159+0000 7f53fe400640 1 --2- 172.21.15.43:0/1523083228 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f53f8101f80 0x7f53f8198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.159+0000 7f53fe400640 1 --2- 172.21.15.43:0/1523083228 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53f8107660 0x7f53f819c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.159+0000 7f53fe400640 1 --2- 172.21.15.43:0/1523083228 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f53f81015c0 0x7f53f8197e10 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.159+0000 7f53fe400640 1 -- 172.21.15.43:0/1523083228 >> 172.21.15.43:0/1523083228 conn(0x7f53f80fd0f0 msgr2=0x7f53f80fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:05.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.159+0000 7f53fe400640 1 -- 172.21.15.43:0/1523083228 shutdown_connections 2024-03-28T22:38:05.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.159+0000 7f53fe400640 1 -- 172.21.15.43:0/1523083228 wait complete. 2024-03-28T22:38:05.174 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:38:05.174 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:38:05.233 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:05 smithi043 ceph-mon[31823]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 29 op/s 2024-03-28T22:38:05.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.352+0000 7faf8fe00640 1 -- 172.21.15.43:0/623333192 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faf90074410 msgr2=0x7faf90112010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:05.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.352+0000 7faf8fe00640 1 --2- 172.21.15.43:0/623333192 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faf90074410 0x7faf90112010 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7faf7c009900 tx=0x7faf7c02bcd0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.352+0000 7faf8fe00640 1 -- 172.21.15.43:0/623333192 shutdown_connections 2024-03-28T22:38:05.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.352+0000 7faf8fe00640 1 --2- 172.21.15.43:0/623333192 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faf90074410 0x7faf90112010 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.352+0000 7faf8fe00640 1 --2- 172.21.15.43:0/623333192 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf90112550 0x7faf90114940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.352+0000 7faf8fe00640 1 --2- 172.21.15.43:0/623333192 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf90073af0 0x7faf90073ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.352+0000 7faf8fe00640 1 -- 172.21.15.43:0/623333192 >> 172.21.15.43:0/623333192 conn(0x7faf900fd030 msgr2=0x7faf900ff450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:05.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.352+0000 7faf8fe00640 1 -- 172.21.15.43:0/623333192 shutdown_connections 2024-03-28T22:38:05.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.352+0000 7faf8fe00640 1 -- 172.21.15.43:0/623333192 wait complete. 2024-03-28T22:38:05.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.354+0000 7faf8fe00640 1 Processor -- start 2024-03-28T22:38:05.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.354+0000 7faf8fe00640 1 -- start start 2024-03-28T22:38:05.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.354+0000 7faf8fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf90073af0 0x7faf9019e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.354+0000 7faf8fe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf90074410 0x7faf9019e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.354+0000 7faf8fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faf90112550 0x7faf901a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.354+0000 7faf8fe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7faf90117fa0 con 0x7faf90073af0 2024-03-28T22:38:05.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.354+0000 7faf8fe00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7faf90117e20 con 0x7faf90112550 2024-03-28T22:38:05.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.354+0000 7faf8fe00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7faf90118120 con 0x7faf90074410 2024-03-28T22:38:05.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.355+0000 7faf8ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf90073af0 0x7faf9019e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.355+0000 7faf8e000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf90074410 0x7faf9019e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.355+0000 7faf8e000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf90074410 0x7faf9019e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57230/0 (socket says 172.21.15.43:57230) 2024-03-28T22:38:05.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.355+0000 7faf8e000640 1 -- 172.21.15.43:0/1857160756 learned_addr learned my addr 172.21.15.43:0/1857160756 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:05.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.355+0000 7faf8f400640 1 --2- 172.21.15.43:0/1857160756 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faf90112550 0x7faf901a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.355+0000 7faf8e000640 1 -- 172.21.15.43:0/1857160756 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faf90112550 msgr2=0x7faf901a2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:05.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.355+0000 7faf8e000640 1 --2- 172.21.15.43:0/1857160756 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faf90112550 0x7faf901a2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.355+0000 7faf8e000640 1 -- 172.21.15.43:0/1857160756 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf90073af0 msgr2=0x7faf9019e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:05.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.355+0000 7faf8e000640 1 --2- 172.21.15.43:0/1857160756 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf90073af0 0x7faf9019e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.355+0000 7faf8e000640 1 -- 172.21.15.43:0/1857160756 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf901a3440 con 0x7faf90074410 2024-03-28T22:38:05.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.355+0000 7faf8ea00640 1 --2- 172.21.15.43:0/1857160756 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf90073af0 0x7faf9019e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:05.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.355+0000 7faf8e000640 1 --2- 172.21.15.43:0/1857160756 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf90074410 0x7faf9019e9e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7faf7c000c00 tx=0x7faf7c0043c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:05.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.356+0000 7faf87400640 1 -- 172.21.15.43:0/1857160756 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf7c042020 con 0x7faf90074410 2024-03-28T22:38:05.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.356+0000 7faf87400640 1 -- 172.21.15.43:0/1857160756 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf7c0054e0 con 0x7faf90074410 2024-03-28T22:38:05.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.356+0000 7faf87400640 1 -- 172.21.15.43:0/1857160756 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf7c003af0 con 0x7faf90074410 2024-03-28T22:38:05.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.356+0000 7faf8fe00640 1 -- 172.21.15.43:0/1857160756 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf901a36d0 con 0x7faf90074410 2024-03-28T22:38:05.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.356+0000 7faf8fe00640 1 -- 172.21.15.43:0/1857160756 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf901aafc0 con 0x7faf90074410 2024-03-28T22:38:05.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.357+0000 7faf8fe00640 1 -- 172.21.15.43:0/1857160756 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf90068460 con 0x7faf90074410 2024-03-28T22:38:05.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.360+0000 7faf87400640 1 -- 172.21.15.43:0/1857160756 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 27) ==== 136989+0+0 (secure 0 0 0) 0x7faf7c045020 con 0x7faf90074410 2024-03-28T22:38:05.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.361+0000 7faf87400640 1 --2- 172.21.15.43:0/1857160756 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7faf740a2090 0x7faf740a4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.361+0000 7faf87400640 1 -- 172.21.15.43:0/1857160756 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7faf7c0f0830 con 0x7faf90074410 2024-03-28T22:38:05.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.361+0000 7faf8ea00640 1 --2- 172.21.15.43:0/1857160756 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7faf740a2090 0x7faf740a4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.362+0000 7faf8ea00640 1 --2- 172.21.15.43:0/1857160756 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7faf740a2090 0x7faf740a4550 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7faf8000bfd0 tx=0x7faf8000b390 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:05.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.363+0000 7faf87400640 1 -- 172.21.15.43:0/1857160756 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7faf7c0bd4b0 con 0x7faf90074410 2024-03-28T22:38:05.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:05 smithi172 ceph-mon[36566]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 29 op/s 2024-03-28T22:38:05.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:05 smithi082 ceph-mon[36471]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 29 op/s 2024-03-28T22:38:05.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.613+0000 7faf8fe00640 1 -- 172.21.15.43:0/1857160756 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf9019f7a0 con 0x7faf90074410 2024-03-28T22:38:05.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.614+0000 7faf87400640 1 -- 172.21.15.43:0/1857160756 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11934 (secure 0 0 0) 0x7faf9019f7a0 con 0x7faf90074410 2024-03-28T22:38:05.613 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:38:05.613 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:33:28.906396+0000","modified":"2024-03-28T22:36:49.597670+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24572,"mds_1":24556,"mds_2":24535,"mds_3":24523,"mds_4":24518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/1084988072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1084988072},{"type":"v1","addr":"172.21.15.82:6835","nonce":1084988072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572,24523,24518,24535,24556]},"id":1}]} 2024-03-28T22:38:05.613 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 29 2024-03-28T22:38:05.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.617+0000 7faf8fe00640 1 -- 172.21.15.43:0/1857160756 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7faf740a2090 msgr2=0x7faf740a4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:05.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.617+0000 7faf8fe00640 1 --2- 172.21.15.43:0/1857160756 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7faf740a2090 0x7faf740a4550 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7faf8000bfd0 tx=0x7faf8000b390 comp rx=0 tx=0).stop 2024-03-28T22:38:05.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.617+0000 7faf8fe00640 1 -- 172.21.15.43:0/1857160756 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf90074410 msgr2=0x7faf9019e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:05.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.617+0000 7faf8fe00640 1 --2- 172.21.15.43:0/1857160756 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf90074410 0x7faf9019e9e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7faf7c000c00 tx=0x7faf7c0043c0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.617+0000 7faf8fe00640 1 -- 172.21.15.43:0/1857160756 shutdown_connections 2024-03-28T22:38:05.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.617+0000 7faf8fe00640 1 --2- 172.21.15.43:0/1857160756 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7faf740a2090 0x7faf740a4550 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.617+0000 7faf8fe00640 1 --2- 172.21.15.43:0/1857160756 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faf90112550 0x7faf901a2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.617+0000 7faf8fe00640 1 --2- 172.21.15.43:0/1857160756 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf90074410 0x7faf9019e9e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.617+0000 7faf8fe00640 1 --2- 172.21.15.43:0/1857160756 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf90073af0 0x7faf9019e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.617+0000 7faf8fe00640 1 -- 172.21.15.43:0/1857160756 >> 172.21.15.43:0/1857160756 conn(0x7faf900fd030 msgr2=0x7faf900fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:05.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.618+0000 7faf8fe00640 1 -- 172.21.15.43:0/1857160756 shutdown_connections 2024-03-28T22:38:05.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.618+0000 7faf8fe00640 1 -- 172.21.15.43:0/1857160756 wait complete. 2024-03-28T22:38:05.634 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T22:38:05.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.830+0000 7f55aa600640 1 -- 172.21.15.43:0/4184666240 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55a4107660 msgr2=0x7f55a4107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:05.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.830+0000 7f55aa600640 1 --2- 172.21.15.43:0/4184666240 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55a4107660 0x7f55a4107a40 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f5598009900 tx=0x7f559802bc30 comp rx=0 tx=0).stop 2024-03-28T22:38:05.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.831+0000 7f55aa600640 1 -- 172.21.15.43:0/4184666240 shutdown_connections 2024-03-28T22:38:05.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.831+0000 7f55aa600640 1 --2- 172.21.15.43:0/4184666240 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f55a4101f80 0x7f55a410e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.831+0000 7f55aa600640 1 --2- 172.21.15.43:0/4184666240 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55a41015c0 0x7f55a4101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.831+0000 7f55aa600640 1 --2- 172.21.15.43:0/4184666240 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55a4107660 0x7f55a4107a40 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.831+0000 7f55aa600640 1 -- 172.21.15.43:0/4184666240 >> 172.21.15.43:0/4184666240 conn(0x7f55a40fd0f0 msgr2=0x7f55a40ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:05.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.831+0000 7f55aa600640 1 -- 172.21.15.43:0/4184666240 shutdown_connections 2024-03-28T22:38:05.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.831+0000 7f55aa600640 1 -- 172.21.15.43:0/4184666240 wait complete. 2024-03-28T22:38:05.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.832+0000 7f55aa600640 1 Processor -- start 2024-03-28T22:38:05.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.832+0000 7f55aa600640 1 -- start start 2024-03-28T22:38:05.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.832+0000 7f55aa600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55a41015c0 0x7f55a4073a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.832+0000 7f55aa600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f55a4101f80 0x7f55a4073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.832+0000 7f55aa600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55a4107660 0x7f55a40744d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.832+0000 7f55aa600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f55a41119f0 con 0x7f55a4101f80 2024-03-28T22:38:05.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.832+0000 7f55aa600640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f55a4111870 con 0x7f55a4107660 2024-03-28T22:38:05.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.832+0000 7f55aa600640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f55a4111b70 con 0x7f55a41015c0 2024-03-28T22:38:05.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.832+0000 7f55a9200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55a41015c0 0x7f55a4073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.832+0000 7f55a9200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55a41015c0 0x7f55a4073a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57242/0 (socket says 172.21.15.43:57242) 2024-03-28T22:38:05.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.832+0000 7f55a9200640 1 -- 172.21.15.43:0/2667646500 learned_addr learned my addr 172.21.15.43:0/2667646500 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:05.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.833+0000 7f55a9c00640 1 --2- 172.21.15.43:0/2667646500 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55a4107660 0x7f55a40744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.833+0000 7f55a3e00640 1 --2- 172.21.15.43:0/2667646500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f55a4101f80 0x7f55a4073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.833+0000 7f55a9200640 1 -- 172.21.15.43:0/2667646500 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55a4107660 msgr2=0x7f55a40744d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:05.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.833+0000 7f55a9200640 1 --2- 172.21.15.43:0/2667646500 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55a4107660 0x7f55a40744d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.833+0000 7f55a9200640 1 -- 172.21.15.43:0/2667646500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f55a4101f80 msgr2=0x7f55a4073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:05.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.833+0000 7f55a9200640 1 --2- 172.21.15.43:0/2667646500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f55a4101f80 0x7f55a4073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.833+0000 7f55a9200640 1 -- 172.21.15.43:0/2667646500 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55a41a62b0 con 0x7f55a41015c0 2024-03-28T22:38:05.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.833+0000 7f55a9c00640 1 --2- 172.21.15.43:0/2667646500 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55a4107660 0x7f55a40744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:05.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.833+0000 7f55a9200640 1 --2- 172.21.15.43:0/2667646500 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55a41015c0 0x7f55a4073a50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f559802c240 tx=0x7f55980042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:05.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.834+0000 7f55a1600640 1 -- 172.21.15.43:0/2667646500 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5598041070 con 0x7f55a41015c0 2024-03-28T22:38:05.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.834+0000 7f55a1600640 1 -- 172.21.15.43:0/2667646500 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f559802ccc0 con 0x7f55a41015c0 2024-03-28T22:38:05.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.834+0000 7f55a1600640 1 -- 172.21.15.43:0/2667646500 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5598003cd0 con 0x7f55a41015c0 2024-03-28T22:38:05.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.834+0000 7f55aa600640 1 -- 172.21.15.43:0/2667646500 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55a41a6540 con 0x7f55a41015c0 2024-03-28T22:38:05.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.834+0000 7f55aa600640 1 -- 172.21.15.43:0/2667646500 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55a41a6a80 con 0x7f55a41015c0 2024-03-28T22:38:05.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.837+0000 7f55ae861740 1 Processor -- start 2024-03-28T22:38:05.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.838+0000 7f55ae861740 1 -- start start 2024-03-28T22:38:05.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.838+0000 7f55ae861740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559bd6198550 0x559bd6198930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.838+0000 7f55ae861740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559bd6198e70 0x559bd61a1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.838+0000 7f55ae861740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559bd61a1a50 0x559bd61a3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.838+0000 7f55ae861740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x559bd60fc050 con 0x559bd61a1a50 2024-03-28T22:38:05.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.838+0000 7f55ae861740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x559bd60a4ce0 con 0x559bd6198550 2024-03-28T22:38:05.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.838+0000 7f55ae861740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x559bd60f80f0 con 0x559bd6198e70 2024-03-28T22:38:05.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.838+0000 7f55a9c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559bd6198550 0x559bd6198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.838+0000 7f55a9c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559bd6198550 0x559bd6198930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34056/0 (socket says 172.21.15.43:34056) 2024-03-28T22:38:05.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.838+0000 7f55a9c00640 1 -- 172.21.15.43:0/2800445021 learned_addr learned my addr 172.21.15.43:0/2800445021 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:05.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.839+0000 7f55a9c00640 1 -- 172.21.15.43:0/2800445021 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559bd6198e70 msgr2=0x559bd61a1510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:38:05.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.839+0000 7f55a9c00640 1 --2- 172.21.15.43:0/2800445021 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559bd6198e70 0x559bd61a1510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.839+0000 7f55a9c00640 1 -- 172.21.15.43:0/2800445021 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559bd61a1a50 msgr2=0x559bd61a3e40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:38:05.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.839+0000 7f55a9c00640 1 --2- 172.21.15.43:0/2800445021 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559bd61a1a50 0x559bd61a3e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.839+0000 7f55a9c00640 1 -- 172.21.15.43:0/2800445021 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559bd5e103f0 con 0x559bd6198550 2024-03-28T22:38:05.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.839+0000 7f55a9c00640 1 --2- 172.21.15.43:0/2800445021 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559bd6198550 0x559bd6198930 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f559c010710 tx=0x7f559c010bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9a899775146fc9d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:05.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.839+0000 7f55a1600640 1 -- 172.21.15.43:0/2667646500 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 27) ==== 136989+0+0 (secure 0 0 0) 0x7f5598044070 con 0x7f55a41015c0 2024-03-28T22:38:05.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.839+0000 7f5596000640 1 -- 172.21.15.43:0/2800445021 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559c00e8d0 con 0x559bd6198550 2024-03-28T22:38:05.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.839+0000 7f5596000640 1 -- 172.21.15.43:0/2800445021 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f559c012450 con 0x559bd6198550 2024-03-28T22:38:05.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.839+0000 7f5596000640 1 -- 172.21.15.43:0/2800445021 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559c01c650 con 0x559bd6198550 2024-03-28T22:38:05.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.840+0000 7f55ae861740 1 -- 172.21.15.43:0/2800445021 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559bd6198550 msgr2=0x559bd6198930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:05.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.840+0000 7f55ae861740 1 --2- 172.21.15.43:0/2800445021 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559bd6198550 0x559bd6198930 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f559c010710 tx=0x7f559c010bd0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.840+0000 7f55ae861740 1 -- 172.21.15.43:0/2800445021 shutdown_connections 2024-03-28T22:38:05.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.840+0000 7f55ae861740 1 --2- 172.21.15.43:0/2800445021 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559bd61a1a50 0x559bd61a3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.840+0000 7f55ae861740 1 --2- 172.21.15.43:0/2800445021 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559bd6198e70 0x559bd61a1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.840+0000 7f55ae861740 1 --2- 172.21.15.43:0/2800445021 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559bd6198550 0x559bd6198930 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.840+0000 7f55ae861740 1 -- 172.21.15.43:0/2800445021 >> 172.21.15.43:0/2800445021 conn(0x559bd6192c80 msgr2=0x559bd6193060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:05.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.840+0000 7f55ae861740 1 -- 172.21.15.43:0/2800445021 shutdown_connections 2024-03-28T22:38:05.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.841+0000 7f55ae861740 1 -- 172.21.15.43:0/2800445021 wait complete. 2024-03-28T22:38:05.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.841+0000 7f55ae861740 1 -- start start 2024-03-28T22:38:05.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.841+0000 7f55ae861740 1 Processor -- start 2024-03-28T22:38:05.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.841+0000 7f55a1600640 1 --2- 172.21.15.43:0/2667646500 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f558c0a1fd0 0x7f558c0a4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.842+0000 7f55a1600640 1 -- 172.21.15.43:0/2667646500 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f55980f00b0 con 0x7f55a41015c0 2024-03-28T22:38:05.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.842+0000 7f55a9c00640 1 --2- 172.21.15.43:0/2667646500 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f558c0a1fd0 0x7f558c0a4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.842+0000 7f55ae861740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559bd6198550 0x559bd619eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.842+0000 7f55ae861740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559bd6198e70 0x559bd619f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.842+0000 7f55ae861740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559bd61a1a50 0x559bd619f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.842+0000 7f55ae861740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x559bd61a4840 con 0x559bd61a1a50 2024-03-28T22:38:05.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.842+0000 7f55ae861740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x559bd609a2f0 con 0x559bd6198550 2024-03-28T22:38:05.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.842+0000 7f55ae861740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x559bd61a49c0 con 0x559bd6198e70 2024-03-28T22:38:05.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.842+0000 7f55ae861740 10 client.0 _ll_register_callbacks cb 0x559bd5e0fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:05.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.842+0000 7f55a3e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559bd6198e70 0x559bd619f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.842+0000 7f55a3e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559bd6198e70 0x559bd619f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57256/0 (socket says 172.21.15.43:57256) 2024-03-28T22:38:05.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.842+0000 7f55a3e00640 1 -- 172.21.15.43:0/1822246518 learned_addr learned my addr 172.21.15.43:0/1822246518 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:05.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.842+0000 7f55a9200640 1 --2- 172.21.15.43:0/1822246518 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559bd6198550 0x559bd619eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.843+0000 7f55a3e00640 1 -- 172.21.15.43:0/1822246518 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559bd6198550 msgr2=0x559bd619eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:05.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.843+0000 7f55a3e00640 1 --2- 172.21.15.43:0/1822246518 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559bd6198550 0x559bd619eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.843+0000 7f55a3e00640 1 -- 172.21.15.43:0/1822246518 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559bd61a1a50 msgr2=0x559bd619f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:05.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.843+0000 7f55a9c00640 1 --2- 172.21.15.43:0/1822246518 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559bd61a1a50 0x559bd619f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.843+0000 7f55a3e00640 1 --2- 172.21.15.43:0/1822246518 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559bd61a1a50 0x559bd619f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.843+0000 7f55a3e00640 1 -- 172.21.15.43:0/1822246518 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559bd5eda7d0 con 0x559bd6198e70 2024-03-28T22:38:05.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.843+0000 7f55a9c00640 1 --2- 172.21.15.43:0/1822246518 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559bd61a1a50 0x559bd619f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:38:05.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.843+0000 7f55a3e00640 1 --2- 172.21.15.43:0/1822246518 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559bd6198e70 0x559bd619f0e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5590011430 tx=0x7f5590011830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5073d10c85ac4eb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:05.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.843+0000 7f558aa00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:38:05.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.843+0000 7f558aa00640 1 -- 172.21.15.43:0/1822246518 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5590012860 con 0x559bd6198e70 2024-03-28T22:38:05.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.844+0000 7f55a9c00640 1 --2- 172.21.15.43:0/2667646500 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f558c0a1fd0 0x7f558c0a4490 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f559c012d90 tx=0x7f559c012df0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:05.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.844+0000 7f558aa00640 1 -- 172.21.15.43:0/1822246518 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f5590012a00 con 0x559bd6198e70 2024-03-28T22:38:05.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.844+0000 7f55ae861740 1 -- 172.21.15.43:0/1822246518 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f559c012450 con 0x559bd6198e70 2024-03-28T22:38:05.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.844+0000 7f558aa00640 1 -- 172.21.15.43:0/1822246518 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5590025050 con 0x559bd6198e70 2024-03-28T22:38:05.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.844+0000 7f55ae861740 10 client.25927 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:05.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.844+0000 7f55ae861740 1 -- 172.21.15.43:0/1822246518 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559bd61c1bf0 con 0x559bd6198e70 2024-03-28T22:38:05.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.844+0000 7f558aa00640 1 -- 172.21.15.43:0/1822246518 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f559c012450 con 0x559bd6198e70 2024-03-28T22:38:05.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.844+0000 7f558aa00640 1 -- 172.21.15.43:0/1822246518 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f559001a3f0 con 0x559bd6198e70 2024-03-28T22:38:05.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.845+0000 7f55ae861740 10 client.25927 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:05.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.845+0000 7f55ae861740 10 client.25927 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:38:05.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.845+0000 7f55ae861740 20 client.25927 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:05.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.845+0000 7f55ae861740 1 --2- 172.21.15.43:0/1822246518 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x559bd61c30b0 0x559bd61c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.845+0000 7f55ae861740 4 client.25927 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:38:05.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.845+0000 7f55ae861740 1 -- 172.21.15.43:0/1822246518 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559bd60a5680 con 0x559bd61c30b0 2024-03-28T22:38:05.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.845+0000 7f55a9200640 1 --2- 172.21.15.43:0/1822246518 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x559bd61c30b0 0x559bd61c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.846+0000 7f55a9200640 1 --2- 172.21.15.43:0/1822246518 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x559bd61c30b0 0x559bd61c54f0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a26c6a05ed9a29e2 server_cookie=a6a642f1c053a670 in_seq=0 out_seq=0 2024-03-28T22:38:05.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.846+0000 7f558aa00640 10 client.25927 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:38:05.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.846+0000 7f558aa00640 1 -- 172.21.15.43:0/1822246518 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559bd60a5680 con 0x559bd61c30b0 2024-03-28T22:38:05.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.846+0000 7f558aa00640 10 client.25927 handle_command_reply: tid=0 2024-03-28T22:38:05.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.847+0000 7f55ae861740 1 client.25927 shutdown 2024-03-28T22:38:05.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.847+0000 7f55ae861740 1 -- 172.21.15.43:0/1822246518 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559bd6198e70 msgr2=0x559bd619f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:05.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.847+0000 7f55ae861740 1 --2- 172.21.15.43:0/1822246518 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559bd6198e70 0x559bd619f0e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5590011430 tx=0x7f5590011830 comp rx=0 tx=0).stop 2024-03-28T22:38:05.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.847+0000 7f55ae861740 1 -- 172.21.15.43:0/1822246518 shutdown_connections 2024-03-28T22:38:05.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.847+0000 7f55ae861740 1 --2- 172.21.15.43:0/1822246518 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x559bd61c30b0 0x559bd61c54f0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.847+0000 7f55ae861740 1 --2- 172.21.15.43:0/1822246518 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559bd61a1a50 0x559bd619f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.847+0000 7f55ae861740 1 --2- 172.21.15.43:0/1822246518 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559bd6198e70 0x559bd619f0e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.847+0000 7f55ae861740 1 --2- 172.21.15.43:0/1822246518 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559bd6198550 0x559bd619eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.847+0000 7f558aa00640 0 client.25927 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:38:05.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.847+0000 7f55ae861740 1 -- 172.21.15.43:0/1822246518 >> 172.21.15.43:0/1822246518 conn(0x559bd6192c80 msgr2=0x559bd61949d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:05.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.848+0000 7f55ae861740 1 -- 172.21.15.43:0/1822246518 shutdown_connections 2024-03-28T22:38:05.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.848+0000 7f55ae861740 1 -- 172.21.15.43:0/1822246518 wait complete. 2024-03-28T22:38:05.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.848+0000 7f55ae861740 20 client.25927 trim_cache size 0 max 16384 2024-03-28T22:38:05.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.876+0000 7f55ae861740 1 Processor -- start 2024-03-28T22:38:05.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.876+0000 7f55ae861740 1 -- start start 2024-03-28T22:38:05.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.876+0000 7f55ae861740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559bd6198550 0x559bd61a12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.876+0000 7f55ae861740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559bd6198e70 0x559bd618dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.876+0000 7f55ae861740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559bd61a1a50 0x559bd61cadc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.876+0000 7f55ae861740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f55a41a8360 con 0x559bd61a1a50 2024-03-28T22:38:05.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.876+0000 7f55ae861740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f55a41a9ea0 con 0x559bd6198550 2024-03-28T22:38:05.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.876+0000 7f55ae861740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f55a41a8a30 con 0x559bd6198e70 2024-03-28T22:38:05.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.877+0000 7f55a9200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559bd6198550 0x559bd61a12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.877+0000 7f55a9200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559bd6198550 0x559bd61a12d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34066/0 (socket says 172.21.15.43:34066) 2024-03-28T22:38:05.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.877+0000 7f55a9200640 1 -- 172.21.15.43:0/1805940915 learned_addr learned my addr 172.21.15.43:0/1805940915 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:05.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.877+0000 7f55a3e00640 1 --2- 172.21.15.43:0/1805940915 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559bd6198e70 0x559bd618dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.877+0000 7f55a9c00640 1 --2- 172.21.15.43:0/1805940915 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559bd61a1a50 0x559bd61cadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.877+0000 7f55a9200640 1 -- 172.21.15.43:0/1805940915 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559bd6198e70 msgr2=0x559bd618dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:05.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.877+0000 7f55a9200640 1 --2- 172.21.15.43:0/1805940915 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559bd6198e70 0x559bd618dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.877+0000 7f55a9200640 1 -- 172.21.15.43:0/1805940915 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559bd61a1a50 msgr2=0x559bd61cadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:05.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.877+0000 7f55a9200640 1 --2- 172.21.15.43:0/1805940915 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559bd61a1a50 0x559bd61cadc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.877+0000 7f55a9200640 1 -- 172.21.15.43:0/1805940915 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559bd619a780 con 0x559bd6198550 2024-03-28T22:38:05.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.877+0000 7f55a9200640 1 --2- 172.21.15.43:0/1805940915 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559bd6198550 0x559bd61a12d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f559802cfd0 tx=0x7f55980dc040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0c8584c7f1b5dba server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:05.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.878+0000 7f5595600640 1 -- 172.21.15.43:0/1805940915 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559804b040 con 0x559bd6198550 2024-03-28T22:38:05.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.878+0000 7f5595600640 1 -- 172.21.15.43:0/1805940915 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5598004c80 con 0x559bd6198550 2024-03-28T22:38:05.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.878+0000 7f5595600640 1 -- 172.21.15.43:0/1805940915 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55980c0710 con 0x559bd6198550 2024-03-28T22:38:05.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.879+0000 7f55ae861740 1 -- 172.21.15.43:0/1805940915 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559bd6198550 msgr2=0x559bd61a12d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:05.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.879+0000 7f55ae861740 1 --2- 172.21.15.43:0/1805940915 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559bd6198550 0x559bd61a12d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f559802cfd0 tx=0x7f55980dc040 comp rx=0 tx=0).stop 2024-03-28T22:38:05.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.879+0000 7f55ae861740 1 -- 172.21.15.43:0/1805940915 shutdown_connections 2024-03-28T22:38:05.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.879+0000 7f55ae861740 1 --2- 172.21.15.43:0/1805940915 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559bd61a1a50 0x559bd61cadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.879+0000 7f55ae861740 1 --2- 172.21.15.43:0/1805940915 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559bd6198e70 0x559bd618dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.879+0000 7f55ae861740 1 --2- 172.21.15.43:0/1805940915 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559bd6198550 0x559bd61a12d0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.879+0000 7f55ae861740 1 -- 172.21.15.43:0/1805940915 >> 172.21.15.43:0/1805940915 conn(0x559bd6192c80 msgr2=0x559bd6194050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:05.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.879+0000 7f55ae861740 1 -- 172.21.15.43:0/1805940915 shutdown_connections 2024-03-28T22:38:05.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.879+0000 7f55ae861740 1 -- 172.21.15.43:0/1805940915 wait complete. 2024-03-28T22:38:05.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.879+0000 7f55ae861740 1 -- start start 2024-03-28T22:38:05.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.880+0000 7f55ae861740 1 Processor -- start 2024-03-28T22:38:05.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.880+0000 7f55ae861740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559bd6198550 0x559bd61a6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.880+0000 7f55ae861740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559bd6198e70 0x559bd61a7300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.880+0000 7f55ae861740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559bd61a1a50 0x559bd61a7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.880+0000 7f55ae861740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f55a41a97d0 con 0x559bd6198550 2024-03-28T22:38:05.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.880+0000 7f55ae861740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f55a41a9100 con 0x559bd6198e70 2024-03-28T22:38:05.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.880+0000 7f55ae861740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f55a4074f30 con 0x559bd61a1a50 2024-03-28T22:38:05.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.880+0000 7f55ae861740 10 client.0 _ll_register_callbacks cb 0x7f5590009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:05.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.881+0000 7f55a9200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559bd6198550 0x559bd61a6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.881+0000 7f55a9200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559bd6198550 0x559bd61a6dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55760/0 (socket says 172.21.15.43:55760) 2024-03-28T22:38:05.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.881+0000 7f55a9200640 1 -- 172.21.15.43:0/3076393749 learned_addr learned my addr 172.21.15.43:0/3076393749 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:05.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.881+0000 7f55a3e00640 1 --2- 172.21.15.43:0/3076393749 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559bd6198e70 0x559bd61a7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.881+0000 7f55a9c00640 1 --2- 172.21.15.43:0/3076393749 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559bd61a1a50 0x559bd61a7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.881+0000 7f55a9200640 1 -- 172.21.15.43:0/3076393749 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559bd61a1a50 msgr2=0x559bd61a7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:05.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.881+0000 7f55a9200640 1 --2- 172.21.15.43:0/3076393749 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559bd61a1a50 0x559bd61a7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.881+0000 7f55a9200640 1 -- 172.21.15.43:0/3076393749 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559bd6198e70 msgr2=0x559bd61a7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:05.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.881+0000 7f55a9200640 1 --2- 172.21.15.43:0/3076393749 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559bd6198e70 0x559bd61a7300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.881+0000 7f55a9200640 1 -- 172.21.15.43:0/3076393749 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559bd62285c0 con 0x559bd6198550 2024-03-28T22:38:05.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.881+0000 7f55a3e00640 1 --2- 172.21.15.43:0/3076393749 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559bd6198e70 0x559bd61a7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:05.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.882+0000 7f55a9200640 1 --2- 172.21.15.43:0/3076393749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559bd6198550 0x559bd61a6dc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f55980b8a20 tx=0x7f55980b8990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca0b42aaaa61e485 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:05.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.882+0000 7f558aa00640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:38:05.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.882+0000 7f558aa00640 1 -- 172.21.15.43:0/3076393749 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559804b040 con 0x559bd6198550 2024-03-28T22:38:05.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.882+0000 7f558aa00640 1 -- 172.21.15.43:0/3076393749 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f55980dc0b0 con 0x559bd6198550 2024-03-28T22:38:05.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.882+0000 7f55ae861740 1 -- 172.21.15.43:0/3076393749 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5598004c80 con 0x559bd6198550 2024-03-28T22:38:05.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.882+0000 7f558aa00640 1 -- 172.21.15.43:0/3076393749 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f55980bccc0 con 0x559bd6198550 2024-03-28T22:38:05.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.882+0000 7f55ae861740 10 client.15840 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:05.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.882+0000 7f55ae861740 1 -- 172.21.15.43:0/3076393749 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559bd6229880 con 0x559bd6198550 2024-03-28T22:38:05.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.883+0000 7f558aa00640 1 -- 172.21.15.43:0/3076393749 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5598004c80 con 0x559bd6198550 2024-03-28T22:38:05.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.883+0000 7f558aa00640 1 -- 172.21.15.43:0/3076393749 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f55980d6080 con 0x559bd6198550 2024-03-28T22:38:05.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.884+0000 7f55ae861740 10 client.15840 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:05.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.884+0000 7f55ae861740 10 client.15840 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:38:05.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.884+0000 7f55ae861740 20 client.15840 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:05.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.884+0000 7f55ae861740 1 --2- 172.21.15.43:0/3076393749 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x559bd6229ea0 0x559bd622c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:05.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.884+0000 7f55ae861740 4 client.15840 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:38:05.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.884+0000 7f55ae861740 1 -- 172.21.15.43:0/3076393749 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559bd6109590 con 0x559bd6229ea0 2024-03-28T22:38:05.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.884+0000 7f55a3e00640 1 --2- 172.21.15.43:0/3076393749 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x559bd6229ea0 0x559bd622c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:05.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.884+0000 7f55a3e00640 1 --2- 172.21.15.43:0/3076393749 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x559bd6229ea0 0x559bd622c290 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5027b5a47d6f07de server_cookie=9a120d49c6071ef9 in_seq=0 out_seq=0 2024-03-28T22:38:05.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.884+0000 7f558aa00640 10 client.15840 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:38:05.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.885+0000 7f558aa00640 1 -- 172.21.15.43:0/3076393749 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559bd6109590 con 0x559bd6229ea0 2024-03-28T22:38:05.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.885+0000 7f558aa00640 10 client.15840 handle_command_reply: tid=0 2024-03-28T22:38:05.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.885+0000 7f55ae861740 1 client.15840 shutdown 2024-03-28T22:38:05.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.885+0000 7f55ae861740 1 -- 172.21.15.43:0/3076393749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559bd6198550 msgr2=0x559bd61a6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:05.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.885+0000 7f55ae861740 1 --2- 172.21.15.43:0/3076393749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559bd6198550 0x559bd61a6dc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f55980b8a20 tx=0x7f55980b8990 comp rx=0 tx=0).stop 2024-03-28T22:38:05.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.885+0000 7f55ae861740 1 -- 172.21.15.43:0/3076393749 shutdown_connections 2024-03-28T22:38:05.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.885+0000 7f55ae861740 1 --2- 172.21.15.43:0/3076393749 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x559bd6229ea0 0x559bd622c290 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.885+0000 7f55ae861740 1 --2- 172.21.15.43:0/3076393749 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559bd6198e70 0x559bd61a7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.885+0000 7f55ae861740 1 --2- 172.21.15.43:0/3076393749 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559bd61a1a50 0x559bd61a7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.885+0000 7f55ae861740 1 --2- 172.21.15.43:0/3076393749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559bd6198550 0x559bd61a6dc0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.885+0000 7f55ae861740 1 -- 172.21.15.43:0/3076393749 >> 172.21.15.43:0/3076393749 conn(0x559bd6192c80 msgr2=0x559bd6191530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:05.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.885+0000 7f558aa00640 0 client.15840 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:38:05.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.885+0000 7f55ae861740 1 -- 172.21.15.43:0/3076393749 shutdown_connections 2024-03-28T22:38:05.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.885+0000 7f55ae861740 1 -- 172.21.15.43:0/3076393749 wait complete. 2024-03-28T22:38:05.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.885+0000 7f55ae861740 20 client.15840 trim_cache size 0 max 16384 2024-03-28T22:38:05.887 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:38:05.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.886+0000 7f55aa600640 1 -- 172.21.15.43:0/2667646500 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f558c0a1fd0 msgr2=0x7f558c0a4490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:05.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.886+0000 7f55aa600640 1 --2- 172.21.15.43:0/2667646500 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f558c0a1fd0 0x7f558c0a4490 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f559c012d90 tx=0x7f559c012df0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.886+0000 7f55aa600640 1 -- 172.21.15.43:0/2667646500 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55a41015c0 msgr2=0x7f55a4073a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:05.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.886+0000 7f55aa600640 1 --2- 172.21.15.43:0/2667646500 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55a41015c0 0x7f55a4073a50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f559802c240 tx=0x7f55980042f0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.886+0000 7f55aa600640 1 -- 172.21.15.43:0/2667646500 shutdown_connections 2024-03-28T22:38:05.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.886+0000 7f55aa600640 1 --2- 172.21.15.43:0/2667646500 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f558c0a1fd0 0x7f558c0a4490 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.886+0000 7f55aa600640 1 --2- 172.21.15.43:0/2667646500 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55a4107660 0x7f55a40744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.886+0000 7f55aa600640 1 --2- 172.21.15.43:0/2667646500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f55a4101f80 0x7f55a4073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.886+0000 7f55aa600640 1 --2- 172.21.15.43:0/2667646500 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55a41015c0 0x7f55a4073a50 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:05.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.886+0000 7f55aa600640 1 -- 172.21.15.43:0/2667646500 >> 172.21.15.43:0/2667646500 conn(0x7f55a40fd0f0 msgr2=0x7f55a40fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:05.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.886+0000 7f55aa600640 1 -- 172.21.15.43:0/2667646500 shutdown_connections 2024-03-28T22:38:05.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:05.886+0000 7f55aa600640 1 -- 172.21.15.43:0/2667646500 wait complete. 2024-03-28T22:38:05.903 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T22:38:06.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.092+0000 7fa4bfe00640 1 -- 172.21.15.43:0/228087372 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa4c01015c0 msgr2=0x7fa4c0101a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.092+0000 7fa4bfe00640 1 --2- 172.21.15.43:0/228087372 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa4c01015c0 0x7fa4c0101a40 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac009900 tx=0x7fa4ac02bc30 comp rx=0 tx=0).stop 2024-03-28T22:38:06.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.092+0000 7fa4bfe00640 1 -- 172.21.15.43:0/228087372 shutdown_connections 2024-03-28T22:38:06.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.092+0000 7fa4bfe00640 1 --2- 172.21.15.43:0/228087372 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa4c01015c0 0x7fa4c0101a40 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.092+0000 7fa4bfe00640 1 --2- 172.21.15.43:0/228087372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4c0101f80 0x7fa4c010e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.092+0000 7fa4bfe00640 1 --2- 172.21.15.43:0/228087372 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4c0107660 0x7fa4c0107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.092+0000 7fa4bfe00640 1 -- 172.21.15.43:0/228087372 >> 172.21.15.43:0/228087372 conn(0x7fa4c00fd0f0 msgr2=0x7fa4c00ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:06.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.093+0000 7fa4bfe00640 1 -- 172.21.15.43:0/228087372 shutdown_connections 2024-03-28T22:38:06.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.093+0000 7fa4bfe00640 1 -- 172.21.15.43:0/228087372 wait complete. 2024-03-28T22:38:06.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.093+0000 7fa4bfe00640 1 Processor -- start 2024-03-28T22:38:06.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.094+0000 7fa4bfe00640 1 -- start start 2024-03-28T22:38:06.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.094+0000 7fa4bfe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa4c01015c0 0x7fa4c019a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.094+0000 7fa4bfe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4c0101f80 0x7fa4c019a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.094+0000 7fa4bfe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4c0107660 0x7fa4c019e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.094+0000 7fa4bfe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c01119f0 con 0x7fa4c0101f80 2024-03-28T22:38:06.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.094+0000 7fa4bfe00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c0111870 con 0x7fa4c01015c0 2024-03-28T22:38:06.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.094+0000 7fa4bfe00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c0111b70 con 0x7fa4c0107660 2024-03-28T22:38:06.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.094+0000 7fa4bea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa4c01015c0 0x7fa4c019a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.094+0000 7fa4bea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa4c01015c0 0x7fa4c019a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34092/0 (socket says 172.21.15.43:34092) 2024-03-28T22:38:06.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.094+0000 7fa4bea00640 1 -- 172.21.15.43:0/1592078643 learned_addr learned my addr 172.21.15.43:0/1592078643 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:06.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.094+0000 7fa4bf400640 1 --2- 172.21.15.43:0/1592078643 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4c0107660 0x7fa4c019e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.094+0000 7fa4be000640 1 --2- 172.21.15.43:0/1592078643 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4c0101f80 0x7fa4c019a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.095+0000 7fa4bea00640 1 -- 172.21.15.43:0/1592078643 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4c0107660 msgr2=0x7fa4c019e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.095+0000 7fa4bea00640 1 --2- 172.21.15.43:0/1592078643 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4c0107660 0x7fa4c019e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.095+0000 7fa4bea00640 1 -- 172.21.15.43:0/1592078643 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4c0101f80 msgr2=0x7fa4c019a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.095+0000 7fa4bea00640 1 --2- 172.21.15.43:0/1592078643 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4c0101f80 0x7fa4c019a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.095+0000 7fa4bea00640 1 -- 172.21.15.43:0/1592078643 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4c019f040 con 0x7fa4c01015c0 2024-03-28T22:38:06.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.095+0000 7fa4bea00640 1 --2- 172.21.15.43:0/1592078643 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa4c01015c0 0x7fa4c019a050 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa4b000e9c0 tx=0x7fa4b000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:06.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.095+0000 7fa4b7400640 1 -- 172.21.15.43:0/1592078643 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b000cd80 con 0x7fa4c01015c0 2024-03-28T22:38:06.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.096+0000 7fa4bfe00640 1 -- 172.21.15.43:0/1592078643 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4c019f330 con 0x7fa4c01015c0 2024-03-28T22:38:06.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.096+0000 7fa4b7400640 1 -- 172.21.15.43:0/1592078643 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4b00035f0 con 0x7fa4c01015c0 2024-03-28T22:38:06.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.096+0000 7fa4b7400640 1 -- 172.21.15.43:0/1592078643 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b0010690 con 0x7fa4c01015c0 2024-03-28T22:38:06.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.096+0000 7fa4bfe00640 1 -- 172.21.15.43:0/1592078643 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4c01a6bc0 con 0x7fa4c01015c0 2024-03-28T22:38:06.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.099+0000 7fa4c8409740 1 Processor -- start 2024-03-28T22:38:06.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.099+0000 7fa4c8409740 1 -- start start 2024-03-28T22:38:06.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.099+0000 7fa4c8409740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625eed21550 0x5625eed21930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.100+0000 7fa4c8409740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625eed21e70 0x5625eed2a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.100+0000 7fa4c8409740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625eed2aa50 0x5625eed2ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.100+0000 7fa4c8409740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5625eec85050 con 0x5625eed21e70 2024-03-28T22:38:06.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.100+0000 7fa4c8409740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5625eec2dce0 con 0x5625eed21550 2024-03-28T22:38:06.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.100+0000 7fa4c8409740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5625eec810f0 con 0x5625eed2aa50 2024-03-28T22:38:06.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.100+0000 7fa4be000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625eed2aa50 0x5625eed2ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.100+0000 7fa4bf400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625eed21550 0x5625eed21930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.100+0000 7fa4bf400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625eed21550 0x5625eed21930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34096/0 (socket says 172.21.15.43:34096) 2024-03-28T22:38:06.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.100+0000 7fa4bf400640 1 -- 172.21.15.43:0/3415445871 learned_addr learned my addr 172.21.15.43:0/3415445871 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:06.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.100+0000 7fa4be000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625eed2aa50 0x5625eed2ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57292/0 (socket says 172.21.15.43:57292) 2024-03-28T22:38:06.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.100+0000 7fa4bf400640 1 -- 172.21.15.43:0/3415445871 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625eed2aa50 msgr2=0x5625eed2ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:06.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.100+0000 7fa4bf400640 1 --2- 172.21.15.43:0/3415445871 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625eed2aa50 0x5625eed2ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.100+0000 7fa4bf400640 1 -- 172.21.15.43:0/3415445871 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625eed21e70 msgr2=0x5625eed2a510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:38:06.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.100+0000 7fa4bf400640 1 --2- 172.21.15.43:0/3415445871 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625eed21e70 0x5625eed2a510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.100+0000 7fa4bf400640 1 -- 172.21.15.43:0/3415445871 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625ee9993f0 con 0x5625eed21550 2024-03-28T22:38:06.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.100+0000 7fa4be000640 1 --2- 172.21.15.43:0/3415445871 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625eed2aa50 0x5625eed2ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:06.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.101+0000 7fa4bf400640 1 --2- 172.21.15.43:0/3415445871 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625eed21550 0x5625eed21930 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8009fd0 tx=0x7fa4b80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=682d8a14ead45c2d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:06.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.101+0000 7fa4b7400640 1 -- 172.21.15.43:0/1592078643 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 27) ==== 136989+0+0 (secure 0 0 0) 0x7fa4b00037a0 con 0x7fa4c01015c0 2024-03-28T22:38:06.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.101+0000 7fa4ab400640 1 -- 172.21.15.43:0/3415445871 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b8004d20 con 0x5625eed21550 2024-03-28T22:38:06.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.101+0000 7fa4ab400640 1 -- 172.21.15.43:0/3415445871 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4b8004050 con 0x5625eed21550 2024-03-28T22:38:06.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.101+0000 7fa4ab400640 1 -- 172.21.15.43:0/3415445871 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b8005260 con 0x5625eed21550 2024-03-28T22:38:06.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.102+0000 7fa4c8409740 1 -- 172.21.15.43:0/3415445871 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625eed21550 msgr2=0x5625eed21930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.102+0000 7fa4c8409740 1 --2- 172.21.15.43:0/3415445871 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625eed21550 0x5625eed21930 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8009fd0 tx=0x7fa4b80118d0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.102+0000 7fa4c8409740 1 -- 172.21.15.43:0/3415445871 shutdown_connections 2024-03-28T22:38:06.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.102+0000 7fa4c8409740 1 --2- 172.21.15.43:0/3415445871 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625eed2aa50 0x5625eed2ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.102+0000 7fa4c8409740 1 --2- 172.21.15.43:0/3415445871 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625eed21e70 0x5625eed2a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.102+0000 7fa4c8409740 1 --2- 172.21.15.43:0/3415445871 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625eed21550 0x5625eed21930 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.102+0000 7fa4c8409740 1 -- 172.21.15.43:0/3415445871 >> 172.21.15.43:0/3415445871 conn(0x5625eed1bc80 msgr2=0x5625eed1c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:06.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.102+0000 7fa4c8409740 1 -- 172.21.15.43:0/3415445871 shutdown_connections 2024-03-28T22:38:06.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.102+0000 7fa4c8409740 1 -- 172.21.15.43:0/3415445871 wait complete. 2024-03-28T22:38:06.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.103+0000 7fa4c8409740 1 -- start start 2024-03-28T22:38:06.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.103+0000 7fa4c8409740 1 Processor -- start 2024-03-28T22:38:06.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.103+0000 7fa4b7400640 1 --2- 172.21.15.43:0/1592078643 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa4a40a21a0 0x7fa4a40a4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.103+0000 7fa4bf400640 1 --2- 172.21.15.43:0/1592078643 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa4a40a21a0 0x7fa4a40a4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.103+0000 7fa4c8409740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625eed21550 0x5625eed27b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.104+0000 7fa4c8409740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625eed21e70 0x5625eed28050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.104+0000 7fa4b7400640 1 -- 172.21.15.43:0/1592078643 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fa4b0010430 con 0x7fa4c01015c0 2024-03-28T22:38:06.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.104+0000 7fa4c8409740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625eed2aa50 0x5625eed28590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.104+0000 7fa4c8409740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5625eed2d890 con 0x5625eed21550 2024-03-28T22:38:06.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.104+0000 7fa4c8409740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5625eec232f0 con 0x5625eed2aa50 2024-03-28T22:38:06.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.104+0000 7fa4c8409740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5625eed2da10 con 0x5625eed21e70 2024-03-28T22:38:06.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.104+0000 7fa4c8409740 10 client.0 _ll_register_callbacks cb 0x5625ee998be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:06.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.104+0000 7fa4bf400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625eed2aa50 0x5625eed28590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.104+0000 7fa4bf400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625eed2aa50 0x5625eed28590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34104/0 (socket says 172.21.15.43:34104) 2024-03-28T22:38:06.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.104+0000 7fa4bf400640 1 -- 172.21.15.43:0/2712312567 learned_addr learned my addr 172.21.15.43:0/2712312567 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:06.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.104+0000 7fa4bf400640 1 -- 172.21.15.43:0/2712312567 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625eed21e70 msgr2=0x5625eed28050 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T22:38:06.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.104+0000 7fa4bf400640 1 --2- 172.21.15.43:0/2712312567 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625eed21e70 0x5625eed28050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.105+0000 7fa4bf400640 1 -- 172.21.15.43:0/2712312567 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625eed21550 msgr2=0x5625eed27b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:38:06.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.105+0000 7fa4bea00640 1 --2- 172.21.15.43:0/2712312567 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625eed21550 0x5625eed27b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.105+0000 7fa4bf400640 1 --2- 172.21.15.43:0/2712312567 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625eed21550 0x5625eed27b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.105+0000 7fa4bf400640 1 -- 172.21.15.43:0/2712312567 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625eea637d0 con 0x5625eed2aa50 2024-03-28T22:38:06.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.105+0000 7fa4bea00640 1 --2- 172.21.15.43:0/2712312567 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625eed21550 0x5625eed27b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:38:06.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.105+0000 7fa4bf400640 1 --2- 172.21.15.43:0/1592078643 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa4a40a21a0 0x7fa4a40a4660 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8009de0 tx=0x7fa4b8009be0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:06.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.105+0000 7fa4bf400640 1 --2- 172.21.15.43:0/2712312567 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625eed2aa50 0x5625eed28590 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8009fd0 tx=0x7fa4b8009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50886fc198a1aba3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:06.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.106+0000 7fa49fe00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:38:06.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.106+0000 7fa49fe00640 1 -- 172.21.15.43:0/2712312567 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b8035060 con 0x5625eed2aa50 2024-03-28T22:38:06.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.106+0000 7fa4c8409740 1 -- 172.21.15.43:0/2712312567 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4b8004050 con 0x5625eed2aa50 2024-03-28T22:38:06.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.106+0000 7fa49fe00640 1 -- 172.21.15.43:0/2712312567 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fa4b8005920 con 0x5625eed2aa50 2024-03-28T22:38:06.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.106+0000 7fa49fe00640 1 -- 172.21.15.43:0/2712312567 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa4b800f4e0 con 0x5625eed2aa50 2024-03-28T22:38:06.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.106+0000 7fa4c8409740 10 client.25997 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:06.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.106+0000 7fa4c8409740 1 -- 172.21.15.43:0/2712312567 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5625eed30b30 con 0x5625eed2aa50 2024-03-28T22:38:06.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.107+0000 7fa49fe00640 1 -- 172.21.15.43:0/2712312567 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4b8004050 con 0x5625eed2aa50 2024-03-28T22:38:06.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.107+0000 7fa49fe00640 1 -- 172.21.15.43:0/2712312567 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fa4b80257b0 con 0x5625eed2aa50 2024-03-28T22:38:06.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.107+0000 7fa4c8409740 10 client.25997 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:06.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.107+0000 7fa4c8409740 10 client.25997 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:38:06.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.107+0000 7fa4c8409740 20 client.25997 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:06.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.107+0000 7fa4c8409740 1 --2- 172.21.15.43:0/2712312567 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5625eed31f40 0x5625eed343a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.107+0000 7fa4c8409740 4 client.25997 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:38:06.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.107+0000 7fa4c8409740 1 -- 172.21.15.43:0/2712312567 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625eec2e680 con 0x5625eed31f40 2024-03-28T22:38:06.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.107+0000 7fa4bea00640 1 --2- 172.21.15.43:0/2712312567 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5625eed31f40 0x5625eed343a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.108+0000 7fa4bea00640 1 --2- 172.21.15.43:0/2712312567 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5625eed31f40 0x5625eed343a0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c92f2bbdcb07fc21 server_cookie=f4380382c200d983 in_seq=0 out_seq=0 2024-03-28T22:38:06.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.108+0000 7fa49fe00640 10 client.25997 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:38:06.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.109+0000 7fa49fe00640 1 -- 172.21.15.43:0/2712312567 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5625eec2e680 con 0x5625eed31f40 2024-03-28T22:38:06.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.109+0000 7fa49fe00640 10 client.25997 handle_command_reply: tid=0 2024-03-28T22:38:06.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.109+0000 7fa4c8409740 1 client.25997 shutdown 2024-03-28T22:38:06.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.109+0000 7fa4c8409740 1 -- 172.21.15.43:0/2712312567 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625eed2aa50 msgr2=0x5625eed28590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.109+0000 7fa4c8409740 1 --2- 172.21.15.43:0/2712312567 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625eed2aa50 0x5625eed28590 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8009fd0 tx=0x7fa4b8009500 comp rx=0 tx=0).stop 2024-03-28T22:38:06.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.109+0000 7fa4c8409740 1 -- 172.21.15.43:0/2712312567 shutdown_connections 2024-03-28T22:38:06.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.109+0000 7fa4c8409740 1 --2- 172.21.15.43:0/2712312567 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5625eed31f40 0x5625eed343a0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.109+0000 7fa4c8409740 1 --2- 172.21.15.43:0/2712312567 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625eed2aa50 0x5625eed28590 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.109+0000 7fa4c8409740 1 --2- 172.21.15.43:0/2712312567 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625eed21e70 0x5625eed28050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.109+0000 7fa4c8409740 1 --2- 172.21.15.43:0/2712312567 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625eed21550 0x5625eed27b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.109+0000 7fa49fe00640 0 client.25997 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:38:06.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.109+0000 7fa4c8409740 1 -- 172.21.15.43:0/2712312567 >> 172.21.15.43:0/2712312567 conn(0x5625eed1bc80 msgr2=0x5625eed1d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:06.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.109+0000 7fa4c8409740 1 -- 172.21.15.43:0/2712312567 shutdown_connections 2024-03-28T22:38:06.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.109+0000 7fa4c8409740 1 -- 172.21.15.43:0/2712312567 wait complete. 2024-03-28T22:38:06.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.109+0000 7fa4c8409740 20 client.25997 trim_cache size 0 max 16384 2024-03-28T22:38:06.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.124+0000 7fa4c8409740 1 Processor -- start 2024-03-28T22:38:06.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.124+0000 7fa4c8409740 1 -- start start 2024-03-28T22:38:06.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.124+0000 7fa4c8409740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625eed21550 0x5625eed2a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.124+0000 7fa4c8409740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625eed21e70 0x5625eed26af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.124+0000 7fa4c8409740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625eed2aa50 0x5625eed55550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.124+0000 7fa4c8409740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c01ac900 con 0x5625eed2aa50 2024-03-28T22:38:06.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.124+0000 7fa4c8409740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c01ae440 con 0x5625eed21e70 2024-03-28T22:38:06.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.124+0000 7fa4c8409740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c01acfd0 con 0x5625eed21550 2024-03-28T22:38:06.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.124+0000 7fa4be000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625eed21e70 0x5625eed26af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.124+0000 7fa4bea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625eed21550 0x5625eed2a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.124+0000 7fa4bea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625eed21550 0x5625eed2a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57306/0 (socket says 172.21.15.43:57306) 2024-03-28T22:38:06.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.124+0000 7fa4bea00640 1 -- 172.21.15.43:0/3324392120 learned_addr learned my addr 172.21.15.43:0/3324392120 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:06.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.124+0000 7fa4be000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625eed21e70 0x5625eed26af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34110/0 (socket says 172.21.15.43:34110) 2024-03-28T22:38:06.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.125+0000 7fa4bf400640 1 --2- 172.21.15.43:0/3324392120 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625eed2aa50 0x5625eed55550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.125+0000 7fa4bea00640 1 -- 172.21.15.43:0/3324392120 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625eed21e70 msgr2=0x5625eed26af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:06.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.125+0000 7fa4bea00640 1 --2- 172.21.15.43:0/3324392120 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625eed21e70 0x5625eed26af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.125+0000 7fa4bea00640 1 -- 172.21.15.43:0/3324392120 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625eed2aa50 msgr2=0x5625eed55550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:06.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.125+0000 7fa4bea00640 1 --2- 172.21.15.43:0/3324392120 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625eed2aa50 0x5625eed55550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.125+0000 7fa4bea00640 1 -- 172.21.15.43:0/3324392120 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625eed21ca0 con 0x5625eed21550 2024-03-28T22:38:06.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.125+0000 7fa4be000640 1 --2- 172.21.15.43:0/3324392120 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625eed21e70 0x5625eed26af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:06.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.125+0000 7fa4bea00640 1 --2- 172.21.15.43:0/3324392120 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625eed21550 0x5625eed2a240 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa4b00058c0 tx=0x7fa4b0020000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50b9e898f7c4e0a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:06.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.125+0000 7fa4aaa00640 1 -- 172.21.15.43:0/3324392120 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b00a3630 con 0x5625eed21550 2024-03-28T22:38:06.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.125+0000 7fa4aaa00640 1 -- 172.21.15.43:0/3324392120 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4b0099190 con 0x5625eed21550 2024-03-28T22:38:06.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.125+0000 7fa4aaa00640 1 -- 172.21.15.43:0/3324392120 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b009a2e0 con 0x5625eed21550 2024-03-28T22:38:06.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.126+0000 7fa4c8409740 1 -- 172.21.15.43:0/3324392120 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625eed21550 msgr2=0x5625eed2a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.126+0000 7fa4c8409740 1 --2- 172.21.15.43:0/3324392120 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625eed21550 0x5625eed2a240 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa4b00058c0 tx=0x7fa4b0020000 comp rx=0 tx=0).stop 2024-03-28T22:38:06.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.126+0000 7fa4c8409740 1 -- 172.21.15.43:0/3324392120 shutdown_connections 2024-03-28T22:38:06.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.126+0000 7fa4c8409740 1 --2- 172.21.15.43:0/3324392120 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625eed21e70 0x5625eed26af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.126+0000 7fa4c8409740 1 --2- 172.21.15.43:0/3324392120 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625eed2aa50 0x5625eed55550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.126+0000 7fa4c8409740 1 --2- 172.21.15.43:0/3324392120 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625eed21550 0x5625eed2a240 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.126+0000 7fa4c8409740 1 -- 172.21.15.43:0/3324392120 >> 172.21.15.43:0/3324392120 conn(0x5625eed1bc80 msgr2=0x5625eed263e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:06.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.126+0000 7fa4c8409740 1 -- 172.21.15.43:0/3324392120 shutdown_connections 2024-03-28T22:38:06.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.127+0000 7fa4c8409740 1 -- 172.21.15.43:0/3324392120 wait complete. 2024-03-28T22:38:06.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.127+0000 7fa4c8409740 1 -- start start 2024-03-28T22:38:06.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.127+0000 7fa4c8409740 1 Processor -- start 2024-03-28T22:38:06.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.127+0000 7fa4c8409740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625eed21550 0x5625eed35740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.127+0000 7fa4c8409740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625eed21e70 0x5625eed35c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.127+0000 7fa4c8409740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625eed2aa50 0x5625eed36200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.128+0000 7fa4c8409740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c01add70 con 0x5625eed2aa50 2024-03-28T22:38:06.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.128+0000 7fa4c8409740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c01ad6a0 con 0x5625eed21550 2024-03-28T22:38:06.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.128+0000 7fa4c8409740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c01aa8e0 con 0x5625eed21e70 2024-03-28T22:38:06.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.128+0000 7fa4c8409740 10 client.0 _ll_register_callbacks cb 0x7fa4b0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:06.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.128+0000 7fa4be000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625eed21e70 0x5625eed35c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.128+0000 7fa4be000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625eed21e70 0x5625eed35c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57320/0 (socket says 172.21.15.43:57320) 2024-03-28T22:38:06.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.128+0000 7fa4be000640 1 -- 172.21.15.43:0/910573704 learned_addr learned my addr 172.21.15.43:0/910573704 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:06.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.128+0000 7fa4bf400640 1 --2- 172.21.15.43:0/910573704 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625eed2aa50 0x5625eed36200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.128+0000 7fa4bea00640 1 --2- 172.21.15.43:0/910573704 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625eed21550 0x5625eed35740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.128+0000 7fa4be000640 1 -- 172.21.15.43:0/910573704 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625eed21550 msgr2=0x5625eed35740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:06.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.128+0000 7fa4be000640 1 --2- 172.21.15.43:0/910573704 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625eed21550 0x5625eed35740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.128+0000 7fa4be000640 1 -- 172.21.15.43:0/910573704 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625eed2aa50 msgr2=0x5625eed36200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:06.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.128+0000 7fa4be000640 1 --2- 172.21.15.43:0/910573704 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625eed2aa50 0x5625eed36200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.128+0000 7fa4be000640 1 -- 172.21.15.43:0/910573704 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625eedb0e60 con 0x5625eed21e70 2024-03-28T22:38:06.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.128+0000 7fa4bf400640 1 --2- 172.21.15.43:0/910573704 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625eed2aa50 0x5625eed36200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:06.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.129+0000 7fa4be000640 1 --2- 172.21.15.43:0/910573704 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625eed21e70 0x5625eed35c80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac005c40 tx=0x7fa4ac02c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9c4e85ff5d9666 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:06.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.129+0000 7fa49fe00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:38:06.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.129+0000 7fa49fe00640 1 -- 172.21.15.43:0/910573704 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac02cd40 con 0x5625eed21e70 2024-03-28T22:38:06.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.129+0000 7fa4c8409740 1 -- 172.21.15.43:0/910573704 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4b0099190 con 0x5625eed21e70 2024-03-28T22:38:06.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.129+0000 7fa49fe00640 1 -- 172.21.15.43:0/910573704 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fa4ac005040 con 0x5625eed21e70 2024-03-28T22:38:06.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.130+0000 7fa49fe00640 1 -- 172.21.15.43:0/910573704 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa4ac04e050 con 0x5625eed21e70 2024-03-28T22:38:06.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.130+0000 7fa4c8409740 10 client.25948 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:06.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.130+0000 7fa4c8409740 1 -- 172.21.15.43:0/910573704 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5625eedb2160 con 0x5625eed21e70 2024-03-28T22:38:06.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.130+0000 7fa49fe00640 1 -- 172.21.15.43:0/910573704 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4b0099190 con 0x5625eed21e70 2024-03-28T22:38:06.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.130+0000 7fa49fe00640 1 -- 172.21.15.43:0/910573704 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fa4ac0035f0 con 0x5625eed21e70 2024-03-28T22:38:06.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.130+0000 7fa4c8409740 10 client.25948 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:06.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.130+0000 7fa4c8409740 10 client.25948 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:38:06.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.130+0000 7fa4c8409740 20 client.25948 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:06.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.130+0000 7fa4c8409740 1 --2- 172.21.15.43:0/910573704 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5625eedb2980 0x5625eedb4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.130+0000 7fa4c8409740 4 client.25948 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:38:06.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.130+0000 7fa4c8409740 1 -- 172.21.15.43:0/910573704 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5625eec92590 con 0x5625eedb2980 2024-03-28T22:38:06.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.131+0000 7fa4bea00640 1 --2- 172.21.15.43:0/910573704 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5625eedb2980 0x5625eedb4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.131+0000 7fa4bea00640 1 --2- 172.21.15.43:0/910573704 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5625eedb2980 0x5625eedb4da0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1aa0b4aaeee81765 server_cookie=40357c41ee0cb7e4 in_seq=0 out_seq=0 2024-03-28T22:38:06.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.131+0000 7fa49fe00640 10 client.25948 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:38:06.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.131+0000 7fa49fe00640 1 -- 172.21.15.43:0/910573704 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5625eec92590 con 0x5625eedb2980 2024-03-28T22:38:06.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.131+0000 7fa49fe00640 10 client.25948 handle_command_reply: tid=0 2024-03-28T22:38:06.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.131+0000 7fa4c8409740 1 client.25948 shutdown 2024-03-28T22:38:06.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.131+0000 7fa4c8409740 1 -- 172.21.15.43:0/910573704 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625eed21e70 msgr2=0x5625eed35c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.131+0000 7fa4c8409740 1 --2- 172.21.15.43:0/910573704 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625eed21e70 0x5625eed35c80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac005c40 tx=0x7fa4ac02c800 comp rx=0 tx=0).stop 2024-03-28T22:38:06.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.132+0000 7fa4c8409740 1 -- 172.21.15.43:0/910573704 shutdown_connections 2024-03-28T22:38:06.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.132+0000 7fa4c8409740 1 --2- 172.21.15.43:0/910573704 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5625eedb2980 0x5625eedb4da0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.132+0000 7fa4c8409740 1 --2- 172.21.15.43:0/910573704 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625eed2aa50 0x5625eed36200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.132+0000 7fa4c8409740 1 --2- 172.21.15.43:0/910573704 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625eed21e70 0x5625eed35c80 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.132+0000 7fa4c8409740 1 --2- 172.21.15.43:0/910573704 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625eed21550 0x5625eed35740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.132+0000 7fa49fe00640 0 client.25948 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:38:06.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.132+0000 7fa4c8409740 1 -- 172.21.15.43:0/910573704 >> 172.21.15.43:0/910573704 conn(0x5625eed1bc80 msgr2=0x5625eed26c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:06.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.132+0000 7fa4c8409740 1 -- 172.21.15.43:0/910573704 shutdown_connections 2024-03-28T22:38:06.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.132+0000 7fa4c8409740 1 -- 172.21.15.43:0/910573704 wait complete. 2024-03-28T22:38:06.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.132+0000 7fa4c8409740 20 client.25948 trim_cache size 0 max 16384 2024-03-28T22:38:06.135 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:38:06.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.133+0000 7fa4bfe00640 1 -- 172.21.15.43:0/1592078643 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa4a40a21a0 msgr2=0x7fa4a40a4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.133+0000 7fa4bfe00640 1 --2- 172.21.15.43:0/1592078643 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa4a40a21a0 0x7fa4a40a4660 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8009de0 tx=0x7fa4b8009be0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.133+0000 7fa4bfe00640 1 -- 172.21.15.43:0/1592078643 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa4c01015c0 msgr2=0x7fa4c019a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.133+0000 7fa4bfe00640 1 --2- 172.21.15.43:0/1592078643 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa4c01015c0 0x7fa4c019a050 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa4b000e9c0 tx=0x7fa4b000ee80 comp rx=0 tx=0).stop 2024-03-28T22:38:06.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.133+0000 7fa4bfe00640 1 -- 172.21.15.43:0/1592078643 shutdown_connections 2024-03-28T22:38:06.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.133+0000 7fa4bfe00640 1 --2- 172.21.15.43:0/1592078643 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa4a40a21a0 0x7fa4a40a4660 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.133+0000 7fa4bfe00640 1 --2- 172.21.15.43:0/1592078643 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4c0107660 0x7fa4c019e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.133+0000 7fa4bfe00640 1 --2- 172.21.15.43:0/1592078643 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4c0101f80 0x7fa4c019a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.133+0000 7fa4bfe00640 1 --2- 172.21.15.43:0/1592078643 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa4c01015c0 0x7fa4c019a050 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.133+0000 7fa4bfe00640 1 -- 172.21.15.43:0/1592078643 >> 172.21.15.43:0/1592078643 conn(0x7fa4c00fd0f0 msgr2=0x7fa4c00fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:06.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.133+0000 7fa4bfe00640 1 -- 172.21.15.43:0/1592078643 shutdown_connections 2024-03-28T22:38:06.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.133+0000 7fa4bfe00640 1 -- 172.21.15.43:0/1592078643 wait complete. 2024-03-28T22:38:06.150 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:38:06.225 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:06 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1857160756' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:38:06.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.350+0000 7fe92de00640 1 -- 172.21.15.43:0/4121451184 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9281015c0 msgr2=0x7fe928101a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.350+0000 7fe92de00640 1 --2- 172.21.15.43:0/4121451184 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9281015c0 0x7fe928101a40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe914009930 tx=0x7fe91402bc20 comp rx=0 tx=0).stop 2024-03-28T22:38:06.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.350+0000 7fe92de00640 1 -- 172.21.15.43:0/4121451184 shutdown_connections 2024-03-28T22:38:06.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.350+0000 7fe92de00640 1 --2- 172.21.15.43:0/4121451184 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe928101f80 0x7fe92810e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.350+0000 7fe92de00640 1 --2- 172.21.15.43:0/4121451184 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9281015c0 0x7fe928101a40 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.350+0000 7fe92de00640 1 --2- 172.21.15.43:0/4121451184 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe928107660 0x7fe928107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.350+0000 7fe92de00640 1 -- 172.21.15.43:0/4121451184 >> 172.21.15.43:0/4121451184 conn(0x7fe9280fd0f0 msgr2=0x7fe9280ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:06.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.350+0000 7fe92de00640 1 -- 172.21.15.43:0/4121451184 shutdown_connections 2024-03-28T22:38:06.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.350+0000 7fe92de00640 1 -- 172.21.15.43:0/4121451184 wait complete. 2024-03-28T22:38:06.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.351+0000 7fe92de00640 1 Processor -- start 2024-03-28T22:38:06.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.351+0000 7fe92de00640 1 -- start start 2024-03-28T22:38:06.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.351+0000 7fe92de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe9281015c0 0x7fe92819a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.351+0000 7fe92de00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe928101f80 0x7fe92819a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.352+0000 7fe92de00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe928107660 0x7fe92819e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.352+0000 7fe92de00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe9281119f0 con 0x7fe9281015c0 2024-03-28T22:38:06.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.352+0000 7fe92de00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe928111870 con 0x7fe928101f80 2024-03-28T22:38:06.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.352+0000 7fe92de00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe928111b70 con 0x7fe928107660 2024-03-28T22:38:06.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.352+0000 7fe927e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe928101f80 0x7fe92819a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.352+0000 7fe927e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe928101f80 0x7fe92819a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34126/0 (socket says 172.21.15.43:34126) 2024-03-28T22:38:06.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.352+0000 7fe927e00640 1 -- 172.21.15.43:0/3286949661 learned_addr learned my addr 172.21.15.43:0/3286949661 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:06.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.352+0000 7fe92d400640 1 --2- 172.21.15.43:0/3286949661 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe928107660 0x7fe92819e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.352+0000 7fe92ca00640 1 --2- 172.21.15.43:0/3286949661 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe9281015c0 0x7fe92819a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.352+0000 7fe927e00640 1 -- 172.21.15.43:0/3286949661 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe928107660 msgr2=0x7fe92819e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.352+0000 7fe927e00640 1 --2- 172.21.15.43:0/3286949661 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe928107660 0x7fe92819e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.352+0000 7fe927e00640 1 -- 172.21.15.43:0/3286949661 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe9281015c0 msgr2=0x7fe92819a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.352+0000 7fe927e00640 1 --2- 172.21.15.43:0/3286949661 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe9281015c0 0x7fe92819a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.352+0000 7fe927e00640 1 -- 172.21.15.43:0/3286949661 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe92819f040 con 0x7fe928101f80 2024-03-28T22:38:06.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.352+0000 7fe92d400640 1 --2- 172.21.15.43:0/3286949661 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe928107660 0x7fe92819e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:06.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.353+0000 7fe927e00640 1 --2- 172.21.15.43:0/3286949661 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe928101f80 0x7fe92819a590 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe914005ab0 tx=0x7fe9140042e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:06.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.353+0000 7fe925600640 1 -- 172.21.15.43:0/3286949661 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe914043020 con 0x7fe928101f80 2024-03-28T22:38:06.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.353+0000 7fe925600640 1 -- 172.21.15.43:0/3286949661 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9140044d0 con 0x7fe928101f80 2024-03-28T22:38:06.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.353+0000 7fe925600640 1 -- 172.21.15.43:0/3286949661 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe914003bb0 con 0x7fe928101f80 2024-03-28T22:38:06.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.353+0000 7fe92de00640 1 -- 172.21.15.43:0/3286949661 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe92819f2d0 con 0x7fe928101f80 2024-03-28T22:38:06.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.353+0000 7fe92de00640 1 -- 172.21.15.43:0/3286949661 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9281a6bc0 con 0x7fe928101f80 2024-03-28T22:38:06.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.358+0000 7fe925600640 1 -- 172.21.15.43:0/3286949661 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 27) ==== 136989+0+0 (secure 0 0 0) 0x7fe914046020 con 0x7fe928101f80 2024-03-28T22:38:06.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.358+0000 7fe9321a5740 1 Processor -- start 2024-03-28T22:38:06.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.358+0000 7fe9321a5740 1 -- start start 2024-03-28T22:38:06.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.359+0000 7fe9321a5740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55790d401360 0x55790d401740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.359+0000 7fe9321a5740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55790d401d10 0x55790d412d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.359+0000 7fe9321a5740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55790d413260 0x55790d415650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.359+0000 7fe9321a5740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55790d365050 con 0x55790d401d10 2024-03-28T22:38:06.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.359+0000 7fe9321a5740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55790d30dce0 con 0x55790d401360 2024-03-28T22:38:06.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.359+0000 7fe9321a5740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55790d3610f0 con 0x55790d413260 2024-03-28T22:38:06.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.359+0000 7fe92ca00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55790d401d10 0x55790d412d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.359+0000 7fe92d400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55790d401360 0x55790d401740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.359+0000 7fe92d400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55790d401360 0x55790d401740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34132/0 (socket says 172.21.15.43:34132) 2024-03-28T22:38:06.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.359+0000 7fe92d400640 1 -- 172.21.15.43:0/1970807749 learned_addr learned my addr 172.21.15.43:0/1970807749 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:06.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.359+0000 7fe927e00640 1 --2- 172.21.15.43:0/1970807749 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55790d413260 0x55790d415650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.360+0000 7fe92d400640 1 -- 172.21.15.43:0/1970807749 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55790d413260 msgr2=0x55790d415650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:06.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.360+0000 7fe92d400640 1 --2- 172.21.15.43:0/1970807749 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55790d413260 0x55790d415650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.360+0000 7fe92d400640 1 -- 172.21.15.43:0/1970807749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55790d401d10 msgr2=0x55790d412d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:06.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.360+0000 7fe92d400640 1 --2- 172.21.15.43:0/1970807749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55790d401d10 0x55790d412d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.360+0000 7fe92d400640 1 -- 172.21.15.43:0/1970807749 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55790d0793f0 con 0x55790d401360 2024-03-28T22:38:06.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.360+0000 7fe927e00640 1 --2- 172.21.15.43:0/1970807749 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55790d413260 0x55790d415650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:38:06.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.360+0000 7fe92ca00640 1 --2- 172.21.15.43:0/1970807749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55790d401d10 0x55790d412d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:06.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.361+0000 7fe92d400640 1 --2- 172.21.15.43:0/1970807749 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55790d401360 0x55790d401740 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe920011810 tx=0x7fe920011cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f3c02197c6d72a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:06.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.361+0000 7fe919c00640 1 -- 172.21.15.43:0/1970807749 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe920011310 con 0x55790d401360 2024-03-28T22:38:06.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.361+0000 7fe919c00640 1 -- 172.21.15.43:0/1970807749 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe920003e50 con 0x55790d401360 2024-03-28T22:38:06.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.361+0000 7fe919c00640 1 -- 172.21.15.43:0/1970807749 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9200045b0 con 0x55790d401360 2024-03-28T22:38:06.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.361+0000 7fe925600640 1 --2- 172.21.15.43:0/3286949661 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe9100a2010 0x7fe9100a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.361+0000 7fe925600640 1 -- 172.21.15.43:0/3286949661 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fe9140f0710 con 0x7fe928101f80 2024-03-28T22:38:06.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.362+0000 7fe92ca00640 1 --2- 172.21.15.43:0/3286949661 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe9100a2010 0x7fe9100a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.362+0000 7fe92ca00640 1 --2- 172.21.15.43:0/3286949661 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe9100a2010 0x7fe9100a44d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x55790d3fcf10 tx=0x7fe91c00b4d0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:06.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.362+0000 7fe9321a5740 1 -- 172.21.15.43:0/1970807749 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55790d401360 msgr2=0x55790d401740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.363+0000 7fe9321a5740 1 --2- 172.21.15.43:0/1970807749 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55790d401360 0x55790d401740 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe920011810 tx=0x7fe920011cd0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.363+0000 7fe9321a5740 1 -- 172.21.15.43:0/1970807749 shutdown_connections 2024-03-28T22:38:06.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.363+0000 7fe9321a5740 1 --2- 172.21.15.43:0/1970807749 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55790d413260 0x55790d415650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.363+0000 7fe9321a5740 1 --2- 172.21.15.43:0/1970807749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55790d401d10 0x55790d412d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.363+0000 7fe9321a5740 1 --2- 172.21.15.43:0/1970807749 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55790d401360 0x55790d401740 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.363+0000 7fe9321a5740 1 -- 172.21.15.43:0/1970807749 >> 172.21.15.43:0/1970807749 conn(0x55790d3fbc80 msgr2=0x55790d3fc060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:06.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.363+0000 7fe9321a5740 1 -- 172.21.15.43:0/1970807749 shutdown_connections 2024-03-28T22:38:06.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.363+0000 7fe9321a5740 1 -- 172.21.15.43:0/1970807749 wait complete. 2024-03-28T22:38:06.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.363+0000 7fe9321a5740 1 -- start start 2024-03-28T22:38:06.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.364+0000 7fe9321a5740 1 Processor -- start 2024-03-28T22:38:06.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.364+0000 7fe9321a5740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55790d401360 0x55790d40cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.364+0000 7fe9321a5740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55790d401d10 0x55790d412cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.364+0000 7fe9321a5740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55790d413260 0x55790d4083d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.364+0000 7fe9321a5740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55790d4028b0 con 0x55790d413260 2024-03-28T22:38:06.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.364+0000 7fe9321a5740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55790d3032f0 con 0x55790d401d10 2024-03-28T22:38:06.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.364+0000 7fe9321a5740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55790d402a30 con 0x55790d401360 2024-03-28T22:38:06.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.364+0000 7fe9321a5740 10 client.0 _ll_register_callbacks cb 0x55790d078be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:06.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.364+0000 7fe92ca00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55790d401360 0x55790d40cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.364+0000 7fe92ca00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55790d401360 0x55790d40cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57352/0 (socket says 172.21.15.43:57352) 2024-03-28T22:38:06.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.364+0000 7fe92ca00640 1 -- 172.21.15.43:0/1886297930 learned_addr learned my addr 172.21.15.43:0/1886297930 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:06.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.364+0000 7fe927e00640 1 --2- 172.21.15.43:0/1886297930 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55790d401d10 0x55790d412cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.365+0000 7fe92d400640 1 --2- 172.21.15.43:0/1886297930 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55790d413260 0x55790d4083d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.365+0000 7fe92ca00640 1 -- 172.21.15.43:0/1886297930 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55790d401d10 msgr2=0x55790d412cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:06.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.365+0000 7fe92ca00640 1 --2- 172.21.15.43:0/1886297930 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55790d401d10 0x55790d412cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.365+0000 7fe92ca00640 1 -- 172.21.15.43:0/1886297930 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55790d413260 msgr2=0x55790d4083d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:06.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.365+0000 7fe92ca00640 1 --2- 172.21.15.43:0/1886297930 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55790d413260 0x55790d4083d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.365+0000 7fe92ca00640 1 -- 172.21.15.43:0/1886297930 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55790d1437d0 con 0x55790d401360 2024-03-28T22:38:06.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.365+0000 7fe92ca00640 1 --2- 172.21.15.43:0/1886297930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55790d401360 0x55790d40cfd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe91c010700 tx=0x7fe91c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=469c0b04c2bfcbd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:06.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.365+0000 7fe90e000640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:38:06.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.365+0000 7fe90e000640 1 -- 172.21.15.43:0/1886297930 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe91c01f8c0 con 0x55790d401360 2024-03-28T22:38:06.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.365+0000 7fe9321a5740 1 -- 172.21.15.43:0/1886297930 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe920003e50 con 0x55790d401360 2024-03-28T22:38:06.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.365+0000 7fe90e000640 1 -- 172.21.15.43:0/1886297930 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fe91c01fa60 con 0x55790d401360 2024-03-28T22:38:06.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.365+0000 7fe90e000640 1 -- 172.21.15.43:0/1886297930 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe91c030050 con 0x55790d401360 2024-03-28T22:38:06.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.366+0000 7fe9321a5740 10 client.25960 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:06.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.366+0000 7fe9321a5740 1 -- 172.21.15.43:0/1886297930 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55790d42ac90 con 0x55790d401360 2024-03-28T22:38:06.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.366+0000 7fe90e000640 1 -- 172.21.15.43:0/1886297930 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe920003e50 con 0x55790d401360 2024-03-28T22:38:06.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.366+0000 7fe90e000640 1 -- 172.21.15.43:0/1886297930 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe91c01e380 con 0x55790d401360 2024-03-28T22:38:06.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.366+0000 7fe9321a5740 10 client.25960 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:06.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.366+0000 7fe9321a5740 10 client.25960 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:38:06.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.366+0000 7fe9321a5740 20 client.25960 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:06.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.366+0000 7fe9321a5740 1 --2- 172.21.15.43:0/1886297930 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55790d42c0a0 0x55790d42e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.366+0000 7fe9321a5740 4 client.25960 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:38:06.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.366+0000 7fe9321a5740 1 -- 172.21.15.43:0/1886297930 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55790d30e680 con 0x55790d42c0a0 2024-03-28T22:38:06.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.366+0000 7fe927e00640 1 --2- 172.21.15.43:0/1886297930 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55790d42c0a0 0x55790d42e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.367+0000 7fe927e00640 1 --2- 172.21.15.43:0/1886297930 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55790d42c0a0 0x55790d42e510 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f9a2a4396310b752 server_cookie=b58a556843b3f11d in_seq=0 out_seq=0 2024-03-28T22:38:06.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.367+0000 7fe90e000640 10 client.25960 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:38:06.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.368+0000 7fe90e000640 1 -- 172.21.15.43:0/1886297930 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55790d30e680 con 0x55790d42c0a0 2024-03-28T22:38:06.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.368+0000 7fe90e000640 10 client.25960 handle_command_reply: tid=0 2024-03-28T22:38:06.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.368+0000 7fe9321a5740 1 client.25960 shutdown 2024-03-28T22:38:06.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.368+0000 7fe9321a5740 1 -- 172.21.15.43:0/1886297930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55790d401360 msgr2=0x55790d40cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.368+0000 7fe9321a5740 1 --2- 172.21.15.43:0/1886297930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55790d401360 0x55790d40cfd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe91c010700 tx=0x7fe91c0046c0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.368+0000 7fe9321a5740 1 -- 172.21.15.43:0/1886297930 shutdown_connections 2024-03-28T22:38:06.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.368+0000 7fe9321a5740 1 --2- 172.21.15.43:0/1886297930 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55790d42c0a0 0x55790d42e510 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.368+0000 7fe9321a5740 1 --2- 172.21.15.43:0/1886297930 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55790d401d10 0x55790d412cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.368+0000 7fe9321a5740 1 --2- 172.21.15.43:0/1886297930 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55790d413260 0x55790d4083d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.368+0000 7fe9321a5740 1 --2- 172.21.15.43:0/1886297930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55790d401360 0x55790d40cfd0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.368+0000 7fe9321a5740 1 -- 172.21.15.43:0/1886297930 >> 172.21.15.43:0/1886297930 conn(0x55790d3fbc80 msgr2=0x55790d3fd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:06.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.368+0000 7fe90e000640 0 client.25960 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:38:06.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.369+0000 7fe9321a5740 1 -- 172.21.15.43:0/1886297930 shutdown_connections 2024-03-28T22:38:06.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.369+0000 7fe9321a5740 1 -- 172.21.15.43:0/1886297930 wait complete. 2024-03-28T22:38:06.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.369+0000 7fe9321a5740 20 client.25960 trim_cache size 0 max 16384 2024-03-28T22:38:06.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.396+0000 7fe9321a5740 1 Processor -- start 2024-03-28T22:38:06.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.396+0000 7fe9321a5740 1 -- start start 2024-03-28T22:38:06.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.396+0000 7fe9321a5740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55790d401360 0x55790d3fac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.396+0000 7fe9321a5740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55790d401d10 0x55790d433de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.396+0000 7fe9321a5740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55790d413260 0x55790d43f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.396+0000 7fe9321a5740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe9281ac840 con 0x55790d413260 2024-03-28T22:38:06.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.396+0000 7fe9321a5740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe9281ae380 con 0x55790d401d10 2024-03-28T22:38:06.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.396+0000 7fe9321a5740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe9281acf10 con 0x55790d401360 2024-03-28T22:38:06.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:06 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1857160756' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:38:06.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.396+0000 7fe927e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55790d401d10 0x55790d433de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.396+0000 7fe92ca00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55790d401360 0x55790d3fac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.397+0000 7fe92ca00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55790d401360 0x55790d3fac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57356/0 (socket says 172.21.15.43:57356) 2024-03-28T22:38:06.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.397+0000 7fe92ca00640 1 -- 172.21.15.43:0/1282951857 learned_addr learned my addr 172.21.15.43:0/1282951857 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:06.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.397+0000 7fe92d400640 1 --2- 172.21.15.43:0/1282951857 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55790d413260 0x55790d43f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.397+0000 7fe927e00640 1 -- 172.21.15.43:0/1282951857 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55790d401360 msgr2=0x55790d3fac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:06.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.397+0000 7fe927e00640 1 --2- 172.21.15.43:0/1282951857 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55790d401360 0x55790d3fac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.397+0000 7fe927e00640 1 -- 172.21.15.43:0/1282951857 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55790d413260 msgr2=0x55790d43f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:06.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.397+0000 7fe927e00640 1 --2- 172.21.15.43:0/1282951857 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55790d413260 0x55790d43f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.397+0000 7fe927e00640 1 -- 172.21.15.43:0/1282951857 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55790d4159c0 con 0x55790d401d10 2024-03-28T22:38:06.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.397+0000 7fe927e00640 1 --2- 172.21.15.43:0/1282951857 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55790d401d10 0x55790d433de0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe9140eda50 tx=0x7fe914003680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66f824513e51a447 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:06.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.398+0000 7fe919200640 1 -- 172.21.15.43:0/1282951857 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9140bd320 con 0x55790d401d10 2024-03-28T22:38:06.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.398+0000 7fe919200640 1 -- 172.21.15.43:0/1282951857 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9140bcc20 con 0x55790d401d10 2024-03-28T22:38:06.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.398+0000 7fe919200640 1 -- 172.21.15.43:0/1282951857 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9140b9290 con 0x55790d401d10 2024-03-28T22:38:06.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.399+0000 7fe9321a5740 1 -- 172.21.15.43:0/1282951857 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55790d401d10 msgr2=0x55790d433de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.399+0000 7fe9321a5740 1 --2- 172.21.15.43:0/1282951857 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55790d401d10 0x55790d433de0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe9140eda50 tx=0x7fe914003680 comp rx=0 tx=0).stop 2024-03-28T22:38:06.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.399+0000 7fe9321a5740 1 -- 172.21.15.43:0/1282951857 shutdown_connections 2024-03-28T22:38:06.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.399+0000 7fe9321a5740 1 --2- 172.21.15.43:0/1282951857 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55790d401d10 0x55790d433de0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.399+0000 7fe9321a5740 1 --2- 172.21.15.43:0/1282951857 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55790d413260 0x55790d43f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.399+0000 7fe9321a5740 1 --2- 172.21.15.43:0/1282951857 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55790d401360 0x55790d3fac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.399+0000 7fe9321a5740 1 -- 172.21.15.43:0/1282951857 >> 172.21.15.43:0/1282951857 conn(0x55790d3fbc80 msgr2=0x55790d3fa530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:06.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.399+0000 7fe9321a5740 1 -- 172.21.15.43:0/1282951857 shutdown_connections 2024-03-28T22:38:06.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.399+0000 7fe9321a5740 1 -- 172.21.15.43:0/1282951857 wait complete. 2024-03-28T22:38:06.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.399+0000 7fe9321a5740 1 -- start start 2024-03-28T22:38:06.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.400+0000 7fe9321a5740 1 Processor -- start 2024-03-28T22:38:06.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.400+0000 7fe9321a5740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55790d401360 0x55790d405550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.400+0000 7fe9321a5740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55790d401d10 0x55790d405a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.400+0000 7fe9321a5740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55790d413260 0x55790d490dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.400+0000 7fe9321a5740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe9281adcb0 con 0x55790d413260 2024-03-28T22:38:06.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.400+0000 7fe9321a5740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe9281ad5e0 con 0x55790d401d10 2024-03-28T22:38:06.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.400+0000 7fe9321a5740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe9281aa8a0 con 0x55790d401360 2024-03-28T22:38:06.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.400+0000 7fe9321a5740 10 client.0 _ll_register_callbacks cb 0x7fe91c0103d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:06.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.401+0000 7fe92ca00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55790d401360 0x55790d405550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.401+0000 7fe92ca00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55790d401360 0x55790d405550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57364/0 (socket says 172.21.15.43:57364) 2024-03-28T22:38:06.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.401+0000 7fe92ca00640 1 -- 172.21.15.43:0/3727106620 learned_addr learned my addr 172.21.15.43:0/3727106620 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:06.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.401+0000 7fe927e00640 1 --2- 172.21.15.43:0/3727106620 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55790d401d10 0x55790d405a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.401+0000 7fe92ca00640 1 -- 172.21.15.43:0/3727106620 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55790d401d10 msgr2=0x55790d405a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:06.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.401+0000 7fe92ca00640 1 --2- 172.21.15.43:0/3727106620 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55790d401d10 0x55790d405a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.401+0000 7fe92ca00640 1 -- 172.21.15.43:0/3727106620 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55790d413260 msgr2=0x55790d490dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:06.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.401+0000 7fe92ca00640 1 --2- 172.21.15.43:0/3727106620 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55790d413260 0x55790d490dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.401+0000 7fe92ca00640 1 -- 172.21.15.43:0/3727106620 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55790d491310 con 0x55790d401360 2024-03-28T22:38:06.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.401+0000 7fe92ca00640 1 --2- 172.21.15.43:0/3727106620 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55790d401360 0x55790d405550 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe91c00bfd0 tx=0x7fe91c01e630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f559b12dd19ee49 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:06.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.401+0000 7fe90e000640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:38:06.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.402+0000 7fe90e000640 1 -- 172.21.15.43:0/3727106620 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe91c02d040 con 0x55790d401360 2024-03-28T22:38:06.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.402+0000 7fe90e000640 1 -- 172.21.15.43:0/3727106620 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fe91c005020 con 0x55790d401360 2024-03-28T22:38:06.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.402+0000 7fe9321a5740 1 -- 172.21.15.43:0/3727106620 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9140bcc20 con 0x55790d401360 2024-03-28T22:38:06.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.402+0000 7fe90e000640 1 -- 172.21.15.43:0/3727106620 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe91c048050 con 0x55790d401360 2024-03-28T22:38:06.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.402+0000 7fe9321a5740 10 client.25966 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:06.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.402+0000 7fe9321a5740 1 -- 172.21.15.43:0/3727106620 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55790d4924d0 con 0x55790d401360 2024-03-28T22:38:06.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.402+0000 7fe90e000640 1 -- 172.21.15.43:0/3727106620 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9140bcc20 con 0x55790d401360 2024-03-28T22:38:06.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.403+0000 7fe90e000640 1 -- 172.21.15.43:0/3727106620 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe91c01f6c0 con 0x55790d401360 2024-03-28T22:38:06.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.403+0000 7fe9321a5740 10 client.25966 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:06.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.403+0000 7fe9321a5740 10 client.25966 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:38:06.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.403+0000 7fe9321a5740 20 client.25966 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:06.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.403+0000 7fe9321a5740 1 --2- 172.21.15.43:0/3727106620 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55790d492bf0 0x55790d495060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.403+0000 7fe9321a5740 4 client.25966 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:38:06.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.403+0000 7fe9321a5740 1 -- 172.21.15.43:0/3727106620 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55790d372590 con 0x55790d492bf0 2024-03-28T22:38:06.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.403+0000 7fe927e00640 1 --2- 172.21.15.43:0/3727106620 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55790d492bf0 0x55790d495060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.404+0000 7fe927e00640 1 --2- 172.21.15.43:0/3727106620 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55790d492bf0 0x55790d495060 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=163d41d5c665e253 server_cookie=1d630ff5afcddee8 in_seq=0 out_seq=0 2024-03-28T22:38:06.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.404+0000 7fe90e000640 10 client.25966 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:38:06.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.404+0000 7fe90e000640 1 -- 172.21.15.43:0/3727106620 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55790d372590 con 0x55790d492bf0 2024-03-28T22:38:06.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.404+0000 7fe90e000640 10 client.25966 handle_command_reply: tid=0 2024-03-28T22:38:06.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.404+0000 7fe9321a5740 1 client.25966 shutdown 2024-03-28T22:38:06.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.404+0000 7fe9321a5740 1 -- 172.21.15.43:0/3727106620 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55790d401360 msgr2=0x55790d405550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.404+0000 7fe9321a5740 1 --2- 172.21.15.43:0/3727106620 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55790d401360 0x55790d405550 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe91c00bfd0 tx=0x7fe91c01e630 comp rx=0 tx=0).stop 2024-03-28T22:38:06.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.404+0000 7fe9321a5740 1 -- 172.21.15.43:0/3727106620 shutdown_connections 2024-03-28T22:38:06.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.404+0000 7fe9321a5740 1 --2- 172.21.15.43:0/3727106620 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55790d492bf0 0x55790d495060 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.404+0000 7fe9321a5740 1 --2- 172.21.15.43:0/3727106620 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55790d401d10 0x55790d405a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.404+0000 7fe9321a5740 1 --2- 172.21.15.43:0/3727106620 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55790d413260 0x55790d490dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.404+0000 7fe9321a5740 1 --2- 172.21.15.43:0/3727106620 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55790d401360 0x55790d405550 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.404+0000 7fe9321a5740 1 -- 172.21.15.43:0/3727106620 >> 172.21.15.43:0/3727106620 conn(0x55790d3fbc80 msgr2=0x55790d3fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:06.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.404+0000 7fe90e000640 0 client.25966 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:38:06.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.404+0000 7fe9321a5740 1 -- 172.21.15.43:0/3727106620 shutdown_connections 2024-03-28T22:38:06.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.405+0000 7fe9321a5740 1 -- 172.21.15.43:0/3727106620 wait complete. 2024-03-28T22:38:06.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.405+0000 7fe9321a5740 20 client.25966 trim_cache size 0 max 16384 2024-03-28T22:38:06.406 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:38:06.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.405+0000 7fe92de00640 1 -- 172.21.15.43:0/3286949661 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe9100a2010 msgr2=0x7fe9100a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.405+0000 7fe92de00640 1 --2- 172.21.15.43:0/3286949661 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe9100a2010 0x7fe9100a44d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x55790d3fcf10 tx=0x7fe91c00b4d0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.406+0000 7fe92de00640 1 -- 172.21.15.43:0/3286949661 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe928101f80 msgr2=0x7fe92819a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.406+0000 7fe92de00640 1 --2- 172.21.15.43:0/3286949661 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe928101f80 0x7fe92819a590 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe914005ab0 tx=0x7fe9140042e0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.406+0000 7fe92de00640 1 -- 172.21.15.43:0/3286949661 shutdown_connections 2024-03-28T22:38:06.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.406+0000 7fe92de00640 1 --2- 172.21.15.43:0/3286949661 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe9100a2010 0x7fe9100a44d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.406+0000 7fe92de00640 1 --2- 172.21.15.43:0/3286949661 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe928101f80 0x7fe92819a590 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.406+0000 7fe92de00640 1 --2- 172.21.15.43:0/3286949661 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe928107660 0x7fe92819e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.406+0000 7fe92de00640 1 --2- 172.21.15.43:0/3286949661 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe9281015c0 0x7fe92819a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.406+0000 7fe92de00640 1 -- 172.21.15.43:0/3286949661 >> 172.21.15.43:0/3286949661 conn(0x7fe9280fd0f0 msgr2=0x7fe92810d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:06.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.406+0000 7fe92de00640 1 -- 172.21.15.43:0/3286949661 shutdown_connections 2024-03-28T22:38:06.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.406+0000 7fe92de00640 1 -- 172.21.15.43:0/3286949661 wait complete. 2024-03-28T22:38:06.428 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:38:06.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:06 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1857160756' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:38:06.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.610+0000 7f86d8e00640 1 -- 172.21.15.43:0/2700585059 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86d4074440 msgr2=0x7f86d410d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.610+0000 7f86d8e00640 1 --2- 172.21.15.43:0/2700585059 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86d4074440 0x7f86d410d920 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f86c0009900 tx=0x7f86c002bcd0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.610+0000 7f86d8e00640 1 -- 172.21.15.43:0/2700585059 shutdown_connections 2024-03-28T22:38:06.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.610+0000 7f86d8e00640 1 --2- 172.21.15.43:0/2700585059 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86d410de60 0x7f86d4110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.610+0000 7f86d8e00640 1 --2- 172.21.15.43:0/2700585059 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86d4074440 0x7f86d410d920 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.610+0000 7f86d8e00640 1 --2- 172.21.15.43:0/2700585059 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86d4073b20 0x7f86d4073f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.610+0000 7f86d8e00640 1 -- 172.21.15.43:0/2700585059 >> 172.21.15.43:0/2700585059 conn(0x7f86d40fd0f0 msgr2=0x7f86d40ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:06.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.610+0000 7f86d8e00640 1 -- 172.21.15.43:0/2700585059 shutdown_connections 2024-03-28T22:38:06.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.610+0000 7f86d8e00640 1 -- 172.21.15.43:0/2700585059 wait complete. 2024-03-28T22:38:06.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.611+0000 7f86d8e00640 1 Processor -- start 2024-03-28T22:38:06.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.612+0000 7f86d8e00640 1 -- start start 2024-03-28T22:38:06.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.612+0000 7f86d8e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86d4073b20 0x7f86d419a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.612+0000 7f86d8e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86d4074440 0x7f86d419a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.612+0000 7f86d8e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86d410de60 0x7f86d419e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.612+0000 7f86d8e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f86d4113c10 con 0x7f86d4074440 2024-03-28T22:38:06.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.612+0000 7f86d8e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f86d4113a90 con 0x7f86d410de60 2024-03-28T22:38:06.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.612+0000 7f86d8e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f86d4113d90 con 0x7f86d4073b20 2024-03-28T22:38:06.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.612+0000 7f86d3400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86d4073b20 0x7f86d419a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.613+0000 7f86d3400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86d4073b20 0x7f86d419a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57380/0 (socket says 172.21.15.43:57380) 2024-03-28T22:38:06.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.613+0000 7f86d3400640 1 -- 172.21.15.43:0/2767035830 learned_addr learned my addr 172.21.15.43:0/2767035830 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:06.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.613+0000 7f86d3e00640 1 --2- 172.21.15.43:0/2767035830 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86d410de60 0x7f86d419e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.613+0000 7f86d2a00640 1 --2- 172.21.15.43:0/2767035830 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86d4074440 0x7f86d419a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.613+0000 7f86d3e00640 1 -- 172.21.15.43:0/2767035830 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86d4073b20 msgr2=0x7f86d419a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.613+0000 7f86d3e00640 1 --2- 172.21.15.43:0/2767035830 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86d4073b20 0x7f86d419a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.613+0000 7f86d3e00640 1 -- 172.21.15.43:0/2767035830 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86d4074440 msgr2=0x7f86d419a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.613+0000 7f86d3e00640 1 --2- 172.21.15.43:0/2767035830 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86d4074440 0x7f86d419a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.613+0000 7f86d3e00640 1 -- 172.21.15.43:0/2767035830 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86d419f0e0 con 0x7f86d410de60 2024-03-28T22:38:06.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.613+0000 7f86d3400640 1 --2- 172.21.15.43:0/2767035830 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86d4073b20 0x7f86d419a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:06.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.614+0000 7f86d3e00640 1 --2- 172.21.15.43:0/2767035830 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86d410de60 0x7f86d419e9b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f86d4074a70 tx=0x7f86cc00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:06.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.614+0000 7f86cbe00640 1 -- 172.21.15.43:0/2767035830 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86cc00cdf0 con 0x7f86d410de60 2024-03-28T22:38:06.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.614+0000 7f86cbe00640 1 -- 172.21.15.43:0/2767035830 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86cc005020 con 0x7f86d410de60 2024-03-28T22:38:06.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.614+0000 7f86d8e00640 1 -- 172.21.15.43:0/2767035830 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86d419f3d0 con 0x7f86d410de60 2024-03-28T22:38:06.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.614+0000 7f86cbe00640 1 -- 172.21.15.43:0/2767035830 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86cc010670 con 0x7f86d410de60 2024-03-28T22:38:06.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.614+0000 7f86d8e00640 1 -- 172.21.15.43:0/2767035830 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86d41a6c60 con 0x7f86d410de60 2024-03-28T22:38:06.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.618+0000 7f86dd07d740 1 Processor -- start 2024-03-28T22:38:06.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.619+0000 7f86cbe00640 1 -- 172.21.15.43:0/2767035830 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 27) ==== 136989+0+0 (secure 0 0 0) 0x7f86cc0054d0 con 0x7f86d410de60 2024-03-28T22:38:06.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.619+0000 7f86dd07d740 1 -- start start 2024-03-28T22:38:06.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.620+0000 7f86dd07d740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560095333360 0x560095333740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.620+0000 7f86dd07d740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560095333d10 0x560095344d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.620+0000 7f86dd07d740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560095345260 0x560095347650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.620+0000 7f86dd07d740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x560095297050 con 0x560095345260 2024-03-28T22:38:06.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.620+0000 7f86dd07d740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x56009523fce0 con 0x560095333d10 2024-03-28T22:38:06.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.620+0000 7f86dd07d740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5600952930f0 con 0x560095333360 2024-03-28T22:38:06.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.620+0000 7f86d2a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560095333360 0x560095333740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.621+0000 7f86d3400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560095333d10 0x560095344d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.621+0000 7f86d3400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560095333d10 0x560095344d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34168/0 (socket says 172.21.15.43:34168) 2024-03-28T22:38:06.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.621+0000 7f86d3400640 1 -- 172.21.15.43:0/2710889692 learned_addr learned my addr 172.21.15.43:0/2710889692 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:06.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.621+0000 7f86d3400640 1 -- 172.21.15.43:0/2710889692 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560095333360 msgr2=0x560095333740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:06.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.621+0000 7f86d3400640 1 --2- 172.21.15.43:0/2710889692 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560095333360 0x560095333740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.621+0000 7f86d2a00640 1 --2- 172.21.15.43:0/2710889692 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560095345260 0x560095347650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.621+0000 7f86d3400640 1 -- 172.21.15.43:0/2710889692 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560095345260 msgr2=0x560095347650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:06.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.621+0000 7f86d3400640 1 --2- 172.21.15.43:0/2710889692 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560095345260 0x560095347650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.621+0000 7f86d3400640 1 -- 172.21.15.43:0/2710889692 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560094fab3f0 con 0x560095333d10 2024-03-28T22:38:06.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.621+0000 7f86d2a00640 1 --2- 172.21.15.43:0/2710889692 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560095345260 0x560095347650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:38:06.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.622+0000 7f86d3400640 1 --2- 172.21.15.43:0/2710889692 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560095333d10 0x560095344d20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f86c4011680 tx=0x7f86c4009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=282c93a96c731801 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:06.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.622+0000 7f86c8c00640 1 -- 172.21.15.43:0/2710889692 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c401e070 con 0x560095333d10 2024-03-28T22:38:06.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.622+0000 7f86c8c00640 1 -- 172.21.15.43:0/2710889692 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86c400fba0 con 0x560095333d10 2024-03-28T22:38:06.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.622+0000 7f86c8c00640 1 -- 172.21.15.43:0/2710889692 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c4019ca0 con 0x560095333d10 2024-03-28T22:38:06.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.622+0000 7f86cbe00640 1 --2- 172.21.15.43:0/2767035830 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f86b80a2010 0x7f86b80a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.622+0000 7f86cbe00640 1 -- 172.21.15.43:0/2767035830 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f86cc0cee80 con 0x7f86d410de60 2024-03-28T22:38:06.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.623+0000 7f86dd07d740 1 -- 172.21.15.43:0/2710889692 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560095333d10 msgr2=0x560095344d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.623+0000 7f86dd07d740 1 --2- 172.21.15.43:0/2710889692 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560095333d10 0x560095344d20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f86c4011680 tx=0x7f86c4009f90 comp rx=0 tx=0).stop 2024-03-28T22:38:06.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.623+0000 7f86d2a00640 1 --2- 172.21.15.43:0/2767035830 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f86b80a2010 0x7f86b80a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.623+0000 7f86dd07d740 1 -- 172.21.15.43:0/2710889692 shutdown_connections 2024-03-28T22:38:06.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.623+0000 7f86dd07d740 1 --2- 172.21.15.43:0/2710889692 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560095333d10 0x560095344d20 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.623+0000 7f86dd07d740 1 --2- 172.21.15.43:0/2710889692 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560095345260 0x560095347650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.623+0000 7f86dd07d740 1 --2- 172.21.15.43:0/2710889692 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560095333360 0x560095333740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.623+0000 7f86dd07d740 1 -- 172.21.15.43:0/2710889692 >> 172.21.15.43:0/2710889692 conn(0x56009532dc80 msgr2=0x56009532e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:06.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.624+0000 7f86dd07d740 1 -- 172.21.15.43:0/2710889692 shutdown_connections 2024-03-28T22:38:06.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.624+0000 7f86d2a00640 1 --2- 172.21.15.43:0/2767035830 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f86b80a2010 0x7f86b80a44d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f86c0005ab0 tx=0x7f86c0005a20 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:06.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.624+0000 7f86dd07d740 1 -- 172.21.15.43:0/2710889692 wait complete. 2024-03-28T22:38:06.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.624+0000 7f86dd07d740 1 -- start start 2024-03-28T22:38:06.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.624+0000 7f86dd07d740 1 Processor -- start 2024-03-28T22:38:06.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.625+0000 7f86dd07d740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560095333360 0x560095342250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.625+0000 7f86dd07d740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560095333d10 0x560095342790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.625+0000 7f86dd07d740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560095345260 0x560095342cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.625+0000 7f86dd07d740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x560095348020 con 0x560095333d10 2024-03-28T22:38:06.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.625+0000 7f86dd07d740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5600952352f0 con 0x560095333360 2024-03-28T22:38:06.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.625+0000 7f86dd07d740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5600953481a0 con 0x560095345260 2024-03-28T22:38:06.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.625+0000 7f86d3400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560095333360 0x560095342250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.625+0000 7f86d3400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560095333360 0x560095342250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34182/0 (socket says 172.21.15.43:34182) 2024-03-28T22:38:06.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.625+0000 7f86d3400640 1 -- 172.21.15.43:0/3918673522 learned_addr learned my addr 172.21.15.43:0/3918673522 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:06.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.625+0000 7f86dd07d740 10 client.0 _ll_register_callbacks cb 0x560094faabe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:06.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.625+0000 7f86d3e00640 1 --2- 172.21.15.43:0/3918673522 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560095345260 0x560095342cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.625+0000 7f86d2a00640 1 --2- 172.21.15.43:0/3918673522 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560095333d10 0x560095342790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.625+0000 7f86d3400640 1 -- 172.21.15.43:0/3918673522 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560095345260 msgr2=0x560095342cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:06.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.625+0000 7f86d3400640 1 --2- 172.21.15.43:0/3918673522 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560095345260 0x560095342cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.625+0000 7f86d3400640 1 -- 172.21.15.43:0/3918673522 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560095333d10 msgr2=0x560095342790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:06.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.625+0000 7f86d3400640 1 --2- 172.21.15.43:0/3918673522 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560095333d10 0x560095342790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.625+0000 7f86d3400640 1 -- 172.21.15.43:0/3918673522 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600950757d0 con 0x560095333360 2024-03-28T22:38:06.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.626+0000 7f86d2a00640 1 --2- 172.21.15.43:0/3918673522 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560095333d10 0x560095342790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:38:06.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.626+0000 7f86d3400640 1 --2- 172.21.15.43:0/3918673522 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560095333360 0x560095342250 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f86c4000c00 tx=0x7f86c4019720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd5d753b10052bba server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:06.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.626+0000 7f86bd600640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:38:06.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.626+0000 7f86bd600640 1 -- 172.21.15.43:0/3918673522 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c401e040 con 0x560095333360 2024-03-28T22:38:06.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.626+0000 7f86bd600640 1 -- 172.21.15.43:0/3918673522 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f86c4009cc0 con 0x560095333360 2024-03-28T22:38:06.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.626+0000 7f86dd07d740 1 -- 172.21.15.43:0/3918673522 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86c400fba0 con 0x560095333360 2024-03-28T22:38:06.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.627+0000 7f86bd600640 1 -- 172.21.15.43:0/3918673522 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f86c4019a80 con 0x560095333360 2024-03-28T22:38:06.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.627+0000 7f86dd07d740 10 client.26033 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:06.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.627+0000 7f86dd07d740 1 -- 172.21.15.43:0/3918673522 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560095354200 con 0x560095333360 2024-03-28T22:38:06.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.627+0000 7f86bd600640 1 -- 172.21.15.43:0/3918673522 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86c400fba0 con 0x560095333360 2024-03-28T22:38:06.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.627+0000 7f86bd600640 1 -- 172.21.15.43:0/3918673522 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f86c402a030 con 0x560095333360 2024-03-28T22:38:06.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.628+0000 7f86dd07d740 10 client.26033 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:06.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.628+0000 7f86dd07d740 10 client.26033 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:38:06.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.628+0000 7f86dd07d740 20 client.26033 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:06.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.628+0000 7f86dd07d740 1 --2- 172.21.15.43:0/3918673522 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x560095355600 0x5600953579f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.628+0000 7f86dd07d740 4 client.26033 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:38:06.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.628+0000 7f86dd07d740 1 -- 172.21.15.43:0/3918673522 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560095240680 con 0x560095355600 2024-03-28T22:38:06.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.628+0000 7f86d2a00640 1 --2- 172.21.15.43:0/3918673522 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x560095355600 0x5600953579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.628+0000 7f86d2a00640 1 --2- 172.21.15.43:0/3918673522 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x560095355600 0x5600953579f0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7ccc343291bef800 server_cookie=92c1fa29af17ee93 in_seq=0 out_seq=0 2024-03-28T22:38:06.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.628+0000 7f86bd600640 10 client.26033 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:38:06.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.630+0000 7f86bd600640 1 -- 172.21.15.43:0/3918673522 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560095240680 con 0x560095355600 2024-03-28T22:38:06.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.630+0000 7f86bd600640 10 client.26033 handle_command_reply: tid=0 2024-03-28T22:38:06.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.630+0000 7f86dd07d740 1 client.26033 shutdown 2024-03-28T22:38:06.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.630+0000 7f86dd07d740 1 -- 172.21.15.43:0/3918673522 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560095333360 msgr2=0x560095342250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.630+0000 7f86dd07d740 1 --2- 172.21.15.43:0/3918673522 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560095333360 0x560095342250 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f86c4000c00 tx=0x7f86c4019720 comp rx=0 tx=0).stop 2024-03-28T22:38:06.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.630+0000 7f86dd07d740 1 -- 172.21.15.43:0/3918673522 shutdown_connections 2024-03-28T22:38:06.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.630+0000 7f86dd07d740 1 --2- 172.21.15.43:0/3918673522 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x560095355600 0x5600953579f0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.630+0000 7f86dd07d740 1 --2- 172.21.15.43:0/3918673522 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560095345260 0x560095342cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.630+0000 7f86dd07d740 1 --2- 172.21.15.43:0/3918673522 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560095333d10 0x560095342790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.630+0000 7f86dd07d740 1 --2- 172.21.15.43:0/3918673522 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560095333360 0x560095342250 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.630+0000 7f86dd07d740 1 -- 172.21.15.43:0/3918673522 >> 172.21.15.43:0/3918673522 conn(0x56009532dc80 msgr2=0x56009532f9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:06.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.630+0000 7f86bd600640 0 client.26033 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:38:06.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.630+0000 7f86dd07d740 1 -- 172.21.15.43:0/3918673522 shutdown_connections 2024-03-28T22:38:06.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.630+0000 7f86dd07d740 1 -- 172.21.15.43:0/3918673522 wait complete. 2024-03-28T22:38:06.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.630+0000 7f86dd07d740 20 client.26033 trim_cache size 0 max 16384 2024-03-28T22:38:06.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.644+0000 7f86dd07d740 1 Processor -- start 2024-03-28T22:38:06.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.645+0000 7f86dd07d740 1 -- start start 2024-03-28T22:38:06.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.645+0000 7f86dd07d740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560095333360 0x560095350e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.645+0000 7f86dd07d740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560095333d10 0x560095344080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.645+0000 7f86dd07d740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560095345260 0x56009535d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.645+0000 7f86dd07d740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f86d41ac7e0 con 0x560095333d10 2024-03-28T22:38:06.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.645+0000 7f86dd07d740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f86d41ae320 con 0x560095345260 2024-03-28T22:38:06.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.645+0000 7f86dd07d740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f86d41aceb0 con 0x560095333360 2024-03-28T22:38:06.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.645+0000 7f86d3400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560095333360 0x560095350e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.645+0000 7f86d3400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560095333360 0x560095350e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57406/0 (socket says 172.21.15.43:57406) 2024-03-28T22:38:06.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.645+0000 7f86d3400640 1 -- 172.21.15.43:0/4082666347 learned_addr learned my addr 172.21.15.43:0/4082666347 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:06.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.645+0000 7f86d3e00640 1 --2- 172.21.15.43:0/4082666347 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560095345260 0x56009535d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.645+0000 7f86d2a00640 1 --2- 172.21.15.43:0/4082666347 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560095333d10 0x560095344080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.646+0000 7f86d3400640 1 -- 172.21.15.43:0/4082666347 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560095345260 msgr2=0x56009535d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:06.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.646+0000 7f86d3400640 1 --2- 172.21.15.43:0/4082666347 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560095345260 0x56009535d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.646+0000 7f86d3400640 1 -- 172.21.15.43:0/4082666347 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560095333d10 msgr2=0x560095344080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:06.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.646+0000 7f86d3400640 1 --2- 172.21.15.43:0/4082666347 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560095333d10 0x560095344080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.646+0000 7f86d3400640 1 -- 172.21.15.43:0/4082666347 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560095333ab0 con 0x560095333360 2024-03-28T22:38:06.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.646+0000 7f86d3400640 1 --2- 172.21.15.43:0/4082666347 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560095333360 0x560095350e00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f86c4019650 tx=0x7f86c4019680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c66d3d1741e1fe7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:06.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.646+0000 7f86bfe00640 1 -- 172.21.15.43:0/4082666347 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c4033070 con 0x560095333360 2024-03-28T22:38:06.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.646+0000 7f86bfe00640 1 -- 172.21.15.43:0/4082666347 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86c4004030 con 0x560095333360 2024-03-28T22:38:06.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.646+0000 7f86bfe00640 1 -- 172.21.15.43:0/4082666347 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c4015020 con 0x560095333360 2024-03-28T22:38:06.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.647+0000 7f86dd07d740 1 -- 172.21.15.43:0/4082666347 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560095333360 msgr2=0x560095350e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.647+0000 7f86dd07d740 1 --2- 172.21.15.43:0/4082666347 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560095333360 0x560095350e00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f86c4019650 tx=0x7f86c4019680 comp rx=0 tx=0).stop 2024-03-28T22:38:06.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.647+0000 7f86dd07d740 1 -- 172.21.15.43:0/4082666347 shutdown_connections 2024-03-28T22:38:06.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.647+0000 7f86dd07d740 1 --2- 172.21.15.43:0/4082666347 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560095345260 0x56009535d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.647+0000 7f86dd07d740 1 --2- 172.21.15.43:0/4082666347 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560095333d10 0x560095344080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.647+0000 7f86dd07d740 1 --2- 172.21.15.43:0/4082666347 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560095333360 0x560095350e00 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.647+0000 7f86dd07d740 1 -- 172.21.15.43:0/4082666347 >> 172.21.15.43:0/4082666347 conn(0x56009532dc80 msgr2=0x560095343970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:06.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.647+0000 7f86dd07d740 1 -- 172.21.15.43:0/4082666347 shutdown_connections 2024-03-28T22:38:06.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.650+0000 7f86dd07d740 1 -- 172.21.15.43:0/4082666347 wait complete. 2024-03-28T22:38:06.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.650+0000 7f86dd07d740 1 -- start start 2024-03-28T22:38:06.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.650+0000 7f86dd07d740 1 Processor -- start 2024-03-28T22:38:06.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.651+0000 7f86dd07d740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560095333360 0x56009534d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.651+0000 7f86dd07d740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560095333d10 0x56009534d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.651+0000 7f86dd07d740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560095345260 0x5600953486c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.651+0000 7f86dd07d740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f86d41adc50 con 0x560095333d10 2024-03-28T22:38:06.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.651+0000 7f86dd07d740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f86d41ad580 con 0x560095333360 2024-03-28T22:38:06.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.651+0000 7f86dd07d740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f86d41aa7c0 con 0x560095345260 2024-03-28T22:38:06.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.651+0000 7f86dd07d740 10 client.0 _ll_register_callbacks cb 0x7f86c401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:06.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.651+0000 7f86d3400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560095333360 0x56009534d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.651+0000 7f86d3400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560095333360 0x56009534d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34206/0 (socket says 172.21.15.43:34206) 2024-03-28T22:38:06.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.651+0000 7f86d3400640 1 -- 172.21.15.43:0/4241954708 learned_addr learned my addr 172.21.15.43:0/4241954708 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:06.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.651+0000 7f86d2a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560095333d10 0x56009534d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.651+0000 7f86d3e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560095345260 0x5600953486c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.651+0000 7f86d3e00640 1 -- 172.21.15.43:0/4241954708 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560095333360 msgr2=0x56009534d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:06.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.651+0000 7f86d3e00640 1 --2- 172.21.15.43:0/4241954708 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560095333360 0x56009534d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.651+0000 7f86d3e00640 1 -- 172.21.15.43:0/4241954708 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560095333d10 msgr2=0x56009534d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:06.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.651+0000 7f86d3e00640 1 --2- 172.21.15.43:0/4241954708 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560095333d10 0x56009534d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.652+0000 7f86d3e00640 1 -- 172.21.15.43:0/4241954708 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560095348ef0 con 0x560095345260 2024-03-28T22:38:06.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.652+0000 7f86d3e00640 1 --2- 172.21.15.43:0/4241954708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560095345260 0x5600953486c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f86cc0cc050 tx=0x7f86cc09f430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ad5add299b89bc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:06.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.652+0000 7f86bd600640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:38:06.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.652+0000 7f86bd600640 1 -- 172.21.15.43:0/4241954708 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86cc0b7070 con 0x560095345260 2024-03-28T22:38:06.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.652+0000 7f86dd07d740 1 -- 172.21.15.43:0/4241954708 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86c4004030 con 0x560095345260 2024-03-28T22:38:06.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.652+0000 7f86bd600640 1 -- 172.21.15.43:0/4241954708 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f86cc0a3730 con 0x560095345260 2024-03-28T22:38:06.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.652+0000 7f86bd600640 1 -- 172.21.15.43:0/4241954708 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f86cc081050 con 0x560095345260 2024-03-28T22:38:06.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.652+0000 7f86dd07d740 10 client.25984 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:06.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.652+0000 7f86dd07d740 1 -- 172.21.15.43:0/4241954708 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5600953bb430 con 0x560095345260 2024-03-28T22:38:06.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.653+0000 7f86bd600640 1 -- 172.21.15.43:0/4241954708 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86c4004030 con 0x560095345260 2024-03-28T22:38:06.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.653+0000 7f86bd600640 1 -- 172.21.15.43:0/4241954708 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f86cc09a670 con 0x560095345260 2024-03-28T22:38:06.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.653+0000 7f86dd07d740 10 client.25984 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:06.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.653+0000 7f86dd07d740 10 client.25984 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:38:06.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.653+0000 7f86dd07d740 20 client.25984 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:06.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.653+0000 7f86dd07d740 1 --2- 172.21.15.43:0/4241954708 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x5600953bbb00 0x5600953bdef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.653+0000 7f86dd07d740 4 client.25984 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:38:06.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.653+0000 7f86dd07d740 1 -- 172.21.15.43:0/4241954708 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5600952a4590 con 0x5600953bbb00 2024-03-28T22:38:06.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.653+0000 7f86d3400640 1 --2- 172.21.15.43:0/4241954708 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x5600953bbb00 0x5600953bdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.654+0000 7f86d3400640 1 --2- 172.21.15.43:0/4241954708 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x5600953bbb00 0x5600953bdef0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=875e9c7c0b6f063e server_cookie=9f925f438711c4e2 in_seq=0 out_seq=0 2024-03-28T22:38:06.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.654+0000 7f86bd600640 10 client.25984 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:38:06.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.654+0000 7f86bd600640 1 -- 172.21.15.43:0/4241954708 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5600952a4590 con 0x5600953bbb00 2024-03-28T22:38:06.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.654+0000 7f86bd600640 10 client.25984 handle_command_reply: tid=0 2024-03-28T22:38:06.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.654+0000 7f86dd07d740 1 client.25984 shutdown 2024-03-28T22:38:06.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.654+0000 7f86dd07d740 1 -- 172.21.15.43:0/4241954708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560095345260 msgr2=0x5600953486c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.654+0000 7f86dd07d740 1 --2- 172.21.15.43:0/4241954708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560095345260 0x5600953486c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f86cc0cc050 tx=0x7f86cc09f430 comp rx=0 tx=0).stop 2024-03-28T22:38:06.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.654+0000 7f86dd07d740 1 -- 172.21.15.43:0/4241954708 shutdown_connections 2024-03-28T22:38:06.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.654+0000 7f86dd07d740 1 --2- 172.21.15.43:0/4241954708 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x5600953bbb00 0x5600953bdef0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.654+0000 7f86dd07d740 1 --2- 172.21.15.43:0/4241954708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560095345260 0x5600953486c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.654+0000 7f86dd07d740 1 --2- 172.21.15.43:0/4241954708 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560095333d10 0x56009534d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.654+0000 7f86dd07d740 1 --2- 172.21.15.43:0/4241954708 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560095333360 0x56009534d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.654+0000 7f86bd600640 0 client.25984 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:38:06.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.654+0000 7f86dd07d740 1 -- 172.21.15.43:0/4241954708 >> 172.21.15.43:0/4241954708 conn(0x56009532dc80 msgr2=0x560095344200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:06.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.655+0000 7f86dd07d740 1 -- 172.21.15.43:0/4241954708 shutdown_connections 2024-03-28T22:38:06.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.655+0000 7f86dd07d740 1 -- 172.21.15.43:0/4241954708 wait complete. 2024-03-28T22:38:06.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.655+0000 7f86dd07d740 20 client.25984 trim_cache size 0 max 16384 2024-03-28T22:38:06.656 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:38:06.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.656+0000 7f86d8e00640 1 -- 172.21.15.43:0/2767035830 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f86b80a2010 msgr2=0x7f86b80a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.656+0000 7f86d8e00640 1 --2- 172.21.15.43:0/2767035830 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f86b80a2010 0x7f86b80a44d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f86c0005ab0 tx=0x7f86c0005a20 comp rx=0 tx=0).stop 2024-03-28T22:38:06.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.657+0000 7f86d8e00640 1 -- 172.21.15.43:0/2767035830 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86d410de60 msgr2=0x7f86d419e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.657+0000 7f86d8e00640 1 --2- 172.21.15.43:0/2767035830 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86d410de60 0x7f86d419e9b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f86d4074a70 tx=0x7f86cc00ef10 comp rx=0 tx=0).stop 2024-03-28T22:38:06.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.657+0000 7f86d8e00640 1 -- 172.21.15.43:0/2767035830 shutdown_connections 2024-03-28T22:38:06.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.657+0000 7f86d8e00640 1 --2- 172.21.15.43:0/2767035830 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f86b80a2010 0x7f86b80a44d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.657+0000 7f86d8e00640 1 --2- 172.21.15.43:0/2767035830 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86d410de60 0x7f86d419e9b0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.657+0000 7f86d8e00640 1 --2- 172.21.15.43:0/2767035830 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86d4074440 0x7f86d419a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.657+0000 7f86d8e00640 1 --2- 172.21.15.43:0/2767035830 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86d4073b20 0x7f86d419a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.657+0000 7f86d8e00640 1 -- 172.21.15.43:0/2767035830 >> 172.21.15.43:0/2767035830 conn(0x7f86d40fd0f0 msgr2=0x7f86d40fde00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:06.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.657+0000 7f86d8e00640 1 -- 172.21.15.43:0/2767035830 shutdown_connections 2024-03-28T22:38:06.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.657+0000 7f86d8e00640 1 -- 172.21.15.43:0/2767035830 wait complete. 2024-03-28T22:38:06.680 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:38:06.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.879+0000 7fcb0d800640 1 -- 172.21.15.43:0/818559564 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcb08109900 msgr2=0x7fcb08109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.879+0000 7fcb0d800640 1 --2- 172.21.15.43:0/818559564 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcb08109900 0x7fcb08109ce0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fcaf8009900 tx=0x7fcaf802bc30 comp rx=0 tx=0).stop 2024-03-28T22:38:06.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.880+0000 7fcb0d800640 1 -- 172.21.15.43:0/818559564 shutdown_connections 2024-03-28T22:38:06.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.880+0000 7fcb0d800640 1 --2- 172.21.15.43:0/818559564 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb081042b0 0x7fcb081107c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.880+0000 7fcb0d800640 1 --2- 172.21.15.43:0/818559564 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb081038f0 0x7fcb08103d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.880+0000 7fcb0d800640 1 --2- 172.21.15.43:0/818559564 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcb08109900 0x7fcb08109ce0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.880+0000 7fcb0d800640 1 -- 172.21.15.43:0/818559564 >> 172.21.15.43:0/818559564 conn(0x7fcb080ff400 msgr2=0x7fcb08101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:06.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.880+0000 7fcb0d800640 1 -- 172.21.15.43:0/818559564 shutdown_connections 2024-03-28T22:38:06.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.880+0000 7fcb0d800640 1 -- 172.21.15.43:0/818559564 wait complete. 2024-03-28T22:38:06.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.881+0000 7fcb0d800640 1 Processor -- start 2024-03-28T22:38:06.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.881+0000 7fcb0d800640 1 -- start start 2024-03-28T22:38:06.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.882+0000 7fcb0d800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcb081038f0 0x7fcb0819a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.882+0000 7fcb0d800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb081042b0 0x7fcb0819a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.882+0000 7fcb0d800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb08109900 0x7fcb0819eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.882+0000 7fcb0d800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcb08113d60 con 0x7fcb08109900 2024-03-28T22:38:06.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.882+0000 7fcb0d800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fcb08113be0 con 0x7fcb081038f0 2024-03-28T22:38:06.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.882+0000 7fcb0d800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fcb08113ee0 con 0x7fcb081042b0 2024-03-28T22:38:06.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.882+0000 7fcb06a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcb081038f0 0x7fcb0819a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.882+0000 7fcb06a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcb081038f0 0x7fcb0819a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34232/0 (socket says 172.21.15.43:34232) 2024-03-28T22:38:06.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.882+0000 7fcb06a00640 1 -- 172.21.15.43:0/626201985 learned_addr learned my addr 172.21.15.43:0/626201985 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:06.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.882+0000 7fcb06000640 1 --2- 172.21.15.43:0/626201985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb081042b0 0x7fcb0819a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.883+0000 7fcb07400640 1 --2- 172.21.15.43:0/626201985 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb08109900 0x7fcb0819eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.883+0000 7fcb06a00640 1 -- 172.21.15.43:0/626201985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb081042b0 msgr2=0x7fcb0819a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.883+0000 7fcb06a00640 1 --2- 172.21.15.43:0/626201985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb081042b0 0x7fcb0819a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.883+0000 7fcb06a00640 1 -- 172.21.15.43:0/626201985 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb08109900 msgr2=0x7fcb0819eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.883+0000 7fcb06a00640 1 --2- 172.21.15.43:0/626201985 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb08109900 0x7fcb0819eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.883+0000 7fcb06a00640 1 -- 172.21.15.43:0/626201985 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb0819f230 con 0x7fcb081038f0 2024-03-28T22:38:06.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.883+0000 7fcb06000640 1 --2- 172.21.15.43:0/626201985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb081042b0 0x7fcb0819a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:38:06.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.883+0000 7fcb07400640 1 --2- 172.21.15.43:0/626201985 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb08109900 0x7fcb0819eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:38:06.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.883+0000 7fcb06a00640 1 --2- 172.21.15.43:0/626201985 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcb081038f0 0x7fcb0819a240 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fcaf802efd0 tx=0x7fcaf8004430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:06.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.884+0000 7fcaff400640 1 -- 172.21.15.43:0/626201985 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf8041070 con 0x7fcb081038f0 2024-03-28T22:38:06.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.884+0000 7fcaff400640 1 -- 172.21.15.43:0/626201985 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcaf80054e0 con 0x7fcb081038f0 2024-03-28T22:38:06.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.884+0000 7fcb0d800640 1 -- 172.21.15.43:0/626201985 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb0819f4c0 con 0x7fcb081038f0 2024-03-28T22:38:06.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.884+0000 7fcaff400640 1 -- 172.21.15.43:0/626201985 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf8003d40 con 0x7fcb081038f0 2024-03-28T22:38:06.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.884+0000 7fcb0d800640 1 -- 172.21.15.43:0/626201985 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb081a6db0 con 0x7fcb081038f0 2024-03-28T22:38:06.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.888+0000 7fcb10565740 1 Processor -- start 2024-03-28T22:38:06.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.889+0000 7fcb10565740 1 -- start start 2024-03-28T22:38:06.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.889+0000 7fcaff400640 1 -- 172.21.15.43:0/626201985 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 27) ==== 136989+0+0 (secure 0 0 0) 0x7fcaf8044070 con 0x7fcb081038f0 2024-03-28T22:38:06.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.889+0000 7fcb10565740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dbfee46190 0x55dbfee46570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.890+0000 7fcb10565740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dbfee46b40 0x55dbfee57b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.890+0000 7fcb10565740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dbfee58090 0x55dbfee5a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.890+0000 7fcb10565740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55dbfedaa050 con 0x55dbfee46190 2024-03-28T22:38:06.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.890+0000 7fcb10565740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55dbfed52ce0 con 0x55dbfee46b40 2024-03-28T22:38:06.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.890+0000 7fcb10565740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55dbfeda60f0 con 0x55dbfee58090 2024-03-28T22:38:06.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.890+0000 7fcb07400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dbfee46190 0x55dbfee46570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.890+0000 7fcb07400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dbfee46190 0x55dbfee46570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55946/0 (socket says 172.21.15.43:55946) 2024-03-28T22:38:06.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.890+0000 7fcb07400640 1 -- 172.21.15.43:0/629147886 learned_addr learned my addr 172.21.15.43:0/629147886 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:06.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.891+0000 7fcb07400640 1 -- 172.21.15.43:0/629147886 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dbfee58090 msgr2=0x55dbfee5a480 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:38:06.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.891+0000 7fcb07400640 1 --2- 172.21.15.43:0/629147886 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dbfee58090 0x55dbfee5a480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.891+0000 7fcb07400640 1 -- 172.21.15.43:0/629147886 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dbfee46b40 msgr2=0x55dbfee57b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:38:06.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.891+0000 7fcb07400640 1 --2- 172.21.15.43:0/629147886 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dbfee46b40 0x55dbfee57b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.891+0000 7fcb07400640 1 -- 172.21.15.43:0/629147886 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbfeabe3f0 con 0x55dbfee46190 2024-03-28T22:38:06.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.891+0000 7fcaff400640 1 --2- 172.21.15.43:0/626201985 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcaec0a2110 0x7fcaec0a45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.891+0000 7fcb07400640 1 --2- 172.21.15.43:0/629147886 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dbfee46190 0x55dbfee46570 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcb00009f10 tx=0x7fcb0000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36e1b2a6dba9d4de server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:06.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.891+0000 7fcaff400640 1 -- 172.21.15.43:0/626201985 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fcaf80f0300 con 0x7fcb081038f0 2024-03-28T22:38:06.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.892+0000 7fcaf3e00640 1 -- 172.21.15.43:0/629147886 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb00003b70 con 0x55dbfee46190 2024-03-28T22:38:06.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.892+0000 7fcaf3e00640 1 -- 172.21.15.43:0/629147886 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb00003d10 con 0x55dbfee46190 2024-03-28T22:38:06.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.892+0000 7fcaf3e00640 1 -- 172.21.15.43:0/629147886 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb00011aa0 con 0x55dbfee46190 2024-03-28T22:38:06.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.893+0000 7fcb06a00640 1 --2- 172.21.15.43:0/626201985 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcaec0a2110 0x7fcaec0a45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.893+0000 7fcb10565740 1 -- 172.21.15.43:0/629147886 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dbfee46190 msgr2=0x55dbfee46570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.893+0000 7fcb10565740 1 --2- 172.21.15.43:0/629147886 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dbfee46190 0x55dbfee46570 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcb00009f10 tx=0x7fcb0000f730 comp rx=0 tx=0).stop 2024-03-28T22:38:06.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.893+0000 7fcb10565740 1 -- 172.21.15.43:0/629147886 shutdown_connections 2024-03-28T22:38:06.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.893+0000 7fcb10565740 1 --2- 172.21.15.43:0/629147886 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dbfee46b40 0x55dbfee57b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.893+0000 7fcb10565740 1 --2- 172.21.15.43:0/629147886 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dbfee58090 0x55dbfee5a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.893+0000 7fcb10565740 1 --2- 172.21.15.43:0/629147886 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dbfee46190 0x55dbfee46570 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.893+0000 7fcb10565740 1 -- 172.21.15.43:0/629147886 >> 172.21.15.43:0/629147886 conn(0x55dbfee40ab0 msgr2=0x55dbfee40e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:06.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.894+0000 7fcb06a00640 1 --2- 172.21.15.43:0/626201985 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcaec0a2110 0x7fcaec0a45d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcaf80ed640 tx=0x7fcaf8005e60 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:06.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.894+0000 7fcb10565740 1 -- 172.21.15.43:0/629147886 shutdown_connections 2024-03-28T22:38:06.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.894+0000 7fcb10565740 1 -- 172.21.15.43:0/629147886 wait complete. 2024-03-28T22:38:06.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.894+0000 7fcb10565740 1 -- start start 2024-03-28T22:38:06.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.895+0000 7fcb10565740 1 Processor -- start 2024-03-28T22:38:06.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.895+0000 7fcb10565740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dbfee46190 0x55dbfee55140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.895+0000 7fcb10565740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dbfee46b40 0x55dbfee55680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.895+0000 7fcb10565740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dbfee58090 0x55dbfee55bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.895+0000 7fcb10565740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55dbfee5ae50 con 0x55dbfee46190 2024-03-28T22:38:06.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.895+0000 7fcb10565740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55dbfed482f0 con 0x55dbfee46b40 2024-03-28T22:38:06.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.895+0000 7fcb10565740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55dbfee5afd0 con 0x55dbfee58090 2024-03-28T22:38:06.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.896+0000 7fcb10565740 10 client.0 _ll_register_callbacks cb 0x55dbfeabdbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:06.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.896+0000 7fcb06000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dbfee46b40 0x55dbfee55680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.896+0000 7fcb06000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dbfee46b40 0x55dbfee55680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34240/0 (socket says 172.21.15.43:34240) 2024-03-28T22:38:06.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.896+0000 7fcb06000640 1 -- 172.21.15.43:0/3908965713 learned_addr learned my addr 172.21.15.43:0/3908965713 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:06.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.896+0000 7fcb07400640 1 --2- 172.21.15.43:0/3908965713 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dbfee58090 0x55dbfee55bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.896+0000 7fcb07400640 1 --2- 172.21.15.43:0/3908965713 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dbfee46190 0x55dbfee55140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.897+0000 7fcb06000640 1 -- 172.21.15.43:0/3908965713 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dbfee58090 msgr2=0x55dbfee55bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:06.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.897+0000 7fcb06000640 1 --2- 172.21.15.43:0/3908965713 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dbfee58090 0x55dbfee55bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.897+0000 7fcb06000640 1 -- 172.21.15.43:0/3908965713 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dbfee46190 msgr2=0x55dbfee55140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:06.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.897+0000 7fcb06000640 1 --2- 172.21.15.43:0/3908965713 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dbfee46190 0x55dbfee55140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.897+0000 7fcb06000640 1 -- 172.21.15.43:0/3908965713 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbfeb887d0 con 0x55dbfee46b40 2024-03-28T22:38:06.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.897+0000 7fcb07400640 1 --2- 172.21.15.43:0/3908965713 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dbfee46190 0x55dbfee55140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:38:06.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.897+0000 7fcb06000640 1 --2- 172.21.15.43:0/3908965713 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dbfee46b40 0x55dbfee55680 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fcaf400e680 tx=0x7fcaf400ea80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65fc29ee045a99a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:06.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.897+0000 7fcaf0c00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:38:06.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.898+0000 7fcaf0c00640 1 -- 172.21.15.43:0/3908965713 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf4004220 con 0x55dbfee46b40 2024-03-28T22:38:06.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.898+0000 7fcb10565740 1 -- 172.21.15.43:0/3908965713 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb00003d10 con 0x55dbfee46b40 2024-03-28T22:38:06.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.898+0000 7fcaf0c00640 1 -- 172.21.15.43:0/3908965713 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fcaf40043c0 con 0x55dbfee46b40 2024-03-28T22:38:06.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.898+0000 7fcaf0c00640 1 -- 172.21.15.43:0/3908965713 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcaf4005020 con 0x55dbfee46b40 2024-03-28T22:38:06.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.898+0000 7fcb10565740 10 client.26048 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:06.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.898+0000 7fcb10565740 1 -- 172.21.15.43:0/3908965713 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55dbfee67270 con 0x55dbfee46b40 2024-03-28T22:38:06.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.899+0000 7fcaf0c00640 1 -- 172.21.15.43:0/3908965713 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb00003d10 con 0x55dbfee46b40 2024-03-28T22:38:06.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.899+0000 7fcaf0c00640 1 -- 172.21.15.43:0/3908965713 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fcaf40047a0 con 0x55dbfee46b40 2024-03-28T22:38:06.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.899+0000 7fcb10565740 10 client.26048 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:06.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.899+0000 7fcb10565740 10 client.26048 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:38:06.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.899+0000 7fcb10565740 20 client.26048 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:06.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.899+0000 7fcb10565740 1 --2- 172.21.15.43:0/3908965713 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55dbfee68680 0x55dbfee6aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.899+0000 7fcb10565740 4 client.26048 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:38:06.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.899+0000 7fcb10565740 1 -- 172.21.15.43:0/3908965713 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dbfed53680 con 0x55dbfee68680 2024-03-28T22:38:06.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.900+0000 7fcb07400640 1 --2- 172.21.15.43:0/3908965713 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55dbfee68680 0x55dbfee6aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.900+0000 7fcb07400640 1 --2- 172.21.15.43:0/3908965713 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55dbfee68680 0x55dbfee6aaf0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8e957297b10cd12 server_cookie=9d4c788699ca58eb in_seq=0 out_seq=0 2024-03-28T22:38:06.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.900+0000 7fcaf0c00640 10 client.26048 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:06.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.901+0000 7fcaf0c00640 1 -- 172.21.15.43:0/3908965713 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dbfed53680 con 0x55dbfee68680 2024-03-28T22:38:06.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.901+0000 7fcaf0c00640 10 client.26048 handle_command_reply: tid=0 2024-03-28T22:38:06.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.902+0000 7fcb10565740 1 client.26048 shutdown 2024-03-28T22:38:06.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.902+0000 7fcb10565740 1 -- 172.21.15.43:0/3908965713 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dbfee46b40 msgr2=0x55dbfee55680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.902+0000 7fcb10565740 1 --2- 172.21.15.43:0/3908965713 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dbfee46b40 0x55dbfee55680 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fcaf400e680 tx=0x7fcaf400ea80 comp rx=0 tx=0).stop 2024-03-28T22:38:06.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.902+0000 7fcb10565740 1 -- 172.21.15.43:0/3908965713 shutdown_connections 2024-03-28T22:38:06.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.902+0000 7fcb10565740 1 --2- 172.21.15.43:0/3908965713 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55dbfee68680 0x55dbfee6aaf0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.902+0000 7fcb10565740 1 --2- 172.21.15.43:0/3908965713 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dbfee46b40 0x55dbfee55680 secure :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fcaf400e680 tx=0x7fcaf400ea80 comp rx=0 tx=0).stop 2024-03-28T22:38:06.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.902+0000 7fcb10565740 1 --2- 172.21.15.43:0/3908965713 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dbfee58090 0x55dbfee55bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.902+0000 7fcb10565740 1 --2- 172.21.15.43:0/3908965713 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dbfee46190 0x55dbfee55140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.902+0000 7fcb10565740 1 -- 172.21.15.43:0/3908965713 >> 172.21.15.43:0/3908965713 conn(0x55dbfee40ab0 msgr2=0x55dbfee42800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:06.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.902+0000 7fcaf0c00640 0 client.26048 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:06.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.902+0000 7fcb10565740 1 -- 172.21.15.43:0/3908965713 shutdown_connections 2024-03-28T22:38:06.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.903+0000 7fcb10565740 1 -- 172.21.15.43:0/3908965713 wait complete. 2024-03-28T22:38:06.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.903+0000 7fcb10565740 20 client.26048 trim_cache size 0 max 16384 2024-03-28T22:38:06.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.931+0000 7fcb10565740 1 Processor -- start 2024-03-28T22:38:06.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.931+0000 7fcb10565740 1 -- start start 2024-03-28T22:38:06.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.932+0000 7fcb10565740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dbfee46190 0x55dbfee57870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.932+0000 7fcb10565740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dbfee58090 0x55dbfee63c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.932+0000 7fcb10565740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dbfee70400 0x55dbfee70840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.932+0000 7fcb10565740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcb081ac9e0 con 0x55dbfee70400 2024-03-28T22:38:06.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.932+0000 7fcb10565740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fcb081ae520 con 0x55dbfee46190 2024-03-28T22:38:06.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.932+0000 7fcb10565740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fcb081ad0b0 con 0x55dbfee58090 2024-03-28T22:38:06.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.932+0000 7fcb06000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dbfee58090 0x55dbfee63c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.932+0000 7fcb06000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dbfee58090 0x55dbfee63c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57460/0 (socket says 172.21.15.43:57460) 2024-03-28T22:38:06.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.932+0000 7fcb06000640 1 -- 172.21.15.43:0/3670144565 learned_addr learned my addr 172.21.15.43:0/3670144565 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:06.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.932+0000 7fcb07400640 1 --2- 172.21.15.43:0/3670144565 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dbfee46190 0x55dbfee57870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.932+0000 7fcb07400640 1 --2- 172.21.15.43:0/3670144565 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dbfee70400 0x55dbfee70840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.933+0000 7fcb06000640 1 -- 172.21.15.43:0/3670144565 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dbfee46190 msgr2=0x55dbfee57870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:06.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.933+0000 7fcb06000640 1 --2- 172.21.15.43:0/3670144565 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dbfee46190 0x55dbfee57870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.933+0000 7fcb06000640 1 -- 172.21.15.43:0/3670144565 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dbfee70400 msgr2=0x55dbfee70840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:06.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.933+0000 7fcb06000640 1 --2- 172.21.15.43:0/3670144565 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dbfee70400 0x55dbfee70840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.933+0000 7fcb06000640 1 -- 172.21.15.43:0/3670144565 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbfee53690 con 0x55dbfee58090 2024-03-28T22:38:06.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.933+0000 7fcb07400640 1 --2- 172.21.15.43:0/3670144565 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dbfee46190 0x55dbfee57870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:06.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.934+0000 7fcb06000640 1 --2- 172.21.15.43:0/3670144565 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dbfee58090 0x55dbfee63c30 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4004e00 tx=0x7fcaf400a310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8af302d284e5259 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:06.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.934+0000 7fcaf3400640 1 -- 172.21.15.43:0/3670144565 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf402b070 con 0x55dbfee58090 2024-03-28T22:38:06.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.934+0000 7fcaf3400640 1 -- 172.21.15.43:0/3670144565 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcaf4004860 con 0x55dbfee58090 2024-03-28T22:38:06.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.934+0000 7fcaf3400640 1 -- 172.21.15.43:0/3670144565 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf400aa20 con 0x55dbfee58090 2024-03-28T22:38:06.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.935+0000 7fcb10565740 1 -- 172.21.15.43:0/3670144565 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dbfee58090 msgr2=0x55dbfee63c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.935+0000 7fcb10565740 1 --2- 172.21.15.43:0/3670144565 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dbfee58090 0x55dbfee63c30 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4004e00 tx=0x7fcaf400a310 comp rx=0 tx=0).stop 2024-03-28T22:38:06.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.935+0000 7fcb10565740 1 -- 172.21.15.43:0/3670144565 shutdown_connections 2024-03-28T22:38:06.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.935+0000 7fcb10565740 1 --2- 172.21.15.43:0/3670144565 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dbfee70400 0x55dbfee70840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.935+0000 7fcb10565740 1 --2- 172.21.15.43:0/3670144565 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dbfee58090 0x55dbfee63c30 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.935+0000 7fcb10565740 1 --2- 172.21.15.43:0/3670144565 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dbfee46190 0x55dbfee57870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.935+0000 7fcb10565740 1 -- 172.21.15.43:0/3670144565 >> 172.21.15.43:0/3670144565 conn(0x55dbfee40ab0 msgr2=0x55dbfee504b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:06.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.935+0000 7fcb10565740 1 -- 172.21.15.43:0/3670144565 shutdown_connections 2024-03-28T22:38:06.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.935+0000 7fcb10565740 1 -- 172.21.15.43:0/3670144565 wait complete. 2024-03-28T22:38:06.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.936+0000 7fcb10565740 1 -- start start 2024-03-28T22:38:06.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.936+0000 7fcb10565740 1 Processor -- start 2024-03-28T22:38:06.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.937+0000 7fcb10565740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dbfee46190 0x55dbfee51d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.937+0000 7fcb10565740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dbfee58090 0x55dbfee522d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.937+0000 7fcb10565740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dbfee70400 0x55dbfee52810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.937+0000 7fcb10565740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcb081ade50 con 0x55dbfee70400 2024-03-28T22:38:06.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.937+0000 7fcb10565740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fcb081ad780 con 0x55dbfee46190 2024-03-28T22:38:06.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.937+0000 7fcb10565740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fcb081aa9c0 con 0x55dbfee58090 2024-03-28T22:38:06.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.937+0000 7fcb10565740 10 client.0 _ll_register_callbacks cb 0x7fcaf400c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:06.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.937+0000 7fcb06000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dbfee58090 0x55dbfee522d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.937+0000 7fcb07400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dbfee46190 0x55dbfee51d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.937+0000 7fcb06000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dbfee58090 0x55dbfee522d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57476/0 (socket says 172.21.15.43:57476) 2024-03-28T22:38:06.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.937+0000 7fcb06000640 1 -- 172.21.15.43:0/1151479952 learned_addr learned my addr 172.21.15.43:0/1151479952 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:06.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.938+0000 7fcb06000640 1 -- 172.21.15.43:0/1151479952 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dbfee46190 msgr2=0x55dbfee51d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:06.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.938+0000 7fcb06000640 1 --2- 172.21.15.43:0/1151479952 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dbfee46190 0x55dbfee51d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.938+0000 7fcb07400640 1 --2- 172.21.15.43:0/1151479952 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dbfee70400 0x55dbfee52810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.938+0000 7fcb06000640 1 -- 172.21.15.43:0/1151479952 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dbfee70400 msgr2=0x55dbfee52810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:06.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.938+0000 7fcb06000640 1 --2- 172.21.15.43:0/1151479952 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dbfee70400 0x55dbfee52810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.938+0000 7fcb06000640 1 -- 172.21.15.43:0/1151479952 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbfeecdb90 con 0x55dbfee58090 2024-03-28T22:38:06.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.938+0000 7fcb07400640 1 --2- 172.21.15.43:0/1151479952 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dbfee70400 0x55dbfee52810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:38:06.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.938+0000 7fcb06000640 1 --2- 172.21.15.43:0/1151479952 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dbfee58090 0x55dbfee522d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fcaf40058c0 tx=0x7fcaf40053f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d054cd1d9d97330 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:06.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.938+0000 7fcaf0c00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:38:06.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.939+0000 7fcaf0c00640 1 -- 172.21.15.43:0/1151479952 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf402b030 con 0x55dbfee58090 2024-03-28T22:38:06.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.939+0000 7fcaf0c00640 1 -- 172.21.15.43:0/1151479952 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fcaf400a7c0 con 0x55dbfee58090 2024-03-28T22:38:06.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.939+0000 7fcb10565740 1 -- 172.21.15.43:0/1151479952 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcaf4004860 con 0x55dbfee58090 2024-03-28T22:38:06.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.939+0000 7fcaf0c00640 1 -- 172.21.15.43:0/1151479952 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcaf4004a10 con 0x55dbfee58090 2024-03-28T22:38:06.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.939+0000 7fcb10565740 10 client.25999 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:06.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.939+0000 7fcb10565740 1 -- 172.21.15.43:0/1151479952 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55dbfeecee50 con 0x55dbfee58090 2024-03-28T22:38:06.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.940+0000 7fcaf0c00640 1 -- 172.21.15.43:0/1151479952 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcaf4004460 con 0x55dbfee58090 2024-03-28T22:38:06.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.940+0000 7fcaf0c00640 1 -- 172.21.15.43:0/1151479952 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fcaf4036020 con 0x55dbfee58090 2024-03-28T22:38:06.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.940+0000 7fcb10565740 10 client.25999 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:06.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.940+0000 7fcb10565740 10 client.25999 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:38:06.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.940+0000 7fcb10565740 20 client.25999 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:06.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.940+0000 7fcb10565740 1 --2- 172.21.15.43:0/1151479952 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55dbfeecf3e0 0x55dbfeed1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.940+0000 7fcb10565740 4 client.25999 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:38:06.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.940+0000 7fcb10565740 1 -- 172.21.15.43:0/1151479952 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fcaf4004860 con 0x55dbfeecf3e0 2024-03-28T22:38:06.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.940+0000 7fcb07400640 1 --2- 172.21.15.43:0/1151479952 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55dbfeecf3e0 0x55dbfeed1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.941+0000 7fcb07400640 1 --2- 172.21.15.43:0/1151479952 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55dbfeecf3e0 0x55dbfeed1800 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4b156633a7b4dec server_cookie=c8abcd866bd89ab5 in_seq=0 out_seq=0 2024-03-28T22:38:06.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.941+0000 7fcaf0c00640 10 client.25999 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:06.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.941+0000 7fcaf0c00640 1 -- 172.21.15.43:0/1151479952 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fcaf4004860 con 0x55dbfeecf3e0 2024-03-28T22:38:06.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.941+0000 7fcaf0c00640 10 client.25999 handle_command_reply: tid=0 2024-03-28T22:38:06.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.941+0000 7fcb10565740 1 client.25999 shutdown 2024-03-28T22:38:06.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.942+0000 7fcb10565740 1 -- 172.21.15.43:0/1151479952 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dbfee58090 msgr2=0x55dbfee522d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.942+0000 7fcb10565740 1 --2- 172.21.15.43:0/1151479952 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dbfee58090 0x55dbfee522d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fcaf40058c0 tx=0x7fcaf40053f0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.942+0000 7fcb10565740 1 -- 172.21.15.43:0/1151479952 shutdown_connections 2024-03-28T22:38:06.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.942+0000 7fcb10565740 1 --2- 172.21.15.43:0/1151479952 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dbfee70400 0x55dbfee52810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.942+0000 7fcb10565740 1 --2- 172.21.15.43:0/1151479952 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dbfee58090 0x55dbfee522d0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.942+0000 7fcb10565740 1 --2- 172.21.15.43:0/1151479952 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55dbfeecf3e0 0x55dbfeed1800 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.942+0000 7fcb10565740 1 --2- 172.21.15.43:0/1151479952 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dbfee46190 0x55dbfee51d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.942+0000 7fcaf0c00640 0 client.25999 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:06.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.942+0000 7fcb10565740 1 -- 172.21.15.43:0/1151479952 >> 172.21.15.43:0/1151479952 conn(0x55dbfee40ab0 msgr2=0x55dbfee6f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:06.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.942+0000 7fcb10565740 1 -- 172.21.15.43:0/1151479952 shutdown_connections 2024-03-28T22:38:06.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.942+0000 7fcb10565740 1 -- 172.21.15.43:0/1151479952 wait complete. 2024-03-28T22:38:06.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.943+0000 7fcb10565740 20 client.25999 trim_cache size 0 max 16384 2024-03-28T22:38:06.943 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:38:06.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.945+0000 7fcb0d800640 1 -- 172.21.15.43:0/626201985 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcaec0a2110 msgr2=0x7fcaec0a45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.945+0000 7fcb0d800640 1 --2- 172.21.15.43:0/626201985 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcaec0a2110 0x7fcaec0a45d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcaf80ed640 tx=0x7fcaf8005e60 comp rx=0 tx=0).stop 2024-03-28T22:38:06.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.945+0000 7fcb0d800640 1 -- 172.21.15.43:0/626201985 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcb081038f0 msgr2=0x7fcb0819a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.945+0000 7fcb0d800640 1 --2- 172.21.15.43:0/626201985 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcb081038f0 0x7fcb0819a240 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fcaf802efd0 tx=0x7fcaf8004430 comp rx=0 tx=0).stop 2024-03-28T22:38:06.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.945+0000 7fcb0d800640 1 -- 172.21.15.43:0/626201985 shutdown_connections 2024-03-28T22:38:06.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.945+0000 7fcb0d800640 1 --2- 172.21.15.43:0/626201985 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcaec0a2110 0x7fcaec0a45d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.945+0000 7fcb0d800640 1 --2- 172.21.15.43:0/626201985 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcb08109900 0x7fcb0819eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.945+0000 7fcb0d800640 1 --2- 172.21.15.43:0/626201985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcb081042b0 0x7fcb0819a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.945+0000 7fcb0d800640 1 --2- 172.21.15.43:0/626201985 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcb081038f0 0x7fcb0819a240 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.945+0000 7fcb0d800640 1 -- 172.21.15.43:0/626201985 >> 172.21.15.43:0/626201985 conn(0x7fcb080ff400 msgr2=0x7fcb080ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:06.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.946+0000 7fcb0d800640 1 -- 172.21.15.43:0/626201985 shutdown_connections 2024-03-28T22:38:06.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:06.946+0000 7fcb0d800640 1 -- 172.21.15.43:0/626201985 wait complete. 2024-03-28T22:38:06.989 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.989+0000 7fc358c00640 1 -- 172.21.15.43:0/3707106826 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc354074420 msgr2=0x7fc35410d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.989 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.989+0000 7fc358c00640 1 --2- 172.21.15.43:0/3707106826 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc354074420 0x7fc35410d8d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc340009900 tx=0x7fc34002bcc0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.989 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.989+0000 7fc358c00640 1 -- 172.21.15.43:0/3707106826 shutdown_connections 2024-03-28T22:38:06.989 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.989+0000 7fc358c00640 1 --2- 172.21.15.43:0/3707106826 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc354074420 0x7fc35410d8d0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.989 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.989+0000 7fc358c00640 1 --2- 172.21.15.43:0/3707106826 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc35410de10 0x7fc354110200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.989 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.989+0000 7fc358c00640 1 --2- 172.21.15.43:0/3707106826 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc354073b00 0x7fc354073ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.990 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.989+0000 7fc358c00640 1 -- 172.21.15.43:0/3707106826 >> 172.21.15.43:0/3707106826 conn(0x7fc3540fd060 msgr2=0x7fc3540ff480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:06.990 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.990+0000 7fc358c00640 1 -- 172.21.15.43:0/3707106826 shutdown_connections 2024-03-28T22:38:06.990 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.990+0000 7fc358c00640 1 -- 172.21.15.43:0/3707106826 wait complete. 2024-03-28T22:38:06.992 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.991+0000 7fc358c00640 1 Processor -- start 2024-03-28T22:38:06.992 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.991+0000 7fc358c00640 1 -- start start 2024-03-28T22:38:06.992 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.991+0000 7fc358c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc354073b00 0x7fc35419a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.992 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.991+0000 7fc358c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc354074420 0x7fc35419a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.992 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.991+0000 7fc358c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc35410de10 0x7fc35419e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.992 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.991+0000 7fc358c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc354113b80 con 0x7fc35410de10 2024-03-28T22:38:06.992 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.991+0000 7fc358c00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fc354113a00 con 0x7fc354073b00 2024-03-28T22:38:06.992 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.991+0000 7fc358c00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fc354113d00 con 0x7fc354074420 2024-03-28T22:38:06.993 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.991+0000 7fc353400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc354073b00 0x7fc35419a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.993 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.991+0000 7fc353400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc354073b00 0x7fc35419a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34264/0 (socket says 172.21.15.43:34264) 2024-03-28T22:38:06.993 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.991+0000 7fc353400640 1 -- 172.21.15.43:0/343439203 learned_addr learned my addr 172.21.15.43:0/343439203 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:06.993 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.992+0000 7fc352a00640 1 --2- 172.21.15.43:0/343439203 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc354074420 0x7fc35419a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.993 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.992+0000 7fc353e00640 1 --2- 172.21.15.43:0/343439203 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc35410de10 0x7fc35419e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.993 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.992+0000 7fc353400640 1 -- 172.21.15.43:0/343439203 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc354074420 msgr2=0x7fc35419a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.993 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.992+0000 7fc353400640 1 --2- 172.21.15.43:0/343439203 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc354074420 0x7fc35419a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.993 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.992+0000 7fc353400640 1 -- 172.21.15.43:0/343439203 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc35410de10 msgr2=0x7fc35419e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.993 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.992+0000 7fc353400640 1 --2- 172.21.15.43:0/343439203 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc35410de10 0x7fc35419e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.993 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.992+0000 7fc353400640 1 -- 172.21.15.43:0/343439203 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc35419f0b0 con 0x7fc354073b00 2024-03-28T22:38:06.993 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.992+0000 7fc353400640 1 --2- 172.21.15.43:0/343439203 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc354073b00 0x7fc35419a0c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc34400bf20 tx=0x7fc34400dd60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:06.994 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.993+0000 7fc34be00640 1 -- 172.21.15.43:0/343439203 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc34401c070 con 0x7fc354073b00 2024-03-28T22:38:06.994 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.993+0000 7fc34be00640 1 -- 172.21.15.43:0/343439203 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc344009d00 con 0x7fc354073b00 2024-03-28T22:38:06.994 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.993+0000 7fc34be00640 1 -- 172.21.15.43:0/343439203 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc34401fd50 con 0x7fc354073b00 2024-03-28T22:38:06.994 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.993+0000 7fc358c00640 1 -- 172.21.15.43:0/343439203 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc35419f3a0 con 0x7fc354073b00 2024-03-28T22:38:06.994 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.993+0000 7fc358c00640 1 -- 172.21.15.43:0/343439203 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc354101460 con 0x7fc354073b00 2024-03-28T22:38:06.995 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.996+0000 7fc35cf6c740 1 Processor -- start 2024-03-28T22:38:06.995 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.996+0000 7fc35cf6c740 1 -- start start 2024-03-28T22:38:06.995 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.997+0000 7fc35cf6c740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556e258e6570 0x556e258e6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.995 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.997+0000 7fc35cf6c740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556e258e6e90 0x556e258ef720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.996 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.997+0000 7fc35cf6c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556e258efc60 0x556e258f2050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.996 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.997+0000 7fc35cf6c740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x556e25849640 con 0x556e258e6570 2024-03-28T22:38:06.996 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.997+0000 7fc35cf6c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x556e257f2810 con 0x556e258efc60 2024-03-28T22:38:06.996 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.997+0000 7fc35cf6c740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x556e258456e0 con 0x556e258e6e90 2024-03-28T22:38:06.996 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.997+0000 7fc352a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556e258efc60 0x556e258f2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.996 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.997+0000 7fc352a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556e258efc60 0x556e258f2050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34274/0 (socket says 172.21.15.43:34274) 2024-03-28T22:38:06.997 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.997+0000 7fc352a00640 1 -- 172.21.15.43:0/45378912 learned_addr learned my addr 172.21.15.43:0/45378912 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:06.997 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.997+0000 7fc353e00640 1 --2- 172.21.15.43:0/45378912 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556e258e6570 0x556e258e6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:06.997 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.997+0000 7fc352a00640 1 -- 172.21.15.43:0/45378912 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556e258e6e90 msgr2=0x556e258ef720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:38:06.997 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.997+0000 7fc352a00640 1 --2- 172.21.15.43:0/45378912 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556e258e6e90 0x556e258ef720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.997 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.997+0000 7fc352a00640 1 -- 172.21.15.43:0/45378912 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556e258e6570 msgr2=0x556e258e6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:06.997 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.997+0000 7fc352a00640 1 --2- 172.21.15.43:0/45378912 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556e258e6570 0x556e258e6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.997 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.997+0000 7fc352a00640 1 -- 172.21.15.43:0/45378912 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e2555e210 con 0x556e258efc60 2024-03-28T22:38:06.997 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.998+0000 7fc352a00640 1 --2- 172.21.15.43:0/45378912 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556e258efc60 0x556e258f2050 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc340009900 tx=0x7fc340005dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c005f6e00f115235 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:06.998 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.998+0000 7fc33fe00640 1 -- 172.21.15.43:0/45378912 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc340004240 con 0x556e258efc60 2024-03-28T22:38:06.998 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.998+0000 7fc34be00640 1 -- 172.21.15.43:0/343439203 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 27) ==== 136989+0+0 (secure 0 0 0) 0x7fc34401f3b0 con 0x7fc354073b00 2024-03-28T22:38:06.998 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.998+0000 7fc33fe00640 1 -- 172.21.15.43:0/45378912 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc34002cdf0 con 0x556e258efc60 2024-03-28T22:38:06.998 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.998+0000 7fc33fe00640 1 -- 172.21.15.43:0/45378912 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc340004920 con 0x556e258efc60 2024-03-28T22:38:06.998 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.999+0000 7fc35cf6c740 1 -- 172.21.15.43:0/45378912 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556e258efc60 msgr2=0x556e258f2050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:06.998 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.999+0000 7fc35cf6c740 1 --2- 172.21.15.43:0/45378912 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556e258efc60 0x556e258f2050 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc340009900 tx=0x7fc340005dd0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.998 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.999+0000 7fc35cf6c740 1 -- 172.21.15.43:0/45378912 shutdown_connections 2024-03-28T22:38:06.999 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.999+0000 7fc35cf6c740 1 --2- 172.21.15.43:0/45378912 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556e258efc60 0x556e258f2050 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.999 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.999+0000 7fc35cf6c740 1 --2- 172.21.15.43:0/45378912 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556e258e6e90 0x556e258ef720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.999 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.999+0000 7fc35cf6c740 1 --2- 172.21.15.43:0/45378912 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556e258e6570 0x556e258e6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:06.999 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:06.999+0000 7fc35cf6c740 1 -- 172.21.15.43:0/45378912 >> 172.21.15.43:0/45378912 conn(0x556e258e0e90 msgr2=0x556e258e1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:06.999 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.000+0000 7fc34be00640 1 --2- 172.21.15.43:0/343439203 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc3380a2100 0x7fc3380a45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:06.999 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.001+0000 7fc34be00640 1 -- 172.21.15.43:0/343439203 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fc3440d6a30 con 0x7fc354073b00 2024-03-28T22:38:07.000 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.001+0000 7fc352a00640 1 --2- 172.21.15.43:0/343439203 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc3380a2100 0x7fc3380a45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:07.000 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.001+0000 7fc35cf6c740 1 -- 172.21.15.43:0/45378912 shutdown_connections 2024-03-28T22:38:07.000 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.002+0000 7fc35cf6c740 1 -- 172.21.15.43:0/45378912 wait complete. 2024-03-28T22:38:07.001 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.002+0000 7fc352a00640 1 --2- 172.21.15.43:0/343439203 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc3380a2100 0x7fc3380a45c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x556e258e22b0 tx=0x7fc34002bcc0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:07.001 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.002+0000 7fc35cf6c740 1 -- start start 2024-03-28T22:38:07.001 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.002+0000 7fc35cf6c740 1 Processor -- start 2024-03-28T22:38:07.005 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.003+0000 7fc35cf6c740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556e258e6570 0x556e258ecca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:07.005 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.003+0000 7fc35cf6c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556e258e6e90 0x556e258ed1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:07.005 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.003+0000 7fc35cf6c740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556e258efc60 0x556e258ed720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:07.005 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.003+0000 7fc35cf6c740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x556e258f2830 con 0x556e258efc60 2024-03-28T22:38:07.005 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.003+0000 7fc35cf6c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x556e257e7d00 con 0x556e258e6e90 2024-03-28T22:38:07.005 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.003+0000 7fc35cf6c740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x556e258f29b0 con 0x556e258e6570 2024-03-28T22:38:07.005 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.003+0000 7fc35cf6c740 10 client.0 _ll_register_callbacks cb 0x556e257d2490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.003+0000 7fc352a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556e258e6e90 0x556e258ed1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.003+0000 7fc352a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556e258e6e90 0x556e258ed1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34278/0 (socket says 172.21.15.43:34278) 2024-03-28T22:38:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.003+0000 7fc352a00640 1 -- 172.21.15.43:0/1957236825 learned_addr learned my addr 172.21.15.43:0/1957236825 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.004+0000 7fc353e00640 1 --2- 172.21.15.43:0/1957236825 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556e258efc60 0x556e258ed720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.004+0000 7fc353400640 1 --2- 172.21.15.43:0/1957236825 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556e258e6570 0x556e258ecca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.004+0000 7fc352a00640 1 -- 172.21.15.43:0/1957236825 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556e258e6570 msgr2=0x556e258ecca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.004+0000 7fc352a00640 1 --2- 172.21.15.43:0/1957236825 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556e258e6570 0x556e258ecca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.004+0000 7fc352a00640 1 -- 172.21.15.43:0/1957236825 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556e258efc60 msgr2=0x556e258ed720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.004+0000 7fc352a00640 1 --2- 172.21.15.43:0/1957236825 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556e258efc60 0x556e258ed720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.004+0000 7fc352a00640 1 -- 172.21.15.43:0/1957236825 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e2562a620 con 0x556e258e6e90 2024-03-28T22:38:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.004+0000 7fc352a00640 1 --2- 172.21.15.43:0/1957236825 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556e258e6e90 0x556e258ed1e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc340043000 tx=0x7fc340005dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18517b6739b3fd04 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.004+0000 7fc33cc00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:38:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.005+0000 7fc33cc00640 1 -- 172.21.15.43:0/1957236825 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc34005a030 con 0x556e258e6e90 2024-03-28T22:38:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.005+0000 7fc35cf6c740 1 -- 172.21.15.43:0/1957236825 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc34002cdf0 con 0x556e258e6e90 2024-03-28T22:38:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.005+0000 7fc33cc00640 1 -- 172.21.15.43:0/1957236825 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fc34004c030 con 0x556e258e6e90 2024-03-28T22:38:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.005+0000 7fc33cc00640 1 -- 172.21.15.43:0/1957236825 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc340065050 con 0x556e258e6e90 2024-03-28T22:38:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.005+0000 7fc35cf6c740 10 client.26066 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.005+0000 7fc35cf6c740 1 -- 172.21.15.43:0/1957236825 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556e258fe760 con 0x556e258e6e90 2024-03-28T22:38:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.005+0000 7fc33cc00640 1 -- 172.21.15.43:0/1957236825 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc34002cdf0 con 0x556e258e6e90 2024-03-28T22:38:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.006+0000 7fc33cc00640 1 -- 172.21.15.43:0/1957236825 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fc340051bc0 con 0x556e258e6e90 2024-03-28T22:38:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.006+0000 7fc35cf6c740 10 client.26066 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.006+0000 7fc35cf6c740 10 client.26066 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:38:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.006+0000 7fc35cf6c740 20 client.26066 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.006+0000 7fc35cf6c740 1 --2- 172.21.15.43:0/1957236825 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x556e258ffab0 0x556e25901ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.006+0000 7fc35cf6c740 4 client.26066 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:38:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.006+0000 7fc35cf6c740 1 -- 172.21.15.43:0/1957236825 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556e2585d030 con 0x556e258ffab0 2024-03-28T22:38:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.006+0000 7fc353400640 1 --2- 172.21.15.43:0/1957236825 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x556e258ffab0 0x556e25901ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:07.009 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.007+0000 7fc353400640 1 --2- 172.21.15.43:0/1957236825 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x556e258ffab0 0x556e25901ea0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65a58eb819e519b2 server_cookie=7c9cfff85c627ff in_seq=0 out_seq=0 2024-03-28T22:38:07.010 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.007+0000 7fc33cc00640 10 client.26066 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:07.010 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.008+0000 7fc33cc00640 1 -- 172.21.15.43:0/1957236825 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556e2585d030 con 0x556e258ffab0 2024-03-28T22:38:07.010 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.008+0000 7fc33cc00640 10 client.26066 handle_command_reply: tid=0 2024-03-28T22:38:07.010 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.008+0000 7fc35cf6c740 1 client.26066 shutdown 2024-03-28T22:38:07.010 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.008+0000 7fc35cf6c740 1 -- 172.21.15.43:0/1957236825 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556e258e6e90 msgr2=0x556e258ed1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:07.010 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.008+0000 7fc35cf6c740 1 --2- 172.21.15.43:0/1957236825 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556e258e6e90 0x556e258ed1e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc340043000 tx=0x7fc340005dd0 comp rx=0 tx=0).stop 2024-03-28T22:38:07.010 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.008+0000 7fc35cf6c740 1 -- 172.21.15.43:0/1957236825 shutdown_connections 2024-03-28T22:38:07.010 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.008+0000 7fc35cf6c740 1 --2- 172.21.15.43:0/1957236825 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x556e258ffab0 0x556e25901ea0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:07.010 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.008+0000 7fc35cf6c740 1 --2- 172.21.15.43:0/1957236825 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556e258e6e90 0x556e258ed1e0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:07.010 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.008+0000 7fc35cf6c740 1 --2- 172.21.15.43:0/1957236825 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556e258efc60 0x556e258ed720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:07.010 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.008+0000 7fc35cf6c740 1 --2- 172.21.15.43:0/1957236825 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556e258e6570 0x556e258ecca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:07.010 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.008+0000 7fc35cf6c740 1 -- 172.21.15.43:0/1957236825 >> 172.21.15.43:0/1957236825 conn(0x556e258e0e90 msgr2=0x556e258e2be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:07.010 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.008+0000 7fc33cc00640 0 client.26066 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:07.010 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.008+0000 7fc35cf6c740 1 -- 172.21.15.43:0/1957236825 shutdown_connections 2024-03-28T22:38:07.011 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.008+0000 7fc35cf6c740 1 -- 172.21.15.43:0/1957236825 wait complete. 2024-03-28T22:38:07.011 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.008+0000 7fc35cf6c740 20 client.26066 trim_cache size 0 max 16384 2024-03-28T22:38:07.037 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.038+0000 7fc35cf6c740 1 Processor -- start 2024-03-28T22:38:07.037 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.038+0000 7fc35cf6c740 1 -- start start 2024-03-28T22:38:07.038 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.038+0000 7fc35cf6c740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556e258e6570 0x556e258ef3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:07.038 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.039+0000 7fc35cf6c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556e258e6e90 0x556e258fb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:07.038 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.039+0000 7fc35cf6c740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556e258efc60 0x556e258e29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:07.038 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.039+0000 7fc35cf6c740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc3541ac940 con 0x556e258efc60 2024-03-28T22:38:07.038 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.039+0000 7fc35cf6c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fc3541ae480 con 0x556e258e6e90 2024-03-28T22:38:07.038 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.039+0000 7fc35cf6c740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fc3541ad010 con 0x556e258e6570 2024-03-28T22:38:07.038 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.039+0000 7fc352a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556e258e6e90 0x556e258fb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.039+0000 7fc353e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556e258efc60 0x556e258e29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.039+0000 7fc352a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556e258e6e90 0x556e258fb5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34290/0 (socket says 172.21.15.43:34290) 2024-03-28T22:38:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.039+0000 7fc353400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556e258e6570 0x556e258ef3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.039+0000 7fc352a00640 1 -- 172.21.15.43:0/451091062 learned_addr learned my addr 172.21.15.43:0/451091062 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.039+0000 7fc352a00640 1 -- 172.21.15.43:0/451091062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556e258e6570 msgr2=0x556e258ef3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.039+0000 7fc352a00640 1 --2- 172.21.15.43:0/451091062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556e258e6570 0x556e258ef3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.039+0000 7fc352a00640 1 -- 172.21.15.43:0/451091062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556e258efc60 msgr2=0x556e258e29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.039+0000 7fc352a00640 1 --2- 172.21.15.43:0/451091062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556e258efc60 0x556e258e29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.039+0000 7fc352a00640 1 -- 172.21.15.43:0/451091062 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e258df310 con 0x556e258e6e90 2024-03-28T22:38:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.040+0000 7fc352a00640 1 --2- 172.21.15.43:0/451091062 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556e258e6e90 0x556e258fb5d0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc34002c430 tx=0x7fc3400344c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac82199bf0b8f93c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.040+0000 7fc33f400640 1 -- 172.21.15.43:0/451091062 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc34005a070 con 0x556e258e6e90 2024-03-28T22:38:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.040+0000 7fc33f400640 1 -- 172.21.15.43:0/451091062 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc340052750 con 0x556e258e6e90 2024-03-28T22:38:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.040+0000 7fc33f400640 1 -- 172.21.15.43:0/451091062 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc340034d00 con 0x556e258e6e90 2024-03-28T22:38:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.041+0000 7fc35cf6c740 1 -- 172.21.15.43:0/451091062 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556e258e6e90 msgr2=0x556e258fb5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.041+0000 7fc35cf6c740 1 --2- 172.21.15.43:0/451091062 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556e258e6e90 0x556e258fb5d0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc34002c430 tx=0x7fc3400344c0 comp rx=0 tx=0).stop 2024-03-28T22:38:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.041+0000 7fc35cf6c740 1 -- 172.21.15.43:0/451091062 shutdown_connections 2024-03-28T22:38:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.041+0000 7fc35cf6c740 1 --2- 172.21.15.43:0/451091062 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556e258e6e90 0x556e258fb5d0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.041+0000 7fc35cf6c740 1 --2- 172.21.15.43:0/451091062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556e258efc60 0x556e258e29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.041+0000 7fc35cf6c740 1 --2- 172.21.15.43:0/451091062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556e258e6570 0x556e258ef3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.041+0000 7fc35cf6c740 1 -- 172.21.15.43:0/451091062 >> 172.21.15.43:0/451091062 conn(0x556e258e0e90 msgr2=0x556e258e22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.041+0000 7fc35cf6c740 1 -- 172.21.15.43:0/451091062 shutdown_connections 2024-03-28T22:38:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.041+0000 7fc35cf6c740 1 -- 172.21.15.43:0/451091062 wait complete. 2024-03-28T22:38:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.041+0000 7fc35cf6c740 1 -- start start 2024-03-28T22:38:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.042+0000 7fc35cf6c740 1 Processor -- start 2024-03-28T22:38:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.042+0000 7fc35cf6c740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556e258e6570 0x556e258f7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.042+0000 7fc35cf6c740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556e258e6e90 0x556e258e9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.042+0000 7fc35cf6c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556e258efc60 0x556e258e98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.042+0000 7fc35cf6c740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc3541addb0 con 0x556e258e6e90 2024-03-28T22:38:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.042+0000 7fc35cf6c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fc3541ad6e0 con 0x556e258efc60 2024-03-28T22:38:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.043+0000 7fc35cf6c740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fc3541aa940 con 0x556e258e6570 2024-03-28T22:38:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.043+0000 7fc35cf6c740 10 client.0 _ll_register_callbacks cb 0x7fc340009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.043+0000 7fc353400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556e258e6570 0x556e258f7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.043+0000 7fc353400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556e258e6570 0x556e258f7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57514/0 (socket says 172.21.15.43:57514) 2024-03-28T22:38:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.043+0000 7fc353400640 1 -- 172.21.15.43:0/995966631 learned_addr learned my addr 172.21.15.43:0/995966631 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.043+0000 7fc353e00640 1 --2- 172.21.15.43:0/995966631 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556e258efc60 0x556e258e98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.043+0000 7fc353400640 1 -- 172.21.15.43:0/995966631 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556e258efc60 msgr2=0x556e258e98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.043+0000 7fc353400640 1 --2- 172.21.15.43:0/995966631 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556e258efc60 0x556e258e98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.043+0000 7fc353400640 1 -- 172.21.15.43:0/995966631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556e258e6e90 msgr2=0x556e258e9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.043+0000 7fc352a00640 1 --2- 172.21.15.43:0/995966631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556e258e6e90 0x556e258e9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.043+0000 7fc353400640 1 --2- 172.21.15.43:0/995966631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556e258e6e90 0x556e258e9360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.043+0000 7fc353400640 1 -- 172.21.15.43:0/995966631 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e258e9e60 con 0x556e258e6570 2024-03-28T22:38:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.044+0000 7fc353400640 1 --2- 172.21.15.43:0/995966631 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556e258e6570 0x556e258f7dc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc344000c00 tx=0x7fc3440a35e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c48dccfb60e50ba8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.044+0000 7fc33cc00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:38:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.044+0000 7fc33cc00640 1 -- 172.21.15.43:0/995966631 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3440ab680 con 0x556e258e6570 2024-03-28T22:38:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.044+0000 7fc33cc00640 1 -- 172.21.15.43:0/995966631 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fc34401dd70 con 0x556e258e6570 2024-03-28T22:38:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.044+0000 7fc35cf6c740 1 -- 172.21.15.43:0/995966631 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc340052750 con 0x556e258e6570 2024-03-28T22:38:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.044+0000 7fc33cc00640 1 -- 172.21.15.43:0/995966631 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc34409f550 con 0x556e258e6570 2024-03-28T22:38:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.044+0000 7fc35cf6c740 10 client.26014 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.044+0000 7fc35cf6c740 1 -- 172.21.15.43:0/995966631 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556e259659a0 con 0x556e258e6570 2024-03-28T22:38:07.045 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.044+0000 7fc33cc00640 1 -- 172.21.15.43:0/995966631 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc340052750 con 0x556e258e6570 2024-03-28T22:38:07.045 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.045+0000 7fc33cc00640 1 -- 172.21.15.43:0/995966631 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fc3440a2330 con 0x556e258e6570 2024-03-28T22:38:07.046 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.045+0000 7fc35cf6c740 10 client.26014 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:07.046 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.045+0000 7fc35cf6c740 10 client.26014 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:38:07.046 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.045+0000 7fc35cf6c740 20 client.26014 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:07.046 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.045+0000 7fc35cf6c740 1 --2- 172.21.15.43:0/995966631 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x556e259660d0 0x556e259684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:07.046 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.045+0000 7fc35cf6c740 4 client.26014 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T22:38:07.046 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.045+0000 7fc35cf6c740 1 -- 172.21.15.43:0/995966631 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556e25856b80 con 0x556e259660d0 2024-03-28T22:38:07.046 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.045+0000 7fc352a00640 1 --2- 172.21.15.43:0/995966631 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x556e259660d0 0x556e259684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:07.046 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.046+0000 7fc352a00640 1 --2- 172.21.15.43:0/995966631 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x556e259660d0 0x556e259684f0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b73856c74ab3f54c server_cookie=e7b3d904820dc89b in_seq=0 out_seq=0 2024-03-28T22:38:07.046 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.046+0000 7fc33cc00640 10 client.26014 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:07.051 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.052+0000 7fc33cc00640 1 -- 172.21.15.43:0/995966631 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556e25856b80 con 0x556e259660d0 2024-03-28T22:38:07.051 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.052+0000 7fc33cc00640 10 client.26014 handle_command_reply: tid=0 2024-03-28T22:38:07.051 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.052+0000 7fc35cf6c740 1 client.26014 shutdown 2024-03-28T22:38:07.051 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.052+0000 7fc35cf6c740 1 -- 172.21.15.43:0/995966631 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556e258e6570 msgr2=0x556e258f7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:07.051 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.052+0000 7fc35cf6c740 1 --2- 172.21.15.43:0/995966631 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556e258e6570 0x556e258f7dc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc344000c00 tx=0x7fc3440a35e0 comp rx=0 tx=0).stop 2024-03-28T22:38:07.051 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.052+0000 7fc35cf6c740 1 -- 172.21.15.43:0/995966631 shutdown_connections 2024-03-28T22:38:07.051 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.052+0000 7fc35cf6c740 1 --2- 172.21.15.43:0/995966631 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x556e259660d0 0x556e259684f0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:07.051 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.052+0000 7fc35cf6c740 1 --2- 172.21.15.43:0/995966631 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556e258efc60 0x556e258e98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:07.051 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.052+0000 7fc35cf6c740 1 --2- 172.21.15.43:0/995966631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556e258e6e90 0x556e258e9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:07.051 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.052+0000 7fc35cf6c740 1 --2- 172.21.15.43:0/995966631 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556e258e6570 0x556e258f7dc0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:07.051 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.052+0000 7fc33cc00640 0 client.26014 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:07.052 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.052+0000 7fc35cf6c740 1 -- 172.21.15.43:0/995966631 >> 172.21.15.43:0/995966631 conn(0x556e258e0e90 msgr2=0x556e258eb190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:07.052 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.052+0000 7fc35cf6c740 1 -- 172.21.15.43:0/995966631 shutdown_connections 2024-03-28T22:38:07.052 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.052+0000 7fc35cf6c740 1 -- 172.21.15.43:0/995966631 wait complete. 2024-03-28T22:38:07.052 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.052+0000 7fc35cf6c740 20 client.26014 trim_cache size 0 max 16384 2024-03-28T22:38:07.052 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:{ 2024-03-28T22:38:07.052 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "message": "", 2024-03-28T22:38:07.052 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "return_code": 0 2024-03-28T22:38:07.052 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:} 2024-03-28T22:38:07.053 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.053+0000 7fc358c00640 1 -- 172.21.15.43:0/343439203 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc3380a2100 msgr2=0x7fc3380a45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:07.053 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.053+0000 7fc358c00640 1 --2- 172.21.15.43:0/343439203 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc3380a2100 0x7fc3380a45c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x556e258e22b0 tx=0x7fc34002bcc0 comp rx=0 tx=0).stop 2024-03-28T22:38:07.053 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.053+0000 7fc358c00640 1 -- 172.21.15.43:0/343439203 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc354073b00 msgr2=0x7fc35419a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:07.053 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.053+0000 7fc358c00640 1 --2- 172.21.15.43:0/343439203 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc354073b00 0x7fc35419a0c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc34400bf20 tx=0x7fc34400dd60 comp rx=0 tx=0).stop 2024-03-28T22:38:07.053 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.053+0000 7fc358c00640 1 -- 172.21.15.43:0/343439203 shutdown_connections 2024-03-28T22:38:07.053 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.053+0000 7fc358c00640 1 --2- 172.21.15.43:0/343439203 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc3380a2100 0x7fc3380a45c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:07.053 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.053+0000 7fc358c00640 1 --2- 172.21.15.43:0/343439203 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc35410de10 0x7fc35419e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:07.054 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.053+0000 7fc358c00640 1 --2- 172.21.15.43:0/343439203 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc354074420 0x7fc35419a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:07.054 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.053+0000 7fc358c00640 1 --2- 172.21.15.43:0/343439203 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc354073b00 0x7fc35419a0c0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:07.054 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.053+0000 7fc358c00640 1 -- 172.21.15.43:0/343439203 >> 172.21.15.43:0/343439203 conn(0x7fc3540fd060 msgr2=0x7fc3540fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:07.054 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.054+0000 7fc358c00640 1 -- 172.21.15.43:0/343439203 shutdown_connections 2024-03-28T22:38:07.054 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:07.054+0000 7fc358c00640 1 -- 172.21.15.43:0/343439203 wait complete. 2024-03-28T22:38:07.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:07 smithi172 ceph-mon[36566]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-03-28T22:38:07.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:07 smithi082 ceph-mon[36471]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-03-28T22:38:07.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:07 smithi043 ceph-mon[31823]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-03-28T22:38:07.971 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:38:07.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:38:07.972 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:38:08.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.268+0000 7f99bee00640 1 -- 172.21.15.43:0/306052975 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99b810de60 msgr2=0x7f99b8110270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:08.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.268+0000 7f99bee00640 1 --2- 172.21.15.43:0/306052975 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99b810de60 0x7f99b8110270 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f99b0009ca0 tx=0x7f99b002c180 comp rx=0 tx=0).stop 2024-03-28T22:38:08.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.268+0000 7f99bee00640 1 -- 172.21.15.43:0/306052975 shutdown_connections 2024-03-28T22:38:08.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.268+0000 7f99bee00640 1 --2- 172.21.15.43:0/306052975 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f99b8074440 0x7f99b810d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.268+0000 7f99bee00640 1 --2- 172.21.15.43:0/306052975 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99b810de60 0x7f99b8110270 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.268+0000 7f99bee00640 1 --2- 172.21.15.43:0/306052975 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99b8073b20 0x7f99b8073f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.268+0000 7f99bee00640 1 -- 172.21.15.43:0/306052975 >> 172.21.15.43:0/306052975 conn(0x7f99b80fd0f0 msgr2=0x7f99b80ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:08.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.268+0000 7f99bee00640 1 -- 172.21.15.43:0/306052975 shutdown_connections 2024-03-28T22:38:08.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.268+0000 7f99bee00640 1 -- 172.21.15.43:0/306052975 wait complete. 2024-03-28T22:38:08.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.270+0000 7f99bee00640 1 Processor -- start 2024-03-28T22:38:08.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.270+0000 7f99bee00640 1 -- start start 2024-03-28T22:38:08.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.270+0000 7f99bee00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99b8073b20 0x7f99b819a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.271+0000 7f99bee00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99b8074440 0x7f99b819a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.271+0000 7f99bee00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f99b810de60 0x7f99b819e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.271+0000 7f99bee00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f99b8113bc0 con 0x7f99b8073b20 2024-03-28T22:38:08.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.271+0000 7f99bee00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f99b8113a40 con 0x7f99b810de60 2024-03-28T22:38:08.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.271+0000 7f99bee00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f99b8113d40 con 0x7f99b8074440 2024-03-28T22:38:08.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.271+0000 7f99be400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f99b810de60 0x7f99b819e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.271+0000 7f99bda00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99b8073b20 0x7f99b819a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.271+0000 7f99bd000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99b8074440 0x7f99b819a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.271+0000 7f99bda00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99b8073b20 0x7f99b819a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56032/0 (socket says 172.21.15.43:56032) 2024-03-28T22:38:08.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.271+0000 7f99be400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f99b810de60 0x7f99b819e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34310/0 (socket says 172.21.15.43:34310) 2024-03-28T22:38:08.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.271+0000 7f99bda00640 1 -- 172.21.15.43:0/2063204070 learned_addr learned my addr 172.21.15.43:0/2063204070 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:08.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.271+0000 7f99bd000640 1 -- 172.21.15.43:0/2063204070 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f99b810de60 msgr2=0x7f99b819e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:08.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.271+0000 7f99bd000640 1 --2- 172.21.15.43:0/2063204070 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f99b810de60 0x7f99b819e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.271+0000 7f99bd000640 1 -- 172.21.15.43:0/2063204070 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99b8073b20 msgr2=0x7f99b819a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:08.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.271+0000 7f99bd000640 1 --2- 172.21.15.43:0/2063204070 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99b8073b20 0x7f99b819a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.271+0000 7f99bd000640 1 -- 172.21.15.43:0/2063204070 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99b819f0b0 con 0x7f99b8074440 2024-03-28T22:38:08.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.272+0000 7f99bd000640 1 --2- 172.21.15.43:0/2063204070 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99b8074440 0x7f99b819a600 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f99a400e9f0 tx=0x7f99a400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:08.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.272+0000 7f99b6000640 1 -- 172.21.15.43:0/2063204070 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a400ccc0 con 0x7f99b8074440 2024-03-28T22:38:08.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.272+0000 7f99b6000640 1 -- 172.21.15.43:0/2063204070 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99a4005020 con 0x7f99b8074440 2024-03-28T22:38:08.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.272+0000 7f99b6000640 1 -- 172.21.15.43:0/2063204070 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a40049e0 con 0x7f99b8074440 2024-03-28T22:38:08.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.272+0000 7f99bee00640 1 -- 172.21.15.43:0/2063204070 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99b819f3a0 con 0x7f99b8074440 2024-03-28T22:38:08.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.272+0000 7f99bee00640 1 -- 172.21.15.43:0/2063204070 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99b81a6c30 con 0x7f99b8074440 2024-03-28T22:38:08.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.277+0000 7f99c301b740 1 Processor -- start 2024-03-28T22:38:08.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.277+0000 7f99c301b740 1 -- start start 2024-03-28T22:38:08.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.278+0000 7f99c301b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dd7810e170 0x55dd7810e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.278+0000 7f99c301b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dd7810ea90 0x55dd78117130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.278+0000 7f99c301b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd78117670 0x55dd78119a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.278+0000 7f99c301b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55dd78071c40 con 0x55dd7810ea90 2024-03-28T22:38:08.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.278+0000 7f99c301b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55dd7801ac40 con 0x55dd78117670 2024-03-28T22:38:08.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.278+0000 7f99c301b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55dd7806dce0 con 0x55dd7810e170 2024-03-28T22:38:08.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.278+0000 7f99b6000640 1 -- 172.21.15.43:0/2063204070 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 27) ==== 136989+0+0 (secure 0 0 0) 0x7f99a4005560 con 0x7f99b8074440 2024-03-28T22:38:08.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.278+0000 7f99be400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dd7810e170 0x55dd7810e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.278+0000 7f99bd000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd78117670 0x55dd78119a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.278+0000 7f99be400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dd7810e170 0x55dd7810e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57540/0 (socket says 172.21.15.43:57540) 2024-03-28T22:38:08.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.278+0000 7f99be400640 1 -- 172.21.15.43:0/1689908158 learned_addr learned my addr 172.21.15.43:0/1689908158 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:08.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.279+0000 7f99bda00640 1 --2- 172.21.15.43:0/1689908158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dd7810ea90 0x55dd78117130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.279+0000 7f99be400640 1 -- 172.21.15.43:0/1689908158 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd78117670 msgr2=0x55dd78119a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:08.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.279+0000 7f99be400640 1 --2- 172.21.15.43:0/1689908158 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd78117670 0x55dd78119a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.279+0000 7f99be400640 1 -- 172.21.15.43:0/1689908158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dd7810ea90 msgr2=0x55dd78117130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:08.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.279+0000 7f99be400640 1 --2- 172.21.15.43:0/1689908158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dd7810ea90 0x55dd78117130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.279+0000 7f99be400640 1 -- 172.21.15.43:0/1689908158 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd77e4e7e0 con 0x55dd7810e170 2024-03-28T22:38:08.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.279+0000 7f99bda00640 1 --2- 172.21.15.43:0/1689908158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dd7810ea90 0x55dd78117130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:38:08.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.279+0000 7f99be400640 1 --2- 172.21.15.43:0/1689908158 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dd7810e170 0x55dd7810e550 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f99b0005e30 tx=0x7f99b0005f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1cafbd3a38a2017d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:08.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.280+0000 7f99aaa00640 1 -- 172.21.15.43:0/1689908158 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b0043070 con 0x55dd7810e170 2024-03-28T22:38:08.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.280+0000 7f99aaa00640 1 -- 172.21.15.43:0/1689908158 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99b0003e50 con 0x55dd7810e170 2024-03-28T22:38:08.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.280+0000 7f99aaa00640 1 -- 172.21.15.43:0/1689908158 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b0039040 con 0x55dd7810e170 2024-03-28T22:38:08.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.280+0000 7f99b6000640 1 --2- 172.21.15.43:0/2063204070 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f99a00a2010 0x7f99a00a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.281+0000 7f99b6000640 1 -- 172.21.15.43:0/2063204070 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f99a40ce6e0 con 0x7f99b8074440 2024-03-28T22:38:08.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.281+0000 7f99c301b740 1 -- 172.21.15.43:0/1689908158 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dd7810e170 msgr2=0x55dd7810e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:08.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.281+0000 7f99c301b740 1 --2- 172.21.15.43:0/1689908158 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dd7810e170 0x55dd7810e550 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f99b0005e30 tx=0x7f99b0005f80 comp rx=0 tx=0).stop 2024-03-28T22:38:08.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.281+0000 7f99bda00640 1 --2- 172.21.15.43:0/2063204070 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f99a00a2010 0x7f99a00a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.281+0000 7f99c301b740 1 -- 172.21.15.43:0/1689908158 shutdown_connections 2024-03-28T22:38:08.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.281+0000 7f99c301b740 1 --2- 172.21.15.43:0/1689908158 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd78117670 0x55dd78119a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.281+0000 7f99c301b740 1 --2- 172.21.15.43:0/1689908158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dd7810ea90 0x55dd78117130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.281+0000 7f99c301b740 1 --2- 172.21.15.43:0/1689908158 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dd7810e170 0x55dd7810e550 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.281+0000 7f99c301b740 1 -- 172.21.15.43:0/1689908158 >> 172.21.15.43:0/1689908158 conn(0x55dd781088a0 msgr2=0x55dd78108c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:08.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.281+0000 7f99c301b740 1 -- 172.21.15.43:0/1689908158 shutdown_connections 2024-03-28T22:38:08.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.283+0000 7f99c301b740 1 -- 172.21.15.43:0/1689908158 wait complete. 2024-03-28T22:38:08.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.284+0000 7f99bda00640 1 --2- 172.21.15.43:0/2063204070 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f99a00a2010 0x7f99a00a44d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x55dd7810dc40 tx=0x7f99ac005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:08.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.284+0000 7f99c301b740 1 -- start start 2024-03-28T22:38:08.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.284+0000 7f99c301b740 1 Processor -- start 2024-03-28T22:38:08.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.285+0000 7f99c301b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dd7810e170 0x55dd781146f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.285+0000 7f99c301b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd7810ea90 0x55dd78114c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.285+0000 7f99c301b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dd78117670 0x55dd78115170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.285+0000 7f99c301b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55dd7811a400 con 0x55dd7810e170 2024-03-28T22:38:08.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.285+0000 7f99c301b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55dd78010250 con 0x55dd7810ea90 2024-03-28T22:38:08.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.285+0000 7f99c301b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55dd7811a580 con 0x55dd78117670 2024-03-28T22:38:08.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.285+0000 7f99c301b740 10 client.0 _ll_register_callbacks cb 0x55dd78031e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:08.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.285+0000 7f99be400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dd78117670 0x55dd78115170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.285+0000 7f99be400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dd78117670 0x55dd78115170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57554/0 (socket says 172.21.15.43:57554) 2024-03-28T22:38:08.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.285+0000 7f99be400640 1 -- 172.21.15.43:0/3542385736 learned_addr learned my addr 172.21.15.43:0/3542385736 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:08.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.285+0000 7f99bda00640 1 --2- 172.21.15.43:0/3542385736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dd7810e170 0x55dd781146f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.286+0000 7f99bd000640 1 --2- 172.21.15.43:0/3542385736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd7810ea90 0x55dd78114c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.286+0000 7f99be400640 1 -- 172.21.15.43:0/3542385736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd7810ea90 msgr2=0x55dd78114c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:08.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.286+0000 7f99be400640 1 --2- 172.21.15.43:0/3542385736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd7810ea90 0x55dd78114c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.286+0000 7f99be400640 1 -- 172.21.15.43:0/3542385736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dd7810e170 msgr2=0x55dd781146f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:08.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.286+0000 7f99be400640 1 --2- 172.21.15.43:0/3542385736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dd7810e170 0x55dd781146f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.286+0000 7f99be400640 1 -- 172.21.15.43:0/3542385736 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd77eaef70 con 0x55dd78117670 2024-03-28T22:38:08.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.286+0000 7f99bd000640 1 --2- 172.21.15.43:0/3542385736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd7810ea90 0x55dd78114c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:38:08.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.286+0000 7f99bda00640 1 --2- 172.21.15.43:0/3542385736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dd7810e170 0x55dd781146f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:38:08.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.286+0000 7f99be400640 1 --2- 172.21.15.43:0/3542385736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dd78117670 0x55dd78115170 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f99b0003d60 tx=0x7f99b0003d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f453b52d75ef04f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:08.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.286+0000 7f999f400640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:38:08.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.286+0000 7f999f400640 1 -- 172.21.15.43:0/3542385736 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b004c030 con 0x55dd78117670 2024-03-28T22:38:08.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.286+0000 7f999f400640 1 -- 172.21.15.43:0/3542385736 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f99b0004000 con 0x55dd78117670 2024-03-28T22:38:08.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.287+0000 7f99c301b740 1 -- 172.21.15.43:0/3542385736 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99b0003e50 con 0x55dd78117670 2024-03-28T22:38:08.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.287+0000 7f999f400640 1 -- 172.21.15.43:0/3542385736 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f99b0057050 con 0x55dd78117670 2024-03-28T22:38:08.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.287+0000 7f99c301b740 10 client.26026 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:08.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.287+0000 7f99c301b740 1 -- 172.21.15.43:0/3542385736 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55dd77d46170 con 0x55dd78117670 2024-03-28T22:38:08.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.288+0000 7f999f400640 1 -- 172.21.15.43:0/3542385736 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99b0003e50 con 0x55dd78117670 2024-03-28T22:38:08.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.288+0000 7f999f400640 1 -- 172.21.15.43:0/3542385736 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f99b0048450 con 0x55dd78117670 2024-03-28T22:38:08.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.288+0000 7f99c301b740 10 client.26026 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:08.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.288+0000 7f99c301b740 10 client.26026 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:38:08.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.288+0000 7f99c301b740 20 client.26026 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:08.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.288+0000 7f99c301b740 1 --2- 172.21.15.43:0/3542385736 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55dd78127870 0x55dd78129c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.288+0000 7f99c301b740 4 client.26026 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:38:08.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.288+0000 7f99c301b740 1 -- 172.21.15.43:0/3542385736 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd7807f180 con 0x55dd78127870 2024-03-28T22:38:08.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.289+0000 7f99bda00640 1 --2- 172.21.15.43:0/3542385736 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55dd78127870 0x55dd78129c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.289+0000 7f99bda00640 1 --2- 172.21.15.43:0/3542385736 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55dd78127870 0x55dd78129c60 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc1a019fae5c2663 server_cookie=127f1044de0e5792 in_seq=0 out_seq=0 2024-03-28T22:38:08.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.289+0000 7f999f400640 10 client.26026 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:08.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.290+0000 7f999f400640 1 -- 172.21.15.43:0/3542385736 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dd7807f180 con 0x55dd78127870 2024-03-28T22:38:08.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.290+0000 7f999f400640 10 client.26026 handle_command_reply: tid=0 2024-03-28T22:38:08.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.290+0000 7f99c301b740 1 client.26026 shutdown 2024-03-28T22:38:08.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.291+0000 7f99c301b740 1 -- 172.21.15.43:0/3542385736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dd78117670 msgr2=0x55dd78115170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:08.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.291+0000 7f99c301b740 1 --2- 172.21.15.43:0/3542385736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dd78117670 0x55dd78115170 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f99b0003d60 tx=0x7f99b0003d90 comp rx=0 tx=0).stop 2024-03-28T22:38:08.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.291+0000 7f99c301b740 1 -- 172.21.15.43:0/3542385736 shutdown_connections 2024-03-28T22:38:08.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.291+0000 7f99c301b740 1 --2- 172.21.15.43:0/3542385736 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55dd78127870 0x55dd78129c60 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.291+0000 7f99c301b740 1 --2- 172.21.15.43:0/3542385736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd7810ea90 0x55dd78114c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.291+0000 7f99c301b740 1 --2- 172.21.15.43:0/3542385736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dd78117670 0x55dd78115170 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.291+0000 7f99c301b740 1 --2- 172.21.15.43:0/3542385736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dd7810e170 0x55dd781146f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.291+0000 7f99c301b740 1 -- 172.21.15.43:0/3542385736 >> 172.21.15.43:0/3542385736 conn(0x55dd781088a0 msgr2=0x55dd7810a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:08.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.291+0000 7f999f400640 0 client.26026 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:08.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.292+0000 7f99c301b740 1 -- 172.21.15.43:0/3542385736 shutdown_connections 2024-03-28T22:38:08.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.292+0000 7f99c301b740 1 -- 172.21.15.43:0/3542385736 wait complete. 2024-03-28T22:38:08.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.292+0000 7f99c301b740 20 client.26026 trim_cache size 0 max 16384 2024-03-28T22:38:08.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.329+0000 7f99c301b740 1 Processor -- start 2024-03-28T22:38:08.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.329+0000 7f99c301b740 1 -- start start 2024-03-28T22:38:08.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.330+0000 7f99c301b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd7810e170 0x55dd781231a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.330+0000 7f99c301b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dd7810ea90 0x55dd78114f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.330+0000 7f99c301b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dd78117670 0x55dd7812f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.330+0000 7f99c301b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f99b81b0b00 con 0x55dd7810ea90 2024-03-28T22:38:08.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.330+0000 7f99c301b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f99b81b2640 con 0x55dd7810e170 2024-03-28T22:38:08.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.330+0000 7f99c301b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f99b81b11d0 con 0x55dd78117670 2024-03-28T22:38:08.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.330+0000 7f99bda00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd7810e170 0x55dd781231a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.330+0000 7f99bda00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd7810e170 0x55dd781231a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34334/0 (socket says 172.21.15.43:34334) 2024-03-28T22:38:08.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.330+0000 7f99bda00640 1 -- 172.21.15.43:0/913722597 learned_addr learned my addr 172.21.15.43:0/913722597 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:08.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.330+0000 7f99be400640 1 --2- 172.21.15.43:0/913722597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dd78117670 0x55dd7812f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.330+0000 7f99bd000640 1 --2- 172.21.15.43:0/913722597 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dd7810ea90 0x55dd78114f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.331+0000 7f99bda00640 1 -- 172.21.15.43:0/913722597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dd78117670 msgr2=0x55dd7812f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:08.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.331+0000 7f99bda00640 1 --2- 172.21.15.43:0/913722597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dd78117670 0x55dd7812f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.331+0000 7f99bda00640 1 -- 172.21.15.43:0/913722597 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dd7810ea90 msgr2=0x55dd78114f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:08.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.331+0000 7f99bda00640 1 --2- 172.21.15.43:0/913722597 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dd7810ea90 0x55dd78114f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.331+0000 7f99bda00640 1 -- 172.21.15.43:0/913722597 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd7811a520 con 0x55dd7810e170 2024-03-28T22:38:08.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.331+0000 7f99be400640 1 --2- 172.21.15.43:0/913722597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dd78117670 0x55dd7812f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:38:08.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.331+0000 7f99bda00640 1 --2- 172.21.15.43:0/913722597 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd7810e170 0x55dd781231a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f99ac005eb0 tx=0x7f99ac019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcbeb330a4d4adf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:08.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.331+0000 7f99aa000640 1 -- 172.21.15.43:0/913722597 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99ac02a500 con 0x55dd7810e170 2024-03-28T22:38:08.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.331+0000 7f99aa000640 1 -- 172.21.15.43:0/913722597 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99ac030070 con 0x55dd7810e170 2024-03-28T22:38:08.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.331+0000 7f99aa000640 1 -- 172.21.15.43:0/913722597 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99ac00f7e0 con 0x55dd7810e170 2024-03-28T22:38:08.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.332+0000 7f99c301b740 1 -- 172.21.15.43:0/913722597 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd7810e170 msgr2=0x55dd781231a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:08.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.332+0000 7f99c301b740 1 --2- 172.21.15.43:0/913722597 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd7810e170 0x55dd781231a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f99ac005eb0 tx=0x7f99ac019000 comp rx=0 tx=0).stop 2024-03-28T22:38:08.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.332+0000 7f99c301b740 1 -- 172.21.15.43:0/913722597 shutdown_connections 2024-03-28T22:38:08.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.332+0000 7f99c301b740 1 --2- 172.21.15.43:0/913722597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dd78117670 0x55dd7812f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.332+0000 7f99c301b740 1 --2- 172.21.15.43:0/913722597 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dd7810ea90 0x55dd78114f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.332+0000 7f99c301b740 1 --2- 172.21.15.43:0/913722597 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd7810e170 0x55dd781231a0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.332+0000 7f99c301b740 1 -- 172.21.15.43:0/913722597 >> 172.21.15.43:0/913722597 conn(0x55dd781088a0 msgr2=0x55dd78107910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:08.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.333+0000 7f99c301b740 1 -- 172.21.15.43:0/913722597 shutdown_connections 2024-03-28T22:38:08.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.333+0000 7f99c301b740 1 -- 172.21.15.43:0/913722597 wait complete. 2024-03-28T22:38:08.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.333+0000 7f99c301b740 1 -- start start 2024-03-28T22:38:08.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.333+0000 7f99c301b740 1 Processor -- start 2024-03-28T22:38:08.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.334+0000 7f99c301b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dd7810e170 0x55dd7818e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.334+0000 7f99c301b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd7810ea90 0x55dd7818e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.334+0000 7f99c301b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dd78117670 0x55dd7811a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.334+0000 7f99c301b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f99b81b1f70 con 0x55dd7810e170 2024-03-28T22:38:08.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.334+0000 7f99c301b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f99b81b18a0 con 0x55dd7810ea90 2024-03-28T22:38:08.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.334+0000 7f99c301b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f99b81aeae0 con 0x55dd78117670 2024-03-28T22:38:08.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.334+0000 7f99c301b740 10 client.0 _ll_register_callbacks cb 0x7f99a40d43e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:08.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.334+0000 7f99bd000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd7810ea90 0x55dd7818e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.335+0000 7f99bd000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd7810ea90 0x55dd7818e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34346/0 (socket says 172.21.15.43:34346) 2024-03-28T22:38:08.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.335+0000 7f99bd000640 1 -- 172.21.15.43:0/3275000376 learned_addr learned my addr 172.21.15.43:0/3275000376 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:08.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.335+0000 7f99be400640 1 --2- 172.21.15.43:0/3275000376 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dd78117670 0x55dd7811a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.335+0000 7f99bda00640 1 --2- 172.21.15.43:0/3275000376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dd7810e170 0x55dd7818e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.335+0000 7f99bd000640 1 -- 172.21.15.43:0/3275000376 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dd78117670 msgr2=0x55dd7811a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:08.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.335+0000 7f99bd000640 1 --2- 172.21.15.43:0/3275000376 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dd78117670 0x55dd7811a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.335+0000 7f99bd000640 1 -- 172.21.15.43:0/3275000376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dd7810e170 msgr2=0x55dd7818e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:08.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.335+0000 7f99bd000640 1 --2- 172.21.15.43:0/3275000376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dd7810e170 0x55dd7818e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.335+0000 7f99bd000640 1 -- 172.21.15.43:0/3275000376 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd7811afe0 con 0x55dd7810ea90 2024-03-28T22:38:08.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.335+0000 7f99be400640 1 --2- 172.21.15.43:0/3275000376 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dd78117670 0x55dd7811a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:08.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.336+0000 7f99bd000640 1 --2- 172.21.15.43:0/3275000376 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd7810ea90 0x55dd7818e980 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f99a40cc090 tx=0x7f99a4096d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e710ba0bda0d68f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:08.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.336+0000 7f999f400640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:38:08.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.336+0000 7f999f400640 1 -- 172.21.15.43:0/3275000376 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a401c030 con 0x55dd7810ea90 2024-03-28T22:38:08.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.336+0000 7f999f400640 1 -- 172.21.15.43:0/3275000376 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f99a409ad40 con 0x55dd7810ea90 2024-03-28T22:38:08.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.336+0000 7f99c301b740 1 -- 172.21.15.43:0/3275000376 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99ac030070 con 0x55dd7810ea90 2024-03-28T22:38:08.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.336+0000 7f999f400640 1 -- 172.21.15.43:0/3275000376 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f99a409a440 con 0x55dd7810ea90 2024-03-28T22:38:08.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.336+0000 7f99c301b740 10 client.26087 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:08.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.337+0000 7f99c301b740 1 -- 172.21.15.43:0/3275000376 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55dd78195c80 con 0x55dd7810ea90 2024-03-28T22:38:08.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.337+0000 7f999f400640 1 -- 172.21.15.43:0/3275000376 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99ac030070 con 0x55dd7810ea90 2024-03-28T22:38:08.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.337+0000 7f999f400640 1 -- 172.21.15.43:0/3275000376 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f99a40a2df0 con 0x55dd7810ea90 2024-03-28T22:38:08.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.337+0000 7f99c301b740 10 client.26087 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:08.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.337+0000 7f99c301b740 10 client.26087 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:38:08.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.337+0000 7f99c301b740 20 client.26087 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:08.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.337+0000 7f99c301b740 1 --2- 172.21.15.43:0/3275000376 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55dd781963d0 0x55dd781987f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.337+0000 7f99c301b740 4 client.26087 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:38:08.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.337+0000 7f99c301b740 1 -- 172.21.15.43:0/3275000376 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55dd780794d0 con 0x55dd781963d0 2024-03-28T22:38:08.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.337+0000 7f99bda00640 1 --2- 172.21.15.43:0/3275000376 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55dd781963d0 0x55dd781987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.338+0000 7f99bda00640 1 --2- 172.21.15.43:0/3275000376 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55dd781963d0 0x55dd781987f0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c228f62195cda1d5 server_cookie=cc4bd3ba5b470575 in_seq=0 out_seq=0 2024-03-28T22:38:08.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.338+0000 7f999f400640 10 client.26087 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:08.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.339+0000 7f999f400640 1 -- 172.21.15.43:0/3275000376 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55dd780794d0 con 0x55dd781963d0 2024-03-28T22:38:08.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.339+0000 7f999f400640 10 client.26087 handle_command_reply: tid=0 2024-03-28T22:38:08.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.339+0000 7f99c301b740 1 client.26087 shutdown 2024-03-28T22:38:08.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.339+0000 7f99c301b740 1 -- 172.21.15.43:0/3275000376 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd7810ea90 msgr2=0x55dd7818e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:08.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.339+0000 7f99c301b740 1 --2- 172.21.15.43:0/3275000376 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd7810ea90 0x55dd7818e980 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f99a40cc090 tx=0x7f99a4096d80 comp rx=0 tx=0).stop 2024-03-28T22:38:08.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.339+0000 7f99c301b740 1 -- 172.21.15.43:0/3275000376 shutdown_connections 2024-03-28T22:38:08.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.339+0000 7f99c301b740 1 --2- 172.21.15.43:0/3275000376 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55dd781963d0 0x55dd781987f0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.339+0000 7f99c301b740 1 --2- 172.21.15.43:0/3275000376 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd7810ea90 0x55dd7818e980 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.339+0000 7f99c301b740 1 --2- 172.21.15.43:0/3275000376 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dd78117670 0x55dd7811a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.339+0000 7f99c301b740 1 --2- 172.21.15.43:0/3275000376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dd7810e170 0x55dd7818e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.339+0000 7f99c301b740 1 -- 172.21.15.43:0/3275000376 >> 172.21.15.43:0/3275000376 conn(0x55dd781088a0 msgr2=0x55dd7810a430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:08.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.339+0000 7f999f400640 0 client.26087 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:08.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.340+0000 7f99c301b740 1 -- 172.21.15.43:0/3275000376 shutdown_connections 2024-03-28T22:38:08.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.340+0000 7f99c301b740 1 -- 172.21.15.43:0/3275000376 wait complete. 2024-03-28T22:38:08.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.340+0000 7f99c301b740 20 client.26087 trim_cache size 0 max 16384 2024-03-28T22:38:08.340 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:38:08.340 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-03-28T22:38:08.341 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "48014d05-475d-4d9d-8b39-95b8d44ed9d7", 2024-03-28T22:38:08.341 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-03-28T22:38:08.341 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:38:08.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.341+0000 7f99bee00640 1 -- 172.21.15.43:0/2063204070 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f99a00a2010 msgr2=0x7f99a00a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:08.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.341+0000 7f99bee00640 1 --2- 172.21.15.43:0/2063204070 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f99a00a2010 0x7f99a00a44d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x55dd7810dc40 tx=0x7f99ac005f90 comp rx=0 tx=0).stop 2024-03-28T22:38:08.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.341+0000 7f99bee00640 1 -- 172.21.15.43:0/2063204070 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99b8074440 msgr2=0x7f99b819a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:08.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.341+0000 7f99bee00640 1 --2- 172.21.15.43:0/2063204070 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99b8074440 0x7f99b819a600 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f99a400e9f0 tx=0x7f99a400eeb0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.341+0000 7f99bee00640 1 -- 172.21.15.43:0/2063204070 shutdown_connections 2024-03-28T22:38:08.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.341+0000 7f99bee00640 1 --2- 172.21.15.43:0/2063204070 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f99a00a2010 0x7f99a00a44d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.341+0000 7f99bee00640 1 --2- 172.21.15.43:0/2063204070 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f99b810de60 0x7f99b819e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.341+0000 7f99bee00640 1 --2- 172.21.15.43:0/2063204070 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f99b8074440 0x7f99b819a600 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.341+0000 7f99bee00640 1 --2- 172.21.15.43:0/2063204070 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f99b8073b20 0x7f99b819a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.341+0000 7f99bee00640 1 -- 172.21.15.43:0/2063204070 >> 172.21.15.43:0/2063204070 conn(0x7f99b80fd0f0 msgr2=0x7f99b80fde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:08.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.342+0000 7f99bee00640 1 -- 172.21.15.43:0/2063204070 shutdown_connections 2024-03-28T22:38:08.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.342+0000 7f99bee00640 1 -- 172.21.15.43:0/2063204070 wait complete. 2024-03-28T22:38:08.364 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:38:08.364 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:38:08.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.558+0000 7fd17ca00640 1 -- 172.21.15.43:0/1857827629 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd178107660 msgr2=0x7fd178107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:08.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.558+0000 7fd17ca00640 1 --2- 172.21.15.43:0/1857827629 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd178107660 0x7fd178107a40 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fd168009900 tx=0x7fd16802bc30 comp rx=0 tx=0).stop 2024-03-28T22:38:08.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.558+0000 7fd17ca00640 1 -- 172.21.15.43:0/1857827629 shutdown_connections 2024-03-28T22:38:08.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.558+0000 7fd17ca00640 1 --2- 172.21.15.43:0/1857827629 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd178101f80 0x7fd17810e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.558+0000 7fd17ca00640 1 --2- 172.21.15.43:0/1857827629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd1781015c0 0x7fd178101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.558+0000 7fd17ca00640 1 --2- 172.21.15.43:0/1857827629 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd178107660 0x7fd178107a40 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.558+0000 7fd17ca00640 1 -- 172.21.15.43:0/1857827629 >> 172.21.15.43:0/1857827629 conn(0x7fd1780fd0f0 msgr2=0x7fd1780ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:08.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.558+0000 7fd17ca00640 1 -- 172.21.15.43:0/1857827629 shutdown_connections 2024-03-28T22:38:08.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.558+0000 7fd17ca00640 1 -- 172.21.15.43:0/1857827629 wait complete. 2024-03-28T22:38:08.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.559+0000 7fd17ca00640 1 Processor -- start 2024-03-28T22:38:08.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.559+0000 7fd17ca00640 1 -- start start 2024-03-28T22:38:08.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.560+0000 7fd17ca00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1781015c0 0x7fd17819a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.560+0000 7fd17ca00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd178101f80 0x7fd17819a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.560+0000 7fd17ca00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd178107660 0x7fd17819e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.560+0000 7fd17ca00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd1781119f0 con 0x7fd178101f80 2024-03-28T22:38:08.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.560+0000 7fd17ca00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd178111870 con 0x7fd178107660 2024-03-28T22:38:08.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.560+0000 7fd17ca00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd178111b70 con 0x7fd1781015c0 2024-03-28T22:38:08.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.560+0000 7fd177400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1781015c0 0x7fd17819a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.560+0000 7fd177400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1781015c0 0x7fd17819a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57602/0 (socket says 172.21.15.43:57602) 2024-03-28T22:38:08.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.560+0000 7fd177400640 1 -- 172.21.15.43:0/4040965328 learned_addr learned my addr 172.21.15.43:0/4040965328 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:08.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.560+0000 7fd176a00640 1 --2- 172.21.15.43:0/4040965328 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd178101f80 0x7fd17819a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.560+0000 7fd177e00640 1 --2- 172.21.15.43:0/4040965328 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd178107660 0x7fd17819e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.560+0000 7fd177400640 1 -- 172.21.15.43:0/4040965328 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd178107660 msgr2=0x7fd17819e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:08.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.560+0000 7fd177400640 1 --2- 172.21.15.43:0/4040965328 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd178107660 0x7fd17819e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.560+0000 7fd177400640 1 -- 172.21.15.43:0/4040965328 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd178101f80 msgr2=0x7fd17819a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:08.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.560+0000 7fd177400640 1 --2- 172.21.15.43:0/4040965328 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd178101f80 0x7fd17819a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.560+0000 7fd177400640 1 -- 172.21.15.43:0/4040965328 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd17819f010 con 0x7fd1781015c0 2024-03-28T22:38:08.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.560+0000 7fd176a00640 1 --2- 172.21.15.43:0/4040965328 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd178101f80 0x7fd17819a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:38:08.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.561+0000 7fd177400640 1 --2- 172.21.15.43:0/4040965328 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1781015c0 0x7fd17819a020 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd16802c240 tx=0x7fd1680043b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:08.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.561+0000 7fd16fe00640 1 -- 172.21.15.43:0/4040965328 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd168041070 con 0x7fd1781015c0 2024-03-28T22:38:08.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.561+0000 7fd17ca00640 1 -- 172.21.15.43:0/4040965328 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd17819f2a0 con 0x7fd1781015c0 2024-03-28T22:38:08.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.561+0000 7fd16fe00640 1 -- 172.21.15.43:0/4040965328 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1680044a0 con 0x7fd1781015c0 2024-03-28T22:38:08.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.561+0000 7fd16fe00640 1 -- 172.21.15.43:0/4040965328 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd168003d10 con 0x7fd1781015c0 2024-03-28T22:38:08.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.561+0000 7fd17ca00640 1 -- 172.21.15.43:0/4040965328 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1781a6b90 con 0x7fd1781015c0 2024-03-28T22:38:08.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.564+0000 7fd180ddb740 1 Processor -- start 2024-03-28T22:38:08.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.565+0000 7fd180ddb740 1 -- start start 2024-03-28T22:38:08.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.565+0000 7fd180ddb740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac8dc35db0 0x55ac8dc34b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.565+0000 7fd180ddb740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac8dc350d0 0x55ac8dc35550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.565+0000 7fd180ddb740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac8dc3e440 0x55ac8dc40830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.565+0000 7fd180ddb740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ac8db99100 con 0x55ac8dc35db0 2024-03-28T22:38:08.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.565+0000 7fd180ddb740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55ac8db41ba0 con 0x55ac8dc350d0 2024-03-28T22:38:08.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.565+0000 7fd180ddb740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55ac8db951a0 con 0x55ac8dc3e440 2024-03-28T22:38:08.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.565+0000 7fd177e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac8dc35db0 0x55ac8dc34b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.566+0000 7fd177e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac8dc35db0 0x55ac8dc34b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56086/0 (socket says 172.21.15.43:56086) 2024-03-28T22:38:08.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.566+0000 7fd177e00640 1 -- 172.21.15.43:0/697917340 learned_addr learned my addr 172.21.15.43:0/697917340 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:08.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.566+0000 7fd176a00640 1 --2- 172.21.15.43:0/697917340 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac8dc3e440 0x55ac8dc40830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.566+0000 7fd177e00640 1 -- 172.21.15.43:0/697917340 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac8dc3e440 msgr2=0x55ac8dc40830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:08.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.566+0000 7fd16fe00640 1 -- 172.21.15.43:0/4040965328 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 27) ==== 136989+0+0 (secure 0 0 0) 0x7fd168044070 con 0x7fd1781015c0 2024-03-28T22:38:08.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.566+0000 7fd177e00640 1 --2- 172.21.15.43:0/697917340 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac8dc3e440 0x55ac8dc40830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.566+0000 7fd177e00640 1 -- 172.21.15.43:0/697917340 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac8dc350d0 msgr2=0x55ac8dc35550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:38:08.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.566+0000 7fd177e00640 1 --2- 172.21.15.43:0/697917340 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac8dc350d0 0x55ac8dc35550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.566+0000 7fd177e00640 1 -- 172.21.15.43:0/697917340 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac8db721a0 con 0x55ac8dc35db0 2024-03-28T22:38:08.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.566+0000 7fd177e00640 1 --2- 172.21.15.43:0/697917340 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac8dc35db0 0x55ac8dc34b00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd170009f50 tx=0x7fd1700118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d52222dd13b895fe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:08.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.567+0000 7fd163e00640 1 -- 172.21.15.43:0/697917340 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd170004b20 con 0x55ac8dc35db0 2024-03-28T22:38:08.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.567+0000 7fd163e00640 1 -- 172.21.15.43:0/697917340 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd170003e50 con 0x55ac8dc35db0 2024-03-28T22:38:08.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.567+0000 7fd163e00640 1 -- 172.21.15.43:0/697917340 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd17000f540 con 0x55ac8dc35db0 2024-03-28T22:38:08.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.568+0000 7fd180ddb740 1 -- 172.21.15.43:0/697917340 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac8dc35db0 msgr2=0x55ac8dc34b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:08.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.568+0000 7fd180ddb740 1 --2- 172.21.15.43:0/697917340 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac8dc35db0 0x55ac8dc34b00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd170009f50 tx=0x7fd1700118d0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.568+0000 7fd16fe00640 1 --2- 172.21.15.43:0/4040965328 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd15c0a2060 0x7fd15c0a4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.568+0000 7fd16fe00640 1 -- 172.21.15.43:0/4040965328 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fd1680f02d0 con 0x7fd1781015c0 2024-03-28T22:38:08.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.569+0000 7fd180ddb740 1 -- 172.21.15.43:0/697917340 shutdown_connections 2024-03-28T22:38:08.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.569+0000 7fd180ddb740 1 --2- 172.21.15.43:0/697917340 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac8dc350d0 0x55ac8dc35550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.569+0000 7fd180ddb740 1 --2- 172.21.15.43:0/697917340 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac8dc3e440 0x55ac8dc40830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.569+0000 7fd180ddb740 1 --2- 172.21.15.43:0/697917340 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac8dc35db0 0x55ac8dc34b00 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.569+0000 7fd180ddb740 1 -- 172.21.15.43:0/697917340 >> 172.21.15.43:0/697917340 conn(0x55ac8dbfba10 msgr2=0x55ac8dc2fb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:08.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.570+0000 7fd180ddb740 1 -- 172.21.15.43:0/697917340 shutdown_connections 2024-03-28T22:38:08.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.570+0000 7fd177e00640 1 --2- 172.21.15.43:0/4040965328 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd15c0a2060 0x7fd15c0a4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.570+0000 7fd180ddb740 1 -- 172.21.15.43:0/697917340 wait complete. 2024-03-28T22:38:08.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.570+0000 7fd180ddb740 1 -- start start 2024-03-28T22:38:08.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.571+0000 7fd180ddb740 1 Processor -- start 2024-03-28T22:38:08.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.571+0000 7fd177e00640 1 --2- 172.21.15.43:0/4040965328 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd15c0a2060 0x7fd15c0a4520 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd170009d60 tx=0x7fd170000f40 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:08.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.572+0000 7fd180ddb740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac8dc350d0 0x55ac8dc3b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.572+0000 7fd180ddb740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac8dc35db0 0x55ac8dc3ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.572+0000 7fd180ddb740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac8dc3e440 0x55ac8dc3bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.572+0000 7fd180ddb740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ac8dc412b0 con 0x55ac8dc35db0 2024-03-28T22:38:08.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.572+0000 7fd180ddb740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55ac8db37250 con 0x55ac8dc3e440 2024-03-28T22:38:08.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.572+0000 7fd180ddb740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55ac8dc41430 con 0x55ac8dc350d0 2024-03-28T22:38:08.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.572+0000 7fd180ddb740 10 client.0 _ll_register_callbacks cb 0x55ac8d8acbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:08.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.572+0000 7fd177400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac8dc350d0 0x55ac8dc3b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.572+0000 7fd177400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac8dc350d0 0x55ac8dc3b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57624/0 (socket says 172.21.15.43:57624) 2024-03-28T22:38:08.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.572+0000 7fd177400640 1 -- 172.21.15.43:0/1719265164 learned_addr learned my addr 172.21.15.43:0/1719265164 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:08.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.572+0000 7fd177e00640 1 --2- 172.21.15.43:0/1719265164 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac8dc3e440 0x55ac8dc3bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.572+0000 7fd176a00640 1 --2- 172.21.15.43:0/1719265164 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac8dc35db0 0x55ac8dc3ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.572+0000 7fd177400640 1 -- 172.21.15.43:0/1719265164 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac8dc3e440 msgr2=0x55ac8dc3bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:08.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.572+0000 7fd177400640 1 --2- 172.21.15.43:0/1719265164 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac8dc3e440 0x55ac8dc3bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.572+0000 7fd177400640 1 -- 172.21.15.43:0/1719265164 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac8dc35db0 msgr2=0x55ac8dc3ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:08.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.573+0000 7fd177400640 1 --2- 172.21.15.43:0/1719265164 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac8dc35db0 0x55ac8dc3ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.573+0000 7fd177400640 1 -- 172.21.15.43:0/1719265164 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac8db53960 con 0x55ac8dc350d0 2024-03-28T22:38:08.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.573+0000 7fd176a00640 1 --2- 172.21.15.43:0/1719265164 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac8dc35db0 0x55ac8dc3ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:38:08.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.573+0000 7fd177400640 1 --2- 172.21.15.43:0/1719265164 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac8dc350d0 0x55ac8dc3b4c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd1680b8b50 tx=0x7fd1680bc2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5cb73d9c91b24bf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:08.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.573+0000 7fd160c00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:38:08.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.573+0000 7fd160c00640 1 -- 172.21.15.43:0/1719265164 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16804b040 con 0x55ac8dc350d0 2024-03-28T22:38:08.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.573+0000 7fd160c00640 1 -- 172.21.15.43:0/1719265164 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fd1680bc9e0 con 0x55ac8dc350d0 2024-03-28T22:38:08.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.573+0000 7fd180ddb740 1 -- 172.21.15.43:0/1719265164 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd170003e50 con 0x55ac8dc350d0 2024-03-28T22:38:08.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.574+0000 7fd160c00640 1 -- 172.21.15.43:0/1719265164 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd168005600 con 0x55ac8dc350d0 2024-03-28T22:38:08.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.574+0000 7fd180ddb740 10 client.26044 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:08.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.574+0000 7fd180ddb740 1 -- 172.21.15.43:0/1719265164 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ac8d9777d0 con 0x55ac8dc350d0 2024-03-28T22:38:08.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.574+0000 7fd160c00640 1 -- 172.21.15.43:0/1719265164 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd170003e50 con 0x55ac8dc350d0 2024-03-28T22:38:08.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.574+0000 7fd160c00640 1 -- 172.21.15.43:0/1719265164 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd1680d50a0 con 0x55ac8dc350d0 2024-03-28T22:38:08.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.574+0000 7fd180ddb740 10 client.26044 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:08.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.574+0000 7fd180ddb740 10 client.26044 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:38:08.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.574+0000 7fd180ddb740 20 client.26044 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:08.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.575+0000 7fd180ddb740 1 --2- 172.21.15.43:0/1719265164 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55ac8dc68010 0x55ac8dc6a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.575+0000 7fd180ddb740 4 client.26044 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:38:08.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.575+0000 7fd180ddb740 1 -- 172.21.15.43:0/1719265164 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac8dba6640 con 0x55ac8dc68010 2024-03-28T22:38:08.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.576+0000 7fd176a00640 1 --2- 172.21.15.43:0/1719265164 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55ac8dc68010 0x55ac8dc6a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.577+0000 7fd176a00640 1 --2- 172.21.15.43:0/1719265164 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55ac8dc68010 0x55ac8dc6a490 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab94d7fc438f2008 server_cookie=66cdae69f7b65a56 in_seq=0 out_seq=0 2024-03-28T22:38:08.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.577+0000 7fd160c00640 10 client.26044 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:08.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.578+0000 7fd160c00640 1 -- 172.21.15.43:0/1719265164 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ac8dba6640 con 0x55ac8dc68010 2024-03-28T22:38:08.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.578+0000 7fd160c00640 10 client.26044 handle_command_reply: tid=0 2024-03-28T22:38:08.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.578+0000 7fd180ddb740 1 client.26044 shutdown 2024-03-28T22:38:08.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.578+0000 7fd180ddb740 1 -- 172.21.15.43:0/1719265164 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac8dc350d0 msgr2=0x55ac8dc3b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:08.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.578+0000 7fd180ddb740 1 --2- 172.21.15.43:0/1719265164 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac8dc350d0 0x55ac8dc3b4c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd1680b8b50 tx=0x7fd1680bc2c0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.578+0000 7fd180ddb740 1 -- 172.21.15.43:0/1719265164 shutdown_connections 2024-03-28T22:38:08.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.578+0000 7fd180ddb740 1 --2- 172.21.15.43:0/1719265164 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55ac8dc68010 0x55ac8dc6a490 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.578+0000 7fd180ddb740 1 --2- 172.21.15.43:0/1719265164 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac8dc3e440 0x55ac8dc3bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.578+0000 7fd180ddb740 1 --2- 172.21.15.43:0/1719265164 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac8dc35db0 0x55ac8dc3ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.578+0000 7fd160c00640 0 client.26044 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:08.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.578+0000 7fd180ddb740 1 --2- 172.21.15.43:0/1719265164 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac8dc350d0 0x55ac8dc3b4c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.578+0000 7fd180ddb740 1 -- 172.21.15.43:0/1719265164 >> 172.21.15.43:0/1719265164 conn(0x55ac8dbfba10 msgr2=0x55ac8dc31870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:08.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.578+0000 7fd180ddb740 1 -- 172.21.15.43:0/1719265164 shutdown_connections 2024-03-28T22:38:08.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.578+0000 7fd180ddb740 1 -- 172.21.15.43:0/1719265164 wait complete. 2024-03-28T22:38:08.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.578+0000 7fd180ddb740 20 client.26044 trim_cache size 0 max 16384 2024-03-28T22:38:08.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.615+0000 7fd180ddb740 1 Processor -- start 2024-03-28T22:38:08.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.615+0000 7fd180ddb740 1 -- start start 2024-03-28T22:38:08.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.615+0000 7fd180ddb740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac8dc350d0 0x55ac8dc3dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.615+0000 7fd180ddb740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac8dc35db0 0x55ac8dc3a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.615+0000 7fd180ddb740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac8dc3e440 0x55ac8dc52920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.615+0000 7fd180ddb740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd178070b40 con 0x55ac8dc3e440 2024-03-28T22:38:08.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.615+0000 7fd180ddb740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd178072680 con 0x55ac8dc350d0 2024-03-28T22:38:08.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.615+0000 7fd180ddb740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd178071210 con 0x55ac8dc35db0 2024-03-28T22:38:08.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.616+0000 7fd177400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac8dc350d0 0x55ac8dc3dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.616+0000 7fd176a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac8dc35db0 0x55ac8dc3a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.616+0000 7fd177400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac8dc350d0 0x55ac8dc3dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34382/0 (socket says 172.21.15.43:34382) 2024-03-28T22:38:08.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.616+0000 7fd177400640 1 -- 172.21.15.43:0/3589961298 learned_addr learned my addr 172.21.15.43:0/3589961298 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:08.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.616+0000 7fd177e00640 1 --2- 172.21.15.43:0/3589961298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac8dc3e440 0x55ac8dc52920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.616+0000 7fd176a00640 1 -- 172.21.15.43:0/3589961298 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac8dc350d0 msgr2=0x55ac8dc3dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:08.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.616+0000 7fd176a00640 1 --2- 172.21.15.43:0/3589961298 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac8dc350d0 0x55ac8dc3dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.616+0000 7fd176a00640 1 -- 172.21.15.43:0/3589961298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac8dc3e440 msgr2=0x55ac8dc52920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:08.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.616+0000 7fd176a00640 1 --2- 172.21.15.43:0/3589961298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac8dc3e440 0x55ac8dc52920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.616+0000 7fd176a00640 1 -- 172.21.15.43:0/3589961298 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac8d8ad3f0 con 0x55ac8dc35db0 2024-03-28T22:38:08.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.616+0000 7fd177400640 1 --2- 172.21.15.43:0/3589961298 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac8dc350d0 0x55ac8dc3dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:08.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.616+0000 7fd176a00640 1 --2- 172.21.15.43:0/3589961298 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac8dc35db0 0x55ac8dc3a420 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x55ac8dc6ad40 tx=0x7fd164004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65c156cd594faee0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:08.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.617+0000 7fd163400640 1 -- 172.21.15.43:0/3589961298 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16401a530 con 0x55ac8dc35db0 2024-03-28T22:38:08.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.617+0000 7fd163400640 1 -- 172.21.15.43:0/3589961298 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd16401e030 con 0x55ac8dc35db0 2024-03-28T22:38:08.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.617+0000 7fd163400640 1 -- 172.21.15.43:0/3589961298 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1640118a0 con 0x55ac8dc35db0 2024-03-28T22:38:08.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.617+0000 7fd180ddb740 1 -- 172.21.15.43:0/3589961298 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac8dc35db0 msgr2=0x55ac8dc3a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:08.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.617+0000 7fd180ddb740 1 --2- 172.21.15.43:0/3589961298 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac8dc35db0 0x55ac8dc3a420 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x55ac8dc6ad40 tx=0x7fd164004040 comp rx=0 tx=0).stop 2024-03-28T22:38:08.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.618+0000 7fd180ddb740 1 -- 172.21.15.43:0/3589961298 shutdown_connections 2024-03-28T22:38:08.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.618+0000 7fd180ddb740 1 --2- 172.21.15.43:0/3589961298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac8dc3e440 0x55ac8dc52920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.618+0000 7fd180ddb740 1 --2- 172.21.15.43:0/3589961298 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac8dc35db0 0x55ac8dc3a420 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.618+0000 7fd180ddb740 1 --2- 172.21.15.43:0/3589961298 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac8dc350d0 0x55ac8dc3dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.618+0000 7fd180ddb740 1 -- 172.21.15.43:0/3589961298 >> 172.21.15.43:0/3589961298 conn(0x55ac8dbfba10 msgr2=0x55ac8dc2ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:08.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.618+0000 7fd180ddb740 1 -- 172.21.15.43:0/3589961298 shutdown_connections 2024-03-28T22:38:08.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.618+0000 7fd180ddb740 1 -- 172.21.15.43:0/3589961298 wait complete. 2024-03-28T22:38:08.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.618+0000 7fd180ddb740 1 -- start start 2024-03-28T22:38:08.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.619+0000 7fd180ddb740 1 Processor -- start 2024-03-28T22:38:08.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.620+0000 7fd180ddb740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac8dc350d0 0x55ac8dc4ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.620+0000 7fd180ddb740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac8dc35db0 0x55ac8dc3f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.620+0000 7fd180ddb740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac8dc3e440 0x55ac8dc3fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.620+0000 7fd180ddb740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd178071fb0 con 0x55ac8dc35db0 2024-03-28T22:38:08.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.620+0000 7fd180ddb740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd1780718e0 con 0x55ac8dc350d0 2024-03-28T22:38:08.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.620+0000 7fd180ddb740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd17806eb60 con 0x55ac8dc3e440 2024-03-28T22:38:08.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.620+0000 7fd180ddb740 10 client.0 _ll_register_callbacks cb 0x7fd168009260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:08.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.620+0000 7fd177e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac8dc3e440 0x55ac8dc3fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.620+0000 7fd177e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac8dc3e440 0x55ac8dc3fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57636/0 (socket says 172.21.15.43:57636) 2024-03-28T22:38:08.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.620+0000 7fd177e00640 1 -- 172.21.15.43:0/4195651414 learned_addr learned my addr 172.21.15.43:0/4195651414 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:08.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.620+0000 7fd176a00640 1 --2- 172.21.15.43:0/4195651414 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac8dc35db0 0x55ac8dc3f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.620+0000 7fd177400640 1 --2- 172.21.15.43:0/4195651414 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac8dc350d0 0x55ac8dc4ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.620+0000 7fd177e00640 1 -- 172.21.15.43:0/4195651414 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac8dc350d0 msgr2=0x55ac8dc4ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:08.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.620+0000 7fd177e00640 1 --2- 172.21.15.43:0/4195651414 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac8dc350d0 0x55ac8dc4ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.620+0000 7fd177e00640 1 -- 172.21.15.43:0/4195651414 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac8dc35db0 msgr2=0x55ac8dc3f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:08.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.620+0000 7fd177e00640 1 --2- 172.21.15.43:0/4195651414 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac8dc35db0 0x55ac8dc3f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.620+0000 7fd177e00640 1 -- 172.21.15.43:0/4195651414 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac8d86d0f0 con 0x55ac8dc3e440 2024-03-28T22:38:08.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.620+0000 7fd176a00640 1 --2- 172.21.15.43:0/4195651414 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac8dc35db0 0x55ac8dc3f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:38:08.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.620+0000 7fd177400640 1 --2- 172.21.15.43:0/4195651414 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac8dc350d0 0x55ac8dc4ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:08.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.621+0000 7fd177e00640 1 --2- 172.21.15.43:0/4195651414 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac8dc3e440 0x55ac8dc3fbe0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd170011dd0 tx=0x7fd170004460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c569615dfd4f2ab server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:08.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.621+0000 7fd160c00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:38:08.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.621+0000 7fd160c00640 1 -- 172.21.15.43:0/4195651414 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd170033030 con 0x55ac8dc3e440 2024-03-28T22:38:08.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.621+0000 7fd160c00640 1 -- 172.21.15.43:0/4195651414 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fd170016040 con 0x55ac8dc3e440 2024-03-28T22:38:08.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.621+0000 7fd180ddb740 1 -- 172.21.15.43:0/4195651414 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd16401e030 con 0x55ac8dc3e440 2024-03-28T22:38:08.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.622+0000 7fd160c00640 1 -- 172.21.15.43:0/4195651414 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd17001d040 con 0x55ac8dc3e440 2024-03-28T22:38:08.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.622+0000 7fd180ddb740 10 client.26050 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:08.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.622+0000 7fd180ddb740 1 -- 172.21.15.43:0/4195651414 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ac8dcceb20 con 0x55ac8dc3e440 2024-03-28T22:38:08.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.622+0000 7fd160c00640 1 -- 172.21.15.43:0/4195651414 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd16401e030 con 0x55ac8dc3e440 2024-03-28T22:38:08.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.622+0000 7fd160c00640 1 -- 172.21.15.43:0/4195651414 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd17002c050 con 0x55ac8dc3e440 2024-03-28T22:38:08.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.623+0000 7fd180ddb740 10 client.26050 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:08.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.623+0000 7fd180ddb740 10 client.26050 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:38:08.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.623+0000 7fd180ddb740 20 client.26050 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:08.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.623+0000 7fd180ddb740 1 --2- 172.21.15.43:0/4195651414 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55ac8dccf130 0x55ac8dcd1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:08.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.623+0000 7fd180ddb740 4 client.26050 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:38:08.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.623+0000 7fd180ddb740 1 -- 172.21.15.43:0/4195651414 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ac8dba0990 con 0x55ac8dccf130 2024-03-28T22:38:08.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.623+0000 7fd177400640 1 --2- 172.21.15.43:0/4195651414 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55ac8dccf130 0x55ac8dcd1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:08.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.623+0000 7fd177400640 1 --2- 172.21.15.43:0/4195651414 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55ac8dccf130 0x55ac8dcd1550 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=601527d919833a87 server_cookie=2c707ac26758038 in_seq=0 out_seq=0 2024-03-28T22:38:08.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.623+0000 7fd160c00640 10 client.26050 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:08.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.623+0000 7fd160c00640 1 -- 172.21.15.43:0/4195651414 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55ac8dba0990 con 0x55ac8dccf130 2024-03-28T22:38:08.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.623+0000 7fd160c00640 10 client.26050 handle_command_reply: tid=0 2024-03-28T22:38:08.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.623+0000 7fd180ddb740 1 client.26050 shutdown 2024-03-28T22:38:08.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.624+0000 7fd180ddb740 1 -- 172.21.15.43:0/4195651414 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac8dc3e440 msgr2=0x55ac8dc3fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:08.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.624+0000 7fd180ddb740 1 --2- 172.21.15.43:0/4195651414 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac8dc3e440 0x55ac8dc3fbe0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd170011dd0 tx=0x7fd170004460 comp rx=0 tx=0).stop 2024-03-28T22:38:08.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.624+0000 7fd180ddb740 1 -- 172.21.15.43:0/4195651414 shutdown_connections 2024-03-28T22:38:08.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.624+0000 7fd180ddb740 1 --2- 172.21.15.43:0/4195651414 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac8dc3e440 0x55ac8dc3fbe0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.624+0000 7fd180ddb740 1 --2- 172.21.15.43:0/4195651414 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac8dc35db0 0x55ac8dc3f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.624+0000 7fd180ddb740 1 --2- 172.21.15.43:0/4195651414 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55ac8dccf130 0x55ac8dcd1550 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.624+0000 7fd180ddb740 1 --2- 172.21.15.43:0/4195651414 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac8dc350d0 0x55ac8dc4ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.624+0000 7fd180ddb740 1 -- 172.21.15.43:0/4195651414 >> 172.21.15.43:0/4195651414 conn(0x55ac8dbfba10 msgr2=0x55ac8dc3b480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:08.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.624+0000 7fd160c00640 0 client.26050 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:08.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.624+0000 7fd180ddb740 1 -- 172.21.15.43:0/4195651414 shutdown_connections 2024-03-28T22:38:08.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.624+0000 7fd180ddb740 1 -- 172.21.15.43:0/4195651414 wait complete. 2024-03-28T22:38:08.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.624+0000 7fd180ddb740 20 client.26050 trim_cache size 0 max 16384 2024-03-28T22:38:08.625 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:38:08.626 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:38:08.626 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-03-28T22:38:08.626 INFO:teuthology.orchestra.run.smithi043.stdout: "48014d05-475d-4d9d-8b39-95b8d44ed9d7": { 2024-03-28T22:38:08.626 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-03-28T22:38:08.626 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "48014d05-475d-4d9d-8b39-95b8d44ed9d7", 2024-03-28T22:38:08.626 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-03-28T22:38:08.626 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:38:08.626 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:38:08.626 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:38:08.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.625+0000 7fd17ca00640 1 -- 172.21.15.43:0/4040965328 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd15c0a2060 msgr2=0x7fd15c0a4520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:08.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.625+0000 7fd17ca00640 1 --2- 172.21.15.43:0/4040965328 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd15c0a2060 0x7fd15c0a4520 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd170009d60 tx=0x7fd170000f40 comp rx=0 tx=0).stop 2024-03-28T22:38:08.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.625+0000 7fd17ca00640 1 -- 172.21.15.43:0/4040965328 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1781015c0 msgr2=0x7fd17819a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:08.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.625+0000 7fd17ca00640 1 --2- 172.21.15.43:0/4040965328 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1781015c0 0x7fd17819a020 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd16802c240 tx=0x7fd1680043b0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.625+0000 7fd17ca00640 1 -- 172.21.15.43:0/4040965328 shutdown_connections 2024-03-28T22:38:08.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.625+0000 7fd17ca00640 1 --2- 172.21.15.43:0/4040965328 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd15c0a2060 0x7fd15c0a4520 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.625+0000 7fd17ca00640 1 --2- 172.21.15.43:0/4040965328 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd178107660 0x7fd17819e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.625+0000 7fd17ca00640 1 --2- 172.21.15.43:0/4040965328 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd178101f80 0x7fd17819a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.625+0000 7fd17ca00640 1 --2- 172.21.15.43:0/4040965328 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd1781015c0 0x7fd17819a020 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:08.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.625+0000 7fd17ca00640 1 -- 172.21.15.43:0/4040965328 >> 172.21.15.43:0/4040965328 conn(0x7fd1780fd0f0 msgr2=0x7fd1780fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:08.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.625+0000 7fd17ca00640 1 -- 172.21.15.43:0/4040965328 shutdown_connections 2024-03-28T22:38:08.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:08.625+0000 7fd17ca00640 1 -- 172.21.15.43:0/4040965328 wait complete. 2024-03-28T22:38:08.641 INFO:tasks.cephfs.filesystem:scrub status for tag:48014d05-475d-4d9d-8b39-95b8d44ed9d7 - {'path': '/', 'tag': '48014d05-475d-4d9d-8b39-95b8d44ed9d7', 'options': 'recursive,force'} 2024-03-28T22:38:09.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:09 smithi172 ceph-mon[36566]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 30 op/s 2024-03-28T22:38:09.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:09 smithi082 ceph-mon[36471]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 30 op/s 2024-03-28T22:38:09.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:09 smithi043 ceph-mon[31823]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 30 op/s 2024-03-28T22:38:10.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:10 smithi172 ceph-mon[36566]: scrub queued for path: / 2024-03-28T22:38:10.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:10 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:38:10.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:10 smithi172 ceph-mon[36566]: scrub summary: active paths [/] 2024-03-28T22:38:10.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:10 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:38:10.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:10 smithi082 ceph-mon[36471]: scrub queued for path: / 2024-03-28T22:38:10.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:10 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:38:10.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:10 smithi082 ceph-mon[36471]: scrub summary: active paths [/] 2024-03-28T22:38:10.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:10 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:38:10.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:10 smithi043 ceph-mon[31823]: scrub queued for path: / 2024-03-28T22:38:10.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:10 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:38:10.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:10 smithi043 ceph-mon[31823]: scrub summary: active paths [/] 2024-03-28T22:38:10.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:10 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:38:11.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:11 smithi172 ceph-mon[36566]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-03-28T22:38:11.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:11 smithi082 ceph-mon[36471]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-03-28T22:38:11.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:11 smithi043 ceph-mon[31823]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-03-28T22:38:13.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:13 smithi172 ceph-mon[36566]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-03-28T22:38:13.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:13 smithi082 ceph-mon[36471]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-03-28T22:38:13.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:13 smithi043 ceph-mon[31823]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-03-28T22:38:14.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:14 smithi172 ceph-mon[36566]: scrub summary: idle 2024-03-28T22:38:14.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:14 smithi082 ceph-mon[36471]: scrub summary: idle 2024-03-28T22:38:14.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:14 smithi043 ceph-mon[31823]: scrub summary: idle 2024-03-28T22:38:15.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:15 smithi172 ceph-mon[36566]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 29 op/s 2024-03-28T22:38:15.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:15 smithi082 ceph-mon[36471]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 29 op/s 2024-03-28T22:38:15.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:15 smithi043 ceph-mon[31823]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 29 op/s 2024-03-28T22:38:17.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:17 smithi172 ceph-mon[36566]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-28T22:38:17.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:17 smithi082 ceph-mon[36471]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-28T22:38:17.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:17 smithi043 ceph-mon[31823]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-28T22:38:19.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:19 smithi172 ceph-mon[36566]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 682 B/s wr, 14 op/s 2024-03-28T22:38:19.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:19 smithi082 ceph-mon[36471]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 682 B/s wr, 14 op/s 2024-03-28T22:38:19.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:19 smithi043 ceph-mon[31823]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 682 B/s wr, 14 op/s 2024-03-28T22:38:20.244 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.245+0000 7f7e0dc00640 1 -- 172.21.15.43:0/2134323234 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e08101570 msgr2=0x7f7e081019f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:20.244 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.245+0000 7f7e0dc00640 1 --2- 172.21.15.43:0/2134323234 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e08101570 0x7f7e081019f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7df4009860 tx=0x7f7df402bb70 comp rx=0 tx=0).stop 2024-03-28T22:38:20.244 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.245+0000 7f7e0dc00640 1 -- 172.21.15.43:0/2134323234 shutdown_connections 2024-03-28T22:38:20.245 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.245+0000 7f7e0dc00640 1 --2- 172.21.15.43:0/2134323234 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7e08101f30 0x7f7e0810e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:20.245 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.245+0000 7f7e0dc00640 1 --2- 172.21.15.43:0/2134323234 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e08101570 0x7f7e081019f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:20.245 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.245+0000 7f7e0dc00640 1 --2- 172.21.15.43:0/2134323234 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e08107630 0x7f7e08107a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:20.245 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.245+0000 7f7e0dc00640 1 -- 172.21.15.43:0/2134323234 >> 172.21.15.43:0/2134323234 conn(0x7f7e080fd060 msgr2=0x7f7e080ff480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:20.245 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.245+0000 7f7e0dc00640 1 -- 172.21.15.43:0/2134323234 shutdown_connections 2024-03-28T22:38:20.245 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.246+0000 7f7e0dc00640 1 -- 172.21.15.43:0/2134323234 wait complete. 2024-03-28T22:38:20.245 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.247+0000 7f7e0dc00640 1 Processor -- start 2024-03-28T22:38:20.246 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.247+0000 7f7e0dc00640 1 -- start start 2024-03-28T22:38:20.246 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.248+0000 7f7e0dc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e08101f30 0x7f7e08197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:20.246 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.248+0000 7f7e0dc00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e08107630 0x7f7e081984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:20.246 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.248+0000 7f7e0dc00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7e0819c860 0x7f7e0819cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:20.246 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.248+0000 7f7e0dc00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7e081118f0 con 0x7f7e08101f30 2024-03-28T22:38:20.246 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.248+0000 7f7e0dc00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7e08111770 con 0x7f7e0819c860 2024-03-28T22:38:20.247 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.248+0000 7f7e0dc00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f7e08111a70 con 0x7f7e08107630 2024-03-28T22:38:20.247 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.248+0000 7f7e07400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e08107630 0x7f7e081984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:20.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.248+0000 7f7e07400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e08107630 0x7f7e081984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:44736/0 (socket says 172.21.15.43:44736) 2024-03-28T22:38:20.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.248+0000 7f7e07400640 1 -- 172.21.15.43:0/1252534351 learned_addr learned my addr 172.21.15.43:0/1252534351 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:20.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.248+0000 7f7e0d200640 1 --2- 172.21.15.43:0/1252534351 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7e0819c860 0x7f7e0819cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:20.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.248+0000 7f7e07e00640 1 --2- 172.21.15.43:0/1252534351 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e08101f30 0x7f7e08197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:20.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.248+0000 7f7e07400640 1 -- 172.21.15.43:0/1252534351 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7e0819c860 msgr2=0x7f7e0819cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:20.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.248+0000 7f7e07400640 1 --2- 172.21.15.43:0/1252534351 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7e0819c860 0x7f7e0819cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:20.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.248+0000 7f7e07400640 1 -- 172.21.15.43:0/1252534351 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e08101f30 msgr2=0x7f7e08197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:20.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.248+0000 7f7e07400640 1 --2- 172.21.15.43:0/1252534351 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e08101f30 0x7f7e08197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:20.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.248+0000 7f7e07400640 1 -- 172.21.15.43:0/1252534351 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e0819d350 con 0x7f7e08107630 2024-03-28T22:38:20.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.248+0000 7f7e0d200640 1 --2- 172.21.15.43:0/1252534351 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7e0819c860 0x7f7e0819cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:38:20.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.248+0000 7f7e07e00640 1 --2- 172.21.15.43:0/1252534351 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e08101f30 0x7f7e08197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:38:20.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.249+0000 7f7e07400640 1 --2- 172.21.15.43:0/1252534351 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e08107630 0x7f7e081984e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7df4037040 tx=0x7f7df40045b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:20.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.249+0000 7f7e04c00640 1 -- 172.21.15.43:0/1252534351 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df4041020 con 0x7f7e08107630 2024-03-28T22:38:20.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.249+0000 7f7e0dc00640 1 -- 172.21.15.43:0/1252534351 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e0819d5e0 con 0x7f7e08107630 2024-03-28T22:38:20.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.249+0000 7f7e04c00640 1 -- 172.21.15.43:0/1252534351 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7df402c9f0 con 0x7f7e08107630 2024-03-28T22:38:20.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.249+0000 7f7e04c00640 1 -- 172.21.15.43:0/1252534351 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df4039040 con 0x7f7e08107630 2024-03-28T22:38:20.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.249+0000 7f7e0dc00640 1 -- 172.21.15.43:0/1252534351 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e08073c10 con 0x7f7e08107630 2024-03-28T22:38:20.251 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.252+0000 7f7e04c00640 1 -- 172.21.15.43:0/1252534351 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 27) ==== 136989+0+0 (secure 0 0 0) 0x7f7df4044020 con 0x7f7e08107630 2024-03-28T22:38:20.252 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.254+0000 7f7e04c00640 1 --2- 172.21.15.43:0/1252534351 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7dec0a2100 0x7f7dec0a45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:20.252 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.254+0000 7f7e11fd3740 1 Processor -- start 2024-03-28T22:38:20.252 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.254+0000 7f7e04c00640 1 -- 172.21.15.43:0/1252534351 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f7df40eff10 con 0x7f7e08107630 2024-03-28T22:38:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.254+0000 7f7e11fd3740 1 -- start start 2024-03-28T22:38:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.254+0000 7f7e07e00640 1 --2- 172.21.15.43:0/1252534351 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7dec0a2100 0x7f7dec0a45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.255+0000 7f7e11fd3740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cabab15570 0x55cabab15950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.255+0000 7f7e11fd3740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cabab15e90 0x55cabab1e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.255+0000 7f7e11fd3740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cabab1ec60 0x55cabab21050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.255+0000 7f7e11fd3740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55cabaa78640 con 0x55cabab1ec60 2024-03-28T22:38:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.255+0000 7f7e11fd3740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55cabaa21810 con 0x55cabab15570 2024-03-28T22:38:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.255+0000 7f7e11fd3740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55cabaa746e0 con 0x55cabab15e90 2024-03-28T22:38:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.255+0000 7f7e07e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cabab15570 0x55cabab15950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.255+0000 7f7e07e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cabab15570 0x55cabab15950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59180/0 (socket says 172.21.15.43:59180) 2024-03-28T22:38:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.255+0000 7f7e07e00640 1 -- 172.21.15.43:0/4132973547 learned_addr learned my addr 172.21.15.43:0/4132973547 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.255+0000 7f7e07400640 1 --2- 172.21.15.43:0/4132973547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cabab15e90 0x55cabab1e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.255+0000 7f7e0d200640 1 --2- 172.21.15.43:0/4132973547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cabab1ec60 0x55cabab21050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.255+0000 7f7e07e00640 1 --2- 172.21.15.43:0/1252534351 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7dec0a2100 0x7f7dec0a45c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7df8005fd0 tx=0x7f7df8009260 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.255+0000 7f7e07e00640 1 -- 172.21.15.43:0/4132973547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cabab15e90 msgr2=0x55cabab1e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.255+0000 7f7e07e00640 1 --2- 172.21.15.43:0/4132973547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cabab15e90 0x55cabab1e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.255+0000 7f7e07e00640 1 -- 172.21.15.43:0/4132973547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cabab1ec60 msgr2=0x55cabab21050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.256+0000 7f7e07e00640 1 --2- 172.21.15.43:0/4132973547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cabab1ec60 0x55cabab21050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.256+0000 7f7e07e00640 1 -- 172.21.15.43:0/4132973547 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55caba78d210 con 0x55cabab15570 2024-03-28T22:38:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.256+0000 7f7e07e00640 1 --2- 172.21.15.43:0/4132973547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cabab15570 0x55cabab15950 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7df8012690 tx=0x7f7df801c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c59812edefcc018 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.256+0000 7f7dfd600640 1 -- 172.21.15.43:0/4132973547 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df8004220 con 0x55cabab15570 2024-03-28T22:38:20.256 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.256+0000 7f7dfd600640 1 -- 172.21.15.43:0/4132973547 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7df8022070 con 0x55cabab15570 2024-03-28T22:38:20.256 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.256+0000 7f7dfd600640 1 -- 172.21.15.43:0/4132973547 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df801e730 con 0x55cabab15570 2024-03-28T22:38:20.256 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.257+0000 7f7e11fd3740 1 -- 172.21.15.43:0/4132973547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cabab15570 msgr2=0x55cabab15950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:20.256 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.257+0000 7f7e11fd3740 1 --2- 172.21.15.43:0/4132973547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cabab15570 0x55cabab15950 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7df8012690 tx=0x7f7df801c000 comp rx=0 tx=0).stop 2024-03-28T22:38:20.256 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.258+0000 7f7e11fd3740 1 -- 172.21.15.43:0/4132973547 shutdown_connections 2024-03-28T22:38:20.257 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.258+0000 7f7e11fd3740 1 --2- 172.21.15.43:0/4132973547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cabab1ec60 0x55cabab21050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:20.257 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.258+0000 7f7e11fd3740 1 --2- 172.21.15.43:0/4132973547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cabab15e90 0x55cabab1e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:20.257 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.258+0000 7f7e11fd3740 1 --2- 172.21.15.43:0/4132973547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cabab15570 0x55cabab15950 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:20.257 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.258+0000 7f7e11fd3740 1 -- 172.21.15.43:0/4132973547 >> 172.21.15.43:0/4132973547 conn(0x55cabab0fe90 msgr2=0x55cabab10270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:20.257 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.258+0000 7f7e11fd3740 1 -- 172.21.15.43:0/4132973547 shutdown_connections 2024-03-28T22:38:20.257 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.258+0000 7f7e11fd3740 1 -- 172.21.15.43:0/4132973547 wait complete. 2024-03-28T22:38:20.257 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.258+0000 7f7e11fd3740 1 -- start start 2024-03-28T22:38:20.258 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.259+0000 7f7e11fd3740 1 Processor -- start 2024-03-28T22:38:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.259+0000 7f7e11fd3740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cabab15570 0x55cabab1bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.259+0000 7f7e11fd3740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cabab15e90 0x55cabab1c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.259+0000 7f7e11fd3740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cabab1ec60 0x55cabab1c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.260+0000 7f7e11fd3740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55cabab21890 con 0x55cabab15570 2024-03-28T22:38:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.260+0000 7f7e11fd3740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55cabaa16d00 con 0x55cabab15e90 2024-03-28T22:38:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.260+0000 7f7e11fd3740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55cabab21a10 con 0x55cabab1ec60 2024-03-28T22:38:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.260+0000 7f7e11fd3740 10 client.0 _ll_register_callbacks cb 0x55cabaa01490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.260+0000 7f7e07400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cabab15e90 0x55cabab1c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.260+0000 7f7e0d200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cabab1ec60 0x55cabab1c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.260+0000 7f7e07400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cabab15e90 0x55cabab1c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59182/0 (socket says 172.21.15.43:59182) 2024-03-28T22:38:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.260+0000 7f7e07400640 1 -- 172.21.15.43:0/1105522764 learned_addr learned my addr 172.21.15.43:0/1105522764 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.260+0000 7f7e07e00640 1 --2- 172.21.15.43:0/1105522764 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cabab15570 0x55cabab1bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.260+0000 7f7e07400640 1 -- 172.21.15.43:0/1105522764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cabab1ec60 msgr2=0x55cabab1c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.260+0000 7f7e07400640 1 --2- 172.21.15.43:0/1105522764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cabab1ec60 0x55cabab1c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.260+0000 7f7e07400640 1 -- 172.21.15.43:0/1105522764 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cabab15570 msgr2=0x55cabab1bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.260+0000 7f7e07400640 1 --2- 172.21.15.43:0/1105522764 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cabab15570 0x55cabab1bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.260+0000 7f7e07400640 1 -- 172.21.15.43:0/1105522764 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55caba859620 con 0x55cabab15e90 2024-03-28T22:38:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.260+0000 7f7e0d200640 1 --2- 172.21.15.43:0/1105522764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cabab1ec60 0x55cabab1c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:38:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.260+0000 7f7e07e00640 1 --2- 172.21.15.43:0/1105522764 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cabab15570 0x55cabab1bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:38:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.261+0000 7f7e07400640 1 --2- 172.21.15.43:0/1105522764 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cabab15e90 0x55cabab1c230 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7df4005940 tx=0x7f7df4005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b41d4b958e281fa8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.261+0000 7f7df2000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:38:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.261+0000 7f7df2000640 1 -- 172.21.15.43:0/1105522764 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df40bc3c0 con 0x55cabab15e90 2024-03-28T22:38:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.261+0000 7f7e11fd3740 1 -- 172.21.15.43:0/1105522764 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7df8022070 con 0x55cabab15e90 2024-03-28T22:38:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.261+0000 7f7df2000640 1 -- 172.21.15.43:0/1105522764 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f7df40c0120 con 0x55cabab15e90 2024-03-28T22:38:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.261+0000 7f7df2000640 1 -- 172.21.15.43:0/1105522764 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7df40c0510 con 0x55cabab15e90 2024-03-28T22:38:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.261+0000 7f7e11fd3740 10 client.26114 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.261+0000 7f7e11fd3740 1 -- 172.21.15.43:0/1105522764 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cabab2d7c0 con 0x55cabab15e90 2024-03-28T22:38:20.263 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.261+0000 7f7df2000640 1 -- 172.21.15.43:0/1105522764 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7df8022070 con 0x55cabab15e90 2024-03-28T22:38:20.263 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.262+0000 7f7df2000640 1 -- 172.21.15.43:0/1105522764 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f7df403c030 con 0x55cabab15e90 2024-03-28T22:38:20.263 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.262+0000 7f7e11fd3740 10 client.26114 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.262+0000 7f7e11fd3740 10 client.26114 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:38:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.262+0000 7f7e11fd3740 20 client.26114 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.262+0000 7f7e11fd3740 1 --2- 172.21.15.43:0/1105522764 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55cabab2ec50 0x55cabab310d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.262+0000 7f7e11fd3740 4 client.26114 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:38:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.262+0000 7f7e11fd3740 1 -- 172.21.15.43:0/1105522764 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cabaa8c030 con 0x55cabab2ec50 2024-03-28T22:38:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.262+0000 7f7e07e00640 1 --2- 172.21.15.43:0/1105522764 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55cabab2ec50 0x55cabab310d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.263+0000 7f7e07e00640 1 --2- 172.21.15.43:0/1105522764 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55cabab2ec50 0x55cabab310d0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8fd2782d465864b9 server_cookie=c24629408f6745e9 in_seq=0 out_seq=0 2024-03-28T22:38:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.263+0000 7f7df2000640 10 client.26114 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.264+0000 7f7df2000640 1 -- 172.21.15.43:0/1105522764 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cabaa8c030 con 0x55cabab2ec50 2024-03-28T22:38:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.264+0000 7f7df2000640 10 client.26114 handle_command_reply: tid=0 2024-03-28T22:38:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.264+0000 7f7e11fd3740 1 client.26114 shutdown 2024-03-28T22:38:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.264+0000 7f7e11fd3740 1 -- 172.21.15.43:0/1105522764 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cabab15e90 msgr2=0x55cabab1c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.264+0000 7f7e11fd3740 1 --2- 172.21.15.43:0/1105522764 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cabab15e90 0x55cabab1c230 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7df4005940 tx=0x7f7df4005f90 comp rx=0 tx=0).stop 2024-03-28T22:38:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.264+0000 7f7e11fd3740 1 -- 172.21.15.43:0/1105522764 shutdown_connections 2024-03-28T22:38:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.264+0000 7f7e11fd3740 1 --2- 172.21.15.43:0/1105522764 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55cabab2ec50 0x55cabab310d0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.264+0000 7f7e11fd3740 1 --2- 172.21.15.43:0/1105522764 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cabab15e90 0x55cabab1c230 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.264+0000 7f7e11fd3740 1 --2- 172.21.15.43:0/1105522764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cabab1ec60 0x55cabab1c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.264+0000 7f7df2000640 0 client.26114 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.264+0000 7f7e11fd3740 1 --2- 172.21.15.43:0/1105522764 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cabab15570 0x55cabab1bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.264+0000 7f7e11fd3740 1 -- 172.21.15.43:0/1105522764 >> 172.21.15.43:0/1105522764 conn(0x55cabab0fe90 msgr2=0x55cabab11be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.264+0000 7f7e11fd3740 1 -- 172.21.15.43:0/1105522764 shutdown_connections 2024-03-28T22:38:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.264+0000 7f7e11fd3740 1 -- 172.21.15.43:0/1105522764 wait complete. 2024-03-28T22:38:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.265+0000 7f7e11fd3740 20 client.26114 trim_cache size 0 max 16384 2024-03-28T22:38:20.279 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.280+0000 7f7e11fd3740 1 Processor -- start 2024-03-28T22:38:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.280+0000 7f7e11fd3740 1 -- start start 2024-03-28T22:38:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.281+0000 7f7e11fd3740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cabab15570 0x55cabab1e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.281+0000 7f7e11fd3740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cabab15e90 0x55cabab2a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.281+0000 7f7e11fd3740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cabab1ec60 0x55cabab119f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.281+0000 7f7e11fd3740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7e081b1030 con 0x55cabab15e90 2024-03-28T22:38:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.281+0000 7f7e11fd3740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7e081b2b70 con 0x55cabab15570 2024-03-28T22:38:20.280 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.281+0000 7f7e11fd3740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f7e081b1700 con 0x55cabab1ec60 2024-03-28T22:38:20.281 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.281+0000 7f7e07e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cabab15570 0x55cabab1e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:20.281 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.281+0000 7f7e07e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cabab15570 0x55cabab1e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59196/0 (socket says 172.21.15.43:59196) 2024-03-28T22:38:20.281 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.281+0000 7f7e07e00640 1 -- 172.21.15.43:0/2032746686 learned_addr learned my addr 172.21.15.43:0/2032746686 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:20.282 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.281+0000 7f7e0d200640 1 --2- 172.21.15.43:0/2032746686 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cabab1ec60 0x55cabab119f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:20.282 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.281+0000 7f7e07400640 1 --2- 172.21.15.43:0/2032746686 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cabab15e90 0x55cabab2a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:20.282 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.281+0000 7f7e07e00640 1 -- 172.21.15.43:0/2032746686 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cabab1ec60 msgr2=0x55cabab119f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:20.282 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.282+0000 7f7e07e00640 1 --2- 172.21.15.43:0/2032746686 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cabab1ec60 0x55cabab119f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:20.282 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.282+0000 7f7e07e00640 1 -- 172.21.15.43:0/2032746686 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cabab15e90 msgr2=0x55cabab2a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:20.282 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.282+0000 7f7e07e00640 1 --2- 172.21.15.43:0/2032746686 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cabab15e90 0x55cabab2a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:20.282 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.282+0000 7f7e07e00640 1 -- 172.21.15.43:0/2032746686 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cabab2ab70 con 0x55cabab15570 2024-03-28T22:38:20.282 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.282+0000 7f7e0d200640 1 --2- 172.21.15.43:0/2032746686 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cabab1ec60 0x55cabab119f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:38:20.282 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.282+0000 7f7e07e00640 1 --2- 172.21.15.43:0/2032746686 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cabab15570 0x55cabab1e420 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f7df8011c40 tx=0x7f7df8020040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=636b5e40ae45ee5e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:20.282 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.282+0000 7f7dfcc00640 1 -- 172.21.15.43:0/2032746686 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df8004280 con 0x55cabab15570 2024-03-28T22:38:20.282 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.282+0000 7f7dfcc00640 1 -- 172.21.15.43:0/2032746686 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7df8022030 con 0x55cabab15570 2024-03-28T22:38:20.282 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.282+0000 7f7dfcc00640 1 -- 172.21.15.43:0/2032746686 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df8005400 con 0x55cabab15570 2024-03-28T22:38:20.284 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.283+0000 7f7e11fd3740 1 -- 172.21.15.43:0/2032746686 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cabab15570 msgr2=0x55cabab1e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:20.284 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.283+0000 7f7e11fd3740 1 --2- 172.21.15.43:0/2032746686 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cabab15570 0x55cabab1e420 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f7df8011c40 tx=0x7f7df8020040 comp rx=0 tx=0).stop 2024-03-28T22:38:20.284 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.283+0000 7f7e11fd3740 1 -- 172.21.15.43:0/2032746686 shutdown_connections 2024-03-28T22:38:20.284 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.283+0000 7f7e11fd3740 1 --2- 172.21.15.43:0/2032746686 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cabab1ec60 0x55cabab119f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:20.284 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.283+0000 7f7e11fd3740 1 --2- 172.21.15.43:0/2032746686 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cabab15e90 0x55cabab2a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.283+0000 7f7e11fd3740 1 --2- 172.21.15.43:0/2032746686 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cabab15570 0x55cabab1e420 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.283+0000 7f7e11fd3740 1 -- 172.21.15.43:0/2032746686 >> 172.21.15.43:0/2032746686 conn(0x55cabab0fe90 msgr2=0x55cabab112e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.283+0000 7f7e11fd3740 1 -- 172.21.15.43:0/2032746686 shutdown_connections 2024-03-28T22:38:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.284+0000 7f7e11fd3740 1 -- 172.21.15.43:0/2032746686 wait complete. 2024-03-28T22:38:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.284+0000 7f7e11fd3740 1 -- start start 2024-03-28T22:38:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.284+0000 7f7e11fd3740 1 Processor -- start 2024-03-28T22:38:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.285+0000 7f7e11fd3740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cabab15570 0x55cabab23160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.285+0000 7f7e11fd3740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cabab15e90 0x55cabab236a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.285+0000 7f7e11fd3740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cabab1ec60 0x55cabab8cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.285+0000 7f7e11fd3740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7e081b24a0 con 0x55cabab15570 2024-03-28T22:38:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.285+0000 7f7e11fd3740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7e081b1dd0 con 0x55cabab15e90 2024-03-28T22:38:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.285+0000 7f7e11fd3740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f7e081af010 con 0x55cabab1ec60 2024-03-28T22:38:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.285+0000 7f7e11fd3740 10 client.0 _ll_register_callbacks cb 0x7f7df40dc0b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.285+0000 7f7e0d200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cabab1ec60 0x55cabab8cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.285+0000 7f7e07e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cabab15570 0x55cabab23160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.285+0000 7f7e0d200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cabab1ec60 0x55cabab8cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:44774/0 (socket says 172.21.15.43:44774) 2024-03-28T22:38:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.285+0000 7f7e0d200640 1 -- 172.21.15.43:0/2897170348 learned_addr learned my addr 172.21.15.43:0/2897170348 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.285+0000 7f7e07400640 1 --2- 172.21.15.43:0/2897170348 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cabab15e90 0x55cabab236a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.285+0000 7f7e0d200640 1 -- 172.21.15.43:0/2897170348 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cabab15e90 msgr2=0x55cabab236a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.285+0000 7f7e0d200640 1 --2- 172.21.15.43:0/2897170348 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cabab15e90 0x55cabab236a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.285+0000 7f7e0d200640 1 -- 172.21.15.43:0/2897170348 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cabab15570 msgr2=0x55cabab23160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.285+0000 7f7e0d200640 1 --2- 172.21.15.43:0/2897170348 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cabab15570 0x55cabab23160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.285+0000 7f7e0d200640 1 -- 172.21.15.43:0/2897170348 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cabab8d7d0 con 0x55cabab1ec60 2024-03-28T22:38:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.285+0000 7f7e07400640 1 --2- 172.21.15.43:0/2897170348 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cabab15e90 0x55cabab236a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:20.289 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.286+0000 7f7e0d200640 1 --2- 172.21.15.43:0/2897170348 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cabab1ec60 0x55cabab8cfd0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7e00012cf0 tx=0x7f7e00010800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c215721413968d5a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:20.289 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.286+0000 7f7df2000640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:38:20.289 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.286+0000 7f7df2000640 1 -- 172.21.15.43:0/2897170348 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e00009c30 con 0x55cabab1ec60 2024-03-28T22:38:20.289 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.286+0000 7f7e11fd3740 1 -- 172.21.15.43:0/2897170348 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7df8022030 con 0x55cabab1ec60 2024-03-28T22:38:20.289 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.286+0000 7f7df2000640 1 -- 172.21.15.43:0/2897170348 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f7e00009dd0 con 0x55cabab1ec60 2024-03-28T22:38:20.289 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.287+0000 7f7df2000640 1 -- 172.21.15.43:0/2897170348 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7e00004b20 con 0x55cabab1ec60 2024-03-28T22:38:20.289 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.287+0000 7f7e11fd3740 10 client.26068 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:20.290 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.287+0000 7f7e11fd3740 1 -- 172.21.15.43:0/2897170348 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cabab95bf0 con 0x55cabab1ec60 2024-03-28T22:38:20.290 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.287+0000 7f7df2000640 1 -- 172.21.15.43:0/2897170348 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7df8022030 con 0x55cabab1ec60 2024-03-28T22:38:20.290 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.288+0000 7f7df2000640 1 -- 172.21.15.43:0/2897170348 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f7e0001c5b0 con 0x55cabab1ec60 2024-03-28T22:38:20.290 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.288+0000 7f7e11fd3740 10 client.26068 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:20.290 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.288+0000 7f7e11fd3740 10 client.26068 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:38:20.290 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.288+0000 7f7e11fd3740 20 client.26068 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:20.290 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.288+0000 7f7e11fd3740 1 --2- 172.21.15.43:0/2897170348 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55cabab96310 0x55cabab98700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:20.290 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.288+0000 7f7e11fd3740 4 client.26068 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T22:38:20.290 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.288+0000 7f7e11fd3740 1 -- 172.21.15.43:0/2897170348 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cabaa85b80 con 0x55cabab96310 2024-03-28T22:38:20.290 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.288+0000 7f7e07e00640 1 --2- 172.21.15.43:0/2897170348 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55cabab96310 0x55cabab98700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:20.290 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.289+0000 7f7df2000640 10 client.26068 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:20.290 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.289+0000 7f7e07e00640 1 --2- 172.21.15.43:0/2897170348 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55cabab96310 0x55cabab98700 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8244eaa4ec005237 server_cookie=d2efdf133880deff in_seq=0 out_seq=0 2024-03-28T22:38:20.291 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.292+0000 7f7df2000640 1 -- 172.21.15.43:0/2897170348 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cabaa85b80 con 0x55cabab96310 2024-03-28T22:38:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.292+0000 7f7df2000640 10 client.26068 handle_command_reply: tid=0 2024-03-28T22:38:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.292+0000 7f7e11fd3740 1 client.26068 shutdown 2024-03-28T22:38:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.292+0000 7f7e11fd3740 1 -- 172.21.15.43:0/2897170348 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cabab1ec60 msgr2=0x55cabab8cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.292+0000 7f7e11fd3740 1 --2- 172.21.15.43:0/2897170348 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cabab1ec60 0x55cabab8cfd0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7e00012cf0 tx=0x7f7e00010800 comp rx=0 tx=0).stop 2024-03-28T22:38:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.292+0000 7f7e11fd3740 1 -- 172.21.15.43:0/2897170348 shutdown_connections 2024-03-28T22:38:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.292+0000 7f7e11fd3740 1 --2- 172.21.15.43:0/2897170348 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55cabab96310 0x55cabab98700 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.292+0000 7f7e11fd3740 1 --2- 172.21.15.43:0/2897170348 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cabab15e90 0x55cabab236a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.292+0000 7f7e11fd3740 1 --2- 172.21.15.43:0/2897170348 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55cabab1ec60 0x55cabab8cfd0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.292+0000 7f7e11fd3740 1 --2- 172.21.15.43:0/2897170348 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55cabab15570 0x55cabab23160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.292+0000 7f7e11fd3740 1 -- 172.21.15.43:0/2897170348 >> 172.21.15.43:0/2897170348 conn(0x55cabab0fe90 msgr2=0x55cabab11b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.292+0000 7f7df2000640 0 client.26068 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.293+0000 7f7e11fd3740 1 -- 172.21.15.43:0/2897170348 shutdown_connections 2024-03-28T22:38:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.293+0000 7f7e11fd3740 1 -- 172.21.15.43:0/2897170348 wait complete. 2024-03-28T22:38:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.293+0000 7f7e11fd3740 20 client.26068 trim_cache size 0 max 16384 2024-03-28T22:38:20.293 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:{ 2024-03-28T22:38:20.293 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "message": "", 2024-03-28T22:38:20.293 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "return_code": 0 2024-03-28T22:38:20.293 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:} 2024-03-28T22:38:20.293 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.294+0000 7f7e0dc00640 1 -- 172.21.15.43:0/1252534351 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7dec0a2100 msgr2=0x7f7dec0a45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:20.293 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.294+0000 7f7e0dc00640 1 --2- 172.21.15.43:0/1252534351 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7dec0a2100 0x7f7dec0a45c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7df8005fd0 tx=0x7f7df8009260 comp rx=0 tx=0).stop 2024-03-28T22:38:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.294+0000 7f7e0dc00640 1 -- 172.21.15.43:0/1252534351 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e08107630 msgr2=0x7f7e081984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.294+0000 7f7e0dc00640 1 --2- 172.21.15.43:0/1252534351 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e08107630 0x7f7e081984e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7df4037040 tx=0x7f7df40045b0 comp rx=0 tx=0).stop 2024-03-28T22:38:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.294+0000 7f7e0dc00640 1 -- 172.21.15.43:0/1252534351 shutdown_connections 2024-03-28T22:38:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.294+0000 7f7e0dc00640 1 --2- 172.21.15.43:0/1252534351 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7dec0a2100 0x7f7dec0a45c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.294+0000 7f7e0dc00640 1 --2- 172.21.15.43:0/1252534351 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7e0819c860 0x7f7e0819cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.294+0000 7f7e0dc00640 1 --2- 172.21.15.43:0/1252534351 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e08107630 0x7f7e081984e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.294+0000 7f7e0dc00640 1 --2- 172.21.15.43:0/1252534351 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e08101f30 0x7f7e08197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.294+0000 7f7e0dc00640 1 -- 172.21.15.43:0/1252534351 >> 172.21.15.43:0/1252534351 conn(0x7f7e080fd060 msgr2=0x7f7e080fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.294+0000 7f7e0dc00640 1 -- 172.21.15.43:0/1252534351 shutdown_connections 2024-03-28T22:38:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:20.294+0000 7f7e0dc00640 1 -- 172.21.15.43:0/1252534351 wait complete. 2024-03-28T22:38:21.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:21 smithi172 ceph-mon[36566]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-28T22:38:21.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:21 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:38:21.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:21 smithi082 ceph-mon[36471]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-28T22:38:21.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:21 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:38:21.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:21 smithi043 ceph-mon[31823]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-28T22:38:21.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:21 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:38:22.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:22 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:38:22.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:22 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:38:22.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:22 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:38:22.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:22 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:38:22.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:22 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:38:22.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:22 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:38:22.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:22 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:38:22.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:22 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:38:22.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:22 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:38:23.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:23 smithi172 ceph-mon[36566]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-03-28T22:38:23.416 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:23 smithi043 ceph-mon[31823]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-03-28T22:38:23.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:23 smithi082 ceph-mon[36471]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-03-28T22:38:25.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:25 smithi172 ceph-mon[36566]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 255 B/s wr, 14 op/s 2024-03-28T22:38:25.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:25 smithi082 ceph-mon[36471]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 255 B/s wr, 14 op/s 2024-03-28T22:38:25.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:25 smithi043 ceph-mon[31823]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 255 B/s wr, 14 op/s 2024-03-28T22:38:27.227 INFO:tasks.workunit.client.0.smithi043.stderr:Updating files: 61% (7690/12435) Updating files: 62% (7710/12435) Updating files: 63% (7835/12435) Updating files: 64% (7959/12435) Updating files: 65% (8083/12435) Updating files: 66% (8208/12435) Updating files: 67% (8332/12435) Updating files: 68% (8456/12435) Updating files: 69% (8581/12435) Updating files: 70% (8705/12435) Updating files: 71% (8829/12435) Updating files: 72% (8954/12435) Updating files: 73% (9078/12435) Updating files: 74% (9202/12435) Updating files: 75% (9327/12435) Updating files: 76% (9451/12435) Updating files: 77% (9575/12435) Updating files: 78% (9700/12435) Updating files: 79% (9824/12435) Updating files: 80% (9948/12435) Updating files: 81% (10073/12435) Updating files: 82% (10197/12435) Updating files: 83% (10322/12435) Updating files: 84% (10446/12435) Updating files: 85% (10570/12435) Updating files: 86% (10695/12435) Updating files: 87% (10819/12435) Updating files: 88% (10943/12435) Updating files: 89% (11068/12435) Updating files: 90% (11192/12435) Updating files: 91% (11316/12435) Updating files: 92% (11441/12435) Updating files: 93% (11565/12435) Updating files: 94% (11689/12435) Updating files: 95% (11814/12435) Updating files: 96% (11938/12435) Updating files: 97% (12062/12435) Updating files: 98% (12187/12435) Updating files: 99% (12311/12435) Updating files: 99% (12385/12435) Updating files: 100% (12435/12435) Updating files: 100% (12435/12435), done. 2024-03-28T22:38:27.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:27 smithi172 ceph-mon[36566]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T22:38:27.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:27 smithi082 ceph-mon[36471]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T22:38:27.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:27 smithi043 ceph-mon[31823]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T22:38:27.661 INFO:tasks.workunit.client.0.smithi043.stderr:Note: switching to '55f5138942aa4866e49329c9103468889cec3f73'. 2024-03-28T22:38:27.661 INFO:tasks.workunit.client.0.smithi043.stderr: 2024-03-28T22:38:27.661 INFO:tasks.workunit.client.0.smithi043.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-28T22:38:27.661 INFO:tasks.workunit.client.0.smithi043.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-28T22:38:27.661 INFO:tasks.workunit.client.0.smithi043.stderr:state without impacting any branches by switching back to a branch. 2024-03-28T22:38:27.661 INFO:tasks.workunit.client.0.smithi043.stderr: 2024-03-28T22:38:27.661 INFO:tasks.workunit.client.0.smithi043.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-28T22:38:27.661 INFO:tasks.workunit.client.0.smithi043.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-28T22:38:27.661 INFO:tasks.workunit.client.0.smithi043.stderr: 2024-03-28T22:38:27.661 INFO:tasks.workunit.client.0.smithi043.stderr: git switch -c 2024-03-28T22:38:27.661 INFO:tasks.workunit.client.0.smithi043.stderr: 2024-03-28T22:38:27.662 INFO:tasks.workunit.client.0.smithi043.stderr:Or undo this operation with: 2024-03-28T22:38:27.662 INFO:tasks.workunit.client.0.smithi043.stderr: 2024-03-28T22:38:27.662 INFO:tasks.workunit.client.0.smithi043.stderr: git switch - 2024-03-28T22:38:27.662 INFO:tasks.workunit.client.0.smithi043.stderr: 2024-03-28T22:38:27.662 INFO:tasks.workunit.client.0.smithi043.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-28T22:38:27.662 INFO:tasks.workunit.client.0.smithi043.stderr: 2024-03-28T22:38:27.662 INFO:tasks.workunit.client.0.smithi043.stderr:HEAD is now at 55f5138942a Merge pull request #56314 from adk3798/wip-64621-squid 2024-03-28T22:38:27.670 DEBUG:teuthology.orchestra.run.smithi043:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-28T22:38:27.778 INFO:tasks.workunit.client.0.smithi043.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-28T22:38:27.786 INFO:tasks.workunit.client.0.smithi043.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-28T22:38:27.786 INFO:tasks.workunit.client.0.smithi043.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-28T22:38:28.749 INFO:tasks.workunit.client.0.smithi043.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-28T22:38:28.889 INFO:tasks.workunit.client.0.smithi043.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-28T22:38:28.976 INFO:tasks.workunit.client.0.smithi043.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-28T22:38:28.983 INFO:tasks.workunit.client.0.smithi043.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-28T22:38:28.983 INFO:tasks.workunit.client.0.smithi043.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-28T22:38:29.077 INFO:tasks.workunit.client.0.smithi043.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-28T22:38:29.088 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-28T22:38:29.088 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-28T22:38:29.141 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-03-28T22:38:29.143 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-03-28T22:38:29.144 DEBUG:teuthology.orchestra.run.smithi043:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=55f5138942aa4866e49329c9103468889cec3f73 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-03-28T22:38:29.215 INFO:tasks.workunit.client.0.smithi043.stderr:+ dbench 1 2024-03-28T22:38:29.229 INFO:tasks.workunit.client.0.smithi043.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-03-28T22:38:29.229 INFO:tasks.workunit.client.0.smithi043.stdout: 2024-03-28T22:38:29.229 INFO:tasks.workunit.client.0.smithi043.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-03-28T22:38:29.237 INFO:tasks.workunit.client.0.smithi043.stdout:0 of 1 processes prepared for launch 0 sec 2024-03-28T22:38:29.338 INFO:tasks.workunit.client.0.smithi043.stdout:1 of 1 processes prepared for launch 0 sec 2024-03-28T22:38:29.338 INFO:tasks.workunit.client.0.smithi043.stdout:releasing clients 2024-03-28T22:38:29.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:29 smithi172 ceph-mon[36566]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T22:38:29.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:29 smithi082 ceph-mon[36471]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T22:38:29.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:29 smithi043 ceph-mon[31823]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T22:38:30.338 INFO:tasks.workunit.client.0.smithi043.stdout: 1 1555 26.09 MB/sec warmup 1 sec latency 12.363 ms 2024-03-28T22:38:31.338 INFO:tasks.workunit.client.0.smithi043.stdout: 1 3836 19.88 MB/sec warmup 2 sec latency 11.169 ms 2024-03-28T22:38:31.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:31 smithi172 ceph-mon[36566]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T22:38:31.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:31 smithi082 ceph-mon[36471]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T22:38:31.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:31 smithi043 ceph-mon[31823]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T22:38:32.338 INFO:tasks.workunit.client.0.smithi043.stdout: 1 5338 15.13 MB/sec warmup 3 sec latency 18.801 ms 2024-03-28T22:38:33.338 INFO:tasks.workunit.client.0.smithi043.stdout: 1 7441 14.79 MB/sec warmup 4 sec latency 10.405 ms 2024-03-28T22:38:33.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:33 smithi172 ceph-mon[36566]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T22:38:33.446 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.448+0000 7f4aea000640 1 -- 172.21.15.43:0/1785015406 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ae4074410 msgr2=0x7f4ae41121c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:33.446 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.448+0000 7f4aea000640 1 --2- 172.21.15.43:0/1785015406 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ae4074410 0x7f4ae41121c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4ad0009900 tx=0x7f4ad002bcc0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.446 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.448+0000 7f4aea000640 1 -- 172.21.15.43:0/1785015406 shutdown_connections 2024-03-28T22:38:33.446 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.448+0000 7f4aea000640 1 --2- 172.21.15.43:0/1785015406 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ae4112700 0x7f4ae4114af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.447 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.448+0000 7f4aea000640 1 --2- 172.21.15.43:0/1785015406 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ae4074410 0x7f4ae41121c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.447 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.448+0000 7f4aea000640 1 --2- 172.21.15.43:0/1785015406 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ae4073af0 0x7f4ae4073ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.447 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.448+0000 7f4aea000640 1 -- 172.21.15.43:0/1785015406 >> 172.21.15.43:0/1785015406 conn(0x7f4ae40fd1e0 msgr2=0x7f4ae40ff600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:33.447 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.448+0000 7f4aea000640 1 -- 172.21.15.43:0/1785015406 shutdown_connections 2024-03-28T22:38:33.447 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.448+0000 7f4aea000640 1 -- 172.21.15.43:0/1785015406 wait complete. 2024-03-28T22:38:33.449 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.449+0000 7f4aea000640 1 Processor -- start 2024-03-28T22:38:33.449 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.449+0000 7f4aea000640 1 -- start start 2024-03-28T22:38:33.449 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.450+0000 7f4aea000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ae4073af0 0x7f4ae419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:33.449 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.450+0000 7f4aea000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ae4074410 0x7f4ae419eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:33.449 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.450+0000 7f4aea000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ae4112700 0x7f4ae41a2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:33.449 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.450+0000 7f4aea000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae4118140 con 0x7f4ae4073af0 2024-03-28T22:38:33.449 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.450+0000 7f4aea000640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae4117fc0 con 0x7f4ae4074410 2024-03-28T22:38:33.449 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.450+0000 7f4aea000640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae41182c0 con 0x7f4ae4112700 2024-03-28T22:38:33.450 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.450+0000 7f4ae3e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ae4112700 0x7f4ae41a2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:33.450 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.450+0000 7f4ae2a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ae4074410 0x7f4ae419eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:33.450 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.450+0000 7f4ae3e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ae4112700 0x7f4ae41a2eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:39132/0 (socket says 172.21.15.43:39132) 2024-03-28T22:38:33.450 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.450+0000 7f4ae3e00640 1 -- 172.21.15.43:0/2996284167 learned_addr learned my addr 172.21.15.43:0/2996284167 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:33.450 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.450+0000 7f4ae3400640 1 --2- 172.21.15.43:0/2996284167 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ae4073af0 0x7f4ae419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:33.450 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.450+0000 7f4ae3e00640 1 -- 172.21.15.43:0/2996284167 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ae4074410 msgr2=0x7f4ae419eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:33.450 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.450+0000 7f4ae3e00640 1 --2- 172.21.15.43:0/2996284167 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ae4074410 0x7f4ae419eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.450 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.450+0000 7f4ae3e00640 1 -- 172.21.15.43:0/2996284167 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ae4073af0 msgr2=0x7f4ae419e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:33.450 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.450+0000 7f4ae3e00640 1 --2- 172.21.15.43:0/2996284167 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ae4073af0 0x7f4ae419e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.450 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.450+0000 7f4ae3e00640 1 -- 172.21.15.43:0/2996284167 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ae41a35e0 con 0x7f4ae4112700 2024-03-28T22:38:33.450 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.451+0000 7f4ae3e00640 1 --2- 172.21.15.43:0/2996284167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ae4112700 0x7f4ae41a2eb0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4adc00dac0 tx=0x7f4adc00df80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:33.450 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.451+0000 7f4adbe00640 1 -- 172.21.15.43:0/2996284167 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4adc017070 con 0x7f4ae4112700 2024-03-28T22:38:33.450 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.451+0000 7f4adbe00640 1 -- 172.21.15.43:0/2996284167 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4adc003e50 con 0x7f4ae4112700 2024-03-28T22:38:33.451 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.451+0000 7f4adbe00640 1 -- 172.21.15.43:0/2996284167 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4adc0106c0 con 0x7f4ae4112700 2024-03-28T22:38:33.451 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.451+0000 7f4aea000640 1 -- 172.21.15.43:0/2996284167 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ae41a38d0 con 0x7f4ae4112700 2024-03-28T22:38:33.451 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.451+0000 7f4aea000640 1 -- 172.21.15.43:0/2996284167 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ae41ab160 con 0x7f4ae4112700 2024-03-28T22:38:33.451 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:33 smithi082 ceph-mon[36471]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T22:38:33.452 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.454+0000 7f4aecc23740 1 Processor -- start 2024-03-28T22:38:33.452 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.454+0000 7f4adbe00640 1 -- 172.21.15.43:0/2996284167 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 27) ==== 136989+0+0 (secure 0 0 0) 0x7f4adc003970 con 0x7f4ae4112700 2024-03-28T22:38:33.452 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.455+0000 7f4aecc23740 1 -- start start 2024-03-28T22:38:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.455+0000 7f4aecc23740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3770fe3a0 0x55c3770fe780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.455+0000 7f4aecc23740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3770fecc0 0x55c377107550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.455+0000 7f4aecc23740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c377107a90 0x55c377109e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:33.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.455+0000 7f4aecc23740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c377061640 con 0x55c3770fe3a0 2024-03-28T22:38:33.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.455+0000 7f4aecc23740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55c37700a810 con 0x55c3770fecc0 2024-03-28T22:38:33.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.455+0000 7f4aecc23740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55c37705d6e0 con 0x55c377107a90 2024-03-28T22:38:33.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.455+0000 7f4ae2a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3770fe3a0 0x55c3770fe780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:33.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.455+0000 7f4ae3400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3770fecc0 0x55c377107550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:33.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.455+0000 7f4ae2a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3770fe3a0 0x55c3770fe780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46030/0 (socket says 172.21.15.43:46030) 2024-03-28T22:38:33.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.455+0000 7f4ae2a00640 1 -- 172.21.15.43:0/3527281104 learned_addr learned my addr 172.21.15.43:0/3527281104 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:33.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.455+0000 7f4ae3400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3770fecc0 0x55c377107550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45452/0 (socket says 172.21.15.43:45452) 2024-03-28T22:38:33.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.455+0000 7f4ae2a00640 1 --2- 172.21.15.43:0/3527281104 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c377107a90 0x55c377109e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:33.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.456+0000 7f4ae3400640 1 -- 172.21.15.43:0/3527281104 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c377107a90 msgr2=0x55c377109e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:33.456 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.456+0000 7f4ae3400640 1 --2- 172.21.15.43:0/3527281104 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c377107a90 0x55c377109e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.456 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.456+0000 7f4ae3400640 1 -- 172.21.15.43:0/3527281104 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3770fe3a0 msgr2=0x55c3770fe780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:33.456 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.456+0000 7f4ae3400640 1 --2- 172.21.15.43:0/3527281104 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3770fe3a0 0x55c3770fe780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.456 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.456+0000 7f4ae3400640 1 -- 172.21.15.43:0/3527281104 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c376d76210 con 0x55c3770fecc0 2024-03-28T22:38:33.456 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.456+0000 7f4ae3400640 1 --2- 172.21.15.43:0/3527281104 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3770fecc0 0x55c377107550 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f4ad4009c00 tx=0x7f4ad4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c25846cd59d73a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:33.456 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.456+0000 7f4acfe00640 1 -- 172.21.15.43:0/3527281104 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad4004230 con 0x55c3770fecc0 2024-03-28T22:38:33.456 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.456+0000 7f4adbe00640 1 --2- 172.21.15.43:0/2996284167 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4ac80a20b0 0x7f4ac80a4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:33.456 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.456+0000 7f4acfe00640 1 -- 172.21.15.43:0/3527281104 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ad40043d0 con 0x55c3770fecc0 2024-03-28T22:38:33.456 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.456+0000 7f4acfe00640 1 -- 172.21.15.43:0/3527281104 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad4004d90 con 0x55c3770fecc0 2024-03-28T22:38:33.456 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.457+0000 7f4adbe00640 1 -- 172.21.15.43:0/2996284167 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f4adc0ce0e0 con 0x7f4ae4112700 2024-03-28T22:38:33.456 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.457+0000 7f4ae2a00640 1 --2- 172.21.15.43:0/2996284167 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4ac80a20b0 0x7f4ac80a4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.457+0000 7f4aecc23740 1 -- 172.21.15.43:0/3527281104 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3770fecc0 msgr2=0x55c377107550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.457+0000 7f4aecc23740 1 --2- 172.21.15.43:0/3527281104 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3770fecc0 0x55c377107550 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f4ad4009c00 tx=0x7f4ad4011810 comp rx=0 tx=0).stop 2024-03-28T22:38:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.457+0000 7f4aecc23740 1 -- 172.21.15.43:0/3527281104 shutdown_connections 2024-03-28T22:38:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.457+0000 7f4aecc23740 1 --2- 172.21.15.43:0/3527281104 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3770fecc0 0x55c377107550 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.457+0000 7f4aecc23740 1 --2- 172.21.15.43:0/3527281104 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c377107a90 0x55c377109e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.457+0000 7f4aecc23740 1 --2- 172.21.15.43:0/3527281104 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3770fe3a0 0x55c3770fe780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.457+0000 7f4aecc23740 1 -- 172.21.15.43:0/3527281104 >> 172.21.15.43:0/3527281104 conn(0x55c3770f8cc0 msgr2=0x55c3770f90a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.458+0000 7f4ae2a00640 1 --2- 172.21.15.43:0/2996284167 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4ac80a20b0 0x7f4ac80a4570 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x55c3770fa0e0 tx=0x7f4ad0005fd0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.458+0000 7f4aecc23740 1 -- 172.21.15.43:0/3527281104 shutdown_connections 2024-03-28T22:38:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.458+0000 7f4aecc23740 1 -- 172.21.15.43:0/3527281104 wait complete. 2024-03-28T22:38:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.458+0000 7f4aecc23740 1 -- start start 2024-03-28T22:38:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.458+0000 7f4aecc23740 1 Processor -- start 2024-03-28T22:38:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.459+0000 7f4aecc23740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3770fe3a0 0x55c377104ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.459+0000 7f4aecc23740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3770fecc0 0x55c377105020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.459+0000 7f4aecc23740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c377107a90 0x55c377105560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.459+0000 7f4aecc23740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c37710a6c0 con 0x55c3770fecc0 2024-03-28T22:38:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.459+0000 7f4aecc23740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55c376fffd00 con 0x55c3770fe3a0 2024-03-28T22:38:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.459+0000 7f4aecc23740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55c37710a840 con 0x55c377107a90 2024-03-28T22:38:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.459+0000 7f4aecc23740 10 client.0 _ll_register_callbacks cb 0x55c376fea490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.459+0000 7f4ae3400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3770fe3a0 0x55c377104ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.459+0000 7f4ae3400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3770fe3a0 0x55c377104ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45468/0 (socket says 172.21.15.43:45468) 2024-03-28T22:38:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.459+0000 7f4ae3400640 1 -- 172.21.15.43:0/81499622 learned_addr learned my addr 172.21.15.43:0/81499622 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.459+0000 7f4ae3e00640 1 --2- 172.21.15.43:0/81499622 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c377107a90 0x55c377105560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.459+0000 7f4ae2a00640 1 --2- 172.21.15.43:0/81499622 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3770fecc0 0x55c377105020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.460+0000 7f4ae3e00640 1 -- 172.21.15.43:0/81499622 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3770fe3a0 msgr2=0x55c377104ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:33.462 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.460+0000 7f4ae3e00640 1 --2- 172.21.15.43:0/81499622 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3770fe3a0 0x55c377104ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.462 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.460+0000 7f4ae3e00640 1 -- 172.21.15.43:0/81499622 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3770fecc0 msgr2=0x55c377105020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:33.462 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.460+0000 7f4ae3e00640 1 --2- 172.21.15.43:0/81499622 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3770fecc0 0x55c377105020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.462 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.460+0000 7f4ae3e00640 1 -- 172.21.15.43:0/81499622 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c376e42620 con 0x55c377107a90 2024-03-28T22:38:33.462 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.460+0000 7f4ae3400640 1 --2- 172.21.15.43:0/81499622 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3770fe3a0 0x55c377104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:33.462 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.460+0000 7f4ae3e00640 1 --2- 172.21.15.43:0/81499622 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c377107a90 0x55c377105560 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4adc0cb2b0 tx=0x7f4adc09a160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39e73a518faf4ec0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:33.462 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.460+0000 7f4accc00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:38:33.462 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.460+0000 7f4accc00640 1 -- 172.21.15.43:0/81499622 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4adc020070 con 0x55c377107a90 2024-03-28T22:38:33.462 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.460+0000 7f4accc00640 1 -- 172.21.15.43:0/81499622 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f4adc09e150 con 0x55c377107a90 2024-03-28T22:38:33.462 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.460+0000 7f4aecc23740 1 -- 172.21.15.43:0/81499622 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ad40043d0 con 0x55c377107a90 2024-03-28T22:38:33.462 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.460+0000 7f4accc00640 1 -- 172.21.15.43:0/81499622 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4adc09a530 con 0x55c377107a90 2024-03-28T22:38:33.462 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.460+0000 7f4aecc23740 10 client.26080 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:33.462 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.460+0000 7f4aecc23740 1 -- 172.21.15.43:0/81499622 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c3771165b0 con 0x55c377107a90 2024-03-28T22:38:33.463 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.461+0000 7f4accc00640 1 -- 172.21.15.43:0/81499622 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ad40043d0 con 0x55c377107a90 2024-03-28T22:38:33.463 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.461+0000 7f4accc00640 1 -- 172.21.15.43:0/81499622 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f4adc023060 con 0x55c377107a90 2024-03-28T22:38:33.463 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.461+0000 7f4aecc23740 10 client.26080 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:33.463 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.461+0000 7f4aecc23740 10 client.26080 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:38:33.463 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.461+0000 7f4aecc23740 20 client.26080 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:33.463 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.461+0000 7f4aecc23740 1 --2- 172.21.15.43:0/81499622 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c377117a20 0x55c377119e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:33.463 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.461+0000 7f4aecc23740 4 client.26080 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:38:33.463 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.461+0000 7f4aecc23740 1 -- 172.21.15.43:0/81499622 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c377075030 con 0x55c377117a20 2024-03-28T22:38:33.463 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.461+0000 7f4ae3400640 1 --2- 172.21.15.43:0/81499622 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c377117a20 0x55c377119e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:33.463 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.462+0000 7f4ae3400640 1 --2- 172.21.15.43:0/81499622 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c377117a20 0x55c377119e10 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82dd20edfbc7a093 server_cookie=411ea8de8347f536 in_seq=0 out_seq=0 2024-03-28T22:38:33.463 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.462+0000 7f4accc00640 10 client.26080 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:33.464 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.463+0000 7f4accc00640 1 -- 172.21.15.43:0/81499622 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c377075030 con 0x55c377117a20 2024-03-28T22:38:33.464 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.463+0000 7f4accc00640 10 client.26080 handle_command_reply: tid=0 2024-03-28T22:38:33.465 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.463+0000 7f4aecc23740 1 client.26080 shutdown 2024-03-28T22:38:33.465 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.464+0000 7f4aecc23740 1 -- 172.21.15.43:0/81499622 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c377107a90 msgr2=0x55c377105560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:33.465 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.464+0000 7f4aecc23740 1 --2- 172.21.15.43:0/81499622 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c377107a90 0x55c377105560 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4adc0cb2b0 tx=0x7f4adc09a160 comp rx=0 tx=0).stop 2024-03-28T22:38:33.465 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.464+0000 7f4aecc23740 1 -- 172.21.15.43:0/81499622 shutdown_connections 2024-03-28T22:38:33.465 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.464+0000 7f4aecc23740 1 --2- 172.21.15.43:0/81499622 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c377107a90 0x55c377105560 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.465 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.464+0000 7f4aecc23740 1 --2- 172.21.15.43:0/81499622 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3770fecc0 0x55c377105020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.465 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.464+0000 7f4aecc23740 1 --2- 172.21.15.43:0/81499622 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c377117a20 0x55c377119e10 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.465 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.464+0000 7f4aecc23740 1 --2- 172.21.15.43:0/81499622 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3770fe3a0 0x55c377104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.466 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.464+0000 7f4aecc23740 1 -- 172.21.15.43:0/81499622 >> 172.21.15.43:0/81499622 conn(0x55c3770f8cc0 msgr2=0x55c3770faa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:33.466 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.464+0000 7f4accc00640 0 client.26080 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:33.466 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.464+0000 7f4aecc23740 1 -- 172.21.15.43:0/81499622 shutdown_connections 2024-03-28T22:38:33.466 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.464+0000 7f4aecc23740 1 -- 172.21.15.43:0/81499622 wait complete. 2024-03-28T22:38:33.466 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.464+0000 7f4aecc23740 20 client.26080 trim_cache size 0 max 16384 2024-03-28T22:38:33.479 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.481+0000 7f4aecc23740 1 Processor -- start 2024-03-28T22:38:33.479 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.481+0000 7f4aecc23740 1 -- start start 2024-03-28T22:38:33.479 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.481+0000 7f4aecc23740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3770fe3a0 0x55c377113460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:33.479 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.481+0000 7f4aecc23740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c3770fecc0 0x55c37711f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:33.479 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.481+0000 7f4aecc23740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c377107a90 0x55c3771033b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:33.479 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.481+0000 7f4aecc23740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae41b5180 con 0x55c377107a90 2024-03-28T22:38:33.479 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.481+0000 7f4aecc23740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae41b6cc0 con 0x55c3770fe3a0 2024-03-28T22:38:33.479 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.481+0000 7f4aecc23740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae41b5850 con 0x55c3770fecc0 2024-03-28T22:38:33.480 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.481+0000 7f4ae3400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3770fe3a0 0x55c377113460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:33.480 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.481+0000 7f4ae3400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3770fe3a0 0x55c377113460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45474/0 (socket says 172.21.15.43:45474) 2024-03-28T22:38:33.481 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.481+0000 7f4ae3400640 1 -- 172.21.15.43:0/2719729268 learned_addr learned my addr 172.21.15.43:0/2719729268 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:33.481 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.481+0000 7f4ae2a00640 1 --2- 172.21.15.43:0/2719729268 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c3770fecc0 0x55c37711f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:33.481 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.482+0000 7f4ae3e00640 1 --2- 172.21.15.43:0/2719729268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c377107a90 0x55c3771033b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:33.481 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.482+0000 7f4ae3400640 1 -- 172.21.15.43:0/2719729268 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c3770fecc0 msgr2=0x55c37711f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:33.481 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.482+0000 7f4ae3400640 1 --2- 172.21.15.43:0/2719729268 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c3770fecc0 0x55c37711f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.481 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.482+0000 7f4ae3400640 1 -- 172.21.15.43:0/2719729268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c377107a90 msgr2=0x55c3771033b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:33.481 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.482+0000 7f4ae3400640 1 --2- 172.21.15.43:0/2719729268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c377107a90 0x55c3771033b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.481 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.482+0000 7f4ae3400640 1 -- 172.21.15.43:0/2719729268 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c37711a180 con 0x55c3770fe3a0 2024-03-28T22:38:33.481 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.482+0000 7f4ae3400640 1 --2- 172.21.15.43:0/2719729268 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3770fe3a0 0x55c377113460 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f4ad4004940 tx=0x7f4ad4004970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf6d690d7265aba9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:33.481 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.482+0000 7f4acf400640 1 -- 172.21.15.43:0/2719729268 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad4032070 con 0x55c3770fe3a0 2024-03-28T22:38:33.481 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.483+0000 7f4acf400640 1 -- 172.21.15.43:0/2719729268 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ad402a070 con 0x55c3770fe3a0 2024-03-28T22:38:33.481 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.483+0000 7f4acf400640 1 -- 172.21.15.43:0/2719729268 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad401e040 con 0x55c3770fe3a0 2024-03-28T22:38:33.484 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.483+0000 7f4aecc23740 1 -- 172.21.15.43:0/2719729268 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3770fe3a0 msgr2=0x55c377113460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:33.484 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.483+0000 7f4aecc23740 1 --2- 172.21.15.43:0/2719729268 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3770fe3a0 0x55c377113460 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f4ad4004940 tx=0x7f4ad4004970 comp rx=0 tx=0).stop 2024-03-28T22:38:33.484 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.483+0000 7f4aecc23740 1 -- 172.21.15.43:0/2719729268 shutdown_connections 2024-03-28T22:38:33.484 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.483+0000 7f4aecc23740 1 --2- 172.21.15.43:0/2719729268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c377107a90 0x55c3771033b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.484 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.483+0000 7f4aecc23740 1 --2- 172.21.15.43:0/2719729268 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c3770fecc0 0x55c37711f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.484 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.483+0000 7f4aecc23740 1 --2- 172.21.15.43:0/2719729268 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3770fe3a0 0x55c377113460 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.484 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.483+0000 7f4aecc23740 1 -- 172.21.15.43:0/2719729268 >> 172.21.15.43:0/2719729268 conn(0x55c3770f8cc0 msgr2=0x55c3770f90a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:33.484 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.484+0000 7f4aecc23740 1 -- 172.21.15.43:0/2719729268 shutdown_connections 2024-03-28T22:38:33.484 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.484+0000 7f4aecc23740 1 -- 172.21.15.43:0/2719729268 wait complete. 2024-03-28T22:38:33.484 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.484+0000 7f4aecc23740 1 -- start start 2024-03-28T22:38:33.484 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.485+0000 7f4aecc23740 1 Processor -- start 2024-03-28T22:38:33.485 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.485+0000 7f4aecc23740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3770fe3a0 0x55c3771015b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:33.485 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.485+0000 7f4aecc23740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3770fecc0 0x55c377101af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:33.485 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.485+0000 7f4aecc23740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c377107a90 0x55c377102030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:33.485 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.485+0000 7f4aecc23740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae41b65f0 con 0x55c3770fecc0 2024-03-28T22:38:33.485 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.485+0000 7f4aecc23740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae41b5f20 con 0x55c3770fe3a0 2024-03-28T22:38:33.485 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.485+0000 7f4aecc23740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae41b3160 con 0x55c377107a90 2024-03-28T22:38:33.485 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.485+0000 7f4aecc23740 10 client.0 _ll_register_callbacks cb 0x7f4adc0092a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:33.485 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.485+0000 7f4ae3400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3770fe3a0 0x55c3771015b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:33.485 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.485+0000 7f4ae3400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3770fe3a0 0x55c3771015b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45478/0 (socket says 172.21.15.43:45478) 2024-03-28T22:38:33.485 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.485+0000 7f4ae3e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c377107a90 0x55c377102030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:33.485 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.485+0000 7f4ae3400640 1 -- 172.21.15.43:0/3738160699 learned_addr learned my addr 172.21.15.43:0/3738160699 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:33.485 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.485+0000 7f4ae2a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3770fecc0 0x55c377101af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:33.485 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.486+0000 7f4ae3400640 1 -- 172.21.15.43:0/3738160699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c377107a90 msgr2=0x55c377102030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:33.485 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.486+0000 7f4ae3400640 1 --2- 172.21.15.43:0/3738160699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c377107a90 0x55c377102030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.485 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.486+0000 7f4ae3400640 1 -- 172.21.15.43:0/3738160699 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3770fecc0 msgr2=0x55c377101af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:33.486 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.486+0000 7f4ae3400640 1 --2- 172.21.15.43:0/3738160699 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3770fecc0 0x55c377101af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.486 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.486+0000 7f4ae3400640 1 -- 172.21.15.43:0/3738160699 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c37717cd20 con 0x55c3770fe3a0 2024-03-28T22:38:33.486 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.486+0000 7f4ae3400640 1 --2- 172.21.15.43:0/3738160699 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3770fe3a0 0x55c3771015b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f4ad400a010 tx=0x7f4ad4005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22cde21aea379228 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:33.486 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.486+0000 7f4accc00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:38:33.486 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.486+0000 7f4accc00640 1 -- 172.21.15.43:0/3738160699 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad403a060 con 0x55c3770fe3a0 2024-03-28T22:38:33.486 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.486+0000 7f4accc00640 1 -- 172.21.15.43:0/3738160699 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f4ad4026440 con 0x55c3770fe3a0 2024-03-28T22:38:33.486 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.486+0000 7f4aecc23740 1 -- 172.21.15.43:0/3738160699 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ad402a070 con 0x55c3770fe3a0 2024-03-28T22:38:33.487 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.487+0000 7f4accc00640 1 -- 172.21.15.43:0/3738160699 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4ad4041050 con 0x55c3770fe3a0 2024-03-28T22:38:33.487 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.487+0000 7f4aecc23740 10 client.26138 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:33.487 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.487+0000 7f4aecc23740 1 -- 172.21.15.43:0/3738160699 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c37717dfb0 con 0x55c3770fe3a0 2024-03-28T22:38:33.487 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.487+0000 7f4accc00640 1 -- 172.21.15.43:0/3738160699 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ad402a070 con 0x55c3770fe3a0 2024-03-28T22:38:33.488 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.487+0000 7f4accc00640 1 -- 172.21.15.43:0/3738160699 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f4ad4024040 con 0x55c3770fe3a0 2024-03-28T22:38:33.488 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.487+0000 7f4aecc23740 10 client.26138 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:33.488 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.487+0000 7f4aecc23740 10 client.26138 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:38:33.488 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.487+0000 7f4aecc23740 20 client.26138 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:33.488 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.487+0000 7f4aecc23740 1 --2- 172.21.15.43:0/3738160699 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c37717e4e0 0x55c377180900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:33.488 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.487+0000 7f4aecc23740 4 client.26138 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T22:38:33.488 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.487+0000 7f4aecc23740 1 -- 172.21.15.43:0/3738160699 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c37706eb80 con 0x55c37717e4e0 2024-03-28T22:38:33.488 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.487+0000 7f4ae2a00640 1 --2- 172.21.15.43:0/3738160699 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c37717e4e0 0x55c377180900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:33.488 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.488+0000 7f4ae2a00640 1 --2- 172.21.15.43:0/3738160699 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c37717e4e0 0x55c377180900 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2e60d79ef691e8e server_cookie=13f452504abf52e4 in_seq=0 out_seq=0 2024-03-28T22:38:33.488 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.488+0000 7f4accc00640 10 client.26138 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:33.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:33 smithi043 ceph-mon[31823]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T22:38:33.648 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.649+0000 7f4accc00640 1 -- 172.21.15.43:0/3738160699 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c37706eb80 con 0x55c37717e4e0 2024-03-28T22:38:33.648 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.649+0000 7f4accc00640 10 client.26138 handle_command_reply: tid=0 2024-03-28T22:38:33.648 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.649+0000 7f4aecc23740 1 client.26138 shutdown 2024-03-28T22:38:33.648 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.649+0000 7f4aecc23740 1 -- 172.21.15.43:0/3738160699 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3770fe3a0 msgr2=0x55c3771015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:33.648 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.649+0000 7f4aecc23740 1 --2- 172.21.15.43:0/3738160699 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3770fe3a0 0x55c3771015b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f4ad400a010 tx=0x7f4ad4005dc0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.648 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.649+0000 7f4aecc23740 1 -- 172.21.15.43:0/3738160699 shutdown_connections 2024-03-28T22:38:33.649 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.649+0000 7f4aecc23740 1 --2- 172.21.15.43:0/3738160699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c377107a90 0x55c377102030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.649 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.649+0000 7f4aecc23740 1 --2- 172.21.15.43:0/3738160699 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3770fecc0 0x55c377101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.649 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.649+0000 7f4aecc23740 1 --2- 172.21.15.43:0/3738160699 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c37717e4e0 0x55c377180900 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.649 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.649+0000 7f4aecc23740 1 --2- 172.21.15.43:0/3738160699 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3770fe3a0 0x55c3771015b0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.649 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.649+0000 7f4aecc23740 1 -- 172.21.15.43:0/3738160699 >> 172.21.15.43:0/3738160699 conn(0x55c3770f8cc0 msgr2=0x55c377103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:33.649 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.649+0000 7f4accc00640 0 client.26138 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:33.649 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.650+0000 7f4aecc23740 1 -- 172.21.15.43:0/3738160699 shutdown_connections 2024-03-28T22:38:33.649 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.650+0000 7f4aecc23740 1 -- 172.21.15.43:0/3738160699 wait complete. 2024-03-28T22:38:33.649 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.650+0000 7f4aecc23740 20 client.26138 trim_cache size 0 max 16384 2024-03-28T22:38:33.650 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:{ 2024-03-28T22:38:33.650 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "message": "", 2024-03-28T22:38:33.650 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "return_code": 0 2024-03-28T22:38:33.650 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:} 2024-03-28T22:38:33.651 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.650+0000 7f4aea000640 1 -- 172.21.15.43:0/2996284167 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4ac80a20b0 msgr2=0x7f4ac80a4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:33.651 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.650+0000 7f4aea000640 1 --2- 172.21.15.43:0/2996284167 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4ac80a20b0 0x7f4ac80a4570 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x55c3770fa0e0 tx=0x7f4ad0005fd0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.651 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.650+0000 7f4aea000640 1 -- 172.21.15.43:0/2996284167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ae4112700 msgr2=0x7f4ae41a2eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:33.651 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.650+0000 7f4aea000640 1 --2- 172.21.15.43:0/2996284167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ae4112700 0x7f4ae41a2eb0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4adc00dac0 tx=0x7f4adc00df80 comp rx=0 tx=0).stop 2024-03-28T22:38:33.651 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.651+0000 7f4aea000640 1 -- 172.21.15.43:0/2996284167 shutdown_connections 2024-03-28T22:38:33.652 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.651+0000 7f4aea000640 1 --2- 172.21.15.43:0/2996284167 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4ac80a20b0 0x7f4ac80a4570 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.652 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.651+0000 7f4aea000640 1 --2- 172.21.15.43:0/2996284167 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ae4074410 0x7f4ae419eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.652 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.651+0000 7f4aea000640 1 --2- 172.21.15.43:0/2996284167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4ae4112700 0x7f4ae41a2eb0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.652 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.651+0000 7f4aea000640 1 --2- 172.21.15.43:0/2996284167 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ae4073af0 0x7f4ae419e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:33.652 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.651+0000 7f4aea000640 1 -- 172.21.15.43:0/2996284167 >> 172.21.15.43:0/2996284167 conn(0x7f4ae40fd1e0 msgr2=0x7f4ae40fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:33.652 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.651+0000 7f4aea000640 1 -- 172.21.15.43:0/2996284167 shutdown_connections 2024-03-28T22:38:33.652 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:33.651+0000 7f4aea000640 1 -- 172.21.15.43:0/2996284167 wait complete. 2024-03-28T22:38:34.338 INFO:tasks.workunit.client.0.smithi043.stdout: 1 8808 12.89 MB/sec warmup 5 sec latency 47.539 ms 2024-03-28T22:38:35.339 INFO:tasks.workunit.client.0.smithi043.stdout: 1 10943 13.00 MB/sec warmup 6 sec latency 13.288 ms 2024-03-28T22:38:35.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:35 smithi172 ceph-mon[36566]: pgmap v44: 129 pgs: 129 active+clean; 4.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 284 KiB/s wr, 45 op/s 2024-03-28T22:38:35.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:35 smithi082 ceph-mon[36471]: pgmap v44: 129 pgs: 129 active+clean; 4.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 284 KiB/s wr, 45 op/s 2024-03-28T22:38:35.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:35 smithi043 ceph-mon[31823]: pgmap v44: 129 pgs: 129 active+clean; 4.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 284 KiB/s wr, 45 op/s 2024-03-28T22:38:36.339 INFO:tasks.workunit.client.0.smithi043.stdout: 1 12476 11.95 MB/sec warmup 7 sec latency 18.094 ms 2024-03-28T22:38:37.339 INFO:tasks.workunit.client.0.smithi043.stdout: 1 14566 12.18 MB/sec warmup 8 sec latency 10.750 ms 2024-03-28T22:38:37.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:37 smithi172 ceph-mon[36566]: pgmap v45: 129 pgs: 129 active+clean; 4.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 284 KiB/s wr, 40 op/s 2024-03-28T22:38:37.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:37 smithi082 ceph-mon[36471]: pgmap v45: 129 pgs: 129 active+clean; 4.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 284 KiB/s wr, 40 op/s 2024-03-28T22:38:37.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:37 smithi043 ceph-mon[31823]: pgmap v45: 129 pgs: 129 active+clean; 4.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 284 KiB/s wr, 40 op/s 2024-03-28T22:38:38.339 INFO:tasks.workunit.client.0.smithi043.stdout: 1 16162 11.44 MB/sec warmup 9 sec latency 23.919 ms 2024-03-28T22:38:38.642 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:38:38.642 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:38:38.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.890+0000 7f12db200640 1 -- 172.21.15.43:0/1953149015 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12dc104190 msgr2=0x7f12dc1106a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:38.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.890+0000 7f12db200640 1 --2- 172.21.15.43:0/1953149015 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12dc104190 0x7f12dc1106a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f12d4009cd0 tx=0x7f12d402c0e0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.890+0000 7f12db200640 1 -- 172.21.15.43:0/1953149015 shutdown_connections 2024-03-28T22:38:38.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.890+0000 7f12db200640 1 --2- 172.21.15.43:0/1953149015 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12dc104190 0x7f12dc1106a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.890+0000 7f12db200640 1 --2- 172.21.15.43:0/1953149015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12dc1037d0 0x7f12dc103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.890+0000 7f12db200640 1 --2- 172.21.15.43:0/1953149015 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f12dc1097e0 0x7f12dc109bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.890+0000 7f12db200640 1 -- 172.21.15.43:0/1953149015 >> 172.21.15.43:0/1953149015 conn(0x7f12dc0ff280 msgr2=0x7f12dc1016a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:38.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.890+0000 7f12db200640 1 -- 172.21.15.43:0/1953149015 shutdown_connections 2024-03-28T22:38:38.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.890+0000 7f12db200640 1 -- 172.21.15.43:0/1953149015 wait complete. 2024-03-28T22:38:38.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.891+0000 7f12db200640 1 Processor -- start 2024-03-28T22:38:38.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.892+0000 7f12db200640 1 -- start start 2024-03-28T22:38:38.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.892+0000 7f12db200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12dc1037d0 0x7f12dc19a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:38.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.892+0000 7f12db200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12dc104190 0x7f12dc19a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:38.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.892+0000 7f12db200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f12dc1097e0 0x7f12dc19e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:38.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.892+0000 7f12db200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f12dc113bf0 con 0x7f12dc104190 2024-03-28T22:38:38.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.892+0000 7f12db200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f12dc113a70 con 0x7f12dc1097e0 2024-03-28T22:38:38.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.892+0000 7f12db200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f12dc113d70 con 0x7f12dc1037d0 2024-03-28T22:38:38.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.892+0000 7f12d9e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12dc1037d0 0x7f12dc19a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:38.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.892+0000 7f12d9e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12dc1037d0 0x7f12dc19a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35924/0 (socket says 172.21.15.43:35924) 2024-03-28T22:38:38.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.892+0000 7f12da800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f12dc1097e0 0x7f12dc19e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:38.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.892+0000 7f12d9e00640 1 -- 172.21.15.43:0/2615492418 learned_addr learned my addr 172.21.15.43:0/2615492418 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:38.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.892+0000 7f12d3e00640 1 --2- 172.21.15.43:0/2615492418 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12dc104190 0x7f12dc19a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:38.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.893+0000 7f12d9e00640 1 -- 172.21.15.43:0/2615492418 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f12dc1097e0 msgr2=0x7f12dc19e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:38.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.893+0000 7f12d9e00640 1 --2- 172.21.15.43:0/2615492418 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f12dc1097e0 0x7f12dc19e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.893+0000 7f12d9e00640 1 -- 172.21.15.43:0/2615492418 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12dc104190 msgr2=0x7f12dc19a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:38.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.893+0000 7f12d9e00640 1 --2- 172.21.15.43:0/2615492418 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12dc104190 0x7f12dc19a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.893+0000 7f12d9e00640 1 -- 172.21.15.43:0/2615492418 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12dc19f100 con 0x7f12dc1037d0 2024-03-28T22:38:38.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.893+0000 7f12da800640 1 --2- 172.21.15.43:0/2615492418 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f12dc1097e0 0x7f12dc19e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:38.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.893+0000 7f12d9e00640 1 --2- 172.21.15.43:0/2615492418 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12dc1037d0 0x7f12dc19a110 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f12cc00e9f0 tx=0x7f12cc00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:38.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.893+0000 7f12d2a00640 1 -- 172.21.15.43:0/2615492418 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12cc00cd80 con 0x7f12dc1037d0 2024-03-28T22:38:38.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.893+0000 7f12d2a00640 1 -- 172.21.15.43:0/2615492418 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12cc005500 con 0x7f12dc1037d0 2024-03-28T22:38:38.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.893+0000 7f12d2a00640 1 -- 172.21.15.43:0/2615492418 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12cc010690 con 0x7f12dc1037d0 2024-03-28T22:38:38.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.893+0000 7f12db200640 1 -- 172.21.15.43:0/2615492418 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12dc19f3f0 con 0x7f12dc1037d0 2024-03-28T22:38:38.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.893+0000 7f12db200640 1 -- 172.21.15.43:0/2615492418 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12dc1a6c80 con 0x7f12dc1037d0 2024-03-28T22:38:38.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.897+0000 7f12d2a00640 1 -- 172.21.15.43:0/2615492418 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 27) ==== 136989+0+0 (secure 0 0 0) 0x7f12cc0046c0 con 0x7f12dc1037d0 2024-03-28T22:38:38.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.897+0000 7f12e3794740 1 Processor -- start 2024-03-28T22:38:38.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.897+0000 7f12e3794740 1 -- start start 2024-03-28T22:38:38.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.898+0000 7f12e3794740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558cc13abdb0 0x558cc13aab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:38.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.898+0000 7f12e3794740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558cc13ab0d0 0x558cc13ab550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:38.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.898+0000 7f12e3794740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558cc13b4440 0x558cc13b6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:38.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.898+0000 7f12e3794740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x558cc130f100 con 0x558cc13b4440 2024-03-28T22:38:38.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.898+0000 7f12e3794740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x558cc12b7ba0 con 0x558cc13ab0d0 2024-03-28T22:38:38.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.898+0000 7f12e3794740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x558cc130b1a0 con 0x558cc13abdb0 2024-03-28T22:38:38.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.898+0000 7f12d9e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558cc13ab0d0 0x558cc13ab550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:38.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.898+0000 7f12da800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558cc13abdb0 0x558cc13aab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:38.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.898+0000 7f12da800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558cc13abdb0 0x558cc13aab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35932/0 (socket says 172.21.15.43:35932) 2024-03-28T22:38:38.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.898+0000 7f12da800640 1 -- 172.21.15.43:0/3729738340 learned_addr learned my addr 172.21.15.43:0/3729738340 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:38.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.898+0000 7f12d3e00640 1 --2- 172.21.15.43:0/3729738340 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558cc13b4440 0x558cc13b6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:38.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.898+0000 7f12da800640 1 -- 172.21.15.43:0/3729738340 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558cc13ab0d0 msgr2=0x558cc13ab550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:38.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.898+0000 7f12da800640 1 --2- 172.21.15.43:0/3729738340 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558cc13ab0d0 0x558cc13ab550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.898+0000 7f12d2a00640 1 --2- 172.21.15.43:0/2615492418 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f12c00a2090 0x7f12c00a4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:38.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.898+0000 7f12da800640 1 -- 172.21.15.43:0/3729738340 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558cc13b4440 msgr2=0x558cc13b6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:38.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.898+0000 7f12da800640 1 --2- 172.21.15.43:0/3729738340 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558cc13b4440 0x558cc13b6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.898+0000 7f12da800640 1 -- 172.21.15.43:0/3729738340 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cc12e81a0 con 0x558cc13abdb0 2024-03-28T22:38:38.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.898+0000 7f12d3e00640 1 --2- 172.21.15.43:0/3729738340 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558cc13b4440 0x558cc13b6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:38:38.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.899+0000 7f12d2a00640 1 -- 172.21.15.43:0/2615492418 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f12cc0ceaf0 con 0x7f12dc1037d0 2024-03-28T22:38:38.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.899+0000 7f12da800640 1 --2- 172.21.15.43:0/2615492418 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f12c00a2090 0x7f12c00a4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:38.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.899+0000 7f12da800640 1 --2- 172.21.15.43:0/3729738340 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558cc13abdb0 0x558cc13aab00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f12d4009ca0 tx=0x7f12d4005f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b61d074bff22c7f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:38.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.899+0000 7f12cb000640 1 -- 172.21.15.43:0/3729738340 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12d4039040 con 0x558cc13abdb0 2024-03-28T22:38:38.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.900+0000 7f12cb000640 1 -- 172.21.15.43:0/3729738340 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12d4043050 con 0x558cc13abdb0 2024-03-28T22:38:38.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.900+0000 7f12cb000640 1 -- 172.21.15.43:0/3729738340 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12d4051070 con 0x558cc13abdb0 2024-03-28T22:38:38.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.900+0000 7f12e3794740 1 -- 172.21.15.43:0/3729738340 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558cc13abdb0 msgr2=0x558cc13aab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:38.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.900+0000 7f12e3794740 1 --2- 172.21.15.43:0/3729738340 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558cc13abdb0 0x558cc13aab00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f12d4009ca0 tx=0x7f12d4005f80 comp rx=0 tx=0).stop 2024-03-28T22:38:38.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.900+0000 7f12da800640 1 --2- 172.21.15.43:0/2615492418 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f12c00a2090 0x7f12c00a4550 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f12d4046000 tx=0x7f12d404b000 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:38.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.901+0000 7f12e3794740 1 -- 172.21.15.43:0/3729738340 shutdown_connections 2024-03-28T22:38:38.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.901+0000 7f12e3794740 1 --2- 172.21.15.43:0/3729738340 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558cc13ab0d0 0x558cc13ab550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.901+0000 7f12e3794740 1 --2- 172.21.15.43:0/3729738340 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558cc13b4440 0x558cc13b6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.901+0000 7f12e3794740 1 --2- 172.21.15.43:0/3729738340 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558cc13abdb0 0x558cc13aab00 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.901+0000 7f12e3794740 1 -- 172.21.15.43:0/3729738340 >> 172.21.15.43:0/3729738340 conn(0x558cc1371a10 msgr2=0x558cc13a5b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:38.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.901+0000 7f12e3794740 1 -- 172.21.15.43:0/3729738340 shutdown_connections 2024-03-28T22:38:38.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.901+0000 7f12e3794740 1 -- 172.21.15.43:0/3729738340 wait complete. 2024-03-28T22:38:38.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.901+0000 7f12e3794740 1 -- start start 2024-03-28T22:38:38.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.901+0000 7f12e3794740 1 Processor -- start 2024-03-28T22:38:38.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.902+0000 7f12e3794740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558cc13ab0d0 0x558cc13b1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:38.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.902+0000 7f12e3794740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558cc13abdb0 0x558cc13b1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:38.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.902+0000 7f12e3794740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558cc13b4440 0x558cc13b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:38.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.902+0000 7f12e3794740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x558cc13b72b0 con 0x558cc13b4440 2024-03-28T22:38:38.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.902+0000 7f12e3794740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x558cc12ad250 con 0x558cc13ab0d0 2024-03-28T22:38:38.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.902+0000 7f12e3794740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x558cc13b7430 con 0x558cc13abdb0 2024-03-28T22:38:38.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.902+0000 7f12e3794740 10 client.0 _ll_register_callbacks cb 0x558cc1022be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:38.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.902+0000 7f12d9e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558cc13ab0d0 0x558cc13b1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:38.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.902+0000 7f12d3e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558cc13abdb0 0x558cc13b1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:38.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.902+0000 7f12d9e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558cc13ab0d0 0x558cc13b1420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:40398/0 (socket says 172.21.15.43:40398) 2024-03-28T22:38:38.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.902+0000 7f12d9e00640 1 -- 172.21.15.43:0/3890403526 learned_addr learned my addr 172.21.15.43:0/3890403526 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:38.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.902+0000 7f12da800640 1 --2- 172.21.15.43:0/3890403526 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558cc13b4440 0x558cc13b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:38.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.903+0000 7f12d9e00640 1 -- 172.21.15.43:0/3890403526 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558cc13abdb0 msgr2=0x558cc13b1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:38.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.903+0000 7f12d9e00640 1 --2- 172.21.15.43:0/3890403526 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558cc13abdb0 0x558cc13b1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.903+0000 7f12d9e00640 1 -- 172.21.15.43:0/3890403526 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558cc13b4440 msgr2=0x558cc13b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:38.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.903+0000 7f12d9e00640 1 --2- 172.21.15.43:0/3890403526 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558cc13b4440 0x558cc13b1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.903+0000 7f12d9e00640 1 -- 172.21.15.43:0/3890403526 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cc12c9960 con 0x558cc13ab0d0 2024-03-28T22:38:38.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.903+0000 7f12da800640 1 --2- 172.21.15.43:0/3890403526 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558cc13b4440 0x558cc13b1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:38:38.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.903+0000 7f12d9e00640 1 --2- 172.21.15.43:0/3890403526 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558cc13ab0d0 0x558cc13b1420 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f12cc00e9c0 tx=0x7f12cc0d2840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c461d893d61f6f10 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:38.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.903+0000 7f12bf400640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:38:38.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.903+0000 7f12bf400640 1 -- 172.21.15.43:0/3890403526 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12cc09acc0 con 0x558cc13ab0d0 2024-03-28T22:38:38.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.903+0000 7f12bf400640 1 -- 172.21.15.43:0/3890403526 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f12cc0a3400 con 0x558cc13ab0d0 2024-03-28T22:38:38.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.903+0000 7f12e3794740 1 -- 172.21.15.43:0/3890403526 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12d4043050 con 0x558cc13ab0d0 2024-03-28T22:38:38.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.904+0000 7f12bf400640 1 -- 172.21.15.43:0/3890403526 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f12cc09a850 con 0x558cc13ab0d0 2024-03-28T22:38:38.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.904+0000 7f12e3794740 10 client.26150 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:38.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.904+0000 7f12e3794740 1 -- 172.21.15.43:0/3890403526 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558cc10ed7d0 con 0x558cc13ab0d0 2024-03-28T22:38:38.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.904+0000 7f12bf400640 1 -- 172.21.15.43:0/3890403526 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12d4043050 con 0x558cc13ab0d0 2024-03-28T22:38:38.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.904+0000 7f12bf400640 1 -- 172.21.15.43:0/3890403526 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f12cc0cc090 con 0x558cc13ab0d0 2024-03-28T22:38:38.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.904+0000 7f12e3794740 10 client.26150 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:38.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.904+0000 7f12e3794740 10 client.26150 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:38:38.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.904+0000 7f12e3794740 20 client.26150 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:38.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.904+0000 7f12e3794740 1 --2- 172.21.15.43:0/3890403526 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x558cc13c4920 0x558cc13c6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:38.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.904+0000 7f12e3794740 4 client.26150 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:38:38.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.904+0000 7f12e3794740 1 -- 172.21.15.43:0/3890403526 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558cc131c640 con 0x558cc13c4920 2024-03-28T22:38:38.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.905+0000 7f12d3e00640 1 --2- 172.21.15.43:0/3890403526 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x558cc13c4920 0x558cc13c6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:38.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.905+0000 7f12d3e00640 1 --2- 172.21.15.43:0/3890403526 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x558cc13c4920 0x558cc13c6dc0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2dbd0910f473128 server_cookie=186312a4bf2e2d5d in_seq=0 out_seq=0 2024-03-28T22:38:38.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.905+0000 7f12bf400640 10 client.26150 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:38.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.906+0000 7f12bf400640 1 -- 172.21.15.43:0/3890403526 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558cc131c640 con 0x558cc13c4920 2024-03-28T22:38:38.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.907+0000 7f12bf400640 10 client.26150 handle_command_reply: tid=0 2024-03-28T22:38:38.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.907+0000 7f12e3794740 1 client.26150 shutdown 2024-03-28T22:38:38.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.907+0000 7f12e3794740 1 -- 172.21.15.43:0/3890403526 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558cc13ab0d0 msgr2=0x558cc13b1420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:38.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.907+0000 7f12e3794740 1 --2- 172.21.15.43:0/3890403526 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558cc13ab0d0 0x558cc13b1420 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f12cc00e9c0 tx=0x7f12cc0d2840 comp rx=0 tx=0).stop 2024-03-28T22:38:38.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.907+0000 7f12e3794740 1 -- 172.21.15.43:0/3890403526 shutdown_connections 2024-03-28T22:38:38.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.907+0000 7f12e3794740 1 --2- 172.21.15.43:0/3890403526 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558cc13b4440 0x558cc13b1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.907+0000 7f12e3794740 1 --2- 172.21.15.43:0/3890403526 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558cc13abdb0 0x558cc13b1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.907+0000 7f12e3794740 1 --2- 172.21.15.43:0/3890403526 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x558cc13c4920 0x558cc13c6dc0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.907+0000 7f12e3794740 1 --2- 172.21.15.43:0/3890403526 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558cc13ab0d0 0x558cc13b1420 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.907+0000 7f12e3794740 1 -- 172.21.15.43:0/3890403526 >> 172.21.15.43:0/3890403526 conn(0x558cc1371a10 msgr2=0x558cc13a7870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:38.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.907+0000 7f12bf400640 0 client.26150 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:38.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.907+0000 7f12e3794740 1 -- 172.21.15.43:0/3890403526 shutdown_connections 2024-03-28T22:38:38.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.907+0000 7f12e3794740 1 -- 172.21.15.43:0/3890403526 wait complete. 2024-03-28T22:38:38.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.907+0000 7f12e3794740 20 client.26150 trim_cache size 0 max 16384 2024-03-28T22:38:38.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.928+0000 7f12e3794740 1 Processor -- start 2024-03-28T22:38:38.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.928+0000 7f12e3794740 1 -- start start 2024-03-28T22:38:38.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.928+0000 7f12e3794740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558cc13ab0d0 0x558cc13b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:38.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.929+0000 7f12e3794740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558cc13abdb0 0x558cc13b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:38.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.929+0000 7f12e3794740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558cc13b4440 0x558cc13c00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:38.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.929+0000 7f12e3794740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f12dc1b0cb0 con 0x558cc13b4440 2024-03-28T22:38:38.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.929+0000 7f12e3794740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f12dc1b27f0 con 0x558cc13ab0d0 2024-03-28T22:38:38.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.929+0000 7f12e3794740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f12dc1b1380 con 0x558cc13abdb0 2024-03-28T22:38:38.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.929+0000 7f12d3e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558cc13abdb0 0x558cc13b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:38.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.929+0000 7f12d3e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558cc13abdb0 0x558cc13b3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35946/0 (socket says 172.21.15.43:35946) 2024-03-28T22:38:38.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.929+0000 7f12d3e00640 1 -- 172.21.15.43:0/1523644026 learned_addr learned my addr 172.21.15.43:0/1523644026 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:38.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.929+0000 7f12d9e00640 1 --2- 172.21.15.43:0/1523644026 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558cc13ab0d0 0x558cc13b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:38.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.929+0000 7f12da800640 1 --2- 172.21.15.43:0/1523644026 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558cc13b4440 0x558cc13c00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.929+0000 7f12d3e00640 1 -- 172.21.15.43:0/1523644026 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558cc13ab0d0 msgr2=0x558cc13b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.929+0000 7f12d3e00640 1 --2- 172.21.15.43:0/1523644026 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558cc13ab0d0 0x558cc13b03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.929+0000 7f12d3e00640 1 -- 172.21.15.43:0/1523644026 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558cc13b4440 msgr2=0x558cc13c00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.929+0000 7f12d3e00640 1 --2- 172.21.15.43:0/1523644026 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558cc13b4440 0x558cc13c00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.929+0000 7f12d3e00640 1 -- 172.21.15.43:0/1523644026 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cc10233f0 con 0x558cc13abdb0 2024-03-28T22:38:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.929+0000 7f12da800640 1 --2- 172.21.15.43:0/1523644026 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558cc13b4440 0x558cc13c00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:38:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.930+0000 7f12d3e00640 1 --2- 172.21.15.43:0/1523644026 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558cc13abdb0 0x558cc13b3a70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f12dc108bb0 tx=0x7f12c4013830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe80a4abe57774e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.930+0000 7f12ca600640 1 -- 172.21.15.43:0/1523644026 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c401dc50 con 0x558cc13abdb0 2024-03-28T22:38:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.930+0000 7f12ca600640 1 -- 172.21.15.43:0/1523644026 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12c401ddf0 con 0x558cc13abdb0 2024-03-28T22:38:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.930+0000 7f12ca600640 1 -- 172.21.15.43:0/1523644026 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c4004a00 con 0x558cc13abdb0 2024-03-28T22:38:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.931+0000 7f12e3794740 1 -- 172.21.15.43:0/1523644026 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558cc13abdb0 msgr2=0x558cc13b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.931+0000 7f12e3794740 1 --2- 172.21.15.43:0/1523644026 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558cc13abdb0 0x558cc13b3a70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f12dc108bb0 tx=0x7f12c4013830 comp rx=0 tx=0).stop 2024-03-28T22:38:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.931+0000 7f12e3794740 1 -- 172.21.15.43:0/1523644026 shutdown_connections 2024-03-28T22:38:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.931+0000 7f12e3794740 1 --2- 172.21.15.43:0/1523644026 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558cc13b4440 0x558cc13c00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.931+0000 7f12e3794740 1 --2- 172.21.15.43:0/1523644026 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558cc13abdb0 0x558cc13b3a70 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.931+0000 7f12e3794740 1 --2- 172.21.15.43:0/1523644026 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558cc13ab0d0 0x558cc13b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.931+0000 7f12e3794740 1 -- 172.21.15.43:0/1523644026 >> 172.21.15.43:0/1523644026 conn(0x558cc1371a10 msgr2=0x558cc13a4d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.931+0000 7f12e3794740 1 -- 172.21.15.43:0/1523644026 shutdown_connections 2024-03-28T22:38:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.931+0000 7f12e3794740 1 -- 172.21.15.43:0/1523644026 wait complete. 2024-03-28T22:38:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.931+0000 7f12e3794740 1 -- start start 2024-03-28T22:38:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.931+0000 7f12e3794740 1 Processor -- start 2024-03-28T22:38:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.932+0000 7f12e3794740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558cc13ab0d0 0x558cc13aeae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.932+0000 7f12e3794740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558cc13abdb0 0x558cc13af020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.932+0000 7f12e3794740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558cc13b4440 0x558cc13b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.932+0000 7f12e3794740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f12dc1b2120 con 0x558cc13b4440 2024-03-28T22:38:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.932+0000 7f12e3794740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f12dc1b1a50 con 0x558cc13ab0d0 2024-03-28T22:38:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.932+0000 7f12e3794740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f12dc1aec90 con 0x558cc13abdb0 2024-03-28T22:38:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.932+0000 7f12e3794740 10 client.0 _ll_register_callbacks cb 0x7f12cc009720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.932+0000 7f12d3e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558cc13abdb0 0x558cc13af020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.932+0000 7f12d9e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558cc13ab0d0 0x558cc13aeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.932+0000 7f12d3e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558cc13abdb0 0x558cc13af020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35962/0 (socket says 172.21.15.43:35962) 2024-03-28T22:38:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.932+0000 7f12d3e00640 1 -- 172.21.15.43:0/2913223249 learned_addr learned my addr 172.21.15.43:0/2913223249 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.932+0000 7f12da800640 1 --2- 172.21.15.43:0/2913223249 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558cc13b4440 0x558cc13b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.933+0000 7f12d3e00640 1 -- 172.21.15.43:0/2913223249 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558cc13ab0d0 msgr2=0x558cc13aeae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.933+0000 7f12d3e00640 1 --2- 172.21.15.43:0/2913223249 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558cc13ab0d0 0x558cc13aeae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.933+0000 7f12d3e00640 1 -- 172.21.15.43:0/2913223249 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558cc13b4440 msgr2=0x558cc13b5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.933+0000 7f12d3e00640 1 --2- 172.21.15.43:0/2913223249 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558cc13b4440 0x558cc13b5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.933+0000 7f12d3e00640 1 -- 172.21.15.43:0/2913223249 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cc0fe30f0 con 0x558cc13abdb0 2024-03-28T22:38:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.933+0000 7f12da800640 1 --2- 172.21.15.43:0/2913223249 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558cc13b4440 0x558cc13b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:38:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.933+0000 7f12d9e00640 1 --2- 172.21.15.43:0/2913223249 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558cc13ab0d0 0x558cc13aeae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.933+0000 7f12d3e00640 1 --2- 172.21.15.43:0/2913223249 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558cc13abdb0 0x558cc13af020 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f12c400bbd0 tx=0x7f12c401dcd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dea296b48e21a42e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.933+0000 7f12bf400640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:38:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.933+0000 7f12bf400640 1 -- 172.21.15.43:0/2913223249 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c401d630 con 0x558cc13abdb0 2024-03-28T22:38:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.933+0000 7f12e3794740 1 -- 172.21.15.43:0/2913223249 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12c401ddf0 con 0x558cc13abdb0 2024-03-28T22:38:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.933+0000 7f12bf400640 1 -- 172.21.15.43:0/2913223249 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f12c4011440 con 0x558cc13abdb0 2024-03-28T22:38:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.933+0000 7f12bf400640 1 -- 172.21.15.43:0/2913223249 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f12c4037050 con 0x558cc13abdb0 2024-03-28T22:38:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.933+0000 7f12e3794740 10 client.26104 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.933+0000 7f12e3794740 1 -- 172.21.15.43:0/2913223249 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558cc142b100 con 0x558cc13abdb0 2024-03-28T22:38:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.934+0000 7f12bf400640 1 -- 172.21.15.43:0/2913223249 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12c401ddf0 con 0x558cc13abdb0 2024-03-28T22:38:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.934+0000 7f12bf400640 1 -- 172.21.15.43:0/2913223249 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f12c40265d0 con 0x558cc13abdb0 2024-03-28T22:38:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.934+0000 7f12e3794740 10 client.26104 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.934+0000 7f12e3794740 10 client.26104 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:38:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.934+0000 7f12e3794740 20 client.26104 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.934+0000 7f12e3794740 1 --2- 172.21.15.43:0/2913223249 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x558cc142b820 0x558cc142dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.934+0000 7f12e3794740 4 client.26104 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:38:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.934+0000 7f12e3794740 1 -- 172.21.15.43:0/2913223249 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558cc1316990 con 0x558cc142b820 2024-03-28T22:38:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.934+0000 7f12d9e00640 1 --2- 172.21.15.43:0/2913223249 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x558cc142b820 0x558cc142dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.935+0000 7f12d9e00640 1 --2- 172.21.15.43:0/2913223249 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x558cc142b820 0x558cc142dc40 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e537f4fbe49805b server_cookie=733dccd56fb034ab in_seq=0 out_seq=0 2024-03-28T22:38:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.935+0000 7f12bf400640 10 client.26104 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.936+0000 7f12bf400640 1 -- 172.21.15.43:0/2913223249 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558cc1316990 con 0x558cc142b820 2024-03-28T22:38:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.936+0000 7f12bf400640 10 client.26104 handle_command_reply: tid=0 2024-03-28T22:38:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.937+0000 7f12e3794740 1 client.26104 shutdown 2024-03-28T22:38:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.937+0000 7f12e3794740 1 -- 172.21.15.43:0/2913223249 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558cc13abdb0 msgr2=0x558cc13af020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.937+0000 7f12e3794740 1 --2- 172.21.15.43:0/2913223249 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558cc13abdb0 0x558cc13af020 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f12c400bbd0 tx=0x7f12c401dcd0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.937+0000 7f12e3794740 1 -- 172.21.15.43:0/2913223249 shutdown_connections 2024-03-28T22:38:38.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.937+0000 7f12e3794740 1 --2- 172.21.15.43:0/2913223249 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558cc13b4440 0x558cc13b5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.937+0000 7f12e3794740 1 --2- 172.21.15.43:0/2913223249 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558cc13abdb0 0x558cc13af020 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.937+0000 7f12e3794740 1 --2- 172.21.15.43:0/2913223249 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x558cc142b820 0x558cc142dc40 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.937+0000 7f12e3794740 1 --2- 172.21.15.43:0/2913223249 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558cc13ab0d0 0x558cc13aeae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.937+0000 7f12bf400640 0 client.26104 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:38.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.937+0000 7f12e3794740 1 -- 172.21.15.43:0/2913223249 >> 172.21.15.43:0/2913223249 conn(0x558cc1371a10 msgr2=0x558cc13a4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:38.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.937+0000 7f12e3794740 1 -- 172.21.15.43:0/2913223249 shutdown_connections 2024-03-28T22:38:38.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.937+0000 7f12e3794740 1 -- 172.21.15.43:0/2913223249 wait complete. 2024-03-28T22:38:38.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.937+0000 7f12e3794740 20 client.26104 trim_cache size 0 max 16384 2024-03-28T22:38:38.942 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:38:38.942 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-03-28T22:38:38.942 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-03-28T22:38:38.942 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:38:38.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.938+0000 7f12db200640 1 -- 172.21.15.43:0/2615492418 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f12c00a2090 msgr2=0x7f12c00a4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:38.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.938+0000 7f12db200640 1 --2- 172.21.15.43:0/2615492418 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f12c00a2090 0x7f12c00a4550 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f12d4046000 tx=0x7f12d404b000 comp rx=0 tx=0).stop 2024-03-28T22:38:38.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.938+0000 7f12db200640 1 -- 172.21.15.43:0/2615492418 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12dc1037d0 msgr2=0x7f12dc19a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:38.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.938+0000 7f12db200640 1 --2- 172.21.15.43:0/2615492418 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12dc1037d0 0x7f12dc19a110 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f12cc00e9f0 tx=0x7f12cc00edf0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.938+0000 7f12db200640 1 -- 172.21.15.43:0/2615492418 shutdown_connections 2024-03-28T22:38:38.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.938+0000 7f12db200640 1 --2- 172.21.15.43:0/2615492418 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f12c00a2090 0x7f12c00a4550 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.938+0000 7f12db200640 1 --2- 172.21.15.43:0/2615492418 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f12dc1097e0 0x7f12dc19e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.938+0000 7f12db200640 1 --2- 172.21.15.43:0/2615492418 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12dc104190 0x7f12dc19a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.938+0000 7f12db200640 1 --2- 172.21.15.43:0/2615492418 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12dc1037d0 0x7f12dc19a110 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:38.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.938+0000 7f12db200640 1 -- 172.21.15.43:0/2615492418 >> 172.21.15.43:0/2615492418 conn(0x7f12dc0ff280 msgr2=0x7f12dc0ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:38.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.938+0000 7f12db200640 1 -- 172.21.15.43:0/2615492418 shutdown_connections 2024-03-28T22:38:38.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:38.938+0000 7f12db200640 1 -- 172.21.15.43:0/2615492418 wait complete. 2024-03-28T22:38:38.954 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:38:38.954 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:38:39.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.158+0000 7f3979600640 1 -- 172.21.15.43:0/571488347 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f39741010c0 msgr2=0x7f397410d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:39.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.158+0000 7f3979600640 1 --2- 172.21.15.43:0/571488347 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f39741010c0 0x7f397410d320 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f396c009be0 tx=0x7f396c02bef0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.159+0000 7f3979600640 1 -- 172.21.15.43:0/571488347 shutdown_connections 2024-03-28T22:38:39.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.159+0000 7f3979600640 1 --2- 172.21.15.43:0/571488347 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f39741010c0 0x7f397410d320 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.159+0000 7f3979600640 1 --2- 172.21.15.43:0/571488347 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3974100700 0x7f3974100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.159+0000 7f3979600640 1 --2- 172.21.15.43:0/571488347 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f39741064f0 0x7f39741068d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.159+0000 7f3979600640 1 -- 172.21.15.43:0/571488347 >> 172.21.15.43:0/571488347 conn(0x7f3974076060 msgr2=0x7f3974076470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:39.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.159+0000 7f3979600640 1 -- 172.21.15.43:0/571488347 shutdown_connections 2024-03-28T22:38:39.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.159+0000 7f3979600640 1 -- 172.21.15.43:0/571488347 wait complete. 2024-03-28T22:38:39.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.160+0000 7f3979600640 1 Processor -- start 2024-03-28T22:38:39.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.160+0000 7f3979600640 1 -- start start 2024-03-28T22:38:39.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.161+0000 7f3979600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3974100700 0x7f397410ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.161+0000 7f3979600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f39741010c0 0x7f397410e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.161+0000 7f3979600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f39741064f0 0x7f397410e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.161+0000 7f3979600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3974111980 con 0x7f39741064f0 2024-03-28T22:38:39.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.161+0000 7f3979600640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f3974111800 con 0x7f39741010c0 2024-03-28T22:38:39.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.161+0000 7f3979600640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f3974111b00 con 0x7f3974100700 2024-03-28T22:38:39.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.161+0000 7f3973e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3974100700 0x7f397410ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.161+0000 7f3973e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3974100700 0x7f397410ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35968/0 (socket says 172.21.15.43:35968) 2024-03-28T22:38:39.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.161+0000 7f3973400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f39741010c0 0x7f397410e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.161+0000 7f3973e00640 1 -- 172.21.15.43:0/4267409592 learned_addr learned my addr 172.21.15.43:0/4267409592 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:39.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.161+0000 7f3978c00640 1 --2- 172.21.15.43:0/4267409592 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f39741064f0 0x7f397410e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.161+0000 7f3973e00640 1 -- 172.21.15.43:0/4267409592 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f39741010c0 msgr2=0x7f397410e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:39.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.161+0000 7f3973e00640 1 --2- 172.21.15.43:0/4267409592 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f39741010c0 0x7f397410e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.161+0000 7f3973e00640 1 -- 172.21.15.43:0/4267409592 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f39741064f0 msgr2=0x7f397410e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:39.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.161+0000 7f3973e00640 1 --2- 172.21.15.43:0/4267409592 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f39741064f0 0x7f397410e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.161+0000 7f3973e00640 1 -- 172.21.15.43:0/4267409592 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39741a6280 con 0x7f3974100700 2024-03-28T22:38:39.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.161+0000 7f3978c00640 1 --2- 172.21.15.43:0/4267409592 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f39741064f0 0x7f397410e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:38:39.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.162+0000 7f3973e00640 1 --2- 172.21.15.43:0/4267409592 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3974100700 0x7f397410ddb0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f396400d950 tx=0x7f396400de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:39.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.162+0000 7f3970c00640 1 -- 172.21.15.43:0/4267409592 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3964017030 con 0x7f3974100700 2024-03-28T22:38:39.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.162+0000 7f3970c00640 1 -- 172.21.15.43:0/4267409592 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3964004020 con 0x7f3974100700 2024-03-28T22:38:39.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.162+0000 7f3979600640 1 -- 172.21.15.43:0/4267409592 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39741a6570 con 0x7f3974100700 2024-03-28T22:38:39.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.162+0000 7f3970c00640 1 -- 172.21.15.43:0/4267409592 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39640047a0 con 0x7f3974100700 2024-03-28T22:38:39.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.162+0000 7f3979600640 1 -- 172.21.15.43:0/4267409592 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39741a6ac0 con 0x7f3974100700 2024-03-28T22:38:39.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.163+0000 7f3979600640 1 -- 172.21.15.43:0/4267409592 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3974068460 con 0x7f3974100700 2024-03-28T22:38:39.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.166+0000 7f3970c00640 1 -- 172.21.15.43:0/4267409592 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 27) ==== 136989+0+0 (secure 0 0 0) 0x7f39640041d0 con 0x7f3974100700 2024-03-28T22:38:39.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.167+0000 7f3970c00640 1 --2- 172.21.15.43:0/4267409592 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f39580a2010 0x7f39580a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.167+0000 7f3970c00640 1 -- 172.21.15.43:0/4267409592 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f39640ce810 con 0x7f3974100700 2024-03-28T22:38:39.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.168+0000 7f3973400640 1 --2- 172.21.15.43:0/4267409592 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f39580a2010 0x7f39580a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.168+0000 7f3973400640 1 --2- 172.21.15.43:0/4267409592 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f39580a2010 0x7f39580a44d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f397410f840 tx=0x7f395c009260 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:39.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.172+0000 7f3970c00640 1 -- 172.21.15.43:0/4267409592 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f396409b310 con 0x7f3974100700 2024-03-28T22:38:39.339 INFO:tasks.workunit.client.0.smithi043.stdout: 1 18141 11.65 MB/sec warmup 10 sec latency 13.988 ms 2024-03-28T22:38:39.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:39 smithi172 ceph-mon[36566]: pgmap v46: 129 pgs: 129 active+clean; 8.4 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 726 KiB/s wr, 198 op/s 2024-03-28T22:38:39.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.425+0000 7f3979600640 1 -- 172.21.15.43:0/4267409592 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f397410f2e0 con 0x7f3974100700 2024-03-28T22:38:39.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.426+0000 7f3970c00640 1 -- 172.21.15.43:0/4267409592 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11934 (secure 0 0 0) 0x7f397410f2e0 con 0x7f3974100700 2024-03-28T22:38:39.424 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:38:39.425 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:33:28.906396+0000","modified":"2024-03-28T22:36:49.597670+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24572,"mds_1":24556,"mds_2":24535,"mds_3":24523,"mds_4":24518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/1084988072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1084988072},{"type":"v1","addr":"172.21.15.82:6835","nonce":1084988072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572,24523,24518,24535,24556]},"id":1}]} 2024-03-28T22:38:39.425 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 29 2024-03-28T22:38:39.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.429+0000 7f3979600640 1 -- 172.21.15.43:0/4267409592 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f39580a2010 msgr2=0x7f39580a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:39.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.429+0000 7f3979600640 1 --2- 172.21.15.43:0/4267409592 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f39580a2010 0x7f39580a44d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f397410f840 tx=0x7f395c009260 comp rx=0 tx=0).stop 2024-03-28T22:38:39.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.429+0000 7f3979600640 1 -- 172.21.15.43:0/4267409592 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3974100700 msgr2=0x7f397410ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:39.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.429+0000 7f3979600640 1 --2- 172.21.15.43:0/4267409592 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3974100700 0x7f397410ddb0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f396400d950 tx=0x7f396400de10 comp rx=0 tx=0).stop 2024-03-28T22:38:39.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.429+0000 7f3979600640 1 -- 172.21.15.43:0/4267409592 shutdown_connections 2024-03-28T22:38:39.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.429+0000 7f3979600640 1 --2- 172.21.15.43:0/4267409592 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f39580a2010 0x7f39580a44d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.429+0000 7f3979600640 1 --2- 172.21.15.43:0/4267409592 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f39741010c0 0x7f397410e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.429+0000 7f3979600640 1 --2- 172.21.15.43:0/4267409592 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f39741064f0 0x7f397410e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.429+0000 7f3979600640 1 --2- 172.21.15.43:0/4267409592 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3974100700 0x7f397410ddb0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.429+0000 7f3979600640 1 -- 172.21.15.43:0/4267409592 >> 172.21.15.43:0/4267409592 conn(0x7f3974076060 msgr2=0x7f39740fe6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:39.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.430+0000 7f3979600640 1 -- 172.21.15.43:0/4267409592 shutdown_connections 2024-03-28T22:38:39.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.430+0000 7f3979600640 1 -- 172.21.15.43:0/4267409592 wait complete. 2024-03-28T22:38:39.443 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:39 smithi043 ceph-mon[31823]: pgmap v46: 129 pgs: 129 active+clean; 8.4 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 726 KiB/s wr, 198 op/s 2024-03-28T22:38:39.448 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T22:38:39.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:39 smithi082 ceph-mon[36471]: pgmap v46: 129 pgs: 129 active+clean; 8.4 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 726 KiB/s wr, 198 op/s 2024-03-28T22:38:39.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.702+0000 7fd802c00640 1 -- 172.21.15.43:0/3427120748 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7fc1076f0 msgr2=0x7fd7fc107ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:39.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.702+0000 7fd802c00640 1 --2- 172.21.15.43:0/3427120748 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7fc1076f0 0x7fd7fc107ad0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec009930 tx=0x7fd7ec02bc20 comp rx=0 tx=0).stop 2024-03-28T22:38:39.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.703+0000 7fd802c00640 1 -- 172.21.15.43:0/3427120748 shutdown_connections 2024-03-28T22:38:39.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.703+0000 7fd802c00640 1 --2- 172.21.15.43:0/3427120748 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7fc1016e0 0x7fd7fc101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.703+0000 7fd802c00640 1 --2- 172.21.15.43:0/3427120748 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7fc1020a0 0x7fd7fc10e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.703+0000 7fd802c00640 1 --2- 172.21.15.43:0/3427120748 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7fc1076f0 0x7fd7fc107ad0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.703+0000 7fd802c00640 1 -- 172.21.15.43:0/3427120748 >> 172.21.15.43:0/3427120748 conn(0x7fd7fc078b50 msgr2=0x7fd7fc078f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:39.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.703+0000 7fd802c00640 1 -- 172.21.15.43:0/3427120748 shutdown_connections 2024-03-28T22:38:39.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.703+0000 7fd802c00640 1 -- 172.21.15.43:0/3427120748 wait complete. 2024-03-28T22:38:39.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.704+0000 7fd802c00640 1 Processor -- start 2024-03-28T22:38:39.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.704+0000 7fd802c00640 1 -- start start 2024-03-28T22:38:39.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.704+0000 7fd802c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7fc1016e0 0x7fd7fc073a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.704+0000 7fd802c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7fc1020a0 0x7fd7fc073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.704+0000 7fd802c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7fc1076f0 0x7fd7fc0744d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.704+0000 7fd802c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd7fc111b50 con 0x7fd7fc1020a0 2024-03-28T22:38:39.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.704+0000 7fd802c00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd7fc1119d0 con 0x7fd7fc1016e0 2024-03-28T22:38:39.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.704+0000 7fd802c00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd7fc111cd0 con 0x7fd7fc1076f0 2024-03-28T22:38:39.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.704+0000 7fd7fbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7fc1016e0 0x7fd7fc073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.705+0000 7fd7fbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7fc1016e0 0x7fd7fc073a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:40430/0 (socket says 172.21.15.43:40430) 2024-03-28T22:38:39.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.705+0000 7fd7fbe00640 1 -- 172.21.15.43:0/3183886391 learned_addr learned my addr 172.21.15.43:0/3183886391 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:39.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.705+0000 7fd7fb400640 1 --2- 172.21.15.43:0/3183886391 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7fc1020a0 0x7fd7fc073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.705+0000 7fd800c00640 1 --2- 172.21.15.43:0/3183886391 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7fc1076f0 0x7fd7fc0744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.705+0000 7fd7fbe00640 1 -- 172.21.15.43:0/3183886391 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7fc1076f0 msgr2=0x7fd7fc0744d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:39.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.705+0000 7fd7fbe00640 1 --2- 172.21.15.43:0/3183886391 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7fc1076f0 0x7fd7fc0744d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.705+0000 7fd7fbe00640 1 -- 172.21.15.43:0/3183886391 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7fc1020a0 msgr2=0x7fd7fc073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:39.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.705+0000 7fd7fbe00640 1 --2- 172.21.15.43:0/3183886391 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7fc1020a0 0x7fd7fc073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.705+0000 7fd7fbe00640 1 -- 172.21.15.43:0/3183886391 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7fc1a64d0 con 0x7fd7fc1016e0 2024-03-28T22:38:39.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.705+0000 7fd7fbe00640 1 --2- 172.21.15.43:0/3183886391 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7fc1016e0 0x7fd7fc073a50 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec005ab0 tx=0x7fd7ec005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:39.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.706+0000 7fd7f8c00640 1 -- 172.21.15.43:0/3183886391 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec043020 con 0x7fd7fc1016e0 2024-03-28T22:38:39.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.706+0000 7fd802c00640 1 -- 172.21.15.43:0/3183886391 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7fc1a6760 con 0x7fd7fc1016e0 2024-03-28T22:38:39.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.706+0000 7fd7f8c00640 1 -- 172.21.15.43:0/3183886391 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7ec005000 con 0x7fd7fc1016e0 2024-03-28T22:38:39.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.706+0000 7fd7f8c00640 1 -- 172.21.15.43:0/3183886391 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec003a80 con 0x7fd7fc1016e0 2024-03-28T22:38:39.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.706+0000 7fd802c00640 1 -- 172.21.15.43:0/3183886391 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7fc1a6d00 con 0x7fd7fc1016e0 2024-03-28T22:38:39.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.709+0000 7fd805982740 1 Processor -- start 2024-03-28T22:38:39.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.709+0000 7fd7f8c00640 1 -- 172.21.15.43:0/3183886391 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 27) ==== 136989+0+0 (secure 0 0 0) 0x7fd7ec046020 con 0x7fd7fc1016e0 2024-03-28T22:38:39.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.709+0000 7fd805982740 1 -- start start 2024-03-28T22:38:39.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.709+0000 7fd805982740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a00f9af380 0x55a00f9af760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.710+0000 7fd805982740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a00f9afca0 0x55a00f9b8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.710+0000 7fd805982740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a00f9b8880 0x55a00f9bac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.710+0000 7fd805982740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55a00f913050 con 0x55a00f9af380 2024-03-28T22:38:39.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.710+0000 7fd805982740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55a00f8bbce0 con 0x55a00f9afca0 2024-03-28T22:38:39.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.710+0000 7fd805982740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55a00f90f0f0 con 0x55a00f9b8880 2024-03-28T22:38:39.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.710+0000 7fd800c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a00f9af380 0x55a00f9af760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.710+0000 7fd7fbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a00f9afca0 0x55a00f9b8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.710+0000 7fd800c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a00f9af380 0x55a00f9af760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35580/0 (socket says 172.21.15.43:35580) 2024-03-28T22:38:39.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.710+0000 7fd800c00640 1 -- 172.21.15.43:0/2409746765 learned_addr learned my addr 172.21.15.43:0/2409746765 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:39.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.710+0000 7fd7fb400640 1 --2- 172.21.15.43:0/2409746765 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a00f9b8880 0x55a00f9bac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.710+0000 7fd800c00640 1 -- 172.21.15.43:0/2409746765 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a00f9b8880 msgr2=0x55a00f9bac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:39.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.710+0000 7fd800c00640 1 --2- 172.21.15.43:0/2409746765 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a00f9b8880 0x55a00f9bac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.710+0000 7fd800c00640 1 -- 172.21.15.43:0/2409746765 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a00f9afca0 msgr2=0x55a00f9b8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:39.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.710+0000 7fd800c00640 1 --2- 172.21.15.43:0/2409746765 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a00f9afca0 0x55a00f9b8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.710+0000 7fd800c00640 1 -- 172.21.15.43:0/2409746765 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a00f6273f0 con 0x55a00f9af380 2024-03-28T22:38:39.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.711+0000 7fd800c00640 1 --2- 172.21.15.43:0/2409746765 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a00f9af380 0x55a00f9af760 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd7f4009f50 tx=0x7fd7f40118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc6c4c186241570c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:39.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.711+0000 7fd7f1200640 1 -- 172.21.15.43:0/2409746765 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f4004d20 con 0x55a00f9af380 2024-03-28T22:38:39.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.711+0000 7fd7f8c00640 1 --2- 172.21.15.43:0/3183886391 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd7e00a2010 0x7fd7e00a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.711+0000 7fd7f1200640 1 -- 172.21.15.43:0/2409746765 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7f4004050 con 0x55a00f9af380 2024-03-28T22:38:39.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.711+0000 7fd7f1200640 1 -- 172.21.15.43:0/2409746765 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f4005280 con 0x55a00f9af380 2024-03-28T22:38:39.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.711+0000 7fd7f8c00640 1 -- 172.21.15.43:0/3183886391 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fd7ec0f0710 con 0x7fd7fc1016e0 2024-03-28T22:38:39.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.711+0000 7fd7fbe00640 1 --2- 172.21.15.43:0/3183886391 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd7e00a2010 0x7fd7e00a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.712+0000 7fd805982740 1 -- 172.21.15.43:0/2409746765 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a00f9af380 msgr2=0x55a00f9af760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:39.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.712+0000 7fd805982740 1 --2- 172.21.15.43:0/2409746765 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a00f9af380 0x55a00f9af760 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd7f4009f50 tx=0x7fd7f40118d0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.712+0000 7fd805982740 1 -- 172.21.15.43:0/2409746765 shutdown_connections 2024-03-28T22:38:39.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.712+0000 7fd805982740 1 --2- 172.21.15.43:0/2409746765 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a00f9afca0 0x55a00f9b8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.712+0000 7fd805982740 1 --2- 172.21.15.43:0/2409746765 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a00f9b8880 0x55a00f9bac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.712+0000 7fd805982740 1 --2- 172.21.15.43:0/2409746765 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a00f9af380 0x55a00f9af760 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.712+0000 7fd805982740 1 -- 172.21.15.43:0/2409746765 >> 172.21.15.43:0/2409746765 conn(0x55a00f9a9ab0 msgr2=0x55a00f9a9e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:39.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.712+0000 7fd805982740 1 -- 172.21.15.43:0/2409746765 shutdown_connections 2024-03-28T22:38:39.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.712+0000 7fd7fbe00640 1 --2- 172.21.15.43:0/3183886391 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd7e00a2010 0x7fd7e00a44d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec0eda50 tx=0x7fd7ec0f6530 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:39.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.712+0000 7fd805982740 1 -- 172.21.15.43:0/2409746765 wait complete. 2024-03-28T22:38:39.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.712+0000 7fd805982740 1 -- start start 2024-03-28T22:38:39.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.713+0000 7fd805982740 1 Processor -- start 2024-03-28T22:38:39.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.713+0000 7fd805982740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a00f9af380 0x55a00f9b5920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.713+0000 7fd805982740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a00f9afca0 0x55a00f9b5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.713+0000 7fd805982740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a00f9b8880 0x55a00f9b63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.713+0000 7fd805982740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55a00f9bb670 con 0x55a00f9afca0 2024-03-28T22:38:39.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.713+0000 7fd805982740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55a00f8b12f0 con 0x55a00f9b8880 2024-03-28T22:38:39.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.713+0000 7fd805982740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55a00f9bb7f0 con 0x55a00f9af380 2024-03-28T22:38:39.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.713+0000 7fd805982740 10 client.0 _ll_register_callbacks cb 0x55a00f626be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:39.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.714+0000 7fd800c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a00f9af380 0x55a00f9b5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.714+0000 7fd7fb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a00f9afca0 0x55a00f9b5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.714+0000 7fd800c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a00f9af380 0x55a00f9b5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35988/0 (socket says 172.21.15.43:35988) 2024-03-28T22:38:39.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.714+0000 7fd7fb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a00f9afca0 0x55a00f9b5e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35586/0 (socket says 172.21.15.43:35586) 2024-03-28T22:38:39.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.714+0000 7fd800c00640 1 -- 172.21.15.43:0/2668532085 learned_addr learned my addr 172.21.15.43:0/2668532085 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.714+0000 7fd800c00640 1 --2- 172.21.15.43:0/2668532085 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a00f9b8880 0x55a00f9b63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.714+0000 7fd7fb400640 1 -- 172.21.15.43:0/2668532085 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a00f9af380 msgr2=0x55a00f9b5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.714+0000 7fd7fb400640 1 --2- 172.21.15.43:0/2668532085 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a00f9af380 0x55a00f9b5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.714+0000 7fd7fb400640 1 -- 172.21.15.43:0/2668532085 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a00f9b8880 msgr2=0x55a00f9b63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.714+0000 7fd7fb400640 1 --2- 172.21.15.43:0/2668532085 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a00f9b8880 0x55a00f9b63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.714+0000 7fd7fb400640 1 -- 172.21.15.43:0/2668532085 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a00f6f17d0 con 0x55a00f9afca0 2024-03-28T22:38:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.714+0000 7fd800c00640 1 --2- 172.21.15.43:0/2668532085 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a00f9b8880 0x55a00f9b63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:38:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.714+0000 7fd7fb400640 1 --2- 172.21.15.43:0/2668532085 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a00f9afca0 0x55a00f9b5e60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc0754e0 tx=0x7fd7e8012b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fecc9ba65acf0920 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.714+0000 7fd7e5600640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:38:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.715+0000 7fd7e5600640 1 -- 172.21.15.43:0/2668532085 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e800f920 con 0x55a00f9afca0 2024-03-28T22:38:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.715+0000 7fd7e5600640 1 -- 172.21.15.43:0/2668532085 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fd7e800fac0 con 0x55a00f9afca0 2024-03-28T22:38:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.715+0000 7fd805982740 1 -- 172.21.15.43:0/2668532085 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd7f4004050 con 0x55a00f9afca0 2024-03-28T22:38:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.715+0000 7fd7e5600640 1 -- 172.21.15.43:0/2668532085 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd7e800fcc0 con 0x55a00f9afca0 2024-03-28T22:38:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.715+0000 7fd805982740 10 client.15999 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.715+0000 7fd805982740 1 -- 172.21.15.43:0/2668532085 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a00f9c79d0 con 0x55a00f9afca0 2024-03-28T22:38:39.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.716+0000 7fd7e5600640 1 -- 172.21.15.43:0/2668532085 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7f4004050 con 0x55a00f9afca0 2024-03-28T22:38:39.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.716+0000 7fd7e5600640 1 -- 172.21.15.43:0/2668532085 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd7e8010430 con 0x55a00f9afca0 2024-03-28T22:38:39.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.717+0000 7fd805982740 10 client.15999 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:39.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.717+0000 7fd805982740 10 client.15999 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:38:39.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.717+0000 7fd805982740 20 client.15999 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:39.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.717+0000 7fd805982740 1 --2- 172.21.15.43:0/2668532085 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55a00f9c8ee0 0x55a00f9cb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.717+0000 7fd805982740 4 client.15999 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:38:39.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.717+0000 7fd805982740 1 -- 172.21.15.43:0/2668532085 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a00f8bc680 con 0x55a00f9c8ee0 2024-03-28T22:38:39.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.717+0000 7fd800c00640 1 --2- 172.21.15.43:0/2668532085 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55a00f9c8ee0 0x55a00f9cb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.717+0000 7fd800c00640 1 --2- 172.21.15.43:0/2668532085 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55a00f9c8ee0 0x55a00f9cb2d0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=bd58830786a58614 server_cookie=c10e7882b5dcf3bd in_seq=0 out_seq=0 2024-03-28T22:38:39.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.717+0000 7fd7e5600640 10 client.15999 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:38:39.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.718+0000 7fd7e5600640 1 -- 172.21.15.43:0/2668532085 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a00f8bc680 con 0x55a00f9c8ee0 2024-03-28T22:38:39.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.718+0000 7fd7e5600640 10 client.15999 handle_command_reply: tid=0 2024-03-28T22:38:39.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.718+0000 7fd805982740 1 client.15999 shutdown 2024-03-28T22:38:39.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.719+0000 7fd805982740 1 -- 172.21.15.43:0/2668532085 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a00f9afca0 msgr2=0x55a00f9b5e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:39.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.719+0000 7fd805982740 1 --2- 172.21.15.43:0/2668532085 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a00f9afca0 0x55a00f9b5e60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc0754e0 tx=0x7fd7e8012b60 comp rx=0 tx=0).stop 2024-03-28T22:38:39.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.719+0000 7fd805982740 1 -- 172.21.15.43:0/2668532085 shutdown_connections 2024-03-28T22:38:39.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.719+0000 7fd805982740 1 --2- 172.21.15.43:0/2668532085 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55a00f9c8ee0 0x55a00f9cb2d0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.719+0000 7fd805982740 1 --2- 172.21.15.43:0/2668532085 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a00f9b8880 0x55a00f9b63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.719+0000 7fd805982740 1 --2- 172.21.15.43:0/2668532085 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a00f9afca0 0x55a00f9b5e60 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.719+0000 7fd805982740 1 --2- 172.21.15.43:0/2668532085 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a00f9af380 0x55a00f9b5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.719+0000 7fd805982740 1 -- 172.21.15.43:0/2668532085 >> 172.21.15.43:0/2668532085 conn(0x55a00f9a9ab0 msgr2=0x55a00f9ab800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:39.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.719+0000 7fd7e5600640 0 client.15999 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:38:39.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.719+0000 7fd805982740 1 -- 172.21.15.43:0/2668532085 shutdown_connections 2024-03-28T22:38:39.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.719+0000 7fd805982740 1 -- 172.21.15.43:0/2668532085 wait complete. 2024-03-28T22:38:39.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.719+0000 7fd805982740 20 client.15999 trim_cache size 0 max 16384 2024-03-28T22:38:39.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.734+0000 7fd805982740 1 Processor -- start 2024-03-28T22:38:39.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.734+0000 7fd805982740 1 -- start start 2024-03-28T22:38:39.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.735+0000 7fd805982740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a00f9af380 0x55a00f9b8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.735+0000 7fd805982740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a00f9afca0 0x55a00f9c4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.735+0000 7fd805982740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a00f9b8880 0x55a00f9b4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.735+0000 7fd805982740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd7fc1b0de0 con 0x55a00f9af380 2024-03-28T22:38:39.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.735+0000 7fd805982740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd7fc1b2920 con 0x55a00f9afca0 2024-03-28T22:38:39.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.735+0000 7fd805982740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd7fc1b14b0 con 0x55a00f9b8880 2024-03-28T22:38:39.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.735+0000 7fd800c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a00f9af380 0x55a00f9b8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.735+0000 7fd7fb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a00f9afca0 0x55a00f9c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.735+0000 7fd800c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a00f9af380 0x55a00f9b8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35598/0 (socket says 172.21.15.43:35598) 2024-03-28T22:38:39.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.735+0000 7fd7fb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a00f9afca0 0x55a00f9c4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:40452/0 (socket says 172.21.15.43:40452) 2024-03-28T22:38:39.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.735+0000 7fd800c00640 1 -- 172.21.15.43:0/1304366168 learned_addr learned my addr 172.21.15.43:0/1304366168 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:39.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.735+0000 7fd800c00640 1 --2- 172.21.15.43:0/1304366168 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a00f9b8880 0x55a00f9b4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.735+0000 7fd7fb400640 1 -- 172.21.15.43:0/1304366168 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a00f9b8880 msgr2=0x55a00f9b4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:39.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.735+0000 7fd7fb400640 1 --2- 172.21.15.43:0/1304366168 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a00f9b8880 0x55a00f9b4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.735+0000 7fd7fb400640 1 -- 172.21.15.43:0/1304366168 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a00f9af380 msgr2=0x55a00f9b8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:39.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.735+0000 7fd7fb400640 1 --2- 172.21.15.43:0/1304366168 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a00f9af380 0x55a00f9b8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.735+0000 7fd7fb400640 1 -- 172.21.15.43:0/1304366168 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a00f9a5370 con 0x55a00f9afca0 2024-03-28T22:38:39.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.735+0000 7fd800c00640 1 --2- 172.21.15.43:0/1304366168 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a00f9af380 0x55a00f9b8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:39.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.736+0000 7fd7fb400640 1 --2- 172.21.15.43:0/1304366168 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a00f9afca0 0x55a00f9c4410 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fd7e80059f0 tx=0x7fd7e8010b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7650e73019b2257b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:39.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.736+0000 7fd7e7e00640 1 -- 172.21.15.43:0/1304366168 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e800fe60 con 0x55a00f9afca0 2024-03-28T22:38:39.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.736+0000 7fd7e7e00640 1 -- 172.21.15.43:0/1304366168 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7e8005590 con 0x55a00f9afca0 2024-03-28T22:38:39.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.736+0000 7fd7e7e00640 1 -- 172.21.15.43:0/1304366168 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e8003850 con 0x55a00f9afca0 2024-03-28T22:38:39.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.737+0000 7fd805982740 1 -- 172.21.15.43:0/1304366168 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a00f9afca0 msgr2=0x55a00f9c4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:39.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.737+0000 7fd805982740 1 --2- 172.21.15.43:0/1304366168 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a00f9afca0 0x55a00f9c4410 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fd7e80059f0 tx=0x7fd7e8010b50 comp rx=0 tx=0).stop 2024-03-28T22:38:39.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.737+0000 7fd805982740 1 -- 172.21.15.43:0/1304366168 shutdown_connections 2024-03-28T22:38:39.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.737+0000 7fd805982740 1 --2- 172.21.15.43:0/1304366168 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a00f9afca0 0x55a00f9c4410 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.737+0000 7fd805982740 1 --2- 172.21.15.43:0/1304366168 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a00f9b8880 0x55a00f9b4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.737+0000 7fd805982740 1 --2- 172.21.15.43:0/1304366168 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a00f9af380 0x55a00f9b8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.737+0000 7fd805982740 1 -- 172.21.15.43:0/1304366168 >> 172.21.15.43:0/1304366168 conn(0x55a00f9a9ab0 msgr2=0x55a00f9a82e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:39.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.737+0000 7fd805982740 1 -- 172.21.15.43:0/1304366168 shutdown_connections 2024-03-28T22:38:39.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.737+0000 7fd805982740 1 -- 172.21.15.43:0/1304366168 wait complete. 2024-03-28T22:38:39.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.737+0000 7fd805982740 1 -- start start 2024-03-28T22:38:39.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.737+0000 7fd805982740 1 Processor -- start 2024-03-28T22:38:39.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.738+0000 7fd805982740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a00f9af380 0x55a00f9c0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.738+0000 7fd805982740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a00f9afca0 0x55a00f9bc000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.738+0000 7fd805982740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a00f9b8880 0x55a00f9bc540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.738+0000 7fd805982740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd7fc1b2250 con 0x55a00f9b8880 2024-03-28T22:38:39.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.738+0000 7fd805982740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd7fc1b1b80 con 0x55a00f9afca0 2024-03-28T22:38:39.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.738+0000 7fd805982740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd7fc1aedc0 con 0x55a00f9af380 2024-03-28T22:38:39.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.738+0000 7fd805982740 10 client.0 _ll_register_callbacks cb 0x7fd7e8009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:39.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.738+0000 7fd7fb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a00f9afca0 0x55a00f9bc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.738+0000 7fd7fb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a00f9afca0 0x55a00f9bc000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:40462/0 (socket says 172.21.15.43:40462) 2024-03-28T22:38:39.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.738+0000 7fd800c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a00f9af380 0x55a00f9c0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.738+0000 7fd7fb400640 1 -- 172.21.15.43:0/1364910727 learned_addr learned my addr 172.21.15.43:0/1364910727 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:39.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.738+0000 7fd800c00640 1 --2- 172.21.15.43:0/1364910727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a00f9b8880 0x55a00f9bc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.739+0000 7fd7fb400640 1 -- 172.21.15.43:0/1364910727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a00f9af380 msgr2=0x55a00f9c0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:39.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.739+0000 7fd7fb400640 1 --2- 172.21.15.43:0/1364910727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a00f9af380 0x55a00f9c0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.739+0000 7fd7fb400640 1 -- 172.21.15.43:0/1364910727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a00f9b8880 msgr2=0x55a00f9bc540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:39.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.739+0000 7fd7fb400640 1 --2- 172.21.15.43:0/1364910727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a00f9b8880 0x55a00f9bc540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.739+0000 7fd7fb400640 1 -- 172.21.15.43:0/1364910727 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a00f9bcc80 con 0x55a00f9afca0 2024-03-28T22:38:39.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.739+0000 7fd7fb400640 1 --2- 172.21.15.43:0/1364910727 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a00f9afca0 0x55a00f9bc000 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd7e8010920 tx=0x7fd7e8005250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f2ca909dd6a631d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:39.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.739+0000 7fd7e5600640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:38:39.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.739+0000 7fd7e5600640 1 -- 172.21.15.43:0/1364910727 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e8000e60 con 0x55a00f9afca0 2024-03-28T22:38:39.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.739+0000 7fd805982740 1 -- 172.21.15.43:0/1364910727 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd7e8005590 con 0x55a00f9afca0 2024-03-28T22:38:39.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.739+0000 7fd7e5600640 1 -- 172.21.15.43:0/1364910727 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fd7e8003800 con 0x55a00f9afca0 2024-03-28T22:38:39.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.740+0000 7fd7e5600640 1 -- 172.21.15.43:0/1364910727 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd7e800fe60 con 0x55a00f9afca0 2024-03-28T22:38:39.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.740+0000 7fd805982740 10 client.26180 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:39.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.740+0000 7fd805982740 1 -- 172.21.15.43:0/1364910727 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a00fa2f100 con 0x55a00f9afca0 2024-03-28T22:38:39.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.740+0000 7fd7e5600640 1 -- 172.21.15.43:0/1364910727 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7e8005590 con 0x55a00f9afca0 2024-03-28T22:38:39.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.740+0000 7fd7e5600640 1 -- 172.21.15.43:0/1364910727 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd7e802a6e0 con 0x55a00f9afca0 2024-03-28T22:38:39.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.740+0000 7fd805982740 10 client.26180 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:39.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.740+0000 7fd805982740 10 client.26180 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:38:39.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.740+0000 7fd805982740 20 client.26180 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:39.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.740+0000 7fd805982740 1 --2- 172.21.15.43:0/1364910727 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55a00fa2f580 0x55a00fa31970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.740+0000 7fd805982740 4 client.26180 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:38:39.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.740+0000 7fd805982740 1 -- 172.21.15.43:0/1364910727 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a00f920590 con 0x55a00fa2f580 2024-03-28T22:38:39.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.740+0000 7fd800c00640 1 --2- 172.21.15.43:0/1364910727 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55a00fa2f580 0x55a00fa31970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.741+0000 7fd800c00640 1 --2- 172.21.15.43:0/1364910727 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55a00fa2f580 0x55a00fa31970 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5eb536cd1b6e67a6 server_cookie=68fb96a216d024c2 in_seq=0 out_seq=0 2024-03-28T22:38:39.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.741+0000 7fd7e5600640 10 client.26180 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:38:39.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.741+0000 7fd7e5600640 1 -- 172.21.15.43:0/1364910727 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a00f920590 con 0x55a00fa2f580 2024-03-28T22:38:39.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.741+0000 7fd7e5600640 10 client.26180 handle_command_reply: tid=0 2024-03-28T22:38:39.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.741+0000 7fd805982740 1 client.26180 shutdown 2024-03-28T22:38:39.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.741+0000 7fd805982740 1 -- 172.21.15.43:0/1364910727 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a00f9afca0 msgr2=0x55a00f9bc000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:39.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.741+0000 7fd805982740 1 --2- 172.21.15.43:0/1364910727 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a00f9afca0 0x55a00f9bc000 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd7e8010920 tx=0x7fd7e8005250 comp rx=0 tx=0).stop 2024-03-28T22:38:39.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.741+0000 7fd805982740 1 -- 172.21.15.43:0/1364910727 shutdown_connections 2024-03-28T22:38:39.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.741+0000 7fd805982740 1 --2- 172.21.15.43:0/1364910727 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55a00fa2f580 0x55a00fa31970 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.741+0000 7fd805982740 1 --2- 172.21.15.43:0/1364910727 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a00f9afca0 0x55a00f9bc000 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.741+0000 7fd805982740 1 --2- 172.21.15.43:0/1364910727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a00f9b8880 0x55a00f9bc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.741+0000 7fd7e5600640 0 client.26180 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:38:39.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.741+0000 7fd805982740 1 --2- 172.21.15.43:0/1364910727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a00f9af380 0x55a00f9c0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.741+0000 7fd805982740 1 -- 172.21.15.43:0/1364910727 >> 172.21.15.43:0/1364910727 conn(0x55a00f9a9ab0 msgr2=0x55a00f9b58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:39.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.742+0000 7fd805982740 1 -- 172.21.15.43:0/1364910727 shutdown_connections 2024-03-28T22:38:39.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.742+0000 7fd805982740 1 -- 172.21.15.43:0/1364910727 wait complete. 2024-03-28T22:38:39.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.742+0000 7fd805982740 20 client.26180 trim_cache size 0 max 16384 2024-03-28T22:38:39.747 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:38:39.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.742+0000 7fd802c00640 1 -- 172.21.15.43:0/3183886391 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd7e00a2010 msgr2=0x7fd7e00a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:39.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.742+0000 7fd802c00640 1 --2- 172.21.15.43:0/3183886391 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd7e00a2010 0x7fd7e00a44d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec0eda50 tx=0x7fd7ec0f6530 comp rx=0 tx=0).stop 2024-03-28T22:38:39.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.742+0000 7fd802c00640 1 -- 172.21.15.43:0/3183886391 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7fc1016e0 msgr2=0x7fd7fc073a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:39.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.742+0000 7fd802c00640 1 --2- 172.21.15.43:0/3183886391 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7fc1016e0 0x7fd7fc073a50 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec005ab0 tx=0x7fd7ec005e80 comp rx=0 tx=0).stop 2024-03-28T22:38:39.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.743+0000 7fd802c00640 1 -- 172.21.15.43:0/3183886391 shutdown_connections 2024-03-28T22:38:39.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.743+0000 7fd802c00640 1 --2- 172.21.15.43:0/3183886391 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd7e00a2010 0x7fd7e00a44d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.743+0000 7fd802c00640 1 --2- 172.21.15.43:0/3183886391 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7fc1076f0 0x7fd7fc0744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.743+0000 7fd802c00640 1 --2- 172.21.15.43:0/3183886391 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7fc1020a0 0x7fd7fc073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.743+0000 7fd802c00640 1 --2- 172.21.15.43:0/3183886391 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7fc1016e0 0x7fd7fc073a50 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.743+0000 7fd802c00640 1 -- 172.21.15.43:0/3183886391 >> 172.21.15.43:0/3183886391 conn(0x7fd7fc078b50 msgr2=0x7fd7fc103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:39.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.743+0000 7fd802c00640 1 -- 172.21.15.43:0/3183886391 shutdown_connections 2024-03-28T22:38:39.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.743+0000 7fd802c00640 1 -- 172.21.15.43:0/3183886391 wait complete. 2024-03-28T22:38:39.758 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T22:38:39.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.977+0000 7fed52e00640 1 -- 172.21.15.43:0/3009037652 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fed4c10df20 msgr2=0x7fed4c110370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:39.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.977+0000 7fed52e00640 1 --2- 172.21.15.43:0/3009037652 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fed4c10df20 0x7fed4c110370 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fed44009900 tx=0x7fed4402bcd0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.977+0000 7fed52e00640 1 -- 172.21.15.43:0/3009037652 shutdown_connections 2024-03-28T22:38:39.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.977+0000 7fed52e00640 1 --2- 172.21.15.43:0/3009037652 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fed4c10df20 0x7fed4c110370 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.977+0000 7fed52e00640 1 --2- 172.21.15.43:0/3009037652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed4c0744a0 0x7fed4c10d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.977+0000 7fed52e00640 1 --2- 172.21.15.43:0/3009037652 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed4c073b80 0x7fed4c073f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.977+0000 7fed52e00640 1 -- 172.21.15.43:0/3009037652 >> 172.21.15.43:0/3009037652 conn(0x7fed4c078b50 msgr2=0x7fed4c078f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:39.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.978+0000 7fed52e00640 1 -- 172.21.15.43:0/3009037652 shutdown_connections 2024-03-28T22:38:39.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.978+0000 7fed52e00640 1 -- 172.21.15.43:0/3009037652 wait complete. 2024-03-28T22:38:39.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.979+0000 7fed52e00640 1 Processor -- start 2024-03-28T22:38:39.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.979+0000 7fed52e00640 1 -- start start 2024-03-28T22:38:39.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.979+0000 7fed52e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed4c073b80 0x7fed4c19a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.979+0000 7fed52e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed4c0744a0 0x7fed4c19a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.979+0000 7fed52e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fed4c10df20 0x7fed4c19ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.979+0000 7fed52e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fed4c113cc0 con 0x7fed4c073b80 2024-03-28T22:38:39.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.979+0000 7fed52e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fed4c113b40 con 0x7fed4c10df20 2024-03-28T22:38:39.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.979+0000 7fed52e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fed4c113e40 con 0x7fed4c0744a0 2024-03-28T22:38:39.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.979+0000 7fed4be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed4c073b80 0x7fed4c19a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.979+0000 7fed4b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed4c0744a0 0x7fed4c19a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.979+0000 7fed50e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fed4c10df20 0x7fed4c19ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.979+0000 7fed4b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed4c0744a0 0x7fed4c19a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:36012/0 (socket says 172.21.15.43:36012) 2024-03-28T22:38:39.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.979+0000 7fed50e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fed4c10df20 0x7fed4c19ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:40478/0 (socket says 172.21.15.43:40478) 2024-03-28T22:38:39.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.980+0000 7fed4b400640 1 -- 172.21.15.43:0/959412185 learned_addr learned my addr 172.21.15.43:0/959412185 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:39.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.980+0000 7fed4be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed4c073b80 0x7fed4c19a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35622/0 (socket says 172.21.15.43:35622) 2024-03-28T22:38:39.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.980+0000 7fed4b400640 1 -- 172.21.15.43:0/959412185 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fed4c10df20 msgr2=0x7fed4c19ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:39.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.980+0000 7fed4b400640 1 --2- 172.21.15.43:0/959412185 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fed4c10df20 0x7fed4c19ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.980+0000 7fed4b400640 1 -- 172.21.15.43:0/959412185 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed4c073b80 msgr2=0x7fed4c19a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:39.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.980+0000 7fed4b400640 1 --2- 172.21.15.43:0/959412185 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed4c073b80 0x7fed4c19a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.980+0000 7fed4b400640 1 -- 172.21.15.43:0/959412185 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed4c19f170 con 0x7fed4c0744a0 2024-03-28T22:38:39.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.980+0000 7fed50e00640 1 --2- 172.21.15.43:0/959412185 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fed4c10df20 0x7fed4c19ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:39.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.980+0000 7fed4be00640 1 --2- 172.21.15.43:0/959412185 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed4c073b80 0x7fed4c19a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:39.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.980+0000 7fed4b400640 1 --2- 172.21.15.43:0/959412185 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed4c0744a0 0x7fed4c19a6c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fed3800e9f0 tx=0x7fed3800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:39.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.981+0000 7fed48c00640 1 -- 172.21.15.43:0/959412185 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3800cd80 con 0x7fed4c0744a0 2024-03-28T22:38:39.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.981+0000 7fed52e00640 1 -- 172.21.15.43:0/959412185 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed4c19f460 con 0x7fed4c0744a0 2024-03-28T22:38:39.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.981+0000 7fed48c00640 1 -- 172.21.15.43:0/959412185 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed38010910 con 0x7fed4c0744a0 2024-03-28T22:38:39.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.981+0000 7fed48c00640 1 -- 172.21.15.43:0/959412185 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed380188f0 con 0x7fed4c0744a0 2024-03-28T22:38:39.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.981+0000 7fed52e00640 1 -- 172.21.15.43:0/959412185 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed4c1a6cf0 con 0x7fed4c0744a0 2024-03-28T22:38:39.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.984+0000 7fed55c0b740 1 Processor -- start 2024-03-28T22:38:39.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.984+0000 7fed55c0b740 1 -- start start 2024-03-28T22:38:39.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.985+0000 7fed55c0b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5596f3887380 0x5596f3887760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.985+0000 7fed55c0b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5596f3887ca0 0x5596f3890340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.985+0000 7fed55c0b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5596f3890880 0x5596f3892c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.985+0000 7fed55c0b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5596f37eb050 con 0x5596f3887380 2024-03-28T22:38:39.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.985+0000 7fed55c0b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5596f3793ce0 con 0x5596f3890880 2024-03-28T22:38:39.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.985+0000 7fed55c0b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5596f37e70f0 con 0x5596f3887ca0 2024-03-28T22:38:39.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.985+0000 7fed48c00640 1 -- 172.21.15.43:0/959412185 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 27) ==== 136989+0+0 (secure 0 0 0) 0x7fed38010ac0 con 0x7fed4c0744a0 2024-03-28T22:38:39.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.985+0000 7fed50e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5596f3887380 0x5596f3887760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.985+0000 7fed4be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5596f3887ca0 0x5596f3890340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.985+0000 7fed4be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5596f3887ca0 0x5596f3890340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:36020/0 (socket says 172.21.15.43:36020) 2024-03-28T22:38:39.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.985+0000 7fed4be00640 1 -- 172.21.15.43:0/334150660 learned_addr learned my addr 172.21.15.43:0/334150660 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:39.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.985+0000 7fed50e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5596f3887380 0x5596f3887760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35638/0 (socket says 172.21.15.43:35638) 2024-03-28T22:38:39.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.985+0000 7fed4b400640 1 --2- 172.21.15.43:0/334150660 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5596f3890880 0x5596f3892c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.986+0000 7fed4be00640 1 -- 172.21.15.43:0/334150660 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5596f3890880 msgr2=0x5596f3892c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:39.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.986+0000 7fed4be00640 1 --2- 172.21.15.43:0/334150660 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5596f3890880 0x5596f3892c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.986+0000 7fed4be00640 1 -- 172.21.15.43:0/334150660 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5596f3887380 msgr2=0x5596f3887760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:39.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.986+0000 7fed4be00640 1 --2- 172.21.15.43:0/334150660 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5596f3887380 0x5596f3887760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.986+0000 7fed4be00640 1 -- 172.21.15.43:0/334150660 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596f34ff3f0 con 0x5596f3887ca0 2024-03-28T22:38:39.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.986+0000 7fed4be00640 1 --2- 172.21.15.43:0/334150660 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5596f3887ca0 0x5596f3890340 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fed3c011860 tx=0x7fed3c011d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=649c479c8dd3782f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:39.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.986+0000 7fed41600640 1 -- 172.21.15.43:0/334150660 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3c0112f0 con 0x5596f3887ca0 2024-03-28T22:38:39.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.986+0000 7fed41600640 1 -- 172.21.15.43:0/334150660 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed3c003e50 con 0x5596f3887ca0 2024-03-28T22:38:39.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.986+0000 7fed41600640 1 -- 172.21.15.43:0/334150660 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3c004590 con 0x5596f3887ca0 2024-03-28T22:38:39.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.987+0000 7fed55c0b740 1 -- 172.21.15.43:0/334150660 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5596f3887ca0 msgr2=0x5596f3890340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:39.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.987+0000 7fed55c0b740 1 --2- 172.21.15.43:0/334150660 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5596f3887ca0 0x5596f3890340 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fed3c011860 tx=0x7fed3c011d20 comp rx=0 tx=0).stop 2024-03-28T22:38:39.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.987+0000 7fed55c0b740 1 -- 172.21.15.43:0/334150660 shutdown_connections 2024-03-28T22:38:39.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.987+0000 7fed55c0b740 1 --2- 172.21.15.43:0/334150660 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5596f3890880 0x5596f3892c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.987+0000 7fed55c0b740 1 --2- 172.21.15.43:0/334150660 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5596f3887ca0 0x5596f3890340 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.987+0000 7fed55c0b740 1 --2- 172.21.15.43:0/334150660 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5596f3887380 0x5596f3887760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.987+0000 7fed55c0b740 1 -- 172.21.15.43:0/334150660 >> 172.21.15.43:0/334150660 conn(0x5596f3881ab0 msgr2=0x5596f3881e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:39.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.987+0000 7fed55c0b740 1 -- 172.21.15.43:0/334150660 shutdown_connections 2024-03-28T22:38:39.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.987+0000 7fed55c0b740 1 -- 172.21.15.43:0/334150660 wait complete. 2024-03-28T22:38:39.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.987+0000 7fed48c00640 1 --2- 172.21.15.43:0/959412185 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fed300a1fa0 0x7fed300a4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.987+0000 7fed48c00640 1 -- 172.21.15.43:0/959412185 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fed380ceaf0 con 0x7fed4c0744a0 2024-03-28T22:38:39.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.988+0000 7fed4be00640 1 --2- 172.21.15.43:0/959412185 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fed300a1fa0 0x7fed300a4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.988+0000 7fed55c0b740 1 -- start start 2024-03-28T22:38:39.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.988+0000 7fed55c0b740 1 Processor -- start 2024-03-28T22:38:39.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.988+0000 7fed4be00640 1 --2- 172.21.15.43:0/959412185 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fed300a1fa0 0x7fed300a4460 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fed3c011620 tx=0x7fed3c009fd0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:39.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.989+0000 7fed55c0b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5596f3887380 0x5596f388d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.989+0000 7fed55c0b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5596f3887ca0 0x5596f388de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.989+0000 7fed55c0b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5596f3890880 0x5596f388e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.989+0000 7fed55c0b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5596f3893670 con 0x5596f3887380 2024-03-28T22:38:39.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.989+0000 7fed55c0b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5596f37892f0 con 0x5596f3887ca0 2024-03-28T22:38:39.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.989+0000 7fed55c0b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5596f38937f0 con 0x5596f3890880 2024-03-28T22:38:39.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.989+0000 7fed55c0b740 10 client.0 _ll_register_callbacks cb 0x5596f34febe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:39.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.989+0000 7fed4b400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5596f3887ca0 0x5596f388de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.989+0000 7fed4be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5596f3887380 0x5596f388d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.989+0000 7fed4b400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5596f3887ca0 0x5596f388de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:40500/0 (socket says 172.21.15.43:40500) 2024-03-28T22:38:39.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.989+0000 7fed4b400640 1 -- 172.21.15.43:0/3607731017 learned_addr learned my addr 172.21.15.43:0/3607731017 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:39.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.989+0000 7fed50e00640 1 --2- 172.21.15.43:0/3607731017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5596f3890880 0x5596f388e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.989+0000 7fed4b400640 1 -- 172.21.15.43:0/3607731017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5596f3890880 msgr2=0x5596f388e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:39.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.989+0000 7fed4b400640 1 --2- 172.21.15.43:0/3607731017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5596f3890880 0x5596f388e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.989+0000 7fed4b400640 1 -- 172.21.15.43:0/3607731017 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5596f3887380 msgr2=0x5596f388d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:39.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.989+0000 7fed4b400640 1 --2- 172.21.15.43:0/3607731017 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5596f3887380 0x5596f388d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.989+0000 7fed4b400640 1 -- 172.21.15.43:0/3607731017 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596f35c97d0 con 0x5596f3887ca0 2024-03-28T22:38:39.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.990+0000 7fed4b400640 1 --2- 172.21.15.43:0/3607731017 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5596f3887ca0 0x5596f388de10 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fed3800e9c0 tx=0x7fed3809ab70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5aad8190ffa9d79 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:39.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.990+0000 7fed36000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:38:39.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.990+0000 7fed36000640 1 -- 172.21.15.43:0/3607731017 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3809efe0 con 0x5596f3887ca0 2024-03-28T22:38:39.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.990+0000 7fed36000640 1 -- 172.21.15.43:0/3607731017 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fed3809a390 con 0x5596f3887ca0 2024-03-28T22:38:39.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.990+0000 7fed55c0b740 1 -- 172.21.15.43:0/3607731017 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed3c003e50 con 0x5596f3887ca0 2024-03-28T22:38:39.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.990+0000 7fed36000640 1 -- 172.21.15.43:0/3607731017 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fed380a3630 con 0x5596f3887ca0 2024-03-28T22:38:39.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.990+0000 7fed55c0b740 10 client.26192 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:39.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.990+0000 7fed55c0b740 1 -- 172.21.15.43:0/3607731017 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5596f389f980 con 0x5596f3887ca0 2024-03-28T22:38:39.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.990+0000 7fed36000640 1 -- 172.21.15.43:0/3607731017 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed3c003e50 con 0x5596f3887ca0 2024-03-28T22:38:39.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.991+0000 7fed36000640 1 -- 172.21.15.43:0/3607731017 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fed38018b60 con 0x5596f3887ca0 2024-03-28T22:38:39.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.991+0000 7fed55c0b740 10 client.26192 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:39.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.991+0000 7fed55c0b740 10 client.26192 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:38:39.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.991+0000 7fed55c0b740 20 client.26192 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:39.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.991+0000 7fed55c0b740 1 --2- 172.21.15.43:0/3607731017 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5596f38a0e40 0x5596f38a3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:39.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.991+0000 7fed55c0b740 4 client.26192 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:38:39.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.991+0000 7fed55c0b740 1 -- 172.21.15.43:0/3607731017 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5596f3794680 con 0x5596f38a0e40 2024-03-28T22:38:39.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.991+0000 7fed4be00640 1 --2- 172.21.15.43:0/3607731017 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5596f38a0e40 0x5596f38a3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:39.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.992+0000 7fed4be00640 1 --2- 172.21.15.43:0/3607731017 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5596f38a0e40 0x5596f38a3290 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=dd063cff95081535 server_cookie=bfa3e84c45431d61 in_seq=0 out_seq=0 2024-03-28T22:38:39.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.992+0000 7fed36000640 10 client.26192 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:38:39.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.993+0000 7fed36000640 1 -- 172.21.15.43:0/3607731017 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5596f3794680 con 0x5596f38a0e40 2024-03-28T22:38:39.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.993+0000 7fed36000640 10 client.26192 handle_command_reply: tid=0 2024-03-28T22:38:39.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.993+0000 7fed55c0b740 1 client.26192 shutdown 2024-03-28T22:38:39.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.993+0000 7fed55c0b740 1 -- 172.21.15.43:0/3607731017 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5596f3887ca0 msgr2=0x5596f388de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:39.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.993+0000 7fed55c0b740 1 --2- 172.21.15.43:0/3607731017 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5596f3887ca0 0x5596f388de10 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fed3800e9c0 tx=0x7fed3809ab70 comp rx=0 tx=0).stop 2024-03-28T22:38:39.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.993+0000 7fed55c0b740 1 -- 172.21.15.43:0/3607731017 shutdown_connections 2024-03-28T22:38:39.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.993+0000 7fed55c0b740 1 --2- 172.21.15.43:0/3607731017 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5596f38a0e40 0x5596f38a3290 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.993+0000 7fed55c0b740 1 --2- 172.21.15.43:0/3607731017 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5596f3887ca0 0x5596f388de10 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.993+0000 7fed55c0b740 1 --2- 172.21.15.43:0/3607731017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5596f3890880 0x5596f388e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.993+0000 7fed55c0b740 1 --2- 172.21.15.43:0/3607731017 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5596f3887380 0x5596f388d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:39.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.993+0000 7fed55c0b740 1 -- 172.21.15.43:0/3607731017 >> 172.21.15.43:0/3607731017 conn(0x5596f3881ab0 msgr2=0x5596f3883800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:39.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.993+0000 7fed36000640 0 client.26192 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:38:39.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.994+0000 7fed55c0b740 1 -- 172.21.15.43:0/3607731017 shutdown_connections 2024-03-28T22:38:39.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.994+0000 7fed55c0b740 1 -- 172.21.15.43:0/3607731017 wait complete. 2024-03-28T22:38:39.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:39.994+0000 7fed55c0b740 20 client.26192 trim_cache size 0 max 16384 2024-03-28T22:38:40.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.009+0000 7fed55c0b740 1 Processor -- start 2024-03-28T22:38:40.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.009+0000 7fed55c0b740 1 -- start start 2024-03-28T22:38:40.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.009+0000 7fed55c0b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5596f3887380 0x5596f3890000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.009+0000 7fed55c0b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5596f3887ca0 0x5596f389c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.009+0000 7fed55c0b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5596f3890880 0x5596f38835f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.009+0000 7fed55c0b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fed4c1b0d20 con 0x5596f3887380 2024-03-28T22:38:40.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.009+0000 7fed55c0b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fed4c1b2860 con 0x5596f3890880 2024-03-28T22:38:40.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.009+0000 7fed55c0b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fed4c1b13f0 con 0x5596f3887ca0 2024-03-28T22:38:40.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.009+0000 7fed4b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5596f3887ca0 0x5596f389c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.010+0000 7fed4b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5596f3887ca0 0x5596f389c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:36030/0 (socket says 172.21.15.43:36030) 2024-03-28T22:38:40.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.010+0000 7fed4b400640 1 -- 172.21.15.43:0/3014239719 learned_addr learned my addr 172.21.15.43:0/3014239719 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:40.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.010+0000 7fed50e00640 1 --2- 172.21.15.43:0/3014239719 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5596f3890880 0x5596f38835f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.010+0000 7fed4b400640 1 -- 172.21.15.43:0/3014239719 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5596f3890880 msgr2=0x5596f38835f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:40.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.010+0000 7fed4b400640 1 --2- 172.21.15.43:0/3014239719 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5596f3890880 0x5596f38835f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.010+0000 7fed4be00640 1 --2- 172.21.15.43:0/3014239719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5596f3887380 0x5596f3890000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.010+0000 7fed4b400640 1 -- 172.21.15.43:0/3014239719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5596f3887380 msgr2=0x5596f3890000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:40.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.010+0000 7fed4b400640 1 --2- 172.21.15.43:0/3014239719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5596f3887380 0x5596f3890000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.010+0000 7fed4b400640 1 -- 172.21.15.43:0/3014239719 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596f388bec0 con 0x5596f3887ca0 2024-03-28T22:38:40.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.010+0000 7fed50e00640 1 --2- 172.21.15.43:0/3014239719 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5596f3890880 0x5596f38835f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:38:40.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.010+0000 7fed4be00640 1 --2- 172.21.15.43:0/3014239719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5596f3887380 0x5596f3890000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:38:40.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.010+0000 7fed4b400640 1 --2- 172.21.15.43:0/3014239719 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5596f3887ca0 0x5596f389c410 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fed380978c0 tx=0x7fed38097310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d7e58934dd9c71a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:40.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.010+0000 7fed40c00640 1 -- 172.21.15.43:0/3014239719 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed380cbc00 con 0x5596f3887ca0 2024-03-28T22:38:40.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.011+0000 7fed40c00640 1 -- 172.21.15.43:0/3014239719 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed380cc1b0 con 0x5596f3887ca0 2024-03-28T22:38:40.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.011+0000 7fed40c00640 1 -- 172.21.15.43:0/3014239719 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed38018cf0 con 0x5596f3887ca0 2024-03-28T22:38:40.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.011+0000 7fed55c0b740 1 -- 172.21.15.43:0/3014239719 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5596f3887ca0 msgr2=0x5596f389c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:40.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.011+0000 7fed55c0b740 1 --2- 172.21.15.43:0/3014239719 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5596f3887ca0 0x5596f389c410 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fed380978c0 tx=0x7fed38097310 comp rx=0 tx=0).stop 2024-03-28T22:38:40.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.011+0000 7fed55c0b740 1 -- 172.21.15.43:0/3014239719 shutdown_connections 2024-03-28T22:38:40.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.011+0000 7fed55c0b740 1 --2- 172.21.15.43:0/3014239719 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5596f3890880 0x5596f38835f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.011+0000 7fed55c0b740 1 --2- 172.21.15.43:0/3014239719 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5596f3887ca0 0x5596f389c410 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.011+0000 7fed55c0b740 1 --2- 172.21.15.43:0/3014239719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5596f3887380 0x5596f3890000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.011+0000 7fed55c0b740 1 -- 172.21.15.43:0/3014239719 >> 172.21.15.43:0/3014239719 conn(0x5596f3881ab0 msgr2=0x5596f3882ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:40.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.011+0000 7fed55c0b740 1 -- 172.21.15.43:0/3014239719 shutdown_connections 2024-03-28T22:38:40.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.011+0000 7fed55c0b740 1 -- 172.21.15.43:0/3014239719 wait complete. 2024-03-28T22:38:40.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.012+0000 7fed55c0b740 1 -- start start 2024-03-28T22:38:40.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.012+0000 7fed55c0b740 1 Processor -- start 2024-03-28T22:38:40.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.012+0000 7fed55c0b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5596f3887380 0x5596f388a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.012+0000 7fed55c0b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5596f3887ca0 0x5596f388aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.012+0000 7fed55c0b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5596f3890880 0x5596f388afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.012+0000 7fed55c0b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fed4c1b2190 con 0x5596f3890880 2024-03-28T22:38:40.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.012+0000 7fed55c0b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fed4c1b1ac0 con 0x5596f3887380 2024-03-28T22:38:40.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.012+0000 7fed55c0b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fed4c1aed00 con 0x5596f3887ca0 2024-03-28T22:38:40.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.013+0000 7fed55c0b740 10 client.0 _ll_register_callbacks cb 0x7fed38009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:40.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.013+0000 7fed4b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5596f3887ca0 0x5596f388aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.013+0000 7fed4be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5596f3887380 0x5596f388a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.013+0000 7fed50e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5596f3890880 0x5596f388afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.013+0000 7fed4b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5596f3887ca0 0x5596f388aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:36036/0 (socket says 172.21.15.43:36036) 2024-03-28T22:38:40.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.013+0000 7fed4b400640 1 -- 172.21.15.43:0/3155913862 learned_addr learned my addr 172.21.15.43:0/3155913862 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:40.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.013+0000 7fed4b400640 1 -- 172.21.15.43:0/3155913862 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5596f3887380 msgr2=0x5596f388a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:40.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.013+0000 7fed4b400640 1 --2- 172.21.15.43:0/3155913862 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5596f3887380 0x5596f388a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.013+0000 7fed4b400640 1 -- 172.21.15.43:0/3155913862 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5596f3890880 msgr2=0x5596f388afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:40.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.013+0000 7fed4b400640 1 --2- 172.21.15.43:0/3155913862 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5596f3890880 0x5596f388afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.013+0000 7fed4b400640 1 -- 172.21.15.43:0/3155913862 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596f3906190 con 0x5596f3887ca0 2024-03-28T22:38:40.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.013+0000 7fed4b400640 1 --2- 172.21.15.43:0/3155913862 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5596f3887ca0 0x5596f388aa60 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fed3809ac40 tx=0x7fed380b9320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd94bc0b693c1412 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:40.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.013+0000 7fed36000640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:38:40.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.014+0000 7fed36000640 1 -- 172.21.15.43:0/3155913862 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed380b93d0 con 0x5596f3887ca0 2024-03-28T22:38:40.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.014+0000 7fed55c0b740 1 -- 172.21.15.43:0/3155913862 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed380cc1b0 con 0x5596f3887ca0 2024-03-28T22:38:40.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.014+0000 7fed36000640 1 -- 172.21.15.43:0/3155913862 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fed38018a90 con 0x5596f3887ca0 2024-03-28T22:38:40.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.014+0000 7fed36000640 1 -- 172.21.15.43:0/3155913862 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fed38018c90 con 0x5596f3887ca0 2024-03-28T22:38:40.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.014+0000 7fed55c0b740 10 client.26146 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:40.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.014+0000 7fed55c0b740 1 -- 172.21.15.43:0/3155913862 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5596f39073e0 con 0x5596f3887ca0 2024-03-28T22:38:40.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.014+0000 7fed36000640 1 -- 172.21.15.43:0/3155913862 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed380cc1b0 con 0x5596f3887ca0 2024-03-28T22:38:40.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.014+0000 7fed36000640 1 -- 172.21.15.43:0/3155913862 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fed38096e90 con 0x5596f3887ca0 2024-03-28T22:38:40.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.014+0000 7fed55c0b740 10 client.26146 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:40.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.014+0000 7fed55c0b740 10 client.26146 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:38:40.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.014+0000 7fed55c0b740 20 client.26146 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:40.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.014+0000 7fed55c0b740 1 --2- 172.21.15.43:0/3155913862 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5596f3907850 0x5596f3909c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.015+0000 7fed55c0b740 4 client.26146 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:38:40.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.015+0000 7fed55c0b740 1 -- 172.21.15.43:0/3155913862 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5596f38a3620 con 0x5596f3907850 2024-03-28T22:38:40.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.015+0000 7fed4be00640 1 --2- 172.21.15.43:0/3155913862 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5596f3907850 0x5596f3909c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.015+0000 7fed4be00640 1 --2- 172.21.15.43:0/3155913862 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5596f3907850 0x5596f3909c70 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e759214cfbd1c6b3 server_cookie=a45f6f9459b1b293 in_seq=0 out_seq=0 2024-03-28T22:38:40.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.015+0000 7fed36000640 10 client.26146 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:38:40.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.015+0000 7fed36000640 1 -- 172.21.15.43:0/3155913862 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5596f38a3620 con 0x5596f3907850 2024-03-28T22:38:40.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.015+0000 7fed36000640 10 client.26146 handle_command_reply: tid=0 2024-03-28T22:38:40.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.015+0000 7fed55c0b740 1 client.26146 shutdown 2024-03-28T22:38:40.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.016+0000 7fed55c0b740 1 -- 172.21.15.43:0/3155913862 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5596f3887ca0 msgr2=0x5596f388aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:40.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.016+0000 7fed55c0b740 1 --2- 172.21.15.43:0/3155913862 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5596f3887ca0 0x5596f388aa60 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fed3809ac40 tx=0x7fed380b9320 comp rx=0 tx=0).stop 2024-03-28T22:38:40.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.016+0000 7fed55c0b740 1 -- 172.21.15.43:0/3155913862 shutdown_connections 2024-03-28T22:38:40.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.016+0000 7fed55c0b740 1 --2- 172.21.15.43:0/3155913862 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5596f3907850 0x5596f3909c70 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.016+0000 7fed55c0b740 1 --2- 172.21.15.43:0/3155913862 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5596f3890880 0x5596f388afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.016+0000 7fed55c0b740 1 --2- 172.21.15.43:0/3155913862 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5596f3887ca0 0x5596f388aa60 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.016+0000 7fed55c0b740 1 --2- 172.21.15.43:0/3155913862 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5596f3887380 0x5596f388a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.016+0000 7fed36000640 0 client.26146 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:38:40.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.016+0000 7fed55c0b740 1 -- 172.21.15.43:0/3155913862 >> 172.21.15.43:0/3155913862 conn(0x5596f3881ab0 msgr2=0x5596f3883770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:40.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.016+0000 7fed55c0b740 1 -- 172.21.15.43:0/3155913862 shutdown_connections 2024-03-28T22:38:40.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.016+0000 7fed55c0b740 1 -- 172.21.15.43:0/3155913862 wait complete. 2024-03-28T22:38:40.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.016+0000 7fed55c0b740 20 client.26146 trim_cache size 0 max 16384 2024-03-28T22:38:40.018 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:38:40.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.017+0000 7fed52e00640 1 -- 172.21.15.43:0/959412185 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fed300a1fa0 msgr2=0x7fed300a4460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:40.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.017+0000 7fed52e00640 1 --2- 172.21.15.43:0/959412185 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fed300a1fa0 0x7fed300a4460 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fed3c011620 tx=0x7fed3c009fd0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.017+0000 7fed52e00640 1 -- 172.21.15.43:0/959412185 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed4c0744a0 msgr2=0x7fed4c19a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:40.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.017+0000 7fed52e00640 1 --2- 172.21.15.43:0/959412185 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed4c0744a0 0x7fed4c19a6c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fed3800e9f0 tx=0x7fed3800eeb0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.017+0000 7fed52e00640 1 -- 172.21.15.43:0/959412185 shutdown_connections 2024-03-28T22:38:40.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.017+0000 7fed52e00640 1 --2- 172.21.15.43:0/959412185 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fed300a1fa0 0x7fed300a4460 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.017+0000 7fed52e00640 1 --2- 172.21.15.43:0/959412185 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fed4c10df20 0x7fed4c19ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.017+0000 7fed52e00640 1 --2- 172.21.15.43:0/959412185 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fed4c0744a0 0x7fed4c19a6c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.017+0000 7fed52e00640 1 --2- 172.21.15.43:0/959412185 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fed4c073b80 0x7fed4c19a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.017+0000 7fed52e00640 1 -- 172.21.15.43:0/959412185 >> 172.21.15.43:0/959412185 conn(0x7fed4c078b50 msgr2=0x7fed4c0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:40.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.017+0000 7fed52e00640 1 -- 172.21.15.43:0/959412185 shutdown_connections 2024-03-28T22:38:40.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.017+0000 7fed52e00640 1 -- 172.21.15.43:0/959412185 wait complete. 2024-03-28T22:38:40.034 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:38:40.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.268+0000 7fd0dae00640 1 -- 172.21.15.43:0/2957937638 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0d4104210 msgr2=0x7fd0d4104690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:40.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.268+0000 7fd0dae00640 1 --2- 172.21.15.43:0/2957937638 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0d4104210 0x7fd0d4104690 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc009cd0 tx=0x7fd0cc02c0e0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.268+0000 7fd0dae00640 1 -- 172.21.15.43:0/2957937638 shutdown_connections 2024-03-28T22:38:40.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.268+0000 7fd0dae00640 1 --2- 172.21.15.43:0/2957937638 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0d4104210 0x7fd0d4104690 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.268+0000 7fd0dae00640 1 --2- 172.21.15.43:0/2957937638 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0d4068e90 0x7fd0d40692f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.268+0000 7fd0dae00640 1 --2- 172.21.15.43:0/2957937638 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0d41066a0 0x7fd0d4068950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.268+0000 7fd0dae00640 1 -- 172.21.15.43:0/2957937638 >> 172.21.15.43:0/2957937638 conn(0x7fd0d4077750 msgr2=0x7fd0d4077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:40.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.269+0000 7fd0dae00640 1 -- 172.21.15.43:0/2957937638 shutdown_connections 2024-03-28T22:38:40.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.269+0000 7fd0dae00640 1 -- 172.21.15.43:0/2957937638 wait complete. 2024-03-28T22:38:40.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.270+0000 7fd0dae00640 1 Processor -- start 2024-03-28T22:38:40.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.270+0000 7fd0dae00640 1 -- start start 2024-03-28T22:38:40.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.271+0000 7fd0dae00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0d4068e90 0x7fd0d419e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.271+0000 7fd0dae00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0d4104210 0x7fd0d419eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.271+0000 7fd0dae00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0d41066a0 0x7fd0d41a2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.271+0000 7fd0dae00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d41180a0 con 0x7fd0d41066a0 2024-03-28T22:38:40.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.271+0000 7fd0dae00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d4117f20 con 0x7fd0d4068e90 2024-03-28T22:38:40.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.271+0000 7fd0dae00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d4118220 con 0x7fd0d4104210 2024-03-28T22:38:40.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.271+0000 7fd0d3e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0d4068e90 0x7fd0d419e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.271+0000 7fd0d3e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0d4068e90 0x7fd0d419e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:40524/0 (socket says 172.21.15.43:40524) 2024-03-28T22:38:40.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.271+0000 7fd0d3400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0d4104210 0x7fd0d419eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.271+0000 7fd0d3e00640 1 -- 172.21.15.43:0/940066000 learned_addr learned my addr 172.21.15.43:0/940066000 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:40.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.271+0000 7fd0d8e00640 1 --2- 172.21.15.43:0/940066000 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0d41066a0 0x7fd0d41a2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.271+0000 7fd0d3e00640 1 -- 172.21.15.43:0/940066000 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0d4104210 msgr2=0x7fd0d419eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:40.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.271+0000 7fd0d3e00640 1 --2- 172.21.15.43:0/940066000 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0d4104210 0x7fd0d419eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.271+0000 7fd0d3e00640 1 -- 172.21.15.43:0/940066000 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0d41066a0 msgr2=0x7fd0d41a2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:40.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.271+0000 7fd0d3e00640 1 --2- 172.21.15.43:0/940066000 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0d41066a0 0x7fd0d41a2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.271+0000 7fd0d3e00640 1 -- 172.21.15.43:0/940066000 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0d41a3580 con 0x7fd0d4068e90 2024-03-28T22:38:40.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.271+0000 7fd0d8e00640 1 --2- 172.21.15.43:0/940066000 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0d41066a0 0x7fd0d41a2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:38:40.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.272+0000 7fd0d3e00640 1 --2- 172.21.15.43:0/940066000 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0d4068e90 0x7fd0d419e5e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fd0c400e9f0 tx=0x7fd0c400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:40.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.272+0000 7fd0d0c00640 1 -- 172.21.15.43:0/940066000 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c400cd80 con 0x7fd0d4068e90 2024-03-28T22:38:40.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.272+0000 7fd0d0c00640 1 -- 172.21.15.43:0/940066000 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0c4005500 con 0x7fd0d4068e90 2024-03-28T22:38:40.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.272+0000 7fd0dae00640 1 -- 172.21.15.43:0/940066000 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0d41a3870 con 0x7fd0d4068e90 2024-03-28T22:38:40.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.272+0000 7fd0d0c00640 1 -- 172.21.15.43:0/940066000 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c4010690 con 0x7fd0d4068e90 2024-03-28T22:38:40.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.272+0000 7fd0dae00640 1 -- 172.21.15.43:0/940066000 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0d41ab100 con 0x7fd0d4068e90 2024-03-28T22:38:40.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.276+0000 7fd0ddb64740 1 Processor -- start 2024-03-28T22:38:40.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.277+0000 7fd0ddb64740 1 -- start start 2024-03-28T22:38:40.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.277+0000 7fd0d0c00640 1 -- 172.21.15.43:0/940066000 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 27) ==== 136989+0+0 (secure 0 0 0) 0x7fd0c40046c0 con 0x7fd0d4068e90 2024-03-28T22:38:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.277+0000 7fd0ddb64740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c7072d8380 0x55c7072d8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.277+0000 7fd0ddb64740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c7072d8ca0 0x55c7072e1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.277+0000 7fd0ddb64740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c7072e1880 0x55c7072e3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.277+0000 7fd0ddb64740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c70723c050 con 0x55c7072e1880 2024-03-28T22:38:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.277+0000 7fd0ddb64740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55c7071e4ce0 con 0x55c7072d8ca0 2024-03-28T22:38:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.277+0000 7fd0ddb64740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55c7072380f0 con 0x55c7072d8380 2024-03-28T22:38:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.277+0000 7fd0d8e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c7072d8380 0x55c7072d8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.277+0000 7fd0d3e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c7072d8ca0 0x55c7072e1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.278+0000 7fd0d8e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c7072d8380 0x55c7072d8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:36056/0 (socket says 172.21.15.43:36056) 2024-03-28T22:38:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.278+0000 7fd0d8e00640 1 -- 172.21.15.43:0/3294839654 learned_addr learned my addr 172.21.15.43:0/3294839654 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.278+0000 7fd0d3400640 1 --2- 172.21.15.43:0/3294839654 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c7072e1880 0x55c7072e3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.278+0000 7fd0d8e00640 1 -- 172.21.15.43:0/3294839654 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c7072d8ca0 msgr2=0x55c7072e1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.278+0000 7fd0d8e00640 1 --2- 172.21.15.43:0/3294839654 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c7072d8ca0 0x55c7072e1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.278+0000 7fd0d8e00640 1 -- 172.21.15.43:0/3294839654 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c7072e1880 msgr2=0x55c7072e3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:40.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.278+0000 7fd0d8e00640 1 --2- 172.21.15.43:0/3294839654 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c7072e1880 0x55c7072e3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.278+0000 7fd0d8e00640 1 -- 172.21.15.43:0/3294839654 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c706f503f0 con 0x55c7072d8380 2024-03-28T22:38:40.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.278+0000 7fd0d3e00640 1 --2- 172.21.15.43:0/3294839654 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c7072d8ca0 0x55c7072e1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:40.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.278+0000 7fd0d3400640 1 --2- 172.21.15.43:0/3294839654 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c7072e1880 0x55c7072e3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:38:40.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.278+0000 7fd0d8e00640 1 --2- 172.21.15.43:0/3294839654 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c7072d8380 0x55c7072d8760 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc02c6f0 tx=0x7fd0cc005fa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6ae393175a3a562 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:40.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.279+0000 7fd0d0c00640 1 --2- 172.21.15.43:0/940066000 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd0b80a2010 0x7fd0b80a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.279+0000 7fd0c9600640 1 -- 172.21.15.43:0/3294839654 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc043020 con 0x55c7072d8380 2024-03-28T22:38:40.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.279+0000 7fd0c9600640 1 -- 172.21.15.43:0/3294839654 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0cc032c70 con 0x55c7072d8380 2024-03-28T22:38:40.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.279+0000 7fd0c9600640 1 -- 172.21.15.43:0/3294839654 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc039040 con 0x55c7072d8380 2024-03-28T22:38:40.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.279+0000 7fd0d0c00640 1 -- 172.21.15.43:0/940066000 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fd0c40cec60 con 0x7fd0d4068e90 2024-03-28T22:38:40.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.279+0000 7fd0d3e00640 1 --2- 172.21.15.43:0/940066000 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd0b80a2010 0x7fd0b80a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.279+0000 7fd0ddb64740 1 -- 172.21.15.43:0/3294839654 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c7072d8380 msgr2=0x55c7072d8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:40.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.279+0000 7fd0ddb64740 1 --2- 172.21.15.43:0/3294839654 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c7072d8380 0x55c7072d8760 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc02c6f0 tx=0x7fd0cc005fa0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.280+0000 7fd0ddb64740 1 -- 172.21.15.43:0/3294839654 shutdown_connections 2024-03-28T22:38:40.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.280+0000 7fd0ddb64740 1 --2- 172.21.15.43:0/3294839654 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c7072d8ca0 0x55c7072e1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.280+0000 7fd0ddb64740 1 --2- 172.21.15.43:0/3294839654 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c7072e1880 0x55c7072e3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.280+0000 7fd0ddb64740 1 --2- 172.21.15.43:0/3294839654 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c7072d8380 0x55c7072d8760 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.280+0000 7fd0ddb64740 1 -- 172.21.15.43:0/3294839654 >> 172.21.15.43:0/3294839654 conn(0x55c7072d2ab0 msgr2=0x55c7072d2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:40.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.280+0000 7fd0d3e00640 1 --2- 172.21.15.43:0/940066000 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd0b80a2010 0x7fd0b80a44d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd0c40cbef0 tx=0x7fd0c40d2770 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:40.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.280+0000 7fd0ddb64740 1 -- 172.21.15.43:0/3294839654 shutdown_connections 2024-03-28T22:38:40.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.280+0000 7fd0ddb64740 1 -- 172.21.15.43:0/3294839654 wait complete. 2024-03-28T22:38:40.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.280+0000 7fd0ddb64740 1 -- start start 2024-03-28T22:38:40.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.280+0000 7fd0ddb64740 1 Processor -- start 2024-03-28T22:38:40.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.281+0000 7fd0ddb64740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c7072d8380 0x55c7072de8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.281+0000 7fd0ddb64740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c7072d8ca0 0x55c7072dee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.281+0000 7fd0ddb64740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c7072e1880 0x55c7072df360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.281+0000 7fd0ddb64740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c7072e4610 con 0x55c7072d8ca0 2024-03-28T22:38:40.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.281+0000 7fd0ddb64740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55c7071da2f0 con 0x55c7072d8380 2024-03-28T22:38:40.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.281+0000 7fd0ddb64740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55c7072e4790 con 0x55c7072e1880 2024-03-28T22:38:40.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.281+0000 7fd0ddb64740 10 client.0 _ll_register_callbacks cb 0x55c706f4fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:40.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.281+0000 7fd0d8e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c7072d8380 0x55c7072de8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.281+0000 7fd0d8e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c7072d8380 0x55c7072de8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:40540/0 (socket says 172.21.15.43:40540) 2024-03-28T22:38:40.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.281+0000 7fd0d8e00640 1 -- 172.21.15.43:0/3834522276 learned_addr learned my addr 172.21.15.43:0/3834522276 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:40.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.281+0000 7fd0d3400640 1 --2- 172.21.15.43:0/3834522276 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c7072d8ca0 0x55c7072dee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.281+0000 7fd0d8e00640 1 --2- 172.21.15.43:0/3834522276 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c7072e1880 0x55c7072df360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.282+0000 7fd0d8e00640 1 -- 172.21.15.43:0/3834522276 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c7072e1880 msgr2=0x55c7072df360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:40.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.282+0000 7fd0d8e00640 1 --2- 172.21.15.43:0/3834522276 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c7072e1880 0x55c7072df360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.282+0000 7fd0d8e00640 1 -- 172.21.15.43:0/3834522276 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c7072d8ca0 msgr2=0x55c7072dee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:40.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.282+0000 7fd0d8e00640 1 --2- 172.21.15.43:0/3834522276 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c7072d8ca0 0x55c7072dee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.282+0000 7fd0d8e00640 1 -- 172.21.15.43:0/3834522276 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c70701a7d0 con 0x55c7072d8380 2024-03-28T22:38:40.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.282+0000 7fd0d8e00640 1 --2- 172.21.15.43:0/3834522276 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c7072d8380 0x55c7072de8e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc009620 tx=0x7fd0cc003f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9791ad5fdbd6a4c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:40.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.282+0000 7fd0be000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:38:40.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.282+0000 7fd0be000640 1 -- 172.21.15.43:0/3834522276 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc043030 con 0x55c7072d8380 2024-03-28T22:38:40.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.282+0000 7fd0be000640 1 -- 172.21.15.43:0/3834522276 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fd0cc004250 con 0x55c7072d8380 2024-03-28T22:38:40.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.282+0000 7fd0ddb64740 1 -- 172.21.15.43:0/3834522276 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0cc032c70 con 0x55c7072d8380 2024-03-28T22:38:40.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.282+0000 7fd0be000640 1 -- 172.21.15.43:0/3834522276 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd0cc039070 con 0x55c7072d8380 2024-03-28T22:38:40.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.282+0000 7fd0ddb64740 10 client.26210 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:40.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.283+0000 7fd0ddb64740 1 -- 172.21.15.43:0/3834522276 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c7072f08e0 con 0x55c7072d8380 2024-03-28T22:38:40.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.283+0000 7fd0be000640 1 -- 172.21.15.43:0/3834522276 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0cc032c70 con 0x55c7072d8380 2024-03-28T22:38:40.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.283+0000 7fd0be000640 1 -- 172.21.15.43:0/3834522276 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd0cc03f870 con 0x55c7072d8380 2024-03-28T22:38:40.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.283+0000 7fd0ddb64740 10 client.26210 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:40.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.283+0000 7fd0ddb64740 10 client.26210 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:38:40.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.283+0000 7fd0ddb64740 20 client.26210 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:40.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.283+0000 7fd0ddb64740 1 --2- 172.21.15.43:0/3834522276 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55c7072f1c30 0x55c7072f4020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.283+0000 7fd0ddb64740 4 client.26210 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:38:40.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.283+0000 7fd0ddb64740 1 -- 172.21.15.43:0/3834522276 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c7071e5680 con 0x55c7072f1c30 2024-03-28T22:38:40.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.284+0000 7fd0d3400640 1 --2- 172.21.15.43:0/3834522276 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55c7072f1c30 0x55c7072f4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.284+0000 7fd0d3400640 1 --2- 172.21.15.43:0/3834522276 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55c7072f1c30 0x55c7072f4020 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4c7aa62dde7ee858 server_cookie=a71c1d363115cc36 in_seq=0 out_seq=0 2024-03-28T22:38:40.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.284+0000 7fd0be000640 10 client.26210 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:38:40.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.285+0000 7fd0be000640 1 -- 172.21.15.43:0/3834522276 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c7071e5680 con 0x55c7072f1c30 2024-03-28T22:38:40.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.285+0000 7fd0be000640 10 client.26210 handle_command_reply: tid=0 2024-03-28T22:38:40.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.285+0000 7fd0ddb64740 1 client.26210 shutdown 2024-03-28T22:38:40.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.285+0000 7fd0ddb64740 1 -- 172.21.15.43:0/3834522276 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c7072d8380 msgr2=0x55c7072de8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:40.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.285+0000 7fd0ddb64740 1 --2- 172.21.15.43:0/3834522276 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c7072d8380 0x55c7072de8e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc009620 tx=0x7fd0cc003f00 comp rx=0 tx=0).stop 2024-03-28T22:38:40.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.285+0000 7fd0ddb64740 1 -- 172.21.15.43:0/3834522276 shutdown_connections 2024-03-28T22:38:40.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.285+0000 7fd0ddb64740 1 --2- 172.21.15.43:0/3834522276 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c7072e1880 0x55c7072df360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.285+0000 7fd0ddb64740 1 --2- 172.21.15.43:0/3834522276 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c7072d8ca0 0x55c7072dee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.285+0000 7fd0ddb64740 1 --2- 172.21.15.43:0/3834522276 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55c7072f1c30 0x55c7072f4020 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.285+0000 7fd0ddb64740 1 --2- 172.21.15.43:0/3834522276 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c7072d8380 0x55c7072de8e0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.285+0000 7fd0ddb64740 1 -- 172.21.15.43:0/3834522276 >> 172.21.15.43:0/3834522276 conn(0x55c7072d2ab0 msgr2=0x55c7072d4800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:40.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.285+0000 7fd0be000640 0 client.26210 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:38:40.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.285+0000 7fd0ddb64740 1 -- 172.21.15.43:0/3834522276 shutdown_connections 2024-03-28T22:38:40.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.285+0000 7fd0ddb64740 1 -- 172.21.15.43:0/3834522276 wait complete. 2024-03-28T22:38:40.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.286+0000 7fd0ddb64740 20 client.26210 trim_cache size 0 max 16384 2024-03-28T22:38:40.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.301+0000 7fd0ddb64740 1 Processor -- start 2024-03-28T22:38:40.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.301+0000 7fd0ddb64740 1 -- start start 2024-03-28T22:38:40.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.301+0000 7fd0ddb64740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c7072d8380 0x55c7072e1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.301+0000 7fd0ddb64740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c7072d8ca0 0x55c7072df180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.301+0000 7fd0ddb64740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c7072e1880 0x55c7072f7040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.301+0000 7fd0ddb64740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d41b51b0 con 0x55c7072d8ca0 2024-03-28T22:38:40.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.301+0000 7fd0ddb64740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d41b6cf0 con 0x55c7072d8380 2024-03-28T22:38:40.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.301+0000 7fd0ddb64740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d41b5880 con 0x55c7072e1880 2024-03-28T22:38:40.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.301+0000 7fd0d3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c7072d8ca0 0x55c7072df180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.301+0000 7fd0d8e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c7072d8380 0x55c7072e1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.301+0000 7fd0d3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c7072d8ca0 0x55c7072df180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35712/0 (socket says 172.21.15.43:35712) 2024-03-28T22:38:40.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.301+0000 7fd0d3400640 1 -- 172.21.15.43:0/1141977840 learned_addr learned my addr 172.21.15.43:0/1141977840 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:40.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.302+0000 7fd0d8e00640 1 --2- 172.21.15.43:0/1141977840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c7072e1880 0x55c7072f7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.302+0000 7fd0d3400640 1 -- 172.21.15.43:0/1141977840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c7072e1880 msgr2=0x55c7072f7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:40.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.302+0000 7fd0d3400640 1 --2- 172.21.15.43:0/1141977840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c7072e1880 0x55c7072f7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.302+0000 7fd0d3400640 1 -- 172.21.15.43:0/1141977840 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c7072d8380 msgr2=0x55c7072e1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:40.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.302+0000 7fd0d3400640 1 --2- 172.21.15.43:0/1141977840 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c7072d8380 0x55c7072e1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.302+0000 7fd0d3400640 1 -- 172.21.15.43:0/1141977840 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7072da4b0 con 0x55c7072d8ca0 2024-03-28T22:38:40.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.302+0000 7fd0d3400640 1 --2- 172.21.15.43:0/1141977840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c7072d8ca0 0x55c7072df180 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4077f70 tx=0x7fd0c0013830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee63d4a6feb0b3f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:40.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.303+0000 7fd0c8c00640 1 -- 172.21.15.43:0/1141977840 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c00130d0 con 0x55c7072d8ca0 2024-03-28T22:38:40.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.303+0000 7fd0c8c00640 1 -- 172.21.15.43:0/1141977840 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0c0013270 con 0x55c7072d8ca0 2024-03-28T22:38:40.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.303+0000 7fd0c8c00640 1 -- 172.21.15.43:0/1141977840 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c0005550 con 0x55c7072d8ca0 2024-03-28T22:38:40.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.303+0000 7fd0ddb64740 1 -- 172.21.15.43:0/1141977840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c7072d8ca0 msgr2=0x55c7072df180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:40.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.303+0000 7fd0ddb64740 1 --2- 172.21.15.43:0/1141977840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c7072d8ca0 0x55c7072df180 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4077f70 tx=0x7fd0c0013830 comp rx=0 tx=0).stop 2024-03-28T22:38:40.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.303+0000 7fd0ddb64740 1 -- 172.21.15.43:0/1141977840 shutdown_connections 2024-03-28T22:38:40.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.303+0000 7fd0ddb64740 1 --2- 172.21.15.43:0/1141977840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c7072e1880 0x55c7072f7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.303+0000 7fd0ddb64740 1 --2- 172.21.15.43:0/1141977840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c7072d8ca0 0x55c7072df180 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.303+0000 7fd0ddb64740 1 --2- 172.21.15.43:0/1141977840 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c7072d8380 0x55c7072e1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.303+0000 7fd0ddb64740 1 -- 172.21.15.43:0/1141977840 >> 172.21.15.43:0/1141977840 conn(0x55c7072d2ab0 msgr2=0x55c7072ec9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:40.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.304+0000 7fd0ddb64740 1 -- 172.21.15.43:0/1141977840 shutdown_connections 2024-03-28T22:38:40.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.304+0000 7fd0ddb64740 1 -- 172.21.15.43:0/1141977840 wait complete. 2024-03-28T22:38:40.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.304+0000 7fd0ddb64740 1 -- start start 2024-03-28T22:38:40.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.304+0000 7fd0ddb64740 1 Processor -- start 2024-03-28T22:38:40.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.305+0000 7fd0ddb64740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c7072d8380 0x55c7072f49c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.305+0000 7fd0ddb64740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c7072d8ca0 0x55c7072f4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.305+0000 7fd0ddb64740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c7072e1880 0x55c7072f5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.305+0000 7fd0ddb64740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d41b6620 con 0x55c7072d8ca0 2024-03-28T22:38:40.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.305+0000 7fd0ddb64740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d41b5f50 con 0x55c7072d8380 2024-03-28T22:38:40.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.305+0000 7fd0ddb64740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d41b3190 con 0x55c7072e1880 2024-03-28T22:38:40.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.305+0000 7fd0ddb64740 10 client.0 _ll_register_callbacks cb 0x7fd0c40d2640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:40.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.305+0000 7fd0d3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c7072d8ca0 0x55c7072f4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.305+0000 7fd0d8e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c7072d8380 0x55c7072f49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.305+0000 7fd0d8e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c7072d8380 0x55c7072f49c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:40560/0 (socket says 172.21.15.43:40560) 2024-03-28T22:38:40.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.305+0000 7fd0d3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c7072d8ca0 0x55c7072f4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35714/0 (socket says 172.21.15.43:35714) 2024-03-28T22:38:40.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.305+0000 7fd0d8e00640 1 -- 172.21.15.43:0/2098484045 learned_addr learned my addr 172.21.15.43:0/2098484045 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:40.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.305+0000 7fd0d8e00640 1 --2- 172.21.15.43:0/2098484045 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c7072e1880 0x55c7072f5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.306+0000 7fd0d3400640 1 -- 172.21.15.43:0/2098484045 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c7072e1880 msgr2=0x55c7072f5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:40.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.306+0000 7fd0d3400640 1 --2- 172.21.15.43:0/2098484045 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c7072e1880 0x55c7072f5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.306+0000 7fd0d3400640 1 -- 172.21.15.43:0/2098484045 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c7072d8380 msgr2=0x55c7072f49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:40.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.306+0000 7fd0d3400640 1 --2- 172.21.15.43:0/2098484045 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c7072d8380 0x55c7072f49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.306+0000 7fd0d3400640 1 -- 172.21.15.43:0/2098484045 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c707356c20 con 0x55c7072d8ca0 2024-03-28T22:38:40.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.306+0000 7fd0d3400640 1 --2- 172.21.15.43:0/2098484045 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c7072d8ca0 0x55c7072f4f00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x55c7072d2650 tx=0x7fd0c000baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cab37c1e706c323 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:40.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.306+0000 7fd0be000640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:38:40.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.306+0000 7fd0be000640 1 -- 172.21.15.43:0/2098484045 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c001d630 con 0x55c7072d8ca0 2024-03-28T22:38:40.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.307+0000 7fd0ddb64740 1 -- 172.21.15.43:0/2098484045 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0c0013270 con 0x55c7072d8ca0 2024-03-28T22:38:40.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.307+0000 7fd0be000640 1 -- 172.21.15.43:0/2098484045 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fd0c0004020 con 0x55c7072d8ca0 2024-03-28T22:38:40.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.307+0000 7fd0be000640 1 -- 172.21.15.43:0/2098484045 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd0c001d7d0 con 0x55c7072d8ca0 2024-03-28T22:38:40.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.307+0000 7fd0ddb64740 10 client.16038 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:40.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.307+0000 7fd0ddb64740 1 -- 172.21.15.43:0/2098484045 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c707357f20 con 0x55c7072d8ca0 2024-03-28T22:38:40.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.307+0000 7fd0be000640 1 -- 172.21.15.43:0/2098484045 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0c0013270 con 0x55c7072d8ca0 2024-03-28T22:38:40.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.307+0000 7fd0be000640 1 -- 172.21.15.43:0/2098484045 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd0c0024070 con 0x55c7072d8ca0 2024-03-28T22:38:40.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.307+0000 7fd0ddb64740 10 client.16038 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:40.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.307+0000 7fd0ddb64740 10 client.16038 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:38:40.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.307+0000 7fd0ddb64740 20 client.16038 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:40.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.307+0000 7fd0ddb64740 1 --2- 172.21.15.43:0/2098484045 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55c7073586c0 0x55c70735aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.307+0000 7fd0ddb64740 4 client.16038 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:38:40.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.307+0000 7fd0ddb64740 1 -- 172.21.15.43:0/2098484045 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c707249590 con 0x55c7073586c0 2024-03-28T22:38:40.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.308+0000 7fd0d8e00640 1 --2- 172.21.15.43:0/2098484045 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55c7073586c0 0x55c70735aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.308+0000 7fd0d8e00640 1 --2- 172.21.15.43:0/2098484045 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55c7073586c0 0x55c70735aae0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e9ee1dfaf82fdda5 server_cookie=6d329fedf87abd0d in_seq=0 out_seq=0 2024-03-28T22:38:40.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.308+0000 7fd0be000640 10 client.16038 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:38:40.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.308+0000 7fd0be000640 1 -- 172.21.15.43:0/2098484045 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c707249590 con 0x55c7073586c0 2024-03-28T22:38:40.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.308+0000 7fd0be000640 10 client.16038 handle_command_reply: tid=0 2024-03-28T22:38:40.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.308+0000 7fd0ddb64740 1 client.16038 shutdown 2024-03-28T22:38:40.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.309+0000 7fd0ddb64740 1 -- 172.21.15.43:0/2098484045 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c7072d8ca0 msgr2=0x55c7072f4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:40.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.309+0000 7fd0ddb64740 1 --2- 172.21.15.43:0/2098484045 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c7072d8ca0 0x55c7072f4f00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x55c7072d2650 tx=0x7fd0c000baf0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.309+0000 7fd0ddb64740 1 -- 172.21.15.43:0/2098484045 shutdown_connections 2024-03-28T22:38:40.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.309+0000 7fd0ddb64740 1 --2- 172.21.15.43:0/2098484045 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c7072e1880 0x55c7072f5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.309+0000 7fd0ddb64740 1 --2- 172.21.15.43:0/2098484045 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c7072d8ca0 0x55c7072f4f00 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.309+0000 7fd0ddb64740 1 --2- 172.21.15.43:0/2098484045 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55c7073586c0 0x55c70735aae0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.309+0000 7fd0ddb64740 1 --2- 172.21.15.43:0/2098484045 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c7072d8380 0x55c7072f49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.309+0000 7fd0ddb64740 1 -- 172.21.15.43:0/2098484045 >> 172.21.15.43:0/2098484045 conn(0x55c7072d2ab0 msgr2=0x55c7072ec9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:40.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.309+0000 7fd0be000640 0 client.16038 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:38:40.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.309+0000 7fd0ddb64740 1 -- 172.21.15.43:0/2098484045 shutdown_connections 2024-03-28T22:38:40.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.309+0000 7fd0ddb64740 1 -- 172.21.15.43:0/2098484045 wait complete. 2024-03-28T22:38:40.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.309+0000 7fd0ddb64740 20 client.16038 trim_cache size 0 max 16384 2024-03-28T22:38:40.311 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:38:40.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.310+0000 7fd0dae00640 1 -- 172.21.15.43:0/940066000 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd0b80a2010 msgr2=0x7fd0b80a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:40.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.310+0000 7fd0dae00640 1 --2- 172.21.15.43:0/940066000 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd0b80a2010 0x7fd0b80a44d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd0c40cbef0 tx=0x7fd0c40d2770 comp rx=0 tx=0).stop 2024-03-28T22:38:40.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.310+0000 7fd0dae00640 1 -- 172.21.15.43:0/940066000 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0d4068e90 msgr2=0x7fd0d419e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:40.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.310+0000 7fd0dae00640 1 --2- 172.21.15.43:0/940066000 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0d4068e90 0x7fd0d419e5e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fd0c400e9f0 tx=0x7fd0c400edf0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.310+0000 7fd0dae00640 1 -- 172.21.15.43:0/940066000 shutdown_connections 2024-03-28T22:38:40.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.310+0000 7fd0dae00640 1 --2- 172.21.15.43:0/940066000 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd0b80a2010 0x7fd0b80a44d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.310+0000 7fd0dae00640 1 --2- 172.21.15.43:0/940066000 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd0d41066a0 0x7fd0d41a2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.310+0000 7fd0dae00640 1 --2- 172.21.15.43:0/940066000 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd0d4104210 0x7fd0d419eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.310+0000 7fd0dae00640 1 --2- 172.21.15.43:0/940066000 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0d4068e90 0x7fd0d419e5e0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.310+0000 7fd0dae00640 1 -- 172.21.15.43:0/940066000 >> 172.21.15.43:0/940066000 conn(0x7fd0d4077750 msgr2=0x7fd0d4101600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:40.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.310+0000 7fd0dae00640 1 -- 172.21.15.43:0/940066000 shutdown_connections 2024-03-28T22:38:40.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.310+0000 7fd0dae00640 1 -- 172.21.15.43:0/940066000 wait complete. 2024-03-28T22:38:40.323 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:40 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/4267409592' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:38:40.326 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:38:40.339 INFO:tasks.workunit.client.0.smithi043.stdout: 1 20201 11.25 MB/sec warmup 11 sec latency 13.409 ms 2024-03-28T22:38:40.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:40 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/4267409592' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:38:40.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:40 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/4267409592' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:38:40.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.546+0000 7f9b60600640 1 -- 172.21.15.43:0/2778423369 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9b581038f0 msgr2=0x7f9b58103d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:40.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.546+0000 7f9b60600640 1 --2- 172.21.15.43:0/2778423369 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9b581038f0 0x7f9b58103d70 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f9b44009900 tx=0x7f9b4402bc30 comp rx=0 tx=0).stop 2024-03-28T22:38:40.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.546+0000 7f9b60600640 1 -- 172.21.15.43:0/2778423369 shutdown_connections 2024-03-28T22:38:40.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.546+0000 7f9b60600640 1 --2- 172.21.15.43:0/2778423369 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9b581038f0 0x7f9b58103d70 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.546+0000 7f9b60600640 1 --2- 172.21.15.43:0/2778423369 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b581042b0 0x7f9b581107c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.546+0000 7f9b60600640 1 --2- 172.21.15.43:0/2778423369 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b58109900 0x7f9b58109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.546+0000 7f9b60600640 1 -- 172.21.15.43:0/2778423369 >> 172.21.15.43:0/2778423369 conn(0x7f9b580ff3a0 msgr2=0x7f9b581017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:40.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.546+0000 7f9b60600640 1 -- 172.21.15.43:0/2778423369 shutdown_connections 2024-03-28T22:38:40.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.546+0000 7f9b60600640 1 -- 172.21.15.43:0/2778423369 wait complete. 2024-03-28T22:38:40.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.548+0000 7f9b60600640 1 Processor -- start 2024-03-28T22:38:40.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.548+0000 7f9b60600640 1 -- start start 2024-03-28T22:38:40.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.548+0000 7f9b60600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9b581038f0 0x7f9b5819a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.548+0000 7f9b60600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b581042b0 0x7f9b5819a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.548+0000 7f9b60600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b58109900 0x7f9b5819eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.548+0000 7f9b60600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9b58113d60 con 0x7f9b58109900 2024-03-28T22:38:40.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.548+0000 7f9b60600640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9b58113be0 con 0x7f9b581038f0 2024-03-28T22:38:40.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.548+0000 7f9b60600640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9b58113ee0 con 0x7f9b581042b0 2024-03-28T22:38:40.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.549+0000 7f9b5dc00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9b581038f0 0x7f9b5819a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.549+0000 7f9b5d200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b581042b0 0x7f9b5819a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.549+0000 7f9b5dc00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9b581038f0 0x7f9b5819a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:40584/0 (socket says 172.21.15.43:40584) 2024-03-28T22:38:40.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.549+0000 7f9b5dc00640 1 -- 172.21.15.43:0/3518045176 learned_addr learned my addr 172.21.15.43:0/3518045176 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:40.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.549+0000 7f9b5e600640 1 --2- 172.21.15.43:0/3518045176 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b58109900 0x7f9b5819eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.549+0000 7f9b5dc00640 1 -- 172.21.15.43:0/3518045176 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b581042b0 msgr2=0x7f9b5819a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:40.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.549+0000 7f9b5dc00640 1 --2- 172.21.15.43:0/3518045176 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b581042b0 0x7f9b5819a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.549+0000 7f9b5dc00640 1 -- 172.21.15.43:0/3518045176 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b58109900 msgr2=0x7f9b5819eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:40.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.549+0000 7f9b5dc00640 1 --2- 172.21.15.43:0/3518045176 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b58109900 0x7f9b5819eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.549+0000 7f9b5dc00640 1 -- 172.21.15.43:0/3518045176 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b5819f1f0 con 0x7f9b581038f0 2024-03-28T22:38:40.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.549+0000 7f9b5d200640 1 --2- 172.21.15.43:0/3518045176 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b581042b0 0x7f9b5819a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:40.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.550+0000 7f9b5dc00640 1 --2- 172.21.15.43:0/3518045176 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9b581038f0 0x7f9b5819a200 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c00d950 tx=0x7f9b4c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:40.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.550+0000 7f9b56000640 1 -- 172.21.15.43:0/3518045176 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b4c017070 con 0x7f9b581038f0 2024-03-28T22:38:40.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.550+0000 7f9b56000640 1 -- 172.21.15.43:0/3518045176 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b4c009c30 con 0x7f9b581038f0 2024-03-28T22:38:40.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.550+0000 7f9b56000640 1 -- 172.21.15.43:0/3518045176 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b4c00f6b0 con 0x7f9b581038f0 2024-03-28T22:38:40.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.550+0000 7f9b60600640 1 -- 172.21.15.43:0/3518045176 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b5819f4e0 con 0x7f9b581038f0 2024-03-28T22:38:40.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.550+0000 7f9b60600640 1 -- 172.21.15.43:0/3518045176 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b581a6d70 con 0x7f9b581038f0 2024-03-28T22:38:40.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.554+0000 7f9b56000640 1 -- 172.21.15.43:0/3518045176 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 27) ==== 136989+0+0 (secure 0 0 0) 0x7f9b4c0109d0 con 0x7f9b581038f0 2024-03-28T22:38:40.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.554+0000 7f9b633ea740 1 Processor -- start 2024-03-28T22:38:40.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.554+0000 7f9b633ea740 1 -- start start 2024-03-28T22:38:40.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.555+0000 7f9b633ea740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a2ff47e380 0x55a2ff47e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.555+0000 7f9b633ea740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a2ff47eca0 0x55a2ff487340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.555+0000 7f9b633ea740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a2ff487880 0x55a2ff489c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.555+0000 7f9b633ea740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55a2ff3e2050 con 0x55a2ff487880 2024-03-28T22:38:40.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.555+0000 7f9b633ea740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55a2ff38ace0 con 0x55a2ff47e380 2024-03-28T22:38:40.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.555+0000 7f9b633ea740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55a2ff3de0f0 con 0x55a2ff47eca0 2024-03-28T22:38:40.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.555+0000 7f9b5e600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a2ff47e380 0x55a2ff47e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.555+0000 7f9b5dc00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a2ff47eca0 0x55a2ff487340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.555+0000 7f9b5e600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a2ff47e380 0x55a2ff47e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:40592/0 (socket says 172.21.15.43:40592) 2024-03-28T22:38:40.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.555+0000 7f9b5e600640 1 -- 172.21.15.43:0/3498304140 learned_addr learned my addr 172.21.15.43:0/3498304140 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:40.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.555+0000 7f9b5d200640 1 --2- 172.21.15.43:0/3498304140 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a2ff487880 0x55a2ff489c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.555+0000 7f9b5e600640 1 -- 172.21.15.43:0/3498304140 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a2ff47eca0 msgr2=0x55a2ff487340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:40.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.555+0000 7f9b5e600640 1 --2- 172.21.15.43:0/3498304140 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a2ff47eca0 0x55a2ff487340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.555+0000 7f9b5e600640 1 -- 172.21.15.43:0/3498304140 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a2ff487880 msgr2=0x55a2ff489c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:40.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.555+0000 7f9b5e600640 1 --2- 172.21.15.43:0/3498304140 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a2ff487880 0x55a2ff489c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.555+0000 7f9b5e600640 1 -- 172.21.15.43:0/3498304140 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2ff0f63f0 con 0x55a2ff47e380 2024-03-28T22:38:40.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.555+0000 7f9b5d200640 1 --2- 172.21.15.43:0/3498304140 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a2ff487880 0x55a2ff489c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:38:40.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.556+0000 7f9b5e600640 1 --2- 172.21.15.43:0/3498304140 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a2ff47e380 0x55a2ff47e760 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f9b50009fd0 tx=0x7f9b500118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53c3c005e1d9f041 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:40.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.556+0000 7f9b56000640 1 --2- 172.21.15.43:0/3518045176 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9b400a2090 0x7f9b400a4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.556+0000 7f9b4aa00640 1 -- 172.21.15.43:0/3498304140 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b50004d20 con 0x55a2ff47e380 2024-03-28T22:38:40.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.556+0000 7f9b56000640 1 -- 172.21.15.43:0/3518045176 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f9b4c0ce2f0 con 0x7f9b581038f0 2024-03-28T22:38:40.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.556+0000 7f9b4aa00640 1 -- 172.21.15.43:0/3498304140 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b50004050 con 0x55a2ff47e380 2024-03-28T22:38:40.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.556+0000 7f9b4aa00640 1 -- 172.21.15.43:0/3498304140 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b50005260 con 0x55a2ff47e380 2024-03-28T22:38:40.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.556+0000 7f9b5dc00640 1 --2- 172.21.15.43:0/3518045176 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9b400a2090 0x7f9b400a4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.557+0000 7f9b633ea740 1 -- 172.21.15.43:0/3498304140 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a2ff47e380 msgr2=0x55a2ff47e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:40.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.557+0000 7f9b633ea740 1 --2- 172.21.15.43:0/3498304140 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a2ff47e380 0x55a2ff47e760 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f9b50009fd0 tx=0x7f9b500118d0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.557+0000 7f9b5dc00640 1 --2- 172.21.15.43:0/3518045176 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9b400a2090 0x7f9b400a4550 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c0cb520 tx=0x7f9b4c0d1d80 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:40.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.557+0000 7f9b633ea740 1 -- 172.21.15.43:0/3498304140 shutdown_connections 2024-03-28T22:38:40.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.557+0000 7f9b633ea740 1 --2- 172.21.15.43:0/3498304140 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a2ff487880 0x55a2ff489c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.557+0000 7f9b633ea740 1 --2- 172.21.15.43:0/3498304140 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a2ff47eca0 0x55a2ff487340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.557+0000 7f9b633ea740 1 --2- 172.21.15.43:0/3498304140 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a2ff47e380 0x55a2ff47e760 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.557+0000 7f9b633ea740 1 -- 172.21.15.43:0/3498304140 >> 172.21.15.43:0/3498304140 conn(0x55a2ff478ab0 msgr2=0x55a2ff478e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:40.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.557+0000 7f9b633ea740 1 -- 172.21.15.43:0/3498304140 shutdown_connections 2024-03-28T22:38:40.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.557+0000 7f9b633ea740 1 -- 172.21.15.43:0/3498304140 wait complete. 2024-03-28T22:38:40.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.558+0000 7f9b633ea740 1 -- start start 2024-03-28T22:38:40.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.558+0000 7f9b633ea740 1 Processor -- start 2024-03-28T22:38:40.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.558+0000 7f9b633ea740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a2ff47e380 0x55a2ff484870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.558+0000 7f9b633ea740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a2ff47eca0 0x55a2ff484db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.558+0000 7f9b633ea740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a2ff487880 0x55a2ff4852f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.558+0000 7f9b633ea740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55a2ff48a670 con 0x55a2ff47e380 2024-03-28T22:38:40.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.558+0000 7f9b633ea740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55a2ff3802f0 con 0x55a2ff487880 2024-03-28T22:38:40.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.558+0000 7f9b633ea740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55a2ff48a7f0 con 0x55a2ff47eca0 2024-03-28T22:38:40.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.558+0000 7f9b633ea740 10 client.0 _ll_register_callbacks cb 0x55a2ff0f5be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:40.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.559+0000 7f9b5d200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a2ff47eca0 0x55a2ff484db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.559+0000 7f9b5e600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a2ff47e380 0x55a2ff484870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.559+0000 7f9b5d200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a2ff47eca0 0x55a2ff484db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:36106/0 (socket says 172.21.15.43:36106) 2024-03-28T22:38:40.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.559+0000 7f9b5d200640 1 -- 172.21.15.43:0/3332210376 learned_addr learned my addr 172.21.15.43:0/3332210376 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:40.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.559+0000 7f9b5e600640 1 --2- 172.21.15.43:0/3332210376 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a2ff487880 0x55a2ff4852f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.559+0000 7f9b5d200640 1 -- 172.21.15.43:0/3332210376 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a2ff487880 msgr2=0x55a2ff4852f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:40.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.559+0000 7f9b5d200640 1 --2- 172.21.15.43:0/3332210376 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a2ff487880 0x55a2ff4852f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.559+0000 7f9b5d200640 1 -- 172.21.15.43:0/3332210376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a2ff47e380 msgr2=0x55a2ff484870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:40.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.559+0000 7f9b5d200640 1 --2- 172.21.15.43:0/3332210376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a2ff47e380 0x55a2ff484870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.559+0000 7f9b5d200640 1 -- 172.21.15.43:0/3332210376 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2ff1c07d0 con 0x55a2ff47eca0 2024-03-28T22:38:40.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.559+0000 7f9b5e600640 1 --2- 172.21.15.43:0/3332210376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a2ff47e380 0x55a2ff484870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:38:40.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.559+0000 7f9b5d200640 1 --2- 172.21.15.43:0/3332210376 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a2ff47eca0 0x55a2ff484db0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9b44006010 tx=0x7f9b44005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3bbb424a1741950 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:40.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.559+0000 7f9b3f400640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:38:40.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.560+0000 7f9b3f400640 1 -- 172.21.15.43:0/3332210376 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b44004380 con 0x55a2ff47eca0 2024-03-28T22:38:40.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.560+0000 7f9b3f400640 1 -- 172.21.15.43:0/3332210376 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f9b4402c8b0 con 0x55a2ff47eca0 2024-03-28T22:38:40.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.560+0000 7f9b633ea740 1 -- 172.21.15.43:0/3332210376 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b50004050 con 0x55a2ff47eca0 2024-03-28T22:38:40.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.560+0000 7f9b3f400640 1 -- 172.21.15.43:0/3332210376 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9b4402cab0 con 0x55a2ff47eca0 2024-03-28T22:38:40.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.560+0000 7f9b633ea740 10 client.26176 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:40.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.560+0000 7f9b633ea740 1 -- 172.21.15.43:0/3332210376 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a2ff4968c0 con 0x55a2ff47eca0 2024-03-28T22:38:40.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.560+0000 7f9b3f400640 1 -- 172.21.15.43:0/3332210376 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b50004050 con 0x55a2ff47eca0 2024-03-28T22:38:40.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.560+0000 7f9b3f400640 1 -- 172.21.15.43:0/3332210376 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f9b44003850 con 0x55a2ff47eca0 2024-03-28T22:38:40.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.561+0000 7f9b633ea740 10 client.26176 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:40.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.561+0000 7f9b633ea740 10 client.26176 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:38:40.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.561+0000 7f9b633ea740 20 client.26176 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:40.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.561+0000 7f9b633ea740 1 --2- 172.21.15.43:0/3332210376 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55a2ff497d80 0x55a2ff49a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.561+0000 7f9b633ea740 4 client.26176 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:38:40.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.561+0000 7f9b633ea740 1 -- 172.21.15.43:0/3332210376 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a2ff38b680 con 0x55a2ff497d80 2024-03-28T22:38:40.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.561+0000 7f9b5e600640 1 --2- 172.21.15.43:0/3332210376 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55a2ff497d80 0x55a2ff49a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.561+0000 7f9b5e600640 1 --2- 172.21.15.43:0/3332210376 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55a2ff497d80 0x55a2ff49a1e0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=14fc249a979dfb56 server_cookie=7fc60fcdc958c6db in_seq=0 out_seq=0 2024-03-28T22:38:40.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.561+0000 7f9b3f400640 10 client.26176 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:38:40.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.562+0000 7f9b3f400640 1 -- 172.21.15.43:0/3332210376 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a2ff38b680 con 0x55a2ff497d80 2024-03-28T22:38:40.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.562+0000 7f9b3f400640 10 client.26176 handle_command_reply: tid=0 2024-03-28T22:38:40.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.562+0000 7f9b633ea740 1 client.26176 shutdown 2024-03-28T22:38:40.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.563+0000 7f9b633ea740 1 -- 172.21.15.43:0/3332210376 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a2ff47eca0 msgr2=0x55a2ff484db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:40.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.563+0000 7f9b633ea740 1 --2- 172.21.15.43:0/3332210376 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a2ff47eca0 0x55a2ff484db0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9b44006010 tx=0x7f9b44005d50 comp rx=0 tx=0).stop 2024-03-28T22:38:40.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.563+0000 7f9b633ea740 1 -- 172.21.15.43:0/3332210376 shutdown_connections 2024-03-28T22:38:40.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.563+0000 7f9b633ea740 1 --2- 172.21.15.43:0/3332210376 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55a2ff497d80 0x55a2ff49a1e0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.563+0000 7f9b633ea740 1 --2- 172.21.15.43:0/3332210376 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a2ff487880 0x55a2ff4852f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.563+0000 7f9b633ea740 1 --2- 172.21.15.43:0/3332210376 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a2ff47eca0 0x55a2ff484db0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.563+0000 7f9b633ea740 1 --2- 172.21.15.43:0/3332210376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a2ff47e380 0x55a2ff484870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.563+0000 7f9b3f400640 0 client.26176 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:38:40.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.563+0000 7f9b633ea740 1 -- 172.21.15.43:0/3332210376 >> 172.21.15.43:0/3332210376 conn(0x55a2ff478ab0 msgr2=0x55a2ff47a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:40.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.563+0000 7f9b633ea740 1 -- 172.21.15.43:0/3332210376 shutdown_connections 2024-03-28T22:38:40.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.563+0000 7f9b633ea740 1 -- 172.21.15.43:0/3332210376 wait complete. 2024-03-28T22:38:40.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.563+0000 7f9b633ea740 20 client.26176 trim_cache size 0 max 16384 2024-03-28T22:38:40.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.579+0000 7f9b633ea740 1 Processor -- start 2024-03-28T22:38:40.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.579+0000 7f9b633ea740 1 -- start start 2024-03-28T22:38:40.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.580+0000 7f9b633ea740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a2ff47e380 0x55a2ff486f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.580+0000 7f9b633ea740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a2ff47eca0 0x55a2ff493410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.580+0000 7f9b633ea740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a2ff487880 0x55a2ff49fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.580+0000 7f9b633ea740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9b581b0dd0 con 0x55a2ff47e380 2024-03-28T22:38:40.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.580+0000 7f9b633ea740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9b581b2910 con 0x55a2ff47eca0 2024-03-28T22:38:40.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.580+0000 7f9b633ea740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9b581b14a0 con 0x55a2ff487880 2024-03-28T22:38:40.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.580+0000 7f9b5e600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a2ff47e380 0x55a2ff486f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.580+0000 7f9b5d200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a2ff47eca0 0x55a2ff493410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.580+0000 7f9b5d200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a2ff47eca0 0x55a2ff493410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:40612/0 (socket says 172.21.15.43:40612) 2024-03-28T22:38:40.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.580+0000 7f9b5d200640 1 -- 172.21.15.43:0/1051999756 learned_addr learned my addr 172.21.15.43:0/1051999756 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:40.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.580+0000 7f9b5e600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a2ff47e380 0x55a2ff486f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35758/0 (socket says 172.21.15.43:35758) 2024-03-28T22:38:40.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.580+0000 7f9b5e600640 1 --2- 172.21.15.43:0/1051999756 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a2ff487880 0x55a2ff49fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.580+0000 7f9b5d200640 1 -- 172.21.15.43:0/1051999756 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a2ff487880 msgr2=0x55a2ff49fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:40.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.580+0000 7f9b5d200640 1 --2- 172.21.15.43:0/1051999756 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a2ff487880 0x55a2ff49fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.580+0000 7f9b5d200640 1 -- 172.21.15.43:0/1051999756 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a2ff47e380 msgr2=0x55a2ff486f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:40.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.581+0000 7f9b5d200640 1 --2- 172.21.15.43:0/1051999756 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a2ff47e380 0x55a2ff486f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.581+0000 7f9b5d200640 1 -- 172.21.15.43:0/1051999756 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2ff482dc0 con 0x55a2ff47eca0 2024-03-28T22:38:40.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.581+0000 7f9b5e600640 1 --2- 172.21.15.43:0/1051999756 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a2ff47e380 0x55a2ff486f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:40.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.581+0000 7f9b5d200640 1 --2- 172.21.15.43:0/1051999756 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a2ff47eca0 0x55a2ff493410 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f9b44003de0 tx=0x7f9b44005c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c9210d0a0d43ec9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:40.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.581+0000 7f9b4a000640 1 -- 172.21.15.43:0/1051999756 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b44009520 con 0x55a2ff47eca0 2024-03-28T22:38:40.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.581+0000 7f9b4a000640 1 -- 172.21.15.43:0/1051999756 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b4405a070 con 0x55a2ff47eca0 2024-03-28T22:38:40.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.581+0000 7f9b4a000640 1 -- 172.21.15.43:0/1051999756 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b44004f80 con 0x55a2ff47eca0 2024-03-28T22:38:40.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.582+0000 7f9b633ea740 1 -- 172.21.15.43:0/1051999756 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a2ff47eca0 msgr2=0x55a2ff493410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:40.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.582+0000 7f9b633ea740 1 --2- 172.21.15.43:0/1051999756 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a2ff47eca0 0x55a2ff493410 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f9b44003de0 tx=0x7f9b44005c00 comp rx=0 tx=0).stop 2024-03-28T22:38:40.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.582+0000 7f9b633ea740 1 -- 172.21.15.43:0/1051999756 shutdown_connections 2024-03-28T22:38:40.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.582+0000 7f9b633ea740 1 --2- 172.21.15.43:0/1051999756 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a2ff47eca0 0x55a2ff493410 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.582+0000 7f9b633ea740 1 --2- 172.21.15.43:0/1051999756 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a2ff487880 0x55a2ff49fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.582+0000 7f9b633ea740 1 --2- 172.21.15.43:0/1051999756 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a2ff47e380 0x55a2ff486f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.582+0000 7f9b633ea740 1 -- 172.21.15.43:0/1051999756 >> 172.21.15.43:0/1051999756 conn(0x55a2ff478ab0 msgr2=0x55a2ff47f080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:40.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.582+0000 7f9b633ea740 1 -- 172.21.15.43:0/1051999756 shutdown_connections 2024-03-28T22:38:40.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.582+0000 7f9b633ea740 1 -- 172.21.15.43:0/1051999756 wait complete. 2024-03-28T22:38:40.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.583+0000 7f9b633ea740 1 -- start start 2024-03-28T22:38:40.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.583+0000 7f9b633ea740 1 Processor -- start 2024-03-28T22:38:40.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.583+0000 7f9b633ea740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a2ff47e380 0x55a2ff481510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.583+0000 7f9b633ea740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a2ff47eca0 0x55a2ff481a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.583+0000 7f9b633ea740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a2ff487880 0x55a2ff481f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.583+0000 7f9b633ea740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9b581b2240 con 0x55a2ff47eca0 2024-03-28T22:38:40.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.583+0000 7f9b633ea740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9b581b1b70 con 0x55a2ff47e380 2024-03-28T22:38:40.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.583+0000 7f9b633ea740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9b581aedb0 con 0x55a2ff487880 2024-03-28T22:38:40.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.584+0000 7f9b633ea740 10 client.0 _ll_register_callbacks cb 0x7f9b44009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:40.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.584+0000 7f9b5e600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a2ff47e380 0x55a2ff481510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.584+0000 7f9b5e600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a2ff47e380 0x55a2ff481510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:40624/0 (socket says 172.21.15.43:40624) 2024-03-28T22:38:40.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.584+0000 7f9b5e600640 1 -- 172.21.15.43:0/150894436 learned_addr learned my addr 172.21.15.43:0/150894436 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:40.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.584+0000 7f9b5e600640 1 --2- 172.21.15.43:0/150894436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a2ff487880 0x55a2ff481f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.584+0000 7f9b5d200640 1 --2- 172.21.15.43:0/150894436 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a2ff47eca0 0x55a2ff481a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.584+0000 7f9b5e600640 1 -- 172.21.15.43:0/150894436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a2ff487880 msgr2=0x55a2ff481f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:40.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.584+0000 7f9b5e600640 1 --2- 172.21.15.43:0/150894436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a2ff487880 0x55a2ff481f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.584+0000 7f9b5e600640 1 -- 172.21.15.43:0/150894436 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a2ff47eca0 msgr2=0x55a2ff481a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:40.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.584+0000 7f9b5e600640 1 --2- 172.21.15.43:0/150894436 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a2ff47eca0 0x55a2ff481a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.584+0000 7f9b5e600640 1 -- 172.21.15.43:0/150894436 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2ff4fd1e0 con 0x55a2ff47e380 2024-03-28T22:38:40.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.585+0000 7f9b5e600640 1 --2- 172.21.15.43:0/150894436 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a2ff47e380 0x55a2ff481510 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f9b50006010 tx=0x7f9b500118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bea2879e4678625 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:40.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.585+0000 7f9b3f400640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:38:40.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.585+0000 7f9b3f400640 1 -- 172.21.15.43:0/150894436 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b5002e3c0 con 0x55a2ff47e380 2024-03-28T22:38:40.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.585+0000 7f9b633ea740 1 -- 172.21.15.43:0/150894436 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b4405a070 con 0x55a2ff47e380 2024-03-28T22:38:40.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.585+0000 7f9b3f400640 1 -- 172.21.15.43:0/150894436 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f9b5002e560 con 0x55a2ff47e380 2024-03-28T22:38:40.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.585+0000 7f9b3f400640 1 -- 172.21.15.43:0/150894436 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9b50042050 con 0x55a2ff47e380 2024-03-28T22:38:40.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.585+0000 7f9b633ea740 10 client.26234 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:40.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.585+0000 7f9b633ea740 1 -- 172.21.15.43:0/150894436 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a2ff4fe510 con 0x55a2ff47e380 2024-03-28T22:38:40.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.586+0000 7f9b3f400640 1 -- 172.21.15.43:0/150894436 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b4405a070 con 0x55a2ff47e380 2024-03-28T22:38:40.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.586+0000 7f9b3f400640 1 -- 172.21.15.43:0/150894436 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f9b5000f720 con 0x55a2ff47e380 2024-03-28T22:38:40.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.586+0000 7f9b633ea740 10 client.26234 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:40.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.586+0000 7f9b633ea740 10 client.26234 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:38:40.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.586+0000 7f9b633ea740 20 client.26234 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:40.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.586+0000 7f9b633ea740 1 --2- 172.21.15.43:0/150894436 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55a2ff4fea60 0x55a2ff500e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.586+0000 7f9b633ea740 4 client.26234 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:38:40.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.586+0000 7f9b633ea740 1 -- 172.21.15.43:0/150894436 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a2ff3ef590 con 0x55a2ff4fea60 2024-03-28T22:38:40.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.586+0000 7f9b5d200640 1 --2- 172.21.15.43:0/150894436 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55a2ff4fea60 0x55a2ff500e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.587+0000 7f9b5d200640 1 --2- 172.21.15.43:0/150894436 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55a2ff4fea60 0x55a2ff500e80 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a62d7171ca3296f4 server_cookie=a4dd6cc3f6ac72f5 in_seq=0 out_seq=0 2024-03-28T22:38:40.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.587+0000 7f9b3f400640 10 client.26234 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:38:40.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.587+0000 7f9b3f400640 1 -- 172.21.15.43:0/150894436 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a2ff3ef590 con 0x55a2ff4fea60 2024-03-28T22:38:40.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.587+0000 7f9b3f400640 10 client.26234 handle_command_reply: tid=0 2024-03-28T22:38:40.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.587+0000 7f9b633ea740 1 client.26234 shutdown 2024-03-28T22:38:40.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.588+0000 7f9b633ea740 1 -- 172.21.15.43:0/150894436 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a2ff47e380 msgr2=0x55a2ff481510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:40.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.588+0000 7f9b633ea740 1 --2- 172.21.15.43:0/150894436 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a2ff47e380 0x55a2ff481510 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f9b50006010 tx=0x7f9b500118d0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.588+0000 7f9b633ea740 1 -- 172.21.15.43:0/150894436 shutdown_connections 2024-03-28T22:38:40.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.588+0000 7f9b633ea740 1 --2- 172.21.15.43:0/150894436 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55a2ff4fea60 0x55a2ff500e80 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.588+0000 7f9b633ea740 1 --2- 172.21.15.43:0/150894436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a2ff487880 0x55a2ff481f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.588+0000 7f9b633ea740 1 --2- 172.21.15.43:0/150894436 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a2ff47eca0 0x55a2ff481a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.588+0000 7f9b633ea740 1 --2- 172.21.15.43:0/150894436 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a2ff47e380 0x55a2ff481510 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.588+0000 7f9b633ea740 1 -- 172.21.15.43:0/150894436 >> 172.21.15.43:0/150894436 conn(0x55a2ff478ab0 msgr2=0x55a2ff4772e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:40.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.588+0000 7f9b3f400640 0 client.26234 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:38:40.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.588+0000 7f9b633ea740 1 -- 172.21.15.43:0/150894436 shutdown_connections 2024-03-28T22:38:40.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.588+0000 7f9b633ea740 1 -- 172.21.15.43:0/150894436 wait complete. 2024-03-28T22:38:40.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.588+0000 7f9b633ea740 20 client.26234 trim_cache size 0 max 16384 2024-03-28T22:38:40.589 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:38:40.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.589+0000 7f9b60600640 1 -- 172.21.15.43:0/3518045176 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9b400a2090 msgr2=0x7f9b400a4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:40.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.589+0000 7f9b60600640 1 --2- 172.21.15.43:0/3518045176 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9b400a2090 0x7f9b400a4550 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c0cb520 tx=0x7f9b4c0d1d80 comp rx=0 tx=0).stop 2024-03-28T22:38:40.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.589+0000 7f9b60600640 1 -- 172.21.15.43:0/3518045176 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9b581038f0 msgr2=0x7f9b5819a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:40.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.589+0000 7f9b60600640 1 --2- 172.21.15.43:0/3518045176 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9b581038f0 0x7f9b5819a200 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c00d950 tx=0x7f9b4c00de10 comp rx=0 tx=0).stop 2024-03-28T22:38:40.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.589+0000 7f9b60600640 1 -- 172.21.15.43:0/3518045176 shutdown_connections 2024-03-28T22:38:40.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.589+0000 7f9b60600640 1 --2- 172.21.15.43:0/3518045176 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9b400a2090 0x7f9b400a4550 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.589+0000 7f9b60600640 1 --2- 172.21.15.43:0/3518045176 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b58109900 0x7f9b5819eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.589+0000 7f9b60600640 1 --2- 172.21.15.43:0/3518045176 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9b581042b0 0x7f9b5819a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.589+0000 7f9b60600640 1 --2- 172.21.15.43:0/3518045176 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9b581038f0 0x7f9b5819a200 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.589+0000 7f9b60600640 1 -- 172.21.15.43:0/3518045176 >> 172.21.15.43:0/3518045176 conn(0x7f9b580ff3a0 msgr2=0x7f9b580ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:40.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.589+0000 7f9b60600640 1 -- 172.21.15.43:0/3518045176 shutdown_connections 2024-03-28T22:38:40.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.590+0000 7f9b60600640 1 -- 172.21.15.43:0/3518045176 wait complete. 2024-03-28T22:38:40.605 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:38:40.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.881+0000 7f4ba0800640 1 -- 172.21.15.43:0/2821089790 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b98108910 msgr2=0x7f4b9810c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:40.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.881+0000 7f4ba0800640 1 --2- 172.21.15.43:0/2821089790 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b98108910 0x7f4b9810c950 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f4b88009900 tx=0x7f4b8802bc30 comp rx=0 tx=0).stop 2024-03-28T22:38:40.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.881+0000 7f4ba0800640 1 -- 172.21.15.43:0/2821089790 shutdown_connections 2024-03-28T22:38:40.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.881+0000 7f4ba0800640 1 --2- 172.21.15.43:0/2821089790 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b98108910 0x7f4b9810c950 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.881+0000 7f4ba0800640 1 --2- 172.21.15.43:0/2821089790 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b9810d030 0x7f4b98114c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.881+0000 7f4ba0800640 1 --2- 172.21.15.43:0/2821089790 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4b98107f60 0x7f4b98108340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.881+0000 7f4ba0800640 1 -- 172.21.15.43:0/2821089790 >> 172.21.15.43:0/2821089790 conn(0x7f4b98075c90 msgr2=0x7f4b980780b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:40.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.881+0000 7f4ba0800640 1 -- 172.21.15.43:0/2821089790 shutdown_connections 2024-03-28T22:38:40.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.881+0000 7f4ba0800640 1 -- 172.21.15.43:0/2821089790 wait complete. 2024-03-28T22:38:40.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.882+0000 7f4ba0800640 1 Processor -- start 2024-03-28T22:38:40.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.883+0000 7f4ba0800640 1 -- start start 2024-03-28T22:38:40.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.883+0000 7f4ba0800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b98107f60 0x7f4b9819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.883+0000 7f4ba0800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b9810d030 0x7f4b9819eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.883+0000 7f4ba0800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4b981a2f10 0x7f4b981a33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.883+0000 7f4ba0800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4b98118110 con 0x7f4b981a2f10 2024-03-28T22:38:40.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.883+0000 7f4ba0800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4b98117f90 con 0x7f4b98107f60 2024-03-28T22:38:40.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.883+0000 7f4ba0800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4b98118290 con 0x7f4b9810d030 2024-03-28T22:38:40.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.883+0000 7f4b9d400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b9810d030 0x7f4b9819eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.883+0000 7f4b9de00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b98107f60 0x7f4b9819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.883+0000 7f4b9d400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b9810d030 0x7f4b9819eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:36148/0 (socket says 172.21.15.43:36148) 2024-03-28T22:38:40.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.883+0000 7f4b9de00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b98107f60 0x7f4b9819e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:40646/0 (socket says 172.21.15.43:40646) 2024-03-28T22:38:40.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.883+0000 7f4b9d400640 1 -- 172.21.15.43:0/3059802075 learned_addr learned my addr 172.21.15.43:0/3059802075 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:40.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.883+0000 7f4b9e800640 1 --2- 172.21.15.43:0/3059802075 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4b981a2f10 0x7f4b981a33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.884+0000 7f4b9d400640 1 -- 172.21.15.43:0/3059802075 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b98107f60 msgr2=0x7f4b9819e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:40.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.884+0000 7f4b9d400640 1 --2- 172.21.15.43:0/3059802075 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b98107f60 0x7f4b9819e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.884+0000 7f4b9d400640 1 -- 172.21.15.43:0/3059802075 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4b981a2f10 msgr2=0x7f4b981a33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:40.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.884+0000 7f4b9d400640 1 --2- 172.21.15.43:0/3059802075 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4b981a2f10 0x7f4b981a33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.884+0000 7f4b9d400640 1 -- 172.21.15.43:0/3059802075 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b981a3a00 con 0x7f4b9810d030 2024-03-28T22:38:40.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.884+0000 7f4b9e800640 1 --2- 172.21.15.43:0/3059802075 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4b981a2f10 0x7f4b981a33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:38:40.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.884+0000 7f4b9d400640 1 --2- 172.21.15.43:0/3059802075 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b9810d030 0x7f4b9819eb90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4b8802c240 tx=0x7f4b88004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:40.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.884+0000 7f4b8ea00640 1 -- 172.21.15.43:0/3059802075 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b88043020 con 0x7f4b9810d030 2024-03-28T22:38:40.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.885+0000 7f4b8ea00640 1 -- 172.21.15.43:0/3059802075 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b880054e0 con 0x7f4b9810d030 2024-03-28T22:38:40.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.885+0000 7f4ba0800640 1 -- 172.21.15.43:0/3059802075 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b981a3c90 con 0x7f4b9810d030 2024-03-28T22:38:40.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.885+0000 7f4b8ea00640 1 -- 172.21.15.43:0/3059802075 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b88003b80 con 0x7f4b9810d030 2024-03-28T22:38:40.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.885+0000 7f4ba0800640 1 -- 172.21.15.43:0/3059802075 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b981ab4e0 con 0x7f4b9810d030 2024-03-28T22:38:40.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.888+0000 7f4b8ea00640 1 -- 172.21.15.43:0/3059802075 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 27) ==== 136989+0+0 (secure 0 0 0) 0x7f4b88046020 con 0x7f4b9810d030 2024-03-28T22:38:40.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.888+0000 7f4ba35e8740 1 Processor -- start 2024-03-28T22:38:40.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.888+0000 7f4ba35e8740 1 -- start start 2024-03-28T22:38:40.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.889+0000 7f4ba35e8740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bfba567380 0x55bfba567760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.889+0000 7f4ba35e8740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bfba567ca0 0x55bfba570340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.889+0000 7f4ba35e8740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55bfba570880 0x55bfba572c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.889+0000 7f4ba35e8740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55bfba4cb050 con 0x55bfba567ca0 2024-03-28T22:38:40.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.889+0000 7f4ba35e8740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55bfba473ce0 con 0x55bfba567380 2024-03-28T22:38:40.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.889+0000 7f4ba35e8740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55bfba4c70f0 con 0x55bfba570880 2024-03-28T22:38:40.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.889+0000 7f4b9e800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bfba567380 0x55bfba567760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.889+0000 7f4b9e800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bfba567380 0x55bfba567760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:40662/0 (socket says 172.21.15.43:40662) 2024-03-28T22:38:40.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.889+0000 7f4b9e800640 1 -- 172.21.15.43:0/3991264306 learned_addr learned my addr 172.21.15.43:0/3991264306 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:40.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.889+0000 7f4b9d400640 1 --2- 172.21.15.43:0/3991264306 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55bfba570880 0x55bfba572c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.889+0000 7f4b9de00640 1 --2- 172.21.15.43:0/3991264306 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bfba567ca0 0x55bfba570340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.889+0000 7f4b9e800640 1 -- 172.21.15.43:0/3991264306 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55bfba570880 msgr2=0x55bfba572c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:40.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.889+0000 7f4b9e800640 1 --2- 172.21.15.43:0/3991264306 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55bfba570880 0x55bfba572c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.889+0000 7f4b9e800640 1 -- 172.21.15.43:0/3991264306 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bfba567ca0 msgr2=0x55bfba570340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:40.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.889+0000 7f4b9e800640 1 --2- 172.21.15.43:0/3991264306 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bfba567ca0 0x55bfba570340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.889+0000 7f4b9e800640 1 -- 172.21.15.43:0/3991264306 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfba1df3f0 con 0x55bfba567380 2024-03-28T22:38:40.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.889+0000 7f4b9de00640 1 --2- 172.21.15.43:0/3991264306 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bfba567ca0 0x55bfba570340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:40.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.890+0000 7f4b8ea00640 1 --2- 172.21.15.43:0/3059802075 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4b840a2090 0x7f4b840a4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.890+0000 7f4b9e800640 1 --2- 172.21.15.43:0/3991264306 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bfba567380 0x55bfba567760 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f4b94009c20 tx=0x7f4b940117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7c0f9fd1db827ee server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:40.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.890+0000 7f4b83000640 1 -- 172.21.15.43:0/3991264306 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b94012450 con 0x55bfba567380 2024-03-28T22:38:40.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.890+0000 7f4b9de00640 1 --2- 172.21.15.43:0/3059802075 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4b840a2090 0x7f4b840a4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.890+0000 7f4b83000640 1 -- 172.21.15.43:0/3991264306 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b94012ad0 con 0x55bfba567380 2024-03-28T22:38:40.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.890+0000 7f4b83000640 1 -- 172.21.15.43:0/3991264306 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b9400fc10 con 0x55bfba567380 2024-03-28T22:38:40.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.890+0000 7f4b8ea00640 1 -- 172.21.15.43:0/3059802075 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f4b880f08a0 con 0x7f4b9810d030 2024-03-28T22:38:40.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.891+0000 7f4ba35e8740 1 -- 172.21.15.43:0/3991264306 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bfba567380 msgr2=0x55bfba567760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:40.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.891+0000 7f4ba35e8740 1 --2- 172.21.15.43:0/3991264306 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bfba567380 0x55bfba567760 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f4b94009c20 tx=0x7f4b940117b0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.891+0000 7f4b9de00640 1 --2- 172.21.15.43:0/3059802075 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4b840a2090 0x7f4b840a4550 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x55bfba566e50 tx=0x7f4b90009680 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:40.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.891+0000 7f4ba35e8740 1 -- 172.21.15.43:0/3991264306 shutdown_connections 2024-03-28T22:38:40.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.891+0000 7f4ba35e8740 1 --2- 172.21.15.43:0/3991264306 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55bfba570880 0x55bfba572c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.891+0000 7f4ba35e8740 1 --2- 172.21.15.43:0/3991264306 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bfba567ca0 0x55bfba570340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.891+0000 7f4ba35e8740 1 --2- 172.21.15.43:0/3991264306 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bfba567380 0x55bfba567760 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.891+0000 7f4ba35e8740 1 -- 172.21.15.43:0/3991264306 >> 172.21.15.43:0/3991264306 conn(0x55bfba561ab0 msgr2=0x55bfba561e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:40.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.891+0000 7f4ba35e8740 1 -- 172.21.15.43:0/3991264306 shutdown_connections 2024-03-28T22:38:40.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.891+0000 7f4ba35e8740 1 -- 172.21.15.43:0/3991264306 wait complete. 2024-03-28T22:38:40.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.892+0000 7f4ba35e8740 1 -- start start 2024-03-28T22:38:40.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.892+0000 7f4ba35e8740 1 Processor -- start 2024-03-28T22:38:40.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.892+0000 7f4ba35e8740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bfba567380 0x55bfba56d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.892+0000 7f4ba35e8740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55bfba567ca0 0x55bfba56dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.892+0000 7f4ba35e8740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bfba570880 0x55bfba56e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.892+0000 7f4ba35e8740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55bfba573670 con 0x55bfba567380 2024-03-28T22:38:40.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.892+0000 7f4ba35e8740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55bfba4692f0 con 0x55bfba570880 2024-03-28T22:38:40.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.892+0000 7f4ba35e8740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55bfba5737f0 con 0x55bfba567ca0 2024-03-28T22:38:40.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.893+0000 7f4ba35e8740 10 client.0 _ll_register_callbacks cb 0x55bfba1debe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:40.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.893+0000 7f4b9d400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55bfba567ca0 0x55bfba56dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.893+0000 7f4b9e800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bfba570880 0x55bfba56e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.893+0000 7f4b9d400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55bfba567ca0 0x55bfba56dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:36162/0 (socket says 172.21.15.43:36162) 2024-03-28T22:38:40.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.893+0000 7f4b9d400640 1 -- 172.21.15.43:0/1789716798 learned_addr learned my addr 172.21.15.43:0/1789716798 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.893+0000 7f4b9de00640 1 --2- 172.21.15.43:0/1789716798 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bfba567380 0x55bfba56d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.893+0000 7f4b9d400640 1 -- 172.21.15.43:0/1789716798 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bfba570880 msgr2=0x55bfba56e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.893+0000 7f4b9d400640 1 --2- 172.21.15.43:0/1789716798 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bfba570880 0x55bfba56e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.893+0000 7f4b9d400640 1 -- 172.21.15.43:0/1789716798 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bfba567380 msgr2=0x55bfba56d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.893+0000 7f4b9d400640 1 --2- 172.21.15.43:0/1789716798 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bfba567380 0x55bfba56d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.893+0000 7f4b9d400640 1 -- 172.21.15.43:0/1789716798 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfba2a97d0 con 0x55bfba567ca0 2024-03-28T22:38:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.893+0000 7f4b9de00640 1 --2- 172.21.15.43:0/1789716798 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bfba567380 0x55bfba56d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:38:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.893+0000 7f4b9d400640 1 --2- 172.21.15.43:0/1789716798 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55bfba567ca0 0x55bfba56dd80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4b88006010 tx=0x7f4b880035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9312d45fa620a482 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.893+0000 7f4b77400640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:38:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.894+0000 7f4b77400640 1 -- 172.21.15.43:0/1789716798 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b880f3e60 con 0x55bfba567ca0 2024-03-28T22:38:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.894+0000 7f4b77400640 1 -- 172.21.15.43:0/1789716798 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f4b880f0440 con 0x55bfba567ca0 2024-03-28T22:38:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.894+0000 7f4ba35e8740 1 -- 172.21.15.43:0/1789716798 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b94012ad0 con 0x55bfba567ca0 2024-03-28T22:38:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.894+0000 7f4b77400640 1 -- 172.21.15.43:0/1789716798 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4b880bcb20 con 0x55bfba567ca0 2024-03-28T22:38:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.894+0000 7f4ba35e8740 10 client.26194 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.894+0000 7f4ba35e8740 1 -- 172.21.15.43:0/1789716798 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bfba573720 con 0x55bfba567ca0 2024-03-28T22:38:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.894+0000 7f4b77400640 1 -- 172.21.15.43:0/1789716798 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b94012ad0 con 0x55bfba567ca0 2024-03-28T22:38:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.894+0000 7f4b77400640 1 -- 172.21.15.43:0/1789716798 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f4b880c5060 con 0x55bfba567ca0 2024-03-28T22:38:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.894+0000 7f4ba35e8740 10 client.26194 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.894+0000 7f4ba35e8740 10 client.26194 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:38:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.895+0000 7f4ba35e8740 20 client.26194 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.895+0000 7f4ba35e8740 1 --2- 172.21.15.43:0/1789716798 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55bfba580ca0 0x55bfba5830d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.895+0000 7f4ba35e8740 4 client.26194 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:38:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.895+0000 7f4ba35e8740 1 -- 172.21.15.43:0/1789716798 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bfba474680 con 0x55bfba580ca0 2024-03-28T22:38:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.895+0000 7f4b9de00640 1 --2- 172.21.15.43:0/1789716798 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55bfba580ca0 0x55bfba5830d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.895+0000 7f4b9de00640 1 --2- 172.21.15.43:0/1789716798 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55bfba580ca0 0x55bfba5830d0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a88c3498f774394e server_cookie=3a7f7ab224a38e67 in_seq=0 out_seq=0 2024-03-28T22:38:40.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.895+0000 7f4b77400640 10 client.26194 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:40.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.896+0000 7f4b77400640 1 -- 172.21.15.43:0/1789716798 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bfba474680 con 0x55bfba580ca0 2024-03-28T22:38:40.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.896+0000 7f4b77400640 10 client.26194 handle_command_reply: tid=0 2024-03-28T22:38:40.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.896+0000 7f4ba35e8740 1 client.26194 shutdown 2024-03-28T22:38:40.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.897+0000 7f4ba35e8740 1 -- 172.21.15.43:0/1789716798 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55bfba567ca0 msgr2=0x55bfba56dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:40.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.897+0000 7f4ba35e8740 1 --2- 172.21.15.43:0/1789716798 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55bfba567ca0 0x55bfba56dd80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4b88006010 tx=0x7f4b880035f0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.897+0000 7f4ba35e8740 1 -- 172.21.15.43:0/1789716798 shutdown_connections 2024-03-28T22:38:40.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.897+0000 7f4ba35e8740 1 --2- 172.21.15.43:0/1789716798 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55bfba580ca0 0x55bfba5830d0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.897+0000 7f4ba35e8740 1 --2- 172.21.15.43:0/1789716798 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bfba570880 0x55bfba56e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.897+0000 7f4ba35e8740 1 --2- 172.21.15.43:0/1789716798 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55bfba567ca0 0x55bfba56dd80 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.897+0000 7f4ba35e8740 1 --2- 172.21.15.43:0/1789716798 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bfba567380 0x55bfba56d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.897+0000 7f4ba35e8740 1 -- 172.21.15.43:0/1789716798 >> 172.21.15.43:0/1789716798 conn(0x55bfba561ab0 msgr2=0x55bfba563800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:40.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.897+0000 7f4b77400640 0 client.26194 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:40.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.897+0000 7f4ba35e8740 1 -- 172.21.15.43:0/1789716798 shutdown_connections 2024-03-28T22:38:40.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.897+0000 7f4ba35e8740 1 -- 172.21.15.43:0/1789716798 wait complete. 2024-03-28T22:38:40.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.897+0000 7f4ba35e8740 20 client.26194 trim_cache size 0 max 16384 2024-03-28T22:38:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.913+0000 7f4ba35e8740 1 Processor -- start 2024-03-28T22:38:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.913+0000 7f4ba35e8740 1 -- start start 2024-03-28T22:38:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.913+0000 7f4ba35e8740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bfba567380 0x55bfba56ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.913+0000 7f4ba35e8740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bfba567ca0 0x55bfba57c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.913+0000 7f4ba35e8740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55bfba570880 0x55bfba5889e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.913+0000 7f4ba35e8740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4b981b5170 con 0x55bfba567ca0 2024-03-28T22:38:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.913+0000 7f4ba35e8740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4b981b6cb0 con 0x55bfba567380 2024-03-28T22:38:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.913+0000 7f4ba35e8740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4b981b5840 con 0x55bfba570880 2024-03-28T22:38:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.914+0000 7f4b9de00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bfba567380 0x55bfba56ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.914+0000 7f4b9d400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bfba567ca0 0x55bfba57c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.914+0000 7f4b9d400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bfba567ca0 0x55bfba57c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35794/0 (socket says 172.21.15.43:35794) 2024-03-28T22:38:40.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.914+0000 7f4b9e800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55bfba570880 0x55bfba5889e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.914+0000 7f4b9d400640 1 -- 172.21.15.43:0/2651283263 learned_addr learned my addr 172.21.15.43:0/2651283263 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:40.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.914+0000 7f4b9de00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bfba567380 0x55bfba56ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:40680/0 (socket says 172.21.15.43:40680) 2024-03-28T22:38:40.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.914+0000 7f4b9d400640 1 -- 172.21.15.43:0/2651283263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55bfba570880 msgr2=0x55bfba5889e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:40.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.914+0000 7f4b9d400640 1 --2- 172.21.15.43:0/2651283263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55bfba570880 0x55bfba5889e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.914+0000 7f4b9d400640 1 -- 172.21.15.43:0/2651283263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bfba567380 msgr2=0x55bfba56ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:40.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.914+0000 7f4b9d400640 1 --2- 172.21.15.43:0/2651283263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bfba567380 0x55bfba56ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.914+0000 7f4b9d400640 1 -- 172.21.15.43:0/2651283263 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfba583750 con 0x55bfba567ca0 2024-03-28T22:38:40.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.914+0000 7f4b9de00640 1 --2- 172.21.15.43:0/2651283263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bfba567380 0x55bfba56ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:40.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.914+0000 7f4b9d400640 1 --2- 172.21.15.43:0/2651283263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bfba567ca0 0x55bfba57c410 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4b880f06d0 tx=0x7f4b88003f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de183a0a98b7dc7c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:40.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.915+0000 7f4b82600640 1 -- 172.21.15.43:0/2651283263 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b880bc6b0 con 0x55bfba567ca0 2024-03-28T22:38:40.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.915+0000 7f4b82600640 1 -- 172.21.15.43:0/2651283263 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b880edac0 con 0x55bfba567ca0 2024-03-28T22:38:40.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.915+0000 7f4b82600640 1 -- 172.21.15.43:0/2651283263 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b880bc890 con 0x55bfba567ca0 2024-03-28T22:38:40.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.916+0000 7f4ba35e8740 1 -- 172.21.15.43:0/2651283263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bfba567ca0 msgr2=0x55bfba57c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:40.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.916+0000 7f4ba35e8740 1 --2- 172.21.15.43:0/2651283263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bfba567ca0 0x55bfba57c410 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4b880f06d0 tx=0x7f4b88003f30 comp rx=0 tx=0).stop 2024-03-28T22:38:40.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.916+0000 7f4ba35e8740 1 -- 172.21.15.43:0/2651283263 shutdown_connections 2024-03-28T22:38:40.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.916+0000 7f4ba35e8740 1 --2- 172.21.15.43:0/2651283263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55bfba570880 0x55bfba5889e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.916+0000 7f4ba35e8740 1 --2- 172.21.15.43:0/2651283263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bfba567ca0 0x55bfba57c410 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.916+0000 7f4ba35e8740 1 --2- 172.21.15.43:0/2651283263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bfba567380 0x55bfba56ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.916+0000 7f4ba35e8740 1 -- 172.21.15.43:0/2651283263 >> 172.21.15.43:0/2651283263 conn(0x55bfba561ab0 msgr2=0x55bfba5602e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:40.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.916+0000 7f4ba35e8740 1 -- 172.21.15.43:0/2651283263 shutdown_connections 2024-03-28T22:38:40.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.916+0000 7f4ba35e8740 1 -- 172.21.15.43:0/2651283263 wait complete. 2024-03-28T22:38:40.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.917+0000 7f4ba35e8740 1 -- start start 2024-03-28T22:38:40.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.917+0000 7f4ba35e8740 1 Processor -- start 2024-03-28T22:38:40.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.917+0000 7f4ba35e8740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55bfba567380 0x55bfba5749a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.917+0000 7f4ba35e8740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bfba567ca0 0x55bfba574ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.917+0000 7f4ba35e8740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bfba570880 0x55bfba5e6010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.917+0000 7f4ba35e8740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4b981b65e0 con 0x55bfba567ca0 2024-03-28T22:38:40.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.917+0000 7f4ba35e8740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4b981b5f10 con 0x55bfba570880 2024-03-28T22:38:40.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.917+0000 7f4ba35e8740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4b981b3150 con 0x55bfba567380 2024-03-28T22:38:40.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.917+0000 7f4ba35e8740 10 client.0 _ll_register_callbacks cb 0x7f4b8802ca90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:40.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.918+0000 7f4b9de00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55bfba567380 0x55bfba5749a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.918+0000 7f4b9e800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bfba570880 0x55bfba5e6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.918+0000 7f4b9de00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55bfba567380 0x55bfba5749a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:36180/0 (socket says 172.21.15.43:36180) 2024-03-28T22:38:40.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.918+0000 7f4b9e800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bfba570880 0x55bfba5e6010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:40686/0 (socket says 172.21.15.43:40686) 2024-03-28T22:38:40.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.918+0000 7f4b9de00640 1 -- 172.21.15.43:0/1413480993 learned_addr learned my addr 172.21.15.43:0/1413480993 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:40.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.918+0000 7f4b9d400640 1 --2- 172.21.15.43:0/1413480993 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bfba567ca0 0x55bfba574ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.918+0000 7f4b9de00640 1 -- 172.21.15.43:0/1413480993 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bfba570880 msgr2=0x55bfba5e6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:40.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.918+0000 7f4b9de00640 1 --2- 172.21.15.43:0/1413480993 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bfba570880 0x55bfba5e6010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.918+0000 7f4b9de00640 1 -- 172.21.15.43:0/1413480993 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bfba567ca0 msgr2=0x55bfba574ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:40.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.918+0000 7f4b9de00640 1 --2- 172.21.15.43:0/1413480993 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bfba567ca0 0x55bfba574ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.918+0000 7f4b9de00640 1 -- 172.21.15.43:0/1413480993 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfba573550 con 0x55bfba567380 2024-03-28T22:38:40.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.918+0000 7f4b9de00640 1 --2- 172.21.15.43:0/1413480993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55bfba567380 0x55bfba5749a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x55bfba583920 tx=0x7f4b90004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbe003d170b9238e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:40.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.918+0000 7f4b77400640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:38:40.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.919+0000 7f4b77400640 1 -- 172.21.15.43:0/1413480993 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b90029610 con 0x55bfba567380 2024-03-28T22:38:40.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.919+0000 7f4ba35e8740 1 -- 172.21.15.43:0/1413480993 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b880edac0 con 0x55bfba567380 2024-03-28T22:38:40.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.919+0000 7f4b77400640 1 -- 172.21.15.43:0/1413480993 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f4b900297b0 con 0x55bfba567380 2024-03-28T22:38:40.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.919+0000 7f4b77400640 1 -- 172.21.15.43:0/1413480993 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4b90039050 con 0x55bfba567380 2024-03-28T22:38:40.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.919+0000 7f4ba35e8740 10 client.26200 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:40.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.919+0000 7f4ba35e8740 1 -- 172.21.15.43:0/1413480993 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bfba5e7610 con 0x55bfba567380 2024-03-28T22:38:40.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.919+0000 7f4b77400640 1 -- 172.21.15.43:0/1413480993 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b880edac0 con 0x55bfba567380 2024-03-28T22:38:40.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.920+0000 7f4b77400640 1 -- 172.21.15.43:0/1413480993 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f4b9001a040 con 0x55bfba567380 2024-03-28T22:38:40.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.920+0000 7f4ba35e8740 10 client.26200 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:40.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.920+0000 7f4ba35e8740 10 client.26200 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:38:40.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.920+0000 7f4ba35e8740 20 client.26200 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:40.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.920+0000 7f4ba35e8740 1 --2- 172.21.15.43:0/1413480993 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55bfba5e7d10 0x55bfba5ea130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:40.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.920+0000 7f4ba35e8740 4 client.26200 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:38:40.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.920+0000 7f4ba35e8740 1 -- 172.21.15.43:0/1413480993 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bfba4d8590 con 0x55bfba5e7d10 2024-03-28T22:38:40.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.920+0000 7f4b9d400640 1 --2- 172.21.15.43:0/1413480993 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55bfba5e7d10 0x55bfba5ea130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:40.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.920+0000 7f4b9d400640 1 --2- 172.21.15.43:0/1413480993 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55bfba5e7d10 0x55bfba5ea130 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d29db8d59754c28 server_cookie=2f03a96cf6c27408 in_seq=0 out_seq=0 2024-03-28T22:38:40.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.920+0000 7f4b77400640 10 client.26200 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:40.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.922+0000 7f4b77400640 1 -- 172.21.15.43:0/1413480993 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bfba4d8590 con 0x55bfba5e7d10 2024-03-28T22:38:40.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.922+0000 7f4b77400640 10 client.26200 handle_command_reply: tid=0 2024-03-28T22:38:40.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.922+0000 7f4ba35e8740 1 client.26200 shutdown 2024-03-28T22:38:40.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.922+0000 7f4ba35e8740 1 -- 172.21.15.43:0/1413480993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55bfba567380 msgr2=0x55bfba5749a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:40.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.922+0000 7f4ba35e8740 1 --2- 172.21.15.43:0/1413480993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55bfba567380 0x55bfba5749a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x55bfba583920 tx=0x7f4b90004a40 comp rx=0 tx=0).stop 2024-03-28T22:38:40.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.922+0000 7f4ba35e8740 1 -- 172.21.15.43:0/1413480993 shutdown_connections 2024-03-28T22:38:40.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.922+0000 7f4ba35e8740 1 --2- 172.21.15.43:0/1413480993 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55bfba5e7d10 0x55bfba5ea130 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.922+0000 7f4ba35e8740 1 --2- 172.21.15.43:0/1413480993 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bfba570880 0x55bfba5e6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.922+0000 7f4ba35e8740 1 --2- 172.21.15.43:0/1413480993 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55bfba567ca0 0x55bfba574ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.922+0000 7f4b77400640 0 client.26200 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:40.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.922+0000 7f4ba35e8740 1 --2- 172.21.15.43:0/1413480993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55bfba567380 0x55bfba5749a0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.922+0000 7f4ba35e8740 1 -- 172.21.15.43:0/1413480993 >> 172.21.15.43:0/1413480993 conn(0x55bfba561ab0 msgr2=0x55bfba56e070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:40.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.922+0000 7f4ba35e8740 1 -- 172.21.15.43:0/1413480993 shutdown_connections 2024-03-28T22:38:40.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.922+0000 7f4ba35e8740 1 -- 172.21.15.43:0/1413480993 wait complete. 2024-03-28T22:38:40.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.922+0000 7f4ba35e8740 20 client.26200 trim_cache size 0 max 16384 2024-03-28T22:38:40.925 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:38:40.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.923+0000 7f4ba0800640 1 -- 172.21.15.43:0/3059802075 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4b840a2090 msgr2=0x7f4b840a4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:40.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.923+0000 7f4ba0800640 1 --2- 172.21.15.43:0/3059802075 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4b840a2090 0x7f4b840a4550 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x55bfba566e50 tx=0x7f4b90009680 comp rx=0 tx=0).stop 2024-03-28T22:38:40.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.923+0000 7f4ba0800640 1 -- 172.21.15.43:0/3059802075 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b9810d030 msgr2=0x7f4b9819eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:40.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.923+0000 7f4ba0800640 1 --2- 172.21.15.43:0/3059802075 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b9810d030 0x7f4b9819eb90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4b8802c240 tx=0x7f4b88004360 comp rx=0 tx=0).stop 2024-03-28T22:38:40.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.923+0000 7f4ba0800640 1 -- 172.21.15.43:0/3059802075 shutdown_connections 2024-03-28T22:38:40.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.923+0000 7f4ba0800640 1 --2- 172.21.15.43:0/3059802075 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4b840a2090 0x7f4b840a4550 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.923+0000 7f4ba0800640 1 --2- 172.21.15.43:0/3059802075 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4b981a2f10 0x7f4b981a33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.923+0000 7f4ba0800640 1 --2- 172.21.15.43:0/3059802075 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b9810d030 0x7f4b9819eb90 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.923+0000 7f4ba0800640 1 --2- 172.21.15.43:0/3059802075 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b98107f60 0x7f4b9819e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:40.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.923+0000 7f4ba0800640 1 -- 172.21.15.43:0/3059802075 >> 172.21.15.43:0/3059802075 conn(0x7f4b98075c90 msgr2=0x7f4b98077fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:40.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.923+0000 7f4ba0800640 1 -- 172.21.15.43:0/3059802075 shutdown_connections 2024-03-28T22:38:40.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:40.923+0000 7f4ba0800640 1 -- 172.21.15.43:0/3059802075 wait complete. 2024-03-28T22:38:41.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:41 smithi043 ceph-mon[31823]: pgmap v47: 129 pgs: 129 active+clean; 8.4 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 726 KiB/s wr, 194 op/s 2024-03-28T22:38:41.339 INFO:tasks.workunit.client.0.smithi043.stdout: 1 21913 11.31 MB/sec warmup 12 sec latency 24.285 ms 2024-03-28T22:38:41.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:41 smithi172 ceph-mon[36566]: pgmap v47: 129 pgs: 129 active+clean; 8.4 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 726 KiB/s wr, 194 op/s 2024-03-28T22:38:41.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:41 smithi082 ceph-mon[36471]: pgmap v47: 129 pgs: 129 active+clean; 8.4 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 726 KiB/s wr, 194 op/s 2024-03-28T22:38:41.942 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:38:41.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:38:41.942 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:38:42.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.195+0000 7f24ebe00640 1 -- 172.21.15.43:0/2093018655 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24ec1037d0 msgr2=0x7f24ec103c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:42.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.195+0000 7f24ebe00640 1 --2- 172.21.15.43:0/2093018655 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24ec1037d0 0x7f24ec103c50 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f24d8009900 tx=0x7f24d802bc30 comp rx=0 tx=0).stop 2024-03-28T22:38:42.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.195+0000 7f24ebe00640 1 -- 172.21.15.43:0/2093018655 shutdown_connections 2024-03-28T22:38:42.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.195+0000 7f24ebe00640 1 --2- 172.21.15.43:0/2093018655 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24ec1037d0 0x7f24ec103c50 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.195+0000 7f24ebe00640 1 --2- 172.21.15.43:0/2093018655 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24ec104190 0x7f24ec1106a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.195+0000 7f24ebe00640 1 --2- 172.21.15.43:0/2093018655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24ec1097e0 0x7f24ec109bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.195+0000 7f24ebe00640 1 -- 172.21.15.43:0/2093018655 >> 172.21.15.43:0/2093018655 conn(0x7f24ec0ff280 msgr2=0x7f24ec1016a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:42.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.195+0000 7f24ebe00640 1 -- 172.21.15.43:0/2093018655 shutdown_connections 2024-03-28T22:38:42.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.195+0000 7f24ebe00640 1 -- 172.21.15.43:0/2093018655 wait complete. 2024-03-28T22:38:42.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.196+0000 7f24ebe00640 1 Processor -- start 2024-03-28T22:38:42.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.196+0000 7f24ebe00640 1 -- start start 2024-03-28T22:38:42.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.196+0000 7f24ebe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24ec1037d0 0x7f24ec19a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.196+0000 7f24ebe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24ec104190 0x7f24ec19a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.196+0000 7f24ebe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24ec1097e0 0x7f24ec19e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.196+0000 7f24ebe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f24ec113bb0 con 0x7f24ec1037d0 2024-03-28T22:38:42.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.196+0000 7f24ebe00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f24ec113a30 con 0x7f24ec104190 2024-03-28T22:38:42.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.196+0000 7f24ebe00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f24ec113d30 con 0x7f24ec1097e0 2024-03-28T22:38:42.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.197+0000 7f24ea000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24ec104190 0x7f24ec19a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.197+0000 7f24eb400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24ec1097e0 0x7f24ec19e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.197+0000 7f24ea000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24ec104190 0x7f24ec19a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:40700/0 (socket says 172.21.15.43:40700) 2024-03-28T22:38:42.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.197+0000 7f24ea000640 1 -- 172.21.15.43:0/3945808490 learned_addr learned my addr 172.21.15.43:0/3945808490 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:42.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.197+0000 7f24eaa00640 1 --2- 172.21.15.43:0/3945808490 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24ec1037d0 0x7f24ec19a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.197+0000 7f24ea000640 1 -- 172.21.15.43:0/3945808490 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24ec1097e0 msgr2=0x7f24ec19e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:42.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.197+0000 7f24ea000640 1 --2- 172.21.15.43:0/3945808490 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24ec1097e0 0x7f24ec19e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.197+0000 7f24ea000640 1 -- 172.21.15.43:0/3945808490 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24ec1037d0 msgr2=0x7f24ec19a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:42.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.197+0000 7f24ea000640 1 --2- 172.21.15.43:0/3945808490 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24ec1037d0 0x7f24ec19a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.197+0000 7f24ea000640 1 -- 172.21.15.43:0/3945808490 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24ec19f0c0 con 0x7f24ec104190 2024-03-28T22:38:42.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.198+0000 7f24ea000640 1 --2- 172.21.15.43:0/3945808490 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24ec104190 0x7f24ec19a610 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f24d8005940 tx=0x7f24d80042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:42.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.198+0000 7f24e3400640 1 -- 172.21.15.43:0/3945808490 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d8041070 con 0x7f24ec104190 2024-03-28T22:38:42.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.198+0000 7f24e3400640 1 -- 172.21.15.43:0/3945808490 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24d802ccc0 con 0x7f24ec104190 2024-03-28T22:38:42.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.198+0000 7f24ebe00640 1 -- 172.21.15.43:0/3945808490 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24ec19f350 con 0x7f24ec104190 2024-03-28T22:38:42.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.198+0000 7f24e3400640 1 -- 172.21.15.43:0/3945808490 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d8003d80 con 0x7f24ec104190 2024-03-28T22:38:42.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.198+0000 7f24ebe00640 1 -- 172.21.15.43:0/3945808490 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24ec1a6c40 con 0x7f24ec104190 2024-03-28T22:38:42.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.201+0000 7f24f4423740 1 Processor -- start 2024-03-28T22:38:42.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.202+0000 7f24f4423740 1 -- start start 2024-03-28T22:38:42.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.202+0000 7f24e3400640 1 -- 172.21.15.43:0/3945808490 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 27) ==== 136989+0+0 (secure 0 0 0) 0x7f24d8044070 con 0x7f24ec104190 2024-03-28T22:38:42.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.202+0000 7f24f4423740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4b120b170 0x55d4b120b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.202+0000 7f24f4423740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4b120ba90 0x55d4b1214130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.202+0000 7f24f4423740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d4b1214670 0x55d4b1216a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.202+0000 7f24f4423740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d4b116ec40 con 0x55d4b120ba90 2024-03-28T22:38:42.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.202+0000 7f24f4423740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55d4b1117c40 con 0x55d4b120b170 2024-03-28T22:38:42.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.202+0000 7f24f4423740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55d4b116ace0 con 0x55d4b1214670 2024-03-28T22:38:42.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.202+0000 7f24eb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4b120b170 0x55d4b120b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.202+0000 7f24eb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4b120b170 0x55d4b120b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:40706/0 (socket says 172.21.15.43:40706) 2024-03-28T22:38:42.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.202+0000 7f24eb400640 1 -- 172.21.15.43:0/3573848318 learned_addr learned my addr 172.21.15.43:0/3573848318 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:42.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.202+0000 7f24eaa00640 1 --2- 172.21.15.43:0/3573848318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4b120ba90 0x55d4b1214130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.202+0000 7f24ea000640 1 --2- 172.21.15.43:0/3573848318 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d4b1214670 0x55d4b1216a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.203+0000 7f24eb400640 1 -- 172.21.15.43:0/3573848318 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d4b1214670 msgr2=0x55d4b1216a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:42.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.203+0000 7f24eb400640 1 --2- 172.21.15.43:0/3573848318 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d4b1214670 0x55d4b1216a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.203+0000 7f24eb400640 1 -- 172.21.15.43:0/3573848318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4b120ba90 msgr2=0x55d4b1214130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:42.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.203+0000 7f24eb400640 1 --2- 172.21.15.43:0/3573848318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4b120ba90 0x55d4b1214130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.203+0000 7f24eb400640 1 -- 172.21.15.43:0/3573848318 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4b0f4b7e0 con 0x55d4b120b170 2024-03-28T22:38:42.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.203+0000 7f24ea000640 1 --2- 172.21.15.43:0/3573848318 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d4b1214670 0x55d4b1216a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:38:42.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.203+0000 7f24eb400640 1 --2- 172.21.15.43:0/3573848318 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4b120b170 0x55d4b120b550 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f24e4010370 tx=0x7f24e4010830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41ca946a2e826eb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:42.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.203+0000 7f24d7400640 1 -- 172.21.15.43:0/3573848318 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e4016070 con 0x55d4b120b170 2024-03-28T22:38:42.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.203+0000 7f24e3400640 1 --2- 172.21.15.43:0/3945808490 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f24d00a2090 0x7f24d00a4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.203+0000 7f24d7400640 1 -- 172.21.15.43:0/3573848318 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24e4004d20 con 0x55d4b120b170 2024-03-28T22:38:42.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.203+0000 7f24d7400640 1 -- 172.21.15.43:0/3573848318 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e4005480 con 0x55d4b120b170 2024-03-28T22:38:42.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.204+0000 7f24e3400640 1 -- 172.21.15.43:0/3945808490 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f24d80f0530 con 0x7f24ec104190 2024-03-28T22:38:42.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.204+0000 7f24eaa00640 1 --2- 172.21.15.43:0/3945808490 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f24d00a2090 0x7f24d00a4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.204+0000 7f24f4423740 1 -- 172.21.15.43:0/3573848318 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4b120b170 msgr2=0x55d4b120b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:42.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.204+0000 7f24f4423740 1 --2- 172.21.15.43:0/3573848318 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4b120b170 0x55d4b120b550 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f24e4010370 tx=0x7f24e4010830 comp rx=0 tx=0).stop 2024-03-28T22:38:42.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.204+0000 7f24f4423740 1 -- 172.21.15.43:0/3573848318 shutdown_connections 2024-03-28T22:38:42.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.204+0000 7f24f4423740 1 --2- 172.21.15.43:0/3573848318 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d4b1214670 0x55d4b1216a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.204+0000 7f24f4423740 1 --2- 172.21.15.43:0/3573848318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4b120ba90 0x55d4b1214130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.204+0000 7f24f4423740 1 --2- 172.21.15.43:0/3573848318 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4b120b170 0x55d4b120b550 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.204+0000 7f24f4423740 1 -- 172.21.15.43:0/3573848318 >> 172.21.15.43:0/3573848318 conn(0x55d4b12058a0 msgr2=0x55d4b1205c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:42.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.204+0000 7f24f4423740 1 -- 172.21.15.43:0/3573848318 shutdown_connections 2024-03-28T22:38:42.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.205+0000 7f24f4423740 1 -- 172.21.15.43:0/3573848318 wait complete. 2024-03-28T22:38:42.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.205+0000 7f24eaa00640 1 --2- 172.21.15.43:0/3945808490 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f24d00a2090 0x7f24d00a4550 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x55d4b120ac40 tx=0x7f24dc005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:42.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.205+0000 7f24f4423740 1 -- start start 2024-03-28T22:38:42.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.205+0000 7f24f4423740 1 Processor -- start 2024-03-28T22:38:42.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.206+0000 7f24f4423740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d4b120b170 0x55d4b12116d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.206+0000 7f24f4423740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4b120ba90 0x55d4b1211c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.206+0000 7f24f4423740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4b1214670 0x55d4b1212150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.206+0000 7f24f4423740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d4b1217460 con 0x55d4b1214670 2024-03-28T22:38:42.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.206+0000 7f24f4423740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55d4b110d250 con 0x55d4b120ba90 2024-03-28T22:38:42.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.206+0000 7f24f4423740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55d4b12175e0 con 0x55d4b120b170 2024-03-28T22:38:42.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.206+0000 7f24f4423740 10 client.0 _ll_register_callbacks cb 0x55d4b112ee70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:42.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.206+0000 7f24eaa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d4b120b170 0x55d4b12116d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.206+0000 7f24eaa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d4b120b170 0x55d4b12116d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:36202/0 (socket says 172.21.15.43:36202) 2024-03-28T22:38:42.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.206+0000 7f24eaa00640 1 -- 172.21.15.43:0/1045629169 learned_addr learned my addr 172.21.15.43:0/1045629169 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:42.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.206+0000 7f24ea000640 1 --2- 172.21.15.43:0/1045629169 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4b120ba90 0x55d4b1211c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.206+0000 7f24eb400640 1 --2- 172.21.15.43:0/1045629169 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4b1214670 0x55d4b1212150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.206+0000 7f24ea000640 1 -- 172.21.15.43:0/1045629169 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d4b120b170 msgr2=0x55d4b12116d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:42.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.206+0000 7f24ea000640 1 --2- 172.21.15.43:0/1045629169 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d4b120b170 0x55d4b12116d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.206+0000 7f24ea000640 1 -- 172.21.15.43:0/1045629169 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4b1214670 msgr2=0x55d4b1212150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:42.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.206+0000 7f24ea000640 1 --2- 172.21.15.43:0/1045629169 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4b1214670 0x55d4b1212150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.206+0000 7f24ea000640 1 -- 172.21.15.43:0/1045629169 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4b0fabf70 con 0x55d4b120ba90 2024-03-28T22:38:42.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.206+0000 7f24eaa00640 1 --2- 172.21.15.43:0/1045629169 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d4b120b170 0x55d4b12116d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:42.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.207+0000 7f24ea000640 1 --2- 172.21.15.43:0/1045629169 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4b120ba90 0x55d4b1211c10 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f24d80ed7e0 tx=0x7f24d8005680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5713a2e6380a6a0c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:42.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.207+0000 7f24cbe00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:38:42.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.207+0000 7f24cbe00640 1 -- 172.21.15.43:0/1045629169 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d804b040 con 0x55d4b120ba90 2024-03-28T22:38:42.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.207+0000 7f24cbe00640 1 -- 172.21.15.43:0/1045629169 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f24d80bccf0 con 0x55d4b120ba90 2024-03-28T22:38:42.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.207+0000 7f24f4423740 1 -- 172.21.15.43:0/1045629169 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24e4004d20 con 0x55d4b120ba90 2024-03-28T22:38:42.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.207+0000 7f24cbe00640 1 -- 172.21.15.43:0/1045629169 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f24d80b8d30 con 0x55d4b120ba90 2024-03-28T22:38:42.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.207+0000 7f24f4423740 10 client.26264 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:42.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.207+0000 7f24f4423740 1 -- 172.21.15.43:0/1045629169 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d4b0e43170 con 0x55d4b120ba90 2024-03-28T22:38:42.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.208+0000 7f24cbe00640 1 -- 172.21.15.43:0/1045629169 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24e4004d20 con 0x55d4b120ba90 2024-03-28T22:38:42.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.208+0000 7f24cbe00640 1 -- 172.21.15.43:0/1045629169 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f24d80c4cf0 con 0x55d4b120ba90 2024-03-28T22:38:42.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.208+0000 7f24f4423740 10 client.26264 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:42.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.208+0000 7f24f4423740 10 client.26264 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:38:42.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.208+0000 7f24f4423740 20 client.26264 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:42.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.208+0000 7f24f4423740 1 --2- 172.21.15.43:0/1045629169 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d4b1224900 0x55d4b1226d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.208+0000 7f24f4423740 4 client.26264 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:38:42.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.208+0000 7f24f4423740 1 -- 172.21.15.43:0/1045629169 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d4b117c180 con 0x55d4b1224900 2024-03-28T22:38:42.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.208+0000 7f24eaa00640 1 --2- 172.21.15.43:0/1045629169 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d4b1224900 0x55d4b1226d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.209+0000 7f24eaa00640 1 --2- 172.21.15.43:0/1045629169 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d4b1224900 0x55d4b1226d80 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a75a95afbacacc15 server_cookie=9118bc05195f48d2 in_seq=0 out_seq=0 2024-03-28T22:38:42.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.209+0000 7f24cbe00640 10 client.26264 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:42.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.211+0000 7f24cbe00640 1 -- 172.21.15.43:0/1045629169 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d4b117c180 con 0x55d4b1224900 2024-03-28T22:38:42.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.211+0000 7f24cbe00640 10 client.26264 handle_command_reply: tid=0 2024-03-28T22:38:42.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.212+0000 7f24f4423740 1 client.26264 shutdown 2024-03-28T22:38:42.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.212+0000 7f24f4423740 1 -- 172.21.15.43:0/1045629169 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4b120ba90 msgr2=0x55d4b1211c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:42.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.212+0000 7f24f4423740 1 --2- 172.21.15.43:0/1045629169 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4b120ba90 0x55d4b1211c10 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f24d80ed7e0 tx=0x7f24d8005680 comp rx=0 tx=0).stop 2024-03-28T22:38:42.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.212+0000 7f24f4423740 1 -- 172.21.15.43:0/1045629169 shutdown_connections 2024-03-28T22:38:42.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.212+0000 7f24f4423740 1 --2- 172.21.15.43:0/1045629169 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d4b1224900 0x55d4b1226d80 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.212+0000 7f24f4423740 1 --2- 172.21.15.43:0/1045629169 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4b120ba90 0x55d4b1211c10 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.212+0000 7f24f4423740 1 --2- 172.21.15.43:0/1045629169 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4b1214670 0x55d4b1212150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.212+0000 7f24f4423740 1 --2- 172.21.15.43:0/1045629169 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d4b120b170 0x55d4b12116d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.212+0000 7f24f4423740 1 -- 172.21.15.43:0/1045629169 >> 172.21.15.43:0/1045629169 conn(0x55d4b12058a0 msgr2=0x55d4b12075f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:42.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.212+0000 7f24cbe00640 0 client.26264 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:42.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.212+0000 7f24f4423740 1 -- 172.21.15.43:0/1045629169 shutdown_connections 2024-03-28T22:38:42.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.212+0000 7f24f4423740 1 -- 172.21.15.43:0/1045629169 wait complete. 2024-03-28T22:38:42.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.212+0000 7f24f4423740 20 client.26264 trim_cache size 0 max 16384 2024-03-28T22:38:42.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.236+0000 7f24f4423740 1 Processor -- start 2024-03-28T22:38:42.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.236+0000 7f24f4423740 1 -- start start 2024-03-28T22:38:42.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.236+0000 7f24f4423740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4b120b170 0x55d4b1220200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.236+0000 7f24f4423740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4b120ba90 0x55d4b122c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.236+0000 7f24f4423740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d4b1214670 0x55d4b121c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.236+0000 7f24f4423740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f24ec1b0c60 con 0x55d4b120b170 2024-03-28T22:38:42.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.236+0000 7f24f4423740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f24ec1b27a0 con 0x55d4b120ba90 2024-03-28T22:38:42.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.236+0000 7f24f4423740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f24ec1b1330 con 0x55d4b1214670 2024-03-28T22:38:42.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.237+0000 7f24eb400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d4b1214670 0x55d4b121c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.237+0000 7f24eaa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4b120b170 0x55d4b1220200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.237+0000 7f24ea000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4b120ba90 0x55d4b122c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.237+0000 7f24eb400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d4b1214670 0x55d4b121c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:36206/0 (socket says 172.21.15.43:36206) 2024-03-28T22:38:42.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.237+0000 7f24eb400640 1 -- 172.21.15.43:0/1552746593 learned_addr learned my addr 172.21.15.43:0/1552746593 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:42.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.237+0000 7f24eb400640 1 -- 172.21.15.43:0/1552746593 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4b120ba90 msgr2=0x55d4b122c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:42.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.237+0000 7f24eb400640 1 --2- 172.21.15.43:0/1552746593 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4b120ba90 0x55d4b122c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.237+0000 7f24eb400640 1 -- 172.21.15.43:0/1552746593 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4b120b170 msgr2=0x55d4b1220200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:42.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.237+0000 7f24eb400640 1 --2- 172.21.15.43:0/1552746593 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4b120b170 0x55d4b1220200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.237+0000 7f24eb400640 1 -- 172.21.15.43:0/1552746593 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4b1220740 con 0x55d4b1214670 2024-03-28T22:38:42.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.237+0000 7f24eb400640 1 --2- 172.21.15.43:0/1552746593 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d4b1214670 0x55d4b121c820 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f24e4010d30 tx=0x7f24e4005bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3205cab96c9f520 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:42.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.238+0000 7f24d6a00640 1 -- 172.21.15.43:0/1552746593 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e4016040 con 0x55d4b1214670 2024-03-28T22:38:42.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.238+0000 7f24d6a00640 1 -- 172.21.15.43:0/1552746593 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24e4003b70 con 0x55d4b1214670 2024-03-28T22:38:42.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.238+0000 7f24d6a00640 1 -- 172.21.15.43:0/1552746593 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e4012860 con 0x55d4b1214670 2024-03-28T22:38:42.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.239+0000 7f24f4423740 1 -- 172.21.15.43:0/1552746593 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d4b1214670 msgr2=0x55d4b121c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:42.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.239+0000 7f24f4423740 1 --2- 172.21.15.43:0/1552746593 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d4b1214670 0x55d4b121c820 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f24e4010d30 tx=0x7f24e4005bd0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.239+0000 7f24f4423740 1 -- 172.21.15.43:0/1552746593 shutdown_connections 2024-03-28T22:38:42.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.239+0000 7f24f4423740 1 --2- 172.21.15.43:0/1552746593 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4b120ba90 0x55d4b122c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.239+0000 7f24f4423740 1 --2- 172.21.15.43:0/1552746593 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d4b1214670 0x55d4b121c820 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.239+0000 7f24f4423740 1 --2- 172.21.15.43:0/1552746593 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4b120b170 0x55d4b1220200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.239+0000 7f24f4423740 1 -- 172.21.15.43:0/1552746593 >> 172.21.15.43:0/1552746593 conn(0x55d4b12058a0 msgr2=0x55d4b1211690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:42.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.239+0000 7f24f4423740 1 -- 172.21.15.43:0/1552746593 shutdown_connections 2024-03-28T22:38:42.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.239+0000 7f24f4423740 1 -- 172.21.15.43:0/1552746593 wait complete. 2024-03-28T22:38:42.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.239+0000 7f24f4423740 1 -- start start 2024-03-28T22:38:42.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.240+0000 7f24f4423740 1 Processor -- start 2024-03-28T22:38:42.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.240+0000 7f24f4423740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d4b120b170 0x55d4b12198c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.240+0000 7f24f4423740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4b120ba90 0x55d4b1219e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.240+0000 7f24f4423740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4b1214670 0x55d4b12934e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.240+0000 7f24f4423740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f24ec1b20d0 con 0x55d4b120ba90 2024-03-28T22:38:42.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.240+0000 7f24f4423740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f24ec1b1a00 con 0x55d4b1214670 2024-03-28T22:38:42.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.240+0000 7f24f4423740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f24ec1aec40 con 0x55d4b120b170 2024-03-28T22:38:42.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.240+0000 7f24f4423740 10 client.0 _ll_register_callbacks cb 0x7f24d80f3cd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:42.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.240+0000 7f24eaa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d4b120b170 0x55d4b12198c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.240+0000 7f24ea000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4b120ba90 0x55d4b1219e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.240+0000 7f24eb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4b1214670 0x55d4b12934e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.240+0000 7f24eaa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d4b120b170 0x55d4b12198c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:36218/0 (socket says 172.21.15.43:36218) 2024-03-28T22:38:42.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.240+0000 7f24eaa00640 1 -- 172.21.15.43:0/2208783808 learned_addr learned my addr 172.21.15.43:0/2208783808 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:42.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.241+0000 7f24eb400640 1 -- 172.21.15.43:0/2208783808 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d4b120b170 msgr2=0x55d4b12198c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:42.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.241+0000 7f24eb400640 1 --2- 172.21.15.43:0/2208783808 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d4b120b170 0x55d4b12198c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.241+0000 7f24eb400640 1 -- 172.21.15.43:0/2208783808 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4b120ba90 msgr2=0x55d4b1219e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:42.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.241+0000 7f24eb400640 1 --2- 172.21.15.43:0/2208783808 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4b120ba90 0x55d4b1219e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.241+0000 7f24eb400640 1 -- 172.21.15.43:0/2208783808 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4b1293a20 con 0x55d4b1214670 2024-03-28T22:38:42.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.241+0000 7f24eaa00640 1 --2- 172.21.15.43:0/2208783808 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d4b120b170 0x55d4b12198c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:42.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.241+0000 7f24ea000640 1 --2- 172.21.15.43:0/2208783808 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4b120ba90 0x55d4b1219e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:42.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.241+0000 7f24eb400640 1 --2- 172.21.15.43:0/2208783808 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4b1214670 0x55d4b12934e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x55d4b1221170 tx=0x7f24e4005fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=619432448a9141a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:42.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.241+0000 7f24cbe00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:38:42.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.241+0000 7f24cbe00640 1 -- 172.21.15.43:0/2208783808 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e4016070 con 0x55d4b1214670 2024-03-28T22:38:42.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.242+0000 7f24f4423740 1 -- 172.21.15.43:0/2208783808 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24e4003b70 con 0x55d4b1214670 2024-03-28T22:38:42.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.242+0000 7f24cbe00640 1 -- 172.21.15.43:0/2208783808 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f24e4003970 con 0x55d4b1214670 2024-03-28T22:38:42.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.242+0000 7f24cbe00640 1 -- 172.21.15.43:0/2208783808 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f24e4037050 con 0x55d4b1214670 2024-03-28T22:38:42.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.242+0000 7f24f4423740 10 client.26270 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:42.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.242+0000 7f24f4423740 1 -- 172.21.15.43:0/2208783808 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d4b1294d60 con 0x55d4b1214670 2024-03-28T22:38:42.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.242+0000 7f24cbe00640 1 -- 172.21.15.43:0/2208783808 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24e4003b70 con 0x55d4b1214670 2024-03-28T22:38:42.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.242+0000 7f24cbe00640 1 -- 172.21.15.43:0/2208783808 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f24e400f040 con 0x55d4b1214670 2024-03-28T22:38:42.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.242+0000 7f24f4423740 10 client.26270 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:42.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.242+0000 7f24f4423740 10 client.26270 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:38:42.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.242+0000 7f24f4423740 20 client.26270 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:42.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.242+0000 7f24f4423740 1 --2- 172.21.15.43:0/2208783808 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d4b1295180 0x55d4b1297570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.242+0000 7f24f4423740 4 client.26270 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:38:42.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.242+0000 7f24f4423740 1 -- 172.21.15.43:0/2208783808 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d4b11764d0 con 0x55d4b1295180 2024-03-28T22:38:42.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.243+0000 7f24eaa00640 1 --2- 172.21.15.43:0/2208783808 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d4b1295180 0x55d4b1297570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.243+0000 7f24eaa00640 1 --2- 172.21.15.43:0/2208783808 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d4b1295180 0x55d4b1297570 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=864272cecff73073 server_cookie=69e78f2f9b36671b in_seq=0 out_seq=0 2024-03-28T22:38:42.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.243+0000 7f24cbe00640 10 client.26270 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:42.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.244+0000 7f24cbe00640 1 -- 172.21.15.43:0/2208783808 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d4b11764d0 con 0x55d4b1295180 2024-03-28T22:38:42.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.244+0000 7f24cbe00640 10 client.26270 handle_command_reply: tid=0 2024-03-28T22:38:42.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.244+0000 7f24f4423740 1 client.26270 shutdown 2024-03-28T22:38:42.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.244+0000 7f24f4423740 1 -- 172.21.15.43:0/2208783808 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4b1214670 msgr2=0x55d4b12934e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:42.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.244+0000 7f24f4423740 1 --2- 172.21.15.43:0/2208783808 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4b1214670 0x55d4b12934e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x55d4b1221170 tx=0x7f24e4005fa0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.244+0000 7f24f4423740 1 -- 172.21.15.43:0/2208783808 shutdown_connections 2024-03-28T22:38:42.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.244+0000 7f24f4423740 1 --2- 172.21.15.43:0/2208783808 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d4b1295180 0x55d4b1297570 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.244+0000 7f24f4423740 1 --2- 172.21.15.43:0/2208783808 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4b1214670 0x55d4b12934e0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.244+0000 7f24f4423740 1 --2- 172.21.15.43:0/2208783808 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d4b120ba90 0x55d4b1219e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.244+0000 7f24f4423740 1 --2- 172.21.15.43:0/2208783808 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d4b120b170 0x55d4b12198c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.244+0000 7f24f4423740 1 -- 172.21.15.43:0/2208783808 >> 172.21.15.43:0/2208783808 conn(0x55d4b12058a0 msgr2=0x55d4b1204900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:42.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.244+0000 7f24cbe00640 0 client.26270 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:42.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.245+0000 7f24f4423740 1 -- 172.21.15.43:0/2208783808 shutdown_connections 2024-03-28T22:38:42.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.245+0000 7f24f4423740 1 -- 172.21.15.43:0/2208783808 wait complete. 2024-03-28T22:38:42.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.245+0000 7f24f4423740 20 client.26270 trim_cache size 0 max 16384 2024-03-28T22:38:42.246 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:38:42.246 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-03-28T22:38:42.246 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "fda80417-69c9-4cfa-9a9b-f3fa52abe301", 2024-03-28T22:38:42.246 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-03-28T22:38:42.246 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:38:42.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.246+0000 7f24ebe00640 1 -- 172.21.15.43:0/3945808490 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f24d00a2090 msgr2=0x7f24d00a4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:42.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.246+0000 7f24ebe00640 1 --2- 172.21.15.43:0/3945808490 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f24d00a2090 0x7f24d00a4550 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x55d4b120ac40 tx=0x7f24dc005f90 comp rx=0 tx=0).stop 2024-03-28T22:38:42.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.246+0000 7f24ebe00640 1 -- 172.21.15.43:0/3945808490 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24ec104190 msgr2=0x7f24ec19a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:42.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.246+0000 7f24ebe00640 1 --2- 172.21.15.43:0/3945808490 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24ec104190 0x7f24ec19a610 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f24d8005940 tx=0x7f24d80042f0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.246+0000 7f24ebe00640 1 -- 172.21.15.43:0/3945808490 shutdown_connections 2024-03-28T22:38:42.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.246+0000 7f24ebe00640 1 --2- 172.21.15.43:0/3945808490 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f24d00a2090 0x7f24d00a4550 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.246+0000 7f24ebe00640 1 --2- 172.21.15.43:0/3945808490 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24ec104190 0x7f24ec19a610 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.246+0000 7f24ebe00640 1 --2- 172.21.15.43:0/3945808490 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24ec1097e0 0x7f24ec19e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.246+0000 7f24ebe00640 1 --2- 172.21.15.43:0/3945808490 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24ec1037d0 0x7f24ec19a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.246+0000 7f24ebe00640 1 -- 172.21.15.43:0/3945808490 >> 172.21.15.43:0/3945808490 conn(0x7f24ec0ff280 msgr2=0x7f24ec0ffa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:42.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.246+0000 7f24ebe00640 1 -- 172.21.15.43:0/3945808490 shutdown_connections 2024-03-28T22:38:42.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.246+0000 7f24ebe00640 1 -- 172.21.15.43:0/3945808490 wait complete. 2024-03-28T22:38:42.261 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:38:42.262 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:38:42.340 INFO:tasks.workunit.client.0.smithi043.stdout: 1 23921 11.06 MB/sec warmup 13 sec latency 12.849 ms 2024-03-28T22:38:42.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.476+0000 7fe096a00640 1 -- 172.21.15.43:0/1431525413 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe0901088b0 msgr2=0x7fe09010c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:42.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.476+0000 7fe096a00640 1 --2- 172.21.15.43:0/1431525413 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe0901088b0 0x7fe09010c870 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe07c009900 tx=0x7fe07c02bc50 comp rx=0 tx=0).stop 2024-03-28T22:38:42.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.476+0000 7fe096a00640 1 -- 172.21.15.43:0/1431525413 shutdown_connections 2024-03-28T22:38:42.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.476+0000 7fe096a00640 1 --2- 172.21.15.43:0/1431525413 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe0901088b0 0x7fe09010c870 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.476+0000 7fe096a00640 1 --2- 172.21.15.43:0/1431525413 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe09010cf50 0x7fe090114b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.476+0000 7fe096a00640 1 --2- 172.21.15.43:0/1431525413 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe090107f00 0x7fe0901082e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.476+0000 7fe096a00640 1 -- 172.21.15.43:0/1431525413 >> 172.21.15.43:0/1431525413 conn(0x7fe090077570 msgr2=0x7fe090106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:42.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.476+0000 7fe096a00640 1 -- 172.21.15.43:0/1431525413 shutdown_connections 2024-03-28T22:38:42.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.476+0000 7fe096a00640 1 -- 172.21.15.43:0/1431525413 wait complete. 2024-03-28T22:38:42.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.477+0000 7fe096a00640 1 Processor -- start 2024-03-28T22:38:42.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.477+0000 7fe096a00640 1 -- start start 2024-03-28T22:38:42.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.478+0000 7fe096a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe090107f00 0x7fe09019e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.478+0000 7fe096a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe09010cf50 0x7fe09019ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.478+0000 7fe096a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe0901a2d90 0x7fe0901a3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.478+0000 7fe096a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe090118050 con 0x7fe090107f00 2024-03-28T22:38:42.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.478+0000 7fe096a00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe090117ed0 con 0x7fe0901a2d90 2024-03-28T22:38:42.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.478+0000 7fe096a00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe0901181d0 con 0x7fe09010cf50 2024-03-28T22:38:42.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.478+0000 7fe096000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe0901a2d90 0x7fe0901a3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.478+0000 7fe094c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe09010cf50 0x7fe09019ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.478+0000 7fe096000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe0901a2d90 0x7fe0901a3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:40750/0 (socket says 172.21.15.43:40750) 2024-03-28T22:38:42.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.478+0000 7fe095600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe090107f00 0x7fe09019e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.478+0000 7fe096000640 1 -- 172.21.15.43:0/2404741684 learned_addr learned my addr 172.21.15.43:0/2404741684 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:42.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.479+0000 7fe096000640 1 -- 172.21.15.43:0/2404741684 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe09010cf50 msgr2=0x7fe09019ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:42.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.479+0000 7fe096000640 1 --2- 172.21.15.43:0/2404741684 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe09010cf50 0x7fe09019ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.479+0000 7fe096000640 1 -- 172.21.15.43:0/2404741684 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe090107f00 msgr2=0x7fe09019e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:42.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.479+0000 7fe096000640 1 --2- 172.21.15.43:0/2404741684 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe090107f00 0x7fe09019e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.479+0000 7fe096000640 1 -- 172.21.15.43:0/2404741684 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0901a3880 con 0x7fe0901a2d90 2024-03-28T22:38:42.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.479+0000 7fe096000640 1 --2- 172.21.15.43:0/2404741684 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe0901a2d90 0x7fe0901a3240 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fe08000e970 tx=0x7fe08000ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:42.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.479+0000 7fe08e000640 1 -- 172.21.15.43:0/2404741684 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe08000ccc0 con 0x7fe0901a2d90 2024-03-28T22:38:42.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.479+0000 7fe08e000640 1 -- 172.21.15.43:0/2404741684 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe08000ce60 con 0x7fe0901a2d90 2024-03-28T22:38:42.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.479+0000 7fe08e000640 1 -- 172.21.15.43:0/2404741684 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe080010430 con 0x7fe0901a2d90 2024-03-28T22:38:42.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.479+0000 7fe096a00640 1 -- 172.21.15.43:0/2404741684 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0901a3b70 con 0x7fe0901a2d90 2024-03-28T22:38:42.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.479+0000 7fe096a00640 1 -- 172.21.15.43:0/2404741684 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0901ab2e0 con 0x7fe0901a2d90 2024-03-28T22:38:42.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.482+0000 7fe08e000640 1 -- 172.21.15.43:0/2404741684 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 27) ==== 136989+0+0 (secure 0 0 0) 0x7fe080005020 con 0x7fe0901a2d90 2024-03-28T22:38:42.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.483+0000 7fe09ad84740 1 Processor -- start 2024-03-28T22:38:42.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.483+0000 7fe09ad84740 1 -- start start 2024-03-28T22:38:42.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.483+0000 7fe09ad84740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56066943edb0 0x56066943db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.483+0000 7fe09ad84740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56066943e0d0 0x56066943e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.483+0000 7fe09ad84740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560669447440 0x560669449830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.483+0000 7fe09ad84740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5606693a2100 con 0x56066943e0d0 2024-03-28T22:38:42.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.483+0000 7fe09ad84740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x56066934aba0 con 0x560669447440 2024-03-28T22:38:42.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.483+0000 7fe09ad84740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x56066939e1a0 con 0x56066943edb0 2024-03-28T22:38:42.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.483+0000 7fe094c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56066943edb0 0x56066943db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.483+0000 7fe095600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56066943e0d0 0x56066943e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.484+0000 7fe094c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56066943edb0 0x56066943db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:36256/0 (socket says 172.21.15.43:36256) 2024-03-28T22:38:42.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.484+0000 7fe095600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56066943e0d0 0x56066943e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35866/0 (socket says 172.21.15.43:35866) 2024-03-28T22:38:42.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.484+0000 7fe094c00640 1 -- 172.21.15.43:0/710994632 learned_addr learned my addr 172.21.15.43:0/710994632 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:42.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.484+0000 7fe094c00640 1 --2- 172.21.15.43:0/710994632 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560669447440 0x560669449830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.484+0000 7fe095600640 1 -- 172.21.15.43:0/710994632 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56066943edb0 msgr2=0x56066943db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:42.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.484+0000 7fe095600640 1 --2- 172.21.15.43:0/710994632 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56066943edb0 0x56066943db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.484+0000 7fe095600640 1 -- 172.21.15.43:0/710994632 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560669447440 msgr2=0x560669449830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:42.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.484+0000 7fe095600640 1 --2- 172.21.15.43:0/710994632 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560669447440 0x560669449830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.484+0000 7fe095600640 1 -- 172.21.15.43:0/710994632 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56066937b1a0 con 0x56066943e0d0 2024-03-28T22:38:42.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.484+0000 7fe095600640 1 --2- 172.21.15.43:0/710994632 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56066943e0d0 0x56066943e550 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe088009fa0 tx=0x7fe0880118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78bcbeef7c4583d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:42.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.484+0000 7fe08e000640 1 --2- 172.21.15.43:0/2404741684 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe0740a2010 0x7fe0740a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.485+0000 7fe08e000640 1 -- 172.21.15.43:0/2404741684 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fe0800ceac0 con 0x7fe0901a2d90 2024-03-28T22:38:42.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.485+0000 7fe094c00640 1 --2- 172.21.15.43:0/2404741684 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe0740a2010 0x7fe0740a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.485+0000 7fe086600640 1 -- 172.21.15.43:0/710994632 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe088003c30 con 0x56066943e0d0 2024-03-28T22:38:42.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.485+0000 7fe086600640 1 -- 172.21.15.43:0/710994632 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe088003dd0 con 0x56066943e0d0 2024-03-28T22:38:42.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.485+0000 7fe086600640 1 -- 172.21.15.43:0/710994632 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe08800fa30 con 0x56066943e0d0 2024-03-28T22:38:42.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.485+0000 7fe09ad84740 1 -- 172.21.15.43:0/710994632 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56066943e0d0 msgr2=0x56066943e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:42.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.485+0000 7fe09ad84740 1 --2- 172.21.15.43:0/710994632 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56066943e0d0 0x56066943e550 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe088009fa0 tx=0x7fe0880118d0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.485+0000 7fe094c00640 1 --2- 172.21.15.43:0/2404741684 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe0740a2010 0x7fe0740a44d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe07c02c840 tx=0x7fe07c000bc0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:42.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.486+0000 7fe09ad84740 1 -- 172.21.15.43:0/710994632 shutdown_connections 2024-03-28T22:38:42.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.486+0000 7fe09ad84740 1 --2- 172.21.15.43:0/710994632 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560669447440 0x560669449830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.486+0000 7fe09ad84740 1 --2- 172.21.15.43:0/710994632 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56066943e0d0 0x56066943e550 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.486+0000 7fe09ad84740 1 --2- 172.21.15.43:0/710994632 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56066943edb0 0x56066943db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.486+0000 7fe09ad84740 1 -- 172.21.15.43:0/710994632 >> 172.21.15.43:0/710994632 conn(0x560669404a10 msgr2=0x560669438b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:42.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.486+0000 7fe09ad84740 1 -- 172.21.15.43:0/710994632 shutdown_connections 2024-03-28T22:38:42.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.486+0000 7fe09ad84740 1 -- 172.21.15.43:0/710994632 wait complete. 2024-03-28T22:38:42.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.486+0000 7fe09ad84740 1 -- start start 2024-03-28T22:38:42.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.486+0000 7fe09ad84740 1 Processor -- start 2024-03-28T22:38:42.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.487+0000 7fe09ad84740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56066943e0d0 0x5606694443d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.487+0000 7fe09ad84740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56066943edb0 0x560669444910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.487+0000 7fe09ad84740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560669447440 0x560669444e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.487+0000 7fe09ad84740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56066944a300 con 0x56066943edb0 2024-03-28T22:38:42.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.487+0000 7fe09ad84740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x560669340250 con 0x560669447440 2024-03-28T22:38:42.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.487+0000 7fe09ad84740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x56066944a480 con 0x56066943e0d0 2024-03-28T22:38:42.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.487+0000 7fe09ad84740 10 client.0 _ll_register_callbacks cb 0x5606690b5be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:42.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.487+0000 7fe095600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56066943e0d0 0x5606694443d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.487+0000 7fe096000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560669447440 0x560669444e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.487+0000 7fe094c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56066943edb0 0x560669444910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.487+0000 7fe095600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56066943e0d0 0x5606694443d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:36270/0 (socket says 172.21.15.43:36270) 2024-03-28T22:38:42.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.487+0000 7fe095600640 1 -- 172.21.15.43:0/3655149941 learned_addr learned my addr 172.21.15.43:0/3655149941 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:42.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.487+0000 7fe096000640 1 -- 172.21.15.43:0/3655149941 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56066943e0d0 msgr2=0x5606694443d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:42.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.487+0000 7fe096000640 1 --2- 172.21.15.43:0/3655149941 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56066943e0d0 0x5606694443d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.488+0000 7fe096000640 1 -- 172.21.15.43:0/3655149941 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56066943edb0 msgr2=0x560669444910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:42.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.488+0000 7fe096000640 1 --2- 172.21.15.43:0/3655149941 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56066943edb0 0x560669444910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.488+0000 7fe096000640 1 -- 172.21.15.43:0/3655149941 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56066935c960 con 0x560669447440 2024-03-28T22:38:42.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.488+0000 7fe095600640 1 --2- 172.21.15.43:0/3655149941 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56066943e0d0 0x5606694443d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:42.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.488+0000 7fe096000640 1 --2- 172.21.15.43:0/3655149941 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560669447440 0x560669444e50 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fe0800cbcf0 tx=0x7fe08009b680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c39e251a78d7b70 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:42.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.488+0000 7fe07aa00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:38:42.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.488+0000 7fe07aa00640 1 -- 172.21.15.43:0/3655149941 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0800b2020 con 0x560669447440 2024-03-28T22:38:42.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.488+0000 7fe07aa00640 1 -- 172.21.15.43:0/3655149941 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fe0800a31d0 con 0x560669447440 2024-03-28T22:38:42.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.488+0000 7fe09ad84740 1 -- 172.21.15.43:0/3655149941 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe088003dd0 con 0x560669447440 2024-03-28T22:38:42.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.488+0000 7fe07aa00640 1 -- 172.21.15.43:0/3655149941 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe08007e050 con 0x560669447440 2024-03-28T22:38:42.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.488+0000 7fe09ad84740 10 client.26282 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:42.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.488+0000 7fe09ad84740 1 -- 172.21.15.43:0/3655149941 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5606691807d0 con 0x560669447440 2024-03-28T22:38:42.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.489+0000 7fe07aa00640 1 -- 172.21.15.43:0/3655149941 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe088003dd0 con 0x560669447440 2024-03-28T22:38:42.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.489+0000 7fe07aa00640 1 -- 172.21.15.43:0/3655149941 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe080079ce0 con 0x560669447440 2024-03-28T22:38:42.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.489+0000 7fe09ad84740 10 client.26282 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:42.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.489+0000 7fe09ad84740 10 client.26282 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:38:42.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.489+0000 7fe09ad84740 20 client.26282 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:42.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.489+0000 7fe09ad84740 1 --2- 172.21.15.43:0/3655149941 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560669457910 0x560669459db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.489+0000 7fe09ad84740 4 client.26282 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:38:42.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.489+0000 7fe09ad84740 1 -- 172.21.15.43:0/3655149941 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5606693af640 con 0x560669457910 2024-03-28T22:38:42.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.489+0000 7fe095600640 1 --2- 172.21.15.43:0/3655149941 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560669457910 0x560669459db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.490+0000 7fe095600640 1 --2- 172.21.15.43:0/3655149941 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560669457910 0x560669459db0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85a3662ebe87a990 server_cookie=6419cd9623aa9cb in_seq=0 out_seq=0 2024-03-28T22:38:42.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.490+0000 7fe07aa00640 10 client.26282 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:42.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.492+0000 7fe07aa00640 1 -- 172.21.15.43:0/3655149941 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5606693af640 con 0x560669457910 2024-03-28T22:38:42.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.492+0000 7fe07aa00640 10 client.26282 handle_command_reply: tid=0 2024-03-28T22:38:42.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.492+0000 7fe09ad84740 1 client.26282 shutdown 2024-03-28T22:38:42.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.492+0000 7fe09ad84740 1 -- 172.21.15.43:0/3655149941 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560669447440 msgr2=0x560669444e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:42.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.492+0000 7fe09ad84740 1 --2- 172.21.15.43:0/3655149941 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560669447440 0x560669444e50 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fe0800cbcf0 tx=0x7fe08009b680 comp rx=0 tx=0).stop 2024-03-28T22:38:42.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.492+0000 7fe09ad84740 1 -- 172.21.15.43:0/3655149941 shutdown_connections 2024-03-28T22:38:42.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.492+0000 7fe09ad84740 1 --2- 172.21.15.43:0/3655149941 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560669457910 0x560669459db0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.492+0000 7fe09ad84740 1 --2- 172.21.15.43:0/3655149941 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560669447440 0x560669444e50 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.492+0000 7fe09ad84740 1 --2- 172.21.15.43:0/3655149941 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56066943edb0 0x560669444910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.492+0000 7fe07aa00640 0 client.26282 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:42.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.492+0000 7fe09ad84740 1 --2- 172.21.15.43:0/3655149941 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56066943e0d0 0x5606694443d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.492+0000 7fe09ad84740 1 -- 172.21.15.43:0/3655149941 >> 172.21.15.43:0/3655149941 conn(0x560669404a10 msgr2=0x56066943a870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:42.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.492+0000 7fe09ad84740 1 -- 172.21.15.43:0/3655149941 shutdown_connections 2024-03-28T22:38:42.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.492+0000 7fe09ad84740 1 -- 172.21.15.43:0/3655149941 wait complete. 2024-03-28T22:38:42.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.492+0000 7fe09ad84740 20 client.26282 trim_cache size 0 max 16384 2024-03-28T22:38:42.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.515+0000 7fe09ad84740 1 Processor -- start 2024-03-28T22:38:42.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.515+0000 7fe09ad84740 1 -- start start 2024-03-28T22:38:42.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.515+0000 7fe09ad84740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56066943e0d0 0x560669446b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.515+0000 7fe09ad84740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56066943edb0 0x5606694530e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.515+0000 7fe09ad84740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560669447440 0x560669443530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.515+0000 7fe09ad84740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe0901b4f80 con 0x56066943e0d0 2024-03-28T22:38:42.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.515+0000 7fe09ad84740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe0901b6ac0 con 0x560669447440 2024-03-28T22:38:42.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.515+0000 7fe09ad84740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe0901b5650 con 0x56066943edb0 2024-03-28T22:38:42.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.515+0000 7fe094c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56066943edb0 0x5606694530e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.515+0000 7fe094c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56066943edb0 0x5606694530e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:36284/0 (socket says 172.21.15.43:36284) 2024-03-28T22:38:42.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.515+0000 7fe094c00640 1 -- 172.21.15.43:0/1708369649 learned_addr learned my addr 172.21.15.43:0/1708369649 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:42.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.516+0000 7fe095600640 1 --2- 172.21.15.43:0/1708369649 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56066943e0d0 0x560669446b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.516+0000 7fe096000640 1 --2- 172.21.15.43:0/1708369649 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560669447440 0x560669443530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.516+0000 7fe094c00640 1 -- 172.21.15.43:0/1708369649 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560669447440 msgr2=0x560669443530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:42.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.516+0000 7fe094c00640 1 --2- 172.21.15.43:0/1708369649 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560669447440 0x560669443530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.516+0000 7fe094c00640 1 -- 172.21.15.43:0/1708369649 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56066943e0d0 msgr2=0x560669446b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:42.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.516+0000 7fe094c00640 1 --2- 172.21.15.43:0/1708369649 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56066943e0d0 0x560669446b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.516+0000 7fe094c00640 1 -- 172.21.15.43:0/1708369649 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606690b63f0 con 0x56066943edb0 2024-03-28T22:38:42.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.516+0000 7fe096000640 1 --2- 172.21.15.43:0/1708369649 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560669447440 0x560669443530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:38:42.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.516+0000 7fe094c00640 1 --2- 172.21.15.43:0/1708369649 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56066943edb0 0x5606694530e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe07c036040 tx=0x7fe07c02bc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfc114f0fb8ad6ab server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:42.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.517+0000 7fe085c00640 1 -- 172.21.15.43:0/1708369649 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe07c0046c0 con 0x56066943edb0 2024-03-28T22:38:42.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.517+0000 7fe085c00640 1 -- 172.21.15.43:0/1708369649 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe07c004ee0 con 0x56066943edb0 2024-03-28T22:38:42.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.517+0000 7fe085c00640 1 -- 172.21.15.43:0/1708369649 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe07c003850 con 0x56066943edb0 2024-03-28T22:38:42.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.517+0000 7fe09ad84740 1 -- 172.21.15.43:0/1708369649 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56066943edb0 msgr2=0x5606694530e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:42.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.517+0000 7fe09ad84740 1 --2- 172.21.15.43:0/1708369649 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56066943edb0 0x5606694530e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe07c036040 tx=0x7fe07c02bc50 comp rx=0 tx=0).stop 2024-03-28T22:38:42.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.517+0000 7fe09ad84740 1 -- 172.21.15.43:0/1708369649 shutdown_connections 2024-03-28T22:38:42.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.517+0000 7fe09ad84740 1 --2- 172.21.15.43:0/1708369649 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560669447440 0x560669443530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.518+0000 7fe09ad84740 1 --2- 172.21.15.43:0/1708369649 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56066943edb0 0x5606694530e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.518+0000 7fe09ad84740 1 --2- 172.21.15.43:0/1708369649 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56066943e0d0 0x560669446b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.518+0000 7fe09ad84740 1 -- 172.21.15.43:0/1708369649 >> 172.21.15.43:0/1708369649 conn(0x560669404a10 msgr2=0x560669437470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:42.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.518+0000 7fe09ad84740 1 -- 172.21.15.43:0/1708369649 shutdown_connections 2024-03-28T22:38:42.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.518+0000 7fe09ad84740 1 -- 172.21.15.43:0/1708369649 wait complete. 2024-03-28T22:38:42.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.518+0000 7fe09ad84740 1 -- start start 2024-03-28T22:38:42.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.518+0000 7fe09ad84740 1 Processor -- start 2024-03-28T22:38:42.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.519+0000 7fe09ad84740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56066943e0d0 0x56066944f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.519+0000 7fe09ad84740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56066943edb0 0x560669440c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.519+0000 7fe09ad84740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560669447440 0x560669441180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.519+0000 7fe09ad84740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe0901b63f0 con 0x56066943e0d0 2024-03-28T22:38:42.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.519+0000 7fe09ad84740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe0901b5d20 con 0x56066943edb0 2024-03-28T22:38:42.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.519+0000 7fe09ad84740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe0901b2f60 con 0x560669447440 2024-03-28T22:38:42.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.519+0000 7fe09ad84740 10 client.0 _ll_register_callbacks cb 0x7fe080004920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:42.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.519+0000 7fe094c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56066943edb0 0x560669440c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.519+0000 7fe096000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560669447440 0x560669441180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.519+0000 7fe094c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56066943edb0 0x560669440c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:40780/0 (socket says 172.21.15.43:40780) 2024-03-28T22:38:42.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.519+0000 7fe094c00640 1 -- 172.21.15.43:0/1657049001 learned_addr learned my addr 172.21.15.43:0/1657049001 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:42.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.519+0000 7fe095600640 1 --2- 172.21.15.43:0/1657049001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56066943e0d0 0x56066944f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.519+0000 7fe094c00640 1 -- 172.21.15.43:0/1657049001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560669447440 msgr2=0x560669441180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:42.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.519+0000 7fe094c00640 1 --2- 172.21.15.43:0/1657049001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560669447440 0x560669441180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.519+0000 7fe094c00640 1 -- 172.21.15.43:0/1657049001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56066943e0d0 msgr2=0x56066944f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:42.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.519+0000 7fe094c00640 1 --2- 172.21.15.43:0/1657049001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56066943e0d0 0x56066944f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.519+0000 7fe094c00640 1 -- 172.21.15.43:0/1657049001 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606690760f0 con 0x56066943edb0 2024-03-28T22:38:42.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.519+0000 7fe095600640 1 --2- 172.21.15.43:0/1657049001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56066943e0d0 0x56066944f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:38:42.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.519+0000 7fe096000640 1 --2- 172.21.15.43:0/1657049001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560669447440 0x560669441180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:42.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.520+0000 7fe094c00640 1 --2- 172.21.15.43:0/1657049001 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56066943edb0 0x560669440c40 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe07c005560 tx=0x7fe07c005590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87438d0d2ab6c30c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:42.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.520+0000 7fe07aa00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:38:42.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.520+0000 7fe07aa00640 1 -- 172.21.15.43:0/1657049001 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe07c0048d0 con 0x56066943edb0 2024-03-28T22:38:42.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.520+0000 7fe09ad84740 1 -- 172.21.15.43:0/1657049001 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe07c004ee0 con 0x56066943edb0 2024-03-28T22:38:42.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.520+0000 7fe07aa00640 1 -- 172.21.15.43:0/1657049001 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fe07c004a70 con 0x56066943edb0 2024-03-28T22:38:42.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.520+0000 7fe07aa00640 1 -- 172.21.15.43:0/1657049001 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe07c067050 con 0x56066943edb0 2024-03-28T22:38:42.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.520+0000 7fe09ad84740 10 client.26288 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:42.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.521+0000 7fe09ad84740 1 -- 172.21.15.43:0/1657049001 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5606694bd9b0 con 0x56066943edb0 2024-03-28T22:38:42.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.521+0000 7fe07aa00640 1 -- 172.21.15.43:0/1657049001 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe07c004ee0 con 0x56066943edb0 2024-03-28T22:38:42.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.521+0000 7fe07aa00640 1 -- 172.21.15.43:0/1657049001 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe07c0344b0 con 0x56066943edb0 2024-03-28T22:38:42.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.521+0000 7fe09ad84740 10 client.26288 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:42.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.521+0000 7fe09ad84740 10 client.26288 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:38:42.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.521+0000 7fe09ad84740 20 client.26288 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:42.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.521+0000 7fe09ad84740 1 --2- 172.21.15.43:0/1657049001 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5606694be0f0 0x5606694c0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:42.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.521+0000 7fe09ad84740 4 client.26288 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:38:42.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.521+0000 7fe09ad84740 1 -- 172.21.15.43:0/1657049001 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5606693a9990 con 0x5606694be0f0 2024-03-28T22:38:42.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.521+0000 7fe095600640 1 --2- 172.21.15.43:0/1657049001 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5606694be0f0 0x5606694c0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:42.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.522+0000 7fe07aa00640 10 client.26288 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:42.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.522+0000 7fe095600640 1 --2- 172.21.15.43:0/1657049001 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5606694be0f0 0x5606694c0510 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=682a1106a2e0f278 server_cookie=2367ea5be8f73890 in_seq=0 out_seq=0 2024-03-28T22:38:42.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.522+0000 7fe07aa00640 1 -- 172.21.15.43:0/1657049001 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5606693a9990 con 0x5606694be0f0 2024-03-28T22:38:42.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.522+0000 7fe07aa00640 10 client.26288 handle_command_reply: tid=0 2024-03-28T22:38:42.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.522+0000 7fe09ad84740 1 client.26288 shutdown 2024-03-28T22:38:42.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.522+0000 7fe09ad84740 1 -- 172.21.15.43:0/1657049001 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56066943edb0 msgr2=0x560669440c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:42.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.522+0000 7fe09ad84740 1 --2- 172.21.15.43:0/1657049001 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56066943edb0 0x560669440c40 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe07c005560 tx=0x7fe07c005590 comp rx=0 tx=0).stop 2024-03-28T22:38:42.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.522+0000 7fe09ad84740 1 -- 172.21.15.43:0/1657049001 shutdown_connections 2024-03-28T22:38:42.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.522+0000 7fe09ad84740 1 --2- 172.21.15.43:0/1657049001 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5606694be0f0 0x5606694c0510 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.522+0000 7fe09ad84740 1 --2- 172.21.15.43:0/1657049001 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56066943edb0 0x560669440c40 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.522+0000 7fe09ad84740 1 --2- 172.21.15.43:0/1657049001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560669447440 0x560669441180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.522+0000 7fe07aa00640 0 client.26288 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:42.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.522+0000 7fe09ad84740 1 --2- 172.21.15.43:0/1657049001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56066943e0d0 0x56066944f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.522+0000 7fe09ad84740 1 -- 172.21.15.43:0/1657049001 >> 172.21.15.43:0/1657049001 conn(0x560669404a10 msgr2=0x560669444390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:42.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.523+0000 7fe09ad84740 1 -- 172.21.15.43:0/1657049001 shutdown_connections 2024-03-28T22:38:42.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.523+0000 7fe09ad84740 1 -- 172.21.15.43:0/1657049001 wait complete. 2024-03-28T22:38:42.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.523+0000 7fe09ad84740 20 client.26288 trim_cache size 0 max 16384 2024-03-28T22:38:42.525 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:38:42.526 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:38:42.526 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-03-28T22:38:42.526 INFO:teuthology.orchestra.run.smithi043.stdout: "fda80417-69c9-4cfa-9a9b-f3fa52abe301": { 2024-03-28T22:38:42.526 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-03-28T22:38:42.526 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "fda80417-69c9-4cfa-9a9b-f3fa52abe301", 2024-03-28T22:38:42.526 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-03-28T22:38:42.526 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:38:42.526 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:38:42.526 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:38:42.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.524+0000 7fe096a00640 1 -- 172.21.15.43:0/2404741684 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe0740a2010 msgr2=0x7fe0740a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:42.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.524+0000 7fe096a00640 1 --2- 172.21.15.43:0/2404741684 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe0740a2010 0x7fe0740a44d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe07c02c840 tx=0x7fe07c000bc0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.524+0000 7fe096a00640 1 -- 172.21.15.43:0/2404741684 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe0901a2d90 msgr2=0x7fe0901a3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:42.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.524+0000 7fe096a00640 1 --2- 172.21.15.43:0/2404741684 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe0901a2d90 0x7fe0901a3240 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fe08000e970 tx=0x7fe08000ee30 comp rx=0 tx=0).stop 2024-03-28T22:38:42.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.524+0000 7fe096a00640 1 -- 172.21.15.43:0/2404741684 shutdown_connections 2024-03-28T22:38:42.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.524+0000 7fe096a00640 1 --2- 172.21.15.43:0/2404741684 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe0740a2010 0x7fe0740a44d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.524+0000 7fe096a00640 1 --2- 172.21.15.43:0/2404741684 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe0901a2d90 0x7fe0901a3240 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.524+0000 7fe096a00640 1 --2- 172.21.15.43:0/2404741684 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe09010cf50 0x7fe09019ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.524+0000 7fe096a00640 1 --2- 172.21.15.43:0/2404741684 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe090107f00 0x7fe09019e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:42.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.524+0000 7fe096a00640 1 -- 172.21.15.43:0/2404741684 >> 172.21.15.43:0/2404741684 conn(0x7fe090077570 msgr2=0x7fe0900791c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:42.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.524+0000 7fe096a00640 1 -- 172.21.15.43:0/2404741684 shutdown_connections 2024-03-28T22:38:42.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:38:42.524+0000 7fe096a00640 1 -- 172.21.15.43:0/2404741684 wait complete. 2024-03-28T22:38:42.541 INFO:tasks.cephfs.filesystem:scrub status for tag:fda80417-69c9-4cfa-9a9b-f3fa52abe301 - {'path': '/', 'tag': 'fda80417-69c9-4cfa-9a9b-f3fa52abe301', 'options': 'recursive,force'} 2024-03-28T22:38:43.340 INFO:tasks.workunit.client.0.smithi043.stdout: 1 25551 11.07 MB/sec warmup 14 sec latency 18.534 ms 2024-03-28T22:38:43.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:43 smithi172 ceph-mon[36566]: pgmap v48: 129 pgs: 129 active+clean; 8.4 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 726 KiB/s wr, 194 op/s 2024-03-28T22:38:43.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:43 smithi082 ceph-mon[36471]: pgmap v48: 129 pgs: 129 active+clean; 8.4 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 726 KiB/s wr, 194 op/s 2024-03-28T22:38:43.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:43 smithi043 ceph-mon[31823]: pgmap v48: 129 pgs: 129 active+clean; 8.4 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 726 KiB/s wr, 194 op/s 2024-03-28T22:38:44.340 INFO:tasks.workunit.client.0.smithi043.stdout: 1 27406 10.85 MB/sec warmup 15 sec latency 16.589 ms 2024-03-28T22:38:44.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:44 smithi172 ceph-mon[36566]: scrub queued for path: / 2024-03-28T22:38:44.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:44 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:38:44.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:44 smithi172 ceph-mon[36566]: scrub summary: active paths [/] 2024-03-28T22:38:44.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:44 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:38:44.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:44 smithi172 ceph-mon[36566]: scrub summary: idle 2024-03-28T22:38:44.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:44 smithi082 ceph-mon[36471]: scrub queued for path: / 2024-03-28T22:38:44.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:44 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:38:44.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:44 smithi082 ceph-mon[36471]: scrub summary: active paths [/] 2024-03-28T22:38:44.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:44 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:38:44.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:44 smithi082 ceph-mon[36471]: scrub summary: idle 2024-03-28T22:38:44.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:44 smithi043 ceph-mon[31823]: scrub queued for path: / 2024-03-28T22:38:44.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:44 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:38:44.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:44 smithi043 ceph-mon[31823]: scrub summary: active paths [/] 2024-03-28T22:38:44.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:44 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:38:44.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:44 smithi043 ceph-mon[31823]: scrub summary: idle 2024-03-28T22:38:45.340 INFO:tasks.workunit.client.0.smithi043.stdout: 1 29022 10.87 MB/sec warmup 16 sec latency 10.184 ms 2024-03-28T22:38:45.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:45 smithi172 ceph-mon[36566]: pgmap v49: 129 pgs: 129 active+clean; 13 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 1.2 MiB/s wr, 265 op/s 2024-03-28T22:38:45.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:45 smithi172 ceph-mon[36566]: mgrmap e28: x(active, since 92s), standbys: y, z 2024-03-28T22:38:45.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:45 smithi082 ceph-mon[36471]: pgmap v49: 129 pgs: 129 active+clean; 13 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 1.2 MiB/s wr, 265 op/s 2024-03-28T22:38:45.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:45 smithi082 ceph-mon[36471]: mgrmap e28: x(active, since 92s), standbys: y, z 2024-03-28T22:38:45.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:45 smithi043 ceph-mon[31823]: pgmap v49: 129 pgs: 129 active+clean; 13 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 1.2 MiB/s wr, 265 op/s 2024-03-28T22:38:45.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:45 smithi043 ceph-mon[31823]: mgrmap e28: x(active, since 92s), standbys: y, z 2024-03-28T22:38:46.340 INFO:tasks.workunit.client.0.smithi043.stdout: 1 30993 10.70 MB/sec warmup 17 sec latency 20.203 ms 2024-03-28T22:38:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.797+0000 7f3340c00640 1 -- 172.21.15.43:0/3110607406 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f333c108610 msgr2=0x7f333c114a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.797+0000 7f3340c00640 1 --2- 172.21.15.43:0/3110607406 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f333c108610 0x7f333c114a20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3334009cd0 tx=0x7f333402c0e0 comp rx=0 tx=0).stop 2024-03-28T22:38:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.797+0000 7f3340c00640 1 -- 172.21.15.43:0/3110607406 shutdown_connections 2024-03-28T22:38:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.797+0000 7f3340c00640 1 --2- 172.21.15.43:0/3110607406 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f333c107c90 0x7f333c1080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.797+0000 7f3340c00640 1 --2- 172.21.15.43:0/3110607406 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f333c108610 0x7f333c114a20 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.797+0000 7f3340c00640 1 --2- 172.21.15.43:0/3110607406 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f333c10dbf0 0x7f333c10dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:46.796 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.797+0000 7f3340c00640 1 -- 172.21.15.43:0/3110607406 >> 172.21.15.43:0/3110607406 conn(0x7f333c077530 msgr2=0x7f333c106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:46.796 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.797+0000 7f3340c00640 1 -- 172.21.15.43:0/3110607406 shutdown_connections 2024-03-28T22:38:46.796 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.797+0000 7f3340c00640 1 -- 172.21.15.43:0/3110607406 wait complete. 2024-03-28T22:38:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.798+0000 7f3340c00640 1 Processor -- start 2024-03-28T22:38:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.798+0000 7f3340c00640 1 -- start start 2024-03-28T22:38:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.798+0000 7f3340c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f333c107c90 0x7f333c19e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.798+0000 7f3340c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f333c108610 0x7f333c19e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.798+0000 7f3340c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f333c10dbf0 0x7f333c1a2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.799+0000 7f3340c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f333c117f60 con 0x7f333c10dbf0 2024-03-28T22:38:46.799 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.799+0000 7f3340c00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f333c117de0 con 0x7f333c108610 2024-03-28T22:38:46.799 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.799+0000 7f3340c00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f333c1180e0 con 0x7f333c107c90 2024-03-28T22:38:46.799 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.799+0000 7f333b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f333c107c90 0x7f333c19e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:46.799 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.799+0000 7f333aa00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f333c108610 0x7f333c19e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:46.799 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.799+0000 7f333b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f333c107c90 0x7f333c19e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:41850/0 (socket says 172.21.15.43:41850) 2024-03-28T22:38:46.799 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.799+0000 7f333b400640 1 -- 172.21.15.43:0/3253693281 learned_addr learned my addr 172.21.15.43:0/3253693281 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:46.799 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.799+0000 7f333aa00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f333c108610 0x7f333c19e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:50602/0 (socket says 172.21.15.43:50602) 2024-03-28T22:38:46.799 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.799+0000 7f333be00640 1 --2- 172.21.15.43:0/3253693281 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f333c10dbf0 0x7f333c1a2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:46.799 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.799+0000 7f333b400640 1 -- 172.21.15.43:0/3253693281 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f333c108610 msgr2=0x7f333c19e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:46.799 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.799+0000 7f333b400640 1 --2- 172.21.15.43:0/3253693281 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f333c108610 0x7f333c19e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:46.799 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.799+0000 7f333b400640 1 -- 172.21.15.43:0/3253693281 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f333c10dbf0 msgr2=0x7f333c1a2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:46.799 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.799+0000 7f333b400640 1 --2- 172.21.15.43:0/3253693281 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f333c10dbf0 0x7f333c1a2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:46.799 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.799+0000 7f333b400640 1 -- 172.21.15.43:0/3253693281 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f333c1a3450 con 0x7f333c107c90 2024-03-28T22:38:46.800 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.799+0000 7f333be00640 1 --2- 172.21.15.43:0/3253693281 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f333c10dbf0 0x7f333c1a2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:38:46.800 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.799+0000 7f333aa00640 1 --2- 172.21.15.43:0/3253693281 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f333c108610 0x7f333c19e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:46.800 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.800+0000 7f333b400640 1 --2- 172.21.15.43:0/3253693281 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f333c107c90 0x7f333c19e460 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f332c00e9f0 tx=0x7f332c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:46.800 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.800+0000 7f3333e00640 1 -- 172.21.15.43:0/3253693281 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f332c00cd80 con 0x7f333c107c90 2024-03-28T22:38:46.800 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.800+0000 7f3340c00640 1 -- 172.21.15.43:0/3253693281 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f333c1a3740 con 0x7f333c107c90 2024-03-28T22:38:46.800 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.800+0000 7f3333e00640 1 -- 172.21.15.43:0/3253693281 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f332c010910 con 0x7f333c107c90 2024-03-28T22:38:46.800 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.800+0000 7f3333e00640 1 -- 172.21.15.43:0/3253693281 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f332c0188f0 con 0x7f333c107c90 2024-03-28T22:38:46.800 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.800+0000 7f3340c00640 1 -- 172.21.15.43:0/3253693281 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f333c1aafd0 con 0x7f333c107c90 2024-03-28T22:38:46.803 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.803+0000 7f3344e4a740 1 Processor -- start 2024-03-28T22:38:46.803 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.803+0000 7f3333e00640 1 -- 172.21.15.43:0/3253693281 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f332c010ac0 con 0x7f333c107c90 2024-03-28T22:38:46.803 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.803+0000 7f3344e4a740 1 -- start start 2024-03-28T22:38:46.803 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.804+0000 7f3344e4a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5592e0b3e570 0x5592e0b3e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:46.803 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.804+0000 7f3344e4a740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592e0b3ee90 0x5592e0b47720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:46.803 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.804+0000 7f3344e4a740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5592e0b47c60 0x5592e0b4a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:46.804 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.804+0000 7f3344e4a740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5592e0aa1640 con 0x5592e0b3ee90 2024-03-28T22:38:46.804 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.804+0000 7f3344e4a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5592e0a4a810 con 0x5592e0b3e570 2024-03-28T22:38:46.804 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.804+0000 7f3344e4a740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5592e0a9d6e0 con 0x5592e0b47c60 2024-03-28T22:38:46.804 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.804+0000 7f333be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5592e0b3e570 0x5592e0b3e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:46.804 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.804+0000 7f333be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5592e0b3e570 0x5592e0b3e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:50614/0 (socket says 172.21.15.43:50614) 2024-03-28T22:38:46.804 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.804+0000 7f333be00640 1 -- 172.21.15.43:0/4195547882 learned_addr learned my addr 172.21.15.43:0/4195547882 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:46.804 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.804+0000 7f333b400640 1 --2- 172.21.15.43:0/4195547882 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592e0b3ee90 0x5592e0b47720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:46.804 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.804+0000 7f333aa00640 1 --2- 172.21.15.43:0/4195547882 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5592e0b47c60 0x5592e0b4a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:46.804 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.805+0000 7f333be00640 1 -- 172.21.15.43:0/4195547882 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5592e0b47c60 msgr2=0x5592e0b4a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:46.804 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.805+0000 7f333be00640 1 --2- 172.21.15.43:0/4195547882 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5592e0b47c60 0x5592e0b4a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:46.804 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.805+0000 7f333be00640 1 -- 172.21.15.43:0/4195547882 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592e0b3ee90 msgr2=0x5592e0b47720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:46.804 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.805+0000 7f333be00640 1 --2- 172.21.15.43:0/4195547882 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592e0b3ee90 0x5592e0b47720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:46.804 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.805+0000 7f333be00640 1 -- 172.21.15.43:0/4195547882 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592e07b6210 con 0x5592e0b3e570 2024-03-28T22:38:46.805 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.805+0000 7f333be00640 1 --2- 172.21.15.43:0/4195547882 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5592e0b3e570 0x5592e0b3e950 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f3334009ca0 tx=0x7f33340323c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95c166a3416fbaeb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:46.805 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.805+0000 7f3333e00640 1 --2- 172.21.15.43:0/3253693281 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f33200a1ee0 0x7f33200a43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:46.805 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.805+0000 7f3333e00640 1 -- 172.21.15.43:0/3253693281 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f332c0ce940 con 0x7f333c107c90 2024-03-28T22:38:46.805 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.805+0000 7f333b400640 1 --2- 172.21.15.43:0/3253693281 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f33200a1ee0 0x7f33200a43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:46.805 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.805+0000 7f3327e00640 1 -- 172.21.15.43:0/4195547882 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3334043020 con 0x5592e0b3e570 2024-03-28T22:38:46.805 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.805+0000 7f3327e00640 1 -- 172.21.15.43:0/4195547882 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3334032c70 con 0x5592e0b3e570 2024-03-28T22:38:46.805 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.805+0000 7f3327e00640 1 -- 172.21.15.43:0/4195547882 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3334039040 con 0x5592e0b3e570 2024-03-28T22:38:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.806+0000 7f3344e4a740 1 -- 172.21.15.43:0/4195547882 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5592e0b3e570 msgr2=0x5592e0b3e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.806+0000 7f333b400640 1 --2- 172.21.15.43:0/3253693281 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f33200a1ee0 0x7f33200a43a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x5592e0b3e020 tx=0x7f332c0d2040 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.806+0000 7f3344e4a740 1 --2- 172.21.15.43:0/4195547882 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5592e0b3e570 0x5592e0b3e950 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f3334009ca0 tx=0x7f33340323c0 comp rx=0 tx=0).stop 2024-03-28T22:38:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.806+0000 7f3344e4a740 1 -- 172.21.15.43:0/4195547882 shutdown_connections 2024-03-28T22:38:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.806+0000 7f3344e4a740 1 --2- 172.21.15.43:0/4195547882 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5592e0b47c60 0x5592e0b4a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.806+0000 7f3344e4a740 1 --2- 172.21.15.43:0/4195547882 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592e0b3ee90 0x5592e0b47720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.806+0000 7f3344e4a740 1 --2- 172.21.15.43:0/4195547882 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5592e0b3e570 0x5592e0b3e950 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.806+0000 7f3344e4a740 1 -- 172.21.15.43:0/4195547882 >> 172.21.15.43:0/4195547882 conn(0x5592e0b38e90 msgr2=0x5592e0b39270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.806+0000 7f3344e4a740 1 -- 172.21.15.43:0/4195547882 shutdown_connections 2024-03-28T22:38:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.807+0000 7f3344e4a740 1 -- 172.21.15.43:0/4195547882 wait complete. 2024-03-28T22:38:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.807+0000 7f3344e4a740 1 -- start start 2024-03-28T22:38:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.807+0000 7f3344e4a740 1 Processor -- start 2024-03-28T22:38:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.808+0000 7f3344e4a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5592e0b3e570 0x5592e0b44c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.808+0000 7f3344e4a740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592e0b3ee90 0x5592e0b451d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.808+0000 7f3344e4a740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5592e0b47c60 0x5592e0b45710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.808+0000 7f3344e4a740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5592e0b4a830 con 0x5592e0b3ee90 2024-03-28T22:38:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.808+0000 7f3344e4a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5592e0a3fd00 con 0x5592e0b3e570 2024-03-28T22:38:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.808+0000 7f3344e4a740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5592e0b4a9b0 con 0x5592e0b47c60 2024-03-28T22:38:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.808+0000 7f3344e4a740 10 client.0 _ll_register_callbacks cb 0x5592e0a2a490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.808+0000 7f333be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5592e0b3e570 0x5592e0b44c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.808+0000 7f333aa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592e0b3ee90 0x5592e0b451d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.808+0000 7f333be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5592e0b3e570 0x5592e0b44c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:50624/0 (socket says 172.21.15.43:50624) 2024-03-28T22:38:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.808+0000 7f333be00640 1 -- 172.21.15.43:0/2450016761 learned_addr learned my addr 172.21.15.43:0/2450016761 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.808+0000 7f333be00640 1 --2- 172.21.15.43:0/2450016761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5592e0b47c60 0x5592e0b45710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:46.811 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.808+0000 7f333aa00640 1 -- 172.21.15.43:0/2450016761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5592e0b47c60 msgr2=0x5592e0b45710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:46.811 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.808+0000 7f333aa00640 1 --2- 172.21.15.43:0/2450016761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5592e0b47c60 0x5592e0b45710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:46.811 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.808+0000 7f333aa00640 1 -- 172.21.15.43:0/2450016761 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5592e0b3e570 msgr2=0x5592e0b44c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:46.811 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.808+0000 7f333aa00640 1 --2- 172.21.15.43:0/2450016761 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5592e0b3e570 0x5592e0b44c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:46.811 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.808+0000 7f333aa00640 1 -- 172.21.15.43:0/2450016761 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592e0882620 con 0x5592e0b3ee90 2024-03-28T22:38:46.811 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.808+0000 7f333be00640 1 --2- 172.21.15.43:0/2450016761 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5592e0b3e570 0x5592e0b44c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:46.811 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.809+0000 7f333aa00640 1 --2- 172.21.15.43:0/2450016761 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592e0b3ee90 0x5592e0b451d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f333c10d380 tx=0x7f3328012e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ba75b73e22e929d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:46.811 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.809+0000 7f3324c00640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:38:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.809+0000 7f3324c00640 1 -- 172.21.15.43:0/2450016761 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f332800f9e0 con 0x5592e0b3ee90 2024-03-28T22:38:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.809+0000 7f3344e4a740 1 -- 172.21.15.43:0/2450016761 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3334032c70 con 0x5592e0b3ee90 2024-03-28T22:38:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.809+0000 7f3324c00640 1 -- 172.21.15.43:0/2450016761 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f332800fb80 con 0x5592e0b3ee90 2024-03-28T22:38:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.809+0000 7f3324c00640 1 -- 172.21.15.43:0/2450016761 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3328025050 con 0x5592e0b3ee90 2024-03-28T22:38:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.810+0000 7f3344e4a740 10 client.16122 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.810+0000 7f3344e4a740 1 -- 172.21.15.43:0/2450016761 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5592e0b567f0 con 0x5592e0b3ee90 2024-03-28T22:38:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.810+0000 7f3324c00640 1 -- 172.21.15.43:0/2450016761 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3334032c70 con 0x5592e0b3ee90 2024-03-28T22:38:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.810+0000 7f3324c00640 1 -- 172.21.15.43:0/2450016761 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f3328003b80 con 0x5592e0b3ee90 2024-03-28T22:38:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.810+0000 7f3344e4a740 10 client.16122 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.810+0000 7f3344e4a740 10 client.16122 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:38:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.810+0000 7f3344e4a740 20 client.16122 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.810+0000 7f3344e4a740 1 --2- 172.21.15.43:0/2450016761 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5592e0b57cb0 0x5592e0b5a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.810+0000 7f3344e4a740 4 client.16122 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:38:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.810+0000 7f3344e4a740 1 -- 172.21.15.43:0/2450016761 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5592e0ab5030 con 0x5592e0b57cb0 2024-03-28T22:38:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.811+0000 7f333be00640 1 --2- 172.21.15.43:0/2450016761 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5592e0b57cb0 0x5592e0b5a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.811+0000 7f333be00640 1 --2- 172.21.15.43:0/2450016761 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5592e0b57cb0 0x5592e0b5a0a0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=290943939a447c2 server_cookie=676c0c8bafa8b527 in_seq=0 out_seq=0 2024-03-28T22:38:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.811+0000 7f3324c00640 10 client.16122 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:46.814 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.813+0000 7f3324c00640 1 -- 172.21.15.43:0/2450016761 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5592e0ab5030 con 0x5592e0b57cb0 2024-03-28T22:38:46.814 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.813+0000 7f3324c00640 10 client.16122 handle_command_reply: tid=0 2024-03-28T22:38:46.814 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.813+0000 7f3344e4a740 1 client.16122 shutdown 2024-03-28T22:38:46.814 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.813+0000 7f3344e4a740 1 -- 172.21.15.43:0/2450016761 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592e0b3ee90 msgr2=0x5592e0b451d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:46.814 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.813+0000 7f3344e4a740 1 --2- 172.21.15.43:0/2450016761 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592e0b3ee90 0x5592e0b451d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f333c10d380 tx=0x7f3328012e70 comp rx=0 tx=0).stop 2024-03-28T22:38:46.814 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.813+0000 7f3344e4a740 1 -- 172.21.15.43:0/2450016761 shutdown_connections 2024-03-28T22:38:46.815 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.813+0000 7f3344e4a740 1 --2- 172.21.15.43:0/2450016761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5592e0b47c60 0x5592e0b45710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:46.815 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.813+0000 7f3344e4a740 1 --2- 172.21.15.43:0/2450016761 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592e0b3ee90 0x5592e0b451d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:46.815 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.813+0000 7f3344e4a740 1 --2- 172.21.15.43:0/2450016761 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5592e0b57cb0 0x5592e0b5a0a0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:46.815 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.813+0000 7f3344e4a740 1 --2- 172.21.15.43:0/2450016761 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5592e0b3e570 0x5592e0b44c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:46.815 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.813+0000 7f3344e4a740 1 -- 172.21.15.43:0/2450016761 >> 172.21.15.43:0/2450016761 conn(0x5592e0b38e90 msgr2=0x5592e0b3abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:46.815 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.813+0000 7f3324c00640 0 client.16122 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:46.815 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.814+0000 7f3344e4a740 1 -- 172.21.15.43:0/2450016761 shutdown_connections 2024-03-28T22:38:46.815 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.814+0000 7f3344e4a740 1 -- 172.21.15.43:0/2450016761 wait complete. 2024-03-28T22:38:46.815 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.814+0000 7f3344e4a740 20 client.16122 trim_cache size 0 max 16384 2024-03-28T22:38:46.830 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.832+0000 7f3344e4a740 1 Processor -- start 2024-03-28T22:38:46.830 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.832+0000 7f3344e4a740 1 -- start start 2024-03-28T22:38:46.830 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.832+0000 7f3344e4a740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5592e0b3e570 0x5592e0b473c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:46.830 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.832+0000 7f3344e4a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5592e0b3ee90 0x5592e0b535d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:46.830 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.832+0000 7f3344e4a740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592e0b47c60 0x5592e0b454e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:46.831 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.832+0000 7f3344e4a740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f333c1b4ff0 con 0x5592e0b47c60 2024-03-28T22:38:46.831 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.832+0000 7f3344e4a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f333c1b6b30 con 0x5592e0b3ee90 2024-03-28T22:38:46.831 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.832+0000 7f3344e4a740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f333c1b56c0 con 0x5592e0b3e570 2024-03-28T22:38:46.832 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.833+0000 7f333be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5592e0b3e570 0x5592e0b473c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:46.832 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.833+0000 7f333aa00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5592e0b3ee90 0x5592e0b535d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:46.832 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.833+0000 7f333be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5592e0b3e570 0x5592e0b473c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:41886/0 (socket says 172.21.15.43:41886) 2024-03-28T22:38:46.832 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.833+0000 7f333be00640 1 -- 172.21.15.43:0/1218538215 learned_addr learned my addr 172.21.15.43:0/1218538215 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:46.832 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.833+0000 7f333be00640 1 --2- 172.21.15.43:0/1218538215 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592e0b47c60 0x5592e0b454e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:46.832 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.833+0000 7f333aa00640 1 -- 172.21.15.43:0/1218538215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5592e0b3e570 msgr2=0x5592e0b473c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:46.832 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.833+0000 7f333aa00640 1 --2- 172.21.15.43:0/1218538215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5592e0b3e570 0x5592e0b473c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:46.833 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.833+0000 7f333aa00640 1 -- 172.21.15.43:0/1218538215 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592e0b47c60 msgr2=0x5592e0b454e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:46.833 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.833+0000 7f333aa00640 1 --2- 172.21.15.43:0/1218538215 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592e0b47c60 0x5592e0b454e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:46.833 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.833+0000 7f333aa00640 1 -- 172.21.15.43:0/1218538215 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592e0b43c40 con 0x5592e0b3ee90 2024-03-28T22:38:46.833 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.833+0000 7f333be00640 1 --2- 172.21.15.43:0/1218538215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5592e0b3e570 0x5592e0b473c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:46.833 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.834+0000 7f333aa00640 1 --2- 172.21.15.43:0/1218538215 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5592e0b3ee90 0x5592e0b535d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f3328000c00 tx=0x7f33280107d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d47c98c1ae32d887 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:46.833 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.834+0000 7f3327400640 1 -- 172.21.15.43:0/1218538215 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f332800f9e0 con 0x5592e0b3ee90 2024-03-28T22:38:46.833 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.834+0000 7f3327400640 1 -- 172.21.15.43:0/1218538215 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3328010e20 con 0x5592e0b3ee90 2024-03-28T22:38:46.833 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.834+0000 7f3327400640 1 -- 172.21.15.43:0/1218538215 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3328005290 con 0x5592e0b3ee90 2024-03-28T22:38:46.837 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.835+0000 7f3344e4a740 1 -- 172.21.15.43:0/1218538215 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5592e0b3ee90 msgr2=0x5592e0b535d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:46.837 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.835+0000 7f3344e4a740 1 --2- 172.21.15.43:0/1218538215 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5592e0b3ee90 0x5592e0b535d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f3328000c00 tx=0x7f33280107d0 comp rx=0 tx=0).stop 2024-03-28T22:38:46.837 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.835+0000 7f3344e4a740 1 -- 172.21.15.43:0/1218538215 shutdown_connections 2024-03-28T22:38:46.838 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.835+0000 7f3344e4a740 1 --2- 172.21.15.43:0/1218538215 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5592e0b3ee90 0x5592e0b535d0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:46.838 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.835+0000 7f3344e4a740 1 --2- 172.21.15.43:0/1218538215 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592e0b47c60 0x5592e0b454e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:46.838 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.835+0000 7f3344e4a740 1 --2- 172.21.15.43:0/1218538215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5592e0b3e570 0x5592e0b473c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:46.838 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.835+0000 7f3344e4a740 1 -- 172.21.15.43:0/1218538215 >> 172.21.15.43:0/1218538215 conn(0x5592e0b38e90 msgr2=0x5592e0b39270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:46.838 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.835+0000 7f3344e4a740 1 -- 172.21.15.43:0/1218538215 shutdown_connections 2024-03-28T22:38:46.838 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.835+0000 7f3344e4a740 1 -- 172.21.15.43:0/1218538215 wait complete. 2024-03-28T22:38:46.838 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.835+0000 7f3344e4a740 1 -- start start 2024-03-28T22:38:46.838 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.836+0000 7f3344e4a740 1 Processor -- start 2024-03-28T22:38:46.838 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.836+0000 7f3344e4a740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5592e0b3e570 0x5592e0b4b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:46.838 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.836+0000 7f3344e4a740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592e0b3ee90 0x5592e0b4bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:46.839 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.836+0000 7f3344e4a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5592e0b47c60 0x5592e0b4c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:46.839 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.836+0000 7f3344e4a740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f333c1b6460 con 0x5592e0b3ee90 2024-03-28T22:38:46.839 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.836+0000 7f3344e4a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f333c1b5d90 con 0x5592e0b47c60 2024-03-28T22:38:46.839 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.836+0000 7f3344e4a740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f333c1b2fd0 con 0x5592e0b3e570 2024-03-28T22:38:46.839 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.836+0000 7f3344e4a740 10 client.0 _ll_register_callbacks cb 0x7f3328009c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:38:46.839 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.836+0000 7f333aa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592e0b3ee90 0x5592e0b4bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:46.839 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.836+0000 7f333be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5592e0b3e570 0x5592e0b4b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:46.839 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.836+0000 7f333aa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592e0b3ee90 0x5592e0b4bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54950/0 (socket says 172.21.15.43:54950) 2024-03-28T22:38:46.839 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.836+0000 7f333aa00640 1 -- 172.21.15.43:0/87671085 learned_addr learned my addr 172.21.15.43:0/87671085 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:38:46.839 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.836+0000 7f333be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5592e0b3e570 0x5592e0b4b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:41888/0 (socket says 172.21.15.43:41888) 2024-03-28T22:38:46.839 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.836+0000 7f333be00640 1 --2- 172.21.15.43:0/87671085 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5592e0b47c60 0x5592e0b4c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:46.839 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.837+0000 7f333be00640 1 -- 172.21.15.43:0/87671085 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5592e0b47c60 msgr2=0x5592e0b4c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:46.839 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.837+0000 7f333be00640 1 --2- 172.21.15.43:0/87671085 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5592e0b47c60 0x5592e0b4c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:46.839 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.837+0000 7f333be00640 1 -- 172.21.15.43:0/87671085 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592e0b3ee90 msgr2=0x5592e0b4bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:38:46.839 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.837+0000 7f333be00640 1 --2- 172.21.15.43:0/87671085 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592e0b3ee90 0x5592e0b4bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:46.840 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.837+0000 7f333be00640 1 -- 172.21.15.43:0/87671085 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592e0bbd230 con 0x5592e0b3e570 2024-03-28T22:38:46.840 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.837+0000 7f333aa00640 1 --2- 172.21.15.43:0/87671085 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592e0b3ee90 0x5592e0b4bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:38:46.840 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.837+0000 7f333be00640 1 --2- 172.21.15.43:0/87671085 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5592e0b3e570 0x5592e0b4b760 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3334032f30 tx=0x7f33340323c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b44dc3e1e9e1061 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:38:46.840 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.837+0000 7f3324c00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:38:46.840 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.837+0000 7f3324c00640 1 -- 172.21.15.43:0/87671085 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3334039040 con 0x5592e0b3e570 2024-03-28T22:38:46.840 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.837+0000 7f3344e4a740 1 -- 172.21.15.43:0/87671085 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3328010e20 con 0x5592e0b3e570 2024-03-28T22:38:46.840 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.837+0000 7f3324c00640 1 -- 172.21.15.43:0/87671085 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f3334032790 con 0x5592e0b3e570 2024-03-28T22:38:46.840 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.838+0000 7f3324c00640 1 -- 172.21.15.43:0/87671085 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3334043040 con 0x5592e0b3e570 2024-03-28T22:38:46.840 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.838+0000 7f3344e4a740 10 client.26254 fetch_fsmap learned FSMap version 29 2024-03-28T22:38:46.840 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.838+0000 7f3344e4a740 1 -- 172.21.15.43:0/87671085 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5592e0bbe5a0 con 0x5592e0b3e570 2024-03-28T22:38:46.840 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.838+0000 7f3324c00640 1 -- 172.21.15.43:0/87671085 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3328010e20 con 0x5592e0b3e570 2024-03-28T22:38:46.840 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.838+0000 7f3324c00640 1 -- 172.21.15.43:0/87671085 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f3328009ce0 con 0x5592e0b3e570 2024-03-28T22:38:46.840 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.838+0000 7f3344e4a740 10 client.26254 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:38:46.840 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.838+0000 7f3344e4a740 10 client.26254 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:38:46.840 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.838+0000 7f3344e4a740 20 client.26254 populate_metadata read hostname 'smithi043' 2024-03-28T22:38:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.838+0000 7f3344e4a740 1 --2- 172.21.15.43:0/87671085 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5592e0bbe890 0x5592e0bc0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:38:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.838+0000 7f3344e4a740 4 client.26254 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T22:38:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.838+0000 7f3344e4a740 1 -- 172.21.15.43:0/87671085 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5592e0aaeb80 con 0x5592e0bbe890 2024-03-28T22:38:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.839+0000 7f333aa00640 1 --2- 172.21.15.43:0/87671085 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5592e0bbe890 0x5592e0bc0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:38:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.839+0000 7f333aa00640 1 --2- 172.21.15.43:0/87671085 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5592e0bbe890 0x5592e0bc0c80 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22a173ea20fce676 server_cookie=aad7deadd95f389 in_seq=0 out_seq=0 2024-03-28T22:38:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:46.839+0000 7f3324c00640 10 client.26254 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:47.064+0000 7f3324c00640 1 -- 172.21.15.43:0/87671085 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5592e0aaeb80 con 0x5592e0bbe890 2024-03-28T22:38:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:47.064+0000 7f3324c00640 10 client.26254 handle_command_reply: tid=0 2024-03-28T22:38:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:47.064+0000 7f3344e4a740 1 client.26254 shutdown 2024-03-28T22:38:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:47.065+0000 7f3344e4a740 1 -- 172.21.15.43:0/87671085 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5592e0b3e570 msgr2=0x5592e0b4b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:47.065+0000 7f3344e4a740 1 --2- 172.21.15.43:0/87671085 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5592e0b3e570 0x5592e0b4b760 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3334032f30 tx=0x7f33340323c0 comp rx=0 tx=0).stop 2024-03-28T22:38:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:47.065+0000 7f3344e4a740 1 -- 172.21.15.43:0/87671085 shutdown_connections 2024-03-28T22:38:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:47.065+0000 7f3344e4a740 1 --2- 172.21.15.43:0/87671085 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5592e0bbe890 0x5592e0bc0c80 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:47.065+0000 7f3344e4a740 1 --2- 172.21.15.43:0/87671085 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5592e0b47c60 0x5592e0b4c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:47.065+0000 7f3344e4a740 1 --2- 172.21.15.43:0/87671085 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5592e0b3ee90 0x5592e0b4bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:47.065+0000 7f3324c00640 0 client.26254 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:38:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:47.065+0000 7f3344e4a740 1 --2- 172.21.15.43:0/87671085 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5592e0b3e570 0x5592e0b4b760 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:47.066 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:47.065+0000 7f3344e4a740 1 -- 172.21.15.43:0/87671085 >> 172.21.15.43:0/87671085 conn(0x5592e0b38e90 msgr2=0x5592e0b454e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:47.066 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:47.065+0000 7f3344e4a740 1 -- 172.21.15.43:0/87671085 shutdown_connections 2024-03-28T22:38:47.066 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:47.065+0000 7f3344e4a740 1 -- 172.21.15.43:0/87671085 wait complete. 2024-03-28T22:38:47.066 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:47.065+0000 7f3344e4a740 20 client.26254 trim_cache size 0 max 16384 2024-03-28T22:38:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:{ 2024-03-28T22:38:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "message": "", 2024-03-28T22:38:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "return_code": 0 2024-03-28T22:38:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:} 2024-03-28T22:38:47.068 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:47.066+0000 7f3340c00640 1 -- 172.21.15.43:0/3253693281 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f33200a1ee0 msgr2=0x7f33200a43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:47.068 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:47.066+0000 7f3340c00640 1 --2- 172.21.15.43:0/3253693281 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f33200a1ee0 0x7f33200a43a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x5592e0b3e020 tx=0x7f332c0d2040 comp rx=0 tx=0).stop 2024-03-28T22:38:47.069 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:47.066+0000 7f3340c00640 1 -- 172.21.15.43:0/3253693281 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f333c107c90 msgr2=0x7f333c19e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:38:47.069 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:47.066+0000 7f3340c00640 1 --2- 172.21.15.43:0/3253693281 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f333c107c90 0x7f333c19e460 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f332c00e9f0 tx=0x7f332c00eeb0 comp rx=0 tx=0).stop 2024-03-28T22:38:47.069 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:47.066+0000 7f3340c00640 1 -- 172.21.15.43:0/3253693281 shutdown_connections 2024-03-28T22:38:47.069 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:47.066+0000 7f3340c00640 1 --2- 172.21.15.43:0/3253693281 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f33200a1ee0 0x7f33200a43a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:47.069 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:47.066+0000 7f3340c00640 1 --2- 172.21.15.43:0/3253693281 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f333c108610 0x7f333c19e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:47.069 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:47.066+0000 7f3340c00640 1 --2- 172.21.15.43:0/3253693281 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f333c10dbf0 0x7f333c1a2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:47.070 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:47.066+0000 7f3340c00640 1 --2- 172.21.15.43:0/3253693281 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f333c107c90 0x7f333c19e460 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:38:47.070 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:47.066+0000 7f3340c00640 1 -- 172.21.15.43:0/3253693281 >> 172.21.15.43:0/3253693281 conn(0x7f333c077530 msgr2=0x7f333c078660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:38:47.070 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:47.066+0000 7f3340c00640 1 -- 172.21.15.43:0/3253693281 shutdown_connections 2024-03-28T22:38:47.070 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:38:47.066+0000 7f3340c00640 1 -- 172.21.15.43:0/3253693281 wait complete. 2024-03-28T22:38:47.340 INFO:tasks.workunit.client.0.smithi043.stdout: 1 32440 10.73 MB/sec warmup 18 sec latency 106.903 ms 2024-03-28T22:38:47.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:47 smithi172 ceph-mon[36566]: pgmap v50: 129 pgs: 129 active+clean; 13 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 936 KiB/s wr, 229 op/s 2024-03-28T22:38:47.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:47 smithi082 ceph-mon[36471]: pgmap v50: 129 pgs: 129 active+clean; 13 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 936 KiB/s wr, 229 op/s 2024-03-28T22:38:47.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:47 smithi043 ceph-mon[31823]: pgmap v50: 129 pgs: 129 active+clean; 13 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 936 KiB/s wr, 229 op/s 2024-03-28T22:38:48.340 INFO:tasks.workunit.client.0.smithi043.stdout: 1 34366 10.53 MB/sec warmup 19 sec latency 18.191 ms 2024-03-28T22:38:49.340 INFO:tasks.workunit.client.0.smithi043.stdout: 1 35991 10.61 MB/sec warmup 20 sec latency 16.239 ms 2024-03-28T22:38:49.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:49 smithi082 ceph-mon[36471]: pgmap v51: 129 pgs: 129 active+clean; 8.6 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 1.3 MiB/s wr, 411 op/s 2024-03-28T22:38:49.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:49 smithi043 ceph-mon[31823]: pgmap v51: 129 pgs: 129 active+clean; 8.6 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 1.3 MiB/s wr, 411 op/s 2024-03-28T22:38:49.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:49 smithi172 ceph-mon[36566]: pgmap v51: 129 pgs: 129 active+clean; 8.6 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 1.3 MiB/s wr, 411 op/s 2024-03-28T22:38:50.340 INFO:tasks.workunit.client.0.smithi043.stdout: 1 37908 10.44 MB/sec warmup 21 sec latency 21.817 ms 2024-03-28T22:38:51.341 INFO:tasks.workunit.client.0.smithi043.stdout: 1 39533 10.51 MB/sec warmup 22 sec latency 9.836 ms 2024-03-28T22:38:51.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:51 smithi082 ceph-mon[36471]: pgmap v52: 129 pgs: 129 active+clean; 8.6 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 933 KiB/s wr, 252 op/s 2024-03-28T22:38:51.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:51 smithi043 ceph-mon[31823]: pgmap v52: 129 pgs: 129 active+clean; 8.6 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 933 KiB/s wr, 252 op/s 2024-03-28T22:38:51.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:51 smithi172 ceph-mon[36566]: pgmap v52: 129 pgs: 129 active+clean; 8.6 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 933 KiB/s wr, 252 op/s 2024-03-28T22:38:52.341 INFO:tasks.workunit.client.0.smithi043.stdout: 1 41459 10.36 MB/sec warmup 23 sec latency 14.749 ms 2024-03-28T22:38:53.341 INFO:tasks.workunit.client.0.smithi043.stdout: 1 42986 10.43 MB/sec warmup 24 sec latency 14.019 ms 2024-03-28T22:38:53.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:53 smithi082 ceph-mon[36471]: pgmap v53: 129 pgs: 129 active+clean; 8.6 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 933 KiB/s wr, 252 op/s 2024-03-28T22:38:53.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:53 smithi043 ceph-mon[31823]: pgmap v53: 129 pgs: 129 active+clean; 8.6 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 933 KiB/s wr, 252 op/s 2024-03-28T22:38:53.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:53 smithi172 ceph-mon[36566]: pgmap v53: 129 pgs: 129 active+clean; 8.6 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 933 KiB/s wr, 252 op/s 2024-03-28T22:38:54.341 INFO:tasks.workunit.client.0.smithi043.stdout: 1 44458 10.23 MB/sec warmup 25 sec latency 16.019 ms 2024-03-28T22:38:55.341 INFO:tasks.workunit.client.0.smithi043.stdout: 1 46388 10.36 MB/sec warmup 26 sec latency 18.613 ms 2024-03-28T22:38:55.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:55 smithi082 ceph-mon[36471]: pgmap v54: 129 pgs: 129 active+clean; 13 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 1.4 MiB/s wr, 325 op/s 2024-03-28T22:38:55.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:55 smithi043 ceph-mon[31823]: pgmap v54: 129 pgs: 129 active+clean; 13 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 1.4 MiB/s wr, 325 op/s 2024-03-28T22:38:55.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:55 smithi172 ceph-mon[36566]: pgmap v54: 129 pgs: 129 active+clean; 13 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 1.4 MiB/s wr, 325 op/s 2024-03-28T22:38:56.341 INFO:tasks.workunit.client.0.smithi043.stdout: 1 47311 10.06 MB/sec warmup 27 sec latency 41.116 ms 2024-03-28T22:38:57.341 INFO:tasks.workunit.client.0.smithi043.stdout: 1 49431 10.14 MB/sec warmup 28 sec latency 10.929 ms 2024-03-28T22:38:57.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:57 smithi082 ceph-mon[36471]: pgmap v55: 129 pgs: 129 active+clean; 13 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 906 KiB/s wr, 254 op/s 2024-03-28T22:38:57.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:57 smithi043 ceph-mon[31823]: pgmap v55: 129 pgs: 129 active+clean; 13 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 906 KiB/s wr, 254 op/s 2024-03-28T22:38:57.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:57 smithi172 ceph-mon[36566]: pgmap v55: 129 pgs: 129 active+clean; 13 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 906 KiB/s wr, 254 op/s 2024-03-28T22:38:58.341 INFO:tasks.workunit.client.0.smithi043.stdout: 1 50515 9.97 MB/sec warmup 29 sec latency 15.836 ms 2024-03-28T22:38:59.341 INFO:tasks.workunit.client.0.smithi043.stdout: 1 52567 10.00 MB/sec warmup 30 sec latency 13.796 ms 2024-03-28T22:38:59.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:38:59 smithi082 ceph-mon[36471]: pgmap v56: 129 pgs: 129 active+clean; 17 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 1.3 MiB/s wr, 305 op/s 2024-03-28T22:38:59.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:38:59 smithi043 ceph-mon[31823]: pgmap v56: 129 pgs: 129 active+clean; 17 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 1.3 MiB/s wr, 305 op/s 2024-03-28T22:38:59.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:38:59 smithi172 ceph-mon[36566]: pgmap v56: 129 pgs: 129 active+clean; 17 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 1.3 MiB/s wr, 305 op/s 2024-03-28T22:39:00.226 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.228+0000 7faf65000640 1 -- 172.21.15.43:0/2462924913 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf60109940 msgr2=0x7faf60109d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:00.226 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.228+0000 7faf65000640 1 --2- 172.21.15.43:0/2462924913 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf60109940 0x7faf60109d20 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7faf50009930 tx=0x7faf5002bc30 comp rx=0 tx=0).stop 2024-03-28T22:39:00.226 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.228+0000 7faf65000640 1 -- 172.21.15.43:0/2462924913 shutdown_connections 2024-03-28T22:39:00.226 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.228+0000 7faf65000640 1 --2- 172.21.15.43:0/2462924913 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faf60103930 0x7faf60103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:00.226 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.228+0000 7faf65000640 1 --2- 172.21.15.43:0/2462924913 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf601042f0 0x7faf60110800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:00.226 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.228+0000 7faf65000640 1 --2- 172.21.15.43:0/2462924913 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf60109940 0x7faf60109d20 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:00.226 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.228+0000 7faf65000640 1 -- 172.21.15.43:0/2462924913 >> 172.21.15.43:0/2462924913 conn(0x7faf600ff400 msgr2=0x7faf60101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:00.226 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.228+0000 7faf65000640 1 -- 172.21.15.43:0/2462924913 shutdown_connections 2024-03-28T22:39:00.230 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.228+0000 7faf65000640 1 -- 172.21.15.43:0/2462924913 wait complete. 2024-03-28T22:39:00.231 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.229+0000 7faf65000640 1 Processor -- start 2024-03-28T22:39:00.231 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.229+0000 7faf65000640 1 -- start start 2024-03-28T22:39:00.232 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.230+0000 7faf65000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf60103930 0x7faf6019a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:00.232 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.230+0000 7faf65000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf601042f0 0x7faf6019a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:00.232 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.230+0000 7faf65000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faf60109940 0x7faf6019ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:00.232 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.230+0000 7faf65000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7faf60113d70 con 0x7faf60103930 2024-03-28T22:39:00.232 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.230+0000 7faf65000640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7faf60113bf0 con 0x7faf60109940 2024-03-28T22:39:00.232 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.230+0000 7faf65000640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7faf60113ef0 con 0x7faf601042f0 2024-03-28T22:39:00.232 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.230+0000 7faf5e800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faf60109940 0x7faf6019ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:00.232 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.230+0000 7faf5e800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faf60109940 0x7faf6019ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:41336/0 (socket says 172.21.15.43:41336) 2024-03-28T22:39:00.232 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.230+0000 7faf5e800640 1 -- 172.21.15.43:0/1827647471 learned_addr learned my addr 172.21.15.43:0/1827647471 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.230+0000 7faf5de00640 1 --2- 172.21.15.43:0/1827647471 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf60103930 0x7faf6019a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.230+0000 7faf5d400640 1 --2- 172.21.15.43:0/1827647471 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf601042f0 0x7faf6019a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.231+0000 7faf5d400640 1 -- 172.21.15.43:0/1827647471 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faf60109940 msgr2=0x7faf6019ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.231+0000 7faf5d400640 1 --2- 172.21.15.43:0/1827647471 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faf60109940 0x7faf6019ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.231+0000 7faf5d400640 1 -- 172.21.15.43:0/1827647471 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf60103930 msgr2=0x7faf6019a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.231+0000 7faf5d400640 1 --2- 172.21.15.43:0/1827647471 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf60103930 0x7faf6019a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.231+0000 7faf5d400640 1 -- 172.21.15.43:0/1827647471 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf6019f1b0 con 0x7faf601042f0 2024-03-28T22:39:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.231+0000 7faf5d400640 1 --2- 172.21.15.43:0/1827647471 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf601042f0 0x7faf6019a750 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7faf4c00d6d0 tx=0x7faf4c00db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.231+0000 7faf56a00640 1 -- 172.21.15.43:0/1827647471 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4c017070 con 0x7faf601042f0 2024-03-28T22:39:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.231+0000 7faf56a00640 1 -- 172.21.15.43:0/1827647471 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf4c009820 con 0x7faf601042f0 2024-03-28T22:39:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.231+0000 7faf65000640 1 -- 172.21.15.43:0/1827647471 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf6019f4a0 con 0x7faf601042f0 2024-03-28T22:39:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.232+0000 7faf56a00640 1 -- 172.21.15.43:0/1827647471 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4c004d00 con 0x7faf601042f0 2024-03-28T22:39:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.232+0000 7faf65000640 1 -- 172.21.15.43:0/1827647471 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf601a6d30 con 0x7faf601042f0 2024-03-28T22:39:00.237 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.235+0000 7faf67dfd740 1 Processor -- start 2024-03-28T22:39:00.237 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.236+0000 7faf67dfd740 1 -- start start 2024-03-28T22:39:00.237 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.236+0000 7faf56a00640 1 -- 172.21.15.43:0/1827647471 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7faf4c0099d0 con 0x7faf601042f0 2024-03-28T22:39:00.237 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.236+0000 7faf67dfd740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cd74383a0 0x564cd7438780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:00.237 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.236+0000 7faf67dfd740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cd7438cc0 0x564cd7441550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:00.237 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.236+0000 7faf67dfd740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cd7441a90 0x564cd7443e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:00.237 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.236+0000 7faf67dfd740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x564cd739b640 con 0x564cd7438cc0 2024-03-28T22:39:00.238 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.236+0000 7faf67dfd740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x564cd7344810 con 0x564cd74383a0 2024-03-28T22:39:00.238 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.236+0000 7faf67dfd740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x564cd73976e0 con 0x564cd7441a90 2024-03-28T22:39:00.238 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.237+0000 7faf5de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cd7438cc0 0x564cd7441550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:00.238 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.237+0000 7faf5e800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cd74383a0 0x564cd7438780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:00.238 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.237+0000 7faf5de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cd7438cc0 0x564cd7441550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48526/0 (socket says 172.21.15.43:48526) 2024-03-28T22:39:00.238 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.237+0000 7faf5de00640 1 -- 172.21.15.43:0/2861308061 learned_addr learned my addr 172.21.15.43:0/2861308061 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:00.238 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.237+0000 7faf5e800640 1 -- 172.21.15.43:0/2861308061 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cd7441a90 msgr2=0x564cd7443e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:00.238 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.237+0000 7faf5d400640 1 --2- 172.21.15.43:0/2861308061 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cd7441a90 0x564cd7443e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:00.238 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.237+0000 7faf5e800640 1 --2- 172.21.15.43:0/2861308061 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cd7441a90 0x564cd7443e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:00.238 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.237+0000 7faf5e800640 1 -- 172.21.15.43:0/2861308061 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cd7438cc0 msgr2=0x564cd7441550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:00.238 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.237+0000 7faf5e800640 1 --2- 172.21.15.43:0/2861308061 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cd7438cc0 0x564cd7441550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:00.238 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.237+0000 7faf5e800640 1 -- 172.21.15.43:0/2861308061 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cd70b0210 con 0x564cd74383a0 2024-03-28T22:39:00.238 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.237+0000 7faf5d400640 1 --2- 172.21.15.43:0/2861308061 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cd7441a90 0x564cd7443e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:39:00.239 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.238+0000 7faf5e800640 1 --2- 172.21.15.43:0/2861308061 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cd74383a0 0x564cd7438780 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7faf58005c40 tx=0x7faf58010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=291b3171cc2a0cc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:00.239 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.238+0000 7faf4b400640 1 -- 172.21.15.43:0/2861308061 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf58003b70 con 0x564cd74383a0 2024-03-28T22:39:00.239 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.238+0000 7faf4b400640 1 -- 172.21.15.43:0/2861308061 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf58003d10 con 0x564cd74383a0 2024-03-28T22:39:00.239 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.238+0000 7faf4b400640 1 -- 172.21.15.43:0/2861308061 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf58011b30 con 0x564cd74383a0 2024-03-28T22:39:00.239 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.238+0000 7faf56a00640 1 --2- 172.21.15.43:0/1827647471 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7faf440a1f90 0x7faf440a4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:00.239 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.238+0000 7faf56a00640 1 -- 172.21.15.43:0/1827647471 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7faf4c0ce050 con 0x7faf601042f0 2024-03-28T22:39:00.239 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.239+0000 7faf5de00640 1 --2- 172.21.15.43:0/1827647471 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7faf440a1f90 0x7faf440a4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:00.239 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.239+0000 7faf67dfd740 1 -- 172.21.15.43:0/2861308061 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cd74383a0 msgr2=0x564cd7438780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:00.239 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.239+0000 7faf67dfd740 1 --2- 172.21.15.43:0/2861308061 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cd74383a0 0x564cd7438780 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7faf58005c40 tx=0x7faf58010860 comp rx=0 tx=0).stop 2024-03-28T22:39:00.239 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.239+0000 7faf67dfd740 1 -- 172.21.15.43:0/2861308061 shutdown_connections 2024-03-28T22:39:00.239 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.239+0000 7faf67dfd740 1 --2- 172.21.15.43:0/2861308061 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cd7441a90 0x564cd7443e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:00.239 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.239+0000 7faf67dfd740 1 --2- 172.21.15.43:0/2861308061 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cd7438cc0 0x564cd7441550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:00.239 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.239+0000 7faf67dfd740 1 --2- 172.21.15.43:0/2861308061 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cd74383a0 0x564cd7438780 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:00.239 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.239+0000 7faf67dfd740 1 -- 172.21.15.43:0/2861308061 >> 172.21.15.43:0/2861308061 conn(0x564cd7432cc0 msgr2=0x564cd74330a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:00.243 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.240+0000 7faf67dfd740 1 -- 172.21.15.43:0/2861308061 shutdown_connections 2024-03-28T22:39:00.243 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.240+0000 7faf67dfd740 1 -- 172.21.15.43:0/2861308061 wait complete. 2024-03-28T22:39:00.243 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.240+0000 7faf5de00640 1 --2- 172.21.15.43:0/1827647471 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7faf440a1f90 0x7faf440a4450 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7faf50005c40 tx=0x7faf5002df90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:00.243 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.240+0000 7faf67dfd740 1 -- start start 2024-03-28T22:39:00.243 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.241+0000 7faf67dfd740 1 Processor -- start 2024-03-28T22:39:00.244 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.241+0000 7faf67dfd740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cd74383a0 0x564cd743eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:00.244 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.241+0000 7faf67dfd740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cd7438cc0 0x564cd743f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:00.244 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.241+0000 7faf67dfd740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cd7441a90 0x564cd743f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:00.244 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.241+0000 7faf67dfd740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x564cd74446c0 con 0x564cd7438cc0 2024-03-28T22:39:00.244 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.241+0000 7faf67dfd740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x564cd7339d00 con 0x564cd7441a90 2024-03-28T22:39:00.244 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.241+0000 7faf67dfd740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x564cd7444840 con 0x564cd74383a0 2024-03-28T22:39:00.244 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.241+0000 7faf67dfd740 10 client.0 _ll_register_callbacks cb 0x564cd7324490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:00.244 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.242+0000 7faf5e800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cd7441a90 0x564cd743f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:00.244 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.242+0000 7faf5d400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cd7438cc0 0x564cd743f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:00.244 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.242+0000 7faf5e800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cd7441a90 0x564cd743f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:41348/0 (socket says 172.21.15.43:41348) 2024-03-28T22:39:00.244 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.242+0000 7faf5e800640 1 -- 172.21.15.43:0/3811993009 learned_addr learned my addr 172.21.15.43:0/3811993009 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:00.244 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.242+0000 7faf5de00640 1 --2- 172.21.15.43:0/3811993009 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cd74383a0 0x564cd743eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:00.244 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.242+0000 7faf5e800640 1 -- 172.21.15.43:0/3811993009 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cd74383a0 msgr2=0x564cd743eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:00.244 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.242+0000 7faf5e800640 1 --2- 172.21.15.43:0/3811993009 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cd74383a0 0x564cd743eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:00.245 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.242+0000 7faf5e800640 1 -- 172.21.15.43:0/3811993009 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cd7438cc0 msgr2=0x564cd743f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:00.245 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.242+0000 7faf5e800640 1 --2- 172.21.15.43:0/3811993009 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cd7438cc0 0x564cd743f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:00.245 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.242+0000 7faf5e800640 1 -- 172.21.15.43:0/3811993009 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cd717c620 con 0x564cd7441a90 2024-03-28T22:39:00.245 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.242+0000 7faf5d400640 1 --2- 172.21.15.43:0/3811993009 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cd7438cc0 0x564cd743f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:00.245 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.242+0000 7faf5e800640 1 --2- 172.21.15.43:0/3811993009 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cd7441a90 0x564cd743f590 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7faf58011010 tx=0x7faf58009bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d0948cc64f07285 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:00.245 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.242+0000 7faf3fe00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:39:00.245 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.243+0000 7faf3fe00640 1 -- 172.21.15.43:0/3811993009 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf58004a40 con 0x564cd7441a90 2024-03-28T22:39:00.245 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.243+0000 7faf3fe00640 1 -- 172.21.15.43:0/3811993009 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7faf58004be0 con 0x564cd7441a90 2024-03-28T22:39:00.245 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.243+0000 7faf67dfd740 1 -- 172.21.15.43:0/3811993009 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf58003d10 con 0x564cd7441a90 2024-03-28T22:39:00.245 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.243+0000 7faf3fe00640 1 -- 172.21.15.43:0/3811993009 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7faf58004de0 con 0x564cd7441a90 2024-03-28T22:39:00.245 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.243+0000 7faf67dfd740 10 client.26318 fetch_fsmap learned FSMap version 29 2024-03-28T22:39:00.246 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.244+0000 7faf67dfd740 1 -- 172.21.15.43:0/3811993009 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564cd7450550 con 0x564cd7441a90 2024-03-28T22:39:00.246 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.244+0000 7faf3fe00640 1 -- 172.21.15.43:0/3811993009 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf58003d10 con 0x564cd7441a90 2024-03-28T22:39:00.246 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.244+0000 7faf3fe00640 1 -- 172.21.15.43:0/3811993009 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7faf580045c0 con 0x564cd7441a90 2024-03-28T22:39:00.246 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.244+0000 7faf67dfd740 10 client.26318 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:39:00.246 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.244+0000 7faf67dfd740 10 client.26318 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:39:00.246 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.244+0000 7faf67dfd740 20 client.26318 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:00.246 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.244+0000 7faf67dfd740 1 --2- 172.21.15.43:0/3811993009 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x564cd7451900 0x564cd7453cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:00.246 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.244+0000 7faf67dfd740 4 client.26318 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:39:00.247 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.244+0000 7faf67dfd740 1 -- 172.21.15.43:0/3811993009 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564cd73af030 con 0x564cd7451900 2024-03-28T22:39:00.247 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.245+0000 7faf5de00640 1 --2- 172.21.15.43:0/3811993009 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x564cd7451900 0x564cd7453cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:00.247 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.245+0000 7faf5de00640 1 --2- 172.21.15.43:0/3811993009 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x564cd7451900 0x564cd7453cf0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c824a2de19f20bf5 server_cookie=7913e781af3a3605 in_seq=0 out_seq=0 2024-03-28T22:39:00.247 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.245+0000 7faf3fe00640 10 client.26318 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.246+0000 7faf3fe00640 1 -- 172.21.15.43:0/3811993009 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564cd73af030 con 0x564cd7451900 2024-03-28T22:39:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.246+0000 7faf3fe00640 10 client.26318 handle_command_reply: tid=0 2024-03-28T22:39:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.246+0000 7faf67dfd740 1 client.26318 shutdown 2024-03-28T22:39:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.246+0000 7faf67dfd740 1 -- 172.21.15.43:0/3811993009 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cd7441a90 msgr2=0x564cd743f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.246+0000 7faf67dfd740 1 --2- 172.21.15.43:0/3811993009 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cd7441a90 0x564cd743f590 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7faf58011010 tx=0x7faf58009bc0 comp rx=0 tx=0).stop 2024-03-28T22:39:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.246+0000 7faf67dfd740 1 -- 172.21.15.43:0/3811993009 shutdown_connections 2024-03-28T22:39:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.246+0000 7faf67dfd740 1 --2- 172.21.15.43:0/3811993009 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x564cd7451900 0x564cd7453cf0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.246+0000 7faf67dfd740 1 --2- 172.21.15.43:0/3811993009 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cd7441a90 0x564cd743f590 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.246+0000 7faf3fe00640 0 client.26318 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:00.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.246+0000 7faf67dfd740 1 --2- 172.21.15.43:0/3811993009 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cd7438cc0 0x564cd743f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:00.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.246+0000 7faf67dfd740 1 --2- 172.21.15.43:0/3811993009 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cd74383a0 0x564cd743eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:00.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.247+0000 7faf67dfd740 1 -- 172.21.15.43:0/3811993009 >> 172.21.15.43:0/3811993009 conn(0x564cd7432cc0 msgr2=0x564cd7434a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:00.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.247+0000 7faf67dfd740 1 -- 172.21.15.43:0/3811993009 shutdown_connections 2024-03-28T22:39:00.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.247+0000 7faf67dfd740 1 -- 172.21.15.43:0/3811993009 wait complete. 2024-03-28T22:39:00.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.247+0000 7faf67dfd740 20 client.26318 trim_cache size 0 max 16384 2024-03-28T22:39:00.269 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.270+0000 7faf67dfd740 1 Processor -- start 2024-03-28T22:39:00.269 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.271+0000 7faf67dfd740 1 -- start start 2024-03-28T22:39:00.269 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.271+0000 7faf67dfd740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cd74383a0 0x564cd7441240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:00.269 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.271+0000 7faf67dfd740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cd7438cc0 0x564cd744d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:00.269 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.271+0000 7faf67dfd740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cd7441a90 0x564cd743e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:00.269 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.271+0000 7faf67dfd740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7faf601b0de0 con 0x564cd74383a0 2024-03-28T22:39:00.270 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.271+0000 7faf67dfd740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7faf601b2920 con 0x564cd7438cc0 2024-03-28T22:39:00.270 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.271+0000 7faf67dfd740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7faf601b14b0 con 0x564cd7441a90 2024-03-28T22:39:00.270 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.271+0000 7faf5e800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cd7441a90 0x564cd743e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:00.270 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.271+0000 7faf5de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cd74383a0 0x564cd7441240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:00.270 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.271+0000 7faf5d400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cd7438cc0 0x564cd744d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:00.270 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.271+0000 7faf5e800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cd7441a90 0x564cd743e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:50614/0 (socket says 172.21.15.43:50614) 2024-03-28T22:39:00.270 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.271+0000 7faf5d400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cd7438cc0 0x564cd744d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:41350/0 (socket says 172.21.15.43:41350) 2024-03-28T22:39:00.270 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.271+0000 7faf5de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cd74383a0 0x564cd7441240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48546/0 (socket says 172.21.15.43:48546) 2024-03-28T22:39:00.270 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.271+0000 7faf5e800640 1 -- 172.21.15.43:0/3609864280 learned_addr learned my addr 172.21.15.43:0/3609864280 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:00.272 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.272+0000 7faf5e800640 1 -- 172.21.15.43:0/3609864280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cd7438cc0 msgr2=0x564cd744d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:00.272 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.272+0000 7faf5e800640 1 --2- 172.21.15.43:0/3609864280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cd7438cc0 0x564cd744d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:00.272 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.272+0000 7faf5e800640 1 -- 172.21.15.43:0/3609864280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cd74383a0 msgr2=0x564cd7441240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:00.272 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.272+0000 7faf5e800640 1 --2- 172.21.15.43:0/3609864280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cd74383a0 0x564cd7441240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:00.272 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.272+0000 7faf5e800640 1 -- 172.21.15.43:0/3609864280 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cd743d080 con 0x564cd7441a90 2024-03-28T22:39:00.272 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.272+0000 7faf5d400640 1 --2- 172.21.15.43:0/3609864280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cd7438cc0 0x564cd744d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:00.272 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.272+0000 7faf5de00640 1 --2- 172.21.15.43:0/3609864280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cd74383a0 0x564cd7441240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:00.272 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.272+0000 7faf5e800640 1 --2- 172.21.15.43:0/3609864280 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cd7441a90 0x564cd743e410 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7faf580055a0 tx=0x7faf58009890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=136e2c3afe7b4fa3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:00.272 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.272+0000 7faf4aa00640 1 -- 172.21.15.43:0/3609864280 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf58010d60 con 0x564cd7441a90 2024-03-28T22:39:00.272 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.272+0000 7faf4aa00640 1 -- 172.21.15.43:0/3609864280 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf58004a00 con 0x564cd7441a90 2024-03-28T22:39:00.272 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.272+0000 7faf4aa00640 1 -- 172.21.15.43:0/3609864280 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf58011990 con 0x564cd7441a90 2024-03-28T22:39:00.272 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.273+0000 7faf67dfd740 1 -- 172.21.15.43:0/3609864280 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cd7441a90 msgr2=0x564cd743e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:00.273 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.273+0000 7faf67dfd740 1 --2- 172.21.15.43:0/3609864280 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cd7441a90 0x564cd743e410 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7faf580055a0 tx=0x7faf58009890 comp rx=0 tx=0).stop 2024-03-28T22:39:00.273 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.273+0000 7faf67dfd740 1 -- 172.21.15.43:0/3609864280 shutdown_connections 2024-03-28T22:39:00.273 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.273+0000 7faf67dfd740 1 --2- 172.21.15.43:0/3609864280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cd7438cc0 0x564cd744d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:00.273 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.273+0000 7faf67dfd740 1 --2- 172.21.15.43:0/3609864280 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cd7441a90 0x564cd743e410 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:00.273 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.273+0000 7faf67dfd740 1 --2- 172.21.15.43:0/3609864280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cd74383a0 0x564cd7441240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:00.273 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.273+0000 7faf67dfd740 1 -- 172.21.15.43:0/3609864280 >> 172.21.15.43:0/3609864280 conn(0x564cd7432cc0 msgr2=0x564cd7431c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:00.273 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.273+0000 7faf67dfd740 1 -- 172.21.15.43:0/3609864280 shutdown_connections 2024-03-28T22:39:00.273 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.273+0000 7faf67dfd740 1 -- 172.21.15.43:0/3609864280 wait complete. 2024-03-28T22:39:00.273 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.274+0000 7faf67dfd740 1 -- start start 2024-03-28T22:39:00.273 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.274+0000 7faf67dfd740 1 Processor -- start 2024-03-28T22:39:00.275 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.275+0000 7faf67dfd740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cd74383a0 0x564cd7449c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:00.275 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.275+0000 7faf67dfd740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cd7438cc0 0x564cd743b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:00.275 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.275+0000 7faf67dfd740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cd7441a90 0x564cd743b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:00.275 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.275+0000 7faf67dfd740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7faf601b2250 con 0x564cd7441a90 2024-03-28T22:39:00.275 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.275+0000 7faf67dfd740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7faf601b1b80 con 0x564cd7438cc0 2024-03-28T22:39:00.275 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.275+0000 7faf67dfd740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7faf601aedc0 con 0x564cd74383a0 2024-03-28T22:39:00.275 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.275+0000 7faf67dfd740 10 client.0 _ll_register_callbacks cb 0x7faf50005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:00.275 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.275+0000 7faf5de00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cd74383a0 0x564cd7449c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.275+0000 7faf5de00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cd74383a0 0x564cd7449c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:50630/0 (socket says 172.21.15.43:50630) 2024-03-28T22:39:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.275+0000 7faf5de00640 1 -- 172.21.15.43:0/76583158 learned_addr learned my addr 172.21.15.43:0/76583158 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.275+0000 7faf5d400640 1 --2- 172.21.15.43:0/76583158 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cd7438cc0 0x564cd743b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.275+0000 7faf5e800640 1 --2- 172.21.15.43:0/76583158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cd7441a90 0x564cd743b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.275+0000 7faf5de00640 1 -- 172.21.15.43:0/76583158 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cd7438cc0 msgr2=0x564cd743b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.275+0000 7faf5de00640 1 --2- 172.21.15.43:0/76583158 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cd7438cc0 0x564cd743b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.275+0000 7faf5de00640 1 -- 172.21.15.43:0/76583158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cd7441a90 msgr2=0x564cd743b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.275+0000 7faf5de00640 1 --2- 172.21.15.43:0/76583158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cd7441a90 0x564cd743b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.275+0000 7faf5de00640 1 -- 172.21.15.43:0/76583158 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cd743bc20 con 0x564cd74383a0 2024-03-28T22:39:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.276+0000 7faf5de00640 1 --2- 172.21.15.43:0/76583158 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cd74383a0 0x564cd7449c80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7faf5002c240 tx=0x7faf50042070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9843e8a2c3af0fd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.276+0000 7faf3fe00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:39:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.276+0000 7faf3fe00640 1 -- 172.21.15.43:0/76583158 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf50046070 con 0x564cd74383a0 2024-03-28T22:39:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.276+0000 7faf3fe00640 1 -- 172.21.15.43:0/76583158 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7faf500583f0 con 0x564cd74383a0 2024-03-28T22:39:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.276+0000 7faf67dfd740 1 -- 172.21.15.43:0/76583158 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf58004a00 con 0x564cd74383a0 2024-03-28T22:39:00.277 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.277+0000 7faf3fe00640 1 -- 172.21.15.43:0/76583158 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7faf500585f0 con 0x564cd74383a0 2024-03-28T22:39:00.277 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.277+0000 7faf67dfd740 10 client.26269 fetch_fsmap learned FSMap version 29 2024-03-28T22:39:00.277 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.277+0000 7faf67dfd740 1 -- 172.21.15.43:0/76583158 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564cd74bfe80 con 0x564cd74383a0 2024-03-28T22:39:00.277 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.277+0000 7faf3fe00640 1 -- 172.21.15.43:0/76583158 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf58004a00 con 0x564cd74383a0 2024-03-28T22:39:00.278 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.278+0000 7faf3fe00640 1 -- 172.21.15.43:0/76583158 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7faf500052f0 con 0x564cd74383a0 2024-03-28T22:39:00.278 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.278+0000 7faf67dfd740 10 client.26269 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:39:00.278 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.278+0000 7faf67dfd740 10 client.26269 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:39:00.278 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.278+0000 7faf67dfd740 20 client.26269 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:00.278 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.278+0000 7faf67dfd740 1 --2- 172.21.15.43:0/76583158 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x564cd7451900 0x564cd74c2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:00.278 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.278+0000 7faf67dfd740 4 client.26269 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T22:39:00.278 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.278+0000 7faf67dfd740 1 -- 172.21.15.43:0/76583158 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564cd73a8b80 con 0x564cd7451900 2024-03-28T22:39:00.278 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.279+0000 7faf5d400640 1 --2- 172.21.15.43:0/76583158 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x564cd7451900 0x564cd74c2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:00.278 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.279+0000 7faf5d400640 1 --2- 172.21.15.43:0/76583158 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x564cd7451900 0x564cd74c2580 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bf59bf763d07307 server_cookie=8ca070e540282cfc in_seq=0 out_seq=0 2024-03-28T22:39:00.278 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.279+0000 7faf3fe00640 10 client.26269 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:00.341 INFO:tasks.workunit.client.0.smithi043.stdout: 1 53661 9.93 MB/sec warmup 31 sec latency 57.779 ms 2024-03-28T22:39:00.527 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.529+0000 7faf3fe00640 1 -- 172.21.15.43:0/76583158 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564cd73a8b80 con 0x564cd7451900 2024-03-28T22:39:00.527 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.529+0000 7faf3fe00640 10 client.26269 handle_command_reply: tid=0 2024-03-28T22:39:00.528 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.529+0000 7faf67dfd740 1 client.26269 shutdown 2024-03-28T22:39:00.528 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.529+0000 7faf67dfd740 1 -- 172.21.15.43:0/76583158 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cd74383a0 msgr2=0x564cd7449c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:00.528 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.529+0000 7faf67dfd740 1 --2- 172.21.15.43:0/76583158 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cd74383a0 0x564cd7449c80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7faf5002c240 tx=0x7faf50042070 comp rx=0 tx=0).stop 2024-03-28T22:39:00.528 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.529+0000 7faf67dfd740 1 -- 172.21.15.43:0/76583158 shutdown_connections 2024-03-28T22:39:00.528 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.529+0000 7faf67dfd740 1 --2- 172.21.15.43:0/76583158 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x564cd7451900 0x564cd74c2580 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:00.528 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.529+0000 7faf67dfd740 1 --2- 172.21.15.43:0/76583158 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cd7438cc0 0x564cd743b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:00.528 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.529+0000 7faf3fe00640 0 client.26269 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:00.528 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.529+0000 7faf67dfd740 1 --2- 172.21.15.43:0/76583158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cd7441a90 0x564cd743b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:00.528 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.529+0000 7faf67dfd740 1 --2- 172.21.15.43:0/76583158 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cd74383a0 0x564cd7449c80 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:00.528 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.529+0000 7faf67dfd740 1 -- 172.21.15.43:0/76583158 >> 172.21.15.43:0/76583158 conn(0x564cd7432cc0 msgr2=0x564cd7433fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:00.528 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.529+0000 7faf67dfd740 1 -- 172.21.15.43:0/76583158 shutdown_connections 2024-03-28T22:39:00.528 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.530+0000 7faf67dfd740 1 -- 172.21.15.43:0/76583158 wait complete. 2024-03-28T22:39:00.528 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.530+0000 7faf67dfd740 20 client.26269 trim_cache size 0 max 16384 2024-03-28T22:39:00.529 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:{ 2024-03-28T22:39:00.529 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "message": "", 2024-03-28T22:39:00.529 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "return_code": 0 2024-03-28T22:39:00.529 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:} 2024-03-28T22:39:00.530 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.530+0000 7faf65000640 1 -- 172.21.15.43:0/1827647471 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7faf440a1f90 msgr2=0x7faf440a4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:00.530 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.530+0000 7faf65000640 1 --2- 172.21.15.43:0/1827647471 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7faf440a1f90 0x7faf440a4450 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7faf50005c40 tx=0x7faf5002df90 comp rx=0 tx=0).stop 2024-03-28T22:39:00.530 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.530+0000 7faf65000640 1 -- 172.21.15.43:0/1827647471 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf601042f0 msgr2=0x7faf6019a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:00.530 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.530+0000 7faf65000640 1 --2- 172.21.15.43:0/1827647471 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf601042f0 0x7faf6019a750 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7faf4c00d6d0 tx=0x7faf4c00db90 comp rx=0 tx=0).stop 2024-03-28T22:39:00.530 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.530+0000 7faf65000640 1 -- 172.21.15.43:0/1827647471 shutdown_connections 2024-03-28T22:39:00.530 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.531+0000 7faf65000640 1 --2- 172.21.15.43:0/1827647471 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7faf440a1f90 0x7faf440a4450 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:00.530 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.531+0000 7faf65000640 1 --2- 172.21.15.43:0/1827647471 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faf60109940 0x7faf6019ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:00.530 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.531+0000 7faf65000640 1 --2- 172.21.15.43:0/1827647471 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf601042f0 0x7faf6019a750 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:00.531 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.531+0000 7faf65000640 1 --2- 172.21.15.43:0/1827647471 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faf60103930 0x7faf6019a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:00.531 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.531+0000 7faf65000640 1 -- 172.21.15.43:0/1827647471 >> 172.21.15.43:0/1827647471 conn(0x7faf600ff400 msgr2=0x7faf600ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:00.531 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.531+0000 7faf65000640 1 -- 172.21.15.43:0/1827647471 shutdown_connections 2024-03-28T22:39:00.531 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:00.531+0000 7faf65000640 1 -- 172.21.15.43:0/1827647471 wait complete. 2024-03-28T22:39:01.342 INFO:tasks.workunit.client.0.smithi043.stdout: 1 54735 9.78 MB/sec warmup 32 sec latency 149.164 ms 2024-03-28T22:39:01.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:01 smithi082 ceph-mon[36471]: pgmap v57: 129 pgs: 129 active+clean; 17 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 861 KiB/s wr, 123 op/s 2024-03-28T22:39:01.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:01 smithi043 ceph-mon[31823]: pgmap v57: 129 pgs: 129 active+clean; 17 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 861 KiB/s wr, 123 op/s 2024-03-28T22:39:01.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:01 smithi172 ceph-mon[36566]: pgmap v57: 129 pgs: 129 active+clean; 17 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 861 KiB/s wr, 123 op/s 2024-03-28T22:39:02.342 INFO:tasks.workunit.client.0.smithi043.stdout: 1 56782 9.86 MB/sec warmup 33 sec latency 24.342 ms 2024-03-28T22:39:02.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:02 smithi082 ceph-mon[36471]: pgmap v58: 129 pgs: 129 active+clean; 17 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 861 KiB/s wr, 123 op/s 2024-03-28T22:39:02.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:02 smithi043 ceph-mon[31823]: pgmap v58: 129 pgs: 129 active+clean; 17 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 861 KiB/s wr, 123 op/s 2024-03-28T22:39:02.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:02 smithi172 ceph-mon[36566]: pgmap v58: 129 pgs: 129 active+clean; 17 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 861 KiB/s wr, 123 op/s 2024-03-28T22:39:03.342 INFO:tasks.workunit.client.0.smithi043.stdout: 1 57683 9.64 MB/sec warmup 34 sec latency 19.528 ms 2024-03-28T22:39:04.342 INFO:tasks.workunit.client.0.smithi043.stdout: 1 59694 9.70 MB/sec warmup 35 sec latency 13.288 ms 2024-03-28T22:39:04.765 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:04 smithi043 ceph-mon[31823]: pgmap v59: 129 pgs: 129 active+clean; 36 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 2.9 MiB/s wr, 352 op/s 2024-03-28T22:39:04.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:04 smithi172 ceph-mon[36566]: pgmap v59: 129 pgs: 129 active+clean; 36 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 2.9 MiB/s wr, 352 op/s 2024-03-28T22:39:04.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:04 smithi082 ceph-mon[36471]: pgmap v59: 129 pgs: 129 active+clean; 36 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 2.9 MiB/s wr, 352 op/s 2024-03-28T22:39:05.342 INFO:tasks.workunit.client.0.smithi043.stdout: 1 60790 9.61 MB/sec warmup 36 sec latency 50.645 ms 2024-03-28T22:39:06.342 INFO:tasks.workunit.client.0.smithi043.stdout: 1 61956 9.49 MB/sec warmup 37 sec latency 62.825 ms 2024-03-28T22:39:06.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:06 smithi043 ceph-mon[31823]: pgmap v60: 129 pgs: 129 active+clean; 36 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 2.5 MiB/s wr, 279 op/s 2024-03-28T22:39:06.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:06 smithi172 ceph-mon[36566]: pgmap v60: 129 pgs: 129 active+clean; 36 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 2.5 MiB/s wr, 279 op/s 2024-03-28T22:39:06.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:06 smithi082 ceph-mon[36471]: pgmap v60: 129 pgs: 129 active+clean; 36 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 2.5 MiB/s wr, 279 op/s 2024-03-28T22:39:07.342 INFO:tasks.workunit.client.0.smithi043.stdout: 1 63961 9.57 MB/sec warmup 38 sec latency 19.994 ms 2024-03-28T22:39:08.342 INFO:tasks.workunit.client.0.smithi043.stdout: 1 64670 9.38 MB/sec warmup 39 sec latency 20.750 ms 2024-03-28T22:39:08.765 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:08 smithi043 ceph-mon[31823]: pgmap v61: 129 pgs: 129 active+clean; 39 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 367 op/s 2024-03-28T22:39:08.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:08 smithi172 ceph-mon[36566]: pgmap v61: 129 pgs: 129 active+clean; 39 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 367 op/s 2024-03-28T22:39:08.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:08 smithi082 ceph-mon[36471]: pgmap v61: 129 pgs: 129 active+clean; 39 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 367 op/s 2024-03-28T22:39:09.343 INFO:tasks.workunit.client.0.smithi043.stdout: 1 66441 9.33 MB/sec warmup 40 sec latency 23.131 ms 2024-03-28T22:39:10.342 INFO:tasks.workunit.client.0.smithi043.stdout: 1 67729 9.36 MB/sec warmup 41 sec latency 29.812 ms 2024-03-28T22:39:10.765 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:10 smithi043 ceph-mon[31823]: pgmap v62: 129 pgs: 129 active+clean; 39 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.2 MiB/s wr, 315 op/s 2024-03-28T22:39:10.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:10 smithi172 ceph-mon[36566]: pgmap v62: 129 pgs: 129 active+clean; 39 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.2 MiB/s wr, 315 op/s 2024-03-28T22:39:10.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:10 smithi082 ceph-mon[36471]: pgmap v62: 129 pgs: 129 active+clean; 39 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.2 MiB/s wr, 315 op/s 2024-03-28T22:39:11.343 INFO:tasks.workunit.client.0.smithi043.stdout: 1 68678 9.20 MB/sec warmup 42 sec latency 47.813 ms 2024-03-28T22:39:12.343 INFO:tasks.workunit.client.0.smithi043.stdout: 1 70682 9.27 MB/sec warmup 43 sec latency 13.046 ms 2024-03-28T22:39:12.542 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:39:12.542 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:39:12.765 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:12 smithi043 ceph-mon[31823]: pgmap v63: 129 pgs: 129 active+clean; 39 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.2 MiB/s wr, 315 op/s 2024-03-28T22:39:12.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.787+0000 7f4dd8000640 1 -- 172.21.15.43:0/2882038343 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dd01038f0 msgr2=0x7f4dd0103d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:12.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.787+0000 7f4dd8000640 1 --2- 172.21.15.43:0/2882038343 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dd01038f0 0x7f4dd0103d70 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc009930 tx=0x7f4dbc02bc30 comp rx=0 tx=0).stop 2024-03-28T22:39:12.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.787+0000 7f4dd8000640 1 -- 172.21.15.43:0/2882038343 shutdown_connections 2024-03-28T22:39:12.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.787+0000 7f4dd8000640 1 --2- 172.21.15.43:0/2882038343 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4dd01042b0 0x7f4dd01107c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:12.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.787+0000 7f4dd8000640 1 --2- 172.21.15.43:0/2882038343 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dd01038f0 0x7f4dd0103d70 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:12.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.787+0000 7f4dd8000640 1 --2- 172.21.15.43:0/2882038343 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4dd0109900 0x7f4dd0109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:12.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.787+0000 7f4dd8000640 1 -- 172.21.15.43:0/2882038343 >> 172.21.15.43:0/2882038343 conn(0x7f4dd00ff3a0 msgr2=0x7f4dd01017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:12.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.788+0000 7f4dd8000640 1 -- 172.21.15.43:0/2882038343 shutdown_connections 2024-03-28T22:39:12.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.788+0000 7f4dd8000640 1 -- 172.21.15.43:0/2882038343 wait complete. 2024-03-28T22:39:12.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.789+0000 7f4dd8000640 1 Processor -- start 2024-03-28T22:39:12.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.789+0000 7f4dd8000640 1 -- start start 2024-03-28T22:39:12.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.789+0000 7f4dd8000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4dd01038f0 0x7f4dd019a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:12.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.789+0000 7f4dd8000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dd01042b0 0x7f4dd019a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:12.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.789+0000 7f4dd8000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4dd0109900 0x7f4dd019eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:12.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.789+0000 7f4dd8000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd0113d30 con 0x7f4dd01038f0 2024-03-28T22:39:12.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.789+0000 7f4dd8000640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd0113bb0 con 0x7f4dd0109900 2024-03-28T22:39:12.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.789+0000 7f4dd8000640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd0113eb0 con 0x7f4dd01042b0 2024-03-28T22:39:12.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.790+0000 7f4dd4c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dd01042b0 0x7f4dd019a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:12.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.790+0000 7f4dd6000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4dd0109900 0x7f4dd019eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:12.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.790+0000 7f4dd5600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4dd01038f0 0x7f4dd019a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:12.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.790+0000 7f4dd4c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dd01042b0 0x7f4dd019a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:42858/0 (socket says 172.21.15.43:42858) 2024-03-28T22:39:12.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.790+0000 7f4dd4c00640 1 -- 172.21.15.43:0/3069390095 learned_addr learned my addr 172.21.15.43:0/3069390095 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:12.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.790+0000 7f4dd4c00640 1 -- 172.21.15.43:0/3069390095 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4dd0109900 msgr2=0x7f4dd019eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:12.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.790+0000 7f4dd4c00640 1 --2- 172.21.15.43:0/3069390095 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4dd0109900 0x7f4dd019eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:12.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.790+0000 7f4dd4c00640 1 -- 172.21.15.43:0/3069390095 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4dd01038f0 msgr2=0x7f4dd019a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:12.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.790+0000 7f4dd4c00640 1 --2- 172.21.15.43:0/3069390095 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4dd01038f0 0x7f4dd019a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:12.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.790+0000 7f4dd4c00640 1 -- 172.21.15.43:0/3069390095 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dd019f260 con 0x7f4dd01042b0 2024-03-28T22:39:12.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.790+0000 7f4dd5600640 1 --2- 172.21.15.43:0/3069390095 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4dd01038f0 0x7f4dd019a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:12.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.790+0000 7f4dd4c00640 1 --2- 172.21.15.43:0/3069390095 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dd01042b0 0x7f4dd019a7b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc009900 tx=0x7f4dbc004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:12.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.791+0000 7f4dce000640 1 -- 172.21.15.43:0/3069390095 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc043020 con 0x7f4dd01042b0 2024-03-28T22:39:12.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.791+0000 7f4dce000640 1 -- 172.21.15.43:0/3069390095 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4dbc0054e0 con 0x7f4dd01042b0 2024-03-28T22:39:12.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.791+0000 7f4dce000640 1 -- 172.21.15.43:0/3069390095 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc003bb0 con 0x7f4dd01042b0 2024-03-28T22:39:12.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.791+0000 7f4dd8000640 1 -- 172.21.15.43:0/3069390095 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dd019f4f0 con 0x7f4dd01042b0 2024-03-28T22:39:12.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.791+0000 7f4dd8000640 1 -- 172.21.15.43:0/3069390095 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dd01a6de0 con 0x7f4dd01042b0 2024-03-28T22:39:12.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.794+0000 7f4dce000640 1 -- 172.21.15.43:0/3069390095 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f4dbc046020 con 0x7f4dd01042b0 2024-03-28T22:39:12.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.794+0000 7f4ddad7d740 1 Processor -- start 2024-03-28T22:39:12.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.795+0000 7f4ddad7d740 1 -- start start 2024-03-28T22:39:12.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.795+0000 7f4ddad7d740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557cb9824bc0 0x557cb9823910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:12.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.795+0000 7f4ddad7d740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557cb9823ee0 0x557cb9824360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:12.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.795+0000 7f4ddad7d740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557cb982d250 0x557cb982f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:12.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.795+0000 7f4ddad7d740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x557cb9788100 con 0x557cb9824bc0 2024-03-28T22:39:12.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.795+0000 7f4ddad7d740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x557cb9730ba0 con 0x557cb9823ee0 2024-03-28T22:39:12.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.795+0000 7f4ddad7d740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x557cb97841a0 con 0x557cb982d250 2024-03-28T22:39:12.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.796+0000 7f4dd5600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557cb9823ee0 0x557cb9824360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:12.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.796+0000 7f4dd4c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557cb982d250 0x557cb982f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:12.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.796+0000 7f4dd5600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557cb9823ee0 0x557cb9824360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59296/0 (socket says 172.21.15.43:59296) 2024-03-28T22:39:12.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.796+0000 7f4dd5600640 1 -- 172.21.15.43:0/3977345855 learned_addr learned my addr 172.21.15.43:0/3977345855 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:12.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.796+0000 7f4dd6000640 1 --2- 172.21.15.43:0/3977345855 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557cb9824bc0 0x557cb9823910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:12.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.796+0000 7f4dd5600640 1 -- 172.21.15.43:0/3977345855 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557cb982d250 msgr2=0x557cb982f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:12.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.796+0000 7f4dd5600640 1 --2- 172.21.15.43:0/3977345855 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557cb982d250 0x557cb982f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:12.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.796+0000 7f4dd5600640 1 -- 172.21.15.43:0/3977345855 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557cb9824bc0 msgr2=0x557cb9823910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:12.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.796+0000 7f4dd5600640 1 --2- 172.21.15.43:0/3977345855 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557cb9824bc0 0x557cb9823910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:12.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.796+0000 7f4dd5600640 1 -- 172.21.15.43:0/3977345855 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557cb97611a0 con 0x557cb9823ee0 2024-03-28T22:39:12.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.796+0000 7f4dce000640 1 --2- 172.21.15.43:0/3069390095 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4db80a1f70 0x7f4db80a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:12.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.796+0000 7f4dce000640 1 -- 172.21.15.43:0/3069390095 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f4dbc0f0460 con 0x7f4dd01042b0 2024-03-28T22:39:12.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.797+0000 7f4dd6000640 1 --2- 172.21.15.43:0/3069390095 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4db80a1f70 0x7f4db80a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:12.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.797+0000 7f4dd5600640 1 --2- 172.21.15.43:0/3977345855 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557cb9823ee0 0x557cb9824360 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f4dc40114c0 tx=0x7f4dc4011980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b48dd062c2b9b9d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:12.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.797+0000 7f4dc2600640 1 -- 172.21.15.43:0/3977345855 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc400f5c0 con 0x557cb9823ee0 2024-03-28T22:39:12.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.797+0000 7f4dc2600640 1 -- 172.21.15.43:0/3977345855 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4dc400fc40 con 0x557cb9823ee0 2024-03-28T22:39:12.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.797+0000 7f4dc2600640 1 -- 172.21.15.43:0/3977345855 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc4019d20 con 0x557cb9823ee0 2024-03-28T22:39:12.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.798+0000 7f4ddad7d740 1 -- 172.21.15.43:0/3977345855 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557cb9823ee0 msgr2=0x557cb9824360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.798+0000 7f4ddad7d740 1 --2- 172.21.15.43:0/3977345855 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557cb9823ee0 0x557cb9824360 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f4dc40114c0 tx=0x7f4dc4011980 comp rx=0 tx=0).stop 2024-03-28T22:39:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.798+0000 7f4dd6000640 1 --2- 172.21.15.43:0/3069390095 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4db80a1f70 0x7f4db80a4430 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4dc8005fd0 tx=0x7f4dc8009500 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.798+0000 7f4ddad7d740 1 -- 172.21.15.43:0/3977345855 shutdown_connections 2024-03-28T22:39:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.798+0000 7f4ddad7d740 1 --2- 172.21.15.43:0/3977345855 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557cb9823ee0 0x557cb9824360 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.798+0000 7f4ddad7d740 1 --2- 172.21.15.43:0/3977345855 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557cb982d250 0x557cb982f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.798+0000 7f4ddad7d740 1 --2- 172.21.15.43:0/3977345855 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557cb9824bc0 0x557cb9823910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.798+0000 7f4ddad7d740 1 -- 172.21.15.43:0/3977345855 >> 172.21.15.43:0/3977345855 conn(0x557cb97eaa10 msgr2=0x557cb981e930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.798+0000 7f4ddad7d740 1 -- 172.21.15.43:0/3977345855 shutdown_connections 2024-03-28T22:39:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.798+0000 7f4ddad7d740 1 -- 172.21.15.43:0/3977345855 wait complete. 2024-03-28T22:39:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.799+0000 7f4ddad7d740 1 -- start start 2024-03-28T22:39:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.799+0000 7f4ddad7d740 1 Processor -- start 2024-03-28T22:39:12.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.799+0000 7f4ddad7d740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557cb9823ee0 0x557cb982a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:12.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.799+0000 7f4ddad7d740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557cb9824bc0 0x557cb982a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:12.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.799+0000 7f4ddad7d740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557cb982d250 0x557cb982ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:12.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.799+0000 7f4ddad7d740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x557cb98388d0 con 0x557cb982d250 2024-03-28T22:39:12.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.799+0000 7f4ddad7d740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x557cb9726250 con 0x557cb9824bc0 2024-03-28T22:39:12.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.799+0000 7f4ddad7d740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x557cb9838a50 con 0x557cb9823ee0 2024-03-28T22:39:12.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.799+0000 7f4ddad7d740 10 client.0 _ll_register_callbacks cb 0x557cb949bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:12.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.800+0000 7f4dd4c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557cb9824bc0 0x557cb982a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:12.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.800+0000 7f4dd5600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557cb9823ee0 0x557cb982a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:12.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.800+0000 7f4dd4c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557cb9824bc0 0x557cb982a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59306/0 (socket says 172.21.15.43:59306) 2024-03-28T22:39:12.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.800+0000 7f4dd4c00640 1 -- 172.21.15.43:0/2164417264 learned_addr learned my addr 172.21.15.43:0/2164417264 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:12.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.800+0000 7f4dd5600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557cb9823ee0 0x557cb982a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:42868/0 (socket says 172.21.15.43:42868) 2024-03-28T22:39:12.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.800+0000 7f4dd6000640 1 --2- 172.21.15.43:0/2164417264 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557cb982d250 0x557cb982ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:12.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.800+0000 7f4dd4c00640 1 -- 172.21.15.43:0/2164417264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557cb9823ee0 msgr2=0x557cb982a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:12.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.800+0000 7f4dd4c00640 1 --2- 172.21.15.43:0/2164417264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557cb9823ee0 0x557cb982a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:12.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.800+0000 7f4dd4c00640 1 -- 172.21.15.43:0/2164417264 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557cb982d250 msgr2=0x557cb982ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:12.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.800+0000 7f4dd4c00640 1 --2- 172.21.15.43:0/2164417264 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557cb982d250 0x557cb982ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:12.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.800+0000 7f4dd4c00640 1 -- 172.21.15.43:0/2164417264 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557cb9742960 con 0x557cb9824bc0 2024-03-28T22:39:12.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.800+0000 7f4dd6000640 1 --2- 172.21.15.43:0/2164417264 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557cb982d250 0x557cb982ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:39:12.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.800+0000 7f4dd5600640 1 --2- 172.21.15.43:0/2164417264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557cb9823ee0 0x557cb982a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:12.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.800+0000 7f4dd4c00640 1 --2- 172.21.15.43:0/2164417264 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557cb9824bc0 0x557cb982a740 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc0bcaf0 tx=0x7f4dbc009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a527f9d13280019 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:12.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.800+0000 7f4db6a00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:39:12.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.801+0000 7f4db6a00640 1 -- 172.21.15.43:0/2164417264 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc0bc6a0 con 0x557cb9824bc0 2024-03-28T22:39:12.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.801+0000 7f4ddad7d740 1 -- 172.21.15.43:0/2164417264 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4dc400fc40 con 0x557cb9824bc0 2024-03-28T22:39:12.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.801+0000 7f4db6a00640 1 -- 172.21.15.43:0/2164417264 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f4dbc0b91d0 con 0x557cb9824bc0 2024-03-28T22:39:12.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.801+0000 7f4db6a00640 1 -- 172.21.15.43:0/2164417264 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4dbc0bc4c0 con 0x557cb9824bc0 2024-03-28T22:39:12.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.801+0000 7f4ddad7d740 10 client.26336 fetch_fsmap learned FSMap version 29 2024-03-28T22:39:12.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.801+0000 7f4ddad7d740 1 -- 172.21.15.43:0/2164417264 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557cb95667d0 con 0x557cb9824bc0 2024-03-28T22:39:12.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.801+0000 7f4db6a00640 1 -- 172.21.15.43:0/2164417264 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4dc400fc40 con 0x557cb9824bc0 2024-03-28T22:39:12.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.801+0000 7f4db6a00640 1 -- 172.21.15.43:0/2164417264 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f4dbc0eda30 con 0x557cb9824bc0 2024-03-28T22:39:12.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.801+0000 7f4ddad7d740 10 client.26336 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:39:12.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.801+0000 7f4ddad7d740 10 client.26336 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:39:12.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.801+0000 7f4ddad7d740 20 client.26336 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:12.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.801+0000 7f4ddad7d740 1 --2- 172.21.15.43:0/2164417264 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557cb9845e60 0x557cb98482b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:12.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.801+0000 7f4ddad7d740 4 client.26336 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:39:12.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.801+0000 7f4ddad7d740 1 -- 172.21.15.43:0/2164417264 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557cb9795640 con 0x557cb9845e60 2024-03-28T22:39:12.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.802+0000 7f4dd5600640 1 --2- 172.21.15.43:0/2164417264 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557cb9845e60 0x557cb98482b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:12.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.802+0000 7f4dd5600640 1 --2- 172.21.15.43:0/2164417264 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557cb9845e60 0x557cb98482b0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd55bf68bbbf30d1 server_cookie=439be49251081426 in_seq=0 out_seq=0 2024-03-28T22:39:12.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.802+0000 7f4db6a00640 10 client.26336 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:12.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.803+0000 7f4db6a00640 1 -- 172.21.15.43:0/2164417264 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557cb9795640 con 0x557cb9845e60 2024-03-28T22:39:12.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.803+0000 7f4db6a00640 10 client.26336 handle_command_reply: tid=0 2024-03-28T22:39:12.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.803+0000 7f4ddad7d740 1 client.26336 shutdown 2024-03-28T22:39:12.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.804+0000 7f4ddad7d740 1 -- 172.21.15.43:0/2164417264 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557cb9824bc0 msgr2=0x557cb982a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:12.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.804+0000 7f4ddad7d740 1 --2- 172.21.15.43:0/2164417264 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557cb9824bc0 0x557cb982a740 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc0bcaf0 tx=0x7f4dbc009560 comp rx=0 tx=0).stop 2024-03-28T22:39:12.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.804+0000 7f4ddad7d740 1 -- 172.21.15.43:0/2164417264 shutdown_connections 2024-03-28T22:39:12.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.804+0000 7f4ddad7d740 1 --2- 172.21.15.43:0/2164417264 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557cb9845e60 0x557cb98482b0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:12.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.804+0000 7f4ddad7d740 1 --2- 172.21.15.43:0/2164417264 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557cb9824bc0 0x557cb982a740 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:12.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.804+0000 7f4ddad7d740 1 --2- 172.21.15.43:0/2164417264 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557cb982d250 0x557cb982ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:12.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.804+0000 7f4ddad7d740 1 --2- 172.21.15.43:0/2164417264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557cb9823ee0 0x557cb982a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:12.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.804+0000 7f4db6a00640 0 client.26336 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:12.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.804+0000 7f4ddad7d740 1 -- 172.21.15.43:0/2164417264 >> 172.21.15.43:0/2164417264 conn(0x557cb97eaa10 msgr2=0x557cb9820680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:12.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.804+0000 7f4ddad7d740 1 -- 172.21.15.43:0/2164417264 shutdown_connections 2024-03-28T22:39:12.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.804+0000 7f4ddad7d740 1 -- 172.21.15.43:0/2164417264 wait complete. 2024-03-28T22:39:12.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.804+0000 7f4ddad7d740 20 client.26336 trim_cache size 0 max 16384 2024-03-28T22:39:12.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.824+0000 7f4ddad7d740 1 Processor -- start 2024-03-28T22:39:12.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.825+0000 7f4ddad7d740 1 -- start start 2024-03-28T22:39:12.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.825+0000 7f4ddad7d740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557cb9823ee0 0x557cb982c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:12.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.825+0000 7f4ddad7d740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557cb9824bc0 0x557cb98416b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:12.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.825+0000 7f4ddad7d740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557cb982d250 0x557cb9820490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:12.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.825+0000 7f4ddad7d740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd01aca50 con 0x557cb9824bc0 2024-03-28T22:39:12.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.825+0000 7f4ddad7d740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd01ae590 con 0x557cb982d250 2024-03-28T22:39:12.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.825+0000 7f4ddad7d740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd01ad120 con 0x557cb9823ee0 2024-03-28T22:39:12.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.825+0000 7f4dd5600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557cb9823ee0 0x557cb982c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:12.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.825+0000 7f4dd5600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557cb9823ee0 0x557cb982c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:42880/0 (socket says 172.21.15.43:42880) 2024-03-28T22:39:12.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.825+0000 7f4dd5600640 1 -- 172.21.15.43:0/3426482732 learned_addr learned my addr 172.21.15.43:0/3426482732 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:12.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.825+0000 7f4dd6000640 1 --2- 172.21.15.43:0/3426482732 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557cb982d250 0x557cb9820490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:12.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.825+0000 7f4dd4c00640 1 --2- 172.21.15.43:0/3426482732 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557cb9824bc0 0x557cb98416b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:12.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.826+0000 7f4dd5600640 1 -- 172.21.15.43:0/3426482732 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557cb982d250 msgr2=0x557cb9820490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:12.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.826+0000 7f4dd5600640 1 --2- 172.21.15.43:0/3426482732 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557cb982d250 0x557cb9820490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:12.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.826+0000 7f4dd5600640 1 -- 172.21.15.43:0/3426482732 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557cb9824bc0 msgr2=0x557cb98416b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:12.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.826+0000 7f4dd5600640 1 --2- 172.21.15.43:0/3426482732 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557cb9824bc0 0x557cb98416b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:12.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.826+0000 7f4dd5600640 1 -- 172.21.15.43:0/3426482732 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557cb949c3f0 con 0x557cb9823ee0 2024-03-28T22:39:12.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.826+0000 7f4dd5600640 1 --2- 172.21.15.43:0/3426482732 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557cb9823ee0 0x557cb982c930 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4009230 tx=0x7f4dc4009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=225b1eab0f0904f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:12.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.826+0000 7f4dc1c00640 1 -- 172.21.15.43:0/3426482732 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc4019bc0 con 0x557cb9823ee0 2024-03-28T22:39:12.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.826+0000 7f4dc1c00640 1 -- 172.21.15.43:0/3426482732 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4dc4019d60 con 0x557cb9823ee0 2024-03-28T22:39:12.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.826+0000 7f4dc1c00640 1 -- 172.21.15.43:0/3426482732 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc400f9c0 con 0x557cb9823ee0 2024-03-28T22:39:12.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.827+0000 7f4ddad7d740 1 -- 172.21.15.43:0/3426482732 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557cb9823ee0 msgr2=0x557cb982c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:12.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.827+0000 7f4ddad7d740 1 --2- 172.21.15.43:0/3426482732 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557cb9823ee0 0x557cb982c930 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4009230 tx=0x7f4dc4009310 comp rx=0 tx=0).stop 2024-03-28T22:39:12.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.827+0000 7f4ddad7d740 1 -- 172.21.15.43:0/3426482732 shutdown_connections 2024-03-28T22:39:12.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.827+0000 7f4ddad7d740 1 --2- 172.21.15.43:0/3426482732 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557cb982d250 0x557cb9820490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:12.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.827+0000 7f4ddad7d740 1 --2- 172.21.15.43:0/3426482732 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557cb9824bc0 0x557cb98416b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:12.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.827+0000 7f4ddad7d740 1 --2- 172.21.15.43:0/3426482732 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557cb9823ee0 0x557cb982c930 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:12.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.827+0000 7f4ddad7d740 1 -- 172.21.15.43:0/3426482732 >> 172.21.15.43:0/3426482732 conn(0x557cb97eaa10 msgr2=0x557cb981fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:12.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.827+0000 7f4ddad7d740 1 -- 172.21.15.43:0/3426482732 shutdown_connections 2024-03-28T22:39:12.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.827+0000 7f4ddad7d740 1 -- 172.21.15.43:0/3426482732 wait complete. 2024-03-28T22:39:12.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.828+0000 7f4ddad7d740 1 -- start start 2024-03-28T22:39:12.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.828+0000 7f4ddad7d740 1 Processor -- start 2024-03-28T22:39:12.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.828+0000 7f4ddad7d740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557cb9823ee0 0x557cb98397d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:12.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.828+0000 7f4ddad7d740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557cb9824bc0 0x557cb9839d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:12.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.828+0000 7f4ddad7d740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557cb982d250 0x557cb983a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:12.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.828+0000 7f4ddad7d740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd01adec0 con 0x557cb9823ee0 2024-03-28T22:39:12.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.828+0000 7f4ddad7d740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd01ad7f0 con 0x557cb982d250 2024-03-28T22:39:12.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.828+0000 7f4ddad7d740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd01aaa50 con 0x557cb9824bc0 2024-03-28T22:39:12.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.828+0000 7f4ddad7d740 10 client.0 _ll_register_callbacks cb 0x7f4dbc02ca90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:12.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.829+0000 7f4dd4c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557cb9824bc0 0x557cb9839d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:12.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.829+0000 7f4dd4c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557cb9824bc0 0x557cb9839d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:42896/0 (socket says 172.21.15.43:42896) 2024-03-28T22:39:12.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.829+0000 7f4dd4c00640 1 -- 172.21.15.43:0/79614215 learned_addr learned my addr 172.21.15.43:0/79614215 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:12.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.829+0000 7f4dd6000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557cb982d250 0x557cb983a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:12.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.829+0000 7f4dd5600640 1 --2- 172.21.15.43:0/79614215 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557cb9823ee0 0x557cb98397d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:12.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.829+0000 7f4dd4c00640 1 -- 172.21.15.43:0/79614215 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557cb982d250 msgr2=0x557cb983a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:12.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.829+0000 7f4dd4c00640 1 --2- 172.21.15.43:0/79614215 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557cb982d250 0x557cb983a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:12.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.829+0000 7f4dd4c00640 1 -- 172.21.15.43:0/79614215 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557cb9823ee0 msgr2=0x557cb98397d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:12.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.829+0000 7f4dd4c00640 1 --2- 172.21.15.43:0/79614215 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557cb9823ee0 0x557cb98397d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:12.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.829+0000 7f4dd4c00640 1 -- 172.21.15.43:0/79614215 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557cb945c0f0 con 0x557cb9824bc0 2024-03-28T22:39:12.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.829+0000 7f4dd4c00640 1 --2- 172.21.15.43:0/79614215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557cb9824bc0 0x557cb9839d10 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc02c820 tx=0x7f4dbc02bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ba7207403ed28bc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:12.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.829+0000 7f4db6a00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:39:12.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.829+0000 7f4db6a00640 1 -- 172.21.15.43:0/79614215 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc0bc6a0 con 0x557cb9824bc0 2024-03-28T22:39:12.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.830+0000 7f4db6a00640 1 -- 172.21.15.43:0/79614215 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f4dbc0c5070 con 0x557cb9824bc0 2024-03-28T22:39:12.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.830+0000 7f4ddad7d740 1 -- 172.21.15.43:0/79614215 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4dc4019d60 con 0x557cb9824bc0 2024-03-28T22:39:12.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.830+0000 7f4db6a00640 1 -- 172.21.15.43:0/79614215 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4dbc0edbb0 con 0x557cb9824bc0 2024-03-28T22:39:12.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.830+0000 7f4ddad7d740 10 client.26287 fetch_fsmap learned FSMap version 29 2024-03-28T22:39:12.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.830+0000 7f4ddad7d740 1 -- 172.21.15.43:0/79614215 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557cb98ac510 con 0x557cb9824bc0 2024-03-28T22:39:12.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.830+0000 7f4db6a00640 1 -- 172.21.15.43:0/79614215 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4dc4019d60 con 0x557cb9824bc0 2024-03-28T22:39:12.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.830+0000 7f4db6a00640 1 -- 172.21.15.43:0/79614215 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f4dbc0d0bb0 con 0x557cb9824bc0 2024-03-28T22:39:12.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.830+0000 7f4ddad7d740 10 client.26287 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:39:12.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.830+0000 7f4ddad7d740 10 client.26287 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:39:12.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.831+0000 7f4ddad7d740 20 client.26287 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:12.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.831+0000 7f4ddad7d740 1 --2- 172.21.15.43:0/79614215 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557cb98acac0 0x557cb98aeee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:12.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.831+0000 7f4ddad7d740 4 client.26287 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:39:12.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.831+0000 7f4ddad7d740 1 -- 172.21.15.43:0/79614215 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557cb978f990 con 0x557cb98acac0 2024-03-28T22:39:12.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.831+0000 7f4dd5600640 1 --2- 172.21.15.43:0/79614215 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557cb98acac0 0x557cb98aeee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:12.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.831+0000 7f4dd5600640 1 --2- 172.21.15.43:0/79614215 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557cb98acac0 0x557cb98aeee0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9ab2cf0837d7ec8 server_cookie=659a210d3d78e538 in_seq=0 out_seq=0 2024-03-28T22:39:12.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.831+0000 7f4db6a00640 10 client.26287 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:12.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.833+0000 7f4db6a00640 1 -- 172.21.15.43:0/79614215 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557cb978f990 con 0x557cb98acac0 2024-03-28T22:39:12.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.833+0000 7f4db6a00640 10 client.26287 handle_command_reply: tid=0 2024-03-28T22:39:12.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.833+0000 7f4ddad7d740 1 client.26287 shutdown 2024-03-28T22:39:12.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.833+0000 7f4ddad7d740 1 -- 172.21.15.43:0/79614215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557cb9824bc0 msgr2=0x557cb9839d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:12.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.833+0000 7f4ddad7d740 1 --2- 172.21.15.43:0/79614215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557cb9824bc0 0x557cb9839d10 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc02c820 tx=0x7f4dbc02bc30 comp rx=0 tx=0).stop 2024-03-28T22:39:12.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.833+0000 7f4ddad7d740 1 -- 172.21.15.43:0/79614215 shutdown_connections 2024-03-28T22:39:12.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.833+0000 7f4ddad7d740 1 --2- 172.21.15.43:0/79614215 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557cb98acac0 0x557cb98aeee0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:12.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.834+0000 7f4ddad7d740 1 --2- 172.21.15.43:0/79614215 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557cb982d250 0x557cb983a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:12.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.834+0000 7f4ddad7d740 1 --2- 172.21.15.43:0/79614215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557cb9824bc0 0x557cb9839d10 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:12.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.834+0000 7f4ddad7d740 1 --2- 172.21.15.43:0/79614215 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557cb9823ee0 0x557cb98397d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:12.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.834+0000 7f4db6a00640 0 client.26287 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:12.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.834+0000 7f4ddad7d740 1 -- 172.21.15.43:0/79614215 >> 172.21.15.43:0/79614215 conn(0x557cb97eaa10 msgr2=0x557cb981dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:12.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.834+0000 7f4ddad7d740 1 -- 172.21.15.43:0/79614215 shutdown_connections 2024-03-28T22:39:12.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.834+0000 7f4ddad7d740 1 -- 172.21.15.43:0/79614215 wait complete. 2024-03-28T22:39:12.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.834+0000 7f4ddad7d740 20 client.26287 trim_cache size 0 max 16384 2024-03-28T22:39:12.833 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:39:12.834 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-03-28T22:39:12.834 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-03-28T22:39:12.834 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:39:12.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.835+0000 7f4dd8000640 1 -- 172.21.15.43:0/3069390095 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4db80a1f70 msgr2=0x7f4db80a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:12.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.835+0000 7f4dd8000640 1 --2- 172.21.15.43:0/3069390095 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4db80a1f70 0x7f4db80a4430 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4dc8005fd0 tx=0x7f4dc8009500 comp rx=0 tx=0).stop 2024-03-28T22:39:12.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.835+0000 7f4dd8000640 1 -- 172.21.15.43:0/3069390095 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dd01042b0 msgr2=0x7f4dd019a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:12.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.835+0000 7f4dd8000640 1 --2- 172.21.15.43:0/3069390095 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dd01042b0 0x7f4dd019a7b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc009900 tx=0x7f4dbc004360 comp rx=0 tx=0).stop 2024-03-28T22:39:12.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.835+0000 7f4dd8000640 1 -- 172.21.15.43:0/3069390095 shutdown_connections 2024-03-28T22:39:12.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.835+0000 7f4dd8000640 1 --2- 172.21.15.43:0/3069390095 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4db80a1f70 0x7f4db80a4430 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:12.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.835+0000 7f4dd8000640 1 --2- 172.21.15.43:0/3069390095 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4dd0109900 0x7f4dd019eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:12.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.835+0000 7f4dd8000640 1 --2- 172.21.15.43:0/3069390095 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dd01042b0 0x7f4dd019a7b0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:12.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.835+0000 7f4dd8000640 1 --2- 172.21.15.43:0/3069390095 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4dd01038f0 0x7f4dd019a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:12.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.835+0000 7f4dd8000640 1 -- 172.21.15.43:0/3069390095 >> 172.21.15.43:0/3069390095 conn(0x7f4dd00ff3a0 msgr2=0x7f4dd00ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:12.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.835+0000 7f4dd8000640 1 -- 172.21.15.43:0/3069390095 shutdown_connections 2024-03-28T22:39:12.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:12.835+0000 7f4dd8000640 1 -- 172.21.15.43:0/3069390095 wait complete. 2024-03-28T22:39:12.852 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:39:12.853 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:39:12.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:12 smithi172 ceph-mon[36566]: pgmap v63: 129 pgs: 129 active+clean; 39 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.2 MiB/s wr, 315 op/s 2024-03-28T22:39:12.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:12 smithi082 ceph-mon[36471]: pgmap v63: 129 pgs: 129 active+clean; 39 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.2 MiB/s wr, 315 op/s 2024-03-28T22:39:13.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.071+0000 7f1f6de00640 1 -- 172.21.15.43:0/3742002424 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1f681042c0 msgr2=0x7f1f681107d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:13.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.071+0000 7f1f6de00640 1 --2- 172.21.15.43:0/3742002424 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1f681042c0 0x7f1f681107d0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1f60009900 tx=0x7f1f6002bc30 comp rx=0 tx=0).stop 2024-03-28T22:39:13.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.071+0000 7f1f6de00640 1 -- 172.21.15.43:0/3742002424 shutdown_connections 2024-03-28T22:39:13.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.071+0000 7f1f6de00640 1 --2- 172.21.15.43:0/3742002424 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1f681042c0 0x7f1f681107d0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.071+0000 7f1f6de00640 1 --2- 172.21.15.43:0/3742002424 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f68103900 0x7f1f68103d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.071+0000 7f1f6de00640 1 --2- 172.21.15.43:0/3742002424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f68109910 0x7f1f68109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.071+0000 7f1f6de00640 1 -- 172.21.15.43:0/3742002424 >> 172.21.15.43:0/3742002424 conn(0x7f1f680ff3d0 msgr2=0x7f1f681017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:13.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.071+0000 7f1f6de00640 1 -- 172.21.15.43:0/3742002424 shutdown_connections 2024-03-28T22:39:13.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.071+0000 7f1f6de00640 1 -- 172.21.15.43:0/3742002424 wait complete. 2024-03-28T22:39:13.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.072+0000 7f1f6de00640 1 Processor -- start 2024-03-28T22:39:13.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.072+0000 7f1f6de00640 1 -- start start 2024-03-28T22:39:13.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.073+0000 7f1f6de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f68103900 0x7f1f6819a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.073+0000 7f1f6de00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f681042c0 0x7f1f6819a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.073+0000 7f1f6de00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1f68109910 0x7f1f6819eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.073+0000 7f1f6de00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1f68113d70 con 0x7f1f68103900 2024-03-28T22:39:13.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.073+0000 7f1f6de00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1f68113bf0 con 0x7f1f68109910 2024-03-28T22:39:13.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.073+0000 7f1f6de00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f1f68113ef0 con 0x7f1f681042c0 2024-03-28T22:39:13.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.073+0000 7f1f66000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f681042c0 0x7f1f6819a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.073+0000 7f1f66000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f681042c0 0x7f1f6819a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:42920/0 (socket says 172.21.15.43:42920) 2024-03-28T22:39:13.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.073+0000 7f1f66000640 1 -- 172.21.15.43:0/4191626269 learned_addr learned my addr 172.21.15.43:0/4191626269 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:13.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.073+0000 7f1f67400640 1 --2- 172.21.15.43:0/4191626269 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1f68109910 0x7f1f6819eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.073+0000 7f1f66a00640 1 --2- 172.21.15.43:0/4191626269 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f68103900 0x7f1f6819a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.073+0000 7f1f66000640 1 -- 172.21.15.43:0/4191626269 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1f68109910 msgr2=0x7f1f6819eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:13.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.073+0000 7f1f66000640 1 --2- 172.21.15.43:0/4191626269 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1f68109910 0x7f1f6819eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.073+0000 7f1f66000640 1 -- 172.21.15.43:0/4191626269 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f68103900 msgr2=0x7f1f6819a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:13.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.073+0000 7f1f66000640 1 --2- 172.21.15.43:0/4191626269 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f68103900 0x7f1f6819a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.073+0000 7f1f66000640 1 -- 172.21.15.43:0/4191626269 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f6819f250 con 0x7f1f681042c0 2024-03-28T22:39:13.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.073+0000 7f1f67400640 1 --2- 172.21.15.43:0/4191626269 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1f68109910 0x7f1f6819eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:13.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.074+0000 7f1f66000640 1 --2- 172.21.15.43:0/4191626269 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f681042c0 0x7f1f6819a7f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1f5000ebc0 tx=0x7f1f5000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:13.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.074+0000 7f1f6ca00640 1 -- 172.21.15.43:0/4191626269 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f50010630 con 0x7f1f681042c0 2024-03-28T22:39:13.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.074+0000 7f1f6ca00640 1 -- 172.21.15.43:0/4191626269 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f50010cb0 con 0x7f1f681042c0 2024-03-28T22:39:13.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.074+0000 7f1f6de00640 1 -- 172.21.15.43:0/4191626269 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f6819f540 con 0x7f1f681042c0 2024-03-28T22:39:13.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.074+0000 7f1f6ca00640 1 -- 172.21.15.43:0/4191626269 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f50018bd0 con 0x7f1f681042c0 2024-03-28T22:39:13.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.074+0000 7f1f6de00640 1 -- 172.21.15.43:0/4191626269 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f681a6dd0 con 0x7f1f681042c0 2024-03-28T22:39:13.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.076+0000 7f1f6de00640 1 -- 172.21.15.43:0/4191626269 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f68068990 con 0x7f1f681042c0 2024-03-28T22:39:13.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.077+0000 7f1f6ca00640 1 -- 172.21.15.43:0/4191626269 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f1f500107d0 con 0x7f1f681042c0 2024-03-28T22:39:13.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.078+0000 7f1f6ca00640 1 --2- 172.21.15.43:0/4191626269 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1f4c0a1ef0 0x7f1f4c0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.079+0000 7f1f6ca00640 1 -- 172.21.15.43:0/4191626269 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f1f500ceb90 con 0x7f1f681042c0 2024-03-28T22:39:13.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.079+0000 7f1f66a00640 1 --2- 172.21.15.43:0/4191626269 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1f4c0a1ef0 0x7f1f4c0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.079+0000 7f1f66a00640 1 --2- 172.21.15.43:0/4191626269 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1f4c0a1ef0 0x7f1f4c0a43b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1f58009b50 tx=0x7f1f58005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:13.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.085+0000 7f1f6ca00640 1 -- 172.21.15.43:0/4191626269 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1f5009b230 con 0x7f1f681042c0 2024-03-28T22:39:13.343 INFO:tasks.workunit.client.0.smithi043.stdout: 1 71644 9.17 MB/sec warmup 44 sec latency 27.860 ms 2024-03-28T22:39:13.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.362+0000 7f1f6de00640 1 -- 172.21.15.43:0/4191626269 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f6810be70 con 0x7f1f681042c0 2024-03-28T22:39:13.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.363+0000 7f1f6ca00640 1 -- 172.21.15.43:0/4191626269 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11934 (secure 0 0 0) 0x7f1f5009f2d0 con 0x7f1f681042c0 2024-03-28T22:39:13.361 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:39:13.361 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:33:28.906396+0000","modified":"2024-03-28T22:36:49.597670+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24572,"mds_1":24556,"mds_2":24535,"mds_3":24523,"mds_4":24518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/1084988072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1084988072},{"type":"v1","addr":"172.21.15.82:6835","nonce":1084988072}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24572,24523,24518,24535,24556]},"id":1}]} 2024-03-28T22:39:13.362 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 29 2024-03-28T22:39:13.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.366+0000 7f1f6de00640 1 -- 172.21.15.43:0/4191626269 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1f4c0a1ef0 msgr2=0x7f1f4c0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:13.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.366+0000 7f1f6de00640 1 --2- 172.21.15.43:0/4191626269 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1f4c0a1ef0 0x7f1f4c0a43b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1f58009b50 tx=0x7f1f58005f90 comp rx=0 tx=0).stop 2024-03-28T22:39:13.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.366+0000 7f1f6de00640 1 -- 172.21.15.43:0/4191626269 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f681042c0 msgr2=0x7f1f6819a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:13.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.366+0000 7f1f6de00640 1 --2- 172.21.15.43:0/4191626269 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f681042c0 0x7f1f6819a7f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1f5000ebc0 tx=0x7f1f5000c6a0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.367+0000 7f1f6de00640 1 -- 172.21.15.43:0/4191626269 shutdown_connections 2024-03-28T22:39:13.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.367+0000 7f1f6de00640 1 --2- 172.21.15.43:0/4191626269 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1f4c0a1ef0 0x7f1f4c0a43b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.367+0000 7f1f6de00640 1 --2- 172.21.15.43:0/4191626269 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1f68109910 0x7f1f6819eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.367+0000 7f1f6de00640 1 --2- 172.21.15.43:0/4191626269 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f681042c0 0x7f1f6819a7f0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.367+0000 7f1f6de00640 1 --2- 172.21.15.43:0/4191626269 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1f68103900 0x7f1f6819a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.367+0000 7f1f6de00640 1 -- 172.21.15.43:0/4191626269 >> 172.21.15.43:0/4191626269 conn(0x7f1f680ff3d0 msgr2=0x7f1f680ff9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:13.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.367+0000 7f1f6de00640 1 -- 172.21.15.43:0/4191626269 shutdown_connections 2024-03-28T22:39:13.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.367+0000 7f1f6de00640 1 -- 172.21.15.43:0/4191626269 wait complete. 2024-03-28T22:39:13.384 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T22:39:13.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.618+0000 7f1fae200640 1 -- 172.21.15.43:0/2508084359 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1fa8107660 msgr2=0x7f1fa8107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:13.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.618+0000 7f1fae200640 1 --2- 172.21.15.43:0/2508084359 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1fa8107660 0x7f1fa8107a40 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c009900 tx=0x7f1f9c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:39:13.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.618+0000 7f1fae200640 1 -- 172.21.15.43:0/2508084359 shutdown_connections 2024-03-28T22:39:13.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.618+0000 7f1fae200640 1 --2- 172.21.15.43:0/2508084359 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1fa8101f80 0x7f1fa810e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.618+0000 7f1fae200640 1 --2- 172.21.15.43:0/2508084359 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1fa81015c0 0x7f1fa8101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.618+0000 7f1fae200640 1 --2- 172.21.15.43:0/2508084359 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1fa8107660 0x7f1fa8107a40 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.618+0000 7f1fae200640 1 -- 172.21.15.43:0/2508084359 >> 172.21.15.43:0/2508084359 conn(0x7f1fa80fd0f0 msgr2=0x7f1fa80ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:13.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.619+0000 7f1fae200640 1 -- 172.21.15.43:0/2508084359 shutdown_connections 2024-03-28T22:39:13.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.619+0000 7f1fae200640 1 -- 172.21.15.43:0/2508084359 wait complete. 2024-03-28T22:39:13.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.620+0000 7f1fae200640 1 Processor -- start 2024-03-28T22:39:13.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.620+0000 7f1fae200640 1 -- start start 2024-03-28T22:39:13.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.621+0000 7f1fae200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1fa81015c0 0x7f1fa8073a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.621+0000 7f1fae200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1fa8101f80 0x7f1fa8073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.621+0000 7f1fae200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1fa8107660 0x7f1fa80744d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.621+0000 7f1fae200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1fa81119f0 con 0x7f1fa81015c0 2024-03-28T22:39:13.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.621+0000 7f1fae200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1fa8111870 con 0x7f1fa8101f80 2024-03-28T22:39:13.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.621+0000 7f1fae200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f1fa8111b70 con 0x7f1fa8107660 2024-03-28T22:39:13.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.621+0000 7f1fa7e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1fa8101f80 0x7f1fa8073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.621+0000 7f1face00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1fa81015c0 0x7f1fa8073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.621+0000 7f1fad800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1fa8107660 0x7f1fa80744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.621+0000 7f1fa7e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1fa8101f80 0x7f1fa8073f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59356/0 (socket says 172.21.15.43:59356) 2024-03-28T22:39:13.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.621+0000 7f1fa7e00640 1 -- 172.21.15.43:0/4048610403 learned_addr learned my addr 172.21.15.43:0/4048610403 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:13.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.622+0000 7f1fa7e00640 1 -- 172.21.15.43:0/4048610403 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1fa8107660 msgr2=0x7f1fa80744d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:13.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.622+0000 7f1fa7e00640 1 --2- 172.21.15.43:0/4048610403 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1fa8107660 0x7f1fa80744d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.622+0000 7f1fa7e00640 1 -- 172.21.15.43:0/4048610403 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1fa81015c0 msgr2=0x7f1fa8073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:13.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.622+0000 7f1fa7e00640 1 --2- 172.21.15.43:0/4048610403 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1fa81015c0 0x7f1fa8073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.622+0000 7f1fa7e00640 1 -- 172.21.15.43:0/4048610403 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fa81a62b0 con 0x7f1fa8101f80 2024-03-28T22:39:13.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.622+0000 7f1fad800640 1 --2- 172.21.15.43:0/4048610403 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1fa8107660 0x7f1fa80744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:13.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.622+0000 7f1fa7e00640 1 --2- 172.21.15.43:0/4048610403 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1fa8101f80 0x7f1fa8073f90 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f1f9400e9f0 tx=0x7f1f9400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:13.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.622+0000 7f1fa5600640 1 -- 172.21.15.43:0/4048610403 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f9400cd80 con 0x7f1fa8101f80 2024-03-28T22:39:13.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.622+0000 7f1fa5600640 1 -- 172.21.15.43:0/4048610403 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f94010910 con 0x7f1fa8101f80 2024-03-28T22:39:13.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.622+0000 7f1fa5600640 1 -- 172.21.15.43:0/4048610403 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f94018990 con 0x7f1fa8101f80 2024-03-28T22:39:13.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.622+0000 7f1fae200640 1 -- 172.21.15.43:0/4048610403 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fa81a65a0 con 0x7f1fa8101f80 2024-03-28T22:39:13.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.623+0000 7f1fae200640 1 -- 172.21.15.43:0/4048610403 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fa81a6b40 con 0x7f1fa8101f80 2024-03-28T22:39:13.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.627+0000 7f1fa5600640 1 -- 172.21.15.43:0/4048610403 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f1f94010ac0 con 0x7f1fa8101f80 2024-03-28T22:39:13.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.627+0000 7f1fb2629740 1 Processor -- start 2024-03-28T22:39:13.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.627+0000 7f1fb2629740 1 -- start start 2024-03-28T22:39:13.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.628+0000 7f1fb2629740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56408a1f6550 0x56408a1f6930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.628+0000 7f1fb2629740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56408a1f6e70 0x56408a1ff510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.628+0000 7f1fb2629740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56408a1ffa50 0x56408a201e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.628+0000 7f1fb2629740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56408a15a050 con 0x56408a1f6e70 2024-03-28T22:39:13.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.628+0000 7f1fb2629740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x56408a102ce0 con 0x56408a1ffa50 2024-03-28T22:39:13.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.628+0000 7f1fb2629740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x56408a1560f0 con 0x56408a1f6550 2024-03-28T22:39:13.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.628+0000 7f1fad800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56408a1f6550 0x56408a1f6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.628+0000 7f1fad800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56408a1f6550 0x56408a1f6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:42948/0 (socket says 172.21.15.43:42948) 2024-03-28T22:39:13.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.628+0000 7f1fad800640 1 -- 172.21.15.43:0/3559855335 learned_addr learned my addr 172.21.15.43:0/3559855335 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:13.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.628+0000 7f1fa7e00640 1 --2- 172.21.15.43:0/3559855335 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56408a1ffa50 0x56408a201e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.628+0000 7f1face00640 1 --2- 172.21.15.43:0/3559855335 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56408a1f6e70 0x56408a1ff510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.629+0000 7f1fad800640 1 -- 172.21.15.43:0/3559855335 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56408a1ffa50 msgr2=0x56408a201e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:13.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.629+0000 7f1fad800640 1 --2- 172.21.15.43:0/3559855335 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56408a1ffa50 0x56408a201e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.629+0000 7f1fad800640 1 -- 172.21.15.43:0/3559855335 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56408a1f6e70 msgr2=0x56408a1ff510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:13.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.629+0000 7f1fad800640 1 --2- 172.21.15.43:0/3559855335 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56408a1f6e70 0x56408a1ff510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.629+0000 7f1fad800640 1 -- 172.21.15.43:0/3559855335 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564089e6e3f0 con 0x56408a1f6550 2024-03-28T22:39:13.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.629+0000 7f1fad800640 1 --2- 172.21.15.43:0/3559855335 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56408a1f6550 0x56408a1f6930 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0011790 tx=0x7f1fa0011c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=561fced12c085c00 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:13.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.629+0000 7f1fa5600640 1 --2- 172.21.15.43:0/4048610403 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1f900a1fa0 0x7f1f900a4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.629+0000 7f1fa5600640 1 -- 172.21.15.43:0/4048610403 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f1f940ce7a0 con 0x7f1fa8101f80 2024-03-28T22:39:13.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.629+0000 7f1f9a000640 1 -- 172.21.15.43:0/3559855335 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa0003c30 con 0x56408a1f6550 2024-03-28T22:39:13.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.629+0000 7f1face00640 1 --2- 172.21.15.43:0/4048610403 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1f900a1fa0 0x7f1f900a4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.629+0000 7f1f9a000640 1 -- 172.21.15.43:0/3559855335 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fa000fe80 con 0x56408a1f6550 2024-03-28T22:39:13.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.629+0000 7f1f9a000640 1 -- 172.21.15.43:0/3559855335 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa00193e0 con 0x56408a1f6550 2024-03-28T22:39:13.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.630+0000 7f1fb2629740 1 -- 172.21.15.43:0/3559855335 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56408a1f6550 msgr2=0x56408a1f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:13.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.630+0000 7f1fb2629740 1 --2- 172.21.15.43:0/3559855335 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56408a1f6550 0x56408a1f6930 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0011790 tx=0x7f1fa0011c50 comp rx=0 tx=0).stop 2024-03-28T22:39:13.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.631+0000 7f1face00640 1 --2- 172.21.15.43:0/4048610403 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1f900a1fa0 0x7f1f900a4460 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c02c240 tx=0x7f1f9c02e040 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:13.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.631+0000 7f1fb2629740 1 -- 172.21.15.43:0/3559855335 shutdown_connections 2024-03-28T22:39:13.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.631+0000 7f1fb2629740 1 --2- 172.21.15.43:0/3559855335 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56408a1ffa50 0x56408a201e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.632+0000 7f1fb2629740 1 --2- 172.21.15.43:0/3559855335 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56408a1f6e70 0x56408a1ff510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.632+0000 7f1fb2629740 1 --2- 172.21.15.43:0/3559855335 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56408a1f6550 0x56408a1f6930 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.632+0000 7f1fb2629740 1 -- 172.21.15.43:0/3559855335 >> 172.21.15.43:0/3559855335 conn(0x56408a1f0c80 msgr2=0x56408a1f1060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:13.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.632+0000 7f1fb2629740 1 -- 172.21.15.43:0/3559855335 shutdown_connections 2024-03-28T22:39:13.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.632+0000 7f1fb2629740 1 -- 172.21.15.43:0/3559855335 wait complete. 2024-03-28T22:39:13.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.633+0000 7f1fb2629740 1 -- start start 2024-03-28T22:39:13.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.633+0000 7f1fb2629740 1 Processor -- start 2024-03-28T22:39:13.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.634+0000 7f1fb2629740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56408a1f6550 0x56408a1fcb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.634+0000 7f1fb2629740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56408a1f6e70 0x56408a1fd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.634+0000 7f1fb2629740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56408a1ffa50 0x56408a1fd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.634+0000 7f1fb2629740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56408a202840 con 0x56408a1ffa50 2024-03-28T22:39:13.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.634+0000 7f1fb2629740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x56408a0f82f0 con 0x56408a1f6550 2024-03-28T22:39:13.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.634+0000 7f1fb2629740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x56408a2029c0 con 0x56408a1f6e70 2024-03-28T22:39:13.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.634+0000 7f1fb2629740 10 client.0 _ll_register_callbacks cb 0x564089e6dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:13.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.634+0000 7f1fad800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56408a1ffa50 0x56408a1fd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.634+0000 7f1face00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56408a1f6550 0x56408a1fcb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.634+0000 7f1fad800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56408a1ffa50 0x56408a1fd5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53308/0 (socket says 172.21.15.43:53308) 2024-03-28T22:39:13.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.634+0000 7f1fad800640 1 -- 172.21.15.43:0/602793925 learned_addr learned my addr 172.21.15.43:0/602793925 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:13.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.634+0000 7f1fa7e00640 1 --2- 172.21.15.43:0/602793925 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56408a1f6e70 0x56408a1fd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.635+0000 7f1fad800640 1 -- 172.21.15.43:0/602793925 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56408a1f6e70 msgr2=0x56408a1fd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:13.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.635+0000 7f1fad800640 1 --2- 172.21.15.43:0/602793925 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56408a1f6e70 0x56408a1fd080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.635+0000 7f1fad800640 1 -- 172.21.15.43:0/602793925 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56408a1f6550 msgr2=0x56408a1fcb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:13.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.635+0000 7f1fad800640 1 --2- 172.21.15.43:0/602793925 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56408a1f6550 0x56408a1fcb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.635+0000 7f1fad800640 1 -- 172.21.15.43:0/602793925 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564089f387d0 con 0x56408a1ffa50 2024-03-28T22:39:13.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.635+0000 7f1fad800640 1 --2- 172.21.15.43:0/602793925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56408a1ffa50 0x56408a1fd5c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1fa000a010 tx=0x7f1fa0004970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecca42257254145d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:13.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.635+0000 7f1f8ea00640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:39:13.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.635+0000 7f1f8ea00640 1 -- 172.21.15.43:0/602793925 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa0028020 con 0x56408a1ffa50 2024-03-28T22:39:13.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.635+0000 7f1f8ea00640 1 -- 172.21.15.43:0/602793925 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f1fa0003a30 con 0x56408a1ffa50 2024-03-28T22:39:13.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.635+0000 7f1fb2629740 1 -- 172.21.15.43:0/602793925 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56408a103680 con 0x56408a1ffa50 2024-03-28T22:39:13.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.636+0000 7f1f8ea00640 1 -- 172.21.15.43:0/602793925 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1fa0033050 con 0x56408a1ffa50 2024-03-28T22:39:13.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.636+0000 7f1fb2629740 10 client.16182 fetch_fsmap learned FSMap version 29 2024-03-28T22:39:13.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.636+0000 7f1fb2629740 1 -- 172.21.15.43:0/602793925 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56408a2173a0 con 0x56408a1ffa50 2024-03-28T22:39:13.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.636+0000 7f1f8ea00640 1 -- 172.21.15.43:0/602793925 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56408a103680 con 0x56408a1ffa50 2024-03-28T22:39:13.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.636+0000 7f1f8ea00640 1 -- 172.21.15.43:0/602793925 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f1fa0018070 con 0x56408a1ffa50 2024-03-28T22:39:13.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.636+0000 7f1fb2629740 10 client.16182 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:39:13.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.636+0000 7f1fb2629740 10 client.16182 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:39:13.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.636+0000 7f1fb2629740 20 client.16182 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:13.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.636+0000 7f1fb2629740 1 --2- 172.21.15.43:0/602793925 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x56408a218800 0x56408a21abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.636+0000 7f1fb2629740 4 client.16182 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:39:13.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.636+0000 7f1fb2629740 1 -- 172.21.15.43:0/602793925 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56408a167590 con 0x56408a218800 2024-03-28T22:39:13.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.637+0000 7f1face00640 1 --2- 172.21.15.43:0/602793925 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x56408a218800 0x56408a21abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.638+0000 7f1face00640 1 --2- 172.21.15.43:0/602793925 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x56408a218800 0x56408a21abf0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4dbb3219c2c0de33 server_cookie=ddbe5d4c02c84fe9 in_seq=0 out_seq=0 2024-03-28T22:39:13.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.638+0000 7f1f8ea00640 10 client.16182 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:39:13.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.639+0000 7f1f8ea00640 1 -- 172.21.15.43:0/602793925 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56408a167590 con 0x56408a218800 2024-03-28T22:39:13.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.639+0000 7f1f8ea00640 10 client.16182 handle_command_reply: tid=0 2024-03-28T22:39:13.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.639+0000 7f1fb2629740 1 client.16182 shutdown 2024-03-28T22:39:13.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.639+0000 7f1fb2629740 1 -- 172.21.15.43:0/602793925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56408a1ffa50 msgr2=0x56408a1fd5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:13.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.639+0000 7f1fb2629740 1 --2- 172.21.15.43:0/602793925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56408a1ffa50 0x56408a1fd5c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1fa000a010 tx=0x7f1fa0004970 comp rx=0 tx=0).stop 2024-03-28T22:39:13.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.639+0000 7f1fb2629740 1 -- 172.21.15.43:0/602793925 shutdown_connections 2024-03-28T22:39:13.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.639+0000 7f1fb2629740 1 --2- 172.21.15.43:0/602793925 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x56408a218800 0x56408a21abf0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.639+0000 7f1fb2629740 1 --2- 172.21.15.43:0/602793925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56408a1ffa50 0x56408a1fd5c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.639+0000 7f1fb2629740 1 --2- 172.21.15.43:0/602793925 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56408a1f6e70 0x56408a1fd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.639+0000 7f1fb2629740 1 --2- 172.21.15.43:0/602793925 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56408a1f6550 0x56408a1fcb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.639+0000 7f1fb2629740 1 -- 172.21.15.43:0/602793925 >> 172.21.15.43:0/602793925 conn(0x56408a1f0c80 msgr2=0x56408a1f29d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:13.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.639+0000 7f1f8ea00640 0 client.16182 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:39:13.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.640+0000 7f1fb2629740 1 -- 172.21.15.43:0/602793925 shutdown_connections 2024-03-28T22:39:13.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.640+0000 7f1fb2629740 1 -- 172.21.15.43:0/602793925 wait complete. 2024-03-28T22:39:13.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.640+0000 7f1fb2629740 20 client.16182 trim_cache size 0 max 16384 2024-03-28T22:39:13.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.658+0000 7f1fb2629740 1 Processor -- start 2024-03-28T22:39:13.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.659+0000 7f1fb2629740 1 -- start start 2024-03-28T22:39:13.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.659+0000 7f1fb2629740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56408a1f6550 0x56408a213e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.659+0000 7f1fb2629740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56408a1f6e70 0x56408a1fe970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.659+0000 7f1fb2629740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56408a1ffa50 0x56408a2203b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.659+0000 7f1fb2629740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1fa81ac860 con 0x56408a1f6550 2024-03-28T22:39:13.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.659+0000 7f1fb2629740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1fa81ae3a0 con 0x56408a1f6e70 2024-03-28T22:39:13.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.659+0000 7f1fb2629740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f1fa81acf30 con 0x56408a1ffa50 2024-03-28T22:39:13.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.659+0000 7f1fad800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56408a1ffa50 0x56408a2203b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.659+0000 7f1fad800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56408a1ffa50 0x56408a2203b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:42962/0 (socket says 172.21.15.43:42962) 2024-03-28T22:39:13.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.659+0000 7f1fad800640 1 -- 172.21.15.43:0/201787249 learned_addr learned my addr 172.21.15.43:0/201787249 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:13.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.659+0000 7f1face00640 1 --2- 172.21.15.43:0/201787249 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56408a1f6550 0x56408a213e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.660+0000 7f1fa7e00640 1 --2- 172.21.15.43:0/201787249 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56408a1f6e70 0x56408a1fe970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.660+0000 7f1fad800640 1 -- 172.21.15.43:0/201787249 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56408a1f6e70 msgr2=0x56408a1fe970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:13.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.660+0000 7f1fad800640 1 --2- 172.21.15.43:0/201787249 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56408a1f6e70 0x56408a1fe970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.660+0000 7f1fad800640 1 -- 172.21.15.43:0/201787249 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56408a1f6550 msgr2=0x56408a213e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:13.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.660+0000 7f1fad800640 1 --2- 172.21.15.43:0/201787249 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56408a1f6550 0x56408a213e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.660+0000 7f1fad800640 1 -- 172.21.15.43:0/201787249 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56408a1f1850 con 0x56408a1ffa50 2024-03-28T22:39:13.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.660+0000 7f1fa7e00640 1 --2- 172.21.15.43:0/201787249 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56408a1f6e70 0x56408a1fe970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:39:13.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.660+0000 7f1face00640 1 --2- 172.21.15.43:0/201787249 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56408a1f6550 0x56408a213e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:39:13.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.660+0000 7f1fad800640 1 --2- 172.21.15.43:0/201787249 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56408a1ffa50 0x56408a2203b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1fa001a890 tx=0x7f1fa0009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef373c3fe9cff203 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:13.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.660+0000 7f1f99600640 1 -- 172.21.15.43:0/201787249 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa0028040 con 0x56408a1ffa50 2024-03-28T22:39:13.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.660+0000 7f1f99600640 1 -- 172.21.15.43:0/201787249 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fa0011400 con 0x56408a1ffa50 2024-03-28T22:39:13.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.660+0000 7f1f99600640 1 -- 172.21.15.43:0/201787249 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa001f040 con 0x56408a1ffa50 2024-03-28T22:39:13.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.661+0000 7f1fb2629740 1 -- 172.21.15.43:0/201787249 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56408a1ffa50 msgr2=0x56408a2203b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:13.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.661+0000 7f1fb2629740 1 --2- 172.21.15.43:0/201787249 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56408a1ffa50 0x56408a2203b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1fa001a890 tx=0x7f1fa0009500 comp rx=0 tx=0).stop 2024-03-28T22:39:13.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.661+0000 7f1fb2629740 1 -- 172.21.15.43:0/201787249 shutdown_connections 2024-03-28T22:39:13.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.661+0000 7f1fb2629740 1 --2- 172.21.15.43:0/201787249 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56408a1f6e70 0x56408a1fe970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.661+0000 7f1fb2629740 1 --2- 172.21.15.43:0/201787249 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56408a1ffa50 0x56408a2203b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.661+0000 7f1fb2629740 1 --2- 172.21.15.43:0/201787249 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56408a1f6550 0x56408a213e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.661+0000 7f1fb2629740 1 -- 172.21.15.43:0/201787249 >> 172.21.15.43:0/201787249 conn(0x56408a1f0c80 msgr2=0x56408a1fe260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:13.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.661+0000 7f1fb2629740 1 -- 172.21.15.43:0/201787249 shutdown_connections 2024-03-28T22:39:13.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.662+0000 7f1fb2629740 1 -- 172.21.15.43:0/201787249 wait complete. 2024-03-28T22:39:13.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.662+0000 7f1fb2629740 1 -- start start 2024-03-28T22:39:13.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.662+0000 7f1fb2629740 1 Processor -- start 2024-03-28T22:39:13.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.663+0000 7f1fb2629740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56408a1f6550 0x56408a210690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.663+0000 7f1fb2629740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56408a1f6e70 0x56408a1f91a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.663+0000 7f1fb2629740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56408a1ffa50 0x56408a1f96e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.663+0000 7f1fb2629740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1fa81adcd0 con 0x56408a1ffa50 2024-03-28T22:39:13.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.663+0000 7f1fb2629740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1fa81ad600 con 0x56408a1f6550 2024-03-28T22:39:13.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.663+0000 7f1fb2629740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f1fa81aa860 con 0x56408a1f6e70 2024-03-28T22:39:13.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.663+0000 7f1fb2629740 10 client.0 _ll_register_callbacks cb 0x7f1f940d4420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:13.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.663+0000 7f1face00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56408a1f6550 0x56408a210690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.663+0000 7f1face00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56408a1f6550 0x56408a210690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59400/0 (socket says 172.21.15.43:59400) 2024-03-28T22:39:13.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.663+0000 7f1face00640 1 -- 172.21.15.43:0/1063104270 learned_addr learned my addr 172.21.15.43:0/1063104270 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:13.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.663+0000 7f1fa7e00640 1 --2- 172.21.15.43:0/1063104270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56408a1f6e70 0x56408a1f91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.663+0000 7f1fad800640 1 --2- 172.21.15.43:0/1063104270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56408a1ffa50 0x56408a1f96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.664+0000 7f1face00640 1 -- 172.21.15.43:0/1063104270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56408a1f6e70 msgr2=0x56408a1f91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:13.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.664+0000 7f1face00640 1 --2- 172.21.15.43:0/1063104270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56408a1f6e70 0x56408a1f91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.664+0000 7f1face00640 1 -- 172.21.15.43:0/1063104270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56408a1ffa50 msgr2=0x56408a1f96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:13.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.664+0000 7f1face00640 1 --2- 172.21.15.43:0/1063104270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56408a1ffa50 0x56408a1f96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.664+0000 7f1face00640 1 -- 172.21.15.43:0/1063104270 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56408a1f9cd0 con 0x56408a1f6550 2024-03-28T22:39:13.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.664+0000 7f1fa7e00640 1 --2- 172.21.15.43:0/1063104270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56408a1f6e70 0x56408a1f91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:13.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.664+0000 7f1face00640 1 --2- 172.21.15.43:0/1063104270 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56408a1f6550 0x56408a210690 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c02e010 tx=0x7f1f9c02bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78dd0b698cbe0d37 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:13.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.664+0000 7f1f8ea00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:39:13.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.664+0000 7f1f8ea00640 1 -- 172.21.15.43:0/1063104270 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c055d10 con 0x56408a1f6550 2024-03-28T22:39:13.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.664+0000 7f1fb2629740 1 -- 172.21.15.43:0/1063104270 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fa0011400 con 0x56408a1f6550 2024-03-28T22:39:13.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.664+0000 7f1f8ea00640 1 -- 172.21.15.43:0/1063104270 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f1f9c02c8b0 con 0x56408a1f6550 2024-03-28T22:39:13.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.665+0000 7f1f8ea00640 1 -- 172.21.15.43:0/1063104270 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1f9c02cab0 con 0x56408a1f6550 2024-03-28T22:39:13.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.665+0000 7f1fb2629740 10 client.26363 fetch_fsmap learned FSMap version 29 2024-03-28T22:39:13.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.665+0000 7f1fb2629740 1 -- 172.21.15.43:0/1063104270 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56408a27e7d0 con 0x56408a1f6550 2024-03-28T22:39:13.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.665+0000 7f1f8ea00640 1 -- 172.21.15.43:0/1063104270 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fa0011400 con 0x56408a1f6550 2024-03-28T22:39:13.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.665+0000 7f1f8ea00640 1 -- 172.21.15.43:0/1063104270 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f1f9c004910 con 0x56408a1f6550 2024-03-28T22:39:13.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.665+0000 7f1fb2629740 10 client.26363 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:39:13.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.665+0000 7f1fb2629740 10 client.26363 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:39:13.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.665+0000 7f1fb2629740 20 client.26363 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:13.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.665+0000 7f1fb2629740 1 --2- 172.21.15.43:0/1063104270 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x56408a27ef20 0x56408a281360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.665+0000 7f1fb2629740 4 client.26363 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:39:13.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.665+0000 7f1fb2629740 1 -- 172.21.15.43:0/1063104270 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56408a1618e0 con 0x56408a27ef20 2024-03-28T22:39:13.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.666+0000 7f1fa7e00640 1 --2- 172.21.15.43:0/1063104270 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x56408a27ef20 0x56408a281360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.666+0000 7f1fa7e00640 1 --2- 172.21.15.43:0/1063104270 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x56408a27ef20 0x56408a281360 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9307001215119d8f server_cookie=59837d3d291a0eb2 in_seq=0 out_seq=0 2024-03-28T22:39:13.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.666+0000 7f1f8ea00640 10 client.26363 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:39:13.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.666+0000 7f1f8ea00640 1 -- 172.21.15.43:0/1063104270 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56408a1618e0 con 0x56408a27ef20 2024-03-28T22:39:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.666+0000 7f1f8ea00640 10 client.26363 handle_command_reply: tid=0 2024-03-28T22:39:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.666+0000 7f1fb2629740 1 client.26363 shutdown 2024-03-28T22:39:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.666+0000 7f1fb2629740 1 -- 172.21.15.43:0/1063104270 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56408a1f6550 msgr2=0x56408a210690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.666+0000 7f1fb2629740 1 --2- 172.21.15.43:0/1063104270 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56408a1f6550 0x56408a210690 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c02e010 tx=0x7f1f9c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:39:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.666+0000 7f1fb2629740 1 -- 172.21.15.43:0/1063104270 shutdown_connections 2024-03-28T22:39:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.666+0000 7f1fb2629740 1 --2- 172.21.15.43:0/1063104270 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x56408a27ef20 0x56408a281360 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.666+0000 7f1fb2629740 1 --2- 172.21.15.43:0/1063104270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56408a1ffa50 0x56408a1f96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.667+0000 7f1fb2629740 1 --2- 172.21.15.43:0/1063104270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56408a1f6e70 0x56408a1f91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.667+0000 7f1f8ea00640 0 client.26363 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:39:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.667+0000 7f1fb2629740 1 --2- 172.21.15.43:0/1063104270 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56408a1f6550 0x56408a210690 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.667+0000 7f1fb2629740 1 -- 172.21.15.43:0/1063104270 >> 172.21.15.43:0/1063104270 conn(0x56408a1f0c80 msgr2=0x56408a1efdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.667+0000 7f1fb2629740 1 -- 172.21.15.43:0/1063104270 shutdown_connections 2024-03-28T22:39:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.667+0000 7f1fb2629740 1 -- 172.21.15.43:0/1063104270 wait complete. 2024-03-28T22:39:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.667+0000 7f1fb2629740 20 client.26363 trim_cache size 0 max 16384 2024-03-28T22:39:13.669 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:39:13.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.668+0000 7f1fae200640 1 -- 172.21.15.43:0/4048610403 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1f900a1fa0 msgr2=0x7f1f900a4460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:13.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.668+0000 7f1fae200640 1 --2- 172.21.15.43:0/4048610403 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1f900a1fa0 0x7f1f900a4460 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c02c240 tx=0x7f1f9c02e040 comp rx=0 tx=0).stop 2024-03-28T22:39:13.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.668+0000 7f1fae200640 1 -- 172.21.15.43:0/4048610403 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1fa8101f80 msgr2=0x7f1fa8073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:13.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.668+0000 7f1fae200640 1 --2- 172.21.15.43:0/4048610403 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1fa8101f80 0x7f1fa8073f90 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f1f9400e9f0 tx=0x7f1f9400eeb0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.668+0000 7f1fae200640 1 -- 172.21.15.43:0/4048610403 shutdown_connections 2024-03-28T22:39:13.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.668+0000 7f1fae200640 1 --2- 172.21.15.43:0/4048610403 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1f900a1fa0 0x7f1f900a4460 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.668+0000 7f1fae200640 1 --2- 172.21.15.43:0/4048610403 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1fa8101f80 0x7f1fa8073f90 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.668+0000 7f1fae200640 1 --2- 172.21.15.43:0/4048610403 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1fa8107660 0x7f1fa80744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.668+0000 7f1fae200640 1 --2- 172.21.15.43:0/4048610403 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1fa81015c0 0x7f1fa8073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.668+0000 7f1fae200640 1 -- 172.21.15.43:0/4048610403 >> 172.21.15.43:0/4048610403 conn(0x7f1fa80fd0f0 msgr2=0x7f1fa80fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:13.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.668+0000 7f1fae200640 1 -- 172.21.15.43:0/4048610403 shutdown_connections 2024-03-28T22:39:13.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.668+0000 7f1fae200640 1 -- 172.21.15.43:0/4048610403 wait complete. 2024-03-28T22:39:13.681 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:13 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/4191626269' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:39:13.684 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T22:39:13.690 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.689+0000 7f8049800640 1 -- 172.21.15.43:0/1804159127 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f80441125c0 msgr2=0x7f80441149b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:13.691 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.689+0000 7f8049800640 1 --2- 172.21.15.43:0/1804159127 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f80441125c0 0x7f80441149b0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f803c009cd0 tx=0x7f803c02c0f0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.691 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.689+0000 7f8049800640 1 -- 172.21.15.43:0/1804159127 shutdown_connections 2024-03-28T22:39:13.691 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.689+0000 7f8049800640 1 --2- 172.21.15.43:0/1804159127 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f80441125c0 0x7f80441149b0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.691 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.689+0000 7f8049800640 1 --2- 172.21.15.43:0/1804159127 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80440765a0 0x7f8044112080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.691 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.689+0000 7f8049800640 1 --2- 172.21.15.43:0/1804159127 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8044075c80 0x7f8044076060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.691 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.689+0000 7f8049800640 1 -- 172.21.15.43:0/1804159127 >> 172.21.15.43:0/1804159127 conn(0x7f80440ff410 msgr2=0x7f8044101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.689+0000 7f8049800640 1 -- 172.21.15.43:0/1804159127 shutdown_connections 2024-03-28T22:39:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.689+0000 7f8049800640 1 -- 172.21.15.43:0/1804159127 wait complete. 2024-03-28T22:39:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.690+0000 7f8049800640 1 Processor -- start 2024-03-28T22:39:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.690+0000 7f8049800640 1 -- start start 2024-03-28T22:39:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.691+0000 7f8049800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8044075c80 0x7f804419e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.691+0000 7f8049800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f80440765a0 0x7f804419eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.691+0000 7f8049800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80441125c0 0x7f80441a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.691+0000 7f8049800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f80441180e0 con 0x7f80441125c0 2024-03-28T22:39:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.691+0000 7f8049800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8044117f60 con 0x7f8044075c80 2024-03-28T22:39:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.691+0000 7f8049800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f8044118260 con 0x7f80440765a0 2024-03-28T22:39:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.691+0000 7f8042a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8044075c80 0x7f804419e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.691+0000 7f8042a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8044075c80 0x7f804419e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59426/0 (socket says 172.21.15.43:59426) 2024-03-28T22:39:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.691+0000 7f8042a00640 1 -- 172.21.15.43:0/3489823456 learned_addr learned my addr 172.21.15.43:0/3489823456 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.691+0000 7f8042000640 1 --2- 172.21.15.43:0/3489823456 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f80440765a0 0x7f804419eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.692+0000 7f8042a00640 1 -- 172.21.15.43:0/3489823456 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f80440765a0 msgr2=0x7f804419eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.692+0000 7f8042a00640 1 --2- 172.21.15.43:0/3489823456 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f80440765a0 0x7f804419eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.692+0000 7f8042a00640 1 -- 172.21.15.43:0/3489823456 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80441125c0 msgr2=0x7f80441a2e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T22:39:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.692+0000 7f8042a00640 1 --2- 172.21.15.43:0/3489823456 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80441125c0 0x7f80441a2e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.692+0000 7f8042a00640 1 -- 172.21.15.43:0/3489823456 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80441a3550 con 0x7f8044075c80 2024-03-28T22:39:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.692+0000 7f8042a00640 1 --2- 172.21.15.43:0/3489823456 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8044075c80 0x7f804419e560 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f803400ea60 tx=0x7f803400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.693+0000 7f803b400640 1 -- 172.21.15.43:0/3489823456 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f803400cde0 con 0x7f8044075c80 2024-03-28T22:39:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.693+0000 7f803b400640 1 -- 172.21.15.43:0/3489823456 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80340035f0 con 0x7f8044075c80 2024-03-28T22:39:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.693+0000 7f803b400640 1 -- 172.21.15.43:0/3489823456 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8034010690 con 0x7f8044075c80 2024-03-28T22:39:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.693+0000 7f8049800640 1 -- 172.21.15.43:0/3489823456 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80441a3840 con 0x7f8044075c80 2024-03-28T22:39:13.694 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.693+0000 7f8049800640 1 -- 172.21.15.43:0/3489823456 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80441ab0d0 con 0x7f8044075c80 2024-03-28T22:39:13.695 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.698+0000 7f803b400640 1 -- 172.21.15.43:0/3489823456 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f8034004020 con 0x7f8044075c80 2024-03-28T22:39:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.699+0000 7f803b400640 1 --2- 172.21.15.43:0/3489823456 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f80280a20f0 0x7f80280a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.699+0000 7f803b400640 1 -- 172.21.15.43:0/3489823456 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f80340ce730 con 0x7f8044075c80 2024-03-28T22:39:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.700+0000 7f8042000640 1 --2- 172.21.15.43:0/3489823456 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f80280a20f0 0x7f80280a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.700+0000 7f804c5ee740 1 Processor -- start 2024-03-28T22:39:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.700+0000 7f804c5ee740 1 -- start start 2024-03-28T22:39:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.700+0000 7f8042000640 1 --2- 172.21.15.43:0/3489823456 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f80280a20f0 0x7f80280a45b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8044078bf0 tx=0x7f803000b390 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:13.699 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.701+0000 7f804c5ee740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d13bcce3a0 0x55d13bcce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.699 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.701+0000 7f804c5ee740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d13bccecc0 0x55d13bcd7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.699 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.701+0000 7f804c5ee740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d13bcd7a90 0x55d13bcd9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.699 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.701+0000 7f804c5ee740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d13bc31640 con 0x55d13bccecc0 2024-03-28T22:39:13.699 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.701+0000 7f804c5ee740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55d13bbda810 con 0x55d13bcce3a0 2024-03-28T22:39:13.699 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.701+0000 7f804c5ee740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55d13bc2d6e0 con 0x55d13bcd7a90 2024-03-28T22:39:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.701+0000 7f8042a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d13bcce3a0 0x55d13bcce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.701+0000 7f8042a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d13bcce3a0 0x55d13bcce780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59442/0 (socket says 172.21.15.43:59442) 2024-03-28T22:39:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.701+0000 7f8042a00640 1 -- 172.21.15.43:0/345660183 learned_addr learned my addr 172.21.15.43:0/345660183 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.701+0000 7f8042000640 1 --2- 172.21.15.43:0/345660183 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d13bccecc0 0x55d13bcd7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.701+0000 7f8043400640 1 --2- 172.21.15.43:0/345660183 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d13bcd7a90 0x55d13bcd9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.701+0000 7f8042a00640 1 -- 172.21.15.43:0/345660183 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d13bcd7a90 msgr2=0x55d13bcd9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.701+0000 7f8042a00640 1 --2- 172.21.15.43:0/345660183 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d13bcd7a90 0x55d13bcd9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.701+0000 7f8042a00640 1 -- 172.21.15.43:0/345660183 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d13bccecc0 msgr2=0x55d13bcd7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.701+0000 7f8042a00640 1 --2- 172.21.15.43:0/345660183 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d13bccecc0 0x55d13bcd7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.701+0000 7f8042a00640 1 -- 172.21.15.43:0/345660183 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d13b946210 con 0x55d13bcce3a0 2024-03-28T22:39:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.702+0000 7f8043400640 1 --2- 172.21.15.43:0/345660183 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d13bcd7a90 0x55d13bcd9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:39:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.703+0000 7f8042a00640 1 --2- 172.21.15.43:0/345660183 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d13bcce3a0 0x55d13bcce780 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f8034097180 tx=0x7f803409ac00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9178fba6f5fbcb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:13.702 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.704+0000 7f802fe00640 1 -- 172.21.15.43:0/345660183 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f803409ee70 con 0x55d13bcce3a0 2024-03-28T22:39:13.702 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.704+0000 7f802fe00640 1 -- 172.21.15.43:0/345660183 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80340a34c0 con 0x55d13bcce3a0 2024-03-28T22:39:13.702 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.704+0000 7f802fe00640 1 -- 172.21.15.43:0/345660183 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80340ce2e0 con 0x55d13bcce3a0 2024-03-28T22:39:13.703 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.706+0000 7f804c5ee740 1 -- 172.21.15.43:0/345660183 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d13bcce3a0 msgr2=0x55d13bcce780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:13.705 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.706+0000 7f804c5ee740 1 --2- 172.21.15.43:0/345660183 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d13bcce3a0 0x55d13bcce780 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f8034097180 tx=0x7f803409ac00 comp rx=0 tx=0).stop 2024-03-28T22:39:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.707+0000 7f804c5ee740 1 -- 172.21.15.43:0/345660183 shutdown_connections 2024-03-28T22:39:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.707+0000 7f804c5ee740 1 --2- 172.21.15.43:0/345660183 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d13bcd7a90 0x55d13bcd9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.707+0000 7f804c5ee740 1 --2- 172.21.15.43:0/345660183 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d13bccecc0 0x55d13bcd7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.707+0000 7f804c5ee740 1 --2- 172.21.15.43:0/345660183 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d13bcce3a0 0x55d13bcce780 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.707+0000 7f804c5ee740 1 -- 172.21.15.43:0/345660183 >> 172.21.15.43:0/345660183 conn(0x55d13bcc8cc0 msgr2=0x55d13bcc90a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.707+0000 7f804c5ee740 1 -- 172.21.15.43:0/345660183 shutdown_connections 2024-03-28T22:39:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.707+0000 7f804c5ee740 1 -- 172.21.15.43:0/345660183 wait complete. 2024-03-28T22:39:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.708+0000 7f804c5ee740 1 -- start start 2024-03-28T22:39:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.709+0000 7f804c5ee740 1 Processor -- start 2024-03-28T22:39:13.707 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.709+0000 7f804c5ee740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d13bcce3a0 0x55d13bcd4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.707 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.709+0000 7f804c5ee740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d13bccecc0 0x55d13bcd5070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.707 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.709+0000 7f804c5ee740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d13bcd7a90 0x55d13bcd55b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.708 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.709+0000 7f804c5ee740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d13bceb6d0 con 0x55d13bccecc0 2024-03-28T22:39:13.708 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.709+0000 7f804c5ee740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55d13bbcfd00 con 0x55d13bcd7a90 2024-03-28T22:39:13.708 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.709+0000 7f804c5ee740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55d13bceb850 con 0x55d13bcce3a0 2024-03-28T22:39:13.708 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.709+0000 7f804c5ee740 10 client.0 _ll_register_callbacks cb 0x55d13bbba490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:13.708 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.709+0000 7f8042000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d13bccecc0 0x55d13bcd5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.708 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.710+0000 7f8042000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d13bccecc0 0x55d13bcd5070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53344/0 (socket says 172.21.15.43:53344) 2024-03-28T22:39:13.708 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.710+0000 7f8042000640 1 -- 172.21.15.43:0/2984263377 learned_addr learned my addr 172.21.15.43:0/2984263377 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:13.708 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.710+0000 7f8043400640 1 --2- 172.21.15.43:0/2984263377 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d13bcd7a90 0x55d13bcd55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.708 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.710+0000 7f8042a00640 1 --2- 172.21.15.43:0/2984263377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d13bcce3a0 0x55d13bcd4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.709 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.710+0000 7f8042000640 1 -- 172.21.15.43:0/2984263377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d13bcce3a0 msgr2=0x55d13bcd4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:13.709 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.710+0000 7f8042000640 1 --2- 172.21.15.43:0/2984263377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d13bcce3a0 0x55d13bcd4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.709 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.710+0000 7f8042000640 1 -- 172.21.15.43:0/2984263377 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d13bcd7a90 msgr2=0x55d13bcd55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:13.709 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.710+0000 7f8042000640 1 --2- 172.21.15.43:0/2984263377 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d13bcd7a90 0x55d13bcd55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.709 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.710+0000 7f8042000640 1 -- 172.21.15.43:0/2984263377 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d13ba12620 con 0x55d13bccecc0 2024-03-28T22:39:13.709 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.711+0000 7f8042000640 1 --2- 172.21.15.43:0/2984263377 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d13bccecc0 0x55d13bcd5070 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f80300116f0 tx=0x7f8030004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc7add965aebe61f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:13.709 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.711+0000 7f802cc00640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:39:13.712 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.715+0000 7f802cc00640 1 -- 172.21.15.43:0/2984263377 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8030005220 con 0x55d13bccecc0 2024-03-28T22:39:13.713 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.715+0000 7f802cc00640 1 -- 172.21.15.43:0/2984263377 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f80300053c0 con 0x55d13bccecc0 2024-03-28T22:39:13.713 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.715+0000 7f804c5ee740 1 -- 172.21.15.43:0/2984263377 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80340a34c0 con 0x55d13bccecc0 2024-03-28T22:39:13.714 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.717+0000 7f802cc00640 1 -- 172.21.15.43:0/2984263377 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f80300055c0 con 0x55d13bccecc0 2024-03-28T22:39:13.714 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.717+0000 7f804c5ee740 10 client.16197 fetch_fsmap learned FSMap version 29 2024-03-28T22:39:13.715 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.717+0000 7f804c5ee740 1 -- 172.21.15.43:0/2984263377 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d13bcf7640 con 0x55d13bccecc0 2024-03-28T22:39:13.715 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.718+0000 7f802cc00640 1 -- 172.21.15.43:0/2984263377 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80340a34c0 con 0x55d13bccecc0 2024-03-28T22:39:13.715 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.718+0000 7f802cc00640 1 -- 172.21.15.43:0/2984263377 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f803001eb10 con 0x55d13bccecc0 2024-03-28T22:39:13.715 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.718+0000 7f804c5ee740 10 client.16197 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:39:13.716 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.718+0000 7f804c5ee740 10 client.16197 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:39:13.716 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.718+0000 7f804c5ee740 20 client.16197 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:13.716 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.718+0000 7f804c5ee740 1 --2- 172.21.15.43:0/2984263377 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d13bcf8b00 0x55d13bcfaf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.716 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.718+0000 7f804c5ee740 4 client.16197 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:39:13.716 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.718+0000 7f804c5ee740 1 -- 172.21.15.43:0/2984263377 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d13bc45030 con 0x55d13bcf8b00 2024-03-28T22:39:13.716 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.719+0000 7f8042a00640 1 --2- 172.21.15.43:0/2984263377 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d13bcf8b00 0x55d13bcfaf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.717 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.720+0000 7f8042a00640 1 --2- 172.21.15.43:0/2984263377 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d13bcf8b00 0x55d13bcfaf60 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2bc5b70dd3d3c43f server_cookie=ff77a1008ffbe579 in_seq=0 out_seq=0 2024-03-28T22:39:13.717 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.720+0000 7f802cc00640 10 client.16197 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:13.720 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.722+0000 7f802cc00640 1 -- 172.21.15.43:0/2984263377 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d13bc45030 con 0x55d13bcf8b00 2024-03-28T22:39:13.720 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.723+0000 7f802cc00640 10 client.16197 handle_command_reply: tid=0 2024-03-28T22:39:13.720 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.723+0000 7f804c5ee740 1 client.16197 shutdown 2024-03-28T22:39:13.721 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.723+0000 7f804c5ee740 1 -- 172.21.15.43:0/2984263377 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d13bccecc0 msgr2=0x55d13bcd5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:13.721 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.723+0000 7f804c5ee740 1 --2- 172.21.15.43:0/2984263377 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d13bccecc0 0x55d13bcd5070 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f80300116f0 tx=0x7f8030004730 comp rx=0 tx=0).stop 2024-03-28T22:39:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.724+0000 7f804c5ee740 1 -- 172.21.15.43:0/2984263377 shutdown_connections 2024-03-28T22:39:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.724+0000 7f804c5ee740 1 --2- 172.21.15.43:0/2984263377 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d13bcf8b00 0x55d13bcfaf60 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.724+0000 7f802cc00640 0 client.16197 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.724+0000 7f804c5ee740 1 --2- 172.21.15.43:0/2984263377 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d13bcd7a90 0x55d13bcd55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.724+0000 7f804c5ee740 1 --2- 172.21.15.43:0/2984263377 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d13bccecc0 0x55d13bcd5070 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.724+0000 7f804c5ee740 1 --2- 172.21.15.43:0/2984263377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d13bcce3a0 0x55d13bcd4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.724+0000 7f804c5ee740 1 -- 172.21.15.43:0/2984263377 >> 172.21.15.43:0/2984263377 conn(0x55d13bcc8cc0 msgr2=0x55d13bccaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.724+0000 7f804c5ee740 1 -- 172.21.15.43:0/2984263377 shutdown_connections 2024-03-28T22:39:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.725+0000 7f804c5ee740 1 -- 172.21.15.43:0/2984263377 wait complete. 2024-03-28T22:39:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.725+0000 7f804c5ee740 20 client.16197 trim_cache size 0 max 16384 2024-03-28T22:39:13.764 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.766+0000 7f804c5ee740 1 Processor -- start 2024-03-28T22:39:13.765 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.767+0000 7f804c5ee740 1 -- start start 2024-03-28T22:39:13.765 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.768+0000 7f804c5ee740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d13bcce3a0 0x55d13bcf4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.765 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.768+0000 7f804c5ee740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d13bcd7a90 0x55d13bd00870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.766 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.768+0000 7f804c5ee740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d13bcf8b00 0x55d13bcc73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.766 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.768+0000 7f804c5ee740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f802400ff90 con 0x55d13bcd7a90 2024-03-28T22:39:13.766 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.768+0000 7f804c5ee740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8024011ad0 con 0x55d13bcce3a0 2024-03-28T22:39:13.766 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.768+0000 7f804c5ee740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f8024010660 con 0x55d13bcf8b00 2024-03-28T22:39:13.766 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.769+0000 7f8042000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d13bcd7a90 0x55d13bd00870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.766 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.769+0000 7f8042000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d13bcd7a90 0x55d13bd00870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53348/0 (socket says 172.21.15.43:53348) 2024-03-28T22:39:13.766 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.769+0000 7f8042000640 1 -- 172.21.15.43:0/1022844638 learned_addr learned my addr 172.21.15.43:0/1022844638 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:13.767 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.769+0000 7f8043400640 1 --2- 172.21.15.43:0/1022844638 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d13bcf8b00 0x55d13bcc73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.767 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.770+0000 7f8042000640 1 -- 172.21.15.43:0/1022844638 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d13bcf8b00 msgr2=0x55d13bcc73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:13.767 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.770+0000 7f8042000640 1 --2- 172.21.15.43:0/1022844638 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d13bcf8b00 0x55d13bcc73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.767 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.770+0000 7f8042000640 1 -- 172.21.15.43:0/1022844638 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d13bcce3a0 msgr2=0x55d13bcf4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:13.767 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.770+0000 7f8042a00640 1 --2- 172.21.15.43:0/1022844638 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d13bcce3a0 0x55d13bcf4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.768 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.770+0000 7f8042000640 1 --2- 172.21.15.43:0/1022844638 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d13bcce3a0 0x55d13bcf4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.768 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.770+0000 7f8042000640 1 -- 172.21.15.43:0/1022844638 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d13bcf5750 con 0x55d13bcd7a90 2024-03-28T22:39:13.768 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.770+0000 7f8042a00640 1 --2- 172.21.15.43:0/1022844638 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d13bcce3a0 0x55d13bcf4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:39:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.772+0000 7f8042000640 1 --2- 172.21.15.43:0/1022844638 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d13bcd7a90 0x55d13bd00870 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f803001ef60 tx=0x7f803000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9a45fa20676fa90 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:13.770 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.772+0000 7f802f400640 1 -- 172.21.15.43:0/1022844638 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8030041070 con 0x55d13bcd7a90 2024-03-28T22:39:13.770 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.772+0000 7f802f400640 1 -- 172.21.15.43:0/1022844638 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8030038040 con 0x55d13bcd7a90 2024-03-28T22:39:13.770 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.772+0000 7f802f400640 1 -- 172.21.15.43:0/1022844638 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f803001e8a0 con 0x55d13bcd7a90 2024-03-28T22:39:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.774+0000 7f804c5ee740 1 -- 172.21.15.43:0/1022844638 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d13bcd7a90 msgr2=0x55d13bd00870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.774+0000 7f804c5ee740 1 --2- 172.21.15.43:0/1022844638 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d13bcd7a90 0x55d13bd00870 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f803001ef60 tx=0x7f803000b480 comp rx=0 tx=0).stop 2024-03-28T22:39:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.775+0000 7f804c5ee740 1 -- 172.21.15.43:0/1022844638 shutdown_connections 2024-03-28T22:39:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.775+0000 7f804c5ee740 1 --2- 172.21.15.43:0/1022844638 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d13bcf8b00 0x55d13bcc73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.773 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.775+0000 7f804c5ee740 1 --2- 172.21.15.43:0/1022844638 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d13bcd7a90 0x55d13bd00870 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.773 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.775+0000 7f804c5ee740 1 --2- 172.21.15.43:0/1022844638 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d13bcce3a0 0x55d13bcf4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.773 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.775+0000 7f804c5ee740 1 -- 172.21.15.43:0/1022844638 >> 172.21.15.43:0/1022844638 conn(0x55d13bcc8cc0 msgr2=0x55d13bccfe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:13.773 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.775+0000 7f804c5ee740 1 -- 172.21.15.43:0/1022844638 shutdown_connections 2024-03-28T22:39:13.773 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.775+0000 7f804c5ee740 1 -- 172.21.15.43:0/1022844638 wait complete. 2024-03-28T22:39:13.773 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.775+0000 7f804c5ee740 1 -- start start 2024-03-28T22:39:13.774 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.776+0000 7f804c5ee740 1 Processor -- start 2024-03-28T22:39:13.774 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.777+0000 7f804c5ee740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d13bcce3a0 0x55d13bceca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.775 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.777+0000 7f804c5ee740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d13bcd7a90 0x55d13bcecfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.778+0000 7f8042a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d13bcce3a0 0x55d13bceca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.778+0000 7f804c5ee740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d13bcf8b00 0x55d13bd5dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.778+0000 7f804c5ee740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8024011400 con 0x55d13bcd7a90 2024-03-28T22:39:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.778+0000 7f804c5ee740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8024010d30 con 0x55d13bcf8b00 2024-03-28T22:39:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.778+0000 7f804c5ee740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f802400df70 con 0x55d13bcce3a0 2024-03-28T22:39:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.778+0000 7f8042000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d13bcd7a90 0x55d13bcecfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.778+0000 7f804c5ee740 10 client.0 _ll_register_callbacks cb 0x7f8030010b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.778+0000 7f8042000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d13bcd7a90 0x55d13bcecfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53362/0 (socket says 172.21.15.43:53362) 2024-03-28T22:39:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.778+0000 7f8042000640 1 -- 172.21.15.43:0/3116007852 learned_addr learned my addr 172.21.15.43:0/3116007852 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.779+0000 7f8042000640 1 -- 172.21.15.43:0/3116007852 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d13bcce3a0 msgr2=0x55d13bceca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.779+0000 7f8042000640 1 --2- 172.21.15.43:0/3116007852 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d13bcce3a0 0x55d13bceca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.779+0000 7f8042000640 1 -- 172.21.15.43:0/3116007852 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d13bcf8b00 msgr2=0x55d13bd5dea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:39:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.779+0000 7f8042000640 1 --2- 172.21.15.43:0/3116007852 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d13bcf8b00 0x55d13bd5dea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.779+0000 7f8042000640 1 -- 172.21.15.43:0/3116007852 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d13bd5e3e0 con 0x55d13bcd7a90 2024-03-28T22:39:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.779+0000 7f8042a00640 1 --2- 172.21.15.43:0/3116007852 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d13bcce3a0 0x55d13bceca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.780+0000 7f8042000640 1 --2- 172.21.15.43:0/3116007852 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d13bcd7a90 0x55d13bcecfc0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8030000c00 tx=0x7f803000b580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3708d13f6bcc908 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.780+0000 7f802cc00640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:39:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.780+0000 7f802cc00640 1 -- 172.21.15.43:0/3116007852 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8030041040 con 0x55d13bcd7a90 2024-03-28T22:39:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.781+0000 7f804c5ee740 1 -- 172.21.15.43:0/3116007852 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8030038040 con 0x55d13bcd7a90 2024-03-28T22:39:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.781+0000 7f802cc00640 1 -- 172.21.15.43:0/3116007852 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f803001fb20 con 0x55d13bcd7a90 2024-03-28T22:39:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.781+0000 7f802cc00640 1 -- 172.21.15.43:0/3116007852 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8030010e90 con 0x55d13bcd7a90 2024-03-28T22:39:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.781+0000 7f804c5ee740 10 client.16203 fetch_fsmap learned FSMap version 29 2024-03-28T22:39:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.781+0000 7f804c5ee740 1 -- 172.21.15.43:0/3116007852 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d13bd5f610 con 0x55d13bcd7a90 2024-03-28T22:39:13.781 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.782+0000 7f802cc00640 1 -- 172.21.15.43:0/3116007852 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8030038040 con 0x55d13bcd7a90 2024-03-28T22:39:13.781 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.783+0000 7f802cc00640 1 -- 172.21.15.43:0/3116007852 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f8030028680 con 0x55d13bcd7a90 2024-03-28T22:39:13.781 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.783+0000 7f804c5ee740 10 client.16203 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:39:13.781 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.783+0000 7f804c5ee740 10 client.16203 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:39:13.781 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.783+0000 7f804c5ee740 20 client.16203 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:13.781 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.783+0000 7f804c5ee740 1 --2- 172.21.15.43:0/3116007852 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d13bd5fc10 0x55d13bd62000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.781 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.783+0000 7f804c5ee740 4 client.16203 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T22:39:13.782 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.783+0000 7f804c5ee740 1 -- 172.21.15.43:0/3116007852 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d13bc3eb80 con 0x55d13bd5fc10 2024-03-28T22:39:13.782 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.784+0000 7f8042a00640 1 --2- 172.21.15.43:0/3116007852 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d13bd5fc10 0x55d13bd62000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.782 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.784+0000 7f8042a00640 1 --2- 172.21.15.43:0/3116007852 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d13bd5fc10 0x55d13bd62000 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bd520431b20ccc server_cookie=5334b26304859590 in_seq=0 out_seq=0 2024-03-28T22:39:13.782 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.784+0000 7f802cc00640 10 client.16203 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:13.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:13 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/4191626269' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:39:13.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:13 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/4191626269' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:39:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.979+0000 7f802cc00640 1 -- 172.21.15.43:0/3116007852 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d13bc3eb80 con 0x55d13bd5fc10 2024-03-28T22:39:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.979+0000 7f802cc00640 10 client.16203 handle_command_reply: tid=0 2024-03-28T22:39:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.979+0000 7f804c5ee740 1 client.16203 shutdown 2024-03-28T22:39:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.979+0000 7f804c5ee740 1 -- 172.21.15.43:0/3116007852 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d13bcd7a90 msgr2=0x55d13bcecfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:13.977 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.979+0000 7f804c5ee740 1 --2- 172.21.15.43:0/3116007852 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d13bcd7a90 0x55d13bcecfc0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8030000c00 tx=0x7f803000b580 comp rx=0 tx=0).stop 2024-03-28T22:39:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.980+0000 7f804c5ee740 1 -- 172.21.15.43:0/3116007852 shutdown_connections 2024-03-28T22:39:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.980+0000 7f804c5ee740 1 --2- 172.21.15.43:0/3116007852 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d13bd5fc10 0x55d13bd62000 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.980+0000 7f804c5ee740 1 --2- 172.21.15.43:0/3116007852 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d13bcf8b00 0x55d13bd5dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.980+0000 7f804c5ee740 1 --2- 172.21.15.43:0/3116007852 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d13bcd7a90 0x55d13bcecfc0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.980+0000 7f804c5ee740 1 --2- 172.21.15.43:0/3116007852 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d13bcce3a0 0x55d13bceca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.980+0000 7f802cc00640 0 client.16203 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.980+0000 7f804c5ee740 1 -- 172.21.15.43:0/3116007852 >> 172.21.15.43:0/3116007852 conn(0x55d13bcc8cc0 msgr2=0x55d13bcca110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.980+0000 7f804c5ee740 1 -- 172.21.15.43:0/3116007852 shutdown_connections 2024-03-28T22:39:13.978 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.980+0000 7f804c5ee740 1 -- 172.21.15.43:0/3116007852 wait complete. 2024-03-28T22:39:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.980+0000 7f804c5ee740 20 client.16203 trim_cache size 0 max 16384 2024-03-28T22:39:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:{ 2024-03-28T22:39:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "message": "", 2024-03-28T22:39:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "return_code": 0 2024-03-28T22:39:13.979 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:} 2024-03-28T22:39:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.981+0000 7f8049800640 1 -- 172.21.15.43:0/3489823456 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f80280a20f0 msgr2=0x7f80280a45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.981+0000 7f8049800640 1 --2- 172.21.15.43:0/3489823456 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f80280a20f0 0x7f80280a45b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8044078bf0 tx=0x7f803000b390 comp rx=0 tx=0).stop 2024-03-28T22:39:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.981+0000 7f8049800640 1 -- 172.21.15.43:0/3489823456 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8044075c80 msgr2=0x7f804419e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.981+0000 7f8049800640 1 --2- 172.21.15.43:0/3489823456 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8044075c80 0x7f804419e560 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f803400ea60 tx=0x7f803400ef20 comp rx=0 tx=0).stop 2024-03-28T22:39:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.981+0000 7f8049800640 1 -- 172.21.15.43:0/3489823456 shutdown_connections 2024-03-28T22:39:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.981+0000 7f8049800640 1 --2- 172.21.15.43:0/3489823456 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f80280a20f0 0x7f80280a45b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.981+0000 7f8049800640 1 --2- 172.21.15.43:0/3489823456 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f80441125c0 0x7f80441a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.981+0000 7f8049800640 1 --2- 172.21.15.43:0/3489823456 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f80440765a0 0x7f804419eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.980 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.981+0000 7f8049800640 1 --2- 172.21.15.43:0/3489823456 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8044075c80 0x7f804419e560 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.981 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.981+0000 7f8049800640 1 -- 172.21.15.43:0/3489823456 >> 172.21.15.43:0/3489823456 conn(0x7f80440ff410 msgr2=0x7f8044101800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:13.981 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.982+0000 7f8049800640 1 -- 172.21.15.43:0/3489823456 shutdown_connections 2024-03-28T22:39:13.981 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:13.982+0000 7f8049800640 1 -- 172.21.15.43:0/3489823456 wait complete. 2024-03-28T22:39:13.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.997+0000 7f5569600640 1 -- 172.21.15.43:0/2033804497 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55641016f0 msgr2=0x7f5564101b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:13.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.997+0000 7f5569600640 1 --2- 172.21.15.43:0/2033804497 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55641016f0 0x7f5564101b70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f5550009900 tx=0x7f555002bc30 comp rx=0 tx=0).stop 2024-03-28T22:39:13.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.998+0000 7f5569600640 1 -- 172.21.15.43:0/2033804497 shutdown_connections 2024-03-28T22:39:13.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.998+0000 7f5569600640 1 --2- 172.21.15.43:0/2033804497 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55641016f0 0x7f5564101b70 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.998+0000 7f5569600640 1 --2- 172.21.15.43:0/2033804497 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f55641020b0 0x7f556410e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.998+0000 7f5569600640 1 --2- 172.21.15.43:0/2033804497 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5564107700 0x7f5564107ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.998+0000 7f5569600640 1 -- 172.21.15.43:0/2033804497 >> 172.21.15.43:0/2033804497 conn(0x7f55640fd180 msgr2=0x7f55640ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:13.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.998+0000 7f5569600640 1 -- 172.21.15.43:0/2033804497 shutdown_connections 2024-03-28T22:39:13.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.998+0000 7f5569600640 1 -- 172.21.15.43:0/2033804497 wait complete. 2024-03-28T22:39:13.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.999+0000 7f5569600640 1 Processor -- start 2024-03-28T22:39:13.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.999+0000 7f5569600640 1 -- start start 2024-03-28T22:39:13.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.999+0000 7f5569600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f55641016f0 0x7f5564198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.999+0000 7f5569600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55641020b0 0x7f5564198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.999+0000 7f5569600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5564107700 0x7f556419c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:13.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.999+0000 7f5569600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5564111b60 con 0x7f55641016f0 2024-03-28T22:39:13.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.999+0000 7f5569600640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f55641119e0 con 0x7f5564107700 2024-03-28T22:39:13.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:13.999+0000 7f5569600640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f5564111ce0 con 0x7f55641020b0 2024-03-28T22:39:13.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.000+0000 7f5562000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55641020b0 0x7f5564198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.000+0000 7f5562a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f55641016f0 0x7f5564198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.000+0000 7f5563400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5564107700 0x7f556419c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:13.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.000+0000 7f5562000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55641020b0 0x7f5564198580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43062/0 (socket says 172.21.15.43:43062) 2024-03-28T22:39:13.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.000+0000 7f5562000640 1 -- 172.21.15.43:0/3706947294 learned_addr learned my addr 172.21.15.43:0/3706947294 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:13.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.000+0000 7f5562000640 1 -- 172.21.15.43:0/3706947294 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5564107700 msgr2=0x7f556419c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:13.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.000+0000 7f5562000640 1 --2- 172.21.15.43:0/3706947294 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5564107700 0x7f556419c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.000+0000 7f5562000640 1 -- 172.21.15.43:0/3706947294 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f55641016f0 msgr2=0x7f5564198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:13.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.000+0000 7f5562000640 1 --2- 172.21.15.43:0/3706947294 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f55641016f0 0x7f5564198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:13.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.000+0000 7f5562000640 1 -- 172.21.15.43:0/3706947294 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f556419d030 con 0x7f55641020b0 2024-03-28T22:39:13.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.001+0000 7f5562000640 1 --2- 172.21.15.43:0/3706947294 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55641020b0 0x7f5564198580 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f555002efd0 tx=0x7f5550004430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:13.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.001+0000 7f555b400640 1 -- 172.21.15.43:0/3706947294 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5550041070 con 0x7f55641020b0 2024-03-28T22:39:14.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.001+0000 7f555b400640 1 -- 172.21.15.43:0/3706947294 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55500054e0 con 0x7f55641020b0 2024-03-28T22:39:14.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.001+0000 7f555b400640 1 -- 172.21.15.43:0/3706947294 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5550003d70 con 0x7f55641020b0 2024-03-28T22:39:14.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.001+0000 7f5569600640 1 -- 172.21.15.43:0/3706947294 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f556419d2c0 con 0x7f55641020b0 2024-03-28T22:39:14.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.002+0000 7f5569600640 1 -- 172.21.15.43:0/3706947294 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55641a4bb0 con 0x7f55641020b0 2024-03-28T22:39:14.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.006+0000 7f555b400640 1 -- 172.21.15.43:0/3706947294 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f5550044070 con 0x7f55641020b0 2024-03-28T22:39:14.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.006+0000 7f556c3a9740 1 Processor -- start 2024-03-28T22:39:14.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.007+0000 7f556c3a9740 1 -- start start 2024-03-28T22:39:14.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.007+0000 7f556c3a9740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55645c05ba20 0x55645c05be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.007+0000 7f556c3a9740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55645c05c3d0 0x55645c048560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.007+0000 7f556c3a9740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55645c048b80 0x55645c048fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.007+0000 7f556c3a9740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55645bfb2050 con 0x55645c048b80 2024-03-28T22:39:14.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.007+0000 7f556c3a9740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55645bf5ace0 con 0x55645c05ba20 2024-03-28T22:39:14.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.007+0000 7f556c3a9740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55645bfae0f0 con 0x55645c05c3d0 2024-03-28T22:39:14.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.007+0000 7f5563400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55645c05ba20 0x55645c05be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.007+0000 7f5563400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55645c05ba20 0x55645c05be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59480/0 (socket says 172.21.15.43:59480) 2024-03-28T22:39:14.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.007+0000 7f5563400640 1 -- 172.21.15.43:0/979503518 learned_addr learned my addr 172.21.15.43:0/979503518 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:14.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.007+0000 7f5562a00640 1 --2- 172.21.15.43:0/979503518 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55645c05c3d0 0x55645c048560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.008+0000 7f5562000640 1 --2- 172.21.15.43:0/979503518 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55645c048b80 0x55645c048fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.008+0000 7f5563400640 1 -- 172.21.15.43:0/979503518 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55645c05c3d0 msgr2=0x55645c048560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:14.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.008+0000 7f5563400640 1 --2- 172.21.15.43:0/979503518 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55645c05c3d0 0x55645c048560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.008+0000 7f5563400640 1 -- 172.21.15.43:0/979503518 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55645c048b80 msgr2=0x55645c048fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:14.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.008+0000 7f5563400640 1 --2- 172.21.15.43:0/979503518 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55645c048b80 0x55645c048fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.008+0000 7f5563400640 1 -- 172.21.15.43:0/979503518 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55645bcc63f0 con 0x55645c05ba20 2024-03-28T22:39:14.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.008+0000 7f5563400640 1 --2- 172.21.15.43:0/979503518 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55645c05ba20 0x55645c05be00 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f555c0039f0 tx=0x7f555c003eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=807bf32711438d9c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:14.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.008+0000 7f555b400640 1 --2- 172.21.15.43:0/3706947294 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f55480a1f70 0x7f55480a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.008+0000 7f554f400640 1 -- 172.21.15.43:0/979503518 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555c01a470 con 0x55645c05ba20 2024-03-28T22:39:14.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.009+0000 7f554f400640 1 -- 172.21.15.43:0/979503518 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f555c015070 con 0x55645c05ba20 2024-03-28T22:39:14.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.009+0000 7f554f400640 1 -- 172.21.15.43:0/979503518 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555c010900 con 0x55645c05ba20 2024-03-28T22:39:14.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.009+0000 7f555b400640 1 -- 172.21.15.43:0/3706947294 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f55500efed0 con 0x7f55641020b0 2024-03-28T22:39:14.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.009+0000 7f5562a00640 1 --2- 172.21.15.43:0/3706947294 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f55480a1f70 0x7f55480a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.009+0000 7f556c3a9740 1 -- 172.21.15.43:0/979503518 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55645c05ba20 msgr2=0x55645c05be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.009+0000 7f556c3a9740 1 --2- 172.21.15.43:0/979503518 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55645c05ba20 0x55645c05be00 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f555c0039f0 tx=0x7f555c003eb0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.010+0000 7f5562a00640 1 --2- 172.21.15.43:0/3706947294 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f55480a1f70 0x7f55480a4430 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x55645c0496a0 tx=0x7f5554005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:14.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.010+0000 7f556c3a9740 1 -- 172.21.15.43:0/979503518 shutdown_connections 2024-03-28T22:39:14.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.010+0000 7f556c3a9740 1 --2- 172.21.15.43:0/979503518 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55645c048b80 0x55645c048fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.010+0000 7f556c3a9740 1 --2- 172.21.15.43:0/979503518 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55645c05c3d0 0x55645c048560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.010+0000 7f556c3a9740 1 --2- 172.21.15.43:0/979503518 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55645c05ba20 0x55645c05be00 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.010+0000 7f556c3a9740 1 -- 172.21.15.43:0/979503518 >> 172.21.15.43:0/979503518 conn(0x55645c04b020 msgr2=0x55645c04b400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:14.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.010+0000 7f556c3a9740 1 -- 172.21.15.43:0/979503518 shutdown_connections 2024-03-28T22:39:14.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.010+0000 7f556c3a9740 1 -- 172.21.15.43:0/979503518 wait complete. 2024-03-28T22:39:14.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.010+0000 7f556c3a9740 1 -- start start 2024-03-28T22:39:14.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.011+0000 7f556c3a9740 1 Processor -- start 2024-03-28T22:39:14.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.012+0000 7f556c3a9740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55645c048b80 0x55645c05fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.012+0000 7f556c3a9740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55645c05ba20 0x55645c05ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.012+0000 7f556c3a9740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55645c05c3d0 0x55645c060500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.012+0000 7f556c3a9740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55645c062e60 con 0x55645c048b80 2024-03-28T22:39:14.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.012+0000 7f556c3a9740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55645bf502f0 con 0x55645c05c3d0 2024-03-28T22:39:14.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.012+0000 7f556c3a9740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55645c062fe0 con 0x55645c05ba20 2024-03-28T22:39:14.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.012+0000 7f5562a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55645c048b80 0x55645c05fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.012+0000 7f5562a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55645c048b80 0x55645c05fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53404/0 (socket says 172.21.15.43:53404) 2024-03-28T22:39:14.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.012+0000 7f5562a00640 1 -- 172.21.15.43:0/1618140337 learned_addr learned my addr 172.21.15.43:0/1618140337 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:14.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.012+0000 7f5563400640 1 --2- 172.21.15.43:0/1618140337 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55645c05c3d0 0x55645c060500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.012+0000 7f556c3a9740 10 client.0 _ll_register_callbacks cb 0x55645bcc5be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:14.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.012+0000 7f5562000640 1 --2- 172.21.15.43:0/1618140337 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55645c05ba20 0x55645c05ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.012+0000 7f5562a00640 1 -- 172.21.15.43:0/1618140337 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55645c05ba20 msgr2=0x55645c05ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:14.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.012+0000 7f5562a00640 1 --2- 172.21.15.43:0/1618140337 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55645c05ba20 0x55645c05ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.012+0000 7f5562a00640 1 -- 172.21.15.43:0/1618140337 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55645c05c3d0 msgr2=0x55645c060500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:14.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.012+0000 7f5562a00640 1 --2- 172.21.15.43:0/1618140337 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55645c05c3d0 0x55645c060500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.012+0000 7f5562a00640 1 -- 172.21.15.43:0/1618140337 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55645bd907d0 con 0x55645c048b80 2024-03-28T22:39:14.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.013+0000 7f5562a00640 1 --2- 172.21.15.43:0/1618140337 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55645c048b80 0x55645c05fa80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5554013010 tx=0x7f5554004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=105ff5e2d6908f11 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:14.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.013+0000 7f5543e00640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:39:14.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.013+0000 7f5543e00640 1 -- 172.21.15.43:0/1618140337 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555401e680 con 0x55645c048b80 2024-03-28T22:39:14.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.013+0000 7f5543e00640 1 -- 172.21.15.43:0/1618140337 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f555401e820 con 0x55645c048b80 2024-03-28T22:39:14.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.013+0000 7f556c3a9740 1 -- 172.21.15.43:0/1618140337 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f555c015070 con 0x55645c048b80 2024-03-28T22:39:14.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.014+0000 7f5543e00640 1 -- 172.21.15.43:0/1618140337 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f555401ea20 con 0x55645c048b80 2024-03-28T22:39:14.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.014+0000 7f556c3a9740 10 client.16215 fetch_fsmap learned FSMap version 29 2024-03-28T22:39:14.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.014+0000 7f556c3a9740 1 -- 172.21.15.43:0/1618140337 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55645c06f260 con 0x55645c048b80 2024-03-28T22:39:14.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.014+0000 7f5543e00640 1 -- 172.21.15.43:0/1618140337 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f555c015070 con 0x55645c048b80 2024-03-28T22:39:14.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.015+0000 7f5543e00640 1 -- 172.21.15.43:0/1618140337 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f5554027070 con 0x55645c048b80 2024-03-28T22:39:14.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.015+0000 7f556c3a9740 10 client.16215 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:39:14.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.015+0000 7f556c3a9740 10 client.16215 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:39:14.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.015+0000 7f556c3a9740 20 client.16215 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:14.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.015+0000 7f556c3a9740 1 --2- 172.21.15.43:0/1618140337 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55645c0706c0 0x55645c072b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.015+0000 7f556c3a9740 4 client.16215 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:39:14.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.015+0000 7f556c3a9740 1 -- 172.21.15.43:0/1618140337 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55645bf5b680 con 0x55645c0706c0 2024-03-28T22:39:14.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.015+0000 7f5562000640 1 --2- 172.21.15.43:0/1618140337 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55645c0706c0 0x55645c072b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.016+0000 7f5562000640 1 --2- 172.21.15.43:0/1618140337 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55645c0706c0 0x55645c072b50 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ceb75cf415951f0f server_cookie=44e6dea70ead1e8c in_seq=0 out_seq=0 2024-03-28T22:39:14.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.016+0000 7f5543e00640 10 client.16215 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:39:14.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.016+0000 7f5543e00640 1 -- 172.21.15.43:0/1618140337 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55645bf5b680 con 0x55645c0706c0 2024-03-28T22:39:14.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.016+0000 7f5543e00640 10 client.16215 handle_command_reply: tid=0 2024-03-28T22:39:14.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.016+0000 7f556c3a9740 1 client.16215 shutdown 2024-03-28T22:39:14.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.017+0000 7f556c3a9740 1 -- 172.21.15.43:0/1618140337 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55645c048b80 msgr2=0x55645c05fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.017+0000 7f556c3a9740 1 --2- 172.21.15.43:0/1618140337 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55645c048b80 0x55645c05fa80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5554013010 tx=0x7f5554004730 comp rx=0 tx=0).stop 2024-03-28T22:39:14.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.017+0000 7f556c3a9740 1 -- 172.21.15.43:0/1618140337 shutdown_connections 2024-03-28T22:39:14.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.017+0000 7f556c3a9740 1 --2- 172.21.15.43:0/1618140337 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55645c0706c0 0x55645c072b50 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.017+0000 7f556c3a9740 1 --2- 172.21.15.43:0/1618140337 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55645c05c3d0 0x55645c060500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.017+0000 7f556c3a9740 1 --2- 172.21.15.43:0/1618140337 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55645c05ba20 0x55645c05ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.017+0000 7f556c3a9740 1 --2- 172.21.15.43:0/1618140337 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55645c048b80 0x55645c05fa80 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.017+0000 7f556c3a9740 1 -- 172.21.15.43:0/1618140337 >> 172.21.15.43:0/1618140337 conn(0x55645c04b020 msgr2=0x55645c04a780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:14.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.017+0000 7f5543e00640 0 client.16215 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:39:14.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.017+0000 7f556c3a9740 1 -- 172.21.15.43:0/1618140337 shutdown_connections 2024-03-28T22:39:14.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.017+0000 7f556c3a9740 1 -- 172.21.15.43:0/1618140337 wait complete. 2024-03-28T22:39:14.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.017+0000 7f556c3a9740 20 client.16215 trim_cache size 0 max 16384 2024-03-28T22:39:14.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.034+0000 7f556c3a9740 1 Processor -- start 2024-03-28T22:39:14.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.034+0000 7f556c3a9740 1 -- start start 2024-03-28T22:39:14.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.035+0000 7f556c3a9740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55645c04b020 0x55645c0621b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.035+0000 7f556c3a9740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55645c048b80 0x55645c06bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.035+0000 7f556c3a9740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55645c05ba20 0x55645c05fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.035+0000 7f556c3a9740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f55641aca00 con 0x55645c05ba20 2024-03-28T22:39:14.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.035+0000 7f556c3a9740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f55641ae540 con 0x55645c048b80 2024-03-28T22:39:14.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.035+0000 7f556c3a9740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f55641ad0d0 con 0x55645c04b020 2024-03-28T22:39:14.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.035+0000 7f5562a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55645c04b020 0x55645c0621b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.035+0000 7f5562000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55645c048b80 0x55645c06bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.035+0000 7f5562a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55645c04b020 0x55645c0621b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43090/0 (socket says 172.21.15.43:43090) 2024-03-28T22:39:14.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.035+0000 7f5562a00640 1 -- 172.21.15.43:0/251663908 learned_addr learned my addr 172.21.15.43:0/251663908 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:14.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.035+0000 7f5563400640 1 --2- 172.21.15.43:0/251663908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55645c05ba20 0x55645c05fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.035+0000 7f5562a00640 1 -- 172.21.15.43:0/251663908 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55645c048b80 msgr2=0x55645c06bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:14.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.035+0000 7f5562a00640 1 --2- 172.21.15.43:0/251663908 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55645c048b80 0x55645c06bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.035+0000 7f5562a00640 1 -- 172.21.15.43:0/251663908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55645c05ba20 msgr2=0x55645c05fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:14.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.035+0000 7f5562a00640 1 --2- 172.21.15.43:0/251663908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55645c05ba20 0x55645c05fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.035+0000 7f5562a00640 1 -- 172.21.15.43:0/251663908 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55645c048f60 con 0x55645c04b020 2024-03-28T22:39:14.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.036+0000 7f5563400640 1 --2- 172.21.15.43:0/251663908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55645c05ba20 0x55645c05fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:39:14.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.036+0000 7f5562a00640 1 --2- 172.21.15.43:0/251663908 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55645c04b020 0x55645c0621b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5554009da0 tx=0x7f5554009dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb46b2fb3009f106 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:14.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.036+0000 7f554ea00640 1 -- 172.21.15.43:0/251663908 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555401e700 con 0x55645c04b020 2024-03-28T22:39:14.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.036+0000 7f554ea00640 1 -- 172.21.15.43:0/251663908 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5554038020 con 0x55645c04b020 2024-03-28T22:39:14.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.036+0000 7f554ea00640 1 -- 172.21.15.43:0/251663908 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5554004910 con 0x55645c04b020 2024-03-28T22:39:14.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.037+0000 7f556c3a9740 1 -- 172.21.15.43:0/251663908 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55645c04b020 msgr2=0x55645c0621b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.037+0000 7f556c3a9740 1 --2- 172.21.15.43:0/251663908 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55645c04b020 0x55645c0621b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5554009da0 tx=0x7f5554009dd0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.037+0000 7f556c3a9740 1 -- 172.21.15.43:0/251663908 shutdown_connections 2024-03-28T22:39:14.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.037+0000 7f556c3a9740 1 --2- 172.21.15.43:0/251663908 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55645c048b80 0x55645c06bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.037+0000 7f556c3a9740 1 --2- 172.21.15.43:0/251663908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55645c05ba20 0x55645c05fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.037+0000 7f556c3a9740 1 --2- 172.21.15.43:0/251663908 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55645c04b020 0x55645c0621b0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.037+0000 7f556c3a9740 1 -- 172.21.15.43:0/251663908 >> 172.21.15.43:0/251663908 conn(0x55645c043f40 msgr2=0x55645c05ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:14.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.037+0000 7f556c3a9740 1 -- 172.21.15.43:0/251663908 shutdown_connections 2024-03-28T22:39:14.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.037+0000 7f556c3a9740 1 -- 172.21.15.43:0/251663908 wait complete. 2024-03-28T22:39:14.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.037+0000 7f556c3a9740 1 -- start start 2024-03-28T22:39:14.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.038+0000 7f556c3a9740 1 Processor -- start 2024-03-28T22:39:14.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.038+0000 7f556c3a9740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55645c048b80 0x55645c068270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.038+0000 7f556c3a9740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55645c04b020 0x55645c065240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.038+0000 7f556c3a9740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55645c05ba20 0x55645c065780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.038+0000 7f556c3a9740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f55641ade70 con 0x55645c048b80 2024-03-28T22:39:14.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.038+0000 7f556c3a9740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f55641ad7a0 con 0x55645c04b020 2024-03-28T22:39:14.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.038+0000 7f556c3a9740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f55641aa9e0 con 0x55645c05ba20 2024-03-28T22:39:14.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.038+0000 7f556c3a9740 10 client.0 _ll_register_callbacks cb 0x7f5554009a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:14.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.038+0000 7f5562000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55645c04b020 0x55645c065240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.038+0000 7f5562a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55645c048b80 0x55645c068270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.038+0000 7f5563400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55645c05ba20 0x55645c065780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.038+0000 7f5562a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55645c048b80 0x55645c068270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53420/0 (socket says 172.21.15.43:53420) 2024-03-28T22:39:14.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.038+0000 7f5562a00640 1 -- 172.21.15.43:0/1049589730 learned_addr learned my addr 172.21.15.43:0/1049589730 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:14.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.038+0000 7f5562000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55645c04b020 0x55645c065240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59502/0 (socket says 172.21.15.43:59502) 2024-03-28T22:39:14.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.039+0000 7f5562000640 1 -- 172.21.15.43:0/1049589730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55645c05ba20 msgr2=0x55645c065780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:14.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.039+0000 7f5562000640 1 --2- 172.21.15.43:0/1049589730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55645c05ba20 0x55645c065780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.039+0000 7f5562000640 1 -- 172.21.15.43:0/1049589730 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55645c048b80 msgr2=0x55645c068270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:14.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.039+0000 7f5562000640 1 --2- 172.21.15.43:0/1049589730 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55645c048b80 0x55645c068270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.039+0000 7f5562000640 1 -- 172.21.15.43:0/1049589730 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55645c065cc0 con 0x55645c04b020 2024-03-28T22:39:14.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.039+0000 7f5563400640 1 --2- 172.21.15.43:0/1049589730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55645c05ba20 0x55645c065780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:14.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.039+0000 7f5562a00640 1 --2- 172.21.15.43:0/1049589730 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55645c048b80 0x55645c068270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:14.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.039+0000 7f5562000640 1 --2- 172.21.15.43:0/1049589730 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55645c04b020 0x55645c065240 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f55500ed330 tx=0x7f55500dc040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6d5fb09d9547e38 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:14.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.039+0000 7f5543e00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:39:14.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.039+0000 7f5543e00640 1 -- 172.21.15.43:0/1049589730 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555004b040 con 0x55645c04b020 2024-03-28T22:39:14.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.040+0000 7f5543e00640 1 -- 172.21.15.43:0/1049589730 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f55500dc0b0 con 0x55645c04b020 2024-03-28T22:39:14.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.040+0000 7f556c3a9740 1 -- 172.21.15.43:0/1049589730 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5554038020 con 0x55645c04b020 2024-03-28T22:39:14.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.040+0000 7f5543e00640 1 -- 172.21.15.43:0/1049589730 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f55500bc1c0 con 0x55645c04b020 2024-03-28T22:39:14.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.040+0000 7f556c3a9740 10 client.26393 fetch_fsmap learned FSMap version 29 2024-03-28T22:39:14.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.040+0000 7f556c3a9740 1 -- 172.21.15.43:0/1049589730 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55645c0d7730 con 0x55645c04b020 2024-03-28T22:39:14.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.040+0000 7f5543e00640 1 -- 172.21.15.43:0/1049589730 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5554038020 con 0x55645c04b020 2024-03-28T22:39:14.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.040+0000 7f5543e00640 1 -- 172.21.15.43:0/1049589730 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f55500b8cf0 con 0x55645c04b020 2024-03-28T22:39:14.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.040+0000 7f556c3a9740 10 client.26393 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:39:14.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.040+0000 7f556c3a9740 10 client.26393 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:39:14.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.040+0000 7f556c3a9740 20 client.26393 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:14.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.040+0000 7f556c3a9740 1 --2- 172.21.15.43:0/1049589730 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55645c05c3d0 0x55645c0d9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.040+0000 7f556c3a9740 4 client.26393 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:39:14.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.040+0000 7f556c3a9740 1 -- 172.21.15.43:0/1049589730 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55645bfbf590 con 0x55645c05c3d0 2024-03-28T22:39:14.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.041+0000 7f5562a00640 1 --2- 172.21.15.43:0/1049589730 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55645c05c3d0 0x55645c0d9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.041+0000 7f5562a00640 1 --2- 172.21.15.43:0/1049589730 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55645c05c3d0 0x55645c0d9ef0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b73155fb3345564d server_cookie=842b4f595e21cdd7 in_seq=0 out_seq=0 2024-03-28T22:39:14.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.041+0000 7f5543e00640 10 client.26393 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:39:14.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.041+0000 7f5543e00640 1 -- 172.21.15.43:0/1049589730 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55645bfbf590 con 0x55645c05c3d0 2024-03-28T22:39:14.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.041+0000 7f5543e00640 10 client.26393 handle_command_reply: tid=0 2024-03-28T22:39:14.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.041+0000 7f556c3a9740 1 client.26393 shutdown 2024-03-28T22:39:14.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.042+0000 7f556c3a9740 1 -- 172.21.15.43:0/1049589730 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55645c04b020 msgr2=0x55645c065240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.042+0000 7f556c3a9740 1 --2- 172.21.15.43:0/1049589730 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55645c04b020 0x55645c065240 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f55500ed330 tx=0x7f55500dc040 comp rx=0 tx=0).stop 2024-03-28T22:39:14.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.042+0000 7f556c3a9740 1 -- 172.21.15.43:0/1049589730 shutdown_connections 2024-03-28T22:39:14.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.042+0000 7f556c3a9740 1 --2- 172.21.15.43:0/1049589730 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55645c05c3d0 0x55645c0d9ef0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.042+0000 7f556c3a9740 1 --2- 172.21.15.43:0/1049589730 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55645c04b020 0x55645c065240 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.042+0000 7f556c3a9740 1 --2- 172.21.15.43:0/1049589730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55645c05ba20 0x55645c065780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.042+0000 7f556c3a9740 1 --2- 172.21.15.43:0/1049589730 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55645c048b80 0x55645c068270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.042+0000 7f556c3a9740 1 -- 172.21.15.43:0/1049589730 >> 172.21.15.43:0/1049589730 conn(0x55645c043f40 msgr2=0x55645c05f9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:14.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.042+0000 7f5543e00640 0 client.26393 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:39:14.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.042+0000 7f556c3a9740 1 -- 172.21.15.43:0/1049589730 shutdown_connections 2024-03-28T22:39:14.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.042+0000 7f556c3a9740 1 -- 172.21.15.43:0/1049589730 wait complete. 2024-03-28T22:39:14.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.042+0000 7f556c3a9740 20 client.26393 trim_cache size 0 max 16384 2024-03-28T22:39:14.044 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:39:14.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.043+0000 7f5569600640 1 -- 172.21.15.43:0/3706947294 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f55480a1f70 msgr2=0x7f55480a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.043+0000 7f5569600640 1 --2- 172.21.15.43:0/3706947294 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f55480a1f70 0x7f55480a4430 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x55645c0496a0 tx=0x7f5554005f90 comp rx=0 tx=0).stop 2024-03-28T22:39:14.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.043+0000 7f5569600640 1 -- 172.21.15.43:0/3706947294 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55641020b0 msgr2=0x7f5564198580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.043+0000 7f5569600640 1 --2- 172.21.15.43:0/3706947294 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55641020b0 0x7f5564198580 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f555002efd0 tx=0x7f5550004430 comp rx=0 tx=0).stop 2024-03-28T22:39:14.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.043+0000 7f5569600640 1 -- 172.21.15.43:0/3706947294 shutdown_connections 2024-03-28T22:39:14.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.043+0000 7f5569600640 1 --2- 172.21.15.43:0/3706947294 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f55480a1f70 0x7f55480a4430 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.043+0000 7f5569600640 1 --2- 172.21.15.43:0/3706947294 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5564107700 0x7f556419c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.043+0000 7f5569600640 1 --2- 172.21.15.43:0/3706947294 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f55641020b0 0x7f5564198580 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.043+0000 7f5569600640 1 --2- 172.21.15.43:0/3706947294 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f55641016f0 0x7f5564198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.043+0000 7f5569600640 1 -- 172.21.15.43:0/3706947294 >> 172.21.15.43:0/3706947294 conn(0x7f55640fd180 msgr2=0x7f55640fd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:14.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.043+0000 7f5569600640 1 -- 172.21.15.43:0/3706947294 shutdown_connections 2024-03-28T22:39:14.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.043+0000 7f5569600640 1 -- 172.21.15.43:0/3706947294 wait complete. 2024-03-28T22:39:14.058 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:39:14.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.315+0000 7f7d83e00640 1 -- 172.21.15.43:0/2213829152 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d841097d0 msgr2=0x7f7d84109bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.315+0000 7f7d83e00640 1 --2- 172.21.15.43:0/2213829152 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d841097d0 0x7f7d84109bb0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7d74009930 tx=0x7f7d7402bc20 comp rx=0 tx=0).stop 2024-03-28T22:39:14.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.315+0000 7f7d83e00640 1 -- 172.21.15.43:0/2213829152 shutdown_connections 2024-03-28T22:39:14.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.315+0000 7f7d83e00640 1 --2- 172.21.15.43:0/2213829152 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7d841037c0 0x7f7d84103c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.315+0000 7f7d83e00640 1 --2- 172.21.15.43:0/2213829152 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d84104180 0x7f7d84110690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.315+0000 7f7d83e00640 1 --2- 172.21.15.43:0/2213829152 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d841097d0 0x7f7d84109bb0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.315+0000 7f7d83e00640 1 -- 172.21.15.43:0/2213829152 >> 172.21.15.43:0/2213829152 conn(0x7f7d840ff230 msgr2=0x7f7d84101650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:14.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.315+0000 7f7d83e00640 1 -- 172.21.15.43:0/2213829152 shutdown_connections 2024-03-28T22:39:14.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.315+0000 7f7d83e00640 1 -- 172.21.15.43:0/2213829152 wait complete. 2024-03-28T22:39:14.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.316+0000 7f7d83e00640 1 Processor -- start 2024-03-28T22:39:14.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.317+0000 7f7d83e00640 1 -- start start 2024-03-28T22:39:14.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.317+0000 7f7d83e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d841037c0 0x7f7d8419a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.317+0000 7f7d83e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7d84104180 0x7f7d8419a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.317+0000 7f7d83e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d841097d0 0x7f7d8419e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.317+0000 7f7d83e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7d84113bf0 con 0x7f7d841097d0 2024-03-28T22:39:14.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.317+0000 7f7d83e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7d84113a70 con 0x7f7d84104180 2024-03-28T22:39:14.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.317+0000 7f7d83e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f7d84113d70 con 0x7f7d841037c0 2024-03-28T22:39:14.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.317+0000 7f7d82a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d841037c0 0x7f7d8419a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.317+0000 7f7d82a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d841037c0 0x7f7d8419a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43116/0 (socket says 172.21.15.43:43116) 2024-03-28T22:39:14.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.317+0000 7f7d82a00640 1 -- 172.21.15.43:0/465922004 learned_addr learned my addr 172.21.15.43:0/465922004 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:14.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.317+0000 7f7d82000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7d84104180 0x7f7d8419a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.317+0000 7f7d83400640 1 --2- 172.21.15.43:0/465922004 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d841097d0 0x7f7d8419e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.318+0000 7f7d82a00640 1 -- 172.21.15.43:0/465922004 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7d84104180 msgr2=0x7f7d8419a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.318+0000 7f7d82a00640 1 --2- 172.21.15.43:0/465922004 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7d84104180 0x7f7d8419a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.318+0000 7f7d82a00640 1 -- 172.21.15.43:0/465922004 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d841097d0 msgr2=0x7f7d8419e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.318+0000 7f7d82a00640 1 --2- 172.21.15.43:0/465922004 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d841097d0 0x7f7d8419e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.318+0000 7f7d82a00640 1 -- 172.21.15.43:0/465922004 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d8419f0e0 con 0x7f7d841037c0 2024-03-28T22:39:14.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.318+0000 7f7d82000640 1 --2- 172.21.15.43:0/465922004 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7d84104180 0x7f7d8419a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:14.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.318+0000 7f7d82a00640 1 --2- 172.21.15.43:0/465922004 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d841037c0 0x7f7d8419a0f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7d74005ab0 tx=0x7f7d74005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:14.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.318+0000 7f7d7b400640 1 -- 172.21.15.43:0/465922004 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d74043020 con 0x7f7d841037c0 2024-03-28T22:39:14.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.319+0000 7f7d83e00640 1 -- 172.21.15.43:0/465922004 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d8419f370 con 0x7f7d841037c0 2024-03-28T22:39:14.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.319+0000 7f7d7b400640 1 -- 172.21.15.43:0/465922004 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d74005000 con 0x7f7d841037c0 2024-03-28T22:39:14.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.319+0000 7f7d7b400640 1 -- 172.21.15.43:0/465922004 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d74003a80 con 0x7f7d841037c0 2024-03-28T22:39:14.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.319+0000 7f7d83e00640 1 -- 172.21.15.43:0/465922004 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d841a6c60 con 0x7f7d841037c0 2024-03-28T22:39:14.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.323+0000 7f7d8c62c740 1 Processor -- start 2024-03-28T22:39:14.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.323+0000 7f7d7b400640 1 -- 172.21.15.43:0/465922004 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f7d74046020 con 0x7f7d841037c0 2024-03-28T22:39:14.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.324+0000 7f7d8c62c740 1 -- start start 2024-03-28T22:39:14.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.324+0000 7f7d8c62c740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559c5867b550 0x559c5867b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.324+0000 7f7d8c62c740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559c5867be70 0x559c58684510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.324+0000 7f7d8c62c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559c58684a50 0x559c58686e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.324+0000 7f7d8c62c740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x559c585df050 con 0x559c5867be70 2024-03-28T22:39:14.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.324+0000 7f7d8c62c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x559c58587ce0 con 0x559c58684a50 2024-03-28T22:39:14.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.324+0000 7f7d8c62c740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x559c585db0f0 con 0x559c5867b550 2024-03-28T22:39:14.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.324+0000 7f7d83400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559c5867b550 0x559c5867b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.324+0000 7f7d82a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559c5867be70 0x559c58684510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.324+0000 7f7d82000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559c58684a50 0x559c58686e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.324+0000 7f7d83400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559c5867b550 0x559c5867b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43122/0 (socket says 172.21.15.43:43122) 2024-03-28T22:39:14.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.324+0000 7f7d83400640 1 -- 172.21.15.43:0/963555297 learned_addr learned my addr 172.21.15.43:0/963555297 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:14.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.325+0000 7f7d82000640 1 -- 172.21.15.43:0/963555297 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559c5867b550 msgr2=0x559c5867b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:14.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.325+0000 7f7d82000640 1 --2- 172.21.15.43:0/963555297 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559c5867b550 0x559c5867b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.325+0000 7f7d82000640 1 -- 172.21.15.43:0/963555297 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559c5867be70 msgr2=0x559c58684510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:14.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.325+0000 7f7d82000640 1 --2- 172.21.15.43:0/963555297 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559c5867be70 0x559c58684510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.325+0000 7f7d82000640 1 -- 172.21.15.43:0/963555297 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c582f33f0 con 0x559c58684a50 2024-03-28T22:39:14.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.325+0000 7f7d83400640 1 --2- 172.21.15.43:0/963555297 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559c5867b550 0x559c5867b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:14.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.325+0000 7f7d82000640 1 --2- 172.21.15.43:0/963555297 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559c58684a50 0x559c58686e40 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f7d70009fa0 tx=0x7f7d700118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=399a9a4243a15d1c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:14.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.325+0000 7f7d7b400640 1 --2- 172.21.15.43:0/465922004 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7d680a1f70 0x7f7d680a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.325+0000 7f7d6fe00640 1 -- 172.21.15.43:0/963555297 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d70004020 con 0x559c58684a50 2024-03-28T22:39:14.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.326+0000 7f7d7b400640 1 -- 172.21.15.43:0/465922004 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f7d740f0390 con 0x7f7d841037c0 2024-03-28T22:39:14.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.326+0000 7f7d6fe00640 1 -- 172.21.15.43:0/963555297 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d700041c0 con 0x559c58684a50 2024-03-28T22:39:14.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.326+0000 7f7d6fe00640 1 -- 172.21.15.43:0/963555297 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d7000f960 con 0x559c58684a50 2024-03-28T22:39:14.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.326+0000 7f7d83400640 1 --2- 172.21.15.43:0/465922004 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7d680a1f70 0x7f7d680a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.326+0000 7f7d8c62c740 1 -- 172.21.15.43:0/963555297 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559c58684a50 msgr2=0x559c58686e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.326+0000 7f7d83400640 1 --2- 172.21.15.43:0/465922004 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7d680a1f70 0x7f7d680a4430 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c009fd0 tx=0x7f7d7c009500 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:14.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.326+0000 7f7d8c62c740 1 --2- 172.21.15.43:0/963555297 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559c58684a50 0x559c58686e40 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f7d70009fa0 tx=0x7f7d700118d0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.327+0000 7f7d8c62c740 1 -- 172.21.15.43:0/963555297 shutdown_connections 2024-03-28T22:39:14.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.327+0000 7f7d8c62c740 1 --2- 172.21.15.43:0/963555297 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559c58684a50 0x559c58686e40 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.327+0000 7f7d8c62c740 1 --2- 172.21.15.43:0/963555297 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559c5867be70 0x559c58684510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.327+0000 7f7d8c62c740 1 --2- 172.21.15.43:0/963555297 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559c5867b550 0x559c5867b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.327+0000 7f7d8c62c740 1 -- 172.21.15.43:0/963555297 >> 172.21.15.43:0/963555297 conn(0x559c58675c80 msgr2=0x559c58676060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:14.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.327+0000 7f7d8c62c740 1 -- 172.21.15.43:0/963555297 shutdown_connections 2024-03-28T22:39:14.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.327+0000 7f7d8c62c740 1 -- 172.21.15.43:0/963555297 wait complete. 2024-03-28T22:39:14.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.327+0000 7f7d8c62c740 1 -- start start 2024-03-28T22:39:14.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.327+0000 7f7d8c62c740 1 Processor -- start 2024-03-28T22:39:14.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.328+0000 7f7d8c62c740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559c5867b550 0x559c58681ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.328+0000 7f7d8c62c740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559c5867be70 0x559c58682010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.328+0000 7f7d8c62c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559c58684a50 0x559c58682550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.328+0000 7f7d8c62c740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x559c58687840 con 0x559c5867be70 2024-03-28T22:39:14.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.328+0000 7f7d8c62c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x559c5857d2f0 con 0x559c58684a50 2024-03-28T22:39:14.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.328+0000 7f7d8c62c740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x559c586879c0 con 0x559c5867b550 2024-03-28T22:39:14.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.328+0000 7f7d82a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559c5867b550 0x559c58681ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.328+0000 7f7d82a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559c5867b550 0x559c58681ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43130/0 (socket says 172.21.15.43:43130) 2024-03-28T22:39:14.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.328+0000 7f7d82a00640 1 -- 172.21.15.43:0/137313169 learned_addr learned my addr 172.21.15.43:0/137313169 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:14.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.328+0000 7f7d83400640 1 --2- 172.21.15.43:0/137313169 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559c58684a50 0x559c58682550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.328+0000 7f7d8c62c740 10 client.0 _ll_register_callbacks cb 0x559c582f2be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:14.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.328+0000 7f7d82000640 1 --2- 172.21.15.43:0/137313169 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559c5867be70 0x559c58682010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.329+0000 7f7d82a00640 1 -- 172.21.15.43:0/137313169 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559c58684a50 msgr2=0x559c58682550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:14.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.329+0000 7f7d82a00640 1 --2- 172.21.15.43:0/137313169 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559c58684a50 0x559c58682550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.329+0000 7f7d82a00640 1 -- 172.21.15.43:0/137313169 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559c5867be70 msgr2=0x559c58682010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:14.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.329+0000 7f7d82a00640 1 --2- 172.21.15.43:0/137313169 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559c5867be70 0x559c58682010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.329+0000 7f7d82a00640 1 -- 172.21.15.43:0/137313169 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c583bd7d0 con 0x559c5867b550 2024-03-28T22:39:14.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.329+0000 7f7d82a00640 1 --2- 172.21.15.43:0/137313169 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559c5867b550 0x559c58681ad0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7d74000c00 tx=0x7f7d740b8df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbea1157188b97bd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:14.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.329+0000 7f7d6cc00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:39:14.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.329+0000 7f7d6cc00640 1 -- 172.21.15.43:0/137313169 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d740bd190 con 0x559c5867b550 2024-03-28T22:39:14.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.329+0000 7f7d6cc00640 1 -- 172.21.15.43:0/137313169 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f7d740054e0 con 0x559c5867b550 2024-03-28T22:39:14.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.329+0000 7f7d8c62c740 1 -- 172.21.15.43:0/137313169 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d700041c0 con 0x559c5867b550 2024-03-28T22:39:14.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.329+0000 7f7d6cc00640 1 -- 172.21.15.43:0/137313169 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7d740bca90 con 0x559c5867b550 2024-03-28T22:39:14.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.329+0000 7f7d8c62c740 10 client.26356 fetch_fsmap learned FSMap version 29 2024-03-28T22:39:14.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.329+0000 7f7d8c62c740 1 -- 172.21.15.43:0/137313169 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559c586878a0 con 0x559c5867b550 2024-03-28T22:39:14.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.330+0000 7f7d6cc00640 1 -- 172.21.15.43:0/137313169 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d700041c0 con 0x559c5867b550 2024-03-28T22:39:14.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.330+0000 7f7d6cc00640 1 -- 172.21.15.43:0/137313169 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f7d740bccf0 con 0x559c5867b550 2024-03-28T22:39:14.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.330+0000 7f7d8c62c740 10 client.26356 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:39:14.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.330+0000 7f7d8c62c740 10 client.26356 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:39:14.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.330+0000 7f7d8c62c740 20 client.26356 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:14.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.330+0000 7f7d8c62c740 1 --2- 172.21.15.43:0/137313169 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x559c58695000 0x559c58697470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.330+0000 7f7d8c62c740 4 client.26356 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:39:14.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.330+0000 7f7d8c62c740 1 -- 172.21.15.43:0/137313169 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559c58588680 con 0x559c58695000 2024-03-28T22:39:14.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.331+0000 7f7d82000640 1 --2- 172.21.15.43:0/137313169 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x559c58695000 0x559c58697470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.332+0000 7f7d6cc00640 10 client.26356 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:39:14.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.332+0000 7f7d82000640 1 --2- 172.21.15.43:0/137313169 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x559c58695000 0x559c58697470 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d8649879071d6c6e server_cookie=89c86d0b63bec0b3 in_seq=0 out_seq=0 2024-03-28T22:39:14.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.332+0000 7f7d6cc00640 1 -- 172.21.15.43:0/137313169 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559c58588680 con 0x559c58695000 2024-03-28T22:39:14.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.332+0000 7f7d6cc00640 10 client.26356 handle_command_reply: tid=0 2024-03-28T22:39:14.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.332+0000 7f7d8c62c740 1 client.26356 shutdown 2024-03-28T22:39:14.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.333+0000 7f7d8c62c740 1 -- 172.21.15.43:0/137313169 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559c5867b550 msgr2=0x559c58681ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.333+0000 7f7d8c62c740 1 --2- 172.21.15.43:0/137313169 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559c5867b550 0x559c58681ad0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7d74000c00 tx=0x7f7d740b8df0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.333+0000 7f7d8c62c740 1 -- 172.21.15.43:0/137313169 shutdown_connections 2024-03-28T22:39:14.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.333+0000 7f7d8c62c740 1 --2- 172.21.15.43:0/137313169 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x559c58695000 0x559c58697470 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.333+0000 7f7d8c62c740 1 --2- 172.21.15.43:0/137313169 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559c58684a50 0x559c58682550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.333+0000 7f7d8c62c740 1 --2- 172.21.15.43:0/137313169 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559c5867be70 0x559c58682010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.333+0000 7f7d6cc00640 0 client.26356 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:39:14.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.333+0000 7f7d8c62c740 1 --2- 172.21.15.43:0/137313169 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559c5867b550 0x559c58681ad0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.333+0000 7f7d8c62c740 1 -- 172.21.15.43:0/137313169 >> 172.21.15.43:0/137313169 conn(0x559c58675c80 msgr2=0x559c586779d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:14.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.333+0000 7f7d8c62c740 1 -- 172.21.15.43:0/137313169 shutdown_connections 2024-03-28T22:39:14.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.333+0000 7f7d8c62c740 1 -- 172.21.15.43:0/137313169 wait complete. 2024-03-28T22:39:14.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.333+0000 7f7d8c62c740 20 client.26356 trim_cache size 0 max 16384 2024-03-28T22:39:14.343 INFO:tasks.workunit.client.0.smithi043.stdout: 1 73414 9.12 MB/sec warmup 45 sec latency 87.697 ms 2024-03-28T22:39:14.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.350+0000 7f7d8c62c740 1 Processor -- start 2024-03-28T22:39:14.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.350+0000 7f7d8c62c740 1 -- start start 2024-03-28T22:39:14.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.350+0000 7f7d8c62c740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559c5867b550 0x559c58684200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.350+0000 7f7d8c62c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559c5867be70 0x559c586777c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.350+0000 7f7d8c62c740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559c58684a50 0x559c5869cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.350+0000 7f7d8c62c740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7d841b0c20 con 0x559c5867b550 2024-03-28T22:39:14.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.350+0000 7f7d8c62c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7d841b2760 con 0x559c5867be70 2024-03-28T22:39:14.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.350+0000 7f7d8c62c740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f7d841b12f0 con 0x559c58684a50 2024-03-28T22:39:14.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.351+0000 7f7d82a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559c5867b550 0x559c58684200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.351+0000 7f7d82000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559c5867be70 0x559c586777c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.351+0000 7f7d83400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559c58684a50 0x559c5869cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.351+0000 7f7d83400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559c58684a50 0x559c5869cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43132/0 (socket says 172.21.15.43:43132) 2024-03-28T22:39:14.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.351+0000 7f7d82000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559c5867be70 0x559c586777c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59548/0 (socket says 172.21.15.43:59548) 2024-03-28T22:39:14.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.351+0000 7f7d83400640 1 -- 172.21.15.43:0/2965176584 learned_addr learned my addr 172.21.15.43:0/2965176584 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:14.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.351+0000 7f7d82a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559c5867b550 0x559c58684200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53448/0 (socket says 172.21.15.43:53448) 2024-03-28T22:39:14.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.351+0000 7f7d82000640 1 -- 172.21.15.43:0/2965176584 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559c58684a50 msgr2=0x559c5869cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:14.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.351+0000 7f7d82000640 1 --2- 172.21.15.43:0/2965176584 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559c58684a50 0x559c5869cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.351+0000 7f7d82000640 1 -- 172.21.15.43:0/2965176584 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559c5867b550 msgr2=0x559c58684200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:14.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.351+0000 7f7d82000640 1 --2- 172.21.15.43:0/2965176584 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559c5867b550 0x559c58684200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.351+0000 7f7d82000640 1 -- 172.21.15.43:0/2965176584 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c58686e40 con 0x559c5867be70 2024-03-28T22:39:14.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.352+0000 7f7d82000640 1 --2- 172.21.15.43:0/2965176584 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559c5867be70 0x559c586777c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f7d7000a010 tx=0x7f7d70003c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=359ea3d3adb19195 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:14.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.352+0000 7f7d6f400640 1 -- 172.21.15.43:0/2965176584 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d7001d040 con 0x559c5867be70 2024-03-28T22:39:14.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.352+0000 7f7d6f400640 1 -- 172.21.15.43:0/2965176584 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d70030070 con 0x559c5867be70 2024-03-28T22:39:14.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.352+0000 7f7d6f400640 1 -- 172.21.15.43:0/2965176584 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d700054c0 con 0x559c5867be70 2024-03-28T22:39:14.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.353+0000 7f7d8c62c740 1 -- 172.21.15.43:0/2965176584 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559c5867be70 msgr2=0x559c586777c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.353+0000 7f7d8c62c740 1 --2- 172.21.15.43:0/2965176584 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559c5867be70 0x559c586777c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f7d7000a010 tx=0x7f7d70003c60 comp rx=0 tx=0).stop 2024-03-28T22:39:14.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.353+0000 7f7d8c62c740 1 -- 172.21.15.43:0/2965176584 shutdown_connections 2024-03-28T22:39:14.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.353+0000 7f7d8c62c740 1 --2- 172.21.15.43:0/2965176584 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559c5867be70 0x559c586777c0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.353+0000 7f7d8c62c740 1 --2- 172.21.15.43:0/2965176584 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559c58684a50 0x559c5869cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.353+0000 7f7d8c62c740 1 --2- 172.21.15.43:0/2965176584 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559c5867b550 0x559c58684200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.353+0000 7f7d8c62c740 1 -- 172.21.15.43:0/2965176584 >> 172.21.15.43:0/2965176584 conn(0x559c58675c80 msgr2=0x559c586770b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:14.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.353+0000 7f7d8c62c740 1 -- 172.21.15.43:0/2965176584 shutdown_connections 2024-03-28T22:39:14.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.353+0000 7f7d8c62c740 1 -- 172.21.15.43:0/2965176584 wait complete. 2024-03-28T22:39:14.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.353+0000 7f7d8c62c740 1 -- start start 2024-03-28T22:39:14.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.354+0000 7f7d8c62c740 1 Processor -- start 2024-03-28T22:39:14.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.354+0000 7f7d8c62c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559c5867b550 0x559c5868d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.354+0000 7f7d8c62c740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559c5867be70 0x559c586883b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.354+0000 7f7d8c62c740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559c58684a50 0x559c58688af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.354+0000 7f7d8c62c740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7d841b2090 con 0x559c5867be70 2024-03-28T22:39:14.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.354+0000 7f7d8c62c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7d841b19c0 con 0x559c5867b550 2024-03-28T22:39:14.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.354+0000 7f7d8c62c740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f7d841aec00 con 0x559c58684a50 2024-03-28T22:39:14.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.354+0000 7f7d8c62c740 10 client.0 _ll_register_callbacks cb 0x7f7d74009260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:14.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.354+0000 7f7d82a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559c5867b550 0x559c5868d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.354+0000 7f7d82a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559c5867b550 0x559c5868d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59560/0 (socket says 172.21.15.43:59560) 2024-03-28T22:39:14.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.354+0000 7f7d82a00640 1 -- 172.21.15.43:0/3880136733 learned_addr learned my addr 172.21.15.43:0/3880136733 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:14.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.354+0000 7f7d83400640 1 --2- 172.21.15.43:0/3880136733 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559c58684a50 0x559c58688af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.354+0000 7f7d82000640 1 --2- 172.21.15.43:0/3880136733 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559c5867be70 0x559c586883b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.355+0000 7f7d82a00640 1 -- 172.21.15.43:0/3880136733 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559c58684a50 msgr2=0x559c58688af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:14.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.355+0000 7f7d82a00640 1 --2- 172.21.15.43:0/3880136733 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559c58684a50 0x559c58688af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.355+0000 7f7d82a00640 1 -- 172.21.15.43:0/3880136733 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559c5867be70 msgr2=0x559c586883b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:14.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.355+0000 7f7d82a00640 1 --2- 172.21.15.43:0/3880136733 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559c5867be70 0x559c586883b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.355+0000 7f7d82a00640 1 -- 172.21.15.43:0/3880136733 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c586fa340 con 0x559c5867b550 2024-03-28T22:39:14.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.355+0000 7f7d82000640 1 --2- 172.21.15.43:0/3880136733 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559c5867be70 0x559c586883b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:39:14.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.355+0000 7f7d82a00640 1 --2- 172.21.15.43:0/3880136733 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559c5867b550 0x559c5868d040 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7d74006010 tx=0x7f7d7402c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddf31ed511119e5f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:14.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.355+0000 7f7d6cc00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:39:14.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.355+0000 7f7d6cc00640 1 -- 172.21.15.43:0/3880136733 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d740bd190 con 0x559c5867b550 2024-03-28T22:39:14.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.355+0000 7f7d6cc00640 1 -- 172.21.15.43:0/3880136733 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f7d740ed8d0 con 0x559c5867b550 2024-03-28T22:39:14.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.355+0000 7f7d8c62c740 1 -- 172.21.15.43:0/3880136733 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d70030070 con 0x559c5867b550 2024-03-28T22:39:14.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.355+0000 7f7d6cc00640 1 -- 172.21.15.43:0/3880136733 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7d740f3b30 con 0x559c5867b550 2024-03-28T22:39:14.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.356+0000 7f7d8c62c740 10 client.26411 fetch_fsmap learned FSMap version 29 2024-03-28T22:39:14.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.356+0000 7f7d8c62c740 1 -- 172.21.15.43:0/3880136733 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559c586fa340 con 0x559c5867b550 2024-03-28T22:39:14.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.356+0000 7f7d6cc00640 1 -- 172.21.15.43:0/3880136733 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d70030070 con 0x559c5867b550 2024-03-28T22:39:14.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.356+0000 7f7d6cc00640 1 -- 172.21.15.43:0/3880136733 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f7d740d0b70 con 0x559c5867b550 2024-03-28T22:39:14.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.356+0000 7f7d8c62c740 10 client.26411 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:39:14.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.356+0000 7f7d8c62c740 10 client.26411 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:39:14.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.356+0000 7f7d8c62c740 20 client.26411 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:14.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.356+0000 7f7d8c62c740 1 --2- 172.21.15.43:0/3880136733 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x559c586fb840 0x559c586fdc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.356+0000 7f7d8c62c740 4 client.26411 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:39:14.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.356+0000 7f7d8c62c740 1 -- 172.21.15.43:0/3880136733 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559c585ec590 con 0x559c586fb840 2024-03-28T22:39:14.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.357+0000 7f7d82000640 1 --2- 172.21.15.43:0/3880136733 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x559c586fb840 0x559c586fdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.357+0000 7f7d82000640 1 --2- 172.21.15.43:0/3880136733 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x559c586fb840 0x559c586fdc30 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ee028fc61c53a782 server_cookie=31ff8e1c39986053 in_seq=0 out_seq=0 2024-03-28T22:39:14.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.357+0000 7f7d6cc00640 10 client.26411 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:39:14.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.357+0000 7f7d6cc00640 1 -- 172.21.15.43:0/3880136733 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559c585ec590 con 0x559c586fb840 2024-03-28T22:39:14.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.357+0000 7f7d6cc00640 10 client.26411 handle_command_reply: tid=0 2024-03-28T22:39:14.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.357+0000 7f7d8c62c740 1 client.26411 shutdown 2024-03-28T22:39:14.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.358+0000 7f7d8c62c740 1 -- 172.21.15.43:0/3880136733 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559c5867b550 msgr2=0x559c5868d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.358+0000 7f7d8c62c740 1 --2- 172.21.15.43:0/3880136733 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559c5867b550 0x559c5868d040 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7d74006010 tx=0x7f7d7402c320 comp rx=0 tx=0).stop 2024-03-28T22:39:14.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.358+0000 7f7d8c62c740 1 -- 172.21.15.43:0/3880136733 shutdown_connections 2024-03-28T22:39:14.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.358+0000 7f7d8c62c740 1 --2- 172.21.15.43:0/3880136733 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559c58684a50 0x559c58688af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.358+0000 7f7d8c62c740 1 --2- 172.21.15.43:0/3880136733 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559c5867be70 0x559c586883b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.358+0000 7f7d8c62c740 1 --2- 172.21.15.43:0/3880136733 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x559c586fb840 0x559c586fdc30 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.358+0000 7f7d8c62c740 1 --2- 172.21.15.43:0/3880136733 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559c5867b550 0x559c5868d040 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.358+0000 7f7d8c62c740 1 -- 172.21.15.43:0/3880136733 >> 172.21.15.43:0/3880136733 conn(0x559c58675c80 msgr2=0x559c586770b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:14.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.358+0000 7f7d6cc00640 0 client.26411 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:39:14.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.358+0000 7f7d8c62c740 1 -- 172.21.15.43:0/3880136733 shutdown_connections 2024-03-28T22:39:14.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.358+0000 7f7d8c62c740 1 -- 172.21.15.43:0/3880136733 wait complete. 2024-03-28T22:39:14.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.358+0000 7f7d8c62c740 20 client.26411 trim_cache size 0 max 16384 2024-03-28T22:39:14.363 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:39:14.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.359+0000 7f7d83e00640 1 -- 172.21.15.43:0/465922004 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7d680a1f70 msgr2=0x7f7d680a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.359+0000 7f7d83e00640 1 --2- 172.21.15.43:0/465922004 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7d680a1f70 0x7f7d680a4430 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c009fd0 tx=0x7f7d7c009500 comp rx=0 tx=0).stop 2024-03-28T22:39:14.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.359+0000 7f7d83e00640 1 -- 172.21.15.43:0/465922004 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d841037c0 msgr2=0x7f7d8419a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.359+0000 7f7d83e00640 1 --2- 172.21.15.43:0/465922004 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d841037c0 0x7f7d8419a0f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7d74005ab0 tx=0x7f7d74005e80 comp rx=0 tx=0).stop 2024-03-28T22:39:14.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.359+0000 7f7d83e00640 1 -- 172.21.15.43:0/465922004 shutdown_connections 2024-03-28T22:39:14.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.359+0000 7f7d83e00640 1 --2- 172.21.15.43:0/465922004 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7d680a1f70 0x7f7d680a4430 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.359+0000 7f7d83e00640 1 --2- 172.21.15.43:0/465922004 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7d84104180 0x7f7d8419a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.359+0000 7f7d83e00640 1 --2- 172.21.15.43:0/465922004 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7d841097d0 0x7f7d8419e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.359+0000 7f7d83e00640 1 --2- 172.21.15.43:0/465922004 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d841037c0 0x7f7d8419a0f0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.359+0000 7f7d83e00640 1 -- 172.21.15.43:0/465922004 >> 172.21.15.43:0/465922004 conn(0x7f7d840ff230 msgr2=0x7f7d840ffa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:14.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.359+0000 7f7d83e00640 1 -- 172.21.15.43:0/465922004 shutdown_connections 2024-03-28T22:39:14.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.359+0000 7f7d83e00640 1 -- 172.21.15.43:0/465922004 wait complete. 2024-03-28T22:39:14.374 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:39:14.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.582+0000 7fb33d400640 1 -- 172.21.15.43:0/271237727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3381032e0 msgr2=0x7fb33810b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.582+0000 7fb33d400640 1 --2- 172.21.15.43:0/271237727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3381032e0 0x7fb33810b7c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb324009900 tx=0x7fb32402bcc0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.582+0000 7fb33d400640 1 -- 172.21.15.43:0/271237727 shutdown_connections 2024-03-28T22:39:14.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.582+0000 7fb33d400640 1 --2- 172.21.15.43:0/271237727 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb33810bd90 0x7fb33810e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.582+0000 7fb33d400640 1 --2- 172.21.15.43:0/271237727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3381032e0 0x7fb33810b7c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.582+0000 7fb33d400640 1 --2- 172.21.15.43:0/271237727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb3381029c0 0x7fb338102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.582+0000 7fb33d400640 1 -- 172.21.15.43:0/271237727 >> 172.21.15.43:0/271237727 conn(0x7fb3380777e0 msgr2=0x7fb338077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:14.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.582+0000 7fb33d400640 1 -- 172.21.15.43:0/271237727 shutdown_connections 2024-03-28T22:39:14.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.582+0000 7fb33d400640 1 -- 172.21.15.43:0/271237727 wait complete. 2024-03-28T22:39:14.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.583+0000 7fb33d400640 1 Processor -- start 2024-03-28T22:39:14.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.583+0000 7fb33d400640 1 -- start start 2024-03-28T22:39:14.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.584+0000 7fb33d400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb3381029c0 0x7fb33819a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.584+0000 7fb33d400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3381032e0 0x7fb33819a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.584+0000 7fb33d400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb33810bd90 0x7fb33819eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.584+0000 7fb33d400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb338111bb0 con 0x7fb33810bd90 2024-03-28T22:39:14.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.584+0000 7fb33d400640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb338111a30 con 0x7fb3381029c0 2024-03-28T22:39:14.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.584+0000 7fb33d400640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fb338111d30 con 0x7fb3381032e0 2024-03-28T22:39:14.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.584+0000 7fb335e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3381032e0 0x7fb33819a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.584+0000 7fb336800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb3381029c0 0x7fb33819a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.584+0000 7fb335e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3381032e0 0x7fb33819a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43156/0 (socket says 172.21.15.43:43156) 2024-03-28T22:39:14.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.584+0000 7fb335e00640 1 -- 172.21.15.43:0/2841672466 learned_addr learned my addr 172.21.15.43:0/2841672466 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:14.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.584+0000 7fb337200640 1 --2- 172.21.15.43:0/2841672466 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb33810bd90 0x7fb33819eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.585+0000 7fb335e00640 1 -- 172.21.15.43:0/2841672466 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb3381029c0 msgr2=0x7fb33819a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.585+0000 7fb335e00640 1 --2- 172.21.15.43:0/2841672466 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb3381029c0 0x7fb33819a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.585+0000 7fb335e00640 1 -- 172.21.15.43:0/2841672466 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb33810bd90 msgr2=0x7fb33819eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.585+0000 7fb335e00640 1 --2- 172.21.15.43:0/2841672466 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb33810bd90 0x7fb33819eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.585+0000 7fb335e00640 1 -- 172.21.15.43:0/2841672466 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb33819f240 con 0x7fb3381032e0 2024-03-28T22:39:14.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.585+0000 7fb337200640 1 --2- 172.21.15.43:0/2841672466 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb33810bd90 0x7fb33819eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:39:14.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.585+0000 7fb336800640 1 --2- 172.21.15.43:0/2841672466 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb3381029c0 0x7fb33819a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:14.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.585+0000 7fb335e00640 1 --2- 172.21.15.43:0/2841672466 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3381032e0 0x7fb33819a790 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb324000c00 tx=0x7fb3240043c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:14.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.585+0000 7fb32f400640 1 -- 172.21.15.43:0/2841672466 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb324042020 con 0x7fb3381032e0 2024-03-28T22:39:14.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.585+0000 7fb32f400640 1 -- 172.21.15.43:0/2841672466 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb324005000 con 0x7fb3381032e0 2024-03-28T22:39:14.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.585+0000 7fb32f400640 1 -- 172.21.15.43:0/2841672466 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb324003a90 con 0x7fb3381032e0 2024-03-28T22:39:14.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.585+0000 7fb33d400640 1 -- 172.21.15.43:0/2841672466 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb33819f4d0 con 0x7fb3381032e0 2024-03-28T22:39:14.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.586+0000 7fb33d400640 1 -- 172.21.15.43:0/2841672466 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3381a6dc0 con 0x7fb3381032e0 2024-03-28T22:39:14.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.589+0000 7fb340180740 1 Processor -- start 2024-03-28T22:39:14.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.589+0000 7fb340180740 1 -- start start 2024-03-28T22:39:14.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.589+0000 7fb32f400640 1 -- 172.21.15.43:0/2841672466 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fb324045020 con 0x7fb3381032e0 2024-03-28T22:39:14.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.590+0000 7fb340180740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5612277e2380 0x5612277e2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.590+0000 7fb340180740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5612277e2ca0 0x5612277eb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.590+0000 7fb340180740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5612277eb880 0x5612277edc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.590+0000 7fb340180740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x561227746050 con 0x5612277eb880 2024-03-28T22:39:14.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.590+0000 7fb340180740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5612276eece0 con 0x5612277e2380 2024-03-28T22:39:14.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.590+0000 7fb340180740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5612277420f0 con 0x5612277e2ca0 2024-03-28T22:39:14.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.590+0000 7fb336800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5612277e2ca0 0x5612277eb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.590+0000 7fb336800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5612277e2ca0 0x5612277eb340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43172/0 (socket says 172.21.15.43:43172) 2024-03-28T22:39:14.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.590+0000 7fb337200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5612277e2380 0x5612277e2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.590+0000 7fb336800640 1 -- 172.21.15.43:0/666521572 learned_addr learned my addr 172.21.15.43:0/666521572 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:14.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.590+0000 7fb335e00640 1 --2- 172.21.15.43:0/666521572 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5612277eb880 0x5612277edc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.590+0000 7fb336800640 1 -- 172.21.15.43:0/666521572 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5612277e2380 msgr2=0x5612277e2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:14.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.590+0000 7fb336800640 1 --2- 172.21.15.43:0/666521572 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5612277e2380 0x5612277e2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.590+0000 7fb336800640 1 -- 172.21.15.43:0/666521572 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5612277eb880 msgr2=0x5612277edc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:14.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.590+0000 7fb336800640 1 --2- 172.21.15.43:0/666521572 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5612277eb880 0x5612277edc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.590+0000 7fb336800640 1 -- 172.21.15.43:0/666521572 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56122745a3f0 con 0x5612277e2ca0 2024-03-28T22:39:14.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.591+0000 7fb336800640 1 --2- 172.21.15.43:0/666521572 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5612277e2ca0 0x5612277eb340 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb32800f4c0 tx=0x7fb32800f980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d51cb3a24a3b7f90 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:14.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.591+0000 7fb323400640 1 -- 172.21.15.43:0/666521572 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb328005220 con 0x5612277e2ca0 2024-03-28T22:39:14.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.591+0000 7fb323400640 1 -- 172.21.15.43:0/666521572 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb328004c80 con 0x5612277e2ca0 2024-03-28T22:39:14.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.591+0000 7fb323400640 1 -- 172.21.15.43:0/666521572 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb328010ca0 con 0x5612277e2ca0 2024-03-28T22:39:14.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.591+0000 7fb32f400640 1 --2- 172.21.15.43:0/2841672466 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb31c0a1ef0 0x7fb31c0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.591+0000 7fb32f400640 1 -- 172.21.15.43:0/2841672466 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fb3240f01b0 con 0x7fb3381032e0 2024-03-28T22:39:14.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.592+0000 7fb337200640 1 --2- 172.21.15.43:0/2841672466 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb31c0a1ef0 0x7fb31c0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.592+0000 7fb340180740 1 -- 172.21.15.43:0/666521572 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5612277e2ca0 msgr2=0x5612277eb340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.592+0000 7fb340180740 1 --2- 172.21.15.43:0/666521572 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5612277e2ca0 0x5612277eb340 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb32800f4c0 tx=0x7fb32800f980 comp rx=0 tx=0).stop 2024-03-28T22:39:14.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.592+0000 7fb340180740 1 -- 172.21.15.43:0/666521572 shutdown_connections 2024-03-28T22:39:14.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.592+0000 7fb340180740 1 --2- 172.21.15.43:0/666521572 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5612277eb880 0x5612277edc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.592+0000 7fb340180740 1 --2- 172.21.15.43:0/666521572 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5612277e2ca0 0x5612277eb340 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.592+0000 7fb340180740 1 --2- 172.21.15.43:0/666521572 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5612277e2380 0x5612277e2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.592+0000 7fb340180740 1 -- 172.21.15.43:0/666521572 >> 172.21.15.43:0/666521572 conn(0x5612277dcab0 msgr2=0x5612277dce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:14.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.592+0000 7fb340180740 1 -- 172.21.15.43:0/666521572 shutdown_connections 2024-03-28T22:39:14.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.592+0000 7fb340180740 1 -- 172.21.15.43:0/666521572 wait complete. 2024-03-28T22:39:14.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.592+0000 7fb337200640 1 --2- 172.21.15.43:0/2841672466 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb31c0a1ef0 0x7fb31c0a43b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb330005fd0 tx=0x7fb330009500 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.593+0000 7fb340180740 1 -- start start 2024-03-28T22:39:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.593+0000 7fb340180740 1 Processor -- start 2024-03-28T22:39:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.593+0000 7fb340180740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5612277e2380 0x5612277e88a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.593+0000 7fb340180740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5612277e2ca0 0x5612277e8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.593+0000 7fb340180740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5612277eb880 0x5612277e9320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.593+0000 7fb340180740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5612277ee670 con 0x5612277e2ca0 2024-03-28T22:39:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.593+0000 7fb340180740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5612276e42f0 con 0x5612277e2380 2024-03-28T22:39:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.593+0000 7fb340180740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5612277ee7f0 con 0x5612277eb880 2024-03-28T22:39:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.594+0000 7fb336800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5612277e2380 0x5612277e88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.594+0000 7fb340180740 10 client.0 _ll_register_callbacks cb 0x561227459be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.594+0000 7fb336800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5612277e2380 0x5612277e88a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59596/0 (socket says 172.21.15.43:59596) 2024-03-28T22:39:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.594+0000 7fb336800640 1 -- 172.21.15.43:0/1775797631 learned_addr learned my addr 172.21.15.43:0/1775797631 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.594+0000 7fb337200640 1 --2- 172.21.15.43:0/1775797631 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5612277eb880 0x5612277e9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.594+0000 7fb335e00640 1 --2- 172.21.15.43:0/1775797631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5612277e2ca0 0x5612277e8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.594+0000 7fb336800640 1 -- 172.21.15.43:0/1775797631 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5612277eb880 msgr2=0x5612277e9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:14.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.594+0000 7fb336800640 1 --2- 172.21.15.43:0/1775797631 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5612277eb880 0x5612277e9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.594+0000 7fb336800640 1 -- 172.21.15.43:0/1775797631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5612277e2ca0 msgr2=0x5612277e8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:14.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.594+0000 7fb336800640 1 --2- 172.21.15.43:0/1775797631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5612277e2ca0 0x5612277e8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.594+0000 7fb336800640 1 -- 172.21.15.43:0/1775797631 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612275247d0 con 0x5612277e2380 2024-03-28T22:39:14.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.594+0000 7fb337200640 1 --2- 172.21.15.43:0/1775797631 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5612277eb880 0x5612277e9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:39:14.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.594+0000 7fb335e00640 1 --2- 172.21.15.43:0/1775797631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5612277e2ca0 0x5612277e8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:39:14.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.594+0000 7fb336800640 1 --2- 172.21.15.43:0/1775797631 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5612277e2380 0x5612277e88a0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fb3280108e0 tx=0x7fb32800adc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65ffaec05256745e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:14.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.594+0000 7fb317e00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:39:14.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.595+0000 7fb317e00640 1 -- 172.21.15.43:0/1775797631 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb328026070 con 0x5612277e2380 2024-03-28T22:39:14.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.595+0000 7fb317e00640 1 -- 172.21.15.43:0/1775797631 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fb328004e30 con 0x5612277e2380 2024-03-28T22:39:14.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.595+0000 7fb340180740 1 -- 172.21.15.43:0/1775797631 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb328004c80 con 0x5612277e2380 2024-03-28T22:39:14.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.595+0000 7fb317e00640 1 -- 172.21.15.43:0/1775797631 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb328032050 con 0x5612277e2380 2024-03-28T22:39:14.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.595+0000 7fb340180740 10 client.26423 fetch_fsmap learned FSMap version 29 2024-03-28T22:39:14.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.595+0000 7fb340180740 1 -- 172.21.15.43:0/1775797631 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5612277fa8f0 con 0x5612277e2380 2024-03-28T22:39:14.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.595+0000 7fb317e00640 1 -- 172.21.15.43:0/1775797631 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb328004c80 con 0x5612277e2380 2024-03-28T22:39:14.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.595+0000 7fb317e00640 1 -- 172.21.15.43:0/1775797631 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb328018930 con 0x5612277e2380 2024-03-28T22:39:14.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.596+0000 7fb340180740 10 client.26423 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:39:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.596+0000 7fb340180740 10 client.26423 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:39:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.596+0000 7fb340180740 20 client.26423 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.596+0000 7fb340180740 1 --2- 172.21.15.43:0/1775797631 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x5612277fbd00 0x5612277fe0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.596+0000 7fb340180740 4 client.26423 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:39:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.596+0000 7fb340180740 1 -- 172.21.15.43:0/1775797631 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612276ef680 con 0x5612277fbd00 2024-03-28T22:39:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.596+0000 7fb335e00640 1 --2- 172.21.15.43:0/1775797631 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x5612277fbd00 0x5612277fe0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.596+0000 7fb335e00640 1 --2- 172.21.15.43:0/1775797631 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x5612277fbd00 0x5612277fe0f0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a0ea3caee0c58830 server_cookie=84523a4d539c9487 in_seq=0 out_seq=0 2024-03-28T22:39:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.596+0000 7fb317e00640 10 client.26423 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:39:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.597+0000 7fb317e00640 1 -- 172.21.15.43:0/1775797631 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5612276ef680 con 0x5612277fbd00 2024-03-28T22:39:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.597+0000 7fb317e00640 10 client.26423 handle_command_reply: tid=0 2024-03-28T22:39:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.597+0000 7fb340180740 1 client.26423 shutdown 2024-03-28T22:39:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.597+0000 7fb340180740 1 -- 172.21.15.43:0/1775797631 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5612277e2380 msgr2=0x5612277e88a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.597+0000 7fb340180740 1 --2- 172.21.15.43:0/1775797631 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5612277e2380 0x5612277e88a0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fb3280108e0 tx=0x7fb32800adc0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.597+0000 7fb340180740 1 -- 172.21.15.43:0/1775797631 shutdown_connections 2024-03-28T22:39:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.597+0000 7fb340180740 1 --2- 172.21.15.43:0/1775797631 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x5612277fbd00 0x5612277fe0f0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.597+0000 7fb340180740 1 --2- 172.21.15.43:0/1775797631 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5612277eb880 0x5612277e9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.597+0000 7fb340180740 1 --2- 172.21.15.43:0/1775797631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5612277e2ca0 0x5612277e8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.597+0000 7fb340180740 1 --2- 172.21.15.43:0/1775797631 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5612277e2380 0x5612277e88a0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.597+0000 7fb317e00640 0 client.26423 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:39:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.597+0000 7fb340180740 1 -- 172.21.15.43:0/1775797631 >> 172.21.15.43:0/1775797631 conn(0x5612277dcab0 msgr2=0x5612277de800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.598+0000 7fb340180740 1 -- 172.21.15.43:0/1775797631 shutdown_connections 2024-03-28T22:39:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.598+0000 7fb340180740 1 -- 172.21.15.43:0/1775797631 wait complete. 2024-03-28T22:39:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.598+0000 7fb340180740 20 client.26423 trim_cache size 0 max 16384 2024-03-28T22:39:14.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.613+0000 7fb340180740 1 Processor -- start 2024-03-28T22:39:14.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.613+0000 7fb340180740 1 -- start start 2024-03-28T22:39:14.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.613+0000 7fb340180740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5612277e2380 0x5612277f7410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.613+0000 7fb340180740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5612277e2ca0 0x5612277e7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.613+0000 7fb340180740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5612277eb880 0x5612277db9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.613+0000 7fb340180740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb3381b0d70 con 0x5612277eb880 2024-03-28T22:39:14.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.613+0000 7fb340180740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb3381b28b0 con 0x5612277e2380 2024-03-28T22:39:14.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.613+0000 7fb340180740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fb3381b1440 con 0x5612277e2ca0 2024-03-28T22:39:14.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.614+0000 7fb336800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5612277e2380 0x5612277f7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.614+0000 7fb336800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5612277e2380 0x5612277f7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59612/0 (socket says 172.21.15.43:59612) 2024-03-28T22:39:14.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.614+0000 7fb336800640 1 -- 172.21.15.43:0/1698341520 learned_addr learned my addr 172.21.15.43:0/1698341520 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:14.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.614+0000 7fb335e00640 1 --2- 172.21.15.43:0/1698341520 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5612277e2ca0 0x5612277e7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.614+0000 7fb337200640 1 --2- 172.21.15.43:0/1698341520 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5612277eb880 0x5612277db9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.614+0000 7fb336800640 1 -- 172.21.15.43:0/1698341520 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5612277e2ca0 msgr2=0x5612277e7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:14.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.614+0000 7fb336800640 1 --2- 172.21.15.43:0/1698341520 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5612277e2ca0 0x5612277e7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.614+0000 7fb336800640 1 -- 172.21.15.43:0/1698341520 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5612277eb880 msgr2=0x5612277db9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:14.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.614+0000 7fb336800640 1 --2- 172.21.15.43:0/1698341520 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5612277eb880 0x5612277db9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.614+0000 7fb336800640 1 -- 172.21.15.43:0/1698341520 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612277e1d50 con 0x5612277e2380 2024-03-28T22:39:14.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.614+0000 7fb336800640 1 --2- 172.21.15.43:0/1698341520 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5612277e2380 0x5612277f7410 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fb32800a330 tx=0x7fb32800f980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a10f08463f96e8a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:14.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.615+0000 7fb322a00640 1 -- 172.21.15.43:0/1698341520 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32803b070 con 0x5612277e2380 2024-03-28T22:39:14.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.615+0000 7fb322a00640 1 -- 172.21.15.43:0/1698341520 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb32800a9e0 con 0x5612277e2380 2024-03-28T22:39:14.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.615+0000 7fb322a00640 1 -- 172.21.15.43:0/1698341520 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb328026040 con 0x5612277e2380 2024-03-28T22:39:14.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.615+0000 7fb340180740 1 -- 172.21.15.43:0/1698341520 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5612277e2380 msgr2=0x5612277f7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.615+0000 7fb340180740 1 --2- 172.21.15.43:0/1698341520 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5612277e2380 0x5612277f7410 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fb32800a330 tx=0x7fb32800f980 comp rx=0 tx=0).stop 2024-03-28T22:39:14.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.615+0000 7fb340180740 1 -- 172.21.15.43:0/1698341520 shutdown_connections 2024-03-28T22:39:14.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.615+0000 7fb340180740 1 --2- 172.21.15.43:0/1698341520 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5612277eb880 0x5612277db9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.615+0000 7fb340180740 1 --2- 172.21.15.43:0/1698341520 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5612277e2ca0 0x5612277e7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.615+0000 7fb340180740 1 --2- 172.21.15.43:0/1698341520 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5612277e2380 0x5612277f7410 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.615+0000 7fb340180740 1 -- 172.21.15.43:0/1698341520 >> 172.21.15.43:0/1698341520 conn(0x5612277dcab0 msgr2=0x5612277db2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:14.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.616+0000 7fb340180740 1 -- 172.21.15.43:0/1698341520 shutdown_connections 2024-03-28T22:39:14.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.616+0000 7fb340180740 1 -- 172.21.15.43:0/1698341520 wait complete. 2024-03-28T22:39:14.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.616+0000 7fb340180740 1 -- start start 2024-03-28T22:39:14.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.616+0000 7fb340180740 1 Processor -- start 2024-03-28T22:39:14.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.617+0000 7fb340180740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5612277e2380 0x5612277f3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.617+0000 7fb340180740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5612277e2ca0 0x5612277f3c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.617+0000 7fb340180740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5612277eb880 0x5612277eeb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.617+0000 7fb340180740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb3381b21e0 con 0x5612277e2380 2024-03-28T22:39:14.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.617+0000 7fb340180740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb3381b1b10 con 0x5612277eb880 2024-03-28T22:39:14.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.617+0000 7fb340180740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fb3381aed50 con 0x5612277e2ca0 2024-03-28T22:39:14.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.617+0000 7fb340180740 10 client.0 _ll_register_callbacks cb 0x7fb3300099c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:14.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.617+0000 7fb337200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5612277eb880 0x5612277eeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.617+0000 7fb335e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5612277e2ca0 0x5612277f3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.617+0000 7fb336800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5612277e2380 0x5612277f3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.617+0000 7fb337200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5612277eb880 0x5612277eeb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59618/0 (socket says 172.21.15.43:59618) 2024-03-28T22:39:14.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.617+0000 7fb337200640 1 -- 172.21.15.43:0/177645511 learned_addr learned my addr 172.21.15.43:0/177645511 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:14.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.617+0000 7fb335e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5612277e2ca0 0x5612277f3c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43200/0 (socket says 172.21.15.43:43200) 2024-03-28T22:39:14.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.617+0000 7fb335e00640 1 -- 172.21.15.43:0/177645511 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5612277eb880 msgr2=0x5612277eeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:14.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.617+0000 7fb335e00640 1 --2- 172.21.15.43:0/177645511 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5612277eb880 0x5612277eeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.617+0000 7fb335e00640 1 -- 172.21.15.43:0/177645511 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5612277e2380 msgr2=0x5612277f3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:14.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.617+0000 7fb335e00640 1 --2- 172.21.15.43:0/177645511 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5612277e2380 0x5612277f3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.617+0000 7fb335e00640 1 -- 172.21.15.43:0/177645511 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612277ef360 con 0x5612277e2ca0 2024-03-28T22:39:14.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.617+0000 7fb337200640 1 --2- 172.21.15.43:0/177645511 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5612277eb880 0x5612277eeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:14.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.618+0000 7fb335e00640 1 --2- 172.21.15.43:0/177645511 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5612277e2ca0 0x5612277f3c50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb3240ed610 tx=0x7fb324005e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60314116828cfeeb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:14.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.618+0000 7fb317e00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:39:14.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.618+0000 7fb317e00640 1 -- 172.21.15.43:0/177645511 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3240bcfb0 con 0x5612277e2ca0 2024-03-28T22:39:14.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.618+0000 7fb340180740 1 -- 172.21.15.43:0/177645511 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb32800a9e0 con 0x5612277e2ca0 2024-03-28T22:39:14.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.618+0000 7fb317e00640 1 -- 172.21.15.43:0/177645511 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7fb3240ba950 con 0x5612277e2ca0 2024-03-28T22:39:14.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.618+0000 7fb317e00640 1 -- 172.21.15.43:0/177645511 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb3240bc8b0 con 0x5612277e2ca0 2024-03-28T22:39:14.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.618+0000 7fb340180740 10 client.26380 fetch_fsmap learned FSMap version 29 2024-03-28T22:39:14.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.618+0000 7fb340180740 1 -- 172.21.15.43:0/177645511 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5612278618a0 con 0x5612277e2ca0 2024-03-28T22:39:14.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.618+0000 7fb317e00640 1 -- 172.21.15.43:0/177645511 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb32800a9e0 con 0x5612277e2ca0 2024-03-28T22:39:14.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.619+0000 7fb317e00640 1 -- 172.21.15.43:0/177645511 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb3240a70b0 con 0x5612277e2ca0 2024-03-28T22:39:14.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.619+0000 7fb340180740 10 client.26380 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:39:14.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.619+0000 7fb340180740 10 client.26380 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:39:14.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.619+0000 7fb340180740 20 client.26380 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:14.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.619+0000 7fb340180740 1 --2- 172.21.15.43:0/177645511 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x561227861f80 0x5612278643a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.619+0000 7fb340180740 4 client.26380 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:39:14.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.619+0000 7fb340180740 1 -- 172.21.15.43:0/177645511 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561227753590 con 0x561227861f80 2024-03-28T22:39:14.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.619+0000 7fb336800640 1 --2- 172.21.15.43:0/177645511 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x561227861f80 0x5612278643a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.619+0000 7fb336800640 1 --2- 172.21.15.43:0/177645511 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x561227861f80 0x5612278643a0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8dcd67999965dbde server_cookie=2c6890365806441b in_seq=0 out_seq=0 2024-03-28T22:39:14.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.619+0000 7fb317e00640 10 client.26380 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:39:14.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.620+0000 7fb317e00640 1 -- 172.21.15.43:0/177645511 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561227753590 con 0x561227861f80 2024-03-28T22:39:14.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.620+0000 7fb317e00640 10 client.26380 handle_command_reply: tid=0 2024-03-28T22:39:14.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.620+0000 7fb340180740 1 client.26380 shutdown 2024-03-28T22:39:14.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.620+0000 7fb340180740 1 -- 172.21.15.43:0/177645511 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5612277e2ca0 msgr2=0x5612277f3c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.620+0000 7fb340180740 1 --2- 172.21.15.43:0/177645511 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5612277e2ca0 0x5612277f3c50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb3240ed610 tx=0x7fb324005e60 comp rx=0 tx=0).stop 2024-03-28T22:39:14.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.620+0000 7fb340180740 1 -- 172.21.15.43:0/177645511 shutdown_connections 2024-03-28T22:39:14.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.620+0000 7fb340180740 1 --2- 172.21.15.43:0/177645511 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x561227861f80 0x5612278643a0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.620+0000 7fb340180740 1 --2- 172.21.15.43:0/177645511 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5612277eb880 0x5612277eeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.620+0000 7fb340180740 1 --2- 172.21.15.43:0/177645511 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5612277e2ca0 0x5612277f3c50 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.620+0000 7fb340180740 1 --2- 172.21.15.43:0/177645511 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5612277e2380 0x5612277f3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.620+0000 7fb317e00640 0 client.26380 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:39:14.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.620+0000 7fb340180740 1 -- 172.21.15.43:0/177645511 >> 172.21.15.43:0/177645511 conn(0x5612277dcab0 msgr2=0x5612277ddab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:14.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.620+0000 7fb340180740 1 -- 172.21.15.43:0/177645511 shutdown_connections 2024-03-28T22:39:14.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.620+0000 7fb340180740 1 -- 172.21.15.43:0/177645511 wait complete. 2024-03-28T22:39:14.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.620+0000 7fb340180740 20 client.26380 trim_cache size 0 max 16384 2024-03-28T22:39:14.623 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:39:14.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.621+0000 7fb33d400640 1 -- 172.21.15.43:0/2841672466 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb31c0a1ef0 msgr2=0x7fb31c0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.621+0000 7fb33d400640 1 --2- 172.21.15.43:0/2841672466 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb31c0a1ef0 0x7fb31c0a43b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb330005fd0 tx=0x7fb330009500 comp rx=0 tx=0).stop 2024-03-28T22:39:14.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.621+0000 7fb33d400640 1 -- 172.21.15.43:0/2841672466 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3381032e0 msgr2=0x7fb33819a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.621+0000 7fb33d400640 1 --2- 172.21.15.43:0/2841672466 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3381032e0 0x7fb33819a790 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb324000c00 tx=0x7fb3240043c0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.621+0000 7fb33d400640 1 -- 172.21.15.43:0/2841672466 shutdown_connections 2024-03-28T22:39:14.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.621+0000 7fb33d400640 1 --2- 172.21.15.43:0/2841672466 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb31c0a1ef0 0x7fb31c0a43b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.621+0000 7fb33d400640 1 --2- 172.21.15.43:0/2841672466 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb33810bd90 0x7fb33819eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.621+0000 7fb33d400640 1 --2- 172.21.15.43:0/2841672466 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb3381032e0 0x7fb33819a790 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.621+0000 7fb33d400640 1 --2- 172.21.15.43:0/2841672466 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb3381029c0 0x7fb33819a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.621+0000 7fb33d400640 1 -- 172.21.15.43:0/2841672466 >> 172.21.15.43:0/2841672466 conn(0x7fb3380777e0 msgr2=0x7fb3380fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:14.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.621+0000 7fb33d400640 1 -- 172.21.15.43:0/2841672466 shutdown_connections 2024-03-28T22:39:14.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.621+0000 7fb33d400640 1 -- 172.21.15.43:0/2841672466 wait complete. 2024-03-28T22:39:14.634 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:14 smithi043 ceph-mon[31823]: pgmap v64: 129 pgs: 129 active+clean; 41 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.1 MiB/s wr, 395 op/s 2024-03-28T22:39:14.637 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:39:14.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.847+0000 7f3f5ec00640 1 -- 172.21.15.43:0/997741665 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3f58107f60 msgr2=0x7f3f58108340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.847+0000 7f3f5ec00640 1 --2- 172.21.15.43:0/997741665 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3f58107f60 0x7f3f58108340 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3f48009900 tx=0x7f3f4802bc40 comp rx=0 tx=0).stop 2024-03-28T22:39:14.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.847+0000 7f3f5ec00640 1 -- 172.21.15.43:0/997741665 shutdown_connections 2024-03-28T22:39:14.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.847+0000 7f3f5ec00640 1 --2- 172.21.15.43:0/997741665 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3f5810d030 0x7f3f58114c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.847+0000 7f3f5ec00640 1 --2- 172.21.15.43:0/997741665 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f58108910 0x7f3f5810c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.847+0000 7f3f5ec00640 1 --2- 172.21.15.43:0/997741665 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3f58107f60 0x7f3f58108340 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.847+0000 7f3f5ec00640 1 -- 172.21.15.43:0/997741665 >> 172.21.15.43:0/997741665 conn(0x7f3f58075c90 msgr2=0x7f3f580780b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:14.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.847+0000 7f3f5ec00640 1 -- 172.21.15.43:0/997741665 shutdown_connections 2024-03-28T22:39:14.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.847+0000 7f3f5ec00640 1 -- 172.21.15.43:0/997741665 wait complete. 2024-03-28T22:39:14.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.848+0000 7f3f5ec00640 1 Processor -- start 2024-03-28T22:39:14.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.848+0000 7f3f5ec00640 1 -- start start 2024-03-28T22:39:14.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.849+0000 7f3f5ec00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3f58107f60 0x7f3f5819e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.849+0000 7f3f5ec00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f5810d030 0x7f3f5819eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.849+0000 7f3f5ec00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3f581a2ea0 0x7f3f581a3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.849+0000 7f3f5ec00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3f581180f0 con 0x7f3f581a2ea0 2024-03-28T22:39:14.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.849+0000 7f3f5ec00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f3f58117f70 con 0x7f3f58107f60 2024-03-28T22:39:14.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.849+0000 7f3f5ec00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f3f58118270 con 0x7f3f5810d030 2024-03-28T22:39:14.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.849+0000 7f3f57e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3f58107f60 0x7f3f5819e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.849+0000 7f3f57e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3f58107f60 0x7f3f5819e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59640/0 (socket says 172.21.15.43:59640) 2024-03-28T22:39:14.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.849+0000 7f3f57e00640 1 -- 172.21.15.43:0/398134098 learned_addr learned my addr 172.21.15.43:0/398134098 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:14.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.849+0000 7f3f57400640 1 --2- 172.21.15.43:0/398134098 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f5810d030 0x7f3f5819eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.849+0000 7f3f5cc00640 1 --2- 172.21.15.43:0/398134098 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3f581a2ea0 0x7f3f581a3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.850+0000 7f3f57e00640 1 -- 172.21.15.43:0/398134098 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f5810d030 msgr2=0x7f3f5819eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.850+0000 7f3f57e00640 1 --2- 172.21.15.43:0/398134098 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f5810d030 0x7f3f5819eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.850+0000 7f3f57e00640 1 -- 172.21.15.43:0/398134098 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3f581a2ea0 msgr2=0x7f3f581a3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.850+0000 7f3f57e00640 1 --2- 172.21.15.43:0/398134098 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3f581a2ea0 0x7f3f581a3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.850+0000 7f3f57e00640 1 -- 172.21.15.43:0/398134098 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f581a3990 con 0x7f3f58107f60 2024-03-28T22:39:14.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.850+0000 7f3f57400640 1 --2- 172.21.15.43:0/398134098 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f5810d030 0x7f3f5819eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:14.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.850+0000 7f3f57e00640 1 --2- 172.21.15.43:0/398134098 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3f58107f60 0x7f3f5819e630 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f3f48006010 tx=0x7f3f4802cb40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:14.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.850+0000 7f3f54c00640 1 -- 172.21.15.43:0/398134098 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f48041070 con 0x7f3f58107f60 2024-03-28T22:39:14.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.850+0000 7f3f54c00640 1 -- 172.21.15.43:0/398134098 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f48004450 con 0x7f3f58107f60 2024-03-28T22:39:14.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.850+0000 7f3f54c00640 1 -- 172.21.15.43:0/398134098 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f48003e30 con 0x7f3f58107f60 2024-03-28T22:39:14.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.850+0000 7f3f5ec00640 1 -- 172.21.15.43:0/398134098 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f581a3c20 con 0x7f3f58107f60 2024-03-28T22:39:14.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.850+0000 7f3f5ec00640 1 -- 172.21.15.43:0/398134098 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f581ab3e0 con 0x7f3f58107f60 2024-03-28T22:39:14.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.855+0000 7f3f617e5740 1 Processor -- start 2024-03-28T22:39:14.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.855+0000 7f3f617e5740 1 -- start start 2024-03-28T22:39:14.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.856+0000 7f3f617e5740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f5c6be8380 0x55f5c6be8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.856+0000 7f3f617e5740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f5c6be8ca0 0x55f5c6bf1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.856+0000 7f3f617e5740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f5c6bf1880 0x55f5c6bf3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.856+0000 7f3f617e5740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55f5c6b4c050 con 0x55f5c6be8380 2024-03-28T22:39:14.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.856+0000 7f3f617e5740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55f5c6af4ce0 con 0x55f5c6be8ca0 2024-03-28T22:39:14.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.856+0000 7f3f617e5740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55f5c6b480f0 con 0x55f5c6bf1880 2024-03-28T22:39:14.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.856+0000 7f3f5cc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f5c6be8380 0x55f5c6be8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.856+0000 7f3f5cc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f5c6be8380 0x55f5c6be8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53530/0 (socket says 172.21.15.43:53530) 2024-03-28T22:39:14.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.856+0000 7f3f5cc00640 1 -- 172.21.15.43:0/1048674930 learned_addr learned my addr 172.21.15.43:0/1048674930 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:14.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.856+0000 7f3f57400640 1 --2- 172.21.15.43:0/1048674930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f5c6bf1880 0x55f5c6bf3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.856+0000 7f3f54c00640 1 -- 172.21.15.43:0/398134098 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f3f48044070 con 0x7f3f58107f60 2024-03-28T22:39:14.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.856+0000 7f3f57e00640 1 --2- 172.21.15.43:0/1048674930 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f5c6be8ca0 0x55f5c6bf1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.856+0000 7f3f57400640 1 -- 172.21.15.43:0/1048674930 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f5c6be8ca0 msgr2=0x55f5c6bf1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:14.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.856+0000 7f3f57400640 1 --2- 172.21.15.43:0/1048674930 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f5c6be8ca0 0x55f5c6bf1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.856+0000 7f3f57400640 1 -- 172.21.15.43:0/1048674930 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f5c6be8380 msgr2=0x55f5c6be8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:14.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.856+0000 7f3f57400640 1 --2- 172.21.15.43:0/1048674930 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f5c6be8380 0x55f5c6be8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.856+0000 7f3f57400640 1 -- 172.21.15.43:0/1048674930 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5c68603f0 con 0x55f5c6bf1880 2024-03-28T22:39:14.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.856+0000 7f3f5cc00640 1 --2- 172.21.15.43:0/1048674930 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f5c6be8380 0x55f5c6be8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:14.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.857+0000 7f3f57400640 1 --2- 172.21.15.43:0/1048674930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f5c6bf1880 0x55f5c6bf3c70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3f5806ada0 tx=0x7f3f440118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fdadd627c8da35e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:14.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.857+0000 7f3f4d200640 1 -- 172.21.15.43:0/1048674930 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f44004310 con 0x55f5c6bf1880 2024-03-28T22:39:14.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.857+0000 7f3f4d200640 1 -- 172.21.15.43:0/1048674930 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f440044b0 con 0x55f5c6bf1880 2024-03-28T22:39:14.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.857+0000 7f3f4d200640 1 -- 172.21.15.43:0/1048674930 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f440035f0 con 0x55f5c6bf1880 2024-03-28T22:39:14.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.858+0000 7f3f54c00640 1 --2- 172.21.15.43:0/398134098 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f3f3c0a1ef0 0x7f3f3c0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.858+0000 7f3f54c00640 1 -- 172.21.15.43:0/398134098 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f3f480f0250 con 0x7f3f58107f60 2024-03-28T22:39:14.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.858+0000 7f3f5cc00640 1 --2- 172.21.15.43:0/398134098 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f3f3c0a1ef0 0x7f3f3c0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.858+0000 7f3f617e5740 1 -- 172.21.15.43:0/1048674930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f5c6bf1880 msgr2=0x55f5c6bf3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.858+0000 7f3f617e5740 1 --2- 172.21.15.43:0/1048674930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f5c6bf1880 0x55f5c6bf3c70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3f5806ada0 tx=0x7f3f440118d0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.858+0000 7f3f617e5740 1 -- 172.21.15.43:0/1048674930 shutdown_connections 2024-03-28T22:39:14.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.858+0000 7f3f617e5740 1 --2- 172.21.15.43:0/1048674930 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f5c6be8ca0 0x55f5c6bf1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.858+0000 7f3f617e5740 1 --2- 172.21.15.43:0/1048674930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f5c6bf1880 0x55f5c6bf3c70 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.858+0000 7f3f617e5740 1 --2- 172.21.15.43:0/1048674930 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f5c6be8380 0x55f5c6be8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.858+0000 7f3f617e5740 1 -- 172.21.15.43:0/1048674930 >> 172.21.15.43:0/1048674930 conn(0x55f5c6be2ab0 msgr2=0x55f5c6be2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:14.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.858+0000 7f3f617e5740 1 -- 172.21.15.43:0/1048674930 shutdown_connections 2024-03-28T22:39:14.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.859+0000 7f3f617e5740 1 -- 172.21.15.43:0/1048674930 wait complete. 2024-03-28T22:39:14.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.859+0000 7f3f5cc00640 1 --2- 172.21.15.43:0/398134098 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f3f3c0a1ef0 0x7f3f3c0a43b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3f50009fd0 tx=0x7f3f50009500 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:14.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.859+0000 7f3f617e5740 1 -- start start 2024-03-28T22:39:14.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.859+0000 7f3f617e5740 1 Processor -- start 2024-03-28T22:39:14.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.860+0000 7f3f617e5740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f5c6be8380 0x55f5c6bee810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.860+0000 7f3f617e5740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f5c6be8ca0 0x55f5c6beed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.860+0000 7f3f617e5740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f5c6bf1880 0x55f5c6bef290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.860+0000 7f3f617e5740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55f5c6bf4670 con 0x55f5c6bf1880 2024-03-28T22:39:14.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.860+0000 7f3f617e5740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55f5c6aea2f0 con 0x55f5c6be8380 2024-03-28T22:39:14.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.860+0000 7f3f617e5740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55f5c6bf47f0 con 0x55f5c6be8ca0 2024-03-28T22:39:14.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.860+0000 7f3f57e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f5c6be8380 0x55f5c6bee810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.860+0000 7f3f57e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f5c6be8380 0x55f5c6bee810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59666/0 (socket says 172.21.15.43:59666) 2024-03-28T22:39:14.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.860+0000 7f3f57400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f5c6be8ca0 0x55f5c6beed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.860+0000 7f3f57e00640 1 -- 172.21.15.43:0/1679395373 learned_addr learned my addr 172.21.15.43:0/1679395373 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:14.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.860+0000 7f3f617e5740 10 client.0 _ll_register_callbacks cb 0x55f5c685fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:14.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.860+0000 7f3f5cc00640 1 --2- 172.21.15.43:0/1679395373 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f5c6bf1880 0x55f5c6bef290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.860+0000 7f3f57e00640 1 -- 172.21.15.43:0/1679395373 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f5c6be8ca0 msgr2=0x55f5c6beed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:14.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.860+0000 7f3f57e00640 1 --2- 172.21.15.43:0/1679395373 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f5c6be8ca0 0x55f5c6beed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.860+0000 7f3f57e00640 1 -- 172.21.15.43:0/1679395373 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f5c6bf1880 msgr2=0x55f5c6bef290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:14.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.860+0000 7f3f57e00640 1 --2- 172.21.15.43:0/1679395373 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f5c6bf1880 0x55f5c6bef290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.860+0000 7f3f57e00640 1 -- 172.21.15.43:0/1679395373 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5c692a7d0 con 0x55f5c6be8380 2024-03-28T22:39:14.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.861+0000 7f3f57e00640 1 --2- 172.21.15.43:0/1679395373 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f5c6be8380 0x55f5c6bee810 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f3f48003fd0 tx=0x7f3f480b8640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a29f6428927a066 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:14.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.861+0000 7f3f41600640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:39:14.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.861+0000 7f3f41600640 1 -- 172.21.15.43:0/1679395373 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4804b040 con 0x55f5c6be8380 2024-03-28T22:39:14.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.861+0000 7f3f41600640 1 -- 172.21.15.43:0/1679395373 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f3f480ed820 con 0x55f5c6be8380 2024-03-28T22:39:14.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.861+0000 7f3f617e5740 1 -- 172.21.15.43:0/1679395373 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f440044b0 con 0x55f5c6be8380 2024-03-28T22:39:14.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.861+0000 7f3f41600640 1 -- 172.21.15.43:0/1679395373 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3f480bc360 con 0x55f5c6be8380 2024-03-28T22:39:14.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.861+0000 7f3f617e5740 10 client.26441 fetch_fsmap learned FSMap version 29 2024-03-28T22:39:14.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.861+0000 7f3f617e5740 1 -- 172.21.15.43:0/1679395373 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f5c6c00870 con 0x55f5c6be8380 2024-03-28T22:39:14.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.861+0000 7f3f41600640 1 -- 172.21.15.43:0/1679395373 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f440044b0 con 0x55f5c6be8380 2024-03-28T22:39:14.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.862+0000 7f3f41600640 1 -- 172.21.15.43:0/1679395373 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f3f480d5210 con 0x55f5c6be8380 2024-03-28T22:39:14.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.862+0000 7f3f617e5740 10 client.26441 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:39:14.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.862+0000 7f3f617e5740 10 client.26441 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:39:14.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.862+0000 7f3f617e5740 20 client.26441 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:14.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.862+0000 7f3f617e5740 1 --2- 172.21.15.43:0/1679395373 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f5c6c01c80 0x55f5c6c040d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.862+0000 7f3f617e5740 4 client.26441 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:39:14.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.862+0000 7f3f617e5740 1 -- 172.21.15.43:0/1679395373 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f5c6af5680 con 0x55f5c6c01c80 2024-03-28T22:39:14.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.862+0000 7f3f57400640 1 --2- 172.21.15.43:0/1679395373 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f5c6c01c80 0x55f5c6c040d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.863+0000 7f3f57400640 1 --2- 172.21.15.43:0/1679395373 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f5c6c01c80 0x55f5c6c040d0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d64ce0d7495ce63 server_cookie=1498ff1b3ec5273b in_seq=0 out_seq=0 2024-03-28T22:39:14.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.863+0000 7f3f41600640 10 client.26441 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:14.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.864+0000 7f3f41600640 1 -- 172.21.15.43:0/1679395373 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f5c6af5680 con 0x55f5c6c01c80 2024-03-28T22:39:14.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.864+0000 7f3f41600640 10 client.26441 handle_command_reply: tid=0 2024-03-28T22:39:14.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.864+0000 7f3f617e5740 1 client.26441 shutdown 2024-03-28T22:39:14.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.864+0000 7f3f617e5740 1 -- 172.21.15.43:0/1679395373 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f5c6be8380 msgr2=0x55f5c6bee810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.864+0000 7f3f617e5740 1 --2- 172.21.15.43:0/1679395373 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f5c6be8380 0x55f5c6bee810 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f3f48003fd0 tx=0x7f3f480b8640 comp rx=0 tx=0).stop 2024-03-28T22:39:14.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.865+0000 7f3f617e5740 1 -- 172.21.15.43:0/1679395373 shutdown_connections 2024-03-28T22:39:14.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.865+0000 7f3f617e5740 1 --2- 172.21.15.43:0/1679395373 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f5c6bf1880 0x55f5c6bef290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.865+0000 7f3f617e5740 1 --2- 172.21.15.43:0/1679395373 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f5c6be8ca0 0x55f5c6beed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.865+0000 7f3f617e5740 1 --2- 172.21.15.43:0/1679395373 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f5c6c01c80 0x55f5c6c040d0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.865+0000 7f3f617e5740 1 --2- 172.21.15.43:0/1679395373 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f5c6be8380 0x55f5c6bee810 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.865+0000 7f3f41600640 0 client.26441 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:14.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.865+0000 7f3f617e5740 1 -- 172.21.15.43:0/1679395373 >> 172.21.15.43:0/1679395373 conn(0x55f5c6be2ab0 msgr2=0x55f5c6be4800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:14.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.865+0000 7f3f617e5740 1 -- 172.21.15.43:0/1679395373 shutdown_connections 2024-03-28T22:39:14.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.865+0000 7f3f617e5740 1 -- 172.21.15.43:0/1679395373 wait complete. 2024-03-28T22:39:14.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.865+0000 7f3f617e5740 20 client.26441 trim_cache size 0 max 16384 2024-03-28T22:39:14.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.882+0000 7f3f617e5740 1 Processor -- start 2024-03-28T22:39:14.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.883+0000 7f3f617e5740 1 -- start start 2024-03-28T22:39:14.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.883+0000 7f3f617e5740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f5c6be8380 0x55f5c6bfd410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.883+0000 7f3f617e5740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f5c6be8ca0 0x55f5c6bef0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.883+0000 7f3f617e5740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f5c6bf1880 0x55f5c6c099a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.883+0000 7f3f617e5740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3f581b5100 con 0x55f5c6bf1880 2024-03-28T22:39:14.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.883+0000 7f3f617e5740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f3f581b6c40 con 0x55f5c6be8380 2024-03-28T22:39:14.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.883+0000 7f3f617e5740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f3f581b57d0 con 0x55f5c6be8ca0 2024-03-28T22:39:14.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.883+0000 7f3f57e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f5c6be8380 0x55f5c6bfd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.883+0000 7f3f57400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f5c6be8ca0 0x55f5c6bef0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.883+0000 7f3f5cc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f5c6bf1880 0x55f5c6c099a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.883+0000 7f3f57400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f5c6be8ca0 0x55f5c6bef0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:54710/0 (socket says 172.21.15.43:54710) 2024-03-28T22:39:14.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.883+0000 7f3f57400640 1 -- 172.21.15.43:0/2012725106 learned_addr learned my addr 172.21.15.43:0/2012725106 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:14.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.883+0000 7f3f57e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f5c6be8380 0x55f5c6bfd410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:57022/0 (socket says 172.21.15.43:57022) 2024-03-28T22:39:14.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.884+0000 7f3f57400640 1 -- 172.21.15.43:0/2012725106 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f5c6be8380 msgr2=0x55f5c6bfd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:14.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.884+0000 7f3f57400640 1 --2- 172.21.15.43:0/2012725106 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f5c6be8380 0x55f5c6bfd410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.884+0000 7f3f57400640 1 -- 172.21.15.43:0/2012725106 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f5c6bf1880 msgr2=0x55f5c6c099a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:14.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.884+0000 7f3f57400640 1 --2- 172.21.15.43:0/2012725106 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f5c6bf1880 0x55f5c6c099a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.884+0000 7f3f57400640 1 -- 172.21.15.43:0/2012725106 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5c6be95e0 con 0x55f5c6be8ca0 2024-03-28T22:39:14.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.884+0000 7f3f57e00640 1 --2- 172.21.15.43:0/2012725106 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f5c6be8380 0x55f5c6bfd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:14.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.884+0000 7f3f57400640 1 --2- 172.21.15.43:0/2012725106 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f5c6be8ca0 0x55f5c6bef0b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3f44014040 tx=0x7f3f44005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92aab5b240553f16 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:14.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.884+0000 7f3f43e00640 1 -- 172.21.15.43:0/2012725106 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f44033070 con 0x55f5c6be8ca0 2024-03-28T22:39:14.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.884+0000 7f3f43e00640 1 -- 172.21.15.43:0/2012725106 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f44009b30 con 0x55f5c6be8ca0 2024-03-28T22:39:14.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.884+0000 7f3f43e00640 1 -- 172.21.15.43:0/2012725106 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4402e790 con 0x55f5c6be8ca0 2024-03-28T22:39:14.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.885+0000 7f3f617e5740 1 -- 172.21.15.43:0/2012725106 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f5c6be8ca0 msgr2=0x55f5c6bef0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.885+0000 7f3f617e5740 1 --2- 172.21.15.43:0/2012725106 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f5c6be8ca0 0x55f5c6bef0b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3f44014040 tx=0x7f3f44005f90 comp rx=0 tx=0).stop 2024-03-28T22:39:14.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.885+0000 7f3f617e5740 1 -- 172.21.15.43:0/2012725106 shutdown_connections 2024-03-28T22:39:14.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.885+0000 7f3f617e5740 1 --2- 172.21.15.43:0/2012725106 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f5c6bf1880 0x55f5c6c099a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.885+0000 7f3f617e5740 1 --2- 172.21.15.43:0/2012725106 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f5c6be8ca0 0x55f5c6bef0b0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.885+0000 7f3f617e5740 1 --2- 172.21.15.43:0/2012725106 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f5c6be8380 0x55f5c6bfd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.885+0000 7f3f617e5740 1 -- 172.21.15.43:0/2012725106 >> 172.21.15.43:0/2012725106 conn(0x55f5c6be2ab0 msgr2=0x55f5c6be12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:14.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.885+0000 7f3f617e5740 1 -- 172.21.15.43:0/2012725106 shutdown_connections 2024-03-28T22:39:14.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.885+0000 7f3f617e5740 1 -- 172.21.15.43:0/2012725106 wait complete. 2024-03-28T22:39:14.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.886+0000 7f3f617e5740 1 -- start start 2024-03-28T22:39:14.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.886+0000 7f3f617e5740 1 Processor -- start 2024-03-28T22:39:14.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.886+0000 7f3f617e5740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f5c6be8380 0x55f5c6bf9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.886+0000 7f3f617e5740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f5c6be8ca0 0x55f5c6bf5000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.886+0000 7f3f617e5740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f5c6bf1880 0x55f5c6bf5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.886+0000 7f3f617e5740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3f581b6570 con 0x55f5c6bf1880 2024-03-28T22:39:14.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.886+0000 7f3f617e5740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f3f581b5ea0 con 0x55f5c6be8380 2024-03-28T22:39:14.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.886+0000 7f3f617e5740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f3f581b30e0 con 0x55f5c6be8ca0 2024-03-28T22:39:14.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.886+0000 7f3f617e5740 10 client.0 _ll_register_callbacks cb 0x7f3f48009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:14.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.887+0000 7f3f57e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f5c6be8380 0x55f5c6bf9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.887+0000 7f3f57400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f5c6be8ca0 0x55f5c6bf5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.887+0000 7f3f57e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f5c6be8380 0x55f5c6bf9bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:57034/0 (socket says 172.21.15.43:57034) 2024-03-28T22:39:14.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.887+0000 7f3f57e00640 1 -- 172.21.15.43:0/4074858519 learned_addr learned my addr 172.21.15.43:0/4074858519 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:14.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.887+0000 7f3f5cc00640 1 --2- 172.21.15.43:0/4074858519 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f5c6bf1880 0x55f5c6bf5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.887+0000 7f3f57e00640 1 -- 172.21.15.43:0/4074858519 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f5c6be8ca0 msgr2=0x55f5c6bf5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:14.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.887+0000 7f3f57e00640 1 --2- 172.21.15.43:0/4074858519 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f5c6be8ca0 0x55f5c6bf5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.887+0000 7f3f57e00640 1 -- 172.21.15.43:0/4074858519 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f5c6bf1880 msgr2=0x55f5c6bf5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:14.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.887+0000 7f3f57e00640 1 --2- 172.21.15.43:0/4074858519 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f5c6bf1880 0x55f5c6bf5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.887+0000 7f3f57e00640 1 -- 172.21.15.43:0/4074858519 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5c6bf5c30 con 0x55f5c6be8380 2024-03-28T22:39:14.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.887+0000 7f3f5cc00640 1 --2- 172.21.15.43:0/4074858519 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f5c6bf1880 0x55f5c6bf5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:39:14.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.887+0000 7f3f57e00640 1 --2- 172.21.15.43:0/4074858519 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f5c6be8380 0x55f5c6bf9bd0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f3f48005940 tx=0x7f3f480bcce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be4710d0de289fc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:14.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.887+0000 7f3f41600640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:39:14.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.887+0000 7f3f41600640 1 -- 172.21.15.43:0/4074858519 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4804b040 con 0x55f5c6be8380 2024-03-28T22:39:14.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.888+0000 7f3f617e5740 1 -- 172.21.15.43:0/4074858519 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f44009b30 con 0x55f5c6be8380 2024-03-28T22:39:14.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.888+0000 7f3f41600640 1 -- 172.21.15.43:0/4074858519 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f3f480b9070 con 0x55f5c6be8380 2024-03-28T22:39:14.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.888+0000 7f3f41600640 1 -- 172.21.15.43:0/4074858519 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3f480c50f0 con 0x55f5c6be8380 2024-03-28T22:39:14.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.888+0000 7f3f617e5740 10 client.26447 fetch_fsmap learned FSMap version 29 2024-03-28T22:39:14.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.888+0000 7f3f617e5740 1 -- 172.21.15.43:0/4074858519 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f5c6c67ed0 con 0x55f5c6be8380 2024-03-28T22:39:14.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.888+0000 7f3f41600640 1 -- 172.21.15.43:0/4074858519 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f44009b30 con 0x55f5c6be8380 2024-03-28T22:39:14.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.888+0000 7f3f41600640 1 -- 172.21.15.43:0/4074858519 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f3f480f3b50 con 0x55f5c6be8380 2024-03-28T22:39:14.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.888+0000 7f3f617e5740 10 client.26447 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:39:14.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.888+0000 7f3f617e5740 10 client.26447 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:39:14.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.888+0000 7f3f617e5740 20 client.26447 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:14.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.888+0000 7f3f617e5740 1 --2- 172.21.15.43:0/4074858519 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f5c6c683a0 0x55f5c6c6a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:14.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.888+0000 7f3f617e5740 4 client.26447 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:39:14.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.888+0000 7f3f617e5740 1 -- 172.21.15.43:0/4074858519 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f5c6bea410 con 0x55f5c6c683a0 2024-03-28T22:39:14.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.889+0000 7f3f57400640 1 --2- 172.21.15.43:0/4074858519 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f5c6c683a0 0x55f5c6c6a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:14.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.889+0000 7f3f57400640 1 --2- 172.21.15.43:0/4074858519 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f5c6c683a0 0x55f5c6c6a790 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe7c249896a0b667 server_cookie=6dab5c9ae3edce8b in_seq=0 out_seq=0 2024-03-28T22:39:14.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.889+0000 7f3f41600640 10 client.26447 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:14.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.889+0000 7f3f41600640 1 -- 172.21.15.43:0/4074858519 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f5c6bea410 con 0x55f5c6c683a0 2024-03-28T22:39:14.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.889+0000 7f3f41600640 10 client.26447 handle_command_reply: tid=0 2024-03-28T22:39:14.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.889+0000 7f3f617e5740 1 client.26447 shutdown 2024-03-28T22:39:14.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.889+0000 7f3f617e5740 1 -- 172.21.15.43:0/4074858519 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f5c6be8380 msgr2=0x55f5c6bf9bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.889+0000 7f3f617e5740 1 --2- 172.21.15.43:0/4074858519 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f5c6be8380 0x55f5c6bf9bd0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f3f48005940 tx=0x7f3f480bcce0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.889+0000 7f3f617e5740 1 -- 172.21.15.43:0/4074858519 shutdown_connections 2024-03-28T22:39:14.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.890+0000 7f3f617e5740 1 --2- 172.21.15.43:0/4074858519 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f5c6bf1880 0x55f5c6bf5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.890+0000 7f3f617e5740 1 --2- 172.21.15.43:0/4074858519 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f5c6be8ca0 0x55f5c6bf5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.890+0000 7f3f617e5740 1 --2- 172.21.15.43:0/4074858519 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f5c6c683a0 0x55f5c6c6a790 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.890+0000 7f3f617e5740 1 --2- 172.21.15.43:0/4074858519 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f5c6be8380 0x55f5c6bf9bd0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.890+0000 7f3f617e5740 1 -- 172.21.15.43:0/4074858519 >> 172.21.15.43:0/4074858519 conn(0x55f5c6be2ab0 msgr2=0x55f5c6be1b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:14.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.890+0000 7f3f41600640 0 client.26447 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:14.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.890+0000 7f3f617e5740 1 -- 172.21.15.43:0/4074858519 shutdown_connections 2024-03-28T22:39:14.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.890+0000 7f3f617e5740 1 -- 172.21.15.43:0/4074858519 wait complete. 2024-03-28T22:39:14.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.890+0000 7f3f617e5740 20 client.26447 trim_cache size 0 max 16384 2024-03-28T22:39:14.892 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:39:14.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.890+0000 7f3f5ec00640 1 -- 172.21.15.43:0/398134098 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f3f3c0a1ef0 msgr2=0x7f3f3c0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.891+0000 7f3f5ec00640 1 --2- 172.21.15.43:0/398134098 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f3f3c0a1ef0 0x7f3f3c0a43b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3f50009fd0 tx=0x7f3f50009500 comp rx=0 tx=0).stop 2024-03-28T22:39:14.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.891+0000 7f3f5ec00640 1 -- 172.21.15.43:0/398134098 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3f58107f60 msgr2=0x7f3f5819e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:14.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.891+0000 7f3f5ec00640 1 --2- 172.21.15.43:0/398134098 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3f58107f60 0x7f3f5819e630 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f3f48006010 tx=0x7f3f4802cb40 comp rx=0 tx=0).stop 2024-03-28T22:39:14.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.891+0000 7f3f5ec00640 1 -- 172.21.15.43:0/398134098 shutdown_connections 2024-03-28T22:39:14.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.891+0000 7f3f5ec00640 1 --2- 172.21.15.43:0/398134098 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f3f3c0a1ef0 0x7f3f3c0a43b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.891+0000 7f3f5ec00640 1 --2- 172.21.15.43:0/398134098 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3f581a2ea0 0x7f3f581a3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.891+0000 7f3f5ec00640 1 --2- 172.21.15.43:0/398134098 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3f5810d030 0x7f3f5819eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.891+0000 7f3f5ec00640 1 --2- 172.21.15.43:0/398134098 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3f58107f60 0x7f3f5819e630 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:14.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.891+0000 7f3f5ec00640 1 -- 172.21.15.43:0/398134098 >> 172.21.15.43:0/398134098 conn(0x7f3f58075c90 msgr2=0x7f3f58076fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:14.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.891+0000 7f3f5ec00640 1 -- 172.21.15.43:0/398134098 shutdown_connections 2024-03-28T22:39:14.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:14.891+0000 7f3f5ec00640 1 -- 172.21.15.43:0/398134098 wait complete. 2024-03-28T22:39:14.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:14 smithi172 ceph-mon[36566]: pgmap v64: 129 pgs: 129 active+clean; 41 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.1 MiB/s wr, 395 op/s 2024-03-28T22:39:14.948 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:14 smithi082 ceph-mon[36471]: pgmap v64: 129 pgs: 129 active+clean; 41 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.1 MiB/s wr, 395 op/s 2024-03-28T22:39:15.343 INFO:tasks.workunit.client.0.smithi043.stdout: 1 75093 9.19 MB/sec warmup 46 sec latency 9.005 ms 2024-03-28T22:39:15.907 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:39:15.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:39:15.907 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:39:16.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.111+0000 7f506ac00640 1 -- 172.21.15.43:0/2301214823 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5064073b20 msgr2=0x7f5064073f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:16.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.111+0000 7f506ac00640 1 --2- 172.21.15.43:0/2301214823 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5064073b20 0x7f5064073f00 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f5058009900 tx=0x7f505802bcd0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.111+0000 7f506ac00640 1 -- 172.21.15.43:0/2301214823 shutdown_connections 2024-03-28T22:39:16.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.111+0000 7f506ac00640 1 --2- 172.21.15.43:0/2301214823 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f506410de60 0x7f5064110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.111+0000 7f506ac00640 1 --2- 172.21.15.43:0/2301214823 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5064074440 0x7f506410d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.111+0000 7f506ac00640 1 --2- 172.21.15.43:0/2301214823 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5064073b20 0x7f5064073f00 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.111+0000 7f506ac00640 1 -- 172.21.15.43:0/2301214823 >> 172.21.15.43:0/2301214823 conn(0x7f50640fd0b0 msgr2=0x7f50640ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:16.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.111+0000 7f506ac00640 1 -- 172.21.15.43:0/2301214823 shutdown_connections 2024-03-28T22:39:16.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.111+0000 7f506ac00640 1 -- 172.21.15.43:0/2301214823 wait complete. 2024-03-28T22:39:16.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.112+0000 7f506ac00640 1 Processor -- start 2024-03-28T22:39:16.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.113+0000 7f506ac00640 1 -- start start 2024-03-28T22:39:16.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.113+0000 7f506ac00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5064073b20 0x7f506410d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.113+0000 7f506ac00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5064074440 0x7f506410a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.113+0000 7f506ac00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f506410de60 0x7f506410b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.113+0000 7f506ac00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5064113bc0 con 0x7f5064074440 2024-03-28T22:39:16.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.113+0000 7f506ac00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5064113a40 con 0x7f5064073b20 2024-03-28T22:39:16.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.113+0000 7f506ac00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f5064113d40 con 0x7f506410de60 2024-03-28T22:39:16.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.113+0000 7f5069800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5064073b20 0x7f506410d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.113+0000 7f5069800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5064073b20 0x7f506410d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:57044/0 (socket says 172.21.15.43:57044) 2024-03-28T22:39:16.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.113+0000 7f5069800640 1 -- 172.21.15.43:0/2709460192 learned_addr learned my addr 172.21.15.43:0/2709460192 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:16.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.113+0000 7f506a200640 1 --2- 172.21.15.43:0/2709460192 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f506410de60 0x7f506410b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.113+0000 7f5068e00640 1 --2- 172.21.15.43:0/2709460192 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5064074440 0x7f506410a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.114+0000 7f5069800640 1 -- 172.21.15.43:0/2709460192 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f506410de60 msgr2=0x7f506410b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:16.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.114+0000 7f5069800640 1 --2- 172.21.15.43:0/2709460192 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f506410de60 0x7f506410b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.114+0000 7f5069800640 1 -- 172.21.15.43:0/2709460192 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5064074440 msgr2=0x7f506410a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:16.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.114+0000 7f5069800640 1 --2- 172.21.15.43:0/2709460192 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5064074440 0x7f506410a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.114+0000 7f5069800640 1 -- 172.21.15.43:0/2709460192 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50641aa900 con 0x7f5064073b20 2024-03-28T22:39:16.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.114+0000 7f5069800640 1 --2- 172.21.15.43:0/2709460192 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5064073b20 0x7f506410d8f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f5058006010 tx=0x7f505802cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:16.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.114+0000 7f5062000640 1 -- 172.21.15.43:0/2709460192 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5058041070 con 0x7f5064073b20 2024-03-28T22:39:16.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.114+0000 7f5062000640 1 -- 172.21.15.43:0/2709460192 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5058004450 con 0x7f5064073b20 2024-03-28T22:39:16.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.114+0000 7f506ac00640 1 -- 172.21.15.43:0/2709460192 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50641aab30 con 0x7f5064073b20 2024-03-28T22:39:16.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.115+0000 7f5062000640 1 -- 172.21.15.43:0/2709460192 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5058003df0 con 0x7f5064073b20 2024-03-28T22:39:16.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.115+0000 7f506ac00640 1 -- 172.21.15.43:0/2709460192 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50641ab080 con 0x7f5064073b20 2024-03-28T22:39:16.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.119+0000 7f506edff740 1 Processor -- start 2024-03-28T22:39:16.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.119+0000 7f506edff740 1 -- start start 2024-03-28T22:39:16.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.120+0000 7f506edff740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55763c72f170 0x55763c72f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.120+0000 7f5062000640 1 -- 172.21.15.43:0/2709460192 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f5058044070 con 0x7f5064073b20 2024-03-28T22:39:16.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.120+0000 7f506edff740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55763c72fa90 0x55763c738130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.120+0000 7f506edff740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55763c738670 0x55763c73aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.120+0000 7f506edff740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55763c692c40 con 0x55763c72fa90 2024-03-28T22:39:16.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.120+0000 7f506edff740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55763c63bc40 con 0x55763c738670 2024-03-28T22:39:16.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.120+0000 7f506edff740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55763c68ece0 con 0x55763c72f170 2024-03-28T22:39:16.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.120+0000 7f506a200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55763c72f170 0x55763c72f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.120+0000 7f506a200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55763c72f170 0x55763c72f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:54740/0 (socket says 172.21.15.43:54740) 2024-03-28T22:39:16.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.120+0000 7f506a200640 1 -- 172.21.15.43:0/2379286953 learned_addr learned my addr 172.21.15.43:0/2379286953 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:16.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.120+0000 7f5068e00640 1 --2- 172.21.15.43:0/2379286953 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55763c738670 0x55763c73aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.120+0000 7f5069800640 1 --2- 172.21.15.43:0/2379286953 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55763c72fa90 0x55763c738130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.120+0000 7f5068e00640 1 -- 172.21.15.43:0/2379286953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55763c72f170 msgr2=0x55763c72f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:16.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.120+0000 7f5068e00640 1 --2- 172.21.15.43:0/2379286953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55763c72f170 0x55763c72f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.121+0000 7f5068e00640 1 -- 172.21.15.43:0/2379286953 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55763c72fa90 msgr2=0x55763c738130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:16.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.121+0000 7f5068e00640 1 --2- 172.21.15.43:0/2379286953 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55763c72fa90 0x55763c738130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.121+0000 7f5068e00640 1 -- 172.21.15.43:0/2379286953 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55763c46f7e0 con 0x55763c738670 2024-03-28T22:39:16.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.121+0000 7f5068e00640 1 --2- 172.21.15.43:0/2379286953 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55763c738670 0x55763c73aa60 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f506410de30 tx=0x7f5050011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a98190e04cacba server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:16.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.121+0000 7f5056a00640 1 -- 172.21.15.43:0/2379286953 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5050004050 con 0x55763c738670 2024-03-28T22:39:16.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.121+0000 7f5056a00640 1 -- 172.21.15.43:0/2379286953 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5050005500 con 0x55763c738670 2024-03-28T22:39:16.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.121+0000 7f5056a00640 1 -- 172.21.15.43:0/2379286953 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5050004d60 con 0x55763c738670 2024-03-28T22:39:16.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.121+0000 7f5062000640 1 --2- 172.21.15.43:0/2709460192 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f504c0a1ef0 0x7f504c0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.122+0000 7f5062000640 1 -- 172.21.15.43:0/2709460192 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f50580eff50 con 0x7f5064073b20 2024-03-28T22:39:16.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.122+0000 7f506a200640 1 --2- 172.21.15.43:0/2709460192 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f504c0a1ef0 0x7f504c0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.122+0000 7f506edff740 1 -- 172.21.15.43:0/2379286953 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55763c738670 msgr2=0x55763c73aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:16.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.122+0000 7f506edff740 1 --2- 172.21.15.43:0/2379286953 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55763c738670 0x55763c73aa60 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f506410de30 tx=0x7f5050011810 comp rx=0 tx=0).stop 2024-03-28T22:39:16.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.122+0000 7f506edff740 1 -- 172.21.15.43:0/2379286953 shutdown_connections 2024-03-28T22:39:16.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.122+0000 7f506edff740 1 --2- 172.21.15.43:0/2379286953 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55763c738670 0x55763c73aa60 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.122+0000 7f506edff740 1 --2- 172.21.15.43:0/2379286953 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55763c72fa90 0x55763c738130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.122+0000 7f506edff740 1 --2- 172.21.15.43:0/2379286953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55763c72f170 0x55763c72f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.122+0000 7f506edff740 1 -- 172.21.15.43:0/2379286953 >> 172.21.15.43:0/2379286953 conn(0x55763c7298a0 msgr2=0x55763c729c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:16.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.122+0000 7f506edff740 1 -- 172.21.15.43:0/2379286953 shutdown_connections 2024-03-28T22:39:16.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.122+0000 7f506a200640 1 --2- 172.21.15.43:0/2709460192 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f504c0a1ef0 0x7f504c0a43b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f505c0049d0 tx=0x7f505c009600 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:16.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.122+0000 7f506edff740 1 -- 172.21.15.43:0/2379286953 wait complete. 2024-03-28T22:39:16.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.123+0000 7f506edff740 1 -- start start 2024-03-28T22:39:16.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.123+0000 7f506edff740 1 Processor -- start 2024-03-28T22:39:16.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.124+0000 7f506edff740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55763c72f170 0x55763c735670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.124+0000 7f506edff740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55763c72fa90 0x55763c735bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.124+0000 7f506edff740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55763c738670 0x55763c7360f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.124+0000 7f506edff740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55763c73b460 con 0x55763c738670 2024-03-28T22:39:16.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.124+0000 7f506edff740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55763c631250 con 0x55763c72f170 2024-03-28T22:39:16.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.124+0000 7f506edff740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55763c73b5e0 con 0x55763c72fa90 2024-03-28T22:39:16.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.124+0000 7f506edff740 10 client.0 _ll_register_callbacks cb 0x55763c652e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:16.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.124+0000 7f5069800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55763c72f170 0x55763c735670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.124+0000 7f5068e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55763c72fa90 0x55763c735bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.124+0000 7f5069800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55763c72f170 0x55763c735670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:57060/0 (socket says 172.21.15.43:57060) 2024-03-28T22:39:16.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.124+0000 7f5069800640 1 -- 172.21.15.43:0/964623707 learned_addr learned my addr 172.21.15.43:0/964623707 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:16.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.124+0000 7f506a200640 1 --2- 172.21.15.43:0/964623707 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55763c738670 0x55763c7360f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.124+0000 7f5069800640 1 -- 172.21.15.43:0/964623707 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55763c72fa90 msgr2=0x55763c735bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:16.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.124+0000 7f5069800640 1 --2- 172.21.15.43:0/964623707 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55763c72fa90 0x55763c735bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.124+0000 7f5069800640 1 -- 172.21.15.43:0/964623707 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55763c738670 msgr2=0x55763c7360f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:16.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.124+0000 7f5069800640 1 --2- 172.21.15.43:0/964623707 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55763c738670 0x55763c7360f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.124+0000 7f5069800640 1 -- 172.21.15.43:0/964623707 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55763c4cff70 con 0x55763c72f170 2024-03-28T22:39:16.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.125+0000 7f5069800640 1 --2- 172.21.15.43:0/964623707 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55763c72f170 0x55763c735670 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f5058005940 tx=0x7f5058005b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa775cd5fb062ae9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:16.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.125+0000 7f504b400640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:39:16.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.125+0000 7f504b400640 1 -- 172.21.15.43:0/964623707 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f505804b030 con 0x55763c72f170 2024-03-28T22:39:16.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.125+0000 7f504b400640 1 -- 172.21.15.43:0/964623707 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f50580b8dd0 con 0x55763c72f170 2024-03-28T22:39:16.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.125+0000 7f506edff740 1 -- 172.21.15.43:0/964623707 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5050005500 con 0x55763c72f170 2024-03-28T22:39:16.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.125+0000 7f504b400640 1 -- 172.21.15.43:0/964623707 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f50580bc530 con 0x55763c72f170 2024-03-28T22:39:16.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.125+0000 7f506edff740 10 client.26459 fetch_fsmap learned FSMap version 29 2024-03-28T22:39:16.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.125+0000 7f506edff740 1 -- 172.21.15.43:0/964623707 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55763c367170 con 0x55763c72f170 2024-03-28T22:39:16.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.125+0000 7f504b400640 1 -- 172.21.15.43:0/964623707 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5050005500 con 0x55763c72f170 2024-03-28T22:39:16.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.126+0000 7f504b400640 1 -- 172.21.15.43:0/964623707 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f50580f20d0 con 0x55763c72f170 2024-03-28T22:39:16.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.126+0000 7f506edff740 10 client.26459 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:39:16.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.126+0000 7f506edff740 10 client.26459 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:39:16.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.126+0000 7f506edff740 20 client.26459 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:16.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.126+0000 7f506edff740 1 --2- 172.21.15.43:0/964623707 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55763c748930 0x55763c74adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.126+0000 7f506edff740 4 client.26459 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:39:16.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.126+0000 7f506edff740 1 -- 172.21.15.43:0/964623707 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55763c6a0180 con 0x55763c748930 2024-03-28T22:39:16.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.126+0000 7f5068e00640 1 --2- 172.21.15.43:0/964623707 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55763c748930 0x55763c74adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.126+0000 7f5068e00640 1 --2- 172.21.15.43:0/964623707 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55763c748930 0x55763c74adc0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab66153a48618dc4 server_cookie=afefb809867ad0e7 in_seq=0 out_seq=0 2024-03-28T22:39:16.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.126+0000 7f504b400640 10 client.26459 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:16.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.128+0000 7f504b400640 1 -- 172.21.15.43:0/964623707 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55763c6a0180 con 0x55763c748930 2024-03-28T22:39:16.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.128+0000 7f504b400640 10 client.26459 handle_command_reply: tid=0 2024-03-28T22:39:16.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.128+0000 7f506edff740 1 client.26459 shutdown 2024-03-28T22:39:16.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.128+0000 7f506edff740 1 -- 172.21.15.43:0/964623707 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55763c72f170 msgr2=0x55763c735670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:16.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.128+0000 7f506edff740 1 --2- 172.21.15.43:0/964623707 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55763c72f170 0x55763c735670 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f5058005940 tx=0x7f5058005b70 comp rx=0 tx=0).stop 2024-03-28T22:39:16.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.128+0000 7f506edff740 1 -- 172.21.15.43:0/964623707 shutdown_connections 2024-03-28T22:39:16.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.128+0000 7f506edff740 1 --2- 172.21.15.43:0/964623707 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55763c738670 0x55763c7360f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.128+0000 7f506edff740 1 --2- 172.21.15.43:0/964623707 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55763c72fa90 0x55763c735bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.128+0000 7f506edff740 1 --2- 172.21.15.43:0/964623707 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55763c748930 0x55763c74adc0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.128+0000 7f506edff740 1 --2- 172.21.15.43:0/964623707 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55763c72f170 0x55763c735670 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.128+0000 7f506edff740 1 -- 172.21.15.43:0/964623707 >> 172.21.15.43:0/964623707 conn(0x55763c7298a0 msgr2=0x55763c72b5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:16.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.128+0000 7f504b400640 0 client.26459 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:16.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.128+0000 7f506edff740 1 -- 172.21.15.43:0/964623707 shutdown_connections 2024-03-28T22:39:16.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.128+0000 7f506edff740 1 -- 172.21.15.43:0/964623707 wait complete. 2024-03-28T22:39:16.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.128+0000 7f506edff740 20 client.26459 trim_cache size 0 max 16384 2024-03-28T22:39:16.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.149+0000 7f506edff740 1 Processor -- start 2024-03-28T22:39:16.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.149+0000 7f506edff740 1 -- start start 2024-03-28T22:39:16.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.149+0000 7f506edff740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55763c72f170 0x55763c737da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.149+0000 7f506edff740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55763c72fa90 0x55763c744200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.149+0000 7f506edff740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55763c738670 0x55763c728780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.149+0000 7f506edff740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f50641b5020 con 0x55763c72fa90 2024-03-28T22:39:16.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.149+0000 7f506edff740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f50641b6b60 con 0x55763c738670 2024-03-28T22:39:16.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.149+0000 7f506edff740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f50641b56f0 con 0x55763c72f170 2024-03-28T22:39:16.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.149+0000 7f5068e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55763c72fa90 0x55763c744200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.149+0000 7f5069800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55763c72f170 0x55763c737da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.149+0000 7f506a200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55763c738670 0x55763c728780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.149+0000 7f5069800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55763c72f170 0x55763c737da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:54768/0 (socket says 172.21.15.43:54768) 2024-03-28T22:39:16.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.149+0000 7f5069800640 1 -- 172.21.15.43:0/1417846347 learned_addr learned my addr 172.21.15.43:0/1417846347 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:16.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.150+0000 7f5069800640 1 -- 172.21.15.43:0/1417846347 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55763c738670 msgr2=0x55763c728780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:16.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.150+0000 7f5069800640 1 --2- 172.21.15.43:0/1417846347 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55763c738670 0x55763c728780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.150+0000 7f5069800640 1 -- 172.21.15.43:0/1417846347 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55763c72fa90 msgr2=0x55763c744200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:16.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.150+0000 7f5069800640 1 --2- 172.21.15.43:0/1417846347 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55763c72fa90 0x55763c744200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.150+0000 7f5069800640 1 -- 172.21.15.43:0/1417846347 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55763c73aa60 con 0x55763c72f170 2024-03-28T22:39:16.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.150+0000 7f506a200640 1 --2- 172.21.15.43:0/1417846347 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55763c738670 0x55763c728780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:16.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.150+0000 7f5068e00640 1 --2- 172.21.15.43:0/1417846347 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55763c72fa90 0x55763c744200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:16.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.150+0000 7f5069800640 1 --2- 172.21.15.43:0/1417846347 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55763c72f170 0x55763c737da0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f50580b9130 tx=0x7f50580c4ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b84ffc79b5543384 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:16.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.150+0000 7f5056000640 1 -- 172.21.15.43:0/1417846347 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f505804b020 con 0x55763c72f170 2024-03-28T22:39:16.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.150+0000 7f5056000640 1 -- 172.21.15.43:0/1417846347 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50580f3790 con 0x55763c72f170 2024-03-28T22:39:16.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.150+0000 7f5056000640 1 -- 172.21.15.43:0/1417846347 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50580bc0c0 con 0x55763c72f170 2024-03-28T22:39:16.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.151+0000 7f506edff740 1 -- 172.21.15.43:0/1417846347 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55763c72f170 msgr2=0x55763c737da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:16.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.151+0000 7f506edff740 1 --2- 172.21.15.43:0/1417846347 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55763c72f170 0x55763c737da0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f50580b9130 tx=0x7f50580c4ea0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.151+0000 7f506edff740 1 -- 172.21.15.43:0/1417846347 shutdown_connections 2024-03-28T22:39:16.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.151+0000 7f506edff740 1 --2- 172.21.15.43:0/1417846347 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55763c738670 0x55763c728780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.151+0000 7f506edff740 1 --2- 172.21.15.43:0/1417846347 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55763c72fa90 0x55763c744200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.151+0000 7f506edff740 1 --2- 172.21.15.43:0/1417846347 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55763c72f170 0x55763c737da0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.151+0000 7f506edff740 1 -- 172.21.15.43:0/1417846347 >> 172.21.15.43:0/1417846347 conn(0x55763c7298a0 msgr2=0x55763c728070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:16.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.151+0000 7f506edff740 1 -- 172.21.15.43:0/1417846347 shutdown_connections 2024-03-28T22:39:16.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.151+0000 7f506edff740 1 -- 172.21.15.43:0/1417846347 wait complete. 2024-03-28T22:39:16.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.152+0000 7f506edff740 1 -- start start 2024-03-28T22:39:16.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.152+0000 7f506edff740 1 Processor -- start 2024-03-28T22:39:16.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.152+0000 7f506edff740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55763c72f170 0x55763c740830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.152+0000 7f506edff740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55763c72fa90 0x55763c73d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.152+0000 7f506edff740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55763c738670 0x55763c73dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.152+0000 7f506edff740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f50641b6490 con 0x55763c72fa90 2024-03-28T22:39:16.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.152+0000 7f506edff740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f50641b5dc0 con 0x55763c738670 2024-03-28T22:39:16.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.152+0000 7f506edff740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f50641b3000 con 0x55763c72f170 2024-03-28T22:39:16.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.152+0000 7f506edff740 10 client.0 _ll_register_callbacks cb 0x7f5058005670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:16.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.153+0000 7f5069800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55763c72f170 0x55763c740830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.153+0000 7f506a200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55763c738670 0x55763c73dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.153+0000 7f5069800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55763c72f170 0x55763c740830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:54780/0 (socket says 172.21.15.43:54780) 2024-03-28T22:39:16.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.153+0000 7f506a200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55763c738670 0x55763c73dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:57070/0 (socket says 172.21.15.43:57070) 2024-03-28T22:39:16.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.153+0000 7f5069800640 1 -- 172.21.15.43:0/578771274 learned_addr learned my addr 172.21.15.43:0/578771274 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:16.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.153+0000 7f5068e00640 1 --2- 172.21.15.43:0/578771274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55763c72fa90 0x55763c73d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.153+0000 7f5069800640 1 -- 172.21.15.43:0/578771274 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55763c738670 msgr2=0x55763c73dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:16.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.153+0000 7f5069800640 1 --2- 172.21.15.43:0/578771274 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55763c738670 0x55763c73dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.153+0000 7f5069800640 1 -- 172.21.15.43:0/578771274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55763c72fa90 msgr2=0x55763c73d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:16.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.153+0000 7f5069800640 1 --2- 172.21.15.43:0/578771274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55763c72fa90 0x55763c73d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.153+0000 7f5069800640 1 -- 172.21.15.43:0/578771274 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55763c740f70 con 0x55763c72f170 2024-03-28T22:39:16.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.153+0000 7f506a200640 1 --2- 172.21.15.43:0/578771274 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55763c738670 0x55763c73dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:16.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.153+0000 7f5068e00640 1 --2- 172.21.15.43:0/578771274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55763c72fa90 0x55763c73d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:39:16.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.153+0000 7f5069800640 1 --2- 172.21.15.43:0/578771274 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55763c72f170 0x55763c740830 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5058005720 tx=0x7f50580b83c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d127e25dde64875 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:16.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.153+0000 7f504b400640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:39:16.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.153+0000 7f504b400640 1 -- 172.21.15.43:0/578771274 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f505804b030 con 0x55763c72f170 2024-03-28T22:39:16.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.154+0000 7f504b400640 1 -- 172.21.15.43:0/578771274 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f50580c49c0 con 0x55763c72f170 2024-03-28T22:39:16.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.154+0000 7f506edff740 1 -- 172.21.15.43:0/578771274 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50580f3790 con 0x55763c72f170 2024-03-28T22:39:16.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.154+0000 7f504b400640 1 -- 172.21.15.43:0/578771274 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f50580bc2a0 con 0x55763c72f170 2024-03-28T22:39:16.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.154+0000 7f506edff740 10 client.26416 fetch_fsmap learned FSMap version 29 2024-03-28T22:39:16.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.154+0000 7f506edff740 1 -- 172.21.15.43:0/578771274 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55763c740f70 con 0x55763c72f170 2024-03-28T22:39:16.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.154+0000 7f504b400640 1 -- 172.21.15.43:0/578771274 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50580f3790 con 0x55763c72f170 2024-03-28T22:39:16.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.154+0000 7f504b400640 1 -- 172.21.15.43:0/578771274 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f50580bbbb0 con 0x55763c72f170 2024-03-28T22:39:16.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.154+0000 7f506edff740 10 client.26416 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:39:16.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.154+0000 7f506edff740 10 client.26416 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:39:16.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.154+0000 7f506edff740 20 client.26416 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:16.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.154+0000 7f506edff740 1 --2- 172.21.15.43:0/578771274 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55763c7afb50 0x55763c7b1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.154+0000 7f506edff740 4 client.26416 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:39:16.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.154+0000 7f506edff740 1 -- 172.21.15.43:0/578771274 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55763c69a4d0 con 0x55763c7afb50 2024-03-28T22:39:16.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.155+0000 7f5068e00640 1 --2- 172.21.15.43:0/578771274 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55763c7afb50 0x55763c7b1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.155+0000 7f5068e00640 1 --2- 172.21.15.43:0/578771274 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55763c7afb50 0x55763c7b1f40 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aed989c85c662e00 server_cookie=a29e4d1d833b1706 in_seq=0 out_seq=0 2024-03-28T22:39:16.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.155+0000 7f504b400640 10 client.26416 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:16.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.156+0000 7f504b400640 1 -- 172.21.15.43:0/578771274 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55763c69a4d0 con 0x55763c7afb50 2024-03-28T22:39:16.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.156+0000 7f504b400640 10 client.26416 handle_command_reply: tid=0 2024-03-28T22:39:16.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.156+0000 7f506edff740 1 client.26416 shutdown 2024-03-28T22:39:16.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.156+0000 7f506edff740 1 -- 172.21.15.43:0/578771274 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55763c72f170 msgr2=0x55763c740830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:16.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.156+0000 7f506edff740 1 --2- 172.21.15.43:0/578771274 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55763c72f170 0x55763c740830 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5058005720 tx=0x7f50580b83c0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.156+0000 7f506edff740 1 -- 172.21.15.43:0/578771274 shutdown_connections 2024-03-28T22:39:16.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.157+0000 7f506edff740 1 --2- 172.21.15.43:0/578771274 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55763c7afb50 0x55763c7b1f40 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.157+0000 7f506edff740 1 --2- 172.21.15.43:0/578771274 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55763c738670 0x55763c73dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.157+0000 7f506edff740 1 --2- 172.21.15.43:0/578771274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55763c72fa90 0x55763c73d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.157+0000 7f506edff740 1 --2- 172.21.15.43:0/578771274 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55763c72f170 0x55763c740830 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.157+0000 7f504b400640 0 client.26416 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:16.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.157+0000 7f506edff740 1 -- 172.21.15.43:0/578771274 >> 172.21.15.43:0/578771274 conn(0x55763c7298a0 msgr2=0x55763c728900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:16.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.157+0000 7f506edff740 1 -- 172.21.15.43:0/578771274 shutdown_connections 2024-03-28T22:39:16.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.157+0000 7f506edff740 1 -- 172.21.15.43:0/578771274 wait complete. 2024-03-28T22:39:16.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.157+0000 7f506edff740 20 client.26416 trim_cache size 0 max 16384 2024-03-28T22:39:16.163 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:39:16.163 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-03-28T22:39:16.164 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "3e1a9f31-2e9b-41e6-a784-520a8ecbc007", 2024-03-28T22:39:16.164 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-03-28T22:39:16.164 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:39:16.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.158+0000 7f506ac00640 1 -- 172.21.15.43:0/2709460192 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f504c0a1ef0 msgr2=0x7f504c0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:16.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.158+0000 7f506ac00640 1 --2- 172.21.15.43:0/2709460192 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f504c0a1ef0 0x7f504c0a43b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f505c0049d0 tx=0x7f505c009600 comp rx=0 tx=0).stop 2024-03-28T22:39:16.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.158+0000 7f506ac00640 1 -- 172.21.15.43:0/2709460192 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5064073b20 msgr2=0x7f506410d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:16.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.158+0000 7f506ac00640 1 --2- 172.21.15.43:0/2709460192 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5064073b20 0x7f506410d8f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f5058006010 tx=0x7f505802cc40 comp rx=0 tx=0).stop 2024-03-28T22:39:16.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.158+0000 7f506ac00640 1 -- 172.21.15.43:0/2709460192 shutdown_connections 2024-03-28T22:39:16.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.158+0000 7f506ac00640 1 --2- 172.21.15.43:0/2709460192 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f504c0a1ef0 0x7f504c0a43b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.158+0000 7f506ac00640 1 --2- 172.21.15.43:0/2709460192 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f506410de60 0x7f506410b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.158+0000 7f506ac00640 1 --2- 172.21.15.43:0/2709460192 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5064074440 0x7f506410a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.158+0000 7f506ac00640 1 --2- 172.21.15.43:0/2709460192 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5064073b20 0x7f506410d8f0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.158+0000 7f506ac00640 1 -- 172.21.15.43:0/2709460192 >> 172.21.15.43:0/2709460192 conn(0x7f50640fd0b0 msgr2=0x7f50640fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:16.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.158+0000 7f506ac00640 1 -- 172.21.15.43:0/2709460192 shutdown_connections 2024-03-28T22:39:16.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.158+0000 7f506ac00640 1 -- 172.21.15.43:0/2709460192 wait complete. 2024-03-28T22:39:16.175 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:39:16.175 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:39:16.343 INFO:tasks.workunit.client.0.smithi043.stdout: 1 76525 9.11 MB/sec warmup 47 sec latency 43.911 ms 2024-03-28T22:39:16.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.403+0000 7f191fa00640 1 -- 172.21.15.43:0/531392694 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1918108910 msgr2=0x7f191810c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:16.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.403+0000 7f191fa00640 1 --2- 172.21.15.43:0/531392694 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1918108910 0x7f191810c950 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f1900009900 tx=0x7f190002bc30 comp rx=0 tx=0).stop 2024-03-28T22:39:16.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.403+0000 7f191fa00640 1 -- 172.21.15.43:0/531392694 shutdown_connections 2024-03-28T22:39:16.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.403+0000 7f191fa00640 1 --2- 172.21.15.43:0/531392694 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1918108910 0x7f191810c950 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.403+0000 7f191fa00640 1 --2- 172.21.15.43:0/531392694 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f191810d030 0x7f1918114c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.403+0000 7f191fa00640 1 --2- 172.21.15.43:0/531392694 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1918107f60 0x7f1918108340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.403+0000 7f191fa00640 1 -- 172.21.15.43:0/531392694 >> 172.21.15.43:0/531392694 conn(0x7f1918075c90 msgr2=0x7f19180780b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:16.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.403+0000 7f191fa00640 1 -- 172.21.15.43:0/531392694 shutdown_connections 2024-03-28T22:39:16.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.403+0000 7f191fa00640 1 -- 172.21.15.43:0/531392694 wait complete. 2024-03-28T22:39:16.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.404+0000 7f191fa00640 1 Processor -- start 2024-03-28T22:39:16.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.405+0000 7f191fa00640 1 -- start start 2024-03-28T22:39:16.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.405+0000 7f191fa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1918107f60 0x7f191819e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.405+0000 7f191fa00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f191810d030 0x7f191819eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.405+0000 7f191fa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19181a2ec0 0x7f19181a3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.405+0000 7f191fa00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f19181180c0 con 0x7f1918107f60 2024-03-28T22:39:16.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.405+0000 7f191fa00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1918117f40 con 0x7f191810d030 2024-03-28T22:39:16.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.405+0000 7f191fa00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f1918118240 con 0x7f19181a2ec0 2024-03-28T22:39:16.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.405+0000 7f191d000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1918107f60 0x7f191819e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.405+0000 7f191da00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19181a2ec0 0x7f19181a3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.405+0000 7f1917e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f191810d030 0x7f191819eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.405+0000 7f1917e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f191810d030 0x7f191819eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:57086/0 (socket says 172.21.15.43:57086) 2024-03-28T22:39:16.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.405+0000 7f1917e00640 1 -- 172.21.15.43:0/2016675085 learned_addr learned my addr 172.21.15.43:0/2016675085 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:16.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.405+0000 7f191d000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1918107f60 0x7f191819e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53190/0 (socket says 172.21.15.43:53190) 2024-03-28T22:39:16.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.405+0000 7f191da00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19181a2ec0 0x7f19181a3370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:54788/0 (socket says 172.21.15.43:54788) 2024-03-28T22:39:16.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.406+0000 7f1917e00640 1 -- 172.21.15.43:0/2016675085 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19181a2ec0 msgr2=0x7f19181a3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:16.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.406+0000 7f1917e00640 1 --2- 172.21.15.43:0/2016675085 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19181a2ec0 0x7f19181a3370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.406+0000 7f1917e00640 1 -- 172.21.15.43:0/2016675085 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1918107f60 msgr2=0x7f191819e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:16.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.406+0000 7f1917e00640 1 --2- 172.21.15.43:0/2016675085 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1918107f60 0x7f191819e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.406+0000 7f1917e00640 1 -- 172.21.15.43:0/2016675085 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19181a39b0 con 0x7f191810d030 2024-03-28T22:39:16.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.406+0000 7f191d000640 1 --2- 172.21.15.43:0/2016675085 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1918107f60 0x7f191819e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:16.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.406+0000 7f1917e00640 1 --2- 172.21.15.43:0/2016675085 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f191810d030 0x7f191819eb40 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f1900036040 tx=0x7f190002cb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:16.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.407+0000 7f1915600640 1 -- 172.21.15.43:0/2016675085 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1900041070 con 0x7f191810d030 2024-03-28T22:39:16.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.407+0000 7f1915600640 1 -- 172.21.15.43:0/2016675085 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1900004450 con 0x7f191810d030 2024-03-28T22:39:16.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.407+0000 7f1915600640 1 -- 172.21.15.43:0/2016675085 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1900003dc0 con 0x7f191810d030 2024-03-28T22:39:16.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.407+0000 7f191fa00640 1 -- 172.21.15.43:0/2016675085 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19181a3c40 con 0x7f191810d030 2024-03-28T22:39:16.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.407+0000 7f191fa00640 1 -- 172.21.15.43:0/2016675085 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19181ab460 con 0x7f191810d030 2024-03-28T22:39:16.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.410+0000 7f19225fe740 1 Processor -- start 2024-03-28T22:39:16.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.410+0000 7f1915600640 1 -- 172.21.15.43:0/2016675085 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f1900044070 con 0x7f191810d030 2024-03-28T22:39:16.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.410+0000 7f19225fe740 1 -- start start 2024-03-28T22:39:16.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.411+0000 7f19225fe740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dc5df2dbc0 0x55dc5df2c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.411+0000 7f19225fe740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dc5df2cee0 0x55dc5df2d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.411+0000 7f19225fe740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dc5df36250 0x55dc5df38640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.411+0000 7f19225fe740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55dc5de91100 con 0x55dc5df2cee0 2024-03-28T22:39:16.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.411+0000 7f19225fe740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55dc5de39ba0 con 0x55dc5df36250 2024-03-28T22:39:16.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.411+0000 7f19225fe740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55dc5de8d1a0 con 0x55dc5df2dbc0 2024-03-28T22:39:16.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.411+0000 7f191da00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dc5df2dbc0 0x55dc5df2c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.411+0000 7f1917e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dc5df36250 0x55dc5df38640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.411+0000 7f191d000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dc5df2cee0 0x55dc5df2d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.411+0000 7f191da00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dc5df2dbc0 0x55dc5df2c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:54792/0 (socket says 172.21.15.43:54792) 2024-03-28T22:39:16.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.411+0000 7f191da00640 1 -- 172.21.15.43:0/524035908 learned_addr learned my addr 172.21.15.43:0/524035908 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:16.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.412+0000 7f191da00640 1 -- 172.21.15.43:0/524035908 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dc5df36250 msgr2=0x55dc5df38640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:16.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.412+0000 7f191da00640 1 --2- 172.21.15.43:0/524035908 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dc5df36250 0x55dc5df38640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.412+0000 7f191da00640 1 -- 172.21.15.43:0/524035908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dc5df2cee0 msgr2=0x55dc5df2d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:16.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.412+0000 7f191da00640 1 --2- 172.21.15.43:0/524035908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dc5df2cee0 0x55dc5df2d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.412+0000 7f191da00640 1 -- 172.21.15.43:0/524035908 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc5de6a1a0 con 0x55dc5df2dbc0 2024-03-28T22:39:16.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.412+0000 7f191d000640 1 --2- 172.21.15.43:0/524035908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dc5df2cee0 0x55dc5df2d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:16.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.412+0000 7f191da00640 1 --2- 172.21.15.43:0/524035908 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dc5df2dbc0 0x55dc5df2c910 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1910009f50 tx=0x7f19100118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8388e3d7922b7542 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:16.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.412+0000 7f190e000640 1 -- 172.21.15.43:0/524035908 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1910004d20 con 0x55dc5df2dbc0 2024-03-28T22:39:16.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.412+0000 7f1915600640 1 --2- 172.21.15.43:0/2016675085 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f18f80a1f70 0x7f18f80a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.412+0000 7f190e000640 1 -- 172.21.15.43:0/524035908 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1910004050 con 0x55dc5df2dbc0 2024-03-28T22:39:16.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.412+0000 7f190e000640 1 -- 172.21.15.43:0/524035908 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1910005260 con 0x55dc5df2dbc0 2024-03-28T22:39:16.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.413+0000 7f1915600640 1 -- 172.21.15.43:0/2016675085 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f19000efec0 con 0x7f191810d030 2024-03-28T22:39:16.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.413+0000 7f191d000640 1 --2- 172.21.15.43:0/2016675085 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f18f80a1f70 0x7f18f80a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.413+0000 7f19225fe740 1 -- 172.21.15.43:0/524035908 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dc5df2dbc0 msgr2=0x55dc5df2c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:16.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.413+0000 7f19225fe740 1 --2- 172.21.15.43:0/524035908 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dc5df2dbc0 0x55dc5df2c910 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1910009f50 tx=0x7f19100118d0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.413+0000 7f19225fe740 1 -- 172.21.15.43:0/524035908 shutdown_connections 2024-03-28T22:39:16.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.413+0000 7f19225fe740 1 --2- 172.21.15.43:0/524035908 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dc5df36250 0x55dc5df38640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.413+0000 7f19225fe740 1 --2- 172.21.15.43:0/524035908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dc5df2cee0 0x55dc5df2d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.413+0000 7f19225fe740 1 --2- 172.21.15.43:0/524035908 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dc5df2dbc0 0x55dc5df2c910 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.413+0000 7f19225fe740 1 -- 172.21.15.43:0/524035908 >> 172.21.15.43:0/524035908 conn(0x55dc5def3a10 msgr2=0x55dc5df27930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:16.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.413+0000 7f19225fe740 1 -- 172.21.15.43:0/524035908 shutdown_connections 2024-03-28T22:39:16.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.414+0000 7f19225fe740 1 -- 172.21.15.43:0/524035908 wait complete. 2024-03-28T22:39:16.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.414+0000 7f191d000640 1 --2- 172.21.15.43:0/2016675085 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f18f80a1f70 0x7f18f80a4430 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1908009fd0 tx=0x7f1908005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:16.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.414+0000 7f19225fe740 1 -- start start 2024-03-28T22:39:16.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.414+0000 7f19225fe740 1 Processor -- start 2024-03-28T22:39:16.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.415+0000 7f19225fe740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dc5df2cee0 0x55dc5df33230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.415+0000 7f19225fe740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dc5df2dbc0 0x55dc5df33770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.415+0000 7f19225fe740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dc5df36250 0x55dc5df33cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.415+0000 7f19225fe740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55dc5df390c0 con 0x55dc5df2cee0 2024-03-28T22:39:16.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.415+0000 7f19225fe740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55dc5de2f250 con 0x55dc5df36250 2024-03-28T22:39:16.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.415+0000 7f19225fe740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55dc5df39240 con 0x55dc5df2dbc0 2024-03-28T22:39:16.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.415+0000 7f19225fe740 10 client.0 _ll_register_callbacks cb 0x55dc5dba4be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:16.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.415+0000 7f191da00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dc5df36250 0x55dc5df33cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.415+0000 7f191d000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dc5df2cee0 0x55dc5df33230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.415+0000 7f191da00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dc5df36250 0x55dc5df33cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:57106/0 (socket says 172.21.15.43:57106) 2024-03-28T22:39:16.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.415+0000 7f191da00640 1 -- 172.21.15.43:0/608248128 learned_addr learned my addr 172.21.15.43:0/608248128 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:16.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.415+0000 7f1917e00640 1 --2- 172.21.15.43:0/608248128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dc5df2dbc0 0x55dc5df33770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.415+0000 7f191da00640 1 -- 172.21.15.43:0/608248128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dc5df2dbc0 msgr2=0x55dc5df33770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:16.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.415+0000 7f191da00640 1 --2- 172.21.15.43:0/608248128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dc5df2dbc0 0x55dc5df33770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.415+0000 7f191da00640 1 -- 172.21.15.43:0/608248128 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dc5df2cee0 msgr2=0x55dc5df33230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:16.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.415+0000 7f191da00640 1 --2- 172.21.15.43:0/608248128 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dc5df2cee0 0x55dc5df33230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.415+0000 7f191da00640 1 -- 172.21.15.43:0/608248128 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc5de4b960 con 0x55dc5df36250 2024-03-28T22:39:16.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.415+0000 7f1917e00640 1 --2- 172.21.15.43:0/608248128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dc5df2dbc0 0x55dc5df33770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:16.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.416+0000 7f191da00640 1 --2- 172.21.15.43:0/608248128 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dc5df36250 0x55dc5df33cb0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f1910009f20 tx=0x7f1910004430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10071c8d45aafb7f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:16.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.416+0000 7f1906a00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:39:16.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.416+0000 7f1906a00640 1 -- 172.21.15.43:0/608248128 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1910028030 con 0x55dc5df36250 2024-03-28T22:39:16.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.416+0000 7f1906a00640 1 -- 172.21.15.43:0/608248128 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f1910004c50 con 0x55dc5df36250 2024-03-28T22:39:16.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.416+0000 7f19225fe740 1 -- 172.21.15.43:0/608248128 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1910004050 con 0x55dc5df36250 2024-03-28T22:39:16.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.416+0000 7f1906a00640 1 -- 172.21.15.43:0/608248128 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1910009be0 con 0x55dc5df36250 2024-03-28T22:39:16.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.416+0000 7f19225fe740 10 client.26477 fetch_fsmap learned FSMap version 29 2024-03-28T22:39:16.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.416+0000 7f19225fe740 1 -- 172.21.15.43:0/608248128 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55dc5dc6f7d0 con 0x55dc5df36250 2024-03-28T22:39:16.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.417+0000 7f1906a00640 1 -- 172.21.15.43:0/608248128 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1910004050 con 0x55dc5df36250 2024-03-28T22:39:16.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.417+0000 7f1906a00640 1 -- 172.21.15.43:0/608248128 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f1910024460 con 0x55dc5df36250 2024-03-28T22:39:16.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.417+0000 7f19225fe740 10 client.26477 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:39:16.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.417+0000 7f19225fe740 10 client.26477 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:39:16.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.417+0000 7f19225fe740 20 client.26477 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:16.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.417+0000 7f19225fe740 1 --2- 172.21.15.43:0/608248128 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55dc5df466b0 0x55dc5df48aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.417+0000 7f19225fe740 4 client.26477 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:39:16.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.417+0000 7f19225fe740 1 -- 172.21.15.43:0/608248128 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dc5de9e640 con 0x55dc5df466b0 2024-03-28T22:39:16.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.417+0000 7f191d000640 1 --2- 172.21.15.43:0/608248128 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55dc5df466b0 0x55dc5df48aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.418+0000 7f191d000640 1 --2- 172.21.15.43:0/608248128 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55dc5df466b0 0x55dc5df48aa0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=472d1fb32a451259 server_cookie=a1d47725006512e2 in_seq=0 out_seq=0 2024-03-28T22:39:16.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.418+0000 7f1906a00640 10 client.26477 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:16.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.420+0000 7f1906a00640 1 -- 172.21.15.43:0/608248128 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dc5de9e640 con 0x55dc5df466b0 2024-03-28T22:39:16.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.420+0000 7f1906a00640 10 client.26477 handle_command_reply: tid=0 2024-03-28T22:39:16.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.420+0000 7f19225fe740 1 client.26477 shutdown 2024-03-28T22:39:16.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.420+0000 7f19225fe740 1 -- 172.21.15.43:0/608248128 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dc5df36250 msgr2=0x55dc5df33cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:16.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.420+0000 7f19225fe740 1 --2- 172.21.15.43:0/608248128 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dc5df36250 0x55dc5df33cb0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f1910009f20 tx=0x7f1910004430 comp rx=0 tx=0).stop 2024-03-28T22:39:16.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.420+0000 7f19225fe740 1 -- 172.21.15.43:0/608248128 shutdown_connections 2024-03-28T22:39:16.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.420+0000 7f19225fe740 1 --2- 172.21.15.43:0/608248128 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55dc5df466b0 0x55dc5df48aa0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.420+0000 7f19225fe740 1 --2- 172.21.15.43:0/608248128 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dc5df36250 0x55dc5df33cb0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.420+0000 7f19225fe740 1 --2- 172.21.15.43:0/608248128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dc5df2dbc0 0x55dc5df33770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.420+0000 7f19225fe740 1 --2- 172.21.15.43:0/608248128 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dc5df2cee0 0x55dc5df33230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.420+0000 7f1906a00640 0 client.26477 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:16.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.420+0000 7f19225fe740 1 -- 172.21.15.43:0/608248128 >> 172.21.15.43:0/608248128 conn(0x55dc5def3a10 msgr2=0x55dc5df29680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:16.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.420+0000 7f19225fe740 1 -- 172.21.15.43:0/608248128 shutdown_connections 2024-03-28T22:39:16.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.420+0000 7f19225fe740 1 -- 172.21.15.43:0/608248128 wait complete. 2024-03-28T22:39:16.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.420+0000 7f19225fe740 20 client.26477 trim_cache size 0 max 16384 2024-03-28T22:39:16.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.442+0000 7f19225fe740 1 Processor -- start 2024-03-28T22:39:16.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.442+0000 7f19225fe740 1 -- start start 2024-03-28T22:39:16.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.443+0000 7f19225fe740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dc5df2cee0 0x55dc5df41ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.443+0000 7f19225fe740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dc5df2dbc0 0x55dc5df33ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.443+0000 7f19225fe740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dc5df36250 0x55dc5df26930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.443+0000 7f19225fe740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f19181b5150 con 0x55dc5df36250 2024-03-28T22:39:16.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.443+0000 7f19225fe740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f19181b6c90 con 0x55dc5df2dbc0 2024-03-28T22:39:16.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.443+0000 7f19225fe740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f19181b5820 con 0x55dc5df2cee0 2024-03-28T22:39:16.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.443+0000 7f191d000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dc5df2cee0 0x55dc5df41ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.443+0000 7f191d000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dc5df2cee0 0x55dc5df41ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:54802/0 (socket says 172.21.15.43:54802) 2024-03-28T22:39:16.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.443+0000 7f191d000640 1 -- 172.21.15.43:0/2525253058 learned_addr learned my addr 172.21.15.43:0/2525253058 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:16.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.443+0000 7f1917e00640 1 --2- 172.21.15.43:0/2525253058 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dc5df2dbc0 0x55dc5df33ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.443+0000 7f191da00640 1 --2- 172.21.15.43:0/2525253058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dc5df36250 0x55dc5df26930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.443+0000 7f191d000640 1 -- 172.21.15.43:0/2525253058 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dc5df2dbc0 msgr2=0x55dc5df33ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:16.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.443+0000 7f191d000640 1 --2- 172.21.15.43:0/2525253058 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dc5df2dbc0 0x55dc5df33ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.443+0000 7f191d000640 1 -- 172.21.15.43:0/2525253058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dc5df36250 msgr2=0x55dc5df26930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:16.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.443+0000 7f191d000640 1 --2- 172.21.15.43:0/2525253058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dc5df36250 0x55dc5df26930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.444+0000 7f191d000640 1 -- 172.21.15.43:0/2525253058 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc5dba53f0 con 0x55dc5df2cee0 2024-03-28T22:39:16.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.444+0000 7f191da00640 1 --2- 172.21.15.43:0/2525253058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dc5df36250 0x55dc5df26930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:39:16.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.444+0000 7f191d000640 1 --2- 172.21.15.43:0/2525253058 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dc5df2cee0 0x55dc5df41ed0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x55dc5df49130 tx=0x7f190801a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e214961662c45e65 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:16.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.444+0000 7f190d600640 1 -- 172.21.15.43:0/2525253058 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f190802bb90 con 0x55dc5df2cee0 2024-03-28T22:39:16.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.444+0000 7f190d600640 1 -- 172.21.15.43:0/2525253058 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1908004050 con 0x55dc5df2cee0 2024-03-28T22:39:16.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.444+0000 7f190d600640 1 -- 172.21.15.43:0/2525253058 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1908003bf0 con 0x55dc5df2cee0 2024-03-28T22:39:16.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.445+0000 7f19225fe740 1 -- 172.21.15.43:0/2525253058 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dc5df2cee0 msgr2=0x55dc5df41ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:16.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.445+0000 7f19225fe740 1 --2- 172.21.15.43:0/2525253058 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dc5df2cee0 0x55dc5df41ed0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x55dc5df49130 tx=0x7f190801a000 comp rx=0 tx=0).stop 2024-03-28T22:39:16.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.445+0000 7f19225fe740 1 -- 172.21.15.43:0/2525253058 shutdown_connections 2024-03-28T22:39:16.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.445+0000 7f19225fe740 1 --2- 172.21.15.43:0/2525253058 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dc5df2dbc0 0x55dc5df33ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.445+0000 7f19225fe740 1 --2- 172.21.15.43:0/2525253058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dc5df36250 0x55dc5df26930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.445+0000 7f19225fe740 1 --2- 172.21.15.43:0/2525253058 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dc5df2cee0 0x55dc5df41ed0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.445+0000 7f19225fe740 1 -- 172.21.15.43:0/2525253058 >> 172.21.15.43:0/2525253058 conn(0x55dc5def3a10 msgr2=0x55dc5df26220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:16.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.445+0000 7f19225fe740 1 -- 172.21.15.43:0/2525253058 shutdown_connections 2024-03-28T22:39:16.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.445+0000 7f19225fe740 1 -- 172.21.15.43:0/2525253058 wait complete. 2024-03-28T22:39:16.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.446+0000 7f19225fe740 1 -- start start 2024-03-28T22:39:16.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.446+0000 7f19225fe740 1 Processor -- start 2024-03-28T22:39:16.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.446+0000 7f19225fe740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dc5df2cee0 0x55dc5df2fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.446+0000 7f19225fe740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dc5df2dbc0 0x55dc5df30190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.446+0000 7f19225fe740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dc5df36250 0x55dc5df3d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.446+0000 7f19225fe740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f19181b65c0 con 0x55dc5df2dbc0 2024-03-28T22:39:16.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.446+0000 7f19225fe740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f19181b5ef0 con 0x55dc5df36250 2024-03-28T22:39:16.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.446+0000 7f19225fe740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f19181b3130 con 0x55dc5df2cee0 2024-03-28T22:39:16.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.446+0000 7f19225fe740 10 client.0 _ll_register_callbacks cb 0x7f1908009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:16.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.446+0000 7f191d000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dc5df2cee0 0x55dc5df2fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.446+0000 7f191da00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dc5df36250 0x55dc5df3d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.447+0000 7f191d000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dc5df2cee0 0x55dc5df2fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:54810/0 (socket says 172.21.15.43:54810) 2024-03-28T22:39:16.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.447+0000 7f191d000640 1 -- 172.21.15.43:0/2153651835 learned_addr learned my addr 172.21.15.43:0/2153651835 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:16.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.447+0000 7f1917e00640 1 --2- 172.21.15.43:0/2153651835 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dc5df2dbc0 0x55dc5df30190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.447+0000 7f191d000640 1 -- 172.21.15.43:0/2153651835 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dc5df36250 msgr2=0x55dc5df3d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:16.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.447+0000 7f191d000640 1 --2- 172.21.15.43:0/2153651835 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dc5df36250 0x55dc5df3d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.447+0000 7f191d000640 1 -- 172.21.15.43:0/2153651835 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dc5df2dbc0 msgr2=0x55dc5df30190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:16.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.447+0000 7f191d000640 1 --2- 172.21.15.43:0/2153651835 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dc5df2dbc0 0x55dc5df30190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.447+0000 7f191d000640 1 -- 172.21.15.43:0/2153651835 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc5db650f0 con 0x55dc5df2cee0 2024-03-28T22:39:16.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.447+0000 7f191da00640 1 --2- 172.21.15.43:0/2153651835 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dc5df36250 0x55dc5df3d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:16.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.447+0000 7f1917e00640 1 --2- 172.21.15.43:0/2153651835 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dc5df2dbc0 0x55dc5df30190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:39:16.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.447+0000 7f191d000640 1 --2- 172.21.15.43:0/2153651835 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dc5df2cee0 0x55dc5df2fc50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f19080125a0 tx=0x7f19080056d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=200e4cb495632ddb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:16.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.447+0000 7f1906a00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:39:16.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.448+0000 7f1906a00640 1 -- 172.21.15.43:0/2153651835 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f190802bb90 con 0x55dc5df2cee0 2024-03-28T22:39:16.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.448+0000 7f1906a00640 1 -- 172.21.15.43:0/2153651835 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f190802bd30 con 0x55dc5df2cee0 2024-03-28T22:39:16.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.448+0000 7f19225fe740 1 -- 172.21.15.43:0/2153651835 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1908004050 con 0x55dc5df2cee0 2024-03-28T22:39:16.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.448+0000 7f1906a00640 1 -- 172.21.15.43:0/2153651835 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1908004200 con 0x55dc5df2cee0 2024-03-28T22:39:16.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.448+0000 7f19225fe740 10 client.26434 fetch_fsmap learned FSMap version 29 2024-03-28T22:39:16.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.448+0000 7f19225fe740 1 -- 172.21.15.43:0/2153651835 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55dc5df3eb80 con 0x55dc5df2cee0 2024-03-28T22:39:16.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.448+0000 7f1906a00640 1 -- 172.21.15.43:0/2153651835 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1908004050 con 0x55dc5df2cee0 2024-03-28T22:39:16.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.448+0000 7f1906a00640 1 -- 172.21.15.43:0/2153651835 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f1908003c00 con 0x55dc5df2cee0 2024-03-28T22:39:16.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.448+0000 7f19225fe740 10 client.26434 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:39:16.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.448+0000 7f19225fe740 10 client.26434 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:39:16.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.449+0000 7f19225fe740 20 client.26434 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:16.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.449+0000 7f19225fe740 1 --2- 172.21.15.43:0/2153651835 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55dc5dfacb50 0x55dc5dfaef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:16.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.449+0000 7f19225fe740 4 client.26434 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:39:16.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.449+0000 7f19225fe740 1 -- 172.21.15.43:0/2153651835 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55dc5de98990 con 0x55dc5dfacb50 2024-03-28T22:39:16.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.449+0000 7f1917e00640 1 --2- 172.21.15.43:0/2153651835 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55dc5dfacb50 0x55dc5dfaef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:16.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.449+0000 7f1917e00640 1 --2- 172.21.15.43:0/2153651835 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55dc5dfacb50 0x55dc5dfaef70 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b83bcd58db60b0e5 server_cookie=58238c752b281b79 in_seq=0 out_seq=0 2024-03-28T22:39:16.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.449+0000 7f1906a00640 10 client.26434 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:16.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.450+0000 7f1906a00640 1 -- 172.21.15.43:0/2153651835 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55dc5de98990 con 0x55dc5dfacb50 2024-03-28T22:39:16.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.450+0000 7f1906a00640 10 client.26434 handle_command_reply: tid=0 2024-03-28T22:39:16.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.450+0000 7f19225fe740 1 client.26434 shutdown 2024-03-28T22:39:16.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.450+0000 7f19225fe740 1 -- 172.21.15.43:0/2153651835 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dc5df2cee0 msgr2=0x55dc5df2fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:16.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.450+0000 7f19225fe740 1 --2- 172.21.15.43:0/2153651835 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dc5df2cee0 0x55dc5df2fc50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f19080125a0 tx=0x7f19080056d0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.450+0000 7f19225fe740 1 -- 172.21.15.43:0/2153651835 shutdown_connections 2024-03-28T22:39:16.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.450+0000 7f19225fe740 1 --2- 172.21.15.43:0/2153651835 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55dc5dfacb50 0x55dc5dfaef70 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.450+0000 7f19225fe740 1 --2- 172.21.15.43:0/2153651835 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dc5df36250 0x55dc5df3d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.450+0000 7f19225fe740 1 --2- 172.21.15.43:0/2153651835 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dc5df2dbc0 0x55dc5df30190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.450+0000 7f19225fe740 1 --2- 172.21.15.43:0/2153651835 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dc5df2cee0 0x55dc5df2fc50 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.450+0000 7f1906a00640 0 client.26434 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:16.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.450+0000 7f19225fe740 1 -- 172.21.15.43:0/2153651835 >> 172.21.15.43:0/2153651835 conn(0x55dc5def3a10 msgr2=0x55dc5df26220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:16.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.450+0000 7f19225fe740 1 -- 172.21.15.43:0/2153651835 shutdown_connections 2024-03-28T22:39:16.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.450+0000 7f19225fe740 1 -- 172.21.15.43:0/2153651835 wait complete. 2024-03-28T22:39:16.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.450+0000 7f19225fe740 20 client.26434 trim_cache size 0 max 16384 2024-03-28T22:39:16.455 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:39:16.455 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:39:16.455 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-03-28T22:39:16.455 INFO:teuthology.orchestra.run.smithi043.stdout: "3e1a9f31-2e9b-41e6-a784-520a8ecbc007": { 2024-03-28T22:39:16.455 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-03-28T22:39:16.455 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3e1a9f31-2e9b-41e6-a784-520a8ecbc007", 2024-03-28T22:39:16.455 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-03-28T22:39:16.455 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:39:16.455 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:39:16.456 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:39:16.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.451+0000 7f191fa00640 1 -- 172.21.15.43:0/2016675085 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f18f80a1f70 msgr2=0x7f18f80a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:16.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.451+0000 7f191fa00640 1 --2- 172.21.15.43:0/2016675085 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f18f80a1f70 0x7f18f80a4430 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1908009fd0 tx=0x7f1908005f90 comp rx=0 tx=0).stop 2024-03-28T22:39:16.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.451+0000 7f191fa00640 1 -- 172.21.15.43:0/2016675085 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f191810d030 msgr2=0x7f191819eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:16.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.451+0000 7f191fa00640 1 --2- 172.21.15.43:0/2016675085 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f191810d030 0x7f191819eb40 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f1900036040 tx=0x7f190002cb20 comp rx=0 tx=0).stop 2024-03-28T22:39:16.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.451+0000 7f191fa00640 1 -- 172.21.15.43:0/2016675085 shutdown_connections 2024-03-28T22:39:16.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.451+0000 7f191fa00640 1 --2- 172.21.15.43:0/2016675085 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f18f80a1f70 0x7f18f80a4430 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.451+0000 7f191fa00640 1 --2- 172.21.15.43:0/2016675085 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f191810d030 0x7f191819eb40 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.451+0000 7f191fa00640 1 --2- 172.21.15.43:0/2016675085 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f19181a2ec0 0x7f19181a3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.451+0000 7f191fa00640 1 --2- 172.21.15.43:0/2016675085 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1918107f60 0x7f191819e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:16.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.451+0000 7f191fa00640 1 -- 172.21.15.43:0/2016675085 >> 172.21.15.43:0/2016675085 conn(0x7f1918075c90 msgr2=0x7f1918076fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:16.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.451+0000 7f191fa00640 1 -- 172.21.15.43:0/2016675085 shutdown_connections 2024-03-28T22:39:16.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:16.451+0000 7f191fa00640 1 -- 172.21.15.43:0/2016675085 wait complete. 2024-03-28T22:39:16.467 INFO:tasks.cephfs.filesystem:scrub status for tag:3e1a9f31-2e9b-41e6-a784-520a8ecbc007 - {'path': '/', 'tag': '3e1a9f31-2e9b-41e6-a784-520a8ecbc007', 'options': 'recursive,force'} 2024-03-28T22:39:16.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:16 smithi043 ceph-mon[31823]: pgmap v65: 129 pgs: 129 active+clean; 41 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 167 op/s 2024-03-28T22:39:16.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:16 smithi172 ceph-mon[36566]: pgmap v65: 129 pgs: 129 active+clean; 41 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 167 op/s 2024-03-28T22:39:16.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:16 smithi082 ceph-mon[36471]: pgmap v65: 129 pgs: 129 active+clean; 41 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 167 op/s 2024-03-28T22:39:17.343 INFO:tasks.workunit.client.0.smithi043.stdout: 1 78461 9.20 MB/sec warmup 48 sec latency 14.052 ms 2024-03-28T22:39:17.765 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:17 smithi043 ceph-mon[31823]: scrub queued for path: / 2024-03-28T22:39:17.765 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:17 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:39:17.765 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:17 smithi043 ceph-mon[31823]: scrub summary: active paths [/] 2024-03-28T22:39:17.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:17 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:39:17.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:17 smithi172 ceph-mon[36566]: scrub queued for path: / 2024-03-28T22:39:17.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:17 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:39:17.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:17 smithi172 ceph-mon[36566]: scrub summary: active paths [/] 2024-03-28T22:39:17.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:17 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:39:17.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:17 smithi082 ceph-mon[36471]: scrub queued for path: / 2024-03-28T22:39:17.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:17 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:39:17.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:17 smithi082 ceph-mon[36471]: scrub summary: active paths [/] 2024-03-28T22:39:17.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:17 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:39:18.343 INFO:tasks.workunit.client.0.smithi043.stdout: 1 79981 9.12 MB/sec warmup 49 sec latency 19.247 ms 2024-03-28T22:39:18.765 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:18 smithi043 ceph-mon[31823]: pgmap v66: 129 pgs: 129 active+clean; 39 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 347 op/s 2024-03-28T22:39:18.765 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:18 smithi043 ceph-mon[31823]: scrub summary: idle 2024-03-28T22:39:18.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:18 smithi172 ceph-mon[36566]: pgmap v66: 129 pgs: 129 active+clean; 39 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 347 op/s 2024-03-28T22:39:18.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:18 smithi172 ceph-mon[36566]: scrub summary: idle 2024-03-28T22:39:18.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:18 smithi082 ceph-mon[36471]: pgmap v66: 129 pgs: 129 active+clean; 39 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 347 op/s 2024-03-28T22:39:18.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:18 smithi082 ceph-mon[36471]: scrub summary: idle 2024-03-28T22:39:19.344 INFO:tasks.workunit.client.0.smithi043.stdout: 1 81894 9.21 MB/sec warmup 50 sec latency 9.373 ms 2024-03-28T22:39:20.344 INFO:tasks.workunit.client.0.smithi043.stdout: 1 83244 9.14 MB/sec warmup 51 sec latency 21.706 ms 2024-03-28T22:39:20.765 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:20 smithi043 ceph-mon[31823]: pgmap v67: 129 pgs: 129 active+clean; 39 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 260 op/s 2024-03-28T22:39:20.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:20 smithi172 ceph-mon[36566]: pgmap v67: 129 pgs: 129 active+clean; 39 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 260 op/s 2024-03-28T22:39:20.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:20 smithi082 ceph-mon[36471]: pgmap v67: 129 pgs: 129 active+clean; 39 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 260 op/s 2024-03-28T22:39:21.344 INFO:tasks.workunit.client.0.smithi043.stdout: 1 85189 9.20 MB/sec warmup 52 sec latency 49.089 ms 2024-03-28T22:39:22.344 INFO:tasks.workunit.client.0.smithi043.stdout: 1 86121 9.07 MB/sec warmup 53 sec latency 19.370 ms 2024-03-28T22:39:22.765 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:22 smithi043 ceph-mon[31823]: pgmap v68: 129 pgs: 129 active+clean; 39 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 260 op/s 2024-03-28T22:39:22.765 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:22 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:39:22.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:22 smithi172 ceph-mon[36566]: pgmap v68: 129 pgs: 129 active+clean; 39 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 260 op/s 2024-03-28T22:39:22.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:22 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:39:22.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:22 smithi082 ceph-mon[36471]: pgmap v68: 129 pgs: 129 active+clean; 39 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 260 op/s 2024-03-28T22:39:22.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:22 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:39:23.344 INFO:tasks.workunit.client.0.smithi043.stdout: 1 88374 9.13 MB/sec warmup 54 sec latency 13.664 ms 2024-03-28T22:39:24.344 INFO:tasks.workunit.client.0.smithi043.stdout: 1 89506 9.08 MB/sec warmup 55 sec latency 16.646 ms 2024-03-28T22:39:24.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:24 smithi043 ceph-mon[31823]: pgmap v69: 129 pgs: 129 active+clean; 42 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 351 op/s 2024-03-28T22:39:24.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:24 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:39:24.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:24 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:39:24.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:24 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:39:24.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:24 smithi172 ceph-mon[36566]: pgmap v69: 129 pgs: 129 active+clean; 42 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 351 op/s 2024-03-28T22:39:24.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:24 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:39:24.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:24 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:39:24.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:24 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:39:24.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:24 smithi082 ceph-mon[36471]: pgmap v69: 129 pgs: 129 active+clean; 42 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 351 op/s 2024-03-28T22:39:24.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:24 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:39:24.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:24 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:39:24.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:24 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:39:25.344 INFO:tasks.workunit.client.0.smithi043.stdout: 1 91640 9.13 MB/sec warmup 56 sec latency 11.730 ms 2024-03-28T22:39:26.345 INFO:tasks.workunit.client.0.smithi043.stdout: 1 92754 9.09 MB/sec warmup 57 sec latency 56.294 ms 2024-03-28T22:39:26.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:26 smithi043 ceph-mon[31823]: pgmap v70: 129 pgs: 129 active+clean; 42 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 271 op/s 2024-03-28T22:39:26.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:26 smithi172 ceph-mon[36566]: pgmap v70: 129 pgs: 129 active+clean; 42 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 271 op/s 2024-03-28T22:39:26.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:26 smithi082 ceph-mon[36471]: pgmap v70: 129 pgs: 129 active+clean; 42 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 271 op/s 2024-03-28T22:39:27.128 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.129+0000 7f60b9e00640 1 -- 172.21.15.43:0/2205066287 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f60b4109940 msgr2=0x7f60b4109d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:27.128 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.129+0000 7f60b9e00640 1 --2- 172.21.15.43:0/2205066287 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f60b4109940 0x7f60b4109d20 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f60a4009900 tx=0x7f60a402bc30 comp rx=0 tx=0).stop 2024-03-28T22:39:27.128 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.129+0000 7f60b9e00640 1 -- 172.21.15.43:0/2205066287 shutdown_connections 2024-03-28T22:39:27.128 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.129+0000 7f60b9e00640 1 --2- 172.21.15.43:0/2205066287 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60b41042f0 0x7f60b4110800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:27.128 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.129+0000 7f60b9e00640 1 --2- 172.21.15.43:0/2205066287 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f60b4103930 0x7f60b4103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:27.128 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.129+0000 7f60b9e00640 1 --2- 172.21.15.43:0/2205066287 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f60b4109940 0x7f60b4109d20 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:27.128 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.129+0000 7f60b9e00640 1 -- 172.21.15.43:0/2205066287 >> 172.21.15.43:0/2205066287 conn(0x7f60b40ff400 msgr2=0x7f60b4101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:27.128 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.129+0000 7f60b9e00640 1 -- 172.21.15.43:0/2205066287 shutdown_connections 2024-03-28T22:39:27.128 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.129+0000 7f60b9e00640 1 -- 172.21.15.43:0/2205066287 wait complete. 2024-03-28T22:39:27.130 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.130+0000 7f60b9e00640 1 Processor -- start 2024-03-28T22:39:27.131 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.130+0000 7f60b9e00640 1 -- start start 2024-03-28T22:39:27.131 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.131+0000 7f60b9e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f60b4103930 0x7f60b419a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:27.131 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.131+0000 7f60b9e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60b41042f0 0x7f60b419a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:27.131 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.131+0000 7f60b9e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f60b4109940 0x7f60b419ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:27.131 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.131+0000 7f60b9e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f60b4113d20 con 0x7f60b41042f0 2024-03-28T22:39:27.131 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.131+0000 7f60b9e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f60b4113ba0 con 0x7f60b4103930 2024-03-28T22:39:27.131 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.131+0000 7f60b9e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f60b4113ea0 con 0x7f60b4109940 2024-03-28T22:39:27.131 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.131+0000 7f60b2a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f60b4103930 0x7f60b419a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:27.131 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.131+0000 7f60b2a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f60b4103930 0x7f60b419a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52572/0 (socket says 172.21.15.43:52572) 2024-03-28T22:39:27.131 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.131+0000 7f60b2a00640 1 -- 172.21.15.43:0/1490326569 learned_addr learned my addr 172.21.15.43:0/1490326569 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:27.131 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.131+0000 7f60b2000640 1 --2- 172.21.15.43:0/1490326569 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60b41042f0 0x7f60b419a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:27.131 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.131+0000 7f60b3400640 1 --2- 172.21.15.43:0/1490326569 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f60b4109940 0x7f60b419ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:27.132 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.131+0000 7f60b2a00640 1 -- 172.21.15.43:0/1490326569 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f60b4109940 msgr2=0x7f60b419ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:27.132 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.131+0000 7f60b2a00640 1 --2- 172.21.15.43:0/1490326569 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f60b4109940 0x7f60b419ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:27.132 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.131+0000 7f60b2a00640 1 -- 172.21.15.43:0/1490326569 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60b41042f0 msgr2=0x7f60b419a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:27.132 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.131+0000 7f60b2a00640 1 --2- 172.21.15.43:0/1490326569 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60b41042f0 0x7f60b419a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:27.132 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.131+0000 7f60b2a00640 1 -- 172.21.15.43:0/1490326569 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60b419f180 con 0x7f60b4103930 2024-03-28T22:39:27.132 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.132+0000 7f60b2a00640 1 --2- 172.21.15.43:0/1490326569 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f60b4103930 0x7f60b419a1e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f60a4005940 tx=0x7f60a4004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:27.132 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.132+0000 7f60b8a00640 1 -- 172.21.15.43:0/1490326569 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a4041070 con 0x7f60b4103930 2024-03-28T22:39:27.132 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.132+0000 7f60b8a00640 1 -- 172.21.15.43:0/1490326569 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60a40054e0 con 0x7f60b4103930 2024-03-28T22:39:27.132 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.132+0000 7f60b8a00640 1 -- 172.21.15.43:0/1490326569 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a4003d00 con 0x7f60b4103930 2024-03-28T22:39:27.132 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.132+0000 7f60b9e00640 1 -- 172.21.15.43:0/1490326569 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60b419f410 con 0x7f60b4103930 2024-03-28T22:39:27.132 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.132+0000 7f60b9e00640 1 -- 172.21.15.43:0/1490326569 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60b41a6d00 con 0x7f60b4103930 2024-03-28T22:39:27.134 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.135+0000 7f60b8a00640 1 -- 172.21.15.43:0/1490326569 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f60a4044070 con 0x7f60b4103930 2024-03-28T22:39:27.134 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.136+0000 7f60bcb97740 1 Processor -- start 2024-03-28T22:39:27.134 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.136+0000 7f60bcb97740 1 -- start start 2024-03-28T22:39:27.136 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.136+0000 7f60bcb97740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b66bf553a0 0x55b66bf55780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:27.136 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.136+0000 7f60bcb97740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b66bf55cc0 0x55b66bf5e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:27.136 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.136+0000 7f60bcb97740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b66bf5ea90 0x55b66bf60e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:27.136 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.136+0000 7f60bcb97740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b66beb8640 con 0x55b66bf5ea90 2024-03-28T22:39:27.136 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.136+0000 7f60bcb97740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55b66be61810 con 0x55b66bf55cc0 2024-03-28T22:39:27.136 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.136+0000 7f60bcb97740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55b66beb46e0 con 0x55b66bf553a0 2024-03-28T22:39:27.136 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.137+0000 7f60b3400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b66bf553a0 0x55b66bf55780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:27.136 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.137+0000 7f60b2a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b66bf55cc0 0x55b66bf5e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:27.136 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.137+0000 7f60b3400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b66bf553a0 0x55b66bf55780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58672/0 (socket says 172.21.15.43:58672) 2024-03-28T22:39:27.137 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.137+0000 7f60b3400640 1 -- 172.21.15.43:0/2876256375 learned_addr learned my addr 172.21.15.43:0/2876256375 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:27.137 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.137+0000 7f60b2000640 1 --2- 172.21.15.43:0/2876256375 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b66bf5ea90 0x55b66bf60e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:27.137 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.137+0000 7f60b8a00640 1 --2- 172.21.15.43:0/1490326569 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f60980a2010 0x7f60980a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:27.137 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.137+0000 7f60b3400640 1 -- 172.21.15.43:0/2876256375 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b66bf55cc0 msgr2=0x55b66bf5e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:27.137 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.137+0000 7f60b8a00640 1 -- 172.21.15.43:0/1490326569 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f60a40eff40 con 0x7f60b4103930 2024-03-28T22:39:27.137 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.137+0000 7f60b3400640 1 --2- 172.21.15.43:0/2876256375 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b66bf55cc0 0x55b66bf5e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:27.137 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.137+0000 7f60b3400640 1 -- 172.21.15.43:0/2876256375 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b66bf5ea90 msgr2=0x55b66bf60e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:27.137 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.137+0000 7f60b3400640 1 --2- 172.21.15.43:0/2876256375 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b66bf5ea90 0x55b66bf60e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:27.137 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.137+0000 7f60b3400640 1 -- 172.21.15.43:0/2876256375 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b66bbcd210 con 0x55b66bf553a0 2024-03-28T22:39:27.137 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.137+0000 7f60b2000640 1 --2- 172.21.15.43:0/2876256375 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b66bf5ea90 0x55b66bf60e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:27.137 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.137+0000 7f60b2a00640 1 --2- 172.21.15.43:0/2876256375 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b66bf55cc0 0x55b66bf5e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:27.137 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.137+0000 7f60b3400640 1 --2- 172.21.15.43:0/1490326569 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f60980a2010 0x7f60980a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:27.138 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.138+0000 7f60b3400640 1 --2- 172.21.15.43:0/2876256375 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b66bf553a0 0x55b66bf55780 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f60ac010520 tx=0x7f60ac0109e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20eef0e64a57f990 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:27.139 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.138+0000 7f609fe00640 1 -- 172.21.15.43:0/2876256375 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60ac019070 con 0x55b66bf553a0 2024-03-28T22:39:27.139 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.138+0000 7f609fe00640 1 -- 172.21.15.43:0/2876256375 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60ac003e10 con 0x55b66bf553a0 2024-03-28T22:39:27.139 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.138+0000 7f609fe00640 1 -- 172.21.15.43:0/2876256375 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60ac01d760 con 0x55b66bf553a0 2024-03-28T22:39:27.139 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.139+0000 7f60bcb97740 1 -- 172.21.15.43:0/2876256375 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b66bf553a0 msgr2=0x55b66bf55780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:27.139 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.139+0000 7f60bcb97740 1 --2- 172.21.15.43:0/2876256375 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b66bf553a0 0x55b66bf55780 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f60ac010520 tx=0x7f60ac0109e0 comp rx=0 tx=0).stop 2024-03-28T22:39:27.139 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.139+0000 7f60b3400640 1 --2- 172.21.15.43:0/1490326569 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f60980a2010 0x7f60980a44d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f60ac021000 tx=0x7f60ac01e000 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:27.139 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.139+0000 7f60bcb97740 1 -- 172.21.15.43:0/2876256375 shutdown_connections 2024-03-28T22:39:27.139 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.139+0000 7f60bcb97740 1 --2- 172.21.15.43:0/2876256375 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b66bf55cc0 0x55b66bf5e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:27.139 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.139+0000 7f60bcb97740 1 --2- 172.21.15.43:0/2876256375 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b66bf5ea90 0x55b66bf60e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:27.139 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.139+0000 7f60bcb97740 1 --2- 172.21.15.43:0/2876256375 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b66bf553a0 0x55b66bf55780 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:27.139 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.139+0000 7f60bcb97740 1 -- 172.21.15.43:0/2876256375 >> 172.21.15.43:0/2876256375 conn(0x55b66bf4fcc0 msgr2=0x55b66bf500a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:27.139 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.139+0000 7f60bcb97740 1 -- 172.21.15.43:0/2876256375 shutdown_connections 2024-03-28T22:39:27.139 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.139+0000 7f60bcb97740 1 -- 172.21.15.43:0/2876256375 wait complete. 2024-03-28T22:39:27.139 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.140+0000 7f60bcb97740 1 -- start start 2024-03-28T22:39:27.140 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.140+0000 7f60bcb97740 1 Processor -- start 2024-03-28T22:39:27.143 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.140+0000 7f60bcb97740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b66bf553a0 0x55b66bf5ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:27.143 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.140+0000 7f60bcb97740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b66bf55cc0 0x55b66bf5bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:27.143 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.140+0000 7f60bcb97740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b66bf5ea90 0x55b66bf5c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:27.143 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.141+0000 7f60bcb97740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b66bf616c0 con 0x55b66bf5ea90 2024-03-28T22:39:27.143 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.141+0000 7f60bcb97740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55b66be56d00 con 0x55b66bf553a0 2024-03-28T22:39:27.143 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.141+0000 7f60bcb97740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55b66bf61840 con 0x55b66bf55cc0 2024-03-28T22:39:27.143 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.141+0000 7f60b2a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b66bf553a0 0x55b66bf5ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:27.143 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.141+0000 7f60b2000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b66bf55cc0 0x55b66bf5bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:27.143 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.141+0000 7f60b2a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b66bf553a0 0x55b66bf5ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52594/0 (socket says 172.21.15.43:52594) 2024-03-28T22:39:27.143 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.141+0000 7f60b2a00640 1 -- 172.21.15.43:0/1413379382 learned_addr learned my addr 172.21.15.43:0/1413379382 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:27.144 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.141+0000 7f60bcb97740 10 client.0 _ll_register_callbacks cb 0x55b66be41490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:27.144 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.141+0000 7f60b3400640 1 --2- 172.21.15.43:0/1413379382 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b66bf5ea90 0x55b66bf5c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:27.144 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.141+0000 7f60b2a00640 1 -- 172.21.15.43:0/1413379382 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b66bf55cc0 msgr2=0x55b66bf5bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:27.144 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.141+0000 7f60b2a00640 1 --2- 172.21.15.43:0/1413379382 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b66bf55cc0 0x55b66bf5bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:27.144 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.141+0000 7f60b2a00640 1 -- 172.21.15.43:0/1413379382 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b66bf5ea90 msgr2=0x55b66bf5c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:27.144 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.141+0000 7f60b2a00640 1 --2- 172.21.15.43:0/1413379382 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b66bf5ea90 0x55b66bf5c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:27.144 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.141+0000 7f60b2a00640 1 -- 172.21.15.43:0/1413379382 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b66bc99620 con 0x55b66bf553a0 2024-03-28T22:39:27.144 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.141+0000 7f60b2a00640 1 --2- 172.21.15.43:0/1413379382 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b66bf553a0 0x55b66bf5ba80 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f60a4000c00 tx=0x7f60a4003f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc3d51eb6aa89290 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:27.144 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.141+0000 7f609cc00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:39:27.144 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.142+0000 7f609cc00640 1 -- 172.21.15.43:0/1413379382 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a404b030 con 0x55b66bf553a0 2024-03-28T22:39:27.144 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.142+0000 7f609cc00640 1 -- 172.21.15.43:0/1413379382 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f60a40ba940 con 0x55b66bf553a0 2024-03-28T22:39:27.144 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.142+0000 7f60bcb97740 1 -- 172.21.15.43:0/1413379382 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60ac003e10 con 0x55b66bf553a0 2024-03-28T22:39:27.144 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.142+0000 7f609cc00640 1 -- 172.21.15.43:0/1413379382 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f60a40bc410 con 0x55b66bf553a0 2024-03-28T22:39:27.145 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.142+0000 7f60bcb97740 10 client.26495 fetch_fsmap learned FSMap version 29 2024-03-28T22:39:27.145 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.142+0000 7f60bcb97740 1 -- 172.21.15.43:0/1413379382 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b66bf6d5c0 con 0x55b66bf553a0 2024-03-28T22:39:27.145 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.142+0000 7f609cc00640 1 -- 172.21.15.43:0/1413379382 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60ac003e10 con 0x55b66bf553a0 2024-03-28T22:39:27.145 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.142+0000 7f609cc00640 1 -- 172.21.15.43:0/1413379382 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f60a40dc0c0 con 0x55b66bf553a0 2024-03-28T22:39:27.145 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.142+0000 7f60bcb97740 10 client.26495 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:39:27.145 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.142+0000 7f60bcb97740 10 client.26495 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:39:27.145 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.143+0000 7f60bcb97740 20 client.26495 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:27.145 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.143+0000 7f60bcb97740 1 --2- 172.21.15.43:0/1413379382 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b66bf6ea70 0x55b66bf70f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:27.145 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.143+0000 7f60bcb97740 4 client.26495 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:39:27.145 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.143+0000 7f60bcb97740 1 -- 172.21.15.43:0/1413379382 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b66becc030 con 0x55b66bf6ea70 2024-03-28T22:39:27.145 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.143+0000 7f60b2000640 1 --2- 172.21.15.43:0/1413379382 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b66bf6ea70 0x55b66bf70f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:27.145 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.143+0000 7f60b2000640 1 --2- 172.21.15.43:0/1413379382 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b66bf6ea70 0x55b66bf70f00 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e28c81dc6314d81c server_cookie=fb4b5fbf35cc7fc2 in_seq=0 out_seq=0 2024-03-28T22:39:27.145 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.143+0000 7f609cc00640 10 client.26495 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:27.146 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.145+0000 7f609cc00640 1 -- 172.21.15.43:0/1413379382 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b66becc030 con 0x55b66bf6ea70 2024-03-28T22:39:27.147 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.145+0000 7f609cc00640 10 client.26495 handle_command_reply: tid=0 2024-03-28T22:39:27.147 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.145+0000 7f60bcb97740 1 client.26495 shutdown 2024-03-28T22:39:27.147 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.145+0000 7f60bcb97740 1 -- 172.21.15.43:0/1413379382 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b66bf553a0 msgr2=0x55b66bf5ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:27.147 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.145+0000 7f60bcb97740 1 --2- 172.21.15.43:0/1413379382 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b66bf553a0 0x55b66bf5ba80 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f60a4000c00 tx=0x7f60a4003f20 comp rx=0 tx=0).stop 2024-03-28T22:39:27.147 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.145+0000 7f60bcb97740 1 -- 172.21.15.43:0/1413379382 shutdown_connections 2024-03-28T22:39:27.147 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.145+0000 7f60bcb97740 1 --2- 172.21.15.43:0/1413379382 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b66bf5ea90 0x55b66bf5c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:27.147 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.145+0000 7f60bcb97740 1 --2- 172.21.15.43:0/1413379382 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b66bf55cc0 0x55b66bf5bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:27.147 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.145+0000 7f60bcb97740 1 --2- 172.21.15.43:0/1413379382 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b66bf6ea70 0x55b66bf70f00 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:27.147 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.146+0000 7f609cc00640 0 client.26495 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:27.147 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.146+0000 7f60bcb97740 1 --2- 172.21.15.43:0/1413379382 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b66bf553a0 0x55b66bf5ba80 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:27.147 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.146+0000 7f60bcb97740 1 -- 172.21.15.43:0/1413379382 >> 172.21.15.43:0/1413379382 conn(0x55b66bf4fcc0 msgr2=0x55b66bf51a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:27.147 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.146+0000 7f60bcb97740 1 -- 172.21.15.43:0/1413379382 shutdown_connections 2024-03-28T22:39:27.147 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.146+0000 7f60bcb97740 1 -- 172.21.15.43:0/1413379382 wait complete. 2024-03-28T22:39:27.147 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.146+0000 7f60bcb97740 20 client.26495 trim_cache size 0 max 16384 2024-03-28T22:39:27.170 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.163+0000 7f60bcb97740 1 Processor -- start 2024-03-28T22:39:27.170 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.163+0000 7f60bcb97740 1 -- start start 2024-03-28T22:39:27.171 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.163+0000 7f60bcb97740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b66bf553a0 0x55b66bf5e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:27.171 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.163+0000 7f60bcb97740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b66bf55cc0 0x55b66bf6a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:27.171 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.163+0000 7f60bcb97740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b66bf5ea90 0x55b66bf76810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:27.171 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.163+0000 7f60bcb97740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f60b41b0dc0 con 0x55b66bf55cc0 2024-03-28T22:39:27.171 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.163+0000 7f60bcb97740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f60b41b2900 con 0x55b66bf5ea90 2024-03-28T22:39:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.163+0000 7f60bcb97740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f60b41b1490 con 0x55b66bf553a0 2024-03-28T22:39:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.163+0000 7f60b2a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b66bf553a0 0x55b66bf5e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.163+0000 7f60b2a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b66bf553a0 0x55b66bf5e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58682/0 (socket says 172.21.15.43:58682) 2024-03-28T22:39:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.163+0000 7f60b2a00640 1 -- 172.21.15.43:0/17756259 learned_addr learned my addr 172.21.15.43:0/17756259 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.164+0000 7f60b2000640 1 --2- 172.21.15.43:0/17756259 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b66bf55cc0 0x55b66bf6a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.164+0000 7f60b3400640 1 --2- 172.21.15.43:0/17756259 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b66bf5ea90 0x55b66bf76810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.164+0000 7f60b2a00640 1 -- 172.21.15.43:0/17756259 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b66bf5ea90 msgr2=0x55b66bf76810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.164+0000 7f60b2a00640 1 --2- 172.21.15.43:0/17756259 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b66bf5ea90 0x55b66bf76810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.164+0000 7f60b2a00640 1 -- 172.21.15.43:0/17756259 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b66bf55cc0 msgr2=0x55b66bf6a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.164+0000 7f60b2a00640 1 --2- 172.21.15.43:0/17756259 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b66bf55cc0 0x55b66bf6a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.164+0000 7f60b2a00640 1 -- 172.21.15.43:0/17756259 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b66bf71580 con 0x55b66bf553a0 2024-03-28T22:39:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.164+0000 7f60b2a00640 1 --2- 172.21.15.43:0/17756259 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b66bf553a0 0x55b66bf5e140 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f60a402cfd0 tx=0x7f60a402bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e0a2ac702d4e50c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.164+0000 7f609f400640 1 -- 172.21.15.43:0/17756259 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a404b030 con 0x55b66bf553a0 2024-03-28T22:39:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.165+0000 7f609f400640 1 -- 172.21.15.43:0/17756259 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60a4005690 con 0x55b66bf553a0 2024-03-28T22:39:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.165+0000 7f609f400640 1 -- 172.21.15.43:0/17756259 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a40bbfa0 con 0x55b66bf553a0 2024-03-28T22:39:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.165+0000 7f60bcb97740 1 -- 172.21.15.43:0/17756259 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b66bf553a0 msgr2=0x55b66bf5e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.165+0000 7f60bcb97740 1 --2- 172.21.15.43:0/17756259 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b66bf553a0 0x55b66bf5e140 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f60a402cfd0 tx=0x7f60a402bc30 comp rx=0 tx=0).stop 2024-03-28T22:39:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.165+0000 7f60bcb97740 1 -- 172.21.15.43:0/17756259 shutdown_connections 2024-03-28T22:39:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.165+0000 7f60bcb97740 1 --2- 172.21.15.43:0/17756259 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b66bf5ea90 0x55b66bf76810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.165+0000 7f60bcb97740 1 --2- 172.21.15.43:0/17756259 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b66bf55cc0 0x55b66bf6a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.165+0000 7f60bcb97740 1 --2- 172.21.15.43:0/17756259 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b66bf553a0 0x55b66bf5e140 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.165+0000 7f60bcb97740 1 -- 172.21.15.43:0/17756259 >> 172.21.15.43:0/17756259 conn(0x55b66bf4fcc0 msgr2=0x55b66bf4ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.165+0000 7f60bcb97740 1 -- 172.21.15.43:0/17756259 shutdown_connections 2024-03-28T22:39:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.165+0000 7f60bcb97740 1 -- 172.21.15.43:0/17756259 wait complete. 2024-03-28T22:39:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.166+0000 7f60bcb97740 1 -- start start 2024-03-28T22:39:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.166+0000 7f60bcb97740 1 Processor -- start 2024-03-28T22:39:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.166+0000 7f60bcb97740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b66bf553a0 0x55b66bf66c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.166+0000 7f60bcb97740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b66bf55cc0 0x55b66bf62f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.166+0000 7f60bcb97740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b66bf5ea90 0x55b66bf634d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.166+0000 7f60bcb97740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f60b41b2230 con 0x55b66bf55cc0 2024-03-28T22:39:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.166+0000 7f60bcb97740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f60b41b1b60 con 0x55b66bf553a0 2024-03-28T22:39:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.166+0000 7f60bcb97740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f60b41aeda0 con 0x55b66bf5ea90 2024-03-28T22:39:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.166+0000 7f60bcb97740 10 client.0 _ll_register_callbacks cb 0x7f60a40035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.167+0000 7f60b2a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b66bf553a0 0x55b66bf66c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.167+0000 7f60b2a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b66bf553a0 0x55b66bf66c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52612/0 (socket says 172.21.15.43:52612) 2024-03-28T22:39:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.167+0000 7f60b2a00640 1 -- 172.21.15.43:0/3267262658 learned_addr learned my addr 172.21.15.43:0/3267262658 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.167+0000 7f60b3400640 1 --2- 172.21.15.43:0/3267262658 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b66bf5ea90 0x55b66bf634d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.167+0000 7f60b2000640 1 --2- 172.21.15.43:0/3267262658 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b66bf55cc0 0x55b66bf62f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.167+0000 7f60b2a00640 1 -- 172.21.15.43:0/3267262658 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b66bf5ea90 msgr2=0x55b66bf634d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.167+0000 7f60b2a00640 1 --2- 172.21.15.43:0/3267262658 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b66bf5ea90 0x55b66bf634d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.167+0000 7f60b2a00640 1 -- 172.21.15.43:0/3267262658 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b66bf55cc0 msgr2=0x55b66bf62f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.167+0000 7f60b2a00640 1 --2- 172.21.15.43:0/3267262658 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b66bf55cc0 0x55b66bf62f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.167+0000 7f60b2a00640 1 -- 172.21.15.43:0/3267262658 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b66bf615a0 con 0x55b66bf553a0 2024-03-28T22:39:27.177 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.167+0000 7f60b2a00640 1 --2- 172.21.15.43:0/3267262658 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b66bf553a0 0x55b66bf66c10 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f60a40b8aa0 tx=0x7f60a40c0860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b246cc45c3151fa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:27.177 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.167+0000 7f609cc00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:39:27.177 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.168+0000 7f609cc00640 1 -- 172.21.15.43:0/3267262658 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a404b070 con 0x55b66bf553a0 2024-03-28T22:39:27.177 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.168+0000 7f609cc00640 1 -- 172.21.15.43:0/3267262658 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9451+0+0 (secure 0 0 0) 0x7f60a40c4b60 con 0x55b66bf553a0 2024-03-28T22:39:27.177 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.168+0000 7f60bcb97740 1 -- 172.21.15.43:0/3267262658 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b66bec5b80 con 0x55b66bf553a0 2024-03-28T22:39:27.177 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.168+0000 7f609cc00640 1 -- 172.21.15.43:0/3267262658 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f60a40bc180 con 0x55b66bf553a0 2024-03-28T22:39:27.177 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.168+0000 7f60bcb97740 10 client.26501 fetch_fsmap learned FSMap version 29 2024-03-28T22:39:27.177 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.168+0000 7f60bcb97740 1 -- 172.21.15.43:0/3267262658 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b66bf615a0 con 0x55b66bf553a0 2024-03-28T22:39:27.177 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.168+0000 7f609cc00640 1 -- 172.21.15.43:0/3267262658 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55b66bec5b80 con 0x55b66bf553a0 2024-03-28T22:39:27.178 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.168+0000 7f609cc00640 1 -- 172.21.15.43:0/3267262658 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f60a40a5b80 con 0x55b66bf553a0 2024-03-28T22:39:27.178 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.168+0000 7f60bcb97740 10 client.26501 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T22:39:27.178 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.168+0000 7f60bcb97740 10 client.26501 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:39:27.178 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.168+0000 7f60bcb97740 20 client.26501 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:27.178 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.168+0000 7f60bcb97740 1 --2- 172.21.15.43:0/3267262658 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b66bfd5d60 0x55b66bfd8150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:27.178 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.168+0000 7f60bcb97740 4 client.26501 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T22:39:27.178 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.168+0000 7f60bcb97740 1 -- 172.21.15.43:0/3267262658 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b66bebfed0 con 0x55b66bfd5d60 2024-03-28T22:39:27.178 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.169+0000 7f60b2000640 1 --2- 172.21.15.43:0/3267262658 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b66bfd5d60 0x55b66bfd8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:27.178 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.169+0000 7f60b2000640 1 --2- 172.21.15.43:0/3267262658 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b66bfd5d60 0x55b66bfd8150 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d4ca53950b4e0e3 server_cookie=baf0aae6b10868b8 in_seq=0 out_seq=0 2024-03-28T22:39:27.179 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.169+0000 7f609cc00640 10 client.26501 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:27.345 INFO:tasks.workunit.client.0.smithi043.stdout: 1 94143 9.03 MB/sec warmup 58 sec latency 65.374 ms 2024-03-28T22:39:27.371 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.372+0000 7f609cc00640 1 -- 172.21.15.43:0/3267262658 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b66bebfed0 con 0x55b66bfd5d60 2024-03-28T22:39:27.371 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.372+0000 7f609cc00640 10 client.26501 handle_command_reply: tid=0 2024-03-28T22:39:27.371 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.372+0000 7f60bcb97740 1 client.26501 shutdown 2024-03-28T22:39:27.371 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.372+0000 7f60bcb97740 1 -- 172.21.15.43:0/3267262658 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b66bf553a0 msgr2=0x55b66bf66c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:27.371 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.372+0000 7f60bcb97740 1 --2- 172.21.15.43:0/3267262658 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b66bf553a0 0x55b66bf66c10 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f60a40b8aa0 tx=0x7f60a40c0860 comp rx=0 tx=0).stop 2024-03-28T22:39:27.372 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.373+0000 7f60bcb97740 1 -- 172.21.15.43:0/3267262658 shutdown_connections 2024-03-28T22:39:27.372 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.373+0000 7f60bcb97740 1 --2- 172.21.15.43:0/3267262658 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b66bf5ea90 0x55b66bf634d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:27.372 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.373+0000 7f60bcb97740 1 --2- 172.21.15.43:0/3267262658 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b66bf55cc0 0x55b66bf62f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:27.372 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.373+0000 7f60bcb97740 1 --2- 172.21.15.43:0/3267262658 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b66bfd5d60 0x55b66bfd8150 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:27.372 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.373+0000 7f60bcb97740 1 --2- 172.21.15.43:0/3267262658 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b66bf553a0 0x55b66bf66c10 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:27.373 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.373+0000 7f60bcb97740 1 -- 172.21.15.43:0/3267262658 >> 172.21.15.43:0/3267262658 conn(0x55b66bf4fcc0 msgr2=0x55b66bf51110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:27.373 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.373+0000 7f609cc00640 0 client.26501 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:27.373 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.373+0000 7f60bcb97740 1 -- 172.21.15.43:0/3267262658 shutdown_connections 2024-03-28T22:39:27.373 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.373+0000 7f60bcb97740 1 -- 172.21.15.43:0/3267262658 wait complete. 2024-03-28T22:39:27.373 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.373+0000 7f60bcb97740 20 client.26501 trim_cache size 0 max 16384 2024-03-28T22:39:27.373 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:{ 2024-03-28T22:39:27.373 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "message": "", 2024-03-28T22:39:27.373 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "return_code": 0 2024-03-28T22:39:27.373 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:} 2024-03-28T22:39:27.374 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.374+0000 7f60b9e00640 1 -- 172.21.15.43:0/1490326569 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f60980a2010 msgr2=0x7f60980a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:27.374 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.374+0000 7f60b9e00640 1 --2- 172.21.15.43:0/1490326569 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f60980a2010 0x7f60980a44d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f60ac021000 tx=0x7f60ac01e000 comp rx=0 tx=0).stop 2024-03-28T22:39:27.375 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.374+0000 7f60b9e00640 1 -- 172.21.15.43:0/1490326569 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f60b4103930 msgr2=0x7f60b419a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:27.375 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.374+0000 7f60b9e00640 1 --2- 172.21.15.43:0/1490326569 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f60b4103930 0x7f60b419a1e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f60a4005940 tx=0x7f60a4004270 comp rx=0 tx=0).stop 2024-03-28T22:39:27.375 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.374+0000 7f60b9e00640 1 -- 172.21.15.43:0/1490326569 shutdown_connections 2024-03-28T22:39:27.375 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.374+0000 7f60b9e00640 1 --2- 172.21.15.43:0/1490326569 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f60980a2010 0x7f60980a44d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:27.375 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.374+0000 7f60b9e00640 1 --2- 172.21.15.43:0/1490326569 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f60b4109940 0x7f60b419ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:27.375 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.374+0000 7f60b9e00640 1 --2- 172.21.15.43:0/1490326569 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f60b41042f0 0x7f60b419a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:27.375 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.374+0000 7f60b9e00640 1 --2- 172.21.15.43:0/1490326569 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f60b4103930 0x7f60b419a1e0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:27.375 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.374+0000 7f60b9e00640 1 -- 172.21.15.43:0/1490326569 >> 172.21.15.43:0/1490326569 conn(0x7f60b40ff400 msgr2=0x7f60b40ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:27.375 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.374+0000 7f60b9e00640 1 -- 172.21.15.43:0/1490326569 shutdown_connections 2024-03-28T22:39:27.375 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:27.375+0000 7f60b9e00640 1 -- 172.21.15.43:0/1490326569 wait complete. 2024-03-28T22:39:27.949 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 28 22:39:27 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-e[66639]: ignoring --setuser ceph since I am not root 2024-03-28T22:39:27.949 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 28 22:39:27 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-e[66639]: ignoring --setgroup ceph since I am not root 2024-03-28T22:39:27.949 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 28 22:39:27 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-e[66639]: starting mds.e at 2024-03-28T22:39:28.345 INFO:tasks.workunit.client.0.smithi043.stdout: 1 96079 9.10 MB/sec warmup 59 sec latency 84.839 ms 2024-03-28T22:39:28.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:28 smithi172 ceph-mon[36566]: pgmap v71: 129 pgs: 129 active+clean; 41 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.5 MiB/s wr, 463 op/s 2024-03-28T22:39:28.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:28 smithi172 ceph-mon[36566]: daemon mds.e restarted 2024-03-28T22:39:28.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:28 smithi172 ceph-mon[36566]: osdmap e98: 12 total, 12 up, 12 in 2024-03-28T22:39:28.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:28 smithi172 ceph-mon[36566]: mds.? [v2:172.21.15.82:6834/4245880955,v1:172.21.15.82:6835/4245880955] up:boot 2024-03-28T22:39:28.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:28 smithi172 ceph-mon[36566]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T22:39:28.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:28 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T22:39:28.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:28 smithi082 ceph-mon[36471]: pgmap v71: 129 pgs: 129 active+clean; 41 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.5 MiB/s wr, 463 op/s 2024-03-28T22:39:28.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:28 smithi082 ceph-mon[36471]: daemon mds.e restarted 2024-03-28T22:39:28.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:28 smithi082 ceph-mon[36471]: osdmap e98: 12 total, 12 up, 12 in 2024-03-28T22:39:28.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:28 smithi082 ceph-mon[36471]: mds.? [v2:172.21.15.82:6834/4245880955,v1:172.21.15.82:6835/4245880955] up:boot 2024-03-28T22:39:28.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:28 smithi082 ceph-mon[36471]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T22:39:28.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:28 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T22:39:29.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:28 smithi043 ceph-mon[31823]: pgmap v71: 129 pgs: 129 active+clean; 41 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.5 MiB/s wr, 463 op/s 2024-03-28T22:39:29.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:28 smithi043 ceph-mon[31823]: daemon mds.e restarted 2024-03-28T22:39:29.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:28 smithi043 ceph-mon[31823]: osdmap e98: 12 total, 12 up, 12 in 2024-03-28T22:39:29.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:28 smithi043 ceph-mon[31823]: mds.? [v2:172.21.15.82:6834/4245880955,v1:172.21.15.82:6835/4245880955] up:boot 2024-03-28T22:39:29.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:28 smithi043 ceph-mon[31823]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T22:39:29.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:28 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T22:39:29.345 INFO:tasks.workunit.client.0.smithi043.stdout: 1 97374 9.04 MB/sec warmup 60 sec latency 20.087 ms 2024-03-28T22:39:30.345 INFO:tasks.workunit.client.0.smithi043.stdout: 1 99576 9.12 MB/sec warmup 61 sec latency 9.589 ms 2024-03-28T22:39:30.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:30 smithi172 ceph-mon[36566]: pgmap v73: 129 pgs: 129 active+clean; 41 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 339 op/s 2024-03-28T22:39:30.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:30 smithi082 ceph-mon[36471]: pgmap v73: 129 pgs: 129 active+clean; 41 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 339 op/s 2024-03-28T22:39:31.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:30 smithi043 ceph-mon[31823]: pgmap v73: 129 pgs: 129 active+clean; 41 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 339 op/s 2024-03-28T22:39:31.345 INFO:tasks.workunit.client.0.smithi043.stdout: 1 100429 9.01 MB/sec warmup 62 sec latency 36.671 ms 2024-03-28T22:39:32.345 INFO:tasks.workunit.client.0.smithi043.stdout: 1 102722 9.06 MB/sec warmup 63 sec latency 16.125 ms 2024-03-28T22:39:32.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:32 smithi172 ceph-mon[36566]: pgmap v74: 129 pgs: 129 active+clean; 41 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 339 op/s 2024-03-28T22:39:32.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:32 smithi082 ceph-mon[36471]: pgmap v74: 129 pgs: 129 active+clean; 41 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 339 op/s 2024-03-28T22:39:33.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:32 smithi043 ceph-mon[31823]: pgmap v74: 129 pgs: 129 active+clean; 41 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 339 op/s 2024-03-28T22:39:33.345 INFO:tasks.workunit.client.0.smithi043.stdout: 1 103719 9.00 MB/sec warmup 64 sec latency 13.874 ms 2024-03-28T22:39:34.345 INFO:tasks.workunit.client.0.smithi043.stdout: 1 105669 9.02 MB/sec warmup 65 sec latency 17.666 ms 2024-03-28T22:39:34.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:34 smithi172 ceph-mon[36566]: pgmap v75: 129 pgs: 129 active+clean; 36 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 334 op/s 2024-03-28T22:39:34.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:34 smithi082 ceph-mon[36471]: pgmap v75: 129 pgs: 129 active+clean; 36 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 334 op/s 2024-03-28T22:39:35.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:34 smithi043 ceph-mon[31823]: pgmap v75: 129 pgs: 129 active+clean; 36 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 334 op/s 2024-03-28T22:39:35.345 INFO:tasks.workunit.client.0.smithi043.stdout: 1 106921 9.01 MB/sec warmup 66 sec latency 13.525 ms 2024-03-28T22:39:36.346 INFO:tasks.workunit.client.0.smithi043.stdout: 1 108142 8.95 MB/sec warmup 67 sec latency 69.646 ms 2024-03-28T22:39:36.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:36 smithi172 ceph-mon[36566]: pgmap v76: 129 pgs: 129 active+clean; 36 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 334 op/s 2024-03-28T22:39:36.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:36 smithi082 ceph-mon[36471]: pgmap v76: 129 pgs: 129 active+clean; 36 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 334 op/s 2024-03-28T22:39:37.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:36 smithi043 ceph-mon[31823]: pgmap v76: 129 pgs: 129 active+clean; 36 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 334 op/s 2024-03-28T22:39:37.346 INFO:tasks.workunit.client.0.smithi043.stdout: 1 110086 9.00 MB/sec warmup 68 sec latency 13.984 ms 2024-03-28T22:39:38.346 INFO:tasks.workunit.client.0.smithi043.stdout: 1 110924 8.91 MB/sec warmup 69 sec latency 18.768 ms 2024-03-28T22:39:38.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:38 smithi172 ceph-mon[36566]: pgmap v77: 129 pgs: 129 active+clean; 40 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 199 op/s 2024-03-28T22:39:38.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:38 smithi082 ceph-mon[36471]: pgmap v77: 129 pgs: 129 active+clean; 40 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 199 op/s 2024-03-28T22:39:39.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:38 smithi043 ceph-mon[31823]: pgmap v77: 129 pgs: 129 active+clean; 40 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 199 op/s 2024-03-28T22:39:39.346 INFO:tasks.workunit.client.0.smithi043.stdout: 1 112930 8.95 MB/sec warmup 70 sec latency 11.672 ms 2024-03-28T22:39:40.346 INFO:tasks.workunit.client.0.smithi043.stdout: 1 114121 8.91 MB/sec warmup 71 sec latency 13.058 ms 2024-03-28T22:39:40.535 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.536+0000 7f11ed800640 1 -- 172.21.15.43:0/900243408 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11e80744a0 msgr2=0x7f11e810d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:40.535 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.536+0000 7f11ed800640 1 --2- 172.21.15.43:0/900243408 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11e80744a0 0x7f11e810d830 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f11d4009900 tx=0x7f11d402bcd0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.535 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.536+0000 7f11ed800640 1 -- 172.21.15.43:0/900243408 shutdown_connections 2024-03-28T22:39:40.535 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.536+0000 7f11ed800640 1 --2- 172.21.15.43:0/900243408 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11e810dd70 0x7f11e81101c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.535 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.536+0000 7f11ed800640 1 --2- 172.21.15.43:0/900243408 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11e80744a0 0x7f11e810d830 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.536 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.536+0000 7f11ed800640 1 --2- 172.21.15.43:0/900243408 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f11e8073b80 0x7f11e8073f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.536 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.536+0000 7f11ed800640 1 -- 172.21.15.43:0/900243408 >> 172.21.15.43:0/900243408 conn(0x7f11e80fd0a0 msgr2=0x7f11e80ff4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:40.536 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.536+0000 7f11ed800640 1 -- 172.21.15.43:0/900243408 shutdown_connections 2024-03-28T22:39:40.536 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.536+0000 7f11ed800640 1 -- 172.21.15.43:0/900243408 wait complete. 2024-03-28T22:39:40.537 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.537+0000 7f11ed800640 1 Processor -- start 2024-03-28T22:39:40.537 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.538+0000 7f11ed800640 1 -- start start 2024-03-28T22:39:40.537 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.538+0000 7f11ed800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11e8073b80 0x7f11e819a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:40.537 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.538+0000 7f11ed800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11e80744a0 0x7f11e819a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:40.537 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.538+0000 7f11ed800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f11e810dd70 0x7f11e819e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:40.537 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.538+0000 7f11ed800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f11e8113b20 con 0x7f11e810dd70 2024-03-28T22:39:40.537 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.538+0000 7f11ed800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f11e81139a0 con 0x7f11e8073b80 2024-03-28T22:39:40.537 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.538+0000 7f11ed800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f11e8113ca0 con 0x7f11e80744a0 2024-03-28T22:39:40.537 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.538+0000 7f11e7e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11e8073b80 0x7f11e819a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:40.537 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.538+0000 7f11e7400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11e80744a0 0x7f11e819a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:40.538 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.538+0000 7f11e7e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11e8073b80 0x7f11e819a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:38080/0 (socket says 172.21.15.43:38080) 2024-03-28T22:39:40.538 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.538+0000 7f11e7e00640 1 -- 172.21.15.43:0/1240747648 learned_addr learned my addr 172.21.15.43:0/1240747648 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:40.538 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.538+0000 7f11ece00640 1 --2- 172.21.15.43:0/1240747648 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f11e810dd70 0x7f11e819e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.539+0000 7f11e7e00640 1 -- 172.21.15.43:0/1240747648 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11e80744a0 msgr2=0x7f11e819a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.539+0000 7f11e7e00640 1 --2- 172.21.15.43:0/1240747648 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11e80744a0 0x7f11e819a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.539+0000 7f11e7e00640 1 -- 172.21.15.43:0/1240747648 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f11e810dd70 msgr2=0x7f11e819e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.539+0000 7f11e7e00640 1 --2- 172.21.15.43:0/1240747648 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f11e810dd70 0x7f11e819e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.539+0000 7f11e7e00640 1 -- 172.21.15.43:0/1240747648 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11e819f060 con 0x7f11e8073b80 2024-03-28T22:39:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.539+0000 7f11e7400640 1 --2- 172.21.15.43:0/1240747648 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11e80744a0 0x7f11e819a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.539+0000 7f11e7e00640 1 --2- 172.21.15.43:0/1240747648 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11e8073b80 0x7f11e819a0c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f11d800c920 tx=0x7f11d800cde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.539+0000 7f11e4c00640 1 -- 172.21.15.43:0/1240747648 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d8007c70 con 0x7f11e8073b80 2024-03-28T22:39:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.540+0000 7f11ed800640 1 -- 172.21.15.43:0/1240747648 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11e819f350 con 0x7f11e8073b80 2024-03-28T22:39:40.540 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.540+0000 7f11e4c00640 1 -- 172.21.15.43:0/1240747648 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11d8007e10 con 0x7f11e8073b80 2024-03-28T22:39:40.540 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.540+0000 7f11e4c00640 1 -- 172.21.15.43:0/1240747648 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d8018890 con 0x7f11e8073b80 2024-03-28T22:39:40.540 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.540+0000 7f11ed800640 1 -- 172.21.15.43:0/1240747648 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11e81a6be0 con 0x7f11e8073b80 2024-03-28T22:39:40.543 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.544+0000 7f11f1a83740 1 Processor -- start 2024-03-28T22:39:40.543 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.544+0000 7f11f1a83740 1 -- start start 2024-03-28T22:39:40.543 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.544+0000 7f11e4c00640 1 -- 172.21.15.43:0/1240747648 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f11d8018a70 con 0x7f11e8073b80 2024-03-28T22:39:40.543 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.544+0000 7f11f1a83740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641fbd5bd80 0x5641fbd5c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:40.543 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.544+0000 7f11f1a83740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641fbd5c6a0 0x5641fbd64f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:40.543 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.544+0000 7f11f1a83740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5641fbd65470 0x5641fbd67860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:40.543 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.544+0000 7f11f1a83740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5641fbcb6640 con 0x5641fbd65470 2024-03-28T22:39:40.543 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.544+0000 7f11f1a83740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5641fbc5f810 con 0x5641fbd5bd80 2024-03-28T22:39:40.543 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.544+0000 7f11f1a83740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5641fbcb26e0 con 0x5641fbd5c6a0 2024-03-28T22:39:40.544 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.545+0000 7f11ece00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641fbd5bd80 0x5641fbd5c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:40.544 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.545+0000 7f11e7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641fbd5c6a0 0x5641fbd64f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:40.545 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.545+0000 7f11ece00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641fbd5bd80 0x5641fbd5c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:38086/0 (socket says 172.21.15.43:38086) 2024-03-28T22:39:40.545 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.545+0000 7f11ece00640 1 -- 172.21.15.43:0/2767008687 learned_addr learned my addr 172.21.15.43:0/2767008687 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:40.545 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.545+0000 7f11e7400640 1 --2- 172.21.15.43:0/2767008687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5641fbd65470 0x5641fbd67860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:40.545 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.545+0000 7f11ece00640 1 -- 172.21.15.43:0/2767008687 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641fbd5c6a0 msgr2=0x5641fbd64f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:40.545 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.545+0000 7f11ece00640 1 --2- 172.21.15.43:0/2767008687 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641fbd5c6a0 0x5641fbd64f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.545 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.545+0000 7f11ece00640 1 -- 172.21.15.43:0/2767008687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5641fbd65470 msgr2=0x5641fbd67860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:40.545 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.545+0000 7f11ece00640 1 --2- 172.21.15.43:0/2767008687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5641fbd65470 0x5641fbd67860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.545 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.545+0000 7f11ece00640 1 -- 172.21.15.43:0/2767008687 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641fb9cb210 con 0x5641fbd5bd80 2024-03-28T22:39:40.545 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.545+0000 7f11e7e00640 1 --2- 172.21.15.43:0/2767008687 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641fbd5c6a0 0x5641fbd64f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:39:40.545 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.545+0000 7f11ece00640 1 --2- 172.21.15.43:0/2767008687 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641fbd5bd80 0x5641fbd5c160 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f11e0009f70 tx=0x7f11e0010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b6d16565ff86ef1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:40.545 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.546+0000 7f11dd600640 1 -- 172.21.15.43:0/2767008687 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e0016070 con 0x5641fbd5bd80 2024-03-28T22:39:40.545 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.546+0000 7f11dd600640 1 -- 172.21.15.43:0/2767008687 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11e0004d20 con 0x5641fbd5bd80 2024-03-28T22:39:40.545 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.546+0000 7f11dd600640 1 -- 172.21.15.43:0/2767008687 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e00128f0 con 0x5641fbd5bd80 2024-03-28T22:39:40.545 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.546+0000 7f11e4c00640 1 --2- 172.21.15.43:0/1240747648 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f11cc0a1ee0 0x7f11cc0a43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:40.547 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.546+0000 7f11e7e00640 1 --2- 172.21.15.43:0/1240747648 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f11cc0a1ee0 0x7f11cc0a43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:40.547 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.546+0000 7f11e4c00640 1 -- 172.21.15.43:0/1240747648 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7f11d80ce600 con 0x7f11e8073b80 2024-03-28T22:39:40.547 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.547+0000 7f11f1a83740 1 -- 172.21.15.43:0/2767008687 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641fbd5bd80 msgr2=0x5641fbd5c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:40.547 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.547+0000 7f11f1a83740 1 --2- 172.21.15.43:0/2767008687 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641fbd5bd80 0x5641fbd5c160 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f11e0009f70 tx=0x7f11e0010800 comp rx=0 tx=0).stop 2024-03-28T22:39:40.547 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.547+0000 7f11f1a83740 1 -- 172.21.15.43:0/2767008687 shutdown_connections 2024-03-28T22:39:40.547 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.547+0000 7f11f1a83740 1 --2- 172.21.15.43:0/2767008687 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5641fbd65470 0x5641fbd67860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.547 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.547+0000 7f11f1a83740 1 --2- 172.21.15.43:0/2767008687 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641fbd5c6a0 0x5641fbd64f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.547 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.547+0000 7f11f1a83740 1 --2- 172.21.15.43:0/2767008687 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641fbd5bd80 0x5641fbd5c160 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.547 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.547+0000 7f11f1a83740 1 -- 172.21.15.43:0/2767008687 >> 172.21.15.43:0/2767008687 conn(0x5641fbd566a0 msgr2=0x5641fbd56a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:40.547 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.547+0000 7f11f1a83740 1 -- 172.21.15.43:0/2767008687 shutdown_connections 2024-03-28T22:39:40.547 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.547+0000 7f11f1a83740 1 -- 172.21.15.43:0/2767008687 wait complete. 2024-03-28T22:39:40.547 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.547+0000 7f11e7e00640 1 --2- 172.21.15.43:0/1240747648 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f11cc0a1ee0 0x7f11cc0a43a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f11d80cba60 tx=0x7f11d8000bc0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:40.547 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.548+0000 7f11f1a83740 1 -- start start 2024-03-28T22:39:40.548 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.548+0000 7f11f1a83740 1 Processor -- start 2024-03-28T22:39:40.550 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.549+0000 7f11f1a83740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5641fbd5bd80 0x5641fbd62500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:40.550 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.549+0000 7f11f1a83740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641fbd5c6a0 0x5641fbd62a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:40.550 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.549+0000 7f11f1a83740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641fbd65470 0x5641fbd62f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:40.550 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.549+0000 7f11f1a83740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5641fbd680a0 con 0x5641fbd5bd80 2024-03-28T22:39:40.550 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.549+0000 7f11f1a83740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5641fbc54d00 con 0x5641fbd5c6a0 2024-03-28T22:39:40.550 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.549+0000 7f11f1a83740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5641fbd68220 con 0x5641fbd65470 2024-03-28T22:39:40.551 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.549+0000 7f11ece00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5641fbd5bd80 0x5641fbd62500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:40.551 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.549+0000 7f11e7400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641fbd5c6a0 0x5641fbd62a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:40.551 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.549+0000 7f11ece00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5641fbd5bd80 0x5641fbd62500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47702/0 (socket says 172.21.15.43:47702) 2024-03-28T22:39:40.551 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.549+0000 7f11ece00640 1 -- 172.21.15.43:0/3738320765 learned_addr learned my addr 172.21.15.43:0/3738320765 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:40.551 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.549+0000 7f11f1a83740 10 client.0 _ll_register_callbacks cb 0x5641fbc3f490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:40.551 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.549+0000 7f11ece00640 1 --2- 172.21.15.43:0/3738320765 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641fbd65470 0x5641fbd62f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:40.551 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.549+0000 7f11e7400640 1 -- 172.21.15.43:0/3738320765 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641fbd65470 msgr2=0x5641fbd62f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:40.551 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.549+0000 7f11e7400640 1 --2- 172.21.15.43:0/3738320765 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641fbd65470 0x5641fbd62f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.551 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.549+0000 7f11e7400640 1 -- 172.21.15.43:0/3738320765 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5641fbd5bd80 msgr2=0x5641fbd62500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:40.551 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.549+0000 7f11e7400640 1 --2- 172.21.15.43:0/3738320765 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5641fbd5bd80 0x5641fbd62500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.551 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.549+0000 7f11e7400640 1 -- 172.21.15.43:0/3738320765 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641fba97620 con 0x5641fbd5c6a0 2024-03-28T22:39:40.552 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.550+0000 7f11e7400640 1 --2- 172.21.15.43:0/3738320765 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641fbd5c6a0 0x5641fbd62a40 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f11d402c210 tx=0x7f11d4004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ffa5bf93b70599c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:40.552 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.550+0000 7f11d2000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:39:40.552 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.550+0000 7f11d2000640 1 -- 172.21.15.43:0/3738320765 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d4044070 con 0x5641fbd5c6a0 2024-03-28T22:39:40.552 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.550+0000 7f11d2000640 1 -- 172.21.15.43:0/3738320765 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7f11d402cb30 con 0x5641fbd5c6a0 2024-03-28T22:39:40.552 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.550+0000 7f11f1a83740 1 -- 172.21.15.43:0/3738320765 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11e0004d20 con 0x5641fbd5c6a0 2024-03-28T22:39:40.552 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.550+0000 7f11d2000640 1 -- 172.21.15.43:0/3738320765 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f11d4051050 con 0x5641fbd5c6a0 2024-03-28T22:39:40.552 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.550+0000 7f11f1a83740 10 client.26519 fetch_fsmap learned FSMap version 30 2024-03-28T22:39:40.552 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.550+0000 7f11f1a83740 1 -- 172.21.15.43:0/3738320765 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5641fbd73fb0 con 0x5641fbd5c6a0 2024-03-28T22:39:40.553 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.551+0000 7f11d2000640 1 -- 172.21.15.43:0/3738320765 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11e0004d20 con 0x5641fbd5c6a0 2024-03-28T22:39:40.553 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.551+0000 7f11d2000640 1 -- 172.21.15.43:0/3738320765 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f11d4003710 con 0x5641fbd5c6a0 2024-03-28T22:39:40.553 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.551+0000 7f11f1a83740 10 client.26519 fetch_fsmap finished waiting for FSMap version 30 2024-03-28T22:39:40.553 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.551+0000 7f11f1a83740 10 client.26519 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:39:40.553 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.551+0000 7f11f1a83740 20 client.26519 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:40.553 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.551+0000 7f11f1a83740 1 --2- 172.21.15.43:0/3738320765 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5641fbd75470 0x5641fbd77860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:40.553 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.551+0000 7f11f1a83740 4 client.26519 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:39:40.553 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.551+0000 7f11f1a83740 1 -- 172.21.15.43:0/3738320765 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5641fbcca030 con 0x5641fbd75470 2024-03-28T22:39:40.553 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.551+0000 7f11ece00640 1 --2- 172.21.15.43:0/3738320765 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5641fbd75470 0x5641fbd77860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:40.553 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.552+0000 7f11ece00640 1 --2- 172.21.15.43:0/3738320765 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5641fbd75470 0x5641fbd77860 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebdcdd1f33028c08 server_cookie=c41485e2ae74f497 in_seq=0 out_seq=0 2024-03-28T22:39:40.553 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.552+0000 7f11d2000640 10 client.26519 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:40.554 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.553+0000 7f11d2000640 1 -- 172.21.15.43:0/3738320765 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5641fbcca030 con 0x5641fbd75470 2024-03-28T22:39:40.555 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.553+0000 7f11d2000640 10 client.26519 handle_command_reply: tid=0 2024-03-28T22:39:40.555 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.554+0000 7f11f1a83740 1 client.26519 shutdown 2024-03-28T22:39:40.555 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.554+0000 7f11f1a83740 1 -- 172.21.15.43:0/3738320765 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641fbd5c6a0 msgr2=0x5641fbd62a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:40.555 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.554+0000 7f11f1a83740 1 --2- 172.21.15.43:0/3738320765 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641fbd5c6a0 0x5641fbd62a40 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f11d402c210 tx=0x7f11d4004270 comp rx=0 tx=0).stop 2024-03-28T22:39:40.555 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.554+0000 7f11f1a83740 1 -- 172.21.15.43:0/3738320765 shutdown_connections 2024-03-28T22:39:40.555 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.554+0000 7f11f1a83740 1 --2- 172.21.15.43:0/3738320765 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5641fbd75470 0x5641fbd77860 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.555 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.554+0000 7f11f1a83740 1 --2- 172.21.15.43:0/3738320765 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641fbd5c6a0 0x5641fbd62a40 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.555 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.554+0000 7f11f1a83740 1 --2- 172.21.15.43:0/3738320765 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641fbd65470 0x5641fbd62f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.555 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.554+0000 7f11f1a83740 1 --2- 172.21.15.43:0/3738320765 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5641fbd5bd80 0x5641fbd62500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.555 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.554+0000 7f11f1a83740 1 -- 172.21.15.43:0/3738320765 >> 172.21.15.43:0/3738320765 conn(0x5641fbd566a0 msgr2=0x5641fbd583f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:40.555 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.554+0000 7f11d2000640 0 client.26519 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:40.555 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.554+0000 7f11f1a83740 1 -- 172.21.15.43:0/3738320765 shutdown_connections 2024-03-28T22:39:40.555 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.554+0000 7f11f1a83740 1 -- 172.21.15.43:0/3738320765 wait complete. 2024-03-28T22:39:40.555 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.554+0000 7f11f1a83740 20 client.26519 trim_cache size 0 max 16384 2024-03-28T22:39:40.572 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.573+0000 7f11f1a83740 1 Processor -- start 2024-03-28T22:39:40.572 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.573+0000 7f11f1a83740 1 -- start start 2024-03-28T22:39:40.573 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.573+0000 7f11f1a83740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641fbd5bd80 0x5641fbd64c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:40.573 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.573+0000 7f11f1a83740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641fbd5c6a0 0x5641fbd70e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:40.573 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.573+0000 7f11f1a83740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5641fbd65470 0x5641fbd7d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:40.573 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.573+0000 7f11f1a83740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f11e81ac840 con 0x5641fbd65470 2024-03-28T22:39:40.573 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.573+0000 7f11f1a83740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f11e81ae380 con 0x5641fbd5bd80 2024-03-28T22:39:40.573 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.573+0000 7f11f1a83740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f11e81acf10 con 0x5641fbd5c6a0 2024-03-28T22:39:40.573 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.573+0000 7f11e7400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641fbd5c6a0 0x5641fbd70e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:40.573 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.573+0000 7f11ece00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641fbd5bd80 0x5641fbd64c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:40.573 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.573+0000 7f11e7400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641fbd5c6a0 0x5641fbd70e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:38888/0 (socket says 172.21.15.43:38888) 2024-03-28T22:39:40.573 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.573+0000 7f11e7400640 1 -- 172.21.15.43:0/179930806 learned_addr learned my addr 172.21.15.43:0/179930806 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:40.573 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.574+0000 7f11ece00640 1 --2- 172.21.15.43:0/179930806 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5641fbd65470 0x5641fbd7d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:40.573 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.574+0000 7f11ece00640 1 -- 172.21.15.43:0/179930806 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641fbd5c6a0 msgr2=0x5641fbd70e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:40.573 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.574+0000 7f11ece00640 1 --2- 172.21.15.43:0/179930806 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641fbd5c6a0 0x5641fbd70e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.573 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.574+0000 7f11ece00640 1 -- 172.21.15.43:0/179930806 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5641fbd65470 msgr2=0x5641fbd7d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:40.574 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.574+0000 7f11ece00640 1 --2- 172.21.15.43:0/179930806 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5641fbd65470 0x5641fbd7d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.574 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.574+0000 7f11ece00640 1 -- 172.21.15.43:0/179930806 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641fbd490a0 con 0x5641fbd5bd80 2024-03-28T22:39:40.574 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.574+0000 7f11ece00640 1 --2- 172.21.15.43:0/179930806 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641fbd5bd80 0x5641fbd64c30 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f11e0010d00 tx=0x7f11e0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f95dcb5c1430ae84 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:40.578 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.574+0000 7f11dcc00640 1 -- 172.21.15.43:0/179930806 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e0016040 con 0x5641fbd5bd80 2024-03-28T22:39:40.578 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.574+0000 7f11dcc00640 1 -- 172.21.15.43:0/179930806 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11e0025040 con 0x5641fbd5bd80 2024-03-28T22:39:40.578 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.574+0000 7f11dcc00640 1 -- 172.21.15.43:0/179930806 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e0024040 con 0x5641fbd5bd80 2024-03-28T22:39:40.578 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.575+0000 7f11f1a83740 1 -- 172.21.15.43:0/179930806 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641fbd5bd80 msgr2=0x5641fbd64c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:40.578 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.575+0000 7f11f1a83740 1 --2- 172.21.15.43:0/179930806 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641fbd5bd80 0x5641fbd64c30 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f11e0010d00 tx=0x7f11e0004a40 comp rx=0 tx=0).stop 2024-03-28T22:39:40.579 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.575+0000 7f11f1a83740 1 -- 172.21.15.43:0/179930806 shutdown_connections 2024-03-28T22:39:40.579 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.575+0000 7f11f1a83740 1 --2- 172.21.15.43:0/179930806 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5641fbd65470 0x5641fbd7d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.579 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.575+0000 7f11f1a83740 1 --2- 172.21.15.43:0/179930806 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641fbd5c6a0 0x5641fbd70e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.579 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.575+0000 7f11f1a83740 1 --2- 172.21.15.43:0/179930806 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641fbd5bd80 0x5641fbd64c30 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.579 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.575+0000 7f11f1a83740 1 -- 172.21.15.43:0/179930806 >> 172.21.15.43:0/179930806 conn(0x5641fbd566a0 msgr2=0x5641fbd5d890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:40.579 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.575+0000 7f11f1a83740 1 -- 172.21.15.43:0/179930806 shutdown_connections 2024-03-28T22:39:40.579 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.575+0000 7f11f1a83740 1 -- 172.21.15.43:0/179930806 wait complete. 2024-03-28T22:39:40.579 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.576+0000 7f11f1a83740 1 -- start start 2024-03-28T22:39:40.579 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.576+0000 7f11f1a83740 1 Processor -- start 2024-03-28T22:39:40.579 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.576+0000 7f11f1a83740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641fbd5bd80 0x5641fbd5f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:40.579 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.576+0000 7f11f1a83740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5641fbd5c6a0 0x5641fbd5f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:40.579 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.576+0000 7f11f1a83740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641fbd65470 0x5641fbdda7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:40.579 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.576+0000 7f11f1a83740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f11e81adcb0 con 0x5641fbd5c6a0 2024-03-28T22:39:40.579 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.576+0000 7f11f1a83740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f11e81ad5e0 con 0x5641fbd5bd80 2024-03-28T22:39:40.580 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.576+0000 7f11f1a83740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f11e81aa820 con 0x5641fbd65470 2024-03-28T22:39:40.580 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.576+0000 7f11f1a83740 10 client.0 _ll_register_callbacks cb 0x7f11d402c670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:40.580 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.577+0000 7f11e7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5641fbd5c6a0 0x5641fbd5f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:40.580 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.577+0000 7f11ece00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641fbd5bd80 0x5641fbd5f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:40.580 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.577+0000 7f11e7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5641fbd5c6a0 0x5641fbd5f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:47708/0 (socket says 172.21.15.43:47708) 2024-03-28T22:39:40.580 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.577+0000 7f11e7400640 1 -- 172.21.15.43:0/2280557870 learned_addr learned my addr 172.21.15.43:0/2280557870 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:40.580 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.577+0000 7f11ece00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641fbd5bd80 0x5641fbd5f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:38114/0 (socket says 172.21.15.43:38114) 2024-03-28T22:39:40.580 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.577+0000 7f11ece00640 1 --2- 172.21.15.43:0/2280557870 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641fbd65470 0x5641fbdda7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:40.580 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.577+0000 7f11e7400640 1 -- 172.21.15.43:0/2280557870 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641fbd65470 msgr2=0x5641fbdda7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:40.580 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.577+0000 7f11e7400640 1 --2- 172.21.15.43:0/2280557870 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641fbd65470 0x5641fbdda7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.580 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.577+0000 7f11e7400640 1 -- 172.21.15.43:0/2280557870 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641fbd5bd80 msgr2=0x5641fbd5f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:40.580 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.577+0000 7f11e7400640 1 --2- 172.21.15.43:0/2280557870 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641fbd5bd80 0x5641fbd5f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.580 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.577+0000 7f11e7400640 1 -- 172.21.15.43:0/2280557870 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641fbd60280 con 0x5641fbd5c6a0 2024-03-28T22:39:40.580 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.577+0000 7f11ece00640 1 --2- 172.21.15.43:0/2280557870 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641fbd5bd80 0x5641fbd5f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:40.580 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.577+0000 7f11e7400640 1 --2- 172.21.15.43:0/2280557870 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5641fbd5c6a0 0x5641fbd5f980 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f11d402c210 tx=0x7f11d4003f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4819a98ff95c192 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:40.581 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.577+0000 7f11d2000640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:39:40.581 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.578+0000 7f11d2000640 1 -- 172.21.15.43:0/2280557870 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d405a060 con 0x5641fbd5c6a0 2024-03-28T22:39:40.581 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.578+0000 7f11d2000640 1 -- 172.21.15.43:0/2280557870 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7f11d40035f0 con 0x5641fbd5c6a0 2024-03-28T22:39:40.581 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.578+0000 7f11f1a83740 1 -- 172.21.15.43:0/2280557870 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11e0025040 con 0x5641fbd5c6a0 2024-03-28T22:39:40.581 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.578+0000 7f11d2000640 1 -- 172.21.15.43:0/2280557870 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f11d4061050 con 0x5641fbd5c6a0 2024-03-28T22:39:40.581 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.578+0000 7f11f1a83740 10 client.16335 fetch_fsmap learned FSMap version 30 2024-03-28T22:39:40.581 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.578+0000 7f11f1a83740 1 -- 172.21.15.43:0/2280557870 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5641fbddbe20 con 0x5641fbd5c6a0 2024-03-28T22:39:40.581 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.578+0000 7f11d2000640 1 -- 172.21.15.43:0/2280557870 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11e0025040 con 0x5641fbd5c6a0 2024-03-28T22:39:40.581 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.579+0000 7f11d2000640 1 -- 172.21.15.43:0/2280557870 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f11d4034440 con 0x5641fbd5c6a0 2024-03-28T22:39:40.581 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.579+0000 7f11f1a83740 10 client.16335 fetch_fsmap finished waiting for FSMap version 30 2024-03-28T22:39:40.581 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.579+0000 7f11f1a83740 10 client.16335 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:39:40.581 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.579+0000 7f11f1a83740 20 client.16335 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:40.581 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.579+0000 7f11f1a83740 1 --2- 172.21.15.43:0/2280557870 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5641fbddc540 0x5641fbdde960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:40.581 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.579+0000 7f11f1a83740 4 client.16335 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T22:39:40.581 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.579+0000 7f11f1a83740 1 -- 172.21.15.43:0/2280557870 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5641fbcc3b80 con 0x5641fbddc540 2024-03-28T22:39:40.582 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.579+0000 7f11ece00640 1 --2- 172.21.15.43:0/2280557870 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5641fbddc540 0x5641fbdde960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:40.582 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.580+0000 7f11ece00640 1 --2- 172.21.15.43:0/2280557870 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5641fbddc540 0x5641fbdde960 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62f4ef367137755b server_cookie=85eb0c25fd2b49e5 in_seq=0 out_seq=0 2024-03-28T22:39:40.582 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.580+0000 7f11d2000640 10 client.16335 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:40.781 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.782+0000 7f11d2000640 1 -- 172.21.15.43:0/2280557870 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5641fbcc3b80 con 0x5641fbddc540 2024-03-28T22:39:40.781 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.782+0000 7f11d2000640 10 client.16335 handle_command_reply: tid=0 2024-03-28T22:39:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.782+0000 7f11f1a83740 1 client.16335 shutdown 2024-03-28T22:39:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.782+0000 7f11f1a83740 1 -- 172.21.15.43:0/2280557870 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5641fbd5c6a0 msgr2=0x5641fbd5f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.782+0000 7f11f1a83740 1 --2- 172.21.15.43:0/2280557870 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5641fbd5c6a0 0x5641fbd5f980 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f11d402c210 tx=0x7f11d4003f30 comp rx=0 tx=0).stop 2024-03-28T22:39:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.782+0000 7f11f1a83740 1 -- 172.21.15.43:0/2280557870 shutdown_connections 2024-03-28T22:39:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.782+0000 7f11f1a83740 1 --2- 172.21.15.43:0/2280557870 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641fbd65470 0x5641fbdda7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.782+0000 7f11f1a83740 1 --2- 172.21.15.43:0/2280557870 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5641fbd5c6a0 0x5641fbd5f980 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.782+0000 7f11f1a83740 1 --2- 172.21.15.43:0/2280557870 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5641fbddc540 0x5641fbdde960 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.782+0000 7f11f1a83740 1 --2- 172.21.15.43:0/2280557870 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641fbd5bd80 0x5641fbd5f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.782+0000 7f11f1a83740 1 -- 172.21.15.43:0/2280557870 >> 172.21.15.43:0/2280557870 conn(0x5641fbd566a0 msgr2=0x5641fbd57980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.782+0000 7f11d2000640 0 client.16335 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.783+0000 7f11f1a83740 1 -- 172.21.15.43:0/2280557870 shutdown_connections 2024-03-28T22:39:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.783+0000 7f11f1a83740 1 -- 172.21.15.43:0/2280557870 wait complete. 2024-03-28T22:39:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.783+0000 7f11f1a83740 20 client.16335 trim_cache size 0 max 16384 2024-03-28T22:39:40.783 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:{ 2024-03-28T22:39:40.783 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "message": "", 2024-03-28T22:39:40.783 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "return_code": 0 2024-03-28T22:39:40.783 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:} 2024-03-28T22:39:40.784 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.784+0000 7f11ed800640 1 -- 172.21.15.43:0/1240747648 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f11cc0a1ee0 msgr2=0x7f11cc0a43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:40.784 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.784+0000 7f11ed800640 1 --2- 172.21.15.43:0/1240747648 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f11cc0a1ee0 0x7f11cc0a43a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f11d80cba60 tx=0x7f11d8000bc0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.784 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.784+0000 7f11ed800640 1 -- 172.21.15.43:0/1240747648 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11e8073b80 msgr2=0x7f11e819a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:40.784 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.784+0000 7f11ed800640 1 --2- 172.21.15.43:0/1240747648 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11e8073b80 0x7f11e819a0c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f11d800c920 tx=0x7f11d800cde0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.784 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.784+0000 7f11ed800640 1 -- 172.21.15.43:0/1240747648 shutdown_connections 2024-03-28T22:39:40.784 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.784+0000 7f11ed800640 1 --2- 172.21.15.43:0/1240747648 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f11cc0a1ee0 0x7f11cc0a43a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.784 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.784+0000 7f11ed800640 1 --2- 172.21.15.43:0/1240747648 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f11e810dd70 0x7f11e819e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.784 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.784+0000 7f11ed800640 1 --2- 172.21.15.43:0/1240747648 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11e80744a0 0x7f11e819a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.784 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.784+0000 7f11ed800640 1 --2- 172.21.15.43:0/1240747648 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11e8073b80 0x7f11e819a0c0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:40.785 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.784+0000 7f11ed800640 1 -- 172.21.15.43:0/1240747648 >> 172.21.15.43:0/1240747648 conn(0x7f11e80fd0a0 msgr2=0x7f11e80fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:40.785 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.784+0000 7f11ed800640 1 -- 172.21.15.43:0/1240747648 shutdown_connections 2024-03-28T22:39:40.785 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:40.784+0000 7f11ed800640 1 -- 172.21.15.43:0/1240747648 wait complete. 2024-03-28T22:39:40.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:40 smithi172 ceph-mon[36566]: pgmap v78: 129 pgs: 129 active+clean; 40 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 169 op/s 2024-03-28T22:39:40.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:40 smithi082 ceph-mon[36471]: pgmap v78: 129 pgs: 129 active+clean; 40 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 169 op/s 2024-03-28T22:39:41.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:40 smithi043 ceph-mon[31823]: pgmap v78: 129 pgs: 129 active+clean; 40 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 169 op/s 2024-03-28T22:39:41.346 INFO:tasks.workunit.client.0.smithi043.stdout: 1 115418 8.87 MB/sec warmup 72 sec latency 130.992 ms 2024-03-28T22:39:42.346 INFO:tasks.workunit.client.0.smithi043.stdout: 1 117246 8.91 MB/sec warmup 73 sec latency 29.505 ms 2024-03-28T22:39:42.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:42 smithi172 ceph-mon[36566]: pgmap v79: 129 pgs: 129 active+clean; 40 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 165 op/s 2024-03-28T22:39:42.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:42 smithi082 ceph-mon[36471]: pgmap v79: 129 pgs: 129 active+clean; 40 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 165 op/s 2024-03-28T22:39:43.019 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:42 smithi043 ceph-mon[31823]: pgmap v79: 129 pgs: 129 active+clean; 40 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 165 op/s 2024-03-28T22:39:43.346 INFO:tasks.workunit.client.0.smithi043.stdout: 1 118068 8.82 MB/sec warmup 74 sec latency 21.444 ms 2024-03-28T22:39:44.347 INFO:tasks.workunit.client.0.smithi043.stdout: 1 120085 8.86 MB/sec warmup 75 sec latency 12.081 ms 2024-03-28T22:39:44.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:44 smithi172 ceph-mon[36566]: pgmap v80: 129 pgs: 129 active+clean; 39 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 364 op/s 2024-03-28T22:39:44.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:44 smithi082 ceph-mon[36471]: pgmap v80: 129 pgs: 129 active+clean; 39 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 364 op/s 2024-03-28T22:39:45.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:44 smithi043 ceph-mon[31823]: pgmap v80: 129 pgs: 129 active+clean; 39 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 364 op/s 2024-03-28T22:39:45.347 INFO:tasks.workunit.client.0.smithi043.stdout: 1 121247 8.83 MB/sec warmup 76 sec latency 11.120 ms 2024-03-28T22:39:46.347 INFO:tasks.workunit.client.0.smithi043.stdout: 1 122774 8.79 MB/sec warmup 77 sec latency 58.260 ms 2024-03-28T22:39:46.469 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:39:46.469 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:39:46.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.710+0000 7f9a8a400640 1 -- 172.21.15.43:0/1936252993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a841077a0 msgr2=0x7f9a84107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:46.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.710+0000 7f9a8a400640 1 --2- 172.21.15.43:0/1936252993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a841077a0 0x7f9a84107b80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9a74009930 tx=0x7f9a7402bc30 comp rx=0 tx=0).stop 2024-03-28T22:39:46.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.711+0000 7f9a8a400640 1 -- 172.21.15.43:0/1936252993 shutdown_connections 2024-03-28T22:39:46.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.711+0000 7f9a8a400640 1 --2- 172.21.15.43:0/1936252993 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a84068950 0x7f9a84068dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.711+0000 7f9a8a400640 1 --2- 172.21.15.43:0/1936252993 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a84069310 0x7f9a8410e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.711+0000 7f9a8a400640 1 --2- 172.21.15.43:0/1936252993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a841077a0 0x7f9a84107b80 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.711+0000 7f9a8a400640 1 -- 172.21.15.43:0/1936252993 >> 172.21.15.43:0/1936252993 conn(0x7f9a84077750 msgr2=0x7f9a84077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:46.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.711+0000 7f9a8a400640 1 -- 172.21.15.43:0/1936252993 shutdown_connections 2024-03-28T22:39:46.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.711+0000 7f9a8a400640 1 -- 172.21.15.43:0/1936252993 wait complete. 2024-03-28T22:39:46.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.712+0000 7f9a8a400640 1 Processor -- start 2024-03-28T22:39:46.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.712+0000 7f9a8a400640 1 -- start start 2024-03-28T22:39:46.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.712+0000 7f9a8a400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a84068950 0x7f9a84100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:46.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.712+0000 7f9a8a400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a84069310 0x7f9a84100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:46.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.712+0000 7f9a8a400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a841077a0 0x7f9a84101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:46.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.712+0000 7f9a8a400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9a84111b00 con 0x7f9a841077a0 2024-03-28T22:39:46.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.712+0000 7f9a8a400640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9a84111980 con 0x7f9a84068950 2024-03-28T22:39:46.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.712+0000 7f9a8a400640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9a84111c80 con 0x7f9a84069310 2024-03-28T22:39:46.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.713+0000 7f9a83400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a84068950 0x7f9a84100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:46.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.713+0000 7f9a83400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a84068950 0x7f9a84100610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52514/0 (socket says 172.21.15.43:52514) 2024-03-28T22:39:46.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.713+0000 7f9a83400640 1 -- 172.21.15.43:0/3897791934 learned_addr learned my addr 172.21.15.43:0/3897791934 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:46.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.713+0000 7f9a82a00640 1 --2- 172.21.15.43:0/3897791934 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a84069310 0x7f9a84100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:46.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.713+0000 7f9a83e00640 1 --2- 172.21.15.43:0/3897791934 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a841077a0 0x7f9a84101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:46.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.713+0000 7f9a83400640 1 -- 172.21.15.43:0/3897791934 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a84069310 msgr2=0x7f9a84100b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:46.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.713+0000 7f9a83400640 1 --2- 172.21.15.43:0/3897791934 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a84069310 0x7f9a84100b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.713+0000 7f9a83400640 1 -- 172.21.15.43:0/3897791934 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a841077a0 msgr2=0x7f9a84101090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:46.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.713+0000 7f9a83400640 1 --2- 172.21.15.43:0/3897791934 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a841077a0 0x7f9a84101090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.713+0000 7f9a83400640 1 -- 172.21.15.43:0/3897791934 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a841aa940 con 0x7f9a84068950 2024-03-28T22:39:46.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.713+0000 7f9a83400640 1 --2- 172.21.15.43:0/3897791934 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a84068950 0x7f9a84100610 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f9a74009900 tx=0x7f9a74004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:46.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.714+0000 7f9a7be00640 1 -- 172.21.15.43:0/3897791934 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a74041070 con 0x7f9a84068950 2024-03-28T22:39:46.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.714+0000 7f9a8a400640 1 -- 172.21.15.43:0/3897791934 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a841aab10 con 0x7f9a84068950 2024-03-28T22:39:46.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.714+0000 7f9a7be00640 1 -- 172.21.15.43:0/3897791934 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a740054e0 con 0x7f9a84068950 2024-03-28T22:39:46.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.714+0000 7f9a7be00640 1 -- 172.21.15.43:0/3897791934 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a74003c20 con 0x7f9a84068950 2024-03-28T22:39:46.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.714+0000 7f9a8a400640 1 -- 172.21.15.43:0/3897791934 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a841ab050 con 0x7f9a84068950 2024-03-28T22:39:46.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.717+0000 7f9a7be00640 1 -- 172.21.15.43:0/3897791934 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f9a74044070 con 0x7f9a84068950 2024-03-28T22:39:46.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.717+0000 7f9a8d174740 1 Processor -- start 2024-03-28T22:39:46.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.718+0000 7f9a8d174740 1 -- start start 2024-03-28T22:39:46.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.718+0000 7f9a8d174740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555fb8693bc0 0x555fb8692910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:46.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.718+0000 7f9a8d174740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555fb8692ee0 0x555fb8693360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:46.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.718+0000 7f9a8d174740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555fb869c250 0x555fb869e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:46.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.718+0000 7f9a8d174740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x555fb85f7100 con 0x555fb869c250 2024-03-28T22:39:46.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.718+0000 7f9a8d174740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x555fb859fba0 con 0x555fb8693bc0 2024-03-28T22:39:46.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.718+0000 7f9a8d174740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x555fb85f31a0 con 0x555fb8692ee0 2024-03-28T22:39:46.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.718+0000 7f9a83400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555fb8692ee0 0x555fb8693360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:46.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.718+0000 7f9a83e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555fb8693bc0 0x555fb8692910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:46.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.718+0000 7f9a82a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555fb869c250 0x555fb869e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:46.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.718+0000 7f9a83400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555fb8692ee0 0x555fb8693360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:55922/0 (socket says 172.21.15.43:55922) 2024-03-28T22:39:46.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.718+0000 7f9a83e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555fb8693bc0 0x555fb8692910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52524/0 (socket says 172.21.15.43:52524) 2024-03-28T22:39:46.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.718+0000 7f9a83400640 1 -- 172.21.15.43:0/2534526498 learned_addr learned my addr 172.21.15.43:0/2534526498 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:46.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.719+0000 7f9a83400640 1 -- 172.21.15.43:0/2534526498 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555fb8693bc0 msgr2=0x555fb8692910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:46.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.719+0000 7f9a83400640 1 --2- 172.21.15.43:0/2534526498 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555fb8693bc0 0x555fb8692910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.719+0000 7f9a83400640 1 -- 172.21.15.43:0/2534526498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555fb869c250 msgr2=0x555fb869e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:46.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.719+0000 7f9a83400640 1 --2- 172.21.15.43:0/2534526498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555fb869c250 0x555fb869e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.719+0000 7f9a83400640 1 -- 172.21.15.43:0/2534526498 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555fb85d01a0 con 0x555fb8692ee0 2024-03-28T22:39:46.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.719+0000 7f9a83e00640 1 --2- 172.21.15.43:0/2534526498 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555fb8693bc0 0x555fb8692910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:46.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.719+0000 7f9a7be00640 1 --2- 172.21.15.43:0/3897791934 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9a680a1ef0 0x7f9a680a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:46.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.719+0000 7f9a83400640 1 --2- 172.21.15.43:0/2534526498 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555fb8692ee0 0x555fb8693360 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9a740ed480 tx=0x7f9a740f8220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=514dc76ecd8acd15 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:46.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.720+0000 7f9a7be00640 1 -- 172.21.15.43:0/3897791934 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7f9a740f00e0 con 0x7f9a84068950 2024-03-28T22:39:46.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.720+0000 7f9a83e00640 1 --2- 172.21.15.43:0/3897791934 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9a680a1ef0 0x7f9a680a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:46.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.720+0000 7f9a78c00640 1 -- 172.21.15.43:0/2534526498 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a740b8a80 con 0x555fb8692ee0 2024-03-28T22:39:46.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.720+0000 7f9a78c00640 1 -- 172.21.15.43:0/2534526498 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a74004c80 con 0x555fb8692ee0 2024-03-28T22:39:46.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.720+0000 7f9a78c00640 1 -- 172.21.15.43:0/2534526498 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a740f3c80 con 0x555fb8692ee0 2024-03-28T22:39:46.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.721+0000 7f9a8d174740 1 -- 172.21.15.43:0/2534526498 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555fb8692ee0 msgr2=0x555fb8693360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:46.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.721+0000 7f9a83e00640 1 --2- 172.21.15.43:0/3897791934 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9a680a1ef0 0x7f9a680a43b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c005fd0 tx=0x7f9a7c009500 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:46.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.721+0000 7f9a8d174740 1 --2- 172.21.15.43:0/2534526498 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555fb8692ee0 0x555fb8693360 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9a740ed480 tx=0x7f9a740f8220 comp rx=0 tx=0).stop 2024-03-28T22:39:46.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.721+0000 7f9a8d174740 1 -- 172.21.15.43:0/2534526498 shutdown_connections 2024-03-28T22:39:46.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.721+0000 7f9a8d174740 1 --2- 172.21.15.43:0/2534526498 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555fb869c250 0x555fb869e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.721+0000 7f9a8d174740 1 --2- 172.21.15.43:0/2534526498 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555fb8692ee0 0x555fb8693360 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.721+0000 7f9a8d174740 1 --2- 172.21.15.43:0/2534526498 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555fb8693bc0 0x555fb8692910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.721+0000 7f9a8d174740 1 -- 172.21.15.43:0/2534526498 >> 172.21.15.43:0/2534526498 conn(0x555fb8659a10 msgr2=0x555fb868d930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:46.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.721+0000 7f9a8d174740 1 -- 172.21.15.43:0/2534526498 shutdown_connections 2024-03-28T22:39:46.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.721+0000 7f9a8d174740 1 -- 172.21.15.43:0/2534526498 wait complete. 2024-03-28T22:39:46.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.721+0000 7f9a8d174740 1 -- start start 2024-03-28T22:39:46.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.722+0000 7f9a8d174740 1 Processor -- start 2024-03-28T22:39:46.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.722+0000 7f9a8d174740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555fb8692ee0 0x555fb86991e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:46.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.722+0000 7f9a8d174740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555fb8693bc0 0x555fb8699720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:46.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.722+0000 7f9a8d174740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555fb869c250 0x555fb8699c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:46.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.722+0000 7f9a8d174740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x555fb85f31a0 con 0x555fb869c250 2024-03-28T22:39:46.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.722+0000 7f9a8d174740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x555fb8595250 con 0x555fb8693bc0 2024-03-28T22:39:46.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.722+0000 7f9a8d174740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x555fb869ee80 con 0x555fb8692ee0 2024-03-28T22:39:46.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.722+0000 7f9a8d174740 10 client.0 _ll_register_callbacks cb 0x555fb830abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:46.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.722+0000 7f9a83400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555fb8692ee0 0x555fb86991e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:46.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.722+0000 7f9a83e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555fb869c250 0x555fb8699c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:46.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.722+0000 7f9a82a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555fb8693bc0 0x555fb8699720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:46.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.722+0000 7f9a83400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555fb8692ee0 0x555fb86991e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:55924/0 (socket says 172.21.15.43:55924) 2024-03-28T22:39:46.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.723+0000 7f9a82a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555fb8693bc0 0x555fb8699720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52528/0 (socket says 172.21.15.43:52528) 2024-03-28T22:39:46.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.723+0000 7f9a83400640 1 -- 172.21.15.43:0/4007898718 learned_addr learned my addr 172.21.15.43:0/4007898718 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:46.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.723+0000 7f9a83400640 1 -- 172.21.15.43:0/4007898718 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555fb8693bc0 msgr2=0x555fb8699720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:46.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.723+0000 7f9a83400640 1 --2- 172.21.15.43:0/4007898718 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555fb8693bc0 0x555fb8699720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.723+0000 7f9a83400640 1 -- 172.21.15.43:0/4007898718 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555fb869c250 msgr2=0x555fb8699c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:46.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.723+0000 7f9a83400640 1 --2- 172.21.15.43:0/4007898718 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555fb869c250 0x555fb8699c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.723+0000 7f9a83400640 1 -- 172.21.15.43:0/4007898718 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555fb85b1960 con 0x555fb8692ee0 2024-03-28T22:39:46.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.723+0000 7f9a83400640 1 --2- 172.21.15.43:0/4007898718 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555fb8692ee0 0x555fb86991e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f9a740f8720 tx=0x7f9a74004ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e53ecb25664c4d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:46.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.723+0000 7f9a6d600640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:39:46.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.723+0000 7f9a6d600640 1 -- 172.21.15.43:0/4007898718 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a7404b030 con 0x555fb8692ee0 2024-03-28T22:39:46.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.723+0000 7f9a6d600640 1 -- 172.21.15.43:0/4007898718 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7f9a740a7070 con 0x555fb8692ee0 2024-03-28T22:39:46.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.723+0000 7f9a8d174740 1 -- 172.21.15.43:0/4007898718 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a74004c80 con 0x555fb8692ee0 2024-03-28T22:39:46.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.724+0000 7f9a6d600640 1 -- 172.21.15.43:0/4007898718 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9a740bc140 con 0x555fb8692ee0 2024-03-28T22:39:46.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.724+0000 7f9a8d174740 10 client.26488 fetch_fsmap learned FSMap version 30 2024-03-28T22:39:46.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.724+0000 7f9a8d174740 1 -- 172.21.15.43:0/4007898718 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555fb83d57d0 con 0x555fb8692ee0 2024-03-28T22:39:46.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.724+0000 7f9a6d600640 1 -- 172.21.15.43:0/4007898718 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a740da070 con 0x555fb8692ee0 2024-03-28T22:39:46.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.724+0000 7f9a6d600640 1 -- 172.21.15.43:0/4007898718 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f9a740bca40 con 0x555fb8692ee0 2024-03-28T22:39:46.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.724+0000 7f9a8d174740 10 client.26488 fetch_fsmap finished waiting for FSMap version 30 2024-03-28T22:39:46.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.724+0000 7f9a8d174740 10 client.26488 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:39:46.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.724+0000 7f9a8d174740 20 client.26488 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:46.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.724+0000 7f9a8d174740 1 --2- 172.21.15.43:0/4007898718 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x555fb86ac1a0 0x555fb86ae590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:46.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.724+0000 7f9a8d174740 4 client.26488 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:39:46.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.724+0000 7f9a8d174740 1 -- 172.21.15.43:0/4007898718 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555fb8604640 con 0x555fb86ac1a0 2024-03-28T22:39:46.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.724+0000 7f9a82a00640 1 --2- 172.21.15.43:0/4007898718 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x555fb86ac1a0 0x555fb86ae590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:46.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.725+0000 7f9a82a00640 1 --2- 172.21.15.43:0/4007898718 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x555fb86ac1a0 0x555fb86ae590 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f368984f6230814b server_cookie=d744252918ba2626 in_seq=0 out_seq=0 2024-03-28T22:39:46.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.725+0000 7f9a6d600640 10 client.26488 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:46.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.726+0000 7f9a6d600640 1 -- 172.21.15.43:0/4007898718 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555fb8604640 con 0x555fb86ac1a0 2024-03-28T22:39:46.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.726+0000 7f9a6d600640 10 client.26488 handle_command_reply: tid=0 2024-03-28T22:39:46.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.726+0000 7f9a8d174740 1 client.26488 shutdown 2024-03-28T22:39:46.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.726+0000 7f9a8d174740 1 -- 172.21.15.43:0/4007898718 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555fb8692ee0 msgr2=0x555fb86991e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:46.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.726+0000 7f9a8d174740 1 --2- 172.21.15.43:0/4007898718 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555fb8692ee0 0x555fb86991e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f9a740f8720 tx=0x7f9a74004ee0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.726+0000 7f9a8d174740 1 -- 172.21.15.43:0/4007898718 shutdown_connections 2024-03-28T22:39:46.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.726+0000 7f9a8d174740 1 --2- 172.21.15.43:0/4007898718 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x555fb86ac1a0 0x555fb86ae590 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.726+0000 7f9a8d174740 1 --2- 172.21.15.43:0/4007898718 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555fb8693bc0 0x555fb8699720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.726+0000 7f9a8d174740 1 --2- 172.21.15.43:0/4007898718 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555fb869c250 0x555fb8699c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.726+0000 7f9a8d174740 1 --2- 172.21.15.43:0/4007898718 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555fb8692ee0 0x555fb86991e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.727+0000 7f9a8d174740 1 -- 172.21.15.43:0/4007898718 >> 172.21.15.43:0/4007898718 conn(0x555fb8659a10 msgr2=0x555fb868f680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:46.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.727+0000 7f9a6d600640 0 client.26488 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:46.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.727+0000 7f9a8d174740 1 -- 172.21.15.43:0/4007898718 shutdown_connections 2024-03-28T22:39:46.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.727+0000 7f9a8d174740 1 -- 172.21.15.43:0/4007898718 wait complete. 2024-03-28T22:39:46.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.727+0000 7f9a8d174740 20 client.26488 trim_cache size 0 max 16384 2024-03-28T22:39:46.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.748+0000 7f9a8d174740 1 Processor -- start 2024-03-28T22:39:46.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.748+0000 7f9a8d174740 1 -- start start 2024-03-28T22:39:46.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.748+0000 7f9a8d174740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555fb8692ee0 0x555fb869b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:46.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.748+0000 7f9a8d174740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555fb8693bc0 0x555fb8698380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:46.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.748+0000 7f9a8d174740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555fb869c250 0x555fb86a7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:46.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.748+0000 7f9a8d174740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9a841b5290 con 0x555fb8692ee0 2024-03-28T22:39:46.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.748+0000 7f9a8d174740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9a841b6dd0 con 0x555fb869c250 2024-03-28T22:39:46.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.748+0000 7f9a8d174740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9a841b5960 con 0x555fb8693bc0 2024-03-28T22:39:46.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.748+0000 7f9a82a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555fb8693bc0 0x555fb8698380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:46.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.748+0000 7f9a83e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555fb869c250 0x555fb86a7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:46.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.748+0000 7f9a83400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555fb8692ee0 0x555fb869b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:46.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.748+0000 7f9a82a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555fb8693bc0 0x555fb8698380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:55938/0 (socket says 172.21.15.43:55938) 2024-03-28T22:39:46.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.748+0000 7f9a83e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555fb869c250 0x555fb86a7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52530/0 (socket says 172.21.15.43:52530) 2024-03-28T22:39:46.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.748+0000 7f9a82a00640 1 -- 172.21.15.43:0/155019881 learned_addr learned my addr 172.21.15.43:0/155019881 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:46.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.749+0000 7f9a82a00640 1 -- 172.21.15.43:0/155019881 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555fb869c250 msgr2=0x555fb86a7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:46.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.749+0000 7f9a82a00640 1 --2- 172.21.15.43:0/155019881 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555fb869c250 0x555fb86a7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.749+0000 7f9a82a00640 1 -- 172.21.15.43:0/155019881 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555fb8692ee0 msgr2=0x555fb869b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:46.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.749+0000 7f9a82a00640 1 --2- 172.21.15.43:0/155019881 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555fb8692ee0 0x555fb869b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.749+0000 7f9a82a00640 1 -- 172.21.15.43:0/155019881 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555fb830b3f0 con 0x555fb8693bc0 2024-03-28T22:39:46.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.749+0000 7f9a82a00640 1 --2- 172.21.15.43:0/155019881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555fb8693bc0 0x555fb8698380 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f9a70004700 tx=0x7f9a70005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8eb94ea814d31281 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:46.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.749+0000 7f9a6fe00640 1 -- 172.21.15.43:0/155019881 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a7001f310 con 0x555fb8693bc0 2024-03-28T22:39:46.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.749+0000 7f9a6fe00640 1 -- 172.21.15.43:0/155019881 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a7001e420 con 0x555fb8693bc0 2024-03-28T22:39:46.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.749+0000 7f9a6fe00640 1 -- 172.21.15.43:0/155019881 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a7001d5e0 con 0x555fb8693bc0 2024-03-28T22:39:46.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.750+0000 7f9a8d174740 1 -- 172.21.15.43:0/155019881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555fb8693bc0 msgr2=0x555fb8698380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:46.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.750+0000 7f9a8d174740 1 --2- 172.21.15.43:0/155019881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555fb8693bc0 0x555fb8698380 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f9a70004700 tx=0x7f9a70005f90 comp rx=0 tx=0).stop 2024-03-28T22:39:46.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.750+0000 7f9a8d174740 1 -- 172.21.15.43:0/155019881 shutdown_connections 2024-03-28T22:39:46.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.750+0000 7f9a8d174740 1 --2- 172.21.15.43:0/155019881 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555fb869c250 0x555fb86a7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.750+0000 7f9a8d174740 1 --2- 172.21.15.43:0/155019881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555fb8693bc0 0x555fb8698380 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.750+0000 7f9a8d174740 1 --2- 172.21.15.43:0/155019881 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555fb8692ee0 0x555fb869b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.750+0000 7f9a8d174740 1 -- 172.21.15.43:0/155019881 >> 172.21.15.43:0/155019881 conn(0x555fb8659a10 msgr2=0x555fb868c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:46.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.750+0000 7f9a8d174740 1 -- 172.21.15.43:0/155019881 shutdown_connections 2024-03-28T22:39:46.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.750+0000 7f9a8d174740 1 -- 172.21.15.43:0/155019881 wait complete. 2024-03-28T22:39:46.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.751+0000 7f9a8d174740 1 -- start start 2024-03-28T22:39:46.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.751+0000 7f9a8d174740 1 Processor -- start 2024-03-28T22:39:46.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.751+0000 7f9a8d174740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555fb8692ee0 0x555fb86a4360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:46.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.752+0000 7f9a8d174740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555fb8693bc0 0x555fb86964a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:46.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.752+0000 7f9a8d174740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555fb869c250 0x555fb86969e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:46.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.752+0000 7f9a8d174740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9a841b6700 con 0x555fb8693bc0 2024-03-28T22:39:46.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.752+0000 7f9a8d174740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9a841b6030 con 0x555fb869c250 2024-03-28T22:39:46.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.752+0000 7f9a8d174740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9a841b3270 con 0x555fb8692ee0 2024-03-28T22:39:46.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.752+0000 7f9a8d174740 10 client.0 _ll_register_callbacks cb 0x7f9a70009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:46.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.752+0000 7f9a83400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555fb8692ee0 0x555fb86a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:46.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.752+0000 7f9a83400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555fb8692ee0 0x555fb86a4360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:55942/0 (socket says 172.21.15.43:55942) 2024-03-28T22:39:46.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.752+0000 7f9a83400640 1 -- 172.21.15.43:0/3331104052 learned_addr learned my addr 172.21.15.43:0/3331104052 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:46.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.752+0000 7f9a82a00640 1 --2- 172.21.15.43:0/3331104052 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555fb8693bc0 0x555fb86964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:46.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.752+0000 7f9a83e00640 1 --2- 172.21.15.43:0/3331104052 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555fb869c250 0x555fb86969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:46.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.752+0000 7f9a83400640 1 -- 172.21.15.43:0/3331104052 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555fb869c250 msgr2=0x555fb86969e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:46.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.752+0000 7f9a83400640 1 --2- 172.21.15.43:0/3331104052 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555fb869c250 0x555fb86969e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.752+0000 7f9a83400640 1 -- 172.21.15.43:0/3331104052 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555fb8693bc0 msgr2=0x555fb86964a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:46.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.752+0000 7f9a83400640 1 --2- 172.21.15.43:0/3331104052 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555fb8693bc0 0x555fb86964a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.752+0000 7f9a83400640 1 -- 172.21.15.43:0/3331104052 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555fb82cb0f0 con 0x555fb8692ee0 2024-03-28T22:39:46.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.752+0000 7f9a82a00640 1 --2- 172.21.15.43:0/3331104052 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555fb8693bc0 0x555fb86964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:39:46.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.753+0000 7f9a83400640 1 --2- 172.21.15.43:0/3331104052 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555fb8692ee0 0x555fb86a4360 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9a74009610 tx=0x7f9a740a72e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f7f61b6e9d397eb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:46.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.753+0000 7f9a6d600640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:39:46.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.753+0000 7f9a6d600640 1 -- 172.21.15.43:0/3331104052 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a7404b070 con 0x555fb8692ee0 2024-03-28T22:39:46.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.753+0000 7f9a6d600640 1 -- 172.21.15.43:0/3331104052 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7f9a740c4af0 con 0x555fb8692ee0 2024-03-28T22:39:46.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.753+0000 7f9a8d174740 1 -- 172.21.15.43:0/3331104052 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a7001e420 con 0x555fb8692ee0 2024-03-28T22:39:46.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.753+0000 7f9a6d600640 1 -- 172.21.15.43:0/3331104052 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9a740bc320 con 0x555fb8692ee0 2024-03-28T22:39:46.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.753+0000 7f9a8d174740 10 client.26494 fetch_fsmap learned FSMap version 30 2024-03-28T22:39:46.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.753+0000 7f9a8d174740 1 -- 172.21.15.43:0/3331104052 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555fb82cb0f0 con 0x555fb8692ee0 2024-03-28T22:39:46.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.753+0000 7f9a6d600640 1 -- 172.21.15.43:0/3331104052 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a7001e420 con 0x555fb8692ee0 2024-03-28T22:39:46.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.753+0000 7f9a6d600640 1 -- 172.21.15.43:0/3331104052 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f9a740edb80 con 0x555fb8692ee0 2024-03-28T22:39:46.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.754+0000 7f9a8d174740 10 client.26494 fetch_fsmap finished waiting for FSMap version 30 2024-03-28T22:39:46.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.754+0000 7f9a8d174740 10 client.26494 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:39:46.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.754+0000 7f9a8d174740 20 client.26494 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:46.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.754+0000 7f9a8d174740 1 --2- 172.21.15.43:0/3331104052 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x555fb87123f0 0x555fb87147e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:46.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.754+0000 7f9a8d174740 4 client.26494 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:39:46.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.754+0000 7f9a8d174740 1 -- 172.21.15.43:0/3331104052 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555fb85fe990 con 0x555fb87123f0 2024-03-28T22:39:46.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.754+0000 7f9a82a00640 1 --2- 172.21.15.43:0/3331104052 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x555fb87123f0 0x555fb87147e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:46.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.754+0000 7f9a82a00640 1 --2- 172.21.15.43:0/3331104052 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x555fb87123f0 0x555fb87147e0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f45f3202da1fcaab server_cookie=f8486df7d5398280 in_seq=0 out_seq=0 2024-03-28T22:39:46.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.754+0000 7f9a6d600640 10 client.26494 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:46.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.755+0000 7f9a6d600640 1 -- 172.21.15.43:0/3331104052 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x555fb85fe990 con 0x555fb87123f0 2024-03-28T22:39:46.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.755+0000 7f9a6d600640 10 client.26494 handle_command_reply: tid=0 2024-03-28T22:39:46.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.755+0000 7f9a8d174740 1 client.26494 shutdown 2024-03-28T22:39:46.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.755+0000 7f9a8d174740 1 -- 172.21.15.43:0/3331104052 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555fb8692ee0 msgr2=0x555fb86a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:46.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.755+0000 7f9a8d174740 1 --2- 172.21.15.43:0/3331104052 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555fb8692ee0 0x555fb86a4360 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9a74009610 tx=0x7f9a740a72e0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.755+0000 7f9a8d174740 1 -- 172.21.15.43:0/3331104052 shutdown_connections 2024-03-28T22:39:46.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.755+0000 7f9a8d174740 1 --2- 172.21.15.43:0/3331104052 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x555fb87123f0 0x555fb87147e0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.755+0000 7f9a8d174740 1 --2- 172.21.15.43:0/3331104052 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555fb869c250 0x555fb86969e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.755+0000 7f9a8d174740 1 --2- 172.21.15.43:0/3331104052 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555fb8693bc0 0x555fb86964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.755+0000 7f9a8d174740 1 --2- 172.21.15.43:0/3331104052 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555fb8692ee0 0x555fb86a4360 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.755+0000 7f9a8d174740 1 -- 172.21.15.43:0/3331104052 >> 172.21.15.43:0/3331104052 conn(0x555fb8659a10 msgr2=0x555fb868cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:46.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.755+0000 7f9a6d600640 0 client.26494 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:46.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.755+0000 7f9a8d174740 1 -- 172.21.15.43:0/3331104052 shutdown_connections 2024-03-28T22:39:46.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.755+0000 7f9a8d174740 1 -- 172.21.15.43:0/3331104052 wait complete. 2024-03-28T22:39:46.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.756+0000 7f9a8d174740 20 client.26494 trim_cache size 0 max 16384 2024-03-28T22:39:46.759 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:39:46.759 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-03-28T22:39:46.759 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-03-28T22:39:46.759 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:39:46.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.756+0000 7f9a8a400640 1 -- 172.21.15.43:0/3897791934 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9a680a1ef0 msgr2=0x7f9a680a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:46.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.756+0000 7f9a8a400640 1 --2- 172.21.15.43:0/3897791934 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9a680a1ef0 0x7f9a680a43b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c005fd0 tx=0x7f9a7c009500 comp rx=0 tx=0).stop 2024-03-28T22:39:46.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.756+0000 7f9a8a400640 1 -- 172.21.15.43:0/3897791934 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a84068950 msgr2=0x7f9a84100610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:46.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.756+0000 7f9a8a400640 1 --2- 172.21.15.43:0/3897791934 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a84068950 0x7f9a84100610 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f9a74009900 tx=0x7f9a74004270 comp rx=0 tx=0).stop 2024-03-28T22:39:46.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.756+0000 7f9a8a400640 1 -- 172.21.15.43:0/3897791934 shutdown_connections 2024-03-28T22:39:46.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.756+0000 7f9a8a400640 1 --2- 172.21.15.43:0/3897791934 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9a680a1ef0 0x7f9a680a43b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.756+0000 7f9a8a400640 1 --2- 172.21.15.43:0/3897791934 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9a841077a0 0x7f9a84101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.756+0000 7f9a8a400640 1 --2- 172.21.15.43:0/3897791934 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9a84069310 0x7f9a84100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.756+0000 7f9a8a400640 1 --2- 172.21.15.43:0/3897791934 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a84068950 0x7f9a84100610 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:46.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.756+0000 7f9a8a400640 1 -- 172.21.15.43:0/3897791934 >> 172.21.15.43:0/3897791934 conn(0x7f9a84077750 msgr2=0x7f9a840fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:46.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.756+0000 7f9a8a400640 1 -- 172.21.15.43:0/3897791934 shutdown_connections 2024-03-28T22:39:46.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:46.757+0000 7f9a8a400640 1 -- 172.21.15.43:0/3897791934 wait complete. 2024-03-28T22:39:46.774 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:39:46.774 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:39:46.872 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:46 smithi043 ceph-mon[31823]: pgmap v81: 129 pgs: 129 active+clean; 39 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 278 op/s 2024-03-28T22:39:46.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:46 smithi172 ceph-mon[36566]: pgmap v81: 129 pgs: 129 active+clean; 39 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 278 op/s 2024-03-28T22:39:46.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:46 smithi082 ceph-mon[36471]: pgmap v81: 129 pgs: 129 active+clean; 39 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 278 op/s 2024-03-28T22:39:47.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.031+0000 7fc317400640 1 -- 172.21.15.43:0/3271255858 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc310107710 msgr2=0x7fc310107af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:47.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.031+0000 7fc317400640 1 --2- 172.21.15.43:0/3271255858 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc310107710 0x7fc310107af0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc304009930 tx=0x7fc30402bc20 comp rx=0 tx=0).stop 2024-03-28T22:39:47.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.031+0000 7fc317400640 1 -- 172.21.15.43:0/3271255858 shutdown_connections 2024-03-28T22:39:47.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.031+0000 7fc317400640 1 --2- 172.21.15.43:0/3271255858 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc310069d20 0x7fc31006a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.031+0000 7fc317400640 1 --2- 172.21.15.43:0/3271255858 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc31006a6a0 0x7fc31010e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.031+0000 7fc317400640 1 --2- 172.21.15.43:0/3271255858 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc310107710 0x7fc310107af0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.031+0000 7fc317400640 1 -- 172.21.15.43:0/3271255858 >> 172.21.15.43:0/3271255858 conn(0x7fc310077630 msgr2=0x7fc310077a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:47.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.032+0000 7fc317400640 1 -- 172.21.15.43:0/3271255858 shutdown_connections 2024-03-28T22:39:47.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.032+0000 7fc317400640 1 -- 172.21.15.43:0/3271255858 wait complete. 2024-03-28T22:39:47.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.033+0000 7fc317400640 1 Processor -- start 2024-03-28T22:39:47.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.033+0000 7fc317400640 1 -- start start 2024-03-28T22:39:47.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.033+0000 7fc317400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc310069d20 0x7fc31019a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.033+0000 7fc317400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc31006a6a0 0x7fc31019a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.033+0000 7fc317400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc310107710 0x7fc31019eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.033+0000 7fc317400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc310111b60 con 0x7fc31006a6a0 2024-03-28T22:39:47.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.033+0000 7fc317400640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fc3101119e0 con 0x7fc310107710 2024-03-28T22:39:47.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.033+0000 7fc317400640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fc310111ce0 con 0x7fc310069d20 2024-03-28T22:39:47.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.033+0000 7fc30fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc31006a6a0 0x7fc31019a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.033+0000 7fc315400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc310107710 0x7fc31019eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.033+0000 7fc315400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc310107710 0x7fc31019eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52564/0 (socket says 172.21.15.43:52564) 2024-03-28T22:39:47.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.033+0000 7fc315400640 1 -- 172.21.15.43:0/118689753 learned_addr learned my addr 172.21.15.43:0/118689753 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:47.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.033+0000 7fc314a00640 1 --2- 172.21.15.43:0/118689753 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc310069d20 0x7fc31019a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.034+0000 7fc315400640 1 -- 172.21.15.43:0/118689753 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc310069d20 msgr2=0x7fc31019a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:47.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.034+0000 7fc315400640 1 --2- 172.21.15.43:0/118689753 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc310069d20 0x7fc31019a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.034+0000 7fc315400640 1 -- 172.21.15.43:0/118689753 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc31006a6a0 msgr2=0x7fc31019a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:47.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.034+0000 7fc315400640 1 --2- 172.21.15.43:0/118689753 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc31006a6a0 0x7fc31019a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.034+0000 7fc315400640 1 -- 172.21.15.43:0/118689753 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc31019f250 con 0x7fc310107710 2024-03-28T22:39:47.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.034+0000 7fc315400640 1 --2- 172.21.15.43:0/118689753 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc310107710 0x7fc31019eb20 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fc30800eb10 tx=0x7fc30800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:47.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.034+0000 7fc30d600640 1 -- 172.21.15.43:0/118689753 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc308004b20 con 0x7fc310107710 2024-03-28T22:39:47.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.034+0000 7fc30d600640 1 -- 172.21.15.43:0/118689753 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc308004cc0 con 0x7fc310107710 2024-03-28T22:39:47.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.034+0000 7fc317400640 1 -- 172.21.15.43:0/118689753 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc31019f540 con 0x7fc310107710 2024-03-28T22:39:47.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.035+0000 7fc30d600640 1 -- 172.21.15.43:0/118689753 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc308010760 con 0x7fc310107710 2024-03-28T22:39:47.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.035+0000 7fc317400640 1 -- 172.21.15.43:0/118689753 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3101a6dd0 con 0x7fc310107710 2024-03-28T22:39:47.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.036+0000 7fc317400640 1 -- 172.21.15.43:0/118689753 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc310102e40 con 0x7fc310107710 2024-03-28T22:39:47.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.038+0000 7fc30d600640 1 -- 172.21.15.43:0/118689753 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fc308005020 con 0x7fc310107710 2024-03-28T22:39:47.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.039+0000 7fc30d600640 1 --2- 172.21.15.43:0/118689753 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc2f80a1f70 0x7fc2f80a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.040+0000 7fc30d600640 1 -- 172.21.15.43:0/118689753 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7fc3080cea40 con 0x7fc310107710 2024-03-28T22:39:47.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.040+0000 7fc314a00640 1 --2- 172.21.15.43:0/118689753 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc2f80a1f70 0x7fc2f80a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.041+0000 7fc314a00640 1 --2- 172.21.15.43:0/118689753 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc2f80a1f70 0x7fc2f80a4430 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc304009900 tx=0x7fc304005a20 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:47.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.045+0000 7fc30d600640 1 -- 172.21.15.43:0/118689753 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc30809b730 con 0x7fc310107710 2024-03-28T22:39:47.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.300+0000 7fc317400640 1 -- 172.21.15.43:0/118689753 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc31019b5b0 con 0x7fc310107710 2024-03-28T22:39:47.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.302+0000 7fc30d600640 1 -- 172.21.15.43:0/118689753 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11934 (secure 0 0 0) 0x7fc30809b030 con 0x7fc310107710 2024-03-28T22:39:47.301 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:39:47.301 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:33:28.906396+0000","modified":"2024-03-28T22:39:27.651799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24572,"mds_1":24556,"mds_2":24535,"mds_3":24523,"mds_4":24518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26507":{"gid":26507,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/4245880955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":4245880955},{"type":"v1","addr":"172.21.15.82:6835","nonce":4245880955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24556,24535,24518,24572,24523]},"id":1}]} 2024-03-28T22:39:47.301 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 30 2024-03-28T22:39:47.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.305+0000 7fc317400640 1 -- 172.21.15.43:0/118689753 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc2f80a1f70 msgr2=0x7fc2f80a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:47.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.305+0000 7fc317400640 1 --2- 172.21.15.43:0/118689753 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc2f80a1f70 0x7fc2f80a4430 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc304009900 tx=0x7fc304005a20 comp rx=0 tx=0).stop 2024-03-28T22:39:47.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.305+0000 7fc317400640 1 -- 172.21.15.43:0/118689753 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc310107710 msgr2=0x7fc31019eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:47.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.305+0000 7fc317400640 1 --2- 172.21.15.43:0/118689753 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc310107710 0x7fc31019eb20 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fc30800eb10 tx=0x7fc30800c2d0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.305+0000 7fc317400640 1 -- 172.21.15.43:0/118689753 shutdown_connections 2024-03-28T22:39:47.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.305+0000 7fc317400640 1 --2- 172.21.15.43:0/118689753 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc2f80a1f70 0x7fc2f80a4430 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.305+0000 7fc317400640 1 --2- 172.21.15.43:0/118689753 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc310107710 0x7fc31019eb20 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.305+0000 7fc317400640 1 --2- 172.21.15.43:0/118689753 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc31006a6a0 0x7fc31019a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.305+0000 7fc317400640 1 --2- 172.21.15.43:0/118689753 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc310069d20 0x7fc31019a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.305+0000 7fc317400640 1 -- 172.21.15.43:0/118689753 >> 172.21.15.43:0/118689753 conn(0x7fc310077630 msgr2=0x7fc31010c920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:47.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.305+0000 7fc317400640 1 -- 172.21.15.43:0/118689753 shutdown_connections 2024-03-28T22:39:47.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.305+0000 7fc317400640 1 -- 172.21.15.43:0/118689753 wait complete. 2024-03-28T22:39:47.328 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T22:39:47.347 INFO:tasks.workunit.client.0.smithi043.stdout: 1 124449 8.85 MB/sec warmup 78 sec latency 25.641 ms 2024-03-28T22:39:47.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.563+0000 7f4df4800640 1 -- 172.21.15.43:0/128566938 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4dec1016e0 msgr2=0x7f4dec101b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:47.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.563+0000 7f4df4800640 1 --2- 172.21.15.43:0/128566938 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4dec1016e0 0x7f4dec101b60 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc00fb60 tx=0x7f4ddc02fcf0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.564+0000 7f4df4800640 1 -- 172.21.15.43:0/128566938 shutdown_connections 2024-03-28T22:39:47.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.564+0000 7f4df4800640 1 --2- 172.21.15.43:0/128566938 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4dec1016e0 0x7f4dec101b60 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.564+0000 7f4df4800640 1 --2- 172.21.15.43:0/128566938 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4dec1020a0 0x7f4dec10e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.564+0000 7f4df4800640 1 --2- 172.21.15.43:0/128566938 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dec1076f0 0x7f4dec107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.564+0000 7f4df4800640 1 -- 172.21.15.43:0/128566938 >> 172.21.15.43:0/128566938 conn(0x7f4dec078b50 msgr2=0x7f4dec078f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:47.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.564+0000 7f4df4800640 1 -- 172.21.15.43:0/128566938 shutdown_connections 2024-03-28T22:39:47.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.564+0000 7f4df4800640 1 -- 172.21.15.43:0/128566938 wait complete. 2024-03-28T22:39:47.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.565+0000 7f4df4800640 1 Processor -- start 2024-03-28T22:39:47.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.565+0000 7f4df4800640 1 -- start start 2024-03-28T22:39:47.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.565+0000 7f4df4800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4dec1016e0 0x7f4dec195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.565+0000 7f4df4800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dec1020a0 0x7f4dec196290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.565+0000 7f4df4800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4dec1076f0 0x7f4dec19a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.565+0000 7f4df4800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4dec10f8f0 con 0x7f4dec1016e0 2024-03-28T22:39:47.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.565+0000 7f4df4800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4dec10f770 con 0x7f4dec1076f0 2024-03-28T22:39:47.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.565+0000 7f4df4800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4dec10fa70 con 0x7f4dec1020a0 2024-03-28T22:39:47.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.566+0000 7f4df2800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4dec1076f0 0x7f4dec19a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.566+0000 7f4df1400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dec1020a0 0x7f4dec196290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.566+0000 7f4df2800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4dec1076f0 0x7f4dec19a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52590/0 (socket says 172.21.15.43:52590) 2024-03-28T22:39:47.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.566+0000 7f4df2800640 1 -- 172.21.15.43:0/214551270 learned_addr learned my addr 172.21.15.43:0/214551270 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:47.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.566+0000 7f4df1e00640 1 --2- 172.21.15.43:0/214551270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4dec1016e0 0x7f4dec195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.566+0000 7f4df1400640 1 -- 172.21.15.43:0/214551270 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4dec1076f0 msgr2=0x7f4dec19a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:47.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.566+0000 7f4df1400640 1 --2- 172.21.15.43:0/214551270 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4dec1076f0 0x7f4dec19a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.566+0000 7f4df1400640 1 -- 172.21.15.43:0/214551270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4dec1016e0 msgr2=0x7f4dec195d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:47.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.566+0000 7f4df1400640 1 --2- 172.21.15.43:0/214551270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4dec1016e0 0x7f4dec195d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.566+0000 7f4df1400640 1 -- 172.21.15.43:0/214551270 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dec19ad40 con 0x7f4dec1020a0 2024-03-28T22:39:47.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.566+0000 7f4df1400640 1 --2- 172.21.15.43:0/214551270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dec1020a0 0x7f4dec196290 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc030230 tx=0x7f4ddc0045b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:47.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.567+0000 7f4de6a00640 1 -- 172.21.15.43:0/214551270 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc045020 con 0x7f4dec1020a0 2024-03-28T22:39:47.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.567+0000 7f4de6a00640 1 -- 172.21.15.43:0/214551270 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ddc038920 con 0x7f4dec1020a0 2024-03-28T22:39:47.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.567+0000 7f4df4800640 1 -- 172.21.15.43:0/214551270 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dec19afd0 con 0x7f4dec1020a0 2024-03-28T22:39:47.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.567+0000 7f4de6a00640 1 -- 172.21.15.43:0/214551270 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc03d040 con 0x7f4dec1020a0 2024-03-28T22:39:47.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.567+0000 7f4df4800640 1 -- 172.21.15.43:0/214551270 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dec1a28c0 con 0x7f4dec1020a0 2024-03-28T22:39:47.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.570+0000 7f4de6a00640 1 -- 172.21.15.43:0/214551270 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f4ddc048020 con 0x7f4dec1020a0 2024-03-28T22:39:47.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.571+0000 7f4df761a740 1 Processor -- start 2024-03-28T22:39:47.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.571+0000 7f4df761a740 1 -- start start 2024-03-28T22:39:47.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.571+0000 7f4df761a740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561e8d318380 0x561e8d318760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.571+0000 7f4df761a740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561e8d318ca0 0x561e8d321340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.571+0000 7f4df761a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561e8d321880 0x561e8d323c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.571+0000 7f4df761a740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x561e8d27c050 con 0x561e8d318380 2024-03-28T22:39:47.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.571+0000 7f4df761a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x561e8d224ce0 con 0x561e8d321880 2024-03-28T22:39:47.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.571+0000 7f4df761a740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x561e8d2780f0 con 0x561e8d318ca0 2024-03-28T22:39:47.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.571+0000 7f4df1e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561e8d318ca0 0x561e8d321340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.571+0000 7f4df1400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561e8d321880 0x561e8d323c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.572+0000 7f4df1e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561e8d318ca0 0x561e8d321340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:55978/0 (socket says 172.21.15.43:55978) 2024-03-28T22:39:47.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.572+0000 7f4df1e00640 1 -- 172.21.15.43:0/553424481 learned_addr learned my addr 172.21.15.43:0/553424481 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:47.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.572+0000 7f4df2800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561e8d318380 0x561e8d318760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.572+0000 7f4df1e00640 1 -- 172.21.15.43:0/553424481 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561e8d321880 msgr2=0x561e8d323c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:47.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.572+0000 7f4df1e00640 1 --2- 172.21.15.43:0/553424481 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561e8d321880 0x561e8d323c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.572+0000 7f4df1e00640 1 -- 172.21.15.43:0/553424481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561e8d318380 msgr2=0x561e8d318760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:47.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.572+0000 7f4df1e00640 1 --2- 172.21.15.43:0/553424481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561e8d318380 0x561e8d318760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.572+0000 7f4df1e00640 1 -- 172.21.15.43:0/553424481 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e8cf903f0 con 0x561e8d318ca0 2024-03-28T22:39:47.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.572+0000 7f4de6a00640 1 --2- 172.21.15.43:0/214551270 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4dd40a1ef0 0x7f4dd40a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.572+0000 7f4df1e00640 1 --2- 172.21.15.43:0/553424481 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561e8d318ca0 0x561e8d321340 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4de0009c50 tx=0x7f4de0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6ddf46b103ec468 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:47.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.572+0000 7f4de6a00640 1 -- 172.21.15.43:0/214551270 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7f4ddc0f3ba0 con 0x7f4dec1020a0 2024-03-28T22:39:47.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.572+0000 7f4df2800640 1 --2- 172.21.15.43:0/214551270 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4dd40a1ef0 0x7f4dd40a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.573+0000 7f4ddb000640 1 -- 172.21.15.43:0/553424481 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de001e070 con 0x561e8d318ca0 2024-03-28T22:39:47.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.573+0000 7f4ddb000640 1 -- 172.21.15.43:0/553424481 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4de0019910 con 0x561e8d318ca0 2024-03-28T22:39:47.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.573+0000 7f4ddb000640 1 -- 172.21.15.43:0/553424481 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de000e9b0 con 0x561e8d318ca0 2024-03-28T22:39:47.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.573+0000 7f4df761a740 1 -- 172.21.15.43:0/553424481 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561e8d318ca0 msgr2=0x561e8d321340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:47.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.573+0000 7f4df761a740 1 --2- 172.21.15.43:0/553424481 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561e8d318ca0 0x561e8d321340 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4de0009c50 tx=0x7f4de0011810 comp rx=0 tx=0).stop 2024-03-28T22:39:47.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.573+0000 7f4df2800640 1 --2- 172.21.15.43:0/214551270 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4dd40a1ef0 0x7f4dd40a43b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4de80049d0 tx=0x7f4de8009500 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:47.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.574+0000 7f4df761a740 1 -- 172.21.15.43:0/553424481 shutdown_connections 2024-03-28T22:39:47.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.574+0000 7f4df761a740 1 --2- 172.21.15.43:0/553424481 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561e8d321880 0x561e8d323c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.574+0000 7f4df761a740 1 --2- 172.21.15.43:0/553424481 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561e8d318ca0 0x561e8d321340 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.574+0000 7f4df761a740 1 --2- 172.21.15.43:0/553424481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561e8d318380 0x561e8d318760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.574+0000 7f4df761a740 1 -- 172.21.15.43:0/553424481 >> 172.21.15.43:0/553424481 conn(0x561e8d312ab0 msgr2=0x561e8d312e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:47.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.574+0000 7f4df761a740 1 -- 172.21.15.43:0/553424481 shutdown_connections 2024-03-28T22:39:47.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.574+0000 7f4df761a740 1 -- 172.21.15.43:0/553424481 wait complete. 2024-03-28T22:39:47.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.574+0000 7f4df761a740 1 -- start start 2024-03-28T22:39:47.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.575+0000 7f4df761a740 1 Processor -- start 2024-03-28T22:39:47.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.575+0000 7f4df761a740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561e8d318380 0x561e8d31e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.575+0000 7f4df761a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561e8d318ca0 0x561e8d31ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.575+0000 7f4df761a740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561e8d321880 0x561e8d31f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.575+0000 7f4df761a740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x561e8d324670 con 0x561e8d321880 2024-03-28T22:39:47.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.575+0000 7f4df761a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x561e8d21a2f0 con 0x561e8d318ca0 2024-03-28T22:39:47.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.575+0000 7f4df761a740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x561e8d3247f0 con 0x561e8d318380 2024-03-28T22:39:47.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.575+0000 7f4df761a740 10 client.0 _ll_register_callbacks cb 0x561e8cf8fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:47.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.575+0000 7f4df1e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561e8d318380 0x561e8d31e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.576+0000 7f4df1400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561e8d318ca0 0x561e8d31ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.576+0000 7f4df1e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561e8d318380 0x561e8d31e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:55992/0 (socket says 172.21.15.43:55992) 2024-03-28T22:39:47.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.576+0000 7f4df1e00640 1 -- 172.21.15.43:0/1070729842 learned_addr learned my addr 172.21.15.43:0/1070729842 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:47.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.576+0000 7f4df2800640 1 --2- 172.21.15.43:0/1070729842 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561e8d321880 0x561e8d31f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.576+0000 7f4df1e00640 1 -- 172.21.15.43:0/1070729842 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561e8d318ca0 msgr2=0x561e8d31ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:47.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.576+0000 7f4df1e00640 1 --2- 172.21.15.43:0/1070729842 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561e8d318ca0 0x561e8d31ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.576+0000 7f4df1e00640 1 -- 172.21.15.43:0/1070729842 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561e8d321880 msgr2=0x561e8d31f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:47.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.576+0000 7f4df1e00640 1 --2- 172.21.15.43:0/1070729842 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561e8d321880 0x561e8d31f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.576+0000 7f4df1e00640 1 -- 172.21.15.43:0/1070729842 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e8d05a7d0 con 0x561e8d318380 2024-03-28T22:39:47.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.576+0000 7f4df1400640 1 --2- 172.21.15.43:0/1070729842 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561e8d318ca0 0x561e8d31ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:47.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.577+0000 7f4df1e00640 1 --2- 172.21.15.43:0/1070729842 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561e8d318380 0x561e8d31e920 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4de0009660 tx=0x7f4de0019f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85b98b4499e8a403 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:47.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.577+0000 7f4dcf400640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:39:47.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.580+0000 7f4dcf400640 1 -- 172.21.15.43:0/1070729842 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de0029020 con 0x561e8d318380 2024-03-28T22:39:47.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.580+0000 7f4dcf400640 1 -- 172.21.15.43:0/1070729842 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7f4de0019630 con 0x561e8d318380 2024-03-28T22:39:47.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.580+0000 7f4df761a740 1 -- 172.21.15.43:0/1070729842 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4de0019910 con 0x561e8d318380 2024-03-28T22:39:47.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.584+0000 7f4dcf400640 1 -- 172.21.15.43:0/1070729842 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4de0034050 con 0x561e8d318380 2024-03-28T22:39:47.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.584+0000 7f4df761a740 10 client.26512 fetch_fsmap learned FSMap version 30 2024-03-28T22:39:47.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.584+0000 7f4df761a740 1 -- 172.21.15.43:0/1070729842 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561e8d3309a0 con 0x561e8d318380 2024-03-28T22:39:47.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.585+0000 7f4dcf400640 1 -- 172.21.15.43:0/1070729842 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4de0019910 con 0x561e8d318380 2024-03-28T22:39:47.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.585+0000 7f4dcf400640 1 -- 172.21.15.43:0/1070729842 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f4de000eb60 con 0x561e8d318380 2024-03-28T22:39:47.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.585+0000 7f4df761a740 10 client.26512 fetch_fsmap finished waiting for FSMap version 30 2024-03-28T22:39:47.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.585+0000 7f4df761a740 10 client.26512 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:39:47.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.585+0000 7f4df761a740 20 client.26512 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:47.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.585+0000 7f4df761a740 1 --2- 172.21.15.43:0/1070729842 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x561e8d331e00 0x561e8d3341f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.585+0000 7f4df761a740 4 client.26512 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:39:47.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.585+0000 7f4df761a740 1 -- 172.21.15.43:0/1070729842 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561e8d225680 con 0x561e8d331e00 2024-03-28T22:39:47.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.585+0000 7f4df1400640 1 --2- 172.21.15.43:0/1070729842 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x561e8d331e00 0x561e8d3341f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.586+0000 7f4df1400640 1 --2- 172.21.15.43:0/1070729842 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x561e8d331e00 0x561e8d3341f0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=24910520478b840e server_cookie=14a365a0943cd8b7 in_seq=0 out_seq=0 2024-03-28T22:39:47.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.586+0000 7f4dcf400640 10 client.26512 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:39:47.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.587+0000 7f4dcf400640 1 -- 172.21.15.43:0/1070729842 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561e8d225680 con 0x561e8d331e00 2024-03-28T22:39:47.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.587+0000 7f4dcf400640 10 client.26512 handle_command_reply: tid=0 2024-03-28T22:39:47.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.587+0000 7f4df761a740 1 client.26512 shutdown 2024-03-28T22:39:47.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.587+0000 7f4df761a740 1 -- 172.21.15.43:0/1070729842 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561e8d318380 msgr2=0x561e8d31e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:47.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.587+0000 7f4df761a740 1 --2- 172.21.15.43:0/1070729842 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561e8d318380 0x561e8d31e920 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4de0009660 tx=0x7f4de0019f10 comp rx=0 tx=0).stop 2024-03-28T22:39:47.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.587+0000 7f4df761a740 1 -- 172.21.15.43:0/1070729842 shutdown_connections 2024-03-28T22:39:47.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.587+0000 7f4df761a740 1 --2- 172.21.15.43:0/1070729842 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x561e8d331e00 0x561e8d3341f0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.587+0000 7f4df761a740 1 --2- 172.21.15.43:0/1070729842 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561e8d318ca0 0x561e8d31ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.587+0000 7f4df761a740 1 --2- 172.21.15.43:0/1070729842 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561e8d321880 0x561e8d31f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.587+0000 7f4df761a740 1 --2- 172.21.15.43:0/1070729842 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561e8d318380 0x561e8d31e920 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.587+0000 7f4dcf400640 0 client.26512 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:39:47.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.587+0000 7f4df761a740 1 -- 172.21.15.43:0/1070729842 >> 172.21.15.43:0/1070729842 conn(0x561e8d312ab0 msgr2=0x561e8d314800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:47.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.588+0000 7f4df761a740 1 -- 172.21.15.43:0/1070729842 shutdown_connections 2024-03-28T22:39:47.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.588+0000 7f4df761a740 1 -- 172.21.15.43:0/1070729842 wait complete. 2024-03-28T22:39:47.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.588+0000 7f4df761a740 20 client.26512 trim_cache size 0 max 16384 2024-03-28T22:39:47.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.604+0000 7f4df761a740 1 Processor -- start 2024-03-28T22:39:47.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.604+0000 7f4df761a740 1 -- start start 2024-03-28T22:39:47.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.605+0000 7f4df761a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561e8d318380 0x561e8d32d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.605+0000 7f4df761a740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561e8d318ca0 0x561e8d31f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.605+0000 7f4df761a740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561e8d321880 0x561e8d3119f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.605+0000 7f4df761a740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4dec1ac920 con 0x561e8d321880 2024-03-28T22:39:47.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.605+0000 7f4df761a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4dec1ae460 con 0x561e8d318380 2024-03-28T22:39:47.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.605+0000 7f4df761a740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4dec1acff0 con 0x561e8d318ca0 2024-03-28T22:39:47.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.605+0000 7f4df2800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561e8d321880 0x561e8d3119f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.605+0000 7f4df1400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561e8d318ca0 0x561e8d31f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.605+0000 7f4df1e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561e8d318380 0x561e8d32d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.605+0000 7f4df2800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561e8d321880 0x561e8d3119f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44062/0 (socket says 172.21.15.43:44062) 2024-03-28T22:39:47.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.605+0000 7f4df1400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561e8d318ca0 0x561e8d31f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:56006/0 (socket says 172.21.15.43:56006) 2024-03-28T22:39:47.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.605+0000 7f4df2800640 1 -- 172.21.15.43:0/2059457798 learned_addr learned my addr 172.21.15.43:0/2059457798 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:47.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.605+0000 7f4df2800640 1 -- 172.21.15.43:0/2059457798 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561e8d318ca0 msgr2=0x561e8d31f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:47.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.605+0000 7f4df2800640 1 --2- 172.21.15.43:0/2059457798 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561e8d318ca0 0x561e8d31f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.606+0000 7f4df2800640 1 -- 172.21.15.43:0/2059457798 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561e8d318380 msgr2=0x561e8d32d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:47.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.606+0000 7f4df2800640 1 --2- 172.21.15.43:0/2059457798 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561e8d318380 0x561e8d32d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.606+0000 7f4df2800640 1 -- 172.21.15.43:0/2059457798 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e8d318ad0 con 0x561e8d321880 2024-03-28T22:39:47.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.606+0000 7f4df2800640 1 --2- 172.21.15.43:0/2059457798 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561e8d321880 0x561e8d3119f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4de800f9f0 tx=0x7f4de8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeff468cb6e70ace server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:47.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.606+0000 7f4dda600640 1 -- 172.21.15.43:0/2059457798 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de8012020 con 0x561e8d321880 2024-03-28T22:39:47.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.606+0000 7f4dda600640 1 -- 172.21.15.43:0/2059457798 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4de80121c0 con 0x561e8d321880 2024-03-28T22:39:47.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.606+0000 7f4dda600640 1 -- 172.21.15.43:0/2059457798 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de80296e0 con 0x561e8d321880 2024-03-28T22:39:47.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.607+0000 7f4df761a740 1 -- 172.21.15.43:0/2059457798 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561e8d321880 msgr2=0x561e8d3119f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:47.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.607+0000 7f4df761a740 1 --2- 172.21.15.43:0/2059457798 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561e8d321880 0x561e8d3119f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4de800f9f0 tx=0x7f4de8017000 comp rx=0 tx=0).stop 2024-03-28T22:39:47.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.607+0000 7f4df761a740 1 -- 172.21.15.43:0/2059457798 shutdown_connections 2024-03-28T22:39:47.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.607+0000 7f4df761a740 1 --2- 172.21.15.43:0/2059457798 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561e8d321880 0x561e8d3119f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.607+0000 7f4df761a740 1 --2- 172.21.15.43:0/2059457798 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561e8d318ca0 0x561e8d31f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.607+0000 7f4df761a740 1 --2- 172.21.15.43:0/2059457798 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561e8d318380 0x561e8d32d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.607+0000 7f4df761a740 1 -- 172.21.15.43:0/2059457798 >> 172.21.15.43:0/2059457798 conn(0x561e8d312ab0 msgr2=0x561e8d3112e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:47.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.607+0000 7f4df761a740 1 -- 172.21.15.43:0/2059457798 shutdown_connections 2024-03-28T22:39:47.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.607+0000 7f4df761a740 1 -- 172.21.15.43:0/2059457798 wait complete. 2024-03-28T22:39:47.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.608+0000 7f4df761a740 1 -- start start 2024-03-28T22:39:47.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.608+0000 7f4df761a740 1 Processor -- start 2024-03-28T22:39:47.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.608+0000 7f4df761a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561e8d318380 0x561e8d3299c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.608+0000 7f4df761a740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561e8d318ca0 0x561e8d324e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.608+0000 7f4df761a740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561e8d321880 0x561e8d325360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.608+0000 7f4df761a740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4dec1add90 con 0x561e8d321880 2024-03-28T22:39:47.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.608+0000 7f4df761a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4dec1ad6c0 con 0x561e8d318380 2024-03-28T22:39:47.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.608+0000 7f4df761a740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4dec1aa900 con 0x561e8d318ca0 2024-03-28T22:39:47.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.608+0000 7f4df761a740 10 client.0 _ll_register_callbacks cb 0x7f4de001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:47.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.609+0000 7f4df1e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561e8d318380 0x561e8d3299c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.609+0000 7f4df1400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561e8d318ca0 0x561e8d324e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.609+0000 7f4df1e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561e8d318380 0x561e8d3299c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52622/0 (socket says 172.21.15.43:52622) 2024-03-28T22:39:47.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.609+0000 7f4df2800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561e8d321880 0x561e8d325360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.609+0000 7f4df1e00640 1 -- 172.21.15.43:0/3111296727 learned_addr learned my addr 172.21.15.43:0/3111296727 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:47.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.609+0000 7f4df1e00640 1 -- 172.21.15.43:0/3111296727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561e8d318ca0 msgr2=0x561e8d324e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:47.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.609+0000 7f4df1e00640 1 --2- 172.21.15.43:0/3111296727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561e8d318ca0 0x561e8d324e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.609+0000 7f4df1e00640 1 -- 172.21.15.43:0/3111296727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561e8d321880 msgr2=0x561e8d325360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:47.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.609+0000 7f4df1e00640 1 --2- 172.21.15.43:0/3111296727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561e8d321880 0x561e8d325360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.609+0000 7f4df1e00640 1 -- 172.21.15.43:0/3111296727 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e8d3258a0 con 0x561e8d318380 2024-03-28T22:39:47.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.609+0000 7f4df1e00640 1 --2- 172.21.15.43:0/3111296727 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561e8d318380 0x561e8d3299c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4de0009c50 tx=0x7f4de0005df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50506508daf22895 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:47.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.609+0000 7f4dcf400640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:39:47.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.609+0000 7f4dcf400640 1 -- 172.21.15.43:0/3111296727 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de003d060 con 0x561e8d318380 2024-03-28T22:39:47.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.609+0000 7f4df761a740 1 -- 172.21.15.43:0/3111296727 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4de80121c0 con 0x561e8d318380 2024-03-28T22:39:47.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.609+0000 7f4dcf400640 1 -- 172.21.15.43:0/3111296727 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7f4de0019630 con 0x561e8d318380 2024-03-28T22:39:47.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.610+0000 7f4dcf400640 1 -- 172.21.15.43:0/3111296727 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4de0044050 con 0x561e8d318380 2024-03-28T22:39:47.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.610+0000 7f4df761a740 10 client.26567 fetch_fsmap learned FSMap version 30 2024-03-28T22:39:47.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.610+0000 7f4df761a740 1 -- 172.21.15.43:0/3111296727 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561e8d397cb0 con 0x561e8d318380 2024-03-28T22:39:47.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.610+0000 7f4dcf400640 1 -- 172.21.15.43:0/3111296727 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4de80121c0 con 0x561e8d318380 2024-03-28T22:39:47.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.610+0000 7f4dcf400640 1 -- 172.21.15.43:0/3111296727 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f4de0038430 con 0x561e8d318380 2024-03-28T22:39:47.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.610+0000 7f4df761a740 10 client.26567 fetch_fsmap finished waiting for FSMap version 30 2024-03-28T22:39:47.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.610+0000 7f4df761a740 10 client.26567 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:39:47.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.610+0000 7f4df761a740 20 client.26567 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:47.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.610+0000 7f4df761a740 1 --2- 172.21.15.43:0/3111296727 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x561e8d3983a0 0x561e8d39a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.610+0000 7f4df761a740 4 client.26567 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:39:47.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.610+0000 7f4df761a740 1 -- 172.21.15.43:0/3111296727 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561e8d289590 con 0x561e8d3983a0 2024-03-28T22:39:47.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.611+0000 7f4df1400640 1 --2- 172.21.15.43:0/3111296727 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x561e8d3983a0 0x561e8d39a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.611+0000 7f4df1400640 1 --2- 172.21.15.43:0/3111296727 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x561e8d3983a0 0x561e8d39a7c0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ff0f4edd8fcc3df0 server_cookie=cb6f84dc5f3c9e67 in_seq=0 out_seq=0 2024-03-28T22:39:47.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.611+0000 7f4dcf400640 10 client.26567 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:39:47.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.611+0000 7f4dcf400640 1 -- 172.21.15.43:0/3111296727 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561e8d289590 con 0x561e8d3983a0 2024-03-28T22:39:47.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.611+0000 7f4dcf400640 10 client.26567 handle_command_reply: tid=0 2024-03-28T22:39:47.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.611+0000 7f4df761a740 1 client.26567 shutdown 2024-03-28T22:39:47.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.612+0000 7f4df761a740 1 -- 172.21.15.43:0/3111296727 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561e8d318380 msgr2=0x561e8d3299c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:47.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.612+0000 7f4df761a740 1 --2- 172.21.15.43:0/3111296727 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561e8d318380 0x561e8d3299c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4de0009c50 tx=0x7f4de0005df0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.612+0000 7f4df761a740 1 -- 172.21.15.43:0/3111296727 shutdown_connections 2024-03-28T22:39:47.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.612+0000 7f4df761a740 1 --2- 172.21.15.43:0/3111296727 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x561e8d3983a0 0x561e8d39a7c0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.612+0000 7f4df761a740 1 --2- 172.21.15.43:0/3111296727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561e8d321880 0x561e8d325360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.612+0000 7f4df761a740 1 --2- 172.21.15.43:0/3111296727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561e8d318ca0 0x561e8d324e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.612+0000 7f4df761a740 1 --2- 172.21.15.43:0/3111296727 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561e8d318380 0x561e8d3299c0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.612+0000 7f4dcf400640 0 client.26567 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:39:47.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.612+0000 7f4df761a740 1 -- 172.21.15.43:0/3111296727 >> 172.21.15.43:0/3111296727 conn(0x561e8d312ab0 msgr2=0x561e8d311b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:47.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.612+0000 7f4df761a740 1 -- 172.21.15.43:0/3111296727 shutdown_connections 2024-03-28T22:39:47.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.612+0000 7f4df761a740 1 -- 172.21.15.43:0/3111296727 wait complete. 2024-03-28T22:39:47.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.612+0000 7f4df761a740 20 client.26567 trim_cache size 0 max 16384 2024-03-28T22:39:47.615 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:39:47.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.613+0000 7f4df4800640 1 -- 172.21.15.43:0/214551270 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4dd40a1ef0 msgr2=0x7f4dd40a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:47.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.613+0000 7f4df4800640 1 --2- 172.21.15.43:0/214551270 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4dd40a1ef0 0x7f4dd40a43b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4de80049d0 tx=0x7f4de8009500 comp rx=0 tx=0).stop 2024-03-28T22:39:47.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.613+0000 7f4df4800640 1 -- 172.21.15.43:0/214551270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dec1020a0 msgr2=0x7f4dec196290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:47.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.613+0000 7f4df4800640 1 --2- 172.21.15.43:0/214551270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dec1020a0 0x7f4dec196290 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc030230 tx=0x7f4ddc0045b0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.613+0000 7f4df4800640 1 -- 172.21.15.43:0/214551270 shutdown_connections 2024-03-28T22:39:47.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.613+0000 7f4df4800640 1 --2- 172.21.15.43:0/214551270 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4dd40a1ef0 0x7f4dd40a43b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.613+0000 7f4df4800640 1 --2- 172.21.15.43:0/214551270 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4dec1076f0 0x7f4dec19a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.613+0000 7f4df4800640 1 --2- 172.21.15.43:0/214551270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dec1020a0 0x7f4dec196290 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.613+0000 7f4df4800640 1 --2- 172.21.15.43:0/214551270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4dec1016e0 0x7f4dec195d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.613+0000 7f4df4800640 1 -- 172.21.15.43:0/214551270 >> 172.21.15.43:0/214551270 conn(0x7f4dec078b50 msgr2=0x7f4dec103ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:47.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.613+0000 7f4df4800640 1 -- 172.21.15.43:0/214551270 shutdown_connections 2024-03-28T22:39:47.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.613+0000 7f4df4800640 1 -- 172.21.15.43:0/214551270 wait complete. 2024-03-28T22:39:47.630 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T22:39:47.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:47 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/118689753' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:39:47.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.914+0000 7f5622400640 1 -- 172.21.15.43:0/1758503900 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f561c106730 msgr2=0x7f561c106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:47.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.914+0000 7f5622400640 1 --2- 172.21.15.43:0/1758503900 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f561c106730 0x7f561c106b10 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f560c009930 tx=0x7f560c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:39:47.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.914+0000 7f5622400640 1 -- 172.21.15.43:0/1758503900 shutdown_connections 2024-03-28T22:39:47.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.914+0000 7f5622400640 1 --2- 172.21.15.43:0/1758503900 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f561c100050 0x7f561c1004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.914+0000 7f5622400640 1 --2- 172.21.15.43:0/1758503900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f561c100a10 0x7f561c10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.914+0000 7f5622400640 1 --2- 172.21.15.43:0/1758503900 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f561c106730 0x7f561c106b10 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.914+0000 7f5622400640 1 -- 172.21.15.43:0/1758503900 >> 172.21.15.43:0/1758503900 conn(0x7f561c075930 msgr2=0x7f561c075d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:47.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.914+0000 7f5622400640 1 -- 172.21.15.43:0/1758503900 shutdown_connections 2024-03-28T22:39:47.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.914+0000 7f5622400640 1 -- 172.21.15.43:0/1758503900 wait complete. 2024-03-28T22:39:47.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.915+0000 7f5622400640 1 Processor -- start 2024-03-28T22:39:47.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.915+0000 7f5622400640 1 -- start start 2024-03-28T22:39:47.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.916+0000 7f5622400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f561c100050 0x7f561c19a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.916+0000 7f5622400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f561c100a10 0x7f561c19a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.916+0000 7f5622400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f561c106730 0x7f561c19eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.916+0000 7f5622400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f561c111c50 con 0x7f561c100a10 2024-03-28T22:39:47.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.916+0000 7f5622400640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f561c111ad0 con 0x7f561c100050 2024-03-28T22:39:47.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.916+0000 7f5622400640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f561c111dd0 con 0x7f561c106730 2024-03-28T22:39:47.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.916+0000 7f561b400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f561c100050 0x7f561c19a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.916+0000 7f561b400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f561c100050 0x7f561c19a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52636/0 (socket says 172.21.15.43:52636) 2024-03-28T22:39:47.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.916+0000 7f561b400640 1 -- 172.21.15.43:0/1447203246 learned_addr learned my addr 172.21.15.43:0/1447203246 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:47.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.916+0000 7f561be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f561c106730 0x7f561c19eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.917+0000 7f5613e00640 1 --2- 172.21.15.43:0/1447203246 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f561c100a10 0x7f561c19a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.917+0000 7f561b400640 1 -- 172.21.15.43:0/1447203246 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f561c106730 msgr2=0x7f561c19eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:47.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.917+0000 7f561b400640 1 --2- 172.21.15.43:0/1447203246 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f561c106730 0x7f561c19eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.917+0000 7f561b400640 1 -- 172.21.15.43:0/1447203246 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f561c100a10 msgr2=0x7f561c19a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:47.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.917+0000 7f561b400640 1 --2- 172.21.15.43:0/1447203246 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f561c100a10 0x7f561c19a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.917+0000 7f561b400640 1 -- 172.21.15.43:0/1447203246 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f561c19f2d0 con 0x7f561c100050 2024-03-28T22:39:47.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.917+0000 7f561b400640 1 --2- 172.21.15.43:0/1447203246 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f561c100050 0x7f561c19a330 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f560c009900 tx=0x7f560c004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:47.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.917+0000 7f5618c00640 1 -- 172.21.15.43:0/1447203246 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f560c043020 con 0x7f561c100050 2024-03-28T22:39:47.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.917+0000 7f5618c00640 1 -- 172.21.15.43:0/1447203246 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f560c0054e0 con 0x7f561c100050 2024-03-28T22:39:47.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.917+0000 7f5622400640 1 -- 172.21.15.43:0/1447203246 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f561c19f560 con 0x7f561c100050 2024-03-28T22:39:47.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.917+0000 7f5618c00640 1 -- 172.21.15.43:0/1447203246 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f560c003b80 con 0x7f561c100050 2024-03-28T22:39:47.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.918+0000 7f5622400640 1 -- 172.21.15.43:0/1447203246 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f561c1a6e50 con 0x7f561c100050 2024-03-28T22:39:47.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.922+0000 7f5625010740 1 Processor -- start 2024-03-28T22:39:47.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.922+0000 7f5618c00640 1 -- 172.21.15.43:0/1447203246 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f560c046020 con 0x7f561c100050 2024-03-28T22:39:47.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.922+0000 7f5625010740 1 -- start start 2024-03-28T22:39:47.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.922+0000 7f5625010740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557de8f32380 0x557de8f32760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.922+0000 7f5625010740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557de8f32ca0 0x557de8f3b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.923+0000 7f5625010740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557de8f3b880 0x557de8f3dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.923+0000 7f5625010740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x557de8e96050 con 0x557de8f32ca0 2024-03-28T22:39:47.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.923+0000 7f5625010740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x557de8e3ece0 con 0x557de8f3b880 2024-03-28T22:39:47.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.923+0000 7f5625010740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x557de8e920f0 con 0x557de8f32380 2024-03-28T22:39:47.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.923+0000 7f561be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557de8f32380 0x557de8f32760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.923+0000 7f5613e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557de8f3b880 0x557de8f3dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.923+0000 7f561b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557de8f32ca0 0x557de8f3b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.923+0000 7f5613e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557de8f3b880 0x557de8f3dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52640/0 (socket says 172.21.15.43:52640) 2024-03-28T22:39:47.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.923+0000 7f5613e00640 1 -- 172.21.15.43:0/2028259601 learned_addr learned my addr 172.21.15.43:0/2028259601 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:47.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.923+0000 7f561be00640 1 -- 172.21.15.43:0/2028259601 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557de8f3b880 msgr2=0x557de8f3dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:47.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.923+0000 7f561be00640 1 --2- 172.21.15.43:0/2028259601 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557de8f3b880 0x557de8f3dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.923+0000 7f561be00640 1 -- 172.21.15.43:0/2028259601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557de8f32ca0 msgr2=0x557de8f3b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:47.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.923+0000 7f561be00640 1 --2- 172.21.15.43:0/2028259601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557de8f32ca0 0x557de8f3b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.923+0000 7f561be00640 1 -- 172.21.15.43:0/2028259601 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557de8baa3f0 con 0x557de8f32380 2024-03-28T22:39:47.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.924+0000 7f561b400640 1 --2- 172.21.15.43:0/2028259601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557de8f32ca0 0x557de8f3b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:39:47.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.924+0000 7f561be00640 1 --2- 172.21.15.43:0/2028259601 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557de8f32380 0x557de8f32760 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f5614009f30 tx=0x7f5614010800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3f112f36065e13f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:47.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.924+0000 7f5618c00640 1 --2- 172.21.15.43:0/1447203246 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f56000a1f70 0x7f56000a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.924+0000 7f5610c00640 1 -- 172.21.15.43:0/2028259601 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5614016070 con 0x557de8f32380 2024-03-28T22:39:47.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.924+0000 7f5610c00640 1 -- 172.21.15.43:0/2028259601 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5614004d20 con 0x557de8f32380 2024-03-28T22:39:47.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.924+0000 7f5610c00640 1 -- 172.21.15.43:0/2028259601 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5614012810 con 0x557de8f32380 2024-03-28T22:39:47.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.924+0000 7f5618c00640 1 -- 172.21.15.43:0/1447203246 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7f560c0f0640 con 0x7f561c100050 2024-03-28T22:39:47.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.924+0000 7f561b400640 1 --2- 172.21.15.43:0/1447203246 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f56000a1f70 0x7f56000a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.925+0000 7f5625010740 1 -- 172.21.15.43:0/2028259601 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557de8f32380 msgr2=0x557de8f32760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:47.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.925+0000 7f5625010740 1 --2- 172.21.15.43:0/2028259601 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557de8f32380 0x557de8f32760 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f5614009f30 tx=0x7f5614010800 comp rx=0 tx=0).stop 2024-03-28T22:39:47.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.925+0000 7f561b400640 1 --2- 172.21.15.43:0/1447203246 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f56000a1f70 0x7f56000a4430 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x557de8f31e50 tx=0x7f560c009260 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:47.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.925+0000 7f5625010740 1 -- 172.21.15.43:0/2028259601 shutdown_connections 2024-03-28T22:39:47.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.925+0000 7f5625010740 1 --2- 172.21.15.43:0/2028259601 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557de8f3b880 0x557de8f3dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.925+0000 7f5625010740 1 --2- 172.21.15.43:0/2028259601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557de8f32ca0 0x557de8f3b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.925+0000 7f5625010740 1 --2- 172.21.15.43:0/2028259601 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557de8f32380 0x557de8f32760 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.925+0000 7f5625010740 1 -- 172.21.15.43:0/2028259601 >> 172.21.15.43:0/2028259601 conn(0x557de8f2cab0 msgr2=0x557de8f2ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:47.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.925+0000 7f5625010740 1 -- 172.21.15.43:0/2028259601 shutdown_connections 2024-03-28T22:39:47.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.926+0000 7f5625010740 1 -- 172.21.15.43:0/2028259601 wait complete. 2024-03-28T22:39:47.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.926+0000 7f5625010740 1 -- start start 2024-03-28T22:39:47.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.926+0000 7f5625010740 1 Processor -- start 2024-03-28T22:39:47.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.927+0000 7f5625010740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557de8f32380 0x557de8f38950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.927+0000 7f5625010740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557de8f32ca0 0x557de8f38e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.927+0000 7f5625010740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557de8f3b880 0x557de8f393d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.927+0000 7f5625010740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x557de8f3e670 con 0x557de8f3b880 2024-03-28T22:39:47.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.927+0000 7f5625010740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x557de8e342f0 con 0x557de8f32ca0 2024-03-28T22:39:47.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.927+0000 7f5625010740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x557de8f3e7f0 con 0x557de8f32380 2024-03-28T22:39:47.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.927+0000 7f5625010740 10 client.0 _ll_register_callbacks cb 0x557de8ba9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:47.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.927+0000 7f5613e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557de8f32ca0 0x557de8f38e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.927+0000 7f5613e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557de8f32ca0 0x557de8f38e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52656/0 (socket says 172.21.15.43:52656) 2024-03-28T22:39:47.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.927+0000 7f5613e00640 1 -- 172.21.15.43:0/4240624601 learned_addr learned my addr 172.21.15.43:0/4240624601 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:47.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.927+0000 7f561be00640 1 --2- 172.21.15.43:0/4240624601 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557de8f32380 0x557de8f38950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.927+0000 7f561be00640 1 --2- 172.21.15.43:0/4240624601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557de8f3b880 0x557de8f393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.928+0000 7f5613e00640 1 -- 172.21.15.43:0/4240624601 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557de8f32380 msgr2=0x557de8f38950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:47.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.928+0000 7f5613e00640 1 --2- 172.21.15.43:0/4240624601 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557de8f32380 0x557de8f38950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.928+0000 7f5613e00640 1 -- 172.21.15.43:0/4240624601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557de8f3b880 msgr2=0x557de8f393d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:47.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.928+0000 7f5613e00640 1 --2- 172.21.15.43:0/4240624601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557de8f3b880 0x557de8f393d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.928+0000 7f5613e00640 1 -- 172.21.15.43:0/4240624601 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557de8c747d0 con 0x557de8f32ca0 2024-03-28T22:39:47.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.928+0000 7f5613e00640 1 --2- 172.21.15.43:0/4240624601 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557de8f32ca0 0x557de8f38e90 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f5604012700 tx=0x7f5604012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1ea43718e3dfb1e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:47.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.928+0000 7f5609600640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:39:47.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.928+0000 7f5609600640 1 -- 172.21.15.43:0/4240624601 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5604011c70 con 0x557de8f32ca0 2024-03-28T22:39:47.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.928+0000 7f5609600640 1 -- 172.21.15.43:0/4240624601 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7f5604011e10 con 0x557de8f32ca0 2024-03-28T22:39:47.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.928+0000 7f5625010740 1 -- 172.21.15.43:0/4240624601 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5614004d20 con 0x557de8f32ca0 2024-03-28T22:39:47.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.929+0000 7f5609600640 1 -- 172.21.15.43:0/4240624601 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5604025050 con 0x557de8f32ca0 2024-03-28T22:39:47.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.929+0000 7f5625010740 10 client.26579 fetch_fsmap learned FSMap version 30 2024-03-28T22:39:47.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.929+0000 7f5625010740 1 -- 172.21.15.43:0/4240624601 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557de8f4aa00 con 0x557de8f32ca0 2024-03-28T22:39:47.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.929+0000 7f5609600640 1 -- 172.21.15.43:0/4240624601 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5614004d20 con 0x557de8f32ca0 2024-03-28T22:39:47.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.929+0000 7f5609600640 1 -- 172.21.15.43:0/4240624601 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f560400f5d0 con 0x557de8f32ca0 2024-03-28T22:39:47.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.929+0000 7f5625010740 10 client.26579 fetch_fsmap finished waiting for FSMap version 30 2024-03-28T22:39:47.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.929+0000 7f5625010740 10 client.26579 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:39:47.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.930+0000 7f5625010740 20 client.26579 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:47.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.930+0000 7f5625010740 1 --2- 172.21.15.43:0/4240624601 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x557de8f4bf10 0x557de8f4e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.930+0000 7f5625010740 4 client.26579 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:39:47.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.930+0000 7f5625010740 1 -- 172.21.15.43:0/4240624601 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557de8e3f680 con 0x557de8f4bf10 2024-03-28T22:39:47.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.931+0000 7f561be00640 1 --2- 172.21.15.43:0/4240624601 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x557de8f4bf10 0x557de8f4e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.931+0000 7f561be00640 1 --2- 172.21.15.43:0/4240624601 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x557de8f4bf10 0x557de8f4e330 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=686a3b5c8b173196 server_cookie=f1060f4cc5989ca0 in_seq=0 out_seq=0 2024-03-28T22:39:47.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.931+0000 7f5609600640 10 client.26579 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:39:47.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.932+0000 7f5609600640 1 -- 172.21.15.43:0/4240624601 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557de8e3f680 con 0x557de8f4bf10 2024-03-28T22:39:47.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.932+0000 7f5609600640 10 client.26579 handle_command_reply: tid=0 2024-03-28T22:39:47.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.932+0000 7f5625010740 1 client.26579 shutdown 2024-03-28T22:39:47.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.932+0000 7f5625010740 1 -- 172.21.15.43:0/4240624601 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557de8f32ca0 msgr2=0x557de8f38e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:47.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.932+0000 7f5625010740 1 --2- 172.21.15.43:0/4240624601 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557de8f32ca0 0x557de8f38e90 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f5604012700 tx=0x7f5604012bc0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.933+0000 7f5625010740 1 -- 172.21.15.43:0/4240624601 shutdown_connections 2024-03-28T22:39:47.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.933+0000 7f5625010740 1 --2- 172.21.15.43:0/4240624601 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x557de8f4bf10 0x557de8f4e330 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.933+0000 7f5625010740 1 --2- 172.21.15.43:0/4240624601 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557de8f32ca0 0x557de8f38e90 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.933+0000 7f5625010740 1 --2- 172.21.15.43:0/4240624601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557de8f3b880 0x557de8f393d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.933+0000 7f5625010740 1 --2- 172.21.15.43:0/4240624601 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557de8f32380 0x557de8f38950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.933+0000 7f5609600640 0 client.26579 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:39:47.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.933+0000 7f5625010740 1 -- 172.21.15.43:0/4240624601 >> 172.21.15.43:0/4240624601 conn(0x557de8f2cab0 msgr2=0x557de8f2e800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:47.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.933+0000 7f5625010740 1 -- 172.21.15.43:0/4240624601 shutdown_connections 2024-03-28T22:39:47.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.933+0000 7f5625010740 1 -- 172.21.15.43:0/4240624601 wait complete. 2024-03-28T22:39:47.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.933+0000 7f5625010740 20 client.26579 trim_cache size 0 max 16384 2024-03-28T22:39:47.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.948+0000 7f5625010740 1 Processor -- start 2024-03-28T22:39:47.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.948+0000 7f5625010740 1 -- start start 2024-03-28T22:39:47.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.948+0000 7f5625010740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557de8f32380 0x557de8f47410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.948+0000 7f5625010740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557de8f32ca0 0x557de8f391f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.948+0000 7f5625010740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557de8f3b880 0x557de8f53b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.948+0000 7f5625010740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f561c1b0d90 con 0x557de8f3b880 2024-03-28T22:39:47.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.948+0000 7f5625010740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f561c1b28d0 con 0x557de8f32ca0 2024-03-28T22:39:47.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.948+0000 7f5625010740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f561c1b1460 con 0x557de8f32380 2024-03-28T22:39:47.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.949+0000 7f5613e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557de8f32ca0 0x557de8f391f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.949+0000 7f561be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557de8f32380 0x557de8f47410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.949+0000 7f5613e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557de8f32ca0 0x557de8f391f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52658/0 (socket says 172.21.15.43:52658) 2024-03-28T22:39:47.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.949+0000 7f5613e00640 1 -- 172.21.15.43:0/3463267054 learned_addr learned my addr 172.21.15.43:0/3463267054 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:47.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.949+0000 7f561be00640 1 --2- 172.21.15.43:0/3463267054 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557de8f3b880 0x557de8f53b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:47 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/118689753' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:39:47.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.949+0000 7f5613e00640 1 -- 172.21.15.43:0/3463267054 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557de8f32380 msgr2=0x557de8f47410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:47.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.949+0000 7f5613e00640 1 --2- 172.21.15.43:0/3463267054 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557de8f32380 0x557de8f47410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.949+0000 7f5613e00640 1 -- 172.21.15.43:0/3463267054 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557de8f3b880 msgr2=0x557de8f53b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:47.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.949+0000 7f5613e00640 1 --2- 172.21.15.43:0/3463267054 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557de8f3b880 0x557de8f53b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.949+0000 7f5613e00640 1 -- 172.21.15.43:0/3463267054 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557de8f32ad0 con 0x557de8f32ca0 2024-03-28T22:39:47.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.949+0000 7f5613e00640 1 --2- 172.21.15.43:0/3463267054 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557de8f32ca0 0x557de8f391f0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f56040092e0 tx=0x7f56040094f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5351a170db616b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:47.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.950+0000 7f560be00640 1 -- 172.21.15.43:0/3463267054 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f560402d020 con 0x557de8f32ca0 2024-03-28T22:39:47.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.950+0000 7f560be00640 1 -- 172.21.15.43:0/3463267054 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5604011c70 con 0x557de8f32ca0 2024-03-28T22:39:47.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.950+0000 7f560be00640 1 -- 172.21.15.43:0/3463267054 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f560401c660 con 0x557de8f32ca0 2024-03-28T22:39:47.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.950+0000 7f5625010740 1 -- 172.21.15.43:0/3463267054 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557de8f32ca0 msgr2=0x557de8f391f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:47.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.950+0000 7f5625010740 1 --2- 172.21.15.43:0/3463267054 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557de8f32ca0 0x557de8f391f0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f56040092e0 tx=0x7f56040094f0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.950+0000 7f5625010740 1 -- 172.21.15.43:0/3463267054 shutdown_connections 2024-03-28T22:39:47.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.950+0000 7f5625010740 1 --2- 172.21.15.43:0/3463267054 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557de8f32ca0 0x557de8f391f0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.950+0000 7f5625010740 1 --2- 172.21.15.43:0/3463267054 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557de8f3b880 0x557de8f53b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.950+0000 7f5625010740 1 --2- 172.21.15.43:0/3463267054 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557de8f32380 0x557de8f47410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.950+0000 7f5625010740 1 -- 172.21.15.43:0/3463267054 >> 172.21.15.43:0/3463267054 conn(0x557de8f2cab0 msgr2=0x557de8f2b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:47.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.951+0000 7f5625010740 1 -- 172.21.15.43:0/3463267054 shutdown_connections 2024-03-28T22:39:47.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.951+0000 7f5625010740 1 -- 172.21.15.43:0/3463267054 wait complete. 2024-03-28T22:39:47.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.951+0000 7f5625010740 1 -- start start 2024-03-28T22:39:47.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.951+0000 7f5625010740 1 Processor -- start 2024-03-28T22:39:47.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.952+0000 7f5625010740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557de8f32380 0x557de8f43ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.952+0000 7f5625010740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557de8f32ca0 0x557de8f3f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.952+0000 7f5625010740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557de8f3b880 0x557de8f3f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.952+0000 7f5625010740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f561c1b2200 con 0x557de8f32380 2024-03-28T22:39:47.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.952+0000 7f5625010740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f561c1b1b30 con 0x557de8f32ca0 2024-03-28T22:39:47.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.952+0000 7f5625010740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f561c1aed70 con 0x557de8f3b880 2024-03-28T22:39:47.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.952+0000 7f5625010740 10 client.0 _ll_register_callbacks cb 0x7f5604009df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:47.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.952+0000 7f561be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557de8f32380 0x557de8f43ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.952+0000 7f5613e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557de8f32ca0 0x557de8f3f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.952+0000 7f561be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557de8f32380 0x557de8f43ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44116/0 (socket says 172.21.15.43:44116) 2024-03-28T22:39:47.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.952+0000 7f561be00640 1 -- 172.21.15.43:0/4023213975 learned_addr learned my addr 172.21.15.43:0/4023213975 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:47.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.952+0000 7f561be00640 1 --2- 172.21.15.43:0/4023213975 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557de8f3b880 0x557de8f3f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.952+0000 7f5613e00640 1 -- 172.21.15.43:0/4023213975 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557de8f3b880 msgr2=0x557de8f3f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:47.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.952+0000 7f5613e00640 1 --2- 172.21.15.43:0/4023213975 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557de8f3b880 0x557de8f3f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.952+0000 7f5613e00640 1 -- 172.21.15.43:0/4023213975 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557de8f32380 msgr2=0x557de8f43ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:47.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.952+0000 7f5613e00640 1 --2- 172.21.15.43:0/4023213975 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557de8f32380 0x557de8f43ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.952+0000 7f5613e00640 1 -- 172.21.15.43:0/4023213975 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557de8f3fda0 con 0x557de8f32ca0 2024-03-28T22:39:47.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.953+0000 7f5613e00640 1 --2- 172.21.15.43:0/4023213975 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557de8f32ca0 0x557de8f3f080 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x557de8f023d0 tx=0x7f560401c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc5c67a43bc60be server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:47.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.953+0000 7f5609600640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:39:47.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.953+0000 7f5609600640 1 -- 172.21.15.43:0/4023213975 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5604038070 con 0x557de8f32ca0 2024-03-28T22:39:47.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.953+0000 7f5625010740 1 -- 172.21.15.43:0/4023213975 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5604011c70 con 0x557de8f32ca0 2024-03-28T22:39:47.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.953+0000 7f5609600640 1 -- 172.21.15.43:0/4023213975 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7f5604022040 con 0x557de8f32ca0 2024-03-28T22:39:47.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.953+0000 7f5609600640 1 -- 172.21.15.43:0/4023213975 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5604041050 con 0x557de8f32ca0 2024-03-28T22:39:47.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.953+0000 7f5625010740 10 client.26585 fetch_fsmap learned FSMap version 30 2024-03-28T22:39:47.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.953+0000 7f5625010740 1 -- 172.21.15.43:0/4023213975 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557de8fb21c0 con 0x557de8f32ca0 2024-03-28T22:39:47.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.954+0000 7f5609600640 1 -- 172.21.15.43:0/4023213975 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5604011c70 con 0x557de8f32ca0 2024-03-28T22:39:47.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.954+0000 7f5609600640 1 -- 172.21.15.43:0/4023213975 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f560403b020 con 0x557de8f32ca0 2024-03-28T22:39:47.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.954+0000 7f5625010740 10 client.26585 fetch_fsmap finished waiting for FSMap version 30 2024-03-28T22:39:47.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.954+0000 7f5625010740 10 client.26585 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:39:47.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.954+0000 7f5625010740 20 client.26585 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:47.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.954+0000 7f5625010740 1 --2- 172.21.15.43:0/4023213975 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x557de8fb2870 0x557de8fb4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:47.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.954+0000 7f5625010740 4 client.26585 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:39:47.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.954+0000 7f5625010740 1 -- 172.21.15.43:0/4023213975 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557de8ea3590 con 0x557de8fb2870 2024-03-28T22:39:47.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.954+0000 7f561be00640 1 --2- 172.21.15.43:0/4023213975 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x557de8fb2870 0x557de8fb4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:47.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.955+0000 7f561be00640 1 --2- 172.21.15.43:0/4023213975 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x557de8fb2870 0x557de8fb4c60 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e7ef770d1527d0cd server_cookie=ba86528aa713c9b3 in_seq=0 out_seq=0 2024-03-28T22:39:47.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.955+0000 7f5609600640 10 client.26585 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:39:47.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.955+0000 7f5609600640 1 -- 172.21.15.43:0/4023213975 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557de8ea3590 con 0x557de8fb2870 2024-03-28T22:39:47.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.955+0000 7f5609600640 10 client.26585 handle_command_reply: tid=0 2024-03-28T22:39:47.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.955+0000 7f5625010740 1 client.26585 shutdown 2024-03-28T22:39:47.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.955+0000 7f5625010740 1 -- 172.21.15.43:0/4023213975 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557de8f32ca0 msgr2=0x557de8f3f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:47.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.955+0000 7f5625010740 1 --2- 172.21.15.43:0/4023213975 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557de8f32ca0 0x557de8f3f080 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x557de8f023d0 tx=0x7f560401c4e0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.955+0000 7f5625010740 1 -- 172.21.15.43:0/4023213975 shutdown_connections 2024-03-28T22:39:47.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.955+0000 7f5625010740 1 --2- 172.21.15.43:0/4023213975 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x557de8fb2870 0x557de8fb4c60 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.955+0000 7f5625010740 1 --2- 172.21.15.43:0/4023213975 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557de8f32ca0 0x557de8f3f080 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.955+0000 7f5625010740 1 --2- 172.21.15.43:0/4023213975 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557de8f3b880 0x557de8f3f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.955+0000 7f5625010740 1 --2- 172.21.15.43:0/4023213975 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557de8f32380 0x557de8f43ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.955+0000 7f5625010740 1 -- 172.21.15.43:0/4023213975 >> 172.21.15.43:0/4023213975 conn(0x557de8f2cab0 msgr2=0x557de8f2dc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:47.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.955+0000 7f5609600640 0 client.26585 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:39:47.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.956+0000 7f5625010740 1 -- 172.21.15.43:0/4023213975 shutdown_connections 2024-03-28T22:39:47.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.956+0000 7f5625010740 1 -- 172.21.15.43:0/4023213975 wait complete. 2024-03-28T22:39:47.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.956+0000 7f5625010740 20 client.26585 trim_cache size 0 max 16384 2024-03-28T22:39:47.960 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:39:47.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.956+0000 7f5622400640 1 -- 172.21.15.43:0/1447203246 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f56000a1f70 msgr2=0x7f56000a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:47.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.956+0000 7f5622400640 1 --2- 172.21.15.43:0/1447203246 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f56000a1f70 0x7f56000a4430 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x557de8f31e50 tx=0x7f560c009260 comp rx=0 tx=0).stop 2024-03-28T22:39:47.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.956+0000 7f5622400640 1 -- 172.21.15.43:0/1447203246 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f561c100050 msgr2=0x7f561c19a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:47.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.956+0000 7f5622400640 1 --2- 172.21.15.43:0/1447203246 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f561c100050 0x7f561c19a330 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f560c009900 tx=0x7f560c004360 comp rx=0 tx=0).stop 2024-03-28T22:39:47.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.956+0000 7f5622400640 1 -- 172.21.15.43:0/1447203246 shutdown_connections 2024-03-28T22:39:47.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.956+0000 7f5622400640 1 --2- 172.21.15.43:0/1447203246 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f56000a1f70 0x7f56000a4430 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.956+0000 7f5622400640 1 --2- 172.21.15.43:0/1447203246 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f561c106730 0x7f561c19eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.956+0000 7f5622400640 1 --2- 172.21.15.43:0/1447203246 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f561c100a10 0x7f561c19a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.956+0000 7f5622400640 1 --2- 172.21.15.43:0/1447203246 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f561c100050 0x7f561c19a330 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:47.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.956+0000 7f5622400640 1 -- 172.21.15.43:0/1447203246 >> 172.21.15.43:0/1447203246 conn(0x7f561c075930 msgr2=0x7f561c0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:47.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.957+0000 7f5622400640 1 -- 172.21.15.43:0/1447203246 shutdown_connections 2024-03-28T22:39:47.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:47.957+0000 7f5622400640 1 -- 172.21.15.43:0/1447203246 wait complete. 2024-03-28T22:39:47.970 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:47 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/118689753' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:39:47.974 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:39:48.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.199+0000 7f818bc00640 1 -- 172.21.15.43:0/2520221658 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8184107f60 msgr2=0x7f8184108340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:48.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.199+0000 7f818bc00640 1 --2- 172.21.15.43:0/2520221658 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8184107f60 0x7f8184108340 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8174009900 tx=0x7f817402bc30 comp rx=0 tx=0).stop 2024-03-28T22:39:48.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.199+0000 7f818bc00640 1 -- 172.21.15.43:0/2520221658 shutdown_connections 2024-03-28T22:39:48.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.199+0000 7f818bc00640 1 --2- 172.21.15.43:0/2520221658 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f818410d030 0x7f8184114c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.199+0000 7f818bc00640 1 --2- 172.21.15.43:0/2520221658 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8184108910 0x7f818410c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.199+0000 7f818bc00640 1 --2- 172.21.15.43:0/2520221658 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8184107f60 0x7f8184108340 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.199+0000 7f818bc00640 1 -- 172.21.15.43:0/2520221658 >> 172.21.15.43:0/2520221658 conn(0x7f8184075c90 msgr2=0x7f81840780b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:48.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.199+0000 7f818bc00640 1 -- 172.21.15.43:0/2520221658 shutdown_connections 2024-03-28T22:39:48.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.199+0000 7f818bc00640 1 -- 172.21.15.43:0/2520221658 wait complete. 2024-03-28T22:39:48.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.200+0000 7f818bc00640 1 Processor -- start 2024-03-28T22:39:48.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.200+0000 7f818bc00640 1 -- start start 2024-03-28T22:39:48.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.201+0000 7f818bc00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8184107f60 0x7f818419e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.201+0000 7f818bc00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f818410d030 0x7f818419eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.201+0000 7f818bc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81841a2ed0 0x7f81841a3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.201+0000 7f818bc00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f81841180d0 con 0x7f81841a2ed0 2024-03-28T22:39:48.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.201+0000 7f818bc00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8184117f50 con 0x7f818410d030 2024-03-28T22:39:48.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.201+0000 7f818bc00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f8184118250 con 0x7f8184107f60 2024-03-28T22:39:48.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.201+0000 7f817be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f818410d030 0x7f818419eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.201+0000 7f8189200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8184107f60 0x7f818419e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.201+0000 7f8189200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8184107f60 0x7f818419e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:56072/0 (socket says 172.21.15.43:56072) 2024-03-28T22:39:48.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.201+0000 7f8189200640 1 -- 172.21.15.43:0/1875976326 learned_addr learned my addr 172.21.15.43:0/1875976326 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:48.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.201+0000 7f817be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f818410d030 0x7f818419eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52682/0 (socket says 172.21.15.43:52682) 2024-03-28T22:39:48.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.201+0000 7f8189c00640 1 --2- 172.21.15.43:0/1875976326 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81841a2ed0 0x7f81841a3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.201+0000 7f817be00640 1 -- 172.21.15.43:0/1875976326 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8184107f60 msgr2=0x7f818419e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:48.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.201+0000 7f817be00640 1 --2- 172.21.15.43:0/1875976326 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8184107f60 0x7f818419e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.201+0000 7f817be00640 1 -- 172.21.15.43:0/1875976326 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81841a2ed0 msgr2=0x7f81841a3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:48.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.201+0000 7f817be00640 1 --2- 172.21.15.43:0/1875976326 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81841a2ed0 0x7f81841a3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.201+0000 7f817be00640 1 -- 172.21.15.43:0/1875976326 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81841a39c0 con 0x7f818410d030 2024-03-28T22:39:48.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.201+0000 7f8189200640 1 --2- 172.21.15.43:0/1875976326 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8184107f60 0x7f818419e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:48.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.202+0000 7f817be00640 1 --2- 172.21.15.43:0/1875976326 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f818410d030 0x7f818419eb50 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f816c00e9c0 tx=0x7f816c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:48.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.202+0000 7f8182000640 1 -- 172.21.15.43:0/1875976326 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816c00ccc0 con 0x7f818410d030 2024-03-28T22:39:48.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.202+0000 7f818bc00640 1 -- 172.21.15.43:0/1875976326 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81841a3cb0 con 0x7f818410d030 2024-03-28T22:39:48.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.202+0000 7f8182000640 1 -- 172.21.15.43:0/1875976326 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f816c010910 con 0x7f818410d030 2024-03-28T22:39:48.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.202+0000 7f8182000640 1 -- 172.21.15.43:0/1875976326 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816c018880 con 0x7f818410d030 2024-03-28T22:39:48.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.202+0000 7f818bc00640 1 -- 172.21.15.43:0/1875976326 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81841ab4a0 con 0x7f818410d030 2024-03-28T22:39:48.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.206+0000 7f818e967740 1 Processor -- start 2024-03-28T22:39:48.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.207+0000 7f818e967740 1 -- start start 2024-03-28T22:39:48.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.207+0000 7f818e967740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5581632d9380 0x5581632d9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.207+0000 7f818e967740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5581632d9ca0 0x5581632e2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.207+0000 7f818e967740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5581632e2880 0x5581632e4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.207+0000 7f818e967740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55816323d050 con 0x5581632d9380 2024-03-28T22:39:48.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.207+0000 7f818e967740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5581631e5ce0 con 0x5581632e2880 2024-03-28T22:39:48.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.207+0000 7f818e967740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5581632390f0 con 0x5581632d9ca0 2024-03-28T22:39:48.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.207+0000 7f8182000640 1 -- 172.21.15.43:0/1875976326 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f816c010ac0 con 0x7f818410d030 2024-03-28T22:39:48.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.207+0000 7f8189200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5581632d9ca0 0x5581632e2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.207+0000 7f8189c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5581632d9380 0x5581632d9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.207+0000 7f8189200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5581632d9ca0 0x5581632e2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:56082/0 (socket says 172.21.15.43:56082) 2024-03-28T22:39:48.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.207+0000 7f8189200640 1 -- 172.21.15.43:0/1883149778 learned_addr learned my addr 172.21.15.43:0/1883149778 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:48.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.207+0000 7f817be00640 1 --2- 172.21.15.43:0/1883149778 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5581632e2880 0x5581632e4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.208+0000 7f8189200640 1 -- 172.21.15.43:0/1883149778 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5581632e2880 msgr2=0x5581632e4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:48.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.208+0000 7f8189200640 1 --2- 172.21.15.43:0/1883149778 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5581632e2880 0x5581632e4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.208+0000 7f8189200640 1 -- 172.21.15.43:0/1883149778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5581632d9380 msgr2=0x5581632d9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:48.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.208+0000 7f8189200640 1 --2- 172.21.15.43:0/1883149778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5581632d9380 0x5581632d9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.208+0000 7f8189200640 1 -- 172.21.15.43:0/1883149778 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558162f513f0 con 0x5581632d9ca0 2024-03-28T22:39:48.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.208+0000 7f817be00640 1 --2- 172.21.15.43:0/1883149778 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5581632e2880 0x5581632e4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:39:48.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.208+0000 7f8189200640 1 --2- 172.21.15.43:0/1883149778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5581632d9ca0 0x5581632e2340 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8174036040 tx=0x7f8174009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec3d7e13414deef3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:48.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.208+0000 7f817a000640 1 -- 172.21.15.43:0/1883149778 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8174042070 con 0x5581632d9ca0 2024-03-28T22:39:48.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.208+0000 7f817a000640 1 -- 172.21.15.43:0/1883149778 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8174004a70 con 0x5581632d9ca0 2024-03-28T22:39:48.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.209+0000 7f817a000640 1 -- 172.21.15.43:0/1883149778 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8174005280 con 0x5581632d9ca0 2024-03-28T22:39:48.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.209+0000 7f8182000640 1 --2- 172.21.15.43:0/1875976326 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f81680a1ef0 0x7f81680a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.209+0000 7f818e967740 1 -- 172.21.15.43:0/1883149778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5581632d9ca0 msgr2=0x5581632e2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:48.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.209+0000 7f818e967740 1 --2- 172.21.15.43:0/1883149778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5581632d9ca0 0x5581632e2340 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8174036040 tx=0x7f8174009560 comp rx=0 tx=0).stop 2024-03-28T22:39:48.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.209+0000 7f818e967740 1 -- 172.21.15.43:0/1883149778 shutdown_connections 2024-03-28T22:39:48.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.209+0000 7f818e967740 1 --2- 172.21.15.43:0/1883149778 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5581632e2880 0x5581632e4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.209+0000 7f818e967740 1 --2- 172.21.15.43:0/1883149778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5581632d9ca0 0x5581632e2340 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.209+0000 7f818e967740 1 --2- 172.21.15.43:0/1883149778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5581632d9380 0x5581632d9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.209+0000 7f8182000640 1 -- 172.21.15.43:0/1875976326 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7f816c0ce910 con 0x7f818410d030 2024-03-28T22:39:48.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.209+0000 7f818e967740 1 -- 172.21.15.43:0/1883149778 >> 172.21.15.43:0/1883149778 conn(0x5581632d3ab0 msgr2=0x5581632d3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:48.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.209+0000 7f8189c00640 1 --2- 172.21.15.43:0/1875976326 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f81680a1ef0 0x7f81680a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.210+0000 7f818e967740 1 -- 172.21.15.43:0/1883149778 shutdown_connections 2024-03-28T22:39:48.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.210+0000 7f818e967740 1 -- 172.21.15.43:0/1883149778 wait complete. 2024-03-28T22:39:48.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.210+0000 7f8189c00640 1 --2- 172.21.15.43:0/1875976326 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f81680a1ef0 0x7f81680a43b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f817c005fd0 tx=0x7f817c009500 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:48.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.210+0000 7f818e967740 1 -- start start 2024-03-28T22:39:48.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.210+0000 7f818e967740 1 Processor -- start 2024-03-28T22:39:48.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.211+0000 7f818e967740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5581632d9380 0x5581632df920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.211+0000 7f818e967740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5581632d9ca0 0x5581632dfe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.211+0000 7f818e967740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5581632e2880 0x5581632e03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.211+0000 7f818e967740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5581632e5610 con 0x5581632d9380 2024-03-28T22:39:48.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.211+0000 7f818e967740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5581631db2f0 con 0x5581632d9ca0 2024-03-28T22:39:48.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.211+0000 7f818e967740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5581632e5790 con 0x5581632e2880 2024-03-28T22:39:48.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.211+0000 7f817be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5581632d9ca0 0x5581632dfe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.211+0000 7f817be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5581632d9ca0 0x5581632dfe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52704/0 (socket says 172.21.15.43:52704) 2024-03-28T22:39:48.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.211+0000 7f817be00640 1 -- 172.21.15.43:0/3130941058 learned_addr learned my addr 172.21.15.43:0/3130941058 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:48.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.211+0000 7f8189200640 1 --2- 172.21.15.43:0/3130941058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5581632d9380 0x5581632df920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.211+0000 7f818e967740 10 client.0 _ll_register_callbacks cb 0x558162f50be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:48.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.211+0000 7f817be00640 1 -- 172.21.15.43:0/3130941058 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5581632e2880 msgr2=0x5581632e03a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T22:39:48.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.211+0000 7f817be00640 1 --2- 172.21.15.43:0/3130941058 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5581632e2880 0x5581632e03a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.212+0000 7f817be00640 1 -- 172.21.15.43:0/3130941058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5581632d9380 msgr2=0x5581632df920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:48.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.212+0000 7f817be00640 1 --2- 172.21.15.43:0/3130941058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5581632d9380 0x5581632df920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.212+0000 7f817be00640 1 -- 172.21.15.43:0/3130941058 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55816301b7d0 con 0x5581632d9ca0 2024-03-28T22:39:48.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.212+0000 7f8189200640 1 --2- 172.21.15.43:0/3130941058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5581632d9380 0x5581632df920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:48.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.212+0000 7f818e967740 1 -- 172.21.15.43:0/3130941058 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8174004a70 con 0x5581632d9ca0 2024-03-28T22:39:48.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.212+0000 7f817be00640 1 --2- 172.21.15.43:0/3130941058 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5581632d9ca0 0x5581632dfe60 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f816c09acb0 tx=0x7f816c09a910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1330622f4e6c422 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:48.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.212+0000 7f8172a00640 10 client.26597 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:39:48.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.212+0000 7f8172a00640 1 -- 172.21.15.43:0/3130941058 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816c0d0e60 con 0x5581632d9ca0 2024-03-28T22:39:48.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.212+0000 7f8172a00640 1 -- 172.21.15.43:0/3130941058 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7f816c09a600 con 0x5581632d9ca0 2024-03-28T22:39:48.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.212+0000 7f8172a00640 1 -- 172.21.15.43:0/3130941058 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f816c0cc0f0 con 0x5581632d9ca0 2024-03-28T22:39:48.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.212+0000 7f818e967740 10 client.26597 fetch_fsmap learned FSMap version 30 2024-03-28T22:39:48.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.212+0000 7f818e967740 1 -- 172.21.15.43:0/3130941058 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({fsmap=30}) -- 0x5581632f1830 con 0x5581632d9ca0 2024-03-28T22:39:48.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.213+0000 7f8172a00640 1 -- 172.21.15.43:0/3130941058 <== mon.2 v2:172.21.15.82:3300/0 4 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f816c095b20 con 0x5581632d9ca0 2024-03-28T22:39:48.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.213+0000 7f818e967740 10 client.26597 fetch_fsmap finished waiting for FSMap version 30 2024-03-28T22:39:48.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.213+0000 7f818e967740 10 client.26597 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:39:48.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.213+0000 7f818e967740 20 client.26597 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:48.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.213+0000 7f818e967740 1 --2- 172.21.15.43:0/3130941058 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5581632f2be0 0x5581632f5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.213+0000 7f818e967740 4 client.26597 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:39:48.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.213+0000 7f818e967740 1 -- 172.21.15.43:0/3130941058 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5581631e6680 con 0x5581632f2be0 2024-03-28T22:39:48.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.213+0000 7f8189200640 1 --2- 172.21.15.43:0/3130941058 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5581632f2be0 0x5581632f5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.214+0000 7f8189200640 1 --2- 172.21.15.43:0/3130941058 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5581632f2be0 0x5581632f5050 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=43b87a62f9f555ae server_cookie=a1f9327456dca4dd in_seq=0 out_seq=0 2024-03-28T22:39:48.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.214+0000 7f8172a00640 10 client.26597 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:39:48.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.215+0000 7f8172a00640 1 -- 172.21.15.43:0/3130941058 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5581631e6680 con 0x5581632f2be0 2024-03-28T22:39:48.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.215+0000 7f8172a00640 10 client.26597 handle_command_reply: tid=0 2024-03-28T22:39:48.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.215+0000 7f818e967740 1 client.26597 shutdown 2024-03-28T22:39:48.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.215+0000 7f818e967740 1 -- 172.21.15.43:0/3130941058 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5581632d9ca0 msgr2=0x5581632dfe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:48.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.215+0000 7f818e967740 1 --2- 172.21.15.43:0/3130941058 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5581632d9ca0 0x5581632dfe60 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f816c09acb0 tx=0x7f816c09a910 comp rx=0 tx=0).stop 2024-03-28T22:39:48.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.215+0000 7f818e967740 1 -- 172.21.15.43:0/3130941058 shutdown_connections 2024-03-28T22:39:48.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.215+0000 7f818e967740 1 --2- 172.21.15.43:0/3130941058 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5581632f2be0 0x5581632f5050 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.215+0000 7f818e967740 1 --2- 172.21.15.43:0/3130941058 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5581632d9ca0 0x5581632dfe60 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.215+0000 7f818e967740 1 --2- 172.21.15.43:0/3130941058 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5581632e2880 0x5581632e03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.215+0000 7f818e967740 1 --2- 172.21.15.43:0/3130941058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5581632d9380 0x5581632df920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.215+0000 7f8172a00640 0 client.26597 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:39:48.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.215+0000 7f818e967740 1 -- 172.21.15.43:0/3130941058 >> 172.21.15.43:0/3130941058 conn(0x5581632d3ab0 msgr2=0x5581632d5800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:48.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.215+0000 7f818e967740 1 -- 172.21.15.43:0/3130941058 shutdown_connections 2024-03-28T22:39:48.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.215+0000 7f818e967740 1 -- 172.21.15.43:0/3130941058 wait complete. 2024-03-28T22:39:48.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.215+0000 7f818e967740 20 client.26597 trim_cache size 0 max 16384 2024-03-28T22:39:48.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.230+0000 7f818e967740 1 Processor -- start 2024-03-28T22:39:48.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.230+0000 7f818e967740 1 -- start start 2024-03-28T22:39:48.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.230+0000 7f818e967740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5581632d9380 0x5581632e2050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.230+0000 7f818e967740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5581632d9ca0 0x5581632ee3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.230+0000 7f818e967740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5581632e2880 0x5581632fa920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.230+0000 7f818e967740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f81841b5160 con 0x5581632e2880 2024-03-28T22:39:48.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.230+0000 7f818e967740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f81841b6ca0 con 0x5581632d9ca0 2024-03-28T22:39:48.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.230+0000 7f818e967740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f81841b5830 con 0x5581632d9380 2024-03-28T22:39:48.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.231+0000 7f8189200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5581632d9380 0x5581632e2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.231+0000 7f817be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5581632d9ca0 0x5581632ee3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.231+0000 7f817be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5581632d9ca0 0x5581632ee3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52720/0 (socket says 172.21.15.43:52720) 2024-03-28T22:39:48.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.231+0000 7f817be00640 1 -- 172.21.15.43:0/237414352 learned_addr learned my addr 172.21.15.43:0/237414352 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:48.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.231+0000 7f8189200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5581632d9380 0x5581632e2050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:56092/0 (socket says 172.21.15.43:56092) 2024-03-28T22:39:48.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.231+0000 7f8189c00640 1 --2- 172.21.15.43:0/237414352 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5581632e2880 0x5581632fa920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.231+0000 7f817be00640 1 -- 172.21.15.43:0/237414352 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5581632d9380 msgr2=0x5581632e2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:48.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.231+0000 7f817be00640 1 --2- 172.21.15.43:0/237414352 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5581632d9380 0x5581632e2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.231+0000 7f817be00640 1 -- 172.21.15.43:0/237414352 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5581632e2880 msgr2=0x5581632fa920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:48.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.231+0000 7f817be00640 1 --2- 172.21.15.43:0/237414352 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5581632e2880 0x5581632fa920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.231+0000 7f817be00640 1 -- 172.21.15.43:0/237414352 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581632cdf80 con 0x5581632d9ca0 2024-03-28T22:39:48.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.231+0000 7f8189c00640 1 --2- 172.21.15.43:0/237414352 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5581632e2880 0x5581632fa920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:39:48.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.231+0000 7f817be00640 1 --2- 172.21.15.43:0/237414352 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5581632d9ca0 0x5581632ee3b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f816c0a3170 tx=0x7f816c0cd000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c454f3cbb952f533 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:48.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.232+0000 7f8179600640 1 -- 172.21.15.43:0/237414352 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816c09a400 con 0x5581632d9ca0 2024-03-28T22:39:48.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.232+0000 7f8179600640 1 -- 172.21.15.43:0/237414352 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8174004a70 con 0x5581632d9ca0 2024-03-28T22:39:48.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.232+0000 7f8179600640 1 -- 172.21.15.43:0/237414352 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816c096f60 con 0x5581632d9ca0 2024-03-28T22:39:48.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.232+0000 7f818e967740 1 -- 172.21.15.43:0/237414352 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5581632d9ca0 msgr2=0x5581632ee3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:48.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.232+0000 7f818e967740 1 --2- 172.21.15.43:0/237414352 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5581632d9ca0 0x5581632ee3b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f816c0a3170 tx=0x7f816c0cd000 comp rx=0 tx=0).stop 2024-03-28T22:39:48.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.232+0000 7f818e967740 1 -- 172.21.15.43:0/237414352 shutdown_connections 2024-03-28T22:39:48.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.232+0000 7f818e967740 1 --2- 172.21.15.43:0/237414352 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5581632d9ca0 0x5581632ee3b0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.232+0000 7f818e967740 1 --2- 172.21.15.43:0/237414352 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5581632e2880 0x5581632fa920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.232+0000 7f818e967740 1 --2- 172.21.15.43:0/237414352 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5581632d9380 0x5581632e2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.232+0000 7f818e967740 1 -- 172.21.15.43:0/237414352 >> 172.21.15.43:0/237414352 conn(0x5581632d3ab0 msgr2=0x5581632e0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:48.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.233+0000 7f818e967740 1 -- 172.21.15.43:0/237414352 shutdown_connections 2024-03-28T22:39:48.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.233+0000 7f818e967740 1 -- 172.21.15.43:0/237414352 wait complete. 2024-03-28T22:39:48.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.233+0000 7f818e967740 1 -- start start 2024-03-28T22:39:48.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.233+0000 7f818e967740 1 Processor -- start 2024-03-28T22:39:48.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.234+0000 7f818e967740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5581632d9380 0x5581632dc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.234+0000 7f818e967740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5581632d9ca0 0x5581632dcca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.234+0000 7f818e967740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5581632e2880 0x5581632dd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.234+0000 7f818e967740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f81841b65d0 con 0x5581632d9ca0 2024-03-28T22:39:48.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.234+0000 7f818e967740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f81841b5f00 con 0x5581632d9380 2024-03-28T22:39:48.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.234+0000 7f818e967740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f81841b3140 con 0x5581632e2880 2024-03-28T22:39:48.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.234+0000 7f818e967740 10 client.0 _ll_register_callbacks cb 0x7f816c09ad80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:48.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.234+0000 7f8189200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5581632d9380 0x5581632dc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.234+0000 7f8189200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5581632d9380 0x5581632dc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52734/0 (socket says 172.21.15.43:52734) 2024-03-28T22:39:48.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.234+0000 7f8189200640 1 -- 172.21.15.43:0/516334286 learned_addr learned my addr 172.21.15.43:0/516334286 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:48.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.234+0000 7f817be00640 1 --2- 172.21.15.43:0/516334286 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5581632d9ca0 0x5581632dcca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.234+0000 7f8189c00640 1 --2- 172.21.15.43:0/516334286 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5581632e2880 0x5581632dd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.234+0000 7f8189200640 1 -- 172.21.15.43:0/516334286 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5581632e2880 msgr2=0x5581632dd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:48.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.234+0000 7f8189200640 1 --2- 172.21.15.43:0/516334286 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5581632e2880 0x5581632dd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.234+0000 7f8189200640 1 -- 172.21.15.43:0/516334286 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5581632d9ca0 msgr2=0x5581632dcca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:48.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.234+0000 7f8189200640 1 --2- 172.21.15.43:0/516334286 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5581632d9ca0 0x5581632dcca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.234+0000 7f8189200640 1 -- 172.21.15.43:0/516334286 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581633581f0 con 0x5581632d9380 2024-03-28T22:39:48.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.235+0000 7f8189200640 1 --2- 172.21.15.43:0/516334286 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5581632d9380 0x5581632dc760 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f8174006010 tx=0x7f8174004590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2750c6a9b336e3d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:48.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.235+0000 7f8172a00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:39:48.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.235+0000 7f8172a00640 1 -- 172.21.15.43:0/516334286 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8174053070 con 0x5581632d9380 2024-03-28T22:39:48.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.235+0000 7f8172a00640 1 -- 172.21.15.43:0/516334286 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7f817404fc30 con 0x5581632d9380 2024-03-28T22:39:48.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.235+0000 7f818e967740 1 -- 172.21.15.43:0/516334286 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8174004a70 con 0x5581632d9380 2024-03-28T22:39:48.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.235+0000 7f8172a00640 1 -- 172.21.15.43:0/516334286 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8174042040 con 0x5581632d9380 2024-03-28T22:39:48.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.235+0000 7f818e967740 10 client.26603 fetch_fsmap learned FSMap version 30 2024-03-28T22:39:48.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.235+0000 7f818e967740 1 -- 172.21.15.43:0/516334286 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558163359530 con 0x5581632d9380 2024-03-28T22:39:48.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.235+0000 7f8172a00640 1 -- 172.21.15.43:0/516334286 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8174004a70 con 0x5581632d9380 2024-03-28T22:39:48.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.236+0000 7f8172a00640 1 -- 172.21.15.43:0/516334286 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f8174058440 con 0x5581632d9380 2024-03-28T22:39:48.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.236+0000 7f818e967740 10 client.26603 fetch_fsmap finished waiting for FSMap version 30 2024-03-28T22:39:48.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.236+0000 7f818e967740 10 client.26603 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:39:48.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.236+0000 7f818e967740 20 client.26603 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:48.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.236+0000 7f818e967740 1 --2- 172.21.15.43:0/516334286 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5581633598e0 0x55816335bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.236+0000 7f818e967740 4 client.26603 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:39:48.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.236+0000 7f818e967740 1 -- 172.21.15.43:0/516334286 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55816324a590 con 0x5581633598e0 2024-03-28T22:39:48.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.236+0000 7f817be00640 1 --2- 172.21.15.43:0/516334286 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5581633598e0 0x55816335bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.236+0000 7f817be00640 1 --2- 172.21.15.43:0/516334286 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5581633598e0 0x55816335bcd0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a9de20c3c4fc3e71 server_cookie=c2e2b5565856c540 in_seq=0 out_seq=0 2024-03-28T22:39:48.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.236+0000 7f8172a00640 10 client.26603 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:39:48.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.237+0000 7f8172a00640 1 -- 172.21.15.43:0/516334286 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55816324a590 con 0x5581633598e0 2024-03-28T22:39:48.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.237+0000 7f8172a00640 10 client.26603 handle_command_reply: tid=0 2024-03-28T22:39:48.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.237+0000 7f818e967740 1 client.26603 shutdown 2024-03-28T22:39:48.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.237+0000 7f818e967740 1 -- 172.21.15.43:0/516334286 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5581632d9380 msgr2=0x5581632dc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:48.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.237+0000 7f818e967740 1 --2- 172.21.15.43:0/516334286 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5581632d9380 0x5581632dc760 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f8174006010 tx=0x7f8174004590 comp rx=0 tx=0).stop 2024-03-28T22:39:48.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.237+0000 7f818e967740 1 -- 172.21.15.43:0/516334286 shutdown_connections 2024-03-28T22:39:48.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.237+0000 7f818e967740 1 --2- 172.21.15.43:0/516334286 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5581632e2880 0x5581632dd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.237+0000 7f818e967740 1 --2- 172.21.15.43:0/516334286 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5581632d9ca0 0x5581632dcca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.237+0000 7f818e967740 1 --2- 172.21.15.43:0/516334286 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5581633598e0 0x55816335bcd0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.237+0000 7f818e967740 1 --2- 172.21.15.43:0/516334286 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5581632d9380 0x5581632dc760 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.237+0000 7f818e967740 1 -- 172.21.15.43:0/516334286 >> 172.21.15.43:0/516334286 conn(0x5581632d3ab0 msgr2=0x5581632df710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:48.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.237+0000 7f8172a00640 0 client.26603 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:39:48.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.237+0000 7f818e967740 1 -- 172.21.15.43:0/516334286 shutdown_connections 2024-03-28T22:39:48.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.237+0000 7f818e967740 1 -- 172.21.15.43:0/516334286 wait complete. 2024-03-28T22:39:48.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.237+0000 7f818e967740 20 client.26603 trim_cache size 0 max 16384 2024-03-28T22:39:48.240 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:39:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.238+0000 7f818bc00640 1 -- 172.21.15.43:0/1875976326 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f81680a1ef0 msgr2=0x7f81680a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.238+0000 7f818bc00640 1 --2- 172.21.15.43:0/1875976326 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f81680a1ef0 0x7f81680a43b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f817c005fd0 tx=0x7f817c009500 comp rx=0 tx=0).stop 2024-03-28T22:39:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.238+0000 7f818bc00640 1 -- 172.21.15.43:0/1875976326 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f818410d030 msgr2=0x7f818419eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.238+0000 7f818bc00640 1 --2- 172.21.15.43:0/1875976326 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f818410d030 0x7f818419eb50 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f816c00e9c0 tx=0x7f816c00ee80 comp rx=0 tx=0).stop 2024-03-28T22:39:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.238+0000 7f818bc00640 1 -- 172.21.15.43:0/1875976326 shutdown_connections 2024-03-28T22:39:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.238+0000 7f818bc00640 1 --2- 172.21.15.43:0/1875976326 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f81680a1ef0 0x7f81680a43b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.238+0000 7f818bc00640 1 --2- 172.21.15.43:0/1875976326 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f818410d030 0x7f818419eb50 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.238+0000 7f818bc00640 1 --2- 172.21.15.43:0/1875976326 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81841a2ed0 0x7f81841a3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.238+0000 7f818bc00640 1 --2- 172.21.15.43:0/1875976326 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8184107f60 0x7f818419e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.238+0000 7f818bc00640 1 -- 172.21.15.43:0/1875976326 >> 172.21.15.43:0/1875976326 conn(0x7f8184075c90 msgr2=0x7f8184078010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.238+0000 7f818bc00640 1 -- 172.21.15.43:0/1875976326 shutdown_connections 2024-03-28T22:39:48.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.238+0000 7f818bc00640 1 -- 172.21.15.43:0/1875976326 wait complete. 2024-03-28T22:39:48.254 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:39:48.347 INFO:tasks.workunit.client.0.smithi043.stdout: 1 125852 8.80 MB/sec warmup 79 sec latency 14.439 ms 2024-03-28T22:39:48.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.471+0000 7fe9f7200640 1 -- 172.21.15.43:0/3626503621 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe9f00744a0 msgr2=0x7fe9f010d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:48.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.471+0000 7fe9f7200640 1 --2- 172.21.15.43:0/3626503621 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe9f00744a0 0x7fe9f010d9e0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc009900 tx=0x7fe9dc02bcc0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.471+0000 7fe9f7200640 1 -- 172.21.15.43:0/3626503621 shutdown_connections 2024-03-28T22:39:48.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.471+0000 7fe9f7200640 1 --2- 172.21.15.43:0/3626503621 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe9f00744a0 0x7fe9f010d9e0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.471+0000 7fe9f7200640 1 --2- 172.21.15.43:0/3626503621 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9f010df20 0x7fe9f0110370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.471+0000 7fe9f7200640 1 --2- 172.21.15.43:0/3626503621 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe9f0073b80 0x7fe9f0073f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.471+0000 7fe9f7200640 1 -- 172.21.15.43:0/3626503621 >> 172.21.15.43:0/3626503621 conn(0x7fe9f0078b50 msgr2=0x7fe9f0078f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:48.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.471+0000 7fe9f7200640 1 -- 172.21.15.43:0/3626503621 shutdown_connections 2024-03-28T22:39:48.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.471+0000 7fe9f7200640 1 -- 172.21.15.43:0/3626503621 wait complete. 2024-03-28T22:39:48.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.472+0000 7fe9f7200640 1 Processor -- start 2024-03-28T22:39:48.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.472+0000 7fe9f7200640 1 -- start start 2024-03-28T22:39:48.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.472+0000 7fe9f7200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9f0073b80 0x7fe9f019a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.472+0000 7fe9f7200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe9f00744a0 0x7fe9f019a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.472+0000 7fe9f7200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe9f010df20 0x7fe9f019ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.472+0000 7fe9f7200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f0113d10 con 0x7fe9f00744a0 2024-03-28T22:39:48.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.472+0000 7fe9f7200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f0113b90 con 0x7fe9f010df20 2024-03-28T22:39:48.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.472+0000 7fe9f7200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f0113e90 con 0x7fe9f0073b80 2024-03-28T22:39:48.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.473+0000 7fe9efe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9f0073b80 0x7fe9f019a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.473+0000 7fe9efe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9f0073b80 0x7fe9f019a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:56124/0 (socket says 172.21.15.43:56124) 2024-03-28T22:39:48.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.473+0000 7fe9efe00640 1 -- 172.21.15.43:0/2071759900 learned_addr learned my addr 172.21.15.43:0/2071759900 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:48.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.473+0000 7fe9f5200640 1 --2- 172.21.15.43:0/2071759900 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe9f010df20 0x7fe9f019ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.473+0000 7fe9ef400640 1 --2- 172.21.15.43:0/2071759900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe9f00744a0 0x7fe9f019a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.473+0000 7fe9efe00640 1 -- 172.21.15.43:0/2071759900 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe9f010df20 msgr2=0x7fe9f019ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:48.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.473+0000 7fe9efe00640 1 --2- 172.21.15.43:0/2071759900 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe9f010df20 0x7fe9f019ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.473+0000 7fe9efe00640 1 -- 172.21.15.43:0/2071759900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe9f00744a0 msgr2=0x7fe9f019a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:48.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.473+0000 7fe9efe00640 1 --2- 172.21.15.43:0/2071759900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe9f00744a0 0x7fe9f019a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.473+0000 7fe9efe00640 1 -- 172.21.15.43:0/2071759900 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9f019f1b0 con 0x7fe9f0073b80 2024-03-28T22:39:48.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.473+0000 7fe9ef400640 1 --2- 172.21.15.43:0/2071759900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe9f00744a0 0x7fe9f019a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:39:48.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.473+0000 7fe9efe00640 1 --2- 172.21.15.43:0/2071759900 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9f0073b80 0x7fe9f019a210 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe9e000e9f0 tx=0x7fe9e000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:48.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.474+0000 7fe9ecc00640 1 -- 172.21.15.43:0/2071759900 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e000cd80 con 0x7fe9f0073b80 2024-03-28T22:39:48.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.474+0000 7fe9f7200640 1 -- 172.21.15.43:0/2071759900 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9f019f4a0 con 0x7fe9f0073b80 2024-03-28T22:39:48.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.474+0000 7fe9ecc00640 1 -- 172.21.15.43:0/2071759900 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9e0010910 con 0x7fe9f0073b80 2024-03-28T22:39:48.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.474+0000 7fe9ecc00640 1 -- 172.21.15.43:0/2071759900 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e00188f0 con 0x7fe9f0073b80 2024-03-28T22:39:48.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.474+0000 7fe9f7200640 1 -- 172.21.15.43:0/2071759900 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9f01a6d30 con 0x7fe9f0073b80 2024-03-28T22:39:48.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.477+0000 7fe9f9e7b740 1 Processor -- start 2024-03-28T22:39:48.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.477+0000 7fe9ecc00640 1 -- 172.21.15.43:0/2071759900 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fe9e0010ac0 con 0x7fe9f0073b80 2024-03-28T22:39:48.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.478+0000 7fe9f9e7b740 1 -- start start 2024-03-28T22:39:48.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.478+0000 7fe9f9e7b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557b05054380 0x557b05054760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.478+0000 7fe9f9e7b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557b05054ca0 0x557b0505d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.478+0000 7fe9f9e7b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557b0505d880 0x557b0505fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.478+0000 7fe9f9e7b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x557b04fb8050 con 0x557b05054ca0 2024-03-28T22:39:48.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.478+0000 7fe9f9e7b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x557b04f60ce0 con 0x557b05054380 2024-03-28T22:39:48.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.478+0000 7fe9f9e7b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x557b04fb40f0 con 0x557b0505d880 2024-03-28T22:39:48.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.478+0000 7fe9f5200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557b05054380 0x557b05054760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.478+0000 7fe9ef400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557b0505d880 0x557b0505fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.478+0000 7fe9f5200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557b05054380 0x557b05054760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52772/0 (socket says 172.21.15.43:52772) 2024-03-28T22:39:48.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.478+0000 7fe9f5200640 1 -- 172.21.15.43:0/2167456423 learned_addr learned my addr 172.21.15.43:0/2167456423 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:48.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.478+0000 7fe9efe00640 1 --2- 172.21.15.43:0/2167456423 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557b05054ca0 0x557b0505d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.479+0000 7fe9f5200640 1 -- 172.21.15.43:0/2167456423 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557b0505d880 msgr2=0x557b0505fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:48.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.479+0000 7fe9f5200640 1 --2- 172.21.15.43:0/2167456423 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557b0505d880 0x557b0505fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.479+0000 7fe9f5200640 1 -- 172.21.15.43:0/2167456423 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557b05054ca0 msgr2=0x557b0505d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:48.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.479+0000 7fe9f5200640 1 --2- 172.21.15.43:0/2167456423 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557b05054ca0 0x557b0505d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.479+0000 7fe9f5200640 1 -- 172.21.15.43:0/2167456423 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b04ccc3f0 con 0x557b05054380 2024-03-28T22:39:48.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.479+0000 7fe9f5200640 1 --2- 172.21.15.43:0/2167456423 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557b05054380 0x557b05054760 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fe9e8009f70 tx=0x7fe9e8010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20426d63512ff94 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:48.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.479+0000 7fe9e5600640 1 -- 172.21.15.43:0/2167456423 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e8016070 con 0x557b05054380 2024-03-28T22:39:48.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.479+0000 7fe9e5600640 1 -- 172.21.15.43:0/2167456423 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9e8004d20 con 0x557b05054380 2024-03-28T22:39:48.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.479+0000 7fe9e5600640 1 -- 172.21.15.43:0/2167456423 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e8012910 con 0x557b05054380 2024-03-28T22:39:48.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.479+0000 7fe9ecc00640 1 --2- 172.21.15.43:0/2071759900 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe9d40a1ef0 0x7fe9d40a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.480+0000 7fe9ecc00640 1 -- 172.21.15.43:0/2071759900 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7fe9e00ce9d0 con 0x7fe9f0073b80 2024-03-28T22:39:48.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.480+0000 7fe9efe00640 1 --2- 172.21.15.43:0/2071759900 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe9d40a1ef0 0x7fe9d40a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.480+0000 7fe9f9e7b740 1 -- 172.21.15.43:0/2167456423 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557b05054380 msgr2=0x557b05054760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:48.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.480+0000 7fe9f9e7b740 1 --2- 172.21.15.43:0/2167456423 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557b05054380 0x557b05054760 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fe9e8009f70 tx=0x7fe9e8010800 comp rx=0 tx=0).stop 2024-03-28T22:39:48.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.480+0000 7fe9f9e7b740 1 -- 172.21.15.43:0/2167456423 shutdown_connections 2024-03-28T22:39:48.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.480+0000 7fe9f9e7b740 1 --2- 172.21.15.43:0/2167456423 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557b0505d880 0x557b0505fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.480+0000 7fe9f9e7b740 1 --2- 172.21.15.43:0/2167456423 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557b05054ca0 0x557b0505d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.480+0000 7fe9f9e7b740 1 --2- 172.21.15.43:0/2167456423 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557b05054380 0x557b05054760 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.480+0000 7fe9f9e7b740 1 -- 172.21.15.43:0/2167456423 >> 172.21.15.43:0/2167456423 conn(0x557b0504eab0 msgr2=0x557b0504ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:48.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.480+0000 7fe9f9e7b740 1 -- 172.21.15.43:0/2167456423 shutdown_connections 2024-03-28T22:39:48.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.480+0000 7fe9f9e7b740 1 -- 172.21.15.43:0/2167456423 wait complete. 2024-03-28T22:39:48.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.480+0000 7fe9efe00640 1 --2- 172.21.15.43:0/2071759900 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe9d40a1ef0 0x7fe9d40a43b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x557b05053e50 tx=0x7fe9e00d4260 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:48.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.481+0000 7fe9f9e7b740 1 -- start start 2024-03-28T22:39:48.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.481+0000 7fe9f9e7b740 1 Processor -- start 2024-03-28T22:39:48.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.482+0000 7fe9f9e7b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557b05054380 0x557b0505a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.482+0000 7fe9f9e7b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557b05054ca0 0x557b0505ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.482+0000 7fe9f9e7b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557b0505d880 0x557b0505b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.482+0000 7fe9f9e7b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x557b05060670 con 0x557b05054ca0 2024-03-28T22:39:48.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.482+0000 7fe9f9e7b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x557b04f562f0 con 0x557b0505d880 2024-03-28T22:39:48.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.482+0000 7fe9f9e7b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x557b050607f0 con 0x557b05054380 2024-03-28T22:39:48.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.482+0000 7fe9f5200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557b05054380 0x557b0505a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.482+0000 7fe9f5200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557b05054380 0x557b0505a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:56140/0 (socket says 172.21.15.43:56140) 2024-03-28T22:39:48.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.482+0000 7fe9f5200640 1 -- 172.21.15.43:0/1506381436 learned_addr learned my addr 172.21.15.43:0/1506381436 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:48.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.482+0000 7fe9f9e7b740 10 client.0 _ll_register_callbacks cb 0x557b04ccbbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:48.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.482+0000 7fe9f5200640 1 --2- 172.21.15.43:0/1506381436 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557b0505d880 0x557b0505b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.482+0000 7fe9ef400640 1 --2- 172.21.15.43:0/1506381436 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557b05054ca0 0x557b0505ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.483+0000 7fe9f5200640 1 -- 172.21.15.43:0/1506381436 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557b0505d880 msgr2=0x557b0505b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:48.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.483+0000 7fe9f5200640 1 --2- 172.21.15.43:0/1506381436 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557b0505d880 0x557b0505b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.483+0000 7fe9f5200640 1 -- 172.21.15.43:0/1506381436 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557b05054ca0 msgr2=0x557b0505ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:48.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.483+0000 7fe9f5200640 1 --2- 172.21.15.43:0/1506381436 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557b05054ca0 0x557b0505ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.483+0000 7fe9f5200640 1 -- 172.21.15.43:0/1506381436 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b04d967d0 con 0x557b05054380 2024-03-28T22:39:48.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.483+0000 7fe9f5200640 1 --2- 172.21.15.43:0/1506381436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557b05054380 0x557b0505a860 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe9e80125f0 tx=0x7fe9e80124f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3b5d5747ed1312 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:48.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.483+0000 7fe9da000640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:39:48.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.483+0000 7fe9da000640 1 -- 172.21.15.43:0/1506381436 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e8016040 con 0x557b05054380 2024-03-28T22:39:48.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.483+0000 7fe9f9e7b740 1 -- 172.21.15.43:0/1506381436 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9e8004d20 con 0x557b05054380 2024-03-28T22:39:48.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.483+0000 7fe9da000640 1 -- 172.21.15.43:0/1506381436 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7fe9e8012d40 con 0x557b05054380 2024-03-28T22:39:48.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.484+0000 7fe9da000640 1 -- 172.21.15.43:0/1506381436 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fe9e8003970 con 0x557b05054380 2024-03-28T22:39:48.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.484+0000 7fe9f9e7b740 10 client.26563 fetch_fsmap learned FSMap version 30 2024-03-28T22:39:48.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.484+0000 7fe9f9e7b740 1 -- 172.21.15.43:0/1506381436 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557b0506c8a0 con 0x557b05054380 2024-03-28T22:39:48.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.484+0000 7fe9da000640 1 -- 172.21.15.43:0/1506381436 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9e8004d20 con 0x557b05054380 2024-03-28T22:39:48.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.484+0000 7fe9da000640 1 -- 172.21.15.43:0/1506381436 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fe9e8028040 con 0x557b05054380 2024-03-28T22:39:48.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.484+0000 7fe9f9e7b740 10 client.26563 fetch_fsmap finished waiting for FSMap version 30 2024-03-28T22:39:48.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.484+0000 7fe9f9e7b740 10 client.26563 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:39:48.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.484+0000 7fe9f9e7b740 20 client.26563 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:48.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.484+0000 7fe9f9e7b740 1 --2- 172.21.15.43:0/1506381436 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x557b0506dc50 0x557b05070040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.484+0000 7fe9f9e7b740 4 client.26563 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:39:48.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.484+0000 7fe9f9e7b740 1 -- 172.21.15.43:0/1506381436 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b04f61680 con 0x557b0506dc50 2024-03-28T22:39:48.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.485+0000 7fe9ef400640 1 --2- 172.21.15.43:0/1506381436 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x557b0506dc50 0x557b05070040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.485+0000 7fe9ef400640 1 --2- 172.21.15.43:0/1506381436 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x557b0506dc50 0x557b05070040 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fe54935bcbde839c server_cookie=2ec22ea9e2032e67 in_seq=0 out_seq=0 2024-03-28T22:39:48.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.485+0000 7fe9da000640 10 client.26563 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:39:48.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.486+0000 7fe9da000640 1 -- 172.21.15.43:0/1506381436 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557b04f61680 con 0x557b0506dc50 2024-03-28T22:39:48.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.486+0000 7fe9da000640 10 client.26563 handle_command_reply: tid=0 2024-03-28T22:39:48.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.486+0000 7fe9f9e7b740 1 client.26563 shutdown 2024-03-28T22:39:48.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.486+0000 7fe9f9e7b740 1 -- 172.21.15.43:0/1506381436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557b05054380 msgr2=0x557b0505a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:48.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.486+0000 7fe9f9e7b740 1 --2- 172.21.15.43:0/1506381436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557b05054380 0x557b0505a860 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe9e80125f0 tx=0x7fe9e80124f0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.486+0000 7fe9f9e7b740 1 -- 172.21.15.43:0/1506381436 shutdown_connections 2024-03-28T22:39:48.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.486+0000 7fe9f9e7b740 1 --2- 172.21.15.43:0/1506381436 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x557b0506dc50 0x557b05070040 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.486+0000 7fe9f9e7b740 1 --2- 172.21.15.43:0/1506381436 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557b0505d880 0x557b0505b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.486+0000 7fe9f9e7b740 1 --2- 172.21.15.43:0/1506381436 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557b05054ca0 0x557b0505ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.486+0000 7fe9da000640 0 client.26563 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:39:48.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.486+0000 7fe9f9e7b740 1 --2- 172.21.15.43:0/1506381436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557b05054380 0x557b0505a860 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.487+0000 7fe9f9e7b740 1 -- 172.21.15.43:0/1506381436 >> 172.21.15.43:0/1506381436 conn(0x557b0504eab0 msgr2=0x557b05050800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:48.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.487+0000 7fe9f9e7b740 1 -- 172.21.15.43:0/1506381436 shutdown_connections 2024-03-28T22:39:48.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.487+0000 7fe9f9e7b740 1 -- 172.21.15.43:0/1506381436 wait complete. 2024-03-28T22:39:48.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.487+0000 7fe9f9e7b740 20 client.26563 trim_cache size 0 max 16384 2024-03-28T22:39:48.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.502+0000 7fe9f9e7b740 1 Processor -- start 2024-03-28T22:39:48.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.502+0000 7fe9f9e7b740 1 -- start start 2024-03-28T22:39:48.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.503+0000 7fe9f9e7b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557b05054380 0x557b0504d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.503+0000 7fe9f9e7b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557b05054ca0 0x557b05069410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.503+0000 7fe9f9e7b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557b0505d880 0x557b0505b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.503+0000 7fe9f9e7b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f01b0ee0 con 0x557b05054ca0 2024-03-28T22:39:48.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.503+0000 7fe9f9e7b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f01b2a20 con 0x557b05054380 2024-03-28T22:39:48.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.503+0000 7fe9f9e7b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f01b15b0 con 0x557b0505d880 2024-03-28T22:39:48.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.503+0000 7fe9f5200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557b05054380 0x557b0504d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.503+0000 7fe9f5200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557b05054380 0x557b0504d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52786/0 (socket says 172.21.15.43:52786) 2024-03-28T22:39:48.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.503+0000 7fe9f5200640 1 -- 172.21.15.43:0/1592023278 learned_addr learned my addr 172.21.15.43:0/1592023278 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:48.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.503+0000 7fe9f5200640 1 --2- 172.21.15.43:0/1592023278 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557b0505d880 0x557b0505b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.503+0000 7fe9ef400640 1 --2- 172.21.15.43:0/1592023278 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557b05054ca0 0x557b05069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.503+0000 7fe9f5200640 1 -- 172.21.15.43:0/1592023278 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557b0505d880 msgr2=0x557b0505b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:48.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.503+0000 7fe9f5200640 1 --2- 172.21.15.43:0/1592023278 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557b0505d880 0x557b0505b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.503+0000 7fe9f5200640 1 -- 172.21.15.43:0/1592023278 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557b05054ca0 msgr2=0x557b05069410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:48.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.503+0000 7fe9f5200640 1 --2- 172.21.15.43:0/1592023278 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557b05054ca0 0x557b05069410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.503+0000 7fe9f5200640 1 -- 172.21.15.43:0/1592023278 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b0504e190 con 0x557b05054380 2024-03-28T22:39:48.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.504+0000 7fe9f5200640 1 --2- 172.21.15.43:0/1592023278 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557b05054380 0x557b0504d2e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fe9e8009870 tx=0x7fe9e8009600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=578d690f2ba4ad63 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:48.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.504+0000 7fe9e4c00640 1 -- 172.21.15.43:0/1592023278 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e8016040 con 0x557b05054380 2024-03-28T22:39:48.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.504+0000 7fe9e4c00640 1 -- 172.21.15.43:0/1592023278 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9e8004920 con 0x557b05054380 2024-03-28T22:39:48.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.504+0000 7fe9e4c00640 1 -- 172.21.15.43:0/1592023278 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e80050b0 con 0x557b05054380 2024-03-28T22:39:48.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.505+0000 7fe9f9e7b740 1 -- 172.21.15.43:0/1592023278 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557b05054380 msgr2=0x557b0504d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:48.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.505+0000 7fe9f9e7b740 1 --2- 172.21.15.43:0/1592023278 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557b05054380 0x557b0504d2e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fe9e8009870 tx=0x7fe9e8009600 comp rx=0 tx=0).stop 2024-03-28T22:39:48.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.505+0000 7fe9f9e7b740 1 -- 172.21.15.43:0/1592023278 shutdown_connections 2024-03-28T22:39:48.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.505+0000 7fe9f9e7b740 1 --2- 172.21.15.43:0/1592023278 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557b0505d880 0x557b0505b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.505+0000 7fe9f9e7b740 1 --2- 172.21.15.43:0/1592023278 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557b05054ca0 0x557b05069410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.505+0000 7fe9f9e7b740 1 --2- 172.21.15.43:0/1592023278 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557b05054380 0x557b0504d2e0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.505+0000 7fe9f9e7b740 1 -- 172.21.15.43:0/1592023278 >> 172.21.15.43:0/1592023278 conn(0x557b0504eab0 msgr2=0x557b05050640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:48.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.505+0000 7fe9f9e7b740 1 -- 172.21.15.43:0/1592023278 shutdown_connections 2024-03-28T22:39:48.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.505+0000 7fe9f9e7b740 1 -- 172.21.15.43:0/1592023278 wait complete. 2024-03-28T22:39:48.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.505+0000 7fe9f9e7b740 1 -- start start 2024-03-28T22:39:48.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.506+0000 7fe9f9e7b740 1 Processor -- start 2024-03-28T22:39:48.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.506+0000 7fe9f9e7b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557b05054380 0x557b05065b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.506+0000 7fe9f9e7b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557b05054ca0 0x557b05061000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.506+0000 7fe9f9e7b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557b0505d880 0x557b05061540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.506+0000 7fe9f9e7b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f01b2350 con 0x557b05054ca0 2024-03-28T22:39:48.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.506+0000 7fe9f9e7b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f01b1c80 con 0x557b05054380 2024-03-28T22:39:48.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.506+0000 7fe9f9e7b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f01aeec0 con 0x557b0505d880 2024-03-28T22:39:48.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.506+0000 7fe9f9e7b740 10 client.0 _ll_register_callbacks cb 0x7fe9e0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:48.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.506+0000 7fe9f5200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557b05054380 0x557b05065b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.506+0000 7fe9f5200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557b05054380 0x557b05065b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52800/0 (socket says 172.21.15.43:52800) 2024-03-28T22:39:48.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.506+0000 7fe9f5200640 1 -- 172.21.15.43:0/2870559536 learned_addr learned my addr 172.21.15.43:0/2870559536 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:48.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.506+0000 7fe9f5200640 1 --2- 172.21.15.43:0/2870559536 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557b0505d880 0x557b05061540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.506+0000 7fe9ef400640 1 --2- 172.21.15.43:0/2870559536 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557b05054ca0 0x557b05061000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.507+0000 7fe9f5200640 1 -- 172.21.15.43:0/2870559536 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557b0505d880 msgr2=0x557b05061540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:48.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.507+0000 7fe9f5200640 1 --2- 172.21.15.43:0/2870559536 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557b0505d880 0x557b05061540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.507+0000 7fe9f5200640 1 -- 172.21.15.43:0/2870559536 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557b05054ca0 msgr2=0x557b05061000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:48.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.507+0000 7fe9f5200640 1 --2- 172.21.15.43:0/2870559536 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557b05054ca0 0x557b05061000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.507+0000 7fe9f5200640 1 -- 172.21.15.43:0/2870559536 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b05054ad0 con 0x557b05054380 2024-03-28T22:39:48.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.507+0000 7fe9f5200640 1 --2- 172.21.15.43:0/2870559536 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557b05054380 0x557b05065b10 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fe9e8004f90 tx=0x7fe9e8005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74ed5d47435a4d5a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:48.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.507+0000 7fe9da000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:39:48.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.507+0000 7fe9da000640 1 -- 172.21.15.43:0/2870559536 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e8016070 con 0x557b05054380 2024-03-28T22:39:48.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.507+0000 7fe9da000640 1 -- 172.21.15.43:0/2870559536 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7fe9e8041070 con 0x557b05054380 2024-03-28T22:39:48.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.507+0000 7fe9f9e7b740 1 -- 172.21.15.43:0/2870559536 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9e8004920 con 0x557b05054380 2024-03-28T22:39:48.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.507+0000 7fe9da000640 1 -- 172.21.15.43:0/2870559536 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fe9e80059e0 con 0x557b05054380 2024-03-28T22:39:48.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.508+0000 7fe9f9e7b740 10 client.26621 fetch_fsmap learned FSMap version 30 2024-03-28T22:39:48.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.508+0000 7fe9f9e7b740 1 -- 172.21.15.43:0/2870559536 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557b050d39c0 con 0x557b05054380 2024-03-28T22:39:48.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.508+0000 7fe9da000640 1 -- 172.21.15.43:0/2870559536 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9e8004920 con 0x557b05054380 2024-03-28T22:39:48.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.508+0000 7fe9da000640 1 -- 172.21.15.43:0/2870559536 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fe9e800f040 con 0x557b05054380 2024-03-28T22:39:48.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.508+0000 7fe9f9e7b740 10 client.26621 fetch_fsmap finished waiting for FSMap version 30 2024-03-28T22:39:48.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.508+0000 7fe9f9e7b740 10 client.26621 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:39:48.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.508+0000 7fe9f9e7b740 20 client.26621 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:48.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.508+0000 7fe9f9e7b740 1 --2- 172.21.15.43:0/2870559536 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x557b0506dc50 0x557b050d5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.508+0000 7fe9f9e7b740 4 client.26621 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:39:48.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.508+0000 7fe9f9e7b740 1 -- 172.21.15.43:0/2870559536 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557b04fc5590 con 0x557b0506dc50 2024-03-28T22:39:48.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.508+0000 7fe9ef400640 1 --2- 172.21.15.43:0/2870559536 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x557b0506dc50 0x557b050d5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.509+0000 7fe9ef400640 1 --2- 172.21.15.43:0/2870559536 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x557b0506dc50 0x557b050d5d90 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fc6c603aec95d71d server_cookie=419c41fdecf115ac in_seq=0 out_seq=0 2024-03-28T22:39:48.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.509+0000 7fe9da000640 10 client.26621 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:39:48.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.509+0000 7fe9da000640 1 -- 172.21.15.43:0/2870559536 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557b04fc5590 con 0x557b0506dc50 2024-03-28T22:39:48.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.509+0000 7fe9da000640 10 client.26621 handle_command_reply: tid=0 2024-03-28T22:39:48.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.509+0000 7fe9f9e7b740 1 client.26621 shutdown 2024-03-28T22:39:48.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.509+0000 7fe9f9e7b740 1 -- 172.21.15.43:0/2870559536 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557b05054380 msgr2=0x557b05065b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:48.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.509+0000 7fe9f9e7b740 1 --2- 172.21.15.43:0/2870559536 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557b05054380 0x557b05065b10 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fe9e8004f90 tx=0x7fe9e8005fd0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.510+0000 7fe9f9e7b740 1 -- 172.21.15.43:0/2870559536 shutdown_connections 2024-03-28T22:39:48.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.510+0000 7fe9f9e7b740 1 --2- 172.21.15.43:0/2870559536 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x557b0506dc50 0x557b050d5d90 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.510+0000 7fe9f9e7b740 1 --2- 172.21.15.43:0/2870559536 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557b0505d880 0x557b05061540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.510+0000 7fe9f9e7b740 1 --2- 172.21.15.43:0/2870559536 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557b05054ca0 0x557b05061000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.510+0000 7fe9f9e7b740 1 --2- 172.21.15.43:0/2870559536 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557b05054380 0x557b05065b10 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.510+0000 7fe9f9e7b740 1 -- 172.21.15.43:0/2870559536 >> 172.21.15.43:0/2870559536 conn(0x557b0504eab0 msgr2=0x557b0505adc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:48.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.510+0000 7fe9da000640 0 client.26621 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:39:48.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.510+0000 7fe9f9e7b740 1 -- 172.21.15.43:0/2870559536 shutdown_connections 2024-03-28T22:39:48.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.510+0000 7fe9f9e7b740 1 -- 172.21.15.43:0/2870559536 wait complete. 2024-03-28T22:39:48.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.510+0000 7fe9f9e7b740 20 client.26621 trim_cache size 0 max 16384 2024-03-28T22:39:48.513 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:39:48.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.511+0000 7fe9f7200640 1 -- 172.21.15.43:0/2071759900 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe9d40a1ef0 msgr2=0x7fe9d40a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:48.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.511+0000 7fe9f7200640 1 --2- 172.21.15.43:0/2071759900 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe9d40a1ef0 0x7fe9d40a43b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x557b05053e50 tx=0x7fe9e00d4260 comp rx=0 tx=0).stop 2024-03-28T22:39:48.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.511+0000 7fe9f7200640 1 -- 172.21.15.43:0/2071759900 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9f0073b80 msgr2=0x7fe9f019a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:48.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.511+0000 7fe9f7200640 1 --2- 172.21.15.43:0/2071759900 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9f0073b80 0x7fe9f019a210 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe9e000e9f0 tx=0x7fe9e000eeb0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.511+0000 7fe9f7200640 1 -- 172.21.15.43:0/2071759900 shutdown_connections 2024-03-28T22:39:48.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.511+0000 7fe9f7200640 1 --2- 172.21.15.43:0/2071759900 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe9d40a1ef0 0x7fe9d40a43b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.511+0000 7fe9f7200640 1 --2- 172.21.15.43:0/2071759900 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe9f010df20 0x7fe9f019ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.511+0000 7fe9f7200640 1 --2- 172.21.15.43:0/2071759900 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe9f00744a0 0x7fe9f019a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.511+0000 7fe9f7200640 1 --2- 172.21.15.43:0/2071759900 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe9f0073b80 0x7fe9f019a210 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.511+0000 7fe9f7200640 1 -- 172.21.15.43:0/2071759900 >> 172.21.15.43:0/2071759900 conn(0x7fe9f0078b50 msgr2=0x7fe9f00fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:48.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.511+0000 7fe9f7200640 1 -- 172.21.15.43:0/2071759900 shutdown_connections 2024-03-28T22:39:48.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.511+0000 7fe9f7200640 1 -- 172.21.15.43:0/2071759900 wait complete. 2024-03-28T22:39:48.531 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:39:48.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.753+0000 7ffa4ee00640 1 -- 172.21.15.43:0/3417629024 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa481032e0 msgr2=0x7ffa4810b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:48.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.753+0000 7ffa4ee00640 1 --2- 172.21.15.43:0/3417629024 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa481032e0 0x7ffa4810b7c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ffa34009900 tx=0x7ffa3402bcd0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.753+0000 7ffa4ee00640 1 -- 172.21.15.43:0/3417629024 shutdown_connections 2024-03-28T22:39:48.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.753+0000 7ffa4ee00640 1 --2- 172.21.15.43:0/3417629024 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffa4810bd90 0x7ffa4810e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.753+0000 7ffa4ee00640 1 --2- 172.21.15.43:0/3417629024 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa481032e0 0x7ffa4810b7c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.753+0000 7ffa4ee00640 1 --2- 172.21.15.43:0/3417629024 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa481029c0 0x7ffa48102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.753+0000 7ffa4ee00640 1 -- 172.21.15.43:0/3417629024 >> 172.21.15.43:0/3417629024 conn(0x7ffa480777b0 msgr2=0x7ffa48077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:48.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.753+0000 7ffa4ee00640 1 -- 172.21.15.43:0/3417629024 shutdown_connections 2024-03-28T22:39:48.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.753+0000 7ffa4ee00640 1 -- 172.21.15.43:0/3417629024 wait complete. 2024-03-28T22:39:48.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.755+0000 7ffa4ee00640 1 Processor -- start 2024-03-28T22:39:48.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.755+0000 7ffa4ee00640 1 -- start start 2024-03-28T22:39:48.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.755+0000 7ffa4ee00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa481029c0 0x7ffa48100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.755+0000 7ffa4ee00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa481032e0 0x7ffa48100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.755+0000 7ffa4ee00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffa4810bd90 0x7ffa48101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.755+0000 7ffa4ee00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ffa48111bb0 con 0x7ffa481032e0 2024-03-28T22:39:48.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.755+0000 7ffa4ee00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ffa48111a30 con 0x7ffa4810bd90 2024-03-28T22:39:48.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.755+0000 7ffa4ee00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7ffa48111d30 con 0x7ffa481029c0 2024-03-28T22:39:48.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.756+0000 7ffa47e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa481029c0 0x7ffa48100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.756+0000 7ffa4ce00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffa4810bd90 0x7ffa48101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.756+0000 7ffa47e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa481029c0 0x7ffa48100610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:56182/0 (socket says 172.21.15.43:56182) 2024-03-28T22:39:48.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.756+0000 7ffa47e00640 1 -- 172.21.15.43:0/1049125464 learned_addr learned my addr 172.21.15.43:0/1049125464 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:48.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.756+0000 7ffa47400640 1 --2- 172.21.15.43:0/1049125464 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa481032e0 0x7ffa48100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.756+0000 7ffa4ce00640 1 -- 172.21.15.43:0/1049125464 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa481029c0 msgr2=0x7ffa48100610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:48.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.756+0000 7ffa4ce00640 1 --2- 172.21.15.43:0/1049125464 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa481029c0 0x7ffa48100610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.756+0000 7ffa4ce00640 1 -- 172.21.15.43:0/1049125464 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa481032e0 msgr2=0x7ffa48100b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:48.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.756+0000 7ffa4ce00640 1 --2- 172.21.15.43:0/1049125464 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa481032e0 0x7ffa48100b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.756+0000 7ffa4ce00640 1 -- 172.21.15.43:0/1049125464 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa481a6590 con 0x7ffa4810bd90 2024-03-28T22:39:48.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.756+0000 7ffa47e00640 1 --2- 172.21.15.43:0/1049125464 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa481029c0 0x7ffa48100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:48.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.756+0000 7ffa4ce00640 1 --2- 172.21.15.43:0/1049125464 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffa4810bd90 0x7ffa48101090 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ffa4000e970 tx=0x7ffa4000ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:48.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.757+0000 7ffa44c00640 1 -- 172.21.15.43:0/1049125464 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4000ccc0 con 0x7ffa4810bd90 2024-03-28T22:39:48.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.757+0000 7ffa44c00640 1 -- 172.21.15.43:0/1049125464 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa400035f0 con 0x7ffa4810bd90 2024-03-28T22:39:48.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.757+0000 7ffa4ee00640 1 -- 172.21.15.43:0/1049125464 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa481a6880 con 0x7ffa4810bd90 2024-03-28T22:39:48.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.757+0000 7ffa44c00640 1 -- 172.21.15.43:0/1049125464 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4000ce60 con 0x7ffa4810bd90 2024-03-28T22:39:48.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.757+0000 7ffa4ee00640 1 -- 172.21.15.43:0/1049125464 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa481a6da0 con 0x7ffa4810bd90 2024-03-28T22:39:48.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.760+0000 7ffa44c00640 1 -- 172.21.15.43:0/1049125464 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7ffa400037a0 con 0x7ffa4810bd90 2024-03-28T22:39:48.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.761+0000 7ffa51c37740 1 Processor -- start 2024-03-28T22:39:48.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.761+0000 7ffa51c37740 1 -- start start 2024-03-28T22:39:48.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.761+0000 7ffa51c37740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5605c25bc380 0x5605c25bc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.761+0000 7ffa51c37740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605c25bcca0 0x5605c25c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.761+0000 7ffa51c37740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5605c25c5880 0x5605c25c7c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.761+0000 7ffa51c37740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5605c2520050 con 0x5605c25bc380 2024-03-28T22:39:48.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.761+0000 7ffa51c37740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5605c24c8ce0 con 0x5605c25bcca0 2024-03-28T22:39:48.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.761+0000 7ffa51c37740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5605c251c0f0 con 0x5605c25c5880 2024-03-28T22:39:48.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.761+0000 7ffa47e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605c25bcca0 0x5605c25c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.762+0000 7ffa47400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5605c25bc380 0x5605c25bc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.762+0000 7ffa47e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605c25bcca0 0x5605c25c5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52828/0 (socket says 172.21.15.43:52828) 2024-03-28T22:39:48.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.762+0000 7ffa47e00640 1 -- 172.21.15.43:0/1548180111 learned_addr learned my addr 172.21.15.43:0/1548180111 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:48.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.762+0000 7ffa47400640 1 --2- 172.21.15.43:0/1548180111 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5605c25c5880 0x5605c25c7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.762+0000 7ffa47e00640 1 -- 172.21.15.43:0/1548180111 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5605c25c5880 msgr2=0x5605c25c7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:48.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.762+0000 7ffa47e00640 1 --2- 172.21.15.43:0/1548180111 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5605c25c5880 0x5605c25c7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.762+0000 7ffa47e00640 1 -- 172.21.15.43:0/1548180111 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5605c25bc380 msgr2=0x5605c25bc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:48.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.762+0000 7ffa47e00640 1 --2- 172.21.15.43:0/1548180111 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5605c25bc380 0x5605c25bc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.762+0000 7ffa47e00640 1 -- 172.21.15.43:0/1548180111 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605c22343f0 con 0x5605c25bcca0 2024-03-28T22:39:48.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.762+0000 7ffa47400640 1 --2- 172.21.15.43:0/1548180111 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5605c25c5880 0x5605c25c7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:39:48.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.762+0000 7ffa44c00640 1 --2- 172.21.15.43:0/1049125464 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ffa2c0a1f70 0x7ffa2c0a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.762+0000 7ffa47e00640 1 --2- 172.21.15.43:0/1548180111 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605c25bcca0 0x5605c25c5340 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ffa3800f8d0 tx=0x7ffa3800fd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ffb7d356fc7019c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:48.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.763+0000 7ffa44c00640 1 -- 172.21.15.43:0/1049125464 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7ffa400ce6e0 con 0x7ffa4810bd90 2024-03-28T22:39:48.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.763+0000 7ffa47400640 1 --2- 172.21.15.43:0/1049125464 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ffa2c0a1f70 0x7ffa2c0a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.763+0000 7ffa3d600640 1 -- 172.21.15.43:0/1548180111 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa38011c20 con 0x5605c25bcca0 2024-03-28T22:39:48.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.763+0000 7ffa3d600640 1 -- 172.21.15.43:0/1548180111 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa38011dc0 con 0x5605c25bcca0 2024-03-28T22:39:48.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.763+0000 7ffa3d600640 1 -- 172.21.15.43:0/1548180111 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa380187d0 con 0x5605c25bcca0 2024-03-28T22:39:48.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.764+0000 7ffa51c37740 1 -- 172.21.15.43:0/1548180111 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605c25bcca0 msgr2=0x5605c25c5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:48.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.764+0000 7ffa47400640 1 --2- 172.21.15.43:0/1049125464 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ffa2c0a1f70 0x7ffa2c0a4430 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ffa34005ab0 tx=0x7ffa34005b50 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:48.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.764+0000 7ffa51c37740 1 --2- 172.21.15.43:0/1548180111 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605c25bcca0 0x5605c25c5340 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ffa3800f8d0 tx=0x7ffa3800fd90 comp rx=0 tx=0).stop 2024-03-28T22:39:48.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.764+0000 7ffa51c37740 1 -- 172.21.15.43:0/1548180111 shutdown_connections 2024-03-28T22:39:48.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.764+0000 7ffa51c37740 1 --2- 172.21.15.43:0/1548180111 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605c25bcca0 0x5605c25c5340 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.764+0000 7ffa51c37740 1 --2- 172.21.15.43:0/1548180111 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5605c25c5880 0x5605c25c7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.764+0000 7ffa51c37740 1 --2- 172.21.15.43:0/1548180111 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5605c25bc380 0x5605c25bc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.764+0000 7ffa51c37740 1 -- 172.21.15.43:0/1548180111 >> 172.21.15.43:0/1548180111 conn(0x5605c25b6ab0 msgr2=0x5605c25b6e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:48.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.764+0000 7ffa51c37740 1 -- 172.21.15.43:0/1548180111 shutdown_connections 2024-03-28T22:39:48.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.764+0000 7ffa51c37740 1 -- 172.21.15.43:0/1548180111 wait complete. 2024-03-28T22:39:48.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.764+0000 7ffa51c37740 1 -- start start 2024-03-28T22:39:48.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.764+0000 7ffa51c37740 1 Processor -- start 2024-03-28T22:39:48.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.765+0000 7ffa51c37740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605c25bc380 0x5605c25c28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.765+0000 7ffa51c37740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5605c25bcca0 0x5605c25c2e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.765+0000 7ffa51c37740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5605c25c5880 0x5605c25c3360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.765+0000 7ffa51c37740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5605c25c8670 con 0x5605c25c5880 2024-03-28T22:39:48.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.765+0000 7ffa51c37740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5605c24be2f0 con 0x5605c25bc380 2024-03-28T22:39:48.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.765+0000 7ffa51c37740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5605c25c87f0 con 0x5605c25bcca0 2024-03-28T22:39:48.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.765+0000 7ffa51c37740 10 client.0 _ll_register_callbacks cb 0x5605c2233be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:48.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.765+0000 7ffa47e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605c25bc380 0x5605c25c28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.765+0000 7ffa47400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5605c25bcca0 0x5605c25c2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.765+0000 7ffa47e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605c25bc380 0x5605c25c28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52832/0 (socket says 172.21.15.43:52832) 2024-03-28T22:39:48.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.765+0000 7ffa47e00640 1 -- 172.21.15.43:0/2741131368 learned_addr learned my addr 172.21.15.43:0/2741131368 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:48.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.765+0000 7ffa4ce00640 1 --2- 172.21.15.43:0/2741131368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5605c25c5880 0x5605c25c3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.766+0000 7ffa47e00640 1 -- 172.21.15.43:0/2741131368 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5605c25bcca0 msgr2=0x5605c25c2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:48.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.766+0000 7ffa47e00640 1 --2- 172.21.15.43:0/2741131368 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5605c25bcca0 0x5605c25c2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.766+0000 7ffa47e00640 1 -- 172.21.15.43:0/2741131368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5605c25c5880 msgr2=0x5605c25c3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:48.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.766+0000 7ffa47e00640 1 --2- 172.21.15.43:0/2741131368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5605c25c5880 0x5605c25c3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.766+0000 7ffa47e00640 1 -- 172.21.15.43:0/2741131368 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605c22fe7d0 con 0x5605c25bc380 2024-03-28T22:39:48.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.766+0000 7ffa4ce00640 1 --2- 172.21.15.43:0/2741131368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5605c25c5880 0x5605c25c3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:39:48.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.766+0000 7ffa47e00640 1 --2- 172.21.15.43:0/2741131368 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605c25bc380 0x5605c25c28e0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ffa38000f80 tx=0x7ffa38003c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29857c1b4f185eb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:48.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.766+0000 7ffa32000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:39:48.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.766+0000 7ffa32000640 1 -- 172.21.15.43:0/2741131368 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa38004450 con 0x5605c25bc380 2024-03-28T22:39:48.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.767+0000 7ffa32000640 1 -- 172.21.15.43:0/2741131368 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7ffa380045f0 con 0x5605c25bc380 2024-03-28T22:39:48.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.767+0000 7ffa51c37740 1 -- 172.21.15.43:0/2741131368 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa38011dc0 con 0x5605c25bc380 2024-03-28T22:39:48.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.767+0000 7ffa32000640 1 -- 172.21.15.43:0/2741131368 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ffa38033050 con 0x5605c25bc380 2024-03-28T22:39:48.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.767+0000 7ffa51c37740 10 client.26633 fetch_fsmap learned FSMap version 30 2024-03-28T22:39:48.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.767+0000 7ffa51c37740 1 -- 172.21.15.43:0/2741131368 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5605c25d4940 con 0x5605c25bc380 2024-03-28T22:39:48.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.767+0000 7ffa32000640 1 -- 172.21.15.43:0/2741131368 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa38011dc0 con 0x5605c25bc380 2024-03-28T22:39:48.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.767+0000 7ffa32000640 1 -- 172.21.15.43:0/2741131368 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7ffa38019420 con 0x5605c25bc380 2024-03-28T22:39:48.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.767+0000 7ffa51c37740 10 client.26633 fetch_fsmap finished waiting for FSMap version 30 2024-03-28T22:39:48.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.767+0000 7ffa51c37740 10 client.26633 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:39:48.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.767+0000 7ffa51c37740 20 client.26633 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:48.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.767+0000 7ffa51c37740 1 --2- 172.21.15.43:0/2741131368 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5605c25d5e00 0x5605c25d81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.767+0000 7ffa51c37740 4 client.26633 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:39:48.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.768+0000 7ffa51c37740 1 -- 172.21.15.43:0/2741131368 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5605c24c9680 con 0x5605c25d5e00 2024-03-28T22:39:48.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.768+0000 7ffa47400640 1 --2- 172.21.15.43:0/2741131368 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5605c25d5e00 0x5605c25d81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.768+0000 7ffa47400640 1 --2- 172.21.15.43:0/2741131368 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5605c25d5e00 0x5605c25d81f0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ed990fb00151d52 server_cookie=4b1ca731dfd83e19 in_seq=0 out_seq=0 2024-03-28T22:39:48.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.768+0000 7ffa32000640 10 client.26633 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:48.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.769+0000 7ffa32000640 1 -- 172.21.15.43:0/2741131368 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5605c24c9680 con 0x5605c25d5e00 2024-03-28T22:39:48.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.769+0000 7ffa32000640 10 client.26633 handle_command_reply: tid=0 2024-03-28T22:39:48.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.770+0000 7ffa51c37740 1 client.26633 shutdown 2024-03-28T22:39:48.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.770+0000 7ffa51c37740 1 -- 172.21.15.43:0/2741131368 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605c25bc380 msgr2=0x5605c25c28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:48.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.770+0000 7ffa51c37740 1 --2- 172.21.15.43:0/2741131368 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605c25bc380 0x5605c25c28e0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ffa38000f80 tx=0x7ffa38003c20 comp rx=0 tx=0).stop 2024-03-28T22:39:48.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.770+0000 7ffa51c37740 1 -- 172.21.15.43:0/2741131368 shutdown_connections 2024-03-28T22:39:48.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.770+0000 7ffa51c37740 1 --2- 172.21.15.43:0/2741131368 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5605c25c5880 0x5605c25c3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.770+0000 7ffa51c37740 1 --2- 172.21.15.43:0/2741131368 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5605c25bcca0 0x5605c25c2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.770+0000 7ffa51c37740 1 --2- 172.21.15.43:0/2741131368 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5605c25d5e00 0x5605c25d81f0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.770+0000 7ffa51c37740 1 --2- 172.21.15.43:0/2741131368 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605c25bc380 0x5605c25c28e0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.770+0000 7ffa32000640 0 client.26633 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:48.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.770+0000 7ffa51c37740 1 -- 172.21.15.43:0/2741131368 >> 172.21.15.43:0/2741131368 conn(0x5605c25b6ab0 msgr2=0x5605c25b8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:48.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.770+0000 7ffa51c37740 1 -- 172.21.15.43:0/2741131368 shutdown_connections 2024-03-28T22:39:48.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.770+0000 7ffa51c37740 1 -- 172.21.15.43:0/2741131368 wait complete. 2024-03-28T22:39:48.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.770+0000 7ffa51c37740 20 client.26633 trim_cache size 0 max 16384 2024-03-28T22:39:48.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.787+0000 7ffa51c37740 1 Processor -- start 2024-03-28T22:39:48.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.787+0000 7ffa51c37740 1 -- start start 2024-03-28T22:39:48.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.787+0000 7ffa51c37740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5605c25bcca0 0x5605c25c10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.787+0000 7ffa51c37740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605c25c5880 0x5605c25d1410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.787+0000 7ffa51c37740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5605c25c31d0 0x5605c25dd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.787+0000 7ffa51c37740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ffa481b0e20 con 0x5605c25bcca0 2024-03-28T22:39:48.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.787+0000 7ffa51c37740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ffa481b2960 con 0x5605c25c5880 2024-03-28T22:39:48.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.787+0000 7ffa51c37740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7ffa481b14f0 con 0x5605c25c31d0 2024-03-28T22:39:48.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.787+0000 7ffa47e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5605c25bcca0 0x5605c25c10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.787+0000 7ffa4ce00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5605c25c31d0 0x5605c25dd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.787+0000 7ffa47400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605c25c5880 0x5605c25d1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.787+0000 7ffa47400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605c25c5880 0x5605c25d1410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52842/0 (socket says 172.21.15.43:52842) 2024-03-28T22:39:48.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.787+0000 7ffa47400640 1 -- 172.21.15.43:0/2679341746 learned_addr learned my addr 172.21.15.43:0/2679341746 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:48.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.788+0000 7ffa47400640 1 -- 172.21.15.43:0/2679341746 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5605c25c31d0 msgr2=0x5605c25dd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:48.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.788+0000 7ffa47400640 1 --2- 172.21.15.43:0/2679341746 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5605c25c31d0 0x5605c25dd9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.788+0000 7ffa47400640 1 -- 172.21.15.43:0/2679341746 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5605c25bcca0 msgr2=0x5605c25c10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:48.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.788+0000 7ffa47400640 1 --2- 172.21.15.43:0/2679341746 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5605c25bcca0 0x5605c25c10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.788+0000 7ffa47400640 1 -- 172.21.15.43:0/2679341746 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605c25c1f90 con 0x5605c25c5880 2024-03-28T22:39:48.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.788+0000 7ffa47400640 1 --2- 172.21.15.43:0/2679341746 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605c25c5880 0x5605c25d1410 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ffa34009230 tx=0x7ffa34009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=231ad235df52cb45 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:48.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.788+0000 7ffa3cc00640 1 -- 172.21.15.43:0/2679341746 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3405c070 con 0x5605c25c5880 2024-03-28T22:39:48.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.788+0000 7ffa3cc00640 1 -- 172.21.15.43:0/2679341746 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa34047030 con 0x5605c25c5880 2024-03-28T22:39:48.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.788+0000 7ffa3cc00640 1 -- 172.21.15.43:0/2679341746 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa340047b0 con 0x5605c25c5880 2024-03-28T22:39:48.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.789+0000 7ffa51c37740 1 -- 172.21.15.43:0/2679341746 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605c25c5880 msgr2=0x5605c25d1410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:48.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.789+0000 7ffa51c37740 1 --2- 172.21.15.43:0/2679341746 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605c25c5880 0x5605c25d1410 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ffa34009230 tx=0x7ffa34009260 comp rx=0 tx=0).stop 2024-03-28T22:39:48.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.789+0000 7ffa51c37740 1 -- 172.21.15.43:0/2679341746 shutdown_connections 2024-03-28T22:39:48.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.789+0000 7ffa51c37740 1 --2- 172.21.15.43:0/2679341746 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605c25c5880 0x5605c25d1410 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.789+0000 7ffa51c37740 1 --2- 172.21.15.43:0/2679341746 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5605c25c31d0 0x5605c25dd9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.789+0000 7ffa51c37740 1 --2- 172.21.15.43:0/2679341746 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5605c25bcca0 0x5605c25c10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.789+0000 7ffa51c37740 1 -- 172.21.15.43:0/2679341746 >> 172.21.15.43:0/2679341746 conn(0x5605c25b6ab0 msgr2=0x5605c25b8700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:48.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.789+0000 7ffa51c37740 1 -- 172.21.15.43:0/2679341746 shutdown_connections 2024-03-28T22:39:48.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.789+0000 7ffa51c37740 1 -- 172.21.15.43:0/2679341746 wait complete. 2024-03-28T22:39:48.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.790+0000 7ffa51c37740 1 -- start start 2024-03-28T22:39:48.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.790+0000 7ffa51c37740 1 Processor -- start 2024-03-28T22:39:48.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.790+0000 7ffa51c37740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5605c25bcca0 0x5605c25bf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.790+0000 7ffa51c37740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605c25c31d0 0x5605c25bfc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.790+0000 7ffa51c37740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5605c25c5880 0x5605c25c01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.790+0000 7ffa51c37740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ffa481b2290 con 0x5605c25bcca0 2024-03-28T22:39:48.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.791+0000 7ffa51c37740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ffa481b1bc0 con 0x5605c25c31d0 2024-03-28T22:39:48.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.791+0000 7ffa51c37740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7ffa481aee00 con 0x5605c25c5880 2024-03-28T22:39:48.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.791+0000 7ffa51c37740 10 client.0 _ll_register_callbacks cb 0x7ffa40005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:48.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.791+0000 7ffa47400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605c25c31d0 0x5605c25bfc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.791+0000 7ffa4ce00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5605c25c5880 0x5605c25c01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.791+0000 7ffa47400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605c25c31d0 0x5605c25bfc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52846/0 (socket says 172.21.15.43:52846) 2024-03-28T22:39:48.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.791+0000 7ffa47400640 1 -- 172.21.15.43:0/1456978266 learned_addr learned my addr 172.21.15.43:0/1456978266 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:48.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.791+0000 7ffa47e00640 1 --2- 172.21.15.43:0/1456978266 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5605c25bcca0 0x5605c25bf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.791+0000 7ffa47400640 1 -- 172.21.15.43:0/1456978266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5605c25c5880 msgr2=0x5605c25c01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:48.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.791+0000 7ffa47400640 1 --2- 172.21.15.43:0/1456978266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5605c25c5880 0x5605c25c01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.791+0000 7ffa47400640 1 -- 172.21.15.43:0/1456978266 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5605c25bcca0 msgr2=0x5605c25bf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:48.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.791+0000 7ffa47400640 1 --2- 172.21.15.43:0/1456978266 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5605c25bcca0 0x5605c25bf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.791+0000 7ffa47400640 1 -- 172.21.15.43:0/1456978266 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605c263b230 con 0x5605c25c31d0 2024-03-28T22:39:48.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.792+0000 7ffa47400640 1 --2- 172.21.15.43:0/1456978266 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605c25c31d0 0x5605c25bfc60 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7ffa3402c2e0 tx=0x7ffa3402c7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2eed62d10a3036d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:48.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.792+0000 7ffa32000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:39:48.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.792+0000 7ffa32000640 1 -- 172.21.15.43:0/1456978266 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3405c070 con 0x5605c25c31d0 2024-03-28T22:39:48.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.792+0000 7ffa32000640 1 -- 172.21.15.43:0/1456978266 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7ffa34004930 con 0x5605c25c31d0 2024-03-28T22:39:48.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.792+0000 7ffa51c37740 1 -- 172.21.15.43:0/1456978266 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa34047030 con 0x5605c25c31d0 2024-03-28T22:39:48.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.792+0000 7ffa32000640 1 -- 172.21.15.43:0/1456978266 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ffa34004b30 con 0x5605c25c31d0 2024-03-28T22:39:48.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.792+0000 7ffa51c37740 10 client.26639 fetch_fsmap learned FSMap version 30 2024-03-28T22:39:48.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.792+0000 7ffa51c37740 1 -- 172.21.15.43:0/1456978266 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5605c263c510 con 0x5605c25c31d0 2024-03-28T22:39:48.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.793+0000 7ffa32000640 1 -- 172.21.15.43:0/1456978266 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa34047030 con 0x5605c25c31d0 2024-03-28T22:39:48.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.793+0000 7ffa32000640 1 -- 172.21.15.43:0/1456978266 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7ffa34005320 con 0x5605c25c31d0 2024-03-28T22:39:48.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.793+0000 7ffa51c37740 10 client.26639 fetch_fsmap finished waiting for FSMap version 30 2024-03-28T22:39:48.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.793+0000 7ffa51c37740 10 client.26639 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:39:48.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.793+0000 7ffa51c37740 20 client.26639 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:48.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.793+0000 7ffa51c37740 1 --2- 172.21.15.43:0/1456978266 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5605c263cc00 0x5605c263f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:48.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.793+0000 7ffa51c37740 4 client.26639 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:39:48.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.793+0000 7ffa51c37740 1 -- 172.21.15.43:0/1456978266 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5605c252d590 con 0x5605c263cc00 2024-03-28T22:39:48.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.794+0000 7ffa47e00640 1 --2- 172.21.15.43:0/1456978266 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5605c263cc00 0x5605c263f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:48.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.794+0000 7ffa47e00640 1 --2- 172.21.15.43:0/1456978266 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5605c263cc00 0x5605c263f020 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f281316e35a75eae server_cookie=2a8df0deef4a6c69 in_seq=0 out_seq=0 2024-03-28T22:39:48.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.794+0000 7ffa32000640 10 client.26639 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:48.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.796+0000 7ffa32000640 1 -- 172.21.15.43:0/1456978266 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5605c252d590 con 0x5605c263cc00 2024-03-28T22:39:48.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.796+0000 7ffa32000640 10 client.26639 handle_command_reply: tid=0 2024-03-28T22:39:48.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.796+0000 7ffa51c37740 1 client.26639 shutdown 2024-03-28T22:39:48.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.796+0000 7ffa51c37740 1 -- 172.21.15.43:0/1456978266 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605c25c31d0 msgr2=0x5605c25bfc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:48.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.796+0000 7ffa51c37740 1 --2- 172.21.15.43:0/1456978266 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605c25c31d0 0x5605c25bfc60 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7ffa3402c2e0 tx=0x7ffa3402c7a0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.796+0000 7ffa51c37740 1 -- 172.21.15.43:0/1456978266 shutdown_connections 2024-03-28T22:39:48.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.796+0000 7ffa51c37740 1 --2- 172.21.15.43:0/1456978266 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5605c263cc00 0x5605c263f020 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.796+0000 7ffa51c37740 1 --2- 172.21.15.43:0/1456978266 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605c25c31d0 0x5605c25bfc60 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.796+0000 7ffa51c37740 1 --2- 172.21.15.43:0/1456978266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5605c25c5880 0x5605c25c01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.796+0000 7ffa51c37740 1 --2- 172.21.15.43:0/1456978266 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5605c25bcca0 0x5605c25bf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.796+0000 7ffa51c37740 1 -- 172.21.15.43:0/1456978266 >> 172.21.15.43:0/1456978266 conn(0x5605c25b6ab0 msgr2=0x5605c25c21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:48.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.796+0000 7ffa32000640 0 client.26639 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:48.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.796+0000 7ffa51c37740 1 -- 172.21.15.43:0/1456978266 shutdown_connections 2024-03-28T22:39:48.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.796+0000 7ffa51c37740 1 -- 172.21.15.43:0/1456978266 wait complete. 2024-03-28T22:39:48.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.797+0000 7ffa51c37740 20 client.26639 trim_cache size 0 max 16384 2024-03-28T22:39:48.798 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:39:48.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.797+0000 7ffa4ee00640 1 -- 172.21.15.43:0/1049125464 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ffa2c0a1f70 msgr2=0x7ffa2c0a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:48.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.797+0000 7ffa4ee00640 1 --2- 172.21.15.43:0/1049125464 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ffa2c0a1f70 0x7ffa2c0a4430 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ffa34005ab0 tx=0x7ffa34005b50 comp rx=0 tx=0).stop 2024-03-28T22:39:48.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.797+0000 7ffa4ee00640 1 -- 172.21.15.43:0/1049125464 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffa4810bd90 msgr2=0x7ffa48101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:48.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.797+0000 7ffa4ee00640 1 --2- 172.21.15.43:0/1049125464 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffa4810bd90 0x7ffa48101090 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ffa4000e970 tx=0x7ffa4000ee30 comp rx=0 tx=0).stop 2024-03-28T22:39:48.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.797+0000 7ffa4ee00640 1 -- 172.21.15.43:0/1049125464 shutdown_connections 2024-03-28T22:39:48.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.797+0000 7ffa4ee00640 1 --2- 172.21.15.43:0/1049125464 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ffa2c0a1f70 0x7ffa2c0a4430 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.797+0000 7ffa4ee00640 1 --2- 172.21.15.43:0/1049125464 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffa4810bd90 0x7ffa48101090 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.797+0000 7ffa4ee00640 1 --2- 172.21.15.43:0/1049125464 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffa481032e0 0x7ffa48100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.797+0000 7ffa4ee00640 1 --2- 172.21.15.43:0/1049125464 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa481029c0 0x7ffa48100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:48.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.797+0000 7ffa4ee00640 1 -- 172.21.15.43:0/1049125464 >> 172.21.15.43:0/1049125464 conn(0x7ffa480777b0 msgr2=0x7ffa480fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:48.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.798+0000 7ffa4ee00640 1 -- 172.21.15.43:0/1049125464 shutdown_connections 2024-03-28T22:39:48.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:48.798+0000 7ffa4ee00640 1 -- 172.21.15.43:0/1049125464 wait complete. 2024-03-28T22:39:48.819 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:48 smithi043 ceph-mon[31823]: pgmap v82: 129 pgs: 129 active+clean; 41 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 355 op/s 2024-03-28T22:39:48.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:48 smithi172 ceph-mon[36566]: pgmap v82: 129 pgs: 129 active+clean; 41 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 355 op/s 2024-03-28T22:39:48.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:48 smithi082 ceph-mon[36471]: pgmap v82: 129 pgs: 129 active+clean; 41 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 355 op/s 2024-03-28T22:39:49.347 INFO:tasks.workunit.client.0.smithi043.stdout: 1 128039 8.86 MB/sec warmup 80 sec latency 12.354 ms 2024-03-28T22:39:49.824 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:39:49.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:39:49.824 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:39:50.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.079+0000 7f3a50e00640 1 -- 172.21.15.43:0/3971990813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a4c10cf50 msgr2=0x7f3a4c114b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:50.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.079+0000 7f3a50e00640 1 --2- 172.21.15.43:0/3971990813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a4c10cf50 0x7f3a4c114b20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f3a44009be0 tx=0x7f3a4402bef0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.079+0000 7f3a50e00640 1 -- 172.21.15.43:0/3971990813 shutdown_connections 2024-03-28T22:39:50.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.079+0000 7f3a50e00640 1 --2- 172.21.15.43:0/3971990813 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a4c1088b0 0x7f3a4c10c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.079+0000 7f3a50e00640 1 --2- 172.21.15.43:0/3971990813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a4c10cf50 0x7f3a4c114b20 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.079+0000 7f3a50e00640 1 --2- 172.21.15.43:0/3971990813 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a4c107f00 0x7f3a4c1082e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.079+0000 7f3a50e00640 1 -- 172.21.15.43:0/3971990813 >> 172.21.15.43:0/3971990813 conn(0x7f3a4c077570 msgr2=0x7f3a4c106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:50.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.080+0000 7f3a50e00640 1 -- 172.21.15.43:0/3971990813 shutdown_connections 2024-03-28T22:39:50.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.080+0000 7f3a50e00640 1 -- 172.21.15.43:0/3971990813 wait complete. 2024-03-28T22:39:50.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.081+0000 7f3a50e00640 1 Processor -- start 2024-03-28T22:39:50.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.081+0000 7f3a50e00640 1 -- start start 2024-03-28T22:39:50.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.081+0000 7f3a50e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a4c107f00 0x7f3a4c19e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.081+0000 7f3a50e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a4c10cf50 0x7f3a4c19e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.081+0000 7f3a50e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a4c1a2d50 0x7f3a4c1a3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.081+0000 7f3a50e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c118050 con 0x7f3a4c10cf50 2024-03-28T22:39:50.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.081+0000 7f3a50e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c117ed0 con 0x7f3a4c1a2d50 2024-03-28T22:39:50.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.081+0000 7f3a50e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c1181d0 con 0x7f3a4c107f00 2024-03-28T22:39:50.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.082+0000 7f3a4be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a4c1a2d50 0x7f3a4c1a3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.082+0000 7f3a4b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a4c107f00 0x7f3a4c19e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.082+0000 7f3a4be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a4c1a2d50 0x7f3a4c1a3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52856/0 (socket says 172.21.15.43:52856) 2024-03-28T22:39:50.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.082+0000 7f3a4be00640 1 -- 172.21.15.43:0/1457061514 learned_addr learned my addr 172.21.15.43:0/1457061514 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:50.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.082+0000 7f3a4aa00640 1 --2- 172.21.15.43:0/1457061514 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a4c10cf50 0x7f3a4c19e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.082+0000 7f3a4be00640 1 -- 172.21.15.43:0/1457061514 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a4c107f00 msgr2=0x7f3a4c19e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:50.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.082+0000 7f3a4be00640 1 --2- 172.21.15.43:0/1457061514 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a4c107f00 0x7f3a4c19e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.082+0000 7f3a4be00640 1 -- 172.21.15.43:0/1457061514 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a4c10cf50 msgr2=0x7f3a4c19e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:50.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.082+0000 7f3a4be00640 1 --2- 172.21.15.43:0/1457061514 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a4c10cf50 0x7f3a4c19e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.082+0000 7f3a4be00640 1 -- 172.21.15.43:0/1457061514 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a4c1a3840 con 0x7f3a4c1a2d50 2024-03-28T22:39:50.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.082+0000 7f3a4be00640 1 --2- 172.21.15.43:0/1457061514 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a4c1a2d50 0x7f3a4c1a3200 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f3a44004af0 tx=0x7f3a440336b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:50.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.083+0000 7f3a43e00640 1 -- 172.21.15.43:0/1457061514 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a44043020 con 0x7f3a4c1a2d50 2024-03-28T22:39:50.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.083+0000 7f3a43e00640 1 -- 172.21.15.43:0/1457061514 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a44003e50 con 0x7f3a4c1a2d50 2024-03-28T22:39:50.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.083+0000 7f3a50e00640 1 -- 172.21.15.43:0/1457061514 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a4c1a3ad0 con 0x7f3a4c1a2d50 2024-03-28T22:39:50.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.083+0000 7f3a43e00640 1 -- 172.21.15.43:0/1457061514 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4403e450 con 0x7f3a4c1a2d50 2024-03-28T22:39:50.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.083+0000 7f3a50e00640 1 -- 172.21.15.43:0/1457061514 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a4c1ab320 con 0x7f3a4c1a2d50 2024-03-28T22:39:50.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.087+0000 7f3a43e00640 1 -- 172.21.15.43:0/1457061514 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f3a44046020 con 0x7f3a4c1a2d50 2024-03-28T22:39:50.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.087+0000 7f3a55024740 1 Processor -- start 2024-03-28T22:39:50.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.088+0000 7f3a55024740 1 -- start start 2024-03-28T22:39:50.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.088+0000 7f3a55024740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625ae58d170 0x5625ae58d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.088+0000 7f3a55024740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625ae58da90 0x5625ae596130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.088+0000 7f3a55024740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625ae596670 0x5625ae598a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.088+0000 7f3a55024740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5625ae4f0c40 con 0x5625ae58da90 2024-03-28T22:39:50.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.088+0000 7f3a55024740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5625ae499c40 con 0x5625ae58d170 2024-03-28T22:39:50.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.088+0000 7f3a55024740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5625ae4ecce0 con 0x5625ae596670 2024-03-28T22:39:50.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.088+0000 7f3a4aa00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625ae58d170 0x5625ae58d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.088+0000 7f3a4aa00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625ae58d170 0x5625ae58d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52866/0 (socket says 172.21.15.43:52866) 2024-03-28T22:39:50.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.088+0000 7f3a4aa00640 1 -- 172.21.15.43:0/2036258531 learned_addr learned my addr 172.21.15.43:0/2036258531 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:50.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.088+0000 7f3a4b400640 1 --2- 172.21.15.43:0/2036258531 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625ae58da90 0x5625ae596130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.089+0000 7f3a4aa00640 1 --2- 172.21.15.43:0/2036258531 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625ae596670 0x5625ae598a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.089+0000 7f3a4aa00640 1 -- 172.21.15.43:0/2036258531 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625ae596670 msgr2=0x5625ae598a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:50.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.089+0000 7f3a4aa00640 1 --2- 172.21.15.43:0/2036258531 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625ae596670 0x5625ae598a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.089+0000 7f3a4aa00640 1 -- 172.21.15.43:0/2036258531 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625ae58da90 msgr2=0x5625ae596130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:50.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.089+0000 7f3a4aa00640 1 --2- 172.21.15.43:0/2036258531 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625ae58da90 0x5625ae596130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.089+0000 7f3a4aa00640 1 -- 172.21.15.43:0/2036258531 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625ae2cd7e0 con 0x5625ae58d170 2024-03-28T22:39:50.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.089+0000 7f3a43e00640 1 --2- 172.21.15.43:0/1457061514 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f3a300a1f70 0x7f3a300a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.089+0000 7f3a4b400640 1 --2- 172.21.15.43:0/2036258531 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625ae58da90 0x5625ae596130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:39:50.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.089+0000 7f3a43e00640 1 -- 172.21.15.43:0/1457061514 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7f3a440f0930 con 0x7f3a4c1a2d50 2024-03-28T22:39:50.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.089+0000 7f3a4aa00640 1 --2- 172.21.15.43:0/2036258531 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625ae58d170 0x5625ae58d550 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f3a38013720 tx=0x7f3a38009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=932c7adecdbe03f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:50.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.089+0000 7f3a4b400640 1 --2- 172.21.15.43:0/1457061514 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f3a300a1f70 0x7f3a300a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.089+0000 7f3a40c00640 1 -- 172.21.15.43:0/2036258531 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a38013d60 con 0x5625ae58d170 2024-03-28T22:39:50.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.090+0000 7f3a40c00640 1 -- 172.21.15.43:0/2036258531 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a38013170 con 0x5625ae58d170 2024-03-28T22:39:50.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.090+0000 7f3a40c00640 1 -- 172.21.15.43:0/2036258531 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a380035f0 con 0x5625ae58d170 2024-03-28T22:39:50.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.090+0000 7f3a55024740 1 -- 172.21.15.43:0/2036258531 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625ae58d170 msgr2=0x5625ae58d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:50.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.090+0000 7f3a55024740 1 --2- 172.21.15.43:0/2036258531 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625ae58d170 0x5625ae58d550 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f3a38013720 tx=0x7f3a38009260 comp rx=0 tx=0).stop 2024-03-28T22:39:50.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.090+0000 7f3a4b400640 1 --2- 172.21.15.43:0/1457061514 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f3a300a1f70 0x7f3a300a4430 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c0095f0 tx=0x7f3a3c005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:50.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.091+0000 7f3a55024740 1 -- 172.21.15.43:0/2036258531 shutdown_connections 2024-03-28T22:39:50.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.091+0000 7f3a55024740 1 --2- 172.21.15.43:0/2036258531 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625ae596670 0x5625ae598a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.091+0000 7f3a55024740 1 --2- 172.21.15.43:0/2036258531 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625ae58da90 0x5625ae596130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.091+0000 7f3a55024740 1 --2- 172.21.15.43:0/2036258531 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625ae58d170 0x5625ae58d550 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.091+0000 7f3a55024740 1 -- 172.21.15.43:0/2036258531 >> 172.21.15.43:0/2036258531 conn(0x5625ae5878a0 msgr2=0x5625ae587c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:50.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.091+0000 7f3a55024740 1 -- 172.21.15.43:0/2036258531 shutdown_connections 2024-03-28T22:39:50.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.091+0000 7f3a55024740 1 -- 172.21.15.43:0/2036258531 wait complete. 2024-03-28T22:39:50.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.091+0000 7f3a55024740 1 -- start start 2024-03-28T22:39:50.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.092+0000 7f3a55024740 1 Processor -- start 2024-03-28T22:39:50.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.092+0000 7f3a55024740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625ae58d170 0x5625ae593680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.092+0000 7f3a55024740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625ae58da90 0x5625ae593bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.092+0000 7f3a55024740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625ae596670 0x5625ae594100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.092+0000 7f3a55024740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5625ae599460 con 0x5625ae58da90 2024-03-28T22:39:50.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.092+0000 7f3a55024740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5625ae48f250 con 0x5625ae596670 2024-03-28T22:39:50.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.092+0000 7f3a55024740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5625ae5995e0 con 0x5625ae58d170 2024-03-28T22:39:50.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.092+0000 7f3a55024740 10 client.0 _ll_register_callbacks cb 0x5625ae4b0e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:50.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.092+0000 7f3a4b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625ae58d170 0x5625ae593680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.092+0000 7f3a4b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625ae58d170 0x5625ae593680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:56260/0 (socket says 172.21.15.43:56260) 2024-03-28T22:39:50.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.092+0000 7f3a4b400640 1 -- 172.21.15.43:0/2264650837 learned_addr learned my addr 172.21.15.43:0/2264650837 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:50.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.092+0000 7f3a4aa00640 1 --2- 172.21.15.43:0/2264650837 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625ae58da90 0x5625ae593bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.092+0000 7f3a4be00640 1 --2- 172.21.15.43:0/2264650837 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625ae596670 0x5625ae594100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.093+0000 7f3a4b400640 1 -- 172.21.15.43:0/2264650837 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625ae596670 msgr2=0x5625ae594100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:50.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.093+0000 7f3a4b400640 1 --2- 172.21.15.43:0/2264650837 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625ae596670 0x5625ae594100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.093+0000 7f3a4b400640 1 -- 172.21.15.43:0/2264650837 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625ae58da90 msgr2=0x5625ae593bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:50.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.093+0000 7f3a4b400640 1 --2- 172.21.15.43:0/2264650837 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625ae58da90 0x5625ae593bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.093+0000 7f3a4b400640 1 -- 172.21.15.43:0/2264650837 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625ae32df70 con 0x5625ae58d170 2024-03-28T22:39:50.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.093+0000 7f3a4be00640 1 --2- 172.21.15.43:0/2264650837 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625ae596670 0x5625ae594100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:50.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.093+0000 7f3a4b400640 1 --2- 172.21.15.43:0/2264650837 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625ae58d170 0x5625ae593680 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c01e9d0 tx=0x7f3a3c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7282709dd99263 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:50.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.093+0000 7f3a35600640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:39:50.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.094+0000 7f3a35600640 1 -- 172.21.15.43:0/2264650837 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c02e060 con 0x5625ae58d170 2024-03-28T22:39:50.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.094+0000 7f3a35600640 1 -- 172.21.15.43:0/2264650837 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7f3a3c018030 con 0x5625ae58d170 2024-03-28T22:39:50.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.094+0000 7f3a55024740 1 -- 172.21.15.43:0/2264650837 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a38013170 con 0x5625ae58d170 2024-03-28T22:39:50.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.094+0000 7f3a35600640 1 -- 172.21.15.43:0/2264650837 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3a3c035050 con 0x5625ae58d170 2024-03-28T22:39:50.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.094+0000 7f3a55024740 10 client.26599 fetch_fsmap learned FSMap version 30 2024-03-28T22:39:50.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.094+0000 7f3a55024740 1 -- 172.21.15.43:0/2264650837 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5625ae1c5170 con 0x5625ae58d170 2024-03-28T22:39:50.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.094+0000 7f3a35600640 1 -- 172.21.15.43:0/2264650837 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a38013170 con 0x5625ae58d170 2024-03-28T22:39:50.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.095+0000 7f3a35600640 1 -- 172.21.15.43:0/2264650837 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f3a3c01c410 con 0x5625ae58d170 2024-03-28T22:39:50.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.095+0000 7f3a55024740 10 client.26599 fetch_fsmap finished waiting for FSMap version 30 2024-03-28T22:39:50.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.095+0000 7f3a55024740 10 client.26599 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:39:50.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.095+0000 7f3a55024740 20 client.26599 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:50.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.095+0000 7f3a55024740 1 --2- 172.21.15.43:0/2264650837 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5625ae5a6ab0 0x5625ae5a8f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.095+0000 7f3a55024740 4 client.26599 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:39:50.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.095+0000 7f3a55024740 1 -- 172.21.15.43:0/2264650837 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625ae4fe180 con 0x5625ae5a6ab0 2024-03-28T22:39:50.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.095+0000 7f3a4aa00640 1 --2- 172.21.15.43:0/2264650837 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5625ae5a6ab0 0x5625ae5a8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.096+0000 7f3a4aa00640 1 --2- 172.21.15.43:0/2264650837 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5625ae5a6ab0 0x5625ae5a8f10 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22ed7e78205e9dd0 server_cookie=406f32eb0734419f in_seq=0 out_seq=0 2024-03-28T22:39:50.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.096+0000 7f3a35600640 10 client.26599 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:50.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.096+0000 7f3a35600640 1 -- 172.21.15.43:0/2264650837 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5625ae4fe180 con 0x5625ae5a6ab0 2024-03-28T22:39:50.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.097+0000 7f3a35600640 10 client.26599 handle_command_reply: tid=0 2024-03-28T22:39:50.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.097+0000 7f3a55024740 1 client.26599 shutdown 2024-03-28T22:39:50.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.097+0000 7f3a55024740 1 -- 172.21.15.43:0/2264650837 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625ae58d170 msgr2=0x5625ae593680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:50.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.097+0000 7f3a55024740 1 --2- 172.21.15.43:0/2264650837 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625ae58d170 0x5625ae593680 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c01e9d0 tx=0x7f3a3c004730 comp rx=0 tx=0).stop 2024-03-28T22:39:50.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.097+0000 7f3a55024740 1 -- 172.21.15.43:0/2264650837 shutdown_connections 2024-03-28T22:39:50.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.097+0000 7f3a55024740 1 --2- 172.21.15.43:0/2264650837 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5625ae5a6ab0 0x5625ae5a8f10 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.097+0000 7f3a55024740 1 --2- 172.21.15.43:0/2264650837 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625ae596670 0x5625ae594100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.097+0000 7f3a55024740 1 --2- 172.21.15.43:0/2264650837 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625ae58da90 0x5625ae593bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.097+0000 7f3a55024740 1 --2- 172.21.15.43:0/2264650837 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625ae58d170 0x5625ae593680 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.097+0000 7f3a55024740 1 -- 172.21.15.43:0/2264650837 >> 172.21.15.43:0/2264650837 conn(0x5625ae5878a0 msgr2=0x5625ae5895f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:50.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.097+0000 7f3a35600640 0 client.26599 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:50.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.097+0000 7f3a55024740 1 -- 172.21.15.43:0/2264650837 shutdown_connections 2024-03-28T22:39:50.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.097+0000 7f3a55024740 1 -- 172.21.15.43:0/2264650837 wait complete. 2024-03-28T22:39:50.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.097+0000 7f3a55024740 20 client.26599 trim_cache size 0 max 16384 2024-03-28T22:39:50.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.117+0000 7f3a55024740 1 Processor -- start 2024-03-28T22:39:50.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.117+0000 7f3a55024740 1 -- start start 2024-03-28T22:39:50.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.118+0000 7f3a55024740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625ae58d170 0x5625ae595db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.118+0000 7f3a55024740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625ae58da90 0x5625ae5a2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.118+0000 7f3a55024740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625ae596670 0x5625ae5ae7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.118+0000 7f3a55024740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c1b4f90 con 0x5625ae58da90 2024-03-28T22:39:50.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.118+0000 7f3a55024740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c1b6ad0 con 0x5625ae58d170 2024-03-28T22:39:50.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.118+0000 7f3a55024740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c1b5660 con 0x5625ae596670 2024-03-28T22:39:50.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.118+0000 7f3a4b400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625ae58d170 0x5625ae595db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.118+0000 7f3a4be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625ae596670 0x5625ae5ae7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.118+0000 7f3a4b400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625ae58d170 0x5625ae595db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52874/0 (socket says 172.21.15.43:52874) 2024-03-28T22:39:50.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.118+0000 7f3a4b400640 1 -- 172.21.15.43:0/3901987978 learned_addr learned my addr 172.21.15.43:0/3901987978 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:50.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.118+0000 7f3a4aa00640 1 --2- 172.21.15.43:0/3901987978 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625ae58da90 0x5625ae5a2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.118+0000 7f3a4b400640 1 -- 172.21.15.43:0/3901987978 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625ae596670 msgr2=0x5625ae5ae7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:50.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.118+0000 7f3a4b400640 1 --2- 172.21.15.43:0/3901987978 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625ae596670 0x5625ae5ae7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.118+0000 7f3a4b400640 1 -- 172.21.15.43:0/3901987978 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625ae58da90 msgr2=0x5625ae5a2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:50.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.118+0000 7f3a4b400640 1 --2- 172.21.15.43:0/3901987978 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625ae58da90 0x5625ae5a2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.118+0000 7f3a4b400640 1 -- 172.21.15.43:0/3901987978 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625ae581b40 con 0x5625ae58d170 2024-03-28T22:39:50.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.118+0000 7f3a4be00640 1 --2- 172.21.15.43:0/3901987978 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625ae596670 0x5625ae5ae7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:50.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.119+0000 7f3a4b400640 1 --2- 172.21.15.43:0/3901987978 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625ae58d170 0x5625ae595db0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c019040 tx=0x7f3a3c01ee90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d979be0329702714 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:50.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.119+0000 7f3a37e00640 1 -- 172.21.15.43:0/3901987978 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c03d020 con 0x5625ae58d170 2024-03-28T22:39:50.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.119+0000 7f3a37e00640 1 -- 172.21.15.43:0/3901987978 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a3c009de0 con 0x5625ae58d170 2024-03-28T22:39:50.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.119+0000 7f3a37e00640 1 -- 172.21.15.43:0/3901987978 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c029ac0 con 0x5625ae58d170 2024-03-28T22:39:50.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.120+0000 7f3a55024740 1 -- 172.21.15.43:0/3901987978 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625ae58d170 msgr2=0x5625ae595db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:50.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.120+0000 7f3a55024740 1 --2- 172.21.15.43:0/3901987978 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625ae58d170 0x5625ae595db0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c019040 tx=0x7f3a3c01ee90 comp rx=0 tx=0).stop 2024-03-28T22:39:50.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.120+0000 7f3a55024740 1 -- 172.21.15.43:0/3901987978 shutdown_connections 2024-03-28T22:39:50.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.120+0000 7f3a55024740 1 --2- 172.21.15.43:0/3901987978 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625ae596670 0x5625ae5ae7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.120+0000 7f3a55024740 1 --2- 172.21.15.43:0/3901987978 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625ae58da90 0x5625ae5a2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.120+0000 7f3a55024740 1 --2- 172.21.15.43:0/3901987978 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625ae58d170 0x5625ae595db0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.120+0000 7f3a55024740 1 -- 172.21.15.43:0/3901987978 >> 172.21.15.43:0/3901987978 conn(0x5625ae5878a0 msgr2=0x5625ae586070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:50.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.120+0000 7f3a55024740 1 -- 172.21.15.43:0/3901987978 shutdown_connections 2024-03-28T22:39:50.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.120+0000 7f3a55024740 1 -- 172.21.15.43:0/3901987978 wait complete. 2024-03-28T22:39:50.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.120+0000 7f3a55024740 1 -- start start 2024-03-28T22:39:50.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.120+0000 7f3a55024740 1 Processor -- start 2024-03-28T22:39:50.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.121+0000 7f3a55024740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625ae58d170 0x5625ae59a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.121+0000 7f3a55024740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625ae58da90 0x5625ae59aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.121+0000 7f3a55024740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625ae596670 0x5625ae60be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.121+0000 7f3a55024740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c1b6400 con 0x5625ae58d170 2024-03-28T22:39:50.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.121+0000 7f3a55024740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c1b5d30 con 0x5625ae596670 2024-03-28T22:39:50.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.121+0000 7f3a55024740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c1b2f70 con 0x5625ae58da90 2024-03-28T22:39:50.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.121+0000 7f3a55024740 10 client.0 _ll_register_callbacks cb 0x7f3a3c005d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:50.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.121+0000 7f3a4be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625ae596670 0x5625ae60be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.121+0000 7f3a4aa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625ae58da90 0x5625ae59aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.121+0000 7f3a4b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625ae58d170 0x5625ae59a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.121+0000 7f3a4be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625ae596670 0x5625ae60be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52882/0 (socket says 172.21.15.43:52882) 2024-03-28T22:39:50.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.121+0000 7f3a4be00640 1 -- 172.21.15.43:0/413215017 learned_addr learned my addr 172.21.15.43:0/413215017 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:50.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.121+0000 7f3a4aa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625ae58da90 0x5625ae59aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:56280/0 (socket says 172.21.15.43:56280) 2024-03-28T22:39:50.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.122+0000 7f3a4be00640 1 -- 172.21.15.43:0/413215017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625ae58da90 msgr2=0x5625ae59aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:50.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.122+0000 7f3a4be00640 1 --2- 172.21.15.43:0/413215017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625ae58da90 0x5625ae59aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.122+0000 7f3a4be00640 1 -- 172.21.15.43:0/413215017 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625ae58d170 msgr2=0x5625ae59a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:50.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.122+0000 7f3a4be00640 1 --2- 172.21.15.43:0/413215017 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625ae58d170 0x5625ae59a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.122+0000 7f3a4be00640 1 -- 172.21.15.43:0/413215017 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625ae60c350 con 0x5625ae596670 2024-03-28T22:39:50.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.122+0000 7f3a4aa00640 1 --2- 172.21.15.43:0/413215017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625ae58da90 0x5625ae59aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:50.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.122+0000 7f3a4be00640 1 --2- 172.21.15.43:0/413215017 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625ae596670 0x5625ae60be10 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3a44033420 tx=0x7f3a440bca50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cff74a06a8601f30 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:50.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.122+0000 7f3a35600640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:39:50.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.122+0000 7f3a35600640 1 -- 172.21.15.43:0/413215017 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a440d4060 con 0x5625ae596670 2024-03-28T22:39:50.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.122+0000 7f3a35600640 1 -- 172.21.15.43:0/413215017 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7f3a440c5540 con 0x5625ae596670 2024-03-28T22:39:50.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.122+0000 7f3a55024740 1 -- 172.21.15.43:0/413215017 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a3c009de0 con 0x5625ae596670 2024-03-28T22:39:50.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.122+0000 7f3a35600640 1 -- 172.21.15.43:0/413215017 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3a440c1180 con 0x5625ae596670 2024-03-28T22:39:50.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.122+0000 7f3a55024740 10 client.26657 fetch_fsmap learned FSMap version 30 2024-03-28T22:39:50.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.122+0000 7f3a55024740 1 -- 172.21.15.43:0/413215017 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5625ae60d570 con 0x5625ae596670 2024-03-28T22:39:50.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.123+0000 7f3a35600640 1 -- 172.21.15.43:0/413215017 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a4409be50 con 0x5625ae596670 2024-03-28T22:39:50.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.123+0000 7f3a35600640 1 -- 172.21.15.43:0/413215017 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f3a44004b40 con 0x5625ae596670 2024-03-28T22:39:50.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.123+0000 7f3a55024740 10 client.26657 fetch_fsmap finished waiting for FSMap version 30 2024-03-28T22:39:50.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.123+0000 7f3a55024740 10 client.26657 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:39:50.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.123+0000 7f3a55024740 20 client.26657 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:50.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.123+0000 7f3a55024740 1 --2- 172.21.15.43:0/413215017 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5625ae60dce0 0x5625ae610100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.123+0000 7f3a55024740 4 client.26657 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:39:50.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.123+0000 7f3a55024740 1 -- 172.21.15.43:0/413215017 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5625ae4f84d0 con 0x5625ae60dce0 2024-03-28T22:39:50.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.123+0000 7f3a4b400640 1 --2- 172.21.15.43:0/413215017 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5625ae60dce0 0x5625ae610100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.124+0000 7f3a4b400640 1 --2- 172.21.15.43:0/413215017 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5625ae60dce0 0x5625ae610100 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24c0faa160097e89 server_cookie=c944314012eee56 in_seq=0 out_seq=0 2024-03-28T22:39:50.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.124+0000 7f3a35600640 10 client.26657 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:50.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.127+0000 7f3a35600640 1 -- 172.21.15.43:0/413215017 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5625ae4f84d0 con 0x5625ae60dce0 2024-03-28T22:39:50.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.127+0000 7f3a35600640 10 client.26657 handle_command_reply: tid=0 2024-03-28T22:39:50.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.127+0000 7f3a55024740 1 client.26657 shutdown 2024-03-28T22:39:50.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.127+0000 7f3a55024740 1 -- 172.21.15.43:0/413215017 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625ae596670 msgr2=0x5625ae60be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:50.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.127+0000 7f3a55024740 1 --2- 172.21.15.43:0/413215017 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625ae596670 0x5625ae60be10 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3a44033420 tx=0x7f3a440bca50 comp rx=0 tx=0).stop 2024-03-28T22:39:50.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.127+0000 7f3a55024740 1 -- 172.21.15.43:0/413215017 shutdown_connections 2024-03-28T22:39:50.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.127+0000 7f3a55024740 1 --2- 172.21.15.43:0/413215017 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5625ae60dce0 0x5625ae610100 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.127+0000 7f3a55024740 1 --2- 172.21.15.43:0/413215017 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625ae596670 0x5625ae60be10 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.127+0000 7f3a55024740 1 --2- 172.21.15.43:0/413215017 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625ae58da90 0x5625ae59aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.127+0000 7f3a55024740 1 --2- 172.21.15.43:0/413215017 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625ae58d170 0x5625ae59a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.127+0000 7f3a35600640 0 client.26657 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:50.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.127+0000 7f3a55024740 1 -- 172.21.15.43:0/413215017 >> 172.21.15.43:0/413215017 conn(0x5625ae5878a0 msgr2=0x5625ae589430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:50.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.127+0000 7f3a55024740 1 -- 172.21.15.43:0/413215017 shutdown_connections 2024-03-28T22:39:50.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.127+0000 7f3a55024740 1 -- 172.21.15.43:0/413215017 wait complete. 2024-03-28T22:39:50.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.127+0000 7f3a55024740 20 client.26657 trim_cache size 0 max 16384 2024-03-28T22:39:50.128 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:39:50.128 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-03-28T22:39:50.128 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "1bd6dd92-11fd-4621-9932-739815bf9cdd", 2024-03-28T22:39:50.128 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-03-28T22:39:50.128 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:39:50.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.128+0000 7f3a50e00640 1 -- 172.21.15.43:0/1457061514 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f3a300a1f70 msgr2=0x7f3a300a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:50.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.128+0000 7f3a50e00640 1 --2- 172.21.15.43:0/1457061514 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f3a300a1f70 0x7f3a300a4430 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c0095f0 tx=0x7f3a3c005f90 comp rx=0 tx=0).stop 2024-03-28T22:39:50.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.128+0000 7f3a50e00640 1 -- 172.21.15.43:0/1457061514 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a4c1a2d50 msgr2=0x7f3a4c1a3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:50.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.128+0000 7f3a50e00640 1 --2- 172.21.15.43:0/1457061514 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a4c1a2d50 0x7f3a4c1a3200 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f3a44004af0 tx=0x7f3a440336b0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.128+0000 7f3a50e00640 1 -- 172.21.15.43:0/1457061514 shutdown_connections 2024-03-28T22:39:50.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.128+0000 7f3a50e00640 1 --2- 172.21.15.43:0/1457061514 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f3a300a1f70 0x7f3a300a4430 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.128+0000 7f3a50e00640 1 --2- 172.21.15.43:0/1457061514 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a4c1a2d50 0x7f3a4c1a3200 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.128+0000 7f3a50e00640 1 --2- 172.21.15.43:0/1457061514 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3a4c10cf50 0x7f3a4c19e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.128+0000 7f3a50e00640 1 --2- 172.21.15.43:0/1457061514 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3a4c107f00 0x7f3a4c19e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.128+0000 7f3a50e00640 1 -- 172.21.15.43:0/1457061514 >> 172.21.15.43:0/1457061514 conn(0x7f3a4c077570 msgr2=0x7f3a4c077af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:50.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.129+0000 7f3a50e00640 1 -- 172.21.15.43:0/1457061514 shutdown_connections 2024-03-28T22:39:50.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.129+0000 7f3a50e00640 1 -- 172.21.15.43:0/1457061514 wait complete. 2024-03-28T22:39:50.146 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:39:50.146 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:39:50.347 INFO:tasks.workunit.client.0.smithi043.stdout: 1 129250 8.82 MB/sec warmup 81 sec latency 64.309 ms 2024-03-28T22:39:50.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.366+0000 7f8312a00640 1 -- 172.21.15.43:0/243643682 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f830c103300 msgr2=0x7f830c10b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:50.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.366+0000 7f8312a00640 1 --2- 172.21.15.43:0/243643682 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f830c103300 0x7f830c10b7e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f82f4009900 tx=0x7f82f402bcc0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.366+0000 7f8312a00640 1 -- 172.21.15.43:0/243643682 shutdown_connections 2024-03-28T22:39:50.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.366+0000 7f8312a00640 1 --2- 172.21.15.43:0/243643682 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f830c103300 0x7f830c10b7e0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.366+0000 7f8312a00640 1 --2- 172.21.15.43:0/243643682 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f830c10bdb0 0x7f830c10e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.366+0000 7f8312a00640 1 --2- 172.21.15.43:0/243643682 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f830c1029e0 0x7f830c102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.366+0000 7f8312a00640 1 -- 172.21.15.43:0/243643682 >> 172.21.15.43:0/243643682 conn(0x7f830c0777b0 msgr2=0x7f830c077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:50.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.366+0000 7f8312a00640 1 -- 172.21.15.43:0/243643682 shutdown_connections 2024-03-28T22:39:50.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.367+0000 7f8312a00640 1 -- 172.21.15.43:0/243643682 wait complete. 2024-03-28T22:39:50.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.368+0000 7f8312a00640 1 Processor -- start 2024-03-28T22:39:50.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.368+0000 7f8312a00640 1 -- start start 2024-03-28T22:39:50.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.368+0000 7f8312a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f830c1029e0 0x7f830c19a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.368+0000 7f8312a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f830c103300 0x7f830c19a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.368+0000 7f8312a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f830c10bdb0 0x7f830c19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.369+0000 7f8312a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f830c111b90 con 0x7f830c10bdb0 2024-03-28T22:39:50.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.369+0000 7f8312a00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f830c111a10 con 0x7f830c1029e0 2024-03-28T22:39:50.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.369+0000 7f8312a00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f830c111d10 con 0x7f830c103300 2024-03-28T22:39:50.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.369+0000 7f830be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f830c1029e0 0x7f830c19a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.369+0000 7f8303400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f830c103300 0x7f830c19a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.369+0000 7f8303400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f830c103300 0x7f830c19a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:56306/0 (socket says 172.21.15.43:56306) 2024-03-28T22:39:50.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.369+0000 7f8303400640 1 -- 172.21.15.43:0/3246199534 learned_addr learned my addr 172.21.15.43:0/3246199534 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:50.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.369+0000 7f830be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f830c1029e0 0x7f830c19a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52906/0 (socket says 172.21.15.43:52906) 2024-03-28T22:39:50.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.369+0000 7f8310a00640 1 --2- 172.21.15.43:0/3246199534 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f830c10bdb0 0x7f830c19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.369+0000 7f8303400640 1 -- 172.21.15.43:0/3246199534 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f830c1029e0 msgr2=0x7f830c19a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:50.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.369+0000 7f8303400640 1 --2- 172.21.15.43:0/3246199534 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f830c1029e0 0x7f830c19a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.369+0000 7f8303400640 1 -- 172.21.15.43:0/3246199534 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f830c10bdb0 msgr2=0x7f830c19eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:50.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.369+0000 7f8303400640 1 --2- 172.21.15.43:0/3246199534 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f830c10bdb0 0x7f830c19eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.369+0000 7f8303400640 1 -- 172.21.15.43:0/3246199534 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f830c19f2c0 con 0x7f830c103300 2024-03-28T22:39:50.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.369+0000 7f8310a00640 1 --2- 172.21.15.43:0/3246199534 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f830c10bdb0 0x7f830c19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:39:50.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.370+0000 7f8303400640 1 --2- 172.21.15.43:0/3246199534 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f830c103300 0x7f830c19a810 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f82f4000c00 tx=0x7f82f4004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:50.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.370+0000 7f8309600640 1 -- 172.21.15.43:0/3246199534 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f4042020 con 0x7f830c103300 2024-03-28T22:39:50.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.370+0000 7f8312a00640 1 -- 172.21.15.43:0/3246199534 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f830c19f550 con 0x7f830c103300 2024-03-28T22:39:50.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.370+0000 7f8309600640 1 -- 172.21.15.43:0/3246199534 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82f4005000 con 0x7f830c103300 2024-03-28T22:39:50.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.370+0000 7f8309600640 1 -- 172.21.15.43:0/3246199534 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f4003b20 con 0x7f830c103300 2024-03-28T22:39:50.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.370+0000 7f8312a00640 1 -- 172.21.15.43:0/3246199534 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f830c1a6e40 con 0x7f830c103300 2024-03-28T22:39:50.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.374+0000 7f83157aa740 1 Processor -- start 2024-03-28T22:39:50.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.374+0000 7f8309600640 1 -- 172.21.15.43:0/3246199534 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f82f4045020 con 0x7f830c103300 2024-03-28T22:39:50.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.374+0000 7f83157aa740 1 -- start start 2024-03-28T22:39:50.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.375+0000 7f83157aa740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5652f7676bc0 0x5652f7675910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.375+0000 7f83157aa740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5652f7675ee0 0x5652f7676360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.375+0000 7f83157aa740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5652f767f250 0x5652f7681640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.375+0000 7f83157aa740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5652f75da100 con 0x5652f767f250 2024-03-28T22:39:50.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.375+0000 7f83157aa740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5652f7582ba0 con 0x5652f7676bc0 2024-03-28T22:39:50.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.375+0000 7f83157aa740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5652f75d61a0 con 0x5652f7675ee0 2024-03-28T22:39:50.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.375+0000 7f8310a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5652f7676bc0 0x5652f7675910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.375+0000 7f830be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5652f7675ee0 0x5652f7676360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.375+0000 7f8310a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5652f7676bc0 0x5652f7675910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52920/0 (socket says 172.21.15.43:52920) 2024-03-28T22:39:50.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.375+0000 7f8303400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5652f767f250 0x5652f7681640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.375+0000 7f8310a00640 1 -- 172.21.15.43:0/2020353910 learned_addr learned my addr 172.21.15.43:0/2020353910 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:50.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.375+0000 7f830be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5652f7675ee0 0x5652f7676360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:56318/0 (socket says 172.21.15.43:56318) 2024-03-28T22:39:50.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.375+0000 7f8310a00640 1 -- 172.21.15.43:0/2020353910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5652f7675ee0 msgr2=0x5652f7676360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:50.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.375+0000 7f8310a00640 1 --2- 172.21.15.43:0/2020353910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5652f7675ee0 0x5652f7676360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.375+0000 7f8310a00640 1 -- 172.21.15.43:0/2020353910 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5652f767f250 msgr2=0x5652f7681640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:50.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.375+0000 7f8310a00640 1 --2- 172.21.15.43:0/2020353910 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5652f767f250 0x5652f7681640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.375+0000 7f8310a00640 1 -- 172.21.15.43:0/2020353910 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652f75b31a0 con 0x5652f7676bc0 2024-03-28T22:39:50.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.376+0000 7f8309600640 1 --2- 172.21.15.43:0/3246199534 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f82f00a1ef0 0x7f82f00a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.376+0000 7f8310a00640 1 --2- 172.21.15.43:0/2020353910 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5652f7676bc0 0x5652f7675910 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8304009f10 tx=0x7f830400f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14094fce0e87d1c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:50.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.376+0000 7f8309600640 1 -- 172.21.15.43:0/3246199534 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7f82f40f00f0 con 0x7f830c103300 2024-03-28T22:39:50.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.376+0000 7f8301200640 1 -- 172.21.15.43:0/2020353910 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8304003b70 con 0x5652f7676bc0 2024-03-28T22:39:50.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.376+0000 7f830be00640 1 --2- 172.21.15.43:0/3246199534 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f82f00a1ef0 0x7f82f00a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.376+0000 7f8301200640 1 -- 172.21.15.43:0/2020353910 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8304003d10 con 0x5652f7676bc0 2024-03-28T22:39:50.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.376+0000 7f8301200640 1 -- 172.21.15.43:0/2020353910 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8304011a70 con 0x5652f7676bc0 2024-03-28T22:39:50.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.377+0000 7f83157aa740 1 -- 172.21.15.43:0/2020353910 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5652f7676bc0 msgr2=0x5652f7675910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:50.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.377+0000 7f83157aa740 1 --2- 172.21.15.43:0/2020353910 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5652f7676bc0 0x5652f7675910 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8304009f10 tx=0x7f830400f730 comp rx=0 tx=0).stop 2024-03-28T22:39:50.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.377+0000 7f830be00640 1 --2- 172.21.15.43:0/3246199534 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f82f00a1ef0 0x7f82f00a43b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x5652f7677110 tx=0x7f82fc005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:50.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.377+0000 7f83157aa740 1 -- 172.21.15.43:0/2020353910 shutdown_connections 2024-03-28T22:39:50.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.377+0000 7f83157aa740 1 --2- 172.21.15.43:0/2020353910 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5652f767f250 0x5652f7681640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.377+0000 7f83157aa740 1 --2- 172.21.15.43:0/2020353910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5652f7675ee0 0x5652f7676360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.377+0000 7f83157aa740 1 --2- 172.21.15.43:0/2020353910 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5652f7676bc0 0x5652f7675910 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.377+0000 7f83157aa740 1 -- 172.21.15.43:0/2020353910 >> 172.21.15.43:0/2020353910 conn(0x5652f763ca10 msgr2=0x5652f7670930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:50.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.377+0000 7f83157aa740 1 -- 172.21.15.43:0/2020353910 shutdown_connections 2024-03-28T22:39:50.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.378+0000 7f83157aa740 1 -- 172.21.15.43:0/2020353910 wait complete. 2024-03-28T22:39:50.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.378+0000 7f83157aa740 1 -- start start 2024-03-28T22:39:50.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.378+0000 7f83157aa740 1 Processor -- start 2024-03-28T22:39:50.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.379+0000 7f83157aa740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5652f7675ee0 0x5652f767c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.379+0000 7f83157aa740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5652f7676bc0 0x5652f767c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.379+0000 7f83157aa740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5652f767f250 0x5652f767cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.379+0000 7f83157aa740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5652f76820c0 con 0x5652f7675ee0 2024-03-28T22:39:50.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.379+0000 7f83157aa740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5652f7578250 con 0x5652f767f250 2024-03-28T22:39:50.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.379+0000 7f83157aa740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5652f7682240 con 0x5652f7676bc0 2024-03-28T22:39:50.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.379+0000 7f83157aa740 10 client.0 _ll_register_callbacks cb 0x5652f72edbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:50.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.379+0000 7f8310a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5652f767f250 0x5652f767cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.379+0000 7f830be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5652f7675ee0 0x5652f767c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.379+0000 7f8303400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5652f7676bc0 0x5652f767c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.379+0000 7f8310a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5652f767f250 0x5652f767cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52926/0 (socket says 172.21.15.43:52926) 2024-03-28T22:39:50.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.379+0000 7f8310a00640 1 -- 172.21.15.43:0/3553573943 learned_addr learned my addr 172.21.15.43:0/3553573943 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:50.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.380+0000 7f8310a00640 1 -- 172.21.15.43:0/3553573943 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5652f7676bc0 msgr2=0x5652f767c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:50.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.380+0000 7f8310a00640 1 --2- 172.21.15.43:0/3553573943 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5652f7676bc0 0x5652f767c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.380+0000 7f8310a00640 1 -- 172.21.15.43:0/3553573943 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5652f7675ee0 msgr2=0x5652f767c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:50.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.380+0000 7f8310a00640 1 --2- 172.21.15.43:0/3553573943 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5652f7675ee0 0x5652f767c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.380+0000 7f8310a00640 1 -- 172.21.15.43:0/3553573943 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652f7594960 con 0x5652f767f250 2024-03-28T22:39:50.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.380+0000 7f8303400640 1 --2- 172.21.15.43:0/3553573943 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5652f7676bc0 0x5652f767c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:50.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.380+0000 7f8310a00640 1 --2- 172.21.15.43:0/3553573943 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5652f767f250 0x5652f767cd20 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f8304000f80 tx=0x7f8304009cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c908fe43b6df1b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:50.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.380+0000 7f82f9600640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:39:50.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.380+0000 7f82f9600640 1 -- 172.21.15.43:0/3553573943 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8304004a40 con 0x5652f767f250 2024-03-28T22:39:50.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.380+0000 7f83157aa740 1 -- 172.21.15.43:0/3553573943 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8304003d10 con 0x5652f767f250 2024-03-28T22:39:50.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.380+0000 7f82f9600640 1 -- 172.21.15.43:0/3553573943 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7f8304004be0 con 0x5652f767f250 2024-03-28T22:39:50.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.381+0000 7f82f9600640 1 -- 172.21.15.43:0/3553573943 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8304030050 con 0x5652f767f250 2024-03-28T22:39:50.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.381+0000 7f83157aa740 10 client.26669 fetch_fsmap learned FSMap version 30 2024-03-28T22:39:50.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.381+0000 7f83157aa740 1 -- 172.21.15.43:0/3553573943 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5652f73b87d0 con 0x5652f767f250 2024-03-28T22:39:50.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.381+0000 7f82f9600640 1 -- 172.21.15.43:0/3553573943 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8304003d10 con 0x5652f767f250 2024-03-28T22:39:50.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.381+0000 7f82f9600640 1 -- 172.21.15.43:0/3553573943 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f8304004080 con 0x5652f767f250 2024-03-28T22:39:50.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.381+0000 7f83157aa740 10 client.26669 fetch_fsmap finished waiting for FSMap version 30 2024-03-28T22:39:50.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.381+0000 7f83157aa740 10 client.26669 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:39:50.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.381+0000 7f83157aa740 20 client.26669 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:50.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.381+0000 7f83157aa740 1 --2- 172.21.15.43:0/3553573943 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5652f768f640 0x5652f7691a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.381+0000 7f83157aa740 4 client.26669 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:39:50.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.381+0000 7f83157aa740 1 -- 172.21.15.43:0/3553573943 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5652f75e7640 con 0x5652f768f640 2024-03-28T22:39:50.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.382+0000 7f830be00640 1 --2- 172.21.15.43:0/3553573943 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5652f768f640 0x5652f7691a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.383+0000 7f830be00640 1 --2- 172.21.15.43:0/3553573943 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5652f768f640 0x5652f7691a30 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1ac7e110062dd89 server_cookie=33e761a2e7d95aa8 in_seq=0 out_seq=0 2024-03-28T22:39:50.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.383+0000 7f82f9600640 10 client.26669 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:50.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.384+0000 7f82f9600640 1 -- 172.21.15.43:0/3553573943 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5652f75e7640 con 0x5652f768f640 2024-03-28T22:39:50.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.384+0000 7f82f9600640 10 client.26669 handle_command_reply: tid=0 2024-03-28T22:39:50.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.384+0000 7f83157aa740 1 client.26669 shutdown 2024-03-28T22:39:50.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.384+0000 7f83157aa740 1 -- 172.21.15.43:0/3553573943 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5652f767f250 msgr2=0x5652f767cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:50.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.384+0000 7f83157aa740 1 --2- 172.21.15.43:0/3553573943 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5652f767f250 0x5652f767cd20 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f8304000f80 tx=0x7f8304009cd0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.384+0000 7f83157aa740 1 -- 172.21.15.43:0/3553573943 shutdown_connections 2024-03-28T22:39:50.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.384+0000 7f83157aa740 1 --2- 172.21.15.43:0/3553573943 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5652f768f640 0x5652f7691a30 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.384+0000 7f83157aa740 1 --2- 172.21.15.43:0/3553573943 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5652f767f250 0x5652f767cd20 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.384+0000 7f83157aa740 1 --2- 172.21.15.43:0/3553573943 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5652f7676bc0 0x5652f767c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.384+0000 7f83157aa740 1 --2- 172.21.15.43:0/3553573943 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5652f7675ee0 0x5652f767c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.384+0000 7f82f9600640 0 client.26669 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:50.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.384+0000 7f83157aa740 1 -- 172.21.15.43:0/3553573943 >> 172.21.15.43:0/3553573943 conn(0x5652f763ca10 msgr2=0x5652f7672680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:50.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.385+0000 7f83157aa740 1 -- 172.21.15.43:0/3553573943 shutdown_connections 2024-03-28T22:39:50.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.385+0000 7f83157aa740 1 -- 172.21.15.43:0/3553573943 wait complete. 2024-03-28T22:39:50.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.385+0000 7f83157aa740 20 client.26669 trim_cache size 0 max 16384 2024-03-28T22:39:50.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.408+0000 7f83157aa740 1 Processor -- start 2024-03-28T22:39:50.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.408+0000 7f83157aa740 1 -- start start 2024-03-28T22:39:50.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.409+0000 7f83157aa740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5652f7675ee0 0x5652f767e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.409+0000 7f83157aa740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5652f7676bc0 0x5652f768aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.409+0000 7f83157aa740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5652f767f250 0x5652f7697030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.409+0000 7f83157aa740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f830c1b0ea0 con 0x5652f7675ee0 2024-03-28T22:39:50.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.409+0000 7f83157aa740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f830c1b29e0 con 0x5652f767f250 2024-03-28T22:39:50.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.409+0000 7f83157aa740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f830c1b1570 con 0x5652f7676bc0 2024-03-28T22:39:50.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.409+0000 7f8303400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5652f7676bc0 0x5652f768aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.409+0000 7f8310a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5652f767f250 0x5652f7697030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.409+0000 7f8303400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5652f7676bc0 0x5652f768aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:56336/0 (socket says 172.21.15.43:56336) 2024-03-28T22:39:50.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.409+0000 7f8303400640 1 -- 172.21.15.43:0/4032058775 learned_addr learned my addr 172.21.15.43:0/4032058775 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:50.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.409+0000 7f830be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5652f7675ee0 0x5652f767e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.409+0000 7f8303400640 1 -- 172.21.15.43:0/4032058775 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5652f767f250 msgr2=0x5652f7697030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:50.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.409+0000 7f8303400640 1 --2- 172.21.15.43:0/4032058775 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5652f767f250 0x5652f7697030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.409+0000 7f8303400640 1 -- 172.21.15.43:0/4032058775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5652f7675ee0 msgr2=0x5652f767e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:50.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.409+0000 7f8303400640 1 --2- 172.21.15.43:0/4032058775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5652f7675ee0 0x5652f767e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.409+0000 7f8303400640 1 -- 172.21.15.43:0/4032058775 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652f72ee3f0 con 0x5652f7676bc0 2024-03-28T22:39:50.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.409+0000 7f8310a00640 1 --2- 172.21.15.43:0/4032058775 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5652f767f250 0x5652f7697030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:50.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.410+0000 7f8303400640 1 --2- 172.21.15.43:0/4032058775 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5652f7676bc0 0x5652f768aea0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f82f40ed5b0 tx=0x7f82f40dc040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec4e188c452680fe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:50.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.410+0000 7f82fbe00640 1 -- 172.21.15.43:0/4032058775 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f40bc5c0 con 0x5652f7676bc0 2024-03-28T22:39:50.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.410+0000 7f82fbe00640 1 -- 172.21.15.43:0/4032058775 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82f40bcef0 con 0x5652f7676bc0 2024-03-28T22:39:50.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.410+0000 7f82fbe00640 1 -- 172.21.15.43:0/4032058775 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f40bc150 con 0x5652f7676bc0 2024-03-28T22:39:50.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.411+0000 7f83157aa740 1 -- 172.21.15.43:0/4032058775 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5652f7676bc0 msgr2=0x5652f768aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:50.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.411+0000 7f83157aa740 1 --2- 172.21.15.43:0/4032058775 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5652f7676bc0 0x5652f768aea0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f82f40ed5b0 tx=0x7f82f40dc040 comp rx=0 tx=0).stop 2024-03-28T22:39:50.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.411+0000 7f83157aa740 1 -- 172.21.15.43:0/4032058775 shutdown_connections 2024-03-28T22:39:50.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.411+0000 7f83157aa740 1 --2- 172.21.15.43:0/4032058775 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5652f767f250 0x5652f7697030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.411+0000 7f83157aa740 1 --2- 172.21.15.43:0/4032058775 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5652f7676bc0 0x5652f768aea0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.411+0000 7f83157aa740 1 --2- 172.21.15.43:0/4032058775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5652f7675ee0 0x5652f767e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.411+0000 7f83157aa740 1 -- 172.21.15.43:0/4032058775 >> 172.21.15.43:0/4032058775 conn(0x5652f763ca10 msgr2=0x5652f7672610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:50.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.412+0000 7f83157aa740 1 -- 172.21.15.43:0/4032058775 shutdown_connections 2024-03-28T22:39:50.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.412+0000 7f83157aa740 1 -- 172.21.15.43:0/4032058775 wait complete. 2024-03-28T22:39:50.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.412+0000 7f83157aa740 1 -- start start 2024-03-28T22:39:50.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.412+0000 7f83157aa740 1 Processor -- start 2024-03-28T22:39:50.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.413+0000 7f83157aa740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5652f7675ee0 0x5652f767fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.413+0000 7f83157aa740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5652f7676bc0 0x5652f7680170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.413+0000 7f83157aa740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5652f767f250 0x5652f76864d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.413+0000 7f83157aa740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f830c1b2310 con 0x5652f7675ee0 2024-03-28T22:39:50.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.413+0000 7f83157aa740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f830c1b1c40 con 0x5652f7676bc0 2024-03-28T22:39:50.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.413+0000 7f83157aa740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f830c1aee80 con 0x5652f767f250 2024-03-28T22:39:50.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.413+0000 7f83157aa740 10 client.0 _ll_register_callbacks cb 0x7f8304015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:50.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.413+0000 7f8303400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5652f7676bc0 0x5652f7680170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.413+0000 7f8303400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5652f7676bc0 0x5652f7680170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52954/0 (socket says 172.21.15.43:52954) 2024-03-28T22:39:50.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.413+0000 7f8303400640 1 -- 172.21.15.43:0/1393151320 learned_addr learned my addr 172.21.15.43:0/1393151320 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:50.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.413+0000 7f830be00640 1 --2- 172.21.15.43:0/1393151320 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5652f7675ee0 0x5652f767fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.414+0000 7f8310a00640 1 --2- 172.21.15.43:0/1393151320 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5652f767f250 0x5652f76864d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.414+0000 7f8303400640 1 -- 172.21.15.43:0/1393151320 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5652f767f250 msgr2=0x5652f76864d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:50.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.414+0000 7f8303400640 1 --2- 172.21.15.43:0/1393151320 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5652f767f250 0x5652f76864d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.414+0000 7f8303400640 1 -- 172.21.15.43:0/1393151320 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5652f7675ee0 msgr2=0x5652f767fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:50.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.414+0000 7f8303400640 1 --2- 172.21.15.43:0/1393151320 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5652f7675ee0 0x5652f767fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.414+0000 7f8303400640 1 -- 172.21.15.43:0/1393151320 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652f72ae0f0 con 0x5652f7676bc0 2024-03-28T22:39:50.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.414+0000 7f8303400640 1 --2- 172.21.15.43:0/1393151320 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5652f7676bc0 0x5652f7680170 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f82f40bc780 tx=0x7f82f402bcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf15d595089e60fa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:50.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.414+0000 7f82f9600640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:39:50.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.415+0000 7f82f9600640 1 -- 172.21.15.43:0/1393151320 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f40bc330 con 0x5652f7676bc0 2024-03-28T22:39:50.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.415+0000 7f82f9600640 1 -- 172.21.15.43:0/1393151320 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7f82f40a70b0 con 0x5652f7676bc0 2024-03-28T22:39:50.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.415+0000 7f83157aa740 1 -- 172.21.15.43:0/1393151320 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82f40bcef0 con 0x5652f7676bc0 2024-03-28T22:39:50.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.415+0000 7f82f9600640 1 -- 172.21.15.43:0/1393151320 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f82f40bc7f0 con 0x5652f7676bc0 2024-03-28T22:39:50.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.415+0000 7f83157aa740 10 client.26675 fetch_fsmap learned FSMap version 30 2024-03-28T22:39:50.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.415+0000 7f83157aa740 1 -- 172.21.15.43:0/1393151320 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5652f7687b90 con 0x5652f7676bc0 2024-03-28T22:39:50.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.416+0000 7f82f9600640 1 -- 172.21.15.43:0/1393151320 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82f40bcef0 con 0x5652f7676bc0 2024-03-28T22:39:50.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.416+0000 7f82f9600640 1 -- 172.21.15.43:0/1393151320 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f82f40ed6c0 con 0x5652f7676bc0 2024-03-28T22:39:50.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.416+0000 7f83157aa740 10 client.26675 fetch_fsmap finished waiting for FSMap version 30 2024-03-28T22:39:50.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.416+0000 7f83157aa740 10 client.26675 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:39:50.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.416+0000 7f83157aa740 20 client.26675 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:50.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.416+0000 7f83157aa740 1 --2- 172.21.15.43:0/1393151320 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5652f76fe260 0x5652f7700680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:50.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.416+0000 7f83157aa740 4 client.26675 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:39:50.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.416+0000 7f83157aa740 1 -- 172.21.15.43:0/1393151320 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5652f75e1990 con 0x5652f76fe260 2024-03-28T22:39:50.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.416+0000 7f830be00640 1 --2- 172.21.15.43:0/1393151320 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5652f76fe260 0x5652f7700680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:50.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.417+0000 7f830be00640 1 --2- 172.21.15.43:0/1393151320 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5652f76fe260 0x5652f7700680 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a443184ad0d5dab7 server_cookie=e82ff81ca9d80e0a in_seq=0 out_seq=0 2024-03-28T22:39:50.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.417+0000 7f82f9600640 10 client.26675 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:50.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.419+0000 7f82f9600640 1 -- 172.21.15.43:0/1393151320 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5652f75e1990 con 0x5652f76fe260 2024-03-28T22:39:50.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.419+0000 7f82f9600640 10 client.26675 handle_command_reply: tid=0 2024-03-28T22:39:50.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.419+0000 7f83157aa740 1 client.26675 shutdown 2024-03-28T22:39:50.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.419+0000 7f83157aa740 1 -- 172.21.15.43:0/1393151320 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5652f7676bc0 msgr2=0x5652f7680170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:50.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.419+0000 7f83157aa740 1 --2- 172.21.15.43:0/1393151320 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5652f7676bc0 0x5652f7680170 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f82f40bc780 tx=0x7f82f402bcc0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.419+0000 7f83157aa740 1 -- 172.21.15.43:0/1393151320 shutdown_connections 2024-03-28T22:39:50.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.419+0000 7f83157aa740 1 --2- 172.21.15.43:0/1393151320 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5652f76fe260 0x5652f7700680 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.419+0000 7f83157aa740 1 --2- 172.21.15.43:0/1393151320 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5652f7676bc0 0x5652f7680170 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.419+0000 7f83157aa740 1 --2- 172.21.15.43:0/1393151320 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5652f767f250 0x5652f76864d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.419+0000 7f83157aa740 1 --2- 172.21.15.43:0/1393151320 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5652f7675ee0 0x5652f767fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.419+0000 7f82f9600640 0 client.26675 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:50.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.419+0000 7f83157aa740 1 -- 172.21.15.43:0/1393151320 >> 172.21.15.43:0/1393151320 conn(0x5652f763ca10 msgr2=0x5652f766f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:50.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.419+0000 7f83157aa740 1 -- 172.21.15.43:0/1393151320 shutdown_connections 2024-03-28T22:39:50.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.419+0000 7f83157aa740 1 -- 172.21.15.43:0/1393151320 wait complete. 2024-03-28T22:39:50.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.420+0000 7f83157aa740 20 client.26675 trim_cache size 0 max 16384 2024-03-28T22:39:50.420 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:39:50.420 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:39:50.420 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-03-28T22:39:50.420 INFO:teuthology.orchestra.run.smithi043.stdout: "1bd6dd92-11fd-4621-9932-739815bf9cdd": { 2024-03-28T22:39:50.420 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-03-28T22:39:50.420 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "1bd6dd92-11fd-4621-9932-739815bf9cdd", 2024-03-28T22:39:50.420 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-03-28T22:39:50.420 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:39:50.420 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:39:50.420 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:39:50.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.420+0000 7f8312a00640 1 -- 172.21.15.43:0/3246199534 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f82f00a1ef0 msgr2=0x7f82f00a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:50.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.420+0000 7f8312a00640 1 --2- 172.21.15.43:0/3246199534 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f82f00a1ef0 0x7f82f00a43b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x5652f7677110 tx=0x7f82fc005f90 comp rx=0 tx=0).stop 2024-03-28T22:39:50.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.420+0000 7f8312a00640 1 -- 172.21.15.43:0/3246199534 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f830c103300 msgr2=0x7f830c19a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:50.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.420+0000 7f8312a00640 1 --2- 172.21.15.43:0/3246199534 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f830c103300 0x7f830c19a810 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f82f4000c00 tx=0x7f82f4004360 comp rx=0 tx=0).stop 2024-03-28T22:39:50.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.420+0000 7f8312a00640 1 -- 172.21.15.43:0/3246199534 shutdown_connections 2024-03-28T22:39:50.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.420+0000 7f8312a00640 1 --2- 172.21.15.43:0/3246199534 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f82f00a1ef0 0x7f82f00a43b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.421+0000 7f8312a00640 1 --2- 172.21.15.43:0/3246199534 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f830c10bdb0 0x7f830c19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.421+0000 7f8312a00640 1 --2- 172.21.15.43:0/3246199534 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f830c103300 0x7f830c19a810 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.421+0000 7f8312a00640 1 --2- 172.21.15.43:0/3246199534 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f830c1029e0 0x7f830c19a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:50.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.421+0000 7f8312a00640 1 -- 172.21.15.43:0/3246199534 >> 172.21.15.43:0/3246199534 conn(0x7f830c0777b0 msgr2=0x7f830c107c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:50.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.421+0000 7f8312a00640 1 -- 172.21.15.43:0/3246199534 shutdown_connections 2024-03-28T22:39:50.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:39:50.421+0000 7f8312a00640 1 -- 172.21.15.43:0/3246199534 wait complete. 2024-03-28T22:39:50.440 INFO:tasks.cephfs.filesystem:scrub status for tag:1bd6dd92-11fd-4621-9932-739815bf9cdd - {'path': '/', 'tag': '1bd6dd92-11fd-4621-9932-739815bf9cdd', 'options': 'recursive,force'} 2024-03-28T22:39:50.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:50 smithi172 ceph-mon[36566]: pgmap v83: 129 pgs: 129 active+clean; 41 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 276 op/s 2024-03-28T22:39:50.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:50 smithi172 ceph-mon[36566]: scrub queued for path: / 2024-03-28T22:39:50.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:50 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:39:50.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:50 smithi172 ceph-mon[36566]: scrub summary: active paths [/] 2024-03-28T22:39:50.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:50 smithi082 ceph-mon[36471]: pgmap v83: 129 pgs: 129 active+clean; 41 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 276 op/s 2024-03-28T22:39:50.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:50 smithi082 ceph-mon[36471]: scrub queued for path: / 2024-03-28T22:39:50.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:50 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:39:50.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:50 smithi082 ceph-mon[36471]: scrub summary: active paths [/] 2024-03-28T22:39:51.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:50 smithi043 ceph-mon[31823]: pgmap v83: 129 pgs: 129 active+clean; 41 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 276 op/s 2024-03-28T22:39:51.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:50 smithi043 ceph-mon[31823]: scrub queued for path: / 2024-03-28T22:39:51.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:50 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:39:51.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:50 smithi043 ceph-mon[31823]: scrub summary: active paths [/] 2024-03-28T22:39:51.348 INFO:tasks.workunit.client.0.smithi043.stdout: 1 131256 8.86 MB/sec warmup 82 sec latency 38.424 ms 2024-03-28T22:39:51.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:51 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:39:51.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:51 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:39:52.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:51 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:39:52.348 INFO:tasks.workunit.client.0.smithi043.stdout: 1 132189 8.79 MB/sec warmup 83 sec latency 40.408 ms 2024-03-28T22:39:52.849 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:52 smithi082 ceph-mon[36471]: pgmap v84: 129 pgs: 129 active+clean; 41 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 276 op/s 2024-03-28T22:39:52.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:52 smithi172 ceph-mon[36566]: pgmap v84: 129 pgs: 129 active+clean; 41 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 276 op/s 2024-03-28T22:39:53.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:52 smithi043 ceph-mon[31823]: pgmap v84: 129 pgs: 129 active+clean; 41 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 276 op/s 2024-03-28T22:39:53.348 INFO:tasks.workunit.client.0.smithi043.stdout: 1 134416 8.83 MB/sec warmup 84 sec latency 12.044 ms 2024-03-28T22:39:53.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:53 smithi172 ceph-mon[36566]: scrub summary: idle 2024-03-28T22:39:53.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:53 smithi082 ceph-mon[36471]: scrub summary: idle 2024-03-28T22:39:53.953 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.955+0000 7fb1ae800640 1 -- 172.21.15.43:0/2996262233 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1a80765a0 msgr2=0x7fb1a8112080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:53.953 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.955+0000 7fb1ae800640 1 --2- 172.21.15.43:0/2996262233 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1a80765a0 0x7fb1a8112080 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb194009930 tx=0x7fb19402bc30 comp rx=0 tx=0).stop 2024-03-28T22:39:53.953 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.955+0000 7fb1ae800640 1 -- 172.21.15.43:0/2996262233 shutdown_connections 2024-03-28T22:39:53.953 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.955+0000 7fb1ae800640 1 --2- 172.21.15.43:0/2996262233 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1a81125c0 0x7fb1a81149b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:53.954 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.955+0000 7fb1ae800640 1 --2- 172.21.15.43:0/2996262233 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1a80765a0 0x7fb1a8112080 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:53.954 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.955+0000 7fb1ae800640 1 --2- 172.21.15.43:0/2996262233 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1a8075c80 0x7fb1a8076060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:53.954 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.955+0000 7fb1ae800640 1 -- 172.21.15.43:0/2996262233 >> 172.21.15.43:0/2996262233 conn(0x7fb1a80ff410 msgr2=0x7fb1a8101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:53.954 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.956+0000 7fb1ae800640 1 -- 172.21.15.43:0/2996262233 shutdown_connections 2024-03-28T22:39:53.954 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.956+0000 7fb1ae800640 1 -- 172.21.15.43:0/2996262233 wait complete. 2024-03-28T22:39:53.955 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.957+0000 7fb1ae800640 1 Processor -- start 2024-03-28T22:39:53.955 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.957+0000 7fb1ae800640 1 -- start start 2024-03-28T22:39:53.955 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.957+0000 7fb1ae800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1a8075c80 0x7fb1a819e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:53.955 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.957+0000 7fb1ae800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1a80765a0 0x7fb1a819eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:53.955 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.957+0000 7fb1ae800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1a81125c0 0x7fb1a81a2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:53.955 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.957+0000 7fb1ae800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a8118170 con 0x7fb1a8075c80 2024-03-28T22:39:53.955 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.957+0000 7fb1ae800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a8117ff0 con 0x7fb1a81125c0 2024-03-28T22:39:53.956 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.957+0000 7fb1ae800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a81182f0 con 0x7fb1a80765a0 2024-03-28T22:39:53.956 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.958+0000 7fb1a6a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1a80765a0 0x7fb1a819eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:53.956 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.958+0000 7fb1a7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1a8075c80 0x7fb1a819e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:53.956 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.958+0000 7fb1a6a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1a80765a0 0x7fb1a819eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:56370/0 (socket says 172.21.15.43:56370) 2024-03-28T22:39:53.956 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.958+0000 7fb1a6a00640 1 -- 172.21.15.43:0/3559534612 learned_addr learned my addr 172.21.15.43:0/3559534612 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:53.956 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.958+0000 7fb1a7e00640 1 --2- 172.21.15.43:0/3559534612 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1a81125c0 0x7fb1a81a2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.958+0000 7fb1a6a00640 1 -- 172.21.15.43:0/3559534612 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1a81125c0 msgr2=0x7fb1a81a2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.958+0000 7fb1a6a00640 1 --2- 172.21.15.43:0/3559534612 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1a81125c0 0x7fb1a81a2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.958+0000 7fb1a6a00640 1 -- 172.21.15.43:0/3559534612 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1a8075c80 msgr2=0x7fb1a819e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.958+0000 7fb1a6a00640 1 --2- 172.21.15.43:0/3559534612 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1a8075c80 0x7fb1a819e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.958+0000 7fb1a6a00640 1 -- 172.21.15.43:0/3559534612 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1a81a35a0 con 0x7fb1a80765a0 2024-03-28T22:39:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.958+0000 7fb1a6a00640 1 --2- 172.21.15.43:0/3559534612 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1a80765a0 0x7fb1a819eaf0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb194009900 tx=0x7fb194004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.959+0000 7fb19fe00640 1 -- 172.21.15.43:0/3559534612 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb194043020 con 0x7fb1a80765a0 2024-03-28T22:39:53.958 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.959+0000 7fb19fe00640 1 -- 172.21.15.43:0/3559534612 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1940054e0 con 0x7fb1a80765a0 2024-03-28T22:39:53.958 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.959+0000 7fb1ae800640 1 -- 172.21.15.43:0/3559534612 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1a81a3830 con 0x7fb1a80765a0 2024-03-28T22:39:53.958 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.959+0000 7fb19fe00640 1 -- 172.21.15.43:0/3559534612 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb194003b80 con 0x7fb1a80765a0 2024-03-28T22:39:53.958 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.959+0000 7fb1ae800640 1 -- 172.21.15.43:0/3559534612 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1a81ab120 con 0x7fb1a80765a0 2024-03-28T22:39:53.959 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.962+0000 7fb1b147b740 1 Processor -- start 2024-03-28T22:39:53.960 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.962+0000 7fb1b147b740 1 -- start start 2024-03-28T22:39:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.962+0000 7fb19fe00640 1 -- 172.21.15.43:0/3559534612 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fb194046020 con 0x7fb1a80765a0 2024-03-28T22:39:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.963+0000 7fb1b147b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55816b99d3a0 0x55816b99d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.963+0000 7fb1b147b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55816b99dcc0 0x55816b9a6550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.963+0000 7fb1b147b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55816b9a6a90 0x55816b9a8e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.963+0000 7fb1b147b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55816b900640 con 0x55816b99d3a0 2024-03-28T22:39:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.963+0000 7fb1b147b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55816b8a9810 con 0x55816b99dcc0 2024-03-28T22:39:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.963+0000 7fb1b147b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55816b8fc6e0 con 0x55816b9a6a90 2024-03-28T22:39:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.963+0000 7fb1a7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55816b99d3a0 0x55816b99d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.963+0000 7fb1a7400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55816b99dcc0 0x55816b9a6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.963+0000 7fb1a7400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55816b99dcc0 0x55816b9a6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52990/0 (socket says 172.21.15.43:52990) 2024-03-28T22:39:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.963+0000 7fb1a7400640 1 -- 172.21.15.43:0/1125584118 learned_addr learned my addr 172.21.15.43:0/1125584118 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.963+0000 7fb1a6a00640 1 --2- 172.21.15.43:0/1125584118 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55816b9a6a90 0x55816b9a8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.963+0000 7fb1a7400640 1 -- 172.21.15.43:0/1125584118 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55816b9a6a90 msgr2=0x55816b9a8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.963+0000 7fb1a7400640 1 --2- 172.21.15.43:0/1125584118 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55816b9a6a90 0x55816b9a8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.963+0000 7fb1a7400640 1 -- 172.21.15.43:0/1125584118 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55816b99d3a0 msgr2=0x55816b99d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.963+0000 7fb1a7400640 1 --2- 172.21.15.43:0/1125584118 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55816b99d3a0 0x55816b99d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.963+0000 7fb1a7400640 1 -- 172.21.15.43:0/1125584118 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55816b615210 con 0x55816b99dcc0 2024-03-28T22:39:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.963+0000 7fb1a6a00640 1 --2- 172.21.15.43:0/1125584118 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55816b9a6a90 0x55816b9a8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:39:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.964+0000 7fb1a7400640 1 --2- 172.21.15.43:0/1125584118 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55816b99dcc0 0x55816b9a6550 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb19800bd10 tx=0x7fb1980107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c84e97ab9e57f807 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.964+0000 7fb19cc00640 1 -- 172.21.15.43:0/1125584118 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb198004030 con 0x55816b99dcc0 2024-03-28T22:39:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.964+0000 7fb19cc00640 1 -- 172.21.15.43:0/1125584118 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1980041d0 con 0x55816b99dcc0 2024-03-28T22:39:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.964+0000 7fb19cc00640 1 -- 172.21.15.43:0/1125584118 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb198011840 con 0x55816b99dcc0 2024-03-28T22:39:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.964+0000 7fb19fe00640 1 --2- 172.21.15.43:0/3559534612 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb18c0a2010 0x7fb18c0a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.965+0000 7fb19fe00640 1 -- 172.21.15.43:0/3559534612 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7fb1940f0530 con 0x7fb1a80765a0 2024-03-28T22:39:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.965+0000 7fb1b147b740 1 -- 172.21.15.43:0/1125584118 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55816b99dcc0 msgr2=0x55816b9a6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.965+0000 7fb1b147b740 1 --2- 172.21.15.43:0/1125584118 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55816b99dcc0 0x55816b9a6550 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb19800bd10 tx=0x7fb1980107a0 comp rx=0 tx=0).stop 2024-03-28T22:39:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.965+0000 7fb1a7e00640 1 --2- 172.21.15.43:0/3559534612 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb18c0a2010 0x7fb18c0a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.965+0000 7fb1b147b740 1 -- 172.21.15.43:0/1125584118 shutdown_connections 2024-03-28T22:39:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.965+0000 7fb1b147b740 1 --2- 172.21.15.43:0/1125584118 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55816b99dcc0 0x55816b9a6550 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.965+0000 7fb1b147b740 1 --2- 172.21.15.43:0/1125584118 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55816b9a6a90 0x55816b9a8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.965+0000 7fb1b147b740 1 --2- 172.21.15.43:0/1125584118 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55816b99d3a0 0x55816b99d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.965+0000 7fb1b147b740 1 -- 172.21.15.43:0/1125584118 >> 172.21.15.43:0/1125584118 conn(0x55816b997cc0 msgr2=0x55816b9980a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.965+0000 7fb1b147b740 1 -- 172.21.15.43:0/1125584118 shutdown_connections 2024-03-28T22:39:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.965+0000 7fb1b147b740 1 -- 172.21.15.43:0/1125584118 wait complete. 2024-03-28T22:39:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.965+0000 7fb1a7e00640 1 --2- 172.21.15.43:0/3559534612 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb18c0a2010 0x7fb18c0a44d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb1a806ac00 tx=0x7fb1a0009500 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.965+0000 7fb1b147b740 1 -- start start 2024-03-28T22:39:53.969 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.966+0000 7fb1b147b740 1 Processor -- start 2024-03-28T22:39:53.969 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.966+0000 7fb1b147b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55816b99d3a0 0x55816b9a3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:53.969 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.966+0000 7fb1b147b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55816b99dcc0 0x55816b9a4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:53.969 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.966+0000 7fb1b147b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55816b9a6a90 0x55816b9a45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.966+0000 7fb1b147b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55816b9a96c0 con 0x55816b99d3a0 2024-03-28T22:39:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.966+0000 7fb1b147b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55816b89ed00 con 0x55816b99dcc0 2024-03-28T22:39:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.966+0000 7fb1b147b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55816b9a9840 con 0x55816b9a6a90 2024-03-28T22:39:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.966+0000 7fb1b147b740 10 client.0 _ll_register_callbacks cb 0x55816b889490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.966+0000 7fb1a6a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55816b99dcc0 0x55816b9a4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.966+0000 7fb1a7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55816b99d3a0 0x55816b9a3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.966+0000 7fb1a7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55816b9a6a90 0x55816b9a45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.966+0000 7fb1a6a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55816b99dcc0 0x55816b9a4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:53000/0 (socket says 172.21.15.43:53000) 2024-03-28T22:39:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.966+0000 7fb1a6a00640 1 -- 172.21.15.43:0/775189393 learned_addr learned my addr 172.21.15.43:0/775189393 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.967+0000 7fb1a6a00640 1 -- 172.21.15.43:0/775189393 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55816b9a6a90 msgr2=0x55816b9a45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.967+0000 7fb1a6a00640 1 --2- 172.21.15.43:0/775189393 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55816b9a6a90 0x55816b9a45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.967+0000 7fb1a6a00640 1 -- 172.21.15.43:0/775189393 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55816b99d3a0 msgr2=0x55816b9a3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.967+0000 7fb1a6a00640 1 --2- 172.21.15.43:0/775189393 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55816b99d3a0 0x55816b9a3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.967+0000 7fb1a6a00640 1 -- 172.21.15.43:0/775189393 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55816b6e1620 con 0x55816b99dcc0 2024-03-28T22:39:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.967+0000 7fb1a6a00640 1 --2- 172.21.15.43:0/775189393 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55816b99dcc0 0x55816b9a4080 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fb1940ed9f0 tx=0x7fb1940035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4398aab0a97af474 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.967+0000 7fb191600640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:39:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.967+0000 7fb191600640 1 -- 172.21.15.43:0/775189393 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1940a7040 con 0x55816b99dcc0 2024-03-28T22:39:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.967+0000 7fb191600640 1 -- 172.21.15.43:0/775189393 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7fb1940f0170 con 0x55816b99dcc0 2024-03-28T22:39:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.967+0000 7fb1b147b740 1 -- 172.21.15.43:0/775189393 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1980041d0 con 0x55816b99dcc0 2024-03-28T22:39:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.967+0000 7fb191600640 1 -- 172.21.15.43:0/775189393 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb1940bc770 con 0x55816b99dcc0 2024-03-28T22:39:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.967+0000 7fb1b147b740 10 client.26687 fetch_fsmap learned FSMap version 30 2024-03-28T22:39:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.967+0000 7fb1b147b740 1 -- 172.21.15.43:0/775189393 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55816b9b5620 con 0x55816b99dcc0 2024-03-28T22:39:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.968+0000 7fb191600640 1 -- 172.21.15.43:0/775189393 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1980041d0 con 0x55816b99dcc0 2024-03-28T22:39:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.968+0000 7fb191600640 1 -- 172.21.15.43:0/775189393 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fb1940edb00 con 0x55816b99dcc0 2024-03-28T22:39:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.968+0000 7fb1b147b740 10 client.26687 fetch_fsmap finished waiting for FSMap version 30 2024-03-28T22:39:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.968+0000 7fb1b147b740 10 client.26687 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:39:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.968+0000 7fb1b147b740 20 client.26687 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.968+0000 7fb1b147b740 1 --2- 172.21.15.43:0/775189393 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55816b9b6b00 0x55816b9b8f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.968+0000 7fb1b147b740 4 client.26687 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:39:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.968+0000 7fb1b147b740 1 -- 172.21.15.43:0/775189393 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55816b914030 con 0x55816b9b6b00 2024-03-28T22:39:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.968+0000 7fb1a7400640 1 --2- 172.21.15.43:0/775189393 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55816b9b6b00 0x55816b9b8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.969+0000 7fb1a7400640 1 --2- 172.21.15.43:0/775189393 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55816b9b6b00 0x55816b9b8f40 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=edce1e8437362438 server_cookie=9b1f5506df9393af in_seq=0 out_seq=0 2024-03-28T22:39:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.969+0000 7fb191600640 10 client.26687 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.971+0000 7fb191600640 1 -- 172.21.15.43:0/775189393 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55816b914030 con 0x55816b9b6b00 2024-03-28T22:39:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.971+0000 7fb191600640 10 client.26687 handle_command_reply: tid=0 2024-03-28T22:39:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.971+0000 7fb1b147b740 1 client.26687 shutdown 2024-03-28T22:39:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.971+0000 7fb1b147b740 1 -- 172.21.15.43:0/775189393 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55816b99dcc0 msgr2=0x55816b9a4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.971+0000 7fb1b147b740 1 --2- 172.21.15.43:0/775189393 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55816b99dcc0 0x55816b9a4080 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fb1940ed9f0 tx=0x7fb1940035f0 comp rx=0 tx=0).stop 2024-03-28T22:39:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.971+0000 7fb1b147b740 1 -- 172.21.15.43:0/775189393 shutdown_connections 2024-03-28T22:39:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.971+0000 7fb1b147b740 1 --2- 172.21.15.43:0/775189393 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55816b9b6b00 0x55816b9b8f40 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.971+0000 7fb1b147b740 1 --2- 172.21.15.43:0/775189393 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55816b99dcc0 0x55816b9a4080 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.971+0000 7fb1b147b740 1 --2- 172.21.15.43:0/775189393 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55816b9a6a90 0x55816b9a45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.971+0000 7fb1b147b740 1 --2- 172.21.15.43:0/775189393 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55816b99d3a0 0x55816b9a3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.971+0000 7fb191600640 0 client.26687 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.971+0000 7fb1b147b740 1 -- 172.21.15.43:0/775189393 >> 172.21.15.43:0/775189393 conn(0x55816b997cc0 msgr2=0x55816b999a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.971+0000 7fb1b147b740 1 -- 172.21.15.43:0/775189393 shutdown_connections 2024-03-28T22:39:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.971+0000 7fb1b147b740 1 -- 172.21.15.43:0/775189393 wait complete. 2024-03-28T22:39:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.972+0000 7fb1b147b740 20 client.26687 trim_cache size 0 max 16384 2024-03-28T22:39:53.988 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.990+0000 7fb1b147b740 1 Processor -- start 2024-03-28T22:39:53.988 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.990+0000 7fb1b147b740 1 -- start start 2024-03-28T22:39:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.990+0000 7fb1b147b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55816b99d3a0 0x55816b9a6270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.990+0000 7fb1b147b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55816b99dcc0 0x55816b9b2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.990+0000 7fb1b147b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55816b9a6a90 0x55816b996ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.990+0000 7fb1b147b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a81b5200 con 0x55816b99d3a0 2024-03-28T22:39:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.990+0000 7fb1b147b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a81b6d40 con 0x55816b99dcc0 2024-03-28T22:39:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.990+0000 7fb1b147b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a81b58d0 con 0x55816b9a6a90 2024-03-28T22:39:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.991+0000 7fb1a6a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55816b99dcc0 0x55816b9b2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.991+0000 7fb1a7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55816b99d3a0 0x55816b9a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.991+0000 7fb1a6a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55816b99dcc0 0x55816b9b2460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:53004/0 (socket says 172.21.15.43:53004) 2024-03-28T22:39:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.991+0000 7fb1a6a00640 1 -- 172.21.15.43:0/1130232031 learned_addr learned my addr 172.21.15.43:0/1130232031 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.991+0000 7fb1a7e00640 1 --2- 172.21.15.43:0/1130232031 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55816b9a6a90 0x55816b996ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.991+0000 7fb1a6a00640 1 -- 172.21.15.43:0/1130232031 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55816b9a6a90 msgr2=0x55816b996ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.991+0000 7fb1a6a00640 1 --2- 172.21.15.43:0/1130232031 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55816b9a6a90 0x55816b996ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:53.989 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.991+0000 7fb1a6a00640 1 -- 172.21.15.43:0/1130232031 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55816b99d3a0 msgr2=0x55816b9a6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:53.990 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.991+0000 7fb1a6a00640 1 --2- 172.21.15.43:0/1130232031 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55816b99d3a0 0x55816b9a6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:53.990 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.991+0000 7fb1a6a00640 1 -- 172.21.15.43:0/1130232031 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55816b9a95a0 con 0x55816b99dcc0 2024-03-28T22:39:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.991+0000 7fb1a6a00640 1 --2- 172.21.15.43:0/1130232031 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55816b99dcc0 0x55816b9b2460 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fb1940bcbc0 tx=0x7fb194009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e59d6075e7ddc020 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.991+0000 7fb193e00640 1 -- 172.21.15.43:0/1130232031 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1940da070 con 0x55816b99dcc0 2024-03-28T22:39:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.992+0000 7fb193e00640 1 -- 172.21.15.43:0/1130232031 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1940d4070 con 0x55816b99dcc0 2024-03-28T22:39:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.992+0000 7fb193e00640 1 -- 172.21.15.43:0/1130232031 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1940a7040 con 0x55816b99dcc0 2024-03-28T22:39:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.992+0000 7fb1b147b740 1 -- 172.21.15.43:0/1130232031 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55816b99dcc0 msgr2=0x55816b9b2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.992+0000 7fb1b147b740 1 --2- 172.21.15.43:0/1130232031 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55816b99dcc0 0x55816b9b2460 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fb1940bcbc0 tx=0x7fb194009260 comp rx=0 tx=0).stop 2024-03-28T22:39:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.992+0000 7fb1b147b740 1 -- 172.21.15.43:0/1130232031 shutdown_connections 2024-03-28T22:39:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.992+0000 7fb1b147b740 1 --2- 172.21.15.43:0/1130232031 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55816b99dcc0 0x55816b9b2460 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.992+0000 7fb1b147b740 1 --2- 172.21.15.43:0/1130232031 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55816b9a6a90 0x55816b996ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.992+0000 7fb1b147b740 1 --2- 172.21.15.43:0/1130232031 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55816b99d3a0 0x55816b9a6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.992+0000 7fb1b147b740 1 -- 172.21.15.43:0/1130232031 >> 172.21.15.43:0/1130232031 conn(0x55816b997cc0 msgr2=0x55816b9963d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.992+0000 7fb1b147b740 1 -- 172.21.15.43:0/1130232031 shutdown_connections 2024-03-28T22:39:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.992+0000 7fb1b147b740 1 -- 172.21.15.43:0/1130232031 wait complete. 2024-03-28T22:39:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.993+0000 7fb1b147b740 1 -- start start 2024-03-28T22:39:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.993+0000 7fb1b147b740 1 Processor -- start 2024-03-28T22:39:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.993+0000 7fb1b147b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55816b99d3a0 0x55816b9aa370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.993+0000 7fb1b147b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55816b99dcc0 0x55816b9aa8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.993+0000 7fb1b147b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55816b9a6a90 0x55816b9aadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.993+0000 7fb1b147b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a81b6670 con 0x55816b99d3a0 2024-03-28T22:39:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.994+0000 7fb1b147b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a81b5fa0 con 0x55816b9a6a90 2024-03-28T22:39:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.994+0000 7fb1b147b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a81b31e0 con 0x55816b99dcc0 2024-03-28T22:39:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.994+0000 7fb1b147b740 10 client.0 _ll_register_callbacks cb 0x7fb1940f3ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:39:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.994+0000 7fb1a7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55816b99d3a0 0x55816b9aa370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.994+0000 7fb1a7e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55816b9a6a90 0x55816b9aadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.994+0000 7fb1a6a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55816b99dcc0 0x55816b9aa8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.994+0000 7fb1a6a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55816b99dcc0 0x55816b9aa8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:56410/0 (socket says 172.21.15.43:56410) 2024-03-28T22:39:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.994+0000 7fb1a6a00640 1 -- 172.21.15.43:0/1557099015 learned_addr learned my addr 172.21.15.43:0/1557099015 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:39:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.994+0000 7fb1a7e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55816b9a6a90 0x55816b9aadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:53012/0 (socket says 172.21.15.43:53012) 2024-03-28T22:39:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.994+0000 7fb1a6a00640 1 -- 172.21.15.43:0/1557099015 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55816b9a6a90 msgr2=0x55816b9aadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.994+0000 7fb1a6a00640 1 --2- 172.21.15.43:0/1557099015 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55816b9a6a90 0x55816b9aadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.994+0000 7fb1a6a00640 1 -- 172.21.15.43:0/1557099015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55816b99d3a0 msgr2=0x55816b9aa370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:39:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.994+0000 7fb1a6a00640 1 --2- 172.21.15.43:0/1557099015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55816b99d3a0 0x55816b9aa370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.994+0000 7fb1a6a00640 1 -- 172.21.15.43:0/1557099015 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55816ba1be80 con 0x55816b99dcc0 2024-03-28T22:39:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.994+0000 7fb1a7400640 1 --2- 172.21.15.43:0/1557099015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55816b99d3a0 0x55816b9aa370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:39:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.994+0000 7fb1a6a00640 1 --2- 172.21.15.43:0/1557099015 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55816b99dcc0 0x55816b9aa8b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb1940b9090 tx=0x7fb1940035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5671a756a57d01c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:39:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.994+0000 7fb191600640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:39:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.995+0000 7fb191600640 1 -- 172.21.15.43:0/1557099015 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1940a1060 con 0x55816b99dcc0 2024-03-28T22:39:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.995+0000 7fb191600640 1 -- 172.21.15.43:0/1557099015 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7fb1940c5140 con 0x55816b99dcc0 2024-03-28T22:39:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.995+0000 7fb1b147b740 1 -- 172.21.15.43:0/1557099015 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1940d4070 con 0x55816b99dcc0 2024-03-28T22:39:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.995+0000 7fb191600640 1 -- 172.21.15.43:0/1557099015 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb1940da070 con 0x55816b99dcc0 2024-03-28T22:39:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.995+0000 7fb1b147b740 10 client.26641 fetch_fsmap learned FSMap version 30 2024-03-28T22:39:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.995+0000 7fb1b147b740 1 -- 172.21.15.43:0/1557099015 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55816ba1be80 con 0x55816b99dcc0 2024-03-28T22:39:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.995+0000 7fb191600640 1 -- 172.21.15.43:0/1557099015 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1940d4070 con 0x55816b99dcc0 2024-03-28T22:39:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.995+0000 7fb191600640 1 -- 172.21.15.43:0/1557099015 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fb1940bcf20 con 0x55816b99dcc0 2024-03-28T22:39:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.995+0000 7fb1b147b740 10 client.26641 fetch_fsmap finished waiting for FSMap version 30 2024-03-28T22:39:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.995+0000 7fb1b147b740 10 client.26641 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:39:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.995+0000 7fb1b147b740 20 client.26641 populate_metadata read hostname 'smithi043' 2024-03-28T22:39:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.995+0000 7fb1b147b740 1 --2- 172.21.15.43:0/1557099015 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55816ba1d4c0 0x55816ba1f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:39:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.995+0000 7fb1b147b740 4 client.26641 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T22:39:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.995+0000 7fb1b147b740 1 -- 172.21.15.43:0/1557099015 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55816b90db80 con 0x55816ba1d4c0 2024-03-28T22:39:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.996+0000 7fb1a7400640 1 --2- 172.21.15.43:0/1557099015 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55816ba1d4c0 0x55816ba1f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:39:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.996+0000 7fb1a7400640 1 --2- 172.21.15.43:0/1557099015 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55816ba1d4c0 0x55816ba1f8e0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6107a83ef7504956 server_cookie=b31c4828bf1c9bc4 in_seq=0 out_seq=0 2024-03-28T22:39:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:53.996+0000 7fb191600640 10 client.26641 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:54.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:53 smithi043 ceph-mon[31823]: scrub summary: idle 2024-03-28T22:39:54.189 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:54.191+0000 7fb191600640 1 -- 172.21.15.43:0/1557099015 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55816b90db80 con 0x55816ba1d4c0 2024-03-28T22:39:54.189 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:54.191+0000 7fb191600640 10 client.26641 handle_command_reply: tid=0 2024-03-28T22:39:54.189 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:54.191+0000 7fb1b147b740 1 client.26641 shutdown 2024-03-28T22:39:54.189 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:54.191+0000 7fb1b147b740 1 -- 172.21.15.43:0/1557099015 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55816b99dcc0 msgr2=0x55816b9aa8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:54.189 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:54.191+0000 7fb1b147b740 1 --2- 172.21.15.43:0/1557099015 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55816b99dcc0 0x55816b9aa8b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb1940b9090 tx=0x7fb1940035f0 comp rx=0 tx=0).stop 2024-03-28T22:39:54.189 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:54.191+0000 7fb1b147b740 1 -- 172.21.15.43:0/1557099015 shutdown_connections 2024-03-28T22:39:54.189 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:54.191+0000 7fb1b147b740 1 --2- 172.21.15.43:0/1557099015 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55816ba1d4c0 0x55816ba1f8e0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:54.189 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:54.191+0000 7fb1b147b740 1 --2- 172.21.15.43:0/1557099015 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55816b9a6a90 0x55816b9aadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:54.190 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:54.191+0000 7fb1b147b740 1 --2- 172.21.15.43:0/1557099015 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55816b99dcc0 0x55816b9aa8b0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:54.190 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:54.191+0000 7fb1b147b740 1 --2- 172.21.15.43:0/1557099015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55816b99d3a0 0x55816b9aa370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:54.190 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:54.191+0000 7fb191600640 0 client.26641 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:39:54.190 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:54.191+0000 7fb1b147b740 1 -- 172.21.15.43:0/1557099015 >> 172.21.15.43:0/1557099015 conn(0x55816b997cc0 msgr2=0x55816b996c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:54.190 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:54.191+0000 7fb1b147b740 1 -- 172.21.15.43:0/1557099015 shutdown_connections 2024-03-28T22:39:54.190 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:54.191+0000 7fb1b147b740 1 -- 172.21.15.43:0/1557099015 wait complete. 2024-03-28T22:39:54.190 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:54.192+0000 7fb1b147b740 20 client.26641 trim_cache size 0 max 16384 2024-03-28T22:39:54.191 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:{ 2024-03-28T22:39:54.191 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "message": "", 2024-03-28T22:39:54.191 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "return_code": 0 2024-03-28T22:39:54.191 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:} 2024-03-28T22:39:54.191 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:54.193+0000 7fb1ae800640 1 -- 172.21.15.43:0/3559534612 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb18c0a2010 msgr2=0x7fb18c0a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:54.192 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:54.193+0000 7fb1ae800640 1 --2- 172.21.15.43:0/3559534612 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb18c0a2010 0x7fb18c0a44d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb1a806ac00 tx=0x7fb1a0009500 comp rx=0 tx=0).stop 2024-03-28T22:39:54.192 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:54.193+0000 7fb1ae800640 1 -- 172.21.15.43:0/3559534612 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1a80765a0 msgr2=0x7fb1a819eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:39:54.192 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:54.193+0000 7fb1ae800640 1 --2- 172.21.15.43:0/3559534612 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1a80765a0 0x7fb1a819eaf0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb194009900 tx=0x7fb194004360 comp rx=0 tx=0).stop 2024-03-28T22:39:54.192 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:54.193+0000 7fb1ae800640 1 -- 172.21.15.43:0/3559534612 shutdown_connections 2024-03-28T22:39:54.192 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:54.193+0000 7fb1ae800640 1 --2- 172.21.15.43:0/3559534612 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb18c0a2010 0x7fb18c0a44d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:54.192 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:54.193+0000 7fb1ae800640 1 --2- 172.21.15.43:0/3559534612 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1a81125c0 0x7fb1a81a2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:54.192 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:54.193+0000 7fb1ae800640 1 --2- 172.21.15.43:0/3559534612 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb1a80765a0 0x7fb1a819eaf0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:54.192 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:54.193+0000 7fb1ae800640 1 --2- 172.21.15.43:0/3559534612 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1a8075c80 0x7fb1a819e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:39:54.192 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:54.193+0000 7fb1ae800640 1 -- 172.21.15.43:0/3559534612 >> 172.21.15.43:0/3559534612 conn(0x7fb1a80ff410 msgr2=0x7fb1a8101800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:39:54.192 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:54.193+0000 7fb1ae800640 1 -- 172.21.15.43:0/3559534612 shutdown_connections 2024-03-28T22:39:54.192 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:39:54.193+0000 7fb1ae800640 1 -- 172.21.15.43:0/3559534612 wait complete. 2024-03-28T22:39:54.348 INFO:tasks.workunit.client.0.smithi043.stdout: 1 135386 8.80 MB/sec warmup 85 sec latency 93.985 ms 2024-03-28T22:39:54.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:54 smithi172 ceph-mon[36566]: pgmap v85: 129 pgs: 129 active+clean; 45 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 376 op/s 2024-03-28T22:39:54.917 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:54 smithi043 ceph-mon[31823]: pgmap v85: 129 pgs: 129 active+clean; 45 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 376 op/s 2024-03-28T22:39:54.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:54 smithi082 ceph-mon[36471]: pgmap v85: 129 pgs: 129 active+clean; 45 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 376 op/s 2024-03-28T22:39:55.348 INFO:tasks.workunit.client.0.smithi043.stdout: 1 137051 8.76 MB/sec warmup 86 sec latency 18.028 ms 2024-03-28T22:39:56.348 INFO:tasks.workunit.client.0.smithi043.stdout: 1 138601 8.81 MB/sec warmup 87 sec latency 25.695 ms 2024-03-28T22:39:56.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:56 smithi172 ceph-mon[36566]: pgmap v86: 129 pgs: 129 active+clean; 45 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 177 op/s 2024-03-28T22:39:56.950 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:56 smithi082 ceph-mon[36471]: pgmap v86: 129 pgs: 129 active+clean; 45 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 177 op/s 2024-03-28T22:39:57.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:56 smithi043 ceph-mon[31823]: pgmap v86: 129 pgs: 129 active+clean; 45 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 177 op/s 2024-03-28T22:39:57.348 INFO:tasks.workunit.client.0.smithi043.stdout: 1 139423 8.73 MB/sec warmup 88 sec latency 52.726 ms 2024-03-28T22:39:58.348 INFO:tasks.workunit.client.0.smithi043.stdout: 1 141381 8.76 MB/sec warmup 89 sec latency 12.145 ms 2024-03-28T22:39:58.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:39:58 smithi172 ceph-mon[36566]: pgmap v87: 129 pgs: 129 active+clean; 40 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 365 op/s 2024-03-28T22:39:58.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:39:58 smithi082 ceph-mon[36471]: pgmap v87: 129 pgs: 129 active+clean; 40 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 365 op/s 2024-03-28T22:39:59.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:39:58 smithi043 ceph-mon[31823]: pgmap v87: 129 pgs: 129 active+clean; 40 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 365 op/s 2024-03-28T22:39:59.349 INFO:tasks.workunit.client.0.smithi043.stdout: 1 142504 8.73 MB/sec warmup 90 sec latency 21.033 ms 2024-03-28T22:40:00.349 INFO:tasks.workunit.client.0.smithi043.stdout: 1 143909 8.70 MB/sec warmup 91 sec latency 16.132 ms 2024-03-28T22:40:00.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:00 smithi172 ceph-mon[36566]: pgmap v88: 129 pgs: 129 active+clean; 40 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 288 op/s 2024-03-28T22:40:00.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:00 smithi172 ceph-mon[36566]: overall HEALTH_OK 2024-03-28T22:40:00.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:00 smithi082 ceph-mon[36471]: pgmap v88: 129 pgs: 129 active+clean; 40 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 288 op/s 2024-03-28T22:40:00.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:00 smithi082 ceph-mon[36471]: overall HEALTH_OK 2024-03-28T22:40:01.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:00 smithi043 ceph-mon[31823]: pgmap v88: 129 pgs: 129 active+clean; 40 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 288 op/s 2024-03-28T22:40:01.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:00 smithi043 ceph-mon[31823]: overall HEALTH_OK 2024-03-28T22:40:01.349 INFO:tasks.workunit.client.0.smithi043.stdout: 1 144201 8.61 MB/sec warmup 92 sec latency 78.555 ms 2024-03-28T22:40:02.349 INFO:tasks.workunit.client.0.smithi043.stdout: 1 145593 8.64 MB/sec warmup 93 sec latency 71.108 ms 2024-03-28T22:40:02.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:02 smithi172 ceph-mon[36566]: pgmap v89: 129 pgs: 129 active+clean; 40 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 288 op/s 2024-03-28T22:40:02.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:02 smithi082 ceph-mon[36471]: pgmap v89: 129 pgs: 129 active+clean; 40 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 288 op/s 2024-03-28T22:40:03.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:02 smithi043 ceph-mon[31823]: pgmap v89: 129 pgs: 129 active+clean; 40 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 288 op/s 2024-03-28T22:40:03.349 INFO:tasks.workunit.client.0.smithi043.stdout: 1 146625 8.59 MB/sec warmup 94 sec latency 19.732 ms 2024-03-28T22:40:04.349 INFO:tasks.workunit.client.0.smithi043.stdout: 1 148986 8.66 MB/sec warmup 95 sec latency 9.600 ms 2024-03-28T22:40:04.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:04 smithi172 ceph-mon[36566]: pgmap v90: 129 pgs: 129 active+clean; 41 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 392 op/s 2024-03-28T22:40:04.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:04 smithi082 ceph-mon[36471]: pgmap v90: 129 pgs: 129 active+clean; 41 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 392 op/s 2024-03-28T22:40:05.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:04 smithi043 ceph-mon[31823]: pgmap v90: 129 pgs: 129 active+clean; 41 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 392 op/s 2024-03-28T22:40:05.349 INFO:tasks.workunit.client.0.smithi043.stdout: 1 149914 8.59 MB/sec warmup 96 sec latency 17.768 ms 2024-03-28T22:40:06.349 INFO:tasks.workunit.client.0.smithi043.stdout: 1 152188 8.63 MB/sec warmup 97 sec latency 10.691 ms 2024-03-28T22:40:06.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:06 smithi172 ceph-mon[36566]: pgmap v91: 129 pgs: 129 active+clean; 41 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 292 op/s 2024-03-28T22:40:06.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:06 smithi082 ceph-mon[36471]: pgmap v91: 129 pgs: 129 active+clean; 41 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 292 op/s 2024-03-28T22:40:07.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:06 smithi043 ceph-mon[31823]: pgmap v91: 129 pgs: 129 active+clean; 41 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 292 op/s 2024-03-28T22:40:07.340 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.342+0000 7f49f4e00640 1 -- 172.21.15.43:0/1925572501 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49f01066b0 msgr2=0x7f49f0106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:07.340 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.342+0000 7f49f4e00640 1 --2- 172.21.15.43:0/1925572501 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49f01066b0 0x7f49f0106a90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f49e0009930 tx=0x7f49e002bc20 comp rx=0 tx=0).stop 2024-03-28T22:40:07.340 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.342+0000 7f49f4e00640 1 -- 172.21.15.43:0/1925572501 shutdown_connections 2024-03-28T22:40:07.340 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.342+0000 7f49f4e00640 1 --2- 172.21.15.43:0/1925572501 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f49f0100610 0x7f49f0100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.340 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.342+0000 7f49f4e00640 1 --2- 172.21.15.43:0/1925572501 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49f0100fd0 0x7f49f010d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.340 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.342+0000 7f49f4e00640 1 --2- 172.21.15.43:0/1925572501 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49f01066b0 0x7f49f0106a90 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.340 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.342+0000 7f49f4e00640 1 -- 172.21.15.43:0/1925572501 >> 172.21.15.43:0/1925572501 conn(0x7f49f0077770 msgr2=0x7f49f0077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:07.341 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.342+0000 7f49f4e00640 1 -- 172.21.15.43:0/1925572501 shutdown_connections 2024-03-28T22:40:07.341 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.342+0000 7f49f4e00640 1 -- 172.21.15.43:0/1925572501 wait complete. 2024-03-28T22:40:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.343+0000 7f49f4e00640 1 Processor -- start 2024-03-28T22:40:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.344+0000 7f49f4e00640 1 -- start start 2024-03-28T22:40:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.344+0000 7f49f4e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49f0100610 0x7f49f019a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.344+0000 7f49f4e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f49f0100fd0 0x7f49f019a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.344+0000 7f49f4e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49f01066b0 0x7f49f019ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.344+0000 7f49f4e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f49f0111b80 con 0x7f49f01066b0 2024-03-28T22:40:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.344+0000 7f49f4e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f49f0111a00 con 0x7f49f0100fd0 2024-03-28T22:40:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.344+0000 7f49f4e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f49f0111d00 con 0x7f49f0100610 2024-03-28T22:40:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.344+0000 7f49ede00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49f0100610 0x7f49f019a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.344+0000 7f49ed400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f49f0100fd0 0x7f49f019a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.344+0000 7f49ed400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f49f0100fd0 0x7f49f019a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:57020/0 (socket says 172.21.15.43:57020) 2024-03-28T22:40:07.343 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.344+0000 7f49ede00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49f0100610 0x7f49f019a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40710/0 (socket says 172.21.15.43:40710) 2024-03-28T22:40:07.343 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.344+0000 7f49ed400640 1 -- 172.21.15.43:0/1013500016 learned_addr learned my addr 172.21.15.43:0/1013500016 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:07.343 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.344+0000 7f49ee800640 1 --2- 172.21.15.43:0/1013500016 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49f01066b0 0x7f49f019ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:07.344 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.345+0000 7f49ede00640 1 -- 172.21.15.43:0/1013500016 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f49f0100fd0 msgr2=0x7f49f019a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:07.344 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.345+0000 7f49ede00640 1 --2- 172.21.15.43:0/1013500016 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f49f0100fd0 0x7f49f019a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.345 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.345+0000 7f49ede00640 1 -- 172.21.15.43:0/1013500016 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49f01066b0 msgr2=0x7f49f019ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:07.345 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.345+0000 7f49ede00640 1 --2- 172.21.15.43:0/1013500016 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49f01066b0 0x7f49f019ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.345 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.345+0000 7f49ede00640 1 -- 172.21.15.43:0/1013500016 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49f019f1c0 con 0x7f49f0100610 2024-03-28T22:40:07.345 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.345+0000 7f49ed400640 1 --2- 172.21.15.43:0/1013500016 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f49f0100fd0 0x7f49f019a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:07.345 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.345+0000 7f49ede00640 1 --2- 172.21.15.43:0/1013500016 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49f0100610 0x7f49f019a220 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f49e0009230 tx=0x7f49e0004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:07.345 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.345+0000 7f49e6a00640 1 -- 172.21.15.43:0/1013500016 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e0043020 con 0x7f49f0100610 2024-03-28T22:40:07.345 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.345+0000 7f49e6a00640 1 -- 172.21.15.43:0/1013500016 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49e0005000 con 0x7f49f0100610 2024-03-28T22:40:07.345 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.345+0000 7f49e6a00640 1 -- 172.21.15.43:0/1013500016 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e0003b20 con 0x7f49f0100610 2024-03-28T22:40:07.345 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.345+0000 7f49f4e00640 1 -- 172.21.15.43:0/1013500016 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49f019f450 con 0x7f49f0100610 2024-03-28T22:40:07.346 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.346+0000 7f49f4e00640 1 -- 172.21.15.43:0/1013500016 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49f01a6d40 con 0x7f49f0100610 2024-03-28T22:40:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.350+0000 7f49f7c2a740 1 Processor -- start 2024-03-28T22:40:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.350+0000 7f49f7c2a740 1 -- start start 2024-03-28T22:40:07.350 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.350+0000 7f49f7c2a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5577279173a0 0x557727917780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:07.350 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.351+0000 7f49e6a00640 1 -- 172.21.15.43:0/1013500016 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f49e0046020 con 0x7f49f0100610 2024-03-28T22:40:07.350 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.351+0000 7f49f7c2a740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557727917cc0 0x557727920550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:07.350 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.351+0000 7f49f7c2a740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557727920a90 0x557727922e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:07.350 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.351+0000 7f49f7c2a740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55772787a640 con 0x557727917cc0 2024-03-28T22:40:07.350 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.351+0000 7f49f7c2a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x557727823810 con 0x5577279173a0 2024-03-28T22:40:07.350 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.351+0000 7f49f7c2a740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5577278766e0 con 0x557727920a90 2024-03-28T22:40:07.350 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.351+0000 7f49ee800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5577279173a0 0x557727917780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:07.350 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.351+0000 7f49ee800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5577279173a0 0x557727917780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:57032/0 (socket says 172.21.15.43:57032) 2024-03-28T22:40:07.350 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.351+0000 7f49ee800640 1 -- 172.21.15.43:0/1207563301 learned_addr learned my addr 172.21.15.43:0/1207563301 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.351+0000 7f49ed400640 1 --2- 172.21.15.43:0/1207563301 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557727920a90 0x557727922e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.351+0000 7f49ede00640 1 --2- 172.21.15.43:0/1207563301 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557727917cc0 0x557727920550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.351+0000 7f49ee800640 1 -- 172.21.15.43:0/1207563301 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557727920a90 msgr2=0x557727922e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.351+0000 7f49ee800640 1 --2- 172.21.15.43:0/1207563301 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557727920a90 0x557727922e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.351+0000 7f49ee800640 1 -- 172.21.15.43:0/1207563301 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557727917cc0 msgr2=0x557727920550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.351+0000 7f49ee800640 1 --2- 172.21.15.43:0/1207563301 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557727917cc0 0x557727920550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.351+0000 7f49ee800640 1 -- 172.21.15.43:0/1207563301 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55772758f210 con 0x5577279173a0 2024-03-28T22:40:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.351+0000 7f49ed400640 1 --2- 172.21.15.43:0/1207563301 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557727920a90 0x557727922e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.352+0000 7f49ee800640 1 --2- 172.21.15.43:0/1207563301 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5577279173a0 0x557727917780 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f49e8009f50 tx=0x7f49e80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e569928756832d7a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:07.351 INFO:tasks.workunit.client.0.smithi043.stdout: 1 153120 8.60 MB/sec warmup 98 sec latency 51.738 ms 2024-03-28T22:40:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.352+0000 7f49db400640 1 -- 172.21.15.43:0/1207563301 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e8004d20 con 0x5577279173a0 2024-03-28T22:40:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.352+0000 7f49db400640 1 -- 172.21.15.43:0/1207563301 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49e8004050 con 0x5577279173a0 2024-03-28T22:40:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.352+0000 7f49db400640 1 -- 172.21.15.43:0/1207563301 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e8005240 con 0x5577279173a0 2024-03-28T22:40:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.352+0000 7f49e6a00640 1 --2- 172.21.15.43:0/1013500016 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f49d40a2010 0x7f49d40a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.352+0000 7f49e6a00640 1 -- 172.21.15.43:0/1013500016 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7f49e00f0450 con 0x7f49f0100610 2024-03-28T22:40:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.352+0000 7f49ede00640 1 --2- 172.21.15.43:0/1013500016 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f49d40a2010 0x7f49d40a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.353+0000 7f49f7c2a740 1 -- 172.21.15.43:0/1207563301 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5577279173a0 msgr2=0x557727917780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.353+0000 7f49f7c2a740 1 --2- 172.21.15.43:0/1207563301 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5577279173a0 0x557727917780 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f49e8009f50 tx=0x7f49e80118d0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.353+0000 7f49f7c2a740 1 -- 172.21.15.43:0/1207563301 shutdown_connections 2024-03-28T22:40:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.353+0000 7f49f7c2a740 1 --2- 172.21.15.43:0/1207563301 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557727920a90 0x557727922e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.353+0000 7f49f7c2a740 1 --2- 172.21.15.43:0/1207563301 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557727917cc0 0x557727920550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.353+0000 7f49f7c2a740 1 --2- 172.21.15.43:0/1207563301 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5577279173a0 0x557727917780 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.353+0000 7f49f7c2a740 1 -- 172.21.15.43:0/1207563301 >> 172.21.15.43:0/1207563301 conn(0x557727911cc0 msgr2=0x5577279120a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.353+0000 7f49f7c2a740 1 -- 172.21.15.43:0/1207563301 shutdown_connections 2024-03-28T22:40:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.353+0000 7f49f7c2a740 1 -- 172.21.15.43:0/1207563301 wait complete. 2024-03-28T22:40:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.353+0000 7f49ede00640 1 --2- 172.21.15.43:0/1013500016 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f49d40a2010 0x7f49d40a44d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x557727916e50 tx=0x7f49e00f3f00 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.354+0000 7f49f7c2a740 1 -- start start 2024-03-28T22:40:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.354+0000 7f49f7c2a740 1 Processor -- start 2024-03-28T22:40:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.354+0000 7f49f7c2a740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5577279173a0 0x55772791dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:07.359 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.354+0000 7f49f7c2a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557727917cc0 0x55772791e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:07.359 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.354+0000 7f49f7c2a740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557727920a90 0x55772791e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:07.359 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.354+0000 7f49f7c2a740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5577279236c0 con 0x5577279173a0 2024-03-28T22:40:07.359 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.354+0000 7f49f7c2a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x557727818d00 con 0x557727917cc0 2024-03-28T22:40:07.359 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.354+0000 7f49f7c2a740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x557727923840 con 0x557727920a90 2024-03-28T22:40:07.359 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.355+0000 7f49ee800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5577279173a0 0x55772791dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:07.359 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.355+0000 7f49f7c2a740 10 client.0 _ll_register_callbacks cb 0x557727803490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:07.359 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.355+0000 7f49ed400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557727917cc0 0x55772791e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:07.359 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.355+0000 7f49ee800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5577279173a0 0x55772791dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38330/0 (socket says 172.21.15.43:38330) 2024-03-28T22:40:07.359 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.355+0000 7f49ee800640 1 -- 172.21.15.43:0/1169365790 learned_addr learned my addr 172.21.15.43:0/1169365790 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:07.359 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.355+0000 7f49ee800640 1 --2- 172.21.15.43:0/1169365790 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557727920a90 0x55772791e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:07.359 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.355+0000 7f49ed400640 1 -- 172.21.15.43:0/1169365790 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557727920a90 msgr2=0x55772791e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:07.359 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.355+0000 7f49ed400640 1 --2- 172.21.15.43:0/1169365790 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557727920a90 0x55772791e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.359 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.355+0000 7f49ed400640 1 -- 172.21.15.43:0/1169365790 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5577279173a0 msgr2=0x55772791dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.355+0000 7f49ed400640 1 --2- 172.21.15.43:0/1169365790 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5577279173a0 0x55772791dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.355+0000 7f49ed400640 1 -- 172.21.15.43:0/1169365790 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55772765b620 con 0x557727917cc0 2024-03-28T22:40:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.355+0000 7f49ee800640 1 --2- 172.21.15.43:0/1169365790 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5577279173a0 0x55772791dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.355+0000 7f49ed400640 1 --2- 172.21.15.43:0/1169365790 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557727917cc0 0x55772791e020 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f49dc009620 tx=0x7f49dc00f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c638dae16473c3da server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.355+0000 7f49cfe00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:40:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.356+0000 7f49cfe00640 1 -- 172.21.15.43:0/1169365790 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49dc0120d0 con 0x557727917cc0 2024-03-28T22:40:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.356+0000 7f49f7c2a740 1 -- 172.21.15.43:0/1169365790 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49e8004050 con 0x557727917cc0 2024-03-28T22:40:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.356+0000 7f49cfe00640 1 -- 172.21.15.43:0/1169365790 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7f49dc004020 con 0x557727917cc0 2024-03-28T22:40:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.356+0000 7f49cfe00640 1 -- 172.21.15.43:0/1169365790 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f49dc012270 con 0x557727917cc0 2024-03-28T22:40:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.356+0000 7f49f7c2a740 10 client.26705 fetch_fsmap learned FSMap version 30 2024-03-28T22:40:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.356+0000 7f49f7c2a740 1 -- 172.21.15.43:0/1169365790 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55772792f660 con 0x557727917cc0 2024-03-28T22:40:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.356+0000 7f49cfe00640 1 -- 172.21.15.43:0/1169365790 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49e8004050 con 0x557727917cc0 2024-03-28T22:40:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.357+0000 7f49cfe00640 1 -- 172.21.15.43:0/1169365790 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f49dc01c470 con 0x557727917cc0 2024-03-28T22:40:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.357+0000 7f49f7c2a740 10 client.26705 fetch_fsmap finished waiting for FSMap version 30 2024-03-28T22:40:07.361 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.357+0000 7f49f7c2a740 10 client.26705 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:40:07.361 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.357+0000 7f49f7c2a740 20 client.26705 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:07.361 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.357+0000 7f49f7c2a740 1 --2- 172.21.15.43:0/1169365790 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557727930ad0 0x557727932ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:07.361 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.357+0000 7f49f7c2a740 4 client.26705 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:40:07.361 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.357+0000 7f49f7c2a740 1 -- 172.21.15.43:0/1169365790 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55772788e030 con 0x557727930ad0 2024-03-28T22:40:07.361 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.357+0000 7f49ee800640 1 --2- 172.21.15.43:0/1169365790 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557727930ad0 0x557727932ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:07.361 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.357+0000 7f49ee800640 1 --2- 172.21.15.43:0/1169365790 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557727930ad0 0x557727932ec0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa405921c439fca9 server_cookie=12f8a493b817a3e5 in_seq=0 out_seq=0 2024-03-28T22:40:07.361 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.357+0000 7f49cfe00640 10 client.26705 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:07.361 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.358+0000 7f49cfe00640 1 -- 172.21.15.43:0/1169365790 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55772788e030 con 0x557727930ad0 2024-03-28T22:40:07.361 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.358+0000 7f49cfe00640 10 client.26705 handle_command_reply: tid=0 2024-03-28T22:40:07.361 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.358+0000 7f49f7c2a740 1 client.26705 shutdown 2024-03-28T22:40:07.361 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.359+0000 7f49f7c2a740 1 -- 172.21.15.43:0/1169365790 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557727917cc0 msgr2=0x55772791e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:07.361 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.359+0000 7f49f7c2a740 1 --2- 172.21.15.43:0/1169365790 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557727917cc0 0x55772791e020 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f49dc009620 tx=0x7f49dc00f790 comp rx=0 tx=0).stop 2024-03-28T22:40:07.361 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.359+0000 7f49f7c2a740 1 -- 172.21.15.43:0/1169365790 shutdown_connections 2024-03-28T22:40:07.361 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.359+0000 7f49f7c2a740 1 --2- 172.21.15.43:0/1169365790 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557727930ad0 0x557727932ec0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.362 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.359+0000 7f49f7c2a740 1 --2- 172.21.15.43:0/1169365790 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557727917cc0 0x55772791e020 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.362 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.359+0000 7f49cfe00640 0 client.26705 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:07.362 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.359+0000 7f49f7c2a740 1 --2- 172.21.15.43:0/1169365790 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557727920a90 0x55772791e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.362 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.359+0000 7f49f7c2a740 1 --2- 172.21.15.43:0/1169365790 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5577279173a0 0x55772791dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.362 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.359+0000 7f49f7c2a740 1 -- 172.21.15.43:0/1169365790 >> 172.21.15.43:0/1169365790 conn(0x557727911cc0 msgr2=0x557727913a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:07.362 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.359+0000 7f49f7c2a740 1 -- 172.21.15.43:0/1169365790 shutdown_connections 2024-03-28T22:40:07.362 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.359+0000 7f49f7c2a740 1 -- 172.21.15.43:0/1169365790 wait complete. 2024-03-28T22:40:07.362 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.359+0000 7f49f7c2a740 20 client.26705 trim_cache size 0 max 16384 2024-03-28T22:40:07.374 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.376+0000 7f49f7c2a740 1 Processor -- start 2024-03-28T22:40:07.374 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.376+0000 7f49f7c2a740 1 -- start start 2024-03-28T22:40:07.374 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.377+0000 7f49f7c2a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5577279173a0 0x557727920210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.377+0000 7f49f7c2a740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557727917cc0 0x55772792c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.377+0000 7f49f7c2a740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557727920a90 0x5577279387d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.377+0000 7f49f7c2a740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f49f01b0db0 con 0x557727917cc0 2024-03-28T22:40:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.377+0000 7f49f7c2a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f49f01b28f0 con 0x5577279173a0 2024-03-28T22:40:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.377+0000 7f49f7c2a740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f49f01b1480 con 0x557727920a90 2024-03-28T22:40:07.376 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.377+0000 7f49ed400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557727917cc0 0x55772792c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:07.376 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.377+0000 7f49ed400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557727917cc0 0x55772792c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38346/0 (socket says 172.21.15.43:38346) 2024-03-28T22:40:07.376 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.377+0000 7f49ed400640 1 -- 172.21.15.43:0/3782135841 learned_addr learned my addr 172.21.15.43:0/3782135841 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:07.376 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.377+0000 7f49ee800640 1 --2- 172.21.15.43:0/3782135841 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5577279173a0 0x557727920210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:07.376 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.377+0000 7f49ed400640 1 -- 172.21.15.43:0/3782135841 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557727920a90 msgr2=0x5577279387d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:07.376 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.377+0000 7f49ee800640 1 --2- 172.21.15.43:0/3782135841 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557727920a90 0x5577279387d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:07.376 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.378+0000 7f49ed400640 1 --2- 172.21.15.43:0/3782135841 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557727920a90 0x5577279387d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.376 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.378+0000 7f49ed400640 1 -- 172.21.15.43:0/3782135841 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5577279173a0 msgr2=0x557727920210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:07.376 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.378+0000 7f49ed400640 1 --2- 172.21.15.43:0/3782135841 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5577279173a0 0x557727920210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.377 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.378+0000 7f49ed400640 1 -- 172.21.15.43:0/3782135841 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557727933230 con 0x557727917cc0 2024-03-28T22:40:07.377 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.378+0000 7f49ee800640 1 --2- 172.21.15.43:0/3782135841 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557727920a90 0x5577279387d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:40:07.377 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.378+0000 7f49ed400640 1 --2- 172.21.15.43:0/3782135841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557727917cc0 0x55772792c460 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f49dc00a010 tx=0x7f49dc005db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1244a2da6756c22 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:07.377 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.378+0000 7f49daa00640 1 -- 172.21.15.43:0/3782135841 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49dc01c470 con 0x557727917cc0 2024-03-28T22:40:07.377 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.378+0000 7f49daa00640 1 -- 172.21.15.43:0/3782135841 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49dc01c610 con 0x557727917cc0 2024-03-28T22:40:07.377 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.378+0000 7f49daa00640 1 -- 172.21.15.43:0/3782135841 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49dc003850 con 0x557727917cc0 2024-03-28T22:40:07.379 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.379+0000 7f49f7c2a740 1 -- 172.21.15.43:0/3782135841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557727917cc0 msgr2=0x55772792c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:07.379 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.379+0000 7f49f7c2a740 1 --2- 172.21.15.43:0/3782135841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557727917cc0 0x55772792c460 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f49dc00a010 tx=0x7f49dc005db0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.379 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.379+0000 7f49f7c2a740 1 -- 172.21.15.43:0/3782135841 shutdown_connections 2024-03-28T22:40:07.379 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.379+0000 7f49f7c2a740 1 --2- 172.21.15.43:0/3782135841 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557727920a90 0x5577279387d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.379 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.379+0000 7f49f7c2a740 1 --2- 172.21.15.43:0/3782135841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557727917cc0 0x55772792c460 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.379 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.379+0000 7f49f7c2a740 1 --2- 172.21.15.43:0/3782135841 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5577279173a0 0x557727920210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.380 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.379+0000 7f49f7c2a740 1 -- 172.21.15.43:0/3782135841 >> 172.21.15.43:0/3782135841 conn(0x557727911cc0 msgr2=0x5577279120a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:07.380 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.379+0000 7f49f7c2a740 1 -- 172.21.15.43:0/3782135841 shutdown_connections 2024-03-28T22:40:07.380 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.379+0000 7f49f7c2a740 1 -- 172.21.15.43:0/3782135841 wait complete. 2024-03-28T22:40:07.380 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.380+0000 7f49f7c2a740 1 -- start start 2024-03-28T22:40:07.380 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.380+0000 7f49f7c2a740 1 Processor -- start 2024-03-28T22:40:07.380 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.381+0000 7f49f7c2a740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5577279173a0 0x557727928a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:07.380 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.381+0000 7f49f7c2a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557727917cc0 0x557727919ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:07.380 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.381+0000 7f49f7c2a740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557727920a90 0x55772791a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:07.380 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.381+0000 7f49f7c2a740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f49f01b2220 con 0x5577279173a0 2024-03-28T22:40:07.380 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.381+0000 7f49f7c2a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f49f01b1b50 con 0x557727917cc0 2024-03-28T22:40:07.380 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.381+0000 7f49f7c2a740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f49f01aed90 con 0x557727920a90 2024-03-28T22:40:07.380 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.381+0000 7f49f7c2a740 10 client.0 _ll_register_callbacks cb 0x7f49dc009cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:07.380 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.381+0000 7f49ed400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557727917cc0 0x557727919ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:07.380 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.381+0000 7f49ed400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557727917cc0 0x557727919ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:57062/0 (socket says 172.21.15.43:57062) 2024-03-28T22:40:07.381 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.381+0000 7f49ed400640 1 -- 172.21.15.43:0/3606366762 learned_addr learned my addr 172.21.15.43:0/3606366762 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:07.381 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.381+0000 7f49ee800640 1 --2- 172.21.15.43:0/3606366762 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5577279173a0 0x557727928a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:07.381 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.381+0000 7f49ee800640 1 --2- 172.21.15.43:0/3606366762 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557727920a90 0x55772791a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:07.381 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.381+0000 7f49ed400640 1 -- 172.21.15.43:0/3606366762 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557727920a90 msgr2=0x55772791a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:07.381 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.381+0000 7f49ed400640 1 --2- 172.21.15.43:0/3606366762 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557727920a90 0x55772791a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.381 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.381+0000 7f49ed400640 1 -- 172.21.15.43:0/3606366762 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5577279173a0 msgr2=0x557727928a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:07.381 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.381+0000 7f49ed400640 1 --2- 172.21.15.43:0/3606366762 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5577279173a0 0x557727928a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.381 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.381+0000 7f49ed400640 1 -- 172.21.15.43:0/3606366762 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55772791a970 con 0x557727917cc0 2024-03-28T22:40:07.381 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.382+0000 7f49ee800640 1 --2- 172.21.15.43:0/3606366762 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5577279173a0 0x557727928a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.382+0000 7f49ed400640 1 --2- 172.21.15.43:0/3606366762 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557727917cc0 0x557727919ef0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f49dc00a010 tx=0x7f49dc0052c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69e74fc47ab0512d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.382+0000 7f49cfe00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:40:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.382+0000 7f49cfe00640 1 -- 172.21.15.43:0/3606366762 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49dc01c470 con 0x557727917cc0 2024-03-28T22:40:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.382+0000 7f49cfe00640 1 -- 172.21.15.43:0/3606366762 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9451+0+0 (secure 0 0 0) 0x7f49dc003800 con 0x557727917cc0 2024-03-28T22:40:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.382+0000 7f49f7c2a740 1 -- 172.21.15.43:0/3606366762 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49dc01c610 con 0x557727917cc0 2024-03-28T22:40:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.382+0000 7f49cfe00640 1 -- 172.21.15.43:0/3606366762 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f49dc005670 con 0x557727917cc0 2024-03-28T22:40:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.382+0000 7f49f7c2a740 10 client.26711 fetch_fsmap learned FSMap version 30 2024-03-28T22:40:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.382+0000 7f49f7c2a740 1 -- 172.21.15.43:0/3606366762 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5577279968e0 con 0x557727917cc0 2024-03-28T22:40:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.383+0000 7f49cfe00640 1 -- 172.21.15.43:0/3606366762 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49dc01c610 con 0x557727917cc0 2024-03-28T22:40:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.383+0000 7f49cfe00640 1 -- 172.21.15.43:0/3606366762 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f49dc010a30 con 0x557727917cc0 2024-03-28T22:40:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.383+0000 7f49f7c2a740 10 client.26711 fetch_fsmap finished waiting for FSMap version 30 2024-03-28T22:40:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.383+0000 7f49f7c2a740 10 client.26711 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:40:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.383+0000 7f49f7c2a740 20 client.26711 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.383+0000 7f49f7c2a740 1 --2- 172.21.15.43:0/3606366762 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557727996fc0 0x5577279993e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.383+0000 7f49f7c2a740 4 client.26711 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T22:40:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.383+0000 7f49f7c2a740 1 -- 172.21.15.43:0/3606366762 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557727887b80 con 0x557727996fc0 2024-03-28T22:40:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.383+0000 7f49ee800640 1 --2- 172.21.15.43:0/3606366762 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557727996fc0 0x5577279993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.384+0000 7f49ee800640 1 --2- 172.21.15.43:0/3606366762 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557727996fc0 0x5577279993e0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ede9c41664edf82e server_cookie=47d47c66ff15fb0d in_seq=0 out_seq=0 2024-03-28T22:40:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.384+0000 7f49cfe00640 10 client.26711 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:07.613 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.615+0000 7f49cfe00640 1 -- 172.21.15.43:0/3606366762 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557727887b80 con 0x557727996fc0 2024-03-28T22:40:07.613 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.615+0000 7f49cfe00640 10 client.26711 handle_command_reply: tid=0 2024-03-28T22:40:07.613 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.615+0000 7f49f7c2a740 1 client.26711 shutdown 2024-03-28T22:40:07.613 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.615+0000 7f49f7c2a740 1 -- 172.21.15.43:0/3606366762 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557727917cc0 msgr2=0x557727919ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:07.613 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.615+0000 7f49f7c2a740 1 --2- 172.21.15.43:0/3606366762 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557727917cc0 0x557727919ef0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f49dc00a010 tx=0x7f49dc0052c0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.613 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.615+0000 7f49f7c2a740 1 -- 172.21.15.43:0/3606366762 shutdown_connections 2024-03-28T22:40:07.613 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.615+0000 7f49f7c2a740 1 --2- 172.21.15.43:0/3606366762 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557727996fc0 0x5577279993e0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.613 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.615+0000 7f49f7c2a740 1 --2- 172.21.15.43:0/3606366762 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557727917cc0 0x557727919ef0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.613 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.615+0000 7f49f7c2a740 1 --2- 172.21.15.43:0/3606366762 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557727920a90 0x55772791a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.613 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.615+0000 7f49f7c2a740 1 --2- 172.21.15.43:0/3606366762 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5577279173a0 0x557727928a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.614 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.615+0000 7f49f7c2a740 1 -- 172.21.15.43:0/3606366762 >> 172.21.15.43:0/3606366762 conn(0x557727911cc0 msgr2=0x55772791c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:07.614 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.615+0000 7f49cfe00640 0 client.26711 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:07.614 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.616+0000 7f49f7c2a740 1 -- 172.21.15.43:0/3606366762 shutdown_connections 2024-03-28T22:40:07.614 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.616+0000 7f49f7c2a740 1 -- 172.21.15.43:0/3606366762 wait complete. 2024-03-28T22:40:07.614 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.616+0000 7f49f7c2a740 20 client.26711 trim_cache size 0 max 16384 2024-03-28T22:40:07.615 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:{ 2024-03-28T22:40:07.615 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "message": "", 2024-03-28T22:40:07.615 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "return_code": 0 2024-03-28T22:40:07.615 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:} 2024-03-28T22:40:07.615 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.617+0000 7f49f4e00640 1 -- 172.21.15.43:0/1013500016 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f49d40a2010 msgr2=0x7f49d40a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.617+0000 7f49f4e00640 1 --2- 172.21.15.43:0/1013500016 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f49d40a2010 0x7f49d40a44d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x557727916e50 tx=0x7f49e00f3f00 comp rx=0 tx=0).stop 2024-03-28T22:40:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.617+0000 7f49f4e00640 1 -- 172.21.15.43:0/1013500016 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49f0100610 msgr2=0x7f49f019a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.617+0000 7f49f4e00640 1 --2- 172.21.15.43:0/1013500016 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49f0100610 0x7f49f019a220 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f49e0009230 tx=0x7f49e0004270 comp rx=0 tx=0).stop 2024-03-28T22:40:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.617+0000 7f49f4e00640 1 -- 172.21.15.43:0/1013500016 shutdown_connections 2024-03-28T22:40:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.617+0000 7f49f4e00640 1 --2- 172.21.15.43:0/1013500016 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f49d40a2010 0x7f49d40a44d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.617+0000 7f49f4e00640 1 --2- 172.21.15.43:0/1013500016 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f49f0100fd0 0x7f49f019a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.617+0000 7f49f4e00640 1 --2- 172.21.15.43:0/1013500016 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f49f01066b0 0x7f49f019ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.617+0000 7f49f4e00640 1 --2- 172.21.15.43:0/1013500016 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f49f0100610 0x7f49f019a220 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.617+0000 7f49f4e00640 1 -- 172.21.15.43:0/1013500016 >> 172.21.15.43:0/1013500016 conn(0x7f49f0077770 msgr2=0x7f49f01021e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.617+0000 7f49f4e00640 1 -- 172.21.15.43:0/1013500016 shutdown_connections 2024-03-28T22:40:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:07.617+0000 7f49f4e00640 1 -- 172.21.15.43:0/1013500016 wait complete. 2024-03-28T22:40:08.350 INFO:tasks.workunit.client.0.smithi043.stdout: 1 154641 8.57 MB/sec warmup 99 sec latency 93.870 ms 2024-03-28T22:40:08.449 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 28 22:40:08 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-e[66639]: ignoring --setuser ceph since I am not root 2024-03-28T22:40:08.449 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 28 22:40:08 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-e[66639]: ignoring --setgroup ceph since I am not root 2024-03-28T22:40:08.449 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 28 22:40:08 smithi082 ceph-mds[66643]: -- 172.21.15.82:0/1885941878 <== mon.2 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560582c7da40 con 0x560582c51000 2024-03-28T22:40:08.449 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 28 22:40:08 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-e[66639]: starting mds.e at 2024-03-28T22:40:08.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:08 smithi082 ceph-mon[36471]: pgmap v92: 129 pgs: 129 active+clean; 46 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 376 op/s 2024-03-28T22:40:08.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:08 smithi082 ceph-mon[36471]: daemon mds.e restarted 2024-03-28T22:40:09.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:08 smithi043 ceph-mon[31823]: pgmap v92: 129 pgs: 129 active+clean; 46 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 376 op/s 2024-03-28T22:40:09.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:08 smithi043 ceph-mon[31823]: daemon mds.e restarted 2024-03-28T22:40:09.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:08 smithi172 ceph-mon[36566]: pgmap v92: 129 pgs: 129 active+clean; 46 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 376 op/s 2024-03-28T22:40:09.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:08 smithi172 ceph-mon[36566]: daemon mds.e restarted 2024-03-28T22:40:09.350 INFO:tasks.workunit.client.0.smithi043.stdout: 1 156294 8.62 MB/sec warmup 100 sec latency 16.013 ms 2024-03-28T22:40:09.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:09 smithi082 ceph-mon[36471]: osdmap e99: 12 total, 12 up, 12 in 2024-03-28T22:40:09.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:09 smithi082 ceph-mon[36471]: mds.? [v2:172.21.15.82:6834/215779342,v1:172.21.15.82:6835/215779342] up:boot 2024-03-28T22:40:09.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:09 smithi082 ceph-mon[36471]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-03-28T22:40:09.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:09 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T22:40:09.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:09 smithi082 ceph-mon[36471]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T22:40:10.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:09 smithi043 ceph-mon[31823]: osdmap e99: 12 total, 12 up, 12 in 2024-03-28T22:40:10.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:09 smithi043 ceph-mon[31823]: mds.? [v2:172.21.15.82:6834/215779342,v1:172.21.15.82:6835/215779342] up:boot 2024-03-28T22:40:10.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:09 smithi043 ceph-mon[31823]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-03-28T22:40:10.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:09 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T22:40:10.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:09 smithi043 ceph-mon[31823]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T22:40:10.046 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:09 smithi172 ceph-mon[36566]: osdmap e99: 12 total, 12 up, 12 in 2024-03-28T22:40:10.046 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:09 smithi172 ceph-mon[36566]: mds.? [v2:172.21.15.82:6834/215779342,v1:172.21.15.82:6835/215779342] up:boot 2024-03-28T22:40:10.046 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:09 smithi172 ceph-mon[36566]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-03-28T22:40:10.046 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:09 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T22:40:10.046 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:09 smithi172 ceph-mon[36566]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T22:40:10.350 INFO:tasks.workunit.client.0.smithi043.stdout: 1 157200 8.56 MB/sec warmup 101 sec latency 21.153 ms 2024-03-28T22:40:10.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:10 smithi082 ceph-mon[36471]: pgmap v94: 129 pgs: 129 active+clean; 46 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 226 op/s 2024-03-28T22:40:11.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:10 smithi043 ceph-mon[31823]: pgmap v94: 129 pgs: 129 active+clean; 46 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 226 op/s 2024-03-28T22:40:11.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:10 smithi172 ceph-mon[36566]: pgmap v94: 129 pgs: 129 active+clean; 46 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 226 op/s 2024-03-28T22:40:11.350 INFO:tasks.workunit.client.0.smithi043.stdout: 1 159314 8.59 MB/sec warmup 102 sec latency 11.134 ms 2024-03-28T22:40:12.350 INFO:tasks.workunit.client.0.smithi043.stdout: 1 160259 8.56 MB/sec warmup 103 sec latency 52.888 ms 2024-03-28T22:40:13.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:12 smithi043 ceph-mon[31823]: pgmap v95: 129 pgs: 129 active+clean; 46 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 226 op/s 2024-03-28T22:40:13.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:12 smithi172 ceph-mon[36566]: pgmap v95: 129 pgs: 129 active+clean; 46 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 226 op/s 2024-03-28T22:40:13.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:12 smithi082 ceph-mon[36471]: pgmap v95: 129 pgs: 129 active+clean; 46 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 226 op/s 2024-03-28T22:40:13.350 INFO:tasks.workunit.client.0.smithi043.stdout: 1 162009 8.54 MB/sec warmup 104 sec latency 20.396 ms 2024-03-28T22:40:14.351 INFO:tasks.workunit.client.0.smithi043.stdout: 1 163459 8.57 MB/sec warmup 105 sec latency 10.444 ms 2024-03-28T22:40:15.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:14 smithi043 ceph-mon[31823]: pgmap v96: 129 pgs: 129 active+clean; 37 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 309 op/s 2024-03-28T22:40:15.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:14 smithi172 ceph-mon[36566]: pgmap v96: 129 pgs: 129 active+clean; 37 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 309 op/s 2024-03-28T22:40:15.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:14 smithi082 ceph-mon[36471]: pgmap v96: 129 pgs: 129 active+clean; 37 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 309 op/s 2024-03-28T22:40:15.351 INFO:tasks.workunit.client.0.smithi043.stdout: 1 164618 8.54 MB/sec warmup 106 sec latency 15.971 ms 2024-03-28T22:40:16.351 INFO:tasks.workunit.client.0.smithi043.stdout: 1 166692 8.58 MB/sec warmup 107 sec latency 14.049 ms 2024-03-28T22:40:17.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:16 smithi043 ceph-mon[31823]: pgmap v97: 129 pgs: 129 active+clean; 37 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 309 op/s 2024-03-28T22:40:17.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:16 smithi172 ceph-mon[36566]: pgmap v97: 129 pgs: 129 active+clean; 37 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 309 op/s 2024-03-28T22:40:17.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:16 smithi082 ceph-mon[36471]: pgmap v97: 129 pgs: 129 active+clean; 37 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 309 op/s 2024-03-28T22:40:17.351 INFO:tasks.workunit.client.0.smithi043.stdout: 1 167409 8.52 MB/sec warmup 108 sec latency 38.249 ms 2024-03-28T22:40:18.351 INFO:tasks.workunit.client.0.smithi043.stdout: 1 169110 8.50 MB/sec warmup 109 sec latency 77.991 ms 2024-03-28T22:40:19.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:18 smithi043 ceph-mon[31823]: pgmap v98: 129 pgs: 129 active+clean; 49 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 283 op/s 2024-03-28T22:40:19.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:18 smithi172 ceph-mon[36566]: pgmap v98: 129 pgs: 129 active+clean; 49 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 283 op/s 2024-03-28T22:40:19.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:18 smithi082 ceph-mon[36471]: pgmap v98: 129 pgs: 129 active+clean; 49 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 283 op/s 2024-03-28T22:40:19.351 INFO:tasks.workunit.client.0.smithi043.stdout: 1 170597 8.53 MB/sec warmup 110 sec latency 10.768 ms 2024-03-28T22:40:20.351 INFO:tasks.workunit.client.0.smithi043.stdout: 1 171769 8.50 MB/sec warmup 111 sec latency 15.462 ms 2024-03-28T22:40:20.441 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:40:20.441 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:40:20.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.687+0000 7fa8c1800640 1 -- 172.21.15.43:0/3691271746 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa8bc073b00 msgr2=0x7fa8bc073ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:20.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.687+0000 7fa8c1800640 1 --2- 172.21.15.43:0/3691271746 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa8bc073b00 0x7fa8bc073ee0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac009900 tx=0x7fa8ac02bcd0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.687+0000 7fa8c1800640 1 -- 172.21.15.43:0/3691271746 shutdown_connections 2024-03-28T22:40:20.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.687+0000 7fa8c1800640 1 --2- 172.21.15.43:0/3691271746 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8bc10de80 0x7fa8bc110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.687+0000 7fa8c1800640 1 --2- 172.21.15.43:0/3691271746 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8bc074420 0x7fa8bc10d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.687+0000 7fa8c1800640 1 --2- 172.21.15.43:0/3691271746 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa8bc073b00 0x7fa8bc073ee0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.687+0000 7fa8c1800640 1 -- 172.21.15.43:0/3691271746 >> 172.21.15.43:0/3691271746 conn(0x7fa8bc0fd0f0 msgr2=0x7fa8bc0ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:20.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.687+0000 7fa8c1800640 1 -- 172.21.15.43:0/3691271746 shutdown_connections 2024-03-28T22:40:20.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.687+0000 7fa8c1800640 1 -- 172.21.15.43:0/3691271746 wait complete. 2024-03-28T22:40:20.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.688+0000 7fa8c1800640 1 Processor -- start 2024-03-28T22:40:20.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.689+0000 7fa8c1800640 1 -- start start 2024-03-28T22:40:20.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.689+0000 7fa8c1800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8bc073b00 0x7fa8bc19a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.689+0000 7fa8c1800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa8bc074420 0x7fa8bc19a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.689+0000 7fa8c1800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8bc10de80 0x7fa8bc19e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.689+0000 7fa8c1800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc113bf0 con 0x7fa8bc073b00 2024-03-28T22:40:20.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.689+0000 7fa8c1800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc113a70 con 0x7fa8bc074420 2024-03-28T22:40:20.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.689+0000 7fa8c1800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc113d70 con 0x7fa8bc10de80 2024-03-28T22:40:20.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.689+0000 7fa8bb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa8bc074420 0x7fa8bc19a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.689+0000 7fa8bb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa8bc074420 0x7fa8bc19a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45122/0 (socket says 172.21.15.43:45122) 2024-03-28T22:40:20.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.689+0000 7fa8bb400640 1 -- 172.21.15.43:0/1361707164 learned_addr learned my addr 172.21.15.43:0/1361707164 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:20.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.689+0000 7fa8c0e00640 1 --2- 172.21.15.43:0/1361707164 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8bc10de80 0x7fa8bc19e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.689+0000 7fa8bbe00640 1 --2- 172.21.15.43:0/1361707164 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8bc073b00 0x7fa8bc19a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.690+0000 7fa8c0e00640 1 -- 172.21.15.43:0/1361707164 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa8bc074420 msgr2=0x7fa8bc19a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:20.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.690+0000 7fa8c0e00640 1 --2- 172.21.15.43:0/1361707164 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa8bc074420 0x7fa8bc19a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.690+0000 7fa8c0e00640 1 -- 172.21.15.43:0/1361707164 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8bc073b00 msgr2=0x7fa8bc19a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:20.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.690+0000 7fa8c0e00640 1 --2- 172.21.15.43:0/1361707164 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8bc073b00 0x7fa8bc19a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.690+0000 7fa8c0e00640 1 -- 172.21.15.43:0/1361707164 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8bc19f080 con 0x7fa8bc10de80 2024-03-28T22:40:20.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.690+0000 7fa8c0e00640 1 --2- 172.21.15.43:0/1361707164 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8bc10de80 0x7fa8bc19e950 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4009fd0 tx=0x7fa8b400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:20.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.690+0000 7fa8b8c00640 1 -- 172.21.15.43:0/1361707164 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b400ce20 con 0x7fa8bc10de80 2024-03-28T22:40:20.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.690+0000 7fa8b8c00640 1 -- 172.21.15.43:0/1361707164 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8b4005020 con 0x7fa8bc10de80 2024-03-28T22:40:20.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.690+0000 7fa8c1800640 1 -- 172.21.15.43:0/1361707164 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8bc19f370 con 0x7fa8bc10de80 2024-03-28T22:40:20.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.691+0000 7fa8b8c00640 1 -- 172.21.15.43:0/1361707164 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b4010720 con 0x7fa8bc10de80 2024-03-28T22:40:20.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.691+0000 7fa8c1800640 1 -- 172.21.15.43:0/1361707164 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8bc1a6c00 con 0x7fa8bc10de80 2024-03-28T22:40:20.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.696+0000 7fa8c5a10740 1 Processor -- start 2024-03-28T22:40:20.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.696+0000 7fa8b8c00640 1 -- 172.21.15.43:0/1361707164 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fa8b4005540 con 0x7fa8bc10de80 2024-03-28T22:40:20.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.696+0000 7fa8c5a10740 1 -- start start 2024-03-28T22:40:20.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.697+0000 7fa8c5a10740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c94fa21db0 0x55c94fa20b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.697+0000 7fa8c5a10740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c94fa210d0 0x55c94fa21550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.697+0000 7fa8c5a10740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c94fa32c50 0x55c94fa35040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.697+0000 7fa8c5a10740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c94f985100 con 0x55c94fa32c50 2024-03-28T22:40:20.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.697+0000 7fa8c5a10740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55c94f92dba0 con 0x55c94fa210d0 2024-03-28T22:40:20.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.697+0000 7fa8c5a10740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55c94f9811a0 con 0x55c94fa21db0 2024-03-28T22:40:20.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.698+0000 7fa8bb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c94fa32c50 0x55c94fa35040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.698+0000 7fa8bb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c94fa32c50 0x55c94fa35040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37732/0 (socket says 172.21.15.43:37732) 2024-03-28T22:40:20.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.698+0000 7fa8bb400640 1 -- 172.21.15.43:0/846760196 learned_addr learned my addr 172.21.15.43:0/846760196 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:20.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.698+0000 7fa8bbe00640 1 --2- 172.21.15.43:0/846760196 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c94fa210d0 0x55c94fa21550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.698+0000 7fa8bb400640 1 --2- 172.21.15.43:0/846760196 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c94fa21db0 0x55c94fa20b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.698+0000 7fa8b8c00640 1 --2- 172.21.15.43:0/1361707164 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa8a00a1ef0 0x7fa8a00a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.698+0000 7fa8bbe00640 1 -- 172.21.15.43:0/846760196 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c94fa21db0 msgr2=0x55c94fa20b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:20.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.698+0000 7fa8bbe00640 1 --2- 172.21.15.43:0/846760196 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c94fa21db0 0x55c94fa20b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.698+0000 7fa8bbe00640 1 -- 172.21.15.43:0/846760196 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c94fa32c50 msgr2=0x55c94fa35040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:20.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.698+0000 7fa8bbe00640 1 --2- 172.21.15.43:0/846760196 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c94fa32c50 0x55c94fa35040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.698+0000 7fa8bbe00640 1 -- 172.21.15.43:0/846760196 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c94f95e1a0 con 0x55c94fa210d0 2024-03-28T22:40:20.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.698+0000 7fa8b8c00640 1 -- 172.21.15.43:0/1361707164 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fa8b40ceb70 con 0x7fa8bc10de80 2024-03-28T22:40:20.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.699+0000 7fa8c0e00640 1 --2- 172.21.15.43:0/1361707164 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa8a00a1ef0 0x7fa8a00a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.699+0000 7fa8bbe00640 1 --2- 172.21.15.43:0/846760196 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c94fa210d0 0x55c94fa21550 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac02c210 tx=0x7fa8ac009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=383fb367db38d10f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:20.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.699+0000 7fa8b1600640 1 -- 172.21.15.43:0/846760196 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac043070 con 0x55c94fa210d0 2024-03-28T22:40:20.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.699+0000 7fa8b1600640 1 -- 172.21.15.43:0/846760196 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8ac02c6f0 con 0x55c94fa210d0 2024-03-28T22:40:20.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.699+0000 7fa8b1600640 1 -- 172.21.15.43:0/846760196 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac0049e0 con 0x55c94fa210d0 2024-03-28T22:40:20.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.700+0000 7fa8c5a10740 1 -- 172.21.15.43:0/846760196 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c94fa210d0 msgr2=0x55c94fa21550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:20.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.700+0000 7fa8c5a10740 1 --2- 172.21.15.43:0/846760196 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c94fa210d0 0x55c94fa21550 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac02c210 tx=0x7fa8ac009560 comp rx=0 tx=0).stop 2024-03-28T22:40:20.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.700+0000 7fa8c0e00640 1 --2- 172.21.15.43:0/1361707164 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa8a00a1ef0 0x7fa8a00a43b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa8b40a33d0 tx=0x7fa8b4005f70 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:20.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.700+0000 7fa8c5a10740 1 -- 172.21.15.43:0/846760196 shutdown_connections 2024-03-28T22:40:20.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.700+0000 7fa8c5a10740 1 --2- 172.21.15.43:0/846760196 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c94fa210d0 0x55c94fa21550 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.700+0000 7fa8c5a10740 1 --2- 172.21.15.43:0/846760196 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c94fa32c50 0x55c94fa35040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.700+0000 7fa8c5a10740 1 --2- 172.21.15.43:0/846760196 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c94fa21db0 0x55c94fa20b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.700+0000 7fa8c5a10740 1 -- 172.21.15.43:0/846760196 >> 172.21.15.43:0/846760196 conn(0x55c94f9e7a10 msgr2=0x55c94fa1bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:20.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.701+0000 7fa8c5a10740 1 -- 172.21.15.43:0/846760196 shutdown_connections 2024-03-28T22:40:20.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.701+0000 7fa8c5a10740 1 -- 172.21.15.43:0/846760196 wait complete. 2024-03-28T22:40:20.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.701+0000 7fa8c5a10740 1 -- start start 2024-03-28T22:40:20.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.701+0000 7fa8c5a10740 1 Processor -- start 2024-03-28T22:40:20.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.702+0000 7fa8c5a10740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c94fa210d0 0x55c94fa2fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.702+0000 7fa8c5a10740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c94fa21db0 0x55c94fa300e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.702+0000 7fa8c5a10740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c94fa32c50 0x55c94fa30620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.702+0000 7fa8c5a10740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c94fa35a60 con 0x55c94fa21db0 2024-03-28T22:40:20.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.702+0000 7fa8c5a10740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55c94f923250 con 0x55c94fa210d0 2024-03-28T22:40:20.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.702+0000 7fa8c5a10740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55c94fa35be0 con 0x55c94fa32c50 2024-03-28T22:40:20.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.702+0000 7fa8c5a10740 10 client.0 _ll_register_callbacks cb 0x55c94f698be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:20.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.702+0000 7fa8bb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c94fa210d0 0x55c94fa2fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.702+0000 7fa8bb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c94fa210d0 0x55c94fa2fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45146/0 (socket says 172.21.15.43:45146) 2024-03-28T22:40:20.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.702+0000 7fa8bb400640 1 -- 172.21.15.43:0/974299423 learned_addr learned my addr 172.21.15.43:0/974299423 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:20.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.702+0000 7fa8bb400640 1 --2- 172.21.15.43:0/974299423 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c94fa32c50 0x55c94fa30620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.702+0000 7fa8bbe00640 1 --2- 172.21.15.43:0/974299423 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c94fa21db0 0x55c94fa300e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.702+0000 7fa8bb400640 1 -- 172.21.15.43:0/974299423 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c94fa32c50 msgr2=0x55c94fa30620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:20.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.702+0000 7fa8bb400640 1 --2- 172.21.15.43:0/974299423 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c94fa32c50 0x55c94fa30620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.703+0000 7fa8bb400640 1 -- 172.21.15.43:0/974299423 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c94fa21db0 msgr2=0x55c94fa300e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:20.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.703+0000 7fa8bb400640 1 --2- 172.21.15.43:0/974299423 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c94fa21db0 0x55c94fa300e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.703+0000 7fa8bb400640 1 -- 172.21.15.43:0/974299423 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c94f93f960 con 0x55c94fa210d0 2024-03-28T22:40:20.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.703+0000 7fa8bbe00640 1 --2- 172.21.15.43:0/974299423 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c94fa21db0 0x55c94fa300e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:40:20.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.703+0000 7fa8bb400640 1 --2- 172.21.15.43:0/974299423 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c94fa210d0 0x55c94fa2fba0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa8a80139d0 tx=0x7fa8a8005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52d53b718f0f9edf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:20.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.703+0000 7fa8a6000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:40:20.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.703+0000 7fa8a6000640 1 -- 172.21.15.43:0/974299423 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a8004260 con 0x55c94fa210d0 2024-03-28T22:40:20.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.703+0000 7fa8c5a10740 1 -- 172.21.15.43:0/974299423 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8ac02c6f0 con 0x55c94fa210d0 2024-03-28T22:40:20.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.703+0000 7fa8a6000640 1 -- 172.21.15.43:0/974299423 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fa8a8004400 con 0x55c94fa210d0 2024-03-28T22:40:20.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.703+0000 7fa8a6000640 1 -- 172.21.15.43:0/974299423 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa8a8005020 con 0x55c94fa210d0 2024-03-28T22:40:20.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.703+0000 7fa8c5a10740 10 client.26729 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:20.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.703+0000 7fa8c5a10740 1 -- 172.21.15.43:0/974299423 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c94f7637d0 con 0x55c94fa210d0 2024-03-28T22:40:20.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.704+0000 7fa8a6000640 1 -- 172.21.15.43:0/974299423 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8ac02c6f0 con 0x55c94fa210d0 2024-03-28T22:40:20.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.704+0000 7fa8a6000640 1 -- 172.21.15.43:0/974299423 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa8a80047a0 con 0x55c94fa210d0 2024-03-28T22:40:20.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.704+0000 7fa8c5a10740 10 client.26729 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:20.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.704+0000 7fa8c5a10740 10 client.26729 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:40:20.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.704+0000 7fa8c5a10740 20 client.26729 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:20.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.704+0000 7fa8c5a10740 1 --2- 172.21.15.43:0/974299423 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c94fa42f80 0x55c94fa453b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.704+0000 7fa8c5a10740 4 client.26729 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:40:20.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.704+0000 7fa8c5a10740 1 -- 172.21.15.43:0/974299423 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c94f992640 con 0x55c94fa42f80 2024-03-28T22:40:20.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.705+0000 7fa8bbe00640 1 --2- 172.21.15.43:0/974299423 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c94fa42f80 0x55c94fa453b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.705+0000 7fa8bbe00640 1 --2- 172.21.15.43:0/974299423 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c94fa42f80 0x55c94fa453b0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4f60c3d57661b27 server_cookie=ac0971140f9aa970 in_seq=0 out_seq=0 2024-03-28T22:40:20.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.705+0000 7fa8a6000640 10 client.26729 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:20.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.708+0000 7fa8a6000640 1 -- 172.21.15.43:0/974299423 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c94f992640 con 0x55c94fa42f80 2024-03-28T22:40:20.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.708+0000 7fa8a6000640 10 client.26729 handle_command_reply: tid=0 2024-03-28T22:40:20.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.708+0000 7fa8c5a10740 1 client.26729 shutdown 2024-03-28T22:40:20.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.708+0000 7fa8c5a10740 1 -- 172.21.15.43:0/974299423 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c94fa210d0 msgr2=0x55c94fa2fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:20.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.708+0000 7fa8c5a10740 1 --2- 172.21.15.43:0/974299423 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c94fa210d0 0x55c94fa2fba0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa8a80139d0 tx=0x7fa8a8005f90 comp rx=0 tx=0).stop 2024-03-28T22:40:20.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.709+0000 7fa8c5a10740 1 -- 172.21.15.43:0/974299423 shutdown_connections 2024-03-28T22:40:20.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.709+0000 7fa8c5a10740 1 --2- 172.21.15.43:0/974299423 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c94fa32c50 0x55c94fa30620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.709+0000 7fa8c5a10740 1 --2- 172.21.15.43:0/974299423 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c94fa21db0 0x55c94fa300e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.709+0000 7fa8c5a10740 1 --2- 172.21.15.43:0/974299423 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c94fa42f80 0x55c94fa453b0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.709+0000 7fa8c5a10740 1 --2- 172.21.15.43:0/974299423 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c94fa210d0 0x55c94fa2fba0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.709+0000 7fa8c5a10740 1 -- 172.21.15.43:0/974299423 >> 172.21.15.43:0/974299423 conn(0x55c94f9e7a10 msgr2=0x55c94fa1d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:20.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.709+0000 7fa8a6000640 0 client.26729 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:20.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.709+0000 7fa8c5a10740 1 -- 172.21.15.43:0/974299423 shutdown_connections 2024-03-28T22:40:20.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.709+0000 7fa8c5a10740 1 -- 172.21.15.43:0/974299423 wait complete. 2024-03-28T22:40:20.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.709+0000 7fa8c5a10740 20 client.26729 trim_cache size 0 max 16384 2024-03-28T22:40:20.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.732+0000 7fa8c5a10740 1 Processor -- start 2024-03-28T22:40:20.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.732+0000 7fa8c5a10740 1 -- start start 2024-03-28T22:40:20.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.732+0000 7fa8c5a10740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c94fa210d0 0x55c94fa1d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.732+0000 7fa8c5a10740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c94fa21db0 0x55c94fa2f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.732+0000 7fa8c5a10740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c94fa32c50 0x55c94fa3e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.732+0000 7fa8c5a10740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc1ac740 con 0x55c94fa21db0 2024-03-28T22:40:20.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.732+0000 7fa8c5a10740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc1ae280 con 0x55c94fa210d0 2024-03-28T22:40:20.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.732+0000 7fa8c5a10740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc1ace10 con 0x55c94fa32c50 2024-03-28T22:40:20.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.733+0000 7fa8bb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c94fa210d0 0x55c94fa1d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.733+0000 7fa8bb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c94fa210d0 0x55c94fa1d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45158/0 (socket says 172.21.15.43:45158) 2024-03-28T22:40:20.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.733+0000 7fa8bbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c94fa21db0 0x55c94fa2f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.733+0000 7fa8bb400640 1 -- 172.21.15.43:0/4025021612 learned_addr learned my addr 172.21.15.43:0/4025021612 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:20.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.733+0000 7fa8bb400640 1 --2- 172.21.15.43:0/4025021612 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c94fa32c50 0x55c94fa3e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.733+0000 7fa8bbe00640 1 -- 172.21.15.43:0/4025021612 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c94fa32c50 msgr2=0x55c94fa3e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:20.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.733+0000 7fa8bbe00640 1 --2- 172.21.15.43:0/4025021612 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c94fa32c50 0x55c94fa3e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.733+0000 7fa8bbe00640 1 -- 172.21.15.43:0/4025021612 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c94fa210d0 msgr2=0x55c94fa1d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:20.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.733+0000 7fa8bbe00640 1 --2- 172.21.15.43:0/4025021612 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c94fa210d0 0x55c94fa1d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.733+0000 7fa8bbe00640 1 -- 172.21.15.43:0/4025021612 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c94f6993f0 con 0x55c94fa21db0 2024-03-28T22:40:20.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.733+0000 7fa8bb400640 1 --2- 172.21.15.43:0/4025021612 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c94fa210d0 0x55c94fa1d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:20.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.734+0000 7fa8bbe00640 1 --2- 172.21.15.43:0/4025021612 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c94fa21db0 0x55c94fa2f480 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac02c8c0 tx=0x7fa8ac005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3562667ac9179af server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:20.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.734+0000 7fa8b0c00640 1 -- 172.21.15.43:0/4025021612 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac055070 con 0x55c94fa21db0 2024-03-28T22:40:20.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.734+0000 7fa8b0c00640 1 -- 172.21.15.43:0/4025021612 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8ac02c6f0 con 0x55c94fa21db0 2024-03-28T22:40:20.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.734+0000 7fa8b0c00640 1 -- 172.21.15.43:0/4025021612 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac043040 con 0x55c94fa21db0 2024-03-28T22:40:20.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.735+0000 7fa8c5a10740 1 -- 172.21.15.43:0/4025021612 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c94fa21db0 msgr2=0x55c94fa2f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:20.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.735+0000 7fa8c5a10740 1 --2- 172.21.15.43:0/4025021612 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c94fa21db0 0x55c94fa2f480 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac02c8c0 tx=0x7fa8ac005fd0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.735+0000 7fa8c5a10740 1 -- 172.21.15.43:0/4025021612 shutdown_connections 2024-03-28T22:40:20.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.735+0000 7fa8c5a10740 1 --2- 172.21.15.43:0/4025021612 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c94fa32c50 0x55c94fa3e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.735+0000 7fa8c5a10740 1 --2- 172.21.15.43:0/4025021612 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c94fa21db0 0x55c94fa2f480 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.735+0000 7fa8c5a10740 1 --2- 172.21.15.43:0/4025021612 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c94fa210d0 0x55c94fa1d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.735+0000 7fa8c5a10740 1 -- 172.21.15.43:0/4025021612 >> 172.21.15.43:0/4025021612 conn(0x55c94f9e7a10 msgr2=0x55c94fa1a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:20.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.735+0000 7fa8c5a10740 1 -- 172.21.15.43:0/4025021612 shutdown_connections 2024-03-28T22:40:20.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.735+0000 7fa8c5a10740 1 -- 172.21.15.43:0/4025021612 wait complete. 2024-03-28T22:40:20.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.736+0000 7fa8c5a10740 1 -- start start 2024-03-28T22:40:20.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.736+0000 7fa8c5a10740 1 Processor -- start 2024-03-28T22:40:20.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.736+0000 7fa8c5a10740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c94fa210d0 0x55c94fa3b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.736+0000 7fa8c5a10740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c94fa21db0 0x55c94fa2c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.736+0000 7fa8c5a10740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c94fa32c50 0x55c94fa2c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.736+0000 7fa8c5a10740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc1adbb0 con 0x55c94fa21db0 2024-03-28T22:40:20.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.736+0000 7fa8c5a10740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc1ad4e0 con 0x55c94fa210d0 2024-03-28T22:40:20.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.736+0000 7fa8c5a10740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc1aa720 con 0x55c94fa32c50 2024-03-28T22:40:20.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.736+0000 7fa8c5a10740 10 client.0 _ll_register_callbacks cb 0x7fa8a8013050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:20.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.737+0000 7fa8bbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c94fa21db0 0x55c94fa2c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.737+0000 7fa8bb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c94fa210d0 0x55c94fa3b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.737+0000 7fa8bbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c94fa21db0 0x55c94fa2c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37760/0 (socket says 172.21.15.43:37760) 2024-03-28T22:40:20.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.737+0000 7fa8bb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c94fa210d0 0x55c94fa3b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45170/0 (socket says 172.21.15.43:45170) 2024-03-28T22:40:20.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.737+0000 7fa8bbe00640 1 -- 172.21.15.43:0/3673090508 learned_addr learned my addr 172.21.15.43:0/3673090508 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:20.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.737+0000 7fa8bb400640 1 --2- 172.21.15.43:0/3673090508 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c94fa32c50 0x55c94fa2c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.737+0000 7fa8bb400640 1 -- 172.21.15.43:0/3673090508 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c94fa32c50 msgr2=0x55c94fa2c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:20.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.737+0000 7fa8bb400640 1 --2- 172.21.15.43:0/3673090508 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c94fa32c50 0x55c94fa2c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.737+0000 7fa8bb400640 1 -- 172.21.15.43:0/3673090508 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c94fa21db0 msgr2=0x55c94fa2c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:20.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.737+0000 7fa8bb400640 1 --2- 172.21.15.43:0/3673090508 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c94fa21db0 0x55c94fa2c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.737+0000 7fa8bb400640 1 -- 172.21.15.43:0/3673090508 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c94f6590f0 con 0x55c94fa210d0 2024-03-28T22:40:20.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.737+0000 7fa8bb400640 1 --2- 172.21.15.43:0/3673090508 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c94fa210d0 0x55c94fa3b060 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fa8a8001eb0 tx=0x7fa8a8009cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cac737e78f34ffe8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:20.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.737+0000 7fa8a6000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:40:20.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.738+0000 7fa8a6000640 1 -- 172.21.15.43:0/3673090508 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a80051c0 con 0x55c94fa210d0 2024-03-28T22:40:20.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.738+0000 7fa8a6000640 1 -- 172.21.15.43:0/3673090508 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fa8a8005360 con 0x55c94fa210d0 2024-03-28T22:40:20.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.738+0000 7fa8c5a10740 1 -- 172.21.15.43:0/3673090508 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8ac02c6f0 con 0x55c94fa210d0 2024-03-28T22:40:20.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.738+0000 7fa8a6000640 1 -- 172.21.15.43:0/3673090508 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa8a8005560 con 0x55c94fa210d0 2024-03-28T22:40:20.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.738+0000 7fa8c5a10740 10 client.26735 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:20.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.738+0000 7fa8c5a10740 1 -- 172.21.15.43:0/3673090508 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c94faa9020 con 0x55c94fa210d0 2024-03-28T22:40:20.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.738+0000 7fa8a6000640 1 -- 172.21.15.43:0/3673090508 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8ac02c6f0 con 0x55c94fa210d0 2024-03-28T22:40:20.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.739+0000 7fa8a6000640 1 -- 172.21.15.43:0/3673090508 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa8a80035f0 con 0x55c94fa210d0 2024-03-28T22:40:20.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.739+0000 7fa8c5a10740 10 client.26735 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:20.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.739+0000 7fa8c5a10740 10 client.26735 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:40:20.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.739+0000 7fa8c5a10740 20 client.26735 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:20.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.739+0000 7fa8c5a10740 1 --2- 172.21.15.43:0/3673090508 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c94faa9660 0x55c94faaba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.739+0000 7fa8c5a10740 4 client.26735 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:40:20.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.739+0000 7fa8c5a10740 1 -- 172.21.15.43:0/3673090508 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c94f98c990 con 0x55c94faa9660 2024-03-28T22:40:20.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.739+0000 7fa8bbe00640 1 --2- 172.21.15.43:0/3673090508 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c94faa9660 0x55c94faaba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.740+0000 7fa8bbe00640 1 --2- 172.21.15.43:0/3673090508 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c94faa9660 0x55c94faaba80 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d85a7a9256e3d443 server_cookie=bde36c81444671e7 in_seq=0 out_seq=0 2024-03-28T22:40:20.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.740+0000 7fa8a6000640 10 client.26735 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:20.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.740+0000 7fa8a6000640 1 -- 172.21.15.43:0/3673090508 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c94f98c990 con 0x55c94faa9660 2024-03-28T22:40:20.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.740+0000 7fa8a6000640 10 client.26735 handle_command_reply: tid=0 2024-03-28T22:40:20.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.740+0000 7fa8c5a10740 1 client.26735 shutdown 2024-03-28T22:40:20.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.740+0000 7fa8c5a10740 1 -- 172.21.15.43:0/3673090508 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c94fa210d0 msgr2=0x55c94fa3b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:20.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.740+0000 7fa8c5a10740 1 --2- 172.21.15.43:0/3673090508 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c94fa210d0 0x55c94fa3b060 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fa8a8001eb0 tx=0x7fa8a8009cc0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.740+0000 7fa8c5a10740 1 -- 172.21.15.43:0/3673090508 shutdown_connections 2024-03-28T22:40:20.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.740+0000 7fa8c5a10740 1 --2- 172.21.15.43:0/3673090508 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c94fa32c50 0x55c94fa2c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.740+0000 7fa8c5a10740 1 --2- 172.21.15.43:0/3673090508 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c94fa21db0 0x55c94fa2c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.740+0000 7fa8c5a10740 1 --2- 172.21.15.43:0/3673090508 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c94faa9660 0x55c94faaba80 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.740+0000 7fa8c5a10740 1 --2- 172.21.15.43:0/3673090508 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c94fa210d0 0x55c94fa3b060 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.740+0000 7fa8c5a10740 1 -- 172.21.15.43:0/3673090508 >> 172.21.15.43:0/3673090508 conn(0x55c94f9e7a10 msgr2=0x55c94fa1ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:20.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.740+0000 7fa8a6000640 0 client.26735 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:20.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.740+0000 7fa8c5a10740 1 -- 172.21.15.43:0/3673090508 shutdown_connections 2024-03-28T22:40:20.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.740+0000 7fa8c5a10740 1 -- 172.21.15.43:0/3673090508 wait complete. 2024-03-28T22:40:20.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.741+0000 7fa8c5a10740 20 client.26735 trim_cache size 0 max 16384 2024-03-28T22:40:20.742 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:40:20.742 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-03-28T22:40:20.742 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-03-28T22:40:20.742 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:40:20.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.741+0000 7fa8c1800640 1 -- 172.21.15.43:0/1361707164 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa8a00a1ef0 msgr2=0x7fa8a00a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:20.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.741+0000 7fa8c1800640 1 --2- 172.21.15.43:0/1361707164 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa8a00a1ef0 0x7fa8a00a43b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa8b40a33d0 tx=0x7fa8b4005f70 comp rx=0 tx=0).stop 2024-03-28T22:40:20.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.741+0000 7fa8c1800640 1 -- 172.21.15.43:0/1361707164 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8bc10de80 msgr2=0x7fa8bc19e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:20.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.741+0000 7fa8c1800640 1 --2- 172.21.15.43:0/1361707164 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8bc10de80 0x7fa8bc19e950 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4009fd0 tx=0x7fa8b400ef10 comp rx=0 tx=0).stop 2024-03-28T22:40:20.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.741+0000 7fa8c1800640 1 -- 172.21.15.43:0/1361707164 shutdown_connections 2024-03-28T22:40:20.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.741+0000 7fa8c1800640 1 --2- 172.21.15.43:0/1361707164 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa8a00a1ef0 0x7fa8a00a43b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.741+0000 7fa8c1800640 1 --2- 172.21.15.43:0/1361707164 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa8bc074420 0x7fa8bc19a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.741+0000 7fa8c1800640 1 --2- 172.21.15.43:0/1361707164 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa8bc10de80 0x7fa8bc19e950 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.741+0000 7fa8c1800640 1 --2- 172.21.15.43:0/1361707164 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8bc073b00 0x7fa8bc19a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.741+0000 7fa8c1800640 1 -- 172.21.15.43:0/1361707164 >> 172.21.15.43:0/1361707164 conn(0x7fa8bc0fd0f0 msgr2=0x7fa8bc0fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:20.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.742+0000 7fa8c1800640 1 -- 172.21.15.43:0/1361707164 shutdown_connections 2024-03-28T22:40:20.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:20.742+0000 7fa8c1800640 1 -- 172.21.15.43:0/1361707164 wait complete. 2024-03-28T22:40:20.758 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:40:20.759 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:40:20.766 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.768+0000 7fee4bc00640 1 -- 172.21.15.43:0/474095490 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee44108c90 msgr2=0x7fee441090f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:20.766 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.768+0000 7fee4bc00640 1 --2- 172.21.15.43:0/474095490 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee44108c90 0x7fee441090f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fee30009900 tx=0x7fee3002bc30 comp rx=0 tx=0).stop 2024-03-28T22:40:20.767 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.769+0000 7fee4bc00640 1 -- 172.21.15.43:0/474095490 shutdown_connections 2024-03-28T22:40:20.767 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.769+0000 7fee4bc00640 1 --2- 172.21.15.43:0/474095490 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fee441167e0 0x7fee44118bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.767 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.769+0000 7fee4bc00640 1 --2- 172.21.15.43:0/474095490 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee44108c90 0x7fee441090f0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.767 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.769+0000 7fee4bc00640 1 --2- 172.21.15.43:0/474095490 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee4410be90 0x7fee44108750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.767 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.769+0000 7fee4bc00640 1 -- 172.21.15.43:0/474095490 >> 172.21.15.43:0/474095490 conn(0x7fee4406cc80 msgr2=0x7fee4406d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:20.767 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.769+0000 7fee4bc00640 1 -- 172.21.15.43:0/474095490 shutdown_connections 2024-03-28T22:40:20.767 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.769+0000 7fee4bc00640 1 -- 172.21.15.43:0/474095490 wait complete. 2024-03-28T22:40:20.767 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.770+0000 7fee4bc00640 1 Processor -- start 2024-03-28T22:40:20.768 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.770+0000 7fee4bc00640 1 -- start start 2024-03-28T22:40:20.769 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.770+0000 7fee4bc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee44108c90 0x7fee441a86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.769 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.770+0000 7fee4bc00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee4410be90 0x7fee441a8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.769 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.770+0000 7fee4bc00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fee441167e0 0x7fee441a2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.770+0000 7fee4bc00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fee4411c5a0 con 0x7fee44108c90 2024-03-28T22:40:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.770+0000 7fee4bc00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fee4411c420 con 0x7fee441167e0 2024-03-28T22:40:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.770+0000 7fee4bc00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fee4411c720 con 0x7fee4410be90 2024-03-28T22:40:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.771+0000 7fee43e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee4410be90 0x7fee441a8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.771+0000 7fee49c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fee441167e0 0x7fee441a2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.771+0000 7fee49200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee44108c90 0x7fee441a86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.771+0000 7fee43e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee4410be90 0x7fee441a8bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:48034/0 (socket says 172.21.15.43:48034) 2024-03-28T22:40:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.771+0000 7fee43e00640 1 -- 172.21.15.43:0/2541385631 learned_addr learned my addr 172.21.15.43:0/2541385631 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.771+0000 7fee49c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fee441167e0 0x7fee441a2820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45192/0 (socket says 172.21.15.43:45192) 2024-03-28T22:40:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.771+0000 7fee43e00640 1 -- 172.21.15.43:0/2541385631 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fee441167e0 msgr2=0x7fee441a2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.771+0000 7fee43e00640 1 --2- 172.21.15.43:0/2541385631 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fee441167e0 0x7fee441a2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.771+0000 7fee43e00640 1 -- 172.21.15.43:0/2541385631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee44108c90 msgr2=0x7fee441a86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.771+0000 7fee43e00640 1 --2- 172.21.15.43:0/2541385631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee44108c90 0x7fee441a86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.771+0000 7fee43e00640 1 -- 172.21.15.43:0/2541385631 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee441a2ff0 con 0x7fee4410be90 2024-03-28T22:40:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.771+0000 7fee43e00640 1 --2- 172.21.15.43:0/2541385631 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee4410be90 0x7fee441a8bf0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fee3002c240 tx=0x7fee300042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.772+0000 7fee41600640 1 -- 172.21.15.43:0/2541385631 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee30043020 con 0x7fee4410be90 2024-03-28T22:40:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.772+0000 7fee41600640 1 -- 172.21.15.43:0/2541385631 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee300054e0 con 0x7fee4410be90 2024-03-28T22:40:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.772+0000 7fee4bc00640 1 -- 172.21.15.43:0/2541385631 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee441a3280 con 0x7fee4410be90 2024-03-28T22:40:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.772+0000 7fee41600640 1 -- 172.21.15.43:0/2541385631 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee30003b30 con 0x7fee4410be90 2024-03-28T22:40:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.772+0000 7fee4bc00640 1 -- 172.21.15.43:0/2541385631 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee441af500 con 0x7fee4410be90 2024-03-28T22:40:20.773 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.775+0000 7fee4e9c6740 1 Processor -- start 2024-03-28T22:40:20.773 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.776+0000 7fee4e9c6740 1 -- start start 2024-03-28T22:40:20.773 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.776+0000 7fee4e9c6740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e8b01f33a0 0x55e8b01f3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.773 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.776+0000 7fee4e9c6740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e8b01f3cc0 0x55e8b01fc550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.774 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.776+0000 7fee4e9c6740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8b01fca90 0x55e8b01fee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.774 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.776+0000 7fee4e9c6740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e8b0156640 con 0x55e8b01f3cc0 2024-03-28T22:40:20.774 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.776+0000 7fee4e9c6740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55e8b00ff810 con 0x55e8b01fca90 2024-03-28T22:40:20.774 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.776+0000 7fee4e9c6740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55e8b01526e0 con 0x55e8b01f33a0 2024-03-28T22:40:20.775 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.776+0000 7fee49c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e8b01f33a0 0x55e8b01f3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.775 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.776+0000 7fee49200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e8b01f3cc0 0x55e8b01fc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.775 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.776+0000 7fee49c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e8b01f33a0 0x55e8b01f3780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:48038/0 (socket says 172.21.15.43:48038) 2024-03-28T22:40:20.775 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.776+0000 7fee49c00640 1 -- 172.21.15.43:0/1145062887 learned_addr learned my addr 172.21.15.43:0/1145062887 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:20.775 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.777+0000 7fee41600640 1 -- 172.21.15.43:0/2541385631 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fee30046020 con 0x7fee4410be90 2024-03-28T22:40:20.775 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.777+0000 7fee49c00640 1 -- 172.21.15.43:0/1145062887 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8b01fca90 msgr2=0x55e8b01fee80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:40:20.776 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.777+0000 7fee49c00640 1 --2- 172.21.15.43:0/1145062887 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8b01fca90 0x55e8b01fee80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.776 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.777+0000 7fee49c00640 1 -- 172.21.15.43:0/1145062887 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e8b01f3cc0 msgr2=0x55e8b01fc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:20.776 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.777+0000 7fee49c00640 1 --2- 172.21.15.43:0/1145062887 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e8b01f3cc0 0x55e8b01fc550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.776 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.777+0000 7fee49c00640 1 -- 172.21.15.43:0/1145062887 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8afe6b210 con 0x55e8b01f33a0 2024-03-28T22:40:20.776 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.777+0000 7fee49c00640 1 --2- 172.21.15.43:0/1145062887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e8b01f33a0 0x55e8b01f3780 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fee3c009f30 tx=0x7fee3c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eea5fdb46cd27a9b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:20.776 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.777+0000 7fee36000640 1 -- 172.21.15.43:0/1145062887 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3c004d20 con 0x55e8b01f33a0 2024-03-28T22:40:20.776 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.777+0000 7fee36000640 1 -- 172.21.15.43:0/1145062887 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee3c004050 con 0x55e8b01f33a0 2024-03-28T22:40:20.776 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.777+0000 7fee36000640 1 -- 172.21.15.43:0/1145062887 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3c005260 con 0x55e8b01f33a0 2024-03-28T22:40:20.776 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.778+0000 7fee4e9c6740 1 -- 172.21.15.43:0/1145062887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e8b01f33a0 msgr2=0x55e8b01f3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:20.776 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.778+0000 7fee4e9c6740 1 --2- 172.21.15.43:0/1145062887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e8b01f33a0 0x55e8b01f3780 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fee3c009f30 tx=0x7fee3c0118d0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.780 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.778+0000 7fee41600640 1 --2- 172.21.15.43:0/2541385631 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fee2c0a2010 0x7fee2c0a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.780 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.778+0000 7fee4e9c6740 1 -- 172.21.15.43:0/1145062887 shutdown_connections 2024-03-28T22:40:20.780 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.778+0000 7fee4e9c6740 1 --2- 172.21.15.43:0/1145062887 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8b01fca90 0x55e8b01fee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.780 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.778+0000 7fee4e9c6740 1 --2- 172.21.15.43:0/1145062887 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e8b01f3cc0 0x55e8b01fc550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.780 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.778+0000 7fee4e9c6740 1 --2- 172.21.15.43:0/1145062887 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e8b01f33a0 0x55e8b01f3780 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.780 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.778+0000 7fee4e9c6740 1 -- 172.21.15.43:0/1145062887 >> 172.21.15.43:0/1145062887 conn(0x55e8b01edcc0 msgr2=0x55e8b01ee0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:20.781 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.778+0000 7fee41600640 1 -- 172.21.15.43:0/2541385631 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fee300f0310 con 0x7fee4410be90 2024-03-28T22:40:20.781 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.779+0000 7fee4e9c6740 1 -- 172.21.15.43:0/1145062887 shutdown_connections 2024-03-28T22:40:20.781 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.779+0000 7fee49c00640 1 --2- 172.21.15.43:0/2541385631 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fee2c0a2010 0x7fee2c0a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.781 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.779+0000 7fee4e9c6740 1 -- 172.21.15.43:0/1145062887 wait complete. 2024-03-28T22:40:20.781 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.779+0000 7fee4e9c6740 1 -- start start 2024-03-28T22:40:20.781 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.780+0000 7fee49c00640 1 --2- 172.21.15.43:0/2541385631 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fee2c0a2010 0x7fee2c0a44d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fee3c009d40 tx=0x7fee3c0096f0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:20.781 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.780+0000 7fee4e9c6740 1 Processor -- start 2024-03-28T22:40:20.781 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.780+0000 7fee4e9c6740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8b01f33a0 0x55e8b01f9b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.781 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.780+0000 7fee4e9c6740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e8b01f3cc0 0x55e8b01fa0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.781 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.780+0000 7fee4e9c6740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e8b01fca90 0x55e8b01fa5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.781 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.780+0000 7fee4e9c6740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e8b01ff6c0 con 0x55e8b01f3cc0 2024-03-28T22:40:20.781 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.780+0000 7fee4e9c6740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55e8b00f4d00 con 0x55e8b01f33a0 2024-03-28T22:40:20.781 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.780+0000 7fee4e9c6740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55e8b01ff840 con 0x55e8b01fca90 2024-03-28T22:40:20.781 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.780+0000 7fee4e9c6740 10 client.0 _ll_register_callbacks cb 0x55e8b00df490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:20.782 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.780+0000 7fee49200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8b01f33a0 0x55e8b01f9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.782 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.780+0000 7fee49c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e8b01fca90 0x55e8b01fa5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.782 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.780+0000 7fee49200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8b01f33a0 0x55e8b01f9b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45196/0 (socket says 172.21.15.43:45196) 2024-03-28T22:40:20.782 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.780+0000 7fee49200640 1 -- 172.21.15.43:0/2383177039 learned_addr learned my addr 172.21.15.43:0/2383177039 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:20.782 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.781+0000 7fee43e00640 1 --2- 172.21.15.43:0/2383177039 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e8b01f3cc0 0x55e8b01fa0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.782 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.781+0000 7fee49200640 1 -- 172.21.15.43:0/2383177039 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e8b01fca90 msgr2=0x55e8b01fa5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:20.782 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.781+0000 7fee49200640 1 --2- 172.21.15.43:0/2383177039 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e8b01fca90 0x55e8b01fa5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.782 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.781+0000 7fee49200640 1 -- 172.21.15.43:0/2383177039 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e8b01f3cc0 msgr2=0x55e8b01fa0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:20.782 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.781+0000 7fee49200640 1 --2- 172.21.15.43:0/2383177039 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e8b01f3cc0 0x55e8b01fa0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.782 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.781+0000 7fee49200640 1 -- 172.21.15.43:0/2383177039 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8aff37620 con 0x55e8b01f33a0 2024-03-28T22:40:20.782 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.781+0000 7fee43e00640 1 --2- 172.21.15.43:0/2383177039 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e8b01f3cc0 0x55e8b01fa0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:40:20.782 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.781+0000 7fee49c00640 1 --2- 172.21.15.43:0/2383177039 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e8b01fca90 0x55e8b01fa5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:20.782 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.781+0000 7fee49200640 1 --2- 172.21.15.43:0/2383177039 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8b01f33a0 0x55e8b01f9b60 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fee380126e0 tx=0x7fee38012ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d497d98e283bf79a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:20.782 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.781+0000 7fee2aa00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:40:20.782 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.781+0000 7fee2aa00640 1 -- 172.21.15.43:0/2383177039 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee38012190 con 0x55e8b01f33a0 2024-03-28T22:40:20.783 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.781+0000 7fee4e9c6740 1 -- 172.21.15.43:0/2383177039 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee3c004050 con 0x55e8b01f33a0 2024-03-28T22:40:20.783 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.781+0000 7fee2aa00640 1 -- 172.21.15.43:0/2383177039 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fee38004020 con 0x55e8b01f33a0 2024-03-28T22:40:20.783 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.782+0000 7fee2aa00640 1 -- 172.21.15.43:0/2383177039 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fee38025050 con 0x55e8b01f33a0 2024-03-28T22:40:20.783 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.782+0000 7fee4e9c6740 10 client.26744 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:20.783 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.782+0000 7fee4e9c6740 1 -- 172.21.15.43:0/2383177039 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e8b020b6d0 con 0x55e8b01f33a0 2024-03-28T22:40:20.783 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.782+0000 7fee2aa00640 1 -- 172.21.15.43:0/2383177039 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee3c004050 con 0x55e8b01f33a0 2024-03-28T22:40:20.783 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.782+0000 7fee2aa00640 1 -- 172.21.15.43:0/2383177039 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fee38003790 con 0x55e8b01f33a0 2024-03-28T22:40:20.783 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.782+0000 7fee4e9c6740 10 client.26744 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:20.783 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.782+0000 7fee4e9c6740 10 client.26744 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:40:20.783 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.782+0000 7fee4e9c6740 20 client.26744 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:20.783 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.782+0000 7fee4e9c6740 1 --2- 172.21.15.43:0/2383177039 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55e8b020cb40 0x55e8b020ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.783 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.782+0000 7fee4e9c6740 4 client.26744 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:40:20.783 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.782+0000 7fee4e9c6740 1 -- 172.21.15.43:0/2383177039 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e8b016a030 con 0x55e8b020cb40 2024-03-28T22:40:20.783 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.783+0000 7fee43e00640 1 --2- 172.21.15.43:0/2383177039 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55e8b020cb40 0x55e8b020ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.783+0000 7fee43e00640 1 --2- 172.21.15.43:0/2383177039 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55e8b020cb40 0x55e8b020ef30 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3de0e009f5e19ed server_cookie=f59a3c17eb597500 in_seq=0 out_seq=0 2024-03-28T22:40:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.783+0000 7fee2aa00640 10 client.26744 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.784+0000 7fee2aa00640 1 -- 172.21.15.43:0/2383177039 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e8b016a030 con 0x55e8b020cb40 2024-03-28T22:40:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.784+0000 7fee2aa00640 10 client.26744 handle_command_reply: tid=0 2024-03-28T22:40:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.784+0000 7fee4e9c6740 1 client.26744 shutdown 2024-03-28T22:40:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.784+0000 7fee4e9c6740 1 -- 172.21.15.43:0/2383177039 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8b01f33a0 msgr2=0x55e8b01f9b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.784+0000 7fee4e9c6740 1 --2- 172.21.15.43:0/2383177039 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8b01f33a0 0x55e8b01f9b60 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fee380126e0 tx=0x7fee38012ba0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.784+0000 7fee4e9c6740 1 -- 172.21.15.43:0/2383177039 shutdown_connections 2024-03-28T22:40:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.784+0000 7fee4e9c6740 1 --2- 172.21.15.43:0/2383177039 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e8b01fca90 0x55e8b01fa5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.784+0000 7fee4e9c6740 1 --2- 172.21.15.43:0/2383177039 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e8b01f3cc0 0x55e8b01fa0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.784+0000 7fee4e9c6740 1 --2- 172.21.15.43:0/2383177039 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55e8b020cb40 0x55e8b020ef30 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.784+0000 7fee4e9c6740 1 --2- 172.21.15.43:0/2383177039 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8b01f33a0 0x55e8b01f9b60 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.784+0000 7fee2aa00640 0 client.26744 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:20.786 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.784+0000 7fee4e9c6740 1 -- 172.21.15.43:0/2383177039 >> 172.21.15.43:0/2383177039 conn(0x55e8b01edcc0 msgr2=0x55e8b01efa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:20.786 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.784+0000 7fee4e9c6740 1 -- 172.21.15.43:0/2383177039 shutdown_connections 2024-03-28T22:40:20.786 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.784+0000 7fee4e9c6740 1 -- 172.21.15.43:0/2383177039 wait complete. 2024-03-28T22:40:20.786 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.784+0000 7fee4e9c6740 20 client.26744 trim_cache size 0 max 16384 2024-03-28T22:40:20.803 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.804+0000 7fee4e9c6740 1 Processor -- start 2024-03-28T22:40:20.803 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.804+0000 7fee4e9c6740 1 -- start start 2024-03-28T22:40:20.803 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.805+0000 7fee4e9c6740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e8b01f33a0 0x55e8b01fc290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.803 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.805+0000 7fee4e9c6740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8b01f3cc0 0x55e8b0208460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.803 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.805+0000 7fee4e9c6740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e8b01fca90 0x55e8b0214800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.803 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.805+0000 7fee4e9c6740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fee441b94f0 con 0x55e8b01fca90 2024-03-28T22:40:20.803 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.805+0000 7fee4e9c6740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fee441bb030 con 0x55e8b01f3cc0 2024-03-28T22:40:20.803 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.805+0000 7fee4e9c6740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fee441b9bc0 con 0x55e8b01f33a0 2024-03-28T22:40:20.803 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.805+0000 7fee49200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e8b01f33a0 0x55e8b01fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.803 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.805+0000 7fee43e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8b01f3cc0 0x55e8b0208460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.804 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.805+0000 7fee49200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e8b01f33a0 0x55e8b01fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:48058/0 (socket says 172.21.15.43:48058) 2024-03-28T22:40:20.804 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.805+0000 7fee49200640 1 -- 172.21.15.43:0/1294444748 learned_addr learned my addr 172.21.15.43:0/1294444748 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:20.804 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.805+0000 7fee49c00640 1 --2- 172.21.15.43:0/1294444748 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e8b01fca90 0x55e8b0214800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.804 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.805+0000 7fee49200640 1 -- 172.21.15.43:0/1294444748 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8b01f3cc0 msgr2=0x55e8b0208460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:20.804 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.805+0000 7fee49200640 1 --2- 172.21.15.43:0/1294444748 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8b01f3cc0 0x55e8b0208460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.804 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.805+0000 7fee49200640 1 -- 172.21.15.43:0/1294444748 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e8b01fca90 msgr2=0x55e8b0214800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:20.804 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.805+0000 7fee49200640 1 --2- 172.21.15.43:0/1294444748 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e8b01fca90 0x55e8b0214800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.804 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.805+0000 7fee49200640 1 -- 172.21.15.43:0/1294444748 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8b01f80f0 con 0x55e8b01f33a0 2024-03-28T22:40:20.804 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.805+0000 7fee49c00640 1 --2- 172.21.15.43:0/1294444748 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e8b01fca90 0x55e8b0214800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:40:20.805 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.806+0000 7fee49200640 1 --2- 172.21.15.43:0/1294444748 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e8b01f33a0 0x55e8b01fc290 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fee38003d50 tx=0x7fee38009f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5821794b1abfd1ad server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:20.805 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.806+0000 7fee35600640 1 -- 172.21.15.43:0/1294444748 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee38012190 con 0x55e8b01f33a0 2024-03-28T22:40:20.806 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.806+0000 7fee35600640 1 -- 172.21.15.43:0/1294444748 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee3800f920 con 0x55e8b01f33a0 2024-03-28T22:40:20.806 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.806+0000 7fee35600640 1 -- 172.21.15.43:0/1294444748 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee38005450 con 0x55e8b01f33a0 2024-03-28T22:40:20.806 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.807+0000 7fee4e9c6740 1 -- 172.21.15.43:0/1294444748 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e8b01f33a0 msgr2=0x55e8b01fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:20.806 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.807+0000 7fee4e9c6740 1 --2- 172.21.15.43:0/1294444748 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e8b01f33a0 0x55e8b01fc290 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fee38003d50 tx=0x7fee38009f00 comp rx=0 tx=0).stop 2024-03-28T22:40:20.806 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.807+0000 7fee4e9c6740 1 -- 172.21.15.43:0/1294444748 shutdown_connections 2024-03-28T22:40:20.806 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.807+0000 7fee4e9c6740 1 --2- 172.21.15.43:0/1294444748 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8b01f3cc0 0x55e8b0208460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.806 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.807+0000 7fee4e9c6740 1 --2- 172.21.15.43:0/1294444748 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e8b01fca90 0x55e8b0214800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.806 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.807+0000 7fee4e9c6740 1 --2- 172.21.15.43:0/1294444748 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e8b01f33a0 0x55e8b01fc290 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.806 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.807+0000 7fee4e9c6740 1 -- 172.21.15.43:0/1294444748 >> 172.21.15.43:0/1294444748 conn(0x55e8b01edcc0 msgr2=0x55e8b01f4f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:20.806 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.807+0000 7fee4e9c6740 1 -- 172.21.15.43:0/1294444748 shutdown_connections 2024-03-28T22:40:20.806 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.807+0000 7fee4e9c6740 1 -- 172.21.15.43:0/1294444748 wait complete. 2024-03-28T22:40:20.806 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.808+0000 7fee4e9c6740 1 -- start start 2024-03-28T22:40:20.806 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.808+0000 7fee4e9c6740 1 Processor -- start 2024-03-28T22:40:20.809 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.809+0000 7fee4e9c6740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e8b01f33a0 0x55e8b01f6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.809 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.809+0000 7fee4e9c6740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8b01f3cc0 0x55e8b01f6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.809 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.809+0000 7fee4e9c6740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e8b01fca90 0x55e8b01f70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.809 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.809+0000 7fee4e9c6740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fee441ba960 con 0x55e8b01fca90 2024-03-28T22:40:20.809 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.809+0000 7fee4e9c6740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fee441ba290 con 0x55e8b01f3cc0 2024-03-28T22:40:20.809 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.809+0000 7fee4e9c6740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fee441b74d0 con 0x55e8b01f33a0 2024-03-28T22:40:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.809+0000 7fee4e9c6740 10 client.0 _ll_register_callbacks cb 0x7fee38009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.809+0000 7fee49200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e8b01f33a0 0x55e8b01f6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.809+0000 7fee49200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e8b01f33a0 0x55e8b01f6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:48060/0 (socket says 172.21.15.43:48060) 2024-03-28T22:40:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.809+0000 7fee49200640 1 -- 172.21.15.43:0/162694850 learned_addr learned my addr 172.21.15.43:0/162694850 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.809+0000 7fee43e00640 1 --2- 172.21.15.43:0/162694850 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8b01f3cc0 0x55e8b01f6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.809+0000 7fee49c00640 1 --2- 172.21.15.43:0/162694850 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e8b01fca90 0x55e8b01f70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.809+0000 7fee49200640 1 -- 172.21.15.43:0/162694850 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8b01f3cc0 msgr2=0x55e8b01f6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.809+0000 7fee49200640 1 --2- 172.21.15.43:0/162694850 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8b01f3cc0 0x55e8b01f6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.809+0000 7fee49200640 1 -- 172.21.15.43:0/162694850 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e8b01fca90 msgr2=0x55e8b01f70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.809+0000 7fee49200640 1 --2- 172.21.15.43:0/162694850 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e8b01fca90 0x55e8b01f70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.809+0000 7fee49200640 1 -- 172.21.15.43:0/162694850 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8b0271e30 con 0x55e8b01f33a0 2024-03-28T22:40:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.810+0000 7fee49200640 1 --2- 172.21.15.43:0/162694850 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e8b01f33a0 0x55e8b01f6670 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fee38005fe0 tx=0x7fee38012190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=328c60e3f229650e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.810+0000 7fee2aa00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:40:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.810+0000 7fee2aa00640 1 -- 172.21.15.43:0/162694850 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee380055e0 con 0x55e8b01f33a0 2024-03-28T22:40:20.811 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.810+0000 7fee4e9c6740 1 -- 172.21.15.43:0/162694850 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee3800f920 con 0x55e8b01f33a0 2024-03-28T22:40:20.811 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.810+0000 7fee2aa00640 1 -- 172.21.15.43:0/162694850 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fee38018040 con 0x55e8b01f33a0 2024-03-28T22:40:20.811 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.810+0000 7fee2aa00640 1 -- 172.21.15.43:0/162694850 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fee3803b050 con 0x55e8b01f33a0 2024-03-28T22:40:20.811 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.810+0000 7fee4e9c6740 10 client.26698 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:20.811 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.810+0000 7fee4e9c6740 1 -- 172.21.15.43:0/162694850 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e8b0273040 con 0x55e8b01f33a0 2024-03-28T22:40:20.811 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.810+0000 7fee2aa00640 1 -- 172.21.15.43:0/162694850 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee3800f920 con 0x55e8b01f33a0 2024-03-28T22:40:20.811 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.810+0000 7fee2aa00640 1 -- 172.21.15.43:0/162694850 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fee3801c850 con 0x55e8b01f33a0 2024-03-28T22:40:20.811 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.811+0000 7fee4e9c6740 10 client.26698 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:20.811 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.811+0000 7fee4e9c6740 10 client.26698 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:40:20.812 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.811+0000 7fee4e9c6740 20 client.26698 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:20.812 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.811+0000 7fee4e9c6740 1 --2- 172.21.15.43:0/162694850 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55e8b0273670 0x55e8b0275a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:20.812 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.811+0000 7fee4e9c6740 4 client.26698 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T22:40:20.812 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.811+0000 7fee4e9c6740 1 -- 172.21.15.43:0/162694850 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e8b0163b80 con 0x55e8b0273670 2024-03-28T22:40:20.812 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.811+0000 7fee43e00640 1 --2- 172.21.15.43:0/162694850 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55e8b0273670 0x55e8b0275a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:20.813 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.811+0000 7fee43e00640 1 --2- 172.21.15.43:0/162694850 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55e8b0273670 0x55e8b0275a90 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=593f4d8fb0fe7dc5 server_cookie=92cb623e0212820a in_seq=0 out_seq=0 2024-03-28T22:40:20.813 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:20.811+0000 7fee2aa00640 10 client.26698 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:20.829 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:20 smithi043 ceph-mon[31823]: pgmap v99: 129 pgs: 129 active+clean; 49 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.2 MiB/s wr, 262 op/s 2024-03-28T22:40:21.030 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:21.032+0000 7fee2aa00640 1 -- 172.21.15.43:0/162694850 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e8b0163b80 con 0x55e8b0273670 2024-03-28T22:40:21.030 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:21.032+0000 7fee2aa00640 10 client.26698 handle_command_reply: tid=0 2024-03-28T22:40:21.030 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:21.032+0000 7fee4e9c6740 1 client.26698 shutdown 2024-03-28T22:40:21.032 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:21.034+0000 7fee4e9c6740 1 -- 172.21.15.43:0/162694850 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e8b01f33a0 msgr2=0x55e8b01f6670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:21.033 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:21.034+0000 7fee4e9c6740 1 --2- 172.21.15.43:0/162694850 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e8b01f33a0 0x55e8b01f6670 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fee38005fe0 tx=0x7fee38012190 comp rx=0 tx=0).stop 2024-03-28T22:40:21.033 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:21.034+0000 7fee4e9c6740 1 -- 172.21.15.43:0/162694850 shutdown_connections 2024-03-28T22:40:21.033 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:21.034+0000 7fee4e9c6740 1 --2- 172.21.15.43:0/162694850 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55e8b0273670 0x55e8b0275a90 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.033 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:21.034+0000 7fee4e9c6740 1 --2- 172.21.15.43:0/162694850 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8b01f3cc0 0x55e8b01f6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.033 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:21.034+0000 7fee4e9c6740 1 --2- 172.21.15.43:0/162694850 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e8b01fca90 0x55e8b01f70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.033 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:21.034+0000 7fee4e9c6740 1 --2- 172.21.15.43:0/162694850 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e8b01f33a0 0x55e8b01f6670 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.033 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:21.034+0000 7fee4e9c6740 1 -- 172.21.15.43:0/162694850 >> 172.21.15.43:0/162694850 conn(0x55e8b01edcc0 msgr2=0x55e8b01ec3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:21.033 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:21.034+0000 7fee2aa00640 0 client.26698 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:21.033 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:21.034+0000 7fee4e9c6740 1 -- 172.21.15.43:0/162694850 shutdown_connections 2024-03-28T22:40:21.033 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:21.034+0000 7fee4e9c6740 1 -- 172.21.15.43:0/162694850 wait complete. 2024-03-28T22:40:21.033 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:21.035+0000 7fee4e9c6740 20 client.26698 trim_cache size 0 max 16384 2024-03-28T22:40:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:{ 2024-03-28T22:40:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "message": "", 2024-03-28T22:40:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "return_code": 0 2024-03-28T22:40:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:} 2024-03-28T22:40:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:21.035+0000 7fee4bc00640 1 -- 172.21.15.43:0/2541385631 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fee2c0a2010 msgr2=0x7fee2c0a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:21.035 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:21.035+0000 7fee4bc00640 1 --2- 172.21.15.43:0/2541385631 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fee2c0a2010 0x7fee2c0a44d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fee3c009d40 tx=0x7fee3c0096f0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.035 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:21.035+0000 7fee4bc00640 1 -- 172.21.15.43:0/2541385631 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee4410be90 msgr2=0x7fee441a8bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:21.035 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:21.035+0000 7fee4bc00640 1 --2- 172.21.15.43:0/2541385631 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee4410be90 0x7fee441a8bf0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fee3002c240 tx=0x7fee300042f0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.035 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:21.035+0000 7fee4bc00640 1 -- 172.21.15.43:0/2541385631 shutdown_connections 2024-03-28T22:40:21.035 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:21.035+0000 7fee4bc00640 1 --2- 172.21.15.43:0/2541385631 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fee2c0a2010 0x7fee2c0a44d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.035 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:21.035+0000 7fee4bc00640 1 --2- 172.21.15.43:0/2541385631 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fee441167e0 0x7fee441a2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.035 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:21.035+0000 7fee4bc00640 1 --2- 172.21.15.43:0/2541385631 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee4410be90 0x7fee441a8bf0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.035 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:21.035+0000 7fee4bc00640 1 --2- 172.21.15.43:0/2541385631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee44108c90 0x7fee441a86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.035 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:21.035+0000 7fee4bc00640 1 -- 172.21.15.43:0/2541385631 >> 172.21.15.43:0/2541385631 conn(0x7fee4406cc80 msgr2=0x7fee44116be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:21.035 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:21.036+0000 7fee4bc00640 1 -- 172.21.15.43:0/2541385631 shutdown_connections 2024-03-28T22:40:21.035 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:21.036+0000 7fee4bc00640 1 -- 172.21.15.43:0/2541385631 wait complete. 2024-03-28T22:40:21.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.043+0000 7fc0c2e00640 1 -- 172.21.15.43:0/3916900262 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0bc073b80 msgr2=0x7fc0bc073f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:21.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.043+0000 7fc0c2e00640 1 --2- 172.21.15.43:0/3916900262 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0bc073b80 0x7fc0bc073f60 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac009900 tx=0x7fc0ac02bcc0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.043+0000 7fc0c2e00640 1 -- 172.21.15.43:0/3916900262 shutdown_connections 2024-03-28T22:40:21.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.043+0000 7fc0c2e00640 1 --2- 172.21.15.43:0/3916900262 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0bc10df20 0x7fc0bc110370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.043+0000 7fc0c2e00640 1 --2- 172.21.15.43:0/3916900262 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0bc0744a0 0x7fc0bc10d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.043+0000 7fc0c2e00640 1 --2- 172.21.15.43:0/3916900262 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0bc073b80 0x7fc0bc073f60 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.043+0000 7fc0c2e00640 1 -- 172.21.15.43:0/3916900262 >> 172.21.15.43:0/3916900262 conn(0x7fc0bc0fd1f0 msgr2=0x7fc0bc0ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:21.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.043+0000 7fc0c2e00640 1 -- 172.21.15.43:0/3916900262 shutdown_connections 2024-03-28T22:40:21.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.043+0000 7fc0c2e00640 1 -- 172.21.15.43:0/3916900262 wait complete. 2024-03-28T22:40:21.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.044+0000 7fc0c2e00640 1 Processor -- start 2024-03-28T22:40:21.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.045+0000 7fc0c2e00640 1 -- start start 2024-03-28T22:40:21.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.045+0000 7fc0c2e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0bc073b80 0x7fc0bc19a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.045+0000 7fc0c2e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0bc0744a0 0x7fc0bc19a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.045+0000 7fc0c2e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0bc10df20 0x7fc0bc19eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.045+0000 7fc0c2e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc0bc113d10 con 0x7fc0bc073b80 2024-03-28T22:40:21.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.045+0000 7fc0c2e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fc0bc113b90 con 0x7fc0bc10df20 2024-03-28T22:40:21.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.045+0000 7fc0c2e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fc0bc113e90 con 0x7fc0bc0744a0 2024-03-28T22:40:21.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.046+0000 7fc0bb400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0bc0744a0 0x7fc0bc19a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.046+0000 7fc0c0e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0bc10df20 0x7fc0bc19eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.046+0000 7fc0bb400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0bc0744a0 0x7fc0bc19a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:48078/0 (socket says 172.21.15.43:48078) 2024-03-28T22:40:21.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.046+0000 7fc0bb400640 1 -- 172.21.15.43:0/301163613 learned_addr learned my addr 172.21.15.43:0/301163613 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:21.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.046+0000 7fc0bbe00640 1 --2- 172.21.15.43:0/301163613 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0bc073b80 0x7fc0bc19a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.046+0000 7fc0bb400640 1 -- 172.21.15.43:0/301163613 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0bc10df20 msgr2=0x7fc0bc19eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:21.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.046+0000 7fc0bb400640 1 --2- 172.21.15.43:0/301163613 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0bc10df20 0x7fc0bc19eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.046+0000 7fc0bb400640 1 -- 172.21.15.43:0/301163613 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0bc073b80 msgr2=0x7fc0bc19a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:21.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.046+0000 7fc0bb400640 1 --2- 172.21.15.43:0/301163613 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0bc073b80 0x7fc0bc19a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.046+0000 7fc0bb400640 1 -- 172.21.15.43:0/301163613 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0bc19f230 con 0x7fc0bc0744a0 2024-03-28T22:40:21.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.046+0000 7fc0bbe00640 1 --2- 172.21.15.43:0/301163613 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0bc073b80 0x7fc0bc19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:40:21.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.046+0000 7fc0c0e00640 1 --2- 172.21.15.43:0/301163613 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0bc10df20 0x7fc0bc19eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:21.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.047+0000 7fc0bb400640 1 --2- 172.21.15.43:0/301163613 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0bc0744a0 0x7fc0bc19a7d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc0a800e9f0 tx=0x7fc0a800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:21.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.047+0000 7fc0b8c00640 1 -- 172.21.15.43:0/301163613 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0a800cd80 con 0x7fc0bc0744a0 2024-03-28T22:40:21.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.047+0000 7fc0b8c00640 1 -- 172.21.15.43:0/301163613 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0a8005500 con 0x7fc0bc0744a0 2024-03-28T22:40:21.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.047+0000 7fc0b8c00640 1 -- 172.21.15.43:0/301163613 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0a8010690 con 0x7fc0bc0744a0 2024-03-28T22:40:21.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.047+0000 7fc0c2e00640 1 -- 172.21.15.43:0/301163613 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0bc19f520 con 0x7fc0bc0744a0 2024-03-28T22:40:21.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.047+0000 7fc0c2e00640 1 -- 172.21.15.43:0/301163613 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0bc1a6db0 con 0x7fc0bc0744a0 2024-03-28T22:40:21.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.049+0000 7fc0c2e00640 1 -- 172.21.15.43:0/301163613 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0bc068990 con 0x7fc0bc0744a0 2024-03-28T22:40:21.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.053+0000 7fc0b8c00640 1 -- 172.21.15.43:0/301163613 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fc0a80046c0 con 0x7fc0bc0744a0 2024-03-28T22:40:21.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.055+0000 7fc0b8c00640 1 --2- 172.21.15.43:0/301163613 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc0a00a1f70 0x7fc0a00a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.055+0000 7fc0b8c00640 1 -- 172.21.15.43:0/301163613 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fc0a80ce980 con 0x7fc0bc0744a0 2024-03-28T22:40:21.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.055+0000 7fc0bbe00640 1 --2- 172.21.15.43:0/301163613 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc0a00a1f70 0x7fc0a00a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.056+0000 7fc0b8c00640 1 -- 172.21.15.43:0/301163613 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc0a809b670 con 0x7fc0bc0744a0 2024-03-28T22:40:21.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.057+0000 7fc0bbe00640 1 --2- 172.21.15.43:0/301163613 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc0a00a1f70 0x7fc0a00a4430 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac036040 tx=0x7fc0ac005fd0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:21.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:20 smithi172 ceph-mon[36566]: pgmap v99: 129 pgs: 129 active+clean; 49 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.2 MiB/s wr, 262 op/s 2024-03-28T22:40:21.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:20 smithi082 ceph-mon[36471]: pgmap v99: 129 pgs: 129 active+clean; 49 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.2 MiB/s wr, 262 op/s 2024-03-28T22:40:21.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.328+0000 7fc0c2e00640 1 -- 172.21.15.43:0/301163613 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc0bc19b590 con 0x7fc0bc0744a0 2024-03-28T22:40:21.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.329+0000 7fc0b8c00640 1 -- 172.21.15.43:0/301163613 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11931 (secure 0 0 0) 0x7fc0bc19b590 con 0x7fc0bc0744a0 2024-03-28T22:40:21.327 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:40:21.327 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:33:28.906396+0000","modified":"2024-03-28T22:40:08.672168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24572,"mds_1":24556,"mds_2":24535,"mds_3":24523,"mds_4":24518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26717":{"gid":26717,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/215779342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":215779342},{"type":"v1","addr":"172.21.15.82:6835","nonce":215779342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24535,24556,24518,24523,24572]},"id":1}]} 2024-03-28T22:40:21.327 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 32 2024-03-28T22:40:21.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.332+0000 7fc0c2e00640 1 -- 172.21.15.43:0/301163613 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc0a00a1f70 msgr2=0x7fc0a00a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:21.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.332+0000 7fc0c2e00640 1 --2- 172.21.15.43:0/301163613 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc0a00a1f70 0x7fc0a00a4430 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac036040 tx=0x7fc0ac005fd0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.332+0000 7fc0c2e00640 1 -- 172.21.15.43:0/301163613 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0bc0744a0 msgr2=0x7fc0bc19a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:21.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.332+0000 7fc0c2e00640 1 --2- 172.21.15.43:0/301163613 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0bc0744a0 0x7fc0bc19a7d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc0a800e9f0 tx=0x7fc0a800edf0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.332+0000 7fc0c2e00640 1 -- 172.21.15.43:0/301163613 shutdown_connections 2024-03-28T22:40:21.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.332+0000 7fc0c2e00640 1 --2- 172.21.15.43:0/301163613 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc0a00a1f70 0x7fc0a00a4430 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.332+0000 7fc0c2e00640 1 --2- 172.21.15.43:0/301163613 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0bc10df20 0x7fc0bc19eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.332+0000 7fc0c2e00640 1 --2- 172.21.15.43:0/301163613 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0bc0744a0 0x7fc0bc19a7d0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.332+0000 7fc0c2e00640 1 --2- 172.21.15.43:0/301163613 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0bc073b80 0x7fc0bc19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.332+0000 7fc0c2e00640 1 -- 172.21.15.43:0/301163613 >> 172.21.15.43:0/301163613 conn(0x7fc0bc0fd1f0 msgr2=0x7fc0bc0fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:21.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.332+0000 7fc0c2e00640 1 -- 172.21.15.43:0/301163613 shutdown_connections 2024-03-28T22:40:21.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.332+0000 7fc0c2e00640 1 -- 172.21.15.43:0/301163613 wait complete. 2024-03-28T22:40:21.348 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T22:40:21.351 INFO:tasks.workunit.client.0.smithi043.stdout: 1 173784 8.54 MB/sec warmup 112 sec latency 99.684 ms 2024-03-28T22:40:21.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.570+0000 7fa22c800640 1 -- 172.21.15.43:0/3067317948 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa224106660 msgr2=0x7fa224106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:21.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.570+0000 7fa22c800640 1 --2- 172.21.15.43:0/3067317948 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa224106660 0x7fa224106a40 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fa218009900 tx=0x7fa21802bc30 comp rx=0 tx=0).stop 2024-03-28T22:40:21.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.570+0000 7fa22c800640 1 -- 172.21.15.43:0/3067317948 shutdown_connections 2024-03-28T22:40:21.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.570+0000 7fa22c800640 1 --2- 172.21.15.43:0/3067317948 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa224100f80 0x7fa22410d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.570+0000 7fa22c800640 1 --2- 172.21.15.43:0/3067317948 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2241005c0 0x7fa224100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.570+0000 7fa22c800640 1 --2- 172.21.15.43:0/3067317948 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa224106660 0x7fa224106a40 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.570+0000 7fa22c800640 1 -- 172.21.15.43:0/3067317948 >> 172.21.15.43:0/3067317948 conn(0x7fa224077750 msgr2=0x7fa224077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:21.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.570+0000 7fa22c800640 1 -- 172.21.15.43:0/3067317948 shutdown_connections 2024-03-28T22:40:21.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.570+0000 7fa22c800640 1 -- 172.21.15.43:0/3067317948 wait complete. 2024-03-28T22:40:21.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.571+0000 7fa22c800640 1 Processor -- start 2024-03-28T22:40:21.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.571+0000 7fa22c800640 1 -- start start 2024-03-28T22:40:21.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.571+0000 7fa22c800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2241005c0 0x7fa224197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.571+0000 7fa22c800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa224100f80 0x7fa224198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.571+0000 7fa22c800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa224106660 0x7fa22419c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.571+0000 7fa22c800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa224111ae0 con 0x7fa2241005c0 2024-03-28T22:40:21.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.571+0000 7fa22c800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa224111960 con 0x7fa224106660 2024-03-28T22:40:21.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.572+0000 7fa22c800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fa224111c60 con 0x7fa224100f80 2024-03-28T22:40:21.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.572+0000 7fa229e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2241005c0 0x7fa224197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.572+0000 7fa229400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa224100f80 0x7fa224198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.572+0000 7fa229400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa224100f80 0x7fa224198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:48096/0 (socket says 172.21.15.43:48096) 2024-03-28T22:40:21.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.572+0000 7fa229400640 1 -- 172.21.15.43:0/59092045 learned_addr learned my addr 172.21.15.43:0/59092045 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:21.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.572+0000 7fa22a800640 1 --2- 172.21.15.43:0/59092045 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa224106660 0x7fa22419c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.572+0000 7fa229400640 1 -- 172.21.15.43:0/59092045 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa224106660 msgr2=0x7fa22419c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:21.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.572+0000 7fa229400640 1 --2- 172.21.15.43:0/59092045 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa224106660 0x7fa22419c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.572+0000 7fa229400640 1 -- 172.21.15.43:0/59092045 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2241005c0 msgr2=0x7fa224197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:21.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.572+0000 7fa229400640 1 --2- 172.21.15.43:0/59092045 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2241005c0 0x7fa224197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.572+0000 7fa229400640 1 -- 172.21.15.43:0/59092045 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa22419cf10 con 0x7fa224100f80 2024-03-28T22:40:21.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.572+0000 7fa229400640 1 --2- 172.21.15.43:0/59092045 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa224100f80 0x7fa224198460 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa21400e9f0 tx=0x7fa21400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:21.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.573+0000 7fa21ea00640 1 -- 172.21.15.43:0/59092045 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa21400cd80 con 0x7fa224100f80 2024-03-28T22:40:21.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.573+0000 7fa22c800640 1 -- 172.21.15.43:0/59092045 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa22419d200 con 0x7fa224100f80 2024-03-28T22:40:21.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.573+0000 7fa21ea00640 1 -- 172.21.15.43:0/59092045 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa214010910 con 0x7fa224100f80 2024-03-28T22:40:21.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.573+0000 7fa21ea00640 1 -- 172.21.15.43:0/59092045 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2140188f0 con 0x7fa224100f80 2024-03-28T22:40:21.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.573+0000 7fa22c800640 1 -- 172.21.15.43:0/59092045 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2241a4a90 con 0x7fa224100f80 2024-03-28T22:40:21.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.576+0000 7fa21ea00640 1 -- 172.21.15.43:0/59092045 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fa214010ac0 con 0x7fa224100f80 2024-03-28T22:40:21.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.576+0000 7fa22f436740 1 Processor -- start 2024-03-28T22:40:21.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.577+0000 7fa22f436740 1 -- start start 2024-03-28T22:40:21.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.577+0000 7fa22f436740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561909e1a790 0x561909e1ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.577+0000 7fa22f436740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561909e1b0b0 0x561909e23750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.577+0000 7fa22f436740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561909e23c90 0x561909e26080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.577+0000 7fa22f436740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x561909d7a050 con 0x561909e23c90 2024-03-28T22:40:21.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.577+0000 7fa22f436740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x561909d22ce0 con 0x561909e1b0b0 2024-03-28T22:40:21.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.577+0000 7fa22f436740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x561909d760f0 con 0x561909e1a790 2024-03-28T22:40:21.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.577+0000 7fa22a800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561909e1a790 0x561909e1ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.577+0000 7fa22a800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561909e1a790 0x561909e1ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:48108/0 (socket says 172.21.15.43:48108) 2024-03-28T22:40:21.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.577+0000 7fa22a800640 1 -- 172.21.15.43:0/2046097247 learned_addr learned my addr 172.21.15.43:0/2046097247 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:21.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.577+0000 7fa229e00640 1 --2- 172.21.15.43:0/2046097247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561909e1b0b0 0x561909e23750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.577+0000 7fa229400640 1 --2- 172.21.15.43:0/2046097247 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561909e23c90 0x561909e26080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.577+0000 7fa22a800640 1 -- 172.21.15.43:0/2046097247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561909e1b0b0 msgr2=0x561909e23750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:21.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.577+0000 7fa22a800640 1 --2- 172.21.15.43:0/2046097247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561909e1b0b0 0x561909e23750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.577+0000 7fa21ea00640 1 --2- 172.21.15.43:0/59092045 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa20c0a1f70 0x7fa20c0a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.578+0000 7fa22a800640 1 -- 172.21.15.43:0/2046097247 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561909e23c90 msgr2=0x561909e26080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:21.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.578+0000 7fa22a800640 1 --2- 172.21.15.43:0/2046097247 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561909e23c90 0x561909e26080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.578+0000 7fa22a800640 1 -- 172.21.15.43:0/2046097247 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561909a8e3f0 con 0x561909e1a790 2024-03-28T22:40:21.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.578+0000 7fa21ea00640 1 -- 172.21.15.43:0/59092045 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fa2140ce8c0 con 0x7fa224100f80 2024-03-28T22:40:21.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.578+0000 7fa22a800640 1 --2- 172.21.15.43:0/59092045 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa20c0a1f70 0x7fa20c0a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.578+0000 7fa22a800640 1 --2- 172.21.15.43:0/2046097247 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561909e1a790 0x561909e1ab70 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa220009fd0 tx=0x7fa2200118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0e3198df7bc21be server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:21.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.578+0000 7fa213000640 1 -- 172.21.15.43:0/2046097247 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa220003bf0 con 0x561909e1a790 2024-03-28T22:40:21.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.578+0000 7fa213000640 1 -- 172.21.15.43:0/2046097247 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa220004630 con 0x561909e1a790 2024-03-28T22:40:21.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.578+0000 7fa213000640 1 -- 172.21.15.43:0/2046097247 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa220019600 con 0x561909e1a790 2024-03-28T22:40:21.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.579+0000 7fa22f436740 1 -- 172.21.15.43:0/2046097247 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561909e1a790 msgr2=0x561909e1ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:21.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.579+0000 7fa22f436740 1 --2- 172.21.15.43:0/2046097247 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561909e1a790 0x561909e1ab70 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa220009fd0 tx=0x7fa2200118d0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.579+0000 7fa22a800640 1 --2- 172.21.15.43:0/59092045 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa20c0a1f70 0x7fa20c0a4430 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa22001d000 tx=0x7fa220019860 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:21.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.579+0000 7fa22f436740 1 -- 172.21.15.43:0/2046097247 shutdown_connections 2024-03-28T22:40:21.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.579+0000 7fa22f436740 1 --2- 172.21.15.43:0/2046097247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561909e1b0b0 0x561909e23750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.579+0000 7fa22f436740 1 --2- 172.21.15.43:0/2046097247 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561909e23c90 0x561909e26080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.579+0000 7fa22f436740 1 --2- 172.21.15.43:0/2046097247 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561909e1a790 0x561909e1ab70 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.579+0000 7fa22f436740 1 -- 172.21.15.43:0/2046097247 >> 172.21.15.43:0/2046097247 conn(0x561909e14ec0 msgr2=0x561909e152a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:21.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.579+0000 7fa22f436740 1 -- 172.21.15.43:0/2046097247 shutdown_connections 2024-03-28T22:40:21.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.580+0000 7fa22f436740 1 -- 172.21.15.43:0/2046097247 wait complete. 2024-03-28T22:40:21.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.580+0000 7fa22f436740 1 -- start start 2024-03-28T22:40:21.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.580+0000 7fa22f436740 1 Processor -- start 2024-03-28T22:40:21.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.580+0000 7fa22f436740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561909e1a790 0x561909e20c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.581+0000 7fa22f436740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561909e1b0b0 0x561909e21170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.581+0000 7fa22f436740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561909e23c90 0x561909e216b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.581+0000 7fa22f436740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x561909e26a80 con 0x561909e1b0b0 2024-03-28T22:40:21.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.581+0000 7fa22f436740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x561909d182f0 con 0x561909e23c90 2024-03-28T22:40:21.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.581+0000 7fa22f436740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x561909e26c00 con 0x561909e1a790 2024-03-28T22:40:21.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.581+0000 7fa22f436740 10 client.0 _ll_register_callbacks cb 0x561909a8dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:21.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.581+0000 7fa229e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561909e1a790 0x561909e20c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.581+0000 7fa229e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561909e1a790 0x561909e20c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:48112/0 (socket says 172.21.15.43:48112) 2024-03-28T22:40:21.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.581+0000 7fa229e00640 1 -- 172.21.15.43:0/2256135247 learned_addr learned my addr 172.21.15.43:0/2256135247 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:21.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.581+0000 7fa22a800640 1 --2- 172.21.15.43:0/2256135247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561909e23c90 0x561909e216b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.581+0000 7fa229400640 1 --2- 172.21.15.43:0/2256135247 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561909e1b0b0 0x561909e21170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.581+0000 7fa229e00640 1 -- 172.21.15.43:0/2256135247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561909e23c90 msgr2=0x561909e216b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:21.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.581+0000 7fa229e00640 1 --2- 172.21.15.43:0/2256135247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561909e23c90 0x561909e216b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.581+0000 7fa229e00640 1 -- 172.21.15.43:0/2256135247 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561909e1b0b0 msgr2=0x561909e21170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:21.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.581+0000 7fa229e00640 1 --2- 172.21.15.43:0/2256135247 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561909e1b0b0 0x561909e21170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.581+0000 7fa229e00640 1 -- 172.21.15.43:0/2256135247 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561909b587d0 con 0x561909e1a790 2024-03-28T22:40:21.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.581+0000 7fa22a800640 1 --2- 172.21.15.43:0/2256135247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561909e23c90 0x561909e216b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:21.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.581+0000 7fa229400640 1 --2- 172.21.15.43:0/2256135247 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561909e1b0b0 0x561909e21170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:21.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.582+0000 7fa229e00640 1 --2- 172.21.15.43:0/2256135247 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561909e1a790 0x561909e20c30 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa218009900 tx=0x7fa21802c7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a9fe22662f265c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:21.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.582+0000 7fa207400640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:40:21.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.582+0000 7fa207400640 1 -- 172.21.15.43:0/2256135247 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa21802ccc0 con 0x561909e1a790 2024-03-28T22:40:21.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.582+0000 7fa22f436740 1 -- 172.21.15.43:0/2256135247 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa220004630 con 0x561909e1a790 2024-03-28T22:40:21.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.582+0000 7fa207400640 1 -- 172.21.15.43:0/2256135247 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fa2180046c0 con 0x561909e1a790 2024-03-28T22:40:21.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.582+0000 7fa207400640 1 -- 172.21.15.43:0/2256135247 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa2180048c0 con 0x561909e1a790 2024-03-28T22:40:21.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.582+0000 7fa22f436740 10 client.26716 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:21.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.582+0000 7fa22f436740 1 -- 172.21.15.43:0/2256135247 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561909e32cd0 con 0x561909e1a790 2024-03-28T22:40:21.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.583+0000 7fa207400640 1 -- 172.21.15.43:0/2256135247 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa220004630 con 0x561909e1a790 2024-03-28T22:40:21.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.583+0000 7fa207400640 1 -- 172.21.15.43:0/2256135247 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa218046070 con 0x561909e1a790 2024-03-28T22:40:21.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.583+0000 7fa22f436740 10 client.26716 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:21.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.583+0000 7fa22f436740 10 client.26716 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:40:21.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.583+0000 7fa22f436740 20 client.26716 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:21.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.583+0000 7fa22f436740 1 --2- 172.21.15.43:0/2256135247 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x561909e34180 0x561909e365d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.583+0000 7fa22f436740 4 client.26716 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:40:21.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.583+0000 7fa22f436740 1 -- 172.21.15.43:0/2256135247 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561909d23680 con 0x561909e34180 2024-03-28T22:40:21.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.583+0000 7fa229400640 1 --2- 172.21.15.43:0/2256135247 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x561909e34180 0x561909e365d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.584+0000 7fa229400640 1 --2- 172.21.15.43:0/2256135247 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x561909e34180 0x561909e365d0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8ceddffd7663d3e1 server_cookie=45b934f3eda378a6 in_seq=0 out_seq=0 2024-03-28T22:40:21.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.584+0000 7fa207400640 10 client.26716 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:40:21.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.584+0000 7fa207400640 1 -- 172.21.15.43:0/2256135247 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561909d23680 con 0x561909e34180 2024-03-28T22:40:21.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.584+0000 7fa207400640 10 client.26716 handle_command_reply: tid=0 2024-03-28T22:40:21.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.584+0000 7fa22f436740 1 client.26716 shutdown 2024-03-28T22:40:21.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.585+0000 7fa22f436740 1 -- 172.21.15.43:0/2256135247 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561909e1a790 msgr2=0x561909e20c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:21.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.585+0000 7fa22f436740 1 --2- 172.21.15.43:0/2256135247 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561909e1a790 0x561909e20c30 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa218009900 tx=0x7fa21802c7e0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.585+0000 7fa22f436740 1 -- 172.21.15.43:0/2256135247 shutdown_connections 2024-03-28T22:40:21.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.585+0000 7fa22f436740 1 --2- 172.21.15.43:0/2256135247 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x561909e34180 0x561909e365d0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.585+0000 7fa22f436740 1 --2- 172.21.15.43:0/2256135247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561909e23c90 0x561909e216b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.585+0000 7fa207400640 0 client.26716 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:40:21.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.585+0000 7fa22f436740 1 --2- 172.21.15.43:0/2256135247 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561909e1b0b0 0x561909e21170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.585+0000 7fa22f436740 1 --2- 172.21.15.43:0/2256135247 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561909e1a790 0x561909e20c30 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.585+0000 7fa22f436740 1 -- 172.21.15.43:0/2256135247 >> 172.21.15.43:0/2256135247 conn(0x561909e14ec0 msgr2=0x561909e16c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:21.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.585+0000 7fa22f436740 1 -- 172.21.15.43:0/2256135247 shutdown_connections 2024-03-28T22:40:21.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.585+0000 7fa22f436740 1 -- 172.21.15.43:0/2256135247 wait complete. 2024-03-28T22:40:21.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.585+0000 7fa22f436740 20 client.26716 trim_cache size 0 max 16384 2024-03-28T22:40:21.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.601+0000 7fa22f436740 1 Processor -- start 2024-03-28T22:40:21.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.601+0000 7fa22f436740 1 -- start start 2024-03-28T22:40:21.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.601+0000 7fa22f436740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561909e1a790 0x561909e23360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.601+0000 7fa22f436740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561909e1b0b0 0x561909e2f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.601+0000 7fa22f436740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561909e23c90 0x561909e16a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.601+0000 7fa22f436740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa2241ac8a0 con 0x561909e23c90 2024-03-28T22:40:21.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.601+0000 7fa22f436740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa2241ae3e0 con 0x561909e1a790 2024-03-28T22:40:21.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.601+0000 7fa22f436740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fa2241acf70 con 0x561909e1b0b0 2024-03-28T22:40:21.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.602+0000 7fa229400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561909e1b0b0 0x561909e2f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.602+0000 7fa229400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561909e1b0b0 0x561909e2f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:48120/0 (socket says 172.21.15.43:48120) 2024-03-28T22:40:21.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.602+0000 7fa229400640 1 -- 172.21.15.43:0/4105929547 learned_addr learned my addr 172.21.15.43:0/4105929547 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:21.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.602+0000 7fa229e00640 1 --2- 172.21.15.43:0/4105929547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561909e1a790 0x561909e23360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.602+0000 7fa22a800640 1 --2- 172.21.15.43:0/4105929547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561909e23c90 0x561909e16a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.602+0000 7fa229400640 1 -- 172.21.15.43:0/4105929547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561909e1a790 msgr2=0x561909e23360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:21.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.602+0000 7fa229400640 1 --2- 172.21.15.43:0/4105929547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561909e1a790 0x561909e23360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.602+0000 7fa229400640 1 -- 172.21.15.43:0/4105929547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561909e23c90 msgr2=0x561909e16a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:21.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.602+0000 7fa229400640 1 --2- 172.21.15.43:0/4105929547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561909e23c90 0x561909e16a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.602+0000 7fa229400640 1 -- 172.21.15.43:0/4105929547 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561909e1f180 con 0x561909e1b0b0 2024-03-28T22:40:21.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.602+0000 7fa22a800640 1 --2- 172.21.15.43:0/4105929547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561909e23c90 0x561909e16a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:40:21.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.602+0000 7fa229400640 1 --2- 172.21.15.43:0/4105929547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561909e1b0b0 0x561909e2f820 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa214097930 tx=0x7fa214097280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77549122c4b9e81 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:21.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.603+0000 7fa212600640 1 -- 172.21.15.43:0/4105929547 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2140cbe90 con 0x561909e1b0b0 2024-03-28T22:40:21.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.603+0000 7fa212600640 1 -- 172.21.15.43:0/4105929547 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2140cc030 con 0x561909e1b0b0 2024-03-28T22:40:21.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.603+0000 7fa212600640 1 -- 172.21.15.43:0/4105929547 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa214096cb0 con 0x561909e1b0b0 2024-03-28T22:40:21.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.603+0000 7fa22f436740 1 -- 172.21.15.43:0/4105929547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561909e1b0b0 msgr2=0x561909e2f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:21.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.603+0000 7fa22f436740 1 --2- 172.21.15.43:0/4105929547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561909e1b0b0 0x561909e2f820 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa214097930 tx=0x7fa214097280 comp rx=0 tx=0).stop 2024-03-28T22:40:21.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.604+0000 7fa22f436740 1 -- 172.21.15.43:0/4105929547 shutdown_connections 2024-03-28T22:40:21.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.604+0000 7fa22f436740 1 --2- 172.21.15.43:0/4105929547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561909e23c90 0x561909e16a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.604+0000 7fa22f436740 1 --2- 172.21.15.43:0/4105929547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561909e1b0b0 0x561909e2f820 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.604+0000 7fa22f436740 1 --2- 172.21.15.43:0/4105929547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561909e1a790 0x561909e23360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.604+0000 7fa22f436740 1 -- 172.21.15.43:0/4105929547 >> 172.21.15.43:0/4105929547 conn(0x561909e14ec0 msgr2=0x561909e162f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:21.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.604+0000 7fa22f436740 1 -- 172.21.15.43:0/4105929547 shutdown_connections 2024-03-28T22:40:21.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.604+0000 7fa22f436740 1 -- 172.21.15.43:0/4105929547 wait complete. 2024-03-28T22:40:21.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.604+0000 7fa22f436740 1 -- start start 2024-03-28T22:40:21.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.604+0000 7fa22f436740 1 Processor -- start 2024-03-28T22:40:21.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.605+0000 7fa22f436740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561909e1a790 0x561909e27750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.605+0000 7fa22f436740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561909e1b0b0 0x561909e27c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.605+0000 7fa22f436740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561909e23c90 0x561909e281d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.605+0000 7fa22f436740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa2241add10 con 0x561909e1a790 2024-03-28T22:40:21.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.605+0000 7fa22f436740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa2241ad640 con 0x561909e1b0b0 2024-03-28T22:40:21.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.605+0000 7fa22f436740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fa2241aa880 con 0x561909e23c90 2024-03-28T22:40:21.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.605+0000 7fa22f436740 10 client.0 _ll_register_callbacks cb 0x7fa218009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:21.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.605+0000 7fa229400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561909e1b0b0 0x561909e27c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.605+0000 7fa229e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561909e1a790 0x561909e27750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.605+0000 7fa229400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561909e1b0b0 0x561909e27c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45294/0 (socket says 172.21.15.43:45294) 2024-03-28T22:40:21.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.605+0000 7fa229400640 1 -- 172.21.15.43:0/437609841 learned_addr learned my addr 172.21.15.43:0/437609841 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:21.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.605+0000 7fa22a800640 1 --2- 172.21.15.43:0/437609841 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561909e23c90 0x561909e281d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.605+0000 7fa229e00640 1 -- 172.21.15.43:0/437609841 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561909e23c90 msgr2=0x561909e281d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:21.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.605+0000 7fa229e00640 1 --2- 172.21.15.43:0/437609841 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561909e23c90 0x561909e281d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.605+0000 7fa229e00640 1 -- 172.21.15.43:0/437609841 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561909e1b0b0 msgr2=0x561909e27c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:21.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.605+0000 7fa229e00640 1 --2- 172.21.15.43:0/437609841 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561909e1b0b0 0x561909e27c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.605+0000 7fa229e00640 1 -- 172.21.15.43:0/437609841 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561909e994d0 con 0x561909e1a790 2024-03-28T22:40:21.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.606+0000 7fa229400640 1 --2- 172.21.15.43:0/437609841 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561909e1b0b0 0x561909e27c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:21.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.606+0000 7fa229e00640 1 --2- 172.21.15.43:0/437609841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561909e1a790 0x561909e27750 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa218003930 tx=0x7fa2180045f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c80532167a28b12 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:21.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.606+0000 7fa207400640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:40:21.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.606+0000 7fa207400640 1 -- 172.21.15.43:0/437609841 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa218009520 con 0x561909e1a790 2024-03-28T22:40:21.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.606+0000 7fa22f436740 1 -- 172.21.15.43:0/437609841 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2140cc030 con 0x561909e1a790 2024-03-28T22:40:21.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.606+0000 7fa207400640 1 -- 172.21.15.43:0/437609841 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fa218003b70 con 0x561909e1a790 2024-03-28T22:40:21.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.606+0000 7fa207400640 1 -- 172.21.15.43:0/437609841 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa218003d70 con 0x561909e1a790 2024-03-28T22:40:21.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.606+0000 7fa22f436740 10 client.16575 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:21.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.606+0000 7fa22f436740 1 -- 172.21.15.43:0/437609841 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561909e9a810 con 0x561909e1a790 2024-03-28T22:40:21.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.607+0000 7fa207400640 1 -- 172.21.15.43:0/437609841 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2140cc030 con 0x561909e1a790 2024-03-28T22:40:21.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.607+0000 7fa207400640 1 -- 172.21.15.43:0/437609841 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa218054030 con 0x561909e1a790 2024-03-28T22:40:21.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.607+0000 7fa22f436740 10 client.16575 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:21.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.607+0000 7fa22f436740 10 client.16575 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:40:21.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.607+0000 7fa22f436740 20 client.16575 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:21.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.607+0000 7fa22f436740 1 --2- 172.21.15.43:0/437609841 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x561909e34180 0x561909e9cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.607+0000 7fa22f436740 4 client.16575 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:40:21.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.607+0000 7fa22f436740 1 -- 172.21.15.43:0/437609841 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561909d87590 con 0x561909e34180 2024-03-28T22:40:21.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.607+0000 7fa229400640 1 --2- 172.21.15.43:0/437609841 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x561909e34180 0x561909e9cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.608+0000 7fa229400640 1 --2- 172.21.15.43:0/437609841 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x561909e34180 0x561909e9cfa0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=31ebe5ada4b8e328 server_cookie=1a1c212ef11d0105 in_seq=0 out_seq=0 2024-03-28T22:40:21.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.608+0000 7fa207400640 10 client.16575 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:40:21.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.608+0000 7fa207400640 1 -- 172.21.15.43:0/437609841 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561909d87590 con 0x561909e34180 2024-03-28T22:40:21.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.608+0000 7fa207400640 10 client.16575 handle_command_reply: tid=0 2024-03-28T22:40:21.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.608+0000 7fa22f436740 1 client.16575 shutdown 2024-03-28T22:40:21.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.608+0000 7fa22f436740 1 -- 172.21.15.43:0/437609841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561909e1a790 msgr2=0x561909e27750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:21.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.608+0000 7fa22f436740 1 --2- 172.21.15.43:0/437609841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561909e1a790 0x561909e27750 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa218003930 tx=0x7fa2180045f0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.608+0000 7fa22f436740 1 -- 172.21.15.43:0/437609841 shutdown_connections 2024-03-28T22:40:21.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.608+0000 7fa22f436740 1 --2- 172.21.15.43:0/437609841 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x561909e34180 0x561909e9cfa0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.608+0000 7fa22f436740 1 --2- 172.21.15.43:0/437609841 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561909e1b0b0 0x561909e27c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.608+0000 7fa22f436740 1 --2- 172.21.15.43:0/437609841 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561909e23c90 0x561909e281d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.608+0000 7fa22f436740 1 --2- 172.21.15.43:0/437609841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561909e1a790 0x561909e27750 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.608+0000 7fa207400640 0 client.16575 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:40:21.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.608+0000 7fa22f436740 1 -- 172.21.15.43:0/437609841 >> 172.21.15.43:0/437609841 conn(0x561909e14ec0 msgr2=0x561909e16b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:21.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.608+0000 7fa22f436740 1 -- 172.21.15.43:0/437609841 shutdown_connections 2024-03-28T22:40:21.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.608+0000 7fa22f436740 1 -- 172.21.15.43:0/437609841 wait complete. 2024-03-28T22:40:21.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.608+0000 7fa22f436740 20 client.16575 trim_cache size 0 max 16384 2024-03-28T22:40:21.613 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:40:21.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.609+0000 7fa22c800640 1 -- 172.21.15.43:0/59092045 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa20c0a1f70 msgr2=0x7fa20c0a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:21.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.609+0000 7fa22c800640 1 --2- 172.21.15.43:0/59092045 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa20c0a1f70 0x7fa20c0a4430 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa22001d000 tx=0x7fa220019860 comp rx=0 tx=0).stop 2024-03-28T22:40:21.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.609+0000 7fa22c800640 1 -- 172.21.15.43:0/59092045 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa224100f80 msgr2=0x7fa224198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:21.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.609+0000 7fa22c800640 1 --2- 172.21.15.43:0/59092045 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa224100f80 0x7fa224198460 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa21400e9f0 tx=0x7fa21400eeb0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.609+0000 7fa22c800640 1 -- 172.21.15.43:0/59092045 shutdown_connections 2024-03-28T22:40:21.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.609+0000 7fa22c800640 1 --2- 172.21.15.43:0/59092045 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa20c0a1f70 0x7fa20c0a4430 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.609+0000 7fa22c800640 1 --2- 172.21.15.43:0/59092045 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa224106660 0x7fa22419c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.609+0000 7fa22c800640 1 --2- 172.21.15.43:0/59092045 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa224100f80 0x7fa224198460 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.609+0000 7fa22c800640 1 --2- 172.21.15.43:0/59092045 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa2241005c0 0x7fa224197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.609+0000 7fa22c800640 1 -- 172.21.15.43:0/59092045 >> 172.21.15.43:0/59092045 conn(0x7fa224077750 msgr2=0x7fa2240ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:21.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.609+0000 7fa22c800640 1 -- 172.21.15.43:0/59092045 shutdown_connections 2024-03-28T22:40:21.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.609+0000 7fa22c800640 1 -- 172.21.15.43:0/59092045 wait complete. 2024-03-28T22:40:21.624 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T22:40:21.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.840+0000 7f4091400640 1 -- 172.21.15.43:0/2566087790 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f408c1037d0 msgr2=0x7f408c103c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:21.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.840+0000 7f4091400640 1 --2- 172.21.15.43:0/2566087790 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f408c1037d0 0x7f408c103c50 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4078009930 tx=0x7f407802bc30 comp rx=0 tx=0).stop 2024-03-28T22:40:21.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.840+0000 7f4091400640 1 -- 172.21.15.43:0/2566087790 shutdown_connections 2024-03-28T22:40:21.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.840+0000 7f4091400640 1 --2- 172.21.15.43:0/2566087790 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f408c104190 0x7f408c1106a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.840+0000 7f4091400640 1 --2- 172.21.15.43:0/2566087790 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f408c1037d0 0x7f408c103c50 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.840+0000 7f4091400640 1 --2- 172.21.15.43:0/2566087790 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f408c1097e0 0x7f408c109bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.840+0000 7f4091400640 1 -- 172.21.15.43:0/2566087790 >> 172.21.15.43:0/2566087790 conn(0x7f408c0ff280 msgr2=0x7f408c1016a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:21.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.840+0000 7f4091400640 1 -- 172.21.15.43:0/2566087790 shutdown_connections 2024-03-28T22:40:21.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.840+0000 7f4091400640 1 -- 172.21.15.43:0/2566087790 wait complete. 2024-03-28T22:40:21.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.841+0000 7f4091400640 1 Processor -- start 2024-03-28T22:40:21.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.842+0000 7f4091400640 1 -- start start 2024-03-28T22:40:21.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.842+0000 7f4091400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f408c1037d0 0x7f408c19a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.842+0000 7f4091400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f408c104190 0x7f408c19a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.842+0000 7f4091400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f408c1097e0 0x7f408c19e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.842+0000 7f4091400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f408c113bf0 con 0x7f408c1097e0 2024-03-28T22:40:21.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.842+0000 7f4091400640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f408c113a70 con 0x7f408c1037d0 2024-03-28T22:40:21.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.842+0000 7f4091400640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f408c113d70 con 0x7f408c104190 2024-03-28T22:40:21.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.842+0000 7f408be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f408c1037d0 0x7f408c19a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.842+0000 7f408be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f408c1037d0 0x7f408c19a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45322/0 (socket says 172.21.15.43:45322) 2024-03-28T22:40:21.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.842+0000 7f408be00640 1 -- 172.21.15.43:0/68788679 learned_addr learned my addr 172.21.15.43:0/68788679 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:21.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.842+0000 7f408b400640 1 --2- 172.21.15.43:0/68788679 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f408c104190 0x7f408c19a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.842+0000 7f4090a00640 1 --2- 172.21.15.43:0/68788679 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f408c1097e0 0x7f408c19e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.843+0000 7f408be00640 1 -- 172.21.15.43:0/68788679 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f408c104190 msgr2=0x7f408c19a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:21.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.843+0000 7f408be00640 1 --2- 172.21.15.43:0/68788679 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f408c104190 0x7f408c19a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.843+0000 7f408be00640 1 -- 172.21.15.43:0/68788679 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f408c1097e0 msgr2=0x7f408c19e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:21.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.843+0000 7f408be00640 1 --2- 172.21.15.43:0/68788679 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f408c1097e0 0x7f408c19e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.843+0000 7f408be00640 1 -- 172.21.15.43:0/68788679 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f408c19f100 con 0x7f408c1037d0 2024-03-28T22:40:21.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.843+0000 7f4090a00640 1 --2- 172.21.15.43:0/68788679 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f408c1097e0 0x7f408c19e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:40:21.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.843+0000 7f408be00640 1 --2- 172.21.15.43:0/68788679 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f408c1037d0 0x7f408c19a110 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f407c00e9f0 tx=0x7f407c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:21.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.843+0000 7f4088c00640 1 -- 172.21.15.43:0/68788679 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f407c00cd80 con 0x7f408c1037d0 2024-03-28T22:40:21.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.843+0000 7f4091400640 1 -- 172.21.15.43:0/68788679 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f408c19f3f0 con 0x7f408c1037d0 2024-03-28T22:40:21.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.843+0000 7f4088c00640 1 -- 172.21.15.43:0/68788679 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f407c010910 con 0x7f408c1037d0 2024-03-28T22:40:21.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.843+0000 7f4088c00640 1 -- 172.21.15.43:0/68788679 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f407c0188f0 con 0x7f408c1037d0 2024-03-28T22:40:21.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.844+0000 7f4091400640 1 -- 172.21.15.43:0/68788679 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f408c1a6c80 con 0x7f408c1037d0 2024-03-28T22:40:21.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.847+0000 7f409564b740 1 Processor -- start 2024-03-28T22:40:21.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.847+0000 7f4088c00640 1 -- 172.21.15.43:0/68788679 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f407c010ac0 con 0x7f408c1037d0 2024-03-28T22:40:21.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.847+0000 7f409564b740 1 -- start start 2024-03-28T22:40:21.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.847+0000 7f409564b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2b57550 0x562bd2b57930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.847+0000 7f409564b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2b57e70 0x562bd2b60510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.847+0000 7f409564b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2b60a50 0x562bd2b62e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.847+0000 7f409564b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562bd2abb050 con 0x562bd2b57e70 2024-03-28T22:40:21.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.847+0000 7f409564b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x562bd2a63ce0 con 0x562bd2b60a50 2024-03-28T22:40:21.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.847+0000 7f409564b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x562bd2ab70f0 con 0x562bd2b57550 2024-03-28T22:40:21.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.847+0000 7f4090a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2b57550 0x562bd2b57930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.847+0000 7f4090a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2b57550 0x562bd2b57930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:48162/0 (socket says 172.21.15.43:48162) 2024-03-28T22:40:21.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.847+0000 7f4090a00640 1 -- 172.21.15.43:0/4189386863 learned_addr learned my addr 172.21.15.43:0/4189386863 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:21.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.847+0000 7f408b400640 1 --2- 172.21.15.43:0/4189386863 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2b60a50 0x562bd2b62e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.847+0000 7f408be00640 1 --2- 172.21.15.43:0/4189386863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2b57e70 0x562bd2b60510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.848+0000 7f4090a00640 1 -- 172.21.15.43:0/4189386863 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2b60a50 msgr2=0x562bd2b62e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:21.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.848+0000 7f4090a00640 1 --2- 172.21.15.43:0/4189386863 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2b60a50 0x562bd2b62e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.848+0000 7f4090a00640 1 -- 172.21.15.43:0/4189386863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2b57e70 msgr2=0x562bd2b60510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:21.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.848+0000 7f4090a00640 1 --2- 172.21.15.43:0/4189386863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2b57e70 0x562bd2b60510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.848+0000 7f4090a00640 1 -- 172.21.15.43:0/4189386863 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bd27cf3f0 con 0x562bd2b57550 2024-03-28T22:40:21.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.848+0000 7f408be00640 1 --2- 172.21.15.43:0/4189386863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2b57e70 0x562bd2b60510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:40:21.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.848+0000 7f4090a00640 1 --2- 172.21.15.43:0/4189386863 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2b57550 0x562bd2b57930 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f408400f410 tx=0x7f408400f8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da975d18a46824f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:21.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.848+0000 7f4081200640 1 -- 172.21.15.43:0/4189386863 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4084015070 con 0x562bd2b57550 2024-03-28T22:40:21.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.848+0000 7f4081200640 1 -- 172.21.15.43:0/4189386863 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4084011be0 con 0x562bd2b57550 2024-03-28T22:40:21.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.848+0000 7f4081200640 1 -- 172.21.15.43:0/4189386863 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4084010cd0 con 0x562bd2b57550 2024-03-28T22:40:21.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.849+0000 7f4088c00640 1 --2- 172.21.15.43:0/68788679 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f40700a1ef0 0x7f40700a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.849+0000 7f4088c00640 1 -- 172.21.15.43:0/68788679 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f407c0ce900 con 0x7f408c1037d0 2024-03-28T22:40:21.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.849+0000 7f408be00640 1 --2- 172.21.15.43:0/68788679 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f40700a1ef0 0x7f40700a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.849+0000 7f409564b740 1 -- 172.21.15.43:0/4189386863 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2b57550 msgr2=0x562bd2b57930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:21.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.849+0000 7f409564b740 1 --2- 172.21.15.43:0/4189386863 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2b57550 0x562bd2b57930 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f408400f410 tx=0x7f408400f8d0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.849+0000 7f409564b740 1 -- 172.21.15.43:0/4189386863 shutdown_connections 2024-03-28T22:40:21.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.849+0000 7f409564b740 1 --2- 172.21.15.43:0/4189386863 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2b60a50 0x562bd2b62e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.849+0000 7f409564b740 1 --2- 172.21.15.43:0/4189386863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2b57e70 0x562bd2b60510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.849+0000 7f409564b740 1 --2- 172.21.15.43:0/4189386863 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2b57550 0x562bd2b57930 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.849+0000 7f409564b740 1 -- 172.21.15.43:0/4189386863 >> 172.21.15.43:0/4189386863 conn(0x562bd2b51c80 msgr2=0x562bd2b52060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:21.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.849+0000 7f409564b740 1 -- 172.21.15.43:0/4189386863 shutdown_connections 2024-03-28T22:40:21.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.849+0000 7f409564b740 1 -- 172.21.15.43:0/4189386863 wait complete. 2024-03-28T22:40:21.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.850+0000 7f408be00640 1 --2- 172.21.15.43:0/68788679 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f40700a1ef0 0x7f40700a43b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f407c0cbd60 tx=0x7f407c01f000 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:21.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.850+0000 7f409564b740 1 -- start start 2024-03-28T22:40:21.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.850+0000 7f409564b740 1 Processor -- start 2024-03-28T22:40:21.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.850+0000 7f409564b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2b57550 0x562bd2b5daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.850+0000 7f409564b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2b57e70 0x562bd2b5e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.850+0000 7f409564b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2b60a50 0x562bd2b5e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.850+0000 7f409564b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562bd2b63840 con 0x562bd2b60a50 2024-03-28T22:40:21.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.850+0000 7f409564b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x562bd2a592f0 con 0x562bd2b57e70 2024-03-28T22:40:21.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.850+0000 7f409564b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x562bd2b639c0 con 0x562bd2b57550 2024-03-28T22:40:21.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.850+0000 7f409564b740 10 client.0 _ll_register_callbacks cb 0x562bd27cebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:21.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.851+0000 7f408b400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2b57e70 0x562bd2b5e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.851+0000 7f408b400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2b57e70 0x562bd2b5e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45330/0 (socket says 172.21.15.43:45330) 2024-03-28T22:40:21.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.851+0000 7f408b400640 1 -- 172.21.15.43:0/798674837 learned_addr learned my addr 172.21.15.43:0/798674837 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:21.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.851+0000 7f4090a00640 1 --2- 172.21.15.43:0/798674837 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2b57550 0x562bd2b5daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.851+0000 7f4090a00640 1 --2- 172.21.15.43:0/798674837 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2b60a50 0x562bd2b5e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.851+0000 7f408b400640 1 -- 172.21.15.43:0/798674837 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2b57550 msgr2=0x562bd2b5daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:21.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.851+0000 7f408b400640 1 --2- 172.21.15.43:0/798674837 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2b57550 0x562bd2b5daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.851+0000 7f408b400640 1 -- 172.21.15.43:0/798674837 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2b60a50 msgr2=0x562bd2b5e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:21.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.851+0000 7f408b400640 1 --2- 172.21.15.43:0/798674837 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2b60a50 0x562bd2b5e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.851+0000 7f408b400640 1 -- 172.21.15.43:0/798674837 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bd28997d0 con 0x562bd2b57e70 2024-03-28T22:40:21.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.851+0000 7f408b400640 1 --2- 172.21.15.43:0/798674837 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2b57e70 0x562bd2b5e030 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f407802cfd0 tx=0x7f407802ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aaffe9726272943 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:21.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.852+0000 7f4075600640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:40:21.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.852+0000 7f4075600640 1 -- 172.21.15.43:0/798674837 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4078004240 con 0x562bd2b57e70 2024-03-28T22:40:21.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.852+0000 7f409564b740 1 -- 172.21.15.43:0/798674837 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4084011be0 con 0x562bd2b57e70 2024-03-28T22:40:21.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.852+0000 7f4075600640 1 -- 172.21.15.43:0/798674837 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f40780043e0 con 0x562bd2b57e70 2024-03-28T22:40:21.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.852+0000 7f4075600640 1 -- 172.21.15.43:0/798674837 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f40780046c0 con 0x562bd2b57e70 2024-03-28T22:40:21.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.852+0000 7f409564b740 10 client.26786 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:21.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.852+0000 7f409564b740 1 -- 172.21.15.43:0/798674837 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562bd2b6fb90 con 0x562bd2b57e70 2024-03-28T22:40:21.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.852+0000 7f4075600640 1 -- 172.21.15.43:0/798674837 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4084011be0 con 0x562bd2b57e70 2024-03-28T22:40:21.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.853+0000 7f4075600640 1 -- 172.21.15.43:0/798674837 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f40780035f0 con 0x562bd2b57e70 2024-03-28T22:40:21.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.853+0000 7f409564b740 10 client.26786 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:21.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.853+0000 7f409564b740 10 client.26786 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:40:21.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.853+0000 7f409564b740 20 client.26786 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:21.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.853+0000 7f409564b740 1 --2- 172.21.15.43:0/798674837 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x562bd2b70ff0 0x562bd2b73410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.853+0000 7f409564b740 4 client.26786 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:40:21.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.853+0000 7f409564b740 1 -- 172.21.15.43:0/798674837 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562bd2a64680 con 0x562bd2b70ff0 2024-03-28T22:40:21.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.853+0000 7f4090a00640 1 --2- 172.21.15.43:0/798674837 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x562bd2b70ff0 0x562bd2b73410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.854+0000 7f4090a00640 1 --2- 172.21.15.43:0/798674837 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x562bd2b70ff0 0x562bd2b73410 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=64dd53296989b48e server_cookie=4c7d750b4f2a0ce2 in_seq=0 out_seq=0 2024-03-28T22:40:21.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.854+0000 7f4075600640 10 client.26786 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:40:21.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.854+0000 7f4075600640 1 -- 172.21.15.43:0/798674837 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562bd2a64680 con 0x562bd2b70ff0 2024-03-28T22:40:21.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.854+0000 7f4075600640 10 client.26786 handle_command_reply: tid=0 2024-03-28T22:40:21.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.854+0000 7f409564b740 1 client.26786 shutdown 2024-03-28T22:40:21.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.855+0000 7f409564b740 1 -- 172.21.15.43:0/798674837 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2b57e70 msgr2=0x562bd2b5e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:21.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.855+0000 7f409564b740 1 --2- 172.21.15.43:0/798674837 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2b57e70 0x562bd2b5e030 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f407802cfd0 tx=0x7f407802ce30 comp rx=0 tx=0).stop 2024-03-28T22:40:21.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.855+0000 7f409564b740 1 -- 172.21.15.43:0/798674837 shutdown_connections 2024-03-28T22:40:21.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.855+0000 7f409564b740 1 --2- 172.21.15.43:0/798674837 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x562bd2b70ff0 0x562bd2b73410 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.855+0000 7f409564b740 1 --2- 172.21.15.43:0/798674837 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2b57e70 0x562bd2b5e030 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.855+0000 7f4075600640 0 client.26786 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:40:21.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.855+0000 7f409564b740 1 --2- 172.21.15.43:0/798674837 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2b60a50 0x562bd2b5e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.855+0000 7f409564b740 1 --2- 172.21.15.43:0/798674837 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2b57550 0x562bd2b5daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.855+0000 7f409564b740 1 -- 172.21.15.43:0/798674837 >> 172.21.15.43:0/798674837 conn(0x562bd2b51c80 msgr2=0x562bd2b539d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:21.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.855+0000 7f409564b740 1 -- 172.21.15.43:0/798674837 shutdown_connections 2024-03-28T22:40:21.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.855+0000 7f409564b740 1 -- 172.21.15.43:0/798674837 wait complete. 2024-03-28T22:40:21.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.855+0000 7f409564b740 20 client.26786 trim_cache size 0 max 16384 2024-03-28T22:40:21.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.871+0000 7f409564b740 1 Processor -- start 2024-03-28T22:40:21.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.871+0000 7f409564b740 1 -- start start 2024-03-28T22:40:21.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.871+0000 7f409564b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2b57550 0x562bd2b6c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.871+0000 7f409564b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2b57e70 0x562bd2b78ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.871+0000 7f409564b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2b60a50 0x562bd2b50c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.871+0000 7f409564b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f408c1b0d50 con 0x562bd2b57e70 2024-03-28T22:40:21.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.871+0000 7f409564b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f408c1b2890 con 0x562bd2b57550 2024-03-28T22:40:21.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.871+0000 7f409564b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f408c1b1420 con 0x562bd2b60a50 2024-03-28T22:40:21.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.872+0000 7f4090a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2b57550 0x562bd2b6c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.872+0000 7f4090a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2b57550 0x562bd2b6c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45340/0 (socket says 172.21.15.43:45340) 2024-03-28T22:40:21.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.872+0000 7f4090a00640 1 -- 172.21.15.43:0/2113977375 learned_addr learned my addr 172.21.15.43:0/2113977375 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:21.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.872+0000 7f4090a00640 1 --2- 172.21.15.43:0/2113977375 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2b60a50 0x562bd2b50c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.872+0000 7f408b400640 1 --2- 172.21.15.43:0/2113977375 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2b57e70 0x562bd2b78ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.872+0000 7f4090a00640 1 -- 172.21.15.43:0/2113977375 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2b60a50 msgr2=0x562bd2b50c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.872+0000 7f4090a00640 1 --2- 172.21.15.43:0/2113977375 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2b60a50 0x562bd2b50c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.872+0000 7f4090a00640 1 -- 172.21.15.43:0/2113977375 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2b57e70 msgr2=0x562bd2b78ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.872+0000 7f4090a00640 1 --2- 172.21.15.43:0/2113977375 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2b57e70 0x562bd2b78ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.872+0000 7f4090a00640 1 -- 172.21.15.43:0/2113977375 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bd2b525c0 con 0x562bd2b57550 2024-03-28T22:40:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.872+0000 7f408b400640 1 --2- 172.21.15.43:0/2113977375 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2b57e70 0x562bd2b78ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:40:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.872+0000 7f4090a00640 1 --2- 172.21.15.43:0/2113977375 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2b57550 0x562bd2b6c5e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f4084009f70 tx=0x7f4084000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c0096296505100f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.873+0000 7f4077e00640 1 -- 172.21.15.43:0/2113977375 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4084033070 con 0x562bd2b57550 2024-03-28T22:40:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.873+0000 7f4077e00640 1 -- 172.21.15.43:0/2113977375 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40840105d0 con 0x562bd2b57550 2024-03-28T22:40:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.873+0000 7f4077e00640 1 -- 172.21.15.43:0/2113977375 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4084024040 con 0x562bd2b57550 2024-03-28T22:40:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.873+0000 7f409564b740 1 -- 172.21.15.43:0/2113977375 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2b57550 msgr2=0x562bd2b6c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.873+0000 7f409564b740 1 --2- 172.21.15.43:0/2113977375 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2b57550 0x562bd2b6c5e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f4084009f70 tx=0x7f4084000f40 comp rx=0 tx=0).stop 2024-03-28T22:40:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.874+0000 7f409564b740 1 -- 172.21.15.43:0/2113977375 shutdown_connections 2024-03-28T22:40:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.874+0000 7f409564b740 1 --2- 172.21.15.43:0/2113977375 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2b60a50 0x562bd2b50c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.874+0000 7f409564b740 1 --2- 172.21.15.43:0/2113977375 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2b57e70 0x562bd2b78ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.874+0000 7f409564b740 1 --2- 172.21.15.43:0/2113977375 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2b57550 0x562bd2b6c5e0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.874+0000 7f409564b740 1 -- 172.21.15.43:0/2113977375 >> 172.21.15.43:0/2113977375 conn(0x562bd2b51c80 msgr2=0x562bd2b50530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.874+0000 7f409564b740 1 -- 172.21.15.43:0/2113977375 shutdown_connections 2024-03-28T22:40:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.874+0000 7f409564b740 1 -- 172.21.15.43:0/2113977375 wait complete. 2024-03-28T22:40:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.874+0000 7f409564b740 1 -- start start 2024-03-28T22:40:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.874+0000 7f409564b740 1 Processor -- start 2024-03-28T22:40:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.875+0000 7f409564b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2b57550 0x562bd2b69030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.875+0000 7f409564b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2b57e70 0x562bd2b643a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.875+0000 7f409564b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2b60a50 0x562bd2b64ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.875+0000 7f409564b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f408c1b21c0 con 0x562bd2b57550 2024-03-28T22:40:21.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.875+0000 7f409564b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f408c1b1af0 con 0x562bd2b60a50 2024-03-28T22:40:21.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.875+0000 7f409564b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f408c1aed30 con 0x562bd2b57e70 2024-03-28T22:40:21.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.875+0000 7f409564b740 10 client.0 _ll_register_callbacks cb 0x7f4078009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:21.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.875+0000 7f4090a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2b57550 0x562bd2b69030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.875+0000 7f408b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2b57e70 0x562bd2b643a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.875+0000 7f408b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2b57e70 0x562bd2b643a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:48200/0 (socket says 172.21.15.43:48200) 2024-03-28T22:40:21.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.875+0000 7f408b400640 1 -- 172.21.15.43:0/559738028 learned_addr learned my addr 172.21.15.43:0/559738028 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:21.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.875+0000 7f4090a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2b57550 0x562bd2b69030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37940/0 (socket says 172.21.15.43:37940) 2024-03-28T22:40:21.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.875+0000 7f4090a00640 1 --2- 172.21.15.43:0/559738028 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2b60a50 0x562bd2b64ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.875+0000 7f408b400640 1 -- 172.21.15.43:0/559738028 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2b60a50 msgr2=0x562bd2b64ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:21.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.875+0000 7f408b400640 1 --2- 172.21.15.43:0/559738028 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2b60a50 0x562bd2b64ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.875+0000 7f408b400640 1 -- 172.21.15.43:0/559738028 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2b57550 msgr2=0x562bd2b69030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:21.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.875+0000 7f408b400640 1 --2- 172.21.15.43:0/559738028 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2b57550 0x562bd2b69030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.875+0000 7f408b400640 1 -- 172.21.15.43:0/559738028 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bd2bd62d0 con 0x562bd2b57e70 2024-03-28T22:40:21.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.876+0000 7f408b400640 1 --2- 172.21.15.43:0/559738028 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2b57e70 0x562bd2b643a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4078000c00 tx=0x7f407802c3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f458cc8a9259cc4f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:21.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.876+0000 7f4075600640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:40:21.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.876+0000 7f4075600640 1 -- 172.21.15.43:0/559738028 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f407805b020 con 0x562bd2b57e70 2024-03-28T22:40:21.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.876+0000 7f409564b740 1 -- 172.21.15.43:0/559738028 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40840105d0 con 0x562bd2b57e70 2024-03-28T22:40:21.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.876+0000 7f4075600640 1 -- 172.21.15.43:0/559738028 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f4078004a60 con 0x562bd2b57e70 2024-03-28T22:40:21.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.876+0000 7f4075600640 1 -- 172.21.15.43:0/559738028 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4078066050 con 0x562bd2b57e70 2024-03-28T22:40:21.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.876+0000 7f409564b740 10 client.26740 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:21.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.876+0000 7f409564b740 1 -- 172.21.15.43:0/559738028 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562bd2bd74d0 con 0x562bd2b57e70 2024-03-28T22:40:21.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.876+0000 7f4075600640 1 -- 172.21.15.43:0/559738028 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40840105d0 con 0x562bd2b57e70 2024-03-28T22:40:21.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.877+0000 7f4075600640 1 -- 172.21.15.43:0/559738028 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4078003d70 con 0x562bd2b57e70 2024-03-28T22:40:21.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.877+0000 7f409564b740 10 client.26740 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:21.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.877+0000 7f409564b740 10 client.26740 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:40:21.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.877+0000 7f409564b740 20 client.26740 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:21.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.877+0000 7f409564b740 1 --2- 172.21.15.43:0/559738028 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x562bd2bd7c40 0x562bd2bda060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:21.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.877+0000 7f409564b740 4 client.26740 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:40:21.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.877+0000 7f409564b740 1 -- 172.21.15.43:0/559738028 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562bd2ac8590 con 0x562bd2bd7c40 2024-03-28T22:40:21.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.877+0000 7f4090a00640 1 --2- 172.21.15.43:0/559738028 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x562bd2bd7c40 0x562bd2bda060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:21.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.878+0000 7f4090a00640 1 --2- 172.21.15.43:0/559738028 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x562bd2bd7c40 0x562bd2bda060 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c469585eaeaf40c6 server_cookie=d76fa4dce7a62c5b in_seq=0 out_seq=0 2024-03-28T22:40:21.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.878+0000 7f4075600640 10 client.26740 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:40:21.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.878+0000 7f4075600640 1 -- 172.21.15.43:0/559738028 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562bd2ac8590 con 0x562bd2bd7c40 2024-03-28T22:40:21.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.878+0000 7f4075600640 10 client.26740 handle_command_reply: tid=0 2024-03-28T22:40:21.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.878+0000 7f409564b740 1 client.26740 shutdown 2024-03-28T22:40:21.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.878+0000 7f409564b740 1 -- 172.21.15.43:0/559738028 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2b57e70 msgr2=0x562bd2b643a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:21.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.878+0000 7f409564b740 1 --2- 172.21.15.43:0/559738028 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2b57e70 0x562bd2b643a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4078000c00 tx=0x7f407802c3b0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.878+0000 7f409564b740 1 -- 172.21.15.43:0/559738028 shutdown_connections 2024-03-28T22:40:21.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.878+0000 7f409564b740 1 --2- 172.21.15.43:0/559738028 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x562bd2bd7c40 0x562bd2bda060 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.878+0000 7f409564b740 1 --2- 172.21.15.43:0/559738028 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2b60a50 0x562bd2b64ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.878+0000 7f409564b740 1 --2- 172.21.15.43:0/559738028 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2b57e70 0x562bd2b643a0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.878+0000 7f409564b740 1 --2- 172.21.15.43:0/559738028 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2b57550 0x562bd2b69030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.878+0000 7f4075600640 0 client.26740 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:40:21.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.878+0000 7f409564b740 1 -- 172.21.15.43:0/559738028 >> 172.21.15.43:0/559738028 conn(0x562bd2b51c80 msgr2=0x562bd2b5e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:21.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.878+0000 7f409564b740 1 -- 172.21.15.43:0/559738028 shutdown_connections 2024-03-28T22:40:21.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.878+0000 7f409564b740 1 -- 172.21.15.43:0/559738028 wait complete. 2024-03-28T22:40:21.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.879+0000 7f409564b740 20 client.26740 trim_cache size 0 max 16384 2024-03-28T22:40:21.884 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:40:21.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.880+0000 7f4091400640 1 -- 172.21.15.43:0/68788679 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f40700a1ef0 msgr2=0x7f40700a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:21.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.880+0000 7f4091400640 1 --2- 172.21.15.43:0/68788679 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f40700a1ef0 0x7f40700a43b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f407c0cbd60 tx=0x7f407c01f000 comp rx=0 tx=0).stop 2024-03-28T22:40:21.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.880+0000 7f4091400640 1 -- 172.21.15.43:0/68788679 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f408c1037d0 msgr2=0x7f408c19a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:21.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.880+0000 7f4091400640 1 --2- 172.21.15.43:0/68788679 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f408c1037d0 0x7f408c19a110 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f407c00e9f0 tx=0x7f407c00eeb0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.880+0000 7f4091400640 1 -- 172.21.15.43:0/68788679 shutdown_connections 2024-03-28T22:40:21.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.880+0000 7f4091400640 1 --2- 172.21.15.43:0/68788679 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f40700a1ef0 0x7f40700a43b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.880+0000 7f4091400640 1 --2- 172.21.15.43:0/68788679 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f408c1097e0 0x7f408c19e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.880+0000 7f4091400640 1 --2- 172.21.15.43:0/68788679 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f408c104190 0x7f408c19a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.880+0000 7f4091400640 1 --2- 172.21.15.43:0/68788679 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f408c1037d0 0x7f408c19a110 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:21.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.880+0000 7f4091400640 1 -- 172.21.15.43:0/68788679 >> 172.21.15.43:0/68788679 conn(0x7f408c0ff280 msgr2=0x7f408c0ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:21.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.880+0000 7f4091400640 1 -- 172.21.15.43:0/68788679 shutdown_connections 2024-03-28T22:40:21.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:21.880+0000 7f4091400640 1 -- 172.21.15.43:0/68788679 wait complete. 2024-03-28T22:40:21.897 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:21 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/301163613' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:40:21.902 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:40:22.099 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:21 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/301163613' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:40:22.189 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:21 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/301163613' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:40:22.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.159+0000 7ff978a00640 1 -- 172.21.15.43:0/1004119233 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9741097e0 msgr2=0x7ff974109bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:22.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.159+0000 7ff978a00640 1 --2- 172.21.15.43:0/1004119233 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9741097e0 0x7ff974109bc0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff964009930 tx=0x7ff96402bc30 comp rx=0 tx=0).stop 2024-03-28T22:40:22.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.159+0000 7ff978a00640 1 -- 172.21.15.43:0/1004119233 shutdown_connections 2024-03-28T22:40:22.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.159+0000 7ff978a00640 1 --2- 172.21.15.43:0/1004119233 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9741037d0 0x7ff974103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.159+0000 7ff978a00640 1 --2- 172.21.15.43:0/1004119233 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff974104190 0x7ff9741106a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.159+0000 7ff978a00640 1 --2- 172.21.15.43:0/1004119233 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9741097e0 0x7ff974109bc0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.159+0000 7ff978a00640 1 -- 172.21.15.43:0/1004119233 >> 172.21.15.43:0/1004119233 conn(0x7ff9740ff260 msgr2=0x7ff974101680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:22.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.159+0000 7ff978a00640 1 -- 172.21.15.43:0/1004119233 shutdown_connections 2024-03-28T22:40:22.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.159+0000 7ff978a00640 1 -- 172.21.15.43:0/1004119233 wait complete. 2024-03-28T22:40:22.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.160+0000 7ff978a00640 1 Processor -- start 2024-03-28T22:40:22.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.160+0000 7ff978a00640 1 -- start start 2024-03-28T22:40:22.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.160+0000 7ff978a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9741037d0 0x7ff9741105c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.160+0000 7ff978a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff974104190 0x7ff97410d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.160+0000 7ff978a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9741097e0 0x7ff97410dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.160+0000 7ff978a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff974113c00 con 0x7ff9741097e0 2024-03-28T22:40:22.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.160+0000 7ff978a00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ff974113a80 con 0x7ff974104190 2024-03-28T22:40:22.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.160+0000 7ff978a00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7ff974113d80 con 0x7ff9741037d0 2024-03-28T22:40:22.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.161+0000 7ff971e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff974104190 0x7ff97410d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.161+0000 7ff973200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9741097e0 0x7ff97410dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.161+0000 7ff972800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9741037d0 0x7ff9741105c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.161+0000 7ff971e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff974104190 0x7ff97410d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45370/0 (socket says 172.21.15.43:45370) 2024-03-28T22:40:22.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.161+0000 7ff971e00640 1 -- 172.21.15.43:0/2136145214 learned_addr learned my addr 172.21.15.43:0/2136145214 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:22.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.161+0000 7ff971e00640 1 -- 172.21.15.43:0/2136145214 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9741037d0 msgr2=0x7ff9741105c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:22.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.161+0000 7ff971e00640 1 --2- 172.21.15.43:0/2136145214 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9741037d0 0x7ff9741105c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.161+0000 7ff971e00640 1 -- 172.21.15.43:0/2136145214 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9741097e0 msgr2=0x7ff97410dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:22.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.161+0000 7ff971e00640 1 --2- 172.21.15.43:0/2136145214 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9741097e0 0x7ff97410dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.161+0000 7ff971e00640 1 -- 172.21.15.43:0/2136145214 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9741aa900 con 0x7ff974104190 2024-03-28T22:40:22.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.161+0000 7ff972800640 1 --2- 172.21.15.43:0/2136145214 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9741037d0 0x7ff9741105c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:22.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.161+0000 7ff973200640 1 --2- 172.21.15.43:0/2136145214 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9741097e0 0x7ff97410dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:22.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.161+0000 7ff971e00640 1 --2- 172.21.15.43:0/2136145214 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff974104190 0x7ff97410d660 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7ff96000e9f0 tx=0x7ff96000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:22.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.162+0000 7ff96b400640 1 -- 172.21.15.43:0/2136145214 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff96000cd80 con 0x7ff974104190 2024-03-28T22:40:22.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.162+0000 7ff96b400640 1 -- 172.21.15.43:0/2136145214 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff960010910 con 0x7ff974104190 2024-03-28T22:40:22.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.162+0000 7ff978a00640 1 -- 172.21.15.43:0/2136145214 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9741aaad0 con 0x7ff974104190 2024-03-28T22:40:22.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.162+0000 7ff96b400640 1 -- 172.21.15.43:0/2136145214 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff960018990 con 0x7ff974104190 2024-03-28T22:40:22.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.162+0000 7ff978a00640 1 -- 172.21.15.43:0/2136145214 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9741aafc0 con 0x7ff974104190 2024-03-28T22:40:22.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.165+0000 7ff97c060740 1 Processor -- start 2024-03-28T22:40:22.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.165+0000 7ff97c060740 1 -- start start 2024-03-28T22:40:22.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.165+0000 7ff96b400640 1 -- 172.21.15.43:0/2136145214 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7ff960010ac0 con 0x7ff974104190 2024-03-28T22:40:22.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.165+0000 7ff97c060740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a737c5550 0x564a737c5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.165+0000 7ff97c060740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a737c5e70 0x564a737ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.165+0000 7ff97c060740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a737cea50 0x564a737d0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.165+0000 7ff97c060740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x564a73729050 con 0x564a737c5550 2024-03-28T22:40:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.165+0000 7ff97c060740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x564a736d1ce0 con 0x564a737cea50 2024-03-28T22:40:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.165+0000 7ff97c060740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x564a737250f0 con 0x564a737c5e70 2024-03-28T22:40:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.165+0000 7ff973200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a737c5550 0x564a737c5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.165+0000 7ff971e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a737cea50 0x564a737d0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.165+0000 7ff972800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a737c5e70 0x564a737ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.165+0000 7ff972800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a737c5e70 0x564a737ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:48218/0 (socket says 172.21.15.43:48218) 2024-03-28T22:40:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.165+0000 7ff972800640 1 -- 172.21.15.43:0/1210091884 learned_addr learned my addr 172.21.15.43:0/1210091884 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.166+0000 7ff973200640 1 -- 172.21.15.43:0/1210091884 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a737c5e70 msgr2=0x564a737ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.166+0000 7ff973200640 1 --2- 172.21.15.43:0/1210091884 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a737c5e70 0x564a737ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.166+0000 7ff973200640 1 -- 172.21.15.43:0/1210091884 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a737cea50 msgr2=0x564a737d0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.166+0000 7ff973200640 1 --2- 172.21.15.43:0/1210091884 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a737cea50 0x564a737d0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.166+0000 7ff973200640 1 -- 172.21.15.43:0/1210091884 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a7343d3f0 con 0x564a737c5550 2024-03-28T22:40:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.166+0000 7ff971e00640 1 --2- 172.21.15.43:0/1210091884 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a737cea50 0x564a737d0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.166+0000 7ff973200640 1 --2- 172.21.15.43:0/1210091884 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a737c5550 0x564a737c5930 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff96c005fd0 tx=0x7ff96c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aac279c7e08b4829 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.166+0000 7ff95f400640 1 -- 172.21.15.43:0/1210091884 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff96c004d20 con 0x564a737c5550 2024-03-28T22:40:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.167+0000 7ff95f400640 1 -- 172.21.15.43:0/1210091884 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff96c00f9a0 con 0x564a737c5550 2024-03-28T22:40:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.167+0000 7ff95f400640 1 -- 172.21.15.43:0/1210091884 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff96c004470 con 0x564a737c5550 2024-03-28T22:40:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.167+0000 7ff96b400640 1 --2- 172.21.15.43:0/2136145214 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ff9580a1ef0 0x7ff9580a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.167+0000 7ff96b400640 1 -- 172.21.15.43:0/2136145214 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7ff9600ce8d0 con 0x7ff974104190 2024-03-28T22:40:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.167+0000 7ff972800640 1 --2- 172.21.15.43:0/2136145214 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ff9580a1ef0 0x7ff9580a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.167+0000 7ff97c060740 1 -- 172.21.15.43:0/1210091884 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a737c5550 msgr2=0x564a737c5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.167+0000 7ff97c060740 1 --2- 172.21.15.43:0/1210091884 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a737c5550 0x564a737c5930 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff96c005fd0 tx=0x7ff96c011bf0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.167+0000 7ff97c060740 1 -- 172.21.15.43:0/1210091884 shutdown_connections 2024-03-28T22:40:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.167+0000 7ff97c060740 1 --2- 172.21.15.43:0/1210091884 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a737cea50 0x564a737d0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.167+0000 7ff97c060740 1 --2- 172.21.15.43:0/1210091884 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a737c5e70 0x564a737ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.167+0000 7ff97c060740 1 --2- 172.21.15.43:0/1210091884 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a737c5550 0x564a737c5930 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.167+0000 7ff97c060740 1 -- 172.21.15.43:0/1210091884 >> 172.21.15.43:0/1210091884 conn(0x564a737bfc80 msgr2=0x564a737c0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.168+0000 7ff97c060740 1 -- 172.21.15.43:0/1210091884 shutdown_connections 2024-03-28T22:40:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.168+0000 7ff97c060740 1 -- 172.21.15.43:0/1210091884 wait complete. 2024-03-28T22:40:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.168+0000 7ff972800640 1 --2- 172.21.15.43:0/2136145214 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ff9580a1ef0 0x7ff9580a43b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff9640096c0 tx=0x7ff96402df90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.168+0000 7ff97c060740 1 -- start start 2024-03-28T22:40:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.168+0000 7ff97c060740 1 Processor -- start 2024-03-28T22:40:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.169+0000 7ff97c060740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a737c5550 0x564a737cbac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.169+0000 7ff97c060740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a737c5e70 0x564a737cc000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.169+0000 7ff97c060740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a737cea50 0x564a737cc540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.169+0000 7ff97c060740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x564a737d1840 con 0x564a737c5550 2024-03-28T22:40:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.169+0000 7ff97c060740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x564a736c72f0 con 0x564a737c5e70 2024-03-28T22:40:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.169+0000 7ff97c060740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x564a737d19c0 con 0x564a737cea50 2024-03-28T22:40:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.169+0000 7ff97c060740 10 client.0 _ll_register_callbacks cb 0x564a7343cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.169+0000 7ff972800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a737c5550 0x564a737cbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.169+0000 7ff971e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a737c5e70 0x564a737cc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.169+0000 7ff971e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a737c5e70 0x564a737cc000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45390/0 (socket says 172.21.15.43:45390) 2024-03-28T22:40:22.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.169+0000 7ff971e00640 1 -- 172.21.15.43:0/728773384 learned_addr learned my addr 172.21.15.43:0/728773384 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:22.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.169+0000 7ff972800640 1 --2- 172.21.15.43:0/728773384 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a737c5550 0x564a737cbac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37972/0 (socket says 172.21.15.43:37972) 2024-03-28T22:40:22.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.169+0000 7ff973200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a737cea50 0x564a737cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.169+0000 7ff971e00640 1 -- 172.21.15.43:0/728773384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a737cea50 msgr2=0x564a737cc540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:22.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.169+0000 7ff971e00640 1 --2- 172.21.15.43:0/728773384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a737cea50 0x564a737cc540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.169+0000 7ff971e00640 1 -- 172.21.15.43:0/728773384 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a737c5550 msgr2=0x564a737cbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:22.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.169+0000 7ff971e00640 1 --2- 172.21.15.43:0/728773384 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a737c5550 0x564a737cbac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.169+0000 7ff971e00640 1 -- 172.21.15.43:0/728773384 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a735077d0 con 0x564a737c5e70 2024-03-28T22:40:22.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.170+0000 7ff971e00640 1 --2- 172.21.15.43:0/728773384 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a737c5e70 0x564a737cc000 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff96000c6a0 tx=0x7ff9600972d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=892e5cba60dc95ab server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:22.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.170+0000 7ff953e00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:40:22.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.170+0000 7ff953e00640 1 -- 172.21.15.43:0/728773384 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9600cc0b0 con 0x564a737c5e70 2024-03-28T22:40:22.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.170+0000 7ff953e00640 1 -- 172.21.15.43:0/728773384 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7ff9600ce650 con 0x564a737c5e70 2024-03-28T22:40:22.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.170+0000 7ff97c060740 1 -- 172.21.15.43:0/728773384 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff96c00f9a0 con 0x564a737c5e70 2024-03-28T22:40:22.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.170+0000 7ff953e00640 1 -- 172.21.15.43:0/728773384 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff96009af70 con 0x564a737c5e70 2024-03-28T22:40:22.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.170+0000 7ff97c060740 10 client.26804 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:22.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.170+0000 7ff97c060740 1 -- 172.21.15.43:0/728773384 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564a737ddb30 con 0x564a737c5e70 2024-03-28T22:40:22.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.171+0000 7ff953e00640 1 -- 172.21.15.43:0/728773384 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff96c00f9a0 con 0x564a737c5e70 2024-03-28T22:40:22.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.171+0000 7ff953e00640 1 -- 172.21.15.43:0/728773384 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7ff960095ae0 con 0x564a737c5e70 2024-03-28T22:40:22.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.171+0000 7ff97c060740 10 client.26804 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:22.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.171+0000 7ff97c060740 10 client.26804 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:40:22.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.171+0000 7ff97c060740 20 client.26804 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:22.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.171+0000 7ff97c060740 1 --2- 172.21.15.43:0/728773384 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x564a737def40 0x564a737e1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.171+0000 7ff97c060740 4 client.26804 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:40:22.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.171+0000 7ff97c060740 1 -- 172.21.15.43:0/728773384 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a736d2680 con 0x564a737def40 2024-03-28T22:40:22.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.171+0000 7ff972800640 1 --2- 172.21.15.43:0/728773384 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x564a737def40 0x564a737e1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.172+0000 7ff972800640 1 --2- 172.21.15.43:0/728773384 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x564a737def40 0x564a737e1390 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6ac5642ddf543510 server_cookie=110347be18a94a06 in_seq=0 out_seq=0 2024-03-28T22:40:22.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.172+0000 7ff953e00640 10 client.26804 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:40:22.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.173+0000 7ff953e00640 1 -- 172.21.15.43:0/728773384 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564a736d2680 con 0x564a737def40 2024-03-28T22:40:22.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.173+0000 7ff953e00640 10 client.26804 handle_command_reply: tid=0 2024-03-28T22:40:22.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.173+0000 7ff97c060740 1 client.26804 shutdown 2024-03-28T22:40:22.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.173+0000 7ff97c060740 1 -- 172.21.15.43:0/728773384 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a737c5e70 msgr2=0x564a737cc000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:22.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.173+0000 7ff97c060740 1 --2- 172.21.15.43:0/728773384 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a737c5e70 0x564a737cc000 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff96000c6a0 tx=0x7ff9600972d0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.173+0000 7ff97c060740 1 -- 172.21.15.43:0/728773384 shutdown_connections 2024-03-28T22:40:22.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.173+0000 7ff97c060740 1 --2- 172.21.15.43:0/728773384 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x564a737def40 0x564a737e1390 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.173+0000 7ff97c060740 1 --2- 172.21.15.43:0/728773384 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a737c5e70 0x564a737cc000 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.173+0000 7ff97c060740 1 --2- 172.21.15.43:0/728773384 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a737cea50 0x564a737cc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.173+0000 7ff953e00640 0 client.26804 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:40:22.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.173+0000 7ff97c060740 1 --2- 172.21.15.43:0/728773384 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a737c5550 0x564a737cbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.173+0000 7ff97c060740 1 -- 172.21.15.43:0/728773384 >> 172.21.15.43:0/728773384 conn(0x564a737bfc80 msgr2=0x564a737c19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:22.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.173+0000 7ff97c060740 1 -- 172.21.15.43:0/728773384 shutdown_connections 2024-03-28T22:40:22.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.173+0000 7ff97c060740 1 -- 172.21.15.43:0/728773384 wait complete. 2024-03-28T22:40:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.173+0000 7ff97c060740 20 client.26804 trim_cache size 0 max 16384 2024-03-28T22:40:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.188+0000 7ff97c060740 1 Processor -- start 2024-03-28T22:40:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.188+0000 7ff97c060740 1 -- start start 2024-03-28T22:40:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.188+0000 7ff97c060740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a737c5550 0x564a737ce1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.188+0000 7ff97c060740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a737c5e70 0x564a737bac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.188+0000 7ff97c060740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a737cea50 0x564a737da5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.188+0000 7ff97c060740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff9741b4fd0 con 0x564a737c5e70 2024-03-28T22:40:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.188+0000 7ff97c060740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ff9741b6b10 con 0x564a737cea50 2024-03-28T22:40:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.188+0000 7ff97c060740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7ff9741b56a0 con 0x564a737c5550 2024-03-28T22:40:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.189+0000 7ff971e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a737c5e70 0x564a737bac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.189+0000 7ff972800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a737c5550 0x564a737ce1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.189+0000 7ff973200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a737cea50 0x564a737da5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.189+0000 7ff971e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a737c5e70 0x564a737bac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37984/0 (socket says 172.21.15.43:37984) 2024-03-28T22:40:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.189+0000 7ff971e00640 1 -- 172.21.15.43:0/2390737785 learned_addr learned my addr 172.21.15.43:0/2390737785 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.189+0000 7ff971e00640 1 -- 172.21.15.43:0/2390737785 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a737c5550 msgr2=0x564a737ce1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.189+0000 7ff971e00640 1 --2- 172.21.15.43:0/2390737785 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a737c5550 0x564a737ce1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.189+0000 7ff971e00640 1 -- 172.21.15.43:0/2390737785 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a737cea50 msgr2=0x564a737da5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.189+0000 7ff971e00640 1 --2- 172.21.15.43:0/2390737785 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a737cea50 0x564a737da5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.189+0000 7ff971e00640 1 -- 172.21.15.43:0/2390737785 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a737caa70 con 0x564a737c5e70 2024-03-28T22:40:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.189+0000 7ff972800640 1 --2- 172.21.15.43:0/2390737785 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a737c5550 0x564a737ce1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.189+0000 7ff971e00640 1 --2- 172.21.15.43:0/2390737785 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a737c5e70 0x564a737bac50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff96009af00 tx=0x7ff960097340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79afbd5b9ab495a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.190+0000 7ff95ea00640 1 -- 172.21.15.43:0/2390737785 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9600d4260 con 0x564a737c5e70 2024-03-28T22:40:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.190+0000 7ff95ea00640 1 -- 172.21.15.43:0/2390737785 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9600b3020 con 0x564a737c5e70 2024-03-28T22:40:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.190+0000 7ff95ea00640 1 -- 172.21.15.43:0/2390737785 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9600ce6b0 con 0x564a737c5e70 2024-03-28T22:40:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.190+0000 7ff97c060740 1 -- 172.21.15.43:0/2390737785 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a737c5e70 msgr2=0x564a737bac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.190+0000 7ff97c060740 1 --2- 172.21.15.43:0/2390737785 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a737c5e70 0x564a737bac50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff96009af00 tx=0x7ff960097340 comp rx=0 tx=0).stop 2024-03-28T22:40:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.190+0000 7ff97c060740 1 -- 172.21.15.43:0/2390737785 shutdown_connections 2024-03-28T22:40:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.190+0000 7ff97c060740 1 --2- 172.21.15.43:0/2390737785 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a737cea50 0x564a737da5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.190+0000 7ff97c060740 1 --2- 172.21.15.43:0/2390737785 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a737c5e70 0x564a737bac50 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.190+0000 7ff97c060740 1 --2- 172.21.15.43:0/2390737785 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a737c5550 0x564a737ce1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.190+0000 7ff97c060740 1 -- 172.21.15.43:0/2390737785 >> 172.21.15.43:0/2390737785 conn(0x564a737bfc80 msgr2=0x564a737cc310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.191+0000 7ff97c060740 1 -- 172.21.15.43:0/2390737785 shutdown_connections 2024-03-28T22:40:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.191+0000 7ff97c060740 1 -- 172.21.15.43:0/2390737785 wait complete. 2024-03-28T22:40:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.191+0000 7ff97c060740 1 -- start start 2024-03-28T22:40:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.191+0000 7ff97c060740 1 Processor -- start 2024-03-28T22:40:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.191+0000 7ff97c060740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a737c5550 0x564a737d2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.191+0000 7ff97c060740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a737c5e70 0x564a737d3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.191+0000 7ff97c060740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a737cea50 0x564a7383e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.191+0000 7ff97c060740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff9741b6440 con 0x564a737c5550 2024-03-28T22:40:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.191+0000 7ff97c060740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ff9741b5d70 con 0x564a737c5e70 2024-03-28T22:40:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.191+0000 7ff97c060740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7ff9741b2fb0 con 0x564a737cea50 2024-03-28T22:40:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.191+0000 7ff97c060740 10 client.0 _ll_register_callbacks cb 0x7ff96000e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.192+0000 7ff972800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a737c5550 0x564a737d2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.192+0000 7ff973200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a737cea50 0x564a7383e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.192+0000 7ff971e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a737c5e70 0x564a737d3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.192+0000 7ff972800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a737c5550 0x564a737d2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37986/0 (socket says 172.21.15.43:37986) 2024-03-28T22:40:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.192+0000 7ff973200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a737cea50 0x564a7383e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:48242/0 (socket says 172.21.15.43:48242) 2024-03-28T22:40:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.192+0000 7ff972800640 1 -- 172.21.15.43:0/1589761892 learned_addr learned my addr 172.21.15.43:0/1589761892 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.192+0000 7ff972800640 1 -- 172.21.15.43:0/1589761892 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a737cea50 msgr2=0x564a7383e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.192+0000 7ff972800640 1 --2- 172.21.15.43:0/1589761892 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a737cea50 0x564a7383e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.192+0000 7ff972800640 1 -- 172.21.15.43:0/1589761892 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a737c5e70 msgr2=0x564a737d3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.192+0000 7ff972800640 1 --2- 172.21.15.43:0/1589761892 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a737c5e70 0x564a737d3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.192+0000 7ff972800640 1 -- 172.21.15.43:0/1589761892 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a7383e8a0 con 0x564a737c5550 2024-03-28T22:40:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.192+0000 7ff971e00640 1 --2- 172.21.15.43:0/1589761892 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a737c5e70 0x564a737d3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.192+0000 7ff973200640 1 --2- 172.21.15.43:0/1589761892 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a737cea50 0x564a7383e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.192+0000 7ff972800640 1 --2- 172.21.15.43:0/1589761892 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a737c5550 0x564a737d2dd0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff964009230 tx=0x7ff964009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c9f19a8302009d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.192+0000 7ff953e00640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:40:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.193+0000 7ff953e00640 1 -- 172.21.15.43:0/1589761892 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff964046070 con 0x564a737c5550 2024-03-28T22:40:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.193+0000 7ff953e00640 1 -- 172.21.15.43:0/1589761892 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7ff964058690 con 0x564a737c5550 2024-03-28T22:40:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.193+0000 7ff97c060740 1 -- 172.21.15.43:0/1589761892 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9600b3020 con 0x564a737c5550 2024-03-28T22:40:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.193+0000 7ff953e00640 1 -- 172.21.15.43:0/1589761892 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff96406a050 con 0x564a737c5550 2024-03-28T22:40:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.193+0000 7ff97c060740 10 client.16611 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.193+0000 7ff97c060740 1 -- 172.21.15.43:0/1589761892 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564a73845c70 con 0x564a737c5550 2024-03-28T22:40:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.193+0000 7ff953e00640 1 -- 172.21.15.43:0/1589761892 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9600b3020 con 0x564a737c5550 2024-03-28T22:40:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.193+0000 7ff953e00640 1 -- 172.21.15.43:0/1589761892 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7ff964005400 con 0x564a737c5550 2024-03-28T22:40:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.193+0000 7ff97c060740 10 client.16611 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.193+0000 7ff97c060740 10 client.16611 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:40:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.193+0000 7ff97c060740 20 client.16611 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.193+0000 7ff97c060740 1 --2- 172.21.15.43:0/1589761892 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x564a73846370 0x564a73848790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.194+0000 7ff97c060740 4 client.16611 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:40:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.194+0000 7ff97c060740 1 -- 172.21.15.43:0/1589761892 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564a73736590 con 0x564a73846370 2024-03-28T22:40:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.194+0000 7ff971e00640 1 --2- 172.21.15.43:0/1589761892 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x564a73846370 0x564a73848790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.194+0000 7ff971e00640 1 --2- 172.21.15.43:0/1589761892 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x564a73846370 0x564a73848790 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=196a607a481eeea2 server_cookie=a6042dee20b8c847 in_seq=0 out_seq=0 2024-03-28T22:40:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.194+0000 7ff953e00640 10 client.16611 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:40:22.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.194+0000 7ff953e00640 1 -- 172.21.15.43:0/1589761892 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564a73736590 con 0x564a73846370 2024-03-28T22:40:22.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.194+0000 7ff953e00640 10 client.16611 handle_command_reply: tid=0 2024-03-28T22:40:22.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.194+0000 7ff97c060740 1 client.16611 shutdown 2024-03-28T22:40:22.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.195+0000 7ff97c060740 1 -- 172.21.15.43:0/1589761892 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a737c5550 msgr2=0x564a737d2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:22.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.195+0000 7ff97c060740 1 --2- 172.21.15.43:0/1589761892 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a737c5550 0x564a737d2dd0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff964009230 tx=0x7ff964009260 comp rx=0 tx=0).stop 2024-03-28T22:40:22.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.195+0000 7ff97c060740 1 -- 172.21.15.43:0/1589761892 shutdown_connections 2024-03-28T22:40:22.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.195+0000 7ff97c060740 1 --2- 172.21.15.43:0/1589761892 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x564a73846370 0x564a73848790 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.195+0000 7ff97c060740 1 --2- 172.21.15.43:0/1589761892 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a737c5e70 0x564a737d3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.195+0000 7ff97c060740 1 --2- 172.21.15.43:0/1589761892 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a737cea50 0x564a7383e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.195+0000 7ff97c060740 1 --2- 172.21.15.43:0/1589761892 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a737c5550 0x564a737d2dd0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.195+0000 7ff97c060740 1 -- 172.21.15.43:0/1589761892 >> 172.21.15.43:0/1589761892 conn(0x564a737bfc80 msgr2=0x564a737be530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:22.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.195+0000 7ff953e00640 0 client.16611 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:40:22.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.195+0000 7ff97c060740 1 -- 172.21.15.43:0/1589761892 shutdown_connections 2024-03-28T22:40:22.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.195+0000 7ff97c060740 1 -- 172.21.15.43:0/1589761892 wait complete. 2024-03-28T22:40:22.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.195+0000 7ff97c060740 20 client.16611 trim_cache size 0 max 16384 2024-03-28T22:40:22.228 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:40:22.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.196+0000 7ff978a00640 1 -- 172.21.15.43:0/2136145214 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ff9580a1ef0 msgr2=0x7ff9580a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:22.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.196+0000 7ff978a00640 1 --2- 172.21.15.43:0/2136145214 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ff9580a1ef0 0x7ff9580a43b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff9640096c0 tx=0x7ff96402df90 comp rx=0 tx=0).stop 2024-03-28T22:40:22.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.196+0000 7ff978a00640 1 -- 172.21.15.43:0/2136145214 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff974104190 msgr2=0x7ff97410d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:22.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.196+0000 7ff978a00640 1 --2- 172.21.15.43:0/2136145214 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff974104190 0x7ff97410d660 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7ff96000e9f0 tx=0x7ff96000eeb0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.196+0000 7ff978a00640 1 -- 172.21.15.43:0/2136145214 shutdown_connections 2024-03-28T22:40:22.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.196+0000 7ff978a00640 1 --2- 172.21.15.43:0/2136145214 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ff9580a1ef0 0x7ff9580a43b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.196+0000 7ff978a00640 1 --2- 172.21.15.43:0/2136145214 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff974104190 0x7ff97410d660 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.196+0000 7ff978a00640 1 --2- 172.21.15.43:0/2136145214 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff9741097e0 0x7ff97410dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.196+0000 7ff978a00640 1 --2- 172.21.15.43:0/2136145214 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff9741037d0 0x7ff9741105c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.196+0000 7ff978a00640 1 -- 172.21.15.43:0/2136145214 >> 172.21.15.43:0/2136145214 conn(0x7ff9740ff260 msgr2=0x7ff9740ffa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:22.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.196+0000 7ff978a00640 1 -- 172.21.15.43:0/2136145214 shutdown_connections 2024-03-28T22:40:22.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.196+0000 7ff978a00640 1 -- 172.21.15.43:0/2136145214 wait complete. 2024-03-28T22:40:22.236 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:40:22.351 INFO:tasks.workunit.client.0.smithi043.stdout: 1 174607 8.48 MB/sec warmup 113 sec latency 53.526 ms 2024-03-28T22:40:22.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.449+0000 7fccd0c00640 1 -- 172.21.15.43:0/499297025 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcccc1097e0 msgr2=0x7fcccc109bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:22.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.449+0000 7fccd0c00640 1 --2- 172.21.15.43:0/499297025 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcccc1097e0 0x7fcccc109bc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fccbc009930 tx=0x7fccbc02bc30 comp rx=0 tx=0).stop 2024-03-28T22:40:22.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.449+0000 7fccd0c00640 1 -- 172.21.15.43:0/499297025 shutdown_connections 2024-03-28T22:40:22.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.449+0000 7fccd0c00640 1 --2- 172.21.15.43:0/499297025 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcccc104190 0x7fcccc1106a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.449+0000 7fccd0c00640 1 --2- 172.21.15.43:0/499297025 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcccc1037d0 0x7fcccc103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.449+0000 7fccd0c00640 1 --2- 172.21.15.43:0/499297025 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcccc1097e0 0x7fcccc109bc0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.449+0000 7fccd0c00640 1 -- 172.21.15.43:0/499297025 >> 172.21.15.43:0/499297025 conn(0x7fcccc0ff280 msgr2=0x7fcccc1016a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:22.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.449+0000 7fccd0c00640 1 -- 172.21.15.43:0/499297025 shutdown_connections 2024-03-28T22:40:22.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.449+0000 7fccd0c00640 1 -- 172.21.15.43:0/499297025 wait complete. 2024-03-28T22:40:22.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.450+0000 7fccd0c00640 1 Processor -- start 2024-03-28T22:40:22.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.451+0000 7fccd0c00640 1 -- start start 2024-03-28T22:40:22.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.451+0000 7fccd0c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcccc1037d0 0x7fcccc19a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.451+0000 7fccd0c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcccc104190 0x7fcccc19a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.451+0000 7fccd0c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcccc1097e0 0x7fcccc19e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.451+0000 7fccd0c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcccc113bb0 con 0x7fcccc104190 2024-03-28T22:40:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.451+0000 7fccd0c00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fcccc113a30 con 0x7fcccc1037d0 2024-03-28T22:40:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.451+0000 7fccd0c00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fcccc113d30 con 0x7fcccc1097e0 2024-03-28T22:40:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.451+0000 7fcccb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcccc1037d0 0x7fcccc19a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.451+0000 7fcccb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcccc1037d0 0x7fcccc19a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45426/0 (socket says 172.21.15.43:45426) 2024-03-28T22:40:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.451+0000 7fcccb400640 1 -- 172.21.15.43:0/267571218 learned_addr learned my addr 172.21.15.43:0/267571218 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.451+0000 7fcccaa00640 1 --2- 172.21.15.43:0/267571218 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcccc104190 0x7fcccc19a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.451+0000 7fcccbe00640 1 --2- 172.21.15.43:0/267571218 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcccc1097e0 0x7fcccc19e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.452+0000 7fcccb400640 1 -- 172.21.15.43:0/267571218 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcccc1097e0 msgr2=0x7fcccc19e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.452+0000 7fcccb400640 1 --2- 172.21.15.43:0/267571218 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcccc1097e0 0x7fcccc19e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.452+0000 7fcccb400640 1 -- 172.21.15.43:0/267571218 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcccc104190 msgr2=0x7fcccc19a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.452+0000 7fcccb400640 1 --2- 172.21.15.43:0/267571218 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcccc104190 0x7fcccc19a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.452+0000 7fcccb400640 1 -- 172.21.15.43:0/267571218 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcccc19f0a0 con 0x7fcccc1037d0 2024-03-28T22:40:22.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.452+0000 7fcccb400640 1 --2- 172.21.15.43:0/267571218 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcccc1037d0 0x7fcccc19a0b0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fccbc004f50 tx=0x7fccbc02ce20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:22.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.452+0000 7fccc3e00640 1 -- 172.21.15.43:0/267571218 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccbc041070 con 0x7fcccc1037d0 2024-03-28T22:40:22.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.452+0000 7fccd0c00640 1 -- 172.21.15.43:0/267571218 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcccc19f330 con 0x7fcccc1037d0 2024-03-28T22:40:22.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.452+0000 7fccc3e00640 1 -- 172.21.15.43:0/267571218 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccbc0044d0 con 0x7fcccc1037d0 2024-03-28T22:40:22.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.453+0000 7fccc3e00640 1 -- 172.21.15.43:0/267571218 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccbc034440 con 0x7fcccc1037d0 2024-03-28T22:40:22.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.453+0000 7fccd0c00640 1 -- 172.21.15.43:0/267571218 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcccc1a6c20 con 0x7fcccc1037d0 2024-03-28T22:40:22.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.456+0000 7fccd4f94740 1 Processor -- start 2024-03-28T22:40:22.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.456+0000 7fccd4f94740 1 -- start start 2024-03-28T22:40:22.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.456+0000 7fccd4f94740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556f650f1550 0x556f650f1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.456+0000 7fccd4f94740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556f650f1e70 0x556f650fa510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.456+0000 7fccd4f94740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556f650faa50 0x556f650fce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.456+0000 7fccd4f94740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x556f65055050 con 0x556f650f1e70 2024-03-28T22:40:22.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.456+0000 7fccd4f94740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x556f64ffdce0 con 0x556f650faa50 2024-03-28T22:40:22.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.456+0000 7fccd4f94740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x556f650510f0 con 0x556f650f1550 2024-03-28T22:40:22.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.456+0000 7fcccbe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556f650f1550 0x556f650f1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.456+0000 7fccc3e00640 1 -- 172.21.15.43:0/267571218 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fccbc044070 con 0x7fcccc1037d0 2024-03-28T22:40:22.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.456+0000 7fcccaa00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556f650faa50 0x556f650fce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.456+0000 7fcccbe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556f650f1550 0x556f650f1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:48270/0 (socket says 172.21.15.43:48270) 2024-03-28T22:40:22.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.456+0000 7fcccaa00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556f650faa50 0x556f650fce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45438/0 (socket says 172.21.15.43:45438) 2024-03-28T22:40:22.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.456+0000 7fcccbe00640 1 -- 172.21.15.43:0/2840272629 learned_addr learned my addr 172.21.15.43:0/2840272629 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:22.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.457+0000 7fcccb400640 1 --2- 172.21.15.43:0/2840272629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556f650f1e70 0x556f650fa510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.457+0000 7fcccaa00640 1 -- 172.21.15.43:0/2840272629 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556f650f1550 msgr2=0x556f650f1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:22.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.457+0000 7fcccaa00640 1 --2- 172.21.15.43:0/2840272629 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556f650f1550 0x556f650f1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.457+0000 7fcccaa00640 1 -- 172.21.15.43:0/2840272629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556f650f1e70 msgr2=0x556f650fa510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:22.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.457+0000 7fcccaa00640 1 --2- 172.21.15.43:0/2840272629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556f650f1e70 0x556f650fa510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.457+0000 7fcccaa00640 1 -- 172.21.15.43:0/2840272629 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f64d693f0 con 0x556f650faa50 2024-03-28T22:40:22.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.457+0000 7fcccaa00640 1 --2- 172.21.15.43:0/2840272629 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556f650faa50 0x556f650fce40 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fccb800abc0 tx=0x7fccb800f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=266ea41439f25480 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:22.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.457+0000 7fccb7e00640 1 -- 172.21.15.43:0/2840272629 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb8005220 con 0x556f650faa50 2024-03-28T22:40:22.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.457+0000 7fccb7e00640 1 -- 172.21.15.43:0/2840272629 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccb8004c80 con 0x556f650faa50 2024-03-28T22:40:22.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.457+0000 7fccb7e00640 1 -- 172.21.15.43:0/2840272629 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb8018870 con 0x556f650faa50 2024-03-28T22:40:22.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.458+0000 7fccc3e00640 1 --2- 172.21.15.43:0/267571218 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fccb00a1f70 0x7fccb00a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.458+0000 7fccc3e00640 1 -- 172.21.15.43:0/267571218 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fccbc0eff50 con 0x7fcccc1037d0 2024-03-28T22:40:22.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.458+0000 7fcccbe00640 1 --2- 172.21.15.43:0/267571218 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fccb00a1f70 0x7fccb00a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.458+0000 7fccd4f94740 1 -- 172.21.15.43:0/2840272629 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556f650faa50 msgr2=0x556f650fce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:22.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.458+0000 7fccd4f94740 1 --2- 172.21.15.43:0/2840272629 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556f650faa50 0x556f650fce40 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fccb800abc0 tx=0x7fccb800f790 comp rx=0 tx=0).stop 2024-03-28T22:40:22.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.459+0000 7fccd4f94740 1 -- 172.21.15.43:0/2840272629 shutdown_connections 2024-03-28T22:40:22.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.459+0000 7fccd4f94740 1 --2- 172.21.15.43:0/2840272629 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556f650faa50 0x556f650fce40 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.459+0000 7fccd4f94740 1 --2- 172.21.15.43:0/2840272629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556f650f1e70 0x556f650fa510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.459+0000 7fccd4f94740 1 --2- 172.21.15.43:0/2840272629 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556f650f1550 0x556f650f1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.459+0000 7fccd4f94740 1 -- 172.21.15.43:0/2840272629 >> 172.21.15.43:0/2840272629 conn(0x556f650ebc80 msgr2=0x556f650ec060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:22.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.459+0000 7fccd4f94740 1 -- 172.21.15.43:0/2840272629 shutdown_connections 2024-03-28T22:40:22.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.459+0000 7fccd4f94740 1 -- 172.21.15.43:0/2840272629 wait complete. 2024-03-28T22:40:22.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.459+0000 7fcccbe00640 1 --2- 172.21.15.43:0/267571218 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fccb00a1f70 0x7fccb00a4430 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fccc4010430 tx=0x7fccc4007520 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:22.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.459+0000 7fccd4f94740 1 -- start start 2024-03-28T22:40:22.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.459+0000 7fccd4f94740 1 Processor -- start 2024-03-28T22:40:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.460+0000 7fccd4f94740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556f650f1550 0x556f650f7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.460+0000 7fccd4f94740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556f650f1e70 0x556f650f7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.460+0000 7fccd4f94740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556f650faa50 0x556f650f84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.460+0000 7fccd4f94740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x556f650fd840 con 0x556f650f1e70 2024-03-28T22:40:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.460+0000 7fccd4f94740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x556f64ff32f0 con 0x556f650f1550 2024-03-28T22:40:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.460+0000 7fccd4f94740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x556f650fd9c0 con 0x556f650faa50 2024-03-28T22:40:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.460+0000 7fccd4f94740 10 client.0 _ll_register_callbacks cb 0x556f64d68be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.460+0000 7fcccb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556f650f1550 0x556f650f7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.460+0000 7fcccaa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556f650f1e70 0x556f650f7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.460+0000 7fcccb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556f650f1550 0x556f650f7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45440/0 (socket says 172.21.15.43:45440) 2024-03-28T22:40:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.460+0000 7fcccbe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556f650faa50 0x556f650f84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.460+0000 7fcccb400640 1 -- 172.21.15.43:0/1911293117 learned_addr learned my addr 172.21.15.43:0/1911293117 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.461+0000 7fcccb400640 1 -- 172.21.15.43:0/1911293117 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556f650faa50 msgr2=0x556f650f84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.461+0000 7fcccb400640 1 --2- 172.21.15.43:0/1911293117 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556f650faa50 0x556f650f84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.461+0000 7fcccb400640 1 -- 172.21.15.43:0/1911293117 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556f650f1e70 msgr2=0x556f650f7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:22.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.461+0000 7fcccb400640 1 --2- 172.21.15.43:0/1911293117 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556f650f1e70 0x556f650f7fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.461+0000 7fcccb400640 1 -- 172.21.15.43:0/1911293117 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f64e337d0 con 0x556f650f1550 2024-03-28T22:40:22.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.461+0000 7fcccbe00640 1 --2- 172.21.15.43:0/1911293117 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556f650faa50 0x556f650f84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:22.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.461+0000 7fcccb400640 1 --2- 172.21.15.43:0/1911293117 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556f650f1550 0x556f650f7a70 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fccbc0ed410 tx=0x7fccbc003990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a98a0a0ea518088 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:22.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.461+0000 7fccb4c00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:40:22.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.461+0000 7fccb4c00640 1 -- 172.21.15.43:0/1911293117 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccbc04b030 con 0x556f650f1550 2024-03-28T22:40:22.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.461+0000 7fccb4c00640 1 -- 172.21.15.43:0/1911293117 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fccbc0dc070 con 0x556f650f1550 2024-03-28T22:40:22.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.461+0000 7fccd4f94740 1 -- 172.21.15.43:0/1911293117 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fccb8004c80 con 0x556f650f1550 2024-03-28T22:40:22.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.462+0000 7fccb4c00640 1 -- 172.21.15.43:0/1911293117 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fccbc0bc190 con 0x556f650f1550 2024-03-28T22:40:22.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.462+0000 7fccd4f94740 10 client.26819 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:22.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.462+0000 7fccd4f94740 1 -- 172.21.15.43:0/1911293117 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556f65109af0 con 0x556f650f1550 2024-03-28T22:40:22.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.462+0000 7fccb4c00640 1 -- 172.21.15.43:0/1911293117 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccb8004c80 con 0x556f650f1550 2024-03-28T22:40:22.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.462+0000 7fccb4c00640 1 -- 172.21.15.43:0/1911293117 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fccbc0c4960 con 0x556f650f1550 2024-03-28T22:40:22.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.462+0000 7fccd4f94740 10 client.26819 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:22.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.462+0000 7fccd4f94740 10 client.26819 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:40:22.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.462+0000 7fccd4f94740 20 client.26819 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:22.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.462+0000 7fccd4f94740 1 --2- 172.21.15.43:0/1911293117 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x556f6510af00 0x556f6510d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.462+0000 7fccd4f94740 4 client.26819 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:40:22.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.462+0000 7fccd4f94740 1 -- 172.21.15.43:0/1911293117 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f64ffe680 con 0x556f6510af00 2024-03-28T22:40:22.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.462+0000 7fcccaa00640 1 --2- 172.21.15.43:0/1911293117 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x556f6510af00 0x556f6510d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.463+0000 7fcccaa00640 1 --2- 172.21.15.43:0/1911293117 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x556f6510af00 0x556f6510d390 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=346ba5043c8fb9df server_cookie=490373105ee205c3 in_seq=0 out_seq=0 2024-03-28T22:40:22.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.463+0000 7fccb4c00640 10 client.26819 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:40:22.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.464+0000 7fccb4c00640 1 -- 172.21.15.43:0/1911293117 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556f64ffe680 con 0x556f6510af00 2024-03-28T22:40:22.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.464+0000 7fccb4c00640 10 client.26819 handle_command_reply: tid=0 2024-03-28T22:40:22.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.464+0000 7fccd4f94740 1 client.26819 shutdown 2024-03-28T22:40:22.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.464+0000 7fccd4f94740 1 -- 172.21.15.43:0/1911293117 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556f650f1550 msgr2=0x556f650f7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:22.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.464+0000 7fccd4f94740 1 --2- 172.21.15.43:0/1911293117 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556f650f1550 0x556f650f7a70 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fccbc0ed410 tx=0x7fccbc003990 comp rx=0 tx=0).stop 2024-03-28T22:40:22.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.464+0000 7fccd4f94740 1 -- 172.21.15.43:0/1911293117 shutdown_connections 2024-03-28T22:40:22.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.464+0000 7fccd4f94740 1 --2- 172.21.15.43:0/1911293117 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x556f6510af00 0x556f6510d390 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.464+0000 7fccd4f94740 1 --2- 172.21.15.43:0/1911293117 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556f650faa50 0x556f650f84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.464+0000 7fccd4f94740 1 --2- 172.21.15.43:0/1911293117 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556f650f1e70 0x556f650f7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.464+0000 7fccd4f94740 1 --2- 172.21.15.43:0/1911293117 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556f650f1550 0x556f650f7a70 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.464+0000 7fccd4f94740 1 -- 172.21.15.43:0/1911293117 >> 172.21.15.43:0/1911293117 conn(0x556f650ebc80 msgr2=0x556f650ed9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:22.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.464+0000 7fccb4c00640 0 client.26819 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:40:22.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.464+0000 7fccd4f94740 1 -- 172.21.15.43:0/1911293117 shutdown_connections 2024-03-28T22:40:22.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.464+0000 7fccd4f94740 1 -- 172.21.15.43:0/1911293117 wait complete. 2024-03-28T22:40:22.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.464+0000 7fccd4f94740 20 client.26819 trim_cache size 0 max 16384 2024-03-28T22:40:22.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.481+0000 7fccd4f94740 1 Processor -- start 2024-03-28T22:40:22.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.481+0000 7fccd4f94740 1 -- start start 2024-03-28T22:40:22.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.481+0000 7fccd4f94740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556f650f1550 0x556f65106610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.481+0000 7fccd4f94740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556f650f1e70 0x556f650eac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.481+0000 7fccd4f94740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556f650faa50 0x556f65112c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.481+0000 7fccd4f94740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcccc1b0b40 con 0x556f650f1550 2024-03-28T22:40:22.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.481+0000 7fccd4f94740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fcccc1b2680 con 0x556f650faa50 2024-03-28T22:40:22.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.481+0000 7fccd4f94740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fcccc1b1210 con 0x556f650f1e70 2024-03-28T22:40:22.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.482+0000 7fcccaa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556f650f1e70 0x556f650eac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.482+0000 7fcccb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556f650f1550 0x556f65106610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.482+0000 7fcccaa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556f650f1e70 0x556f650eac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:48288/0 (socket says 172.21.15.43:48288) 2024-03-28T22:40:22.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.482+0000 7fcccaa00640 1 -- 172.21.15.43:0/4227416734 learned_addr learned my addr 172.21.15.43:0/4227416734 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:22.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.482+0000 7fcccbe00640 1 --2- 172.21.15.43:0/4227416734 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556f650faa50 0x556f65112c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.482+0000 7fcccb400640 1 -- 172.21.15.43:0/4227416734 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556f650f1e70 msgr2=0x556f650eac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:22.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.482+0000 7fcccb400640 1 --2- 172.21.15.43:0/4227416734 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556f650f1e70 0x556f650eac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.482+0000 7fcccb400640 1 -- 172.21.15.43:0/4227416734 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556f650faa50 msgr2=0x556f65112c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:22.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.482+0000 7fcccb400640 1 --2- 172.21.15.43:0/4227416734 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556f650faa50 0x556f65112c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.482+0000 7fcccb400640 1 -- 172.21.15.43:0/4227416734 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f650fce40 con 0x556f650f1550 2024-03-28T22:40:22.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.482+0000 7fcccbe00640 1 --2- 172.21.15.43:0/4227416734 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556f650faa50 0x556f65112c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:40:22.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.483+0000 7fcccb400640 1 --2- 172.21.15.43:0/4227416734 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556f650f1550 0x556f65106610 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fccbc0f6040 tx=0x7fccbc02c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85271da5017c54d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:22.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.483+0000 7fccb7400640 1 -- 172.21.15.43:0/4227416734 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccbc04b070 con 0x556f650f1550 2024-03-28T22:40:22.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.483+0000 7fccb7400640 1 -- 172.21.15.43:0/4227416734 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccbc0bcd50 con 0x556f650f1550 2024-03-28T22:40:22.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.483+0000 7fccb7400640 1 -- 172.21.15.43:0/4227416734 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccbc0bbfb0 con 0x556f650f1550 2024-03-28T22:40:22.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.484+0000 7fccd4f94740 1 -- 172.21.15.43:0/4227416734 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556f650f1550 msgr2=0x556f65106610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:22.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.484+0000 7fccd4f94740 1 --2- 172.21.15.43:0/4227416734 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556f650f1550 0x556f65106610 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fccbc0f6040 tx=0x7fccbc02c450 comp rx=0 tx=0).stop 2024-03-28T22:40:22.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.484+0000 7fccd4f94740 1 -- 172.21.15.43:0/4227416734 shutdown_connections 2024-03-28T22:40:22.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.484+0000 7fccd4f94740 1 --2- 172.21.15.43:0/4227416734 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556f650faa50 0x556f65112c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.484+0000 7fccd4f94740 1 --2- 172.21.15.43:0/4227416734 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556f650f1e70 0x556f650eac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.484+0000 7fccd4f94740 1 --2- 172.21.15.43:0/4227416734 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556f650f1550 0x556f65106610 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.484+0000 7fccd4f94740 1 -- 172.21.15.43:0/4227416734 >> 172.21.15.43:0/4227416734 conn(0x556f650ebc80 msgr2=0x556f650ea530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:22.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.484+0000 7fccd4f94740 1 -- 172.21.15.43:0/4227416734 shutdown_connections 2024-03-28T22:40:22.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.484+0000 7fccd4f94740 1 -- 172.21.15.43:0/4227416734 wait complete. 2024-03-28T22:40:22.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.485+0000 7fccd4f94740 1 -- start start 2024-03-28T22:40:22.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.485+0000 7fccd4f94740 1 Processor -- start 2024-03-28T22:40:22.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.485+0000 7fccd4f94740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556f650f1550 0x556f65102c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.485+0000 7fccd4f94740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556f650f1e70 0x556f650fe180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.485+0000 7fccd4f94740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556f650faa50 0x556f650fe6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.485+0000 7fccd4f94740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcccc1b1fb0 con 0x556f650f1550 2024-03-28T22:40:22.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.485+0000 7fccd4f94740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fcccc1b18e0 con 0x556f650faa50 2024-03-28T22:40:22.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.485+0000 7fccd4f94740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fcccc1aeb20 con 0x556f650f1e70 2024-03-28T22:40:22.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.485+0000 7fccd4f94740 10 client.0 _ll_register_callbacks cb 0x7fccbc009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:22.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.486+0000 7fcccbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556f650faa50 0x556f650fe6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.486+0000 7fcccb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556f650f1550 0x556f65102c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.486+0000 7fcccaa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556f650f1e70 0x556f650fe180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.486+0000 7fcccbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556f650faa50 0x556f650fe6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45456/0 (socket says 172.21.15.43:45456) 2024-03-28T22:40:22.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.486+0000 7fcccbe00640 1 -- 172.21.15.43:0/552031982 learned_addr learned my addr 172.21.15.43:0/552031982 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:22.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.486+0000 7fcccaa00640 1 -- 172.21.15.43:0/552031982 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556f650faa50 msgr2=0x556f650fe6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:22.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.486+0000 7fcccaa00640 1 --2- 172.21.15.43:0/552031982 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556f650faa50 0x556f650fe6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.486+0000 7fcccaa00640 1 -- 172.21.15.43:0/552031982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556f650f1550 msgr2=0x556f65102c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:22.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.486+0000 7fcccaa00640 1 --2- 172.21.15.43:0/552031982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556f650f1550 0x556f65102c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.486+0000 7fcccaa00640 1 -- 172.21.15.43:0/552031982 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f650fec80 con 0x556f650f1e70 2024-03-28T22:40:22.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.486+0000 7fcccbe00640 1 --2- 172.21.15.43:0/552031982 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556f650faa50 0x556f650fe6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:22.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.486+0000 7fcccaa00640 1 --2- 172.21.15.43:0/552031982 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556f650f1e70 0x556f650fe180 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fccb8004f50 tx=0x7fccb800a880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9f8b567c6bc1823 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:22.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.486+0000 7fccb4c00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:40:22.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.487+0000 7fccb4c00640 1 -- 172.21.15.43:0/552031982 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb8004870 con 0x556f650f1e70 2024-03-28T22:40:22.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.487+0000 7fccd4f94740 1 -- 172.21.15.43:0/552031982 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fccbc0bcd50 con 0x556f650f1e70 2024-03-28T22:40:22.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.487+0000 7fccb4c00640 1 -- 172.21.15.43:0/552031982 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fccb8004a10 con 0x556f650f1e70 2024-03-28T22:40:22.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.487+0000 7fccb4c00640 1 -- 172.21.15.43:0/552031982 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fccb803d050 con 0x556f650f1e70 2024-03-28T22:40:22.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.487+0000 7fccd4f94740 10 client.26776 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:22.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.487+0000 7fccd4f94740 1 -- 172.21.15.43:0/552031982 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556f65170f60 con 0x556f650f1e70 2024-03-28T22:40:22.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.487+0000 7fccb4c00640 1 -- 172.21.15.43:0/552031982 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccbc0bcd50 con 0x556f650f1e70 2024-03-28T22:40:22.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.487+0000 7fccb4c00640 1 -- 172.21.15.43:0/552031982 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fccb80186d0 con 0x556f650f1e70 2024-03-28T22:40:22.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.487+0000 7fccd4f94740 10 client.26776 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:22.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.488+0000 7fccd4f94740 10 client.26776 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:40:22.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.488+0000 7fccd4f94740 20 client.26776 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:22.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.488+0000 7fccd4f94740 1 --2- 172.21.15.43:0/552031982 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x556f65171440 0x556f65173830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.488+0000 7fccd4f94740 4 client.26776 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:40:22.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.488+0000 7fccd4f94740 1 -- 172.21.15.43:0/552031982 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556f65062590 con 0x556f65171440 2024-03-28T22:40:22.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.488+0000 7fcccb400640 1 --2- 172.21.15.43:0/552031982 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x556f65171440 0x556f65173830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.488+0000 7fcccb400640 1 --2- 172.21.15.43:0/552031982 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x556f65171440 0x556f65173830 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=12eb588c6657d32c server_cookie=dea1d47caa86d7fd in_seq=0 out_seq=0 2024-03-28T22:40:22.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.488+0000 7fccb4c00640 10 client.26776 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:40:22.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.488+0000 7fccb4c00640 1 -- 172.21.15.43:0/552031982 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556f65062590 con 0x556f65171440 2024-03-28T22:40:22.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.489+0000 7fccb4c00640 10 client.26776 handle_command_reply: tid=0 2024-03-28T22:40:22.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.489+0000 7fccd4f94740 1 client.26776 shutdown 2024-03-28T22:40:22.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.489+0000 7fccd4f94740 1 -- 172.21.15.43:0/552031982 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556f650f1e70 msgr2=0x556f650fe180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:22.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.489+0000 7fccd4f94740 1 --2- 172.21.15.43:0/552031982 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556f650f1e70 0x556f650fe180 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fccb8004f50 tx=0x7fccb800a880 comp rx=0 tx=0).stop 2024-03-28T22:40:22.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.489+0000 7fccd4f94740 1 -- 172.21.15.43:0/552031982 shutdown_connections 2024-03-28T22:40:22.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.489+0000 7fccd4f94740 1 --2- 172.21.15.43:0/552031982 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x556f65171440 0x556f65173830 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.489+0000 7fccd4f94740 1 --2- 172.21.15.43:0/552031982 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556f650faa50 0x556f650fe6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.489+0000 7fccd4f94740 1 --2- 172.21.15.43:0/552031982 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556f650f1e70 0x556f650fe180 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.489+0000 7fccd4f94740 1 --2- 172.21.15.43:0/552031982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556f650f1550 0x556f65102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.489+0000 7fccd4f94740 1 -- 172.21.15.43:0/552031982 >> 172.21.15.43:0/552031982 conn(0x556f650ebc80 msgr2=0x556f650ed860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:22.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.489+0000 7fccb4c00640 0 client.26776 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:40:22.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.489+0000 7fccd4f94740 1 -- 172.21.15.43:0/552031982 shutdown_connections 2024-03-28T22:40:22.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.489+0000 7fccd4f94740 1 -- 172.21.15.43:0/552031982 wait complete. 2024-03-28T22:40:22.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.489+0000 7fccd4f94740 20 client.26776 trim_cache size 0 max 16384 2024-03-28T22:40:22.491 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:40:22.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.490+0000 7fccd0c00640 1 -- 172.21.15.43:0/267571218 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fccb00a1f70 msgr2=0x7fccb00a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:22.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.490+0000 7fccd0c00640 1 --2- 172.21.15.43:0/267571218 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fccb00a1f70 0x7fccb00a4430 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fccc4010430 tx=0x7fccc4007520 comp rx=0 tx=0).stop 2024-03-28T22:40:22.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.490+0000 7fccd0c00640 1 -- 172.21.15.43:0/267571218 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcccc1037d0 msgr2=0x7fcccc19a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:22.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.490+0000 7fccd0c00640 1 --2- 172.21.15.43:0/267571218 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcccc1037d0 0x7fcccc19a0b0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fccbc004f50 tx=0x7fccbc02ce20 comp rx=0 tx=0).stop 2024-03-28T22:40:22.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.490+0000 7fccd0c00640 1 -- 172.21.15.43:0/267571218 shutdown_connections 2024-03-28T22:40:22.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.490+0000 7fccd0c00640 1 --2- 172.21.15.43:0/267571218 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fccb00a1f70 0x7fccb00a4430 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.490+0000 7fccd0c00640 1 --2- 172.21.15.43:0/267571218 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcccc1097e0 0x7fcccc19e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.490+0000 7fccd0c00640 1 --2- 172.21.15.43:0/267571218 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcccc104190 0x7fcccc19a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.490+0000 7fccd0c00640 1 --2- 172.21.15.43:0/267571218 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcccc1037d0 0x7fcccc19a0b0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.490+0000 7fccd0c00640 1 -- 172.21.15.43:0/267571218 >> 172.21.15.43:0/267571218 conn(0x7fcccc0ff280 msgr2=0x7fcccc0ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:22.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.490+0000 7fccd0c00640 1 -- 172.21.15.43:0/267571218 shutdown_connections 2024-03-28T22:40:22.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.490+0000 7fccd0c00640 1 -- 172.21.15.43:0/267571218 wait complete. 2024-03-28T22:40:22.505 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:40:22.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.719+0000 7f4fa6e00640 1 -- 172.21.15.43:0/1557833122 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fa0102780 msgr2=0x7f4fa0102b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:22.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.719+0000 7f4fa6e00640 1 --2- 172.21.15.43:0/1557833122 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fa0102780 0x7f4fa0102b60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4f90009ca0 tx=0x7f4f9002c180 comp rx=0 tx=0).stop 2024-03-28T22:40:22.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.719+0000 7f4fa6e00640 1 -- 172.21.15.43:0/1557833122 shutdown_connections 2024-03-28T22:40:22.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.719+0000 7f4fa6e00640 1 --2- 172.21.15.43:0/1557833122 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4fa010bba0 0x7f4fa010e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.719+0000 7f4fa6e00640 1 --2- 172.21.15.43:0/1557833122 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4fa01030a0 0x7f4fa010b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.719+0000 7f4fa6e00640 1 --2- 172.21.15.43:0/1557833122 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fa0102780 0x7f4fa0102b60 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.719+0000 7f4fa6e00640 1 -- 172.21.15.43:0/1557833122 >> 172.21.15.43:0/1557833122 conn(0x7f4fa00fbfa0 msgr2=0x7f4fa00fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:22.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.719+0000 7f4fa6e00640 1 -- 172.21.15.43:0/1557833122 shutdown_connections 2024-03-28T22:40:22.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.720+0000 7f4fa6e00640 1 -- 172.21.15.43:0/1557833122 wait complete. 2024-03-28T22:40:22.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.720+0000 7f4fa6e00640 1 Processor -- start 2024-03-28T22:40:22.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.721+0000 7f4fa6e00640 1 -- start start 2024-03-28T22:40:22.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.721+0000 7f4fa6e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4fa0102780 0x7f4fa0197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.721+0000 7f4fa6e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4fa01030a0 0x7f4fa0198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.721+0000 7f4fa6e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fa010bba0 0x7f4fa019c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.721+0000 7f4fa6e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa01119a0 con 0x7f4fa010bba0 2024-03-28T22:40:22.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.721+0000 7f4fa6e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa0111820 con 0x7f4fa01030a0 2024-03-28T22:40:22.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.721+0000 7f4fa6e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa0111b20 con 0x7f4fa0102780 2024-03-28T22:40:22.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.721+0000 7f4fa5000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4fa01030a0 0x7f4fa0198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.721+0000 7f4fa5a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4fa0102780 0x7f4fa0197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.721+0000 7f4fa5000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4fa01030a0 0x7f4fa0198380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45478/0 (socket says 172.21.15.43:45478) 2024-03-28T22:40:22.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.721+0000 7f4fa5000640 1 -- 172.21.15.43:0/2833173436 learned_addr learned my addr 172.21.15.43:0/2833173436 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:22.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.722+0000 7f4fa6400640 1 --2- 172.21.15.43:0/2833173436 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fa010bba0 0x7f4fa019c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.722+0000 7f4fa5000640 1 -- 172.21.15.43:0/2833173436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4fa0102780 msgr2=0x7f4fa0197e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:22.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.722+0000 7f4fa5000640 1 --2- 172.21.15.43:0/2833173436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4fa0102780 0x7f4fa0197e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.722+0000 7f4fa5000640 1 -- 172.21.15.43:0/2833173436 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fa010bba0 msgr2=0x7f4fa019c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:22.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.722+0000 7f4fa5000640 1 --2- 172.21.15.43:0/2833173436 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fa010bba0 0x7f4fa019c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.722+0000 7f4fa5000640 1 -- 172.21.15.43:0/2833173436 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fa019ce30 con 0x7f4fa01030a0 2024-03-28T22:40:22.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.722+0000 7f4fa5a00640 1 --2- 172.21.15.43:0/2833173436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4fa0102780 0x7f4fa0197e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:22.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.722+0000 7f4fa5000640 1 --2- 172.21.15.43:0/2833173436 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4fa01030a0 0x7f4fa0198380 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c00e9f0 tx=0x7f4f8c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:22.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.722+0000 7f4f9e000640 1 -- 172.21.15.43:0/2833173436 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c00ccc0 con 0x7f4fa01030a0 2024-03-28T22:40:22.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.722+0000 7f4fa6e00640 1 -- 172.21.15.43:0/2833173436 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fa019d120 con 0x7f4fa01030a0 2024-03-28T22:40:22.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.722+0000 7f4f9e000640 1 -- 172.21.15.43:0/2833173436 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f8c010910 con 0x7f4fa01030a0 2024-03-28T22:40:22.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.722+0000 7f4f9e000640 1 -- 172.21.15.43:0/2833173436 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c018920 con 0x7f4fa01030a0 2024-03-28T22:40:22.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.723+0000 7f4fa6e00640 1 -- 172.21.15.43:0/2833173436 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fa01a49b0 con 0x7f4fa01030a0 2024-03-28T22:40:22.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.726+0000 7f4fab1a8740 1 Processor -- start 2024-03-28T22:40:22.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.726+0000 7f4fab1a8740 1 -- start start 2024-03-28T22:40:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.727+0000 7f4f9e000640 1 -- 172.21.15.43:0/2833173436 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f4f8c010430 con 0x7f4fa01030a0 2024-03-28T22:40:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.727+0000 7f4fab1a8740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d3e7357550 0x55d3e7357930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.727+0000 7f4fab1a8740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d3e7357e70 0x55d3e7360510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.727+0000 7f4fab1a8740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3e7360a50 0x55d3e7362e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.727+0000 7f4fab1a8740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d3e72bb050 con 0x55d3e7357550 2024-03-28T22:40:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.727+0000 7f4fab1a8740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55d3e7263ce0 con 0x55d3e7360a50 2024-03-28T22:40:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.727+0000 7f4fab1a8740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55d3e72b70f0 con 0x55d3e7357e70 2024-03-28T22:40:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.727+0000 7f4fa5000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3e7360a50 0x55d3e7362e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.727+0000 7f4fa5a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d3e7357e70 0x55d3e7360510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.727+0000 7f4fa6400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d3e7357550 0x55d3e7357930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.727+0000 7f4fa5a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d3e7357e70 0x55d3e7360510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:48330/0 (socket says 172.21.15.43:48330) 2024-03-28T22:40:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.727+0000 7f4fa5a00640 1 -- 172.21.15.43:0/4210763390 learned_addr learned my addr 172.21.15.43:0/4210763390 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.727+0000 7f4fa5a00640 1 -- 172.21.15.43:0/4210763390 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3e7360a50 msgr2=0x55d3e7362e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.727+0000 7f4fa5a00640 1 --2- 172.21.15.43:0/4210763390 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3e7360a50 0x55d3e7362e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.727+0000 7f4fa5a00640 1 -- 172.21.15.43:0/4210763390 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d3e7357550 msgr2=0x55d3e7357930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.727+0000 7f4fa5a00640 1 --2- 172.21.15.43:0/4210763390 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d3e7357550 0x55d3e7357930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.728+0000 7f4fa5a00640 1 -- 172.21.15.43:0/4210763390 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3e6fcf3f0 con 0x55d3e7357e70 2024-03-28T22:40:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.728+0000 7f4fa5a00640 1 --2- 172.21.15.43:0/4210763390 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d3e7357e70 0x55d3e7360510 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4f90037040 tx=0x7f4f900098f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=882381f185c6057 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.728+0000 7f4f96a00640 1 -- 172.21.15.43:0/4210763390 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f90043070 con 0x55d3e7357e70 2024-03-28T22:40:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.728+0000 7f4f96a00640 1 -- 172.21.15.43:0/4210763390 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f90003e50 con 0x55d3e7357e70 2024-03-28T22:40:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.728+0000 7f4f96a00640 1 -- 172.21.15.43:0/4210763390 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f90039040 con 0x55d3e7357e70 2024-03-28T22:40:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.729+0000 7f4f9e000640 1 --2- 172.21.15.43:0/2833173436 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4f840a1ef0 0x7f4f840a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.729+0000 7f4f9e000640 1 -- 172.21.15.43:0/2833173436 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f4f8c0ce8b0 con 0x7f4fa01030a0 2024-03-28T22:40:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.729+0000 7f4fa6400640 1 --2- 172.21.15.43:0/2833173436 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4f840a1ef0 0x7f4f840a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.729+0000 7f4fab1a8740 1 -- 172.21.15.43:0/4210763390 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d3e7357e70 msgr2=0x55d3e7360510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.729+0000 7f4fab1a8740 1 --2- 172.21.15.43:0/4210763390 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d3e7357e70 0x55d3e7360510 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4f90037040 tx=0x7f4f900098f0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.730+0000 7f4fab1a8740 1 -- 172.21.15.43:0/4210763390 shutdown_connections 2024-03-28T22:40:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.730+0000 7f4fab1a8740 1 --2- 172.21.15.43:0/4210763390 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3e7360a50 0x55d3e7362e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.730+0000 7f4fab1a8740 1 --2- 172.21.15.43:0/4210763390 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d3e7357e70 0x55d3e7360510 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.730+0000 7f4fab1a8740 1 --2- 172.21.15.43:0/4210763390 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d3e7357550 0x55d3e7357930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.730+0000 7f4fab1a8740 1 -- 172.21.15.43:0/4210763390 >> 172.21.15.43:0/4210763390 conn(0x55d3e7351c80 msgr2=0x55d3e7352060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.730+0000 7f4fab1a8740 1 -- 172.21.15.43:0/4210763390 shutdown_connections 2024-03-28T22:40:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.730+0000 7f4fa6400640 1 --2- 172.21.15.43:0/2833173436 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4f840a1ef0 0x7f4f840a43b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4f98009c70 tx=0x7f4f98005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.730+0000 7f4fab1a8740 1 -- 172.21.15.43:0/4210763390 wait complete. 2024-03-28T22:40:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.730+0000 7f4fab1a8740 1 -- start start 2024-03-28T22:40:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.730+0000 7f4fab1a8740 1 Processor -- start 2024-03-28T22:40:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.731+0000 7f4fab1a8740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d3e7357550 0x55d3e735daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.731+0000 7f4fab1a8740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3e7357e70 0x55d3e735e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.731+0000 7f4fab1a8740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d3e7360a50 0x55d3e735e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.731+0000 7f4fab1a8740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d3e73637e0 con 0x55d3e7360a50 2024-03-28T22:40:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.731+0000 7f4fab1a8740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55d3e72592f0 con 0x55d3e7357e70 2024-03-28T22:40:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.731+0000 7f4fab1a8740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55d3e7363960 con 0x55d3e7357550 2024-03-28T22:40:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.731+0000 7f4fab1a8740 10 client.0 _ll_register_callbacks cb 0x55d3e6fcebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.731+0000 7f4fa5a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d3e7357550 0x55d3e735daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.731+0000 7f4fa5000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3e7357e70 0x55d3e735e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.731+0000 7f4fa5a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d3e7357550 0x55d3e735daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:48332/0 (socket says 172.21.15.43:48332) 2024-03-28T22:40:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.731+0000 7f4fa5a00640 1 -- 172.21.15.43:0/4124750198 learned_addr learned my addr 172.21.15.43:0/4124750198 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.732+0000 7f4fa6400640 1 --2- 172.21.15.43:0/4124750198 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d3e7360a50 0x55d3e735e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.732+0000 7f4fa5a00640 1 -- 172.21.15.43:0/4124750198 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3e7357e70 msgr2=0x55d3e735e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.732+0000 7f4fa5a00640 1 --2- 172.21.15.43:0/4124750198 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3e7357e70 0x55d3e735e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.732+0000 7f4fa5a00640 1 -- 172.21.15.43:0/4124750198 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d3e7360a50 msgr2=0x55d3e735e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:22.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.732+0000 7f4fa5a00640 1 --2- 172.21.15.43:0/4124750198 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d3e7360a50 0x55d3e735e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.732+0000 7f4fa5a00640 1 -- 172.21.15.43:0/4124750198 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3e70997d0 con 0x55d3e7357550 2024-03-28T22:40:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.732+0000 7f4fa5a00640 1 --2- 172.21.15.43:0/4124750198 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d3e7357550 0x55d3e735daf0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4f90009bf0 tx=0x7f4f9002c180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e209c8845872774 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.732+0000 7f4f8b400640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:40:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.734+0000 7f4f8b400640 1 -- 172.21.15.43:0/4124750198 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f9004c070 con 0x55d3e7357550 2024-03-28T22:40:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.734+0000 7f4f8b400640 1 -- 172.21.15.43:0/4124750198 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f4f900349f0 con 0x55d3e7357550 2024-03-28T22:40:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.734+0000 7f4fab1a8740 1 -- 172.21.15.43:0/4124750198 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f90003e50 con 0x55d3e7357550 2024-03-28T22:40:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.735+0000 7f4f8b400640 1 -- 172.21.15.43:0/4124750198 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4f90058050 con 0x55d3e7357550 2024-03-28T22:40:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.735+0000 7f4fab1a8740 10 client.26788 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.735+0000 7f4fab1a8740 1 -- 172.21.15.43:0/4124750198 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d3e73637e0 con 0x55d3e7357550 2024-03-28T22:40:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.735+0000 7f4f8b400640 1 -- 172.21.15.43:0/4124750198 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f90003e50 con 0x55d3e7357550 2024-03-28T22:40:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.735+0000 7f4f8b400640 1 -- 172.21.15.43:0/4124750198 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4f90032970 con 0x55d3e7357550 2024-03-28T22:40:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.735+0000 7f4fab1a8740 10 client.26788 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.735+0000 7f4fab1a8740 10 client.26788 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:40:22.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.735+0000 7f4fab1a8740 20 client.26788 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:22.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.735+0000 7f4fab1a8740 1 --2- 172.21.15.43:0/4124750198 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d3e7370c60 0x55d3e7373050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.735+0000 7f4fab1a8740 4 client.26788 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:40:22.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.735+0000 7f4fab1a8740 1 -- 172.21.15.43:0/4124750198 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d3e7264680 con 0x55d3e7370c60 2024-03-28T22:40:22.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.736+0000 7f4fa5000640 1 --2- 172.21.15.43:0/4124750198 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d3e7370c60 0x55d3e7373050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.736+0000 7f4fa5000640 1 --2- 172.21.15.43:0/4124750198 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d3e7370c60 0x55d3e7373050 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9560491c73411c81 server_cookie=5027fa35a0990cc8 in_seq=0 out_seq=0 2024-03-28T22:40:22.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.736+0000 7f4f8b400640 10 client.26788 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:22.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.738+0000 7f4f8b400640 1 -- 172.21.15.43:0/4124750198 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d3e7264680 con 0x55d3e7370c60 2024-03-28T22:40:22.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.738+0000 7f4f8b400640 10 client.26788 handle_command_reply: tid=0 2024-03-28T22:40:22.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.738+0000 7f4fab1a8740 1 client.26788 shutdown 2024-03-28T22:40:22.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.739+0000 7f4fab1a8740 1 -- 172.21.15.43:0/4124750198 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d3e7357550 msgr2=0x55d3e735daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:22.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.739+0000 7f4fab1a8740 1 --2- 172.21.15.43:0/4124750198 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d3e7357550 0x55d3e735daf0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4f90009bf0 tx=0x7f4f9002c180 comp rx=0 tx=0).stop 2024-03-28T22:40:22.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.739+0000 7f4fab1a8740 1 -- 172.21.15.43:0/4124750198 shutdown_connections 2024-03-28T22:40:22.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.739+0000 7f4fab1a8740 1 --2- 172.21.15.43:0/4124750198 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d3e7370c60 0x55d3e7373050 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.739+0000 7f4fab1a8740 1 --2- 172.21.15.43:0/4124750198 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3e7357e70 0x55d3e735e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.739+0000 7f4fab1a8740 1 --2- 172.21.15.43:0/4124750198 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d3e7360a50 0x55d3e735e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.739+0000 7f4fab1a8740 1 --2- 172.21.15.43:0/4124750198 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d3e7357550 0x55d3e735daf0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.739+0000 7f4fab1a8740 1 -- 172.21.15.43:0/4124750198 >> 172.21.15.43:0/4124750198 conn(0x55d3e7351c80 msgr2=0x55d3e73539d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:22.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.739+0000 7f4f8b400640 0 client.26788 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:22.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.739+0000 7f4fab1a8740 1 -- 172.21.15.43:0/4124750198 shutdown_connections 2024-03-28T22:40:22.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.739+0000 7f4fab1a8740 1 -- 172.21.15.43:0/4124750198 wait complete. 2024-03-28T22:40:22.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.739+0000 7f4fab1a8740 20 client.26788 trim_cache size 0 max 16384 2024-03-28T22:40:22.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.756+0000 7f4fab1a8740 1 Processor -- start 2024-03-28T22:40:22.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.756+0000 7f4fab1a8740 1 -- start start 2024-03-28T22:40:22.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.756+0000 7f4fab1a8740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d3e7357550 0x55d3e7351000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.756+0000 7f4fab1a8740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3e7357e70 0x55d3e736c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.756+0000 7f4fab1a8740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d3e7360a50 0x55d3e735e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.756+0000 7f4fab1a8740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa01b0bb0 con 0x55d3e7357550 2024-03-28T22:40:22.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.756+0000 7f4fab1a8740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa01b26f0 con 0x55d3e7357e70 2024-03-28T22:40:22.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.756+0000 7f4fab1a8740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa01b1280 con 0x55d3e7360a50 2024-03-28T22:40:22.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.756+0000 7f4fa5a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d3e7357550 0x55d3e7351000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.756+0000 7f4fa5a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d3e7357550 0x55d3e7351000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38064/0 (socket says 172.21.15.43:38064) 2024-03-28T22:40:22.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.756+0000 7f4fa5a00640 1 -- 172.21.15.43:0/143883844 learned_addr learned my addr 172.21.15.43:0/143883844 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:22.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.756+0000 7f4fa6400640 1 --2- 172.21.15.43:0/143883844 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d3e7360a50 0x55d3e735e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.757+0000 7f4fa5000640 1 --2- 172.21.15.43:0/143883844 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3e7357e70 0x55d3e736c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.757+0000 7f4fa5a00640 1 -- 172.21.15.43:0/143883844 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d3e7360a50 msgr2=0x55d3e735e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.757+0000 7f4fa5a00640 1 --2- 172.21.15.43:0/143883844 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d3e7360a50 0x55d3e735e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.757+0000 7f4fa5a00640 1 -- 172.21.15.43:0/143883844 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3e7357e70 msgr2=0x55d3e736c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.757+0000 7f4fa5a00640 1 --2- 172.21.15.43:0/143883844 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3e7357e70 0x55d3e736c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.757+0000 7f4fa5a00640 1 -- 172.21.15.43:0/143883844 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3e734d540 con 0x55d3e7357550 2024-03-28T22:40:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.757+0000 7f4fa5000640 1 --2- 172.21.15.43:0/143883844 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3e7357e70 0x55d3e736c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:40:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.757+0000 7f4fa5a00640 1 --2- 172.21.15.43:0/143883844 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d3e7357550 0x55d3e7351000 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4f9002c6c0 tx=0x7f4f90009940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=148dcc8c80a95cd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.757+0000 7f4f96000640 1 -- 172.21.15.43:0/143883844 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f90055020 con 0x55d3e7357550 2024-03-28T22:40:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.757+0000 7f4f96000640 1 -- 172.21.15.43:0/143883844 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f90034cd0 con 0x55d3e7357550 2024-03-28T22:40:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.758+0000 7f4f96000640 1 -- 172.21.15.43:0/143883844 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f9004c070 con 0x55d3e7357550 2024-03-28T22:40:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.758+0000 7f4fab1a8740 1 -- 172.21.15.43:0/143883844 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d3e7357550 msgr2=0x55d3e7351000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.758+0000 7f4fab1a8740 1 --2- 172.21.15.43:0/143883844 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d3e7357550 0x55d3e7351000 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4f9002c6c0 tx=0x7f4f90009940 comp rx=0 tx=0).stop 2024-03-28T22:40:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.758+0000 7f4fab1a8740 1 -- 172.21.15.43:0/143883844 shutdown_connections 2024-03-28T22:40:22.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.758+0000 7f4fab1a8740 1 --2- 172.21.15.43:0/143883844 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3e7357e70 0x55d3e736c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.758+0000 7f4fab1a8740 1 --2- 172.21.15.43:0/143883844 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d3e7360a50 0x55d3e735e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.758+0000 7f4fab1a8740 1 --2- 172.21.15.43:0/143883844 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d3e7357550 0x55d3e7351000 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.758+0000 7f4fab1a8740 1 -- 172.21.15.43:0/143883844 >> 172.21.15.43:0/143883844 conn(0x55d3e7351c80 msgr2=0x55d3e735cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:22.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.759+0000 7f4fab1a8740 1 -- 172.21.15.43:0/143883844 shutdown_connections 2024-03-28T22:40:22.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.759+0000 7f4fab1a8740 1 -- 172.21.15.43:0/143883844 wait complete. 2024-03-28T22:40:22.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.759+0000 7f4fab1a8740 1 -- start start 2024-03-28T22:40:22.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.759+0000 7f4fab1a8740 1 Processor -- start 2024-03-28T22:40:22.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.760+0000 7f4fab1a8740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3e7357550 0x55d3e7364430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.760+0000 7f4fab1a8740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d3e7357e70 0x55d3e7364970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.760+0000 7f4fab1a8740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d3e7360a50 0x55d3e7364eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.760+0000 7f4fab1a8740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa01b2020 con 0x55d3e7357e70 2024-03-28T22:40:22.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.760+0000 7f4fab1a8740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa01b1950 con 0x55d3e7357550 2024-03-28T22:40:22.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.760+0000 7f4fab1a8740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa01aeb90 con 0x55d3e7360a50 2024-03-28T22:40:22.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.760+0000 7f4fab1a8740 10 client.0 _ll_register_callbacks cb 0x7f4f98009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:22.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.760+0000 7f4fa5a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3e7357550 0x55d3e7364430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.760+0000 7f4fa5a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3e7357550 0x55d3e7364430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45520/0 (socket says 172.21.15.43:45520) 2024-03-28T22:40:22.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.760+0000 7f4fa5a00640 1 -- 172.21.15.43:0/1081970980 learned_addr learned my addr 172.21.15.43:0/1081970980 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:22.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.760+0000 7f4fa6400640 1 --2- 172.21.15.43:0/1081970980 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d3e7360a50 0x55d3e7364eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.760+0000 7f4fa5000640 1 --2- 172.21.15.43:0/1081970980 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d3e7357e70 0x55d3e7364970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.760+0000 7f4fa5a00640 1 -- 172.21.15.43:0/1081970980 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d3e7360a50 msgr2=0x55d3e7364eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:22.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.760+0000 7f4fa5a00640 1 --2- 172.21.15.43:0/1081970980 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d3e7360a50 0x55d3e7364eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.760+0000 7f4fa5a00640 1 -- 172.21.15.43:0/1081970980 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d3e7357e70 msgr2=0x55d3e7364970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:22.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.760+0000 7f4fa5a00640 1 --2- 172.21.15.43:0/1081970980 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d3e7357e70 0x55d3e7364970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.760+0000 7f4fa5a00640 1 -- 172.21.15.43:0/1081970980 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3e73de600 con 0x55d3e7357550 2024-03-28T22:40:22.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.761+0000 7f4fa5a00640 1 --2- 172.21.15.43:0/1081970980 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3e7357550 0x55d3e7364430 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f4f90009bf0 tx=0x7f4f90034a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cac5fea76e8eb367 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:22.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.761+0000 7f4f8b400640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:40:22.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.761+0000 7f4f8b400640 1 -- 172.21.15.43:0/1081970980 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f90055030 con 0x55d3e7357550 2024-03-28T22:40:22.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.761+0000 7f4fab1a8740 1 -- 172.21.15.43:0/1081970980 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f90034cd0 con 0x55d3e7357550 2024-03-28T22:40:22.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.761+0000 7f4f8b400640 1 -- 172.21.15.43:0/1081970980 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f4f90032a80 con 0x55d3e7357550 2024-03-28T22:40:22.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.761+0000 7f4f8b400640 1 -- 172.21.15.43:0/1081970980 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4f9006d050 con 0x55d3e7357550 2024-03-28T22:40:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.761+0000 7f4fab1a8740 10 client.26840 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.761+0000 7f4fab1a8740 1 -- 172.21.15.43:0/1081970980 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d3e73df860 con 0x55d3e7357550 2024-03-28T22:40:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.762+0000 7f4f8b400640 1 -- 172.21.15.43:0/1081970980 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f90034cd0 con 0x55d3e7357550 2024-03-28T22:40:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.762+0000 7f4f8b400640 1 -- 172.21.15.43:0/1081970980 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4f90063030 con 0x55d3e7357550 2024-03-28T22:40:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.762+0000 7f4fab1a8740 10 client.26840 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.762+0000 7f4fab1a8740 10 client.26840 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:40:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.762+0000 7f4fab1a8740 20 client.26840 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.762+0000 7f4fab1a8740 1 --2- 172.21.15.43:0/1081970980 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d3e73dfe10 0x55d3e73e2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.762+0000 7f4fab1a8740 4 client.26840 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:40:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.762+0000 7f4fab1a8740 1 -- 172.21.15.43:0/1081970980 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d3e7357060 con 0x55d3e73dfe10 2024-03-28T22:40:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.762+0000 7f4fa5000640 1 --2- 172.21.15.43:0/1081970980 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d3e73dfe10 0x55d3e73e2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.762+0000 7f4fa5000640 1 --2- 172.21.15.43:0/1081970980 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d3e73dfe10 0x55d3e73e2230 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e722a8c70c665dbf server_cookie=f83de4136d9db8e8 in_seq=0 out_seq=0 2024-03-28T22:40:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.762+0000 7f4f8b400640 10 client.26840 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.763+0000 7f4f8b400640 1 -- 172.21.15.43:0/1081970980 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d3e7357060 con 0x55d3e73dfe10 2024-03-28T22:40:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.763+0000 7f4f8b400640 10 client.26840 handle_command_reply: tid=0 2024-03-28T22:40:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.763+0000 7f4fab1a8740 1 client.26840 shutdown 2024-03-28T22:40:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.763+0000 7f4fab1a8740 1 -- 172.21.15.43:0/1081970980 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3e7357550 msgr2=0x55d3e7364430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.763+0000 7f4fab1a8740 1 --2- 172.21.15.43:0/1081970980 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3e7357550 0x55d3e7364430 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f4f90009bf0 tx=0x7f4f90034a80 comp rx=0 tx=0).stop 2024-03-28T22:40:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.763+0000 7f4fab1a8740 1 -- 172.21.15.43:0/1081970980 shutdown_connections 2024-03-28T22:40:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.763+0000 7f4fab1a8740 1 --2- 172.21.15.43:0/1081970980 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d3e7360a50 0x55d3e7364eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.763+0000 7f4fab1a8740 1 --2- 172.21.15.43:0/1081970980 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d3e7357e70 0x55d3e7364970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.763+0000 7f4fab1a8740 1 --2- 172.21.15.43:0/1081970980 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d3e73dfe10 0x55d3e73e2230 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.763+0000 7f4fab1a8740 1 --2- 172.21.15.43:0/1081970980 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3e7357550 0x55d3e7364430 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.763+0000 7f4f8b400640 0 client.26840 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.763+0000 7f4fab1a8740 1 -- 172.21.15.43:0/1081970980 >> 172.21.15.43:0/1081970980 conn(0x55d3e7351c80 msgr2=0x55d3e7352e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.763+0000 7f4fab1a8740 1 -- 172.21.15.43:0/1081970980 shutdown_connections 2024-03-28T22:40:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.763+0000 7f4fab1a8740 1 -- 172.21.15.43:0/1081970980 wait complete. 2024-03-28T22:40:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.763+0000 7f4fab1a8740 20 client.26840 trim_cache size 0 max 16384 2024-03-28T22:40:22.766 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:40:22.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.764+0000 7f4fa6e00640 1 -- 172.21.15.43:0/2833173436 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4f840a1ef0 msgr2=0x7f4f840a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:22.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.764+0000 7f4fa6e00640 1 --2- 172.21.15.43:0/2833173436 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4f840a1ef0 0x7f4f840a43b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4f98009c70 tx=0x7f4f98005f90 comp rx=0 tx=0).stop 2024-03-28T22:40:22.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.764+0000 7f4fa6e00640 1 -- 172.21.15.43:0/2833173436 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4fa01030a0 msgr2=0x7f4fa0198380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:22.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.764+0000 7f4fa6e00640 1 --2- 172.21.15.43:0/2833173436 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4fa01030a0 0x7f4fa0198380 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c00e9f0 tx=0x7f4f8c00eeb0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.764+0000 7f4fa6e00640 1 -- 172.21.15.43:0/2833173436 shutdown_connections 2024-03-28T22:40:22.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.764+0000 7f4fa6e00640 1 --2- 172.21.15.43:0/2833173436 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4f840a1ef0 0x7f4f840a43b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.764+0000 7f4fa6e00640 1 --2- 172.21.15.43:0/2833173436 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4fa01030a0 0x7f4fa0198380 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.764+0000 7f4fa6e00640 1 --2- 172.21.15.43:0/2833173436 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fa010bba0 0x7f4fa019c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.764+0000 7f4fa6e00640 1 --2- 172.21.15.43:0/2833173436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4fa0102780 0x7f4fa0197e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:22.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.764+0000 7f4fa6e00640 1 -- 172.21.15.43:0/2833173436 >> 172.21.15.43:0/2833173436 conn(0x7f4fa00fbfa0 msgr2=0x7f4fa00fccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:22.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.764+0000 7f4fa6e00640 1 -- 172.21.15.43:0/2833173436 shutdown_connections 2024-03-28T22:40:22.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:22.764+0000 7f4fa6e00640 1 -- 172.21.15.43:0/2833173436 wait complete. 2024-03-28T22:40:22.767 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:22 smithi043 ceph-mon[31823]: pgmap v100: 129 pgs: 129 active+clean; 49 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 235 op/s 2024-03-28T22:40:23.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:22 smithi172 ceph-mon[36566]: pgmap v100: 129 pgs: 129 active+clean; 49 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 235 op/s 2024-03-28T22:40:23.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:22 smithi082 ceph-mon[36471]: pgmap v100: 129 pgs: 129 active+clean; 49 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 235 op/s 2024-03-28T22:40:23.352 INFO:tasks.workunit.client.0.smithi043.stdout: 1 176604 8.50 MB/sec warmup 114 sec latency 34.172 ms 2024-03-28T22:40:23.780 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:40:23.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:40:23.780 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:40:24.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.017+0000 7fc6ad400640 1 -- 172.21.15.43:0/2946061446 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6a81098f0 msgr2=0x7fc6a8109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:24.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.017+0000 7fc6ad400640 1 --2- 172.21.15.43:0/2946061446 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6a81098f0 0x7fc6a8109cd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc698009930 tx=0x7fc69802bc20 comp rx=0 tx=0).stop 2024-03-28T22:40:24.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.017+0000 7fc6ad400640 1 -- 172.21.15.43:0/2946061446 shutdown_connections 2024-03-28T22:40:24.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.017+0000 7fc6ad400640 1 --2- 172.21.15.43:0/2946061446 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc6a81042a0 0x7fc6a81107b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.017+0000 7fc6ad400640 1 --2- 172.21.15.43:0/2946061446 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6a81038e0 0x7fc6a8103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.017+0000 7fc6ad400640 1 --2- 172.21.15.43:0/2946061446 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6a81098f0 0x7fc6a8109cd0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.017+0000 7fc6ad400640 1 -- 172.21.15.43:0/2946061446 >> 172.21.15.43:0/2946061446 conn(0x7fc6a80ff3d0 msgr2=0x7fc6a81017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:24.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.017+0000 7fc6ad400640 1 -- 172.21.15.43:0/2946061446 shutdown_connections 2024-03-28T22:40:24.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.017+0000 7fc6ad400640 1 -- 172.21.15.43:0/2946061446 wait complete. 2024-03-28T22:40:24.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.018+0000 7fc6ad400640 1 Processor -- start 2024-03-28T22:40:24.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.018+0000 7fc6ad400640 1 -- start start 2024-03-28T22:40:24.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.019+0000 7fc6ad400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6a81038e0 0x7fc6a819a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.019+0000 7fc6ad400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6a81042a0 0x7fc6a819a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.019+0000 7fc6ad400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc6a81098f0 0x7fc6a819eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.019+0000 7fc6ad400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a8113cc0 con 0x7fc6a81038e0 2024-03-28T22:40:24.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.019+0000 7fc6ad400640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a8113b40 con 0x7fc6a81098f0 2024-03-28T22:40:24.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.019+0000 7fc6ad400640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a8113e40 con 0x7fc6a81042a0 2024-03-28T22:40:24.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.019+0000 7fc6a5e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6a81042a0 0x7fc6a819a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.019+0000 7fc6a7200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc6a81098f0 0x7fc6a819eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.019+0000 7fc6a5e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6a81042a0 0x7fc6a819a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:48374/0 (socket says 172.21.15.43:48374) 2024-03-28T22:40:24.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.019+0000 7fc6a6800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6a81038e0 0x7fc6a819a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.019+0000 7fc6a7200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc6a81098f0 0x7fc6a819eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45536/0 (socket says 172.21.15.43:45536) 2024-03-28T22:40:24.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.019+0000 7fc6a5e00640 1 -- 172.21.15.43:0/4233723092 learned_addr learned my addr 172.21.15.43:0/4233723092 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:24.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.020+0000 7fc6a5e00640 1 -- 172.21.15.43:0/4233723092 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc6a81098f0 msgr2=0x7fc6a819eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:24.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.020+0000 7fc6a5e00640 1 --2- 172.21.15.43:0/4233723092 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc6a81098f0 0x7fc6a819eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.020+0000 7fc6a5e00640 1 -- 172.21.15.43:0/4233723092 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6a81038e0 msgr2=0x7fc6a819a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:24.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.020+0000 7fc6a5e00640 1 --2- 172.21.15.43:0/4233723092 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6a81038e0 0x7fc6a819a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.020+0000 7fc6a5e00640 1 -- 172.21.15.43:0/4233723092 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6a819f220 con 0x7fc6a81042a0 2024-03-28T22:40:24.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.020+0000 7fc6a7200640 1 --2- 172.21.15.43:0/4233723092 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc6a81098f0 0x7fc6a819eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:24.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.020+0000 7fc6a5e00640 1 --2- 172.21.15.43:0/4233723092 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6a81042a0 0x7fc6a819a7c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc69400e9f0 tx=0x7fc69400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:24.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.020+0000 7fc69f400640 1 -- 172.21.15.43:0/4233723092 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc69400cd80 con 0x7fc6a81042a0 2024-03-28T22:40:24.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.020+0000 7fc69f400640 1 -- 172.21.15.43:0/4233723092 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc694010910 con 0x7fc6a81042a0 2024-03-28T22:40:24.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.020+0000 7fc6ad400640 1 -- 172.21.15.43:0/4233723092 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6a819f510 con 0x7fc6a81042a0 2024-03-28T22:40:24.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.020+0000 7fc69f400640 1 -- 172.21.15.43:0/4233723092 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc694018990 con 0x7fc6a81042a0 2024-03-28T22:40:24.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.021+0000 7fc6ad400640 1 -- 172.21.15.43:0/4233723092 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6a81a6da0 con 0x7fc6a81042a0 2024-03-28T22:40:24.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.024+0000 7fc69f400640 1 -- 172.21.15.43:0/4233723092 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fc694010ac0 con 0x7fc6a81042a0 2024-03-28T22:40:24.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.025+0000 7fc6b018f740 1 Processor -- start 2024-03-28T22:40:24.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.025+0000 7fc6b018f740 1 -- start start 2024-03-28T22:40:24.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.025+0000 7fc6b018f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f26d552f70 0x55f26d553350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.025+0000 7fc6b018f740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f26d553890 0x55f26d55bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.025+0000 7fc6b018f740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f26d55c470 0x55f26d55e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.025+0000 7fc6b018f740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55f26d4b6c40 con 0x55f26d55c470 2024-03-28T22:40:24.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.025+0000 7fc6b018f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55f26d45fc40 con 0x55f26d552f70 2024-03-28T22:40:24.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.025+0000 7fc6b018f740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55f26d4b2ce0 con 0x55f26d553890 2024-03-28T22:40:24.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.025+0000 7fc6a7200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f26d552f70 0x55f26d553350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.026+0000 7fc6a6800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f26d553890 0x55f26d55bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.026+0000 7fc6a7200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f26d552f70 0x55f26d553350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45552/0 (socket says 172.21.15.43:45552) 2024-03-28T22:40:24.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.026+0000 7fc6a7200640 1 -- 172.21.15.43:0/3905107298 learned_addr learned my addr 172.21.15.43:0/3905107298 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:24.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.026+0000 7fc6a5e00640 1 --2- 172.21.15.43:0/3905107298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f26d55c470 0x55f26d55e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.026+0000 7fc6a7200640 1 -- 172.21.15.43:0/3905107298 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f26d553890 msgr2=0x55f26d55bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:24.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.026+0000 7fc6a7200640 1 --2- 172.21.15.43:0/3905107298 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f26d553890 0x55f26d55bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.026+0000 7fc6a7200640 1 -- 172.21.15.43:0/3905107298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f26d55c470 msgr2=0x55f26d55e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:24.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.026+0000 7fc6a7200640 1 --2- 172.21.15.43:0/3905107298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f26d55c470 0x55f26d55e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.026+0000 7fc6a7200640 1 -- 172.21.15.43:0/3905107298 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f26d2937e0 con 0x55f26d552f70 2024-03-28T22:40:24.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.026+0000 7fc6a7200640 1 --2- 172.21.15.43:0/3905107298 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f26d552f70 0x55f26d553350 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0011790 tx=0x7fc6a0011c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8b3902d66b25403 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:24.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.027+0000 7fc69f400640 1 --2- 172.21.15.43:0/4233723092 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc68c0a1f70 0x7fc68c0a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.027+0000 7fc693400640 1 -- 172.21.15.43:0/3905107298 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a0003c30 con 0x55f26d552f70 2024-03-28T22:40:24.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.027+0000 7fc693400640 1 -- 172.21.15.43:0/3905107298 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6a000fe80 con 0x55f26d552f70 2024-03-28T22:40:24.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.027+0000 7fc693400640 1 -- 172.21.15.43:0/3905107298 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a00193e0 con 0x55f26d552f70 2024-03-28T22:40:24.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.027+0000 7fc69f400640 1 -- 172.21.15.43:0/4233723092 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fc6940ce860 con 0x7fc6a81042a0 2024-03-28T22:40:24.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.027+0000 7fc6a6800640 1 --2- 172.21.15.43:0/4233723092 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc68c0a1f70 0x7fc68c0a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.028+0000 7fc6b018f740 1 -- 172.21.15.43:0/3905107298 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f26d552f70 msgr2=0x55f26d553350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:24.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.028+0000 7fc6b018f740 1 --2- 172.21.15.43:0/3905107298 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f26d552f70 0x55f26d553350 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0011790 tx=0x7fc6a0011c50 comp rx=0 tx=0).stop 2024-03-28T22:40:24.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.028+0000 7fc6a6800640 1 --2- 172.21.15.43:0/4233723092 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc68c0a1f70 0x7fc68c0a4430 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc698009900 tx=0x7fc69802e040 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:24.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.028+0000 7fc6b018f740 1 -- 172.21.15.43:0/3905107298 shutdown_connections 2024-03-28T22:40:24.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.028+0000 7fc6b018f740 1 --2- 172.21.15.43:0/3905107298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f26d55c470 0x55f26d55e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.028+0000 7fc6b018f740 1 --2- 172.21.15.43:0/3905107298 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f26d553890 0x55f26d55bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.028+0000 7fc6b018f740 1 --2- 172.21.15.43:0/3905107298 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f26d552f70 0x55f26d553350 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.028+0000 7fc6b018f740 1 -- 172.21.15.43:0/3905107298 >> 172.21.15.43:0/3905107298 conn(0x55f26d54d6a0 msgr2=0x55f26d54da80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:24.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.028+0000 7fc6b018f740 1 -- 172.21.15.43:0/3905107298 shutdown_connections 2024-03-28T22:40:24.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.028+0000 7fc6b018f740 1 -- 172.21.15.43:0/3905107298 wait complete. 2024-03-28T22:40:24.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.029+0000 7fc6b018f740 1 -- start start 2024-03-28T22:40:24.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.029+0000 7fc6b018f740 1 Processor -- start 2024-03-28T22:40:24.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.029+0000 7fc6b018f740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f26d552f70 0x55f26d559470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.029+0000 7fc6b018f740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f26d553890 0x55f26d5599b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.029+0000 7fc6b018f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f26d55c470 0x55f26d559ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.029+0000 7fc6b018f740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55f26d55f260 con 0x55f26d553890 2024-03-28T22:40:24.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.030+0000 7fc6b018f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55f26d455250 con 0x55f26d55c470 2024-03-28T22:40:24.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.030+0000 7fc6b018f740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55f26d55f3e0 con 0x55f26d552f70 2024-03-28T22:40:24.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.030+0000 7fc6b018f740 10 client.0 _ll_register_callbacks cb 0x55f26d476e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:24.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.030+0000 7fc6a6800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f26d552f70 0x55f26d559470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.030+0000 7fc6a7200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f26d55c470 0x55f26d559ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.030+0000 7fc6a6800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f26d552f70 0x55f26d559470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:48394/0 (socket says 172.21.15.43:48394) 2024-03-28T22:40:24.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.030+0000 7fc6a7200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f26d55c470 0x55f26d559ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45566/0 (socket says 172.21.15.43:45566) 2024-03-28T22:40:24.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.030+0000 7fc6a6800640 1 -- 172.21.15.43:0/1669423307 learned_addr learned my addr 172.21.15.43:0/1669423307 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:24.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.030+0000 7fc6a5e00640 1 --2- 172.21.15.43:0/1669423307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f26d553890 0x55f26d5599b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.030+0000 7fc6a6800640 1 -- 172.21.15.43:0/1669423307 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f26d55c470 msgr2=0x55f26d559ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:24.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.030+0000 7fc6a6800640 1 --2- 172.21.15.43:0/1669423307 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f26d55c470 0x55f26d559ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.030+0000 7fc6a6800640 1 -- 172.21.15.43:0/1669423307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f26d553890 msgr2=0x55f26d5599b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:24.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.030+0000 7fc6a6800640 1 --2- 172.21.15.43:0/1669423307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f26d553890 0x55f26d5599b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.030+0000 7fc6a6800640 1 -- 172.21.15.43:0/1669423307 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f26d2f3f70 con 0x55f26d552f70 2024-03-28T22:40:24.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.031+0000 7fc6a5e00640 1 --2- 172.21.15.43:0/1669423307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f26d553890 0x55f26d5599b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:24.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.031+0000 7fc6a6800640 1 --2- 172.21.15.43:0/1669423307 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f26d552f70 0x55f26d559470 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc698035000 tx=0x7fc69802bc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b240434a69055711 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:24.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.031+0000 7fc687e00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:40:24.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.031+0000 7fc687e00640 1 -- 172.21.15.43:0/1669423307 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc698050070 con 0x55f26d552f70 2024-03-28T22:40:24.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.031+0000 7fc6b018f740 1 -- 172.21.15.43:0/1669423307 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f26d4c4180 con 0x55f26d552f70 2024-03-28T22:40:24.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.031+0000 7fc687e00640 1 -- 172.21.15.43:0/1669423307 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fc698004ad0 con 0x55f26d552f70 2024-03-28T22:40:24.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.032+0000 7fc687e00640 1 -- 172.21.15.43:0/1669423307 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc69805a050 con 0x55f26d552f70 2024-03-28T22:40:24.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.032+0000 7fc6b018f740 10 client.26806 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:24.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.032+0000 7fc6b018f740 1 -- 172.21.15.43:0/1669423307 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f26d18b170 con 0x55f26d552f70 2024-03-28T22:40:24.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.032+0000 7fc687e00640 1 -- 172.21.15.43:0/1669423307 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f26d4c4180 con 0x55f26d552f70 2024-03-28T22:40:24.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.032+0000 7fc687e00640 1 -- 172.21.15.43:0/1669423307 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc6980037b0 con 0x55f26d552f70 2024-03-28T22:40:24.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.032+0000 7fc6b018f740 10 client.26806 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:24.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.032+0000 7fc6b018f740 10 client.26806 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:40:24.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.032+0000 7fc6b018f740 20 client.26806 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:24.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.033+0000 7fc6b018f740 1 --2- 172.21.15.43:0/1669423307 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f26d56c880 0x55f26d56ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.033+0000 7fc6b018f740 4 client.26806 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:40:24.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.033+0000 7fc6b018f740 1 -- 172.21.15.43:0/1669423307 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f26d4be4d0 con 0x55f26d56c880 2024-03-28T22:40:24.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.033+0000 7fc6a5e00640 1 --2- 172.21.15.43:0/1669423307 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f26d56c880 0x55f26d56ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.033+0000 7fc6a5e00640 1 --2- 172.21.15.43:0/1669423307 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f26d56c880 0x55f26d56ecd0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c510940272ce5a server_cookie=6b28c7f5d81e6546 in_seq=0 out_seq=0 2024-03-28T22:40:24.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.033+0000 7fc687e00640 10 client.26806 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:24.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.036+0000 7fc687e00640 1 -- 172.21.15.43:0/1669423307 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f26d4be4d0 con 0x55f26d56c880 2024-03-28T22:40:24.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.036+0000 7fc687e00640 10 client.26806 handle_command_reply: tid=0 2024-03-28T22:40:24.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.036+0000 7fc6b018f740 1 client.26806 shutdown 2024-03-28T22:40:24.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.036+0000 7fc6b018f740 1 -- 172.21.15.43:0/1669423307 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f26d552f70 msgr2=0x55f26d559470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:24.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.036+0000 7fc6b018f740 1 --2- 172.21.15.43:0/1669423307 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f26d552f70 0x55f26d559470 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc698035000 tx=0x7fc69802bc20 comp rx=0 tx=0).stop 2024-03-28T22:40:24.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.036+0000 7fc6b018f740 1 -- 172.21.15.43:0/1669423307 shutdown_connections 2024-03-28T22:40:24.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.036+0000 7fc6b018f740 1 --2- 172.21.15.43:0/1669423307 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f26d56c880 0x55f26d56ecd0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.036+0000 7fc6b018f740 1 --2- 172.21.15.43:0/1669423307 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f26d55c470 0x55f26d559ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.036+0000 7fc6b018f740 1 --2- 172.21.15.43:0/1669423307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f26d553890 0x55f26d5599b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.036+0000 7fc6b018f740 1 --2- 172.21.15.43:0/1669423307 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f26d552f70 0x55f26d559470 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.036+0000 7fc6b018f740 1 -- 172.21.15.43:0/1669423307 >> 172.21.15.43:0/1669423307 conn(0x55f26d54d6a0 msgr2=0x55f26d54f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:24.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.036+0000 7fc687e00640 0 client.26806 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:24.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.036+0000 7fc6b018f740 1 -- 172.21.15.43:0/1669423307 shutdown_connections 2024-03-28T22:40:24.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.036+0000 7fc6b018f740 1 -- 172.21.15.43:0/1669423307 wait complete. 2024-03-28T22:40:24.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.036+0000 7fc6b018f740 20 client.26806 trim_cache size 0 max 16384 2024-03-28T22:40:24.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.057+0000 7fc6b018f740 1 Processor -- start 2024-03-28T22:40:24.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.057+0000 7fc6b018f740 1 -- start start 2024-03-28T22:40:24.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.057+0000 7fc6b018f740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f26d552f70 0x55f26d55bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.058+0000 7fc6b018f740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f26d553890 0x55f26d568030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.058+0000 7fc6b018f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f26d55c470 0x55f26d5745a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.058+0000 7fc6b018f740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a81b0d50 con 0x55f26d553890 2024-03-28T22:40:24.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.058+0000 7fc6b018f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a81b2890 con 0x55f26d55c470 2024-03-28T22:40:24.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.058+0000 7fc6b018f740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a81b1420 con 0x55f26d552f70 2024-03-28T22:40:24.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.058+0000 7fc6a7200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f26d55c470 0x55f26d5745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.058+0000 7fc6a7200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f26d55c470 0x55f26d5745a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45580/0 (socket says 172.21.15.43:45580) 2024-03-28T22:40:24.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.058+0000 7fc6a7200640 1 -- 172.21.15.43:0/2597552526 learned_addr learned my addr 172.21.15.43:0/2597552526 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:24.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.058+0000 7fc6a6800640 1 --2- 172.21.15.43:0/2597552526 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f26d552f70 0x55f26d55bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.058+0000 7fc6a5e00640 1 --2- 172.21.15.43:0/2597552526 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f26d553890 0x55f26d568030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.058+0000 7fc6a7200640 1 -- 172.21.15.43:0/2597552526 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f26d552f70 msgr2=0x55f26d55bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:24.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.058+0000 7fc6a7200640 1 --2- 172.21.15.43:0/2597552526 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f26d552f70 0x55f26d55bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.058+0000 7fc6a7200640 1 -- 172.21.15.43:0/2597552526 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f26d553890 msgr2=0x55f26d568030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:24.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.058+0000 7fc6a7200640 1 --2- 172.21.15.43:0/2597552526 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f26d553890 0x55f26d568030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.058+0000 7fc6a7200640 1 -- 172.21.15.43:0/2597552526 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f26d56f350 con 0x55f26d55c470 2024-03-28T22:40:24.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.059+0000 7fc6a6800640 1 --2- 172.21.15.43:0/2597552526 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f26d552f70 0x55f26d55bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:24.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.059+0000 7fc6a7200640 1 --2- 172.21.15.43:0/2597552526 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f26d55c470 0x55f26d5745a0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0004590 tx=0x7fc6a00045c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd1bfb293ba049ca server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:24.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.059+0000 7fc692a00640 1 -- 172.21.15.43:0/2597552526 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a002a070 con 0x55f26d55c470 2024-03-28T22:40:24.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.059+0000 7fc692a00640 1 -- 172.21.15.43:0/2597552526 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6a0004050 con 0x55f26d55c470 2024-03-28T22:40:24.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.059+0000 7fc692a00640 1 -- 172.21.15.43:0/2597552526 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a001f040 con 0x55f26d55c470 2024-03-28T22:40:24.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.060+0000 7fc6b018f740 1 -- 172.21.15.43:0/2597552526 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f26d55c470 msgr2=0x55f26d5745a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:24.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.060+0000 7fc6b018f740 1 --2- 172.21.15.43:0/2597552526 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f26d55c470 0x55f26d5745a0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0004590 tx=0x7fc6a00045c0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.060+0000 7fc6b018f740 1 -- 172.21.15.43:0/2597552526 shutdown_connections 2024-03-28T22:40:24.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.060+0000 7fc6b018f740 1 --2- 172.21.15.43:0/2597552526 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f26d55c470 0x55f26d5745a0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.060+0000 7fc6b018f740 1 --2- 172.21.15.43:0/2597552526 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f26d553890 0x55f26d568030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.060+0000 7fc6b018f740 1 --2- 172.21.15.43:0/2597552526 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f26d552f70 0x55f26d55bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.060+0000 7fc6b018f740 1 -- 172.21.15.43:0/2597552526 >> 172.21.15.43:0/2597552526 conn(0x55f26d54d6a0 msgr2=0x55f26d54f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:24.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.060+0000 7fc6b018f740 1 -- 172.21.15.43:0/2597552526 shutdown_connections 2024-03-28T22:40:24.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.060+0000 7fc6b018f740 1 -- 172.21.15.43:0/2597552526 wait complete. 2024-03-28T22:40:24.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.061+0000 7fc6b018f740 1 -- start start 2024-03-28T22:40:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.061+0000 7fc6b018f740 1 Processor -- start 2024-03-28T22:40:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.061+0000 7fc6b018f740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f26d552f70 0x55f26d564850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.061+0000 7fc6b018f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f26d553890 0x55f26d561630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.061+0000 7fc6b018f740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f26d55c470 0x55f26d561b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.061+0000 7fc6b018f740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a81b21c0 con 0x55f26d552f70 2024-03-28T22:40:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.061+0000 7fc6b018f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a81b1af0 con 0x55f26d553890 2024-03-28T22:40:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.061+0000 7fc6b018f740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a81aed30 con 0x55f26d55c470 2024-03-28T22:40:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.061+0000 7fc6b018f740 10 client.0 _ll_register_callbacks cb 0x7fc698009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.062+0000 7fc6a5e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f26d553890 0x55f26d561630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.062+0000 7fc6a6800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f26d552f70 0x55f26d564850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.062+0000 7fc6a7200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f26d55c470 0x55f26d561b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.062+0000 7fc6a5e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f26d553890 0x55f26d561630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45588/0 (socket says 172.21.15.43:45588) 2024-03-28T22:40:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.062+0000 7fc6a5e00640 1 -- 172.21.15.43:0/1828111037 learned_addr learned my addr 172.21.15.43:0/1828111037 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:24.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.062+0000 7fc6a7200640 1 -- 172.21.15.43:0/1828111037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f26d553890 msgr2=0x55f26d561630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:24.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.062+0000 7fc6a7200640 1 --2- 172.21.15.43:0/1828111037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f26d553890 0x55f26d561630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.062+0000 7fc6a7200640 1 -- 172.21.15.43:0/1828111037 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f26d552f70 msgr2=0x55f26d564850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:24.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.062+0000 7fc6a7200640 1 --2- 172.21.15.43:0/1828111037 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f26d552f70 0x55f26d564850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.062+0000 7fc6a7200640 1 -- 172.21.15.43:0/1828111037 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f26d562260 con 0x55f26d55c470 2024-03-28T22:40:24.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.062+0000 7fc6a5e00640 1 --2- 172.21.15.43:0/1828111037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f26d553890 0x55f26d561630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:24.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.062+0000 7fc6a7200640 1 --2- 172.21.15.43:0/1828111037 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f26d55c470 0x55f26d561b70 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0028040 tx=0x7fc6a0005c40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=834accb53a621d89 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:24.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.062+0000 7fc687e00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:40:24.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.063+0000 7fc687e00640 1 -- 172.21.15.43:0/1828111037 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a0032020 con 0x55f26d55c470 2024-03-28T22:40:24.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.063+0000 7fc687e00640 1 -- 172.21.15.43:0/1828111037 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fc6a0005880 con 0x55f26d55c470 2024-03-28T22:40:24.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.063+0000 7fc6b018f740 1 -- 172.21.15.43:0/1828111037 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6a0004050 con 0x55f26d55c470 2024-03-28T22:40:24.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.063+0000 7fc687e00640 1 -- 172.21.15.43:0/1828111037 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc6a003d050 con 0x55f26d55c470 2024-03-28T22:40:24.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.063+0000 7fc6b018f740 10 client.26812 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:24.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.063+0000 7fc6b018f740 1 -- 172.21.15.43:0/1828111037 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f26d5d3b00 con 0x55f26d55c470 2024-03-28T22:40:24.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.063+0000 7fc687e00640 1 -- 172.21.15.43:0/1828111037 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6a0004050 con 0x55f26d55c470 2024-03-28T22:40:24.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.064+0000 7fc687e00640 1 -- 172.21.15.43:0/1828111037 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc6a002d070 con 0x55f26d55c470 2024-03-28T22:40:24.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.064+0000 7fc6b018f740 10 client.26812 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:24.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.064+0000 7fc6b018f740 10 client.26812 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:40:24.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.064+0000 7fc6b018f740 20 client.26812 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:24.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.064+0000 7fc6b018f740 1 --2- 172.21.15.43:0/1828111037 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f26d5d3fc0 0x55f26d5d63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.064+0000 7fc6b018f740 4 client.26812 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:40:24.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.064+0000 7fc6b018f740 1 -- 172.21.15.43:0/1828111037 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f26d43caa0 con 0x55f26d5d3fc0 2024-03-28T22:40:24.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.064+0000 7fc6a6800640 1 --2- 172.21.15.43:0/1828111037 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f26d5d3fc0 0x55f26d5d63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.065+0000 7fc6a6800640 1 --2- 172.21.15.43:0/1828111037 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f26d5d3fc0 0x55f26d5d63b0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eec65c0e8675cec0 server_cookie=af88529c7fb0b7fb in_seq=0 out_seq=0 2024-03-28T22:40:24.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.065+0000 7fc687e00640 10 client.26812 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:24.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.067+0000 7fc687e00640 1 -- 172.21.15.43:0/1828111037 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f26d43caa0 con 0x55f26d5d3fc0 2024-03-28T22:40:24.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.067+0000 7fc687e00640 10 client.26812 handle_command_reply: tid=0 2024-03-28T22:40:24.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.068+0000 7fc6b018f740 1 client.26812 shutdown 2024-03-28T22:40:24.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.068+0000 7fc6b018f740 1 -- 172.21.15.43:0/1828111037 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f26d55c470 msgr2=0x55f26d561b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:24.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.068+0000 7fc6b018f740 1 --2- 172.21.15.43:0/1828111037 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f26d55c470 0x55f26d561b70 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0028040 tx=0x7fc6a0005c40 comp rx=0 tx=0).stop 2024-03-28T22:40:24.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.068+0000 7fc6b018f740 1 -- 172.21.15.43:0/1828111037 shutdown_connections 2024-03-28T22:40:24.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.068+0000 7fc6b018f740 1 --2- 172.21.15.43:0/1828111037 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f26d5d3fc0 0x55f26d5d63b0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.068+0000 7fc6b018f740 1 --2- 172.21.15.43:0/1828111037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f26d553890 0x55f26d561630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.068+0000 7fc6b018f740 1 --2- 172.21.15.43:0/1828111037 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f26d55c470 0x55f26d561b70 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.068+0000 7fc6b018f740 1 --2- 172.21.15.43:0/1828111037 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f26d552f70 0x55f26d564850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.068+0000 7fc687e00640 0 client.26812 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:24.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.068+0000 7fc6b018f740 1 -- 172.21.15.43:0/1828111037 >> 172.21.15.43:0/1828111037 conn(0x55f26d54d6a0 msgr2=0x55f26d559410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:24.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.068+0000 7fc6b018f740 1 -- 172.21.15.43:0/1828111037 shutdown_connections 2024-03-28T22:40:24.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.068+0000 7fc6b018f740 1 -- 172.21.15.43:0/1828111037 wait complete. 2024-03-28T22:40:24.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.068+0000 7fc6b018f740 20 client.26812 trim_cache size 0 max 16384 2024-03-28T22:40:24.068 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:40:24.068 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-03-28T22:40:24.068 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "cd29bd9b-d867-4e57-8fb9-09cc194a9aa0", 2024-03-28T22:40:24.068 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-03-28T22:40:24.068 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:40:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.069+0000 7fc6ad400640 1 -- 172.21.15.43:0/4233723092 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc68c0a1f70 msgr2=0x7fc68c0a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.069+0000 7fc6ad400640 1 --2- 172.21.15.43:0/4233723092 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc68c0a1f70 0x7fc68c0a4430 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc698009900 tx=0x7fc69802e040 comp rx=0 tx=0).stop 2024-03-28T22:40:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.069+0000 7fc6ad400640 1 -- 172.21.15.43:0/4233723092 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6a81042a0 msgr2=0x7fc6a819a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.069+0000 7fc6ad400640 1 --2- 172.21.15.43:0/4233723092 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6a81042a0 0x7fc6a819a7c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc69400e9f0 tx=0x7fc69400eeb0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.069+0000 7fc6ad400640 1 -- 172.21.15.43:0/4233723092 shutdown_connections 2024-03-28T22:40:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.069+0000 7fc6ad400640 1 --2- 172.21.15.43:0/4233723092 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc68c0a1f70 0x7fc68c0a4430 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.069+0000 7fc6ad400640 1 --2- 172.21.15.43:0/4233723092 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc6a81098f0 0x7fc6a819eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.069+0000 7fc6ad400640 1 --2- 172.21.15.43:0/4233723092 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc6a81042a0 0x7fc6a819a7c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.069+0000 7fc6ad400640 1 --2- 172.21.15.43:0/4233723092 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6a81038e0 0x7fc6a819a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.069+0000 7fc6ad400640 1 -- 172.21.15.43:0/4233723092 >> 172.21.15.43:0/4233723092 conn(0x7fc6a80ff3d0 msgr2=0x7fc6a80ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:24.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.069+0000 7fc6ad400640 1 -- 172.21.15.43:0/4233723092 shutdown_connections 2024-03-28T22:40:24.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.069+0000 7fc6ad400640 1 -- 172.21.15.43:0/4233723092 wait complete. 2024-03-28T22:40:24.085 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:40:24.085 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:40:24.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.304+0000 7fb343400640 1 -- 172.21.15.43:0/3896452329 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb33c0765a0 msgr2=0x7fb33c112050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:24.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.304+0000 7fb343400640 1 --2- 172.21.15.43:0/3896452329 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb33c0765a0 0x7fb33c112050 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb328009930 tx=0x7fb32802bc30 comp rx=0 tx=0).stop 2024-03-28T22:40:24.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.304+0000 7fb343400640 1 -- 172.21.15.43:0/3896452329 shutdown_connections 2024-03-28T22:40:24.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.304+0000 7fb343400640 1 --2- 172.21.15.43:0/3896452329 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb33c112590 0x7fb33c114980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.304+0000 7fb343400640 1 --2- 172.21.15.43:0/3896452329 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb33c0765a0 0x7fb33c112050 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.304+0000 7fb343400640 1 --2- 172.21.15.43:0/3896452329 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb33c075c80 0x7fb33c076060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.304+0000 7fb343400640 1 -- 172.21.15.43:0/3896452329 >> 172.21.15.43:0/3896452329 conn(0x7fb33c0ff400 msgr2=0x7fb33c101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:24.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.304+0000 7fb343400640 1 -- 172.21.15.43:0/3896452329 shutdown_connections 2024-03-28T22:40:24.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.304+0000 7fb343400640 1 -- 172.21.15.43:0/3896452329 wait complete. 2024-03-28T22:40:24.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.305+0000 7fb343400640 1 Processor -- start 2024-03-28T22:40:24.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.305+0000 7fb343400640 1 -- start start 2024-03-28T22:40:24.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.306+0000 7fb343400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb33c075c80 0x7fb33c19e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.306+0000 7fb343400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb33c0765a0 0x7fb33c19eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.306+0000 7fb343400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb33c112590 0x7fb33c1a2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.306+0000 7fb343400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb33c1180e0 con 0x7fb33c0765a0 2024-03-28T22:40:24.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.306+0000 7fb343400640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb33c117f60 con 0x7fb33c112590 2024-03-28T22:40:24.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.306+0000 7fb343400640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fb33c118260 con 0x7fb33c075c80 2024-03-28T22:40:24.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.306+0000 7fb340a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb33c075c80 0x7fb33c19e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.306+0000 7fb33be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb33c0765a0 0x7fb33c19eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.306+0000 7fb341400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb33c112590 0x7fb33c1a2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.306+0000 7fb340a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb33c075c80 0x7fb33c19e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:48436/0 (socket says 172.21.15.43:48436) 2024-03-28T22:40:24.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.306+0000 7fb340a00640 1 -- 172.21.15.43:0/2442876001 learned_addr learned my addr 172.21.15.43:0/2442876001 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:24.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.307+0000 7fb340a00640 1 -- 172.21.15.43:0/2442876001 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb33c112590 msgr2=0x7fb33c1a2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:24.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.307+0000 7fb340a00640 1 --2- 172.21.15.43:0/2442876001 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb33c112590 0x7fb33c1a2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.307+0000 7fb340a00640 1 -- 172.21.15.43:0/2442876001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb33c0765a0 msgr2=0x7fb33c19eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:24.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.307+0000 7fb340a00640 1 --2- 172.21.15.43:0/2442876001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb33c0765a0 0x7fb33c19eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.307+0000 7fb340a00640 1 -- 172.21.15.43:0/2442876001 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb33c1a3600 con 0x7fb33c075c80 2024-03-28T22:40:24.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.307+0000 7fb341400640 1 --2- 172.21.15.43:0/2442876001 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb33c112590 0x7fb33c1a2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:24.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.307+0000 7fb340a00640 1 --2- 172.21.15.43:0/2442876001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb33c075c80 0x7fb33c19e660 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb33000e9f0 tx=0x7fb33000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:24.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.307+0000 7fb339600640 1 -- 172.21.15.43:0/2442876001 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb33000cd80 con 0x7fb33c075c80 2024-03-28T22:40:24.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.307+0000 7fb339600640 1 -- 172.21.15.43:0/2442876001 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb330010910 con 0x7fb33c075c80 2024-03-28T22:40:24.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.307+0000 7fb339600640 1 -- 172.21.15.43:0/2442876001 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb330018990 con 0x7fb33c075c80 2024-03-28T22:40:24.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.307+0000 7fb343400640 1 -- 172.21.15.43:0/2442876001 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb33c1a38f0 con 0x7fb33c075c80 2024-03-28T22:40:24.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.307+0000 7fb343400640 1 -- 172.21.15.43:0/2442876001 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb33c1ab180 con 0x7fb33c075c80 2024-03-28T22:40:24.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.311+0000 7fb3461bc740 1 Processor -- start 2024-03-28T22:40:24.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.311+0000 7fb339600640 1 -- 172.21.15.43:0/2442876001 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fb330018b30 con 0x7fb33c075c80 2024-03-28T22:40:24.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.311+0000 7fb3461bc740 1 -- start start 2024-03-28T22:40:24.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.311+0000 7fb3461bc740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5564601cdbc0 0x5564601cc910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.311+0000 7fb3461bc740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5564601ccee0 0x5564601cd360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.311+0000 7fb3461bc740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564601d6250 0x5564601d8640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.311+0000 7fb3461bc740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x556460131100 con 0x5564601ccee0 2024-03-28T22:40:24.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.311+0000 7fb3461bc740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5564600d9ba0 con 0x5564601d6250 2024-03-28T22:40:24.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.311+0000 7fb3461bc740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55646012d1a0 con 0x5564601cdbc0 2024-03-28T22:40:24.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.312+0000 7fb341400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5564601cdbc0 0x5564601cc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.312+0000 7fb341400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5564601cdbc0 0x5564601cc910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:48440/0 (socket says 172.21.15.43:48440) 2024-03-28T22:40:24.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.312+0000 7fb341400640 1 -- 172.21.15.43:0/2572157263 learned_addr learned my addr 172.21.15.43:0/2572157263 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:24.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.312+0000 7fb340a00640 1 --2- 172.21.15.43:0/2572157263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5564601ccee0 0x5564601cd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.312+0000 7fb33be00640 1 --2- 172.21.15.43:0/2572157263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564601d6250 0x5564601d8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.312+0000 7fb341400640 1 -- 172.21.15.43:0/2572157263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564601d6250 msgr2=0x5564601d8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:24.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.312+0000 7fb341400640 1 --2- 172.21.15.43:0/2572157263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564601d6250 0x5564601d8640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.312+0000 7fb341400640 1 -- 172.21.15.43:0/2572157263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5564601ccee0 msgr2=0x5564601cd360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:24.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.312+0000 7fb341400640 1 --2- 172.21.15.43:0/2572157263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5564601ccee0 0x5564601cd360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.312+0000 7fb341400640 1 -- 172.21.15.43:0/2572157263 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55646010a1a0 con 0x5564601cdbc0 2024-03-28T22:40:24.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.312+0000 7fb33be00640 1 --2- 172.21.15.43:0/2572157263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564601d6250 0x5564601d8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:40:24.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.312+0000 7fb340a00640 1 --2- 172.21.15.43:0/2572157263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5564601ccee0 0x5564601cd360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:40:24.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.313+0000 7fb341400640 1 --2- 172.21.15.43:0/2572157263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5564601cdbc0 0x5564601cc910 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb334011760 tx=0x7fb334011c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f03185a4c5d2430 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:24.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.313+0000 7fb339600640 1 --2- 172.21.15.43:0/2442876001 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb3240a1ef0 0x7fb3240a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.313+0000 7fb32dc00640 1 -- 172.21.15.43:0/2572157263 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb334004d20 con 0x5564601cdbc0 2024-03-28T22:40:24.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.313+0000 7fb32dc00640 1 -- 172.21.15.43:0/2572157263 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb33400f9a0 con 0x5564601cdbc0 2024-03-28T22:40:24.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.313+0000 7fb339600640 1 -- 172.21.15.43:0/2442876001 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fb3300ce8f0 con 0x7fb33c075c80 2024-03-28T22:40:24.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.313+0000 7fb32dc00640 1 -- 172.21.15.43:0/2572157263 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb334004590 con 0x5564601cdbc0 2024-03-28T22:40:24.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.313+0000 7fb340a00640 1 --2- 172.21.15.43:0/2442876001 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb3240a1ef0 0x7fb3240a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.314+0000 7fb3461bc740 1 -- 172.21.15.43:0/2572157263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5564601cdbc0 msgr2=0x5564601cc910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:24.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.314+0000 7fb3461bc740 1 --2- 172.21.15.43:0/2572157263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5564601cdbc0 0x5564601cc910 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb334011760 tx=0x7fb334011c20 comp rx=0 tx=0).stop 2024-03-28T22:40:24.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.314+0000 7fb340a00640 1 --2- 172.21.15.43:0/2442876001 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb3240a1ef0 0x7fb3240a43b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x5564601ce110 tx=0x7fb330000bc0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:24.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.314+0000 7fb3461bc740 1 -- 172.21.15.43:0/2572157263 shutdown_connections 2024-03-28T22:40:24.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.314+0000 7fb3461bc740 1 --2- 172.21.15.43:0/2572157263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564601d6250 0x5564601d8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.314+0000 7fb3461bc740 1 --2- 172.21.15.43:0/2572157263 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5564601ccee0 0x5564601cd360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.314+0000 7fb3461bc740 1 --2- 172.21.15.43:0/2572157263 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5564601cdbc0 0x5564601cc910 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.314+0000 7fb3461bc740 1 -- 172.21.15.43:0/2572157263 >> 172.21.15.43:0/2572157263 conn(0x556460193a10 msgr2=0x5564601c7930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:24.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.314+0000 7fb3461bc740 1 -- 172.21.15.43:0/2572157263 shutdown_connections 2024-03-28T22:40:24.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.314+0000 7fb3461bc740 1 -- 172.21.15.43:0/2572157263 wait complete. 2024-03-28T22:40:24.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.315+0000 7fb3461bc740 1 -- start start 2024-03-28T22:40:24.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.315+0000 7fb3461bc740 1 Processor -- start 2024-03-28T22:40:24.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.315+0000 7fb3461bc740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5564601ccee0 0x5564601d32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.315+0000 7fb3461bc740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5564601cdbc0 0x5564601d3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.315+0000 7fb3461bc740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564601d6250 0x5564601d3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.315+0000 7fb3461bc740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5564601d90c0 con 0x5564601ccee0 2024-03-28T22:40:24.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.315+0000 7fb3461bc740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5564600cf250 con 0x5564601d6250 2024-03-28T22:40:24.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.315+0000 7fb3461bc740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5564601d9240 con 0x5564601cdbc0 2024-03-28T22:40:24.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.315+0000 7fb3461bc740 10 client.0 _ll_register_callbacks cb 0x55645fe44be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:24.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.316+0000 7fb341400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5564601ccee0 0x5564601d32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.316+0000 7fb33be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5564601cdbc0 0x5564601d3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.316+0000 7fb33be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5564601cdbc0 0x5564601d3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:48454/0 (socket says 172.21.15.43:48454) 2024-03-28T22:40:24.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.316+0000 7fb33be00640 1 -- 172.21.15.43:0/2482861686 learned_addr learned my addr 172.21.15.43:0/2482861686 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:24.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.316+0000 7fb341400640 1 --2- 172.21.15.43:0/2482861686 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564601d6250 0x5564601d3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.316+0000 7fb33be00640 1 -- 172.21.15.43:0/2482861686 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564601d6250 msgr2=0x5564601d3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:24.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.316+0000 7fb33be00640 1 --2- 172.21.15.43:0/2482861686 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564601d6250 0x5564601d3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.316+0000 7fb33be00640 1 -- 172.21.15.43:0/2482861686 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5564601ccee0 msgr2=0x5564601d32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:24.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.316+0000 7fb33be00640 1 --2- 172.21.15.43:0/2482861686 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5564601ccee0 0x5564601d32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.316+0000 7fb33be00640 1 -- 172.21.15.43:0/2482861686 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564600eb960 con 0x5564601cdbc0 2024-03-28T22:40:24.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.316+0000 7fb341400640 1 --2- 172.21.15.43:0/2482861686 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5564601ccee0 0x5564601d32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:40:24.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.316+0000 7fb33be00640 1 --2- 172.21.15.43:0/2482861686 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5564601cdbc0 0x5564601d3800 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb32802cfd0 tx=0x7fb32802ce10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46e61d4dbf847039 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:24.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.316+0000 7fb322000640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:40:24.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.317+0000 7fb322000640 1 -- 172.21.15.43:0/2482861686 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb328005d50 con 0x5564601cdbc0 2024-03-28T22:40:24.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.317+0000 7fb322000640 1 -- 172.21.15.43:0/2482861686 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fb328004240 con 0x5564601cdbc0 2024-03-28T22:40:24.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.317+0000 7fb3461bc740 1 -- 172.21.15.43:0/2482861686 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb33400f9a0 con 0x5564601cdbc0 2024-03-28T22:40:24.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.317+0000 7fb322000640 1 -- 172.21.15.43:0/2482861686 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb328004440 con 0x5564601cdbc0 2024-03-28T22:40:24.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.317+0000 7fb3461bc740 10 client.26824 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:24.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.317+0000 7fb3461bc740 1 -- 172.21.15.43:0/2482861686 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55645ff0f7d0 con 0x5564601cdbc0 2024-03-28T22:40:24.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.318+0000 7fb322000640 1 -- 172.21.15.43:0/2482861686 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb33400f9a0 con 0x5564601cdbc0 2024-03-28T22:40:24.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.318+0000 7fb322000640 1 -- 172.21.15.43:0/2482861686 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb3280035f0 con 0x5564601cdbc0 2024-03-28T22:40:24.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.318+0000 7fb3461bc740 10 client.26824 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:24.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.318+0000 7fb3461bc740 10 client.26824 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:40:24.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.318+0000 7fb3461bc740 20 client.26824 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:24.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.318+0000 7fb3461bc740 1 --2- 172.21.15.43:0/2482861686 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5564601e6770 0x5564601e8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.318+0000 7fb3461bc740 4 client.26824 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:40:24.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.318+0000 7fb3461bc740 1 -- 172.21.15.43:0/2482861686 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55646013e640 con 0x5564601e6770 2024-03-28T22:40:24.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.318+0000 7fb341400640 1 --2- 172.21.15.43:0/2482861686 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5564601e6770 0x5564601e8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.319+0000 7fb341400640 1 --2- 172.21.15.43:0/2482861686 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5564601e6770 0x5564601e8b80 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2871b70a6189f552 server_cookie=676f7ff174b542a7 in_seq=0 out_seq=0 2024-03-28T22:40:24.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.319+0000 7fb322000640 10 client.26824 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:24.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.322+0000 7fb322000640 1 -- 172.21.15.43:0/2482861686 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55646013e640 con 0x5564601e6770 2024-03-28T22:40:24.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.322+0000 7fb322000640 10 client.26824 handle_command_reply: tid=0 2024-03-28T22:40:24.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.322+0000 7fb3461bc740 1 client.26824 shutdown 2024-03-28T22:40:24.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.322+0000 7fb3461bc740 1 -- 172.21.15.43:0/2482861686 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5564601cdbc0 msgr2=0x5564601d3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:24.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.322+0000 7fb3461bc740 1 --2- 172.21.15.43:0/2482861686 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5564601cdbc0 0x5564601d3800 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb32802cfd0 tx=0x7fb32802ce10 comp rx=0 tx=0).stop 2024-03-28T22:40:24.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.322+0000 7fb3461bc740 1 -- 172.21.15.43:0/2482861686 shutdown_connections 2024-03-28T22:40:24.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.322+0000 7fb3461bc740 1 --2- 172.21.15.43:0/2482861686 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5564601e6770 0x5564601e8b80 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.322+0000 7fb3461bc740 1 --2- 172.21.15.43:0/2482861686 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564601d6250 0x5564601d3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.322+0000 7fb3461bc740 1 --2- 172.21.15.43:0/2482861686 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5564601cdbc0 0x5564601d3800 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.322+0000 7fb3461bc740 1 --2- 172.21.15.43:0/2482861686 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5564601ccee0 0x5564601d32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.322+0000 7fb3461bc740 1 -- 172.21.15.43:0/2482861686 >> 172.21.15.43:0/2482861686 conn(0x556460193a10 msgr2=0x5564601c9680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:24.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.322+0000 7fb322000640 0 client.26824 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:24.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.323+0000 7fb3461bc740 1 -- 172.21.15.43:0/2482861686 shutdown_connections 2024-03-28T22:40:24.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.323+0000 7fb3461bc740 1 -- 172.21.15.43:0/2482861686 wait complete. 2024-03-28T22:40:24.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.323+0000 7fb3461bc740 20 client.26824 trim_cache size 0 max 16384 2024-03-28T22:40:24.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.344+0000 7fb3461bc740 1 Processor -- start 2024-03-28T22:40:24.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.345+0000 7fb3461bc740 1 -- start start 2024-03-28T22:40:24.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.345+0000 7fb3461bc740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5564601ccee0 0x5564601d59f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.345+0000 7fb3461bc740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564601cdbc0 0x5564601e1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.345+0000 7fb3461bc740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5564601d6250 0x5564601c9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.345+0000 7fb3461bc740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb33c1b51a0 con 0x5564601ccee0 2024-03-28T22:40:24.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.345+0000 7fb3461bc740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb33c1b6ce0 con 0x5564601cdbc0 2024-03-28T22:40:24.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.345+0000 7fb3461bc740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fb33c1b5870 con 0x5564601d6250 2024-03-28T22:40:24.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.345+0000 7fb33be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564601cdbc0 0x5564601e1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.346+0000 7fb33be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564601cdbc0 0x5564601e1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45648/0 (socket says 172.21.15.43:45648) 2024-03-28T22:40:24.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.346+0000 7fb33be00640 1 -- 172.21.15.43:0/2867389296 learned_addr learned my addr 172.21.15.43:0/2867389296 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:24.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.346+0000 7fb341400640 1 --2- 172.21.15.43:0/2867389296 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5564601ccee0 0x5564601d59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.346+0000 7fb341400640 1 --2- 172.21.15.43:0/2867389296 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5564601d6250 0x5564601c9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.346+0000 7fb33be00640 1 -- 172.21.15.43:0/2867389296 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5564601d6250 msgr2=0x5564601c9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:24.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.346+0000 7fb33be00640 1 --2- 172.21.15.43:0/2867389296 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5564601d6250 0x5564601c9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.346+0000 7fb33be00640 1 -- 172.21.15.43:0/2867389296 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5564601ccee0 msgr2=0x5564601d59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:24.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.346+0000 7fb33be00640 1 --2- 172.21.15.43:0/2867389296 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5564601ccee0 0x5564601d59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.346+0000 7fb33be00640 1 -- 172.21.15.43:0/2867389296 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55645fe453f0 con 0x5564601cdbc0 2024-03-28T22:40:24.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.346+0000 7fb341400640 1 --2- 172.21.15.43:0/2867389296 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5564601d6250 0x5564601c9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:40:24.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.346+0000 7fb33be00640 1 --2- 172.21.15.43:0/2867389296 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564601cdbc0 0x5564601e1ea0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fb32802bc70 tx=0x7fb328009520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a1fbb792a1d7533 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:24.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.347+0000 7fb32d200640 1 -- 172.21.15.43:0/2867389296 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32805a070 con 0x5564601cdbc0 2024-03-28T22:40:24.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.347+0000 7fb32d200640 1 -- 172.21.15.43:0/2867389296 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb328005620 con 0x5564601cdbc0 2024-03-28T22:40:24.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.347+0000 7fb32d200640 1 -- 172.21.15.43:0/2867389296 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb328004000 con 0x5564601cdbc0 2024-03-28T22:40:24.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.348+0000 7fb3461bc740 1 -- 172.21.15.43:0/2867389296 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564601cdbc0 msgr2=0x5564601e1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:24.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.348+0000 7fb3461bc740 1 --2- 172.21.15.43:0/2867389296 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564601cdbc0 0x5564601e1ea0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fb32802bc70 tx=0x7fb328009520 comp rx=0 tx=0).stop 2024-03-28T22:40:24.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.348+0000 7fb3461bc740 1 -- 172.21.15.43:0/2867389296 shutdown_connections 2024-03-28T22:40:24.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.348+0000 7fb3461bc740 1 --2- 172.21.15.43:0/2867389296 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564601cdbc0 0x5564601e1ea0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.348+0000 7fb3461bc740 1 --2- 172.21.15.43:0/2867389296 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5564601d6250 0x5564601c9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.348+0000 7fb3461bc740 1 --2- 172.21.15.43:0/2867389296 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5564601ccee0 0x5564601d59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.348+0000 7fb3461bc740 1 -- 172.21.15.43:0/2867389296 >> 172.21.15.43:0/2867389296 conn(0x556460193a10 msgr2=0x5564601c8d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:24.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.348+0000 7fb3461bc740 1 -- 172.21.15.43:0/2867389296 shutdown_connections 2024-03-28T22:40:24.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.348+0000 7fb3461bc740 1 -- 172.21.15.43:0/2867389296 wait complete. 2024-03-28T22:40:24.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.348+0000 7fb3461bc740 1 -- start start 2024-03-28T22:40:24.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.348+0000 7fb3461bc740 1 Processor -- start 2024-03-28T22:40:24.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.349+0000 7fb3461bc740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564601ccee0 0x5564601d05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.349+0000 7fb3461bc740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5564601cdbc0 0x5564601d0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.349+0000 7fb3461bc740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5564601d6250 0x5564601d6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.349+0000 7fb3461bc740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb33c1b6610 con 0x5564601cdbc0 2024-03-28T22:40:24.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.349+0000 7fb3461bc740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb33c1b5f40 con 0x5564601ccee0 2024-03-28T22:40:24.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.349+0000 7fb3461bc740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fb33c1b3180 con 0x5564601d6250 2024-03-28T22:40:24.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.349+0000 7fb3461bc740 10 client.0 _ll_register_callbacks cb 0x7fb330005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:24.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.349+0000 7fb341400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564601ccee0 0x5564601d05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.349+0000 7fb341400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564601ccee0 0x5564601d05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:45650/0 (socket says 172.21.15.43:45650) 2024-03-28T22:40:24.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.349+0000 7fb341400640 1 -- 172.21.15.43:0/1046838986 learned_addr learned my addr 172.21.15.43:0/1046838986 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:24.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.349+0000 7fb33be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5564601cdbc0 0x5564601d0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.349+0000 7fb341400640 1 --2- 172.21.15.43:0/1046838986 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5564601d6250 0x5564601d6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.350+0000 7fb33be00640 1 -- 172.21.15.43:0/1046838986 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5564601d6250 msgr2=0x5564601d6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:24.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.350+0000 7fb33be00640 1 --2- 172.21.15.43:0/1046838986 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5564601d6250 0x5564601d6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.350+0000 7fb33be00640 1 -- 172.21.15.43:0/1046838986 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564601ccee0 msgr2=0x5564601d05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:24.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.350+0000 7fb33be00640 1 --2- 172.21.15.43:0/1046838986 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564601ccee0 0x5564601d05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.350+0000 7fb33be00640 1 -- 172.21.15.43:0/1046838986 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55645fe050f0 con 0x5564601cdbc0 2024-03-28T22:40:24.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.350+0000 7fb33be00640 1 --2- 172.21.15.43:0/1046838986 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5564601cdbc0 0x5564601d0af0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x5564601c29f0 tx=0x7fb32802ce10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f9bd4d7c0f7250c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:24.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.350+0000 7fb322000640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:40:24.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.350+0000 7fb322000640 1 -- 172.21.15.43:0/1046838986 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32805a070 con 0x5564601cdbc0 2024-03-28T22:40:24.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.350+0000 7fb3461bc740 1 -- 172.21.15.43:0/1046838986 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb328005620 con 0x5564601cdbc0 2024-03-28T22:40:24.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.350+0000 7fb322000640 1 -- 172.21.15.43:0/1046838986 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fb328005360 con 0x5564601cdbc0 2024-03-28T22:40:24.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.351+0000 7fb322000640 1 -- 172.21.15.43:0/1046838986 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb328009780 con 0x5564601cdbc0 2024-03-28T22:40:24.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.351+0000 7fb3461bc740 10 client.16680 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:24.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.351+0000 7fb3461bc740 1 -- 172.21.15.43:0/1046838986 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55646024cb20 con 0x5564601cdbc0 2024-03-28T22:40:24.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.351+0000 7fb322000640 1 -- 172.21.15.43:0/1046838986 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb328005620 con 0x5564601cdbc0 2024-03-28T22:40:24.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.351+0000 7fb322000640 1 -- 172.21.15.43:0/1046838986 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb328004a90 con 0x5564601cdbc0 2024-03-28T22:40:24.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.351+0000 7fb3461bc740 10 client.16680 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:24.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.351+0000 7fb3461bc740 10 client.16680 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:40:24.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.351+0000 7fb3461bc740 20 client.16680 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:24.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.351+0000 7fb3461bc740 1 --2- 172.21.15.43:0/1046838986 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55646024cfa0 0x55646024f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:24.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.351+0000 7fb3461bc740 4 client.16680 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:40:24.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.351+0000 7fb3461bc740 1 -- 172.21.15.43:0/1046838986 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556460138990 con 0x55646024cfa0 2024-03-28T22:40:24.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.352+0000 7fb341400640 1 --2- 172.21.15.43:0/1046838986 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55646024cfa0 0x55646024f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:24.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.352+0000 7fb341400640 1 --2- 172.21.15.43:0/1046838986 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55646024cfa0 0x55646024f390 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3c5c6c58be8904 server_cookie=dd0f72dace40b78c in_seq=0 out_seq=0 2024-03-28T22:40:24.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.352+0000 7fb322000640 10 client.16680 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:24.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.352+0000 7fb322000640 1 -- 172.21.15.43:0/1046838986 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x556460138990 con 0x55646024cfa0 2024-03-28T22:40:24.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.352+0000 7fb322000640 10 client.16680 handle_command_reply: tid=0 2024-03-28T22:40:24.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.352+0000 7fb3461bc740 1 client.16680 shutdown 2024-03-28T22:40:24.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.352+0000 7fb3461bc740 1 -- 172.21.15.43:0/1046838986 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5564601cdbc0 msgr2=0x5564601d0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:24.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.352+0000 7fb3461bc740 1 --2- 172.21.15.43:0/1046838986 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5564601cdbc0 0x5564601d0af0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x5564601c29f0 tx=0x7fb32802ce10 comp rx=0 tx=0).stop 2024-03-28T22:40:24.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.352+0000 7fb3461bc740 1 -- 172.21.15.43:0/1046838986 shutdown_connections 2024-03-28T22:40:24.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.352+0000 7fb3461bc740 1 --2- 172.21.15.43:0/1046838986 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5564601d6250 0x5564601d6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.352+0000 7fb3461bc740 1 --2- 172.21.15.43:0/1046838986 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5564601cdbc0 0x5564601d0af0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.352+0000 7fb3461bc740 1 --2- 172.21.15.43:0/1046838986 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55646024cfa0 0x55646024f390 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.352+0000 7fb3461bc740 1 --2- 172.21.15.43:0/1046838986 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564601ccee0 0x5564601d05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.353+0000 7fb322000640 0 client.16680 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:24.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.353+0000 7fb3461bc740 1 -- 172.21.15.43:0/1046838986 >> 172.21.15.43:0/1046838986 conn(0x556460193a10 msgr2=0x5564601c6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:24.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.353+0000 7fb3461bc740 1 -- 172.21.15.43:0/1046838986 shutdown_connections 2024-03-28T22:40:24.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.353+0000 7fb3461bc740 1 -- 172.21.15.43:0/1046838986 wait complete. 2024-03-28T22:40:24.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.353+0000 7fb3461bc740 20 client.16680 trim_cache size 0 max 16384 2024-03-28T22:40:24.355 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:40:24.355 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:40:24.355 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-03-28T22:40:24.355 INFO:teuthology.orchestra.run.smithi043.stdout: "cd29bd9b-d867-4e57-8fb9-09cc194a9aa0": { 2024-03-28T22:40:24.355 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-03-28T22:40:24.356 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "cd29bd9b-d867-4e57-8fb9-09cc194a9aa0", 2024-03-28T22:40:24.356 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-03-28T22:40:24.356 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:40:24.356 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:40:24.356 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:40:24.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.354+0000 7fb343400640 1 -- 172.21.15.43:0/2442876001 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb3240a1ef0 msgr2=0x7fb3240a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:24.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.354+0000 7fb343400640 1 --2- 172.21.15.43:0/2442876001 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb3240a1ef0 0x7fb3240a43b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x5564601ce110 tx=0x7fb330000bc0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.354+0000 7fb343400640 1 -- 172.21.15.43:0/2442876001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb33c075c80 msgr2=0x7fb33c19e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:24.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.354+0000 7fb343400640 1 --2- 172.21.15.43:0/2442876001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb33c075c80 0x7fb33c19e660 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb33000e9f0 tx=0x7fb33000eeb0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.354+0000 7fb343400640 1 -- 172.21.15.43:0/2442876001 shutdown_connections 2024-03-28T22:40:24.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.354+0000 7fb343400640 1 --2- 172.21.15.43:0/2442876001 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb3240a1ef0 0x7fb3240a43b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.354+0000 7fb343400640 1 --2- 172.21.15.43:0/2442876001 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb33c112590 0x7fb33c1a2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.354+0000 7fb343400640 1 --2- 172.21.15.43:0/2442876001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb33c0765a0 0x7fb33c19eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.354+0000 7fb343400640 1 --2- 172.21.15.43:0/2442876001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb33c075c80 0x7fb33c19e660 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:24.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.354+0000 7fb343400640 1 -- 172.21.15.43:0/2442876001 >> 172.21.15.43:0/2442876001 conn(0x7fb33c0ff400 msgr2=0x7fb33c1017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:24.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.354+0000 7fb343400640 1 -- 172.21.15.43:0/2442876001 shutdown_connections 2024-03-28T22:40:24.358 INFO:tasks.workunit.client.0.smithi043.stdout: 1 177847 8.50 MB/sec warmup 115 sec latency 11.403 ms 2024-03-28T22:40:24.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:24.354+0000 7fb343400640 1 -- 172.21.15.43:0/2442876001 wait complete. 2024-03-28T22:40:24.374 INFO:tasks.cephfs.filesystem:scrub status for tag:cd29bd9b-d867-4e57-8fb9-09cc194a9aa0 - {'path': '/', 'tag': 'cd29bd9b-d867-4e57-8fb9-09cc194a9aa0', 'options': 'recursive,force'} 2024-03-28T22:40:25.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:24 smithi043 ceph-mon[31823]: pgmap v101: 129 pgs: 129 active+clean; 43 MiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 432 op/s 2024-03-28T22:40:25.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:24 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:40:25.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:24 smithi043 ceph-mon[31823]: scrub queued for path: / 2024-03-28T22:40:25.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:24 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:40:25.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:24 smithi043 ceph-mon[31823]: scrub summary: active paths [/] 2024-03-28T22:40:25.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:24 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:40:25.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:24 smithi172 ceph-mon[36566]: pgmap v101: 129 pgs: 129 active+clean; 43 MiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 432 op/s 2024-03-28T22:40:25.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:24 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:40:25.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:24 smithi172 ceph-mon[36566]: scrub queued for path: / 2024-03-28T22:40:25.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:24 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:40:25.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:24 smithi172 ceph-mon[36566]: scrub summary: active paths [/] 2024-03-28T22:40:25.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:24 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:40:25.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:24 smithi082 ceph-mon[36471]: pgmap v101: 129 pgs: 129 active+clean; 43 MiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 432 op/s 2024-03-28T22:40:25.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:24 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:40:25.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:24 smithi082 ceph-mon[36471]: scrub queued for path: / 2024-03-28T22:40:25.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:24 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:40:25.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:24 smithi082 ceph-mon[36471]: scrub summary: active paths [/] 2024-03-28T22:40:25.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:24 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:40:25.352 INFO:tasks.workunit.client.0.smithi043.stdout: 1 179688 8.48 MB/sec warmup 116 sec latency 17.838 ms 2024-03-28T22:40:26.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:25 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:40:26.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:25 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:40:26.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:25 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:40:26.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:25 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:40:26.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:25 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:40:26.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:25 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:40:26.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:25 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:40:26.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:25 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:40:26.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:25 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:40:26.352 INFO:tasks.workunit.client.0.smithi043.stdout: 1 181268 8.51 MB/sec warmup 117 sec latency 10.879 ms 2024-03-28T22:40:27.352 INFO:tasks.workunit.client.0.smithi043.stdout: 1 182400 8.48 MB/sec warmup 118 sec latency 33.565 ms 2024-03-28T22:40:27.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:27 smithi172 ceph-mon[36566]: pgmap v102: 129 pgs: 129 active+clean; 43 MiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 259 op/s 2024-03-28T22:40:27.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:27 smithi082 ceph-mon[36471]: pgmap v102: 129 pgs: 129 active+clean; 43 MiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 259 op/s 2024-03-28T22:40:27.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:27 smithi043 ceph-mon[31823]: pgmap v102: 129 pgs: 129 active+clean; 43 MiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 259 op/s 2024-03-28T22:40:28.352 INFO:tasks.workunit.client.0.smithi043.stdout: 1 184530 8.52 MB/sec warmup 119 sec latency 19.087 ms 2024-03-28T22:40:29.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:29 smithi172 ceph-mon[36566]: pgmap v103: 129 pgs: 129 active+clean; 46 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 346 op/s 2024-03-28T22:40:29.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:29 smithi172 ceph-mon[36566]: scrub summary: idle 2024-03-28T22:40:29.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:29 smithi082 ceph-mon[36471]: pgmap v103: 129 pgs: 129 active+clean; 46 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 346 op/s 2024-03-28T22:40:29.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:29 smithi082 ceph-mon[36471]: scrub summary: idle 2024-03-28T22:40:29.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:29 smithi043 ceph-mon[31823]: pgmap v103: 129 pgs: 129 active+clean; 46 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 346 op/s 2024-03-28T22:40:29.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:29 smithi043 ceph-mon[31823]: scrub summary: idle 2024-03-28T22:40:30.352 INFO:tasks.workunit.client.0.smithi043.stdout: 1 187805 12.49 MB/sec execute 1 sec latency 21.831 ms 2024-03-28T22:40:31.352 INFO:tasks.workunit.client.0.smithi043.stdout: 1 188765 8.77 MB/sec execute 2 sec latency 21.574 ms 2024-03-28T22:40:31.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:31 smithi172 ceph-mon[36566]: pgmap v104: 129 pgs: 129 active+clean; 46 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 284 op/s 2024-03-28T22:40:31.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:31 smithi082 ceph-mon[36471]: pgmap v104: 129 pgs: 129 active+clean; 46 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 284 op/s 2024-03-28T22:40:31.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:31 smithi043 ceph-mon[31823]: pgmap v104: 129 pgs: 129 active+clean; 46 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 284 op/s 2024-03-28T22:40:32.359 INFO:tasks.workunit.client.0.smithi043.stdout: 1 190474 8.28 MB/sec execute 3 sec latency 38.132 ms 2024-03-28T22:40:33.353 INFO:tasks.workunit.client.0.smithi043.stdout: 1 191876 9.00 MB/sec execute 4 sec latency 17.866 ms 2024-03-28T22:40:33.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:33 smithi172 ceph-mon[36566]: pgmap v105: 129 pgs: 129 active+clean; 46 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 284 op/s 2024-03-28T22:40:33.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:33 smithi082 ceph-mon[36471]: pgmap v105: 129 pgs: 129 active+clean; 46 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 284 op/s 2024-03-28T22:40:33.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:33 smithi043 ceph-mon[31823]: pgmap v105: 129 pgs: 129 active+clean; 46 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 284 op/s 2024-03-28T22:40:34.200 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.202+0000 7f4cc1600640 1 -- 172.21.15.43:0/4237317040 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4cbc107c90 msgr2=0x7f4cbc1080d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:34.201 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.202+0000 7f4cc1600640 1 --2- 172.21.15.43:0/4237317040 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4cbc107c90 0x7f4cbc1080d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f4ca8009930 tx=0x7f4ca802bc40 comp rx=0 tx=0).stop 2024-03-28T22:40:34.201 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.203+0000 7f4cc1600640 1 -- 172.21.15.43:0/4237317040 shutdown_connections 2024-03-28T22:40:34.201 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.203+0000 7f4cc1600640 1 --2- 172.21.15.43:0/4237317040 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4cbc108610 0x7f4cbc114a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.201 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.203+0000 7f4cc1600640 1 --2- 172.21.15.43:0/4237317040 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4cbc107c90 0x7f4cbc1080d0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.201 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.203+0000 7f4cc1600640 1 --2- 172.21.15.43:0/4237317040 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cbc10dbf0 0x7f4cbc10dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.201 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.203+0000 7f4cc1600640 1 -- 172.21.15.43:0/4237317040 >> 172.21.15.43:0/4237317040 conn(0x7f4cbc077530 msgr2=0x7f4cbc106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:34.201 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.203+0000 7f4cc1600640 1 -- 172.21.15.43:0/4237317040 shutdown_connections 2024-03-28T22:40:34.201 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.203+0000 7f4cc1600640 1 -- 172.21.15.43:0/4237317040 wait complete. 2024-03-28T22:40:34.202 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.204+0000 7f4cc1600640 1 Processor -- start 2024-03-28T22:40:34.202 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.204+0000 7f4cc1600640 1 -- start start 2024-03-28T22:40:34.203 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.205+0000 7f4cc1600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cbc107c90 0x7f4cbc19e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:34.203 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.205+0000 7f4cc1600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4cbc108610 0x7f4cbc19e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:34.203 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.205+0000 7f4cc1600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4cbc10dbf0 0x7f4cbc1a2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:34.203 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.205+0000 7f4cc1600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc117fb0 con 0x7f4cbc107c90 2024-03-28T22:40:34.203 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.205+0000 7f4cc1600640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc117e30 con 0x7f4cbc10dbf0 2024-03-28T22:40:34.203 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.205+0000 7f4cc1600640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc118130 con 0x7f4cbc108610 2024-03-28T22:40:34.203 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.205+0000 7f4cbb400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4cbc108610 0x7f4cbc19e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:34.203 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.205+0000 7f4cbbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cbc107c90 0x7f4cbc19e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:34.203 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.205+0000 7f4cc0c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4cbc10dbf0 0x7f4cbc1a2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:34.204 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.205+0000 7f4cbb400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4cbc108610 0x7f4cbc19e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:38448/0 (socket says 172.21.15.43:38448) 2024-03-28T22:40:34.204 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.205+0000 7f4cbbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cbc107c90 0x7f4cbc19e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51652/0 (socket says 172.21.15.43:51652) 2024-03-28T22:40:34.204 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.205+0000 7f4cbb400640 1 -- 172.21.15.43:0/3674214562 learned_addr learned my addr 172.21.15.43:0/3674214562 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:34.205 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.206+0000 7f4cc0c00640 1 -- 172.21.15.43:0/3674214562 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4cbc108610 msgr2=0x7f4cbc19e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:34.205 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.206+0000 7f4cc0c00640 1 --2- 172.21.15.43:0/3674214562 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4cbc108610 0x7f4cbc19e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.205 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.206+0000 7f4cc0c00640 1 -- 172.21.15.43:0/3674214562 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cbc107c90 msgr2=0x7f4cbc19e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:34.205 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.206+0000 7f4cc0c00640 1 --2- 172.21.15.43:0/3674214562 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cbc107c90 0x7f4cbc19e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.206 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.206+0000 7f4cc0c00640 1 -- 172.21.15.43:0/3674214562 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cbc1a3460 con 0x7f4cbc10dbf0 2024-03-28T22:40:34.206 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.206+0000 7f4cbb400640 1 --2- 172.21.15.43:0/3674214562 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4cbc108610 0x7f4cbc19e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:34.206 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.206+0000 7f4cc0c00640 1 --2- 172.21.15.43:0/3674214562 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4cbc10dbf0 0x7f4cbc1a2d30 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f4cb400ec50 tx=0x7f4cb400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:34.206 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.206+0000 7f4cb8c00640 1 -- 172.21.15.43:0/3674214562 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb4003b70 con 0x7f4cbc10dbf0 2024-03-28T22:40:34.206 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.207+0000 7f4cb8c00640 1 -- 172.21.15.43:0/3674214562 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cb4003d10 con 0x7f4cbc10dbf0 2024-03-28T22:40:34.206 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.207+0000 7f4cc1600640 1 -- 172.21.15.43:0/3674214562 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cbc1a3750 con 0x7f4cbc10dbf0 2024-03-28T22:40:34.206 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.207+0000 7f4cb8c00640 1 -- 172.21.15.43:0/3674214562 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb4010890 con 0x7f4cbc10dbf0 2024-03-28T22:40:34.207 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.207+0000 7f4cc1600640 1 -- 172.21.15.43:0/3674214562 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cbc1aafe0 con 0x7f4cbc10dbf0 2024-03-28T22:40:34.207 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.210+0000 7f4cb8c00640 1 -- 172.21.15.43:0/3674214562 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f4cb4003fc0 con 0x7f4cbc10dbf0 2024-03-28T22:40:34.208 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.211+0000 7f4cb8c00640 1 --2- 172.21.15.43:0/3674214562 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4ca00a1ee0 0x7f4ca00a43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:34.209 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.211+0000 7f4cb8c00640 1 -- 172.21.15.43:0/3674214562 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f4cb40ceb20 con 0x7f4cbc10dbf0 2024-03-28T22:40:34.209 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.211+0000 7f4cbbe00640 1 --2- 172.21.15.43:0/3674214562 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4ca00a1ee0 0x7f4ca00a43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:34.209 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.212+0000 7f4cc5968740 1 Processor -- start 2024-03-28T22:40:34.211 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.212+0000 7f4cc5968740 1 -- start start 2024-03-28T22:40:34.211 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.212+0000 7f4cbbe00640 1 --2- 172.21.15.43:0/3674214562 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4ca00a1ee0 0x7f4ca00a43a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4cac009b50 tx=0x7f4cac005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:34.211 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.212+0000 7f4cc5968740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55daa6ad6570 0x55daa6ad6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:34.211 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.212+0000 7f4cc5968740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55daa6ad6e90 0x55daa6adf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:34.211 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.212+0000 7f4cc5968740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55daa6adfc60 0x55daa6ae2050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:34.211 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.212+0000 7f4cc5968740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55daa6a39640 con 0x55daa6ad6570 2024-03-28T22:40:34.211 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.212+0000 7f4cc5968740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55daa69e2810 con 0x55daa6ad6e90 2024-03-28T22:40:34.211 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.212+0000 7f4cc5968740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55daa6a356e0 con 0x55daa6adfc60 2024-03-28T22:40:34.211 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.213+0000 7f4cc0c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55daa6adfc60 0x55daa6ae2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:34.211 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.213+0000 7f4cbb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55daa6ad6e90 0x55daa6adf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:34.211 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.213+0000 7f4cbbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55daa6ad6570 0x55daa6ad6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:34.211 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.213+0000 7f4cc0c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55daa6adfc60 0x55daa6ae2050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:38454/0 (socket says 172.21.15.43:38454) 2024-03-28T22:40:34.211 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.213+0000 7f4cbb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55daa6ad6e90 0x55daa6adf720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:40136/0 (socket says 172.21.15.43:40136) 2024-03-28T22:40:34.212 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.213+0000 7f4cc0c00640 1 -- 172.21.15.43:0/2154265861 learned_addr learned my addr 172.21.15.43:0/2154265861 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:34.212 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.213+0000 7f4cc0c00640 1 -- 172.21.15.43:0/2154265861 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55daa6ad6e90 msgr2=0x55daa6adf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:34.212 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.213+0000 7f4cc0c00640 1 --2- 172.21.15.43:0/2154265861 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55daa6ad6e90 0x55daa6adf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.212 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.213+0000 7f4cc0c00640 1 -- 172.21.15.43:0/2154265861 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55daa6ad6570 msgr2=0x55daa6ad6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:34.212 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.213+0000 7f4cc0c00640 1 --2- 172.21.15.43:0/2154265861 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55daa6ad6570 0x55daa6ad6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.212 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.213+0000 7f4cc0c00640 1 -- 172.21.15.43:0/2154265861 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55daa674e210 con 0x55daa6adfc60 2024-03-28T22:40:34.214 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.213+0000 7f4cc0c00640 1 --2- 172.21.15.43:0/2154265861 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55daa6adfc60 0x55daa6ae2050 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4097ae0 tx=0x7f4cb4096db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f397a143015db25e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:34.214 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.214+0000 7f4cb1200640 1 -- 172.21.15.43:0/2154265861 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb409b810 con 0x55daa6adfc60 2024-03-28T22:40:34.214 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.214+0000 7f4cb1200640 1 -- 172.21.15.43:0/2154265861 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cb409b110 con 0x55daa6adfc60 2024-03-28T22:40:34.214 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.214+0000 7f4cb1200640 1 -- 172.21.15.43:0/2154265861 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb40a3420 con 0x55daa6adfc60 2024-03-28T22:40:34.214 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.215+0000 7f4cc5968740 1 -- 172.21.15.43:0/2154265861 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55daa6adfc60 msgr2=0x55daa6ae2050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:34.214 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.215+0000 7f4cc5968740 1 --2- 172.21.15.43:0/2154265861 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55daa6adfc60 0x55daa6ae2050 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4097ae0 tx=0x7f4cb4096db0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.214 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.215+0000 7f4cc5968740 1 -- 172.21.15.43:0/2154265861 shutdown_connections 2024-03-28T22:40:34.214 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.215+0000 7f4cc5968740 1 --2- 172.21.15.43:0/2154265861 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55daa6ad6e90 0x55daa6adf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.214 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.215+0000 7f4cc5968740 1 --2- 172.21.15.43:0/2154265861 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55daa6adfc60 0x55daa6ae2050 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.214 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.215+0000 7f4cc5968740 1 --2- 172.21.15.43:0/2154265861 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55daa6ad6570 0x55daa6ad6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.215 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.215+0000 7f4cc5968740 1 -- 172.21.15.43:0/2154265861 >> 172.21.15.43:0/2154265861 conn(0x55daa6ad0e90 msgr2=0x55daa6ad1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:34.215 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.215+0000 7f4cc5968740 1 -- 172.21.15.43:0/2154265861 shutdown_connections 2024-03-28T22:40:34.215 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.215+0000 7f4cc5968740 1 -- 172.21.15.43:0/2154265861 wait complete. 2024-03-28T22:40:34.215 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.216+0000 7f4cc5968740 1 -- start start 2024-03-28T22:40:34.215 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.216+0000 7f4cc5968740 1 Processor -- start 2024-03-28T22:40:34.219 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.217+0000 7f4cc5968740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55daa6ad6570 0x55daa6adcce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:34.219 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.217+0000 7f4cc5968740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55daa6ad6e90 0x55daa6add220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:34.219 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.217+0000 7f4cc5968740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55daa6adfc60 0x55daa6add760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:34.219 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.217+0000 7f4cc5968740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55daa6ae2890 con 0x55daa6ad6570 2024-03-28T22:40:34.219 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.217+0000 7f4cc5968740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55daa69d7d00 con 0x55daa6ad6e90 2024-03-28T22:40:34.219 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.217+0000 7f4cc5968740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55daa6ae2a10 con 0x55daa6adfc60 2024-03-28T22:40:34.219 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.217+0000 7f4cc5968740 10 client.0 _ll_register_callbacks cb 0x55daa69c2490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:34.219 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.217+0000 7f4cbbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55daa6ad6570 0x55daa6adcce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:34.219 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.217+0000 7f4cc0c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55daa6adfc60 0x55daa6add760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:34.219 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.217+0000 7f4cc0c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55daa6adfc60 0x55daa6add760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:38460/0 (socket says 172.21.15.43:38460) 2024-03-28T22:40:34.220 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.217+0000 7f4cc0c00640 1 -- 172.21.15.43:0/3622692130 learned_addr learned my addr 172.21.15.43:0/3622692130 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:34.220 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.217+0000 7f4cbb400640 1 --2- 172.21.15.43:0/3622692130 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55daa6ad6e90 0x55daa6add220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:34.220 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.217+0000 7f4cc0c00640 1 -- 172.21.15.43:0/3622692130 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55daa6ad6e90 msgr2=0x55daa6add220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:34.220 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.217+0000 7f4cc0c00640 1 --2- 172.21.15.43:0/3622692130 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55daa6ad6e90 0x55daa6add220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.220 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.217+0000 7f4cc0c00640 1 -- 172.21.15.43:0/3622692130 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55daa6ad6570 msgr2=0x55daa6adcce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:34.220 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.217+0000 7f4cc0c00640 1 --2- 172.21.15.43:0/3622692130 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55daa6ad6570 0x55daa6adcce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.220 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.218+0000 7f4cc0c00640 1 -- 172.21.15.43:0/3622692130 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55daa681a620 con 0x55daa6adfc60 2024-03-28T22:40:34.220 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.218+0000 7f4cbbe00640 1 --2- 172.21.15.43:0/3622692130 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55daa6ad6570 0x55daa6adcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:34.220 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.218+0000 7f4cc0c00640 1 --2- 172.21.15.43:0/3622692130 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55daa6adfc60 0x55daa6add760 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4cb409ae10 tx=0x7f4cb40cbde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b55d31731d416ed server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:34.220 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.218+0000 7f4ca5600640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:40:34.220 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.218+0000 7f4ca5600640 1 -- 172.21.15.43:0/3622692130 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb409b810 con 0x55daa6adfc60 2024-03-28T22:40:34.220 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.218+0000 7f4cc5968740 1 -- 172.21.15.43:0/3622692130 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4cb409b110 con 0x55daa6adfc60 2024-03-28T22:40:34.220 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.218+0000 7f4ca5600640 1 -- 172.21.15.43:0/3622692130 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f4cb40a3630 con 0x55daa6adfc60 2024-03-28T22:40:34.221 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.218+0000 7f4ca5600640 1 -- 172.21.15.43:0/3622692130 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4cb4081050 con 0x55daa6adfc60 2024-03-28T22:40:34.221 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.218+0000 7f4cc5968740 10 client.26839 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:34.221 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.218+0000 7f4cc5968740 1 -- 172.21.15.43:0/3622692130 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55daa6ae2890 con 0x55daa6adfc60 2024-03-28T22:40:34.221 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.219+0000 7f4ca5600640 1 -- 172.21.15.43:0/3622692130 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cb409b110 con 0x55daa6adfc60 2024-03-28T22:40:34.221 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.219+0000 7f4ca5600640 1 -- 172.21.15.43:0/3622692130 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4cb4099fd0 con 0x55daa6adfc60 2024-03-28T22:40:34.221 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.219+0000 7f4cc5968740 10 client.26839 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:34.221 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.219+0000 7f4cc5968740 10 client.26839 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:40:34.221 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.219+0000 7f4cc5968740 20 client.26839 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:34.221 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.219+0000 7f4cc5968740 1 --2- 172.21.15.43:0/3622692130 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55daa6aef930 0x55daa6af1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:34.221 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.219+0000 7f4cc5968740 4 client.26839 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:40:34.221 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.219+0000 7f4cc5968740 1 -- 172.21.15.43:0/3622692130 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55daa6a4d030 con 0x55daa6aef930 2024-03-28T22:40:34.221 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.219+0000 7f4cbbe00640 1 --2- 172.21.15.43:0/3622692130 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55daa6aef930 0x55daa6af1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:34.221 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.220+0000 7f4cbbe00640 1 --2- 172.21.15.43:0/3622692130 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55daa6aef930 0x55daa6af1d20 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d748a632680f4f6a server_cookie=bf9c8798fca54ef6 in_seq=0 out_seq=0 2024-03-28T22:40:34.221 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.220+0000 7f4ca5600640 10 client.26839 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.221+0000 7f4ca5600640 1 -- 172.21.15.43:0/3622692130 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55daa6a4d030 con 0x55daa6aef930 2024-03-28T22:40:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.221+0000 7f4ca5600640 10 client.26839 handle_command_reply: tid=0 2024-03-28T22:40:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.221+0000 7f4cc5968740 1 client.26839 shutdown 2024-03-28T22:40:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.221+0000 7f4cc5968740 1 -- 172.21.15.43:0/3622692130 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55daa6adfc60 msgr2=0x55daa6add760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.221+0000 7f4cc5968740 1 --2- 172.21.15.43:0/3622692130 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55daa6adfc60 0x55daa6add760 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4cb409ae10 tx=0x7f4cb40cbde0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.221+0000 7f4cc5968740 1 -- 172.21.15.43:0/3622692130 shutdown_connections 2024-03-28T22:40:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.221+0000 7f4cc5968740 1 --2- 172.21.15.43:0/3622692130 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55daa6aef930 0x55daa6af1d20 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.221+0000 7f4cc5968740 1 --2- 172.21.15.43:0/3622692130 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55daa6ad6e90 0x55daa6add220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.221+0000 7f4cc5968740 1 --2- 172.21.15.43:0/3622692130 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55daa6adfc60 0x55daa6add760 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.221+0000 7f4cc5968740 1 --2- 172.21.15.43:0/3622692130 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55daa6ad6570 0x55daa6adcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.221+0000 7f4ca5600640 0 client.26839 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.221+0000 7f4cc5968740 1 -- 172.21.15.43:0/3622692130 >> 172.21.15.43:0/3622692130 conn(0x55daa6ad0e90 msgr2=0x55daa6ad2be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.221+0000 7f4cc5968740 1 -- 172.21.15.43:0/3622692130 shutdown_connections 2024-03-28T22:40:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.221+0000 7f4cc5968740 1 -- 172.21.15.43:0/3622692130 wait complete. 2024-03-28T22:40:34.223 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.221+0000 7f4cc5968740 20 client.26839 trim_cache size 0 max 16384 2024-03-28T22:40:34.238 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.240+0000 7f4cc5968740 1 Processor -- start 2024-03-28T22:40:34.239 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.240+0000 7f4cc5968740 1 -- start start 2024-03-28T22:40:34.239 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.240+0000 7f4cc5968740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55daa6ad6570 0x55daa6ad0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:34.239 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.240+0000 7f4cc5968740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55daa6ad6e90 0x55daa6aeb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:34.239 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.240+0000 7f4cc5968740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55daa6adfc60 0x55daa6add580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:34.239 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.240+0000 7f4cc5968740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc1b50f0 con 0x55daa6adfc60 2024-03-28T22:40:34.239 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.240+0000 7f4cc5968740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc1b6c30 con 0x55daa6ad6570 2024-03-28T22:40:34.239 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.240+0000 7f4cc5968740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc1b57c0 con 0x55daa6ad6e90 2024-03-28T22:40:34.239 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.240+0000 7f4cbb400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55daa6ad6e90 0x55daa6aeb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:34.239 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.240+0000 7f4cbb400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55daa6ad6e90 0x55daa6aeb630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:38474/0 (socket says 172.21.15.43:38474) 2024-03-28T22:40:34.239 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.240+0000 7f4cbb400640 1 -- 172.21.15.43:0/2222210361 learned_addr learned my addr 172.21.15.43:0/2222210361 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:34.239 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.240+0000 7f4cbbe00640 1 --2- 172.21.15.43:0/2222210361 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55daa6ad6570 0x55daa6ad0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:34.239 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.241+0000 7f4cc0c00640 1 --2- 172.21.15.43:0/2222210361 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55daa6adfc60 0x55daa6add580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:34.240 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.241+0000 7f4cbb400640 1 -- 172.21.15.43:0/2222210361 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55daa6ad6570 msgr2=0x55daa6ad0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:34.240 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.241+0000 7f4cbb400640 1 --2- 172.21.15.43:0/2222210361 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55daa6ad6570 0x55daa6ad0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.240 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.241+0000 7f4cbb400640 1 -- 172.21.15.43:0/2222210361 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55daa6adfc60 msgr2=0x55daa6add580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:34.240 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.241+0000 7f4cbb400640 1 --2- 172.21.15.43:0/2222210361 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55daa6adfc60 0x55daa6add580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.240 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.241+0000 7f4cbb400640 1 -- 172.21.15.43:0/2222210361 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55daa6ad7b50 con 0x55daa6ad6e90 2024-03-28T22:40:34.240 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.241+0000 7f4cbbe00640 1 --2- 172.21.15.43:0/2222210361 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55daa6ad6570 0x55daa6ad0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:34.240 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.241+0000 7f4cc0c00640 1 --2- 172.21.15.43:0/2222210361 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55daa6adfc60 0x55daa6add580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:40:34.240 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.241+0000 7f4cbb400640 1 --2- 172.21.15.43:0/2222210361 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55daa6ad6e90 0x55daa6aeb630 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4ca8005ab0 tx=0x7f4ca8005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2fe8014d91e37ce server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:34.247 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.241+0000 7f4ca7e00640 1 -- 172.21.15.43:0/2222210361 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca804d070 con 0x55daa6ad6e90 2024-03-28T22:40:34.247 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.242+0000 7f4ca7e00640 1 -- 172.21.15.43:0/2222210361 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ca802c8c0 con 0x55daa6ad6e90 2024-03-28T22:40:34.247 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.242+0000 7f4ca7e00640 1 -- 172.21.15.43:0/2222210361 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca8003c90 con 0x55daa6ad6e90 2024-03-28T22:40:34.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.242+0000 7f4cc5968740 1 -- 172.21.15.43:0/2222210361 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55daa6ad6e90 msgr2=0x55daa6aeb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:34.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.242+0000 7f4cc5968740 1 --2- 172.21.15.43:0/2222210361 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55daa6ad6e90 0x55daa6aeb630 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4ca8005ab0 tx=0x7f4ca8005a20 comp rx=0 tx=0).stop 2024-03-28T22:40:34.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.242+0000 7f4cc5968740 1 -- 172.21.15.43:0/2222210361 shutdown_connections 2024-03-28T22:40:34.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.242+0000 7f4cc5968740 1 --2- 172.21.15.43:0/2222210361 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55daa6adfc60 0x55daa6add580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.243+0000 7f4cc5968740 1 --2- 172.21.15.43:0/2222210361 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55daa6ad6e90 0x55daa6aeb630 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.243+0000 7f4cc5968740 1 --2- 172.21.15.43:0/2222210361 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55daa6ad6570 0x55daa6ad0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.243+0000 7f4cc5968740 1 -- 172.21.15.43:0/2222210361 >> 172.21.15.43:0/2222210361 conn(0x55daa6ad0e90 msgr2=0x55daa6adbac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:34.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.243+0000 7f4cc5968740 1 -- 172.21.15.43:0/2222210361 shutdown_connections 2024-03-28T22:40:34.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.243+0000 7f4cc5968740 1 -- 172.21.15.43:0/2222210361 wait complete. 2024-03-28T22:40:34.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.243+0000 7f4cc5968740 1 -- start start 2024-03-28T22:40:34.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.243+0000 7f4cc5968740 1 Processor -- start 2024-03-28T22:40:34.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.244+0000 7f4cc5968740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55daa6ad6570 0x55daa6ae3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:34.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.244+0000 7f4cc5968740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55daa6ad6e90 0x55daa6ae3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:34.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.244+0000 7f4cc5968740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55daa6adfc60 0x55daa6b54a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:34.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.244+0000 7f4cc5968740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc1b6560 con 0x55daa6ad6570 2024-03-28T22:40:34.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.244+0000 7f4cc5968740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc1b5e90 con 0x55daa6adfc60 2024-03-28T22:40:34.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.244+0000 7f4cc5968740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc1b30d0 con 0x55daa6ad6e90 2024-03-28T22:40:34.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.244+0000 7f4cc5968740 10 client.0 _ll_register_callbacks cb 0x7f4cac0098c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:34.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.244+0000 7f4cc0c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55daa6adfc60 0x55daa6b54a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:34.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.244+0000 7f4cbb400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55daa6ad6e90 0x55daa6ae3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:34.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.244+0000 7f4cc0c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55daa6adfc60 0x55daa6b54a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:40170/0 (socket says 172.21.15.43:40170) 2024-03-28T22:40:34.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.244+0000 7f4cbb400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55daa6ad6e90 0x55daa6ae3ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:38480/0 (socket says 172.21.15.43:38480) 2024-03-28T22:40:34.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.244+0000 7f4cc0c00640 1 -- 172.21.15.43:0/1018908560 learned_addr learned my addr 172.21.15.43:0/1018908560 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:34.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.244+0000 7f4cbbe00640 1 --2- 172.21.15.43:0/1018908560 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55daa6ad6570 0x55daa6ae3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:34.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.244+0000 7f4cc0c00640 1 -- 172.21.15.43:0/1018908560 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55daa6ad6e90 msgr2=0x55daa6ae3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:34.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.244+0000 7f4cc0c00640 1 --2- 172.21.15.43:0/1018908560 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55daa6ad6e90 0x55daa6ae3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.244+0000 7f4cc0c00640 1 -- 172.21.15.43:0/1018908560 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55daa6ad6570 msgr2=0x55daa6ae3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:34.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.244+0000 7f4cc0c00640 1 --2- 172.21.15.43:0/1018908560 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55daa6ad6570 0x55daa6ae3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.250 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.244+0000 7f4cc0c00640 1 -- 172.21.15.43:0/1018908560 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55daa6b54f80 con 0x55daa6adfc60 2024-03-28T22:40:34.250 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.244+0000 7f4cbb400640 1 --2- 172.21.15.43:0/1018908560 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55daa6ad6e90 0x55daa6ae3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:34.250 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.245+0000 7f4cc0c00640 1 --2- 172.21.15.43:0/1018908560 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55daa6adfc60 0x55daa6b54a40 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4009bc0 tx=0x7f4cb400ccd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37d74eed51a115bb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:34.250 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.245+0000 7f4ca5600640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:40:34.250 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.245+0000 7f4ca5600640 1 -- 172.21.15.43:0/1018908560 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb409aee0 con 0x55daa6adfc60 2024-03-28T22:40:34.250 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.245+0000 7f4ca5600640 1 -- 172.21.15.43:0/1018908560 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f4cb4099fd0 con 0x55daa6adfc60 2024-03-28T22:40:34.250 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.245+0000 7f4cc5968740 1 -- 172.21.15.43:0/1018908560 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ca802c8c0 con 0x55daa6adfc60 2024-03-28T22:40:34.250 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.245+0000 7f4ca5600640 1 -- 172.21.15.43:0/1018908560 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4cb409b810 con 0x55daa6adfc60 2024-03-28T22:40:34.250 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.245+0000 7f4cc5968740 10 client.26894 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:34.250 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.245+0000 7f4cc5968740 1 -- 172.21.15.43:0/1018908560 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55daa6b56130 con 0x55daa6adfc60 2024-03-28T22:40:34.250 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.246+0000 7f4ca5600640 1 -- 172.21.15.43:0/1018908560 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ca802c8c0 con 0x55daa6adfc60 2024-03-28T22:40:34.250 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.246+0000 7f4ca5600640 1 -- 172.21.15.43:0/1018908560 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4cb4084040 con 0x55daa6adfc60 2024-03-28T22:40:34.250 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.246+0000 7f4cc5968740 10 client.26894 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:34.251 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.246+0000 7f4cc5968740 10 client.26894 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:40:34.251 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.246+0000 7f4cc5968740 20 client.26894 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:34.251 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.246+0000 7f4cc5968740 1 --2- 172.21.15.43:0/1018908560 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55daa6b56660 0x55daa6b58a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:34.251 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.246+0000 7f4cc5968740 4 client.26894 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T22:40:34.251 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.246+0000 7f4cc5968740 1 -- 172.21.15.43:0/1018908560 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55daa6a46b80 con 0x55daa6b56660 2024-03-28T22:40:34.251 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.246+0000 7f4cbbe00640 1 --2- 172.21.15.43:0/1018908560 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55daa6b56660 0x55daa6b58a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:34.251 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.247+0000 7f4cbbe00640 1 --2- 172.21.15.43:0/1018908560 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55daa6b56660 0x55daa6b58a50 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c74041ab7a476a6 server_cookie=b8cfca4f4099f43e in_seq=0 out_seq=0 2024-03-28T22:40:34.251 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.247+0000 7f4ca5600640 10 client.26894 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:34.353 INFO:tasks.workunit.client.0.smithi043.stdout: 1 192682 7.67 MB/sec execute 5 sec latency 48.158 ms 2024-03-28T22:40:34.440 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.441+0000 7f4ca5600640 1 -- 172.21.15.43:0/1018908560 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55daa6a46b80 con 0x55daa6b56660 2024-03-28T22:40:34.440 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.441+0000 7f4ca5600640 10 client.26894 handle_command_reply: tid=0 2024-03-28T22:40:34.440 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.441+0000 7f4cc5968740 1 client.26894 shutdown 2024-03-28T22:40:34.440 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.442+0000 7f4cc5968740 1 -- 172.21.15.43:0/1018908560 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55daa6adfc60 msgr2=0x55daa6b54a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:34.440 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.442+0000 7f4cc5968740 1 --2- 172.21.15.43:0/1018908560 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55daa6adfc60 0x55daa6b54a40 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4009bc0 tx=0x7f4cb400ccd0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.440 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.442+0000 7f4cc5968740 1 -- 172.21.15.43:0/1018908560 shutdown_connections 2024-03-28T22:40:34.440 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.442+0000 7f4cc5968740 1 --2- 172.21.15.43:0/1018908560 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55daa6b56660 0x55daa6b58a50 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.440 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.442+0000 7f4cc5968740 1 --2- 172.21.15.43:0/1018908560 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55daa6adfc60 0x55daa6b54a40 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.440 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.442+0000 7f4cc5968740 1 --2- 172.21.15.43:0/1018908560 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55daa6ad6e90 0x55daa6ae3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.441 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.442+0000 7f4ca5600640 0 client.26894 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:34.441 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.442+0000 7f4cc5968740 1 --2- 172.21.15.43:0/1018908560 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55daa6ad6570 0x55daa6ae3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.441 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.442+0000 7f4cc5968740 1 -- 172.21.15.43:0/1018908560 >> 172.21.15.43:0/1018908560 conn(0x55daa6ad0e90 msgr2=0x55daa6add240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:34.441 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.442+0000 7f4cc5968740 1 -- 172.21.15.43:0/1018908560 shutdown_connections 2024-03-28T22:40:34.441 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.442+0000 7f4cc5968740 1 -- 172.21.15.43:0/1018908560 wait complete. 2024-03-28T22:40:34.441 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.442+0000 7f4cc5968740 20 client.26894 trim_cache size 0 max 16384 2024-03-28T22:40:34.441 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:{ 2024-03-28T22:40:34.441 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "message": "", 2024-03-28T22:40:34.441 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "return_code": 0 2024-03-28T22:40:34.441 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:} 2024-03-28T22:40:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.443+0000 7f4cc1600640 1 -- 172.21.15.43:0/3674214562 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4ca00a1ee0 msgr2=0x7f4ca00a43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.443+0000 7f4cc1600640 1 --2- 172.21.15.43:0/3674214562 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4ca00a1ee0 0x7f4ca00a43a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4cac009b50 tx=0x7f4cac005f90 comp rx=0 tx=0).stop 2024-03-28T22:40:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.443+0000 7f4cc1600640 1 -- 172.21.15.43:0/3674214562 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4cbc10dbf0 msgr2=0x7f4cbc1a2d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:34.442 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.443+0000 7f4cc1600640 1 --2- 172.21.15.43:0/3674214562 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4cbc10dbf0 0x7f4cbc1a2d30 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f4cb400ec50 tx=0x7f4cb400c6a0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.443+0000 7f4cc1600640 1 -- 172.21.15.43:0/3674214562 shutdown_connections 2024-03-28T22:40:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.443+0000 7f4cc1600640 1 --2- 172.21.15.43:0/3674214562 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4ca00a1ee0 0x7f4ca00a43a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.443+0000 7f4cc1600640 1 --2- 172.21.15.43:0/3674214562 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4cbc10dbf0 0x7f4cbc1a2d30 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.443+0000 7f4cc1600640 1 --2- 172.21.15.43:0/3674214562 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4cbc108610 0x7f4cbc19e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.443+0000 7f4cc1600640 1 --2- 172.21.15.43:0/3674214562 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cbc107c90 0x7f4cbc19e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.443+0000 7f4cc1600640 1 -- 172.21.15.43:0/3674214562 >> 172.21.15.43:0/3674214562 conn(0x7f4cbc077530 msgr2=0x7f4cbc0785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.443+0000 7f4cc1600640 1 -- 172.21.15.43:0/3674214562 shutdown_connections 2024-03-28T22:40:34.443 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:34.443+0000 7f4cc1600640 1 -- 172.21.15.43:0/3674214562 wait complete. 2024-03-28T22:40:35.353 INFO:tasks.workunit.client.0.smithi043.stdout: 1 194538 8.14 MB/sec execute 6 sec latency 104.072 ms 2024-03-28T22:40:35.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:35 smithi172 ceph-mon[36566]: pgmap v106: 129 pgs: 129 active+clean; 49 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 362 op/s 2024-03-28T22:40:35.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:35 smithi082 ceph-mon[36471]: pgmap v106: 129 pgs: 129 active+clean; 49 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 362 op/s 2024-03-28T22:40:35.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:35 smithi043 ceph-mon[31823]: pgmap v106: 129 pgs: 129 active+clean; 49 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 362 op/s 2024-03-28T22:40:36.353 INFO:tasks.workunit.client.0.smithi043.stdout: 1 195633 7.92 MB/sec execute 7 sec latency 11.593 ms 2024-03-28T22:40:37.353 INFO:tasks.workunit.client.0.smithi043.stdout: 1 196755 7.58 MB/sec execute 8 sec latency 39.210 ms 2024-03-28T22:40:37.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:37 smithi172 ceph-mon[36566]: pgmap v107: 129 pgs: 129 active+clean; 49 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 165 op/s 2024-03-28T22:40:37.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:37 smithi082 ceph-mon[36471]: pgmap v107: 129 pgs: 129 active+clean; 49 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 165 op/s 2024-03-28T22:40:37.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:37 smithi043 ceph-mon[31823]: pgmap v107: 129 pgs: 129 active+clean; 49 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 165 op/s 2024-03-28T22:40:38.353 INFO:tasks.workunit.client.0.smithi043.stdout: 1 198784 8.14 MB/sec execute 9 sec latency 69.540 ms 2024-03-28T22:40:39.353 INFO:tasks.workunit.client.0.smithi043.stdout: 1 199860 7.66 MB/sec execute 10 sec latency 17.448 ms 2024-03-28T22:40:39.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:39 smithi172 ceph-mon[36566]: pgmap v108: 129 pgs: 129 active+clean; 45 MiB data, 789 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 346 op/s 2024-03-28T22:40:39.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:39 smithi082 ceph-mon[36471]: pgmap v108: 129 pgs: 129 active+clean; 45 MiB data, 789 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 346 op/s 2024-03-28T22:40:39.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:39 smithi043 ceph-mon[31823]: pgmap v108: 129 pgs: 129 active+clean; 45 MiB data, 789 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 346 op/s 2024-03-28T22:40:40.354 INFO:tasks.workunit.client.0.smithi043.stdout: 1 202151 8.36 MB/sec execute 11 sec latency 14.646 ms 2024-03-28T22:40:41.354 INFO:tasks.workunit.client.0.smithi043.stdout: 1 203059 7.86 MB/sec execute 12 sec latency 19.552 ms 2024-03-28T22:40:41.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:41 smithi172 ceph-mon[36566]: pgmap v109: 129 pgs: 129 active+clean; 45 MiB data, 789 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 258 op/s 2024-03-28T22:40:41.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:41 smithi082 ceph-mon[36471]: pgmap v109: 129 pgs: 129 active+clean; 45 MiB data, 789 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 258 op/s 2024-03-28T22:40:41.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:41 smithi043 ceph-mon[31823]: pgmap v109: 129 pgs: 129 active+clean; 45 MiB data, 789 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 258 op/s 2024-03-28T22:40:42.354 INFO:tasks.workunit.client.0.smithi043.stdout: 1 205181 8.17 MB/sec execute 13 sec latency 10.776 ms 2024-03-28T22:40:43.354 INFO:tasks.workunit.client.0.smithi043.stdout: 1 206396 8.06 MB/sec execute 14 sec latency 44.909 ms 2024-03-28T22:40:43.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:43 smithi172 ceph-mon[36566]: pgmap v110: 129 pgs: 129 active+clean; 45 MiB data, 789 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 258 op/s 2024-03-28T22:40:43.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:43 smithi082 ceph-mon[36471]: pgmap v110: 129 pgs: 129 active+clean; 45 MiB data, 789 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 258 op/s 2024-03-28T22:40:43.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:43 smithi043 ceph-mon[31823]: pgmap v110: 129 pgs: 129 active+clean; 45 MiB data, 789 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 258 op/s 2024-03-28T22:40:44.354 INFO:tasks.workunit.client.0.smithi043.stdout: 1 208207 8.04 MB/sec execute 15 sec latency 16.873 ms 2024-03-28T22:40:45.354 INFO:tasks.workunit.client.0.smithi043.stdout: 1 209728 8.24 MB/sec execute 16 sec latency 17.113 ms 2024-03-28T22:40:45.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:45 smithi172 ceph-mon[36566]: pgmap v111: 129 pgs: 129 active+clean; 43 MiB data, 828 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 330 op/s 2024-03-28T22:40:45.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:45 smithi082 ceph-mon[36471]: pgmap v111: 129 pgs: 129 active+clean; 43 MiB data, 828 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 330 op/s 2024-03-28T22:40:45.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:45 smithi043 ceph-mon[31823]: pgmap v111: 129 pgs: 129 active+clean; 43 MiB data, 828 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.3 MiB/s wr, 330 op/s 2024-03-28T22:40:46.354 INFO:tasks.workunit.client.0.smithi043.stdout: 1 211502 8.10 MB/sec execute 17 sec latency 17.622 ms 2024-03-28T22:40:47.354 INFO:tasks.workunit.client.0.smithi043.stdout: 1 213131 8.37 MB/sec execute 18 sec latency 15.325 ms 2024-03-28T22:40:47.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:47 smithi172 ceph-mon[36566]: pgmap v112: 129 pgs: 129 active+clean; 43 MiB data, 828 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 252 op/s 2024-03-28T22:40:47.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:47 smithi082 ceph-mon[36471]: pgmap v112: 129 pgs: 129 active+clean; 43 MiB data, 828 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 252 op/s 2024-03-28T22:40:47.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:47 smithi043 ceph-mon[31823]: pgmap v112: 129 pgs: 129 active+clean; 43 MiB data, 828 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 252 op/s 2024-03-28T22:40:47.633 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.634+0000 7f5b81e00640 1 -- 172.21.15.43:0/2559873650 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5b7c109930 msgr2=0x7f5b7c109d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:47.633 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.634+0000 7f5b81e00640 1 --2- 172.21.15.43:0/2559873650 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5b7c109930 0x7f5b7c109d10 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c009900 tx=0x7f5b6c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:40:47.633 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.634+0000 7f5b81e00640 1 -- 172.21.15.43:0/2559873650 shutdown_connections 2024-03-28T22:40:47.633 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.634+0000 7f5b81e00640 1 --2- 172.21.15.43:0/2559873650 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b7c1042e0 0x7f5b7c1107f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.633 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.634+0000 7f5b81e00640 1 --2- 172.21.15.43:0/2559873650 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5b7c103920 0x7f5b7c103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.634 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.634+0000 7f5b81e00640 1 --2- 172.21.15.43:0/2559873650 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5b7c109930 0x7f5b7c109d10 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.634 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.634+0000 7f5b81e00640 1 -- 172.21.15.43:0/2559873650 >> 172.21.15.43:0/2559873650 conn(0x7f5b7c0ff410 msgr2=0x7f5b7c101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:47.634 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.634+0000 7f5b81e00640 1 -- 172.21.15.43:0/2559873650 shutdown_connections 2024-03-28T22:40:47.634 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.634+0000 7f5b81e00640 1 -- 172.21.15.43:0/2559873650 wait complete. 2024-03-28T22:40:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.635+0000 7f5b81e00640 1 Processor -- start 2024-03-28T22:40:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.635+0000 7f5b81e00640 1 -- start start 2024-03-28T22:40:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.636+0000 7f5b81e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5b7c103920 0x7f5b7c19a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.636+0000 7f5b81e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5b7c1042e0 0x7f5b7c19a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.636+0000 7f5b81e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b7c109930 0x7f5b7c19ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.636+0000 7f5b81e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7c113d90 con 0x7f5b7c1042e0 2024-03-28T22:40:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.636+0000 7f5b81e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7c113c10 con 0x7f5b7c103920 2024-03-28T22:40:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.636+0000 7f5b81e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7c113f10 con 0x7f5b7c109930 2024-03-28T22:40:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.636+0000 7f5b7aa00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5b7c103920 0x7f5b7c19a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.636+0000 7f5b7aa00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5b7c103920 0x7f5b7c19a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59290/0 (socket says 172.21.15.43:59290) 2024-03-28T22:40:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.636+0000 7f5b7aa00640 1 -- 172.21.15.43:0/3908441195 learned_addr learned my addr 172.21.15.43:0/3908441195 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.636+0000 7f5b7b400640 1 --2- 172.21.15.43:0/3908441195 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b7c109930 0x7f5b7c19ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:47.637 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.636+0000 7f5b7a000640 1 --2- 172.21.15.43:0/3908441195 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5b7c1042e0 0x7f5b7c19a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.636+0000 7f5b7aa00640 1 -- 172.21.15.43:0/3908441195 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b7c109930 msgr2=0x7f5b7c19ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.636+0000 7f5b7aa00640 1 --2- 172.21.15.43:0/3908441195 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b7c109930 0x7f5b7c19ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.636+0000 7f5b7aa00640 1 -- 172.21.15.43:0/3908441195 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5b7c1042e0 msgr2=0x7f5b7c19a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.636+0000 7f5b7aa00640 1 --2- 172.21.15.43:0/3908441195 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5b7c1042e0 0x7f5b7c19a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.636+0000 7f5b7aa00640 1 -- 172.21.15.43:0/3908441195 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b7c19f1b0 con 0x7f5b7c103920 2024-03-28T22:40:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.637+0000 7f5b7aa00640 1 --2- 172.21.15.43:0/3908441195 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5b7c103920 0x7f5b7c19a210 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c02efd0 tx=0x7f5b6c004430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.637+0000 7f5b80a00640 1 -- 172.21.15.43:0/3908441195 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c041070 con 0x7f5b7c103920 2024-03-28T22:40:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.637+0000 7f5b80a00640 1 -- 172.21.15.43:0/3908441195 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b6c0054e0 con 0x7f5b7c103920 2024-03-28T22:40:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.637+0000 7f5b80a00640 1 -- 172.21.15.43:0/3908441195 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c003d70 con 0x7f5b7c103920 2024-03-28T22:40:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.637+0000 7f5b81e00640 1 -- 172.21.15.43:0/3908441195 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b7c19f440 con 0x7f5b7c103920 2024-03-28T22:40:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.637+0000 7f5b81e00640 1 -- 172.21.15.43:0/3908441195 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b7c1a6d30 con 0x7f5b7c103920 2024-03-28T22:40:47.639 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.641+0000 7f5b84a72740 1 Processor -- start 2024-03-28T22:40:47.639 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.641+0000 7f5b84a72740 1 -- start start 2024-03-28T22:40:47.639 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.641+0000 7f5b80a00640 1 -- 172.21.15.43:0/3908441195 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f5b6c044070 con 0x7f5b7c103920 2024-03-28T22:40:47.640 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.641+0000 7f5b84a72740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555a0271d3a0 0x555a0271d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:47.640 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.641+0000 7f5b84a72740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555a0271dcc0 0x555a02726550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:47.640 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.641+0000 7f5b84a72740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555a02726a90 0x555a02728e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:47.640 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.641+0000 7f5b84a72740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x555a02680640 con 0x555a02726a90 2024-03-28T22:40:47.640 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.641+0000 7f5b84a72740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x555a02629810 con 0x555a0271dcc0 2024-03-28T22:40:47.640 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.641+0000 7f5b84a72740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x555a0267c6e0 con 0x555a0271d3a0 2024-03-28T22:40:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.641+0000 7f5b7b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555a0271d3a0 0x555a0271d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.641+0000 7f5b7aa00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555a0271dcc0 0x555a02726550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.641+0000 7f5b7b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555a0271d3a0 0x555a0271d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:49750/0 (socket says 172.21.15.43:49750) 2024-03-28T22:40:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.641+0000 7f5b7a000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555a02726a90 0x555a02728e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.641+0000 7f5b7b400640 1 -- 172.21.15.43:0/1330431008 learned_addr learned my addr 172.21.15.43:0/1330431008 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.642+0000 7f5b7b400640 1 -- 172.21.15.43:0/1330431008 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555a0271dcc0 msgr2=0x555a02726550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.642+0000 7f5b7b400640 1 --2- 172.21.15.43:0/1330431008 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555a0271dcc0 0x555a02726550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.642+0000 7f5b7b400640 1 -- 172.21.15.43:0/1330431008 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555a02726a90 msgr2=0x555a02728e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.642+0000 7f5b7b400640 1 --2- 172.21.15.43:0/1330431008 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555a02726a90 0x555a02728e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.642+0000 7f5b7b400640 1 -- 172.21.15.43:0/1330431008 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a02395210 con 0x555a0271d3a0 2024-03-28T22:40:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.642+0000 7f5b7b400640 1 --2- 172.21.15.43:0/1330431008 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555a0271d3a0 0x555a0271d780 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5b74009f50 tx=0x7f5b740118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f9546aca4516778 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.642+0000 7f5b67e00640 1 -- 172.21.15.43:0/1330431008 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b74004d20 con 0x555a0271d3a0 2024-03-28T22:40:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.642+0000 7f5b67e00640 1 -- 172.21.15.43:0/1330431008 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b74004050 con 0x555a0271d3a0 2024-03-28T22:40:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.642+0000 7f5b67e00640 1 -- 172.21.15.43:0/1330431008 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b74005280 con 0x555a0271d3a0 2024-03-28T22:40:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.643+0000 7f5b80a00640 1 --2- 172.21.15.43:0/3908441195 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f5b600a2010 0x7f5b600a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:47.645 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.643+0000 7f5b80a00640 1 -- 172.21.15.43:0/3908441195 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f5b6c0eff10 con 0x7f5b7c103920 2024-03-28T22:40:47.646 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.643+0000 7f5b7aa00640 1 --2- 172.21.15.43:0/3908441195 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f5b600a2010 0x7f5b600a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:47.646 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.643+0000 7f5b84a72740 1 -- 172.21.15.43:0/1330431008 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555a0271d3a0 msgr2=0x555a0271d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:47.646 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.643+0000 7f5b84a72740 1 --2- 172.21.15.43:0/1330431008 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555a0271d3a0 0x555a0271d780 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5b74009f50 tx=0x7f5b740118d0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.646 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.643+0000 7f5b84a72740 1 -- 172.21.15.43:0/1330431008 shutdown_connections 2024-03-28T22:40:47.646 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.643+0000 7f5b84a72740 1 --2- 172.21.15.43:0/1330431008 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555a0271dcc0 0x555a02726550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.646 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.643+0000 7f5b84a72740 1 --2- 172.21.15.43:0/1330431008 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555a02726a90 0x555a02728e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.646 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.643+0000 7f5b84a72740 1 --2- 172.21.15.43:0/1330431008 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555a0271d3a0 0x555a0271d780 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.646 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.643+0000 7f5b84a72740 1 -- 172.21.15.43:0/1330431008 >> 172.21.15.43:0/1330431008 conn(0x555a02717cc0 msgr2=0x555a027180a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:47.646 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.643+0000 7f5b84a72740 1 -- 172.21.15.43:0/1330431008 shutdown_connections 2024-03-28T22:40:47.646 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.644+0000 7f5b84a72740 1 -- 172.21.15.43:0/1330431008 wait complete. 2024-03-28T22:40:47.646 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.644+0000 7f5b7aa00640 1 --2- 172.21.15.43:0/3908441195 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f5b600a2010 0x7f5b600a44d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x555a0271ce50 tx=0x7f5b6c0dc040 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:47.646 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.644+0000 7f5b84a72740 1 -- start start 2024-03-28T22:40:47.646 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.644+0000 7f5b84a72740 1 Processor -- start 2024-03-28T22:40:47.646 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.645+0000 7f5b84a72740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555a0271d3a0 0x555a02723af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:47.647 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.645+0000 7f5b84a72740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555a0271dcc0 0x555a02724030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:47.647 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.645+0000 7f5b84a72740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555a02726a90 0x555a02724570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:47.647 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.645+0000 7f5b84a72740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x555a027296c0 con 0x555a02726a90 2024-03-28T22:40:47.647 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.645+0000 7f5b84a72740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x555a0261ed00 con 0x555a0271d3a0 2024-03-28T22:40:47.647 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.645+0000 7f5b84a72740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x555a02729840 con 0x555a0271dcc0 2024-03-28T22:40:47.647 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.645+0000 7f5b84a72740 10 client.0 _ll_register_callbacks cb 0x555a02609490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:47.647 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.645+0000 7f5b7a000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555a0271dcc0 0x555a02724030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:47.647 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.645+0000 7f5b7a000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555a0271dcc0 0x555a02724030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:49764/0 (socket says 172.21.15.43:49764) 2024-03-28T22:40:47.647 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.645+0000 7f5b7a000640 1 -- 172.21.15.43:0/3511637505 learned_addr learned my addr 172.21.15.43:0/3511637505 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:47.647 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.645+0000 7f5b7b400640 1 --2- 172.21.15.43:0/3511637505 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555a0271d3a0 0x555a02723af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:47.647 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.645+0000 7f5b7b400640 1 --2- 172.21.15.43:0/3511637505 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555a02726a90 0x555a02724570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:47.647 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.645+0000 7f5b7a000640 1 -- 172.21.15.43:0/3511637505 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555a0271d3a0 msgr2=0x555a02723af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:47.648 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.645+0000 7f5b7a000640 1 --2- 172.21.15.43:0/3511637505 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555a0271d3a0 0x555a02723af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.648 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.645+0000 7f5b7a000640 1 -- 172.21.15.43:0/3511637505 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555a02726a90 msgr2=0x555a02724570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:47.648 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.645+0000 7f5b7a000640 1 --2- 172.21.15.43:0/3511637505 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555a02726a90 0x555a02724570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.648 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.645+0000 7f5b7a000640 1 -- 172.21.15.43:0/3511637505 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a02461620 con 0x555a0271dcc0 2024-03-28T22:40:47.648 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.646+0000 7f5b7a000640 1 --2- 172.21.15.43:0/3511637505 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555a0271dcc0 0x555a02724030 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f5b6800bb00 tx=0x7f5b680044c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=854d817531a85d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:47.648 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.646+0000 7f5b64c00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:40:47.648 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.646+0000 7f5b64c00640 1 -- 172.21.15.43:0/3511637505 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b68003e20 con 0x555a0271dcc0 2024-03-28T22:40:47.648 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.646+0000 7f5b84a72740 1 -- 172.21.15.43:0/3511637505 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b74004050 con 0x555a0271dcc0 2024-03-28T22:40:47.648 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.646+0000 7f5b64c00640 1 -- 172.21.15.43:0/3511637505 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f5b68011440 con 0x555a0271dcc0 2024-03-28T22:40:47.648 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.646+0000 7f5b64c00640 1 -- 172.21.15.43:0/3511637505 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5b68011640 con 0x555a0271dcc0 2024-03-28T22:40:47.648 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.646+0000 7f5b84a72740 10 client.26857 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:47.648 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.646+0000 7f5b84a72740 1 -- 172.21.15.43:0/3511637505 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x555a027356e0 con 0x555a0271dcc0 2024-03-28T22:40:47.648 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.647+0000 7f5b64c00640 1 -- 172.21.15.43:0/3511637505 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b74004050 con 0x555a0271dcc0 2024-03-28T22:40:47.648 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.647+0000 7f5b64c00640 1 -- 172.21.15.43:0/3511637505 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f5b68010980 con 0x555a0271dcc0 2024-03-28T22:40:47.648 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.647+0000 7f5b84a72740 10 client.26857 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:47.649 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.647+0000 7f5b84a72740 10 client.26857 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:40:47.649 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.647+0000 7f5b84a72740 20 client.26857 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:47.649 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.647+0000 7f5b84a72740 1 --2- 172.21.15.43:0/3511637505 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x555a02736bf0 0x555a02738fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:47.649 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.647+0000 7f5b84a72740 4 client.26857 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:40:47.649 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.647+0000 7f5b84a72740 1 -- 172.21.15.43:0/3511637505 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a02694030 con 0x555a02736bf0 2024-03-28T22:40:47.649 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.647+0000 7f5b7b400640 1 --2- 172.21.15.43:0/3511637505 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x555a02736bf0 0x555a02738fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:47.650 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.648+0000 7f5b7b400640 1 --2- 172.21.15.43:0/3511637505 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x555a02736bf0 0x555a02738fe0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b0037bd6a977938 server_cookie=168252ab31fc0e1 in_seq=0 out_seq=0 2024-03-28T22:40:47.650 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.648+0000 7f5b64c00640 10 client.26857 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:47.650 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.649+0000 7f5b64c00640 1 -- 172.21.15.43:0/3511637505 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555a02694030 con 0x555a02736bf0 2024-03-28T22:40:47.650 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.649+0000 7f5b64c00640 10 client.26857 handle_command_reply: tid=0 2024-03-28T22:40:47.650 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.649+0000 7f5b84a72740 1 client.26857 shutdown 2024-03-28T22:40:47.651 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.649+0000 7f5b84a72740 1 -- 172.21.15.43:0/3511637505 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555a0271dcc0 msgr2=0x555a02724030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:47.651 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.649+0000 7f5b84a72740 1 --2- 172.21.15.43:0/3511637505 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555a0271dcc0 0x555a02724030 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f5b6800bb00 tx=0x7f5b680044c0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.651 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.649+0000 7f5b84a72740 1 -- 172.21.15.43:0/3511637505 shutdown_connections 2024-03-28T22:40:47.651 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.649+0000 7f5b84a72740 1 --2- 172.21.15.43:0/3511637505 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555a02726a90 0x555a02724570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.651 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.649+0000 7f5b84a72740 1 --2- 172.21.15.43:0/3511637505 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555a0271dcc0 0x555a02724030 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.651 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.649+0000 7f5b84a72740 1 --2- 172.21.15.43:0/3511637505 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x555a02736bf0 0x555a02738fe0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.651 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.649+0000 7f5b84a72740 1 --2- 172.21.15.43:0/3511637505 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555a0271d3a0 0x555a02723af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.651 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.649+0000 7f5b64c00640 0 client.26857 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:47.651 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.650+0000 7f5b84a72740 1 -- 172.21.15.43:0/3511637505 >> 172.21.15.43:0/3511637505 conn(0x555a02717cc0 msgr2=0x555a02719a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:47.651 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.650+0000 7f5b84a72740 1 -- 172.21.15.43:0/3511637505 shutdown_connections 2024-03-28T22:40:47.651 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.650+0000 7f5b84a72740 1 -- 172.21.15.43:0/3511637505 wait complete. 2024-03-28T22:40:47.651 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.650+0000 7f5b84a72740 20 client.26857 trim_cache size 0 max 16384 2024-03-28T22:40:47.666 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.666+0000 7f5b84a72740 1 Processor -- start 2024-03-28T22:40:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.666+0000 7f5b84a72740 1 -- start start 2024-03-28T22:40:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.667+0000 7f5b84a72740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555a0271d3a0 0x555a02726220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.667+0000 7f5b84a72740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555a0271dcc0 0x555a02732490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.667+0000 7f5b84a72740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555a02726a90 0x555a0273e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.667+0000 7f5b84a72740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7c1b0da0 con 0x555a02726a90 2024-03-28T22:40:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.667+0000 7f5b84a72740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7c1b28e0 con 0x555a0271dcc0 2024-03-28T22:40:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.667+0000 7f5b84a72740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7c1b1470 con 0x555a0271d3a0 2024-03-28T22:40:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.667+0000 7f5b7a000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555a0271dcc0 0x555a02732490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.667+0000 7f5b7a000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555a0271dcc0 0x555a02732490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59312/0 (socket says 172.21.15.43:59312) 2024-03-28T22:40:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.667+0000 7f5b7b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555a0271d3a0 0x555a02726220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.667+0000 7f5b7a000640 1 -- 172.21.15.43:0/281044717 learned_addr learned my addr 172.21.15.43:0/281044717 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.667+0000 7f5b7b400640 1 --2- 172.21.15.43:0/281044717 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555a02726a90 0x555a0273e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.667+0000 7f5b7a000640 1 -- 172.21.15.43:0/281044717 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555a0271d3a0 msgr2=0x555a02726220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.667+0000 7f5b7a000640 1 --2- 172.21.15.43:0/281044717 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555a0271d3a0 0x555a02726220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.667+0000 7f5b7a000640 1 -- 172.21.15.43:0/281044717 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555a02726a90 msgr2=0x555a0273e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.667+0000 7f5b7a000640 1 --2- 172.21.15.43:0/281044717 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555a02726a90 0x555a0273e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.667+0000 7f5b7a000640 1 -- 172.21.15.43:0/281044717 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a027291f0 con 0x555a0271dcc0 2024-03-28T22:40:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.667+0000 7f5b7b400640 1 --2- 172.21.15.43:0/281044717 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555a02726a90 0x555a0273e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:40:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.668+0000 7f5b7a000640 1 --2- 172.21.15.43:0/281044717 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555a0271dcc0 0x555a02732490 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f5b68010e70 tx=0x7f5b68004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26acd6d79fe70bc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.668+0000 7f5b67400640 1 -- 172.21.15.43:0/281044717 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b68011440 con 0x555a0271dcc0 2024-03-28T22:40:47.669 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.668+0000 7f5b67400640 1 -- 172.21.15.43:0/281044717 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b6802e070 con 0x555a0271dcc0 2024-03-28T22:40:47.669 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.668+0000 7f5b67400640 1 -- 172.21.15.43:0/281044717 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b6801a9e0 con 0x555a0271dcc0 2024-03-28T22:40:47.673 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.669+0000 7f5b84a72740 1 -- 172.21.15.43:0/281044717 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555a0271dcc0 msgr2=0x555a02732490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:47.673 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.669+0000 7f5b84a72740 1 --2- 172.21.15.43:0/281044717 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555a0271dcc0 0x555a02732490 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f5b68010e70 tx=0x7f5b68004360 comp rx=0 tx=0).stop 2024-03-28T22:40:47.673 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.669+0000 7f5b84a72740 1 -- 172.21.15.43:0/281044717 shutdown_connections 2024-03-28T22:40:47.673 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.669+0000 7f5b84a72740 1 --2- 172.21.15.43:0/281044717 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555a0271dcc0 0x555a02732490 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.673 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.669+0000 7f5b84a72740 1 --2- 172.21.15.43:0/281044717 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555a02726a90 0x555a0273e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.673 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.669+0000 7f5b84a72740 1 --2- 172.21.15.43:0/281044717 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555a0271d3a0 0x555a02726220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.673 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.669+0000 7f5b84a72740 1 -- 172.21.15.43:0/281044717 >> 172.21.15.43:0/281044717 conn(0x555a02717cc0 msgr2=0x555a02716c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:47.673 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.669+0000 7f5b84a72740 1 -- 172.21.15.43:0/281044717 shutdown_connections 2024-03-28T22:40:47.673 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.669+0000 7f5b84a72740 1 -- 172.21.15.43:0/281044717 wait complete. 2024-03-28T22:40:47.673 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.669+0000 7f5b84a72740 1 -- start start 2024-03-28T22:40:47.673 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.669+0000 7f5b84a72740 1 Processor -- start 2024-03-28T22:40:47.673 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.670+0000 7f5b84a72740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555a0271d3a0 0x555a02720f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:47.673 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.670+0000 7f5b84a72740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555a0271dcc0 0x555a027214a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:47.673 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.670+0000 7f5b84a72740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555a02726a90 0x555a02795ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.670+0000 7f5b84a72740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7c1b2210 con 0x555a0271d3a0 2024-03-28T22:40:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.670+0000 7f5b84a72740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7c1b1b40 con 0x555a02726a90 2024-03-28T22:40:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.670+0000 7f5b84a72740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7c1aed80 con 0x555a0271dcc0 2024-03-28T22:40:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.670+0000 7f5b84a72740 10 client.0 _ll_register_callbacks cb 0x7f5b68004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.670+0000 7f5b7a000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555a0271dcc0 0x555a027214a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.670+0000 7f5b7a000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555a0271dcc0 0x555a027214a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:49784/0 (socket says 172.21.15.43:49784) 2024-03-28T22:40:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.670+0000 7f5b7a000640 1 -- 172.21.15.43:0/2660289545 learned_addr learned my addr 172.21.15.43:0/2660289545 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.670+0000 7f5b7b400640 1 --2- 172.21.15.43:0/2660289545 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555a0271d3a0 0x555a02720f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.670+0000 7f5b7b400640 1 --2- 172.21.15.43:0/2660289545 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555a02726a90 0x555a02795ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.671+0000 7f5b7a000640 1 -- 172.21.15.43:0/2660289545 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555a02726a90 msgr2=0x555a02795ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.671+0000 7f5b7a000640 1 --2- 172.21.15.43:0/2660289545 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555a02726a90 0x555a02795ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.671+0000 7f5b7a000640 1 -- 172.21.15.43:0/2660289545 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555a0271d3a0 msgr2=0x555a02720f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.671+0000 7f5b7a000640 1 --2- 172.21.15.43:0/2660289545 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555a0271d3a0 0x555a02720f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.671+0000 7f5b7a000640 1 -- 172.21.15.43:0/2660289545 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a02796560 con 0x555a0271dcc0 2024-03-28T22:40:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.671+0000 7f5b7a000640 1 --2- 172.21.15.43:0/2660289545 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555a0271dcc0 0x555a027214a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5b6802b040 tx=0x7f5b6800b690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bfb03f96197f36d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.671+0000 7f5b64c00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:40:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.671+0000 7f5b64c00640 1 -- 172.21.15.43:0/2660289545 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b6801a590 con 0x555a0271dcc0 2024-03-28T22:40:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.671+0000 7f5b84a72740 1 -- 172.21.15.43:0/2660289545 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b6802e070 con 0x555a0271dcc0 2024-03-28T22:40:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.671+0000 7f5b64c00640 1 -- 172.21.15.43:0/2660289545 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f5b68011440 con 0x555a0271dcc0 2024-03-28T22:40:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.672+0000 7f5b64c00640 1 -- 172.21.15.43:0/2660289545 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5b6801ae60 con 0x555a0271dcc0 2024-03-28T22:40:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.672+0000 7f5b84a72740 10 client.26863 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.672+0000 7f5b84a72740 1 -- 172.21.15.43:0/2660289545 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x555a0279d8b0 con 0x555a0271dcc0 2024-03-28T22:40:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.672+0000 7f5b64c00640 1 -- 172.21.15.43:0/2660289545 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b6802e070 con 0x555a0271dcc0 2024-03-28T22:40:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.672+0000 7f5b64c00640 1 -- 172.21.15.43:0/2660289545 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f5b68029450 con 0x555a0271dcc0 2024-03-28T22:40:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.672+0000 7f5b84a72740 10 client.26863 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.672+0000 7f5b84a72740 10 client.26863 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:40:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.672+0000 7f5b84a72740 20 client.26863 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.672+0000 7f5b84a72740 1 --2- 172.21.15.43:0/2660289545 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x555a0279dd30 0x555a027a0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.672+0000 7f5b84a72740 4 client.26863 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T22:40:47.676 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.672+0000 7f5b84a72740 1 -- 172.21.15.43:0/2660289545 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555a0268db80 con 0x555a0279dd30 2024-03-28T22:40:47.676 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.672+0000 7f5b7b400640 1 --2- 172.21.15.43:0/2660289545 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x555a0279dd30 0x555a027a0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:47.676 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.673+0000 7f5b7b400640 1 --2- 172.21.15.43:0/2660289545 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x555a0279dd30 0x555a027a0120 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe15f389adfbdaa8 server_cookie=dd0d8617d18a7e2c in_seq=0 out_seq=0 2024-03-28T22:40:47.676 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.673+0000 7f5b64c00640 10 client.26863 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:47.889 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.889+0000 7f5b64c00640 1 -- 172.21.15.43:0/2660289545 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555a0268db80 con 0x555a0279dd30 2024-03-28T22:40:47.889 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.889+0000 7f5b64c00640 10 client.26863 handle_command_reply: tid=0 2024-03-28T22:40:47.889 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.889+0000 7f5b84a72740 1 client.26863 shutdown 2024-03-28T22:40:47.889 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.889+0000 7f5b84a72740 1 -- 172.21.15.43:0/2660289545 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555a0271dcc0 msgr2=0x555a027214a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:47.889 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.889+0000 7f5b84a72740 1 --2- 172.21.15.43:0/2660289545 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555a0271dcc0 0x555a027214a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5b6802b040 tx=0x7f5b6800b690 comp rx=0 tx=0).stop 2024-03-28T22:40:47.889 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.890+0000 7f5b84a72740 1 -- 172.21.15.43:0/2660289545 shutdown_connections 2024-03-28T22:40:47.889 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.890+0000 7f5b84a72740 1 --2- 172.21.15.43:0/2660289545 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x555a0279dd30 0x555a027a0120 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.889 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.890+0000 7f5b84a72740 1 --2- 172.21.15.43:0/2660289545 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555a02726a90 0x555a02795ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.889 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.890+0000 7f5b84a72740 1 --2- 172.21.15.43:0/2660289545 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555a0271dcc0 0x555a027214a0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.890+0000 7f5b64c00640 0 client.26863 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.890+0000 7f5b84a72740 1 --2- 172.21.15.43:0/2660289545 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555a0271d3a0 0x555a02720f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.890+0000 7f5b84a72740 1 -- 172.21.15.43:0/2660289545 >> 172.21.15.43:0/2660289545 conn(0x555a02717cc0 msgr2=0x555a02723ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.890+0000 7f5b84a72740 1 -- 172.21.15.43:0/2660289545 shutdown_connections 2024-03-28T22:40:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.890+0000 7f5b84a72740 1 -- 172.21.15.43:0/2660289545 wait complete. 2024-03-28T22:40:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.890+0000 7f5b84a72740 20 client.26863 trim_cache size 0 max 16384 2024-03-28T22:40:47.891 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:{ 2024-03-28T22:40:47.891 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "message": "", 2024-03-28T22:40:47.891 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "return_code": 0 2024-03-28T22:40:47.891 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:} 2024-03-28T22:40:47.892 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.891+0000 7f5b81e00640 1 -- 172.21.15.43:0/3908441195 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f5b600a2010 msgr2=0x7f5b600a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:47.892 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.891+0000 7f5b81e00640 1 --2- 172.21.15.43:0/3908441195 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f5b600a2010 0x7f5b600a44d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x555a0271ce50 tx=0x7f5b6c0dc040 comp rx=0 tx=0).stop 2024-03-28T22:40:47.892 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.891+0000 7f5b81e00640 1 -- 172.21.15.43:0/3908441195 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5b7c103920 msgr2=0x7f5b7c19a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:47.892 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.891+0000 7f5b81e00640 1 --2- 172.21.15.43:0/3908441195 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5b7c103920 0x7f5b7c19a210 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c02efd0 tx=0x7f5b6c004430 comp rx=0 tx=0).stop 2024-03-28T22:40:47.892 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.891+0000 7f5b81e00640 1 -- 172.21.15.43:0/3908441195 shutdown_connections 2024-03-28T22:40:47.892 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.891+0000 7f5b81e00640 1 --2- 172.21.15.43:0/3908441195 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f5b600a2010 0x7f5b600a44d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.892 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.891+0000 7f5b81e00640 1 --2- 172.21.15.43:0/3908441195 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5b7c109930 0x7f5b7c19ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.892 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.891+0000 7f5b81e00640 1 --2- 172.21.15.43:0/3908441195 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5b7c1042e0 0x7f5b7c19a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.892 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.891+0000 7f5b81e00640 1 --2- 172.21.15.43:0/3908441195 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5b7c103920 0x7f5b7c19a210 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:47.892 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.891+0000 7f5b81e00640 1 -- 172.21.15.43:0/3908441195 >> 172.21.15.43:0/3908441195 conn(0x7f5b7c0ff410 msgr2=0x7f5b7c0ff9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:47.892 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.892+0000 7f5b81e00640 1 -- 172.21.15.43:0/3908441195 shutdown_connections 2024-03-28T22:40:47.892 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:40:47.892+0000 7f5b81e00640 1 -- 172.21.15.43:0/3908441195 wait complete. 2024-03-28T22:40:48.355 INFO:tasks.workunit.client.0.smithi043.stdout: 1 213877 8.05 MB/sec execute 19 sec latency 39.644 ms 2024-03-28T22:40:49.355 INFO:tasks.workunit.client.0.smithi043.stdout: 1 216224 8.28 MB/sec execute 20 sec latency 10.928 ms 2024-03-28T22:40:49.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:49 smithi172 ceph-mon[36566]: pgmap v113: 129 pgs: 129 active+clean; 46 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 333 op/s 2024-03-28T22:40:49.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:49 smithi082 ceph-mon[36471]: pgmap v113: 129 pgs: 129 active+clean; 46 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 333 op/s 2024-03-28T22:40:49.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:49 smithi043 ceph-mon[31823]: pgmap v113: 129 pgs: 129 active+clean; 46 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 333 op/s 2024-03-28T22:40:50.355 INFO:tasks.workunit.client.0.smithi043.stdout: 1 217235 8.13 MB/sec execute 21 sec latency 22.219 ms 2024-03-28T22:40:51.355 INFO:tasks.workunit.client.0.smithi043.stdout: 1 219404 8.31 MB/sec execute 22 sec latency 10.560 ms 2024-03-28T22:40:51.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:51 smithi082 ceph-mon[36471]: pgmap v114: 129 pgs: 129 active+clean; 46 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 152 op/s 2024-03-28T22:40:51.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:51 smithi043 ceph-mon[31823]: pgmap v114: 129 pgs: 129 active+clean; 46 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 152 op/s 2024-03-28T22:40:51.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:51 smithi172 ceph-mon[36566]: pgmap v114: 129 pgs: 129 active+clean; 46 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 152 op/s 2024-03-28T22:40:52.355 INFO:tasks.workunit.client.0.smithi043.stdout: 1 220651 8.24 MB/sec execute 23 sec latency 12.701 ms 2024-03-28T22:40:53.355 INFO:tasks.workunit.client.0.smithi043.stdout: 1 222383 8.19 MB/sec execute 24 sec latency 77.926 ms 2024-03-28T22:40:53.468 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:53 smithi082 ceph-mon[36471]: pgmap v115: 129 pgs: 129 active+clean; 46 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 152 op/s 2024-03-28T22:40:53.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:53 smithi043 ceph-mon[31823]: pgmap v115: 129 pgs: 129 active+clean; 46 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 152 op/s 2024-03-28T22:40:53.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:53 smithi172 ceph-mon[36566]: pgmap v115: 129 pgs: 129 active+clean; 46 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 152 op/s 2024-03-28T22:40:54.355 INFO:tasks.workunit.client.0.smithi043.stdout: 1 223969 8.34 MB/sec execute 25 sec latency 10.772 ms 2024-03-28T22:40:54.375 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:40:54.375 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:40:54.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.631+0000 7f9062400640 1 -- 172.21.15.43:0/2890208778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f905c10de60 msgr2=0x7f905c110270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:54.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.631+0000 7f9062400640 1 --2- 172.21.15.43:0/2890208778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f905c10de60 0x7f905c110270 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f9054009be0 tx=0x7f905402bf80 comp rx=0 tx=0).stop 2024-03-28T22:40:54.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.631+0000 7f9062400640 1 -- 172.21.15.43:0/2890208778 shutdown_connections 2024-03-28T22:40:54.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.631+0000 7f9062400640 1 --2- 172.21.15.43:0/2890208778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f905c10de60 0x7f905c110270 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.631+0000 7f9062400640 1 --2- 172.21.15.43:0/2890208778 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f905c074440 0x7f905c10d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.631+0000 7f9062400640 1 --2- 172.21.15.43:0/2890208778 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f905c073b20 0x7f905c073f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.631+0000 7f9062400640 1 -- 172.21.15.43:0/2890208778 >> 172.21.15.43:0/2890208778 conn(0x7f905c0fd0b0 msgr2=0x7f905c0ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:54.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.631+0000 7f9062400640 1 -- 172.21.15.43:0/2890208778 shutdown_connections 2024-03-28T22:40:54.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.631+0000 7f9062400640 1 -- 172.21.15.43:0/2890208778 wait complete. 2024-03-28T22:40:54.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.632+0000 7f9062400640 1 Processor -- start 2024-03-28T22:40:54.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.632+0000 7f9062400640 1 -- start start 2024-03-28T22:40:54.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.633+0000 7f9062400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f905c073b20 0x7f905c19a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:54.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.633+0000 7f9062400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f905c074440 0x7f905c19a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:54.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.633+0000 7f9062400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f905c10de60 0x7f905c19e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:54.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.633+0000 7f9062400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f905c113c10 con 0x7f905c074440 2024-03-28T22:40:54.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.633+0000 7f9062400640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f905c113a90 con 0x7f905c073b20 2024-03-28T22:40:54.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.633+0000 7f9062400640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f905c113d90 con 0x7f905c10de60 2024-03-28T22:40:54.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.633+0000 7f9061000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f905c073b20 0x7f905c19a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:54.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.633+0000 7f9061000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f905c073b20 0x7f905c19a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59344/0 (socket says 172.21.15.43:59344) 2024-03-28T22:40:54.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.633+0000 7f9061000640 1 -- 172.21.15.43:0/3031037071 learned_addr learned my addr 172.21.15.43:0/3031037071 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:54.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.633+0000 7f9061a00640 1 --2- 172.21.15.43:0/3031037071 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f905c10de60 0x7f905c19e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:54.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.633+0000 7f905be00640 1 --2- 172.21.15.43:0/3031037071 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f905c074440 0x7f905c19a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:54.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.633+0000 7f9061000640 1 -- 172.21.15.43:0/3031037071 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f905c10de60 msgr2=0x7f905c19e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:54.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.633+0000 7f9061000640 1 --2- 172.21.15.43:0/3031037071 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f905c10de60 0x7f905c19e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.633+0000 7f9061000640 1 -- 172.21.15.43:0/3031037071 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f905c074440 msgr2=0x7f905c19a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:54.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.633+0000 7f9061000640 1 --2- 172.21.15.43:0/3031037071 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f905c074440 0x7f905c19a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.633+0000 7f9061000640 1 -- 172.21.15.43:0/3031037071 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f905c19f090 con 0x7f905c073b20 2024-03-28T22:40:54.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.634+0000 7f9061000640 1 --2- 172.21.15.43:0/3031037071 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f905c073b20 0x7f905c19a0f0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f904c00e9c0 tx=0x7f904c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:54.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.634+0000 7f9059600640 1 -- 172.21.15.43:0/3031037071 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904c00ccc0 con 0x7f905c073b20 2024-03-28T22:40:54.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.634+0000 7f9062400640 1 -- 172.21.15.43:0/3031037071 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f905c19f380 con 0x7f905c073b20 2024-03-28T22:40:54.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.634+0000 7f9059600640 1 -- 172.21.15.43:0/3031037071 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f904c005500 con 0x7f905c073b20 2024-03-28T22:40:54.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.634+0000 7f9059600640 1 -- 172.21.15.43:0/3031037071 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904c010430 con 0x7f905c073b20 2024-03-28T22:40:54.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.634+0000 7f9062400640 1 -- 172.21.15.43:0/3031037071 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f905c1a6c10 con 0x7f905c073b20 2024-03-28T22:40:54.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.637+0000 7f9066767740 1 Processor -- start 2024-03-28T22:40:54.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.638+0000 7f9066767740 1 -- start start 2024-03-28T22:40:54.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.638+0000 7f9059600640 1 -- 172.21.15.43:0/3031037071 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f904c0046c0 con 0x7f905c073b20 2024-03-28T22:40:54.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.638+0000 7f9066767740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b83732db0 0x560b83731b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:54.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.638+0000 7f9066767740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560b837320d0 0x560b83732550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:54.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.638+0000 7f9066767740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560b8373b440 0x560b8373d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:54.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.638+0000 7f9066767740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x560b83696100 con 0x560b83732db0 2024-03-28T22:40:54.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.638+0000 7f9066767740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x560b8363eba0 con 0x560b8373b440 2024-03-28T22:40:54.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.638+0000 7f9066767740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x560b836921a0 con 0x560b837320d0 2024-03-28T22:40:54.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.638+0000 7f9061000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560b837320d0 0x560b83732550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:54.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.638+0000 7f905be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560b8373b440 0x560b8373d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:54.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.638+0000 7f9061a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b83732db0 0x560b83731b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:54.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.638+0000 7f905be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560b8373b440 0x560b8373d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59354/0 (socket says 172.21.15.43:59354) 2024-03-28T22:40:54.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.638+0000 7f905be00640 1 -- 172.21.15.43:0/3043871391 learned_addr learned my addr 172.21.15.43:0/3043871391 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:54.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.639+0000 7f905be00640 1 -- 172.21.15.43:0/3043871391 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560b837320d0 msgr2=0x560b83732550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:54.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.639+0000 7f905be00640 1 --2- 172.21.15.43:0/3043871391 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560b837320d0 0x560b83732550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.639+0000 7f905be00640 1 -- 172.21.15.43:0/3043871391 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b83732db0 msgr2=0x560b83731b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:54.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.639+0000 7f905be00640 1 --2- 172.21.15.43:0/3043871391 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b83732db0 0x560b83731b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.639+0000 7f905be00640 1 -- 172.21.15.43:0/3043871391 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b8366f1a0 con 0x560b8373b440 2024-03-28T22:40:54.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.639+0000 7f905be00640 1 --2- 172.21.15.43:0/3043871391 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560b8373b440 0x560b8373d830 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f9044009c10 tx=0x7f9044011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9294cd0383ee6f69 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:54.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.639+0000 7f9052000640 1 -- 172.21.15.43:0/3043871391 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9044004230 con 0x560b8373b440 2024-03-28T22:40:54.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.639+0000 7f9059600640 1 --2- 172.21.15.43:0/3031037071 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f90400a1ef0 0x7f90400a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:54.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.639+0000 7f9052000640 1 -- 172.21.15.43:0/3043871391 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90440043d0 con 0x560b8373b440 2024-03-28T22:40:54.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.639+0000 7f9052000640 1 -- 172.21.15.43:0/3043871391 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9044004e00 con 0x560b8373b440 2024-03-28T22:40:54.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.640+0000 7f9059600640 1 -- 172.21.15.43:0/3031037071 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f904c0ce8b0 con 0x7f905c073b20 2024-03-28T22:40:54.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.640+0000 7f9061a00640 1 --2- 172.21.15.43:0/3031037071 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f90400a1ef0 0x7f90400a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:54.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.640+0000 7f9066767740 1 -- 172.21.15.43:0/3043871391 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560b8373b440 msgr2=0x560b8373d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:54.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.640+0000 7f9066767740 1 --2- 172.21.15.43:0/3043871391 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560b8373b440 0x560b8373d830 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f9044009c10 tx=0x7f9044011810 comp rx=0 tx=0).stop 2024-03-28T22:40:54.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.640+0000 7f9066767740 1 -- 172.21.15.43:0/3043871391 shutdown_connections 2024-03-28T22:40:54.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.640+0000 7f9066767740 1 --2- 172.21.15.43:0/3043871391 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560b8373b440 0x560b8373d830 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.640+0000 7f9066767740 1 --2- 172.21.15.43:0/3043871391 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560b837320d0 0x560b83732550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.640+0000 7f9066767740 1 --2- 172.21.15.43:0/3043871391 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b83732db0 0x560b83731b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.640+0000 7f9066767740 1 -- 172.21.15.43:0/3043871391 >> 172.21.15.43:0/3043871391 conn(0x560b836f8a10 msgr2=0x560b8372cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:54.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.640+0000 7f9066767740 1 -- 172.21.15.43:0/3043871391 shutdown_connections 2024-03-28T22:40:54.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.641+0000 7f9066767740 1 -- 172.21.15.43:0/3043871391 wait complete. 2024-03-28T22:40:54.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.641+0000 7f9061a00640 1 --2- 172.21.15.43:0/3031037071 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f90400a1ef0 0x7f90400a43b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f905402c4c0 tx=0x7f9054005fd0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:54.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.641+0000 7f9066767740 1 -- start start 2024-03-28T22:40:54.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.641+0000 7f9066767740 1 Processor -- start 2024-03-28T22:40:54.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.642+0000 7f9066767740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560b837320d0 0x560b83738380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:54.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.642+0000 7f9066767740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560b83732db0 0x560b837388c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:54.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.642+0000 7f9066767740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b8373b440 0x560b83738e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:54.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.642+0000 7f9066767740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x560b8373e2b0 con 0x560b8373b440 2024-03-28T22:40:54.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.642+0000 7f9066767740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x560b83634250 con 0x560b837320d0 2024-03-28T22:40:54.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.642+0000 7f9066767740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x560b8373e430 con 0x560b83732db0 2024-03-28T22:40:54.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.642+0000 7f9066767740 10 client.0 _ll_register_callbacks cb 0x560b833a9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:54.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.642+0000 7f9061000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560b837320d0 0x560b83738380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:54.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.642+0000 7f9061000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560b837320d0 0x560b83738380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59362/0 (socket says 172.21.15.43:59362) 2024-03-28T22:40:54.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.642+0000 7f9061000640 1 -- 172.21.15.43:0/2302008822 learned_addr learned my addr 172.21.15.43:0/2302008822 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:54.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.642+0000 7f905be00640 1 --2- 172.21.15.43:0/2302008822 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560b83732db0 0x560b837388c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:54.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.642+0000 7f9061a00640 1 --2- 172.21.15.43:0/2302008822 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b8373b440 0x560b83738e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:54.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.642+0000 7f9061000640 1 -- 172.21.15.43:0/2302008822 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560b83732db0 msgr2=0x560b837388c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:54.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.642+0000 7f9061000640 1 --2- 172.21.15.43:0/2302008822 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560b83732db0 0x560b837388c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.642+0000 7f9061000640 1 -- 172.21.15.43:0/2302008822 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b8373b440 msgr2=0x560b83738e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:54.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.642+0000 7f9061000640 1 --2- 172.21.15.43:0/2302008822 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b8373b440 0x560b83738e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.642+0000 7f9061000640 1 -- 172.21.15.43:0/2302008822 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b83650960 con 0x560b837320d0 2024-03-28T22:40:54.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.642+0000 7f9061a00640 1 --2- 172.21.15.43:0/2302008822 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b8373b440 0x560b83738e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:40:54.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.643+0000 7f9061000640 1 --2- 172.21.15.43:0/2302008822 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560b837320d0 0x560b83738380 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f904c000c00 tx=0x7f904c097200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66b3f81ac7fcf8c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:54.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.643+0000 7f904aa00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:40:54.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.643+0000 7f904aa00640 1 -- 172.21.15.43:0/2302008822 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904c0d4020 con 0x560b837320d0 2024-03-28T22:40:54.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.643+0000 7f904aa00640 1 -- 172.21.15.43:0/2302008822 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f904c0a31b0 con 0x560b837320d0 2024-03-28T22:40:54.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.643+0000 7f9066767740 1 -- 172.21.15.43:0/2302008822 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90440043d0 con 0x560b837320d0 2024-03-28T22:40:54.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.643+0000 7f904aa00640 1 -- 172.21.15.43:0/2302008822 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f904c09a800 con 0x560b837320d0 2024-03-28T22:40:54.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.643+0000 7f9066767740 10 client.26924 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:54.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.643+0000 7f9066767740 1 -- 172.21.15.43:0/2302008822 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560b834747d0 con 0x560b837320d0 2024-03-28T22:40:54.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.644+0000 7f904aa00640 1 -- 172.21.15.43:0/2302008822 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90440043d0 con 0x560b837320d0 2024-03-28T22:40:54.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.644+0000 7f904aa00640 1 -- 172.21.15.43:0/2302008822 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f904c09a2f0 con 0x560b837320d0 2024-03-28T22:40:54.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.644+0000 7f9066767740 10 client.26924 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:54.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.644+0000 7f9066767740 10 client.26924 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:40:54.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.644+0000 7f9066767740 20 client.26924 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:54.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.644+0000 7f9066767740 1 --2- 172.21.15.43:0/2302008822 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560b8374b770 0x560b8374dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:54.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.644+0000 7f9066767740 4 client.26924 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:40:54.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.644+0000 7f9066767740 1 -- 172.21.15.43:0/2302008822 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560b836a3640 con 0x560b8374b770 2024-03-28T22:40:54.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.644+0000 7f905be00640 1 --2- 172.21.15.43:0/2302008822 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560b8374b770 0x560b8374dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:54.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.645+0000 7f905be00640 1 --2- 172.21.15.43:0/2302008822 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560b8374b770 0x560b8374dbe0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf9def54d5231306 server_cookie=c10ee10b6b7904b2 in_seq=0 out_seq=0 2024-03-28T22:40:54.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.645+0000 7f904aa00640 10 client.26924 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:54.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.646+0000 7f904aa00640 1 -- 172.21.15.43:0/2302008822 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560b836a3640 con 0x560b8374b770 2024-03-28T22:40:54.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.646+0000 7f904aa00640 10 client.26924 handle_command_reply: tid=0 2024-03-28T22:40:54.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.646+0000 7f9066767740 1 client.26924 shutdown 2024-03-28T22:40:54.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.646+0000 7f9066767740 1 -- 172.21.15.43:0/2302008822 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560b837320d0 msgr2=0x560b83738380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:54.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.646+0000 7f9066767740 1 --2- 172.21.15.43:0/2302008822 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560b837320d0 0x560b83738380 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f904c000c00 tx=0x7f904c097200 comp rx=0 tx=0).stop 2024-03-28T22:40:54.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.647+0000 7f9066767740 1 -- 172.21.15.43:0/2302008822 shutdown_connections 2024-03-28T22:40:54.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.647+0000 7f9066767740 1 --2- 172.21.15.43:0/2302008822 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b8373b440 0x560b83738e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.647+0000 7f9066767740 1 --2- 172.21.15.43:0/2302008822 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560b83732db0 0x560b837388c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.647+0000 7f9066767740 1 --2- 172.21.15.43:0/2302008822 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560b8374b770 0x560b8374dbe0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.647+0000 7f9066767740 1 --2- 172.21.15.43:0/2302008822 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560b837320d0 0x560b83738380 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.647+0000 7f9066767740 1 -- 172.21.15.43:0/2302008822 >> 172.21.15.43:0/2302008822 conn(0x560b836f8a10 msgr2=0x560b8372e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:54.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.647+0000 7f904aa00640 0 client.26924 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:54.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.647+0000 7f9066767740 1 -- 172.21.15.43:0/2302008822 shutdown_connections 2024-03-28T22:40:54.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.647+0000 7f9066767740 1 -- 172.21.15.43:0/2302008822 wait complete. 2024-03-28T22:40:54.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.647+0000 7f9066767740 20 client.26924 trim_cache size 0 max 16384 2024-03-28T22:40:54.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.668+0000 7f9066767740 1 Processor -- start 2024-03-28T22:40:54.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.668+0000 7f9066767740 1 -- start start 2024-03-28T22:40:54.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.669+0000 7f9066767740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b837320d0 0x560b8373aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:54.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.669+0000 7f9066767740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560b83732db0 0x560b83747090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:54.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.669+0000 7f9066767740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560b8373b440 0x560b8372e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:54.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.669+0000 7f9066767740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f905c1b0c90 con 0x560b837320d0 2024-03-28T22:40:54.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.669+0000 7f9066767740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f905c1b27d0 con 0x560b8373b440 2024-03-28T22:40:54.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.669+0000 7f9066767740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f905c1b1360 con 0x560b83732db0 2024-03-28T22:40:54.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.669+0000 7f905be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560b83732db0 0x560b83747090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:54.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.669+0000 7f905be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560b83732db0 0x560b83747090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:49826/0 (socket says 172.21.15.43:49826) 2024-03-28T22:40:54.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.669+0000 7f905be00640 1 -- 172.21.15.43:0/2811139978 learned_addr learned my addr 172.21.15.43:0/2811139978 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:54.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.669+0000 7f9061000640 1 --2- 172.21.15.43:0/2811139978 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b837320d0 0x560b8373aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:54.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.669+0000 7f9061a00640 1 --2- 172.21.15.43:0/2811139978 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560b8373b440 0x560b8372e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:54.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.669+0000 7f905be00640 1 -- 172.21.15.43:0/2811139978 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560b8373b440 msgr2=0x560b8372e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:54.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.669+0000 7f905be00640 1 --2- 172.21.15.43:0/2811139978 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560b8373b440 0x560b8372e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.669+0000 7f905be00640 1 -- 172.21.15.43:0/2811139978 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b837320d0 msgr2=0x560b8373aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:54.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.669+0000 7f905be00640 1 --2- 172.21.15.43:0/2811139978 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b837320d0 0x560b8373aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.669+0000 7f905be00640 1 -- 172.21.15.43:0/2811139978 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b833aa3f0 con 0x560b83732db0 2024-03-28T22:40:54.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.669+0000 7f9061a00640 1 --2- 172.21.15.43:0/2811139978 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560b8373b440 0x560b8372e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:40:54.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.670+0000 7f905be00640 1 --2- 172.21.15.43:0/2811139978 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560b83732db0 0x560b83747090 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f9044009730 tx=0x7f9044004920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b09142db5f366841 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:54.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.670+0000 7f9051600640 1 -- 172.21.15.43:0/2811139978 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904401e040 con 0x560b83732db0 2024-03-28T22:40:54.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.670+0000 7f9051600640 1 -- 172.21.15.43:0/2811139978 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f904401f040 con 0x560b83732db0 2024-03-28T22:40:54.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.670+0000 7f9051600640 1 -- 172.21.15.43:0/2811139978 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9044003850 con 0x560b83732db0 2024-03-28T22:40:54.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.671+0000 7f9066767740 1 -- 172.21.15.43:0/2811139978 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560b83732db0 msgr2=0x560b83747090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:54.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.671+0000 7f9066767740 1 --2- 172.21.15.43:0/2811139978 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560b83732db0 0x560b83747090 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f9044009730 tx=0x7f9044004920 comp rx=0 tx=0).stop 2024-03-28T22:40:54.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.671+0000 7f9066767740 1 -- 172.21.15.43:0/2811139978 shutdown_connections 2024-03-28T22:40:54.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.671+0000 7f9066767740 1 --2- 172.21.15.43:0/2811139978 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560b8373b440 0x560b8372e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.671+0000 7f9066767740 1 --2- 172.21.15.43:0/2811139978 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560b83732db0 0x560b83747090 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.671+0000 7f9066767740 1 --2- 172.21.15.43:0/2811139978 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b837320d0 0x560b8373aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.671+0000 7f9066767740 1 -- 172.21.15.43:0/2811139978 >> 172.21.15.43:0/2811139978 conn(0x560b836f8a10 msgr2=0x560b8372df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:54.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.671+0000 7f9066767740 1 -- 172.21.15.43:0/2811139978 shutdown_connections 2024-03-28T22:40:54.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.671+0000 7f9066767740 1 -- 172.21.15.43:0/2811139978 wait complete. 2024-03-28T22:40:54.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.672+0000 7f9066767740 1 -- start start 2024-03-28T22:40:54.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.672+0000 7f9066767740 1 Processor -- start 2024-03-28T22:40:54.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.672+0000 7f9066767740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560b837320d0 0x560b8373f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:54.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.672+0000 7f9066767740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b83732db0 0x560b8373f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:54.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.672+0000 7f9066767740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560b8373b440 0x560b8373fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:54.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.672+0000 7f9066767740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f905c1b2100 con 0x560b83732db0 2024-03-28T22:40:54.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.672+0000 7f9066767740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f905c1b1a30 con 0x560b8373b440 2024-03-28T22:40:54.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.672+0000 7f9066767740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f905c1aec70 con 0x560b837320d0 2024-03-28T22:40:54.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.672+0000 7f9066767740 10 client.0 _ll_register_callbacks cb 0x7f904c0d4220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:54.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.673+0000 7f9061000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560b837320d0 0x560b8373f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:54.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.673+0000 7f9061000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560b837320d0 0x560b8373f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:49838/0 (socket says 172.21.15.43:49838) 2024-03-28T22:40:54.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.673+0000 7f9061000640 1 -- 172.21.15.43:0/646001684 learned_addr learned my addr 172.21.15.43:0/646001684 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:54.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.673+0000 7f9061a00640 1 --2- 172.21.15.43:0/646001684 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560b8373b440 0x560b8373fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:54.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.673+0000 7f905be00640 1 --2- 172.21.15.43:0/646001684 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b83732db0 0x560b8373f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:54.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.673+0000 7f9061000640 1 -- 172.21.15.43:0/646001684 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560b8373b440 msgr2=0x560b8373fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:54.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.673+0000 7f9061000640 1 --2- 172.21.15.43:0/646001684 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560b8373b440 0x560b8373fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.673+0000 7f9061000640 1 -- 172.21.15.43:0/646001684 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b83732db0 msgr2=0x560b8373f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:54.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.673+0000 7f9061000640 1 --2- 172.21.15.43:0/646001684 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b83732db0 0x560b8373f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.673+0000 7f9061000640 1 -- 172.21.15.43:0/646001684 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b8336a0f0 con 0x560b837320d0 2024-03-28T22:40:54.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.673+0000 7f905be00640 1 --2- 172.21.15.43:0/646001684 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b83732db0 0x560b8373f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:40:54.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.673+0000 7f9061000640 1 --2- 172.21.15.43:0/646001684 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560b837320d0 0x560b8373f000 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f904c097820 tx=0x7f904c0a3460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bbe70f3d8dbdd6b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:54.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.673+0000 7f904aa00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:40:54.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.673+0000 7f904aa00640 1 -- 172.21.15.43:0/646001684 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904c0cc090 con 0x560b837320d0 2024-03-28T22:40:54.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.674+0000 7f9066767740 1 -- 172.21.15.43:0/646001684 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f904401f040 con 0x560b837320d0 2024-03-28T22:40:54.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.674+0000 7f904aa00640 1 -- 172.21.15.43:0/646001684 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f904c096e50 con 0x560b837320d0 2024-03-28T22:40:54.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.674+0000 7f904aa00640 1 -- 172.21.15.43:0/646001684 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f904c0056b0 con 0x560b837320d0 2024-03-28T22:40:54.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.674+0000 7f9066767740 10 client.26881 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:54.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.674+0000 7f9066767740 1 -- 172.21.15.43:0/646001684 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560b837b1c50 con 0x560b837320d0 2024-03-28T22:40:54.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.674+0000 7f904aa00640 1 -- 172.21.15.43:0/646001684 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f904401f040 con 0x560b837320d0 2024-03-28T22:40:54.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.674+0000 7f904aa00640 1 -- 172.21.15.43:0/646001684 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f904c0b3bd0 con 0x560b837320d0 2024-03-28T22:40:54.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.674+0000 7f9066767740 10 client.26881 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:54.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.674+0000 7f9066767740 10 client.26881 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:40:54.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.674+0000 7f9066767740 20 client.26881 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:54.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.674+0000 7f9066767740 1 --2- 172.21.15.43:0/646001684 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560b837b22b0 0x560b837b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:54.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.674+0000 7f9066767740 4 client.26881 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:40:54.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.674+0000 7f9066767740 1 -- 172.21.15.43:0/646001684 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560b8369d990 con 0x560b837b22b0 2024-03-28T22:40:54.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.675+0000 7f905be00640 1 --2- 172.21.15.43:0/646001684 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560b837b22b0 0x560b837b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:54.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.675+0000 7f905be00640 1 --2- 172.21.15.43:0/646001684 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560b837b22b0 0x560b837b46d0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=673ea9b4e8f65f78 server_cookie=9923b7518b241aaf in_seq=0 out_seq=0 2024-03-28T22:40:54.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.675+0000 7f904aa00640 10 client.26881 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:54.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.675+0000 7f904aa00640 1 -- 172.21.15.43:0/646001684 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560b8369d990 con 0x560b837b22b0 2024-03-28T22:40:54.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.675+0000 7f904aa00640 10 client.26881 handle_command_reply: tid=0 2024-03-28T22:40:54.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.675+0000 7f9066767740 1 client.26881 shutdown 2024-03-28T22:40:54.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.676+0000 7f9066767740 1 -- 172.21.15.43:0/646001684 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560b837320d0 msgr2=0x560b8373f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:54.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.676+0000 7f9066767740 1 --2- 172.21.15.43:0/646001684 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560b837320d0 0x560b8373f000 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f904c097820 tx=0x7f904c0a3460 comp rx=0 tx=0).stop 2024-03-28T22:40:54.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.676+0000 7f9066767740 1 -- 172.21.15.43:0/646001684 shutdown_connections 2024-03-28T22:40:54.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.676+0000 7f9066767740 1 --2- 172.21.15.43:0/646001684 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560b837b22b0 0x560b837b46d0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.676+0000 7f9066767740 1 --2- 172.21.15.43:0/646001684 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560b8373b440 0x560b8373fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.676+0000 7f9066767740 1 --2- 172.21.15.43:0/646001684 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560b83732db0 0x560b8373f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.676+0000 7f904aa00640 0 client.26881 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:54.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.676+0000 7f9066767740 1 --2- 172.21.15.43:0/646001684 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560b837320d0 0x560b8373f000 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.676+0000 7f9066767740 1 -- 172.21.15.43:0/646001684 >> 172.21.15.43:0/646001684 conn(0x560b836f8a10 msgr2=0x560b83738bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:54.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.676+0000 7f9066767740 1 -- 172.21.15.43:0/646001684 shutdown_connections 2024-03-28T22:40:54.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.676+0000 7f9066767740 1 -- 172.21.15.43:0/646001684 wait complete. 2024-03-28T22:40:54.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.676+0000 7f9066767740 20 client.26881 trim_cache size 0 max 16384 2024-03-28T22:40:54.678 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:40:54.679 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-03-28T22:40:54.679 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-03-28T22:40:54.679 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:40:54.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.677+0000 7f9062400640 1 -- 172.21.15.43:0/3031037071 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f90400a1ef0 msgr2=0x7f90400a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:54.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.677+0000 7f9062400640 1 --2- 172.21.15.43:0/3031037071 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f90400a1ef0 0x7f90400a43b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f905402c4c0 tx=0x7f9054005fd0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.677+0000 7f9062400640 1 -- 172.21.15.43:0/3031037071 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f905c073b20 msgr2=0x7f905c19a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:54.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.677+0000 7f9062400640 1 --2- 172.21.15.43:0/3031037071 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f905c073b20 0x7f905c19a0f0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f904c00e9c0 tx=0x7f904c00ee80 comp rx=0 tx=0).stop 2024-03-28T22:40:54.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.677+0000 7f9062400640 1 -- 172.21.15.43:0/3031037071 shutdown_connections 2024-03-28T22:40:54.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.677+0000 7f9062400640 1 --2- 172.21.15.43:0/3031037071 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f90400a1ef0 0x7f90400a43b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.677+0000 7f9062400640 1 --2- 172.21.15.43:0/3031037071 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f905c10de60 0x7f905c19e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.677+0000 7f9062400640 1 --2- 172.21.15.43:0/3031037071 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f905c074440 0x7f905c19a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.677+0000 7f9062400640 1 --2- 172.21.15.43:0/3031037071 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f905c073b20 0x7f905c19a0f0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.677+0000 7f9062400640 1 -- 172.21.15.43:0/3031037071 >> 172.21.15.43:0/3031037071 conn(0x7f905c0fd0b0 msgr2=0x7f905c0fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:54.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.677+0000 7f9062400640 1 -- 172.21.15.43:0/3031037071 shutdown_connections 2024-03-28T22:40:54.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.677+0000 7f9062400640 1 -- 172.21.15.43:0/3031037071 wait complete. 2024-03-28T22:40:54.694 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:40:54.694 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:40:54.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.919+0000 7f50eb200640 1 -- 172.21.15.43:0/2377513503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50ec10dbf0 msgr2=0x7f50ec10dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:54.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.919+0000 7f50eb200640 1 --2- 172.21.15.43:0/2377513503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50ec10dbf0 0x7f50ec10dfd0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f50dc009930 tx=0x7f50dc02bc30 comp rx=0 tx=0).stop 2024-03-28T22:40:54.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.919+0000 7f50eb200640 1 -- 172.21.15.43:0/2377513503 shutdown_connections 2024-03-28T22:40:54.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.919+0000 7f50eb200640 1 --2- 172.21.15.43:0/2377513503 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f50ec1084f0 0x7f50ec1149b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.919+0000 7f50eb200640 1 --2- 172.21.15.43:0/2377513503 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f50ec107b30 0x7f50ec107fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.919+0000 7f50eb200640 1 --2- 172.21.15.43:0/2377513503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50ec10dbf0 0x7f50ec10dfd0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.919+0000 7f50eb200640 1 -- 172.21.15.43:0/2377513503 >> 172.21.15.43:0/2377513503 conn(0x7f50ec077270 msgr2=0x7f50ec079690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:54.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.919+0000 7f50eb200640 1 -- 172.21.15.43:0/2377513503 shutdown_connections 2024-03-28T22:40:54.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.919+0000 7f50eb200640 1 -- 172.21.15.43:0/2377513503 wait complete. 2024-03-28T22:40:54.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.920+0000 7f50eb200640 1 Processor -- start 2024-03-28T22:40:54.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.921+0000 7f50eb200640 1 -- start start 2024-03-28T22:40:54.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.921+0000 7f50eb200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50ec107b30 0x7f50ec19e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:54.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.921+0000 7f50eb200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f50ec1084f0 0x7f50ec19e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:54.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.921+0000 7f50eb200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f50ec10dbf0 0x7f50ec1a2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:54.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.921+0000 7f50eb200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f50ec117f30 con 0x7f50ec1084f0 2024-03-28T22:40:54.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.921+0000 7f50eb200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f50ec117db0 con 0x7f50ec10dbf0 2024-03-28T22:40:54.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.921+0000 7f50eb200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f50ec1180b0 con 0x7f50ec107b30 2024-03-28T22:40:54.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.921+0000 7f50e9e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50ec107b30 0x7f50ec19e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:54.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.921+0000 7f50ea800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f50ec10dbf0 0x7f50ec1a2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:54.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.921+0000 7f50e9e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50ec107b30 0x7f50ec19e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58394/0 (socket says 172.21.15.43:58394) 2024-03-28T22:40:54.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.921+0000 7f50e9e00640 1 -- 172.21.15.43:0/1935535343 learned_addr learned my addr 172.21.15.43:0/1935535343 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:54.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.921+0000 7f50e9400640 1 --2- 172.21.15.43:0/1935535343 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f50ec1084f0 0x7f50ec19e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:54.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.922+0000 7f50e9e00640 1 -- 172.21.15.43:0/1935535343 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f50ec10dbf0 msgr2=0x7f50ec1a2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:54.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.922+0000 7f50e9e00640 1 --2- 172.21.15.43:0/1935535343 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f50ec10dbf0 0x7f50ec1a2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.922+0000 7f50e9e00640 1 -- 172.21.15.43:0/1935535343 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f50ec1084f0 msgr2=0x7f50ec19e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:54.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.922+0000 7f50e9e00640 1 --2- 172.21.15.43:0/1935535343 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f50ec1084f0 0x7f50ec19e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:54.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.922+0000 7f50e9e00640 1 -- 172.21.15.43:0/1935535343 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50ec1a3340 con 0x7f50ec107b30 2024-03-28T22:40:54.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.922+0000 7f50ea800640 1 --2- 172.21.15.43:0/1935535343 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f50ec10dbf0 0x7f50ec1a2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:54.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.922+0000 7f50e9e00640 1 --2- 172.21.15.43:0/1935535343 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50ec107b30 0x7f50ec19e350 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f50dc009230 tx=0x7f50dc02cba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:54.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.922+0000 7f50e2a00640 1 -- 172.21.15.43:0/1935535343 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50dc041070 con 0x7f50ec107b30 2024-03-28T22:40:54.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.922+0000 7f50e2a00640 1 -- 172.21.15.43:0/1935535343 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50dc004450 con 0x7f50ec107b30 2024-03-28T22:40:54.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.922+0000 7f50e2a00640 1 -- 172.21.15.43:0/1935535343 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50dc034440 con 0x7f50ec107b30 2024-03-28T22:40:54.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.922+0000 7f50eb200640 1 -- 172.21.15.43:0/1935535343 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50ec1a35d0 con 0x7f50ec107b30 2024-03-28T22:40:54.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.923+0000 7f50eb200640 1 -- 172.21.15.43:0/1935535343 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50ec1aaec0 con 0x7f50ec107b30 2024-03-28T22:40:54.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.924+0000 7f50eb200640 1 -- 172.21.15.43:0/1935535343 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50ec1091e0 con 0x7f50ec107b30 2024-03-28T22:40:54.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.926+0000 7f50e2a00640 1 -- 172.21.15.43:0/1935535343 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f50dc044070 con 0x7f50ec107b30 2024-03-28T22:40:54.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.927+0000 7f50e2a00640 1 --2- 172.21.15.43:0/1935535343 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f50d00a1ef0 0x7f50d00a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:54.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.927+0000 7f50e2a00640 1 -- 172.21.15.43:0/1935535343 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f50dc0f01a0 con 0x7f50ec107b30 2024-03-28T22:40:54.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.927+0000 7f50e9400640 1 --2- 172.21.15.43:0/1935535343 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f50d00a1ef0 0x7f50d00a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:54.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.928+0000 7f50e9400640 1 --2- 172.21.15.43:0/1935535343 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f50d00a1ef0 0x7f50d00a43b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f50ec10d250 tx=0x7f50d4005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:54.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:54.930+0000 7f50e2a00640 1 -- 172.21.15.43:0/1935535343 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f50dc0bc950 con 0x7f50ec107b30 2024-03-28T22:40:55.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.182+0000 7f50eb200640 1 -- 172.21.15.43:0/1935535343 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f50ec10ffe0 con 0x7f50ec107b30 2024-03-28T22:40:55.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.183+0000 7f50e2a00640 1 -- 172.21.15.43:0/1935535343 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11931 (secure 0 0 0) 0x7f50dc0c09f0 con 0x7f50ec107b30 2024-03-28T22:40:55.182 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:40:55.182 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:33:28.906396+0000","modified":"2024-03-28T22:40:08.672168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24572,"mds_1":24556,"mds_2":24535,"mds_3":24523,"mds_4":24518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26717":{"gid":26717,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/215779342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":215779342},{"type":"v1","addr":"172.21.15.82:6835","nonce":215779342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24535,24556,24518,24523,24572]},"id":1}]} 2024-03-28T22:40:55.183 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 32 2024-03-28T22:40:55.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.186+0000 7f50eb200640 1 -- 172.21.15.43:0/1935535343 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f50d00a1ef0 msgr2=0x7f50d00a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:55.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.186+0000 7f50eb200640 1 --2- 172.21.15.43:0/1935535343 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f50d00a1ef0 0x7f50d00a43b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f50ec10d250 tx=0x7f50d4005f90 comp rx=0 tx=0).stop 2024-03-28T22:40:55.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.186+0000 7f50eb200640 1 -- 172.21.15.43:0/1935535343 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50ec107b30 msgr2=0x7f50ec19e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:55.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.186+0000 7f50eb200640 1 --2- 172.21.15.43:0/1935535343 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50ec107b30 0x7f50ec19e350 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f50dc009230 tx=0x7f50dc02cba0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.186+0000 7f50eb200640 1 -- 172.21.15.43:0/1935535343 shutdown_connections 2024-03-28T22:40:55.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.186+0000 7f50eb200640 1 --2- 172.21.15.43:0/1935535343 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f50d00a1ef0 0x7f50d00a43b0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.186+0000 7f50eb200640 1 --2- 172.21.15.43:0/1935535343 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f50ec10dbf0 0x7f50ec1a2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.186+0000 7f50eb200640 1 --2- 172.21.15.43:0/1935535343 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f50ec1084f0 0x7f50ec19e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.186+0000 7f50eb200640 1 --2- 172.21.15.43:0/1935535343 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f50ec107b30 0x7f50ec19e350 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.186+0000 7f50eb200640 1 -- 172.21.15.43:0/1935535343 >> 172.21.15.43:0/1935535343 conn(0x7f50ec077270 msgr2=0x7f50ec078960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:55.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.186+0000 7f50eb200640 1 -- 172.21.15.43:0/1935535343 shutdown_connections 2024-03-28T22:40:55.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.186+0000 7f50eb200640 1 -- 172.21.15.43:0/1935535343 wait complete. 2024-03-28T22:40:55.203 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T22:40:55.355 INFO:tasks.workunit.client.0.smithi043.stdout: 1 225714 8.24 MB/sec execute 26 sec latency 32.816 ms 2024-03-28T22:40:55.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.426+0000 7f594f600640 1 -- 172.21.15.43:0/625705816 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f59481076f0 msgr2=0x7f5948107ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:55.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.426+0000 7f594f600640 1 --2- 172.21.15.43:0/625705816 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f59481076f0 0x7f5948107ad0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f593c009900 tx=0x7f593c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:40:55.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.427+0000 7f594f600640 1 -- 172.21.15.43:0/625705816 shutdown_connections 2024-03-28T22:40:55.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.427+0000 7f594f600640 1 --2- 172.21.15.43:0/625705816 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f59481020a0 0x7f594810e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.427+0000 7f594f600640 1 --2- 172.21.15.43:0/625705816 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59481016e0 0x7f5948101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.427+0000 7f594f600640 1 --2- 172.21.15.43:0/625705816 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f59481076f0 0x7f5948107ad0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.427+0000 7f594f600640 1 -- 172.21.15.43:0/625705816 >> 172.21.15.43:0/625705816 conn(0x7f5948078b50 msgr2=0x7f5948078f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:55.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.427+0000 7f594f600640 1 -- 172.21.15.43:0/625705816 shutdown_connections 2024-03-28T22:40:55.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.427+0000 7f594f600640 1 -- 172.21.15.43:0/625705816 wait complete. 2024-03-28T22:40:55.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.428+0000 7f594f600640 1 Processor -- start 2024-03-28T22:40:55.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.428+0000 7f594f600640 1 -- start start 2024-03-28T22:40:55.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.428+0000 7f594f600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f59481016e0 0x7f594819a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.428+0000 7f594f600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f59481020a0 0x7f594819a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.428+0000 7f594f600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59481076f0 0x7f594819eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.428+0000 7f594f600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5948111b00 con 0x7f59481020a0 2024-03-28T22:40:55.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.428+0000 7f594f600640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5948111980 con 0x7f59481016e0 2024-03-28T22:40:55.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.428+0000 7f594f600640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f5948111c80 con 0x7f59481076f0 2024-03-28T22:40:55.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.429+0000 7f594cc00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f59481016e0 0x7f594819a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.429+0000 7f594cc00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f59481016e0 0x7f594819a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59676/0 (socket says 172.21.15.43:59676) 2024-03-28T22:40:55.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.429+0000 7f594cc00640 1 -- 172.21.15.43:0/2355946300 learned_addr learned my addr 172.21.15.43:0/2355946300 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:55.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.429+0000 7f594d600640 1 --2- 172.21.15.43:0/2355946300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59481076f0 0x7f594819eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.429+0000 7f5947e00640 1 --2- 172.21.15.43:0/2355946300 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f59481020a0 0x7f594819a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.429+0000 7f594cc00640 1 -- 172.21.15.43:0/2355946300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59481076f0 msgr2=0x7f594819eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:55.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.429+0000 7f594cc00640 1 --2- 172.21.15.43:0/2355946300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59481076f0 0x7f594819eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.429+0000 7f594cc00640 1 -- 172.21.15.43:0/2355946300 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f59481020a0 msgr2=0x7f594819a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:55.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.429+0000 7f594cc00640 1 --2- 172.21.15.43:0/2355946300 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f59481020a0 0x7f594819a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.429+0000 7f594cc00640 1 -- 172.21.15.43:0/2355946300 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f594819f210 con 0x7f59481016e0 2024-03-28T22:40:55.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.429+0000 7f594d600640 1 --2- 172.21.15.43:0/2355946300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59481076f0 0x7f594819eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:55.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.429+0000 7f594cc00640 1 --2- 172.21.15.43:0/2355946300 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f59481016e0 0x7f594819a220 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f593c005940 tx=0x7f593c0042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:55.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.430+0000 7f5945600640 1 -- 172.21.15.43:0/2355946300 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f593c041070 con 0x7f59481016e0 2024-03-28T22:40:55.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.430+0000 7f594f600640 1 -- 172.21.15.43:0/2355946300 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f594819f4a0 con 0x7f59481016e0 2024-03-28T22:40:55.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.430+0000 7f5945600640 1 -- 172.21.15.43:0/2355946300 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f593c02ccc0 con 0x7f59481016e0 2024-03-28T22:40:55.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.430+0000 7f5945600640 1 -- 172.21.15.43:0/2355946300 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f593c003d80 con 0x7f59481016e0 2024-03-28T22:40:55.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.430+0000 7f594f600640 1 -- 172.21.15.43:0/2355946300 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59481a6d90 con 0x7f59481016e0 2024-03-28T22:40:55.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.433+0000 7f5945600640 1 -- 172.21.15.43:0/2355946300 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f593c044070 con 0x7f59481016e0 2024-03-28T22:40:55.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.433+0000 7f59523a3740 1 Processor -- start 2024-03-28T22:40:55.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.434+0000 7f59523a3740 1 -- start start 2024-03-28T22:40:55.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.434+0000 7f59523a3740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ebc7e10380 0x55ebc7e10760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.434+0000 7f59523a3740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ebc7e10ca0 0x55ebc7e19340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.434+0000 7f59523a3740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ebc7e19880 0x55ebc7e1bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.434+0000 7f59523a3740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ebc7d74050 con 0x55ebc7e10ca0 2024-03-28T22:40:55.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.434+0000 7f59523a3740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55ebc7d1cce0 con 0x55ebc7e19880 2024-03-28T22:40:55.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.434+0000 7f59523a3740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55ebc7d700f0 con 0x55ebc7e10380 2024-03-28T22:40:55.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.434+0000 7f594d600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ebc7e10380 0x55ebc7e10760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.434+0000 7f594d600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ebc7e10380 0x55ebc7e10760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58428/0 (socket says 172.21.15.43:58428) 2024-03-28T22:40:55.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.434+0000 7f594d600640 1 -- 172.21.15.43:0/2707455742 learned_addr learned my addr 172.21.15.43:0/2707455742 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:55.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.434+0000 7f5947e00640 1 --2- 172.21.15.43:0/2707455742 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ebc7e19880 0x55ebc7e1bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.435+0000 7f594d600640 1 -- 172.21.15.43:0/2707455742 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ebc7e19880 msgr2=0x55ebc7e1bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:55.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.435+0000 7f594d600640 1 --2- 172.21.15.43:0/2707455742 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ebc7e19880 0x55ebc7e1bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.435+0000 7f594d600640 1 -- 172.21.15.43:0/2707455742 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ebc7e10ca0 msgr2=0x55ebc7e19340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:55.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.435+0000 7f594d600640 1 --2- 172.21.15.43:0/2707455742 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ebc7e10ca0 0x55ebc7e19340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.435+0000 7f594d600640 1 -- 172.21.15.43:0/2707455742 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebc7a883f0 con 0x55ebc7e10380 2024-03-28T22:40:55.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.435+0000 7f5947e00640 1 --2- 172.21.15.43:0/2707455742 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ebc7e19880 0x55ebc7e1bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:55.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.435+0000 7f594d600640 1 --2- 172.21.15.43:0/2707455742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ebc7e10380 0x55ebc7e10760 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f59400107a0 tx=0x7f5940010c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cff256d11e7d7134 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:55.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.435+0000 7f5945600640 1 --2- 172.21.15.43:0/2355946300 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f59300a1f70 0x7f59300a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.435+0000 7f5939c00640 1 -- 172.21.15.43:0/2707455742 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5940003b70 con 0x55ebc7e10380 2024-03-28T22:40:55.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.436+0000 7f5939c00640 1 -- 172.21.15.43:0/2707455742 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5940010e50 con 0x55ebc7e10380 2024-03-28T22:40:55.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.436+0000 7f5939c00640 1 -- 172.21.15.43:0/2707455742 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59400046d0 con 0x55ebc7e10380 2024-03-28T22:40:55.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.436+0000 7f594cc00640 1 --2- 172.21.15.43:0/2355946300 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f59300a1f70 0x7f59300a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.436+0000 7f5945600640 1 -- 172.21.15.43:0/2355946300 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f593c0f0220 con 0x7f59481016e0 2024-03-28T22:40:55.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.436+0000 7f59523a3740 1 -- 172.21.15.43:0/2707455742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ebc7e10380 msgr2=0x55ebc7e10760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:55.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.436+0000 7f59523a3740 1 --2- 172.21.15.43:0/2707455742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ebc7e10380 0x55ebc7e10760 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f59400107a0 tx=0x7f5940010c60 comp rx=0 tx=0).stop 2024-03-28T22:40:55.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.436+0000 7f594cc00640 1 --2- 172.21.15.43:0/2355946300 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f59300a1f70 0x7f59300a4430 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x55ebc7e0fe50 tx=0x7f593c02ce70 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:55.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.437+0000 7f59523a3740 1 -- 172.21.15.43:0/2707455742 shutdown_connections 2024-03-28T22:40:55.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.437+0000 7f59523a3740 1 --2- 172.21.15.43:0/2707455742 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ebc7e19880 0x55ebc7e1bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.437+0000 7f59523a3740 1 --2- 172.21.15.43:0/2707455742 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ebc7e10ca0 0x55ebc7e19340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.437+0000 7f59523a3740 1 --2- 172.21.15.43:0/2707455742 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ebc7e10380 0x55ebc7e10760 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.437+0000 7f59523a3740 1 -- 172.21.15.43:0/2707455742 >> 172.21.15.43:0/2707455742 conn(0x55ebc7e0aab0 msgr2=0x55ebc7e0ae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:55.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.437+0000 7f59523a3740 1 -- 172.21.15.43:0/2707455742 shutdown_connections 2024-03-28T22:40:55.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.437+0000 7f59523a3740 1 -- 172.21.15.43:0/2707455742 wait complete. 2024-03-28T22:40:55.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.437+0000 7f59523a3740 1 -- start start 2024-03-28T22:40:55.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.437+0000 7f59523a3740 1 Processor -- start 2024-03-28T22:40:55.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.438+0000 7f59523a3740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ebc7e10380 0x55ebc7e168b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.438+0000 7f59523a3740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ebc7e10ca0 0x55ebc7e16df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.438+0000 7f59523a3740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ebc7e19880 0x55ebc7e17330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.438+0000 7f59523a3740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ebc7e1c670 con 0x55ebc7e19880 2024-03-28T22:40:55.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.438+0000 7f59523a3740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55ebc7d122f0 con 0x55ebc7e10ca0 2024-03-28T22:40:55.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.438+0000 7f59523a3740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55ebc7e1c7f0 con 0x55ebc7e10380 2024-03-28T22:40:55.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.438+0000 7f59523a3740 10 client.0 _ll_register_callbacks cb 0x55ebc7a87be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:55.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.438+0000 7f594d600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ebc7e10380 0x55ebc7e168b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.438+0000 7f5947e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ebc7e10ca0 0x55ebc7e16df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.438+0000 7f5947e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ebc7e10ca0 0x55ebc7e16df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59684/0 (socket says 172.21.15.43:59684) 2024-03-28T22:40:55.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.438+0000 7f5947e00640 1 -- 172.21.15.43:0/3665001967 learned_addr learned my addr 172.21.15.43:0/3665001967 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:55.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.438+0000 7f594d600640 1 --2- 172.21.15.43:0/3665001967 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ebc7e19880 0x55ebc7e17330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.439+0000 7f5947e00640 1 -- 172.21.15.43:0/3665001967 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ebc7e10380 msgr2=0x55ebc7e168b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:55.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.439+0000 7f5947e00640 1 --2- 172.21.15.43:0/3665001967 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ebc7e10380 0x55ebc7e168b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.439+0000 7f5947e00640 1 -- 172.21.15.43:0/3665001967 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ebc7e19880 msgr2=0x55ebc7e17330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:55.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.439+0000 7f5947e00640 1 --2- 172.21.15.43:0/3665001967 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ebc7e19880 0x55ebc7e17330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.439+0000 7f5947e00640 1 -- 172.21.15.43:0/3665001967 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebc7b527d0 con 0x55ebc7e10ca0 2024-03-28T22:40:55.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.439+0000 7f594d600640 1 --2- 172.21.15.43:0/3665001967 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ebc7e19880 0x55ebc7e17330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:40:55.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.439+0000 7f5947e00640 1 --2- 172.21.15.43:0/3665001967 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ebc7e10ca0 0x55ebc7e16df0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f59340126e0 tx=0x7f5934012ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=347adca80a6e79b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:55.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.439+0000 7f592e000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:40:55.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.439+0000 7f592e000640 1 -- 172.21.15.43:0/3665001967 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5934004050 con 0x55ebc7e10ca0 2024-03-28T22:40:55.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.439+0000 7f59523a3740 1 -- 172.21.15.43:0/3665001967 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5940010e50 con 0x55ebc7e10ca0 2024-03-28T22:40:55.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.439+0000 7f592e000640 1 -- 172.21.15.43:0/3665001967 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f59340041f0 con 0x55ebc7e10ca0 2024-03-28T22:40:55.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.440+0000 7f592e000640 1 -- 172.21.15.43:0/3665001967 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5934025050 con 0x55ebc7e10ca0 2024-03-28T22:40:55.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.440+0000 7f59523a3740 10 client.26948 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:55.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.440+0000 7f59523a3740 1 -- 172.21.15.43:0/3665001967 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ebc7e28a50 con 0x55ebc7e10ca0 2024-03-28T22:40:55.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.440+0000 7f592e000640 1 -- 172.21.15.43:0/3665001967 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5940010e50 con 0x55ebc7e10ca0 2024-03-28T22:40:55.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.440+0000 7f592e000640 1 -- 172.21.15.43:0/3665001967 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f5934003af0 con 0x55ebc7e10ca0 2024-03-28T22:40:55.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.440+0000 7f59523a3740 10 client.26948 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:55.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.440+0000 7f59523a3740 10 client.26948 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:40:55.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.440+0000 7f59523a3740 20 client.26948 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:55.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.440+0000 7f59523a3740 1 --2- 172.21.15.43:0/3665001967 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55ebc7e29f00 0x55ebc7e2c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.440+0000 7f59523a3740 4 client.26948 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:40:55.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.440+0000 7f59523a3740 1 -- 172.21.15.43:0/3665001967 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ebc7d1d680 con 0x55ebc7e29f00 2024-03-28T22:40:55.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.441+0000 7f594d600640 1 --2- 172.21.15.43:0/3665001967 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55ebc7e29f00 0x55ebc7e2c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.441+0000 7f594d600640 1 --2- 172.21.15.43:0/3665001967 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55ebc7e29f00 0x55ebc7e2c2f0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=32b53894e4575162 server_cookie=e4d31b99ac2943b9 in_seq=0 out_seq=0 2024-03-28T22:40:55.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.441+0000 7f592e000640 10 client.26948 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:40:55.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.442+0000 7f592e000640 1 -- 172.21.15.43:0/3665001967 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ebc7d1d680 con 0x55ebc7e29f00 2024-03-28T22:40:55.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.442+0000 7f592e000640 10 client.26948 handle_command_reply: tid=0 2024-03-28T22:40:55.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.442+0000 7f59523a3740 1 client.26948 shutdown 2024-03-28T22:40:55.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.442+0000 7f59523a3740 1 -- 172.21.15.43:0/3665001967 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ebc7e10ca0 msgr2=0x55ebc7e16df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:55.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.442+0000 7f59523a3740 1 --2- 172.21.15.43:0/3665001967 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ebc7e10ca0 0x55ebc7e16df0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f59340126e0 tx=0x7f5934012ba0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.442+0000 7f59523a3740 1 -- 172.21.15.43:0/3665001967 shutdown_connections 2024-03-28T22:40:55.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.442+0000 7f59523a3740 1 --2- 172.21.15.43:0/3665001967 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55ebc7e29f00 0x55ebc7e2c2f0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.442+0000 7f59523a3740 1 --2- 172.21.15.43:0/3665001967 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ebc7e10ca0 0x55ebc7e16df0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.442+0000 7f59523a3740 1 --2- 172.21.15.43:0/3665001967 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ebc7e19880 0x55ebc7e17330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.442+0000 7f59523a3740 1 --2- 172.21.15.43:0/3665001967 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ebc7e10380 0x55ebc7e168b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.442+0000 7f592e000640 0 client.26948 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:40:55.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.442+0000 7f59523a3740 1 -- 172.21.15.43:0/3665001967 >> 172.21.15.43:0/3665001967 conn(0x55ebc7e0aab0 msgr2=0x55ebc7e0c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:55.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.442+0000 7f59523a3740 1 -- 172.21.15.43:0/3665001967 shutdown_connections 2024-03-28T22:40:55.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.442+0000 7f59523a3740 1 -- 172.21.15.43:0/3665001967 wait complete. 2024-03-28T22:40:55.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.443+0000 7f59523a3740 20 client.26948 trim_cache size 0 max 16384 2024-03-28T22:40:55.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:55 smithi082 ceph-mon[36471]: pgmap v116: 129 pgs: 129 active+clean; 42 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 347 op/s 2024-03-28T22:40:55.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.458+0000 7f59523a3740 1 Processor -- start 2024-03-28T22:40:55.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.458+0000 7f59523a3740 1 -- start start 2024-03-28T22:40:55.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.459+0000 7f59523a3740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ebc7e10380 0x55ebc7e18fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.459+0000 7f59523a3740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ebc7e10ca0 0x55ebc7e254a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.459+0000 7f59523a3740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ebc7e19880 0x55ebc7e31b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.459+0000 7f59523a3740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f59481b0db0 con 0x55ebc7e19880 2024-03-28T22:40:55.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.459+0000 7f59523a3740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f59481b28f0 con 0x55ebc7e10380 2024-03-28T22:40:55.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.459+0000 7f59523a3740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f59481b1480 con 0x55ebc7e10ca0 2024-03-28T22:40:55.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.459+0000 7f5947e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ebc7e10ca0 0x55ebc7e254a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.459+0000 7f5947e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ebc7e10ca0 0x55ebc7e254a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58434/0 (socket says 172.21.15.43:58434) 2024-03-28T22:40:55.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.459+0000 7f5947e00640 1 -- 172.21.15.43:0/210548421 learned_addr learned my addr 172.21.15.43:0/210548421 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:55.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.459+0000 7f594d600640 1 --2- 172.21.15.43:0/210548421 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ebc7e10380 0x55ebc7e18fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.459+0000 7f594d600640 1 --2- 172.21.15.43:0/210548421 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ebc7e19880 0x55ebc7e31b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.460+0000 7f5947e00640 1 -- 172.21.15.43:0/210548421 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ebc7e10380 msgr2=0x55ebc7e18fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:55.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.460+0000 7f5947e00640 1 --2- 172.21.15.43:0/210548421 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ebc7e10380 0x55ebc7e18fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.460+0000 7f5947e00640 1 -- 172.21.15.43:0/210548421 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ebc7e19880 msgr2=0x55ebc7e31b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:55.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.460+0000 7f5947e00640 1 --2- 172.21.15.43:0/210548421 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ebc7e19880 0x55ebc7e31b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.460+0000 7f5947e00640 1 -- 172.21.15.43:0/210548421 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebc7e2c660 con 0x55ebc7e10ca0 2024-03-28T22:40:55.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.460+0000 7f594d600640 1 --2- 172.21.15.43:0/210548421 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ebc7e10380 0x55ebc7e18fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:55.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.460+0000 7f5947e00640 1 --2- 172.21.15.43:0/210548421 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ebc7e10ca0 0x55ebc7e254a0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f593400a010 tx=0x7f5934009830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0b87b223c596469 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:55.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.461+0000 7f5939200640 1 -- 172.21.15.43:0/210548421 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f593402e070 con 0x55ebc7e10ca0 2024-03-28T22:40:55.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.461+0000 7f5939200640 1 -- 172.21.15.43:0/210548421 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5934010640 con 0x55ebc7e10ca0 2024-03-28T22:40:55.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.461+0000 7f5939200640 1 -- 172.21.15.43:0/210548421 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f593400f920 con 0x55ebc7e10ca0 2024-03-28T22:40:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.461+0000 7f59523a3740 1 -- 172.21.15.43:0/210548421 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ebc7e10ca0 msgr2=0x55ebc7e254a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.461+0000 7f59523a3740 1 --2- 172.21.15.43:0/210548421 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ebc7e10ca0 0x55ebc7e254a0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f593400a010 tx=0x7f5934009830 comp rx=0 tx=0).stop 2024-03-28T22:40:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.462+0000 7f59523a3740 1 -- 172.21.15.43:0/210548421 shutdown_connections 2024-03-28T22:40:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.462+0000 7f59523a3740 1 --2- 172.21.15.43:0/210548421 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ebc7e19880 0x55ebc7e31b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.462+0000 7f59523a3740 1 --2- 172.21.15.43:0/210548421 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ebc7e10ca0 0x55ebc7e254a0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.462+0000 7f59523a3740 1 --2- 172.21.15.43:0/210548421 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ebc7e10380 0x55ebc7e18fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.462+0000 7f59523a3740 1 -- 172.21.15.43:0/210548421 >> 172.21.15.43:0/210548421 conn(0x55ebc7e0aab0 msgr2=0x55ebc7e0c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.462+0000 7f59523a3740 1 -- 172.21.15.43:0/210548421 shutdown_connections 2024-03-28T22:40:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.462+0000 7f59523a3740 1 -- 172.21.15.43:0/210548421 wait complete. 2024-03-28T22:40:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.462+0000 7f59523a3740 1 -- start start 2024-03-28T22:40:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.462+0000 7f59523a3740 1 Processor -- start 2024-03-28T22:40:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.463+0000 7f59523a3740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ebc7e10380 0x55ebc7e13a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.463+0000 7f59523a3740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ebc7e10ca0 0x55ebc7e13f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.463+0000 7f59523a3740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ebc7e19880 0x55ebc7e8f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.463+0000 7f59523a3740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f59481b2220 con 0x55ebc7e19880 2024-03-28T22:40:55.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.463+0000 7f59523a3740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f59481b1b50 con 0x55ebc7e10380 2024-03-28T22:40:55.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.463+0000 7f59523a3740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f59481aed90 con 0x55ebc7e10ca0 2024-03-28T22:40:55.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.463+0000 7f59523a3740 10 client.0 _ll_register_callbacks cb 0x7f5934009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:55.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.463+0000 7f594d600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ebc7e10380 0x55ebc7e13a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.463+0000 7f5947e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ebc7e10ca0 0x55ebc7e13f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.463+0000 7f5947e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ebc7e10ca0 0x55ebc7e13f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58442/0 (socket says 172.21.15.43:58442) 2024-03-28T22:40:55.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.463+0000 7f5947e00640 1 -- 172.21.15.43:0/3670766584 learned_addr learned my addr 172.21.15.43:0/3670766584 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:55.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.463+0000 7f594d600640 1 --2- 172.21.15.43:0/3670766584 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ebc7e19880 0x55ebc7e8f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.464+0000 7f5947e00640 1 -- 172.21.15.43:0/3670766584 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ebc7e10380 msgr2=0x55ebc7e13a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:55.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.464+0000 7f5947e00640 1 --2- 172.21.15.43:0/3670766584 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ebc7e10380 0x55ebc7e13a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.464+0000 7f5947e00640 1 -- 172.21.15.43:0/3670766584 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ebc7e19880 msgr2=0x55ebc7e8f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:55.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.464+0000 7f5947e00640 1 --2- 172.21.15.43:0/3670766584 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ebc7e19880 0x55ebc7e8f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.464+0000 7f5947e00640 1 -- 172.21.15.43:0/3670766584 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebc7e8f6e0 con 0x55ebc7e10ca0 2024-03-28T22:40:55.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.464+0000 7f5947e00640 1 --2- 172.21.15.43:0/3670766584 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ebc7e10ca0 0x55ebc7e13f70 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f5934019040 tx=0x7f59340096c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24f2fcc78597d15e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:55.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.464+0000 7f592e000640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:40:55.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.464+0000 7f592e000640 1 -- 172.21.15.43:0/3670766584 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f593402e050 con 0x55ebc7e10ca0 2024-03-28T22:40:55.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.464+0000 7f592e000640 1 -- 172.21.15.43:0/3670766584 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f5934005640 con 0x55ebc7e10ca0 2024-03-28T22:40:55.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.464+0000 7f59523a3740 1 -- 172.21.15.43:0/3670766584 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5934010640 con 0x55ebc7e10ca0 2024-03-28T22:40:55.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.464+0000 7f592e000640 1 -- 172.21.15.43:0/3670766584 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f593403f050 con 0x55ebc7e10ca0 2024-03-28T22:40:55.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.465+0000 7f59523a3740 10 client.26905 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:55.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.465+0000 7f59523a3740 1 -- 172.21.15.43:0/3670766584 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ebc7e90880 con 0x55ebc7e10ca0 2024-03-28T22:40:55.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.465+0000 7f592e000640 1 -- 172.21.15.43:0/3670766584 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5934010640 con 0x55ebc7e10ca0 2024-03-28T22:40:55.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.465+0000 7f592e000640 1 -- 172.21.15.43:0/3670766584 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f593403a030 con 0x55ebc7e10ca0 2024-03-28T22:40:55.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.465+0000 7f59523a3740 10 client.26905 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:55.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.465+0000 7f59523a3740 10 client.26905 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:40:55.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.465+0000 7f59523a3740 20 client.26905 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:55.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.465+0000 7f59523a3740 1 --2- 172.21.15.43:0/3670766584 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55ebc7e90e80 0x55ebc7e932a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.465+0000 7f59523a3740 4 client.26905 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:40:55.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.465+0000 7f59523a3740 1 -- 172.21.15.43:0/3670766584 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ebc7d81590 con 0x55ebc7e90e80 2024-03-28T22:40:55.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.465+0000 7f594d600640 1 --2- 172.21.15.43:0/3670766584 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55ebc7e90e80 0x55ebc7e932a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.466+0000 7f594d600640 1 --2- 172.21.15.43:0/3670766584 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55ebc7e90e80 0x55ebc7e932a0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c43bd1cf55aaec9c server_cookie=9910ed4be854e485 in_seq=0 out_seq=0 2024-03-28T22:40:55.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.466+0000 7f592e000640 10 client.26905 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:40:55.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.466+0000 7f592e000640 1 -- 172.21.15.43:0/3670766584 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ebc7d81590 con 0x55ebc7e90e80 2024-03-28T22:40:55.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.466+0000 7f592e000640 10 client.26905 handle_command_reply: tid=0 2024-03-28T22:40:55.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.466+0000 7f59523a3740 1 client.26905 shutdown 2024-03-28T22:40:55.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.466+0000 7f59523a3740 1 -- 172.21.15.43:0/3670766584 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ebc7e10ca0 msgr2=0x55ebc7e13f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:55.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.466+0000 7f59523a3740 1 --2- 172.21.15.43:0/3670766584 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ebc7e10ca0 0x55ebc7e13f70 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f5934019040 tx=0x7f59340096c0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.466+0000 7f59523a3740 1 -- 172.21.15.43:0/3670766584 shutdown_connections 2024-03-28T22:40:55.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.466+0000 7f59523a3740 1 --2- 172.21.15.43:0/3670766584 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55ebc7e90e80 0x55ebc7e932a0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.466+0000 7f59523a3740 1 --2- 172.21.15.43:0/3670766584 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ebc7e19880 0x55ebc7e8f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.466+0000 7f59523a3740 1 --2- 172.21.15.43:0/3670766584 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ebc7e10ca0 0x55ebc7e13f70 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.466+0000 7f59523a3740 1 --2- 172.21.15.43:0/3670766584 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ebc7e10380 0x55ebc7e13a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.466+0000 7f59523a3740 1 -- 172.21.15.43:0/3670766584 >> 172.21.15.43:0/3670766584 conn(0x55ebc7e0aab0 msgr2=0x55ebc7e0c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:55.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.466+0000 7f592e000640 0 client.26905 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:40:55.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.467+0000 7f59523a3740 1 -- 172.21.15.43:0/3670766584 shutdown_connections 2024-03-28T22:40:55.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.467+0000 7f59523a3740 1 -- 172.21.15.43:0/3670766584 wait complete. 2024-03-28T22:40:55.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.467+0000 7f59523a3740 20 client.26905 trim_cache size 0 max 16384 2024-03-28T22:40:55.470 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:40:55.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.467+0000 7f594f600640 1 -- 172.21.15.43:0/2355946300 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f59300a1f70 msgr2=0x7f59300a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:55.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.467+0000 7f594f600640 1 --2- 172.21.15.43:0/2355946300 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f59300a1f70 0x7f59300a4430 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x55ebc7e0fe50 tx=0x7f593c02ce70 comp rx=0 tx=0).stop 2024-03-28T22:40:55.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.467+0000 7f594f600640 1 -- 172.21.15.43:0/2355946300 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f59481016e0 msgr2=0x7f594819a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:55.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.467+0000 7f594f600640 1 --2- 172.21.15.43:0/2355946300 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f59481016e0 0x7f594819a220 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f593c005940 tx=0x7f593c0042f0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.467+0000 7f594f600640 1 -- 172.21.15.43:0/2355946300 shutdown_connections 2024-03-28T22:40:55.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.467+0000 7f594f600640 1 --2- 172.21.15.43:0/2355946300 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f59300a1f70 0x7f59300a4430 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.468+0000 7f594f600640 1 --2- 172.21.15.43:0/2355946300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f59481076f0 0x7f594819eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.468+0000 7f594f600640 1 --2- 172.21.15.43:0/2355946300 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f59481020a0 0x7f594819a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.468+0000 7f594f600640 1 --2- 172.21.15.43:0/2355946300 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f59481016e0 0x7f594819a220 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.468+0000 7f594f600640 1 -- 172.21.15.43:0/2355946300 >> 172.21.15.43:0/2355946300 conn(0x7f5948078b50 msgr2=0x7f594810ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:55.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.468+0000 7f594f600640 1 -- 172.21.15.43:0/2355946300 shutdown_connections 2024-03-28T22:40:55.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.468+0000 7f594f600640 1 -- 172.21.15.43:0/2355946300 wait complete. 2024-03-28T22:40:55.481 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:55 smithi043 ceph-mon[31823]: pgmap v116: 129 pgs: 129 active+clean; 42 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 347 op/s 2024-03-28T22:40:55.484 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T22:40:55.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:55 smithi172 ceph-mon[36566]: pgmap v116: 129 pgs: 129 active+clean; 42 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 347 op/s 2024-03-28T22:40:55.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.704+0000 7f6f72800640 1 -- 172.21.15.43:0/4256776642 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f6c074420 msgr2=0x7f6f6c10d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:55.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.704+0000 7f6f72800640 1 --2- 172.21.15.43:0/4256776642 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f6c074420 0x7f6f6c10d940 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f6f58009900 tx=0x7f6f5802bcd0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.704+0000 7f6f72800640 1 -- 172.21.15.43:0/4256776642 shutdown_connections 2024-03-28T22:40:55.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.704+0000 7f6f72800640 1 --2- 172.21.15.43:0/4256776642 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f6c10de80 0x7f6f6c110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.704+0000 7f6f72800640 1 --2- 172.21.15.43:0/4256776642 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f6c074420 0x7f6f6c10d940 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.704+0000 7f6f72800640 1 --2- 172.21.15.43:0/4256776642 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f6c073b00 0x7f6f6c073ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.704+0000 7f6f72800640 1 -- 172.21.15.43:0/4256776642 >> 172.21.15.43:0/4256776642 conn(0x7f6f6c0fd0f0 msgr2=0x7f6f6c0ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:55.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.704+0000 7f6f72800640 1 -- 172.21.15.43:0/4256776642 shutdown_connections 2024-03-28T22:40:55.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.704+0000 7f6f72800640 1 -- 172.21.15.43:0/4256776642 wait complete. 2024-03-28T22:40:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.705+0000 7f6f72800640 1 Processor -- start 2024-03-28T22:40:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.705+0000 7f6f72800640 1 -- start start 2024-03-28T22:40:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.706+0000 7f6f72800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f6c073b00 0x7f6f6c19a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.706+0000 7f6f72800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f6c074420 0x7f6f6c19a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.706+0000 7f6f72800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f6c10de80 0x7f6f6c19e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.706+0000 7f6f72800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6c113ba0 con 0x7f6f6c074420 2024-03-28T22:40:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.706+0000 7f6f72800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6c113a20 con 0x7f6f6c073b00 2024-03-28T22:40:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.706+0000 7f6f72800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6c113d20 con 0x7f6f6c10de80 2024-03-28T22:40:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.706+0000 7f6f71400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f6c073b00 0x7f6f6c19a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.706+0000 7f6f70a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f6c074420 0x7f6f6c19a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.706+0000 7f6f71e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f6c10de80 0x7f6f6c19e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.706+0000 7f6f71400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f6c073b00 0x7f6f6c19a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59714/0 (socket says 172.21.15.43:59714) 2024-03-28T22:40:55.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.706+0000 7f6f71400640 1 -- 172.21.15.43:0/2245965183 learned_addr learned my addr 172.21.15.43:0/2245965183 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:55.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.706+0000 7f6f71e00640 1 --2- 172.21.15.43:0/2245965183 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f6c10de80 0x7f6f6c19e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58464/0 (socket says 172.21.15.43:58464) 2024-03-28T22:40:55.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.706+0000 7f6f71400640 1 -- 172.21.15.43:0/2245965183 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f6c10de80 msgr2=0x7f6f6c19e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:55.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.706+0000 7f6f71400640 1 --2- 172.21.15.43:0/2245965183 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f6c10de80 0x7f6f6c19e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.706+0000 7f6f71400640 1 -- 172.21.15.43:0/2245965183 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f6c074420 msgr2=0x7f6f6c19a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:55.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.706+0000 7f6f71400640 1 --2- 172.21.15.43:0/2245965183 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f6c074420 0x7f6f6c19a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.707+0000 7f6f71400640 1 -- 172.21.15.43:0/2245965183 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f6c19f0b0 con 0x7f6f6c073b00 2024-03-28T22:40:55.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.707+0000 7f6f70a00640 1 --2- 172.21.15.43:0/2245965183 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f6c074420 0x7f6f6c19a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:55.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.707+0000 7f6f71e00640 1 --2- 172.21.15.43:0/2245965183 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f6c10de80 0x7f6f6c19e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:55.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.707+0000 7f6f71400640 1 --2- 172.21.15.43:0/2245965183 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f6c073b00 0x7f6f6c19a0c0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f6f6000d770 tx=0x7f6f6000dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:55.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.707+0000 7f6f6a000640 1 -- 172.21.15.43:0/2245965183 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f60017070 con 0x7f6f6c073b00 2024-03-28T22:40:55.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.707+0000 7f6f72800640 1 -- 172.21.15.43:0/2245965183 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f6c19f3a0 con 0x7f6f6c073b00 2024-03-28T22:40:55.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.707+0000 7f6f6a000640 1 -- 172.21.15.43:0/2245965183 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f60005500 con 0x7f6f6c073b00 2024-03-28T22:40:55.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.707+0000 7f6f6a000640 1 -- 172.21.15.43:0/2245965183 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f60004e00 con 0x7f6f6c073b00 2024-03-28T22:40:55.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.707+0000 7f6f72800640 1 -- 172.21.15.43:0/2245965183 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f6c1a6c30 con 0x7f6f6c073b00 2024-03-28T22:40:55.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.711+0000 7f6f76c31740 1 Processor -- start 2024-03-28T22:40:55.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.712+0000 7f6f76c31740 1 -- start start 2024-03-28T22:40:55.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.712+0000 7f6f6a000640 1 -- 172.21.15.43:0/2245965183 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f6f60004fa0 con 0x7f6f6c073b00 2024-03-28T22:40:55.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.712+0000 7f6f76c31740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55caf730b550 0x55caf730b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.712+0000 7f6f76c31740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55caf730be70 0x55caf7314510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.712+0000 7f6f76c31740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55caf7314a50 0x55caf7316e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.712+0000 7f6f76c31740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55caf726f050 con 0x55caf730b550 2024-03-28T22:40:55.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.712+0000 7f6f76c31740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55caf7217ce0 con 0x55caf7314a50 2024-03-28T22:40:55.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.712+0000 7f6f76c31740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55caf726b0f0 con 0x55caf730be70 2024-03-28T22:40:55.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.712+0000 7f6f71400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55caf730be70 0x55caf7314510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.712+0000 7f6f70a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55caf7314a50 0x55caf7316e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.712+0000 7f6f71400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55caf730be70 0x55caf7314510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58474/0 (socket says 172.21.15.43:58474) 2024-03-28T22:40:55.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.712+0000 7f6f71400640 1 -- 172.21.15.43:0/1996278643 learned_addr learned my addr 172.21.15.43:0/1996278643 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:55.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.713+0000 7f6f71e00640 1 --2- 172.21.15.43:0/1996278643 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55caf730b550 0x55caf730b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.713+0000 7f6f71400640 1 -- 172.21.15.43:0/1996278643 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55caf7314a50 msgr2=0x55caf7316e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:55.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.713+0000 7f6f71400640 1 --2- 172.21.15.43:0/1996278643 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55caf7314a50 0x55caf7316e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.713+0000 7f6f71400640 1 -- 172.21.15.43:0/1996278643 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55caf730b550 msgr2=0x55caf730b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:55.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.713+0000 7f6f71400640 1 --2- 172.21.15.43:0/1996278643 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55caf730b550 0x55caf730b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.713+0000 7f6f71400640 1 -- 172.21.15.43:0/1996278643 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55caf6f833f0 con 0x55caf730be70 2024-03-28T22:40:55.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.713+0000 7f6f71400640 1 --2- 172.21.15.43:0/1996278643 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55caf730be70 0x55caf7314510 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6f600cb210 tx=0x7f6f600d65f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c2b43b4e828e885 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:55.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.713+0000 7f6f5e600640 1 -- 172.21.15.43:0/1996278643 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f600e1070 con 0x55caf730be70 2024-03-28T22:40:55.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.713+0000 7f6f5e600640 1 -- 172.21.15.43:0/1996278643 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f600dcae0 con 0x55caf730be70 2024-03-28T22:40:55.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.714+0000 7f6f5e600640 1 -- 172.21.15.43:0/1996278643 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f600dbc40 con 0x55caf730be70 2024-03-28T22:40:55.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.714+0000 7f6f6a000640 1 --2- 172.21.15.43:0/2245965183 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6f540a1ef0 0x7f6f540a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.714+0000 7f6f6a000640 1 -- 172.21.15.43:0/2245965183 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f6f600cdef0 con 0x7f6f6c073b00 2024-03-28T22:40:55.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.714+0000 7f6f71e00640 1 --2- 172.21.15.43:0/2245965183 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6f540a1ef0 0x7f6f540a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.714+0000 7f6f76c31740 1 -- 172.21.15.43:0/1996278643 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55caf730be70 msgr2=0x55caf7314510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:55.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.714+0000 7f6f76c31740 1 --2- 172.21.15.43:0/1996278643 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55caf730be70 0x55caf7314510 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6f600cb210 tx=0x7f6f600d65f0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.715+0000 7f6f76c31740 1 -- 172.21.15.43:0/1996278643 shutdown_connections 2024-03-28T22:40:55.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.715+0000 7f6f76c31740 1 --2- 172.21.15.43:0/1996278643 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55caf7314a50 0x55caf7316e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.715+0000 7f6f76c31740 1 --2- 172.21.15.43:0/1996278643 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55caf730be70 0x55caf7314510 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.715+0000 7f6f76c31740 1 --2- 172.21.15.43:0/1996278643 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55caf730b550 0x55caf730b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.715+0000 7f6f76c31740 1 -- 172.21.15.43:0/1996278643 >> 172.21.15.43:0/1996278643 conn(0x55caf7305c80 msgr2=0x55caf7306060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:55.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.715+0000 7f6f76c31740 1 -- 172.21.15.43:0/1996278643 shutdown_connections 2024-03-28T22:40:55.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.715+0000 7f6f76c31740 1 -- 172.21.15.43:0/1996278643 wait complete. 2024-03-28T22:40:55.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.715+0000 7f6f71e00640 1 --2- 172.21.15.43:0/2245965183 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6f540a1ef0 0x7f6f540a43b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6f64005fd0 tx=0x7f6f64009500 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:55.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.715+0000 7f6f76c31740 1 -- start start 2024-03-28T22:40:55.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.716+0000 7f6f76c31740 1 Processor -- start 2024-03-28T22:40:55.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.716+0000 7f6f76c31740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55caf730b550 0x55caf7311a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.716+0000 7f6f76c31740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55caf730be70 0x55caf7311fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.716+0000 7f6f76c31740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55caf7314a50 0x55caf7312510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.716+0000 7f6f76c31740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55caf7317840 con 0x55caf730be70 2024-03-28T22:40:55.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.716+0000 7f6f76c31740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55caf720d2f0 con 0x55caf730b550 2024-03-28T22:40:55.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.716+0000 7f6f76c31740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55caf73179c0 con 0x55caf7314a50 2024-03-28T22:40:55.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.716+0000 7f6f76c31740 10 client.0 _ll_register_callbacks cb 0x55caf6f82be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:55.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.716+0000 7f6f71400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55caf730b550 0x55caf7311a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.716+0000 7f6f71400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55caf730b550 0x55caf7311a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59736/0 (socket says 172.21.15.43:59736) 2024-03-28T22:40:55.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.716+0000 7f6f71400640 1 -- 172.21.15.43:0/1410046790 learned_addr learned my addr 172.21.15.43:0/1410046790 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:55.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.716+0000 7f6f70a00640 1 --2- 172.21.15.43:0/1410046790 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55caf730be70 0x55caf7311fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.716+0000 7f6f71e00640 1 --2- 172.21.15.43:0/1410046790 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55caf7314a50 0x55caf7312510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.717+0000 7f6f71400640 1 -- 172.21.15.43:0/1410046790 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55caf7314a50 msgr2=0x55caf7312510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:55.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.717+0000 7f6f71400640 1 --2- 172.21.15.43:0/1410046790 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55caf7314a50 0x55caf7312510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.717+0000 7f6f71400640 1 -- 172.21.15.43:0/1410046790 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55caf730be70 msgr2=0x55caf7311fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:55.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.717+0000 7f6f71400640 1 --2- 172.21.15.43:0/1410046790 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55caf730be70 0x55caf7311fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.717+0000 7f6f71400640 1 -- 172.21.15.43:0/1410046790 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55caf704d7d0 con 0x55caf730b550 2024-03-28T22:40:55.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.717+0000 7f6f71400640 1 --2- 172.21.15.43:0/1410046790 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55caf730b550 0x55caf7311a90 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f6f60096fc0 tx=0x7f6f600db4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19736c156bac9a79 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:55.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.717+0000 7f6f52a00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:40:55.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.717+0000 7f6f52a00640 1 -- 172.21.15.43:0/1410046790 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f60020070 con 0x55caf730b550 2024-03-28T22:40:55.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.717+0000 7f6f52a00640 1 -- 172.21.15.43:0/1410046790 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f6f600d1bf0 con 0x55caf730b550 2024-03-28T22:40:55.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.717+0000 7f6f76c31740 1 -- 172.21.15.43:0/1410046790 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f600dcae0 con 0x55caf730b550 2024-03-28T22:40:55.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.718+0000 7f6f52a00640 1 -- 172.21.15.43:0/1410046790 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6f60096890 con 0x55caf730b550 2024-03-28T22:40:55.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.718+0000 7f6f76c31740 10 client.26966 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:55.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.718+0000 7f6f76c31740 1 -- 172.21.15.43:0/1410046790 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55caf7317840 con 0x55caf730b550 2024-03-28T22:40:55.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.718+0000 7f6f52a00640 1 -- 172.21.15.43:0/1410046790 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f600dcae0 con 0x55caf730b550 2024-03-28T22:40:55.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.718+0000 7f6f52a00640 1 -- 172.21.15.43:0/1410046790 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6f600cb520 con 0x55caf730b550 2024-03-28T22:40:55.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.718+0000 7f6f76c31740 10 client.26966 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:55.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.718+0000 7f6f76c31740 10 client.26966 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:40:55.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.718+0000 7f6f76c31740 20 client.26966 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:55.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.718+0000 7f6f76c31740 1 --2- 172.21.15.43:0/1410046790 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55caf7324c80 0x55caf7327070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.718+0000 7f6f76c31740 4 client.26966 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:40:55.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.718+0000 7f6f76c31740 1 -- 172.21.15.43:0/1410046790 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55caf7218680 con 0x55caf7324c80 2024-03-28T22:40:55.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.719+0000 7f6f70a00640 1 --2- 172.21.15.43:0/1410046790 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55caf7324c80 0x55caf7327070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.719+0000 7f6f70a00640 1 --2- 172.21.15.43:0/1410046790 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55caf7324c80 0x55caf7327070 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e10b85ac72bf60b1 server_cookie=e0c17bbefa43f93e in_seq=0 out_seq=0 2024-03-28T22:40:55.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.719+0000 7f6f52a00640 10 client.26966 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:40:55.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.720+0000 7f6f52a00640 1 -- 172.21.15.43:0/1410046790 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55caf7218680 con 0x55caf7324c80 2024-03-28T22:40:55.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.720+0000 7f6f52a00640 10 client.26966 handle_command_reply: tid=0 2024-03-28T22:40:55.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.720+0000 7f6f76c31740 1 client.26966 shutdown 2024-03-28T22:40:55.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.720+0000 7f6f76c31740 1 -- 172.21.15.43:0/1410046790 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55caf730b550 msgr2=0x55caf7311a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:55.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.720+0000 7f6f76c31740 1 --2- 172.21.15.43:0/1410046790 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55caf730b550 0x55caf7311a90 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f6f60096fc0 tx=0x7f6f600db4c0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.720+0000 7f6f76c31740 1 -- 172.21.15.43:0/1410046790 shutdown_connections 2024-03-28T22:40:55.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.720+0000 7f6f76c31740 1 --2- 172.21.15.43:0/1410046790 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55caf7324c80 0x55caf7327070 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.720+0000 7f6f76c31740 1 --2- 172.21.15.43:0/1410046790 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55caf7314a50 0x55caf7312510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.720+0000 7f6f76c31740 1 --2- 172.21.15.43:0/1410046790 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55caf730be70 0x55caf7311fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.720+0000 7f6f76c31740 1 --2- 172.21.15.43:0/1410046790 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55caf730b550 0x55caf7311a90 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.720+0000 7f6f76c31740 1 -- 172.21.15.43:0/1410046790 >> 172.21.15.43:0/1410046790 conn(0x55caf7305c80 msgr2=0x55caf73079d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:55.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.720+0000 7f6f52a00640 0 client.26966 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:40:55.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.721+0000 7f6f76c31740 1 -- 172.21.15.43:0/1410046790 shutdown_connections 2024-03-28T22:40:55.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.721+0000 7f6f76c31740 1 -- 172.21.15.43:0/1410046790 wait complete. 2024-03-28T22:40:55.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.721+0000 7f6f76c31740 20 client.26966 trim_cache size 0 max 16384 2024-03-28T22:40:55.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.736+0000 7f6f76c31740 1 Processor -- start 2024-03-28T22:40:55.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.736+0000 7f6f76c31740 1 -- start start 2024-03-28T22:40:55.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.736+0000 7f6f76c31740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55caf730b550 0x55caf7312330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.737+0000 7f6f76c31740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55caf730be70 0x55caf7304c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.737+0000 7f6f76c31740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55caf7314a50 0x55caf730ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.737+0000 7f6f76c31740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6c1b0bd0 con 0x55caf730be70 2024-03-28T22:40:55.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.737+0000 7f6f76c31740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6c1b2710 con 0x55caf730b550 2024-03-28T22:40:55.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.737+0000 7f6f76c31740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6c1b12a0 con 0x55caf7314a50 2024-03-28T22:40:55.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.737+0000 7f6f71400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55caf730b550 0x55caf7312330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.737+0000 7f6f71e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55caf7314a50 0x55caf730ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.737+0000 7f6f71400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55caf730b550 0x55caf7312330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59742/0 (socket says 172.21.15.43:59742) 2024-03-28T22:40:55.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.737+0000 7f6f71400640 1 -- 172.21.15.43:0/572005753 learned_addr learned my addr 172.21.15.43:0/572005753 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:55.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.737+0000 7f6f70a00640 1 --2- 172.21.15.43:0/572005753 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55caf730be70 0x55caf7304c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.737+0000 7f6f71e00640 1 -- 172.21.15.43:0/572005753 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55caf730b550 msgr2=0x55caf7312330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:55.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.737+0000 7f6f71e00640 1 --2- 172.21.15.43:0/572005753 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55caf730b550 0x55caf7312330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.737+0000 7f6f71e00640 1 -- 172.21.15.43:0/572005753 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55caf730be70 msgr2=0x55caf7304c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:55.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.737+0000 7f6f71e00640 1 --2- 172.21.15.43:0/572005753 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55caf730be70 0x55caf7304c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.737+0000 7f6f71e00640 1 -- 172.21.15.43:0/572005753 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55caf7320b10 con 0x55caf7314a50 2024-03-28T22:40:55.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.737+0000 7f6f71400640 1 --2- 172.21.15.43:0/572005753 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55caf730b550 0x55caf7312330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:55.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.738+0000 7f6f71e00640 1 --2- 172.21.15.43:0/572005753 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55caf7314a50 0x55caf730ed10 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f6f64003c90 tx=0x7f6f6401b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4c64a83d9762ed7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:55.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.738+0000 7f6f5dc00640 1 -- 172.21.15.43:0/572005753 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f64003cc0 con 0x55caf7314a50 2024-03-28T22:40:55.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.738+0000 7f6f5dc00640 1 -- 172.21.15.43:0/572005753 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f640045c0 con 0x55caf7314a50 2024-03-28T22:40:55.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.738+0000 7f6f5dc00640 1 -- 172.21.15.43:0/572005753 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6401d660 con 0x55caf7314a50 2024-03-28T22:40:55.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.739+0000 7f6f76c31740 1 -- 172.21.15.43:0/572005753 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55caf7314a50 msgr2=0x55caf730ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:55.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.739+0000 7f6f76c31740 1 --2- 172.21.15.43:0/572005753 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55caf7314a50 0x55caf730ed10 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f6f64003c90 tx=0x7f6f6401b000 comp rx=0 tx=0).stop 2024-03-28T22:40:55.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.739+0000 7f6f76c31740 1 -- 172.21.15.43:0/572005753 shutdown_connections 2024-03-28T22:40:55.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.739+0000 7f6f76c31740 1 --2- 172.21.15.43:0/572005753 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55caf7314a50 0x55caf730ed10 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.739+0000 7f6f76c31740 1 --2- 172.21.15.43:0/572005753 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55caf730be70 0x55caf7304c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.739+0000 7f6f76c31740 1 --2- 172.21.15.43:0/572005753 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55caf730b550 0x55caf7312330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.739+0000 7f6f76c31740 1 -- 172.21.15.43:0/572005753 >> 172.21.15.43:0/572005753 conn(0x55caf7305c80 msgr2=0x55caf7304530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:55.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.739+0000 7f6f76c31740 1 -- 172.21.15.43:0/572005753 shutdown_connections 2024-03-28T22:40:55.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.739+0000 7f6f76c31740 1 -- 172.21.15.43:0/572005753 wait complete. 2024-03-28T22:40:55.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.739+0000 7f6f76c31740 1 -- start start 2024-03-28T22:40:55.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.740+0000 7f6f76c31740 1 Processor -- start 2024-03-28T22:40:55.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.740+0000 7f6f76c31740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55caf730b550 0x55caf7327c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.740+0000 7f6f76c31740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55caf730be70 0x55caf7328190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.740+0000 7f6f76c31740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55caf7314a50 0x55caf73286d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.740+0000 7f6f76c31740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6c1b2040 con 0x55caf730be70 2024-03-28T22:40:55.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.740+0000 7f6f76c31740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6c1b1970 con 0x55caf7314a50 2024-03-28T22:40:55.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.740+0000 7f6f76c31740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6c1aebb0 con 0x55caf730b550 2024-03-28T22:40:55.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.740+0000 7f6f76c31740 10 client.0 _ll_register_callbacks cb 0x7f6f600df070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:55.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.740+0000 7f6f71400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55caf730b550 0x55caf7327c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.740+0000 7f6f71400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55caf730b550 0x55caf7327c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58508/0 (socket says 172.21.15.43:58508) 2024-03-28T22:40:55.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.740+0000 7f6f71400640 1 -- 172.21.15.43:0/1009550119 learned_addr learned my addr 172.21.15.43:0/1009550119 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:55.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.741+0000 7f6f71e00640 1 --2- 172.21.15.43:0/1009550119 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55caf7314a50 0x55caf73286d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.741+0000 7f6f70a00640 1 --2- 172.21.15.43:0/1009550119 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55caf730be70 0x55caf7328190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.741+0000 7f6f71400640 1 -- 172.21.15.43:0/1009550119 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55caf7314a50 msgr2=0x55caf73286d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:55.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.741+0000 7f6f71400640 1 --2- 172.21.15.43:0/1009550119 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55caf7314a50 0x55caf73286d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.741+0000 7f6f71400640 1 -- 172.21.15.43:0/1009550119 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55caf730be70 msgr2=0x55caf7328190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:55.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.741+0000 7f6f71400640 1 --2- 172.21.15.43:0/1009550119 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55caf730be70 0x55caf7328190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.741+0000 7f6f71400640 1 -- 172.21.15.43:0/1009550119 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55caf7389ec0 con 0x55caf730b550 2024-03-28T22:40:55.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.741+0000 7f6f70a00640 1 --2- 172.21.15.43:0/1009550119 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55caf730be70 0x55caf7328190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:40:55.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.741+0000 7f6f71400640 1 --2- 172.21.15.43:0/1009550119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55caf730b550 0x55caf7327c50 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f6f60009610 tx=0x7f6f6009e790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e5bea9cefa0db3c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:55.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.741+0000 7f6f52a00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:40:55.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.742+0000 7f6f52a00640 1 -- 172.21.15.43:0/1009550119 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f60020070 con 0x55caf730b550 2024-03-28T22:40:55.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.742+0000 7f6f76c31740 1 -- 172.21.15.43:0/1009550119 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f640045c0 con 0x55caf730b550 2024-03-28T22:40:55.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.742+0000 7f6f52a00640 1 -- 172.21.15.43:0/1009550119 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f6f60009cc0 con 0x55caf730b550 2024-03-28T22:40:55.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.742+0000 7f6f52a00640 1 -- 172.21.15.43:0/1009550119 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6f600056b0 con 0x55caf730b550 2024-03-28T22:40:55.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.742+0000 7f6f76c31740 10 client.26923 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:55.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.742+0000 7f6f76c31740 1 -- 172.21.15.43:0/1009550119 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55caf738b1c0 con 0x55caf730b550 2024-03-28T22:40:55.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.742+0000 7f6f52a00640 1 -- 172.21.15.43:0/1009550119 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f640045c0 con 0x55caf730b550 2024-03-28T22:40:55.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.743+0000 7f6f52a00640 1 -- 172.21.15.43:0/1009550119 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6f600b5080 con 0x55caf730b550 2024-03-28T22:40:55.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.743+0000 7f6f76c31740 10 client.26923 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:55.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.743+0000 7f6f76c31740 10 client.26923 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:40:55.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.743+0000 7f6f76c31740 20 client.26923 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:55.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.743+0000 7f6f76c31740 1 --2- 172.21.15.43:0/1009550119 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55caf738b740 0x55caf738db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:55.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.743+0000 7f6f76c31740 4 client.26923 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:40:55.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.743+0000 7f6f76c31740 1 -- 172.21.15.43:0/1009550119 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55caf727c590 con 0x55caf738b740 2024-03-28T22:40:55.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.743+0000 7f6f70a00640 1 --2- 172.21.15.43:0/1009550119 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55caf738b740 0x55caf738db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:55.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.744+0000 7f6f70a00640 1 --2- 172.21.15.43:0/1009550119 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55caf738b740 0x55caf738db30 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=cc85a81383a7f26a server_cookie=29fc0b921cc7bc91 in_seq=0 out_seq=0 2024-03-28T22:40:55.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.744+0000 7f6f52a00640 10 client.26923 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:40:55.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.744+0000 7f6f52a00640 1 -- 172.21.15.43:0/1009550119 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55caf727c590 con 0x55caf738b740 2024-03-28T22:40:55.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.744+0000 7f6f52a00640 10 client.26923 handle_command_reply: tid=0 2024-03-28T22:40:55.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.744+0000 7f6f76c31740 1 client.26923 shutdown 2024-03-28T22:40:55.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.744+0000 7f6f76c31740 1 -- 172.21.15.43:0/1009550119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55caf730b550 msgr2=0x55caf7327c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:55.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.744+0000 7f6f76c31740 1 --2- 172.21.15.43:0/1009550119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55caf730b550 0x55caf7327c50 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f6f60009610 tx=0x7f6f6009e790 comp rx=0 tx=0).stop 2024-03-28T22:40:55.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.744+0000 7f6f76c31740 1 -- 172.21.15.43:0/1009550119 shutdown_connections 2024-03-28T22:40:55.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.744+0000 7f6f76c31740 1 --2- 172.21.15.43:0/1009550119 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55caf738b740 0x55caf738db30 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.744+0000 7f6f76c31740 1 --2- 172.21.15.43:0/1009550119 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55caf7314a50 0x55caf73286d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.744+0000 7f6f76c31740 1 --2- 172.21.15.43:0/1009550119 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55caf730be70 0x55caf7328190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.744+0000 7f6f76c31740 1 --2- 172.21.15.43:0/1009550119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55caf730b550 0x55caf7327c50 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.744+0000 7f6f76c31740 1 -- 172.21.15.43:0/1009550119 >> 172.21.15.43:0/1009550119 conn(0x55caf7305c80 msgr2=0x55caf7304dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:55.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.744+0000 7f6f52a00640 0 client.26923 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:40:55.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.744+0000 7f6f76c31740 1 -- 172.21.15.43:0/1009550119 shutdown_connections 2024-03-28T22:40:55.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.745+0000 7f6f76c31740 1 -- 172.21.15.43:0/1009550119 wait complete. 2024-03-28T22:40:55.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.745+0000 7f6f76c31740 20 client.26923 trim_cache size 0 max 16384 2024-03-28T22:40:55.749 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:40:55.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.745+0000 7f6f72800640 1 -- 172.21.15.43:0/2245965183 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6f540a1ef0 msgr2=0x7f6f540a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:55.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.745+0000 7f6f72800640 1 --2- 172.21.15.43:0/2245965183 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6f540a1ef0 0x7f6f540a43b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6f64005fd0 tx=0x7f6f64009500 comp rx=0 tx=0).stop 2024-03-28T22:40:55.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.745+0000 7f6f72800640 1 -- 172.21.15.43:0/2245965183 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f6c073b00 msgr2=0x7f6f6c19a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:55.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.745+0000 7f6f72800640 1 --2- 172.21.15.43:0/2245965183 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f6c073b00 0x7f6f6c19a0c0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f6f6000d770 tx=0x7f6f6000dc30 comp rx=0 tx=0).stop 2024-03-28T22:40:55.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.746+0000 7f6f72800640 1 -- 172.21.15.43:0/2245965183 shutdown_connections 2024-03-28T22:40:55.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.746+0000 7f6f72800640 1 --2- 172.21.15.43:0/2245965183 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6f540a1ef0 0x7f6f540a43b0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.746+0000 7f6f72800640 1 --2- 172.21.15.43:0/2245965183 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f6c10de80 0x7f6f6c19e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.746+0000 7f6f72800640 1 --2- 172.21.15.43:0/2245965183 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f6c074420 0x7f6f6c19a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.746+0000 7f6f72800640 1 --2- 172.21.15.43:0/2245965183 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f6c073b00 0x7f6f6c19a0c0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:55.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.746+0000 7f6f72800640 1 -- 172.21.15.43:0/2245965183 >> 172.21.15.43:0/2245965183 conn(0x7f6f6c0fd0f0 msgr2=0x7f6f6c0fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:55.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.746+0000 7f6f72800640 1 -- 172.21.15.43:0/2245965183 shutdown_connections 2024-03-28T22:40:55.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:55.746+0000 7f6f72800640 1 -- 172.21.15.43:0/2245965183 wait complete. 2024-03-28T22:40:55.764 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:40:56.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.012+0000 7f1952000640 1 -- 172.21.15.43:0/3516563782 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f194c1136a0 msgr2=0x7f194c115a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:56.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.012+0000 7f1952000640 1 --2- 172.21.15.43:0/3516563782 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f194c1136a0 0x7f194c115a90 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1944009ca0 tx=0x7f194402c100 comp rx=0 tx=0).stop 2024-03-28T22:40:56.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.012+0000 7f1952000640 1 -- 172.21.15.43:0/3516563782 shutdown_connections 2024-03-28T22:40:56.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.012+0000 7f1952000640 1 --2- 172.21.15.43:0/3516563782 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f194c10aa10 0x7f194c113160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.012+0000 7f1952000640 1 --2- 172.21.15.43:0/3516563782 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f194c1136a0 0x7f194c115a90 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.012+0000 7f1952000640 1 --2- 172.21.15.43:0/3516563782 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f194c10a0f0 0x7f194c10a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.012+0000 7f1952000640 1 -- 172.21.15.43:0/3516563782 >> 172.21.15.43:0/3516563782 conn(0x7f194c06c830 msgr2=0x7f194c06cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:56.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.012+0000 7f1952000640 1 -- 172.21.15.43:0/3516563782 shutdown_connections 2024-03-28T22:40:56.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.012+0000 7f1952000640 1 -- 172.21.15.43:0/3516563782 wait complete. 2024-03-28T22:40:56.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.014+0000 7f1952000640 1 Processor -- start 2024-03-28T22:40:56.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.014+0000 7f1952000640 1 -- start start 2024-03-28T22:40:56.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.014+0000 7f1952000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f194c10a0f0 0x7f194c1a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.014+0000 7f1952000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f194c10aa10 0x7f194c1a6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.014+0000 7f1952000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f194c1136a0 0x7f194c1a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.014+0000 7f1952000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f194c11a380 con 0x7f194c1136a0 2024-03-28T22:40:56.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.014+0000 7f1952000640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f194c11a200 con 0x7f194c10aa10 2024-03-28T22:40:56.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.014+0000 7f1952000640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f194c11a500 con 0x7f194c10a0f0 2024-03-28T22:40:56.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.014+0000 7f194b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f194c10a0f0 0x7f194c1a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.015+0000 7f1943e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f194c10aa10 0x7f194c1a6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.015+0000 7f194b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f194c10a0f0 0x7f194c1a6430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58520/0 (socket says 172.21.15.43:58520) 2024-03-28T22:40:56.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.015+0000 7f194b400640 1 -- 172.21.15.43:0/3162496112 learned_addr learned my addr 172.21.15.43:0/3162496112 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:56.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.015+0000 7f194be00640 1 --2- 172.21.15.43:0/3162496112 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f194c1136a0 0x7f194c1a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.015+0000 7f1943e00640 1 -- 172.21.15.43:0/3162496112 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f194c10a0f0 msgr2=0x7f194c1a6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:56.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.015+0000 7f1943e00640 1 --2- 172.21.15.43:0/3162496112 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f194c10a0f0 0x7f194c1a6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.015+0000 7f1943e00640 1 -- 172.21.15.43:0/3162496112 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f194c1136a0 msgr2=0x7f194c1a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:56.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.015+0000 7f1943e00640 1 --2- 172.21.15.43:0/3162496112 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f194c1136a0 0x7f194c1a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.015+0000 7f1943e00640 1 -- 172.21.15.43:0/3162496112 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f194c1a0d60 con 0x7f194c10aa10 2024-03-28T22:40:56.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.016+0000 7f1943e00640 1 --2- 172.21.15.43:0/3162496112 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f194c10aa10 0x7f194c1a6970 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f193400d650 tx=0x7f193400db10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:56.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.017+0000 7f1948c00640 1 -- 172.21.15.43:0/3162496112 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1934017070 con 0x7f194c10aa10 2024-03-28T22:40:56.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.017+0000 7f1952000640 1 -- 172.21.15.43:0/3162496112 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f194c1a1050 con 0x7f194c10aa10 2024-03-28T22:40:56.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.017+0000 7f1952000640 1 -- 172.21.15.43:0/3162496112 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f194c1ad2f0 con 0x7f194c10aa10 2024-03-28T22:40:56.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.017+0000 7f1948c00640 1 -- 172.21.15.43:0/3162496112 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1934009ca0 con 0x7f194c10aa10 2024-03-28T22:40:56.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.017+0000 7f1948c00640 1 -- 172.21.15.43:0/3162496112 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1934003b60 con 0x7f194c10aa10 2024-03-28T22:40:56.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.022+0000 7f1954bf3740 1 Processor -- start 2024-03-28T22:40:56.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.022+0000 7f1954bf3740 1 -- start start 2024-03-28T22:40:56.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.022+0000 7f1948c00640 1 -- 172.21.15.43:0/3162496112 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f1934003d00 con 0x7f194c10aa10 2024-03-28T22:40:56.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.023+0000 7f1954bf3740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cca780a20 0x564cca780e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.023+0000 7f1954bf3740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cca7813d0 0x564cca76d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.023+0000 7f1954bf3740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cca76db80 0x564cca76dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.023+0000 7f1954bf3740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x564cca6d7050 con 0x564cca780a20 2024-03-28T22:40:56.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.023+0000 7f1954bf3740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x564cca67fce0 con 0x564cca76db80 2024-03-28T22:40:56.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.023+0000 7f1954bf3740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x564cca6d30f0 con 0x564cca7813d0 2024-03-28T22:40:56.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.024+0000 7f194be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cca780a20 0x564cca780e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.024+0000 7f194b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cca7813d0 0x564cca76d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.024+0000 7f194b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cca7813d0 0x564cca76d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58522/0 (socket says 172.21.15.43:58522) 2024-03-28T22:40:56.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.024+0000 7f194b400640 1 -- 172.21.15.43:0/1737267022 learned_addr learned my addr 172.21.15.43:0/1737267022 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:56.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.024+0000 7f1943e00640 1 --2- 172.21.15.43:0/1737267022 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cca76db80 0x564cca76dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.025+0000 7f194b400640 1 -- 172.21.15.43:0/1737267022 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cca76db80 msgr2=0x564cca76dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:56.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.025+0000 7f194b400640 1 --2- 172.21.15.43:0/1737267022 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cca76db80 0x564cca76dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.025+0000 7f194b400640 1 -- 172.21.15.43:0/1737267022 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cca780a20 msgr2=0x564cca780e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:56.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.025+0000 7f194b400640 1 --2- 172.21.15.43:0/1737267022 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cca780a20 0x564cca780e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.025+0000 7f194b400640 1 -- 172.21.15.43:0/1737267022 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cca3eb3f0 con 0x564cca7813d0 2024-03-28T22:40:56.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.025+0000 7f194be00640 1 --2- 172.21.15.43:0/1737267022 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cca780a20 0x564cca780e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:40:56.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.026+0000 7f194b400640 1 --2- 172.21.15.43:0/1737267022 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cca7813d0 0x564cca76d560 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f193c009e10 tx=0x7f193c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b971eb10663a2fae server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:56.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.026+0000 7f1948c00640 1 --2- 172.21.15.43:0/3162496112 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f19300a1f70 0x7f19300a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.027+0000 7f193be00640 1 -- 172.21.15.43:0/1737267022 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193c01e070 con 0x564cca7813d0 2024-03-28T22:40:56.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.027+0000 7f1948c00640 1 -- 172.21.15.43:0/3162496112 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f19340cdc00 con 0x7f194c10aa10 2024-03-28T22:40:56.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.027+0000 7f193be00640 1 -- 172.21.15.43:0/1737267022 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f193c019b10 con 0x564cca7813d0 2024-03-28T22:40:56.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.027+0000 7f193be00640 1 -- 172.21.15.43:0/1737267022 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193c00fc30 con 0x564cca7813d0 2024-03-28T22:40:56.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.027+0000 7f194be00640 1 --2- 172.21.15.43:0/3162496112 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f19300a1f70 0x7f19300a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.030+0000 7f1954bf3740 1 -- 172.21.15.43:0/1737267022 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cca7813d0 msgr2=0x564cca76d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:56.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.030+0000 7f194be00640 1 --2- 172.21.15.43:0/3162496112 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f19300a1f70 0x7f19300a4430 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f194402c640 tx=0x7f194402e040 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:56.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.031+0000 7f1954bf3740 1 --2- 172.21.15.43:0/1737267022 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cca7813d0 0x564cca76d560 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f193c009e10 tx=0x7f193c011810 comp rx=0 tx=0).stop 2024-03-28T22:40:56.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.031+0000 7f1954bf3740 1 -- 172.21.15.43:0/1737267022 shutdown_connections 2024-03-28T22:40:56.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.031+0000 7f1954bf3740 1 --2- 172.21.15.43:0/1737267022 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cca76db80 0x564cca76dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.031+0000 7f1954bf3740 1 --2- 172.21.15.43:0/1737267022 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cca7813d0 0x564cca76d560 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.031+0000 7f1954bf3740 1 --2- 172.21.15.43:0/1737267022 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cca780a20 0x564cca780e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.031+0000 7f1954bf3740 1 -- 172.21.15.43:0/1737267022 >> 172.21.15.43:0/1737267022 conn(0x564cca770020 msgr2=0x564cca770400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:56.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.032+0000 7f1954bf3740 1 -- 172.21.15.43:0/1737267022 shutdown_connections 2024-03-28T22:40:56.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.032+0000 7f1954bf3740 1 -- 172.21.15.43:0/1737267022 wait complete. 2024-03-28T22:40:56.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.033+0000 7f1954bf3740 1 -- start start 2024-03-28T22:40:56.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.033+0000 7f1954bf3740 1 Processor -- start 2024-03-28T22:40:56.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.035+0000 7f1954bf3740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cca76db80 0x564cca7849e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.036+0000 7f1954bf3740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cca780a20 0x564cca784f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.036+0000 7f1954bf3740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cca7813d0 0x564cca785460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.036+0000 7f1954bf3740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x564cca787e60 con 0x564cca76db80 2024-03-28T22:40:56.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.036+0000 7f1954bf3740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x564cca6752f0 con 0x564cca7813d0 2024-03-28T22:40:56.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.036+0000 7f1954bf3740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x564cca787fe0 con 0x564cca780a20 2024-03-28T22:40:56.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.036+0000 7f1943e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cca780a20 0x564cca784f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.036+0000 7f1943e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cca780a20 0x564cca784f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58534/0 (socket says 172.21.15.43:58534) 2024-03-28T22:40:56.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.036+0000 7f1943e00640 1 -- 172.21.15.43:0/2976208832 learned_addr learned my addr 172.21.15.43:0/2976208832 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:56.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.037+0000 7f1943e00640 1 -- 172.21.15.43:0/2976208832 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cca7813d0 msgr2=0x564cca785460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:56.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.037+0000 7f194b400640 1 --2- 172.21.15.43:0/2976208832 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cca76db80 0x564cca7849e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.038+0000 7f1954bf3740 10 client.0 _ll_register_callbacks cb 0x564cca3eabe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:56.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.037+0000 7f194be00640 1 --2- 172.21.15.43:0/2976208832 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cca7813d0 0x564cca785460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.038+0000 7f1943e00640 1 --2- 172.21.15.43:0/2976208832 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cca7813d0 0x564cca785460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.038+0000 7f1943e00640 1 -- 172.21.15.43:0/2976208832 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cca76db80 msgr2=0x564cca7849e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:56.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.038+0000 7f1943e00640 1 --2- 172.21.15.43:0/2976208832 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cca76db80 0x564cca7849e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.038+0000 7f1943e00640 1 -- 172.21.15.43:0/2976208832 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cca4b57d0 con 0x564cca780a20 2024-03-28T22:40:56.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.038+0000 7f194b400640 1 --2- 172.21.15.43:0/2976208832 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cca76db80 0x564cca7849e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:40:56.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.039+0000 7f1954bf3740 1 -- 172.21.15.43:0/2976208832 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f193c019b10 con 0x564cca780a20 2024-03-28T22:40:56.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.039+0000 7f194be00640 1 --2- 172.21.15.43:0/2976208832 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cca7813d0 0x564cca785460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:40:56.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.039+0000 7f1943e00640 1 --2- 172.21.15.43:0/2976208832 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cca780a20 0x564cca784f20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f1934003770 tx=0x7f1934099d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed2981c14b694bd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:56.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.039+0000 7f1938c00640 10 client.26935 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:40:56.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.040+0000 7f1938c00640 1 -- 172.21.15.43:0/2976208832 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1934020070 con 0x564cca780a20 2024-03-28T22:40:56.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.040+0000 7f1938c00640 1 -- 172.21.15.43:0/2976208832 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f19340cb1d0 con 0x564cca780a20 2024-03-28T22:40:56.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.040+0000 7f1938c00640 1 -- 172.21.15.43:0/2976208832 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f19340cb3d0 con 0x564cca780a20 2024-03-28T22:40:56.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.040+0000 7f1954bf3740 10 client.26935 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:56.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.040+0000 7f1954bf3740 1 -- 172.21.15.43:0/2976208832 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({fsmap=32}) -- 0x564cca794170 con 0x564cca780a20 2024-03-28T22:40:56.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.040+0000 7f1938c00640 1 -- 172.21.15.43:0/2976208832 <== mon.1 v2:172.21.15.172:3300/0 4 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f1934099840 con 0x564cca780a20 2024-03-28T22:40:56.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.040+0000 7f1954bf3740 10 client.26935 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:56.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.041+0000 7f1954bf3740 10 client.26935 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:40:56.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.041+0000 7f1954bf3740 20 client.26935 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:56.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.041+0000 7f1954bf3740 1 --2- 172.21.15.43:0/2976208832 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x564cca7954c0 0x564cca7978d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.041+0000 7f1954bf3740 4 client.26935 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:40:56.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.041+0000 7f1954bf3740 1 -- 172.21.15.43:0/2976208832 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564cca680680 con 0x564cca7954c0 2024-03-28T22:40:56.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.041+0000 7f194b400640 1 --2- 172.21.15.43:0/2976208832 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x564cca7954c0 0x564cca7978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.042+0000 7f194b400640 1 --2- 172.21.15.43:0/2976208832 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x564cca7954c0 0x564cca7978d0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=34915fa595f8d49 server_cookie=9dd8e6bc1ec79572 in_seq=0 out_seq=0 2024-03-28T22:40:56.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.042+0000 7f1938c00640 10 client.26935 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:40:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.043+0000 7f1938c00640 1 -- 172.21.15.43:0/2976208832 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564cca680680 con 0x564cca7954c0 2024-03-28T22:40:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.043+0000 7f1938c00640 10 client.26935 handle_command_reply: tid=0 2024-03-28T22:40:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.043+0000 7f1954bf3740 1 client.26935 shutdown 2024-03-28T22:40:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.043+0000 7f1954bf3740 1 -- 172.21.15.43:0/2976208832 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cca780a20 msgr2=0x564cca784f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.043+0000 7f1954bf3740 1 --2- 172.21.15.43:0/2976208832 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cca780a20 0x564cca784f20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f1934003770 tx=0x7f1934099d50 comp rx=0 tx=0).stop 2024-03-28T22:40:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.043+0000 7f1954bf3740 1 -- 172.21.15.43:0/2976208832 shutdown_connections 2024-03-28T22:40:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.043+0000 7f1954bf3740 1 --2- 172.21.15.43:0/2976208832 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x564cca7954c0 0x564cca7978d0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.043+0000 7f1954bf3740 1 --2- 172.21.15.43:0/2976208832 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cca7813d0 0x564cca785460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.043+0000 7f1954bf3740 1 --2- 172.21.15.43:0/2976208832 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cca780a20 0x564cca784f20 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.043+0000 7f1954bf3740 1 --2- 172.21.15.43:0/2976208832 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cca76db80 0x564cca7849e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.043+0000 7f1954bf3740 1 -- 172.21.15.43:0/2976208832 >> 172.21.15.43:0/2976208832 conn(0x564cca770020 msgr2=0x564cca76f780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:56.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.043+0000 7f1938c00640 0 client.26935 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:40:56.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.044+0000 7f1954bf3740 1 -- 172.21.15.43:0/2976208832 shutdown_connections 2024-03-28T22:40:56.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.044+0000 7f1954bf3740 1 -- 172.21.15.43:0/2976208832 wait complete. 2024-03-28T22:40:56.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.044+0000 7f1954bf3740 20 client.26935 trim_cache size 0 max 16384 2024-03-28T22:40:56.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.061+0000 7f1954bf3740 1 Processor -- start 2024-03-28T22:40:56.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.061+0000 7f1954bf3740 1 -- start start 2024-03-28T22:40:56.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.061+0000 7f1954bf3740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cca76db80 0x564cca787110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.061+0000 7f1954bf3740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cca780a20 0x564cca790ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.061+0000 7f1954bf3740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cca7813d0 0x564cca77fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.061+0000 7f1954bf3740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f192800ff80 con 0x564cca780a20 2024-03-28T22:40:56.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.061+0000 7f1954bf3740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1928011ac0 con 0x564cca7813d0 2024-03-28T22:40:56.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.061+0000 7f1954bf3740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f1928010650 con 0x564cca76db80 2024-03-28T22:40:56.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.061+0000 7f194b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cca76db80 0x564cca787110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.061+0000 7f194b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cca76db80 0x564cca787110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58542/0 (socket says 172.21.15.43:58542) 2024-03-28T22:40:56.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.061+0000 7f194b400640 1 -- 172.21.15.43:0/2434873370 learned_addr learned my addr 172.21.15.43:0/2434873370 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:56.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.061+0000 7f194be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cca7813d0 0x564cca77fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.061+0000 7f1943e00640 1 --2- 172.21.15.43:0/2434873370 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cca780a20 0x564cca790ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.062+0000 7f194be00640 1 -- 172.21.15.43:0/2434873370 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cca76db80 msgr2=0x564cca787110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:56.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.062+0000 7f194be00640 1 --2- 172.21.15.43:0/2434873370 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cca76db80 0x564cca787110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.062+0000 7f194be00640 1 -- 172.21.15.43:0/2434873370 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cca780a20 msgr2=0x564cca790ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:56.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.062+0000 7f194be00640 1 --2- 172.21.15.43:0/2434873370 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cca780a20 0x564cca790ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.062+0000 7f194be00640 1 -- 172.21.15.43:0/2434873370 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cca7911e0 con 0x564cca7813d0 2024-03-28T22:40:56.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.062+0000 7f194b400640 1 --2- 172.21.15.43:0/2434873370 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cca76db80 0x564cca787110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:56.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.062+0000 7f1943e00640 1 --2- 172.21.15.43:0/2434873370 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cca780a20 0x564cca790ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:56.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.062+0000 7f194be00640 1 --2- 172.21.15.43:0/2434873370 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cca7813d0 0x564cca77fa00 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f1944037040 tx=0x7f1944040000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc50eb6257da4054 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:56.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.062+0000 7f193b400640 1 -- 172.21.15.43:0/2434873370 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1944046070 con 0x564cca7813d0 2024-03-28T22:40:56.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.063+0000 7f193b400640 1 -- 172.21.15.43:0/2434873370 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1944003e50 con 0x564cca7813d0 2024-03-28T22:40:56.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.063+0000 7f193b400640 1 -- 172.21.15.43:0/2434873370 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1944039040 con 0x564cca7813d0 2024-03-28T22:40:56.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.063+0000 7f1954bf3740 1 -- 172.21.15.43:0/2434873370 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cca7813d0 msgr2=0x564cca77fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:56.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.063+0000 7f1954bf3740 1 --2- 172.21.15.43:0/2434873370 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cca7813d0 0x564cca77fa00 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f1944037040 tx=0x7f1944040000 comp rx=0 tx=0).stop 2024-03-28T22:40:56.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.063+0000 7f1954bf3740 1 -- 172.21.15.43:0/2434873370 shutdown_connections 2024-03-28T22:40:56.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.063+0000 7f1954bf3740 1 --2- 172.21.15.43:0/2434873370 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cca7813d0 0x564cca77fa00 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.063+0000 7f1954bf3740 1 --2- 172.21.15.43:0/2434873370 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cca780a20 0x564cca790ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.063+0000 7f1954bf3740 1 --2- 172.21.15.43:0/2434873370 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cca76db80 0x564cca787110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.063+0000 7f1954bf3740 1 -- 172.21.15.43:0/2434873370 >> 172.21.15.43:0/2434873370 conn(0x564cca770020 msgr2=0x564cca77ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:56.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.064+0000 7f1954bf3740 1 -- 172.21.15.43:0/2434873370 shutdown_connections 2024-03-28T22:40:56.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.064+0000 7f1954bf3740 1 -- 172.21.15.43:0/2434873370 wait complete. 2024-03-28T22:40:56.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.064+0000 7f1954bf3740 1 -- start start 2024-03-28T22:40:56.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.064+0000 7f1954bf3740 1 Processor -- start 2024-03-28T22:40:56.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.065+0000 7f1954bf3740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cca76db80 0x564cca78d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.065+0000 7f1954bf3740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cca780a20 0x564cca78a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.065+0000 7f1954bf3740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cca7813d0 0x564cca78a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.065+0000 7f1954bf3740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f19280113f0 con 0x564cca780a20 2024-03-28T22:40:56.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.065+0000 7f1954bf3740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1928010d20 con 0x564cca7813d0 2024-03-28T22:40:56.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.065+0000 7f1954bf3740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f192800df60 con 0x564cca76db80 2024-03-28T22:40:56.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.065+0000 7f1954bf3740 10 client.0 _ll_register_callbacks cb 0x7f193409a270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:56.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.065+0000 7f194b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cca76db80 0x564cca78d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.065+0000 7f194be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cca7813d0 0x564cca78a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.065+0000 7f194b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cca76db80 0x564cca78d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58550/0 (socket says 172.21.15.43:58550) 2024-03-28T22:40:56.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.065+0000 7f194b400640 1 -- 172.21.15.43:0/3358445061 learned_addr learned my addr 172.21.15.43:0/3358445061 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:56.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.065+0000 7f1943e00640 1 --2- 172.21.15.43:0/3358445061 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cca780a20 0x564cca78a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.065+0000 7f194be00640 1 -- 172.21.15.43:0/3358445061 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cca76db80 msgr2=0x564cca78d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:56.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.065+0000 7f194be00640 1 --2- 172.21.15.43:0/3358445061 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cca76db80 0x564cca78d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.066+0000 7f194be00640 1 -- 172.21.15.43:0/3358445061 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cca780a20 msgr2=0x564cca78a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:56.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.066+0000 7f194be00640 1 --2- 172.21.15.43:0/3358445061 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cca780a20 0x564cca78a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.066+0000 7f194be00640 1 -- 172.21.15.43:0/3358445061 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cca78d9c0 con 0x564cca7813d0 2024-03-28T22:40:56.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.066+0000 7f194b400640 1 --2- 172.21.15.43:0/3358445061 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cca76db80 0x564cca78d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:56.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.066+0000 7f194be00640 1 --2- 172.21.15.43:0/3358445061 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cca7813d0 0x564cca78a7e0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f1944004af0 tx=0x7f1944005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab4ffa38722bfabf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:56.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.066+0000 7f1938c00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:40:56.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.066+0000 7f1938c00640 1 -- 172.21.15.43:0/3358445061 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194405a060 con 0x564cca7813d0 2024-03-28T22:40:56.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.066+0000 7f1954bf3740 1 -- 172.21.15.43:0/3358445061 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1944003e50 con 0x564cca7813d0 2024-03-28T22:40:56.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.066+0000 7f1938c00640 1 -- 172.21.15.43:0/3358445061 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f1944003970 con 0x564cca7813d0 2024-03-28T22:40:56.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.066+0000 7f1938c00640 1 -- 172.21.15.43:0/3358445061 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1944061050 con 0x564cca7813d0 2024-03-28T22:40:56.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.066+0000 7f1954bf3740 10 client.26987 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:56.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.066+0000 7f1954bf3740 1 -- 172.21.15.43:0/3358445061 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564cca7fc2a0 con 0x564cca7813d0 2024-03-28T22:40:56.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.067+0000 7f1938c00640 1 -- 172.21.15.43:0/3358445061 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1944003e50 con 0x564cca7813d0 2024-03-28T22:40:56.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.067+0000 7f1938c00640 1 -- 172.21.15.43:0/3358445061 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f194404dc30 con 0x564cca7813d0 2024-03-28T22:40:56.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.067+0000 7f1954bf3740 10 client.26987 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:56.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.067+0000 7f1954bf3740 10 client.26987 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:40:56.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.067+0000 7f1954bf3740 20 client.26987 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:56.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.067+0000 7f1954bf3740 1 --2- 172.21.15.43:0/3358445061 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x564cca7fc9a0 0x564cca7fedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.067+0000 7f1954bf3740 4 client.26987 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:40:56.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.067+0000 7f1954bf3740 1 -- 172.21.15.43:0/3358445061 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564cca6e4590 con 0x564cca7fc9a0 2024-03-28T22:40:56.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.067+0000 7f194b400640 1 --2- 172.21.15.43:0/3358445061 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x564cca7fc9a0 0x564cca7fedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.068+0000 7f194b400640 1 --2- 172.21.15.43:0/3358445061 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x564cca7fc9a0 0x564cca7fedc0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cd2c96c3a00d01f5 server_cookie=482e376e83bae214 in_seq=0 out_seq=0 2024-03-28T22:40:56.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.068+0000 7f1938c00640 10 client.26987 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:40:56.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.068+0000 7f1938c00640 1 -- 172.21.15.43:0/3358445061 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564cca6e4590 con 0x564cca7fc9a0 2024-03-28T22:40:56.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.068+0000 7f1938c00640 10 client.26987 handle_command_reply: tid=0 2024-03-28T22:40:56.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.068+0000 7f1954bf3740 1 client.26987 shutdown 2024-03-28T22:40:56.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.068+0000 7f1954bf3740 1 -- 172.21.15.43:0/3358445061 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cca7813d0 msgr2=0x564cca78a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:56.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.068+0000 7f1954bf3740 1 --2- 172.21.15.43:0/3358445061 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cca7813d0 0x564cca78a7e0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f1944004af0 tx=0x7f1944005fd0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.068+0000 7f1954bf3740 1 -- 172.21.15.43:0/3358445061 shutdown_connections 2024-03-28T22:40:56.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.068+0000 7f1954bf3740 1 --2- 172.21.15.43:0/3358445061 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x564cca7fc9a0 0x564cca7fedc0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.068+0000 7f1954bf3740 1 --2- 172.21.15.43:0/3358445061 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564cca7813d0 0x564cca78a7e0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.068+0000 7f1954bf3740 1 --2- 172.21.15.43:0/3358445061 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564cca780a20 0x564cca78a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.068+0000 7f1938c00640 0 client.26987 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:40:56.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.068+0000 7f1954bf3740 1 --2- 172.21.15.43:0/3358445061 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564cca76db80 0x564cca78d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.068+0000 7f1954bf3740 1 -- 172.21.15.43:0/3358445061 >> 172.21.15.43:0/3358445061 conn(0x564cca770020 msgr2=0x564cca77fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:56.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.069+0000 7f1954bf3740 1 -- 172.21.15.43:0/3358445061 shutdown_connections 2024-03-28T22:40:56.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.069+0000 7f1954bf3740 1 -- 172.21.15.43:0/3358445061 wait complete. 2024-03-28T22:40:56.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.069+0000 7f1954bf3740 20 client.26987 trim_cache size 0 max 16384 2024-03-28T22:40:56.072 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:40:56.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.069+0000 7f1952000640 1 -- 172.21.15.43:0/3162496112 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f19300a1f70 msgr2=0x7f19300a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:56.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.069+0000 7f1952000640 1 --2- 172.21.15.43:0/3162496112 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f19300a1f70 0x7f19300a4430 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f194402c640 tx=0x7f194402e040 comp rx=0 tx=0).stop 2024-03-28T22:40:56.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.069+0000 7f1952000640 1 -- 172.21.15.43:0/3162496112 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f194c10aa10 msgr2=0x7f194c1a6970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:56.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.069+0000 7f1952000640 1 --2- 172.21.15.43:0/3162496112 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f194c10aa10 0x7f194c1a6970 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f193400d650 tx=0x7f193400db10 comp rx=0 tx=0).stop 2024-03-28T22:40:56.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.070+0000 7f1952000640 1 -- 172.21.15.43:0/3162496112 shutdown_connections 2024-03-28T22:40:56.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.070+0000 7f1952000640 1 --2- 172.21.15.43:0/3162496112 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f19300a1f70 0x7f19300a4430 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.070+0000 7f1952000640 1 --2- 172.21.15.43:0/3162496112 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f194c10aa10 0x7f194c1a6970 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.070+0000 7f1952000640 1 --2- 172.21.15.43:0/3162496112 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f194c1136a0 0x7f194c1a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.070+0000 7f1952000640 1 --2- 172.21.15.43:0/3162496112 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f194c10a0f0 0x7f194c1a6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.070+0000 7f1952000640 1 -- 172.21.15.43:0/3162496112 >> 172.21.15.43:0/3162496112 conn(0x7f194c06c830 msgr2=0x7f194c070350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:56.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.070+0000 7f1952000640 1 -- 172.21.15.43:0/3162496112 shutdown_connections 2024-03-28T22:40:56.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.070+0000 7f1952000640 1 -- 172.21.15.43:0/3162496112 wait complete. 2024-03-28T22:40:56.086 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:40:56.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.289+0000 7f9d3d400640 1 -- 172.21.15.43:0/169773012 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d38100e90 msgr2=0x7f9d38105350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:56.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.289+0000 7f9d3d400640 1 --2- 172.21.15.43:0/169773012 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d38100e90 0x7f9d38105350 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f9d24009900 tx=0x7f9d2402bd10 comp rx=0 tx=0).stop 2024-03-28T22:40:56.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.289+0000 7f9d3d400640 1 -- 172.21.15.43:0/169773012 shutdown_connections 2024-03-28T22:40:56.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.289+0000 7f9d3d400640 1 --2- 172.21.15.43:0/169773012 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9d38105970 0x7f9d38105e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.289+0000 7f9d3d400640 1 --2- 172.21.15.43:0/169773012 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d38100e90 0x7f9d38105350 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.289+0000 7f9d3d400640 1 --2- 172.21.15.43:0/169773012 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9d38100570 0x7f9d38100950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.289+0000 7f9d3d400640 1 -- 172.21.15.43:0/169773012 >> 172.21.15.43:0/169773012 conn(0x7f9d380fbfa0 msgr2=0x7f9d380fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:56.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.289+0000 7f9d3d400640 1 -- 172.21.15.43:0/169773012 shutdown_connections 2024-03-28T22:40:56.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.289+0000 7f9d3d400640 1 -- 172.21.15.43:0/169773012 wait complete. 2024-03-28T22:40:56.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.290+0000 7f9d3d400640 1 Processor -- start 2024-03-28T22:40:56.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.291+0000 7f9d3d400640 1 -- start start 2024-03-28T22:40:56.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.291+0000 7f9d3d400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9d38100570 0x7f9d3819e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.291+0000 7f9d3d400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9d38100e90 0x7f9d3819ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.291+0000 7f9d3d400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d38105970 0x7f9d381a2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.291+0000 7f9d3d400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9d38117fb0 con 0x7f9d38100570 2024-03-28T22:40:56.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.291+0000 7f9d3d400640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9d38117e30 con 0x7f9d38100e90 2024-03-28T22:40:56.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.291+0000 7f9d3d400640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9d38118130 con 0x7f9d38105970 2024-03-28T22:40:56.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.291+0000 7f9d37400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9d38100e90 0x7f9d3819ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.291+0000 7f9d37400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9d38100e90 0x7f9d3819ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59816/0 (socket says 172.21.15.43:59816) 2024-03-28T22:40:56.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.291+0000 7f9d37400640 1 -- 172.21.15.43:0/1453150483 learned_addr learned my addr 172.21.15.43:0/1453150483 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:56.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.291+0000 7f9d37e00640 1 --2- 172.21.15.43:0/1453150483 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9d38100570 0x7f9d3819e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.291+0000 7f9d3ca00640 1 --2- 172.21.15.43:0/1453150483 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d38105970 0x7f9d381a2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.292+0000 7f9d37400640 1 -- 172.21.15.43:0/1453150483 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d38105970 msgr2=0x7f9d381a2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:56.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.292+0000 7f9d37400640 1 --2- 172.21.15.43:0/1453150483 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d38105970 0x7f9d381a2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.292+0000 7f9d37400640 1 -- 172.21.15.43:0/1453150483 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9d38100570 msgr2=0x7f9d3819e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:56.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.292+0000 7f9d37400640 1 --2- 172.21.15.43:0/1453150483 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9d38100570 0x7f9d3819e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.292+0000 7f9d37400640 1 -- 172.21.15.43:0/1453150483 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d381a3490 con 0x7f9d38100e90 2024-03-28T22:40:56.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.292+0000 7f9d3ca00640 1 --2- 172.21.15.43:0/1453150483 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d38105970 0x7f9d381a2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:56.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.292+0000 7f9d37400640 1 --2- 172.21.15.43:0/1453150483 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9d38100e90 0x7f9d3819ea30 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f9d24006010 tx=0x7f9d2402cbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:56.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.292+0000 7f9d34c00640 1 -- 172.21.15.43:0/1453150483 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d24043020 con 0x7f9d38100e90 2024-03-28T22:40:56.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.292+0000 7f9d3d400640 1 -- 172.21.15.43:0/1453150483 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d381a3720 con 0x7f9d38100e90 2024-03-28T22:40:56.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.292+0000 7f9d34c00640 1 -- 172.21.15.43:0/1453150483 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d240054e0 con 0x7f9d38100e90 2024-03-28T22:40:56.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.292+0000 7f9d34c00640 1 -- 172.21.15.43:0/1453150483 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d24003aa0 con 0x7f9d38100e90 2024-03-28T22:40:56.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.293+0000 7f9d3d400640 1 -- 172.21.15.43:0/1453150483 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d381ab010 con 0x7f9d38100e90 2024-03-28T22:40:56.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.296+0000 7f9d41638740 1 Processor -- start 2024-03-28T22:40:56.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.296+0000 7f9d41638740 1 -- start start 2024-03-28T22:40:56.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.296+0000 7f9d34c00640 1 -- 172.21.15.43:0/1453150483 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f9d24046020 con 0x7f9d38100e90 2024-03-28T22:40:56.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.297+0000 7f9d41638740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dfb9a1b550 0x55dfb9a1b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.297+0000 7f9d41638740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dfb9a1be70 0x55dfb9a24510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.297+0000 7f9d41638740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfb9a24a50 0x55dfb9a26e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.297+0000 7f9d41638740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55dfb997f050 con 0x55dfb9a24a50 2024-03-28T22:40:56.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.297+0000 7f9d41638740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55dfb9927ce0 con 0x55dfb9a1be70 2024-03-28T22:40:56.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.297+0000 7f9d41638740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55dfb997b0f0 con 0x55dfb9a1b550 2024-03-28T22:40:56.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.297+0000 7f9d3ca00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dfb9a1b550 0x55dfb9a1b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.297+0000 7f9d37e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dfb9a1be70 0x55dfb9a24510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.297+0000 7f9d37e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dfb9a1be70 0x55dfb9a24510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59832/0 (socket says 172.21.15.43:59832) 2024-03-28T22:40:56.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.297+0000 7f9d3ca00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dfb9a1b550 0x55dfb9a1b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58578/0 (socket says 172.21.15.43:58578) 2024-03-28T22:40:56.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.297+0000 7f9d37e00640 1 -- 172.21.15.43:0/1049883072 learned_addr learned my addr 172.21.15.43:0/1049883072 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:56.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.297+0000 7f9d37400640 1 --2- 172.21.15.43:0/1049883072 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfb9a24a50 0x55dfb9a26e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.297+0000 7f9d37e00640 1 -- 172.21.15.43:0/1049883072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dfb9a1b550 msgr2=0x55dfb9a1b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:56.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.297+0000 7f9d37e00640 1 --2- 172.21.15.43:0/1049883072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dfb9a1b550 0x55dfb9a1b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.297+0000 7f9d37e00640 1 -- 172.21.15.43:0/1049883072 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfb9a24a50 msgr2=0x55dfb9a26e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:56.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.297+0000 7f9d37e00640 1 --2- 172.21.15.43:0/1049883072 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfb9a24a50 0x55dfb9a26e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.297+0000 7f9d37e00640 1 -- 172.21.15.43:0/1049883072 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfb96933f0 con 0x55dfb9a1be70 2024-03-28T22:40:56.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.297+0000 7f9d37400640 1 --2- 172.21.15.43:0/1049883072 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfb9a24a50 0x55dfb9a26e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:40:56.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.298+0000 7f9d37e00640 1 --2- 172.21.15.43:0/1049883072 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dfb9a1be70 0x55dfb9a24510 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f9d28009d20 tx=0x7f9d28011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=993e548731482e67 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:56.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.298+0000 7f9d2d200640 1 -- 172.21.15.43:0/1049883072 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d2801e070 con 0x55dfb9a1be70 2024-03-28T22:40:56.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.298+0000 7f9d2d200640 1 -- 172.21.15.43:0/1049883072 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d28019b10 con 0x55dfb9a1be70 2024-03-28T22:40:56.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.298+0000 7f9d2d200640 1 -- 172.21.15.43:0/1049883072 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d2800eb90 con 0x55dfb9a1be70 2024-03-28T22:40:56.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.298+0000 7f9d34c00640 1 --2- 172.21.15.43:0/1453150483 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9d1c0a1ef0 0x7f9d1c0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.299+0000 7f9d41638740 1 -- 172.21.15.43:0/1049883072 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dfb9a1be70 msgr2=0x55dfb9a24510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:56.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.299+0000 7f9d41638740 1 --2- 172.21.15.43:0/1049883072 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dfb9a1be70 0x55dfb9a24510 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f9d28009d20 tx=0x7f9d28011810 comp rx=0 tx=0).stop 2024-03-28T22:40:56.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.299+0000 7f9d34c00640 1 -- 172.21.15.43:0/1453150483 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f9d240f0460 con 0x7f9d38100e90 2024-03-28T22:40:56.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.299+0000 7f9d3ca00640 1 --2- 172.21.15.43:0/1453150483 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9d1c0a1ef0 0x7f9d1c0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.299+0000 7f9d41638740 1 -- 172.21.15.43:0/1049883072 shutdown_connections 2024-03-28T22:40:56.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.299+0000 7f9d41638740 1 --2- 172.21.15.43:0/1049883072 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dfb9a1be70 0x55dfb9a24510 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.299+0000 7f9d41638740 1 --2- 172.21.15.43:0/1049883072 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfb9a24a50 0x55dfb9a26e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.299+0000 7f9d41638740 1 --2- 172.21.15.43:0/1049883072 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dfb9a1b550 0x55dfb9a1b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.299+0000 7f9d41638740 1 -- 172.21.15.43:0/1049883072 >> 172.21.15.43:0/1049883072 conn(0x55dfb9a15c80 msgr2=0x55dfb9a16060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:56.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.299+0000 7f9d41638740 1 -- 172.21.15.43:0/1049883072 shutdown_connections 2024-03-28T22:40:56.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.299+0000 7f9d41638740 1 -- 172.21.15.43:0/1049883072 wait complete. 2024-03-28T22:40:56.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.300+0000 7f9d3ca00640 1 --2- 172.21.15.43:0/1453150483 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9d1c0a1ef0 0x7f9d1c0a43b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9d30009fd0 tx=0x7f9d30009500 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:56.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.300+0000 7f9d41638740 1 -- start start 2024-03-28T22:40:56.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.300+0000 7f9d41638740 1 Processor -- start 2024-03-28T22:40:56.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.301+0000 7f9d41638740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dfb9a1b550 0x55dfb9a21ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.301+0000 7f9d41638740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfb9a1be70 0x55dfb9a22020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.301+0000 7f9d41638740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dfb9a24a50 0x55dfb9a22560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.301+0000 7f9d41638740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55dfb9a27840 con 0x55dfb9a1be70 2024-03-28T22:40:56.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.301+0000 7f9d41638740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55dfb991d2f0 con 0x55dfb9a1b550 2024-03-28T22:40:56.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.301+0000 7f9d41638740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55dfb9a279c0 con 0x55dfb9a24a50 2024-03-28T22:40:56.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.301+0000 7f9d41638740 10 client.0 _ll_register_callbacks cb 0x55dfb9692be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:56.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.301+0000 7f9d37e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dfb9a1b550 0x55dfb9a21ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.301+0000 7f9d3ca00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dfb9a24a50 0x55dfb9a22560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.301+0000 7f9d37e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dfb9a1b550 0x55dfb9a21ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59844/0 (socket says 172.21.15.43:59844) 2024-03-28T22:40:56.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.301+0000 7f9d37400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfb9a1be70 0x55dfb9a22020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.301+0000 7f9d37e00640 1 -- 172.21.15.43:0/4186020020 learned_addr learned my addr 172.21.15.43:0/4186020020 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:56.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.301+0000 7f9d37e00640 1 -- 172.21.15.43:0/4186020020 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dfb9a24a50 msgr2=0x55dfb9a22560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:56.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.301+0000 7f9d37e00640 1 --2- 172.21.15.43:0/4186020020 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dfb9a24a50 0x55dfb9a22560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.301+0000 7f9d37e00640 1 -- 172.21.15.43:0/4186020020 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfb9a1be70 msgr2=0x55dfb9a22020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:56.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.301+0000 7f9d37e00640 1 --2- 172.21.15.43:0/4186020020 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfb9a1be70 0x55dfb9a22020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.301+0000 7f9d37e00640 1 -- 172.21.15.43:0/4186020020 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfb975d7d0 con 0x55dfb9a1b550 2024-03-28T22:40:56.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.301+0000 7f9d3ca00640 1 --2- 172.21.15.43:0/4186020020 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dfb9a24a50 0x55dfb9a22560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:56.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.301+0000 7f9d37400640 1 --2- 172.21.15.43:0/4186020020 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfb9a1be70 0x55dfb9a22020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:56.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.302+0000 7f9d37e00640 1 --2- 172.21.15.43:0/4186020020 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dfb9a1b550 0x55dfb9a21ae0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f9d28011d10 tx=0x7f9d28019660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d8041ef28d0fda5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:56.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.302+0000 7f9d21600640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:40:56.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.302+0000 7f9d21600640 1 -- 172.21.15.43:0/4186020020 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d28027020 con 0x55dfb9a1b550 2024-03-28T22:40:56.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.302+0000 7f9d21600640 1 -- 172.21.15.43:0/4186020020 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f9d2800eaf0 con 0x55dfb9a1b550 2024-03-28T22:40:56.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.302+0000 7f9d41638740 1 -- 172.21.15.43:0/4186020020 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d28019b10 con 0x55dfb9a1b550 2024-03-28T22:40:56.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.302+0000 7f9d21600640 1 -- 172.21.15.43:0/4186020020 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9d28032050 con 0x55dfb9a1b550 2024-03-28T22:40:56.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.302+0000 7f9d41638740 10 client.26999 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:56.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.302+0000 7f9d41638740 1 -- 172.21.15.43:0/4186020020 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55dfb9a33b40 con 0x55dfb9a1b550 2024-03-28T22:40:56.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.303+0000 7f9d21600640 1 -- 172.21.15.43:0/4186020020 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d28019b10 con 0x55dfb9a1b550 2024-03-28T22:40:56.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.303+0000 7f9d21600640 1 -- 172.21.15.43:0/4186020020 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f9d280046c0 con 0x55dfb9a1b550 2024-03-28T22:40:56.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.303+0000 7f9d41638740 10 client.26999 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:56.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.303+0000 7f9d41638740 10 client.26999 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:40:56.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.303+0000 7f9d41638740 20 client.26999 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:56.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.303+0000 7f9d41638740 1 --2- 172.21.15.43:0/4186020020 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55dfb9a34fa0 0x55dfb9a37390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.303+0000 7f9d41638740 4 client.26999 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:40:56.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.303+0000 7f9d41638740 1 -- 172.21.15.43:0/4186020020 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dfb9928680 con 0x55dfb9a34fa0 2024-03-28T22:40:56.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.303+0000 7f9d37400640 1 --2- 172.21.15.43:0/4186020020 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55dfb9a34fa0 0x55dfb9a37390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.304+0000 7f9d37400640 1 --2- 172.21.15.43:0/4186020020 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55dfb9a34fa0 0x55dfb9a37390 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a44e8902aed297f3 server_cookie=831b519160a518f0 in_seq=0 out_seq=0 2024-03-28T22:40:56.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.304+0000 7f9d21600640 10 client.26999 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:40:56.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.305+0000 7f9d21600640 1 -- 172.21.15.43:0/4186020020 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dfb9928680 con 0x55dfb9a34fa0 2024-03-28T22:40:56.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.305+0000 7f9d21600640 10 client.26999 handle_command_reply: tid=0 2024-03-28T22:40:56.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.305+0000 7f9d41638740 1 client.26999 shutdown 2024-03-28T22:40:56.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.305+0000 7f9d41638740 1 -- 172.21.15.43:0/4186020020 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dfb9a1b550 msgr2=0x55dfb9a21ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:56.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.305+0000 7f9d41638740 1 --2- 172.21.15.43:0/4186020020 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dfb9a1b550 0x55dfb9a21ae0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f9d28011d10 tx=0x7f9d28019660 comp rx=0 tx=0).stop 2024-03-28T22:40:56.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.305+0000 7f9d41638740 1 -- 172.21.15.43:0/4186020020 shutdown_connections 2024-03-28T22:40:56.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.305+0000 7f9d41638740 1 --2- 172.21.15.43:0/4186020020 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55dfb9a34fa0 0x55dfb9a37390 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.305+0000 7f9d41638740 1 --2- 172.21.15.43:0/4186020020 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dfb9a24a50 0x55dfb9a22560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.305+0000 7f9d41638740 1 --2- 172.21.15.43:0/4186020020 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfb9a1be70 0x55dfb9a22020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.305+0000 7f9d41638740 1 --2- 172.21.15.43:0/4186020020 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dfb9a1b550 0x55dfb9a21ae0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.305+0000 7f9d41638740 1 -- 172.21.15.43:0/4186020020 >> 172.21.15.43:0/4186020020 conn(0x55dfb9a15c80 msgr2=0x55dfb9a179d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:56.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.305+0000 7f9d21600640 0 client.26999 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:40:56.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.305+0000 7f9d41638740 1 -- 172.21.15.43:0/4186020020 shutdown_connections 2024-03-28T22:40:56.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.305+0000 7f9d41638740 1 -- 172.21.15.43:0/4186020020 wait complete. 2024-03-28T22:40:56.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.305+0000 7f9d41638740 20 client.26999 trim_cache size 0 max 16384 2024-03-28T22:40:56.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.323+0000 7f9d41638740 1 Processor -- start 2024-03-28T22:40:56.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.323+0000 7f9d41638740 1 -- start start 2024-03-28T22:40:56.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.324+0000 7f9d41638740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfb9a1b550 0x55dfb9a10c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.324+0000 7f9d41638740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dfb9a1be70 0x55dfb9a305e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.324+0000 7f9d41638740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dfb9a24a50 0x55dfb9a22380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.324+0000 7f9d41638740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9d381b4fa0 con 0x55dfb9a1b550 2024-03-28T22:40:56.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.324+0000 7f9d41638740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9d381b6ae0 con 0x55dfb9a24a50 2024-03-28T22:40:56.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.324+0000 7f9d41638740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9d381b5670 con 0x55dfb9a1be70 2024-03-28T22:40:56.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.324+0000 7f9d37400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dfb9a1be70 0x55dfb9a305e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.324+0000 7f9d3ca00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dfb9a24a50 0x55dfb9a22380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.324+0000 7f9d37400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dfb9a1be70 0x55dfb9a305e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58600/0 (socket says 172.21.15.43:58600) 2024-03-28T22:40:56.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.324+0000 7f9d37400640 1 -- 172.21.15.43:0/3420682910 learned_addr learned my addr 172.21.15.43:0/3420682910 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:56.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.324+0000 7f9d3ca00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dfb9a24a50 0x55dfb9a22380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59860/0 (socket says 172.21.15.43:59860) 2024-03-28T22:40:56.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.324+0000 7f9d37e00640 1 --2- 172.21.15.43:0/3420682910 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfb9a1b550 0x55dfb9a10c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.324+0000 7f9d37400640 1 -- 172.21.15.43:0/3420682910 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dfb9a24a50 msgr2=0x55dfb9a22380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:56.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.324+0000 7f9d37400640 1 --2- 172.21.15.43:0/3420682910 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dfb9a24a50 0x55dfb9a22380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.324+0000 7f9d37400640 1 -- 172.21.15.43:0/3420682910 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfb9a1b550 msgr2=0x55dfb9a10c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:56.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.324+0000 7f9d37400640 1 --2- 172.21.15.43:0/3420682910 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfb9a1b550 0x55dfb9a10c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.324+0000 7f9d37400640 1 -- 172.21.15.43:0/3420682910 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfb9a20a90 con 0x55dfb9a1be70 2024-03-28T22:40:56.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.325+0000 7f9d3ca00640 1 --2- 172.21.15.43:0/3420682910 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dfb9a24a50 0x55dfb9a22380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:56.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.325+0000 7f9d37400640 1 --2- 172.21.15.43:0/3420682910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dfb9a1be70 0x55dfb9a305e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f9d240bcb50 tx=0x7f9d2402ce60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e2b908de392d06c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:56.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.325+0000 7f9d23e00640 1 -- 172.21.15.43:0/3420682910 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d240bc700 con 0x55dfb9a1be70 2024-03-28T22:40:56.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.325+0000 7f9d23e00640 1 -- 172.21.15.43:0/3420682910 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d24005690 con 0x55dfb9a1be70 2024-03-28T22:40:56.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.325+0000 7f9d23e00640 1 -- 172.21.15.43:0/3420682910 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d240bc520 con 0x55dfb9a1be70 2024-03-28T22:40:56.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.326+0000 7f9d41638740 1 -- 172.21.15.43:0/3420682910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dfb9a1be70 msgr2=0x55dfb9a305e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:56.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.326+0000 7f9d41638740 1 --2- 172.21.15.43:0/3420682910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dfb9a1be70 0x55dfb9a305e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f9d240bcb50 tx=0x7f9d2402ce60 comp rx=0 tx=0).stop 2024-03-28T22:40:56.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.326+0000 7f9d41638740 1 -- 172.21.15.43:0/3420682910 shutdown_connections 2024-03-28T22:40:56.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.326+0000 7f9d41638740 1 --2- 172.21.15.43:0/3420682910 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dfb9a24a50 0x55dfb9a22380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.326+0000 7f9d41638740 1 --2- 172.21.15.43:0/3420682910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dfb9a1be70 0x55dfb9a305e0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.326+0000 7f9d41638740 1 --2- 172.21.15.43:0/3420682910 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfb9a1b550 0x55dfb9a10c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.326+0000 7f9d41638740 1 -- 172.21.15.43:0/3420682910 >> 172.21.15.43:0/3420682910 conn(0x55dfb9a15c80 msgr2=0x55dfb9a14dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:56.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.326+0000 7f9d41638740 1 -- 172.21.15.43:0/3420682910 shutdown_connections 2024-03-28T22:40:56.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.326+0000 7f9d41638740 1 -- 172.21.15.43:0/3420682910 wait complete. 2024-03-28T22:40:56.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.327+0000 7f9d41638740 1 -- start start 2024-03-28T22:40:56.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.327+0000 7f9d41638740 1 Processor -- start 2024-03-28T22:40:56.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.328+0000 7f9d41638740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dfb9a1b550 0x55dfb9a1e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.328+0000 7f9d41638740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dfb9a1be70 0x55dfb9a1eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.328+0000 7f9d41638740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfb9a24a50 0x55dfb9a1f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.328+0000 7f9d41638740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9d381b6410 con 0x55dfb9a24a50 2024-03-28T22:40:56.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.328+0000 7f9d41638740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9d381b5d40 con 0x55dfb9a1be70 2024-03-28T22:40:56.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.328+0000 7f9d41638740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9d381b2f80 con 0x55dfb9a1b550 2024-03-28T22:40:56.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.328+0000 7f9d41638740 10 client.0 _ll_register_callbacks cb 0x7f9d2801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:56.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.328+0000 7f9d37e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dfb9a1b550 0x55dfb9a1e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.328+0000 7f9d37400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dfb9a1be70 0x55dfb9a1eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.328+0000 7f9d3ca00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfb9a24a50 0x55dfb9a1f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.328+0000 7f9d37400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dfb9a1be70 0x55dfb9a1eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59862/0 (socket says 172.21.15.43:59862) 2024-03-28T22:40:56.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.328+0000 7f9d37e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dfb9a1b550 0x55dfb9a1e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58602/0 (socket says 172.21.15.43:58602) 2024-03-28T22:40:56.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.328+0000 7f9d37400640 1 -- 172.21.15.43:0/2679863515 learned_addr learned my addr 172.21.15.43:0/2679863515 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:56.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.328+0000 7f9d37400640 1 -- 172.21.15.43:0/2679863515 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dfb9a1b550 msgr2=0x55dfb9a1e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:56.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.328+0000 7f9d37400640 1 --2- 172.21.15.43:0/2679863515 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dfb9a1b550 0x55dfb9a1e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.328+0000 7f9d37400640 1 -- 172.21.15.43:0/2679863515 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfb9a24a50 msgr2=0x55dfb9a1f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:56.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.328+0000 7f9d37400640 1 --2- 172.21.15.43:0/2679863515 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfb9a24a50 0x55dfb9a1f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.328+0000 7f9d37400640 1 -- 172.21.15.43:0/2679863515 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfb9a1bca0 con 0x55dfb9a1be70 2024-03-28T22:40:56.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.328+0000 7f9d37e00640 1 --2- 172.21.15.43:0/2679863515 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dfb9a1b550 0x55dfb9a1e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:56.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.329+0000 7f9d37400640 1 --2- 172.21.15.43:0/2679863515 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dfb9a1be70 0x55dfb9a1eb50 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f9d24005940 tx=0x7f9d24004ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aaf9b804355bf9b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:56.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.329+0000 7f9d21600640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:40:56.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.329+0000 7f9d21600640 1 -- 172.21.15.43:0/2679863515 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d240bc520 con 0x55dfb9a1be70 2024-03-28T22:40:56.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.329+0000 7f9d21600640 1 -- 172.21.15.43:0/2679863515 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f9d24004250 con 0x55dfb9a1be70 2024-03-28T22:40:56.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.329+0000 7f9d41638740 1 -- 172.21.15.43:0/2679863515 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55dfb998c590 con 0x55dfb9a1be70 2024-03-28T22:40:56.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.329+0000 7f9d21600640 1 -- 172.21.15.43:0/2679863515 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9d240bc700 con 0x55dfb9a1be70 2024-03-28T22:40:56.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.329+0000 7f9d41638740 10 client.27005 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:56.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.329+0000 7f9d41638740 1 -- 172.21.15.43:0/2679863515 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55dfb9a9b240 con 0x55dfb9a1be70 2024-03-28T22:40:56.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.329+0000 7f9d21600640 1 -- 172.21.15.43:0/2679863515 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55dfb998c590 con 0x55dfb9a1be70 2024-03-28T22:40:56.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.330+0000 7f9d21600640 1 -- 172.21.15.43:0/2679863515 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f9d24003c40 con 0x55dfb9a1be70 2024-03-28T22:40:56.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.330+0000 7f9d41638740 10 client.27005 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:56.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.330+0000 7f9d41638740 10 client.27005 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:40:56.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.330+0000 7f9d41638740 20 client.27005 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:56.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.330+0000 7f9d41638740 1 --2- 172.21.15.43:0/2679863515 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55dfb9a9b9e0 0x55dfb9a9de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.330+0000 7f9d41638740 4 client.27005 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:40:56.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.330+0000 7f9d41638740 1 -- 172.21.15.43:0/2679863515 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dfb99868e0 con 0x55dfb9a9b9e0 2024-03-28T22:40:56.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.330+0000 7f9d37e00640 1 --2- 172.21.15.43:0/2679863515 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55dfb9a9b9e0 0x55dfb9a9de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.330+0000 7f9d37e00640 1 --2- 172.21.15.43:0/2679863515 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55dfb9a9b9e0 0x55dfb9a9de00 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=596c9c58ed681f0d server_cookie=e1300c232e1d243d in_seq=0 out_seq=0 2024-03-28T22:40:56.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.330+0000 7f9d21600640 10 client.27005 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:40:56.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.331+0000 7f9d21600640 1 -- 172.21.15.43:0/2679863515 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dfb99868e0 con 0x55dfb9a9b9e0 2024-03-28T22:40:56.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.331+0000 7f9d21600640 10 client.27005 handle_command_reply: tid=0 2024-03-28T22:40:56.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.331+0000 7f9d41638740 1 client.27005 shutdown 2024-03-28T22:40:56.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.331+0000 7f9d41638740 1 -- 172.21.15.43:0/2679863515 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dfb9a1be70 msgr2=0x55dfb9a1eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:56.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.331+0000 7f9d41638740 1 --2- 172.21.15.43:0/2679863515 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dfb9a1be70 0x55dfb9a1eb50 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f9d24005940 tx=0x7f9d24004ee0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.331+0000 7f9d41638740 1 -- 172.21.15.43:0/2679863515 shutdown_connections 2024-03-28T22:40:56.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.331+0000 7f9d41638740 1 --2- 172.21.15.43:0/2679863515 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55dfb9a9b9e0 0x55dfb9a9de00 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.331+0000 7f9d41638740 1 --2- 172.21.15.43:0/2679863515 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dfb9a1be70 0x55dfb9a1eb50 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.331+0000 7f9d41638740 1 --2- 172.21.15.43:0/2679863515 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dfb9a24a50 0x55dfb9a1f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.331+0000 7f9d41638740 1 --2- 172.21.15.43:0/2679863515 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dfb9a1b550 0x55dfb9a1e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.331+0000 7f9d41638740 1 -- 172.21.15.43:0/2679863515 >> 172.21.15.43:0/2679863515 conn(0x55dfb9a15c80 msgr2=0x55dfb9a16cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:56.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.331+0000 7f9d21600640 0 client.27005 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:40:56.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.331+0000 7f9d41638740 1 -- 172.21.15.43:0/2679863515 shutdown_connections 2024-03-28T22:40:56.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.331+0000 7f9d41638740 1 -- 172.21.15.43:0/2679863515 wait complete. 2024-03-28T22:40:56.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.331+0000 7f9d41638740 20 client.27005 trim_cache size 0 max 16384 2024-03-28T22:40:56.335 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:40:56.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.332+0000 7f9d3d400640 1 -- 172.21.15.43:0/1453150483 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9d1c0a1ef0 msgr2=0x7f9d1c0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:56.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.332+0000 7f9d3d400640 1 --2- 172.21.15.43:0/1453150483 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9d1c0a1ef0 0x7f9d1c0a43b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9d30009fd0 tx=0x7f9d30009500 comp rx=0 tx=0).stop 2024-03-28T22:40:56.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.332+0000 7f9d3d400640 1 -- 172.21.15.43:0/1453150483 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9d38100e90 msgr2=0x7f9d3819ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:56.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.332+0000 7f9d3d400640 1 --2- 172.21.15.43:0/1453150483 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9d38100e90 0x7f9d3819ea30 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f9d24006010 tx=0x7f9d2402cbb0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.332+0000 7f9d3d400640 1 -- 172.21.15.43:0/1453150483 shutdown_connections 2024-03-28T22:40:56.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.332+0000 7f9d3d400640 1 --2- 172.21.15.43:0/1453150483 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9d1c0a1ef0 0x7f9d1c0a43b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.332+0000 7f9d3d400640 1 --2- 172.21.15.43:0/1453150483 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9d38100e90 0x7f9d3819ea30 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.332+0000 7f9d3d400640 1 --2- 172.21.15.43:0/1453150483 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9d38105970 0x7f9d381a2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.332+0000 7f9d3d400640 1 --2- 172.21.15.43:0/1453150483 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9d38100570 0x7f9d3819e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.332+0000 7f9d3d400640 1 -- 172.21.15.43:0/1453150483 >> 172.21.15.43:0/1453150483 conn(0x7f9d380fbfa0 msgr2=0x7f9d38103320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:56.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.332+0000 7f9d3d400640 1 -- 172.21.15.43:0/1453150483 shutdown_connections 2024-03-28T22:40:56.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.332+0000 7f9d3d400640 1 -- 172.21.15.43:0/1453150483 wait complete. 2024-03-28T22:40:56.347 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:56 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1935535343' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:40:56.351 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:40:56.355 INFO:tasks.workunit.client.0.smithi043.stdout: 1 227399 8.43 MB/sec execute 27 sec latency 11.418 ms 2024-03-28T22:40:56.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:56 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1935535343' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:40:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.567+0000 7f03e3000640 1 -- 172.21.15.43:0/3785992524 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03dc0765a0 msgr2=0x7f03dc111f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.567+0000 7f03e3000640 1 --2- 172.21.15.43:0/3785992524 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03dc0765a0 0x7f03dc111f30 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f03c8009930 tx=0x7f03c802bc30 comp rx=0 tx=0).stop 2024-03-28T22:40:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.567+0000 7f03e3000640 1 -- 172.21.15.43:0/3785992524 shutdown_connections 2024-03-28T22:40:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.567+0000 7f03e3000640 1 --2- 172.21.15.43:0/3785992524 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03dc112470 0x7f03dc114860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.567+0000 7f03e3000640 1 --2- 172.21.15.43:0/3785992524 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03dc0765a0 0x7f03dc111f30 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.567+0000 7f03e3000640 1 --2- 172.21.15.43:0/3785992524 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03dc075c80 0x7f03dc076060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.567+0000 7f03e3000640 1 -- 172.21.15.43:0/3785992524 >> 172.21.15.43:0/3785992524 conn(0x7f03dc0ff260 msgr2=0x7f03dc101680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:56.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.567+0000 7f03e3000640 1 -- 172.21.15.43:0/3785992524 shutdown_connections 2024-03-28T22:40:56.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.567+0000 7f03e3000640 1 -- 172.21.15.43:0/3785992524 wait complete. 2024-03-28T22:40:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.568+0000 7f03e3000640 1 Processor -- start 2024-03-28T22:40:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.569+0000 7f03e3000640 1 -- start start 2024-03-28T22:40:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.569+0000 7f03e3000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03dc075c80 0x7f03dc19e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.569+0000 7f03e3000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03dc0765a0 0x7f03dc19ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.569+0000 7f03e3000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03dc112470 0x7f03dc1a2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.569+0000 7f03e3000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f03dc118020 con 0x7f03dc075c80 2024-03-28T22:40:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.569+0000 7f03e3000640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f03dc117ea0 con 0x7f03dc112470 2024-03-28T22:40:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.569+0000 7f03e3000640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f03dc1181a0 con 0x7f03dc0765a0 2024-03-28T22:40:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.570+0000 7f03e1200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03dc0765a0 0x7f03dc19ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.570+0000 7f03e2600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03dc112470 0x7f03dc1a2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.570+0000 7f03e1200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03dc0765a0 0x7f03dc19ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58620/0 (socket says 172.21.15.43:58620) 2024-03-28T22:40:56.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.570+0000 7f03e1200640 1 -- 172.21.15.43:0/3989473149 learned_addr learned my addr 172.21.15.43:0/3989473149 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:56.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.570+0000 7f03e1c00640 1 --2- 172.21.15.43:0/3989473149 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03dc075c80 0x7f03dc19e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.570+0000 7f03e1200640 1 -- 172.21.15.43:0/3989473149 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03dc112470 msgr2=0x7f03dc1a2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:56.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.570+0000 7f03e1200640 1 --2- 172.21.15.43:0/3989473149 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03dc112470 0x7f03dc1a2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.570+0000 7f03e1200640 1 -- 172.21.15.43:0/3989473149 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03dc075c80 msgr2=0x7f03dc19e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:56.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.570+0000 7f03e1200640 1 --2- 172.21.15.43:0/3989473149 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03dc075c80 0x7f03dc19e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.570+0000 7f03e1200640 1 -- 172.21.15.43:0/3989473149 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03dc1a3530 con 0x7f03dc0765a0 2024-03-28T22:40:56.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.570+0000 7f03e2600640 1 --2- 172.21.15.43:0/3989473149 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03dc112470 0x7f03dc1a2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:56.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.570+0000 7f03e1c00640 1 --2- 172.21.15.43:0/3989473149 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03dc075c80 0x7f03dc19e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:56.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.571+0000 7f03e1200640 1 --2- 172.21.15.43:0/3989473149 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03dc0765a0 0x7f03dc19ea80 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f03c8009900 tx=0x7f03c8004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:56.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.571+0000 7f03da000640 1 -- 172.21.15.43:0/3989473149 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c8043020 con 0x7f03dc0765a0 2024-03-28T22:40:56.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.571+0000 7f03da000640 1 -- 172.21.15.43:0/3989473149 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03c80054e0 con 0x7f03dc0765a0 2024-03-28T22:40:56.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.571+0000 7f03da000640 1 -- 172.21.15.43:0/3989473149 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c8003bb0 con 0x7f03dc0765a0 2024-03-28T22:40:56.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.571+0000 7f03e3000640 1 -- 172.21.15.43:0/3989473149 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03dc1a37c0 con 0x7f03dc0765a0 2024-03-28T22:40:56.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.571+0000 7f03e3000640 1 -- 172.21.15.43:0/3989473149 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03dc1ab0b0 con 0x7f03dc0765a0 2024-03-28T22:40:56.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.574+0000 7f03da000640 1 -- 172.21.15.43:0/3989473149 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f03c8046020 con 0x7f03dc0765a0 2024-03-28T22:40:56.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.574+0000 7f03e73d4740 1 Processor -- start 2024-03-28T22:40:56.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.575+0000 7f03e73d4740 1 -- start start 2024-03-28T22:40:56.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.575+0000 7f03e73d4740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ab033cb550 0x55ab033cb930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.575+0000 7f03e73d4740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ab033cbe70 0x55ab033d4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.575+0000 7f03e73d4740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ab033d4a50 0x55ab033d6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.575+0000 7f03e73d4740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ab0332f050 con 0x55ab033cbe70 2024-03-28T22:40:56.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.575+0000 7f03e73d4740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55ab032d7ce0 con 0x55ab033d4a50 2024-03-28T22:40:56.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.575+0000 7f03e73d4740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55ab0332b0f0 con 0x55ab033cb550 2024-03-28T22:40:56.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.575+0000 7f03e2600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ab033cb550 0x55ab033cb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.575+0000 7f03e1200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ab033d4a50 0x55ab033d6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.575+0000 7f03e2600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ab033cb550 0x55ab033cb930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58636/0 (socket says 172.21.15.43:58636) 2024-03-28T22:40:56.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.575+0000 7f03e1c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ab033cbe70 0x55ab033d4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.575+0000 7f03e2600640 1 -- 172.21.15.43:0/542347870 learned_addr learned my addr 172.21.15.43:0/542347870 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.576+0000 7f03e2600640 1 -- 172.21.15.43:0/542347870 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ab033d4a50 msgr2=0x55ab033d6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.576+0000 7f03e2600640 1 --2- 172.21.15.43:0/542347870 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ab033d4a50 0x55ab033d6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.576+0000 7f03e2600640 1 -- 172.21.15.43:0/542347870 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ab033cbe70 msgr2=0x55ab033d4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.576+0000 7f03e2600640 1 --2- 172.21.15.43:0/542347870 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ab033cbe70 0x55ab033d4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.576+0000 7f03e2600640 1 -- 172.21.15.43:0/542347870 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab030433f0 con 0x55ab033cb550 2024-03-28T22:40:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.576+0000 7f03da000640 1 --2- 172.21.15.43:0/3989473149 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f03c40a1ef0 0x7f03c40a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.576+0000 7f03e1200640 1 --2- 172.21.15.43:0/542347870 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ab033d4a50 0x55ab033d6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.576+0000 7f03da000640 1 -- 172.21.15.43:0/3989473149 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f03c80f04c0 con 0x7f03dc0765a0 2024-03-28T22:40:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.576+0000 7f03e2600640 1 --2- 172.21.15.43:0/542347870 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ab033cb550 0x55ab033cb930 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f03d4011760 tx=0x7f03d4011c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53f1eb310f496bde server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.576+0000 7f03e1c00640 1 --2- 172.21.15.43:0/3989473149 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f03c40a1ef0 0x7f03c40a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.576+0000 7f03cea00640 1 -- 172.21.15.43:0/542347870 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d4004d20 con 0x55ab033cb550 2024-03-28T22:40:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.576+0000 7f03cea00640 1 -- 172.21.15.43:0/542347870 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03d400f9a0 con 0x55ab033cb550 2024-03-28T22:40:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.576+0000 7f03cea00640 1 -- 172.21.15.43:0/542347870 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d4004590 con 0x55ab033cb550 2024-03-28T22:40:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.577+0000 7f03e73d4740 1 -- 172.21.15.43:0/542347870 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ab033cb550 msgr2=0x55ab033cb930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.577+0000 7f03e1c00640 1 --2- 172.21.15.43:0/3989473149 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f03c40a1ef0 0x7f03c40a43b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x55ab033cb020 tx=0x7f03d0005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.577+0000 7f03e73d4740 1 --2- 172.21.15.43:0/542347870 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ab033cb550 0x55ab033cb930 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f03d4011760 tx=0x7f03d4011c20 comp rx=0 tx=0).stop 2024-03-28T22:40:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.577+0000 7f03e73d4740 1 -- 172.21.15.43:0/542347870 shutdown_connections 2024-03-28T22:40:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.577+0000 7f03e73d4740 1 --2- 172.21.15.43:0/542347870 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ab033d4a50 0x55ab033d6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.577+0000 7f03e73d4740 1 --2- 172.21.15.43:0/542347870 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ab033cbe70 0x55ab033d4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.577+0000 7f03e73d4740 1 --2- 172.21.15.43:0/542347870 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ab033cb550 0x55ab033cb930 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.577+0000 7f03e73d4740 1 -- 172.21.15.43:0/542347870 >> 172.21.15.43:0/542347870 conn(0x55ab033c5c80 msgr2=0x55ab033c6060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.577+0000 7f03e73d4740 1 -- 172.21.15.43:0/542347870 shutdown_connections 2024-03-28T22:40:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.577+0000 7f03e73d4740 1 -- 172.21.15.43:0/542347870 wait complete. 2024-03-28T22:40:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.578+0000 7f03e73d4740 1 -- start start 2024-03-28T22:40:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.578+0000 7f03e73d4740 1 Processor -- start 2024-03-28T22:40:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.579+0000 7f03e73d4740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ab033cb550 0x55ab033d1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.579+0000 7f03e73d4740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ab033cbe70 0x55ab033d2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.579+0000 7f03e73d4740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ab033d4a50 0x55ab033d25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.579+0000 7f03e73d4740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ab033d7840 con 0x55ab033cb550 2024-03-28T22:40:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.579+0000 7f03e73d4740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55ab032cd2f0 con 0x55ab033cbe70 2024-03-28T22:40:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.579+0000 7f03e73d4740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55ab033d79c0 con 0x55ab033d4a50 2024-03-28T22:40:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.579+0000 7f03e73d4740 10 client.0 _ll_register_callbacks cb 0x55ab03042be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.579+0000 7f03e2600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ab033d4a50 0x55ab033d25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.579+0000 7f03e1c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ab033cb550 0x55ab033d1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.579+0000 7f03e2600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ab033d4a50 0x55ab033d25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58648/0 (socket says 172.21.15.43:58648) 2024-03-28T22:40:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.579+0000 7f03e1200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ab033cbe70 0x55ab033d2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.579+0000 7f03e2600640 1 -- 172.21.15.43:0/638950854 learned_addr learned my addr 172.21.15.43:0/638950854 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.579+0000 7f03e2600640 1 -- 172.21.15.43:0/638950854 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ab033cbe70 msgr2=0x55ab033d2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.579+0000 7f03e2600640 1 --2- 172.21.15.43:0/638950854 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ab033cbe70 0x55ab033d2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.579+0000 7f03e2600640 1 -- 172.21.15.43:0/638950854 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ab033cb550 msgr2=0x55ab033d1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.579+0000 7f03e2600640 1 --2- 172.21.15.43:0/638950854 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ab033cb550 0x55ab033d1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.579+0000 7f03e2600640 1 -- 172.21.15.43:0/638950854 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab0310d7d0 con 0x55ab033d4a50 2024-03-28T22:40:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.579+0000 7f03e1c00640 1 --2- 172.21.15.43:0/638950854 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ab033cb550 0x55ab033d1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:40:56.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.580+0000 7f03e2600640 1 --2- 172.21.15.43:0/638950854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ab033d4a50 0x55ab033d25d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f03d4000f80 tx=0x7f03d4003c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40dc981b219b11ef server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:56.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.580+0000 7f03c3400640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:40:56.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.580+0000 7f03c3400640 1 -- 172.21.15.43:0/638950854 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d4005000 con 0x55ab033d4a50 2024-03-28T22:40:56.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.580+0000 7f03e73d4740 1 -- 172.21.15.43:0/638950854 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03d400f9a0 con 0x55ab033d4a50 2024-03-28T22:40:56.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.580+0000 7f03c3400640 1 -- 172.21.15.43:0/638950854 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f03d40051a0 con 0x55ab033d4a50 2024-03-28T22:40:56.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.580+0000 7f03c3400640 1 -- 172.21.15.43:0/638950854 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f03d4032050 con 0x55ab033d4a50 2024-03-28T22:40:56.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.580+0000 7f03e73d4740 10 client.26971 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:56.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.580+0000 7f03e73d4740 1 -- 172.21.15.43:0/638950854 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ab033e3c10 con 0x55ab033d4a50 2024-03-28T22:40:56.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.581+0000 7f03c3400640 1 -- 172.21.15.43:0/638950854 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03d400f9a0 con 0x55ab033d4a50 2024-03-28T22:40:56.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.581+0000 7f03c3400640 1 -- 172.21.15.43:0/638950854 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f03d4018070 con 0x55ab033d4a50 2024-03-28T22:40:56.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.581+0000 7f03e73d4740 10 client.26971 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:56.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.581+0000 7f03e73d4740 10 client.26971 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:40:56.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.581+0000 7f03e73d4740 20 client.26971 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:56.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.581+0000 7f03e73d4740 1 --2- 172.21.15.43:0/638950854 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55ab033e4fc0 0x55ab033e73b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.581+0000 7f03e73d4740 4 client.26971 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:40:56.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.581+0000 7f03e73d4740 1 -- 172.21.15.43:0/638950854 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ab032d8680 con 0x55ab033e4fc0 2024-03-28T22:40:56.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.581+0000 7f03e1c00640 1 --2- 172.21.15.43:0/638950854 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55ab033e4fc0 0x55ab033e73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.582+0000 7f03e1c00640 1 --2- 172.21.15.43:0/638950854 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55ab033e4fc0 0x55ab033e73b0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=674fbd722affcb2b server_cookie=efed0c6227df8291 in_seq=0 out_seq=0 2024-03-28T22:40:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.582+0000 7f03c3400640 10 client.26971 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.584+0000 7f03c3400640 1 -- 172.21.15.43:0/638950854 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ab032d8680 con 0x55ab033e4fc0 2024-03-28T22:40:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.584+0000 7f03c3400640 10 client.26971 handle_command_reply: tid=0 2024-03-28T22:40:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.584+0000 7f03e73d4740 1 client.26971 shutdown 2024-03-28T22:40:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.584+0000 7f03e73d4740 1 -- 172.21.15.43:0/638950854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ab033d4a50 msgr2=0x55ab033d25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.584+0000 7f03e73d4740 1 --2- 172.21.15.43:0/638950854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ab033d4a50 0x55ab033d25d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f03d4000f80 tx=0x7f03d4003c60 comp rx=0 tx=0).stop 2024-03-28T22:40:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.584+0000 7f03e73d4740 1 -- 172.21.15.43:0/638950854 shutdown_connections 2024-03-28T22:40:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.584+0000 7f03e73d4740 1 --2- 172.21.15.43:0/638950854 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55ab033e4fc0 0x55ab033e73b0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.584+0000 7f03e73d4740 1 --2- 172.21.15.43:0/638950854 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ab033cbe70 0x55ab033d2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.584+0000 7f03e73d4740 1 --2- 172.21.15.43:0/638950854 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ab033d4a50 0x55ab033d25d0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.584+0000 7f03e73d4740 1 --2- 172.21.15.43:0/638950854 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ab033cb550 0x55ab033d1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.584+0000 7f03e73d4740 1 -- 172.21.15.43:0/638950854 >> 172.21.15.43:0/638950854 conn(0x55ab033c5c80 msgr2=0x55ab033c79d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:56.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.584+0000 7f03c3400640 0 client.26971 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:56.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.585+0000 7f03e73d4740 1 -- 172.21.15.43:0/638950854 shutdown_connections 2024-03-28T22:40:56.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.585+0000 7f03e73d4740 1 -- 172.21.15.43:0/638950854 wait complete. 2024-03-28T22:40:56.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.585+0000 7f03e73d4740 20 client.26971 trim_cache size 0 max 16384 2024-03-28T22:40:56.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.599+0000 7f03e73d4740 1 Processor -- start 2024-03-28T22:40:56.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.600+0000 7f03e73d4740 1 -- start start 2024-03-28T22:40:56.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.600+0000 7f03e73d4740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ab033cb550 0x55ab033d4280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.600+0000 7f03e73d4740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ab033cbe70 0x55ab033e05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.600+0000 7f03e73d4740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ab033d4a50 0x55ab033ecb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.600+0000 7f03e73d4740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f03dc1b5020 con 0x55ab033cb550 2024-03-28T22:40:56.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.600+0000 7f03e73d4740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f03dc1b6b60 con 0x55ab033cbe70 2024-03-28T22:40:56.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.600+0000 7f03e73d4740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f03dc1b56f0 con 0x55ab033d4a50 2024-03-28T22:40:56.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.600+0000 7f03e1200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ab033cbe70 0x55ab033e05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.600+0000 7f03e2600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ab033d4a50 0x55ab033ecb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.600+0000 7f03e1c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ab033cb550 0x55ab033d4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.600+0000 7f03e2600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ab033d4a50 0x55ab033ecb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58660/0 (socket says 172.21.15.43:58660) 2024-03-28T22:40:56.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.600+0000 7f03e1200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ab033cbe70 0x55ab033e05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59896/0 (socket says 172.21.15.43:59896) 2024-03-28T22:40:56.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.600+0000 7f03e2600640 1 -- 172.21.15.43:0/221220322 learned_addr learned my addr 172.21.15.43:0/221220322 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:56.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.601+0000 7f03e2600640 1 -- 172.21.15.43:0/221220322 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ab033cbe70 msgr2=0x55ab033e05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:56.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.601+0000 7f03e2600640 1 --2- 172.21.15.43:0/221220322 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ab033cbe70 0x55ab033e05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.601+0000 7f03e2600640 1 -- 172.21.15.43:0/221220322 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ab033cb550 msgr2=0x55ab033d4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:56.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.601+0000 7f03e2600640 1 --2- 172.21.15.43:0/221220322 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ab033cb550 0x55ab033d4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.601+0000 7f03e2600640 1 -- 172.21.15.43:0/221220322 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab033e7720 con 0x55ab033d4a50 2024-03-28T22:40:56.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.601+0000 7f03e1200640 1 --2- 172.21.15.43:0/221220322 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ab033cbe70 0x55ab033e05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:56.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.601+0000 7f03e2600640 1 --2- 172.21.15.43:0/221220322 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ab033d4a50 0x55ab033ecb70 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f03d4009480 tx=0x7f03d40021d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50c1431fbd3820af server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:56.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.601+0000 7f03ce000640 1 -- 172.21.15.43:0/221220322 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d4003cd0 con 0x55ab033d4a50 2024-03-28T22:40:56.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.601+0000 7f03ce000640 1 -- 172.21.15.43:0/221220322 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03d4029070 con 0x55ab033d4a50 2024-03-28T22:40:56.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.601+0000 7f03ce000640 1 -- 172.21.15.43:0/221220322 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d4010430 con 0x55ab033d4a50 2024-03-28T22:40:56.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.602+0000 7f03e73d4740 1 -- 172.21.15.43:0/221220322 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ab033d4a50 msgr2=0x55ab033ecb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:56.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.602+0000 7f03e73d4740 1 --2- 172.21.15.43:0/221220322 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ab033d4a50 0x55ab033ecb70 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f03d4009480 tx=0x7f03d40021d0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.602+0000 7f03e73d4740 1 -- 172.21.15.43:0/221220322 shutdown_connections 2024-03-28T22:40:56.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.602+0000 7f03e73d4740 1 --2- 172.21.15.43:0/221220322 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ab033cbe70 0x55ab033e05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.602+0000 7f03e73d4740 1 --2- 172.21.15.43:0/221220322 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ab033d4a50 0x55ab033ecb70 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.602+0000 7f03e73d4740 1 --2- 172.21.15.43:0/221220322 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ab033cb550 0x55ab033d4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.602+0000 7f03e73d4740 1 -- 172.21.15.43:0/221220322 >> 172.21.15.43:0/221220322 conn(0x55ab033c5c80 msgr2=0x55ab033c4dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:56.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.602+0000 7f03e73d4740 1 -- 172.21.15.43:0/221220322 shutdown_connections 2024-03-28T22:40:56.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.602+0000 7f03e73d4740 1 -- 172.21.15.43:0/221220322 wait complete. 2024-03-28T22:40:56.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.602+0000 7f03e73d4740 1 -- start start 2024-03-28T22:40:56.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.603+0000 7f03e73d4740 1 Processor -- start 2024-03-28T22:40:56.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.603+0000 7f03e73d4740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ab033cb550 0x55ab033dcbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.603+0000 7f03e73d4740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ab033cbe70 0x55ab033d7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.603+0000 7f03e73d4740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ab033d4a50 0x55ab033d8520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.603+0000 7f03e73d4740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f03dc1b6490 con 0x55ab033cb550 2024-03-28T22:40:56.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.603+0000 7f03e73d4740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f03dc1b5dc0 con 0x55ab033cbe70 2024-03-28T22:40:56.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.603+0000 7f03e73d4740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f03dc1b3000 con 0x55ab033d4a50 2024-03-28T22:40:56.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.603+0000 7f03e73d4740 10 client.0 _ll_register_callbacks cb 0x7f03d0009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:56.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.603+0000 7f03e1c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ab033cb550 0x55ab033dcbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.603+0000 7f03e1200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ab033cbe70 0x55ab033d7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.603+0000 7f03e2600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ab033d4a50 0x55ab033d8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.603+0000 7f03e2600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ab033d4a50 0x55ab033d8520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58662/0 (socket says 172.21.15.43:58662) 2024-03-28T22:40:56.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.603+0000 7f03e2600640 1 -- 172.21.15.43:0/1453607806 learned_addr learned my addr 172.21.15.43:0/1453607806 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:56.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.603+0000 7f03e1c00640 1 --2- 172.21.15.43:0/1453607806 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ab033cb550 0x55ab033dcbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41834/0 (socket says 172.21.15.43:41834) 2024-03-28T22:40:56.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.603+0000 7f03e1200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ab033cbe70 0x55ab033d7fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59912/0 (socket says 172.21.15.43:59912) 2024-03-28T22:40:56.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.604+0000 7f03e1200640 1 -- 172.21.15.43:0/1453607806 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ab033d4a50 msgr2=0x55ab033d8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:56.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.604+0000 7f03e1200640 1 --2- 172.21.15.43:0/1453607806 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ab033d4a50 0x55ab033d8520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.604+0000 7f03e1200640 1 -- 172.21.15.43:0/1453607806 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ab033cb550 msgr2=0x55ab033dcbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:56.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.604+0000 7f03e1200640 1 --2- 172.21.15.43:0/1453607806 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ab033cb550 0x55ab033dcbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.604+0000 7f03e1200640 1 -- 172.21.15.43:0/1453607806 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab033d8a60 con 0x55ab033cbe70 2024-03-28T22:40:56.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.604+0000 7f03e1200640 1 --2- 172.21.15.43:0/1453607806 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ab033cbe70 0x55ab033d7fe0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f03c80ed920 tx=0x7f03c80035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95b5836ea8bbee8c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:56.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.604+0000 7f03c3400640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:40:56.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.604+0000 7f03c3400640 1 -- 172.21.15.43:0/1453607806 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c80bc990 con 0x55ab033cbe70 2024-03-28T22:40:56.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.604+0000 7f03e73d4740 1 -- 172.21.15.43:0/1453607806 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03d4029070 con 0x55ab033cbe70 2024-03-28T22:40:56.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.604+0000 7f03c3400640 1 -- 172.21.15.43:0/1453607806 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f03c80b9230 con 0x55ab033cbe70 2024-03-28T22:40:56.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.605+0000 7f03c3400640 1 -- 172.21.15.43:0/1453607806 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f03c80bc520 con 0x55ab033cbe70 2024-03-28T22:40:56.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.605+0000 7f03e73d4740 10 client.27023 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:56.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.605+0000 7f03e73d4740 1 -- 172.21.15.43:0/1453607806 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ab0344af30 con 0x55ab033cbe70 2024-03-28T22:40:56.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.605+0000 7f03c3400640 1 -- 172.21.15.43:0/1453607806 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03d4029070 con 0x55ab033cbe70 2024-03-28T22:40:56.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.605+0000 7f03c3400640 1 -- 172.21.15.43:0/1453607806 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f03c80eda90 con 0x55ab033cbe70 2024-03-28T22:40:56.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.605+0000 7f03e73d4740 10 client.27023 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:56.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.605+0000 7f03e73d4740 10 client.27023 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:40:56.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.605+0000 7f03e73d4740 20 client.27023 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:56.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.605+0000 7f03e73d4740 1 --2- 172.21.15.43:0/1453607806 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55ab0344b680 0x55ab0344daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:56.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.605+0000 7f03e73d4740 4 client.27023 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:40:56.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.605+0000 7f03e73d4740 1 -- 172.21.15.43:0/1453607806 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ab0333c590 con 0x55ab0344b680 2024-03-28T22:40:56.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.605+0000 7f03e1c00640 1 --2- 172.21.15.43:0/1453607806 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55ab0344b680 0x55ab0344daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:56.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.606+0000 7f03e1c00640 1 --2- 172.21.15.43:0/1453607806 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55ab0344b680 0x55ab0344daa0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec6a53c83f9f5473 server_cookie=78b9fef2133d17a in_seq=0 out_seq=0 2024-03-28T22:40:56.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.606+0000 7f03c3400640 10 client.27023 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:56.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.608+0000 7f03c3400640 1 -- 172.21.15.43:0/1453607806 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ab0333c590 con 0x55ab0344b680 2024-03-28T22:40:56.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.608+0000 7f03c3400640 10 client.27023 handle_command_reply: tid=0 2024-03-28T22:40:56.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.608+0000 7f03e73d4740 1 client.27023 shutdown 2024-03-28T22:40:56.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.608+0000 7f03e73d4740 1 -- 172.21.15.43:0/1453607806 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ab033cbe70 msgr2=0x55ab033d7fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:56.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.608+0000 7f03e73d4740 1 --2- 172.21.15.43:0/1453607806 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ab033cbe70 0x55ab033d7fe0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f03c80ed920 tx=0x7f03c80035f0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.608+0000 7f03e73d4740 1 -- 172.21.15.43:0/1453607806 shutdown_connections 2024-03-28T22:40:56.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.608+0000 7f03e73d4740 1 --2- 172.21.15.43:0/1453607806 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55ab0344b680 0x55ab0344daa0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.608+0000 7f03e73d4740 1 --2- 172.21.15.43:0/1453607806 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ab033cbe70 0x55ab033d7fe0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.608+0000 7f03e73d4740 1 --2- 172.21.15.43:0/1453607806 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ab033d4a50 0x55ab033d8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.608+0000 7f03e73d4740 1 --2- 172.21.15.43:0/1453607806 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ab033cb550 0x55ab033dcbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.608+0000 7f03c3400640 0 client.27023 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:56.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.608+0000 7f03e73d4740 1 -- 172.21.15.43:0/1453607806 >> 172.21.15.43:0/1453607806 conn(0x55ab033c5c80 msgr2=0x55ab033d1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:56.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.608+0000 7f03e73d4740 1 -- 172.21.15.43:0/1453607806 shutdown_connections 2024-03-28T22:40:56.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.608+0000 7f03e73d4740 1 -- 172.21.15.43:0/1453607806 wait complete. 2024-03-28T22:40:56.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.608+0000 7f03e73d4740 20 client.27023 trim_cache size 0 max 16384 2024-03-28T22:40:56.609 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:40:56.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.609+0000 7f03e3000640 1 -- 172.21.15.43:0/3989473149 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f03c40a1ef0 msgr2=0x7f03c40a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:56.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.609+0000 7f03e3000640 1 --2- 172.21.15.43:0/3989473149 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f03c40a1ef0 0x7f03c40a43b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x55ab033cb020 tx=0x7f03d0005f90 comp rx=0 tx=0).stop 2024-03-28T22:40:56.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.609+0000 7f03e3000640 1 -- 172.21.15.43:0/3989473149 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03dc0765a0 msgr2=0x7f03dc19ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:56.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.609+0000 7f03e3000640 1 --2- 172.21.15.43:0/3989473149 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03dc0765a0 0x7f03dc19ea80 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f03c8009900 tx=0x7f03c8004360 comp rx=0 tx=0).stop 2024-03-28T22:40:56.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.609+0000 7f03e3000640 1 -- 172.21.15.43:0/3989473149 shutdown_connections 2024-03-28T22:40:56.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.609+0000 7f03e3000640 1 --2- 172.21.15.43:0/3989473149 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f03c40a1ef0 0x7f03c40a43b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.609+0000 7f03e3000640 1 --2- 172.21.15.43:0/3989473149 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03dc112470 0x7f03dc1a2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.609+0000 7f03e3000640 1 --2- 172.21.15.43:0/3989473149 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03dc0765a0 0x7f03dc19ea80 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.609+0000 7f03e3000640 1 --2- 172.21.15.43:0/3989473149 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03dc075c80 0x7f03dc19e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:56.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.609+0000 7f03e3000640 1 -- 172.21.15.43:0/3989473149 >> 172.21.15.43:0/3989473149 conn(0x7f03dc0ff260 msgr2=0x7f03dc113af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:56.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.609+0000 7f03e3000640 1 -- 172.21.15.43:0/3989473149 shutdown_connections 2024-03-28T22:40:56.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:56.609+0000 7f03e3000640 1 -- 172.21.15.43:0/3989473149 wait complete. 2024-03-28T22:40:56.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:56 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1935535343' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:40:57.356 INFO:tasks.workunit.client.0.smithi043.stdout: 1 228665 8.31 MB/sec execute 28 sec latency 15.447 ms 2024-03-28T22:40:57.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:57 smithi082 ceph-mon[36471]: pgmap v117: 129 pgs: 129 active+clean; 42 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 276 op/s 2024-03-28T22:40:57.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:57 smithi043 ceph-mon[31823]: pgmap v117: 129 pgs: 129 active+clean; 42 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 276 op/s 2024-03-28T22:40:57.626 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:40:57.626 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:40:57.626 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:40:57.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:57 smithi172 ceph-mon[36566]: pgmap v117: 129 pgs: 129 active+clean; 42 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 276 op/s 2024-03-28T22:40:57.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.905+0000 7fd5ab800640 1 -- 172.21.15.43:0/3416945080 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd5a41016e0 msgr2=0x7fd5a4101b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:57.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.905+0000 7fd5ab800640 1 --2- 172.21.15.43:0/3416945080 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd5a41016e0 0x7fd5a4101b60 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fd590009900 tx=0x7fd59002bc30 comp rx=0 tx=0).stop 2024-03-28T22:40:57.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.905+0000 7fd5ab800640 1 -- 172.21.15.43:0/3416945080 shutdown_connections 2024-03-28T22:40:57.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.905+0000 7fd5ab800640 1 --2- 172.21.15.43:0/3416945080 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd5a41016e0 0x7fd5a4101b60 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:57.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.905+0000 7fd5ab800640 1 --2- 172.21.15.43:0/3416945080 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5a41020a0 0x7fd5a410e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:57.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.905+0000 7fd5ab800640 1 --2- 172.21.15.43:0/3416945080 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5a41076f0 0x7fd5a4107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:57.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.905+0000 7fd5ab800640 1 -- 172.21.15.43:0/3416945080 >> 172.21.15.43:0/3416945080 conn(0x7fd5a4078b50 msgr2=0x7fd5a4078f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:57.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.905+0000 7fd5ab800640 1 -- 172.21.15.43:0/3416945080 shutdown_connections 2024-03-28T22:40:57.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.905+0000 7fd5ab800640 1 -- 172.21.15.43:0/3416945080 wait complete. 2024-03-28T22:40:57.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.906+0000 7fd5ab800640 1 Processor -- start 2024-03-28T22:40:57.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.907+0000 7fd5ab800640 1 -- start start 2024-03-28T22:40:57.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.907+0000 7fd5ab800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd5a41016e0 0x7fd5a4197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:57.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.907+0000 7fd5ab800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5a41020a0 0x7fd5a4198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:57.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.907+0000 7fd5ab800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5a41076f0 0x7fd5a419c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:57.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.907+0000 7fd5ab800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a4111b50 con 0x7fd5a41076f0 2024-03-28T22:40:57.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.907+0000 7fd5ab800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a41119d0 con 0x7fd5a41016e0 2024-03-28T22:40:57.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.907+0000 7fd5ab800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a4111cd0 con 0x7fd5a41020a0 2024-03-28T22:40:57.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.907+0000 7fd5a8e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd5a41016e0 0x7fd5a4197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:57.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.907+0000 7fd5a8e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd5a41016e0 0x7fd5a4197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59942/0 (socket says 172.21.15.43:59942) 2024-03-28T22:40:57.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.907+0000 7fd5a8e00640 1 -- 172.21.15.43:0/1437085310 learned_addr learned my addr 172.21.15.43:0/1437085310 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:57.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.907+0000 7fd5a3e00640 1 --2- 172.21.15.43:0/1437085310 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5a41020a0 0x7fd5a4198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:57.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.907+0000 7fd5a9800640 1 --2- 172.21.15.43:0/1437085310 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5a41076f0 0x7fd5a419c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:57.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.908+0000 7fd5a8e00640 1 -- 172.21.15.43:0/1437085310 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5a41020a0 msgr2=0x7fd5a4198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:57.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.908+0000 7fd5a8e00640 1 --2- 172.21.15.43:0/1437085310 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5a41020a0 0x7fd5a4198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:57.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.908+0000 7fd5a8e00640 1 -- 172.21.15.43:0/1437085310 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5a41076f0 msgr2=0x7fd5a419c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:57.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.908+0000 7fd5a8e00640 1 --2- 172.21.15.43:0/1437085310 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5a41076f0 0x7fd5a419c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:57.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.908+0000 7fd5a8e00640 1 -- 172.21.15.43:0/1437085310 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5a419cfc0 con 0x7fd5a41016e0 2024-03-28T22:40:57.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.908+0000 7fd5a8e00640 1 --2- 172.21.15.43:0/1437085310 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd5a41016e0 0x7fd5a4197fd0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fd59800ea60 tx=0x7fd59800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:57.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.908+0000 7fd5a1600640 1 -- 172.21.15.43:0/1437085310 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59800cde0 con 0x7fd5a41016e0 2024-03-28T22:40:57.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.909+0000 7fd5a1600640 1 -- 172.21.15.43:0/1437085310 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5980035f0 con 0x7fd5a41016e0 2024-03-28T22:40:57.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.909+0000 7fd5a1600640 1 -- 172.21.15.43:0/1437085310 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd598010670 con 0x7fd5a41016e0 2024-03-28T22:40:57.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.909+0000 7fd5ab800640 1 -- 172.21.15.43:0/1437085310 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5a419d2b0 con 0x7fd5a41016e0 2024-03-28T22:40:57.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.909+0000 7fd5ab800640 1 -- 172.21.15.43:0/1437085310 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5a41a4b40 con 0x7fd5a41016e0 2024-03-28T22:40:57.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.912+0000 7fd5a1600640 1 -- 172.21.15.43:0/1437085310 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fd5980037a0 con 0x7fd5a41016e0 2024-03-28T22:40:57.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.912+0000 7fd5ae56f740 1 Processor -- start 2024-03-28T22:40:57.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.912+0000 7fd5ae56f740 1 -- start start 2024-03-28T22:40:57.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.913+0000 7fd5ae56f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560d26b0b600 0x560d26b0b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:57.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.913+0000 7fd5ae56f740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560d26af9150 0x560d26af95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:57.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.913+0000 7fd5ae56f740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560d26af9b10 0x560d26b134d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:57.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.913+0000 7fd5ae56f740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x560d26a62c40 con 0x560d26af9150 2024-03-28T22:40:57.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.913+0000 7fd5ae56f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x560d26a0bc40 con 0x560d26b0b600 2024-03-28T22:40:57.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.913+0000 7fd5ae56f740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x560d26a5ece0 con 0x560d26af9b10 2024-03-28T22:40:57.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.913+0000 7fd5a9800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560d26b0b600 0x560d26b0b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:57.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.913+0000 7fd5a3e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560d26af9b10 0x560d26b134d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:57.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.913+0000 7fd5a9800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560d26b0b600 0x560d26b0b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59950/0 (socket says 172.21.15.43:59950) 2024-03-28T22:40:57.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.913+0000 7fd5a9800640 1 -- 172.21.15.43:0/157014981 learned_addr learned my addr 172.21.15.43:0/157014981 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:57.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.913+0000 7fd5a8e00640 1 --2- 172.21.15.43:0/157014981 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560d26af9150 0x560d26af95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:57.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.914+0000 7fd5a3e00640 1 -- 172.21.15.43:0/157014981 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560d26b0b600 msgr2=0x560d26b0b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:57.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.914+0000 7fd5a3e00640 1 --2- 172.21.15.43:0/157014981 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560d26b0b600 0x560d26b0b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:57.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.914+0000 7fd5a3e00640 1 -- 172.21.15.43:0/157014981 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560d26af9150 msgr2=0x560d26af95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:57.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.914+0000 7fd5a3e00640 1 --2- 172.21.15.43:0/157014981 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560d26af9150 0x560d26af95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:57.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.914+0000 7fd5a3e00640 1 -- 172.21.15.43:0/157014981 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d2683f7e0 con 0x560d26af9b10 2024-03-28T22:40:57.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.914+0000 7fd5a1600640 1 --2- 172.21.15.43:0/1437085310 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd58c0a1f70 0x7fd58c0a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:57.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.914+0000 7fd5a1600640 1 -- 172.21.15.43:0/1437085310 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fd5980ce7a0 con 0x7fd5a41016e0 2024-03-28T22:40:57.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.914+0000 7fd5a3e00640 1 --2- 172.21.15.43:0/157014981 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560d26af9b10 0x560d26b134d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fd59002c170 tx=0x7fd590005df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=317c4f4b6a834462 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:57.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.914+0000 7fd5a9800640 1 --2- 172.21.15.43:0/1437085310 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd58c0a1f70 0x7fd58c0a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:57.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.914+0000 7fd596000640 1 -- 172.21.15.43:0/157014981 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd590042070 con 0x560d26af9b10 2024-03-28T22:40:57.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.914+0000 7fd596000640 1 -- 172.21.15.43:0/157014981 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd59002c690 con 0x560d26af9b10 2024-03-28T22:40:57.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.914+0000 7fd596000640 1 -- 172.21.15.43:0/157014981 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd590004bf0 con 0x560d26af9b10 2024-03-28T22:40:57.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.915+0000 7fd5ae56f740 1 -- 172.21.15.43:0/157014981 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560d26af9b10 msgr2=0x560d26b134d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:57.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.915+0000 7fd5ae56f740 1 --2- 172.21.15.43:0/157014981 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560d26af9b10 0x560d26b134d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fd59002c170 tx=0x7fd590005df0 comp rx=0 tx=0).stop 2024-03-28T22:40:57.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.915+0000 7fd5a9800640 1 --2- 172.21.15.43:0/1437085310 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd58c0a1f70 0x7fd58c0a4430 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x560d26b07c30 tx=0x7fd59c009500 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:57.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.915+0000 7fd5ae56f740 1 -- 172.21.15.43:0/157014981 shutdown_connections 2024-03-28T22:40:57.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.915+0000 7fd5ae56f740 1 --2- 172.21.15.43:0/157014981 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560d26af9b10 0x560d26b134d0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:57.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.915+0000 7fd5ae56f740 1 --2- 172.21.15.43:0/157014981 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560d26af9150 0x560d26af95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:57.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.915+0000 7fd5ae56f740 1 --2- 172.21.15.43:0/157014981 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560d26b0b600 0x560d26b0b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:57.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.915+0000 7fd5ae56f740 1 -- 172.21.15.43:0/157014981 >> 172.21.15.43:0/157014981 conn(0x560d26afbc10 msgr2=0x560d26afbff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:57.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.915+0000 7fd5ae56f740 1 -- 172.21.15.43:0/157014981 shutdown_connections 2024-03-28T22:40:57.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.916+0000 7fd5ae56f740 1 -- 172.21.15.43:0/157014981 wait complete. 2024-03-28T22:40:57.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.916+0000 7fd5ae56f740 1 -- start start 2024-03-28T22:40:57.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.916+0000 7fd5ae56f740 1 Processor -- start 2024-03-28T22:40:57.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.917+0000 7fd5ae56f740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560d26af9150 0x560d26b109b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:57.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.917+0000 7fd5ae56f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560d26af9b10 0x560d26b10ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:57.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.917+0000 7fd5ae56f740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560d26b0b600 0x560d26b11430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:57.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.917+0000 7fd5ae56f740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x560d26b13ad0 con 0x560d26af9150 2024-03-28T22:40:57.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.917+0000 7fd5ae56f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x560d26a01250 con 0x560d26af9b10 2024-03-28T22:40:57.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.917+0000 7fd5ae56f740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x560d26b13c50 con 0x560d26b0b600 2024-03-28T22:40:57.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.917+0000 7fd5ae56f740 10 client.0 _ll_register_callbacks cb 0x560d26a22e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:57.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.917+0000 7fd5a3e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560d26af9b10 0x560d26b10ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:57.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.917+0000 7fd5a3e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560d26af9b10 0x560d26b10ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59966/0 (socket says 172.21.15.43:59966) 2024-03-28T22:40:57.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.917+0000 7fd5a3e00640 1 -- 172.21.15.43:0/3333639156 learned_addr learned my addr 172.21.15.43:0/3333639156 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:57.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.917+0000 7fd5a8e00640 1 --2- 172.21.15.43:0/3333639156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560d26af9150 0x560d26b109b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:57.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.917+0000 7fd5a9800640 1 --2- 172.21.15.43:0/3333639156 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560d26b0b600 0x560d26b11430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:57.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.918+0000 7fd5a3e00640 1 -- 172.21.15.43:0/3333639156 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560d26b0b600 msgr2=0x560d26b11430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:57.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.918+0000 7fd5a3e00640 1 --2- 172.21.15.43:0/3333639156 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560d26b0b600 0x560d26b11430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:57.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.918+0000 7fd5a3e00640 1 -- 172.21.15.43:0/3333639156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560d26af9150 msgr2=0x560d26b109b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:57.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.918+0000 7fd5a3e00640 1 --2- 172.21.15.43:0/3333639156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560d26af9150 0x560d26b109b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:57.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.918+0000 7fd5a3e00640 1 -- 172.21.15.43:0/3333639156 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d2689ff70 con 0x560d26af9b10 2024-03-28T22:40:57.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.918+0000 7fd5a8e00640 1 --2- 172.21.15.43:0/3333639156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560d26af9150 0x560d26b109b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:57.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.918+0000 7fd5a3e00640 1 --2- 172.21.15.43:0/3333639156 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560d26af9b10 0x560d26b10ef0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fd590009900 tx=0x7fd590005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4945f082f84be0a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:57.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.918+0000 7fd58aa00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:40:57.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.918+0000 7fd58aa00640 1 -- 172.21.15.43:0/3333639156 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59004c070 con 0x560d26af9b10 2024-03-28T22:40:57.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.918+0000 7fd5ae56f740 1 -- 172.21.15.43:0/3333639156 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd59002c690 con 0x560d26af9b10 2024-03-28T22:40:57.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.918+0000 7fd58aa00640 1 -- 172.21.15.43:0/3333639156 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fd590005180 con 0x560d26af9b10 2024-03-28T22:40:57.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.918+0000 7fd58aa00640 1 -- 172.21.15.43:0/3333639156 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd590058050 con 0x560d26af9b10 2024-03-28T22:40:57.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.918+0000 7fd5ae56f740 10 client.27035 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:57.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.918+0000 7fd5ae56f740 1 -- 172.21.15.43:0/3333639156 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560d26737170 con 0x560d26af9b10 2024-03-28T22:40:57.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.919+0000 7fd58aa00640 1 -- 172.21.15.43:0/3333639156 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd59002c690 con 0x560d26af9b10 2024-03-28T22:40:57.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.920+0000 7fd58aa00640 1 -- 172.21.15.43:0/3333639156 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd590047be0 con 0x560d26af9b10 2024-03-28T22:40:57.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.920+0000 7fd5ae56f740 10 client.27035 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:57.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.920+0000 7fd5ae56f740 10 client.27035 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:40:57.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.920+0000 7fd5ae56f740 20 client.27035 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:57.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.920+0000 7fd5ae56f740 1 --2- 172.21.15.43:0/3333639156 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560d26b20ec0 0x560d26b232b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:57.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.920+0000 7fd5ae56f740 4 client.27035 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:40:57.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.920+0000 7fd5ae56f740 1 -- 172.21.15.43:0/3333639156 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d26a70180 con 0x560d26b20ec0 2024-03-28T22:40:57.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.920+0000 7fd5a8e00640 1 --2- 172.21.15.43:0/3333639156 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560d26b20ec0 0x560d26b232b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:57.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.920+0000 7fd5a8e00640 1 --2- 172.21.15.43:0/3333639156 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560d26b20ec0 0x560d26b232b0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fae049474a5898a4 server_cookie=deb787bf4b87b0cc in_seq=0 out_seq=0 2024-03-28T22:40:57.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.920+0000 7fd58aa00640 10 client.27035 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:57.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.922+0000 7fd58aa00640 1 -- 172.21.15.43:0/3333639156 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560d26a70180 con 0x560d26b20ec0 2024-03-28T22:40:57.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.922+0000 7fd58aa00640 10 client.27035 handle_command_reply: tid=0 2024-03-28T22:40:57.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.922+0000 7fd5ae56f740 1 client.27035 shutdown 2024-03-28T22:40:57.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.922+0000 7fd5ae56f740 1 -- 172.21.15.43:0/3333639156 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560d26af9b10 msgr2=0x560d26b10ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:57.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.922+0000 7fd5ae56f740 1 --2- 172.21.15.43:0/3333639156 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560d26af9b10 0x560d26b10ef0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fd590009900 tx=0x7fd590005a20 comp rx=0 tx=0).stop 2024-03-28T22:40:57.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.922+0000 7fd5ae56f740 1 -- 172.21.15.43:0/3333639156 shutdown_connections 2024-03-28T22:40:57.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.922+0000 7fd5ae56f740 1 --2- 172.21.15.43:0/3333639156 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560d26b20ec0 0x560d26b232b0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:57.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.922+0000 7fd5ae56f740 1 --2- 172.21.15.43:0/3333639156 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560d26af9b10 0x560d26b10ef0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:57.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.922+0000 7fd5ae56f740 1 --2- 172.21.15.43:0/3333639156 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560d26b0b600 0x560d26b11430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:57.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.922+0000 7fd5ae56f740 1 --2- 172.21.15.43:0/3333639156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560d26af9150 0x560d26b109b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:57.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.922+0000 7fd5ae56f740 1 -- 172.21.15.43:0/3333639156 >> 172.21.15.43:0/3333639156 conn(0x560d26afbc10 msgr2=0x560d26b0ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:57.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.922+0000 7fd58aa00640 0 client.27035 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:57.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.922+0000 7fd5ae56f740 1 -- 172.21.15.43:0/3333639156 shutdown_connections 2024-03-28T22:40:57.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.922+0000 7fd5ae56f740 1 -- 172.21.15.43:0/3333639156 wait complete. 2024-03-28T22:40:57.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.923+0000 7fd5ae56f740 20 client.27035 trim_cache size 0 max 16384 2024-03-28T22:40:57.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.942+0000 7fd5ae56f740 1 Processor -- start 2024-03-28T22:40:57.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.942+0000 7fd5ae56f740 1 -- start start 2024-03-28T22:40:57.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.943+0000 7fd5ae56f740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560d26af9150 0x560d26b0e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:57.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.943+0000 7fd5ae56f740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560d26af9b10 0x560d26b1c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:57.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.943+0000 7fd5ae56f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560d26b0b600 0x560d26b120d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:57.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.943+0000 7fd5ae56f740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a41ac8c0 con 0x560d26af9b10 2024-03-28T22:40:57.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.943+0000 7fd5ae56f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a41ae400 con 0x560d26b0b600 2024-03-28T22:40:57.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.943+0000 7fd5ae56f740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a41acf90 con 0x560d26af9150 2024-03-28T22:40:57.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.943+0000 7fd5a9800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560d26b0b600 0x560d26b120d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:57.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.943+0000 7fd5a8e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560d26af9150 0x560d26b0e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:57.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.943+0000 7fd5a3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560d26af9b10 0x560d26b1c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:57.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.943+0000 7fd5a9800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560d26b0b600 0x560d26b120d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59972/0 (socket says 172.21.15.43:59972) 2024-03-28T22:40:57.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.943+0000 7fd5a8e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560d26af9150 0x560d26b0e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58696/0 (socket says 172.21.15.43:58696) 2024-03-28T22:40:57.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.943+0000 7fd5a9800640 1 -- 172.21.15.43:0/778014288 learned_addr learned my addr 172.21.15.43:0/778014288 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:57.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.944+0000 7fd5a9800640 1 -- 172.21.15.43:0/778014288 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560d26af9150 msgr2=0x560d26b0e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:57.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.944+0000 7fd5a9800640 1 --2- 172.21.15.43:0/778014288 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560d26af9150 0x560d26b0e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:57.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.944+0000 7fd5a9800640 1 -- 172.21.15.43:0/778014288 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560d26af9b10 msgr2=0x560d26b1c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:57.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.944+0000 7fd5a9800640 1 --2- 172.21.15.43:0/778014288 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560d26af9b10 0x560d26b1c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:57.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.944+0000 7fd5a9800640 1 -- 172.21.15.43:0/778014288 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d26afd1d0 con 0x560d26b0b600 2024-03-28T22:40:57.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.944+0000 7fd5a8e00640 1 --2- 172.21.15.43:0/778014288 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560d26af9150 0x560d26b0e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:57.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.944+0000 7fd5a3e00640 1 --2- 172.21.15.43:0/778014288 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560d26af9b10 0x560d26b1c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:57.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.944+0000 7fd5a9800640 1 --2- 172.21.15.43:0/778014288 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560d26b0b600 0x560d26b120d0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fd59c0112c0 tx=0x7fd59c019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74e913321033e29e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:57.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.944+0000 7fd595600640 1 -- 172.21.15.43:0/778014288 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59c003970 con 0x560d26b0b600 2024-03-28T22:40:57.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.944+0000 7fd595600640 1 -- 172.21.15.43:0/778014288 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd59c004030 con 0x560d26b0b600 2024-03-28T22:40:57.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.944+0000 7fd595600640 1 -- 172.21.15.43:0/778014288 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59c0107d0 con 0x560d26b0b600 2024-03-28T22:40:57.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.945+0000 7fd5ae56f740 1 -- 172.21.15.43:0/778014288 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560d26b0b600 msgr2=0x560d26b120d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:57.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.945+0000 7fd5ae56f740 1 --2- 172.21.15.43:0/778014288 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560d26b0b600 0x560d26b120d0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fd59c0112c0 tx=0x7fd59c019000 comp rx=0 tx=0).stop 2024-03-28T22:40:57.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.945+0000 7fd5ae56f740 1 -- 172.21.15.43:0/778014288 shutdown_connections 2024-03-28T22:40:57.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.945+0000 7fd5ae56f740 1 --2- 172.21.15.43:0/778014288 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560d26b0b600 0x560d26b120d0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:57.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.945+0000 7fd5ae56f740 1 --2- 172.21.15.43:0/778014288 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560d26af9b10 0x560d26b1c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:57.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.945+0000 7fd5ae56f740 1 --2- 172.21.15.43:0/778014288 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560d26af9150 0x560d26b0e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:57.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.945+0000 7fd5ae56f740 1 -- 172.21.15.43:0/778014288 >> 172.21.15.43:0/778014288 conn(0x560d26afbc10 msgr2=0x560d26afb2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:57.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.945+0000 7fd5ae56f740 1 -- 172.21.15.43:0/778014288 shutdown_connections 2024-03-28T22:40:57.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.945+0000 7fd5ae56f740 1 -- 172.21.15.43:0/778014288 wait complete. 2024-03-28T22:40:57.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.946+0000 7fd5ae56f740 1 -- start start 2024-03-28T22:40:57.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.946+0000 7fd5ae56f740 1 Processor -- start 2024-03-28T22:40:57.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.946+0000 7fd5ae56f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560d26af9150 0x560d26b0d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:57.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.946+0000 7fd5ae56f740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560d26af9b10 0x560d26b0d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:57.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.946+0000 7fd5ae56f740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560d26b0b600 0x560d26b17f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:57.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.946+0000 7fd5ae56f740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a41add30 con 0x560d26b0b600 2024-03-28T22:40:57.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.946+0000 7fd5ae56f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a41ad660 con 0x560d26af9150 2024-03-28T22:40:57.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.946+0000 7fd5ae56f740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a41aa8a0 con 0x560d26af9b10 2024-03-28T22:40:57.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.946+0000 7fd5ae56f740 10 client.0 _ll_register_callbacks cb 0x7fd5980d4400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:57.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.946+0000 7fd5a8e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560d26af9150 0x560d26b0d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:57.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.946+0000 7fd5a8e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560d26af9150 0x560d26b0d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59976/0 (socket says 172.21.15.43:59976) 2024-03-28T22:40:57.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.946+0000 7fd5a8e00640 1 -- 172.21.15.43:0/3344240691 learned_addr learned my addr 172.21.15.43:0/3344240691 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:57.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.946+0000 7fd5a3e00640 1 --2- 172.21.15.43:0/3344240691 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560d26af9b10 0x560d26b0d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:57.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.947+0000 7fd5a9800640 1 --2- 172.21.15.43:0/3344240691 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560d26b0b600 0x560d26b17f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:57.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.947+0000 7fd5a8e00640 1 -- 172.21.15.43:0/3344240691 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560d26af9b10 msgr2=0x560d26b0d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:57.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.947+0000 7fd5a8e00640 1 --2- 172.21.15.43:0/3344240691 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560d26af9b10 0x560d26b0d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:57.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.947+0000 7fd5a8e00640 1 -- 172.21.15.43:0/3344240691 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560d26b0b600 msgr2=0x560d26b17f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:57.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.947+0000 7fd5a8e00640 1 --2- 172.21.15.43:0/3344240691 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560d26b0b600 0x560d26b17f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:57.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.947+0000 7fd5a8e00640 1 -- 172.21.15.43:0/3344240691 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d26b184d0 con 0x560d26af9150 2024-03-28T22:40:57.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.947+0000 7fd5a3e00640 1 --2- 172.21.15.43:0/3344240691 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560d26af9b10 0x560d26b0d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:57.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.947+0000 7fd5a8e00640 1 --2- 172.21.15.43:0/3344240691 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560d26af9150 0x560d26b0d330 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fd59809ad20 tx=0x7fd598005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a856364cc6d822e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:57.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.947+0000 7fd58aa00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:40:57.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.947+0000 7fd58aa00640 1 -- 172.21.15.43:0/3344240691 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59809ad90 con 0x560d26af9150 2024-03-28T22:40:57.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.947+0000 7fd58aa00640 1 -- 172.21.15.43:0/3344240691 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fd598097400 con 0x560d26af9150 2024-03-28T22:40:57.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.947+0000 7fd5ae56f740 1 -- 172.21.15.43:0/3344240691 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd59c004030 con 0x560d26af9150 2024-03-28T22:40:57.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.948+0000 7fd58aa00640 1 -- 172.21.15.43:0/3344240691 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd59809eee0 con 0x560d26af9150 2024-03-28T22:40:57.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.948+0000 7fd5ae56f740 10 client.27041 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:57.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.948+0000 7fd5ae56f740 1 -- 172.21.15.43:0/3344240691 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560d26b86f70 con 0x560d26af9150 2024-03-28T22:40:57.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.948+0000 7fd58aa00640 1 -- 172.21.15.43:0/3344240691 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd59c004030 con 0x560d26af9150 2024-03-28T22:40:57.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.948+0000 7fd58aa00640 1 -- 172.21.15.43:0/3344240691 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd5980ceb90 con 0x560d26af9150 2024-03-28T22:40:57.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.948+0000 7fd5ae56f740 10 client.27041 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:57.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.948+0000 7fd5ae56f740 10 client.27041 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:40:57.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.948+0000 7fd5ae56f740 20 client.27041 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:57.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.948+0000 7fd5ae56f740 1 --2- 172.21.15.43:0/3344240691 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560d26b875e0 0x560d26b899d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:57.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.948+0000 7fd5ae56f740 4 client.27041 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:40:57.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.948+0000 7fd5ae56f740 1 -- 172.21.15.43:0/3344240691 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560d26a6a4d0 con 0x560d26b875e0 2024-03-28T22:40:57.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.948+0000 7fd5a3e00640 1 --2- 172.21.15.43:0/3344240691 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560d26b875e0 0x560d26b899d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:57.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.949+0000 7fd5a3e00640 1 --2- 172.21.15.43:0/3344240691 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560d26b875e0 0x560d26b899d0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44dd7c683eb07b4c server_cookie=be8ad883a0f0f2cc in_seq=0 out_seq=0 2024-03-28T22:40:57.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.949+0000 7fd58aa00640 10 client.27041 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:57.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.953+0000 7fd58aa00640 1 -- 172.21.15.43:0/3344240691 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560d26a6a4d0 con 0x560d26b875e0 2024-03-28T22:40:57.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.953+0000 7fd58aa00640 10 client.27041 handle_command_reply: tid=0 2024-03-28T22:40:57.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.953+0000 7fd5ae56f740 1 client.27041 shutdown 2024-03-28T22:40:57.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.953+0000 7fd5ae56f740 1 -- 172.21.15.43:0/3344240691 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560d26af9150 msgr2=0x560d26b0d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:57.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.953+0000 7fd5ae56f740 1 --2- 172.21.15.43:0/3344240691 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560d26af9150 0x560d26b0d330 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fd59809ad20 tx=0x7fd598005f10 comp rx=0 tx=0).stop 2024-03-28T22:40:57.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.953+0000 7fd5ae56f740 1 -- 172.21.15.43:0/3344240691 shutdown_connections 2024-03-28T22:40:57.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.953+0000 7fd5ae56f740 1 --2- 172.21.15.43:0/3344240691 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560d26b0b600 0x560d26b17f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:57.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.953+0000 7fd5ae56f740 1 --2- 172.21.15.43:0/3344240691 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560d26af9b10 0x560d26b0d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:57.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.953+0000 7fd5ae56f740 1 --2- 172.21.15.43:0/3344240691 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560d26b875e0 0x560d26b899d0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:57.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.953+0000 7fd5ae56f740 1 --2- 172.21.15.43:0/3344240691 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560d26af9150 0x560d26b0d330 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:57.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.953+0000 7fd58aa00640 0 client.27041 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:57.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.953+0000 7fd5ae56f740 1 -- 172.21.15.43:0/3344240691 >> 172.21.15.43:0/3344240691 conn(0x560d26afbc10 msgr2=0x560d26b11160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:57.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.954+0000 7fd5ae56f740 1 -- 172.21.15.43:0/3344240691 shutdown_connections 2024-03-28T22:40:57.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.954+0000 7fd5ae56f740 1 -- 172.21.15.43:0/3344240691 wait complete. 2024-03-28T22:40:57.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.954+0000 7fd5ae56f740 20 client.27041 trim_cache size 0 max 16384 2024-03-28T22:40:57.953 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:40:57.954 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-03-28T22:40:57.954 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "24cfdf77-b426-48fa-90e1-931666d3e793", 2024-03-28T22:40:57.954 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-03-28T22:40:57.954 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:40:57.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.954+0000 7fd5ab800640 1 -- 172.21.15.43:0/1437085310 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd58c0a1f70 msgr2=0x7fd58c0a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:57.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.954+0000 7fd5ab800640 1 --2- 172.21.15.43:0/1437085310 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd58c0a1f70 0x7fd58c0a4430 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x560d26b07c30 tx=0x7fd59c009500 comp rx=0 tx=0).stop 2024-03-28T22:40:57.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.954+0000 7fd5ab800640 1 -- 172.21.15.43:0/1437085310 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd5a41016e0 msgr2=0x7fd5a4197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:57.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.954+0000 7fd5ab800640 1 --2- 172.21.15.43:0/1437085310 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd5a41016e0 0x7fd5a4197fd0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fd59800ea60 tx=0x7fd59800ef20 comp rx=0 tx=0).stop 2024-03-28T22:40:57.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.955+0000 7fd5ab800640 1 -- 172.21.15.43:0/1437085310 shutdown_connections 2024-03-28T22:40:57.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.955+0000 7fd5ab800640 1 --2- 172.21.15.43:0/1437085310 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd58c0a1f70 0x7fd58c0a4430 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:57.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.955+0000 7fd5ab800640 1 --2- 172.21.15.43:0/1437085310 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5a41076f0 0x7fd5a419c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:57.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.955+0000 7fd5ab800640 1 --2- 172.21.15.43:0/1437085310 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5a41020a0 0x7fd5a4198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:57.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.955+0000 7fd5ab800640 1 --2- 172.21.15.43:0/1437085310 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd5a41016e0 0x7fd5a4197fd0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:57.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.955+0000 7fd5ab800640 1 -- 172.21.15.43:0/1437085310 >> 172.21.15.43:0/1437085310 conn(0x7fd5a4078b50 msgr2=0x7fd5a4103f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:57.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.955+0000 7fd5ab800640 1 -- 172.21.15.43:0/1437085310 shutdown_connections 2024-03-28T22:40:57.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:57.955+0000 7fd5ab800640 1 -- 172.21.15.43:0/1437085310 wait complete. 2024-03-28T22:40:57.973 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:40:57.973 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:40:58.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.242+0000 7f1edbe00640 1 -- 172.21.15.43:0/163329819 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1ed4109900 msgr2=0x7f1ed4109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:58.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.242+0000 7f1edbe00640 1 --2- 172.21.15.43:0/163329819 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1ed4109900 0x7f1ed4109ce0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8009900 tx=0x7f1ec802bc40 comp rx=0 tx=0).stop 2024-03-28T22:40:58.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.243+0000 7f1edbe00640 1 -- 172.21.15.43:0/163329819 shutdown_connections 2024-03-28T22:40:58.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.243+0000 7f1edbe00640 1 --2- 172.21.15.43:0/163329819 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ed41042b0 0x7f1ed41107c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.243+0000 7f1edbe00640 1 --2- 172.21.15.43:0/163329819 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ed41038f0 0x7f1ed4103d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.243+0000 7f1edbe00640 1 --2- 172.21.15.43:0/163329819 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1ed4109900 0x7f1ed4109ce0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.243+0000 7f1edbe00640 1 -- 172.21.15.43:0/163329819 >> 172.21.15.43:0/163329819 conn(0x7f1ed40ff3c0 msgr2=0x7f1ed41017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:58.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.243+0000 7f1edbe00640 1 -- 172.21.15.43:0/163329819 shutdown_connections 2024-03-28T22:40:58.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.243+0000 7f1edbe00640 1 -- 172.21.15.43:0/163329819 wait complete. 2024-03-28T22:40:58.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.244+0000 7f1edbe00640 1 Processor -- start 2024-03-28T22:40:58.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.244+0000 7f1edbe00640 1 -- start start 2024-03-28T22:40:58.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.244+0000 7f1edbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ed41038f0 0x7f1ed419a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:58.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.244+0000 7f1edbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1ed41042b0 0x7f1ed419a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:58.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.244+0000 7f1edbe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ed4109900 0x7f1ed419eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:58.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.244+0000 7f1edbe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed4113d30 con 0x7f1ed41038f0 2024-03-28T22:40:58.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.244+0000 7f1edbe00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed4113bb0 con 0x7f1ed41042b0 2024-03-28T22:40:58.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.244+0000 7f1edbe00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed4113eb0 con 0x7f1ed4109900 2024-03-28T22:40:58.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.245+0000 7f1ed8a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1ed41042b0 0x7f1ed419a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:58.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.245+0000 7f1ed9e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ed4109900 0x7f1ed419eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:58.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.245+0000 7f1ed9e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ed4109900 0x7f1ed419eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58728/0 (socket says 172.21.15.43:58728) 2024-03-28T22:40:58.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.245+0000 7f1ed9e00640 1 -- 172.21.15.43:0/1969276257 learned_addr learned my addr 172.21.15.43:0/1969276257 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:58.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.245+0000 7f1ed8a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1ed41042b0 0x7f1ed419a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59996/0 (socket says 172.21.15.43:59996) 2024-03-28T22:40:58.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.245+0000 7f1ed9400640 1 --2- 172.21.15.43:0/1969276257 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ed41038f0 0x7f1ed419a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:58.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.245+0000 7f1ed8a00640 1 -- 172.21.15.43:0/1969276257 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ed4109900 msgr2=0x7f1ed419eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:58.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.245+0000 7f1ed8a00640 1 --2- 172.21.15.43:0/1969276257 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ed4109900 0x7f1ed419eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.245+0000 7f1ed8a00640 1 -- 172.21.15.43:0/1969276257 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ed41038f0 msgr2=0x7f1ed419a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:58.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.245+0000 7f1ed8a00640 1 --2- 172.21.15.43:0/1969276257 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ed41038f0 0x7f1ed419a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.245+0000 7f1ed8a00640 1 -- 172.21.15.43:0/1969276257 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ed419f280 con 0x7f1ed41042b0 2024-03-28T22:40:58.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.245+0000 7f1ed9400640 1 --2- 172.21.15.43:0/1969276257 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ed41038f0 0x7f1ed419a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:40:58.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.245+0000 7f1ed8a00640 1 --2- 172.21.15.43:0/1969276257 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1ed41042b0 0x7f1ed419a7d0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f1ec000e9f0 tx=0x7f1ec000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:58.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.246+0000 7f1ed2000640 1 -- 172.21.15.43:0/1969276257 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec000ccc0 con 0x7f1ed41042b0 2024-03-28T22:40:58.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.246+0000 7f1ed2000640 1 -- 172.21.15.43:0/1969276257 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ec0010910 con 0x7f1ed41042b0 2024-03-28T22:40:58.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.246+0000 7f1ed2000640 1 -- 172.21.15.43:0/1969276257 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec0018950 con 0x7f1ed41042b0 2024-03-28T22:40:58.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.246+0000 7f1edbe00640 1 -- 172.21.15.43:0/1969276257 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ed419f570 con 0x7f1ed41042b0 2024-03-28T22:40:58.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.246+0000 7f1edbe00640 1 -- 172.21.15.43:0/1969276257 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ed41a6e00 con 0x7f1ed41042b0 2024-03-28T22:40:58.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.249+0000 7f1edeb63740 1 Processor -- start 2024-03-28T22:40:58.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.249+0000 7f1edeb63740 1 -- start start 2024-03-28T22:40:58.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.249+0000 7f1ed2000640 1 -- 172.21.15.43:0/1969276257 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f1ec0010ac0 con 0x7f1ed41042b0 2024-03-28T22:40:58.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.249+0000 7f1edeb63740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e92beacbc0 0x55e92beab910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:58.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.250+0000 7f1edeb63740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e92beabee0 0x55e92beac360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:58.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.250+0000 7f1edeb63740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e92beb5250 0x55e92beb7640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:58.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.250+0000 7f1edeb63740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e92be10100 con 0x55e92beabee0 2024-03-28T22:40:58.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.250+0000 7f1edeb63740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55e92bdb8ba0 con 0x55e92beacbc0 2024-03-28T22:40:58.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.250+0000 7f1edeb63740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55e92be0c1a0 con 0x55e92beb5250 2024-03-28T22:40:58.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.250+0000 7f1ed9e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e92beacbc0 0x55e92beab910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:58.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.250+0000 7f1ed9e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e92beacbc0 0x55e92beab910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:60006/0 (socket says 172.21.15.43:60006) 2024-03-28T22:40:58.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.250+0000 7f1ed9e00640 1 -- 172.21.15.43:0/447187161 learned_addr learned my addr 172.21.15.43:0/447187161 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:58.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.250+0000 7f1ed8a00640 1 --2- 172.21.15.43:0/447187161 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e92beb5250 0x55e92beb7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:58.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.250+0000 7f1ed9400640 1 --2- 172.21.15.43:0/447187161 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e92beabee0 0x55e92beac360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:58.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.250+0000 7f1ed9e00640 1 -- 172.21.15.43:0/447187161 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e92beb5250 msgr2=0x55e92beb7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:58.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.250+0000 7f1ed9e00640 1 --2- 172.21.15.43:0/447187161 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e92beb5250 0x55e92beb7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.250+0000 7f1ed9e00640 1 -- 172.21.15.43:0/447187161 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e92beabee0 msgr2=0x55e92beac360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:58.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.250+0000 7f1ed9e00640 1 --2- 172.21.15.43:0/447187161 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e92beabee0 0x55e92beac360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.250+0000 7f1ed9e00640 1 -- 172.21.15.43:0/447187161 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e92bde91a0 con 0x55e92beacbc0 2024-03-28T22:40:58.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.250+0000 7f1ed8a00640 1 --2- 172.21.15.43:0/447187161 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e92beb5250 0x55e92beb7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:40:58.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.251+0000 7f1ed9e00640 1 --2- 172.21.15.43:0/447187161 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e92beacbc0 0x55e92beab910 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc011440 tx=0x7f1ecc011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4c6bc0dd3affa9c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:58.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.251+0000 7f1ec6600640 1 -- 172.21.15.43:0/447187161 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc004b20 con 0x55e92beacbc0 2024-03-28T22:40:58.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.251+0000 7f1ec6600640 1 -- 172.21.15.43:0/447187161 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ecc004cc0 con 0x55e92beacbc0 2024-03-28T22:40:58.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.251+0000 7f1ec6600640 1 -- 172.21.15.43:0/447187161 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc005450 con 0x55e92beacbc0 2024-03-28T22:40:58.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.251+0000 7f1ed2000640 1 --2- 172.21.15.43:0/1969276257 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1ebc0a1f70 0x7f1ebc0a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:58.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.251+0000 7f1ed2000640 1 -- 172.21.15.43:0/1969276257 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f1ec00ce8a0 con 0x7f1ed41042b0 2024-03-28T22:40:58.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.251+0000 7f1ed9400640 1 --2- 172.21.15.43:0/1969276257 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1ebc0a1f70 0x7f1ebc0a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:58.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.252+0000 7f1edeb63740 1 -- 172.21.15.43:0/447187161 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e92beacbc0 msgr2=0x55e92beab910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:58.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.252+0000 7f1edeb63740 1 --2- 172.21.15.43:0/447187161 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e92beacbc0 0x55e92beab910 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc011440 tx=0x7f1ecc011900 comp rx=0 tx=0).stop 2024-03-28T22:40:58.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.252+0000 7f1edeb63740 1 -- 172.21.15.43:0/447187161 shutdown_connections 2024-03-28T22:40:58.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.252+0000 7f1edeb63740 1 --2- 172.21.15.43:0/447187161 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e92beb5250 0x55e92beb7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.252+0000 7f1edeb63740 1 --2- 172.21.15.43:0/447187161 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e92beabee0 0x55e92beac360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.252+0000 7f1edeb63740 1 --2- 172.21.15.43:0/447187161 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e92beacbc0 0x55e92beab910 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.252+0000 7f1edeb63740 1 -- 172.21.15.43:0/447187161 >> 172.21.15.43:0/447187161 conn(0x55e92be72a10 msgr2=0x55e92bea6930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:58.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.252+0000 7f1edeb63740 1 -- 172.21.15.43:0/447187161 shutdown_connections 2024-03-28T22:40:58.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.252+0000 7f1edeb63740 1 -- 172.21.15.43:0/447187161 wait complete. 2024-03-28T22:40:58.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.252+0000 7f1ed9400640 1 --2- 172.21.15.43:0/1969276257 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1ebc0a1f70 0x7f1ebc0a4430 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8006010 tx=0x7f1ec8005b50 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:58.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.253+0000 7f1edeb63740 1 -- start start 2024-03-28T22:40:58.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.253+0000 7f1edeb63740 1 Processor -- start 2024-03-28T22:40:58.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.253+0000 7f1edeb63740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e92beabee0 0x55e92beb20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:58.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.253+0000 7f1edeb63740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e92beacbc0 0x55e92beb2610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:58.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.253+0000 7f1edeb63740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e92beb5250 0x55e92beb2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:58.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.253+0000 7f1edeb63740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e92beb80c0 con 0x55e92beb5250 2024-03-28T22:40:58.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.253+0000 7f1edeb63740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55e92bdae250 con 0x55e92beacbc0 2024-03-28T22:40:58.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.253+0000 7f1edeb63740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55e92beb8240 con 0x55e92beabee0 2024-03-28T22:40:58.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.253+0000 7f1edeb63740 10 client.0 _ll_register_callbacks cb 0x55e92bb23be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:58.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.254+0000 7f1ed8a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e92beacbc0 0x55e92beb2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:58.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.254+0000 7f1ed8a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e92beacbc0 0x55e92beb2610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:60012/0 (socket says 172.21.15.43:60012) 2024-03-28T22:40:58.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.254+0000 7f1ed8a00640 1 -- 172.21.15.43:0/2867438846 learned_addr learned my addr 172.21.15.43:0/2867438846 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:58.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.254+0000 7f1ed9400640 1 --2- 172.21.15.43:0/2867438846 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e92beabee0 0x55e92beb20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:58.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.254+0000 7f1ed9e00640 1 --2- 172.21.15.43:0/2867438846 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e92beb5250 0x55e92beb2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:58.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.254+0000 7f1ed8a00640 1 -- 172.21.15.43:0/2867438846 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e92beabee0 msgr2=0x55e92beb20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:58.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.254+0000 7f1ed8a00640 1 --2- 172.21.15.43:0/2867438846 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e92beabee0 0x55e92beb20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.254+0000 7f1ed8a00640 1 -- 172.21.15.43:0/2867438846 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e92beb5250 msgr2=0x55e92beb2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:58.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.254+0000 7f1ed8a00640 1 --2- 172.21.15.43:0/2867438846 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e92beb5250 0x55e92beb2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.254+0000 7f1ed8a00640 1 -- 172.21.15.43:0/2867438846 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e92bdca960 con 0x55e92beacbc0 2024-03-28T22:40:58.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.254+0000 7f1ed9e00640 1 --2- 172.21.15.43:0/2867438846 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e92beb5250 0x55e92beb2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:40:58.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.254+0000 7f1ed8a00640 1 --2- 172.21.15.43:0/2867438846 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e92beacbc0 0x55e92beb2610 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f1ec009abf0 tx=0x7f1ec009aaf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36d7e2a7d364381e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:58.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.254+0000 7f1ebaa00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:40:58.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.255+0000 7f1ebaa00640 1 -- 172.21.15.43:0/2867438846 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec009a540 con 0x55e92beacbc0 2024-03-28T22:40:58.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.255+0000 7f1ebaa00640 1 -- 172.21.15.43:0/2867438846 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f1ec0010430 con 0x55e92beacbc0 2024-03-28T22:40:58.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.255+0000 7f1edeb63740 1 -- 172.21.15.43:0/2867438846 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ecc004cc0 con 0x55e92beacbc0 2024-03-28T22:40:58.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.255+0000 7f1ebaa00640 1 -- 172.21.15.43:0/2867438846 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1ec009aef0 con 0x55e92beacbc0 2024-03-28T22:40:58.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.255+0000 7f1edeb63740 10 client.27053 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:58.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.255+0000 7f1edeb63740 1 -- 172.21.15.43:0/2867438846 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e92bbee7d0 con 0x55e92beacbc0 2024-03-28T22:40:58.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.255+0000 7f1ebaa00640 1 -- 172.21.15.43:0/2867438846 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ecc004cc0 con 0x55e92beacbc0 2024-03-28T22:40:58.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.256+0000 7f1ebaa00640 1 -- 172.21.15.43:0/2867438846 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f1ec0096c40 con 0x55e92beacbc0 2024-03-28T22:40:58.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.256+0000 7f1edeb63740 10 client.27053 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:58.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.256+0000 7f1edeb63740 10 client.27053 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:40:58.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.256+0000 7f1edeb63740 20 client.27053 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:58.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.256+0000 7f1edeb63740 1 --2- 172.21.15.43:0/2867438846 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55e92bec54c0 0x55e92bec7940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:58.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.256+0000 7f1edeb63740 4 client.27053 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:40:58.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.256+0000 7f1edeb63740 1 -- 172.21.15.43:0/2867438846 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e92be1d640 con 0x55e92bec54c0 2024-03-28T22:40:58.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.256+0000 7f1ed9400640 1 --2- 172.21.15.43:0/2867438846 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55e92bec54c0 0x55e92bec7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:58.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.257+0000 7f1ed9400640 1 --2- 172.21.15.43:0/2867438846 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55e92bec54c0 0x55e92bec7940 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e3bc796346bd0cc server_cookie=e65e7976ef7abfd6 in_seq=0 out_seq=0 2024-03-28T22:40:58.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.257+0000 7f1ebaa00640 10 client.27053 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:58.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.257+0000 7f1ebaa00640 1 -- 172.21.15.43:0/2867438846 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e92be1d640 con 0x55e92bec54c0 2024-03-28T22:40:58.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.257+0000 7f1ebaa00640 10 client.27053 handle_command_reply: tid=0 2024-03-28T22:40:58.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.257+0000 7f1edeb63740 1 client.27053 shutdown 2024-03-28T22:40:58.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.258+0000 7f1edeb63740 1 -- 172.21.15.43:0/2867438846 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e92beacbc0 msgr2=0x55e92beb2610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:58.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.258+0000 7f1edeb63740 1 --2- 172.21.15.43:0/2867438846 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e92beacbc0 0x55e92beb2610 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f1ec009abf0 tx=0x7f1ec009aaf0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.258+0000 7f1edeb63740 1 -- 172.21.15.43:0/2867438846 shutdown_connections 2024-03-28T22:40:58.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.258+0000 7f1edeb63740 1 --2- 172.21.15.43:0/2867438846 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55e92bec54c0 0x55e92bec7940 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.258+0000 7f1edeb63740 1 --2- 172.21.15.43:0/2867438846 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e92beacbc0 0x55e92beb2610 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.258+0000 7f1ebaa00640 0 client.27053 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:58.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.258+0000 7f1edeb63740 1 --2- 172.21.15.43:0/2867438846 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e92beb5250 0x55e92beb2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.258+0000 7f1edeb63740 1 --2- 172.21.15.43:0/2867438846 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e92beabee0 0x55e92beb20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.258+0000 7f1edeb63740 1 -- 172.21.15.43:0/2867438846 >> 172.21.15.43:0/2867438846 conn(0x55e92be72a10 msgr2=0x55e92bea8680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:58.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.258+0000 7f1edeb63740 1 -- 172.21.15.43:0/2867438846 shutdown_connections 2024-03-28T22:40:58.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.258+0000 7f1edeb63740 1 -- 172.21.15.43:0/2867438846 wait complete. 2024-03-28T22:40:58.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.258+0000 7f1edeb63740 20 client.27053 trim_cache size 0 max 16384 2024-03-28T22:40:58.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:58 smithi043 ceph-mon[31823]: scrub queued for path: / 2024-03-28T22:40:58.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:58 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:40:58.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:58 smithi043 ceph-mon[31823]: scrub summary: active paths [/] 2024-03-28T22:40:58.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:58 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:40:58.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:58 smithi043 ceph-mon[31823]: scrub summary: idle 2024-03-28T22:40:58.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.278+0000 7f1edeb63740 1 Processor -- start 2024-03-28T22:40:58.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.278+0000 7f1edeb63740 1 -- start start 2024-03-28T22:40:58.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.278+0000 7f1edeb63740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e92beabee0 0x55e92beb4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:58.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.278+0000 7f1edeb63740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e92beacbc0 0x55e92bec0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:58.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.278+0000 7f1edeb63740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e92beb5250 0x55e92bea8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:58.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.278+0000 7f1edeb63740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed41b0dd0 con 0x55e92beb5250 2024-03-28T22:40:58.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.278+0000 7f1edeb63740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed41b2910 con 0x55e92beacbc0 2024-03-28T22:40:58.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.278+0000 7f1edeb63740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed41b14a0 con 0x55e92beabee0 2024-03-28T22:40:58.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.279+0000 7f1ed9400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e92beabee0 0x55e92beb4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:58.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.279+0000 7f1ed9400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e92beabee0 0x55e92beb4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58762/0 (socket says 172.21.15.43:58762) 2024-03-28T22:40:58.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.279+0000 7f1ed8a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e92beacbc0 0x55e92bec0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:58.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.279+0000 7f1ed9400640 1 -- 172.21.15.43:0/1278125622 learned_addr learned my addr 172.21.15.43:0/1278125622 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:58.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.279+0000 7f1ed9e00640 1 --2- 172.21.15.43:0/1278125622 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e92beb5250 0x55e92bea8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:58.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.279+0000 7f1ed9400640 1 -- 172.21.15.43:0/1278125622 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e92beacbc0 msgr2=0x55e92bec0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:58.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.279+0000 7f1ed9400640 1 --2- 172.21.15.43:0/1278125622 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e92beacbc0 0x55e92bec0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.279+0000 7f1ed9400640 1 -- 172.21.15.43:0/1278125622 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e92beb5250 msgr2=0x55e92bea8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:58.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.279+0000 7f1ed9400640 1 --2- 172.21.15.43:0/1278125622 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e92beb5250 0x55e92bea8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.279+0000 7f1ed9400640 1 -- 172.21.15.43:0/1278125622 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e92bb243f0 con 0x55e92beabee0 2024-03-28T22:40:58.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.279+0000 7f1ed8a00640 1 --2- 172.21.15.43:0/1278125622 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e92beacbc0 0x55e92bec0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:58.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.279+0000 7f1ed9400640 1 --2- 172.21.15.43:0/1278125622 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e92beabee0 0x55e92beb4800 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8009900 tx=0x7f1ec8040000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7329beeda702fc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:58.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.280+0000 7f1ec5c00640 1 -- 172.21.15.43:0/1278125622 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec8056070 con 0x55e92beabee0 2024-03-28T22:40:58.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.280+0000 7f1ec5c00640 1 -- 172.21.15.43:0/1278125622 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ec802cd10 con 0x55e92beabee0 2024-03-28T22:40:58.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.280+0000 7f1ec5c00640 1 -- 172.21.15.43:0/1278125622 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec8005460 con 0x55e92beabee0 2024-03-28T22:40:58.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.280+0000 7f1edeb63740 1 -- 172.21.15.43:0/1278125622 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e92beabee0 msgr2=0x55e92beb4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:58.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.280+0000 7f1edeb63740 1 --2- 172.21.15.43:0/1278125622 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e92beabee0 0x55e92beb4800 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8009900 tx=0x7f1ec8040000 comp rx=0 tx=0).stop 2024-03-28T22:40:58.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.281+0000 7f1edeb63740 1 -- 172.21.15.43:0/1278125622 shutdown_connections 2024-03-28T22:40:58.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.281+0000 7f1edeb63740 1 --2- 172.21.15.43:0/1278125622 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e92beacbc0 0x55e92bec0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.281+0000 7f1edeb63740 1 --2- 172.21.15.43:0/1278125622 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e92beb5250 0x55e92bea8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.281+0000 7f1edeb63740 1 --2- 172.21.15.43:0/1278125622 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e92beabee0 0x55e92beb4800 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.281+0000 7f1edeb63740 1 -- 172.21.15.43:0/1278125622 >> 172.21.15.43:0/1278125622 conn(0x55e92be72a10 msgr2=0x55e92bea7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:58.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.281+0000 7f1edeb63740 1 -- 172.21.15.43:0/1278125622 shutdown_connections 2024-03-28T22:40:58.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.281+0000 7f1edeb63740 1 -- 172.21.15.43:0/1278125622 wait complete. 2024-03-28T22:40:58.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.281+0000 7f1edeb63740 1 -- start start 2024-03-28T22:40:58.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.281+0000 7f1edeb63740 1 Processor -- start 2024-03-28T22:40:58.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.282+0000 7f1edeb63740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e92beabee0 0x55e92beaece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:58.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.282+0000 7f1edeb63740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e92beacbc0 0x55e92beaf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:58.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.282+0000 7f1edeb63740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e92beb5250 0x55e92beaf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:58.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.282+0000 7f1edeb63740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed41b2240 con 0x55e92beacbc0 2024-03-28T22:40:58.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.282+0000 7f1edeb63740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed41b1b70 con 0x55e92beb5250 2024-03-28T22:40:58.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.282+0000 7f1edeb63740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed41aedb0 con 0x55e92beabee0 2024-03-28T22:40:58.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.282+0000 7f1edeb63740 10 client.0 _ll_register_callbacks cb 0x7f1ec009acc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:58.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.282+0000 7f1ed9400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e92beabee0 0x55e92beaece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:58.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.282+0000 7f1ed9400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e92beabee0 0x55e92beaece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58768/0 (socket says 172.21.15.43:58768) 2024-03-28T22:40:58.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.282+0000 7f1ed9400640 1 -- 172.21.15.43:0/2421932046 learned_addr learned my addr 172.21.15.43:0/2421932046 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:58.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.282+0000 7f1ed9e00640 1 --2- 172.21.15.43:0/2421932046 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e92beb5250 0x55e92beaf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:58.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.282+0000 7f1ed8a00640 1 --2- 172.21.15.43:0/2421932046 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e92beacbc0 0x55e92beaf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:58.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.282+0000 7f1ed9400640 1 -- 172.21.15.43:0/2421932046 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e92beb5250 msgr2=0x55e92beaf760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:58.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.282+0000 7f1ed9400640 1 --2- 172.21.15.43:0/2421932046 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e92beb5250 0x55e92beaf760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.282+0000 7f1ed9400640 1 -- 172.21.15.43:0/2421932046 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e92beacbc0 msgr2=0x55e92beaf220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:58.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.282+0000 7f1ed9400640 1 --2- 172.21.15.43:0/2421932046 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e92beacbc0 0x55e92beaf220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.282+0000 7f1ed9400640 1 -- 172.21.15.43:0/2421932046 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e92bae40f0 con 0x55e92beabee0 2024-03-28T22:40:58.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.282+0000 7f1ed8a00640 1 --2- 172.21.15.43:0/2421932046 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e92beacbc0 0x55e92beaf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:40:58.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.283+0000 7f1ed9400640 1 --2- 172.21.15.43:0/2421932046 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e92beabee0 0x55e92beaece0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8000c00 tx=0x7f1ec80052f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bd888acda896dd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:58.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.283+0000 7f1ebaa00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:40:58.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.283+0000 7f1ebaa00640 1 -- 172.21.15.43:0/2421932046 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec8056070 con 0x55e92beabee0 2024-03-28T22:40:58.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.283+0000 7f1edeb63740 1 -- 172.21.15.43:0/2421932046 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ec802cd10 con 0x55e92beabee0 2024-03-28T22:40:58.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.283+0000 7f1ebaa00640 1 -- 172.21.15.43:0/2421932046 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f1ec80046c0 con 0x55e92beabee0 2024-03-28T22:40:58.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.283+0000 7f1ebaa00640 1 -- 172.21.15.43:0/2421932046 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1ec8064050 con 0x55e92beabee0 2024-03-28T22:40:58.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.283+0000 7f1edeb63740 10 client.27013 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:58.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.283+0000 7f1edeb63740 1 -- 172.21.15.43:0/2421932046 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e92bf2ba50 con 0x55e92beabee0 2024-03-28T22:40:58.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.284+0000 7f1ebaa00640 1 -- 172.21.15.43:0/2421932046 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ec802cd10 con 0x55e92beabee0 2024-03-28T22:40:58.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.284+0000 7f1ebaa00640 1 -- 172.21.15.43:0/2421932046 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f1ec8003bc0 con 0x55e92beabee0 2024-03-28T22:40:58.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.284+0000 7f1edeb63740 10 client.27013 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:58.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.284+0000 7f1edeb63740 10 client.27013 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:40:58.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.284+0000 7f1edeb63740 20 client.27013 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:58.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.284+0000 7f1edeb63740 1 --2- 172.21.15.43:0/2421932046 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55e92bf2c0c0 0x55e92bf2e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:58.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.284+0000 7f1edeb63740 4 client.27013 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:40:58.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.284+0000 7f1edeb63740 1 -- 172.21.15.43:0/2421932046 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e92be17990 con 0x55e92bf2c0c0 2024-03-28T22:40:58.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.284+0000 7f1ed8a00640 1 --2- 172.21.15.43:0/2421932046 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55e92bf2c0c0 0x55e92bf2e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:58.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.285+0000 7f1ebaa00640 10 client.27013 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:58.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.285+0000 7f1ed8a00640 1 --2- 172.21.15.43:0/2421932046 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55e92bf2c0c0 0x55e92bf2e4e0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47a52776b8f05bd7 server_cookie=5c5e7e818e0e3b47 in_seq=0 out_seq=0 2024-03-28T22:40:58.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.285+0000 7f1ebaa00640 1 -- 172.21.15.43:0/2421932046 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e92be17990 con 0x55e92bf2c0c0 2024-03-28T22:40:58.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.285+0000 7f1ebaa00640 10 client.27013 handle_command_reply: tid=0 2024-03-28T22:40:58.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.285+0000 7f1edeb63740 1 client.27013 shutdown 2024-03-28T22:40:58.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.285+0000 7f1edeb63740 1 -- 172.21.15.43:0/2421932046 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e92beabee0 msgr2=0x55e92beaece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:58.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.285+0000 7f1edeb63740 1 --2- 172.21.15.43:0/2421932046 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e92beabee0 0x55e92beaece0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8000c00 tx=0x7f1ec80052f0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.285+0000 7f1edeb63740 1 -- 172.21.15.43:0/2421932046 shutdown_connections 2024-03-28T22:40:58.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.285+0000 7f1edeb63740 1 --2- 172.21.15.43:0/2421932046 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55e92bf2c0c0 0x55e92bf2e4e0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.285+0000 7f1edeb63740 1 --2- 172.21.15.43:0/2421932046 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e92beb5250 0x55e92beaf760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.285+0000 7f1edeb63740 1 --2- 172.21.15.43:0/2421932046 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e92beacbc0 0x55e92beaf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.285+0000 7f1edeb63740 1 --2- 172.21.15.43:0/2421932046 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e92beabee0 0x55e92beaece0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.285+0000 7f1ebaa00640 0 client.27013 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:40:58.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.285+0000 7f1edeb63740 1 -- 172.21.15.43:0/2421932046 >> 172.21.15.43:0/2421932046 conn(0x55e92be72a10 msgr2=0x55e92bea5ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:58.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.285+0000 7f1edeb63740 1 -- 172.21.15.43:0/2421932046 shutdown_connections 2024-03-28T22:40:58.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.285+0000 7f1edeb63740 1 -- 172.21.15.43:0/2421932046 wait complete. 2024-03-28T22:40:58.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.286+0000 7f1edeb63740 20 client.27013 trim_cache size 0 max 16384 2024-03-28T22:40:58.290 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:40:58.290 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-03-28T22:40:58.290 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-03-28T22:40:58.290 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:40:58.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.286+0000 7f1edbe00640 1 -- 172.21.15.43:0/1969276257 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1ebc0a1f70 msgr2=0x7f1ebc0a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:58.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.286+0000 7f1edbe00640 1 --2- 172.21.15.43:0/1969276257 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1ebc0a1f70 0x7f1ebc0a4430 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8006010 tx=0x7f1ec8005b50 comp rx=0 tx=0).stop 2024-03-28T22:40:58.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.286+0000 7f1edbe00640 1 -- 172.21.15.43:0/1969276257 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1ed41042b0 msgr2=0x7f1ed419a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:58.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.286+0000 7f1edbe00640 1 --2- 172.21.15.43:0/1969276257 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1ed41042b0 0x7f1ed419a7d0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f1ec000e9f0 tx=0x7f1ec000eeb0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.286+0000 7f1edbe00640 1 -- 172.21.15.43:0/1969276257 shutdown_connections 2024-03-28T22:40:58.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.286+0000 7f1edbe00640 1 --2- 172.21.15.43:0/1969276257 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1ebc0a1f70 0x7f1ebc0a4430 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.286+0000 7f1edbe00640 1 --2- 172.21.15.43:0/1969276257 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1ed41042b0 0x7f1ed419a7d0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.286+0000 7f1edbe00640 1 --2- 172.21.15.43:0/1969276257 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ed4109900 0x7f1ed419eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.286+0000 7f1edbe00640 1 --2- 172.21.15.43:0/1969276257 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ed41038f0 0x7f1ed419a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.286+0000 7f1edbe00640 1 -- 172.21.15.43:0/1969276257 >> 172.21.15.43:0/1969276257 conn(0x7f1ed40ff3c0 msgr2=0x7f1ed40ffa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:58.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.287+0000 7f1edbe00640 1 -- 172.21.15.43:0/1969276257 shutdown_connections 2024-03-28T22:40:58.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.287+0000 7f1edbe00640 1 -- 172.21.15.43:0/1969276257 wait complete. 2024-03-28T22:40:58.305 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:40:58.305 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:40:58.355 INFO:tasks.workunit.client.0.smithi043.stdout: 1 230555 8.45 MB/sec execute 29 sec latency 83.369 ms 2024-03-28T22:40:58.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:58 smithi082 ceph-mon[36471]: scrub queued for path: / 2024-03-28T22:40:58.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:58 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:40:58.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:58 smithi082 ceph-mon[36471]: scrub summary: active paths [/] 2024-03-28T22:40:58.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:58 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:40:58.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:58 smithi082 ceph-mon[36471]: scrub summary: idle 2024-03-28T22:40:58.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.514+0000 7f9045e00640 1 -- 172.21.15.43:0/581462783 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9040103720 msgr2=0x7f9040103ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:58.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.514+0000 7f9045e00640 1 --2- 172.21.15.43:0/581462783 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9040103720 0x7f9040103ba0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f902c009900 tx=0x7f902c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:40:58.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.514+0000 7f9045e00640 1 -- 172.21.15.43:0/581462783 shutdown_connections 2024-03-28T22:40:58.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.514+0000 7f9045e00640 1 --2- 172.21.15.43:0/581462783 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9040103720 0x7f9040103ba0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.514+0000 7f9045e00640 1 --2- 172.21.15.43:0/581462783 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f90401040e0 0x7f90401105f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.514+0000 7f9045e00640 1 --2- 172.21.15.43:0/581462783 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9040109730 0x7f9040109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.514+0000 7f9045e00640 1 -- 172.21.15.43:0/581462783 >> 172.21.15.43:0/581462783 conn(0x7f90400ff1f0 msgr2=0x7f9040101610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:58.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.514+0000 7f9045e00640 1 -- 172.21.15.43:0/581462783 shutdown_connections 2024-03-28T22:40:58.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.514+0000 7f9045e00640 1 -- 172.21.15.43:0/581462783 wait complete. 2024-03-28T22:40:58.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.516+0000 7f9045e00640 1 Processor -- start 2024-03-28T22:40:58.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.516+0000 7f9045e00640 1 -- start start 2024-03-28T22:40:58.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.517+0000 7f9045e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9040103720 0x7f9040199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:58.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.517+0000 7f9045e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90401040e0 0x7f904019a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:58.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.517+0000 7f9045e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9040109730 0x7f904019e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:58.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.517+0000 7f9045e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9040113b50 con 0x7f9040109730 2024-03-28T22:40:58.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.517+0000 7f9045e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f90401139d0 con 0x7f9040103720 2024-03-28T22:40:58.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.517+0000 7f9045e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9040113cd0 con 0x7f90401040e0 2024-03-28T22:40:58.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.517+0000 7f9044a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9040103720 0x7f9040199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:58.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.517+0000 7f9044a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9040103720 0x7f9040199fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:60034/0 (socket says 172.21.15.43:60034) 2024-03-28T22:40:58.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.517+0000 7f9044a00640 1 -- 172.21.15.43:0/2585352206 learned_addr learned my addr 172.21.15.43:0/2585352206 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:58.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.517+0000 7f903fe00640 1 --2- 172.21.15.43:0/2585352206 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90401040e0 0x7f904019a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:58.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.517+0000 7f9045400640 1 --2- 172.21.15.43:0/2585352206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9040109730 0x7f904019e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:58.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.517+0000 7f9044a00640 1 -- 172.21.15.43:0/2585352206 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90401040e0 msgr2=0x7f904019a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:58.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.517+0000 7f9044a00640 1 --2- 172.21.15.43:0/2585352206 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90401040e0 0x7f904019a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.517+0000 7f9044a00640 1 -- 172.21.15.43:0/2585352206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9040109730 msgr2=0x7f904019e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:58.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.517+0000 7f9044a00640 1 --2- 172.21.15.43:0/2585352206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9040109730 0x7f904019e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.517+0000 7f9044a00640 1 -- 172.21.15.43:0/2585352206 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f904019efc0 con 0x7f9040103720 2024-03-28T22:40:58.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.517+0000 7f903fe00640 1 --2- 172.21.15.43:0/2585352206 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90401040e0 0x7f904019a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:58.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.518+0000 7f9044a00640 1 --2- 172.21.15.43:0/2585352206 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9040103720 0x7f9040199fd0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f903400ebc0 tx=0x7f903400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:58.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.518+0000 7f903d600640 1 -- 172.21.15.43:0/2585352206 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9034010630 con 0x7f9040103720 2024-03-28T22:40:58.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.518+0000 7f903d600640 1 -- 172.21.15.43:0/2585352206 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9034010cb0 con 0x7f9040103720 2024-03-28T22:40:58.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.518+0000 7f9045e00640 1 -- 172.21.15.43:0/2585352206 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f904019f2b0 con 0x7f9040103720 2024-03-28T22:40:58.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.518+0000 7f903d600640 1 -- 172.21.15.43:0/2585352206 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9034018bd0 con 0x7f9040103720 2024-03-28T22:40:58.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.518+0000 7f9045e00640 1 -- 172.21.15.43:0/2585352206 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90401a6b40 con 0x7f9040103720 2024-03-28T22:40:58.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.520+0000 7f9045e00640 1 -- 172.21.15.43:0/2585352206 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90400688b0 con 0x7f9040103720 2024-03-28T22:40:58.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.522+0000 7f903d600640 1 -- 172.21.15.43:0/2585352206 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f90340107d0 con 0x7f9040103720 2024-03-28T22:40:58.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.523+0000 7f903d600640 1 --2- 172.21.15.43:0/2585352206 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f90280a1ef0 0x7f90280a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:58.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.523+0000 7f903d600640 1 -- 172.21.15.43:0/2585352206 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f90340ceb20 con 0x7f9040103720 2024-03-28T22:40:58.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.523+0000 7f903fe00640 1 --2- 172.21.15.43:0/2585352206 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f90280a1ef0 0x7f90280a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:58.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.524+0000 7f903fe00640 1 --2- 172.21.15.43:0/2585352206 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f90280a1ef0 0x7f90280a43b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f902c02efd0 tx=0x7f902c000bc0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:58.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.530+0000 7f903d600640 1 -- 172.21.15.43:0/2585352206 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f903409b1c0 con 0x7f9040103720 2024-03-28T22:40:58.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:58 smithi172 ceph-mon[36566]: scrub queued for path: / 2024-03-28T22:40:58.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:58 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:40:58.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:58 smithi172 ceph-mon[36566]: scrub summary: active paths [/] 2024-03-28T22:40:58.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:58 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:40:58.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:58 smithi172 ceph-mon[36566]: scrub summary: idle 2024-03-28T22:40:58.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.791+0000 7f9045e00640 1 -- 172.21.15.43:0/2585352206 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f90401a6e30 con 0x7f9040103720 2024-03-28T22:40:58.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.792+0000 7f903d600640 1 -- 172.21.15.43:0/2585352206 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11931 (secure 0 0 0) 0x7f903409f260 con 0x7f9040103720 2024-03-28T22:40:58.791 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:40:58.791 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:33:28.906396+0000","modified":"2024-03-28T22:40:08.672168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24572,"mds_1":24556,"mds_2":24535,"mds_3":24523,"mds_4":24518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26717":{"gid":26717,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/215779342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":215779342},{"type":"v1","addr":"172.21.15.82:6835","nonce":215779342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24535,24556,24518,24523,24572]},"id":1}]} 2024-03-28T22:40:58.791 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 32 2024-03-28T22:40:58.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.795+0000 7f9045e00640 1 -- 172.21.15.43:0/2585352206 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f90280a1ef0 msgr2=0x7f90280a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:58.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.795+0000 7f9045e00640 1 --2- 172.21.15.43:0/2585352206 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f90280a1ef0 0x7f90280a43b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f902c02efd0 tx=0x7f902c000bc0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.795+0000 7f9045e00640 1 -- 172.21.15.43:0/2585352206 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9040103720 msgr2=0x7f9040199fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:58.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.795+0000 7f9045e00640 1 --2- 172.21.15.43:0/2585352206 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9040103720 0x7f9040199fd0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f903400ebc0 tx=0x7f903400c6a0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.795+0000 7f9045e00640 1 -- 172.21.15.43:0/2585352206 shutdown_connections 2024-03-28T22:40:58.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.795+0000 7f9045e00640 1 --2- 172.21.15.43:0/2585352206 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f90280a1ef0 0x7f90280a43b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.795+0000 7f9045e00640 1 --2- 172.21.15.43:0/2585352206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9040109730 0x7f904019e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.795+0000 7f9045e00640 1 --2- 172.21.15.43:0/2585352206 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f90401040e0 0x7f904019a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.795+0000 7f9045e00640 1 --2- 172.21.15.43:0/2585352206 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9040103720 0x7f9040199fd0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:58.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.795+0000 7f9045e00640 1 -- 172.21.15.43:0/2585352206 >> 172.21.15.43:0/2585352206 conn(0x7f90400ff1f0 msgr2=0x7f90400ff730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:58.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.796+0000 7f9045e00640 1 -- 172.21.15.43:0/2585352206 shutdown_connections 2024-03-28T22:40:58.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:58.796+0000 7f9045e00640 1 -- 172.21.15.43:0/2585352206 wait complete. 2024-03-28T22:40:58.813 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T22:40:59.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.050+0000 7f1167e00640 1 -- 172.21.15.43:0/708822312 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f116010df60 msgr2=0x7f1160110370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.050+0000 7f1167e00640 1 --2- 172.21.15.43:0/708822312 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f116010df60 0x7f1160110370 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1158009ca0 tx=0x7f115802c190 comp rx=0 tx=0).stop 2024-03-28T22:40:59.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.050+0000 7f1167e00640 1 -- 172.21.15.43:0/708822312 shutdown_connections 2024-03-28T22:40:59.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.050+0000 7f1167e00640 1 --2- 172.21.15.43:0/708822312 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f116010df60 0x7f1160110370 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.050+0000 7f1167e00640 1 --2- 172.21.15.43:0/708822312 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1160074440 0x7f116010da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.050+0000 7f1167e00640 1 --2- 172.21.15.43:0/708822312 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1160073b20 0x7f1160073f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.050+0000 7f1167e00640 1 -- 172.21.15.43:0/708822312 >> 172.21.15.43:0/708822312 conn(0x7f11600fd1b0 msgr2=0x7f11600ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:59.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.051+0000 7f1167e00640 1 -- 172.21.15.43:0/708822312 shutdown_connections 2024-03-28T22:40:59.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.051+0000 7f1167e00640 1 -- 172.21.15.43:0/708822312 wait complete. 2024-03-28T22:40:59.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.052+0000 7f1167e00640 1 Processor -- start 2024-03-28T22:40:59.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.052+0000 7f1167e00640 1 -- start start 2024-03-28T22:40:59.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.052+0000 7f1167e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1160073b20 0x7f116019a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.052+0000 7f1167e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1160074440 0x7f116019a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.052+0000 7f1167e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f116010df60 0x7f116019eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.052+0000 7f1167e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1160113cc0 con 0x7f116010df60 2024-03-28T22:40:59.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.052+0000 7f1167e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1160113b40 con 0x7f1160073b20 2024-03-28T22:40:59.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.052+0000 7f1167e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f1160113e40 con 0x7f1160074440 2024-03-28T22:40:59.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.052+0000 7f1165400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1160073b20 0x7f116019a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.052+0000 7f1165400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1160073b20 0x7f116019a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:60060/0 (socket says 172.21.15.43:60060) 2024-03-28T22:40:59.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.052+0000 7f1164a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1160074440 0x7f116019a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.052+0000 7f1165400640 1 -- 172.21.15.43:0/3992846104 learned_addr learned my addr 172.21.15.43:0/3992846104 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:59.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.052+0000 7f1165e00640 1 --2- 172.21.15.43:0/3992846104 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f116010df60 0x7f116019eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.053+0000 7f1165400640 1 -- 172.21.15.43:0/3992846104 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1160074440 msgr2=0x7f116019a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.053+0000 7f1165400640 1 --2- 172.21.15.43:0/3992846104 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1160074440 0x7f116019a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.053+0000 7f1165400640 1 -- 172.21.15.43:0/3992846104 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f116010df60 msgr2=0x7f116019eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.053+0000 7f1165400640 1 --2- 172.21.15.43:0/3992846104 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f116010df60 0x7f116019eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.053+0000 7f1165400640 1 -- 172.21.15.43:0/3992846104 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f116019f230 con 0x7f1160073b20 2024-03-28T22:40:59.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.053+0000 7f1165400640 1 --2- 172.21.15.43:0/3992846104 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1160073b20 0x7f116019a240 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f115400e9f0 tx=0x7f115400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:59.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.053+0000 7f115e000640 1 -- 172.21.15.43:0/3992846104 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f115400cd80 con 0x7f1160073b20 2024-03-28T22:40:59.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.054+0000 7f115e000640 1 -- 172.21.15.43:0/3992846104 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1154005500 con 0x7f1160073b20 2024-03-28T22:40:59.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.054+0000 7f115e000640 1 -- 172.21.15.43:0/3992846104 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1154010690 con 0x7f1160073b20 2024-03-28T22:40:59.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.054+0000 7f1167e00640 1 -- 172.21.15.43:0/3992846104 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f116019f520 con 0x7f1160073b20 2024-03-28T22:40:59.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.054+0000 7f1167e00640 1 -- 172.21.15.43:0/3992846104 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11601a6db0 con 0x7f1160073b20 2024-03-28T22:40:59.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.058+0000 7f116ab97740 1 Processor -- start 2024-03-28T22:40:59.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.058+0000 7f116ab97740 1 -- start start 2024-03-28T22:40:59.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.059+0000 7f116ab97740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562eede5b190 0x562eede5b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.059+0000 7f116ab97740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562eede5bb40 0x562eede6cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.059+0000 7f116ab97740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562eede6d090 0x562eede6f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.059+0000 7f116ab97740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562eeddbf050 con 0x562eede5bb40 2024-03-28T22:40:59.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.059+0000 7f116ab97740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x562eedd67ce0 con 0x562eede6d090 2024-03-28T22:40:59.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.059+0000 7f116ab97740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x562eeddbb0f0 con 0x562eede5b190 2024-03-28T22:40:59.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.059+0000 7f115e000640 1 -- 172.21.15.43:0/3992846104 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f11540046c0 con 0x7f1160073b20 2024-03-28T22:40:59.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.059+0000 7f1165e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562eede5b190 0x562eede5b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.059+0000 7f1165e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562eede5b190 0x562eede5b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58808/0 (socket says 172.21.15.43:58808) 2024-03-28T22:40:59.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.059+0000 7f1165e00640 1 -- 172.21.15.43:0/866682448 learned_addr learned my addr 172.21.15.43:0/866682448 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:59.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.059+0000 7f1164a00640 1 --2- 172.21.15.43:0/866682448 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562eede6d090 0x562eede6f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.060+0000 7f1165400640 1 --2- 172.21.15.43:0/866682448 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562eede5bb40 0x562eede6cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.060+0000 7f1165e00640 1 -- 172.21.15.43:0/866682448 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562eede6d090 msgr2=0x562eede6f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:59.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.060+0000 7f1165e00640 1 --2- 172.21.15.43:0/866682448 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562eede6d090 0x562eede6f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.060+0000 7f1165e00640 1 -- 172.21.15.43:0/866682448 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562eede5bb40 msgr2=0x562eede6cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:59.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.060+0000 7f1165e00640 1 --2- 172.21.15.43:0/866682448 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562eede5bb40 0x562eede6cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.060+0000 7f1165e00640 1 -- 172.21.15.43:0/866682448 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562eedad33f0 con 0x562eede5b190 2024-03-28T22:40:59.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.060+0000 7f1164a00640 1 --2- 172.21.15.43:0/866682448 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562eede6d090 0x562eede6f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:59.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.060+0000 7f1165400640 1 --2- 172.21.15.43:0/866682448 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562eede5bb40 0x562eede6cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:40:59.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.060+0000 7f1165e00640 1 --2- 172.21.15.43:0/866682448 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562eede5b190 0x562eede5b570 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f1158004af0 tx=0x7f1158005f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7845bae472f9560 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:59.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.060+0000 7f1152600640 1 -- 172.21.15.43:0/866682448 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1158043020 con 0x562eede5b190 2024-03-28T22:40:59.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.060+0000 7f1152600640 1 -- 172.21.15.43:0/866682448 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1158003e50 con 0x562eede5b190 2024-03-28T22:40:59.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.060+0000 7f1152600640 1 -- 172.21.15.43:0/866682448 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1158039040 con 0x562eede5b190 2024-03-28T22:40:59.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.061+0000 7f115e000640 1 --2- 172.21.15.43:0/3992846104 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f11480a1f70 0x7f11480a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.061+0000 7f116ab97740 1 -- 172.21.15.43:0/866682448 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562eede5b190 msgr2=0x562eede5b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.061+0000 7f116ab97740 1 --2- 172.21.15.43:0/866682448 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562eede5b190 0x562eede5b570 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f1158004af0 tx=0x7f1158005f80 comp rx=0 tx=0).stop 2024-03-28T22:40:59.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.061+0000 7f115e000640 1 -- 172.21.15.43:0/3992846104 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f11540ce920 con 0x7f1160073b20 2024-03-28T22:40:59.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.061+0000 7f1165400640 1 --2- 172.21.15.43:0/3992846104 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f11480a1f70 0x7f11480a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.061+0000 7f116ab97740 1 -- 172.21.15.43:0/866682448 shutdown_connections 2024-03-28T22:40:59.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.061+0000 7f116ab97740 1 --2- 172.21.15.43:0/866682448 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562eede6d090 0x562eede6f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.061+0000 7f116ab97740 1 --2- 172.21.15.43:0/866682448 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562eede5bb40 0x562eede6cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.061+0000 7f116ab97740 1 --2- 172.21.15.43:0/866682448 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562eede5b190 0x562eede5b570 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.061+0000 7f116ab97740 1 -- 172.21.15.43:0/866682448 >> 172.21.15.43:0/866682448 conn(0x562eede55ab0 msgr2=0x562eede55e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:59.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.062+0000 7f116ab97740 1 -- 172.21.15.43:0/866682448 shutdown_connections 2024-03-28T22:40:59.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.062+0000 7f116ab97740 1 -- 172.21.15.43:0/866682448 wait complete. 2024-03-28T22:40:59.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.062+0000 7f1165400640 1 --2- 172.21.15.43:0/3992846104 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f11480a1f70 0x7f11480a4430 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x562eede56d40 tx=0x7f1154004a40 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:59.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.062+0000 7f116ab97740 1 -- start start 2024-03-28T22:40:59.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.062+0000 7f116ab97740 1 Processor -- start 2024-03-28T22:40:59.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.063+0000 7f116ab97740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562eede5b190 0x562eede6a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.063+0000 7f116ab97740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562eede5bb40 0x562eede6a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.063+0000 7f116ab97740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562eede6d090 0x562eede6abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.063+0000 7f116ab97740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562eede6fdf0 con 0x562eede5bb40 2024-03-28T22:40:59.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.063+0000 7f116ab97740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x562eedd5d2f0 con 0x562eede5b190 2024-03-28T22:40:59.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.063+0000 7f116ab97740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x562eede6ff70 con 0x562eede6d090 2024-03-28T22:40:59.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.063+0000 7f116ab97740 10 client.0 _ll_register_callbacks cb 0x562eedad2be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:59.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.063+0000 7f1165e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562eede5b190 0x562eede6a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.063+0000 7f1164a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562eede5bb40 0x562eede6a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.063+0000 7f1165e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562eede5b190 0x562eede6a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:60066/0 (socket says 172.21.15.43:60066) 2024-03-28T22:40:59.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.063+0000 7f1165e00640 1 -- 172.21.15.43:0/3449320045 learned_addr learned my addr 172.21.15.43:0/3449320045 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:59.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.063+0000 7f1165e00640 1 --2- 172.21.15.43:0/3449320045 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562eede6d090 0x562eede6abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.064+0000 7f1165e00640 1 -- 172.21.15.43:0/3449320045 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562eede6d090 msgr2=0x562eede6abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:59.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.064+0000 7f1165e00640 1 --2- 172.21.15.43:0/3449320045 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562eede6d090 0x562eede6abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.064+0000 7f1165e00640 1 -- 172.21.15.43:0/3449320045 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562eede5bb40 msgr2=0x562eede6a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:59.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.064+0000 7f1165e00640 1 --2- 172.21.15.43:0/3449320045 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562eede5bb40 0x562eede6a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.064+0000 7f1165e00640 1 -- 172.21.15.43:0/3449320045 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562eedb9d7d0 con 0x562eede5b190 2024-03-28T22:40:59.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.064+0000 7f1164a00640 1 --2- 172.21.15.43:0/3449320045 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562eede5bb40 0x562eede6a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:59.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.064+0000 7f1165e00640 1 --2- 172.21.15.43:0/3449320045 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562eede5b190 0x562eede6a160 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f1158004800 tx=0x7f1158004700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df407e96896f5224 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:59.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.064+0000 7f1146a00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:40:59.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.064+0000 7f1146a00640 1 -- 172.21.15.43:0/3449320045 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f115804c070 con 0x562eede5b190 2024-03-28T22:40:59.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.064+0000 7f116ab97740 1 -- 172.21.15.43:0/3449320045 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1158003e50 con 0x562eede5b190 2024-03-28T22:40:59.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.064+0000 7f1146a00640 1 -- 172.21.15.43:0/3449320045 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f11580043a0 con 0x562eede5b190 2024-03-28T22:40:59.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.064+0000 7f1146a00640 1 -- 172.21.15.43:0/3449320045 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1158043040 con 0x562eede5b190 2024-03-28T22:40:59.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.065+0000 7f116ab97740 10 client.27077 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:59.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.065+0000 7f116ab97740 1 -- 172.21.15.43:0/3449320045 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562eede7c100 con 0x562eede5b190 2024-03-28T22:40:59.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.065+0000 7f1146a00640 1 -- 172.21.15.43:0/3449320045 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1158003e50 con 0x562eede5b190 2024-03-28T22:40:59.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.065+0000 7f1146a00640 1 -- 172.21.15.43:0/3449320045 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f115803f6e0 con 0x562eede5b190 2024-03-28T22:40:59.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.065+0000 7f116ab97740 10 client.27077 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:59.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.065+0000 7f116ab97740 10 client.27077 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:40:59.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.065+0000 7f116ab97740 20 client.27077 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:59.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.065+0000 7f116ab97740 1 --2- 172.21.15.43:0/3449320045 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x562eede7d450 0x562eede7f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.065+0000 7f116ab97740 4 client.27077 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:40:59.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.065+0000 7f116ab97740 1 -- 172.21.15.43:0/3449320045 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562eedd68680 con 0x562eede7d450 2024-03-28T22:40:59.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.065+0000 7f1164a00640 1 --2- 172.21.15.43:0/3449320045 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x562eede7d450 0x562eede7f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.066+0000 7f1164a00640 1 --2- 172.21.15.43:0/3449320045 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x562eede7d450 0x562eede7f840 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=be962f68f1af4009 server_cookie=864769415b830511 in_seq=0 out_seq=0 2024-03-28T22:40:59.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.066+0000 7f1146a00640 10 client.27077 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:40:59.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.067+0000 7f1146a00640 1 -- 172.21.15.43:0/3449320045 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562eedd68680 con 0x562eede7d450 2024-03-28T22:40:59.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.067+0000 7f1146a00640 10 client.27077 handle_command_reply: tid=0 2024-03-28T22:40:59.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.067+0000 7f116ab97740 1 client.27077 shutdown 2024-03-28T22:40:59.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.067+0000 7f116ab97740 1 -- 172.21.15.43:0/3449320045 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562eede5b190 msgr2=0x562eede6a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.067+0000 7f116ab97740 1 --2- 172.21.15.43:0/3449320045 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562eede5b190 0x562eede6a160 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f1158004800 tx=0x7f1158004700 comp rx=0 tx=0).stop 2024-03-28T22:40:59.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.067+0000 7f116ab97740 1 -- 172.21.15.43:0/3449320045 shutdown_connections 2024-03-28T22:40:59.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.067+0000 7f116ab97740 1 --2- 172.21.15.43:0/3449320045 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x562eede7d450 0x562eede7f840 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.067+0000 7f116ab97740 1 --2- 172.21.15.43:0/3449320045 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562eede6d090 0x562eede6abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.067+0000 7f116ab97740 1 --2- 172.21.15.43:0/3449320045 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562eede5bb40 0x562eede6a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.067+0000 7f1146a00640 0 client.27077 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:40:59.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.067+0000 7f116ab97740 1 --2- 172.21.15.43:0/3449320045 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562eede5b190 0x562eede6a160 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.067+0000 7f116ab97740 1 -- 172.21.15.43:0/3449320045 >> 172.21.15.43:0/3449320045 conn(0x562eede55ab0 msgr2=0x562eede57800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:59.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.067+0000 7f116ab97740 1 -- 172.21.15.43:0/3449320045 shutdown_connections 2024-03-28T22:40:59.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.067+0000 7f116ab97740 1 -- 172.21.15.43:0/3449320045 wait complete. 2024-03-28T22:40:59.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.067+0000 7f116ab97740 20 client.27077 trim_cache size 0 max 16384 2024-03-28T22:40:59.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.084+0000 7f116ab97740 1 Processor -- start 2024-03-28T22:40:59.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.084+0000 7f116ab97740 1 -- start start 2024-03-28T22:40:59.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.085+0000 7f116ab97740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562eede5b190 0x562eede6c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.085+0000 7f116ab97740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562eede5bb40 0x562eede78c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.085+0000 7f116ab97740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562eede6d090 0x562eede57610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.085+0000 7f116ab97740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f11601aca10 con 0x562eede5b190 2024-03-28T22:40:59.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.085+0000 7f116ab97740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f11601ae550 con 0x562eede6d090 2024-03-28T22:40:59.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.085+0000 7f116ab97740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f11601ad0e0 con 0x562eede5bb40 2024-03-28T22:40:59.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.085+0000 7f1165e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562eede5b190 0x562eede6c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.085+0000 7f1164a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562eede5bb40 0x562eede78c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.085+0000 7f1164a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562eede5bb40 0x562eede78c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58832/0 (socket says 172.21.15.43:58832) 2024-03-28T22:40:59.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.085+0000 7f1164a00640 1 -- 172.21.15.43:0/4289377065 learned_addr learned my addr 172.21.15.43:0/4289377065 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:59.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.085+0000 7f1165e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562eede5b190 0x562eede6c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42016/0 (socket says 172.21.15.43:42016) 2024-03-28T22:40:59.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.085+0000 7f1165e00640 1 --2- 172.21.15.43:0/4289377065 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562eede6d090 0x562eede57610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.085+0000 7f1164a00640 1 -- 172.21.15.43:0/4289377065 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562eede6d090 msgr2=0x562eede57610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:59.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.085+0000 7f1164a00640 1 --2- 172.21.15.43:0/4289377065 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562eede6d090 0x562eede57610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.085+0000 7f1164a00640 1 -- 172.21.15.43:0/4289377065 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562eede5b190 msgr2=0x562eede6c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:59.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.085+0000 7f1164a00640 1 --2- 172.21.15.43:0/4289377065 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562eede5b190 0x562eede6c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.085+0000 7f1164a00640 1 -- 172.21.15.43:0/4289377065 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562eede6f7f0 con 0x562eede5bb40 2024-03-28T22:40:59.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.085+0000 7f1165e00640 1 --2- 172.21.15.43:0/4289377065 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562eede6d090 0x562eede57610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:40:59.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.086+0000 7f1164a00640 1 --2- 172.21.15.43:0/4289377065 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562eede5bb40 0x562eede78c00 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f114c005d00 tx=0x7f114c005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c00ee505239dc36 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:59.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.086+0000 7f1151c00640 1 -- 172.21.15.43:0/4289377065 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114c005020 con 0x562eede5bb40 2024-03-28T22:40:59.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.086+0000 7f1151c00640 1 -- 172.21.15.43:0/4289377065 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f114c022170 con 0x562eede5bb40 2024-03-28T22:40:59.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.086+0000 7f1151c00640 1 -- 172.21.15.43:0/4289377065 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114c00fa50 con 0x562eede5bb40 2024-03-28T22:40:59.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.087+0000 7f116ab97740 1 -- 172.21.15.43:0/4289377065 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562eede5bb40 msgr2=0x562eede78c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.087+0000 7f116ab97740 1 --2- 172.21.15.43:0/4289377065 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562eede5bb40 0x562eede78c00 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f114c005d00 tx=0x7f114c005f90 comp rx=0 tx=0).stop 2024-03-28T22:40:59.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.087+0000 7f116ab97740 1 -- 172.21.15.43:0/4289377065 shutdown_connections 2024-03-28T22:40:59.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.087+0000 7f116ab97740 1 --2- 172.21.15.43:0/4289377065 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562eede6d090 0x562eede57610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.087+0000 7f116ab97740 1 --2- 172.21.15.43:0/4289377065 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562eede5bb40 0x562eede78c00 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.087+0000 7f116ab97740 1 --2- 172.21.15.43:0/4289377065 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562eede5b190 0x562eede6c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.087+0000 7f116ab97740 1 -- 172.21.15.43:0/4289377065 >> 172.21.15.43:0/4289377065 conn(0x562eede55ab0 msgr2=0x562eede56f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:59.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.087+0000 7f116ab97740 1 -- 172.21.15.43:0/4289377065 shutdown_connections 2024-03-28T22:40:59.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.087+0000 7f116ab97740 1 -- 172.21.15.43:0/4289377065 wait complete. 2024-03-28T22:40:59.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.087+0000 7f116ab97740 1 -- start start 2024-03-28T22:40:59.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.087+0000 7f116ab97740 1 Processor -- start 2024-03-28T22:40:59.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.088+0000 7f116ab97740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562eede5b190 0x562eede70a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.088+0000 7f116ab97740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562eede5bb40 0x562eede70fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.088+0000 7f116ab97740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562eede6d090 0x562eede71510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.088+0000 7f116ab97740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f11601ade80 con 0x562eede5bb40 2024-03-28T22:40:59.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.088+0000 7f116ab97740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f11601ad7b0 con 0x562eede5b190 2024-03-28T22:40:59.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.088+0000 7f116ab97740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f11601aa9f0 con 0x562eede6d090 2024-03-28T22:40:59.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.088+0000 7f116ab97740 10 client.0 _ll_register_callbacks cb 0x7f114c0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:59.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.088+0000 7f1165e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562eede5b190 0x562eede70a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.088+0000 7f1165e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562eede5b190 0x562eede70a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:60090/0 (socket says 172.21.15.43:60090) 2024-03-28T22:40:59.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.088+0000 7f1165e00640 1 -- 172.21.15.43:0/4184361197 learned_addr learned my addr 172.21.15.43:0/4184361197 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:59.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.088+0000 7f1165e00640 1 --2- 172.21.15.43:0/4184361197 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562eede6d090 0x562eede71510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.089+0000 7f1164a00640 1 --2- 172.21.15.43:0/4184361197 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562eede5bb40 0x562eede70fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.089+0000 7f1165e00640 1 -- 172.21.15.43:0/4184361197 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562eede6d090 msgr2=0x562eede71510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:59.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.089+0000 7f1165e00640 1 --2- 172.21.15.43:0/4184361197 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562eede6d090 0x562eede71510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.089+0000 7f1165e00640 1 -- 172.21.15.43:0/4184361197 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562eede5bb40 msgr2=0x562eede70fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:59.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.089+0000 7f1165e00640 1 --2- 172.21.15.43:0/4184361197 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562eede5bb40 0x562eede70fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.089+0000 7f1165e00640 1 -- 172.21.15.43:0/4184361197 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562eede5a490 con 0x562eede5b190 2024-03-28T22:40:59.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.089+0000 7f1165e00640 1 --2- 172.21.15.43:0/4184361197 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562eede5b190 0x562eede70a90 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f115802c1d0 tx=0x7f1158000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4c71b9304adee4c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:59.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.089+0000 7f1146a00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:40:59.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.089+0000 7f1146a00640 1 -- 172.21.15.43:0/4184361197 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f115804c040 con 0x562eede5b190 2024-03-28T22:40:59.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.090+0000 7f116ab97740 1 -- 172.21.15.43:0/4184361197 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f114c022170 con 0x562eede5b190 2024-03-28T22:40:59.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.090+0000 7f1146a00640 1 -- 172.21.15.43:0/4184361197 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f115803f480 con 0x562eede5b190 2024-03-28T22:40:59.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.090+0000 7f1146a00640 1 -- 172.21.15.43:0/4184361197 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1158039040 con 0x562eede5b190 2024-03-28T22:40:59.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.090+0000 7f116ab97740 10 client.27080 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:59.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.090+0000 7f116ab97740 1 -- 172.21.15.43:0/4184361197 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562eedee34b0 con 0x562eede5b190 2024-03-28T22:40:59.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.090+0000 7f1146a00640 1 -- 172.21.15.43:0/4184361197 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f114c022170 con 0x562eede5b190 2024-03-28T22:40:59.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.090+0000 7f1146a00640 1 -- 172.21.15.43:0/4184361197 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f115805e030 con 0x562eede5b190 2024-03-28T22:40:59.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.090+0000 7f116ab97740 10 client.27080 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:59.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.090+0000 7f116ab97740 10 client.27080 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:40:59.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.090+0000 7f116ab97740 20 client.27080 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:59.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.090+0000 7f116ab97740 1 --2- 172.21.15.43:0/4184361197 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x562eedee3990 0x562eedee5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.090+0000 7f116ab97740 4 client.27080 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:40:59.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.090+0000 7f116ab97740 1 -- 172.21.15.43:0/4184361197 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562eeddcc590 con 0x562eedee3990 2024-03-28T22:40:59.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.091+0000 7f1164a00640 1 --2- 172.21.15.43:0/4184361197 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x562eedee3990 0x562eedee5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.091+0000 7f1164a00640 1 --2- 172.21.15.43:0/4184361197 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x562eedee3990 0x562eedee5d80 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=70c5b6f545c83f63 server_cookie=e2afff6996968ba8 in_seq=0 out_seq=0 2024-03-28T22:40:59.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.091+0000 7f1146a00640 10 client.27080 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:40:59.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.091+0000 7f1146a00640 1 -- 172.21.15.43:0/4184361197 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562eeddcc590 con 0x562eedee3990 2024-03-28T22:40:59.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.091+0000 7f1146a00640 10 client.27080 handle_command_reply: tid=0 2024-03-28T22:40:59.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.091+0000 7f116ab97740 1 client.27080 shutdown 2024-03-28T22:40:59.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.092+0000 7f116ab97740 1 -- 172.21.15.43:0/4184361197 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562eede5b190 msgr2=0x562eede70a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.092+0000 7f116ab97740 1 --2- 172.21.15.43:0/4184361197 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562eede5b190 0x562eede70a90 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f115802c1d0 tx=0x7f1158000f40 comp rx=0 tx=0).stop 2024-03-28T22:40:59.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.092+0000 7f116ab97740 1 -- 172.21.15.43:0/4184361197 shutdown_connections 2024-03-28T22:40:59.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.092+0000 7f116ab97740 1 --2- 172.21.15.43:0/4184361197 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x562eedee3990 0x562eedee5d80 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.092+0000 7f116ab97740 1 --2- 172.21.15.43:0/4184361197 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562eede6d090 0x562eede71510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.092+0000 7f116ab97740 1 --2- 172.21.15.43:0/4184361197 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562eede5bb40 0x562eede70fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.092+0000 7f1146a00640 0 client.27080 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:40:59.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.092+0000 7f116ab97740 1 --2- 172.21.15.43:0/4184361197 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562eede5b190 0x562eede70a90 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.092+0000 7f116ab97740 1 -- 172.21.15.43:0/4184361197 >> 172.21.15.43:0/4184361197 conn(0x562eede55ab0 msgr2=0x562eede6a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:59.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.092+0000 7f116ab97740 1 -- 172.21.15.43:0/4184361197 shutdown_connections 2024-03-28T22:40:59.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.092+0000 7f116ab97740 1 -- 172.21.15.43:0/4184361197 wait complete. 2024-03-28T22:40:59.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.092+0000 7f116ab97740 20 client.27080 trim_cache size 0 max 16384 2024-03-28T22:40:59.099 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:40:59.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.093+0000 7f1167e00640 1 -- 172.21.15.43:0/3992846104 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f11480a1f70 msgr2=0x7f11480a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.093+0000 7f1167e00640 1 --2- 172.21.15.43:0/3992846104 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f11480a1f70 0x7f11480a4430 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x562eede56d40 tx=0x7f1154004a40 comp rx=0 tx=0).stop 2024-03-28T22:40:59.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.093+0000 7f1167e00640 1 -- 172.21.15.43:0/3992846104 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1160073b20 msgr2=0x7f116019a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.093+0000 7f1167e00640 1 --2- 172.21.15.43:0/3992846104 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1160073b20 0x7f116019a240 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f115400e9f0 tx=0x7f115400edf0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.093+0000 7f1167e00640 1 -- 172.21.15.43:0/3992846104 shutdown_connections 2024-03-28T22:40:59.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.093+0000 7f1167e00640 1 --2- 172.21.15.43:0/3992846104 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f11480a1f70 0x7f11480a4430 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.093+0000 7f1167e00640 1 --2- 172.21.15.43:0/3992846104 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f116010df60 0x7f116019eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.093+0000 7f1167e00640 1 --2- 172.21.15.43:0/3992846104 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1160074440 0x7f116019a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.093+0000 7f1167e00640 1 --2- 172.21.15.43:0/3992846104 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1160073b20 0x7f116019a240 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.093+0000 7f1167e00640 1 -- 172.21.15.43:0/3992846104 >> 172.21.15.43:0/3992846104 conn(0x7f11600fd1b0 msgr2=0x7f11600fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:59.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.093+0000 7f1167e00640 1 -- 172.21.15.43:0/3992846104 shutdown_connections 2024-03-28T22:40:59.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.093+0000 7f1167e00640 1 -- 172.21.15.43:0/3992846104 wait complete. 2024-03-28T22:40:59.110 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T22:40:59.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.345+0000 7fd794c00640 1 -- 172.21.15.43:0/4228903948 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7901076f0 msgr2=0x7fd790107ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.345+0000 7fd794c00640 1 --2- 172.21.15.43:0/4228903948 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7901076f0 0x7fd790107ad0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd780009930 tx=0x7fd78002bc30 comp rx=0 tx=0).stop 2024-03-28T22:40:59.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.345+0000 7fd794c00640 1 -- 172.21.15.43:0/4228903948 shutdown_connections 2024-03-28T22:40:59.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.345+0000 7fd794c00640 1 --2- 172.21.15.43:0/4228903948 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7901016e0 0x7fd790101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.345+0000 7fd794c00640 1 --2- 172.21.15.43:0/4228903948 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7901020a0 0x7fd79010e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.345+0000 7fd794c00640 1 --2- 172.21.15.43:0/4228903948 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7901076f0 0x7fd790107ad0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.345+0000 7fd794c00640 1 -- 172.21.15.43:0/4228903948 >> 172.21.15.43:0/4228903948 conn(0x7fd790078b50 msgr2=0x7fd790078f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:59.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.345+0000 7fd794c00640 1 -- 172.21.15.43:0/4228903948 shutdown_connections 2024-03-28T22:40:59.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.345+0000 7fd794c00640 1 -- 172.21.15.43:0/4228903948 wait complete. 2024-03-28T22:40:59.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.346+0000 7fd794c00640 1 Processor -- start 2024-03-28T22:40:59.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.346+0000 7fd794c00640 1 -- start start 2024-03-28T22:40:59.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.346+0000 7fd794c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7901016e0 0x7fd790073a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.346+0000 7fd794c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7901020a0 0x7fd790073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.346+0000 7fd794c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7901076f0 0x7fd7900744d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.346+0000 7fd794c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd790111b50 con 0x7fd7901016e0 2024-03-28T22:40:59.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.347+0000 7fd794c00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd7901119d0 con 0x7fd7901076f0 2024-03-28T22:40:59.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.347+0000 7fd794c00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd790111cd0 con 0x7fd7901020a0 2024-03-28T22:40:59.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.347+0000 7fd78de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7901016e0 0x7fd790073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.347+0000 7fd78d400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7901020a0 0x7fd790073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.347+0000 7fd78d400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7901020a0 0x7fd790073f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58854/0 (socket says 172.21.15.43:58854) 2024-03-28T22:40:59.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.347+0000 7fd78e800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7901076f0 0x7fd7900744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.347+0000 7fd78d400640 1 -- 172.21.15.43:0/2714781980 learned_addr learned my addr 172.21.15.43:0/2714781980 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:59.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.347+0000 7fd78d400640 1 -- 172.21.15.43:0/2714781980 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7901076f0 msgr2=0x7fd7900744d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.347+0000 7fd78d400640 1 --2- 172.21.15.43:0/2714781980 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7901076f0 0x7fd7900744d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.347+0000 7fd78d400640 1 -- 172.21.15.43:0/2714781980 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7901016e0 msgr2=0x7fd790073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.347+0000 7fd78d400640 1 --2- 172.21.15.43:0/2714781980 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7901016e0 0x7fd790073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.347+0000 7fd78d400640 1 -- 172.21.15.43:0/2714781980 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7901a6530 con 0x7fd7901020a0 2024-03-28T22:40:59.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.347+0000 7fd78e800640 1 --2- 172.21.15.43:0/2714781980 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7901076f0 0x7fd7900744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:59.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.348+0000 7fd78d400640 1 --2- 172.21.15.43:0/2714781980 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7901020a0 0x7fd790073f90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd77c00e9f0 tx=0x7fd77c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:59.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.348+0000 7fd786a00640 1 -- 172.21.15.43:0/2714781980 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd77c00cd80 con 0x7fd7901020a0 2024-03-28T22:40:59.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.348+0000 7fd794c00640 1 -- 172.21.15.43:0/2714781980 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7901a6820 con 0x7fd7901020a0 2024-03-28T22:40:59.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.348+0000 7fd786a00640 1 -- 172.21.15.43:0/2714781980 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd77c005500 con 0x7fd7901020a0 2024-03-28T22:40:59.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.348+0000 7fd786a00640 1 -- 172.21.15.43:0/2714781980 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd77c010690 con 0x7fd7901020a0 2024-03-28T22:40:59.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.348+0000 7fd794c00640 1 -- 172.21.15.43:0/2714781980 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7901a6d70 con 0x7fd7901020a0 2024-03-28T22:40:59.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.352+0000 7fd7977ea740 1 Processor -- start 2024-03-28T22:40:59.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.352+0000 7fd786a00640 1 -- 172.21.15.43:0/2714781980 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fd77c0046c0 con 0x7fd7901020a0 2024-03-28T22:40:59.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.352+0000 7fd7977ea740 1 -- start start 2024-03-28T22:40:59.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.352+0000 7fd7977ea740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587845dd380 0x5587845dd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.352+0000 7fd7977ea740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587845ddca0 0x5587845e6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.352+0000 7fd7977ea740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587845e6880 0x5587845e8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.352+0000 7fd7977ea740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x558784541050 con 0x5587845e6880 2024-03-28T22:40:59.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.352+0000 7fd7977ea740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5587844e9ce0 con 0x5587845dd380 2024-03-28T22:40:59.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.352+0000 7fd7977ea740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55878453d0f0 con 0x5587845ddca0 2024-03-28T22:40:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.352+0000 7fd78e800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587845dd380 0x5587845dd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.352+0000 7fd78d400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587845e6880 0x5587845e8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.352+0000 7fd78e800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587845dd380 0x5587845dd760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:60110/0 (socket says 172.21.15.43:60110) 2024-03-28T22:40:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.352+0000 7fd78e800640 1 -- 172.21.15.43:0/2772088727 learned_addr learned my addr 172.21.15.43:0/2772088727 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.352+0000 7fd78de00640 1 --2- 172.21.15.43:0/2772088727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587845ddca0 0x5587845e6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.353+0000 7fd78e800640 1 -- 172.21.15.43:0/2772088727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587845ddca0 msgr2=0x5587845e6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.353+0000 7fd78e800640 1 --2- 172.21.15.43:0/2772088727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587845ddca0 0x5587845e6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.353+0000 7fd78e800640 1 -- 172.21.15.43:0/2772088727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587845e6880 msgr2=0x5587845e8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.353+0000 7fd78e800640 1 --2- 172.21.15.43:0/2772088727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587845e6880 0x5587845e8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.353+0000 7fd78e800640 1 -- 172.21.15.43:0/2772088727 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587842553f0 con 0x5587845dd380 2024-03-28T22:40:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.353+0000 7fd78e800640 1 --2- 172.21.15.43:0/2772088727 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587845dd380 0x5587845dd760 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fd78800f600 tx=0x7fd78800fac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66fef2eabc01e92d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.353+0000 7fd77b000640 1 -- 172.21.15.43:0/2772088727 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd788003b70 con 0x5587845dd380 2024-03-28T22:40:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.354+0000 7fd77b000640 1 -- 172.21.15.43:0/2772088727 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd78800dd60 con 0x5587845dd380 2024-03-28T22:40:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.354+0000 7fd77b000640 1 -- 172.21.15.43:0/2772088727 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd788004810 con 0x5587845dd380 2024-03-28T22:40:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.354+0000 7fd786a00640 1 --2- 172.21.15.43:0/2714781980 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd7740a1ef0 0x7fd7740a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.354+0000 7fd786a00640 1 -- 172.21.15.43:0/2714781980 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fd77c0ce9b0 con 0x7fd7901020a0 2024-03-28T22:40:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.354+0000 7fd78de00640 1 --2- 172.21.15.43:0/2714781980 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd7740a1ef0 0x7fd7740a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.354+0000 7fd7977ea740 1 -- 172.21.15.43:0/2772088727 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587845dd380 msgr2=0x5587845dd760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.354+0000 7fd7977ea740 1 --2- 172.21.15.43:0/2772088727 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587845dd380 0x5587845dd760 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fd78800f600 tx=0x7fd78800fac0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.355+0000 7fd7977ea740 1 -- 172.21.15.43:0/2772088727 shutdown_connections 2024-03-28T22:40:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.355+0000 7fd7977ea740 1 --2- 172.21.15.43:0/2772088727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587845e6880 0x5587845e8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.355+0000 7fd7977ea740 1 --2- 172.21.15.43:0/2772088727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587845ddca0 0x5587845e6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.355+0000 7fd7977ea740 1 --2- 172.21.15.43:0/2772088727 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587845dd380 0x5587845dd760 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.355+0000 7fd7977ea740 1 -- 172.21.15.43:0/2772088727 >> 172.21.15.43:0/2772088727 conn(0x5587845d7ab0 msgr2=0x5587845d7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.355+0000 7fd7977ea740 1 -- 172.21.15.43:0/2772088727 shutdown_connections 2024-03-28T22:40:59.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.355+0000 7fd7977ea740 1 -- 172.21.15.43:0/2772088727 wait complete. 2024-03-28T22:40:59.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.355+0000 7fd78de00640 1 --2- 172.21.15.43:0/2714781980 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd7740a1ef0 0x7fd7740a43b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd780009900 tx=0x7fd78002df90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:59.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.355+0000 7fd7977ea740 1 -- start start 2024-03-28T22:40:59.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.356+0000 7fd7977ea740 1 Processor -- start 2024-03-28T22:40:59.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.356+0000 7fd7977ea740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587845dd380 0x5587845e37e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.356+0000 7fd7977ea740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587845ddca0 0x5587845e3d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.356+0000 7fd7977ea740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587845e6880 0x5587845e4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.356+0000 7fd7977ea740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5587845e9670 con 0x5587845dd380 2024-03-28T22:40:59.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.356+0000 7fd7977ea740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5587844df2f0 con 0x5587845ddca0 2024-03-28T22:40:59.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.356+0000 7fd7977ea740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5587845e97f0 con 0x5587845e6880 2024-03-28T22:40:59.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.356+0000 7fd7977ea740 10 client.0 _ll_register_callbacks cb 0x558784254be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:59.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.356+0000 7fd78d400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587845ddca0 0x5587845e3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.356+0000 7fd78de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587845dd380 0x5587845e37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.356+0000 7fd78e800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587845e6880 0x5587845e4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.356+0000 7fd78d400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587845ddca0 0x5587845e3d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:60112/0 (socket says 172.21.15.43:60112) 2024-03-28T22:40:59.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.356+0000 7fd78d400640 1 -- 172.21.15.43:0/1445711614 learned_addr learned my addr 172.21.15.43:0/1445711614 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:59.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.357+0000 7fd78d400640 1 -- 172.21.15.43:0/1445711614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587845e6880 msgr2=0x5587845e4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:59.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.357+0000 7fd78d400640 1 --2- 172.21.15.43:0/1445711614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587845e6880 0x5587845e4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.357+0000 7fd78d400640 1 -- 172.21.15.43:0/1445711614 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587845dd380 msgr2=0x5587845e37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:59.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.357+0000 7fd78d400640 1 --2- 172.21.15.43:0/1445711614 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587845dd380 0x5587845e37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.357+0000 7fd78d400640 1 -- 172.21.15.43:0/1445711614 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55878431f7d0 con 0x5587845ddca0 2024-03-28T22:40:59.359 INFO:tasks.workunit.client.0.smithi043.stdout: 1 231528 8.25 MB/sec execute 30 sec latency 16.450 ms 2024-03-28T22:40:59.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.357+0000 7fd78d400640 1 --2- 172.21.15.43:0/1445711614 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587845ddca0 0x5587845e3d20 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fd77c00c6a0 tx=0x7fd77c004ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71cf9689200b5b96 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:59.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.357+0000 7fd76f400640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:40:59.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.357+0000 7fd76f400640 1 -- 172.21.15.43:0/1445711614 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd77c09a900 con 0x5587845ddca0 2024-03-28T22:40:59.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.357+0000 7fd76f400640 1 -- 172.21.15.43:0/1445711614 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fd77c0a34b0 con 0x5587845ddca0 2024-03-28T22:40:59.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.357+0000 7fd7977ea740 1 -- 172.21.15.43:0/1445711614 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd78800dd60 con 0x5587845ddca0 2024-03-28T22:40:59.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.357+0000 7fd76f400640 1 -- 172.21.15.43:0/1445711614 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd77c0cc190 con 0x5587845ddca0 2024-03-28T22:40:59.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.358+0000 7fd7977ea740 10 client.27092 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:59.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.358+0000 7fd7977ea740 1 -- 172.21.15.43:0/1445711614 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5587845f58d0 con 0x5587845ddca0 2024-03-28T22:40:59.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.358+0000 7fd76f400640 1 -- 172.21.15.43:0/1445711614 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd78800dd60 con 0x5587845ddca0 2024-03-28T22:40:59.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.358+0000 7fd76f400640 1 -- 172.21.15.43:0/1445711614 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd77c096cf0 con 0x5587845ddca0 2024-03-28T22:40:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.358+0000 7fd7977ea740 10 client.27092 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.358+0000 7fd7977ea740 10 client.27092 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:40:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.358+0000 7fd7977ea740 20 client.27092 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.358+0000 7fd7977ea740 1 --2- 172.21.15.43:0/1445711614 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5587845f6d90 0x5587845f9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.358+0000 7fd7977ea740 4 client.27092 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:40:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.358+0000 7fd7977ea740 1 -- 172.21.15.43:0/1445711614 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5587844ea680 con 0x5587845f6d90 2024-03-28T22:40:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.358+0000 7fd78de00640 1 --2- 172.21.15.43:0/1445711614 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5587845f6d90 0x5587845f9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.359+0000 7fd78de00640 1 --2- 172.21.15.43:0/1445711614 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5587845f6d90 0x5587845f9180 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=18595d41448dd410 server_cookie=343adb52373f5639 in_seq=0 out_seq=0 2024-03-28T22:40:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.359+0000 7fd76f400640 10 client.27092 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:40:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.360+0000 7fd76f400640 1 -- 172.21.15.43:0/1445711614 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5587844ea680 con 0x5587845f6d90 2024-03-28T22:40:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.360+0000 7fd76f400640 10 client.27092 handle_command_reply: tid=0 2024-03-28T22:40:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.360+0000 7fd7977ea740 1 client.27092 shutdown 2024-03-28T22:40:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.360+0000 7fd7977ea740 1 -- 172.21.15.43:0/1445711614 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587845ddca0 msgr2=0x5587845e3d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.360+0000 7fd7977ea740 1 --2- 172.21.15.43:0/1445711614 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587845ddca0 0x5587845e3d20 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fd77c00c6a0 tx=0x7fd77c004ba0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.361+0000 7fd7977ea740 1 -- 172.21.15.43:0/1445711614 shutdown_connections 2024-03-28T22:40:59.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.361+0000 7fd7977ea740 1 --2- 172.21.15.43:0/1445711614 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5587845f6d90 0x5587845f9180 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.361+0000 7fd7977ea740 1 --2- 172.21.15.43:0/1445711614 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587845ddca0 0x5587845e3d20 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.361+0000 7fd7977ea740 1 --2- 172.21.15.43:0/1445711614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587845e6880 0x5587845e4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.361+0000 7fd7977ea740 1 --2- 172.21.15.43:0/1445711614 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587845dd380 0x5587845e37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.361+0000 7fd7977ea740 1 -- 172.21.15.43:0/1445711614 >> 172.21.15.43:0/1445711614 conn(0x5587845d7ab0 msgr2=0x5587845d9800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:59.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.361+0000 7fd76f400640 0 client.27092 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:40:59.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.361+0000 7fd7977ea740 1 -- 172.21.15.43:0/1445711614 shutdown_connections 2024-03-28T22:40:59.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.361+0000 7fd7977ea740 1 -- 172.21.15.43:0/1445711614 wait complete. 2024-03-28T22:40:59.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.361+0000 7fd7977ea740 20 client.27092 trim_cache size 0 max 16384 2024-03-28T22:40:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.379+0000 7fd7977ea740 1 Processor -- start 2024-03-28T22:40:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.379+0000 7fd7977ea740 1 -- start start 2024-03-28T22:40:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.379+0000 7fd7977ea740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587845dd380 0x5587845e5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.379+0000 7fd7977ea740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587845ddca0 0x5587845f2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.379+0000 7fd7977ea740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587845e6880 0x5587845fe960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.379+0000 7fd7977ea740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd7901b0d90 con 0x5587845e6880 2024-03-28T22:40:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.379+0000 7fd7977ea740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd7901b28d0 con 0x5587845ddca0 2024-03-28T22:40:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.379+0000 7fd7977ea740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd7901b1460 con 0x5587845dd380 2024-03-28T22:40:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.380+0000 7fd78d400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587845ddca0 0x5587845f2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.380+0000 7fd78d400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587845ddca0 0x5587845f2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:60114/0 (socket says 172.21.15.43:60114) 2024-03-28T22:40:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.380+0000 7fd78d400640 1 -- 172.21.15.43:0/733404940 learned_addr learned my addr 172.21.15.43:0/733404940 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.380+0000 7fd78de00640 1 --2- 172.21.15.43:0/733404940 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587845dd380 0x5587845e5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.380+0000 7fd78e800640 1 --2- 172.21.15.43:0/733404940 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587845e6880 0x5587845fe960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.380+0000 7fd78d400640 1 -- 172.21.15.43:0/733404940 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587845dd380 msgr2=0x5587845e5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.380+0000 7fd78d400640 1 --2- 172.21.15.43:0/733404940 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587845dd380 0x5587845e5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.380+0000 7fd78d400640 1 -- 172.21.15.43:0/733404940 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587845e6880 msgr2=0x5587845fe960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.380+0000 7fd78d400640 1 --2- 172.21.15.43:0/733404940 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587845e6880 0x5587845fe960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.380+0000 7fd78d400640 1 -- 172.21.15.43:0/733404940 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587845d1f80 con 0x5587845ddca0 2024-03-28T22:40:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.380+0000 7fd78d400640 1 --2- 172.21.15.43:0/733404940 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587845ddca0 0x5587845f2440 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fd77c09aca0 tx=0x7fd77c0971a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1290e92c8535810f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.381+0000 7fd77a600640 1 -- 172.21.15.43:0/733404940 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd77c09ad70 con 0x5587845ddca0 2024-03-28T22:40:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.381+0000 7fd77a600640 1 -- 172.21.15.43:0/733404940 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd77c09b6a0 con 0x5587845ddca0 2024-03-28T22:40:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.381+0000 7fd77a600640 1 -- 172.21.15.43:0/733404940 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd77c0a32b0 con 0x5587845ddca0 2024-03-28T22:40:59.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.381+0000 7fd7977ea740 1 -- 172.21.15.43:0/733404940 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587845ddca0 msgr2=0x5587845f2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.381+0000 7fd7977ea740 1 --2- 172.21.15.43:0/733404940 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587845ddca0 0x5587845f2440 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fd77c09aca0 tx=0x7fd77c0971a0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.382+0000 7fd7977ea740 1 -- 172.21.15.43:0/733404940 shutdown_connections 2024-03-28T22:40:59.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.382+0000 7fd7977ea740 1 --2- 172.21.15.43:0/733404940 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587845ddca0 0x5587845f2440 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.382+0000 7fd7977ea740 1 --2- 172.21.15.43:0/733404940 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587845e6880 0x5587845fe960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.382+0000 7fd7977ea740 1 --2- 172.21.15.43:0/733404940 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587845dd380 0x5587845e5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.382+0000 7fd7977ea740 1 -- 172.21.15.43:0/733404940 >> 172.21.15.43:0/733404940 conn(0x5587845d7ab0 msgr2=0x5587845d62e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:59.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.382+0000 7fd7977ea740 1 -- 172.21.15.43:0/733404940 shutdown_connections 2024-03-28T22:40:59.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.382+0000 7fd7977ea740 1 -- 172.21.15.43:0/733404940 wait complete. 2024-03-28T22:40:59.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.382+0000 7fd7977ea740 1 -- start start 2024-03-28T22:40:59.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.383+0000 7fd7977ea740 1 Processor -- start 2024-03-28T22:40:59.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.383+0000 7fd7977ea740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587845dd380 0x5587845ea6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.383+0000 7fd7977ea740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587845ddca0 0x5587845eac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.383+0000 7fd7977ea740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587845e6880 0x55878465bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.383+0000 7fd7977ea740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd7901b2200 con 0x5587845e6880 2024-03-28T22:40:59.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.383+0000 7fd7977ea740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd7901b1b30 con 0x5587845ddca0 2024-03-28T22:40:59.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.383+0000 7fd7977ea740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd7901aed70 con 0x5587845dd380 2024-03-28T22:40:59.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.383+0000 7fd7977ea740 10 client.0 _ll_register_callbacks cb 0x7fd77c0d4220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:59.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.383+0000 7fd78de00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587845dd380 0x5587845ea6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.383+0000 7fd78d400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587845ddca0 0x5587845eac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.383+0000 7fd78de00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587845dd380 0x5587845ea6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58884/0 (socket says 172.21.15.43:58884) 2024-03-28T22:40:59.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.383+0000 7fd78de00640 1 -- 172.21.15.43:0/1024174410 learned_addr learned my addr 172.21.15.43:0/1024174410 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:59.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.384+0000 7fd78e800640 1 --2- 172.21.15.43:0/1024174410 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587845e6880 0x55878465bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.384+0000 7fd78d400640 1 -- 172.21.15.43:0/1024174410 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587845dd380 msgr2=0x5587845ea6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:59.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.384+0000 7fd78d400640 1 --2- 172.21.15.43:0/1024174410 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587845dd380 0x5587845ea6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.384+0000 7fd78d400640 1 -- 172.21.15.43:0/1024174410 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587845e6880 msgr2=0x55878465bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:59.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.384+0000 7fd78d400640 1 --2- 172.21.15.43:0/1024174410 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587845e6880 0x55878465bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.384+0000 7fd78d400640 1 -- 172.21.15.43:0/1024174410 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587845eb4d0 con 0x5587845ddca0 2024-03-28T22:40:59.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.384+0000 7fd78d400640 1 --2- 172.21.15.43:0/1024174410 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587845ddca0 0x5587845eac00 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x5587845e3370 tx=0x7fd77c096f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ee52eac1523dfd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:59.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.384+0000 7fd76f400640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:40:59.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.384+0000 7fd76f400640 1 -- 172.21.15.43:0/1024174410 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd77c0ab060 con 0x5587845ddca0 2024-03-28T22:40:59.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.384+0000 7fd7977ea740 1 -- 172.21.15.43:0/1024174410 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd77c09b6a0 con 0x5587845ddca0 2024-03-28T22:40:59.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.384+0000 7fd76f400640 1 -- 172.21.15.43:0/1024174410 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fd77c09a3f0 con 0x5587845ddca0 2024-03-28T22:40:59.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.385+0000 7fd76f400640 1 -- 172.21.15.43:0/1024174410 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd77c09ad70 con 0x5587845ddca0 2024-03-28T22:40:59.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.385+0000 7fd7977ea740 10 client.27098 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:59.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.385+0000 7fd7977ea740 1 -- 172.21.15.43:0/1024174410 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5587845eb4d0 con 0x5587845ddca0 2024-03-28T22:40:59.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.385+0000 7fd76f400640 1 -- 172.21.15.43:0/1024174410 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd77c09b6a0 con 0x5587845ddca0 2024-03-28T22:40:59.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.385+0000 7fd76f400640 1 -- 172.21.15.43:0/1024174410 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd77c0ae080 con 0x5587845ddca0 2024-03-28T22:40:59.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.385+0000 7fd7977ea740 10 client.27098 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:59.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.385+0000 7fd7977ea740 10 client.27098 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:40:59.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.385+0000 7fd7977ea740 20 client.27098 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:59.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.385+0000 7fd7977ea740 1 --2- 172.21.15.43:0/1024174410 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55878465d8d0 0x55878465fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.385+0000 7fd7977ea740 4 client.27098 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:40:59.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.385+0000 7fd7977ea740 1 -- 172.21.15.43:0/1024174410 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55878454e590 con 0x55878465d8d0 2024-03-28T22:40:59.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.385+0000 7fd78de00640 1 --2- 172.21.15.43:0/1024174410 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55878465d8d0 0x55878465fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.386+0000 7fd78de00640 1 --2- 172.21.15.43:0/1024174410 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55878465d8d0 0x55878465fcc0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=22a359c69a292db5 server_cookie=51679af10c3d76c3 in_seq=0 out_seq=0 2024-03-28T22:40:59.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.386+0000 7fd76f400640 10 client.27098 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:40:59.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.386+0000 7fd76f400640 1 -- 172.21.15.43:0/1024174410 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55878454e590 con 0x55878465d8d0 2024-03-28T22:40:59.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.386+0000 7fd76f400640 10 client.27098 handle_command_reply: tid=0 2024-03-28T22:40:59.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.386+0000 7fd7977ea740 1 client.27098 shutdown 2024-03-28T22:40:59.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.387+0000 7fd7977ea740 1 -- 172.21.15.43:0/1024174410 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587845ddca0 msgr2=0x5587845eac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.387+0000 7fd7977ea740 1 --2- 172.21.15.43:0/1024174410 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587845ddca0 0x5587845eac00 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x5587845e3370 tx=0x7fd77c096f00 comp rx=0 tx=0).stop 2024-03-28T22:40:59.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.387+0000 7fd7977ea740 1 -- 172.21.15.43:0/1024174410 shutdown_connections 2024-03-28T22:40:59.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.387+0000 7fd7977ea740 1 --2- 172.21.15.43:0/1024174410 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55878465d8d0 0x55878465fcc0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.387+0000 7fd7977ea740 1 --2- 172.21.15.43:0/1024174410 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587845ddca0 0x5587845eac00 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.387+0000 7fd76f400640 0 client.27098 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:40:59.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.387+0000 7fd7977ea740 1 --2- 172.21.15.43:0/1024174410 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587845e6880 0x55878465bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.387+0000 7fd7977ea740 1 --2- 172.21.15.43:0/1024174410 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587845dd380 0x5587845ea6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.387+0000 7fd7977ea740 1 -- 172.21.15.43:0/1024174410 >> 172.21.15.43:0/1024174410 conn(0x5587845d7ab0 msgr2=0x5587845e4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:59.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.387+0000 7fd7977ea740 1 -- 172.21.15.43:0/1024174410 shutdown_connections 2024-03-28T22:40:59.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.387+0000 7fd7977ea740 1 -- 172.21.15.43:0/1024174410 wait complete. 2024-03-28T22:40:59.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.387+0000 7fd7977ea740 20 client.27098 trim_cache size 0 max 16384 2024-03-28T22:40:59.390 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:40:59.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.388+0000 7fd794c00640 1 -- 172.21.15.43:0/2714781980 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd7740a1ef0 msgr2=0x7fd7740a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.388+0000 7fd794c00640 1 --2- 172.21.15.43:0/2714781980 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd7740a1ef0 0x7fd7740a43b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd780009900 tx=0x7fd78002df90 comp rx=0 tx=0).stop 2024-03-28T22:40:59.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.388+0000 7fd794c00640 1 -- 172.21.15.43:0/2714781980 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7901020a0 msgr2=0x7fd790073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.388+0000 7fd794c00640 1 --2- 172.21.15.43:0/2714781980 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7901020a0 0x7fd790073f90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd77c00e9f0 tx=0x7fd77c00edf0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.388+0000 7fd794c00640 1 -- 172.21.15.43:0/2714781980 shutdown_connections 2024-03-28T22:40:59.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.388+0000 7fd794c00640 1 --2- 172.21.15.43:0/2714781980 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd7740a1ef0 0x7fd7740a43b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.388+0000 7fd794c00640 1 --2- 172.21.15.43:0/2714781980 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7901076f0 0x7fd7900744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.388+0000 7fd794c00640 1 --2- 172.21.15.43:0/2714781980 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7901020a0 0x7fd790073f90 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.388+0000 7fd794c00640 1 --2- 172.21.15.43:0/2714781980 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7901016e0 0x7fd790073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.388+0000 7fd794c00640 1 -- 172.21.15.43:0/2714781980 >> 172.21.15.43:0/2714781980 conn(0x7fd790078b50 msgr2=0x7fd790103ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:59.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.388+0000 7fd794c00640 1 -- 172.21.15.43:0/2714781980 shutdown_connections 2024-03-28T22:40:59.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.388+0000 7fd794c00640 1 -- 172.21.15.43:0/2714781980 wait complete. 2024-03-28T22:40:59.403 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:59 smithi043 ceph-mon[31823]: pgmap v118: 129 pgs: 129 active+clean; 45 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 359 op/s 2024-03-28T22:40:59.403 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:40:59 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2585352206' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:40:59.407 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:40:59.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:59 smithi082 ceph-mon[36471]: pgmap v118: 129 pgs: 129 active+clean; 45 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 359 op/s 2024-03-28T22:40:59.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:40:59 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2585352206' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:40:59.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.642+0000 7f7ad9c00640 1 -- 172.21.15.43:0/661537705 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ad4109900 msgr2=0x7f7ad4109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.642+0000 7f7ad9c00640 1 --2- 172.21.15.43:0/661537705 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ad4109900 0x7f7ad4109ce0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f7ac4009900 tx=0x7f7ac402bc40 comp rx=0 tx=0).stop 2024-03-28T22:40:59.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.642+0000 7f7ad9c00640 1 -- 172.21.15.43:0/661537705 shutdown_connections 2024-03-28T22:40:59.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.642+0000 7f7ad9c00640 1 --2- 172.21.15.43:0/661537705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ad41042b0 0x7f7ad41107c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.642+0000 7f7ad9c00640 1 --2- 172.21.15.43:0/661537705 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ad41038f0 0x7f7ad4103d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.642+0000 7f7ad9c00640 1 --2- 172.21.15.43:0/661537705 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ad4109900 0x7f7ad4109ce0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.642+0000 7f7ad9c00640 1 -- 172.21.15.43:0/661537705 >> 172.21.15.43:0/661537705 conn(0x7f7ad40ff3a0 msgr2=0x7f7ad41017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:59.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.642+0000 7f7ad9c00640 1 -- 172.21.15.43:0/661537705 shutdown_connections 2024-03-28T22:40:59.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.642+0000 7f7ad9c00640 1 -- 172.21.15.43:0/661537705 wait complete. 2024-03-28T22:40:59.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.643+0000 7f7ad9c00640 1 Processor -- start 2024-03-28T22:40:59.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.643+0000 7f7ad9c00640 1 -- start start 2024-03-28T22:40:59.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.644+0000 7f7ad9c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ad41038f0 0x7f7ad419a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.644+0000 7f7ad9c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ad41042b0 0x7f7ad419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.644+0000 7f7ad9c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ad4109900 0x7f7ad419ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.644+0000 7f7ad9c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad4113d30 con 0x7f7ad41038f0 2024-03-28T22:40:59.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.644+0000 7f7ad9c00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad4113bb0 con 0x7f7ad4109900 2024-03-28T22:40:59.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.644+0000 7f7ad9c00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad4113eb0 con 0x7f7ad41042b0 2024-03-28T22:40:59.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.644+0000 7f7ad2000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ad41042b0 0x7f7ad419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.644+0000 7f7ad2a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ad41038f0 0x7f7ad419a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.644+0000 7f7ad2000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ad41042b0 0x7f7ad419a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58900/0 (socket says 172.21.15.43:58900) 2024-03-28T22:40:59.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.644+0000 7f7ad2000640 1 -- 172.21.15.43:0/1997675074 learned_addr learned my addr 172.21.15.43:0/1997675074 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:59.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.644+0000 7f7ad3400640 1 --2- 172.21.15.43:0/1997675074 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ad4109900 0x7f7ad419ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.645+0000 7f7ad2000640 1 -- 172.21.15.43:0/1997675074 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ad4109900 msgr2=0x7f7ad419ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.645+0000 7f7ad2000640 1 --2- 172.21.15.43:0/1997675074 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ad4109900 0x7f7ad419ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.645+0000 7f7ad2000640 1 -- 172.21.15.43:0/1997675074 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ad41038f0 msgr2=0x7f7ad419a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.645+0000 7f7ad2000640 1 --2- 172.21.15.43:0/1997675074 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ad41038f0 0x7f7ad419a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.645+0000 7f7ad2000640 1 -- 172.21.15.43:0/1997675074 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ad419f200 con 0x7f7ad41042b0 2024-03-28T22:40:59.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.645+0000 7f7ad3400640 1 --2- 172.21.15.43:0/1997675074 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ad4109900 0x7f7ad419ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:40:59.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.645+0000 7f7ad2000640 1 --2- 172.21.15.43:0/1997675074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ad41042b0 0x7f7ad419a750 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7ac000d950 tx=0x7f7ac000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:59.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.645+0000 7f7acb400640 1 -- 172.21.15.43:0/1997675074 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac0017070 con 0x7f7ad41042b0 2024-03-28T22:40:59.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.646+0000 7f7ad9c00640 1 -- 172.21.15.43:0/1997675074 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ad419f4f0 con 0x7f7ad41042b0 2024-03-28T22:40:59.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.646+0000 7f7acb400640 1 -- 172.21.15.43:0/1997675074 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ac0009c30 con 0x7f7ad41042b0 2024-03-28T22:40:59.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.646+0000 7f7acb400640 1 -- 172.21.15.43:0/1997675074 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac000f6b0 con 0x7f7ad41042b0 2024-03-28T22:40:59.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.646+0000 7f7ad9c00640 1 -- 172.21.15.43:0/1997675074 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ad41a6d80 con 0x7f7ad41042b0 2024-03-28T22:40:59.651 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:59 smithi172 ceph-mon[36566]: pgmap v118: 129 pgs: 129 active+clean; 45 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 359 op/s 2024-03-28T22:40:59.652 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:40:59 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2585352206' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:40:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.649+0000 7f7adc993740 1 Processor -- start 2024-03-28T22:40:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.649+0000 7f7adc993740 1 -- start start 2024-03-28T22:40:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.650+0000 7f7adc993740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562800b0d380 0x562800b0d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.650+0000 7f7adc993740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562800b0dca0 0x562800b16340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.650+0000 7f7adc993740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562800b16880 0x562800b18c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.650+0000 7f7adc993740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562800a71050 con 0x562800b0d380 2024-03-28T22:40:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.650+0000 7f7adc993740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x562800a19ce0 con 0x562800b16880 2024-03-28T22:40:59.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.650+0000 7f7adc993740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x562800a6d0f0 con 0x562800b0dca0 2024-03-28T22:40:59.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.650+0000 7f7ad3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562800b0d380 0x562800b0d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.650+0000 7f7ad2a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562800b0dca0 0x562800b16340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.650+0000 7f7ad3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562800b0d380 0x562800b0d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42116/0 (socket says 172.21.15.43:42116) 2024-03-28T22:40:59.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.650+0000 7f7ad3400640 1 -- 172.21.15.43:0/2996108451 learned_addr learned my addr 172.21.15.43:0/2996108451 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:59.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.650+0000 7f7acb400640 1 -- 172.21.15.43:0/1997675074 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f7ac00109d0 con 0x7f7ad41042b0 2024-03-28T22:40:59.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.651+0000 7f7ad2a00640 1 -- 172.21.15.43:0/2996108451 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562800b16880 msgr2=0x562800b18c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:59.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.651+0000 7f7ad2a00640 1 --2- 172.21.15.43:0/2996108451 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562800b16880 0x562800b18c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.651+0000 7f7ad2a00640 1 -- 172.21.15.43:0/2996108451 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562800b0d380 msgr2=0x562800b0d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:59.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.651+0000 7f7ad2a00640 1 --2- 172.21.15.43:0/2996108451 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562800b0d380 0x562800b0d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.651+0000 7f7ad2a00640 1 -- 172.21.15.43:0/2996108451 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628007853f0 con 0x562800b0dca0 2024-03-28T22:40:59.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.651+0000 7f7ad2a00640 1 --2- 172.21.15.43:0/2996108451 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562800b0dca0 0x562800b16340 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7ac402c180 tx=0x7f7ac4005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=950fc099261e5bab server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:59.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.651+0000 7f7abfe00640 1 -- 172.21.15.43:0/2996108451 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac4004240 con 0x562800b0dca0 2024-03-28T22:40:59.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.651+0000 7f7abfe00640 1 -- 172.21.15.43:0/2996108451 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ac40043e0 con 0x562800b0dca0 2024-03-28T22:40:59.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.651+0000 7f7abfe00640 1 -- 172.21.15.43:0/2996108451 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac40051c0 con 0x562800b0dca0 2024-03-28T22:40:59.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.652+0000 7f7adc993740 1 -- 172.21.15.43:0/2996108451 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562800b0dca0 msgr2=0x562800b16340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.652+0000 7f7adc993740 1 --2- 172.21.15.43:0/2996108451 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562800b0dca0 0x562800b16340 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7ac402c180 tx=0x7f7ac4005d50 comp rx=0 tx=0).stop 2024-03-28T22:40:59.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.652+0000 7f7acb400640 1 --2- 172.21.15.43:0/1997675074 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7ab80a1f70 0x7f7ab80a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.652+0000 7f7adc993740 1 -- 172.21.15.43:0/2996108451 shutdown_connections 2024-03-28T22:40:59.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.652+0000 7f7adc993740 1 --2- 172.21.15.43:0/2996108451 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562800b16880 0x562800b18c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.652+0000 7f7adc993740 1 --2- 172.21.15.43:0/2996108451 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562800b0dca0 0x562800b16340 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.652+0000 7f7adc993740 1 --2- 172.21.15.43:0/2996108451 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562800b0d380 0x562800b0d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.652+0000 7f7adc993740 1 -- 172.21.15.43:0/2996108451 >> 172.21.15.43:0/2996108451 conn(0x562800b07ab0 msgr2=0x562800b07e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:59.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.652+0000 7f7acb400640 1 -- 172.21.15.43:0/1997675074 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f7ac00ce170 con 0x7f7ad41042b0 2024-03-28T22:40:59.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.652+0000 7f7ad3400640 1 --2- 172.21.15.43:0/1997675074 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7ab80a1f70 0x7f7ab80a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.652+0000 7f7adc993740 1 -- 172.21.15.43:0/2996108451 shutdown_connections 2024-03-28T22:40:59.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.652+0000 7f7adc993740 1 -- 172.21.15.43:0/2996108451 wait complete. 2024-03-28T22:40:59.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.653+0000 7f7adc993740 1 -- start start 2024-03-28T22:40:59.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.653+0000 7f7adc993740 1 Processor -- start 2024-03-28T22:40:59.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.653+0000 7f7ad3400640 1 --2- 172.21.15.43:0/1997675074 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7ab80a1f70 0x7f7ab80a4430 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7acc005fd0 tx=0x7f7acc009500 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:59.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.654+0000 7f7adc993740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562800b0d380 0x562800b13980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.654+0000 7f7adc993740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562800b0dca0 0x562800b13ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.654+0000 7f7adc993740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562800b16880 0x562800b14400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.654+0000 7f7adc993740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562800b19610 con 0x562800b0d380 2024-03-28T22:40:59.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.654+0000 7f7adc993740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x562800a0f2f0 con 0x562800b0dca0 2024-03-28T22:40:59.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.654+0000 7f7adc993740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x562800b19790 con 0x562800b16880 2024-03-28T22:40:59.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.654+0000 7f7adc993740 10 client.0 _ll_register_callbacks cb 0x562800784be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:59.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.654+0000 7f7ad2000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562800b0dca0 0x562800b13ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.654+0000 7f7ad2a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562800b0d380 0x562800b13980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.654+0000 7f7ad3400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562800b16880 0x562800b14400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.654+0000 7f7ad2000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562800b0dca0 0x562800b13ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:60160/0 (socket says 172.21.15.43:60160) 2024-03-28T22:40:59.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.654+0000 7f7ad2000640 1 -- 172.21.15.43:0/1215656055 learned_addr learned my addr 172.21.15.43:0/1215656055 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:59.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.654+0000 7f7ad2a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562800b0d380 0x562800b13980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42124/0 (socket says 172.21.15.43:42124) 2024-03-28T22:40:59.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.654+0000 7f7ad2000640 1 -- 172.21.15.43:0/1215656055 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562800b16880 msgr2=0x562800b14400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:59.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.654+0000 7f7ad2000640 1 --2- 172.21.15.43:0/1215656055 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562800b16880 0x562800b14400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.654+0000 7f7ad2000640 1 -- 172.21.15.43:0/1215656055 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562800b0d380 msgr2=0x562800b13980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:59.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.654+0000 7f7ad2000640 1 --2- 172.21.15.43:0/1215656055 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562800b0d380 0x562800b13980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.654+0000 7f7ad2000640 1 -- 172.21.15.43:0/1215656055 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56280084f7d0 con 0x562800b0dca0 2024-03-28T22:40:59.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.655+0000 7f7ad2000640 1 --2- 172.21.15.43:0/1215656055 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562800b0dca0 0x562800b13ec0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0009610 tx=0x7f7ac009a3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a23f10dfabdff2b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:59.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.655+0000 7f7abcc00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:40:59.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.655+0000 7f7abcc00640 1 -- 172.21.15.43:0/1215656055 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac0020030 con 0x562800b0dca0 2024-03-28T22:40:59.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.655+0000 7f7abcc00640 1 -- 172.21.15.43:0/1215656055 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f7ac00cb740 con 0x562800b0dca0 2024-03-28T22:40:59.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.655+0000 7f7adc993740 1 -- 172.21.15.43:0/1215656055 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ac40043e0 con 0x562800b0dca0 2024-03-28T22:40:59.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.655+0000 7f7abcc00640 1 -- 172.21.15.43:0/1215656055 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7ac00cb940 con 0x562800b0dca0 2024-03-28T22:40:59.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.655+0000 7f7adc993740 10 client.27107 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:59.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.655+0000 7f7adc993740 1 -- 172.21.15.43:0/1215656055 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562800b259e0 con 0x562800b0dca0 2024-03-28T22:40:59.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.656+0000 7f7abcc00640 1 -- 172.21.15.43:0/1215656055 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ac40043e0 con 0x562800b0dca0 2024-03-28T22:40:59.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.656+0000 7f7abcc00640 1 -- 172.21.15.43:0/1215656055 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7ac00d00d0 con 0x562800b0dca0 2024-03-28T22:40:59.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.656+0000 7f7adc993740 10 client.27107 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:59.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.656+0000 7f7adc993740 10 client.27107 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:40:59.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.656+0000 7f7adc993740 20 client.27107 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:59.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.656+0000 7f7adc993740 1 --2- 172.21.15.43:0/1215656055 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x562800b26ea0 0x562800b29290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.656+0000 7f7adc993740 4 client.27107 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:40:59.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.656+0000 7f7adc993740 1 -- 172.21.15.43:0/1215656055 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562800a1a680 con 0x562800b26ea0 2024-03-28T22:40:59.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.656+0000 7f7ad2a00640 1 --2- 172.21.15.43:0/1215656055 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x562800b26ea0 0x562800b29290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.657+0000 7f7ad2a00640 1 --2- 172.21.15.43:0/1215656055 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x562800b26ea0 0x562800b29290 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=619a6ec9bd56cf1b server_cookie=4462517a941e89a7 in_seq=0 out_seq=0 2024-03-28T22:40:59.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.657+0000 7f7abcc00640 10 client.27107 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:40:59.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.658+0000 7f7abcc00640 1 -- 172.21.15.43:0/1215656055 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562800a1a680 con 0x562800b26ea0 2024-03-28T22:40:59.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.658+0000 7f7abcc00640 10 client.27107 handle_command_reply: tid=0 2024-03-28T22:40:59.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.658+0000 7f7adc993740 1 client.27107 shutdown 2024-03-28T22:40:59.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.658+0000 7f7adc993740 1 -- 172.21.15.43:0/1215656055 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562800b0dca0 msgr2=0x562800b13ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.658+0000 7f7adc993740 1 --2- 172.21.15.43:0/1215656055 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562800b0dca0 0x562800b13ec0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0009610 tx=0x7f7ac009a3a0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.658+0000 7f7adc993740 1 -- 172.21.15.43:0/1215656055 shutdown_connections 2024-03-28T22:40:59.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.658+0000 7f7adc993740 1 --2- 172.21.15.43:0/1215656055 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x562800b26ea0 0x562800b29290 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.658+0000 7f7adc993740 1 --2- 172.21.15.43:0/1215656055 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562800b0dca0 0x562800b13ec0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.658+0000 7f7adc993740 1 --2- 172.21.15.43:0/1215656055 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562800b16880 0x562800b14400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.658+0000 7f7adc993740 1 --2- 172.21.15.43:0/1215656055 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562800b0d380 0x562800b13980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.658+0000 7f7adc993740 1 -- 172.21.15.43:0/1215656055 >> 172.21.15.43:0/1215656055 conn(0x562800b07ab0 msgr2=0x562800b09800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:59.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.658+0000 7f7abcc00640 0 client.27107 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:40:59.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.658+0000 7f7adc993740 1 -- 172.21.15.43:0/1215656055 shutdown_connections 2024-03-28T22:40:59.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.658+0000 7f7adc993740 1 -- 172.21.15.43:0/1215656055 wait complete. 2024-03-28T22:40:59.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.658+0000 7f7adc993740 20 client.27107 trim_cache size 0 max 16384 2024-03-28T22:40:59.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.676+0000 7f7adc993740 1 Processor -- start 2024-03-28T22:40:59.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.676+0000 7f7adc993740 1 -- start start 2024-03-28T22:40:59.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.676+0000 7f7adc993740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562800b0d380 0x562800b160b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.676+0000 7f7adc993740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562800b0dca0 0x562800b223b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.676+0000 7f7adc993740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562800b16880 0x562800b2eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.676+0000 7f7adc993740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad41b0ce0 con 0x562800b0dca0 2024-03-28T22:40:59.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.676+0000 7f7adc993740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad41b2820 con 0x562800b0d380 2024-03-28T22:40:59.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.676+0000 7f7adc993740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad41b13b0 con 0x562800b16880 2024-03-28T22:40:59.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.677+0000 7f7ad2a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562800b0d380 0x562800b160b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.677+0000 7f7ad2a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562800b0d380 0x562800b160b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:60162/0 (socket says 172.21.15.43:60162) 2024-03-28T22:40:59.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.677+0000 7f7ad2a00640 1 -- 172.21.15.43:0/3924270353 learned_addr learned my addr 172.21.15.43:0/3924270353 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:59.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.677+0000 7f7ad3400640 1 --2- 172.21.15.43:0/3924270353 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562800b16880 0x562800b2eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.677+0000 7f7ad2000640 1 --2- 172.21.15.43:0/3924270353 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562800b0dca0 0x562800b223b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.677+0000 7f7ad2a00640 1 -- 172.21.15.43:0/3924270353 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562800b16880 msgr2=0x562800b2eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:59.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.677+0000 7f7ad2a00640 1 --2- 172.21.15.43:0/3924270353 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562800b16880 0x562800b2eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.677+0000 7f7ad2a00640 1 -- 172.21.15.43:0/3924270353 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562800b0dca0 msgr2=0x562800b223b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:59.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.677+0000 7f7ad2a00640 1 --2- 172.21.15.43:0/3924270353 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562800b0dca0 0x562800b223b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.677+0000 7f7ad2a00640 1 -- 172.21.15.43:0/3924270353 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562800b0dad0 con 0x562800b0d380 2024-03-28T22:40:59.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.677+0000 7f7ad2a00640 1 --2- 172.21.15.43:0/3924270353 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562800b0d380 0x562800b160b0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x562800b0ce50 tx=0x7f7ac4004b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=412749ab700f15db server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:59.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.678+0000 7f7abf400640 1 -- 172.21.15.43:0/3924270353 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac402c8c0 con 0x562800b0d380 2024-03-28T22:40:59.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.678+0000 7f7abf400640 1 -- 172.21.15.43:0/3924270353 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ac404c040 con 0x562800b0d380 2024-03-28T22:40:59.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.678+0000 7f7abf400640 1 -- 172.21.15.43:0/3924270353 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac4054030 con 0x562800b0d380 2024-03-28T22:40:59.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.678+0000 7f7adc993740 1 -- 172.21.15.43:0/3924270353 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562800b0d380 msgr2=0x562800b160b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.678+0000 7f7adc993740 1 --2- 172.21.15.43:0/3924270353 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562800b0d380 0x562800b160b0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x562800b0ce50 tx=0x7f7ac4004b30 comp rx=0 tx=0).stop 2024-03-28T22:40:59.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.678+0000 7f7adc993740 1 -- 172.21.15.43:0/3924270353 shutdown_connections 2024-03-28T22:40:59.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.678+0000 7f7adc993740 1 --2- 172.21.15.43:0/3924270353 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562800b16880 0x562800b2eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.678+0000 7f7adc993740 1 --2- 172.21.15.43:0/3924270353 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562800b0dca0 0x562800b223b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.678+0000 7f7adc993740 1 --2- 172.21.15.43:0/3924270353 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562800b0d380 0x562800b160b0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.678+0000 7f7adc993740 1 -- 172.21.15.43:0/3924270353 >> 172.21.15.43:0/3924270353 conn(0x562800b07ab0 msgr2=0x562800b09790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:59.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.678+0000 7f7adc993740 1 -- 172.21.15.43:0/3924270353 shutdown_connections 2024-03-28T22:40:59.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.679+0000 7f7adc993740 1 -- 172.21.15.43:0/3924270353 wait complete. 2024-03-28T22:40:59.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.679+0000 7f7adc993740 1 -- start start 2024-03-28T22:40:59.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.679+0000 7f7adc993740 1 Processor -- start 2024-03-28T22:40:59.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.679+0000 7f7adc993740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562800b0d380 0x562800b1e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.680+0000 7f7adc993740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562800b0dca0 0x562800b1b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.680+0000 7f7adc993740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562800b16880 0x562800b1bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.680+0000 7f7adc993740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad41b2150 con 0x562800b0d380 2024-03-28T22:40:59.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.680+0000 7f7adc993740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad41b1a80 con 0x562800b0dca0 2024-03-28T22:40:59.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.680+0000 7f7adc993740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad41aecc0 con 0x562800b16880 2024-03-28T22:40:59.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.680+0000 7f7adc993740 10 client.0 _ll_register_callbacks cb 0x7f7ac000df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:59.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.680+0000 7f7ad3400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562800b16880 0x562800b1bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.680+0000 7f7ad2a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562800b0d380 0x562800b1e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.680+0000 7f7ad2000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562800b0dca0 0x562800b1b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.680+0000 7f7ad3400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562800b16880 0x562800b1bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58920/0 (socket says 172.21.15.43:58920) 2024-03-28T22:40:59.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.680+0000 7f7ad3400640 1 -- 172.21.15.43:0/2415708538 learned_addr learned my addr 172.21.15.43:0/2415708538 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:59.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.680+0000 7f7ad2000640 1 -- 172.21.15.43:0/2415708538 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562800b16880 msgr2=0x562800b1bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:59.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.680+0000 7f7ad2000640 1 --2- 172.21.15.43:0/2415708538 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562800b16880 0x562800b1bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.680+0000 7f7ad2000640 1 -- 172.21.15.43:0/2415708538 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562800b0d380 msgr2=0x562800b1e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:59.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.680+0000 7f7ad2000640 1 --2- 172.21.15.43:0/2415708538 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562800b0d380 0x562800b1e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.680+0000 7f7ad2000640 1 -- 172.21.15.43:0/2415708538 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562800b1f050 con 0x562800b0dca0 2024-03-28T22:40:59.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.680+0000 7f7ad2a00640 1 --2- 172.21.15.43:0/2415708538 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562800b0d380 0x562800b1e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:59.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.680+0000 7f7ad3400640 1 --2- 172.21.15.43:0/2415708538 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562800b16880 0x562800b1bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:59.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.681+0000 7f7ad2000640 1 --2- 172.21.15.43:0/2415708538 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562800b0dca0 0x562800b1b9b0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f7ac009a410 tx=0x7f7ac0009e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68d0f20e3844d5ce server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:59.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.681+0000 7f7abcc00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:40:59.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.681+0000 7f7abcc00640 1 -- 172.21.15.43:0/2415708538 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac00ba390 con 0x562800b0dca0 2024-03-28T22:40:59.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.681+0000 7f7abcc00640 1 -- 172.21.15.43:0/2415708538 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f7ac00cb740 con 0x562800b0dca0 2024-03-28T22:40:59.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.681+0000 7f7adc993740 1 -- 172.21.15.43:0/2415708538 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ac404c040 con 0x562800b0dca0 2024-03-28T22:40:59.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.681+0000 7f7abcc00640 1 -- 172.21.15.43:0/2415708538 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7ac00cb940 con 0x562800b0dca0 2024-03-28T22:40:59.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.681+0000 7f7adc993740 10 client.27113 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:59.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.681+0000 7f7adc993740 1 -- 172.21.15.43:0/2415708538 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562800b8dc20 con 0x562800b0dca0 2024-03-28T22:40:59.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.681+0000 7f7abcc00640 1 -- 172.21.15.43:0/2415708538 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ac404c040 con 0x562800b0dca0 2024-03-28T22:40:59.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.682+0000 7f7abcc00640 1 -- 172.21.15.43:0/2415708538 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7ac0099bb0 con 0x562800b0dca0 2024-03-28T22:40:59.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.682+0000 7f7adc993740 10 client.27113 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:59.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.682+0000 7f7adc993740 10 client.27113 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:40:59.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.682+0000 7f7adc993740 20 client.27113 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:59.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.682+0000 7f7adc993740 1 --2- 172.21.15.43:0/2415708538 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x562800b8e260 0x562800b90680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.682+0000 7f7adc993740 4 client.27113 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:40:59.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.682+0000 7f7adc993740 1 -- 172.21.15.43:0/2415708538 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562800a7e590 con 0x562800b8e260 2024-03-28T22:40:59.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.682+0000 7f7ad2a00640 1 --2- 172.21.15.43:0/2415708538 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x562800b8e260 0x562800b90680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.683+0000 7f7ad2a00640 1 --2- 172.21.15.43:0/2415708538 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x562800b8e260 0x562800b90680 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8fc0f6754be7bd77 server_cookie=69c2168b63a59b8d in_seq=0 out_seq=0 2024-03-28T22:40:59.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.683+0000 7f7abcc00640 10 client.27113 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:40:59.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.683+0000 7f7abcc00640 1 -- 172.21.15.43:0/2415708538 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562800a7e590 con 0x562800b8e260 2024-03-28T22:40:59.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.683+0000 7f7abcc00640 10 client.27113 handle_command_reply: tid=0 2024-03-28T22:40:59.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.683+0000 7f7adc993740 1 client.27113 shutdown 2024-03-28T22:40:59.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.683+0000 7f7adc993740 1 -- 172.21.15.43:0/2415708538 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562800b0dca0 msgr2=0x562800b1b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.683+0000 7f7adc993740 1 --2- 172.21.15.43:0/2415708538 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562800b0dca0 0x562800b1b9b0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f7ac009a410 tx=0x7f7ac0009e50 comp rx=0 tx=0).stop 2024-03-28T22:40:59.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.683+0000 7f7adc993740 1 -- 172.21.15.43:0/2415708538 shutdown_connections 2024-03-28T22:40:59.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.683+0000 7f7adc993740 1 --2- 172.21.15.43:0/2415708538 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x562800b8e260 0x562800b90680 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.683+0000 7f7adc993740 1 --2- 172.21.15.43:0/2415708538 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562800b0dca0 0x562800b1b9b0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.683+0000 7f7adc993740 1 --2- 172.21.15.43:0/2415708538 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562800b16880 0x562800b1bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.683+0000 7f7abcc00640 0 client.27113 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:40:59.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.683+0000 7f7adc993740 1 --2- 172.21.15.43:0/2415708538 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562800b0d380 0x562800b1e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.683+0000 7f7adc993740 1 -- 172.21.15.43:0/2415708538 >> 172.21.15.43:0/2415708538 conn(0x562800b07ab0 msgr2=0x562800b08e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:59.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.683+0000 7f7adc993740 1 -- 172.21.15.43:0/2415708538 shutdown_connections 2024-03-28T22:40:59.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.683+0000 7f7adc993740 1 -- 172.21.15.43:0/2415708538 wait complete. 2024-03-28T22:40:59.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.683+0000 7f7adc993740 20 client.27113 trim_cache size 0 max 16384 2024-03-28T22:40:59.700 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:40:59.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.684+0000 7f7ad9c00640 1 -- 172.21.15.43:0/1997675074 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7ab80a1f70 msgr2=0x7f7ab80a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.684+0000 7f7ad9c00640 1 --2- 172.21.15.43:0/1997675074 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7ab80a1f70 0x7f7ab80a4430 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7acc005fd0 tx=0x7f7acc009500 comp rx=0 tx=0).stop 2024-03-28T22:40:59.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.684+0000 7f7ad9c00640 1 -- 172.21.15.43:0/1997675074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ad41042b0 msgr2=0x7f7ad419a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.684+0000 7f7ad9c00640 1 --2- 172.21.15.43:0/1997675074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ad41042b0 0x7f7ad419a750 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7ac000d950 tx=0x7f7ac000de10 comp rx=0 tx=0).stop 2024-03-28T22:40:59.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.684+0000 7f7ad9c00640 1 -- 172.21.15.43:0/1997675074 shutdown_connections 2024-03-28T22:40:59.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.684+0000 7f7ad9c00640 1 --2- 172.21.15.43:0/1997675074 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7ab80a1f70 0x7f7ab80a4430 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.684+0000 7f7ad9c00640 1 --2- 172.21.15.43:0/1997675074 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ad4109900 0x7f7ad419ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.684+0000 7f7ad9c00640 1 --2- 172.21.15.43:0/1997675074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ad41042b0 0x7f7ad419a750 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.684+0000 7f7ad9c00640 1 --2- 172.21.15.43:0/1997675074 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ad41038f0 0x7f7ad419a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.684+0000 7f7ad9c00640 1 -- 172.21.15.43:0/1997675074 >> 172.21.15.43:0/1997675074 conn(0x7f7ad40ff3a0 msgr2=0x7f7ad40ffb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:59.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.684+0000 7f7ad9c00640 1 -- 172.21.15.43:0/1997675074 shutdown_connections 2024-03-28T22:40:59.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.684+0000 7f7ad9c00640 1 -- 172.21.15.43:0/1997675074 wait complete. 2024-03-28T22:40:59.703 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:40:59.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.924+0000 7fe329800640 1 -- 172.21.15.43:0/616329011 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe324106660 msgr2=0x7fe324106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.924+0000 7fe329800640 1 --2- 172.21.15.43:0/616329011 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe324106660 0x7fe324106a40 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe314009930 tx=0x7fe31402bc20 comp rx=0 tx=0).stop 2024-03-28T22:40:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.924+0000 7fe329800640 1 -- 172.21.15.43:0/616329011 shutdown_connections 2024-03-28T22:40:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.924+0000 7fe329800640 1 --2- 172.21.15.43:0/616329011 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe3241005c0 0x7fe324100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.924+0000 7fe329800640 1 --2- 172.21.15.43:0/616329011 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe324100f80 0x7fe32410d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.924+0000 7fe329800640 1 --2- 172.21.15.43:0/616329011 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe324106660 0x7fe324106a40 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.924+0000 7fe329800640 1 -- 172.21.15.43:0/616329011 >> 172.21.15.43:0/616329011 conn(0x7fe324077750 msgr2=0x7fe324077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.924+0000 7fe329800640 1 -- 172.21.15.43:0/616329011 shutdown_connections 2024-03-28T22:40:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.924+0000 7fe329800640 1 -- 172.21.15.43:0/616329011 wait complete. 2024-03-28T22:40:59.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.925+0000 7fe329800640 1 Processor -- start 2024-03-28T22:40:59.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.925+0000 7fe329800640 1 -- start start 2024-03-28T22:40:59.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.926+0000 7fe329800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe3241005c0 0x7fe32419a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.926+0000 7fe329800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe324100f80 0x7fe32419a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.926+0000 7fe329800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe324106660 0x7fe32419ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.926+0000 7fe329800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe3241109e0 con 0x7fe3241005c0 2024-03-28T22:40:59.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.926+0000 7fe329800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe324110860 con 0x7fe324100f80 2024-03-28T22:40:59.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.926+0000 7fe329800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe324110b60 con 0x7fe324106660 2024-03-28T22:40:59.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.926+0000 7fe322a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe3241005c0 0x7fe32419a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.926+0000 7fe323400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe324106660 0x7fe32419ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.926+0000 7fe322a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe3241005c0 0x7fe32419a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42166/0 (socket says 172.21.15.43:42166) 2024-03-28T22:40:59.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.926+0000 7fe322a00640 1 -- 172.21.15.43:0/765205693 learned_addr learned my addr 172.21.15.43:0/765205693 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:59.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.926+0000 7fe322000640 1 --2- 172.21.15.43:0/765205693 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe324100f80 0x7fe32419a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.927+0000 7fe322a00640 1 -- 172.21.15.43:0/765205693 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe324106660 msgr2=0x7fe32419ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.927+0000 7fe322a00640 1 --2- 172.21.15.43:0/765205693 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe324106660 0x7fe32419ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.927+0000 7fe322a00640 1 -- 172.21.15.43:0/765205693 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe324100f80 msgr2=0x7fe32419a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.927+0000 7fe322a00640 1 --2- 172.21.15.43:0/765205693 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe324100f80 0x7fe32419a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.927+0000 7fe322a00640 1 -- 172.21.15.43:0/765205693 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe32419f190 con 0x7fe3241005c0 2024-03-28T22:40:59.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.927+0000 7fe322a00640 1 --2- 172.21.15.43:0/765205693 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe3241005c0 0x7fe32419a1a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe314000c00 tx=0x7fe31402cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:59.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.927+0000 7fe31b400640 1 -- 172.21.15.43:0/765205693 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe314041070 con 0x7fe3241005c0 2024-03-28T22:40:59.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.927+0000 7fe31b400640 1 -- 172.21.15.43:0/765205693 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe314004450 con 0x7fe3241005c0 2024-03-28T22:40:59.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.927+0000 7fe31b400640 1 -- 172.21.15.43:0/765205693 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe314034440 con 0x7fe3241005c0 2024-03-28T22:40:59.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.927+0000 7fe329800640 1 -- 172.21.15.43:0/765205693 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe32419f420 con 0x7fe3241005c0 2024-03-28T22:40:59.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.927+0000 7fe329800640 1 -- 172.21.15.43:0/765205693 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3241a6d10 con 0x7fe3241005c0 2024-03-28T22:40:59.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.932+0000 7fe32c5da740 1 Processor -- start 2024-03-28T22:40:59.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.932+0000 7fe32c5da740 1 -- start start 2024-03-28T22:40:59.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.932+0000 7fe31b400640 1 -- 172.21.15.43:0/765205693 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fe314044070 con 0x7fe3241005c0 2024-03-28T22:40:59.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.932+0000 7fe32c5da740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc302ab380 0x55fc302ab760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.932+0000 7fe32c5da740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc302abca0 0x55fc302b4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.932+0000 7fe32c5da740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc302b4880 0x55fc302b6c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.932+0000 7fe32c5da740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55fc3020f050 con 0x55fc302b4880 2024-03-28T22:40:59.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.933+0000 7fe32c5da740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55fc301b7ce0 con 0x55fc302ab380 2024-03-28T22:40:59.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.933+0000 7fe32c5da740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55fc3020b0f0 con 0x55fc302abca0 2024-03-28T22:40:59.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.933+0000 7fe323400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc302ab380 0x55fc302ab760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.933+0000 7fe323400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc302ab380 0x55fc302ab760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:60210/0 (socket says 172.21.15.43:60210) 2024-03-28T22:40:59.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.933+0000 7fe322a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc302abca0 0x55fc302b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.933+0000 7fe323400640 1 -- 172.21.15.43:0/1083022486 learned_addr learned my addr 172.21.15.43:0/1083022486 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:59.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.933+0000 7fe322000640 1 --2- 172.21.15.43:0/1083022486 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc302b4880 0x55fc302b6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.933+0000 7fe323400640 1 -- 172.21.15.43:0/1083022486 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc302abca0 msgr2=0x55fc302b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:59.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.933+0000 7fe323400640 1 --2- 172.21.15.43:0/1083022486 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc302abca0 0x55fc302b4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.933+0000 7fe323400640 1 -- 172.21.15.43:0/1083022486 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc302b4880 msgr2=0x55fc302b6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:59.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.933+0000 7fe323400640 1 --2- 172.21.15.43:0/1083022486 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc302b4880 0x55fc302b6c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.933+0000 7fe323400640 1 -- 172.21.15.43:0/1083022486 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc2ff233f0 con 0x55fc302ab380 2024-03-28T22:40:59.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.933+0000 7fe322000640 1 --2- 172.21.15.43:0/1083022486 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc302b4880 0x55fc302b6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:40:59.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.934+0000 7fe323400640 1 --2- 172.21.15.43:0/1083022486 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc302ab380 0x55fc302ab760 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fe31c009f70 tx=0x7fe31c010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76af17096768473b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:59.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.934+0000 7fe30fe00640 1 -- 172.21.15.43:0/1083022486 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31c016070 con 0x55fc302ab380 2024-03-28T22:40:59.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.934+0000 7fe30fe00640 1 -- 172.21.15.43:0/1083022486 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe31c004d20 con 0x55fc302ab380 2024-03-28T22:40:59.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.934+0000 7fe30fe00640 1 -- 172.21.15.43:0/1083022486 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31c012910 con 0x55fc302ab380 2024-03-28T22:40:59.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.934+0000 7fe31b400640 1 --2- 172.21.15.43:0/765205693 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe3080a1f70 0x7fe3080a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.934+0000 7fe322a00640 1 --2- 172.21.15.43:0/765205693 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe3080a1f70 0x7fe3080a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.934+0000 7fe31b400640 1 -- 172.21.15.43:0/765205693 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fe3140efe80 con 0x7fe3241005c0 2024-03-28T22:40:59.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.935+0000 7fe32c5da740 1 -- 172.21.15.43:0/1083022486 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc302ab380 msgr2=0x55fc302ab760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.935+0000 7fe32c5da740 1 --2- 172.21.15.43:0/1083022486 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc302ab380 0x55fc302ab760 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fe31c009f70 tx=0x7fe31c010800 comp rx=0 tx=0).stop 2024-03-28T22:40:59.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.935+0000 7fe322a00640 1 --2- 172.21.15.43:0/765205693 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe3080a1f70 0x7fe3080a4430 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe3140ed3f0 tx=0x7fe3140f39d0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:59.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.935+0000 7fe32c5da740 1 -- 172.21.15.43:0/1083022486 shutdown_connections 2024-03-28T22:40:59.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.935+0000 7fe32c5da740 1 --2- 172.21.15.43:0/1083022486 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc302b4880 0x55fc302b6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.935+0000 7fe32c5da740 1 --2- 172.21.15.43:0/1083022486 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc302abca0 0x55fc302b4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.935+0000 7fe32c5da740 1 --2- 172.21.15.43:0/1083022486 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc302ab380 0x55fc302ab760 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.935+0000 7fe32c5da740 1 -- 172.21.15.43:0/1083022486 >> 172.21.15.43:0/1083022486 conn(0x55fc302a5ab0 msgr2=0x55fc302a5e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:59.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.935+0000 7fe32c5da740 1 -- 172.21.15.43:0/1083022486 shutdown_connections 2024-03-28T22:40:59.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.935+0000 7fe32c5da740 1 -- 172.21.15.43:0/1083022486 wait complete. 2024-03-28T22:40:59.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.936+0000 7fe32c5da740 1 -- start start 2024-03-28T22:40:59.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.936+0000 7fe32c5da740 1 Processor -- start 2024-03-28T22:40:59.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.937+0000 7fe32c5da740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc302ab380 0x55fc302b1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.937+0000 7fe32c5da740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc302abca0 0x55fc302b1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.937+0000 7fe32c5da740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc302b4880 0x55fc302b22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.937+0000 7fe32c5da740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55fc302b7670 con 0x55fc302ab380 2024-03-28T22:40:59.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.937+0000 7fe32c5da740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55fc301ad2f0 con 0x55fc302b4880 2024-03-28T22:40:59.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.937+0000 7fe32c5da740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55fc302b77f0 con 0x55fc302abca0 2024-03-28T22:40:59.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.937+0000 7fe32c5da740 10 client.0 _ll_register_callbacks cb 0x55fc2ff22be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:59.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.937+0000 7fe323400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc302ab380 0x55fc302b1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.937+0000 7fe322000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc302abca0 0x55fc302b1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.937+0000 7fe323400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc302ab380 0x55fc302b1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42180/0 (socket says 172.21.15.43:42180) 2024-03-28T22:40:59.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.937+0000 7fe323400640 1 -- 172.21.15.43:0/993723140 learned_addr learned my addr 172.21.15.43:0/993723140 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:59.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.937+0000 7fe323400640 1 --2- 172.21.15.43:0/993723140 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc302b4880 0x55fc302b22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.937+0000 7fe322000640 1 -- 172.21.15.43:0/993723140 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc302b4880 msgr2=0x55fc302b22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:59.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.937+0000 7fe322000640 1 --2- 172.21.15.43:0/993723140 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc302b4880 0x55fc302b22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.937+0000 7fe322000640 1 -- 172.21.15.43:0/993723140 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc302ab380 msgr2=0x55fc302b1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:59.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.937+0000 7fe322000640 1 --2- 172.21.15.43:0/993723140 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc302ab380 0x55fc302b1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.937+0000 7fe322000640 1 -- 172.21.15.43:0/993723140 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc2ffed7d0 con 0x55fc302abca0 2024-03-28T22:40:59.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.938+0000 7fe322000640 1 --2- 172.21.15.43:0/993723140 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc302abca0 0x55fc302b1db0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fe3240ffe30 tx=0x7fe310012820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7cf80b850dc6891 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:59.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.938+0000 7fe30cc00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:40:59.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.938+0000 7fe30cc00640 1 -- 172.21.15.43:0/993723140 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe310004220 con 0x55fc302abca0 2024-03-28T22:40:59.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.938+0000 7fe32c5da740 1 -- 172.21.15.43:0/993723140 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe31c004d20 con 0x55fc302abca0 2024-03-28T22:40:59.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.938+0000 7fe30cc00640 1 -- 172.21.15.43:0/993723140 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fe3100043c0 con 0x55fc302abca0 2024-03-28T22:40:59.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.938+0000 7fe30cc00640 1 -- 172.21.15.43:0/993723140 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe310024050 con 0x55fc302abca0 2024-03-28T22:40:59.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.938+0000 7fe32c5da740 10 client.27085 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:59.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.938+0000 7fe32c5da740 1 -- 172.21.15.43:0/993723140 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55fc302c3920 con 0x55fc302abca0 2024-03-28T22:40:59.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.939+0000 7fe30cc00640 1 -- 172.21.15.43:0/993723140 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe31c004d20 con 0x55fc302abca0 2024-03-28T22:40:59.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.939+0000 7fe30cc00640 1 -- 172.21.15.43:0/993723140 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe310019030 con 0x55fc302abca0 2024-03-28T22:40:59.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.939+0000 7fe32c5da740 10 client.27085 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:59.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.939+0000 7fe32c5da740 10 client.27085 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:40:59.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.939+0000 7fe32c5da740 20 client.27085 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:59.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.939+0000 7fe32c5da740 1 --2- 172.21.15.43:0/993723140 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55fc302c4de0 0x55fc302c7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.939+0000 7fe32c5da740 4 client.27085 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:40:59.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.939+0000 7fe32c5da740 1 -- 172.21.15.43:0/993723140 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fc301b8680 con 0x55fc302c4de0 2024-03-28T22:40:59.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.939+0000 7fe323400640 1 --2- 172.21.15.43:0/993723140 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55fc302c4de0 0x55fc302c7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.940+0000 7fe323400640 1 --2- 172.21.15.43:0/993723140 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55fc302c4de0 0x55fc302c7210 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1fd64be6dcc98815 server_cookie=f50030346abc9a1f in_seq=0 out_seq=0 2024-03-28T22:40:59.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.940+0000 7fe30cc00640 10 client.27085 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:40:59.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.941+0000 7fe30cc00640 1 -- 172.21.15.43:0/993723140 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fc301b8680 con 0x55fc302c4de0 2024-03-28T22:40:59.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.941+0000 7fe30cc00640 10 client.27085 handle_command_reply: tid=0 2024-03-28T22:40:59.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.941+0000 7fe32c5da740 1 client.27085 shutdown 2024-03-28T22:40:59.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.941+0000 7fe32c5da740 1 -- 172.21.15.43:0/993723140 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc302abca0 msgr2=0x55fc302b1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.941+0000 7fe32c5da740 1 --2- 172.21.15.43:0/993723140 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc302abca0 0x55fc302b1db0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fe3240ffe30 tx=0x7fe310012820 comp rx=0 tx=0).stop 2024-03-28T22:40:59.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.941+0000 7fe32c5da740 1 -- 172.21.15.43:0/993723140 shutdown_connections 2024-03-28T22:40:59.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.941+0000 7fe32c5da740 1 --2- 172.21.15.43:0/993723140 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55fc302c4de0 0x55fc302c7210 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.941+0000 7fe32c5da740 1 --2- 172.21.15.43:0/993723140 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc302b4880 0x55fc302b22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.941+0000 7fe32c5da740 1 --2- 172.21.15.43:0/993723140 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc302abca0 0x55fc302b1db0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.941+0000 7fe32c5da740 1 --2- 172.21.15.43:0/993723140 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc302ab380 0x55fc302b1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.941+0000 7fe30cc00640 0 client.27085 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:40:59.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.941+0000 7fe32c5da740 1 -- 172.21.15.43:0/993723140 >> 172.21.15.43:0/993723140 conn(0x55fc302a5ab0 msgr2=0x55fc302a7800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:59.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.942+0000 7fe32c5da740 1 -- 172.21.15.43:0/993723140 shutdown_connections 2024-03-28T22:40:59.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.942+0000 7fe32c5da740 1 -- 172.21.15.43:0/993723140 wait complete. 2024-03-28T22:40:59.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.942+0000 7fe32c5da740 20 client.27085 trim_cache size 0 max 16384 2024-03-28T22:40:59.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.958+0000 7fe32c5da740 1 Processor -- start 2024-03-28T22:40:59.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.958+0000 7fe32c5da740 1 -- start start 2024-03-28T22:40:59.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.958+0000 7fe32c5da740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc302ab380 0x55fc302b3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.958+0000 7fe32c5da740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc302abca0 0x55fc302c0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.958+0000 7fe32c5da740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc302b4880 0x55fc302a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.958+0000 7fe32c5da740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe3241b0de0 con 0x55fc302abca0 2024-03-28T22:40:59.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.958+0000 7fe32c5da740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe3241b2920 con 0x55fc302b4880 2024-03-28T22:40:59.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.958+0000 7fe32c5da740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe3241b14b0 con 0x55fc302ab380 2024-03-28T22:40:59.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.958+0000 7fe323400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc302ab380 0x55fc302b3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.958+0000 7fe323400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc302ab380 0x55fc302b3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:58966/0 (socket says 172.21.15.43:58966) 2024-03-28T22:40:59.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.958+0000 7fe322000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc302abca0 0x55fc302c0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.958+0000 7fe323400640 1 -- 172.21.15.43:0/3900489346 learned_addr learned my addr 172.21.15.43:0/3900489346 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:59.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.958+0000 7fe323400640 1 --2- 172.21.15.43:0/3900489346 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc302b4880 0x55fc302a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.959+0000 7fe323400640 1 -- 172.21.15.43:0/3900489346 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc302b4880 msgr2=0x55fc302a49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:59.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.959+0000 7fe323400640 1 --2- 172.21.15.43:0/3900489346 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc302b4880 0x55fc302a49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.959+0000 7fe323400640 1 -- 172.21.15.43:0/3900489346 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc302abca0 msgr2=0x55fc302c0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:59.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.959+0000 7fe323400640 1 --2- 172.21.15.43:0/3900489346 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc302abca0 0x55fc302c0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.959+0000 7fe323400640 1 -- 172.21.15.43:0/3900489346 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc302aad50 con 0x55fc302ab380 2024-03-28T22:40:59.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.959+0000 7fe322000640 1 --2- 172.21.15.43:0/3900489346 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc302abca0 0x55fc302c0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:59.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.959+0000 7fe323400640 1 --2- 172.21.15.43:0/3900489346 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc302ab380 0x55fc302b3fa0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe31c005b40 tx=0x7fe31c005a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e92d4b8795020a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:59.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.959+0000 7fe30f400640 1 -- 172.21.15.43:0/3900489346 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31c016050 con 0x55fc302ab380 2024-03-28T22:40:59.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.959+0000 7fe30f400640 1 -- 172.21.15.43:0/3900489346 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe31c025040 con 0x55fc302ab380 2024-03-28T22:40:59.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.959+0000 7fe30f400640 1 -- 172.21.15.43:0/3900489346 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31c024040 con 0x55fc302ab380 2024-03-28T22:40:59.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.960+0000 7fe32c5da740 1 -- 172.21.15.43:0/3900489346 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc302ab380 msgr2=0x55fc302b3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.960+0000 7fe32c5da740 1 --2- 172.21.15.43:0/3900489346 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc302ab380 0x55fc302b3fa0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe31c005b40 tx=0x7fe31c005a70 comp rx=0 tx=0).stop 2024-03-28T22:40:59.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.960+0000 7fe32c5da740 1 -- 172.21.15.43:0/3900489346 shutdown_connections 2024-03-28T22:40:59.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.960+0000 7fe32c5da740 1 --2- 172.21.15.43:0/3900489346 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc302b4880 0x55fc302a49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.960+0000 7fe32c5da740 1 --2- 172.21.15.43:0/3900489346 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc302abca0 0x55fc302c0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.960+0000 7fe32c5da740 1 --2- 172.21.15.43:0/3900489346 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc302ab380 0x55fc302b3fa0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.960+0000 7fe32c5da740 1 -- 172.21.15.43:0/3900489346 >> 172.21.15.43:0/3900489346 conn(0x55fc302a5ab0 msgr2=0x55fc302a42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:59.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.961+0000 7fe32c5da740 1 -- 172.21.15.43:0/3900489346 shutdown_connections 2024-03-28T22:40:59.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.961+0000 7fe32c5da740 1 -- 172.21.15.43:0/3900489346 wait complete. 2024-03-28T22:40:59.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.961+0000 7fe32c5da740 1 -- start start 2024-03-28T22:40:59.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.961+0000 7fe32c5da740 1 Processor -- start 2024-03-28T22:40:59.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.962+0000 7fe32c5da740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc302ab380 0x55fc302ae660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.962+0000 7fe32c5da740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc302abca0 0x55fc302aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.962+0000 7fe32c5da740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc302b4880 0x55fc302af0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.962+0000 7fe32c5da740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe3241b2250 con 0x55fc302abca0 2024-03-28T22:40:59.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.962+0000 7fe32c5da740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe3241b1b80 con 0x55fc302ab380 2024-03-28T22:40:59.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.962+0000 7fe32c5da740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe3241aedc0 con 0x55fc302b4880 2024-03-28T22:40:59.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.962+0000 7fe32c5da740 10 client.0 _ll_register_callbacks cb 0x7fe31000bac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:40:59.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.962+0000 7fe322000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc302abca0 0x55fc302aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.962+0000 7fe323400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc302ab380 0x55fc302ae660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.962+0000 7fe322000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc302abca0 0x55fc302aeba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42202/0 (socket says 172.21.15.43:42202) 2024-03-28T22:40:59.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.962+0000 7fe322000640 1 -- 172.21.15.43:0/2104970245 learned_addr learned my addr 172.21.15.43:0/2104970245 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:40:59.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.962+0000 7fe323400640 1 --2- 172.21.15.43:0/2104970245 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc302b4880 0x55fc302af0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.963+0000 7fe322000640 1 -- 172.21.15.43:0/2104970245 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc302b4880 msgr2=0x55fc302af0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:59.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.963+0000 7fe322000640 1 --2- 172.21.15.43:0/2104970245 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc302b4880 0x55fc302af0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.963+0000 7fe322000640 1 -- 172.21.15.43:0/2104970245 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc302ab380 msgr2=0x55fc302ae660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:40:59.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.963+0000 7fe322000640 1 --2- 172.21.15.43:0/2104970245 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc302ab380 0x55fc302ae660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.963+0000 7fe322000640 1 -- 172.21.15.43:0/2104970245 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc3032a360 con 0x55fc302abca0 2024-03-28T22:40:59.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.963+0000 7fe323400640 1 --2- 172.21.15.43:0/2104970245 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc302ab380 0x55fc302ae660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:40:59.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.963+0000 7fe322000640 1 --2- 172.21.15.43:0/2104970245 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc302abca0 0x55fc302aeba0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x55fc302b10d0 tx=0x7fe310010990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6bb6f1c3c0c7c5e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:40:59.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.963+0000 7fe30cc00640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:40:59.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.963+0000 7fe30cc00640 1 -- 172.21.15.43:0/2104970245 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe310005850 con 0x55fc302abca0 2024-03-28T22:40:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.964+0000 7fe32c5da740 1 -- 172.21.15.43:0/2104970245 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe31c025040 con 0x55fc302abca0 2024-03-28T22:40:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.964+0000 7fe30cc00640 1 -- 172.21.15.43:0/2104970245 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fe310004020 con 0x55fc302abca0 2024-03-28T22:40:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.964+0000 7fe30cc00640 1 -- 172.21.15.43:0/2104970245 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe310039050 con 0x55fc302abca0 2024-03-28T22:40:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.964+0000 7fe32c5da740 10 client.16929 fetch_fsmap learned FSMap version 32 2024-03-28T22:40:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.964+0000 7fe32c5da740 1 -- 172.21.15.43:0/2104970245 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55fc3032b690 con 0x55fc302abca0 2024-03-28T22:40:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.964+0000 7fe30cc00640 1 -- 172.21.15.43:0/2104970245 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe31c025040 con 0x55fc302abca0 2024-03-28T22:40:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.964+0000 7fe30cc00640 1 -- 172.21.15.43:0/2104970245 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe310019040 con 0x55fc302abca0 2024-03-28T22:40:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.964+0000 7fe32c5da740 10 client.16929 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:40:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.964+0000 7fe32c5da740 10 client.16929 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:40:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.964+0000 7fe32c5da740 20 client.16929 populate_metadata read hostname 'smithi043' 2024-03-28T22:40:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.964+0000 7fe32c5da740 1 --2- 172.21.15.43:0/2104970245 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55fc3032be40 0x55fc3032e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:40:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.964+0000 7fe32c5da740 4 client.16929 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:40:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.964+0000 7fe32c5da740 1 -- 172.21.15.43:0/2104970245 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fc3021c590 con 0x55fc3032be40 2024-03-28T22:40:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.965+0000 7fe323400640 1 --2- 172.21.15.43:0/2104970245 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55fc3032be40 0x55fc3032e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:40:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.965+0000 7fe323400640 1 --2- 172.21.15.43:0/2104970245 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55fc3032be40 0x55fc3032e260 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a67fb4287a1d852f server_cookie=258e7342c0480b in_seq=0 out_seq=0 2024-03-28T22:40:59.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.965+0000 7fe30cc00640 10 client.16929 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:40:59.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.966+0000 7fe30cc00640 1 -- 172.21.15.43:0/2104970245 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fc3021c590 con 0x55fc3032be40 2024-03-28T22:40:59.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.966+0000 7fe30cc00640 10 client.16929 handle_command_reply: tid=0 2024-03-28T22:40:59.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.966+0000 7fe32c5da740 1 client.16929 shutdown 2024-03-28T22:40:59.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.966+0000 7fe32c5da740 1 -- 172.21.15.43:0/2104970245 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc302abca0 msgr2=0x55fc302aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.966+0000 7fe32c5da740 1 --2- 172.21.15.43:0/2104970245 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc302abca0 0x55fc302aeba0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x55fc302b10d0 tx=0x7fe310010990 comp rx=0 tx=0).stop 2024-03-28T22:40:59.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.966+0000 7fe32c5da740 1 -- 172.21.15.43:0/2104970245 shutdown_connections 2024-03-28T22:40:59.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.966+0000 7fe32c5da740 1 --2- 172.21.15.43:0/2104970245 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55fc3032be40 0x55fc3032e260 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.966+0000 7fe32c5da740 1 --2- 172.21.15.43:0/2104970245 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc302b4880 0x55fc302af0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.966+0000 7fe32c5da740 1 --2- 172.21.15.43:0/2104970245 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc302abca0 0x55fc302aeba0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.966+0000 7fe32c5da740 1 --2- 172.21.15.43:0/2104970245 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc302ab380 0x55fc302ae660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.966+0000 7fe32c5da740 1 -- 172.21.15.43:0/2104970245 >> 172.21.15.43:0/2104970245 conn(0x55fc302a5ab0 msgr2=0x55fc302a4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:59.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.966+0000 7fe30cc00640 0 client.16929 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:40:59.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.966+0000 7fe32c5da740 1 -- 172.21.15.43:0/2104970245 shutdown_connections 2024-03-28T22:40:59.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.966+0000 7fe32c5da740 1 -- 172.21.15.43:0/2104970245 wait complete. 2024-03-28T22:40:59.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.966+0000 7fe32c5da740 20 client.16929 trim_cache size 0 max 16384 2024-03-28T22:40:59.969 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:40:59.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.967+0000 7fe329800640 1 -- 172.21.15.43:0/765205693 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe3080a1f70 msgr2=0x7fe3080a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.967+0000 7fe329800640 1 --2- 172.21.15.43:0/765205693 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe3080a1f70 0x7fe3080a4430 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe3140ed3f0 tx=0x7fe3140f39d0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.967+0000 7fe329800640 1 -- 172.21.15.43:0/765205693 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe3241005c0 msgr2=0x7fe32419a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:40:59.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.967+0000 7fe329800640 1 --2- 172.21.15.43:0/765205693 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe3241005c0 0x7fe32419a1a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe314000c00 tx=0x7fe31402cbe0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.967+0000 7fe329800640 1 -- 172.21.15.43:0/765205693 shutdown_connections 2024-03-28T22:40:59.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.967+0000 7fe329800640 1 --2- 172.21.15.43:0/765205693 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe3080a1f70 0x7fe3080a4430 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.967+0000 7fe329800640 1 --2- 172.21.15.43:0/765205693 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe324100f80 0x7fe32419a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.967+0000 7fe329800640 1 --2- 172.21.15.43:0/765205693 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe324106660 0x7fe32419ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.967+0000 7fe329800640 1 --2- 172.21.15.43:0/765205693 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe3241005c0 0x7fe32419a1a0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:40:59.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.967+0000 7fe329800640 1 -- 172.21.15.43:0/765205693 >> 172.21.15.43:0/765205693 conn(0x7fe324077750 msgr2=0x7fe3240688b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:40:59.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.967+0000 7fe329800640 1 -- 172.21.15.43:0/765205693 shutdown_connections 2024-03-28T22:40:59.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:40:59.967+0000 7fe329800640 1 -- 172.21.15.43:0/765205693 wait complete. 2024-03-28T22:40:59.984 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:41:00.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.204+0000 7f03eb400640 1 -- 172.21.15.43:0/3062515112 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03e41076f0 msgr2=0x7f03e4107ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:00.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.204+0000 7f03eb400640 1 --2- 172.21.15.43:0/3062515112 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03e41076f0 0x7f03e4107ad0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f03d8009890 tx=0x7f03d802bb70 comp rx=0 tx=0).stop 2024-03-28T22:41:00.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.204+0000 7f03eb400640 1 -- 172.21.15.43:0/3062515112 shutdown_connections 2024-03-28T22:41:00.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.204+0000 7f03eb400640 1 --2- 172.21.15.43:0/3062515112 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03e41020a0 0x7f03e410e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:00.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.204+0000 7f03eb400640 1 --2- 172.21.15.43:0/3062515112 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03e41016e0 0x7f03e4101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:00.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.204+0000 7f03eb400640 1 --2- 172.21.15.43:0/3062515112 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03e41076f0 0x7f03e4107ad0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:00.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.204+0000 7f03eb400640 1 -- 172.21.15.43:0/3062515112 >> 172.21.15.43:0/3062515112 conn(0x7f03e4078b50 msgr2=0x7f03e4078f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:00.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.205+0000 7f03eb400640 1 -- 172.21.15.43:0/3062515112 shutdown_connections 2024-03-28T22:41:00.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.205+0000 7f03eb400640 1 -- 172.21.15.43:0/3062515112 wait complete. 2024-03-28T22:41:00.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.206+0000 7f03eb400640 1 Processor -- start 2024-03-28T22:41:00.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.206+0000 7f03eb400640 1 -- start start 2024-03-28T22:41:00.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.206+0000 7f03eb400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03e41016e0 0x7f03e4073a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:00.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.206+0000 7f03eb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03e41020a0 0x7f03e4073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:00.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.206+0000 7f03eb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03e41076f0 0x7f03e40744d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:00.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.206+0000 7f03eb400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f03e4111ac0 con 0x7f03e41020a0 2024-03-28T22:41:00.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.206+0000 7f03eb400640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f03e4111940 con 0x7f03e41076f0 2024-03-28T22:41:00.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.206+0000 7f03eb400640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f03e4111c40 con 0x7f03e41016e0 2024-03-28T22:41:00.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.206+0000 7f03e8a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03e41016e0 0x7f03e4073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:00.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.206+0000 7f03e8a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03e41016e0 0x7f03e4073a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:59012/0 (socket says 172.21.15.43:59012) 2024-03-28T22:41:00.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.206+0000 7f03e8a00640 1 -- 172.21.15.43:0/2950352730 learned_addr learned my addr 172.21.15.43:0/2950352730 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:00.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.206+0000 7f03e9400640 1 --2- 172.21.15.43:0/2950352730 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03e41076f0 0x7f03e40744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:00.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.207+0000 7f03e3e00640 1 --2- 172.21.15.43:0/2950352730 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03e41020a0 0x7f03e4073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:00.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.207+0000 7f03e8a00640 1 -- 172.21.15.43:0/2950352730 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03e41076f0 msgr2=0x7f03e40744d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:00.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.207+0000 7f03e8a00640 1 --2- 172.21.15.43:0/2950352730 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03e41076f0 0x7f03e40744d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:00.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.207+0000 7f03e8a00640 1 -- 172.21.15.43:0/2950352730 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03e41020a0 msgr2=0x7f03e4073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:00.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.207+0000 7f03e8a00640 1 --2- 172.21.15.43:0/2950352730 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03e41020a0 0x7f03e4073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:00.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.207+0000 7f03e8a00640 1 -- 172.21.15.43:0/2950352730 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03e41a6500 con 0x7f03e41016e0 2024-03-28T22:41:00.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.207+0000 7f03e3e00640 1 --2- 172.21.15.43:0/2950352730 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03e41020a0 0x7f03e4073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:41:00.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.207+0000 7f03e8a00640 1 --2- 172.21.15.43:0/2950352730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03e41016e0 0x7f03e4073a50 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f03d80042f0 tx=0x7f03d8004320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:00.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.207+0000 7f03e1600640 1 -- 172.21.15.43:0/2950352730 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d8041020 con 0x7f03e41016e0 2024-03-28T22:41:00.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.207+0000 7f03eb400640 1 -- 172.21.15.43:0/2950352730 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03e41a6790 con 0x7f03e41016e0 2024-03-28T22:41:00.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.207+0000 7f03e1600640 1 -- 172.21.15.43:0/2950352730 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03d802c9f0 con 0x7f03e41016e0 2024-03-28T22:41:00.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.207+0000 7f03e1600640 1 -- 172.21.15.43:0/2950352730 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d8039040 con 0x7f03e41016e0 2024-03-28T22:41:00.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.208+0000 7f03eb400640 1 -- 172.21.15.43:0/2950352730 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03e41a6c80 con 0x7f03e41016e0 2024-03-28T22:41:00.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.211+0000 7f03e1600640 1 -- 172.21.15.43:0/2950352730 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f03d8044020 con 0x7f03e41016e0 2024-03-28T22:41:00.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.211+0000 7f03edfdd740 1 Processor -- start 2024-03-28T22:41:00.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.211+0000 7f03edfdd740 1 -- start start 2024-03-28T22:41:00.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.212+0000 7f03edfdd740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632c07f8380 0x5632c07f8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:00.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.212+0000 7f03edfdd740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5632c07f8ca0 0x5632c0801340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:00.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.212+0000 7f03edfdd740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5632c0801880 0x5632c0803c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:00.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.212+0000 7f03edfdd740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5632c075c050 con 0x5632c07f8380 2024-03-28T22:41:00.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.212+0000 7f03edfdd740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5632c0704ce0 con 0x5632c0801880 2024-03-28T22:41:00.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.212+0000 7f03edfdd740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5632c07580f0 con 0x5632c07f8ca0 2024-03-28T22:41:00.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.212+0000 7f03e3e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5632c0801880 0x5632c0803c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:00.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.212+0000 7f03e8a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5632c07f8ca0 0x5632c0801340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:00.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.212+0000 7f03e3e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5632c0801880 0x5632c0803c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:60244/0 (socket says 172.21.15.43:60244) 2024-03-28T22:41:00.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.212+0000 7f03e3e00640 1 -- 172.21.15.43:0/3877157047 learned_addr learned my addr 172.21.15.43:0/3877157047 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:00.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.212+0000 7f03e9400640 1 --2- 172.21.15.43:0/3877157047 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632c07f8380 0x5632c07f8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:00.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.212+0000 7f03e1600640 1 --2- 172.21.15.43:0/2950352730 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f03cc0a1ef0 0x7f03cc0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:00.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.213+0000 7f03e3e00640 1 -- 172.21.15.43:0/3877157047 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5632c07f8ca0 msgr2=0x5632c0801340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:00.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.213+0000 7f03e3e00640 1 --2- 172.21.15.43:0/3877157047 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5632c07f8ca0 0x5632c0801340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:00.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.213+0000 7f03e3e00640 1 -- 172.21.15.43:0/3877157047 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632c07f8380 msgr2=0x5632c07f8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:00.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.213+0000 7f03e3e00640 1 --2- 172.21.15.43:0/3877157047 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632c07f8380 0x5632c07f8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:00.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.213+0000 7f03e3e00640 1 -- 172.21.15.43:0/3877157047 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632c04703f0 con 0x5632c0801880 2024-03-28T22:41:00.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.213+0000 7f03e8a00640 1 --2- 172.21.15.43:0/3877157047 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5632c07f8ca0 0x5632c0801340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:41:00.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.213+0000 7f03e1600640 1 -- 172.21.15.43:0/2950352730 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f03d80efee0 con 0x7f03e41016e0 2024-03-28T22:41:00.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.213+0000 7f03e9400640 1 --2- 172.21.15.43:0/2950352730 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f03cc0a1ef0 0x7f03cc0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:00.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.213+0000 7f03e3e00640 1 --2- 172.21.15.43:0/3877157047 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5632c0801880 0x5632c0803c70 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f03d0009cf0 tx=0x7f03d00117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb7c9426f2012f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:00.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.213+0000 7f03d5c00640 1 -- 172.21.15.43:0/3877157047 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d0012450 con 0x5632c0801880 2024-03-28T22:41:00.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.214+0000 7f03d5c00640 1 -- 172.21.15.43:0/3877157047 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03d0012ad0 con 0x5632c0801880 2024-03-28T22:41:00.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.214+0000 7f03d5c00640 1 -- 172.21.15.43:0/3877157047 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d000fca0 con 0x5632c0801880 2024-03-28T22:41:00.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.214+0000 7f03edfdd740 1 -- 172.21.15.43:0/3877157047 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5632c0801880 msgr2=0x5632c0803c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:00.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.214+0000 7f03edfdd740 1 --2- 172.21.15.43:0/3877157047 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5632c0801880 0x5632c0803c70 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f03d0009cf0 tx=0x7f03d00117b0 comp rx=0 tx=0).stop 2024-03-28T22:41:00.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.214+0000 7f03e9400640 1 --2- 172.21.15.43:0/2950352730 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f03cc0a1ef0 0x7f03cc0a43b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f03dc0049d0 tx=0x7f03dc004a40 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:00.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.214+0000 7f03edfdd740 1 -- 172.21.15.43:0/3877157047 shutdown_connections 2024-03-28T22:41:00.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.214+0000 7f03edfdd740 1 --2- 172.21.15.43:0/3877157047 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5632c0801880 0x5632c0803c70 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:00.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.214+0000 7f03edfdd740 1 --2- 172.21.15.43:0/3877157047 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5632c07f8ca0 0x5632c0801340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:00.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.214+0000 7f03edfdd740 1 --2- 172.21.15.43:0/3877157047 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632c07f8380 0x5632c07f8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:00.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.214+0000 7f03edfdd740 1 -- 172.21.15.43:0/3877157047 >> 172.21.15.43:0/3877157047 conn(0x5632c07f2ab0 msgr2=0x5632c07f2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:00.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.215+0000 7f03edfdd740 1 -- 172.21.15.43:0/3877157047 shutdown_connections 2024-03-28T22:41:00.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.215+0000 7f03edfdd740 1 -- 172.21.15.43:0/3877157047 wait complete. 2024-03-28T22:41:00.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.215+0000 7f03edfdd740 1 -- start start 2024-03-28T22:41:00.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.215+0000 7f03edfdd740 1 Processor -- start 2024-03-28T22:41:00.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.216+0000 7f03edfdd740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5632c07f8380 0x5632c07fe850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:00.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.216+0000 7f03edfdd740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632c07f8ca0 0x5632c07fed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:00.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.216+0000 7f03edfdd740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5632c0801880 0x5632c07ff2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:00.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.216+0000 7f03edfdd740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5632c0804670 con 0x5632c07f8ca0 2024-03-28T22:41:00.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.216+0000 7f03edfdd740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5632c06fa2f0 con 0x5632c0801880 2024-03-28T22:41:00.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.216+0000 7f03edfdd740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5632c08047f0 con 0x5632c07f8380 2024-03-28T22:41:00.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.216+0000 7f03edfdd740 10 client.0 _ll_register_callbacks cb 0x5632c046fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:41:00.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.216+0000 7f03e3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632c07f8ca0 0x5632c07fed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:00.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.216+0000 7f03e9400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5632c0801880 0x5632c07ff2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:00.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.216+0000 7f03e8a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5632c07f8380 0x5632c07fe850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:00.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.216+0000 7f03e3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632c07f8ca0 0x5632c07fed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42236/0 (socket says 172.21.15.43:42236) 2024-03-28T22:41:00.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.216+0000 7f03e3e00640 1 -- 172.21.15.43:0/3600808878 learned_addr learned my addr 172.21.15.43:0/3600808878 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:00.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.216+0000 7f03e8a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5632c07f8380 0x5632c07fe850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:59036/0 (socket says 172.21.15.43:59036) 2024-03-28T22:41:00.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.216+0000 7f03e9400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5632c0801880 0x5632c07ff2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:60252/0 (socket says 172.21.15.43:60252) 2024-03-28T22:41:00.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.217+0000 7f03e9400640 1 -- 172.21.15.43:0/3600808878 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5632c07f8380 msgr2=0x5632c07fe850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:00.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.217+0000 7f03e9400640 1 --2- 172.21.15.43:0/3600808878 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5632c07f8380 0x5632c07fe850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:00.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.217+0000 7f03e9400640 1 -- 172.21.15.43:0/3600808878 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632c07f8ca0 msgr2=0x5632c07fed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:00.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.217+0000 7f03e9400640 1 --2- 172.21.15.43:0/3600808878 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632c07f8ca0 0x5632c07fed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:00.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.217+0000 7f03e9400640 1 -- 172.21.15.43:0/3600808878 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632c053a7d0 con 0x5632c0801880 2024-03-28T22:41:00.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.217+0000 7f03e3e00640 1 --2- 172.21.15.43:0/3600808878 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632c07f8ca0 0x5632c07fed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:41:00.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.217+0000 7f03e9400640 1 --2- 172.21.15.43:0/3600808878 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5632c0801880 0x5632c07ff2d0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f03dc01ada0 tx=0x7f03dc010040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11ee149d035ea06d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:00.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.217+0000 7f03ca000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:41:00.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.217+0000 7f03ca000640 1 -- 172.21.15.43:0/3600808878 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03dc02a060 con 0x5632c0801880 2024-03-28T22:41:00.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.218+0000 7f03ca000640 1 -- 172.21.15.43:0/3600808878 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f03dc01a230 con 0x5632c0801880 2024-03-28T22:41:00.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.218+0000 7f03edfdd740 1 -- 172.21.15.43:0/3600808878 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03d0012ad0 con 0x5632c0801880 2024-03-28T22:41:00.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.218+0000 7f03ca000640 1 -- 172.21.15.43:0/3600808878 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f03dc031050 con 0x5632c0801880 2024-03-28T22:41:00.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.218+0000 7f03edfdd740 10 client.27143 fetch_fsmap learned FSMap version 32 2024-03-28T22:41:00.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.218+0000 7f03edfdd740 1 -- 172.21.15.43:0/3600808878 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5632c0810910 con 0x5632c0801880 2024-03-28T22:41:00.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.218+0000 7f03ca000640 1 -- 172.21.15.43:0/3600808878 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03d0012ad0 con 0x5632c0801880 2024-03-28T22:41:00.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.218+0000 7f03ca000640 1 -- 172.21.15.43:0/3600808878 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f03dc00b040 con 0x5632c0801880 2024-03-28T22:41:00.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.219+0000 7f03edfdd740 10 client.27143 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:41:00.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.219+0000 7f03edfdd740 10 client.27143 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:41:00.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.219+0000 7f03edfdd740 20 client.27143 populate_metadata read hostname 'smithi043' 2024-03-28T22:41:00.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.219+0000 7f03edfdd740 1 --2- 172.21.15.43:0/3600808878 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5632c0811dd0 0x5632c0814200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:00.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.219+0000 7f03edfdd740 4 client.27143 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:41:00.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.219+0000 7f03edfdd740 1 -- 172.21.15.43:0/3600808878 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632c0705680 con 0x5632c0811dd0 2024-03-28T22:41:00.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.219+0000 7f03e8a00640 1 --2- 172.21.15.43:0/3600808878 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5632c0811dd0 0x5632c0814200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:00.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.220+0000 7f03e8a00640 1 --2- 172.21.15.43:0/3600808878 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5632c0811dd0 0x5632c0814200 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37da659b6720ae0a server_cookie=8ee5822abff5bde7 in_seq=0 out_seq=0 2024-03-28T22:41:00.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.220+0000 7f03ca000640 10 client.27143 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:00.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.220+0000 7f03ca000640 1 -- 172.21.15.43:0/3600808878 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5632c0705680 con 0x5632c0811dd0 2024-03-28T22:41:00.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.220+0000 7f03ca000640 10 client.27143 handle_command_reply: tid=0 2024-03-28T22:41:00.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.220+0000 7f03edfdd740 1 client.27143 shutdown 2024-03-28T22:41:00.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.220+0000 7f03edfdd740 1 -- 172.21.15.43:0/3600808878 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5632c0801880 msgr2=0x5632c07ff2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:00.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.221+0000 7f03edfdd740 1 --2- 172.21.15.43:0/3600808878 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5632c0801880 0x5632c07ff2d0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f03dc01ada0 tx=0x7f03dc010040 comp rx=0 tx=0).stop 2024-03-28T22:41:00.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.221+0000 7f03edfdd740 1 -- 172.21.15.43:0/3600808878 shutdown_connections 2024-03-28T22:41:00.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.221+0000 7f03edfdd740 1 --2- 172.21.15.43:0/3600808878 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5632c0811dd0 0x5632c0814200 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:00.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.221+0000 7f03edfdd740 1 --2- 172.21.15.43:0/3600808878 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5632c0801880 0x5632c07ff2d0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:00.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.221+0000 7f03edfdd740 1 --2- 172.21.15.43:0/3600808878 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632c07f8ca0 0x5632c07fed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:00.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.221+0000 7f03edfdd740 1 --2- 172.21.15.43:0/3600808878 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5632c07f8380 0x5632c07fe850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:00.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.221+0000 7f03ca000640 0 client.27143 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:00.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.221+0000 7f03edfdd740 1 -- 172.21.15.43:0/3600808878 >> 172.21.15.43:0/3600808878 conn(0x5632c07f2ab0 msgr2=0x5632c07f4800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:00.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.221+0000 7f03edfdd740 1 -- 172.21.15.43:0/3600808878 shutdown_connections 2024-03-28T22:41:00.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.221+0000 7f03edfdd740 1 -- 172.21.15.43:0/3600808878 wait complete. 2024-03-28T22:41:00.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.221+0000 7f03edfdd740 20 client.27143 trim_cache size 0 max 16384 2024-03-28T22:41:00.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.236+0000 7f03edfdd740 1 Processor -- start 2024-03-28T22:41:00.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.237+0000 7f03edfdd740 1 -- start start 2024-03-28T22:41:00.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.237+0000 7f03edfdd740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632c07f8380 0x5632c0800f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:00.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.237+0000 7f03edfdd740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5632c07f8ca0 0x5632c080d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:00.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.237+0000 7f03edfdd740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5632c0801880 0x5632c07f19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:00.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.237+0000 7f03edfdd740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f03e41b0e50 con 0x5632c07f8380 2024-03-28T22:41:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.237+0000 7f03edfdd740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f03e41b2990 con 0x5632c0801880 2024-03-28T22:41:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.237+0000 7f03edfdd740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f03e41b1520 con 0x5632c07f8ca0 2024-03-28T22:41:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.237+0000 7f03e3e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5632c07f8ca0 0x5632c080d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.237+0000 7f03e9400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5632c0801880 0x5632c07f19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.237+0000 7f03e3e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5632c07f8ca0 0x5632c080d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:59052/0 (socket says 172.21.15.43:59052) 2024-03-28T22:41:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.237+0000 7f03e3e00640 1 -- 172.21.15.43:0/3009708653 learned_addr learned my addr 172.21.15.43:0/3009708653 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.237+0000 7f03e8a00640 1 --2- 172.21.15.43:0/3009708653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632c07f8380 0x5632c0800f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.238+0000 7f03e9400640 1 -- 172.21.15.43:0/3009708653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5632c07f8ca0 msgr2=0x5632c080d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.238+0000 7f03e9400640 1 --2- 172.21.15.43:0/3009708653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5632c07f8ca0 0x5632c080d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.238+0000 7f03e9400640 1 -- 172.21.15.43:0/3009708653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632c07f8380 msgr2=0x5632c0800f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.238+0000 7f03e9400640 1 --2- 172.21.15.43:0/3009708653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632c07f8380 0x5632c0800f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.238+0000 7f03e9400640 1 -- 172.21.15.43:0/3009708653 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632c0803c70 con 0x5632c0801880 2024-03-28T22:41:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.238+0000 7f03e3e00640 1 --2- 172.21.15.43:0/3009708653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5632c07f8ca0 0x5632c080d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:41:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.238+0000 7f03e8a00640 1 --2- 172.21.15.43:0/3009708653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632c07f8380 0x5632c0800f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:41:00.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.238+0000 7f03e9400640 1 --2- 172.21.15.43:0/3009708653 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5632c0801880 0x5632c07f19f0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f03dc0099f0 tx=0x7f03dc00df40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b92c44572dc752ed server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:00.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.238+0000 7f03d5200640 1 -- 172.21.15.43:0/3009708653 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03dc03a070 con 0x5632c0801880 2024-03-28T22:41:00.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.238+0000 7f03d5200640 1 -- 172.21.15.43:0/3009708653 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03dc01e070 con 0x5632c0801880 2024-03-28T22:41:00.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.238+0000 7f03d5200640 1 -- 172.21.15.43:0/3009708653 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03dc005040 con 0x5632c0801880 2024-03-28T22:41:00.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.239+0000 7f03edfdd740 1 -- 172.21.15.43:0/3009708653 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5632c0801880 msgr2=0x5632c07f19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:00.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.239+0000 7f03edfdd740 1 --2- 172.21.15.43:0/3009708653 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5632c0801880 0x5632c07f19f0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f03dc0099f0 tx=0x7f03dc00df40 comp rx=0 tx=0).stop 2024-03-28T22:41:00.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.239+0000 7f03edfdd740 1 -- 172.21.15.43:0/3009708653 shutdown_connections 2024-03-28T22:41:00.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.239+0000 7f03edfdd740 1 --2- 172.21.15.43:0/3009708653 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5632c0801880 0x5632c07f19f0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:00.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.239+0000 7f03edfdd740 1 --2- 172.21.15.43:0/3009708653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5632c07f8ca0 0x5632c080d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:00.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.239+0000 7f03edfdd740 1 --2- 172.21.15.43:0/3009708653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632c07f8380 0x5632c0800f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:00.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.239+0000 7f03edfdd740 1 -- 172.21.15.43:0/3009708653 >> 172.21.15.43:0/3009708653 conn(0x5632c07f2ab0 msgr2=0x5632c07f12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:00.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.239+0000 7f03edfdd740 1 -- 172.21.15.43:0/3009708653 shutdown_connections 2024-03-28T22:41:00.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.239+0000 7f03edfdd740 1 -- 172.21.15.43:0/3009708653 wait complete. 2024-03-28T22:41:00.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.239+0000 7f03edfdd740 1 -- start start 2024-03-28T22:41:00.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.240+0000 7f03edfdd740 1 Processor -- start 2024-03-28T22:41:00.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.240+0000 7f03edfdd740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5632c07f8380 0x5632c08056c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:00.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.240+0000 7f03edfdd740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5632c07f8ca0 0x5632c0805c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:00.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.240+0000 7f03edfdd740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632c0801880 0x5632c0877100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:00.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.240+0000 7f03edfdd740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f03e41b22c0 con 0x5632c0801880 2024-03-28T22:41:00.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.240+0000 7f03edfdd740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f03e41b1bf0 con 0x5632c07f8ca0 2024-03-28T22:41:00.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.240+0000 7f03edfdd740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f03e41aee30 con 0x5632c07f8380 2024-03-28T22:41:00.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.240+0000 7f03edfdd740 10 client.0 _ll_register_callbacks cb 0x7f03dc009840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:41:00.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.241+0000 7f03e3e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5632c07f8ca0 0x5632c0805c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:00.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.241+0000 7f03e8a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5632c07f8380 0x5632c08056c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:00.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.241+0000 7f03e8a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5632c07f8380 0x5632c08056c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:59056/0 (socket says 172.21.15.43:59056) 2024-03-28T22:41:00.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.241+0000 7f03e8a00640 1 -- 172.21.15.43:0/373677623 learned_addr learned my addr 172.21.15.43:0/373677623 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:00.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.241+0000 7f03e3e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5632c07f8ca0 0x5632c0805c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:60268/0 (socket says 172.21.15.43:60268) 2024-03-28T22:41:00.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.241+0000 7f03e9400640 1 --2- 172.21.15.43:0/373677623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632c0801880 0x5632c0877100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:00.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.241+0000 7f03e8a00640 1 -- 172.21.15.43:0/373677623 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5632c07f8ca0 msgr2=0x5632c0805c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:00.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.241+0000 7f03e8a00640 1 --2- 172.21.15.43:0/373677623 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5632c07f8ca0 0x5632c0805c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:00.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.241+0000 7f03e8a00640 1 -- 172.21.15.43:0/373677623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632c0801880 msgr2=0x5632c0877100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:00.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.241+0000 7f03e8a00640 1 --2- 172.21.15.43:0/373677623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632c0801880 0x5632c0877100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:00.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.241+0000 7f03e8a00640 1 -- 172.21.15.43:0/373677623 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632c0806500 con 0x5632c07f8380 2024-03-28T22:41:00.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.241+0000 7f03e8a00640 1 --2- 172.21.15.43:0/373677623 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5632c07f8380 0x5632c08056c0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f03d802c180 tx=0x7f03d80f3990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a90c6b2d3871bc99 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:00.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.241+0000 7f03ca000640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:41:00.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.241+0000 7f03ca000640 1 -- 172.21.15.43:0/373677623 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d80bc3b0 con 0x5632c07f8380 2024-03-28T22:41:00.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.242+0000 7f03ca000640 1 -- 172.21.15.43:0/373677623 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f03d80dc0b0 con 0x5632c07f8380 2024-03-28T22:41:00.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.242+0000 7f03edfdd740 1 -- 172.21.15.43:0/373677623 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03dc01e070 con 0x5632c07f8380 2024-03-28T22:41:00.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.242+0000 7f03ca000640 1 -- 172.21.15.43:0/373677623 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f03d80bbf40 con 0x5632c07f8380 2024-03-28T22:41:00.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.242+0000 7f03edfdd740 10 client.27109 fetch_fsmap learned FSMap version 32 2024-03-28T22:41:00.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.242+0000 7f03edfdd740 1 -- 172.21.15.43:0/373677623 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5632c08787d0 con 0x5632c07f8380 2024-03-28T22:41:00.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.242+0000 7f03ca000640 1 -- 172.21.15.43:0/373677623 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03dc01e070 con 0x5632c07f8380 2024-03-28T22:41:00.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.242+0000 7f03ca000640 1 -- 172.21.15.43:0/373677623 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f03d803c030 con 0x5632c07f8380 2024-03-28T22:41:00.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.242+0000 7f03edfdd740 10 client.27109 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:41:00.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.242+0000 7f03edfdd740 10 client.27109 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:41:00.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.242+0000 7f03edfdd740 20 client.27109 populate_metadata read hostname 'smithi043' 2024-03-28T22:41:00.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.242+0000 7f03edfdd740 1 --2- 172.21.15.43:0/373677623 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5632c0878ef0 0x5632c087b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:00.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.242+0000 7f03edfdd740 4 client.27109 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:41:00.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.242+0000 7f03edfdd740 1 -- 172.21.15.43:0/373677623 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5632c0769590 con 0x5632c0878ef0 2024-03-28T22:41:00.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.242+0000 7f03e3e00640 1 --2- 172.21.15.43:0/373677623 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5632c0878ef0 0x5632c087b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:00.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.243+0000 7f03e3e00640 1 --2- 172.21.15.43:0/373677623 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5632c0878ef0 0x5632c087b2e0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90c28d7f66a78563 server_cookie=edd6fb79c2d8271 in_seq=0 out_seq=0 2024-03-28T22:41:00.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.243+0000 7f03ca000640 10 client.27109 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:00.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.244+0000 7f03ca000640 1 -- 172.21.15.43:0/373677623 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5632c0769590 con 0x5632c0878ef0 2024-03-28T22:41:00.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.244+0000 7f03ca000640 10 client.27109 handle_command_reply: tid=0 2024-03-28T22:41:00.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.245+0000 7f03edfdd740 1 client.27109 shutdown 2024-03-28T22:41:00.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.245+0000 7f03edfdd740 1 -- 172.21.15.43:0/373677623 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5632c07f8380 msgr2=0x5632c08056c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:00.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.245+0000 7f03edfdd740 1 --2- 172.21.15.43:0/373677623 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5632c07f8380 0x5632c08056c0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f03d802c180 tx=0x7f03d80f3990 comp rx=0 tx=0).stop 2024-03-28T22:41:00.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.245+0000 7f03edfdd740 1 -- 172.21.15.43:0/373677623 shutdown_connections 2024-03-28T22:41:00.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.245+0000 7f03edfdd740 1 --2- 172.21.15.43:0/373677623 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5632c0878ef0 0x5632c087b2e0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:00.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.245+0000 7f03edfdd740 1 --2- 172.21.15.43:0/373677623 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5632c07f8ca0 0x5632c0805c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:00.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.245+0000 7f03edfdd740 1 --2- 172.21.15.43:0/373677623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5632c0801880 0x5632c0877100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:00.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.245+0000 7f03ca000640 0 client.27109 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:00.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.245+0000 7f03edfdd740 1 --2- 172.21.15.43:0/373677623 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5632c07f8380 0x5632c08056c0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:00.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.245+0000 7f03edfdd740 1 -- 172.21.15.43:0/373677623 >> 172.21.15.43:0/373677623 conn(0x5632c07f2ab0 msgr2=0x5632c07ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:00.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.245+0000 7f03edfdd740 1 -- 172.21.15.43:0/373677623 shutdown_connections 2024-03-28T22:41:00.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.245+0000 7f03edfdd740 1 -- 172.21.15.43:0/373677623 wait complete. 2024-03-28T22:41:00.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.245+0000 7f03edfdd740 20 client.27109 trim_cache size 0 max 16384 2024-03-28T22:41:00.247 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:41:00.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.246+0000 7f03eb400640 1 -- 172.21.15.43:0/2950352730 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f03cc0a1ef0 msgr2=0x7f03cc0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:00.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.246+0000 7f03eb400640 1 --2- 172.21.15.43:0/2950352730 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f03cc0a1ef0 0x7f03cc0a43b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f03dc0049d0 tx=0x7f03dc004a40 comp rx=0 tx=0).stop 2024-03-28T22:41:00.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.246+0000 7f03eb400640 1 -- 172.21.15.43:0/2950352730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03e41016e0 msgr2=0x7f03e4073a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:00.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.246+0000 7f03eb400640 1 --2- 172.21.15.43:0/2950352730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03e41016e0 0x7f03e4073a50 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f03d80042f0 tx=0x7f03d8004320 comp rx=0 tx=0).stop 2024-03-28T22:41:00.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.246+0000 7f03eb400640 1 -- 172.21.15.43:0/2950352730 shutdown_connections 2024-03-28T22:41:00.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.246+0000 7f03eb400640 1 --2- 172.21.15.43:0/2950352730 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f03cc0a1ef0 0x7f03cc0a43b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:00.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.246+0000 7f03eb400640 1 --2- 172.21.15.43:0/2950352730 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f03e41076f0 0x7f03e40744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:00.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.246+0000 7f03eb400640 1 --2- 172.21.15.43:0/2950352730 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f03e41020a0 0x7f03e4073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:00.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.246+0000 7f03eb400640 1 --2- 172.21.15.43:0/2950352730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f03e41016e0 0x7f03e4073a50 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:00.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.246+0000 7f03eb400640 1 -- 172.21.15.43:0/2950352730 >> 172.21.15.43:0/2950352730 conn(0x7f03e4078b50 msgr2=0x7f03e4104100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:00.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.246+0000 7f03eb400640 1 -- 172.21.15.43:0/2950352730 shutdown_connections 2024-03-28T22:41:00.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:00.246+0000 7f03eb400640 1 -- 172.21.15.43:0/2950352730 wait complete. 2024-03-28T22:41:00.356 INFO:tasks.workunit.client.0.smithi043.stdout: 1 233815 8.39 MB/sec execute 31 sec latency 10.812 ms 2024-03-28T22:41:01.044 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.045+0000 7f3475200640 1 -- 172.21.15.43:0/1883547315 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34701064d0 msgr2=0x7f34701068b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:01.044 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.045+0000 7f3475200640 1 --2- 172.21.15.43:0/1883547315 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34701064d0 0x7f34701068b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3460009930 tx=0x7f346002bc30 comp rx=0 tx=0).stop 2024-03-28T22:41:01.045 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.046+0000 7f3475200640 1 -- 172.21.15.43:0/1883547315 shutdown_connections 2024-03-28T22:41:01.045 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.046+0000 7f3475200640 1 --2- 172.21.15.43:0/1883547315 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3470100df0 0x7f347010d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.045 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.046+0000 7f3475200640 1 --2- 172.21.15.43:0/1883547315 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3470100430 0x7f34701008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.045 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.046+0000 7f3475200640 1 --2- 172.21.15.43:0/1883547315 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f34701064d0 0x7f34701068b0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.045 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.046+0000 7f3475200640 1 -- 172.21.15.43:0/1883547315 >> 172.21.15.43:0/1883547315 conn(0x7f34700fbf60 msgr2=0x7f34700fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:01.045 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.046+0000 7f3475200640 1 -- 172.21.15.43:0/1883547315 shutdown_connections 2024-03-28T22:41:01.045 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.046+0000 7f3475200640 1 -- 172.21.15.43:0/1883547315 wait complete. 2024-03-28T22:41:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.047+0000 7f3475200640 1 Processor -- start 2024-03-28T22:41:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.047+0000 7f3475200640 1 -- start start 2024-03-28T22:41:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.047+0000 7f3475200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3470100430 0x7f347010dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.047+0000 7f3475200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3470100df0 0x7f347010e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.047+0000 7f3475200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34701064d0 0x7f347010e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.047+0000 7f3475200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3470111910 con 0x7f34701064d0 2024-03-28T22:41:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.047+0000 7f3475200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f3470111790 con 0x7f3470100df0 2024-03-28T22:41:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.047+0000 7f3475200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f3470111a90 con 0x7f3470100430 2024-03-28T22:41:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.048+0000 7f346f400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3470100430 0x7f347010dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.048+0000 7f346ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3470100df0 0x7f347010e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.048+0000 7f346f400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3470100430 0x7f347010dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:59078/0 (socket says 172.21.15.43:59078) 2024-03-28T22:41:01.048 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.048+0000 7f346f400640 1 -- 172.21.15.43:0/1092008752 learned_addr learned my addr 172.21.15.43:0/1092008752 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:01.048 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.048+0000 7f346ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3470100df0 0x7f347010e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:60284/0 (socket says 172.21.15.43:60284) 2024-03-28T22:41:01.048 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.048+0000 7f346ea00640 1 -- 172.21.15.43:0/1092008752 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3470100430 msgr2=0x7f347010dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:01.048 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.048+0000 7f346ea00640 1 --2- 172.21.15.43:0/1092008752 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3470100430 0x7f347010dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.048 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.048+0000 7f346ea00640 1 -- 172.21.15.43:0/1092008752 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34701064d0 msgr2=0x7f347010e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:01.048 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.048+0000 7f346fe00640 1 --2- 172.21.15.43:0/1092008752 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34701064d0 0x7f347010e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.048 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.048+0000 7f346ea00640 1 --2- 172.21.15.43:0/1092008752 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34701064d0 0x7f347010e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.048 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.048+0000 7f346ea00640 1 -- 172.21.15.43:0/1092008752 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34701a63a0 con 0x7f3470100df0 2024-03-28T22:41:01.048 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.048+0000 7f346fe00640 1 --2- 172.21.15.43:0/1092008752 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34701064d0 0x7f347010e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:41:01.050 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.049+0000 7f346ea00640 1 --2- 172.21.15.43:0/1092008752 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3470100df0 0x7f347010e2d0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f345c00e9f0 tx=0x7f345c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:01.050 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.049+0000 7f3467e00640 1 -- 172.21.15.43:0/1092008752 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f345c00ccc0 con 0x7f3470100df0 2024-03-28T22:41:01.050 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.049+0000 7f3467e00640 1 -- 172.21.15.43:0/1092008752 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f345c010910 con 0x7f3470100df0 2024-03-28T22:41:01.050 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.049+0000 7f3475200640 1 -- 172.21.15.43:0/1092008752 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34701a6690 con 0x7f3470100df0 2024-03-28T22:41:01.050 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.049+0000 7f3467e00640 1 -- 172.21.15.43:0/1092008752 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f345c018960 con 0x7f3470100df0 2024-03-28T22:41:01.050 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.049+0000 7f3475200640 1 -- 172.21.15.43:0/1092008752 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34701a6c30 con 0x7f3470100df0 2024-03-28T22:41:01.051 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.053+0000 7f3479608740 1 Processor -- start 2024-03-28T22:41:01.051 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.053+0000 7f3479608740 1 -- start start 2024-03-28T22:41:01.052 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.053+0000 7f3479608740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563b3c5b1570 0x563b3c5b1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.053 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.053+0000 7f3479608740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563b3c5b1e90 0x563b3c5ba720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.053 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.053+0000 7f3479608740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563b3c5bac60 0x563b3c5bd050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.053 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.053+0000 7f3479608740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x563b3c514640 con 0x563b3c5b1570 2024-03-28T22:41:01.053 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.053+0000 7f3479608740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x563b3c4bd810 con 0x563b3c5b1e90 2024-03-28T22:41:01.053 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.053+0000 7f3479608740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x563b3c5106e0 con 0x563b3c5bac60 2024-03-28T22:41:01.053 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.054+0000 7f346fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563b3c5b1570 0x563b3c5b1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.053 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.054+0000 7f346f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563b3c5b1e90 0x563b3c5ba720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.053 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.054+0000 7f346fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563b3c5b1570 0x563b3c5b1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42280/0 (socket says 172.21.15.43:42280) 2024-03-28T22:41:01.053 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.054+0000 7f346fe00640 1 -- 172.21.15.43:0/838322918 learned_addr learned my addr 172.21.15.43:0/838322918 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:01.053 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.054+0000 7f3467e00640 1 -- 172.21.15.43:0/1092008752 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f345c010ac0 con 0x7f3470100df0 2024-03-28T22:41:01.053 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.054+0000 7f346f400640 1 -- 172.21.15.43:0/838322918 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563b3c5bac60 msgr2=0x563b3c5bd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:01.053 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.054+0000 7f346f400640 1 --2- 172.21.15.43:0/838322918 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563b3c5bac60 0x563b3c5bd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.053 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.054+0000 7f346f400640 1 -- 172.21.15.43:0/838322918 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563b3c5b1570 msgr2=0x563b3c5b1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:01.054 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.054+0000 7f346f400640 1 --2- 172.21.15.43:0/838322918 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563b3c5b1570 0x563b3c5b1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.054 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.054+0000 7f346f400640 1 -- 172.21.15.43:0/838322918 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b3c229210 con 0x563b3c5b1e90 2024-03-28T22:41:01.054 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.054+0000 7f346fe00640 1 --2- 172.21.15.43:0/838322918 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563b3c5b1570 0x563b3c5b1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:41:01.055 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.055+0000 7f346f400640 1 --2- 172.21.15.43:0/838322918 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563b3c5b1e90 0x563b3c5ba720 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f3460009900 tx=0x7f3460005df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d64c30b6ef1a067 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:01.055 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.055+0000 7f3464c00640 1 -- 172.21.15.43:0/838322918 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3460042070 con 0x563b3c5b1e90 2024-03-28T22:41:01.055 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.055+0000 7f3464c00640 1 -- 172.21.15.43:0/838322918 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f346002c690 con 0x563b3c5b1e90 2024-03-28T22:41:01.055 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.055+0000 7f3464c00640 1 -- 172.21.15.43:0/838322918 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3460004b80 con 0x563b3c5b1e90 2024-03-28T22:41:01.055 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.056+0000 7f3467e00640 1 --2- 172.21.15.43:0/1092008752 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f34540a1f60 0x7f34540a4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.055 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.056+0000 7f3479608740 1 -- 172.21.15.43:0/838322918 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563b3c5b1e90 msgr2=0x563b3c5ba720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:01.056 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.056+0000 7f3479608740 1 --2- 172.21.15.43:0/838322918 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563b3c5b1e90 0x563b3c5ba720 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f3460009900 tx=0x7f3460005df0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.056 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.056+0000 7f3467e00640 1 -- 172.21.15.43:0/1092008752 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f345c0ce990 con 0x7f3470100df0 2024-03-28T22:41:01.056 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.056+0000 7f346fe00640 1 --2- 172.21.15.43:0/1092008752 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f34540a1f60 0x7f34540a4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.056 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.056+0000 7f3479608740 1 -- 172.21.15.43:0/838322918 shutdown_connections 2024-03-28T22:41:01.056 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.056+0000 7f3479608740 1 --2- 172.21.15.43:0/838322918 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563b3c5b1e90 0x563b3c5ba720 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.056 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.056+0000 7f3479608740 1 --2- 172.21.15.43:0/838322918 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563b3c5bac60 0x563b3c5bd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.056 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.056+0000 7f3479608740 1 --2- 172.21.15.43:0/838322918 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563b3c5b1570 0x563b3c5b1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.056 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.056+0000 7f3479608740 1 -- 172.21.15.43:0/838322918 >> 172.21.15.43:0/838322918 conn(0x563b3c5abe90 msgr2=0x563b3c5ac270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:01.056 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.057+0000 7f3479608740 1 -- 172.21.15.43:0/838322918 shutdown_connections 2024-03-28T22:41:01.056 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.057+0000 7f3479608740 1 -- 172.21.15.43:0/838322918 wait complete. 2024-03-28T22:41:01.058 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.057+0000 7f346fe00640 1 --2- 172.21.15.43:0/1092008752 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f34540a1f60 0x7f34540a4420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3468009d20 tx=0x7f3468009500 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:01.058 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.057+0000 7f3479608740 1 -- start start 2024-03-28T22:41:01.058 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.057+0000 7f3479608740 1 Processor -- start 2024-03-28T22:41:01.058 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.058+0000 7f3479608740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563b3c5b1570 0x563b3c5b7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.058+0000 7f3479608740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563b3c5b1e90 0x563b3c5b8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.058+0000 7f3479608740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563b3c5bac60 0x563b3c5b8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.058+0000 7f3479608740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x563b3c5bd830 con 0x563b3c5bac60 2024-03-28T22:41:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.058+0000 7f3479608740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x563b3c4b2d00 con 0x563b3c5b1570 2024-03-28T22:41:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.058+0000 7f3479608740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x563b3c5bd9b0 con 0x563b3c5b1e90 2024-03-28T22:41:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.058+0000 7f3479608740 10 client.0 _ll_register_callbacks cb 0x563b3c49d490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:41:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.058+0000 7f346f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563b3c5b1570 0x563b3c5b7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.058+0000 7f346f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563b3c5b1570 0x563b3c5b7ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:60312/0 (socket says 172.21.15.43:60312) 2024-03-28T22:41:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.058+0000 7f346ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563b3c5b1e90 0x563b3c5b8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.058+0000 7f346f400640 1 -- 172.21.15.43:0/1338799616 learned_addr learned my addr 172.21.15.43:0/1338799616 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.058+0000 7f346fe00640 1 --2- 172.21.15.43:0/1338799616 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563b3c5bac60 0x563b3c5b8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.059+0000 7f346f400640 1 -- 172.21.15.43:0/1338799616 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563b3c5b1e90 msgr2=0x563b3c5b8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.059+0000 7f346f400640 1 --2- 172.21.15.43:0/1338799616 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563b3c5b1e90 0x563b3c5b8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.059+0000 7f346f400640 1 -- 172.21.15.43:0/1338799616 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563b3c5bac60 msgr2=0x563b3c5b8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:01.060 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.059+0000 7f346f400640 1 --2- 172.21.15.43:0/1338799616 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563b3c5bac60 0x563b3c5b8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.060 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.059+0000 7f346f400640 1 -- 172.21.15.43:0/1338799616 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b3c2f5620 con 0x563b3c5b1570 2024-03-28T22:41:01.060 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.059+0000 7f346f400640 1 --2- 172.21.15.43:0/1338799616 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563b3c5b1570 0x563b3c5b7ce0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f346002cb30 tx=0x7f346002cb60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da22c6c259c66bf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:01.060 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.059+0000 7f3459600640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:41:01.060 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.059+0000 7f3459600640 1 -- 172.21.15.43:0/1338799616 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3460052020 con 0x563b3c5b1570 2024-03-28T22:41:01.060 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.060+0000 7f3459600640 1 -- 172.21.15.43:0/1338799616 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f3460004440 con 0x563b3c5b1570 2024-03-28T22:41:01.060 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.060+0000 7f3479608740 1 -- 172.21.15.43:0/1338799616 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563b3c528030 con 0x563b3c5b1570 2024-03-28T22:41:01.060 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.060+0000 7f3459600640 1 -- 172.21.15.43:0/1338799616 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f346005d050 con 0x563b3c5b1570 2024-03-28T22:41:01.060 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.060+0000 7f3479608740 10 client.27161 fetch_fsmap learned FSMap version 32 2024-03-28T22:41:01.060 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.060+0000 7f3479608740 1 -- 172.21.15.43:0/1338799616 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563b3c5c9750 con 0x563b3c5b1570 2024-03-28T22:41:01.063 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.060+0000 7f3459600640 1 -- 172.21.15.43:0/1338799616 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x563b3c528030 con 0x563b3c5b1570 2024-03-28T22:41:01.063 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.061+0000 7f3459600640 1 -- 172.21.15.43:0/1338799616 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f3460003af0 con 0x563b3c5b1570 2024-03-28T22:41:01.063 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.061+0000 7f3479608740 10 client.27161 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:41:01.063 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.061+0000 7f3479608740 10 client.27161 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:41:01.063 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.061+0000 7f3479608740 20 client.27161 populate_metadata read hostname 'smithi043' 2024-03-28T22:41:01.063 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.061+0000 7f3479608740 1 --2- 172.21.15.43:0/1338799616 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x563b3c5caaa0 0x563b3c5cce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.063 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.061+0000 7f3479608740 4 client.27161 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:41:01.063 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.061+0000 7f3479608740 1 -- 172.21.15.43:0/1338799616 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563b3c521b80 con 0x563b3c5caaa0 2024-03-28T22:41:01.063 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.061+0000 7f346ea00640 1 --2- 172.21.15.43:0/1338799616 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x563b3c5caaa0 0x563b3c5cce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.062+0000 7f346ea00640 1 --2- 172.21.15.43:0/1338799616 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x563b3c5caaa0 0x563b3c5cce90 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ce66b7243991315 server_cookie=4886bb67e8d9b50e in_seq=0 out_seq=0 2024-03-28T22:41:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.062+0000 7f3459600640 10 client.27161 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.063+0000 7f3459600640 1 -- 172.21.15.43:0/1338799616 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563b3c521b80 con 0x563b3c5caaa0 2024-03-28T22:41:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.063+0000 7f3459600640 10 client.27161 handle_command_reply: tid=0 2024-03-28T22:41:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.063+0000 7f3479608740 1 client.27161 shutdown 2024-03-28T22:41:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.063+0000 7f3479608740 1 -- 172.21.15.43:0/1338799616 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563b3c5b1570 msgr2=0x563b3c5b7ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.063+0000 7f3479608740 1 --2- 172.21.15.43:0/1338799616 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563b3c5b1570 0x563b3c5b7ce0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f346002cb30 tx=0x7f346002cb60 comp rx=0 tx=0).stop 2024-03-28T22:41:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.063+0000 7f3479608740 1 -- 172.21.15.43:0/1338799616 shutdown_connections 2024-03-28T22:41:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.063+0000 7f3479608740 1 --2- 172.21.15.43:0/1338799616 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563b3c5bac60 0x563b3c5b8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.063+0000 7f3479608740 1 --2- 172.21.15.43:0/1338799616 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563b3c5b1e90 0x563b3c5b8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.063+0000 7f3479608740 1 --2- 172.21.15.43:0/1338799616 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x563b3c5caaa0 0x563b3c5cce90 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.063+0000 7f3479608740 1 --2- 172.21.15.43:0/1338799616 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563b3c5b1570 0x563b3c5b7ce0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.063+0000 7f3479608740 1 -- 172.21.15.43:0/1338799616 >> 172.21.15.43:0/1338799616 conn(0x563b3c5abe90 msgr2=0x563b3c5adbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.063+0000 7f3459600640 0 client.27161 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:01.065 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.063+0000 7f3479608740 1 -- 172.21.15.43:0/1338799616 shutdown_connections 2024-03-28T22:41:01.065 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.063+0000 7f3479608740 1 -- 172.21.15.43:0/1338799616 wait complete. 2024-03-28T22:41:01.065 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.064+0000 7f3479608740 20 client.27161 trim_cache size 0 max 16384 2024-03-28T22:41:01.080 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.081+0000 7f3479608740 1 Processor -- start 2024-03-28T22:41:01.080 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.081+0000 7f3479608740 1 -- start start 2024-03-28T22:41:01.080 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.081+0000 7f3479608740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563b3c5b1570 0x563b3c5ba410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.080 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.081+0000 7f3479608740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563b3c5b1e90 0x563b3c5c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.080 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.081+0000 7f3479608740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563b3c5bac60 0x563b3c5d2540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.080 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.081+0000 7f3479608740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f34701ac750 con 0x563b3c5b1570 2024-03-28T22:41:01.081 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.081+0000 7f3479608740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f34701ae290 con 0x563b3c5bac60 2024-03-28T22:41:01.081 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.081+0000 7f3479608740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f34701ace20 con 0x563b3c5b1e90 2024-03-28T22:41:01.081 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.082+0000 7f346ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563b3c5b1e90 0x563b3c5c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.081 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.082+0000 7f346ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563b3c5b1e90 0x563b3c5c65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:59102/0 (socket says 172.21.15.43:59102) 2024-03-28T22:41:01.081 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.082+0000 7f346ea00640 1 -- 172.21.15.43:0/1864417239 learned_addr learned my addr 172.21.15.43:0/1864417239 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:01.081 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.082+0000 7f346fe00640 1 --2- 172.21.15.43:0/1864417239 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563b3c5bac60 0x563b3c5d2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.081 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.082+0000 7f346f400640 1 --2- 172.21.15.43:0/1864417239 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563b3c5b1570 0x563b3c5ba410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.082 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.082+0000 7f346ea00640 1 -- 172.21.15.43:0/1864417239 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563b3c5bac60 msgr2=0x563b3c5d2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:01.082 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.082+0000 7f346ea00640 1 --2- 172.21.15.43:0/1864417239 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563b3c5bac60 0x563b3c5d2540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.082+0000 7f346ea00640 1 -- 172.21.15.43:0/1864417239 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563b3c5b1570 msgr2=0x563b3c5ba410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.082+0000 7f346ea00640 1 --2- 172.21.15.43:0/1864417239 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563b3c5b1570 0x563b3c5ba410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.082+0000 7f346ea00640 1 -- 172.21.15.43:0/1864417239 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b3c5b1cc0 con 0x563b3c5b1e90 2024-03-28T22:41:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.082+0000 7f346f400640 1 --2- 172.21.15.43:0/1864417239 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563b3c5b1570 0x563b3c5ba410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:41:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.082+0000 7f346ea00640 1 --2- 172.21.15.43:0/1864417239 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563b3c5b1e90 0x563b3c5c65d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f345c00cf10 tx=0x7f345c010eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9aa53ea7c2d67ebc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.083+0000 7f345be00640 1 -- 172.21.15.43:0/1864417239 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f345c099020 con 0x563b3c5b1e90 2024-03-28T22:41:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.083+0000 7f345be00640 1 -- 172.21.15.43:0/1864417239 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f345c099280 con 0x563b3c5b1e90 2024-03-28T22:41:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.083+0000 7f345be00640 1 -- 172.21.15.43:0/1864417239 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f345c09a630 con 0x563b3c5b1e90 2024-03-28T22:41:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.084+0000 7f3479608740 1 -- 172.21.15.43:0/1864417239 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563b3c5b1e90 msgr2=0x563b3c5c65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.084+0000 7f3479608740 1 --2- 172.21.15.43:0/1864417239 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563b3c5b1e90 0x563b3c5c65d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f345c00cf10 tx=0x7f345c010eb0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.084+0000 7f3479608740 1 -- 172.21.15.43:0/1864417239 shutdown_connections 2024-03-28T22:41:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.084+0000 7f3479608740 1 --2- 172.21.15.43:0/1864417239 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563b3c5bac60 0x563b3c5d2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.084+0000 7f3479608740 1 --2- 172.21.15.43:0/1864417239 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563b3c5b1e90 0x563b3c5c65d0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.084+0000 7f3479608740 1 --2- 172.21.15.43:0/1864417239 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563b3c5b1570 0x563b3c5ba410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.084+0000 7f3479608740 1 -- 172.21.15.43:0/1864417239 >> 172.21.15.43:0/1864417239 conn(0x563b3c5abe90 msgr2=0x563b3c5b6d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:01.084 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.084+0000 7f3479608740 1 -- 172.21.15.43:0/1864417239 shutdown_connections 2024-03-28T22:41:01.084 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.084+0000 7f3479608740 1 -- 172.21.15.43:0/1864417239 wait complete. 2024-03-28T22:41:01.087 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.084+0000 7f3479608740 1 -- start start 2024-03-28T22:41:01.087 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.085+0000 7f3479608740 1 Processor -- start 2024-03-28T22:41:01.087 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.085+0000 7f3479608740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563b3c5b1570 0x563b3c5b4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.087 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.085+0000 7f3479608740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563b3c5b1e90 0x563b3c5b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.087 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.085+0000 7f3479608740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563b3c5bac60 0x563b3c5b54d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.087 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.085+0000 7f3479608740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f34701adbc0 con 0x563b3c5bac60 2024-03-28T22:41:01.087 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.085+0000 7f3479608740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f34701ad4f0 con 0x563b3c5b1570 2024-03-28T22:41:01.087 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.085+0000 7f3479608740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f34701aa730 con 0x563b3c5b1e90 2024-03-28T22:41:01.087 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.085+0000 7f3479608740 10 client.0 _ll_register_callbacks cb 0x7f3468009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:41:01.087 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.085+0000 7f346f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563b3c5b1570 0x563b3c5b4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.088 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.085+0000 7f346ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563b3c5b1e90 0x563b3c5b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.088 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.085+0000 7f346ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563b3c5b1e90 0x563b3c5b4f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:59114/0 (socket says 172.21.15.43:59114) 2024-03-28T22:41:01.088 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.085+0000 7f346f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563b3c5b1570 0x563b3c5b4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:60332/0 (socket says 172.21.15.43:60332) 2024-03-28T22:41:01.088 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.085+0000 7f346ea00640 1 -- 172.21.15.43:0/2773114386 learned_addr learned my addr 172.21.15.43:0/2773114386 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:01.088 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.085+0000 7f346fe00640 1 --2- 172.21.15.43:0/2773114386 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563b3c5bac60 0x563b3c5b54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.088 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.086+0000 7f346ea00640 1 -- 172.21.15.43:0/2773114386 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563b3c5b1570 msgr2=0x563b3c5b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:01.088 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.086+0000 7f346ea00640 1 --2- 172.21.15.43:0/2773114386 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563b3c5b1570 0x563b3c5b4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.088 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.086+0000 7f346ea00640 1 -- 172.21.15.43:0/2773114386 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563b3c5bac60 msgr2=0x563b3c5b54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:01.088 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.086+0000 7f346ea00640 1 --2- 172.21.15.43:0/2773114386 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563b3c5bac60 0x563b3c5b54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.088 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.086+0000 7f346ea00640 1 -- 172.21.15.43:0/2773114386 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b3c62fd70 con 0x563b3c5b1e90 2024-03-28T22:41:01.088 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.086+0000 7f346f400640 1 --2- 172.21.15.43:0/2773114386 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563b3c5b1570 0x563b3c5b4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:41:01.088 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.086+0000 7f346ea00640 1 --2- 172.21.15.43:0/2773114386 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563b3c5b1e90 0x563b3c5b4f90 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f345c09af10 tx=0x7f345c0a3860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d48d7372d2faa2c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:01.088 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.086+0000 7f3459600640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:41:01.088 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.086+0000 7f3459600640 1 -- 172.21.15.43:0/2773114386 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f345c0cbf60 con 0x563b3c5b1e90 2024-03-28T22:41:01.088 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.086+0000 7f3459600640 1 -- 172.21.15.43:0/2773114386 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f345c09a3d0 con 0x563b3c5b1e90 2024-03-28T22:41:01.089 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.086+0000 7f3479608740 1 -- 172.21.15.43:0/2773114386 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f345c099280 con 0x563b3c5b1e90 2024-03-28T22:41:01.089 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.087+0000 7f3459600640 1 -- 172.21.15.43:0/2773114386 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f345c09af80 con 0x563b3c5b1e90 2024-03-28T22:41:01.089 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.087+0000 7f3479608740 10 client.27124 fetch_fsmap learned FSMap version 32 2024-03-28T22:41:01.089 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.087+0000 7f3479608740 1 -- 172.21.15.43:0/2773114386 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563b3c631040 con 0x563b3c5b1e90 2024-03-28T22:41:01.089 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.087+0000 7f3459600640 1 -- 172.21.15.43:0/2773114386 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f345c099280 con 0x563b3c5b1e90 2024-03-28T22:41:01.089 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.087+0000 7f3459600640 1 -- 172.21.15.43:0/2773114386 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f345c018c90 con 0x563b3c5b1e90 2024-03-28T22:41:01.089 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.087+0000 7f3479608740 10 client.27124 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:41:01.089 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.087+0000 7f3479608740 10 client.27124 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:41:01.089 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.087+0000 7f3479608740 20 client.27124 populate_metadata read hostname 'smithi043' 2024-03-28T22:41:01.089 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.087+0000 7f3479608740 1 --2- 172.21.15.43:0/2773114386 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x563b3c6316b0 0x563b3c633ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.089 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.087+0000 7f3479608740 4 client.27124 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T22:41:01.089 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.087+0000 7f3479608740 1 -- 172.21.15.43:0/2773114386 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563b3c51bed0 con 0x563b3c6316b0 2024-03-28T22:41:01.089 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.088+0000 7f346f400640 1 --2- 172.21.15.43:0/2773114386 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x563b3c6316b0 0x563b3c633ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.089 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.088+0000 7f3459600640 10 client.27124 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:01.089 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.088+0000 7f346f400640 1 --2- 172.21.15.43:0/2773114386 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x563b3c6316b0 0x563b3c633ad0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1096e3840242cbe server_cookie=afb333f47d65e397 in_seq=0 out_seq=0 2024-03-28T22:41:01.266 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:41:01.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:41:01.266 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:41:01.300 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.300+0000 7f3459600640 1 -- 172.21.15.43:0/2773114386 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563b3c51bed0 con 0x563b3c6316b0 2024-03-28T22:41:01.300 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.300+0000 7f3459600640 10 client.27124 handle_command_reply: tid=0 2024-03-28T22:41:01.300 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.301+0000 7f3479608740 1 client.27124 shutdown 2024-03-28T22:41:01.300 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.301+0000 7f3479608740 1 -- 172.21.15.43:0/2773114386 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563b3c5b1e90 msgr2=0x563b3c5b4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:01.300 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.301+0000 7f3479608740 1 --2- 172.21.15.43:0/2773114386 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563b3c5b1e90 0x563b3c5b4f90 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f345c09af10 tx=0x7f345c0a3860 comp rx=0 tx=0).stop 2024-03-28T22:41:01.300 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.301+0000 7f3479608740 1 -- 172.21.15.43:0/2773114386 shutdown_connections 2024-03-28T22:41:01.300 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.301+0000 7f3479608740 1 --2- 172.21.15.43:0/2773114386 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563b3c5bac60 0x563b3c5b54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.300 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.301+0000 7f3479608740 1 --2- 172.21.15.43:0/2773114386 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563b3c5b1e90 0x563b3c5b4f90 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.300 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.301+0000 7f3479608740 1 --2- 172.21.15.43:0/2773114386 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x563b3c6316b0 0x563b3c633ad0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.300 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.301+0000 7f3479608740 1 --2- 172.21.15.43:0/2773114386 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563b3c5b1570 0x563b3c5b4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.301 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.301+0000 7f3459600640 0 client.27124 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:01.301 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.301+0000 7f3479608740 1 -- 172.21.15.43:0/2773114386 >> 172.21.15.43:0/2773114386 conn(0x563b3c5abe90 msgr2=0x563b3c5b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:01.301 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.301+0000 7f3479608740 1 -- 172.21.15.43:0/2773114386 shutdown_connections 2024-03-28T22:41:01.301 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.301+0000 7f3479608740 1 -- 172.21.15.43:0/2773114386 wait complete. 2024-03-28T22:41:01.301 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.301+0000 7f3479608740 20 client.27124 trim_cache size 0 max 16384 2024-03-28T22:41:01.301 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:{ 2024-03-28T22:41:01.301 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "message": "", 2024-03-28T22:41:01.301 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "return_code": 0 2024-03-28T22:41:01.301 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:} 2024-03-28T22:41:01.302 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.302+0000 7f3475200640 1 -- 172.21.15.43:0/1092008752 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f34540a1f60 msgr2=0x7f34540a4420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:01.302 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.302+0000 7f3475200640 1 --2- 172.21.15.43:0/1092008752 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f34540a1f60 0x7f34540a4420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3468009d20 tx=0x7f3468009500 comp rx=0 tx=0).stop 2024-03-28T22:41:01.302 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.302+0000 7f3475200640 1 -- 172.21.15.43:0/1092008752 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3470100df0 msgr2=0x7f347010e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:01.303 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.302+0000 7f3475200640 1 --2- 172.21.15.43:0/1092008752 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3470100df0 0x7f347010e2d0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f345c00e9f0 tx=0x7f345c00eeb0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.303 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.303+0000 7f3475200640 1 -- 172.21.15.43:0/1092008752 shutdown_connections 2024-03-28T22:41:01.303 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.303+0000 7f3475200640 1 --2- 172.21.15.43:0/1092008752 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f34540a1f60 0x7f34540a4420 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.303 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.303+0000 7f3475200640 1 --2- 172.21.15.43:0/1092008752 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3470100df0 0x7f347010e2d0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.303 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.303+0000 7f3475200640 1 --2- 172.21.15.43:0/1092008752 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34701064d0 0x7f347010e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.303 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.303+0000 7f3475200640 1 --2- 172.21.15.43:0/1092008752 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3470100430 0x7f347010dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.303 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.303+0000 7f3475200640 1 -- 172.21.15.43:0/1092008752 >> 172.21.15.43:0/1092008752 conn(0x7f34700fbf60 msgr2=0x7f34700fc930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:01.303 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.303+0000 7f3475200640 1 -- 172.21.15.43:0/1092008752 shutdown_connections 2024-03-28T22:41:01.303 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:01.303+0000 7f3475200640 1 -- 172.21.15.43:0/1092008752 wait complete. 2024-03-28T22:41:01.356 INFO:tasks.workunit.client.0.smithi043.stdout: 1 234339 8.28 MB/sec execute 32 sec latency 224.241 ms 2024-03-28T22:41:01.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:01 smithi082 ceph-mon[36471]: pgmap v119: 129 pgs: 129 active+clean; 45 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 277 op/s 2024-03-28T22:41:01.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.483+0000 7ff6ce800640 1 -- 172.21.15.43:0/3763563729 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6c8109900 msgr2=0x7ff6c8109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:01.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.483+0000 7ff6ce800640 1 --2- 172.21.15.43:0/3763563729 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6c8109900 0x7ff6c8109ce0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8009900 tx=0x7ff6b802bc30 comp rx=0 tx=0).stop 2024-03-28T22:41:01.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.483+0000 7ff6ce800640 1 -- 172.21.15.43:0/3763563729 shutdown_connections 2024-03-28T22:41:01.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.483+0000 7ff6ce800640 1 --2- 172.21.15.43:0/3763563729 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6c81042b0 0x7ff6c81107c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.483+0000 7ff6ce800640 1 --2- 172.21.15.43:0/3763563729 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6c81038f0 0x7ff6c8103d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.483+0000 7ff6ce800640 1 --2- 172.21.15.43:0/3763563729 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6c8109900 0x7ff6c8109ce0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.484+0000 7ff6ce800640 1 -- 172.21.15.43:0/3763563729 >> 172.21.15.43:0/3763563729 conn(0x7ff6c80ff3a0 msgr2=0x7ff6c81017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:01.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.484+0000 7ff6ce800640 1 -- 172.21.15.43:0/3763563729 shutdown_connections 2024-03-28T22:41:01.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.484+0000 7ff6ce800640 1 -- 172.21.15.43:0/3763563729 wait complete. 2024-03-28T22:41:01.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.485+0000 7ff6ce800640 1 Processor -- start 2024-03-28T22:41:01.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.485+0000 7ff6ce800640 1 -- start start 2024-03-28T22:41:01.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.485+0000 7ff6ce800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6c81038f0 0x7ff6c819a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.485+0000 7ff6ce800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6c81042b0 0x7ff6c819a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.485+0000 7ff6ce800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6c8109900 0x7ff6c819ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.485+0000 7ff6ce800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c8113cd0 con 0x7ff6c81038f0 2024-03-28T22:41:01.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.485+0000 7ff6ce800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c8113b50 con 0x7ff6c81042b0 2024-03-28T22:41:01.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.485+0000 7ff6ce800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c8113e50 con 0x7ff6c8109900 2024-03-28T22:41:01.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.486+0000 7ff6c6a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6c81042b0 0x7ff6c819a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.486+0000 7ff6c6a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6c81042b0 0x7ff6c819a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:60348/0 (socket says 172.21.15.43:60348) 2024-03-28T22:41:01.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.486+0000 7ff6c6a00640 1 -- 172.21.15.43:0/2212261272 learned_addr learned my addr 172.21.15.43:0/2212261272 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:01.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.486+0000 7ff6c7400640 1 --2- 172.21.15.43:0/2212261272 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6c81038f0 0x7ff6c819a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.486+0000 7ff6c7e00640 1 --2- 172.21.15.43:0/2212261272 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6c8109900 0x7ff6c819ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.486+0000 7ff6c6a00640 1 -- 172.21.15.43:0/2212261272 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6c8109900 msgr2=0x7ff6c819ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:01.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.486+0000 7ff6c6a00640 1 --2- 172.21.15.43:0/2212261272 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6c8109900 0x7ff6c819ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.486+0000 7ff6c6a00640 1 -- 172.21.15.43:0/2212261272 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6c81038f0 msgr2=0x7ff6c819a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:01.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.486+0000 7ff6c6a00640 1 --2- 172.21.15.43:0/2212261272 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6c81038f0 0x7ff6c819a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.486+0000 7ff6c6a00640 1 -- 172.21.15.43:0/2212261272 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6c819f200 con 0x7ff6c81042b0 2024-03-28T22:41:01.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.486+0000 7ff6c6a00640 1 --2- 172.21.15.43:0/2212261272 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6c81042b0 0x7ff6c819a750 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7ff6b400d950 tx=0x7ff6b400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:01.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.487+0000 7ff6bfe00640 1 -- 172.21.15.43:0/2212261272 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b4017070 con 0x7ff6c81042b0 2024-03-28T22:41:01.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.487+0000 7ff6bfe00640 1 -- 172.21.15.43:0/2212261272 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6b4009c30 con 0x7ff6c81042b0 2024-03-28T22:41:01.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.487+0000 7ff6bfe00640 1 -- 172.21.15.43:0/2212261272 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b400f6b0 con 0x7ff6c81042b0 2024-03-28T22:41:01.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.487+0000 7ff6ce800640 1 -- 172.21.15.43:0/2212261272 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6c819f4f0 con 0x7ff6c81042b0 2024-03-28T22:41:01.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.487+0000 7ff6ce800640 1 -- 172.21.15.43:0/2212261272 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6c81a6d80 con 0x7ff6c81042b0 2024-03-28T22:41:01.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.490+0000 7ff6bfe00640 1 -- 172.21.15.43:0/2212261272 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7ff6b40109d0 con 0x7ff6c81042b0 2024-03-28T22:41:01.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.490+0000 7ff6d1595740 1 Processor -- start 2024-03-28T22:41:01.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.491+0000 7ff6d1595740 1 -- start start 2024-03-28T22:41:01.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.491+0000 7ff6d1595740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6959aef70 0x55c6959af350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.491+0000 7ff6d1595740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6959af890 0x55c6959b7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.491+0000 7ff6d1595740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6959b8470 0x55c6959ba860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.491+0000 7ff6d1595740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c695912c40 con 0x55c6959af890 2024-03-28T22:41:01.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.491+0000 7ff6d1595740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55c6958bbc40 con 0x55c6959aef70 2024-03-28T22:41:01.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.491+0000 7ff6d1595740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55c69590ece0 con 0x55c6959b8470 2024-03-28T22:41:01.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.491+0000 7ff6c7e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6959aef70 0x55c6959af350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.492+0000 7ff6c7e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6959aef70 0x55c6959af350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:60350/0 (socket says 172.21.15.43:60350) 2024-03-28T22:41:01.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.492+0000 7ff6c7e00640 1 -- 172.21.15.43:0/1409387575 learned_addr learned my addr 172.21.15.43:0/1409387575 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:01.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.492+0000 7ff6c7400640 1 --2- 172.21.15.43:0/1409387575 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6959af890 0x55c6959b7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.492+0000 7ff6c7e00640 1 -- 172.21.15.43:0/1409387575 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6959b8470 msgr2=0x55c6959ba860 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T22:41:01.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.492+0000 7ff6c7e00640 1 --2- 172.21.15.43:0/1409387575 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6959b8470 0x55c6959ba860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.492+0000 7ff6c7e00640 1 -- 172.21.15.43:0/1409387575 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6959af890 msgr2=0x55c6959b7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:01.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.492+0000 7ff6c7e00640 1 --2- 172.21.15.43:0/1409387575 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6959af890 0x55c6959b7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.492+0000 7ff6c7e00640 1 -- 172.21.15.43:0/1409387575 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6956ef7e0 con 0x55c6959aef70 2024-03-28T22:41:01.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.492+0000 7ff6c7e00640 1 --2- 172.21.15.43:0/1409387575 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6959aef70 0x55c6959af350 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7ff6c0009f90 tx=0x7ff6c0010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c886e6adff0d167a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:01.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.492+0000 7ff6bfe00640 1 --2- 172.21.15.43:0/2212261272 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ff6ac0a1f70 0x7ff6ac0a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.493+0000 7ff6bcc00640 1 -- 172.21.15.43:0/1409387575 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c0011700 con 0x55c6959aef70 2024-03-28T22:41:01.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.493+0000 7ff6bcc00640 1 -- 172.21.15.43:0/1409387575 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6c0011d80 con 0x55c6959aef70 2024-03-28T22:41:01.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.493+0000 7ff6bcc00640 1 -- 172.21.15.43:0/1409387575 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c0018380 con 0x55c6959aef70 2024-03-28T22:41:01.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.493+0000 7ff6bfe00640 1 -- 172.21.15.43:0/2212261272 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7ff6b40ce050 con 0x7ff6c81042b0 2024-03-28T22:41:01.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.493+0000 7ff6c7400640 1 --2- 172.21.15.43:0/2212261272 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ff6ac0a1f70 0x7ff6ac0a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.494+0000 7ff6d1595740 1 -- 172.21.15.43:0/1409387575 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6959aef70 msgr2=0x55c6959af350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:01.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.494+0000 7ff6d1595740 1 --2- 172.21.15.43:0/1409387575 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6959aef70 0x55c6959af350 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7ff6c0009f90 tx=0x7ff6c0010860 comp rx=0 tx=0).stop 2024-03-28T22:41:01.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.494+0000 7ff6d1595740 1 -- 172.21.15.43:0/1409387575 shutdown_connections 2024-03-28T22:41:01.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.494+0000 7ff6d1595740 1 --2- 172.21.15.43:0/1409387575 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6959b8470 0x55c6959ba860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.494+0000 7ff6d1595740 1 --2- 172.21.15.43:0/1409387575 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6959af890 0x55c6959b7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.494+0000 7ff6d1595740 1 --2- 172.21.15.43:0/1409387575 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6959aef70 0x55c6959af350 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.494+0000 7ff6d1595740 1 -- 172.21.15.43:0/1409387575 >> 172.21.15.43:0/1409387575 conn(0x55c6959a96a0 msgr2=0x55c6959a9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:01.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.494+0000 7ff6c7400640 1 --2- 172.21.15.43:0/2212261272 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ff6ac0a1f70 0x7ff6ac0a4430 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8009230 tx=0x7ff6b8005fd0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:01.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.494+0000 7ff6d1595740 1 -- 172.21.15.43:0/1409387575 shutdown_connections 2024-03-28T22:41:01.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.494+0000 7ff6d1595740 1 -- 172.21.15.43:0/1409387575 wait complete. 2024-03-28T22:41:01.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.495+0000 7ff6d1595740 1 -- start start 2024-03-28T22:41:01.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.495+0000 7ff6d1595740 1 Processor -- start 2024-03-28T22:41:01.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.495+0000 7ff6d1595740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6959aef70 0x55c6959b5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.495+0000 7ff6d1595740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6959af890 0x55c6959b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.495+0000 7ff6d1595740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6959b8470 0x55c6959b5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.495+0000 7ff6d1595740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c6959bb260 con 0x55c6959af890 2024-03-28T22:41:01.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.495+0000 7ff6d1595740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55c6958b1250 con 0x55c6959aef70 2024-03-28T22:41:01.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.495+0000 7ff6d1595740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55c6959bb3e0 con 0x55c6959b8470 2024-03-28T22:41:01.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.495+0000 7ff6c7400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6959aef70 0x55c6959b5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.496+0000 7ff6c6a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6959af890 0x55c6959b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.495+0000 7ff6d1595740 10 client.0 _ll_register_callbacks cb 0x55c6958d2e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:41:01.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.496+0000 7ff6c7400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6959aef70 0x55c6959b5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:60352/0 (socket says 172.21.15.43:60352) 2024-03-28T22:41:01.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.496+0000 7ff6c6a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6959af890 0x55c6959b5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42342/0 (socket says 172.21.15.43:42342) 2024-03-28T22:41:01.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.496+0000 7ff6c7400640 1 -- 172.21.15.43:0/369194583 learned_addr learned my addr 172.21.15.43:0/369194583 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:01.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.496+0000 7ff6c7e00640 1 --2- 172.21.15.43:0/369194583 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6959b8470 0x55c6959b5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.496+0000 7ff6c7400640 1 -- 172.21.15.43:0/369194583 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6959b8470 msgr2=0x55c6959b5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:01.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.496+0000 7ff6c7400640 1 --2- 172.21.15.43:0/369194583 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6959b8470 0x55c6959b5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.496+0000 7ff6c7400640 1 -- 172.21.15.43:0/369194583 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6959af890 msgr2=0x55c6959b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:01.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.496+0000 7ff6c7400640 1 --2- 172.21.15.43:0/369194583 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6959af890 0x55c6959b5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.496+0000 7ff6c7400640 1 -- 172.21.15.43:0/369194583 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c69574ff70 con 0x55c6959aef70 2024-03-28T22:41:01.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.496+0000 7ff6c7400640 1 --2- 172.21.15.43:0/369194583 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6959aef70 0x55c6959b5520 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8009900 tx=0x7ff6b802bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdeac5cb891760f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:01.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.496+0000 7ff6b1600640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:41:01.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.497+0000 7ff6b1600640 1 -- 172.21.15.43:0/369194583 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b80050c0 con 0x55c6959aef70 2024-03-28T22:41:01.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.497+0000 7ff6b1600640 1 -- 172.21.15.43:0/369194583 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7ff6b8005260 con 0x55c6959aef70 2024-03-28T22:41:01.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.497+0000 7ff6d1595740 1 -- 172.21.15.43:0/369194583 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6c0011d80 con 0x55c6959aef70 2024-03-28T22:41:01.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.497+0000 7ff6b1600640 1 -- 172.21.15.43:0/369194583 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff6b8004c50 con 0x55c6959aef70 2024-03-28T22:41:01.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.497+0000 7ff6d1595740 10 client.27179 fetch_fsmap learned FSMap version 32 2024-03-28T22:41:01.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.497+0000 7ff6d1595740 1 -- 172.21.15.43:0/369194583 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c6955e7170 con 0x55c6959aef70 2024-03-28T22:41:01.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.497+0000 7ff6b1600640 1 -- 172.21.15.43:0/369194583 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6c0011d80 con 0x55c6959aef70 2024-03-28T22:41:01.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.497+0000 7ff6b1600640 1 -- 172.21.15.43:0/369194583 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7ff6b8051070 con 0x55c6959aef70 2024-03-28T22:41:01.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.497+0000 7ff6d1595740 10 client.27179 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:41:01.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.497+0000 7ff6d1595740 10 client.27179 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:41:01.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.497+0000 7ff6d1595740 20 client.27179 populate_metadata read hostname 'smithi043' 2024-03-28T22:41:01.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.497+0000 7ff6d1595740 1 --2- 172.21.15.43:0/369194583 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c6959c8880 0x55c6959cace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.498+0000 7ff6d1595740 4 client.27179 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:41:01.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.498+0000 7ff6d1595740 1 -- 172.21.15.43:0/369194583 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c695920180 con 0x55c6959c8880 2024-03-28T22:41:01.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.498+0000 7ff6c6a00640 1 --2- 172.21.15.43:0/369194583 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c6959c8880 0x55c6959cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.498+0000 7ff6c6a00640 1 --2- 172.21.15.43:0/369194583 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c6959c8880 0x55c6959cace0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65f97a8aa3d3fc61 server_cookie=71380a13675ca084 in_seq=0 out_seq=0 2024-03-28T22:41:01.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.498+0000 7ff6b1600640 10 client.27179 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:01.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.499+0000 7ff6b1600640 1 -- 172.21.15.43:0/369194583 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c695920180 con 0x55c6959c8880 2024-03-28T22:41:01.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.499+0000 7ff6b1600640 10 client.27179 handle_command_reply: tid=0 2024-03-28T22:41:01.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.499+0000 7ff6d1595740 1 client.27179 shutdown 2024-03-28T22:41:01.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.499+0000 7ff6d1595740 1 -- 172.21.15.43:0/369194583 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6959aef70 msgr2=0x55c6959b5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:01.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.499+0000 7ff6d1595740 1 --2- 172.21.15.43:0/369194583 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6959aef70 0x55c6959b5520 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8009900 tx=0x7ff6b802bc30 comp rx=0 tx=0).stop 2024-03-28T22:41:01.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.499+0000 7ff6d1595740 1 -- 172.21.15.43:0/369194583 shutdown_connections 2024-03-28T22:41:01.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.499+0000 7ff6d1595740 1 --2- 172.21.15.43:0/369194583 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6959b8470 0x55c6959b5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.500+0000 7ff6d1595740 1 --2- 172.21.15.43:0/369194583 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6959af890 0x55c6959b5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.500+0000 7ff6d1595740 1 --2- 172.21.15.43:0/369194583 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c6959c8880 0x55c6959cace0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.500+0000 7ff6d1595740 1 --2- 172.21.15.43:0/369194583 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6959aef70 0x55c6959b5520 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.500+0000 7ff6d1595740 1 -- 172.21.15.43:0/369194583 >> 172.21.15.43:0/369194583 conn(0x55c6959a96a0 msgr2=0x55c6959ab3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:01.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.500+0000 7ff6b1600640 0 client.27179 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:01.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.500+0000 7ff6d1595740 1 -- 172.21.15.43:0/369194583 shutdown_connections 2024-03-28T22:41:01.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.500+0000 7ff6d1595740 1 -- 172.21.15.43:0/369194583 wait complete. 2024-03-28T22:41:01.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.500+0000 7ff6d1595740 20 client.27179 trim_cache size 0 max 16384 2024-03-28T22:41:01.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:01 smithi043 ceph-mon[31823]: pgmap v119: 129 pgs: 129 active+clean; 45 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 277 op/s 2024-03-28T22:41:01.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.520+0000 7ff6d1595740 1 Processor -- start 2024-03-28T22:41:01.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.520+0000 7ff6d1595740 1 -- start start 2024-03-28T22:41:01.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.520+0000 7ff6d1595740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6959aef70 0x55c6959b7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.520+0000 7ff6d1595740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6959af890 0x55c6959c4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.520+0000 7ff6d1595740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6959b8470 0x55c6959ab1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.520+0000 7ff6d1595740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c81b0d50 con 0x55c6959b8470 2024-03-28T22:41:01.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.520+0000 7ff6d1595740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c81b2890 con 0x55c6959aef70 2024-03-28T22:41:01.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.520+0000 7ff6d1595740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c81b1420 con 0x55c6959af890 2024-03-28T22:41:01.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.521+0000 7ff6c7400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6959aef70 0x55c6959b7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.521+0000 7ff6c7400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6959aef70 0x55c6959b7c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:60358/0 (socket says 172.21.15.43:60358) 2024-03-28T22:41:01.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.521+0000 7ff6c6a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6959af890 0x55c6959c4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.521+0000 7ff6c7400640 1 -- 172.21.15.43:0/706877096 learned_addr learned my addr 172.21.15.43:0/706877096 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:01.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.521+0000 7ff6c7e00640 1 --2- 172.21.15.43:0/706877096 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6959b8470 0x55c6959ab1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.521+0000 7ff6c7400640 1 -- 172.21.15.43:0/706877096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6959af890 msgr2=0x55c6959c4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:01.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.521+0000 7ff6c7400640 1 --2- 172.21.15.43:0/706877096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6959af890 0x55c6959c4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.521+0000 7ff6c7400640 1 -- 172.21.15.43:0/706877096 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6959b8470 msgr2=0x55c6959ab1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:01.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.521+0000 7ff6c7400640 1 --2- 172.21.15.43:0/706877096 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6959b8470 0x55c6959ab1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.521+0000 7ff6c7400640 1 -- 172.21.15.43:0/706877096 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6959b3ab0 con 0x55c6959aef70 2024-03-28T22:41:01.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.521+0000 7ff6c6a00640 1 --2- 172.21.15.43:0/706877096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6959af890 0x55c6959c4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:41:01.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.521+0000 7ff6c7e00640 1 --2- 172.21.15.43:0/706877096 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6959b8470 0x55c6959ab1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:41:01.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.522+0000 7ff6c7400640 1 --2- 172.21.15.43:0/706877096 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6959aef70 0x55c6959b7c50 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8000c00 tx=0x7ff6b802ce10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=185331631a524a38 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:01.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.522+0000 7ff6b3e00640 1 -- 172.21.15.43:0/706877096 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b80489e0 con 0x55c6959aef70 2024-03-28T22:41:01.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.522+0000 7ff6b3e00640 1 -- 172.21.15.43:0/706877096 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6b8048b80 con 0x55c6959aef70 2024-03-28T22:41:01.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.522+0000 7ff6b3e00640 1 -- 172.21.15.43:0/706877096 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b8053800 con 0x55c6959aef70 2024-03-28T22:41:01.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.523+0000 7ff6d1595740 1 -- 172.21.15.43:0/706877096 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6959aef70 msgr2=0x55c6959b7c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:01.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.523+0000 7ff6d1595740 1 --2- 172.21.15.43:0/706877096 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6959aef70 0x55c6959b7c50 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8000c00 tx=0x7ff6b802ce10 comp rx=0 tx=0).stop 2024-03-28T22:41:01.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.523+0000 7ff6d1595740 1 -- 172.21.15.43:0/706877096 shutdown_connections 2024-03-28T22:41:01.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.523+0000 7ff6d1595740 1 --2- 172.21.15.43:0/706877096 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6959b8470 0x55c6959ab1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.523+0000 7ff6d1595740 1 --2- 172.21.15.43:0/706877096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6959af890 0x55c6959c4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.523+0000 7ff6d1595740 1 --2- 172.21.15.43:0/706877096 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6959aef70 0x55c6959b7c50 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.523+0000 7ff6d1595740 1 -- 172.21.15.43:0/706877096 >> 172.21.15.43:0/706877096 conn(0x55c6959a96a0 msgr2=0x55c6959aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:01.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.523+0000 7ff6d1595740 1 -- 172.21.15.43:0/706877096 shutdown_connections 2024-03-28T22:41:01.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.523+0000 7ff6d1595740 1 -- 172.21.15.43:0/706877096 wait complete. 2024-03-28T22:41:01.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.523+0000 7ff6d1595740 1 -- start start 2024-03-28T22:41:01.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.523+0000 7ff6d1595740 1 Processor -- start 2024-03-28T22:41:01.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.524+0000 7ff6d1595740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6959aef70 0x55c6959c0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.524+0000 7ff6d1595740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6959af890 0x55c6959bd660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.524+0000 7ff6d1595740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6959b8470 0x55c6959bdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.524+0000 7ff6d1595740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c81b21c0 con 0x55c6959af890 2024-03-28T22:41:01.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.524+0000 7ff6d1595740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c81b1af0 con 0x55c6959b8470 2024-03-28T22:41:01.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.524+0000 7ff6d1595740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c81aed30 con 0x55c6959aef70 2024-03-28T22:41:01.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.524+0000 7ff6d1595740 10 client.0 _ll_register_callbacks cb 0x7ff6b8005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:41:01.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.524+0000 7ff6c7400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6959aef70 0x55c6959c0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.524+0000 7ff6c7400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6959aef70 0x55c6959c0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:59182/0 (socket says 172.21.15.43:59182) 2024-03-28T22:41:01.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.524+0000 7ff6c7400640 1 -- 172.21.15.43:0/2260227074 learned_addr learned my addr 172.21.15.43:0/2260227074 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:01.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.524+0000 7ff6c7e00640 1 --2- 172.21.15.43:0/2260227074 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6959b8470 0x55c6959bdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.524+0000 7ff6c6a00640 1 --2- 172.21.15.43:0/2260227074 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6959af890 0x55c6959bd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.525+0000 7ff6c7400640 1 -- 172.21.15.43:0/2260227074 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6959b8470 msgr2=0x55c6959bdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:01.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.525+0000 7ff6c7400640 1 --2- 172.21.15.43:0/2260227074 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6959b8470 0x55c6959bdba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.525+0000 7ff6c7400640 1 -- 172.21.15.43:0/2260227074 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6959af890 msgr2=0x55c6959bd660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:01.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.525+0000 7ff6c7400640 1 --2- 172.21.15.43:0/2260227074 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6959af890 0x55c6959bd660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.525+0000 7ff6c7400640 1 -- 172.21.15.43:0/2260227074 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6959be0e0 con 0x55c6959aef70 2024-03-28T22:41:01.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.525+0000 7ff6c7400640 1 --2- 172.21.15.43:0/2260227074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6959aef70 0x55c6959c0690 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff6b80484f0 tx=0x7ff6b8005990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28b5d102ce835f47 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:01.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.525+0000 7ff6b1600640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:41:01.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.525+0000 7ff6b1600640 1 -- 172.21.15.43:0/2260227074 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b804d040 con 0x55c6959aef70 2024-03-28T22:41:01.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.525+0000 7ff6d1595740 1 -- 172.21.15.43:0/2260227074 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6b8048b80 con 0x55c6959aef70 2024-03-28T22:41:01.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.525+0000 7ff6b1600640 1 -- 172.21.15.43:0/2260227074 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7ff6b8005380 con 0x55c6959aef70 2024-03-28T22:41:01.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.526+0000 7ff6b1600640 1 -- 172.21.15.43:0/2260227074 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff6b8053540 con 0x55c6959aef70 2024-03-28T22:41:01.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.526+0000 7ff6d1595740 10 client.27139 fetch_fsmap learned FSMap version 32 2024-03-28T22:41:01.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.526+0000 7ff6d1595740 1 -- 172.21.15.43:0/2260227074 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c695a2f760 con 0x55c6959aef70 2024-03-28T22:41:01.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.526+0000 7ff6b1600640 1 -- 172.21.15.43:0/2260227074 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6b8048b80 con 0x55c6959aef70 2024-03-28T22:41:01.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.526+0000 7ff6b1600640 1 -- 172.21.15.43:0/2260227074 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7ff6b805a020 con 0x55c6959aef70 2024-03-28T22:41:01.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.526+0000 7ff6d1595740 10 client.27139 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:41:01.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.526+0000 7ff6d1595740 10 client.27139 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:41:01.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.526+0000 7ff6d1595740 20 client.27139 populate_metadata read hostname 'smithi043' 2024-03-28T22:41:01.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.526+0000 7ff6d1595740 1 --2- 172.21.15.43:0/2260227074 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c695a2fbe0 0x55c695a31fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.526+0000 7ff6d1595740 4 client.27139 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:41:01.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.526+0000 7ff6d1595740 1 -- 172.21.15.43:0/2260227074 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c69591a4d0 con 0x55c695a2fbe0 2024-03-28T22:41:01.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.527+0000 7ff6c6a00640 1 --2- 172.21.15.43:0/2260227074 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c695a2fbe0 0x55c695a31fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.527+0000 7ff6c6a00640 1 --2- 172.21.15.43:0/2260227074 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c695a2fbe0 0x55c695a31fd0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6cc18d027b45468 server_cookie=aa8dd33360152386 in_seq=0 out_seq=0 2024-03-28T22:41:01.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.527+0000 7ff6b1600640 10 client.27139 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:01.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.529+0000 7ff6b1600640 1 -- 172.21.15.43:0/2260227074 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c69591a4d0 con 0x55c695a2fbe0 2024-03-28T22:41:01.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.529+0000 7ff6b1600640 10 client.27139 handle_command_reply: tid=0 2024-03-28T22:41:01.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.529+0000 7ff6d1595740 1 client.27139 shutdown 2024-03-28T22:41:01.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.529+0000 7ff6d1595740 1 -- 172.21.15.43:0/2260227074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6959aef70 msgr2=0x55c6959c0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:01.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.529+0000 7ff6d1595740 1 --2- 172.21.15.43:0/2260227074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6959aef70 0x55c6959c0690 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff6b80484f0 tx=0x7ff6b8005990 comp rx=0 tx=0).stop 2024-03-28T22:41:01.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.529+0000 7ff6d1595740 1 -- 172.21.15.43:0/2260227074 shutdown_connections 2024-03-28T22:41:01.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.529+0000 7ff6d1595740 1 --2- 172.21.15.43:0/2260227074 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c695a2fbe0 0x55c695a31fd0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.529+0000 7ff6d1595740 1 --2- 172.21.15.43:0/2260227074 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6959b8470 0x55c6959bdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.529+0000 7ff6d1595740 1 --2- 172.21.15.43:0/2260227074 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c6959af890 0x55c6959bd660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.529+0000 7ff6d1595740 1 --2- 172.21.15.43:0/2260227074 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c6959aef70 0x55c6959c0690 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.529+0000 7ff6b1600640 0 client.27139 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:01.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.529+0000 7ff6d1595740 1 -- 172.21.15.43:0/2260227074 >> 172.21.15.43:0/2260227074 conn(0x55c6959a96a0 msgr2=0x55c6959ab360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:01.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.529+0000 7ff6d1595740 1 -- 172.21.15.43:0/2260227074 shutdown_connections 2024-03-28T22:41:01.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.529+0000 7ff6d1595740 1 -- 172.21.15.43:0/2260227074 wait complete. 2024-03-28T22:41:01.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.529+0000 7ff6d1595740 20 client.27139 trim_cache size 0 max 16384 2024-03-28T22:41:01.530 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:41:01.530 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-03-28T22:41:01.530 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "6bae7f7e-b886-49c4-9dd1-60296c3f5f91", 2024-03-28T22:41:01.530 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-03-28T22:41:01.530 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:41:01.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.530+0000 7ff6ce800640 1 -- 172.21.15.43:0/2212261272 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ff6ac0a1f70 msgr2=0x7ff6ac0a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:01.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.530+0000 7ff6ce800640 1 --2- 172.21.15.43:0/2212261272 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ff6ac0a1f70 0x7ff6ac0a4430 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8009230 tx=0x7ff6b8005fd0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.530+0000 7ff6ce800640 1 -- 172.21.15.43:0/2212261272 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6c81042b0 msgr2=0x7ff6c819a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:01.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.530+0000 7ff6ce800640 1 --2- 172.21.15.43:0/2212261272 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6c81042b0 0x7ff6c819a750 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7ff6b400d950 tx=0x7ff6b400de10 comp rx=0 tx=0).stop 2024-03-28T22:41:01.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.530+0000 7ff6ce800640 1 -- 172.21.15.43:0/2212261272 shutdown_connections 2024-03-28T22:41:01.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.530+0000 7ff6ce800640 1 --2- 172.21.15.43:0/2212261272 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ff6ac0a1f70 0x7ff6ac0a4430 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.530+0000 7ff6ce800640 1 --2- 172.21.15.43:0/2212261272 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6c81042b0 0x7ff6c819a750 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.530+0000 7ff6ce800640 1 --2- 172.21.15.43:0/2212261272 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff6c8109900 0x7ff6c819ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.530+0000 7ff6ce800640 1 --2- 172.21.15.43:0/2212261272 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6c81038f0 0x7ff6c819a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.530+0000 7ff6ce800640 1 -- 172.21.15.43:0/2212261272 >> 172.21.15.43:0/2212261272 conn(0x7ff6c80ff3a0 msgr2=0x7ff6c80ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:01.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.531+0000 7ff6ce800640 1 -- 172.21.15.43:0/2212261272 shutdown_connections 2024-03-28T22:41:01.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.531+0000 7ff6ce800640 1 -- 172.21.15.43:0/2212261272 wait complete. 2024-03-28T22:41:01.551 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:41:01.551 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:41:01.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:01 smithi172 ceph-mon[36566]: pgmap v119: 129 pgs: 129 active+clean; 45 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 277 op/s 2024-03-28T22:41:01.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.834+0000 7f7862c00640 1 -- 172.21.15.43:0/2078077361 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f785c073b00 msgr2=0x7f785c073ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:01.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.834+0000 7f7862c00640 1 --2- 172.21.15.43:0/2078077361 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f785c073b00 0x7f785c073ee0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f784c009900 tx=0x7f784c02bcc0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.835+0000 7f7862c00640 1 -- 172.21.15.43:0/2078077361 shutdown_connections 2024-03-28T22:41:01.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.835+0000 7f7862c00640 1 --2- 172.21.15.43:0/2078077361 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f785c10dfb0 0x7f785c1103a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.835+0000 7f7862c00640 1 --2- 172.21.15.43:0/2078077361 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f785c074420 0x7f785c10da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.835+0000 7f7862c00640 1 --2- 172.21.15.43:0/2078077361 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f785c073b00 0x7f785c073ee0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.835+0000 7f7862c00640 1 -- 172.21.15.43:0/2078077361 >> 172.21.15.43:0/2078077361 conn(0x7f785c0fd180 msgr2=0x7f785c0ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:01.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.835+0000 7f7862c00640 1 -- 172.21.15.43:0/2078077361 shutdown_connections 2024-03-28T22:41:01.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.835+0000 7f7862c00640 1 -- 172.21.15.43:0/2078077361 wait complete. 2024-03-28T22:41:01.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.836+0000 7f7862c00640 1 Processor -- start 2024-03-28T22:41:01.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.836+0000 7f7862c00640 1 -- start start 2024-03-28T22:41:01.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.837+0000 7f7862c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f785c073b00 0x7f785c19a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.837+0000 7f7862c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f785c074420 0x7f785c19a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.837+0000 7f7862c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f785c10dfb0 0x7f785c19eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.837+0000 7f7862c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f785c113d20 con 0x7f785c074420 2024-03-28T22:41:01.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.837+0000 7f7862c00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f785c113ba0 con 0x7f785c10dfb0 2024-03-28T22:41:01.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.837+0000 7f7862c00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f785c113ea0 con 0x7f785c073b00 2024-03-28T22:41:01.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.837+0000 7f785be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f785c073b00 0x7f785c19a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.837+0000 7f785be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f785c073b00 0x7f785c19a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:59200/0 (socket says 172.21.15.43:59200) 2024-03-28T22:41:01.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.837+0000 7f785be00640 1 -- 172.21.15.43:0/2847852944 learned_addr learned my addr 172.21.15.43:0/2847852944 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:01.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.837+0000 7f7860c00640 1 --2- 172.21.15.43:0/2847852944 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f785c10dfb0 0x7f785c19eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.837+0000 7f785b400640 1 --2- 172.21.15.43:0/2847852944 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f785c074420 0x7f785c19a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.837+0000 7f7860c00640 1 -- 172.21.15.43:0/2847852944 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f785c073b00 msgr2=0x7f785c19a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:01.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.837+0000 7f7860c00640 1 --2- 172.21.15.43:0/2847852944 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f785c073b00 0x7f785c19a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.837+0000 7f7860c00640 1 -- 172.21.15.43:0/2847852944 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f785c074420 msgr2=0x7f785c19a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:01.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.838+0000 7f7860c00640 1 --2- 172.21.15.43:0/2847852944 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f785c074420 0x7f785c19a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.838+0000 7f7860c00640 1 -- 172.21.15.43:0/2847852944 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f785c19f1d0 con 0x7f785c10dfb0 2024-03-28T22:41:01.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.838+0000 7f785be00640 1 --2- 172.21.15.43:0/2847852944 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f785c073b00 0x7f785c19a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:41:01.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.838+0000 7f785b400640 1 --2- 172.21.15.43:0/2847852944 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f785c074420 0x7f785c19a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:41:01.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.838+0000 7f7860c00640 1 --2- 172.21.15.43:0/2847852944 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f785c10dfb0 0x7f785c19eaa0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f7854009fd0 tx=0x7f785400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:01.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.838+0000 7f7858c00640 1 -- 172.21.15.43:0/2847852944 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785400ce20 con 0x7f785c10dfb0 2024-03-28T22:41:01.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.838+0000 7f7858c00640 1 -- 172.21.15.43:0/2847852944 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7854005020 con 0x7f785c10dfb0 2024-03-28T22:41:01.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.838+0000 7f7862c00640 1 -- 172.21.15.43:0/2847852944 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f785c19f4c0 con 0x7f785c10dfb0 2024-03-28T22:41:01.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.838+0000 7f7858c00640 1 -- 172.21.15.43:0/2847852944 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7854010720 con 0x7f785c10dfb0 2024-03-28T22:41:01.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.838+0000 7f7862c00640 1 -- 172.21.15.43:0/2847852944 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f785c1a6d50 con 0x7f785c10dfb0 2024-03-28T22:41:01.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.842+0000 7f7858c00640 1 -- 172.21.15.43:0/2847852944 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f7854005540 con 0x7f785c10dfb0 2024-03-28T22:41:01.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.842+0000 7f786580b740 1 Processor -- start 2024-03-28T22:41:01.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.842+0000 7f786580b740 1 -- start start 2024-03-28T22:41:01.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.843+0000 7f786580b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559f018b4bc0 0x559f018b3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.843+0000 7f786580b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559f018b3ee0 0x559f018b4360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.843+0000 7f786580b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559f018bd250 0x559f018bf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.843+0000 7f786580b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x559f01818100 con 0x559f018b3ee0 2024-03-28T22:41:01.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.843+0000 7f786580b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x559f017c0ba0 con 0x559f018b4bc0 2024-03-28T22:41:01.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.843+0000 7f786580b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x559f018141a0 con 0x559f018bd250 2024-03-28T22:41:01.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.843+0000 7f785b400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559f018b4bc0 0x559f018b3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.843+0000 7f785b400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559f018b4bc0 0x559f018b3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:60400/0 (socket says 172.21.15.43:60400) 2024-03-28T22:41:01.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.843+0000 7f785b400640 1 -- 172.21.15.43:0/2644451823 learned_addr learned my addr 172.21.15.43:0/2644451823 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:01.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.843+0000 7f785be00640 1 --2- 172.21.15.43:0/2644451823 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559f018b3ee0 0x559f018b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.843+0000 7f785b400640 1 --2- 172.21.15.43:0/2644451823 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559f018bd250 0x559f018bf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.844+0000 7f785b400640 1 -- 172.21.15.43:0/2644451823 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559f018bd250 msgr2=0x559f018bf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:01.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.844+0000 7f785b400640 1 --2- 172.21.15.43:0/2644451823 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559f018bd250 0x559f018bf640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.844+0000 7f785b400640 1 -- 172.21.15.43:0/2644451823 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559f018b3ee0 msgr2=0x559f018b4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:01.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.844+0000 7f785b400640 1 --2- 172.21.15.43:0/2644451823 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559f018b3ee0 0x559f018b4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.844+0000 7f785b400640 1 -- 172.21.15.43:0/2644451823 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f017f11a0 con 0x559f018b4bc0 2024-03-28T22:41:01.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.844+0000 7f7858c00640 1 --2- 172.21.15.43:0/2847852944 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f78400a1ef0 0x7f78400a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.844+0000 7f785b400640 1 --2- 172.21.15.43:0/2644451823 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559f018b4bc0 0x559f018b3910 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f7848014a30 tx=0x7f784800c570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e206f40664e555b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:01.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.844+0000 7f7858c00640 1 -- 172.21.15.43:0/2847852944 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f78540cecd0 con 0x7f785c10dfb0 2024-03-28T22:41:01.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.844+0000 7f785be00640 1 --2- 172.21.15.43:0/2847852944 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f78400a1ef0 0x7f78400a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.844+0000 7f7851200640 1 -- 172.21.15.43:0/2644451823 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78480142d0 con 0x559f018b4bc0 2024-03-28T22:41:01.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.844+0000 7f7851200640 1 -- 172.21.15.43:0/2644451823 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7848005500 con 0x559f018b4bc0 2024-03-28T22:41:01.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.844+0000 7f7851200640 1 -- 172.21.15.43:0/2644451823 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78480047a0 con 0x559f018b4bc0 2024-03-28T22:41:01.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.845+0000 7f786580b740 1 -- 172.21.15.43:0/2644451823 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559f018b4bc0 msgr2=0x559f018b3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:01.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.845+0000 7f785be00640 1 --2- 172.21.15.43:0/2847852944 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f78400a1ef0 0x7f78400a43b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f784c0096c0 tx=0x7f784c02e040 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:01.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.845+0000 7f786580b740 1 --2- 172.21.15.43:0/2644451823 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559f018b4bc0 0x559f018b3910 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f7848014a30 tx=0x7f784800c570 comp rx=0 tx=0).stop 2024-03-28T22:41:01.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.845+0000 7f786580b740 1 -- 172.21.15.43:0/2644451823 shutdown_connections 2024-03-28T22:41:01.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.845+0000 7f786580b740 1 --2- 172.21.15.43:0/2644451823 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559f018bd250 0x559f018bf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.845+0000 7f786580b740 1 --2- 172.21.15.43:0/2644451823 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559f018b3ee0 0x559f018b4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.845+0000 7f786580b740 1 --2- 172.21.15.43:0/2644451823 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559f018b4bc0 0x559f018b3910 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.845+0000 7f786580b740 1 -- 172.21.15.43:0/2644451823 >> 172.21.15.43:0/2644451823 conn(0x559f0187aa10 msgr2=0x559f018ae930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:01.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.845+0000 7f786580b740 1 -- 172.21.15.43:0/2644451823 shutdown_connections 2024-03-28T22:41:01.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.846+0000 7f786580b740 1 -- 172.21.15.43:0/2644451823 wait complete. 2024-03-28T22:41:01.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.846+0000 7f786580b740 1 -- start start 2024-03-28T22:41:01.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.846+0000 7f786580b740 1 Processor -- start 2024-03-28T22:41:01.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.847+0000 7f786580b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559f018b3ee0 0x559f018ba2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.847+0000 7f786580b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559f018b4bc0 0x559f018ba820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.847+0000 7f786580b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559f018bd250 0x559f018bad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.847+0000 7f786580b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x559f018c0060 con 0x559f018b4bc0 2024-03-28T22:41:01.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.847+0000 7f786580b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x559f017b6250 con 0x559f018b3ee0 2024-03-28T22:41:01.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.847+0000 7f786580b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x559f018c01e0 con 0x559f018bd250 2024-03-28T22:41:01.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.847+0000 7f786580b740 10 client.0 _ll_register_callbacks cb 0x559f0152bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:41:01.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.847+0000 7f785b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559f018b4bc0 0x559f018ba820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.847+0000 7f7860c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559f018bd250 0x559f018bad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.847+0000 7f785be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559f018b3ee0 0x559f018ba2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.847+0000 7f785b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559f018b4bc0 0x559f018ba820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42390/0 (socket says 172.21.15.43:42390) 2024-03-28T22:41:01.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.847+0000 7f785b400640 1 -- 172.21.15.43:0/2654176701 learned_addr learned my addr 172.21.15.43:0/2654176701 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:01.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.848+0000 7f785b400640 1 -- 172.21.15.43:0/2654176701 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559f018bd250 msgr2=0x559f018bad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:01.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.848+0000 7f785b400640 1 --2- 172.21.15.43:0/2654176701 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559f018bd250 0x559f018bad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.848+0000 7f785b400640 1 -- 172.21.15.43:0/2654176701 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559f018b3ee0 msgr2=0x559f018ba2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:01.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.848+0000 7f785b400640 1 --2- 172.21.15.43:0/2654176701 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559f018b3ee0 0x559f018ba2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.848+0000 7f785b400640 1 -- 172.21.15.43:0/2654176701 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f017d2960 con 0x559f018b4bc0 2024-03-28T22:41:01.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.848+0000 7f785be00640 1 --2- 172.21.15.43:0/2654176701 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559f018b3ee0 0x559f018ba2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:41:01.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.848+0000 7f785b400640 1 --2- 172.21.15.43:0/2654176701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559f018b4bc0 0x559f018ba820 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f784800f9e0 tx=0x7f784800fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=863528f2ff548473 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:01.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.848+0000 7f7845600640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:41:01.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.848+0000 7f7845600640 1 -- 172.21.15.43:0/2654176701 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7848012650 con 0x559f018b4bc0 2024-03-28T22:41:01.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.848+0000 7f7845600640 1 -- 172.21.15.43:0/2654176701 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f78480127f0 con 0x559f018b4bc0 2024-03-28T22:41:01.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.848+0000 7f786580b740 1 -- 172.21.15.43:0/2654176701 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7848005500 con 0x559f018b4bc0 2024-03-28T22:41:01.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.849+0000 7f7845600640 1 -- 172.21.15.43:0/2654176701 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f78480129f0 con 0x559f018b4bc0 2024-03-28T22:41:01.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.849+0000 7f786580b740 10 client.16989 fetch_fsmap learned FSMap version 32 2024-03-28T22:41:01.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.849+0000 7f786580b740 1 -- 172.21.15.43:0/2654176701 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559f015f67d0 con 0x559f018b4bc0 2024-03-28T22:41:01.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.849+0000 7f7845600640 1 -- 172.21.15.43:0/2654176701 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7848005500 con 0x559f018b4bc0 2024-03-28T22:41:01.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.849+0000 7f7845600640 1 -- 172.21.15.43:0/2654176701 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f784801f8f0 con 0x559f018b4bc0 2024-03-28T22:41:01.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.849+0000 7f786580b740 10 client.16989 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:41:01.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.849+0000 7f786580b740 10 client.16989 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:41:01.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.849+0000 7f786580b740 20 client.16989 populate_metadata read hostname 'smithi043' 2024-03-28T22:41:01.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.849+0000 7f786580b740 1 --2- 172.21.15.43:0/2654176701 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559f018cd5b0 0x559f018cf9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.849+0000 7f786580b740 4 client.16989 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:41:01.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.849+0000 7f786580b740 1 -- 172.21.15.43:0/2654176701 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f01825640 con 0x559f018cd5b0 2024-03-28T22:41:01.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.850+0000 7f785be00640 1 --2- 172.21.15.43:0/2654176701 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559f018cd5b0 0x559f018cf9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.850+0000 7f785be00640 1 --2- 172.21.15.43:0/2654176701 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559f018cd5b0 0x559f018cf9a0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2a5626ea530ba0f server_cookie=69acef868a90c505 in_seq=0 out_seq=0 2024-03-28T22:41:01.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.850+0000 7f7845600640 10 client.16989 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:01.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.851+0000 7f7845600640 1 -- 172.21.15.43:0/2654176701 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559f01825640 con 0x559f018cd5b0 2024-03-28T22:41:01.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.851+0000 7f7845600640 10 client.16989 handle_command_reply: tid=0 2024-03-28T22:41:01.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.851+0000 7f786580b740 1 client.16989 shutdown 2024-03-28T22:41:01.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.851+0000 7f786580b740 1 -- 172.21.15.43:0/2654176701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559f018b4bc0 msgr2=0x559f018ba820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:01.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.851+0000 7f786580b740 1 --2- 172.21.15.43:0/2654176701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559f018b4bc0 0x559f018ba820 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f784800f9e0 tx=0x7f784800fbe0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.851+0000 7f786580b740 1 -- 172.21.15.43:0/2654176701 shutdown_connections 2024-03-28T22:41:01.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.851+0000 7f786580b740 1 --2- 172.21.15.43:0/2654176701 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559f018bd250 0x559f018bad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.851+0000 7f786580b740 1 --2- 172.21.15.43:0/2654176701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559f018b4bc0 0x559f018ba820 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.851+0000 7f786580b740 1 --2- 172.21.15.43:0/2654176701 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559f018cd5b0 0x559f018cf9a0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.851+0000 7f786580b740 1 --2- 172.21.15.43:0/2654176701 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559f018b3ee0 0x559f018ba2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.851+0000 7f7845600640 0 client.16989 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:01.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.851+0000 7f786580b740 1 -- 172.21.15.43:0/2654176701 >> 172.21.15.43:0/2654176701 conn(0x559f0187aa10 msgr2=0x559f018b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:01.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.852+0000 7f786580b740 1 -- 172.21.15.43:0/2654176701 shutdown_connections 2024-03-28T22:41:01.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.852+0000 7f786580b740 1 -- 172.21.15.43:0/2654176701 wait complete. 2024-03-28T22:41:01.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.852+0000 7f786580b740 20 client.16989 trim_cache size 0 max 16384 2024-03-28T22:41:01.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.875+0000 7f786580b740 1 Processor -- start 2024-03-28T22:41:01.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.875+0000 7f786580b740 1 -- start start 2024-03-28T22:41:01.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.875+0000 7f786580b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559f018b3ee0 0x559f018bca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.875+0000 7f786580b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559f018b4bc0 0x559f018b9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.875+0000 7f786580b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559f018bd250 0x559f018c8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.876+0000 7f786580b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f785c1b0cc0 con 0x559f018bd250 2024-03-28T22:41:01.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.876+0000 7f786580b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f785c1b2800 con 0x559f018b4bc0 2024-03-28T22:41:01.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.876+0000 7f786580b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f785c1b1390 con 0x559f018b3ee0 2024-03-28T22:41:01.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.876+0000 7f785b400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559f018b4bc0 0x559f018b9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.876+0000 7f785b400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559f018b4bc0 0x559f018b9440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:60420/0 (socket says 172.21.15.43:60420) 2024-03-28T22:41:01.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.876+0000 7f785b400640 1 -- 172.21.15.43:0/2168951673 learned_addr learned my addr 172.21.15.43:0/2168951673 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:01.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.876+0000 7f785be00640 1 --2- 172.21.15.43:0/2168951673 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559f018b3ee0 0x559f018bca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.876+0000 7f7860c00640 1 --2- 172.21.15.43:0/2168951673 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559f018bd250 0x559f018c8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.876+0000 7f785b400640 1 -- 172.21.15.43:0/2168951673 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559f018b3ee0 msgr2=0x559f018bca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:01.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.876+0000 7f785b400640 1 --2- 172.21.15.43:0/2168951673 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559f018b3ee0 0x559f018bca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.876+0000 7f785b400640 1 -- 172.21.15.43:0/2168951673 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559f018bd250 msgr2=0x559f018c8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:01.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.876+0000 7f785b400640 1 --2- 172.21.15.43:0/2168951673 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559f018bd250 0x559f018c8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.876+0000 7f785b400640 1 -- 172.21.15.43:0/2168951673 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f0152c3f0 con 0x559f018b4bc0 2024-03-28T22:41:01.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.876+0000 7f785b400640 1 --2- 172.21.15.43:0/2168951673 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559f018b4bc0 0x559f018b9440 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f7848000c00 tx=0x7f7848005870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55cbac2c05fba752 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:01.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.877+0000 7f7847e00640 1 -- 172.21.15.43:0/2168951673 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7848038070 con 0x559f018b4bc0 2024-03-28T22:41:01.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.877+0000 7f7847e00640 1 -- 172.21.15.43:0/2168951673 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7848023030 con 0x559f018b4bc0 2024-03-28T22:41:01.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.877+0000 7f7847e00640 1 -- 172.21.15.43:0/2168951673 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7848012df0 con 0x559f018b4bc0 2024-03-28T22:41:01.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.877+0000 7f786580b740 1 -- 172.21.15.43:0/2168951673 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559f018b4bc0 msgr2=0x559f018b9440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:01.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.878+0000 7f786580b740 1 --2- 172.21.15.43:0/2168951673 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559f018b4bc0 0x559f018b9440 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f7848000c00 tx=0x7f7848005870 comp rx=0 tx=0).stop 2024-03-28T22:41:01.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.878+0000 7f786580b740 1 -- 172.21.15.43:0/2168951673 shutdown_connections 2024-03-28T22:41:01.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.878+0000 7f786580b740 1 --2- 172.21.15.43:0/2168951673 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559f018b4bc0 0x559f018b9440 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.878+0000 7f786580b740 1 --2- 172.21.15.43:0/2168951673 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559f018bd250 0x559f018c8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.878+0000 7f786580b740 1 --2- 172.21.15.43:0/2168951673 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559f018b3ee0 0x559f018bca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.878+0000 7f786580b740 1 -- 172.21.15.43:0/2168951673 >> 172.21.15.43:0/2168951673 conn(0x559f0187aa10 msgr2=0x559f018c8470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:01.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.878+0000 7f786580b740 1 -- 172.21.15.43:0/2168951673 shutdown_connections 2024-03-28T22:41:01.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.878+0000 7f786580b740 1 -- 172.21.15.43:0/2168951673 wait complete. 2024-03-28T22:41:01.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.878+0000 7f786580b740 1 -- start start 2024-03-28T22:41:01.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.879+0000 7f786580b740 1 Processor -- start 2024-03-28T22:41:01.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.879+0000 7f786580b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559f018b3ee0 0x559f018d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.879+0000 7f786580b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559f018b4bc0 0x559f018b6690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.879+0000 7f786580b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559f018bd250 0x559f018b6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.879+0000 7f786580b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f785c1b2130 con 0x559f018b3ee0 2024-03-28T22:41:01.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.879+0000 7f786580b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f785c1b1a60 con 0x559f018b4bc0 2024-03-28T22:41:01.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.879+0000 7f786580b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f785c1aeca0 con 0x559f018bd250 2024-03-28T22:41:01.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.879+0000 7f786580b740 10 client.0 _ll_register_callbacks cb 0x7f784c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:41:01.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.879+0000 7f785be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559f018b3ee0 0x559f018d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.879+0000 7f7860c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559f018bd250 0x559f018b6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.879+0000 7f785b400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559f018b4bc0 0x559f018b6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.879+0000 7f7860c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559f018bd250 0x559f018b6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:59218/0 (socket says 172.21.15.43:59218) 2024-03-28T22:41:01.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.879+0000 7f7860c00640 1 -- 172.21.15.43:0/887735585 learned_addr learned my addr 172.21.15.43:0/887735585 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:01.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.880+0000 7f785b400640 1 -- 172.21.15.43:0/887735585 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559f018bd250 msgr2=0x559f018b6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:01.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.880+0000 7f785b400640 1 --2- 172.21.15.43:0/887735585 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559f018bd250 0x559f018b6bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.880+0000 7f785b400640 1 -- 172.21.15.43:0/887735585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559f018b3ee0 msgr2=0x559f018d29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:01.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.880+0000 7f785b400640 1 --2- 172.21.15.43:0/887735585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559f018b3ee0 0x559f018d29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.880+0000 7f785b400640 1 -- 172.21.15.43:0/887735585 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f014ec0f0 con 0x559f018b4bc0 2024-03-28T22:41:01.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.880+0000 7f7860c00640 1 --2- 172.21.15.43:0/887735585 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559f018bd250 0x559f018b6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:41:01.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.880+0000 7f785b400640 1 --2- 172.21.15.43:0/887735585 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559f018b4bc0 0x559f018b6690 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f784801fe50 tx=0x7f78480149b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b9ad5ca3628ca01 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:01.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.880+0000 7f7845600640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:41:01.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.880+0000 7f7845600640 1 -- 172.21.15.43:0/887735585 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7848038070 con 0x559f018b4bc0 2024-03-28T22:41:01.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.880+0000 7f786580b740 1 -- 172.21.15.43:0/887735585 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7848023030 con 0x559f018b4bc0 2024-03-28T22:41:01.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.880+0000 7f7845600640 1 -- 172.21.15.43:0/887735585 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f7848012450 con 0x559f018b4bc0 2024-03-28T22:41:01.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.881+0000 7f7845600640 1 -- 172.21.15.43:0/887735585 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f784802d030 con 0x559f018b4bc0 2024-03-28T22:41:01.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.881+0000 7f786580b740 10 client.27203 fetch_fsmap learned FSMap version 32 2024-03-28T22:41:01.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.881+0000 7f786580b740 1 -- 172.21.15.43:0/887735585 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559f01932e70 con 0x559f018b4bc0 2024-03-28T22:41:01.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.881+0000 7f7845600640 1 -- 172.21.15.43:0/887735585 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7848023030 con 0x559f018b4bc0 2024-03-28T22:41:01.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.881+0000 7f7845600640 1 -- 172.21.15.43:0/887735585 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7848032030 con 0x559f018b4bc0 2024-03-28T22:41:01.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.881+0000 7f786580b740 10 client.27203 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:41:01.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.881+0000 7f786580b740 10 client.27203 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:41:01.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.881+0000 7f786580b740 20 client.27203 populate_metadata read hostname 'smithi043' 2024-03-28T22:41:01.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.881+0000 7f786580b740 1 --2- 172.21.15.43:0/887735585 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559f019333d0 0x559f019357f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:01.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.881+0000 7f786580b740 4 client.27203 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:41:01.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.881+0000 7f786580b740 1 -- 172.21.15.43:0/887735585 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559f0181f990 con 0x559f019333d0 2024-03-28T22:41:01.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.882+0000 7f785be00640 1 --2- 172.21.15.43:0/887735585 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559f019333d0 0x559f019357f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:01.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.882+0000 7f785be00640 1 --2- 172.21.15.43:0/887735585 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559f019333d0 0x559f019357f0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=746dc827d52a133c server_cookie=2f4f6eb8d29f604e in_seq=0 out_seq=0 2024-03-28T22:41:01.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.882+0000 7f7845600640 10 client.27203 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:01.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.883+0000 7f7845600640 1 -- 172.21.15.43:0/887735585 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x559f0181f990 con 0x559f019333d0 2024-03-28T22:41:01.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.883+0000 7f7845600640 10 client.27203 handle_command_reply: tid=0 2024-03-28T22:41:01.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.883+0000 7f786580b740 1 client.27203 shutdown 2024-03-28T22:41:01.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.883+0000 7f786580b740 1 -- 172.21.15.43:0/887735585 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559f018b4bc0 msgr2=0x559f018b6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:01.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.883+0000 7f786580b740 1 --2- 172.21.15.43:0/887735585 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559f018b4bc0 0x559f018b6690 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f784801fe50 tx=0x7f78480149b0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.883+0000 7f786580b740 1 -- 172.21.15.43:0/887735585 shutdown_connections 2024-03-28T22:41:01.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.883+0000 7f786580b740 1 --2- 172.21.15.43:0/887735585 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559f019333d0 0x559f019357f0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.883+0000 7f786580b740 1 --2- 172.21.15.43:0/887735585 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559f018b4bc0 0x559f018b6690 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.883+0000 7f786580b740 1 --2- 172.21.15.43:0/887735585 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559f018bd250 0x559f018b6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.883+0000 7f786580b740 1 --2- 172.21.15.43:0/887735585 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559f018b3ee0 0x559f018d29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.883+0000 7f7845600640 0 client.27203 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:01.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.883+0000 7f786580b740 1 -- 172.21.15.43:0/887735585 >> 172.21.15.43:0/887735585 conn(0x559f0187aa10 msgr2=0x559f018c8470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:01.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.883+0000 7f786580b740 1 -- 172.21.15.43:0/887735585 shutdown_connections 2024-03-28T22:41:01.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.883+0000 7f786580b740 1 -- 172.21.15.43:0/887735585 wait complete. 2024-03-28T22:41:01.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.883+0000 7f786580b740 20 client.27203 trim_cache size 0 max 16384 2024-03-28T22:41:01.886 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:41:01.886 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:41:01.886 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-03-28T22:41:01.886 INFO:teuthology.orchestra.run.smithi043.stdout: "6bae7f7e-b886-49c4-9dd1-60296c3f5f91": { 2024-03-28T22:41:01.886 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-03-28T22:41:01.886 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "6bae7f7e-b886-49c4-9dd1-60296c3f5f91", 2024-03-28T22:41:01.886 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-03-28T22:41:01.886 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:41:01.886 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:41:01.886 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:41:01.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.884+0000 7f7862c00640 1 -- 172.21.15.43:0/2847852944 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f78400a1ef0 msgr2=0x7f78400a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:01.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.884+0000 7f7862c00640 1 --2- 172.21.15.43:0/2847852944 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f78400a1ef0 0x7f78400a43b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f784c0096c0 tx=0x7f784c02e040 comp rx=0 tx=0).stop 2024-03-28T22:41:01.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.884+0000 7f7862c00640 1 -- 172.21.15.43:0/2847852944 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f785c10dfb0 msgr2=0x7f785c19eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:01.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.884+0000 7f7862c00640 1 --2- 172.21.15.43:0/2847852944 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f785c10dfb0 0x7f785c19eaa0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f7854009fd0 tx=0x7f785400ef10 comp rx=0 tx=0).stop 2024-03-28T22:41:01.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.884+0000 7f7862c00640 1 -- 172.21.15.43:0/2847852944 shutdown_connections 2024-03-28T22:41:01.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.884+0000 7f7862c00640 1 --2- 172.21.15.43:0/2847852944 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f78400a1ef0 0x7f78400a43b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.884+0000 7f7862c00640 1 --2- 172.21.15.43:0/2847852944 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f785c10dfb0 0x7f785c19eaa0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.884+0000 7f7862c00640 1 --2- 172.21.15.43:0/2847852944 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f785c074420 0x7f785c19a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.884+0000 7f7862c00640 1 --2- 172.21.15.43:0/2847852944 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f785c073b00 0x7f785c19a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:01.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.884+0000 7f7862c00640 1 -- 172.21.15.43:0/2847852944 >> 172.21.15.43:0/2847852944 conn(0x7f785c0fd180 msgr2=0x7f785c0fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:01.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.884+0000 7f7862c00640 1 -- 172.21.15.43:0/2847852944 shutdown_connections 2024-03-28T22:41:01.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:01.884+0000 7f7862c00640 1 -- 172.21.15.43:0/2847852944 wait complete. 2024-03-28T22:41:01.903 INFO:tasks.cephfs.filesystem:scrub status for tag:6bae7f7e-b886-49c4-9dd1-60296c3f5f91 - {'path': '/', 'tag': '6bae7f7e-b886-49c4-9dd1-60296c3f5f91', 'options': 'recursive,force'} 2024-03-28T22:41:02.356 INFO:tasks.workunit.client.0.smithi043.stdout: 1 234806 8.06 MB/sec execute 33 sec latency 68.116 ms 2024-03-28T22:41:02.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:02 smithi082 ceph-mon[36471]: scrub queued for path: / 2024-03-28T22:41:02.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:02 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:41:02.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:02 smithi082 ceph-mon[36471]: scrub summary: active paths [/] 2024-03-28T22:41:02.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:02 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:41:02.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:02 smithi043 ceph-mon[31823]: scrub queued for path: / 2024-03-28T22:41:02.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:02 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:41:02.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:02 smithi043 ceph-mon[31823]: scrub summary: active paths [/] 2024-03-28T22:41:02.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:02 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:41:02.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:02 smithi172 ceph-mon[36566]: scrub queued for path: / 2024-03-28T22:41:02.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:02 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:41:02.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:02 smithi172 ceph-mon[36566]: scrub summary: active paths [/] 2024-03-28T22:41:02.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:02 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:41:03.356 INFO:tasks.workunit.client.0.smithi043.stdout: 1 235796 7.97 MB/sec execute 34 sec latency 110.547 ms 2024-03-28T22:41:03.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:03 smithi082 ceph-mon[36471]: pgmap v120: 129 pgs: 129 active+clean; 45 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 277 op/s 2024-03-28T22:41:03.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:03 smithi082 ceph-mon[36471]: scrub summary: idle 2024-03-28T22:41:03.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:03 smithi043 ceph-mon[31823]: pgmap v120: 129 pgs: 129 active+clean; 45 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 277 op/s 2024-03-28T22:41:03.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:03 smithi043 ceph-mon[31823]: scrub summary: idle 2024-03-28T22:41:03.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:03 smithi172 ceph-mon[36566]: pgmap v120: 129 pgs: 129 active+clean; 45 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 277 op/s 2024-03-28T22:41:03.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:03 smithi172 ceph-mon[36566]: scrub summary: idle 2024-03-28T22:41:04.356 INFO:tasks.workunit.client.0.smithi043.stdout: 1 237702 8.09 MB/sec execute 35 sec latency 24.828 ms 2024-03-28T22:41:05.356 INFO:tasks.workunit.client.0.smithi043.stdout: 1 238571 7.93 MB/sec execute 36 sec latency 23.056 ms 2024-03-28T22:41:05.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:05 smithi043 ceph-mon[31823]: pgmap v121: 129 pgs: 129 active+clean; 40 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 512 op/s 2024-03-28T22:41:05.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:05 smithi172 ceph-mon[36566]: pgmap v121: 129 pgs: 129 active+clean; 40 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 512 op/s 2024-03-28T22:41:05.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:05 smithi082 ceph-mon[36471]: pgmap v121: 129 pgs: 129 active+clean; 40 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 512 op/s 2024-03-28T22:41:06.356 INFO:tasks.workunit.client.0.smithi043.stdout: 1 240573 8.00 MB/sec execute 37 sec latency 17.530 ms 2024-03-28T22:41:06.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:06 smithi043 ceph-mon[31823]: pgmap v122: 129 pgs: 129 active+clean; 40 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 317 op/s 2024-03-28T22:41:06.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:06 smithi172 ceph-mon[36566]: pgmap v122: 129 pgs: 129 active+clean; 40 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 317 op/s 2024-03-28T22:41:06.698 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:06 smithi082 ceph-mon[36471]: pgmap v122: 129 pgs: 129 active+clean; 40 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 317 op/s 2024-03-28T22:41:07.356 INFO:tasks.workunit.client.0.smithi043.stdout: 1 241715 8.00 MB/sec execute 38 sec latency 11.524 ms 2024-03-28T22:41:08.357 INFO:tasks.workunit.client.0.smithi043.stdout: 1 243015 7.93 MB/sec execute 39 sec latency 48.379 ms 2024-03-28T22:41:08.765 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:08 smithi043 ceph-mon[31823]: pgmap v123: 129 pgs: 129 active+clean; 43 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 398 op/s 2024-03-28T22:41:08.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:08 smithi172 ceph-mon[36566]: pgmap v123: 129 pgs: 129 active+clean; 43 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 398 op/s 2024-03-28T22:41:08.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:08 smithi082 ceph-mon[36471]: pgmap v123: 129 pgs: 129 active+clean; 43 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 398 op/s 2024-03-28T22:41:09.357 INFO:tasks.workunit.client.0.smithi043.stdout: 1 245012 8.07 MB/sec execute 40 sec latency 13.479 ms 2024-03-28T22:41:10.357 INFO:tasks.workunit.client.0.smithi043.stdout: 1 246175 8.00 MB/sec execute 41 sec latency 15.142 ms 2024-03-28T22:41:10.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:10 smithi172 ceph-mon[36566]: pgmap v124: 129 pgs: 129 active+clean; 43 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 315 op/s 2024-03-28T22:41:10.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:10 smithi082 ceph-mon[36471]: pgmap v124: 129 pgs: 129 active+clean; 43 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 315 op/s 2024-03-28T22:41:11.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:10 smithi043 ceph-mon[31823]: pgmap v124: 129 pgs: 129 active+clean; 43 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 315 op/s 2024-03-28T22:41:11.357 INFO:tasks.workunit.client.0.smithi043.stdout: 1 248354 8.11 MB/sec execute 42 sec latency 9.473 ms 2024-03-28T22:41:12.357 INFO:tasks.workunit.client.0.smithi043.stdout: 1 249388 8.00 MB/sec execute 43 sec latency 17.675 ms 2024-03-28T22:41:12.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:12 smithi172 ceph-mon[36566]: pgmap v125: 129 pgs: 129 active+clean; 43 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 315 op/s 2024-03-28T22:41:12.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:12 smithi082 ceph-mon[36471]: pgmap v125: 129 pgs: 129 active+clean; 43 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 315 op/s 2024-03-28T22:41:13.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:12 smithi043 ceph-mon[31823]: pgmap v125: 129 pgs: 129 active+clean; 43 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 315 op/s 2024-03-28T22:41:13.357 INFO:tasks.workunit.client.0.smithi043.stdout: 1 251257 8.06 MB/sec execute 44 sec latency 46.638 ms 2024-03-28T22:41:14.358 INFO:tasks.workunit.client.0.smithi043.stdout: 1 252432 8.03 MB/sec execute 45 sec latency 27.084 ms 2024-03-28T22:41:14.451 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.448+0000 7fa1ab800640 1 -- 172.21.15.43:0/1356496744 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1a4108750 msgr2=0x7fa1a4108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:14.451 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.448+0000 7fa1ab800640 1 --2- 172.21.15.43:0/1356496744 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1a4108750 0x7fa1a4108bb0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa190009930 tx=0x7fa19002bc80 comp rx=0 tx=0).stop 2024-03-28T22:41:14.451 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.448+0000 7fa1ab800640 1 -- 172.21.15.43:0/1356496744 shutdown_connections 2024-03-28T22:41:14.451 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.448+0000 7fa1ab800640 1 --2- 172.21.15.43:0/1356496744 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa1a41090f0 0x7fa1a4114fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.451 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.448+0000 7fa1ab800640 1 --2- 172.21.15.43:0/1356496744 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1a4108750 0x7fa1a4108bb0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.451 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.448+0000 7fa1ab800640 1 --2- 172.21.15.43:0/1356496744 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1a410d130 0x7fa1a410d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.451 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.448+0000 7fa1ab800640 1 -- 172.21.15.43:0/1356496744 >> 172.21.15.43:0/1356496744 conn(0x7fa1a406cc80 msgr2=0x7fa1a406d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:14.451 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.448+0000 7fa1ab800640 1 -- 172.21.15.43:0/1356496744 shutdown_connections 2024-03-28T22:41:14.452 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.448+0000 7fa1ab800640 1 -- 172.21.15.43:0/1356496744 wait complete. 2024-03-28T22:41:14.452 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.449+0000 7fa1ab800640 1 Processor -- start 2024-03-28T22:41:14.452 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.449+0000 7fa1ab800640 1 -- start start 2024-03-28T22:41:14.452 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.449+0000 7fa1ab800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1a4108750 0x7fa1a41a8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:14.452 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.449+0000 7fa1ab800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa1a41090f0 0x7fa1a41a8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:14.452 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.449+0000 7fa1ab800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1a410d130 0x7fa1a41a2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:14.452 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.449+0000 7fa1ab800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa1a41181a0 con 0x7fa1a4108750 2024-03-28T22:41:14.452 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.449+0000 7fa1ab800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa1a4118020 con 0x7fa1a41090f0 2024-03-28T22:41:14.452 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.449+0000 7fa1ab800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fa1a4118320 con 0x7fa1a410d130 2024-03-28T22:41:14.452 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.450+0000 7fa1a8e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1a4108750 0x7fa1a41a8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:14.452 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.450+0000 7fa1a3e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa1a41090f0 0x7fa1a41a8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:14.452 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.450+0000 7fa1a8e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1a4108750 0x7fa1a41a8680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46766/0 (socket says 172.21.15.43:46766) 2024-03-28T22:41:14.452 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.450+0000 7fa1a3e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa1a41090f0 0x7fa1a41a8bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:56396/0 (socket says 172.21.15.43:56396) 2024-03-28T22:41:14.452 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.450+0000 7fa1a8e00640 1 -- 172.21.15.43:0/2136692469 learned_addr learned my addr 172.21.15.43:0/2136692469 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:14.453 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.450+0000 7fa1a9800640 1 --2- 172.21.15.43:0/2136692469 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1a410d130 0x7fa1a41a2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:14.454 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.450+0000 7fa1a8e00640 1 -- 172.21.15.43:0/2136692469 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1a410d130 msgr2=0x7fa1a41a2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:14.454 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.450+0000 7fa1a8e00640 1 --2- 172.21.15.43:0/2136692469 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1a410d130 0x7fa1a41a2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.450+0000 7fa1a8e00640 1 -- 172.21.15.43:0/2136692469 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa1a41090f0 msgr2=0x7fa1a41a8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.450+0000 7fa1a8e00640 1 --2- 172.21.15.43:0/2136692469 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa1a41090f0 0x7fa1a41a8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.450+0000 7fa1a8e00640 1 -- 172.21.15.43:0/2136692469 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1a41a3010 con 0x7fa1a4108750 2024-03-28T22:41:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.450+0000 7fa1a3e00640 1 --2- 172.21.15.43:0/2136692469 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa1a41090f0 0x7fa1a41a8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:41:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.451+0000 7fa1a8e00640 1 --2- 172.21.15.43:0/2136692469 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1a4108750 0x7fa1a41a8680 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa19400e9f0 tx=0x7fa19400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.451+0000 7fa1a1600640 1 -- 172.21.15.43:0/2136692469 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa19400cd80 con 0x7fa1a4108750 2024-03-28T22:41:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.451+0000 7fa1a1600640 1 -- 172.21.15.43:0/2136692469 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa194010910 con 0x7fa1a4108750 2024-03-28T22:41:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.451+0000 7fa1ab800640 1 -- 172.21.15.43:0/2136692469 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1a41a3300 con 0x7fa1a4108750 2024-03-28T22:41:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.451+0000 7fa1a1600640 1 -- 172.21.15.43:0/2136692469 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa194018990 con 0x7fa1a4108750 2024-03-28T22:41:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.451+0000 7fa1ab800640 1 -- 172.21.15.43:0/2136692469 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1a41af4d0 con 0x7fa1a4108750 2024-03-28T22:41:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.455+0000 7fa1a1600640 1 -- 172.21.15.43:0/2136692469 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fa194010ac0 con 0x7fa1a4108750 2024-03-28T22:41:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.455+0000 7fa1ae42f740 1 Processor -- start 2024-03-28T22:41:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.455+0000 7fa1ae42f740 1 -- start start 2024-03-28T22:41:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.455+0000 7fa1ae42f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5606349213a0 0x560634921780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:14.456 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.456+0000 7fa1ae42f740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560634921cc0 0x56063492a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:14.456 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.456+0000 7fa1ae42f740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56063492aa90 0x56063492ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:14.456 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.456+0000 7fa1ae42f740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x560634884640 con 0x560634921cc0 2024-03-28T22:41:14.456 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.456+0000 7fa1ae42f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x56063482d810 con 0x5606349213a0 2024-03-28T22:41:14.456 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.456+0000 7fa1ae42f740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5606348806e0 con 0x56063492aa90 2024-03-28T22:41:14.456 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.456+0000 7fa1a9800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5606349213a0 0x560634921780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:14.456 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.456+0000 7fa1a8e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560634921cc0 0x56063492a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:14.456 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.456+0000 7fa1a9800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5606349213a0 0x560634921780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:56408/0 (socket says 172.21.15.43:56408) 2024-03-28T22:41:14.456 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.456+0000 7fa1a9800640 1 -- 172.21.15.43:0/1492141378 learned_addr learned my addr 172.21.15.43:0/1492141378 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:14.456 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.456+0000 7fa1a3e00640 1 --2- 172.21.15.43:0/1492141378 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56063492aa90 0x56063492ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:14.461 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.456+0000 7fa1a9800640 1 -- 172.21.15.43:0/1492141378 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56063492aa90 msgr2=0x56063492ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:14.461 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.456+0000 7fa1a9800640 1 --2- 172.21.15.43:0/1492141378 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56063492aa90 0x56063492ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.461 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.456+0000 7fa1a9800640 1 -- 172.21.15.43:0/1492141378 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560634921cc0 msgr2=0x56063492a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:14.461 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.456+0000 7fa1a9800640 1 --2- 172.21.15.43:0/1492141378 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560634921cc0 0x56063492a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.461 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.456+0000 7fa1a9800640 1 -- 172.21.15.43:0/1492141378 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560634599210 con 0x5606349213a0 2024-03-28T22:41:14.461 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.456+0000 7fa1a1600640 1 --2- 172.21.15.43:0/2136692469 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa1880a2010 0x7fa1880a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:14.461 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.457+0000 7fa1a9800640 1 --2- 172.21.15.43:0/1492141378 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5606349213a0 0x560634921780 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa19c009b20 tx=0x7fa19c00aea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=664597942cb03dee server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:14.461 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.457+0000 7fa1a1600640 1 -- 172.21.15.43:0/2136692469 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fa1940ce860 con 0x7fa1a4108750 2024-03-28T22:41:14.461 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.457+0000 7fa1a8e00640 1 --2- 172.21.15.43:0/2136692469 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa1880a2010 0x7fa1880a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:14.461 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.457+0000 7fa19a000640 1 -- 172.21.15.43:0/1492141378 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa19c00a980 con 0x5606349213a0 2024-03-28T22:41:14.461 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.457+0000 7fa19a000640 1 -- 172.21.15.43:0/1492141378 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa19c003e50 con 0x5606349213a0 2024-03-28T22:41:14.462 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.457+0000 7fa19a000640 1 -- 172.21.15.43:0/1492141378 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa19c004860 con 0x5606349213a0 2024-03-28T22:41:14.462 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.458+0000 7fa1ae42f740 1 -- 172.21.15.43:0/1492141378 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5606349213a0 msgr2=0x560634921780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:14.462 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.458+0000 7fa1ae42f740 1 --2- 172.21.15.43:0/1492141378 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5606349213a0 0x560634921780 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa19c009b20 tx=0x7fa19c00aea0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.462 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.458+0000 7fa1a8e00640 1 --2- 172.21.15.43:0/2136692469 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa1880a2010 0x7fa1880a44d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x560634920e50 tx=0x7fa1940d4530 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:14.462 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.458+0000 7fa1ae42f740 1 -- 172.21.15.43:0/1492141378 shutdown_connections 2024-03-28T22:41:14.462 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.458+0000 7fa1ae42f740 1 --2- 172.21.15.43:0/1492141378 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56063492aa90 0x56063492ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.462 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.458+0000 7fa1ae42f740 1 --2- 172.21.15.43:0/1492141378 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560634921cc0 0x56063492a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.462 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.458+0000 7fa1ae42f740 1 --2- 172.21.15.43:0/1492141378 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5606349213a0 0x560634921780 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.462 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.458+0000 7fa1ae42f740 1 -- 172.21.15.43:0/1492141378 >> 172.21.15.43:0/1492141378 conn(0x56063491bcc0 msgr2=0x56063491c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:14.462 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.458+0000 7fa1ae42f740 1 -- 172.21.15.43:0/1492141378 shutdown_connections 2024-03-28T22:41:14.462 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.458+0000 7fa1ae42f740 1 -- 172.21.15.43:0/1492141378 wait complete. 2024-03-28T22:41:14.462 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.459+0000 7fa1ae42f740 1 -- start start 2024-03-28T22:41:14.462 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.459+0000 7fa1ae42f740 1 Processor -- start 2024-03-28T22:41:14.462 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.459+0000 7fa1ae42f740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5606349213a0 0x560634927ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:14.463 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.460+0000 7fa1ae42f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560634921cc0 0x560634928010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:14.463 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.460+0000 7fa1ae42f740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56063492aa90 0x560634928550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:14.463 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.460+0000 7fa1ae42f740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56063492d6c0 con 0x56063492aa90 2024-03-28T22:41:14.463 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.460+0000 7fa1a9800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5606349213a0 0x560634927ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:14.463 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.460+0000 7fa1a3e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560634921cc0 0x560634928010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:14.463 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.460+0000 7fa1a9800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5606349213a0 0x560634927ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:55588/0 (socket says 172.21.15.43:55588) 2024-03-28T22:41:14.463 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.460+0000 7fa1a9800640 1 -- 172.21.15.43:0/2374593701 learned_addr learned my addr 172.21.15.43:0/2374593701 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:14.463 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.460+0000 7fa1a3e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560634921cc0 0x560634928010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:56424/0 (socket says 172.21.15.43:56424) 2024-03-28T22:41:14.463 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.460+0000 7fa1ae42f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x560634822d00 con 0x560634921cc0 2024-03-28T22:41:14.463 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.460+0000 7fa1ae42f740 1 -- 172.21.15.43:0/2374593701 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x56063492d840 con 0x5606349213a0 2024-03-28T22:41:14.463 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.460+0000 7fa1ae42f740 10 client.0 _ll_register_callbacks cb 0x56063480d490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:41:14.463 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.460+0000 7fa1a9800640 1 --2- 172.21.15.43:0/2374593701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56063492aa90 0x560634928550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:14.463 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.460+0000 7fa1a3e00640 1 -- 172.21.15.43:0/2374593701 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5606349213a0 msgr2=0x560634927ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:14.463 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.460+0000 7fa1a3e00640 1 --2- 172.21.15.43:0/2374593701 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5606349213a0 0x560634927ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.463 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.460+0000 7fa1a3e00640 1 -- 172.21.15.43:0/2374593701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56063492aa90 msgr2=0x560634928550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.460+0000 7fa1a3e00640 1 --2- 172.21.15.43:0/2374593701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56063492aa90 0x560634928550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.460+0000 7fa1a3e00640 1 -- 172.21.15.43:0/2374593701 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560634665620 con 0x560634921cc0 2024-03-28T22:41:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.460+0000 7fa1a9800640 1 --2- 172.21.15.43:0/2374593701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56063492aa90 0x560634928550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:41:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.461+0000 7fa1a3e00640 1 --2- 172.21.15.43:0/2374593701 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560634921cc0 0x560634928010 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa190009230 tx=0x7fa190004e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=300edfcd80d4538 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.461+0000 7fa18ea00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:41:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.461+0000 7fa18ea00640 1 -- 172.21.15.43:0/2374593701 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa19002ccb0 con 0x560634921cc0 2024-03-28T22:41:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.461+0000 7fa1ae42f740 1 -- 172.21.15.43:0/2374593701 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa19c003e50 con 0x560634921cc0 2024-03-28T22:41:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.461+0000 7fa18ea00640 1 -- 172.21.15.43:0/2374593701 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fa190004040 con 0x560634921cc0 2024-03-28T22:41:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.461+0000 7fa18ea00640 1 -- 172.21.15.43:0/2374593701 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa190004240 con 0x560634921cc0 2024-03-28T22:41:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.461+0000 7fa1ae42f740 10 client.27215 fetch_fsmap learned FSMap version 32 2024-03-28T22:41:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.461+0000 7fa1ae42f740 1 -- 172.21.15.43:0/2374593701 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560634939600 con 0x560634921cc0 2024-03-28T22:41:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.462+0000 7fa18ea00640 1 -- 172.21.15.43:0/2374593701 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa19c003e50 con 0x560634921cc0 2024-03-28T22:41:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.462+0000 7fa18ea00640 1 -- 172.21.15.43:0/2374593701 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa190003850 con 0x560634921cc0 2024-03-28T22:41:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.462+0000 7fa1ae42f740 10 client.27215 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:41:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.462+0000 7fa1ae42f740 10 client.27215 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:41:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.462+0000 7fa1ae42f740 20 client.27215 populate_metadata read hostname 'smithi043' 2024-03-28T22:41:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.462+0000 7fa1ae42f740 1 --2- 172.21.15.43:0/2374593701 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56063493aac0 0x56063493ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.462+0000 7fa1ae42f740 4 client.27215 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:41:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.462+0000 7fa1ae42f740 1 -- 172.21.15.43:0/2374593701 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560634898030 con 0x56063493aac0 2024-03-28T22:41:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.462+0000 7fa1a9800640 1 --2- 172.21.15.43:0/2374593701 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56063493aac0 0x56063493ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.463+0000 7fa1a9800640 1 --2- 172.21.15.43:0/2374593701 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56063493aac0 0x56063493ceb0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aceb1cc64bb575a server_cookie=424c6004bf375967 in_seq=0 out_seq=0 2024-03-28T22:41:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.463+0000 7fa18ea00640 10 client.27215 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:14.466 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.465+0000 7fa18ea00640 1 -- 172.21.15.43:0/2374593701 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560634898030 con 0x56063493aac0 2024-03-28T22:41:14.466 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.465+0000 7fa18ea00640 10 client.27215 handle_command_reply: tid=0 2024-03-28T22:41:14.466 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.465+0000 7fa1ae42f740 1 client.27215 shutdown 2024-03-28T22:41:14.467 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.465+0000 7fa1ae42f740 1 -- 172.21.15.43:0/2374593701 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560634921cc0 msgr2=0x560634928010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:14.467 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.465+0000 7fa1ae42f740 1 --2- 172.21.15.43:0/2374593701 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560634921cc0 0x560634928010 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa190009230 tx=0x7fa190004e50 comp rx=0 tx=0).stop 2024-03-28T22:41:14.467 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.465+0000 7fa1ae42f740 1 -- 172.21.15.43:0/2374593701 shutdown_connections 2024-03-28T22:41:14.467 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.465+0000 7fa1ae42f740 1 --2- 172.21.15.43:0/2374593701 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56063493aac0 0x56063493ceb0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.467 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.465+0000 7fa1ae42f740 1 --2- 172.21.15.43:0/2374593701 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560634921cc0 0x560634928010 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.467 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.465+0000 7fa1ae42f740 1 --2- 172.21.15.43:0/2374593701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56063492aa90 0x560634928550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.467 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.465+0000 7fa1ae42f740 1 --2- 172.21.15.43:0/2374593701 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5606349213a0 0x560634927ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.467 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.465+0000 7fa1ae42f740 1 -- 172.21.15.43:0/2374593701 >> 172.21.15.43:0/2374593701 conn(0x56063491bcc0 msgr2=0x56063491da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:14.467 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.465+0000 7fa18ea00640 0 client.27215 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:14.467 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.465+0000 7fa1ae42f740 1 -- 172.21.15.43:0/2374593701 shutdown_connections 2024-03-28T22:41:14.467 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.465+0000 7fa1ae42f740 1 -- 172.21.15.43:0/2374593701 wait complete. 2024-03-28T22:41:14.467 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.466+0000 7fa1ae42f740 20 client.27215 trim_cache size 0 max 16384 2024-03-28T22:41:14.480 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.482+0000 7fa1ae42f740 1 Processor -- start 2024-03-28T22:41:14.480 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.482+0000 7fa1ae42f740 1 -- start start 2024-03-28T22:41:14.480 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.483+0000 7fa1ae42f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5606349213a0 0x56063492a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.483+0000 7fa1ae42f740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560634921cc0 0x560634936460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.483+0000 7fa1ae42f740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56063492aa90 0x560634942780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.483+0000 7fa1ae42f740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa1a41b94f0 con 0x56063492aa90 2024-03-28T22:41:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.483+0000 7fa1ae42f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa1a41bb030 con 0x5606349213a0 2024-03-28T22:41:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.483+0000 7fa1ae42f740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fa1a41b9bc0 con 0x560634921cc0 2024-03-28T22:41:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.483+0000 7fa1a9800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5606349213a0 0x56063492a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.483+0000 7fa1a3e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560634921cc0 0x560634936460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.483+0000 7fa1a9800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5606349213a0 0x56063492a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:56430/0 (socket says 172.21.15.43:56430) 2024-03-28T22:41:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.483+0000 7fa1a9800640 1 -- 172.21.15.43:0/4183141140 learned_addr learned my addr 172.21.15.43:0/4183141140 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.483+0000 7fa1a9800640 1 --2- 172.21.15.43:0/4183141140 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56063492aa90 0x560634942780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.483+0000 7fa1a3e00640 1 -- 172.21.15.43:0/4183141140 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5606349213a0 msgr2=0x56063492a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.483+0000 7fa1a3e00640 1 --2- 172.21.15.43:0/4183141140 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5606349213a0 0x56063492a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.483+0000 7fa1a3e00640 1 -- 172.21.15.43:0/4183141140 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56063492aa90 msgr2=0x560634942780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.483+0000 7fa1a3e00640 1 --2- 172.21.15.43:0/4183141140 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56063492aa90 0x560634942780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.483+0000 7fa1a3e00640 1 -- 172.21.15.43:0/4183141140 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56063493d220 con 0x560634921cc0 2024-03-28T22:41:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.484+0000 7fa1a3e00640 1 --2- 172.21.15.43:0/4183141140 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560634921cc0 0x560634936460 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fa190003d20 tx=0x7fa19002bc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6a505cc9dae0abd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.484+0000 7fa199600640 1 -- 172.21.15.43:0/4183141140 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa190009520 con 0x560634921cc0 2024-03-28T22:41:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.484+0000 7fa199600640 1 -- 172.21.15.43:0/4183141140 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa19002cab0 con 0x560634921cc0 2024-03-28T22:41:14.483 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.484+0000 7fa199600640 1 -- 172.21.15.43:0/4183141140 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa190004ef0 con 0x560634921cc0 2024-03-28T22:41:14.483 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.485+0000 7fa1ae42f740 1 -- 172.21.15.43:0/4183141140 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560634921cc0 msgr2=0x560634936460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:14.483 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.485+0000 7fa1ae42f740 1 --2- 172.21.15.43:0/4183141140 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560634921cc0 0x560634936460 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fa190003d20 tx=0x7fa19002bc80 comp rx=0 tx=0).stop 2024-03-28T22:41:14.483 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.485+0000 7fa1ae42f740 1 -- 172.21.15.43:0/4183141140 shutdown_connections 2024-03-28T22:41:14.483 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.485+0000 7fa1ae42f740 1 --2- 172.21.15.43:0/4183141140 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56063492aa90 0x560634942780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.484 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.485+0000 7fa1ae42f740 1 --2- 172.21.15.43:0/4183141140 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560634921cc0 0x560634936460 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.484 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.485+0000 7fa1ae42f740 1 --2- 172.21.15.43:0/4183141140 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5606349213a0 0x56063492a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.484 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.485+0000 7fa1ae42f740 1 -- 172.21.15.43:0/4183141140 >> 172.21.15.43:0/4183141140 conn(0x56063491bcc0 msgr2=0x56063491ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:14.484 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.485+0000 7fa1ae42f740 1 -- 172.21.15.43:0/4183141140 shutdown_connections 2024-03-28T22:41:14.484 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.486+0000 7fa1ae42f740 1 -- 172.21.15.43:0/4183141140 wait complete. 2024-03-28T22:41:14.484 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.486+0000 7fa1ae42f740 1 -- start start 2024-03-28T22:41:14.485 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.486+0000 7fa1ae42f740 1 Processor -- start 2024-03-28T22:41:14.485 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.487+0000 7fa1ae42f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5606349213a0 0x56063492e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:14.485 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.487+0000 7fa1ae42f740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560634921cc0 0x56063492eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:14.488 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.487+0000 7fa1a9800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5606349213a0 0x56063492e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:14.488 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.487+0000 7fa1a9800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5606349213a0 0x56063492e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:56446/0 (socket says 172.21.15.43:56446) 2024-03-28T22:41:14.488 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.487+0000 7fa1a9800640 1 -- 172.21.15.43:0/2152965307 learned_addr learned my addr 172.21.15.43:0/2152965307 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:14.489 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.487+0000 7fa1a3e00640 1 --2- 172.21.15.43:0/2152965307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560634921cc0 0x56063492eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:14.489 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.487+0000 7fa1ae42f740 1 --2- 172.21.15.43:0/2152965307 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56063492aa90 0x56063492f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:14.489 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.487+0000 7fa1ae42f740 1 -- 172.21.15.43:0/2152965307 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa1a41ba960 con 0x560634921cc0 2024-03-28T22:41:14.489 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.487+0000 7fa1ae42f740 1 -- 172.21.15.43:0/2152965307 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa1a41ba290 con 0x5606349213a0 2024-03-28T22:41:14.489 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.487+0000 7fa1ae42f740 1 -- 172.21.15.43:0/2152965307 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fa1a41b74d0 con 0x56063492aa90 2024-03-28T22:41:14.489 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.488+0000 7fa1ae42f740 10 client.0 _ll_register_callbacks cb 0x7fa190009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:41:14.489 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.488+0000 7fa1a9800640 1 --2- 172.21.15.43:0/2152965307 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56063492aa90 0x56063492f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:14.489 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.488+0000 7fa1a9800640 1 -- 172.21.15.43:0/2152965307 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56063492aa90 msgr2=0x56063492f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:14.489 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.488+0000 7fa1a9800640 1 --2- 172.21.15.43:0/2152965307 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56063492aa90 0x56063492f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.489 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.488+0000 7fa1a9800640 1 -- 172.21.15.43:0/2152965307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560634921cc0 msgr2=0x56063492eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:14.489 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.488+0000 7fa1a9800640 1 --2- 172.21.15.43:0/2152965307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560634921cc0 0x56063492eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.489 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.488+0000 7fa1a9800640 1 -- 172.21.15.43:0/2152965307 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606349a0000 con 0x5606349213a0 2024-03-28T22:41:14.489 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.488+0000 7fa1a3e00640 1 --2- 172.21.15.43:0/2152965307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560634921cc0 0x56063492eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:41:14.489 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.488+0000 7fa1a9800640 1 --2- 172.21.15.43:0/2152965307 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5606349213a0 0x56063492e5a0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa19c009af0 tx=0x7fa19c025040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b568df9a9abf796e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:14.490 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.488+0000 7fa18ea00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:41:14.490 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.489+0000 7fa18ea00640 1 -- 172.21.15.43:0/2152965307 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa19c00a780 con 0x5606349213a0 2024-03-28T22:41:14.490 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.489+0000 7fa18ea00640 1 -- 172.21.15.43:0/2152965307 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fa19c00a920 con 0x5606349213a0 2024-03-28T22:41:14.490 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.489+0000 7fa1ae42f740 1 -- 172.21.15.43:0/2152965307 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa19002cab0 con 0x5606349213a0 2024-03-28T22:41:14.490 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.489+0000 7fa18ea00640 1 -- 172.21.15.43:0/2152965307 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa19c03c050 con 0x5606349213a0 2024-03-28T22:41:14.490 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.489+0000 7fa1ae42f740 10 client.27221 fetch_fsmap learned FSMap version 32 2024-03-28T22:41:14.490 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.489+0000 7fa1ae42f740 1 -- 172.21.15.43:0/2152965307 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5606349a12b0 con 0x5606349213a0 2024-03-28T22:41:14.490 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.490+0000 7fa18ea00640 1 -- 172.21.15.43:0/2152965307 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa19002cab0 con 0x5606349213a0 2024-03-28T22:41:14.490 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.490+0000 7fa18ea00640 1 -- 172.21.15.43:0/2152965307 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa19c007450 con 0x5606349213a0 2024-03-28T22:41:14.490 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.490+0000 7fa1ae42f740 10 client.27221 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:41:14.490 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.490+0000 7fa1ae42f740 10 client.27221 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:41:14.490 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.490+0000 7fa1ae42f740 20 client.27221 populate_metadata read hostname 'smithi043' 2024-03-28T22:41:14.490 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.490+0000 7fa1ae42f740 1 --2- 172.21.15.43:0/2152965307 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5606349a1790 0x5606349a3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:14.490 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.490+0000 7fa1ae42f740 4 client.27221 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T22:41:14.491 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.490+0000 7fa1ae42f740 1 -- 172.21.15.43:0/2152965307 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560634891b80 con 0x5606349a1790 2024-03-28T22:41:14.491 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.490+0000 7fa1a3e00640 1 --2- 172.21.15.43:0/2152965307 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5606349a1790 0x5606349a3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:14.491 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.491+0000 7fa1a3e00640 1 --2- 172.21.15.43:0/2152965307 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5606349a1790 0x5606349a3b80 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f48ef620d949d588 server_cookie=c9ffae39ba1328cf in_seq=0 out_seq=0 2024-03-28T22:41:14.491 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.491+0000 7fa18ea00640 10 client.27221 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:14.676 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.677+0000 7fa18ea00640 1 -- 172.21.15.43:0/2152965307 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560634891b80 con 0x5606349a1790 2024-03-28T22:41:14.676 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.677+0000 7fa18ea00640 10 client.27221 handle_command_reply: tid=0 2024-03-28T22:41:14.676 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.677+0000 7fa1ae42f740 1 client.27221 shutdown 2024-03-28T22:41:14.676 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.678+0000 7fa1ae42f740 1 -- 172.21.15.43:0/2152965307 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5606349213a0 msgr2=0x56063492e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.678+0000 7fa1ae42f740 1 --2- 172.21.15.43:0/2152965307 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5606349213a0 0x56063492e5a0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa19c009af0 tx=0x7fa19c025040 comp rx=0 tx=0).stop 2024-03-28T22:41:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.678+0000 7fa1ae42f740 1 -- 172.21.15.43:0/2152965307 shutdown_connections 2024-03-28T22:41:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.678+0000 7fa1ae42f740 1 --2- 172.21.15.43:0/2152965307 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56063492aa90 0x56063492f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.678+0000 7fa1ae42f740 1 --2- 172.21.15.43:0/2152965307 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560634921cc0 0x56063492eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.678+0000 7fa1ae42f740 1 --2- 172.21.15.43:0/2152965307 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5606349a1790 0x5606349a3b80 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.678+0000 7fa1ae42f740 1 --2- 172.21.15.43:0/2152965307 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5606349213a0 0x56063492e5a0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.678+0000 7fa18ea00640 0 client.27221 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.678+0000 7fa1ae42f740 1 -- 172.21.15.43:0/2152965307 >> 172.21.15.43:0/2152965307 conn(0x56063491bcc0 msgr2=0x560634927a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.678+0000 7fa1ae42f740 1 -- 172.21.15.43:0/2152965307 shutdown_connections 2024-03-28T22:41:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.678+0000 7fa1ae42f740 1 -- 172.21.15.43:0/2152965307 wait complete. 2024-03-28T22:41:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.678+0000 7fa1ae42f740 20 client.27221 trim_cache size 0 max 16384 2024-03-28T22:41:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:{ 2024-03-28T22:41:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "message": "", 2024-03-28T22:41:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "return_code": 0 2024-03-28T22:41:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:} 2024-03-28T22:41:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.679+0000 7fa1ab800640 1 -- 172.21.15.43:0/2136692469 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa1880a2010 msgr2=0x7fa1880a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.679+0000 7fa1ab800640 1 --2- 172.21.15.43:0/2136692469 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa1880a2010 0x7fa1880a44d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x560634920e50 tx=0x7fa1940d4530 comp rx=0 tx=0).stop 2024-03-28T22:41:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.679+0000 7fa1ab800640 1 -- 172.21.15.43:0/2136692469 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1a4108750 msgr2=0x7fa1a41a8680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:14.680 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.679+0000 7fa1ab800640 1 --2- 172.21.15.43:0/2136692469 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1a4108750 0x7fa1a41a8680 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa19400e9f0 tx=0x7fa19400eeb0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.680 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.679+0000 7fa1ab800640 1 -- 172.21.15.43:0/2136692469 shutdown_connections 2024-03-28T22:41:14.680 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.679+0000 7fa1ab800640 1 --2- 172.21.15.43:0/2136692469 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa1880a2010 0x7fa1880a44d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.680 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.679+0000 7fa1ab800640 1 --2- 172.21.15.43:0/2136692469 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa1a41090f0 0x7fa1a41a8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.680 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.679+0000 7fa1ab800640 1 --2- 172.21.15.43:0/2136692469 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa1a410d130 0x7fa1a41a2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.680 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.679+0000 7fa1ab800640 1 --2- 172.21.15.43:0/2136692469 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa1a4108750 0x7fa1a41a8680 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:14.680 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.679+0000 7fa1ab800640 1 -- 172.21.15.43:0/2136692469 >> 172.21.15.43:0/2136692469 conn(0x7fa1a406cc80 msgr2=0x7fa1a4113340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:14.680 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.679+0000 7fa1ab800640 1 -- 172.21.15.43:0/2136692469 shutdown_connections 2024-03-28T22:41:14.680 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:14.680+0000 7fa1ab800640 1 -- 172.21.15.43:0/2136692469 wait complete. 2024-03-28T22:41:14.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:14 smithi172 ceph-mon[36566]: pgmap v126: 129 pgs: 129 active+clean; 49 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 395 op/s 2024-03-28T22:41:14.948 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:14 smithi082 ceph-mon[36471]: pgmap v126: 129 pgs: 129 active+clean; 49 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 395 op/s 2024-03-28T22:41:15.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:14 smithi043 ceph-mon[31823]: pgmap v126: 129 pgs: 129 active+clean; 49 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 395 op/s 2024-03-28T22:41:15.357 INFO:tasks.workunit.client.0.smithi043.stdout: 1 254173 8.00 MB/sec execute 46 sec latency 62.083 ms 2024-03-28T22:41:16.358 INFO:tasks.workunit.client.0.smithi043.stdout: 1 255816 8.10 MB/sec execute 47 sec latency 9.088 ms 2024-03-28T22:41:16.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:16 smithi172 ceph-mon[36566]: pgmap v127: 129 pgs: 129 active+clean; 49 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 161 op/s 2024-03-28T22:41:16.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:16 smithi082 ceph-mon[36471]: pgmap v127: 129 pgs: 129 active+clean; 49 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 161 op/s 2024-03-28T22:41:17.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:16 smithi043 ceph-mon[31823]: pgmap v127: 129 pgs: 129 active+clean; 49 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 161 op/s 2024-03-28T22:41:17.358 INFO:tasks.workunit.client.0.smithi043.stdout: 1 257237 8.04 MB/sec execute 48 sec latency 16.195 ms 2024-03-28T22:41:18.358 INFO:tasks.workunit.client.0.smithi043.stdout: 1 258806 8.06 MB/sec execute 49 sec latency 92.001 ms 2024-03-28T22:41:18.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:18 smithi172 ceph-mon[36566]: pgmap v128: 129 pgs: 129 active+clean; 43 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 347 op/s 2024-03-28T22:41:18.948 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:18 smithi082 ceph-mon[36471]: pgmap v128: 129 pgs: 129 active+clean; 43 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 347 op/s 2024-03-28T22:41:19.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:18 smithi043 ceph-mon[31823]: pgmap v128: 129 pgs: 129 active+clean; 43 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 347 op/s 2024-03-28T22:41:19.358 INFO:tasks.workunit.client.0.smithi043.stdout: 1 259653 8.00 MB/sec execute 50 sec latency 33.593 ms 2024-03-28T22:41:20.358 INFO:tasks.workunit.client.0.smithi043.stdout: 1 261315 7.97 MB/sec execute 51 sec latency 42.088 ms 2024-03-28T22:41:20.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:20 smithi172 ceph-mon[36566]: pgmap v129: 129 pgs: 129 active+clean; 43 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 267 op/s 2024-03-28T22:41:20.948 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:20 smithi082 ceph-mon[36471]: pgmap v129: 129 pgs: 129 active+clean; 43 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 267 op/s 2024-03-28T22:41:21.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:20 smithi043 ceph-mon[31823]: pgmap v129: 129 pgs: 129 active+clean; 43 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 267 op/s 2024-03-28T22:41:21.358 INFO:tasks.workunit.client.0.smithi043.stdout: 1 262760 8.05 MB/sec execute 52 sec latency 13.362 ms 2024-03-28T22:41:22.358 INFO:tasks.workunit.client.0.smithi043.stdout: 1 263676 7.94 MB/sec execute 53 sec latency 21.330 ms 2024-03-28T22:41:22.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:22 smithi172 ceph-mon[36566]: pgmap v130: 129 pgs: 129 active+clean; 43 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 267 op/s 2024-03-28T22:41:22.948 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:22 smithi082 ceph-mon[36471]: pgmap v130: 129 pgs: 129 active+clean; 43 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 267 op/s 2024-03-28T22:41:23.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:22 smithi043 ceph-mon[31823]: pgmap v130: 129 pgs: 129 active+clean; 43 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.3 MiB/s wr, 267 op/s 2024-03-28T22:41:23.359 INFO:tasks.workunit.client.0.smithi043.stdout: 1 265509 7.99 MB/sec execute 54 sec latency 13.927 ms 2024-03-28T22:41:24.358 INFO:tasks.workunit.client.0.smithi043.stdout: 1 266621 7.96 MB/sec execute 55 sec latency 130.541 ms 2024-03-28T22:41:24.818 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:24 smithi043 ceph-mon[31823]: pgmap v131: 129 pgs: 129 active+clean; 46 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 341 op/s 2024-03-28T22:41:24.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:24 smithi172 ceph-mon[36566]: pgmap v131: 129 pgs: 129 active+clean; 46 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 341 op/s 2024-03-28T22:41:24.948 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:24 smithi082 ceph-mon[36471]: pgmap v131: 129 pgs: 129 active+clean; 46 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 341 op/s 2024-03-28T22:41:25.358 INFO:tasks.workunit.client.0.smithi043.stdout: 1 268509 7.95 MB/sec execute 56 sec latency 18.756 ms 2024-03-28T22:41:25.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:25 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:41:25.948 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:25 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:41:26.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:25 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:41:26.359 INFO:tasks.workunit.client.0.smithi043.stdout: 1 270059 8.02 MB/sec execute 57 sec latency 14.972 ms 2024-03-28T22:41:26.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:26 smithi172 ceph-mon[36566]: pgmap v132: 129 pgs: 129 active+clean; 46 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.2 MiB/s wr, 260 op/s 2024-03-28T22:41:26.948 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:26 smithi082 ceph-mon[36471]: pgmap v132: 129 pgs: 129 active+clean; 46 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.2 MiB/s wr, 260 op/s 2024-03-28T22:41:27.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:26 smithi043 ceph-mon[31823]: pgmap v132: 129 pgs: 129 active+clean; 46 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.2 MiB/s wr, 260 op/s 2024-03-28T22:41:27.359 INFO:tasks.workunit.client.0.smithi043.stdout: 1 271805 7.98 MB/sec execute 58 sec latency 64.320 ms 2024-03-28T22:41:27.829 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.831+0000 7faafea00640 1 -- 172.21.15.43:0/2317774111 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faaf8107f90 msgr2=0x7faaf8108370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:27.830 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.831+0000 7faafea00640 1 --2- 172.21.15.43:0/2317774111 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faaf8107f90 0x7faaf8108370 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7faae8009860 tx=0x7faae802bb70 comp rx=0 tx=0).stop 2024-03-28T22:41:27.830 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.832+0000 7faafea00640 1 -- 172.21.15.43:0/2317774111 shutdown_connections 2024-03-28T22:41:27.830 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.832+0000 7faafea00640 1 --2- 172.21.15.43:0/2317774111 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf810d060 0x7faaf8114c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:27.830 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.832+0000 7faafea00640 1 --2- 172.21.15.43:0/2317774111 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faaf8108940 0x7faaf810c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:27.830 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.832+0000 7faafea00640 1 --2- 172.21.15.43:0/2317774111 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faaf8107f90 0x7faaf8108370 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:27.830 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.832+0000 7faafea00640 1 -- 172.21.15.43:0/2317774111 >> 172.21.15.43:0/2317774111 conn(0x7faaf8075cc0 msgr2=0x7faaf80780e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:27.830 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.832+0000 7faafea00640 1 -- 172.21.15.43:0/2317774111 shutdown_connections 2024-03-28T22:41:27.830 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.832+0000 7faafea00640 1 -- 172.21.15.43:0/2317774111 wait complete. 2024-03-28T22:41:27.831 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.833+0000 7faafea00640 1 Processor -- start 2024-03-28T22:41:27.831 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.833+0000 7faafea00640 1 -- start start 2024-03-28T22:41:27.831 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.834+0000 7faafea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faaf8107f90 0x7faaf819e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:27.831 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.834+0000 7faafea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faaf810d060 0x7faaf819eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:27.831 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.834+0000 7faafea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf81a2e30 0x7faaf81a32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:27.832 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.834+0000 7faafea00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7faaf81180f0 con 0x7faaf81a2e30 2024-03-28T22:41:27.832 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.834+0000 7faafea00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7faaf8117f70 con 0x7faaf810d060 2024-03-28T22:41:27.832 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.834+0000 7faafea00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7faaf8118270 con 0x7faaf8107f90 2024-03-28T22:41:27.832 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.834+0000 7faaf7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faaf8107f90 0x7faaf819e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:27.832 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.834+0000 7faaf7400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faaf810d060 0x7faaf819eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:27.832 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.834+0000 7faaf7400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faaf810d060 0x7faaf819eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33534/0 (socket says 172.21.15.43:33534) 2024-03-28T22:41:27.832 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.834+0000 7faaf7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faaf8107f90 0x7faaf819e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:50144/0 (socket says 172.21.15.43:50144) 2024-03-28T22:41:27.832 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.834+0000 7faaf7400640 1 -- 172.21.15.43:0/1349101750 learned_addr learned my addr 172.21.15.43:0/1349101750 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:27.832 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.834+0000 7faafca00640 1 --2- 172.21.15.43:0/1349101750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf81a2e30 0x7faaf81a32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.834+0000 7faaf7e00640 1 -- 172.21.15.43:0/1349101750 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faaf810d060 msgr2=0x7faaf819eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.834+0000 7faaf7e00640 1 --2- 172.21.15.43:0/1349101750 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faaf810d060 0x7faaf819eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.834+0000 7faaf7e00640 1 -- 172.21.15.43:0/1349101750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf81a2e30 msgr2=0x7faaf81a32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.834+0000 7faaf7e00640 1 --2- 172.21.15.43:0/1349101750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf81a2e30 0x7faaf81a32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.834+0000 7faaf7e00640 1 -- 172.21.15.43:0/1349101750 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaf81a3920 con 0x7faaf8107f90 2024-03-28T22:41:27.834 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.835+0000 7faaf7e00640 1 --2- 172.21.15.43:0/1349101750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faaf8107f90 0x7faaf819e570 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7faae802c0b0 tx=0x7faae80045b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:27.834 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.835+0000 7faaf4c00640 1 -- 172.21.15.43:0/1349101750 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae8041020 con 0x7faaf8107f90 2024-03-28T22:41:27.834 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.835+0000 7faaf4c00640 1 -- 172.21.15.43:0/1349101750 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faae802c9f0 con 0x7faaf8107f90 2024-03-28T22:41:27.834 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.835+0000 7faaf4c00640 1 -- 172.21.15.43:0/1349101750 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae8039040 con 0x7faaf8107f90 2024-03-28T22:41:27.834 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.835+0000 7faafea00640 1 -- 172.21.15.43:0/1349101750 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaf81a3bb0 con 0x7faaf8107f90 2024-03-28T22:41:27.834 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.835+0000 7faafea00640 1 -- 172.21.15.43:0/1349101750 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faaf81ab320 con 0x7faaf8107f90 2024-03-28T22:41:27.836 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.839+0000 7fab01832740 1 Processor -- start 2024-03-28T22:41:27.836 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.839+0000 7faaf4c00640 1 -- 172.21.15.43:0/1349101750 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7faae8044020 con 0x7faaf8107f90 2024-03-28T22:41:27.836 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.839+0000 7fab01832740 1 -- start start 2024-03-28T22:41:27.838 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.839+0000 7fab01832740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5610711b53a0 0x5610711b5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:27.838 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.839+0000 7fab01832740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5610711b5cc0 0x5610711be550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:27.838 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.839+0000 7fab01832740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5610711bea90 0x5610711c0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:27.838 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.839+0000 7fab01832740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x561071118640 con 0x5610711bea90 2024-03-28T22:41:27.838 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.839+0000 7fab01832740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5610710c1810 con 0x5610711b5cc0 2024-03-28T22:41:27.838 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.839+0000 7fab01832740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5610711146e0 con 0x5610711b53a0 2024-03-28T22:41:27.838 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.839+0000 7faaf7e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5610711b5cc0 0x5610711be550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:27.838 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.839+0000 7faafca00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5610711b53a0 0x5610711b5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:27.839 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.840+0000 7faafca00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5610711b53a0 0x5610711b5780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:50146/0 (socket says 172.21.15.43:50146) 2024-03-28T22:41:27.839 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.840+0000 7faafca00640 1 -- 172.21.15.43:0/1137096963 learned_addr learned my addr 172.21.15.43:0/1137096963 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:27.839 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.840+0000 7faaf7400640 1 --2- 172.21.15.43:0/1137096963 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5610711bea90 0x5610711c0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:27.839 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.840+0000 7faafca00640 1 -- 172.21.15.43:0/1137096963 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5610711b5cc0 msgr2=0x5610711be550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:27.839 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.840+0000 7faafca00640 1 --2- 172.21.15.43:0/1137096963 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5610711b5cc0 0x5610711be550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:27.839 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.840+0000 7faafca00640 1 -- 172.21.15.43:0/1137096963 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5610711bea90 msgr2=0x5610711c0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:27.839 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.840+0000 7faafca00640 1 --2- 172.21.15.43:0/1137096963 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5610711bea90 0x5610711c0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:27.839 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.840+0000 7faafca00640 1 -- 172.21.15.43:0/1137096963 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561070e2d210 con 0x5610711b53a0 2024-03-28T22:41:27.839 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.840+0000 7faafca00640 1 --2- 172.21.15.43:0/1137096963 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5610711b53a0 0x5610711b5780 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7faaf0009f50 tx=0x7faaf00118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4e48e8c3f05e0db server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:27.839 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.841+0000 7faaed200640 1 -- 172.21.15.43:0/1137096963 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf0004d20 con 0x5610711b53a0 2024-03-28T22:41:27.839 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.841+0000 7faaed200640 1 -- 172.21.15.43:0/1137096963 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faaf0004050 con 0x5610711b53a0 2024-03-28T22:41:27.840 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.841+0000 7faaed200640 1 -- 172.21.15.43:0/1137096963 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf0005280 con 0x5610711b53a0 2024-03-28T22:41:27.840 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.841+0000 7faaf4c00640 1 --2- 172.21.15.43:0/1349101750 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7faadc0a2010 0x7faadc0a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:27.843 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.841+0000 7faaf4c00640 1 -- 172.21.15.43:0/1349101750 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7faae80efb70 con 0x7faaf8107f90 2024-03-28T22:41:27.843 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.841+0000 7faaf7e00640 1 --2- 172.21.15.43:0/1349101750 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7faadc0a2010 0x7faadc0a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:27.844 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.841+0000 7fab01832740 1 -- 172.21.15.43:0/1137096963 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5610711b53a0 msgr2=0x5610711b5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:27.844 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.841+0000 7fab01832740 1 --2- 172.21.15.43:0/1137096963 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5610711b53a0 0x5610711b5780 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7faaf0009f50 tx=0x7faaf00118d0 comp rx=0 tx=0).stop 2024-03-28T22:41:27.844 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.842+0000 7fab01832740 1 -- 172.21.15.43:0/1137096963 shutdown_connections 2024-03-28T22:41:27.844 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.842+0000 7fab01832740 1 --2- 172.21.15.43:0/1137096963 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5610711b5cc0 0x5610711be550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:27.844 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.842+0000 7fab01832740 1 --2- 172.21.15.43:0/1137096963 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5610711bea90 0x5610711c0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:27.844 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.842+0000 7fab01832740 1 --2- 172.21.15.43:0/1137096963 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5610711b53a0 0x5610711b5780 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:27.844 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.842+0000 7fab01832740 1 -- 172.21.15.43:0/1137096963 >> 172.21.15.43:0/1137096963 conn(0x5610711afcc0 msgr2=0x5610711b00a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:27.844 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.842+0000 7fab01832740 1 -- 172.21.15.43:0/1137096963 shutdown_connections 2024-03-28T22:41:27.844 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.842+0000 7fab01832740 1 -- 172.21.15.43:0/1137096963 wait complete. 2024-03-28T22:41:27.844 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.842+0000 7faaf7e00640 1 --2- 172.21.15.43:0/1349101750 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7faadc0a2010 0x7faadc0a44d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x5610711b4e50 tx=0x7faae80f3620 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:27.844 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.842+0000 7fab01832740 1 -- start start 2024-03-28T22:41:27.844 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.842+0000 7fab01832740 1 Processor -- start 2024-03-28T22:41:27.844 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.843+0000 7fab01832740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5610711b53a0 0x5610711bbb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:27.845 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.843+0000 7fab01832740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5610711b5cc0 0x5610711bc050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:27.845 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.843+0000 7fab01832740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5610711bea90 0x5610711bc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:27.845 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.843+0000 7fab01832740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5610711c16c0 con 0x5610711bea90 2024-03-28T22:41:27.845 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.843+0000 7fab01832740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5610710b6d00 con 0x5610711b5cc0 2024-03-28T22:41:27.845 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.843+0000 7fab01832740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5610711c1840 con 0x5610711b53a0 2024-03-28T22:41:27.845 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.843+0000 7fab01832740 10 client.0 _ll_register_callbacks cb 0x5610710a1490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:41:27.845 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.843+0000 7faaf7400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5610711b5cc0 0x5610711bc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:27.845 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.843+0000 7faafca00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5610711b53a0 0x5610711bbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:27.845 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.843+0000 7faaf7400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5610711b5cc0 0x5610711bc050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33546/0 (socket says 172.21.15.43:33546) 2024-03-28T22:41:27.845 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.843+0000 7faaf7400640 1 -- 172.21.15.43:0/1252837957 learned_addr learned my addr 172.21.15.43:0/1252837957 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:27.845 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.843+0000 7faafca00640 1 --2- 172.21.15.43:0/1252837957 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5610711bea90 0x5610711bc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:27.845 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.844+0000 7faaf7400640 1 -- 172.21.15.43:0/1252837957 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5610711b53a0 msgr2=0x5610711bbb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:27.845 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.844+0000 7faaf7400640 1 --2- 172.21.15.43:0/1252837957 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5610711b53a0 0x5610711bbb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:27.845 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.844+0000 7faaf7400640 1 -- 172.21.15.43:0/1252837957 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5610711bea90 msgr2=0x5610711bc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:27.845 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.844+0000 7faaf7400640 1 --2- 172.21.15.43:0/1252837957 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5610711bea90 0x5610711bc590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:27.846 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.844+0000 7faaf7400640 1 -- 172.21.15.43:0/1252837957 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561070ef9620 con 0x5610711b5cc0 2024-03-28T22:41:27.846 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.844+0000 7faafca00640 1 --2- 172.21.15.43:0/1252837957 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5610711bea90 0x5610711bc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:41:27.846 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.844+0000 7faaf7400640 1 --2- 172.21.15.43:0/1252837957 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5610711b5cc0 0x5610711bc050 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7faae4011760 tx=0x7faae4011c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86b77b7b553bf848 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:27.846 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.844+0000 7faae1600640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:41:27.846 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.844+0000 7faae1600640 1 -- 172.21.15.43:0/1252837957 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae4004300 con 0x5610711b5cc0 2024-03-28T22:41:27.846 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.844+0000 7fab01832740 1 -- 172.21.15.43:0/1252837957 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faaf0004050 con 0x5610711b5cc0 2024-03-28T22:41:27.846 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.844+0000 7faae1600640 1 -- 172.21.15.43:0/1252837957 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7faae400f920 con 0x5610711b5cc0 2024-03-28T22:41:27.846 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.844+0000 7faae1600640 1 -- 172.21.15.43:0/1252837957 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7faae4025050 con 0x5610711b5cc0 2024-03-28T22:41:27.846 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.845+0000 7fab01832740 10 client.27233 fetch_fsmap learned FSMap version 32 2024-03-28T22:41:27.846 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.845+0000 7fab01832740 1 -- 172.21.15.43:0/1252837957 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5610711cd6d0 con 0x5610711b5cc0 2024-03-28T22:41:27.846 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.845+0000 7faae1600640 1 -- 172.21.15.43:0/1252837957 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faaf0004050 con 0x5610711b5cc0 2024-03-28T22:41:27.846 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.845+0000 7faae1600640 1 -- 172.21.15.43:0/1252837957 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7faae4019030 con 0x5610711b5cc0 2024-03-28T22:41:27.846 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.845+0000 7fab01832740 10 client.27233 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:41:27.846 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.845+0000 7fab01832740 10 client.27233 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:41:27.847 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.845+0000 7fab01832740 20 client.27233 populate_metadata read hostname 'smithi043' 2024-03-28T22:41:27.847 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.845+0000 7fab01832740 1 --2- 172.21.15.43:0/1252837957 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5610711cebe0 0x5610711d0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:27.847 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.845+0000 7fab01832740 4 client.27233 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:41:27.847 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.845+0000 7fab01832740 1 -- 172.21.15.43:0/1252837957 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56107112c030 con 0x5610711cebe0 2024-03-28T22:41:27.847 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.846+0000 7faafca00640 1 --2- 172.21.15.43:0/1252837957 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5610711cebe0 0x5610711d0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:27.847 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.846+0000 7faafca00640 1 --2- 172.21.15.43:0/1252837957 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5610711cebe0 0x5610711d0fd0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87806c6ee29e7b9f server_cookie=1a1bcffac777ebfd in_seq=0 out_seq=0 2024-03-28T22:41:27.847 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.846+0000 7faae1600640 10 client.27233 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:27.848 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.847+0000 7faae1600640 1 -- 172.21.15.43:0/1252837957 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56107112c030 con 0x5610711cebe0 2024-03-28T22:41:27.848 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.847+0000 7faae1600640 10 client.27233 handle_command_reply: tid=0 2024-03-28T22:41:27.848 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.847+0000 7fab01832740 1 client.27233 shutdown 2024-03-28T22:41:27.848 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.847+0000 7fab01832740 1 -- 172.21.15.43:0/1252837957 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5610711b5cc0 msgr2=0x5610711bc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:27.848 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.847+0000 7fab01832740 1 --2- 172.21.15.43:0/1252837957 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5610711b5cc0 0x5610711bc050 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7faae4011760 tx=0x7faae4011c20 comp rx=0 tx=0).stop 2024-03-28T22:41:27.848 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.848+0000 7fab01832740 1 -- 172.21.15.43:0/1252837957 shutdown_connections 2024-03-28T22:41:27.848 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.848+0000 7fab01832740 1 --2- 172.21.15.43:0/1252837957 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5610711cebe0 0x5610711d0fd0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:27.849 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.848+0000 7fab01832740 1 --2- 172.21.15.43:0/1252837957 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5610711b5cc0 0x5610711bc050 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:27.849 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.848+0000 7fab01832740 1 --2- 172.21.15.43:0/1252837957 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5610711bea90 0x5610711bc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:27.849 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.848+0000 7fab01832740 1 --2- 172.21.15.43:0/1252837957 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5610711b53a0 0x5610711bbb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:27.849 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.848+0000 7faae1600640 0 client.27233 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:27.849 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.848+0000 7fab01832740 1 -- 172.21.15.43:0/1252837957 >> 172.21.15.43:0/1252837957 conn(0x5610711afcc0 msgr2=0x5610711b1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:27.849 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.848+0000 7fab01832740 1 -- 172.21.15.43:0/1252837957 shutdown_connections 2024-03-28T22:41:27.849 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.848+0000 7fab01832740 1 -- 172.21.15.43:0/1252837957 wait complete. 2024-03-28T22:41:27.849 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.848+0000 7fab01832740 20 client.27233 trim_cache size 0 max 16384 2024-03-28T22:41:27.864 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.866+0000 7fab01832740 1 Processor -- start 2024-03-28T22:41:27.864 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.866+0000 7fab01832740 1 -- start start 2024-03-28T22:41:27.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.867+0000 7fab01832740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5610711b53a0 0x5610711be240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:27.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.867+0000 7fab01832740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5610711b5cc0 0x5610711ca460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:27.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.867+0000 7fab01832740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5610711bea90 0x5610711d68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:27.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.867+0000 7fab01832740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7faaf81b5140 con 0x5610711bea90 2024-03-28T22:41:27.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.867+0000 7fab01832740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7faaf81b6c80 con 0x5610711b53a0 2024-03-28T22:41:27.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.867+0000 7fab01832740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7faaf81b5810 con 0x5610711b5cc0 2024-03-28T22:41:27.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.867+0000 7faafca00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5610711b53a0 0x5610711be240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:27.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.867+0000 7faaf7400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5610711b5cc0 0x5610711ca460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:27.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.867+0000 7faafca00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5610711b53a0 0x5610711be240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33562/0 (socket says 172.21.15.43:33562) 2024-03-28T22:41:27.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.867+0000 7faafca00640 1 -- 172.21.15.43:0/766016619 learned_addr learned my addr 172.21.15.43:0/766016619 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:27.866 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.867+0000 7faafca00640 1 --2- 172.21.15.43:0/766016619 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5610711bea90 0x5610711d68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.868+0000 7faafca00640 1 -- 172.21.15.43:0/766016619 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5610711b5cc0 msgr2=0x5610711ca460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.868+0000 7faafca00640 1 --2- 172.21.15.43:0/766016619 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5610711b5cc0 0x5610711ca460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.868+0000 7faafca00640 1 -- 172.21.15.43:0/766016619 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5610711bea90 msgr2=0x5610711d68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.868+0000 7faafca00640 1 --2- 172.21.15.43:0/766016619 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5610711bea90 0x5610711d68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.868+0000 7faafca00640 1 -- 172.21.15.43:0/766016619 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610711b4b60 con 0x5610711b53a0 2024-03-28T22:41:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.868+0000 7faaf7400640 1 --2- 172.21.15.43:0/766016619 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5610711b5cc0 0x5610711ca460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:41:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.868+0000 7faafca00640 1 --2- 172.21.15.43:0/766016619 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5610711b53a0 0x5610711be240 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7faaf0009f20 tx=0x7faaf0005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=670893c9ac1c86ad server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.868+0000 7faae3e00640 1 -- 172.21.15.43:0/766016619 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf002dc30 con 0x5610711b53a0 2024-03-28T22:41:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.868+0000 7faae3e00640 1 -- 172.21.15.43:0/766016619 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faaf0003e50 con 0x5610711b53a0 2024-03-28T22:41:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.868+0000 7faae3e00640 1 -- 172.21.15.43:0/766016619 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf002d1c0 con 0x5610711b53a0 2024-03-28T22:41:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.869+0000 7fab01832740 1 -- 172.21.15.43:0/766016619 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5610711b53a0 msgr2=0x5610711be240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.869+0000 7fab01832740 1 --2- 172.21.15.43:0/766016619 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5610711b53a0 0x5610711be240 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7faaf0009f20 tx=0x7faaf0005fd0 comp rx=0 tx=0).stop 2024-03-28T22:41:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.869+0000 7fab01832740 1 -- 172.21.15.43:0/766016619 shutdown_connections 2024-03-28T22:41:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.869+0000 7fab01832740 1 --2- 172.21.15.43:0/766016619 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5610711bea90 0x5610711d68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.869+0000 7fab01832740 1 --2- 172.21.15.43:0/766016619 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5610711b5cc0 0x5610711ca460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.869+0000 7fab01832740 1 --2- 172.21.15.43:0/766016619 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5610711b53a0 0x5610711be240 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.869+0000 7fab01832740 1 -- 172.21.15.43:0/766016619 >> 172.21.15.43:0/766016619 conn(0x5610711afcc0 msgr2=0x5610711b00a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.869+0000 7fab01832740 1 -- 172.21.15.43:0/766016619 shutdown_connections 2024-03-28T22:41:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.869+0000 7fab01832740 1 -- 172.21.15.43:0/766016619 wait complete. 2024-03-28T22:41:27.871 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.870+0000 7fab01832740 1 -- start start 2024-03-28T22:41:27.871 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.870+0000 7fab01832740 1 Processor -- start 2024-03-28T22:41:27.871 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.871+0000 7fab01832740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5610711b53a0 0x5610711b8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:27.871 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.871+0000 7fab01832740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5610711b5cc0 0x5610711b8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:27.871 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.871+0000 7fab01832740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5610711bea90 0x561071233f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:27.871 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.871+0000 7fab01832740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7faaf81b65b0 con 0x5610711b5cc0 2024-03-28T22:41:27.871 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.871+0000 7fab01832740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7faaf81b5ee0 con 0x5610711bea90 2024-03-28T22:41:27.871 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.871+0000 7fab01832740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7faaf81b3120 con 0x5610711b53a0 2024-03-28T22:41:27.871 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.871+0000 7fab01832740 10 client.0 _ll_register_callbacks cb 0x7faae40098f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:41:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.871+0000 7faafca00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5610711b53a0 0x5610711b8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.871+0000 7faafca00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5610711b53a0 0x5610711b8a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:50174/0 (socket says 172.21.15.43:50174) 2024-03-28T22:41:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.871+0000 7faafca00640 1 -- 172.21.15.43:0/3640058591 learned_addr learned my addr 172.21.15.43:0/3640058591 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.871+0000 7faafca00640 1 --2- 172.21.15.43:0/3640058591 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5610711bea90 0x561071233f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.871+0000 7faaf7400640 1 --2- 172.21.15.43:0/3640058591 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5610711b5cc0 0x5610711b8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.871+0000 7faafca00640 1 -- 172.21.15.43:0/3640058591 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5610711bea90 msgr2=0x561071233f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.871+0000 7faafca00640 1 --2- 172.21.15.43:0/3640058591 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5610711bea90 0x561071233f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.871+0000 7faafca00640 1 -- 172.21.15.43:0/3640058591 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5610711b5cc0 msgr2=0x5610711b8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.871+0000 7faafca00640 1 --2- 172.21.15.43:0/3640058591 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5610711b5cc0 0x5610711b8fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.872+0000 7faafca00640 1 -- 172.21.15.43:0/3640058591 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561071234450 con 0x5610711b53a0 2024-03-28T22:41:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.872+0000 7faafca00640 1 --2- 172.21.15.43:0/3640058591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5610711b53a0 0x5610711b8a90 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7faaf0014020 tx=0x7faaf0028040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2302a603f9b8b05e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.872+0000 7faae1600640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:41:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.872+0000 7faae1600640 1 -- 172.21.15.43:0/3640058591 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf00044b0 con 0x5610711b53a0 2024-03-28T22:41:27.872 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.872+0000 7fab01832740 1 -- 172.21.15.43:0/3640058591 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faaf0003e50 con 0x5610711b53a0 2024-03-28T22:41:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.872+0000 7faae1600640 1 -- 172.21.15.43:0/3640058591 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7faaf0003970 con 0x5610711b53a0 2024-03-28T22:41:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.872+0000 7faae1600640 1 -- 172.21.15.43:0/3640058591 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7faaf0003b70 con 0x5610711b53a0 2024-03-28T22:41:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.872+0000 7fab01832740 10 client.27193 fetch_fsmap learned FSMap version 32 2024-03-28T22:41:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.873+0000 7fab01832740 1 -- 172.21.15.43:0/3640058591 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5610712355a0 con 0x5610711b53a0 2024-03-28T22:41:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.873+0000 7faae1600640 1 -- 172.21.15.43:0/3640058591 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faaf0003e50 con 0x5610711b53a0 2024-03-28T22:41:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.873+0000 7faae1600640 1 -- 172.21.15.43:0/3640058591 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7faaf0026430 con 0x5610711b53a0 2024-03-28T22:41:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.873+0000 7fab01832740 10 client.27193 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:41:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.873+0000 7fab01832740 10 client.27193 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:41:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.873+0000 7fab01832740 20 client.27193 populate_metadata read hostname 'smithi043' 2024-03-28T22:41:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.873+0000 7fab01832740 1 --2- 172.21.15.43:0/3640058591 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x561071235a10 0x561071237e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.873+0000 7fab01832740 4 client.27193 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T22:41:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.873+0000 7fab01832740 1 -- 172.21.15.43:0/3640058591 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561071125b80 con 0x561071235a10 2024-03-28T22:41:27.873 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.873+0000 7faaf7400640 1 --2- 172.21.15.43:0/3640058591 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x561071235a10 0x561071237e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:27.874 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.874+0000 7faaf7400640 1 --2- 172.21.15.43:0/3640058591 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x561071235a10 0x561071237e00 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c54844a102059464 server_cookie=ced8d0c23d8302bd in_seq=0 out_seq=0 2024-03-28T22:41:27.874 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:27.874+0000 7faae1600640 10 client.27193 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:27.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:27 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:41:27.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:27 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:41:27.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:27 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:41:27.948 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:27 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:41:27.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:27 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:41:27.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:27 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:41:28.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:27 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:41:28.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:27 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:41:28.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:27 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:41:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:28.098+0000 7faae1600640 1 -- 172.21.15.43:0/3640058591 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561071125b80 con 0x561071235a10 2024-03-28T22:41:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:28.098+0000 7faae1600640 10 client.27193 handle_command_reply: tid=0 2024-03-28T22:41:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:28.098+0000 7fab01832740 1 client.27193 shutdown 2024-03-28T22:41:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:28.098+0000 7fab01832740 1 -- 172.21.15.43:0/3640058591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5610711b53a0 msgr2=0x5610711b8a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:28.098+0000 7fab01832740 1 --2- 172.21.15.43:0/3640058591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5610711b53a0 0x5610711b8a90 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7faaf0014020 tx=0x7faaf0028040 comp rx=0 tx=0).stop 2024-03-28T22:41:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:28.098+0000 7fab01832740 1 -- 172.21.15.43:0/3640058591 shutdown_connections 2024-03-28T22:41:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:28.098+0000 7fab01832740 1 --2- 172.21.15.43:0/3640058591 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x561071235a10 0x561071237e00 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:28.098+0000 7fab01832740 1 --2- 172.21.15.43:0/3640058591 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5610711bea90 0x561071233f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:28.098+0000 7fab01832740 1 --2- 172.21.15.43:0/3640058591 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5610711b5cc0 0x5610711b8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:28.098+0000 7fab01832740 1 --2- 172.21.15.43:0/3640058591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5610711b53a0 0x5610711b8a90 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:28.098+0000 7faae1600640 0 client.27193 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:28.098+0000 7fab01832740 1 -- 172.21.15.43:0/3640058591 >> 172.21.15.43:0/3640058591 conn(0x5610711afcc0 msgr2=0x5610711ae3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:28.098+0000 7fab01832740 1 -- 172.21.15.43:0/3640058591 shutdown_connections 2024-03-28T22:41:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:28.098+0000 7fab01832740 1 -- 172.21.15.43:0/3640058591 wait complete. 2024-03-28T22:41:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:28.098+0000 7fab01832740 20 client.27193 trim_cache size 0 max 16384 2024-03-28T22:41:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:{ 2024-03-28T22:41:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "message": "", 2024-03-28T22:41:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "return_code": 0 2024-03-28T22:41:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:} 2024-03-28T22:41:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:28.099+0000 7faafea00640 1 -- 172.21.15.43:0/1349101750 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7faadc0a2010 msgr2=0x7faadc0a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:28.099+0000 7faafea00640 1 --2- 172.21.15.43:0/1349101750 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7faadc0a2010 0x7faadc0a44d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x5610711b4e50 tx=0x7faae80f3620 comp rx=0 tx=0).stop 2024-03-28T22:41:28.099 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:28.099+0000 7faafea00640 1 -- 172.21.15.43:0/1349101750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faaf8107f90 msgr2=0x7faaf819e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:28.099 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:28.099+0000 7faafea00640 1 --2- 172.21.15.43:0/1349101750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faaf8107f90 0x7faaf819e570 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7faae802c0b0 tx=0x7faae80045b0 comp rx=0 tx=0).stop 2024-03-28T22:41:28.099 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:28.099+0000 7faafea00640 1 -- 172.21.15.43:0/1349101750 shutdown_connections 2024-03-28T22:41:28.099 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:28.099+0000 7faafea00640 1 --2- 172.21.15.43:0/1349101750 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7faadc0a2010 0x7faadc0a44d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:28.099 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:28.099+0000 7faafea00640 1 --2- 172.21.15.43:0/1349101750 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faaf810d060 0x7faaf819eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:28.099 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:28.099+0000 7faafea00640 1 --2- 172.21.15.43:0/1349101750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faaf81a2e30 0x7faaf81a32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:28.099 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:28.099+0000 7faafea00640 1 --2- 172.21.15.43:0/1349101750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faaf8107f90 0x7faaf819e570 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:28.099 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:28.099+0000 7faafea00640 1 -- 172.21.15.43:0/1349101750 >> 172.21.15.43:0/1349101750 conn(0x7faaf8075cc0 msgr2=0x7faaf8077f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:28.099 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:28.100+0000 7faafea00640 1 -- 172.21.15.43:0/1349101750 shutdown_connections 2024-03-28T22:41:28.099 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:28.100+0000 7faafea00640 1 -- 172.21.15.43:0/1349101750 wait complete. 2024-03-28T22:41:28.359 INFO:tasks.workunit.client.0.smithi043.stdout: 1 273396 8.06 MB/sec execute 59 sec latency 94.179 ms 2024-03-28T22:41:28.948 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:28 smithi082 ceph-mon[36471]: pgmap v133: 129 pgs: 129 active+clean; 50 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 345 op/s 2024-03-28T22:41:29.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:28 smithi043 ceph-mon[31823]: pgmap v133: 129 pgs: 129 active+clean; 50 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 345 op/s 2024-03-28T22:41:29.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:28 smithi172 ceph-mon[36566]: pgmap v133: 129 pgs: 129 active+clean; 50 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 345 op/s 2024-03-28T22:41:29.359 INFO:tasks.workunit.client.0.smithi043.stdout: 1 274144 7.95 MB/sec execute 60 sec latency 41.410 ms 2024-03-28T22:41:30.359 INFO:tasks.workunit.client.0.smithi043.stdout: 1 276171 8.01 MB/sec execute 61 sec latency 15.926 ms 2024-03-28T22:41:30.948 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:30 smithi082 ceph-mon[36471]: pgmap v134: 129 pgs: 129 active+clean; 50 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 158 op/s 2024-03-28T22:41:31.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:30 smithi043 ceph-mon[31823]: pgmap v134: 129 pgs: 129 active+clean; 50 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 158 op/s 2024-03-28T22:41:31.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:30 smithi172 ceph-mon[36566]: pgmap v134: 129 pgs: 129 active+clean; 50 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 158 op/s 2024-03-28T22:41:31.361 INFO:tasks.workunit.client.0.smithi043.stdout: 1 277350 7.99 MB/sec execute 62 sec latency 15.406 ms 2024-03-28T22:41:31.904 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:41:31.904 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:41:32.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.164+0000 7fc846000640 1 -- 172.21.15.43:0/2748335898 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8401076f0 msgr2=0x7fc840107ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:32.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.164+0000 7fc846000640 1 --2- 172.21.15.43:0/2748335898 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8401076f0 0x7fc840107ad0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fc830009900 tx=0x7fc83002bc30 comp rx=0 tx=0).stop 2024-03-28T22:41:32.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.164+0000 7fc846000640 1 -- 172.21.15.43:0/2748335898 shutdown_connections 2024-03-28T22:41:32.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.164+0000 7fc846000640 1 --2- 172.21.15.43:0/2748335898 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8401020a0 0x7fc84010e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.164+0000 7fc846000640 1 --2- 172.21.15.43:0/2748335898 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8401016e0 0x7fc840101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.164+0000 7fc846000640 1 --2- 172.21.15.43:0/2748335898 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8401076f0 0x7fc840107ad0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.164+0000 7fc846000640 1 -- 172.21.15.43:0/2748335898 >> 172.21.15.43:0/2748335898 conn(0x7fc840078b50 msgr2=0x7fc840078f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:32.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.164+0000 7fc846000640 1 -- 172.21.15.43:0/2748335898 shutdown_connections 2024-03-28T22:41:32.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.164+0000 7fc846000640 1 -- 172.21.15.43:0/2748335898 wait complete. 2024-03-28T22:41:32.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.165+0000 7fc846000640 1 Processor -- start 2024-03-28T22:41:32.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.165+0000 7fc846000640 1 -- start start 2024-03-28T22:41:32.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.166+0000 7fc846000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8401016e0 0x7fc84010b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:32.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.166+0000 7fc846000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8401020a0 0x7fc84010bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:32.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.166+0000 7fc846000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8401076f0 0x7fc84010c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:32.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.166+0000 7fc846000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc840075600 con 0x7fc8401020a0 2024-03-28T22:41:32.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.166+0000 7fc846000640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fc840075480 con 0x7fc8401016e0 2024-03-28T22:41:32.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.166+0000 7fc846000640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fc840075780 con 0x7fc8401076f0 2024-03-28T22:41:32.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.166+0000 7fc83f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8401016e0 0x7fc84010b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:32.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.166+0000 7fc83ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8401020a0 0x7fc84010bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:32.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.166+0000 7fc83f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8401016e0 0x7fc84010b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33596/0 (socket says 172.21.15.43:33596) 2024-03-28T22:41:32.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.166+0000 7fc83f400640 1 -- 172.21.15.43:0/2705510677 learned_addr learned my addr 172.21.15.43:0/2705510677 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:32.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.166+0000 7fc83fe00640 1 --2- 172.21.15.43:0/2705510677 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8401076f0 0x7fc84010c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:32.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.166+0000 7fc83f400640 1 -- 172.21.15.43:0/2705510677 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8401076f0 msgr2=0x7fc84010c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:32.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.166+0000 7fc83f400640 1 --2- 172.21.15.43:0/2705510677 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8401076f0 0x7fc84010c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.166+0000 7fc83f400640 1 -- 172.21.15.43:0/2705510677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8401020a0 msgr2=0x7fc84010bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:32.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.166+0000 7fc83f400640 1 --2- 172.21.15.43:0/2705510677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8401020a0 0x7fc84010bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.166+0000 7fc83f400640 1 -- 172.21.15.43:0/2705510677 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8401aa850 con 0x7fc8401016e0 2024-03-28T22:41:32.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.167+0000 7fc83f400640 1 --2- 172.21.15.43:0/2705510677 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8401016e0 0x7fc84010b6f0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fc83002efd0 tx=0x7fc8300043b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:32.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.167+0000 7fc837e00640 1 -- 172.21.15.43:0/2705510677 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc830041070 con 0x7fc8401016e0 2024-03-28T22:41:32.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.167+0000 7fc837e00640 1 -- 172.21.15.43:0/2705510677 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8300044a0 con 0x7fc8401016e0 2024-03-28T22:41:32.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.167+0000 7fc837e00640 1 -- 172.21.15.43:0/2705510677 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc830003ca0 con 0x7fc8401016e0 2024-03-28T22:41:32.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.167+0000 7fc846000640 1 -- 172.21.15.43:0/2705510677 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8401aaae0 con 0x7fc8401016e0 2024-03-28T22:41:32.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.167+0000 7fc846000640 1 -- 172.21.15.43:0/2705510677 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8401ab020 con 0x7fc8401016e0 2024-03-28T22:41:32.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.170+0000 7fc848d95740 1 Processor -- start 2024-03-28T22:41:32.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.171+0000 7fc848d95740 1 -- start start 2024-03-28T22:41:32.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.171+0000 7fc837e00640 1 -- 172.21.15.43:0/2705510677 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fc830044070 con 0x7fc8401016e0 2024-03-28T22:41:32.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.171+0000 7fc848d95740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55627fd90bc0 0x55627fd8f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:32.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.171+0000 7fc848d95740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55627fd8fee0 0x55627fd90360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:32.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.171+0000 7fc848d95740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55627fd99250 0x55627fd9b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:32.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.171+0000 7fc848d95740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55627fcf4100 con 0x55627fd90bc0 2024-03-28T22:41:32.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.171+0000 7fc848d95740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55627fc9cba0 con 0x55627fd99250 2024-03-28T22:41:32.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.171+0000 7fc848d95740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55627fcf01a0 con 0x55627fd8fee0 2024-03-28T22:41:32.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.171+0000 7fc83ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55627fd99250 0x55627fd9b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:32.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.171+0000 7fc83f400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55627fd8fee0 0x55627fd90360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:32.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.171+0000 7fc83ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55627fd99250 0x55627fd9b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33610/0 (socket says 172.21.15.43:33610) 2024-03-28T22:41:32.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.171+0000 7fc83ea00640 1 -- 172.21.15.43:0/1314976437 learned_addr learned my addr 172.21.15.43:0/1314976437 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:32.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.172+0000 7fc83ea00640 1 -- 172.21.15.43:0/1314976437 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55627fd8fee0 msgr2=0x55627fd90360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:32.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.172+0000 7fc83ea00640 1 --2- 172.21.15.43:0/1314976437 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55627fd8fee0 0x55627fd90360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.172+0000 7fc83ea00640 1 -- 172.21.15.43:0/1314976437 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55627fd90bc0 msgr2=0x55627fd8f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:32.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.172+0000 7fc83ea00640 1 --2- 172.21.15.43:0/1314976437 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55627fd90bc0 0x55627fd8f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.172+0000 7fc83ea00640 1 -- 172.21.15.43:0/1314976437 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55627fccd1a0 con 0x55627fd99250 2024-03-28T22:41:32.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.172+0000 7fc83f400640 1 --2- 172.21.15.43:0/1314976437 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55627fd8fee0 0x55627fd90360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:41:32.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.172+0000 7fc83ea00640 1 --2- 172.21.15.43:0/1314976437 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55627fd99250 0x55627fd9b640 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fc82c009c60 tx=0x7fc82c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec0183df7eef4153 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:32.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.173+0000 7fc837e00640 1 --2- 172.21.15.43:0/2705510677 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc8240a1fb0 0x7fc8240a4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:32.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.173+0000 7fc82be00640 1 -- 172.21.15.43:0/1314976437 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82c004230 con 0x55627fd99250 2024-03-28T22:41:32.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.173+0000 7fc82be00640 1 -- 172.21.15.43:0/1314976437 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc82c0043d0 con 0x55627fd99250 2024-03-28T22:41:32.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.173+0000 7fc82be00640 1 -- 172.21.15.43:0/1314976437 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82c004db0 con 0x55627fd99250 2024-03-28T22:41:32.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.173+0000 7fc837e00640 1 -- 172.21.15.43:0/2705510677 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fc8300efd60 con 0x7fc8401016e0 2024-03-28T22:41:32.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.173+0000 7fc83fe00640 1 --2- 172.21.15.43:0/2705510677 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc8240a1fb0 0x7fc8240a4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:32.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.174+0000 7fc848d95740 1 -- 172.21.15.43:0/1314976437 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55627fd99250 msgr2=0x55627fd9b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:32.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.174+0000 7fc848d95740 1 --2- 172.21.15.43:0/1314976437 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55627fd99250 0x55627fd9b640 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fc82c009c60 tx=0x7fc82c011810 comp rx=0 tx=0).stop 2024-03-28T22:41:32.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.174+0000 7fc848d95740 1 -- 172.21.15.43:0/1314976437 shutdown_connections 2024-03-28T22:41:32.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.174+0000 7fc83fe00640 1 --2- 172.21.15.43:0/2705510677 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc8240a1fb0 0x7fc8240a4470 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc838005fd0 tx=0x7fc838009a90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:32.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.174+0000 7fc848d95740 1 --2- 172.21.15.43:0/1314976437 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55627fd99250 0x55627fd9b640 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.174+0000 7fc848d95740 1 --2- 172.21.15.43:0/1314976437 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55627fd8fee0 0x55627fd90360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.174+0000 7fc848d95740 1 --2- 172.21.15.43:0/1314976437 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55627fd90bc0 0x55627fd8f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.174+0000 7fc848d95740 1 -- 172.21.15.43:0/1314976437 >> 172.21.15.43:0/1314976437 conn(0x55627fd56a10 msgr2=0x55627fd8a930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:32.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.174+0000 7fc848d95740 1 -- 172.21.15.43:0/1314976437 shutdown_connections 2024-03-28T22:41:32.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.174+0000 7fc848d95740 1 -- 172.21.15.43:0/1314976437 wait complete. 2024-03-28T22:41:32.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.174+0000 7fc848d95740 1 -- start start 2024-03-28T22:41:32.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.175+0000 7fc848d95740 1 Processor -- start 2024-03-28T22:41:32.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.175+0000 7fc848d95740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55627fd8fee0 0x55627fd961e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:32.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.175+0000 7fc848d95740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55627fd90bc0 0x55627fd96720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:32.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.175+0000 7fc848d95740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55627fd99250 0x55627fd96c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:32.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.175+0000 7fc848d95740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55627fd9c0c0 con 0x55627fd99250 2024-03-28T22:41:32.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.175+0000 7fc848d95740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55627fc92250 con 0x55627fd8fee0 2024-03-28T22:41:32.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.175+0000 7fc848d95740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55627fd9c240 con 0x55627fd90bc0 2024-03-28T22:41:32.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.175+0000 7fc848d95740 10 client.0 _ll_register_callbacks cb 0x55627fa07be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:41:32.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.175+0000 7fc83ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55627fd90bc0 0x55627fd96720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:32.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.175+0000 7fc83f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55627fd8fee0 0x55627fd961e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:32.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.175+0000 7fc83ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55627fd90bc0 0x55627fd96720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:50198/0 (socket says 172.21.15.43:50198) 2024-03-28T22:41:32.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.175+0000 7fc83ea00640 1 -- 172.21.15.43:0/3822064128 learned_addr learned my addr 172.21.15.43:0/3822064128 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:32.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.176+0000 7fc83fe00640 1 --2- 172.21.15.43:0/3822064128 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55627fd99250 0x55627fd96c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:32.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.176+0000 7fc83f400640 1 -- 172.21.15.43:0/3822064128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55627fd90bc0 msgr2=0x55627fd96720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:32.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.176+0000 7fc83f400640 1 --2- 172.21.15.43:0/3822064128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55627fd90bc0 0x55627fd96720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.176+0000 7fc83f400640 1 -- 172.21.15.43:0/3822064128 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55627fd99250 msgr2=0x55627fd96c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:32.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.176+0000 7fc83f400640 1 --2- 172.21.15.43:0/3822064128 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55627fd99250 0x55627fd96c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.176+0000 7fc83f400640 1 -- 172.21.15.43:0/3822064128 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55627fcae960 con 0x55627fd8fee0 2024-03-28T22:41:32.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.176+0000 7fc83f400640 1 --2- 172.21.15.43:0/3822064128 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55627fd8fee0 0x55627fd961e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fc8300ed220 tx=0x7fc83002bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=697d608da6a805da server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:32.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.176+0000 7fc828c00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:41:32.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.176+0000 7fc828c00640 1 -- 172.21.15.43:0/3822064128 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83004b070 con 0x55627fd8fee0 2024-03-28T22:41:32.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.176+0000 7fc828c00640 1 -- 172.21.15.43:0/3822064128 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fc8300b8be0 con 0x55627fd8fee0 2024-03-28T22:41:32.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.176+0000 7fc848d95740 1 -- 172.21.15.43:0/3822064128 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc82c0043d0 con 0x55627fd8fee0 2024-03-28T22:41:32.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.177+0000 7fc828c00640 1 -- 172.21.15.43:0/3822064128 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc8300b8810 con 0x55627fd8fee0 2024-03-28T22:41:32.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.177+0000 7fc848d95740 10 client.27251 fetch_fsmap learned FSMap version 32 2024-03-28T22:41:32.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.177+0000 7fc848d95740 1 -- 172.21.15.43:0/3822064128 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55627fad27d0 con 0x55627fd8fee0 2024-03-28T22:41:32.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.177+0000 7fc828c00640 1 -- 172.21.15.43:0/3822064128 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc82c0043d0 con 0x55627fd8fee0 2024-03-28T22:41:32.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.177+0000 7fc828c00640 1 -- 172.21.15.43:0/3822064128 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc8300c47d0 con 0x55627fd8fee0 2024-03-28T22:41:32.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.177+0000 7fc848d95740 10 client.27251 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:41:32.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.177+0000 7fc848d95740 10 client.27251 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:41:32.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.177+0000 7fc848d95740 20 client.27251 populate_metadata read hostname 'smithi043' 2024-03-28T22:41:32.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.177+0000 7fc848d95740 1 --2- 172.21.15.43:0/3822064128 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55627fda96a0 0x55627fdabb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:32.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.177+0000 7fc848d95740 4 client.27251 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:41:32.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.177+0000 7fc848d95740 1 -- 172.21.15.43:0/3822064128 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55627fd01640 con 0x55627fda96a0 2024-03-28T22:41:32.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.178+0000 7fc83ea00640 1 --2- 172.21.15.43:0/3822064128 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55627fda96a0 0x55627fdabb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:32.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.178+0000 7fc83ea00640 1 --2- 172.21.15.43:0/3822064128 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55627fda96a0 0x55627fdabb40 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10d0125fd58cc2a6 server_cookie=fc9b1a8a636e60c8 in_seq=0 out_seq=0 2024-03-28T22:41:32.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.178+0000 7fc828c00640 10 client.27251 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:32.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.179+0000 7fc828c00640 1 -- 172.21.15.43:0/3822064128 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55627fd01640 con 0x55627fda96a0 2024-03-28T22:41:32.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.179+0000 7fc828c00640 10 client.27251 handle_command_reply: tid=0 2024-03-28T22:41:32.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.179+0000 7fc848d95740 1 client.27251 shutdown 2024-03-28T22:41:32.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.180+0000 7fc848d95740 1 -- 172.21.15.43:0/3822064128 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55627fd8fee0 msgr2=0x55627fd961e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:32.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.180+0000 7fc848d95740 1 --2- 172.21.15.43:0/3822064128 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55627fd8fee0 0x55627fd961e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fc8300ed220 tx=0x7fc83002bc30 comp rx=0 tx=0).stop 2024-03-28T22:41:32.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.180+0000 7fc848d95740 1 -- 172.21.15.43:0/3822064128 shutdown_connections 2024-03-28T22:41:32.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.180+0000 7fc848d95740 1 --2- 172.21.15.43:0/3822064128 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55627fd99250 0x55627fd96c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.180+0000 7fc848d95740 1 --2- 172.21.15.43:0/3822064128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55627fd90bc0 0x55627fd96720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.180+0000 7fc848d95740 1 --2- 172.21.15.43:0/3822064128 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55627fda96a0 0x55627fdabb40 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.180+0000 7fc848d95740 1 --2- 172.21.15.43:0/3822064128 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55627fd8fee0 0x55627fd961e0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.180+0000 7fc848d95740 1 -- 172.21.15.43:0/3822064128 >> 172.21.15.43:0/3822064128 conn(0x55627fd56a10 msgr2=0x55627fd8c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:32.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.180+0000 7fc828c00640 0 client.27251 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:32.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.180+0000 7fc848d95740 1 -- 172.21.15.43:0/3822064128 shutdown_connections 2024-03-28T22:41:32.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.180+0000 7fc848d95740 1 -- 172.21.15.43:0/3822064128 wait complete. 2024-03-28T22:41:32.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.180+0000 7fc848d95740 20 client.27251 trim_cache size 0 max 16384 2024-03-28T22:41:32.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.203+0000 7fc848d95740 1 Processor -- start 2024-03-28T22:41:32.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.203+0000 7fc848d95740 1 -- start start 2024-03-28T22:41:32.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.204+0000 7fc848d95740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55627fd8fee0 0x55627fd98910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:32.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.204+0000 7fc848d95740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55627fd90bc0 0x55627fd95120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:32.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.204+0000 7fc848d95740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55627fd99250 0x55627fd8a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:32.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.204+0000 7fc848d95740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc8401b51a0 con 0x55627fd8fee0 2024-03-28T22:41:32.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.204+0000 7fc848d95740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fc8401b6ce0 con 0x55627fd90bc0 2024-03-28T22:41:32.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.204+0000 7fc848d95740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fc8401b5870 con 0x55627fd99250 2024-03-28T22:41:32.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.204+0000 7fc83ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55627fd90bc0 0x55627fd95120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:32.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.204+0000 7fc83fe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55627fd99250 0x55627fd8a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:32.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.204+0000 7fc83ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55627fd90bc0 0x55627fd95120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33634/0 (socket says 172.21.15.43:33634) 2024-03-28T22:41:32.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.204+0000 7fc83ea00640 1 -- 172.21.15.43:0/1332949016 learned_addr learned my addr 172.21.15.43:0/1332949016 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:32.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.204+0000 7fc83f400640 1 --2- 172.21.15.43:0/1332949016 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55627fd8fee0 0x55627fd98910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:32.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.204+0000 7fc83ea00640 1 -- 172.21.15.43:0/1332949016 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55627fd99250 msgr2=0x55627fd8a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:32.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.204+0000 7fc83ea00640 1 --2- 172.21.15.43:0/1332949016 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55627fd99250 0x55627fd8a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.204+0000 7fc83ea00640 1 -- 172.21.15.43:0/1332949016 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55627fd8fee0 msgr2=0x55627fd98910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:32.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.204+0000 7fc83ea00640 1 --2- 172.21.15.43:0/1332949016 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55627fd8fee0 0x55627fd98910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.204+0000 7fc83ea00640 1 -- 172.21.15.43:0/1332949016 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55627fa083f0 con 0x55627fd90bc0 2024-03-28T22:41:32.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.205+0000 7fc83ea00640 1 --2- 172.21.15.43:0/1332949016 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55627fd90bc0 0x55627fd95120 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fc82c009c60 tx=0x7fc82c004600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fe80542b3c39338 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:32.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.205+0000 7fc82b400640 1 -- 172.21.15.43:0/1332949016 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82c030070 con 0x55627fd90bc0 2024-03-28T22:41:32.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.205+0000 7fc82b400640 1 -- 172.21.15.43:0/1332949016 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc82c01f040 con 0x55627fd90bc0 2024-03-28T22:41:32.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.205+0000 7fc82b400640 1 -- 172.21.15.43:0/1332949016 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82c01e040 con 0x55627fd90bc0 2024-03-28T22:41:32.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.206+0000 7fc848d95740 1 -- 172.21.15.43:0/1332949016 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55627fd90bc0 msgr2=0x55627fd95120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:32.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.206+0000 7fc848d95740 1 --2- 172.21.15.43:0/1332949016 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55627fd90bc0 0x55627fd95120 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fc82c009c60 tx=0x7fc82c004600 comp rx=0 tx=0).stop 2024-03-28T22:41:32.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.206+0000 7fc848d95740 1 -- 172.21.15.43:0/1332949016 shutdown_connections 2024-03-28T22:41:32.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.206+0000 7fc848d95740 1 --2- 172.21.15.43:0/1332949016 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55627fd90bc0 0x55627fd95120 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.206+0000 7fc848d95740 1 --2- 172.21.15.43:0/1332949016 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55627fd99250 0x55627fd8a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.206+0000 7fc848d95740 1 --2- 172.21.15.43:0/1332949016 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55627fd8fee0 0x55627fd98910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.206+0000 7fc848d95740 1 -- 172.21.15.43:0/1332949016 >> 172.21.15.43:0/1332949016 conn(0x55627fd56a10 msgr2=0x55627fd8c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.206+0000 7fc848d95740 1 -- 172.21.15.43:0/1332949016 shutdown_connections 2024-03-28T22:41:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.206+0000 7fc848d95740 1 -- 172.21.15.43:0/1332949016 wait complete. 2024-03-28T22:41:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.207+0000 7fc848d95740 1 -- start start 2024-03-28T22:41:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.207+0000 7fc848d95740 1 Processor -- start 2024-03-28T22:41:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.207+0000 7fc848d95740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55627fd8fee0 0x55627fda1530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.207+0000 7fc848d95740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55627fd90bc0 0x55627fd92450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.207+0000 7fc848d95740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55627fd99250 0x55627fd92990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.207+0000 7fc848d95740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc8401b6610 con 0x55627fd99250 2024-03-28T22:41:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.207+0000 7fc848d95740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fc8401b5f40 con 0x55627fd90bc0 2024-03-28T22:41:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.207+0000 7fc848d95740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fc8401b3180 con 0x55627fd8fee0 2024-03-28T22:41:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.207+0000 7fc848d95740 10 client.0 _ll_register_callbacks cb 0x7fc830005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:41:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.208+0000 7fc83f400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55627fd8fee0 0x55627fda1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.208+0000 7fc83f400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55627fd8fee0 0x55627fda1530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:50220/0 (socket says 172.21.15.43:50220) 2024-03-28T22:41:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.208+0000 7fc83f400640 1 -- 172.21.15.43:0/4088346300 learned_addr learned my addr 172.21.15.43:0/4088346300 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.208+0000 7fc83ea00640 1 --2- 172.21.15.43:0/4088346300 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55627fd90bc0 0x55627fd92450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.208+0000 7fc83fe00640 1 --2- 172.21.15.43:0/4088346300 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55627fd99250 0x55627fd92990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.208+0000 7fc83f400640 1 -- 172.21.15.43:0/4088346300 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55627fd90bc0 msgr2=0x55627fd92450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.208+0000 7fc83f400640 1 --2- 172.21.15.43:0/4088346300 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55627fd90bc0 0x55627fd92450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.208+0000 7fc83f400640 1 -- 172.21.15.43:0/4088346300 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55627fd99250 msgr2=0x55627fd92990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.208+0000 7fc83f400640 1 --2- 172.21.15.43:0/4088346300 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55627fd99250 0x55627fd92990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.208+0000 7fc83f400640 1 -- 172.21.15.43:0/4088346300 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55627f9c80f0 con 0x55627fd8fee0 2024-03-28T22:41:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.208+0000 7fc83ea00640 1 --2- 172.21.15.43:0/4088346300 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55627fd90bc0 0x55627fd92450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:41:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.208+0000 7fc83f400640 1 --2- 172.21.15.43:0/4088346300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55627fd8fee0 0x55627fda1530 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc8300bc730 tx=0x7fc8300baa80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=989b73c1292de284 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.208+0000 7fc828c00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:41:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.209+0000 7fc828c00640 1 -- 172.21.15.43:0/4088346300 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83004b070 con 0x55627fd8fee0 2024-03-28T22:41:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.209+0000 7fc848d95740 1 -- 172.21.15.43:0/4088346300 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc82c01f040 con 0x55627fd8fee0 2024-03-28T22:41:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.209+0000 7fc828c00640 1 -- 172.21.15.43:0/4088346300 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fc8300bc9d0 con 0x55627fd8fee0 2024-03-28T22:41:32.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.209+0000 7fc828c00640 1 -- 172.21.15.43:0/4088346300 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc8300bbed0 con 0x55627fd8fee0 2024-03-28T22:41:32.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.209+0000 7fc848d95740 10 client.27211 fetch_fsmap learned FSMap version 32 2024-03-28T22:41:32.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.209+0000 7fc848d95740 1 -- 172.21.15.43:0/4088346300 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55627fd93e70 con 0x55627fd8fee0 2024-03-28T22:41:32.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.209+0000 7fc828c00640 1 -- 172.21.15.43:0/4088346300 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc82c01f040 con 0x55627fd8fee0 2024-03-28T22:41:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.209+0000 7fc828c00640 1 -- 172.21.15.43:0/4088346300 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc8300f3bb0 con 0x55627fd8fee0 2024-03-28T22:41:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.209+0000 7fc848d95740 10 client.27211 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:41:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.209+0000 7fc848d95740 10 client.27211 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:41:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.209+0000 7fc848d95740 20 client.27211 populate_metadata read hostname 'smithi043' 2024-03-28T22:41:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.209+0000 7fc848d95740 1 --2- 172.21.15.43:0/4088346300 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55627fe10a50 0x55627fe12e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.209+0000 7fc848d95740 4 client.27211 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:41:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.209+0000 7fc848d95740 1 -- 172.21.15.43:0/4088346300 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55627fcfb990 con 0x55627fe10a50 2024-03-28T22:41:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.210+0000 7fc83ea00640 1 --2- 172.21.15.43:0/4088346300 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55627fe10a50 0x55627fe12e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.210+0000 7fc83ea00640 1 --2- 172.21.15.43:0/4088346300 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55627fe10a50 0x55627fe12e40 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f5632612ab2d67b server_cookie=112112d960351298 in_seq=0 out_seq=0 2024-03-28T22:41:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.210+0000 7fc828c00640 10 client.27211 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.211+0000 7fc828c00640 1 -- 172.21.15.43:0/4088346300 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55627fcfb990 con 0x55627fe10a50 2024-03-28T22:41:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.211+0000 7fc828c00640 10 client.27211 handle_command_reply: tid=0 2024-03-28T22:41:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.211+0000 7fc848d95740 1 client.27211 shutdown 2024-03-28T22:41:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.211+0000 7fc848d95740 1 -- 172.21.15.43:0/4088346300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55627fd8fee0 msgr2=0x55627fda1530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.211+0000 7fc848d95740 1 --2- 172.21.15.43:0/4088346300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55627fd8fee0 0x55627fda1530 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc8300bc730 tx=0x7fc8300baa80 comp rx=0 tx=0).stop 2024-03-28T22:41:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.211+0000 7fc848d95740 1 -- 172.21.15.43:0/4088346300 shutdown_connections 2024-03-28T22:41:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.211+0000 7fc848d95740 1 --2- 172.21.15.43:0/4088346300 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55627fe10a50 0x55627fe12e40 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.211+0000 7fc848d95740 1 --2- 172.21.15.43:0/4088346300 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55627fd90bc0 0x55627fd92450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.211+0000 7fc848d95740 1 --2- 172.21.15.43:0/4088346300 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55627fd99250 0x55627fd92990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.211+0000 7fc848d95740 1 --2- 172.21.15.43:0/4088346300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55627fd8fee0 0x55627fda1530 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.211+0000 7fc848d95740 1 -- 172.21.15.43:0/4088346300 >> 172.21.15.43:0/4088346300 conn(0x55627fd56a10 msgr2=0x55627fd961a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.211+0000 7fc828c00640 0 client.27211 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.211+0000 7fc848d95740 1 -- 172.21.15.43:0/4088346300 shutdown_connections 2024-03-28T22:41:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.211+0000 7fc848d95740 1 -- 172.21.15.43:0/4088346300 wait complete. 2024-03-28T22:41:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.211+0000 7fc848d95740 20 client.27211 trim_cache size 0 max 16384 2024-03-28T22:41:32.213 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:41:32.213 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-03-28T22:41:32.213 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-03-28T22:41:32.213 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:41:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.212+0000 7fc846000640 1 -- 172.21.15.43:0/2705510677 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc8240a1fb0 msgr2=0x7fc8240a4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.212+0000 7fc846000640 1 --2- 172.21.15.43:0/2705510677 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc8240a1fb0 0x7fc8240a4470 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc838005fd0 tx=0x7fc838009a90 comp rx=0 tx=0).stop 2024-03-28T22:41:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.212+0000 7fc846000640 1 -- 172.21.15.43:0/2705510677 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8401016e0 msgr2=0x7fc84010b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.212+0000 7fc846000640 1 --2- 172.21.15.43:0/2705510677 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8401016e0 0x7fc84010b6f0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fc83002efd0 tx=0x7fc8300043b0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.212+0000 7fc846000640 1 -- 172.21.15.43:0/2705510677 shutdown_connections 2024-03-28T22:41:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.212+0000 7fc846000640 1 --2- 172.21.15.43:0/2705510677 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc8240a1fb0 0x7fc8240a4470 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.212+0000 7fc846000640 1 --2- 172.21.15.43:0/2705510677 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8401076f0 0x7fc84010c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.212+0000 7fc846000640 1 --2- 172.21.15.43:0/2705510677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8401020a0 0x7fc84010bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.212+0000 7fc846000640 1 --2- 172.21.15.43:0/2705510677 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8401016e0 0x7fc84010b6f0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.212+0000 7fc846000640 1 -- 172.21.15.43:0/2705510677 >> 172.21.15.43:0/2705510677 conn(0x7fc840078b50 msgr2=0x7fc840103f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.212+0000 7fc846000640 1 -- 172.21.15.43:0/2705510677 shutdown_connections 2024-03-28T22:41:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.212+0000 7fc846000640 1 -- 172.21.15.43:0/2705510677 wait complete. 2024-03-28T22:41:32.227 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:41:32.227 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:41:32.359 INFO:tasks.workunit.client.0.smithi043.stdout: 1 279134 7.97 MB/sec execute 63 sec latency 14.402 ms 2024-03-28T22:41:32.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.440+0000 7fd292e00640 1 -- 172.21.15.43:0/3466915078 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd28c073b80 msgr2=0x7fd28c073f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:32.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.440+0000 7fd292e00640 1 --2- 172.21.15.43:0/3466915078 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd28c073b80 0x7fd28c073f60 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fd284009900 tx=0x7fd28402bcc0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.440+0000 7fd292e00640 1 -- 172.21.15.43:0/3466915078 shutdown_connections 2024-03-28T22:41:32.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.440+0000 7fd292e00640 1 --2- 172.21.15.43:0/3466915078 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd28c10dd80 0x7fd28c1101d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.440+0000 7fd292e00640 1 --2- 172.21.15.43:0/3466915078 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd28c0744a0 0x7fd28c10d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.440+0000 7fd292e00640 1 --2- 172.21.15.43:0/3466915078 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd28c073b80 0x7fd28c073f60 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.440+0000 7fd292e00640 1 -- 172.21.15.43:0/3466915078 >> 172.21.15.43:0/3466915078 conn(0x7fd28c0fd030 msgr2=0x7fd28c0ff450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:32.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.440+0000 7fd292e00640 1 -- 172.21.15.43:0/3466915078 shutdown_connections 2024-03-28T22:41:32.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.441+0000 7fd292e00640 1 -- 172.21.15.43:0/3466915078 wait complete. 2024-03-28T22:41:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.441+0000 7fd292e00640 1 Processor -- start 2024-03-28T22:41:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.442+0000 7fd292e00640 1 -- start start 2024-03-28T22:41:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.442+0000 7fd292e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd28c073b80 0x7fd28c19a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.442+0000 7fd292e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd28c0744a0 0x7fd28c19a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.442+0000 7fd292e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd28c10dd80 0x7fd28c19e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.442+0000 7fd292e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd28c113b70 con 0x7fd28c10dd80 2024-03-28T22:41:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.442+0000 7fd292e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd28c1139f0 con 0x7fd28c073b80 2024-03-28T22:41:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.442+0000 7fd292e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd28c113cf0 con 0x7fd28c0744a0 2024-03-28T22:41:32.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.442+0000 7fd291a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd28c073b80 0x7fd28c19a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:32.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.442+0000 7fd291a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd28c073b80 0x7fd28c19a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33664/0 (socket says 172.21.15.43:33664) 2024-03-28T22:41:32.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.442+0000 7fd291a00640 1 -- 172.21.15.43:0/956325153 learned_addr learned my addr 172.21.15.43:0/956325153 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:32.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.442+0000 7fd291000640 1 --2- 172.21.15.43:0/956325153 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd28c0744a0 0x7fd28c19a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:32.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.442+0000 7fd292400640 1 --2- 172.21.15.43:0/956325153 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd28c10dd80 0x7fd28c19e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:32.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.443+0000 7fd291a00640 1 -- 172.21.15.43:0/956325153 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd28c0744a0 msgr2=0x7fd28c19a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:32.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.443+0000 7fd291a00640 1 --2- 172.21.15.43:0/956325153 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd28c0744a0 0x7fd28c19a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.443+0000 7fd291a00640 1 -- 172.21.15.43:0/956325153 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd28c10dd80 msgr2=0x7fd28c19e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:32.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.443+0000 7fd291a00640 1 --2- 172.21.15.43:0/956325153 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd28c10dd80 0x7fd28c19e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.443+0000 7fd291a00640 1 -- 172.21.15.43:0/956325153 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd28c19efd0 con 0x7fd28c073b80 2024-03-28T22:41:32.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.443+0000 7fd292400640 1 --2- 172.21.15.43:0/956325153 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd28c10dd80 0x7fd28c19e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:41:32.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.443+0000 7fd291a00640 1 --2- 172.21.15.43:0/956325153 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd28c073b80 0x7fd28c19a030 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fd284000c00 tx=0x7fd284004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:32.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.443+0000 7fd28a000640 1 -- 172.21.15.43:0/956325153 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd284042020 con 0x7fd28c073b80 2024-03-28T22:41:32.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.443+0000 7fd28a000640 1 -- 172.21.15.43:0/956325153 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd284005000 con 0x7fd28c073b80 2024-03-28T22:41:32.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.443+0000 7fd292e00640 1 -- 172.21.15.43:0/956325153 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd28c19f260 con 0x7fd28c073b80 2024-03-28T22:41:32.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.443+0000 7fd28a000640 1 -- 172.21.15.43:0/956325153 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd284003b80 con 0x7fd28c073b80 2024-03-28T22:41:32.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.443+0000 7fd292e00640 1 -- 172.21.15.43:0/956325153 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd28c1a6b50 con 0x7fd28c073b80 2024-03-28T22:41:32.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.445+0000 7fd292e00640 1 -- 172.21.15.43:0/956325153 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd28c068460 con 0x7fd28c073b80 2024-03-28T22:41:32.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.447+0000 7fd28a000640 1 -- 172.21.15.43:0/956325153 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fd284045020 con 0x7fd28c073b80 2024-03-28T22:41:32.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.448+0000 7fd28a000640 1 --2- 172.21.15.43:0/956325153 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd2700a1ef0 0x7fd2700a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:32.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.448+0000 7fd28a000640 1 -- 172.21.15.43:0/956325153 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fd2840f00e0 con 0x7fd28c073b80 2024-03-28T22:41:32.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.449+0000 7fd291000640 1 --2- 172.21.15.43:0/956325153 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd2700a1ef0 0x7fd2700a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:32.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.449+0000 7fd291000640 1 --2- 172.21.15.43:0/956325153 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd2700a1ef0 0x7fd2700a43b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd2780099a0 tx=0x7fd278009500 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:32.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.452+0000 7fd28a000640 1 -- 172.21.15.43:0/956325153 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd28404c020 con 0x7fd28c073b80 2024-03-28T22:41:32.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.741+0000 7fd292e00640 1 -- 172.21.15.43:0/956325153 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd28c108f00 con 0x7fd28c073b80 2024-03-28T22:41:32.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.742+0000 7fd28a000640 1 -- 172.21.15.43:0/956325153 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11931 (secure 0 0 0) 0x7fd2840bcee0 con 0x7fd28c073b80 2024-03-28T22:41:32.740 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:41:32.740 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:33:28.906396+0000","modified":"2024-03-28T22:40:08.672168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24572,"mds_1":24556,"mds_2":24535,"mds_3":24523,"mds_4":24518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26717":{"gid":26717,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/215779342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":215779342},{"type":"v1","addr":"172.21.15.82:6835","nonce":215779342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24535,24556,24518,24523,24572]},"id":1}]} 2024-03-28T22:41:32.740 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 32 2024-03-28T22:41:32.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.745+0000 7fd292e00640 1 -- 172.21.15.43:0/956325153 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd2700a1ef0 msgr2=0x7fd2700a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:32.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.745+0000 7fd292e00640 1 --2- 172.21.15.43:0/956325153 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd2700a1ef0 0x7fd2700a43b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd2780099a0 tx=0x7fd278009500 comp rx=0 tx=0).stop 2024-03-28T22:41:32.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.745+0000 7fd292e00640 1 -- 172.21.15.43:0/956325153 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd28c073b80 msgr2=0x7fd28c19a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:32.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.745+0000 7fd292e00640 1 --2- 172.21.15.43:0/956325153 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd28c073b80 0x7fd28c19a030 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fd284000c00 tx=0x7fd284004360 comp rx=0 tx=0).stop 2024-03-28T22:41:32.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.745+0000 7fd292e00640 1 -- 172.21.15.43:0/956325153 shutdown_connections 2024-03-28T22:41:32.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.745+0000 7fd292e00640 1 --2- 172.21.15.43:0/956325153 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd2700a1ef0 0x7fd2700a43b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.745+0000 7fd292e00640 1 --2- 172.21.15.43:0/956325153 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd28c10dd80 0x7fd28c19e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.745+0000 7fd292e00640 1 --2- 172.21.15.43:0/956325153 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd28c0744a0 0x7fd28c19a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.745+0000 7fd292e00640 1 --2- 172.21.15.43:0/956325153 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd28c073b80 0x7fd28c19a030 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.745+0000 7fd292e00640 1 -- 172.21.15.43:0/956325153 >> 172.21.15.43:0/956325153 conn(0x7fd28c0fd030 msgr2=0x7fd28c0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:32.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.745+0000 7fd292e00640 1 -- 172.21.15.43:0/956325153 shutdown_connections 2024-03-28T22:41:32.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.746+0000 7fd292e00640 1 -- 172.21.15.43:0/956325153 wait complete. 2024-03-28T22:41:32.763 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T22:41:32.948 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:32 smithi082 ceph-mon[36471]: pgmap v135: 129 pgs: 129 active+clean; 50 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 158 op/s 2024-03-28T22:41:32.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.991+0000 7f23f4200640 1 -- 172.21.15.43:0/1603176261 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23ec100080 msgr2=0x7f23ec100460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:32.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.991+0000 7f23f4200640 1 --2- 172.21.15.43:0/1603176261 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23ec100080 0x7f23ec100460 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f23e0009930 tx=0x7f23e002bcd0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.991+0000 7f23f4200640 1 -- 172.21.15.43:0/1603176261 shutdown_connections 2024-03-28T22:41:32.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.991+0000 7f23f4200640 1 --2- 172.21.15.43:0/1603176261 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f23ec1009a0 0x7f23ec111fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.991+0000 7f23f4200640 1 --2- 172.21.15.43:0/1603176261 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f23ec112510 0x7f23ec114900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.991+0000 7f23f4200640 1 --2- 172.21.15.43:0/1603176261 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23ec100080 0x7f23ec100460 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.991+0000 7f23f4200640 1 -- 172.21.15.43:0/1603176261 >> 172.21.15.43:0/1603176261 conn(0x7f23ec075a00 msgr2=0x7f23ec075e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:32.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.991+0000 7f23f4200640 1 -- 172.21.15.43:0/1603176261 shutdown_connections 2024-03-28T22:41:32.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.991+0000 7f23f4200640 1 -- 172.21.15.43:0/1603176261 wait complete. 2024-03-28T22:41:32.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.992+0000 7f23f4200640 1 Processor -- start 2024-03-28T22:41:32.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.992+0000 7f23f4200640 1 -- start start 2024-03-28T22:41:32.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.992+0000 7f23f4200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f23ec100080 0x7f23ec19e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:32.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.992+0000 7f23f4200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f23ec1009a0 0x7f23ec19eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:32.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.992+0000 7f23f4200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23ec112510 0x7f23ec1a2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:32.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.993+0000 7f23f4200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f23ec118180 con 0x7f23ec100080 2024-03-28T22:41:32.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.993+0000 7f23f4200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f23ec118000 con 0x7f23ec1009a0 2024-03-28T22:41:32.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.993+0000 7f23f4200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f23ec118300 con 0x7f23ec112510 2024-03-28T22:41:32.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.993+0000 7f23f0e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f23ec1009a0 0x7f23ec19eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:32.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.993+0000 7f23f0e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f23ec1009a0 0x7f23ec19eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33696/0 (socket says 172.21.15.43:33696) 2024-03-28T22:41:32.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.993+0000 7f23f0e00640 1 -- 172.21.15.43:0/4116028527 learned_addr learned my addr 172.21.15.43:0/4116028527 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:32.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.993+0000 7f23f1800640 1 --2- 172.21.15.43:0/4116028527 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f23ec100080 0x7f23ec19e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:32.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.993+0000 7f23f2200640 1 --2- 172.21.15.43:0/4116028527 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23ec112510 0x7f23ec1a2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:32.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.993+0000 7f23f0e00640 1 -- 172.21.15.43:0/4116028527 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23ec112510 msgr2=0x7f23ec1a2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:32.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.993+0000 7f23f0e00640 1 --2- 172.21.15.43:0/4116028527 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23ec112510 0x7f23ec1a2ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.993+0000 7f23f0e00640 1 -- 172.21.15.43:0/4116028527 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f23ec100080 msgr2=0x7f23ec19e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:32.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.993+0000 7f23f0e00640 1 --2- 172.21.15.43:0/4116028527 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f23ec100080 0x7f23ec19e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.993+0000 7f23f0e00640 1 -- 172.21.15.43:0/4116028527 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23ec1a35f0 con 0x7f23ec1009a0 2024-03-28T22:41:32.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.994+0000 7f23f0e00640 1 --2- 172.21.15.43:0/4116028527 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f23ec1009a0 0x7f23ec19eb40 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f23d800e9f0 tx=0x7f23d800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:32.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.994+0000 7f23ea000640 1 -- 172.21.15.43:0/4116028527 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d800cd80 con 0x7f23ec1009a0 2024-03-28T22:41:32.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.994+0000 7f23f4200640 1 -- 172.21.15.43:0/4116028527 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23ec1a38e0 con 0x7f23ec1009a0 2024-03-28T22:41:32.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.994+0000 7f23ea000640 1 -- 172.21.15.43:0/4116028527 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23d8010910 con 0x7f23ec1009a0 2024-03-28T22:41:32.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.994+0000 7f23ea000640 1 -- 172.21.15.43:0/4116028527 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d80188f0 con 0x7f23ec1009a0 2024-03-28T22:41:32.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.994+0000 7f23f4200640 1 -- 172.21.15.43:0/4116028527 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23ec1ab170 con 0x7f23ec1009a0 2024-03-28T22:41:32.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.998+0000 7f23f6e56740 1 Processor -- start 2024-03-28T22:41:32.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.998+0000 7f23f6e56740 1 -- start start 2024-03-28T22:41:32.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.999+0000 7f23f6e56740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55841bdc5380 0x55841bdc5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:32.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.999+0000 7f23f6e56740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55841bdc5ca0 0x55841bdce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:32.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.999+0000 7f23f6e56740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55841bdce880 0x55841bdd0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:32.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.999+0000 7f23f6e56740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55841bd29050 con 0x55841bdce880 2024-03-28T22:41:32.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.999+0000 7f23f6e56740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55841bcd1ce0 con 0x55841bdc5380 2024-03-28T22:41:32.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.999+0000 7f23f6e56740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55841bd250f0 con 0x55841bdc5ca0 2024-03-28T22:41:32.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.999+0000 7f23ea000640 1 -- 172.21.15.43:0/4116028527 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f23d8010ac0 con 0x7f23ec1009a0 2024-03-28T22:41:32.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.999+0000 7f23f2200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55841bdc5380 0x55841bdc5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:32.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.999+0000 7f23f2200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55841bdc5380 0x55841bdc5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33706/0 (socket says 172.21.15.43:33706) 2024-03-28T22:41:32.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.999+0000 7f23f2200640 1 -- 172.21.15.43:0/1801345088 learned_addr learned my addr 172.21.15.43:0/1801345088 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:32.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.999+0000 7f23f1800640 1 --2- 172.21.15.43:0/1801345088 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55841bdc5ca0 0x55841bdce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:32.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:32.999+0000 7f23f0e00640 1 --2- 172.21.15.43:0/1801345088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55841bdce880 0x55841bdd0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:32.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.000+0000 7f23f2200640 1 -- 172.21.15.43:0/1801345088 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55841bdc5ca0 msgr2=0x55841bdce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:32.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.000+0000 7f23f2200640 1 --2- 172.21.15.43:0/1801345088 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55841bdc5ca0 0x55841bdce340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.000+0000 7f23f2200640 1 -- 172.21.15.43:0/1801345088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55841bdce880 msgr2=0x55841bdd0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:32.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.000+0000 7f23f2200640 1 --2- 172.21.15.43:0/1801345088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55841bdce880 0x55841bdd0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:32.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.000+0000 7f23f2200640 1 -- 172.21.15.43:0/1801345088 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55841ba3d3f0 con 0x55841bdc5380 2024-03-28T22:41:32.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.000+0000 7f23f0e00640 1 --2- 172.21.15.43:0/1801345088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55841bdce880 0x55841bdd0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:41:32.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.000+0000 7f23f2200640 1 --2- 172.21.15.43:0/1801345088 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55841bdc5380 0x55841bdc5760 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f23e4009f90 tx=0x7f23e40118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d769e0e772ce4a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:32.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.000+0000 7f23dea00640 1 -- 172.21.15.43:0/1801345088 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e4004d20 con 0x55841bdc5380 2024-03-28T22:41:32.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.000+0000 7f23dea00640 1 -- 172.21.15.43:0/1801345088 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23e4004050 con 0x55841bdc5380 2024-03-28T22:41:32.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.000+0000 7f23dea00640 1 -- 172.21.15.43:0/1801345088 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e4005240 con 0x55841bdc5380 2024-03-28T22:41:33.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.001+0000 7f23ea000640 1 --2- 172.21.15.43:0/4116028527 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f23d40a1ef0 0x7f23d40a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.001+0000 7f23f6e56740 1 -- 172.21.15.43:0/1801345088 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55841bdc5380 msgr2=0x55841bdc5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.001+0000 7f23f6e56740 1 --2- 172.21.15.43:0/1801345088 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55841bdc5380 0x55841bdc5760 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f23e4009f90 tx=0x7f23e40118d0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.001+0000 7f23ea000640 1 -- 172.21.15.43:0/4116028527 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f23d80ce910 con 0x7f23ec1009a0 2024-03-28T22:41:33.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.001+0000 7f23f6e56740 1 -- 172.21.15.43:0/1801345088 shutdown_connections 2024-03-28T22:41:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.001+0000 7f23f6e56740 1 --2- 172.21.15.43:0/1801345088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55841bdce880 0x55841bdd0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.001+0000 7f23f6e56740 1 --2- 172.21.15.43:0/1801345088 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55841bdc5ca0 0x55841bdce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.001+0000 7f23f6e56740 1 --2- 172.21.15.43:0/1801345088 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55841bdc5380 0x55841bdc5760 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.001+0000 7f23f1800640 1 --2- 172.21.15.43:0/4116028527 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f23d40a1ef0 0x7f23d40a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.001+0000 7f23f6e56740 1 -- 172.21.15.43:0/1801345088 >> 172.21.15.43:0/1801345088 conn(0x55841bdbfab0 msgr2=0x55841bdbfe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.002+0000 7f23f6e56740 1 -- 172.21.15.43:0/1801345088 shutdown_connections 2024-03-28T22:41:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.002+0000 7f23f6e56740 1 -- 172.21.15.43:0/1801345088 wait complete. 2024-03-28T22:41:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.002+0000 7f23f6e56740 1 -- start start 2024-03-28T22:41:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.002+0000 7f23f1800640 1 --2- 172.21.15.43:0/4116028527 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f23d40a1ef0 0x7f23d40a43b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f23e0009900 tx=0x7f23e002e040 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.002+0000 7f23f6e56740 1 Processor -- start 2024-03-28T22:41:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.003+0000 7f23f6e56740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55841bdc5380 0x55841bdcb850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.003+0000 7f23f6e56740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55841bdc5ca0 0x55841bdcbd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.003+0000 7f23f6e56740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55841bdce880 0x55841bdcc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.003+0000 7f23f6e56740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55841bdd1670 con 0x55841bdc5ca0 2024-03-28T22:41:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.003+0000 7f23f6e56740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55841bcc72f0 con 0x55841bdce880 2024-03-28T22:41:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.003+0000 7f23f6e56740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55841bdd17f0 con 0x55841bdc5380 2024-03-28T22:41:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.003+0000 7f23f6e56740 10 client.0 _ll_register_callbacks cb 0x55841ba3cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:41:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.003+0000 7f23f1800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55841bdc5380 0x55841bdcb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.003+0000 7f23f1800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55841bdc5380 0x55841bdcb850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:50258/0 (socket says 172.21.15.43:50258) 2024-03-28T22:41:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.003+0000 7f23f1800640 1 -- 172.21.15.43:0/148280639 learned_addr learned my addr 172.21.15.43:0/148280639 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.003+0000 7f23f2200640 1 --2- 172.21.15.43:0/148280639 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55841bdce880 0x55841bdcc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.003+0000 7f23f0e00640 1 --2- 172.21.15.43:0/148280639 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55841bdc5ca0 0x55841bdcbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.004+0000 7f23f1800640 1 -- 172.21.15.43:0/148280639 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55841bdce880 msgr2=0x55841bdcc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.004+0000 7f23f1800640 1 --2- 172.21.15.43:0/148280639 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55841bdce880 0x55841bdcc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.004+0000 7f23f1800640 1 -- 172.21.15.43:0/148280639 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55841bdc5ca0 msgr2=0x55841bdcbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.004+0000 7f23f1800640 1 --2- 172.21.15.43:0/148280639 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55841bdc5ca0 0x55841bdcbd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.004+0000 7f23f1800640 1 -- 172.21.15.43:0/148280639 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55841bb077d0 con 0x55841bdc5380 2024-03-28T22:41:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.004+0000 7f23f2200640 1 --2- 172.21.15.43:0/148280639 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55841bdce880 0x55841bdcc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:41:33.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.004+0000 7f23f1800640 1 --2- 172.21.15.43:0/148280639 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55841bdc5380 0x55841bdcb850 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f23e0009930 tx=0x7f23e002bcd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64ca42c9a2a61a60 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:33.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.004+0000 7f23d3400640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:41:33.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.004+0000 7f23d3400640 1 -- 172.21.15.43:0/148280639 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e0051020 con 0x55841bdc5380 2024-03-28T22:41:33.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.004+0000 7f23d3400640 1 -- 172.21.15.43:0/148280639 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f23e0004a50 con 0x55841bdc5380 2024-03-28T22:41:33.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.004+0000 7f23f6e56740 1 -- 172.21.15.43:0/148280639 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23e4004050 con 0x55841bdc5380 2024-03-28T22:41:33.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.005+0000 7f23d3400640 1 -- 172.21.15.43:0/148280639 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f23e005c050 con 0x55841bdc5380 2024-03-28T22:41:33.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.005+0000 7f23f6e56740 10 client.27229 fetch_fsmap learned FSMap version 32 2024-03-28T22:41:33.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.005+0000 7f23f6e56740 1 -- 172.21.15.43:0/148280639 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55841bddd8b0 con 0x55841bdc5380 2024-03-28T22:41:33.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.005+0000 7f23d3400640 1 -- 172.21.15.43:0/148280639 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23e4004050 con 0x55841bdc5380 2024-03-28T22:41:33.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.005+0000 7f23d3400640 1 -- 172.21.15.43:0/148280639 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f23e0003890 con 0x55841bdc5380 2024-03-28T22:41:33.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.005+0000 7f23f6e56740 10 client.27229 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:41:33.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.005+0000 7f23f6e56740 10 client.27229 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:41:33.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.005+0000 7f23f6e56740 20 client.27229 populate_metadata read hostname 'smithi043' 2024-03-28T22:41:33.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.005+0000 7f23f6e56740 1 --2- 172.21.15.43:0/148280639 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55841bdded70 0x55841bde11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.005+0000 7f23f6e56740 4 client.27229 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:41:33.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.005+0000 7f23f6e56740 1 -- 172.21.15.43:0/148280639 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55841bcd2680 con 0x55841bdded70 2024-03-28T22:41:33.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.006+0000 7f23f0e00640 1 --2- 172.21.15.43:0/148280639 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55841bdded70 0x55841bde11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.006+0000 7f23f0e00640 1 --2- 172.21.15.43:0/148280639 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55841bdded70 0x55841bde11f0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=89395d7c856e13ca server_cookie=8a70626306d7732a in_seq=0 out_seq=0 2024-03-28T22:41:33.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.006+0000 7f23d3400640 10 client.27229 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:41:33.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.007+0000 7f23d3400640 1 -- 172.21.15.43:0/148280639 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55841bcd2680 con 0x55841bdded70 2024-03-28T22:41:33.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.007+0000 7f23d3400640 10 client.27229 handle_command_reply: tid=0 2024-03-28T22:41:33.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.008+0000 7f23f6e56740 1 client.27229 shutdown 2024-03-28T22:41:33.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.008+0000 7f23f6e56740 1 -- 172.21.15.43:0/148280639 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55841bdc5380 msgr2=0x55841bdcb850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.008+0000 7f23f6e56740 1 --2- 172.21.15.43:0/148280639 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55841bdc5380 0x55841bdcb850 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f23e0009930 tx=0x7f23e002bcd0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.008+0000 7f23f6e56740 1 -- 172.21.15.43:0/148280639 shutdown_connections 2024-03-28T22:41:33.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.008+0000 7f23f6e56740 1 --2- 172.21.15.43:0/148280639 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55841bdded70 0x55841bde11f0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.008+0000 7f23f6e56740 1 --2- 172.21.15.43:0/148280639 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55841bdce880 0x55841bdcc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.008+0000 7f23f6e56740 1 --2- 172.21.15.43:0/148280639 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55841bdc5ca0 0x55841bdcbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.008+0000 7f23d3400640 0 client.27229 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:41:33.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.008+0000 7f23f6e56740 1 --2- 172.21.15.43:0/148280639 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55841bdc5380 0x55841bdcb850 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.008+0000 7f23f6e56740 1 -- 172.21.15.43:0/148280639 >> 172.21.15.43:0/148280639 conn(0x55841bdbfab0 msgr2=0x55841bdc1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:33.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.008+0000 7f23f6e56740 1 -- 172.21.15.43:0/148280639 shutdown_connections 2024-03-28T22:41:33.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.008+0000 7f23f6e56740 1 -- 172.21.15.43:0/148280639 wait complete. 2024-03-28T22:41:33.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.008+0000 7f23f6e56740 20 client.27229 trim_cache size 0 max 16384 2024-03-28T22:41:33.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:32 smithi043 ceph-mon[31823]: pgmap v135: 129 pgs: 129 active+clean; 50 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 158 op/s 2024-03-28T22:41:33.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.026+0000 7f23f6e56740 1 Processor -- start 2024-03-28T22:41:33.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.026+0000 7f23f6e56740 1 -- start start 2024-03-28T22:41:33.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.026+0000 7f23f6e56740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55841bdc5380 0x55841bdca9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.026+0000 7f23f6e56740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55841bdc5ca0 0x55841bdda410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.026+0000 7f23f6e56740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55841bdce880 0x55841bdcc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.026+0000 7f23f6e56740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f23ec1b5190 con 0x55841bdc5380 2024-03-28T22:41:33.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.026+0000 7f23f6e56740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f23ec1b6cd0 con 0x55841bdce880 2024-03-28T22:41:33.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.026+0000 7f23f6e56740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f23ec1b5860 con 0x55841bdc5ca0 2024-03-28T22:41:33.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.027+0000 7f23f2200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55841bdce880 0x55841bdcc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.027+0000 7f23f0e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55841bdc5ca0 0x55841bdda410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.027+0000 7f23f2200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55841bdce880 0x55841bdcc0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33724/0 (socket says 172.21.15.43:33724) 2024-03-28T22:41:33.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.027+0000 7f23f2200640 1 -- 172.21.15.43:0/159061509 learned_addr learned my addr 172.21.15.43:0/159061509 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:33.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.027+0000 7f23f0e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55841bdc5ca0 0x55841bdda410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:50262/0 (socket says 172.21.15.43:50262) 2024-03-28T22:41:33.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.027+0000 7f23f1800640 1 --2- 172.21.15.43:0/159061509 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55841bdc5380 0x55841bdca9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.027+0000 7f23f2200640 1 -- 172.21.15.43:0/159061509 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55841bdc5ca0 msgr2=0x55841bdda410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:33.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.027+0000 7f23f2200640 1 --2- 172.21.15.43:0/159061509 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55841bdc5ca0 0x55841bdda410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.027+0000 7f23f2200640 1 -- 172.21.15.43:0/159061509 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55841bdc5380 msgr2=0x55841bdca9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:33.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.027+0000 7f23f2200640 1 --2- 172.21.15.43:0/159061509 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55841bdc5380 0x55841bdca9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.027+0000 7f23f2200640 1 -- 172.21.15.43:0/159061509 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55841bdbf2f0 con 0x55841bdce880 2024-03-28T22:41:33.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.027+0000 7f23f1800640 1 --2- 172.21.15.43:0/159061509 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55841bdc5380 0x55841bdca9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:41:33.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.027+0000 7f23f0e00640 1 --2- 172.21.15.43:0/159061509 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55841bdc5ca0 0x55841bdda410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:41:33.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.027+0000 7f23f2200640 1 --2- 172.21.15.43:0/159061509 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55841bdce880 0x55841bdcc0f0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f23e4011dd0 tx=0x7f23e4009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d26a18891d3c9aec server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:33.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.028+0000 7f23de000640 1 -- 172.21.15.43:0/159061509 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e4005240 con 0x55841bdce880 2024-03-28T22:41:33.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.028+0000 7f23de000640 1 -- 172.21.15.43:0/159061509 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23e40058c0 con 0x55841bdce880 2024-03-28T22:41:33.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.028+0000 7f23de000640 1 -- 172.21.15.43:0/159061509 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e4019430 con 0x55841bdce880 2024-03-28T22:41:33.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.028+0000 7f23f6e56740 1 -- 172.21.15.43:0/159061509 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55841bdce880 msgr2=0x55841bdcc0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.028+0000 7f23f6e56740 1 --2- 172.21.15.43:0/159061509 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55841bdce880 0x55841bdcc0f0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f23e4011dd0 tx=0x7f23e4009b30 comp rx=0 tx=0).stop 2024-03-28T22:41:33.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.029+0000 7f23f6e56740 1 -- 172.21.15.43:0/159061509 shutdown_connections 2024-03-28T22:41:33.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.029+0000 7f23f6e56740 1 --2- 172.21.15.43:0/159061509 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55841bdce880 0x55841bdcc0f0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.029+0000 7f23f6e56740 1 --2- 172.21.15.43:0/159061509 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55841bdc5ca0 0x55841bdda410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.029+0000 7f23f6e56740 1 --2- 172.21.15.43:0/159061509 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55841bdc5380 0x55841bdca9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.029+0000 7f23f6e56740 1 -- 172.21.15.43:0/159061509 >> 172.21.15.43:0/159061509 conn(0x55841bdbfab0 msgr2=0x55841bdc0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:33.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.029+0000 7f23f6e56740 1 -- 172.21.15.43:0/159061509 shutdown_connections 2024-03-28T22:41:33.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.029+0000 7f23f6e56740 1 -- 172.21.15.43:0/159061509 wait complete. 2024-03-28T22:41:33.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.029+0000 7f23f6e56740 1 -- start start 2024-03-28T22:41:33.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.029+0000 7f23f6e56740 1 Processor -- start 2024-03-28T22:41:33.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.030+0000 7f23f6e56740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55841bdc5380 0x55841bdd6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.030+0000 7f23f6e56740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55841bdc5ca0 0x55841bdd1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.030+0000 7f23f6e56740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55841bdce880 0x55841bdd2350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.030+0000 7f23f6e56740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f23ec1b6600 con 0x55841bdc5ca0 2024-03-28T22:41:33.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.030+0000 7f23f6e56740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f23ec1b5f30 con 0x55841bdce880 2024-03-28T22:41:33.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.030+0000 7f23f6e56740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f23ec1b3170 con 0x55841bdc5380 2024-03-28T22:41:33.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.030+0000 7f23f6e56740 10 client.0 _ll_register_callbacks cb 0x7f23e0044070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:41:33.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.030+0000 7f23f2200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55841bdce880 0x55841bdd2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.030+0000 7f23f0e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55841bdc5ca0 0x55841bdd1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.030+0000 7f23f1800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55841bdc5380 0x55841bdd6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.030+0000 7f23f2200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55841bdce880 0x55841bdd2350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33738/0 (socket says 172.21.15.43:33738) 2024-03-28T22:41:33.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.030+0000 7f23f1800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55841bdc5380 0x55841bdd6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:50268/0 (socket says 172.21.15.43:50268) 2024-03-28T22:41:33.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.030+0000 7f23f2200640 1 -- 172.21.15.43:0/2626675823 learned_addr learned my addr 172.21.15.43:0/2626675823 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:33.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.030+0000 7f23f2200640 1 -- 172.21.15.43:0/2626675823 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55841bdc5380 msgr2=0x55841bdd6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:33.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.031+0000 7f23f2200640 1 --2- 172.21.15.43:0/2626675823 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55841bdc5380 0x55841bdd6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.031+0000 7f23f2200640 1 -- 172.21.15.43:0/2626675823 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55841bdc5ca0 msgr2=0x55841bdd1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:33.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.031+0000 7f23f2200640 1 --2- 172.21.15.43:0/2626675823 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55841bdc5ca0 0x55841bdd1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.031+0000 7f23f2200640 1 -- 172.21.15.43:0/2626675823 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55841bdd2890 con 0x55841bdce880 2024-03-28T22:41:33.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.031+0000 7f23f1800640 1 --2- 172.21.15.43:0/2626675823 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55841bdc5380 0x55841bdd6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:41:33.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.031+0000 7f23f2200640 1 --2- 172.21.15.43:0/2626675823 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55841bdce880 0x55841bdd2350 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f23e4006010 tx=0x7f23e400f710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53294ede2d4dc22 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:33.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.031+0000 7f23d3400640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:41:33.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.031+0000 7f23d3400640 1 -- 172.21.15.43:0/2626675823 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e4037060 con 0x55841bdce880 2024-03-28T22:41:33.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.031+0000 7f23d3400640 1 -- 172.21.15.43:0/2626675823 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f23e4005480 con 0x55841bdce880 2024-03-28T22:41:33.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.031+0000 7f23f6e56740 1 -- 172.21.15.43:0/2626675823 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23e40058c0 con 0x55841bdce880 2024-03-28T22:41:33.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.032+0000 7f23d3400640 1 -- 172.21.15.43:0/2626675823 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f23e403e050 con 0x55841bdce880 2024-03-28T22:41:33.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.032+0000 7f23f6e56740 10 client.27281 fetch_fsmap learned FSMap version 32 2024-03-28T22:41:33.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.032+0000 7f23f6e56740 1 -- 172.21.15.43:0/2626675823 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55841be44e70 con 0x55841bdce880 2024-03-28T22:41:33.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.032+0000 7f23d3400640 1 -- 172.21.15.43:0/2626675823 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23e40058c0 con 0x55841bdce880 2024-03-28T22:41:33.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.032+0000 7f23d3400640 1 -- 172.21.15.43:0/2626675823 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f23e4019ac0 con 0x55841bdce880 2024-03-28T22:41:33.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.032+0000 7f23f6e56740 10 client.27281 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:41:33.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.032+0000 7f23f6e56740 10 client.27281 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:41:33.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.032+0000 7f23f6e56740 20 client.27281 populate_metadata read hostname 'smithi043' 2024-03-28T22:41:33.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.032+0000 7f23f6e56740 1 --2- 172.21.15.43:0/2626675823 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55841be45330 0x55841be47720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.032+0000 7f23f6e56740 4 client.27281 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:41:33.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.032+0000 7f23f6e56740 1 -- 172.21.15.43:0/2626675823 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55841bd36590 con 0x55841be45330 2024-03-28T22:41:33.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.033+0000 7f23f1800640 1 --2- 172.21.15.43:0/2626675823 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55841be45330 0x55841be47720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.033+0000 7f23f1800640 1 --2- 172.21.15.43:0/2626675823 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55841be45330 0x55841be47720 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=69efb10046dddee8 server_cookie=b40b84ddf057dbe in_seq=0 out_seq=0 2024-03-28T22:41:33.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.033+0000 7f23d3400640 10 client.27281 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:41:33.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.033+0000 7f23d3400640 1 -- 172.21.15.43:0/2626675823 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55841bd36590 con 0x55841be45330 2024-03-28T22:41:33.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.033+0000 7f23d3400640 10 client.27281 handle_command_reply: tid=0 2024-03-28T22:41:33.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.033+0000 7f23f6e56740 1 client.27281 shutdown 2024-03-28T22:41:33.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.034+0000 7f23f6e56740 1 -- 172.21.15.43:0/2626675823 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55841bdce880 msgr2=0x55841bdd2350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.034+0000 7f23f6e56740 1 --2- 172.21.15.43:0/2626675823 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55841bdce880 0x55841bdd2350 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f23e4006010 tx=0x7f23e400f710 comp rx=0 tx=0).stop 2024-03-28T22:41:33.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.034+0000 7f23f6e56740 1 -- 172.21.15.43:0/2626675823 shutdown_connections 2024-03-28T22:41:33.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.034+0000 7f23f6e56740 1 --2- 172.21.15.43:0/2626675823 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55841be45330 0x55841be47720 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.034+0000 7f23f6e56740 1 --2- 172.21.15.43:0/2626675823 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55841bdce880 0x55841bdd2350 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.034+0000 7f23f6e56740 1 --2- 172.21.15.43:0/2626675823 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55841bdc5ca0 0x55841bdd1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.034+0000 7f23f6e56740 1 --2- 172.21.15.43:0/2626675823 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55841bdc5380 0x55841bdd6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.034+0000 7f23f6e56740 1 -- 172.21.15.43:0/2626675823 >> 172.21.15.43:0/2626675823 conn(0x55841bdbfab0 msgr2=0x55841bdbeb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:33.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.034+0000 7f23d3400640 0 client.27281 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:41:33.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.034+0000 7f23f6e56740 1 -- 172.21.15.43:0/2626675823 shutdown_connections 2024-03-28T22:41:33.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.034+0000 7f23f6e56740 1 -- 172.21.15.43:0/2626675823 wait complete. 2024-03-28T22:41:33.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.034+0000 7f23f6e56740 20 client.27281 trim_cache size 0 max 16384 2024-03-28T22:41:33.035 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:41:33.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.035+0000 7f23f4200640 1 -- 172.21.15.43:0/4116028527 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f23d40a1ef0 msgr2=0x7f23d40a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.035+0000 7f23f4200640 1 --2- 172.21.15.43:0/4116028527 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f23d40a1ef0 0x7f23d40a43b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f23e0009900 tx=0x7f23e002e040 comp rx=0 tx=0).stop 2024-03-28T22:41:33.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.035+0000 7f23f4200640 1 -- 172.21.15.43:0/4116028527 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f23ec1009a0 msgr2=0x7f23ec19eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.035+0000 7f23f4200640 1 --2- 172.21.15.43:0/4116028527 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f23ec1009a0 0x7f23ec19eb40 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f23d800e9f0 tx=0x7f23d800eeb0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.035+0000 7f23f4200640 1 -- 172.21.15.43:0/4116028527 shutdown_connections 2024-03-28T22:41:33.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.035+0000 7f23f4200640 1 --2- 172.21.15.43:0/4116028527 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f23d40a1ef0 0x7f23d40a43b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.035+0000 7f23f4200640 1 --2- 172.21.15.43:0/4116028527 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f23ec1009a0 0x7f23ec19eb40 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.035+0000 7f23f4200640 1 --2- 172.21.15.43:0/4116028527 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23ec112510 0x7f23ec1a2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.035+0000 7f23f4200640 1 --2- 172.21.15.43:0/4116028527 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f23ec100080 0x7f23ec19e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.035+0000 7f23f4200640 1 -- 172.21.15.43:0/4116028527 >> 172.21.15.43:0/4116028527 conn(0x7f23ec075a00 msgr2=0x7f23ec10ffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:33.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.035+0000 7f23f4200640 1 -- 172.21.15.43:0/4116028527 shutdown_connections 2024-03-28T22:41:33.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.035+0000 7f23f4200640 1 -- 172.21.15.43:0/4116028527 wait complete. 2024-03-28T22:41:33.051 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T22:41:33.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:32 smithi172 ceph-mon[36566]: pgmap v135: 129 pgs: 129 active+clean; 50 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 158 op/s 2024-03-28T22:41:33.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.282+0000 7f26a2200640 1 -- 172.21.15.43:0/3090202674 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f269c107680 msgr2=0x7f269c107a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.282+0000 7f26a2200640 1 --2- 172.21.15.43:0/3090202674 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f269c107680 0x7f269c107a60 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f268c009930 tx=0x7f268c02bc20 comp rx=0 tx=0).stop 2024-03-28T22:41:33.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.282+0000 7f26a2200640 1 -- 172.21.15.43:0/3090202674 shutdown_connections 2024-03-28T22:41:33.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.282+0000 7f26a2200640 1 --2- 172.21.15.43:0/3090202674 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f269c1015e0 0x7f269c101a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.282+0000 7f26a2200640 1 --2- 172.21.15.43:0/3090202674 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f269c101fa0 0x7f269c10e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.282+0000 7f26a2200640 1 --2- 172.21.15.43:0/3090202674 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f269c107680 0x7f269c107a60 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.282+0000 7f26a2200640 1 -- 172.21.15.43:0/3090202674 >> 172.21.15.43:0/3090202674 conn(0x7f269c0fd0f0 msgr2=0x7f269c0ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:33.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.283+0000 7f26a2200640 1 -- 172.21.15.43:0/3090202674 shutdown_connections 2024-03-28T22:41:33.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.283+0000 7f26a2200640 1 -- 172.21.15.43:0/3090202674 wait complete. 2024-03-28T22:41:33.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.284+0000 7f26a2200640 1 Processor -- start 2024-03-28T22:41:33.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.284+0000 7f26a2200640 1 -- start start 2024-03-28T22:41:33.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.284+0000 7f26a2200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f269c1015e0 0x7f269c197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.284+0000 7f26a2200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f269c101fa0 0x7f269c1983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.284+0000 7f26a2200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f269c107680 0x7f269c19c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.284+0000 7f26a2200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f269c111a10 con 0x7f269c1015e0 2024-03-28T22:41:33.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.284+0000 7f26a2200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f269c111890 con 0x7f269c101fa0 2024-03-28T22:41:33.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.284+0000 7f26a2200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f269c111b90 con 0x7f269c107680 2024-03-28T22:41:33.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.284+0000 7f26a0e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f269c1015e0 0x7f269c197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.284+0000 7f269be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f269c101fa0 0x7f269c1983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.285+0000 7f26a0e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f269c1015e0 0x7f269c197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37564/0 (socket says 172.21.15.43:37564) 2024-03-28T22:41:33.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.285+0000 7f26a1800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f269c107680 0x7f269c19c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.285+0000 7f26a0e00640 1 -- 172.21.15.43:0/1341130563 learned_addr learned my addr 172.21.15.43:0/1341130563 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:33.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.285+0000 7f26a0e00640 1 -- 172.21.15.43:0/1341130563 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f269c107680 msgr2=0x7f269c19c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.285+0000 7f26a0e00640 1 --2- 172.21.15.43:0/1341130563 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f269c107680 0x7f269c19c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.285+0000 7f26a0e00640 1 -- 172.21.15.43:0/1341130563 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f269c101fa0 msgr2=0x7f269c1983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.285+0000 7f26a0e00640 1 --2- 172.21.15.43:0/1341130563 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f269c101fa0 0x7f269c1983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.285+0000 7f26a0e00640 1 -- 172.21.15.43:0/1341130563 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f269c19ce80 con 0x7f269c1015e0 2024-03-28T22:41:33.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.285+0000 7f269be00640 1 --2- 172.21.15.43:0/1341130563 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f269c101fa0 0x7f269c1983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:41:33.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.285+0000 7f26a0e00640 1 --2- 172.21.15.43:0/1341130563 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f269c1015e0 0x7f269c197e90 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f268c005ab0 tx=0x7f268c005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:33.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.286+0000 7f2699600640 1 -- 172.21.15.43:0/1341130563 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f268c043020 con 0x7f269c1015e0 2024-03-28T22:41:33.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.286+0000 7f26a2200640 1 -- 172.21.15.43:0/1341130563 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f269c19d110 con 0x7f269c1015e0 2024-03-28T22:41:33.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.286+0000 7f2699600640 1 -- 172.21.15.43:0/1341130563 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f268c005000 con 0x7f269c1015e0 2024-03-28T22:41:33.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.286+0000 7f2699600640 1 -- 172.21.15.43:0/1341130563 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f268c003af0 con 0x7f269c1015e0 2024-03-28T22:41:33.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.286+0000 7f26a2200640 1 -- 172.21.15.43:0/1341130563 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f269c1a4a00 con 0x7f269c1015e0 2024-03-28T22:41:33.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.289+0000 7f2699600640 1 -- 172.21.15.43:0/1341130563 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f268c046020 con 0x7f269c1015e0 2024-03-28T22:41:33.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.290+0000 7f26a65c2740 1 Processor -- start 2024-03-28T22:41:33.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.290+0000 7f26a65c2740 1 -- start start 2024-03-28T22:41:33.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.290+0000 7f26a65c2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5636ff465fc0 0x5636ff4663a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.290+0000 7f26a65c2740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5636ff466970 0x5636ff466df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.290+0000 7f26a65c2740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5636ff478180 0x5636ff47a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.290+0000 7f26a65c2740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5636ff3ca050 con 0x5636ff466970 2024-03-28T22:41:33.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.290+0000 7f26a65c2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5636ff372ce0 con 0x5636ff465fc0 2024-03-28T22:41:33.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.290+0000 7f26a65c2740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5636ff3c60f0 con 0x5636ff478180 2024-03-28T22:41:33.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.291+0000 7f26a1800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5636ff465fc0 0x5636ff4663a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.291+0000 7f26a1800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5636ff465fc0 0x5636ff4663a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33756/0 (socket says 172.21.15.43:33756) 2024-03-28T22:41:33.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.291+0000 7f26a1800640 1 -- 172.21.15.43:0/743960712 learned_addr learned my addr 172.21.15.43:0/743960712 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:33.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.291+0000 7f26a0e00640 1 --2- 172.21.15.43:0/743960712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5636ff466970 0x5636ff466df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.291+0000 7f269be00640 1 --2- 172.21.15.43:0/743960712 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5636ff478180 0x5636ff47a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.291+0000 7f26a1800640 1 -- 172.21.15.43:0/743960712 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5636ff478180 msgr2=0x5636ff47a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:33.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.291+0000 7f26a1800640 1 --2- 172.21.15.43:0/743960712 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5636ff478180 0x5636ff47a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.291+0000 7f26a1800640 1 -- 172.21.15.43:0/743960712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5636ff466970 msgr2=0x5636ff466df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:33.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.291+0000 7f26a1800640 1 --2- 172.21.15.43:0/743960712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5636ff466970 0x5636ff466df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.291+0000 7f26a1800640 1 -- 172.21.15.43:0/743960712 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636ff0de3f0 con 0x5636ff465fc0 2024-03-28T22:41:33.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.291+0000 7f269be00640 1 --2- 172.21.15.43:0/743960712 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5636ff478180 0x5636ff47a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:41:33.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.291+0000 7f26a0e00640 1 --2- 172.21.15.43:0/743960712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5636ff466970 0x5636ff466df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:41:33.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.292+0000 7f2699600640 1 --2- 172.21.15.43:0/1341130563 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f26800a1ef0 0x7f26800a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.292+0000 7f26a1800640 1 --2- 172.21.15.43:0/743960712 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5636ff465fc0 0x5636ff4663a0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2694009fd0 tx=0x7f26940118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5dd1ffe794de6ed server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:33.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.292+0000 7f2699600640 1 -- 172.21.15.43:0/1341130563 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f268c0f03c0 con 0x7f269c1015e0 2024-03-28T22:41:33.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.292+0000 7f2692000640 1 -- 172.21.15.43:0/743960712 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2694004d20 con 0x5636ff465fc0 2024-03-28T22:41:33.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.292+0000 7f26a0e00640 1 --2- 172.21.15.43:0/1341130563 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f26800a1ef0 0x7f26800a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.292+0000 7f2692000640 1 -- 172.21.15.43:0/743960712 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2694004050 con 0x5636ff465fc0 2024-03-28T22:41:33.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.292+0000 7f2692000640 1 -- 172.21.15.43:0/743960712 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2694005240 con 0x5636ff465fc0 2024-03-28T22:41:33.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.293+0000 7f26a65c2740 1 -- 172.21.15.43:0/743960712 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5636ff465fc0 msgr2=0x5636ff4663a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.293+0000 7f26a65c2740 1 --2- 172.21.15.43:0/743960712 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5636ff465fc0 0x5636ff4663a0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2694009fd0 tx=0x7f26940118d0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.293+0000 7f26a0e00640 1 --2- 172.21.15.43:0/1341130563 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f26800a1ef0 0x7f26800a43b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x5636ff467460 tx=0x7f268c02bc20 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:33.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.293+0000 7f26a65c2740 1 -- 172.21.15.43:0/743960712 shutdown_connections 2024-03-28T22:41:33.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.293+0000 7f26a65c2740 1 --2- 172.21.15.43:0/743960712 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5636ff478180 0x5636ff47a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.293+0000 7f26a65c2740 1 --2- 172.21.15.43:0/743960712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5636ff466970 0x5636ff466df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.293+0000 7f26a65c2740 1 --2- 172.21.15.43:0/743960712 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5636ff465fc0 0x5636ff4663a0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.293+0000 7f26a65c2740 1 -- 172.21.15.43:0/743960712 >> 172.21.15.43:0/743960712 conn(0x5636ff460c80 msgr2=0x5636ff461060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:33.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.293+0000 7f26a65c2740 1 -- 172.21.15.43:0/743960712 shutdown_connections 2024-03-28T22:41:33.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.293+0000 7f26a65c2740 1 -- 172.21.15.43:0/743960712 wait complete. 2024-03-28T22:41:33.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.294+0000 7f26a65c2740 1 -- start start 2024-03-28T22:41:33.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.294+0000 7f26a65c2740 1 Processor -- start 2024-03-28T22:41:33.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.294+0000 7f26a65c2740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5636ff465fc0 0x5636ff468500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.294+0000 7f26a65c2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5636ff466970 0x5636ff468a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.294+0000 7f26a65c2740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5636ff478180 0x5636ff468f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.294+0000 7f26a65c2740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5636ff47b030 con 0x5636ff465fc0 2024-03-28T22:41:33.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.294+0000 7f26a65c2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5636ff3682f0 con 0x5636ff466970 2024-03-28T22:41:33.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.294+0000 7f26a65c2740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5636ff47b1b0 con 0x5636ff478180 2024-03-28T22:41:33.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.294+0000 7f26a65c2740 10 client.0 _ll_register_callbacks cb 0x5636ff0ddbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:41:33.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.294+0000 7f269be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5636ff466970 0x5636ff468a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.295+0000 7f26a1800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5636ff465fc0 0x5636ff468500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.295+0000 7f269be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5636ff466970 0x5636ff468a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33762/0 (socket says 172.21.15.43:33762) 2024-03-28T22:41:33.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.295+0000 7f269be00640 1 -- 172.21.15.43:0/4203859001 learned_addr learned my addr 172.21.15.43:0/4203859001 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:33.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.295+0000 7f26a1800640 1 --2- 172.21.15.43:0/4203859001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5636ff478180 0x5636ff468f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.295+0000 7f269be00640 1 -- 172.21.15.43:0/4203859001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5636ff478180 msgr2=0x5636ff468f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:33.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.295+0000 7f269be00640 1 --2- 172.21.15.43:0/4203859001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5636ff478180 0x5636ff468f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.295+0000 7f269be00640 1 -- 172.21.15.43:0/4203859001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5636ff465fc0 msgr2=0x5636ff468500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:33.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.295+0000 7f269be00640 1 --2- 172.21.15.43:0/4203859001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5636ff465fc0 0x5636ff468500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.295+0000 7f269be00640 1 -- 172.21.15.43:0/4203859001 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636ff1a87d0 con 0x5636ff466970 2024-03-28T22:41:33.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.295+0000 7f269be00640 1 --2- 172.21.15.43:0/4203859001 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5636ff466970 0x5636ff468a40 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f2688012a10 tx=0x7f2688012ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a82abb87fa538700 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:33.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.295+0000 7f2686a00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:41:33.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.295+0000 7f2686a00640 1 -- 172.21.15.43:0/4203859001 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f268800fac0 con 0x5636ff466970 2024-03-28T22:41:33.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.296+0000 7f26a65c2740 1 -- 172.21.15.43:0/4203859001 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2694004050 con 0x5636ff466970 2024-03-28T22:41:33.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.296+0000 7f2686a00640 1 -- 172.21.15.43:0/4203859001 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f268800fc60 con 0x5636ff466970 2024-03-28T22:41:33.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.296+0000 7f2686a00640 1 -- 172.21.15.43:0/4203859001 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2688024050 con 0x5636ff466970 2024-03-28T22:41:33.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.296+0000 7f26a65c2740 10 client.27293 fetch_fsmap learned FSMap version 32 2024-03-28T22:41:33.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.296+0000 7f26a65c2740 1 -- 172.21.15.43:0/4203859001 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5636ff4872a0 con 0x5636ff466970 2024-03-28T22:41:33.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.296+0000 7f2686a00640 1 -- 172.21.15.43:0/4203859001 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2694004050 con 0x5636ff466970 2024-03-28T22:41:33.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.296+0000 7f2686a00640 1 -- 172.21.15.43:0/4203859001 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f26880104c0 con 0x5636ff466970 2024-03-28T22:41:33.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.296+0000 7f26a65c2740 10 client.27293 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:41:33.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.297+0000 7f26a65c2740 10 client.27293 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:41:33.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.297+0000 7f26a65c2740 20 client.27293 populate_metadata read hostname 'smithi043' 2024-03-28T22:41:33.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.297+0000 7f26a65c2740 1 --2- 172.21.15.43:0/4203859001 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5636ff488760 0x5636ff48abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.297+0000 7f26a65c2740 4 client.27293 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:41:33.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.297+0000 7f26a65c2740 1 -- 172.21.15.43:0/4203859001 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5636ff373680 con 0x5636ff488760 2024-03-28T22:41:33.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.297+0000 7f26a1800640 1 --2- 172.21.15.43:0/4203859001 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5636ff488760 0x5636ff48abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.297+0000 7f26a1800640 1 --2- 172.21.15.43:0/4203859001 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5636ff488760 0x5636ff48abc0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f2ecd09de738f2fd server_cookie=be252038f63b4bb5 in_seq=0 out_seq=0 2024-03-28T22:41:33.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.297+0000 7f2686a00640 10 client.27293 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:41:33.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.298+0000 7f2686a00640 1 -- 172.21.15.43:0/4203859001 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5636ff373680 con 0x5636ff488760 2024-03-28T22:41:33.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.298+0000 7f2686a00640 10 client.27293 handle_command_reply: tid=0 2024-03-28T22:41:33.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.298+0000 7f26a65c2740 1 client.27293 shutdown 2024-03-28T22:41:33.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.299+0000 7f26a65c2740 1 -- 172.21.15.43:0/4203859001 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5636ff466970 msgr2=0x5636ff468a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.299+0000 7f26a65c2740 1 --2- 172.21.15.43:0/4203859001 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5636ff466970 0x5636ff468a40 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f2688012a10 tx=0x7f2688012ed0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.299+0000 7f26a65c2740 1 -- 172.21.15.43:0/4203859001 shutdown_connections 2024-03-28T22:41:33.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.299+0000 7f26a65c2740 1 --2- 172.21.15.43:0/4203859001 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5636ff488760 0x5636ff48abc0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.299+0000 7f26a65c2740 1 --2- 172.21.15.43:0/4203859001 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5636ff466970 0x5636ff468a40 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.299+0000 7f26a65c2740 1 --2- 172.21.15.43:0/4203859001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5636ff478180 0x5636ff468f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.299+0000 7f26a65c2740 1 --2- 172.21.15.43:0/4203859001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5636ff465fc0 0x5636ff468500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.299+0000 7f26a65c2740 1 -- 172.21.15.43:0/4203859001 >> 172.21.15.43:0/4203859001 conn(0x5636ff460c80 msgr2=0x5636ff4629d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:33.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.299+0000 7f2686a00640 0 client.27293 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:41:33.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.299+0000 7f26a65c2740 1 -- 172.21.15.43:0/4203859001 shutdown_connections 2024-03-28T22:41:33.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.299+0000 7f26a65c2740 1 -- 172.21.15.43:0/4203859001 wait complete. 2024-03-28T22:41:33.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.299+0000 7f26a65c2740 20 client.27293 trim_cache size 0 max 16384 2024-03-28T22:41:33.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.315+0000 7f26a65c2740 1 Processor -- start 2024-03-28T22:41:33.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.315+0000 7f26a65c2740 1 -- start start 2024-03-28T22:41:33.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.315+0000 7f26a65c2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5636ff465fc0 0x5636ff46abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.315+0000 7f26a65c2740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5636ff466970 0x5636ff483e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.315+0000 7f26a65c2740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5636ff478180 0x5636ff45bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.315+0000 7f26a65c2740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f269c1ac870 con 0x5636ff478180 2024-03-28T22:41:33.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.315+0000 7f26a65c2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f269c1ae3b0 con 0x5636ff465fc0 2024-03-28T22:41:33.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.315+0000 7f26a65c2740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f269c1acf40 con 0x5636ff466970 2024-03-28T22:41:33.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.316+0000 7f269be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5636ff466970 0x5636ff483e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.316+0000 7f269be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5636ff466970 0x5636ff483e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:50310/0 (socket says 172.21.15.43:50310) 2024-03-28T22:41:33.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.316+0000 7f269be00640 1 -- 172.21.15.43:0/1816409631 learned_addr learned my addr 172.21.15.43:0/1816409631 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:33.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.316+0000 7f26a1800640 1 --2- 172.21.15.43:0/1816409631 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5636ff465fc0 0x5636ff46abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.316+0000 7f26a1800640 1 --2- 172.21.15.43:0/1816409631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5636ff478180 0x5636ff45bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.316+0000 7f269be00640 1 -- 172.21.15.43:0/1816409631 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5636ff465fc0 msgr2=0x5636ff46abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:33.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.316+0000 7f269be00640 1 --2- 172.21.15.43:0/1816409631 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5636ff465fc0 0x5636ff46abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.316+0000 7f269be00640 1 -- 172.21.15.43:0/1816409631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5636ff478180 msgr2=0x5636ff45bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:33.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.316+0000 7f269be00640 1 --2- 172.21.15.43:0/1816409631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5636ff478180 0x5636ff45bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.316+0000 7f269be00640 1 -- 172.21.15.43:0/1816409631 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636ff47a9d0 con 0x5636ff466970 2024-03-28T22:41:33.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.316+0000 7f26a1800640 1 --2- 172.21.15.43:0/1816409631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5636ff478180 0x5636ff45bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:41:33.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.316+0000 7f269be00640 1 --2- 172.21.15.43:0/1816409631 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5636ff466970 0x5636ff483e40 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f2688010ad0 tx=0x7f2688012680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f3217d226ae0930 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:33.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.317+0000 7f2691600640 1 -- 172.21.15.43:0/1816409631 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2688010430 con 0x5636ff466970 2024-03-28T22:41:33.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.317+0000 7f2691600640 1 -- 172.21.15.43:0/1816409631 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26880105d0 con 0x5636ff466970 2024-03-28T22:41:33.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.317+0000 7f2691600640 1 -- 172.21.15.43:0/1816409631 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2688005400 con 0x5636ff466970 2024-03-28T22:41:33.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.317+0000 7f26a65c2740 1 -- 172.21.15.43:0/1816409631 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5636ff466970 msgr2=0x5636ff483e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.317+0000 7f26a65c2740 1 --2- 172.21.15.43:0/1816409631 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5636ff466970 0x5636ff483e40 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f2688010ad0 tx=0x7f2688012680 comp rx=0 tx=0).stop 2024-03-28T22:41:33.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.318+0000 7f26a65c2740 1 -- 172.21.15.43:0/1816409631 shutdown_connections 2024-03-28T22:41:33.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.318+0000 7f26a65c2740 1 --2- 172.21.15.43:0/1816409631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5636ff478180 0x5636ff45bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.318+0000 7f26a65c2740 1 --2- 172.21.15.43:0/1816409631 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5636ff466970 0x5636ff483e40 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.318+0000 7f26a65c2740 1 --2- 172.21.15.43:0/1816409631 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5636ff465fc0 0x5636ff46abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.318+0000 7f26a65c2740 1 -- 172.21.15.43:0/1816409631 >> 172.21.15.43:0/1816409631 conn(0x5636ff460c80 msgr2=0x5636ff45f530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:33.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.318+0000 7f26a65c2740 1 -- 172.21.15.43:0/1816409631 shutdown_connections 2024-03-28T22:41:33.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.318+0000 7f26a65c2740 1 -- 172.21.15.43:0/1816409631 wait complete. 2024-03-28T22:41:33.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.318+0000 7f26a65c2740 1 -- start start 2024-03-28T22:41:33.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.318+0000 7f26a65c2740 1 Processor -- start 2024-03-28T22:41:33.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.319+0000 7f26a65c2740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5636ff465fc0 0x5636ff475e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.319+0000 7f26a65c2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5636ff466970 0x5636ff476380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.319+0000 7f26a65c2740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5636ff478180 0x5636ff4768c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.319+0000 7f26a65c2740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f269c1adce0 con 0x5636ff478180 2024-03-28T22:41:33.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.319+0000 7f26a65c2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f269c1ad610 con 0x5636ff466970 2024-03-28T22:41:33.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.319+0000 7f26a65c2740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f269c1aa850 con 0x5636ff465fc0 2024-03-28T22:41:33.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.319+0000 7f26a65c2740 10 client.0 _ll_register_callbacks cb 0x7f2688009c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:41:33.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.319+0000 7f269be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5636ff466970 0x5636ff476380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.319+0000 7f269be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5636ff466970 0x5636ff476380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33782/0 (socket says 172.21.15.43:33782) 2024-03-28T22:41:33.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.319+0000 7f269be00640 1 -- 172.21.15.43:0/3585156349 learned_addr learned my addr 172.21.15.43:0/3585156349 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:33.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.319+0000 7f26a1800640 1 --2- 172.21.15.43:0/3585156349 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5636ff465fc0 0x5636ff475e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.319+0000 7f26a1800640 1 --2- 172.21.15.43:0/3585156349 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5636ff478180 0x5636ff4768c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.319+0000 7f269be00640 1 -- 172.21.15.43:0/3585156349 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5636ff465fc0 msgr2=0x5636ff475e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:33.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.319+0000 7f269be00640 1 --2- 172.21.15.43:0/3585156349 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5636ff465fc0 0x5636ff475e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.319+0000 7f269be00640 1 -- 172.21.15.43:0/3585156349 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5636ff478180 msgr2=0x5636ff4768c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:33.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.319+0000 7f269be00640 1 --2- 172.21.15.43:0/3585156349 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5636ff478180 0x5636ff4768c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.320+0000 7f269be00640 1 -- 172.21.15.43:0/3585156349 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636ff4edc60 con 0x5636ff466970 2024-03-28T22:41:33.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.320+0000 7f26a1800640 1 --2- 172.21.15.43:0/3585156349 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5636ff478180 0x5636ff4768c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:41:33.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.320+0000 7f269be00640 1 --2- 172.21.15.43:0/3585156349 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5636ff466970 0x5636ff476380 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f26880050a0 tx=0x7f2688012ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a14e56d53716cb38 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:33.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.320+0000 7f2686a00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:41:33.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.320+0000 7f2686a00640 1 -- 172.21.15.43:0/3585156349 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2688037060 con 0x5636ff466970 2024-03-28T22:41:33.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.320+0000 7f26a65c2740 1 -- 172.21.15.43:0/3585156349 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26880105d0 con 0x5636ff466970 2024-03-28T22:41:33.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.320+0000 7f2686a00640 1 -- 172.21.15.43:0/3585156349 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f2688018040 con 0x5636ff466970 2024-03-28T22:41:33.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.320+0000 7f2686a00640 1 -- 172.21.15.43:0/3585156349 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2688010430 con 0x5636ff466970 2024-03-28T22:41:33.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.320+0000 7f26a65c2740 10 client.27296 fetch_fsmap learned FSMap version 32 2024-03-28T22:41:33.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.320+0000 7f26a65c2740 1 -- 172.21.15.43:0/3585156349 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5636ff4eef20 con 0x5636ff466970 2024-03-28T22:41:33.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.321+0000 7f2686a00640 1 -- 172.21.15.43:0/3585156349 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26880105d0 con 0x5636ff466970 2024-03-28T22:41:33.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.321+0000 7f2686a00640 1 -- 172.21.15.43:0/3585156349 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f2688003a60 con 0x5636ff466970 2024-03-28T22:41:33.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.321+0000 7f26a65c2740 10 client.27296 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:41:33.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.321+0000 7f26a65c2740 10 client.27296 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:41:33.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.321+0000 7f26a65c2740 20 client.27296 populate_metadata read hostname 'smithi043' 2024-03-28T22:41:33.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.321+0000 7f26a65c2740 1 --2- 172.21.15.43:0/3585156349 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5636ff4ef500 0x5636ff4f1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.321+0000 7f26a65c2740 4 client.27296 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:41:33.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.321+0000 7f26a65c2740 1 -- 172.21.15.43:0/3585156349 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5636ff3d7590 con 0x5636ff4ef500 2024-03-28T22:41:33.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.321+0000 7f26a1800640 1 --2- 172.21.15.43:0/3585156349 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5636ff4ef500 0x5636ff4f1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.322+0000 7f26a1800640 1 --2- 172.21.15.43:0/3585156349 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5636ff4ef500 0x5636ff4f1920 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=735967ac97e6fbed server_cookie=1a6901bb5e3dcf71 in_seq=0 out_seq=0 2024-03-28T22:41:33.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.322+0000 7f2686a00640 10 client.27296 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:41:33.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.322+0000 7f2686a00640 1 -- 172.21.15.43:0/3585156349 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5636ff3d7590 con 0x5636ff4ef500 2024-03-28T22:41:33.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.322+0000 7f2686a00640 10 client.27296 handle_command_reply: tid=0 2024-03-28T22:41:33.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.322+0000 7f26a65c2740 1 client.27296 shutdown 2024-03-28T22:41:33.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.322+0000 7f26a65c2740 1 -- 172.21.15.43:0/3585156349 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5636ff466970 msgr2=0x5636ff476380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.322+0000 7f26a65c2740 1 --2- 172.21.15.43:0/3585156349 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5636ff466970 0x5636ff476380 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f26880050a0 tx=0x7f2688012ed0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.322+0000 7f26a65c2740 1 -- 172.21.15.43:0/3585156349 shutdown_connections 2024-03-28T22:41:33.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.322+0000 7f26a65c2740 1 --2- 172.21.15.43:0/3585156349 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5636ff4ef500 0x5636ff4f1920 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.322+0000 7f26a65c2740 1 --2- 172.21.15.43:0/3585156349 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5636ff466970 0x5636ff476380 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.322+0000 7f26a65c2740 1 --2- 172.21.15.43:0/3585156349 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5636ff478180 0x5636ff4768c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.322+0000 7f2686a00640 0 client.27296 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:41:33.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.322+0000 7f26a65c2740 1 --2- 172.21.15.43:0/3585156349 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5636ff465fc0 0x5636ff475e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.322+0000 7f26a65c2740 1 -- 172.21.15.43:0/3585156349 >> 172.21.15.43:0/3585156349 conn(0x5636ff460c80 msgr2=0x5636ff45fdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:33.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.323+0000 7f26a65c2740 1 -- 172.21.15.43:0/3585156349 shutdown_connections 2024-03-28T22:41:33.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.323+0000 7f26a65c2740 1 -- 172.21.15.43:0/3585156349 wait complete. 2024-03-28T22:41:33.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.323+0000 7f26a65c2740 20 client.27296 trim_cache size 0 max 16384 2024-03-28T22:41:33.325 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:41:33.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.323+0000 7f26a2200640 1 -- 172.21.15.43:0/1341130563 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f26800a1ef0 msgr2=0x7f26800a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.323+0000 7f26a2200640 1 --2- 172.21.15.43:0/1341130563 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f26800a1ef0 0x7f26800a43b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x5636ff467460 tx=0x7f268c02bc20 comp rx=0 tx=0).stop 2024-03-28T22:41:33.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.323+0000 7f26a2200640 1 -- 172.21.15.43:0/1341130563 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f269c1015e0 msgr2=0x7f269c197e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.323+0000 7f26a2200640 1 --2- 172.21.15.43:0/1341130563 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f269c1015e0 0x7f269c197e90 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f268c005ab0 tx=0x7f268c005e80 comp rx=0 tx=0).stop 2024-03-28T22:41:33.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.324+0000 7f26a2200640 1 -- 172.21.15.43:0/1341130563 shutdown_connections 2024-03-28T22:41:33.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.324+0000 7f26a2200640 1 --2- 172.21.15.43:0/1341130563 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f26800a1ef0 0x7f26800a43b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.324+0000 7f26a2200640 1 --2- 172.21.15.43:0/1341130563 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f269c101fa0 0x7f269c1983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.324+0000 7f26a2200640 1 --2- 172.21.15.43:0/1341130563 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f269c107680 0x7f269c19c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.324+0000 7f26a2200640 1 --2- 172.21.15.43:0/1341130563 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f269c1015e0 0x7f269c197e90 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.324+0000 7f26a2200640 1 -- 172.21.15.43:0/1341130563 >> 172.21.15.43:0/1341130563 conn(0x7f269c0fd0f0 msgr2=0x7f269c0fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:33.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.324+0000 7f26a2200640 1 -- 172.21.15.43:0/1341130563 shutdown_connections 2024-03-28T22:41:33.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.324+0000 7f26a2200640 1 -- 172.21.15.43:0/1341130563 wait complete. 2024-03-28T22:41:33.339 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:41:33.359 INFO:tasks.workunit.client.0.smithi043.stdout: 1 280634 8.04 MB/sec execute 64 sec latency 21.414 ms 2024-03-28T22:41:33.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.558+0000 7f58ebe00640 1 -- 172.21.15.43:0/4195952971 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58ec104190 msgr2=0x7f58ec1106a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.558+0000 7f58ebe00640 1 --2- 172.21.15.43:0/4195952971 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58ec104190 0x7f58ec1106a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f58e4009cd0 tx=0x7f58e402c0f0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.558+0000 7f58ebe00640 1 -- 172.21.15.43:0/4195952971 shutdown_connections 2024-03-28T22:41:33.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.558+0000 7f58ebe00640 1 --2- 172.21.15.43:0/4195952971 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58ec104190 0x7f58ec1106a0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.558+0000 7f58ebe00640 1 --2- 172.21.15.43:0/4195952971 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58ec1037d0 0x7f58ec103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.558+0000 7f58ebe00640 1 --2- 172.21.15.43:0/4195952971 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f58ec1097e0 0x7f58ec109bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.558+0000 7f58ebe00640 1 -- 172.21.15.43:0/4195952971 >> 172.21.15.43:0/4195952971 conn(0x7f58ec0ff260 msgr2=0x7f58ec101680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:33.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.558+0000 7f58ebe00640 1 -- 172.21.15.43:0/4195952971 shutdown_connections 2024-03-28T22:41:33.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.558+0000 7f58ebe00640 1 -- 172.21.15.43:0/4195952971 wait complete. 2024-03-28T22:41:33.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.559+0000 7f58ebe00640 1 Processor -- start 2024-03-28T22:41:33.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.559+0000 7f58ebe00640 1 -- start start 2024-03-28T22:41:33.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.560+0000 7f58ebe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58ec1037d0 0x7f58ec19a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.560+0000 7f58ebe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f58ec104190 0x7f58ec19a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.560+0000 7f58ebe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58ec1097e0 0x7f58ec19ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.560+0000 7f58ebe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f58ec113bf0 con 0x7f58ec1097e0 2024-03-28T22:41:33.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.560+0000 7f58ebe00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f58ec113a70 con 0x7f58ec104190 2024-03-28T22:41:33.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.560+0000 7f58ebe00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f58ec113d70 con 0x7f58ec1037d0 2024-03-28T22:41:33.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.560+0000 7f58ea000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f58ec104190 0x7f58ec19a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.560+0000 7f58ea000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f58ec104190 0x7f58ec19a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33798/0 (socket says 172.21.15.43:33798) 2024-03-28T22:41:33.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.560+0000 7f58ea000640 1 -- 172.21.15.43:0/1074596633 learned_addr learned my addr 172.21.15.43:0/1074596633 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:33.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.560+0000 7f58eb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58ec1097e0 0x7f58ec19ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.560+0000 7f58eaa00640 1 --2- 172.21.15.43:0/1074596633 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58ec1037d0 0x7f58ec19a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.560+0000 7f58ea000640 1 -- 172.21.15.43:0/1074596633 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58ec1037d0 msgr2=0x7f58ec19a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.560+0000 7f58ea000640 1 --2- 172.21.15.43:0/1074596633 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58ec1037d0 0x7f58ec19a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.560+0000 7f58ea000640 1 -- 172.21.15.43:0/1074596633 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58ec1097e0 msgr2=0x7f58ec19ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.560+0000 7f58ea000640 1 --2- 172.21.15.43:0/1074596633 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58ec1097e0 0x7f58ec19ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.560+0000 7f58ea000640 1 -- 172.21.15.43:0/1074596633 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58ec19f360 con 0x7f58ec104190 2024-03-28T22:41:33.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.561+0000 7f58ea000640 1 --2- 172.21.15.43:0/1074596633 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f58ec104190 0x7f58ec19a8b0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f58d800ea60 tx=0x7f58d800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:33.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.561+0000 7f58e3400640 1 -- 172.21.15.43:0/1074596633 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d800cde0 con 0x7f58ec104190 2024-03-28T22:41:33.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.561+0000 7f58e3400640 1 -- 172.21.15.43:0/1074596633 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58d80035f0 con 0x7f58ec104190 2024-03-28T22:41:33.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.561+0000 7f58ebe00640 1 -- 172.21.15.43:0/1074596633 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58ec19f650 con 0x7f58ec104190 2024-03-28T22:41:33.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.561+0000 7f58e3400640 1 -- 172.21.15.43:0/1074596633 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d8010670 con 0x7f58ec104190 2024-03-28T22:41:33.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.561+0000 7f58ebe00640 1 -- 172.21.15.43:0/1074596633 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58ec1a6ee0 con 0x7f58ec104190 2024-03-28T22:41:33.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.565+0000 7f58f43fe740 1 Processor -- start 2024-03-28T22:41:33.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.566+0000 7f58f43fe740 1 -- start start 2024-03-28T22:41:33.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.566+0000 7f58e3400640 1 -- 172.21.15.43:0/1074596633 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f58d80037a0 con 0x7f58ec104190 2024-03-28T22:41:33.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.566+0000 7f58f43fe740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561b2a80550 0x5561b2a80930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.566+0000 7f58f43fe740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5561b2a80e70 0x5561b2a89510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.566+0000 7f58f43fe740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561b2a89a50 0x5561b2a8be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.566+0000 7f58f43fe740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5561b29e4050 con 0x5561b2a80550 2024-03-28T22:41:33.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.566+0000 7f58f43fe740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5561b298cce0 con 0x5561b2a89a50 2024-03-28T22:41:33.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.566+0000 7f58f43fe740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5561b29e00f0 con 0x5561b2a80e70 2024-03-28T22:41:33.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.566+0000 7f58eb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561b2a80550 0x5561b2a80930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.567+0000 7f58eaa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5561b2a80e70 0x5561b2a89510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.567+0000 7f58eb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561b2a80550 0x5561b2a80930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37616/0 (socket says 172.21.15.43:37616) 2024-03-28T22:41:33.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.567+0000 7f58eb400640 1 -- 172.21.15.43:0/495254321 learned_addr learned my addr 172.21.15.43:0/495254321 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:33.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.567+0000 7f58eaa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5561b2a80e70 0x5561b2a89510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:50342/0 (socket says 172.21.15.43:50342) 2024-03-28T22:41:33.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.567+0000 7f58ea000640 1 --2- 172.21.15.43:0/495254321 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561b2a89a50 0x5561b2a8be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.567+0000 7f58eaa00640 1 -- 172.21.15.43:0/495254321 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561b2a89a50 msgr2=0x5561b2a8be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:33.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.567+0000 7f58eaa00640 1 --2- 172.21.15.43:0/495254321 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561b2a89a50 0x5561b2a8be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.567+0000 7f58eaa00640 1 -- 172.21.15.43:0/495254321 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561b2a80550 msgr2=0x5561b2a80930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:33.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.567+0000 7f58eaa00640 1 --2- 172.21.15.43:0/495254321 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561b2a80550 0x5561b2a80930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.567+0000 7f58eaa00640 1 -- 172.21.15.43:0/495254321 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561b26f83f0 con 0x5561b2a80e70 2024-03-28T22:41:33.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.567+0000 7f58eaa00640 1 --2- 172.21.15.43:0/495254321 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5561b2a80e70 0x5561b2a89510 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f58dc009c20 tx=0x7f58dc011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81f123f7f51e5cdf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:33.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.567+0000 7f58d7400640 1 -- 172.21.15.43:0/495254321 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58dc004050 con 0x5561b2a80e70 2024-03-28T22:41:33.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.568+0000 7f58d7400640 1 -- 172.21.15.43:0/495254321 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58dc005500 con 0x5561b2a80e70 2024-03-28T22:41:33.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.568+0000 7f58d7400640 1 -- 172.21.15.43:0/495254321 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58dc004d50 con 0x5561b2a80e70 2024-03-28T22:41:33.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.568+0000 7f58e3400640 1 --2- 172.21.15.43:0/1074596633 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f58d00a1ef0 0x7f58d00a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.568+0000 7f58f43fe740 1 -- 172.21.15.43:0/495254321 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5561b2a80e70 msgr2=0x5561b2a89510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.568+0000 7f58f43fe740 1 --2- 172.21.15.43:0/495254321 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5561b2a80e70 0x5561b2a89510 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f58dc009c20 tx=0x7f58dc011810 comp rx=0 tx=0).stop 2024-03-28T22:41:33.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.568+0000 7f58e3400640 1 -- 172.21.15.43:0/1074596633 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f58d80ce870 con 0x7f58ec104190 2024-03-28T22:41:33.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.568+0000 7f58f43fe740 1 -- 172.21.15.43:0/495254321 shutdown_connections 2024-03-28T22:41:33.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.568+0000 7f58f43fe740 1 --2- 172.21.15.43:0/495254321 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561b2a89a50 0x5561b2a8be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.568+0000 7f58f43fe740 1 --2- 172.21.15.43:0/495254321 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5561b2a80e70 0x5561b2a89510 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.568+0000 7f58f43fe740 1 --2- 172.21.15.43:0/495254321 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561b2a80550 0x5561b2a80930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.568+0000 7f58f43fe740 1 -- 172.21.15.43:0/495254321 >> 172.21.15.43:0/495254321 conn(0x5561b2a7ac80 msgr2=0x5561b2a7b060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:33.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.568+0000 7f58eb400640 1 --2- 172.21.15.43:0/1074596633 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f58d00a1ef0 0x7f58d00a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.568+0000 7f58f43fe740 1 -- 172.21.15.43:0/495254321 shutdown_connections 2024-03-28T22:41:33.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.569+0000 7f58f43fe740 1 -- 172.21.15.43:0/495254321 wait complete. 2024-03-28T22:41:33.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.569+0000 7f58f43fe740 1 -- start start 2024-03-28T22:41:33.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.569+0000 7f58f43fe740 1 Processor -- start 2024-03-28T22:41:33.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.570+0000 7f58eb400640 1 --2- 172.21.15.43:0/1074596633 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f58d00a1ef0 0x7f58d00a43b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f58e4009bf0 tx=0x7f58e4005d80 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:33.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.570+0000 7f58f43fe740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561b2a80550 0x5561b2a86a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.570+0000 7f58f43fe740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5561b2a80e70 0x5561b2a86fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.570+0000 7f58f43fe740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561b2a89a50 0x5561b2a87500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.570+0000 7f58f43fe740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5561b2a8c840 con 0x5561b2a80550 2024-03-28T22:41:33.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.570+0000 7f58f43fe740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5561b29822f0 con 0x5561b2a89a50 2024-03-28T22:41:33.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.570+0000 7f58f43fe740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5561b2a8c9c0 con 0x5561b2a80e70 2024-03-28T22:41:33.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.570+0000 7f58f43fe740 10 client.0 _ll_register_callbacks cb 0x5561b26f7be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:41:33.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.570+0000 7f58ea000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5561b2a80e70 0x5561b2a86fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.570+0000 7f58eaa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561b2a80550 0x5561b2a86a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.570+0000 7f58ea000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5561b2a80e70 0x5561b2a86fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:50350/0 (socket says 172.21.15.43:50350) 2024-03-28T22:41:33.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.570+0000 7f58eb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561b2a89a50 0x5561b2a87500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.570+0000 7f58ea000640 1 -- 172.21.15.43:0/974114857 learned_addr learned my addr 172.21.15.43:0/974114857 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:33.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.571+0000 7f58ea000640 1 -- 172.21.15.43:0/974114857 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561b2a89a50 msgr2=0x5561b2a87500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:33.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.571+0000 7f58ea000640 1 --2- 172.21.15.43:0/974114857 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561b2a89a50 0x5561b2a87500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.571+0000 7f58ea000640 1 -- 172.21.15.43:0/974114857 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561b2a80550 msgr2=0x5561b2a86a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:33.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.571+0000 7f58ea000640 1 --2- 172.21.15.43:0/974114857 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561b2a80550 0x5561b2a86a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.571+0000 7f58ea000640 1 -- 172.21.15.43:0/974114857 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561b27c27d0 con 0x5561b2a80e70 2024-03-28T22:41:33.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.571+0000 7f58ea000640 1 --2- 172.21.15.43:0/974114857 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5561b2a80e70 0x5561b2a86fc0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f58d809ab60 tx=0x7f58d809ac30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0d48c94c1635296 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:33.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.571+0000 7f58cbe00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:41:33.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.571+0000 7f58cbe00640 1 -- 172.21.15.43:0/974114857 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d809a7c0 con 0x5561b2a80e70 2024-03-28T22:41:33.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.571+0000 7f58cbe00640 1 -- 172.21.15.43:0/974114857 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f58d80cbe40 con 0x5561b2a80e70 2024-03-28T22:41:33.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.571+0000 7f58f43fe740 1 -- 172.21.15.43:0/974114857 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58dc005500 con 0x5561b2a80e70 2024-03-28T22:41:33.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.571+0000 7f58cbe00640 1 -- 172.21.15.43:0/974114857 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f58d80cc040 con 0x5561b2a80e70 2024-03-28T22:41:33.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.572+0000 7f58f43fe740 10 client.27265 fetch_fsmap learned FSMap version 32 2024-03-28T22:41:33.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.572+0000 7f58f43fe740 1 -- 172.21.15.43:0/974114857 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5561b2a98b40 con 0x5561b2a80e70 2024-03-28T22:41:33.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.572+0000 7f58cbe00640 1 -- 172.21.15.43:0/974114857 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58dc005500 con 0x5561b2a80e70 2024-03-28T22:41:33.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.572+0000 7f58cbe00640 1 -- 172.21.15.43:0/974114857 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f58d80ce450 con 0x5561b2a80e70 2024-03-28T22:41:33.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.572+0000 7f58f43fe740 10 client.27265 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:41:33.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.572+0000 7f58f43fe740 10 client.27265 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:41:33.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.572+0000 7f58f43fe740 20 client.27265 populate_metadata read hostname 'smithi043' 2024-03-28T22:41:33.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.572+0000 7f58f43fe740 1 --2- 172.21.15.43:0/974114857 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5561b2a9a000 0x5561b2a9c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.572+0000 7f58f43fe740 4 client.27265 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:41:33.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.572+0000 7f58f43fe740 1 -- 172.21.15.43:0/974114857 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5561b298d680 con 0x5561b2a9a000 2024-03-28T22:41:33.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.572+0000 7f58eaa00640 1 --2- 172.21.15.43:0/974114857 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5561b2a9a000 0x5561b2a9c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.573+0000 7f58eaa00640 1 --2- 172.21.15.43:0/974114857 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5561b2a9a000 0x5561b2a9c450 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c55d8f57ed68d260 server_cookie=7c17d991933c61bd in_seq=0 out_seq=0 2024-03-28T22:41:33.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.573+0000 7f58cbe00640 10 client.27265 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:41:33.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.574+0000 7f58cbe00640 1 -- 172.21.15.43:0/974114857 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5561b298d680 con 0x5561b2a9a000 2024-03-28T22:41:33.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.574+0000 7f58cbe00640 10 client.27265 handle_command_reply: tid=0 2024-03-28T22:41:33.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.574+0000 7f58f43fe740 1 client.27265 shutdown 2024-03-28T22:41:33.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.574+0000 7f58f43fe740 1 -- 172.21.15.43:0/974114857 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5561b2a80e70 msgr2=0x5561b2a86fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.574+0000 7f58f43fe740 1 --2- 172.21.15.43:0/974114857 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5561b2a80e70 0x5561b2a86fc0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f58d809ab60 tx=0x7f58d809ac30 comp rx=0 tx=0).stop 2024-03-28T22:41:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.574+0000 7f58f43fe740 1 -- 172.21.15.43:0/974114857 shutdown_connections 2024-03-28T22:41:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.574+0000 7f58f43fe740 1 --2- 172.21.15.43:0/974114857 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5561b2a9a000 0x5561b2a9c450 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.574+0000 7f58f43fe740 1 --2- 172.21.15.43:0/974114857 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561b2a89a50 0x5561b2a87500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.574+0000 7f58f43fe740 1 --2- 172.21.15.43:0/974114857 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5561b2a80e70 0x5561b2a86fc0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.574+0000 7f58f43fe740 1 --2- 172.21.15.43:0/974114857 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561b2a80550 0x5561b2a86a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.574+0000 7f58f43fe740 1 -- 172.21.15.43:0/974114857 >> 172.21.15.43:0/974114857 conn(0x5561b2a7ac80 msgr2=0x5561b2a7c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.574+0000 7f58cbe00640 0 client.27265 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:41:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.575+0000 7f58f43fe740 1 -- 172.21.15.43:0/974114857 shutdown_connections 2024-03-28T22:41:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.575+0000 7f58f43fe740 1 -- 172.21.15.43:0/974114857 wait complete. 2024-03-28T22:41:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.575+0000 7f58f43fe740 20 client.27265 trim_cache size 0 max 16384 2024-03-28T22:41:33.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.594+0000 7f58f43fe740 1 Processor -- start 2024-03-28T22:41:33.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.595+0000 7f58f43fe740 1 -- start start 2024-03-28T22:41:33.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.598+0000 7f58f43fe740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5561b2a80550 0x5561b2a891b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.598+0000 7f58f43fe740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561b2a80e70 0x5561b2a955e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.598+0000 7f58f43fe740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561b2a89a50 0x5561b2a79c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.598+0000 7f58f43fe740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f58ec1b0b60 con 0x5561b2a80e70 2024-03-28T22:41:33.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.598+0000 7f58f43fe740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f58ec1b26a0 con 0x5561b2a89a50 2024-03-28T22:41:33.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.598+0000 7f58f43fe740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f58ec1b1230 con 0x5561b2a80550 2024-03-28T22:41:33.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.599+0000 7f58eb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561b2a89a50 0x5561b2a79c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.599+0000 7f58eb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561b2a89a50 0x5561b2a79c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33836/0 (socket says 172.21.15.43:33836) 2024-03-28T22:41:33.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.599+0000 7f58eb400640 1 -- 172.21.15.43:0/4009873970 learned_addr learned my addr 172.21.15.43:0/4009873970 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:33.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.600+0000 7f58eb400640 1 -- 172.21.15.43:0/4009873970 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5561b2a80550 msgr2=0x5561b2a891b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:33.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.600+0000 7f58eb400640 1 --2- 172.21.15.43:0/4009873970 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5561b2a80550 0x5561b2a891b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.601+0000 7f58eb400640 1 -- 172.21.15.43:0/4009873970 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561b2a80e70 msgr2=0x5561b2a955e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:41:33.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.601+0000 7f58eb400640 1 --2- 172.21.15.43:0/4009873970 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561b2a80e70 0x5561b2a955e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.601+0000 7f58eb400640 1 -- 172.21.15.43:0/4009873970 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561b2a751d0 con 0x5561b2a89a50 2024-03-28T22:41:33.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.604+0000 7f58eb400640 1 --2- 172.21.15.43:0/4009873970 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561b2a89a50 0x5561b2a79c40 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f58e4006010 tx=0x7f58e4041000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddefb1834f230ea7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:33.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.604+0000 7f58d6a00640 1 -- 172.21.15.43:0/4009873970 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e4039040 con 0x5561b2a89a50 2024-03-28T22:41:33.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.605+0000 7f58d6a00640 1 -- 172.21.15.43:0/4009873970 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58e4003e50 con 0x5561b2a89a50 2024-03-28T22:41:33.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.605+0000 7f58d6a00640 1 -- 172.21.15.43:0/4009873970 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e40046a0 con 0x5561b2a89a50 2024-03-28T22:41:33.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.610+0000 7f58f43fe740 1 -- 172.21.15.43:0/4009873970 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561b2a89a50 msgr2=0x5561b2a79c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.610+0000 7f58f43fe740 1 --2- 172.21.15.43:0/4009873970 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561b2a89a50 0x5561b2a79c40 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f58e4006010 tx=0x7f58e4041000 comp rx=0 tx=0).stop 2024-03-28T22:41:33.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.612+0000 7f58f43fe740 1 -- 172.21.15.43:0/4009873970 shutdown_connections 2024-03-28T22:41:33.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.612+0000 7f58f43fe740 1 --2- 172.21.15.43:0/4009873970 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561b2a89a50 0x5561b2a79c40 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.612+0000 7f58f43fe740 1 --2- 172.21.15.43:0/4009873970 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561b2a80e70 0x5561b2a955e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.612+0000 7f58f43fe740 1 --2- 172.21.15.43:0/4009873970 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5561b2a80550 0x5561b2a891b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.612+0000 7f58f43fe740 1 -- 172.21.15.43:0/4009873970 >> 172.21.15.43:0/4009873970 conn(0x5561b2a7ac80 msgr2=0x5561b2a79530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:33.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.612+0000 7f58f43fe740 1 -- 172.21.15.43:0/4009873970 shutdown_connections 2024-03-28T22:41:33.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.613+0000 7f58f43fe740 1 -- 172.21.15.43:0/4009873970 wait complete. 2024-03-28T22:41:33.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.614+0000 7f58f43fe740 1 -- start start 2024-03-28T22:41:33.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.614+0000 7f58f43fe740 1 Processor -- start 2024-03-28T22:41:33.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.617+0000 7f58f43fe740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561b2a80550 0x5561b2a91ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.617+0000 7f58f43fe740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561b2a80e70 0x5561b2a8d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.617+0000 7f58f43fe740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5561b2a89a50 0x5561b2a8d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.617+0000 7f58f43fe740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f58ec1b1fd0 con 0x5561b2a80550 2024-03-28T22:41:33.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.617+0000 7f58f43fe740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f58ec1b1900 con 0x5561b2a80e70 2024-03-28T22:41:33.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.617+0000 7f58f43fe740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f58ec1aeb40 con 0x5561b2a89a50 2024-03-28T22:41:33.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.617+0000 7f58f43fe740 10 client.0 _ll_register_callbacks cb 0x7f58d80d4220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:41:33.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.617+0000 7f58ea000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561b2a80e70 0x5561b2a8d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.617+0000 7f58ea000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561b2a80e70 0x5561b2a8d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33842/0 (socket says 172.21.15.43:33842) 2024-03-28T22:41:33.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.617+0000 7f58ea000640 1 -- 172.21.15.43:0/87844354 learned_addr learned my addr 172.21.15.43:0/87844354 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:33.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.618+0000 7f58eb400640 1 --2- 172.21.15.43:0/87844354 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5561b2a89a50 0x5561b2a8d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.618+0000 7f58eb400640 1 -- 172.21.15.43:0/87844354 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561b2a80e70 msgr2=0x5561b2a8d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:33.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.618+0000 7f58eb400640 1 --2- 172.21.15.43:0/87844354 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561b2a80e70 0x5561b2a8d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.618+0000 7f58eb400640 1 -- 172.21.15.43:0/87844354 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561b2a80550 msgr2=0x5561b2a91ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:33.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.618+0000 7f58eb400640 1 --2- 172.21.15.43:0/87844354 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561b2a80550 0x5561b2a91ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.618+0000 7f58eb400640 1 -- 172.21.15.43:0/87844354 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561b2a8e020 con 0x5561b2a89a50 2024-03-28T22:41:33.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.622+0000 7f58eb400640 1 --2- 172.21.15.43:0/87844354 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5561b2a89a50 0x5561b2a8d7f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f58e4037040 tx=0x7f58e4032ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1d49d5793fcfb93 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:33.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.623+0000 7f58cbe00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:41:33.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.623+0000 7f58cbe00640 1 -- 172.21.15.43:0/87844354 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e405d020 con 0x5561b2a89a50 2024-03-28T22:41:33.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.624+0000 7f58f43fe740 1 -- 172.21.15.43:0/87844354 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58e4003e50 con 0x5561b2a89a50 2024-03-28T22:41:33.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.624+0000 7f58cbe00640 1 -- 172.21.15.43:0/87844354 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f58e4032c90 con 0x5561b2a89a50 2024-03-28T22:41:33.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.626+0000 7f58cbe00640 1 -- 172.21.15.43:0/87844354 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f58e405d020 con 0x5561b2a89a50 2024-03-28T22:41:33.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.626+0000 7f58f43fe740 10 client.27268 fetch_fsmap learned FSMap version 32 2024-03-28T22:41:33.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.626+0000 7f58f43fe740 1 -- 172.21.15.43:0/87844354 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5561b2b19bd0 con 0x5561b2a89a50 2024-03-28T22:41:33.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.627+0000 7f58cbe00640 1 -- 172.21.15.43:0/87844354 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58e4003e50 con 0x5561b2a89a50 2024-03-28T22:41:33.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.627+0000 7f58cbe00640 1 -- 172.21.15.43:0/87844354 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f58e404e570 con 0x5561b2a89a50 2024-03-28T22:41:33.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.628+0000 7f58f43fe740 10 client.27268 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:41:33.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.628+0000 7f58f43fe740 10 client.27268 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:41:33.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.628+0000 7f58f43fe740 20 client.27268 populate_metadata read hostname 'smithi043' 2024-03-28T22:41:33.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.628+0000 7f58f43fe740 1 --2- 172.21.15.43:0/87844354 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5561b2b1a230 0x5561b2b1c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.628+0000 7f58f43fe740 4 client.27268 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:41:33.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.628+0000 7f58f43fe740 1 -- 172.21.15.43:0/87844354 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5561b29f1590 con 0x5561b2b1a230 2024-03-28T22:41:33.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.630+0000 7f58eaa00640 1 --2- 172.21.15.43:0/87844354 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5561b2b1a230 0x5561b2b1c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.631+0000 7f58eaa00640 1 --2- 172.21.15.43:0/87844354 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5561b2b1a230 0x5561b2b1c650 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=193d63a235bb5004 server_cookie=ea2f0382c0668d92 in_seq=0 out_seq=0 2024-03-28T22:41:33.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.631+0000 7f58cbe00640 10 client.27268 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:41:33.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.632+0000 7f58cbe00640 1 -- 172.21.15.43:0/87844354 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5561b29f1590 con 0x5561b2b1a230 2024-03-28T22:41:33.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.632+0000 7f58cbe00640 10 client.27268 handle_command_reply: tid=0 2024-03-28T22:41:33.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.632+0000 7f58f43fe740 1 client.27268 shutdown 2024-03-28T22:41:33.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.633+0000 7f58f43fe740 1 -- 172.21.15.43:0/87844354 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5561b2a89a50 msgr2=0x5561b2a8d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.633+0000 7f58f43fe740 1 --2- 172.21.15.43:0/87844354 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5561b2a89a50 0x5561b2a8d7f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f58e4037040 tx=0x7f58e4032ae0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.633+0000 7f58f43fe740 1 -- 172.21.15.43:0/87844354 shutdown_connections 2024-03-28T22:41:33.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.633+0000 7f58f43fe740 1 --2- 172.21.15.43:0/87844354 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5561b2b1a230 0x5561b2b1c650 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.633+0000 7f58cbe00640 0 client.27268 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:41:33.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.633+0000 7f58f43fe740 1 --2- 172.21.15.43:0/87844354 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561b2a80e70 0x5561b2a8d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.633+0000 7f58f43fe740 1 --2- 172.21.15.43:0/87844354 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5561b2a89a50 0x5561b2a8d7f0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.633+0000 7f58f43fe740 1 --2- 172.21.15.43:0/87844354 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5561b2a80550 0x5561b2a91ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.633+0000 7f58f43fe740 1 -- 172.21.15.43:0/87844354 >> 172.21.15.43:0/87844354 conn(0x5561b2a7ac80 msgr2=0x5561b2a84f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:33.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.634+0000 7f58f43fe740 1 -- 172.21.15.43:0/87844354 shutdown_connections 2024-03-28T22:41:33.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.634+0000 7f58f43fe740 1 -- 172.21.15.43:0/87844354 wait complete. 2024-03-28T22:41:33.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.635+0000 7f58f43fe740 20 client.27268 trim_cache size 0 max 16384 2024-03-28T22:41:33.633 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:41:33.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.637+0000 7f58ebe00640 1 -- 172.21.15.43:0/1074596633 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f58d00a1ef0 msgr2=0x7f58d00a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.637+0000 7f58ebe00640 1 --2- 172.21.15.43:0/1074596633 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f58d00a1ef0 0x7f58d00a43b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f58e4009bf0 tx=0x7f58e4005d80 comp rx=0 tx=0).stop 2024-03-28T22:41:33.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.637+0000 7f58ebe00640 1 -- 172.21.15.43:0/1074596633 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f58ec104190 msgr2=0x7f58ec19a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.637+0000 7f58ebe00640 1 --2- 172.21.15.43:0/1074596633 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f58ec104190 0x7f58ec19a8b0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f58d800ea60 tx=0x7f58d800ef20 comp rx=0 tx=0).stop 2024-03-28T22:41:33.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.637+0000 7f58ebe00640 1 -- 172.21.15.43:0/1074596633 shutdown_connections 2024-03-28T22:41:33.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.637+0000 7f58ebe00640 1 --2- 172.21.15.43:0/1074596633 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f58d00a1ef0 0x7f58d00a43b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.637+0000 7f58ebe00640 1 --2- 172.21.15.43:0/1074596633 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f58ec104190 0x7f58ec19a8b0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.637+0000 7f58ebe00640 1 --2- 172.21.15.43:0/1074596633 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f58ec1097e0 0x7f58ec19ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.637+0000 7f58ebe00640 1 --2- 172.21.15.43:0/1074596633 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f58ec1037d0 0x7f58ec19a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.637+0000 7f58ebe00640 1 -- 172.21.15.43:0/1074596633 >> 172.21.15.43:0/1074596633 conn(0x7f58ec0ff260 msgr2=0x7f58ec0ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:33.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.637+0000 7f58ebe00640 1 -- 172.21.15.43:0/1074596633 shutdown_connections 2024-03-28T22:41:33.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.638+0000 7f58ebe00640 1 -- 172.21.15.43:0/1074596633 wait complete. 2024-03-28T22:41:33.659 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:41:33.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.920+0000 7fc7faa00640 1 -- 172.21.15.43:0/2429943328 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7f4107710 msgr2=0x7fc7f4107af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.920+0000 7fc7faa00640 1 --2- 172.21.15.43:0/2429943328 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7f4107710 0x7fc7f4107af0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fc7e4009900 tx=0x7fc7e402bc40 comp rx=0 tx=0).stop 2024-03-28T22:41:33.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.920+0000 7fc7faa00640 1 -- 172.21.15.43:0/2429943328 shutdown_connections 2024-03-28T22:41:33.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.920+0000 7fc7faa00640 1 --2- 172.21.15.43:0/2429943328 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7f406a640 0x7fc7f410e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.920+0000 7fc7faa00640 1 --2- 172.21.15.43:0/2429943328 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7f4069cc0 0x7fc7f406a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.920+0000 7fc7faa00640 1 --2- 172.21.15.43:0/2429943328 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7f4107710 0x7fc7f4107af0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.920+0000 7fc7faa00640 1 -- 172.21.15.43:0/2429943328 >> 172.21.15.43:0/2429943328 conn(0x7fc7f4077630 msgr2=0x7fc7f4077a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:33.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.920+0000 7fc7faa00640 1 -- 172.21.15.43:0/2429943328 shutdown_connections 2024-03-28T22:41:33.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.920+0000 7fc7faa00640 1 -- 172.21.15.43:0/2429943328 wait complete. 2024-03-28T22:41:33.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.921+0000 7fc7faa00640 1 Processor -- start 2024-03-28T22:41:33.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.922+0000 7fc7faa00640 1 -- start start 2024-03-28T22:41:33.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.922+0000 7fc7faa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7f4069cc0 0x7fc7f4197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.922+0000 7fc7faa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7f406a640 0x7fc7f4198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.922+0000 7fc7faa00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7f4107710 0x7fc7f419c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.922+0000 7fc7faa00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f4111b20 con 0x7fc7f4069cc0 2024-03-28T22:41:33.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.922+0000 7fc7faa00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f41119a0 con 0x7fc7f4107710 2024-03-28T22:41:33.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.922+0000 7fc7faa00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f4111ca0 con 0x7fc7f406a640 2024-03-28T22:41:33.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.922+0000 7fc7f3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7f4069cc0 0x7fc7f4197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.922+0000 7fc7f3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7f4069cc0 0x7fc7f4197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37660/0 (socket says 172.21.15.43:37660) 2024-03-28T22:41:33.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.922+0000 7fc7f3e00640 1 -- 172.21.15.43:0/1483131223 learned_addr learned my addr 172.21.15.43:0/1483131223 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:33.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.922+0000 7fc7f8a00640 1 --2- 172.21.15.43:0/1483131223 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7f4107710 0x7fc7f419c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.922+0000 7fc7f3400640 1 --2- 172.21.15.43:0/1483131223 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7f406a640 0x7fc7f4198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.923+0000 7fc7f3e00640 1 -- 172.21.15.43:0/1483131223 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7f406a640 msgr2=0x7fc7f4198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.923+0000 7fc7f3e00640 1 --2- 172.21.15.43:0/1483131223 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7f406a640 0x7fc7f4198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.923+0000 7fc7f3e00640 1 -- 172.21.15.43:0/1483131223 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7f4107710 msgr2=0x7fc7f419c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.923+0000 7fc7f3e00640 1 --2- 172.21.15.43:0/1483131223 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7f4107710 0x7fc7f419c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.923+0000 7fc7f3e00640 1 -- 172.21.15.43:0/1483131223 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7f419cfd0 con 0x7fc7f4069cc0 2024-03-28T22:41:33.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.923+0000 7fc7f8a00640 1 --2- 172.21.15.43:0/1483131223 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7f4107710 0x7fc7f419c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:41:33.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.923+0000 7fc7f3e00640 1 --2- 172.21.15.43:0/1483131223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7f4069cc0 0x7fc7f4197fe0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc7e4005ab0 tx=0x7fc7e40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:33.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.924+0000 7fc7f0c00640 1 -- 172.21.15.43:0/1483131223 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e4041070 con 0x7fc7f4069cc0 2024-03-28T22:41:33.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.924+0000 7fc7faa00640 1 -- 172.21.15.43:0/1483131223 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7f419d260 con 0x7fc7f4069cc0 2024-03-28T22:41:33.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.924+0000 7fc7f0c00640 1 -- 172.21.15.43:0/1483131223 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7e402ccd0 con 0x7fc7f4069cc0 2024-03-28T22:41:33.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.924+0000 7fc7f0c00640 1 -- 172.21.15.43:0/1483131223 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e4003d10 con 0x7fc7f4069cc0 2024-03-28T22:41:33.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.924+0000 7fc7faa00640 1 -- 172.21.15.43:0/1483131223 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7f41a4b50 con 0x7fc7f4069cc0 2024-03-28T22:41:33.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.927+0000 7fc7f0c00640 1 -- 172.21.15.43:0/1483131223 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fc7e4044070 con 0x7fc7f4069cc0 2024-03-28T22:41:33.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.928+0000 7fc7fd799740 1 Processor -- start 2024-03-28T22:41:33.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.928+0000 7fc7fd799740 1 -- start start 2024-03-28T22:41:33.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.928+0000 7fc7fd799740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fdfbefed60 0x55fdfbeff140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.928+0000 7fc7fd799740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fdfbeff710 0x55fdfbf0c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.928+0000 7fc7fd799740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fdfbf0cc90 0x55fdfbf0f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.928+0000 7fc7fd799740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55fdfbe63050 con 0x55fdfbf0cc90 2024-03-28T22:41:33.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.928+0000 7fc7fd799740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55fdfbe0bce0 con 0x55fdfbeff710 2024-03-28T22:41:33.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.928+0000 7fc7fd799740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55fdfbe5f0f0 con 0x55fdfbefed60 2024-03-28T22:41:33.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.928+0000 7fc7f8a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fdfbefed60 0x55fdfbeff140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.929+0000 7fc7f3e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fdfbeff710 0x55fdfbf0c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.929+0000 7fc7f8a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fdfbefed60 0x55fdfbeff140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:50414/0 (socket says 172.21.15.43:50414) 2024-03-28T22:41:33.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.929+0000 7fc7f8a00640 1 -- 172.21.15.43:0/1972680736 learned_addr learned my addr 172.21.15.43:0/1972680736 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:33.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.929+0000 7fc7f3400640 1 --2- 172.21.15.43:0/1972680736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fdfbf0cc90 0x55fdfbf0f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.929+0000 7fc7f8a00640 1 -- 172.21.15.43:0/1972680736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fdfbeff710 msgr2=0x55fdfbf0c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:33.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.929+0000 7fc7f8a00640 1 --2- 172.21.15.43:0/1972680736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fdfbeff710 0x55fdfbf0c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.929+0000 7fc7f8a00640 1 -- 172.21.15.43:0/1972680736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fdfbf0cc90 msgr2=0x55fdfbf0f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:33.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.929+0000 7fc7f8a00640 1 --2- 172.21.15.43:0/1972680736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fdfbf0cc90 0x55fdfbf0f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.929+0000 7fc7f8a00640 1 -- 172.21.15.43:0/1972680736 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdfbb773f0 con 0x55fdfbefed60 2024-03-28T22:41:33.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.929+0000 7fc7f0c00640 1 --2- 172.21.15.43:0/1483131223 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc7d80a1ef0 0x7fc7d80a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.929+0000 7fc7f8a00640 1 --2- 172.21.15.43:0/1972680736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fdfbefed60 0x55fdfbeff140 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec011790 tx=0x7fc7ec011c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddd7b1f2006bb9ff server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:33.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.929+0000 7fc7f0c00640 1 -- 172.21.15.43:0/1483131223 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fc7e40efed0 con 0x7fc7f4069cc0 2024-03-28T22:41:33.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.930+0000 7fc7f3e00640 1 --2- 172.21.15.43:0/1483131223 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc7d80a1ef0 0x7fc7d80a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.930+0000 7fc7e9200640 1 -- 172.21.15.43:0/1972680736 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ec003c30 con 0x55fdfbefed60 2024-03-28T22:41:33.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.930+0000 7fc7e9200640 1 -- 172.21.15.43:0/1972680736 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7ec00fe80 con 0x55fdfbefed60 2024-03-28T22:41:33.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.930+0000 7fc7e9200640 1 -- 172.21.15.43:0/1972680736 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ec0193e0 con 0x55fdfbefed60 2024-03-28T22:41:33.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.930+0000 7fc7fd799740 1 -- 172.21.15.43:0/1972680736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fdfbefed60 msgr2=0x55fdfbeff140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.930+0000 7fc7f3e00640 1 --2- 172.21.15.43:0/1483131223 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc7d80a1ef0 0x7fc7d80a43b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc7e40bc610 tx=0x7fc7e40dc040 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:33.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.930+0000 7fc7fd799740 1 --2- 172.21.15.43:0/1972680736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fdfbefed60 0x55fdfbeff140 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec011790 tx=0x7fc7ec011c50 comp rx=0 tx=0).stop 2024-03-28T22:41:33.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.931+0000 7fc7fd799740 1 -- 172.21.15.43:0/1972680736 shutdown_connections 2024-03-28T22:41:33.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.931+0000 7fc7fd799740 1 --2- 172.21.15.43:0/1972680736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fdfbeff710 0x55fdfbf0c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.931+0000 7fc7fd799740 1 --2- 172.21.15.43:0/1972680736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fdfbf0cc90 0x55fdfbf0f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.931+0000 7fc7fd799740 1 --2- 172.21.15.43:0/1972680736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fdfbefed60 0x55fdfbeff140 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.931+0000 7fc7fd799740 1 -- 172.21.15.43:0/1972680736 >> 172.21.15.43:0/1972680736 conn(0x55fdfbef9ab0 msgr2=0x55fdfbef9e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.931+0000 7fc7fd799740 1 -- 172.21.15.43:0/1972680736 shutdown_connections 2024-03-28T22:41:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.931+0000 7fc7fd799740 1 -- 172.21.15.43:0/1972680736 wait complete. 2024-03-28T22:41:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.931+0000 7fc7fd799740 1 -- start start 2024-03-28T22:41:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.931+0000 7fc7fd799740 1 Processor -- start 2024-03-28T22:41:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.932+0000 7fc7fd799740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fdfbefed60 0x55fdfbf09c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.932+0000 7fc7fd799740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fdfbeff710 0x55fdfbf0a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.932+0000 7fc7fd799740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fdfbf0cc90 0x55fdfbf0a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.932+0000 7fc7fd799740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55fdfbf0fa50 con 0x55fdfbeff710 2024-03-28T22:41:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.932+0000 7fc7fd799740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55fdfbe012f0 con 0x55fdfbefed60 2024-03-28T22:41:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.932+0000 7fc7fd799740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55fdfbf0fbd0 con 0x55fdfbf0cc90 2024-03-28T22:41:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.932+0000 7fc7fd799740 10 client.0 _ll_register_callbacks cb 0x55fdfbb76be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:41:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.932+0000 7fc7f8a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fdfbefed60 0x55fdfbf09c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.932+0000 7fc7f8a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fdfbefed60 0x55fdfbf09c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33872/0 (socket says 172.21.15.43:33872) 2024-03-28T22:41:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.932+0000 7fc7f8a00640 1 -- 172.21.15.43:0/3868813642 learned_addr learned my addr 172.21.15.43:0/3868813642 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.932+0000 7fc7f3400640 1 --2- 172.21.15.43:0/3868813642 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fdfbeff710 0x55fdfbf0a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.932+0000 7fc7f8a00640 1 --2- 172.21.15.43:0/3868813642 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fdfbf0cc90 0x55fdfbf0a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.933+0000 7fc7f8a00640 1 -- 172.21.15.43:0/3868813642 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fdfbf0cc90 msgr2=0x55fdfbf0a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.933+0000 7fc7f8a00640 1 --2- 172.21.15.43:0/3868813642 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fdfbf0cc90 0x55fdfbf0a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.933+0000 7fc7f8a00640 1 -- 172.21.15.43:0/3868813642 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fdfbeff710 msgr2=0x55fdfbf0a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.933+0000 7fc7f8a00640 1 --2- 172.21.15.43:0/3868813642 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fdfbeff710 0x55fdfbf0a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.933+0000 7fc7f8a00640 1 -- 172.21.15.43:0/3868813642 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdfbc417d0 con 0x55fdfbefed60 2024-03-28T22:41:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.933+0000 7fc7f8a00640 1 --2- 172.21.15.43:0/3868813642 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fdfbefed60 0x55fdfbf09c20 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec0046c0 tx=0x7fc7ec0045c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=218b9f5124e3261f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.933+0000 7fc7dd600640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:41:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.933+0000 7fc7dd600640 1 -- 172.21.15.43:0/3868813642 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ec01f040 con 0x55fdfbefed60 2024-03-28T22:41:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.933+0000 7fc7fd799740 1 -- 172.21.15.43:0/3868813642 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fdfbe0c680 con 0x55fdfbefed60 2024-03-28T22:41:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.933+0000 7fc7dd600640 1 -- 172.21.15.43:0/3868813642 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fc7ec004c40 con 0x55fdfbefed60 2024-03-28T22:41:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.934+0000 7fc7dd600640 1 -- 172.21.15.43:0/3868813642 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc7ec035050 con 0x55fdfbefed60 2024-03-28T22:41:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.934+0000 7fc7fd799740 10 client.27326 fetch_fsmap learned FSMap version 32 2024-03-28T22:41:33.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.934+0000 7fc7fd799740 1 -- 172.21.15.43:0/3868813642 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55fdfbf1bcf0 con 0x55fdfbefed60 2024-03-28T22:41:33.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.934+0000 7fc7dd600640 1 -- 172.21.15.43:0/3868813642 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55fdfbe0c680 con 0x55fdfbefed60 2024-03-28T22:41:33.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.934+0000 7fc7dd600640 1 -- 172.21.15.43:0/3868813642 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc7ec018070 con 0x55fdfbefed60 2024-03-28T22:41:33.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.935+0000 7fc7fd799740 10 client.27326 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:41:33.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.935+0000 7fc7fd799740 10 client.27326 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:41:33.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.935+0000 7fc7fd799740 20 client.27326 populate_metadata read hostname 'smithi043' 2024-03-28T22:41:33.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.935+0000 7fc7fd799740 1 --2- 172.21.15.43:0/3868813642 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55fdfbf1d100 0x55fdfbf1f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.935+0000 7fc7fd799740 4 client.27326 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:41:33.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.935+0000 7fc7fd799740 1 -- 172.21.15.43:0/3868813642 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fdfbe70590 con 0x55fdfbf1d100 2024-03-28T22:41:33.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.935+0000 7fc7f3400640 1 --2- 172.21.15.43:0/3868813642 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55fdfbf1d100 0x55fdfbf1f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.936+0000 7fc7f3400640 1 --2- 172.21.15.43:0/3868813642 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55fdfbf1d100 0x55fdfbf1f4f0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c5fbd80a1a5cec76 server_cookie=c0383a9c56421ac0 in_seq=0 out_seq=0 2024-03-28T22:41:33.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.936+0000 7fc7dd600640 10 client.27326 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:41:33.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.937+0000 7fc7dd600640 1 -- 172.21.15.43:0/3868813642 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fdfbe70590 con 0x55fdfbf1d100 2024-03-28T22:41:33.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.937+0000 7fc7dd600640 10 client.27326 handle_command_reply: tid=0 2024-03-28T22:41:33.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.937+0000 7fc7fd799740 1 client.27326 shutdown 2024-03-28T22:41:33.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.937+0000 7fc7fd799740 1 -- 172.21.15.43:0/3868813642 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fdfbefed60 msgr2=0x55fdfbf09c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.937+0000 7fc7fd799740 1 --2- 172.21.15.43:0/3868813642 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fdfbefed60 0x55fdfbf09c20 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec0046c0 tx=0x7fc7ec0045c0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.937+0000 7fc7fd799740 1 -- 172.21.15.43:0/3868813642 shutdown_connections 2024-03-28T22:41:33.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.937+0000 7fc7fd799740 1 --2- 172.21.15.43:0/3868813642 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55fdfbf1d100 0x55fdfbf1f4f0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.937+0000 7fc7fd799740 1 --2- 172.21.15.43:0/3868813642 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fdfbf0cc90 0x55fdfbf0a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.937+0000 7fc7fd799740 1 --2- 172.21.15.43:0/3868813642 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fdfbeff710 0x55fdfbf0a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.937+0000 7fc7fd799740 1 --2- 172.21.15.43:0/3868813642 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fdfbefed60 0x55fdfbf09c20 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.937+0000 7fc7fd799740 1 -- 172.21.15.43:0/3868813642 >> 172.21.15.43:0/3868813642 conn(0x55fdfbef9ab0 msgr2=0x55fdfbefb800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:33.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.937+0000 7fc7dd600640 0 client.27326 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:41:33.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.937+0000 7fc7fd799740 1 -- 172.21.15.43:0/3868813642 shutdown_connections 2024-03-28T22:41:33.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.937+0000 7fc7fd799740 1 -- 172.21.15.43:0/3868813642 wait complete. 2024-03-28T22:41:33.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.938+0000 7fc7fd799740 20 client.27326 trim_cache size 0 max 16384 2024-03-28T22:41:33.948 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:33 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/956325153' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:41:33.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.954+0000 7fc7fd799740 1 Processor -- start 2024-03-28T22:41:33.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.954+0000 7fc7fd799740 1 -- start start 2024-03-28T22:41:33.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.954+0000 7fc7fd799740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fdfbefed60 0x55fdfbf0c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.955+0000 7fc7fd799740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fdfbeff710 0x55fdfbf18890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.955+0000 7fc7fd799740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fdfbf0cc90 0x55fdfbf24d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.955+0000 7fc7fd799740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f41ac9f0 con 0x55fdfbeff710 2024-03-28T22:41:33.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.955+0000 7fc7fd799740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f41ae530 con 0x55fdfbefed60 2024-03-28T22:41:33.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.955+0000 7fc7fd799740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f41ad0c0 con 0x55fdfbf0cc90 2024-03-28T22:41:33.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.955+0000 7fc7f8a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fdfbefed60 0x55fdfbf0c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.955+0000 7fc7f8a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fdfbefed60 0x55fdfbf0c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33880/0 (socket says 172.21.15.43:33880) 2024-03-28T22:41:33.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.955+0000 7fc7f8a00640 1 -- 172.21.15.43:0/925796007 learned_addr learned my addr 172.21.15.43:0/925796007 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:33.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.955+0000 7fc7f8a00640 1 --2- 172.21.15.43:0/925796007 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fdfbf0cc90 0x55fdfbf24d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.955+0000 7fc7f3400640 1 --2- 172.21.15.43:0/925796007 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fdfbeff710 0x55fdfbf18890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.955+0000 7fc7f8a00640 1 -- 172.21.15.43:0/925796007 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fdfbf0cc90 msgr2=0x55fdfbf24d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:33.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.955+0000 7fc7f8a00640 1 --2- 172.21.15.43:0/925796007 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fdfbf0cc90 0x55fdfbf24d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.955+0000 7fc7f8a00640 1 -- 172.21.15.43:0/925796007 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fdfbeff710 msgr2=0x55fdfbf18890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:33.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.955+0000 7fc7f8a00640 1 --2- 172.21.15.43:0/925796007 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fdfbeff710 0x55fdfbf18890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.955+0000 7fc7f8a00640 1 -- 172.21.15.43:0/925796007 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdfbef98e0 con 0x55fdfbefed60 2024-03-28T22:41:33.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.956+0000 7fc7f8a00640 1 --2- 172.21.15.43:0/925796007 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fdfbefed60 0x55fdfbf0c350 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec0055a0 tx=0x7fc7ec011c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=476a4af20914cfce server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:33.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.956+0000 7fc7dfe00640 1 -- 172.21.15.43:0/925796007 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ec00f790 con 0x55fdfbefed60 2024-03-28T22:41:33.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.956+0000 7fc7dfe00640 1 -- 172.21.15.43:0/925796007 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7ec005980 con 0x55fdfbefed60 2024-03-28T22:41:33.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.956+0000 7fc7dfe00640 1 -- 172.21.15.43:0/925796007 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ec004c20 con 0x55fdfbefed60 2024-03-28T22:41:33.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.957+0000 7fc7fd799740 1 -- 172.21.15.43:0/925796007 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fdfbefed60 msgr2=0x55fdfbf0c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.957+0000 7fc7fd799740 1 --2- 172.21.15.43:0/925796007 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fdfbefed60 0x55fdfbf0c350 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec0055a0 tx=0x7fc7ec011c50 comp rx=0 tx=0).stop 2024-03-28T22:41:33.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.957+0000 7fc7fd799740 1 -- 172.21.15.43:0/925796007 shutdown_connections 2024-03-28T22:41:33.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.957+0000 7fc7fd799740 1 --2- 172.21.15.43:0/925796007 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fdfbf0cc90 0x55fdfbf24d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.957+0000 7fc7fd799740 1 --2- 172.21.15.43:0/925796007 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fdfbeff710 0x55fdfbf18890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.957+0000 7fc7fd799740 1 --2- 172.21.15.43:0/925796007 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fdfbefed60 0x55fdfbf0c350 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.957+0000 7fc7fd799740 1 -- 172.21.15.43:0/925796007 >> 172.21.15.43:0/925796007 conn(0x55fdfbef9ab0 msgr2=0x55fdfbefb790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:33.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.957+0000 7fc7fd799740 1 -- 172.21.15.43:0/925796007 shutdown_connections 2024-03-28T22:41:33.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.957+0000 7fc7fd799740 1 -- 172.21.15.43:0/925796007 wait complete. 2024-03-28T22:41:33.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.957+0000 7fc7fd799740 1 -- start start 2024-03-28T22:41:33.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.958+0000 7fc7fd799740 1 Processor -- start 2024-03-28T22:41:33.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.958+0000 7fc7fd799740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fdfbefed60 0x55fdfbf152b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.958+0000 7fc7fd799740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fdfbeff710 0x55fdfbf06640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.958+0000 7fc7fd799740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fdfbf0cc90 0x55fdfbf06d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.958+0000 7fc7fd799740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f41ade60 con 0x55fdfbf0cc90 2024-03-28T22:41:33.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.958+0000 7fc7fd799740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f41ad790 con 0x55fdfbefed60 2024-03-28T22:41:33.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.958+0000 7fc7fd799740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f41aa9d0 con 0x55fdfbeff710 2024-03-28T22:41:33.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.958+0000 7fc7fd799740 10 client.0 _ll_register_callbacks cb 0x7fc7e40f3850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:41:33.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.958+0000 7fc7f3400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fdfbeff710 0x55fdfbf06640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.958+0000 7fc7f8a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fdfbefed60 0x55fdfbf152b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.958+0000 7fc7f3400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fdfbeff710 0x55fdfbf06640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:50434/0 (socket says 172.21.15.43:50434) 2024-03-28T22:41:33.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.958+0000 7fc7f3400640 1 -- 172.21.15.43:0/899403015 learned_addr learned my addr 172.21.15.43:0/899403015 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:33.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.959+0000 7fc7f8a00640 1 --2- 172.21.15.43:0/899403015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fdfbf0cc90 0x55fdfbf06d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.959+0000 7fc7f3400640 1 -- 172.21.15.43:0/899403015 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fdfbefed60 msgr2=0x55fdfbf152b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:33.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.959+0000 7fc7f3400640 1 --2- 172.21.15.43:0/899403015 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fdfbefed60 0x55fdfbf152b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.959+0000 7fc7f3400640 1 -- 172.21.15.43:0/899403015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fdfbf0cc90 msgr2=0x55fdfbf06d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:33.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.959+0000 7fc7f3400640 1 --2- 172.21.15.43:0/899403015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fdfbf0cc90 0x55fdfbf06d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.959+0000 7fc7f3400640 1 -- 172.21.15.43:0/899403015 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdfbefe470 con 0x55fdfbeff710 2024-03-28T22:41:33.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.959+0000 7fc7f3400640 1 --2- 172.21.15.43:0/899403015 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fdfbeff710 0x55fdfbf06640 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0005d80 tx=0x7fc7e000bc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4271432daa1efe67 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:33.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.959+0000 7fc7dd600640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:41:33.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.959+0000 7fc7dd600640 1 -- 172.21.15.43:0/899403015 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e000be60 con 0x55fdfbeff710 2024-03-28T22:41:33.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.959+0000 7fc7fd799740 1 -- 172.21.15.43:0/899403015 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7ec005980 con 0x55fdfbeff710 2024-03-28T22:41:33.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.959+0000 7fc7dd600640 1 -- 172.21.15.43:0/899403015 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fc7e001d7c0 con 0x55fdfbeff710 2024-03-28T22:41:33.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.960+0000 7fc7dd600640 1 -- 172.21.15.43:0/899403015 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc7e002d050 con 0x55fdfbeff710 2024-03-28T22:41:33.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.960+0000 7fc7fd799740 10 client.27286 fetch_fsmap learned FSMap version 32 2024-03-28T22:41:33.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.960+0000 7fc7fd799740 1 -- 172.21.15.43:0/899403015 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55fdfbf833c0 con 0x55fdfbeff710 2024-03-28T22:41:33.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.960+0000 7fc7dd600640 1 -- 172.21.15.43:0/899403015 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7ec005980 con 0x55fdfbeff710 2024-03-28T22:41:33.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.960+0000 7fc7dd600640 1 -- 172.21.15.43:0/899403015 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc7e0026bd0 con 0x55fdfbeff710 2024-03-28T22:41:33.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.960+0000 7fc7fd799740 10 client.27286 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:41:33.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.960+0000 7fc7fd799740 10 client.27286 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:41:33.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.960+0000 7fc7fd799740 20 client.27286 populate_metadata read hostname 'smithi043' 2024-03-28T22:41:33.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.960+0000 7fc7fd799740 1 --2- 172.21.15.43:0/899403015 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55fdfbf83b20 0x55fdfbf85f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:33.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.960+0000 7fc7fd799740 4 client.27286 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:41:33.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.960+0000 7fc7fd799740 1 -- 172.21.15.43:0/899403015 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fdfbe6a8e0 con 0x55fdfbf83b20 2024-03-28T22:41:33.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.960+0000 7fc7f8a00640 1 --2- 172.21.15.43:0/899403015 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55fdfbf83b20 0x55fdfbf85f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:33.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.961+0000 7fc7f8a00640 1 --2- 172.21.15.43:0/899403015 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55fdfbf83b20 0x55fdfbf85f40 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3dd7ab70bd132466 server_cookie=9162de639e6393ef in_seq=0 out_seq=0 2024-03-28T22:41:33.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.961+0000 7fc7dd600640 10 client.27286 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:41:33.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.961+0000 7fc7dd600640 1 -- 172.21.15.43:0/899403015 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fdfbe6a8e0 con 0x55fdfbf83b20 2024-03-28T22:41:33.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.961+0000 7fc7dd600640 10 client.27286 handle_command_reply: tid=0 2024-03-28T22:41:33.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.961+0000 7fc7fd799740 1 client.27286 shutdown 2024-03-28T22:41:33.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.961+0000 7fc7fd799740 1 -- 172.21.15.43:0/899403015 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fdfbeff710 msgr2=0x55fdfbf06640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.961+0000 7fc7fd799740 1 --2- 172.21.15.43:0/899403015 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fdfbeff710 0x55fdfbf06640 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0005d80 tx=0x7fc7e000bc70 comp rx=0 tx=0).stop 2024-03-28T22:41:33.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.961+0000 7fc7fd799740 1 -- 172.21.15.43:0/899403015 shutdown_connections 2024-03-28T22:41:33.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.961+0000 7fc7fd799740 1 --2- 172.21.15.43:0/899403015 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55fdfbf83b20 0x55fdfbf85f40 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.961+0000 7fc7fd799740 1 --2- 172.21.15.43:0/899403015 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fdfbf0cc90 0x55fdfbf06d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.961+0000 7fc7fd799740 1 --2- 172.21.15.43:0/899403015 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fdfbeff710 0x55fdfbf06640 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.961+0000 7fc7fd799740 1 --2- 172.21.15.43:0/899403015 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fdfbefed60 0x55fdfbf152b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.961+0000 7fc7dd600640 0 client.27286 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:41:33.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.961+0000 7fc7fd799740 1 -- 172.21.15.43:0/899403015 >> 172.21.15.43:0/899403015 conn(0x55fdfbef9ab0 msgr2=0x55fdfbef82e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:33.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.962+0000 7fc7fd799740 1 -- 172.21.15.43:0/899403015 shutdown_connections 2024-03-28T22:41:33.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.962+0000 7fc7fd799740 1 -- 172.21.15.43:0/899403015 wait complete. 2024-03-28T22:41:33.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.962+0000 7fc7fd799740 20 client.27286 trim_cache size 0 max 16384 2024-03-28T22:41:33.963 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:41:33.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.962+0000 7fc7faa00640 1 -- 172.21.15.43:0/1483131223 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc7d80a1ef0 msgr2=0x7fc7d80a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.962+0000 7fc7faa00640 1 --2- 172.21.15.43:0/1483131223 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc7d80a1ef0 0x7fc7d80a43b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc7e40bc610 tx=0x7fc7e40dc040 comp rx=0 tx=0).stop 2024-03-28T22:41:33.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.962+0000 7fc7faa00640 1 -- 172.21.15.43:0/1483131223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7f4069cc0 msgr2=0x7fc7f4197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:33.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.962+0000 7fc7faa00640 1 --2- 172.21.15.43:0/1483131223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7f4069cc0 0x7fc7f4197fe0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc7e4005ab0 tx=0x7fc7e40042f0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.962+0000 7fc7faa00640 1 -- 172.21.15.43:0/1483131223 shutdown_connections 2024-03-28T22:41:33.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.962+0000 7fc7faa00640 1 --2- 172.21.15.43:0/1483131223 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc7d80a1ef0 0x7fc7d80a43b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.962+0000 7fc7faa00640 1 --2- 172.21.15.43:0/1483131223 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc7f4107710 0x7fc7f419c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.962+0000 7fc7faa00640 1 --2- 172.21.15.43:0/1483131223 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7f406a640 0x7fc7f4198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.963+0000 7fc7faa00640 1 --2- 172.21.15.43:0/1483131223 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc7f4069cc0 0x7fc7f4197fe0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:33.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.963+0000 7fc7faa00640 1 -- 172.21.15.43:0/1483131223 >> 172.21.15.43:0/1483131223 conn(0x7fc7f4077630 msgr2=0x7fc7f40fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:33.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.963+0000 7fc7faa00640 1 -- 172.21.15.43:0/1483131223 shutdown_connections 2024-03-28T22:41:33.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:33.963+0000 7fc7faa00640 1 -- 172.21.15.43:0/1483131223 wait complete. 2024-03-28T22:41:33.976 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:33 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/956325153' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:41:33.980 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:41:34.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:33 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/956325153' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:41:34.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.240+0000 7fb02f400640 1 -- 172.21.15.43:0/1935754816 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb028109900 msgr2=0x7fb028109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:34.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.240+0000 7fb02f400640 1 --2- 172.21.15.43:0/1935754816 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb028109900 0x7fb028109ce0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fb01c009930 tx=0x7fb01c02bc20 comp rx=0 tx=0).stop 2024-03-28T22:41:34.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.240+0000 7fb02f400640 1 -- 172.21.15.43:0/1935754816 shutdown_connections 2024-03-28T22:41:34.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.240+0000 7fb02f400640 1 --2- 172.21.15.43:0/1935754816 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb0281038f0 0x7fb028103d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:34.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.240+0000 7fb02f400640 1 --2- 172.21.15.43:0/1935754816 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb0281042b0 0x7fb0281107c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:34.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.240+0000 7fb02f400640 1 --2- 172.21.15.43:0/1935754816 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb028109900 0x7fb028109ce0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:34.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.240+0000 7fb02f400640 1 -- 172.21.15.43:0/1935754816 >> 172.21.15.43:0/1935754816 conn(0x7fb0280ff3c0 msgr2=0x7fb0281017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:34.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.240+0000 7fb02f400640 1 -- 172.21.15.43:0/1935754816 shutdown_connections 2024-03-28T22:41:34.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.240+0000 7fb02f400640 1 -- 172.21.15.43:0/1935754816 wait complete. 2024-03-28T22:41:34.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.242+0000 7fb02f400640 1 Processor -- start 2024-03-28T22:41:34.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.242+0000 7fb02f400640 1 -- start start 2024-03-28T22:41:34.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.242+0000 7fb02f400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0281038f0 0x7fb02819a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:34.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.242+0000 7fb02f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb0281042b0 0x7fb02819a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:34.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.242+0000 7fb02f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb028109900 0x7fb02819eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:34.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.242+0000 7fb02f400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb028113cd0 con 0x7fb028109900 2024-03-28T22:41:34.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.242+0000 7fb02f400640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb028113b50 con 0x7fb0281042b0 2024-03-28T22:41:34.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.242+0000 7fb02f400640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fb028113e50 con 0x7fb0281038f0 2024-03-28T22:41:34.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.242+0000 7fb027e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb0281042b0 0x7fb02819a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:34.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.242+0000 7fb02ca00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0281038f0 0x7fb02819a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:34.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.242+0000 7fb027e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb0281042b0 0x7fb02819a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33922/0 (socket says 172.21.15.43:33922) 2024-03-28T22:41:34.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.242+0000 7fb027e00640 1 -- 172.21.15.43:0/1837800946 learned_addr learned my addr 172.21.15.43:0/1837800946 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:34.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.242+0000 7fb02ca00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0281038f0 0x7fb02819a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:50462/0 (socket says 172.21.15.43:50462) 2024-03-28T22:41:34.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.243+0000 7fb02d400640 1 --2- 172.21.15.43:0/1837800946 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb028109900 0x7fb02819eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.243+0000 7fb027e00640 1 -- 172.21.15.43:0/1837800946 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0281038f0 msgr2=0x7fb02819a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.243+0000 7fb027e00640 1 --2- 172.21.15.43:0/1837800946 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0281038f0 0x7fb02819a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.243+0000 7fb027e00640 1 -- 172.21.15.43:0/1837800946 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb028109900 msgr2=0x7fb02819eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.243+0000 7fb027e00640 1 --2- 172.21.15.43:0/1837800946 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb028109900 0x7fb02819eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.243+0000 7fb027e00640 1 -- 172.21.15.43:0/1837800946 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb02819f240 con 0x7fb0281042b0 2024-03-28T22:41:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.243+0000 7fb02ca00640 1 --2- 172.21.15.43:0/1837800946 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0281038f0 0x7fb02819a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:41:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.243+0000 7fb02d400640 1 --2- 172.21.15.43:0/1837800946 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb028109900 0x7fb02819eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:41:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.243+0000 7fb027e00640 1 --2- 172.21.15.43:0/1837800946 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb0281042b0 0x7fb02819a790 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fb01400ea60 tx=0x7fb01400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.243+0000 7fb025600640 1 -- 172.21.15.43:0/1837800946 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb01400cde0 con 0x7fb0281042b0 2024-03-28T22:41:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.244+0000 7fb025600640 1 -- 172.21.15.43:0/1837800946 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0140035f0 con 0x7fb0281042b0 2024-03-28T22:41:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.244+0000 7fb02f400640 1 -- 172.21.15.43:0/1837800946 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb02819f530 con 0x7fb0281042b0 2024-03-28T22:41:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.244+0000 7fb025600640 1 -- 172.21.15.43:0/1837800946 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb014010690 con 0x7fb0281042b0 2024-03-28T22:41:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.244+0000 7fb02f400640 1 -- 172.21.15.43:0/1837800946 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0281a6dc0 con 0x7fb0281042b0 2024-03-28T22:41:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.247+0000 7fb032228740 1 Processor -- start 2024-03-28T22:41:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.247+0000 7fb032228740 1 -- start start 2024-03-28T22:41:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.247+0000 7fb025600640 1 -- 172.21.15.43:0/1837800946 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fb0140037a0 con 0x7fb0281042b0 2024-03-28T22:41:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.247+0000 7fb032228740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55711484f380 0x55711484f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.247+0000 7fb032228740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711484fca0 0x557114858340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.247+0000 7fb032228740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557114858880 0x55711485ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.247+0000 7fb032228740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5571147b3050 con 0x55711484f380 2024-03-28T22:41:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.247+0000 7fb032228740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55711475bce0 con 0x55711484fca0 2024-03-28T22:41:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.247+0000 7fb032228740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5571147af0f0 con 0x557114858880 2024-03-28T22:41:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.248+0000 7fb02ca00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711484fca0 0x557114858340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.248+0000 7fb027e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557114858880 0x55711485ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.248+0000 7fb02ca00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711484fca0 0x557114858340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33930/0 (socket says 172.21.15.43:33930) 2024-03-28T22:41:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.248+0000 7fb02ca00640 1 -- 172.21.15.43:0/1838393605 learned_addr learned my addr 172.21.15.43:0/1838393605 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.248+0000 7fb02d400640 1 --2- 172.21.15.43:0/1838393605 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55711484f380 0x55711484f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.248+0000 7fb027e00640 1 -- 172.21.15.43:0/1838393605 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711484fca0 msgr2=0x557114858340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.248+0000 7fb027e00640 1 --2- 172.21.15.43:0/1838393605 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711484fca0 0x557114858340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.248+0000 7fb027e00640 1 -- 172.21.15.43:0/1838393605 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55711484f380 msgr2=0x55711484f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.248+0000 7fb027e00640 1 --2- 172.21.15.43:0/1838393605 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55711484f380 0x55711484f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.248+0000 7fb027e00640 1 -- 172.21.15.43:0/1838393605 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571144c73f0 con 0x557114858880 2024-03-28T22:41:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.249+0000 7fb027e00640 1 --2- 172.21.15.43:0/1838393605 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557114858880 0x55711485ac70 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb0140cbba0 tx=0x7fb0140d4ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab204f3dffe65ea5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.249+0000 7fb019c00640 1 -- 172.21.15.43:0/1838393605 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0140dd070 con 0x557114858880 2024-03-28T22:41:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.249+0000 7fb019c00640 1 -- 172.21.15.43:0/1838393605 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0140d6de0 con 0x557114858880 2024-03-28T22:41:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.249+0000 7fb019c00640 1 -- 172.21.15.43:0/1838393605 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0140d97e0 con 0x557114858880 2024-03-28T22:41:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.249+0000 7fb025600640 1 --2- 172.21.15.43:0/1837800946 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb0100a1f70 0x7fb0100a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.249+0000 7fb025600640 1 -- 172.21.15.43:0/1837800946 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fb0140ce850 con 0x7fb0281042b0 2024-03-28T22:41:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.250+0000 7fb032228740 1 -- 172.21.15.43:0/1838393605 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557114858880 msgr2=0x55711485ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.250+0000 7fb032228740 1 --2- 172.21.15.43:0/1838393605 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557114858880 0x55711485ac70 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb0140cbba0 tx=0x7fb0140d4ef0 comp rx=0 tx=0).stop 2024-03-28T22:41:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.250+0000 7fb02d400640 1 --2- 172.21.15.43:0/1837800946 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb0100a1f70 0x7fb0100a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.250+0000 7fb032228740 1 -- 172.21.15.43:0/1838393605 shutdown_connections 2024-03-28T22:41:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.250+0000 7fb032228740 1 --2- 172.21.15.43:0/1838393605 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711484fca0 0x557114858340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.250+0000 7fb032228740 1 --2- 172.21.15.43:0/1838393605 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557114858880 0x55711485ac70 secure :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb0140cbba0 tx=0x7fb0140d4ef0 comp rx=0 tx=0).stop 2024-03-28T22:41:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.250+0000 7fb032228740 1 --2- 172.21.15.43:0/1838393605 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55711484f380 0x55711484f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:34.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.250+0000 7fb032228740 1 -- 172.21.15.43:0/1838393605 >> 172.21.15.43:0/1838393605 conn(0x557114849ab0 msgr2=0x557114849e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:34.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.250+0000 7fb032228740 1 -- 172.21.15.43:0/1838393605 shutdown_connections 2024-03-28T22:41:34.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.250+0000 7fb032228740 1 -- 172.21.15.43:0/1838393605 wait complete. 2024-03-28T22:41:34.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.251+0000 7fb032228740 1 -- start start 2024-03-28T22:41:34.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.251+0000 7fb02d400640 1 --2- 172.21.15.43:0/1837800946 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb0100a1f70 0x7fb0100a4430 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb020005fd0 tx=0x7fb020009500 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:34.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.251+0000 7fb032228740 1 Processor -- start 2024-03-28T22:41:34.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.252+0000 7fb032228740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711484f380 0x5571148559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:34.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.252+0000 7fb032228740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711484fca0 0x557114855ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:34.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.252+0000 7fb032228740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557114856430 0x557114864b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:34.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.252+0000 7fb032228740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55711485b670 con 0x557114856430 2024-03-28T22:41:34.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.252+0000 7fb032228740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5571147512f0 con 0x55711484fca0 2024-03-28T22:41:34.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.252+0000 7fb032228740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55711485b7f0 con 0x55711484f380 2024-03-28T22:41:34.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.252+0000 7fb032228740 10 client.0 _ll_register_callbacks cb 0x5571144c6be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:41:34.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.252+0000 7fb02ca00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711484f380 0x5571148559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:34.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.252+0000 7fb027e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711484fca0 0x557114855ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:34.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.252+0000 7fb02ca00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711484f380 0x5571148559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:50484/0 (socket says 172.21.15.43:50484) 2024-03-28T22:41:34.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.252+0000 7fb02ca00640 1 -- 172.21.15.43:0/1393371446 learned_addr learned my addr 172.21.15.43:0/1393371446 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:34.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.252+0000 7fb02d400640 1 --2- 172.21.15.43:0/1393371446 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557114856430 0x557114864b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:34.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.252+0000 7fb02ca00640 1 -- 172.21.15.43:0/1393371446 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711484fca0 msgr2=0x557114855ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:34.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.252+0000 7fb02ca00640 1 --2- 172.21.15.43:0/1393371446 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711484fca0 0x557114855ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:34.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.252+0000 7fb02ca00640 1 -- 172.21.15.43:0/1393371446 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557114856430 msgr2=0x557114864b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:34.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.252+0000 7fb02ca00640 1 --2- 172.21.15.43:0/1393371446 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557114856430 0x557114864b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:34.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.252+0000 7fb02ca00640 1 -- 172.21.15.43:0/1393371446 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571145917d0 con 0x55711484f380 2024-03-28T22:41:34.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.252+0000 7fb02d400640 1 --2- 172.21.15.43:0/1393371446 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557114856430 0x557114864b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:41:34.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.253+0000 7fb02ca00640 1 --2- 172.21.15.43:0/1393371446 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711484f380 0x5571148559b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb01c02c7f0 tx=0x7fb01c004e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c71e9f0dba413f23 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:34.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.253+0000 7fb00e000640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:41:34.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.253+0000 7fb00e000640 1 -- 172.21.15.43:0/1393371446 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb01c004250 con 0x55711484f380 2024-03-28T22:41:34.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.253+0000 7fb032228740 1 -- 172.21.15.43:0/1393371446 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0140d6de0 con 0x55711484f380 2024-03-28T22:41:34.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.253+0000 7fb00e000640 1 -- 172.21.15.43:0/1393371446 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fb01c0043f0 con 0x55711484f380 2024-03-28T22:41:34.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.253+0000 7fb00e000640 1 -- 172.21.15.43:0/1393371446 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb01c02ccc0 con 0x55711484f380 2024-03-28T22:41:34.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.253+0000 7fb032228740 10 client.27298 fetch_fsmap learned FSMap version 32 2024-03-28T22:41:34.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.253+0000 7fb032228740 1 -- 172.21.15.43:0/1393371446 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557114867f20 con 0x55711484f380 2024-03-28T22:41:34.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.254+0000 7fb00e000640 1 -- 172.21.15.43:0/1393371446 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0140d6de0 con 0x55711484f380 2024-03-28T22:41:34.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.254+0000 7fb00e000640 1 -- 172.21.15.43:0/1393371446 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb01c003ae0 con 0x55711484f380 2024-03-28T22:41:34.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.254+0000 7fb032228740 10 client.27298 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:41:34.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.254+0000 7fb032228740 10 client.27298 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:41:34.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.254+0000 7fb032228740 20 client.27298 populate_metadata read hostname 'smithi043' 2024-03-28T22:41:34.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.254+0000 7fb032228740 1 --2- 172.21.15.43:0/1393371446 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557114869330 0x55711486b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:34.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.254+0000 7fb032228740 4 client.27298 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:41:34.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.254+0000 7fb032228740 1 -- 172.21.15.43:0/1393371446 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55711475c680 con 0x557114869330 2024-03-28T22:41:34.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.254+0000 7fb027e00640 1 --2- 172.21.15.43:0/1393371446 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557114869330 0x55711486b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:34.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.255+0000 7fb027e00640 1 --2- 172.21.15.43:0/1393371446 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557114869330 0x55711486b780 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6dc05ae5a81f6fa9 server_cookie=47ad557a6adb80a in_seq=0 out_seq=0 2024-03-28T22:41:34.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.255+0000 7fb00e000640 10 client.27298 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:34.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.257+0000 7fb00e000640 1 -- 172.21.15.43:0/1393371446 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55711475c680 con 0x557114869330 2024-03-28T22:41:34.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.257+0000 7fb00e000640 10 client.27298 handle_command_reply: tid=0 2024-03-28T22:41:34.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.257+0000 7fb032228740 1 client.27298 shutdown 2024-03-28T22:41:34.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.257+0000 7fb032228740 1 -- 172.21.15.43:0/1393371446 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711484f380 msgr2=0x5571148559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:34.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.257+0000 7fb032228740 1 --2- 172.21.15.43:0/1393371446 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711484f380 0x5571148559b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb01c02c7f0 tx=0x7fb01c004e30 comp rx=0 tx=0).stop 2024-03-28T22:41:34.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.257+0000 7fb032228740 1 -- 172.21.15.43:0/1393371446 shutdown_connections 2024-03-28T22:41:34.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.257+0000 7fb032228740 1 --2- 172.21.15.43:0/1393371446 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557114869330 0x55711486b780 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:34.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.257+0000 7fb032228740 1 --2- 172.21.15.43:0/1393371446 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711484fca0 0x557114855ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:34.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.257+0000 7fb032228740 1 --2- 172.21.15.43:0/1393371446 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557114856430 0x557114864b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:34.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.257+0000 7fb00e000640 0 client.27298 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:34.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.257+0000 7fb032228740 1 --2- 172.21.15.43:0/1393371446 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711484f380 0x5571148559b0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:34.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.257+0000 7fb032228740 1 -- 172.21.15.43:0/1393371446 >> 172.21.15.43:0/1393371446 conn(0x557114849ab0 msgr2=0x55711484b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:34.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.257+0000 7fb032228740 1 -- 172.21.15.43:0/1393371446 shutdown_connections 2024-03-28T22:41:34.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.257+0000 7fb032228740 1 -- 172.21.15.43:0/1393371446 wait complete. 2024-03-28T22:41:34.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.257+0000 7fb032228740 20 client.27298 trim_cache size 0 max 16384 2024-03-28T22:41:34.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.273+0000 7fb032228740 1 Processor -- start 2024-03-28T22:41:34.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.273+0000 7fb032228740 1 -- start start 2024-03-28T22:41:34.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.273+0000 7fb032228740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711484f380 0x557114871050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:34.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.273+0000 7fb032228740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711484fca0 0x55711484b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:34.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.273+0000 7fb032228740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557114856430 0x557114864440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:34.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.273+0000 7fb032228740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb0281b0f20 con 0x557114856430 2024-03-28T22:41:34.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.273+0000 7fb032228740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb0281b2a60 con 0x55711484f380 2024-03-28T22:41:34.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.273+0000 7fb032228740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fb0281b15f0 con 0x55711484fca0 2024-03-28T22:41:34.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.273+0000 7fb02ca00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711484f380 0x557114871050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:34.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.273+0000 7fb027e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711484fca0 0x55711484b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:34.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.274+0000 7fb02ca00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711484f380 0x557114871050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33950/0 (socket says 172.21.15.43:33950) 2024-03-28T22:41:34.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.274+0000 7fb02ca00640 1 -- 172.21.15.43:0/1795588659 learned_addr learned my addr 172.21.15.43:0/1795588659 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:34.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.274+0000 7fb027e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711484fca0 0x55711484b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:50486/0 (socket says 172.21.15.43:50486) 2024-03-28T22:41:34.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.274+0000 7fb02d400640 1 --2- 172.21.15.43:0/1795588659 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557114856430 0x557114864440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:34.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.274+0000 7fb02ca00640 1 -- 172.21.15.43:0/1795588659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711484fca0 msgr2=0x55711484b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:34.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.274+0000 7fb02ca00640 1 --2- 172.21.15.43:0/1795588659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711484fca0 0x55711484b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:34.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.274+0000 7fb02ca00640 1 -- 172.21.15.43:0/1795588659 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557114856430 msgr2=0x557114864440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:34.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.274+0000 7fb02ca00640 1 --2- 172.21.15.43:0/1795588659 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557114856430 0x557114864440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:34.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.274+0000 7fb02ca00640 1 -- 172.21.15.43:0/1795588659 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557114856250 con 0x55711484f380 2024-03-28T22:41:34.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.274+0000 7fb027e00640 1 --2- 172.21.15.43:0/1795588659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711484fca0 0x55711484b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:41:34.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.274+0000 7fb02ca00640 1 --2- 172.21.15.43:0/1795588659 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711484f380 0x557114871050 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fb01c02c160 tx=0x7fb01c02bc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6a640665f08ed8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:34.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.275+0000 7fb019200640 1 -- 172.21.15.43:0/1795588659 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb01c05a070 con 0x55711484f380 2024-03-28T22:41:34.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.275+0000 7fb019200640 1 -- 172.21.15.43:0/1795588659 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb01c043070 con 0x55711484f380 2024-03-28T22:41:34.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.275+0000 7fb019200640 1 -- 172.21.15.43:0/1795588659 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb01c0035f0 con 0x55711484f380 2024-03-28T22:41:34.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.275+0000 7fb032228740 1 -- 172.21.15.43:0/1795588659 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711484f380 msgr2=0x557114871050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:34.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.275+0000 7fb032228740 1 --2- 172.21.15.43:0/1795588659 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711484f380 0x557114871050 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fb01c02c160 tx=0x7fb01c02bc20 comp rx=0 tx=0).stop 2024-03-28T22:41:34.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.276+0000 7fb032228740 1 -- 172.21.15.43:0/1795588659 shutdown_connections 2024-03-28T22:41:34.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.276+0000 7fb032228740 1 --2- 172.21.15.43:0/1795588659 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557114856430 0x557114864440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:34.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.276+0000 7fb032228740 1 --2- 172.21.15.43:0/1795588659 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711484fca0 0x55711484b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:34.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.276+0000 7fb032228740 1 --2- 172.21.15.43:0/1795588659 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711484f380 0x557114871050 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:34.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.276+0000 7fb032228740 1 -- 172.21.15.43:0/1795588659 >> 172.21.15.43:0/1795588659 conn(0x557114849ab0 msgr2=0x55711484ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:34.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.276+0000 7fb032228740 1 -- 172.21.15.43:0/1795588659 shutdown_connections 2024-03-28T22:41:34.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.276+0000 7fb032228740 1 -- 172.21.15.43:0/1795588659 wait complete. 2024-03-28T22:41:34.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.276+0000 7fb032228740 1 -- start start 2024-03-28T22:41:34.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.276+0000 7fb032228740 1 Processor -- start 2024-03-28T22:41:34.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.277+0000 7fb032228740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711484f380 0x55711485bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:34.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.277+0000 7fb032228740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55711484fca0 0x55711485c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:34.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.277+0000 7fb032228740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557114856430 0x5571148603f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:34.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.277+0000 7fb032228740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb0281b2390 con 0x55711484fca0 2024-03-28T22:41:34.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.277+0000 7fb032228740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb0281b1cc0 con 0x55711484f380 2024-03-28T22:41:34.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.277+0000 7fb032228740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fb0281aef00 con 0x557114856430 2024-03-28T22:41:34.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.277+0000 7fb032228740 10 client.0 _ll_register_callbacks cb 0x7fb01c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:41:34.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.277+0000 7fb02ca00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711484f380 0x55711485bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:34.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.277+0000 7fb02ca00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711484f380 0x55711485bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33958/0 (socket says 172.21.15.43:33958) 2024-03-28T22:41:34.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.277+0000 7fb02ca00640 1 -- 172.21.15.43:0/4035900925 learned_addr learned my addr 172.21.15.43:0/4035900925 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:34.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.277+0000 7fb02d400640 1 --2- 172.21.15.43:0/4035900925 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557114856430 0x5571148603f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:34.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.277+0000 7fb027e00640 1 --2- 172.21.15.43:0/4035900925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55711484fca0 0x55711485c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:34.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.277+0000 7fb02ca00640 1 -- 172.21.15.43:0/4035900925 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557114856430 msgr2=0x5571148603f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:34.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.277+0000 7fb02ca00640 1 --2- 172.21.15.43:0/4035900925 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557114856430 0x5571148603f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:34.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.277+0000 7fb02ca00640 1 -- 172.21.15.43:0/4035900925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55711484fca0 msgr2=0x55711485c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:34.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.277+0000 7fb02ca00640 1 --2- 172.21.15.43:0/4035900925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55711484fca0 0x55711485c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:34.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.277+0000 7fb02ca00640 1 -- 172.21.15.43:0/4035900925 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557114860a60 con 0x55711484f380 2024-03-28T22:41:34.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.278+0000 7fb02ca00640 1 --2- 172.21.15.43:0/4035900925 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711484f380 0x55711485bfd0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fb01c058040 tx=0x7fb01c048a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3d634a938e10d38 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:34.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.278+0000 7fb00e000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:41:34.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.278+0000 7fb00e000640 1 -- 172.21.15.43:0/4035900925 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb01c05a070 con 0x55711484f380 2024-03-28T22:41:34.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.278+0000 7fb032228740 1 -- 172.21.15.43:0/4035900925 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb01c043070 con 0x55711484f380 2024-03-28T22:41:34.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.278+0000 7fb00e000640 1 -- 172.21.15.43:0/4035900925 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fb01c004000 con 0x55711484f380 2024-03-28T22:41:34.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.278+0000 7fb00e000640 1 -- 172.21.15.43:0/4035900925 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb01c066050 con 0x55711484f380 2024-03-28T22:41:34.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.278+0000 7fb032228740 10 client.27350 fetch_fsmap learned FSMap version 32 2024-03-28T22:41:34.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.278+0000 7fb032228740 1 -- 172.21.15.43:0/4035900925 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5571148cf220 con 0x55711484f380 2024-03-28T22:41:34.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.279+0000 7fb00e000640 1 -- 172.21.15.43:0/4035900925 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb01c043070 con 0x55711484f380 2024-03-28T22:41:34.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.279+0000 7fb00e000640 1 -- 172.21.15.43:0/4035900925 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb01c004ef0 con 0x55711484f380 2024-03-28T22:41:34.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.279+0000 7fb032228740 10 client.27350 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:41:34.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.279+0000 7fb032228740 10 client.27350 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:41:34.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.279+0000 7fb032228740 20 client.27350 populate_metadata read hostname 'smithi043' 2024-03-28T22:41:34.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.279+0000 7fb032228740 1 --2- 172.21.15.43:0/4035900925 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5571148cf860 0x5571148d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:34.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.279+0000 7fb032228740 4 client.27350 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:41:34.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.279+0000 7fb032228740 1 -- 172.21.15.43:0/4035900925 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5571147c0590 con 0x5571148cf860 2024-03-28T22:41:34.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.279+0000 7fb027e00640 1 --2- 172.21.15.43:0/4035900925 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5571148cf860 0x5571148d1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:34.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.280+0000 7fb027e00640 1 --2- 172.21.15.43:0/4035900925 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5571148cf860 0x5571148d1c80 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d111fce6a3ba5b9 server_cookie=98c0853eabd82971 in_seq=0 out_seq=0 2024-03-28T22:41:34.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.280+0000 7fb00e000640 10 client.27350 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:34.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.282+0000 7fb00e000640 1 -- 172.21.15.43:0/4035900925 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5571147c0590 con 0x5571148cf860 2024-03-28T22:41:34.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.282+0000 7fb00e000640 10 client.27350 handle_command_reply: tid=0 2024-03-28T22:41:34.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.282+0000 7fb032228740 1 client.27350 shutdown 2024-03-28T22:41:34.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.282+0000 7fb032228740 1 -- 172.21.15.43:0/4035900925 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711484f380 msgr2=0x55711485bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:34.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.282+0000 7fb032228740 1 --2- 172.21.15.43:0/4035900925 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711484f380 0x55711485bfd0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fb01c058040 tx=0x7fb01c048a20 comp rx=0 tx=0).stop 2024-03-28T22:41:34.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.282+0000 7fb032228740 1 -- 172.21.15.43:0/4035900925 shutdown_connections 2024-03-28T22:41:34.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.282+0000 7fb032228740 1 --2- 172.21.15.43:0/4035900925 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557114856430 0x5571148603f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:34.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.282+0000 7fb032228740 1 --2- 172.21.15.43:0/4035900925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55711484fca0 0x55711485c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:34.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.282+0000 7fb032228740 1 --2- 172.21.15.43:0/4035900925 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5571148cf860 0x5571148d1c80 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:34.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.282+0000 7fb032228740 1 --2- 172.21.15.43:0/4035900925 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711484f380 0x55711485bfd0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:34.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.282+0000 7fb032228740 1 -- 172.21.15.43:0/4035900925 >> 172.21.15.43:0/4035900925 conn(0x557114849ab0 msgr2=0x5571148482e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:34.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.282+0000 7fb00e000640 0 client.27350 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:34.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.282+0000 7fb032228740 1 -- 172.21.15.43:0/4035900925 shutdown_connections 2024-03-28T22:41:34.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.282+0000 7fb032228740 1 -- 172.21.15.43:0/4035900925 wait complete. 2024-03-28T22:41:34.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.282+0000 7fb032228740 20 client.27350 trim_cache size 0 max 16384 2024-03-28T22:41:34.284 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:41:34.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.283+0000 7fb02f400640 1 -- 172.21.15.43:0/1837800946 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb0100a1f70 msgr2=0x7fb0100a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:34.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.283+0000 7fb02f400640 1 --2- 172.21.15.43:0/1837800946 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb0100a1f70 0x7fb0100a4430 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb020005fd0 tx=0x7fb020009500 comp rx=0 tx=0).stop 2024-03-28T22:41:34.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.283+0000 7fb02f400640 1 -- 172.21.15.43:0/1837800946 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb0281042b0 msgr2=0x7fb02819a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:34.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.283+0000 7fb02f400640 1 --2- 172.21.15.43:0/1837800946 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb0281042b0 0x7fb02819a790 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fb01400ea60 tx=0x7fb01400ef20 comp rx=0 tx=0).stop 2024-03-28T22:41:34.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.283+0000 7fb02f400640 1 -- 172.21.15.43:0/1837800946 shutdown_connections 2024-03-28T22:41:34.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.283+0000 7fb02f400640 1 --2- 172.21.15.43:0/1837800946 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb0100a1f70 0x7fb0100a4430 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:34.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.283+0000 7fb02f400640 1 --2- 172.21.15.43:0/1837800946 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb0281042b0 0x7fb02819a790 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:34.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.283+0000 7fb02f400640 1 --2- 172.21.15.43:0/1837800946 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb028109900 0x7fb02819eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:34.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.283+0000 7fb02f400640 1 --2- 172.21.15.43:0/1837800946 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb0281038f0 0x7fb02819a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:34.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.283+0000 7fb02f400640 1 -- 172.21.15.43:0/1837800946 >> 172.21.15.43:0/1837800946 conn(0x7fb0280ff3c0 msgr2=0x7fb0280ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:34.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.283+0000 7fb02f400640 1 -- 172.21.15.43:0/1837800946 shutdown_connections 2024-03-28T22:41:34.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:34.283+0000 7fb02f400640 1 -- 172.21.15.43:0/1837800946 wait complete. 2024-03-28T22:41:34.360 INFO:tasks.workunit.client.0.smithi043.stdout: 1 281937 7.99 MB/sec execute 65 sec latency 47.129 ms 2024-03-28T22:41:34.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:34 smithi082 ceph-mon[36471]: pgmap v136: 129 pgs: 129 active+clean; 46 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 349 op/s 2024-03-28T22:41:35.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:34 smithi043 ceph-mon[31823]: pgmap v136: 129 pgs: 129 active+clean; 46 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 349 op/s 2024-03-28T22:41:35.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:34 smithi172 ceph-mon[36566]: pgmap v136: 129 pgs: 129 active+clean; 46 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.1 MiB/s wr, 349 op/s 2024-03-28T22:41:35.299 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:41:35.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:41:35.300 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:41:35.360 INFO:tasks.workunit.client.0.smithi043.stdout: 1 283893 8.06 MB/sec execute 66 sec latency 10.221 ms 2024-03-28T22:41:35.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.552+0000 7f647fc00640 1 -- 172.21.15.43:0/3260608563 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f64781038f0 msgr2=0x7f6478103d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:35.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.552+0000 7f647fc00640 1 --2- 172.21.15.43:0/3260608563 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f64781038f0 0x7f6478103d70 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f6464009900 tx=0x7f646402bc30 comp rx=0 tx=0).stop 2024-03-28T22:41:35.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.553+0000 7f647fc00640 1 -- 172.21.15.43:0/3260608563 shutdown_connections 2024-03-28T22:41:35.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.553+0000 7f647fc00640 1 --2- 172.21.15.43:0/3260608563 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f64781038f0 0x7f6478103d70 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.553+0000 7f647fc00640 1 --2- 172.21.15.43:0/3260608563 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64781042b0 0x7f64781107c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.553+0000 7f647fc00640 1 --2- 172.21.15.43:0/3260608563 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6478109900 0x7f6478109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.553+0000 7f647fc00640 1 -- 172.21.15.43:0/3260608563 >> 172.21.15.43:0/3260608563 conn(0x7f64780ff400 msgr2=0x7f6478101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:35.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.553+0000 7f647fc00640 1 -- 172.21.15.43:0/3260608563 shutdown_connections 2024-03-28T22:41:35.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.553+0000 7f647fc00640 1 -- 172.21.15.43:0/3260608563 wait complete. 2024-03-28T22:41:35.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.554+0000 7f647fc00640 1 Processor -- start 2024-03-28T22:41:35.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.554+0000 7f647fc00640 1 -- start start 2024-03-28T22:41:35.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.554+0000 7f647fc00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64781038f0 0x7f647819a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.554+0000 7f647fc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64781042b0 0x7f647819a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.554+0000 7f647fc00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6478109900 0x7f647819eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.554+0000 7f647fc00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6478113d60 con 0x7f64781042b0 2024-03-28T22:41:35.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.555+0000 7f647fc00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f6478113be0 con 0x7f6478109900 2024-03-28T22:41:35.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.555+0000 7f647fc00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f6478113ee0 con 0x7f64781038f0 2024-03-28T22:41:35.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.555+0000 7f647d200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64781038f0 0x7f647819a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.555+0000 7f647d200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64781038f0 0x7f647819a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43758/0 (socket says 172.21.15.43:43758) 2024-03-28T22:41:35.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.555+0000 7f647d200640 1 -- 172.21.15.43:0/470352156 learned_addr learned my addr 172.21.15.43:0/470352156 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:35.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.555+0000 7f647dc00640 1 --2- 172.21.15.43:0/470352156 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6478109900 0x7f647819eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.555+0000 7f6477e00640 1 --2- 172.21.15.43:0/470352156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64781042b0 0x7f647819a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.555+0000 7f647d200640 1 -- 172.21.15.43:0/470352156 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6478109900 msgr2=0x7f647819eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:35.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.555+0000 7f647d200640 1 --2- 172.21.15.43:0/470352156 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6478109900 0x7f647819eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.555+0000 7f647d200640 1 -- 172.21.15.43:0/470352156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64781042b0 msgr2=0x7f647819a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:35.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.555+0000 7f647d200640 1 --2- 172.21.15.43:0/470352156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64781042b0 0x7f647819a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.555+0000 7f647d200640 1 -- 172.21.15.43:0/470352156 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f647819f290 con 0x7f64781038f0 2024-03-28T22:41:35.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.556+0000 7f647d200640 1 --2- 172.21.15.43:0/470352156 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64781038f0 0x7f647819a2a0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f646c00e9f0 tx=0x7f646c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:35.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.556+0000 7f6475600640 1 -- 172.21.15.43:0/470352156 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f646c00cd80 con 0x7f64781038f0 2024-03-28T22:41:35.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.556+0000 7f6475600640 1 -- 172.21.15.43:0/470352156 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f646c010910 con 0x7f64781038f0 2024-03-28T22:41:35.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.556+0000 7f647fc00640 1 -- 172.21.15.43:0/470352156 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f647819f580 con 0x7f64781038f0 2024-03-28T22:41:35.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.556+0000 7f6475600640 1 -- 172.21.15.43:0/470352156 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f646c018990 con 0x7f64781038f0 2024-03-28T22:41:35.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.556+0000 7f647fc00640 1 -- 172.21.15.43:0/470352156 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64781a6e10 con 0x7f64781038f0 2024-03-28T22:41:35.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.560+0000 7f6475600640 1 -- 172.21.15.43:0/470352156 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f646c010ac0 con 0x7f64781038f0 2024-03-28T22:41:35.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.561+0000 7f648299e740 1 Processor -- start 2024-03-28T22:41:35.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.561+0000 7f648299e740 1 -- start start 2024-03-28T22:41:35.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.561+0000 7f648299e740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639007acf70 0x5639007ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.561+0000 7f648299e740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639007ad890 0x5639007b5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.561+0000 7f648299e740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639007b6470 0x5639007b8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.561+0000 7f648299e740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x563900710c40 con 0x5639007ad890 2024-03-28T22:41:35.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.561+0000 7f648299e740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5639006b9c40 con 0x5639007acf70 2024-03-28T22:41:35.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.561+0000 7f648299e740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x56390070cce0 con 0x5639007b6470 2024-03-28T22:41:35.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.561+0000 7f647dc00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639007acf70 0x5639007ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.561+0000 7f647dc00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639007acf70 0x5639007ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48056/0 (socket says 172.21.15.43:48056) 2024-03-28T22:41:35.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.562+0000 7f647dc00640 1 -- 172.21.15.43:0/114500185 learned_addr learned my addr 172.21.15.43:0/114500185 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:35.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.562+0000 7f6477e00640 1 --2- 172.21.15.43:0/114500185 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639007b6470 0x5639007b8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.562+0000 7f647d200640 1 --2- 172.21.15.43:0/114500185 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639007ad890 0x5639007b5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.562+0000 7f647dc00640 1 -- 172.21.15.43:0/114500185 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639007b6470 msgr2=0x5639007b8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:35.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.562+0000 7f647dc00640 1 --2- 172.21.15.43:0/114500185 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639007b6470 0x5639007b8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.562+0000 7f647dc00640 1 -- 172.21.15.43:0/114500185 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639007ad890 msgr2=0x5639007b5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:35.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.562+0000 7f647dc00640 1 --2- 172.21.15.43:0/114500185 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639007ad890 0x5639007b5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.562+0000 7f647dc00640 1 -- 172.21.15.43:0/114500185 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639004ed7e0 con 0x5639007acf70 2024-03-28T22:41:35.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.562+0000 7f647d200640 1 --2- 172.21.15.43:0/114500185 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639007ad890 0x5639007b5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:41:35.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.562+0000 7f647dc00640 1 --2- 172.21.15.43:0/114500185 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639007acf70 0x5639007ad350 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f647000a430 tx=0x7f6470009650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68bb8eb7c040db37 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:35.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.562+0000 7f6475600640 1 --2- 172.21.15.43:0/470352156 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f64600a1ef0 0x7f64600a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.562+0000 7f646a000640 1 -- 172.21.15.43:0/114500185 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6470013790 con 0x5639007acf70 2024-03-28T22:41:35.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.563+0000 7f646a000640 1 -- 172.21.15.43:0/114500185 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6470019070 con 0x5639007acf70 2024-03-28T22:41:35.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.563+0000 7f646a000640 1 -- 172.21.15.43:0/114500185 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6470014de0 con 0x5639007acf70 2024-03-28T22:41:35.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.563+0000 7f6475600640 1 -- 172.21.15.43:0/470352156 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f646c0ce890 con 0x7f64781038f0 2024-03-28T22:41:35.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.563+0000 7f647d200640 1 --2- 172.21.15.43:0/470352156 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f64600a1ef0 0x7f64600a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.563+0000 7f647d200640 1 --2- 172.21.15.43:0/470352156 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f64600a1ef0 0x7f64600a43b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x5639007aca40 tx=0x7f646c000bc0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:35.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.563+0000 7f648299e740 1 -- 172.21.15.43:0/114500185 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639007acf70 msgr2=0x5639007ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:35.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.563+0000 7f648299e740 1 --2- 172.21.15.43:0/114500185 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639007acf70 0x5639007ad350 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f647000a430 tx=0x7f6470009650 comp rx=0 tx=0).stop 2024-03-28T22:41:35.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.564+0000 7f648299e740 1 -- 172.21.15.43:0/114500185 shutdown_connections 2024-03-28T22:41:35.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.564+0000 7f648299e740 1 --2- 172.21.15.43:0/114500185 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639007b6470 0x5639007b8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.564+0000 7f648299e740 1 --2- 172.21.15.43:0/114500185 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639007ad890 0x5639007b5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.564+0000 7f648299e740 1 --2- 172.21.15.43:0/114500185 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639007acf70 0x5639007ad350 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.564+0000 7f648299e740 1 -- 172.21.15.43:0/114500185 >> 172.21.15.43:0/114500185 conn(0x5639007a76a0 msgr2=0x5639007a7a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:35.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.564+0000 7f648299e740 1 -- 172.21.15.43:0/114500185 shutdown_connections 2024-03-28T22:41:35.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.564+0000 7f648299e740 1 -- 172.21.15.43:0/114500185 wait complete. 2024-03-28T22:41:35.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.564+0000 7f648299e740 1 -- start start 2024-03-28T22:41:35.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.564+0000 7f648299e740 1 Processor -- start 2024-03-28T22:41:35.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.565+0000 7f648299e740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639007acf70 0x5639007b34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.565+0000 7f648299e740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639007ad890 0x5639007b39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.565+0000 7f648299e740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639007b6470 0x5639007b3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.565+0000 7f648299e740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5639007b9260 con 0x5639007acf70 2024-03-28T22:41:35.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.565+0000 7f648299e740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5639006af250 con 0x5639007b6470 2024-03-28T22:41:35.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.565+0000 7f648299e740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5639007b93e0 con 0x5639007ad890 2024-03-28T22:41:35.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.565+0000 7f648299e740 10 client.0 _ll_register_callbacks cb 0x5639006d0e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:41:35.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.565+0000 7f6477e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639007ad890 0x5639007b39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.565+0000 7f6477e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639007ad890 0x5639007b39e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43778/0 (socket says 172.21.15.43:43778) 2024-03-28T22:41:35.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.565+0000 7f6477e00640 1 -- 172.21.15.43:0/1432597656 learned_addr learned my addr 172.21.15.43:0/1432597656 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:35.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.565+0000 7f647dc00640 1 --2- 172.21.15.43:0/1432597656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639007acf70 0x5639007b34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.566+0000 7f647dc00640 1 --2- 172.21.15.43:0/1432597656 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639007b6470 0x5639007b3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.566+0000 7f6477e00640 1 -- 172.21.15.43:0/1432597656 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639007b6470 msgr2=0x5639007b3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:35.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.566+0000 7f6477e00640 1 --2- 172.21.15.43:0/1432597656 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639007b6470 0x5639007b3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.566+0000 7f6477e00640 1 -- 172.21.15.43:0/1432597656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639007acf70 msgr2=0x5639007b34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:35.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.566+0000 7f6477e00640 1 --2- 172.21.15.43:0/1432597656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639007acf70 0x5639007b34a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.566+0000 7f6477e00640 1 -- 172.21.15.43:0/1432597656 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56390054df70 con 0x5639007ad890 2024-03-28T22:41:35.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.566+0000 7f647dc00640 1 --2- 172.21.15.43:0/1432597656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639007acf70 0x5639007b34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:41:35.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.566+0000 7f6477e00640 1 --2- 172.21.15.43:0/1432597656 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639007ad890 0x5639007b39e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6464009900 tx=0x7f646402c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ff2b58c2f112b9a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:35.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.566+0000 7f645ea00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:41:35.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.566+0000 7f645ea00640 1 -- 172.21.15.43:0/1432597656 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6464043070 con 0x5639007ad890 2024-03-28T22:41:35.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.566+0000 7f645ea00640 1 -- 172.21.15.43:0/1432597656 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f646402ccc0 con 0x5639007ad890 2024-03-28T22:41:35.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.566+0000 7f648299e740 1 -- 172.21.15.43:0/1432597656 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6470019070 con 0x5639007ad890 2024-03-28T22:41:35.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.567+0000 7f645ea00640 1 -- 172.21.15.43:0/1432597656 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6464051050 con 0x5639007ad890 2024-03-28T22:41:35.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.567+0000 7f648299e740 10 client.27313 fetch_fsmap learned FSMap version 32 2024-03-28T22:41:35.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.567+0000 7f648299e740 1 -- 172.21.15.43:0/1432597656 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5639003e5170 con 0x5639007ad890 2024-03-28T22:41:35.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.567+0000 7f645ea00640 1 -- 172.21.15.43:0/1432597656 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6470019070 con 0x5639007ad890 2024-03-28T22:41:35.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.567+0000 7f645ea00640 1 -- 172.21.15.43:0/1432597656 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6464005470 con 0x5639007ad890 2024-03-28T22:41:35.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.567+0000 7f648299e740 10 client.27313 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:41:35.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.567+0000 7f648299e740 10 client.27313 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:41:35.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.567+0000 7f648299e740 20 client.27313 populate_metadata read hostname 'smithi043' 2024-03-28T22:41:35.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.567+0000 7f648299e740 1 --2- 172.21.15.43:0/1432597656 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5639007c6870 0x5639007c8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.567+0000 7f648299e740 4 client.27313 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:41:35.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.567+0000 7f648299e740 1 -- 172.21.15.43:0/1432597656 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56390071e180 con 0x5639007c6870 2024-03-28T22:41:35.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.568+0000 7f647dc00640 1 --2- 172.21.15.43:0/1432597656 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5639007c6870 0x5639007c8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.568+0000 7f645ea00640 10 client.27313 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:35.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.568+0000 7f647dc00640 1 --2- 172.21.15.43:0/1432597656 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5639007c6870 0x5639007c8cb0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60f9e846ea1ae398 server_cookie=7aaeb8d1df16a8ca in_seq=0 out_seq=0 2024-03-28T22:41:35.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.569+0000 7f645ea00640 1 -- 172.21.15.43:0/1432597656 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56390071e180 con 0x5639007c6870 2024-03-28T22:41:35.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.569+0000 7f645ea00640 10 client.27313 handle_command_reply: tid=0 2024-03-28T22:41:35.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.569+0000 7f648299e740 1 client.27313 shutdown 2024-03-28T22:41:35.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.570+0000 7f648299e740 1 -- 172.21.15.43:0/1432597656 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639007ad890 msgr2=0x5639007b39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:35.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.570+0000 7f648299e740 1 --2- 172.21.15.43:0/1432597656 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639007ad890 0x5639007b39e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6464009900 tx=0x7f646402c800 comp rx=0 tx=0).stop 2024-03-28T22:41:35.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.570+0000 7f648299e740 1 -- 172.21.15.43:0/1432597656 shutdown_connections 2024-03-28T22:41:35.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.570+0000 7f648299e740 1 --2- 172.21.15.43:0/1432597656 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5639007c6870 0x5639007c8cb0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.570+0000 7f648299e740 1 --2- 172.21.15.43:0/1432597656 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639007b6470 0x5639007b3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.570+0000 7f648299e740 1 --2- 172.21.15.43:0/1432597656 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639007ad890 0x5639007b39e0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.570+0000 7f645ea00640 0 client.27313 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:35.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.570+0000 7f648299e740 1 --2- 172.21.15.43:0/1432597656 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639007acf70 0x5639007b34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.570+0000 7f648299e740 1 -- 172.21.15.43:0/1432597656 >> 172.21.15.43:0/1432597656 conn(0x5639007a76a0 msgr2=0x5639007a93f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:35.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.570+0000 7f648299e740 1 -- 172.21.15.43:0/1432597656 shutdown_connections 2024-03-28T22:41:35.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.570+0000 7f648299e740 1 -- 172.21.15.43:0/1432597656 wait complete. 2024-03-28T22:41:35.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.570+0000 7f648299e740 20 client.27313 trim_cache size 0 max 16384 2024-03-28T22:41:35.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.592+0000 7f648299e740 1 Processor -- start 2024-03-28T22:41:35.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.592+0000 7f648299e740 1 -- start start 2024-03-28T22:41:35.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.592+0000 7f648299e740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639007acf70 0x5639007b5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.592+0000 7f648299e740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639007ad890 0x5639007c2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.592+0000 7f648299e740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639007b6470 0x5639007ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.592+0000 7f648299e740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f64781b0e40 con 0x5639007b6470 2024-03-28T22:41:35.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.592+0000 7f648299e740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f64781b2980 con 0x5639007ad890 2024-03-28T22:41:35.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.592+0000 7f648299e740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f64781b1510 con 0x5639007acf70 2024-03-28T22:41:35.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.593+0000 7f647dc00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639007acf70 0x5639007b5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.593+0000 7f6477e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639007ad890 0x5639007c2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.593+0000 7f647dc00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639007acf70 0x5639007b5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43784/0 (socket says 172.21.15.43:43784) 2024-03-28T22:41:35.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.593+0000 7f647dc00640 1 -- 172.21.15.43:0/1941534476 learned_addr learned my addr 172.21.15.43:0/1941534476 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:35.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.593+0000 7f647dc00640 1 --2- 172.21.15.43:0/1941534476 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639007b6470 0x5639007ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.593+0000 7f647dc00640 1 -- 172.21.15.43:0/1941534476 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639007ad890 msgr2=0x5639007c2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:35.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.593+0000 7f647dc00640 1 --2- 172.21.15.43:0/1941534476 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639007ad890 0x5639007c2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.593+0000 7f647dc00640 1 -- 172.21.15.43:0/1941534476 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639007b6470 msgr2=0x5639007ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:35.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.593+0000 7f647dc00640 1 --2- 172.21.15.43:0/1941534476 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639007b6470 0x5639007ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.593+0000 7f647dc00640 1 -- 172.21.15.43:0/1941534476 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639007acd60 con 0x5639007acf70 2024-03-28T22:41:35.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.593+0000 7f647dc00640 1 --2- 172.21.15.43:0/1941534476 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639007acf70 0x5639007b5bd0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6470009f50 tx=0x7f6470004b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b349b5be8b5ac36 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:35.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.594+0000 7f6469600640 1 -- 172.21.15.43:0/1941534476 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6470029a30 con 0x5639007acf70 2024-03-28T22:41:35.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.594+0000 7f6469600640 1 -- 172.21.15.43:0/1941534476 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6470019070 con 0x5639007acf70 2024-03-28T22:41:35.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.594+0000 7f6469600640 1 -- 172.21.15.43:0/1941534476 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64700137f0 con 0x5639007acf70 2024-03-28T22:41:35.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.594+0000 7f648299e740 1 -- 172.21.15.43:0/1941534476 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639007acf70 msgr2=0x5639007b5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:35.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.594+0000 7f648299e740 1 --2- 172.21.15.43:0/1941534476 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639007acf70 0x5639007b5bd0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6470009f50 tx=0x7f6470004b50 comp rx=0 tx=0).stop 2024-03-28T22:41:35.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.595+0000 7f648299e740 1 -- 172.21.15.43:0/1941534476 shutdown_connections 2024-03-28T22:41:35.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.595+0000 7f648299e740 1 --2- 172.21.15.43:0/1941534476 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639007ad890 0x5639007c2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.595+0000 7f648299e740 1 --2- 172.21.15.43:0/1941534476 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639007b6470 0x5639007ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.595+0000 7f648299e740 1 --2- 172.21.15.43:0/1941534476 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639007acf70 0x5639007b5bd0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.595+0000 7f648299e740 1 -- 172.21.15.43:0/1941534476 >> 172.21.15.43:0/1941534476 conn(0x5639007a76a0 msgr2=0x5639007b3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:35.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.595+0000 7f648299e740 1 -- 172.21.15.43:0/1941534476 shutdown_connections 2024-03-28T22:41:35.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.595+0000 7f648299e740 1 -- 172.21.15.43:0/1941534476 wait complete. 2024-03-28T22:41:35.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.595+0000 7f648299e740 1 -- start start 2024-03-28T22:41:35.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.595+0000 7f648299e740 1 Processor -- start 2024-03-28T22:41:35.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.596+0000 7f648299e740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639007acf70 0x5639007be660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.596+0000 7f648299e740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639007ad890 0x5639007bb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.596+0000 7f648299e740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639007b6470 0x5639007bbb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.596+0000 7f648299e740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f64781b22b0 con 0x5639007acf70 2024-03-28T22:41:35.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.596+0000 7f648299e740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f64781b1be0 con 0x5639007b6470 2024-03-28T22:41:35.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.596+0000 7f648299e740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f64781aee20 con 0x5639007ad890 2024-03-28T22:41:35.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.596+0000 7f648299e740 10 client.0 _ll_register_callbacks cb 0x7f6464009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:41:35.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.596+0000 7f6477e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639007ad890 0x5639007bb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.596+0000 7f6477e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639007ad890 0x5639007bb630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43792/0 (socket says 172.21.15.43:43792) 2024-03-28T22:41:35.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.596+0000 7f6477e00640 1 -- 172.21.15.43:0/3821520953 learned_addr learned my addr 172.21.15.43:0/3821520953 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:35.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.596+0000 7f647dc00640 1 --2- 172.21.15.43:0/3821520953 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639007acf70 0x5639007be660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.596+0000 7f647dc00640 1 --2- 172.21.15.43:0/3821520953 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639007b6470 0x5639007bbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.596+0000 7f6477e00640 1 -- 172.21.15.43:0/3821520953 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639007b6470 msgr2=0x5639007bbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:35.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.596+0000 7f6477e00640 1 --2- 172.21.15.43:0/3821520953 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639007b6470 0x5639007bbb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.596+0000 7f6477e00640 1 -- 172.21.15.43:0/3821520953 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639007acf70 msgr2=0x5639007be660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:35.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.596+0000 7f6477e00640 1 --2- 172.21.15.43:0/3821520953 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639007acf70 0x5639007be660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.596+0000 7f6477e00640 1 -- 172.21.15.43:0/3821520953 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639007bc0b0 con 0x5639007ad890 2024-03-28T22:41:35.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.596+0000 7f647dc00640 1 --2- 172.21.15.43:0/3821520953 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639007acf70 0x5639007be660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:41:35.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.597+0000 7f6477e00640 1 --2- 172.21.15.43:0/3821520953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639007ad890 0x5639007bb630 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x5639007b4a70 tx=0x7f64640095b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e4434a6e6a97071 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:35.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.597+0000 7f645ea00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:41:35.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.597+0000 7f645ea00640 1 -- 172.21.15.43:0/3821520953 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6464043040 con 0x5639007ad890 2024-03-28T22:41:35.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.597+0000 7f648299e740 1 -- 172.21.15.43:0/3821520953 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6470019070 con 0x5639007ad890 2024-03-28T22:41:35.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.597+0000 7f645ea00640 1 -- 172.21.15.43:0/3821520953 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f64640037e0 con 0x5639007ad890 2024-03-28T22:41:35.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.597+0000 7f645ea00640 1 -- 172.21.15.43:0/3821520953 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6464065050 con 0x5639007ad890 2024-03-28T22:41:35.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.597+0000 7f648299e740 10 client.27319 fetch_fsmap learned FSMap version 32 2024-03-28T22:41:35.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.597+0000 7f648299e740 1 -- 172.21.15.43:0/3821520953 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56390082d8a0 con 0x5639007ad890 2024-03-28T22:41:35.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.598+0000 7f645ea00640 1 -- 172.21.15.43:0/3821520953 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6470019070 con 0x5639007ad890 2024-03-28T22:41:35.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.598+0000 7f645ea00640 1 -- 172.21.15.43:0/3821520953 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6464004f80 con 0x5639007ad890 2024-03-28T22:41:35.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.598+0000 7f648299e740 10 client.27319 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:41:35.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.598+0000 7f648299e740 10 client.27319 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:41:35.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.598+0000 7f648299e740 20 client.27319 populate_metadata read hostname 'smithi043' 2024-03-28T22:41:35.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.598+0000 7f648299e740 1 --2- 172.21.15.43:0/3821520953 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56390082e050 0x563900830470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.598+0000 7f648299e740 4 client.27319 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:41:35.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.598+0000 7f648299e740 1 -- 172.21.15.43:0/3821520953 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5639007184d0 con 0x56390082e050 2024-03-28T22:41:35.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.598+0000 7f647dc00640 1 --2- 172.21.15.43:0/3821520953 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56390082e050 0x563900830470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.599+0000 7f647dc00640 1 --2- 172.21.15.43:0/3821520953 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56390082e050 0x563900830470 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=670a64f19aaa743f server_cookie=cee40830fe1ed5df in_seq=0 out_seq=0 2024-03-28T22:41:35.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.599+0000 7f645ea00640 10 client.27319 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:35.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.602+0000 7f645ea00640 1 -- 172.21.15.43:0/3821520953 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5639007184d0 con 0x56390082e050 2024-03-28T22:41:35.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.602+0000 7f645ea00640 10 client.27319 handle_command_reply: tid=0 2024-03-28T22:41:35.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.602+0000 7f648299e740 1 client.27319 shutdown 2024-03-28T22:41:35.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.602+0000 7f648299e740 1 -- 172.21.15.43:0/3821520953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639007ad890 msgr2=0x5639007bb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:35.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.602+0000 7f648299e740 1 --2- 172.21.15.43:0/3821520953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639007ad890 0x5639007bb630 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x5639007b4a70 tx=0x7f64640095b0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.602+0000 7f648299e740 1 -- 172.21.15.43:0/3821520953 shutdown_connections 2024-03-28T22:41:35.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.602+0000 7f648299e740 1 --2- 172.21.15.43:0/3821520953 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56390082e050 0x563900830470 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.602+0000 7f648299e740 1 --2- 172.21.15.43:0/3821520953 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639007b6470 0x5639007bbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.602+0000 7f648299e740 1 --2- 172.21.15.43:0/3821520953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639007ad890 0x5639007bb630 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.602+0000 7f648299e740 1 --2- 172.21.15.43:0/3821520953 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639007acf70 0x5639007be660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.602+0000 7f645ea00640 0 client.27319 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:35.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.602+0000 7f648299e740 1 -- 172.21.15.43:0/3821520953 >> 172.21.15.43:0/3821520953 conn(0x5639007a76a0 msgr2=0x5639007b3290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:35.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.602+0000 7f648299e740 1 -- 172.21.15.43:0/3821520953 shutdown_connections 2024-03-28T22:41:35.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.602+0000 7f648299e740 1 -- 172.21.15.43:0/3821520953 wait complete. 2024-03-28T22:41:35.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.602+0000 7f648299e740 20 client.27319 trim_cache size 0 max 16384 2024-03-28T22:41:35.601 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:41:35.601 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-03-28T22:41:35.601 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "0f3c2fac-22bb-48fa-bb59-fba2be76df74", 2024-03-28T22:41:35.601 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-03-28T22:41:35.601 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:41:35.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.603+0000 7f647fc00640 1 -- 172.21.15.43:0/470352156 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f64600a1ef0 msgr2=0x7f64600a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:35.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.603+0000 7f647fc00640 1 --2- 172.21.15.43:0/470352156 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f64600a1ef0 0x7f64600a43b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x5639007aca40 tx=0x7f646c000bc0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.603+0000 7f647fc00640 1 -- 172.21.15.43:0/470352156 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64781038f0 msgr2=0x7f647819a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:35.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.603+0000 7f647fc00640 1 --2- 172.21.15.43:0/470352156 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64781038f0 0x7f647819a2a0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f646c00e9f0 tx=0x7f646c00eeb0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.603+0000 7f647fc00640 1 -- 172.21.15.43:0/470352156 shutdown_connections 2024-03-28T22:41:35.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.603+0000 7f647fc00640 1 --2- 172.21.15.43:0/470352156 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f64600a1ef0 0x7f64600a43b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.603+0000 7f647fc00640 1 --2- 172.21.15.43:0/470352156 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6478109900 0x7f647819eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.603+0000 7f647fc00640 1 --2- 172.21.15.43:0/470352156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f64781042b0 0x7f647819a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.603+0000 7f647fc00640 1 --2- 172.21.15.43:0/470352156 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64781038f0 0x7f647819a2a0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.603+0000 7f647fc00640 1 -- 172.21.15.43:0/470352156 >> 172.21.15.43:0/470352156 conn(0x7f64780ff400 msgr2=0x7f64780ffc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:35.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.603+0000 7f647fc00640 1 -- 172.21.15.43:0/470352156 shutdown_connections 2024-03-28T22:41:35.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.604+0000 7f647fc00640 1 -- 172.21.15.43:0/470352156 wait complete. 2024-03-28T22:41:35.619 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:41:35.619 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:41:35.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.843+0000 7fb94b200640 1 -- 172.21.15.43:0/649407579 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb94c106590 msgr2=0x7fb94c106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:35.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.843+0000 7fb94b200640 1 --2- 172.21.15.43:0/649407579 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb94c106590 0x7fb94c106970 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fb93c009930 tx=0x7fb93c02bc20 comp rx=0 tx=0).stop 2024-03-28T22:41:35.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.843+0000 7fb94b200640 1 -- 172.21.15.43:0/649407579 shutdown_connections 2024-03-28T22:41:35.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.843+0000 7fb94b200640 1 --2- 172.21.15.43:0/649407579 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb94c100e90 0x7fb94c10d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.843+0000 7fb94b200640 1 --2- 172.21.15.43:0/649407579 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb94c1004d0 0x7fb94c100950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.843+0000 7fb94b200640 1 --2- 172.21.15.43:0/649407579 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb94c106590 0x7fb94c106970 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.843+0000 7fb94b200640 1 -- 172.21.15.43:0/649407579 >> 172.21.15.43:0/649407579 conn(0x7fb94c0fbfa0 msgr2=0x7fb94c0fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:35.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.843+0000 7fb94b200640 1 -- 172.21.15.43:0/649407579 shutdown_connections 2024-03-28T22:41:35.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.843+0000 7fb94b200640 1 -- 172.21.15.43:0/649407579 wait complete. 2024-03-28T22:41:35.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.845+0000 7fb94b200640 1 Processor -- start 2024-03-28T22:41:35.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.845+0000 7fb94b200640 1 -- start start 2024-03-28T22:41:35.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.845+0000 7fb94b200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb94c1004d0 0x7fb94c197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.845+0000 7fb94b200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb94c100e90 0x7fb94c1983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.845+0000 7fb94b200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb94c106590 0x7fb94c19c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.845+0000 7fb94b200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb94c1119f0 con 0x7fb94c1004d0 2024-03-28T22:41:35.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.845+0000 7fb94b200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb94c111870 con 0x7fb94c100e90 2024-03-28T22:41:35.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.845+0000 7fb94b200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fb94c111b70 con 0x7fb94c106590 2024-03-28T22:41:35.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.846+0000 7fb949e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb94c1004d0 0x7fb94c197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.846+0000 7fb949400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb94c100e90 0x7fb94c1983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.846+0000 7fb94a800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb94c106590 0x7fb94c19c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.846+0000 7fb949400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb94c100e90 0x7fb94c1983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48114/0 (socket says 172.21.15.43:48114) 2024-03-28T22:41:35.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.846+0000 7fb949400640 1 -- 172.21.15.43:0/2980099530 learned_addr learned my addr 172.21.15.43:0/2980099530 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:35.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.846+0000 7fb949400640 1 -- 172.21.15.43:0/2980099530 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb94c106590 msgr2=0x7fb94c19c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:35.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.846+0000 7fb949400640 1 --2- 172.21.15.43:0/2980099530 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb94c106590 0x7fb94c19c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.846+0000 7fb949400640 1 -- 172.21.15.43:0/2980099530 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb94c1004d0 msgr2=0x7fb94c197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:35.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.846+0000 7fb949400640 1 --2- 172.21.15.43:0/2980099530 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb94c1004d0 0x7fb94c197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.846+0000 7fb949400640 1 -- 172.21.15.43:0/2980099530 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb94c19ce80 con 0x7fb94c100e90 2024-03-28T22:41:35.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.846+0000 7fb94a800640 1 --2- 172.21.15.43:0/2980099530 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb94c106590 0x7fb94c19c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:41:35.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.846+0000 7fb949400640 1 --2- 172.21.15.43:0/2980099530 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb94c100e90 0x7fb94c1983d0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fb93800d950 tx=0x7fb93800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:35.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.847+0000 7fb942a00640 1 -- 172.21.15.43:0/2980099530 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb938017070 con 0x7fb94c100e90 2024-03-28T22:41:35.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.847+0000 7fb942a00640 1 -- 172.21.15.43:0/2980099530 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb938009c30 con 0x7fb94c100e90 2024-03-28T22:41:35.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.847+0000 7fb94b200640 1 -- 172.21.15.43:0/2980099530 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb94c19d170 con 0x7fb94c100e90 2024-03-28T22:41:35.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.847+0000 7fb942a00640 1 -- 172.21.15.43:0/2980099530 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb93800f6b0 con 0x7fb94c100e90 2024-03-28T22:41:35.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.847+0000 7fb94b200640 1 -- 172.21.15.43:0/2980099530 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb94c1a4a00 con 0x7fb94c100e90 2024-03-28T22:41:35.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.850+0000 7fb942a00640 1 -- 172.21.15.43:0/2980099530 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fb9380109d0 con 0x7fb94c100e90 2024-03-28T22:41:35.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.850+0000 7fb953a60740 1 Processor -- start 2024-03-28T22:41:35.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.850+0000 7fb953a60740 1 -- start start 2024-03-28T22:41:35.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.851+0000 7fb953a60740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555930471db0 0x555930470b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.851+0000 7fb953a60740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5559304710d0 0x555930471550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.851+0000 7fb953a60740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55593047a440 0x55593047c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.851+0000 7fb953a60740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5559303d5100 con 0x5559304710d0 2024-03-28T22:41:35.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.851+0000 7fb953a60740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55593037dba0 con 0x55593047a440 2024-03-28T22:41:35.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.851+0000 7fb953a60740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5559303d11a0 con 0x555930471db0 2024-03-28T22:41:35.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.851+0000 7fb94a800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555930471db0 0x555930470b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.851+0000 7fb94a800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555930471db0 0x555930470b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43810/0 (socket says 172.21.15.43:43810) 2024-03-28T22:41:35.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.851+0000 7fb94a800640 1 -- 172.21.15.43:0/899359754 learned_addr learned my addr 172.21.15.43:0/899359754 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:35.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.851+0000 7fb949e00640 1 --2- 172.21.15.43:0/899359754 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5559304710d0 0x555930471550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.851+0000 7fb949400640 1 --2- 172.21.15.43:0/899359754 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55593047a440 0x55593047c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.852+0000 7fb94a800640 1 -- 172.21.15.43:0/899359754 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55593047a440 msgr2=0x55593047c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:35.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.852+0000 7fb94a800640 1 --2- 172.21.15.43:0/899359754 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55593047a440 0x55593047c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.852+0000 7fb94a800640 1 -- 172.21.15.43:0/899359754 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5559304710d0 msgr2=0x555930471550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:35.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.852+0000 7fb94a800640 1 --2- 172.21.15.43:0/899359754 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5559304710d0 0x555930471550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.852+0000 7fb94a800640 1 -- 172.21.15.43:0/899359754 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559303ae1a0 con 0x555930471db0 2024-03-28T22:41:35.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.852+0000 7fb949e00640 1 --2- 172.21.15.43:0/899359754 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5559304710d0 0x555930471550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:41:35.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.852+0000 7fb94a800640 1 --2- 172.21.15.43:0/899359754 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555930471db0 0x555930470b00 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fb944010710 tx=0x7fb944010bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=466bf7e71b122c9b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:35.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.852+0000 7fb937400640 1 -- 172.21.15.43:0/899359754 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94400e8d0 con 0x555930471db0 2024-03-28T22:41:35.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.852+0000 7fb937400640 1 -- 172.21.15.43:0/899359754 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb944012450 con 0x555930471db0 2024-03-28T22:41:35.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.852+0000 7fb937400640 1 -- 172.21.15.43:0/899359754 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94401c650 con 0x555930471db0 2024-03-28T22:41:35.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.852+0000 7fb942a00640 1 --2- 172.21.15.43:0/2980099530 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb9300a1f70 0x7fb9300a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.853+0000 7fb942a00640 1 -- 172.21.15.43:0/2980099530 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fb9380ce0c0 con 0x7fb94c100e90 2024-03-28T22:41:35.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.853+0000 7fb953a60740 1 -- 172.21.15.43:0/899359754 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555930471db0 msgr2=0x555930470b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:35.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.853+0000 7fb953a60740 1 --2- 172.21.15.43:0/899359754 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555930471db0 0x555930470b00 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fb944010710 tx=0x7fb944010bd0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.853+0000 7fb949e00640 1 --2- 172.21.15.43:0/2980099530 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb9300a1f70 0x7fb9300a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.853+0000 7fb953a60740 1 -- 172.21.15.43:0/899359754 shutdown_connections 2024-03-28T22:41:35.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.853+0000 7fb953a60740 1 --2- 172.21.15.43:0/899359754 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55593047a440 0x55593047c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.853+0000 7fb953a60740 1 --2- 172.21.15.43:0/899359754 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5559304710d0 0x555930471550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.853+0000 7fb953a60740 1 --2- 172.21.15.43:0/899359754 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555930471db0 0x555930470b00 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.853+0000 7fb953a60740 1 -- 172.21.15.43:0/899359754 >> 172.21.15.43:0/899359754 conn(0x555930437a10 msgr2=0x55593046bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:35.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.853+0000 7fb953a60740 1 -- 172.21.15.43:0/899359754 shutdown_connections 2024-03-28T22:41:35.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.853+0000 7fb953a60740 1 -- 172.21.15.43:0/899359754 wait complete. 2024-03-28T22:41:35.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.854+0000 7fb953a60740 1 -- start start 2024-03-28T22:41:35.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.854+0000 7fb949e00640 1 --2- 172.21.15.43:0/2980099530 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb9300a1f70 0x7fb9300a4430 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb93c009900 tx=0x7fb93c000bc0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:35.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.854+0000 7fb953a60740 1 Processor -- start 2024-03-28T22:41:35.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.855+0000 7fb953a60740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5559304710d0 0x555930477430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.855+0000 7fb953a60740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555930471db0 0x555930477970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.855+0000 7fb953a60740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55593047a440 0x555930477eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.855+0000 7fb953a60740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55593047d2b0 con 0x555930471db0 2024-03-28T22:41:35.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.855+0000 7fb953a60740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x555930373250 con 0x55593047a440 2024-03-28T22:41:35.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.855+0000 7fb953a60740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55593047d430 con 0x5559304710d0 2024-03-28T22:41:35.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.855+0000 7fb953a60740 10 client.0 _ll_register_callbacks cb 0x5559300e8be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:41:35.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.855+0000 7fb949e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5559304710d0 0x555930477430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.855+0000 7fb949e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5559304710d0 0x555930477430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43812/0 (socket says 172.21.15.43:43812) 2024-03-28T22:41:35.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.855+0000 7fb949e00640 1 -- 172.21.15.43:0/813375078 learned_addr learned my addr 172.21.15.43:0/813375078 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:35.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.855+0000 7fb949400640 1 --2- 172.21.15.43:0/813375078 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555930471db0 0x555930477970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.855+0000 7fb94a800640 1 --2- 172.21.15.43:0/813375078 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55593047a440 0x555930477eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.856+0000 7fb949e00640 1 -- 172.21.15.43:0/813375078 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55593047a440 msgr2=0x555930477eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:35.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.856+0000 7fb949e00640 1 --2- 172.21.15.43:0/813375078 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55593047a440 0x555930477eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.856+0000 7fb949e00640 1 -- 172.21.15.43:0/813375078 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555930471db0 msgr2=0x555930477970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:35.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.856+0000 7fb949e00640 1 --2- 172.21.15.43:0/813375078 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555930471db0 0x555930477970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.856+0000 7fb949e00640 1 -- 172.21.15.43:0/813375078 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55593038f960 con 0x5559304710d0 2024-03-28T22:41:35.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.856+0000 7fb94a800640 1 --2- 172.21.15.43:0/813375078 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55593047a440 0x555930477eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:41:35.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.856+0000 7fb949e00640 1 --2- 172.21.15.43:0/813375078 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5559304710d0 0x555930477430 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fb93c009930 tx=0x7fb93c009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d2b05033200ff8e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:35.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.856+0000 7fb92be00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:41:35.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.856+0000 7fb92be00640 1 -- 172.21.15.43:0/813375078 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb93c04d950 con 0x5559304710d0 2024-03-28T22:41:35.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.856+0000 7fb953a60740 1 -- 172.21.15.43:0/813375078 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb944012450 con 0x5559304710d0 2024-03-28T22:41:35.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.856+0000 7fb92be00640 1 -- 172.21.15.43:0/813375078 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fb93c04daf0 con 0x5559304710d0 2024-03-28T22:41:35.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.856+0000 7fb92be00640 1 -- 172.21.15.43:0/813375078 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb93c05f050 con 0x5559304710d0 2024-03-28T22:41:35.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.856+0000 7fb953a60740 10 client.27331 fetch_fsmap learned FSMap version 32 2024-03-28T22:41:35.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.857+0000 7fb953a60740 1 -- 172.21.15.43:0/813375078 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5559301b37d0 con 0x5559304710d0 2024-03-28T22:41:35.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.857+0000 7fb92be00640 1 -- 172.21.15.43:0/813375078 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb944012450 con 0x5559304710d0 2024-03-28T22:41:35.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.857+0000 7fb92be00640 1 -- 172.21.15.43:0/813375078 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb93c0035f0 con 0x5559304710d0 2024-03-28T22:41:35.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.857+0000 7fb953a60740 10 client.27331 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:41:35.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.857+0000 7fb953a60740 10 client.27331 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:41:35.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.857+0000 7fb953a60740 20 client.27331 populate_metadata read hostname 'smithi043' 2024-03-28T22:41:35.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.857+0000 7fb953a60740 1 --2- 172.21.15.43:0/813375078 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55593048a810 0x55593048cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.857+0000 7fb953a60740 4 client.27331 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:41:35.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.857+0000 7fb953a60740 1 -- 172.21.15.43:0/813375078 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5559303e2640 con 0x55593048a810 2024-03-28T22:41:35.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.857+0000 7fb949400640 1 --2- 172.21.15.43:0/813375078 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55593048a810 0x55593048cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.858+0000 7fb949400640 1 --2- 172.21.15.43:0/813375078 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55593048a810 0x55593048cc90 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b1151002f771640 server_cookie=fbd2553d6c71814 in_seq=0 out_seq=0 2024-03-28T22:41:35.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.858+0000 7fb92be00640 10 client.27331 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:35.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.861+0000 7fb92be00640 1 -- 172.21.15.43:0/813375078 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5559303e2640 con 0x55593048a810 2024-03-28T22:41:35.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.861+0000 7fb92be00640 10 client.27331 handle_command_reply: tid=0 2024-03-28T22:41:35.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.861+0000 7fb953a60740 1 client.27331 shutdown 2024-03-28T22:41:35.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.862+0000 7fb953a60740 1 -- 172.21.15.43:0/813375078 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5559304710d0 msgr2=0x555930477430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:35.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.862+0000 7fb953a60740 1 --2- 172.21.15.43:0/813375078 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5559304710d0 0x555930477430 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fb93c009930 tx=0x7fb93c009260 comp rx=0 tx=0).stop 2024-03-28T22:41:35.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.862+0000 7fb953a60740 1 -- 172.21.15.43:0/813375078 shutdown_connections 2024-03-28T22:41:35.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.862+0000 7fb953a60740 1 --2- 172.21.15.43:0/813375078 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55593048a810 0x55593048cc90 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.862+0000 7fb953a60740 1 --2- 172.21.15.43:0/813375078 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55593047a440 0x555930477eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.862+0000 7fb953a60740 1 --2- 172.21.15.43:0/813375078 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x555930471db0 0x555930477970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.862+0000 7fb953a60740 1 --2- 172.21.15.43:0/813375078 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5559304710d0 0x555930477430 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.862+0000 7fb953a60740 1 -- 172.21.15.43:0/813375078 >> 172.21.15.43:0/813375078 conn(0x555930437a10 msgr2=0x55593046d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:35.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.862+0000 7fb92be00640 0 client.27331 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:35.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.862+0000 7fb953a60740 1 -- 172.21.15.43:0/813375078 shutdown_connections 2024-03-28T22:41:35.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.862+0000 7fb953a60740 1 -- 172.21.15.43:0/813375078 wait complete. 2024-03-28T22:41:35.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.862+0000 7fb953a60740 20 client.27331 trim_cache size 0 max 16384 2024-03-28T22:41:35.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.882+0000 7fb953a60740 1 Processor -- start 2024-03-28T22:41:35.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.882+0000 7fb953a60740 1 -- start start 2024-03-28T22:41:35.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.883+0000 7fb953a60740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5559304710d0 0x55593046ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.883+0000 7fb953a60740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555930471db0 0x555930476d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.883+0000 7fb953a60740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55593047a440 0x555930486090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.883+0000 7fb953a60740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb94c1ac810 con 0x5559304710d0 2024-03-28T22:41:35.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.883+0000 7fb953a60740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb94c1ae350 con 0x555930471db0 2024-03-28T22:41:35.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.883+0000 7fb953a60740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fb94c1acee0 con 0x55593047a440 2024-03-28T22:41:35.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.883+0000 7fb94a800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55593047a440 0x555930486090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.883+0000 7fb949e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5559304710d0 0x55593046ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.883+0000 7fb94a800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55593047a440 0x555930486090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43816/0 (socket says 172.21.15.43:43816) 2024-03-28T22:41:35.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.883+0000 7fb94a800640 1 -- 172.21.15.43:0/1167201280 learned_addr learned my addr 172.21.15.43:0/1167201280 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:35.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.883+0000 7fb949400640 1 --2- 172.21.15.43:0/1167201280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555930471db0 0x555930476d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.884+0000 7fb94a800640 1 -- 172.21.15.43:0/1167201280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555930471db0 msgr2=0x555930476d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:35.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.884+0000 7fb94a800640 1 --2- 172.21.15.43:0/1167201280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555930471db0 0x555930476d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.884+0000 7fb94a800640 1 -- 172.21.15.43:0/1167201280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5559304710d0 msgr2=0x55593046ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:35.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.884+0000 7fb94a800640 1 --2- 172.21.15.43:0/1167201280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5559304710d0 0x55593046ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.884+0000 7fb94a800640 1 -- 172.21.15.43:0/1167201280 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559300e93f0 con 0x55593047a440 2024-03-28T22:41:35.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.884+0000 7fb94a800640 1 --2- 172.21.15.43:0/1167201280 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55593047a440 0x555930486090 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fb94400e6c0 tx=0x7fb944012b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88c6fe805dc3aa7d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:35.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.884+0000 7fb936a00640 1 -- 172.21.15.43:0/1167201280 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94402b070 con 0x55593047a440 2024-03-28T22:41:35.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.884+0000 7fb936a00640 1 -- 172.21.15.43:0/1167201280 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb944010dc0 con 0x55593047a440 2024-03-28T22:41:35.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.884+0000 7fb936a00640 1 -- 172.21.15.43:0/1167201280 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94400eea0 con 0x55593047a440 2024-03-28T22:41:35.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.885+0000 7fb953a60740 1 -- 172.21.15.43:0/1167201280 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55593047a440 msgr2=0x555930486090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:35.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.885+0000 7fb953a60740 1 --2- 172.21.15.43:0/1167201280 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55593047a440 0x555930486090 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fb94400e6c0 tx=0x7fb944012b40 comp rx=0 tx=0).stop 2024-03-28T22:41:35.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.885+0000 7fb953a60740 1 -- 172.21.15.43:0/1167201280 shutdown_connections 2024-03-28T22:41:35.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.885+0000 7fb953a60740 1 --2- 172.21.15.43:0/1167201280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555930471db0 0x555930476d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.885+0000 7fb953a60740 1 --2- 172.21.15.43:0/1167201280 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55593047a440 0x555930486090 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.885+0000 7fb953a60740 1 --2- 172.21.15.43:0/1167201280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5559304710d0 0x55593046ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.885+0000 7fb953a60740 1 -- 172.21.15.43:0/1167201280 >> 172.21.15.43:0/1167201280 conn(0x555930437a10 msgr2=0x55593046cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:35.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.885+0000 7fb953a60740 1 -- 172.21.15.43:0/1167201280 shutdown_connections 2024-03-28T22:41:35.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.885+0000 7fb953a60740 1 -- 172.21.15.43:0/1167201280 wait complete. 2024-03-28T22:41:35.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.886+0000 7fb953a60740 1 -- start start 2024-03-28T22:41:35.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.886+0000 7fb953a60740 1 Processor -- start 2024-03-28T22:41:35.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.886+0000 7fb953a60740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5559304710d0 0x555930482b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.886+0000 7fb953a60740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555930471db0 0x555930473dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.886+0000 7fb953a60740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55593047a440 0x555930474500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.886+0000 7fb953a60740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb94c1adc80 con 0x5559304710d0 2024-03-28T22:41:35.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.886+0000 7fb953a60740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb94c1ad5b0 con 0x55593047a440 2024-03-28T22:41:35.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.886+0000 7fb953a60740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fb94c1aa7f0 con 0x555930471db0 2024-03-28T22:41:35.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.886+0000 7fb953a60740 10 client.0 _ll_register_callbacks cb 0x7fb93c02c340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:41:35.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.886+0000 7fb949e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5559304710d0 0x555930482b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.886+0000 7fb94a800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55593047a440 0x555930474500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.886+0000 7fb949400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555930471db0 0x555930473dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.887+0000 7fb94a800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55593047a440 0x555930474500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48160/0 (socket says 172.21.15.43:48160) 2024-03-28T22:41:35.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.887+0000 7fb94a800640 1 -- 172.21.15.43:0/4239216959 learned_addr learned my addr 172.21.15.43:0/4239216959 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:35.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.887+0000 7fb949400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555930471db0 0x555930473dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43824/0 (socket says 172.21.15.43:43824) 2024-03-28T22:41:35.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.887+0000 7fb94a800640 1 -- 172.21.15.43:0/4239216959 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555930471db0 msgr2=0x555930473dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:35.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.887+0000 7fb94a800640 1 --2- 172.21.15.43:0/4239216959 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555930471db0 0x555930473dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.887+0000 7fb94a800640 1 -- 172.21.15.43:0/4239216959 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5559304710d0 msgr2=0x555930482b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:35.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.887+0000 7fb94a800640 1 --2- 172.21.15.43:0/4239216959 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5559304710d0 0x555930482b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.887+0000 7fb94a800640 1 -- 172.21.15.43:0/4239216959 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559300a90f0 con 0x55593047a440 2024-03-28T22:41:35.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.887+0000 7fb949400640 1 --2- 172.21.15.43:0/4239216959 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555930471db0 0x555930473dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:41:35.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.887+0000 7fb94a800640 1 --2- 172.21.15.43:0/4239216959 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55593047a440 0x555930474500 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fb944010680 tx=0x7fb944012dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=153e19e8eb360fcb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:35.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.887+0000 7fb92be00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:41:35.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.888+0000 7fb92be00640 1 -- 172.21.15.43:0/4239216959 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94402b040 con 0x55593047a440 2024-03-28T22:41:35.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.888+0000 7fb953a60740 1 -- 172.21.15.43:0/4239216959 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb944010dc0 con 0x55593047a440 2024-03-28T22:41:35.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.888+0000 7fb92be00640 1 -- 172.21.15.43:0/4239216959 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fb944012910 con 0x55593047a440 2024-03-28T22:41:35.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.888+0000 7fb92be00640 1 -- 172.21.15.43:0/4239216959 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb94403d050 con 0x55593047a440 2024-03-28T22:41:35.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.888+0000 7fb953a60740 10 client.27386 fetch_fsmap learned FSMap version 32 2024-03-28T22:41:35.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.888+0000 7fb953a60740 1 -- 172.21.15.43:0/4239216959 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5559304f93a0 con 0x55593047a440 2024-03-28T22:41:35.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.888+0000 7fb92be00640 1 -- 172.21.15.43:0/4239216959 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb944010dc0 con 0x55593047a440 2024-03-28T22:41:35.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.888+0000 7fb92be00640 1 -- 172.21.15.43:0/4239216959 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb94402e030 con 0x55593047a440 2024-03-28T22:41:35.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.889+0000 7fb953a60740 10 client.27386 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:41:35.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.889+0000 7fb953a60740 10 client.27386 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:41:35.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.889+0000 7fb953a60740 20 client.27386 populate_metadata read hostname 'smithi043' 2024-03-28T22:41:35.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.889+0000 7fb953a60740 1 --2- 172.21.15.43:0/4239216959 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5559304f96d0 0x5559304fbac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:35.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.889+0000 7fb953a60740 4 client.27386 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:41:35.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.889+0000 7fb953a60740 1 -- 172.21.15.43:0/4239216959 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5559303dc990 con 0x5559304f96d0 2024-03-28T22:41:35.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.889+0000 7fb949e00640 1 --2- 172.21.15.43:0/4239216959 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5559304f96d0 0x5559304fbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:35.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.889+0000 7fb949e00640 1 --2- 172.21.15.43:0/4239216959 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5559304f96d0 0x5559304fbac0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9800c1495322e335 server_cookie=d7b55c4528a7aa58 in_seq=0 out_seq=0 2024-03-28T22:41:35.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.889+0000 7fb92be00640 10 client.27386 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:35.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.891+0000 7fb92be00640 1 -- 172.21.15.43:0/4239216959 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5559303dc990 con 0x5559304f96d0 2024-03-28T22:41:35.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.891+0000 7fb92be00640 10 client.27386 handle_command_reply: tid=0 2024-03-28T22:41:35.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.891+0000 7fb953a60740 1 client.27386 shutdown 2024-03-28T22:41:35.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.892+0000 7fb953a60740 1 -- 172.21.15.43:0/4239216959 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55593047a440 msgr2=0x555930474500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:35.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.892+0000 7fb953a60740 1 --2- 172.21.15.43:0/4239216959 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55593047a440 0x555930474500 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fb944010680 tx=0x7fb944012dc0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.892+0000 7fb953a60740 1 -- 172.21.15.43:0/4239216959 shutdown_connections 2024-03-28T22:41:35.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.892+0000 7fb953a60740 1 --2- 172.21.15.43:0/4239216959 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5559304f96d0 0x5559304fbac0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.892+0000 7fb953a60740 1 --2- 172.21.15.43:0/4239216959 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55593047a440 0x555930474500 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.892+0000 7fb953a60740 1 --2- 172.21.15.43:0/4239216959 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x555930471db0 0x555930473dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.892+0000 7fb953a60740 1 --2- 172.21.15.43:0/4239216959 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5559304710d0 0x555930482b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.892+0000 7fb92be00640 0 client.27386 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:35.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.892+0000 7fb953a60740 1 -- 172.21.15.43:0/4239216959 >> 172.21.15.43:0/4239216959 conn(0x555930437a10 msgr2=0x55593046ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:35.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.892+0000 7fb953a60740 1 -- 172.21.15.43:0/4239216959 shutdown_connections 2024-03-28T22:41:35.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.892+0000 7fb953a60740 1 -- 172.21.15.43:0/4239216959 wait complete. 2024-03-28T22:41:35.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.892+0000 7fb953a60740 20 client.27386 trim_cache size 0 max 16384 2024-03-28T22:41:35.893 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:41:35.893 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:41:35.893 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-03-28T22:41:35.893 INFO:teuthology.orchestra.run.smithi043.stdout: "0f3c2fac-22bb-48fa-bb59-fba2be76df74": { 2024-03-28T22:41:35.893 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-03-28T22:41:35.893 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "0f3c2fac-22bb-48fa-bb59-fba2be76df74", 2024-03-28T22:41:35.893 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-03-28T22:41:35.893 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:41:35.893 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:41:35.893 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:41:35.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.893+0000 7fb94b200640 1 -- 172.21.15.43:0/2980099530 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb9300a1f70 msgr2=0x7fb9300a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:35.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.893+0000 7fb94b200640 1 --2- 172.21.15.43:0/2980099530 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb9300a1f70 0x7fb9300a4430 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb93c009900 tx=0x7fb93c000bc0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.893+0000 7fb94b200640 1 -- 172.21.15.43:0/2980099530 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb94c100e90 msgr2=0x7fb94c1983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:35.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.893+0000 7fb94b200640 1 --2- 172.21.15.43:0/2980099530 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb94c100e90 0x7fb94c1983d0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fb93800d950 tx=0x7fb93800de10 comp rx=0 tx=0).stop 2024-03-28T22:41:35.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.893+0000 7fb94b200640 1 -- 172.21.15.43:0/2980099530 shutdown_connections 2024-03-28T22:41:35.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.893+0000 7fb94b200640 1 --2- 172.21.15.43:0/2980099530 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb9300a1f70 0x7fb9300a4430 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.893+0000 7fb94b200640 1 --2- 172.21.15.43:0/2980099530 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb94c100e90 0x7fb94c1983d0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.893+0000 7fb94b200640 1 --2- 172.21.15.43:0/2980099530 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb94c106590 0x7fb94c19c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.893+0000 7fb94b200640 1 --2- 172.21.15.43:0/2980099530 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb94c1004d0 0x7fb94c197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:35.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.893+0000 7fb94b200640 1 -- 172.21.15.43:0/2980099530 >> 172.21.15.43:0/2980099530 conn(0x7fb94c0fbfa0 msgr2=0x7fb94c0fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:35.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.893+0000 7fb94b200640 1 -- 172.21.15.43:0/2980099530 shutdown_connections 2024-03-28T22:41:35.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:41:35.893+0000 7fb94b200640 1 -- 172.21.15.43:0/2980099530 wait complete. 2024-03-28T22:41:35.909 INFO:tasks.cephfs.filesystem:scrub status for tag:0f3c2fac-22bb-48fa-bb59-fba2be76df74 - {'path': '/', 'tag': '0f3c2fac-22bb-48fa-bb59-fba2be76df74', 'options': 'recursive,force'} 2024-03-28T22:41:36.360 INFO:tasks.workunit.client.0.smithi043.stdout: 1 284774 7.98 MB/sec execute 67 sec latency 15.880 ms 2024-03-28T22:41:36.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:36 smithi082 ceph-mon[36471]: pgmap v137: 129 pgs: 129 active+clean; 46 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 275 op/s 2024-03-28T22:41:36.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:36 smithi082 ceph-mon[36471]: scrub queued for path: / 2024-03-28T22:41:36.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:36 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:41:36.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:36 smithi082 ceph-mon[36471]: scrub summary: active paths [/] 2024-03-28T22:41:36.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:36 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:41:37.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:36 smithi043 ceph-mon[31823]: pgmap v137: 129 pgs: 129 active+clean; 46 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 275 op/s 2024-03-28T22:41:37.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:36 smithi043 ceph-mon[31823]: scrub queued for path: / 2024-03-28T22:41:37.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:36 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:41:37.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:36 smithi043 ceph-mon[31823]: scrub summary: active paths [/] 2024-03-28T22:41:37.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:36 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:41:37.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:36 smithi172 ceph-mon[36566]: pgmap v137: 129 pgs: 129 active+clean; 46 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 275 op/s 2024-03-28T22:41:37.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:36 smithi172 ceph-mon[36566]: scrub queued for path: / 2024-03-28T22:41:37.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:36 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:41:37.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:36 smithi172 ceph-mon[36566]: scrub summary: active paths [/] 2024-03-28T22:41:37.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:36 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:41:37.360 INFO:tasks.workunit.client.0.smithi043.stdout: 1 286760 8.03 MB/sec execute 68 sec latency 21.867 ms 2024-03-28T22:41:38.360 INFO:tasks.workunit.client.0.smithi043.stdout: 1 288009 8.01 MB/sec execute 69 sec latency 16.301 ms 2024-03-28T22:41:38.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:38 smithi082 ceph-mon[36471]: pgmap v138: 129 pgs: 129 active+clean; 50 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 351 op/s 2024-03-28T22:41:38.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:38 smithi082 ceph-mon[36471]: scrub summary: idle 2024-03-28T22:41:39.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:38 smithi043 ceph-mon[31823]: pgmap v138: 129 pgs: 129 active+clean; 50 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 351 op/s 2024-03-28T22:41:39.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:38 smithi043 ceph-mon[31823]: scrub summary: idle 2024-03-28T22:41:39.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:38 smithi172 ceph-mon[36566]: pgmap v138: 129 pgs: 129 active+clean; 50 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 351 op/s 2024-03-28T22:41:39.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:38 smithi172 ceph-mon[36566]: scrub summary: idle 2024-03-28T22:41:39.360 INFO:tasks.workunit.client.0.smithi043.stdout: 1 289299 7.97 MB/sec execute 70 sec latency 49.397 ms 2024-03-28T22:41:40.361 INFO:tasks.workunit.client.0.smithi043.stdout: 1 291162 8.05 MB/sec execute 71 sec latency 11.885 ms 2024-03-28T22:41:40.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:40 smithi082 ceph-mon[36471]: pgmap v139: 129 pgs: 129 active+clean; 50 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 267 op/s 2024-03-28T22:41:41.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:40 smithi043 ceph-mon[31823]: pgmap v139: 129 pgs: 129 active+clean; 50 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 267 op/s 2024-03-28T22:41:41.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:40 smithi172 ceph-mon[36566]: pgmap v139: 129 pgs: 129 active+clean; 50 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 267 op/s 2024-03-28T22:41:41.256 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.258+0000 7f8dcee00640 1 -- 172.21.15.43:0/3807570264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8dc8107730 msgr2=0x7f8dc8107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:41.256 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.258+0000 7f8dcee00640 1 --2- 172.21.15.43:0/3807570264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8dc8107730 0x7f8dc8107b10 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f8db8009cd0 tx=0x7f8db802c0e0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.256 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.258+0000 7f8dcee00640 1 -- 172.21.15.43:0/3807570264 shutdown_connections 2024-03-28T22:41:41.257 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.258+0000 7f8dcee00640 1 --2- 172.21.15.43:0/3807570264 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8dc81020e0 0x7f8dc810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.257 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.258+0000 7f8dcee00640 1 --2- 172.21.15.43:0/3807570264 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8dc8101720 0x7f8dc8101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.257 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.258+0000 7f8dcee00640 1 --2- 172.21.15.43:0/3807570264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8dc8107730 0x7f8dc8107b10 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.257 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.258+0000 7f8dcee00640 1 -- 172.21.15.43:0/3807570264 >> 172.21.15.43:0/3807570264 conn(0x7f8dc80fd210 msgr2=0x7f8dc80ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:41.257 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.258+0000 7f8dcee00640 1 -- 172.21.15.43:0/3807570264 shutdown_connections 2024-03-28T22:41:41.257 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.258+0000 7f8dcee00640 1 -- 172.21.15.43:0/3807570264 wait complete. 2024-03-28T22:41:41.260 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.260+0000 7f8dcee00640 1 Processor -- start 2024-03-28T22:41:41.260 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.260+0000 7f8dcee00640 1 -- start start 2024-03-28T22:41:41.260 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.260+0000 7f8dcee00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8dc8101720 0x7f8dc8073a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:41.260 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.260+0000 7f8dcee00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8dc81020e0 0x7f8dc8073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:41.260 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.260+0000 7f8dcee00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8dc8107730 0x7f8dc80744d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:41.260 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.260+0000 7f8dcee00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc8111b00 con 0x7f8dc8101720 2024-03-28T22:41:41.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.260+0000 7f8dcee00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc8111980 con 0x7f8dc8107730 2024-03-28T22:41:41.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.260+0000 7f8dcee00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc8111c80 con 0x7f8dc81020e0 2024-03-28T22:41:41.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.261+0000 7f8dc7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8dc8101720 0x7f8dc8073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:41.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.261+0000 7f8dcce00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8dc8107730 0x7f8dc80744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:41.262 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.261+0000 7f8dc7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8dc8101720 0x7f8dc8073a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:41820/0 (socket says 172.21.15.43:41820) 2024-03-28T22:41:41.262 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.261+0000 7f8dc7e00640 1 -- 172.21.15.43:0/187961591 learned_addr learned my addr 172.21.15.43:0/187961591 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:41.262 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.261+0000 7f8dbf400640 1 --2- 172.21.15.43:0/187961591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8dc81020e0 0x7f8dc8073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:41.262 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.261+0000 7f8dc7e00640 1 -- 172.21.15.43:0/187961591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8dc81020e0 msgr2=0x7f8dc8073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:41.262 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.261+0000 7f8dc7e00640 1 --2- 172.21.15.43:0/187961591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8dc81020e0 0x7f8dc8073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.262 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.261+0000 7f8dc7e00640 1 -- 172.21.15.43:0/187961591 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8dc8107730 msgr2=0x7f8dc80744d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:41.262 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.261+0000 7f8dc7e00640 1 --2- 172.21.15.43:0/187961591 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8dc8107730 0x7f8dc80744d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.262 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.261+0000 7f8dc7e00640 1 -- 172.21.15.43:0/187961591 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dc81a64a0 con 0x7f8dc8101720 2024-03-28T22:41:41.263 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.261+0000 7f8dc7e00640 1 --2- 172.21.15.43:0/187961591 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8dc8101720 0x7f8dc8073a50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8db8000f80 tx=0x7f8db8032460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:41.263 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.262+0000 7f8dc5600640 1 -- 172.21.15.43:0/187961591 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db8039040 con 0x7f8dc8101720 2024-03-28T22:41:41.263 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.262+0000 7f8dc5600640 1 -- 172.21.15.43:0/187961591 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8db8044070 con 0x7f8dc8101720 2024-03-28T22:41:41.263 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.262+0000 7f8dcee00640 1 -- 172.21.15.43:0/187961591 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dc81a6730 con 0x7f8dc8101720 2024-03-28T22:41:41.263 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.262+0000 7f8dc5600640 1 -- 172.21.15.43:0/187961591 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db80042a0 con 0x7f8dc8101720 2024-03-28T22:41:41.263 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.262+0000 7f8dcee00640 1 -- 172.21.15.43:0/187961591 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dc81a6c70 con 0x7f8dc8101720 2024-03-28T22:41:41.271 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.265+0000 7f8dc5600640 1 -- 172.21.15.43:0/187961591 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f8db8003970 con 0x7f8dc8101720 2024-03-28T22:41:41.271 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.265+0000 7f8dd1b64740 1 Processor -- start 2024-03-28T22:41:41.271 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.266+0000 7f8dd1b64740 1 -- start start 2024-03-28T22:41:41.271 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.266+0000 7f8dd1b64740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580635233a0 0x558063523780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:41.271 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.266+0000 7f8dd1b64740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558063523cc0 0x55806352c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:41.271 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.266+0000 7f8dd1b64740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55806352ca90 0x55806352ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:41.271 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.266+0000 7f8dd1b64740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x558063486640 con 0x5580635233a0 2024-03-28T22:41:41.271 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.266+0000 7f8dd1b64740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55806342f810 con 0x558063523cc0 2024-03-28T22:41:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.266+0000 7f8dd1b64740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5580634826e0 con 0x55806352ca90 2024-03-28T22:41:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.266+0000 7f8dc7e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558063523cc0 0x55806352c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.266+0000 7f8dbf400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55806352ca90 0x55806352ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.266+0000 7f8dcce00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580635233a0 0x558063523780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.266+0000 7f8dc7e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558063523cc0 0x55806352c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48182/0 (socket says 172.21.15.43:48182) 2024-03-28T22:41:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.266+0000 7f8dc7e00640 1 -- 172.21.15.43:0/702074138 learned_addr learned my addr 172.21.15.43:0/702074138 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.266+0000 7f8dbf400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55806352ca90 0x55806352ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43858/0 (socket says 172.21.15.43:43858) 2024-03-28T22:41:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.267+0000 7f8dc7e00640 1 -- 172.21.15.43:0/702074138 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55806352ca90 msgr2=0x55806352ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.267+0000 7f8dc7e00640 1 --2- 172.21.15.43:0/702074138 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55806352ca90 0x55806352ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.267+0000 7f8dc7e00640 1 -- 172.21.15.43:0/702074138 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580635233a0 msgr2=0x558063523780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.267+0000 7f8dc7e00640 1 --2- 172.21.15.43:0/702074138 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580635233a0 0x558063523780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.267+0000 7f8dc7e00640 1 -- 172.21.15.43:0/702074138 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55806319b210 con 0x558063523cc0 2024-03-28T22:41:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.267+0000 7f8dc5600640 1 --2- 172.21.15.43:0/187961591 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f8dac0a2010 0x7f8dac0a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.267+0000 7f8dc7e00640 1 --2- 172.21.15.43:0/702074138 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558063523cc0 0x55806352c550 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f8db80edc70 tx=0x7f8db80f6b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6c63ccfc66fd364 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.267+0000 7f8dc5600640 1 -- 172.21.15.43:0/187961591 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f8db80f08d0 con 0x7f8dc8101720 2024-03-28T22:41:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.267+0000 7f8dbd600640 1 -- 172.21.15.43:0/702074138 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db80b9300 con 0x558063523cc0 2024-03-28T22:41:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.267+0000 7f8dcce00640 1 --2- 172.21.15.43:0/187961591 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f8dac0a2010 0x7f8dac0a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.268+0000 7f8dbd600640 1 -- 172.21.15.43:0/702074138 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8db8035de0 con 0x558063523cc0 2024-03-28T22:41:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.268+0000 7f8dbd600640 1 -- 172.21.15.43:0/702074138 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db80bc9c0 con 0x558063523cc0 2024-03-28T22:41:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.268+0000 7f8dd1b64740 1 -- 172.21.15.43:0/702074138 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558063523cc0 msgr2=0x55806352c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:41.274 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.268+0000 7f8dcce00640 1 --2- 172.21.15.43:0/187961591 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f8dac0a2010 0x7f8dac0a44d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0009d20 tx=0x7f8dc0005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:41.274 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.268+0000 7f8dd1b64740 1 --2- 172.21.15.43:0/702074138 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558063523cc0 0x55806352c550 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f8db80edc70 tx=0x7f8db80f6b20 comp rx=0 tx=0).stop 2024-03-28T22:41:41.274 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.268+0000 7f8dd1b64740 1 -- 172.21.15.43:0/702074138 shutdown_connections 2024-03-28T22:41:41.274 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.268+0000 7f8dd1b64740 1 --2- 172.21.15.43:0/702074138 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558063523cc0 0x55806352c550 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.274 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.268+0000 7f8dd1b64740 1 --2- 172.21.15.43:0/702074138 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55806352ca90 0x55806352ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.274 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.268+0000 7f8dd1b64740 1 --2- 172.21.15.43:0/702074138 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580635233a0 0x558063523780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.274 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.268+0000 7f8dd1b64740 1 -- 172.21.15.43:0/702074138 >> 172.21.15.43:0/702074138 conn(0x55806351dcc0 msgr2=0x55806351e0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:41.274 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.268+0000 7f8dd1b64740 1 -- 172.21.15.43:0/702074138 shutdown_connections 2024-03-28T22:41:41.275 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.269+0000 7f8dd1b64740 1 -- 172.21.15.43:0/702074138 wait complete. 2024-03-28T22:41:41.275 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.269+0000 7f8dd1b64740 1 -- start start 2024-03-28T22:41:41.275 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.269+0000 7f8dd1b64740 1 Processor -- start 2024-03-28T22:41:41.275 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.270+0000 7f8dd1b64740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580635233a0 0x558063529b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:41.275 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.270+0000 7f8dd1b64740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558063523cc0 0x55806352a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:41.275 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.270+0000 7f8dd1b64740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55806352ca90 0x55806352a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:41.275 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.270+0000 7f8dd1b64740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55806352f480 con 0x5580635233a0 2024-03-28T22:41:41.275 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.270+0000 7f8dd1b64740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x558063424d00 con 0x558063523cc0 2024-03-28T22:41:41.275 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.270+0000 7f8dd1b64740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55806352f600 con 0x55806352ca90 2024-03-28T22:41:41.275 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.270+0000 7f8dd1b64740 10 client.0 _ll_register_callbacks cb 0x55806340f490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:41:41.276 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.270+0000 7f8dc7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580635233a0 0x558063529b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:41.276 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.270+0000 7f8dcce00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55806352ca90 0x55806352a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:41.276 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.270+0000 7f8dbf400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558063523cc0 0x55806352a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:41.276 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.270+0000 7f8dcce00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55806352ca90 0x55806352a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43874/0 (socket says 172.21.15.43:43874) 2024-03-28T22:41:41.276 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.270+0000 7f8dbf400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558063523cc0 0x55806352a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48196/0 (socket says 172.21.15.43:48196) 2024-03-28T22:41:41.276 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.270+0000 7f8dcce00640 1 -- 172.21.15.43:0/1009883425 learned_addr learned my addr 172.21.15.43:0/1009883425 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:41.276 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.270+0000 7f8dc7e00640 1 -- 172.21.15.43:0/1009883425 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55806352ca90 msgr2=0x55806352a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:41.276 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.270+0000 7f8dc7e00640 1 --2- 172.21.15.43:0/1009883425 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55806352ca90 0x55806352a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.276 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.270+0000 7f8dc7e00640 1 -- 172.21.15.43:0/1009883425 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558063523cc0 msgr2=0x55806352a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:41.276 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.270+0000 7f8dc7e00640 1 --2- 172.21.15.43:0/1009883425 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558063523cc0 0x55806352a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.277 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.270+0000 7f8dc7e00640 1 -- 172.21.15.43:0/1009883425 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558063267620 con 0x5580635233a0 2024-03-28T22:41:41.277 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.271+0000 7f8dc7e00640 1 --2- 172.21.15.43:0/1009883425 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580635233a0 0x558063529b10 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x558063522e50 tx=0x7f8db802c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97b6ea2c74b6dd87 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:41.277 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.271+0000 7f8db6000640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:41:41.277 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.271+0000 7f8db6000640 1 -- 172.21.15.43:0/1009883425 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db80bcba0 con 0x5580635233a0 2024-03-28T22:41:41.277 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.271+0000 7f8db6000640 1 -- 172.21.15.43:0/1009883425 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f8db8032790 con 0x5580635233a0 2024-03-28T22:41:41.277 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.271+0000 7f8dd1b64740 1 -- 172.21.15.43:0/1009883425 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55806349a030 con 0x5580635233a0 2024-03-28T22:41:41.277 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.271+0000 7f8db6000640 1 -- 172.21.15.43:0/1009883425 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8db80bce30 con 0x5580635233a0 2024-03-28T22:41:41.277 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.271+0000 7f8dd1b64740 10 client.17178 fetch_fsmap learned FSMap version 32 2024-03-28T22:41:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.271+0000 7f8dd1b64740 1 -- 172.21.15.43:0/1009883425 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55806352f480 con 0x5580635233a0 2024-03-28T22:41:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.272+0000 7f8db6000640 1 -- 172.21.15.43:0/1009883425 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55806349a030 con 0x5580635233a0 2024-03-28T22:41:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.272+0000 7f8db6000640 1 -- 172.21.15.43:0/1009883425 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f8db803c030 con 0x5580635233a0 2024-03-28T22:41:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.272+0000 7f8dd1b64740 10 client.17178 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:41:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.272+0000 7f8dd1b64740 10 client.17178 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:41:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.272+0000 7f8dd1b64740 20 client.17178 populate_metadata read hostname 'smithi043' 2024-03-28T22:41:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.272+0000 7f8dd1b64740 1 --2- 172.21.15.43:0/1009883425 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55806353c5e0 0x55806353e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.272+0000 7f8dd1b64740 4 client.17178 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:41:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.272+0000 7f8dd1b64740 1 -- 172.21.15.43:0/1009883425 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558063493b80 con 0x55806353c5e0 2024-03-28T22:41:41.278 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.272+0000 7f8dbf400640 1 --2- 172.21.15.43:0/1009883425 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55806353c5e0 0x55806353e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.273+0000 7f8dbf400640 1 --2- 172.21.15.43:0/1009883425 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55806353c5e0 0x55806353e9d0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=518451e64629b8c server_cookie=7efc56ef556f7a3b in_seq=0 out_seq=0 2024-03-28T22:41:41.279 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.273+0000 7f8db6000640 10 client.17178 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:41.280 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.276+0000 7f8db6000640 1 -- 172.21.15.43:0/1009883425 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558063493b80 con 0x55806353c5e0 2024-03-28T22:41:41.280 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.276+0000 7f8db6000640 10 client.17178 handle_command_reply: tid=0 2024-03-28T22:41:41.280 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.276+0000 7f8dd1b64740 1 client.17178 shutdown 2024-03-28T22:41:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.276+0000 7f8dd1b64740 1 -- 172.21.15.43:0/1009883425 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580635233a0 msgr2=0x558063529b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.276+0000 7f8dd1b64740 1 --2- 172.21.15.43:0/1009883425 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580635233a0 0x558063529b10 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x558063522e50 tx=0x7f8db802c900 comp rx=0 tx=0).stop 2024-03-28T22:41:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.276+0000 7f8dd1b64740 1 -- 172.21.15.43:0/1009883425 shutdown_connections 2024-03-28T22:41:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.276+0000 7f8dd1b64740 1 --2- 172.21.15.43:0/1009883425 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55806353c5e0 0x55806353e9d0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.276+0000 7f8dd1b64740 1 --2- 172.21.15.43:0/1009883425 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558063523cc0 0x55806352a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.276+0000 7f8db6000640 0 client.17178 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.276+0000 7f8dd1b64740 1 --2- 172.21.15.43:0/1009883425 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55806352ca90 0x55806352a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.276+0000 7f8dd1b64740 1 --2- 172.21.15.43:0/1009883425 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580635233a0 0x558063529b10 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.276+0000 7f8dd1b64740 1 -- 172.21.15.43:0/1009883425 >> 172.21.15.43:0/1009883425 conn(0x55806351dcc0 msgr2=0x55806351fa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.276+0000 7f8dd1b64740 1 -- 172.21.15.43:0/1009883425 shutdown_connections 2024-03-28T22:41:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.277+0000 7f8dd1b64740 1 -- 172.21.15.43:0/1009883425 wait complete. 2024-03-28T22:41:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.277+0000 7f8dd1b64740 20 client.17178 trim_cache size 0 max 16384 2024-03-28T22:41:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.293+0000 7f8dd1b64740 1 Processor -- start 2024-03-28T22:41:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.293+0000 7f8dd1b64740 1 -- start start 2024-03-28T22:41:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.294+0000 7f8dd1b64740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580635233a0 0x55806352c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.294+0000 7f8dd1b64740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558063523cc0 0x558063538190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.294+0000 7f8dd1b64740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55806352ca90 0x558063544290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.294+0000 7f8dd1b64740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc81b0d70 con 0x5580635233a0 2024-03-28T22:41:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.294+0000 7f8dd1b64740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc81b28b0 con 0x55806352ca90 2024-03-28T22:41:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.294+0000 7f8dd1b64740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc81b1440 con 0x558063523cc0 2024-03-28T22:41:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.294+0000 7f8dbf400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558063523cc0 0x558063538190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.294+0000 7f8dc7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580635233a0 0x55806352c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.294+0000 7f8dbf400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558063523cc0 0x558063538190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43886/0 (socket says 172.21.15.43:43886) 2024-03-28T22:41:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.294+0000 7f8dbf400640 1 -- 172.21.15.43:0/4088120567 learned_addr learned my addr 172.21.15.43:0/4088120567 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.294+0000 7f8dcce00640 1 --2- 172.21.15.43:0/4088120567 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55806352ca90 0x558063544290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.294+0000 7f8dbf400640 1 -- 172.21.15.43:0/4088120567 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55806352ca90 msgr2=0x558063544290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.294+0000 7f8dbf400640 1 --2- 172.21.15.43:0/4088120567 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55806352ca90 0x558063544290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.294+0000 7f8dbf400640 1 -- 172.21.15.43:0/4088120567 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580635233a0 msgr2=0x55806352c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.294+0000 7f8dbf400640 1 --2- 172.21.15.43:0/4088120567 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580635233a0 0x55806352c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.294+0000 7f8dbf400640 1 -- 172.21.15.43:0/4088120567 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55806353f000 con 0x558063523cc0 2024-03-28T22:41:41.294 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.294+0000 7f8dc7e00640 1 --2- 172.21.15.43:0/4088120567 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580635233a0 0x55806352c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:41:41.295 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.295+0000 7f8dbf400640 1 --2- 172.21.15.43:0/4088120567 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558063523cc0 0x558063538190 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f8db0009dd0 tx=0x7f8db0005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f40930153e60db5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:41.295 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.295+0000 7f8dbcc00640 1 -- 172.21.15.43:0/4088120567 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db0023030 con 0x558063523cc0 2024-03-28T22:41:41.295 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.295+0000 7f8dbcc00640 1 -- 172.21.15.43:0/4088120567 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8db0005500 con 0x558063523cc0 2024-03-28T22:41:41.295 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.295+0000 7f8dbcc00640 1 -- 172.21.15.43:0/4088120567 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db001f2b0 con 0x558063523cc0 2024-03-28T22:41:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.296+0000 7f8dd1b64740 1 -- 172.21.15.43:0/4088120567 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558063523cc0 msgr2=0x558063538190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.296+0000 7f8dd1b64740 1 --2- 172.21.15.43:0/4088120567 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558063523cc0 0x558063538190 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f8db0009dd0 tx=0x7f8db0005f90 comp rx=0 tx=0).stop 2024-03-28T22:41:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.296+0000 7f8dd1b64740 1 -- 172.21.15.43:0/4088120567 shutdown_connections 2024-03-28T22:41:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.296+0000 7f8dd1b64740 1 --2- 172.21.15.43:0/4088120567 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55806352ca90 0x558063544290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.296+0000 7f8dd1b64740 1 --2- 172.21.15.43:0/4088120567 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558063523cc0 0x558063538190 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.296+0000 7f8dd1b64740 1 --2- 172.21.15.43:0/4088120567 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580635233a0 0x55806352c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.296+0000 7f8dd1b64740 1 -- 172.21.15.43:0/4088120567 >> 172.21.15.43:0/4088120567 conn(0x55806351dcc0 msgr2=0x5580635188d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.296+0000 7f8dd1b64740 1 -- 172.21.15.43:0/4088120567 shutdown_connections 2024-03-28T22:41:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.296+0000 7f8dd1b64740 1 -- 172.21.15.43:0/4088120567 wait complete. 2024-03-28T22:41:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.296+0000 7f8dd1b64740 1 -- start start 2024-03-28T22:41:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.296+0000 7f8dd1b64740 1 Processor -- start 2024-03-28T22:41:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.297+0000 7f8dd1b64740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580635233a0 0x5580635265e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.297+0000 7f8dd1b64740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558063523cc0 0x558063526b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.297+0000 7f8dd1b64740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55806352ca90 0x558063527060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.297+0000 7f8dd1b64740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc81b21e0 con 0x5580635233a0 2024-03-28T22:41:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.297+0000 7f8dd1b64740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc81b1b10 con 0x55806352ca90 2024-03-28T22:41:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.297+0000 7f8dd1b64740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc81aed50 con 0x558063523cc0 2024-03-28T22:41:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.297+0000 7f8dd1b64740 10 client.0 _ll_register_callbacks cb 0x7f8dc0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:41:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.297+0000 7f8dcce00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55806352ca90 0x558063527060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.297+0000 7f8dbf400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558063523cc0 0x558063526b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.297+0000 7f8dc7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580635233a0 0x5580635265e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.297+0000 7f8dcce00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55806352ca90 0x558063527060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48204/0 (socket says 172.21.15.43:48204) 2024-03-28T22:41:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.297+0000 7f8dcce00640 1 -- 172.21.15.43:0/381727324 learned_addr learned my addr 172.21.15.43:0/381727324 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.297+0000 7f8dcce00640 1 -- 172.21.15.43:0/381727324 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558063523cc0 msgr2=0x558063526b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.297+0000 7f8dcce00640 1 --2- 172.21.15.43:0/381727324 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558063523cc0 0x558063526b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.297+0000 7f8dcce00640 1 -- 172.21.15.43:0/381727324 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580635233a0 msgr2=0x5580635265e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.297+0000 7f8dcce00640 1 --2- 172.21.15.43:0/381727324 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580635233a0 0x5580635265e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.298+0000 7f8dcce00640 1 -- 172.21.15.43:0/381727324 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580635a18c0 con 0x55806352ca90 2024-03-28T22:41:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.298+0000 7f8dcce00640 1 --2- 172.21.15.43:0/381727324 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55806352ca90 0x558063527060 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f8dc00045c0 tx=0x7f8dc00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3571e7e73c99398a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.298+0000 7f8db6000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:41:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.298+0000 7f8db6000640 1 -- 172.21.15.43:0/381727324 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc002f060 con 0x55806352ca90 2024-03-28T22:41:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.298+0000 7f8dd1b64740 1 -- 172.21.15.43:0/381727324 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8db0005500 con 0x55806352ca90 2024-03-28T22:41:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.298+0000 7f8db6000640 1 -- 172.21.15.43:0/381727324 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f8dc001f6d0 con 0x55806352ca90 2024-03-28T22:41:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.298+0000 7f8db6000640 1 -- 172.21.15.43:0/381727324 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8dc0036050 con 0x55806352ca90 2024-03-28T22:41:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.299+0000 7f8dd1b64740 10 client.27404 fetch_fsmap learned FSMap version 32 2024-03-28T22:41:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.299+0000 7f8dd1b64740 1 -- 172.21.15.43:0/381727324 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5580635a2b20 con 0x55806352ca90 2024-03-28T22:41:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.299+0000 7f8db6000640 1 -- 172.21.15.43:0/381727324 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8db0005500 con 0x55806352ca90 2024-03-28T22:41:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.299+0000 7f8db6000640 1 -- 172.21.15.43:0/381727324 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f8dc002aaf0 con 0x55806352ca90 2024-03-28T22:41:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.299+0000 7f8dd1b64740 10 client.27404 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:41:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.299+0000 7f8dd1b64740 10 client.27404 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:41:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.299+0000 7f8dd1b64740 20 client.27404 populate_metadata read hostname 'smithi043' 2024-03-28T22:41:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.299+0000 7f8dd1b64740 1 --2- 172.21.15.43:0/381727324 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5580635a32b0 0x5580635a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.299+0000 7f8dd1b64740 4 client.27404 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T22:41:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.299+0000 7f8dd1b64740 1 -- 172.21.15.43:0/381727324 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55806348ded0 con 0x5580635a32b0 2024-03-28T22:41:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.299+0000 7f8dc7e00640 1 --2- 172.21.15.43:0/381727324 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5580635a32b0 0x5580635a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.300+0000 7f8dc7e00640 1 --2- 172.21.15.43:0/381727324 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5580635a32b0 0x5580635a56d0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9f65a6ed17fbd07 server_cookie=7541c4938c9c0da8 in_seq=0 out_seq=0 2024-03-28T22:41:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.300+0000 7f8db6000640 10 client.27404 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:41.361 INFO:tasks.workunit.client.0.smithi043.stdout: 1 292363 8.01 MB/sec execute 72 sec latency 31.365 ms 2024-03-28T22:41:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.508+0000 7f8db6000640 1 -- 172.21.15.43:0/381727324 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55806348ded0 con 0x5580635a32b0 2024-03-28T22:41:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.508+0000 7f8db6000640 10 client.27404 handle_command_reply: tid=0 2024-03-28T22:41:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.508+0000 7f8dd1b64740 1 client.27404 shutdown 2024-03-28T22:41:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.508+0000 7f8dd1b64740 1 -- 172.21.15.43:0/381727324 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55806352ca90 msgr2=0x558063527060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.508+0000 7f8dd1b64740 1 --2- 172.21.15.43:0/381727324 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55806352ca90 0x558063527060 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f8dc00045c0 tx=0x7f8dc00046c0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.508+0000 7f8dd1b64740 1 -- 172.21.15.43:0/381727324 shutdown_connections 2024-03-28T22:41:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.508+0000 7f8dd1b64740 1 --2- 172.21.15.43:0/381727324 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5580635a32b0 0x5580635a56d0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.508+0000 7f8dd1b64740 1 --2- 172.21.15.43:0/381727324 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55806352ca90 0x558063527060 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.508+0000 7f8dd1b64740 1 --2- 172.21.15.43:0/381727324 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558063523cc0 0x558063526b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.508+0000 7f8dd1b64740 1 --2- 172.21.15.43:0/381727324 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580635233a0 0x5580635265e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.508+0000 7f8db6000640 0 client.27404 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.508+0000 7f8dd1b64740 1 -- 172.21.15.43:0/381727324 >> 172.21.15.43:0/381727324 conn(0x55806351dcc0 msgr2=0x55806351c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.509+0000 7f8dd1b64740 1 -- 172.21.15.43:0/381727324 shutdown_connections 2024-03-28T22:41:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.509+0000 7f8dd1b64740 1 -- 172.21.15.43:0/381727324 wait complete. 2024-03-28T22:41:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.509+0000 7f8dd1b64740 20 client.27404 trim_cache size 0 max 16384 2024-03-28T22:41:41.509 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:{ 2024-03-28T22:41:41.509 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "message": "", 2024-03-28T22:41:41.509 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "return_code": 0 2024-03-28T22:41:41.509 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:} 2024-03-28T22:41:41.510 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.509+0000 7f8dcee00640 1 -- 172.21.15.43:0/187961591 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f8dac0a2010 msgr2=0x7f8dac0a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:41.510 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.509+0000 7f8dcee00640 1 --2- 172.21.15.43:0/187961591 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f8dac0a2010 0x7f8dac0a44d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0009d20 tx=0x7f8dc0005f90 comp rx=0 tx=0).stop 2024-03-28T22:41:41.510 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.510+0000 7f8dcee00640 1 -- 172.21.15.43:0/187961591 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8dc8101720 msgr2=0x7f8dc8073a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:41.510 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.510+0000 7f8dcee00640 1 --2- 172.21.15.43:0/187961591 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8dc8101720 0x7f8dc8073a50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8db8000f80 tx=0x7f8db8032460 comp rx=0 tx=0).stop 2024-03-28T22:41:41.510 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.510+0000 7f8dcee00640 1 -- 172.21.15.43:0/187961591 shutdown_connections 2024-03-28T22:41:41.510 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.510+0000 7f8dcee00640 1 --2- 172.21.15.43:0/187961591 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f8dac0a2010 0x7f8dac0a44d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.511 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.510+0000 7f8dcee00640 1 --2- 172.21.15.43:0/187961591 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8dc8107730 0x7f8dc80744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.511 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.510+0000 7f8dcee00640 1 --2- 172.21.15.43:0/187961591 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8dc81020e0 0x7f8dc8073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.511 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.510+0000 7f8dcee00640 1 --2- 172.21.15.43:0/187961591 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8dc8101720 0x7f8dc8073a50 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:41.511 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.510+0000 7f8dcee00640 1 -- 172.21.15.43:0/187961591 >> 172.21.15.43:0/187961591 conn(0x7f8dc80fd210 msgr2=0x7f8dc80fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:41.511 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.510+0000 7f8dcee00640 1 -- 172.21.15.43:0/187961591 shutdown_connections 2024-03-28T22:41:41.511 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:41.510+0000 7f8dcee00640 1 -- 172.21.15.43:0/187961591 wait complete. 2024-03-28T22:41:42.360 INFO:tasks.workunit.client.0.smithi043.stdout: 1 294403 8.07 MB/sec execute 73 sec latency 125.964 ms 2024-03-28T22:41:42.946 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:42 smithi172 ceph-mon[36566]: pgmap v140: 129 pgs: 129 active+clean; 50 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 267 op/s 2024-03-28T22:41:42.948 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:42 smithi082 ceph-mon[36471]: pgmap v140: 129 pgs: 129 active+clean; 50 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 267 op/s 2024-03-28T22:41:43.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:42 smithi043 ceph-mon[31823]: pgmap v140: 129 pgs: 129 active+clean; 50 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 267 op/s 2024-03-28T22:41:43.361 INFO:tasks.workunit.client.0.smithi043.stdout: 1 295351 7.99 MB/sec execute 74 sec latency 21.112 ms 2024-03-28T22:41:44.361 INFO:tasks.workunit.client.0.smithi043.stdout: 1 297309 8.03 MB/sec execute 75 sec latency 47.220 ms 2024-03-28T22:41:44.949 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:44 smithi082 ceph-mon[36471]: pgmap v141: 129 pgs: 129 active+clean; 41 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 444 op/s 2024-03-28T22:41:45.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:44 smithi043 ceph-mon[31823]: pgmap v141: 129 pgs: 129 active+clean; 41 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 444 op/s 2024-03-28T22:41:45.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:44 smithi172 ceph-mon[36566]: pgmap v141: 129 pgs: 129 active+clean; 41 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 444 op/s 2024-03-28T22:41:45.361 INFO:tasks.workunit.client.0.smithi043.stdout: 1 298480 8.03 MB/sec execute 76 sec latency 11.048 ms 2024-03-28T22:41:46.361 INFO:tasks.workunit.client.0.smithi043.stdout: 1 299929 7.99 MB/sec execute 77 sec latency 47.390 ms 2024-03-28T22:41:46.948 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:46 smithi082 ceph-mon[36471]: pgmap v142: 129 pgs: 129 active+clean; 41 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 253 op/s 2024-03-28T22:41:47.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:46 smithi043 ceph-mon[31823]: pgmap v142: 129 pgs: 129 active+clean; 41 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 253 op/s 2024-03-28T22:41:47.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:46 smithi172 ceph-mon[36566]: pgmap v142: 129 pgs: 129 active+clean; 41 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 253 op/s 2024-03-28T22:41:47.361 INFO:tasks.workunit.client.0.smithi043.stdout: 1 301678 8.05 MB/sec execute 78 sec latency 13.211 ms 2024-03-28T22:41:48.361 INFO:tasks.workunit.client.0.smithi043.stdout: 1 302512 7.97 MB/sec execute 79 sec latency 15.234 ms 2024-03-28T22:41:49.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:48 smithi043 ceph-mon[31823]: pgmap v143: 129 pgs: 129 active+clean; 45 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 340 op/s 2024-03-28T22:41:49.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:48 smithi172 ceph-mon[36566]: pgmap v143: 129 pgs: 129 active+clean; 45 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 340 op/s 2024-03-28T22:41:49.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:48 smithi082 ceph-mon[36471]: pgmap v143: 129 pgs: 129 active+clean; 45 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 340 op/s 2024-03-28T22:41:49.361 INFO:tasks.workunit.client.0.smithi043.stdout: 1 304129 7.96 MB/sec execute 80 sec latency 96.743 ms 2024-03-28T22:41:50.361 INFO:tasks.workunit.client.0.smithi043.stdout: 1 305542 8.00 MB/sec execute 81 sec latency 10.222 ms 2024-03-28T22:41:51.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:50 smithi043 ceph-mon[31823]: pgmap v144: 129 pgs: 129 active+clean; 45 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 264 op/s 2024-03-28T22:41:51.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:50 smithi172 ceph-mon[36566]: pgmap v144: 129 pgs: 129 active+clean; 45 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 264 op/s 2024-03-28T22:41:51.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:50 smithi082 ceph-mon[36471]: pgmap v144: 129 pgs: 129 active+clean; 45 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 264 op/s 2024-03-28T22:41:51.362 INFO:tasks.workunit.client.0.smithi043.stdout: 1 306980 7.97 MB/sec execute 82 sec latency 20.247 ms 2024-03-28T22:41:52.362 INFO:tasks.workunit.client.0.smithi043.stdout: 1 308826 8.04 MB/sec execute 83 sec latency 10.544 ms 2024-03-28T22:41:53.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:52 smithi043 ceph-mon[31823]: pgmap v145: 129 pgs: 129 active+clean; 45 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 264 op/s 2024-03-28T22:41:53.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:52 smithi172 ceph-mon[36566]: pgmap v145: 129 pgs: 129 active+clean; 45 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 264 op/s 2024-03-28T22:41:53.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:52 smithi082 ceph-mon[36471]: pgmap v145: 129 pgs: 129 active+clean; 45 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 264 op/s 2024-03-28T22:41:53.362 INFO:tasks.workunit.client.0.smithi043.stdout: 1 309792 7.97 MB/sec execute 84 sec latency 14.190 ms 2024-03-28T22:41:54.362 INFO:tasks.workunit.client.0.smithi043.stdout: 1 311611 8.00 MB/sec execute 85 sec latency 51.712 ms 2024-03-28T22:41:54.661 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.663+0000 7f6233200640 1 -- 172.21.15.43:0/1924161305 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f622c103740 msgr2=0x7f622c103bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:54.661 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.663+0000 7f6233200640 1 --2- 172.21.15.43:0/1924161305 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f622c103740 0x7f622c103bc0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f621c009900 tx=0x7f621c02bc50 comp rx=0 tx=0).stop 2024-03-28T22:41:54.661 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.664+0000 7f6233200640 1 -- 172.21.15.43:0/1924161305 shutdown_connections 2024-03-28T22:41:54.662 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.664+0000 7f6233200640 1 --2- 172.21.15.43:0/1924161305 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f622c103740 0x7f622c103bc0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.662 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.664+0000 7f6233200640 1 --2- 172.21.15.43:0/1924161305 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f622c104100 0x7f622c110610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.662 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.664+0000 7f6233200640 1 --2- 172.21.15.43:0/1924161305 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f622c109750 0x7f622c109b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.662 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.664+0000 7f6233200640 1 -- 172.21.15.43:0/1924161305 >> 172.21.15.43:0/1924161305 conn(0x7f622c0ff230 msgr2=0x7f622c101650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:54.662 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.664+0000 7f6233200640 1 -- 172.21.15.43:0/1924161305 shutdown_connections 2024-03-28T22:41:54.662 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.664+0000 7f6233200640 1 -- 172.21.15.43:0/1924161305 wait complete. 2024-03-28T22:41:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.665+0000 7f6233200640 1 Processor -- start 2024-03-28T22:41:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.665+0000 7f6233200640 1 -- start start 2024-03-28T22:41:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.665+0000 7f6233200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f622c103740 0x7f622c19a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.665+0000 7f6233200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f622c104100 0x7f622c19a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.666+0000 7f6233200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f622c109750 0x7f622c19e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:54.663 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.666+0000 7f6233200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f622c113bb0 con 0x7f622c103740 2024-03-28T22:41:54.664 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.666+0000 7f6233200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f622c113a30 con 0x7f622c109750 2024-03-28T22:41:54.664 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.666+0000 7f6233200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f622c113d30 con 0x7f622c104100 2024-03-28T22:41:54.664 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.666+0000 7f6231400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f622c104100 0x7f622c19a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:54.664 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.666+0000 7f6232800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f622c109750 0x7f622c19e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:54.664 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.666+0000 7f6231e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f622c103740 0x7f622c19a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:54.664 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.666+0000 7f6231400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f622c104100 0x7f622c19a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:38492/0 (socket says 172.21.15.43:38492) 2024-03-28T22:41:54.664 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.666+0000 7f6231400640 1 -- 172.21.15.43:0/3190432208 learned_addr learned my addr 172.21.15.43:0/3190432208 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:54.665 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.666+0000 7f6232800640 1 -- 172.21.15.43:0/3190432208 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f622c104100 msgr2=0x7f622c19a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:54.665 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.666+0000 7f6232800640 1 --2- 172.21.15.43:0/3190432208 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f622c104100 0x7f622c19a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.665 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.666+0000 7f6232800640 1 -- 172.21.15.43:0/3190432208 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f622c103740 msgr2=0x7f622c19a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:54.665 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.666+0000 7f6232800640 1 --2- 172.21.15.43:0/3190432208 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f622c103740 0x7f622c19a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.665 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.666+0000 7f6232800640 1 -- 172.21.15.43:0/3190432208 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f622c19eff0 con 0x7f622c109750 2024-03-28T22:41:54.665 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.667+0000 7f6232800640 1 --2- 172.21.15.43:0/3190432208 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f622c109750 0x7f622c19e8c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f6228009fd0 tx=0x7f622800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:54.665 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.667+0000 7f6226a00640 1 -- 172.21.15.43:0/3190432208 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f622800ce20 con 0x7f622c109750 2024-03-28T22:41:54.666 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.667+0000 7f6226a00640 1 -- 172.21.15.43:0/3190432208 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6228005020 con 0x7f622c109750 2024-03-28T22:41:54.666 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.667+0000 7f6226a00640 1 -- 172.21.15.43:0/3190432208 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6228010720 con 0x7f622c109750 2024-03-28T22:41:54.666 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.667+0000 7f6233200640 1 -- 172.21.15.43:0/3190432208 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f622c19f2e0 con 0x7f622c109750 2024-03-28T22:41:54.666 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.667+0000 7f6233200640 1 -- 172.21.15.43:0/3190432208 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f622c1a6b70 con 0x7f622c109750 2024-03-28T22:41:54.669 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.671+0000 7f6237595740 1 Processor -- start 2024-03-28T22:41:54.669 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.672+0000 7f6226a00640 1 -- 172.21.15.43:0/3190432208 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f6228005540 con 0x7f622c109750 2024-03-28T22:41:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.672+0000 7f6237595740 1 -- start start 2024-03-28T22:41:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.672+0000 7f6237595740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fae8ec570 0x562fae8ec950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.672+0000 7f6237595740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fae8ece90 0x562fae8f5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.672+0000 7f6237595740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fae8f5c60 0x562fae8f8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.672+0000 7f6237595740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562fae84f640 con 0x562fae8ec570 2024-03-28T22:41:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.672+0000 7f6237595740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x562fae7f8810 con 0x562fae8ece90 2024-03-28T22:41:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.672+0000 7f6237595740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x562fae84b6e0 con 0x562fae8f5c60 2024-03-28T22:41:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.673+0000 7f6231e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fae8ece90 0x562fae8f5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.673+0000 7f6231e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fae8ece90 0x562fae8f5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52484/0 (socket says 172.21.15.43:52484) 2024-03-28T22:41:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.673+0000 7f6231e00640 1 -- 172.21.15.43:0/3000785025 learned_addr learned my addr 172.21.15.43:0/3000785025 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.673+0000 7f6231400640 1 --2- 172.21.15.43:0/3000785025 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fae8ec570 0x562fae8ec950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.673+0000 7f6231400640 1 --2- 172.21.15.43:0/3000785025 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fae8f5c60 0x562fae8f8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.673+0000 7f6231e00640 1 -- 172.21.15.43:0/3000785025 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fae8f5c60 msgr2=0x562fae8f8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.673+0000 7f6231e00640 1 --2- 172.21.15.43:0/3000785025 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fae8f5c60 0x562fae8f8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.673+0000 7f6231e00640 1 -- 172.21.15.43:0/3000785025 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fae8ec570 msgr2=0x562fae8ec950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.673+0000 7f6231e00640 1 --2- 172.21.15.43:0/3000785025 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fae8ec570 0x562fae8ec950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.673+0000 7f6231e00640 1 -- 172.21.15.43:0/3000785025 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fae564210 con 0x562fae8ece90 2024-03-28T22:41:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.673+0000 7f6231400640 1 --2- 172.21.15.43:0/3000785025 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fae8f5c60 0x562fae8f8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:41:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.673+0000 7f6231e00640 1 --2- 172.21.15.43:0/3000785025 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fae8ece90 0x562fae8f5720 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f6220009c70 tx=0x7f6220011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a33384e88a037c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.673+0000 7f6226a00640 1 --2- 172.21.15.43:0/3190432208 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f62140a1ee0 0x7f62140a43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.674+0000 7f6226a00640 1 -- 172.21.15.43:0/3190432208 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f62280cec10 con 0x7f622c109750 2024-03-28T22:41:54.673 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.674+0000 7f621b000640 1 -- 172.21.15.43:0/3000785025 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6220004230 con 0x562fae8ece90 2024-03-28T22:41:54.673 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.674+0000 7f621b000640 1 -- 172.21.15.43:0/3000785025 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62200043d0 con 0x562fae8ece90 2024-03-28T22:41:54.673 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.674+0000 7f621b000640 1 -- 172.21.15.43:0/3000785025 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6220019800 con 0x562fae8ece90 2024-03-28T22:41:54.673 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.674+0000 7f6231400640 1 --2- 172.21.15.43:0/3190432208 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f62140a1ee0 0x7f62140a43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:54.676 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.674+0000 7f6237595740 1 -- 172.21.15.43:0/3000785025 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fae8ece90 msgr2=0x562fae8f5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:54.676 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.674+0000 7f6231400640 1 --2- 172.21.15.43:0/3190432208 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f62140a1ee0 0x7f62140a43a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x562fae8e8020 tx=0x7f621c005df0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:54.676 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.674+0000 7f6237595740 1 --2- 172.21.15.43:0/3000785025 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fae8ece90 0x562fae8f5720 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f6220009c70 tx=0x7f6220011810 comp rx=0 tx=0).stop 2024-03-28T22:41:54.676 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.675+0000 7f6237595740 1 -- 172.21.15.43:0/3000785025 shutdown_connections 2024-03-28T22:41:54.676 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.675+0000 7f6237595740 1 --2- 172.21.15.43:0/3000785025 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fae8ece90 0x562fae8f5720 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.676 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.675+0000 7f6237595740 1 --2- 172.21.15.43:0/3000785025 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fae8f5c60 0x562fae8f8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.676 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.675+0000 7f6237595740 1 --2- 172.21.15.43:0/3000785025 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fae8ec570 0x562fae8ec950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.676 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.675+0000 7f6237595740 1 -- 172.21.15.43:0/3000785025 >> 172.21.15.43:0/3000785025 conn(0x562fae8e6e90 msgr2=0x562fae8e7270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:54.677 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.675+0000 7f6237595740 1 -- 172.21.15.43:0/3000785025 shutdown_connections 2024-03-28T22:41:54.677 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.675+0000 7f6237595740 1 -- 172.21.15.43:0/3000785025 wait complete. 2024-03-28T22:41:54.677 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.675+0000 7f6237595740 1 -- start start 2024-03-28T22:41:54.677 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.675+0000 7f6237595740 1 Processor -- start 2024-03-28T22:41:54.677 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.676+0000 7f6237595740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fae8ec570 0x562fae8f2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:54.677 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.676+0000 7f6237595740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fae8ece90 0x562fae8f31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:54.677 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.676+0000 7f6237595740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fae8f5c60 0x562fae8f3720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:54.677 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.676+0000 7f6237595740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562fae8f8890 con 0x562fae8f5c60 2024-03-28T22:41:54.677 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.676+0000 7f6237595740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x562fae7edd00 con 0x562fae8ec570 2024-03-28T22:41:54.677 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.676+0000 7f6237595740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x562fae8f8a10 con 0x562fae8ece90 2024-03-28T22:41:54.677 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.676+0000 7f6237595740 10 client.0 _ll_register_callbacks cb 0x562fae7d8490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:41:54.677 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.676+0000 7f6231e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fae8ec570 0x562fae8f2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:54.677 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.676+0000 7f6231400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fae8ece90 0x562fae8f31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:54.677 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.676+0000 7f6231e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fae8ec570 0x562fae8f2ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52498/0 (socket says 172.21.15.43:52498) 2024-03-28T22:41:54.677 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.676+0000 7f6231e00640 1 -- 172.21.15.43:0/2452308634 learned_addr learned my addr 172.21.15.43:0/2452308634 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.676+0000 7f6232800640 1 --2- 172.21.15.43:0/2452308634 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fae8f5c60 0x562fae8f3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.677+0000 7f6231e00640 1 -- 172.21.15.43:0/2452308634 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fae8ece90 msgr2=0x562fae8f31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.677+0000 7f6231e00640 1 --2- 172.21.15.43:0/2452308634 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fae8ece90 0x562fae8f31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.677+0000 7f6231e00640 1 -- 172.21.15.43:0/2452308634 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fae8f5c60 msgr2=0x562fae8f3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.677+0000 7f6231e00640 1 --2- 172.21.15.43:0/2452308634 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fae8f5c60 0x562fae8f3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.677+0000 7f6231e00640 1 -- 172.21.15.43:0/2452308634 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fae630620 con 0x562fae8ec570 2024-03-28T22:41:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.677+0000 7f6231e00640 1 --2- 172.21.15.43:0/2452308634 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fae8ec570 0x562fae8f2ca0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f6220000c00 tx=0x7f6220003da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3536cc606c6d2f47 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.677+0000 7f620f400640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:41:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.677+0000 7f620f400640 1 -- 172.21.15.43:0/2452308634 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6220027070 con 0x562fae8ec570 2024-03-28T22:41:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.677+0000 7f620f400640 1 -- 172.21.15.43:0/2452308634 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f6220003800 con 0x562fae8ec570 2024-03-28T22:41:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.677+0000 7f6237595740 1 -- 172.21.15.43:0/2452308634 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62200043d0 con 0x562fae8ec570 2024-03-28T22:41:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.678+0000 7f620f400640 1 -- 172.21.15.43:0/2452308634 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6220033050 con 0x562fae8ec570 2024-03-28T22:41:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.678+0000 7f6237595740 10 client.27416 fetch_fsmap learned FSMap version 32 2024-03-28T22:41:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.678+0000 7f6237595740 1 -- 172.21.15.43:0/2452308634 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562fae9046f0 con 0x562fae8ec570 2024-03-28T22:41:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.678+0000 7f620f400640 1 -- 172.21.15.43:0/2452308634 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62200043d0 con 0x562fae8ec570 2024-03-28T22:41:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.678+0000 7f620f400640 1 -- 172.21.15.43:0/2452308634 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6220018070 con 0x562fae8ec570 2024-03-28T22:41:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.678+0000 7f6237595740 10 client.27416 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:41:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.678+0000 7f6237595740 10 client.27416 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:41:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.678+0000 7f6237595740 20 client.27416 populate_metadata read hostname 'smithi043' 2024-03-28T22:41:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.678+0000 7f6237595740 1 --2- 172.21.15.43:0/2452308634 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x562fae905aa0 0x562fae907e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.678+0000 7f6237595740 4 client.27416 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:41:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.678+0000 7f6237595740 1 -- 172.21.15.43:0/2452308634 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562fae863030 con 0x562fae905aa0 2024-03-28T22:41:54.679 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.679+0000 7f6231400640 1 --2- 172.21.15.43:0/2452308634 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x562fae905aa0 0x562fae907e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:54.680 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.679+0000 7f6231400640 1 --2- 172.21.15.43:0/2452308634 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x562fae905aa0 0x562fae907e90 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25fbcbf9542a0b3c server_cookie=74a800f1b10ddbdc in_seq=0 out_seq=0 2024-03-28T22:41:54.680 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.679+0000 7f620f400640 10 client.27416 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:54.680 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.681+0000 7f620f400640 1 -- 172.21.15.43:0/2452308634 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562fae863030 con 0x562fae905aa0 2024-03-28T22:41:54.680 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.681+0000 7f620f400640 10 client.27416 handle_command_reply: tid=0 2024-03-28T22:41:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.681+0000 7f6237595740 1 client.27416 shutdown 2024-03-28T22:41:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.682+0000 7f6237595740 1 -- 172.21.15.43:0/2452308634 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fae8ec570 msgr2=0x562fae8f2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.682+0000 7f6237595740 1 --2- 172.21.15.43:0/2452308634 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fae8ec570 0x562fae8f2ca0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f6220000c00 tx=0x7f6220003da0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.682+0000 7f6237595740 1 -- 172.21.15.43:0/2452308634 shutdown_connections 2024-03-28T22:41:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.682+0000 7f6237595740 1 --2- 172.21.15.43:0/2452308634 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fae8f5c60 0x562fae8f3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.682+0000 7f6237595740 1 --2- 172.21.15.43:0/2452308634 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fae8ece90 0x562fae8f31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.682+0000 7f6237595740 1 --2- 172.21.15.43:0/2452308634 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x562fae905aa0 0x562fae907e90 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.682+0000 7f6237595740 1 --2- 172.21.15.43:0/2452308634 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fae8ec570 0x562fae8f2ca0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.682+0000 7f6237595740 1 -- 172.21.15.43:0/2452308634 >> 172.21.15.43:0/2452308634 conn(0x562fae8e6e90 msgr2=0x562fae8e8be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.682+0000 7f620f400640 0 client.27416 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.682+0000 7f6237595740 1 -- 172.21.15.43:0/2452308634 shutdown_connections 2024-03-28T22:41:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.682+0000 7f6237595740 1 -- 172.21.15.43:0/2452308634 wait complete. 2024-03-28T22:41:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.682+0000 7f6237595740 20 client.27416 trim_cache size 0 max 16384 2024-03-28T22:41:54.700 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.701+0000 7f6237595740 1 Processor -- start 2024-03-28T22:41:54.700 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.701+0000 7f6237595740 1 -- start start 2024-03-28T22:41:54.700 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.701+0000 7f6237595740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fae8ec570 0x562fae8f53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:54.700 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.701+0000 7f6237595740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fae8ece90 0x562fae901630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:54.700 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.702+0000 7f6237595740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fae8f5c60 0x562fae8e89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:54.700 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.702+0000 7f6237595740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f622c1b0c90 con 0x562fae8f5c60 2024-03-28T22:41:54.700 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.702+0000 7f6237595740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f622c1b27d0 con 0x562fae8ec570 2024-03-28T22:41:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.702+0000 7f6237595740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f622c1b1360 con 0x562fae8ece90 2024-03-28T22:41:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.702+0000 7f6231400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fae8ece90 0x562fae901630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.702+0000 7f6231e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fae8ec570 0x562fae8f53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.702+0000 7f6231400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fae8ece90 0x562fae901630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:38518/0 (socket says 172.21.15.43:38518) 2024-03-28T22:41:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.702+0000 7f6231400640 1 -- 172.21.15.43:0/4255453345 learned_addr learned my addr 172.21.15.43:0/4255453345 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.702+0000 7f6232800640 1 --2- 172.21.15.43:0/4255453345 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fae8f5c60 0x562fae8e89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.702+0000 7f6231400640 1 -- 172.21.15.43:0/4255453345 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fae8ec570 msgr2=0x562fae8f53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.702+0000 7f6231400640 1 --2- 172.21.15.43:0/4255453345 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fae8ec570 0x562fae8f53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.702+0000 7f6231400640 1 -- 172.21.15.43:0/4255453345 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fae8f5c60 msgr2=0x562fae8e89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.702+0000 7f6231400640 1 --2- 172.21.15.43:0/4255453345 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fae8f5c60 0x562fae8e89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.702+0000 7f6231400640 1 -- 172.21.15.43:0/4255453345 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fae9083f0 con 0x562fae8ece90 2024-03-28T22:41:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.702+0000 7f6231400640 1 --2- 172.21.15.43:0/4255453345 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fae8ece90 0x562fae901630 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f621c000c00 tx=0x7f621c041040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a19505b3f7013351 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.703+0000 7f621a600640 1 -- 172.21.15.43:0/4255453345 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621c055d90 con 0x562fae8ece90 2024-03-28T22:41:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.703+0000 7f621a600640 1 -- 172.21.15.43:0/4255453345 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f621c0046c0 con 0x562fae8ece90 2024-03-28T22:41:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.703+0000 7f621a600640 1 -- 172.21.15.43:0/4255453345 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621c005240 con 0x562fae8ece90 2024-03-28T22:41:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.704+0000 7f6237595740 1 -- 172.21.15.43:0/4255453345 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fae8ece90 msgr2=0x562fae901630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.704+0000 7f6237595740 1 --2- 172.21.15.43:0/4255453345 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fae8ece90 0x562fae901630 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f621c000c00 tx=0x7f621c041040 comp rx=0 tx=0).stop 2024-03-28T22:41:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.704+0000 7f6237595740 1 -- 172.21.15.43:0/4255453345 shutdown_connections 2024-03-28T22:41:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.704+0000 7f6237595740 1 --2- 172.21.15.43:0/4255453345 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fae8f5c60 0x562fae8e89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.704+0000 7f6237595740 1 --2- 172.21.15.43:0/4255453345 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fae8ece90 0x562fae901630 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.704+0000 7f6237595740 1 --2- 172.21.15.43:0/4255453345 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fae8ec570 0x562fae8f53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.704+0000 7f6237595740 1 -- 172.21.15.43:0/4255453345 >> 172.21.15.43:0/4255453345 conn(0x562fae8e6e90 msgr2=0x562fae8e8290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.704+0000 7f6237595740 1 -- 172.21.15.43:0/4255453345 shutdown_connections 2024-03-28T22:41:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.704+0000 7f6237595740 1 -- 172.21.15.43:0/4255453345 wait complete. 2024-03-28T22:41:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.704+0000 7f6237595740 1 -- start start 2024-03-28T22:41:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.704+0000 7f6237595740 1 Processor -- start 2024-03-28T22:41:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.705+0000 7f6237595740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fae8ec570 0x562fae8f0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.705+0000 7f6237595740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fae8ece90 0x562fae8f0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.705+0000 7f6237595740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fae8f5c60 0x562fae96ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:54.706 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.705+0000 7f6237595740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f622c1b2100 con 0x562fae8ece90 2024-03-28T22:41:54.706 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.705+0000 7f6237595740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f622c1b1a30 con 0x562fae8ec570 2024-03-28T22:41:54.706 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.705+0000 7f6237595740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f622c1aec70 con 0x562fae8f5c60 2024-03-28T22:41:54.706 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.705+0000 7f6237595740 10 client.0 _ll_register_callbacks cb 0x7f6220009af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:41:54.706 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.705+0000 7f6231e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fae8ec570 0x562fae8f0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:54.706 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.705+0000 7f6231e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fae8ec570 0x562fae8f0050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52520/0 (socket says 172.21.15.43:52520) 2024-03-28T22:41:54.706 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.705+0000 7f6231e00640 1 -- 172.21.15.43:0/1744262676 learned_addr learned my addr 172.21.15.43:0/1744262676 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:41:54.706 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.705+0000 7f6232800640 1 --2- 172.21.15.43:0/1744262676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fae8f5c60 0x562fae96ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:54.706 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.705+0000 7f6231400640 1 --2- 172.21.15.43:0/1744262676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fae8ece90 0x562fae8f0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.706+0000 7f6231e00640 1 -- 172.21.15.43:0/1744262676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fae8f5c60 msgr2=0x562fae96ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.706+0000 7f6231e00640 1 --2- 172.21.15.43:0/1744262676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fae8f5c60 0x562fae96ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.706+0000 7f6231e00640 1 -- 172.21.15.43:0/1744262676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fae8ece90 msgr2=0x562fae8f0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:41:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.706+0000 7f6231e00640 1 --2- 172.21.15.43:0/1744262676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fae8ece90 0x562fae8f0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.706+0000 7f6231e00640 1 -- 172.21.15.43:0/1744262676 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fae96b1e0 con 0x562fae8ec570 2024-03-28T22:41:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.706+0000 7f6232800640 1 --2- 172.21.15.43:0/1744262676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fae8f5c60 0x562fae96ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:41:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.706+0000 7f6231e00640 1 --2- 172.21.15.43:0/1744262676 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fae8ec570 0x562fae8f0050 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f622001a800 tx=0x7f622001a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=884c7235048c79ac server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:41:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.706+0000 7f620f400640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:41:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.706+0000 7f620f400640 1 -- 172.21.15.43:0/1744262676 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6220030020 con 0x562fae8ec570 2024-03-28T22:41:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.706+0000 7f620f400640 1 -- 172.21.15.43:0/1744262676 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f6220009e20 con 0x562fae8ec570 2024-03-28T22:41:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.706+0000 7f6237595740 1 -- 172.21.15.43:0/1744262676 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f621c0046c0 con 0x562fae8ec570 2024-03-28T22:41:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.707+0000 7f620f400640 1 -- 172.21.15.43:0/1744262676 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6220027070 con 0x562fae8ec570 2024-03-28T22:41:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.707+0000 7f6237595740 10 client.27422 fetch_fsmap learned FSMap version 32 2024-03-28T22:41:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.707+0000 7f6237595740 1 -- 172.21.15.43:0/1744262676 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562fae96c4f0 con 0x562fae8ec570 2024-03-28T22:41:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.707+0000 7f620f400640 1 -- 172.21.15.43:0/1744262676 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f621c0046c0 con 0x562fae8ec570 2024-03-28T22:41:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.707+0000 7f620f400640 1 -- 172.21.15.43:0/1744262676 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6220018040 con 0x562fae8ec570 2024-03-28T22:41:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.707+0000 7f6237595740 10 client.27422 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:41:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.707+0000 7f6237595740 10 client.27422 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:41:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.707+0000 7f6237595740 20 client.27422 populate_metadata read hostname 'smithi043' 2024-03-28T22:41:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.707+0000 7f6237595740 1 --2- 172.21.15.43:0/1744262676 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x562fae96cba0 0x562fae96efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:41:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.707+0000 7f6237595740 4 client.27422 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T22:41:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.707+0000 7f6237595740 1 -- 172.21.15.43:0/1744262676 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562fae85cb80 con 0x562fae96cba0 2024-03-28T22:41:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.707+0000 7f6231400640 1 --2- 172.21.15.43:0/1744262676 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x562fae96cba0 0x562fae96efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:41:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.708+0000 7f6231400640 1 --2- 172.21.15.43:0/1744262676 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x562fae96cba0 0x562fae96efc0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fee5e09ad0661567 server_cookie=39f9fbafae0e8fa5 in_seq=0 out_seq=0 2024-03-28T22:41:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.708+0000 7f620f400640 10 client.27422 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:54.896 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.899+0000 7f620f400640 1 -- 172.21.15.43:0/1744262676 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562fae85cb80 con 0x562fae96cba0 2024-03-28T22:41:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.899+0000 7f620f400640 10 client.27422 handle_command_reply: tid=0 2024-03-28T22:41:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.899+0000 7f6237595740 1 client.27422 shutdown 2024-03-28T22:41:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.899+0000 7f6237595740 1 -- 172.21.15.43:0/1744262676 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fae8ec570 msgr2=0x562fae8f0050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.899+0000 7f6237595740 1 --2- 172.21.15.43:0/1744262676 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fae8ec570 0x562fae8f0050 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f622001a800 tx=0x7f622001a5d0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.899+0000 7f6237595740 1 -- 172.21.15.43:0/1744262676 shutdown_connections 2024-03-28T22:41:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.899+0000 7f6237595740 1 --2- 172.21.15.43:0/1744262676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fae8f5c60 0x562fae96ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.899+0000 7f6237595740 1 --2- 172.21.15.43:0/1744262676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fae8ece90 0x562fae8f0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.899+0000 7f6237595740 1 --2- 172.21.15.43:0/1744262676 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x562fae96cba0 0x562fae96efc0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.899+0000 7f6237595740 1 --2- 172.21.15.43:0/1744262676 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fae8ec570 0x562fae8f0050 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.899+0000 7f6237595740 1 -- 172.21.15.43:0/1744262676 >> 172.21.15.43:0/1744262676 conn(0x562fae8e6e90 msgr2=0x562fae8e8b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.899+0000 7f620f400640 0 client.27422 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:41:54.898 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.899+0000 7f6237595740 1 -- 172.21.15.43:0/1744262676 shutdown_connections 2024-03-28T22:41:54.898 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.899+0000 7f6237595740 1 -- 172.21.15.43:0/1744262676 wait complete. 2024-03-28T22:41:54.898 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.899+0000 7f6237595740 20 client.27422 trim_cache size 0 max 16384 2024-03-28T22:41:54.898 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:{ 2024-03-28T22:41:54.898 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "message": "", 2024-03-28T22:41:54.898 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "return_code": 0 2024-03-28T22:41:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:} 2024-03-28T22:41:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.900+0000 7f6233200640 1 -- 172.21.15.43:0/3190432208 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f62140a1ee0 msgr2=0x7f62140a43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.900+0000 7f6233200640 1 --2- 172.21.15.43:0/3190432208 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f62140a1ee0 0x7f62140a43a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x562fae8e8020 tx=0x7f621c005df0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.900+0000 7f6233200640 1 -- 172.21.15.43:0/3190432208 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f622c109750 msgr2=0x7f622c19e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:41:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.900+0000 7f6233200640 1 --2- 172.21.15.43:0/3190432208 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f622c109750 0x7f622c19e8c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f6228009fd0 tx=0x7f622800ef10 comp rx=0 tx=0).stop 2024-03-28T22:41:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.900+0000 7f6233200640 1 -- 172.21.15.43:0/3190432208 shutdown_connections 2024-03-28T22:41:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.900+0000 7f6233200640 1 --2- 172.21.15.43:0/3190432208 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f62140a1ee0 0x7f62140a43a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.900+0000 7f6233200640 1 --2- 172.21.15.43:0/3190432208 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f622c109750 0x7f622c19e8c0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.900+0000 7f6233200640 1 --2- 172.21.15.43:0/3190432208 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f622c104100 0x7f622c19a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.900+0000 7f6233200640 1 --2- 172.21.15.43:0/3190432208 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f622c103740 0x7f622c19a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:41:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.900+0000 7f6233200640 1 -- 172.21.15.43:0/3190432208 >> 172.21.15.43:0/3190432208 conn(0x7f622c0ff230 msgr2=0x7f622c0ff8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:41:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.901+0000 7f6233200640 1 -- 172.21.15.43:0/3190432208 shutdown_connections 2024-03-28T22:41:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:41:54.901+0000 7f6233200640 1 -- 172.21.15.43:0/3190432208 wait complete. 2024-03-28T22:41:54.999 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:54 smithi082 ceph-mon[36471]: pgmap v146: 129 pgs: 129 active+clean; 48 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 341 op/s 2024-03-28T22:41:55.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:54 smithi043 ceph-mon[31823]: pgmap v146: 129 pgs: 129 active+clean; 48 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 341 op/s 2024-03-28T22:41:55.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:54 smithi172 ceph-mon[36566]: pgmap v146: 129 pgs: 129 active+clean; 48 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 341 op/s 2024-03-28T22:41:55.362 INFO:tasks.workunit.client.0.smithi043.stdout: 1 312609 7.98 MB/sec execute 86 sec latency 87.165 ms 2024-03-28T22:41:56.362 INFO:tasks.workunit.client.0.smithi043.stdout: 1 313847 7.95 MB/sec execute 87 sec latency 58.371 ms 2024-03-28T22:41:57.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:56 smithi043 ceph-mon[31823]: pgmap v147: 129 pgs: 129 active+clean; 48 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 164 op/s 2024-03-28T22:41:57.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:56 smithi172 ceph-mon[36566]: pgmap v147: 129 pgs: 129 active+clean; 48 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 164 op/s 2024-03-28T22:41:57.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:56 smithi082 ceph-mon[36471]: pgmap v147: 129 pgs: 129 active+clean; 48 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 164 op/s 2024-03-28T22:41:57.362 INFO:tasks.workunit.client.0.smithi043.stdout: 1 315779 8.00 MB/sec execute 88 sec latency 17.092 ms 2024-03-28T22:41:58.362 INFO:tasks.workunit.client.0.smithi043.stdout: 1 316607 7.94 MB/sec execute 89 sec latency 15.676 ms 2024-03-28T22:41:59.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:41:58 smithi043 ceph-mon[31823]: pgmap v148: 129 pgs: 129 active+clean; 44 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 328 op/s 2024-03-28T22:41:59.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:41:58 smithi172 ceph-mon[36566]: pgmap v148: 129 pgs: 129 active+clean; 44 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 328 op/s 2024-03-28T22:41:59.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:41:58 smithi082 ceph-mon[36471]: pgmap v148: 129 pgs: 129 active+clean; 44 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 328 op/s 2024-03-28T22:41:59.362 INFO:tasks.workunit.client.0.smithi043.stdout: 1 318286 7.93 MB/sec execute 90 sec latency 60.061 ms 2024-03-28T22:42:00.362 INFO:tasks.workunit.client.0.smithi043.stdout: 1 319642 7.96 MB/sec execute 91 sec latency 16.874 ms 2024-03-28T22:42:01.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:00 smithi043 ceph-mon[31823]: pgmap v149: 129 pgs: 129 active+clean; 44 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 240 op/s 2024-03-28T22:42:01.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:00 smithi172 ceph-mon[36566]: pgmap v149: 129 pgs: 129 active+clean; 44 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 240 op/s 2024-03-28T22:42:01.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:00 smithi082 ceph-mon[36471]: pgmap v149: 129 pgs: 129 active+clean; 44 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 240 op/s 2024-03-28T22:42:01.362 INFO:tasks.workunit.client.0.smithi043.stdout: 1 320263 7.89 MB/sec execute 92 sec latency 87.749 ms 2024-03-28T22:42:02.363 INFO:tasks.workunit.client.0.smithi043.stdout: 1 320675 7.85 MB/sec execute 93 sec latency 90.603 ms 2024-03-28T22:42:03.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:02 smithi043 ceph-mon[31823]: pgmap v150: 129 pgs: 129 active+clean; 44 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 240 op/s 2024-03-28T22:42:03.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:02 smithi172 ceph-mon[36566]: pgmap v150: 129 pgs: 129 active+clean; 44 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 240 op/s 2024-03-28T22:42:03.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:02 smithi082 ceph-mon[36471]: pgmap v150: 129 pgs: 129 active+clean; 44 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 240 op/s 2024-03-28T22:42:03.363 INFO:tasks.workunit.client.0.smithi043.stdout: 1 322448 7.85 MB/sec execute 94 sec latency 15.308 ms 2024-03-28T22:42:04.364 INFO:tasks.workunit.client.0.smithi043.stdout: 1 323329 7.83 MB/sec execute 95 sec latency 19.828 ms 2024-03-28T22:42:05.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:04 smithi043 ceph-mon[31823]: pgmap v151: 129 pgs: 129 active+clean; 46 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 340 op/s 2024-03-28T22:42:05.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:04 smithi172 ceph-mon[36566]: pgmap v151: 129 pgs: 129 active+clean; 46 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 340 op/s 2024-03-28T22:42:05.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:04 smithi082 ceph-mon[36471]: pgmap v151: 129 pgs: 129 active+clean; 46 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 340 op/s 2024-03-28T22:42:05.363 INFO:tasks.workunit.client.0.smithi043.stdout: 1 324433 7.80 MB/sec execute 96 sec latency 31.148 ms 2024-03-28T22:42:05.910 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:42:05.910 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:42:06.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.161+0000 7f9ed2200640 1 -- 172.21.15.43:0/1191025911 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ecc1076d0 msgr2=0x7f9ecc107ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:06.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.161+0000 7f9ed2200640 1 --2- 172.21.15.43:0/1191025911 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ecc1076d0 0x7f9ecc107ab0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc009930 tx=0x7f9ebc02bc30 comp rx=0 tx=0).stop 2024-03-28T22:42:06.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.162+0000 7f9ed2200640 1 -- 172.21.15.43:0/1191025911 shutdown_connections 2024-03-28T22:42:06.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.162+0000 7f9ed2200640 1 --2- 172.21.15.43:0/1191025911 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ecc1016c0 0x7f9ecc101b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.162+0000 7f9ed2200640 1 --2- 172.21.15.43:0/1191025911 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ecc102080 0x7f9ecc10e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.162+0000 7f9ed2200640 1 --2- 172.21.15.43:0/1191025911 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ecc1076d0 0x7f9ecc107ab0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.162+0000 7f9ed2200640 1 -- 172.21.15.43:0/1191025911 >> 172.21.15.43:0/1191025911 conn(0x7f9ecc0fd1b0 msgr2=0x7f9ecc0ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:06.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.162+0000 7f9ed2200640 1 -- 172.21.15.43:0/1191025911 shutdown_connections 2024-03-28T22:42:06.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.162+0000 7f9ed2200640 1 -- 172.21.15.43:0/1191025911 wait complete. 2024-03-28T22:42:06.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.163+0000 7f9ed2200640 1 Processor -- start 2024-03-28T22:42:06.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.163+0000 7f9ed2200640 1 -- start start 2024-03-28T22:42:06.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.163+0000 7f9ed2200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ecc1016c0 0x7f9ecc19a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.163+0000 7f9ed2200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ecc102080 0x7f9ecc19a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.163+0000 7f9ed2200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ecc1076d0 0x7f9ecc19eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.164+0000 7f9ed2200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc111b30 con 0x7f9ecc102080 2024-03-28T22:42:06.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.164+0000 7f9ed2200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc1119b0 con 0x7f9ecc1016c0 2024-03-28T22:42:06.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.164+0000 7f9ed2200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc111cb0 con 0x7f9ecc1076d0 2024-03-28T22:42:06.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.164+0000 7f9ecb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ecc1016c0 0x7f9ecc19a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.164+0000 7f9ecbe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ecc1076d0 0x7f9ecc19eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.164+0000 7f9ecb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ecc1016c0 0x7f9ecc19a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44378/0 (socket says 172.21.15.43:44378) 2024-03-28T22:42:06.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.164+0000 7f9ecb400640 1 -- 172.21.15.43:0/200099758 learned_addr learned my addr 172.21.15.43:0/200099758 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:06.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.164+0000 7f9ecbe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ecc1076d0 0x7f9ecc19eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:44170/0 (socket says 172.21.15.43:44170) 2024-03-28T22:42:06.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.164+0000 7f9ecaa00640 1 --2- 172.21.15.43:0/200099758 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ecc102080 0x7f9ecc19a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.164+0000 7f9ecbe00640 1 -- 172.21.15.43:0/200099758 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ecc1016c0 msgr2=0x7f9ecc19a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:06.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.164+0000 7f9ecbe00640 1 --2- 172.21.15.43:0/200099758 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ecc1016c0 0x7f9ecc19a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.164+0000 7f9ecbe00640 1 -- 172.21.15.43:0/200099758 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ecc102080 msgr2=0x7f9ecc19a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:06.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.164+0000 7f9ecbe00640 1 --2- 172.21.15.43:0/200099758 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ecc102080 0x7f9ecc19a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.164+0000 7f9ecbe00640 1 -- 172.21.15.43:0/200099758 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ecc19f280 con 0x7f9ecc1076d0 2024-03-28T22:42:06.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.164+0000 7f9ecaa00640 1 --2- 172.21.15.43:0/200099758 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ecc102080 0x7f9ecc19a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:42:06.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.164+0000 7f9ecb400640 1 --2- 172.21.15.43:0/200099758 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ecc1016c0 0x7f9ecc19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:06.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.165+0000 7f9ecbe00640 1 --2- 172.21.15.43:0/200099758 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ecc1076d0 0x7f9ecc19eb50 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4009fd0 tx=0x7f9ec400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:06.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.165+0000 7f9ec3e00640 1 -- 172.21.15.43:0/200099758 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec400ce20 con 0x7f9ecc1076d0 2024-03-28T22:42:06.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.165+0000 7f9ec3e00640 1 -- 172.21.15.43:0/200099758 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ec4005020 con 0x7f9ecc1076d0 2024-03-28T22:42:06.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.165+0000 7f9ed2200640 1 -- 172.21.15.43:0/200099758 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ecc19f570 con 0x7f9ecc1076d0 2024-03-28T22:42:06.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.165+0000 7f9ec3e00640 1 -- 172.21.15.43:0/200099758 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec4010720 con 0x7f9ecc1076d0 2024-03-28T22:42:06.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.165+0000 7f9ed2200640 1 -- 172.21.15.43:0/200099758 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ecc1a6e00 con 0x7f9ecc1076d0 2024-03-28T22:42:06.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.169+0000 7f9ec3e00640 1 -- 172.21.15.43:0/200099758 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f9ec4005540 con 0x7f9ecc1076d0 2024-03-28T22:42:06.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.169+0000 7f9ed4e84740 1 Processor -- start 2024-03-28T22:42:06.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.169+0000 7f9ed4e84740 1 -- start start 2024-03-28T22:42:06.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.169+0000 7f9ed4e84740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640d4122bc0 0x5640d4121910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.169+0000 7f9ed4e84740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640d4121ee0 0x5640d4122360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.169+0000 7f9ed4e84740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640d412b250 0x5640d412d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.169+0000 7f9ed4e84740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5640d4086100 con 0x5640d4122bc0 2024-03-28T22:42:06.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.169+0000 7f9ed4e84740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5640d402eba0 con 0x5640d4121ee0 2024-03-28T22:42:06.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.169+0000 7f9ed4e84740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5640d40821a0 con 0x5640d412b250 2024-03-28T22:42:06.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.170+0000 7f9ecb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640d4121ee0 0x5640d4122360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.170+0000 7f9ecb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640d4121ee0 0x5640d4122360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44386/0 (socket says 172.21.15.43:44386) 2024-03-28T22:42:06.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.170+0000 7f9ecb400640 1 -- 172.21.15.43:0/1198941274 learned_addr learned my addr 172.21.15.43:0/1198941274 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:06.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.170+0000 7f9ecaa00640 1 --2- 172.21.15.43:0/1198941274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640d4122bc0 0x5640d4121910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.170+0000 7f9ecaa00640 1 --2- 172.21.15.43:0/1198941274 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640d412b250 0x5640d412d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.170+0000 7f9ecb400640 1 -- 172.21.15.43:0/1198941274 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640d412b250 msgr2=0x5640d412d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:06.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.170+0000 7f9ecb400640 1 --2- 172.21.15.43:0/1198941274 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640d412b250 0x5640d412d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.170+0000 7f9ecb400640 1 -- 172.21.15.43:0/1198941274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640d4122bc0 msgr2=0x5640d4121910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:06.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.170+0000 7f9ecb400640 1 --2- 172.21.15.43:0/1198941274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640d4122bc0 0x5640d4121910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.170+0000 7f9ecb400640 1 -- 172.21.15.43:0/1198941274 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640d405f1a0 con 0x5640d4121ee0 2024-03-28T22:42:06.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.170+0000 7f9ecaa00640 1 --2- 172.21.15.43:0/1198941274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640d4122bc0 0x5640d4121910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:42:06.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.170+0000 7f9ec3e00640 1 --2- 172.21.15.43:0/200099758 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9eb00a1f70 0x7f9eb00a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.171+0000 7f9ecb400640 1 --2- 172.21.15.43:0/1198941274 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640d4121ee0 0x5640d4122360 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc005ec0 tx=0x7f9ebc009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6218d3950fb0dba server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:06.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.171+0000 7f9ec3e00640 1 -- 172.21.15.43:0/200099758 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f9ec40ceb30 con 0x7f9ecc1076d0 2024-03-28T22:42:06.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.171+0000 7f9ecaa00640 1 --2- 172.21.15.43:0/200099758 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9eb00a1f70 0x7f9eb00a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.171+0000 7f9eb7e00640 1 -- 172.21.15.43:0/1198941274 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc045070 con 0x5640d4121ee0 2024-03-28T22:42:06.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.171+0000 7f9eb7e00640 1 -- 172.21.15.43:0/1198941274 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ebc004a70 con 0x5640d4121ee0 2024-03-28T22:42:06.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.171+0000 7f9eb7e00640 1 -- 172.21.15.43:0/1198941274 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc005180 con 0x5640d4121ee0 2024-03-28T22:42:06.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.172+0000 7f9ed4e84740 1 -- 172.21.15.43:0/1198941274 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640d4121ee0 msgr2=0x5640d4122360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:06.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.172+0000 7f9ecaa00640 1 --2- 172.21.15.43:0/200099758 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9eb00a1f70 0x7f9eb00a4430 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc0759e0 tx=0x7f9eb80096c0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:06.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.172+0000 7f9ed4e84740 1 --2- 172.21.15.43:0/1198941274 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640d4121ee0 0x5640d4122360 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc005ec0 tx=0x7f9ebc009560 comp rx=0 tx=0).stop 2024-03-28T22:42:06.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.172+0000 7f9ed4e84740 1 -- 172.21.15.43:0/1198941274 shutdown_connections 2024-03-28T22:42:06.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.172+0000 7f9ed4e84740 1 --2- 172.21.15.43:0/1198941274 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640d4121ee0 0x5640d4122360 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.172+0000 7f9ed4e84740 1 --2- 172.21.15.43:0/1198941274 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640d412b250 0x5640d412d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.172+0000 7f9ed4e84740 1 --2- 172.21.15.43:0/1198941274 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640d4122bc0 0x5640d4121910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.172+0000 7f9ed4e84740 1 -- 172.21.15.43:0/1198941274 >> 172.21.15.43:0/1198941274 conn(0x5640d40e8a10 msgr2=0x5640d411c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:06.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.172+0000 7f9ed4e84740 1 -- 172.21.15.43:0/1198941274 shutdown_connections 2024-03-28T22:42:06.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.172+0000 7f9ed4e84740 1 -- 172.21.15.43:0/1198941274 wait complete. 2024-03-28T22:42:06.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.173+0000 7f9ed4e84740 1 -- start start 2024-03-28T22:42:06.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.173+0000 7f9ed4e84740 1 Processor -- start 2024-03-28T22:42:06.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.173+0000 7f9ed4e84740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640d4121ee0 0x5640d41281f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.173+0000 7f9ed4e84740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640d4122bc0 0x5640d4128730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.173+0000 7f9ed4e84740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640d412b250 0x5640d4128c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.173+0000 7f9ed4e84740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5640d412e060 con 0x5640d4121ee0 2024-03-28T22:42:06.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.173+0000 7f9ed4e84740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5640d4024250 con 0x5640d412b250 2024-03-28T22:42:06.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.173+0000 7f9ed4e84740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5640d412e1e0 con 0x5640d4122bc0 2024-03-28T22:42:06.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.173+0000 7f9ed4e84740 10 client.0 _ll_register_callbacks cb 0x5640d3d99be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:06.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.174+0000 7f9ecb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640d4121ee0 0x5640d41281f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.174+0000 7f9ecaa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640d4122bc0 0x5640d4128730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.174+0000 7f9ecaa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640d4122bc0 0x5640d4128730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:44194/0 (socket says 172.21.15.43:44194) 2024-03-28T22:42:06.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.174+0000 7f9ecaa00640 1 -- 172.21.15.43:0/2410513831 learned_addr learned my addr 172.21.15.43:0/2410513831 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:06.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.174+0000 7f9ecbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640d412b250 0x5640d4128c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.174+0000 7f9ecaa00640 1 -- 172.21.15.43:0/2410513831 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640d412b250 msgr2=0x5640d4128c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:06.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.174+0000 7f9ecaa00640 1 --2- 172.21.15.43:0/2410513831 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640d412b250 0x5640d4128c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.174+0000 7f9ecaa00640 1 -- 172.21.15.43:0/2410513831 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640d4121ee0 msgr2=0x5640d41281f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:06.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.174+0000 7f9ecaa00640 1 --2- 172.21.15.43:0/2410513831 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640d4121ee0 0x5640d41281f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.174+0000 7f9ecaa00640 1 -- 172.21.15.43:0/2410513831 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640d4040960 con 0x5640d4122bc0 2024-03-28T22:42:06.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.174+0000 7f9ecaa00640 1 --2- 172.21.15.43:0/2410513831 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640d4122bc0 0x5640d4128730 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f9eb80108c0 tx=0x7f9eb8004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2dc6d1bef8ce1077 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:06.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.174+0000 7f9eb4c00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:42:06.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.175+0000 7f9eb4c00640 1 -- 172.21.15.43:0/2410513831 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eb8019030 con 0x5640d4122bc0 2024-03-28T22:42:06.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.175+0000 7f9eb4c00640 1 -- 172.21.15.43:0/2410513831 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f9eb800fb50 con 0x5640d4122bc0 2024-03-28T22:42:06.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.175+0000 7f9ed4e84740 1 -- 172.21.15.43:0/2410513831 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ebc004a70 con 0x5640d4122bc0 2024-03-28T22:42:06.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.175+0000 7f9eb4c00640 1 -- 172.21.15.43:0/2410513831 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9eb800fd90 con 0x5640d4122bc0 2024-03-28T22:42:06.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.175+0000 7f9ed4e84740 10 client.27382 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:06.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.175+0000 7f9ed4e84740 1 -- 172.21.15.43:0/2410513831 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5640d3e647d0 con 0x5640d4122bc0 2024-03-28T22:42:06.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.175+0000 7f9eb4c00640 1 -- 172.21.15.43:0/2410513831 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ebc004a70 con 0x5640d4122bc0 2024-03-28T22:42:06.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.176+0000 7f9eb4c00640 1 -- 172.21.15.43:0/2410513831 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f9eb8004c00 con 0x5640d4122bc0 2024-03-28T22:42:06.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.176+0000 7f9ed4e84740 10 client.27382 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:06.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.176+0000 7f9ed4e84740 10 client.27382 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:42:06.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.176+0000 7f9ed4e84740 20 client.27382 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:06.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.176+0000 7f9ed4e84740 1 --2- 172.21.15.43:0/2410513831 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5640d413b6c0 0x5640d413db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.176+0000 7f9ed4e84740 4 client.27382 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:42:06.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.176+0000 7f9ed4e84740 1 -- 172.21.15.43:0/2410513831 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640d4093640 con 0x5640d413b6c0 2024-03-28T22:42:06.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.176+0000 7f9ecb400640 1 --2- 172.21.15.43:0/2410513831 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5640d413b6c0 0x5640d413db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.177+0000 7f9ecb400640 1 --2- 172.21.15.43:0/2410513831 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5640d413b6c0 0x5640d413db80 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5dc8462097cb1fe server_cookie=b3a1a963a66ba6b5 in_seq=0 out_seq=0 2024-03-28T22:42:06.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.177+0000 7f9eb4c00640 10 client.27382 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:06.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.178+0000 7f9eb4c00640 1 -- 172.21.15.43:0/2410513831 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5640d4093640 con 0x5640d413b6c0 2024-03-28T22:42:06.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.178+0000 7f9eb4c00640 10 client.27382 handle_command_reply: tid=0 2024-03-28T22:42:06.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.178+0000 7f9ed4e84740 1 client.27382 shutdown 2024-03-28T22:42:06.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.179+0000 7f9ed4e84740 1 -- 172.21.15.43:0/2410513831 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640d4122bc0 msgr2=0x5640d4128730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:06.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.179+0000 7f9ed4e84740 1 --2- 172.21.15.43:0/2410513831 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640d4122bc0 0x5640d4128730 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f9eb80108c0 tx=0x7f9eb8004730 comp rx=0 tx=0).stop 2024-03-28T22:42:06.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.179+0000 7f9ed4e84740 1 -- 172.21.15.43:0/2410513831 shutdown_connections 2024-03-28T22:42:06.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.179+0000 7f9ed4e84740 1 --2- 172.21.15.43:0/2410513831 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5640d413b6c0 0x5640d413db80 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.179+0000 7f9ed4e84740 1 --2- 172.21.15.43:0/2410513831 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640d412b250 0x5640d4128c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.179+0000 7f9ed4e84740 1 --2- 172.21.15.43:0/2410513831 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640d4122bc0 0x5640d4128730 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.179+0000 7f9ed4e84740 1 --2- 172.21.15.43:0/2410513831 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640d4121ee0 0x5640d41281f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.179+0000 7f9ed4e84740 1 -- 172.21.15.43:0/2410513831 >> 172.21.15.43:0/2410513831 conn(0x5640d40e8a10 msgr2=0x5640d411e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:06.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.179+0000 7f9eb4c00640 0 client.27382 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:06.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.179+0000 7f9ed4e84740 1 -- 172.21.15.43:0/2410513831 shutdown_connections 2024-03-28T22:42:06.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.179+0000 7f9ed4e84740 1 -- 172.21.15.43:0/2410513831 wait complete. 2024-03-28T22:42:06.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.179+0000 7f9ed4e84740 20 client.27382 trim_cache size 0 max 16384 2024-03-28T22:42:06.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.199+0000 7f9ed4e84740 1 Processor -- start 2024-03-28T22:42:06.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.199+0000 7f9ed4e84740 1 -- start start 2024-03-28T22:42:06.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.200+0000 7f9ed4e84740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640d4121ee0 0x5640d412a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.200+0000 7f9ed4e84740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640d4122bc0 0x5640d411cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.200+0000 7f9ed4e84740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640d412b250 0x5640d4136e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.200+0000 7f9ed4e84740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc1b0e10 con 0x5640d4122bc0 2024-03-28T22:42:06.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.200+0000 7f9ed4e84740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc1b2950 con 0x5640d4121ee0 2024-03-28T22:42:06.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.200+0000 7f9ed4e84740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc1b14e0 con 0x5640d412b250 2024-03-28T22:42:06.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.200+0000 7f9ecb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640d4121ee0 0x5640d412a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.200+0000 7f9ecaa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640d4122bc0 0x5640d411cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.200+0000 7f9ecb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640d4121ee0 0x5640d412a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44402/0 (socket says 172.21.15.43:44402) 2024-03-28T22:42:06.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.200+0000 7f9ecb400640 1 -- 172.21.15.43:0/799477363 learned_addr learned my addr 172.21.15.43:0/799477363 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:06.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.200+0000 7f9ecbe00640 1 --2- 172.21.15.43:0/799477363 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640d412b250 0x5640d4136e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.200+0000 7f9ecb400640 1 -- 172.21.15.43:0/799477363 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640d412b250 msgr2=0x5640d4136e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:06.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.200+0000 7f9ecb400640 1 --2- 172.21.15.43:0/799477363 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640d412b250 0x5640d4136e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.200+0000 7f9ecb400640 1 -- 172.21.15.43:0/799477363 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640d4122bc0 msgr2=0x5640d411cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:06.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.200+0000 7f9ecb400640 1 --2- 172.21.15.43:0/799477363 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640d4122bc0 0x5640d411cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.200+0000 7f9ecb400640 1 -- 172.21.15.43:0/799477363 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640d3d9a3f0 con 0x5640d4121ee0 2024-03-28T22:42:06.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.200+0000 7f9ecbe00640 1 --2- 172.21.15.43:0/799477363 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640d412b250 0x5640d4136e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:06.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.200+0000 7f9ecaa00640 1 --2- 172.21.15.43:0/799477363 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640d4122bc0 0x5640d411cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:06.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.201+0000 7f9ecb400640 1 --2- 172.21.15.43:0/799477363 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640d4121ee0 0x5640d412a920 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc000c00 tx=0x7f9ebc02cb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be9adb3ed6f76dea server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:06.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.201+0000 7f9eb7400640 1 -- 172.21.15.43:0/799477363 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc045040 con 0x5640d4121ee0 2024-03-28T22:42:06.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.201+0000 7f9eb7400640 1 -- 172.21.15.43:0/799477363 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ebc050070 con 0x5640d4121ee0 2024-03-28T22:42:06.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.201+0000 7f9eb7400640 1 -- 172.21.15.43:0/799477363 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc005230 con 0x5640d4121ee0 2024-03-28T22:42:06.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.202+0000 7f9ed4e84740 1 -- 172.21.15.43:0/799477363 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640d4121ee0 msgr2=0x5640d412a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:06.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.202+0000 7f9ed4e84740 1 --2- 172.21.15.43:0/799477363 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640d4121ee0 0x5640d412a920 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc000c00 tx=0x7f9ebc02cb70 comp rx=0 tx=0).stop 2024-03-28T22:42:06.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.202+0000 7f9ed4e84740 1 -- 172.21.15.43:0/799477363 shutdown_connections 2024-03-28T22:42:06.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.202+0000 7f9ed4e84740 1 --2- 172.21.15.43:0/799477363 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640d412b250 0x5640d4136e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.202+0000 7f9ed4e84740 1 --2- 172.21.15.43:0/799477363 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640d4122bc0 0x5640d411cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.202+0000 7f9ed4e84740 1 --2- 172.21.15.43:0/799477363 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640d4121ee0 0x5640d412a920 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.202+0000 7f9ed4e84740 1 -- 172.21.15.43:0/799477363 >> 172.21.15.43:0/799477363 conn(0x5640d40e8a10 msgr2=0x5640d411dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:06.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.202+0000 7f9ed4e84740 1 -- 172.21.15.43:0/799477363 shutdown_connections 2024-03-28T22:42:06.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.202+0000 7f9ed4e84740 1 -- 172.21.15.43:0/799477363 wait complete. 2024-03-28T22:42:06.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.202+0000 7f9ed4e84740 1 -- start start 2024-03-28T22:42:06.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.203+0000 7f9ed4e84740 1 Processor -- start 2024-03-28T22:42:06.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.203+0000 7f9ed4e84740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640d4121ee0 0x5640d4133740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.203+0000 7f9ed4e84740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640d4122bc0 0x5640d4125470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.203+0000 7f9ed4e84740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640d412b250 0x5640d41259b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.203+0000 7f9ed4e84740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc1b2280 con 0x5640d4121ee0 2024-03-28T22:42:06.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.203+0000 7f9ed4e84740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc1b1bb0 con 0x5640d4122bc0 2024-03-28T22:42:06.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.203+0000 7f9ed4e84740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc1aedf0 con 0x5640d412b250 2024-03-28T22:42:06.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.203+0000 7f9ed4e84740 10 client.0 _ll_register_callbacks cb 0x7f9eb8009c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:06.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.203+0000 7f9ecaa00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640d4122bc0 0x5640d4125470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.203+0000 7f9ecaa00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640d4122bc0 0x5640d4125470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44406/0 (socket says 172.21.15.43:44406) 2024-03-28T22:42:06.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.203+0000 7f9ecaa00640 1 -- 172.21.15.43:0/1887157151 learned_addr learned my addr 172.21.15.43:0/1887157151 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:06.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.203+0000 7f9ecbe00640 1 --2- 172.21.15.43:0/1887157151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640d412b250 0x5640d41259b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.203+0000 7f9ecb400640 1 --2- 172.21.15.43:0/1887157151 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640d4121ee0 0x5640d4133740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.204+0000 7f9ecaa00640 1 -- 172.21.15.43:0/1887157151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640d412b250 msgr2=0x5640d41259b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:06.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.204+0000 7f9ecaa00640 1 --2- 172.21.15.43:0/1887157151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640d412b250 0x5640d41259b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.204+0000 7f9ecaa00640 1 -- 172.21.15.43:0/1887157151 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640d4121ee0 msgr2=0x5640d4133740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:06.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.204+0000 7f9ecaa00640 1 --2- 172.21.15.43:0/1887157151 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640d4121ee0 0x5640d4133740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.204+0000 7f9ecaa00640 1 -- 172.21.15.43:0/1887157151 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640d3d5a0f0 con 0x5640d4122bc0 2024-03-28T22:42:06.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.204+0000 7f9ecbe00640 1 --2- 172.21.15.43:0/1887157151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640d412b250 0x5640d41259b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:06.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.204+0000 7f9ecaa00640 1 --2- 172.21.15.43:0/1887157151 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640d4122bc0 0x5640d4125470 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f9eb8022540 tx=0x7f9eb8010ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3e3b27f042c6204 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:06.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.204+0000 7f9eb4c00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:42:06.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.204+0000 7f9eb4c00640 1 -- 172.21.15.43:0/1887157151 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eb8010430 con 0x5640d4122bc0 2024-03-28T22:42:06.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.204+0000 7f9ed4e84740 1 -- 172.21.15.43:0/1887157151 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ebc050070 con 0x5640d4122bc0 2024-03-28T22:42:06.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.204+0000 7f9eb4c00640 1 -- 172.21.15.43:0/1887157151 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f9eb8005000 con 0x5640d4122bc0 2024-03-28T22:42:06.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.205+0000 7f9eb4c00640 1 -- 172.21.15.43:0/1887157151 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9eb8009ce0 con 0x5640d4122bc0 2024-03-28T22:42:06.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.205+0000 7f9ed4e84740 10 client.27440 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:06.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.205+0000 7f9ed4e84740 1 -- 172.21.15.43:0/1887157151 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5640d412c840 con 0x5640d4122bc0 2024-03-28T22:42:06.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.205+0000 7f9eb4c00640 1 -- 172.21.15.43:0/1887157151 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ebc050070 con 0x5640d4122bc0 2024-03-28T22:42:06.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.205+0000 7f9eb4c00640 1 -- 172.21.15.43:0/1887157151 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f9eb8037430 con 0x5640d4122bc0 2024-03-28T22:42:06.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.205+0000 7f9ed4e84740 10 client.27440 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:06.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.205+0000 7f9ed4e84740 10 client.27440 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:42:06.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.205+0000 7f9ed4e84740 20 client.27440 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:06.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.205+0000 7f9ed4e84740 1 --2- 172.21.15.43:0/1887157151 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5640d41a1f80 0x5640d41a43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.205+0000 7f9ed4e84740 4 client.27440 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:42:06.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.205+0000 7f9ed4e84740 1 -- 172.21.15.43:0/1887157151 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5640d408d990 con 0x5640d41a1f80 2024-03-28T22:42:06.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.206+0000 7f9ecb400640 1 --2- 172.21.15.43:0/1887157151 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5640d41a1f80 0x5640d41a43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.206+0000 7f9ecb400640 1 --2- 172.21.15.43:0/1887157151 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5640d41a1f80 0x5640d41a43a0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9006f53931ba9c6 server_cookie=d291a57565c85cf2 in_seq=0 out_seq=0 2024-03-28T22:42:06.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.206+0000 7f9eb4c00640 10 client.27440 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:06.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.206+0000 7f9eb4c00640 1 -- 172.21.15.43:0/1887157151 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5640d408d990 con 0x5640d41a1f80 2024-03-28T22:42:06.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.206+0000 7f9eb4c00640 10 client.27440 handle_command_reply: tid=0 2024-03-28T22:42:06.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.206+0000 7f9ed4e84740 1 client.27440 shutdown 2024-03-28T22:42:06.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.206+0000 7f9ed4e84740 1 -- 172.21.15.43:0/1887157151 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640d4122bc0 msgr2=0x5640d4125470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:06.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.206+0000 7f9ed4e84740 1 --2- 172.21.15.43:0/1887157151 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640d4122bc0 0x5640d4125470 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f9eb8022540 tx=0x7f9eb8010ea0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.206+0000 7f9ed4e84740 1 -- 172.21.15.43:0/1887157151 shutdown_connections 2024-03-28T22:42:06.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.206+0000 7f9ed4e84740 1 --2- 172.21.15.43:0/1887157151 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5640d41a1f80 0x5640d41a43a0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.206+0000 7f9ed4e84740 1 --2- 172.21.15.43:0/1887157151 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640d4122bc0 0x5640d4125470 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.206+0000 7f9ed4e84740 1 --2- 172.21.15.43:0/1887157151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640d412b250 0x5640d41259b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.206+0000 7f9ed4e84740 1 --2- 172.21.15.43:0/1887157151 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640d4121ee0 0x5640d4133740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.206+0000 7f9eb4c00640 0 client.27440 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:06.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.206+0000 7f9ed4e84740 1 -- 172.21.15.43:0/1887157151 >> 172.21.15.43:0/1887157151 conn(0x5640d40e8a10 msgr2=0x5640d4128a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:06.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.207+0000 7f9ed4e84740 1 -- 172.21.15.43:0/1887157151 shutdown_connections 2024-03-28T22:42:06.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.207+0000 7f9ed4e84740 1 -- 172.21.15.43:0/1887157151 wait complete. 2024-03-28T22:42:06.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.207+0000 7f9ed4e84740 20 client.27440 trim_cache size 0 max 16384 2024-03-28T22:42:06.210 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:42:06.210 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-03-28T22:42:06.210 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-03-28T22:42:06.211 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:42:06.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.207+0000 7f9ed2200640 1 -- 172.21.15.43:0/200099758 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9eb00a1f70 msgr2=0x7f9eb00a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:06.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.207+0000 7f9ed2200640 1 --2- 172.21.15.43:0/200099758 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9eb00a1f70 0x7f9eb00a4430 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc0759e0 tx=0x7f9eb80096c0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.208+0000 7f9ed2200640 1 -- 172.21.15.43:0/200099758 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ecc1076d0 msgr2=0x7f9ecc19eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:06.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.208+0000 7f9ed2200640 1 --2- 172.21.15.43:0/200099758 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ecc1076d0 0x7f9ecc19eb50 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4009fd0 tx=0x7f9ec400ef10 comp rx=0 tx=0).stop 2024-03-28T22:42:06.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.208+0000 7f9ed2200640 1 -- 172.21.15.43:0/200099758 shutdown_connections 2024-03-28T22:42:06.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.208+0000 7f9ed2200640 1 --2- 172.21.15.43:0/200099758 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9eb00a1f70 0x7f9eb00a4430 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.208+0000 7f9ed2200640 1 --2- 172.21.15.43:0/200099758 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ecc1076d0 0x7f9ecc19eb50 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.208+0000 7f9ed2200640 1 --2- 172.21.15.43:0/200099758 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ecc102080 0x7f9ecc19a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.208+0000 7f9ed2200640 1 --2- 172.21.15.43:0/200099758 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ecc1016c0 0x7f9ecc19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.208+0000 7f9ed2200640 1 -- 172.21.15.43:0/200099758 >> 172.21.15.43:0/200099758 conn(0x7f9ecc0fd1b0 msgr2=0x7f9ecc10cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:06.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.208+0000 7f9ed2200640 1 -- 172.21.15.43:0/200099758 shutdown_connections 2024-03-28T22:42:06.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.208+0000 7f9ed2200640 1 -- 172.21.15.43:0/200099758 wait complete. 2024-03-28T22:42:06.224 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:42:06.225 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:42:06.364 INFO:tasks.workunit.client.0.smithi043.stdout: 1 326453 7.85 MB/sec execute 97 sec latency 12.981 ms 2024-03-28T22:42:06.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.453+0000 7fad16e00640 1 -- 172.21.15.43:0/2487136449 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad101075f0 msgr2=0x7fad101079d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:06.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.453+0000 7fad16e00640 1 --2- 172.21.15.43:0/2487136449 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad101075f0 0x7fad101079d0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fad04009cd0 tx=0x7fad0402c0f0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.453+0000 7fad16e00640 1 -- 172.21.15.43:0/2487136449 shutdown_connections 2024-03-28T22:42:06.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.453+0000 7fad16e00640 1 --2- 172.21.15.43:0/2487136449 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fad10101530 0x7fad101019b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.453+0000 7fad16e00640 1 --2- 172.21.15.43:0/2487136449 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad10101ef0 0x7fad1010e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.453+0000 7fad16e00640 1 --2- 172.21.15.43:0/2487136449 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad101075f0 0x7fad101079d0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.453+0000 7fad16e00640 1 -- 172.21.15.43:0/2487136449 >> 172.21.15.43:0/2487136449 conn(0x7fad100fd020 msgr2=0x7fad100ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:06.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.453+0000 7fad16e00640 1 -- 172.21.15.43:0/2487136449 shutdown_connections 2024-03-28T22:42:06.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.453+0000 7fad16e00640 1 -- 172.21.15.43:0/2487136449 wait complete. 2024-03-28T22:42:06.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.454+0000 7fad16e00640 1 Processor -- start 2024-03-28T22:42:06.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.455+0000 7fad16e00640 1 -- start start 2024-03-28T22:42:06.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.455+0000 7fad16e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fad10101530 0x7fad10199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.455+0000 7fad16e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad10101ef0 0x7fad1019a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.455+0000 7fad16e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad101075f0 0x7fad1019e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.455+0000 7fad16e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fad101118b0 con 0x7fad10101ef0 2024-03-28T22:42:06.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.455+0000 7fad16e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fad10111730 con 0x7fad10101530 2024-03-28T22:42:06.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.455+0000 7fad16e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fad10111a30 con 0x7fad101075f0 2024-03-28T22:42:06.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.455+0000 7fad15a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fad10101530 0x7fad10199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.455+0000 7fad15a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fad10101530 0x7fad10199fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44418/0 (socket says 172.21.15.43:44418) 2024-03-28T22:42:06.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.455+0000 7fad15a00640 1 -- 172.21.15.43:0/2815740441 learned_addr learned my addr 172.21.15.43:0/2815740441 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:06.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.455+0000 7fad15000640 1 --2- 172.21.15.43:0/2815740441 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad10101ef0 0x7fad1019a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.455+0000 7fad16400640 1 --2- 172.21.15.43:0/2815740441 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad101075f0 0x7fad1019e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.456+0000 7fad15a00640 1 -- 172.21.15.43:0/2815740441 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad101075f0 msgr2=0x7fad1019e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:06.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.456+0000 7fad15a00640 1 --2- 172.21.15.43:0/2815740441 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad101075f0 0x7fad1019e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.456+0000 7fad15a00640 1 -- 172.21.15.43:0/2815740441 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad10101ef0 msgr2=0x7fad1019a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:06.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.456+0000 7fad15a00640 1 --2- 172.21.15.43:0/2815740441 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad10101ef0 0x7fad1019a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.456+0000 7fad15a00640 1 -- 172.21.15.43:0/2815740441 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad1019ef90 con 0x7fad10101530 2024-03-28T22:42:06.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.456+0000 7fad15000640 1 --2- 172.21.15.43:0/2815740441 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad10101ef0 0x7fad1019a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:42:06.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.456+0000 7fad15a00640 1 --2- 172.21.15.43:0/2815740441 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fad10101530 0x7fad10199fa0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fad04004af0 tx=0x7fad04032ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:06.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.456+0000 7fad0e000640 1 -- 172.21.15.43:0/2815740441 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad04041020 con 0x7fad10101530 2024-03-28T22:42:06.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.456+0000 7fad0e000640 1 -- 172.21.15.43:0/2815740441 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad04034cd0 con 0x7fad10101530 2024-03-28T22:42:06.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.456+0000 7fad16e00640 1 -- 172.21.15.43:0/2815740441 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad1019f220 con 0x7fad10101530 2024-03-28T22:42:06.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.457+0000 7fad0e000640 1 -- 172.21.15.43:0/2815740441 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad04039040 con 0x7fad10101530 2024-03-28T22:42:06.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.457+0000 7fad16e00640 1 -- 172.21.15.43:0/2815740441 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad101a6b10 con 0x7fad10101530 2024-03-28T22:42:06.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.458+0000 7fad16e00640 1 -- 172.21.15.43:0/2815740441 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad100688b0 con 0x7fad10101530 2024-03-28T22:42:06.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.461+0000 7fad0e000640 1 -- 172.21.15.43:0/2815740441 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fad04044020 con 0x7fad10101530 2024-03-28T22:42:06.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.462+0000 7fad0e000640 1 --2- 172.21.15.43:0/2815740441 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7facf80a1f70 0x7facf80a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.462+0000 7fad0e000640 1 -- 172.21.15.43:0/2815740441 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fad040effe0 con 0x7fad10101530 2024-03-28T22:42:06.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.462+0000 7fad15000640 1 --2- 172.21.15.43:0/2815740441 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7facf80a1f70 0x7facf80a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.463+0000 7fad15000640 1 --2- 172.21.15.43:0/2815740441 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7facf80a1f70 0x7facf80a4430 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fad1010e090 tx=0x7facfc005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:06.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.466+0000 7fad0e000640 1 -- 172.21.15.43:0/2815740441 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fad0404b020 con 0x7fad10101530 2024-03-28T22:42:06.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.738+0000 7fad16e00640 1 -- 172.21.15.43:0/2815740441 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad101a6ce0 con 0x7fad10101530 2024-03-28T22:42:06.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.739+0000 7fad0e000640 1 -- 172.21.15.43:0/2815740441 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11931 (secure 0 0 0) 0x7fad040bce40 con 0x7fad10101530 2024-03-28T22:42:06.738 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:42:06.738 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:33:28.906396+0000","modified":"2024-03-28T22:40:08.672168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24572,"mds_1":24556,"mds_2":24535,"mds_3":24523,"mds_4":24518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26717":{"gid":26717,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/215779342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":215779342},{"type":"v1","addr":"172.21.15.82:6835","nonce":215779342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24535,24556,24518,24523,24572]},"id":1}]} 2024-03-28T22:42:06.738 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 32 2024-03-28T22:42:06.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.742+0000 7fad16e00640 1 -- 172.21.15.43:0/2815740441 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7facf80a1f70 msgr2=0x7facf80a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:06.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.742+0000 7fad16e00640 1 --2- 172.21.15.43:0/2815740441 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7facf80a1f70 0x7facf80a4430 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fad1010e090 tx=0x7facfc005f90 comp rx=0 tx=0).stop 2024-03-28T22:42:06.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.742+0000 7fad16e00640 1 -- 172.21.15.43:0/2815740441 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fad10101530 msgr2=0x7fad10199fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:06.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.742+0000 7fad16e00640 1 --2- 172.21.15.43:0/2815740441 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fad10101530 0x7fad10199fa0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fad04004af0 tx=0x7fad04032ea0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.742+0000 7fad16e00640 1 -- 172.21.15.43:0/2815740441 shutdown_connections 2024-03-28T22:42:06.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.742+0000 7fad16e00640 1 --2- 172.21.15.43:0/2815740441 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7facf80a1f70 0x7facf80a4430 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.742+0000 7fad16e00640 1 --2- 172.21.15.43:0/2815740441 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad101075f0 0x7fad1019e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.742+0000 7fad16e00640 1 --2- 172.21.15.43:0/2815740441 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fad10101ef0 0x7fad1019a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.742+0000 7fad16e00640 1 --2- 172.21.15.43:0/2815740441 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fad10101530 0x7fad10199fa0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.742+0000 7fad16e00640 1 -- 172.21.15.43:0/2815740441 >> 172.21.15.43:0/2815740441 conn(0x7fad100fd020 msgr2=0x7fad100fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:06.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.742+0000 7fad16e00640 1 -- 172.21.15.43:0/2815740441 shutdown_connections 2024-03-28T22:42:06.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.742+0000 7fad16e00640 1 -- 172.21.15.43:0/2815740441 wait complete. 2024-03-28T22:42:06.761 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T22:42:06.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.980+0000 7f7326000640 1 -- 172.21.15.43:0/4130276436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7320104190 msgr2=0x7f73201106a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:06.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.980+0000 7f7326000640 1 --2- 172.21.15.43:0/4130276436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7320104190 0x7f73201106a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f7318009cd0 tx=0x7f731802c0e0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.980+0000 7f7326000640 1 -- 172.21.15.43:0/4130276436 shutdown_connections 2024-03-28T22:42:06.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.980+0000 7f7326000640 1 --2- 172.21.15.43:0/4130276436 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f73201037d0 0x7f7320103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.980+0000 7f7326000640 1 --2- 172.21.15.43:0/4130276436 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7320104190 0x7f73201106a0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.980+0000 7f7326000640 1 --2- 172.21.15.43:0/4130276436 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73201097e0 0x7f7320109bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.980+0000 7f7326000640 1 -- 172.21.15.43:0/4130276436 >> 172.21.15.43:0/4130276436 conn(0x7f73200ff260 msgr2=0x7f7320101680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:06.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.980+0000 7f7326000640 1 -- 172.21.15.43:0/4130276436 shutdown_connections 2024-03-28T22:42:06.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.980+0000 7f7326000640 1 -- 172.21.15.43:0/4130276436 wait complete. 2024-03-28T22:42:06.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.981+0000 7f7326000640 1 Processor -- start 2024-03-28T22:42:06.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.981+0000 7f7326000640 1 -- start start 2024-03-28T22:42:06.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.982+0000 7f7326000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73201037d0 0x7f732019a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.982+0000 7f7326000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7320104190 0x7f732019a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.982+0000 7f7326000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f73201097e0 0x7f732019e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.982+0000 7f7326000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7320113bb0 con 0x7f73201037d0 2024-03-28T22:42:06.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.982+0000 7f7326000640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7320113a30 con 0x7f73201097e0 2024-03-28T22:42:06.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.982+0000 7f7326000640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f7320113d30 con 0x7f7320104190 2024-03-28T22:42:06.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.982+0000 7f731fe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7320104190 0x7f732019a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.982+0000 7f7324c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73201037d0 0x7f732019a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.982+0000 7f731fe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7320104190 0x7f732019a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:44248/0 (socket says 172.21.15.43:44248) 2024-03-28T22:42:06.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.982+0000 7f731fe00640 1 -- 172.21.15.43:0/1813021020 learned_addr learned my addr 172.21.15.43:0/1813021020 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:06.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.982+0000 7f7325600640 1 --2- 172.21.15.43:0/1813021020 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f73201097e0 0x7f732019e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.982+0000 7f731fe00640 1 -- 172.21.15.43:0/1813021020 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f73201097e0 msgr2=0x7f732019e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:06.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.982+0000 7f731fe00640 1 --2- 172.21.15.43:0/1813021020 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f73201097e0 0x7f732019e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.982+0000 7f731fe00640 1 -- 172.21.15.43:0/1813021020 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73201037d0 msgr2=0x7f732019a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:06.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.982+0000 7f731fe00640 1 --2- 172.21.15.43:0/1813021020 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73201037d0 0x7f732019a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.982+0000 7f731fe00640 1 -- 172.21.15.43:0/1813021020 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f732019f060 con 0x7f7320104190 2024-03-28T22:42:06.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.982+0000 7f7325600640 1 --2- 172.21.15.43:0/1813021020 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f73201097e0 0x7f732019e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:06.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.983+0000 7f731fe00640 1 --2- 172.21.15.43:0/1813021020 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7320104190 0x7f732019a5b0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f730800e9f0 tx=0x7f730800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:06.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.983+0000 7f731d600640 1 -- 172.21.15.43:0/1813021020 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f730800cd80 con 0x7f7320104190 2024-03-28T22:42:06.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.983+0000 7f7326000640 1 -- 172.21.15.43:0/1813021020 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f732019f350 con 0x7f7320104190 2024-03-28T22:42:06.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.983+0000 7f731d600640 1 -- 172.21.15.43:0/1813021020 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7308010910 con 0x7f7320104190 2024-03-28T22:42:06.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.983+0000 7f731d600640 1 -- 172.21.15.43:0/1813021020 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7308018990 con 0x7f7320104190 2024-03-28T22:42:06.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.983+0000 7f7326000640 1 -- 172.21.15.43:0/1813021020 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73201a6be0 con 0x7f7320104190 2024-03-28T22:42:06.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.986+0000 7f732a3a3740 1 Processor -- start 2024-03-28T22:42:06.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.986+0000 7f731d600640 1 -- 172.21.15.43:0/1813021020 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f7308010ac0 con 0x7f7320104190 2024-03-28T22:42:06.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.987+0000 7f732a3a3740 1 -- start start 2024-03-28T22:42:06.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.987+0000 7f732a3a3740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e5ca8f3550 0x55e5ca8f3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.987+0000 7f732a3a3740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e5ca8f3e70 0x55e5ca8fc510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.987+0000 7f732a3a3740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e5ca8fca50 0x55e5ca8fee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.987+0000 7f732a3a3740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e5ca857050 con 0x55e5ca8f3550 2024-03-28T22:42:06.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.987+0000 7f732a3a3740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55e5ca7ffce0 con 0x55e5ca8f3e70 2024-03-28T22:42:06.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.987+0000 7f732a3a3740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55e5ca8530f0 con 0x55e5ca8fca50 2024-03-28T22:42:06.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.987+0000 7f7324c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e5ca8f3e70 0x55e5ca8fc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.987+0000 7f7325600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e5ca8f3550 0x55e5ca8f3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.987+0000 7f7324c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e5ca8f3e70 0x55e5ca8fc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44438/0 (socket says 172.21.15.43:44438) 2024-03-28T22:42:06.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.987+0000 7f7324c00640 1 -- 172.21.15.43:0/1822131114 learned_addr learned my addr 172.21.15.43:0/1822131114 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:06.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.987+0000 7f731fe00640 1 --2- 172.21.15.43:0/1822131114 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e5ca8fca50 0x55e5ca8fee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.988+0000 7f7324c00640 1 -- 172.21.15.43:0/1822131114 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e5ca8fca50 msgr2=0x55e5ca8fee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:06.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.988+0000 7f7324c00640 1 --2- 172.21.15.43:0/1822131114 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e5ca8fca50 0x55e5ca8fee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.988+0000 7f7324c00640 1 -- 172.21.15.43:0/1822131114 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e5ca8f3550 msgr2=0x55e5ca8f3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:06.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.988+0000 7f7324c00640 1 --2- 172.21.15.43:0/1822131114 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e5ca8f3550 0x55e5ca8f3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.988+0000 7f7324c00640 1 -- 172.21.15.43:0/1822131114 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5ca56b3f0 con 0x55e5ca8f3e70 2024-03-28T22:42:06.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.988+0000 7f7324c00640 1 --2- 172.21.15.43:0/1822131114 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e5ca8f3e70 0x55e5ca8fc510 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f731000bc90 tx=0x7f73100107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a41d7c048703bbb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:06.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.988+0000 7f731d600640 1 --2- 172.21.15.43:0/1813021020 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f73040a1ef0 0x7f73040a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.988+0000 7f731d600640 1 -- 172.21.15.43:0/1813021020 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f73080ce890 con 0x7f7320104190 2024-03-28T22:42:06.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.988+0000 7f7315c00640 1 -- 172.21.15.43:0/1822131114 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f731001c480 con 0x55e5ca8f3e70 2024-03-28T22:42:06.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.988+0000 7f7315c00640 1 -- 172.21.15.43:0/1822131114 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f731001cb00 con 0x55e5ca8f3e70 2024-03-28T22:42:06.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.988+0000 7f7315c00640 1 -- 172.21.15.43:0/1822131114 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7310011bb0 con 0x55e5ca8f3e70 2024-03-28T22:42:06.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.988+0000 7f7325600640 1 --2- 172.21.15.43:0/1813021020 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f73040a1ef0 0x7f73040a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.989+0000 7f732a3a3740 1 -- 172.21.15.43:0/1822131114 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e5ca8f3e70 msgr2=0x55e5ca8fc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:06.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.989+0000 7f732a3a3740 1 --2- 172.21.15.43:0/1822131114 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e5ca8f3e70 0x55e5ca8fc510 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f731000bc90 tx=0x7f73100107a0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.989+0000 7f7325600640 1 --2- 172.21.15.43:0/1813021020 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f73040a1ef0 0x7f73040a43b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f731802c620 tx=0x7f7318005d80 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:06.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.989+0000 7f732a3a3740 1 -- 172.21.15.43:0/1822131114 shutdown_connections 2024-03-28T22:42:06.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.989+0000 7f732a3a3740 1 --2- 172.21.15.43:0/1822131114 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e5ca8f3e70 0x55e5ca8fc510 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.989+0000 7f732a3a3740 1 --2- 172.21.15.43:0/1822131114 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e5ca8fca50 0x55e5ca8fee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.989+0000 7f732a3a3740 1 --2- 172.21.15.43:0/1822131114 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e5ca8f3550 0x55e5ca8f3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.989+0000 7f732a3a3740 1 -- 172.21.15.43:0/1822131114 >> 172.21.15.43:0/1822131114 conn(0x55e5ca8edc80 msgr2=0x55e5ca8ee060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:06.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.989+0000 7f732a3a3740 1 -- 172.21.15.43:0/1822131114 shutdown_connections 2024-03-28T22:42:06.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.990+0000 7f732a3a3740 1 -- 172.21.15.43:0/1822131114 wait complete. 2024-03-28T22:42:06.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.990+0000 7f732a3a3740 1 -- start start 2024-03-28T22:42:06.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.990+0000 7f732a3a3740 1 Processor -- start 2024-03-28T22:42:06.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.991+0000 7f732a3a3740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e5ca8f3550 0x55e5ca8f9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.991+0000 7f732a3a3740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e5ca8f3e70 0x55e5ca8f9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.991+0000 7f732a3a3740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e5ca8fca50 0x55e5ca8fa520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.991+0000 7f732a3a3740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e5ca8ff840 con 0x55e5ca8f3550 2024-03-28T22:42:06.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.991+0000 7f732a3a3740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55e5ca7f52f0 con 0x55e5ca8f3e70 2024-03-28T22:42:06.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.991+0000 7f732a3a3740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55e5ca8ff9c0 con 0x55e5ca8fca50 2024-03-28T22:42:06.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.991+0000 7f732a3a3740 10 client.0 _ll_register_callbacks cb 0x55e5ca56abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:06.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.991+0000 7f7324c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e5ca8f3550 0x55e5ca8f9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.991+0000 7f731fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e5ca8f3e70 0x55e5ca8f9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.991+0000 7f7325600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e5ca8fca50 0x55e5ca8fa520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.991+0000 7f731fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e5ca8f3e70 0x55e5ca8f9fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44444/0 (socket says 172.21.15.43:44444) 2024-03-28T22:42:06.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.991+0000 7f731fe00640 1 -- 172.21.15.43:0/525993541 learned_addr learned my addr 172.21.15.43:0/525993541 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:06.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.992+0000 7f7324c00640 1 -- 172.21.15.43:0/525993541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e5ca8fca50 msgr2=0x55e5ca8fa520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:06.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.992+0000 7f7324c00640 1 --2- 172.21.15.43:0/525993541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e5ca8fca50 0x55e5ca8fa520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.992+0000 7f7324c00640 1 -- 172.21.15.43:0/525993541 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e5ca8f3e70 msgr2=0x55e5ca8f9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:06.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.992+0000 7f7324c00640 1 --2- 172.21.15.43:0/525993541 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e5ca8f3e70 0x55e5ca8f9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.992+0000 7f7324c00640 1 -- 172.21.15.43:0/525993541 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5ca6357d0 con 0x55e5ca8f3550 2024-03-28T22:42:06.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.992+0000 7f7324c00640 1 --2- 172.21.15.43:0/525993541 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e5ca8f3550 0x55e5ca8f9aa0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7310000c00 tx=0x7f731001c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=570b7f140b5c5e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:06.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.992+0000 7f730e000640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:42:06.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.992+0000 7f730e000640 1 -- 172.21.15.43:0/525993541 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f731001c920 con 0x55e5ca8f3550 2024-03-28T22:42:06.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.992+0000 7f730e000640 1 -- 172.21.15.43:0/525993541 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f731000bd80 con 0x55e5ca8f3550 2024-03-28T22:42:06.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.992+0000 7f732a3a3740 1 -- 172.21.15.43:0/525993541 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f731001cb00 con 0x55e5ca8f3550 2024-03-28T22:42:06.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.993+0000 7f730e000640 1 -- 172.21.15.43:0/525993541 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7310011b40 con 0x55e5ca8f3550 2024-03-28T22:42:06.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.993+0000 7f732a3a3740 10 client.17238 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:06.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.993+0000 7f732a3a3740 1 -- 172.21.15.43:0/525993541 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e5ca90bb50 con 0x55e5ca8f3550 2024-03-28T22:42:06.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.993+0000 7f730e000640 1 -- 172.21.15.43:0/525993541 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f731001cb00 con 0x55e5ca8f3550 2024-03-28T22:42:06.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.993+0000 7f730e000640 1 -- 172.21.15.43:0/525993541 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f73100046c0 con 0x55e5ca8f3550 2024-03-28T22:42:06.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.993+0000 7f732a3a3740 10 client.17238 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:06.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.993+0000 7f732a3a3740 10 client.17238 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:42:06.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.993+0000 7f732a3a3740 20 client.17238 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:06.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.993+0000 7f732a3a3740 1 --2- 172.21.15.43:0/525993541 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55e5ca90cfb0 0x55e5ca90f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:06.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.993+0000 7f732a3a3740 4 client.17238 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:42:06.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.993+0000 7f732a3a3740 1 -- 172.21.15.43:0/525993541 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e5ca800680 con 0x55e5ca90cfb0 2024-03-28T22:42:06.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.994+0000 7f731fe00640 1 --2- 172.21.15.43:0/525993541 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55e5ca90cfb0 0x55e5ca90f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:06.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.994+0000 7f731fe00640 1 --2- 172.21.15.43:0/525993541 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55e5ca90cfb0 0x55e5ca90f3a0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7e18793d791f4ddd server_cookie=55e712f3137f31f4 in_seq=0 out_seq=0 2024-03-28T22:42:06.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.994+0000 7f730e000640 10 client.17238 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:42:06.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.995+0000 7f730e000640 1 -- 172.21.15.43:0/525993541 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e5ca800680 con 0x55e5ca90cfb0 2024-03-28T22:42:06.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.995+0000 7f730e000640 10 client.17238 handle_command_reply: tid=0 2024-03-28T22:42:06.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.995+0000 7f732a3a3740 1 client.17238 shutdown 2024-03-28T22:42:06.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.995+0000 7f732a3a3740 1 -- 172.21.15.43:0/525993541 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e5ca8f3550 msgr2=0x55e5ca8f9aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:06.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.996+0000 7f732a3a3740 1 --2- 172.21.15.43:0/525993541 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e5ca8f3550 0x55e5ca8f9aa0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7310000c00 tx=0x7f731001c730 comp rx=0 tx=0).stop 2024-03-28T22:42:06.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.996+0000 7f732a3a3740 1 -- 172.21.15.43:0/525993541 shutdown_connections 2024-03-28T22:42:06.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.996+0000 7f732a3a3740 1 --2- 172.21.15.43:0/525993541 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55e5ca90cfb0 0x55e5ca90f3a0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.996+0000 7f732a3a3740 1 --2- 172.21.15.43:0/525993541 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e5ca8f3e70 0x55e5ca8f9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.996+0000 7f732a3a3740 1 --2- 172.21.15.43:0/525993541 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e5ca8fca50 0x55e5ca8fa520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.996+0000 7f732a3a3740 1 --2- 172.21.15.43:0/525993541 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e5ca8f3550 0x55e5ca8f9aa0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:06.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.996+0000 7f730e000640 0 client.17238 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:42:06.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.996+0000 7f732a3a3740 1 -- 172.21.15.43:0/525993541 >> 172.21.15.43:0/525993541 conn(0x55e5ca8edc80 msgr2=0x55e5ca8ef9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:06.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.996+0000 7f732a3a3740 1 -- 172.21.15.43:0/525993541 shutdown_connections 2024-03-28T22:42:06.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.996+0000 7f732a3a3740 1 -- 172.21.15.43:0/525993541 wait complete. 2024-03-28T22:42:06.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:06.996+0000 7f732a3a3740 20 client.17238 trim_cache size 0 max 16384 2024-03-28T22:42:07.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.012+0000 7f732a3a3740 1 Processor -- start 2024-03-28T22:42:07.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.012+0000 7f732a3a3740 1 -- start start 2024-03-28T22:42:07.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.012+0000 7f732a3a3740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e5ca8f3550 0x55e5ca9085e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.012+0000 7f732a3a3740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e5ca8f3e70 0x55e5ca914ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.013+0000 7f732a3a3740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e5ca8fca50 0x55e5ca904c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.013+0000 7f732a3a3740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f73201b0bb0 con 0x55e5ca8fca50 2024-03-28T22:42:07.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.013+0000 7f732a3a3740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f73201b26f0 con 0x55e5ca8f3550 2024-03-28T22:42:07.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.013+0000 7f732a3a3740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f73201b1280 con 0x55e5ca8f3e70 2024-03-28T22:42:07.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.013+0000 7f7324c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e5ca8f3550 0x55e5ca9085e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.013+0000 7f7324c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e5ca8f3550 0x55e5ca9085e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44450/0 (socket says 172.21.15.43:44450) 2024-03-28T22:42:07.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.013+0000 7f7324c00640 1 -- 172.21.15.43:0/1878564909 learned_addr learned my addr 172.21.15.43:0/1878564909 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:07.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.013+0000 7f7325600640 1 --2- 172.21.15.43:0/1878564909 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e5ca8fca50 0x55e5ca904c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.013+0000 7f731fe00640 1 --2- 172.21.15.43:0/1878564909 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e5ca8f3e70 0x55e5ca914ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.014+0000 7f7324c00640 1 -- 172.21.15.43:0/1878564909 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e5ca8f3e70 msgr2=0x55e5ca914ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:07.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.014+0000 7f7324c00640 1 --2- 172.21.15.43:0/1878564909 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e5ca8f3e70 0x55e5ca914ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.014+0000 7f7324c00640 1 -- 172.21.15.43:0/1878564909 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e5ca8fca50 msgr2=0x55e5ca904c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:07.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.014+0000 7f7324c00640 1 --2- 172.21.15.43:0/1878564909 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e5ca8fca50 0x55e5ca904c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.014+0000 7f7324c00640 1 -- 172.21.15.43:0/1878564909 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5ca908b20 con 0x55e5ca8f3550 2024-03-28T22:42:07.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.014+0000 7f7324c00640 1 --2- 172.21.15.43:0/1878564909 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e5ca8f3550 0x55e5ca9085e0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f7310010ca0 tx=0x7f7310005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c06f6c48a4c3d7a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:07.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.014+0000 7f7315200640 1 -- 172.21.15.43:0/1878564909 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73100114b0 con 0x55e5ca8f3550 2024-03-28T22:42:07.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.015+0000 7f7315200640 1 -- 172.21.15.43:0/1878564909 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f731001cc80 con 0x55e5ca8f3550 2024-03-28T22:42:07.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.015+0000 7f7315200640 1 -- 172.21.15.43:0/1878564909 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73100050f0 con 0x55e5ca8f3550 2024-03-28T22:42:07.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.015+0000 7f732a3a3740 1 -- 172.21.15.43:0/1878564909 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e5ca8f3550 msgr2=0x55e5ca9085e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:07.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.015+0000 7f732a3a3740 1 --2- 172.21.15.43:0/1878564909 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e5ca8f3550 0x55e5ca9085e0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f7310010ca0 tx=0x7f7310005850 comp rx=0 tx=0).stop 2024-03-28T22:42:07.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.015+0000 7f732a3a3740 1 -- 172.21.15.43:0/1878564909 shutdown_connections 2024-03-28T22:42:07.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.015+0000 7f732a3a3740 1 --2- 172.21.15.43:0/1878564909 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e5ca8fca50 0x55e5ca904c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.015+0000 7f732a3a3740 1 --2- 172.21.15.43:0/1878564909 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e5ca8f3e70 0x55e5ca914ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.015+0000 7f732a3a3740 1 --2- 172.21.15.43:0/1878564909 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e5ca8f3550 0x55e5ca9085e0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.015+0000 7f732a3a3740 1 -- 172.21.15.43:0/1878564909 >> 172.21.15.43:0/1878564909 conn(0x55e5ca8edc80 msgr2=0x55e5ca8ec530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:07.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.015+0000 7f732a3a3740 1 -- 172.21.15.43:0/1878564909 shutdown_connections 2024-03-28T22:42:07.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.016+0000 7f732a3a3740 1 -- 172.21.15.43:0/1878564909 wait complete. 2024-03-28T22:42:07.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.016+0000 7f732a3a3740 1 -- start start 2024-03-28T22:42:07.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.016+0000 7f732a3a3740 1 Processor -- start 2024-03-28T22:42:07.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:06 smithi043 ceph-mon[31823]: pgmap v152: 129 pgs: 129 active+clean; 46 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 263 op/s 2024-03-28T22:42:07.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:06 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2815740441' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:42:07.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.017+0000 7f732a3a3740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e5ca8f3550 0x55e5ca904b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.017+0000 7f732a3a3740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e5ca8f3e70 0x55e5ca905050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.017+0000 7f732a3a3740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e5ca8fca50 0x55e5ca902190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.017+0000 7f732a3a3740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f73201b2020 con 0x55e5ca8f3550 2024-03-28T22:42:07.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.017+0000 7f732a3a3740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f73201b1950 con 0x55e5ca8fca50 2024-03-28T22:42:07.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.017+0000 7f732a3a3740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f73201aeb90 con 0x55e5ca8f3e70 2024-03-28T22:42:07.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.017+0000 7f732a3a3740 10 client.0 _ll_register_callbacks cb 0x7f7308005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:07.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.017+0000 7f731fe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e5ca8f3e70 0x55e5ca905050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.017+0000 7f7324c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e5ca8f3550 0x55e5ca904b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.017+0000 7f731fe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e5ca8f3e70 0x55e5ca905050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:44278/0 (socket says 172.21.15.43:44278) 2024-03-28T22:42:07.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.017+0000 7f731fe00640 1 -- 172.21.15.43:0/1756679487 learned_addr learned my addr 172.21.15.43:0/1756679487 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:07.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.017+0000 7f7325600640 1 --2- 172.21.15.43:0/1756679487 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e5ca8fca50 0x55e5ca902190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.017+0000 7f731fe00640 1 -- 172.21.15.43:0/1756679487 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e5ca8fca50 msgr2=0x55e5ca902190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:07.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.017+0000 7f731fe00640 1 --2- 172.21.15.43:0/1756679487 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e5ca8fca50 0x55e5ca902190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.017+0000 7f731fe00640 1 -- 172.21.15.43:0/1756679487 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e5ca8f3550 msgr2=0x55e5ca904b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:07.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.017+0000 7f731fe00640 1 --2- 172.21.15.43:0/1756679487 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e5ca8f3550 0x55e5ca904b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.017+0000 7f731fe00640 1 -- 172.21.15.43:0/1756679487 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5ca902960 con 0x55e5ca8f3e70 2024-03-28T22:42:07.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.018+0000 7f731fe00640 1 --2- 172.21.15.43:0/1756679487 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e5ca8f3e70 0x55e5ca905050 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f730809aec0 tx=0x7f73080d4120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2619eec248e0c18 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:07.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.018+0000 7f730e000640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:42:07.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.018+0000 7f730e000640 1 -- 172.21.15.43:0/1756679487 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f730809af30 con 0x55e5ca8f3e70 2024-03-28T22:42:07.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.018+0000 7f730e000640 1 -- 172.21.15.43:0/1756679487 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f73080975a0 con 0x55e5ca8f3e70 2024-03-28T22:42:07.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.018+0000 7f732a3a3740 1 -- 172.21.15.43:0/1756679487 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f731001cc80 con 0x55e5ca8f3e70 2024-03-28T22:42:07.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.018+0000 7f730e000640 1 -- 172.21.15.43:0/1756679487 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f730809efd0 con 0x55e5ca8f3e70 2024-03-28T22:42:07.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.018+0000 7f732a3a3740 10 client.27412 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:07.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.018+0000 7f732a3a3740 1 -- 172.21.15.43:0/1756679487 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e5ca972d60 con 0x55e5ca8f3e70 2024-03-28T22:42:07.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.018+0000 7f730e000640 1 -- 172.21.15.43:0/1756679487 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f731001cc80 con 0x55e5ca8f3e70 2024-03-28T22:42:07.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.019+0000 7f730e000640 1 -- 172.21.15.43:0/1756679487 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f73080ce440 con 0x55e5ca8f3e70 2024-03-28T22:42:07.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.019+0000 7f732a3a3740 10 client.27412 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:07.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.019+0000 7f732a3a3740 10 client.27412 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:42:07.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.019+0000 7f732a3a3740 20 client.27412 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:07.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.019+0000 7f732a3a3740 1 --2- 172.21.15.43:0/1756679487 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55e5ca9734e0 0x55e5ca975900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.019+0000 7f732a3a3740 4 client.27412 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:42:07.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.019+0000 7f732a3a3740 1 -- 172.21.15.43:0/1756679487 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e5ca864590 con 0x55e5ca9734e0 2024-03-28T22:42:07.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.019+0000 7f7324c00640 1 --2- 172.21.15.43:0/1756679487 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55e5ca9734e0 0x55e5ca975900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.019+0000 7f7324c00640 1 --2- 172.21.15.43:0/1756679487 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55e5ca9734e0 0x55e5ca975900 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3d8d1fa181fb8b9e server_cookie=ffa7403afc3a9a6d in_seq=0 out_seq=0 2024-03-28T22:42:07.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.019+0000 7f730e000640 10 client.27412 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:42:07.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.020+0000 7f730e000640 1 -- 172.21.15.43:0/1756679487 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e5ca864590 con 0x55e5ca9734e0 2024-03-28T22:42:07.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.020+0000 7f730e000640 10 client.27412 handle_command_reply: tid=0 2024-03-28T22:42:07.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.020+0000 7f732a3a3740 1 client.27412 shutdown 2024-03-28T22:42:07.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.020+0000 7f732a3a3740 1 -- 172.21.15.43:0/1756679487 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e5ca8f3e70 msgr2=0x55e5ca905050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:07.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.020+0000 7f732a3a3740 1 --2- 172.21.15.43:0/1756679487 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e5ca8f3e70 0x55e5ca905050 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f730809aec0 tx=0x7f73080d4120 comp rx=0 tx=0).stop 2024-03-28T22:42:07.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.020+0000 7f732a3a3740 1 -- 172.21.15.43:0/1756679487 shutdown_connections 2024-03-28T22:42:07.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.020+0000 7f732a3a3740 1 --2- 172.21.15.43:0/1756679487 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55e5ca9734e0 0x55e5ca975900 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.020+0000 7f732a3a3740 1 --2- 172.21.15.43:0/1756679487 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e5ca8fca50 0x55e5ca902190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.020+0000 7f732a3a3740 1 --2- 172.21.15.43:0/1756679487 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e5ca8f3e70 0x55e5ca905050 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.020+0000 7f732a3a3740 1 --2- 172.21.15.43:0/1756679487 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e5ca8f3550 0x55e5ca904b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.020+0000 7f730e000640 0 client.27412 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:42:07.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.020+0000 7f732a3a3740 1 -- 172.21.15.43:0/1756679487 >> 172.21.15.43:0/1756679487 conn(0x55e5ca8edc80 msgr2=0x55e5ca8ef860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:07.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.020+0000 7f732a3a3740 1 -- 172.21.15.43:0/1756679487 shutdown_connections 2024-03-28T22:42:07.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.020+0000 7f732a3a3740 1 -- 172.21.15.43:0/1756679487 wait complete. 2024-03-28T22:42:07.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.020+0000 7f732a3a3740 20 client.27412 trim_cache size 0 max 16384 2024-03-28T22:42:07.023 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:42:07.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.021+0000 7f7326000640 1 -- 172.21.15.43:0/1813021020 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f73040a1ef0 msgr2=0x7f73040a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:07.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.021+0000 7f7326000640 1 --2- 172.21.15.43:0/1813021020 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f73040a1ef0 0x7f73040a43b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f731802c620 tx=0x7f7318005d80 comp rx=0 tx=0).stop 2024-03-28T22:42:07.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.021+0000 7f7326000640 1 -- 172.21.15.43:0/1813021020 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7320104190 msgr2=0x7f732019a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:07.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.021+0000 7f7326000640 1 --2- 172.21.15.43:0/1813021020 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7320104190 0x7f732019a5b0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f730800e9f0 tx=0x7f730800eeb0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.021+0000 7f7326000640 1 -- 172.21.15.43:0/1813021020 shutdown_connections 2024-03-28T22:42:07.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.021+0000 7f7326000640 1 --2- 172.21.15.43:0/1813021020 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f73040a1ef0 0x7f73040a43b0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.021+0000 7f7326000640 1 --2- 172.21.15.43:0/1813021020 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f73201097e0 0x7f732019e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.021+0000 7f7326000640 1 --2- 172.21.15.43:0/1813021020 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7320104190 0x7f732019a5b0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.021+0000 7f7326000640 1 --2- 172.21.15.43:0/1813021020 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73201037d0 0x7f732019a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.021+0000 7f7326000640 1 -- 172.21.15.43:0/1813021020 >> 172.21.15.43:0/1813021020 conn(0x7f73200ff260 msgr2=0x7f73200ff9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:07.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.021+0000 7f7326000640 1 -- 172.21.15.43:0/1813021020 shutdown_connections 2024-03-28T22:42:07.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.021+0000 7f7326000640 1 -- 172.21.15.43:0/1813021020 wait complete. 2024-03-28T22:42:07.037 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T22:42:07.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:06 smithi172 ceph-mon[36566]: pgmap v152: 129 pgs: 129 active+clean; 46 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 263 op/s 2024-03-28T22:42:07.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:06 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2815740441' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:42:07.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:06 smithi082 ceph-mon[36471]: pgmap v152: 129 pgs: 129 active+clean; 46 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 263 op/s 2024-03-28T22:42:07.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:06 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2815740441' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:42:07.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.294+0000 7ffb57c00640 1 -- 172.21.15.43:0/678561254 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb50109900 msgr2=0x7ffb50109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:07.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.294+0000 7ffb57c00640 1 --2- 172.21.15.43:0/678561254 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb50109900 0x7ffb50109ce0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ffb44009930 tx=0x7ffb4402bc30 comp rx=0 tx=0).stop 2024-03-28T22:42:07.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.294+0000 7ffb57c00640 1 -- 172.21.15.43:0/678561254 shutdown_connections 2024-03-28T22:42:07.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.294+0000 7ffb57c00640 1 --2- 172.21.15.43:0/678561254 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffb501038f0 0x7ffb50103d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.294+0000 7ffb57c00640 1 --2- 172.21.15.43:0/678561254 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb501042b0 0x7ffb501107c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.294+0000 7ffb57c00640 1 --2- 172.21.15.43:0/678561254 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb50109900 0x7ffb50109ce0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.294+0000 7ffb57c00640 1 -- 172.21.15.43:0/678561254 >> 172.21.15.43:0/678561254 conn(0x7ffb500ff400 msgr2=0x7ffb50101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:07.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.294+0000 7ffb57c00640 1 -- 172.21.15.43:0/678561254 shutdown_connections 2024-03-28T22:42:07.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.294+0000 7ffb57c00640 1 -- 172.21.15.43:0/678561254 wait complete. 2024-03-28T22:42:07.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.295+0000 7ffb57c00640 1 Processor -- start 2024-03-28T22:42:07.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.295+0000 7ffb57c00640 1 -- start start 2024-03-28T22:42:07.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.295+0000 7ffb57c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb501038f0 0x7ffb50075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.295+0000 7ffb57c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb501042b0 0x7ffb500761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.295+0000 7ffb57c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffb50109900 0x7ffb500766e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.296+0000 7ffb57c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ffb50113d30 con 0x7ffb501042b0 2024-03-28T22:42:07.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.296+0000 7ffb57c00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ffb50113bb0 con 0x7ffb50109900 2024-03-28T22:42:07.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.296+0000 7ffb57c00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7ffb50113eb0 con 0x7ffb501038f0 2024-03-28T22:42:07.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.296+0000 7ffb55200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb501038f0 0x7ffb50075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.296+0000 7ffb55200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb501038f0 0x7ffb50075c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:44288/0 (socket says 172.21.15.43:44288) 2024-03-28T22:42:07.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.296+0000 7ffb55200640 1 -- 172.21.15.43:0/2391922723 learned_addr learned my addr 172.21.15.43:0/2391922723 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:07.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.296+0000 7ffb55c00640 1 --2- 172.21.15.43:0/2391922723 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffb50109900 0x7ffb500766e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.296+0000 7ffb4fe00640 1 --2- 172.21.15.43:0/2391922723 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb501042b0 0x7ffb500761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.296+0000 7ffb55200640 1 -- 172.21.15.43:0/2391922723 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffb50109900 msgr2=0x7ffb500766e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:07.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.296+0000 7ffb55200640 1 --2- 172.21.15.43:0/2391922723 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffb50109900 0x7ffb500766e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.296+0000 7ffb55200640 1 -- 172.21.15.43:0/2391922723 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb501042b0 msgr2=0x7ffb500761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:07.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.296+0000 7ffb55200640 1 --2- 172.21.15.43:0/2391922723 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb501042b0 0x7ffb500761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.296+0000 7ffb55200640 1 -- 172.21.15.43:0/2391922723 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb501aa920 con 0x7ffb501038f0 2024-03-28T22:42:07.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.296+0000 7ffb4fe00640 1 --2- 172.21.15.43:0/2391922723 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb501042b0 0x7ffb500761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:42:07.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.297+0000 7ffb55200640 1 --2- 172.21.15.43:0/2391922723 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb501038f0 0x7ffb50075c60 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ffb44009900 tx=0x7ffb440042e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:07.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.297+0000 7ffb4d600640 1 -- 172.21.15.43:0/2391922723 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb44043020 con 0x7ffb501038f0 2024-03-28T22:42:07.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.297+0000 7ffb4d600640 1 -- 172.21.15.43:0/2391922723 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb440044d0 con 0x7ffb501038f0 2024-03-28T22:42:07.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.297+0000 7ffb4d600640 1 -- 172.21.15.43:0/2391922723 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb44003be0 con 0x7ffb501038f0 2024-03-28T22:42:07.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.297+0000 7ffb57c00640 1 -- 172.21.15.43:0/2391922723 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb501aabb0 con 0x7ffb501038f0 2024-03-28T22:42:07.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.297+0000 7ffb57c00640 1 -- 172.21.15.43:0/2391922723 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb501ab150 con 0x7ffb501038f0 2024-03-28T22:42:07.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.301+0000 7ffb5a980740 1 Processor -- start 2024-03-28T22:42:07.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.301+0000 7ffb5a980740 1 -- start start 2024-03-28T22:42:07.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.301+0000 7ffb4d600640 1 -- 172.21.15.43:0/2391922723 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7ffb44046020 con 0x7ffb501038f0 2024-03-28T22:42:07.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.302+0000 7ffb5a980740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556de03ec380 0x556de03ec760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.302+0000 7ffb5a980740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556de03ecca0 0x556de03f5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.302+0000 7ffb5a980740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556de03f5880 0x556de03f7c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.302+0000 7ffb5a980740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x556de0350050 con 0x556de03ec380 2024-03-28T22:42:07.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.302+0000 7ffb5a980740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x556de02f8ce0 con 0x556de03ecca0 2024-03-28T22:42:07.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.302+0000 7ffb5a980740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x556de034c0f0 con 0x556de03f5880 2024-03-28T22:42:07.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.302+0000 7ffb55200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556de03ecca0 0x556de03f5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.302+0000 7ffb55200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556de03ecca0 0x556de03f5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44478/0 (socket says 172.21.15.43:44478) 2024-03-28T22:42:07.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.302+0000 7ffb55200640 1 -- 172.21.15.43:0/1284439807 learned_addr learned my addr 172.21.15.43:0/1284439807 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:07.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.302+0000 7ffb55c00640 1 --2- 172.21.15.43:0/1284439807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556de03ec380 0x556de03ec760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.302+0000 7ffb4fe00640 1 --2- 172.21.15.43:0/1284439807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556de03f5880 0x556de03f7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.302+0000 7ffb4fe00640 1 -- 172.21.15.43:0/1284439807 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556de03ecca0 msgr2=0x556de03f5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:07.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.302+0000 7ffb4fe00640 1 --2- 172.21.15.43:0/1284439807 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556de03ecca0 0x556de03f5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.302+0000 7ffb4fe00640 1 -- 172.21.15.43:0/1284439807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556de03ec380 msgr2=0x556de03ec760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:07.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.302+0000 7ffb4fe00640 1 --2- 172.21.15.43:0/1284439807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556de03ec380 0x556de03ec760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.302+0000 7ffb4fe00640 1 -- 172.21.15.43:0/1284439807 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556de00643f0 con 0x556de03f5880 2024-03-28T22:42:07.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.303+0000 7ffb4fe00640 1 --2- 172.21.15.43:0/1284439807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556de03f5880 0x556de03f7c70 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ffb500774b0 tx=0x7ffb3c0117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f490a1583462ba15 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:07.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.303+0000 7ffb42000640 1 -- 172.21.15.43:0/1284439807 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c012450 con 0x556de03f5880 2024-03-28T22:42:07.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.303+0000 7ffb42000640 1 -- 172.21.15.43:0/1284439807 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb3c012ad0 con 0x556de03f5880 2024-03-28T22:42:07.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.303+0000 7ffb42000640 1 -- 172.21.15.43:0/1284439807 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c00fb60 con 0x556de03f5880 2024-03-28T22:42:07.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.303+0000 7ffb4d600640 1 --2- 172.21.15.43:0/2391922723 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ffb380a1fb0 0x7ffb380a4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.304+0000 7ffb4d600640 1 -- 172.21.15.43:0/2391922723 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7ffb440f04c0 con 0x7ffb501038f0 2024-03-28T22:42:07.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.304+0000 7ffb55c00640 1 --2- 172.21.15.43:0/2391922723 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ffb380a1fb0 0x7ffb380a4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.304+0000 7ffb5a980740 1 -- 172.21.15.43:0/1284439807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556de03f5880 msgr2=0x556de03f7c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:07.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.304+0000 7ffb5a980740 1 --2- 172.21.15.43:0/1284439807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556de03f5880 0x556de03f7c70 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ffb500774b0 tx=0x7ffb3c0117b0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.304+0000 7ffb5a980740 1 -- 172.21.15.43:0/1284439807 shutdown_connections 2024-03-28T22:42:07.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.304+0000 7ffb5a980740 1 --2- 172.21.15.43:0/1284439807 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556de03ecca0 0x556de03f5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.304+0000 7ffb5a980740 1 --2- 172.21.15.43:0/1284439807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556de03f5880 0x556de03f7c70 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.304+0000 7ffb5a980740 1 --2- 172.21.15.43:0/1284439807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556de03ec380 0x556de03ec760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.304+0000 7ffb5a980740 1 -- 172.21.15.43:0/1284439807 >> 172.21.15.43:0/1284439807 conn(0x556de03e6ab0 msgr2=0x556de03e6e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:07.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.304+0000 7ffb5a980740 1 -- 172.21.15.43:0/1284439807 shutdown_connections 2024-03-28T22:42:07.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.304+0000 7ffb5a980740 1 -- 172.21.15.43:0/1284439807 wait complete. 2024-03-28T22:42:07.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.305+0000 7ffb55c00640 1 --2- 172.21.15.43:0/2391922723 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ffb380a1fb0 0x7ffb380a4470 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ffb48005fd0 tx=0x7ffb480096a0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:07.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.305+0000 7ffb5a980740 1 -- start start 2024-03-28T22:42:07.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.305+0000 7ffb5a980740 1 Processor -- start 2024-03-28T22:42:07.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.306+0000 7ffb5a980740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556de03ec380 0x556de03f2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.306+0000 7ffb5a980740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556de03ecca0 0x556de03f2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.306+0000 7ffb5a980740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556de03f5880 0x556de03f32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.306+0000 7ffb5a980740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x556de03f8670 con 0x556de03f5880 2024-03-28T22:42:07.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.306+0000 7ffb5a980740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x556de02ee2f0 con 0x556de03ecca0 2024-03-28T22:42:07.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.306+0000 7ffb5a980740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x556de03f87f0 con 0x556de03ec380 2024-03-28T22:42:07.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.306+0000 7ffb5a980740 10 client.0 _ll_register_callbacks cb 0x556de0063be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:07.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.306+0000 7ffb55200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556de03ec380 0x556de03f2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.306+0000 7ffb4fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556de03ecca0 0x556de03f2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.306+0000 7ffb55200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556de03ec380 0x556de03f2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:44298/0 (socket says 172.21.15.43:44298) 2024-03-28T22:42:07.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.306+0000 7ffb55200640 1 -- 172.21.15.43:0/1877505152 learned_addr learned my addr 172.21.15.43:0/1877505152 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:07.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.306+0000 7ffb4fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556de03ecca0 0x556de03f2d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44494/0 (socket says 172.21.15.43:44494) 2024-03-28T22:42:07.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.306+0000 7ffb55c00640 1 --2- 172.21.15.43:0/1877505152 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556de03f5880 0x556de03f32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.307+0000 7ffb55200640 1 -- 172.21.15.43:0/1877505152 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556de03ecca0 msgr2=0x556de03f2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:07.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.307+0000 7ffb55200640 1 --2- 172.21.15.43:0/1877505152 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556de03ecca0 0x556de03f2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.307+0000 7ffb55200640 1 -- 172.21.15.43:0/1877505152 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556de03f5880 msgr2=0x556de03f32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:07.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.307+0000 7ffb55200640 1 --2- 172.21.15.43:0/1877505152 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556de03f5880 0x556de03f32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.307+0000 7ffb55200640 1 -- 172.21.15.43:0/1877505152 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556de012e7d0 con 0x556de03ec380 2024-03-28T22:42:07.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.307+0000 7ffb4fe00640 1 --2- 172.21.15.43:0/1877505152 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556de03ecca0 0x556de03f2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:07.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.307+0000 7ffb55c00640 1 --2- 172.21.15.43:0/1877505152 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556de03f5880 0x556de03f32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:42:07.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.307+0000 7ffb55200640 1 --2- 172.21.15.43:0/1877505152 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556de03ec380 0x556de03f2830 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ffb44000c00 tx=0x7ffb44009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bd4f32cc4001946 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:07.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.307+0000 7ffb36a00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:42:07.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.307+0000 7ffb36a00640 1 -- 172.21.15.43:0/1877505152 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb440bc7a0 con 0x556de03ec380 2024-03-28T22:42:07.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.307+0000 7ffb36a00640 1 -- 172.21.15.43:0/1877505152 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7ffb440c4f70 con 0x556de03ec380 2024-03-28T22:42:07.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.307+0000 7ffb5a980740 1 -- 172.21.15.43:0/1877505152 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb3c012ad0 con 0x556de03ec380 2024-03-28T22:42:07.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.308+0000 7ffb36a00640 1 -- 172.21.15.43:0/1877505152 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ffb440bada0 con 0x556de03ec380 2024-03-28T22:42:07.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.308+0000 7ffb5a980740 10 client.27424 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:07.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.308+0000 7ffb5a980740 1 -- 172.21.15.43:0/1877505152 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556de03f86c0 con 0x556de03ec380 2024-03-28T22:42:07.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.308+0000 7ffb36a00640 1 -- 172.21.15.43:0/1877505152 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb3c012ad0 con 0x556de03ec380 2024-03-28T22:42:07.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.308+0000 7ffb36a00640 1 -- 172.21.15.43:0/1877505152 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7ffb4402ca90 con 0x556de03ec380 2024-03-28T22:42:07.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.308+0000 7ffb5a980740 10 client.27424 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:07.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.308+0000 7ffb5a980740 10 client.27424 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:42:07.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.308+0000 7ffb5a980740 20 client.27424 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:07.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.308+0000 7ffb5a980740 1 --2- 172.21.15.43:0/1877505152 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x556de0405cb0 0x556de0408130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.308+0000 7ffb5a980740 4 client.27424 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:42:07.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.308+0000 7ffb5a980740 1 -- 172.21.15.43:0/1877505152 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556de02f9680 con 0x556de0405cb0 2024-03-28T22:42:07.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.309+0000 7ffb4fe00640 1 --2- 172.21.15.43:0/1877505152 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x556de0405cb0 0x556de0408130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.309+0000 7ffb4fe00640 1 --2- 172.21.15.43:0/1877505152 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x556de0405cb0 0x556de0408130 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c028d982f31ffcd1 server_cookie=81856f6f389a537c in_seq=0 out_seq=0 2024-03-28T22:42:07.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.309+0000 7ffb36a00640 10 client.27424 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:42:07.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.310+0000 7ffb36a00640 1 -- 172.21.15.43:0/1877505152 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556de02f9680 con 0x556de0405cb0 2024-03-28T22:42:07.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.310+0000 7ffb36a00640 10 client.27424 handle_command_reply: tid=0 2024-03-28T22:42:07.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.310+0000 7ffb5a980740 1 client.27424 shutdown 2024-03-28T22:42:07.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.310+0000 7ffb5a980740 1 -- 172.21.15.43:0/1877505152 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556de03ec380 msgr2=0x556de03f2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:07.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.310+0000 7ffb5a980740 1 --2- 172.21.15.43:0/1877505152 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556de03ec380 0x556de03f2830 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ffb44000c00 tx=0x7ffb44009560 comp rx=0 tx=0).stop 2024-03-28T22:42:07.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.311+0000 7ffb5a980740 1 -- 172.21.15.43:0/1877505152 shutdown_connections 2024-03-28T22:42:07.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.311+0000 7ffb5a980740 1 --2- 172.21.15.43:0/1877505152 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x556de0405cb0 0x556de0408130 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.311+0000 7ffb5a980740 1 --2- 172.21.15.43:0/1877505152 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556de03ecca0 0x556de03f2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.311+0000 7ffb5a980740 1 --2- 172.21.15.43:0/1877505152 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556de03f5880 0x556de03f32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.311+0000 7ffb5a980740 1 --2- 172.21.15.43:0/1877505152 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556de03ec380 0x556de03f2830 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.311+0000 7ffb5a980740 1 -- 172.21.15.43:0/1877505152 >> 172.21.15.43:0/1877505152 conn(0x556de03e6ab0 msgr2=0x556de03e8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:07.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.311+0000 7ffb36a00640 0 client.27424 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:42:07.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.311+0000 7ffb5a980740 1 -- 172.21.15.43:0/1877505152 shutdown_connections 2024-03-28T22:42:07.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.311+0000 7ffb5a980740 1 -- 172.21.15.43:0/1877505152 wait complete. 2024-03-28T22:42:07.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.311+0000 7ffb5a980740 20 client.27424 trim_cache size 0 max 16384 2024-03-28T22:42:07.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.328+0000 7ffb5a980740 1 Processor -- start 2024-03-28T22:42:07.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.328+0000 7ffb5a980740 1 -- start start 2024-03-28T22:42:07.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.328+0000 7ffb5a980740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556de03ec380 0x556de03f4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.328+0000 7ffb5a980740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556de03ecca0 0x556de03f30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.328+0000 7ffb5a980740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556de03f5880 0x556de040da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.328+0000 7ffb5a980740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ffb501b5250 con 0x556de03f5880 2024-03-28T22:42:07.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.328+0000 7ffb5a980740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ffb501b6d90 con 0x556de03ec380 2024-03-28T22:42:07.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.328+0000 7ffb5a980740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7ffb501b5920 con 0x556de03ecca0 2024-03-28T22:42:07.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.329+0000 7ffb4fe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556de03ecca0 0x556de03f30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.329+0000 7ffb55200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556de03ec380 0x556de03f4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.329+0000 7ffb4fe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556de03ecca0 0x556de03f30d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:44304/0 (socket says 172.21.15.43:44304) 2024-03-28T22:42:07.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.329+0000 7ffb4fe00640 1 -- 172.21.15.43:0/3744656609 learned_addr learned my addr 172.21.15.43:0/3744656609 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:07.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.329+0000 7ffb55c00640 1 --2- 172.21.15.43:0/3744656609 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556de03f5880 0x556de040da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.329+0000 7ffb55200640 1 -- 172.21.15.43:0/3744656609 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556de03ecca0 msgr2=0x556de03f30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:07.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.329+0000 7ffb55200640 1 --2- 172.21.15.43:0/3744656609 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556de03ecca0 0x556de03f30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.329+0000 7ffb55200640 1 -- 172.21.15.43:0/3744656609 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556de03f5880 msgr2=0x556de040da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:07.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.329+0000 7ffb55200640 1 --2- 172.21.15.43:0/3744656609 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556de03f5880 0x556de040da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.329+0000 7ffb55200640 1 -- 172.21.15.43:0/3744656609 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556de03ee410 con 0x556de03ec380 2024-03-28T22:42:07.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.329+0000 7ffb55c00640 1 --2- 172.21.15.43:0/3744656609 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556de03f5880 0x556de040da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:42:07.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.329+0000 7ffb55200640 1 --2- 172.21.15.43:0/3744656609 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556de03ec380 0x556de03f4f60 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7ffb44005c40 tx=0x7ffb44003680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15aea6f1a3ea2e33 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:07.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.329+0000 7ffb41600640 1 -- 172.21.15.43:0/3744656609 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb440bc5c0 con 0x556de03ec380 2024-03-28T22:42:07.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.330+0000 7ffb41600640 1 -- 172.21.15.43:0/3744656609 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb44004c80 con 0x556de03ec380 2024-03-28T22:42:07.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.330+0000 7ffb41600640 1 -- 172.21.15.43:0/3744656609 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb440bd360 con 0x556de03ec380 2024-03-28T22:42:07.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.330+0000 7ffb5a980740 1 -- 172.21.15.43:0/3744656609 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556de03ec380 msgr2=0x556de03f4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:07.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.330+0000 7ffb5a980740 1 --2- 172.21.15.43:0/3744656609 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556de03ec380 0x556de03f4f60 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7ffb44005c40 tx=0x7ffb44003680 comp rx=0 tx=0).stop 2024-03-28T22:42:07.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.330+0000 7ffb5a980740 1 -- 172.21.15.43:0/3744656609 shutdown_connections 2024-03-28T22:42:07.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.330+0000 7ffb5a980740 1 --2- 172.21.15.43:0/3744656609 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556de03f5880 0x556de040da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.331+0000 7ffb5a980740 1 --2- 172.21.15.43:0/3744656609 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556de03ecca0 0x556de03f30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.331+0000 7ffb5a980740 1 --2- 172.21.15.43:0/3744656609 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556de03ec380 0x556de03f4f60 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.331+0000 7ffb5a980740 1 -- 172.21.15.43:0/3744656609 >> 172.21.15.43:0/3744656609 conn(0x556de03e6ab0 msgr2=0x556de03e7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:07.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.331+0000 7ffb5a980740 1 -- 172.21.15.43:0/3744656609 shutdown_connections 2024-03-28T22:42:07.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.331+0000 7ffb5a980740 1 -- 172.21.15.43:0/3744656609 wait complete. 2024-03-28T22:42:07.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.331+0000 7ffb5a980740 1 -- start start 2024-03-28T22:42:07.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.331+0000 7ffb5a980740 1 Processor -- start 2024-03-28T22:42:07.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.332+0000 7ffb5a980740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556de03ec380 0x556de03fdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.332+0000 7ffb5a980740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556de03ecca0 0x556de03ef080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.332+0000 7ffb5a980740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556de03f5880 0x556de03ef5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.332+0000 7ffb5a980740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ffb501b66c0 con 0x556de03f5880 2024-03-28T22:42:07.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.332+0000 7ffb5a980740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ffb501b5ff0 con 0x556de03ec380 2024-03-28T22:42:07.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.332+0000 7ffb5a980740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7ffb501b3230 con 0x556de03ecca0 2024-03-28T22:42:07.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.332+0000 7ffb5a980740 10 client.0 _ll_register_callbacks cb 0x7ffb440f3d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:07.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.332+0000 7ffb55200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556de03ec380 0x556de03fdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.332+0000 7ffb55200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556de03ec380 0x556de03fdcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44506/0 (socket says 172.21.15.43:44506) 2024-03-28T22:42:07.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.332+0000 7ffb4fe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556de03ecca0 0x556de03ef080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.332+0000 7ffb55200640 1 -- 172.21.15.43:0/470568921 learned_addr learned my addr 172.21.15.43:0/470568921 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:07.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.333+0000 7ffb55c00640 1 --2- 172.21.15.43:0/470568921 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556de03f5880 0x556de03ef5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.333+0000 7ffb55200640 1 -- 172.21.15.43:0/470568921 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556de03ecca0 msgr2=0x556de03ef080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:07.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.333+0000 7ffb55200640 1 --2- 172.21.15.43:0/470568921 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556de03ecca0 0x556de03ef080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.333+0000 7ffb55200640 1 -- 172.21.15.43:0/470568921 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556de03f5880 msgr2=0x556de03ef5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:07.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.333+0000 7ffb55200640 1 --2- 172.21.15.43:0/470568921 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556de03f5880 0x556de03ef5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.333+0000 7ffb55200640 1 -- 172.21.15.43:0/470568921 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556de03efdf0 con 0x556de03ec380 2024-03-28T22:42:07.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.333+0000 7ffb55200640 1 --2- 172.21.15.43:0/470568921 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556de03ec380 0x556de03fdcc0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ffb440b9030 tx=0x7ffb4402bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9107decf734cf6ca server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:07.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.333+0000 7ffb36a00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:42:07.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.333+0000 7ffb36a00640 1 -- 172.21.15.43:0/470568921 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb440bc5c0 con 0x556de03ec380 2024-03-28T22:42:07.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.333+0000 7ffb36a00640 1 -- 172.21.15.43:0/470568921 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7ffb440bc2c0 con 0x556de03ec380 2024-03-28T22:42:07.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.333+0000 7ffb5a980740 1 -- 172.21.15.43:0/470568921 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb44004c80 con 0x556de03ec380 2024-03-28T22:42:07.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.334+0000 7ffb36a00640 1 -- 172.21.15.43:0/470568921 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ffb440b8e70 con 0x556de03ec380 2024-03-28T22:42:07.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.334+0000 7ffb5a980740 10 client.27482 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:07.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.334+0000 7ffb5a980740 1 -- 172.21.15.43:0/470568921 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556de03efdf0 con 0x556de03ec380 2024-03-28T22:42:07.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.334+0000 7ffb36a00640 1 -- 172.21.15.43:0/470568921 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb44004c80 con 0x556de03ec380 2024-03-28T22:42:07.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.334+0000 7ffb36a00640 1 -- 172.21.15.43:0/470568921 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7ffb440bbb20 con 0x556de03ec380 2024-03-28T22:42:07.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.334+0000 7ffb5a980740 10 client.27482 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:07.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.334+0000 7ffb5a980740 10 client.27482 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:42:07.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.334+0000 7ffb5a980740 20 client.27482 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:07.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.334+0000 7ffb5a980740 1 --2- 172.21.15.43:0/470568921 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x556de046c250 0x556de046e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.334+0000 7ffb5a980740 4 client.27482 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:42:07.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.334+0000 7ffb5a980740 1 -- 172.21.15.43:0/470568921 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556de035d590 con 0x556de046c250 2024-03-28T22:42:07.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.334+0000 7ffb4fe00640 1 --2- 172.21.15.43:0/470568921 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x556de046c250 0x556de046e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.335+0000 7ffb4fe00640 1 --2- 172.21.15.43:0/470568921 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x556de046c250 0x556de046e640 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=76898ac225c523e9 server_cookie=2db3d5503c244b4d in_seq=0 out_seq=0 2024-03-28T22:42:07.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.335+0000 7ffb36a00640 10 client.27482 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:42:07.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.335+0000 7ffb36a00640 1 -- 172.21.15.43:0/470568921 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556de035d590 con 0x556de046c250 2024-03-28T22:42:07.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.335+0000 7ffb36a00640 10 client.27482 handle_command_reply: tid=0 2024-03-28T22:42:07.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.335+0000 7ffb5a980740 1 client.27482 shutdown 2024-03-28T22:42:07.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.335+0000 7ffb5a980740 1 -- 172.21.15.43:0/470568921 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556de03ec380 msgr2=0x556de03fdcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:07.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.335+0000 7ffb5a980740 1 --2- 172.21.15.43:0/470568921 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556de03ec380 0x556de03fdcc0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ffb440b9030 tx=0x7ffb4402bc30 comp rx=0 tx=0).stop 2024-03-28T22:42:07.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.335+0000 7ffb5a980740 1 -- 172.21.15.43:0/470568921 shutdown_connections 2024-03-28T22:42:07.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.335+0000 7ffb5a980740 1 --2- 172.21.15.43:0/470568921 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x556de046c250 0x556de046e640 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.335+0000 7ffb5a980740 1 --2- 172.21.15.43:0/470568921 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556de03f5880 0x556de03ef5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.335+0000 7ffb5a980740 1 --2- 172.21.15.43:0/470568921 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556de03ecca0 0x556de03ef080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.335+0000 7ffb5a980740 1 --2- 172.21.15.43:0/470568921 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556de03ec380 0x556de03fdcc0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.335+0000 7ffb36a00640 0 client.27482 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:42:07.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.335+0000 7ffb5a980740 1 -- 172.21.15.43:0/470568921 >> 172.21.15.43:0/470568921 conn(0x556de03e6ab0 msgr2=0x556de03e8680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:07.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.336+0000 7ffb5a980740 1 -- 172.21.15.43:0/470568921 shutdown_connections 2024-03-28T22:42:07.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.336+0000 7ffb5a980740 1 -- 172.21.15.43:0/470568921 wait complete. 2024-03-28T22:42:07.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.336+0000 7ffb5a980740 20 client.27482 trim_cache size 0 max 16384 2024-03-28T22:42:07.338 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:42:07.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.336+0000 7ffb57c00640 1 -- 172.21.15.43:0/2391922723 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ffb380a1fb0 msgr2=0x7ffb380a4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:07.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.336+0000 7ffb57c00640 1 --2- 172.21.15.43:0/2391922723 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ffb380a1fb0 0x7ffb380a4470 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ffb48005fd0 tx=0x7ffb480096a0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.336+0000 7ffb57c00640 1 -- 172.21.15.43:0/2391922723 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb501038f0 msgr2=0x7ffb50075c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:07.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.336+0000 7ffb57c00640 1 --2- 172.21.15.43:0/2391922723 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb501038f0 0x7ffb50075c60 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ffb44009900 tx=0x7ffb440042e0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.337+0000 7ffb57c00640 1 -- 172.21.15.43:0/2391922723 shutdown_connections 2024-03-28T22:42:07.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.337+0000 7ffb57c00640 1 --2- 172.21.15.43:0/2391922723 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ffb380a1fb0 0x7ffb380a4470 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.337+0000 7ffb57c00640 1 --2- 172.21.15.43:0/2391922723 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffb50109900 0x7ffb500766e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.337+0000 7ffb57c00640 1 --2- 172.21.15.43:0/2391922723 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb501042b0 0x7ffb500761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.337+0000 7ffb57c00640 1 --2- 172.21.15.43:0/2391922723 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffb501038f0 0x7ffb50075c60 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.337+0000 7ffb57c00640 1 -- 172.21.15.43:0/2391922723 >> 172.21.15.43:0/2391922723 conn(0x7ffb500ff400 msgr2=0x7ffb500ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:07.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.337+0000 7ffb57c00640 1 -- 172.21.15.43:0/2391922723 shutdown_connections 2024-03-28T22:42:07.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.337+0000 7ffb57c00640 1 -- 172.21.15.43:0/2391922723 wait complete. 2024-03-28T22:42:07.353 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:42:07.363 INFO:tasks.workunit.client.0.smithi043.stdout: 1 327327 7.79 MB/sec execute 98 sec latency 21.131 ms 2024-03-28T22:42:07.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.561+0000 7f2d67e00640 1 -- 172.21.15.43:0/1722884227 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d68107f00 msgr2=0x7f2d681082e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:07.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.561+0000 7f2d67e00640 1 --2- 172.21.15.43:0/1722884227 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d68107f00 0x7f2d681082e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f2d58009900 tx=0x7f2d5802bc30 comp rx=0 tx=0).stop 2024-03-28T22:42:07.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.561+0000 7f2d67e00640 1 -- 172.21.15.43:0/1722884227 shutdown_connections 2024-03-28T22:42:07.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.561+0000 7f2d67e00640 1 --2- 172.21.15.43:0/1722884227 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2d6810cf50 0x7f2d68114b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.561+0000 7f2d67e00640 1 --2- 172.21.15.43:0/1722884227 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d681088b0 0x7f2d6810c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.561+0000 7f2d67e00640 1 --2- 172.21.15.43:0/1722884227 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d68107f00 0x7f2d681082e0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.561+0000 7f2d67e00640 1 -- 172.21.15.43:0/1722884227 >> 172.21.15.43:0/1722884227 conn(0x7f2d68077570 msgr2=0x7f2d68106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:07.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.561+0000 7f2d67e00640 1 -- 172.21.15.43:0/1722884227 shutdown_connections 2024-03-28T22:42:07.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.561+0000 7f2d67e00640 1 -- 172.21.15.43:0/1722884227 wait complete. 2024-03-28T22:42:07.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.563+0000 7f2d67e00640 1 Processor -- start 2024-03-28T22:42:07.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.563+0000 7f2d67e00640 1 -- start start 2024-03-28T22:42:07.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.563+0000 7f2d67e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2d68107f00 0x7f2d6819e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.563+0000 7f2d67e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d6810cf50 0x7f2d6819eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.563+0000 7f2d67e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d681a2e40 0x7f2d681a32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.563+0000 7f2d67e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2d68118000 con 0x7f2d681a2e40 2024-03-28T22:42:07.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.563+0000 7f2d67e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f2d68117e80 con 0x7f2d68107f00 2024-03-28T22:42:07.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.563+0000 7f2d67e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f2d68118180 con 0x7f2d6810cf50 2024-03-28T22:42:07.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.563+0000 7f2d66a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2d68107f00 0x7f2d6819e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.563+0000 7f2d66a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2d68107f00 0x7f2d6819e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44518/0 (socket says 172.21.15.43:44518) 2024-03-28T22:42:07.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.563+0000 7f2d66a00640 1 -- 172.21.15.43:0/387104982 learned_addr learned my addr 172.21.15.43:0/387104982 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:07.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.563+0000 7f2d66000640 1 --2- 172.21.15.43:0/387104982 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d6810cf50 0x7f2d6819eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.564+0000 7f2d67400640 1 --2- 172.21.15.43:0/387104982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d681a2e40 0x7f2d681a32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.564+0000 7f2d66a00640 1 -- 172.21.15.43:0/387104982 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d6810cf50 msgr2=0x7f2d6819eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:07.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.564+0000 7f2d66a00640 1 --2- 172.21.15.43:0/387104982 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d6810cf50 0x7f2d6819eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.564+0000 7f2d66a00640 1 -- 172.21.15.43:0/387104982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d681a2e40 msgr2=0x7f2d681a32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:07.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.564+0000 7f2d66a00640 1 --2- 172.21.15.43:0/387104982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d681a2e40 0x7f2d681a32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.564+0000 7f2d66a00640 1 -- 172.21.15.43:0/387104982 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d681a3930 con 0x7f2d68107f00 2024-03-28T22:42:07.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.564+0000 7f2d66a00640 1 --2- 172.21.15.43:0/387104982 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2d68107f00 0x7f2d6819e580 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f2d58006010 tx=0x7f2d5802cb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:07.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.565+0000 7f2d5f400640 1 -- 172.21.15.43:0/387104982 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d58041070 con 0x7f2d68107f00 2024-03-28T22:42:07.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.565+0000 7f2d5f400640 1 -- 172.21.15.43:0/387104982 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d58004450 con 0x7f2d68107f00 2024-03-28T22:42:07.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.565+0000 7f2d67e00640 1 -- 172.21.15.43:0/387104982 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d681a3bc0 con 0x7f2d68107f00 2024-03-28T22:42:07.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.565+0000 7f2d5f400640 1 -- 172.21.15.43:0/387104982 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d58003e00 con 0x7f2d68107f00 2024-03-28T22:42:07.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.565+0000 7f2d67e00640 1 -- 172.21.15.43:0/387104982 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d681ab3e0 con 0x7f2d68107f00 2024-03-28T22:42:07.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.568+0000 7f2d5f400640 1 -- 172.21.15.43:0/387104982 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f2d58044070 con 0x7f2d68107f00 2024-03-28T22:42:07.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.568+0000 7f2d70363740 1 Processor -- start 2024-03-28T22:42:07.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.568+0000 7f2d70363740 1 -- start start 2024-03-28T22:42:07.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.569+0000 7f2d70363740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574c7340550 0x5574c7340930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.569+0000 7f2d70363740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5574c7340e70 0x5574c7349510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.569+0000 7f2d70363740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5574c7349a50 0x5574c734be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.569+0000 7f2d70363740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5574c72a4050 con 0x5574c7340e70 2024-03-28T22:42:07.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.569+0000 7f2d70363740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5574c724cce0 con 0x5574c7340550 2024-03-28T22:42:07.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.569+0000 7f2d70363740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5574c72a00f0 con 0x5574c7349a50 2024-03-28T22:42:07.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.569+0000 7f2d67400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574c7340550 0x5574c7340930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.569+0000 7f2d67400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574c7340550 0x5574c7340930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44524/0 (socket says 172.21.15.43:44524) 2024-03-28T22:42:07.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.569+0000 7f2d66a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5574c7340e70 0x5574c7349510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.569+0000 7f2d67400640 1 -- 172.21.15.43:0/1650044841 learned_addr learned my addr 172.21.15.43:0/1650044841 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:07.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.569+0000 7f2d66000640 1 --2- 172.21.15.43:0/1650044841 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5574c7349a50 0x5574c734be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.569+0000 7f2d67400640 1 -- 172.21.15.43:0/1650044841 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5574c7349a50 msgr2=0x5574c734be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:07.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.569+0000 7f2d67400640 1 --2- 172.21.15.43:0/1650044841 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5574c7349a50 0x5574c734be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.570+0000 7f2d67400640 1 -- 172.21.15.43:0/1650044841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5574c7340e70 msgr2=0x5574c7349510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:07.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.570+0000 7f2d67400640 1 --2- 172.21.15.43:0/1650044841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5574c7340e70 0x5574c7349510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.570+0000 7f2d67400640 1 -- 172.21.15.43:0/1650044841 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574c6fb83f0 con 0x5574c7340550 2024-03-28T22:42:07.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.570+0000 7f2d67400640 1 --2- 172.21.15.43:0/1650044841 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574c7340550 0x5574c7340930 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f2d60009f40 tx=0x7f2d600118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bef210231059998 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:07.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.570+0000 7f2d53400640 1 -- 172.21.15.43:0/1650044841 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d6001e070 con 0x5574c7340550 2024-03-28T22:42:07.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.570+0000 7f2d53400640 1 -- 172.21.15.43:0/1650044841 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d60003c30 con 0x5574c7340550 2024-03-28T22:42:07.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.570+0000 7f2d53400640 1 -- 172.21.15.43:0/1650044841 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d6000f930 con 0x5574c7340550 2024-03-28T22:42:07.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.570+0000 7f2d5f400640 1 --2- 172.21.15.43:0/387104982 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f2d4c0a1f70 0x7f2d4c0a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.571+0000 7f2d66a00640 1 --2- 172.21.15.43:0/387104982 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f2d4c0a1f70 0x7f2d4c0a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.571+0000 7f2d5f400640 1 -- 172.21.15.43:0/387104982 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f2d580efe10 con 0x7f2d68107f00 2024-03-28T22:42:07.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.571+0000 7f2d70363740 1 -- 172.21.15.43:0/1650044841 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574c7340550 msgr2=0x5574c7340930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:07.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.571+0000 7f2d70363740 1 --2- 172.21.15.43:0/1650044841 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574c7340550 0x5574c7340930 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f2d60009f40 tx=0x7f2d600118d0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.571+0000 7f2d70363740 1 -- 172.21.15.43:0/1650044841 shutdown_connections 2024-03-28T22:42:07.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.571+0000 7f2d70363740 1 --2- 172.21.15.43:0/1650044841 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5574c7349a50 0x5574c734be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.571+0000 7f2d70363740 1 --2- 172.21.15.43:0/1650044841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5574c7340e70 0x5574c7349510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.571+0000 7f2d70363740 1 --2- 172.21.15.43:0/1650044841 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574c7340550 0x5574c7340930 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.571+0000 7f2d70363740 1 -- 172.21.15.43:0/1650044841 >> 172.21.15.43:0/1650044841 conn(0x5574c733ac80 msgr2=0x5574c733b060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:07.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.571+0000 7f2d66a00640 1 --2- 172.21.15.43:0/387104982 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f2d4c0a1f70 0x7f2d4c0a4430 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2d580ed380 tx=0x7f2d580f3960 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:07.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.571+0000 7f2d70363740 1 -- 172.21.15.43:0/1650044841 shutdown_connections 2024-03-28T22:42:07.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.571+0000 7f2d70363740 1 -- 172.21.15.43:0/1650044841 wait complete. 2024-03-28T22:42:07.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.572+0000 7f2d70363740 1 -- start start 2024-03-28T22:42:07.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.572+0000 7f2d70363740 1 Processor -- start 2024-03-28T22:42:07.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.573+0000 7f2d70363740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574c7340550 0x5574c7346ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.573+0000 7f2d70363740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5574c7340e70 0x5574c7347000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.573+0000 7f2d70363740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5574c7349a50 0x5574c7347540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.573+0000 7f2d70363740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5574c734c840 con 0x5574c7340e70 2024-03-28T22:42:07.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.573+0000 7f2d70363740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5574c72422f0 con 0x5574c7340550 2024-03-28T22:42:07.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.573+0000 7f2d70363740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5574c734c9c0 con 0x5574c7349a50 2024-03-28T22:42:07.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.573+0000 7f2d70363740 10 client.0 _ll_register_callbacks cb 0x5574c6fb7be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:07.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.573+0000 7f2d66000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5574c7340e70 0x5574c7347000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.573+0000 7f2d67400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574c7340550 0x5574c7346ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.573+0000 7f2d66000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5574c7340e70 0x5574c7347000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48276/0 (socket says 172.21.15.43:48276) 2024-03-28T22:42:07.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.573+0000 7f2d66000640 1 -- 172.21.15.43:0/2334380639 learned_addr learned my addr 172.21.15.43:0/2334380639 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:07.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.573+0000 7f2d67400640 1 --2- 172.21.15.43:0/2334380639 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5574c7349a50 0x5574c7347540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.573+0000 7f2d66000640 1 -- 172.21.15.43:0/2334380639 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5574c7349a50 msgr2=0x5574c7347540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:07.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.573+0000 7f2d66000640 1 --2- 172.21.15.43:0/2334380639 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5574c7349a50 0x5574c7347540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.573+0000 7f2d66000640 1 -- 172.21.15.43:0/2334380639 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574c7340550 msgr2=0x5574c7346ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:07.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.573+0000 7f2d66000640 1 --2- 172.21.15.43:0/2334380639 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574c7340550 0x5574c7346ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.573+0000 7f2d66000640 1 -- 172.21.15.43:0/2334380639 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574c70827d0 con 0x5574c7340e70 2024-03-28T22:42:07.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.573+0000 7f2d67400640 1 --2- 172.21.15.43:0/2334380639 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5574c7349a50 0x5574c7347540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:42:07.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.574+0000 7f2d66000640 1 --2- 172.21.15.43:0/2334380639 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5574c7340e70 0x5574c7347000 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2d54012700 tx=0x7f2d54012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=963803931607a71f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:07.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.574+0000 7f2d47e00640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:42:07.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.574+0000 7f2d47e00640 1 -- 172.21.15.43:0/2334380639 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d5400f920 con 0x5574c7340e70 2024-03-28T22:42:07.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.574+0000 7f2d47e00640 1 -- 172.21.15.43:0/2334380639 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f2d5400fac0 con 0x5574c7340e70 2024-03-28T22:42:07.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.574+0000 7f2d70363740 1 -- 172.21.15.43:0/2334380639 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d60003c30 con 0x5574c7340e70 2024-03-28T22:42:07.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.574+0000 7f2d47e00640 1 -- 172.21.15.43:0/2334380639 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2d5400fcc0 con 0x5574c7340e70 2024-03-28T22:42:07.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.574+0000 7f2d70363740 10 client.17274 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:07.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.574+0000 7f2d70363740 1 -- 172.21.15.43:0/2334380639 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5574c7358b90 con 0x5574c7340e70 2024-03-28T22:42:07.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.575+0000 7f2d47e00640 1 -- 172.21.15.43:0/2334380639 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d60003c30 con 0x5574c7340e70 2024-03-28T22:42:07.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.575+0000 7f2d47e00640 1 -- 172.21.15.43:0/2334380639 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f2d5401c420 con 0x5574c7340e70 2024-03-28T22:42:07.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.575+0000 7f2d70363740 10 client.17274 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:07.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.575+0000 7f2d70363740 10 client.17274 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:42:07.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.575+0000 7f2d70363740 20 client.17274 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:07.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.575+0000 7f2d70363740 1 --2- 172.21.15.43:0/2334380639 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5574c735a050 0x5574c735c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.575+0000 7f2d70363740 4 client.17274 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:42:07.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.575+0000 7f2d70363740 1 -- 172.21.15.43:0/2334380639 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5574c724d680 con 0x5574c735a050 2024-03-28T22:42:07.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.575+0000 7f2d67400640 1 --2- 172.21.15.43:0/2334380639 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5574c735a050 0x5574c735c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.576+0000 7f2d67400640 1 --2- 172.21.15.43:0/2334380639 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5574c735a050 0x5574c735c480 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2231002af5407227 server_cookie=cf64ac5bf7504bd1 in_seq=0 out_seq=0 2024-03-28T22:42:07.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.576+0000 7f2d47e00640 10 client.17274 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:42:07.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.577+0000 7f2d47e00640 1 -- 172.21.15.43:0/2334380639 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5574c724d680 con 0x5574c735a050 2024-03-28T22:42:07.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.577+0000 7f2d47e00640 10 client.17274 handle_command_reply: tid=0 2024-03-28T22:42:07.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.577+0000 7f2d70363740 1 client.17274 shutdown 2024-03-28T22:42:07.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.577+0000 7f2d70363740 1 -- 172.21.15.43:0/2334380639 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5574c7340e70 msgr2=0x5574c7347000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:07.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.577+0000 7f2d70363740 1 --2- 172.21.15.43:0/2334380639 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5574c7340e70 0x5574c7347000 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2d54012700 tx=0x7f2d54012bc0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.577+0000 7f2d70363740 1 -- 172.21.15.43:0/2334380639 shutdown_connections 2024-03-28T22:42:07.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.577+0000 7f2d70363740 1 --2- 172.21.15.43:0/2334380639 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5574c7349a50 0x5574c7347540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.577+0000 7f2d70363740 1 --2- 172.21.15.43:0/2334380639 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5574c7340e70 0x5574c7347000 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.577+0000 7f2d70363740 1 --2- 172.21.15.43:0/2334380639 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5574c735a050 0x5574c735c480 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.577+0000 7f2d70363740 1 --2- 172.21.15.43:0/2334380639 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574c7340550 0x5574c7346ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.577+0000 7f2d70363740 1 -- 172.21.15.43:0/2334380639 >> 172.21.15.43:0/2334380639 conn(0x5574c733ac80 msgr2=0x5574c733c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:07.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.577+0000 7f2d47e00640 0 client.17274 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:42:07.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.577+0000 7f2d70363740 1 -- 172.21.15.43:0/2334380639 shutdown_connections 2024-03-28T22:42:07.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.577+0000 7f2d70363740 1 -- 172.21.15.43:0/2334380639 wait complete. 2024-03-28T22:42:07.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.578+0000 7f2d70363740 20 client.17274 trim_cache size 0 max 16384 2024-03-28T22:42:07.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.594+0000 7f2d70363740 1 Processor -- start 2024-03-28T22:42:07.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.594+0000 7f2d70363740 1 -- start start 2024-03-28T22:42:07.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.594+0000 7f2d70363740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5574c7340550 0x5574c73491f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.594+0000 7f2d70363740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574c7340e70 0x5574c73555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.594+0000 7f2d70363740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5574c7349a50 0x5574c7361d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.594+0000 7f2d70363740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2d681b5030 con 0x5574c7340550 2024-03-28T22:42:07.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.594+0000 7f2d70363740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f2d681b6b70 con 0x5574c7340e70 2024-03-28T22:42:07.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.594+0000 7f2d70363740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f2d681b5700 con 0x5574c7349a50 2024-03-28T22:42:07.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.595+0000 7f2d66000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574c7340e70 0x5574c73555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.595+0000 7f2d66000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574c7340e70 0x5574c73555e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44550/0 (socket says 172.21.15.43:44550) 2024-03-28T22:42:07.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.595+0000 7f2d66000640 1 -- 172.21.15.43:0/647228658 learned_addr learned my addr 172.21.15.43:0/647228658 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:07.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.595+0000 7f2d67400640 1 --2- 172.21.15.43:0/647228658 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5574c7340550 0x5574c73491f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.595+0000 7f2d67400640 1 --2- 172.21.15.43:0/647228658 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5574c7349a50 0x5574c7361d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.595+0000 7f2d66000640 1 -- 172.21.15.43:0/647228658 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5574c7349a50 msgr2=0x5574c7361d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:07.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.595+0000 7f2d66000640 1 --2- 172.21.15.43:0/647228658 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5574c7349a50 0x5574c7361d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.595+0000 7f2d66000640 1 -- 172.21.15.43:0/647228658 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5574c7340550 msgr2=0x5574c73491f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:07.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.595+0000 7f2d66000640 1 --2- 172.21.15.43:0/647228658 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5574c7340550 0x5574c73491f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.595+0000 7f2d66000640 1 -- 172.21.15.43:0/647228658 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574c734c1b0 con 0x5574c7340e70 2024-03-28T22:42:07.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.595+0000 7f2d66000640 1 --2- 172.21.15.43:0/647228658 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574c7340e70 0x5574c73555e0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f2d5401ca50 tx=0x7f2d540057d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b3c42bbf59344ff server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:07.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.596+0000 7f2d52a00640 1 -- 172.21.15.43:0/647228658 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d5400fe60 con 0x5574c7340e70 2024-03-28T22:42:07.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.596+0000 7f2d52a00640 1 -- 172.21.15.43:0/647228658 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d5400f920 con 0x5574c7340e70 2024-03-28T22:42:07.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.596+0000 7f2d52a00640 1 -- 172.21.15.43:0/647228658 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d540055e0 con 0x5574c7340e70 2024-03-28T22:42:07.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.596+0000 7f2d70363740 1 -- 172.21.15.43:0/647228658 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574c7340e70 msgr2=0x5574c73555e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:07.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.596+0000 7f2d70363740 1 --2- 172.21.15.43:0/647228658 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574c7340e70 0x5574c73555e0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f2d5401ca50 tx=0x7f2d540057d0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.597+0000 7f2d70363740 1 -- 172.21.15.43:0/647228658 shutdown_connections 2024-03-28T22:42:07.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.597+0000 7f2d70363740 1 --2- 172.21.15.43:0/647228658 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574c7340e70 0x5574c73555e0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.597+0000 7f2d70363740 1 --2- 172.21.15.43:0/647228658 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5574c7349a50 0x5574c7361d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.597+0000 7f2d70363740 1 --2- 172.21.15.43:0/647228658 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5574c7340550 0x5574c73491f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.597+0000 7f2d70363740 1 -- 172.21.15.43:0/647228658 >> 172.21.15.43:0/647228658 conn(0x5574c733ac80 msgr2=0x5574c7346a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:07.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.597+0000 7f2d70363740 1 -- 172.21.15.43:0/647228658 shutdown_connections 2024-03-28T22:42:07.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.597+0000 7f2d70363740 1 -- 172.21.15.43:0/647228658 wait complete. 2024-03-28T22:42:07.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.597+0000 7f2d70363740 1 -- start start 2024-03-28T22:42:07.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.597+0000 7f2d70363740 1 Processor -- start 2024-03-28T22:42:07.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.598+0000 7f2d70363740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5574c7340550 0x5574c734d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.598+0000 7f2d70363740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574c7340e70 0x5574c734dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.598+0000 7f2d70363740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5574c7349a50 0x5574c734e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.598+0000 7f2d70363740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2d681b64a0 con 0x5574c7340550 2024-03-28T22:42:07.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.598+0000 7f2d70363740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f2d681b5dd0 con 0x5574c7340e70 2024-03-28T22:42:07.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.598+0000 7f2d70363740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f2d681b3010 con 0x5574c7349a50 2024-03-28T22:42:07.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.598+0000 7f2d70363740 10 client.0 _ll_register_callbacks cb 0x7f2d54009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:07.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.598+0000 7f2d66000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574c7340e70 0x5574c734dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.598+0000 7f2d67400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5574c7340550 0x5574c734d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.598+0000 7f2d66000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574c7340e70 0x5574c734dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44560/0 (socket says 172.21.15.43:44560) 2024-03-28T22:42:07.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.598+0000 7f2d66000640 1 -- 172.21.15.43:0/227409388 learned_addr learned my addr 172.21.15.43:0/227409388 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:07.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.598+0000 7f2d67400640 1 --2- 172.21.15.43:0/227409388 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5574c7349a50 0x5574c734e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.598+0000 7f2d66000640 1 -- 172.21.15.43:0/227409388 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5574c7349a50 msgr2=0x5574c734e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:07.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.598+0000 7f2d66000640 1 --2- 172.21.15.43:0/227409388 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5574c7349a50 0x5574c734e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.598+0000 7f2d66000640 1 -- 172.21.15.43:0/227409388 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5574c7340550 msgr2=0x5574c734d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:07.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.598+0000 7f2d66000640 1 --2- 172.21.15.43:0/227409388 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5574c7340550 0x5574c734d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.598+0000 7f2d66000640 1 -- 172.21.15.43:0/227409388 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574c73bf660 con 0x5574c7340e70 2024-03-28T22:42:07.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.599+0000 7f2d66000640 1 --2- 172.21.15.43:0/227409388 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574c7340e70 0x5574c734dcd0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f2d54000c00 tx=0x7f2d5401c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=195d25e37f50b9ee server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:07.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.599+0000 7f2d47e00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:42:07.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.599+0000 7f2d47e00640 1 -- 172.21.15.43:0/227409388 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d54037060 con 0x5574c7340e70 2024-03-28T22:42:07.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.599+0000 7f2d70363740 1 -- 172.21.15.43:0/227409388 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d5400f920 con 0x5574c7340e70 2024-03-28T22:42:07.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.599+0000 7f2d47e00640 1 -- 172.21.15.43:0/227409388 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f2d54005530 con 0x5574c7340e70 2024-03-28T22:42:07.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.599+0000 7f2d47e00640 1 -- 172.21.15.43:0/227409388 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2d5403e050 con 0x5574c7340e70 2024-03-28T22:42:07.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.599+0000 7f2d70363740 10 client.27500 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:07.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.599+0000 7f2d70363740 1 -- 172.21.15.43:0/227409388 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5574c73c0940 con 0x5574c7340e70 2024-03-28T22:42:07.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.600+0000 7f2d47e00640 1 -- 172.21.15.43:0/227409388 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d5400f920 con 0x5574c7340e70 2024-03-28T22:42:07.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.600+0000 7f2d47e00640 1 -- 172.21.15.43:0/227409388 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f2d54003cc0 con 0x5574c7340e70 2024-03-28T22:42:07.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.600+0000 7f2d70363740 10 client.27500 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:07.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.600+0000 7f2d70363740 10 client.27500 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:42:07.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.600+0000 7f2d70363740 20 client.27500 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:07.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.600+0000 7f2d70363740 1 --2- 172.21.15.43:0/227409388 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5574c73c0fe0 0x5574c73c3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.600+0000 7f2d70363740 4 client.27500 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:42:07.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.600+0000 7f2d70363740 1 -- 172.21.15.43:0/227409388 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5574c72b1590 con 0x5574c73c0fe0 2024-03-28T22:42:07.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.600+0000 7f2d67400640 1 --2- 172.21.15.43:0/227409388 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5574c73c0fe0 0x5574c73c3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.601+0000 7f2d67400640 1 --2- 172.21.15.43:0/227409388 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5574c73c0fe0 0x5574c73c3400 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f5f69a544a48bc3 server_cookie=6633657d57cbfe35 in_seq=0 out_seq=0 2024-03-28T22:42:07.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.601+0000 7f2d47e00640 10 client.27500 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:42:07.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.601+0000 7f2d47e00640 1 -- 172.21.15.43:0/227409388 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5574c72b1590 con 0x5574c73c0fe0 2024-03-28T22:42:07.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.601+0000 7f2d47e00640 10 client.27500 handle_command_reply: tid=0 2024-03-28T22:42:07.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.601+0000 7f2d70363740 1 client.27500 shutdown 2024-03-28T22:42:07.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.601+0000 7f2d70363740 1 -- 172.21.15.43:0/227409388 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574c7340e70 msgr2=0x5574c734dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:07.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.601+0000 7f2d70363740 1 --2- 172.21.15.43:0/227409388 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574c7340e70 0x5574c734dcd0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f2d54000c00 tx=0x7f2d5401c960 comp rx=0 tx=0).stop 2024-03-28T22:42:07.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.601+0000 7f2d70363740 1 -- 172.21.15.43:0/227409388 shutdown_connections 2024-03-28T22:42:07.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.601+0000 7f2d70363740 1 --2- 172.21.15.43:0/227409388 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5574c73c0fe0 0x5574c73c3400 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.601+0000 7f2d70363740 1 --2- 172.21.15.43:0/227409388 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574c7340e70 0x5574c734dcd0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.601+0000 7f2d70363740 1 --2- 172.21.15.43:0/227409388 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5574c7349a50 0x5574c734e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.601+0000 7f2d70363740 1 --2- 172.21.15.43:0/227409388 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5574c7340550 0x5574c734d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.601+0000 7f2d70363740 1 -- 172.21.15.43:0/227409388 >> 172.21.15.43:0/227409388 conn(0x5574c733ac80 msgr2=0x5574c7339dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:07.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.601+0000 7f2d47e00640 0 client.27500 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:42:07.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.601+0000 7f2d70363740 1 -- 172.21.15.43:0/227409388 shutdown_connections 2024-03-28T22:42:07.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.601+0000 7f2d70363740 1 -- 172.21.15.43:0/227409388 wait complete. 2024-03-28T22:42:07.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.602+0000 7f2d70363740 20 client.27500 trim_cache size 0 max 16384 2024-03-28T22:42:07.604 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:42:07.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.602+0000 7f2d67e00640 1 -- 172.21.15.43:0/387104982 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f2d4c0a1f70 msgr2=0x7f2d4c0a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:07.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.602+0000 7f2d67e00640 1 --2- 172.21.15.43:0/387104982 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f2d4c0a1f70 0x7f2d4c0a4430 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2d580ed380 tx=0x7f2d580f3960 comp rx=0 tx=0).stop 2024-03-28T22:42:07.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.602+0000 7f2d67e00640 1 -- 172.21.15.43:0/387104982 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2d68107f00 msgr2=0x7f2d6819e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:07.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.602+0000 7f2d67e00640 1 --2- 172.21.15.43:0/387104982 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2d68107f00 0x7f2d6819e580 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f2d58006010 tx=0x7f2d5802cb30 comp rx=0 tx=0).stop 2024-03-28T22:42:07.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.602+0000 7f2d67e00640 1 -- 172.21.15.43:0/387104982 shutdown_connections 2024-03-28T22:42:07.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.602+0000 7f2d67e00640 1 --2- 172.21.15.43:0/387104982 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f2d4c0a1f70 0x7f2d4c0a4430 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.602+0000 7f2d67e00640 1 --2- 172.21.15.43:0/387104982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2d681a2e40 0x7f2d681a32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.602+0000 7f2d67e00640 1 --2- 172.21.15.43:0/387104982 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2d6810cf50 0x7f2d6819eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.602+0000 7f2d67e00640 1 --2- 172.21.15.43:0/387104982 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2d68107f00 0x7f2d6819e580 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.602+0000 7f2d67e00640 1 -- 172.21.15.43:0/387104982 >> 172.21.15.43:0/387104982 conn(0x7f2d68077570 msgr2=0x7f2d680791c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:07.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.602+0000 7f2d67e00640 1 -- 172.21.15.43:0/387104982 shutdown_connections 2024-03-28T22:42:07.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.603+0000 7f2d67e00640 1 -- 172.21.15.43:0/387104982 wait complete. 2024-03-28T22:42:07.618 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:42:07.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.894+0000 7f88dfe00640 1 -- 172.21.15.43:0/3939581489 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88d81038f0 msgr2=0x7f88d8103d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:07.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.894+0000 7f88dfe00640 1 --2- 172.21.15.43:0/3939581489 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88d81038f0 0x7f88d8103d70 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f88c4009930 tx=0x7f88c402bc20 comp rx=0 tx=0).stop 2024-03-28T22:42:07.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.894+0000 7f88dfe00640 1 -- 172.21.15.43:0/3939581489 shutdown_connections 2024-03-28T22:42:07.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.894+0000 7f88dfe00640 1 --2- 172.21.15.43:0/3939581489 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f88d81042b0 0x7f88d81107c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.894+0000 7f88dfe00640 1 --2- 172.21.15.43:0/3939581489 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88d81038f0 0x7f88d8103d70 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.894+0000 7f88dfe00640 1 --2- 172.21.15.43:0/3939581489 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88d8109900 0x7f88d8109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.894+0000 7f88dfe00640 1 -- 172.21.15.43:0/3939581489 >> 172.21.15.43:0/3939581489 conn(0x7f88d80ff400 msgr2=0x7f88d8101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:07.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.894+0000 7f88dfe00640 1 -- 172.21.15.43:0/3939581489 shutdown_connections 2024-03-28T22:42:07.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.894+0000 7f88dfe00640 1 -- 172.21.15.43:0/3939581489 wait complete. 2024-03-28T22:42:07.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.895+0000 7f88dfe00640 1 Processor -- start 2024-03-28T22:42:07.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.895+0000 7f88dfe00640 1 -- start start 2024-03-28T22:42:07.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.896+0000 7f88dfe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88d81038f0 0x7f88d819a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.896+0000 7f88dfe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88d81042b0 0x7f88d819a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.896+0000 7f88dfe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f88d8109900 0x7f88d819eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.896+0000 7f88dfe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f88d8113d30 con 0x7f88d81038f0 2024-03-28T22:42:07.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.896+0000 7f88dfe00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f88d8113bb0 con 0x7f88d8109900 2024-03-28T22:42:07.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.896+0000 7f88dfe00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f88d8113eb0 con 0x7f88d81042b0 2024-03-28T22:42:07.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.896+0000 7f88dca00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88d81042b0 0x7f88d819a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.896+0000 7f88dd400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88d81038f0 0x7f88d819a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.896+0000 7f88dde00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f88d8109900 0x7f88d819eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.896+0000 7f88dca00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88d81042b0 0x7f88d819a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:44376/0 (socket says 172.21.15.43:44376) 2024-03-28T22:42:07.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.896+0000 7f88dca00640 1 -- 172.21.15.43:0/471906519 learned_addr learned my addr 172.21.15.43:0/471906519 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:07.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.896+0000 7f88dca00640 1 -- 172.21.15.43:0/471906519 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f88d8109900 msgr2=0x7f88d819eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:07.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.896+0000 7f88dca00640 1 --2- 172.21.15.43:0/471906519 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f88d8109900 0x7f88d819eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.896+0000 7f88dca00640 1 -- 172.21.15.43:0/471906519 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88d81038f0 msgr2=0x7f88d819a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:07.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.896+0000 7f88dca00640 1 --2- 172.21.15.43:0/471906519 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88d81038f0 0x7f88d819a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.897+0000 7f88dca00640 1 -- 172.21.15.43:0/471906519 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88d819f240 con 0x7f88d81042b0 2024-03-28T22:42:07.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.897+0000 7f88dde00640 1 --2- 172.21.15.43:0/471906519 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f88d8109900 0x7f88d819eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:07.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.897+0000 7f88dca00640 1 --2- 172.21.15.43:0/471906519 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88d81042b0 0x7f88d819a790 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f88c4005ab0 tx=0x7f88c4005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:07.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.897+0000 7f88d6000640 1 -- 172.21.15.43:0/471906519 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c4043020 con 0x7f88d81042b0 2024-03-28T22:42:07.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.897+0000 7f88d6000640 1 -- 172.21.15.43:0/471906519 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88c4005000 con 0x7f88d81042b0 2024-03-28T22:42:07.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.897+0000 7f88dfe00640 1 -- 172.21.15.43:0/471906519 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88d819f4d0 con 0x7f88d81042b0 2024-03-28T22:42:07.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.897+0000 7f88d6000640 1 -- 172.21.15.43:0/471906519 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c4003a80 con 0x7f88d81042b0 2024-03-28T22:42:07.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.897+0000 7f88dfe00640 1 -- 172.21.15.43:0/471906519 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88d81a6dc0 con 0x7f88d81042b0 2024-03-28T22:42:07.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.901+0000 7f88d6000640 1 -- 172.21.15.43:0/471906519 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f88c4046020 con 0x7f88d81042b0 2024-03-28T22:42:07.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.901+0000 7f88e2bef740 1 Processor -- start 2024-03-28T22:42:07.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.901+0000 7f88e2bef740 1 -- start start 2024-03-28T22:42:07.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.901+0000 7f88e2bef740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565445af9380 0x565445af9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.901+0000 7f88e2bef740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x565445af9ca0 0x565445b02340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.902+0000 7f88e2bef740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565445b02880 0x565445b04c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.902+0000 7f88e2bef740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x565445a5d050 con 0x565445b02880 2024-03-28T22:42:07.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.902+0000 7f88e2bef740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x565445a05ce0 con 0x565445af9380 2024-03-28T22:42:07.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.902+0000 7f88e2bef740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x565445a590f0 con 0x565445af9ca0 2024-03-28T22:42:07.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.902+0000 7f88dde00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565445af9380 0x565445af9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.902+0000 7f88dd400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x565445af9ca0 0x565445b02340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.902+0000 7f88dde00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565445af9380 0x565445af9760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44590/0 (socket says 172.21.15.43:44590) 2024-03-28T22:42:07.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.902+0000 7f88dde00640 1 -- 172.21.15.43:0/409574904 learned_addr learned my addr 172.21.15.43:0/409574904 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:07.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.902+0000 7f88dca00640 1 --2- 172.21.15.43:0/409574904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565445b02880 0x565445b04c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.902+0000 7f88dd400640 1 -- 172.21.15.43:0/409574904 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565445af9380 msgr2=0x565445af9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:07.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.902+0000 7f88dd400640 1 --2- 172.21.15.43:0/409574904 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565445af9380 0x565445af9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.902+0000 7f88dd400640 1 -- 172.21.15.43:0/409574904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565445b02880 msgr2=0x565445b04c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:07.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.902+0000 7f88dd400640 1 --2- 172.21.15.43:0/409574904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565445b02880 0x565445b04c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.902+0000 7f88dd400640 1 -- 172.21.15.43:0/409574904 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654457713f0 con 0x565445af9ca0 2024-03-28T22:42:07.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.903+0000 7f88dd400640 1 --2- 172.21.15.43:0/409574904 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x565445af9ca0 0x565445b02340 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f88c8009fa0 tx=0x7f88c80118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bfa04aec6aaf7f9d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:07.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.903+0000 7f88ce600640 1 -- 172.21.15.43:0/409574904 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c8003c30 con 0x565445af9ca0 2024-03-28T22:42:07.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.903+0000 7f88ce600640 1 -- 172.21.15.43:0/409574904 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88c8003dd0 con 0x565445af9ca0 2024-03-28T22:42:07.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.903+0000 7f88ce600640 1 -- 172.21.15.43:0/409574904 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c800fb20 con 0x565445af9ca0 2024-03-28T22:42:07.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.903+0000 7f88d6000640 1 --2- 172.21.15.43:0/471906519 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f88bc0a1f70 0x7f88bc0a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.903+0000 7f88d6000640 1 -- 172.21.15.43:0/471906519 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f88c40f0390 con 0x7f88d81042b0 2024-03-28T22:42:07.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.903+0000 7f88dde00640 1 --2- 172.21.15.43:0/471906519 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f88bc0a1f70 0x7f88bc0a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.904+0000 7f88e2bef740 1 -- 172.21.15.43:0/409574904 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x565445af9ca0 msgr2=0x565445b02340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:07.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.904+0000 7f88e2bef740 1 --2- 172.21.15.43:0/409574904 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x565445af9ca0 0x565445b02340 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f88c8009fa0 tx=0x7f88c80118d0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.904+0000 7f88e2bef740 1 -- 172.21.15.43:0/409574904 shutdown_connections 2024-03-28T22:42:07.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.904+0000 7f88e2bef740 1 --2- 172.21.15.43:0/409574904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565445b02880 0x565445b04c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.904+0000 7f88e2bef740 1 --2- 172.21.15.43:0/409574904 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x565445af9ca0 0x565445b02340 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.904+0000 7f88e2bef740 1 --2- 172.21.15.43:0/409574904 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565445af9380 0x565445af9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.904+0000 7f88e2bef740 1 -- 172.21.15.43:0/409574904 >> 172.21.15.43:0/409574904 conn(0x565445af3ab0 msgr2=0x565445af3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:07.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.904+0000 7f88e2bef740 1 -- 172.21.15.43:0/409574904 shutdown_connections 2024-03-28T22:42:07.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.904+0000 7f88e2bef740 1 -- 172.21.15.43:0/409574904 wait complete. 2024-03-28T22:42:07.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.904+0000 7f88dde00640 1 --2- 172.21.15.43:0/471906519 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f88bc0a1f70 0x7f88bc0a4430 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f88d0004650 tx=0x7f88d0005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:07.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.905+0000 7f88e2bef740 1 -- start start 2024-03-28T22:42:07.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.905+0000 7f88e2bef740 1 Processor -- start 2024-03-28T22:42:07.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.906+0000 7f88e2bef740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565445af9380 0x565445aff8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.906+0000 7f88e2bef740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565445af9ca0 0x565445affde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.906+0000 7f88e2bef740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x565445b02880 0x565445b00320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.906+0000 7f88e2bef740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x565445b05670 con 0x565445af9380 2024-03-28T22:42:07.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.906+0000 7f88e2bef740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5654459fb2f0 con 0x565445af9ca0 2024-03-28T22:42:07.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.906+0000 7f88e2bef740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x565445b057f0 con 0x565445b02880 2024-03-28T22:42:07.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.906+0000 7f88e2bef740 10 client.0 _ll_register_callbacks cb 0x565445770be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:07.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.906+0000 7f88dca00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565445af9ca0 0x565445affde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.906+0000 7f88dca00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565445af9ca0 0x565445affde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44606/0 (socket says 172.21.15.43:44606) 2024-03-28T22:42:07.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.906+0000 7f88dca00640 1 -- 172.21.15.43:0/3841223675 learned_addr learned my addr 172.21.15.43:0/3841223675 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:07.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.906+0000 7f88dde00640 1 --2- 172.21.15.43:0/3841223675 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x565445b02880 0x565445b00320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.906+0000 7f88dd400640 1 --2- 172.21.15.43:0/3841223675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565445af9380 0x565445aff8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.906+0000 7f88dca00640 1 -- 172.21.15.43:0/3841223675 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x565445b02880 msgr2=0x565445b00320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:07.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.906+0000 7f88dca00640 1 --2- 172.21.15.43:0/3841223675 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x565445b02880 0x565445b00320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.906+0000 7f88dca00640 1 -- 172.21.15.43:0/3841223675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565445af9380 msgr2=0x565445aff8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:07.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.906+0000 7f88dca00640 1 --2- 172.21.15.43:0/3841223675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565445af9380 0x565445aff8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.906+0000 7f88dca00640 1 -- 172.21.15.43:0/3841223675 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56544583b7d0 con 0x565445af9ca0 2024-03-28T22:42:07.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.907+0000 7f88dca00640 1 --2- 172.21.15.43:0/3841223675 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565445af9ca0 0x565445affde0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f88c4006010 tx=0x7f88c40bc920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31c8bb861bfc7d19 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:07.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.907+0000 7f88c2a00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:42:07.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.907+0000 7f88c2a00640 1 -- 172.21.15.43:0/3841223675 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c40bca90 con 0x565445af9ca0 2024-03-28T22:42:07.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.907+0000 7f88c2a00640 1 -- 172.21.15.43:0/3841223675 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f88c40054e0 con 0x565445af9ca0 2024-03-28T22:42:07.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.907+0000 7f88e2bef740 1 -- 172.21.15.43:0/3841223675 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88c8003dd0 con 0x565445af9ca0 2024-03-28T22:42:07.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.907+0000 7f88c2a00640 1 -- 172.21.15.43:0/3841223675 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f88c40c0be0 con 0x565445af9ca0 2024-03-28T22:42:07.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.907+0000 7f88e2bef740 10 client.27512 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:07.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.907+0000 7f88e2bef740 1 -- 172.21.15.43:0/3841223675 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x565445b11950 con 0x565445af9ca0 2024-03-28T22:42:07.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.908+0000 7f88c2a00640 1 -- 172.21.15.43:0/3841223675 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88c8003dd0 con 0x565445af9ca0 2024-03-28T22:42:07.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.908+0000 7f88c2a00640 1 -- 172.21.15.43:0/3841223675 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f88c40bccf0 con 0x565445af9ca0 2024-03-28T22:42:07.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.908+0000 7f88e2bef740 10 client.27512 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:07.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.908+0000 7f88e2bef740 10 client.27512 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:42:07.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.908+0000 7f88e2bef740 20 client.27512 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:07.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.908+0000 7f88e2bef740 1 --2- 172.21.15.43:0/3841223675 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x565445b12d40 0x565445b15160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.908+0000 7f88e2bef740 4 client.27512 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:42:07.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.908+0000 7f88e2bef740 1 -- 172.21.15.43:0/3841223675 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565445a06680 con 0x565445b12d40 2024-03-28T22:42:07.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.908+0000 7f88dd400640 1 --2- 172.21.15.43:0/3841223675 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x565445b12d40 0x565445b15160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.909+0000 7f88dd400640 1 --2- 172.21.15.43:0/3841223675 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x565445b12d40 0x565445b15160 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5180f87c8e71a8b server_cookie=f3b77fc2d96f837f in_seq=0 out_seq=0 2024-03-28T22:42:07.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.909+0000 7f88c2a00640 10 client.27512 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:42:07.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.910+0000 7f88c2a00640 1 -- 172.21.15.43:0/3841223675 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x565445a06680 con 0x565445b12d40 2024-03-28T22:42:07.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.910+0000 7f88c2a00640 10 client.27512 handle_command_reply: tid=0 2024-03-28T22:42:07.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.910+0000 7f88e2bef740 1 client.27512 shutdown 2024-03-28T22:42:07.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.910+0000 7f88e2bef740 1 -- 172.21.15.43:0/3841223675 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565445af9ca0 msgr2=0x565445affde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:07.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.910+0000 7f88e2bef740 1 --2- 172.21.15.43:0/3841223675 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565445af9ca0 0x565445affde0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f88c4006010 tx=0x7f88c40bc920 comp rx=0 tx=0).stop 2024-03-28T22:42:07.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.910+0000 7f88e2bef740 1 -- 172.21.15.43:0/3841223675 shutdown_connections 2024-03-28T22:42:07.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.910+0000 7f88e2bef740 1 --2- 172.21.15.43:0/3841223675 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x565445b12d40 0x565445b15160 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.910+0000 7f88e2bef740 1 --2- 172.21.15.43:0/3841223675 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565445af9ca0 0x565445affde0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.910+0000 7f88e2bef740 1 --2- 172.21.15.43:0/3841223675 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x565445b02880 0x565445b00320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.910+0000 7f88e2bef740 1 --2- 172.21.15.43:0/3841223675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565445af9380 0x565445aff8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.910+0000 7f88c2a00640 0 client.27512 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:42:07.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.910+0000 7f88e2bef740 1 -- 172.21.15.43:0/3841223675 >> 172.21.15.43:0/3841223675 conn(0x565445af3ab0 msgr2=0x565445af5800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:07.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.910+0000 7f88e2bef740 1 -- 172.21.15.43:0/3841223675 shutdown_connections 2024-03-28T22:42:07.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.910+0000 7f88e2bef740 1 -- 172.21.15.43:0/3841223675 wait complete. 2024-03-28T22:42:07.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.910+0000 7f88e2bef740 20 client.27512 trim_cache size 0 max 16384 2024-03-28T22:42:07.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.926+0000 7f88e2bef740 1 Processor -- start 2024-03-28T22:42:07.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.926+0000 7f88e2bef740 1 -- start start 2024-03-28T22:42:07.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.927+0000 7f88e2bef740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565445af9380 0x565445b0e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.927+0000 7f88e2bef740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565445af9ca0 0x565445b00140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.927+0000 7f88e2bef740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x565445b02880 0x565445b1aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.927+0000 7f88e2bef740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f88d81b0d00 con 0x565445af9380 2024-03-28T22:42:07.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.927+0000 7f88e2bef740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f88d81b2840 con 0x565445af9ca0 2024-03-28T22:42:07.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.927+0000 7f88e2bef740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f88d81b13d0 con 0x565445b02880 2024-03-28T22:42:07.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.927+0000 7f88dde00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x565445b02880 0x565445b1aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.927+0000 7f88dca00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565445af9ca0 0x565445b00140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.927+0000 7f88dd400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565445af9380 0x565445b0e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.927+0000 7f88dca00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565445af9ca0 0x565445b00140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44612/0 (socket says 172.21.15.43:44612) 2024-03-28T22:42:07.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.927+0000 7f88dca00640 1 -- 172.21.15.43:0/1895367955 learned_addr learned my addr 172.21.15.43:0/1895367955 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:07.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.927+0000 7f88dde00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x565445b02880 0x565445b1aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:44412/0 (socket says 172.21.15.43:44412) 2024-03-28T22:42:07.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.928+0000 7f88dca00640 1 -- 172.21.15.43:0/1895367955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x565445b02880 msgr2=0x565445b1aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:07.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.928+0000 7f88dca00640 1 --2- 172.21.15.43:0/1895367955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x565445b02880 0x565445b1aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.928+0000 7f88dca00640 1 -- 172.21.15.43:0/1895367955 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565445af9380 msgr2=0x565445b0e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:07.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.928+0000 7f88dca00640 1 --2- 172.21.15.43:0/1895367955 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565445af9380 0x565445b0e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.928+0000 7f88dca00640 1 -- 172.21.15.43:0/1895367955 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565445b04fa0 con 0x565445af9ca0 2024-03-28T22:42:07.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.928+0000 7f88dd400640 1 --2- 172.21.15.43:0/1895367955 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565445af9380 0x565445b0e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:07.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.928+0000 7f88dca00640 1 --2- 172.21.15.43:0/1895367955 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565445af9ca0 0x565445b00140 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f88c40b8ef0 tx=0x7f88c40f0610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88940af9bc6b3042 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:07.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.928+0000 7f88cdc00640 1 -- 172.21.15.43:0/1895367955 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c40bca90 con 0x565445af9ca0 2024-03-28T22:42:07.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.928+0000 7f88cdc00640 1 -- 172.21.15.43:0/1895367955 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88c40c5230 con 0x565445af9ca0 2024-03-28T22:42:07.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.928+0000 7f88cdc00640 1 -- 172.21.15.43:0/1895367955 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c40c4df0 con 0x565445af9ca0 2024-03-28T22:42:07.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.929+0000 7f88e2bef740 1 -- 172.21.15.43:0/1895367955 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565445af9ca0 msgr2=0x565445b00140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:07.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.929+0000 7f88e2bef740 1 --2- 172.21.15.43:0/1895367955 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565445af9ca0 0x565445b00140 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f88c40b8ef0 tx=0x7f88c40f0610 comp rx=0 tx=0).stop 2024-03-28T22:42:07.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.929+0000 7f88e2bef740 1 -- 172.21.15.43:0/1895367955 shutdown_connections 2024-03-28T22:42:07.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.929+0000 7f88e2bef740 1 --2- 172.21.15.43:0/1895367955 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565445af9ca0 0x565445b00140 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.929+0000 7f88e2bef740 1 --2- 172.21.15.43:0/1895367955 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x565445b02880 0x565445b1aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.929+0000 7f88e2bef740 1 --2- 172.21.15.43:0/1895367955 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565445af9380 0x565445b0e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.929+0000 7f88e2bef740 1 -- 172.21.15.43:0/1895367955 >> 172.21.15.43:0/1895367955 conn(0x565445af3ab0 msgr2=0x565445af22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:07.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.929+0000 7f88e2bef740 1 -- 172.21.15.43:0/1895367955 shutdown_connections 2024-03-28T22:42:07.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.929+0000 7f88e2bef740 1 -- 172.21.15.43:0/1895367955 wait complete. 2024-03-28T22:42:07.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.930+0000 7f88e2bef740 1 -- start start 2024-03-28T22:42:07.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.930+0000 7f88e2bef740 1 Processor -- start 2024-03-28T22:42:07.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.930+0000 7f88e2bef740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565445af9380 0x565445b0ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.930+0000 7f88e2bef740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x565445af9ca0 0x565445afc110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.930+0000 7f88e2bef740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565445b02880 0x565445afc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.930+0000 7f88e2bef740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f88d81b2170 con 0x565445b02880 2024-03-28T22:42:07.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.930+0000 7f88e2bef740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f88d81b1aa0 con 0x565445af9380 2024-03-28T22:42:07.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.930+0000 7f88e2bef740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f88d81aece0 con 0x565445af9ca0 2024-03-28T22:42:07.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.930+0000 7f88e2bef740 10 client.0 _ll_register_callbacks cb 0x7f88c4009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:07.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.930+0000 7f88dd400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565445af9380 0x565445b0ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.931+0000 7f88dd400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565445af9380 0x565445b0ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44620/0 (socket says 172.21.15.43:44620) 2024-03-28T22:42:07.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.931+0000 7f88dd400640 1 -- 172.21.15.43:0/3875575755 learned_addr learned my addr 172.21.15.43:0/3875575755 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:07.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.931+0000 7f88dca00640 1 --2- 172.21.15.43:0/3875575755 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x565445af9ca0 0x565445afc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.931+0000 7f88dde00640 1 --2- 172.21.15.43:0/3875575755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565445b02880 0x565445afc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.931+0000 7f88dd400640 1 -- 172.21.15.43:0/3875575755 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x565445af9ca0 msgr2=0x565445afc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:07.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.931+0000 7f88dd400640 1 --2- 172.21.15.43:0/3875575755 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x565445af9ca0 0x565445afc110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.931+0000 7f88dd400640 1 -- 172.21.15.43:0/3875575755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565445b02880 msgr2=0x565445afc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:07.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.931+0000 7f88dd400640 1 --2- 172.21.15.43:0/3875575755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565445b02880 0x565445afc650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.931+0000 7f88dd400640 1 -- 172.21.15.43:0/3875575755 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565445afce80 con 0x565445af9380 2024-03-28T22:42:07.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.931+0000 7f88dd400640 1 --2- 172.21.15.43:0/3875575755 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565445af9380 0x565445b0ac90 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f88c8009d60 tx=0x7f88c800f6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f2614e41db47029 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:07.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.931+0000 7f88c2a00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:42:07.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.932+0000 7f88c2a00640 1 -- 172.21.15.43:0/3875575755 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c8003d30 con 0x565445af9380 2024-03-28T22:42:07.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.932+0000 7f88c2a00640 1 -- 172.21.15.43:0/3875575755 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f88c800f730 con 0x565445af9380 2024-03-28T22:42:07.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.932+0000 7f88e2bef740 1 -- 172.21.15.43:0/3875575755 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88c40c5230 con 0x565445af9380 2024-03-28T22:42:07.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.932+0000 7f88c2a00640 1 -- 172.21.15.43:0/3875575755 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f88c800f950 con 0x565445af9380 2024-03-28T22:42:07.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.932+0000 7f88e2bef740 10 client.27518 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:07.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.932+0000 7f88e2bef740 1 -- 172.21.15.43:0/3875575755 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x565445b78f50 con 0x565445af9380 2024-03-28T22:42:07.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.932+0000 7f88c2a00640 1 -- 172.21.15.43:0/3875575755 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88c40c5230 con 0x565445af9380 2024-03-28T22:42:07.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.932+0000 7f88c2a00640 1 -- 172.21.15.43:0/3875575755 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f88c8005770 con 0x565445af9380 2024-03-28T22:42:07.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.933+0000 7f88e2bef740 10 client.27518 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:07.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.933+0000 7f88e2bef740 10 client.27518 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:42:07.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.933+0000 7f88e2bef740 20 client.27518 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:07.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.933+0000 7f88e2bef740 1 --2- 172.21.15.43:0/3875575755 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x565445b794e0 0x565445b7b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:07.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.933+0000 7f88e2bef740 4 client.27518 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:42:07.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.933+0000 7f88e2bef740 1 -- 172.21.15.43:0/3875575755 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "damage ls"}) -- 0x565445a6a590 con 0x565445b794e0 2024-03-28T22:42:07.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.933+0000 7f88dca00640 1 --2- 172.21.15.43:0/3875575755 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x565445b794e0 0x565445b7b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:07.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.934+0000 7f88dca00640 1 --2- 172.21.15.43:0/3875575755 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x565445b794e0 0x565445b7b900 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=10d9ba70c9580a9a server_cookie=28bfdcd9f57a121d in_seq=0 out_seq=0 2024-03-28T22:42:07.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.934+0000 7f88c2a00640 10 client.27518 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:42:07.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.934+0000 7f88c2a00640 1 -- 172.21.15.43:0/3875575755 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x565445a6a590 con 0x565445b794e0 2024-03-28T22:42:07.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.934+0000 7f88c2a00640 10 client.27518 handle_command_reply: tid=0 2024-03-28T22:42:07.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.934+0000 7f88e2bef740 1 client.27518 shutdown 2024-03-28T22:42:07.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.934+0000 7f88e2bef740 1 -- 172.21.15.43:0/3875575755 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565445af9380 msgr2=0x565445b0ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:07.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.934+0000 7f88e2bef740 1 --2- 172.21.15.43:0/3875575755 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565445af9380 0x565445b0ac90 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f88c8009d60 tx=0x7f88c800f6c0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.934+0000 7f88e2bef740 1 -- 172.21.15.43:0/3875575755 shutdown_connections 2024-03-28T22:42:07.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.934+0000 7f88e2bef740 1 --2- 172.21.15.43:0/3875575755 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x565445b794e0 0x565445b7b900 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.934+0000 7f88e2bef740 1 --2- 172.21.15.43:0/3875575755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x565445b02880 0x565445afc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.934+0000 7f88e2bef740 1 --2- 172.21.15.43:0/3875575755 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x565445af9ca0 0x565445afc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.934+0000 7f88e2bef740 1 --2- 172.21.15.43:0/3875575755 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565445af9380 0x565445b0ac90 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.934+0000 7f88c2a00640 0 client.27518 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:42:07.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.934+0000 7f88e2bef740 1 -- 172.21.15.43:0/3875575755 >> 172.21.15.43:0/3875575755 conn(0x565445af3ab0 msgr2=0x565445af2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:07.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.935+0000 7f88e2bef740 1 -- 172.21.15.43:0/3875575755 shutdown_connections 2024-03-28T22:42:07.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.935+0000 7f88e2bef740 1 -- 172.21.15.43:0/3875575755 wait complete. 2024-03-28T22:42:07.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.935+0000 7f88e2bef740 20 client.27518 trim_cache size 0 max 16384 2024-03-28T22:42:07.936 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:42:07.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.936+0000 7f88dfe00640 1 -- 172.21.15.43:0/471906519 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f88bc0a1f70 msgr2=0x7f88bc0a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:07.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.936+0000 7f88dfe00640 1 --2- 172.21.15.43:0/471906519 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f88bc0a1f70 0x7f88bc0a4430 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f88d0004650 tx=0x7f88d0005f90 comp rx=0 tx=0).stop 2024-03-28T22:42:07.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.936+0000 7f88dfe00640 1 -- 172.21.15.43:0/471906519 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88d81042b0 msgr2=0x7f88d819a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:07.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.936+0000 7f88dfe00640 1 --2- 172.21.15.43:0/471906519 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88d81042b0 0x7f88d819a790 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f88c4005ab0 tx=0x7f88c4005e80 comp rx=0 tx=0).stop 2024-03-28T22:42:07.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.936+0000 7f88dfe00640 1 -- 172.21.15.43:0/471906519 shutdown_connections 2024-03-28T22:42:07.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.936+0000 7f88dfe00640 1 --2- 172.21.15.43:0/471906519 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f88bc0a1f70 0x7f88bc0a4430 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.936+0000 7f88dfe00640 1 --2- 172.21.15.43:0/471906519 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f88d8109900 0x7f88d819eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.936+0000 7f88dfe00640 1 --2- 172.21.15.43:0/471906519 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f88d81042b0 0x7f88d819a790 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.936+0000 7f88dfe00640 1 --2- 172.21.15.43:0/471906519 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88d81038f0 0x7f88d819a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:07.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.936+0000 7f88dfe00640 1 -- 172.21.15.43:0/471906519 >> 172.21.15.43:0/471906519 conn(0x7f88d80ff400 msgr2=0x7f88d80ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:07.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.936+0000 7f88dfe00640 1 -- 172.21.15.43:0/471906519 shutdown_connections 2024-03-28T22:42:07.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:07.936+0000 7f88dfe00640 1 -- 172.21.15.43:0/471906519 wait complete. 2024-03-28T22:42:07.954 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:42:08.042 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.043+0000 7ff49de00640 1 -- 172.21.15.43:0/2043365062 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff4981042e0 msgr2=0x7ff4981107f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:08.042 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.043+0000 7ff49de00640 1 --2- 172.21.15.43:0/2043365062 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff4981042e0 0x7ff4981107f0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7ff490009ca0 tx=0x7ff49002c100 comp rx=0 tx=0).stop 2024-03-28T22:42:08.042 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.043+0000 7ff49de00640 1 -- 172.21.15.43:0/2043365062 shutdown_connections 2024-03-28T22:42:08.043 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.043+0000 7ff49de00640 1 --2- 172.21.15.43:0/2043365062 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff4981042e0 0x7ff4981107f0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.043 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.043+0000 7ff49de00640 1 --2- 172.21.15.43:0/2043365062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff498103920 0x7ff498103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.043 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.043+0000 7ff49de00640 1 --2- 172.21.15.43:0/2043365062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff498109930 0x7ff498109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.043 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.043+0000 7ff49de00640 1 -- 172.21.15.43:0/2043365062 >> 172.21.15.43:0/2043365062 conn(0x7ff4980ff410 msgr2=0x7ff498101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:08.043 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.043+0000 7ff49de00640 1 -- 172.21.15.43:0/2043365062 shutdown_connections 2024-03-28T22:42:08.043 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.043+0000 7ff49de00640 1 -- 172.21.15.43:0/2043365062 wait complete. 2024-03-28T22:42:08.045 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.044+0000 7ff49de00640 1 Processor -- start 2024-03-28T22:42:08.045 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.044+0000 7ff49de00640 1 -- start start 2024-03-28T22:42:08.045 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.045+0000 7ff49de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff498103920 0x7ff49819a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.045 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.045+0000 7ff49de00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4981042e0 0x7ff49819a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.045 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.045+0000 7ff49de00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff498109930 0x7ff49819e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.045 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.045+0000 7ff49de00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff498113d40 con 0x7ff498103920 2024-03-28T22:42:08.045 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.045+0000 7ff49de00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ff498113bc0 con 0x7ff498109930 2024-03-28T22:42:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.045+0000 7ff49de00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7ff498113ec0 con 0x7ff4981042e0 2024-03-28T22:42:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.045+0000 7ff496000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4981042e0 0x7ff49819a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.045+0000 7ff496a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff498103920 0x7ff49819a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.045+0000 7ff496000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4981042e0 0x7ff49819a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:44436/0 (socket says 172.21.15.43:44436) 2024-03-28T22:42:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.045+0000 7ff496000640 1 -- 172.21.15.43:0/3126971427 learned_addr learned my addr 172.21.15.43:0/3126971427 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.045+0000 7ff497400640 1 --2- 172.21.15.43:0/3126971427 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff498109930 0x7ff49819e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.046+0000 7ff496000640 1 -- 172.21.15.43:0/3126971427 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff498109930 msgr2=0x7ff49819e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.046+0000 7ff496000640 1 --2- 172.21.15.43:0/3126971427 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff498109930 0x7ff49819e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.046+0000 7ff496000640 1 -- 172.21.15.43:0/3126971427 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff498103920 msgr2=0x7ff49819a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.046+0000 7ff496000640 1 --2- 172.21.15.43:0/3126971427 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff498103920 0x7ff49819a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.046+0000 7ff496000640 1 -- 172.21.15.43:0/3126971427 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff49819f110 con 0x7ff4981042e0 2024-03-28T22:42:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.046+0000 7ff496000640 1 --2- 172.21.15.43:0/3126971427 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4981042e0 0x7ff49819a660 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff48400e9f0 tx=0x7ff48400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:08.046 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.046+0000 7ff49ca00640 1 -- 172.21.15.43:0/3126971427 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff48400cd80 con 0x7ff4981042e0 2024-03-28T22:42:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.046+0000 7ff49de00640 1 -- 172.21.15.43:0/3126971427 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff49819f400 con 0x7ff4981042e0 2024-03-28T22:42:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.046+0000 7ff49ca00640 1 -- 172.21.15.43:0/3126971427 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff484005500 con 0x7ff4981042e0 2024-03-28T22:42:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.046+0000 7ff49ca00640 1 -- 172.21.15.43:0/3126971427 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff484010690 con 0x7ff4981042e0 2024-03-28T22:42:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.046+0000 7ff49de00640 1 -- 172.21.15.43:0/3126971427 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4981a6c90 con 0x7ff4981042e0 2024-03-28T22:42:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.050+0000 7ff4a0bf5740 1 Processor -- start 2024-03-28T22:42:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.050+0000 7ff4a0bf5740 1 -- start start 2024-03-28T22:42:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.050+0000 7ff4a0bf5740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5582926743a0 0x558292674780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.050+0000 7ff4a0bf5740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558292674cc0 0x55829267d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.050+0000 7ff4a0bf5740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55829267da90 0x55829267fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.050+0000 7ff4a0bf5740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5582925d7640 con 0x558292674cc0 2024-03-28T22:42:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.050+0000 7ff4a0bf5740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x558292580810 con 0x55829267da90 2024-03-28T22:42:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.050+0000 7ff4a0bf5740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5582925d36e0 con 0x5582926743a0 2024-03-28T22:42:08.050 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.051+0000 7ff49ca00640 1 -- 172.21.15.43:0/3126971427 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7ff4840046c0 con 0x7ff4981042e0 2024-03-28T22:42:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.051+0000 7ff497400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5582926743a0 0x558292674780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.051+0000 7ff497400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5582926743a0 0x558292674780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:44448/0 (socket says 172.21.15.43:44448) 2024-03-28T22:42:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.051+0000 7ff497400640 1 -- 172.21.15.43:0/550168685 learned_addr learned my addr 172.21.15.43:0/550168685 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.051+0000 7ff496a00640 1 --2- 172.21.15.43:0/550168685 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558292674cc0 0x55829267d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.051+0000 7ff496000640 1 --2- 172.21.15.43:0/550168685 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55829267da90 0x55829267fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.051+0000 7ff497400640 1 -- 172.21.15.43:0/550168685 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55829267da90 msgr2=0x55829267fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.051+0000 7ff497400640 1 --2- 172.21.15.43:0/550168685 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55829267da90 0x55829267fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.051+0000 7ff497400640 1 -- 172.21.15.43:0/550168685 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558292674cc0 msgr2=0x55829267d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.051+0000 7ff497400640 1 --2- 172.21.15.43:0/550168685 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558292674cc0 0x55829267d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.051+0000 7ff497400640 1 -- 172.21.15.43:0/550168685 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582922ec210 con 0x5582926743a0 2024-03-28T22:42:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.052+0000 7ff497400640 1 --2- 172.21.15.43:0/550168685 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5582926743a0 0x558292674780 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ff49002c710 tx=0x7ff490005f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a5618222b3e04bc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.052+0000 7ff483e00640 1 -- 172.21.15.43:0/550168685 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff490045020 con 0x5582926743a0 2024-03-28T22:42:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.052+0000 7ff483e00640 1 -- 172.21.15.43:0/550168685 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff49003a040 con 0x5582926743a0 2024-03-28T22:42:08.051 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.052+0000 7ff483e00640 1 -- 172.21.15.43:0/550168685 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff490039040 con 0x5582926743a0 2024-03-28T22:42:08.052 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.052+0000 7ff49ca00640 1 --2- 172.21.15.43:0/3126971427 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ff47c0a1f90 0x7ff47c0a4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.055 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.053+0000 7ff49ca00640 1 -- 172.21.15.43:0/3126971427 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7ff4840ce9b0 con 0x7ff4981042e0 2024-03-28T22:42:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.053+0000 7ff496a00640 1 --2- 172.21.15.43:0/3126971427 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ff47c0a1f90 0x7ff47c0a4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.053+0000 7ff4a0bf5740 1 -- 172.21.15.43:0/550168685 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5582926743a0 msgr2=0x558292674780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.053+0000 7ff4a0bf5740 1 --2- 172.21.15.43:0/550168685 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5582926743a0 0x558292674780 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ff49002c710 tx=0x7ff490005f80 comp rx=0 tx=0).stop 2024-03-28T22:42:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.053+0000 7ff4a0bf5740 1 -- 172.21.15.43:0/550168685 shutdown_connections 2024-03-28T22:42:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.053+0000 7ff4a0bf5740 1 --2- 172.21.15.43:0/550168685 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55829267da90 0x55829267fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.053+0000 7ff4a0bf5740 1 --2- 172.21.15.43:0/550168685 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558292674cc0 0x55829267d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.053+0000 7ff4a0bf5740 1 --2- 172.21.15.43:0/550168685 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5582926743a0 0x558292674780 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.053+0000 7ff4a0bf5740 1 -- 172.21.15.43:0/550168685 >> 172.21.15.43:0/550168685 conn(0x55829266ecc0 msgr2=0x55829266f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.053+0000 7ff4a0bf5740 1 -- 172.21.15.43:0/550168685 shutdown_connections 2024-03-28T22:42:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.053+0000 7ff4a0bf5740 1 -- 172.21.15.43:0/550168685 wait complete. 2024-03-28T22:42:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.053+0000 7ff496a00640 1 --2- 172.21.15.43:0/3126971427 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ff47c0a1f90 0x7ff47c0a4450 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x558292673e50 tx=0x7ff488005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.054+0000 7ff4a0bf5740 1 -- start start 2024-03-28T22:42:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.054+0000 7ff4a0bf5740 1 Processor -- start 2024-03-28T22:42:08.056 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.055+0000 7ff4a0bf5740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5582926743a0 0x55829267aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.057 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.055+0000 7ff4a0bf5740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558292674cc0 0x55829267b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.057 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.055+0000 7ff4a0bf5740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55829267da90 0x55829267b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.057 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.055+0000 7ff4a0bf5740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x558292680660 con 0x55829267da90 2024-03-28T22:42:08.057 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.055+0000 7ff4a0bf5740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x558292575d00 con 0x558292674cc0 2024-03-28T22:42:08.057 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.055+0000 7ff4a0bf5740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5582926807e0 con 0x5582926743a0 2024-03-28T22:42:08.057 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.055+0000 7ff4a0bf5740 10 client.0 _ll_register_callbacks cb 0x558292560490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:08.057 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.055+0000 7ff496a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5582926743a0 0x55829267aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.057 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.055+0000 7ff496000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558292674cc0 0x55829267b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.057 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.055+0000 7ff496a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5582926743a0 0x55829267aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:44456/0 (socket says 172.21.15.43:44456) 2024-03-28T22:42:08.057 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.055+0000 7ff496a00640 1 -- 172.21.15.43:0/4251657062 learned_addr learned my addr 172.21.15.43:0/4251657062 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:08.057 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.055+0000 7ff496000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558292674cc0 0x55829267b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44654/0 (socket says 172.21.15.43:44654) 2024-03-28T22:42:08.057 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.055+0000 7ff497400640 1 --2- 172.21.15.43:0/4251657062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55829267da90 0x55829267b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.057 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.055+0000 7ff496a00640 1 -- 172.21.15.43:0/4251657062 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558292674cc0 msgr2=0x55829267b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:08.057 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.055+0000 7ff496a00640 1 --2- 172.21.15.43:0/4251657062 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558292674cc0 0x55829267b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.058 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.055+0000 7ff496a00640 1 -- 172.21.15.43:0/4251657062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55829267da90 msgr2=0x55829267b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:08.058 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.055+0000 7ff496a00640 1 --2- 172.21.15.43:0/4251657062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55829267da90 0x55829267b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.058 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.055+0000 7ff496a00640 1 -- 172.21.15.43:0/4251657062 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582923b8620 con 0x5582926743a0 2024-03-28T22:42:08.058 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.055+0000 7ff497400640 1 --2- 172.21.15.43:0/4251657062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55829267da90 0x55829267b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:42:08.058 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.056+0000 7ff496000640 1 --2- 172.21.15.43:0/4251657062 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558292674cc0 0x55829267b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:08.058 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.056+0000 7ff496a00640 1 --2- 172.21.15.43:0/4251657062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5582926743a0 0x55829267aaf0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7ff488009d40 tx=0x7ff488004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e45261e627ecbad2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:08.058 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.056+0000 7ff480c00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:42:08.058 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.056+0000 7ff480c00640 1 -- 172.21.15.43:0/4251657062 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff48801f640 con 0x5582926743a0 2024-03-28T22:42:08.058 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.056+0000 7ff480c00640 1 -- 172.21.15.43:0/4251657062 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7ff48801f7e0 con 0x5582926743a0 2024-03-28T22:42:08.058 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.056+0000 7ff4a0bf5740 1 -- 172.21.15.43:0/4251657062 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff49003a040 con 0x5582926743a0 2024-03-28T22:42:08.058 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.056+0000 7ff480c00640 1 -- 172.21.15.43:0/4251657062 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff488032050 con 0x5582926743a0 2024-03-28T22:42:08.058 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.056+0000 7ff4a0bf5740 10 client.27478 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:08.058 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.056+0000 7ff4a0bf5740 1 -- 172.21.15.43:0/4251657062 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55829268c5c0 con 0x5582926743a0 2024-03-28T22:42:08.058 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.057+0000 7ff480c00640 1 -- 172.21.15.43:0/4251657062 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff49003a040 con 0x5582926743a0 2024-03-28T22:42:08.059 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.057+0000 7ff480c00640 1 -- 172.21.15.43:0/4251657062 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7ff488027020 con 0x5582926743a0 2024-03-28T22:42:08.059 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.057+0000 7ff4a0bf5740 10 client.27478 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:08.059 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.057+0000 7ff4a0bf5740 10 client.27478 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:42:08.059 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.057+0000 7ff4a0bf5740 20 client.27478 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:08.059 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.057+0000 7ff4a0bf5740 1 --2- 172.21.15.43:0/4251657062 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55829268da80 0x55829268fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.059 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.057+0000 7ff4a0bf5740 4 client.27478 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:42:08.059 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.057+0000 7ff4a0bf5740 1 -- 172.21.15.43:0/4251657062 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5582925eb030 con 0x55829268da80 2024-03-28T22:42:08.060 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.057+0000 7ff496000640 1 --2- 172.21.15.43:0/4251657062 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55829268da80 0x55829268fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.060 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.058+0000 7ff496000640 1 --2- 172.21.15.43:0/4251657062 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55829268da80 0x55829268fed0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54b539c0b8f1d247 server_cookie=72507783386c9404 in_seq=0 out_seq=0 2024-03-28T22:42:08.060 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.058+0000 7ff480c00640 10 client.27478 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.059+0000 7ff480c00640 1 -- 172.21.15.43:0/4251657062 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5582925eb030 con 0x55829268da80 2024-03-28T22:42:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.059+0000 7ff480c00640 10 client.27478 handle_command_reply: tid=0 2024-03-28T22:42:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.059+0000 7ff4a0bf5740 1 client.27478 shutdown 2024-03-28T22:42:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.059+0000 7ff4a0bf5740 1 -- 172.21.15.43:0/4251657062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5582926743a0 msgr2=0x55829267aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.059+0000 7ff4a0bf5740 1 --2- 172.21.15.43:0/4251657062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5582926743a0 0x55829267aaf0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7ff488009d40 tx=0x7ff488004730 comp rx=0 tx=0).stop 2024-03-28T22:42:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.059+0000 7ff4a0bf5740 1 -- 172.21.15.43:0/4251657062 shutdown_connections 2024-03-28T22:42:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.059+0000 7ff4a0bf5740 1 --2- 172.21.15.43:0/4251657062 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55829268da80 0x55829268fed0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.059+0000 7ff480c00640 0 client.27478 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.059+0000 7ff4a0bf5740 1 --2- 172.21.15.43:0/4251657062 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558292674cc0 0x55829267b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.059+0000 7ff4a0bf5740 1 --2- 172.21.15.43:0/4251657062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55829267da90 0x55829267b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.059+0000 7ff4a0bf5740 1 --2- 172.21.15.43:0/4251657062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5582926743a0 0x55829267aaf0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.059+0000 7ff4a0bf5740 1 -- 172.21.15.43:0/4251657062 >> 172.21.15.43:0/4251657062 conn(0x55829266ecc0 msgr2=0x558292670a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.060+0000 7ff4a0bf5740 1 -- 172.21.15.43:0/4251657062 shutdown_connections 2024-03-28T22:42:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.060+0000 7ff4a0bf5740 1 -- 172.21.15.43:0/4251657062 wait complete. 2024-03-28T22:42:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.060+0000 7ff4a0bf5740 20 client.27478 trim_cache size 0 max 16384 2024-03-28T22:42:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.078+0000 7ff4a0bf5740 1 Processor -- start 2024-03-28T22:42:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.079+0000 7ff4a0bf5740 1 -- start start 2024-03-28T22:42:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.079+0000 7ff4a0bf5740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5582926743a0 0x55829267d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.079+0000 7ff4a0bf5740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558292674cc0 0x558292689400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.079+0000 7ff4a0bf5740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55829267da90 0x558292675220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.079+0000 7ff4a0bf5740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff4981b0cb0 con 0x558292674cc0 2024-03-28T22:42:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.079+0000 7ff4a0bf5740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ff4981b27f0 con 0x55829267da90 2024-03-28T22:42:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.079+0000 7ff4a0bf5740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7ff4981b1380 con 0x5582926743a0 2024-03-28T22:42:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.079+0000 7ff497400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55829267da90 0x558292675220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.079+0000 7ff497400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55829267da90 0x558292675220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44664/0 (socket says 172.21.15.43:44664) 2024-03-28T22:42:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.079+0000 7ff497400640 1 -- 172.21.15.43:0/371250694 learned_addr learned my addr 172.21.15.43:0/371250694 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.079+0000 7ff496000640 1 --2- 172.21.15.43:0/371250694 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558292674cc0 0x558292689400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.079+0000 7ff496a00640 1 --2- 172.21.15.43:0/371250694 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5582926743a0 0x55829267d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.080+0000 7ff497400640 1 -- 172.21.15.43:0/371250694 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5582926743a0 msgr2=0x55829267d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.080+0000 7ff497400640 1 --2- 172.21.15.43:0/371250694 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5582926743a0 0x55829267d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.080+0000 7ff497400640 1 -- 172.21.15.43:0/371250694 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558292674cc0 msgr2=0x558292689400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.080+0000 7ff497400640 1 --2- 172.21.15.43:0/371250694 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558292674cc0 0x558292689400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.080+0000 7ff497400640 1 -- 172.21.15.43:0/371250694 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582926801f0 con 0x55829267da90 2024-03-28T22:42:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.080+0000 7ff496a00640 1 --2- 172.21.15.43:0/371250694 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5582926743a0 0x55829267d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.080+0000 7ff497400640 1 --2- 172.21.15.43:0/371250694 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55829267da90 0x558292675220 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7ff49002c6e0 tx=0x7ff4900095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6749d368a2bb14f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.080+0000 7ff483400640 1 -- 172.21.15.43:0/371250694 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff490045030 con 0x55829267da90 2024-03-28T22:42:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.080+0000 7ff483400640 1 -- 172.21.15.43:0/371250694 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff490003970 con 0x55829267da90 2024-03-28T22:42:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.080+0000 7ff483400640 1 -- 172.21.15.43:0/371250694 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff490039040 con 0x55829267da90 2024-03-28T22:42:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.081+0000 7ff4a0bf5740 1 -- 172.21.15.43:0/371250694 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55829267da90 msgr2=0x558292675220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.081+0000 7ff4a0bf5740 1 --2- 172.21.15.43:0/371250694 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55829267da90 0x558292675220 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7ff49002c6e0 tx=0x7ff4900095e0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.082+0000 7ff4a0bf5740 1 -- 172.21.15.43:0/371250694 shutdown_connections 2024-03-28T22:42:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.082+0000 7ff4a0bf5740 1 --2- 172.21.15.43:0/371250694 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55829267da90 0x558292675220 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.082+0000 7ff4a0bf5740 1 --2- 172.21.15.43:0/371250694 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558292674cc0 0x558292689400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.082+0000 7ff4a0bf5740 1 --2- 172.21.15.43:0/371250694 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5582926743a0 0x55829267d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.082+0000 7ff4a0bf5740 1 -- 172.21.15.43:0/371250694 >> 172.21.15.43:0/371250694 conn(0x55829266ecc0 msgr2=0x55829266d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.082+0000 7ff4a0bf5740 1 -- 172.21.15.43:0/371250694 shutdown_connections 2024-03-28T22:42:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.082+0000 7ff4a0bf5740 1 -- 172.21.15.43:0/371250694 wait complete. 2024-03-28T22:42:08.083 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.082+0000 7ff4a0bf5740 1 -- start start 2024-03-28T22:42:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.082+0000 7ff4a0bf5740 1 Processor -- start 2024-03-28T22:42:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.083+0000 7ff4a0bf5740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5582926743a0 0x558292681a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.083+0000 7ff4a0bf5740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558292674cc0 0x558292681f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.083+0000 7ff4a0bf5740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55829267da90 0x5582926f2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.083+0000 7ff4a0bf5740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff4981b2120 con 0x55829267da90 2024-03-28T22:42:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.083+0000 7ff4a0bf5740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ff4981b1a50 con 0x558292674cc0 2024-03-28T22:42:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.083+0000 7ff4a0bf5740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7ff4981aec90 con 0x5582926743a0 2024-03-28T22:42:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.083+0000 7ff4a0bf5740 10 client.0 _ll_register_callbacks cb 0x7ff488009920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.083+0000 7ff496a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5582926743a0 0x558292681a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.083+0000 7ff496000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558292674cc0 0x558292681f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.083+0000 7ff496a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5582926743a0 0x558292681a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:44478/0 (socket says 172.21.15.43:44478) 2024-03-28T22:42:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.083+0000 7ff496000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558292674cc0 0x558292681f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44680/0 (socket says 172.21.15.43:44680) 2024-03-28T22:42:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.083+0000 7ff496a00640 1 -- 172.21.15.43:0/4221442696 learned_addr learned my addr 172.21.15.43:0/4221442696 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.083+0000 7ff497400640 1 --2- 172.21.15.43:0/4221442696 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55829267da90 0x5582926f2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.084+0000 7ff496000640 1 -- 172.21.15.43:0/4221442696 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5582926743a0 msgr2=0x558292681a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.084+0000 7ff496000640 1 --2- 172.21.15.43:0/4221442696 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5582926743a0 0x558292681a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.084+0000 7ff496000640 1 -- 172.21.15.43:0/4221442696 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55829267da90 msgr2=0x5582926f2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.084+0000 7ff496000640 1 --2- 172.21.15.43:0/4221442696 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55829267da90 0x5582926f2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.084+0000 7ff496000640 1 -- 172.21.15.43:0/4221442696 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582926f33a0 con 0x558292674cc0 2024-03-28T22:42:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.084+0000 7ff496a00640 1 --2- 172.21.15.43:0/4221442696 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5582926743a0 0x558292681a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.084+0000 7ff496000640 1 --2- 172.21.15.43:0/4221442696 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558292674cc0 0x558292681f80 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7ff4840cbd00 tx=0x7ff484097300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41e35c4063eeb4ae server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.084+0000 7ff480c00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:42:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.084+0000 7ff480c00640 1 -- 172.21.15.43:0/4221442696 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4840cc180 con 0x558292674cc0 2024-03-28T22:42:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.084+0000 7ff480c00640 1 -- 172.21.15.43:0/4221442696 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7ff4840a32b0 con 0x558292674cc0 2024-03-28T22:42:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.084+0000 7ff4a0bf5740 1 -- 172.21.15.43:0/4221442696 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff490003970 con 0x558292674cc0 2024-03-28T22:42:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.084+0000 7ff480c00640 1 -- 172.21.15.43:0/4221442696 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff48409f0f0 con 0x558292674cc0 2024-03-28T22:42:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.084+0000 7ff4a0bf5740 10 client.27536 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.084+0000 7ff4a0bf5740 1 -- 172.21.15.43:0/4221442696 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5582926f46b0 con 0x558292674cc0 2024-03-28T22:42:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.085+0000 7ff480c00640 1 -- 172.21.15.43:0/4221442696 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff490003970 con 0x558292674cc0 2024-03-28T22:42:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.085+0000 7ff480c00640 1 -- 172.21.15.43:0/4221442696 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7ff48409a460 con 0x558292674cc0 2024-03-28T22:42:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.085+0000 7ff4a0bf5740 10 client.27536 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.085+0000 7ff4a0bf5740 10 client.27536 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:42:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.085+0000 7ff4a0bf5740 20 client.27536 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.085+0000 7ff4a0bf5740 1 --2- 172.21.15.43:0/4221442696 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55829268da80 0x5582926f6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.085+0000 7ff4a0bf5740 4 client.27536 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T22:42:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.085+0000 7ff4a0bf5740 1 -- 172.21.15.43:0/4221442696 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5582925e4b80 con 0x55829268da80 2024-03-28T22:42:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.085+0000 7ff496a00640 1 --2- 172.21.15.43:0/4221442696 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55829268da80 0x5582926f6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.086+0000 7ff496a00640 1 --2- 172.21.15.43:0/4221442696 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55829268da80 0x5582926f6dc0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6caed80663177d1b server_cookie=fdadbb9c3b3d2716 in_seq=0 out_seq=0 2024-03-28T22:42:08.087 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.086+0000 7ff480c00640 10 client.27536 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:08.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.215+0000 7f1106800640 1 -- 172.21.15.43:0/1333762732 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f10f800dca0 msgr2=0x7f10f80b1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:08.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.215+0000 7f1106800640 1 --2- 172.21.15.43:0/1333762732 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f10f800dca0 0x7f10f80b1920 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f1100067560 tx=0x7f1100097860 comp rx=0 tx=0).stop 2024-03-28T22:42:08.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.215+0000 7f1106800640 1 -- 172.21.15.43:0/1333762732 shutdown_connections 2024-03-28T22:42:08.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.215+0000 7f1106800640 1 --2- 172.21.15.43:0/1333762732 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f10f800dca0 0x7f10f80b1920 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.215+0000 7f1106800640 1 --2- 172.21.15.43:0/1333762732 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10f800d300 0x7f10f800d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.215+0000 7f1106800640 1 --2- 172.21.15.43:0/1333762732 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10f8014f00 0x7f10f80152e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.215+0000 7f1106800640 1 -- 172.21.15.43:0/1333762732 >> 172.21.15.43:0/1333762732 conn(0x7f10f80092b0 msgr2=0x7f10f80096c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:08.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.216+0000 7f1106800640 1 -- 172.21.15.43:0/1333762732 shutdown_connections 2024-03-28T22:42:08.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.216+0000 7f1106800640 1 -- 172.21.15.43:0/1333762732 wait complete. 2024-03-28T22:42:08.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.218+0000 7f1106800640 1 Processor -- start 2024-03-28T22:42:08.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.218+0000 7f1106800640 1 -- start start 2024-03-28T22:42:08.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.219+0000 7f1106800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10f800d300 0x7f10f813b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.219+0000 7f1106800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f10f800dca0 0x7f10f813b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.220+0000 7f1105400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10f800d300 0x7f10f813b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.220+0000 7f1105400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10f800d300 0x7f10f813b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:44492/0 (socket says 172.21.15.43:44492) 2024-03-28T22:42:08.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.220+0000 7f1105400640 1 -- 172.21.15.43:0/715921905 learned_addr learned my addr 172.21.15.43:0/715921905 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:08.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.221+0000 7f1106800640 1 --2- 172.21.15.43:0/715921905 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10f8014f00 0x7f10f813fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.221+0000 7f1106800640 1 -- 172.21.15.43:0/715921905 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f10f80b4f30 con 0x7f10f8014f00 2024-03-28T22:42:08.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.221+0000 7f1106800640 1 -- 172.21.15.43:0/715921905 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f10f80b4db0 con 0x7f10f800dca0 2024-03-28T22:42:08.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.221+0000 7f1104a00640 1 --2- 172.21.15.43:0/715921905 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f10f800dca0 0x7f10f813b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.221+0000 7f1106800640 1 -- 172.21.15.43:0/715921905 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f10f80b50b0 con 0x7f10f800d300 2024-03-28T22:42:08.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.222+0000 7f1105e00640 1 --2- 172.21.15.43:0/715921905 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10f8014f00 0x7f10f813fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.222+0000 7f1105400640 1 -- 172.21.15.43:0/715921905 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f10f800dca0 msgr2=0x7f10f813b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:08.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.222+0000 7f1105400640 1 --2- 172.21.15.43:0/715921905 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f10f800dca0 0x7f10f813b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.222+0000 7f1105400640 1 -- 172.21.15.43:0/715921905 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10f8014f00 msgr2=0x7f10f813fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:08.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.222+0000 7f1105400640 1 --2- 172.21.15.43:0/715921905 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10f8014f00 0x7f10f813fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.222+0000 7f1105400640 1 -- 172.21.15.43:0/715921905 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10f8140350 con 0x7f10f800d300 2024-03-28T22:42:08.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.222+0000 7f1104a00640 1 --2- 172.21.15.43:0/715921905 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f10f800dca0 0x7f10f813b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:42:08.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.222+0000 7f1105e00640 1 --2- 172.21.15.43:0/715921905 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10f8014f00 0x7f10f813fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:42:08.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.223+0000 7f1105400640 1 --2- 172.21.15.43:0/715921905 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10f800d300 0x7f10f813b3b0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f10f400e9f0 tx=0x7f10f400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:08.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.223+0000 7f10fe000640 1 -- 172.21.15.43:0/715921905 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f400cd80 con 0x7f10f800d300 2024-03-28T22:42:08.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.224+0000 7f1106800640 1 -- 172.21.15.43:0/715921905 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10f8140640 con 0x7f10f800d300 2024-03-28T22:42:08.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.224+0000 7f1106800640 1 -- 172.21.15.43:0/715921905 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10f8147ed0 con 0x7f10f800d300 2024-03-28T22:42:08.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.224+0000 7f10fe000640 1 -- 172.21.15.43:0/715921905 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10f4010910 con 0x7f10f800d300 2024-03-28T22:42:08.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.224+0000 7f10fe000640 1 -- 172.21.15.43:0/715921905 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f400cd80 con 0x7f10f800d300 2024-03-28T22:42:08.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.231+0000 7f110ab92740 1 Processor -- start 2024-03-28T22:42:08.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.232+0000 7f110ab92740 1 -- start start 2024-03-28T22:42:08.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.232+0000 7f110ab92740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55afd2ce9550 0x55afd2ce9930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.232+0000 7f110ab92740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55afd2ce9e70 0x55afd2cf2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.232+0000 7f110ab92740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afd2cf2a50 0x55afd2cf4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.232+0000 7f110ab92740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55afd2c4d050 con 0x55afd2cf2a50 2024-03-28T22:42:08.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.232+0000 7f110ab92740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55afd2bf5ce0 con 0x55afd2ce9550 2024-03-28T22:42:08.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.232+0000 7f110ab92740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55afd2c490f0 con 0x55afd2ce9e70 2024-03-28T22:42:08.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.233+0000 7f1105e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55afd2ce9550 0x55afd2ce9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.233+0000 7f1105e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55afd2ce9550 0x55afd2ce9930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44702/0 (socket says 172.21.15.43:44702) 2024-03-28T22:42:08.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.233+0000 7f1105e00640 1 -- 172.21.15.43:0/1273134229 learned_addr learned my addr 172.21.15.43:0/1273134229 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:08.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.233+0000 7f1104a00640 1 --2- 172.21.15.43:0/1273134229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afd2cf2a50 0x55afd2cf4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.233+0000 7f10fe000640 1 -- 172.21.15.43:0/715921905 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f10f40186b0 con 0x7f10f800d300 2024-03-28T22:42:08.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.234+0000 7f1105400640 1 --2- 172.21.15.43:0/1273134229 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55afd2ce9e70 0x55afd2cf2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.235+0000 7f1105e00640 1 -- 172.21.15.43:0/1273134229 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55afd2ce9e70 msgr2=0x55afd2cf2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:08.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.235+0000 7f1105e00640 1 --2- 172.21.15.43:0/1273134229 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55afd2ce9e70 0x55afd2cf2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.235+0000 7f1105e00640 1 -- 172.21.15.43:0/1273134229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afd2cf2a50 msgr2=0x55afd2cf4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:08.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.235+0000 7f1105e00640 1 --2- 172.21.15.43:0/1273134229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afd2cf2a50 0x55afd2cf4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.235+0000 7f1105e00640 1 -- 172.21.15.43:0/1273134229 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afd29613f0 con 0x55afd2ce9550 2024-03-28T22:42:08.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.235+0000 7f1105e00640 1 --2- 172.21.15.43:0/1273134229 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55afd2ce9550 0x55afd2ce9930 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f1100064c40 tx=0x7f110009c310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f35fb82454f2ef3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:08.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.236+0000 7f10f2600640 1 -- 172.21.15.43:0/1273134229 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11000ae070 con 0x55afd2ce9550 2024-03-28T22:42:08.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.236+0000 7f10f2600640 1 -- 172.21.15.43:0/1273134229 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f110009de70 con 0x55afd2ce9550 2024-03-28T22:42:08.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.236+0000 7f10f2600640 1 -- 172.21.15.43:0/1273134229 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110009ce10 con 0x55afd2ce9550 2024-03-28T22:42:08.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.237+0000 7f10fe000640 1 --2- 172.21.15.43:0/715921905 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f10e80a21f0 0x7f10e80a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.237+0000 7f110ab92740 1 -- 172.21.15.43:0/1273134229 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55afd2ce9550 msgr2=0x55afd2ce9930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:08.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.237+0000 7f110ab92740 1 --2- 172.21.15.43:0/1273134229 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55afd2ce9550 0x55afd2ce9930 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f1100064c40 tx=0x7f110009c310 comp rx=0 tx=0).stop 2024-03-28T22:42:08.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.237+0000 7f10fe000640 1 -- 172.21.15.43:0/715921905 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f10f40ce6d0 con 0x7f10f800d300 2024-03-28T22:42:08.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.238+0000 7f110ab92740 1 -- 172.21.15.43:0/1273134229 shutdown_connections 2024-03-28T22:42:08.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.238+0000 7f110ab92740 1 --2- 172.21.15.43:0/1273134229 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afd2cf2a50 0x55afd2cf4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.238+0000 7f110ab92740 1 --2- 172.21.15.43:0/1273134229 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55afd2ce9e70 0x55afd2cf2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.239+0000 7f110ab92740 1 --2- 172.21.15.43:0/1273134229 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55afd2ce9550 0x55afd2ce9930 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.239+0000 7f110ab92740 1 -- 172.21.15.43:0/1273134229 >> 172.21.15.43:0/1273134229 conn(0x55afd2ce3c80 msgr2=0x55afd2ce4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:08.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.239+0000 7f110ab92740 1 -- 172.21.15.43:0/1273134229 shutdown_connections 2024-03-28T22:42:08.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.240+0000 7f1105400640 1 --2- 172.21.15.43:0/715921905 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f10e80a21f0 0x7f10e80a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.241+0000 7f110ab92740 1 -- 172.21.15.43:0/1273134229 wait complete. 2024-03-28T22:42:08.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.242+0000 7f110ab92740 1 -- start start 2024-03-28T22:42:08.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.242+0000 7f110ab92740 1 Processor -- start 2024-03-28T22:42:08.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.243+0000 7f1105400640 1 --2- 172.21.15.43:0/715921905 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f10e80a21f0 0x7f10e80a46b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x55afd2ce9020 tx=0x7f10f40ce920 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:08.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.245+0000 7f110ab92740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55afd2ce9550 0x55afd2cefa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.246+0000 7f1105e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55afd2ce9550 0x55afd2cefa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.246+0000 7f1105e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55afd2ce9550 0x55afd2cefa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44712/0 (socket says 172.21.15.43:44712) 2024-03-28T22:42:08.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.246+0000 7f1105e00640 1 -- 172.21.15.43:0/1551762465 learned_addr learned my addr 172.21.15.43:0/1551762465 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:08.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.246+0000 7f110ab92740 1 --2- 172.21.15.43:0/1551762465 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afd2ce9e70 0x55afd2ceff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.246+0000 7f110ab92740 1 --2- 172.21.15.43:0/1551762465 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55afd2cf2a50 0x55afd2cf0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.246+0000 7f110ab92740 1 -- 172.21.15.43:0/1551762465 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55afd2cf57e0 con 0x55afd2ce9e70 2024-03-28T22:42:08.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.246+0000 7f110ab92740 1 -- 172.21.15.43:0/1551762465 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55afd2beb2f0 con 0x55afd2ce9550 2024-03-28T22:42:08.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.246+0000 7f110ab92740 1 -- 172.21.15.43:0/1551762465 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55afd2cf5960 con 0x55afd2cf2a50 2024-03-28T22:42:08.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.247+0000 7f110ab92740 10 client.0 _ll_register_callbacks cb 0x55afd2960be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:08.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.247+0000 7f1104a00640 1 --2- 172.21.15.43:0/1551762465 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afd2ce9e70 0x55afd2ceff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.248+0000 7f1104a00640 1 -- 172.21.15.43:0/1551762465 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55afd2cf2a50 msgr2=0x55afd2cf0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:08.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.248+0000 7f1104a00640 1 --2- 172.21.15.43:0/1551762465 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55afd2cf2a50 0x55afd2cf0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.248+0000 7f1104a00640 1 -- 172.21.15.43:0/1551762465 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55afd2ce9550 msgr2=0x55afd2cefa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:08.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.248+0000 7f1104a00640 1 --2- 172.21.15.43:0/1551762465 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55afd2ce9550 0x55afd2cefa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.248+0000 7f1104a00640 1 -- 172.21.15.43:0/1551762465 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afd2a2b7d0 con 0x55afd2ce9e70 2024-03-28T22:42:08.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.249+0000 7f1104a00640 1 --2- 172.21.15.43:0/1551762465 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afd2ce9e70 0x55afd2ceff40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f10ec013760 tx=0x7f10ec013c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83babb2321bf53c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:08.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.249+0000 7f10e6a00640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:42:08.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.250+0000 7f10e6a00640 1 -- 172.21.15.43:0/1551762465 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ec020060 con 0x55afd2ce9e70 2024-03-28T22:42:08.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.250+0000 7f110ab92740 1 -- 172.21.15.43:0/1551762465 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f110009de70 con 0x55afd2ce9e70 2024-03-28T22:42:08.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.250+0000 7f10e6a00640 1 -- 172.21.15.43:0/1551762465 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f10ec011bd0 con 0x55afd2ce9e70 2024-03-28T22:42:08.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.251+0000 7f10e6a00640 1 -- 172.21.15.43:0/1551762465 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f10ec027050 con 0x55afd2ce9e70 2024-03-28T22:42:08.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.251+0000 7f110ab92740 10 client.17328 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:08.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.251+0000 7f110ab92740 1 -- 172.21.15.43:0/1551762465 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55afd2d01b40 con 0x55afd2ce9e70 2024-03-28T22:42:08.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.252+0000 7f10e6a00640 1 -- 172.21.15.43:0/1551762465 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f110009de70 con 0x55afd2ce9e70 2024-03-28T22:42:08.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.253+0000 7f10e6a00640 1 -- 172.21.15.43:0/1551762465 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f10ec01a030 con 0x55afd2ce9e70 2024-03-28T22:42:08.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.253+0000 7f110ab92740 10 client.17328 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:08.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.253+0000 7f110ab92740 10 client.17328 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:42:08.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.253+0000 7f110ab92740 20 client.17328 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:08.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.253+0000 7f110ab92740 1 --2- 172.21.15.43:0/1551762465 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55afd2d03020 0x55afd2d054a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.253+0000 7f110ab92740 4 client.17328 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:42:08.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.253+0000 7f110ab92740 1 -- 172.21.15.43:0/1551762465 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55afd2bf6680 con 0x55afd2d03020 2024-03-28T22:42:08.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.254+0000 7f1105e00640 1 --2- 172.21.15.43:0/1551762465 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55afd2d03020 0x55afd2d054a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.255+0000 7f10e6a00640 10 client.17328 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:08.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.256+0000 7f1105e00640 1 --2- 172.21.15.43:0/1551762465 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55afd2d03020 0x55afd2d054a0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7156b38820f30c0 server_cookie=4c6fb748431627d0 in_seq=0 out_seq=0 2024-03-28T22:42:08.275 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.276+0000 7ff480c00640 1 -- 172.21.15.43:0/4221442696 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5582925e4b80 con 0x55829268da80 2024-03-28T22:42:08.275 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.276+0000 7ff480c00640 10 client.27536 handle_command_reply: tid=0 2024-03-28T22:42:08.275 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.276+0000 7ff4a0bf5740 1 client.27536 shutdown 2024-03-28T22:42:08.275 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.276+0000 7ff4a0bf5740 1 -- 172.21.15.43:0/4221442696 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558292674cc0 msgr2=0x558292681f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:08.275 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.276+0000 7ff4a0bf5740 1 --2- 172.21.15.43:0/4221442696 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558292674cc0 0x558292681f80 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7ff4840cbd00 tx=0x7ff484097300 comp rx=0 tx=0).stop 2024-03-28T22:42:08.275 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.276+0000 7ff4a0bf5740 1 -- 172.21.15.43:0/4221442696 shutdown_connections 2024-03-28T22:42:08.275 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.276+0000 7ff4a0bf5740 1 --2- 172.21.15.43:0/4221442696 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55829268da80 0x5582926f6dc0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.275 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.276+0000 7ff4a0bf5740 1 --2- 172.21.15.43:0/4221442696 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558292674cc0 0x558292681f80 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.275 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.276+0000 7ff480c00640 0 client.27536 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:08.275 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.276+0000 7ff4a0bf5740 1 --2- 172.21.15.43:0/4221442696 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55829267da90 0x5582926f2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.275 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.276+0000 7ff4a0bf5740 1 --2- 172.21.15.43:0/4221442696 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5582926743a0 0x558292681a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.275 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.276+0000 7ff4a0bf5740 1 -- 172.21.15.43:0/4221442696 >> 172.21.15.43:0/4221442696 conn(0x55829266ecc0 msgr2=0x55829266dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:08.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.277+0000 7f10e6a00640 1 -- 172.21.15.43:0/1551762465 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55afd2bf6680 con 0x55afd2d03020 2024-03-28T22:42:08.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.277+0000 7f10e6a00640 10 client.17328 handle_command_reply: tid=0 2024-03-28T22:42:08.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.277+0000 7f110ab92740 1 client.17328 shutdown 2024-03-28T22:42:08.277 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.277+0000 7ff4a0bf5740 1 -- 172.21.15.43:0/4221442696 shutdown_connections 2024-03-28T22:42:08.277 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.277+0000 7ff4a0bf5740 1 -- 172.21.15.43:0/4221442696 wait complete. 2024-03-28T22:42:08.277 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.278+0000 7ff4a0bf5740 20 client.27536 trim_cache size 0 max 16384 2024-03-28T22:42:08.277 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:{ 2024-03-28T22:42:08.277 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "message": "", 2024-03-28T22:42:08.277 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "return_code": 0 2024-03-28T22:42:08.277 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:} 2024-03-28T22:42:08.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.279+0000 7f110ab92740 1 -- 172.21.15.43:0/1551762465 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afd2ce9e70 msgr2=0x55afd2ceff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:08.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.279+0000 7f110ab92740 1 --2- 172.21.15.43:0/1551762465 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afd2ce9e70 0x55afd2ceff40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f10ec013760 tx=0x7f10ec013c20 comp rx=0 tx=0).stop 2024-03-28T22:42:08.278 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.279+0000 7ff49de00640 1 -- 172.21.15.43:0/3126971427 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ff47c0a1f90 msgr2=0x7ff47c0a4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:08.278 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.279+0000 7ff49de00640 1 --2- 172.21.15.43:0/3126971427 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ff47c0a1f90 0x7ff47c0a4450 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x558292673e50 tx=0x7ff488005f90 comp rx=0 tx=0).stop 2024-03-28T22:42:08.278 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.279+0000 7ff49de00640 1 -- 172.21.15.43:0/3126971427 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4981042e0 msgr2=0x7ff49819a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:08.279 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.279+0000 7ff49de00640 1 --2- 172.21.15.43:0/3126971427 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4981042e0 0x7ff49819a660 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff48400e9f0 tx=0x7ff48400edf0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.279 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.279+0000 7ff49de00640 1 -- 172.21.15.43:0/3126971427 shutdown_connections 2024-03-28T22:42:08.279 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.279+0000 7ff49de00640 1 --2- 172.21.15.43:0/3126971427 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7ff47c0a1f90 0x7ff47c0a4450 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.279 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.279+0000 7ff49de00640 1 --2- 172.21.15.43:0/3126971427 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff498109930 0x7ff49819e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.279 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.279+0000 7ff49de00640 1 --2- 172.21.15.43:0/3126971427 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4981042e0 0x7ff49819a660 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.279 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.279+0000 7ff49de00640 1 --2- 172.21.15.43:0/3126971427 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff498103920 0x7ff49819a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.279 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.279+0000 7ff49de00640 1 -- 172.21.15.43:0/3126971427 >> 172.21.15.43:0/3126971427 conn(0x7ff4980ff410 msgr2=0x7ff4980ff8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:08.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.279+0000 7f110ab92740 1 -- 172.21.15.43:0/1551762465 shutdown_connections 2024-03-28T22:42:08.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.279+0000 7f110ab92740 1 --2- 172.21.15.43:0/1551762465 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55afd2cf2a50 0x55afd2cf0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.279+0000 7f110ab92740 1 --2- 172.21.15.43:0/1551762465 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afd2ce9e70 0x55afd2ceff40 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.279+0000 7f110ab92740 1 --2- 172.21.15.43:0/1551762465 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55afd2d03020 0x55afd2d054a0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.279+0000 7f110ab92740 1 --2- 172.21.15.43:0/1551762465 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55afd2ce9550 0x55afd2cefa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.279+0000 7f110ab92740 1 -- 172.21.15.43:0/1551762465 >> 172.21.15.43:0/1551762465 conn(0x55afd2ce3c80 msgr2=0x55afd2ce59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:08.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.279+0000 7f10e6a00640 0 client.17328 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:08.280 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.280+0000 7ff49de00640 1 -- 172.21.15.43:0/3126971427 shutdown_connections 2024-03-28T22:42:08.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.280+0000 7f110ab92740 1 -- 172.21.15.43:0/1551762465 shutdown_connections 2024-03-28T22:42:08.281 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:08.280+0000 7ff49de00640 1 -- 172.21.15.43:0/3126971427 wait complete. 2024-03-28T22:42:08.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.280+0000 7f110ab92740 1 -- 172.21.15.43:0/1551762465 wait complete. 2024-03-28T22:42:08.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.280+0000 7f110ab92740 20 client.17328 trim_cache size 0 max 16384 2024-03-28T22:42:08.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.297+0000 7f110ab92740 1 Processor -- start 2024-03-28T22:42:08.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.297+0000 7f110ab92740 1 -- start start 2024-03-28T22:42:08.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.297+0000 7f110ab92740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55afd2ce9550 0x55afd2cf2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.297+0000 7f110ab92740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afd2ce9e70 0x55afd2cfe5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.297+0000 7f110ab92740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55afd2cf2a50 0x55afd2d0adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.297+0000 7f110ab92740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f10e000ff20 con 0x55afd2ce9e70 2024-03-28T22:42:08.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.297+0000 7f110ab92740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f10e0011a60 con 0x55afd2ce9550 2024-03-28T22:42:08.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.297+0000 7f110ab92740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f10e00105f0 con 0x55afd2cf2a50 2024-03-28T22:42:08.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.298+0000 7f1104a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afd2ce9e70 0x55afd2cfe5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.298+0000 7f1105e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55afd2ce9550 0x55afd2cf2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.298+0000 7f1104a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afd2ce9e70 0x55afd2cfe5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48474/0 (socket says 172.21.15.43:48474) 2024-03-28T22:42:08.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.298+0000 7f1104a00640 1 -- 172.21.15.43:0/861880779 learned_addr learned my addr 172.21.15.43:0/861880779 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:08.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.298+0000 7f1105e00640 1 --2- 172.21.15.43:0/861880779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55afd2cf2a50 0x55afd2d0adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.298+0000 7f1104a00640 1 -- 172.21.15.43:0/861880779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55afd2cf2a50 msgr2=0x55afd2d0adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:08.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.298+0000 7f1104a00640 1 --2- 172.21.15.43:0/861880779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55afd2cf2a50 0x55afd2d0adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.298+0000 7f1104a00640 1 -- 172.21.15.43:0/861880779 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55afd2ce9550 msgr2=0x55afd2cf2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:08.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.298+0000 7f1104a00640 1 --2- 172.21.15.43:0/861880779 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55afd2ce9550 0x55afd2cf2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.298+0000 7f1104a00640 1 -- 172.21.15.43:0/861880779 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afd2ce3ab0 con 0x55afd2ce9e70 2024-03-28T22:42:08.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.298+0000 7f1105e00640 1 --2- 172.21.15.43:0/861880779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55afd2cf2a50 0x55afd2d0adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:08.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.298+0000 7f1104a00640 1 --2- 172.21.15.43:0/861880779 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afd2ce9e70 0x55afd2cfe5e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f10ec01b5f0 tx=0x7f10ec009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75b7b6c989a5b3bd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:08.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.299+0000 7f10f1c00640 1 -- 172.21.15.43:0/861880779 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ec020060 con 0x55afd2ce9e70 2024-03-28T22:42:08.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.299+0000 7f10f1c00640 1 -- 172.21.15.43:0/861880779 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10ec011dd0 con 0x55afd2ce9e70 2024-03-28T22:42:08.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.299+0000 7f10f1c00640 1 -- 172.21.15.43:0/861880779 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ec004660 con 0x55afd2ce9e70 2024-03-28T22:42:08.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.299+0000 7f110ab92740 1 -- 172.21.15.43:0/861880779 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afd2ce9e70 msgr2=0x55afd2cfe5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:08.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.299+0000 7f110ab92740 1 --2- 172.21.15.43:0/861880779 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afd2ce9e70 0x55afd2cfe5e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f10ec01b5f0 tx=0x7f10ec009dc0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.300+0000 7f110ab92740 1 -- 172.21.15.43:0/861880779 shutdown_connections 2024-03-28T22:42:08.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.300+0000 7f110ab92740 1 --2- 172.21.15.43:0/861880779 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55afd2cf2a50 0x55afd2d0adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.300+0000 7f110ab92740 1 --2- 172.21.15.43:0/861880779 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afd2ce9e70 0x55afd2cfe5e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.300+0000 7f110ab92740 1 --2- 172.21.15.43:0/861880779 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55afd2ce9550 0x55afd2cf2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.300+0000 7f110ab92740 1 -- 172.21.15.43:0/861880779 >> 172.21.15.43:0/861880779 conn(0x55afd2ce3c80 msgr2=0x55afd2cef9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:08.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.300+0000 7f110ab92740 1 -- 172.21.15.43:0/861880779 shutdown_connections 2024-03-28T22:42:08.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.300+0000 7f110ab92740 1 -- 172.21.15.43:0/861880779 wait complete. 2024-03-28T22:42:08.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.300+0000 7f110ab92740 1 -- start start 2024-03-28T22:42:08.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.300+0000 7f110ab92740 1 Processor -- start 2024-03-28T22:42:08.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.301+0000 7f110ab92740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55afd2ce9550 0x55afd2cec730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.301+0000 7f110ab92740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afd2ce9e70 0x55afd2cecc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.301+0000 7f110ab92740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55afd2cf2a50 0x55afd2ced1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.301+0000 7f110ab92740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f10e0011390 con 0x55afd2ce9e70 2024-03-28T22:42:08.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.301+0000 7f110ab92740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f10e0010cc0 con 0x55afd2ce9550 2024-03-28T22:42:08.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.301+0000 7f110ab92740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f10e000df00 con 0x55afd2cf2a50 2024-03-28T22:42:08.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.301+0000 7f110ab92740 10 client.0 _ll_register_callbacks cb 0x7f10ec009650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:08.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.301+0000 7f1104a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afd2ce9e70 0x55afd2cecc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.301+0000 7f1104a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afd2ce9e70 0x55afd2cecc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48490/0 (socket says 172.21.15.43:48490) 2024-03-28T22:42:08.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.301+0000 7f1104a00640 1 -- 172.21.15.43:0/807285062 learned_addr learned my addr 172.21.15.43:0/807285062 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:08.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.301+0000 7f1105e00640 1 --2- 172.21.15.43:0/807285062 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55afd2ce9550 0x55afd2cec730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.301+0000 7f1105e00640 1 --2- 172.21.15.43:0/807285062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55afd2cf2a50 0x55afd2ced1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.302+0000 7f1104a00640 1 -- 172.21.15.43:0/807285062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55afd2cf2a50 msgr2=0x55afd2ced1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:08.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.302+0000 7f1104a00640 1 --2- 172.21.15.43:0/807285062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55afd2cf2a50 0x55afd2ced1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.302+0000 7f1104a00640 1 -- 172.21.15.43:0/807285062 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55afd2ce9550 msgr2=0x55afd2cec730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:08.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.302+0000 7f1104a00640 1 --2- 172.21.15.43:0/807285062 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55afd2ce9550 0x55afd2cec730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.302+0000 7f1104a00640 1 -- 172.21.15.43:0/807285062 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afd2d685b0 con 0x55afd2ce9e70 2024-03-28T22:42:08.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.302+0000 7f1105e00640 1 --2- 172.21.15.43:0/807285062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55afd2cf2a50 0x55afd2ced1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:42:08.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.302+0000 7f1104a00640 1 --2- 172.21.15.43:0/807285062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afd2ce9e70 0x55afd2cecc70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f10ec01b690 tx=0x7f10ec013c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bbf052d44de25c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:08.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.302+0000 7f10e6a00640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:42:08.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.302+0000 7f10e6a00640 1 -- 172.21.15.43:0/807285062 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ec020040 con 0x55afd2ce9e70 2024-03-28T22:42:08.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.302+0000 7f10e6a00640 1 -- 172.21.15.43:0/807285062 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f10ec01b9a0 con 0x55afd2ce9e70 2024-03-28T22:42:08.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.302+0000 7f110ab92740 1 -- 172.21.15.43:0/807285062 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10ec011dd0 con 0x55afd2ce9e70 2024-03-28T22:42:08.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.303+0000 7f10e6a00640 1 -- 172.21.15.43:0/807285062 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f10ec01bbe0 con 0x55afd2ce9e70 2024-03-28T22:42:08.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.303+0000 7f110ab92740 10 client.17334 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:08.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.303+0000 7f110ab92740 1 -- 172.21.15.43:0/807285062 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55afd2d69870 con 0x55afd2ce9e70 2024-03-28T22:42:08.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.303+0000 7f10e6a00640 1 -- 172.21.15.43:0/807285062 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10ec011dd0 con 0x55afd2ce9e70 2024-03-28T22:42:08.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.303+0000 7f10e6a00640 1 -- 172.21.15.43:0/807285062 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f10ec01a070 con 0x55afd2ce9e70 2024-03-28T22:42:08.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.303+0000 7f110ab92740 10 client.17334 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:08.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.303+0000 7f110ab92740 10 client.17334 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:42:08.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.303+0000 7f110ab92740 20 client.17334 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:08.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.303+0000 7f110ab92740 1 --2- 172.21.15.43:0/807285062 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55afd2d69f20 0x55afd2d6c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:08.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.303+0000 7f110ab92740 4 client.17334 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:42:08.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.303+0000 7f110ab92740 1 -- 172.21.15.43:0/807285062 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55afd2c5a590 con 0x55afd2d69f20 2024-03-28T22:42:08.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.304+0000 7f1105e00640 1 --2- 172.21.15.43:0/807285062 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55afd2d69f20 0x55afd2d6c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:08.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.304+0000 7f1105e00640 1 --2- 172.21.15.43:0/807285062 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55afd2d69f20 0x55afd2d6c340 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c5f8d4898ec9774 server_cookie=22d135fd9f1afaca in_seq=0 out_seq=0 2024-03-28T22:42:08.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.304+0000 7f10e6a00640 10 client.17334 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:08.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.305+0000 7f10e6a00640 1 -- 172.21.15.43:0/807285062 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55afd2c5a590 con 0x55afd2d69f20 2024-03-28T22:42:08.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.305+0000 7f10e6a00640 10 client.17334 handle_command_reply: tid=0 2024-03-28T22:42:08.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.305+0000 7f110ab92740 1 client.17334 shutdown 2024-03-28T22:42:08.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.305+0000 7f110ab92740 1 -- 172.21.15.43:0/807285062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afd2ce9e70 msgr2=0x55afd2cecc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:08.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.305+0000 7f110ab92740 1 --2- 172.21.15.43:0/807285062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afd2ce9e70 0x55afd2cecc70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f10ec01b690 tx=0x7f10ec013c20 comp rx=0 tx=0).stop 2024-03-28T22:42:08.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.306+0000 7f110ab92740 1 -- 172.21.15.43:0/807285062 shutdown_connections 2024-03-28T22:42:08.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.306+0000 7f110ab92740 1 --2- 172.21.15.43:0/807285062 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55afd2cf2a50 0x55afd2ced1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.306+0000 7f110ab92740 1 --2- 172.21.15.43:0/807285062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55afd2ce9e70 0x55afd2cecc70 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.306+0000 7f110ab92740 1 --2- 172.21.15.43:0/807285062 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55afd2d69f20 0x55afd2d6c340 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.306+0000 7f110ab92740 1 --2- 172.21.15.43:0/807285062 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55afd2ce9550 0x55afd2cec730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.306+0000 7f10e6a00640 0 client.17334 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:08.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.306+0000 7f110ab92740 1 -- 172.21.15.43:0/807285062 >> 172.21.15.43:0/807285062 conn(0x55afd2ce3c80 msgr2=0x55afd2ce2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:08.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.306+0000 7f110ab92740 1 -- 172.21.15.43:0/807285062 shutdown_connections 2024-03-28T22:42:08.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.306+0000 7f110ab92740 1 -- 172.21.15.43:0/807285062 wait complete. 2024-03-28T22:42:08.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.306+0000 7f110ab92740 20 client.17334 trim_cache size 0 max 16384 2024-03-28T22:42:08.308 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:42:08.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.307+0000 7f1106800640 1 -- 172.21.15.43:0/715921905 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f10e80a21f0 msgr2=0x7f10e80a46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:08.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.307+0000 7f1106800640 1 --2- 172.21.15.43:0/715921905 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f10e80a21f0 0x7f10e80a46b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x55afd2ce9020 tx=0x7f10f40ce920 comp rx=0 tx=0).stop 2024-03-28T22:42:08.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.307+0000 7f1106800640 1 -- 172.21.15.43:0/715921905 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10f800d300 msgr2=0x7f10f813b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:08.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.307+0000 7f1106800640 1 --2- 172.21.15.43:0/715921905 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10f800d300 0x7f10f813b3b0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f10f400e9f0 tx=0x7f10f400eeb0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.307+0000 7f1106800640 1 -- 172.21.15.43:0/715921905 shutdown_connections 2024-03-28T22:42:08.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.307+0000 7f1106800640 1 --2- 172.21.15.43:0/715921905 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f10e80a21f0 0x7f10e80a46b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.307+0000 7f1106800640 1 --2- 172.21.15.43:0/715921905 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f10f800dca0 0x7f10f813b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.307+0000 7f1106800640 1 --2- 172.21.15.43:0/715921905 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10f8014f00 0x7f10f813fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.307+0000 7f1106800640 1 --2- 172.21.15.43:0/715921905 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f10f800d300 0x7f10f813b3b0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:08.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.307+0000 7f1106800640 1 -- 172.21.15.43:0/715921905 >> 172.21.15.43:0/715921905 conn(0x7f10f80092b0 msgr2=0x7f10f809ac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:08.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.307+0000 7f1106800640 1 -- 172.21.15.43:0/715921905 shutdown_connections 2024-03-28T22:42:08.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:08.307+0000 7f1106800640 1 -- 172.21.15.43:0/715921905 wait complete. 2024-03-28T22:42:08.364 INFO:tasks.workunit.client.0.smithi043.stdout: 1 329237 7.83 MB/sec execute 99 sec latency 90.601 ms 2024-03-28T22:42:09.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:08 smithi043 ceph-mon[31823]: pgmap v153: 129 pgs: 129 active+clean; 51 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.9 MiB/s wr, 343 op/s 2024-03-28T22:42:09.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:08 smithi172 ceph-mon[36566]: pgmap v153: 129 pgs: 129 active+clean; 51 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.9 MiB/s wr, 343 op/s 2024-03-28T22:42:09.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:08 smithi082 ceph-mon[36471]: pgmap v153: 129 pgs: 129 active+clean; 51 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.9 MiB/s wr, 343 op/s 2024-03-28T22:42:09.324 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:42:09.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:42:09.324 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:42:09.364 INFO:tasks.workunit.client.0.smithi043.stdout: 1 330443 7.82 MB/sec execute 100 sec latency 13.364 ms 2024-03-28T22:42:09.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.600+0000 7f6e77600640 1 -- 172.21.15.43:0/180941462 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6e701010f0 msgr2=0x7f6e701055b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:09.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.600+0000 7f6e77600640 1 --2- 172.21.15.43:0/180941462 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6e701010f0 0x7f6e701055b0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f6e58009900 tx=0x7f6e5802bcc0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.601+0000 7f6e77600640 1 -- 172.21.15.43:0/180941462 shutdown_connections 2024-03-28T22:42:09.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.601+0000 7f6e77600640 1 --2- 172.21.15.43:0/180941462 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6e701010f0 0x7f6e701055b0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.601+0000 7f6e77600640 1 --2- 172.21.15.43:0/180941462 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6e70105bd0 0x7f6e70106080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.601+0000 7f6e77600640 1 --2- 172.21.15.43:0/180941462 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e701007d0 0x7f6e70100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.601+0000 7f6e77600640 1 -- 172.21.15.43:0/180941462 >> 172.21.15.43:0/180941462 conn(0x7f6e700777b0 msgr2=0x7f6e70077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:09.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.601+0000 7f6e77600640 1 -- 172.21.15.43:0/180941462 shutdown_connections 2024-03-28T22:42:09.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.601+0000 7f6e77600640 1 -- 172.21.15.43:0/180941462 wait complete. 2024-03-28T22:42:09.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.602+0000 7f6e77600640 1 Processor -- start 2024-03-28T22:42:09.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.603+0000 7f6e77600640 1 -- start start 2024-03-28T22:42:09.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.603+0000 7f6e77600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6e701007d0 0x7f6e7019e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.603+0000 7f6e77600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6e701010f0 0x7f6e7019eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.603+0000 7f6e77600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e70105bd0 0x7f6e701a2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.603+0000 7f6e77600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6e70118180 con 0x7f6e70105bd0 2024-03-28T22:42:09.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.603+0000 7f6e77600640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f6e70118000 con 0x7f6e701010f0 2024-03-28T22:42:09.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.603+0000 7f6e77600640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f6e70118300 con 0x7f6e701007d0 2024-03-28T22:42:09.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.603+0000 7f6e74c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6e701007d0 0x7f6e7019e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.603+0000 7f6e67e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6e701010f0 0x7f6e7019eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.603+0000 7f6e74c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6e701007d0 0x7f6e7019e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:44522/0 (socket says 172.21.15.43:44522) 2024-03-28T22:42:09.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.603+0000 7f6e74c00640 1 -- 172.21.15.43:0/342133720 learned_addr learned my addr 172.21.15.43:0/342133720 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:09.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.604+0000 7f6e75600640 1 --2- 172.21.15.43:0/342133720 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e70105bd0 0x7f6e701a2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.604+0000 7f6e74c00640 1 -- 172.21.15.43:0/342133720 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6e701010f0 msgr2=0x7f6e7019eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:09.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.604+0000 7f6e74c00640 1 --2- 172.21.15.43:0/342133720 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6e701010f0 0x7f6e7019eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.604+0000 7f6e74c00640 1 -- 172.21.15.43:0/342133720 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e70105bd0 msgr2=0x7f6e701a2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:09.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.604+0000 7f6e74c00640 1 --2- 172.21.15.43:0/342133720 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e70105bd0 0x7f6e701a2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.604+0000 7f6e74c00640 1 -- 172.21.15.43:0/342133720 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e701a3650 con 0x7f6e701007d0 2024-03-28T22:42:09.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.604+0000 7f6e74c00640 1 --2- 172.21.15.43:0/342133720 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6e701007d0 0x7f6e7019e660 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f6e6000e9f0 tx=0x7f6e6000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:09.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.604+0000 7f6e6e000640 1 -- 172.21.15.43:0/342133720 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e6000cd80 con 0x7f6e701007d0 2024-03-28T22:42:09.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.604+0000 7f6e6e000640 1 -- 172.21.15.43:0/342133720 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e60010910 con 0x7f6e701007d0 2024-03-28T22:42:09.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.604+0000 7f6e77600640 1 -- 172.21.15.43:0/342133720 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e701a3940 con 0x7f6e701007d0 2024-03-28T22:42:09.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.604+0000 7f6e6e000640 1 -- 172.21.15.43:0/342133720 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e60018990 con 0x7f6e701007d0 2024-03-28T22:42:09.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.605+0000 7f6e77600640 1 -- 172.21.15.43:0/342133720 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e701ab1d0 con 0x7f6e701007d0 2024-03-28T22:42:09.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.608+0000 7f6e7a3f6740 1 Processor -- start 2024-03-28T22:42:09.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.608+0000 7f6e7a3f6740 1 -- start start 2024-03-28T22:42:09.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.608+0000 7f6e6e000640 1 -- 172.21.15.43:0/342133720 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f6e60010ac0 con 0x7f6e701007d0 2024-03-28T22:42:09.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.609+0000 7f6e7a3f6740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dddf902f70 0x55dddf903350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.609+0000 7f6e7a3f6740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dddf903890 0x55dddf90bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.609+0000 7f6e7a3f6740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dddf90c470 0x55dddf90e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.609+0000 7f6e7a3f6740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55dddf866c40 con 0x55dddf90c470 2024-03-28T22:42:09.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.609+0000 7f6e7a3f6740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55dddf80fc40 con 0x55dddf903890 2024-03-28T22:42:09.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.609+0000 7f6e7a3f6740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55dddf862ce0 con 0x55dddf902f70 2024-03-28T22:42:09.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.609+0000 7f6e75600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dddf902f70 0x55dddf903350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.609+0000 7f6e74c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dddf903890 0x55dddf90bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.609+0000 7f6e75600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dddf902f70 0x55dddf903350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:44524/0 (socket says 172.21.15.43:44524) 2024-03-28T22:42:09.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.609+0000 7f6e75600640 1 -- 172.21.15.43:0/1249321122 learned_addr learned my addr 172.21.15.43:0/1249321122 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:09.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.609+0000 7f6e67e00640 1 --2- 172.21.15.43:0/1249321122 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dddf90c470 0x55dddf90e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.609+0000 7f6e75600640 1 -- 172.21.15.43:0/1249321122 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dddf903890 msgr2=0x55dddf90bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:09.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.609+0000 7f6e75600640 1 --2- 172.21.15.43:0/1249321122 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dddf903890 0x55dddf90bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.609+0000 7f6e75600640 1 -- 172.21.15.43:0/1249321122 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dddf90c470 msgr2=0x55dddf90e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:09.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.609+0000 7f6e75600640 1 --2- 172.21.15.43:0/1249321122 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dddf90c470 0x55dddf90e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.609+0000 7f6e75600640 1 -- 172.21.15.43:0/1249321122 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dddf6437e0 con 0x55dddf902f70 2024-03-28T22:42:09.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.610+0000 7f6e75600640 1 --2- 172.21.15.43:0/1249321122 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dddf902f70 0x55dddf903350 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f6e68009f40 tx=0x7f6e68010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=121eca0e7e4c27ee server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:09.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.610+0000 7f6e65c00640 1 -- 172.21.15.43:0/1249321122 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e68003b70 con 0x55dddf902f70 2024-03-28T22:42:09.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.610+0000 7f6e65c00640 1 -- 172.21.15.43:0/1249321122 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e68003d10 con 0x55dddf902f70 2024-03-28T22:42:09.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.610+0000 7f6e65c00640 1 -- 172.21.15.43:0/1249321122 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e68011a40 con 0x55dddf902f70 2024-03-28T22:42:09.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.610+0000 7f6e6e000640 1 --2- 172.21.15.43:0/342133720 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6e540a1ef0 0x7f6e540a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.611+0000 7f6e6e000640 1 -- 172.21.15.43:0/342133720 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f6e600ce8f0 con 0x7f6e701007d0 2024-03-28T22:42:09.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.611+0000 7f6e74c00640 1 --2- 172.21.15.43:0/342133720 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6e540a1ef0 0x7f6e540a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.611+0000 7f6e7a3f6740 1 -- 172.21.15.43:0/1249321122 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dddf902f70 msgr2=0x55dddf903350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:09.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.611+0000 7f6e7a3f6740 1 --2- 172.21.15.43:0/1249321122 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dddf902f70 0x55dddf903350 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f6e68009f40 tx=0x7f6e68010860 comp rx=0 tx=0).stop 2024-03-28T22:42:09.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.611+0000 7f6e7a3f6740 1 -- 172.21.15.43:0/1249321122 shutdown_connections 2024-03-28T22:42:09.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.611+0000 7f6e7a3f6740 1 --2- 172.21.15.43:0/1249321122 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dddf903890 0x55dddf90bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.611+0000 7f6e7a3f6740 1 --2- 172.21.15.43:0/1249321122 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dddf90c470 0x55dddf90e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.611+0000 7f6e7a3f6740 1 --2- 172.21.15.43:0/1249321122 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dddf902f70 0x55dddf903350 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.611+0000 7f6e7a3f6740 1 -- 172.21.15.43:0/1249321122 >> 172.21.15.43:0/1249321122 conn(0x55dddf8fd6a0 msgr2=0x55dddf8fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:09.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.611+0000 7f6e7a3f6740 1 -- 172.21.15.43:0/1249321122 shutdown_connections 2024-03-28T22:42:09.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.611+0000 7f6e7a3f6740 1 -- 172.21.15.43:0/1249321122 wait complete. 2024-03-28T22:42:09.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.612+0000 7f6e74c00640 1 --2- 172.21.15.43:0/342133720 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6e540a1ef0 0x7f6e540a43b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6e600cbd50 tx=0x7f6e600d44a0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:09.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.612+0000 7f6e7a3f6740 1 -- start start 2024-03-28T22:42:09.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.612+0000 7f6e7a3f6740 1 Processor -- start 2024-03-28T22:42:09.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.613+0000 7f6e7a3f6740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dddf902f70 0x55dddf909460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.613+0000 7f6e7a3f6740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dddf903890 0x55dddf9099a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.613+0000 7f6e7a3f6740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dddf90c470 0x55dddf909ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.613+0000 7f6e7a3f6740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55dddf90f260 con 0x55dddf903890 2024-03-28T22:42:09.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.613+0000 7f6e7a3f6740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55dddf805250 con 0x55dddf902f70 2024-03-28T22:42:09.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.613+0000 7f6e7a3f6740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55dddf90f3e0 con 0x55dddf90c470 2024-03-28T22:42:09.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.613+0000 7f6e7a3f6740 10 client.0 _ll_register_callbacks cb 0x55dddf826e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:09.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.613+0000 7f6e75600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dddf902f70 0x55dddf909460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.613+0000 7f6e75600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dddf902f70 0x55dddf909460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44776/0 (socket says 172.21.15.43:44776) 2024-03-28T22:42:09.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.613+0000 7f6e75600640 1 -- 172.21.15.43:0/3392574454 learned_addr learned my addr 172.21.15.43:0/3392574454 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:09.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.613+0000 7f6e75600640 1 --2- 172.21.15.43:0/3392574454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dddf90c470 0x55dddf909ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.613+0000 7f6e67e00640 1 --2- 172.21.15.43:0/3392574454 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dddf903890 0x55dddf9099a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.613+0000 7f6e75600640 1 -- 172.21.15.43:0/3392574454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dddf90c470 msgr2=0x55dddf909ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:09.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.613+0000 7f6e75600640 1 --2- 172.21.15.43:0/3392574454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dddf90c470 0x55dddf909ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.613+0000 7f6e75600640 1 -- 172.21.15.43:0/3392574454 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dddf903890 msgr2=0x55dddf9099a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:09.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.613+0000 7f6e75600640 1 --2- 172.21.15.43:0/3392574454 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dddf903890 0x55dddf9099a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.613+0000 7f6e75600640 1 -- 172.21.15.43:0/3392574454 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dddf6a3f70 con 0x55dddf902f70 2024-03-28T22:42:09.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.614+0000 7f6e75600640 1 --2- 172.21.15.43:0/3392574454 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dddf902f70 0x55dddf909460 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f6e68003c80 tx=0x7f6e68013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87b3bfcf351eb593 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:09.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.614+0000 7f6e5e000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:42:09.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.614+0000 7f6e5e000640 1 -- 172.21.15.43:0/3392574454 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e680117a0 con 0x55dddf902f70 2024-03-28T22:42:09.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.614+0000 7f6e5e000640 1 -- 172.21.15.43:0/3392574454 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f6e68011940 con 0x55dddf902f70 2024-03-28T22:42:09.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.614+0000 7f6e7a3f6740 1 -- 172.21.15.43:0/3392574454 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e68003d10 con 0x55dddf902f70 2024-03-28T22:42:09.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.614+0000 7f6e5e000640 1 -- 172.21.15.43:0/3392574454 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6e68011b40 con 0x55dddf902f70 2024-03-28T22:42:09.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.614+0000 7f6e7a3f6740 10 client.27566 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:09.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.614+0000 7f6e7a3f6740 1 -- 172.21.15.43:0/3392574454 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55dddf53b170 con 0x55dddf902f70 2024-03-28T22:42:09.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.615+0000 7f6e5e000640 1 -- 172.21.15.43:0/3392574454 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e68003d10 con 0x55dddf902f70 2024-03-28T22:42:09.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.615+0000 7f6e5e000640 1 -- 172.21.15.43:0/3392574454 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6e68029040 con 0x55dddf902f70 2024-03-28T22:42:09.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.615+0000 7f6e7a3f6740 10 client.27566 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:09.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.615+0000 7f6e7a3f6740 10 client.27566 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:42:09.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.615+0000 7f6e7a3f6740 20 client.27566 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:09.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.615+0000 7f6e7a3f6740 1 --2- 172.21.15.43:0/3392574454 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55dddf91c690 0x55dddf91ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.615+0000 7f6e7a3f6740 4 client.27566 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:42:09.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.615+0000 7f6e7a3f6740 1 -- 172.21.15.43:0/3392574454 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dddf874180 con 0x55dddf91c690 2024-03-28T22:42:09.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.616+0000 7f6e67e00640 1 --2- 172.21.15.43:0/3392574454 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55dddf91c690 0x55dddf91ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.616+0000 7f6e5e000640 10 client.27566 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:09.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.616+0000 7f6e67e00640 1 --2- 172.21.15.43:0/3392574454 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55dddf91c690 0x55dddf91ea80 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c18ead8bf630754 server_cookie=fe19836601a10443 in_seq=0 out_seq=0 2024-03-28T22:42:09.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.617+0000 7f6e5e000640 1 -- 172.21.15.43:0/3392574454 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dddf874180 con 0x55dddf91c690 2024-03-28T22:42:09.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.617+0000 7f6e5e000640 10 client.27566 handle_command_reply: tid=0 2024-03-28T22:42:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.617+0000 7f6e7a3f6740 1 client.27566 shutdown 2024-03-28T22:42:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.617+0000 7f6e7a3f6740 1 -- 172.21.15.43:0/3392574454 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dddf902f70 msgr2=0x55dddf909460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.617+0000 7f6e7a3f6740 1 --2- 172.21.15.43:0/3392574454 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dddf902f70 0x55dddf909460 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f6e68003c80 tx=0x7f6e68013040 comp rx=0 tx=0).stop 2024-03-28T22:42:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.617+0000 7f6e7a3f6740 1 -- 172.21.15.43:0/3392574454 shutdown_connections 2024-03-28T22:42:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.617+0000 7f6e7a3f6740 1 --2- 172.21.15.43:0/3392574454 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dddf90c470 0x55dddf909ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.617+0000 7f6e7a3f6740 1 --2- 172.21.15.43:0/3392574454 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dddf903890 0x55dddf9099a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.618+0000 7f6e7a3f6740 1 --2- 172.21.15.43:0/3392574454 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55dddf91c690 0x55dddf91ea80 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.618+0000 7f6e7a3f6740 1 --2- 172.21.15.43:0/3392574454 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dddf902f70 0x55dddf909460 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.618+0000 7f6e7a3f6740 1 -- 172.21.15.43:0/3392574454 >> 172.21.15.43:0/3392574454 conn(0x55dddf8fd6a0 msgr2=0x55dddf8ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.618+0000 7f6e5e000640 0 client.27566 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.618+0000 7f6e7a3f6740 1 -- 172.21.15.43:0/3392574454 shutdown_connections 2024-03-28T22:42:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.618+0000 7f6e7a3f6740 1 -- 172.21.15.43:0/3392574454 wait complete. 2024-03-28T22:42:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.618+0000 7f6e7a3f6740 20 client.27566 trim_cache size 0 max 16384 2024-03-28T22:42:09.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.640+0000 7f6e7a3f6740 1 Processor -- start 2024-03-28T22:42:09.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.640+0000 7f6e7a3f6740 1 -- start start 2024-03-28T22:42:09.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.640+0000 7f6e7a3f6740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dddf902f70 0x55dddf918000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.640+0000 7f6e7a3f6740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dddf903890 0x55dddf908c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.640+0000 7f6e7a3f6740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dddf90c470 0x55dddf924130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.640+0000 7f6e7a3f6740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6e701b52e0 con 0x55dddf902f70 2024-03-28T22:42:09.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.640+0000 7f6e7a3f6740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f6e701b6e20 con 0x55dddf90c470 2024-03-28T22:42:09.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.640+0000 7f6e7a3f6740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f6e701b59b0 con 0x55dddf903890 2024-03-28T22:42:09.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.641+0000 7f6e67e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dddf903890 0x55dddf908c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.641+0000 7f6e67e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dddf903890 0x55dddf908c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:44538/0 (socket says 172.21.15.43:44538) 2024-03-28T22:42:09.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.641+0000 7f6e67e00640 1 -- 172.21.15.43:0/3395536312 learned_addr learned my addr 172.21.15.43:0/3395536312 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:09.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.641+0000 7f6e75600640 1 --2- 172.21.15.43:0/3395536312 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dddf902f70 0x55dddf918000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.641+0000 7f6e75600640 1 --2- 172.21.15.43:0/3395536312 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dddf90c470 0x55dddf924130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.641+0000 7f6e67e00640 1 -- 172.21.15.43:0/3395536312 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dddf90c470 msgr2=0x55dddf924130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:09.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.641+0000 7f6e67e00640 1 --2- 172.21.15.43:0/3395536312 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dddf90c470 0x55dddf924130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.641+0000 7f6e67e00640 1 -- 172.21.15.43:0/3395536312 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dddf902f70 msgr2=0x55dddf918000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:09.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.641+0000 7f6e67e00640 1 --2- 172.21.15.43:0/3395536312 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dddf902f70 0x55dddf918000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.641+0000 7f6e67e00640 1 -- 172.21.15.43:0/3395536312 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dddf8f7790 con 0x55dddf903890 2024-03-28T22:42:09.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.641+0000 7f6e75600640 1 --2- 172.21.15.43:0/3395536312 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dddf90c470 0x55dddf924130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:42:09.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.641+0000 7f6e67e00640 1 --2- 172.21.15.43:0/3395536312 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dddf903890 0x55dddf908c10 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6e5802c8e0 tx=0x7f6e58009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b4bc7dbab2fc46f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:09.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.642+0000 7f6e65200640 1 -- 172.21.15.43:0/3395536312 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e58004180 con 0x55dddf903890 2024-03-28T22:42:09.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.642+0000 7f6e65200640 1 -- 172.21.15.43:0/3395536312 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e58004320 con 0x55dddf903890 2024-03-28T22:42:09.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.642+0000 7f6e65200640 1 -- 172.21.15.43:0/3395536312 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e58048260 con 0x55dddf903890 2024-03-28T22:42:09.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.643+0000 7f6e7a3f6740 1 -- 172.21.15.43:0/3395536312 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dddf903890 msgr2=0x55dddf908c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:09.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.643+0000 7f6e7a3f6740 1 --2- 172.21.15.43:0/3395536312 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dddf903890 0x55dddf908c10 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6e5802c8e0 tx=0x7f6e58009260 comp rx=0 tx=0).stop 2024-03-28T22:42:09.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.643+0000 7f6e7a3f6740 1 -- 172.21.15.43:0/3395536312 shutdown_connections 2024-03-28T22:42:09.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.643+0000 7f6e7a3f6740 1 --2- 172.21.15.43:0/3395536312 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dddf90c470 0x55dddf924130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.643+0000 7f6e7a3f6740 1 --2- 172.21.15.43:0/3395536312 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dddf903890 0x55dddf908c10 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.643+0000 7f6e7a3f6740 1 --2- 172.21.15.43:0/3395536312 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dddf902f70 0x55dddf918000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.643+0000 7f6e7a3f6740 1 -- 172.21.15.43:0/3395536312 >> 172.21.15.43:0/3395536312 conn(0x55dddf8fd6a0 msgr2=0x55dddf8fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:09.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.643+0000 7f6e7a3f6740 1 -- 172.21.15.43:0/3395536312 shutdown_connections 2024-03-28T22:42:09.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.643+0000 7f6e7a3f6740 1 -- 172.21.15.43:0/3395536312 wait complete. 2024-03-28T22:42:09.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.643+0000 7f6e7a3f6740 1 -- start start 2024-03-28T22:42:09.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.644+0000 7f6e7a3f6740 1 Processor -- start 2024-03-28T22:42:09.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.644+0000 7f6e7a3f6740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dddf902f70 0x55dddf906230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.644+0000 7f6e7a3f6740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dddf903890 0x55dddf906770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.644+0000 7f6e7a3f6740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dddf90c470 0x55dddf906cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.644+0000 7f6e7a3f6740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6e701b6750 con 0x55dddf90c470 2024-03-28T22:42:09.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.644+0000 7f6e7a3f6740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f6e701b6080 con 0x55dddf903890 2024-03-28T22:42:09.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.644+0000 7f6e7a3f6740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f6e701b32c0 con 0x55dddf902f70 2024-03-28T22:42:09.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.644+0000 7f6e7a3f6740 10 client.0 _ll_register_callbacks cb 0x7f6e600d1f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:09.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.644+0000 7f6e67e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dddf903890 0x55dddf906770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.644+0000 7f6e75600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dddf902f70 0x55dddf906230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.644+0000 7f6e67e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dddf903890 0x55dddf906770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44790/0 (socket says 172.21.15.43:44790) 2024-03-28T22:42:09.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.644+0000 7f6e67e00640 1 -- 172.21.15.43:0/2118344462 learned_addr learned my addr 172.21.15.43:0/2118344462 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:09.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.644+0000 7f6e75600640 1 --2- 172.21.15.43:0/2118344462 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dddf90c470 0x55dddf906cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.645+0000 7f6e67e00640 1 -- 172.21.15.43:0/2118344462 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dddf902f70 msgr2=0x55dddf906230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:09.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.645+0000 7f6e67e00640 1 --2- 172.21.15.43:0/2118344462 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dddf902f70 0x55dddf906230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.645+0000 7f6e67e00640 1 -- 172.21.15.43:0/2118344462 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dddf90c470 msgr2=0x55dddf906cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:09.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.645+0000 7f6e67e00640 1 --2- 172.21.15.43:0/2118344462 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dddf90c470 0x55dddf906cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.645+0000 7f6e67e00640 1 -- 172.21.15.43:0/2118344462 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dddf981a00 con 0x55dddf903890 2024-03-28T22:42:09.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.645+0000 7f6e67e00640 1 --2- 172.21.15.43:0/2118344462 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dddf903890 0x55dddf906770 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f6e5802c750 tx=0x7f6e5802cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25c00e5b30dd6153 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:09.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.645+0000 7f6e5e000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:42:09.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.645+0000 7f6e5e000640 1 -- 172.21.15.43:0/2118344462 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e58055060 con 0x55dddf903890 2024-03-28T22:42:09.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.645+0000 7f6e7a3f6740 1 -- 172.21.15.43:0/2118344462 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e58004320 con 0x55dddf903890 2024-03-28T22:42:09.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.645+0000 7f6e5e000640 1 -- 172.21.15.43:0/2118344462 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f6e58046040 con 0x55dddf903890 2024-03-28T22:42:09.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.646+0000 7f6e5e000640 1 -- 172.21.15.43:0/2118344462 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6e5805c050 con 0x55dddf903890 2024-03-28T22:42:09.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.646+0000 7f6e7a3f6740 10 client.27572 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:09.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.646+0000 7f6e7a3f6740 1 -- 172.21.15.43:0/2118344462 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55dddf982cc0 con 0x55dddf903890 2024-03-28T22:42:09.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.646+0000 7f6e5e000640 1 -- 172.21.15.43:0/2118344462 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e58004320 con 0x55dddf903890 2024-03-28T22:42:09.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.646+0000 7f6e5e000640 1 -- 172.21.15.43:0/2118344462 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6e58005070 con 0x55dddf903890 2024-03-28T22:42:09.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.646+0000 7f6e7a3f6740 10 client.27572 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:09.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.646+0000 7f6e7a3f6740 10 client.27572 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:42:09.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.646+0000 7f6e7a3f6740 20 client.27572 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:09.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.646+0000 7f6e7a3f6740 1 --2- 172.21.15.43:0/2118344462 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55dddf983310 0x55dddf985730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.646+0000 7f6e7a3f6740 4 client.27572 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:42:09.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.646+0000 7f6e7a3f6740 1 -- 172.21.15.43:0/2118344462 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55dddf86e4d0 con 0x55dddf983310 2024-03-28T22:42:09.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.647+0000 7f6e75600640 1 --2- 172.21.15.43:0/2118344462 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55dddf983310 0x55dddf985730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.647+0000 7f6e75600640 1 --2- 172.21.15.43:0/2118344462 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55dddf983310 0x55dddf985730 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25166be4b1b1c6e4 server_cookie=c9c4e24612b699b0 in_seq=0 out_seq=0 2024-03-28T22:42:09.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.647+0000 7f6e5e000640 10 client.27572 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:09.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.649+0000 7f6e5e000640 1 -- 172.21.15.43:0/2118344462 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55dddf86e4d0 con 0x55dddf983310 2024-03-28T22:42:09.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.649+0000 7f6e5e000640 10 client.27572 handle_command_reply: tid=0 2024-03-28T22:42:09.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.649+0000 7f6e7a3f6740 1 client.27572 shutdown 2024-03-28T22:42:09.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.649+0000 7f6e7a3f6740 1 -- 172.21.15.43:0/2118344462 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dddf903890 msgr2=0x55dddf906770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:09.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.649+0000 7f6e7a3f6740 1 --2- 172.21.15.43:0/2118344462 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dddf903890 0x55dddf906770 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f6e5802c750 tx=0x7f6e5802cc80 comp rx=0 tx=0).stop 2024-03-28T22:42:09.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.649+0000 7f6e7a3f6740 1 -- 172.21.15.43:0/2118344462 shutdown_connections 2024-03-28T22:42:09.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.649+0000 7f6e7a3f6740 1 --2- 172.21.15.43:0/2118344462 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55dddf983310 0x55dddf985730 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.649+0000 7f6e7a3f6740 1 --2- 172.21.15.43:0/2118344462 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dddf903890 0x55dddf906770 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.650+0000 7f6e5e000640 0 client.27572 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:09.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.650+0000 7f6e7a3f6740 1 --2- 172.21.15.43:0/2118344462 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dddf90c470 0x55dddf906cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.650+0000 7f6e7a3f6740 1 --2- 172.21.15.43:0/2118344462 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dddf902f70 0x55dddf906230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.650+0000 7f6e7a3f6740 1 -- 172.21.15.43:0/2118344462 >> 172.21.15.43:0/2118344462 conn(0x55dddf8fd6a0 msgr2=0x55dddf8fbe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:09.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.650+0000 7f6e7a3f6740 1 -- 172.21.15.43:0/2118344462 shutdown_connections 2024-03-28T22:42:09.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.650+0000 7f6e7a3f6740 1 -- 172.21.15.43:0/2118344462 wait complete. 2024-03-28T22:42:09.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.650+0000 7f6e7a3f6740 20 client.27572 trim_cache size 0 max 16384 2024-03-28T22:42:09.651 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:42:09.651 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-03-28T22:42:09.651 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "67ca9a24-1061-4ddc-923a-a67abf02d9a4", 2024-03-28T22:42:09.651 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-03-28T22:42:09.651 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:42:09.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.651+0000 7f6e77600640 1 -- 172.21.15.43:0/342133720 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6e540a1ef0 msgr2=0x7f6e540a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:09.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.651+0000 7f6e77600640 1 --2- 172.21.15.43:0/342133720 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6e540a1ef0 0x7f6e540a43b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6e600cbd50 tx=0x7f6e600d44a0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.651+0000 7f6e77600640 1 -- 172.21.15.43:0/342133720 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6e701007d0 msgr2=0x7f6e7019e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:09.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.651+0000 7f6e77600640 1 --2- 172.21.15.43:0/342133720 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6e701007d0 0x7f6e7019e660 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f6e6000e9f0 tx=0x7f6e6000eeb0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.651+0000 7f6e77600640 1 -- 172.21.15.43:0/342133720 shutdown_connections 2024-03-28T22:42:09.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.651+0000 7f6e77600640 1 --2- 172.21.15.43:0/342133720 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6e540a1ef0 0x7f6e540a43b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.651+0000 7f6e77600640 1 --2- 172.21.15.43:0/342133720 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6e701010f0 0x7f6e7019eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.651+0000 7f6e77600640 1 --2- 172.21.15.43:0/342133720 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6e70105bd0 0x7f6e701a2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.651+0000 7f6e77600640 1 --2- 172.21.15.43:0/342133720 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6e701007d0 0x7f6e7019e660 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.651+0000 7f6e77600640 1 -- 172.21.15.43:0/342133720 >> 172.21.15.43:0/342133720 conn(0x7f6e700777b0 msgr2=0x7f6e70103580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:09.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.651+0000 7f6e77600640 1 -- 172.21.15.43:0/342133720 shutdown_connections 2024-03-28T22:42:09.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.651+0000 7f6e77600640 1 -- 172.21.15.43:0/342133720 wait complete. 2024-03-28T22:42:09.671 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:42:09.671 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:42:09.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.938+0000 7fcc66800640 1 -- 172.21.15.43:0/3547665254 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc60109900 msgr2=0x7fcc60109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:09.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.938+0000 7fcc66800640 1 --2- 172.21.15.43:0/3547665254 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc60109900 0x7fcc60109ce0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fcc50009930 tx=0x7fcc5002bc30 comp rx=0 tx=0).stop 2024-03-28T22:42:09.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.938+0000 7fcc66800640 1 -- 172.21.15.43:0/3547665254 shutdown_connections 2024-03-28T22:42:09.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.938+0000 7fcc66800640 1 --2- 172.21.15.43:0/3547665254 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc601042b0 0x7fcc601107c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.938+0000 7fcc66800640 1 --2- 172.21.15.43:0/3547665254 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc601038f0 0x7fcc60103d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.938+0000 7fcc66800640 1 --2- 172.21.15.43:0/3547665254 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc60109900 0x7fcc60109ce0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.938+0000 7fcc66800640 1 -- 172.21.15.43:0/3547665254 >> 172.21.15.43:0/3547665254 conn(0x7fcc600ff3a0 msgr2=0x7fcc601017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:09.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.938+0000 7fcc66800640 1 -- 172.21.15.43:0/3547665254 shutdown_connections 2024-03-28T22:42:09.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.938+0000 7fcc66800640 1 -- 172.21.15.43:0/3547665254 wait complete. 2024-03-28T22:42:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.939+0000 7fcc66800640 1 Processor -- start 2024-03-28T22:42:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.939+0000 7fcc66800640 1 -- start start 2024-03-28T22:42:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.940+0000 7fcc66800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc601038f0 0x7fcc6010e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.940+0000 7fcc66800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc601042b0 0x7fcc6010ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.940+0000 7fcc66800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc60109900 0x7fcc6010f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.940+0000 7fcc66800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcc600765a0 con 0x7fcc60109900 2024-03-28T22:42:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.940+0000 7fcc66800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fcc60076420 con 0x7fcc601038f0 2024-03-28T22:42:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.940+0000 7fcc66800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fcc60076720 con 0x7fcc601042b0 2024-03-28T22:42:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.940+0000 7fcc5f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc601038f0 0x7fcc6010e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.940+0000 7fcc5ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc601042b0 0x7fcc6010ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.940+0000 7fcc5f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc601038f0 0x7fcc6010e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44804/0 (socket says 172.21.15.43:44804) 2024-03-28T22:42:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.940+0000 7fcc5f400640 1 -- 172.21.15.43:0/2951896070 learned_addr learned my addr 172.21.15.43:0/2951896070 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:09.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.940+0000 7fcc5ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc601042b0 0x7fcc6010ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:44574/0 (socket says 172.21.15.43:44574) 2024-03-28T22:42:09.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.940+0000 7fcc5fe00640 1 --2- 172.21.15.43:0/2951896070 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc60109900 0x7fcc6010f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.940+0000 7fcc5f400640 1 -- 172.21.15.43:0/2951896070 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc601042b0 msgr2=0x7fcc6010ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:09.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.940+0000 7fcc5f400640 1 --2- 172.21.15.43:0/2951896070 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc601042b0 0x7fcc6010ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.940+0000 7fcc5f400640 1 -- 172.21.15.43:0/2951896070 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc60109900 msgr2=0x7fcc6010f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:09.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.940+0000 7fcc5f400640 1 --2- 172.21.15.43:0/2951896070 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc60109900 0x7fcc6010f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.940+0000 7fcc5f400640 1 -- 172.21.15.43:0/2951896070 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc601aa900 con 0x7fcc601038f0 2024-03-28T22:42:09.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.940+0000 7fcc5ea00640 1 --2- 172.21.15.43:0/2951896070 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc601042b0 0x7fcc6010ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:09.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.941+0000 7fcc5f400640 1 --2- 172.21.15.43:0/2951896070 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc601038f0 0x7fcc6010e910 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fcc5002c170 tx=0x7fcc50004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:09.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.941+0000 7fcc57e00640 1 -- 172.21.15.43:0/2951896070 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc50043020 con 0x7fcc601038f0 2024-03-28T22:42:09.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.941+0000 7fcc66800640 1 -- 172.21.15.43:0/2951896070 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc601aab90 con 0x7fcc601038f0 2024-03-28T22:42:09.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.941+0000 7fcc57e00640 1 -- 172.21.15.43:0/2951896070 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc500054e0 con 0x7fcc601038f0 2024-03-28T22:42:09.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.941+0000 7fcc57e00640 1 -- 172.21.15.43:0/2951896070 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc50003be0 con 0x7fcc601038f0 2024-03-28T22:42:09.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.941+0000 7fcc66800640 1 -- 172.21.15.43:0/2951896070 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc601ab130 con 0x7fcc601038f0 2024-03-28T22:42:09.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.945+0000 7fcc57e00640 1 -- 172.21.15.43:0/2951896070 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fcc50046020 con 0x7fcc601038f0 2024-03-28T22:42:09.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.946+0000 7fcc69602740 1 Processor -- start 2024-03-28T22:42:09.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.947+0000 7fcc69602740 1 -- start start 2024-03-28T22:42:09.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.947+0000 7fcc57e00640 1 --2- 172.21.15.43:0/2951896070 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcc440a1ef0 0x7fcc440a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.947+0000 7fcc69602740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587c305abc0 0x5587c3059910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.947+0000 7fcc69602740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587c3059ee0 0x5587c305a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.947+0000 7fcc57e00640 1 -- 172.21.15.43:0/2951896070 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fcc500efea0 con 0x7fcc601038f0 2024-03-28T22:42:09.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.947+0000 7fcc69602740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587c3063250 0x5587c3065640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.947+0000 7fcc69602740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5587c2fbe100 con 0x5587c3059ee0 2024-03-28T22:42:09.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.947+0000 7fcc69602740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5587c2f66ba0 con 0x5587c3063250 2024-03-28T22:42:09.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.947+0000 7fcc69602740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5587c2fba1a0 con 0x5587c305abc0 2024-03-28T22:42:09.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.947+0000 7fcc5fe00640 1 --2- 172.21.15.43:0/2951896070 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcc440a1ef0 0x7fcc440a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.948+0000 7fcc5f400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587c305abc0 0x5587c3059910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.948+0000 7fcc5f400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587c305abc0 0x5587c3059910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:44578/0 (socket says 172.21.15.43:44578) 2024-03-28T22:42:09.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.948+0000 7fcc5f400640 1 -- 172.21.15.43:0/4210102862 learned_addr learned my addr 172.21.15.43:0/4210102862 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:09.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.948+0000 7fcc5fe00640 1 --2- 172.21.15.43:0/4210102862 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587c3063250 0x5587c3065640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.948+0000 7fcc5ea00640 1 --2- 172.21.15.43:0/4210102862 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587c3059ee0 0x5587c305a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.948+0000 7fcc5f400640 1 -- 172.21.15.43:0/4210102862 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587c3063250 msgr2=0x5587c3065640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:09.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.948+0000 7fcc5f400640 1 --2- 172.21.15.43:0/4210102862 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587c3063250 0x5587c3065640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.948+0000 7fcc5f400640 1 -- 172.21.15.43:0/4210102862 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587c3059ee0 msgr2=0x5587c305a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:09.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.948+0000 7fcc5f400640 1 --2- 172.21.15.43:0/4210102862 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587c3059ee0 0x5587c305a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.948+0000 7fcc5f400640 1 -- 172.21.15.43:0/4210102862 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587c2f971a0 con 0x5587c305abc0 2024-03-28T22:42:09.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.948+0000 7fcc5ea00640 1 --2- 172.21.15.43:0/4210102862 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587c3059ee0 0x5587c305a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:42:09.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.948+0000 7fcc5f400640 1 --2- 172.21.15.43:0/4210102862 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587c305abc0 0x5587c3059910 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fcc500c4810 tx=0x7fcc50004ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4bfb147f724711e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:09.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.949+0000 7fcc54c00640 1 -- 172.21.15.43:0/4210102862 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc50005690 con 0x5587c305abc0 2024-03-28T22:42:09.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.949+0000 7fcc54c00640 1 -- 172.21.15.43:0/4210102862 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc5002cb90 con 0x5587c305abc0 2024-03-28T22:42:09.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.949+0000 7fcc54c00640 1 -- 172.21.15.43:0/4210102862 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc500dc0b0 con 0x5587c305abc0 2024-03-28T22:42:09.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.949+0000 7fcc69602740 1 -- 172.21.15.43:0/4210102862 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587c305abc0 msgr2=0x5587c3059910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:09.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.949+0000 7fcc5fe00640 1 --2- 172.21.15.43:0/2951896070 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcc440a1ef0 0x7fcc440a43b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fcc58009fd0 tx=0x7fcc58009500 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:09.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.949+0000 7fcc69602740 1 --2- 172.21.15.43:0/4210102862 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587c305abc0 0x5587c3059910 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fcc500c4810 tx=0x7fcc50004ee0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.950+0000 7fcc69602740 1 -- 172.21.15.43:0/4210102862 shutdown_connections 2024-03-28T22:42:09.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.950+0000 7fcc69602740 1 --2- 172.21.15.43:0/4210102862 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587c3063250 0x5587c3065640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.950+0000 7fcc69602740 1 --2- 172.21.15.43:0/4210102862 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587c3059ee0 0x5587c305a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.950+0000 7fcc69602740 1 --2- 172.21.15.43:0/4210102862 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587c305abc0 0x5587c3059910 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.950+0000 7fcc69602740 1 -- 172.21.15.43:0/4210102862 >> 172.21.15.43:0/4210102862 conn(0x5587c3020a10 msgr2=0x5587c3054930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:09.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.950+0000 7fcc69602740 1 -- 172.21.15.43:0/4210102862 shutdown_connections 2024-03-28T22:42:09.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.950+0000 7fcc69602740 1 -- 172.21.15.43:0/4210102862 wait complete. 2024-03-28T22:42:09.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.950+0000 7fcc69602740 1 -- start start 2024-03-28T22:42:09.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.950+0000 7fcc69602740 1 Processor -- start 2024-03-28T22:42:09.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.951+0000 7fcc69602740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587c3059ee0 0x5587c3060240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.951+0000 7fcc69602740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587c305abc0 0x5587c3060780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.951+0000 7fcc69602740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587c3063250 0x5587c3060cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.951+0000 7fcc69602740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5587c2fba1a0 con 0x5587c3063250 2024-03-28T22:42:09.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.951+0000 7fcc69602740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5587c2f5c250 con 0x5587c3059ee0 2024-03-28T22:42:09.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.951+0000 7fcc69602740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5587c3065e80 con 0x5587c305abc0 2024-03-28T22:42:09.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.951+0000 7fcc69602740 10 client.0 _ll_register_callbacks cb 0x5587c2cd1be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:09.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.951+0000 7fcc5f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587c3059ee0 0x5587c3060240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.951+0000 7fcc5ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587c305abc0 0x5587c3060780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.951+0000 7fcc5ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587c305abc0 0x5587c3060780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:44584/0 (socket says 172.21.15.43:44584) 2024-03-28T22:42:09.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.951+0000 7fcc5ea00640 1 -- 172.21.15.43:0/534563549 learned_addr learned my addr 172.21.15.43:0/534563549 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:09.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.951+0000 7fcc5f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587c3059ee0 0x5587c3060240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44832/0 (socket says 172.21.15.43:44832) 2024-03-28T22:42:09.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.952+0000 7fcc5fe00640 1 --2- 172.21.15.43:0/534563549 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587c3063250 0x5587c3060cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.952+0000 7fcc5ea00640 1 -- 172.21.15.43:0/534563549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587c3059ee0 msgr2=0x5587c3060240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:09.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.952+0000 7fcc5ea00640 1 --2- 172.21.15.43:0/534563549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587c3059ee0 0x5587c3060240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.952+0000 7fcc5ea00640 1 -- 172.21.15.43:0/534563549 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587c3063250 msgr2=0x5587c3060cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:09.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.952+0000 7fcc5ea00640 1 --2- 172.21.15.43:0/534563549 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587c3063250 0x5587c3060cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.952+0000 7fcc5ea00640 1 -- 172.21.15.43:0/534563549 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587c2f78960 con 0x5587c305abc0 2024-03-28T22:42:09.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.952+0000 7fcc5f400640 1 --2- 172.21.15.43:0/534563549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587c3059ee0 0x5587c3060240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:09.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.952+0000 7fcc5ea00640 1 --2- 172.21.15.43:0/534563549 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587c305abc0 0x5587c3060780 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c0129b0 tx=0x7fcc4c012e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2900320c73ab8a78 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:09.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.952+0000 7fcc49600640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:42:09.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.952+0000 7fcc49600640 1 -- 172.21.15.43:0/534563549 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c00fa00 con 0x5587c305abc0 2024-03-28T22:42:09.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.952+0000 7fcc69602740 1 -- 172.21.15.43:0/534563549 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc5002cb90 con 0x5587c305abc0 2024-03-28T22:42:09.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.952+0000 7fcc49600640 1 -- 172.21.15.43:0/534563549 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fcc4c00fba0 con 0x5587c305abc0 2024-03-28T22:42:09.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.953+0000 7fcc49600640 1 -- 172.21.15.43:0/534563549 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcc4c00fda0 con 0x5587c305abc0 2024-03-28T22:42:09.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.953+0000 7fcc69602740 10 client.27529 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:09.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.953+0000 7fcc69602740 1 -- 172.21.15.43:0/534563549 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5587c2d9c7d0 con 0x5587c305abc0 2024-03-28T22:42:09.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.953+0000 7fcc49600640 1 -- 172.21.15.43:0/534563549 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc5002cb90 con 0x5587c305abc0 2024-03-28T22:42:09.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.953+0000 7fcc49600640 1 -- 172.21.15.43:0/534563549 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fcc4c01c540 con 0x5587c305abc0 2024-03-28T22:42:09.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.953+0000 7fcc69602740 10 client.27529 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:09.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.953+0000 7fcc69602740 10 client.27529 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:42:09.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.953+0000 7fcc69602740 20 client.27529 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:09.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.953+0000 7fcc69602740 1 --2- 172.21.15.43:0/534563549 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5587c30733a0 0x5587c3075820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.953+0000 7fcc69602740 4 client.27529 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:42:09.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.953+0000 7fcc69602740 1 -- 172.21.15.43:0/534563549 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5587c2fcb640 con 0x5587c30733a0 2024-03-28T22:42:09.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.954+0000 7fcc5f400640 1 --2- 172.21.15.43:0/534563549 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5587c30733a0 0x5587c3075820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.954+0000 7fcc5f400640 1 --2- 172.21.15.43:0/534563549 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5587c30733a0 0x5587c3075820 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0df78d07b6e6063 server_cookie=c00673e2cf2dfe2 in_seq=0 out_seq=0 2024-03-28T22:42:09.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.954+0000 7fcc49600640 10 client.27529 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:09.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.956+0000 7fcc49600640 1 -- 172.21.15.43:0/534563549 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5587c2fcb640 con 0x5587c30733a0 2024-03-28T22:42:09.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.956+0000 7fcc49600640 10 client.27529 handle_command_reply: tid=0 2024-03-28T22:42:09.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.956+0000 7fcc69602740 1 client.27529 shutdown 2024-03-28T22:42:09.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.956+0000 7fcc69602740 1 -- 172.21.15.43:0/534563549 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587c305abc0 msgr2=0x5587c3060780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:09.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.956+0000 7fcc69602740 1 --2- 172.21.15.43:0/534563549 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587c305abc0 0x5587c3060780 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c0129b0 tx=0x7fcc4c012e70 comp rx=0 tx=0).stop 2024-03-28T22:42:09.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.956+0000 7fcc69602740 1 -- 172.21.15.43:0/534563549 shutdown_connections 2024-03-28T22:42:09.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.956+0000 7fcc69602740 1 --2- 172.21.15.43:0/534563549 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587c3063250 0x5587c3060cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.956+0000 7fcc69602740 1 --2- 172.21.15.43:0/534563549 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587c305abc0 0x5587c3060780 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.956+0000 7fcc69602740 1 --2- 172.21.15.43:0/534563549 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5587c30733a0 0x5587c3075820 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.956+0000 7fcc69602740 1 --2- 172.21.15.43:0/534563549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587c3059ee0 0x5587c3060240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.956+0000 7fcc69602740 1 -- 172.21.15.43:0/534563549 >> 172.21.15.43:0/534563549 conn(0x5587c3020a10 msgr2=0x5587c3056680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:09.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.956+0000 7fcc49600640 0 client.27529 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:09.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.956+0000 7fcc69602740 1 -- 172.21.15.43:0/534563549 shutdown_connections 2024-03-28T22:42:09.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.956+0000 7fcc69602740 1 -- 172.21.15.43:0/534563549 wait complete. 2024-03-28T22:42:09.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.957+0000 7fcc69602740 20 client.27529 trim_cache size 0 max 16384 2024-03-28T22:42:09.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.978+0000 7fcc69602740 1 Processor -- start 2024-03-28T22:42:09.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.978+0000 7fcc69602740 1 -- start start 2024-03-28T22:42:09.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.978+0000 7fcc69602740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587c3059ee0 0x5587c3062970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.978+0000 7fcc69602740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587c305abc0 0x5587c306eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.978+0000 7fcc69602740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587c3063250 0x5587c3056490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.978+0000 7fcc69602740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcc601b51a0 con 0x5587c305abc0 2024-03-28T22:42:09.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.978+0000 7fcc69602740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fcc601b6ce0 con 0x5587c3059ee0 2024-03-28T22:42:09.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.978+0000 7fcc69602740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fcc601b5870 con 0x5587c3063250 2024-03-28T22:42:09.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.978+0000 7fcc5f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587c3059ee0 0x5587c3062970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.978+0000 7fcc5f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587c3059ee0 0x5587c3062970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44846/0 (socket says 172.21.15.43:44846) 2024-03-28T22:42:09.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.978+0000 7fcc5f400640 1 -- 172.21.15.43:0/2929611999 learned_addr learned my addr 172.21.15.43:0/2929611999 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:09.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.978+0000 7fcc5ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587c305abc0 0x5587c306eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.979+0000 7fcc5fe00640 1 --2- 172.21.15.43:0/2929611999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587c3063250 0x5587c3056490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.979+0000 7fcc5f400640 1 -- 172.21.15.43:0/2929611999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587c3063250 msgr2=0x5587c3056490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:09.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.979+0000 7fcc5f400640 1 --2- 172.21.15.43:0/2929611999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587c3063250 0x5587c3056490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.979+0000 7fcc5f400640 1 -- 172.21.15.43:0/2929611999 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587c305abc0 msgr2=0x5587c306eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:09.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.979+0000 7fcc5f400640 1 --2- 172.21.15.43:0/2929611999 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587c305abc0 0x5587c306eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.979+0000 7fcc5f400640 1 -- 172.21.15.43:0/2929611999 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587c2cd23f0 con 0x5587c3059ee0 2024-03-28T22:42:09.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.979+0000 7fcc5ea00640 1 --2- 172.21.15.43:0/2929611999 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587c305abc0 0x5587c306eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:09.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.979+0000 7fcc5f400640 1 --2- 172.21.15.43:0/2929611999 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587c3059ee0 0x5587c3062970 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fcc5002bc70 tx=0x7fcc50004ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3530b63bcd561b34 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:09.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.979+0000 7fcc4be00640 1 -- 172.21.15.43:0/2929611999 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc5002cb90 con 0x5587c3059ee0 2024-03-28T22:42:09.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.979+0000 7fcc4be00640 1 -- 172.21.15.43:0/2929611999 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc50005690 con 0x5587c3059ee0 2024-03-28T22:42:09.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.979+0000 7fcc4be00640 1 -- 172.21.15.43:0/2929611999 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc500c06f0 con 0x5587c3059ee0 2024-03-28T22:42:09.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.980+0000 7fcc69602740 1 -- 172.21.15.43:0/2929611999 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587c3059ee0 msgr2=0x5587c3062970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:09.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.980+0000 7fcc69602740 1 --2- 172.21.15.43:0/2929611999 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587c3059ee0 0x5587c3062970 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fcc5002bc70 tx=0x7fcc50004ee0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.980+0000 7fcc69602740 1 -- 172.21.15.43:0/2929611999 shutdown_connections 2024-03-28T22:42:09.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.980+0000 7fcc69602740 1 --2- 172.21.15.43:0/2929611999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587c3063250 0x5587c3056490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.980+0000 7fcc69602740 1 --2- 172.21.15.43:0/2929611999 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587c305abc0 0x5587c306eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.980+0000 7fcc69602740 1 --2- 172.21.15.43:0/2929611999 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587c3059ee0 0x5587c3062970 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.980+0000 7fcc69602740 1 -- 172.21.15.43:0/2929611999 >> 172.21.15.43:0/2929611999 conn(0x5587c3020a10 msgr2=0x5587c3055d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:09.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.981+0000 7fcc69602740 1 -- 172.21.15.43:0/2929611999 shutdown_connections 2024-03-28T22:42:09.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.981+0000 7fcc69602740 1 -- 172.21.15.43:0/2929611999 wait complete. 2024-03-28T22:42:09.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.981+0000 7fcc69602740 1 -- start start 2024-03-28T22:42:09.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.981+0000 7fcc69602740 1 Processor -- start 2024-03-28T22:42:09.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.982+0000 7fcc69602740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587c3059ee0 0x5587c3066c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.982+0000 7fcc69602740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587c305abc0 0x5587c30671a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.982+0000 7fcc69602740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587c3063250 0x5587c30676e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.982+0000 7fcc69602740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcc601b6610 con 0x5587c305abc0 2024-03-28T22:42:09.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.982+0000 7fcc69602740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fcc601b5f40 con 0x5587c3059ee0 2024-03-28T22:42:09.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.982+0000 7fcc69602740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fcc601b3180 con 0x5587c3063250 2024-03-28T22:42:09.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.982+0000 7fcc69602740 10 client.0 _ll_register_callbacks cb 0x7fcc4c009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:09.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.982+0000 7fcc5f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587c3059ee0 0x5587c3066c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.982+0000 7fcc5ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587c305abc0 0x5587c30671a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.982+0000 7fcc5ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587c305abc0 0x5587c30671a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:48602/0 (socket says 172.21.15.43:48602) 2024-03-28T22:42:09.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.982+0000 7fcc5ea00640 1 -- 172.21.15.43:0/689654091 learned_addr learned my addr 172.21.15.43:0/689654091 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:09.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.982+0000 7fcc5f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587c3059ee0 0x5587c3066c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:44856/0 (socket says 172.21.15.43:44856) 2024-03-28T22:42:09.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.982+0000 7fcc5fe00640 1 --2- 172.21.15.43:0/689654091 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587c3063250 0x5587c30676e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.982+0000 7fcc5ea00640 1 -- 172.21.15.43:0/689654091 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587c3063250 msgr2=0x5587c30676e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:09.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.982+0000 7fcc5ea00640 1 --2- 172.21.15.43:0/689654091 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587c3063250 0x5587c30676e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.982+0000 7fcc5ea00640 1 -- 172.21.15.43:0/689654091 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587c3059ee0 msgr2=0x5587c3066c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:09.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.982+0000 7fcc5ea00640 1 --2- 172.21.15.43:0/689654091 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587c3059ee0 0x5587c3066c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.982+0000 7fcc5ea00640 1 -- 172.21.15.43:0/689654091 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587c2c920f0 con 0x5587c305abc0 2024-03-28T22:42:09.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.982+0000 7fcc5f400640 1 --2- 172.21.15.43:0/689654091 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587c3059ee0 0x5587c3066c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:09.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.983+0000 7fcc5ea00640 1 --2- 172.21.15.43:0/689654091 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587c305abc0 0x5587c30671a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c0092e0 tx=0x7fcc4c01cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68380c13ce632b48 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:09.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.983+0000 7fcc49600640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:42:09.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.983+0000 7fcc49600640 1 -- 172.21.15.43:0/689654091 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c02f060 con 0x5587c305abc0 2024-03-28T22:42:09.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.983+0000 7fcc49600640 1 -- 172.21.15.43:0/689654091 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fcc4c01c630 con 0x5587c305abc0 2024-03-28T22:42:09.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.983+0000 7fcc69602740 1 -- 172.21.15.43:0/689654091 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5587c2fc5990 con 0x5587c305abc0 2024-03-28T22:42:09.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.983+0000 7fcc49600640 1 -- 172.21.15.43:0/689654091 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcc4c036050 con 0x5587c305abc0 2024-03-28T22:42:09.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.983+0000 7fcc69602740 10 client.17370 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:09.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.983+0000 7fcc69602740 1 -- 172.21.15.43:0/689654091 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5587c30d9b40 con 0x5587c305abc0 2024-03-28T22:42:09.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.984+0000 7fcc49600640 1 -- 172.21.15.43:0/689654091 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5587c2fc5990 con 0x5587c305abc0 2024-03-28T22:42:09.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.984+0000 7fcc49600640 1 -- 172.21.15.43:0/689654091 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fcc4c003a30 con 0x5587c305abc0 2024-03-28T22:42:09.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.984+0000 7fcc69602740 10 client.17370 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:09.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.984+0000 7fcc69602740 10 client.17370 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:42:09.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.984+0000 7fcc69602740 20 client.17370 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:09.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.984+0000 7fcc69602740 1 --2- 172.21.15.43:0/689654091 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5587c30da240 0x5587c30dc660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:09.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.984+0000 7fcc69602740 4 client.17370 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:42:09.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.984+0000 7fcc69602740 1 -- 172.21.15.43:0/689654091 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5587c2f43400 con 0x5587c30da240 2024-03-28T22:42:09.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.984+0000 7fcc5f400640 1 --2- 172.21.15.43:0/689654091 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5587c30da240 0x5587c30dc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:09.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.985+0000 7fcc5f400640 1 --2- 172.21.15.43:0/689654091 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5587c30da240 0x5587c30dc660 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=651d2b7b27b17ad server_cookie=b874882c7360ecc2 in_seq=0 out_seq=0 2024-03-28T22:42:09.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.985+0000 7fcc49600640 10 client.17370 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:09.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.985+0000 7fcc49600640 1 -- 172.21.15.43:0/689654091 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5587c2f43400 con 0x5587c30da240 2024-03-28T22:42:09.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.985+0000 7fcc49600640 10 client.17370 handle_command_reply: tid=0 2024-03-28T22:42:09.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.985+0000 7fcc69602740 1 client.17370 shutdown 2024-03-28T22:42:09.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.986+0000 7fcc69602740 1 -- 172.21.15.43:0/689654091 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587c305abc0 msgr2=0x5587c30671a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:09.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.986+0000 7fcc69602740 1 --2- 172.21.15.43:0/689654091 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587c305abc0 0x5587c30671a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c0092e0 tx=0x7fcc4c01cef0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.986+0000 7fcc69602740 1 -- 172.21.15.43:0/689654091 shutdown_connections 2024-03-28T22:42:09.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.986+0000 7fcc69602740 1 --2- 172.21.15.43:0/689654091 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5587c3063250 0x5587c30676e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.986+0000 7fcc69602740 1 --2- 172.21.15.43:0/689654091 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5587c305abc0 0x5587c30671a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.986+0000 7fcc69602740 1 --2- 172.21.15.43:0/689654091 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5587c30da240 0x5587c30dc660 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.986+0000 7fcc69602740 1 --2- 172.21.15.43:0/689654091 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5587c3059ee0 0x5587c3066c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.986+0000 7fcc69602740 1 -- 172.21.15.43:0/689654091 >> 172.21.15.43:0/689654091 conn(0x5587c3020a10 msgr2=0x5587c3053ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:09.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.986+0000 7fcc49600640 0 client.17370 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:09.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.986+0000 7fcc69602740 1 -- 172.21.15.43:0/689654091 shutdown_connections 2024-03-28T22:42:09.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.986+0000 7fcc69602740 1 -- 172.21.15.43:0/689654091 wait complete. 2024-03-28T22:42:09.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.986+0000 7fcc69602740 20 client.17370 trim_cache size 0 max 16384 2024-03-28T22:42:09.988 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:42:09.988 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:42:09.988 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-03-28T22:42:09.988 INFO:teuthology.orchestra.run.smithi043.stdout: "67ca9a24-1061-4ddc-923a-a67abf02d9a4": { 2024-03-28T22:42:09.988 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-03-28T22:42:09.988 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "67ca9a24-1061-4ddc-923a-a67abf02d9a4", 2024-03-28T22:42:09.989 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-03-28T22:42:09.989 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:42:09.989 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:42:09.989 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:42:09.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.987+0000 7fcc66800640 1 -- 172.21.15.43:0/2951896070 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcc440a1ef0 msgr2=0x7fcc440a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:09.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.987+0000 7fcc66800640 1 --2- 172.21.15.43:0/2951896070 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcc440a1ef0 0x7fcc440a43b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fcc58009fd0 tx=0x7fcc58009500 comp rx=0 tx=0).stop 2024-03-28T22:42:09.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.987+0000 7fcc66800640 1 -- 172.21.15.43:0/2951896070 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc601038f0 msgr2=0x7fcc6010e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:09.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.987+0000 7fcc66800640 1 --2- 172.21.15.43:0/2951896070 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc601038f0 0x7fcc6010e910 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fcc5002c170 tx=0x7fcc50004360 comp rx=0 tx=0).stop 2024-03-28T22:42:09.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.987+0000 7fcc66800640 1 -- 172.21.15.43:0/2951896070 shutdown_connections 2024-03-28T22:42:09.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.987+0000 7fcc66800640 1 --2- 172.21.15.43:0/2951896070 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcc440a1ef0 0x7fcc440a43b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.987+0000 7fcc66800640 1 --2- 172.21.15.43:0/2951896070 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcc60109900 0x7fcc6010f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.987+0000 7fcc66800640 1 --2- 172.21.15.43:0/2951896070 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcc601042b0 0x7fcc6010ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.987+0000 7fcc66800640 1 --2- 172.21.15.43:0/2951896070 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc601038f0 0x7fcc6010e910 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:09.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.987+0000 7fcc66800640 1 -- 172.21.15.43:0/2951896070 >> 172.21.15.43:0/2951896070 conn(0x7fcc600ff3a0 msgr2=0x7fcc600ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:09.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.987+0000 7fcc66800640 1 -- 172.21.15.43:0/2951896070 shutdown_connections 2024-03-28T22:42:09.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:09.987+0000 7fcc66800640 1 -- 172.21.15.43:0/2951896070 wait complete. 2024-03-28T22:42:10.005 INFO:tasks.cephfs.filesystem:scrub status for tag:67ca9a24-1061-4ddc-923a-a67abf02d9a4 - {'path': '/', 'tag': '67ca9a24-1061-4ddc-923a-a67abf02d9a4', 'options': 'recursive,force'} 2024-03-28T22:42:10.364 INFO:tasks.workunit.client.0.smithi043.stdout: 1 331711 7.80 MB/sec execute 101 sec latency 55.501 ms 2024-03-28T22:42:11.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:10 smithi043 ceph-mon[31823]: pgmap v154: 129 pgs: 129 active+clean; 51 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 178 op/s 2024-03-28T22:42:11.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:10 smithi043 ceph-mon[31823]: scrub queued for path: / 2024-03-28T22:42:11.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:10 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:42:11.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:10 smithi043 ceph-mon[31823]: scrub summary: active paths [/] 2024-03-28T22:42:11.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:10 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:42:11.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:10 smithi172 ceph-mon[36566]: pgmap v154: 129 pgs: 129 active+clean; 51 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 178 op/s 2024-03-28T22:42:11.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:10 smithi172 ceph-mon[36566]: scrub queued for path: / 2024-03-28T22:42:11.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:10 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:42:11.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:10 smithi172 ceph-mon[36566]: scrub summary: active paths [/] 2024-03-28T22:42:11.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:10 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:42:11.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:10 smithi082 ceph-mon[36471]: pgmap v154: 129 pgs: 129 active+clean; 51 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 178 op/s 2024-03-28T22:42:11.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:10 smithi082 ceph-mon[36471]: scrub queued for path: / 2024-03-28T22:42:11.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:10 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:42:11.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:10 smithi082 ceph-mon[36471]: scrub summary: active paths [/] 2024-03-28T22:42:11.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:10 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:42:11.364 INFO:tasks.workunit.client.0.smithi043.stdout: 1 333551 7.84 MB/sec execute 102 sec latency 42.271 ms 2024-03-28T22:42:12.364 INFO:tasks.workunit.client.0.smithi043.stdout: 1 334408 7.79 MB/sec execute 103 sec latency 22.240 ms 2024-03-28T22:42:13.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:12 smithi043 ceph-mon[31823]: pgmap v155: 129 pgs: 129 active+clean; 51 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 178 op/s 2024-03-28T22:42:13.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:12 smithi172 ceph-mon[36566]: pgmap v155: 129 pgs: 129 active+clean; 51 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 178 op/s 2024-03-28T22:42:13.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:12 smithi082 ceph-mon[36471]: pgmap v155: 129 pgs: 129 active+clean; 51 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 178 op/s 2024-03-28T22:42:13.364 INFO:tasks.workunit.client.0.smithi043.stdout: 1 336634 7.83 MB/sec execute 104 sec latency 14.855 ms 2024-03-28T22:42:14.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:13 smithi172 ceph-mon[36566]: scrub summary: idle 2024-03-28T22:42:14.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:13 smithi082 ceph-mon[36471]: scrub summary: idle 2024-03-28T22:42:14.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:13 smithi043 ceph-mon[31823]: scrub summary: idle 2024-03-28T22:42:14.365 INFO:tasks.workunit.client.0.smithi043.stdout: 1 337728 7.82 MB/sec execute 105 sec latency 14.418 ms 2024-03-28T22:42:15.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:14 smithi172 ceph-mon[36566]: pgmap v156: 129 pgs: 129 active+clean; 44 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 345 op/s 2024-03-28T22:42:15.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:14 smithi082 ceph-mon[36471]: pgmap v156: 129 pgs: 129 active+clean; 44 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 345 op/s 2024-03-28T22:42:15.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:14 smithi043 ceph-mon[31823]: pgmap v156: 129 pgs: 129 active+clean; 44 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 345 op/s 2024-03-28T22:42:15.364 INFO:tasks.workunit.client.0.smithi043.stdout: 1 339413 7.81 MB/sec execute 106 sec latency 50.447 ms 2024-03-28T22:42:16.365 INFO:tasks.workunit.client.0.smithi043.stdout: 1 340957 7.84 MB/sec execute 107 sec latency 9.867 ms 2024-03-28T22:42:17.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:16 smithi172 ceph-mon[36566]: pgmap v157: 129 pgs: 129 active+clean; 44 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 246 op/s 2024-03-28T22:42:17.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:16 smithi082 ceph-mon[36471]: pgmap v157: 129 pgs: 129 active+clean; 44 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 246 op/s 2024-03-28T22:42:17.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:16 smithi043 ceph-mon[31823]: pgmap v157: 129 pgs: 129 active+clean; 44 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 246 op/s 2024-03-28T22:42:17.365 INFO:tasks.workunit.client.0.smithi043.stdout: 1 342213 7.82 MB/sec execute 108 sec latency 16.464 ms 2024-03-28T22:42:18.365 INFO:tasks.workunit.client.0.smithi043.stdout: 1 344148 7.86 MB/sec execute 109 sec latency 12.332 ms 2024-03-28T22:42:19.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:18 smithi172 ceph-mon[36566]: pgmap v158: 129 pgs: 129 active+clean; 48 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 327 op/s 2024-03-28T22:42:19.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:18 smithi082 ceph-mon[36471]: pgmap v158: 129 pgs: 129 active+clean; 48 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 327 op/s 2024-03-28T22:42:19.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:18 smithi043 ceph-mon[31823]: pgmap v158: 129 pgs: 129 active+clean; 48 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 327 op/s 2024-03-28T22:42:19.365 INFO:tasks.workunit.client.0.smithi043.stdout: 1 344951 7.81 MB/sec execute 110 sec latency 24.802 ms 2024-03-28T22:42:20.365 INFO:tasks.workunit.client.0.smithi043.stdout: 1 346565 7.80 MB/sec execute 111 sec latency 69.400 ms 2024-03-28T22:42:21.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:20 smithi172 ceph-mon[36566]: pgmap v159: 129 pgs: 129 active+clean; 48 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 248 op/s 2024-03-28T22:42:21.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:20 smithi082 ceph-mon[36471]: pgmap v159: 129 pgs: 129 active+clean; 48 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 248 op/s 2024-03-28T22:42:21.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:20 smithi043 ceph-mon[31823]: pgmap v159: 129 pgs: 129 active+clean; 48 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 248 op/s 2024-03-28T22:42:21.365 INFO:tasks.workunit.client.0.smithi043.stdout: 1 348001 7.84 MB/sec execute 112 sec latency 11.017 ms 2024-03-28T22:42:21.436 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.437+0000 7f56ab200640 1 -- 172.21.15.43:0/3802037999 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f56a4100310 msgr2=0x7f56a41006f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:21.436 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.437+0000 7f56ab200640 1 --2- 172.21.15.43:0/3802037999 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f56a4100310 0x7f56a41006f0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f5698009900 tx=0x7f569802bc40 comp rx=0 tx=0).stop 2024-03-28T22:42:21.436 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.437+0000 7f56ab200640 1 -- 172.21.15.43:0/3802037999 shutdown_connections 2024-03-28T22:42:21.436 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.437+0000 7f56ab200640 1 --2- 172.21.15.43:0/3802037999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56a410dec0 0x7f56a41102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:21.436 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.437+0000 7f56ab200640 1 --2- 172.21.15.43:0/3802037999 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56a4100cc0 0x7f56a410d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:21.436 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.437+0000 7f56ab200640 1 --2- 172.21.15.43:0/3802037999 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f56a4100310 0x7f56a41006f0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:21.436 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.437+0000 7f56ab200640 1 -- 172.21.15.43:0/3802037999 >> 172.21.15.43:0/3802037999 conn(0x7f56a40fbf60 msgr2=0x7f56a40fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:21.437 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.438+0000 7f56ab200640 1 -- 172.21.15.43:0/3802037999 shutdown_connections 2024-03-28T22:42:21.437 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.438+0000 7f56ab200640 1 -- 172.21.15.43:0/3802037999 wait complete. 2024-03-28T22:42:21.437 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.439+0000 7f56ab200640 1 Processor -- start 2024-03-28T22:42:21.437 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.439+0000 7f56ab200640 1 -- start start 2024-03-28T22:42:21.438 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.439+0000 7f56ab200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56a4100310 0x7f56a419a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:21.438 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.439+0000 7f56ab200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56a4100cc0 0x7f56a419a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:21.438 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.439+0000 7f56ab200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f56a410dec0 0x7f56a419e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:21.438 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.439+0000 7f56ab200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f56a4113ba0 con 0x7f56a4100cc0 2024-03-28T22:42:21.438 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.439+0000 7f56ab200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f56a4113a20 con 0x7f56a410dec0 2024-03-28T22:42:21.438 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.439+0000 7f56ab200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f56a4113d20 con 0x7f56a4100310 2024-03-28T22:42:21.438 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.439+0000 7f56a9e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56a4100310 0x7f56a419a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:21.439 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.440+0000 7f56a9e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56a4100310 0x7f56a419a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:56422/0 (socket says 172.21.15.43:56422) 2024-03-28T22:42:21.439 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.440+0000 7f56a9e00640 1 -- 172.21.15.43:0/4016041290 learned_addr learned my addr 172.21.15.43:0/4016041290 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:21.439 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.440+0000 7f56aa800640 1 --2- 172.21.15.43:0/4016041290 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f56a410dec0 0x7f56a419e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:21.439 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.440+0000 7f56a9400640 1 --2- 172.21.15.43:0/4016041290 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56a4100cc0 0x7f56a419a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:21.440 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.440+0000 7f56a9e00640 1 -- 172.21.15.43:0/4016041290 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f56a410dec0 msgr2=0x7f56a419e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:21.440 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.440+0000 7f56a9e00640 1 --2- 172.21.15.43:0/4016041290 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f56a410dec0 0x7f56a419e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:21.440 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.440+0000 7f56a9e00640 1 -- 172.21.15.43:0/4016041290 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56a4100cc0 msgr2=0x7f56a419a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:21.440 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.440+0000 7f56a9e00640 1 --2- 172.21.15.43:0/4016041290 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56a4100cc0 0x7f56a419a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:21.440 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.440+0000 7f56a9e00640 1 -- 172.21.15.43:0/4016041290 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56a419f030 con 0x7f56a4100310 2024-03-28T22:42:21.440 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.440+0000 7f56aa800640 1 --2- 172.21.15.43:0/4016041290 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f56a410dec0 0x7f56a419e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:21.440 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.440+0000 7f56a9e00640 1 --2- 172.21.15.43:0/4016041290 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56a4100310 0x7f56a419a040 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f5698005ab0 tx=0x7f5698004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:21.440 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.441+0000 7f569ea00640 1 -- 172.21.15.43:0/4016041290 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5698041070 con 0x7f56a4100310 2024-03-28T22:42:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.441+0000 7f569ea00640 1 -- 172.21.15.43:0/4016041290 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f569802cca0 con 0x7f56a4100310 2024-03-28T22:42:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.441+0000 7f56ab200640 1 -- 172.21.15.43:0/4016041290 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56a419f2c0 con 0x7f56a4100310 2024-03-28T22:42:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.441+0000 7f569ea00640 1 -- 172.21.15.43:0/4016041290 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5698003e50 con 0x7f56a4100310 2024-03-28T22:42:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.441+0000 7f56ab200640 1 -- 172.21.15.43:0/4016041290 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56a41a6bb0 con 0x7f56a4100310 2024-03-28T22:42:21.442 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.444+0000 7f569ea00640 1 -- 172.21.15.43:0/4016041290 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f5698044070 con 0x7f56a4100310 2024-03-28T22:42:21.443 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.444+0000 7f56af59e740 1 Processor -- start 2024-03-28T22:42:21.443 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.444+0000 7f56af59e740 1 -- start start 2024-03-28T22:42:21.445 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.445+0000 7f56af59e740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f7cd45d570 0x55f7cd45d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:21.445 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.445+0000 7f56af59e740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f7cd45de90 0x55f7cd466720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:21.445 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.445+0000 7f56af59e740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f7cd466c60 0x55f7cd469050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:21.445 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.445+0000 7f56af59e740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55f7cd3c0640 con 0x55f7cd45d570 2024-03-28T22:42:21.445 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.445+0000 7f56af59e740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55f7cd369810 con 0x55f7cd45de90 2024-03-28T22:42:21.445 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.445+0000 7f56af59e740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55f7cd3bc6e0 con 0x55f7cd466c60 2024-03-28T22:42:21.445 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.445+0000 7f56a9e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f7cd45de90 0x55f7cd466720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:21.446 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.445+0000 7f56aa800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f7cd45d570 0x55f7cd45d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:21.446 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.445+0000 7f56a9e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f7cd45de90 0x55f7cd466720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:55280/0 (socket says 172.21.15.43:55280) 2024-03-28T22:42:21.446 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.445+0000 7f56a9e00640 1 -- 172.21.15.43:0/580563300 learned_addr learned my addr 172.21.15.43:0/580563300 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:21.446 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.445+0000 7f56a9400640 1 --2- 172.21.15.43:0/580563300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f7cd466c60 0x55f7cd469050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:21.446 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.446+0000 7f56a9e00640 1 -- 172.21.15.43:0/580563300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f7cd466c60 msgr2=0x55f7cd469050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:21.446 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.446+0000 7f56a9e00640 1 --2- 172.21.15.43:0/580563300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f7cd466c60 0x55f7cd469050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:21.446 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.446+0000 7f56a9e00640 1 -- 172.21.15.43:0/580563300 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f7cd45d570 msgr2=0x55f7cd45d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:21.446 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.446+0000 7f56a9e00640 1 --2- 172.21.15.43:0/580563300 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f7cd45d570 0x55f7cd45d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:21.446 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.446+0000 7f56a9e00640 1 -- 172.21.15.43:0/580563300 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7cd0d5210 con 0x55f7cd45de90 2024-03-28T22:42:21.446 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.446+0000 7f56a9400640 1 --2- 172.21.15.43:0/580563300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f7cd466c60 0x55f7cd469050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:21.446 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.446+0000 7f569ea00640 1 --2- 172.21.15.43:0/4016041290 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f568c0a2100 0x7f568c0a45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:21.446 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.446+0000 7f56a9e00640 1 --2- 172.21.15.43:0/580563300 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f7cd45de90 0x55f7cd466720 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f56980ed200 tx=0x7f56980f3e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d883910369402417 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:21.446 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.446+0000 7f569ea00640 1 -- 172.21.15.43:0/4016041290 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f56980efe60 con 0x7f56a4100310 2024-03-28T22:42:21.447 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.446+0000 7f56aa800640 1 --2- 172.21.15.43:0/4016041290 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f568c0a2100 0x7f568c0a45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:21.447 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.447+0000 7f5693000640 1 -- 172.21.15.43:0/580563300 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56980b88b0 con 0x55f7cd45de90 2024-03-28T22:42:21.447 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.447+0000 7f5693000640 1 -- 172.21.15.43:0/580563300 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56980b8030 con 0x55f7cd45de90 2024-03-28T22:42:21.447 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.447+0000 7f5693000640 1 -- 172.21.15.43:0/580563300 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56980bbf70 con 0x55f7cd45de90 2024-03-28T22:42:21.450 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.448+0000 7f56af59e740 1 -- 172.21.15.43:0/580563300 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f7cd45de90 msgr2=0x55f7cd466720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:21.450 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.448+0000 7f56af59e740 1 --2- 172.21.15.43:0/580563300 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f7cd45de90 0x55f7cd466720 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f56980ed200 tx=0x7f56980f3e70 comp rx=0 tx=0).stop 2024-03-28T22:42:21.450 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.448+0000 7f56aa800640 1 --2- 172.21.15.43:0/4016041290 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f568c0a2100 0x7f568c0a45c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f56a0009fd0 tx=0x7f56a0009500 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:21.450 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.448+0000 7f56af59e740 1 -- 172.21.15.43:0/580563300 shutdown_connections 2024-03-28T22:42:21.450 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.448+0000 7f56af59e740 1 --2- 172.21.15.43:0/580563300 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f7cd45de90 0x55f7cd466720 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.448+0000 7f56af59e740 1 --2- 172.21.15.43:0/580563300 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f7cd466c60 0x55f7cd469050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.448+0000 7f56af59e740 1 --2- 172.21.15.43:0/580563300 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f7cd45d570 0x55f7cd45d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.448+0000 7f56af59e740 1 -- 172.21.15.43:0/580563300 >> 172.21.15.43:0/580563300 conn(0x55f7cd457e90 msgr2=0x55f7cd458270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.448+0000 7f56af59e740 1 -- 172.21.15.43:0/580563300 shutdown_connections 2024-03-28T22:42:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.448+0000 7f56af59e740 1 -- 172.21.15.43:0/580563300 wait complete. 2024-03-28T22:42:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.448+0000 7f56af59e740 1 -- start start 2024-03-28T22:42:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.449+0000 7f56af59e740 1 Processor -- start 2024-03-28T22:42:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.449+0000 7f56af59e740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f7cd45d570 0x55f7cd463cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.449+0000 7f56af59e740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f7cd45de90 0x55f7cd464210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.449+0000 7f56af59e740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f7cd466c60 0x55f7cd464750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.449+0000 7f56af59e740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55f7cd469650 con 0x55f7cd466c60 2024-03-28T22:42:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.449+0000 7f56af59e740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55f7cd35ed00 con 0x55f7cd45d570 2024-03-28T22:42:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.449+0000 7f56af59e740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55f7cd4697d0 con 0x55f7cd45de90 2024-03-28T22:42:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.449+0000 7f56af59e740 10 client.0 _ll_register_callbacks cb 0x55f7cd349490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.449+0000 7f56a9400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f7cd45de90 0x55f7cd464210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.449+0000 7f56a9e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f7cd45d570 0x55f7cd463cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.449+0000 7f56a9e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f7cd45d570 0x55f7cd463cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:55292/0 (socket says 172.21.15.43:55292) 2024-03-28T22:42:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.449+0000 7f56a9e00640 1 -- 172.21.15.43:0/3854858803 learned_addr learned my addr 172.21.15.43:0/3854858803 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.449+0000 7f56a9400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f7cd45de90 0x55f7cd464210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:56438/0 (socket says 172.21.15.43:56438) 2024-03-28T22:42:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.450+0000 7f56aa800640 1 --2- 172.21.15.43:0/3854858803 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f7cd466c60 0x55f7cd464750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.450+0000 7f56a9e00640 1 -- 172.21.15.43:0/3854858803 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f7cd45de90 msgr2=0x55f7cd464210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.450+0000 7f56a9e00640 1 --2- 172.21.15.43:0/3854858803 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f7cd45de90 0x55f7cd464210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.450+0000 7f56a9e00640 1 -- 172.21.15.43:0/3854858803 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f7cd466c60 msgr2=0x55f7cd464750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.450+0000 7f56a9e00640 1 --2- 172.21.15.43:0/3854858803 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f7cd466c60 0x55f7cd464750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.450+0000 7f56a9e00640 1 -- 172.21.15.43:0/3854858803 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7cd1a1620 con 0x55f7cd45d570 2024-03-28T22:42:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.450+0000 7f56aa800640 1 --2- 172.21.15.43:0/3854858803 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f7cd466c60 0x55f7cd464750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:42:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.450+0000 7f56a9e00640 1 --2- 172.21.15.43:0/3854858803 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f7cd45d570 0x55f7cd463cd0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f56980b8fe0 tx=0x7f56980a70c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5288237585b75ae server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.450+0000 7f5687400640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:42:21.453 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.450+0000 7f5687400640 1 -- 172.21.15.43:0/3854858803 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f569804b040 con 0x55f7cd45d570 2024-03-28T22:42:21.453 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.450+0000 7f5687400640 1 -- 172.21.15.43:0/3854858803 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f56980ed430 con 0x55f7cd45d570 2024-03-28T22:42:21.453 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.450+0000 7f56af59e740 1 -- 172.21.15.43:0/3854858803 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56980b8030 con 0x55f7cd45d570 2024-03-28T22:42:21.453 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.451+0000 7f5687400640 1 -- 172.21.15.43:0/3854858803 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f56980bc150 con 0x55f7cd45d570 2024-03-28T22:42:21.453 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.451+0000 7f56af59e740 10 client.27602 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:21.453 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.451+0000 7f56af59e740 1 -- 172.21.15.43:0/3854858803 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f7cd469650 con 0x55f7cd45d570 2024-03-28T22:42:21.453 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.451+0000 7f5687400640 1 -- 172.21.15.43:0/3854858803 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56980b8030 con 0x55f7cd45d570 2024-03-28T22:42:21.453 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.451+0000 7f5687400640 1 -- 172.21.15.43:0/3854858803 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f56980bc870 con 0x55f7cd45d570 2024-03-28T22:42:21.453 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.451+0000 7f56af59e740 10 client.27602 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:21.453 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.451+0000 7f56af59e740 10 client.27602 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:42:21.453 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.451+0000 7f56af59e740 20 client.27602 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:21.453 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.451+0000 7f56af59e740 1 --2- 172.21.15.43:0/3854858803 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f7cd4766d0 0x55f7cd478ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:21.453 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.451+0000 7f56af59e740 4 client.27602 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:42:21.453 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.451+0000 7f56af59e740 1 -- 172.21.15.43:0/3854858803 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f7cd3d4030 con 0x55f7cd4766d0 2024-03-28T22:42:21.453 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.451+0000 7f56a9400640 1 --2- 172.21.15.43:0/3854858803 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f7cd4766d0 0x55f7cd478ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:21.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.452+0000 7f56a9400640 1 --2- 172.21.15.43:0/3854858803 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f7cd4766d0 0x55f7cd478ac0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18e10b9a59fdc555 server_cookie=3c539cf0397b3033 in_seq=0 out_seq=0 2024-03-28T22:42:21.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.452+0000 7f5687400640 10 client.27602 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:21.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.454+0000 7f5687400640 1 -- 172.21.15.43:0/3854858803 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f7cd3d4030 con 0x55f7cd4766d0 2024-03-28T22:42:21.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.454+0000 7f5687400640 10 client.27602 handle_command_reply: tid=0 2024-03-28T22:42:21.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.454+0000 7f56af59e740 1 client.27602 shutdown 2024-03-28T22:42:21.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.454+0000 7f56af59e740 1 -- 172.21.15.43:0/3854858803 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f7cd45d570 msgr2=0x55f7cd463cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:21.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.454+0000 7f56af59e740 1 --2- 172.21.15.43:0/3854858803 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f7cd45d570 0x55f7cd463cd0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f56980b8fe0 tx=0x7f56980a70c0 comp rx=0 tx=0).stop 2024-03-28T22:42:21.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.454+0000 7f56af59e740 1 -- 172.21.15.43:0/3854858803 shutdown_connections 2024-03-28T22:42:21.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.454+0000 7f56af59e740 1 --2- 172.21.15.43:0/3854858803 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f7cd466c60 0x55f7cd464750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:21.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.454+0000 7f56af59e740 1 --2- 172.21.15.43:0/3854858803 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f7cd45de90 0x55f7cd464210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:21.455 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.454+0000 7f56af59e740 1 --2- 172.21.15.43:0/3854858803 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f7cd4766d0 0x55f7cd478ac0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:21.456 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.454+0000 7f56af59e740 1 --2- 172.21.15.43:0/3854858803 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f7cd45d570 0x55f7cd463cd0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:21.456 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.454+0000 7f5687400640 0 client.27602 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:21.456 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.454+0000 7f56af59e740 1 -- 172.21.15.43:0/3854858803 >> 172.21.15.43:0/3854858803 conn(0x55f7cd457e90 msgr2=0x55f7cd459be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:21.456 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.455+0000 7f56af59e740 1 -- 172.21.15.43:0/3854858803 shutdown_connections 2024-03-28T22:42:21.456 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.455+0000 7f56af59e740 1 -- 172.21.15.43:0/3854858803 wait complete. 2024-03-28T22:42:21.456 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.455+0000 7f56af59e740 20 client.27602 trim_cache size 0 max 16384 2024-03-28T22:42:21.470 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.472+0000 7f56af59e740 1 Processor -- start 2024-03-28T22:42:21.470 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.472+0000 7f56af59e740 1 -- start start 2024-03-28T22:42:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.472+0000 7f56af59e740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f7cd45d570 0x55f7cd472360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.472+0000 7f56af59e740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f7cd45de90 0x55f7cd47e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.472+0000 7f56af59e740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f7cd466c60 0x55f7cd46e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.472+0000 7f56af59e740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f56a41b0c40 con 0x55f7cd45d570 2024-03-28T22:42:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.472+0000 7f56af59e740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f56a41b2780 con 0x55f7cd45de90 2024-03-28T22:42:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.472+0000 7f56af59e740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f56a41b1310 con 0x55f7cd466c60 2024-03-28T22:42:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.473+0000 7f56a9400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f7cd45de90 0x55f7cd47e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:21.473 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.473+0000 7f56aa800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f7cd466c60 0x55f7cd46e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:21.473 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.473+0000 7f56a9400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f7cd45de90 0x55f7cd47e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:55300/0 (socket says 172.21.15.43:55300) 2024-03-28T22:42:21.473 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.473+0000 7f56a9400640 1 -- 172.21.15.43:0/3415270280 learned_addr learned my addr 172.21.15.43:0/3415270280 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:21.473 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.473+0000 7f56aa800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f7cd466c60 0x55f7cd46e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:56448/0 (socket says 172.21.15.43:56448) 2024-03-28T22:42:21.473 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.473+0000 7f56a9e00640 1 --2- 172.21.15.43:0/3415270280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f7cd45d570 0x55f7cd472360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:21.473 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.473+0000 7f56a9400640 1 -- 172.21.15.43:0/3415270280 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f7cd466c60 msgr2=0x55f7cd46e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:21.473 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.473+0000 7f56a9400640 1 --2- 172.21.15.43:0/3415270280 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f7cd466c60 0x55f7cd46e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:21.473 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.473+0000 7f56a9400640 1 -- 172.21.15.43:0/3415270280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f7cd45d570 msgr2=0x55f7cd472360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:21.473 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.473+0000 7f56a9400640 1 --2- 172.21.15.43:0/3415270280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f7cd45d570 0x55f7cd472360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:21.473 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.473+0000 7f56a9400640 1 -- 172.21.15.43:0/3415270280 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7cd45c960 con 0x55f7cd45de90 2024-03-28T22:42:21.473 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.473+0000 7f56a9400640 1 --2- 172.21.15.43:0/3415270280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f7cd45de90 0x55f7cd47e3e0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f5694005d00 tx=0x7f569400b410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd3caf1f812b3107 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:21.473 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.474+0000 7f5692600640 1 -- 172.21.15.43:0/3415270280 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5694020430 con 0x55f7cd45de90 2024-03-28T22:42:21.473 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.474+0000 7f5692600640 1 -- 172.21.15.43:0/3415270280 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56940205d0 con 0x55f7cd45de90 2024-03-28T22:42:21.474 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.474+0000 7f5692600640 1 -- 172.21.15.43:0/3415270280 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5694010690 con 0x55f7cd45de90 2024-03-28T22:42:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.474+0000 7f56af59e740 1 -- 172.21.15.43:0/3415270280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f7cd45de90 msgr2=0x55f7cd47e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.474+0000 7f56af59e740 1 --2- 172.21.15.43:0/3415270280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f7cd45de90 0x55f7cd47e3e0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f5694005d00 tx=0x7f569400b410 comp rx=0 tx=0).stop 2024-03-28T22:42:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.475+0000 7f56af59e740 1 -- 172.21.15.43:0/3415270280 shutdown_connections 2024-03-28T22:42:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.475+0000 7f56af59e740 1 --2- 172.21.15.43:0/3415270280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f7cd45de90 0x55f7cd47e3e0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.475+0000 7f56af59e740 1 --2- 172.21.15.43:0/3415270280 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f7cd466c60 0x55f7cd46e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.475+0000 7f56af59e740 1 --2- 172.21.15.43:0/3415270280 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f7cd45d570 0x55f7cd472360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.475+0000 7f56af59e740 1 -- 172.21.15.43:0/3415270280 >> 172.21.15.43:0/3415270280 conn(0x55f7cd457e90 msgr2=0x55f7cd463c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.475+0000 7f56af59e740 1 -- 172.21.15.43:0/3415270280 shutdown_connections 2024-03-28T22:42:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.475+0000 7f56af59e740 1 -- 172.21.15.43:0/3415270280 wait complete. 2024-03-28T22:42:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.475+0000 7f56af59e740 1 -- start start 2024-03-28T22:42:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.475+0000 7f56af59e740 1 Processor -- start 2024-03-28T22:42:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.476+0000 7f56af59e740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f7cd45d570 0x55f7cd46ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.476+0000 7f56af59e740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f7cd45de90 0x55f7cd46af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.476+0000 7f56af59e740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f7cd466c60 0x55f7cd46b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.476+0000 7f56af59e740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f56a41b20b0 con 0x55f7cd466c60 2024-03-28T22:42:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.476+0000 7f56af59e740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f56a41b19e0 con 0x55f7cd45de90 2024-03-28T22:42:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.476+0000 7f56af59e740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f56a41aec20 con 0x55f7cd45d570 2024-03-28T22:42:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.476+0000 7f56af59e740 10 client.0 _ll_register_callbacks cb 0x7f56a0009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.476+0000 7f56a9e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f7cd45d570 0x55f7cd46ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.476+0000 7f56a9400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f7cd45de90 0x55f7cd46af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.476+0000 7f56a9e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f7cd45d570 0x55f7cd46ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:56458/0 (socket says 172.21.15.43:56458) 2024-03-28T22:42:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.476+0000 7f56a9e00640 1 -- 172.21.15.43:0/553360948 learned_addr learned my addr 172.21.15.43:0/553360948 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.476+0000 7f56aa800640 1 --2- 172.21.15.43:0/553360948 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f7cd466c60 0x55f7cd46b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.477+0000 7f56a9e00640 1 -- 172.21.15.43:0/553360948 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f7cd45de90 msgr2=0x55f7cd46af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.477+0000 7f56a9e00640 1 --2- 172.21.15.43:0/553360948 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f7cd45de90 0x55f7cd46af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.477+0000 7f56a9e00640 1 -- 172.21.15.43:0/553360948 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f7cd466c60 msgr2=0x55f7cd46b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:21.479 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.477+0000 7f56a9e00640 1 --2- 172.21.15.43:0/553360948 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f7cd466c60 0x55f7cd46b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:21.479 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.477+0000 7f56a9e00640 1 -- 172.21.15.43:0/553360948 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7cd46bd00 con 0x55f7cd45d570 2024-03-28T22:42:21.479 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.477+0000 7f56a9400640 1 --2- 172.21.15.43:0/553360948 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f7cd45de90 0x55f7cd46af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:21.479 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.477+0000 7f56a9e00640 1 --2- 172.21.15.43:0/553360948 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f7cd45d570 0x55f7cd46ec10 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f56980b8a20 tx=0x7f5698009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cbce26f367a4aed server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:21.479 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.477+0000 7f5687400640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:42:21.479 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.477+0000 7f5687400640 1 -- 172.21.15.43:0/553360948 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f569804b070 con 0x55f7cd45d570 2024-03-28T22:42:21.479 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.477+0000 7f5687400640 1 -- 172.21.15.43:0/553360948 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f56980bc870 con 0x55f7cd45d570 2024-03-28T22:42:21.479 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.477+0000 7f56af59e740 1 -- 172.21.15.43:0/553360948 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56940205d0 con 0x55f7cd45d570 2024-03-28T22:42:21.479 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.478+0000 7f5687400640 1 -- 172.21.15.43:0/553360948 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f56980bc3e0 con 0x55f7cd45d570 2024-03-28T22:42:21.479 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.478+0000 7f56af59e740 10 client.27553 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:21.479 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.478+0000 7f56af59e740 1 -- 172.21.15.43:0/553360948 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f7cd46bd00 con 0x55f7cd45d570 2024-03-28T22:42:21.479 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.478+0000 7f5687400640 1 -- 172.21.15.43:0/553360948 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56940205d0 con 0x55f7cd45d570 2024-03-28T22:42:21.479 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.478+0000 7f5687400640 1 -- 172.21.15.43:0/553360948 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f56980c4870 con 0x55f7cd45d570 2024-03-28T22:42:21.479 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.478+0000 7f56af59e740 10 client.27553 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:21.479 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.478+0000 7f56af59e740 10 client.27553 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:42:21.480 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.478+0000 7f56af59e740 20 client.27553 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:21.480 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.478+0000 7f56af59e740 1 --2- 172.21.15.43:0/553360948 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f7cd4ddd80 0x55f7cd4e0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:21.480 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.478+0000 7f56af59e740 4 client.27553 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T22:42:21.480 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.478+0000 7f56af59e740 1 -- 172.21.15.43:0/553360948 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f7cd3cdb80 con 0x55f7cd4ddd80 2024-03-28T22:42:21.480 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.478+0000 7f56a9400640 1 --2- 172.21.15.43:0/553360948 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f7cd4ddd80 0x55f7cd4e0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:21.480 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.479+0000 7f56a9400640 1 --2- 172.21.15.43:0/553360948 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f7cd4ddd80 0x55f7cd4e0170 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdda08cb1389cf52 server_cookie=7e8a11683ad34bc0 in_seq=0 out_seq=0 2024-03-28T22:42:21.480 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.479+0000 7f5687400640 10 client.27553 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.695+0000 7f5687400640 1 -- 172.21.15.43:0/553360948 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f7cd3cdb80 con 0x55f7cd4ddd80 2024-03-28T22:42:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.695+0000 7f5687400640 10 client.27553 handle_command_reply: tid=0 2024-03-28T22:42:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.695+0000 7f56af59e740 1 client.27553 shutdown 2024-03-28T22:42:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.695+0000 7f56af59e740 1 -- 172.21.15.43:0/553360948 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f7cd45d570 msgr2=0x55f7cd46ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.695+0000 7f56af59e740 1 --2- 172.21.15.43:0/553360948 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f7cd45d570 0x55f7cd46ec10 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f56980b8a20 tx=0x7f5698009260 comp rx=0 tx=0).stop 2024-03-28T22:42:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.696+0000 7f56af59e740 1 -- 172.21.15.43:0/553360948 shutdown_connections 2024-03-28T22:42:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.696+0000 7f56af59e740 1 --2- 172.21.15.43:0/553360948 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f7cd4ddd80 0x55f7cd4e0170 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.696+0000 7f56af59e740 1 --2- 172.21.15.43:0/553360948 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f7cd45de90 0x55f7cd46af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.696+0000 7f56af59e740 1 --2- 172.21.15.43:0/553360948 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f7cd466c60 0x55f7cd46b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.696+0000 7f56af59e740 1 --2- 172.21.15.43:0/553360948 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f7cd45d570 0x55f7cd46ec10 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.696+0000 7f5687400640 0 client.27553 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.696+0000 7f56af59e740 1 -- 172.21.15.43:0/553360948 >> 172.21.15.43:0/553360948 conn(0x55f7cd457e90 msgr2=0x55f7cd456f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.696+0000 7f56af59e740 1 -- 172.21.15.43:0/553360948 shutdown_connections 2024-03-28T22:42:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.696+0000 7f56af59e740 1 -- 172.21.15.43:0/553360948 wait complete. 2024-03-28T22:42:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.696+0000 7f56af59e740 20 client.27553 trim_cache size 0 max 16384 2024-03-28T22:42:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:{ 2024-03-28T22:42:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "message": "", 2024-03-28T22:42:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "return_code": 0 2024-03-28T22:42:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:} 2024-03-28T22:42:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.697+0000 7f56ab200640 1 -- 172.21.15.43:0/4016041290 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f568c0a2100 msgr2=0x7f568c0a45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.697+0000 7f56ab200640 1 --2- 172.21.15.43:0/4016041290 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f568c0a2100 0x7f568c0a45c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f56a0009fd0 tx=0x7f56a0009500 comp rx=0 tx=0).stop 2024-03-28T22:42:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.697+0000 7f56ab200640 1 -- 172.21.15.43:0/4016041290 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56a4100310 msgr2=0x7f56a419a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.697+0000 7f56ab200640 1 --2- 172.21.15.43:0/4016041290 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56a4100310 0x7f56a419a040 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f5698005ab0 tx=0x7f5698004360 comp rx=0 tx=0).stop 2024-03-28T22:42:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.697+0000 7f56ab200640 1 -- 172.21.15.43:0/4016041290 shutdown_connections 2024-03-28T22:42:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.697+0000 7f56ab200640 1 --2- 172.21.15.43:0/4016041290 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f568c0a2100 0x7f568c0a45c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.697+0000 7f56ab200640 1 --2- 172.21.15.43:0/4016041290 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f56a410dec0 0x7f56a419e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.697+0000 7f56ab200640 1 --2- 172.21.15.43:0/4016041290 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f56a4100cc0 0x7f56a419a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.697+0000 7f56ab200640 1 --2- 172.21.15.43:0/4016041290 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f56a4100310 0x7f56a419a040 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.697+0000 7f56ab200640 1 -- 172.21.15.43:0/4016041290 >> 172.21.15.43:0/4016041290 conn(0x7f56a40fbf60 msgr2=0x7f56a40fe350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.697+0000 7f56ab200640 1 -- 172.21.15.43:0/4016041290 shutdown_connections 2024-03-28T22:42:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:21.697+0000 7f56ab200640 1 -- 172.21.15.43:0/4016041290 wait complete. 2024-03-28T22:42:22.365 INFO:tasks.workunit.client.0.smithi043.stdout: 1 348794 7.78 MB/sec execute 113 sec latency 169.265 ms 2024-03-28T22:42:23.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:22 smithi172 ceph-mon[36566]: pgmap v160: 129 pgs: 129 active+clean; 48 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 248 op/s 2024-03-28T22:42:23.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:22 smithi082 ceph-mon[36471]: pgmap v160: 129 pgs: 129 active+clean; 48 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 248 op/s 2024-03-28T22:42:23.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:22 smithi043 ceph-mon[31823]: pgmap v160: 129 pgs: 129 active+clean; 48 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 248 op/s 2024-03-28T22:42:23.365 INFO:tasks.workunit.client.0.smithi043.stdout: 1 350948 7.82 MB/sec execute 114 sec latency 12.169 ms 2024-03-28T22:42:24.365 INFO:tasks.workunit.client.0.smithi043.stdout: 1 351892 7.80 MB/sec execute 115 sec latency 31.539 ms 2024-03-28T22:42:25.068 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:24 smithi043 ceph-mon[31823]: pgmap v161: 129 pgs: 129 active+clean; 44 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 424 op/s 2024-03-28T22:42:25.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:24 smithi172 ceph-mon[36566]: pgmap v161: 129 pgs: 129 active+clean; 44 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 424 op/s 2024-03-28T22:42:25.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:24 smithi082 ceph-mon[36471]: pgmap v161: 129 pgs: 129 active+clean; 44 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 424 op/s 2024-03-28T22:42:25.366 INFO:tasks.workunit.client.0.smithi043.stdout: 1 353588 7.79 MB/sec execute 116 sec latency 40.604 ms 2024-03-28T22:42:26.369 INFO:tasks.workunit.client.0.smithi043.stdout: 1 355092 7.83 MB/sec execute 117 sec latency 19.054 ms 2024-03-28T22:42:27.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:26 smithi172 ceph-mon[36566]: pgmap v162: 129 pgs: 129 active+clean; 44 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 257 op/s 2024-03-28T22:42:27.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:26 smithi082 ceph-mon[36471]: pgmap v162: 129 pgs: 129 active+clean; 44 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 257 op/s 2024-03-28T22:42:27.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:26 smithi043 ceph-mon[31823]: pgmap v162: 129 pgs: 129 active+clean; 44 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 257 op/s 2024-03-28T22:42:27.366 INFO:tasks.workunit.client.0.smithi043.stdout: 1 356386 7.81 MB/sec execute 118 sec latency 12.305 ms 2024-03-28T22:42:28.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:27 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:42:28.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:27 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:42:28.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:27 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:42:28.366 INFO:tasks.workunit.client.0.smithi043.stdout: 1 358324 7.84 MB/sec execute 119 sec latency 13.851 ms 2024-03-28T22:42:29.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:28 smithi172 ceph-mon[36566]: pgmap v163: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 336 op/s 2024-03-28T22:42:29.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:28 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:42:29.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:28 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:42:29.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:28 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:42:29.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:28 smithi082 ceph-mon[36471]: pgmap v163: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 336 op/s 2024-03-28T22:42:29.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:28 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:42:29.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:28 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:42:29.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:28 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:42:29.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:28 smithi043 ceph-mon[31823]: pgmap v163: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 336 op/s 2024-03-28T22:42:29.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:28 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:42:29.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:28 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:42:29.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:28 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:42:29.366 INFO:tasks.workunit.client.0.smithi043.stdout: 1 359062 7.80 MB/sec execute 120 sec latency 23.264 ms 2024-03-28T22:42:30.366 INFO:tasks.workunit.client.0.smithi043.stdout: 1 360687 7.79 MB/sec execute 121 sec latency 71.736 ms 2024-03-28T22:42:31.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:30 smithi172 ceph-mon[36566]: pgmap v164: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 254 op/s 2024-03-28T22:42:31.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:30 smithi082 ceph-mon[36471]: pgmap v164: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 254 op/s 2024-03-28T22:42:31.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:30 smithi043 ceph-mon[31823]: pgmap v164: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 254 op/s 2024-03-28T22:42:31.366 INFO:tasks.workunit.client.0.smithi043.stdout: 1 362081 7.82 MB/sec execute 122 sec latency 14.871 ms 2024-03-28T22:42:32.367 INFO:tasks.workunit.client.0.smithi043.stdout: 1 363012 7.77 MB/sec execute 123 sec latency 19.985 ms 2024-03-28T22:42:33.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:32 smithi172 ceph-mon[36566]: pgmap v165: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 254 op/s 2024-03-28T22:42:33.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:32 smithi082 ceph-mon[36471]: pgmap v165: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 254 op/s 2024-03-28T22:42:33.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:32 smithi043 ceph-mon[31823]: pgmap v165: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 254 op/s 2024-03-28T22:42:33.367 INFO:tasks.workunit.client.0.smithi043.stdout: 1 364986 7.80 MB/sec execute 124 sec latency 13.908 ms 2024-03-28T22:42:34.367 INFO:tasks.workunit.client.0.smithi043.stdout: 1 366021 7.79 MB/sec execute 125 sec latency 17.085 ms 2024-03-28T22:42:34.851 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.852+0000 7f9499e00640 1 -- 172.21.15.43:0/3265940651 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9494101f10 msgr2=0x7f949410e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:34.851 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.852+0000 7f9499e00640 1 --2- 172.21.15.43:0/3265940651 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9494101f10 0x7f949410e3d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f948c009be0 tx=0x7f948c02bef0 comp rx=0 tx=0).stop 2024-03-28T22:42:34.851 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.853+0000 7f9499e00640 1 -- 172.21.15.43:0/3265940651 shutdown_connections 2024-03-28T22:42:34.851 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.853+0000 7f9499e00640 1 --2- 172.21.15.43:0/3265940651 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9494101f10 0x7f949410e3d0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:34.851 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.853+0000 7f9499e00640 1 --2- 172.21.15.43:0/3265940651 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9494101550 0x7f94941019d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:34.851 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.853+0000 7f9499e00640 1 --2- 172.21.15.43:0/3265940651 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9494107610 0x7f94941079f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:34.851 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.853+0000 7f9499e00640 1 -- 172.21.15.43:0/3265940651 >> 172.21.15.43:0/3265940651 conn(0x7f94940fd060 msgr2=0x7f94940ff480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:34.851 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.853+0000 7f9499e00640 1 -- 172.21.15.43:0/3265940651 shutdown_connections 2024-03-28T22:42:34.852 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.853+0000 7f9499e00640 1 -- 172.21.15.43:0/3265940651 wait complete. 2024-03-28T22:42:34.854 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.854+0000 7f9499e00640 1 Processor -- start 2024-03-28T22:42:34.854 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.854+0000 7f9499e00640 1 -- start start 2024-03-28T22:42:34.854 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.854+0000 7f9499e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9494101550 0x7f9494199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.854+0000 7f9499e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9494101f10 0x7f949419a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.854+0000 7f9499e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9494107610 0x7f949419e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.854+0000 7f9499e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9494111940 con 0x7f9494107610 2024-03-28T22:42:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.854+0000 7f9499e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f94941117c0 con 0x7f9494101550 2024-03-28T22:42:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.854+0000 7f9499e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9494111ac0 con 0x7f9494101f10 2024-03-28T22:42:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.855+0000 7f948be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9494101f10 0x7f949419a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.855+0000 7f9498a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9494101550 0x7f9494199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.855+0000 7f948be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9494101f10 0x7f949419a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:52090/0 (socket says 172.21.15.43:52090) 2024-03-28T22:42:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.855+0000 7f948be00640 1 -- 172.21.15.43:0/1244411623 learned_addr learned my addr 172.21.15.43:0/1244411623 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.855+0000 7f9499400640 1 --2- 172.21.15.43:0/1244411623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9494107610 0x7f949419e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.855+0000 7f948be00640 1 -- 172.21.15.43:0/1244411623 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9494101550 msgr2=0x7f9494199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.855+0000 7f948be00640 1 --2- 172.21.15.43:0/1244411623 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9494101550 0x7f9494199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.855+0000 7f948be00640 1 -- 172.21.15.43:0/1244411623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9494107610 msgr2=0x7f949419e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.855+0000 7f948be00640 1 --2- 172.21.15.43:0/1244411623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9494107610 0x7f949419e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:34.856 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.855+0000 7f948be00640 1 -- 172.21.15.43:0/1244411623 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f949419ef50 con 0x7f9494101f10 2024-03-28T22:42:34.856 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.855+0000 7f9498a00640 1 --2- 172.21.15.43:0/1244411623 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9494101550 0x7f9494199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:34.856 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.855+0000 7f9499400640 1 --2- 172.21.15.43:0/1244411623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9494107610 0x7f949419e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:42:34.856 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.855+0000 7f948be00640 1 --2- 172.21.15.43:0/1244411623 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9494101f10 0x7f949419a4a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f947c00e9f0 tx=0x7f947c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:34.856 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.856+0000 7f9492000640 1 -- 172.21.15.43:0/1244411623 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947c00cd80 con 0x7f9494101f10 2024-03-28T22:42:34.856 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.856+0000 7f9492000640 1 -- 172.21.15.43:0/1244411623 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f947c005500 con 0x7f9494101f10 2024-03-28T22:42:34.856 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.856+0000 7f9492000640 1 -- 172.21.15.43:0/1244411623 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947c010690 con 0x7f9494101f10 2024-03-28T22:42:34.856 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.856+0000 7f9499e00640 1 -- 172.21.15.43:0/1244411623 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f949419f240 con 0x7f9494101f10 2024-03-28T22:42:34.856 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.856+0000 7f9499e00640 1 -- 172.21.15.43:0/1244411623 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94941a6ad0 con 0x7f9494101f10 2024-03-28T22:42:34.857 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.859+0000 7f9492000640 1 -- 172.21.15.43:0/1244411623 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f947c0046c0 con 0x7f9494101f10 2024-03-28T22:42:34.857 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.860+0000 7f949e1f7740 1 Processor -- start 2024-03-28T22:42:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.860+0000 7f949e1f7740 1 -- start start 2024-03-28T22:42:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.860+0000 7f949e1f7740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c029d68570 0x55c029d68950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.860+0000 7f949e1f7740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c029d68e90 0x55c029d71720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.860+0000 7f949e1f7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c029d71c60 0x55c029d74050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.860+0000 7f949e1f7740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c029ccb640 con 0x55c029d68e90 2024-03-28T22:42:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.860+0000 7f949e1f7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55c029c74810 con 0x55c029d71c60 2024-03-28T22:42:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.860+0000 7f949e1f7740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55c029cc76e0 con 0x55c029d68570 2024-03-28T22:42:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.861+0000 7f9499400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c029d68570 0x55c029d68950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.861+0000 7f948be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c029d71c60 0x55c029d74050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.861+0000 7f9499400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c029d68570 0x55c029d68950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:52104/0 (socket says 172.21.15.43:52104) 2024-03-28T22:42:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.861+0000 7f9499400640 1 -- 172.21.15.43:0/1567489796 learned_addr learned my addr 172.21.15.43:0/1567489796 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.861+0000 7f9498a00640 1 --2- 172.21.15.43:0/1567489796 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c029d68e90 0x55c029d71720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.861+0000 7f9492000640 1 --2- 172.21.15.43:0/1244411623 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f94780a1f60 0x7f94780a4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.861+0000 7f9492000640 1 -- 172.21.15.43:0/1244411623 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f947c0ce9d0 con 0x7f9494101f10 2024-03-28T22:42:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.861+0000 7f948be00640 1 -- 172.21.15.43:0/1567489796 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c029d68570 msgr2=0x55c029d68950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.861+0000 7f948be00640 1 --2- 172.21.15.43:0/1567489796 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c029d68570 0x55c029d68950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.861+0000 7f948be00640 1 -- 172.21.15.43:0/1567489796 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c029d68e90 msgr2=0x55c029d71720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.861+0000 7f948be00640 1 --2- 172.21.15.43:0/1567489796 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c029d68e90 0x55c029d71720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.861+0000 7f948be00640 1 -- 172.21.15.43:0/1567489796 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0299e0210 con 0x55c029d71c60 2024-03-28T22:42:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.861+0000 7f9498a00640 1 --2- 172.21.15.43:0/1567489796 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c029d68e90 0x55c029d71720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:42:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.861+0000 7f9499400640 1 --2- 172.21.15.43:0/1244411623 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f94780a1f60 0x7f94780a4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:34.861 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.862+0000 7f948be00640 1 --2- 172.21.15.43:0/1567489796 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c029d71c60 0x55c029d74050 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f947c0cbd20 tx=0x7f947c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bf29d0c1a7fddf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:34.861 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.862+0000 7f9489c00640 1 -- 172.21.15.43:0/1567489796 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947c09ab00 con 0x55c029d71c60 2024-03-28T22:42:34.861 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.862+0000 7f9489c00640 1 -- 172.21.15.43:0/1567489796 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f947c09b6c0 con 0x55c029d71c60 2024-03-28T22:42:34.861 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.862+0000 7f9489c00640 1 -- 172.21.15.43:0/1567489796 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947c09a920 con 0x55c029d71c60 2024-03-28T22:42:34.863 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.863+0000 7f949e1f7740 1 -- 172.21.15.43:0/1567489796 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c029d71c60 msgr2=0x55c029d74050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:34.863 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.863+0000 7f949e1f7740 1 --2- 172.21.15.43:0/1567489796 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c029d71c60 0x55c029d74050 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f947c0cbd20 tx=0x7f947c004a40 comp rx=0 tx=0).stop 2024-03-28T22:42:34.864 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.863+0000 7f9499400640 1 --2- 172.21.15.43:0/1244411623 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f94780a1f60 0x7f94780a4420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f948c005f00 tx=0x7f948c009500 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:34.864 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.863+0000 7f949e1f7740 1 -- 172.21.15.43:0/1567489796 shutdown_connections 2024-03-28T22:42:34.864 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.863+0000 7f949e1f7740 1 --2- 172.21.15.43:0/1567489796 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c029d71c60 0x55c029d74050 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:34.864 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.863+0000 7f949e1f7740 1 --2- 172.21.15.43:0/1567489796 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c029d68e90 0x55c029d71720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:34.864 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.863+0000 7f949e1f7740 1 --2- 172.21.15.43:0/1567489796 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c029d68570 0x55c029d68950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:34.864 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.863+0000 7f949e1f7740 1 -- 172.21.15.43:0/1567489796 >> 172.21.15.43:0/1567489796 conn(0x55c029d62e90 msgr2=0x55c029d63270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:34.864 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.863+0000 7f949e1f7740 1 -- 172.21.15.43:0/1567489796 shutdown_connections 2024-03-28T22:42:34.864 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.863+0000 7f949e1f7740 1 -- 172.21.15.43:0/1567489796 wait complete. 2024-03-28T22:42:34.864 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.863+0000 7f949e1f7740 1 -- start start 2024-03-28T22:42:34.864 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.864+0000 7f949e1f7740 1 Processor -- start 2024-03-28T22:42:34.864 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.864+0000 7f949e1f7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c029d68570 0x55c029d6ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:34.864 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.864+0000 7f949e1f7740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c029d68e90 0x55c029d6f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:34.864 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.864+0000 7f949e1f7740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c029d71c60 0x55c029d6f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:34.864 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.864+0000 7f949e1f7740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c029d74830 con 0x55c029d68e90 2024-03-28T22:42:34.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.864+0000 7f949e1f7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55c029c69d00 con 0x55c029d68570 2024-03-28T22:42:34.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.864+0000 7f949e1f7740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55c029d749b0 con 0x55c029d71c60 2024-03-28T22:42:34.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.864+0000 7f949e1f7740 10 client.0 _ll_register_callbacks cb 0x55c029c54490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:34.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.864+0000 7f9498a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c029d68570 0x55c029d6ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:34.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.865+0000 7f9498a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c029d68570 0x55c029d6ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47940/0 (socket says 172.21.15.43:47940) 2024-03-28T22:42:34.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.865+0000 7f9498a00640 1 -- 172.21.15.43:0/1167284081 learned_addr learned my addr 172.21.15.43:0/1167284081 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:34.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.865+0000 7f9499400640 1 --2- 172.21.15.43:0/1167284081 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c029d71c60 0x55c029d6f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:34.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.865+0000 7f948be00640 1 --2- 172.21.15.43:0/1167284081 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c029d68e90 0x55c029d6f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:34.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.865+0000 7f9498a00640 1 -- 172.21.15.43:0/1167284081 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c029d71c60 msgr2=0x55c029d6f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:34.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.865+0000 7f9498a00640 1 --2- 172.21.15.43:0/1167284081 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c029d71c60 0x55c029d6f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:34.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.865+0000 7f9498a00640 1 -- 172.21.15.43:0/1167284081 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c029d68e90 msgr2=0x55c029d6f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:34.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.865+0000 7f9498a00640 1 --2- 172.21.15.43:0/1167284081 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c029d68e90 0x55c029d6f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:34.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.865+0000 7f9498a00640 1 -- 172.21.15.43:0/1167284081 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c029aac620 con 0x55c029d68570 2024-03-28T22:42:34.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.865+0000 7f9498a00640 1 --2- 172.21.15.43:0/1167284081 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c029d68570 0x55c029d6ed70 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f9484012a10 tx=0x7f9484012ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6021591fda408c74 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:34.866 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.865+0000 7f9482000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:42:34.866 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.865+0000 7f9482000640 1 -- 172.21.15.43:0/1167284081 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948400fac0 con 0x55c029d68570 2024-03-28T22:42:34.866 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.866+0000 7f9482000640 1 -- 172.21.15.43:0/1167284081 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f948400fc60 con 0x55c029d68570 2024-03-28T22:42:34.866 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.866+0000 7f949e1f7740 1 -- 172.21.15.43:0/1167284081 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f947c09b6c0 con 0x55c029d68570 2024-03-28T22:42:34.866 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.866+0000 7f9482000640 1 -- 172.21.15.43:0/1167284081 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f948400fe60 con 0x55c029d68570 2024-03-28T22:42:34.866 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.866+0000 7f949e1f7740 10 client.27620 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:34.866 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.866+0000 7f949e1f7740 1 -- 172.21.15.43:0/1167284081 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c029d80840 con 0x55c029d68570 2024-03-28T22:42:34.866 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.866+0000 7f9482000640 1 -- 172.21.15.43:0/1167284081 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f947c09b6c0 con 0x55c029d68570 2024-03-28T22:42:34.866 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.866+0000 7f9482000640 1 -- 172.21.15.43:0/1167284081 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f948401c5e0 con 0x55c029d68570 2024-03-28T22:42:34.867 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.867+0000 7f949e1f7740 10 client.27620 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:34.868 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.867+0000 7f949e1f7740 10 client.27620 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:42:34.868 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.867+0000 7f949e1f7740 20 client.27620 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:34.868 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.867+0000 7f949e1f7740 1 --2- 172.21.15.43:0/1167284081 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c029d81d50 0x55c029d84180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:34.868 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.867+0000 7f949e1f7740 4 client.27620 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:42:34.868 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.867+0000 7f949e1f7740 1 -- 172.21.15.43:0/1167284081 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c029cdf030 con 0x55c029d81d50 2024-03-28T22:42:34.868 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.867+0000 7f948be00640 1 --2- 172.21.15.43:0/1167284081 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c029d81d50 0x55c029d84180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:34.868 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.868+0000 7f948be00640 1 --2- 172.21.15.43:0/1167284081 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c029d81d50 0x55c029d84180 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f63dc272c85edcf6 server_cookie=3975d4da67fc14e6 in_seq=0 out_seq=0 2024-03-28T22:42:34.868 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.868+0000 7f9482000640 10 client.27620 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:34.868 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.869+0000 7f9482000640 1 -- 172.21.15.43:0/1167284081 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c029cdf030 con 0x55c029d81d50 2024-03-28T22:42:34.868 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.869+0000 7f9482000640 10 client.27620 handle_command_reply: tid=0 2024-03-28T22:42:34.868 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.869+0000 7f949e1f7740 1 client.27620 shutdown 2024-03-28T22:42:34.868 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.869+0000 7f949e1f7740 1 -- 172.21.15.43:0/1167284081 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c029d68570 msgr2=0x55c029d6ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:34.868 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.869+0000 7f949e1f7740 1 --2- 172.21.15.43:0/1167284081 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c029d68570 0x55c029d6ed70 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f9484012a10 tx=0x7f9484012ed0 comp rx=0 tx=0).stop 2024-03-28T22:42:34.869 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.869+0000 7f949e1f7740 1 -- 172.21.15.43:0/1167284081 shutdown_connections 2024-03-28T22:42:34.869 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.869+0000 7f949e1f7740 1 --2- 172.21.15.43:0/1167284081 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c029d71c60 0x55c029d6f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:34.869 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.869+0000 7f949e1f7740 1 --2- 172.21.15.43:0/1167284081 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c029d68e90 0x55c029d6f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:34.869 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.869+0000 7f949e1f7740 1 --2- 172.21.15.43:0/1167284081 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c029d81d50 0x55c029d84180 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:34.869 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.869+0000 7f949e1f7740 1 --2- 172.21.15.43:0/1167284081 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c029d68570 0x55c029d6ed70 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:34.869 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.869+0000 7f9482000640 0 client.27620 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:34.869 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.869+0000 7f949e1f7740 1 -- 172.21.15.43:0/1167284081 >> 172.21.15.43:0/1167284081 conn(0x55c029d62e90 msgr2=0x55c029d64be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:34.869 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.869+0000 7f949e1f7740 1 -- 172.21.15.43:0/1167284081 shutdown_connections 2024-03-28T22:42:34.869 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.869+0000 7f949e1f7740 1 -- 172.21.15.43:0/1167284081 wait complete. 2024-03-28T22:42:34.869 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.869+0000 7f949e1f7740 20 client.27620 trim_cache size 0 max 16384 2024-03-28T22:42:34.886 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.887+0000 7f949e1f7740 1 Processor -- start 2024-03-28T22:42:34.886 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.887+0000 7f949e1f7740 1 -- start start 2024-03-28T22:42:34.887 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.888+0000 7f949e1f7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c029d68570 0x55c029d714a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:34.887 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.888+0000 7f949e1f7740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c029d68e90 0x55c029d7d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:34.887 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.888+0000 7f949e1f7740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c029d71c60 0x55c029d89a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:34.887 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.888+0000 7f949e1f7740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f94941b0bd0 con 0x55c029d71c60 2024-03-28T22:42:34.887 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.888+0000 7f949e1f7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f94941b2710 con 0x55c029d68570 2024-03-28T22:42:34.887 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.888+0000 7f949e1f7740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f94941b12a0 con 0x55c029d68e90 2024-03-28T22:42:34.887 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.888+0000 7f9498a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c029d68570 0x55c029d714a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:34.887 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.888+0000 7f9498a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c029d68570 0x55c029d714a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47608/0 (socket says 172.21.15.43:47608) 2024-03-28T22:42:34.887 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.888+0000 7f9498a00640 1 -- 172.21.15.43:0/728236741 learned_addr learned my addr 172.21.15.43:0/728236741 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.888+0000 7f948be00640 1 --2- 172.21.15.43:0/728236741 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c029d68e90 0x55c029d7d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.888+0000 7f9499400640 1 --2- 172.21.15.43:0/728236741 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c029d71c60 0x55c029d89a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.888+0000 7f9498a00640 1 -- 172.21.15.43:0/728236741 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c029d68e90 msgr2=0x55c029d7d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.888+0000 7f9498a00640 1 --2- 172.21.15.43:0/728236741 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c029d68e90 0x55c029d7d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.888+0000 7f9498a00640 1 -- 172.21.15.43:0/728236741 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c029d71c60 msgr2=0x55c029d89a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.888+0000 7f9498a00640 1 --2- 172.21.15.43:0/728236741 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c029d71c60 0x55c029d89a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.888+0000 7f9498a00640 1 -- 172.21.15.43:0/728236741 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c029d63a60 con 0x55c029d68570 2024-03-28T22:42:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.889+0000 7f9498a00640 1 --2- 172.21.15.43:0/728236741 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c029d68570 0x55c029d714a0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f9484012560 tx=0x7f94840126c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=710e860a1ccb991a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.889+0000 7f9489200640 1 -- 172.21.15.43:0/728236741 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948400fac0 con 0x55c029d68570 2024-03-28T22:42:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.889+0000 7f9489200640 1 -- 172.21.15.43:0/728236741 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f948400fc60 con 0x55c029d68570 2024-03-28T22:42:34.889 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.889+0000 7f9489200640 1 -- 172.21.15.43:0/728236741 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94840048c0 con 0x55c029d68570 2024-03-28T22:42:34.892 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.890+0000 7f949e1f7740 1 -- 172.21.15.43:0/728236741 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c029d68570 msgr2=0x55c029d714a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:34.892 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.890+0000 7f949e1f7740 1 --2- 172.21.15.43:0/728236741 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c029d68570 0x55c029d714a0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f9484012560 tx=0x7f94840126c0 comp rx=0 tx=0).stop 2024-03-28T22:42:34.893 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.890+0000 7f949e1f7740 1 -- 172.21.15.43:0/728236741 shutdown_connections 2024-03-28T22:42:34.893 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.890+0000 7f949e1f7740 1 --2- 172.21.15.43:0/728236741 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c029d71c60 0x55c029d89a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:34.893 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.890+0000 7f949e1f7740 1 --2- 172.21.15.43:0/728236741 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c029d68e90 0x55c029d7d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:34.893 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.890+0000 7f949e1f7740 1 --2- 172.21.15.43:0/728236741 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c029d68570 0x55c029d714a0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:34.893 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.890+0000 7f949e1f7740 1 -- 172.21.15.43:0/728236741 >> 172.21.15.43:0/728236741 conn(0x55c029d62e90 msgr2=0x55c029d64b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:34.893 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.890+0000 7f949e1f7740 1 -- 172.21.15.43:0/728236741 shutdown_connections 2024-03-28T22:42:34.893 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.890+0000 7f949e1f7740 1 -- 172.21.15.43:0/728236741 wait complete. 2024-03-28T22:42:34.893 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.891+0000 7f949e1f7740 1 -- start start 2024-03-28T22:42:34.893 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.891+0000 7f949e1f7740 1 Processor -- start 2024-03-28T22:42:34.893 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.891+0000 7f949e1f7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c029d68570 0x55c029d75ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:34.893 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.891+0000 7f949e1f7740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c029d68e90 0x55c029d76220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:34.893 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.891+0000 7f949e1f7740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c029d71c60 0x55c029def8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:34.893 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.891+0000 7f949e1f7740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f94941b2040 con 0x55c029d71c60 2024-03-28T22:42:34.893 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.891+0000 7f949e1f7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f94941b1970 con 0x55c029d68570 2024-03-28T22:42:34.894 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.891+0000 7f949e1f7740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f94941aebb0 con 0x55c029d68e90 2024-03-28T22:42:34.894 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.892+0000 7f949e1f7740 10 client.0 _ll_register_callbacks cb 0x7f9484009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:34.894 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.892+0000 7f9498a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c029d68570 0x55c029d75ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:34.894 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.892+0000 7f9498a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c029d68570 0x55c029d75ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47616/0 (socket says 172.21.15.43:47616) 2024-03-28T22:42:34.894 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.892+0000 7f9498a00640 1 -- 172.21.15.43:0/3209836890 learned_addr learned my addr 172.21.15.43:0/3209836890 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:34.894 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.892+0000 7f948be00640 1 --2- 172.21.15.43:0/3209836890 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c029d68e90 0x55c029d76220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:34.894 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.892+0000 7f9499400640 1 --2- 172.21.15.43:0/3209836890 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c029d71c60 0x55c029def8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:34.894 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.892+0000 7f9498a00640 1 -- 172.21.15.43:0/3209836890 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c029d68e90 msgr2=0x55c029d76220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:34.894 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.892+0000 7f9498a00640 1 --2- 172.21.15.43:0/3209836890 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c029d68e90 0x55c029d76220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:34.894 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.892+0000 7f9498a00640 1 -- 172.21.15.43:0/3209836890 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c029d71c60 msgr2=0x55c029def8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:34.894 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.892+0000 7f9498a00640 1 --2- 172.21.15.43:0/3209836890 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c029d71c60 0x55c029def8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:34.894 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.892+0000 7f9498a00640 1 -- 172.21.15.43:0/3209836890 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c029defe60 con 0x55c029d68570 2024-03-28T22:42:34.894 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.892+0000 7f9498a00640 1 --2- 172.21.15.43:0/3209836890 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c029d68570 0x55c029d75ce0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f948400fe30 tx=0x7f948400f8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8b9a021029d919 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:34.894 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.892+0000 7f9482000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:42:34.895 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.893+0000 7f9482000640 1 -- 172.21.15.43:0/3209836890 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948400fac0 con 0x55c029d68570 2024-03-28T22:42:34.895 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.893+0000 7f9482000640 1 -- 172.21.15.43:0/3209836890 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f9484018040 con 0x55c029d68570 2024-03-28T22:42:34.895 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.893+0000 7f949e1f7740 1 -- 172.21.15.43:0/3209836890 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f948400fc60 con 0x55c029d68570 2024-03-28T22:42:34.895 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.893+0000 7f9482000640 1 -- 172.21.15.43:0/3209836890 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9484009260 con 0x55c029d68570 2024-03-28T22:42:34.895 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.893+0000 7f949e1f7740 10 client.27626 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:34.895 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.893+0000 7f949e1f7740 1 -- 172.21.15.43:0/3209836890 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c029df1170 con 0x55c029d68570 2024-03-28T22:42:34.895 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.893+0000 7f9482000640 1 -- 172.21.15.43:0/3209836890 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f948400fc60 con 0x55c029d68570 2024-03-28T22:42:34.895 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.894+0000 7f9482000640 1 -- 172.21.15.43:0/3209836890 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f948402a920 con 0x55c029d68570 2024-03-28T22:42:34.895 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.894+0000 7f949e1f7740 10 client.27626 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:34.895 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.894+0000 7f949e1f7740 10 client.27626 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:42:34.895 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.894+0000 7f949e1f7740 20 client.27626 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:34.895 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.894+0000 7f949e1f7740 1 --2- 172.21.15.43:0/3209836890 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c029df17d0 0x55c029df3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:34.895 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.894+0000 7f949e1f7740 4 client.27626 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T22:42:34.896 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.894+0000 7f949e1f7740 1 -- 172.21.15.43:0/3209836890 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c029cd8b80 con 0x55c029df17d0 2024-03-28T22:42:34.896 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.894+0000 7f948be00640 1 --2- 172.21.15.43:0/3209836890 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c029df17d0 0x55c029df3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:34.896 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.894+0000 7f948be00640 1 --2- 172.21.15.43:0/3209836890 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c029df17d0 0x55c029df3bc0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4384d7dbfedd4573 server_cookie=139f6a873f987773 in_seq=0 out_seq=0 2024-03-28T22:42:34.896 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:34.894+0000 7f9482000640 10 client.27626 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:35.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:34 smithi172 ceph-mon[36566]: pgmap v166: 129 pgs: 129 active+clean; 52 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 329 op/s 2024-03-28T22:42:35.149 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:35.150+0000 7f9482000640 1 -- 172.21.15.43:0/3209836890 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c029cd8b80 con 0x55c029df17d0 2024-03-28T22:42:35.149 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:35.150+0000 7f9482000640 10 client.27626 handle_command_reply: tid=0 2024-03-28T22:42:35.149 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:35.150+0000 7f949e1f7740 1 client.27626 shutdown 2024-03-28T22:42:35.149 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:35.151+0000 7f949e1f7740 1 -- 172.21.15.43:0/3209836890 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c029d68570 msgr2=0x55c029d75ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:35.149 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:35.151+0000 7f949e1f7740 1 --2- 172.21.15.43:0/3209836890 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c029d68570 0x55c029d75ce0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f948400fe30 tx=0x7f948400f8d0 comp rx=0 tx=0).stop 2024-03-28T22:42:35.150 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:35.151+0000 7f949e1f7740 1 -- 172.21.15.43:0/3209836890 shutdown_connections 2024-03-28T22:42:35.150 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:35.151+0000 7f949e1f7740 1 --2- 172.21.15.43:0/3209836890 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c029d71c60 0x55c029def8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:35.150 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:35.151+0000 7f949e1f7740 1 --2- 172.21.15.43:0/3209836890 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c029d68e90 0x55c029d76220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:35.150 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:35.151+0000 7f949e1f7740 1 --2- 172.21.15.43:0/3209836890 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c029df17d0 0x55c029df3bc0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:35.150 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:35.151+0000 7f949e1f7740 1 --2- 172.21.15.43:0/3209836890 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c029d68570 0x55c029d75ce0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:35.150 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:35.151+0000 7f9482000640 0 client.27626 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:35.151 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:35.151+0000 7f949e1f7740 1 -- 172.21.15.43:0/3209836890 >> 172.21.15.43:0/3209836890 conn(0x55c029d62e90 msgr2=0x55c029d61670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:35.151 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:35.151+0000 7f949e1f7740 1 -- 172.21.15.43:0/3209836890 shutdown_connections 2024-03-28T22:42:35.151 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:35.152+0000 7f949e1f7740 1 -- 172.21.15.43:0/3209836890 wait complete. 2024-03-28T22:42:35.152 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:35.152+0000 7f949e1f7740 20 client.27626 trim_cache size 0 max 16384 2024-03-28T22:42:35.152 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:{ 2024-03-28T22:42:35.152 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "message": "", 2024-03-28T22:42:35.152 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "return_code": 0 2024-03-28T22:42:35.152 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:} 2024-03-28T22:42:35.153 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:35.153+0000 7f9499e00640 1 -- 172.21.15.43:0/1244411623 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f94780a1f60 msgr2=0x7f94780a4420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:35.153 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:35.153+0000 7f9499e00640 1 --2- 172.21.15.43:0/1244411623 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f94780a1f60 0x7f94780a4420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f948c005f00 tx=0x7f948c009500 comp rx=0 tx=0).stop 2024-03-28T22:42:35.153 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:35.153+0000 7f9499e00640 1 -- 172.21.15.43:0/1244411623 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9494101f10 msgr2=0x7f949419a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:35.153 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:35.153+0000 7f9499e00640 1 --2- 172.21.15.43:0/1244411623 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9494101f10 0x7f949419a4a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f947c00e9f0 tx=0x7f947c00edf0 comp rx=0 tx=0).stop 2024-03-28T22:42:35.153 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:35.153+0000 7f9499e00640 1 -- 172.21.15.43:0/1244411623 shutdown_connections 2024-03-28T22:42:35.153 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:35.153+0000 7f9499e00640 1 --2- 172.21.15.43:0/1244411623 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f94780a1f60 0x7f94780a4420 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:35.153 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:35.153+0000 7f9499e00640 1 --2- 172.21.15.43:0/1244411623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9494107610 0x7f949419e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:35.153 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:35.153+0000 7f9499e00640 1 --2- 172.21.15.43:0/1244411623 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9494101f10 0x7f949419a4a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:35.153+0000 7f9499e00640 1 --2- 172.21.15.43:0/1244411623 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9494101550 0x7f9494199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:35.153+0000 7f9499e00640 1 -- 172.21.15.43:0/1244411623 >> 172.21.15.43:0/1244411623 conn(0x7f94940fd060 msgr2=0x7f94940fe4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:35.153+0000 7f9499e00640 1 -- 172.21.15.43:0/1244411623 shutdown_connections 2024-03-28T22:42:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:35.153+0000 7f9499e00640 1 -- 172.21.15.43:0/1244411623 wait complete. 2024-03-28T22:42:35.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:34 smithi082 ceph-mon[36471]: pgmap v166: 129 pgs: 129 active+clean; 52 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 329 op/s 2024-03-28T22:42:35.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:34 smithi043 ceph-mon[31823]: pgmap v166: 129 pgs: 129 active+clean; 52 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 329 op/s 2024-03-28T22:42:35.367 INFO:tasks.workunit.client.0.smithi043.stdout: 1 367152 7.77 MB/sec execute 126 sec latency 86.757 ms 2024-03-28T22:42:36.367 INFO:tasks.workunit.client.0.smithi043.stdout: 1 369001 7.80 MB/sec execute 127 sec latency 11.913 ms 2024-03-28T22:42:37.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:36 smithi172 ceph-mon[36566]: pgmap v167: 129 pgs: 129 active+clean; 52 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 153 op/s 2024-03-28T22:42:37.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:36 smithi082 ceph-mon[36471]: pgmap v167: 129 pgs: 129 active+clean; 52 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 153 op/s 2024-03-28T22:42:37.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:36 smithi043 ceph-mon[31823]: pgmap v167: 129 pgs: 129 active+clean; 52 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 153 op/s 2024-03-28T22:42:37.367 INFO:tasks.workunit.client.0.smithi043.stdout: 1 369810 7.76 MB/sec execute 128 sec latency 22.051 ms 2024-03-28T22:42:38.367 INFO:tasks.workunit.client.0.smithi043.stdout: 1 371877 7.79 MB/sec execute 129 sec latency 11.879 ms 2024-03-28T22:42:39.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:38 smithi172 ceph-mon[36566]: pgmap v168: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 328 op/s 2024-03-28T22:42:39.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:38 smithi082 ceph-mon[36471]: pgmap v168: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 328 op/s 2024-03-28T22:42:39.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:38 smithi043 ceph-mon[31823]: pgmap v168: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 328 op/s 2024-03-28T22:42:39.367 INFO:tasks.workunit.client.0.smithi043.stdout: 1 373110 7.78 MB/sec execute 130 sec latency 23.639 ms 2024-03-28T22:42:40.006 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:42:40.006 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:42:40.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.295+0000 7f2b3a600640 1 -- 172.21.15.43:0/854357433 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b34075c80 msgr2=0x7f2b34076060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:40.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.295+0000 7f2b3a600640 1 --2- 172.21.15.43:0/854357433 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b34075c80 0x7f2b34076060 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f2b24009930 tx=0x7f2b2402bc30 comp rx=0 tx=0).stop 2024-03-28T22:42:40.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.295+0000 7f2b3a600640 1 -- 172.21.15.43:0/854357433 shutdown_connections 2024-03-28T22:42:40.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.295+0000 7f2b3a600640 1 --2- 172.21.15.43:0/854357433 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b34112590 0x7f2b34114980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.295+0000 7f2b3a600640 1 --2- 172.21.15.43:0/854357433 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b340765a0 0x7f2b34112050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.295+0000 7f2b3a600640 1 --2- 172.21.15.43:0/854357433 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b34075c80 0x7f2b34076060 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.295+0000 7f2b3a600640 1 -- 172.21.15.43:0/854357433 >> 172.21.15.43:0/854357433 conn(0x7f2b340ff400 msgr2=0x7f2b34101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:40.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.295+0000 7f2b3a600640 1 -- 172.21.15.43:0/854357433 shutdown_connections 2024-03-28T22:42:40.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.295+0000 7f2b3a600640 1 -- 172.21.15.43:0/854357433 wait complete. 2024-03-28T22:42:40.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.296+0000 7f2b3a600640 1 Processor -- start 2024-03-28T22:42:40.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.296+0000 7f2b3a600640 1 -- start start 2024-03-28T22:42:40.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.296+0000 7f2b3a600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b34075c80 0x7f2b3419e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:40.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.296+0000 7f2b3a600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b340765a0 0x7f2b3419ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:40.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.296+0000 7f2b3a600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b34112590 0x7f2b341a2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:40.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.296+0000 7f2b3a600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2b34118130 con 0x7f2b34075c80 2024-03-28T22:42:40.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.296+0000 7f2b3a600640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f2b34117fb0 con 0x7f2b34112590 2024-03-28T22:42:40.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.296+0000 7f2b3a600640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f2b341182b0 con 0x7f2b340765a0 2024-03-28T22:42:40.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.297+0000 7f2b32a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b340765a0 0x7f2b3419ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:40.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.297+0000 7f2b33400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b34075c80 0x7f2b3419e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:40.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.297+0000 7f2b32a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b340765a0 0x7f2b3419ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:56714/0 (socket says 172.21.15.43:56714) 2024-03-28T22:42:40.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.297+0000 7f2b33e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b34112590 0x7f2b341a2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:40.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.297+0000 7f2b32a00640 1 -- 172.21.15.43:0/30687727 learned_addr learned my addr 172.21.15.43:0/30687727 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:40.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.297+0000 7f2b32a00640 1 -- 172.21.15.43:0/30687727 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b34112590 msgr2=0x7f2b341a2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:40.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.297+0000 7f2b32a00640 1 --2- 172.21.15.43:0/30687727 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b34112590 0x7f2b341a2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.297+0000 7f2b32a00640 1 -- 172.21.15.43:0/30687727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b34075c80 msgr2=0x7f2b3419e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:40.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.297+0000 7f2b32a00640 1 --2- 172.21.15.43:0/30687727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b34075c80 0x7f2b3419e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.297+0000 7f2b32a00640 1 -- 172.21.15.43:0/30687727 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b341a36c0 con 0x7f2b340765a0 2024-03-28T22:42:40.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.297+0000 7f2b33400640 1 --2- 172.21.15.43:0/30687727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b34075c80 0x7f2b3419e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:40.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.297+0000 7f2b32a00640 1 --2- 172.21.15.43:0/30687727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b340765a0 0x7f2b3419ec10 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2b2000ea60 tx=0x7f2b2000ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:40.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.298+0000 7f2b2be00640 1 -- 172.21.15.43:0/30687727 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b2000cd20 con 0x7f2b340765a0 2024-03-28T22:42:40.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.298+0000 7f2b2be00640 1 -- 172.21.15.43:0/30687727 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b200035f0 con 0x7f2b340765a0 2024-03-28T22:42:40.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.298+0000 7f2b2be00640 1 -- 172.21.15.43:0/30687727 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b20010690 con 0x7f2b340765a0 2024-03-28T22:42:40.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.298+0000 7f2b3a600640 1 -- 172.21.15.43:0/30687727 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b341a39b0 con 0x7f2b340765a0 2024-03-28T22:42:40.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.298+0000 7f2b3a600640 1 -- 172.21.15.43:0/30687727 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b341ab240 con 0x7f2b340765a0 2024-03-28T22:42:40.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.301+0000 7f2b3d203740 1 Processor -- start 2024-03-28T22:42:40.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.301+0000 7f2b3d203740 1 -- start start 2024-03-28T22:42:40.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.302+0000 7f2b3d203740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b1546a6bc0 0x55b1546a5910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:40.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.302+0000 7f2b3d203740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b1546a5ee0 0x55b1546a6360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:40.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.302+0000 7f2b3d203740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b1546af250 0x55b1546b1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:40.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.302+0000 7f2b3d203740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b15460a100 con 0x55b1546af250 2024-03-28T22:42:40.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.302+0000 7f2b3d203740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55b1545b2ba0 con 0x55b1546a5ee0 2024-03-28T22:42:40.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.302+0000 7f2b3d203740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55b1546061a0 con 0x55b1546a6bc0 2024-03-28T22:42:40.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.302+0000 7f2b2be00640 1 -- 172.21.15.43:0/30687727 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f2b200037a0 con 0x7f2b340765a0 2024-03-28T22:42:40.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.302+0000 7f2b33e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b1546a6bc0 0x55b1546a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:40.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.302+0000 7f2b33400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b1546a5ee0 0x55b1546a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:40.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.302+0000 7f2b33e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b1546a6bc0 0x55b1546a5910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:56726/0 (socket says 172.21.15.43:56726) 2024-03-28T22:42:40.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.302+0000 7f2b33e00640 1 -- 172.21.15.43:0/4290708082 learned_addr learned my addr 172.21.15.43:0/4290708082 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:40.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.302+0000 7f2b32a00640 1 --2- 172.21.15.43:0/4290708082 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b1546af250 0x55b1546b1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:40.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.302+0000 7f2b33e00640 1 -- 172.21.15.43:0/4290708082 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b1546a5ee0 msgr2=0x55b1546a6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:40.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.302+0000 7f2b33e00640 1 --2- 172.21.15.43:0/4290708082 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b1546a5ee0 0x55b1546a6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.302+0000 7f2b33e00640 1 -- 172.21.15.43:0/4290708082 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b1546af250 msgr2=0x55b1546b1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:40.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.302+0000 7f2b33e00640 1 --2- 172.21.15.43:0/4290708082 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b1546af250 0x55b1546b1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.302+0000 7f2b33e00640 1 -- 172.21.15.43:0/4290708082 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1545e31a0 con 0x55b1546a6bc0 2024-03-28T22:42:40.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.302+0000 7f2b32a00640 1 --2- 172.21.15.43:0/4290708082 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b1546af250 0x55b1546b1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:42:40.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.303+0000 7f2b33e00640 1 --2- 172.21.15.43:0/4290708082 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b1546a6bc0 0x55b1546a5910 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c009f40 tx=0x7f2b2c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11edb881d4222f9f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:40.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.303+0000 7f2b28c00640 1 -- 172.21.15.43:0/4290708082 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c004d20 con 0x55b1546a6bc0 2024-03-28T22:42:40.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.303+0000 7f2b28c00640 1 -- 172.21.15.43:0/4290708082 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b2c004050 con 0x55b1546a6bc0 2024-03-28T22:42:40.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.303+0000 7f2b28c00640 1 -- 172.21.15.43:0/4290708082 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c005280 con 0x55b1546a6bc0 2024-03-28T22:42:40.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.303+0000 7f2b2be00640 1 --2- 172.21.15.43:0/30687727 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f2b180a1ef0 0x7f2b180a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:40.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.303+0000 7f2b3d203740 1 -- 172.21.15.43:0/4290708082 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b1546a6bc0 msgr2=0x55b1546a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:40.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.303+0000 7f2b3d203740 1 --2- 172.21.15.43:0/4290708082 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b1546a6bc0 0x55b1546a5910 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c009f40 tx=0x7f2b2c0118d0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.304+0000 7f2b3d203740 1 -- 172.21.15.43:0/4290708082 shutdown_connections 2024-03-28T22:42:40.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.304+0000 7f2b3d203740 1 --2- 172.21.15.43:0/4290708082 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b1546a5ee0 0x55b1546a6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.304+0000 7f2b2be00640 1 -- 172.21.15.43:0/30687727 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f2b200ce820 con 0x7f2b340765a0 2024-03-28T22:42:40.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.304+0000 7f2b3d203740 1 --2- 172.21.15.43:0/4290708082 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b1546af250 0x55b1546b1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.304+0000 7f2b3d203740 1 --2- 172.21.15.43:0/4290708082 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b1546a6bc0 0x55b1546a5910 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.304+0000 7f2b3d203740 1 -- 172.21.15.43:0/4290708082 >> 172.21.15.43:0/4290708082 conn(0x55b15466ca10 msgr2=0x55b1546a0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:40.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.304+0000 7f2b33400640 1 --2- 172.21.15.43:0/30687727 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f2b180a1ef0 0x7f2b180a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:40.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.304+0000 7f2b3d203740 1 -- 172.21.15.43:0/4290708082 shutdown_connections 2024-03-28T22:42:40.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.304+0000 7f2b3d203740 1 -- 172.21.15.43:0/4290708082 wait complete. 2024-03-28T22:42:40.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.304+0000 7f2b3d203740 1 -- start start 2024-03-28T22:42:40.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.304+0000 7f2b3d203740 1 Processor -- start 2024-03-28T22:42:40.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.304+0000 7f2b33400640 1 --2- 172.21.15.43:0/30687727 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f2b180a1ef0 0x7f2b180a43b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2b24009900 tx=0x7f2b2402df90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:40.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.305+0000 7f2b3d203740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b1546a5ee0 0x55b1546ac230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:40.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.305+0000 7f2b3d203740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b1546a6bc0 0x55b1546ac770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:40.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.305+0000 7f2b3d203740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b1546af250 0x55b1546accb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:40.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.305+0000 7f2b3d203740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b1546b20c0 con 0x55b1546af250 2024-03-28T22:42:40.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.305+0000 7f2b3d203740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55b1545a8250 con 0x55b1546a5ee0 2024-03-28T22:42:40.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.305+0000 7f2b3d203740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55b1546b2240 con 0x55b1546a6bc0 2024-03-28T22:42:40.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.305+0000 7f2b33400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b1546a5ee0 0x55b1546ac230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:40.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.305+0000 7f2b3d203740 10 client.0 _ll_register_callbacks cb 0x55b15431dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:40.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.305+0000 7f2b33400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b1546a5ee0 0x55b1546ac230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47654/0 (socket says 172.21.15.43:47654) 2024-03-28T22:42:40.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.305+0000 7f2b33400640 1 -- 172.21.15.43:0/1502380576 learned_addr learned my addr 172.21.15.43:0/1502380576 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:40.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.305+0000 7f2b32a00640 1 --2- 172.21.15.43:0/1502380576 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b1546a6bc0 0x55b1546ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:40.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.305+0000 7f2b33e00640 1 --2- 172.21.15.43:0/1502380576 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b1546af250 0x55b1546accb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:40.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.306+0000 7f2b33400640 1 -- 172.21.15.43:0/1502380576 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b1546a6bc0 msgr2=0x55b1546ac770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:40.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.306+0000 7f2b33400640 1 --2- 172.21.15.43:0/1502380576 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b1546a6bc0 0x55b1546ac770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.306+0000 7f2b33400640 1 -- 172.21.15.43:0/1502380576 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b1546af250 msgr2=0x55b1546accb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:40.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.306+0000 7f2b33400640 1 --2- 172.21.15.43:0/1502380576 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b1546af250 0x55b1546accb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.306+0000 7f2b33400640 1 -- 172.21.15.43:0/1502380576 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1545c4960 con 0x55b1546a5ee0 2024-03-28T22:42:40.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.306+0000 7f2b33e00640 1 --2- 172.21.15.43:0/1502380576 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b1546af250 0x55b1546accb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:42:40.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.306+0000 7f2b32a00640 1 --2- 172.21.15.43:0/1502380576 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b1546a6bc0 0x55b1546ac770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:40.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.306+0000 7f2b33400640 1 --2- 172.21.15.43:0/1502380576 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b1546a5ee0 0x55b1546ac230 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f2b24009930 tx=0x7f2b24009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98fcd1f3188671b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:40.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.306+0000 7f2b1d600640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:42:40.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.306+0000 7f2b1d600640 1 -- 172.21.15.43:0/1502380576 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b24046070 con 0x55b1546a5ee0 2024-03-28T22:42:40.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.306+0000 7f2b3d203740 1 -- 172.21.15.43:0/1502380576 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b2c004050 con 0x55b1546a5ee0 2024-03-28T22:42:40.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.306+0000 7f2b1d600640 1 -- 172.21.15.43:0/1502380576 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f2b24004450 con 0x55b1546a5ee0 2024-03-28T22:42:40.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.307+0000 7f2b1d600640 1 -- 172.21.15.43:0/1502380576 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2b24004ee0 con 0x55b1546a5ee0 2024-03-28T22:42:40.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.307+0000 7f2b3d203740 10 client.27638 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:40.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.307+0000 7f2b3d203740 1 -- 172.21.15.43:0/1502380576 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b1543e87d0 con 0x55b1546a5ee0 2024-03-28T22:42:40.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.307+0000 7f2b1d600640 1 -- 172.21.15.43:0/1502380576 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b2c004050 con 0x55b1546a5ee0 2024-03-28T22:42:40.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.307+0000 7f2b1d600640 1 -- 172.21.15.43:0/1502380576 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f2b24034440 con 0x55b1546a5ee0 2024-03-28T22:42:40.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.307+0000 7f2b3d203740 10 client.27638 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:40.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.307+0000 7f2b3d203740 10 client.27638 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:42:40.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.307+0000 7f2b3d203740 20 client.27638 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:40.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.307+0000 7f2b3d203740 1 --2- 172.21.15.43:0/1502380576 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b1546bf5d0 0x55b1546c1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:40.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.307+0000 7f2b3d203740 4 client.27638 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:42:40.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.307+0000 7f2b3d203740 1 -- 172.21.15.43:0/1502380576 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b154617640 con 0x55b1546bf5d0 2024-03-28T22:42:40.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.307+0000 7f2b32a00640 1 --2- 172.21.15.43:0/1502380576 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b1546bf5d0 0x55b1546c1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:40.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.308+0000 7f2b32a00640 1 --2- 172.21.15.43:0/1502380576 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b1546bf5d0 0x55b1546c1a90 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f09f1d56fa2867b server_cookie=f3c06590699a196f in_seq=0 out_seq=0 2024-03-28T22:42:40.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.308+0000 7f2b1d600640 10 client.27638 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:40.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.309+0000 7f2b1d600640 1 -- 172.21.15.43:0/1502380576 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b154617640 con 0x55b1546bf5d0 2024-03-28T22:42:40.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.309+0000 7f2b1d600640 10 client.27638 handle_command_reply: tid=0 2024-03-28T22:42:40.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.309+0000 7f2b3d203740 1 client.27638 shutdown 2024-03-28T22:42:40.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.309+0000 7f2b3d203740 1 -- 172.21.15.43:0/1502380576 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b1546a5ee0 msgr2=0x55b1546ac230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:40.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.309+0000 7f2b3d203740 1 --2- 172.21.15.43:0/1502380576 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b1546a5ee0 0x55b1546ac230 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f2b24009930 tx=0x7f2b24009260 comp rx=0 tx=0).stop 2024-03-28T22:42:40.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.309+0000 7f2b3d203740 1 -- 172.21.15.43:0/1502380576 shutdown_connections 2024-03-28T22:42:40.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.309+0000 7f2b3d203740 1 --2- 172.21.15.43:0/1502380576 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b1546af250 0x55b1546accb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.309+0000 7f2b3d203740 1 --2- 172.21.15.43:0/1502380576 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b1546a6bc0 0x55b1546ac770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.309+0000 7f2b3d203740 1 --2- 172.21.15.43:0/1502380576 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b1546bf5d0 0x55b1546c1a90 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.309+0000 7f2b3d203740 1 --2- 172.21.15.43:0/1502380576 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b1546a5ee0 0x55b1546ac230 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.309+0000 7f2b3d203740 1 -- 172.21.15.43:0/1502380576 >> 172.21.15.43:0/1502380576 conn(0x55b15466ca10 msgr2=0x55b1546a1df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:40.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.309+0000 7f2b1d600640 0 client.27638 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:40.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.310+0000 7f2b3d203740 1 -- 172.21.15.43:0/1502380576 shutdown_connections 2024-03-28T22:42:40.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.310+0000 7f2b3d203740 1 -- 172.21.15.43:0/1502380576 wait complete. 2024-03-28T22:42:40.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.310+0000 7f2b3d203740 20 client.27638 trim_cache size 0 max 16384 2024-03-28T22:42:40.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.346+0000 7f2b3d203740 1 Processor -- start 2024-03-28T22:42:40.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.347+0000 7f2b3d203740 1 -- start start 2024-03-28T22:42:40.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.347+0000 7f2b3d203740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b1546a5ee0 0x55b1546abb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:40.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.347+0000 7f2b3d203740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b1546a6bc0 0x55b1546baea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:40.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.347+0000 7f2b3d203740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b1546af250 0x55b15469f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:40.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.347+0000 7f2b3d203740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2b341b5200 con 0x55b1546a6bc0 2024-03-28T22:42:40.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.347+0000 7f2b3d203740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f2b341b6d40 con 0x55b1546a5ee0 2024-03-28T22:42:40.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.347+0000 7f2b3d203740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f2b341b58d0 con 0x55b1546af250 2024-03-28T22:42:40.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.347+0000 7f2b33400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b1546a5ee0 0x55b1546abb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:40.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.347+0000 7f2b33400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b1546a5ee0 0x55b1546abb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47670/0 (socket says 172.21.15.43:47670) 2024-03-28T22:42:40.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.347+0000 7f2b33400640 1 -- 172.21.15.43:0/1405329786 learned_addr learned my addr 172.21.15.43:0/1405329786 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:40.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.347+0000 7f2b32a00640 1 --2- 172.21.15.43:0/1405329786 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b1546a6bc0 0x55b1546baea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:40.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.348+0000 7f2b33e00640 1 --2- 172.21.15.43:0/1405329786 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b1546af250 0x55b15469f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:40.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.348+0000 7f2b33400640 1 -- 172.21.15.43:0/1405329786 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b1546af250 msgr2=0x55b15469f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:40.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.348+0000 7f2b33400640 1 --2- 172.21.15.43:0/1405329786 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b1546af250 0x55b15469f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.348+0000 7f2b33400640 1 -- 172.21.15.43:0/1405329786 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b1546a6bc0 msgr2=0x55b1546baea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:40.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.348+0000 7f2b33400640 1 --2- 172.21.15.43:0/1405329786 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b1546a6bc0 0x55b1546baea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.348+0000 7f2b33400640 1 -- 172.21.15.43:0/1405329786 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b15431e3f0 con 0x55b1546a5ee0 2024-03-28T22:42:40.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.348+0000 7f2b33e00640 1 --2- 172.21.15.43:0/1405329786 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b1546af250 0x55b15469f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:42:40.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.348+0000 7f2b32a00640 1 --2- 172.21.15.43:0/1405329786 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b1546a6bc0 0x55b1546baea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:40.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.349+0000 7f2b33400640 1 --2- 172.21.15.43:0/1405329786 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b1546a5ee0 0x55b1546abb70 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f2b24034880 tx=0x7f2b240040a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=939670d5253bff80 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:40.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.350+0000 7f2b1fe00640 1 -- 172.21.15.43:0/1405329786 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b24034af0 con 0x55b1546a5ee0 2024-03-28T22:42:40.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.350+0000 7f2b1fe00640 1 -- 172.21.15.43:0/1405329786 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b24034c90 con 0x55b1546a5ee0 2024-03-28T22:42:40.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.350+0000 7f2b1fe00640 1 -- 172.21.15.43:0/1405329786 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b24003830 con 0x55b1546a5ee0 2024-03-28T22:42:40.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.350+0000 7f2b3d203740 1 -- 172.21.15.43:0/1405329786 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b1546a5ee0 msgr2=0x55b1546abb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:40.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.350+0000 7f2b3d203740 1 --2- 172.21.15.43:0/1405329786 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b1546a5ee0 0x55b1546abb70 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f2b24034880 tx=0x7f2b240040a0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.351+0000 7f2b3d203740 1 -- 172.21.15.43:0/1405329786 shutdown_connections 2024-03-28T22:42:40.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.351+0000 7f2b3d203740 1 --2- 172.21.15.43:0/1405329786 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b1546af250 0x55b15469f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.351+0000 7f2b3d203740 1 --2- 172.21.15.43:0/1405329786 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b1546a6bc0 0x55b1546baea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.351+0000 7f2b3d203740 1 --2- 172.21.15.43:0/1405329786 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b1546a5ee0 0x55b1546abb70 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.351+0000 7f2b3d203740 1 -- 172.21.15.43:0/1405329786 >> 172.21.15.43:0/1405329786 conn(0x55b15466ca10 msgr2=0x55b15469f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:40.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.351+0000 7f2b3d203740 1 -- 172.21.15.43:0/1405329786 shutdown_connections 2024-03-28T22:42:40.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.352+0000 7f2b3d203740 1 -- 172.21.15.43:0/1405329786 wait complete. 2024-03-28T22:42:40.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.353+0000 7f2b3d203740 1 -- start start 2024-03-28T22:42:40.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.353+0000 7f2b3d203740 1 Processor -- start 2024-03-28T22:42:40.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.354+0000 7f2b3d203740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b1546a5ee0 0x55b1546b76c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:40.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.354+0000 7f2b3d203740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b1546a6bc0 0x55b1546a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:40.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.354+0000 7f2b3d203740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b1546af250 0x55b1546a8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:40.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.354+0000 7f2b3d203740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2b341b6670 con 0x55b1546af250 2024-03-28T22:42:40.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.354+0000 7f2b3d203740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f2b341b5fa0 con 0x55b1546a6bc0 2024-03-28T22:42:40.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.354+0000 7f2b3d203740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f2b341b31e0 con 0x55b1546a5ee0 2024-03-28T22:42:40.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.354+0000 7f2b3d203740 10 client.0 _ll_register_callbacks cb 0x7f2b24009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:40.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.355+0000 7f2b33e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b1546af250 0x55b1546a8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:40.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.355+0000 7f2b33e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b1546af250 0x55b1546a8fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60276/0 (socket says 172.21.15.43:60276) 2024-03-28T22:42:40.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.355+0000 7f2b33400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b1546a5ee0 0x55b1546b76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:40.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.355+0000 7f2b33e00640 1 -- 172.21.15.43:0/2496090220 learned_addr learned my addr 172.21.15.43:0/2496090220 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:40.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.356+0000 7f2b32a00640 1 --2- 172.21.15.43:0/2496090220 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b1546a6bc0 0x55b1546a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:40.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.356+0000 7f2b33e00640 1 -- 172.21.15.43:0/2496090220 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b1546a5ee0 msgr2=0x55b1546b76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:40.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.356+0000 7f2b33e00640 1 --2- 172.21.15.43:0/2496090220 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b1546a5ee0 0x55b1546b76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.356+0000 7f2b33e00640 1 -- 172.21.15.43:0/2496090220 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b1546a6bc0 msgr2=0x55b1546a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:40.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.356+0000 7f2b33e00640 1 --2- 172.21.15.43:0/2496090220 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b1546a6bc0 0x55b1546a8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.356+0000 7f2b33e00640 1 -- 172.21.15.43:0/2496090220 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1542de0f0 con 0x55b1546af250 2024-03-28T22:42:40.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.357+0000 7f2b33e00640 1 --2- 172.21.15.43:0/2496090220 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b1546af250 0x55b1546a8fe0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c005c10 tx=0x7f2b2c005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dd33d17975405b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:40.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.357+0000 7f2b1d600640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:42:40.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.357+0000 7f2b1d600640 1 -- 172.21.15.43:0/2496090220 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c028070 con 0x55b1546af250 2024-03-28T22:42:40.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.357+0000 7f2b1d600640 1 -- 172.21.15.43:0/2496090220 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f2b2c0057f0 con 0x55b1546af250 2024-03-28T22:42:40.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.357+0000 7f2b3d203740 1 -- 172.21.15.43:0/2496090220 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b24034c90 con 0x55b1546af250 2024-03-28T22:42:40.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.358+0000 7f2b1d600640 1 -- 172.21.15.43:0/2496090220 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2b2c02d460 con 0x55b1546af250 2024-03-28T22:42:40.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.358+0000 7f2b3d203740 10 client.17421 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:40.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.358+0000 7f2b3d203740 1 -- 172.21.15.43:0/2496090220 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b154736670 con 0x55b1546af250 2024-03-28T22:42:40.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.359+0000 7f2b1d600640 1 -- 172.21.15.43:0/2496090220 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b24034c90 con 0x55b1546af250 2024-03-28T22:42:40.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.359+0000 7f2b1d600640 1 -- 172.21.15.43:0/2496090220 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f2b2c00f760 con 0x55b1546af250 2024-03-28T22:42:40.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.359+0000 7f2b3d203740 10 client.17421 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:40.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.359+0000 7f2b3d203740 10 client.17421 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:42:40.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.359+0000 7f2b3d203740 20 client.17421 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:40.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.359+0000 7f2b3d203740 1 --2- 172.21.15.43:0/2496090220 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b154736c90 0x55b1547390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:40.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.359+0000 7f2b3d203740 4 client.17421 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:42:40.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.359+0000 7f2b3d203740 1 -- 172.21.15.43:0/2496090220 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b154611990 con 0x55b154736c90 2024-03-28T22:42:40.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.360+0000 7f2b33400640 1 --2- 172.21.15.43:0/2496090220 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b154736c90 0x55b1547390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:40.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.360+0000 7f2b33400640 1 --2- 172.21.15.43:0/2496090220 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b154736c90 0x55b1547390b0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a173dcc4f578cf8 server_cookie=1be4b33f17d5ad1f in_seq=0 out_seq=0 2024-03-28T22:42:40.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.360+0000 7f2b1d600640 10 client.17421 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:40.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.361+0000 7f2b1d600640 1 -- 172.21.15.43:0/2496090220 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b154611990 con 0x55b154736c90 2024-03-28T22:42:40.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.361+0000 7f2b1d600640 10 client.17421 handle_command_reply: tid=0 2024-03-28T22:42:40.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.362+0000 7f2b3d203740 1 client.17421 shutdown 2024-03-28T22:42:40.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.362+0000 7f2b3d203740 1 -- 172.21.15.43:0/2496090220 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b1546af250 msgr2=0x55b1546a8fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:40.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.362+0000 7f2b3d203740 1 --2- 172.21.15.43:0/2496090220 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b1546af250 0x55b1546a8fe0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c005c10 tx=0x7f2b2c005270 comp rx=0 tx=0).stop 2024-03-28T22:42:40.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.362+0000 7f2b3d203740 1 -- 172.21.15.43:0/2496090220 shutdown_connections 2024-03-28T22:42:40.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.362+0000 7f2b3d203740 1 --2- 172.21.15.43:0/2496090220 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b154736c90 0x55b1547390b0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.362+0000 7f2b3d203740 1 --2- 172.21.15.43:0/2496090220 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b1546a6bc0 0x55b1546a8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.362+0000 7f2b3d203740 1 --2- 172.21.15.43:0/2496090220 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b1546af250 0x55b1546a8fe0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.362+0000 7f2b1d600640 0 client.17421 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:40.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.362+0000 7f2b3d203740 1 --2- 172.21.15.43:0/2496090220 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b1546a5ee0 0x55b1546b76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.362+0000 7f2b3d203740 1 -- 172.21.15.43:0/2496090220 >> 172.21.15.43:0/2496090220 conn(0x55b15466ca10 msgr2=0x55b1546ac1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:40.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.362+0000 7f2b3d203740 1 -- 172.21.15.43:0/2496090220 shutdown_connections 2024-03-28T22:42:40.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.363+0000 7f2b3d203740 1 -- 172.21.15.43:0/2496090220 wait complete. 2024-03-28T22:42:40.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.363+0000 7f2b3d203740 20 client.17421 trim_cache size 0 max 16384 2024-03-28T22:42:40.361 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:42:40.361 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-03-28T22:42:40.361 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-03-28T22:42:40.361 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:42:40.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.364+0000 7f2b3a600640 1 -- 172.21.15.43:0/30687727 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f2b180a1ef0 msgr2=0x7f2b180a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:40.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.364+0000 7f2b3a600640 1 --2- 172.21.15.43:0/30687727 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f2b180a1ef0 0x7f2b180a43b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2b24009900 tx=0x7f2b2402df90 comp rx=0 tx=0).stop 2024-03-28T22:42:40.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.364+0000 7f2b3a600640 1 -- 172.21.15.43:0/30687727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b340765a0 msgr2=0x7f2b3419ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:40.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.364+0000 7f2b3a600640 1 --2- 172.21.15.43:0/30687727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b340765a0 0x7f2b3419ec10 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2b2000ea60 tx=0x7f2b2000ee60 comp rx=0 tx=0).stop 2024-03-28T22:42:40.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.364+0000 7f2b3a600640 1 -- 172.21.15.43:0/30687727 shutdown_connections 2024-03-28T22:42:40.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.364+0000 7f2b3a600640 1 --2- 172.21.15.43:0/30687727 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f2b180a1ef0 0x7f2b180a43b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.364+0000 7f2b3a600640 1 --2- 172.21.15.43:0/30687727 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b34112590 0x7f2b341a2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.364+0000 7f2b3a600640 1 --2- 172.21.15.43:0/30687727 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2b340765a0 0x7f2b3419ec10 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.364+0000 7f2b3a600640 1 --2- 172.21.15.43:0/30687727 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2b34075c80 0x7f2b3419e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.364+0000 7f2b3a600640 1 -- 172.21.15.43:0/30687727 >> 172.21.15.43:0/30687727 conn(0x7f2b340ff400 msgr2=0x7f2b341017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:40.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.364+0000 7f2b3a600640 1 -- 172.21.15.43:0/30687727 shutdown_connections 2024-03-28T22:42:40.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.365+0000 7f2b3a600640 1 -- 172.21.15.43:0/30687727 wait complete. 2024-03-28T22:42:40.367 INFO:tasks.workunit.client.0.smithi043.stdout: 1 374826 7.77 MB/sec execute 131 sec latency 48.065 ms 2024-03-28T22:42:40.389 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:42:40.389 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:42:40.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.592+0000 7fc09a200640 1 -- 172.21.15.43:0/1200779250 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0941042a0 msgr2=0x7fc0941107b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:40.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.592+0000 7fc09a200640 1 --2- 172.21.15.43:0/1200779250 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0941042a0 0x7fc0941107b0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc08c009ca0 tx=0x7fc08c02c100 comp rx=0 tx=0).stop 2024-03-28T22:42:40.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.592+0000 7fc09a200640 1 -- 172.21.15.43:0/1200779250 shutdown_connections 2024-03-28T22:42:40.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.592+0000 7fc09a200640 1 --2- 172.21.15.43:0/1200779250 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0941042a0 0x7fc0941107b0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.592+0000 7fc09a200640 1 --2- 172.21.15.43:0/1200779250 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0941038e0 0x7fc094103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.592+0000 7fc09a200640 1 --2- 172.21.15.43:0/1200779250 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0941098f0 0x7fc094109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.592+0000 7fc09a200640 1 -- 172.21.15.43:0/1200779250 >> 172.21.15.43:0/1200779250 conn(0x7fc0940ff3b0 msgr2=0x7fc0941017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:40.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.592+0000 7fc09a200640 1 -- 172.21.15.43:0/1200779250 shutdown_connections 2024-03-28T22:42:40.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.592+0000 7fc09a200640 1 -- 172.21.15.43:0/1200779250 wait complete. 2024-03-28T22:42:40.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.593+0000 7fc09a200640 1 Processor -- start 2024-03-28T22:42:40.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.594+0000 7fc09a200640 1 -- start start 2024-03-28T22:42:40.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.594+0000 7fc09a200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0941038e0 0x7fc09410e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:40.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.594+0000 7fc09a200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0941042a0 0x7fc09410eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:40.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.594+0000 7fc09a200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0941098f0 0x7fc09410f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:40.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.594+0000 7fc09a200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc0940766c0 con 0x7fc0941038e0 2024-03-28T22:42:40.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.594+0000 7fc09a200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fc094076540 con 0x7fc0941098f0 2024-03-28T22:42:40.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.594+0000 7fc09a200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fc094076840 con 0x7fc0941042a0 2024-03-28T22:42:40.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.594+0000 7fc092a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0941042a0 0x7fc09410eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:40.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.594+0000 7fc093e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0941098f0 0x7fc09410f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:40.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.594+0000 7fc092a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0941042a0 0x7fc09410eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:56774/0 (socket says 172.21.15.43:56774) 2024-03-28T22:42:40.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.594+0000 7fc093400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0941038e0 0x7fc09410e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:40.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.594+0000 7fc092a00640 1 -- 172.21.15.43:0/3848555051 learned_addr learned my addr 172.21.15.43:0/3848555051 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:40.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.595+0000 7fc093e00640 1 -- 172.21.15.43:0/3848555051 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0941042a0 msgr2=0x7fc09410eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:40.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.595+0000 7fc093e00640 1 --2- 172.21.15.43:0/3848555051 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0941042a0 0x7fc09410eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.595+0000 7fc093e00640 1 -- 172.21.15.43:0/3848555051 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0941038e0 msgr2=0x7fc09410e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:40.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.595+0000 7fc093e00640 1 --2- 172.21.15.43:0/3848555051 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0941038e0 0x7fc09410e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.595+0000 7fc093e00640 1 -- 172.21.15.43:0/3848555051 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0941aa860 con 0x7fc0941098f0 2024-03-28T22:42:40.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.595+0000 7fc092a00640 1 --2- 172.21.15.43:0/3848555051 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0941042a0 0x7fc09410eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:40.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.595+0000 7fc093e00640 1 --2- 172.21.15.43:0/3848555051 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0941098f0 0x7fc09410f420 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fc08c009bf0 tx=0x7fc08c034670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:40.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.595+0000 7fc08be00640 1 -- 172.21.15.43:0/3848555051 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc08c039050 con 0x7fc0941098f0 2024-03-28T22:42:40.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.595+0000 7fc08be00640 1 -- 172.21.15.43:0/3848555051 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc08c045070 con 0x7fc0941098f0 2024-03-28T22:42:40.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.595+0000 7fc08be00640 1 -- 172.21.15.43:0/3848555051 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc08c0045e0 con 0x7fc0941098f0 2024-03-28T22:42:40.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.595+0000 7fc09a200640 1 -- 172.21.15.43:0/3848555051 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0941aaaf0 con 0x7fc0941098f0 2024-03-28T22:42:40.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.596+0000 7fc09a200640 1 -- 172.21.15.43:0/3848555051 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0941aaf60 con 0x7fc0941098f0 2024-03-28T22:42:40.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.597+0000 7fc09a200640 1 -- 172.21.15.43:0/3848555051 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc094068990 con 0x7fc0941098f0 2024-03-28T22:42:40.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.599+0000 7fc08be00640 1 -- 172.21.15.43:0/3848555051 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fc08c003970 con 0x7fc0941098f0 2024-03-28T22:42:40.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.600+0000 7fc08be00640 1 --2- 172.21.15.43:0/3848555051 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc0780a1ef0 0x7fc0780a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:40.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.600+0000 7fc08be00640 1 -- 172.21.15.43:0/3848555051 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fc08c0f0b30 con 0x7fc0941098f0 2024-03-28T22:42:40.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.600+0000 7fc093400640 1 --2- 172.21.15.43:0/3848555051 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc0780a1ef0 0x7fc0780a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:40.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.601+0000 7fc093400640 1 --2- 172.21.15.43:0/3848555051 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc0780a1ef0 0x7fc0780a43b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc084009b50 tx=0x7fc084005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:40.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.605+0000 7fc08be00640 1 -- 172.21.15.43:0/3848555051 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc08c0bd930 con 0x7fc0941098f0 2024-03-28T22:42:40.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.884+0000 7fc09a200640 1 -- 172.21.15.43:0/3848555051 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc094078530 con 0x7fc0941098f0 2024-03-28T22:42:40.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.885+0000 7fc08be00640 1 -- 172.21.15.43:0/3848555051 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11931 (secure 0 0 0) 0x7fc08c0bd230 con 0x7fc0941098f0 2024-03-28T22:42:40.883 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:42:40.883 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:33:28.906396+0000","modified":"2024-03-28T22:40:08.672168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24572,"mds_1":24556,"mds_2":24535,"mds_3":24523,"mds_4":24518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26717":{"gid":26717,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/215779342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":215779342},{"type":"v1","addr":"172.21.15.82:6835","nonce":215779342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24535,24556,24518,24523,24572]},"id":1}]} 2024-03-28T22:42:40.883 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 32 2024-03-28T22:42:40.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.888+0000 7fc09a200640 1 -- 172.21.15.43:0/3848555051 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc0780a1ef0 msgr2=0x7fc0780a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:40.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.888+0000 7fc09a200640 1 --2- 172.21.15.43:0/3848555051 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc0780a1ef0 0x7fc0780a43b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc084009b50 tx=0x7fc084005f90 comp rx=0 tx=0).stop 2024-03-28T22:42:40.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.888+0000 7fc09a200640 1 -- 172.21.15.43:0/3848555051 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0941098f0 msgr2=0x7fc09410f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:40.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.888+0000 7fc09a200640 1 --2- 172.21.15.43:0/3848555051 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0941098f0 0x7fc09410f420 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fc08c009bf0 tx=0x7fc08c034670 comp rx=0 tx=0).stop 2024-03-28T22:42:40.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.888+0000 7fc09a200640 1 -- 172.21.15.43:0/3848555051 shutdown_connections 2024-03-28T22:42:40.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.888+0000 7fc09a200640 1 --2- 172.21.15.43:0/3848555051 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc0780a1ef0 0x7fc0780a43b0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.888+0000 7fc09a200640 1 --2- 172.21.15.43:0/3848555051 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0941098f0 0x7fc09410f420 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.888+0000 7fc09a200640 1 --2- 172.21.15.43:0/3848555051 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc0941042a0 0x7fc09410eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.888+0000 7fc09a200640 1 --2- 172.21.15.43:0/3848555051 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0941038e0 0x7fc09410e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:40.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.888+0000 7fc09a200640 1 -- 172.21.15.43:0/3848555051 >> 172.21.15.43:0/3848555051 conn(0x7fc0940ff3b0 msgr2=0x7fc0940ffab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:40.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.888+0000 7fc09a200640 1 -- 172.21.15.43:0/3848555051 shutdown_connections 2024-03-28T22:42:40.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:40.888+0000 7fc09a200640 1 -- 172.21.15.43:0/3848555051 wait complete. 2024-03-28T22:42:40.905 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T22:42:41.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.130+0000 7fd402800640 1 -- 172.21.15.43:0/2231129438 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3fc10df40 msgr2=0x7fd3fc110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:41.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.130+0000 7fd402800640 1 --2- 172.21.15.43:0/2231129438 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3fc10df40 0x7fd3fc110330 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4009900 tx=0x7fd3f402bc40 comp rx=0 tx=0).stop 2024-03-28T22:42:41.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.130+0000 7fd402800640 1 -- 172.21.15.43:0/2231129438 shutdown_connections 2024-03-28T22:42:41.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.130+0000 7fd402800640 1 --2- 172.21.15.43:0/2231129438 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3fc10df40 0x7fd3fc110330 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.130+0000 7fd402800640 1 --2- 172.21.15.43:0/2231129438 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3fc100d40 0x7fd3fc10d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.130+0000 7fd402800640 1 --2- 172.21.15.43:0/2231129438 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd3fc100390 0x7fd3fc100770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.130+0000 7fd402800640 1 -- 172.21.15.43:0/2231129438 >> 172.21.15.43:0/2231129438 conn(0x7fd3fc0fbfa0 msgr2=0x7fd3fc0fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:41.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.131+0000 7fd402800640 1 -- 172.21.15.43:0/2231129438 shutdown_connections 2024-03-28T22:42:41.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.131+0000 7fd402800640 1 -- 172.21.15.43:0/2231129438 wait complete. 2024-03-28T22:42:41.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.132+0000 7fd402800640 1 Processor -- start 2024-03-28T22:42:41.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.132+0000 7fd402800640 1 -- start start 2024-03-28T22:42:41.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.132+0000 7fd402800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3fc100390 0x7fd3fc19a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.132+0000 7fd402800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3fc100d40 0x7fd3fc19a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.132+0000 7fd402800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd3fc10df40 0x7fd3fc19e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.132+0000 7fd402800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd3fc113bb0 con 0x7fd3fc100390 2024-03-28T22:42:41.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.132+0000 7fd402800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd3fc113a30 con 0x7fd3fc10df40 2024-03-28T22:42:41.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.132+0000 7fd402800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd3fc113d30 con 0x7fd3fc100d40 2024-03-28T22:42:41.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.133+0000 7fd401400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3fc100390 0x7fd3fc19a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.133+0000 7fd401e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd3fc10df40 0x7fd3fc19e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.133+0000 7fd401e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd3fc10df40 0x7fd3fc19e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47724/0 (socket says 172.21.15.43:47724) 2024-03-28T22:42:41.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.133+0000 7fd401e00640 1 -- 172.21.15.43:0/3979897989 learned_addr learned my addr 172.21.15.43:0/3979897989 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:41.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.133+0000 7fd401400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3fc100390 0x7fd3fc19a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60316/0 (socket says 172.21.15.43:60316) 2024-03-28T22:42:41.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.133+0000 7fd400a00640 1 --2- 172.21.15.43:0/3979897989 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3fc100d40 0x7fd3fc19a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.133+0000 7fd401e00640 1 -- 172.21.15.43:0/3979897989 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3fc100d40 msgr2=0x7fd3fc19a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:41.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.133+0000 7fd401e00640 1 --2- 172.21.15.43:0/3979897989 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3fc100d40 0x7fd3fc19a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.133+0000 7fd401e00640 1 -- 172.21.15.43:0/3979897989 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3fc100390 msgr2=0x7fd3fc19a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:41.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.133+0000 7fd401e00640 1 --2- 172.21.15.43:0/3979897989 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3fc100390 0x7fd3fc19a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.133+0000 7fd401e00640 1 -- 172.21.15.43:0/3979897989 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3fc19f070 con 0x7fd3fc10df40 2024-03-28T22:42:41.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.133+0000 7fd401e00640 1 --2- 172.21.15.43:0/3979897989 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd3fc10df40 0x7fd3fc19e940 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4005940 tx=0x7fd3f402cb40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:41.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.134+0000 7fd3fa000640 1 -- 172.21.15.43:0/3979897989 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f4041070 con 0x7fd3fc10df40 2024-03-28T22:42:41.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.134+0000 7fd402800640 1 -- 172.21.15.43:0/3979897989 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3fc19f300 con 0x7fd3fc10df40 2024-03-28T22:42:41.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.134+0000 7fd3fa000640 1 -- 172.21.15.43:0/3979897989 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3f4004450 con 0x7fd3fc10df40 2024-03-28T22:42:41.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.134+0000 7fd3fa000640 1 -- 172.21.15.43:0/3979897989 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f4003c90 con 0x7fd3fc10df40 2024-03-28T22:42:41.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.134+0000 7fd402800640 1 -- 172.21.15.43:0/3979897989 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3fc1a6bf0 con 0x7fd3fc10df40 2024-03-28T22:42:41.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.137+0000 7fd406b90740 1 Processor -- start 2024-03-28T22:42:41.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.137+0000 7fd3fa000640 1 -- 172.21.15.43:0/3979897989 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fd3f4044070 con 0x7fd3fc10df40 2024-03-28T22:42:41.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.137+0000 7fd406b90740 1 -- start start 2024-03-28T22:42:41.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.138+0000 7fd406b90740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557592f0e550 0x557592f0e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.138+0000 7fd406b90740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557592f0ee70 0x557592f17510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.138+0000 7fd406b90740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557592f17a50 0x557592f19e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.138+0000 7fd406b90740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x557592e72050 con 0x557592f0ee70 2024-03-28T22:42:41.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.138+0000 7fd406b90740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x557592e1ace0 con 0x557592f17a50 2024-03-28T22:42:41.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.138+0000 7fd406b90740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x557592e6e0f0 con 0x557592f0e550 2024-03-28T22:42:41.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.138+0000 7fd400a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557592f0e550 0x557592f0e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.138+0000 7fd400a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557592f0e550 0x557592f0e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:56804/0 (socket says 172.21.15.43:56804) 2024-03-28T22:42:41.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.138+0000 7fd400a00640 1 -- 172.21.15.43:0/550452936 learned_addr learned my addr 172.21.15.43:0/550452936 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:41.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.138+0000 7fd400a00640 1 --2- 172.21.15.43:0/550452936 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557592f17a50 0x557592f19e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.138+0000 7fd401400640 1 --2- 172.21.15.43:0/550452936 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557592f0ee70 0x557592f17510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.138+0000 7fd400a00640 1 -- 172.21.15.43:0/550452936 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557592f17a50 msgr2=0x557592f19e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:41.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.138+0000 7fd400a00640 1 --2- 172.21.15.43:0/550452936 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557592f17a50 0x557592f19e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.138+0000 7fd400a00640 1 -- 172.21.15.43:0/550452936 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557592f0ee70 msgr2=0x557592f17510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:41.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.138+0000 7fd400a00640 1 --2- 172.21.15.43:0/550452936 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557592f0ee70 0x557592f17510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.138+0000 7fd400a00640 1 -- 172.21.15.43:0/550452936 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557592b863f0 con 0x557592f0e550 2024-03-28T22:42:41.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.139+0000 7fd400a00640 1 --2- 172.21.15.43:0/550452936 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557592f0e550 0x557592f0e930 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fd3f0005f40 tx=0x7fd3f0009990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e88cc0a13f51dfb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:41.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.139+0000 7fd3ea600640 1 -- 172.21.15.43:0/550452936 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f0003970 con 0x557592f0e550 2024-03-28T22:42:41.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.139+0000 7fd3ea600640 1 -- 172.21.15.43:0/550452936 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3f000f880 con 0x557592f0e550 2024-03-28T22:42:41.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.139+0000 7fd3ea600640 1 -- 172.21.15.43:0/550452936 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f00045e0 con 0x557592f0e550 2024-03-28T22:42:41.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.139+0000 7fd3fa000640 1 --2- 172.21.15.43:0/3979897989 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd3e40a1ef0 0x7fd3e40a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.139+0000 7fd3fa000640 1 -- 172.21.15.43:0/3979897989 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fd3f40eff90 con 0x7fd3fc10df40 2024-03-28T22:42:41.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.139+0000 7fd401400640 1 --2- 172.21.15.43:0/3979897989 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd3e40a1ef0 0x7fd3e40a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.140+0000 7fd406b90740 1 -- 172.21.15.43:0/550452936 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557592f0e550 msgr2=0x557592f0e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:41.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.140+0000 7fd406b90740 1 --2- 172.21.15.43:0/550452936 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557592f0e550 0x557592f0e930 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fd3f0005f40 tx=0x7fd3f0009990 comp rx=0 tx=0).stop 2024-03-28T22:42:41.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.140+0000 7fd406b90740 1 -- 172.21.15.43:0/550452936 shutdown_connections 2024-03-28T22:42:41.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.140+0000 7fd406b90740 1 --2- 172.21.15.43:0/550452936 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557592f17a50 0x557592f19e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.140+0000 7fd406b90740 1 --2- 172.21.15.43:0/550452936 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557592f0ee70 0x557592f17510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.140+0000 7fd406b90740 1 --2- 172.21.15.43:0/550452936 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557592f0e550 0x557592f0e930 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.140+0000 7fd406b90740 1 -- 172.21.15.43:0/550452936 >> 172.21.15.43:0/550452936 conn(0x557592f08c80 msgr2=0x557592f09060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:41.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.140+0000 7fd406b90740 1 -- 172.21.15.43:0/550452936 shutdown_connections 2024-03-28T22:42:41.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.140+0000 7fd401400640 1 --2- 172.21.15.43:0/3979897989 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd3e40a1ef0 0x7fd3e40a43b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x557592f0e020 tx=0x7fd3ec005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:41.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.140+0000 7fd406b90740 1 -- 172.21.15.43:0/550452936 wait complete. 2024-03-28T22:42:41.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.140+0000 7fd406b90740 1 -- start start 2024-03-28T22:42:41.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.141+0000 7fd406b90740 1 Processor -- start 2024-03-28T22:42:41.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.141+0000 7fd406b90740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557592f0e550 0x557592f14a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.141+0000 7fd406b90740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557592f0ee70 0x557592f14fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.141+0000 7fd406b90740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557592f17a50 0x557592f15510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.141+0000 7fd406b90740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x557592f1a7e0 con 0x557592f17a50 2024-03-28T22:42:41.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.141+0000 7fd406b90740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x557592e102f0 con 0x557592f0ee70 2024-03-28T22:42:41.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.141+0000 7fd406b90740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x557592f1a960 con 0x557592f0e550 2024-03-28T22:42:41.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.141+0000 7fd406b90740 10 client.0 _ll_register_callbacks cb 0x557592b85be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:41.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.141+0000 7fd401400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557592f0e550 0x557592f14a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.141+0000 7fd401400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557592f0e550 0x557592f14a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:56808/0 (socket says 172.21.15.43:56808) 2024-03-28T22:42:41.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.141+0000 7fd400a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557592f0ee70 0x557592f14fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.141+0000 7fd401400640 1 -- 172.21.15.43:0/3276738090 learned_addr learned my addr 172.21.15.43:0/3276738090 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:41.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.141+0000 7fd401e00640 1 --2- 172.21.15.43:0/3276738090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557592f17a50 0x557592f15510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.142+0000 7fd401400640 1 -- 172.21.15.43:0/3276738090 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557592f0ee70 msgr2=0x557592f14fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:41.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.142+0000 7fd401400640 1 --2- 172.21.15.43:0/3276738090 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557592f0ee70 0x557592f14fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.142+0000 7fd401400640 1 -- 172.21.15.43:0/3276738090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557592f17a50 msgr2=0x557592f15510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:41.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.142+0000 7fd401400640 1 --2- 172.21.15.43:0/3276738090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557592f17a50 0x557592f15510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.142+0000 7fd401400640 1 -- 172.21.15.43:0/3276738090 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557592c507d0 con 0x557592f0e550 2024-03-28T22:42:41.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.142+0000 7fd400a00640 1 --2- 172.21.15.43:0/3276738090 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557592f0ee70 0x557592f14fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:41.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.142+0000 7fd401e00640 1 --2- 172.21.15.43:0/3276738090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557592f17a50 0x557592f15510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:42:41.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.142+0000 7fd401400640 1 --2- 172.21.15.43:0/3276738090 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557592f0e550 0x557592f14a90 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec01ea00 tx=0x7fd3ec004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1887da489c231ef server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:41.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.142+0000 7fd3dea00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:42:41.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.142+0000 7fd3dea00640 1 -- 172.21.15.43:0/3276738090 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec005020 con 0x557592f0e550 2024-03-28T22:42:41.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.142+0000 7fd3dea00640 1 -- 172.21.15.43:0/3276738090 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fd3ec018030 con 0x557592f0e550 2024-03-28T22:42:41.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.142+0000 7fd406b90740 1 -- 172.21.15.43:0/3276738090 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3f000f880 con 0x557592f0e550 2024-03-28T22:42:41.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.143+0000 7fd3dea00640 1 -- 172.21.15.43:0/3276738090 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd3ec034050 con 0x557592f0e550 2024-03-28T22:42:41.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.143+0000 7fd406b90740 10 client.27604 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:41.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.143+0000 7fd406b90740 1 -- 172.21.15.43:0/3276738090 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557592f26b50 con 0x557592f0e550 2024-03-28T22:42:41.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.143+0000 7fd3dea00640 1 -- 172.21.15.43:0/3276738090 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3f000f880 con 0x557592f0e550 2024-03-28T22:42:41.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.143+0000 7fd3dea00640 1 -- 172.21.15.43:0/3276738090 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd3ec029420 con 0x557592f0e550 2024-03-28T22:42:41.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.143+0000 7fd406b90740 10 client.27604 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:41.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.143+0000 7fd406b90740 10 client.27604 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:42:41.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.143+0000 7fd406b90740 20 client.27604 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:41.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.143+0000 7fd406b90740 1 --2- 172.21.15.43:0/3276738090 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x557592f28060 0x557592f2a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.143+0000 7fd406b90740 4 client.27604 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:42:41.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.143+0000 7fd406b90740 1 -- 172.21.15.43:0/3276738090 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557592e1b680 con 0x557592f28060 2024-03-28T22:42:41.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.144+0000 7fd400a00640 1 --2- 172.21.15.43:0/3276738090 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x557592f28060 0x557592f2a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.144+0000 7fd400a00640 1 --2- 172.21.15.43:0/3276738090 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x557592f28060 0x557592f2a4e0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=309779e98f2c6f23 server_cookie=f338972ec3bc8c35 in_seq=0 out_seq=0 2024-03-28T22:42:41.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.144+0000 7fd3dea00640 10 client.27604 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:42:41.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.145+0000 7fd3dea00640 1 -- 172.21.15.43:0/3276738090 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557592e1b680 con 0x557592f28060 2024-03-28T22:42:41.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.145+0000 7fd3dea00640 10 client.27604 handle_command_reply: tid=0 2024-03-28T22:42:41.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.145+0000 7fd406b90740 1 client.27604 shutdown 2024-03-28T22:42:41.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.145+0000 7fd406b90740 1 -- 172.21.15.43:0/3276738090 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557592f0e550 msgr2=0x557592f14a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:41.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.145+0000 7fd406b90740 1 --2- 172.21.15.43:0/3276738090 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557592f0e550 0x557592f14a90 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec01ea00 tx=0x7fd3ec004730 comp rx=0 tx=0).stop 2024-03-28T22:42:41.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.145+0000 7fd406b90740 1 -- 172.21.15.43:0/3276738090 shutdown_connections 2024-03-28T22:42:41.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.145+0000 7fd406b90740 1 --2- 172.21.15.43:0/3276738090 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x557592f28060 0x557592f2a4e0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.145+0000 7fd406b90740 1 --2- 172.21.15.43:0/3276738090 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557592f0ee70 0x557592f14fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.145+0000 7fd406b90740 1 --2- 172.21.15.43:0/3276738090 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557592f17a50 0x557592f15510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.145+0000 7fd3dea00640 0 client.27604 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:42:41.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.145+0000 7fd406b90740 1 --2- 172.21.15.43:0/3276738090 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557592f0e550 0x557592f14a90 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.145+0000 7fd406b90740 1 -- 172.21.15.43:0/3276738090 >> 172.21.15.43:0/3276738090 conn(0x557592f08c80 msgr2=0x557592f0a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:41.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.145+0000 7fd406b90740 1 -- 172.21.15.43:0/3276738090 shutdown_connections 2024-03-28T22:42:41.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.145+0000 7fd406b90740 1 -- 172.21.15.43:0/3276738090 wait complete. 2024-03-28T22:42:41.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.145+0000 7fd406b90740 20 client.27604 trim_cache size 0 max 16384 2024-03-28T22:42:41.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:40 smithi172 ceph-mon[36566]: pgmap v169: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 250 op/s 2024-03-28T22:42:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.160+0000 7fd406b90740 1 Processor -- start 2024-03-28T22:42:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.160+0000 7fd406b90740 1 -- start start 2024-03-28T22:42:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.161+0000 7fd406b90740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557592f0e550 0x557592f13bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.161+0000 7fd406b90740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557592f0ee70 0x557592f23580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.161+0000 7fd406b90740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557592f17a50 0x557592f15330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.161+0000 7fd406b90740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd3fc1b0b60 con 0x557592f0e550 2024-03-28T22:42:41.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.161+0000 7fd406b90740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd3fc1b26a0 con 0x557592f0ee70 2024-03-28T22:42:41.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.161+0000 7fd406b90740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd3fc1b1230 con 0x557592f17a50 2024-03-28T22:42:41.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.161+0000 7fd401e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557592f17a50 0x557592f15330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.161+0000 7fd400a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557592f0ee70 0x557592f23580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.161+0000 7fd401e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557592f17a50 0x557592f15330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:56816/0 (socket says 172.21.15.43:56816) 2024-03-28T22:42:41.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.161+0000 7fd401400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557592f0e550 0x557592f13bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.161+0000 7fd401e00640 1 -- 172.21.15.43:0/1427460899 learned_addr learned my addr 172.21.15.43:0/1427460899 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:41.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.161+0000 7fd401e00640 1 -- 172.21.15.43:0/1427460899 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557592f0ee70 msgr2=0x557592f23580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:41.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.161+0000 7fd401e00640 1 --2- 172.21.15.43:0/1427460899 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557592f0ee70 0x557592f23580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.161+0000 7fd401e00640 1 -- 172.21.15.43:0/1427460899 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557592f0e550 msgr2=0x557592f13bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:41.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.161+0000 7fd401e00640 1 --2- 172.21.15.43:0/1427460899 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557592f0e550 0x557592f13bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.161+0000 7fd401e00640 1 -- 172.21.15.43:0/1427460899 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557592f19e40 con 0x557592f17a50 2024-03-28T22:42:41.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.161+0000 7fd400a00640 1 --2- 172.21.15.43:0/1427460899 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557592f0ee70 0x557592f23580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:41.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.162+0000 7fd401e00640 1 --2- 172.21.15.43:0/1427460899 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557592f17a50 0x557592f15330 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd3f402c180 tx=0x7fd3f40bc340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=812784450e1ddaf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:41.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.162+0000 7fd3e9c00640 1 -- 172.21.15.43:0/1427460899 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f404b040 con 0x557592f17a50 2024-03-28T22:42:41.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.162+0000 7fd3e9c00640 1 -- 172.21.15.43:0/1427460899 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3f40c07e0 con 0x557592f17a50 2024-03-28T22:42:41.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.162+0000 7fd3e9c00640 1 -- 172.21.15.43:0/1427460899 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f40b8f40 con 0x557592f17a50 2024-03-28T22:42:41.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.162+0000 7fd406b90740 1 -- 172.21.15.43:0/1427460899 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557592f17a50 msgr2=0x557592f15330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:41.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.162+0000 7fd406b90740 1 --2- 172.21.15.43:0/1427460899 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557592f17a50 0x557592f15330 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd3f402c180 tx=0x7fd3f40bc340 comp rx=0 tx=0).stop 2024-03-28T22:42:41.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.162+0000 7fd406b90740 1 -- 172.21.15.43:0/1427460899 shutdown_connections 2024-03-28T22:42:41.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.162+0000 7fd406b90740 1 --2- 172.21.15.43:0/1427460899 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557592f0ee70 0x557592f23580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.162+0000 7fd406b90740 1 --2- 172.21.15.43:0/1427460899 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557592f17a50 0x557592f15330 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.162+0000 7fd406b90740 1 --2- 172.21.15.43:0/1427460899 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557592f0e550 0x557592f13bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.162+0000 7fd406b90740 1 -- 172.21.15.43:0/1427460899 >> 172.21.15.43:0/1427460899 conn(0x557592f08c80 msgr2=0x557592f07530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:41.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.163+0000 7fd406b90740 1 -- 172.21.15.43:0/1427460899 shutdown_connections 2024-03-28T22:42:41.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.163+0000 7fd406b90740 1 -- 172.21.15.43:0/1427460899 wait complete. 2024-03-28T22:42:41.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.163+0000 7fd406b90740 1 -- start start 2024-03-28T22:42:41.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.163+0000 7fd406b90740 1 Processor -- start 2024-03-28T22:42:41.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.163+0000 7fd406b90740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557592f0e550 0x557592f1fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.163+0000 7fd406b90740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557592f0ee70 0x557592f1af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.164+0000 7fd406b90740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557592f17a50 0x557592f1b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.164+0000 7fd406b90740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd3fc1b1fd0 con 0x557592f0ee70 2024-03-28T22:42:41.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.164+0000 7fd406b90740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd3fc1b1900 con 0x557592f0e550 2024-03-28T22:42:41.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.164+0000 7fd406b90740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd3fc1aeb40 con 0x557592f17a50 2024-03-28T22:42:41.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.164+0000 7fd406b90740 10 client.0 _ll_register_callbacks cb 0x7fd3ec009a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:41.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.164+0000 7fd401400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557592f0e550 0x557592f1fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.164+0000 7fd401400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557592f0e550 0x557592f1fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47768/0 (socket says 172.21.15.43:47768) 2024-03-28T22:42:41.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.164+0000 7fd401400640 1 -- 172.21.15.43:0/462525440 learned_addr learned my addr 172.21.15.43:0/462525440 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:41.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.164+0000 7fd401e00640 1 --2- 172.21.15.43:0/462525440 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557592f17a50 0x557592f1b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.164+0000 7fd400a00640 1 --2- 172.21.15.43:0/462525440 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557592f0ee70 0x557592f1af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.164+0000 7fd401400640 1 -- 172.21.15.43:0/462525440 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557592f17a50 msgr2=0x557592f1b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:41.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.164+0000 7fd401400640 1 --2- 172.21.15.43:0/462525440 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557592f17a50 0x557592f1b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.164+0000 7fd401400640 1 -- 172.21.15.43:0/462525440 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557592f0ee70 msgr2=0x557592f1af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:41.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.164+0000 7fd401400640 1 --2- 172.21.15.43:0/462525440 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557592f0ee70 0x557592f1af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.164+0000 7fd401400640 1 -- 172.21.15.43:0/462525440 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557592f1ba00 con 0x557592f0e550 2024-03-28T22:42:41.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.164+0000 7fd400a00640 1 --2- 172.21.15.43:0/462525440 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557592f0ee70 0x557592f1af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:42:41.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.164+0000 7fd401400640 1 --2- 172.21.15.43:0/462525440 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557592f0e550 0x557592f1fc20 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec029a30 tx=0x7fd3ec0296d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be946c288a430f88 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:41.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.164+0000 7fd3dea00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:42:41.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.165+0000 7fd3dea00640 1 -- 172.21.15.43:0/462525440 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec01e7c0 con 0x557592f0e550 2024-03-28T22:42:41.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.165+0000 7fd406b90740 1 -- 172.21.15.43:0/462525440 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3f40c07e0 con 0x557592f0e550 2024-03-28T22:42:41.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.165+0000 7fd3dea00640 1 -- 172.21.15.43:0/462525440 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fd3ec0047a0 con 0x557592f0e550 2024-03-28T22:42:41.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.165+0000 7fd3dea00640 1 -- 172.21.15.43:0/462525440 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd3ec047050 con 0x557592f0e550 2024-03-28T22:42:41.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.165+0000 7fd406b90740 10 client.27668 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:41.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.165+0000 7fd406b90740 1 -- 172.21.15.43:0/462525440 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557592f8e0e0 con 0x557592f0e550 2024-03-28T22:42:41.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.165+0000 7fd3dea00640 1 -- 172.21.15.43:0/462525440 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3f40c07e0 con 0x557592f0e550 2024-03-28T22:42:41.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.165+0000 7fd3dea00640 1 -- 172.21.15.43:0/462525440 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd3ec042030 con 0x557592f0e550 2024-03-28T22:42:41.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.165+0000 7fd406b90740 10 client.27668 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:41.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.165+0000 7fd406b90740 10 client.27668 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:42:41.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.165+0000 7fd406b90740 20 client.27668 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:41.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.165+0000 7fd406b90740 1 --2- 172.21.15.43:0/462525440 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x557592f8e7c0 0x557592f90be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.165+0000 7fd406b90740 4 client.27668 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:42:41.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.165+0000 7fd406b90740 1 -- 172.21.15.43:0/462525440 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557592e7f590 con 0x557592f8e7c0 2024-03-28T22:42:41.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.166+0000 7fd400a00640 1 --2- 172.21.15.43:0/462525440 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x557592f8e7c0 0x557592f90be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.166+0000 7fd400a00640 1 --2- 172.21.15.43:0/462525440 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x557592f8e7c0 0x557592f90be0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c04bad00cf6a9961 server_cookie=6ede663e5a9d4389 in_seq=0 out_seq=0 2024-03-28T22:42:41.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.166+0000 7fd3dea00640 10 client.27668 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:42:41.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.166+0000 7fd3dea00640 1 -- 172.21.15.43:0/462525440 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557592e7f590 con 0x557592f8e7c0 2024-03-28T22:42:41.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.166+0000 7fd3dea00640 10 client.27668 handle_command_reply: tid=0 2024-03-28T22:42:41.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.166+0000 7fd406b90740 1 client.27668 shutdown 2024-03-28T22:42:41.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.167+0000 7fd406b90740 1 -- 172.21.15.43:0/462525440 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557592f0e550 msgr2=0x557592f1fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:41.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.167+0000 7fd406b90740 1 --2- 172.21.15.43:0/462525440 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557592f0e550 0x557592f1fc20 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec029a30 tx=0x7fd3ec0296d0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.167+0000 7fd406b90740 1 -- 172.21.15.43:0/462525440 shutdown_connections 2024-03-28T22:42:41.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.167+0000 7fd406b90740 1 --2- 172.21.15.43:0/462525440 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x557592f8e7c0 0x557592f90be0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.167+0000 7fd406b90740 1 --2- 172.21.15.43:0/462525440 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557592f17a50 0x557592f1b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.167+0000 7fd406b90740 1 --2- 172.21.15.43:0/462525440 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557592f0ee70 0x557592f1af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.167+0000 7fd406b90740 1 --2- 172.21.15.43:0/462525440 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557592f0e550 0x557592f1fc20 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.167+0000 7fd3dea00640 0 client.27668 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:42:41.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.167+0000 7fd406b90740 1 -- 172.21.15.43:0/462525440 >> 172.21.15.43:0/462525440 conn(0x557592f08c80 msgr2=0x557592f0a810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:41.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.167+0000 7fd406b90740 1 -- 172.21.15.43:0/462525440 shutdown_connections 2024-03-28T22:42:41.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.167+0000 7fd406b90740 1 -- 172.21.15.43:0/462525440 wait complete. 2024-03-28T22:42:41.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.167+0000 7fd406b90740 20 client.27668 trim_cache size 0 max 16384 2024-03-28T22:42:41.173 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:42:41.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.167+0000 7fd402800640 1 -- 172.21.15.43:0/3979897989 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd3e40a1ef0 msgr2=0x7fd3e40a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:41.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.167+0000 7fd402800640 1 --2- 172.21.15.43:0/3979897989 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd3e40a1ef0 0x7fd3e40a43b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x557592f0e020 tx=0x7fd3ec005f90 comp rx=0 tx=0).stop 2024-03-28T22:42:41.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.167+0000 7fd402800640 1 -- 172.21.15.43:0/3979897989 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd3fc10df40 msgr2=0x7fd3fc19e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:41.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.167+0000 7fd402800640 1 --2- 172.21.15.43:0/3979897989 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd3fc10df40 0x7fd3fc19e940 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4005940 tx=0x7fd3f402cb40 comp rx=0 tx=0).stop 2024-03-28T22:42:41.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.168+0000 7fd402800640 1 -- 172.21.15.43:0/3979897989 shutdown_connections 2024-03-28T22:42:41.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.168+0000 7fd402800640 1 --2- 172.21.15.43:0/3979897989 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd3e40a1ef0 0x7fd3e40a43b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.168+0000 7fd402800640 1 --2- 172.21.15.43:0/3979897989 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd3fc10df40 0x7fd3fc19e940 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.168+0000 7fd402800640 1 --2- 172.21.15.43:0/3979897989 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3fc100d40 0x7fd3fc19a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.168+0000 7fd402800640 1 --2- 172.21.15.43:0/3979897989 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3fc100390 0x7fd3fc19a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.168+0000 7fd402800640 1 -- 172.21.15.43:0/3979897989 >> 172.21.15.43:0/3979897989 conn(0x7fd3fc0fbfa0 msgr2=0x7fd3fc0fe390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:41.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.168+0000 7fd402800640 1 -- 172.21.15.43:0/3979897989 shutdown_connections 2024-03-28T22:42:41.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.168+0000 7fd402800640 1 -- 172.21.15.43:0/3979897989 wait complete. 2024-03-28T22:42:41.178 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:40 smithi043 ceph-mon[31823]: pgmap v169: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 250 op/s 2024-03-28T22:42:41.181 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T22:42:41.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:40 smithi082 ceph-mon[36471]: pgmap v169: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 250 op/s 2024-03-28T22:42:41.367 INFO:tasks.workunit.client.0.smithi043.stdout: 1 376362 7.81 MB/sec execute 132 sec latency 19.022 ms 2024-03-28T22:42:41.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.421+0000 7f38c1a00640 1 -- 172.21.15.43:0/2596160939 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f38bc1126d0 msgr2=0x7f38bc114ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:41.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.421+0000 7f38c1a00640 1 --2- 172.21.15.43:0/2596160939 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f38bc1126d0 0x7f38bc114ac0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f38b4009ca0 tx=0x7f38b402c190 comp rx=0 tx=0).stop 2024-03-28T22:42:41.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.421+0000 7f38c1a00640 1 -- 172.21.15.43:0/2596160939 shutdown_connections 2024-03-28T22:42:41.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.421+0000 7f38c1a00640 1 --2- 172.21.15.43:0/2596160939 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f38bc1126d0 0x7f38bc114ac0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.421+0000 7f38c1a00640 1 --2- 172.21.15.43:0/2596160939 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f38bc074410 0x7f38bc112190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.421+0000 7f38c1a00640 1 --2- 172.21.15.43:0/2596160939 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38bc073af0 0x7f38bc073ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.421+0000 7f38c1a00640 1 -- 172.21.15.43:0/2596160939 >> 172.21.15.43:0/2596160939 conn(0x7f38bc0fd1b0 msgr2=0x7f38bc0ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:41.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.421+0000 7f38c1a00640 1 -- 172.21.15.43:0/2596160939 shutdown_connections 2024-03-28T22:42:41.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.421+0000 7f38c1a00640 1 -- 172.21.15.43:0/2596160939 wait complete. 2024-03-28T22:42:41.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.422+0000 7f38c1a00640 1 Processor -- start 2024-03-28T22:42:41.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.422+0000 7f38c1a00640 1 -- start start 2024-03-28T22:42:41.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.422+0000 7f38c1a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f38bc073af0 0x7f38bc19e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.422+0000 7f38c1a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f38bc074410 0x7f38bc19eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.422+0000 7f38c1a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38bc1126d0 0x7f38bc1a2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.422+0000 7f38c1a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f38bc118070 con 0x7f38bc1126d0 2024-03-28T22:42:41.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.422+0000 7f38c1a00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f38bc117ef0 con 0x7f38bc074410 2024-03-28T22:42:41.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.422+0000 7f38c1a00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f38bc1181f0 con 0x7f38bc073af0 2024-03-28T22:42:41.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.423+0000 7f38ba000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f38bc074410 0x7f38bc19eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.423+0000 7f38bb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38bc1126d0 0x7f38bc1a2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.423+0000 7f38baa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f38bc073af0 0x7f38bc19e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.423+0000 7f38ba000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f38bc074410 0x7f38bc19eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47772/0 (socket says 172.21.15.43:47772) 2024-03-28T22:42:41.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.423+0000 7f38ba000640 1 -- 172.21.15.43:0/2635395762 learned_addr learned my addr 172.21.15.43:0/2635395762 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:41.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.423+0000 7f38ba000640 1 -- 172.21.15.43:0/2635395762 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f38bc073af0 msgr2=0x7f38bc19e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:41.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.423+0000 7f38ba000640 1 --2- 172.21.15.43:0/2635395762 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f38bc073af0 0x7f38bc19e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.423+0000 7f38ba000640 1 -- 172.21.15.43:0/2635395762 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38bc1126d0 msgr2=0x7f38bc1a2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:41.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.423+0000 7f38ba000640 1 --2- 172.21.15.43:0/2635395762 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38bc1126d0 0x7f38bc1a2ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.423+0000 7f38ba000640 1 -- 172.21.15.43:0/2635395762 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38bc1a3620 con 0x7f38bc074410 2024-03-28T22:42:41.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.423+0000 7f38ba000640 1 --2- 172.21.15.43:0/2635395762 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f38bc074410 0x7f38bc19eb70 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f38a800e9f0 tx=0x7f38a800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:41.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.423+0000 7f38b3400640 1 -- 172.21.15.43:0/2635395762 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a800cd80 con 0x7f38bc074410 2024-03-28T22:42:41.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.424+0000 7f38b3400640 1 -- 172.21.15.43:0/2635395762 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38a8010910 con 0x7f38bc074410 2024-03-28T22:42:41.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.424+0000 7f38b3400640 1 -- 172.21.15.43:0/2635395762 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a8018990 con 0x7f38bc074410 2024-03-28T22:42:41.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.424+0000 7f38c1a00640 1 -- 172.21.15.43:0/2635395762 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38bc1a3910 con 0x7f38bc074410 2024-03-28T22:42:41.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.424+0000 7f38c1a00640 1 -- 172.21.15.43:0/2635395762 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38bc1ab1a0 con 0x7f38bc074410 2024-03-28T22:42:41.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.427+0000 7f38c4630740 1 Processor -- start 2024-03-28T22:42:41.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.427+0000 7f38c4630740 1 -- start start 2024-03-28T22:42:41.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.427+0000 7f38c4630740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622f9d6c380 0x5622f9d6c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.427+0000 7f38c4630740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5622f9d6cca0 0x5622f9d75340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.427+0000 7f38c4630740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5622f9d75880 0x5622f9d77c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.427+0000 7f38c4630740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5622f9cd0050 con 0x5622f9d6c380 2024-03-28T22:42:41.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.427+0000 7f38c4630740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5622f9c78ce0 con 0x5622f9d75880 2024-03-28T22:42:41.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.427+0000 7f38c4630740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5622f9ccc0f0 con 0x5622f9d6cca0 2024-03-28T22:42:41.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.427+0000 7f38b3400640 1 -- 172.21.15.43:0/2635395762 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f38a8010ac0 con 0x7f38bc074410 2024-03-28T22:42:41.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.427+0000 7f38baa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5622f9d6cca0 0x5622f9d75340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.427+0000 7f38bb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622f9d6c380 0x5622f9d6c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.427+0000 7f38baa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5622f9d6cca0 0x5622f9d75340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:56844/0 (socket says 172.21.15.43:56844) 2024-03-28T22:42:41.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.427+0000 7f38baa00640 1 -- 172.21.15.43:0/3277725706 learned_addr learned my addr 172.21.15.43:0/3277725706 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:41.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.428+0000 7f38ba000640 1 --2- 172.21.15.43:0/3277725706 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5622f9d75880 0x5622f9d77c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.428+0000 7f38baa00640 1 -- 172.21.15.43:0/3277725706 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5622f9d75880 msgr2=0x5622f9d77c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:41.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.428+0000 7f38baa00640 1 --2- 172.21.15.43:0/3277725706 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5622f9d75880 0x5622f9d77c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.428+0000 7f38baa00640 1 -- 172.21.15.43:0/3277725706 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622f9d6c380 msgr2=0x5622f9d6c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:41.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.428+0000 7f38baa00640 1 --2- 172.21.15.43:0/3277725706 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622f9d6c380 0x5622f9d6c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.428+0000 7f38baa00640 1 -- 172.21.15.43:0/3277725706 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622f99e43f0 con 0x5622f9d6cca0 2024-03-28T22:42:41.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.428+0000 7f38baa00640 1 --2- 172.21.15.43:0/3277725706 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5622f9d6cca0 0x5622f9d75340 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f38ac009cf0 tx=0x7f38ac011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77420181bfdbb827 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:41.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.428+0000 7f38a7e00640 1 -- 172.21.15.43:0/3277725706 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38ac004230 con 0x5622f9d6cca0 2024-03-28T22:42:41.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.428+0000 7f38a7e00640 1 -- 172.21.15.43:0/3277725706 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38ac0043d0 con 0x5622f9d6cca0 2024-03-28T22:42:41.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.428+0000 7f38a7e00640 1 -- 172.21.15.43:0/3277725706 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38ac019960 con 0x5622f9d6cca0 2024-03-28T22:42:41.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.429+0000 7f38b3400640 1 --2- 172.21.15.43:0/2635395762 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f38a00a1ef0 0x7f38a00a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.429+0000 7f38b3400640 1 -- 172.21.15.43:0/2635395762 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f38a80ce840 con 0x7f38bc074410 2024-03-28T22:42:41.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.429+0000 7f38bb400640 1 --2- 172.21.15.43:0/2635395762 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f38a00a1ef0 0x7f38a00a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.429+0000 7f38c4630740 1 -- 172.21.15.43:0/3277725706 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5622f9d6cca0 msgr2=0x5622f9d75340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:41.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.429+0000 7f38c4630740 1 --2- 172.21.15.43:0/3277725706 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5622f9d6cca0 0x5622f9d75340 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f38ac009cf0 tx=0x7f38ac011810 comp rx=0 tx=0).stop 2024-03-28T22:42:41.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.429+0000 7f38c4630740 1 -- 172.21.15.43:0/3277725706 shutdown_connections 2024-03-28T22:42:41.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.429+0000 7f38c4630740 1 --2- 172.21.15.43:0/3277725706 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5622f9d75880 0x5622f9d77c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.429+0000 7f38c4630740 1 --2- 172.21.15.43:0/3277725706 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5622f9d6cca0 0x5622f9d75340 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.429+0000 7f38c4630740 1 --2- 172.21.15.43:0/3277725706 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622f9d6c380 0x5622f9d6c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.429+0000 7f38c4630740 1 -- 172.21.15.43:0/3277725706 >> 172.21.15.43:0/3277725706 conn(0x5622f9d66ab0 msgr2=0x5622f9d66e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:41.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.429+0000 7f38c4630740 1 -- 172.21.15.43:0/3277725706 shutdown_connections 2024-03-28T22:42:41.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.430+0000 7f38c4630740 1 -- 172.21.15.43:0/3277725706 wait complete. 2024-03-28T22:42:41.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.430+0000 7f38bb400640 1 --2- 172.21.15.43:0/2635395762 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f38a00a1ef0 0x7f38a00a43b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f38b402c6d0 tx=0x7f38b4005d80 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:41.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.430+0000 7f38c4630740 1 -- start start 2024-03-28T22:42:41.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.430+0000 7f38c4630740 1 Processor -- start 2024-03-28T22:42:41.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.431+0000 7f38c4630740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622f9d6c380 0x5622f9d728d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.431+0000 7f38c4630740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5622f9d6cca0 0x5622f9d72e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.431+0000 7f38c4630740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5622f9d75880 0x5622f9d73350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.431+0000 7f38c4630740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5622f9d78670 con 0x5622f9d6c380 2024-03-28T22:42:41.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.431+0000 7f38c4630740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5622f9c6e2f0 con 0x5622f9d75880 2024-03-28T22:42:41.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.431+0000 7f38c4630740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5622f9d787f0 con 0x5622f9d6cca0 2024-03-28T22:42:41.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.431+0000 7f38c4630740 10 client.0 _ll_register_callbacks cb 0x5622f99e3be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:41.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.431+0000 7f38ba000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5622f9d6cca0 0x5622f9d72e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.431+0000 7f38baa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622f9d6c380 0x5622f9d728d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.431+0000 7f38ba000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5622f9d6cca0 0x5622f9d72e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:56858/0 (socket says 172.21.15.43:56858) 2024-03-28T22:42:41.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.431+0000 7f38ba000640 1 -- 172.21.15.43:0/1518157393 learned_addr learned my addr 172.21.15.43:0/1518157393 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:41.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.431+0000 7f38bb400640 1 --2- 172.21.15.43:0/1518157393 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5622f9d75880 0x5622f9d73350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.431+0000 7f38ba000640 1 -- 172.21.15.43:0/1518157393 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5622f9d75880 msgr2=0x5622f9d73350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:41.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.431+0000 7f38ba000640 1 --2- 172.21.15.43:0/1518157393 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5622f9d75880 0x5622f9d73350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.431+0000 7f38ba000640 1 -- 172.21.15.43:0/1518157393 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622f9d6c380 msgr2=0x5622f9d728d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:41.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.431+0000 7f38ba000640 1 --2- 172.21.15.43:0/1518157393 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622f9d6c380 0x5622f9d728d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.431+0000 7f38ba000640 1 -- 172.21.15.43:0/1518157393 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622f9aae7d0 con 0x5622f9d6cca0 2024-03-28T22:42:41.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.431+0000 7f38baa00640 1 --2- 172.21.15.43:0/1518157393 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622f9d6c380 0x5622f9d728d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:41.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.432+0000 7f38ba000640 1 --2- 172.21.15.43:0/1518157393 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5622f9d6cca0 0x5622f9d72e10 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f38a800e9c0 tx=0x7f38a809ada0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a1ba945739f4416 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:41.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.432+0000 7f38a4c00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:42:41.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.432+0000 7f38a4c00640 1 -- 172.21.15.43:0/1518157393 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a80cc020 con 0x5622f9d6cca0 2024-03-28T22:42:41.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.432+0000 7f38a4c00640 1 -- 172.21.15.43:0/1518157393 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f38a80ce5c0 con 0x5622f9d6cca0 2024-03-28T22:42:41.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.432+0000 7f38c4630740 1 -- 172.21.15.43:0/1518157393 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38ac0043d0 con 0x5622f9d6cca0 2024-03-28T22:42:41.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.432+0000 7f38a4c00640 1 -- 172.21.15.43:0/1518157393 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f38a809ef80 con 0x5622f9d6cca0 2024-03-28T22:42:41.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.432+0000 7f38c4630740 10 client.27622 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:41.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.432+0000 7f38c4630740 1 -- 172.21.15.43:0/1518157393 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5622f9d849c0 con 0x5622f9d6cca0 2024-03-28T22:42:41.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.432+0000 7f38a4c00640 1 -- 172.21.15.43:0/1518157393 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38ac0043d0 con 0x5622f9d6cca0 2024-03-28T22:42:41.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.432+0000 7f38a4c00640 1 -- 172.21.15.43:0/1518157393 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f38a809a450 con 0x5622f9d6cca0 2024-03-28T22:42:41.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.432+0000 7f38c4630740 10 client.27622 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:41.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.432+0000 7f38c4630740 10 client.27622 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:42:41.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.432+0000 7f38c4630740 20 client.27622 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:41.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.432+0000 7f38c4630740 1 --2- 172.21.15.43:0/1518157393 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5622f9d85dd0 0x5622f9d88240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.432+0000 7f38c4630740 4 client.27622 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:42:41.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.433+0000 7f38c4630740 1 -- 172.21.15.43:0/1518157393 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5622f9c79680 con 0x5622f9d85dd0 2024-03-28T22:42:41.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.433+0000 7f38baa00640 1 --2- 172.21.15.43:0/1518157393 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5622f9d85dd0 0x5622f9d88240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.433+0000 7f38baa00640 1 --2- 172.21.15.43:0/1518157393 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5622f9d85dd0 0x5622f9d88240 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9507685dde8ea995 server_cookie=d031b2719cb10f5a in_seq=0 out_seq=0 2024-03-28T22:42:41.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.433+0000 7f38a4c00640 10 client.27622 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:42:41.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.434+0000 7f38a4c00640 1 -- 172.21.15.43:0/1518157393 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5622f9c79680 con 0x5622f9d85dd0 2024-03-28T22:42:41.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.434+0000 7f38a4c00640 10 client.27622 handle_command_reply: tid=0 2024-03-28T22:42:41.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.434+0000 7f38c4630740 1 client.27622 shutdown 2024-03-28T22:42:41.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.434+0000 7f38c4630740 1 -- 172.21.15.43:0/1518157393 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5622f9d6cca0 msgr2=0x5622f9d72e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:41.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.434+0000 7f38c4630740 1 --2- 172.21.15.43:0/1518157393 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5622f9d6cca0 0x5622f9d72e10 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f38a800e9c0 tx=0x7f38a809ada0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.434+0000 7f38c4630740 1 -- 172.21.15.43:0/1518157393 shutdown_connections 2024-03-28T22:42:41.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.434+0000 7f38c4630740 1 --2- 172.21.15.43:0/1518157393 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5622f9d85dd0 0x5622f9d88240 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.434+0000 7f38c4630740 1 --2- 172.21.15.43:0/1518157393 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5622f9d75880 0x5622f9d73350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.434+0000 7f38c4630740 1 --2- 172.21.15.43:0/1518157393 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5622f9d6cca0 0x5622f9d72e10 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.434+0000 7f38c4630740 1 --2- 172.21.15.43:0/1518157393 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622f9d6c380 0x5622f9d728d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.434+0000 7f38a4c00640 0 client.27622 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:42:41.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.434+0000 7f38c4630740 1 -- 172.21.15.43:0/1518157393 >> 172.21.15.43:0/1518157393 conn(0x5622f9d66ab0 msgr2=0x5622f9d68800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:41.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.435+0000 7f38c4630740 1 -- 172.21.15.43:0/1518157393 shutdown_connections 2024-03-28T22:42:41.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.435+0000 7f38c4630740 1 -- 172.21.15.43:0/1518157393 wait complete. 2024-03-28T22:42:41.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.435+0000 7f38c4630740 20 client.27622 trim_cache size 0 max 16384 2024-03-28T22:42:41.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.455+0000 7f38c4630740 1 Processor -- start 2024-03-28T22:42:41.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.455+0000 7f38c4630740 1 -- start start 2024-03-28T22:42:41.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.455+0000 7f38c4630740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5622f9d6c380 0x5622f9d75000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.455+0000 7f38c4630740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622f9d6cca0 0x5622f9d81440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.455+0000 7f38c4630740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5622f9d75880 0x5622f9d8db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.455+0000 7f38c4630740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f38bc1b52f0 con 0x5622f9d6cca0 2024-03-28T22:42:41.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.455+0000 7f38c4630740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f38bc1b6e30 con 0x5622f9d75880 2024-03-28T22:42:41.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.455+0000 7f38c4630740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f38bc1b59c0 con 0x5622f9d6c380 2024-03-28T22:42:41.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.456+0000 7f38baa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5622f9d6c380 0x5622f9d75000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.456+0000 7f38bb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5622f9d75880 0x5622f9d8db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.456+0000 7f38baa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5622f9d6c380 0x5622f9d75000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:56868/0 (socket says 172.21.15.43:56868) 2024-03-28T22:42:41.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.456+0000 7f38baa00640 1 -- 172.21.15.43:0/3225653963 learned_addr learned my addr 172.21.15.43:0/3225653963 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:41.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.456+0000 7f38baa00640 1 -- 172.21.15.43:0/3225653963 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5622f9d75880 msgr2=0x5622f9d8db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:41.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.456+0000 7f38baa00640 1 --2- 172.21.15.43:0/3225653963 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5622f9d75880 0x5622f9d8db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.456+0000 7f38baa00640 1 -- 172.21.15.43:0/3225653963 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622f9d6cca0 msgr2=0x5622f9d81440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:41.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.456+0000 7f38ba000640 1 --2- 172.21.15.43:0/3225653963 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622f9d6cca0 0x5622f9d81440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.457+0000 7f38baa00640 1 --2- 172.21.15.43:0/3225653963 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622f9d6cca0 0x5622f9d81440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.457+0000 7f38baa00640 1 -- 172.21.15.43:0/3225653963 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622f9d77c70 con 0x5622f9d6c380 2024-03-28T22:42:41.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.457+0000 7f38ba000640 1 --2- 172.21.15.43:0/3225653963 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622f9d6cca0 0x5622f9d81440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:42:41.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.457+0000 7f38baa00640 1 --2- 172.21.15.43:0/3225653963 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5622f9d6c380 0x5622f9d75000 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f38ac009cf0 tx=0x7f38ac005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9972d0c81c43d44 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:41.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.459+0000 7f38a7400640 1 -- 172.21.15.43:0/3225653963 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38ac004310 con 0x5622f9d6c380 2024-03-28T22:42:41.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.459+0000 7f38a7400640 1 -- 172.21.15.43:0/3225653963 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38ac0097a0 con 0x5622f9d6c380 2024-03-28T22:42:41.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.459+0000 7f38a7400640 1 -- 172.21.15.43:0/3225653963 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38ac0052a0 con 0x5622f9d6c380 2024-03-28T22:42:41.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.460+0000 7f38c4630740 1 -- 172.21.15.43:0/3225653963 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5622f9d6c380 msgr2=0x5622f9d75000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:41.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.460+0000 7f38c4630740 1 --2- 172.21.15.43:0/3225653963 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5622f9d6c380 0x5622f9d75000 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f38ac009cf0 tx=0x7f38ac005f90 comp rx=0 tx=0).stop 2024-03-28T22:42:41.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.460+0000 7f38c4630740 1 -- 172.21.15.43:0/3225653963 shutdown_connections 2024-03-28T22:42:41.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.460+0000 7f38c4630740 1 --2- 172.21.15.43:0/3225653963 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5622f9d75880 0x5622f9d8db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.460+0000 7f38c4630740 1 --2- 172.21.15.43:0/3225653963 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622f9d6cca0 0x5622f9d81440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.460+0000 7f38c4630740 1 --2- 172.21.15.43:0/3225653963 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5622f9d6c380 0x5622f9d75000 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.460+0000 7f38c4630740 1 -- 172.21.15.43:0/3225653963 >> 172.21.15.43:0/3225653963 conn(0x5622f9d66ab0 msgr2=0x5622f9d66e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:41.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.461+0000 7f38c4630740 1 -- 172.21.15.43:0/3225653963 shutdown_connections 2024-03-28T22:42:41.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.461+0000 7f38c4630740 1 -- 172.21.15.43:0/3225653963 wait complete. 2024-03-28T22:42:41.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.461+0000 7f38c4630740 1 -- start start 2024-03-28T22:42:41.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.462+0000 7f38c4630740 1 Processor -- start 2024-03-28T22:42:41.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.463+0000 7f38c4630740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622f9d6c380 0x5622f9d7dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.463+0000 7f38c4630740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5622f9d6cca0 0x5622f9d6f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.463+0000 7f38c4630740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5622f9d75880 0x5622f9d6f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.463+0000 7f38c4630740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f38bc1b6760 con 0x5622f9d6c380 2024-03-28T22:42:41.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.463+0000 7f38c4630740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f38bc1b6090 con 0x5622f9d6cca0 2024-03-28T22:42:41.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.463+0000 7f38c4630740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f38bc1b32d0 con 0x5622f9d75880 2024-03-28T22:42:41.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.464+0000 7f38ba000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5622f9d6cca0 0x5622f9d6f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.464+0000 7f38c4630740 10 client.0 _ll_register_callbacks cb 0x7f38a8009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:41.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.464+0000 7f38ba000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5622f9d6cca0 0x5622f9d6f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47816/0 (socket says 172.21.15.43:47816) 2024-03-28T22:42:41.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.464+0000 7f38ba000640 1 -- 172.21.15.43:0/910160001 learned_addr learned my addr 172.21.15.43:0/910160001 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:41.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.464+0000 7f38bb400640 1 --2- 172.21.15.43:0/910160001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5622f9d75880 0x5622f9d6f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.465+0000 7f38baa00640 1 --2- 172.21.15.43:0/910160001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622f9d6c380 0x5622f9d7dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.465+0000 7f38ba000640 1 -- 172.21.15.43:0/910160001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5622f9d75880 msgr2=0x5622f9d6f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:41.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.465+0000 7f38ba000640 1 --2- 172.21.15.43:0/910160001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5622f9d75880 0x5622f9d6f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.465+0000 7f38ba000640 1 -- 172.21.15.43:0/910160001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622f9d6c380 msgr2=0x5622f9d7dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:41.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.466+0000 7f38ba000640 1 --2- 172.21.15.43:0/910160001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622f9d6c380 0x5622f9d7dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.466+0000 7f38ba000640 1 -- 172.21.15.43:0/910160001 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622f9d6fd30 con 0x5622f9d6cca0 2024-03-28T22:42:41.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.466+0000 7f38baa00640 1 --2- 172.21.15.43:0/910160001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622f9d6c380 0x5622f9d7dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:41.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.466+0000 7f38ba000640 1 --2- 172.21.15.43:0/910160001 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5622f9d6cca0 0x5622f9d6f100 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f38a809abe0 tx=0x7f38a80ce5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66302746ae061ef1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:41.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.466+0000 7f38a4c00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:42:41.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.466+0000 7f38a4c00640 1 -- 172.21.15.43:0/910160001 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a80cc020 con 0x5622f9d6cca0 2024-03-28T22:42:41.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.467+0000 7f38a4c00640 1 -- 172.21.15.43:0/910160001 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f38a8018b30 con 0x5622f9d6cca0 2024-03-28T22:42:41.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.467+0000 7f38c4630740 1 -- 172.21.15.43:0/910160001 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38ac0097a0 con 0x5622f9d6cca0 2024-03-28T22:42:41.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.467+0000 7f38a4c00640 1 -- 172.21.15.43:0/910160001 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f38a8097180 con 0x5622f9d6cca0 2024-03-28T22:42:41.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.467+0000 7f38c4630740 10 client.27686 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:41.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.467+0000 7f38c4630740 1 -- 172.21.15.43:0/910160001 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5622f9dfcfe0 con 0x5622f9d6cca0 2024-03-28T22:42:41.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.469+0000 7f38a4c00640 1 -- 172.21.15.43:0/910160001 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38ac0097a0 con 0x5622f9d6cca0 2024-03-28T22:42:41.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.469+0000 7f38a4c00640 1 -- 172.21.15.43:0/910160001 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f38a80b3b60 con 0x5622f9d6cca0 2024-03-28T22:42:41.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.469+0000 7f38c4630740 10 client.27686 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:41.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.469+0000 7f38c4630740 10 client.27686 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:42:41.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.469+0000 7f38c4630740 20 client.27686 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:41.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.469+0000 7f38c4630740 1 --2- 172.21.15.43:0/910160001 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5622f9dfd690 0x5622f9dffab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.469+0000 7f38c4630740 4 client.27686 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:42:41.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.469+0000 7f38c4630740 1 -- 172.21.15.43:0/910160001 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5622f9cdd590 con 0x5622f9dfd690 2024-03-28T22:42:41.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.470+0000 7f38baa00640 1 --2- 172.21.15.43:0/910160001 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5622f9dfd690 0x5622f9dffab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.471+0000 7f38baa00640 1 --2- 172.21.15.43:0/910160001 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5622f9dfd690 0x5622f9dffab0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3b1e8816367799f2 server_cookie=916919f8d0ae4dad in_seq=0 out_seq=0 2024-03-28T22:42:41.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.471+0000 7f38a4c00640 10 client.27686 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:42:41.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.471+0000 7f38a4c00640 1 -- 172.21.15.43:0/910160001 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5622f9cdd590 con 0x5622f9dfd690 2024-03-28T22:42:41.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.471+0000 7f38a4c00640 10 client.27686 handle_command_reply: tid=0 2024-03-28T22:42:41.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.471+0000 7f38c4630740 1 client.27686 shutdown 2024-03-28T22:42:41.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.472+0000 7f38c4630740 1 -- 172.21.15.43:0/910160001 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5622f9d6cca0 msgr2=0x5622f9d6f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:41.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.472+0000 7f38c4630740 1 --2- 172.21.15.43:0/910160001 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5622f9d6cca0 0x5622f9d6f100 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f38a809abe0 tx=0x7f38a80ce5d0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.472+0000 7f38c4630740 1 -- 172.21.15.43:0/910160001 shutdown_connections 2024-03-28T22:42:41.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.472+0000 7f38c4630740 1 --2- 172.21.15.43:0/910160001 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5622f9dfd690 0x5622f9dffab0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.472+0000 7f38c4630740 1 --2- 172.21.15.43:0/910160001 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5622f9d6cca0 0x5622f9d6f100 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.472+0000 7f38c4630740 1 --2- 172.21.15.43:0/910160001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5622f9d75880 0x5622f9d6f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.472+0000 7f38c4630740 1 --2- 172.21.15.43:0/910160001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5622f9d6c380 0x5622f9d7dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.472+0000 7f38c4630740 1 -- 172.21.15.43:0/910160001 >> 172.21.15.43:0/910160001 conn(0x5622f9d66ab0 msgr2=0x5622f9d652e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:41.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.473+0000 7f38a4c00640 0 client.27686 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:42:41.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.473+0000 7f38c4630740 1 -- 172.21.15.43:0/910160001 shutdown_connections 2024-03-28T22:42:41.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.473+0000 7f38c4630740 1 -- 172.21.15.43:0/910160001 wait complete. 2024-03-28T22:42:41.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.473+0000 7f38c4630740 20 client.27686 trim_cache size 0 max 16384 2024-03-28T22:42:41.471 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:42:41.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.474+0000 7f38c1a00640 1 -- 172.21.15.43:0/2635395762 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f38a00a1ef0 msgr2=0x7f38a00a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:41.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.474+0000 7f38c1a00640 1 --2- 172.21.15.43:0/2635395762 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f38a00a1ef0 0x7f38a00a43b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f38b402c6d0 tx=0x7f38b4005d80 comp rx=0 tx=0).stop 2024-03-28T22:42:41.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.474+0000 7f38c1a00640 1 -- 172.21.15.43:0/2635395762 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f38bc074410 msgr2=0x7f38bc19eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:41.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.474+0000 7f38c1a00640 1 --2- 172.21.15.43:0/2635395762 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f38bc074410 0x7f38bc19eb70 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f38a800e9f0 tx=0x7f38a800eeb0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.475+0000 7f38c1a00640 1 -- 172.21.15.43:0/2635395762 shutdown_connections 2024-03-28T22:42:41.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.475+0000 7f38c1a00640 1 --2- 172.21.15.43:0/2635395762 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f38a00a1ef0 0x7f38a00a43b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.475+0000 7f38c1a00640 1 --2- 172.21.15.43:0/2635395762 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f38bc074410 0x7f38bc19eb70 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.475+0000 7f38c1a00640 1 --2- 172.21.15.43:0/2635395762 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f38bc1126d0 0x7f38bc1a2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.475+0000 7f38c1a00640 1 --2- 172.21.15.43:0/2635395762 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f38bc073af0 0x7f38bc19e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.475+0000 7f38c1a00640 1 -- 172.21.15.43:0/2635395762 >> 172.21.15.43:0/2635395762 conn(0x7f38bc0fd1b0 msgr2=0x7f38bc0feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:41.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.475+0000 7f38c1a00640 1 -- 172.21.15.43:0/2635395762 shutdown_connections 2024-03-28T22:42:41.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.475+0000 7f38c1a00640 1 -- 172.21.15.43:0/2635395762 wait complete. 2024-03-28T22:42:41.488 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:42:41.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.683+0000 7f26f0c00640 1 -- 172.21.15.43:0/2868574150 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f26ec107690 msgr2=0x7f26ec107a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:41.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.683+0000 7f26f0c00640 1 --2- 172.21.15.43:0/2868574150 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f26ec107690 0x7f26ec107a70 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f26e4009cd0 tx=0x7f26e402c0f0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.684+0000 7f26f0c00640 1 -- 172.21.15.43:0/2868574150 shutdown_connections 2024-03-28T22:42:41.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.684+0000 7f26f0c00640 1 --2- 172.21.15.43:0/2868574150 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f26ec101f90 0x7f26ec10e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.684+0000 7f26f0c00640 1 --2- 172.21.15.43:0/2868574150 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26ec1015d0 0x7f26ec101a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.684+0000 7f26f0c00640 1 --2- 172.21.15.43:0/2868574150 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f26ec107690 0x7f26ec107a70 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.684+0000 7f26f0c00640 1 -- 172.21.15.43:0/2868574150 >> 172.21.15.43:0/2868574150 conn(0x7f26ec0fd0c0 msgr2=0x7f26ec0ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:41.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.684+0000 7f26f0c00640 1 -- 172.21.15.43:0/2868574150 shutdown_connections 2024-03-28T22:42:41.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.684+0000 7f26f0c00640 1 -- 172.21.15.43:0/2868574150 wait complete. 2024-03-28T22:42:41.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.685+0000 7f26f0c00640 1 Processor -- start 2024-03-28T22:42:41.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.685+0000 7f26f0c00640 1 -- start start 2024-03-28T22:42:41.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.685+0000 7f26f0c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26ec1015d0 0x7f26ec19a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.685+0000 7f26f0c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f26ec101f90 0x7f26ec19a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.685+0000 7f26f0c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f26ec107690 0x7f26ec19e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.685+0000 7f26f0c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f26ec1119d0 con 0x7f26ec1015d0 2024-03-28T22:42:41.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.685+0000 7f26f0c00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f26ec111850 con 0x7f26ec107690 2024-03-28T22:42:41.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.685+0000 7f26f0c00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f26ec111b50 con 0x7f26ec101f90 2024-03-28T22:42:41.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.685+0000 7f26eb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26ec1015d0 0x7f26ec19a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.685+0000 7f26ebe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f26ec107690 0x7f26ec19e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.685+0000 7f26eaa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f26ec101f90 0x7f26ec19a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.686+0000 7f26eaa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f26ec101f90 0x7f26ec19a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:56896/0 (socket says 172.21.15.43:56896) 2024-03-28T22:42:41.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.686+0000 7f26eaa00640 1 -- 172.21.15.43:0/2611239108 learned_addr learned my addr 172.21.15.43:0/2611239108 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:41.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.686+0000 7f26ebe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f26ec107690 0x7f26ec19e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47836/0 (socket says 172.21.15.43:47836) 2024-03-28T22:42:41.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.686+0000 7f26eaa00640 1 -- 172.21.15.43:0/2611239108 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f26ec107690 msgr2=0x7f26ec19e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:41.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.686+0000 7f26eaa00640 1 --2- 172.21.15.43:0/2611239108 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f26ec107690 0x7f26ec19e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.686+0000 7f26eaa00640 1 -- 172.21.15.43:0/2611239108 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26ec1015d0 msgr2=0x7f26ec19a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:41.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.686+0000 7f26eaa00640 1 --2- 172.21.15.43:0/2611239108 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26ec1015d0 0x7f26ec19a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.686+0000 7f26eaa00640 1 -- 172.21.15.43:0/2611239108 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26ec19f030 con 0x7f26ec101f90 2024-03-28T22:42:41.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.686+0000 7f26eaa00640 1 --2- 172.21.15.43:0/2611239108 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f26ec101f90 0x7f26ec19a580 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f26d800d950 tx=0x7f26d800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:41.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.686+0000 7f26e3e00640 1 -- 172.21.15.43:0/2611239108 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d8017030 con 0x7f26ec101f90 2024-03-28T22:42:41.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.686+0000 7f26f0c00640 1 -- 172.21.15.43:0/2611239108 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26ec19f320 con 0x7f26ec101f90 2024-03-28T22:42:41.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.686+0000 7f26e3e00640 1 -- 172.21.15.43:0/2611239108 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26d8004020 con 0x7f26ec101f90 2024-03-28T22:42:41.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.686+0000 7f26e3e00640 1 -- 172.21.15.43:0/2611239108 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d80047a0 con 0x7f26ec101f90 2024-03-28T22:42:41.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.687+0000 7f26f0c00640 1 -- 172.21.15.43:0/2611239108 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26ec1a6bb0 con 0x7f26ec101f90 2024-03-28T22:42:41.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.690+0000 7f26f4f88740 1 Processor -- start 2024-03-28T22:42:41.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.690+0000 7f26f4f88740 1 -- start start 2024-03-28T22:42:41.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.690+0000 7f26e3e00640 1 -- 172.21.15.43:0/2611239108 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f26d80041d0 con 0x7f26ec101f90 2024-03-28T22:42:41.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.690+0000 7f26f4f88740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e132843550 0x55e132843930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.690+0000 7f26f4f88740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e132843e70 0x55e13284c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.690+0000 7f26f4f88740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e13284ca50 0x55e13284ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.690+0000 7f26f4f88740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e1327a7050 con 0x55e132843e70 2024-03-28T22:42:41.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.690+0000 7f26f4f88740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55e13274fce0 con 0x55e132843550 2024-03-28T22:42:41.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.690+0000 7f26f4f88740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55e1327a30f0 con 0x55e13284ca50 2024-03-28T22:42:41.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.691+0000 7f26ebe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e132843550 0x55e132843930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.691+0000 7f26ebe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e132843550 0x55e132843930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47844/0 (socket says 172.21.15.43:47844) 2024-03-28T22:42:41.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.691+0000 7f26ebe00640 1 -- 172.21.15.43:0/717718119 learned_addr learned my addr 172.21.15.43:0/717718119 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:41.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.691+0000 7f26eb400640 1 --2- 172.21.15.43:0/717718119 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e132843e70 0x55e13284c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.691+0000 7f26eaa00640 1 --2- 172.21.15.43:0/717718119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e13284ca50 0x55e13284ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.691+0000 7f26ebe00640 1 -- 172.21.15.43:0/717718119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e13284ca50 msgr2=0x55e13284ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:41.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.691+0000 7f26ebe00640 1 --2- 172.21.15.43:0/717718119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e13284ca50 0x55e13284ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.691+0000 7f26ebe00640 1 -- 172.21.15.43:0/717718119 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e132843e70 msgr2=0x55e13284c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:41.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.691+0000 7f26ebe00640 1 --2- 172.21.15.43:0/717718119 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e132843e70 0x55e13284c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.691+0000 7f26ebe00640 1 -- 172.21.15.43:0/717718119 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1324bb3f0 con 0x55e132843550 2024-03-28T22:42:41.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.691+0000 7f26eaa00640 1 --2- 172.21.15.43:0/717718119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e13284ca50 0x55e13284ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:42:41.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.691+0000 7f26ebe00640 1 --2- 172.21.15.43:0/717718119 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e132843550 0x55e132843930 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f26dc009c20 tx=0x7f26dc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd93a49ff9eed99c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:41.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.692+0000 7f26d7e00640 1 -- 172.21.15.43:0/717718119 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26dc01e070 con 0x55e132843550 2024-03-28T22:42:41.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.692+0000 7f26d7e00640 1 -- 172.21.15.43:0/717718119 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26dc019b10 con 0x55e132843550 2024-03-28T22:42:41.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.692+0000 7f26d7e00640 1 -- 172.21.15.43:0/717718119 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26dc00eae0 con 0x55e132843550 2024-03-28T22:42:41.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.692+0000 7f26e3e00640 1 --2- 172.21.15.43:0/2611239108 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f26d00a1ef0 0x7f26d00a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.692+0000 7f26e3e00640 1 -- 172.21.15.43:0/2611239108 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f26d80cdf70 con 0x7f26ec101f90 2024-03-28T22:42:41.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.692+0000 7f26f4f88740 1 -- 172.21.15.43:0/717718119 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e132843550 msgr2=0x55e132843930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:41.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.692+0000 7f26f4f88740 1 --2- 172.21.15.43:0/717718119 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e132843550 0x55e132843930 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f26dc009c20 tx=0x7f26dc011810 comp rx=0 tx=0).stop 2024-03-28T22:42:41.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.692+0000 7f26eb400640 1 --2- 172.21.15.43:0/2611239108 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f26d00a1ef0 0x7f26d00a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.692+0000 7f26f4f88740 1 -- 172.21.15.43:0/717718119 shutdown_connections 2024-03-28T22:42:41.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.692+0000 7f26f4f88740 1 --2- 172.21.15.43:0/717718119 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e13284ca50 0x55e13284ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.692+0000 7f26f4f88740 1 --2- 172.21.15.43:0/717718119 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e132843e70 0x55e13284c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.693+0000 7f26f4f88740 1 --2- 172.21.15.43:0/717718119 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e132843550 0x55e132843930 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.693+0000 7f26f4f88740 1 -- 172.21.15.43:0/717718119 >> 172.21.15.43:0/717718119 conn(0x55e13283dc80 msgr2=0x55e13283e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:41.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.693+0000 7f26f4f88740 1 -- 172.21.15.43:0/717718119 shutdown_connections 2024-03-28T22:42:41.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.693+0000 7f26f4f88740 1 -- 172.21.15.43:0/717718119 wait complete. 2024-03-28T22:42:41.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.693+0000 7f26eb400640 1 --2- 172.21.15.43:0/2611239108 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f26d00a1ef0 0x7f26d00a43b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f26e4009ca0 tx=0x7f26e40093a0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:41.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.693+0000 7f26f4f88740 1 -- start start 2024-03-28T22:42:41.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.693+0000 7f26f4f88740 1 Processor -- start 2024-03-28T22:42:41.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.694+0000 7f26f4f88740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e132843550 0x55e132849b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.694+0000 7f26f4f88740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e132843e70 0x55e13284a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.694+0000 7f26f4f88740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e13284ca50 0x55e13284a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.694+0000 7f26f4f88740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e13284f840 con 0x55e132843550 2024-03-28T22:42:41.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.694+0000 7f26f4f88740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55e1327452f0 con 0x55e13284ca50 2024-03-28T22:42:41.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.694+0000 7f26f4f88740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55e13284f9c0 con 0x55e132843e70 2024-03-28T22:42:41.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.694+0000 7f26f4f88740 10 client.0 _ll_register_callbacks cb 0x55e1324babe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:41.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.694+0000 7f26eaa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e132843e70 0x55e13284a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.694+0000 7f26ebe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e13284ca50 0x55e13284a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.694+0000 7f26eaa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e132843e70 0x55e13284a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:56906/0 (socket says 172.21.15.43:56906) 2024-03-28T22:42:41.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.694+0000 7f26eaa00640 1 -- 172.21.15.43:0/39892236 learned_addr learned my addr 172.21.15.43:0/39892236 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:41.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.694+0000 7f26eb400640 1 --2- 172.21.15.43:0/39892236 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e132843550 0x55e132849b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.695+0000 7f26ebe00640 1 -- 172.21.15.43:0/39892236 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e132843e70 msgr2=0x55e13284a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:41.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.695+0000 7f26ebe00640 1 --2- 172.21.15.43:0/39892236 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e132843e70 0x55e13284a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.695+0000 7f26ebe00640 1 -- 172.21.15.43:0/39892236 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e132843550 msgr2=0x55e132849b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:41.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.695+0000 7f26ebe00640 1 --2- 172.21.15.43:0/39892236 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e132843550 0x55e132849b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.695+0000 7f26ebe00640 1 -- 172.21.15.43:0/39892236 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1325857d0 con 0x55e13284ca50 2024-03-28T22:42:41.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.695+0000 7f26eb400640 1 --2- 172.21.15.43:0/39892236 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e132843550 0x55e132849b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:42:41.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.695+0000 7f26ebe00640 1 --2- 172.21.15.43:0/39892236 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e13284ca50 0x55e13284a5c0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f26dc005890 tx=0x7f26dc00e440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0d5c8c04389efd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:41.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.695+0000 7f26d4c00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:42:41.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.695+0000 7f26d4c00640 1 -- 172.21.15.43:0/39892236 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26dc027070 con 0x55e13284ca50 2024-03-28T22:42:41.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.695+0000 7f26d4c00640 1 -- 172.21.15.43:0/39892236 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f26dc019840 con 0x55e13284ca50 2024-03-28T22:42:41.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.695+0000 7f26f4f88740 1 -- 172.21.15.43:0/39892236 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26dc019b10 con 0x55e13284ca50 2024-03-28T22:42:41.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.696+0000 7f26d4c00640 1 -- 172.21.15.43:0/39892236 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f26dc032050 con 0x55e13284ca50 2024-03-28T22:42:41.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.696+0000 7f26f4f88740 10 client.27698 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:41.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.696+0000 7f26f4f88740 1 -- 172.21.15.43:0/39892236 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e13285bba0 con 0x55e13284ca50 2024-03-28T22:42:41.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.696+0000 7f26d4c00640 1 -- 172.21.15.43:0/39892236 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26dc019b10 con 0x55e13284ca50 2024-03-28T22:42:41.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.696+0000 7f26d4c00640 1 -- 172.21.15.43:0/39892236 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f26dc00e7e0 con 0x55e13284ca50 2024-03-28T22:42:41.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.696+0000 7f26f4f88740 10 client.27698 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:41.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.696+0000 7f26f4f88740 10 client.27698 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:42:41.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.696+0000 7f26f4f88740 20 client.27698 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:41.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.696+0000 7f26f4f88740 1 --2- 172.21.15.43:0/39892236 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55e13285cf50 0x55e13285f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.696+0000 7f26f4f88740 4 client.27698 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:42:41.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.696+0000 7f26f4f88740 1 -- 172.21.15.43:0/39892236 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e132750680 con 0x55e13285cf50 2024-03-28T22:42:41.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.697+0000 7f26eb400640 1 --2- 172.21.15.43:0/39892236 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55e13285cf50 0x55e13285f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.697+0000 7f26eb400640 1 --2- 172.21.15.43:0/39892236 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55e13285cf50 0x55e13285f340 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5d9081f47e4714bf server_cookie=25822f5bf5de6e5e in_seq=0 out_seq=0 2024-03-28T22:42:41.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.697+0000 7f26d4c00640 10 client.27698 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:42:41.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.698+0000 7f26d4c00640 1 -- 172.21.15.43:0/39892236 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e132750680 con 0x55e13285cf50 2024-03-28T22:42:41.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.698+0000 7f26d4c00640 10 client.27698 handle_command_reply: tid=0 2024-03-28T22:42:41.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.698+0000 7f26f4f88740 1 client.27698 shutdown 2024-03-28T22:42:41.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.698+0000 7f26f4f88740 1 -- 172.21.15.43:0/39892236 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e13284ca50 msgr2=0x55e13284a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:41.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.699+0000 7f26f4f88740 1 --2- 172.21.15.43:0/39892236 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e13284ca50 0x55e13284a5c0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f26dc005890 tx=0x7f26dc00e440 comp rx=0 tx=0).stop 2024-03-28T22:42:41.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.699+0000 7f26f4f88740 1 -- 172.21.15.43:0/39892236 shutdown_connections 2024-03-28T22:42:41.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.699+0000 7f26f4f88740 1 --2- 172.21.15.43:0/39892236 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55e13285cf50 0x55e13285f340 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.699+0000 7f26f4f88740 1 --2- 172.21.15.43:0/39892236 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e13284ca50 0x55e13284a5c0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.699+0000 7f26f4f88740 1 --2- 172.21.15.43:0/39892236 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e132843e70 0x55e13284a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.699+0000 7f26d4c00640 0 client.27698 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:42:41.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.699+0000 7f26f4f88740 1 --2- 172.21.15.43:0/39892236 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e132843550 0x55e132849b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.699+0000 7f26f4f88740 1 -- 172.21.15.43:0/39892236 >> 172.21.15.43:0/39892236 conn(0x55e13283dc80 msgr2=0x55e13283f9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:41.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.699+0000 7f26f4f88740 1 -- 172.21.15.43:0/39892236 shutdown_connections 2024-03-28T22:42:41.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.699+0000 7f26f4f88740 1 -- 172.21.15.43:0/39892236 wait complete. 2024-03-28T22:42:41.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.699+0000 7f26f4f88740 20 client.27698 trim_cache size 0 max 16384 2024-03-28T22:42:41.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.716+0000 7f26f4f88740 1 Processor -- start 2024-03-28T22:42:41.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.716+0000 7f26f4f88740 1 -- start start 2024-03-28T22:42:41.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.716+0000 7f26f4f88740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e132843550 0x55e13284c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.716+0000 7f26f4f88740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e132843e70 0x55e132848db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.716+0000 7f26f4f88740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e13284ca50 0x55e1328585e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.716+0000 7f26f4f88740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f26ec1b0b90 con 0x55e13284ca50 2024-03-28T22:42:41.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.716+0000 7f26f4f88740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f26ec1b26d0 con 0x55e132843e70 2024-03-28T22:42:41.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.716+0000 7f26f4f88740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f26ec1b1260 con 0x55e132843550 2024-03-28T22:42:41.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.716+0000 7f26eb400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e132843550 0x55e13284c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.716+0000 7f26ebe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e13284ca50 0x55e1328585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.716+0000 7f26eb400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e132843550 0x55e13284c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:56920/0 (socket says 172.21.15.43:56920) 2024-03-28T22:42:41.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.716+0000 7f26eaa00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e132843e70 0x55e132848db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.716+0000 7f26eb400640 1 -- 172.21.15.43:0/3378011430 learned_addr learned my addr 172.21.15.43:0/3378011430 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:41.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.717+0000 7f26eb400640 1 -- 172.21.15.43:0/3378011430 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e132843e70 msgr2=0x55e132848db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:41.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.717+0000 7f26eb400640 1 --2- 172.21.15.43:0/3378011430 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e132843e70 0x55e132848db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.717+0000 7f26eb400640 1 -- 172.21.15.43:0/3378011430 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e13284ca50 msgr2=0x55e1328585e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:41.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.717+0000 7f26eb400640 1 --2- 172.21.15.43:0/3378011430 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e13284ca50 0x55e1328585e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.717+0000 7f26eb400640 1 -- 172.21.15.43:0/3378011430 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e132843ca0 con 0x55e132843550 2024-03-28T22:42:41.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.717+0000 7f26eb400640 1 --2- 172.21.15.43:0/3378011430 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e132843550 0x55e13284c270 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f26e4009a60 tx=0x7f26e4041000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bdd510aee7232f7f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:41.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.717+0000 7f26d7400640 1 -- 172.21.15.43:0/3378011430 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e4039040 con 0x55e132843550 2024-03-28T22:42:41.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.717+0000 7f26d7400640 1 -- 172.21.15.43:0/3378011430 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26e4055340 con 0x55e132843550 2024-03-28T22:42:41.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.717+0000 7f26d7400640 1 -- 172.21.15.43:0/3378011430 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e4003d60 con 0x55e132843550 2024-03-28T22:42:41.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.718+0000 7f26f4f88740 1 -- 172.21.15.43:0/3378011430 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e132843550 msgr2=0x55e13284c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:41.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.718+0000 7f26f4f88740 1 --2- 172.21.15.43:0/3378011430 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e132843550 0x55e13284c270 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f26e4009a60 tx=0x7f26e4041000 comp rx=0 tx=0).stop 2024-03-28T22:42:41.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.718+0000 7f26f4f88740 1 -- 172.21.15.43:0/3378011430 shutdown_connections 2024-03-28T22:42:41.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.718+0000 7f26f4f88740 1 --2- 172.21.15.43:0/3378011430 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e132843e70 0x55e132848db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.718+0000 7f26f4f88740 1 --2- 172.21.15.43:0/3378011430 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e13284ca50 0x55e1328585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.718+0000 7f26f4f88740 1 --2- 172.21.15.43:0/3378011430 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e132843550 0x55e13284c270 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.718+0000 7f26f4f88740 1 -- 172.21.15.43:0/3378011430 >> 172.21.15.43:0/3378011430 conn(0x55e13283dc80 msgr2=0x55e13284a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:41.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.718+0000 7f26f4f88740 1 -- 172.21.15.43:0/3378011430 shutdown_connections 2024-03-28T22:42:41.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.718+0000 7f26f4f88740 1 -- 172.21.15.43:0/3378011430 wait complete. 2024-03-28T22:42:41.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.719+0000 7f26f4f88740 1 -- start start 2024-03-28T22:42:41.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.719+0000 7f26f4f88740 1 Processor -- start 2024-03-28T22:42:41.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.719+0000 7f26f4f88740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e132843550 0x55e132850600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.719+0000 7f26f4f88740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e132843e70 0x55e132850b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.719+0000 7f26f4f88740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e13284ca50 0x55e132851080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.719+0000 7f26f4f88740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f26ec1b2000 con 0x55e132843e70 2024-03-28T22:42:41.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.719+0000 7f26f4f88740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f26ec1b1930 con 0x55e132843550 2024-03-28T22:42:41.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.719+0000 7f26f4f88740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f26ec1aeb70 con 0x55e13284ca50 2024-03-28T22:42:41.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.720+0000 7f26f4f88740 10 client.0 _ll_register_callbacks cb 0x7f26dc01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:41.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.720+0000 7f26eaa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e132843e70 0x55e132850b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.720+0000 7f26eaa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e132843e70 0x55e132850b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60460/0 (socket says 172.21.15.43:60460) 2024-03-28T22:42:41.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.720+0000 7f26eaa00640 1 -- 172.21.15.43:0/2979851989 learned_addr learned my addr 172.21.15.43:0/2979851989 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:41.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.720+0000 7f26ebe00640 1 --2- 172.21.15.43:0/2979851989 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e13284ca50 0x55e132851080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.720+0000 7f26eb400640 1 --2- 172.21.15.43:0/2979851989 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e132843550 0x55e132850600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.720+0000 7f26eaa00640 1 -- 172.21.15.43:0/2979851989 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e13284ca50 msgr2=0x55e132851080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:41.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.720+0000 7f26eaa00640 1 --2- 172.21.15.43:0/2979851989 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e13284ca50 0x55e132851080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.720+0000 7f26eaa00640 1 -- 172.21.15.43:0/2979851989 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e132843550 msgr2=0x55e132850600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:41.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.720+0000 7f26eaa00640 1 --2- 172.21.15.43:0/2979851989 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e132843550 0x55e132850600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.720+0000 7f26eaa00640 1 -- 172.21.15.43:0/2979851989 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1328c2190 con 0x55e132843e70 2024-03-28T22:42:41.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.720+0000 7f26eaa00640 1 --2- 172.21.15.43:0/2979851989 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e132843e70 0x55e132850b40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f26d8096f30 tx=0x7f26d80051e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94720b9a44e9cdf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:41.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.720+0000 7f26d4c00640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:42:41.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.721+0000 7f26d4c00640 1 -- 172.21.15.43:0/2979851989 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d8099ec0 con 0x55e132843e70 2024-03-28T22:42:41.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.721+0000 7f26d4c00640 1 -- 172.21.15.43:0/2979851989 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f26d80cb540 con 0x55e132843e70 2024-03-28T22:42:41.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.721+0000 7f26f4f88740 1 -- 172.21.15.43:0/2979851989 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26e4055340 con 0x55e132843e70 2024-03-28T22:42:41.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.721+0000 7f26d4c00640 1 -- 172.21.15.43:0/2979851989 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f26d80cb740 con 0x55e132843e70 2024-03-28T22:42:41.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.721+0000 7f26f4f88740 10 client.17478 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:41.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.721+0000 7f26f4f88740 1 -- 172.21.15.43:0/2979851989 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e1328c3470 con 0x55e132843e70 2024-03-28T22:42:41.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.721+0000 7f26d4c00640 1 -- 172.21.15.43:0/2979851989 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26e4055340 con 0x55e132843e70 2024-03-28T22:42:41.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.721+0000 7f26d4c00640 1 -- 172.21.15.43:0/2979851989 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f26d80999b0 con 0x55e132843e70 2024-03-28T22:42:41.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.722+0000 7f26f4f88740 10 client.17478 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:41.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.722+0000 7f26f4f88740 10 client.17478 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:42:41.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.722+0000 7f26f4f88740 20 client.17478 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:41.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.722+0000 7f26f4f88740 1 --2- 172.21.15.43:0/2979851989 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55e1328c3cc0 0x55e1328c60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.722+0000 7f26f4f88740 4 client.17478 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:42:41.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.722+0000 7f26f4f88740 1 -- 172.21.15.43:0/2979851989 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e1327b4590 con 0x55e1328c3cc0 2024-03-28T22:42:41.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.722+0000 7f26eb400640 1 --2- 172.21.15.43:0/2979851989 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55e1328c3cc0 0x55e1328c60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.723+0000 7f26eb400640 1 --2- 172.21.15.43:0/2979851989 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55e1328c3cc0 0x55e1328c60e0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e9fab1cd6b77500b server_cookie=b4c042faa1dc5281 in_seq=0 out_seq=0 2024-03-28T22:42:41.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.723+0000 7f26d4c00640 10 client.17478 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:42:41.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.723+0000 7f26d4c00640 1 -- 172.21.15.43:0/2979851989 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e1327b4590 con 0x55e1328c3cc0 2024-03-28T22:42:41.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.723+0000 7f26d4c00640 10 client.17478 handle_command_reply: tid=0 2024-03-28T22:42:41.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.723+0000 7f26f4f88740 1 client.17478 shutdown 2024-03-28T22:42:41.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.723+0000 7f26f4f88740 1 -- 172.21.15.43:0/2979851989 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e132843e70 msgr2=0x55e132850b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:41.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.723+0000 7f26f4f88740 1 --2- 172.21.15.43:0/2979851989 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e132843e70 0x55e132850b40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f26d8096f30 tx=0x7f26d80051e0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.723+0000 7f26f4f88740 1 -- 172.21.15.43:0/2979851989 shutdown_connections 2024-03-28T22:42:41.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.723+0000 7f26f4f88740 1 --2- 172.21.15.43:0/2979851989 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e13284ca50 0x55e132851080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.723+0000 7f26f4f88740 1 --2- 172.21.15.43:0/2979851989 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e132843e70 0x55e132850b40 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.723+0000 7f26f4f88740 1 --2- 172.21.15.43:0/2979851989 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55e1328c3cc0 0x55e1328c60e0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.723+0000 7f26f4f88740 1 --2- 172.21.15.43:0/2979851989 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e132843550 0x55e132850600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.723+0000 7f26f4f88740 1 -- 172.21.15.43:0/2979851989 >> 172.21.15.43:0/2979851989 conn(0x55e13283dc80 msgr2=0x55e13283c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:41.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.723+0000 7f26d4c00640 0 client.17478 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:42:41.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.723+0000 7f26f4f88740 1 -- 172.21.15.43:0/2979851989 shutdown_connections 2024-03-28T22:42:41.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.723+0000 7f26f4f88740 1 -- 172.21.15.43:0/2979851989 wait complete. 2024-03-28T22:42:41.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.723+0000 7f26f4f88740 20 client.17478 trim_cache size 0 max 16384 2024-03-28T22:42:41.729 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:42:41.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.724+0000 7f26f0c00640 1 -- 172.21.15.43:0/2611239108 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f26d00a1ef0 msgr2=0x7f26d00a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:41.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.724+0000 7f26f0c00640 1 --2- 172.21.15.43:0/2611239108 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f26d00a1ef0 0x7f26d00a43b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f26e4009ca0 tx=0x7f26e40093a0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.724+0000 7f26f0c00640 1 -- 172.21.15.43:0/2611239108 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f26ec101f90 msgr2=0x7f26ec19a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:41.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.724+0000 7f26f0c00640 1 --2- 172.21.15.43:0/2611239108 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f26ec101f90 0x7f26ec19a580 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f26d800d950 tx=0x7f26d800de10 comp rx=0 tx=0).stop 2024-03-28T22:42:41.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.724+0000 7f26f0c00640 1 -- 172.21.15.43:0/2611239108 shutdown_connections 2024-03-28T22:42:41.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.724+0000 7f26f0c00640 1 --2- 172.21.15.43:0/2611239108 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f26d00a1ef0 0x7f26d00a43b0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.724+0000 7f26f0c00640 1 --2- 172.21.15.43:0/2611239108 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f26ec107690 0x7f26ec19e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.724+0000 7f26f0c00640 1 --2- 172.21.15.43:0/2611239108 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f26ec101f90 0x7f26ec19a580 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.724+0000 7f26f0c00640 1 --2- 172.21.15.43:0/2611239108 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f26ec1015d0 0x7f26ec19a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.724+0000 7f26f0c00640 1 -- 172.21.15.43:0/2611239108 >> 172.21.15.43:0/2611239108 conn(0x7f26ec0fd0c0 msgr2=0x7f26ec10d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:41.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.724+0000 7f26f0c00640 1 -- 172.21.15.43:0/2611239108 shutdown_connections 2024-03-28T22:42:41.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.724+0000 7f26f0c00640 1 -- 172.21.15.43:0/2611239108 wait complete. 2024-03-28T22:42:41.738 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:42:41.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.971+0000 7f9ac3e00640 1 -- 172.21.15.43:0/1794940972 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ac41097e0 msgr2=0x7f9ac4109bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:41.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.971+0000 7f9ac3e00640 1 --2- 172.21.15.43:0/1794940972 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ac41097e0 0x7f9ac4109bc0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4009900 tx=0x7f9ab402bc40 comp rx=0 tx=0).stop 2024-03-28T22:42:41.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.971+0000 7f9ac3e00640 1 -- 172.21.15.43:0/1794940972 shutdown_connections 2024-03-28T22:42:41.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.971+0000 7f9ac3e00640 1 --2- 172.21.15.43:0/1794940972 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ac4104190 0x7f9ac41106a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.971+0000 7f9ac3e00640 1 --2- 172.21.15.43:0/1794940972 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ac41037d0 0x7f9ac4103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.971+0000 7f9ac3e00640 1 --2- 172.21.15.43:0/1794940972 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ac41097e0 0x7f9ac4109bc0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.971+0000 7f9ac3e00640 1 -- 172.21.15.43:0/1794940972 >> 172.21.15.43:0/1794940972 conn(0x7f9ac40ff280 msgr2=0x7f9ac41016a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:41.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.971+0000 7f9ac3e00640 1 -- 172.21.15.43:0/1794940972 shutdown_connections 2024-03-28T22:42:41.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.971+0000 7f9ac3e00640 1 -- 172.21.15.43:0/1794940972 wait complete. 2024-03-28T22:42:41.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.972+0000 7f9ac3e00640 1 Processor -- start 2024-03-28T22:42:41.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.973+0000 7f9ac3e00640 1 -- start start 2024-03-28T22:42:41.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.973+0000 7f9ac3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ac41037d0 0x7f9ac419a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.973+0000 7f9ac3e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ac4104190 0x7f9ac419a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.973+0000 7f9ac3e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ac41097e0 0x7f9ac419e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.973+0000 7f9ac3e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac4113c00 con 0x7f9ac41037d0 2024-03-28T22:42:41.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.973+0000 7f9ac3e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac4113a80 con 0x7f9ac4104190 2024-03-28T22:42:41.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.973+0000 7f9ac3e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac4113d80 con 0x7f9ac41097e0 2024-03-28T22:42:41.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.973+0000 7f9ac2a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ac41037d0 0x7f9ac419a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.973+0000 7f9ac2a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ac41037d0 0x7f9ac419a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60482/0 (socket says 172.21.15.43:60482) 2024-03-28T22:42:41.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.973+0000 7f9ac2a00640 1 -- 172.21.15.43:0/2946459405 learned_addr learned my addr 172.21.15.43:0/2946459405 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:41.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.973+0000 7f9ac2000640 1 --2- 172.21.15.43:0/2946459405 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ac4104190 0x7f9ac419a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.973+0000 7f9ac3400640 1 --2- 172.21.15.43:0/2946459405 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ac41097e0 0x7f9ac419e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.974+0000 7f9ac2a00640 1 -- 172.21.15.43:0/2946459405 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ac41097e0 msgr2=0x7f9ac419e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:41.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.974+0000 7f9ac2a00640 1 --2- 172.21.15.43:0/2946459405 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ac41097e0 0x7f9ac419e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.974+0000 7f9ac2a00640 1 -- 172.21.15.43:0/2946459405 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ac4104190 msgr2=0x7f9ac419a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:41.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.974+0000 7f9ac2a00640 1 --2- 172.21.15.43:0/2946459405 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ac4104190 0x7f9ac419a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.974+0000 7f9ac2a00640 1 -- 172.21.15.43:0/2946459405 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac419f0b0 con 0x7f9ac41037d0 2024-03-28T22:42:41.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.974+0000 7f9ac2000640 1 --2- 172.21.15.43:0/2946459405 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ac4104190 0x7f9ac419a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:41.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.974+0000 7f9ac3400640 1 --2- 172.21.15.43:0/2946459405 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ac41097e0 0x7f9ac419e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:41.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.975+0000 7f9ac2a00640 1 --2- 172.21.15.43:0/2946459405 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ac41037d0 0x7f9ac419a0c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4005ab0 tx=0x7f9ab4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:41.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.975+0000 7f9abb400640 1 -- 172.21.15.43:0/2946459405 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4041070 con 0x7f9ac41037d0 2024-03-28T22:42:41.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.975+0000 7f9abb400640 1 -- 172.21.15.43:0/2946459405 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ab40054e0 con 0x7f9ac41037d0 2024-03-28T22:42:41.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.975+0000 7f9abb400640 1 -- 172.21.15.43:0/2946459405 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4003bf0 con 0x7f9ac41037d0 2024-03-28T22:42:41.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.975+0000 7f9ac3e00640 1 -- 172.21.15.43:0/2946459405 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ac419f340 con 0x7f9ac41037d0 2024-03-28T22:42:41.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.975+0000 7f9ac3e00640 1 -- 172.21.15.43:0/2946459405 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ac41a6c30 con 0x7f9ac41037d0 2024-03-28T22:42:41.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.979+0000 7f9abb400640 1 -- 172.21.15.43:0/2946459405 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f9ab4044070 con 0x7f9ac41037d0 2024-03-28T22:42:41.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.979+0000 7f9acc390740 1 Processor -- start 2024-03-28T22:42:41.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.979+0000 7f9acc390740 1 -- start start 2024-03-28T22:42:41.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.980+0000 7f9acc390740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55be603f6550 0x55be603f6930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.980+0000 7f9acc390740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55be603f6e70 0x55be603ff510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.980+0000 7f9acc390740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55be603ffa50 0x55be60401e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.980+0000 7f9acc390740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55be6035a050 con 0x55be603f6e70 2024-03-28T22:42:41.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.980+0000 7f9acc390740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55be60302ce0 con 0x55be603f6550 2024-03-28T22:42:41.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.980+0000 7f9acc390740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55be603560f0 con 0x55be603ffa50 2024-03-28T22:42:41.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.980+0000 7f9ac3400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55be603f6550 0x55be603f6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.980+0000 7f9ac3400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55be603f6550 0x55be603f6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47896/0 (socket says 172.21.15.43:47896) 2024-03-28T22:42:41.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.980+0000 7f9ac3400640 1 -- 172.21.15.43:0/4105387344 learned_addr learned my addr 172.21.15.43:0/4105387344 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:41.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.980+0000 7f9ac2a00640 1 --2- 172.21.15.43:0/4105387344 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55be603f6e70 0x55be603ff510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.980+0000 7f9ac2000640 1 --2- 172.21.15.43:0/4105387344 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55be603ffa50 0x55be60401e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.981+0000 7f9ac3400640 1 -- 172.21.15.43:0/4105387344 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55be603ffa50 msgr2=0x55be60401e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:41.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.981+0000 7f9ac3400640 1 --2- 172.21.15.43:0/4105387344 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55be603ffa50 0x55be60401e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.981+0000 7f9ac3400640 1 -- 172.21.15.43:0/4105387344 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55be603f6e70 msgr2=0x55be603ff510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:41.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.981+0000 7f9ac3400640 1 --2- 172.21.15.43:0/4105387344 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55be603f6e70 0x55be603ff510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.981+0000 7f9ac3400640 1 -- 172.21.15.43:0/4105387344 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be6006e3f0 con 0x55be603f6550 2024-03-28T22:42:41.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.981+0000 7f9ac2a00640 1 --2- 172.21.15.43:0/4105387344 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55be603f6e70 0x55be603ff510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:42:41.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.981+0000 7f9ac3400640 1 --2- 172.21.15.43:0/4105387344 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55be603f6550 0x55be603f6930 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f9abc011470 tx=0x7f9abc011930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd9ddf6118f44660 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:41.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.981+0000 7f9ab3400640 1 -- 172.21.15.43:0/4105387344 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9abc01e070 con 0x55be603f6550 2024-03-28T22:42:41.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.981+0000 7f9ab3400640 1 -- 172.21.15.43:0/4105387344 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9abc00fa20 con 0x55be603f6550 2024-03-28T22:42:41.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.981+0000 7f9ab3400640 1 -- 172.21.15.43:0/4105387344 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9abc019b80 con 0x55be603f6550 2024-03-28T22:42:41.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.982+0000 7f9abb400640 1 --2- 172.21.15.43:0/2946459405 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9aa80a1f70 0x7f9aa80a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.982+0000 7f9abb400640 1 -- 172.21.15.43:0/2946459405 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f9ab40effe0 con 0x7f9ac41037d0 2024-03-28T22:42:41.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.982+0000 7f9ac2a00640 1 --2- 172.21.15.43:0/2946459405 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9aa80a1f70 0x7f9aa80a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.983+0000 7f9acc390740 1 -- 172.21.15.43:0/4105387344 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55be603f6550 msgr2=0x55be603f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:41.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.983+0000 7f9acc390740 1 --2- 172.21.15.43:0/4105387344 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55be603f6550 0x55be603f6930 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f9abc011470 tx=0x7f9abc011930 comp rx=0 tx=0).stop 2024-03-28T22:42:41.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.983+0000 7f9acc390740 1 -- 172.21.15.43:0/4105387344 shutdown_connections 2024-03-28T22:42:41.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.983+0000 7f9acc390740 1 --2- 172.21.15.43:0/4105387344 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55be603ffa50 0x55be60401e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.983+0000 7f9acc390740 1 --2- 172.21.15.43:0/4105387344 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55be603f6e70 0x55be603ff510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.983+0000 7f9acc390740 1 --2- 172.21.15.43:0/4105387344 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55be603f6550 0x55be603f6930 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.983+0000 7f9ac2a00640 1 --2- 172.21.15.43:0/2946459405 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9aa80a1f70 0x7f9aa80a4430 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x55be603f6020 tx=0x7f9ab40dc040 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:41.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.983+0000 7f9acc390740 1 -- 172.21.15.43:0/4105387344 >> 172.21.15.43:0/4105387344 conn(0x55be603f0c80 msgr2=0x55be603f1060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:41.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.983+0000 7f9acc390740 1 -- 172.21.15.43:0/4105387344 shutdown_connections 2024-03-28T22:42:41.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.983+0000 7f9acc390740 1 -- 172.21.15.43:0/4105387344 wait complete. 2024-03-28T22:42:41.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.984+0000 7f9acc390740 1 -- start start 2024-03-28T22:42:41.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.984+0000 7f9acc390740 1 Processor -- start 2024-03-28T22:42:41.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.984+0000 7f9acc390740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55be603f6550 0x55be603fcaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.985+0000 7f9acc390740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55be603f6e70 0x55be603fcfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.985+0000 7f9acc390740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55be603ffa50 0x55be603fd520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.985+0000 7f9acc390740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55be60402840 con 0x55be603f6550 2024-03-28T22:42:41.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.985+0000 7f9acc390740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55be602f82f0 con 0x55be603f6e70 2024-03-28T22:42:41.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.985+0000 7f9acc390740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55be604029c0 con 0x55be603ffa50 2024-03-28T22:42:41.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.985+0000 7f9acc390740 10 client.0 _ll_register_callbacks cb 0x55be6006dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:41.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.985+0000 7f9ac2000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55be603f6e70 0x55be603fcfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.985+0000 7f9ac2000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55be603f6e70 0x55be603fcfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47908/0 (socket says 172.21.15.43:47908) 2024-03-28T22:42:41.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.985+0000 7f9ac2000640 1 -- 172.21.15.43:0/2515693596 learned_addr learned my addr 172.21.15.43:0/2515693596 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:41.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.985+0000 7f9ac3400640 1 --2- 172.21.15.43:0/2515693596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55be603f6550 0x55be603fcaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.985+0000 7f9ac3400640 1 --2- 172.21.15.43:0/2515693596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55be603ffa50 0x55be603fd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.985+0000 7f9ac2000640 1 -- 172.21.15.43:0/2515693596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55be603ffa50 msgr2=0x55be603fd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:41.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.985+0000 7f9ac2000640 1 --2- 172.21.15.43:0/2515693596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55be603ffa50 0x55be603fd520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.985+0000 7f9ac2000640 1 -- 172.21.15.43:0/2515693596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55be603f6550 msgr2=0x55be603fcaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:41.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.985+0000 7f9ac2000640 1 --2- 172.21.15.43:0/2515693596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55be603f6550 0x55be603fcaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.985+0000 7f9ac2000640 1 -- 172.21.15.43:0/2515693596 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be601387d0 con 0x55be603f6e70 2024-03-28T22:42:41.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.985+0000 7f9ac3400640 1 --2- 172.21.15.43:0/2515693596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55be603ffa50 0x55be603fd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:42:41.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.986+0000 7f9ac2000640 1 --2- 172.21.15.43:0/2515693596 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55be603f6e70 0x55be603fcfe0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f9aac00fe20 tx=0x7f9aac012820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fdfec2cac9662e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:41.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.986+0000 7f9aa7e00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:42:41.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.986+0000 7f9aa7e00640 1 -- 172.21.15.43:0/2515693596 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aac003970 con 0x55be603f6e70 2024-03-28T22:42:41.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.986+0000 7f9acc390740 1 -- 172.21.15.43:0/2515693596 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9abc00fa20 con 0x55be603f6e70 2024-03-28T22:42:41.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.986+0000 7f9aa7e00640 1 -- 172.21.15.43:0/2515693596 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f9aac003b10 con 0x55be603f6e70 2024-03-28T22:42:41.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.986+0000 7f9aa7e00640 1 -- 172.21.15.43:0/2515693596 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9aac025050 con 0x55be603f6e70 2024-03-28T22:42:41.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.986+0000 7f9acc390740 10 client.27716 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:41.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.986+0000 7f9acc390740 1 -- 172.21.15.43:0/2515693596 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55be60405b70 con 0x55be603f6e70 2024-03-28T22:42:41.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.987+0000 7f9aa7e00640 1 -- 172.21.15.43:0/2515693596 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9abc00fa20 con 0x55be603f6e70 2024-03-28T22:42:41.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.987+0000 7f9aa7e00640 1 -- 172.21.15.43:0/2515693596 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f9aac0104c0 con 0x55be603f6e70 2024-03-28T22:42:41.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.987+0000 7f9acc390740 10 client.27716 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:41.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.987+0000 7f9acc390740 10 client.27716 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:42:41.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.987+0000 7f9acc390740 20 client.27716 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:41.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.987+0000 7f9acc390740 1 --2- 172.21.15.43:0/2515693596 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55be60406f80 0x55be60409400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:41.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.987+0000 7f9acc390740 4 client.27716 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:42:41.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.987+0000 7f9acc390740 1 -- 172.21.15.43:0/2515693596 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55be60303680 con 0x55be60406f80 2024-03-28T22:42:41.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.987+0000 7f9ac3400640 1 --2- 172.21.15.43:0/2515693596 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55be60406f80 0x55be60409400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:41.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.988+0000 7f9ac3400640 1 --2- 172.21.15.43:0/2515693596 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55be60406f80 0x55be60409400 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5f03158a0aeb8198 server_cookie=d25cb895414543fc in_seq=0 out_seq=0 2024-03-28T22:42:41.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.988+0000 7f9aa7e00640 10 client.27716 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:42:41.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.989+0000 7f9aa7e00640 1 -- 172.21.15.43:0/2515693596 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55be60303680 con 0x55be60406f80 2024-03-28T22:42:41.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.989+0000 7f9aa7e00640 10 client.27716 handle_command_reply: tid=0 2024-03-28T22:42:41.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.989+0000 7f9acc390740 1 client.27716 shutdown 2024-03-28T22:42:41.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.989+0000 7f9acc390740 1 -- 172.21.15.43:0/2515693596 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55be603f6e70 msgr2=0x55be603fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:41.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.989+0000 7f9acc390740 1 --2- 172.21.15.43:0/2515693596 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55be603f6e70 0x55be603fcfe0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f9aac00fe20 tx=0x7f9aac012820 comp rx=0 tx=0).stop 2024-03-28T22:42:41.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.989+0000 7f9acc390740 1 -- 172.21.15.43:0/2515693596 shutdown_connections 2024-03-28T22:42:41.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.989+0000 7f9acc390740 1 --2- 172.21.15.43:0/2515693596 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55be60406f80 0x55be60409400 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.989+0000 7f9acc390740 1 --2- 172.21.15.43:0/2515693596 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55be603f6e70 0x55be603fcfe0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.989+0000 7f9acc390740 1 --2- 172.21.15.43:0/2515693596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55be603ffa50 0x55be603fd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.989+0000 7f9acc390740 1 --2- 172.21.15.43:0/2515693596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55be603f6550 0x55be603fcaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:41.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.989+0000 7f9acc390740 1 -- 172.21.15.43:0/2515693596 >> 172.21.15.43:0/2515693596 conn(0x55be603f0c80 msgr2=0x55be603f29d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:41.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.989+0000 7f9aa7e00640 0 client.27716 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:42:41.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.989+0000 7f9acc390740 1 -- 172.21.15.43:0/2515693596 shutdown_connections 2024-03-28T22:42:41.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.989+0000 7f9acc390740 1 -- 172.21.15.43:0/2515693596 wait complete. 2024-03-28T22:42:41.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:41.989+0000 7f9acc390740 20 client.27716 trim_cache size 0 max 16384 2024-03-28T22:42:42.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.005+0000 7f9acc390740 1 Processor -- start 2024-03-28T22:42:42.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.005+0000 7f9acc390740 1 -- start start 2024-03-28T22:42:42.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.005+0000 7f9acc390740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55be603f6550 0x55be603ff1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:42.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.005+0000 7f9acc390740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55be603f6e70 0x55be603ebc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:42.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.005+0000 7f9acc390740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55be603ffa50 0x55be60413dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:42.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.005+0000 7f9acc390740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac41ac860 con 0x55be603f6550 2024-03-28T22:42:42.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.005+0000 7f9acc390740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac41ae3a0 con 0x55be603f6e70 2024-03-28T22:42:42.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.005+0000 7f9acc390740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac41acf30 con 0x55be603ffa50 2024-03-28T22:42:42.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.006+0000 7f9ac2000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55be603f6e70 0x55be603ebc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:42.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.006+0000 7f9ac2000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55be603f6e70 0x55be603ebc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47912/0 (socket says 172.21.15.43:47912) 2024-03-28T22:42:42.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.006+0000 7f9ac2000640 1 -- 172.21.15.43:0/3689600270 learned_addr learned my addr 172.21.15.43:0/3689600270 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:42.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.006+0000 7f9ac3400640 1 --2- 172.21.15.43:0/3689600270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55be603f6550 0x55be603ff1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:42.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.006+0000 7f9ac3400640 1 --2- 172.21.15.43:0/3689600270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55be603ffa50 0x55be60413dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:42.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.006+0000 7f9ac2000640 1 -- 172.21.15.43:0/3689600270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55be603ffa50 msgr2=0x55be60413dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:42.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.006+0000 7f9ac2000640 1 --2- 172.21.15.43:0/3689600270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55be603ffa50 0x55be60413dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.006+0000 7f9ac2000640 1 -- 172.21.15.43:0/3689600270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55be603f6550 msgr2=0x55be603ff1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:42.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.006+0000 7f9ac2000640 1 --2- 172.21.15.43:0/3689600270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55be603f6550 0x55be603ff1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.006+0000 7f9ac2000640 1 -- 172.21.15.43:0/3689600270 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be603fba50 con 0x55be603f6e70 2024-03-28T22:42:42.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.007+0000 7f9ac2000640 1 --2- 172.21.15.43:0/3689600270 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55be603f6e70 0x55be603ebc50 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f9aac006010 tx=0x7f9aac00f930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86a45fff08a5925d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:42.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.007+0000 7f9ab2a00640 1 -- 172.21.15.43:0/3689600270 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aac02f070 con 0x55be603f6e70 2024-03-28T22:42:42.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.007+0000 7f9ab2a00640 1 -- 172.21.15.43:0/3689600270 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9aac003e50 con 0x55be603f6e70 2024-03-28T22:42:42.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.007+0000 7f9ab2a00640 1 -- 172.21.15.43:0/3689600270 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aac005110 con 0x55be603f6e70 2024-03-28T22:42:42.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.008+0000 7f9acc390740 1 -- 172.21.15.43:0/3689600270 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55be603f6e70 msgr2=0x55be603ebc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:42.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.008+0000 7f9acc390740 1 --2- 172.21.15.43:0/3689600270 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55be603f6e70 0x55be603ebc50 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f9aac006010 tx=0x7f9aac00f930 comp rx=0 tx=0).stop 2024-03-28T22:42:42.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.008+0000 7f9acc390740 1 -- 172.21.15.43:0/3689600270 shutdown_connections 2024-03-28T22:42:42.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.008+0000 7f9acc390740 1 --2- 172.21.15.43:0/3689600270 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55be603f6e70 0x55be603ebc50 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.008+0000 7f9acc390740 1 --2- 172.21.15.43:0/3689600270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55be603ffa50 0x55be60413dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.008+0000 7f9acc390740 1 --2- 172.21.15.43:0/3689600270 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55be603f6550 0x55be603ff1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.008+0000 7f9acc390740 1 -- 172.21.15.43:0/3689600270 >> 172.21.15.43:0/3689600270 conn(0x55be603f0c80 msgr2=0x55be604136b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:42.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.008+0000 7f9acc390740 1 -- 172.21.15.43:0/3689600270 shutdown_connections 2024-03-28T22:42:42.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.008+0000 7f9acc390740 1 -- 172.21.15.43:0/3689600270 wait complete. 2024-03-28T22:42:42.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.008+0000 7f9acc390740 1 -- start start 2024-03-28T22:42:42.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.008+0000 7f9acc390740 1 Processor -- start 2024-03-28T22:42:42.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.009+0000 7f9acc390740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55be603f6550 0x55be6040ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:42.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.009+0000 7f9acc390740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55be603f6e70 0x55be6040bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:42.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.009+0000 7f9acc390740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55be603ffa50 0x55be6040c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:42.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.009+0000 7f9acc390740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac41adcd0 con 0x55be603ffa50 2024-03-28T22:42:42.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.009+0000 7f9acc390740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac41ad600 con 0x55be603f6550 2024-03-28T22:42:42.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.009+0000 7f9acc390740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac41aa840 con 0x55be603f6e70 2024-03-28T22:42:42.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.009+0000 7f9acc390740 10 client.0 _ll_register_callbacks cb 0x7f9aac00f4a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:42.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.009+0000 7f9ac3400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55be603f6550 0x55be6040ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:42.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.009+0000 7f9ac3400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55be603f6550 0x55be6040ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47918/0 (socket says 172.21.15.43:47918) 2024-03-28T22:42:42.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.009+0000 7f9ac3400640 1 -- 172.21.15.43:0/3735309467 learned_addr learned my addr 172.21.15.43:0/3735309467 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:42.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.009+0000 7f9ac2000640 1 --2- 172.21.15.43:0/3735309467 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55be603f6e70 0x55be6040bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:42.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.009+0000 7f9ac3400640 1 --2- 172.21.15.43:0/3735309467 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55be603ffa50 0x55be6040c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:42.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.010+0000 7f9ac3400640 1 -- 172.21.15.43:0/3735309467 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55be603f6e70 msgr2=0x55be6040bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:42.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.010+0000 7f9ac3400640 1 --2- 172.21.15.43:0/3735309467 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55be603f6e70 0x55be6040bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.010+0000 7f9ac3400640 1 -- 172.21.15.43:0/3735309467 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55be603ffa50 msgr2=0x55be6040c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:42.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.010+0000 7f9ac3400640 1 --2- 172.21.15.43:0/3735309467 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55be603ffa50 0x55be6040c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.010+0000 7f9ac3400640 1 -- 172.21.15.43:0/3735309467 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be6040c730 con 0x55be603f6550 2024-03-28T22:42:42.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.010+0000 7f9ac2000640 1 --2- 172.21.15.43:0/3735309467 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55be603f6e70 0x55be6040bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:42.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.010+0000 7f9ac3400640 1 --2- 172.21.15.43:0/3735309467 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55be603f6550 0x55be6040ece0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f9abc011970 tx=0x7f9abc02a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd31cdab2ca70555 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:42.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.010+0000 7f9aa7e00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:42:42.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.010+0000 7f9aa7e00640 1 -- 172.21.15.43:0/3735309467 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9abc01e040 con 0x55be603f6550 2024-03-28T22:42:42.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.010+0000 7f9acc390740 1 -- 172.21.15.43:0/3735309467 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9aac003e50 con 0x55be603f6550 2024-03-28T22:42:42.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.010+0000 7f9aa7e00640 1 -- 172.21.15.43:0/3735309467 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f9abc00f6b0 con 0x55be603f6550 2024-03-28T22:42:42.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.011+0000 7f9aa7e00640 1 -- 172.21.15.43:0/3735309467 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9abc00f8d0 con 0x55be603f6550 2024-03-28T22:42:42.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.011+0000 7f9acc390740 10 client.27722 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:42.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.011+0000 7f9acc390740 1 -- 172.21.15.43:0/3735309467 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55be60487730 con 0x55be603f6550 2024-03-28T22:42:42.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.011+0000 7f9aa7e00640 1 -- 172.21.15.43:0/3735309467 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9aac003e50 con 0x55be603f6550 2024-03-28T22:42:42.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.011+0000 7f9aa7e00640 1 -- 172.21.15.43:0/3735309467 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f9abc024040 con 0x55be603f6550 2024-03-28T22:42:42.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.011+0000 7f9acc390740 10 client.27722 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:42.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.011+0000 7f9acc390740 10 client.27722 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:42:42.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.011+0000 7f9acc390740 20 client.27722 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:42.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.011+0000 7f9acc390740 1 --2- 172.21.15.43:0/3735309467 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55be60487c40 0x55be6048a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:42.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.011+0000 7f9acc390740 4 client.27722 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:42:42.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.011+0000 7f9acc390740 1 -- 172.21.15.43:0/3735309467 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55be60367590 con 0x55be60487c40 2024-03-28T22:42:42.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.012+0000 7f9ac2000640 1 --2- 172.21.15.43:0/3735309467 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55be60487c40 0x55be6048a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:42.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.012+0000 7f9ac2000640 1 --2- 172.21.15.43:0/3735309467 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55be60487c40 0x55be6048a060 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=af48ab0cda8b568 server_cookie=a20530077d166c0d in_seq=0 out_seq=0 2024-03-28T22:42:42.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.012+0000 7f9aa7e00640 10 client.27722 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:42:42.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.013+0000 7f9aa7e00640 1 -- 172.21.15.43:0/3735309467 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55be60367590 con 0x55be60487c40 2024-03-28T22:42:42.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.013+0000 7f9aa7e00640 10 client.27722 handle_command_reply: tid=0 2024-03-28T22:42:42.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.013+0000 7f9acc390740 1 client.27722 shutdown 2024-03-28T22:42:42.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.013+0000 7f9acc390740 1 -- 172.21.15.43:0/3735309467 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55be603f6550 msgr2=0x55be6040ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:42.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.013+0000 7f9acc390740 1 --2- 172.21.15.43:0/3735309467 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55be603f6550 0x55be6040ece0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f9abc011970 tx=0x7f9abc02a040 comp rx=0 tx=0).stop 2024-03-28T22:42:42.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.013+0000 7f9acc390740 1 -- 172.21.15.43:0/3735309467 shutdown_connections 2024-03-28T22:42:42.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.013+0000 7f9acc390740 1 --2- 172.21.15.43:0/3735309467 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55be60487c40 0x55be6048a060 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.013+0000 7f9acc390740 1 --2- 172.21.15.43:0/3735309467 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55be603ffa50 0x55be6040c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.013+0000 7f9acc390740 1 --2- 172.21.15.43:0/3735309467 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55be603f6e70 0x55be6040bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.013+0000 7f9acc390740 1 --2- 172.21.15.43:0/3735309467 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55be603f6550 0x55be6040ece0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.013+0000 7f9aa7e00640 0 client.27722 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:42:42.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.013+0000 7f9acc390740 1 -- 172.21.15.43:0/3735309467 >> 172.21.15.43:0/3735309467 conn(0x55be603f0c80 msgr2=0x55be603ef530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:42.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.013+0000 7f9acc390740 1 -- 172.21.15.43:0/3735309467 shutdown_connections 2024-03-28T22:42:42.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.013+0000 7f9acc390740 1 -- 172.21.15.43:0/3735309467 wait complete. 2024-03-28T22:42:42.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.013+0000 7f9acc390740 20 client.27722 trim_cache size 0 max 16384 2024-03-28T22:42:42.016 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:42:42.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.014+0000 7f9ac3e00640 1 -- 172.21.15.43:0/2946459405 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9aa80a1f70 msgr2=0x7f9aa80a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:42.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.014+0000 7f9ac3e00640 1 --2- 172.21.15.43:0/2946459405 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9aa80a1f70 0x7f9aa80a4430 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x55be603f6020 tx=0x7f9ab40dc040 comp rx=0 tx=0).stop 2024-03-28T22:42:42.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.014+0000 7f9ac3e00640 1 -- 172.21.15.43:0/2946459405 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ac41037d0 msgr2=0x7f9ac419a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:42.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.014+0000 7f9ac3e00640 1 --2- 172.21.15.43:0/2946459405 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ac41037d0 0x7f9ac419a0c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4005ab0 tx=0x7f9ab4004270 comp rx=0 tx=0).stop 2024-03-28T22:42:42.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.014+0000 7f9ac3e00640 1 -- 172.21.15.43:0/2946459405 shutdown_connections 2024-03-28T22:42:42.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.014+0000 7f9ac3e00640 1 --2- 172.21.15.43:0/2946459405 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9aa80a1f70 0x7f9aa80a4430 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.014+0000 7f9ac3e00640 1 --2- 172.21.15.43:0/2946459405 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ac4104190 0x7f9ac419a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.014+0000 7f9ac3e00640 1 --2- 172.21.15.43:0/2946459405 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9ac41097e0 0x7f9ac419e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.014+0000 7f9ac3e00640 1 --2- 172.21.15.43:0/2946459405 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9ac41037d0 0x7f9ac419a0c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.014+0000 7f9ac3e00640 1 -- 172.21.15.43:0/2946459405 >> 172.21.15.43:0/2946459405 conn(0x7f9ac40ff280 msgr2=0x7f9ac40ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:42.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.014+0000 7f9ac3e00640 1 -- 172.21.15.43:0/2946459405 shutdown_connections 2024-03-28T22:42:42.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.015+0000 7f9ac3e00640 1 -- 172.21.15.43:0/2946459405 wait complete. 2024-03-28T22:42:42.032 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:42:42.122 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:41 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3848555051' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:42:42.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:41 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3848555051' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:42:42.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:41 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3848555051' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:42:42.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.269+0000 7fa48cc00640 1 -- 172.21.15.43:0/372930613 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa4881016e0 msgr2=0x7fa488101b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:42.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.269+0000 7fa48cc00640 1 --2- 172.21.15.43:0/372930613 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa4881016e0 0x7fa488101b60 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fa470009900 tx=0x7fa47002bc40 comp rx=0 tx=0).stop 2024-03-28T22:42:42.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.269+0000 7fa48cc00640 1 -- 172.21.15.43:0/372930613 shutdown_connections 2024-03-28T22:42:42.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.269+0000 7fa48cc00640 1 --2- 172.21.15.43:0/372930613 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa4881016e0 0x7fa488101b60 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.269+0000 7fa48cc00640 1 --2- 172.21.15.43:0/372930613 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4881020a0 0x7fa48810e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.269+0000 7fa48cc00640 1 --2- 172.21.15.43:0/372930613 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4881076f0 0x7fa488107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.269+0000 7fa48cc00640 1 -- 172.21.15.43:0/372930613 >> 172.21.15.43:0/372930613 conn(0x7fa488078b50 msgr2=0x7fa488078f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:42.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.270+0000 7fa48cc00640 1 -- 172.21.15.43:0/372930613 shutdown_connections 2024-03-28T22:42:42.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.270+0000 7fa48cc00640 1 -- 172.21.15.43:0/372930613 wait complete. 2024-03-28T22:42:42.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.271+0000 7fa48cc00640 1 Processor -- start 2024-03-28T22:42:42.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.271+0000 7fa48cc00640 1 -- start start 2024-03-28T22:42:42.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.272+0000 7fa48cc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4881016e0 0x7fa488197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:42.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.272+0000 7fa48cc00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4881020a0 0x7fa488198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:42.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.272+0000 7fa48cc00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa4881076f0 0x7fa48819c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:42.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.272+0000 7fa48cc00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa488111ac0 con 0x7fa4881016e0 2024-03-28T22:42:42.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.272+0000 7fa48cc00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa488111940 con 0x7fa4881076f0 2024-03-28T22:42:42.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.272+0000 7fa48cc00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fa488111c40 con 0x7fa4881020a0 2024-03-28T22:42:42.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.272+0000 7fa485e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4881016e0 0x7fa488197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:42.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.272+0000 7fa47fe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4881020a0 0x7fa488198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:42.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.272+0000 7fa485e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4881016e0 0x7fa488197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60542/0 (socket says 172.21.15.43:60542) 2024-03-28T22:42:42.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.272+0000 7fa485e00640 1 -- 172.21.15.43:0/508204000 learned_addr learned my addr 172.21.15.43:0/508204000 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:42.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.272+0000 7fa486800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa4881076f0 0x7fa48819c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:42.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.272+0000 7fa485e00640 1 -- 172.21.15.43:0/508204000 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4881020a0 msgr2=0x7fa488198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:42.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.272+0000 7fa485e00640 1 --2- 172.21.15.43:0/508204000 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4881020a0 0x7fa488198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.272+0000 7fa485e00640 1 -- 172.21.15.43:0/508204000 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa4881076f0 msgr2=0x7fa48819c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:42.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.272+0000 7fa485e00640 1 --2- 172.21.15.43:0/508204000 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa4881076f0 0x7fa48819c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.273+0000 7fa485e00640 1 -- 172.21.15.43:0/508204000 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa48819cfb0 con 0x7fa4881016e0 2024-03-28T22:42:42.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.273+0000 7fa486800640 1 --2- 172.21.15.43:0/508204000 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa4881076f0 0x7fa48819c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:42.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.273+0000 7fa47fe00640 1 --2- 172.21.15.43:0/508204000 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4881020a0 0x7fa488198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:42.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.273+0000 7fa485e00640 1 --2- 172.21.15.43:0/508204000 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4881016e0 0x7fa488197fc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa47800e9f0 tx=0x7fa47800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:42.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.273+0000 7fa47ea00640 1 -- 172.21.15.43:0/508204000 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa47800cd80 con 0x7fa4881016e0 2024-03-28T22:42:42.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.273+0000 7fa48cc00640 1 -- 172.21.15.43:0/508204000 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa48819d2a0 con 0x7fa4881016e0 2024-03-28T22:42:42.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.273+0000 7fa47ea00640 1 -- 172.21.15.43:0/508204000 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa478010910 con 0x7fa4881016e0 2024-03-28T22:42:42.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.273+0000 7fa47ea00640 1 -- 172.21.15.43:0/508204000 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4780188f0 con 0x7fa4881016e0 2024-03-28T22:42:42.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.273+0000 7fa48cc00640 1 -- 172.21.15.43:0/508204000 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4881a4b30 con 0x7fa4881016e0 2024-03-28T22:42:42.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.277+0000 7fa48f7ee740 1 Processor -- start 2024-03-28T22:42:42.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.278+0000 7fa48f7ee740 1 -- start start 2024-03-28T22:42:42.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.278+0000 7fa47ea00640 1 -- 172.21.15.43:0/508204000 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fa478010ac0 con 0x7fa4881016e0 2024-03-28T22:42:42.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.278+0000 7fa48f7ee740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d89c90b380 0x55d89c90b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:42.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.278+0000 7fa48f7ee740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d89c90bca0 0x55d89c914340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:42.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.278+0000 7fa48f7ee740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d89c914880 0x55d89c916c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:42.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.278+0000 7fa48f7ee740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d89c86f050 con 0x55d89c914880 2024-03-28T22:42:42.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.278+0000 7fa48f7ee740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55d89c817ce0 con 0x55d89c90b380 2024-03-28T22:42:42.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.278+0000 7fa48f7ee740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55d89c86b0f0 con 0x55d89c90bca0 2024-03-28T22:42:42.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.279+0000 7fa485e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d89c90bca0 0x55d89c914340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:42.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.279+0000 7fa486800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d89c90b380 0x55d89c90b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:42.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.279+0000 7fa485e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d89c90bca0 0x55d89c914340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57032/0 (socket says 172.21.15.43:57032) 2024-03-28T22:42:42.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.279+0000 7fa485e00640 1 -- 172.21.15.43:0/3743863377 learned_addr learned my addr 172.21.15.43:0/3743863377 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.279+0000 7fa47fe00640 1 --2- 172.21.15.43:0/3743863377 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d89c914880 0x55d89c916c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.279+0000 7fa485e00640 1 -- 172.21.15.43:0/3743863377 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d89c90b380 msgr2=0x55d89c90b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.279+0000 7fa485e00640 1 --2- 172.21.15.43:0/3743863377 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d89c90b380 0x55d89c90b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.279+0000 7fa485e00640 1 -- 172.21.15.43:0/3743863377 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d89c914880 msgr2=0x55d89c916c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.279+0000 7fa485e00640 1 --2- 172.21.15.43:0/3743863377 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d89c914880 0x55d89c916c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.279+0000 7fa485e00640 1 -- 172.21.15.43:0/3743863377 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d89c5833f0 con 0x55d89c90bca0 2024-03-28T22:42:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.279+0000 7fa486800640 1 --2- 172.21.15.43:0/3743863377 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d89c90b380 0x55d89c90b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.279+0000 7fa47fe00640 1 --2- 172.21.15.43:0/3743863377 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d89c914880 0x55d89c916c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:42:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.280+0000 7fa485e00640 1 --2- 172.21.15.43:0/3743863377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d89c90bca0 0x55d89c914340 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fa4780cbad0 tx=0x7fa4780d6670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8f0d39711e62921 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.280+0000 7fa47ea00640 1 --2- 172.21.15.43:0/508204000 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa46c0a1f70 0x7fa46c0a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.280+0000 7fa477000640 1 -- 172.21.15.43:0/3743863377 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4780dd070 con 0x55d89c90bca0 2024-03-28T22:42:42.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.280+0000 7fa47ea00640 1 -- 172.21.15.43:0/508204000 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fa4780ce840 con 0x7fa4881016e0 2024-03-28T22:42:42.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.280+0000 7fa477000640 1 -- 172.21.15.43:0/3743863377 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa47809aee0 con 0x55d89c90bca0 2024-03-28T22:42:42.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.280+0000 7fa477000640 1 -- 172.21.15.43:0/3743863377 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa47809acb0 con 0x55d89c90bca0 2024-03-28T22:42:42.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.280+0000 7fa486800640 1 --2- 172.21.15.43:0/508204000 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa46c0a1f70 0x7fa46c0a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:42.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.281+0000 7fa48f7ee740 1 -- 172.21.15.43:0/3743863377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d89c90bca0 msgr2=0x55d89c914340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:42.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.281+0000 7fa486800640 1 --2- 172.21.15.43:0/508204000 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa46c0a1f70 0x7fa46c0a4430 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa480005fd0 tx=0x7fa480009600 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:42.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.281+0000 7fa48f7ee740 1 --2- 172.21.15.43:0/3743863377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d89c90bca0 0x55d89c914340 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fa4780cbad0 tx=0x7fa4780d6670 comp rx=0 tx=0).stop 2024-03-28T22:42:42.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.281+0000 7fa48f7ee740 1 -- 172.21.15.43:0/3743863377 shutdown_connections 2024-03-28T22:42:42.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.281+0000 7fa48f7ee740 1 --2- 172.21.15.43:0/3743863377 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d89c914880 0x55d89c916c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.281+0000 7fa48f7ee740 1 --2- 172.21.15.43:0/3743863377 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d89c90bca0 0x55d89c914340 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.281+0000 7fa48f7ee740 1 --2- 172.21.15.43:0/3743863377 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d89c90b380 0x55d89c90b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.281+0000 7fa48f7ee740 1 -- 172.21.15.43:0/3743863377 >> 172.21.15.43:0/3743863377 conn(0x55d89c905ab0 msgr2=0x55d89c905e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:42.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.281+0000 7fa48f7ee740 1 -- 172.21.15.43:0/3743863377 shutdown_connections 2024-03-28T22:42:42.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.282+0000 7fa48f7ee740 1 -- 172.21.15.43:0/3743863377 wait complete. 2024-03-28T22:42:42.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.282+0000 7fa48f7ee740 1 -- start start 2024-03-28T22:42:42.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.282+0000 7fa48f7ee740 1 Processor -- start 2024-03-28T22:42:42.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.283+0000 7fa48f7ee740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d89c90b380 0x55d89c914260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:42.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.283+0000 7fa48f7ee740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d89c90bca0 0x55d89c911850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:42.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.283+0000 7fa48f7ee740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d89c914880 0x55d89c911d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:42.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.283+0000 7fa48f7ee740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d89c91ba80 con 0x55d89c90b380 2024-03-28T22:42:42.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.283+0000 7fa48f7ee740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55d89c80d2f0 con 0x55d89c914880 2024-03-28T22:42:42.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.283+0000 7fa48f7ee740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55d89c91bc00 con 0x55d89c90bca0 2024-03-28T22:42:42.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.283+0000 7fa48f7ee740 10 client.0 _ll_register_callbacks cb 0x55d89c582be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:42.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.284+0000 7fa486800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d89c914880 0x55d89c911d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:42.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.284+0000 7fa486800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d89c914880 0x55d89c911d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47950/0 (socket says 172.21.15.43:47950) 2024-03-28T22:42:42.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.284+0000 7fa486800640 1 -- 172.21.15.43:0/450647005 learned_addr learned my addr 172.21.15.43:0/450647005 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:42.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.284+0000 7fa47fe00640 1 --2- 172.21.15.43:0/450647005 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d89c90bca0 0x55d89c911850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:42.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.284+0000 7fa486800640 1 -- 172.21.15.43:0/450647005 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d89c90bca0 msgr2=0x55d89c911850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:42.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.284+0000 7fa486800640 1 --2- 172.21.15.43:0/450647005 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d89c90bca0 0x55d89c911850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.284+0000 7fa486800640 1 -- 172.21.15.43:0/450647005 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d89c90b380 msgr2=0x55d89c914260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:42.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.284+0000 7fa486800640 1 --2- 172.21.15.43:0/450647005 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d89c90b380 0x55d89c914260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.284+0000 7fa486800640 1 -- 172.21.15.43:0/450647005 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d89c64d7d0 con 0x55d89c914880 2024-03-28T22:42:42.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.285+0000 7fa486800640 1 --2- 172.21.15.43:0/450647005 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d89c914880 0x55d89c911d90 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fa480011010 tx=0x7fa480004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b89f6c2ff274a071 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:42.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.285+0000 7fa46b400640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:42:42.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.285+0000 7fa46b400640 1 -- 172.21.15.43:0/450647005 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48003c060 con 0x55d89c914880 2024-03-28T22:42:42.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.285+0000 7fa48f7ee740 1 -- 172.21.15.43:0/450647005 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa47809aee0 con 0x55d89c914880 2024-03-28T22:42:42.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.286+0000 7fa46b400640 1 -- 172.21.15.43:0/450647005 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fa48001c660 con 0x55d89c914880 2024-03-28T22:42:42.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.286+0000 7fa46b400640 1 -- 172.21.15.43:0/450647005 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa48001ee10 con 0x55d89c914880 2024-03-28T22:42:42.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.286+0000 7fa48f7ee740 10 client.27734 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:42.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.286+0000 7fa48f7ee740 1 -- 172.21.15.43:0/450647005 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d89c92c1f0 con 0x55d89c914880 2024-03-28T22:42:42.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.286+0000 7fa46b400640 1 -- 172.21.15.43:0/450647005 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa47809aee0 con 0x55d89c914880 2024-03-28T22:42:42.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.287+0000 7fa46b400640 1 -- 172.21.15.43:0/450647005 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa48001a040 con 0x55d89c914880 2024-03-28T22:42:42.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.287+0000 7fa48f7ee740 10 client.27734 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:42.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.287+0000 7fa48f7ee740 10 client.27734 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:42:42.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.287+0000 7fa48f7ee740 20 client.27734 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:42.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.287+0000 7fa48f7ee740 1 --2- 172.21.15.43:0/450647005 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d89c92d600 0x55d89c92fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:42.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.287+0000 7fa48f7ee740 4 client.27734 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:42:42.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.287+0000 7fa48f7ee740 1 -- 172.21.15.43:0/450647005 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d89c818680 con 0x55d89c92d600 2024-03-28T22:42:42.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.288+0000 7fa485e00640 1 --2- 172.21.15.43:0/450647005 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d89c92d600 0x55d89c92fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:42.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.288+0000 7fa485e00640 1 --2- 172.21.15.43:0/450647005 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d89c92d600 0x55d89c92fa80 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25c38d877bf3e9b4 server_cookie=c51e7e4c9575024c in_seq=0 out_seq=0 2024-03-28T22:42:42.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.288+0000 7fa46b400640 10 client.27734 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:42.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.293+0000 7fa46b400640 1 -- 172.21.15.43:0/450647005 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d89c818680 con 0x55d89c92d600 2024-03-28T22:42:42.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.293+0000 7fa46b400640 10 client.27734 handle_command_reply: tid=0 2024-03-28T22:42:42.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.293+0000 7fa48f7ee740 1 client.27734 shutdown 2024-03-28T22:42:42.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.293+0000 7fa48f7ee740 1 -- 172.21.15.43:0/450647005 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d89c914880 msgr2=0x55d89c911d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:42.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.293+0000 7fa48f7ee740 1 --2- 172.21.15.43:0/450647005 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d89c914880 0x55d89c911d90 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fa480011010 tx=0x7fa480004a40 comp rx=0 tx=0).stop 2024-03-28T22:42:42.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.294+0000 7fa48f7ee740 1 -- 172.21.15.43:0/450647005 shutdown_connections 2024-03-28T22:42:42.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.294+0000 7fa48f7ee740 1 --2- 172.21.15.43:0/450647005 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d89c92d600 0x55d89c92fa80 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.294+0000 7fa48f7ee740 1 --2- 172.21.15.43:0/450647005 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d89c914880 0x55d89c911d90 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.294+0000 7fa48f7ee740 1 --2- 172.21.15.43:0/450647005 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d89c90bca0 0x55d89c911850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.294+0000 7fa48f7ee740 1 --2- 172.21.15.43:0/450647005 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d89c90b380 0x55d89c914260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.294+0000 7fa48f7ee740 1 -- 172.21.15.43:0/450647005 >> 172.21.15.43:0/450647005 conn(0x55d89c905ab0 msgr2=0x55d89c907800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:42.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.295+0000 7fa46b400640 0 client.27734 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:42.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.296+0000 7fa48f7ee740 1 -- 172.21.15.43:0/450647005 shutdown_connections 2024-03-28T22:42:42.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.297+0000 7fa48f7ee740 1 -- 172.21.15.43:0/450647005 wait complete. 2024-03-28T22:42:42.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.297+0000 7fa48f7ee740 20 client.27734 trim_cache size 0 max 16384 2024-03-28T22:42:42.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.327+0000 7fa48f7ee740 1 Processor -- start 2024-03-28T22:42:42.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.327+0000 7fa48f7ee740 1 -- start start 2024-03-28T22:42:42.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.327+0000 7fa48f7ee740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d89c90b380 0x55d89c913f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:42.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.327+0000 7fa48f7ee740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d89c90bca0 0x55d89c935350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:42.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.327+0000 7fa48f7ee740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d89c914880 0x55d89c911810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:42.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.327+0000 7fa48f7ee740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa488075010 con 0x55d89c90b380 2024-03-28T22:42:42.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.327+0000 7fa48f7ee740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa4881a5d90 con 0x55d89c90bca0 2024-03-28T22:42:42.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.327+0000 7fa48f7ee740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fa4880756e0 con 0x55d89c914880 2024-03-28T22:42:42.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.328+0000 7fa485e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d89c90b380 0x55d89c913f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:42.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.328+0000 7fa486800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d89c914880 0x55d89c911810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:42.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.328+0000 7fa47fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d89c90bca0 0x55d89c935350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:42.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.328+0000 7fa47fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d89c90bca0 0x55d89c935350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47954/0 (socket says 172.21.15.43:47954) 2024-03-28T22:42:42.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.328+0000 7fa485e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d89c90b380 0x55d89c913f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60566/0 (socket says 172.21.15.43:60566) 2024-03-28T22:42:42.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.328+0000 7fa47fe00640 1 -- 172.21.15.43:0/1880379251 learned_addr learned my addr 172.21.15.43:0/1880379251 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:42.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.328+0000 7fa486800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d89c914880 0x55d89c911810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57058/0 (socket says 172.21.15.43:57058) 2024-03-28T22:42:42.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.328+0000 7fa47fe00640 1 -- 172.21.15.43:0/1880379251 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d89c914880 msgr2=0x55d89c911810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:42.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.328+0000 7fa47fe00640 1 --2- 172.21.15.43:0/1880379251 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d89c914880 0x55d89c911810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.328+0000 7fa47fe00640 1 -- 172.21.15.43:0/1880379251 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d89c90b380 msgr2=0x55d89c913f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:42.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.328+0000 7fa47fe00640 1 --2- 172.21.15.43:0/1880379251 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d89c90b380 0x55d89c913f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.328+0000 7fa47fe00640 1 -- 172.21.15.43:0/1880379251 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d89c912830 con 0x55d89c90bca0 2024-03-28T22:42:42.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.328+0000 7fa486800640 1 --2- 172.21.15.43:0/1880379251 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d89c914880 0x55d89c911810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:42.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.328+0000 7fa485e00640 1 --2- 172.21.15.43:0/1880379251 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d89c90b380 0x55d89c913f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:42.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.329+0000 7fa47fe00640 1 --2- 172.21.15.43:0/1880379251 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d89c90bca0 0x55d89c935350 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fa47002dfd0 tx=0x7fa470005e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=772788bbac677cd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:42.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.329+0000 7fa476600640 1 -- 172.21.15.43:0/1880379251 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa47004c070 con 0x55d89c90bca0 2024-03-28T22:42:42.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.329+0000 7fa476600640 1 -- 172.21.15.43:0/1880379251 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa470004a70 con 0x55d89c90bca0 2024-03-28T22:42:42.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.329+0000 7fa476600640 1 -- 172.21.15.43:0/1880379251 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa470005190 con 0x55d89c90bca0 2024-03-28T22:42:42.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.330+0000 7fa48f7ee740 1 -- 172.21.15.43:0/1880379251 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d89c90bca0 msgr2=0x55d89c935350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:42.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.330+0000 7fa48f7ee740 1 --2- 172.21.15.43:0/1880379251 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d89c90bca0 0x55d89c935350 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fa47002dfd0 tx=0x7fa470005e00 comp rx=0 tx=0).stop 2024-03-28T22:42:42.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.330+0000 7fa48f7ee740 1 -- 172.21.15.43:0/1880379251 shutdown_connections 2024-03-28T22:42:42.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.330+0000 7fa48f7ee740 1 --2- 172.21.15.43:0/1880379251 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d89c90bca0 0x55d89c935350 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.330+0000 7fa48f7ee740 1 --2- 172.21.15.43:0/1880379251 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d89c914880 0x55d89c911810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.330+0000 7fa48f7ee740 1 --2- 172.21.15.43:0/1880379251 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d89c90b380 0x55d89c913f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.330+0000 7fa48f7ee740 1 -- 172.21.15.43:0/1880379251 >> 172.21.15.43:0/1880379251 conn(0x55d89c905ab0 msgr2=0x55d89c905e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:42.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.330+0000 7fa48f7ee740 1 -- 172.21.15.43:0/1880379251 shutdown_connections 2024-03-28T22:42:42.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.330+0000 7fa48f7ee740 1 -- 172.21.15.43:0/1880379251 wait complete. 2024-03-28T22:42:42.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.330+0000 7fa48f7ee740 1 -- start start 2024-03-28T22:42:42.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.331+0000 7fa48f7ee740 1 Processor -- start 2024-03-28T22:42:42.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.331+0000 7fa48f7ee740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d89c90b380 0x55d89c929cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:42.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.331+0000 7fa48f7ee740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d89c90bca0 0x55d89c92a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:42.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.331+0000 7fa48f7ee740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d89c914880 0x55d89c925270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:42.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.331+0000 7fa48f7ee740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa4881a56c0 con 0x55d89c90bca0 2024-03-28T22:42:42.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.331+0000 7fa48f7ee740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa4881a4ff0 con 0x55d89c90b380 2024-03-28T22:42:42.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.331+0000 7fa48f7ee740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fa488072ff0 con 0x55d89c914880 2024-03-28T22:42:42.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.331+0000 7fa48f7ee740 10 client.0 _ll_register_callbacks cb 0x7fa480010720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:42.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.331+0000 7fa485e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d89c90b380 0x55d89c929cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:42.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.331+0000 7fa485e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d89c90b380 0x55d89c929cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47968/0 (socket says 172.21.15.43:47968) 2024-03-28T22:42:42.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.331+0000 7fa485e00640 1 -- 172.21.15.43:0/2624731286 learned_addr learned my addr 172.21.15.43:0/2624731286 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:42.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.331+0000 7fa47fe00640 1 --2- 172.21.15.43:0/2624731286 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d89c90bca0 0x55d89c92a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:42.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.331+0000 7fa486800640 1 --2- 172.21.15.43:0/2624731286 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d89c914880 0x55d89c925270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:42.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.332+0000 7fa485e00640 1 -- 172.21.15.43:0/2624731286 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d89c914880 msgr2=0x55d89c925270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:42.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.332+0000 7fa485e00640 1 --2- 172.21.15.43:0/2624731286 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d89c914880 0x55d89c925270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.332+0000 7fa485e00640 1 -- 172.21.15.43:0/2624731286 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d89c90bca0 msgr2=0x55d89c92a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:42.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.332+0000 7fa485e00640 1 --2- 172.21.15.43:0/2624731286 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d89c90bca0 0x55d89c92a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.332+0000 7fa485e00640 1 -- 172.21.15.43:0/2624731286 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d89c925a70 con 0x55d89c90b380 2024-03-28T22:42:42.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.332+0000 7fa47fe00640 1 --2- 172.21.15.43:0/2624731286 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d89c90bca0 0x55d89c92a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:42:42.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.332+0000 7fa485e00640 1 --2- 172.21.15.43:0/2624731286 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d89c90b380 0x55d89c929cb0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fa47800e9c0 tx=0x7fa4780d6f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0db9bd4e03c81b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:42.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.332+0000 7fa46b400640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:42:42.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.332+0000 7fa46b400640 1 -- 172.21.15.43:0/2624731286 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4780dd040 con 0x55d89c90b380 2024-03-28T22:42:42.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.332+0000 7fa48f7ee740 1 -- 172.21.15.43:0/2624731286 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa470004a70 con 0x55d89c90b380 2024-03-28T22:42:42.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.332+0000 7fa46b400640 1 -- 172.21.15.43:0/2624731286 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fa4780d46d0 con 0x55d89c90b380 2024-03-28T22:42:42.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.332+0000 7fa46b400640 1 -- 172.21.15.43:0/2624731286 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa47809a840 con 0x55d89c90b380 2024-03-28T22:42:42.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.332+0000 7fa48f7ee740 10 client.27740 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:42.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.332+0000 7fa48f7ee740 1 -- 172.21.15.43:0/2624731286 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d89c993610 con 0x55d89c90b380 2024-03-28T22:42:42.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.333+0000 7fa46b400640 1 -- 172.21.15.43:0/2624731286 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa470004a70 con 0x55d89c90b380 2024-03-28T22:42:42.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.333+0000 7fa46b400640 1 -- 172.21.15.43:0/2624731286 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa4780cbe10 con 0x55d89c90b380 2024-03-28T22:42:42.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.333+0000 7fa48f7ee740 10 client.27740 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:42.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.333+0000 7fa48f7ee740 10 client.27740 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:42:42.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.333+0000 7fa48f7ee740 20 client.27740 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:42.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.333+0000 7fa48f7ee740 1 --2- 172.21.15.43:0/2624731286 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d89c993970 0x55d89c995d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:42.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.333+0000 7fa48f7ee740 4 client.27740 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:42:42.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.333+0000 7fa48f7ee740 1 -- 172.21.15.43:0/2624731286 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d89c87c590 con 0x55d89c993970 2024-03-28T22:42:42.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.333+0000 7fa47fe00640 1 --2- 172.21.15.43:0/2624731286 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d89c993970 0x55d89c995d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:42.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.334+0000 7fa47fe00640 1 --2- 172.21.15.43:0/2624731286 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d89c993970 0x55d89c995d60 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da99097000147b17 server_cookie=4ec0f9b38391a81 in_seq=0 out_seq=0 2024-03-28T22:42:42.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.334+0000 7fa46b400640 10 client.27740 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:42.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.334+0000 7fa46b400640 1 -- 172.21.15.43:0/2624731286 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d89c87c590 con 0x55d89c993970 2024-03-28T22:42:42.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.334+0000 7fa46b400640 10 client.27740 handle_command_reply: tid=0 2024-03-28T22:42:42.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.334+0000 7fa48f7ee740 1 client.27740 shutdown 2024-03-28T22:42:42.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.334+0000 7fa48f7ee740 1 -- 172.21.15.43:0/2624731286 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d89c90b380 msgr2=0x55d89c929cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:42.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.334+0000 7fa48f7ee740 1 --2- 172.21.15.43:0/2624731286 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d89c90b380 0x55d89c929cb0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fa47800e9c0 tx=0x7fa4780d6f10 comp rx=0 tx=0).stop 2024-03-28T22:42:42.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.334+0000 7fa48f7ee740 1 -- 172.21.15.43:0/2624731286 shutdown_connections 2024-03-28T22:42:42.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.334+0000 7fa48f7ee740 1 --2- 172.21.15.43:0/2624731286 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d89c914880 0x55d89c925270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.334+0000 7fa48f7ee740 1 --2- 172.21.15.43:0/2624731286 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d89c90bca0 0x55d89c92a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.334+0000 7fa48f7ee740 1 --2- 172.21.15.43:0/2624731286 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d89c993970 0x55d89c995d60 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.334+0000 7fa48f7ee740 1 --2- 172.21.15.43:0/2624731286 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d89c90b380 0x55d89c929cb0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.334+0000 7fa46b400640 0 client.27740 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:42.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.334+0000 7fa48f7ee740 1 -- 172.21.15.43:0/2624731286 >> 172.21.15.43:0/2624731286 conn(0x55d89c905ab0 msgr2=0x55d89c911810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:42.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.334+0000 7fa48f7ee740 1 -- 172.21.15.43:0/2624731286 shutdown_connections 2024-03-28T22:42:42.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.334+0000 7fa48f7ee740 1 -- 172.21.15.43:0/2624731286 wait complete. 2024-03-28T22:42:42.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.334+0000 7fa48f7ee740 20 client.27740 trim_cache size 0 max 16384 2024-03-28T22:42:42.339 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:42:42.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.335+0000 7fa48cc00640 1 -- 172.21.15.43:0/508204000 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa46c0a1f70 msgr2=0x7fa46c0a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:42.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.335+0000 7fa48cc00640 1 --2- 172.21.15.43:0/508204000 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa46c0a1f70 0x7fa46c0a4430 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa480005fd0 tx=0x7fa480009600 comp rx=0 tx=0).stop 2024-03-28T22:42:42.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.335+0000 7fa48cc00640 1 -- 172.21.15.43:0/508204000 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4881016e0 msgr2=0x7fa488197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:42.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.335+0000 7fa48cc00640 1 --2- 172.21.15.43:0/508204000 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4881016e0 0x7fa488197fc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa47800e9f0 tx=0x7fa47800eeb0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.336+0000 7fa48cc00640 1 -- 172.21.15.43:0/508204000 shutdown_connections 2024-03-28T22:42:42.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.336+0000 7fa48cc00640 1 --2- 172.21.15.43:0/508204000 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa46c0a1f70 0x7fa46c0a4430 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.336+0000 7fa48cc00640 1 --2- 172.21.15.43:0/508204000 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa4881076f0 0x7fa48819c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.336+0000 7fa48cc00640 1 --2- 172.21.15.43:0/508204000 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa4881020a0 0x7fa488198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.336+0000 7fa48cc00640 1 --2- 172.21.15.43:0/508204000 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa4881016e0 0x7fa488197fc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:42.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.336+0000 7fa48cc00640 1 -- 172.21.15.43:0/508204000 >> 172.21.15.43:0/508204000 conn(0x7fa488078b50 msgr2=0x7fa488104070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:42.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.336+0000 7fa48cc00640 1 -- 172.21.15.43:0/508204000 shutdown_connections 2024-03-28T22:42:42.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:42.336+0000 7fa48cc00640 1 -- 172.21.15.43:0/508204000 wait complete. 2024-03-28T22:42:42.367 INFO:tasks.workunit.client.0.smithi043.stdout: 1 377614 7.79 MB/sec execute 133 sec latency 108.383 ms 2024-03-28T22:42:43.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:42 smithi172 ceph-mon[36566]: pgmap v170: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 250 op/s 2024-03-28T22:42:43.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:42 smithi082 ceph-mon[36471]: pgmap v170: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 250 op/s 2024-03-28T22:42:43.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:42 smithi043 ceph-mon[31823]: pgmap v170: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 250 op/s 2024-03-28T22:42:43.351 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:42:43.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:42:43.351 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:42:43.367 INFO:tasks.workunit.client.0.smithi043.stdout: 1 379630 7.82 MB/sec execute 134 sec latency 13.665 ms 2024-03-28T22:42:43.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.600+0000 7f6525800640 1 -- 172.21.15.43:0/1906057711 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6520102990 msgr2=0x7f6520102d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:43.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.600+0000 7f6525800640 1 --2- 172.21.15.43:0/1906057711 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6520102990 0x7f6520102d70 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f6510009900 tx=0x7f651002bcd0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.600+0000 7f6525800640 1 -- 172.21.15.43:0/1906057711 shutdown_connections 2024-03-28T22:42:43.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.600+0000 7f6525800640 1 --2- 172.21.15.43:0/1906057711 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f652010bd60 0x7f652010e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.600+0000 7f6525800640 1 --2- 172.21.15.43:0/1906057711 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f65201032b0 0x7f652010b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.600+0000 7f6525800640 1 --2- 172.21.15.43:0/1906057711 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6520102990 0x7f6520102d70 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.600+0000 7f6525800640 1 -- 172.21.15.43:0/1906057711 >> 172.21.15.43:0/1906057711 conn(0x7f65200769d0 msgr2=0x7f6520076de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:43.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.600+0000 7f6525800640 1 -- 172.21.15.43:0/1906057711 shutdown_connections 2024-03-28T22:42:43.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.600+0000 7f6525800640 1 -- 172.21.15.43:0/1906057711 wait complete. 2024-03-28T22:42:43.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.601+0000 7f6525800640 1 Processor -- start 2024-03-28T22:42:43.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.601+0000 7f6525800640 1 -- start start 2024-03-28T22:42:43.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.601+0000 7f6525800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6520102990 0x7f652019a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.602+0000 7f6525800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f65201032b0 0x7f652019a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.602+0000 7f6525800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f652010bd60 0x7f652019eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.602+0000 7f6525800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6520111b40 con 0x7f6520102990 2024-03-28T22:42:43.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.602+0000 7f6525800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f65201119c0 con 0x7f65201032b0 2024-03-28T22:42:43.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.602+0000 7f6525800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f6520111cc0 con 0x7f652010bd60 2024-03-28T22:42:43.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.602+0000 7f651e000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f65201032b0 0x7f652019a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.602+0000 7f651e000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f65201032b0 0x7f652019a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47996/0 (socket says 172.21.15.43:47996) 2024-03-28T22:42:43.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.602+0000 7f651e000640 1 -- 172.21.15.43:0/1236234542 learned_addr learned my addr 172.21.15.43:0/1236234542 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:43.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.602+0000 7f651f400640 1 --2- 172.21.15.43:0/1236234542 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f652010bd60 0x7f652019eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.602+0000 7f651ea00640 1 --2- 172.21.15.43:0/1236234542 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6520102990 0x7f652019a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.602+0000 7f651e000640 1 -- 172.21.15.43:0/1236234542 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f652010bd60 msgr2=0x7f652019eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:43.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.602+0000 7f651e000640 1 --2- 172.21.15.43:0/1236234542 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f652010bd60 0x7f652019eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.602+0000 7f651e000640 1 -- 172.21.15.43:0/1236234542 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6520102990 msgr2=0x7f652019a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:43.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.602+0000 7f651e000640 1 --2- 172.21.15.43:0/1236234542 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6520102990 0x7f652019a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.602+0000 7f651e000640 1 -- 172.21.15.43:0/1236234542 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f652019f1d0 con 0x7f65201032b0 2024-03-28T22:42:43.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.603+0000 7f651e000640 1 --2- 172.21.15.43:0/1236234542 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f65201032b0 0x7f652019a720 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f650c00e9f0 tx=0x7f650c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:43.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.603+0000 7f6517400640 1 -- 172.21.15.43:0/1236234542 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650c00cd80 con 0x7f65201032b0 2024-03-28T22:42:43.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.603+0000 7f6525800640 1 -- 172.21.15.43:0/1236234542 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f652019f4c0 con 0x7f65201032b0 2024-03-28T22:42:43.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.603+0000 7f6517400640 1 -- 172.21.15.43:0/1236234542 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f650c010910 con 0x7f65201032b0 2024-03-28T22:42:43.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.603+0000 7f6517400640 1 -- 172.21.15.43:0/1236234542 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650c0188f0 con 0x7f65201032b0 2024-03-28T22:42:43.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.603+0000 7f6525800640 1 -- 172.21.15.43:0/1236234542 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65201a6d50 con 0x7f65201032b0 2024-03-28T22:42:43.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.606+0000 7f65285fc740 1 Processor -- start 2024-03-28T22:42:43.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.607+0000 7f65285fc740 1 -- start start 2024-03-28T22:42:43.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.607+0000 7f65285fc740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bab966f70 0x557bab967350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.607+0000 7f65285fc740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bab967890 0x557bab96ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.607+0000 7f65285fc740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bab970470 0x557bab972860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.607+0000 7f65285fc740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x557bab8cac40 con 0x557bab966f70 2024-03-28T22:42:43.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.607+0000 7f65285fc740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x557bab873c40 con 0x557bab970470 2024-03-28T22:42:43.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.607+0000 7f65285fc740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x557bab8c6ce0 con 0x557bab967890 2024-03-28T22:42:43.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.607+0000 7f6517400640 1 -- 172.21.15.43:0/1236234542 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f650c010ac0 con 0x7f65201032b0 2024-03-28T22:42:43.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.607+0000 7f651ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bab967890 0x557bab96ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.607+0000 7f651ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bab967890 0x557bab96ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57092/0 (socket says 172.21.15.43:57092) 2024-03-28T22:42:43.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.607+0000 7f651ea00640 1 -- 172.21.15.43:0/1985500542 learned_addr learned my addr 172.21.15.43:0/1985500542 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:43.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.607+0000 7f651f400640 1 --2- 172.21.15.43:0/1985500542 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bab966f70 0x557bab967350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.608+0000 7f651e000640 1 --2- 172.21.15.43:0/1985500542 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bab970470 0x557bab972860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.608+0000 7f651ea00640 1 -- 172.21.15.43:0/1985500542 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bab970470 msgr2=0x557bab972860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:43.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.608+0000 7f651ea00640 1 --2- 172.21.15.43:0/1985500542 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bab970470 0x557bab972860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.608+0000 7f651ea00640 1 -- 172.21.15.43:0/1985500542 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bab966f70 msgr2=0x557bab967350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:43.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.608+0000 7f651ea00640 1 --2- 172.21.15.43:0/1985500542 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bab966f70 0x557bab967350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.608+0000 7f651ea00640 1 -- 172.21.15.43:0/1985500542 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bab6a77e0 con 0x557bab967890 2024-03-28T22:42:43.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.608+0000 7f651f400640 1 --2- 172.21.15.43:0/1985500542 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bab966f70 0x557bab967350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:43.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.608+0000 7f651ea00640 1 --2- 172.21.15.43:0/1985500542 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bab967890 0x557bab96ff30 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6510000c00 tx=0x7f6510009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b2e9b18f4d866d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:43.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.608+0000 7f650be00640 1 -- 172.21.15.43:0/1985500542 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6510043070 con 0x557bab967890 2024-03-28T22:42:43.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.608+0000 7f650be00640 1 -- 172.21.15.43:0/1985500542 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f651002c6f0 con 0x557bab967890 2024-03-28T22:42:43.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.608+0000 7f650be00640 1 -- 172.21.15.43:0/1985500542 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6510004a60 con 0x557bab967890 2024-03-28T22:42:43.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.609+0000 7f6517400640 1 --2- 172.21.15.43:0/1236234542 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f65040a1ef0 0x7f65040a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.609+0000 7f6517400640 1 -- 172.21.15.43:0/1236234542 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f650c0ce920 con 0x7f65201032b0 2024-03-28T22:42:43.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.609+0000 7f65285fc740 1 -- 172.21.15.43:0/1985500542 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bab967890 msgr2=0x557bab96ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:43.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.609+0000 7f65285fc740 1 --2- 172.21.15.43:0/1985500542 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bab967890 0x557bab96ff30 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6510000c00 tx=0x7f6510009560 comp rx=0 tx=0).stop 2024-03-28T22:42:43.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.609+0000 7f651f400640 1 --2- 172.21.15.43:0/1236234542 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f65040a1ef0 0x7f65040a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.609+0000 7f65285fc740 1 -- 172.21.15.43:0/1985500542 shutdown_connections 2024-03-28T22:42:43.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.609+0000 7f65285fc740 1 --2- 172.21.15.43:0/1985500542 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bab970470 0x557bab972860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.609+0000 7f65285fc740 1 --2- 172.21.15.43:0/1985500542 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bab967890 0x557bab96ff30 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.609+0000 7f65285fc740 1 --2- 172.21.15.43:0/1985500542 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bab966f70 0x557bab967350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.609+0000 7f65285fc740 1 -- 172.21.15.43:0/1985500542 >> 172.21.15.43:0/1985500542 conn(0x557bab9616a0 msgr2=0x557bab961a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:43.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.610+0000 7f65285fc740 1 -- 172.21.15.43:0/1985500542 shutdown_connections 2024-03-28T22:42:43.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.610+0000 7f65285fc740 1 -- 172.21.15.43:0/1985500542 wait complete. 2024-03-28T22:42:43.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.610+0000 7f65285fc740 1 -- start start 2024-03-28T22:42:43.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.610+0000 7f651f400640 1 --2- 172.21.15.43:0/1236234542 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f65040a1ef0 0x7f65040a43b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6518009d40 tx=0x7f6518009500 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:43.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.610+0000 7f65285fc740 1 Processor -- start 2024-03-28T22:42:43.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.611+0000 7f65285fc740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bab966f70 0x557bab96d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.611+0000 7f65285fc740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bab967890 0x557bab96da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.611+0000 7f65285fc740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bab970470 0x557bab96df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.611+0000 7f65285fc740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x557bab973200 con 0x557bab970470 2024-03-28T22:42:43.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.611+0000 7f65285fc740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x557bab869250 con 0x557bab967890 2024-03-28T22:42:43.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.611+0000 7f65285fc740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x557bab973380 con 0x557bab966f70 2024-03-28T22:42:43.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.611+0000 7f65285fc740 10 client.0 _ll_register_callbacks cb 0x557bab88ae70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:43.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.611+0000 7f651ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bab966f70 0x557bab96d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.611+0000 7f651e000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bab967890 0x557bab96da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.611+0000 7f651ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bab966f70 0x557bab96d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57096/0 (socket says 172.21.15.43:57096) 2024-03-28T22:42:43.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.611+0000 7f651f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bab970470 0x557bab96df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.611+0000 7f651ea00640 1 -- 172.21.15.43:0/1003059221 learned_addr learned my addr 172.21.15.43:0/1003059221 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:43.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.611+0000 7f651e000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bab967890 0x557bab96da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48018/0 (socket says 172.21.15.43:48018) 2024-03-28T22:42:43.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.612+0000 7f651e000640 1 -- 172.21.15.43:0/1003059221 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bab966f70 msgr2=0x557bab96d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:43.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.612+0000 7f651e000640 1 --2- 172.21.15.43:0/1003059221 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bab966f70 0x557bab96d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.612+0000 7f651e000640 1 -- 172.21.15.43:0/1003059221 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bab970470 msgr2=0x557bab96df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:43.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.612+0000 7f651e000640 1 --2- 172.21.15.43:0/1003059221 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bab970470 0x557bab96df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.612+0000 7f651e000640 1 -- 172.21.15.43:0/1003059221 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bab707f70 con 0x557bab967890 2024-03-28T22:42:43.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.612+0000 7f651ea00640 1 --2- 172.21.15.43:0/1003059221 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bab966f70 0x557bab96d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:43.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.612+0000 7f651e000640 1 --2- 172.21.15.43:0/1003059221 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bab967890 0x557bab96da00 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f650c0cbc70 tx=0x7f650c09a920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b26dcb8342e3e2ba server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:43.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.612+0000 7f6508c00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:42:43.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.612+0000 7f6508c00640 1 -- 172.21.15.43:0/1003059221 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650c0cc100 con 0x557bab967890 2024-03-28T22:42:43.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.612+0000 7f65285fc740 1 -- 172.21.15.43:0/1003059221 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f651002c6f0 con 0x557bab967890 2024-03-28T22:42:43.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.612+0000 7f6508c00640 1 -- 172.21.15.43:0/1003059221 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f650c09a610 con 0x557bab967890 2024-03-28T22:42:43.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.612+0000 7f6508c00640 1 -- 172.21.15.43:0/1003059221 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f650c0a36b0 con 0x557bab967890 2024-03-28T22:42:43.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.612+0000 7f65285fc740 10 client.27752 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:43.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.612+0000 7f65285fc740 1 -- 172.21.15.43:0/1003059221 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557bab59f170 con 0x557bab967890 2024-03-28T22:42:43.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.613+0000 7f6508c00640 1 -- 172.21.15.43:0/1003059221 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f651002c6f0 con 0x557bab967890 2024-03-28T22:42:43.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.613+0000 7f6508c00640 1 -- 172.21.15.43:0/1003059221 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f650c099e80 con 0x557bab967890 2024-03-28T22:42:43.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.613+0000 7f65285fc740 10 client.27752 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:43.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.613+0000 7f65285fc740 10 client.27752 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:42:43.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.613+0000 7f65285fc740 20 client.27752 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:43.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.613+0000 7f65285fc740 1 --2- 172.21.15.43:0/1003059221 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557bab980860 0x557bab982c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.613+0000 7f65285fc740 4 client.27752 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:42:43.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.613+0000 7f65285fc740 1 -- 172.21.15.43:0/1003059221 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557bab8d8180 con 0x557bab980860 2024-03-28T22:42:43.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.613+0000 7f651ea00640 1 --2- 172.21.15.43:0/1003059221 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557bab980860 0x557bab982c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.614+0000 7f651ea00640 1 --2- 172.21.15.43:0/1003059221 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557bab980860 0x557bab982c80 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60b241512af4ffd8 server_cookie=827650a3c1794b20 in_seq=0 out_seq=0 2024-03-28T22:42:43.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.614+0000 7f6508c00640 10 client.27752 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:43.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.614+0000 7f6508c00640 1 -- 172.21.15.43:0/1003059221 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557bab8d8180 con 0x557bab980860 2024-03-28T22:42:43.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.614+0000 7f6508c00640 10 client.27752 handle_command_reply: tid=0 2024-03-28T22:42:43.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.615+0000 7f65285fc740 1 client.27752 shutdown 2024-03-28T22:42:43.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.615+0000 7f65285fc740 1 -- 172.21.15.43:0/1003059221 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bab967890 msgr2=0x557bab96da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:43.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.615+0000 7f65285fc740 1 --2- 172.21.15.43:0/1003059221 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bab967890 0x557bab96da00 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f650c0cbc70 tx=0x7f650c09a920 comp rx=0 tx=0).stop 2024-03-28T22:42:43.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.615+0000 7f65285fc740 1 -- 172.21.15.43:0/1003059221 shutdown_connections 2024-03-28T22:42:43.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.615+0000 7f65285fc740 1 --2- 172.21.15.43:0/1003059221 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557bab980860 0x557bab982c80 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.615+0000 7f65285fc740 1 --2- 172.21.15.43:0/1003059221 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bab967890 0x557bab96da00 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.615+0000 7f6508c00640 0 client.27752 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:43.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.615+0000 7f65285fc740 1 --2- 172.21.15.43:0/1003059221 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bab970470 0x557bab96df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.615+0000 7f65285fc740 1 --2- 172.21.15.43:0/1003059221 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bab966f70 0x557bab96d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.615+0000 7f65285fc740 1 -- 172.21.15.43:0/1003059221 >> 172.21.15.43:0/1003059221 conn(0x557bab9616a0 msgr2=0x557bab9633f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:43.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.615+0000 7f65285fc740 1 -- 172.21.15.43:0/1003059221 shutdown_connections 2024-03-28T22:42:43.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.615+0000 7f65285fc740 1 -- 172.21.15.43:0/1003059221 wait complete. 2024-03-28T22:42:43.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.615+0000 7f65285fc740 20 client.27752 trim_cache size 0 max 16384 2024-03-28T22:42:43.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.636+0000 7f65285fc740 1 Processor -- start 2024-03-28T22:42:43.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.636+0000 7f65285fc740 1 -- start start 2024-03-28T22:42:43.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.637+0000 7f65285fc740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bab966f70 0x557bab97c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.637+0000 7f65285fc740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bab967890 0x557bab988530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.637+0000 7f65285fc740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bab970470 0x557bab978620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.637+0000 7f65285fc740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f65201b0ce0 con 0x557bab970470 2024-03-28T22:42:43.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.637+0000 7f65285fc740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f65201b2820 con 0x557bab966f70 2024-03-28T22:42:43.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.637+0000 7f65285fc740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f65201b13b0 con 0x557bab967890 2024-03-28T22:42:43.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.637+0000 7f651ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bab966f70 0x557bab97c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.637+0000 7f651ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bab966f70 0x557bab97c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48022/0 (socket says 172.21.15.43:48022) 2024-03-28T22:42:43.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.637+0000 7f651ea00640 1 -- 172.21.15.43:0/1283310614 learned_addr learned my addr 172.21.15.43:0/1283310614 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:43.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.637+0000 7f651e000640 1 --2- 172.21.15.43:0/1283310614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bab967890 0x557bab988530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.637+0000 7f651f400640 1 --2- 172.21.15.43:0/1283310614 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bab970470 0x557bab978620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.638+0000 7f651ea00640 1 -- 172.21.15.43:0/1283310614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bab967890 msgr2=0x557bab988530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:43.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.638+0000 7f651ea00640 1 --2- 172.21.15.43:0/1283310614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bab967890 0x557bab988530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.638+0000 7f651ea00640 1 -- 172.21.15.43:0/1283310614 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bab970470 msgr2=0x557bab978620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:43.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.638+0000 7f651ea00640 1 --2- 172.21.15.43:0/1283310614 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bab970470 0x557bab978620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.638+0000 7f651ea00640 1 -- 172.21.15.43:0/1283310614 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bab96c470 con 0x557bab966f70 2024-03-28T22:42:43.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.638+0000 7f651f400640 1 --2- 172.21.15.43:0/1283310614 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bab970470 0x557bab978620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:42:43.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.638+0000 7f651ea00640 1 --2- 172.21.15.43:0/1283310614 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bab966f70 0x557bab97c000 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f6510005ec0 tx=0x7f651002bcd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6055021958dba34 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:43.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.638+0000 7f650b400640 1 -- 172.21.15.43:0/1283310614 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6510043040 con 0x557bab966f70 2024-03-28T22:42:43.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.638+0000 7f650b400640 1 -- 172.21.15.43:0/1283310614 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6510004ee0 con 0x557bab966f70 2024-03-28T22:42:43.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.638+0000 7f650b400640 1 -- 172.21.15.43:0/1283310614 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6510048670 con 0x557bab966f70 2024-03-28T22:42:43.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.639+0000 7f65285fc740 1 -- 172.21.15.43:0/1283310614 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bab966f70 msgr2=0x557bab97c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:43.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.639+0000 7f65285fc740 1 --2- 172.21.15.43:0/1283310614 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bab966f70 0x557bab97c000 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f6510005ec0 tx=0x7f651002bcd0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.639+0000 7f65285fc740 1 -- 172.21.15.43:0/1283310614 shutdown_connections 2024-03-28T22:42:43.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.639+0000 7f65285fc740 1 --2- 172.21.15.43:0/1283310614 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bab970470 0x557bab978620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.639+0000 7f65285fc740 1 --2- 172.21.15.43:0/1283310614 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bab967890 0x557bab988530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.639+0000 7f65285fc740 1 --2- 172.21.15.43:0/1283310614 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bab966f70 0x557bab97c000 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.639+0000 7f65285fc740 1 -- 172.21.15.43:0/1283310614 >> 172.21.15.43:0/1283310614 conn(0x557bab9616a0 msgr2=0x557bab96d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:43.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.639+0000 7f65285fc740 1 -- 172.21.15.43:0/1283310614 shutdown_connections 2024-03-28T22:42:43.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.639+0000 7f65285fc740 1 -- 172.21.15.43:0/1283310614 wait complete. 2024-03-28T22:42:43.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.640+0000 7f65285fc740 1 -- start start 2024-03-28T22:42:43.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.640+0000 7f65285fc740 1 Processor -- start 2024-03-28T22:42:43.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.640+0000 7f65285fc740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bab966f70 0x557bab978820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.641+0000 7f65285fc740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bab967890 0x557bab975600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.641+0000 7f65285fc740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bab970470 0x557bab975b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.641+0000 7f65285fc740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f65201b2150 con 0x557bab966f70 2024-03-28T22:42:43.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.641+0000 7f65285fc740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f65201b1a80 con 0x557bab970470 2024-03-28T22:42:43.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.641+0000 7f65285fc740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f65201aecc0 con 0x557bab967890 2024-03-28T22:42:43.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.641+0000 7f65285fc740 10 client.0 _ll_register_callbacks cb 0x7f650c00cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:43.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.641+0000 7f651e000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bab967890 0x557bab975600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.641+0000 7f651e000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bab967890 0x557bab975600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57114/0 (socket says 172.21.15.43:57114) 2024-03-28T22:42:43.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.641+0000 7f651e000640 1 -- 172.21.15.43:0/2603116544 learned_addr learned my addr 172.21.15.43:0/2603116544 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:43.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.641+0000 7f651f400640 1 --2- 172.21.15.43:0/2603116544 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bab970470 0x557bab975b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.641+0000 7f651ea00640 1 --2- 172.21.15.43:0/2603116544 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bab966f70 0x557bab978820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.641+0000 7f651e000640 1 -- 172.21.15.43:0/2603116544 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bab970470 msgr2=0x557bab975b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:43.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.641+0000 7f651e000640 1 --2- 172.21.15.43:0/2603116544 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bab970470 0x557bab975b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.641+0000 7f651e000640 1 -- 172.21.15.43:0/2603116544 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bab966f70 msgr2=0x557bab978820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:43.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.641+0000 7f651e000640 1 --2- 172.21.15.43:0/2603116544 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bab966f70 0x557bab978820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.641+0000 7f651e000640 1 -- 172.21.15.43:0/2603116544 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bab9761d0 con 0x557bab967890 2024-03-28T22:42:43.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.641+0000 7f651f400640 1 --2- 172.21.15.43:0/2603116544 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bab970470 0x557bab975b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:43.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.642+0000 7f651e000640 1 --2- 172.21.15.43:0/2603116544 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bab967890 0x557bab975600 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f650c09a290 tx=0x7f650c009700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d39bd3ece880f30e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:43.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.642+0000 7f6508c00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:42:43.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.642+0000 7f6508c00640 1 -- 172.21.15.43:0/2603116544 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650c097090 con 0x557bab967890 2024-03-28T22:42:43.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.642+0000 7f6508c00640 1 -- 172.21.15.43:0/2603116544 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f650c09a410 con 0x557bab967890 2024-03-28T22:42:43.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.642+0000 7f65285fc740 1 -- 172.21.15.43:0/2603116544 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6510004ee0 con 0x557bab967890 2024-03-28T22:42:43.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.642+0000 7f6508c00640 1 -- 172.21.15.43:0/2603116544 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f650c09ad90 con 0x557bab967890 2024-03-28T22:42:43.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.642+0000 7f65285fc740 10 client.27700 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:43.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.642+0000 7f65285fc740 1 -- 172.21.15.43:0/2603116544 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557bab9e7a70 con 0x557bab967890 2024-03-28T22:42:43.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.642+0000 7f6508c00640 1 -- 172.21.15.43:0/2603116544 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6510004ee0 con 0x557bab967890 2024-03-28T22:42:43.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.643+0000 7f6508c00640 1 -- 172.21.15.43:0/2603116544 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f650c09e070 con 0x557bab967890 2024-03-28T22:42:43.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.643+0000 7f65285fc740 10 client.27700 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:43.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.643+0000 7f65285fc740 10 client.27700 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:42:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.643+0000 7f65285fc740 20 client.27700 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.643+0000 7f65285fc740 1 --2- 172.21.15.43:0/2603116544 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557bab9e8160 0x557bab9ea580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.643+0000 7f65285fc740 4 client.27700 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:42:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.643+0000 7f65285fc740 1 -- 172.21.15.43:0/2603116544 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557bab8d24d0 con 0x557bab9e8160 2024-03-28T22:42:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.643+0000 7f651ea00640 1 --2- 172.21.15.43:0/2603116544 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557bab9e8160 0x557bab9ea580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.643+0000 7f651ea00640 1 --2- 172.21.15.43:0/2603116544 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557bab9e8160 0x557bab9ea580 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3bd1e8c2b4c9d207 server_cookie=3e6f835e9c7659ce in_seq=0 out_seq=0 2024-03-28T22:42:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.643+0000 7f6508c00640 10 client.27700 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.646+0000 7f6508c00640 1 -- 172.21.15.43:0/2603116544 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557bab8d24d0 con 0x557bab9e8160 2024-03-28T22:42:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.646+0000 7f6508c00640 10 client.27700 handle_command_reply: tid=0 2024-03-28T22:42:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.646+0000 7f65285fc740 1 client.27700 shutdown 2024-03-28T22:42:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.646+0000 7f65285fc740 1 -- 172.21.15.43:0/2603116544 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bab967890 msgr2=0x557bab975600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.646+0000 7f65285fc740 1 --2- 172.21.15.43:0/2603116544 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bab967890 0x557bab975600 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f650c09a290 tx=0x7f650c009700 comp rx=0 tx=0).stop 2024-03-28T22:42:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.646+0000 7f65285fc740 1 -- 172.21.15.43:0/2603116544 shutdown_connections 2024-03-28T22:42:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.646+0000 7f65285fc740 1 --2- 172.21.15.43:0/2603116544 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557bab9e8160 0x557bab9ea580 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.646+0000 7f65285fc740 1 --2- 172.21.15.43:0/2603116544 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bab970470 0x557bab975b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.646+0000 7f65285fc740 1 --2- 172.21.15.43:0/2603116544 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bab967890 0x557bab975600 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.646+0000 7f65285fc740 1 --2- 172.21.15.43:0/2603116544 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bab966f70 0x557bab978820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.646+0000 7f6508c00640 0 client.27700 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.646+0000 7f65285fc740 1 -- 172.21.15.43:0/2603116544 >> 172.21.15.43:0/2603116544 conn(0x557bab9616a0 msgr2=0x557bab96dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.647+0000 7f65285fc740 1 -- 172.21.15.43:0/2603116544 shutdown_connections 2024-03-28T22:42:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.647+0000 7f65285fc740 1 -- 172.21.15.43:0/2603116544 wait complete. 2024-03-28T22:42:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.647+0000 7f65285fc740 20 client.27700 trim_cache size 0 max 16384 2024-03-28T22:42:43.646 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:42:43.647 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-03-28T22:42:43.647 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "cccfa5d6-a983-4ec2-b769-d186e26bc665", 2024-03-28T22:42:43.647 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-03-28T22:42:43.647 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:42:43.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.647+0000 7f6525800640 1 -- 172.21.15.43:0/1236234542 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f65040a1ef0 msgr2=0x7f65040a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:43.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.647+0000 7f6525800640 1 --2- 172.21.15.43:0/1236234542 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f65040a1ef0 0x7f65040a43b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6518009d40 tx=0x7f6518009500 comp rx=0 tx=0).stop 2024-03-28T22:42:43.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.647+0000 7f6525800640 1 -- 172.21.15.43:0/1236234542 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f65201032b0 msgr2=0x7f652019a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:43.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.647+0000 7f6525800640 1 --2- 172.21.15.43:0/1236234542 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f65201032b0 0x7f652019a720 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f650c00e9f0 tx=0x7f650c00eeb0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.648+0000 7f6525800640 1 -- 172.21.15.43:0/1236234542 shutdown_connections 2024-03-28T22:42:43.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.648+0000 7f6525800640 1 --2- 172.21.15.43:0/1236234542 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f65040a1ef0 0x7f65040a43b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.648+0000 7f6525800640 1 --2- 172.21.15.43:0/1236234542 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f65201032b0 0x7f652019a720 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.648+0000 7f6525800640 1 --2- 172.21.15.43:0/1236234542 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f652010bd60 0x7f652019eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.648+0000 7f6525800640 1 --2- 172.21.15.43:0/1236234542 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6520102990 0x7f652019a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.648+0000 7f6525800640 1 -- 172.21.15.43:0/1236234542 >> 172.21.15.43:0/1236234542 conn(0x7f65200769d0 msgr2=0x7f65200ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:43.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.648+0000 7f6525800640 1 -- 172.21.15.43:0/1236234542 shutdown_connections 2024-03-28T22:42:43.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.648+0000 7f6525800640 1 -- 172.21.15.43:0/1236234542 wait complete. 2024-03-28T22:42:43.665 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:42:43.665 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:42:43.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.945+0000 7f3685400640 1 -- 172.21.15.43:0/359549859 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3680109900 msgr2=0x7f3680109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:43.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.945+0000 7f3685400640 1 --2- 172.21.15.43:0/359549859 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3680109900 0x7f3680109ce0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f3668009930 tx=0x7f366802bc30 comp rx=0 tx=0).stop 2024-03-28T22:42:43.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.945+0000 7f3685400640 1 -- 172.21.15.43:0/359549859 shutdown_connections 2024-03-28T22:42:43.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.945+0000 7f3685400640 1 --2- 172.21.15.43:0/359549859 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f36801042b0 0x7f36801107c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.945+0000 7f3685400640 1 --2- 172.21.15.43:0/359549859 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f36801038f0 0x7f3680103d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.945+0000 7f3685400640 1 --2- 172.21.15.43:0/359549859 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3680109900 0x7f3680109ce0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.945+0000 7f3685400640 1 -- 172.21.15.43:0/359549859 >> 172.21.15.43:0/359549859 conn(0x7f36800ff3a0 msgr2=0x7f36801017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:43.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.945+0000 7f3685400640 1 -- 172.21.15.43:0/359549859 shutdown_connections 2024-03-28T22:42:43.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.945+0000 7f3685400640 1 -- 172.21.15.43:0/359549859 wait complete. 2024-03-28T22:42:43.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.946+0000 7f3685400640 1 Processor -- start 2024-03-28T22:42:43.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.946+0000 7f3685400640 1 -- start start 2024-03-28T22:42:43.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.946+0000 7f3685400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f36801038f0 0x7f368019a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.946+0000 7f3685400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f36801042b0 0x7f368019a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.946+0000 7f3685400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3680109900 0x7f368019eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.946+0000 7f3685400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3680113d60 con 0x7f36801038f0 2024-03-28T22:42:43.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.947+0000 7f3685400640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f3680113be0 con 0x7f36801042b0 2024-03-28T22:42:43.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.947+0000 7f3685400640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f3680113ee0 con 0x7f3680109900 2024-03-28T22:42:43.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.947+0000 7f367e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f36801038f0 0x7f368019a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.947+0000 7f367de00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f36801042b0 0x7f368019a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.947+0000 7f367f200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3680109900 0x7f368019eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.947+0000 7f367de00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f36801042b0 0x7f368019a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48056/0 (socket says 172.21.15.43:48056) 2024-03-28T22:42:43.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.947+0000 7f367e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f36801038f0 0x7f368019a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60646/0 (socket says 172.21.15.43:60646) 2024-03-28T22:42:43.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.947+0000 7f367de00640 1 -- 172.21.15.43:0/3170543271 learned_addr learned my addr 172.21.15.43:0/3170543271 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:43.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.947+0000 7f367f200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3680109900 0x7f368019eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57128/0 (socket says 172.21.15.43:57128) 2024-03-28T22:42:43.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.947+0000 7f367f200640 1 -- 172.21.15.43:0/3170543271 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f36801042b0 msgr2=0x7f368019a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:43.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.947+0000 7f367f200640 1 --2- 172.21.15.43:0/3170543271 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f36801042b0 0x7f368019a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.947+0000 7f367f200640 1 -- 172.21.15.43:0/3170543271 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f36801038f0 msgr2=0x7f368019a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:43.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.947+0000 7f367f200640 1 --2- 172.21.15.43:0/3170543271 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f36801038f0 0x7f368019a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.947+0000 7f367f200640 1 -- 172.21.15.43:0/3170543271 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f368019f1f0 con 0x7f3680109900 2024-03-28T22:42:43.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.947+0000 7f367de00640 1 --2- 172.21.15.43:0/3170543271 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f36801042b0 0x7f368019a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:43.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.948+0000 7f367f200640 1 --2- 172.21.15.43:0/3170543271 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3680109900 0x7f368019eac0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f3678009fd0 tx=0x7f367800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:43.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.948+0000 7f3677400640 1 -- 172.21.15.43:0/3170543271 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f367800ce20 con 0x7f3680109900 2024-03-28T22:42:43.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.948+0000 7f3677400640 1 -- 172.21.15.43:0/3170543271 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3678005020 con 0x7f3680109900 2024-03-28T22:42:43.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.948+0000 7f3685400640 1 -- 172.21.15.43:0/3170543271 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f368019f4e0 con 0x7f3680109900 2024-03-28T22:42:43.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.948+0000 7f3677400640 1 -- 172.21.15.43:0/3170543271 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3678010720 con 0x7f3680109900 2024-03-28T22:42:43.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.948+0000 7f3685400640 1 -- 172.21.15.43:0/3170543271 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36801a6d70 con 0x7f3680109900 2024-03-28T22:42:43.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.951+0000 7f3687fea740 1 Processor -- start 2024-03-28T22:42:43.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.951+0000 7f3677400640 1 -- 172.21.15.43:0/3170543271 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f3678005540 con 0x7f3680109900 2024-03-28T22:42:43.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.952+0000 7f3687fea740 1 -- start start 2024-03-28T22:42:43.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.952+0000 7f3687fea740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d709d83bc0 0x55d709d82910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.952+0000 7f3687fea740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d709d82ee0 0x55d709d83360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.952+0000 7f3687fea740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d709d8c250 0x55d709d8e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.952+0000 7f3687fea740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d709ce7100 con 0x55d709d8c250 2024-03-28T22:42:43.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.952+0000 7f3687fea740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55d709c8fba0 con 0x55d709d82ee0 2024-03-28T22:42:43.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.952+0000 7f3687fea740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55d709ce31a0 con 0x55d709d83bc0 2024-03-28T22:42:43.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.952+0000 7f367de00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d709d83bc0 0x55d709d82910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.952+0000 7f367e800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d709d82ee0 0x55d709d83360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.952+0000 7f367de00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d709d83bc0 0x55d709d82910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57132/0 (socket says 172.21.15.43:57132) 2024-03-28T22:42:43.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.952+0000 7f367de00640 1 -- 172.21.15.43:0/2606409129 learned_addr learned my addr 172.21.15.43:0/2606409129 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:43.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.953+0000 7f367de00640 1 --2- 172.21.15.43:0/2606409129 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d709d8c250 0x55d709d8e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.953+0000 7f367de00640 1 -- 172.21.15.43:0/2606409129 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d709d82ee0 msgr2=0x55d709d83360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:43.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.953+0000 7f367de00640 1 --2- 172.21.15.43:0/2606409129 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d709d82ee0 0x55d709d83360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.953+0000 7f367de00640 1 -- 172.21.15.43:0/2606409129 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d709d8c250 msgr2=0x55d709d8e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:43.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.953+0000 7f367de00640 1 --2- 172.21.15.43:0/2606409129 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d709d8c250 0x55d709d8e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.953+0000 7f367de00640 1 -- 172.21.15.43:0/2606409129 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d709cc01a0 con 0x55d709d83bc0 2024-03-28T22:42:43.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.953+0000 7f3677400640 1 --2- 172.21.15.43:0/3170543271 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f36640a1f70 0x7f36640a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.953+0000 7f367e800640 1 --2- 172.21.15.43:0/2606409129 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d709d82ee0 0x55d709d83360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:43.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.953+0000 7f3677400640 1 -- 172.21.15.43:0/3170543271 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f36780cea80 con 0x7f3680109900 2024-03-28T22:42:43.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.953+0000 7f367de00640 1 --2- 172.21.15.43:0/2606409129 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d709d83bc0 0x55d709d82910 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f3670011ca0 tx=0x7f3670009c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aed2c695e48ee9e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:43.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.953+0000 7f367f200640 1 --2- 172.21.15.43:0/3170543271 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f36640a1f70 0x7f36640a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.953+0000 7f366f400640 1 -- 172.21.15.43:0/2606409129 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f367000fc50 con 0x55d709d83bc0 2024-03-28T22:42:43.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.954+0000 7f366f400640 1 -- 172.21.15.43:0/2606409129 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f367000fdf0 con 0x55d709d83bc0 2024-03-28T22:42:43.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.954+0000 7f366f400640 1 -- 172.21.15.43:0/2606409129 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f367000e6b0 con 0x55d709d83bc0 2024-03-28T22:42:43.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.954+0000 7f3687fea740 1 -- 172.21.15.43:0/2606409129 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d709d83bc0 msgr2=0x55d709d82910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:43.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.954+0000 7f367f200640 1 --2- 172.21.15.43:0/3170543271 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f36640a1f70 0x7f36640a4430 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3678097aa0 tx=0x7f3678005f70 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:43.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.954+0000 7f3687fea740 1 --2- 172.21.15.43:0/2606409129 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d709d83bc0 0x55d709d82910 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f3670011ca0 tx=0x7f3670009c60 comp rx=0 tx=0).stop 2024-03-28T22:42:43.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.954+0000 7f3687fea740 1 -- 172.21.15.43:0/2606409129 shutdown_connections 2024-03-28T22:42:43.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.954+0000 7f3687fea740 1 --2- 172.21.15.43:0/2606409129 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d709d82ee0 0x55d709d83360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.954+0000 7f3687fea740 1 --2- 172.21.15.43:0/2606409129 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d709d8c250 0x55d709d8e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.954+0000 7f3687fea740 1 --2- 172.21.15.43:0/2606409129 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d709d83bc0 0x55d709d82910 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.954+0000 7f3687fea740 1 -- 172.21.15.43:0/2606409129 >> 172.21.15.43:0/2606409129 conn(0x55d709d49a10 msgr2=0x55d709d7d930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:43.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.955+0000 7f3687fea740 1 -- 172.21.15.43:0/2606409129 shutdown_connections 2024-03-28T22:42:43.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.955+0000 7f3687fea740 1 -- 172.21.15.43:0/2606409129 wait complete. 2024-03-28T22:42:43.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.955+0000 7f3687fea740 1 -- start start 2024-03-28T22:42:43.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.955+0000 7f3687fea740 1 Processor -- start 2024-03-28T22:42:43.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.956+0000 7f3687fea740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d709d82ee0 0x55d709d89280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.956+0000 7f3687fea740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d709d83bc0 0x55d709d897c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.956+0000 7f3687fea740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d709d8c250 0x55d709d89d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.956+0000 7f3687fea740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d709d8f0c0 con 0x55d709d82ee0 2024-03-28T22:42:43.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.956+0000 7f3687fea740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55d709c85250 con 0x55d709d8c250 2024-03-28T22:42:43.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.956+0000 7f3687fea740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55d709d8f240 con 0x55d709d83bc0 2024-03-28T22:42:43.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.956+0000 7f3687fea740 10 client.0 _ll_register_callbacks cb 0x55d7099fabe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:43.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.956+0000 7f367e800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d709d83bc0 0x55d709d897c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.956+0000 7f367de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d709d82ee0 0x55d709d89280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.956+0000 7f367e800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d709d83bc0 0x55d709d897c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57146/0 (socket says 172.21.15.43:57146) 2024-03-28T22:42:43.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.956+0000 7f367e800640 1 -- 172.21.15.43:0/3008374775 learned_addr learned my addr 172.21.15.43:0/3008374775 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:43.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.956+0000 7f367de00640 1 --2- 172.21.15.43:0/3008374775 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d709d8c250 0x55d709d89d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.956+0000 7f367e800640 1 -- 172.21.15.43:0/3008374775 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d709d8c250 msgr2=0x55d709d89d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:43.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.956+0000 7f367e800640 1 --2- 172.21.15.43:0/3008374775 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d709d8c250 0x55d709d89d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.956+0000 7f367e800640 1 -- 172.21.15.43:0/3008374775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d709d82ee0 msgr2=0x55d709d89280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:43.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.956+0000 7f367e800640 1 --2- 172.21.15.43:0/3008374775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d709d82ee0 0x55d709d89280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.956+0000 7f367e800640 1 -- 172.21.15.43:0/3008374775 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d709ca1960 con 0x55d709d83bc0 2024-03-28T22:42:43.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.956+0000 7f367de00640 1 --2- 172.21.15.43:0/3008374775 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d709d8c250 0x55d709d89d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:42:43.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.957+0000 7f367e800640 1 --2- 172.21.15.43:0/3008374775 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d709d83bc0 0x55d709d897c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f366802c240 tx=0x7f366802c7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5674d065c469cec4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:43.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.957+0000 7f3663e00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:42:43.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.957+0000 7f3663e00640 1 -- 172.21.15.43:0/3008374775 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366802ca90 con 0x55d709d83bc0 2024-03-28T22:42:43.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.957+0000 7f3663e00640 1 -- 172.21.15.43:0/3008374775 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f366802cc30 con 0x55d709d83bc0 2024-03-28T22:42:43.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.957+0000 7f3687fea740 1 -- 172.21.15.43:0/3008374775 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f367000fdf0 con 0x55d709d83bc0 2024-03-28T22:42:43.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.958+0000 7f3663e00640 1 -- 172.21.15.43:0/3008374775 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f36680046c0 con 0x55d709d83bc0 2024-03-28T22:42:43.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.958+0000 7f3687fea740 10 client.27712 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:43.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.958+0000 7f3687fea740 1 -- 172.21.15.43:0/3008374775 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d709ac57d0 con 0x55d709d83bc0 2024-03-28T22:42:43.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.958+0000 7f3663e00640 1 -- 172.21.15.43:0/3008374775 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f367000fdf0 con 0x55d709d83bc0 2024-03-28T22:42:43.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.958+0000 7f3663e00640 1 -- 172.21.15.43:0/3008374775 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f36680035f0 con 0x55d709d83bc0 2024-03-28T22:42:43.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.958+0000 7f3687fea740 10 client.27712 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:43.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.958+0000 7f3687fea740 10 client.27712 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:42:43.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.958+0000 7f3687fea740 20 client.27712 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:43.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.958+0000 7f3687fea740 1 --2- 172.21.15.43:0/3008374775 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d709d9c720 0x55d709d9eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.958+0000 7f3687fea740 4 client.27712 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:42:43.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.958+0000 7f3687fea740 1 -- 172.21.15.43:0/3008374775 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d709cf4640 con 0x55d709d9c720 2024-03-28T22:42:43.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.958+0000 7f367de00640 1 --2- 172.21.15.43:0/3008374775 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d709d9c720 0x55d709d9eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.959+0000 7f367de00640 1 --2- 172.21.15.43:0/3008374775 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d709d9c720 0x55d709d9eb10 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f88ce6f1f2ec690a server_cookie=b19816a0a98d6f40 in_seq=0 out_seq=0 2024-03-28T22:42:43.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.959+0000 7f3663e00640 10 client.27712 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:43.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.961+0000 7f3663e00640 1 -- 172.21.15.43:0/3008374775 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d709cf4640 con 0x55d709d9c720 2024-03-28T22:42:43.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.961+0000 7f3663e00640 10 client.27712 handle_command_reply: tid=0 2024-03-28T22:42:43.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.961+0000 7f3687fea740 1 client.27712 shutdown 2024-03-28T22:42:43.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.961+0000 7f3687fea740 1 -- 172.21.15.43:0/3008374775 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d709d83bc0 msgr2=0x55d709d897c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:43.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.961+0000 7f3687fea740 1 --2- 172.21.15.43:0/3008374775 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d709d83bc0 0x55d709d897c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f366802c240 tx=0x7f366802c7e0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.961+0000 7f3687fea740 1 -- 172.21.15.43:0/3008374775 shutdown_connections 2024-03-28T22:42:43.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.961+0000 7f3687fea740 1 --2- 172.21.15.43:0/3008374775 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d709d9c720 0x55d709d9eb10 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.961+0000 7f3687fea740 1 --2- 172.21.15.43:0/3008374775 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d709d8c250 0x55d709d89d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.961+0000 7f3687fea740 1 --2- 172.21.15.43:0/3008374775 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d709d83bc0 0x55d709d897c0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.961+0000 7f3687fea740 1 --2- 172.21.15.43:0/3008374775 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d709d82ee0 0x55d709d89280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.961+0000 7f3687fea740 1 -- 172.21.15.43:0/3008374775 >> 172.21.15.43:0/3008374775 conn(0x55d709d49a10 msgr2=0x55d709d7f680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:43.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.961+0000 7f3663e00640 0 client.27712 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:43.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.961+0000 7f3687fea740 1 -- 172.21.15.43:0/3008374775 shutdown_connections 2024-03-28T22:42:43.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.962+0000 7f3687fea740 1 -- 172.21.15.43:0/3008374775 wait complete. 2024-03-28T22:42:43.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.962+0000 7f3687fea740 20 client.27712 trim_cache size 0 max 16384 2024-03-28T22:42:43.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.983+0000 7f3687fea740 1 Processor -- start 2024-03-28T22:42:43.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.983+0000 7f3687fea740 1 -- start start 2024-03-28T22:42:43.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.984+0000 7f3687fea740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d709d82ee0 0x55d709d8b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.984+0000 7f3687fea740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d709d83bc0 0x55d709d97ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.984+0000 7f3687fea740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d709d8c250 0x55d709da43e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.984+0000 7f3687fea740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f36801b0e90 con 0x55d709d83bc0 2024-03-28T22:42:43.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.984+0000 7f3687fea740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f36801b29d0 con 0x55d709d82ee0 2024-03-28T22:42:43.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.984+0000 7f3687fea740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f36801b1560 con 0x55d709d8c250 2024-03-28T22:42:43.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.984+0000 7f367de00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d709d82ee0 0x55d709d8b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.984+0000 7f367e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d709d83bc0 0x55d709d97ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.984+0000 7f367de00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d709d82ee0 0x55d709d8b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48086/0 (socket says 172.21.15.43:48086) 2024-03-28T22:42:43.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.984+0000 7f367e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d709d83bc0 0x55d709d97ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60682/0 (socket says 172.21.15.43:60682) 2024-03-28T22:42:43.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.984+0000 7f367de00640 1 -- 172.21.15.43:0/2736028990 learned_addr learned my addr 172.21.15.43:0/2736028990 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:43.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.984+0000 7f367de00640 1 --2- 172.21.15.43:0/2736028990 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d709d8c250 0x55d709da43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.984+0000 7f367e800640 1 -- 172.21.15.43:0/2736028990 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d709d8c250 msgr2=0x55d709da43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:43.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.984+0000 7f367e800640 1 --2- 172.21.15.43:0/2736028990 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d709d8c250 0x55d709da43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.984+0000 7f367e800640 1 -- 172.21.15.43:0/2736028990 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d709d82ee0 msgr2=0x55d709d8b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:43.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.984+0000 7f367e800640 1 --2- 172.21.15.43:0/2736028990 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d709d82ee0 0x55d709d8b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.984+0000 7f367e800640 1 -- 172.21.15.43:0/2736028990 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7099fb3f0 con 0x55d709d83bc0 2024-03-28T22:42:43.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.984+0000 7f367de00640 1 --2- 172.21.15.43:0/2736028990 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d709d82ee0 0x55d709d8b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:43.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.985+0000 7f367e800640 1 --2- 172.21.15.43:0/2736028990 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d709d83bc0 0x55d709d97ea0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f366802c170 tx=0x7f366802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a199011a01f44d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:43.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.985+0000 7f366ea00640 1 -- 172.21.15.43:0/2736028990 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3668003c00 con 0x55d709d83bc0 2024-03-28T22:42:43.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.985+0000 7f366ea00640 1 -- 172.21.15.43:0/2736028990 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3668003da0 con 0x55d709d83bc0 2024-03-28T22:42:43.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.985+0000 7f366ea00640 1 -- 172.21.15.43:0/2736028990 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3668005140 con 0x55d709d83bc0 2024-03-28T22:42:43.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.986+0000 7f3687fea740 1 -- 172.21.15.43:0/2736028990 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d709d83bc0 msgr2=0x55d709d97ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:43.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.986+0000 7f3687fea740 1 --2- 172.21.15.43:0/2736028990 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d709d83bc0 0x55d709d97ea0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f366802c170 tx=0x7f366802bc30 comp rx=0 tx=0).stop 2024-03-28T22:42:43.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.986+0000 7f3687fea740 1 -- 172.21.15.43:0/2736028990 shutdown_connections 2024-03-28T22:42:43.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.986+0000 7f3687fea740 1 --2- 172.21.15.43:0/2736028990 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d709d8c250 0x55d709da43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.986+0000 7f3687fea740 1 --2- 172.21.15.43:0/2736028990 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d709d83bc0 0x55d709d97ea0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.986+0000 7f3687fea740 1 --2- 172.21.15.43:0/2736028990 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d709d82ee0 0x55d709d8b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.986+0000 7f3687fea740 1 -- 172.21.15.43:0/2736028990 >> 172.21.15.43:0/2736028990 conn(0x55d709d49a10 msgr2=0x55d709d7ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:43.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.986+0000 7f3687fea740 1 -- 172.21.15.43:0/2736028990 shutdown_connections 2024-03-28T22:42:43.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.986+0000 7f3687fea740 1 -- 172.21.15.43:0/2736028990 wait complete. 2024-03-28T22:42:43.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.986+0000 7f3687fea740 1 -- start start 2024-03-28T22:42:43.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.987+0000 7f3687fea740 1 Processor -- start 2024-03-28T22:42:43.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.987+0000 7f3687fea740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d709d82ee0 0x55d709d8cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.987+0000 7f3687fea740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d709d83bc0 0x55d709d8d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.987+0000 7f3687fea740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d709d8c250 0x55d709d8da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.987+0000 7f3687fea740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f36801b2300 con 0x55d709d8c250 2024-03-28T22:42:43.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.987+0000 7f3687fea740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f36801b1c30 con 0x55d709d83bc0 2024-03-28T22:42:43.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.987+0000 7f3687fea740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f36801aee70 con 0x55d709d82ee0 2024-03-28T22:42:43.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.987+0000 7f3687fea740 10 client.0 _ll_register_callbacks cb 0x7f3668009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:43.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.987+0000 7f367e800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d709d83bc0 0x55d709d8d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.987+0000 7f367e800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d709d83bc0 0x55d709d8d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48100/0 (socket says 172.21.15.43:48100) 2024-03-28T22:42:43.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.987+0000 7f367e800640 1 -- 172.21.15.43:0/2699570298 learned_addr learned my addr 172.21.15.43:0/2699570298 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:43.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.987+0000 7f367de00640 1 --2- 172.21.15.43:0/2699570298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d709d8c250 0x55d709d8da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.988+0000 7f367e800640 1 -- 172.21.15.43:0/2699570298 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d709d82ee0 msgr2=0x55d709d8cfd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T22:42:43.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.988+0000 7f367de00640 1 --2- 172.21.15.43:0/2699570298 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d709d82ee0 0x55d709d8cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.988+0000 7f367e800640 1 --2- 172.21.15.43:0/2699570298 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d709d82ee0 0x55d709d8cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.988+0000 7f367e800640 1 -- 172.21.15.43:0/2699570298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d709d8c250 msgr2=0x55d709d8da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.988+0000 7f367e800640 1 --2- 172.21.15.43:0/2699570298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d709d8c250 0x55d709d8da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.988+0000 7f367e800640 1 -- 172.21.15.43:0/2699570298 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7099bb0f0 con 0x55d709d83bc0 2024-03-28T22:42:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.988+0000 7f367de00640 1 --2- 172.21.15.43:0/2699570298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d709d8c250 0x55d709d8da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:42:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.988+0000 7f367e800640 1 --2- 172.21.15.43:0/2699570298 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d709d83bc0 0x55d709d8d510 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f3668006010 tx=0x7f366802ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc3fe653dd5902b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.988+0000 7f3663e00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:42:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.988+0000 7f3663e00640 1 -- 172.21.15.43:0/2699570298 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3668003c00 con 0x55d709d83bc0 2024-03-28T22:42:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.988+0000 7f3663e00640 1 -- 172.21.15.43:0/2699570298 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f3668004c70 con 0x55d709d83bc0 2024-03-28T22:42:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.988+0000 7f3687fea740 1 -- 172.21.15.43:0/2699570298 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d709cee990 con 0x55d709d83bc0 2024-03-28T22:42:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.989+0000 7f3663e00640 1 -- 172.21.15.43:0/2699570298 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3668067050 con 0x55d709d83bc0 2024-03-28T22:42:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.989+0000 7f3687fea740 10 client.27776 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.989+0000 7f3687fea740 1 -- 172.21.15.43:0/2699570298 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d709e02da0 con 0x55d709d83bc0 2024-03-28T22:42:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.989+0000 7f3663e00640 1 -- 172.21.15.43:0/2699570298 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d709cee990 con 0x55d709d83bc0 2024-03-28T22:42:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.989+0000 7f3663e00640 1 -- 172.21.15.43:0/2699570298 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f366805e070 con 0x55d709d83bc0 2024-03-28T22:42:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.989+0000 7f3687fea740 10 client.27776 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.989+0000 7f3687fea740 10 client.27776 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:42:43.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.989+0000 7f3687fea740 20 client.27776 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:43.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.989+0000 7f3687fea740 1 --2- 172.21.15.43:0/2699570298 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d709e03350 0x55d709e05770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:43.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.989+0000 7f3687fea740 4 client.27776 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:42:43.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.989+0000 7f3687fea740 1 -- 172.21.15.43:0/2699570298 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d709c6c400 con 0x55d709e03350 2024-03-28T22:42:43.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.989+0000 7f367de00640 1 --2- 172.21.15.43:0/2699570298 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d709e03350 0x55d709e05770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.990+0000 7f367de00640 1 --2- 172.21.15.43:0/2699570298 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d709e03350 0x55d709e05770 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94e2e7e126418e4d server_cookie=deea1fa7bde393b5 in_seq=0 out_seq=0 2024-03-28T22:42:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.990+0000 7f3663e00640 10 client.27776 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.992+0000 7f3663e00640 1 -- 172.21.15.43:0/2699570298 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55d709c6c400 con 0x55d709e03350 2024-03-28T22:42:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.992+0000 7f3663e00640 10 client.27776 handle_command_reply: tid=0 2024-03-28T22:42:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.992+0000 7f3687fea740 1 client.27776 shutdown 2024-03-28T22:42:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.992+0000 7f3687fea740 1 -- 172.21.15.43:0/2699570298 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d709d83bc0 msgr2=0x55d709d8d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.992+0000 7f3687fea740 1 --2- 172.21.15.43:0/2699570298 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d709d83bc0 0x55d709d8d510 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f3668006010 tx=0x7f366802ccc0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.992+0000 7f3687fea740 1 -- 172.21.15.43:0/2699570298 shutdown_connections 2024-03-28T22:42:43.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.992+0000 7f3687fea740 1 --2- 172.21.15.43:0/2699570298 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d709e03350 0x55d709e05770 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.992+0000 7f3687fea740 1 --2- 172.21.15.43:0/2699570298 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d709d83bc0 0x55d709d8d510 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.992+0000 7f3687fea740 1 --2- 172.21.15.43:0/2699570298 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d709d8c250 0x55d709d8da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.992+0000 7f3687fea740 1 --2- 172.21.15.43:0/2699570298 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d709d82ee0 0x55d709d8cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.992+0000 7f3663e00640 0 client.27776 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:43.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.992+0000 7f3687fea740 1 -- 172.21.15.43:0/2699570298 >> 172.21.15.43:0/2699570298 conn(0x55d709d49a10 msgr2=0x55d709d7c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:43.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.992+0000 7f3687fea740 1 -- 172.21.15.43:0/2699570298 shutdown_connections 2024-03-28T22:42:43.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.992+0000 7f3687fea740 1 -- 172.21.15.43:0/2699570298 wait complete. 2024-03-28T22:42:43.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.992+0000 7f3687fea740 20 client.27776 trim_cache size 0 max 16384 2024-03-28T22:42:43.992 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:42:43.992 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:42:43.993 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-03-28T22:42:43.993 INFO:teuthology.orchestra.run.smithi043.stdout: "cccfa5d6-a983-4ec2-b769-d186e26bc665": { 2024-03-28T22:42:43.993 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-03-28T22:42:43.993 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "cccfa5d6-a983-4ec2-b769-d186e26bc665", 2024-03-28T22:42:43.993 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-03-28T22:42:43.993 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:42:43.993 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:42:43.993 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:42:43.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.993+0000 7f3685400640 1 -- 172.21.15.43:0/3170543271 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f36640a1f70 msgr2=0x7f36640a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:43.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.993+0000 7f3685400640 1 --2- 172.21.15.43:0/3170543271 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f36640a1f70 0x7f36640a4430 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3678097aa0 tx=0x7f3678005f70 comp rx=0 tx=0).stop 2024-03-28T22:42:43.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.993+0000 7f3685400640 1 -- 172.21.15.43:0/3170543271 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3680109900 msgr2=0x7f368019eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:43.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.993+0000 7f3685400640 1 --2- 172.21.15.43:0/3170543271 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3680109900 0x7f368019eac0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f3678009fd0 tx=0x7f367800ef10 comp rx=0 tx=0).stop 2024-03-28T22:42:43.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.993+0000 7f3685400640 1 -- 172.21.15.43:0/3170543271 shutdown_connections 2024-03-28T22:42:43.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.993+0000 7f3685400640 1 --2- 172.21.15.43:0/3170543271 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f36640a1f70 0x7f36640a4430 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.993+0000 7f3685400640 1 --2- 172.21.15.43:0/3170543271 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f36801042b0 0x7f368019a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.993+0000 7f3685400640 1 --2- 172.21.15.43:0/3170543271 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3680109900 0x7f368019eac0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.993+0000 7f3685400640 1 --2- 172.21.15.43:0/3170543271 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f36801038f0 0x7f368019a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:43.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.993+0000 7f3685400640 1 -- 172.21.15.43:0/3170543271 >> 172.21.15.43:0/3170543271 conn(0x7f36800ff3a0 msgr2=0x7f36800ffb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:43.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.993+0000 7f3685400640 1 -- 172.21.15.43:0/3170543271 shutdown_connections 2024-03-28T22:42:43.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:42:43.993+0000 7f3685400640 1 -- 172.21.15.43:0/3170543271 wait complete. 2024-03-28T22:42:44.009 INFO:tasks.cephfs.filesystem:scrub status for tag:cccfa5d6-a983-4ec2-b769-d186e26bc665 - {'path': '/', 'tag': 'cccfa5d6-a983-4ec2-b769-d186e26bc665', 'options': 'recursive,force'} 2024-03-28T22:42:44.368 INFO:tasks.workunit.client.0.smithi043.stdout: 1 380476 7.78 MB/sec execute 135 sec latency 21.028 ms 2024-03-28T22:42:45.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:44 smithi172 ceph-mon[36566]: pgmap v171: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 334 op/s 2024-03-28T22:42:45.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:44 smithi172 ceph-mon[36566]: scrub queued for path: / 2024-03-28T22:42:45.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:44 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:42:45.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:44 smithi172 ceph-mon[36566]: scrub summary: active paths [/] 2024-03-28T22:42:45.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:44 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:42:45.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:44 smithi082 ceph-mon[36471]: pgmap v171: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 334 op/s 2024-03-28T22:42:45.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:44 smithi082 ceph-mon[36471]: scrub queued for path: / 2024-03-28T22:42:45.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:44 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:42:45.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:44 smithi082 ceph-mon[36471]: scrub summary: active paths [/] 2024-03-28T22:42:45.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:44 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:42:45.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:44 smithi043 ceph-mon[31823]: pgmap v171: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 334 op/s 2024-03-28T22:42:45.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:44 smithi043 ceph-mon[31823]: scrub queued for path: / 2024-03-28T22:42:45.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:44 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:42:45.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:44 smithi043 ceph-mon[31823]: scrub summary: active paths [/] 2024-03-28T22:42:45.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:44 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:42:45.368 INFO:tasks.workunit.client.0.smithi043.stdout: 1 382320 7.80 MB/sec execute 136 sec latency 38.323 ms 2024-03-28T22:42:46.368 INFO:tasks.workunit.client.0.smithi043.stdout: 1 383496 7.80 MB/sec execute 137 sec latency 55.789 ms 2024-03-28T22:42:47.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:46 smithi172 ceph-mon[36566]: pgmap v172: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 258 op/s 2024-03-28T22:42:47.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:46 smithi082 ceph-mon[36471]: pgmap v172: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 258 op/s 2024-03-28T22:42:47.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:46 smithi043 ceph-mon[31823]: pgmap v172: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 258 op/s 2024-03-28T22:42:47.368 INFO:tasks.workunit.client.0.smithi043.stdout: 1 384623 7.78 MB/sec execute 138 sec latency 18.448 ms 2024-03-28T22:42:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.305+0000 7f996b200640 1 -- 172.21.15.43:0/2187819946 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f996c10dbf0 msgr2=0x7f996c10dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.305+0000 7f996b200640 1 --2- 172.21.15.43:0/2187819946 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f996c10dbf0 0x7f996c10dfd0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f995c009930 tx=0x7f995c02bc20 comp rx=0 tx=0).stop 2024-03-28T22:42:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.305+0000 7f996b200640 1 -- 172.21.15.43:0/2187819946 shutdown_connections 2024-03-28T22:42:48.304 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.305+0000 7f996b200640 1 --2- 172.21.15.43:0/2187819946 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f996c107c90 0x7f996c1080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.304 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.305+0000 7f996b200640 1 --2- 172.21.15.43:0/2187819946 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f996c108610 0x7f996c114a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.304 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.305+0000 7f996b200640 1 --2- 172.21.15.43:0/2187819946 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f996c10dbf0 0x7f996c10dfd0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.304 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.305+0000 7f996b200640 1 -- 172.21.15.43:0/2187819946 >> 172.21.15.43:0/2187819946 conn(0x7f996c077530 msgr2=0x7f996c106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:48.304 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.306+0000 7f996b200640 1 -- 172.21.15.43:0/2187819946 shutdown_connections 2024-03-28T22:42:48.304 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.306+0000 7f996b200640 1 -- 172.21.15.43:0/2187819946 wait complete. 2024-03-28T22:42:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.307+0000 7f996b200640 1 Processor -- start 2024-03-28T22:42:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.307+0000 7f996b200640 1 -- start start 2024-03-28T22:42:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.307+0000 7f996b200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f996c107c90 0x7f996c19e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.307+0000 7f996b200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f996c108610 0x7f996c19e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.307+0000 7f996b200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f996c10dbf0 0x7f996c1a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.307+0000 7f996b200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f996c117fb0 con 0x7f996c108610 2024-03-28T22:42:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.307+0000 7f996b200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f996c117e30 con 0x7f996c107c90 2024-03-28T22:42:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.307+0000 7f996b200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f996c118130 con 0x7f996c10dbf0 2024-03-28T22:42:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.307+0000 7f9969e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f996c107c90 0x7f996c19e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.308+0000 7f9969e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f996c107c90 0x7f996c19e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:49094/0 (socket says 172.21.15.43:49094) 2024-03-28T22:42:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.308+0000 7f9969e00640 1 -- 172.21.15.43:0/2008400222 learned_addr learned my addr 172.21.15.43:0/2008400222 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.308+0000 7f996a800640 1 --2- 172.21.15.43:0/2008400222 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f996c10dbf0 0x7f996c1a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.308+0000 7f9969400640 1 --2- 172.21.15.43:0/2008400222 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f996c108610 0x7f996c19e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.308+0000 7f9969e00640 1 -- 172.21.15.43:0/2008400222 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f996c10dbf0 msgr2=0x7f996c1a2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.308+0000 7f9969e00640 1 --2- 172.21.15.43:0/2008400222 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f996c10dbf0 0x7f996c1a2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.308+0000 7f9969e00640 1 -- 172.21.15.43:0/2008400222 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f996c108610 msgr2=0x7f996c19e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.308+0000 7f9969e00640 1 --2- 172.21.15.43:0/2008400222 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f996c108610 0x7f996c19e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.308+0000 7f9969e00640 1 -- 172.21.15.43:0/2008400222 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f996c1a3480 con 0x7f996c107c90 2024-03-28T22:42:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.308+0000 7f9969400640 1 --2- 172.21.15.43:0/2008400222 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f996c108610 0x7f996c19e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:42:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.308+0000 7f9969e00640 1 --2- 172.21.15.43:0/2008400222 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f996c107c90 0x7f996c19e490 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f995c000c00 tx=0x7f995c0043b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:48.308 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.309+0000 7f9962a00640 1 -- 172.21.15.43:0/2008400222 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f995c043020 con 0x7f996c107c90 2024-03-28T22:42:48.308 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.309+0000 7f9962a00640 1 -- 172.21.15.43:0/2008400222 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f995c0054e0 con 0x7f996c107c90 2024-03-28T22:42:48.308 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.309+0000 7f9962a00640 1 -- 172.21.15.43:0/2008400222 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f995c003b30 con 0x7f996c107c90 2024-03-28T22:42:48.308 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.309+0000 7f996b200640 1 -- 172.21.15.43:0/2008400222 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f996c1a3710 con 0x7f996c107c90 2024-03-28T22:42:48.308 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.309+0000 7f996b200640 1 -- 172.21.15.43:0/2008400222 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f996c1ab000 con 0x7f996c107c90 2024-03-28T22:42:48.309 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.312+0000 7f9962a00640 1 -- 172.21.15.43:0/2008400222 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f995c046020 con 0x7f996c107c90 2024-03-28T22:42:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.312+0000 7f9973e1b740 1 Processor -- start 2024-03-28T22:42:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.312+0000 7f9973e1b740 1 -- start start 2024-03-28T22:42:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.313+0000 7f9973e1b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559d32a12570 0x559d32a12950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.313+0000 7f9973e1b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559d32a12e90 0x559d32a1b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.313+0000 7f9973e1b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d32a1bc60 0x559d32a1e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.313+0000 7f9973e1b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x559d32975640 con 0x559d32a12570 2024-03-28T22:42:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.313+0000 7f9973e1b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x559d3291e810 con 0x559d32a1bc60 2024-03-28T22:42:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.313+0000 7f9973e1b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x559d329716e0 con 0x559d32a12e90 2024-03-28T22:42:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.313+0000 7f9969e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559d32a12e90 0x559d32a1b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.313+0000 7f9969400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d32a1bc60 0x559d32a1e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.313+0000 7f9969400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d32a1bc60 0x559d32a1e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:49098/0 (socket says 172.21.15.43:49098) 2024-03-28T22:42:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.313+0000 7f9969400640 1 -- 172.21.15.43:0/194203553 learned_addr learned my addr 172.21.15.43:0/194203553 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.313+0000 7f9969e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559d32a12e90 0x559d32a1b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:48608/0 (socket says 172.21.15.43:48608) 2024-03-28T22:42:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.313+0000 7f996a800640 1 --2- 172.21.15.43:0/194203553 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559d32a12570 0x559d32a12950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:48.313 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.313+0000 7f9969400640 1 -- 172.21.15.43:0/194203553 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559d32a12e90 msgr2=0x559d32a1b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:48.313 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.313+0000 7f9969400640 1 --2- 172.21.15.43:0/194203553 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559d32a12e90 0x559d32a1b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.313 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.313+0000 7f9969400640 1 -- 172.21.15.43:0/194203553 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559d32a12570 msgr2=0x559d32a12950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:48.313 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.313+0000 7f9969400640 1 --2- 172.21.15.43:0/194203553 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559d32a12570 0x559d32a12950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.313 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.313+0000 7f9969400640 1 -- 172.21.15.43:0/194203553 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d3268a210 con 0x559d32a1bc60 2024-03-28T22:42:48.313 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.314+0000 7f9969400640 1 --2- 172.21.15.43:0/194203553 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d32a1bc60 0x559d32a1e050 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f9958009ce0 tx=0x7f99580117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f7690c8445ee292 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:48.313 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.314+0000 7f9962a00640 1 --2- 172.21.15.43:0/2008400222 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f99500a1fb0 0x7f99500a4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:48.313 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.314+0000 7f9957400640 1 -- 172.21.15.43:0/194203553 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9958012450 con 0x559d32a1bc60 2024-03-28T22:42:48.313 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.314+0000 7f9957400640 1 -- 172.21.15.43:0/194203553 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9958012ad0 con 0x559d32a1bc60 2024-03-28T22:42:48.313 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.314+0000 7f9957400640 1 -- 172.21.15.43:0/194203553 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f995800fc40 con 0x559d32a1bc60 2024-03-28T22:42:48.313 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.314+0000 7f9962a00640 1 -- 172.21.15.43:0/2008400222 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f995c0f0460 con 0x7f996c107c90 2024-03-28T22:42:48.313 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.314+0000 7f996a800640 1 --2- 172.21.15.43:0/2008400222 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f99500a1fb0 0x7f99500a4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:48.317 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.315+0000 7f9973e1b740 1 -- 172.21.15.43:0/194203553 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d32a1bc60 msgr2=0x559d32a1e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:48.317 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.315+0000 7f9973e1b740 1 --2- 172.21.15.43:0/194203553 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d32a1bc60 0x559d32a1e050 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f9958009ce0 tx=0x7f99580117b0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.317 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.315+0000 7f9973e1b740 1 -- 172.21.15.43:0/194203553 shutdown_connections 2024-03-28T22:42:48.317 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.315+0000 7f9973e1b740 1 --2- 172.21.15.43:0/194203553 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d32a1bc60 0x559d32a1e050 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.317 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.315+0000 7f9973e1b740 1 --2- 172.21.15.43:0/194203553 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559d32a12e90 0x559d32a1b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.317 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.315+0000 7f9973e1b740 1 --2- 172.21.15.43:0/194203553 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559d32a12570 0x559d32a12950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.317 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.315+0000 7f9973e1b740 1 -- 172.21.15.43:0/194203553 >> 172.21.15.43:0/194203553 conn(0x559d32a0ce90 msgr2=0x559d32a0d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:48.317 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.315+0000 7f9973e1b740 1 -- 172.21.15.43:0/194203553 shutdown_connections 2024-03-28T22:42:48.317 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.315+0000 7f996a800640 1 --2- 172.21.15.43:0/2008400222 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f99500a1fb0 0x7f99500a4470 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9964005fd0 tx=0x7f99640096a0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.315+0000 7f9973e1b740 1 -- 172.21.15.43:0/194203553 wait complete. 2024-03-28T22:42:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.316+0000 7f9973e1b740 1 -- start start 2024-03-28T22:42:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.316+0000 7f9973e1b740 1 Processor -- start 2024-03-28T22:42:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.317+0000 7f9973e1b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559d32a12570 0x559d32a18c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.317+0000 7f9973e1b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d32a12e90 0x559d32a19160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.317+0000 7f9973e1b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559d32a1bc60 0x559d32a196a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.317+0000 7f9973e1b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x559d32a1e890 con 0x559d32a1bc60 2024-03-28T22:42:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.317+0000 7f9973e1b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x559d32913d00 con 0x559d32a12e90 2024-03-28T22:42:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.317+0000 7f9973e1b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x559d32a1ea10 con 0x559d32a12570 2024-03-28T22:42:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.317+0000 7f9973e1b740 10 client.0 _ll_register_callbacks cb 0x559d328fe490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.317+0000 7f9969400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d32a12e90 0x559d32a19160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.317+0000 7f9969e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559d32a12570 0x559d32a18c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.317+0000 7f996a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559d32a1bc60 0x559d32a196a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:48.318 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.317+0000 7f9969e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559d32a12570 0x559d32a18c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:48624/0 (socket says 172.21.15.43:48624) 2024-03-28T22:42:48.319 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.317+0000 7f9969e00640 1 -- 172.21.15.43:0/2081275108 learned_addr learned my addr 172.21.15.43:0/2081275108 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:48.319 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.317+0000 7f9969400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d32a12e90 0x559d32a19160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:49112/0 (socket says 172.21.15.43:49112) 2024-03-28T22:42:48.319 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.317+0000 7f9969e00640 1 -- 172.21.15.43:0/2081275108 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d32a12e90 msgr2=0x559d32a19160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:48.319 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.317+0000 7f9969e00640 1 --2- 172.21.15.43:0/2081275108 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d32a12e90 0x559d32a19160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.319 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.317+0000 7f9969e00640 1 -- 172.21.15.43:0/2081275108 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559d32a1bc60 msgr2=0x559d32a196a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:48.319 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.317+0000 7f9969e00640 1 --2- 172.21.15.43:0/2081275108 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559d32a1bc60 0x559d32a196a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.319 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.317+0000 7f9969e00640 1 -- 172.21.15.43:0/2081275108 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d32756620 con 0x559d32a12570 2024-03-28T22:42:48.319 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.318+0000 7f9969e00640 1 --2- 172.21.15.43:0/2081275108 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559d32a12570 0x559d32a18c20 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f995c009900 tx=0x7f995c009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65367bae230b7a5c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:48.319 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.318+0000 7f994be00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:42:48.319 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.318+0000 7f994be00640 1 -- 172.21.15.43:0/2081275108 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f995c0bc700 con 0x559d32a12570 2024-03-28T22:42:48.319 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.318+0000 7f994be00640 1 -- 172.21.15.43:0/2081275108 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f995c0eda90 con 0x559d32a12570 2024-03-28T22:42:48.319 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.318+0000 7f9973e1b740 1 -- 172.21.15.43:0/2081275108 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9958012ad0 con 0x559d32a12570 2024-03-28T22:42:48.319 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.318+0000 7f994be00640 1 -- 172.21.15.43:0/2081275108 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f995c0bc520 con 0x559d32a12570 2024-03-28T22:42:48.319 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.318+0000 7f9973e1b740 10 client.27727 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:48.320 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.318+0000 7f9973e1b740 1 -- 172.21.15.43:0/2081275108 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559d32a1e8f0 con 0x559d32a12570 2024-03-28T22:42:48.320 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.318+0000 7f994be00640 1 -- 172.21.15.43:0/2081275108 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9958012ad0 con 0x559d32a12570 2024-03-28T22:42:48.320 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.318+0000 7f994be00640 1 -- 172.21.15.43:0/2081275108 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f995c0b9230 con 0x559d32a12570 2024-03-28T22:42:48.320 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.319+0000 7f9973e1b740 10 client.27727 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:48.320 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.319+0000 7f9973e1b740 10 client.27727 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:42:48.320 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.319+0000 7f9973e1b740 20 client.27727 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:48.320 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.319+0000 7f9973e1b740 1 --2- 172.21.15.43:0/2081275108 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559d32a2bbd0 0x559d32a2dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:48.320 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.319+0000 7f9973e1b740 4 client.27727 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:42:48.320 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.319+0000 7f9973e1b740 1 -- 172.21.15.43:0/2081275108 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559d32989030 con 0x559d32a2bbd0 2024-03-28T22:42:48.320 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.319+0000 7f9969400640 1 --2- 172.21.15.43:0/2081275108 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559d32a2bbd0 0x559d32a2dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:48.320 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.320+0000 7f9969400640 1 --2- 172.21.15.43:0/2081275108 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559d32a2bbd0 0x559d32a2dfc0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fca94d382b781123 server_cookie=6ed78d2257892d6a in_seq=0 out_seq=0 2024-03-28T22:42:48.320 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.320+0000 7f994be00640 10 client.27727 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:48.322 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.320+0000 7f994be00640 1 -- 172.21.15.43:0/2081275108 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559d32989030 con 0x559d32a2bbd0 2024-03-28T22:42:48.322 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.320+0000 7f994be00640 10 client.27727 handle_command_reply: tid=0 2024-03-28T22:42:48.322 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.320+0000 7f9973e1b740 1 client.27727 shutdown 2024-03-28T22:42:48.322 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.320+0000 7f9973e1b740 1 -- 172.21.15.43:0/2081275108 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559d32a12570 msgr2=0x559d32a18c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:48.322 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.320+0000 7f9973e1b740 1 --2- 172.21.15.43:0/2081275108 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559d32a12570 0x559d32a18c20 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f995c009900 tx=0x7f995c009260 comp rx=0 tx=0).stop 2024-03-28T22:42:48.322 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.321+0000 7f9973e1b740 1 -- 172.21.15.43:0/2081275108 shutdown_connections 2024-03-28T22:42:48.322 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.321+0000 7f9973e1b740 1 --2- 172.21.15.43:0/2081275108 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559d32a2bbd0 0x559d32a2dfc0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.322 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.321+0000 7f9973e1b740 1 --2- 172.21.15.43:0/2081275108 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d32a12e90 0x559d32a19160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.322 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.321+0000 7f9973e1b740 1 --2- 172.21.15.43:0/2081275108 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559d32a1bc60 0x559d32a196a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.322 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.321+0000 7f9973e1b740 1 --2- 172.21.15.43:0/2081275108 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559d32a12570 0x559d32a18c20 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.322 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.321+0000 7f994be00640 0 client.27727 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:48.322 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.321+0000 7f9973e1b740 1 -- 172.21.15.43:0/2081275108 >> 172.21.15.43:0/2081275108 conn(0x559d32a0ce90 msgr2=0x559d32a0ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:48.322 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.321+0000 7f9973e1b740 1 -- 172.21.15.43:0/2081275108 shutdown_connections 2024-03-28T22:42:48.323 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.321+0000 7f9973e1b740 1 -- 172.21.15.43:0/2081275108 wait complete. 2024-03-28T22:42:48.323 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.321+0000 7f9973e1b740 20 client.27727 trim_cache size 0 max 16384 2024-03-28T22:42:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.338+0000 7f9973e1b740 1 Processor -- start 2024-03-28T22:42:48.336 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.338+0000 7f9973e1b740 1 -- start start 2024-03-28T22:42:48.338 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.339+0000 7f9973e1b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559d32a12570 0x559d32a1b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:48.338 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.339+0000 7f9973e1b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d32a12e90 0x559d32a27630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.339+0000 7f9973e1b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559d32a1bc60 0x559d32a338d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.339+0000 7f9973e1b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f996c1b5070 con 0x559d32a1bc60 2024-03-28T22:42:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.339+0000 7f9973e1b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f996c1b6bb0 con 0x559d32a12e90 2024-03-28T22:42:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.339+0000 7f9973e1b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f996c1b5740 con 0x559d32a12570 2024-03-28T22:42:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.339+0000 7f9969e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559d32a12570 0x559d32a1b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.339+0000 7f9969e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559d32a12570 0x559d32a1b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:48636/0 (socket says 172.21.15.43:48636) 2024-03-28T22:42:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.339+0000 7f9969e00640 1 -- 172.21.15.43:0/2116500671 learned_addr learned my addr 172.21.15.43:0/2116500671 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.339+0000 7f9969400640 1 --2- 172.21.15.43:0/2116500671 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d32a12e90 0x559d32a27630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.339+0000 7f996a800640 1 --2- 172.21.15.43:0/2116500671 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559d32a1bc60 0x559d32a338d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.339+0000 7f9969e00640 1 -- 172.21.15.43:0/2116500671 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d32a12e90 msgr2=0x559d32a27630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.339+0000 7f9969e00640 1 --2- 172.21.15.43:0/2116500671 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d32a12e90 0x559d32a27630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.339+0000 7f9969e00640 1 -- 172.21.15.43:0/2116500671 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559d32a1bc60 msgr2=0x559d32a338d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.339+0000 7f9969e00640 1 --2- 172.21.15.43:0/2116500671 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559d32a1bc60 0x559d32a338d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.339+0000 7f9969e00640 1 -- 172.21.15.43:0/2116500671 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d32a0b310 con 0x559d32a12570 2024-03-28T22:42:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.339+0000 7f9969400640 1 --2- 172.21.15.43:0/2116500671 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d32a12e90 0x559d32a27630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:42:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.340+0000 7f9969e00640 1 --2- 172.21.15.43:0/2116500671 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559d32a12570 0x559d32a1b2e0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f995c0bc8c0 tx=0x7f995c0035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8231457b58ce57a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.340+0000 7f9956a00640 1 -- 172.21.15.43:0/2116500671 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f995c005690 con 0x559d32a12570 2024-03-28T22:42:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.340+0000 7f9956a00640 1 -- 172.21.15.43:0/2116500671 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f995c0bd2c0 con 0x559d32a12570 2024-03-28T22:42:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.340+0000 7f9956a00640 1 -- 172.21.15.43:0/2116500671 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f995c0ed9d0 con 0x559d32a12570 2024-03-28T22:42:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.341+0000 7f9973e1b740 1 -- 172.21.15.43:0/2116500671 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559d32a12570 msgr2=0x559d32a1b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.341+0000 7f9973e1b740 1 --2- 172.21.15.43:0/2116500671 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559d32a12570 0x559d32a1b2e0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f995c0bc8c0 tx=0x7f995c0035f0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.341+0000 7f9973e1b740 1 -- 172.21.15.43:0/2116500671 shutdown_connections 2024-03-28T22:42:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.341+0000 7f9973e1b740 1 --2- 172.21.15.43:0/2116500671 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d32a12e90 0x559d32a27630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.341+0000 7f9973e1b740 1 --2- 172.21.15.43:0/2116500671 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559d32a1bc60 0x559d32a338d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.341+0000 7f9973e1b740 1 --2- 172.21.15.43:0/2116500671 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559d32a12570 0x559d32a1b2e0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.341+0000 7f9973e1b740 1 -- 172.21.15.43:0/2116500671 >> 172.21.15.43:0/2116500671 conn(0x559d32a0ce90 msgr2=0x559d32a07a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.341+0000 7f9973e1b740 1 -- 172.21.15.43:0/2116500671 shutdown_connections 2024-03-28T22:42:48.340 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.341+0000 7f9973e1b740 1 -- 172.21.15.43:0/2116500671 wait complete. 2024-03-28T22:42:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.341+0000 7f9973e1b740 1 -- start start 2024-03-28T22:42:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.342+0000 7f9973e1b740 1 Processor -- start 2024-03-28T22:42:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.342+0000 7f9973e1b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d32a12570 0x559d32a156b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.342+0000 7f9973e1b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559d32a12e90 0x559d32a15bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.342+0000 7f9973e1b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559d32a1bc60 0x559d32a16130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.342+0000 7f9973e1b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f996c1b64e0 con 0x559d32a12e90 2024-03-28T22:42:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.342+0000 7f9973e1b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f996c1b5e10 con 0x559d32a12570 2024-03-28T22:42:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.342+0000 7f9973e1b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f996c1b3050 con 0x559d32a1bc60 2024-03-28T22:42:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.342+0000 7f9973e1b740 10 client.0 _ll_register_callbacks cb 0x7f995c004560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:42:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.342+0000 7f9969e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d32a12570 0x559d32a156b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.342+0000 7f9969e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d32a12570 0x559d32a156b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:49120/0 (socket says 172.21.15.43:49120) 2024-03-28T22:42:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.342+0000 7f9969e00640 1 -- 172.21.15.43:0/1891730807 learned_addr learned my addr 172.21.15.43:0/1891730807 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:42:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.343+0000 7f9969400640 1 --2- 172.21.15.43:0/1891730807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559d32a12e90 0x559d32a15bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.343+0000 7f996a800640 1 --2- 172.21.15.43:0/1891730807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559d32a1bc60 0x559d32a16130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.343+0000 7f9969e00640 1 -- 172.21.15.43:0/1891730807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559d32a1bc60 msgr2=0x559d32a16130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.343+0000 7f9969e00640 1 --2- 172.21.15.43:0/1891730807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559d32a1bc60 0x559d32a16130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.343+0000 7f9969e00640 1 -- 172.21.15.43:0/1891730807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559d32a12e90 msgr2=0x559d32a15bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:42:48.345 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.343+0000 7f9969e00640 1 --2- 172.21.15.43:0/1891730807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559d32a12e90 0x559d32a15bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.345 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.343+0000 7f9969e00640 1 -- 172.21.15.43:0/1891730807 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d32a90f00 con 0x559d32a12570 2024-03-28T22:42:48.345 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.343+0000 7f996a800640 1 --2- 172.21.15.43:0/1891730807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559d32a1bc60 0x559d32a16130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:42:48.345 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.343+0000 7f9969e00640 1 --2- 172.21.15.43:0/1891730807 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d32a12570 0x559d32a156b0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f995c0b9020 tx=0x7f995c0b94b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f24c3dc90db78f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:42:48.345 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.343+0000 7f994be00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:42:48.345 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.343+0000 7f994be00640 1 -- 172.21.15.43:0/1891730807 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f995c0edbd0 con 0x559d32a12570 2024-03-28T22:42:48.345 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.343+0000 7f994be00640 1 -- 172.21.15.43:0/1891730807 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f995c0c4ed0 con 0x559d32a12570 2024-03-28T22:42:48.345 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.343+0000 7f9973e1b740 1 -- 172.21.15.43:0/1891730807 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f995c0bd2c0 con 0x559d32a12570 2024-03-28T22:42:48.345 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.343+0000 7f994be00640 1 -- 172.21.15.43:0/1891730807 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f995c0c0d10 con 0x559d32a12570 2024-03-28T22:42:48.345 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.344+0000 7f9973e1b740 10 client.27794 fetch_fsmap learned FSMap version 32 2024-03-28T22:42:48.345 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.344+0000 7f9973e1b740 1 -- 172.21.15.43:0/1891730807 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559d32a90f00 con 0x559d32a12570 2024-03-28T22:42:48.345 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.344+0000 7f994be00640 1 -- 172.21.15.43:0/1891730807 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f995c0bd2c0 con 0x559d32a12570 2024-03-28T22:42:48.345 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.344+0000 7f994be00640 1 -- 172.21.15.43:0/1891730807 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f995c0bba80 con 0x559d32a12570 2024-03-28T22:42:48.346 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.344+0000 7f9973e1b740 10 client.27794 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:42:48.346 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.344+0000 7f9973e1b740 10 client.27794 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:42:48.346 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.344+0000 7f9973e1b740 20 client.27794 populate_metadata read hostname 'smithi043' 2024-03-28T22:42:48.346 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.344+0000 7f9973e1b740 1 --2- 172.21.15.43:0/1891730807 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559d32a92390 0x559d32a94780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:42:48.346 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.344+0000 7f9973e1b740 4 client.27794 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T22:42:48.346 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.344+0000 7f9973e1b740 1 -- 172.21.15.43:0/1891730807 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559d32982b80 con 0x559d32a92390 2024-03-28T22:42:48.346 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.344+0000 7f9969400640 1 --2- 172.21.15.43:0/1891730807 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559d32a92390 0x559d32a94780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:42:48.346 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.345+0000 7f9969400640 1 --2- 172.21.15.43:0/1891730807 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559d32a92390 0x559d32a94780 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ed1b51bb84e1c8c server_cookie=20bf3811d0f7ce72 in_seq=0 out_seq=0 2024-03-28T22:42:48.346 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.345+0000 7f994be00640 10 client.27794 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:48.368 INFO:tasks.workunit.client.0.smithi043.stdout: 1 386784 7.82 MB/sec execute 139 sec latency 24.048 ms 2024-03-28T22:42:48.547 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.549+0000 7f994be00640 1 -- 172.21.15.43:0/1891730807 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559d32982b80 con 0x559d32a92390 2024-03-28T22:42:48.547 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.549+0000 7f994be00640 10 client.27794 handle_command_reply: tid=0 2024-03-28T22:42:48.547 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.549+0000 7f9973e1b740 1 client.27794 shutdown 2024-03-28T22:42:48.547 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.549+0000 7f9973e1b740 1 -- 172.21.15.43:0/1891730807 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d32a12570 msgr2=0x559d32a156b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:48.547 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.549+0000 7f9973e1b740 1 --2- 172.21.15.43:0/1891730807 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d32a12570 0x559d32a156b0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f995c0b9020 tx=0x7f995c0b94b0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.548 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.549+0000 7f9973e1b740 1 -- 172.21.15.43:0/1891730807 shutdown_connections 2024-03-28T22:42:48.548 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.549+0000 7f9973e1b740 1 --2- 172.21.15.43:0/1891730807 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559d32a1bc60 0x559d32a16130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.548 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.550+0000 7f9973e1b740 1 --2- 172.21.15.43:0/1891730807 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559d32a12e90 0x559d32a15bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.548 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.550+0000 7f9973e1b740 1 --2- 172.21.15.43:0/1891730807 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559d32a92390 0x559d32a94780 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.548 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.550+0000 7f9973e1b740 1 --2- 172.21.15.43:0/1891730807 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d32a12570 0x559d32a156b0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.548 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.550+0000 7f9973e1b740 1 -- 172.21.15.43:0/1891730807 >> 172.21.15.43:0/1891730807 conn(0x559d32a0ce90 msgr2=0x559d32a0b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:48.548 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.550+0000 7f994be00640 0 client.27794 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:42:48.548 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.550+0000 7f9973e1b740 1 -- 172.21.15.43:0/1891730807 shutdown_connections 2024-03-28T22:42:48.548 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.550+0000 7f9973e1b740 1 -- 172.21.15.43:0/1891730807 wait complete. 2024-03-28T22:42:48.548 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.550+0000 7f9973e1b740 20 client.27794 trim_cache size 0 max 16384 2024-03-28T22:42:48.549 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:{ 2024-03-28T22:42:48.549 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "message": "", 2024-03-28T22:42:48.549 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "return_code": 0 2024-03-28T22:42:48.549 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:} 2024-03-28T22:42:48.550 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.551+0000 7f996b200640 1 -- 172.21.15.43:0/2008400222 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f99500a1fb0 msgr2=0x7f99500a4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:48.550 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.551+0000 7f996b200640 1 --2- 172.21.15.43:0/2008400222 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f99500a1fb0 0x7f99500a4470 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9964005fd0 tx=0x7f99640096a0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.550 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.551+0000 7f996b200640 1 -- 172.21.15.43:0/2008400222 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f996c107c90 msgr2=0x7f996c19e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:42:48.550 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.551+0000 7f996b200640 1 --2- 172.21.15.43:0/2008400222 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f996c107c90 0x7f996c19e490 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f995c000c00 tx=0x7f995c0043b0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.550 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.551+0000 7f996b200640 1 -- 172.21.15.43:0/2008400222 shutdown_connections 2024-03-28T22:42:48.550 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.552+0000 7f996b200640 1 --2- 172.21.15.43:0/2008400222 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f99500a1fb0 0x7f99500a4470 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.550 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.552+0000 7f996b200640 1 --2- 172.21.15.43:0/2008400222 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f996c10dbf0 0x7f996c1a2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.550 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.552+0000 7f996b200640 1 --2- 172.21.15.43:0/2008400222 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f996c108610 0x7f996c19e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.550 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.552+0000 7f996b200640 1 --2- 172.21.15.43:0/2008400222 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f996c107c90 0x7f996c19e490 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:42:48.550 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.552+0000 7f996b200640 1 -- 172.21.15.43:0/2008400222 >> 172.21.15.43:0/2008400222 conn(0x7f996c077530 msgr2=0x7f996c0786e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:42:48.550 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.552+0000 7f996b200640 1 -- 172.21.15.43:0/2008400222 shutdown_connections 2024-03-28T22:42:48.550 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:42:48.552+0000 7f996b200640 1 -- 172.21.15.43:0/2008400222 wait complete. 2024-03-28T22:42:49.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:48 smithi172 ceph-mon[36566]: pgmap v173: 129 pgs: 129 active+clean; 52 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 344 op/s 2024-03-28T22:42:49.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:48 smithi172 ceph-mon[36566]: scrub summary: idle 2024-03-28T22:42:49.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:48 smithi082 ceph-mon[36471]: pgmap v173: 129 pgs: 129 active+clean; 52 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 344 op/s 2024-03-28T22:42:49.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:48 smithi082 ceph-mon[36471]: scrub summary: idle 2024-03-28T22:42:49.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:48 smithi043 ceph-mon[31823]: pgmap v173: 129 pgs: 129 active+clean; 52 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 344 op/s 2024-03-28T22:42:49.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:48 smithi043 ceph-mon[31823]: scrub summary: idle 2024-03-28T22:42:49.368 INFO:tasks.workunit.client.0.smithi043.stdout: 1 387548 7.78 MB/sec execute 140 sec latency 115.106 ms 2024-03-28T22:42:50.368 INFO:tasks.workunit.client.0.smithi043.stdout: 1 389673 7.81 MB/sec execute 141 sec latency 9.520 ms 2024-03-28T22:42:51.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:50 smithi172 ceph-mon[36566]: pgmap v174: 129 pgs: 129 active+clean; 52 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 169 op/s 2024-03-28T22:42:51.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:50 smithi082 ceph-mon[36471]: pgmap v174: 129 pgs: 129 active+clean; 52 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 169 op/s 2024-03-28T22:42:51.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:50 smithi043 ceph-mon[31823]: pgmap v174: 129 pgs: 129 active+clean; 52 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 169 op/s 2024-03-28T22:42:51.368 INFO:tasks.workunit.client.0.smithi043.stdout: 1 390757 7.80 MB/sec execute 142 sec latency 37.181 ms 2024-03-28T22:42:52.368 INFO:tasks.workunit.client.0.smithi043.stdout: 1 392199 7.78 MB/sec execute 143 sec latency 28.221 ms 2024-03-28T22:42:53.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:52 smithi172 ceph-mon[36566]: pgmap v175: 129 pgs: 129 active+clean; 52 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 169 op/s 2024-03-28T22:42:53.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:52 smithi082 ceph-mon[36471]: pgmap v175: 129 pgs: 129 active+clean; 52 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 169 op/s 2024-03-28T22:42:53.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:52 smithi043 ceph-mon[31823]: pgmap v175: 129 pgs: 129 active+clean; 52 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 169 op/s 2024-03-28T22:42:53.368 INFO:tasks.workunit.client.0.smithi043.stdout: 1 393999 7.82 MB/sec execute 144 sec latency 12.702 ms 2024-03-28T22:42:54.369 INFO:tasks.workunit.client.0.smithi043.stdout: 1 394888 7.78 MB/sec execute 145 sec latency 19.520 ms 2024-03-28T22:42:55.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:54 smithi172 ceph-mon[36566]: pgmap v176: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 348 op/s 2024-03-28T22:42:55.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:54 smithi082 ceph-mon[36471]: pgmap v176: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 348 op/s 2024-03-28T22:42:55.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:54 smithi043 ceph-mon[31823]: pgmap v176: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 348 op/s 2024-03-28T22:42:55.369 INFO:tasks.workunit.client.0.smithi043.stdout: 1 396955 7.80 MB/sec execute 146 sec latency 14.301 ms 2024-03-28T22:42:56.369 INFO:tasks.workunit.client.0.smithi043.stdout: 1 397870 7.79 MB/sec execute 147 sec latency 47.964 ms 2024-03-28T22:42:57.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:56 smithi082 ceph-mon[36471]: pgmap v177: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 264 op/s 2024-03-28T22:42:57.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:56 smithi043 ceph-mon[31823]: pgmap v177: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 264 op/s 2024-03-28T22:42:57.370 INFO:tasks.workunit.client.0.smithi043.stdout: 1 399183 7.78 MB/sec execute 148 sec latency 32.827 ms 2024-03-28T22:42:57.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:56 smithi172 ceph-mon[36566]: pgmap v177: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 264 op/s 2024-03-28T22:42:58.369 INFO:tasks.workunit.client.0.smithi043.stdout: 1 400939 7.81 MB/sec execute 149 sec latency 12.867 ms 2024-03-28T22:42:59.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:42:58 smithi082 ceph-mon[36471]: pgmap v178: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 338 op/s 2024-03-28T22:42:59.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:42:58 smithi043 ceph-mon[31823]: pgmap v178: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 338 op/s 2024-03-28T22:42:59.369 INFO:tasks.workunit.client.0.smithi043.stdout: 1 401685 7.77 MB/sec execute 150 sec latency 17.547 ms 2024-03-28T22:42:59.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:42:58 smithi172 ceph-mon[36566]: pgmap v178: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 338 op/s 2024-03-28T22:43:00.370 INFO:tasks.workunit.client.0.smithi043.stdout: 1 403454 7.76 MB/sec execute 151 sec latency 12.798 ms 2024-03-28T22:43:01.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:00 smithi082 ceph-mon[36471]: pgmap v179: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 252 op/s 2024-03-28T22:43:01.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:00 smithi043 ceph-mon[31823]: pgmap v179: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 252 op/s 2024-03-28T22:43:01.370 INFO:tasks.workunit.client.0.smithi043.stdout: 1 404579 7.78 MB/sec execute 152 sec latency 89.096 ms 2024-03-28T22:43:01.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:00 smithi172 ceph-mon[36566]: pgmap v179: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 252 op/s 2024-03-28T22:43:01.704 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.706+0000 7f0cef200640 1 -- 172.21.15.43:0/3250026471 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cf0103040 msgr2=0x7f0cf010b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:01.704 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.706+0000 7f0cef200640 1 --2- 172.21.15.43:0/3250026471 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cf0103040 0x7f0cf010b520 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc009860 tx=0x7f0cdc02bc00 comp rx=0 tx=0).stop 2024-03-28T22:43:01.704 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.706+0000 7f0cef200640 1 -- 172.21.15.43:0/3250026471 shutdown_connections 2024-03-28T22:43:01.704 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.706+0000 7f0cef200640 1 --2- 172.21.15.43:0/3250026471 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0cf010bb40 0x7f0cf010dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.704 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.706+0000 7f0cef200640 1 --2- 172.21.15.43:0/3250026471 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cf0103040 0x7f0cf010b520 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.704 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.706+0000 7f0cef200640 1 --2- 172.21.15.43:0/3250026471 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0cf0102720 0x7f0cf0102b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.704 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.706+0000 7f0cef200640 1 -- 172.21.15.43:0/3250026471 >> 172.21.15.43:0/3250026471 conn(0x7f0cf00fbf60 msgr2=0x7f0cf00fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:01.705 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.706+0000 7f0cef200640 1 -- 172.21.15.43:0/3250026471 shutdown_connections 2024-03-28T22:43:01.705 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.706+0000 7f0cef200640 1 -- 172.21.15.43:0/3250026471 wait complete. 2024-03-28T22:43:01.705 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.708+0000 7f0cef200640 1 Processor -- start 2024-03-28T22:43:01.705 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.708+0000 7f0cef200640 1 -- start start 2024-03-28T22:43:01.706 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.708+0000 7f0cef200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cf0102720 0x7f0cf019b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:01.706 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.708+0000 7f0cef200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0cf0103040 0x7f0cf019be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:01.706 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.708+0000 7f0cef200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0cf010bb40 0x7f0cf0195a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:01.706 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.708+0000 7f0cef200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf010f730 con 0x7f0cf010bb40 2024-03-28T22:43:01.706 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.708+0000 7f0cef200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf010f5b0 con 0x7f0cf0103040 2024-03-28T22:43:01.706 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.708+0000 7f0cef200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf010f8b0 con 0x7f0cf0102720 2024-03-28T22:43:01.707 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.708+0000 7f0cede00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cf0102720 0x7f0cf019b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:01.707 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.708+0000 7f0ced400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0cf0103040 0x7f0cf019be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:01.707 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.709+0000 7f0cede00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cf0102720 0x7f0cf019b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:41504/0 (socket says 172.21.15.43:41504) 2024-03-28T22:43:01.707 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.709+0000 7f0cede00640 1 -- 172.21.15.43:0/3620235262 learned_addr learned my addr 172.21.15.43:0/3620235262 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:01.707 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.709+0000 7f0cee800640 1 --2- 172.21.15.43:0/3620235262 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0cf010bb40 0x7f0cf0195a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:01.707 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.709+0000 7f0cede00640 1 -- 172.21.15.43:0/3620235262 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0cf0103040 msgr2=0x7f0cf019be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:01.707 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.709+0000 7f0cede00640 1 --2- 172.21.15.43:0/3620235262 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0cf0103040 0x7f0cf019be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.708 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.709+0000 7f0cede00640 1 -- 172.21.15.43:0/3620235262 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0cf010bb40 msgr2=0x7f0cf0195a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:01.708 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.709+0000 7f0cede00640 1 --2- 172.21.15.43:0/3620235262 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0cf010bb40 0x7f0cf0195a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.708 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.709+0000 7f0cede00640 1 -- 172.21.15.43:0/3620235262 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cf01961f0 con 0x7f0cf0102720 2024-03-28T22:43:01.708 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.709+0000 7f0cee800640 1 --2- 172.21.15.43:0/3620235262 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0cf010bb40 0x7f0cf0195a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:43:01.708 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.709+0000 7f0ced400640 1 --2- 172.21.15.43:0/3620235262 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0cf0103040 0x7f0cf019be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:01.708 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.709+0000 7f0cede00640 1 --2- 172.21.15.43:0/3620235262 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cf0102720 0x7f0cf019b8c0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0012fb0 tx=0x7f0ce0010490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:01.708 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.710+0000 7f0ce6a00640 1 -- 172.21.15.43:0/3620235262 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce0010db0 con 0x7f0cf0102720 2024-03-28T22:43:01.708 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.710+0000 7f0ce6a00640 1 -- 172.21.15.43:0/3620235262 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ce0004020 con 0x7f0cf0102720 2024-03-28T22:43:01.708 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.710+0000 7f0ce6a00640 1 -- 172.21.15.43:0/3620235262 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce0003bf0 con 0x7f0cf0102720 2024-03-28T22:43:01.708 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.710+0000 7f0cef200640 1 -- 172.21.15.43:0/3620235262 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cf0196480 con 0x7f0cf0102720 2024-03-28T22:43:01.708 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.710+0000 7f0cef200640 1 -- 172.21.15.43:0/3620235262 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cf01a2780 con 0x7f0cf0102720 2024-03-28T22:43:01.711 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.713+0000 7f0cf7e13740 1 Processor -- start 2024-03-28T22:43:01.711 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.714+0000 7f0ce6a00640 1 -- 172.21.15.43:0/3620235262 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f0ce001f030 con 0x7f0cf0102720 2024-03-28T22:43:01.711 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.714+0000 7f0cf7e13740 1 -- start start 2024-03-28T22:43:01.712 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.714+0000 7f0cf7e13740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5613d8324570 0x5613d8324950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:01.712 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.714+0000 7f0cf7e13740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5613d8324e90 0x5613d832d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:01.712 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.714+0000 7f0cf7e13740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5613d832dc60 0x5613d8330050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:01.712 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.714+0000 7f0cf7e13740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5613d8287640 con 0x5613d8324e90 2024-03-28T22:43:01.712 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.714+0000 7f0cf7e13740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5613d8230810 con 0x5613d8324570 2024-03-28T22:43:01.712 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.714+0000 7f0cf7e13740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5613d82836e0 con 0x5613d832dc60 2024-03-28T22:43:01.712 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.714+0000 7f0cee800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5613d8324570 0x5613d8324950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:01.712 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.714+0000 7f0cede00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5613d8324e90 0x5613d832d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:01.712 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.714+0000 7f0cee800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5613d8324570 0x5613d8324950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:42360/0 (socket says 172.21.15.43:42360) 2024-03-28T22:43:01.712 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.714+0000 7f0cee800640 1 -- 172.21.15.43:0/4176574736 learned_addr learned my addr 172.21.15.43:0/4176574736 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:01.712 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.715+0000 7f0ced400640 1 --2- 172.21.15.43:0/4176574736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5613d832dc60 0x5613d8330050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:01.714 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.715+0000 7f0cee800640 1 -- 172.21.15.43:0/4176574736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5613d832dc60 msgr2=0x5613d8330050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:01.714 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.715+0000 7f0cee800640 1 --2- 172.21.15.43:0/4176574736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5613d832dc60 0x5613d8330050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.714 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.715+0000 7f0cee800640 1 -- 172.21.15.43:0/4176574736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5613d8324e90 msgr2=0x5613d832d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:01.714 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.715+0000 7f0cee800640 1 --2- 172.21.15.43:0/4176574736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5613d8324e90 0x5613d832d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.714 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.715+0000 7f0cee800640 1 -- 172.21.15.43:0/4176574736 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613d7f9c210 con 0x5613d8324570 2024-03-28T22:43:01.714 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.715+0000 7f0cee800640 1 --2- 172.21.15.43:0/4176574736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5613d8324570 0x5613d8324950 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8009fd0 tx=0x7f0ce8010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=613a8b2a9024a5c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:01.714 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.716+0000 7f0cdb400640 1 -- 172.21.15.43:0/4176574736 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8011700 con 0x5613d8324570 2024-03-28T22:43:01.714 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.716+0000 7f0cdb400640 1 -- 172.21.15.43:0/4176574736 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ce8011d80 con 0x5613d8324570 2024-03-28T22:43:01.714 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.716+0000 7f0ce6a00640 1 --2- 172.21.15.43:0/3620235262 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0cd40a1ee0 0x7f0cd40a43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:01.714 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.716+0000 7f0cdb400640 1 -- 172.21.15.43:0/4176574736 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce800e3c0 con 0x5613d8324570 2024-03-28T22:43:01.714 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.716+0000 7f0ce6a00640 1 -- 172.21.15.43:0/3620235262 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f0ce00d1c10 con 0x7f0cf0102720 2024-03-28T22:43:01.714 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.716+0000 7f0cede00640 1 --2- 172.21.15.43:0/3620235262 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0cd40a1ee0 0x7f0cd40a43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.717+0000 7f0cf7e13740 1 -- 172.21.15.43:0/4176574736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5613d8324570 msgr2=0x5613d8324950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.717+0000 7f0cf7e13740 1 --2- 172.21.15.43:0/4176574736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5613d8324570 0x5613d8324950 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8009fd0 tx=0x7f0ce8010860 comp rx=0 tx=0).stop 2024-03-28T22:43:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.717+0000 7f0cf7e13740 1 -- 172.21.15.43:0/4176574736 shutdown_connections 2024-03-28T22:43:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.717+0000 7f0cf7e13740 1 --2- 172.21.15.43:0/4176574736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5613d832dc60 0x5613d8330050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.717+0000 7f0cf7e13740 1 --2- 172.21.15.43:0/4176574736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5613d8324e90 0x5613d832d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.717+0000 7f0cf7e13740 1 --2- 172.21.15.43:0/4176574736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5613d8324570 0x5613d8324950 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.717+0000 7f0cf7e13740 1 -- 172.21.15.43:0/4176574736 >> 172.21.15.43:0/4176574736 conn(0x5613d831ee90 msgr2=0x5613d831f270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.717+0000 7f0cf7e13740 1 -- 172.21.15.43:0/4176574736 shutdown_connections 2024-03-28T22:43:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.717+0000 7f0cede00640 1 --2- 172.21.15.43:0/3620235262 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0cd40a1ee0 0x7f0cd40a43a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0ce00cf070 tx=0x7f0ce00d5750 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.717+0000 7f0cf7e13740 1 -- 172.21.15.43:0/4176574736 wait complete. 2024-03-28T22:43:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.718+0000 7f0cf7e13740 1 -- start start 2024-03-28T22:43:01.717 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.718+0000 7f0cf7e13740 1 Processor -- start 2024-03-28T22:43:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.718+0000 7f0cf7e13740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5613d8324570 0x5613d832ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.718+0000 7f0cf7e13740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5613d8324e90 0x5613d832b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.718+0000 7f0cf7e13740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5613d832dc60 0x5613d832b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.718+0000 7f0cf7e13740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5613d8330890 con 0x5613d8324570 2024-03-28T22:43:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.718+0000 7f0cf7e13740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5613d8225d00 con 0x5613d832dc60 2024-03-28T22:43:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.718+0000 7f0cf7e13740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5613d8330a10 con 0x5613d8324e90 2024-03-28T22:43:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.718+0000 7f0cf7e13740 10 client.0 _ll_register_callbacks cb 0x5613d8210490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.719+0000 7f0ced400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5613d8324e90 0x5613d832b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.719+0000 7f0ced400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5613d8324e90 0x5613d832b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:41522/0 (socket says 172.21.15.43:41522) 2024-03-28T22:43:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.719+0000 7f0cee800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5613d8324570 0x5613d832ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.719+0000 7f0ced400640 1 -- 172.21.15.43:0/3769386353 learned_addr learned my addr 172.21.15.43:0/3769386353 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.719+0000 7f0cee800640 1 --2- 172.21.15.43:0/3769386353 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5613d832dc60 0x5613d832b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.719+0000 7f0ced400640 1 -- 172.21.15.43:0/3769386353 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5613d832dc60 msgr2=0x5613d832b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.719+0000 7f0ced400640 1 --2- 172.21.15.43:0/3769386353 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5613d832dc60 0x5613d832b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.719+0000 7f0ced400640 1 -- 172.21.15.43:0/3769386353 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5613d8324570 msgr2=0x5613d832ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.719+0000 7f0ced400640 1 --2- 172.21.15.43:0/3769386353 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5613d8324570 0x5613d832ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.719+0000 7f0ced400640 1 -- 172.21.15.43:0/3769386353 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613d8068620 con 0x5613d8324e90 2024-03-28T22:43:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.719+0000 7f0cee800640 1 --2- 172.21.15.43:0/3769386353 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5613d832dc60 0x5613d832b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:43:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.719+0000 7f0ced400640 1 --2- 172.21.15.43:0/3769386353 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5613d8324e90 0x5613d832b240 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc0057c0 tx=0x7f0cdc0044d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4f1408361c3095e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.719+0000 7f0ccfe00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:43:01.721 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.720+0000 7f0ccfe00640 1 -- 172.21.15.43:0/3769386353 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc045070 con 0x5613d8324e90 2024-03-28T22:43:01.721 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.720+0000 7f0ccfe00640 1 -- 172.21.15.43:0/3769386353 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f0cdc034850 con 0x5613d8324e90 2024-03-28T22:43:01.721 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.720+0000 7f0cf7e13740 1 -- 172.21.15.43:0/3769386353 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ce8011d80 con 0x5613d8324e90 2024-03-28T22:43:01.721 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.720+0000 7f0ccfe00640 1 -- 172.21.15.43:0/3769386353 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0cdc051050 con 0x5613d8324e90 2024-03-28T22:43:01.721 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.720+0000 7f0cf7e13740 10 client.27745 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:01.721 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.720+0000 7f0cf7e13740 1 -- 172.21.15.43:0/3769386353 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5613d833c7c0 con 0x5613d8324e90 2024-03-28T22:43:01.721 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.721+0000 7f0ccfe00640 1 -- 172.21.15.43:0/3769386353 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ce8011d80 con 0x5613d8324e90 2024-03-28T22:43:01.721 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.721+0000 7f0ccfe00640 1 -- 172.21.15.43:0/3769386353 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f0cdc003b10 con 0x5613d8324e90 2024-03-28T22:43:01.721 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.721+0000 7f0cf7e13740 10 client.27745 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:01.721 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.721+0000 7f0cf7e13740 10 client.27745 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:43:01.722 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.721+0000 7f0cf7e13740 20 client.27745 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:01.722 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.721+0000 7f0cf7e13740 1 --2- 172.21.15.43:0/3769386353 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5613d833dc30 0x5613d8340040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:01.722 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.721+0000 7f0cf7e13740 4 client.27745 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:43:01.722 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.721+0000 7f0cf7e13740 1 -- 172.21.15.43:0/3769386353 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5613d829b030 con 0x5613d833dc30 2024-03-28T22:43:01.722 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.721+0000 7f0cee800640 1 --2- 172.21.15.43:0/3769386353 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5613d833dc30 0x5613d8340040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:01.722 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.722+0000 7f0cee800640 1 --2- 172.21.15.43:0/3769386353 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5613d833dc30 0x5613d8340040 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1499108747b47473 server_cookie=d9badd508f479a98 in_seq=0 out_seq=0 2024-03-28T22:43:01.722 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.722+0000 7f0ccfe00640 10 client.27745 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:01.722 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.723+0000 7f0ccfe00640 1 -- 172.21.15.43:0/3769386353 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5613d829b030 con 0x5613d833dc30 2024-03-28T22:43:01.722 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.723+0000 7f0ccfe00640 10 client.27745 handle_command_reply: tid=0 2024-03-28T22:43:01.722 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.723+0000 7f0cf7e13740 1 client.27745 shutdown 2024-03-28T22:43:01.722 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.723+0000 7f0cf7e13740 1 -- 172.21.15.43:0/3769386353 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5613d8324e90 msgr2=0x5613d832b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:01.722 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.723+0000 7f0cf7e13740 1 --2- 172.21.15.43:0/3769386353 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5613d8324e90 0x5613d832b240 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc0057c0 tx=0x7f0cdc0044d0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.722 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.723+0000 7f0cf7e13740 1 -- 172.21.15.43:0/3769386353 shutdown_connections 2024-03-28T22:43:01.722 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.723+0000 7f0cf7e13740 1 --2- 172.21.15.43:0/3769386353 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5613d833dc30 0x5613d8340040 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.723 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.723+0000 7f0cf7e13740 1 --2- 172.21.15.43:0/3769386353 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5613d832dc60 0x5613d832b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.723 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.723+0000 7f0cf7e13740 1 --2- 172.21.15.43:0/3769386353 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5613d8324e90 0x5613d832b240 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.723 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.723+0000 7f0cf7e13740 1 --2- 172.21.15.43:0/3769386353 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5613d8324570 0x5613d832ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.723 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.723+0000 7f0ccfe00640 0 client.27745 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:01.723 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.723+0000 7f0cf7e13740 1 -- 172.21.15.43:0/3769386353 >> 172.21.15.43:0/3769386353 conn(0x5613d831ee90 msgr2=0x5613d8320be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:01.723 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.723+0000 7f0cf7e13740 1 -- 172.21.15.43:0/3769386353 shutdown_connections 2024-03-28T22:43:01.723 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.723+0000 7f0cf7e13740 1 -- 172.21.15.43:0/3769386353 wait complete. 2024-03-28T22:43:01.723 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.724+0000 7f0cf7e13740 20 client.27745 trim_cache size 0 max 16384 2024-03-28T22:43:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.741+0000 7f0cf7e13740 1 Processor -- start 2024-03-28T22:43:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.741+0000 7f0cf7e13740 1 -- start start 2024-03-28T22:43:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.741+0000 7f0cf7e13740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5613d8324570 0x5613d832d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.741+0000 7f0cf7e13740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5613d8324e90 0x5613d8339630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.741+0000 7f0cf7e13740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5613d832dc60 0x5613d8319a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.741+0000 7f0cf7e13740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf01ac860 con 0x5613d8324570 2024-03-28T22:43:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.741+0000 7f0cf7e13740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf01ae3a0 con 0x5613d8324e90 2024-03-28T22:43:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.741+0000 7f0cf7e13740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf01acf30 con 0x5613d832dc60 2024-03-28T22:43:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.742+0000 7f0ced400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5613d8324e90 0x5613d8339630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.742+0000 7f0cee800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5613d8324570 0x5613d832d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:01.740 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.742+0000 7f0ced400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5613d8324e90 0x5613d8339630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:42380/0 (socket says 172.21.15.43:42380) 2024-03-28T22:43:01.741 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.742+0000 7f0ced400640 1 -- 172.21.15.43:0/2505044161 learned_addr learned my addr 172.21.15.43:0/2505044161 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:01.741 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.742+0000 7f0cee800640 1 --2- 172.21.15.43:0/2505044161 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5613d832dc60 0x5613d8319a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:01.741 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.742+0000 7f0ced400640 1 -- 172.21.15.43:0/2505044161 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5613d832dc60 msgr2=0x5613d8319a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:01.741 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.742+0000 7f0ced400640 1 --2- 172.21.15.43:0/2505044161 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5613d832dc60 0x5613d8319a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.741 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.742+0000 7f0ced400640 1 -- 172.21.15.43:0/2505044161 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5613d8324570 msgr2=0x5613d832d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:01.741 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.742+0000 7f0ced400640 1 --2- 172.21.15.43:0/2505044161 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5613d8324570 0x5613d832d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.741 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.742+0000 7f0ced400640 1 -- 172.21.15.43:0/2505044161 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613d8319f70 con 0x5613d8324e90 2024-03-28T22:43:01.744 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.743+0000 7f0ced400640 1 --2- 172.21.15.43:0/2505044161 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5613d8324e90 0x5613d8339630 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc0095f0 tx=0x7f0cdc005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bbd8c4703954cd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:01.744 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.743+0000 7f0cdaa00640 1 -- 172.21.15.43:0/2505044161 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc05a070 con 0x5613d8324e90 2024-03-28T22:43:01.744 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.743+0000 7f0cdaa00640 1 -- 172.21.15.43:0/2505044161 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cdc04e030 con 0x5613d8324e90 2024-03-28T22:43:01.744 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.743+0000 7f0cdaa00640 1 -- 172.21.15.43:0/2505044161 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc045070 con 0x5613d8324e90 2024-03-28T22:43:01.744 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.744+0000 7f0cf7e13740 1 -- 172.21.15.43:0/2505044161 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5613d8324e90 msgr2=0x5613d8339630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:01.745 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.744+0000 7f0cf7e13740 1 --2- 172.21.15.43:0/2505044161 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5613d8324e90 0x5613d8339630 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc0095f0 tx=0x7f0cdc005dc0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.745 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.744+0000 7f0cf7e13740 1 -- 172.21.15.43:0/2505044161 shutdown_connections 2024-03-28T22:43:01.745 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.744+0000 7f0cf7e13740 1 --2- 172.21.15.43:0/2505044161 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5613d8324e90 0x5613d8339630 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.745 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.744+0000 7f0cf7e13740 1 --2- 172.21.15.43:0/2505044161 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5613d832dc60 0x5613d8319a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.745 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.744+0000 7f0cf7e13740 1 --2- 172.21.15.43:0/2505044161 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5613d8324570 0x5613d832d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.745 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.744+0000 7f0cf7e13740 1 -- 172.21.15.43:0/2505044161 >> 172.21.15.43:0/2505044161 conn(0x5613d831ee90 msgr2=0x5613d831f270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:01.745 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.744+0000 7f0cf7e13740 1 -- 172.21.15.43:0/2505044161 shutdown_connections 2024-03-28T22:43:01.745 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.744+0000 7f0cf7e13740 1 -- 172.21.15.43:0/2505044161 wait complete. 2024-03-28T22:43:01.745 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.744+0000 7f0cf7e13740 1 -- start start 2024-03-28T22:43:01.746 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.745+0000 7f0cf7e13740 1 Processor -- start 2024-03-28T22:43:01.746 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.745+0000 7f0cf7e13740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5613d8324570 0x5613d8327f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:01.746 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.745+0000 7f0cf7e13740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5613d8324e90 0x5613d83284a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:01.746 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.745+0000 7f0cf7e13740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5613d832dc60 0x5613d83a2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:01.746 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.745+0000 7f0cf7e13740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf01adcd0 con 0x5613d832dc60 2024-03-28T22:43:01.746 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.745+0000 7f0cf7e13740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf01ad600 con 0x5613d8324570 2024-03-28T22:43:01.746 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.745+0000 7f0cf7e13740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf01aa840 con 0x5613d8324e90 2024-03-28T22:43:01.746 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.745+0000 7f0cf7e13740 10 client.0 _ll_register_callbacks cb 0x7f0ce80093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:01.746 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.745+0000 7f0ced400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5613d8324e90 0x5613d83284a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:01.746 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.745+0000 7f0cee800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5613d8324570 0x5613d8327f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:01.746 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.745+0000 7f0ced400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5613d8324e90 0x5613d83284a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:41540/0 (socket says 172.21.15.43:41540) 2024-03-28T22:43:01.746 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.745+0000 7f0ced400640 1 -- 172.21.15.43:0/1780561305 learned_addr learned my addr 172.21.15.43:0/1780561305 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:01.747 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.745+0000 7f0cee800640 1 --2- 172.21.15.43:0/1780561305 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5613d832dc60 0x5613d83a2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:01.747 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.746+0000 7f0ced400640 1 -- 172.21.15.43:0/1780561305 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5613d8324570 msgr2=0x5613d8327f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:01.747 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.746+0000 7f0ced400640 1 --2- 172.21.15.43:0/1780561305 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5613d8324570 0x5613d8327f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.747 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.746+0000 7f0ced400640 1 -- 172.21.15.43:0/1780561305 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5613d832dc60 msgr2=0x5613d83a2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:01.747 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.746+0000 7f0ced400640 1 --2- 172.21.15.43:0/1780561305 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5613d832dc60 0x5613d83a2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.747 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.746+0000 7f0ced400640 1 -- 172.21.15.43:0/1780561305 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613d83a34c0 con 0x5613d8324e90 2024-03-28T22:43:01.747 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.746+0000 7f0ced400640 1 --2- 172.21.15.43:0/1780561305 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5613d8324e90 0x5613d83284a0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc02c140 tx=0x7f0cdc03f800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2765d8f801784b66 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:01.747 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.746+0000 7f0ccfe00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:43:01.747 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.746+0000 7f0ccfe00640 1 -- 172.21.15.43:0/1780561305 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc05a070 con 0x5613d8324e90 2024-03-28T22:43:01.747 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.746+0000 7f0cf7e13740 1 -- 172.21.15.43:0/1780561305 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cdc04e030 con 0x5613d8324e90 2024-03-28T22:43:01.747 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.746+0000 7f0ccfe00640 1 -- 172.21.15.43:0/1780561305 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f0cdc03fd00 con 0x5613d8324e90 2024-03-28T22:43:01.747 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.747+0000 7f0ccfe00640 1 -- 172.21.15.43:0/1780561305 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0cdc067050 con 0x5613d8324e90 2024-03-28T22:43:01.747 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.747+0000 7f0cf7e13740 10 client.27751 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:01.748 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.747+0000 7f0cf7e13740 1 -- 172.21.15.43:0/1780561305 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5613d83a4730 con 0x5613d8324e90 2024-03-28T22:43:01.748 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.747+0000 7f0ccfe00640 1 -- 172.21.15.43:0/1780561305 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cdc04e030 con 0x5613d8324e90 2024-03-28T22:43:01.749 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.747+0000 7f0ccfe00640 1 -- 172.21.15.43:0/1780561305 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f0cdc0035f0 con 0x5613d8324e90 2024-03-28T22:43:01.749 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.747+0000 7f0cf7e13740 10 client.27751 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:01.749 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.747+0000 7f0cf7e13740 10 client.27751 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:43:01.749 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.747+0000 7f0cf7e13740 20 client.27751 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:01.749 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.747+0000 7f0cf7e13740 1 --2- 172.21.15.43:0/1780561305 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5613d83a4d30 0x5613d83a7150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:01.749 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.748+0000 7f0cf7e13740 4 client.27751 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T22:43:01.749 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.748+0000 7f0cf7e13740 1 -- 172.21.15.43:0/1780561305 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5613d8294b80 con 0x5613d83a4d30 2024-03-28T22:43:01.749 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.748+0000 7f0cee800640 1 --2- 172.21.15.43:0/1780561305 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5613d83a4d30 0x5613d83a7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:01.749 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.748+0000 7f0cee800640 1 --2- 172.21.15.43:0/1780561305 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5613d83a4d30 0x5613d83a7150 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=752be729d5fd1093 server_cookie=8be24a753b857e4 in_seq=0 out_seq=0 2024-03-28T22:43:01.749 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.748+0000 7f0ccfe00640 10 client.27751 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:01.946 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.948+0000 7f0ccfe00640 1 -- 172.21.15.43:0/1780561305 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5613d8294b80 con 0x5613d83a4d30 2024-03-28T22:43:01.946 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.948+0000 7f0ccfe00640 10 client.27751 handle_command_reply: tid=0 2024-03-28T22:43:01.946 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.948+0000 7f0cf7e13740 1 client.27751 shutdown 2024-03-28T22:43:01.947 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.948+0000 7f0cf7e13740 1 -- 172.21.15.43:0/1780561305 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5613d8324e90 msgr2=0x5613d83284a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:01.947 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.948+0000 7f0cf7e13740 1 --2- 172.21.15.43:0/1780561305 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5613d8324e90 0x5613d83284a0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc02c140 tx=0x7f0cdc03f800 comp rx=0 tx=0).stop 2024-03-28T22:43:01.947 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.948+0000 7f0cf7e13740 1 -- 172.21.15.43:0/1780561305 shutdown_connections 2024-03-28T22:43:01.947 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.948+0000 7f0cf7e13740 1 --2- 172.21.15.43:0/1780561305 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5613d832dc60 0x5613d83a2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.947 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.948+0000 7f0cf7e13740 1 --2- 172.21.15.43:0/1780561305 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5613d8324e90 0x5613d83284a0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.947 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.948+0000 7f0cf7e13740 1 --2- 172.21.15.43:0/1780561305 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5613d83a4d30 0x5613d83a7150 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.947 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.948+0000 7f0cf7e13740 1 --2- 172.21.15.43:0/1780561305 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5613d8324570 0x5613d8327f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.947 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.948+0000 7f0ccfe00640 0 client.27751 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:01.947 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.948+0000 7f0cf7e13740 1 -- 172.21.15.43:0/1780561305 >> 172.21.15.43:0/1780561305 conn(0x5613d831ee90 msgr2=0x5613d832aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:01.947 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.948+0000 7f0cf7e13740 1 -- 172.21.15.43:0/1780561305 shutdown_connections 2024-03-28T22:43:01.947 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.949+0000 7f0cf7e13740 1 -- 172.21.15.43:0/1780561305 wait complete. 2024-03-28T22:43:01.947 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.949+0000 7f0cf7e13740 20 client.27751 trim_cache size 0 max 16384 2024-03-28T22:43:01.948 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:{ 2024-03-28T22:43:01.948 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "message": "", 2024-03-28T22:43:01.948 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "return_code": 0 2024-03-28T22:43:01.948 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:} 2024-03-28T22:43:01.949 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.949+0000 7f0cef200640 1 -- 172.21.15.43:0/3620235262 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0cd40a1ee0 msgr2=0x7f0cd40a43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:01.949 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.950+0000 7f0cef200640 1 --2- 172.21.15.43:0/3620235262 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0cd40a1ee0 0x7f0cd40a43a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0ce00cf070 tx=0x7f0ce00d5750 comp rx=0 tx=0).stop 2024-03-28T22:43:01.949 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.950+0000 7f0cef200640 1 -- 172.21.15.43:0/3620235262 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cf0102720 msgr2=0x7f0cf019b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:01.949 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.950+0000 7f0cef200640 1 --2- 172.21.15.43:0/3620235262 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cf0102720 0x7f0cf019b8c0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0012fb0 tx=0x7f0ce0010490 comp rx=0 tx=0).stop 2024-03-28T22:43:01.949 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.950+0000 7f0cef200640 1 -- 172.21.15.43:0/3620235262 shutdown_connections 2024-03-28T22:43:01.949 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.950+0000 7f0cef200640 1 --2- 172.21.15.43:0/3620235262 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0cd40a1ee0 0x7f0cd40a43a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.949 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.950+0000 7f0cef200640 1 --2- 172.21.15.43:0/3620235262 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0cf0103040 0x7f0cf019be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.949 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.950+0000 7f0cef200640 1 --2- 172.21.15.43:0/3620235262 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0cf010bb40 0x7f0cf0195a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.949 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.950+0000 7f0cef200640 1 --2- 172.21.15.43:0/3620235262 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cf0102720 0x7f0cf019b8c0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:01.949 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.950+0000 7f0cef200640 1 -- 172.21.15.43:0/3620235262 >> 172.21.15.43:0/3620235262 conn(0x7f0cf00fbf60 msgr2=0x7f0cf00fccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:01.949 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.950+0000 7f0cef200640 1 -- 172.21.15.43:0/3620235262 shutdown_connections 2024-03-28T22:43:01.949 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:01.950+0000 7f0cef200640 1 -- 172.21.15.43:0/3620235262 wait complete. 2024-03-28T22:43:02.370 INFO:tasks.workunit.client.0.smithi043.stdout: 1 404599 7.73 MB/sec execute 153 sec latency 200.857 ms 2024-03-28T22:43:03.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:02 smithi082 ceph-mon[36471]: pgmap v180: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 252 op/s 2024-03-28T22:43:03.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:02 smithi043 ceph-mon[31823]: pgmap v180: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 252 op/s 2024-03-28T22:43:03.370 INFO:tasks.workunit.client.0.smithi043.stdout: 1 405324 7.69 MB/sec execute 154 sec latency 94.769 ms 2024-03-28T22:43:03.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:02 smithi172 ceph-mon[36566]: pgmap v180: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 252 op/s 2024-03-28T22:43:04.370 INFO:tasks.workunit.client.0.smithi043.stdout: 1 407278 7.71 MB/sec execute 155 sec latency 29.681 ms 2024-03-28T22:43:05.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:04 smithi082 ceph-mon[36471]: pgmap v181: 129 pgs: 129 active+clean; 42 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 433 op/s 2024-03-28T22:43:05.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:04 smithi043 ceph-mon[31823]: pgmap v181: 129 pgs: 129 active+clean; 42 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 433 op/s 2024-03-28T22:43:05.370 INFO:tasks.workunit.client.0.smithi043.stdout: 1 408394 7.71 MB/sec execute 156 sec latency 16.910 ms 2024-03-28T22:43:05.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:04 smithi172 ceph-mon[36566]: pgmap v181: 129 pgs: 129 active+clean; 42 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 433 op/s 2024-03-28T22:43:06.370 INFO:tasks.workunit.client.0.smithi043.stdout: 1 409608 7.69 MB/sec execute 157 sec latency 63.210 ms 2024-03-28T22:43:07.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:06 smithi082 ceph-mon[36471]: pgmap v182: 129 pgs: 129 active+clean; 42 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 255 op/s 2024-03-28T22:43:07.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:06 smithi043 ceph-mon[31823]: pgmap v182: 129 pgs: 129 active+clean; 42 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 255 op/s 2024-03-28T22:43:07.370 INFO:tasks.workunit.client.0.smithi043.stdout: 1 411489 7.72 MB/sec execute 158 sec latency 77.580 ms 2024-03-28T22:43:07.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:06 smithi172 ceph-mon[36566]: pgmap v182: 129 pgs: 129 active+clean; 42 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 255 op/s 2024-03-28T22:43:08.371 INFO:tasks.workunit.client.0.smithi043.stdout: 1 412365 7.69 MB/sec execute 159 sec latency 20.720 ms 2024-03-28T22:43:09.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:08 smithi082 ceph-mon[36471]: pgmap v183: 129 pgs: 129 active+clean; 47 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 327 op/s 2024-03-28T22:43:09.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:08 smithi043 ceph-mon[31823]: pgmap v183: 129 pgs: 129 active+clean; 47 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 327 op/s 2024-03-28T22:43:09.371 INFO:tasks.workunit.client.0.smithi043.stdout: 1 413023 7.67 MB/sec execute 160 sec latency 647.896 ms 2024-03-28T22:43:09.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:08 smithi172 ceph-mon[36566]: pgmap v183: 129 pgs: 129 active+clean; 47 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 327 op/s 2024-03-28T22:43:10.371 INFO:tasks.workunit.client.0.smithi043.stdout: 1 413023 7.62 MB/sec execute 161 sec latency 1648.034 ms 2024-03-28T22:43:11.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:10 smithi082 ceph-mon[36471]: pgmap v184: 129 pgs: 129 active+clean; 47 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 253 op/s 2024-03-28T22:43:11.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:10 smithi043 ceph-mon[31823]: pgmap v184: 129 pgs: 129 active+clean; 47 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 253 op/s 2024-03-28T22:43:11.371 INFO:tasks.workunit.client.0.smithi043.stdout: 1 413023 7.57 MB/sec execute 162 sec latency 2648.133 ms 2024-03-28T22:43:11.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:10 smithi172 ceph-mon[36566]: pgmap v184: 129 pgs: 129 active+clean; 47 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 253 op/s 2024-03-28T22:43:12.371 INFO:tasks.workunit.client.0.smithi043.stdout: 1 413023 7.53 MB/sec execute 163 sec latency 3648.302 ms 2024-03-28T22:43:13.197 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:12 smithi172 ceph-mon[36566]: pgmap v185: 129 pgs: 129 active+clean; 47 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 253 op/s 2024-03-28T22:43:13.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:12 smithi082 ceph-mon[36471]: pgmap v185: 129 pgs: 129 active+clean; 47 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 253 op/s 2024-03-28T22:43:13.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:12 smithi043 ceph-mon[31823]: pgmap v185: 129 pgs: 129 active+clean; 47 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 253 op/s 2024-03-28T22:43:13.371 INFO:tasks.workunit.client.0.smithi043.stdout: 1 413023 7.48 MB/sec execute 164 sec latency 4648.510 ms 2024-03-28T22:43:14.009 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:43:14.010 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:43:14.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.251+0000 7f4c09400640 1 -- 172.21.15.43:0/4291784700 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4c04107f00 msgr2=0x7f4c041082e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:14.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.251+0000 7f4c09400640 1 --2- 172.21.15.43:0/4291784700 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4c04107f00 0x7f4c041082e0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4009900 tx=0x7f4bf402bc30 comp rx=0 tx=0).stop 2024-03-28T22:43:14.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.251+0000 7f4c09400640 1 -- 172.21.15.43:0/4291784700 shutdown_connections 2024-03-28T22:43:14.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.251+0000 7f4c09400640 1 --2- 172.21.15.43:0/4291784700 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c0410cf50 0x7f4c04114b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.251+0000 7f4c09400640 1 --2- 172.21.15.43:0/4291784700 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c041088b0 0x7f4c0410c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.251+0000 7f4c09400640 1 --2- 172.21.15.43:0/4291784700 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4c04107f00 0x7f4c041082e0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.251+0000 7f4c09400640 1 -- 172.21.15.43:0/4291784700 >> 172.21.15.43:0/4291784700 conn(0x7f4c04077570 msgr2=0x7f4c04106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:14.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.251+0000 7f4c09400640 1 -- 172.21.15.43:0/4291784700 shutdown_connections 2024-03-28T22:43:14.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.251+0000 7f4c09400640 1 -- 172.21.15.43:0/4291784700 wait complete. 2024-03-28T22:43:14.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.252+0000 7f4c09400640 1 Processor -- start 2024-03-28T22:43:14.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.252+0000 7f4c09400640 1 -- start start 2024-03-28T22:43:14.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.253+0000 7f4c09400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4c04107f00 0x7f4c0419e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:14.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.253+0000 7f4c09400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c0410cf50 0x7f4c0419ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:14.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.253+0000 7f4c09400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c041a2da0 0x7f4c041a3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:14.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.253+0000 7f4c09400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4c04118020 con 0x7f4c041a2da0 2024-03-28T22:43:14.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.253+0000 7f4c09400640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4c04117ea0 con 0x7f4c04107f00 2024-03-28T22:43:14.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.253+0000 7f4c09400640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041181a0 con 0x7f4c0410cf50 2024-03-28T22:43:14.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.253+0000 7f4c03e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4c04107f00 0x7f4c0419e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:14.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.253+0000 7f4c03e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4c04107f00 0x7f4c0419e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:51380/0 (socket says 172.21.15.43:51380) 2024-03-28T22:43:14.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.253+0000 7f4c03e00640 1 -- 172.21.15.43:0/1601972918 learned_addr learned my addr 172.21.15.43:0/1601972918 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:14.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.253+0000 7f4bfbe00640 1 --2- 172.21.15.43:0/1601972918 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c0410cf50 0x7f4c0419ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:14.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.253+0000 7f4c08a00640 1 --2- 172.21.15.43:0/1601972918 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c041a2da0 0x7f4c041a3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:14.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.253+0000 7f4c03e00640 1 -- 172.21.15.43:0/1601972918 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c0410cf50 msgr2=0x7f4c0419ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:14.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.253+0000 7f4c03e00640 1 --2- 172.21.15.43:0/1601972918 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c0410cf50 0x7f4c0419ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.253+0000 7f4c03e00640 1 -- 172.21.15.43:0/1601972918 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c041a2da0 msgr2=0x7f4c041a3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:14.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.253+0000 7f4c03e00640 1 --2- 172.21.15.43:0/1601972918 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c041a2da0 0x7f4c041a3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.253+0000 7f4c03e00640 1 -- 172.21.15.43:0/1601972918 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c041a3890 con 0x7f4c04107f00 2024-03-28T22:43:14.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.254+0000 7f4c03e00640 1 --2- 172.21.15.43:0/1601972918 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4c04107f00 0x7f4c0419e4e0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4006010 tx=0x7f4bf402cb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:14.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.254+0000 7f4c01600640 1 -- 172.21.15.43:0/1601972918 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf4041070 con 0x7f4c04107f00 2024-03-28T22:43:14.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.254+0000 7f4c01600640 1 -- 172.21.15.43:0/1601972918 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bf4004450 con 0x7f4c04107f00 2024-03-28T22:43:14.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.254+0000 7f4c09400640 1 -- 172.21.15.43:0/1601972918 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c041a3b20 con 0x7f4c04107f00 2024-03-28T22:43:14.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.254+0000 7f4c01600640 1 -- 172.21.15.43:0/1601972918 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf4003e00 con 0x7f4c04107f00 2024-03-28T22:43:14.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.254+0000 7f4c09400640 1 -- 172.21.15.43:0/1601972918 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c041ab2e0 con 0x7f4c04107f00 2024-03-28T22:43:14.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.257+0000 7f4c01600640 1 -- 172.21.15.43:0/1601972918 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f4bf4044070 con 0x7f4c04107f00 2024-03-28T22:43:14.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.257+0000 7f4c0d65d740 1 Processor -- start 2024-03-28T22:43:14.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.258+0000 7f4c0d65d740 1 -- start start 2024-03-28T22:43:14.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.258+0000 7f4c0d65d740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561bec34adb0 0x561bec349b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:14.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.258+0000 7f4c0d65d740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561bec34a0d0 0x561bec34a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:14.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.258+0000 7f4c0d65d740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561bec353440 0x561bec355830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:14.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.258+0000 7f4c0d65d740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x561bec2ae100 con 0x561bec353440 2024-03-28T22:43:14.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.258+0000 7f4c0d65d740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x561bec256ba0 con 0x561bec34a0d0 2024-03-28T22:43:14.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.258+0000 7f4c0d65d740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x561bec2aa1a0 con 0x561bec34adb0 2024-03-28T22:43:14.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.258+0000 7f4c08a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561bec34adb0 0x561bec349b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:14.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.258+0000 7f4c03e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561bec34a0d0 0x561bec34a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:14.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.258+0000 7f4c08a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561bec34adb0 0x561bec349b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:49296/0 (socket says 172.21.15.43:49296) 2024-03-28T22:43:14.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.258+0000 7f4c08a00640 1 -- 172.21.15.43:0/1115554488 learned_addr learned my addr 172.21.15.43:0/1115554488 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:14.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.258+0000 7f4bfbe00640 1 --2- 172.21.15.43:0/1115554488 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561bec353440 0x561bec355830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:14.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.259+0000 7f4c08a00640 1 -- 172.21.15.43:0/1115554488 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561bec34a0d0 msgr2=0x561bec34a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:14.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.259+0000 7f4c08a00640 1 --2- 172.21.15.43:0/1115554488 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561bec34a0d0 0x561bec34a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.259+0000 7f4c08a00640 1 -- 172.21.15.43:0/1115554488 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561bec353440 msgr2=0x561bec355830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:14.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.259+0000 7f4c08a00640 1 --2- 172.21.15.43:0/1115554488 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561bec353440 0x561bec355830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.259+0000 7f4c08a00640 1 -- 172.21.15.43:0/1115554488 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bec2871a0 con 0x561bec34adb0 2024-03-28T22:43:14.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.259+0000 7f4bfbe00640 1 --2- 172.21.15.43:0/1115554488 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561bec353440 0x561bec355830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:43:14.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.259+0000 7f4c01600640 1 --2- 172.21.15.43:0/1601972918 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4be80a1ef0 0x7f4be80a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:14.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.259+0000 7f4c08a00640 1 --2- 172.21.15.43:0/1115554488 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561bec34adb0 0x561bec349b00 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc009f60 tx=0x7f4bfc009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86536b77841d8c5f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:14.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.259+0000 7f4c01600640 1 -- 172.21.15.43:0/1601972918 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f4bf40eff80 con 0x7f4c04107f00 2024-03-28T22:43:14.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.259+0000 7f4c03e00640 1 --2- 172.21.15.43:0/1601972918 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4be80a1ef0 0x7f4be80a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:14.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.259+0000 7f4bf9200640 1 -- 172.21.15.43:0/1115554488 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc016070 con 0x561bec34adb0 2024-03-28T22:43:14.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.259+0000 7f4bf9200640 1 -- 172.21.15.43:0/1115554488 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bfc004d20 con 0x561bec34adb0 2024-03-28T22:43:14.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.259+0000 7f4bf9200640 1 -- 172.21.15.43:0/1115554488 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc012950 con 0x561bec34adb0 2024-03-28T22:43:14.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.260+0000 7f4c0d65d740 1 -- 172.21.15.43:0/1115554488 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561bec34adb0 msgr2=0x561bec349b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:14.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.260+0000 7f4c0d65d740 1 --2- 172.21.15.43:0/1115554488 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561bec34adb0 0x561bec349b00 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc009f60 tx=0x7f4bfc009f90 comp rx=0 tx=0).stop 2024-03-28T22:43:14.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.260+0000 7f4c03e00640 1 --2- 172.21.15.43:0/1601972918 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4be80a1ef0 0x7f4be80a43b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x561bec34b300 tx=0x7f4bf40f3a30 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:14.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.260+0000 7f4c0d65d740 1 -- 172.21.15.43:0/1115554488 shutdown_connections 2024-03-28T22:43:14.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.260+0000 7f4c0d65d740 1 --2- 172.21.15.43:0/1115554488 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561bec34a0d0 0x561bec34a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.260+0000 7f4c0d65d740 1 --2- 172.21.15.43:0/1115554488 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561bec353440 0x561bec355830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.260+0000 7f4c0d65d740 1 --2- 172.21.15.43:0/1115554488 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561bec34adb0 0x561bec349b00 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.260+0000 7f4c0d65d740 1 -- 172.21.15.43:0/1115554488 >> 172.21.15.43:0/1115554488 conn(0x561bec310a10 msgr2=0x561bec344b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:14.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.261+0000 7f4c0d65d740 1 -- 172.21.15.43:0/1115554488 shutdown_connections 2024-03-28T22:43:14.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.261+0000 7f4c0d65d740 1 -- 172.21.15.43:0/1115554488 wait complete. 2024-03-28T22:43:14.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.261+0000 7f4c0d65d740 1 -- start start 2024-03-28T22:43:14.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.261+0000 7f4c0d65d740 1 Processor -- start 2024-03-28T22:43:14.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.262+0000 7f4c0d65d740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561bec34a0d0 0x561bec350490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:14.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.262+0000 7f4c0d65d740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561bec34adb0 0x561bec3509d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:14.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.262+0000 7f4c0d65d740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561bec353440 0x561bec350f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:14.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.262+0000 7f4c0d65d740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x561bec3562b0 con 0x561bec34adb0 2024-03-28T22:43:14.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.262+0000 7f4c0d65d740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x561bec24c250 con 0x561bec353440 2024-03-28T22:43:14.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.262+0000 7f4c0d65d740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x561bec356430 con 0x561bec34a0d0 2024-03-28T22:43:14.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.262+0000 7f4c0d65d740 10 client.0 _ll_register_callbacks cb 0x561bebfc1be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:14.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.262+0000 7f4c08a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561bec34a0d0 0x561bec350490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:14.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.262+0000 7f4bfbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561bec34adb0 0x561bec3509d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:14.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.262+0000 7f4c08a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561bec34a0d0 0x561bec350490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:49298/0 (socket says 172.21.15.43:49298) 2024-03-28T22:43:14.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.262+0000 7f4c08a00640 1 -- 172.21.15.43:0/3239449326 learned_addr learned my addr 172.21.15.43:0/3239449326 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:14.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.262+0000 7f4c08a00640 1 --2- 172.21.15.43:0/3239449326 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561bec353440 0x561bec350f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:14.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.262+0000 7f4bfbe00640 1 -- 172.21.15.43:0/3239449326 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561bec34a0d0 msgr2=0x561bec350490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:14.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.262+0000 7f4bfbe00640 1 --2- 172.21.15.43:0/3239449326 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561bec34a0d0 0x561bec350490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.262+0000 7f4bfbe00640 1 -- 172.21.15.43:0/3239449326 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561bec353440 msgr2=0x561bec350f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:14.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.262+0000 7f4bfbe00640 1 --2- 172.21.15.43:0/3239449326 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561bec353440 0x561bec350f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.262+0000 7f4bfbe00640 1 -- 172.21.15.43:0/3239449326 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bec268960 con 0x561bec34adb0 2024-03-28T22:43:14.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.262+0000 7f4c08a00640 1 --2- 172.21.15.43:0/3239449326 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561bec34a0d0 0x561bec350490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:14.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.263+0000 7f4bfbe00640 1 --2- 172.21.15.43:0/3239449326 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561bec34adb0 0x561bec3509d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4c0406ac80 tx=0x7f4bec012b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d41e7ed1ebb988c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:14.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.263+0000 7f4bf1600640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:43:14.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.263+0000 7f4bf1600640 1 -- 172.21.15.43:0/3239449326 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bec004050 con 0x561bec34adb0 2024-03-28T22:43:14.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.263+0000 7f4bf1600640 1 -- 172.21.15.43:0/3239449326 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f4bec0041f0 con 0x561bec34adb0 2024-03-28T22:43:14.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.263+0000 7f4c0d65d740 1 -- 172.21.15.43:0/3239449326 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bfc004d20 con 0x561bec34adb0 2024-03-28T22:43:14.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.263+0000 7f4bf1600640 1 -- 172.21.15.43:0/3239449326 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4bec024050 con 0x561bec34adb0 2024-03-28T22:43:14.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.263+0000 7f4c0d65d740 10 client.17595 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:14.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.263+0000 7f4c0d65d740 1 -- 172.21.15.43:0/3239449326 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561bec08c7d0 con 0x561bec34adb0 2024-03-28T22:43:14.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.264+0000 7f4bf1600640 1 -- 172.21.15.43:0/3239449326 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bfc004d20 con 0x561bec34adb0 2024-03-28T22:43:14.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.264+0000 7f4bf1600640 1 -- 172.21.15.43:0/3239449326 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4bec0053e0 con 0x561bec34adb0 2024-03-28T22:43:14.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.264+0000 7f4c0d65d740 10 client.17595 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:14.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.264+0000 7f4c0d65d740 10 client.17595 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:43:14.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.264+0000 7f4c0d65d740 20 client.17595 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:14.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.264+0000 7f4c0d65d740 1 --2- 172.21.15.43:0/3239449326 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x561bec3639f0 0x561bec365e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:14.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.264+0000 7f4c0d65d740 4 client.17595 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:43:14.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.264+0000 7f4c0d65d740 1 -- 172.21.15.43:0/3239449326 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561bec2bb640 con 0x561bec3639f0 2024-03-28T22:43:14.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.264+0000 7f4c08a00640 1 --2- 172.21.15.43:0/3239449326 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x561bec3639f0 0x561bec365e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:14.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.265+0000 7f4c08a00640 1 --2- 172.21.15.43:0/3239449326 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x561bec3639f0 0x561bec365e30 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc7b2e5204532ed2 server_cookie=f4254318a41ebb26 in_seq=0 out_seq=0 2024-03-28T22:43:14.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.265+0000 7f4bf1600640 10 client.17595 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:14.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.265+0000 7f4bf1600640 1 -- 172.21.15.43:0/3239449326 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561bec2bb640 con 0x561bec3639f0 2024-03-28T22:43:14.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.265+0000 7f4bf1600640 10 client.17595 handle_command_reply: tid=0 2024-03-28T22:43:14.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.265+0000 7f4c0d65d740 1 client.17595 shutdown 2024-03-28T22:43:14.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.266+0000 7f4c0d65d740 1 -- 172.21.15.43:0/3239449326 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561bec34adb0 msgr2=0x561bec3509d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:14.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.266+0000 7f4c0d65d740 1 --2- 172.21.15.43:0/3239449326 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561bec34adb0 0x561bec3509d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4c0406ac80 tx=0x7f4bec012b60 comp rx=0 tx=0).stop 2024-03-28T22:43:14.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.266+0000 7f4c0d65d740 1 -- 172.21.15.43:0/3239449326 shutdown_connections 2024-03-28T22:43:14.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.266+0000 7f4c0d65d740 1 --2- 172.21.15.43:0/3239449326 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x561bec3639f0 0x561bec365e30 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.266+0000 7f4c0d65d740 1 --2- 172.21.15.43:0/3239449326 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561bec353440 0x561bec350f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.266+0000 7f4c0d65d740 1 --2- 172.21.15.43:0/3239449326 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561bec34adb0 0x561bec3509d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.266+0000 7f4c0d65d740 1 --2- 172.21.15.43:0/3239449326 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561bec34a0d0 0x561bec350490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.266+0000 7f4bf1600640 0 client.17595 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:14.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.266+0000 7f4c0d65d740 1 -- 172.21.15.43:0/3239449326 >> 172.21.15.43:0/3239449326 conn(0x561bec310a10 msgr2=0x561bec346870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:14.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.266+0000 7f4c0d65d740 1 -- 172.21.15.43:0/3239449326 shutdown_connections 2024-03-28T22:43:14.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.266+0000 7f4c0d65d740 1 -- 172.21.15.43:0/3239449326 wait complete. 2024-03-28T22:43:14.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.266+0000 7f4c0d65d740 20 client.17595 trim_cache size 0 max 16384 2024-03-28T22:43:14.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.286+0000 7f4c0d65d740 1 Processor -- start 2024-03-28T22:43:14.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.286+0000 7f4c0d65d740 1 -- start start 2024-03-28T22:43:14.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.286+0000 7f4c0d65d740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561bec34a0d0 0x561bec35f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:14.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.286+0000 7f4c0d65d740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561bec34adb0 0x561bec350d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:14.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.286+0000 7f4c0d65d740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561bec353440 0x561bec346680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:14.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.286+0000 7f4c0d65d740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041b5030 con 0x561bec34adb0 2024-03-28T22:43:14.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.286+0000 7f4c0d65d740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041b6b70 con 0x561bec34a0d0 2024-03-28T22:43:14.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.286+0000 7f4c0d65d740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041b5700 con 0x561bec353440 2024-03-28T22:43:14.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.286+0000 7f4c08a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561bec34a0d0 0x561bec35f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:14.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.286+0000 7f4c08a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561bec34a0d0 0x561bec35f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:51410/0 (socket says 172.21.15.43:51410) 2024-03-28T22:43:14.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.286+0000 7f4c08a00640 1 -- 172.21.15.43:0/2471023549 learned_addr learned my addr 172.21.15.43:0/2471023549 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:14.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.286+0000 7f4c08a00640 1 --2- 172.21.15.43:0/2471023549 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561bec353440 0x561bec346680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:14.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.287+0000 7f4bfbe00640 1 --2- 172.21.15.43:0/2471023549 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561bec34adb0 0x561bec350d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:14.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.287+0000 7f4c08a00640 1 -- 172.21.15.43:0/2471023549 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561bec353440 msgr2=0x561bec346680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:14.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.287+0000 7f4c08a00640 1 --2- 172.21.15.43:0/2471023549 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561bec353440 0x561bec346680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.287+0000 7f4c08a00640 1 -- 172.21.15.43:0/2471023549 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561bec34adb0 msgr2=0x561bec350d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:14.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.287+0000 7f4c08a00640 1 --2- 172.21.15.43:0/2471023549 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561bec34adb0 0x561bec350d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.287+0000 7f4c08a00640 1 -- 172.21.15.43:0/2471023549 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bebfc23f0 con 0x561bec34a0d0 2024-03-28T22:43:14.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.287+0000 7f4c08a00640 1 --2- 172.21.15.43:0/2471023549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561bec34a0d0 0x561bec35f090 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc005b00 tx=0x7f4bfc005970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf5023984f642503 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:14.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.287+0000 7f4bf3e00640 1 -- 172.21.15.43:0/2471023549 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc016070 con 0x561bec34a0d0 2024-03-28T22:43:14.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.287+0000 7f4bf3e00640 1 -- 172.21.15.43:0/2471023549 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bfc02e690 con 0x561bec34a0d0 2024-03-28T22:43:14.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.287+0000 7f4bf3e00640 1 -- 172.21.15.43:0/2471023549 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc012dd0 con 0x561bec34a0d0 2024-03-28T22:43:14.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.288+0000 7f4c0d65d740 1 -- 172.21.15.43:0/2471023549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561bec34a0d0 msgr2=0x561bec35f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:14.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.288+0000 7f4c0d65d740 1 --2- 172.21.15.43:0/2471023549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561bec34a0d0 0x561bec35f090 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc005b00 tx=0x7f4bfc005970 comp rx=0 tx=0).stop 2024-03-28T22:43:14.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.288+0000 7f4c0d65d740 1 -- 172.21.15.43:0/2471023549 shutdown_connections 2024-03-28T22:43:14.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.288+0000 7f4c0d65d740 1 --2- 172.21.15.43:0/2471023549 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561bec353440 0x561bec346680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.288+0000 7f4c0d65d740 1 --2- 172.21.15.43:0/2471023549 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561bec34adb0 0x561bec350d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.288+0000 7f4c0d65d740 1 --2- 172.21.15.43:0/2471023549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561bec34a0d0 0x561bec35f090 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.288+0000 7f4c0d65d740 1 -- 172.21.15.43:0/2471023549 >> 172.21.15.43:0/2471023549 conn(0x561bec310a10 msgr2=0x561bec345f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:14.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.288+0000 7f4c0d65d740 1 -- 172.21.15.43:0/2471023549 shutdown_connections 2024-03-28T22:43:14.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.288+0000 7f4c0d65d740 1 -- 172.21.15.43:0/2471023549 wait complete. 2024-03-28T22:43:14.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.288+0000 7f4c0d65d740 1 -- start start 2024-03-28T22:43:14.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.289+0000 7f4c0d65d740 1 Processor -- start 2024-03-28T22:43:14.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.289+0000 7f4c0d65d740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561bec34a0d0 0x561bec35ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:14.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.289+0000 7f4c0d65d740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561bec34adb0 0x561bec353e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:14.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.289+0000 7f4c0d65d740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561bec353440 0x561bec354450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:14.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.289+0000 7f4c0d65d740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041b64a0 con 0x561bec34adb0 2024-03-28T22:43:14.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.289+0000 7f4c0d65d740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041b5dd0 con 0x561bec34a0d0 2024-03-28T22:43:14.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.289+0000 7f4c0d65d740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041b3010 con 0x561bec353440 2024-03-28T22:43:14.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.289+0000 7f4c0d65d740 10 client.0 _ll_register_callbacks cb 0x7f4bec009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:14.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.289+0000 7f4bfbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561bec34adb0 0x561bec353e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:14.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.289+0000 7f4c08a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561bec34a0d0 0x561bec35ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:14.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.289+0000 7f4bfbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561bec34adb0 0x561bec353e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53166/0 (socket says 172.21.15.43:53166) 2024-03-28T22:43:14.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.289+0000 7f4bfbe00640 1 -- 172.21.15.43:0/821302624 learned_addr learned my addr 172.21.15.43:0/821302624 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:14.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.289+0000 7f4c08a00640 1 --2- 172.21.15.43:0/821302624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561bec353440 0x561bec354450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:14.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.290+0000 7f4bfbe00640 1 -- 172.21.15.43:0/821302624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561bec353440 msgr2=0x561bec354450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:14.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.290+0000 7f4bfbe00640 1 --2- 172.21.15.43:0/821302624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561bec353440 0x561bec354450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.290+0000 7f4bfbe00640 1 -- 172.21.15.43:0/821302624 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561bec34a0d0 msgr2=0x561bec35ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:14.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.290+0000 7f4bfbe00640 1 --2- 172.21.15.43:0/821302624 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561bec34a0d0 0x561bec35ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.290+0000 7f4bfbe00640 1 -- 172.21.15.43:0/821302624 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bebf820f0 con 0x561bec34adb0 2024-03-28T22:43:14.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.290+0000 7f4bfbe00640 1 --2- 172.21.15.43:0/821302624 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561bec34adb0 0x561bec353e40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4c0406ac80 tx=0x7f4bec0123a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b5235b1f73effb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:14.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.290+0000 7f4bf1600640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:43:14.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.290+0000 7f4bf1600640 1 -- 172.21.15.43:0/821302624 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bec02c070 con 0x561bec34adb0 2024-03-28T22:43:14.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.290+0000 7f4bf1600640 1 -- 172.21.15.43:0/821302624 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f4bec00fc40 con 0x561bec34adb0 2024-03-28T22:43:14.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.290+0000 7f4c0d65d740 1 -- 172.21.15.43:0/821302624 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bfc02e690 con 0x561bec34adb0 2024-03-28T22:43:14.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.291+0000 7f4bf1600640 1 -- 172.21.15.43:0/821302624 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4bec00fe40 con 0x561bec34adb0 2024-03-28T22:43:14.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.291+0000 7f4c0d65d740 10 client.17598 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:14.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.291+0000 7f4c0d65d740 1 -- 172.21.15.43:0/821302624 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561bec3c9d30 con 0x561bec34adb0 2024-03-28T22:43:14.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.291+0000 7f4bf1600640 1 -- 172.21.15.43:0/821302624 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bfc02e690 con 0x561bec34adb0 2024-03-28T22:43:14.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.291+0000 7f4bf1600640 1 -- 172.21.15.43:0/821302624 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4bec003b30 con 0x561bec34adb0 2024-03-28T22:43:14.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.291+0000 7f4c0d65d740 10 client.17598 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:14.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.291+0000 7f4c0d65d740 10 client.17598 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:43:14.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.291+0000 7f4c0d65d740 20 client.17598 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:14.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.291+0000 7f4c0d65d740 1 --2- 172.21.15.43:0/821302624 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x561bec3ca3b0 0x561bec3cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:14.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.291+0000 7f4c0d65d740 4 client.17598 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:43:14.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.291+0000 7f4c0d65d740 1 -- 172.21.15.43:0/821302624 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561bec2b5990 con 0x561bec3ca3b0 2024-03-28T22:43:14.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.292+0000 7f4c08a00640 1 --2- 172.21.15.43:0/821302624 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x561bec3ca3b0 0x561bec3cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:14.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.292+0000 7f4c08a00640 1 --2- 172.21.15.43:0/821302624 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x561bec3ca3b0 0x561bec3cc7d0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a0b84632c14f4b2 server_cookie=af9a55b6bc7beb91 in_seq=0 out_seq=0 2024-03-28T22:43:14.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.292+0000 7f4bf1600640 10 client.17598 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:14.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.292+0000 7f4bf1600640 1 -- 172.21.15.43:0/821302624 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561bec2b5990 con 0x561bec3ca3b0 2024-03-28T22:43:14.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.292+0000 7f4bf1600640 10 client.17598 handle_command_reply: tid=0 2024-03-28T22:43:14.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.292+0000 7f4c0d65d740 1 client.17598 shutdown 2024-03-28T22:43:14.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.293+0000 7f4c0d65d740 1 -- 172.21.15.43:0/821302624 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561bec34adb0 msgr2=0x561bec353e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:14.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.293+0000 7f4c0d65d740 1 --2- 172.21.15.43:0/821302624 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561bec34adb0 0x561bec353e40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4c0406ac80 tx=0x7f4bec0123a0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.293+0000 7f4c0d65d740 1 -- 172.21.15.43:0/821302624 shutdown_connections 2024-03-28T22:43:14.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.293+0000 7f4c0d65d740 1 --2- 172.21.15.43:0/821302624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561bec353440 0x561bec354450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.293+0000 7f4c0d65d740 1 --2- 172.21.15.43:0/821302624 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561bec34adb0 0x561bec353e40 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.293+0000 7f4c0d65d740 1 --2- 172.21.15.43:0/821302624 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x561bec3ca3b0 0x561bec3cc7d0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.293+0000 7f4c0d65d740 1 --2- 172.21.15.43:0/821302624 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561bec34a0d0 0x561bec35ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.293+0000 7f4c0d65d740 1 -- 172.21.15.43:0/821302624 >> 172.21.15.43:0/821302624 conn(0x561bec310a10 msgr2=0x561bec343d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:14.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.293+0000 7f4bf1600640 0 client.17598 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:14.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.293+0000 7f4c0d65d740 1 -- 172.21.15.43:0/821302624 shutdown_connections 2024-03-28T22:43:14.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.293+0000 7f4c0d65d740 1 -- 172.21.15.43:0/821302624 wait complete. 2024-03-28T22:43:14.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.293+0000 7f4c0d65d740 20 client.17598 trim_cache size 0 max 16384 2024-03-28T22:43:14.295 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:43:14.295 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-03-28T22:43:14.295 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-03-28T22:43:14.295 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:43:14.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.294+0000 7f4c09400640 1 -- 172.21.15.43:0/1601972918 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4be80a1ef0 msgr2=0x7f4be80a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:14.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.294+0000 7f4c09400640 1 --2- 172.21.15.43:0/1601972918 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4be80a1ef0 0x7f4be80a43b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x561bec34b300 tx=0x7f4bf40f3a30 comp rx=0 tx=0).stop 2024-03-28T22:43:14.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.294+0000 7f4c09400640 1 -- 172.21.15.43:0/1601972918 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4c04107f00 msgr2=0x7f4c0419e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:14.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.294+0000 7f4c09400640 1 --2- 172.21.15.43:0/1601972918 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4c04107f00 0x7f4c0419e4e0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4006010 tx=0x7f4bf402cb30 comp rx=0 tx=0).stop 2024-03-28T22:43:14.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.294+0000 7f4c09400640 1 -- 172.21.15.43:0/1601972918 shutdown_connections 2024-03-28T22:43:14.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.294+0000 7f4c09400640 1 --2- 172.21.15.43:0/1601972918 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4be80a1ef0 0x7f4be80a43b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.294+0000 7f4c09400640 1 --2- 172.21.15.43:0/1601972918 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4c041a2da0 0x7f4c041a3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.294+0000 7f4c09400640 1 --2- 172.21.15.43:0/1601972918 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4c0410cf50 0x7f4c0419ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.294+0000 7f4c09400640 1 --2- 172.21.15.43:0/1601972918 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4c04107f00 0x7f4c0419e4e0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.294+0000 7f4c09400640 1 -- 172.21.15.43:0/1601972918 >> 172.21.15.43:0/1601972918 conn(0x7f4c04077570 msgr2=0x7f4c04079130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:14.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.294+0000 7f4c09400640 1 -- 172.21.15.43:0/1601972918 shutdown_connections 2024-03-28T22:43:14.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.294+0000 7f4c09400640 1 -- 172.21.15.43:0/1601972918 wait complete. 2024-03-28T22:43:14.308 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:43:14.309 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:43:14.371 INFO:tasks.workunit.client.0.smithi043.stdout: 1 413023 7.44 MB/sec execute 165 sec latency 5648.661 ms 2024-03-28T22:43:14.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.537+0000 7fda23200640 1 -- 172.21.15.43:0/157576116 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda1c107b30 msgr2=0x7fda1c107fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:14.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.537+0000 7fda23200640 1 --2- 172.21.15.43:0/157576116 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda1c107b30 0x7fda1c107fb0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fda08009930 tx=0x7fda0802bc20 comp rx=0 tx=0).stop 2024-03-28T22:43:14.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.537+0000 7fda23200640 1 -- 172.21.15.43:0/157576116 shutdown_connections 2024-03-28T22:43:14.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.537+0000 7fda23200640 1 --2- 172.21.15.43:0/157576116 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda1c107b30 0x7fda1c107fb0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.537+0000 7fda23200640 1 --2- 172.21.15.43:0/157576116 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda1c1084f0 0x7fda1c1149b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.537+0000 7fda23200640 1 --2- 172.21.15.43:0/157576116 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda1c10dbf0 0x7fda1c10dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.537+0000 7fda23200640 1 -- 172.21.15.43:0/157576116 >> 172.21.15.43:0/157576116 conn(0x7fda1c077270 msgr2=0x7fda1c079690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:14.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.537+0000 7fda23200640 1 -- 172.21.15.43:0/157576116 shutdown_connections 2024-03-28T22:43:14.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.537+0000 7fda23200640 1 -- 172.21.15.43:0/157576116 wait complete. 2024-03-28T22:43:14.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.538+0000 7fda23200640 1 Processor -- start 2024-03-28T22:43:14.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.539+0000 7fda23200640 1 -- start start 2024-03-28T22:43:14.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.539+0000 7fda23200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda1c107b30 0x7fda1c19e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:14.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.539+0000 7fda23200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda1c1084f0 0x7fda1c19e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:14.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.539+0000 7fda23200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda1c10dbf0 0x7fda1c1a2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:14.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.539+0000 7fda23200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fda1c117f80 con 0x7fda1c10dbf0 2024-03-28T22:43:14.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.539+0000 7fda23200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fda1c117e00 con 0x7fda1c107b30 2024-03-28T22:43:14.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.539+0000 7fda23200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fda1c118100 con 0x7fda1c1084f0 2024-03-28T22:43:14.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.539+0000 7fda21e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda1c107b30 0x7fda1c19e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:14.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.539+0000 7fda21e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda1c107b30 0x7fda1c19e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:51444/0 (socket says 172.21.15.43:51444) 2024-03-28T22:43:14.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.539+0000 7fda21e00640 1 -- 172.21.15.43:0/214692609 learned_addr learned my addr 172.21.15.43:0/214692609 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:14.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.540+0000 7fda21400640 1 --2- 172.21.15.43:0/214692609 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda1c1084f0 0x7fda1c19e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:14.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.540+0000 7fda22800640 1 --2- 172.21.15.43:0/214692609 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda1c10dbf0 0x7fda1c1a2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:14.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.540+0000 7fda21e00640 1 -- 172.21.15.43:0/214692609 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda1c1084f0 msgr2=0x7fda1c19e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:14.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.540+0000 7fda21e00640 1 --2- 172.21.15.43:0/214692609 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda1c1084f0 0x7fda1c19e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.540+0000 7fda21e00640 1 -- 172.21.15.43:0/214692609 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda1c10dbf0 msgr2=0x7fda1c1a2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:14.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.540+0000 7fda21e00640 1 --2- 172.21.15.43:0/214692609 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda1c10dbf0 0x7fda1c1a2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.540+0000 7fda21e00640 1 -- 172.21.15.43:0/214692609 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda1c1a3350 con 0x7fda1c107b30 2024-03-28T22:43:14.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.540+0000 7fda21e00640 1 --2- 172.21.15.43:0/214692609 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda1c107b30 0x7fda1c19e360 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fda1000e9f0 tx=0x7fda1000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:14.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.540+0000 7fda16a00640 1 -- 172.21.15.43:0/214692609 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda1000cd80 con 0x7fda1c107b30 2024-03-28T22:43:14.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.540+0000 7fda16a00640 1 -- 172.21.15.43:0/214692609 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda10010910 con 0x7fda1c107b30 2024-03-28T22:43:14.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.540+0000 7fda16a00640 1 -- 172.21.15.43:0/214692609 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda10018990 con 0x7fda1c107b30 2024-03-28T22:43:14.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.540+0000 7fda23200640 1 -- 172.21.15.43:0/214692609 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda1c1a3640 con 0x7fda1c107b30 2024-03-28T22:43:14.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.541+0000 7fda23200640 1 -- 172.21.15.43:0/214692609 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda1c1aaed0 con 0x7fda1c107b30 2024-03-28T22:43:14.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.542+0000 7fda23200640 1 -- 172.21.15.43:0/214692609 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda1c0688b0 con 0x7fda1c107b30 2024-03-28T22:43:14.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.544+0000 7fda16a00640 1 -- 172.21.15.43:0/214692609 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fda10010ac0 con 0x7fda1c107b30 2024-03-28T22:43:14.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.545+0000 7fda16a00640 1 --2- 172.21.15.43:0/214692609 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fda040a1ef0 0x7fda040a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:14.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.545+0000 7fda16a00640 1 -- 172.21.15.43:0/214692609 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fda100ce940 con 0x7fda1c107b30 2024-03-28T22:43:14.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.545+0000 7fda21400640 1 --2- 172.21.15.43:0/214692609 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fda040a1ef0 0x7fda040a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:14.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.546+0000 7fda21400640 1 --2- 172.21.15.43:0/214692609 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fda040a1ef0 0x7fda040a43b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fda08009900 tx=0x7fda08005a20 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:14.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.549+0000 7fda16a00640 1 -- 172.21.15.43:0/214692609 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fda1009b630 con 0x7fda1c107b30 2024-03-28T22:43:14.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.794+0000 7fda23200640 1 -- 172.21.15.43:0/214692609 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fda1c19f6b0 con 0x7fda1c107b30 2024-03-28T22:43:14.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.795+0000 7fda16a00640 1 -- 172.21.15.43:0/214692609 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11931 (secure 0 0 0) 0x7fda1009af30 con 0x7fda1c107b30 2024-03-28T22:43:14.793 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:43:14.793 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:33:28.906396+0000","modified":"2024-03-28T22:40:08.672168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24572,"mds_1":24556,"mds_2":24535,"mds_3":24523,"mds_4":24518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26717":{"gid":26717,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/215779342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":215779342},{"type":"v1","addr":"172.21.15.82:6835","nonce":215779342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24535,24556,24518,24523,24572]},"id":1}]} 2024-03-28T22:43:14.794 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 32 2024-03-28T22:43:14.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.798+0000 7fda23200640 1 -- 172.21.15.43:0/214692609 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fda040a1ef0 msgr2=0x7fda040a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:14.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.798+0000 7fda23200640 1 --2- 172.21.15.43:0/214692609 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fda040a1ef0 0x7fda040a43b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fda08009900 tx=0x7fda08005a20 comp rx=0 tx=0).stop 2024-03-28T22:43:14.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.798+0000 7fda23200640 1 -- 172.21.15.43:0/214692609 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda1c107b30 msgr2=0x7fda1c19e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:14.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.798+0000 7fda23200640 1 --2- 172.21.15.43:0/214692609 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda1c107b30 0x7fda1c19e360 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fda1000e9f0 tx=0x7fda1000eeb0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.799+0000 7fda23200640 1 -- 172.21.15.43:0/214692609 shutdown_connections 2024-03-28T22:43:14.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.799+0000 7fda23200640 1 --2- 172.21.15.43:0/214692609 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fda040a1ef0 0x7fda040a43b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.799+0000 7fda23200640 1 --2- 172.21.15.43:0/214692609 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fda1c10dbf0 0x7fda1c1a2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.799+0000 7fda23200640 1 --2- 172.21.15.43:0/214692609 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fda1c1084f0 0x7fda1c19e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.799+0000 7fda23200640 1 --2- 172.21.15.43:0/214692609 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda1c107b30 0x7fda1c19e360 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:14.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.799+0000 7fda23200640 1 -- 172.21.15.43:0/214692609 >> 172.21.15.43:0/214692609 conn(0x7fda1c077270 msgr2=0x7fda1c0788f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:14.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.799+0000 7fda23200640 1 -- 172.21.15.43:0/214692609 shutdown_connections 2024-03-28T22:43:14.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:14.799+0000 7fda23200640 1 -- 172.21.15.43:0/214692609 wait complete. 2024-03-28T22:43:14.817 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T22:43:15.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.048+0000 7fe11d800640 1 -- 172.21.15.43:0/3258635149 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe118109900 msgr2=0x7fe118109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.048+0000 7fe11d800640 1 --2- 172.21.15.43:0/3258635149 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe118109900 0x7fe118109ce0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fe108009900 tx=0x7fe10802bc30 comp rx=0 tx=0).stop 2024-03-28T22:43:15.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.048+0000 7fe11d800640 1 -- 172.21.15.43:0/3258635149 shutdown_connections 2024-03-28T22:43:15.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.048+0000 7fe11d800640 1 --2- 172.21.15.43:0/3258635149 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe1181042b0 0x7fe1181107c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.048+0000 7fe11d800640 1 --2- 172.21.15.43:0/3258635149 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1181038f0 0x7fe118103d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.048+0000 7fe11d800640 1 --2- 172.21.15.43:0/3258635149 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe118109900 0x7fe118109ce0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.048+0000 7fe11d800640 1 -- 172.21.15.43:0/3258635149 >> 172.21.15.43:0/3258635149 conn(0x7fe1180ff3c0 msgr2=0x7fe1181017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:15.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.048+0000 7fe11d800640 1 -- 172.21.15.43:0/3258635149 shutdown_connections 2024-03-28T22:43:15.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.048+0000 7fe11d800640 1 -- 172.21.15.43:0/3258635149 wait complete. 2024-03-28T22:43:15.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.049+0000 7fe11d800640 1 Processor -- start 2024-03-28T22:43:15.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.049+0000 7fe11d800640 1 -- start start 2024-03-28T22:43:15.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.049+0000 7fe11d800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1181038f0 0x7fe11819a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.050+0000 7fe11d800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe1181042b0 0x7fe11819a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.050+0000 7fe11d800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe118109900 0x7fe11819eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.050+0000 7fe11d800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe118113d60 con 0x7fe1181042b0 2024-03-28T22:43:15.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.050+0000 7fe11d800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe118113be0 con 0x7fe118109900 2024-03-28T22:43:15.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.050+0000 7fe11d800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe118113ee0 con 0x7fe1181038f0 2024-03-28T22:43:15.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.050+0000 7fe116a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1181038f0 0x7fe11819a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.050+0000 7fe116a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1181038f0 0x7fe11819a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:49314/0 (socket says 172.21.15.43:49314) 2024-03-28T22:43:15.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.050+0000 7fe117400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe118109900 0x7fe11819eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.050+0000 7fe116a00640 1 -- 172.21.15.43:0/2405507862 learned_addr learned my addr 172.21.15.43:0/2405507862 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:15.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.050+0000 7fe116000640 1 --2- 172.21.15.43:0/2405507862 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe1181042b0 0x7fe11819a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.050+0000 7fe117400640 1 -- 172.21.15.43:0/2405507862 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1181038f0 msgr2=0x7fe11819a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.050+0000 7fe117400640 1 --2- 172.21.15.43:0/2405507862 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1181038f0 0x7fe11819a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.050+0000 7fe117400640 1 -- 172.21.15.43:0/2405507862 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe1181042b0 msgr2=0x7fe11819a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.050+0000 7fe117400640 1 --2- 172.21.15.43:0/2405507862 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe1181042b0 0x7fe11819a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.051+0000 7fe117400640 1 -- 172.21.15.43:0/2405507862 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe11819f290 con 0x7fe118109900 2024-03-28T22:43:15.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.051+0000 7fe116a00640 1 --2- 172.21.15.43:0/2405507862 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1181038f0 0x7fe11819a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:15.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.051+0000 7fe116000640 1 --2- 172.21.15.43:0/2405507862 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe1181042b0 0x7fe11819a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:15.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.051+0000 7fe117400640 1 --2- 172.21.15.43:0/2405507862 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe118109900 0x7fe11819eb60 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fe11000e940 tx=0x7fe11000ee00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:15.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.051+0000 7fe10f400640 1 -- 172.21.15.43:0/2405507862 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe110004d20 con 0x7fe118109900 2024-03-28T22:43:15.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.051+0000 7fe10f400640 1 -- 172.21.15.43:0/2405507862 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe110005020 con 0x7fe118109900 2024-03-28T22:43:15.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.051+0000 7fe10f400640 1 -- 172.21.15.43:0/2405507862 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1100116b0 con 0x7fe118109900 2024-03-28T22:43:15.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.051+0000 7fe11d800640 1 -- 172.21.15.43:0/2405507862 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe11819f580 con 0x7fe118109900 2024-03-28T22:43:15.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.051+0000 7fe11d800640 1 -- 172.21.15.43:0/2405507862 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1181a6e10 con 0x7fe118109900 2024-03-28T22:43:15.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.054+0000 7fe10f400640 1 -- 172.21.15.43:0/2405507862 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fe11000cb20 con 0x7fe118109900 2024-03-28T22:43:15.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.055+0000 7fe120602740 1 Processor -- start 2024-03-28T22:43:15.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.055+0000 7fe120602740 1 -- start start 2024-03-28T22:43:15.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.055+0000 7fe120602740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d397d21380 0x55d397d21760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.055+0000 7fe120602740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d397d21ca0 0x55d397d2a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.055+0000 7fe120602740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d397d2a880 0x55d397d2cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.055+0000 7fe120602740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d397c85050 con 0x55d397d2a880 2024-03-28T22:43:15.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.055+0000 7fe120602740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55d397c2dce0 con 0x55d397d21380 2024-03-28T22:43:15.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.055+0000 7fe120602740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55d397c810f0 con 0x55d397d21ca0 2024-03-28T22:43:15.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.055+0000 7fe116a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d397d21ca0 0x55d397d2a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.056+0000 7fe116a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d397d21ca0 0x55d397d2a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:49322/0 (socket says 172.21.15.43:49322) 2024-03-28T22:43:15.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.056+0000 7fe116a00640 1 -- 172.21.15.43:0/3158750856 learned_addr learned my addr 172.21.15.43:0/3158750856 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:15.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.056+0000 7fe116000640 1 --2- 172.21.15.43:0/3158750856 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d397d21380 0x55d397d21760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.056+0000 7fe116000640 1 --2- 172.21.15.43:0/3158750856 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d397d2a880 0x55d397d2cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.056+0000 7fe116a00640 1 -- 172.21.15.43:0/3158750856 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d397d21380 msgr2=0x55d397d21760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.056+0000 7fe116a00640 1 --2- 172.21.15.43:0/3158750856 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d397d21380 0x55d397d21760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.056+0000 7fe116a00640 1 -- 172.21.15.43:0/3158750856 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d397d2a880 msgr2=0x55d397d2cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.056+0000 7fe116a00640 1 --2- 172.21.15.43:0/3158750856 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d397d2a880 0x55d397d2cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.056+0000 7fe116a00640 1 -- 172.21.15.43:0/3158750856 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3979993f0 con 0x55d397d21ca0 2024-03-28T22:43:15.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.056+0000 7fe116000640 1 --2- 172.21.15.43:0/3158750856 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d397d2a880 0x55d397d2cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:43:15.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.056+0000 7fe116a00640 1 --2- 172.21.15.43:0/3158750856 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d397d21ca0 0x55d397d2a340 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fe108036040 tx=0x7fe108005e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=792c185f9739aea4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:15.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.056+0000 7fe10f400640 1 --2- 172.21.15.43:0/2405507862 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe0fc0a1f70 0x7fe0fc0a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.057+0000 7fe107e00640 1 -- 172.21.15.43:0/3158750856 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe108004c70 con 0x55d397d21ca0 2024-03-28T22:43:15.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.057+0000 7fe107e00640 1 -- 172.21.15.43:0/3158750856 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe10802c8b0 con 0x55d397d21ca0 2024-03-28T22:43:15.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.057+0000 7fe10f400640 1 -- 172.21.15.43:0/2405507862 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fe1100ce570 con 0x7fe118109900 2024-03-28T22:43:15.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.057+0000 7fe107e00640 1 -- 172.21.15.43:0/3158750856 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe108005550 con 0x55d397d21ca0 2024-03-28T22:43:15.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.057+0000 7fe116000640 1 --2- 172.21.15.43:0/2405507862 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe0fc0a1f70 0x7fe0fc0a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.058+0000 7fe116000640 1 --2- 172.21.15.43:0/2405507862 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe0fc0a1f70 0x7fe0fc0a4430 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x55d397d1cc40 tx=0x7fe100005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:15.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.058+0000 7fe120602740 1 -- 172.21.15.43:0/3158750856 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d397d21ca0 msgr2=0x55d397d2a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.058+0000 7fe120602740 1 --2- 172.21.15.43:0/3158750856 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d397d21ca0 0x55d397d2a340 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fe108036040 tx=0x7fe108005e60 comp rx=0 tx=0).stop 2024-03-28T22:43:15.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.058+0000 7fe120602740 1 -- 172.21.15.43:0/3158750856 shutdown_connections 2024-03-28T22:43:15.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.058+0000 7fe120602740 1 --2- 172.21.15.43:0/3158750856 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d397d2a880 0x55d397d2cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.058+0000 7fe120602740 1 --2- 172.21.15.43:0/3158750856 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d397d21ca0 0x55d397d2a340 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.058+0000 7fe120602740 1 --2- 172.21.15.43:0/3158750856 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d397d21380 0x55d397d21760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.058+0000 7fe120602740 1 -- 172.21.15.43:0/3158750856 >> 172.21.15.43:0/3158750856 conn(0x55d397d1bab0 msgr2=0x55d397d1be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:15.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.058+0000 7fe120602740 1 -- 172.21.15.43:0/3158750856 shutdown_connections 2024-03-28T22:43:15.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.058+0000 7fe120602740 1 -- 172.21.15.43:0/3158750856 wait complete. 2024-03-28T22:43:15.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.058+0000 7fe120602740 1 -- start start 2024-03-28T22:43:15.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.058+0000 7fe120602740 1 Processor -- start 2024-03-28T22:43:15.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.059+0000 7fe120602740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d397d21380 0x55d397d27890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.059+0000 7fe120602740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d397d21ca0 0x55d397d27dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.059+0000 7fe120602740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d397d2a880 0x55d397d28310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.059+0000 7fe120602740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d397d2d610 con 0x55d397d21380 2024-03-28T22:43:15.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.059+0000 7fe120602740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55d397c232f0 con 0x55d397d21ca0 2024-03-28T22:43:15.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.059+0000 7fe120602740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55d397d2d790 con 0x55d397d2a880 2024-03-28T22:43:15.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.059+0000 7fe120602740 10 client.0 _ll_register_callbacks cb 0x55d397998be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:15.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.059+0000 7fe116000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d397d21ca0 0x55d397d27dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.059+0000 7fe117400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d397d2a880 0x55d397d28310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.059+0000 7fe116a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d397d21380 0x55d397d27890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.059+0000 7fe116000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d397d21ca0 0x55d397d27dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:50342/0 (socket says 172.21.15.43:50342) 2024-03-28T22:43:15.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.059+0000 7fe116000640 1 -- 172.21.15.43:0/3753984121 learned_addr learned my addr 172.21.15.43:0/3753984121 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:15.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.060+0000 7fe116000640 1 -- 172.21.15.43:0/3753984121 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d397d2a880 msgr2=0x55d397d28310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.060+0000 7fe116000640 1 --2- 172.21.15.43:0/3753984121 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d397d2a880 0x55d397d28310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.060+0000 7fe116000640 1 -- 172.21.15.43:0/3753984121 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d397d21380 msgr2=0x55d397d27890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.060+0000 7fe116000640 1 --2- 172.21.15.43:0/3753984121 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d397d21380 0x55d397d27890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.060+0000 7fe116000640 1 -- 172.21.15.43:0/3753984121 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d397a637d0 con 0x55d397d21ca0 2024-03-28T22:43:15.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.060+0000 7fe116000640 1 --2- 172.21.15.43:0/3753984121 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d397d21ca0 0x55d397d27dd0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fe10000fd10 tx=0x7fe100005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3aeac8f9acb6c600 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:15.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.060+0000 7fe104c00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:43:15.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.060+0000 7fe104c00640 1 -- 172.21.15.43:0/3753984121 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10002d060 con 0x55d397d21ca0 2024-03-28T22:43:15.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.060+0000 7fe120602740 1 -- 172.21.15.43:0/3753984121 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe10802c8b0 con 0x55d397d21ca0 2024-03-28T22:43:15.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.060+0000 7fe104c00640 1 -- 172.21.15.43:0/3753984121 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fe10001f9b0 con 0x55d397d21ca0 2024-03-28T22:43:15.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.061+0000 7fe104c00640 1 -- 172.21.15.43:0/3753984121 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe100034050 con 0x55d397d21ca0 2024-03-28T22:43:15.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.061+0000 7fe120602740 10 client.27845 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:15.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.061+0000 7fe120602740 1 -- 172.21.15.43:0/3753984121 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d397d39900 con 0x55d397d21ca0 2024-03-28T22:43:15.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.061+0000 7fe104c00640 1 -- 172.21.15.43:0/3753984121 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe10802c8b0 con 0x55d397d21ca0 2024-03-28T22:43:15.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.061+0000 7fe104c00640 1 -- 172.21.15.43:0/3753984121 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe10001db20 con 0x55d397d21ca0 2024-03-28T22:43:15.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.061+0000 7fe120602740 10 client.27845 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:15.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.061+0000 7fe120602740 10 client.27845 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:43:15.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.061+0000 7fe120602740 20 client.27845 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:15.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.061+0000 7fe120602740 1 --2- 172.21.15.43:0/3753984121 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55d397d3adc0 0x55d397d3d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.061+0000 7fe120602740 4 client.27845 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:43:15.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.061+0000 7fe120602740 1 -- 172.21.15.43:0/3753984121 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d397c2e680 con 0x55d397d3adc0 2024-03-28T22:43:15.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.062+0000 7fe116a00640 1 --2- 172.21.15.43:0/3753984121 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55d397d3adc0 0x55d397d3d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.062+0000 7fe116a00640 1 --2- 172.21.15.43:0/3753984121 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55d397d3adc0 0x55d397d3d1f0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1c0aa3128c85f943 server_cookie=b3753d71de57de39 in_seq=0 out_seq=0 2024-03-28T22:43:15.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.062+0000 7fe104c00640 10 client.27845 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:43:15.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.063+0000 7fe104c00640 1 -- 172.21.15.43:0/3753984121 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d397c2e680 con 0x55d397d3adc0 2024-03-28T22:43:15.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.063+0000 7fe104c00640 10 client.27845 handle_command_reply: tid=0 2024-03-28T22:43:15.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.063+0000 7fe120602740 1 client.27845 shutdown 2024-03-28T22:43:15.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.063+0000 7fe120602740 1 -- 172.21.15.43:0/3753984121 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d397d21ca0 msgr2=0x55d397d27dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.063+0000 7fe120602740 1 --2- 172.21.15.43:0/3753984121 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d397d21ca0 0x55d397d27dd0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fe10000fd10 tx=0x7fe100005850 comp rx=0 tx=0).stop 2024-03-28T22:43:15.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.063+0000 7fe120602740 1 -- 172.21.15.43:0/3753984121 shutdown_connections 2024-03-28T22:43:15.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.063+0000 7fe120602740 1 --2- 172.21.15.43:0/3753984121 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55d397d3adc0 0x55d397d3d1f0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.063+0000 7fe120602740 1 --2- 172.21.15.43:0/3753984121 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d397d21ca0 0x55d397d27dd0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.063+0000 7fe120602740 1 --2- 172.21.15.43:0/3753984121 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d397d2a880 0x55d397d28310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.063+0000 7fe120602740 1 --2- 172.21.15.43:0/3753984121 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d397d21380 0x55d397d27890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.063+0000 7fe104c00640 0 client.27845 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:43:15.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.063+0000 7fe120602740 1 -- 172.21.15.43:0/3753984121 >> 172.21.15.43:0/3753984121 conn(0x55d397d1bab0 msgr2=0x55d397d1d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:15.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.063+0000 7fe120602740 1 -- 172.21.15.43:0/3753984121 shutdown_connections 2024-03-28T22:43:15.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.063+0000 7fe120602740 1 -- 172.21.15.43:0/3753984121 wait complete. 2024-03-28T22:43:15.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.064+0000 7fe120602740 20 client.27845 trim_cache size 0 max 16384 2024-03-28T22:43:15.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.078+0000 7fe120602740 1 Processor -- start 2024-03-28T22:43:15.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.078+0000 7fe120602740 1 -- start start 2024-03-28T22:43:15.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.079+0000 7fe120602740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d397d21380 0x55d397d36410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.079+0000 7fe120602740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d397d21ca0 0x55d397d42b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.079+0000 7fe120602740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d397d2a880 0x55d397d1ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.079+0000 7fe120602740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe1181b0d20 con 0x55d397d21ca0 2024-03-28T22:43:15.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.079+0000 7fe120602740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe1181b2860 con 0x55d397d2a880 2024-03-28T22:43:15.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.079+0000 7fe120602740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe1181b13f0 con 0x55d397d21380 2024-03-28T22:43:15.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.079+0000 7fe117400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d397d2a880 0x55d397d1ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.079+0000 7fe116a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d397d21380 0x55d397d36410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.079+0000 7fe117400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d397d2a880 0x55d397d1ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:50356/0 (socket says 172.21.15.43:50356) 2024-03-28T22:43:15.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.079+0000 7fe117400640 1 -- 172.21.15.43:0/3098414806 learned_addr learned my addr 172.21.15.43:0/3098414806 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:15.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.079+0000 7fe116000640 1 --2- 172.21.15.43:0/3098414806 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d397d21ca0 0x55d397d42b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.079+0000 7fe117400640 1 -- 172.21.15.43:0/3098414806 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d397d21380 msgr2=0x55d397d36410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.079+0000 7fe117400640 1 --2- 172.21.15.43:0/3098414806 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d397d21380 0x55d397d36410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.079+0000 7fe117400640 1 -- 172.21.15.43:0/3098414806 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d397d21ca0 msgr2=0x55d397d42b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.079+0000 7fe117400640 1 --2- 172.21.15.43:0/3098414806 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d397d21ca0 0x55d397d42b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.079+0000 7fe117400640 1 -- 172.21.15.43:0/3098414806 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d397d1b8e0 con 0x55d397d2a880 2024-03-28T22:43:15.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.079+0000 7fe116a00640 1 --2- 172.21.15.43:0/3098414806 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d397d21380 0x55d397d36410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:15.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.080+0000 7fe117400640 1 --2- 172.21.15.43:0/3098414806 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d397d2a880 0x55d397d1ce90 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fe11009aba0 tx=0x7fe11009ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0c7c5b3b3edb917 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:15.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.080+0000 7fe107400640 1 -- 172.21.15.43:0/3098414806 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe11001f030 con 0x55d397d2a880 2024-03-28T22:43:15.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.080+0000 7fe107400640 1 -- 172.21.15.43:0/3098414806 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe11000ce10 con 0x55d397d2a880 2024-03-28T22:43:15.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.080+0000 7fe107400640 1 -- 172.21.15.43:0/3098414806 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1100ce100 con 0x55d397d2a880 2024-03-28T22:43:15.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.081+0000 7fe120602740 1 -- 172.21.15.43:0/3098414806 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d397d2a880 msgr2=0x55d397d1ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.081+0000 7fe120602740 1 --2- 172.21.15.43:0/3098414806 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d397d2a880 0x55d397d1ce90 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fe11009aba0 tx=0x7fe11009ecb0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.081+0000 7fe120602740 1 -- 172.21.15.43:0/3098414806 shutdown_connections 2024-03-28T22:43:15.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.081+0000 7fe120602740 1 --2- 172.21.15.43:0/3098414806 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d397d2a880 0x55d397d1ce90 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.081+0000 7fe120602740 1 --2- 172.21.15.43:0/3098414806 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d397d21ca0 0x55d397d42b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.081+0000 7fe120602740 1 --2- 172.21.15.43:0/3098414806 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d397d21380 0x55d397d36410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.081+0000 7fe120602740 1 -- 172.21.15.43:0/3098414806 >> 172.21.15.43:0/3098414806 conn(0x55d397d1bab0 msgr2=0x55d397d261a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:15.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.081+0000 7fe120602740 1 -- 172.21.15.43:0/3098414806 shutdown_connections 2024-03-28T22:43:15.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.081+0000 7fe120602740 1 -- 172.21.15.43:0/3098414806 wait complete. 2024-03-28T22:43:15.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.081+0000 7fe120602740 1 -- start start 2024-03-28T22:43:15.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.082+0000 7fe120602740 1 Processor -- start 2024-03-28T22:43:15.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.082+0000 7fe120602740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d397d21380 0x55d397d32d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.082+0000 7fe120602740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d397d21ca0 0x55d397d241e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.082+0000 7fe120602740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d397d2a880 0x55d397d24810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.082+0000 7fe120602740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe1181b2190 con 0x55d397d21ca0 2024-03-28T22:43:15.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.082+0000 7fe120602740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe1181b1ac0 con 0x55d397d2a880 2024-03-28T22:43:15.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.082+0000 7fe120602740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe1181aed00 con 0x55d397d21380 2024-03-28T22:43:15.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.082+0000 7fe120602740 10 client.0 _ll_register_callbacks cb 0x7fe100009b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:15.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.082+0000 7fe116a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d397d21380 0x55d397d32d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.082+0000 7fe116a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d397d21380 0x55d397d32d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:49348/0 (socket says 172.21.15.43:49348) 2024-03-28T22:43:15.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.082+0000 7fe117400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d397d2a880 0x55d397d24810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.082+0000 7fe116a00640 1 -- 172.21.15.43:0/2905499880 learned_addr learned my addr 172.21.15.43:0/2905499880 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:15.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.082+0000 7fe116000640 1 --2- 172.21.15.43:0/2905499880 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d397d21ca0 0x55d397d241e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.083+0000 7fe117400640 1 -- 172.21.15.43:0/2905499880 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d397d21380 msgr2=0x55d397d32d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.083+0000 7fe117400640 1 --2- 172.21.15.43:0/2905499880 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d397d21380 0x55d397d32d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.083+0000 7fe117400640 1 -- 172.21.15.43:0/2905499880 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d397d21ca0 msgr2=0x55d397d241e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.083+0000 7fe117400640 1 --2- 172.21.15.43:0/2905499880 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d397d21ca0 0x55d397d241e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.083+0000 7fe117400640 1 -- 172.21.15.43:0/2905499880 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d397d24fe0 con 0x55d397d2a880 2024-03-28T22:43:15.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.083+0000 7fe116a00640 1 --2- 172.21.15.43:0/2905499880 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d397d21380 0x55d397d32d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:15.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.083+0000 7fe117400640 1 --2- 172.21.15.43:0/2905499880 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d397d2a880 0x55d397d24810 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fe11009aba0 tx=0x7fe11000ef30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9ffbf5342d826ae server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:15.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.083+0000 7fe104c00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:43:15.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.083+0000 7fe104c00640 1 -- 172.21.15.43:0/2905499880 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1100b7020 con 0x55d397d2a880 2024-03-28T22:43:15.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.083+0000 7fe104c00640 1 -- 172.21.15.43:0/2905499880 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fe1100a2e70 con 0x55d397d2a880 2024-03-28T22:43:15.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.083+0000 7fe120602740 1 -- 172.21.15.43:0/2905499880 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe11000ce10 con 0x55d397d2a880 2024-03-28T22:43:15.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.083+0000 7fe104c00640 1 -- 172.21.15.43:0/2905499880 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe11001f030 con 0x55d397d2a880 2024-03-28T22:43:15.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.083+0000 7fe120602740 10 client.27851 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:15.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.083+0000 7fe120602740 1 -- 172.21.15.43:0/2905499880 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d397da1230 con 0x55d397d2a880 2024-03-28T22:43:15.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.084+0000 7fe104c00640 1 -- 172.21.15.43:0/2905499880 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe11000ce10 con 0x55d397d2a880 2024-03-28T22:43:15.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.084+0000 7fe104c00640 1 -- 172.21.15.43:0/2905499880 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe11000f040 con 0x55d397d2a880 2024-03-28T22:43:15.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.084+0000 7fe120602740 10 client.27851 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:15.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.084+0000 7fe120602740 10 client.27851 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:43:15.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.084+0000 7fe120602740 20 client.27851 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:15.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.084+0000 7fe120602740 1 --2- 172.21.15.43:0/2905499880 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55d397da18f0 0x55d397da3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.084+0000 7fe120602740 4 client.27851 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:43:15.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.084+0000 7fe120602740 1 -- 172.21.15.43:0/2905499880 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d397c92590 con 0x55d397da18f0 2024-03-28T22:43:15.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.084+0000 7fe116a00640 1 --2- 172.21.15.43:0/2905499880 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55d397da18f0 0x55d397da3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.084+0000 7fe104c00640 10 client.27851 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:43:15.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.084+0000 7fe116a00640 1 --2- 172.21.15.43:0/2905499880 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55d397da18f0 0x55d397da3ce0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ace97155707f974 server_cookie=5a9fd3107b35d7d3 in_seq=0 out_seq=0 2024-03-28T22:43:15.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.085+0000 7fe104c00640 1 -- 172.21.15.43:0/2905499880 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d397c92590 con 0x55d397da18f0 2024-03-28T22:43:15.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.085+0000 7fe104c00640 10 client.27851 handle_command_reply: tid=0 2024-03-28T22:43:15.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.085+0000 7fe120602740 1 client.27851 shutdown 2024-03-28T22:43:15.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.085+0000 7fe120602740 1 -- 172.21.15.43:0/2905499880 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d397d2a880 msgr2=0x55d397d24810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.085+0000 7fe120602740 1 --2- 172.21.15.43:0/2905499880 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d397d2a880 0x55d397d24810 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fe11009aba0 tx=0x7fe11000ef30 comp rx=0 tx=0).stop 2024-03-28T22:43:15.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.085+0000 7fe120602740 1 -- 172.21.15.43:0/2905499880 shutdown_connections 2024-03-28T22:43:15.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.085+0000 7fe120602740 1 --2- 172.21.15.43:0/2905499880 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55d397da18f0 0x55d397da3ce0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.085+0000 7fe120602740 1 --2- 172.21.15.43:0/2905499880 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d397d2a880 0x55d397d24810 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.085+0000 7fe120602740 1 --2- 172.21.15.43:0/2905499880 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d397d21ca0 0x55d397d241e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.085+0000 7fe120602740 1 --2- 172.21.15.43:0/2905499880 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d397d21380 0x55d397d32d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.085+0000 7fe104c00640 0 client.27851 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:43:15.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.085+0000 7fe120602740 1 -- 172.21.15.43:0/2905499880 >> 172.21.15.43:0/2905499880 conn(0x55d397d1bab0 msgr2=0x55d397d27850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:15.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.085+0000 7fe120602740 1 -- 172.21.15.43:0/2905499880 shutdown_connections 2024-03-28T22:43:15.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.085+0000 7fe120602740 1 -- 172.21.15.43:0/2905499880 wait complete. 2024-03-28T22:43:15.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.085+0000 7fe120602740 20 client.27851 trim_cache size 0 max 16384 2024-03-28T22:43:15.089 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:43:15.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.086+0000 7fe11d800640 1 -- 172.21.15.43:0/2405507862 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe0fc0a1f70 msgr2=0x7fe0fc0a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.086+0000 7fe11d800640 1 --2- 172.21.15.43:0/2405507862 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe0fc0a1f70 0x7fe0fc0a4430 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x55d397d1cc40 tx=0x7fe100005f90 comp rx=0 tx=0).stop 2024-03-28T22:43:15.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.086+0000 7fe11d800640 1 -- 172.21.15.43:0/2405507862 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe118109900 msgr2=0x7fe11819eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.086+0000 7fe11d800640 1 --2- 172.21.15.43:0/2405507862 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe118109900 0x7fe11819eb60 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fe11000e940 tx=0x7fe11000ee00 comp rx=0 tx=0).stop 2024-03-28T22:43:15.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.086+0000 7fe11d800640 1 -- 172.21.15.43:0/2405507862 shutdown_connections 2024-03-28T22:43:15.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.086+0000 7fe11d800640 1 --2- 172.21.15.43:0/2405507862 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe0fc0a1f70 0x7fe0fc0a4430 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.086+0000 7fe11d800640 1 --2- 172.21.15.43:0/2405507862 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe118109900 0x7fe11819eb60 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.086+0000 7fe11d800640 1 --2- 172.21.15.43:0/2405507862 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe1181042b0 0x7fe11819a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.086+0000 7fe11d800640 1 --2- 172.21.15.43:0/2405507862 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe1181038f0 0x7fe11819a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.086+0000 7fe11d800640 1 -- 172.21.15.43:0/2405507862 >> 172.21.15.43:0/2405507862 conn(0x7fe1180ff3c0 msgr2=0x7fe1180ffa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:15.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.086+0000 7fe11d800640 1 -- 172.21.15.43:0/2405507862 shutdown_connections 2024-03-28T22:43:15.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.086+0000 7fe11d800640 1 -- 172.21.15.43:0/2405507862 wait complete. 2024-03-28T22:43:15.093 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.091+0000 7fd40ae00640 1 -- 172.21.15.43:0/2291830678 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd404103740 msgr2=0x7fd404103bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.093 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.091+0000 7fd40ae00640 1 --2- 172.21.15.43:0/2291830678 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd404103740 0x7fd404103bc0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fd3f0009900 tx=0x7fd3f002bc40 comp rx=0 tx=0).stop 2024-03-28T22:43:15.093 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.091+0000 7fd40ae00640 1 -- 172.21.15.43:0/2291830678 shutdown_connections 2024-03-28T22:43:15.093 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.091+0000 7fd40ae00640 1 --2- 172.21.15.43:0/2291830678 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd404103740 0x7fd404103bc0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.093 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.091+0000 7fd40ae00640 1 --2- 172.21.15.43:0/2291830678 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd404104100 0x7fd404110610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.093 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.091+0000 7fd40ae00640 1 --2- 172.21.15.43:0/2291830678 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd404109750 0x7fd404109b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.093 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.091+0000 7fd40ae00640 1 -- 172.21.15.43:0/2291830678 >> 172.21.15.43:0/2291830678 conn(0x7fd4040ff230 msgr2=0x7fd404101650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:15.093 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.091+0000 7fd40ae00640 1 -- 172.21.15.43:0/2291830678 shutdown_connections 2024-03-28T22:43:15.093 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.091+0000 7fd40ae00640 1 -- 172.21.15.43:0/2291830678 wait complete. 2024-03-28T22:43:15.094 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.092+0000 7fd40ae00640 1 Processor -- start 2024-03-28T22:43:15.094 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.093+0000 7fd40ae00640 1 -- start start 2024-03-28T22:43:15.094 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.093+0000 7fd40ae00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd404103740 0x7fd40419a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.094 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.093+0000 7fd40ae00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd404104100 0x7fd40419a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.094 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.093+0000 7fd40ae00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd404109750 0x7fd40419e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.094 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.093+0000 7fd40ae00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd404113b70 con 0x7fd404109750 2024-03-28T22:43:15.094 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.093+0000 7fd40ae00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd4041139f0 con 0x7fd404104100 2024-03-28T22:43:15.094 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.093+0000 7fd40ae00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd404113cf0 con 0x7fd404103740 2024-03-28T22:43:15.094 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.093+0000 7fd409000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd404104100 0x7fd40419a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.094 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.093+0000 7fd409000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd404104100 0x7fd40419a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:50382/0 (socket says 172.21.15.43:50382) 2024-03-28T22:43:15.094 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.093+0000 7fd409a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd404103740 0x7fd40419a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.094 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.094+0000 7fd409a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd404103740 0x7fd40419a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:49370/0 (socket says 172.21.15.43:49370) 2024-03-28T22:43:15.094 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.094+0000 7fd409000640 1 -- 172.21.15.43:0/1191838191 learned_addr learned my addr 172.21.15.43:0/1191838191 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:15.095 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.094+0000 7fd40a400640 1 --2- 172.21.15.43:0/1191838191 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd404109750 0x7fd40419e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.095 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.094+0000 7fd409000640 1 -- 172.21.15.43:0/1191838191 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd404103740 msgr2=0x7fd40419a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.095 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.094+0000 7fd409000640 1 --2- 172.21.15.43:0/1191838191 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd404103740 0x7fd40419a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.095 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.094+0000 7fd409000640 1 -- 172.21.15.43:0/1191838191 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd404109750 msgr2=0x7fd40419e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.095 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.094+0000 7fd409000640 1 --2- 172.21.15.43:0/1191838191 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd404109750 0x7fd40419e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.095 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.094+0000 7fd409000640 1 -- 172.21.15.43:0/1191838191 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd40419f020 con 0x7fd404104100 2024-03-28T22:43:15.095 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.094+0000 7fd40a400640 1 --2- 172.21.15.43:0/1191838191 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd404109750 0x7fd40419e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:43:15.095 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.094+0000 7fd409000640 1 --2- 172.21.15.43:0/1191838191 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd404104100 0x7fd40419a570 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fd3f0005ab0 tx=0x7fd3f0004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:15.095 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.094+0000 7fd402000640 1 -- 172.21.15.43:0/1191838191 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f0041070 con 0x7fd404104100 2024-03-28T22:43:15.095 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.095+0000 7fd402000640 1 -- 172.21.15.43:0/1191838191 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3f00054e0 con 0x7fd404104100 2024-03-28T22:43:15.095 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.095+0000 7fd402000640 1 -- 172.21.15.43:0/1191838191 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f0003cc0 con 0x7fd404104100 2024-03-28T22:43:15.095 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.095+0000 7fd40ae00640 1 -- 172.21.15.43:0/1191838191 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd40419f2b0 con 0x7fd404104100 2024-03-28T22:43:15.095 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.095+0000 7fd40ae00640 1 -- 172.21.15.43:0/1191838191 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4041a6ba0 con 0x7fd404104100 2024-03-28T22:43:15.096 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.097+0000 7fd402000640 1 -- 172.21.15.43:0/1191838191 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fd3f0044070 con 0x7fd404104100 2024-03-28T22:43:15.097 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.098+0000 7fd40f187740 1 Processor -- start 2024-03-28T22:43:15.097 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.098+0000 7fd40f187740 1 -- start start 2024-03-28T22:43:15.097 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.099+0000 7fd40f187740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557858582570 0x557858582950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.097 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.099+0000 7fd40f187740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557858582e90 0x55785858b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.097 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.099+0000 7fd40f187740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55785858bc60 0x55785858e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.097 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.099+0000 7fd40f187740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5578584e5640 con 0x557858582570 2024-03-28T22:43:15.097 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.099+0000 7fd40f187740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55785848e810 con 0x557858582e90 2024-03-28T22:43:15.097 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.099+0000 7fd40f187740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5578584e16e0 con 0x55785858bc60 2024-03-28T22:43:15.097 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.099+0000 7fd409000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55785858bc60 0x55785858e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.097 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.099+0000 7fd402000640 1 --2- 172.21.15.43:0/1191838191 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd3ec0a1f60 0x7fd3ec0a4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.097 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.099+0000 7fd40a400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557858582570 0x557858582950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.097 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.099+0000 7fd409000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55785858bc60 0x55785858e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:49386/0 (socket says 172.21.15.43:49386) 2024-03-28T22:43:15.097 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.099+0000 7fd40a400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557858582570 0x557858582950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46004/0 (socket says 172.21.15.43:46004) 2024-03-28T22:43:15.097 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.099+0000 7fd409000640 1 -- 172.21.15.43:0/2161717213 learned_addr learned my addr 172.21.15.43:0/2161717213 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:15.098 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.099+0000 7fd409a00640 1 --2- 172.21.15.43:0/2161717213 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557858582e90 0x55785858b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.099 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.100+0000 7fd402000640 1 -- 172.21.15.43:0/1191838191 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fd3f00f0190 con 0x7fd404104100 2024-03-28T22:43:15.099 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.100+0000 7fd409000640 1 -- 172.21.15.43:0/2161717213 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557858582e90 msgr2=0x55785858b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.099 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.100+0000 7fd409000640 1 --2- 172.21.15.43:0/2161717213 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557858582e90 0x55785858b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.099 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.100+0000 7fd409000640 1 -- 172.21.15.43:0/2161717213 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557858582570 msgr2=0x557858582950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.099 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.100+0000 7fd409000640 1 --2- 172.21.15.43:0/2161717213 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557858582570 0x557858582950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.099 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.100+0000 7fd409000640 1 -- 172.21.15.43:0/2161717213 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578581fa210 con 0x55785858bc60 2024-03-28T22:43:15.099 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.100+0000 7fd40a400640 1 --2- 172.21.15.43:0/1191838191 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd3ec0a1f60 0x7fd3ec0a4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.100 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.100+0000 7fd409000640 1 --2- 172.21.15.43:0/2161717213 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55785858bc60 0x55785858e050 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd3f00ed650 tx=0x7fd3f0003760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cf3b02cb19863c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:15.100 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.100+0000 7fd3f6a00640 1 -- 172.21.15.43:0/2161717213 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f004b070 con 0x55785858bc60 2024-03-28T22:43:15.100 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.100+0000 7fd3f6a00640 1 -- 172.21.15.43:0/2161717213 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3f002cce0 con 0x55785858bc60 2024-03-28T22:43:15.100 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.100+0000 7fd3f6a00640 1 -- 172.21.15.43:0/2161717213 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f00c5030 con 0x55785858bc60 2024-03-28T22:43:15.100 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.101+0000 7fd40f187740 1 -- 172.21.15.43:0/2161717213 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55785858bc60 msgr2=0x55785858e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.100 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.101+0000 7fd40f187740 1 --2- 172.21.15.43:0/2161717213 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55785858bc60 0x55785858e050 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd3f00ed650 tx=0x7fd3f0003760 comp rx=0 tx=0).stop 2024-03-28T22:43:15.100 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.101+0000 7fd40a400640 1 --2- 172.21.15.43:0/1191838191 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd3ec0a1f60 0x7fd3ec0a4420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc005fd0 tx=0x7fd3fc014040 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:15.100 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.101+0000 7fd40f187740 1 -- 172.21.15.43:0/2161717213 shutdown_connections 2024-03-28T22:43:15.100 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.101+0000 7fd40f187740 1 --2- 172.21.15.43:0/2161717213 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557858582e90 0x55785858b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.100 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.101+0000 7fd40f187740 1 --2- 172.21.15.43:0/2161717213 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55785858bc60 0x55785858e050 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.100 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.101+0000 7fd40f187740 1 --2- 172.21.15.43:0/2161717213 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557858582570 0x557858582950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.100 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.101+0000 7fd40f187740 1 -- 172.21.15.43:0/2161717213 >> 172.21.15.43:0/2161717213 conn(0x55785857ce90 msgr2=0x55785857d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:15.100 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.101+0000 7fd40f187740 1 -- 172.21.15.43:0/2161717213 shutdown_connections 2024-03-28T22:43:15.100 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.101+0000 7fd40f187740 1 -- 172.21.15.43:0/2161717213 wait complete. 2024-03-28T22:43:15.100 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.102+0000 7fd40f187740 1 -- start start 2024-03-28T22:43:15.101 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.102+0000 7fd40f187740 1 Processor -- start 2024-03-28T22:43:15.101 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.102+0000 7fd40f187740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557858582570 0x557858588d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.101 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.102+0000 7fd40f187740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557858582e90 0x557858589260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.101 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.102+0000 7fd40f187740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55785858bc60 0x5578585897a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.101 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.102+0000 7fd40f187740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55785858e650 con 0x557858582570 2024-03-28T22:43:15.101 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.103+0000 7fd40f187740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x557858483d00 con 0x55785858bc60 2024-03-28T22:43:15.101 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.103+0000 7fd40f187740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55785858e7d0 con 0x557858582e90 2024-03-28T22:43:15.102 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.103+0000 7fd40f187740 10 client.0 _ll_register_callbacks cb 0x55785846e490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:15.102 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.103+0000 7fd409000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557858582e90 0x557858589260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.102 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.103+0000 7fd409000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557858582e90 0x557858589260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:49388/0 (socket says 172.21.15.43:49388) 2024-03-28T22:43:15.102 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.103+0000 7fd409000640 1 -- 172.21.15.43:0/4057314811 learned_addr learned my addr 172.21.15.43:0/4057314811 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:15.102 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.103+0000 7fd40a400640 1 --2- 172.21.15.43:0/4057314811 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55785858bc60 0x5578585897a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.102 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.103+0000 7fd409a00640 1 --2- 172.21.15.43:0/4057314811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557858582570 0x557858588d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.102 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:14 smithi043 ceph-mon[31823]: pgmap v186: 129 pgs: 129 active+clean; 57 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 283 op/s 2024-03-28T22:43:15.102 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:14 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/214692609' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:43:15.104 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.103+0000 7fd409000640 1 -- 172.21.15.43:0/4057314811 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55785858bc60 msgr2=0x5578585897a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.104 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.103+0000 7fd409000640 1 --2- 172.21.15.43:0/4057314811 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55785858bc60 0x5578585897a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.104 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.103+0000 7fd409000640 1 -- 172.21.15.43:0/4057314811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557858582570 msgr2=0x557858588d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.104 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.103+0000 7fd409000640 1 --2- 172.21.15.43:0/4057314811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557858582570 0x557858588d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.104 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.103+0000 7fd409000640 1 -- 172.21.15.43:0/4057314811 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578582c6620 con 0x557858582e90 2024-03-28T22:43:15.104 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.103+0000 7fd409a00640 1 --2- 172.21.15.43:0/4057314811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557858582570 0x557858588d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:43:15.104 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.103+0000 7fd409000640 1 --2- 172.21.15.43:0/4057314811 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557858582e90 0x557858589260 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fd3f00bc820 tx=0x7fd3f00a72e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c75119511d69215a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:15.104 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.103+0000 7fd3eb400640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:43:15.104 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.104+0000 7fd3eb400640 1 -- 172.21.15.43:0/4057314811 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f004b050 con 0x557858582e90 2024-03-28T22:43:15.104 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.104+0000 7fd3eb400640 1 -- 172.21.15.43:0/4057314811 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fd3f00c4ba0 con 0x557858582e90 2024-03-28T22:43:15.104 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.104+0000 7fd40f187740 1 -- 172.21.15.43:0/4057314811 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5578584f9030 con 0x557858582e90 2024-03-28T22:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.104+0000 7fd3eb400640 1 -- 172.21.15.43:0/4057314811 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd3f00da050 con 0x557858582e90 2024-03-28T22:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.104+0000 7fd40f187740 10 client.27805 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.104+0000 7fd40f187740 1 -- 172.21.15.43:0/4057314811 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55785858e650 con 0x557858582e90 2024-03-28T22:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.104+0000 7fd3eb400640 1 -- 172.21.15.43:0/4057314811 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5578584f9030 con 0x557858582e90 2024-03-28T22:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.104+0000 7fd3eb400640 1 -- 172.21.15.43:0/4057314811 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd3f00d5200 con 0x557858582e90 2024-03-28T22:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.105+0000 7fd40f187740 10 client.27805 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.105+0000 7fd40f187740 10 client.27805 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.105+0000 7fd40f187740 20 client.27805 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.105+0000 7fd40f187740 1 --2- 172.21.15.43:0/4057314811 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55785859b710 0x55785859db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.105+0000 7fd40f187740 4 client.27805 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:43:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.105+0000 7fd40f187740 1 -- 172.21.15.43:0/4057314811 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5578584f2b80 con 0x55785859b710 2024-03-28T22:43:15.106 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T22:43:15.108 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.105+0000 7fd409a00640 1 --2- 172.21.15.43:0/4057314811 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55785859b710 0x55785859db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.108 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.105+0000 7fd409a00640 1 --2- 172.21.15.43:0/4057314811 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55785859b710 0x55785859db20 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40098990e427fd8d server_cookie=c86d3524d1c66820 in_seq=0 out_seq=0 2024-03-28T22:43:15.108 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.105+0000 7fd3eb400640 10 client.27805 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:15.108 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.106+0000 7fd3eb400640 1 -- 172.21.15.43:0/4057314811 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5578584f2b80 con 0x55785859b710 2024-03-28T22:43:15.108 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.106+0000 7fd3eb400640 10 client.27805 handle_command_reply: tid=0 2024-03-28T22:43:15.108 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.106+0000 7fd40f187740 1 client.27805 shutdown 2024-03-28T22:43:15.108 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.106+0000 7fd40f187740 1 -- 172.21.15.43:0/4057314811 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557858582e90 msgr2=0x557858589260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.108 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.106+0000 7fd40f187740 1 --2- 172.21.15.43:0/4057314811 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557858582e90 0x557858589260 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fd3f00bc820 tx=0x7fd3f00a72e0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.106+0000 7fd40f187740 1 -- 172.21.15.43:0/4057314811 shutdown_connections 2024-03-28T22:43:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.106+0000 7fd40f187740 1 --2- 172.21.15.43:0/4057314811 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55785859b710 0x55785859db20 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.106+0000 7fd40f187740 1 --2- 172.21.15.43:0/4057314811 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55785858bc60 0x5578585897a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.106+0000 7fd40f187740 1 --2- 172.21.15.43:0/4057314811 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557858582e90 0x557858589260 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.106+0000 7fd40f187740 1 --2- 172.21.15.43:0/4057314811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557858582570 0x557858588d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.106+0000 7fd3eb400640 0 client.27805 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.106+0000 7fd40f187740 1 -- 172.21.15.43:0/4057314811 >> 172.21.15.43:0/4057314811 conn(0x55785857ce90 msgr2=0x55785857ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.107+0000 7fd40f187740 1 -- 172.21.15.43:0/4057314811 shutdown_connections 2024-03-28T22:43:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.107+0000 7fd40f187740 1 -- 172.21.15.43:0/4057314811 wait complete. 2024-03-28T22:43:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.107+0000 7fd40f187740 20 client.27805 trim_cache size 0 max 16384 2024-03-28T22:43:15.122 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.124+0000 7fd40f187740 1 Processor -- start 2024-03-28T22:43:15.122 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.124+0000 7fd40f187740 1 -- start start 2024-03-28T22:43:15.122 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.124+0000 7fd40f187740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557858582570 0x55785858b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.122 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.124+0000 7fd40f187740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557858582e90 0x557858597360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.122 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.124+0000 7fd40f187740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55785858bc60 0x5578585a3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.122 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.124+0000 7fd40f187740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd4041b0b60 con 0x55785858bc60 2024-03-28T22:43:15.122 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.124+0000 7fd40f187740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd4041b26a0 con 0x557858582570 2024-03-28T22:43:15.122 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.124+0000 7fd40f187740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd4041b1230 con 0x557858582e90 2024-03-28T22:43:15.122 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.124+0000 7fd409000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557858582e90 0x557858597360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.122 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.124+0000 7fd409a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557858582570 0x55785858b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.123 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.124+0000 7fd409000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557858582e90 0x557858597360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:49396/0 (socket says 172.21.15.43:49396) 2024-03-28T22:43:15.123 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.124+0000 7fd409000640 1 -- 172.21.15.43:0/1600990676 learned_addr learned my addr 172.21.15.43:0/1600990676 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:15.123 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.124+0000 7fd40a400640 1 --2- 172.21.15.43:0/1600990676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55785858bc60 0x5578585a3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.123 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.125+0000 7fd409000640 1 -- 172.21.15.43:0/1600990676 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557858582570 msgr2=0x55785858b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.123 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.125+0000 7fd409000640 1 --2- 172.21.15.43:0/1600990676 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557858582570 0x55785858b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.123 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.125+0000 7fd409000640 1 -- 172.21.15.43:0/1600990676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55785858bc60 msgr2=0x5578585a3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.123 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.125+0000 7fd409000640 1 --2- 172.21.15.43:0/1600990676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55785858bc60 0x5578585a3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.123 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.125+0000 7fd409000640 1 -- 172.21.15.43:0/1600990676 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55785858e3c0 con 0x557858582e90 2024-03-28T22:43:15.123 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.125+0000 7fd40a400640 1 --2- 172.21.15.43:0/1600990676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55785858bc60 0x5578585a3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:43:15.127 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.125+0000 7fd409000640 1 --2- 172.21.15.43:0/1600990676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557858582e90 0x557858597360 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fd3f00f0480 tx=0x7fd3f00ed5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=329771349eb54dde server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:15.127 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.125+0000 7fd3f6000640 1 -- 172.21.15.43:0/1600990676 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f004b050 con 0x557858582e90 2024-03-28T22:43:15.127 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.125+0000 7fd3f6000640 1 -- 172.21.15.43:0/1600990676 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3f00a0020 con 0x557858582e90 2024-03-28T22:43:15.127 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.125+0000 7fd3f6000640 1 -- 172.21.15.43:0/1600990676 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f00bc3d0 con 0x557858582e90 2024-03-28T22:43:15.127 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.126+0000 7fd40f187740 1 -- 172.21.15.43:0/1600990676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557858582e90 msgr2=0x557858597360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.127 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.126+0000 7fd40f187740 1 --2- 172.21.15.43:0/1600990676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557858582e90 0x557858597360 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fd3f00f0480 tx=0x7fd3f00ed5d0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.127 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.126+0000 7fd40f187740 1 -- 172.21.15.43:0/1600990676 shutdown_connections 2024-03-28T22:43:15.127 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.126+0000 7fd40f187740 1 --2- 172.21.15.43:0/1600990676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55785858bc60 0x5578585a3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.128 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.126+0000 7fd40f187740 1 --2- 172.21.15.43:0/1600990676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557858582e90 0x557858597360 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.128 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.126+0000 7fd40f187740 1 --2- 172.21.15.43:0/1600990676 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557858582570 0x55785858b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.128 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.126+0000 7fd40f187740 1 -- 172.21.15.43:0/1600990676 >> 172.21.15.43:0/1600990676 conn(0x55785857ce90 msgr2=0x5578585875f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:15.128 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.126+0000 7fd40f187740 1 -- 172.21.15.43:0/1600990676 shutdown_connections 2024-03-28T22:43:15.128 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.126+0000 7fd40f187740 1 -- 172.21.15.43:0/1600990676 wait complete. 2024-03-28T22:43:15.128 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.127+0000 7fd40f187740 1 -- start start 2024-03-28T22:43:15.128 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.127+0000 7fd40f187740 1 Processor -- start 2024-03-28T22:43:15.128 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.127+0000 7fd40f187740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557858582570 0x557858585940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.128 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.127+0000 7fd40f187740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557858582e90 0x557858585e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.128 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.127+0000 7fd40f187740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55785858bc60 0x5578585863c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.128 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.127+0000 7fd40f187740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd4041b1fd0 con 0x557858582570 2024-03-28T22:43:15.128 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.127+0000 7fd40f187740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd4041b1900 con 0x557858582e90 2024-03-28T22:43:15.128 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.127+0000 7fd40f187740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd4041aeb40 con 0x55785858bc60 2024-03-28T22:43:15.128 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.127+0000 7fd40f187740 10 client.0 _ll_register_callbacks cb 0x7fd3fc009a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:15.129 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.128+0000 7fd409000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557858582e90 0x557858585e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.129 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.128+0000 7fd40a400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55785858bc60 0x5578585863c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.129 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.128+0000 7fd409000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557858582e90 0x557858585e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:50422/0 (socket says 172.21.15.43:50422) 2024-03-28T22:43:15.129 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.128+0000 7fd40a400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55785858bc60 0x5578585863c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:49412/0 (socket says 172.21.15.43:49412) 2024-03-28T22:43:15.129 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.128+0000 7fd409000640 1 -- 172.21.15.43:0/78580987 learned_addr learned my addr 172.21.15.43:0/78580987 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:15.129 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.128+0000 7fd409a00640 1 --2- 172.21.15.43:0/78580987 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557858582570 0x557858585940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.129 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.128+0000 7fd409000640 1 -- 172.21.15.43:0/78580987 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55785858bc60 msgr2=0x5578585863c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.129 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.128+0000 7fd409000640 1 --2- 172.21.15.43:0/78580987 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55785858bc60 0x5578585863c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.129 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.128+0000 7fd409000640 1 -- 172.21.15.43:0/78580987 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557858582570 msgr2=0x557858585940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.129 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.128+0000 7fd409000640 1 --2- 172.21.15.43:0/78580987 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557858582570 0x557858585940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.129 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.128+0000 7fd409000640 1 -- 172.21.15.43:0/78580987 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557858600a80 con 0x557858582e90 2024-03-28T22:43:15.129 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.128+0000 7fd40a400640 1 --2- 172.21.15.43:0/78580987 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55785858bc60 0x5578585863c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:15.129 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.128+0000 7fd409000640 1 --2- 172.21.15.43:0/78580987 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557858582e90 0x557858585e80 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fd3f00bc6e0 tx=0x7fd3f0005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c90cd71bafcc9f2a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:15.129 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.128+0000 7fd3eb400640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:43:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.129+0000 7fd3eb400640 1 -- 172.21.15.43:0/78580987 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f004b070 con 0x557858582e90 2024-03-28T22:43:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.129+0000 7fd3eb400640 1 -- 172.21.15.43:0/78580987 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fd3f00bbef0 con 0x557858582e90 2024-03-28T22:43:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.129+0000 7fd40f187740 1 -- 172.21.15.43:0/78580987 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3f00a0020 con 0x557858582e90 2024-03-28T22:43:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.129+0000 7fd3eb400640 1 -- 172.21.15.43:0/78580987 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd3f00ed970 con 0x557858582e90 2024-03-28T22:43:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.129+0000 7fd40f187740 10 client.27869 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.129+0000 7fd40f187740 1 -- 172.21.15.43:0/78580987 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557858600a80 con 0x557858582e90 2024-03-28T22:43:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.129+0000 7fd3eb400640 1 -- 172.21.15.43:0/78580987 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3f00a0020 con 0x557858582e90 2024-03-28T22:43:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.129+0000 7fd3eb400640 1 -- 172.21.15.43:0/78580987 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd3f00bb750 con 0x557858582e90 2024-03-28T22:43:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.129+0000 7fd40f187740 10 client.27869 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.129+0000 7fd40f187740 10 client.27869 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:43:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.129+0000 7fd40f187740 20 client.27869 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.129+0000 7fd40f187740 1 --2- 172.21.15.43:0/78580987 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557858602020 0x557858604440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.129+0000 7fd40f187740 4 client.27869 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T22:43:15.130 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.129+0000 7fd40f187740 1 -- 172.21.15.43:0/78580987 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5578584eced0 con 0x557858602020 2024-03-28T22:43:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.130+0000 7fd409a00640 1 --2- 172.21.15.43:0/78580987 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557858602020 0x557858604440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.130+0000 7fd409a00640 1 --2- 172.21.15.43:0/78580987 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557858602020 0x557858604440 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=290e704fc1a5d2e4 server_cookie=6c65f4d301c2756d in_seq=0 out_seq=0 2024-03-28T22:43:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.130+0000 7fd3eb400640 10 client.27869 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:15.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:14 smithi082 ceph-mon[36471]: pgmap v186: 129 pgs: 129 active+clean; 57 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 283 op/s 2024-03-28T22:43:15.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:14 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/214692609' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:43:15.276 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.278+0000 7fd3eb400640 1 -- 172.21.15.43:0/78580987 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5578584eced0 con 0x557858602020 2024-03-28T22:43:15.276 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.278+0000 7fd3eb400640 10 client.27869 handle_command_reply: tid=0 2024-03-28T22:43:15.276 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.278+0000 7fd40f187740 1 client.27869 shutdown 2024-03-28T22:43:15.276 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.278+0000 7fd40f187740 1 -- 172.21.15.43:0/78580987 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557858582e90 msgr2=0x557858585e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.276 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.278+0000 7fd40f187740 1 --2- 172.21.15.43:0/78580987 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557858582e90 0x557858585e80 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fd3f00bc6e0 tx=0x7fd3f0005fd0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.276 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.278+0000 7fd40f187740 1 -- 172.21.15.43:0/78580987 shutdown_connections 2024-03-28T22:43:15.276 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.278+0000 7fd40f187740 1 --2- 172.21.15.43:0/78580987 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557858602020 0x557858604440 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.277 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.279+0000 7fd40f187740 1 --2- 172.21.15.43:0/78580987 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557858582e90 0x557858585e80 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.277 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.279+0000 7fd40f187740 1 --2- 172.21.15.43:0/78580987 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55785858bc60 0x5578585863c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.277 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.279+0000 7fd40f187740 1 --2- 172.21.15.43:0/78580987 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557858582570 0x557858585940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.277 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.279+0000 7fd40f187740 1 -- 172.21.15.43:0/78580987 >> 172.21.15.43:0/78580987 conn(0x55785857ce90 msgr2=0x557858588cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:15.277 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.279+0000 7fd3eb400640 0 client.27869 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:15.277 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.279+0000 7fd40f187740 1 -- 172.21.15.43:0/78580987 shutdown_connections 2024-03-28T22:43:15.277 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.279+0000 7fd40f187740 1 -- 172.21.15.43:0/78580987 wait complete. 2024-03-28T22:43:15.278 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.280+0000 7fd40f187740 20 client.27869 trim_cache size 0 max 16384 2024-03-28T22:43:15.278 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:{ 2024-03-28T22:43:15.278 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "message": "", 2024-03-28T22:43:15.278 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "return_code": 0 2024-03-28T22:43:15.278 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:} 2024-03-28T22:43:15.279 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.282+0000 7fd40ae00640 1 -- 172.21.15.43:0/1191838191 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd3ec0a1f60 msgr2=0x7fd3ec0a4420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.279 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.282+0000 7fd40ae00640 1 --2- 172.21.15.43:0/1191838191 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd3ec0a1f60 0x7fd3ec0a4420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc005fd0 tx=0x7fd3fc014040 comp rx=0 tx=0).stop 2024-03-28T22:43:15.279 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.282+0000 7fd40ae00640 1 -- 172.21.15.43:0/1191838191 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd404104100 msgr2=0x7fd40419a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.279 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.282+0000 7fd40ae00640 1 --2- 172.21.15.43:0/1191838191 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd404104100 0x7fd40419a570 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fd3f0005ab0 tx=0x7fd3f0004270 comp rx=0 tx=0).stop 2024-03-28T22:43:15.280 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.283+0000 7fd40ae00640 1 -- 172.21.15.43:0/1191838191 shutdown_connections 2024-03-28T22:43:15.280 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.283+0000 7fd40ae00640 1 --2- 172.21.15.43:0/1191838191 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd3ec0a1f60 0x7fd3ec0a4420 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.280 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.283+0000 7fd40ae00640 1 --2- 172.21.15.43:0/1191838191 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd404104100 0x7fd40419a570 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.280 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.283+0000 7fd40ae00640 1 --2- 172.21.15.43:0/1191838191 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd404109750 0x7fd40419e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.280 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.283+0000 7fd40ae00640 1 --2- 172.21.15.43:0/1191838191 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd404103740 0x7fd40419a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.280 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.283+0000 7fd40ae00640 1 -- 172.21.15.43:0/1191838191 >> 172.21.15.43:0/1191838191 conn(0x7fd4040ff230 msgr2=0x7fd4040ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:15.281 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.283+0000 7fd40ae00640 1 -- 172.21.15.43:0/1191838191 shutdown_connections 2024-03-28T22:43:15.281 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:15.283+0000 7fd40ae00640 1 -- 172.21.15.43:0/1191838191 wait complete. 2024-03-28T22:43:15.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.344+0000 7f9e55c00640 1 -- 172.21.15.43:0/3568029685 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e501076d0 msgr2=0x7f9e50107ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.344+0000 7f9e55c00640 1 --2- 172.21.15.43:0/3568029685 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e501076d0 0x7f9e50107ab0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f9e40009930 tx=0x7f9e4002bc30 comp rx=0 tx=0).stop 2024-03-28T22:43:15.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.345+0000 7f9e55c00640 1 -- 172.21.15.43:0/3568029685 shutdown_connections 2024-03-28T22:43:15.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.345+0000 7f9e55c00640 1 --2- 172.21.15.43:0/3568029685 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e50102080 0x7f9e5010e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.345+0000 7f9e55c00640 1 --2- 172.21.15.43:0/3568029685 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e501016c0 0x7f9e50101b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.345+0000 7f9e55c00640 1 --2- 172.21.15.43:0/3568029685 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e501076d0 0x7f9e50107ab0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.345+0000 7f9e55c00640 1 -- 172.21.15.43:0/3568029685 >> 172.21.15.43:0/3568029685 conn(0x7f9e500fd1b0 msgr2=0x7f9e500ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:15.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.345+0000 7f9e55c00640 1 -- 172.21.15.43:0/3568029685 shutdown_connections 2024-03-28T22:43:15.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.345+0000 7f9e55c00640 1 -- 172.21.15.43:0/3568029685 wait complete. 2024-03-28T22:43:15.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.346+0000 7f9e55c00640 1 Processor -- start 2024-03-28T22:43:15.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.346+0000 7f9e55c00640 1 -- start start 2024-03-28T22:43:15.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.346+0000 7f9e55c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e501016c0 0x7f9e50198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.346+0000 7f9e55c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e50102080 0x7f9e50198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.346+0000 7f9e55c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e501076d0 0x7f9e5019c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.346+0000 7f9e55c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9e50111b30 con 0x7f9e50102080 2024-03-28T22:43:15.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.346+0000 7f9e55c00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9e501119b0 con 0x7f9e501016c0 2024-03-28T22:43:15.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.346+0000 7f9e55c00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9e50111cb0 con 0x7f9e501076d0 2024-03-28T22:43:15.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.346+0000 7f9e4ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e501016c0 0x7f9e50198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.346+0000 7f9e4e000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e50102080 0x7f9e50198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.346+0000 7f9e4f400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e501076d0 0x7f9e5019c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.346+0000 7f9e4ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e501016c0 0x7f9e50198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:50434/0 (socket says 172.21.15.43:50434) 2024-03-28T22:43:15.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.346+0000 7f9e4ea00640 1 -- 172.21.15.43:0/3010316988 learned_addr learned my addr 172.21.15.43:0/3010316988 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:15.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.347+0000 7f9e4ea00640 1 -- 172.21.15.43:0/3010316988 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e501076d0 msgr2=0x7f9e5019c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.347+0000 7f9e4ea00640 1 --2- 172.21.15.43:0/3010316988 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e501076d0 0x7f9e5019c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.347+0000 7f9e4ea00640 1 -- 172.21.15.43:0/3010316988 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e50102080 msgr2=0x7f9e50198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.347+0000 7f9e4ea00640 1 --2- 172.21.15.43:0/3010316988 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e50102080 0x7f9e50198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.347+0000 7f9e4ea00640 1 -- 172.21.15.43:0/3010316988 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e5019d020 con 0x7f9e501016c0 2024-03-28T22:43:15.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.347+0000 7f9e4e000640 1 --2- 172.21.15.43:0/3010316988 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e50102080 0x7f9e50198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:15.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.347+0000 7f9e4ea00640 1 --2- 172.21.15.43:0/3010316988 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e501016c0 0x7f9e50198030 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f9e40009900 tx=0x7f9e40004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:15.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.347+0000 7f9e47400640 1 -- 172.21.15.43:0/3010316988 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e40043020 con 0x7f9e501016c0 2024-03-28T22:43:15.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.348+0000 7f9e55c00640 1 -- 172.21.15.43:0/3010316988 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e5019d2b0 con 0x7f9e501016c0 2024-03-28T22:43:15.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.348+0000 7f9e47400640 1 -- 172.21.15.43:0/3010316988 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e400054e0 con 0x7f9e501016c0 2024-03-28T22:43:15.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.348+0000 7f9e47400640 1 -- 172.21.15.43:0/3010316988 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e40003bb0 con 0x7f9e501016c0 2024-03-28T22:43:15.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.348+0000 7f9e55c00640 1 -- 172.21.15.43:0/3010316988 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e501a4ba0 con 0x7f9e501016c0 2024-03-28T22:43:15.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.352+0000 7f9e58963740 1 Processor -- start 2024-03-28T22:43:15.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.352+0000 7f9e58963740 1 -- start start 2024-03-28T22:43:15.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.352+0000 7f9e58963740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecb054f380 0x55ecb054f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.353+0000 7f9e58963740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ecb054fca0 0x55ecb0558340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.353+0000 7f9e47400640 1 -- 172.21.15.43:0/3010316988 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f9e40046020 con 0x7f9e501016c0 2024-03-28T22:43:15.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.353+0000 7f9e58963740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecb0558880 0x55ecb055ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.353+0000 7f9e58963740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ecb04b3050 con 0x55ecb0558880 2024-03-28T22:43:15.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.353+0000 7f9e58963740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55ecb045bce0 con 0x55ecb054f380 2024-03-28T22:43:15.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.353+0000 7f9e58963740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55ecb04af0f0 con 0x55ecb054fca0 2024-03-28T22:43:15.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.353+0000 7f9e4f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecb054f380 0x55ecb054f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.353+0000 7f9e4f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecb054f380 0x55ecb054f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:50450/0 (socket says 172.21.15.43:50450) 2024-03-28T22:43:15.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.353+0000 7f9e4f400640 1 -- 172.21.15.43:0/1164551703 learned_addr learned my addr 172.21.15.43:0/1164551703 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:15.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.353+0000 7f9e4e000640 1 --2- 172.21.15.43:0/1164551703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecb0558880 0x55ecb055ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.353+0000 7f9e4ea00640 1 --2- 172.21.15.43:0/1164551703 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ecb054fca0 0x55ecb0558340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.353+0000 7f9e4f400640 1 -- 172.21.15.43:0/1164551703 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ecb054fca0 msgr2=0x55ecb0558340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.353+0000 7f9e4f400640 1 --2- 172.21.15.43:0/1164551703 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ecb054fca0 0x55ecb0558340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.353+0000 7f9e4f400640 1 -- 172.21.15.43:0/1164551703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecb0558880 msgr2=0x55ecb055ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.353+0000 7f9e4f400640 1 --2- 172.21.15.43:0/1164551703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecb0558880 0x55ecb055ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.353+0000 7f9e4f400640 1 -- 172.21.15.43:0/1164551703 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecb01c73f0 con 0x55ecb054f380 2024-03-28T22:43:15.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.353+0000 7f9e4ea00640 1 --2- 172.21.15.43:0/1164551703 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ecb054fca0 0x55ecb0558340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:43:15.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.353+0000 7f9e4e000640 1 --2- 172.21.15.43:0/1164551703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecb0558880 0x55ecb055ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:43:15.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.353+0000 7f9e4f400640 1 --2- 172.21.15.43:0/1164551703 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecb054f380 0x55ecb054f760 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f9e48009f50 tx=0x7f9e480118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61d07d0e7643fc9f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:15.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.354+0000 7f9e3be00640 1 -- 172.21.15.43:0/1164551703 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e48004d20 con 0x55ecb054f380 2024-03-28T22:43:15.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.354+0000 7f9e3be00640 1 -- 172.21.15.43:0/1164551703 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e48004050 con 0x55ecb054f380 2024-03-28T22:43:15.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.354+0000 7f9e3be00640 1 -- 172.21.15.43:0/1164551703 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e48005240 con 0x55ecb054f380 2024-03-28T22:43:15.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.354+0000 7f9e58963740 1 -- 172.21.15.43:0/1164551703 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecb054f380 msgr2=0x55ecb054f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.354+0000 7f9e58963740 1 --2- 172.21.15.43:0/1164551703 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecb054f380 0x55ecb054f760 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f9e48009f50 tx=0x7f9e480118d0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.355+0000 7f9e47400640 1 --2- 172.21.15.43:0/3010316988 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9e340a1ef0 0x7f9e340a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.355+0000 7f9e58963740 1 -- 172.21.15.43:0/1164551703 shutdown_connections 2024-03-28T22:43:15.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.355+0000 7f9e58963740 1 --2- 172.21.15.43:0/1164551703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecb0558880 0x55ecb055ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.355+0000 7f9e58963740 1 --2- 172.21.15.43:0/1164551703 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ecb054fca0 0x55ecb0558340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.355+0000 7f9e58963740 1 --2- 172.21.15.43:0/1164551703 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecb054f380 0x55ecb054f760 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.355+0000 7f9e58963740 1 -- 172.21.15.43:0/1164551703 >> 172.21.15.43:0/1164551703 conn(0x55ecb0549ab0 msgr2=0x55ecb0549e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:15.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.355+0000 7f9e47400640 1 -- 172.21.15.43:0/3010316988 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f9e400f04e0 con 0x7f9e501016c0 2024-03-28T22:43:15.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.355+0000 7f9e58963740 1 -- 172.21.15.43:0/1164551703 shutdown_connections 2024-03-28T22:43:15.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.355+0000 7f9e4f400640 1 --2- 172.21.15.43:0/3010316988 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9e340a1ef0 0x7f9e340a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.355+0000 7f9e58963740 1 -- 172.21.15.43:0/1164551703 wait complete. 2024-03-28T22:43:15.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.355+0000 7f9e58963740 1 -- start start 2024-03-28T22:43:15.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.356+0000 7f9e4f400640 1 --2- 172.21.15.43:0/3010316988 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9e340a1ef0 0x7f9e340a43b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9e48004990 tx=0x7f9e48004700 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:15.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.356+0000 7f9e58963740 1 Processor -- start 2024-03-28T22:43:15.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.357+0000 7f9e58963740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecb054f380 0x55ecb0562a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.357+0000 7f9e58963740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ecb054fca0 0x55ecb0562f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.357+0000 7f9e58963740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecb0558880 0x55ecb055ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.357+0000 7f9e58963740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ecb055b670 con 0x55ecb054f380 2024-03-28T22:43:15.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.357+0000 7f9e58963740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55ecb04512f0 con 0x55ecb0558880 2024-03-28T22:43:15.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.357+0000 7f9e58963740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55ecb055b7f0 con 0x55ecb054fca0 2024-03-28T22:43:15.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.357+0000 7f9e58963740 10 client.0 _ll_register_callbacks cb 0x55ecb01c6be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:15.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.357+0000 7f9e4f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecb0558880 0x55ecb055ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.357+0000 7f9e4ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecb054f380 0x55ecb0562a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.357+0000 7f9e4f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecb0558880 0x55ecb055ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:50456/0 (socket says 172.21.15.43:50456) 2024-03-28T22:43:15.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.358+0000 7f9e4f400640 1 -- 172.21.15.43:0/2581353568 learned_addr learned my addr 172.21.15.43:0/2581353568 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:15.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.358+0000 7f9e4e000640 1 --2- 172.21.15.43:0/2581353568 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ecb054fca0 0x55ecb0562f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.358+0000 7f9e4f400640 1 -- 172.21.15.43:0/2581353568 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ecb054fca0 msgr2=0x55ecb0562f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.358+0000 7f9e4f400640 1 --2- 172.21.15.43:0/2581353568 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ecb054fca0 0x55ecb0562f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.358+0000 7f9e4f400640 1 -- 172.21.15.43:0/2581353568 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecb054f380 msgr2=0x55ecb0562a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.358+0000 7f9e4f400640 1 --2- 172.21.15.43:0/2581353568 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecb054f380 0x55ecb0562a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.358+0000 7f9e4f400640 1 -- 172.21.15.43:0/2581353568 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecb02917d0 con 0x55ecb0558880 2024-03-28T22:43:15.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.359+0000 7f9e4f400640 1 --2- 172.21.15.43:0/2581353568 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecb0558880 0x55ecb055ce30 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f9e48009f50 tx=0x7f9e48005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6b45888f974366 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:15.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.359+0000 7f9e38c00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:43:15.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.359+0000 7f9e38c00640 1 -- 172.21.15.43:0/2581353568 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e48032070 con 0x55ecb0558880 2024-03-28T22:43:15.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.359+0000 7f9e58963740 1 -- 172.21.15.43:0/2581353568 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e48004050 con 0x55ecb0558880 2024-03-28T22:43:15.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.359+0000 7f9e38c00640 1 -- 172.21.15.43:0/2581353568 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f9e48024440 con 0x55ecb0558880 2024-03-28T22:43:15.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.359+0000 7f9e38c00640 1 -- 172.21.15.43:0/2581353568 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9e4803e050 con 0x55ecb0558880 2024-03-28T22:43:15.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.359+0000 7f9e58963740 10 client.27881 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:15.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.359+0000 7f9e58963740 1 -- 172.21.15.43:0/2581353568 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ecb056bc60 con 0x55ecb0558880 2024-03-28T22:43:15.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.360+0000 7f9e38c00640 1 -- 172.21.15.43:0/2581353568 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e48004050 con 0x55ecb0558880 2024-03-28T22:43:15.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.360+0000 7f9e38c00640 1 -- 172.21.15.43:0/2581353568 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f9e4802d070 con 0x55ecb0558880 2024-03-28T22:43:15.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.360+0000 7f9e58963740 10 client.27881 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:15.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.360+0000 7f9e58963740 10 client.27881 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:43:15.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.360+0000 7f9e58963740 20 client.27881 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:15.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.360+0000 7f9e58963740 1 --2- 172.21.15.43:0/2581353568 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55ecb056d010 0x55ecb056f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.360+0000 7f9e58963740 4 client.27881 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:43:15.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.360+0000 7f9e58963740 1 -- 172.21.15.43:0/2581353568 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ecb045c680 con 0x55ecb056d010 2024-03-28T22:43:15.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.361+0000 7f9e4ea00640 1 --2- 172.21.15.43:0/2581353568 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55ecb056d010 0x55ecb056f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.361+0000 7f9e4ea00640 1 --2- 172.21.15.43:0/2581353568 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55ecb056d010 0x55ecb056f400 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d86fc522423311f9 server_cookie=10eba924fbad26b4 in_seq=0 out_seq=0 2024-03-28T22:43:15.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.361+0000 7f9e38c00640 10 client.27881 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:43:15.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.362+0000 7f9e38c00640 1 -- 172.21.15.43:0/2581353568 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ecb045c680 con 0x55ecb056d010 2024-03-28T22:43:15.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.362+0000 7f9e38c00640 10 client.27881 handle_command_reply: tid=0 2024-03-28T22:43:15.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.362+0000 7f9e58963740 1 client.27881 shutdown 2024-03-28T22:43:15.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.362+0000 7f9e58963740 1 -- 172.21.15.43:0/2581353568 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecb0558880 msgr2=0x55ecb055ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.362+0000 7f9e58963740 1 --2- 172.21.15.43:0/2581353568 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecb0558880 0x55ecb055ce30 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f9e48009f50 tx=0x7f9e48005fd0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.362+0000 7f9e58963740 1 -- 172.21.15.43:0/2581353568 shutdown_connections 2024-03-28T22:43:15.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.362+0000 7f9e58963740 1 --2- 172.21.15.43:0/2581353568 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55ecb056d010 0x55ecb056f400 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.362+0000 7f9e58963740 1 --2- 172.21.15.43:0/2581353568 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecb0558880 0x55ecb055ce30 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.362+0000 7f9e58963740 1 --2- 172.21.15.43:0/2581353568 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ecb054fca0 0x55ecb0562f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.362+0000 7f9e58963740 1 --2- 172.21.15.43:0/2581353568 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecb054f380 0x55ecb0562a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.362+0000 7f9e38c00640 0 client.27881 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:43:15.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.362+0000 7f9e58963740 1 -- 172.21.15.43:0/2581353568 >> 172.21.15.43:0/2581353568 conn(0x55ecb0549ab0 msgr2=0x55ecb054b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:15.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.362+0000 7f9e58963740 1 -- 172.21.15.43:0/2581353568 shutdown_connections 2024-03-28T22:43:15.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.362+0000 7f9e58963740 1 -- 172.21.15.43:0/2581353568 wait complete. 2024-03-28T22:43:15.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.363+0000 7f9e58963740 20 client.27881 trim_cache size 0 max 16384 2024-03-28T22:43:15.372 INFO:tasks.workunit.client.0.smithi043.stdout: 1 413159 7.39 MB/sec execute 166 sec latency 6531.828 ms 2024-03-28T22:43:15.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.390+0000 7f9e58963740 1 Processor -- start 2024-03-28T22:43:15.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.390+0000 7f9e58963740 1 -- start start 2024-03-28T22:43:15.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.390+0000 7f9e58963740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecb054f380 0x55ecb055cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.390+0000 7f9e58963740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecb054fca0 0x55ecb0574c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.390+0000 7f9e58963740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ecb0558880 0x55ecb0556eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.390+0000 7f9e58963740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9e501ac9c0 con 0x55ecb054f380 2024-03-28T22:43:15.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.390+0000 7f9e58963740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9e501ae500 con 0x55ecb054fca0 2024-03-28T22:43:15.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.390+0000 7f9e58963740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9e501ad090 con 0x55ecb0558880 2024-03-28T22:43:15.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.390+0000 7f9e4f400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ecb0558880 0x55ecb0556eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.390+0000 7f9e4e000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecb054fca0 0x55ecb0574c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.390+0000 7f9e4ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecb054f380 0x55ecb055cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.390+0000 7f9e4f400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ecb0558880 0x55ecb0556eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:49462/0 (socket says 172.21.15.43:49462) 2024-03-28T22:43:15.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.390+0000 7f9e4f400640 1 -- 172.21.15.43:0/2440142881 learned_addr learned my addr 172.21.15.43:0/2440142881 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:15.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.391+0000 7f9e4f400640 1 -- 172.21.15.43:0/2440142881 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecb054fca0 msgr2=0x55ecb0574c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.391+0000 7f9e4f400640 1 --2- 172.21.15.43:0/2440142881 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecb054fca0 0x55ecb0574c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.391+0000 7f9e4f400640 1 -- 172.21.15.43:0/2440142881 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecb054f380 msgr2=0x55ecb055cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.391+0000 7f9e4f400640 1 --2- 172.21.15.43:0/2440142881 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecb054f380 0x55ecb055cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.391+0000 7f9e4f400640 1 -- 172.21.15.43:0/2440142881 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecb054fad0 con 0x55ecb0558880 2024-03-28T22:43:15.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.391+0000 7f9e4e000640 1 --2- 172.21.15.43:0/2440142881 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecb054fca0 0x55ecb0574c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:15.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.391+0000 7f9e4ea00640 1 --2- 172.21.15.43:0/2440142881 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecb054f380 0x55ecb055cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:15.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.391+0000 7f9e4f400640 1 --2- 172.21.15.43:0/2440142881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ecb0558880 0x55ecb0556eb0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f9e48009270 tx=0x7f9e48005d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9842c7442f7011 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:15.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.391+0000 7f9e3b400640 1 -- 172.21.15.43:0/2440142881 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e4803b070 con 0x55ecb0558880 2024-03-28T22:43:15.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.391+0000 7f9e3b400640 1 -- 172.21.15.43:0/2440142881 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e48028070 con 0x55ecb0558880 2024-03-28T22:43:15.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.391+0000 7f9e3b400640 1 -- 172.21.15.43:0/2440142881 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e48032040 con 0x55ecb0558880 2024-03-28T22:43:15.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.392+0000 7f9e58963740 1 -- 172.21.15.43:0/2440142881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ecb0558880 msgr2=0x55ecb0556eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.392+0000 7f9e58963740 1 --2- 172.21.15.43:0/2440142881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ecb0558880 0x55ecb0556eb0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f9e48009270 tx=0x7f9e48005d30 comp rx=0 tx=0).stop 2024-03-28T22:43:15.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.392+0000 7f9e58963740 1 -- 172.21.15.43:0/2440142881 shutdown_connections 2024-03-28T22:43:15.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.392+0000 7f9e58963740 1 --2- 172.21.15.43:0/2440142881 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecb054fca0 0x55ecb0574c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.392+0000 7f9e58963740 1 --2- 172.21.15.43:0/2440142881 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ecb0558880 0x55ecb0556eb0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.392+0000 7f9e58963740 1 --2- 172.21.15.43:0/2440142881 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecb054f380 0x55ecb055cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.392+0000 7f9e58963740 1 -- 172.21.15.43:0/2440142881 >> 172.21.15.43:0/2440142881 conn(0x55ecb0549ab0 msgr2=0x55ecb055c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:15.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.392+0000 7f9e58963740 1 -- 172.21.15.43:0/2440142881 shutdown_connections 2024-03-28T22:43:15.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.392+0000 7f9e58963740 1 -- 172.21.15.43:0/2440142881 wait complete. 2024-03-28T22:43:15.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.393+0000 7f9e58963740 1 -- start start 2024-03-28T22:43:15.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.393+0000 7f9e58963740 1 Processor -- start 2024-03-28T22:43:15.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.393+0000 7f9e58963740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecb054f380 0x55ecb0568d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.393+0000 7f9e58963740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecb054fca0 0x55ecb0569260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.393+0000 7f9e58963740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ecb0558880 0x55ecb05661d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.393+0000 7f9e58963740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9e501ade30 con 0x55ecb054fca0 2024-03-28T22:43:15.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.393+0000 7f9e58963740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9e501ad760 con 0x55ecb054f380 2024-03-28T22:43:15.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.393+0000 7f9e58963740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9e501aa9a0 con 0x55ecb0558880 2024-03-28T22:43:15.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.393+0000 7f9e58963740 10 client.0 _ll_register_callbacks cb 0x7f9e48009e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:15.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.394+0000 7f9e4ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecb054f380 0x55ecb0568d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.394+0000 7f9e4f400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ecb0558880 0x55ecb05661d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.394+0000 7f9e4ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecb054f380 0x55ecb0568d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:50480/0 (socket says 172.21.15.43:50480) 2024-03-28T22:43:15.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.394+0000 7f9e4ea00640 1 -- 172.21.15.43:0/1309606459 learned_addr learned my addr 172.21.15.43:0/1309606459 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:15.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.394+0000 7f9e4e000640 1 --2- 172.21.15.43:0/1309606459 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecb054fca0 0x55ecb0569260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.394+0000 7f9e4ea00640 1 -- 172.21.15.43:0/1309606459 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ecb0558880 msgr2=0x55ecb05661d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.394+0000 7f9e4ea00640 1 --2- 172.21.15.43:0/1309606459 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ecb0558880 0x55ecb05661d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.394+0000 7f9e4ea00640 1 -- 172.21.15.43:0/1309606459 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecb054fca0 msgr2=0x55ecb0569260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.394+0000 7f9e4ea00640 1 --2- 172.21.15.43:0/1309606459 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecb054fca0 0x55ecb0569260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.394+0000 7f9e4ea00640 1 -- 172.21.15.43:0/1309606459 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecb05669d0 con 0x55ecb054f380 2024-03-28T22:43:15.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.394+0000 7f9e4e000640 1 --2- 172.21.15.43:0/1309606459 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecb054fca0 0x55ecb0569260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:43:15.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.394+0000 7f9e4ea00640 1 --2- 172.21.15.43:0/1309606459 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecb054f380 0x55ecb0568d20 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f9e400b95b0 tx=0x7f9e40004ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e730086a6834ad90 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:15.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.394+0000 7f9e38c00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:43:15.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.395+0000 7f9e38c00640 1 -- 172.21.15.43:0/1309606459 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e400bd2e0 con 0x55ecb054f380 2024-03-28T22:43:15.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.395+0000 7f9e38c00640 1 -- 172.21.15.43:0/1309606459 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f9e4002cb90 con 0x55ecb054f380 2024-03-28T22:43:15.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.395+0000 7f9e58963740 1 -- 172.21.15.43:0/1309606459 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e48028070 con 0x55ecb054f380 2024-03-28T22:43:15.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.395+0000 7f9e38c00640 1 -- 172.21.15.43:0/1309606459 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9e400bcbe0 con 0x55ecb054f380 2024-03-28T22:43:15.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.395+0000 7f9e58963740 10 client.27887 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:15.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.395+0000 7f9e58963740 1 -- 172.21.15.43:0/1309606459 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ecb05d2dc0 con 0x55ecb054f380 2024-03-28T22:43:15.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.395+0000 7f9e38c00640 1 -- 172.21.15.43:0/1309606459 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e48028070 con 0x55ecb054f380 2024-03-28T22:43:15.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.396+0000 7f9e38c00640 1 -- 172.21.15.43:0/1309606459 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f9e400f0120 con 0x55ecb054f380 2024-03-28T22:43:15.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.396+0000 7f9e58963740 10 client.27887 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:15.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.396+0000 7f9e58963740 10 client.27887 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:43:15.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.396+0000 7f9e58963740 20 client.27887 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:15.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.396+0000 7f9e58963740 1 --2- 172.21.15.43:0/1309606459 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55ecb05d3490 0x55ecb05d5880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.396+0000 7f9e58963740 4 client.27887 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:43:15.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.396+0000 7f9e58963740 1 -- 172.21.15.43:0/1309606459 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ecb04c0590 con 0x55ecb05d3490 2024-03-28T22:43:15.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.396+0000 7f9e4e000640 1 --2- 172.21.15.43:0/1309606459 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55ecb05d3490 0x55ecb05d5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.397+0000 7f9e4e000640 1 --2- 172.21.15.43:0/1309606459 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55ecb05d3490 0x55ecb05d5880 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5f636bfaf3000fff server_cookie=73ff196791bf0da5 in_seq=0 out_seq=0 2024-03-28T22:43:15.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.397+0000 7f9e38c00640 10 client.27887 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:43:15.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.397+0000 7f9e38c00640 1 -- 172.21.15.43:0/1309606459 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ecb04c0590 con 0x55ecb05d3490 2024-03-28T22:43:15.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.397+0000 7f9e38c00640 10 client.27887 handle_command_reply: tid=0 2024-03-28T22:43:15.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.397+0000 7f9e58963740 1 client.27887 shutdown 2024-03-28T22:43:15.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.397+0000 7f9e58963740 1 -- 172.21.15.43:0/1309606459 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecb054f380 msgr2=0x55ecb0568d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.397+0000 7f9e58963740 1 --2- 172.21.15.43:0/1309606459 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecb054f380 0x55ecb0568d20 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f9e400b95b0 tx=0x7f9e40004ee0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.397+0000 7f9e58963740 1 -- 172.21.15.43:0/1309606459 shutdown_connections 2024-03-28T22:43:15.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.397+0000 7f9e58963740 1 --2- 172.21.15.43:0/1309606459 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55ecb05d3490 0x55ecb05d5880 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.397+0000 7f9e58963740 1 --2- 172.21.15.43:0/1309606459 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ecb0558880 0x55ecb05661d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.397+0000 7f9e58963740 1 --2- 172.21.15.43:0/1309606459 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ecb054fca0 0x55ecb0569260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.397+0000 7f9e58963740 1 --2- 172.21.15.43:0/1309606459 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecb054f380 0x55ecb0568d20 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.397+0000 7f9e58963740 1 -- 172.21.15.43:0/1309606459 >> 172.21.15.43:0/1309606459 conn(0x55ecb0549ab0 msgr2=0x55ecb0562570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:15.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.397+0000 7f9e38c00640 0 client.27887 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:43:15.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.397+0000 7f9e58963740 1 -- 172.21.15.43:0/1309606459 shutdown_connections 2024-03-28T22:43:15.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.397+0000 7f9e58963740 1 -- 172.21.15.43:0/1309606459 wait complete. 2024-03-28T22:43:15.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.397+0000 7f9e58963740 20 client.27887 trim_cache size 0 max 16384 2024-03-28T22:43:15.399 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:43:15.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.398+0000 7f9e55c00640 1 -- 172.21.15.43:0/3010316988 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9e340a1ef0 msgr2=0x7f9e340a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.398+0000 7f9e55c00640 1 --2- 172.21.15.43:0/3010316988 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9e340a1ef0 0x7f9e340a43b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9e48004990 tx=0x7f9e48004700 comp rx=0 tx=0).stop 2024-03-28T22:43:15.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.398+0000 7f9e55c00640 1 -- 172.21.15.43:0/3010316988 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e501016c0 msgr2=0x7f9e50198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.398+0000 7f9e55c00640 1 --2- 172.21.15.43:0/3010316988 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e501016c0 0x7f9e50198030 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f9e40009900 tx=0x7f9e40004360 comp rx=0 tx=0).stop 2024-03-28T22:43:15.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.398+0000 7f9e55c00640 1 -- 172.21.15.43:0/3010316988 shutdown_connections 2024-03-28T22:43:15.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.398+0000 7f9e55c00640 1 --2- 172.21.15.43:0/3010316988 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9e340a1ef0 0x7f9e340a43b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.398+0000 7f9e55c00640 1 --2- 172.21.15.43:0/3010316988 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e501076d0 0x7f9e5019c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.398+0000 7f9e55c00640 1 --2- 172.21.15.43:0/3010316988 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e50102080 0x7f9e50198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.398+0000 7f9e55c00640 1 --2- 172.21.15.43:0/3010316988 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e501016c0 0x7f9e50198030 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.398+0000 7f9e55c00640 1 -- 172.21.15.43:0/3010316988 >> 172.21.15.43:0/3010316988 conn(0x7f9e500fd1b0 msgr2=0x7f9e500fd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:15.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.398+0000 7f9e55c00640 1 -- 172.21.15.43:0/3010316988 shutdown_connections 2024-03-28T22:43:15.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.398+0000 7f9e55c00640 1 -- 172.21.15.43:0/3010316988 wait complete. 2024-03-28T22:43:15.401 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:14 smithi172 ceph-mon[36566]: pgmap v186: 129 pgs: 129 active+clean; 57 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 283 op/s 2024-03-28T22:43:15.401 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:14 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/214692609' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:43:15.413 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:43:15.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.683+0000 7f5ec6200640 1 -- 172.21.15.43:0/1915167505 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ec01042b0 msgr2=0x7f5ec01107c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.683+0000 7f5ec6200640 1 --2- 172.21.15.43:0/1915167505 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ec01042b0 0x7f5ec01107c0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8009c10 tx=0x7f5eb802bef0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.683+0000 7f5ec6200640 1 -- 172.21.15.43:0/1915167505 shutdown_connections 2024-03-28T22:43:15.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.683+0000 7f5ec6200640 1 --2- 172.21.15.43:0/1915167505 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ec01038f0 0x7f5ec0103d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.683+0000 7f5ec6200640 1 --2- 172.21.15.43:0/1915167505 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ec01042b0 0x7f5ec01107c0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.683+0000 7f5ec6200640 1 --2- 172.21.15.43:0/1915167505 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ec0109900 0x7f5ec0109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.683+0000 7f5ec6200640 1 -- 172.21.15.43:0/1915167505 >> 172.21.15.43:0/1915167505 conn(0x7f5ec00ff3a0 msgr2=0x7f5ec01017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:15.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.683+0000 7f5ec6200640 1 -- 172.21.15.43:0/1915167505 shutdown_connections 2024-03-28T22:43:15.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.683+0000 7f5ec6200640 1 -- 172.21.15.43:0/1915167505 wait complete. 2024-03-28T22:43:15.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.684+0000 7f5ec6200640 1 Processor -- start 2024-03-28T22:43:15.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.684+0000 7f5ec6200640 1 -- start start 2024-03-28T22:43:15.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.685+0000 7f5ec6200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ec01038f0 0x7f5ec019a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.685+0000 7f5ec6200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ec01042b0 0x7f5ec019a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.685+0000 7f5ec6200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ec0109900 0x7f5ec019eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.685+0000 7f5ec6200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5ec0113d60 con 0x7f5ec0109900 2024-03-28T22:43:15.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.685+0000 7f5ec6200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5ec0113be0 con 0x7f5ec01038f0 2024-03-28T22:43:15.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.685+0000 7f5ec6200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f5ec0113ee0 con 0x7f5ec01042b0 2024-03-28T22:43:15.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.685+0000 7f5ebf400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ec01038f0 0x7f5ec019a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.685+0000 7f5ebea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ec01042b0 0x7f5ec019a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.685+0000 7f5ebf400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ec01038f0 0x7f5ec019a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:50510/0 (socket says 172.21.15.43:50510) 2024-03-28T22:43:15.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.685+0000 7f5ebf400640 1 -- 172.21.15.43:0/2398588241 learned_addr learned my addr 172.21.15.43:0/2398588241 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:15.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.685+0000 7f5ebfe00640 1 --2- 172.21.15.43:0/2398588241 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ec0109900 0x7f5ec019eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.686+0000 7f5ebf400640 1 -- 172.21.15.43:0/2398588241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ec01042b0 msgr2=0x7f5ec019a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.686+0000 7f5ebf400640 1 --2- 172.21.15.43:0/2398588241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ec01042b0 0x7f5ec019a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.686+0000 7f5ebf400640 1 -- 172.21.15.43:0/2398588241 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ec0109900 msgr2=0x7f5ec019eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.686+0000 7f5ebf400640 1 --2- 172.21.15.43:0/2398588241 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ec0109900 0x7f5ec019eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.686+0000 7f5ebf400640 1 -- 172.21.15.43:0/2398588241 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ec019f240 con 0x7f5ec01038f0 2024-03-28T22:43:15.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.686+0000 7f5ebea00640 1 --2- 172.21.15.43:0/2398588241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ec01042b0 0x7f5ec019a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:15.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.686+0000 7f5ebf400640 1 --2- 172.21.15.43:0/2398588241 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ec01038f0 0x7f5ec019a2a0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f5eb000e9f0 tx=0x7f5eb000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:15.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.686+0000 7f5eb7e00640 1 -- 172.21.15.43:0/2398588241 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eb000cd80 con 0x7f5ec01038f0 2024-03-28T22:43:15.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.686+0000 7f5ec6200640 1 -- 172.21.15.43:0/2398588241 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ec019f530 con 0x7f5ec01038f0 2024-03-28T22:43:15.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.686+0000 7f5eb7e00640 1 -- 172.21.15.43:0/2398588241 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5eb0010910 con 0x7f5ec01038f0 2024-03-28T22:43:15.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.686+0000 7f5eb7e00640 1 -- 172.21.15.43:0/2398588241 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eb00188f0 con 0x7f5ec01038f0 2024-03-28T22:43:15.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.686+0000 7f5ec6200640 1 -- 172.21.15.43:0/2398588241 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ec01a6dc0 con 0x7f5ec01038f0 2024-03-28T22:43:15.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.690+0000 7f5ec8e59740 1 Processor -- start 2024-03-28T22:43:15.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.690+0000 7f5ec8e59740 1 -- start start 2024-03-28T22:43:15.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.690+0000 7f5eb7e00640 1 -- 172.21.15.43:0/2398588241 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f5eb0010ac0 con 0x7f5ec01038f0 2024-03-28T22:43:15.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.690+0000 7f5ec8e59740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2371380 0x562bd2371760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.690+0000 7f5ec8e59740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2371ca0 0x562bd237a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.690+0000 7f5ec8e59740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd237a880 0x562bd237cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.690+0000 7f5ec8e59740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562bd22d5050 con 0x562bd2371380 2024-03-28T22:43:15.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.690+0000 7f5ec8e59740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x562bd227dce0 con 0x562bd2371ca0 2024-03-28T22:43:15.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.690+0000 7f5ec8e59740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x562bd22d10f0 con 0x562bd237a880 2024-03-28T22:43:15.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.690+0000 7f5ebf400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2371ca0 0x562bd237a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.690+0000 7f5ebfe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2371380 0x562bd2371760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.690+0000 7f5ebea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd237a880 0x562bd237cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.691+0000 7f5ebf400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2371ca0 0x562bd237a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:50518/0 (socket says 172.21.15.43:50518) 2024-03-28T22:43:15.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.691+0000 7f5ebf400640 1 -- 172.21.15.43:0/1798610785 learned_addr learned my addr 172.21.15.43:0/1798610785 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:15.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.691+0000 7f5ebfe00640 1 -- 172.21.15.43:0/1798610785 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd237a880 msgr2=0x562bd237cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.691+0000 7f5ebfe00640 1 --2- 172.21.15.43:0/1798610785 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd237a880 0x562bd237cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.691+0000 7f5ebfe00640 1 -- 172.21.15.43:0/1798610785 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2371ca0 msgr2=0x562bd237a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.691+0000 7f5ebfe00640 1 --2- 172.21.15.43:0/1798610785 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2371ca0 0x562bd237a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.691+0000 7f5ebfe00640 1 -- 172.21.15.43:0/1798610785 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bd1fe93f0 con 0x562bd2371380 2024-03-28T22:43:15.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.691+0000 7f5ebf400640 1 --2- 172.21.15.43:0/1798610785 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2371ca0 0x562bd237a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:15.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.692+0000 7f5ebfe00640 1 --2- 172.21.15.43:0/1798610785 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2371380 0x562bd2371760 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8036040 tx=0x7f5eb8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=494f519bc7b17b3e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:15.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.692+0000 7f5eb7e00640 1 --2- 172.21.15.43:0/2398588241 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f5ea40a1ef0 0x7f5ea40a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.692+0000 7f5eabe00640 1 -- 172.21.15.43:0/1798610785 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eb8043030 con 0x562bd2371380 2024-03-28T22:43:15.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.692+0000 7f5eb7e00640 1 -- 172.21.15.43:0/2398588241 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f5eb00ce8e0 con 0x7f5ec01038f0 2024-03-28T22:43:15.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.693+0000 7f5eabe00640 1 -- 172.21.15.43:0/1798610785 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5eb8033c00 con 0x562bd2371380 2024-03-28T22:43:15.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.693+0000 7f5ebf400640 1 --2- 172.21.15.43:0/2398588241 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f5ea40a1ef0 0x7f5ea40a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.693+0000 7f5eabe00640 1 -- 172.21.15.43:0/1798610785 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eb802c890 con 0x562bd2371380 2024-03-28T22:43:15.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.693+0000 7f5ec8e59740 1 -- 172.21.15.43:0/1798610785 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2371380 msgr2=0x562bd2371760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.693+0000 7f5ec8e59740 1 --2- 172.21.15.43:0/1798610785 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2371380 0x562bd2371760 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8036040 tx=0x7f5eb8009840 comp rx=0 tx=0).stop 2024-03-28T22:43:15.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.693+0000 7f5ebf400640 1 --2- 172.21.15.43:0/2398588241 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f5ea40a1ef0 0x7f5ea40a43b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5eb000e9c0 tx=0x7f5eb00d4530 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:15.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.694+0000 7f5ec8e59740 1 -- 172.21.15.43:0/1798610785 shutdown_connections 2024-03-28T22:43:15.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.694+0000 7f5ec8e59740 1 --2- 172.21.15.43:0/1798610785 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2371ca0 0x562bd237a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.694+0000 7f5ec8e59740 1 --2- 172.21.15.43:0/1798610785 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd237a880 0x562bd237cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.694+0000 7f5ec8e59740 1 --2- 172.21.15.43:0/1798610785 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2371380 0x562bd2371760 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.694+0000 7f5ec8e59740 1 -- 172.21.15.43:0/1798610785 >> 172.21.15.43:0/1798610785 conn(0x562bd236bab0 msgr2=0x562bd236be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:15.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.694+0000 7f5ec8e59740 1 -- 172.21.15.43:0/1798610785 shutdown_connections 2024-03-28T22:43:15.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.694+0000 7f5ec8e59740 1 -- 172.21.15.43:0/1798610785 wait complete. 2024-03-28T22:43:15.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.694+0000 7f5ec8e59740 1 -- start start 2024-03-28T22:43:15.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.694+0000 7f5ec8e59740 1 Processor -- start 2024-03-28T22:43:15.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.695+0000 7f5ec8e59740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2371380 0x562bd2377850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.695+0000 7f5ec8e59740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2371ca0 0x562bd2377d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.695+0000 7f5ec8e59740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd237a880 0x562bd23782d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.695+0000 7f5ec8e59740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562bd237d610 con 0x562bd237a880 2024-03-28T22:43:15.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.695+0000 7f5ec8e59740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x562bd22732f0 con 0x562bd2371380 2024-03-28T22:43:15.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.695+0000 7f5ec8e59740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x562bd237d790 con 0x562bd2371ca0 2024-03-28T22:43:15.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.695+0000 7f5ec8e59740 10 client.0 _ll_register_callbacks cb 0x562bd1fe8be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:15.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.695+0000 7f5ebea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2371ca0 0x562bd2377d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.695+0000 7f5ebea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2371ca0 0x562bd2377d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:49500/0 (socket says 172.21.15.43:49500) 2024-03-28T22:43:15.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.695+0000 7f5ebea00640 1 -- 172.21.15.43:0/3533559216 learned_addr learned my addr 172.21.15.43:0/3533559216 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:15.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.695+0000 7f5ebfe00640 1 --2- 172.21.15.43:0/3533559216 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2371380 0x562bd2377850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.695+0000 7f5ebfe00640 1 --2- 172.21.15.43:0/3533559216 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd237a880 0x562bd23782d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.696+0000 7f5ebea00640 1 -- 172.21.15.43:0/3533559216 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2371380 msgr2=0x562bd2377850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.696+0000 7f5ebea00640 1 --2- 172.21.15.43:0/3533559216 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2371380 0x562bd2377850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.696+0000 7f5ebea00640 1 -- 172.21.15.43:0/3533559216 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd237a880 msgr2=0x562bd23782d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.696+0000 7f5ebea00640 1 --2- 172.21.15.43:0/3533559216 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd237a880 0x562bd23782d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.696+0000 7f5ebea00640 1 -- 172.21.15.43:0/3533559216 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bd20b37d0 con 0x562bd2371ca0 2024-03-28T22:43:15.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.696+0000 7f5ebfe00640 1 --2- 172.21.15.43:0/3533559216 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd237a880 0x562bd23782d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:43:15.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.696+0000 7f5ebea00640 1 --2- 172.21.15.43:0/3533559216 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2371ca0 0x562bd2377d90 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f5eac011d10 tx=0x7f5eac00f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e38679ef95488940 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:15.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.696+0000 7f5ea8c00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:43:15.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.696+0000 7f5ea8c00640 1 -- 172.21.15.43:0/3533559216 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eac01e020 con 0x562bd2371ca0 2024-03-28T22:43:15.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.696+0000 7f5ec8e59740 1 -- 172.21.15.43:0/3533559216 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5eb8033c00 con 0x562bd2371ca0 2024-03-28T22:43:15.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.696+0000 7f5ea8c00640 1 -- 172.21.15.43:0/3533559216 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f5eac017050 con 0x562bd2371ca0 2024-03-28T22:43:15.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.697+0000 7f5ea8c00640 1 -- 172.21.15.43:0/3533559216 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5eac026050 con 0x562bd2371ca0 2024-03-28T22:43:15.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.697+0000 7f5ec8e59740 10 client.27838 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:15.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.697+0000 7f5ec8e59740 1 -- 172.21.15.43:0/3533559216 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562bd2389930 con 0x562bd2371ca0 2024-03-28T22:43:15.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.697+0000 7f5ea8c00640 1 -- 172.21.15.43:0/3533559216 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5eb8033c00 con 0x562bd2371ca0 2024-03-28T22:43:15.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.697+0000 7f5ea8c00640 1 -- 172.21.15.43:0/3533559216 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f5eac019070 con 0x562bd2371ca0 2024-03-28T22:43:15.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.697+0000 7f5ec8e59740 10 client.27838 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:15.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.697+0000 7f5ec8e59740 10 client.27838 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:43:15.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.697+0000 7f5ec8e59740 20 client.27838 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:15.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.697+0000 7f5ec8e59740 1 --2- 172.21.15.43:0/3533559216 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x562bd238aeb0 0x562bd238d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.697+0000 7f5ec8e59740 4 client.27838 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:43:15.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.697+0000 7f5ec8e59740 1 -- 172.21.15.43:0/3533559216 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562bd227e680 con 0x562bd238aeb0 2024-03-28T22:43:15.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.698+0000 7f5ebfe00640 1 --2- 172.21.15.43:0/3533559216 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x562bd238aeb0 0x562bd238d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.698+0000 7f5ebfe00640 1 --2- 172.21.15.43:0/3533559216 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x562bd238aeb0 0x562bd238d2d0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=53628ebb8989d92 server_cookie=688cb10e3746c9f in_seq=0 out_seq=0 2024-03-28T22:43:15.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.698+0000 7f5ea8c00640 10 client.27838 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:43:15.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.699+0000 7f5ea8c00640 1 -- 172.21.15.43:0/3533559216 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562bd227e680 con 0x562bd238aeb0 2024-03-28T22:43:15.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.699+0000 7f5ea8c00640 10 client.27838 handle_command_reply: tid=0 2024-03-28T22:43:15.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.699+0000 7f5ec8e59740 1 client.27838 shutdown 2024-03-28T22:43:15.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.699+0000 7f5ec8e59740 1 -- 172.21.15.43:0/3533559216 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2371ca0 msgr2=0x562bd2377d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.699+0000 7f5ec8e59740 1 --2- 172.21.15.43:0/3533559216 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2371ca0 0x562bd2377d90 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f5eac011d10 tx=0x7f5eac00f790 comp rx=0 tx=0).stop 2024-03-28T22:43:15.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.699+0000 7f5ec8e59740 1 -- 172.21.15.43:0/3533559216 shutdown_connections 2024-03-28T22:43:15.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.699+0000 7f5ec8e59740 1 --2- 172.21.15.43:0/3533559216 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd237a880 0x562bd23782d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.699+0000 7f5ec8e59740 1 --2- 172.21.15.43:0/3533559216 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2371ca0 0x562bd2377d90 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.699+0000 7f5ec8e59740 1 --2- 172.21.15.43:0/3533559216 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x562bd238aeb0 0x562bd238d2d0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.699+0000 7f5ec8e59740 1 --2- 172.21.15.43:0/3533559216 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2371380 0x562bd2377850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.699+0000 7f5ec8e59740 1 -- 172.21.15.43:0/3533559216 >> 172.21.15.43:0/3533559216 conn(0x562bd236bab0 msgr2=0x562bd236d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:15.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.699+0000 7f5ea8c00640 0 client.27838 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:43:15.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.699+0000 7f5ec8e59740 1 -- 172.21.15.43:0/3533559216 shutdown_connections 2024-03-28T22:43:15.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.699+0000 7f5ec8e59740 1 -- 172.21.15.43:0/3533559216 wait complete. 2024-03-28T22:43:15.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.700+0000 7f5ec8e59740 20 client.27838 trim_cache size 0 max 16384 2024-03-28T22:43:15.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.716+0000 7f5ec8e59740 1 Processor -- start 2024-03-28T22:43:15.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.716+0000 7f5ec8e59740 1 -- start start 2024-03-28T22:43:15.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.717+0000 7f5ec8e59740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2371380 0x562bd23863e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.717+0000 7f5ec8e59740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd237a880 0x562bd23780f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.717+0000 7f5ec8e59740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2392b60 0x562bd2374ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.717+0000 7f5ec8e59740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5ec01b0d20 con 0x562bd2371380 2024-03-28T22:43:15.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.717+0000 7f5ec8e59740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5ec01b2860 con 0x562bd2392b60 2024-03-28T22:43:15.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.717+0000 7f5ec8e59740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f5ec01b13f0 con 0x562bd237a880 2024-03-28T22:43:15.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.717+0000 7f5ebfe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2371380 0x562bd23863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.717+0000 7f5ebfe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2371380 0x562bd23863e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46120/0 (socket says 172.21.15.43:46120) 2024-03-28T22:43:15.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.717+0000 7f5ebfe00640 1 -- 172.21.15.43:0/3889573624 learned_addr learned my addr 172.21.15.43:0/3889573624 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:15.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.717+0000 7f5ebfe00640 1 --2- 172.21.15.43:0/3889573624 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2392b60 0x562bd2374ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.717+0000 7f5ebea00640 1 --2- 172.21.15.43:0/3889573624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd237a880 0x562bd23780f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.717+0000 7f5ebfe00640 1 -- 172.21.15.43:0/3889573624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd237a880 msgr2=0x562bd23780f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.717+0000 7f5ebfe00640 1 --2- 172.21.15.43:0/3889573624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd237a880 0x562bd23780f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.717+0000 7f5ebfe00640 1 -- 172.21.15.43:0/3889573624 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2392b60 msgr2=0x562bd2374ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.717+0000 7f5ebfe00640 1 --2- 172.21.15.43:0/3889573624 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2392b60 0x562bd2374ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.717+0000 7f5ebfe00640 1 -- 172.21.15.43:0/3889573624 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bd238d930 con 0x562bd2371380 2024-03-28T22:43:15.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.718+0000 7f5ebfe00640 1 --2- 172.21.15.43:0/3889573624 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2371380 0x562bd23863e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8009600 tx=0x7f5eb802bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2bc134a683f7790 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:15.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.718+0000 7f5eab400640 1 -- 172.21.15.43:0/3889573624 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eb8043040 con 0x562bd2371380 2024-03-28T22:43:15.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.718+0000 7f5eab400640 1 -- 172.21.15.43:0/3889573624 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5eb8054b40 con 0x562bd2371380 2024-03-28T22:43:15.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.718+0000 7f5eab400640 1 -- 172.21.15.43:0/3889573624 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eb8054150 con 0x562bd2371380 2024-03-28T22:43:15.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.719+0000 7f5ec8e59740 1 -- 172.21.15.43:0/3889573624 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2371380 msgr2=0x562bd23863e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.719+0000 7f5ec8e59740 1 --2- 172.21.15.43:0/3889573624 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2371380 0x562bd23863e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8009600 tx=0x7f5eb802bef0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.719+0000 7f5ec8e59740 1 -- 172.21.15.43:0/3889573624 shutdown_connections 2024-03-28T22:43:15.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.719+0000 7f5ec8e59740 1 --2- 172.21.15.43:0/3889573624 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2392b60 0x562bd2374ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.719+0000 7f5ec8e59740 1 --2- 172.21.15.43:0/3889573624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd237a880 0x562bd23780f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.719+0000 7f5ec8e59740 1 --2- 172.21.15.43:0/3889573624 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd2371380 0x562bd23863e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.719+0000 7f5ec8e59740 1 -- 172.21.15.43:0/3889573624 >> 172.21.15.43:0/3889573624 conn(0x562bd236bab0 msgr2=0x562bd236ab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:15.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.719+0000 7f5ec8e59740 1 -- 172.21.15.43:0/3889573624 shutdown_connections 2024-03-28T22:43:15.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.719+0000 7f5ec8e59740 1 -- 172.21.15.43:0/3889573624 wait complete. 2024-03-28T22:43:15.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.719+0000 7f5ec8e59740 1 -- start start 2024-03-28T22:43:15.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.720+0000 7f5ec8e59740 1 Processor -- start 2024-03-28T22:43:15.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.720+0000 7f5ec8e59740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2371380 0x562bd237ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.720+0000 7f5ec8e59740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd237a880 0x562bd237ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.720+0000 7f5ec8e59740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2392b60 0x562bd23f0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.720+0000 7f5ec8e59740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5ec01b2190 con 0x562bd237a880 2024-03-28T22:43:15.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.720+0000 7f5ec8e59740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5ec01b1ac0 con 0x562bd2371380 2024-03-28T22:43:15.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.720+0000 7f5ec8e59740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f5ec01aed00 con 0x562bd2392b60 2024-03-28T22:43:15.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.720+0000 7f5ec8e59740 10 client.0 _ll_register_callbacks cb 0x7f5eac011920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:15.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.720+0000 7f5ebfe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2371380 0x562bd237ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.720+0000 7f5ebfe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2371380 0x562bd237ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:50548/0 (socket says 172.21.15.43:50548) 2024-03-28T22:43:15.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.720+0000 7f5ebfe00640 1 -- 172.21.15.43:0/2736190241 learned_addr learned my addr 172.21.15.43:0/2736190241 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:15.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.721+0000 7f5ebfe00640 1 --2- 172.21.15.43:0/2736190241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2392b60 0x562bd23f0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.721+0000 7f5ebea00640 1 --2- 172.21.15.43:0/2736190241 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd237a880 0x562bd237ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.721+0000 7f5ebfe00640 1 -- 172.21.15.43:0/2736190241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2392b60 msgr2=0x562bd23f0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.721+0000 7f5ebfe00640 1 --2- 172.21.15.43:0/2736190241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2392b60 0x562bd23f0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.721+0000 7f5ebfe00640 1 -- 172.21.15.43:0/2736190241 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd237a880 msgr2=0x562bd237ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.721+0000 7f5ebfe00640 1 --2- 172.21.15.43:0/2736190241 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd237a880 0x562bd237ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.721+0000 7f5ebfe00640 1 -- 172.21.15.43:0/2736190241 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bd23f06d0 con 0x562bd2371380 2024-03-28T22:43:15.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.721+0000 7f5ebfe00640 1 --2- 172.21.15.43:0/2736190241 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2371380 0x562bd237ea50 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8009d30 tx=0x7f5eb8000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ab1e70d00aa36db server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:15.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.721+0000 7f5ea8c00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:43:15.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.721+0000 7f5ea8c00640 1 -- 172.21.15.43:0/2736190241 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eb8043070 con 0x562bd2371380 2024-03-28T22:43:15.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.721+0000 7f5ea8c00640 1 -- 172.21.15.43:0/2736190241 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f5eb8004d40 con 0x562bd2371380 2024-03-28T22:43:15.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.721+0000 7f5ec8e59740 1 -- 172.21.15.43:0/2736190241 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5eb8054b40 con 0x562bd2371380 2024-03-28T22:43:15.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.722+0000 7f5ea8c00640 1 -- 172.21.15.43:0/2736190241 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5eb802cb00 con 0x562bd2371380 2024-03-28T22:43:15.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.722+0000 7f5ec8e59740 10 client.27905 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:15.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.722+0000 7f5ec8e59740 1 -- 172.21.15.43:0/2736190241 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562bd23f1980 con 0x562bd2371380 2024-03-28T22:43:15.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.722+0000 7f5ea8c00640 1 -- 172.21.15.43:0/2736190241 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5eb8054b40 con 0x562bd2371380 2024-03-28T22:43:15.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.722+0000 7f5ea8c00640 1 -- 172.21.15.43:0/2736190241 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f5eb80544b0 con 0x562bd2371380 2024-03-28T22:43:15.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.722+0000 7f5ec8e59740 10 client.27905 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:15.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.722+0000 7f5ec8e59740 10 client.27905 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:43:15.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.722+0000 7f5ec8e59740 20 client.27905 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:15.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.722+0000 7f5ec8e59740 1 --2- 172.21.15.43:0/2736190241 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x562bd23f1c10 0x562bd23f4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.722+0000 7f5ec8e59740 4 client.27905 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:43:15.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.722+0000 7f5ec8e59740 1 -- 172.21.15.43:0/2736190241 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562bd22e2590 con 0x562bd23f1c10 2024-03-28T22:43:15.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.723+0000 7f5ebea00640 1 --2- 172.21.15.43:0/2736190241 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x562bd23f1c10 0x562bd23f4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.723+0000 7f5ebea00640 1 --2- 172.21.15.43:0/2736190241 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x562bd23f1c10 0x562bd23f4000 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a791867de7e6137f server_cookie=c05c7b38edccdda2 in_seq=0 out_seq=0 2024-03-28T22:43:15.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.723+0000 7f5ea8c00640 10 client.27905 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:43:15.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.723+0000 7f5ea8c00640 1 -- 172.21.15.43:0/2736190241 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562bd22e2590 con 0x562bd23f1c10 2024-03-28T22:43:15.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.723+0000 7f5ea8c00640 10 client.27905 handle_command_reply: tid=0 2024-03-28T22:43:15.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.723+0000 7f5ec8e59740 1 client.27905 shutdown 2024-03-28T22:43:15.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.724+0000 7f5ec8e59740 1 -- 172.21.15.43:0/2736190241 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2371380 msgr2=0x562bd237ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.724+0000 7f5ec8e59740 1 --2- 172.21.15.43:0/2736190241 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2371380 0x562bd237ea50 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8009d30 tx=0x7f5eb8000f40 comp rx=0 tx=0).stop 2024-03-28T22:43:15.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.724+0000 7f5ec8e59740 1 -- 172.21.15.43:0/2736190241 shutdown_connections 2024-03-28T22:43:15.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.724+0000 7f5ec8e59740 1 --2- 172.21.15.43:0/2736190241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562bd2392b60 0x562bd23f0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.724+0000 7f5ec8e59740 1 --2- 172.21.15.43:0/2736190241 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562bd237a880 0x562bd237ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.724+0000 7f5ec8e59740 1 --2- 172.21.15.43:0/2736190241 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x562bd23f1c10 0x562bd23f4000 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.724+0000 7f5ec8e59740 1 --2- 172.21.15.43:0/2736190241 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562bd2371380 0x562bd237ea50 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.724+0000 7f5ec8e59740 1 -- 172.21.15.43:0/2736190241 >> 172.21.15.43:0/2736190241 conn(0x562bd236bab0 msgr2=0x562bd2391b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:15.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.724+0000 7f5ea8c00640 0 client.27905 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:43:15.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.724+0000 7f5ec8e59740 1 -- 172.21.15.43:0/2736190241 shutdown_connections 2024-03-28T22:43:15.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.724+0000 7f5ec8e59740 1 -- 172.21.15.43:0/2736190241 wait complete. 2024-03-28T22:43:15.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.724+0000 7f5ec8e59740 20 client.27905 trim_cache size 0 max 16384 2024-03-28T22:43:15.725 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:43:15.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.725+0000 7f5ec6200640 1 -- 172.21.15.43:0/2398588241 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f5ea40a1ef0 msgr2=0x7f5ea40a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.725+0000 7f5ec6200640 1 --2- 172.21.15.43:0/2398588241 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f5ea40a1ef0 0x7f5ea40a43b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5eb000e9c0 tx=0x7f5eb00d4530 comp rx=0 tx=0).stop 2024-03-28T22:43:15.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.725+0000 7f5ec6200640 1 -- 172.21.15.43:0/2398588241 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ec01038f0 msgr2=0x7f5ec019a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.725+0000 7f5ec6200640 1 --2- 172.21.15.43:0/2398588241 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ec01038f0 0x7f5ec019a2a0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f5eb000e9f0 tx=0x7f5eb000eeb0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.725+0000 7f5ec6200640 1 -- 172.21.15.43:0/2398588241 shutdown_connections 2024-03-28T22:43:15.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.725+0000 7f5ec6200640 1 --2- 172.21.15.43:0/2398588241 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f5ea40a1ef0 0x7f5ea40a43b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.725+0000 7f5ec6200640 1 --2- 172.21.15.43:0/2398588241 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ec0109900 0x7f5ec019eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.725+0000 7f5ec6200640 1 --2- 172.21.15.43:0/2398588241 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ec01042b0 0x7f5ec019a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.725+0000 7f5ec6200640 1 --2- 172.21.15.43:0/2398588241 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ec01038f0 0x7f5ec019a2a0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.725+0000 7f5ec6200640 1 -- 172.21.15.43:0/2398588241 >> 172.21.15.43:0/2398588241 conn(0x7f5ec00ff3a0 msgr2=0x7f5ec00ff9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:15.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.725+0000 7f5ec6200640 1 -- 172.21.15.43:0/2398588241 shutdown_connections 2024-03-28T22:43:15.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.725+0000 7f5ec6200640 1 -- 172.21.15.43:0/2398588241 wait complete. 2024-03-28T22:43:15.741 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:43:15.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.964+0000 7f185e200640 1 -- 172.21.15.43:0/2815656679 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1858068930 msgr2=0x7f1858068d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.964+0000 7f185e200640 1 --2- 172.21.15.43:0/2815656679 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1858068930 0x7f1858068d10 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f1848009900 tx=0x7f184802bc30 comp rx=0 tx=0).stop 2024-03-28T22:43:15.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.965+0000 7f185e200640 1 -- 172.21.15.43:0/2815656679 shutdown_connections 2024-03-28T22:43:15.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.965+0000 7f185e200640 1 --2- 172.21.15.43:0/2815656679 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f185810e0f0 0x7f18581104e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.965+0000 7f185e200640 1 --2- 172.21.15.43:0/2815656679 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1858069250 0x7f185810dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.965+0000 7f185e200640 1 --2- 172.21.15.43:0/2815656679 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1858068930 0x7f1858068d10 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.965+0000 7f185e200640 1 -- 172.21.15.43:0/2815656679 >> 172.21.15.43:0/2815656679 conn(0x7f1858077750 msgr2=0x7f1858077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:15.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.965+0000 7f185e200640 1 -- 172.21.15.43:0/2815656679 shutdown_connections 2024-03-28T22:43:15.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.965+0000 7f185e200640 1 -- 172.21.15.43:0/2815656679 wait complete. 2024-03-28T22:43:15.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.966+0000 7f185e200640 1 Processor -- start 2024-03-28T22:43:15.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.966+0000 7f185e200640 1 -- start start 2024-03-28T22:43:15.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.967+0000 7f185e200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1858068930 0x7f185819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.967+0000 7f185e200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1858069250 0x7f185819a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.967+0000 7f1857400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1858068930 0x7f185819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.967+0000 7f1857400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1858068930 0x7f185819a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:49530/0 (socket says 172.21.15.43:49530) 2024-03-28T22:43:15.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.967+0000 7f185e200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f185810e0f0 0x7f185819ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.967+0000 7f1857400640 1 -- 172.21.15.43:0/3324128032 learned_addr learned my addr 172.21.15.43:0/3324128032 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:15.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.967+0000 7f185e200640 1 -- 172.21.15.43:0/3324128032 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1858113d00 con 0x7f185810e0f0 2024-03-28T22:43:15.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.967+0000 7f185e200640 1 -- 172.21.15.43:0/3324128032 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1858113b80 con 0x7f1858069250 2024-03-28T22:43:15.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.967+0000 7f185e200640 1 -- 172.21.15.43:0/3324128032 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f1858113e80 con 0x7f1858068930 2024-03-28T22:43:15.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.967+0000 7f184ea00640 1 --2- 172.21.15.43:0/3324128032 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1858069250 0x7f185819a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.967+0000 7f1857e00640 1 --2- 172.21.15.43:0/3324128032 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f185810e0f0 0x7f185819ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.968+0000 7f1857400640 1 -- 172.21.15.43:0/3324128032 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1858069250 msgr2=0x7f185819a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.968+0000 7f1857400640 1 --2- 172.21.15.43:0/3324128032 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1858069250 0x7f185819a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.968+0000 7f1857400640 1 -- 172.21.15.43:0/3324128032 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f185810e0f0 msgr2=0x7f185819ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.968+0000 7f1857400640 1 --2- 172.21.15.43:0/3324128032 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f185810e0f0 0x7f185819ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.968+0000 7f1857400640 1 -- 172.21.15.43:0/3324128032 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f185819f1b0 con 0x7f1858068930 2024-03-28T22:43:15.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.968+0000 7f1857e00640 1 --2- 172.21.15.43:0/3324128032 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f185810e0f0 0x7f185819ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:43:15.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.968+0000 7f1857400640 1 --2- 172.21.15.43:0/3324128032 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1858068930 0x7f185819a1c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f184802c240 tx=0x7f18480042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:15.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.968+0000 7f1854c00640 1 -- 172.21.15.43:0/3324128032 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1848043020 con 0x7f1858068930 2024-03-28T22:43:15.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.968+0000 7f1854c00640 1 -- 172.21.15.43:0/3324128032 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18480054e0 con 0x7f1858068930 2024-03-28T22:43:15.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.968+0000 7f185e200640 1 -- 172.21.15.43:0/3324128032 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f185819f440 con 0x7f1858068930 2024-03-28T22:43:15.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.968+0000 7f1854c00640 1 -- 172.21.15.43:0/3324128032 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1848003b20 con 0x7f1858068930 2024-03-28T22:43:15.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.969+0000 7f185e200640 1 -- 172.21.15.43:0/3324128032 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18581a6d30 con 0x7f1858068930 2024-03-28T22:43:15.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.973+0000 7f1860ff7740 1 Processor -- start 2024-03-28T22:43:15.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.973+0000 7f1860ff7740 1 -- start start 2024-03-28T22:43:15.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.973+0000 7f1854c00640 1 -- 172.21.15.43:0/3324128032 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f1848046020 con 0x7f1858068930 2024-03-28T22:43:15.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.974+0000 7f1860ff7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f8d338c380 0x55f8d338c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.974+0000 7f1860ff7740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f8d338cca0 0x55f8d3395340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.974+0000 7f1860ff7740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f8d3395880 0x55f8d3397c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.974+0000 7f1860ff7740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55f8d32f0050 con 0x55f8d3395880 2024-03-28T22:43:15.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.974+0000 7f1860ff7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55f8d3298ce0 con 0x55f8d338c380 2024-03-28T22:43:15.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.974+0000 7f1860ff7740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55f8d32ec0f0 con 0x55f8d338cca0 2024-03-28T22:43:15.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.974+0000 7f1857e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f8d338c380 0x55f8d338c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.974+0000 7f1857400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f8d338cca0 0x55f8d3395340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.974+0000 7f1857e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f8d338c380 0x55f8d338c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:50582/0 (socket says 172.21.15.43:50582) 2024-03-28T22:43:15.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.974+0000 7f1857e00640 1 -- 172.21.15.43:0/62842903 learned_addr learned my addr 172.21.15.43:0/62842903 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:15.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.974+0000 7f184ea00640 1 --2- 172.21.15.43:0/62842903 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f8d3395880 0x55f8d3397c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.974+0000 7f1857e00640 1 -- 172.21.15.43:0/62842903 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f8d338cca0 msgr2=0x55f8d3395340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.975+0000 7f1857e00640 1 --2- 172.21.15.43:0/62842903 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f8d338cca0 0x55f8d3395340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.975+0000 7f1857e00640 1 -- 172.21.15.43:0/62842903 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f8d3395880 msgr2=0x55f8d3397c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.975+0000 7f1857e00640 1 --2- 172.21.15.43:0/62842903 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f8d3395880 0x55f8d3397c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.975+0000 7f1857e00640 1 -- 172.21.15.43:0/62842903 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8d30043f0 con 0x55f8d338c380 2024-03-28T22:43:15.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.975+0000 7f1857e00640 1 --2- 172.21.15.43:0/62842903 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f8d338c380 0x55f8d338c760 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f1850009fc0 tx=0x7f1850010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2157f4e4a5be5d0d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:15.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.975+0000 7f1847e00640 1 -- 172.21.15.43:0/62842903 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1850011700 con 0x55f8d338c380 2024-03-28T22:43:15.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.975+0000 7f1847e00640 1 -- 172.21.15.43:0/62842903 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1850011d80 con 0x55f8d338c380 2024-03-28T22:43:15.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.975+0000 7f1847e00640 1 -- 172.21.15.43:0/62842903 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f185001a380 con 0x55f8d338c380 2024-03-28T22:43:15.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.975+0000 7f1854c00640 1 --2- 172.21.15.43:0/3324128032 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f183c0a1f70 0x7f183c0a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.976+0000 7f1854c00640 1 -- 172.21.15.43:0/3324128032 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f18480f0490 con 0x7f1858068930 2024-03-28T22:43:15.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.976+0000 7f1857400640 1 --2- 172.21.15.43:0/3324128032 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f183c0a1f70 0x7f183c0a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.976+0000 7f1860ff7740 1 -- 172.21.15.43:0/62842903 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f8d338c380 msgr2=0x55f8d338c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.976+0000 7f1860ff7740 1 --2- 172.21.15.43:0/62842903 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f8d338c380 0x55f8d338c760 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f1850009fc0 tx=0x7f1850010860 comp rx=0 tx=0).stop 2024-03-28T22:43:15.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.976+0000 7f1860ff7740 1 -- 172.21.15.43:0/62842903 shutdown_connections 2024-03-28T22:43:15.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.976+0000 7f1860ff7740 1 --2- 172.21.15.43:0/62842903 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f8d3395880 0x55f8d3397c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.976+0000 7f1860ff7740 1 --2- 172.21.15.43:0/62842903 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f8d338cca0 0x55f8d3395340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.976+0000 7f1860ff7740 1 --2- 172.21.15.43:0/62842903 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f8d338c380 0x55f8d338c760 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.976+0000 7f1860ff7740 1 -- 172.21.15.43:0/62842903 >> 172.21.15.43:0/62842903 conn(0x55f8d3386ab0 msgr2=0x55f8d3386e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:15.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.976+0000 7f1860ff7740 1 -- 172.21.15.43:0/62842903 shutdown_connections 2024-03-28T22:43:15.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.976+0000 7f1860ff7740 1 -- 172.21.15.43:0/62842903 wait complete. 2024-03-28T22:43:15.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.976+0000 7f1857400640 1 --2- 172.21.15.43:0/3324128032 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f183c0a1f70 0x7f183c0a4430 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f18480f3fd0 tx=0x7f18480f3f40 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:15.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.977+0000 7f1860ff7740 1 -- start start 2024-03-28T22:43:15.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.977+0000 7f1860ff7740 1 Processor -- start 2024-03-28T22:43:15.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.977+0000 7f1860ff7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f8d338c380 0x55f8d3392860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.978+0000 7f1860ff7740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f8d338cca0 0x55f8d3392da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.978+0000 7f1860ff7740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f8d3395880 0x55f8d33932e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.978+0000 7f1860ff7740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55f8d3398670 con 0x55f8d338cca0 2024-03-28T22:43:15.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.978+0000 7f1860ff7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55f8d328e2f0 con 0x55f8d338c380 2024-03-28T22:43:15.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.978+0000 7f1860ff7740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55f8d33987f0 con 0x55f8d3395880 2024-03-28T22:43:15.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.978+0000 7f1860ff7740 10 client.0 _ll_register_callbacks cb 0x55f8d3003be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:15.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.978+0000 7f1857e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f8d338c380 0x55f8d3392860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.978+0000 7f1857e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f8d338c380 0x55f8d3392860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:50590/0 (socket says 172.21.15.43:50590) 2024-03-28T22:43:15.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.978+0000 7f184ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f8d338cca0 0x55f8d3392da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.978+0000 7f1857e00640 1 -- 172.21.15.43:0/361933294 learned_addr learned my addr 172.21.15.43:0/361933294 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:15.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.978+0000 7f1857e00640 1 --2- 172.21.15.43:0/361933294 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f8d3395880 0x55f8d33932e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.978+0000 7f184ea00640 1 -- 172.21.15.43:0/361933294 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f8d3395880 msgr2=0x55f8d33932e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.978+0000 7f184ea00640 1 --2- 172.21.15.43:0/361933294 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f8d3395880 0x55f8d33932e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.978+0000 7f184ea00640 1 -- 172.21.15.43:0/361933294 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f8d338c380 msgr2=0x55f8d3392860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:15.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.978+0000 7f184ea00640 1 --2- 172.21.15.43:0/361933294 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f8d338c380 0x55f8d3392860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.978+0000 7f184ea00640 1 -- 172.21.15.43:0/361933294 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8d30ce7d0 con 0x55f8d338cca0 2024-03-28T22:43:15.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.979+0000 7f184ea00640 1 --2- 172.21.15.43:0/361933294 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f8d338cca0 0x55f8d3392da0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f18400126d0 tx=0x7f1840012b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=296ea84e1a1d3e58 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:15.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.979+0000 7f1844c00640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:43:15.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.979+0000 7f1844c00640 1 -- 172.21.15.43:0/361933294 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1840004050 con 0x55f8d338cca0 2024-03-28T22:43:15.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.979+0000 7f1844c00640 1 -- 172.21.15.43:0/361933294 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f18400041f0 con 0x55f8d338cca0 2024-03-28T22:43:15.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.979+0000 7f1860ff7740 1 -- 172.21.15.43:0/361933294 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1850011d80 con 0x55f8d338cca0 2024-03-28T22:43:15.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.979+0000 7f1844c00640 1 -- 172.21.15.43:0/361933294 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1840025050 con 0x55f8d338cca0 2024-03-28T22:43:15.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.979+0000 7f1860ff7740 10 client.17679 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:15.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.979+0000 7f1860ff7740 1 -- 172.21.15.43:0/361933294 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f8d33a4920 con 0x55f8d338cca0 2024-03-28T22:43:15.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.980+0000 7f1844c00640 1 -- 172.21.15.43:0/361933294 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1850011d80 con 0x55f8d338cca0 2024-03-28T22:43:15.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.980+0000 7f1844c00640 1 -- 172.21.15.43:0/361933294 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f1840003a50 con 0x55f8d338cca0 2024-03-28T22:43:15.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.980+0000 7f1860ff7740 10 client.17679 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:15.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.980+0000 7f1860ff7740 10 client.17679 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:43:15.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.980+0000 7f1860ff7740 20 client.17679 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:15.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.980+0000 7f1860ff7740 1 --2- 172.21.15.43:0/361933294 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55f8d33a5de0 0x55f8d33a8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:15.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.980+0000 7f1860ff7740 4 client.17679 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:43:15.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.980+0000 7f1860ff7740 1 -- 172.21.15.43:0/361933294 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f8d3299680 con 0x55f8d33a5de0 2024-03-28T22:43:15.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.980+0000 7f1857e00640 1 --2- 172.21.15.43:0/361933294 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55f8d33a5de0 0x55f8d33a8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:15.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.981+0000 7f1857e00640 1 --2- 172.21.15.43:0/361933294 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55f8d33a5de0 0x55f8d33a8210 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4ba43adf679ab74b server_cookie=f55255f60bcd88e2 in_seq=0 out_seq=0 2024-03-28T22:43:15.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.981+0000 7f1844c00640 10 client.17679 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:43:15.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.982+0000 7f1844c00640 1 -- 172.21.15.43:0/361933294 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f8d3299680 con 0x55f8d33a5de0 2024-03-28T22:43:15.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.982+0000 7f1844c00640 10 client.17679 handle_command_reply: tid=0 2024-03-28T22:43:15.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.982+0000 7f1860ff7740 1 client.17679 shutdown 2024-03-28T22:43:15.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.982+0000 7f1860ff7740 1 -- 172.21.15.43:0/361933294 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f8d338cca0 msgr2=0x55f8d3392da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:15.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.982+0000 7f1860ff7740 1 --2- 172.21.15.43:0/361933294 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f8d338cca0 0x55f8d3392da0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f18400126d0 tx=0x7f1840012b90 comp rx=0 tx=0).stop 2024-03-28T22:43:15.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.982+0000 7f1860ff7740 1 -- 172.21.15.43:0/361933294 shutdown_connections 2024-03-28T22:43:15.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.982+0000 7f1860ff7740 1 --2- 172.21.15.43:0/361933294 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55f8d33a5de0 0x55f8d33a8210 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.982+0000 7f1860ff7740 1 --2- 172.21.15.43:0/361933294 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f8d3395880 0x55f8d33932e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.982+0000 7f1860ff7740 1 --2- 172.21.15.43:0/361933294 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f8d338cca0 0x55f8d3392da0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.982+0000 7f1860ff7740 1 --2- 172.21.15.43:0/361933294 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f8d338c380 0x55f8d3392860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:15.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.982+0000 7f1860ff7740 1 -- 172.21.15.43:0/361933294 >> 172.21.15.43:0/361933294 conn(0x55f8d3386ab0 msgr2=0x55f8d3388800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:15.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.982+0000 7f1844c00640 0 client.17679 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:43:15.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.983+0000 7f1860ff7740 1 -- 172.21.15.43:0/361933294 shutdown_connections 2024-03-28T22:43:15.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.983+0000 7f1860ff7740 1 -- 172.21.15.43:0/361933294 wait complete. 2024-03-28T22:43:15.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:15.983+0000 7f1860ff7740 20 client.17679 trim_cache size 0 max 16384 2024-03-28T22:43:16.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.010+0000 7f1860ff7740 1 Processor -- start 2024-03-28T22:43:16.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.010+0000 7f1860ff7740 1 -- start start 2024-03-28T22:43:16.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.010+0000 7f1860ff7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f8d338c380 0x55f8d3394f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:16.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.011+0000 7f1860ff7740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f8d338cca0 0x55f8d33a1410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:16.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.011+0000 7f1860ff7740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f8d3395880 0x55f8d33adb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:16.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.011+0000 7f1860ff7740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f18581b0d50 con 0x55f8d3395880 2024-03-28T22:43:16.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.011+0000 7f1860ff7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f18581b2890 con 0x55f8d338c380 2024-03-28T22:43:16.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.011+0000 7f1860ff7740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f18581b1420 con 0x55f8d338cca0 2024-03-28T22:43:16.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.011+0000 7f1857e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f8d338c380 0x55f8d3394f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:16.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.012+0000 7f1857e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f8d338c380 0x55f8d3394f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:50600/0 (socket says 172.21.15.43:50600) 2024-03-28T22:43:16.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.012+0000 7f1857e00640 1 -- 172.21.15.43:0/3097736110 learned_addr learned my addr 172.21.15.43:0/3097736110 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:16.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.012+0000 7f184ea00640 1 --2- 172.21.15.43:0/3097736110 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f8d338cca0 0x55f8d33a1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:16.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.013+0000 7f1857e00640 1 --2- 172.21.15.43:0/3097736110 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f8d3395880 0x55f8d33adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:16.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.014+0000 7f184ea00640 1 -- 172.21.15.43:0/3097736110 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f8d338c380 msgr2=0x55f8d3394f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:16.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.014+0000 7f184ea00640 1 --2- 172.21.15.43:0/3097736110 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f8d338c380 0x55f8d3394f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.014+0000 7f184ea00640 1 -- 172.21.15.43:0/3097736110 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f8d3395880 msgr2=0x55f8d33adb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:16.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.014+0000 7f184ea00640 1 --2- 172.21.15.43:0/3097736110 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f8d3395880 0x55f8d33adb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.014+0000 7f184ea00640 1 -- 172.21.15.43:0/3097736110 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8d3390db0 con 0x55f8d338cca0 2024-03-28T22:43:16.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.015+0000 7f1857e00640 1 --2- 172.21.15.43:0/3097736110 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f8d3395880 0x55f8d33adb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:16.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.016+0000 7f184ea00640 1 --2- 172.21.15.43:0/3097736110 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f8d338cca0 0x55f8d33a1410 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f1840000c00 tx=0x7f1840010750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc6edc8332e6dd95 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:16.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.017+0000 7f1847400640 1 -- 172.21.15.43:0/3097736110 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f184002e070 con 0x55f8d338cca0 2024-03-28T22:43:16.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.017+0000 7f1847400640 1 -- 172.21.15.43:0/3097736110 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f184000fe00 con 0x55f8d338cca0 2024-03-28T22:43:16.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.017+0000 7f1847400640 1 -- 172.21.15.43:0/3097736110 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18400055f0 con 0x55f8d338cca0 2024-03-28T22:43:16.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.020+0000 7f1860ff7740 1 -- 172.21.15.43:0/3097736110 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f8d338cca0 msgr2=0x55f8d33a1410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:16.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.020+0000 7f1860ff7740 1 --2- 172.21.15.43:0/3097736110 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f8d338cca0 0x55f8d33a1410 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f1840000c00 tx=0x7f1840010750 comp rx=0 tx=0).stop 2024-03-28T22:43:16.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.021+0000 7f1860ff7740 1 -- 172.21.15.43:0/3097736110 shutdown_connections 2024-03-28T22:43:16.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.021+0000 7f1860ff7740 1 --2- 172.21.15.43:0/3097736110 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f8d3395880 0x55f8d33adb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.021+0000 7f1860ff7740 1 --2- 172.21.15.43:0/3097736110 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f8d338cca0 0x55f8d33a1410 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.021+0000 7f1860ff7740 1 --2- 172.21.15.43:0/3097736110 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f8d338c380 0x55f8d3394f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.021+0000 7f1860ff7740 1 -- 172.21.15.43:0/3097736110 >> 172.21.15.43:0/3097736110 conn(0x55f8d3386ab0 msgr2=0x55f8d338d080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:16.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.023+0000 7f1860ff7740 1 -- 172.21.15.43:0/3097736110 shutdown_connections 2024-03-28T22:43:16.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.023+0000 7f1860ff7740 1 -- 172.21.15.43:0/3097736110 wait complete. 2024-03-28T22:43:16.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.024+0000 7f1860ff7740 1 -- start start 2024-03-28T22:43:16.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.025+0000 7f1860ff7740 1 Processor -- start 2024-03-28T22:43:16.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.027+0000 7f1860ff7740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f8d338c380 0x55f8d338fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:16.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.027+0000 7f1860ff7740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f8d338cca0 0x55f8d338ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:16.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.027+0000 7f1860ff7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f8d3395880 0x55f8d340b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:16.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.027+0000 7f1860ff7740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f18581b21c0 con 0x55f8d338c380 2024-03-28T22:43:16.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.027+0000 7f1860ff7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f18581b1af0 con 0x55f8d3395880 2024-03-28T22:43:16.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.027+0000 7f1860ff7740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f18581aed30 con 0x55f8d338cca0 2024-03-28T22:43:16.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.027+0000 7f1860ff7740 10 client.0 _ll_register_callbacks cb 0x7f18400097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:16.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.027+0000 7f184ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f8d338cca0 0x55f8d338ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:16.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.028+0000 7f184ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f8d338cca0 0x55f8d338ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:49570/0 (socket says 172.21.15.43:49570) 2024-03-28T22:43:16.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.028+0000 7f184ea00640 1 -- 172.21.15.43:0/868603502 learned_addr learned my addr 172.21.15.43:0/868603502 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:16.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.028+0000 7f1857e00640 1 --2- 172.21.15.43:0/868603502 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f8d338c380 0x55f8d338fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:16.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.028+0000 7f1857e00640 1 --2- 172.21.15.43:0/868603502 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f8d3395880 0x55f8d340b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:16.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.028+0000 7f184ea00640 1 -- 172.21.15.43:0/868603502 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f8d3395880 msgr2=0x55f8d340b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:16.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.028+0000 7f184ea00640 1 --2- 172.21.15.43:0/868603502 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f8d3395880 0x55f8d340b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.028+0000 7f184ea00640 1 -- 172.21.15.43:0/868603502 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f8d338c380 msgr2=0x55f8d338fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:16.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.028+0000 7f184ea00640 1 --2- 172.21.15.43:0/868603502 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f8d338c380 0x55f8d338fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.028+0000 7f184ea00640 1 -- 172.21.15.43:0/868603502 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8d340b690 con 0x55f8d338cca0 2024-03-28T22:43:16.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.029+0000 7f1857e00640 1 --2- 172.21.15.43:0/868603502 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f8d3395880 0x55f8d340b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:43:16.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.030+0000 7f184ea00640 1 --2- 172.21.15.43:0/868603502 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f8d338cca0 0x55f8d338ff50 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f1840010010 tx=0x7f1840004f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ce4b9ec25438c38 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:16.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.031+0000 7f1844c00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:43:16.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.031+0000 7f1844c00640 1 -- 172.21.15.43:0/868603502 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1840037060 con 0x55f8d338cca0 2024-03-28T22:43:16.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.031+0000 7f1844c00640 1 -- 172.21.15.43:0/868603502 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f1840022050 con 0x55f8d338cca0 2024-03-28T22:43:16.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.032+0000 7f1860ff7740 1 -- 172.21.15.43:0/868603502 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f184000fe00 con 0x55f8d338cca0 2024-03-28T22:43:16.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.032+0000 7f1844c00640 1 -- 172.21.15.43:0/868603502 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1840037060 con 0x55f8d338cca0 2024-03-28T22:43:16.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.033+0000 7f1860ff7740 10 client.27862 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:16.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.033+0000 7f1860ff7740 1 -- 172.21.15.43:0/868603502 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f8d340c840 con 0x55f8d338cca0 2024-03-28T22:43:16.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.034+0000 7f1844c00640 1 -- 172.21.15.43:0/868603502 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f184000fe00 con 0x55f8d338cca0 2024-03-28T22:43:16.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.034+0000 7f1844c00640 1 -- 172.21.15.43:0/868603502 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f1840010cc0 con 0x55f8d338cca0 2024-03-28T22:43:16.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.034+0000 7f1860ff7740 10 client.27862 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:16.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.034+0000 7f1860ff7740 10 client.27862 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:43:16.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.034+0000 7f1860ff7740 20 client.27862 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:16.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.034+0000 7f1860ff7740 1 --2- 172.21.15.43:0/868603502 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55f8d340ce50 0x55f8d340f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:16.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.034+0000 7f1860ff7740 4 client.27862 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:43:16.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.034+0000 7f1860ff7740 1 -- 172.21.15.43:0/868603502 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f8d32fd590 con 0x55f8d340ce50 2024-03-28T22:43:16.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.036+0000 7f1857e00640 1 --2- 172.21.15.43:0/868603502 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55f8d340ce50 0x55f8d340f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:16.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.037+0000 7f1857e00640 1 --2- 172.21.15.43:0/868603502 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55f8d340ce50 0x55f8d340f270 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2ad87ffb2c131ecd server_cookie=44b623a890e6d549 in_seq=0 out_seq=0 2024-03-28T22:43:16.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.037+0000 7f1844c00640 10 client.27862 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:43:16.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.038+0000 7f1844c00640 1 -- 172.21.15.43:0/868603502 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f8d32fd590 con 0x55f8d340ce50 2024-03-28T22:43:16.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.038+0000 7f1844c00640 10 client.27862 handle_command_reply: tid=0 2024-03-28T22:43:16.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.038+0000 7f1860ff7740 1 client.27862 shutdown 2024-03-28T22:43:16.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.038+0000 7f1860ff7740 1 -- 172.21.15.43:0/868603502 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f8d338cca0 msgr2=0x55f8d338ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:16.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.038+0000 7f1860ff7740 1 --2- 172.21.15.43:0/868603502 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f8d338cca0 0x55f8d338ff50 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f1840010010 tx=0x7f1840004f80 comp rx=0 tx=0).stop 2024-03-28T22:43:16.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.038+0000 7f1860ff7740 1 -- 172.21.15.43:0/868603502 shutdown_connections 2024-03-28T22:43:16.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.038+0000 7f1860ff7740 1 --2- 172.21.15.43:0/868603502 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55f8d340ce50 0x55f8d340f270 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.038+0000 7f1860ff7740 1 --2- 172.21.15.43:0/868603502 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f8d3395880 0x55f8d340b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.039+0000 7f1860ff7740 1 --2- 172.21.15.43:0/868603502 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f8d338cca0 0x55f8d338ff50 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.039+0000 7f1860ff7740 1 --2- 172.21.15.43:0/868603502 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f8d338c380 0x55f8d338fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.039+0000 7f1860ff7740 1 -- 172.21.15.43:0/868603502 >> 172.21.15.43:0/868603502 conn(0x55f8d3386ab0 msgr2=0x55f8d33852e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:16.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.039+0000 7f1844c00640 0 client.27862 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:43:16.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.040+0000 7f1860ff7740 1 -- 172.21.15.43:0/868603502 shutdown_connections 2024-03-28T22:43:16.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.040+0000 7f1860ff7740 1 -- 172.21.15.43:0/868603502 wait complete. 2024-03-28T22:43:16.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.040+0000 7f1860ff7740 20 client.27862 trim_cache size 0 max 16384 2024-03-28T22:43:16.038 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:43:16.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.041+0000 7f185e200640 1 -- 172.21.15.43:0/3324128032 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f183c0a1f70 msgr2=0x7f183c0a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:16.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.041+0000 7f185e200640 1 --2- 172.21.15.43:0/3324128032 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f183c0a1f70 0x7f183c0a4430 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f18480f3fd0 tx=0x7f18480f3f40 comp rx=0 tx=0).stop 2024-03-28T22:43:16.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.041+0000 7f185e200640 1 -- 172.21.15.43:0/3324128032 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1858068930 msgr2=0x7f185819a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:16.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.041+0000 7f185e200640 1 --2- 172.21.15.43:0/3324128032 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1858068930 0x7f185819a1c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f184802c240 tx=0x7f18480042f0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.042+0000 7f185e200640 1 -- 172.21.15.43:0/3324128032 shutdown_connections 2024-03-28T22:43:16.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.042+0000 7f185e200640 1 --2- 172.21.15.43:0/3324128032 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f183c0a1f70 0x7f183c0a4430 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.042+0000 7f185e200640 1 --2- 172.21.15.43:0/3324128032 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1858069250 0x7f185819a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.042+0000 7f185e200640 1 --2- 172.21.15.43:0/3324128032 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f185810e0f0 0x7f185819ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.042+0000 7f185e200640 1 --2- 172.21.15.43:0/3324128032 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1858068930 0x7f185819a1c0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.042+0000 7f185e200640 1 -- 172.21.15.43:0/3324128032 >> 172.21.15.43:0/3324128032 conn(0x7f1858077750 msgr2=0x7f185810f570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:16.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.042+0000 7f185e200640 1 -- 172.21.15.43:0/3324128032 shutdown_connections 2024-03-28T22:43:16.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.043+0000 7f185e200640 1 -- 172.21.15.43:0/3324128032 wait complete. 2024-03-28T22:43:16.072 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:43:16.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.274+0000 7fe33ce00640 1 -- 172.21.15.43:0/3953101365 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe338107f00 msgr2=0x7fe3381082e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:16.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.274+0000 7fe33ce00640 1 --2- 172.21.15.43:0/3953101365 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe338107f00 0x7fe3381082e0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fe328009900 tx=0x7fe32802bc30 comp rx=0 tx=0).stop 2024-03-28T22:43:16.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.275+0000 7fe33ce00640 1 -- 172.21.15.43:0/3953101365 shutdown_connections 2024-03-28T22:43:16.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.275+0000 7fe33ce00640 1 --2- 172.21.15.43:0/3953101365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe33810cf50 0x7fe338114b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.275+0000 7fe33ce00640 1 --2- 172.21.15.43:0/3953101365 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3381088b0 0x7fe33810c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.275+0000 7fe33ce00640 1 --2- 172.21.15.43:0/3953101365 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe338107f00 0x7fe3381082e0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.275+0000 7fe33ce00640 1 -- 172.21.15.43:0/3953101365 >> 172.21.15.43:0/3953101365 conn(0x7fe338077570 msgr2=0x7fe338106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:16.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.275+0000 7fe33ce00640 1 -- 172.21.15.43:0/3953101365 shutdown_connections 2024-03-28T22:43:16.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.275+0000 7fe33ce00640 1 -- 172.21.15.43:0/3953101365 wait complete. 2024-03-28T22:43:16.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.276+0000 7fe33ce00640 1 Processor -- start 2024-03-28T22:43:16.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.276+0000 7fe33ce00640 1 -- start start 2024-03-28T22:43:16.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.276+0000 7fe33ce00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe338107f00 0x7fe33819e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:16.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.276+0000 7fe33ce00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe33810cf50 0x7fe33819e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:16.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.276+0000 7fe33ce00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3381a2d10 0x7fe3381a31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:16.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.276+0000 7fe33ce00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe338118000 con 0x7fe338107f00 2024-03-28T22:43:16.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.276+0000 7fe33ce00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe338117e80 con 0x7fe33810cf50 2024-03-28T22:43:16.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.276+0000 7fe33ce00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe338118180 con 0x7fe3381a2d10 2024-03-28T22:43:16.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.277+0000 7fe337400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe338107f00 0x7fe33819e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:16.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.277+0000 7fe336a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe33810cf50 0x7fe33819e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:16.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.277+0000 7fe337400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe338107f00 0x7fe33819e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46222/0 (socket says 172.21.15.43:46222) 2024-03-28T22:43:16.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.277+0000 7fe337400640 1 -- 172.21.15.43:0/555751810 learned_addr learned my addr 172.21.15.43:0/555751810 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:16.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.277+0000 7fe337e00640 1 --2- 172.21.15.43:0/555751810 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3381a2d10 0x7fe3381a31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:16.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.277+0000 7fe336a00640 1 -- 172.21.15.43:0/555751810 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3381a2d10 msgr2=0x7fe3381a31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:16.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.277+0000 7fe336a00640 1 --2- 172.21.15.43:0/555751810 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3381a2d10 0x7fe3381a31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.277+0000 7fe336a00640 1 -- 172.21.15.43:0/555751810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe338107f00 msgr2=0x7fe33819e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:16.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.277+0000 7fe336a00640 1 --2- 172.21.15.43:0/555751810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe338107f00 0x7fe33819e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.277+0000 7fe336a00640 1 -- 172.21.15.43:0/555751810 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3381a3800 con 0x7fe33810cf50 2024-03-28T22:43:16.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.278+0000 7fe336a00640 1 --2- 172.21.15.43:0/555751810 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe33810cf50 0x7fe33819e9e0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fe338108e70 tx=0x7fe32000ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:16.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.278+0000 7fe32fe00640 1 -- 172.21.15.43:0/555751810 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32000cd00 con 0x7fe33810cf50 2024-03-28T22:43:16.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.278+0000 7fe32fe00640 1 -- 172.21.15.43:0/555751810 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3200035f0 con 0x7fe33810cf50 2024-03-28T22:43:16.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.278+0000 7fe33ce00640 1 -- 172.21.15.43:0/555751810 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3381a3af0 con 0x7fe33810cf50 2024-03-28T22:43:16.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.278+0000 7fe32fe00640 1 -- 172.21.15.43:0/555751810 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe320010690 con 0x7fe33810cf50 2024-03-28T22:43:16.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.278+0000 7fe33ce00640 1 -- 172.21.15.43:0/555751810 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3381ab280 con 0x7fe33810cf50 2024-03-28T22:43:16.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.281+0000 7fe341006740 1 Processor -- start 2024-03-28T22:43:16.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.281+0000 7fe341006740 1 -- start start 2024-03-28T22:43:16.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.281+0000 7fe32fe00640 1 -- 172.21.15.43:0/555751810 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fe3200037a0 con 0x7fe33810cf50 2024-03-28T22:43:16.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.282+0000 7fe341006740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562d1f5f3550 0x562d1f5f3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:16.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.282+0000 7fe341006740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d1f5f3e70 0x562d1f5fc510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:16.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.282+0000 7fe341006740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562d1f5fca50 0x562d1f5fee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:16.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.282+0000 7fe341006740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562d1f557050 con 0x562d1f5f3e70 2024-03-28T22:43:16.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.282+0000 7fe341006740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x562d1f4ffce0 con 0x562d1f5fca50 2024-03-28T22:43:16.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.282+0000 7fe341006740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x562d1f5530f0 con 0x562d1f5f3550 2024-03-28T22:43:16.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.282+0000 7fe337e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562d1f5f3550 0x562d1f5f3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:16.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.282+0000 7fe336a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562d1f5fca50 0x562d1f5fee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:16.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.282+0000 7fe337e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562d1f5f3550 0x562d1f5f3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:49596/0 (socket says 172.21.15.43:49596) 2024-03-28T22:43:16.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.282+0000 7fe337e00640 1 -- 172.21.15.43:0/3349605970 learned_addr learned my addr 172.21.15.43:0/3349605970 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:16.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.282+0000 7fe337400640 1 --2- 172.21.15.43:0/3349605970 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d1f5f3e70 0x562d1f5fc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:16.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.282+0000 7fe337e00640 1 -- 172.21.15.43:0/3349605970 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562d1f5fca50 msgr2=0x562d1f5fee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:16.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.282+0000 7fe337e00640 1 --2- 172.21.15.43:0/3349605970 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562d1f5fca50 0x562d1f5fee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.282+0000 7fe337e00640 1 -- 172.21.15.43:0/3349605970 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d1f5f3e70 msgr2=0x562d1f5fc510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:16.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.282+0000 7fe337e00640 1 --2- 172.21.15.43:0/3349605970 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d1f5f3e70 0x562d1f5fc510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.282+0000 7fe337e00640 1 -- 172.21.15.43:0/3349605970 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d1f26b3f0 con 0x562d1f5f3550 2024-03-28T22:43:16.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.283+0000 7fe337e00640 1 --2- 172.21.15.43:0/3349605970 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562d1f5f3550 0x562d1f5f3930 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe330009f40 tx=0x7fe3300118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f7bf7bde9a18b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:16.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.283+0000 7fe32cc00640 1 -- 172.21.15.43:0/3349605970 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe330004d20 con 0x562d1f5f3550 2024-03-28T22:43:16.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.283+0000 7fe32cc00640 1 -- 172.21.15.43:0/3349605970 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe330004050 con 0x562d1f5f3550 2024-03-28T22:43:16.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.283+0000 7fe32cc00640 1 -- 172.21.15.43:0/3349605970 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe330005240 con 0x562d1f5f3550 2024-03-28T22:43:16.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.283+0000 7fe32fe00640 1 --2- 172.21.15.43:0/555751810 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe31c0a1ef0 0x7fe31c0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:16.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.283+0000 7fe32fe00640 1 -- 172.21.15.43:0/555751810 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fe3200ce7e0 con 0x7fe33810cf50 2024-03-28T22:43:16.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.284+0000 7fe337400640 1 --2- 172.21.15.43:0/555751810 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe31c0a1ef0 0x7fe31c0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:16.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.284+0000 7fe341006740 1 -- 172.21.15.43:0/3349605970 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562d1f5f3550 msgr2=0x562d1f5f3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:16.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.284+0000 7fe341006740 1 --2- 172.21.15.43:0/3349605970 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562d1f5f3550 0x562d1f5f3930 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe330009f40 tx=0x7fe3300118d0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.284+0000 7fe341006740 1 -- 172.21.15.43:0/3349605970 shutdown_connections 2024-03-28T22:43:16.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.284+0000 7fe341006740 1 --2- 172.21.15.43:0/3349605970 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562d1f5fca50 0x562d1f5fee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.284+0000 7fe341006740 1 --2- 172.21.15.43:0/3349605970 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d1f5f3e70 0x562d1f5fc510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.284+0000 7fe341006740 1 --2- 172.21.15.43:0/3349605970 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562d1f5f3550 0x562d1f5f3930 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.284+0000 7fe341006740 1 -- 172.21.15.43:0/3349605970 >> 172.21.15.43:0/3349605970 conn(0x562d1f5edc80 msgr2=0x562d1f5ee060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:16.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.284+0000 7fe341006740 1 -- 172.21.15.43:0/3349605970 shutdown_connections 2024-03-28T22:43:16.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.284+0000 7fe341006740 1 -- 172.21.15.43:0/3349605970 wait complete. 2024-03-28T22:43:16.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.284+0000 7fe337400640 1 --2- 172.21.15.43:0/555751810 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe31c0a1ef0 0x7fe31c0a43b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe32802cfd0 tx=0x7fe32802e040 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:16.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.285+0000 7fe341006740 1 -- start start 2024-03-28T22:43:16.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.285+0000 7fe341006740 1 Processor -- start 2024-03-28T22:43:16.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.286+0000 7fe341006740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562d1f5f3550 0x562d1f5f9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:16.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.286+0000 7fe341006740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562d1f5f3e70 0x562d1f5f9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:16.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.286+0000 7fe341006740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d1f5fca50 0x562d1f5fa520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:16.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.286+0000 7fe341006740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562d1f5ff840 con 0x562d1f5fca50 2024-03-28T22:43:16.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.286+0000 7fe341006740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x562d1f4f52f0 con 0x562d1f5f3550 2024-03-28T22:43:16.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.286+0000 7fe341006740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x562d1f5ff9c0 con 0x562d1f5f3e70 2024-03-28T22:43:16.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.286+0000 7fe341006740 10 client.0 _ll_register_callbacks cb 0x562d1f26abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:16.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.286+0000 7fe337400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562d1f5f3550 0x562d1f5f9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:16.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.286+0000 7fe336a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562d1f5f3e70 0x562d1f5f9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:16.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.286+0000 7fe337400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562d1f5f3550 0x562d1f5f9aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:50636/0 (socket says 172.21.15.43:50636) 2024-03-28T22:43:16.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.286+0000 7fe337400640 1 -- 172.21.15.43:0/4126262776 learned_addr learned my addr 172.21.15.43:0/4126262776 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:16.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.286+0000 7fe337e00640 1 --2- 172.21.15.43:0/4126262776 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d1f5fca50 0x562d1f5fa520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:16.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.286+0000 7fe336a00640 1 -- 172.21.15.43:0/4126262776 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562d1f5f3550 msgr2=0x562d1f5f9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:16.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.286+0000 7fe336a00640 1 --2- 172.21.15.43:0/4126262776 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562d1f5f3550 0x562d1f5f9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.286+0000 7fe336a00640 1 -- 172.21.15.43:0/4126262776 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d1f5fca50 msgr2=0x562d1f5fa520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:16.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.286+0000 7fe336a00640 1 --2- 172.21.15.43:0/4126262776 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d1f5fca50 0x562d1f5fa520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.286+0000 7fe336a00640 1 -- 172.21.15.43:0/4126262776 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d1f3357d0 con 0x562d1f5f3e70 2024-03-28T22:43:16.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.286+0000 7fe337e00640 1 --2- 172.21.15.43:0/4126262776 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d1f5fca50 0x562d1f5fa520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:43:16.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.287+0000 7fe336a00640 1 --2- 172.21.15.43:0/4126262776 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562d1f5f3e70 0x562d1f5f9fe0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe3200cbb30 tx=0x7fe320097070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b2f4f06d11e2d08 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:16.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.287+0000 7fe325600640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:43:16.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.287+0000 7fe325600640 1 -- 172.21.15.43:0/4126262776 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32009a730 con 0x562d1f5f3e70 2024-03-28T22:43:16.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.287+0000 7fe341006740 1 -- 172.21.15.43:0/4126262776 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe330004050 con 0x562d1f5f3e70 2024-03-28T22:43:16.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.287+0000 7fe325600640 1 -- 172.21.15.43:0/4126262776 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fe3200cbdb0 con 0x562d1f5f3e70 2024-03-28T22:43:16.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.287+0000 7fe325600640 1 -- 172.21.15.43:0/4126262776 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe3200cbfb0 con 0x562d1f5f3e70 2024-03-28T22:43:16.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.287+0000 7fe341006740 10 client.27874 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:16.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.287+0000 7fe341006740 1 -- 172.21.15.43:0/4126262776 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562d1f60bbb0 con 0x562d1f5f3e70 2024-03-28T22:43:16.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.287+0000 7fe325600640 1 -- 172.21.15.43:0/4126262776 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe330004050 con 0x562d1f5f3e70 2024-03-28T22:43:16.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.288+0000 7fe325600640 1 -- 172.21.15.43:0/4126262776 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe3200ce3c0 con 0x562d1f5f3e70 2024-03-28T22:43:16.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.288+0000 7fe341006740 10 client.27874 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:16.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.288+0000 7fe341006740 10 client.27874 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:43:16.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.288+0000 7fe341006740 20 client.27874 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:16.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.288+0000 7fe341006740 1 --2- 172.21.15.43:0/4126262776 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x562d1f60d0c0 0x562d1f60f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:16.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.288+0000 7fe341006740 4 client.27874 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:43:16.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.288+0000 7fe341006740 1 -- 172.21.15.43:0/4126262776 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d1f500680 con 0x562d1f60d0c0 2024-03-28T22:43:16.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.288+0000 7fe337400640 1 --2- 172.21.15.43:0/4126262776 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x562d1f60d0c0 0x562d1f60f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:16.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.289+0000 7fe337400640 1 --2- 172.21.15.43:0/4126262776 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x562d1f60d0c0 0x562d1f60f4d0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=285b645f6ec88ecd server_cookie=d219fd8b0844d32e in_seq=0 out_seq=0 2024-03-28T22:43:16.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.289+0000 7fe325600640 10 client.27874 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:16.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.291+0000 7fe325600640 1 -- 172.21.15.43:0/4126262776 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562d1f500680 con 0x562d1f60d0c0 2024-03-28T22:43:16.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.291+0000 7fe325600640 10 client.27874 handle_command_reply: tid=0 2024-03-28T22:43:16.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.291+0000 7fe341006740 1 client.27874 shutdown 2024-03-28T22:43:16.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.291+0000 7fe341006740 1 -- 172.21.15.43:0/4126262776 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562d1f5f3e70 msgr2=0x562d1f5f9fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:16.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.291+0000 7fe341006740 1 --2- 172.21.15.43:0/4126262776 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562d1f5f3e70 0x562d1f5f9fe0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe3200cbb30 tx=0x7fe320097070 comp rx=0 tx=0).stop 2024-03-28T22:43:16.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.292+0000 7fe341006740 1 -- 172.21.15.43:0/4126262776 shutdown_connections 2024-03-28T22:43:16.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.292+0000 7fe341006740 1 --2- 172.21.15.43:0/4126262776 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d1f5fca50 0x562d1f5fa520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.292+0000 7fe341006740 1 --2- 172.21.15.43:0/4126262776 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562d1f5f3e70 0x562d1f5f9fe0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.292+0000 7fe341006740 1 --2- 172.21.15.43:0/4126262776 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x562d1f60d0c0 0x562d1f60f4d0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.292+0000 7fe341006740 1 --2- 172.21.15.43:0/4126262776 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562d1f5f3550 0x562d1f5f9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.292+0000 7fe341006740 1 -- 172.21.15.43:0/4126262776 >> 172.21.15.43:0/4126262776 conn(0x562d1f5edc80 msgr2=0x562d1f5ef9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:16.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.292+0000 7fe325600640 0 client.27874 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:16.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.292+0000 7fe341006740 1 -- 172.21.15.43:0/4126262776 shutdown_connections 2024-03-28T22:43:16.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.292+0000 7fe341006740 1 -- 172.21.15.43:0/4126262776 wait complete. 2024-03-28T22:43:16.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.292+0000 7fe341006740 20 client.27874 trim_cache size 0 max 16384 2024-03-28T22:43:16.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.307+0000 7fe341006740 1 Processor -- start 2024-03-28T22:43:16.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.307+0000 7fe341006740 1 -- start start 2024-03-28T22:43:16.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.307+0000 7fe341006740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d1f5f3550 0x562d1f6085e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:16.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.307+0000 7fe341006740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562d1f5f3e70 0x562d1f5fa340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:16.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.307+0000 7fe341006740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562d1f5fca50 0x562d1f614da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:16.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.307+0000 7fe341006740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe3381b4e60 con 0x562d1f5f3550 2024-03-28T22:43:16.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.307+0000 7fe341006740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe3381b69a0 con 0x562d1f5f3e70 2024-03-28T22:43:16.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.307+0000 7fe341006740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe3381b5530 con 0x562d1f5fca50 2024-03-28T22:43:16.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.307+0000 7fe337400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d1f5f3550 0x562d1f6085e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:16.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.307+0000 7fe337400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d1f5f3550 0x562d1f6085e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46268/0 (socket says 172.21.15.43:46268) 2024-03-28T22:43:16.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.307+0000 7fe337e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562d1f5fca50 0x562d1f614da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:16.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.307+0000 7fe337400640 1 -- 172.21.15.43:0/116041979 learned_addr learned my addr 172.21.15.43:0/116041979 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:16.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.307+0000 7fe336a00640 1 --2- 172.21.15.43:0/116041979 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562d1f5f3e70 0x562d1f5fa340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:16.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.308+0000 7fe337400640 1 -- 172.21.15.43:0/116041979 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562d1f5fca50 msgr2=0x562d1f614da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:16.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.308+0000 7fe337400640 1 --2- 172.21.15.43:0/116041979 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562d1f5fca50 0x562d1f614da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.308+0000 7fe337400640 1 -- 172.21.15.43:0/116041979 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562d1f5f3e70 msgr2=0x562d1f5fa340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:16.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.308+0000 7fe337400640 1 --2- 172.21.15.43:0/116041979 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562d1f5f3e70 0x562d1f5fa340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.308+0000 7fe337400640 1 -- 172.21.15.43:0/116041979 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d1f5f3ca0 con 0x562d1f5f3550 2024-03-28T22:43:16.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.308+0000 7fe337400640 1 --2- 172.21.15.43:0/116041979 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d1f5f3550 0x562d1f6085e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe328005940 tx=0x7fe328009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf9bf9c433e8507 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:16.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.308+0000 7fe327e00640 1 -- 172.21.15.43:0/116041979 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3280553b0 con 0x562d1f5f3550 2024-03-28T22:43:16.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.308+0000 7fe327e00640 1 -- 172.21.15.43:0/116041979 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe328055550 con 0x562d1f5f3550 2024-03-28T22:43:16.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.308+0000 7fe327e00640 1 -- 172.21.15.43:0/116041979 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe328004ec0 con 0x562d1f5f3550 2024-03-28T22:43:16.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.309+0000 7fe341006740 1 -- 172.21.15.43:0/116041979 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d1f5f3550 msgr2=0x562d1f6085e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:16.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.309+0000 7fe341006740 1 --2- 172.21.15.43:0/116041979 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d1f5f3550 0x562d1f6085e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe328005940 tx=0x7fe328009260 comp rx=0 tx=0).stop 2024-03-28T22:43:16.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.309+0000 7fe341006740 1 -- 172.21.15.43:0/116041979 shutdown_connections 2024-03-28T22:43:16.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.309+0000 7fe341006740 1 --2- 172.21.15.43:0/116041979 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562d1f5f3e70 0x562d1f5fa340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.309+0000 7fe341006740 1 --2- 172.21.15.43:0/116041979 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562d1f5fca50 0x562d1f614da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.309+0000 7fe341006740 1 --2- 172.21.15.43:0/116041979 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d1f5f3550 0x562d1f6085e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.309+0000 7fe341006740 1 -- 172.21.15.43:0/116041979 >> 172.21.15.43:0/116041979 conn(0x562d1f5edc80 msgr2=0x562d1f5ecdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:16.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.309+0000 7fe341006740 1 -- 172.21.15.43:0/116041979 shutdown_connections 2024-03-28T22:43:16.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.309+0000 7fe341006740 1 -- 172.21.15.43:0/116041979 wait complete. 2024-03-28T22:43:16.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.310+0000 7fe341006740 1 -- start start 2024-03-28T22:43:16.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.310+0000 7fe341006740 1 Processor -- start 2024-03-28T22:43:16.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.310+0000 7fe341006740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d1f5f3550 0x562d1f600890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:16.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.310+0000 7fe341006740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562d1f5f3e70 0x562d1f600dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:16.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.311+0000 7fe341006740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562d1f5fca50 0x562d1f6723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:16.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.311+0000 7fe341006740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe3381b62d0 con 0x562d1f5f3550 2024-03-28T22:43:16.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.311+0000 7fe341006740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe3381b5c00 con 0x562d1f5fca50 2024-03-28T22:43:16.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.311+0000 7fe341006740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe3381b2e40 con 0x562d1f5f3e70 2024-03-28T22:43:16.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.311+0000 7fe341006740 10 client.0 _ll_register_callbacks cb 0x7fe3200d4220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:16.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.311+0000 7fe337e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562d1f5fca50 0x562d1f6723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:16.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.311+0000 7fe337400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d1f5f3550 0x562d1f600890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:16.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.311+0000 7fe336a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562d1f5f3e70 0x562d1f600dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:16.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.311+0000 7fe337e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562d1f5fca50 0x562d1f6723d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:50662/0 (socket says 172.21.15.43:50662) 2024-03-28T22:43:16.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.311+0000 7fe337e00640 1 -- 172.21.15.43:0/755849325 learned_addr learned my addr 172.21.15.43:0/755849325 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:16.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.311+0000 7fe337400640 1 -- 172.21.15.43:0/755849325 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562d1f5f3e70 msgr2=0x562d1f600dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:16.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.311+0000 7fe337400640 1 --2- 172.21.15.43:0/755849325 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562d1f5f3e70 0x562d1f600dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.311+0000 7fe337400640 1 -- 172.21.15.43:0/755849325 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562d1f5fca50 msgr2=0x562d1f6723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:16.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.311+0000 7fe337400640 1 --2- 172.21.15.43:0/755849325 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562d1f5fca50 0x562d1f6723d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.311+0000 7fe337400640 1 -- 172.21.15.43:0/755849325 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d1f6016d0 con 0x562d1f5f3550 2024-03-28T22:43:16.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.311+0000 7fe337e00640 1 --2- 172.21.15.43:0/755849325 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562d1f5fca50 0x562d1f6723d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:16.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.312+0000 7fe337400640 1 --2- 172.21.15.43:0/755849325 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d1f5f3550 0x562d1f600890 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe328006010 tx=0x7fe3280058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6933d55fef574ea server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:16.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.312+0000 7fe325600640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:43:16.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.312+0000 7fe325600640 1 -- 172.21.15.43:0/755849325 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe328004980 con 0x562d1f5f3550 2024-03-28T22:43:16.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.312+0000 7fe325600640 1 -- 172.21.15.43:0/755849325 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fe328004b20 con 0x562d1f5f3550 2024-03-28T22:43:16.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.312+0000 7fe341006740 1 -- 172.21.15.43:0/755849325 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe328055550 con 0x562d1f5f3550 2024-03-28T22:43:16.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.312+0000 7fe325600640 1 -- 172.21.15.43:0/755849325 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe328004040 con 0x562d1f5f3550 2024-03-28T22:43:16.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.312+0000 7fe341006740 10 client.17703 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:16.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.312+0000 7fe341006740 1 -- 172.21.15.43:0/755849325 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562d1f673a40 con 0x562d1f5f3550 2024-03-28T22:43:16.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.312+0000 7fe325600640 1 -- 172.21.15.43:0/755849325 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe328055550 con 0x562d1f5f3550 2024-03-28T22:43:16.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.313+0000 7fe325600640 1 -- 172.21.15.43:0/755849325 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe3280035f0 con 0x562d1f5f3550 2024-03-28T22:43:16.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.313+0000 7fe341006740 10 client.17703 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:16.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.313+0000 7fe341006740 10 client.17703 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:43:16.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.313+0000 7fe341006740 20 client.17703 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:16.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.313+0000 7fe341006740 1 --2- 172.21.15.43:0/755849325 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x562d1f6740a0 0x562d1f676490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:16.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.313+0000 7fe341006740 4 client.17703 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:43:16.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.313+0000 7fe341006740 1 -- 172.21.15.43:0/755849325 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562d1f564590 con 0x562d1f6740a0 2024-03-28T22:43:16.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.313+0000 7fe336a00640 1 --2- 172.21.15.43:0/755849325 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x562d1f6740a0 0x562d1f676490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:16.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.313+0000 7fe336a00640 1 --2- 172.21.15.43:0/755849325 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x562d1f6740a0 0x562d1f676490 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4663bac5618f98a server_cookie=8e4628d9873178da in_seq=0 out_seq=0 2024-03-28T22:43:16.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.313+0000 7fe325600640 10 client.17703 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:16.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.314+0000 7fe325600640 1 -- 172.21.15.43:0/755849325 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562d1f564590 con 0x562d1f6740a0 2024-03-28T22:43:16.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.314+0000 7fe325600640 10 client.17703 handle_command_reply: tid=0 2024-03-28T22:43:16.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.314+0000 7fe341006740 1 client.17703 shutdown 2024-03-28T22:43:16.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.314+0000 7fe341006740 1 -- 172.21.15.43:0/755849325 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d1f5f3550 msgr2=0x562d1f600890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:16.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.314+0000 7fe341006740 1 --2- 172.21.15.43:0/755849325 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d1f5f3550 0x562d1f600890 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe328006010 tx=0x7fe3280058d0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.314+0000 7fe341006740 1 -- 172.21.15.43:0/755849325 shutdown_connections 2024-03-28T22:43:16.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.314+0000 7fe341006740 1 --2- 172.21.15.43:0/755849325 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x562d1f6740a0 0x562d1f676490 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.314+0000 7fe341006740 1 --2- 172.21.15.43:0/755849325 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562d1f5fca50 0x562d1f6723d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.314+0000 7fe341006740 1 --2- 172.21.15.43:0/755849325 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562d1f5f3e70 0x562d1f600dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.314+0000 7fe341006740 1 --2- 172.21.15.43:0/755849325 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562d1f5f3550 0x562d1f600890 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.314+0000 7fe325600640 0 client.17703 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:16.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.314+0000 7fe341006740 1 -- 172.21.15.43:0/755849325 >> 172.21.15.43:0/755849325 conn(0x562d1f5edc80 msgr2=0x562d1f5eed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:16.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.314+0000 7fe341006740 1 -- 172.21.15.43:0/755849325 shutdown_connections 2024-03-28T22:43:16.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.314+0000 7fe341006740 1 -- 172.21.15.43:0/755849325 wait complete. 2024-03-28T22:43:16.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.315+0000 7fe341006740 20 client.17703 trim_cache size 0 max 16384 2024-03-28T22:43:16.318 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:43:16.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.315+0000 7fe33ce00640 1 -- 172.21.15.43:0/555751810 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe31c0a1ef0 msgr2=0x7fe31c0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:16.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.315+0000 7fe33ce00640 1 --2- 172.21.15.43:0/555751810 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe31c0a1ef0 0x7fe31c0a43b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe32802cfd0 tx=0x7fe32802e040 comp rx=0 tx=0).stop 2024-03-28T22:43:16.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.315+0000 7fe33ce00640 1 -- 172.21.15.43:0/555751810 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe33810cf50 msgr2=0x7fe33819e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:16.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.315+0000 7fe33ce00640 1 --2- 172.21.15.43:0/555751810 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe33810cf50 0x7fe33819e9e0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fe338108e70 tx=0x7fe32000ee50 comp rx=0 tx=0).stop 2024-03-28T22:43:16.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.315+0000 7fe33ce00640 1 -- 172.21.15.43:0/555751810 shutdown_connections 2024-03-28T22:43:16.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.315+0000 7fe33ce00640 1 --2- 172.21.15.43:0/555751810 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe31c0a1ef0 0x7fe31c0a43b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.315+0000 7fe33ce00640 1 --2- 172.21.15.43:0/555751810 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe33810cf50 0x7fe33819e9e0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.315+0000 7fe33ce00640 1 --2- 172.21.15.43:0/555751810 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe3381a2d10 0x7fe3381a31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.315+0000 7fe33ce00640 1 --2- 172.21.15.43:0/555751810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe338107f00 0x7fe33819e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:16.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.315+0000 7fe33ce00640 1 -- 172.21.15.43:0/555751810 >> 172.21.15.43:0/555751810 conn(0x7fe338077570 msgr2=0x7fe338077b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:16.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.316+0000 7fe33ce00640 1 -- 172.21.15.43:0/555751810 shutdown_connections 2024-03-28T22:43:16.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:16.316+0000 7fe33ce00640 1 -- 172.21.15.43:0/555751810 wait complete. 2024-03-28T22:43:16.372 INFO:tasks.workunit.client.0.smithi043.stdout: 1 415043 7.42 MB/sec execute 167 sec latency 63.714 ms 2024-03-28T22:43:17.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:16 smithi082 ceph-mon[36471]: pgmap v187: 129 pgs: 129 active+clean; 57 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 1.7 MiB/s wr, 102 op/s 2024-03-28T22:43:17.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:16 smithi043 ceph-mon[31823]: pgmap v187: 129 pgs: 129 active+clean; 57 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 1.7 MiB/s wr, 102 op/s 2024-03-28T22:43:17.334 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:43:17.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:43:17.335 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:43:17.373 INFO:tasks.workunit.client.0.smithi043.stdout: 1 415943 7.39 MB/sec execute 168 sec latency 18.352 ms 2024-03-28T22:43:17.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:16 smithi172 ceph-mon[36566]: pgmap v187: 129 pgs: 129 active+clean; 57 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 1.7 MiB/s wr, 102 op/s 2024-03-28T22:43:17.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.579+0000 7f4dc4600640 1 -- 172.21.15.43:0/4246726743 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4dbc075c80 msgr2=0x7f4dbc076060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:17.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.579+0000 7f4dc4600640 1 --2- 172.21.15.43:0/4246726743 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4dbc075c80 0x7f4dbc076060 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f4db0009930 tx=0x7f4db002bc30 comp rx=0 tx=0).stop 2024-03-28T22:43:17.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.579+0000 7f4dc4600640 1 -- 172.21.15.43:0/4246726743 shutdown_connections 2024-03-28T22:43:17.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.579+0000 7f4dc4600640 1 --2- 172.21.15.43:0/4246726743 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4dbc112590 0x7f4dbc114980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.579+0000 7f4dc4600640 1 --2- 172.21.15.43:0/4246726743 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dbc0765a0 0x7f4dbc112050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.579+0000 7f4dc4600640 1 --2- 172.21.15.43:0/4246726743 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4dbc075c80 0x7f4dbc076060 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.579+0000 7f4dc4600640 1 -- 172.21.15.43:0/4246726743 >> 172.21.15.43:0/4246726743 conn(0x7f4dbc0ff3a0 msgr2=0x7f4dbc1017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:17.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.580+0000 7f4dc4600640 1 -- 172.21.15.43:0/4246726743 shutdown_connections 2024-03-28T22:43:17.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.580+0000 7f4dc4600640 1 -- 172.21.15.43:0/4246726743 wait complete. 2024-03-28T22:43:17.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.580+0000 7f4dc4600640 1 Processor -- start 2024-03-28T22:43:17.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.581+0000 7f4dc4600640 1 -- start start 2024-03-28T22:43:17.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.581+0000 7f4dc4600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4dbc075c80 0x7f4dbc19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.581+0000 7f4dc4600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4dbc0765a0 0x7f4dbc19ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.581+0000 7f4dc4600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dbc112590 0x7f4dbc1a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.581+0000 7f4dc4600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4dbc118150 con 0x7f4dbc075c80 2024-03-28T22:43:17.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.581+0000 7f4dc4600640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4dbc117fd0 con 0x7f4dbc0765a0 2024-03-28T22:43:17.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.581+0000 7f4dc4600640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4dbc1182d0 con 0x7f4dbc112590 2024-03-28T22:43:17.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.581+0000 7f4dc1c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4dbc075c80 0x7f4dbc19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.581+0000 7f4dc1200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4dbc0765a0 0x7f4dbc19ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.581+0000 7f4dc1200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4dbc0765a0 0x7f4dbc19ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:50692/0 (socket says 172.21.15.43:50692) 2024-03-28T22:43:17.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.581+0000 7f4dc1c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4dbc075c80 0x7f4dbc19e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46298/0 (socket says 172.21.15.43:46298) 2024-03-28T22:43:17.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.581+0000 7f4dc1200640 1 -- 172.21.15.43:0/1115150244 learned_addr learned my addr 172.21.15.43:0/1115150244 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:17.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.582+0000 7f4dc2600640 1 --2- 172.21.15.43:0/1115150244 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dbc112590 0x7f4dbc1a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.582+0000 7f4dc1200640 1 -- 172.21.15.43:0/1115150244 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dbc112590 msgr2=0x7f4dbc1a2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:17.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.582+0000 7f4dc1200640 1 --2- 172.21.15.43:0/1115150244 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dbc112590 0x7f4dbc1a2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.582+0000 7f4dc1200640 1 -- 172.21.15.43:0/1115150244 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4dbc075c80 msgr2=0x7f4dbc19e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:17.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.582+0000 7f4dc1200640 1 --2- 172.21.15.43:0/1115150244 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4dbc075c80 0x7f4dbc19e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.582+0000 7f4dc1200640 1 -- 172.21.15.43:0/1115150244 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dbc1a3680 con 0x7f4dbc0765a0 2024-03-28T22:43:17.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.582+0000 7f4dc1c00640 1 --2- 172.21.15.43:0/1115150244 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4dbc075c80 0x7f4dbc19e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:17.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.582+0000 7f4dc1200640 1 --2- 172.21.15.43:0/1115150244 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4dbc0765a0 0x7f4dbc19ebd0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f4da800e9f0 tx=0x7f4da800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:17.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.582+0000 7f4dba000640 1 -- 172.21.15.43:0/1115150244 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da800cd80 con 0x7f4dbc0765a0 2024-03-28T22:43:17.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.582+0000 7f4dba000640 1 -- 172.21.15.43:0/1115150244 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4da8005500 con 0x7f4dbc0765a0 2024-03-28T22:43:17.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.582+0000 7f4dba000640 1 -- 172.21.15.43:0/1115150244 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da8010690 con 0x7f4dbc0765a0 2024-03-28T22:43:17.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.582+0000 7f4dc4600640 1 -- 172.21.15.43:0/1115150244 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dbc1a3970 con 0x7f4dbc0765a0 2024-03-28T22:43:17.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.583+0000 7f4dc4600640 1 -- 172.21.15.43:0/1115150244 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dbc1ab200 con 0x7f4dbc0765a0 2024-03-28T22:43:17.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.588+0000 7f4dc7273740 1 Processor -- start 2024-03-28T22:43:17.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.588+0000 7f4dc7273740 1 -- start start 2024-03-28T22:43:17.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.588+0000 7f4dc7273740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b201a04f70 0x55b201a05350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.588+0000 7f4dc7273740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b201a05890 0x55b201a0df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.588+0000 7f4dc7273740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b201a0e470 0x55b201a10860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.588+0000 7f4dc7273740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b201968c40 con 0x55b201a04f70 2024-03-28T22:43:17.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.588+0000 7f4dc7273740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55b201911c40 con 0x55b201a0e470 2024-03-28T22:43:17.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.588+0000 7f4dc7273740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55b201964ce0 con 0x55b201a05890 2024-03-28T22:43:17.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.589+0000 7f4dc2600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b201a04f70 0x55b201a05350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.589+0000 7f4dc1c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b201a05890 0x55b201a0df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.589+0000 7f4dc2600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b201a04f70 0x55b201a05350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46302/0 (socket says 172.21.15.43:46302) 2024-03-28T22:43:17.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.589+0000 7f4dc1c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b201a05890 0x55b201a0df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:49664/0 (socket says 172.21.15.43:49664) 2024-03-28T22:43:17.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.589+0000 7f4dba000640 1 -- 172.21.15.43:0/1115150244 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f4da80046c0 con 0x7f4dbc0765a0 2024-03-28T22:43:17.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.589+0000 7f4dc2600640 1 -- 172.21.15.43:0/2855256653 learned_addr learned my addr 172.21.15.43:0/2855256653 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:17.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.589+0000 7f4dc1200640 1 --2- 172.21.15.43:0/2855256653 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b201a0e470 0x55b201a10860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.589+0000 7f4dc1c00640 1 -- 172.21.15.43:0/2855256653 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b201a0e470 msgr2=0x55b201a10860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:17.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.589+0000 7f4dc1c00640 1 --2- 172.21.15.43:0/2855256653 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b201a0e470 0x55b201a10860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.589+0000 7f4dc1c00640 1 -- 172.21.15.43:0/2855256653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b201a04f70 msgr2=0x55b201a05350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:17.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.589+0000 7f4dc1c00640 1 --2- 172.21.15.43:0/2855256653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b201a04f70 0x55b201a05350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.589+0000 7f4dc1c00640 1 -- 172.21.15.43:0/2855256653 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2017457e0 con 0x55b201a05890 2024-03-28T22:43:17.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.589+0000 7f4dc2600640 1 --2- 172.21.15.43:0/2855256653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b201a04f70 0x55b201a05350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:17.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.589+0000 7f4dc1200640 1 --2- 172.21.15.43:0/2855256653 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b201a0e470 0x55b201a10860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:43:17.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.590+0000 7f4dc1c00640 1 --2- 172.21.15.43:0/2855256653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b201a05890 0x55b201a0df30 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4db0036040 tx=0x7f4db0009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6e94443fd5ff497 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:17.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.590+0000 7f4daea00640 1 -- 172.21.15.43:0/2855256653 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db0045070 con 0x55b201a05890 2024-03-28T22:43:17.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.590+0000 7f4daea00640 1 -- 172.21.15.43:0/2855256653 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4db002c650 con 0x55b201a05890 2024-03-28T22:43:17.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.590+0000 7f4daea00640 1 -- 172.21.15.43:0/2855256653 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db00050e0 con 0x55b201a05890 2024-03-28T22:43:17.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.591+0000 7f4dc7273740 1 -- 172.21.15.43:0/2855256653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b201a05890 msgr2=0x55b201a0df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:17.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.591+0000 7f4dc7273740 1 --2- 172.21.15.43:0/2855256653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b201a05890 0x55b201a0df30 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4db0036040 tx=0x7f4db0009560 comp rx=0 tx=0).stop 2024-03-28T22:43:17.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.591+0000 7f4dba000640 1 --2- 172.21.15.43:0/1115150244 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4da40a1ef0 0x7f4da40a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.591+0000 7f4dba000640 1 -- 172.21.15.43:0/1115150244 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f4da80ce9b0 con 0x7f4dbc0765a0 2024-03-28T22:43:17.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.591+0000 7f4dc7273740 1 -- 172.21.15.43:0/2855256653 shutdown_connections 2024-03-28T22:43:17.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.591+0000 7f4dc7273740 1 --2- 172.21.15.43:0/2855256653 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b201a0e470 0x55b201a10860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.591+0000 7f4dc7273740 1 --2- 172.21.15.43:0/2855256653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b201a05890 0x55b201a0df30 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.591+0000 7f4dc7273740 1 --2- 172.21.15.43:0/2855256653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b201a04f70 0x55b201a05350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.591+0000 7f4dc7273740 1 -- 172.21.15.43:0/2855256653 >> 172.21.15.43:0/2855256653 conn(0x55b2019ff6a0 msgr2=0x55b2019ffa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:17.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.591+0000 7f4dc2600640 1 --2- 172.21.15.43:0/1115150244 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4da40a1ef0 0x7f4da40a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.591+0000 7f4dc7273740 1 -- 172.21.15.43:0/2855256653 shutdown_connections 2024-03-28T22:43:17.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.591+0000 7f4dc7273740 1 -- 172.21.15.43:0/2855256653 wait complete. 2024-03-28T22:43:17.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.592+0000 7f4dc7273740 1 -- start start 2024-03-28T22:43:17.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.592+0000 7f4dc2600640 1 --2- 172.21.15.43:0/1115150244 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4da40a1ef0 0x7f4da40a43b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4db4005fd0 tx=0x7f4db4009500 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:17.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.592+0000 7f4dc7273740 1 Processor -- start 2024-03-28T22:43:17.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.592+0000 7f4dc7273740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b201a04f70 0x55b201a0b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.592+0000 7f4dc7273740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b201a05890 0x55b201a0ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.592+0000 7f4dc7273740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b201a0e470 0x55b201a0bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.592+0000 7f4dc7273740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b201a11200 con 0x55b201a04f70 2024-03-28T22:43:17.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.592+0000 7f4dc7273740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55b201907250 con 0x55b201a0e470 2024-03-28T22:43:17.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.592+0000 7f4dc7273740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55b201a11380 con 0x55b201a05890 2024-03-28T22:43:17.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.593+0000 7f4dc7273740 10 client.0 _ll_register_callbacks cb 0x55b201928e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:17.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.593+0000 7f4dc1200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b201a05890 0x55b201a0ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.593+0000 7f4dc2600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b201a0e470 0x55b201a0bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.593+0000 7f4dc1c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b201a04f70 0x55b201a0b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.593+0000 7f4dc1200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b201a05890 0x55b201a0ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:49678/0 (socket says 172.21.15.43:49678) 2024-03-28T22:43:17.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.593+0000 7f4dc2600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b201a0e470 0x55b201a0bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:50704/0 (socket says 172.21.15.43:50704) 2024-03-28T22:43:17.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.593+0000 7f4dc1200640 1 -- 172.21.15.43:0/3663489539 learned_addr learned my addr 172.21.15.43:0/3663489539 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:17.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.593+0000 7f4dc1200640 1 -- 172.21.15.43:0/3663489539 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b201a0e470 msgr2=0x55b201a0bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:17.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.593+0000 7f4dc1200640 1 --2- 172.21.15.43:0/3663489539 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b201a0e470 0x55b201a0bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.593+0000 7f4dc1200640 1 -- 172.21.15.43:0/3663489539 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b201a04f70 msgr2=0x55b201a0b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:17.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.593+0000 7f4dc1200640 1 --2- 172.21.15.43:0/3663489539 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b201a04f70 0x55b201a0b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.593+0000 7f4dc1200640 1 -- 172.21.15.43:0/3663489539 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2017a5f70 con 0x55b201a05890 2024-03-28T22:43:17.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.593+0000 7f4dc2600640 1 --2- 172.21.15.43:0/3663489539 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b201a0e470 0x55b201a0bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:17.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.593+0000 7f4dc1c00640 1 --2- 172.21.15.43:0/3663489539 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b201a04f70 0x55b201a0b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:17.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.594+0000 7f4dc1200640 1 --2- 172.21.15.43:0/3663489539 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b201a05890 0x55b201a0ba10 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f4da809af30 tx=0x7f4da809aba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9de6a5aeabb1155 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:17.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.594+0000 7f4da3400640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:43:17.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.594+0000 7f4da3400640 1 -- 172.21.15.43:0/3663489539 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da80cc190 con 0x55b201a05890 2024-03-28T22:43:17.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.594+0000 7f4da3400640 1 -- 172.21.15.43:0/3663489539 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f4da80a34b0 con 0x55b201a05890 2024-03-28T22:43:17.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.594+0000 7f4dc7273740 1 -- 172.21.15.43:0/3663489539 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4db002c650 con 0x55b201a05890 2024-03-28T22:43:17.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.594+0000 7f4da3400640 1 -- 172.21.15.43:0/3663489539 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4da809f0f0 con 0x55b201a05890 2024-03-28T22:43:17.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.594+0000 7f4dc7273740 10 client.27892 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:17.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.594+0000 7f4dc7273740 1 -- 172.21.15.43:0/3663489539 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b20163d170 con 0x55b201a05890 2024-03-28T22:43:17.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.594+0000 7f4da3400640 1 -- 172.21.15.43:0/3663489539 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4db002c650 con 0x55b201a05890 2024-03-28T22:43:17.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.594+0000 7f4da3400640 1 -- 172.21.15.43:0/3663489539 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4da8096d80 con 0x55b201a05890 2024-03-28T22:43:17.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.595+0000 7f4dc7273740 10 client.27892 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:17.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.595+0000 7f4dc7273740 10 client.27892 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:43:17.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.595+0000 7f4dc7273740 20 client.27892 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:17.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.595+0000 7f4dc7273740 1 --2- 172.21.15.43:0/3663489539 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b201a1e730 0x55b201a20b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.595+0000 7f4dc7273740 4 client.27892 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:43:17.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.595+0000 7f4dc7273740 1 -- 172.21.15.43:0/3663489539 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b201976180 con 0x55b201a1e730 2024-03-28T22:43:17.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.595+0000 7f4dc1c00640 1 --2- 172.21.15.43:0/3663489539 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b201a1e730 0x55b201a20b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.595+0000 7f4dc1c00640 1 --2- 172.21.15.43:0/3663489539 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b201a1e730 0x55b201a20b40 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f280c3bdfd68e8b server_cookie=be68e663b83a9569 in_seq=0 out_seq=0 2024-03-28T22:43:17.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.595+0000 7f4da3400640 10 client.27892 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:17.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.596+0000 7f4da3400640 1 -- 172.21.15.43:0/3663489539 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b201976180 con 0x55b201a1e730 2024-03-28T22:43:17.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.596+0000 7f4da3400640 10 client.27892 handle_command_reply: tid=0 2024-03-28T22:43:17.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.597+0000 7f4dc7273740 1 client.27892 shutdown 2024-03-28T22:43:17.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.597+0000 7f4dc7273740 1 -- 172.21.15.43:0/3663489539 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b201a05890 msgr2=0x55b201a0ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:17.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.597+0000 7f4dc7273740 1 --2- 172.21.15.43:0/3663489539 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b201a05890 0x55b201a0ba10 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f4da809af30 tx=0x7f4da809aba0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.597+0000 7f4dc7273740 1 -- 172.21.15.43:0/3663489539 shutdown_connections 2024-03-28T22:43:17.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.597+0000 7f4dc7273740 1 --2- 172.21.15.43:0/3663489539 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b201a1e730 0x55b201a20b40 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.597+0000 7f4dc7273740 1 --2- 172.21.15.43:0/3663489539 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b201a0e470 0x55b201a0bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.597+0000 7f4dc7273740 1 --2- 172.21.15.43:0/3663489539 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b201a05890 0x55b201a0ba10 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.597+0000 7f4da3400640 0 client.27892 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:17.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.597+0000 7f4dc7273740 1 --2- 172.21.15.43:0/3663489539 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b201a04f70 0x55b201a0b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.597+0000 7f4dc7273740 1 -- 172.21.15.43:0/3663489539 >> 172.21.15.43:0/3663489539 conn(0x55b2019ff6a0 msgr2=0x55b201a013f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:17.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.597+0000 7f4dc7273740 1 -- 172.21.15.43:0/3663489539 shutdown_connections 2024-03-28T22:43:17.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.597+0000 7f4dc7273740 1 -- 172.21.15.43:0/3663489539 wait complete. 2024-03-28T22:43:17.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.597+0000 7f4dc7273740 20 client.27892 trim_cache size 0 max 16384 2024-03-28T22:43:17.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.618+0000 7f4dc7273740 1 Processor -- start 2024-03-28T22:43:17.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.618+0000 7f4dc7273740 1 -- start start 2024-03-28T22:43:17.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.618+0000 7f4dc7273740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2019ff6a0 0x55b201a19fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.619+0000 7f4dc7273740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b201a04f70 0x55b201a26410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.619+0000 7f4dc7273740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b201a05890 0x55b201a165c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.619+0000 7f4dc7273740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4dbc1b51a0 con 0x55b2019ff6a0 2024-03-28T22:43:17.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.619+0000 7f4dc7273740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4dbc1b6ce0 con 0x55b201a04f70 2024-03-28T22:43:17.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.619+0000 7f4dc7273740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4dbc1b5870 con 0x55b201a05890 2024-03-28T22:43:17.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.619+0000 7f4dc1c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2019ff6a0 0x55b201a19fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.619+0000 7f4dc1c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2019ff6a0 0x55b201a19fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46316/0 (socket says 172.21.15.43:46316) 2024-03-28T22:43:17.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.619+0000 7f4dc1c00640 1 -- 172.21.15.43:0/2544749849 learned_addr learned my addr 172.21.15.43:0/2544749849 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:17.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.619+0000 7f4dc2600640 1 --2- 172.21.15.43:0/2544749849 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b201a05890 0x55b201a165c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.619+0000 7f4dc1200640 1 --2- 172.21.15.43:0/2544749849 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b201a04f70 0x55b201a26410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.619+0000 7f4dc1c00640 1 -- 172.21.15.43:0/2544749849 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b201a05890 msgr2=0x55b201a165c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:17.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.619+0000 7f4dc1c00640 1 --2- 172.21.15.43:0/2544749849 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b201a05890 0x55b201a165c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.619+0000 7f4dc1c00640 1 -- 172.21.15.43:0/2544749849 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b201a04f70 msgr2=0x55b201a26410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:17.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.619+0000 7f4dc1c00640 1 --2- 172.21.15.43:0/2544749849 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b201a04f70 0x55b201a26410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.619+0000 7f4dc1c00640 1 -- 172.21.15.43:0/2544749849 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b201a1a4e0 con 0x55b2019ff6a0 2024-03-28T22:43:17.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.619+0000 7f4dc1200640 1 --2- 172.21.15.43:0/2544749849 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b201a04f70 0x55b201a26410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:17.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.620+0000 7f4dc1c00640 1 --2- 172.21.15.43:0/2544749849 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2019ff6a0 0x55b201a19fa0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4db002cb00 tx=0x7f4db002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c71a6b8e12b3b4e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:17.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.620+0000 7f4dae000640 1 -- 172.21.15.43:0/2544749849 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db0045040 con 0x55b2019ff6a0 2024-03-28T22:43:17.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.620+0000 7f4dae000640 1 -- 172.21.15.43:0/2544749849 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4db00053c0 con 0x55b2019ff6a0 2024-03-28T22:43:17.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.620+0000 7f4dae000640 1 -- 172.21.15.43:0/2544749849 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db004a9c0 con 0x55b2019ff6a0 2024-03-28T22:43:17.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.621+0000 7f4dc7273740 1 -- 172.21.15.43:0/2544749849 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2019ff6a0 msgr2=0x55b201a19fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:17.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.621+0000 7f4dc7273740 1 --2- 172.21.15.43:0/2544749849 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2019ff6a0 0x55b201a19fa0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4db002cb00 tx=0x7f4db002bc30 comp rx=0 tx=0).stop 2024-03-28T22:43:17.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.621+0000 7f4dc7273740 1 -- 172.21.15.43:0/2544749849 shutdown_connections 2024-03-28T22:43:17.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.621+0000 7f4dc7273740 1 --2- 172.21.15.43:0/2544749849 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b201a04f70 0x55b201a26410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.621+0000 7f4dc7273740 1 --2- 172.21.15.43:0/2544749849 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b201a05890 0x55b201a165c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.621+0000 7f4dc7273740 1 --2- 172.21.15.43:0/2544749849 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2019ff6a0 0x55b201a19fa0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.621+0000 7f4dc7273740 1 -- 172.21.15.43:0/2544749849 >> 172.21.15.43:0/2544749849 conn(0x55b201a0a130 msgr2=0x55b201a0b490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:17.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.621+0000 7f4dc7273740 1 -- 172.21.15.43:0/2544749849 shutdown_connections 2024-03-28T22:43:17.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.621+0000 7f4dc7273740 1 -- 172.21.15.43:0/2544749849 wait complete. 2024-03-28T22:43:17.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.622+0000 7f4dc7273740 1 -- start start 2024-03-28T22:43:17.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.622+0000 7f4dc7273740 1 Processor -- start 2024-03-28T22:43:17.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.623+0000 7f4dc7273740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2019ff6a0 0x55b201a165d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.623+0000 7f4dc7273740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b201a04f70 0x55b201a135a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.623+0000 7f4dc7273740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b201a05890 0x55b201a13ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.623+0000 7f4dc7273740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4dbc1b6610 con 0x55b2019ff6a0 2024-03-28T22:43:17.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.623+0000 7f4dc7273740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4dbc1b5f40 con 0x55b201a04f70 2024-03-28T22:43:17.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.623+0000 7f4dc7273740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4dbc1b3180 con 0x55b201a05890 2024-03-28T22:43:17.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.623+0000 7f4dc7273740 10 client.0 _ll_register_callbacks cb 0x7f4da800cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:17.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.623+0000 7f4dc1200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b201a04f70 0x55b201a135a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.623+0000 7f4dc1c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2019ff6a0 0x55b201a165d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.623+0000 7f4dc1200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b201a04f70 0x55b201a135a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:50734/0 (socket says 172.21.15.43:50734) 2024-03-28T22:43:17.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.623+0000 7f4dc1200640 1 -- 172.21.15.43:0/103501216 learned_addr learned my addr 172.21.15.43:0/103501216 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:17.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.623+0000 7f4dc2600640 1 --2- 172.21.15.43:0/103501216 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b201a05890 0x55b201a13ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.623+0000 7f4dc1200640 1 -- 172.21.15.43:0/103501216 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b201a05890 msgr2=0x55b201a13ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:17.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.623+0000 7f4dc1200640 1 --2- 172.21.15.43:0/103501216 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b201a05890 0x55b201a13ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.623+0000 7f4dc1200640 1 -- 172.21.15.43:0/103501216 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2019ff6a0 msgr2=0x55b201a165d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:17.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.623+0000 7f4dc1200640 1 --2- 172.21.15.43:0/103501216 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2019ff6a0 0x55b201a165d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.623+0000 7f4dc1200640 1 -- 172.21.15.43:0/103501216 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b201a14020 con 0x55b201a04f70 2024-03-28T22:43:17.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.623+0000 7f4dc1c00640 1 --2- 172.21.15.43:0/103501216 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2019ff6a0 0x55b201a165d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:17.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.624+0000 7f4dc1200640 1 --2- 172.21.15.43:0/103501216 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b201a04f70 0x55b201a135a0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f4da80ced70 tx=0x7f4da8099e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1b3e306cb2fad07 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:17.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.624+0000 7f4da3400640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:43:17.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.624+0000 7f4da3400640 1 -- 172.21.15.43:0/103501216 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da80cc190 con 0x55b201a04f70 2024-03-28T22:43:17.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.624+0000 7f4da3400640 1 -- 172.21.15.43:0/103501216 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f4da809a3f0 con 0x55b201a04f70 2024-03-28T22:43:17.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.624+0000 7f4dc7273740 1 -- 172.21.15.43:0/103501216 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4db00053c0 con 0x55b201a04f70 2024-03-28T22:43:17.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.624+0000 7f4da3400640 1 -- 172.21.15.43:0/103501216 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4da809ad70 con 0x55b201a04f70 2024-03-28T22:43:17.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.624+0000 7f4dc7273740 10 client.27956 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:17.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.624+0000 7f4dc7273740 1 -- 172.21.15.43:0/103501216 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b201a85790 con 0x55b201a04f70 2024-03-28T22:43:17.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.624+0000 7f4da3400640 1 -- 172.21.15.43:0/103501216 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4db00053c0 con 0x55b201a04f70 2024-03-28T22:43:17.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.625+0000 7f4da3400640 1 -- 172.21.15.43:0/103501216 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4da80a01a0 con 0x55b201a04f70 2024-03-28T22:43:17.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.625+0000 7f4dc7273740 10 client.27956 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:17.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.625+0000 7f4dc7273740 10 client.27956 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:43:17.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.625+0000 7f4dc7273740 20 client.27956 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:17.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.625+0000 7f4dc7273740 1 --2- 172.21.15.43:0/103501216 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b201a0e470 0x55b201a87f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.625+0000 7f4dc7273740 4 client.27956 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:43:17.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.625+0000 7f4dc7273740 1 -- 172.21.15.43:0/103501216 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b2019704d0 con 0x55b201a0e470 2024-03-28T22:43:17.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.625+0000 7f4dc1c00640 1 --2- 172.21.15.43:0/103501216 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b201a0e470 0x55b201a87f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.625+0000 7f4dc1c00640 1 --2- 172.21.15.43:0/103501216 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b201a0e470 0x55b201a87f50 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9579a4ee74cd9bc server_cookie=114d47230bbdcc84 in_seq=0 out_seq=0 2024-03-28T22:43:17.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.625+0000 7f4da3400640 10 client.27956 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:17.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.627+0000 7f4da3400640 1 -- 172.21.15.43:0/103501216 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b2019704d0 con 0x55b201a0e470 2024-03-28T22:43:17.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.627+0000 7f4da3400640 10 client.27956 handle_command_reply: tid=0 2024-03-28T22:43:17.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.627+0000 7f4dc7273740 1 client.27956 shutdown 2024-03-28T22:43:17.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.627+0000 7f4dc7273740 1 -- 172.21.15.43:0/103501216 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b201a04f70 msgr2=0x55b201a135a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:17.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.627+0000 7f4dc7273740 1 --2- 172.21.15.43:0/103501216 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b201a04f70 0x55b201a135a0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f4da80ced70 tx=0x7f4da8099e60 comp rx=0 tx=0).stop 2024-03-28T22:43:17.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.627+0000 7f4dc7273740 1 -- 172.21.15.43:0/103501216 shutdown_connections 2024-03-28T22:43:17.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.627+0000 7f4dc7273740 1 --2- 172.21.15.43:0/103501216 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b201a0e470 0x55b201a87f50 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.627+0000 7f4dc7273740 1 --2- 172.21.15.43:0/103501216 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b201a04f70 0x55b201a135a0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.627+0000 7f4dc7273740 1 --2- 172.21.15.43:0/103501216 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b201a05890 0x55b201a13ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.627+0000 7f4dc7273740 1 --2- 172.21.15.43:0/103501216 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2019ff6a0 0x55b201a165d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.627+0000 7f4dc7273740 1 -- 172.21.15.43:0/103501216 >> 172.21.15.43:0/103501216 conn(0x55b201a0a130 msgr2=0x55b201a0bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:17.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.627+0000 7f4da3400640 0 client.27956 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:17.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.627+0000 7f4dc7273740 1 -- 172.21.15.43:0/103501216 shutdown_connections 2024-03-28T22:43:17.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.627+0000 7f4dc7273740 1 -- 172.21.15.43:0/103501216 wait complete. 2024-03-28T22:43:17.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.627+0000 7f4dc7273740 20 client.27956 trim_cache size 0 max 16384 2024-03-28T22:43:17.629 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:43:17.629 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-03-28T22:43:17.629 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "14c06235-f0c6-48e9-a3a9-923d2337a193", 2024-03-28T22:43:17.629 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-03-28T22:43:17.629 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:43:17.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.628+0000 7f4dc4600640 1 -- 172.21.15.43:0/1115150244 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4da40a1ef0 msgr2=0x7f4da40a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:17.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.628+0000 7f4dc4600640 1 --2- 172.21.15.43:0/1115150244 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4da40a1ef0 0x7f4da40a43b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4db4005fd0 tx=0x7f4db4009500 comp rx=0 tx=0).stop 2024-03-28T22:43:17.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.628+0000 7f4dc4600640 1 -- 172.21.15.43:0/1115150244 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4dbc0765a0 msgr2=0x7f4dbc19ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:17.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.628+0000 7f4dc4600640 1 --2- 172.21.15.43:0/1115150244 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4dbc0765a0 0x7f4dbc19ebd0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f4da800e9f0 tx=0x7f4da800edf0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.628+0000 7f4dc4600640 1 -- 172.21.15.43:0/1115150244 shutdown_connections 2024-03-28T22:43:17.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.628+0000 7f4dc4600640 1 --2- 172.21.15.43:0/1115150244 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4da40a1ef0 0x7f4da40a43b0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.628+0000 7f4dc4600640 1 --2- 172.21.15.43:0/1115150244 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4dbc0765a0 0x7f4dbc19ebd0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.628+0000 7f4dc4600640 1 --2- 172.21.15.43:0/1115150244 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dbc112590 0x7f4dbc1a2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.628+0000 7f4dc4600640 1 --2- 172.21.15.43:0/1115150244 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4dbc075c80 0x7f4dbc19e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.628+0000 7f4dc4600640 1 -- 172.21.15.43:0/1115150244 >> 172.21.15.43:0/1115150244 conn(0x7f4dbc0ff3a0 msgr2=0x7f4dbc113200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:17.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.628+0000 7f4dc4600640 1 -- 172.21.15.43:0/1115150244 shutdown_connections 2024-03-28T22:43:17.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.628+0000 7f4dc4600640 1 -- 172.21.15.43:0/1115150244 wait complete. 2024-03-28T22:43:17.644 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:43:17.644 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:43:17.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.919+0000 7f72f6600640 1 -- 172.21.15.43:0/1165764821 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72f0073b20 msgr2=0x7f72f0073f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:17.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.919+0000 7f72f6600640 1 --2- 172.21.15.43:0/1165764821 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72f0073b20 0x7f72f0073f00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f72e4009900 tx=0x7f72e402bcd0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.919+0000 7f72f6600640 1 -- 172.21.15.43:0/1165764821 shutdown_connections 2024-03-28T22:43:17.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.919+0000 7f72f6600640 1 --2- 172.21.15.43:0/1165764821 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f72f010de60 0x7f72f0110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.919+0000 7f72f6600640 1 --2- 172.21.15.43:0/1165764821 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72f0074440 0x7f72f010d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.919+0000 7f72f6600640 1 --2- 172.21.15.43:0/1165764821 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72f0073b20 0x7f72f0073f00 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.919+0000 7f72f6600640 1 -- 172.21.15.43:0/1165764821 >> 172.21.15.43:0/1165764821 conn(0x7f72f00fd0f0 msgr2=0x7f72f00ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:17.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.919+0000 7f72f6600640 1 -- 172.21.15.43:0/1165764821 shutdown_connections 2024-03-28T22:43:17.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.919+0000 7f72f6600640 1 -- 172.21.15.43:0/1165764821 wait complete. 2024-03-28T22:43:17.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.920+0000 7f72f6600640 1 Processor -- start 2024-03-28T22:43:17.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.920+0000 7f72f6600640 1 -- start start 2024-03-28T22:43:17.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.921+0000 7f72f6600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72f0073b20 0x7f72f010ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.921+0000 7f72f6600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f72f0074440 0x7f72f010d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.921+0000 7f72f6600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72f010de60 0x7f72f01a2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.921+0000 7f72f6600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f72f0103070 con 0x7f72f0073b20 2024-03-28T22:43:17.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.921+0000 7f72f6600640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f72f0102ef0 con 0x7f72f0074440 2024-03-28T22:43:17.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.921+0000 7f72f6600640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f72f01031f0 con 0x7f72f010de60 2024-03-28T22:43:17.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.921+0000 7f72f5200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72f0073b20 0x7f72f010ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.921+0000 7f72efe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f72f0074440 0x7f72f010d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.921+0000 7f72efe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f72f0074440 0x7f72f010d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:50764/0 (socket says 172.21.15.43:50764) 2024-03-28T22:43:17.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.921+0000 7f72efe00640 1 -- 172.21.15.43:0/4098524676 learned_addr learned my addr 172.21.15.43:0/4098524676 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:17.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.921+0000 7f72f5c00640 1 --2- 172.21.15.43:0/4098524676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72f010de60 0x7f72f01a2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.921+0000 7f72efe00640 1 -- 172.21.15.43:0/4098524676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72f010de60 msgr2=0x7f72f01a2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:17.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.921+0000 7f72efe00640 1 --2- 172.21.15.43:0/4098524676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72f010de60 0x7f72f01a2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.921+0000 7f72efe00640 1 -- 172.21.15.43:0/4098524676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72f0073b20 msgr2=0x7f72f010ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:17.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.921+0000 7f72efe00640 1 --2- 172.21.15.43:0/4098524676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72f0073b20 0x7f72f010ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.921+0000 7f72efe00640 1 -- 172.21.15.43:0/4098524676 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72f01a3520 con 0x7f72f0074440 2024-03-28T22:43:17.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.922+0000 7f72f5200640 1 --2- 172.21.15.43:0/4098524676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72f0073b20 0x7f72f010ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:17.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.922+0000 7f72efe00640 1 --2- 172.21.15.43:0/4098524676 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f72f0074440 0x7f72f010d390 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f72dc00e9f0 tx=0x7f72dc00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:17.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.922+0000 7f72ed600640 1 -- 172.21.15.43:0/4098524676 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72dc00cd80 con 0x7f72f0074440 2024-03-28T22:43:17.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.922+0000 7f72f6600640 1 -- 172.21.15.43:0/4098524676 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72f01a3810 con 0x7f72f0074440 2024-03-28T22:43:17.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.922+0000 7f72ed600640 1 -- 172.21.15.43:0/4098524676 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72dc005500 con 0x7f72f0074440 2024-03-28T22:43:17.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.922+0000 7f72ed600640 1 -- 172.21.15.43:0/4098524676 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72dc010690 con 0x7f72f0074440 2024-03-28T22:43:17.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.922+0000 7f72f6600640 1 -- 172.21.15.43:0/4098524676 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72f01ab0a0 con 0x7f72f0074440 2024-03-28T22:43:17.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.926+0000 7f72faa04740 1 Processor -- start 2024-03-28T22:43:17.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.926+0000 7f72faa04740 1 -- start start 2024-03-28T22:43:17.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.926+0000 7f72faa04740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564272478db0 0x564272477b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.926+0000 7f72faa04740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642724780d0 0x564272478550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.926+0000 7f72faa04740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564272481440 0x564272483830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.926+0000 7f72faa04740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5642723dc100 con 0x5642724780d0 2024-03-28T22:43:17.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.926+0000 7f72faa04740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x564272384ba0 con 0x564272481440 2024-03-28T22:43:17.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.926+0000 7f72faa04740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5642723d81a0 con 0x564272478db0 2024-03-28T22:43:17.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.927+0000 7f72f5c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564272478db0 0x564272477b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.927+0000 7f72ed600640 1 -- 172.21.15.43:0/4098524676 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f72dc0046c0 con 0x7f72f0074440 2024-03-28T22:43:17.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.927+0000 7f72f5c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564272478db0 0x564272477b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:49722/0 (socket says 172.21.15.43:49722) 2024-03-28T22:43:17.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.927+0000 7f72f5c00640 1 -- 172.21.15.43:0/1987582016 learned_addr learned my addr 172.21.15.43:0/1987582016 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:17.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.927+0000 7f72f5200640 1 --2- 172.21.15.43:0/1987582016 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642724780d0 0x564272478550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.927+0000 7f72efe00640 1 --2- 172.21.15.43:0/1987582016 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564272481440 0x564272483830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.927+0000 7f72f5c00640 1 -- 172.21.15.43:0/1987582016 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564272481440 msgr2=0x564272483830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:17.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.927+0000 7f72f5c00640 1 --2- 172.21.15.43:0/1987582016 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564272481440 0x564272483830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.927+0000 7f72f5c00640 1 -- 172.21.15.43:0/1987582016 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642724780d0 msgr2=0x564272478550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:17.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.927+0000 7f72f5c00640 1 --2- 172.21.15.43:0/1987582016 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642724780d0 0x564272478550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.927+0000 7f72f5c00640 1 -- 172.21.15.43:0/1987582016 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642723b51a0 con 0x564272478db0 2024-03-28T22:43:17.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.927+0000 7f72f5c00640 1 --2- 172.21.15.43:0/1987582016 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564272478db0 0x564272477b00 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f72e8009f20 tx=0x7f72e8010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b490201ff21c6fa7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:17.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.928+0000 7f72e2000640 1 -- 172.21.15.43:0/1987582016 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e8003c30 con 0x564272478db0 2024-03-28T22:43:17.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.928+0000 7f72e2000640 1 -- 172.21.15.43:0/1987582016 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72e8015070 con 0x564272478db0 2024-03-28T22:43:17.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.928+0000 7f72e2000640 1 -- 172.21.15.43:0/1987582016 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e8011880 con 0x564272478db0 2024-03-28T22:43:17.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.928+0000 7f72ed600640 1 --2- 172.21.15.43:0/4098524676 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f72d80a1ef0 0x7f72d80a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.928+0000 7f72ed600640 1 -- 172.21.15.43:0/4098524676 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f72dc0ce9b0 con 0x7f72f0074440 2024-03-28T22:43:17.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.928+0000 7f72faa04740 1 -- 172.21.15.43:0/1987582016 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564272478db0 msgr2=0x564272477b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:17.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.928+0000 7f72faa04740 1 --2- 172.21.15.43:0/1987582016 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564272478db0 0x564272477b00 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f72e8009f20 tx=0x7f72e8010860 comp rx=0 tx=0).stop 2024-03-28T22:43:17.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.929+0000 7f72f5200640 1 --2- 172.21.15.43:0/4098524676 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f72d80a1ef0 0x7f72d80a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.929+0000 7f72faa04740 1 -- 172.21.15.43:0/1987582016 shutdown_connections 2024-03-28T22:43:17.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.929+0000 7f72faa04740 1 --2- 172.21.15.43:0/1987582016 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564272481440 0x564272483830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.929+0000 7f72faa04740 1 --2- 172.21.15.43:0/1987582016 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642724780d0 0x564272478550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.929+0000 7f72faa04740 1 --2- 172.21.15.43:0/1987582016 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564272478db0 0x564272477b00 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.929+0000 7f72faa04740 1 -- 172.21.15.43:0/1987582016 >> 172.21.15.43:0/1987582016 conn(0x56427243ea10 msgr2=0x564272472b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:17.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.929+0000 7f72faa04740 1 -- 172.21.15.43:0/1987582016 shutdown_connections 2024-03-28T22:43:17.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.929+0000 7f72faa04740 1 -- 172.21.15.43:0/1987582016 wait complete. 2024-03-28T22:43:17.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.929+0000 7f72faa04740 1 -- start start 2024-03-28T22:43:17.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.929+0000 7f72f5200640 1 --2- 172.21.15.43:0/4098524676 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f72d80a1ef0 0x7f72d80a43b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f72e402efd0 tx=0x7f72e402cf90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:17.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.930+0000 7f72faa04740 1 Processor -- start 2024-03-28T22:43:17.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.930+0000 7f72faa04740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5642724780d0 0x56427247e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.930+0000 7f72faa04740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564272478db0 0x56427247e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.930+0000 7f72faa04740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564272481440 0x56427247eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.930+0000 7f72faa04740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5642724842b0 con 0x564272478db0 2024-03-28T22:43:17.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.930+0000 7f72faa04740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x56427237a250 con 0x5642724780d0 2024-03-28T22:43:17.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.930+0000 7f72faa04740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x564272484430 con 0x564272481440 2024-03-28T22:43:17.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.930+0000 7f72faa04740 10 client.0 _ll_register_callbacks cb 0x5642720efbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:17.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.930+0000 7f72f5200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5642724780d0 0x56427247e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.930+0000 7f72f5c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564272481440 0x56427247eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.930+0000 7f72f5200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5642724780d0 0x56427247e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:50788/0 (socket says 172.21.15.43:50788) 2024-03-28T22:43:17.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.930+0000 7f72f5200640 1 -- 172.21.15.43:0/2217664421 learned_addr learned my addr 172.21.15.43:0/2217664421 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:17.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.930+0000 7f72f5c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564272481440 0x56427247eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:49728/0 (socket says 172.21.15.43:49728) 2024-03-28T22:43:17.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.930+0000 7f72efe00640 1 --2- 172.21.15.43:0/2217664421 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564272478db0 0x56427247e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.931+0000 7f72f5200640 1 -- 172.21.15.43:0/2217664421 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564272481440 msgr2=0x56427247eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:17.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.931+0000 7f72f5200640 1 --2- 172.21.15.43:0/2217664421 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564272481440 0x56427247eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.931+0000 7f72f5200640 1 -- 172.21.15.43:0/2217664421 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564272478db0 msgr2=0x56427247e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:17.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.931+0000 7f72f5200640 1 --2- 172.21.15.43:0/2217664421 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564272478db0 0x56427247e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.931+0000 7f72f5200640 1 -- 172.21.15.43:0/2217664421 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564272396960 con 0x5642724780d0 2024-03-28T22:43:17.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.931+0000 7f72f5200640 1 --2- 172.21.15.43:0/2217664421 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5642724780d0 0x56427247e450 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f72e4009230 tx=0x7f72e4009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78f44db77a6521db server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:17.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.931+0000 7f72d6a00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:43:17.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.932+0000 7f72d6a00640 1 -- 172.21.15.43:0/2217664421 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e4005340 con 0x5642724780d0 2024-03-28T22:43:17.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.932+0000 7f72d6a00640 1 -- 172.21.15.43:0/2217664421 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f72e40054e0 con 0x5642724780d0 2024-03-28T22:43:17.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.932+0000 7f72faa04740 1 -- 172.21.15.43:0/2217664421 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72e8015070 con 0x5642724780d0 2024-03-28T22:43:17.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.932+0000 7f72d6a00640 1 -- 172.21.15.43:0/2217664421 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f72e40035f0 con 0x5642724780d0 2024-03-28T22:43:17.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.932+0000 7f72faa04740 10 client.27968 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:17.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.932+0000 7f72faa04740 1 -- 172.21.15.43:0/2217664421 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5642721ba7d0 con 0x5642724780d0 2024-03-28T22:43:17.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.932+0000 7f72d6a00640 1 -- 172.21.15.43:0/2217664421 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72e8015070 con 0x5642724780d0 2024-03-28T22:43:17.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.932+0000 7f72d6a00640 1 -- 172.21.15.43:0/2217664421 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f72e4051070 con 0x5642724780d0 2024-03-28T22:43:17.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.933+0000 7f72faa04740 10 client.27968 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:17.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.933+0000 7f72faa04740 10 client.27968 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:43:17.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.933+0000 7f72faa04740 20 client.27968 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:17.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.933+0000 7f72faa04740 1 --2- 172.21.15.43:0/2217664421 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x564272491810 0x564272493c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.933+0000 7f72faa04740 4 client.27968 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:43:17.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.933+0000 7f72faa04740 1 -- 172.21.15.43:0/2217664421 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5642723e9640 con 0x564272491810 2024-03-28T22:43:17.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.933+0000 7f72efe00640 1 --2- 172.21.15.43:0/2217664421 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x564272491810 0x564272493c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.933+0000 7f72efe00640 1 --2- 172.21.15.43:0/2217664421 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x564272491810 0x564272493c80 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f1cb6a22fc2540a server_cookie=5c1bc6b256b62c2d in_seq=0 out_seq=0 2024-03-28T22:43:17.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.933+0000 7f72d6a00640 10 client.27968 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:17.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.934+0000 7f72d6a00640 1 -- 172.21.15.43:0/2217664421 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5642723e9640 con 0x564272491810 2024-03-28T22:43:17.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.934+0000 7f72d6a00640 10 client.27968 handle_command_reply: tid=0 2024-03-28T22:43:17.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.934+0000 7f72faa04740 1 client.27968 shutdown 2024-03-28T22:43:17.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.935+0000 7f72faa04740 1 -- 172.21.15.43:0/2217664421 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5642724780d0 msgr2=0x56427247e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:17.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.935+0000 7f72faa04740 1 --2- 172.21.15.43:0/2217664421 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5642724780d0 0x56427247e450 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f72e4009230 tx=0x7f72e4009260 comp rx=0 tx=0).stop 2024-03-28T22:43:17.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.935+0000 7f72faa04740 1 -- 172.21.15.43:0/2217664421 shutdown_connections 2024-03-28T22:43:17.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.935+0000 7f72faa04740 1 --2- 172.21.15.43:0/2217664421 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564272481440 0x56427247eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.935+0000 7f72faa04740 1 --2- 172.21.15.43:0/2217664421 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564272478db0 0x56427247e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.935+0000 7f72faa04740 1 --2- 172.21.15.43:0/2217664421 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x564272491810 0x564272493c80 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.935+0000 7f72faa04740 1 --2- 172.21.15.43:0/2217664421 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5642724780d0 0x56427247e450 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.935+0000 7f72faa04740 1 -- 172.21.15.43:0/2217664421 >> 172.21.15.43:0/2217664421 conn(0x56427243ea10 msgr2=0x564272474870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:17.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.935+0000 7f72d6a00640 0 client.27968 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:17.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.935+0000 7f72faa04740 1 -- 172.21.15.43:0/2217664421 shutdown_connections 2024-03-28T22:43:17.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.935+0000 7f72faa04740 1 -- 172.21.15.43:0/2217664421 wait complete. 2024-03-28T22:43:17.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.935+0000 7f72faa04740 20 client.27968 trim_cache size 0 max 16384 2024-03-28T22:43:17.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.955+0000 7f72faa04740 1 Processor -- start 2024-03-28T22:43:17.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.955+0000 7f72faa04740 1 -- start start 2024-03-28T22:43:17.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.955+0000 7f72faa04740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5642724780d0 0x564272480b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.955+0000 7f72faa04740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564272478db0 0x56427248d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.955+0000 7f72faa04740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564272481440 0x564272474680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.955+0000 7f72faa04740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f72f01b4fc0 con 0x564272481440 2024-03-28T22:43:17.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.955+0000 7f72faa04740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f72f01b6b00 con 0x564272478db0 2024-03-28T22:43:17.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.955+0000 7f72faa04740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f72f01b5690 con 0x5642724780d0 2024-03-28T22:43:17.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.955+0000 7f72efe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564272478db0 0x56427248d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.955+0000 7f72efe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564272478db0 0x56427248d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:50794/0 (socket says 172.21.15.43:50794) 2024-03-28T22:43:17.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.955+0000 7f72efe00640 1 -- 172.21.15.43:0/3852829600 learned_addr learned my addr 172.21.15.43:0/3852829600 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:17.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.955+0000 7f72f5c00640 1 --2- 172.21.15.43:0/3852829600 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564272481440 0x564272474680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.955+0000 7f72f5200640 1 --2- 172.21.15.43:0/3852829600 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5642724780d0 0x564272480b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.956+0000 7f72efe00640 1 -- 172.21.15.43:0/3852829600 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5642724780d0 msgr2=0x564272480b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:17.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.956+0000 7f72efe00640 1 --2- 172.21.15.43:0/3852829600 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5642724780d0 0x564272480b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.956+0000 7f72efe00640 1 -- 172.21.15.43:0/3852829600 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564272481440 msgr2=0x564272474680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:17.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.956+0000 7f72efe00640 1 --2- 172.21.15.43:0/3852829600 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564272481440 0x564272474680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.956+0000 7f72efe00640 1 -- 172.21.15.43:0/3852829600 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642720f03f0 con 0x564272478db0 2024-03-28T22:43:17.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.956+0000 7f72f5c00640 1 --2- 172.21.15.43:0/3852829600 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564272481440 0x564272474680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:43:17.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.956+0000 7f72efe00640 1 --2- 172.21.15.43:0/3852829600 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564272478db0 0x56427248d090 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f72dc09aca0 tx=0x7f72dc0d1f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4976fcc91feaca2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:17.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.956+0000 7f72e1600640 1 -- 172.21.15.43:0/3852829600 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72dc09b6a0 con 0x564272478db0 2024-03-28T22:43:17.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.956+0000 7f72e1600640 1 -- 172.21.15.43:0/3852829600 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72dc09afa0 con 0x564272478db0 2024-03-28T22:43:17.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.956+0000 7f72e1600640 1 -- 172.21.15.43:0/3852829600 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72dc0a32b0 con 0x564272478db0 2024-03-28T22:43:17.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.957+0000 7f72faa04740 1 -- 172.21.15.43:0/3852829600 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564272478db0 msgr2=0x56427248d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:17.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.957+0000 7f72faa04740 1 --2- 172.21.15.43:0/3852829600 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564272478db0 0x56427248d090 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f72dc09aca0 tx=0x7f72dc0d1f90 comp rx=0 tx=0).stop 2024-03-28T22:43:17.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.957+0000 7f72faa04740 1 -- 172.21.15.43:0/3852829600 shutdown_connections 2024-03-28T22:43:17.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.957+0000 7f72faa04740 1 --2- 172.21.15.43:0/3852829600 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564272478db0 0x56427248d090 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.957+0000 7f72faa04740 1 --2- 172.21.15.43:0/3852829600 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564272481440 0x564272474680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.957+0000 7f72faa04740 1 --2- 172.21.15.43:0/3852829600 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5642724780d0 0x564272480b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.957+0000 7f72faa04740 1 -- 172.21.15.43:0/3852829600 >> 172.21.15.43:0/3852829600 conn(0x56427243ea10 msgr2=0x564272473f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:17.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.957+0000 7f72faa04740 1 -- 172.21.15.43:0/3852829600 shutdown_connections 2024-03-28T22:43:17.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.957+0000 7f72faa04740 1 -- 172.21.15.43:0/3852829600 wait complete. 2024-03-28T22:43:17.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.958+0000 7f72faa04740 1 -- start start 2024-03-28T22:43:17.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.958+0000 7f72faa04740 1 Processor -- start 2024-03-28T22:43:17.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.958+0000 7f72faa04740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642724780d0 0x564272482130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.958+0000 7f72faa04740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564272478db0 0x564272482670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.958+0000 7f72faa04740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564272481440 0x564272482bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.958+0000 7f72faa04740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f72f01b6430 con 0x5642724780d0 2024-03-28T22:43:17.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.958+0000 7f72faa04740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f72f01b5d60 con 0x564272478db0 2024-03-28T22:43:17.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.958+0000 7f72faa04740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f72f01b2fa0 con 0x564272481440 2024-03-28T22:43:17.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.958+0000 7f72faa04740 10 client.0 _ll_register_callbacks cb 0x7f72e40095a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:17.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.959+0000 7f72efe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564272478db0 0x564272482670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.959+0000 7f72efe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564272478db0 0x564272482670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:50802/0 (socket says 172.21.15.43:50802) 2024-03-28T22:43:17.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.959+0000 7f72efe00640 1 -- 172.21.15.43:0/2107376676 learned_addr learned my addr 172.21.15.43:0/2107376676 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:17.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.959+0000 7f72f5c00640 1 --2- 172.21.15.43:0/2107376676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564272481440 0x564272482bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.959+0000 7f72f5200640 1 --2- 172.21.15.43:0/2107376676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642724780d0 0x564272482130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.959+0000 7f72efe00640 1 -- 172.21.15.43:0/2107376676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564272481440 msgr2=0x564272482bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:17.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.959+0000 7f72efe00640 1 --2- 172.21.15.43:0/2107376676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564272481440 0x564272482bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.959+0000 7f72efe00640 1 -- 172.21.15.43:0/2107376676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642724780d0 msgr2=0x564272482130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:17.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.959+0000 7f72efe00640 1 --2- 172.21.15.43:0/2107376676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642724780d0 0x564272482130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.959+0000 7f72efe00640 1 -- 172.21.15.43:0/2107376676 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642720b00f0 con 0x564272478db0 2024-03-28T22:43:17.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.959+0000 7f72efe00640 1 --2- 172.21.15.43:0/2107376676 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564272478db0 0x564272482670 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f72dc097970 tx=0x7f72dc09ab60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3f8d4243a7f7a62 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:17.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.959+0000 7f72d6a00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:43:17.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.960+0000 7f72d6a00640 1 -- 172.21.15.43:0/2107376676 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72dc0b9060 con 0x564272478db0 2024-03-28T22:43:17.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.960+0000 7f72faa04740 1 -- 172.21.15.43:0/2107376676 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72dc09afa0 con 0x564272478db0 2024-03-28T22:43:17.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.960+0000 7f72d6a00640 1 -- 172.21.15.43:0/2107376676 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f72dc0a32b0 con 0x564272478db0 2024-03-28T22:43:17.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.964+0000 7f72d6a00640 1 -- 172.21.15.43:0/2107376676 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f72dc0cc180 con 0x564272478db0 2024-03-28T22:43:17.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.964+0000 7f72faa04740 10 client.27974 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:17.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.964+0000 7f72faa04740 1 -- 172.21.15.43:0/2107376676 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5642724f7f60 con 0x564272478db0 2024-03-28T22:43:17.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.964+0000 7f72d6a00640 1 -- 172.21.15.43:0/2107376676 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72dc09afa0 con 0x564272478db0 2024-03-28T22:43:17.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.964+0000 7f72d6a00640 1 -- 172.21.15.43:0/2107376676 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f72dc0b9320 con 0x564272478db0 2024-03-28T22:43:17.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.964+0000 7f72faa04740 10 client.27974 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:17.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.964+0000 7f72faa04740 10 client.27974 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:43:17.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.964+0000 7f72faa04740 20 client.27974 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:17.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.964+0000 7f72faa04740 1 --2- 172.21.15.43:0/2107376676 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5642724f8610 0x5642724faa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:17.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.964+0000 7f72faa04740 4 client.27974 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:43:17.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.964+0000 7f72faa04740 1 -- 172.21.15.43:0/2107376676 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5642723e3990 con 0x5642724f8610 2024-03-28T22:43:17.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.965+0000 7f72f5200640 1 --2- 172.21.15.43:0/2107376676 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5642724f8610 0x5642724faa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:17.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.965+0000 7f72f5200640 1 --2- 172.21.15.43:0/2107376676 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5642724f8610 0x5642724faa30 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7ed6e556a1b1331 server_cookie=fdad361876c402ed in_seq=0 out_seq=0 2024-03-28T22:43:17.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.965+0000 7f72d6a00640 10 client.27974 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:17.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.967+0000 7f72d6a00640 1 -- 172.21.15.43:0/2107376676 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5642723e3990 con 0x5642724f8610 2024-03-28T22:43:17.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.967+0000 7f72d6a00640 10 client.27974 handle_command_reply: tid=0 2024-03-28T22:43:17.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.967+0000 7f72faa04740 1 client.27974 shutdown 2024-03-28T22:43:17.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.967+0000 7f72faa04740 1 -- 172.21.15.43:0/2107376676 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564272478db0 msgr2=0x564272482670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:17.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.967+0000 7f72faa04740 1 --2- 172.21.15.43:0/2107376676 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564272478db0 0x564272482670 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f72dc097970 tx=0x7f72dc09ab60 comp rx=0 tx=0).stop 2024-03-28T22:43:17.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.967+0000 7f72faa04740 1 -- 172.21.15.43:0/2107376676 shutdown_connections 2024-03-28T22:43:17.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.967+0000 7f72faa04740 1 --2- 172.21.15.43:0/2107376676 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5642724f8610 0x5642724faa30 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.967+0000 7f72faa04740 1 --2- 172.21.15.43:0/2107376676 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564272478db0 0x564272482670 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.967+0000 7f72faa04740 1 --2- 172.21.15.43:0/2107376676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564272481440 0x564272482bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.967+0000 7f72d6a00640 0 client.27974 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:17.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.967+0000 7f72faa04740 1 --2- 172.21.15.43:0/2107376676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5642724780d0 0x564272482130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.967+0000 7f72faa04740 1 -- 172.21.15.43:0/2107376676 >> 172.21.15.43:0/2107376676 conn(0x56427243ea10 msgr2=0x564272471d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:17.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.967+0000 7f72faa04740 1 -- 172.21.15.43:0/2107376676 shutdown_connections 2024-03-28T22:43:17.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.967+0000 7f72faa04740 1 -- 172.21.15.43:0/2107376676 wait complete. 2024-03-28T22:43:17.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.968+0000 7f72faa04740 20 client.27974 trim_cache size 0 max 16384 2024-03-28T22:43:17.966 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:43:17.966 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:43:17.966 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-03-28T22:43:17.967 INFO:teuthology.orchestra.run.smithi043.stdout: "14c06235-f0c6-48e9-a3a9-923d2337a193": { 2024-03-28T22:43:17.967 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-03-28T22:43:17.967 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "14c06235-f0c6-48e9-a3a9-923d2337a193", 2024-03-28T22:43:17.967 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-03-28T22:43:17.967 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:43:17.967 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:43:17.967 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:43:17.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.968+0000 7f72f6600640 1 -- 172.21.15.43:0/4098524676 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f72d80a1ef0 msgr2=0x7f72d80a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:17.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.968+0000 7f72f6600640 1 --2- 172.21.15.43:0/4098524676 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f72d80a1ef0 0x7f72d80a43b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f72e402efd0 tx=0x7f72e402cf90 comp rx=0 tx=0).stop 2024-03-28T22:43:17.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.968+0000 7f72f6600640 1 -- 172.21.15.43:0/4098524676 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f72f0074440 msgr2=0x7f72f010d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:17.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.968+0000 7f72f6600640 1 --2- 172.21.15.43:0/4098524676 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f72f0074440 0x7f72f010d390 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f72dc00e9f0 tx=0x7f72dc00edf0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.968+0000 7f72f6600640 1 -- 172.21.15.43:0/4098524676 shutdown_connections 2024-03-28T22:43:17.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.968+0000 7f72f6600640 1 --2- 172.21.15.43:0/4098524676 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f72d80a1ef0 0x7f72d80a43b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.968+0000 7f72f6600640 1 --2- 172.21.15.43:0/4098524676 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f72f0074440 0x7f72f010d390 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.968+0000 7f72f6600640 1 --2- 172.21.15.43:0/4098524676 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f72f010de60 0x7f72f01a2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.968+0000 7f72f6600640 1 --2- 172.21.15.43:0/4098524676 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f72f0073b20 0x7f72f010ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:17.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.968+0000 7f72f6600640 1 -- 172.21.15.43:0/4098524676 >> 172.21.15.43:0/4098524676 conn(0x7f72f00fd0f0 msgr2=0x7f72f00fde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:17.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.969+0000 7f72f6600640 1 -- 172.21.15.43:0/4098524676 shutdown_connections 2024-03-28T22:43:17.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:17.969+0000 7f72f6600640 1 -- 172.21.15.43:0/4098524676 wait complete. 2024-03-28T22:43:17.984 INFO:tasks.cephfs.filesystem:scrub status for tag:14c06235-f0c6-48e9-a3a9-923d2337a193 - {'path': '/', 'tag': '14c06235-f0c6-48e9-a3a9-923d2337a193', 'options': 'recursive,force'} 2024-03-28T22:43:18.374 INFO:tasks.workunit.client.0.smithi043.stdout: 1 417408 7.38 MB/sec execute 169 sec latency 33.117 ms 2024-03-28T22:43:19.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:18 smithi043 ceph-mon[31823]: pgmap v188: 129 pgs: 129 active+clean; 45 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 216 op/s 2024-03-28T22:43:19.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:18 smithi043 ceph-mon[31823]: scrub queued for path: / 2024-03-28T22:43:19.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:18 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:43:19.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:18 smithi043 ceph-mon[31823]: scrub summary: active paths [/] 2024-03-28T22:43:19.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:18 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:43:19.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:18 smithi043 ceph-mon[31823]: scrub summary: idle 2024-03-28T22:43:19.373 INFO:tasks.workunit.client.0.smithi043.stdout: 1 418954 7.41 MB/sec execute 170 sec latency 10.343 ms 2024-03-28T22:43:19.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:18 smithi172 ceph-mon[36566]: pgmap v188: 129 pgs: 129 active+clean; 45 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 216 op/s 2024-03-28T22:43:19.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:18 smithi172 ceph-mon[36566]: scrub queued for path: / 2024-03-28T22:43:19.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:18 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:43:19.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:18 smithi172 ceph-mon[36566]: scrub summary: active paths [/] 2024-03-28T22:43:19.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:18 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:43:19.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:18 smithi172 ceph-mon[36566]: scrub summary: idle 2024-03-28T22:43:19.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:18 smithi082 ceph-mon[36471]: pgmap v188: 129 pgs: 129 active+clean; 45 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 216 op/s 2024-03-28T22:43:19.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:18 smithi082 ceph-mon[36471]: scrub queued for path: / 2024-03-28T22:43:19.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:18 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:43:19.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:18 smithi082 ceph-mon[36471]: scrub summary: active paths [/] 2024-03-28T22:43:19.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:18 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:43:19.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:18 smithi082 ceph-mon[36471]: scrub summary: idle 2024-03-28T22:43:20.374 INFO:tasks.workunit.client.0.smithi043.stdout: 1 420039 7.40 MB/sec execute 171 sec latency 24.262 ms 2024-03-28T22:43:21.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:20 smithi043 ceph-mon[31823]: pgmap v189: 129 pgs: 129 active+clean; 45 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 1.8 MiB/s wr, 144 op/s 2024-03-28T22:43:21.373 INFO:tasks.workunit.client.0.smithi043.stdout: 1 421141 7.37 MB/sec execute 172 sec latency 83.773 ms 2024-03-28T22:43:21.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:20 smithi172 ceph-mon[36566]: pgmap v189: 129 pgs: 129 active+clean; 45 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 1.8 MiB/s wr, 144 op/s 2024-03-28T22:43:21.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:20 smithi082 ceph-mon[36471]: pgmap v189: 129 pgs: 129 active+clean; 45 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 1.8 MiB/s wr, 144 op/s 2024-03-28T22:43:22.374 INFO:tasks.workunit.client.0.smithi043.stdout: 1 422555 7.39 MB/sec execute 173 sec latency 16.660 ms 2024-03-28T22:43:23.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:22 smithi043 ceph-mon[31823]: pgmap v190: 129 pgs: 129 active+clean; 45 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 1.8 MiB/s wr, 144 op/s 2024-03-28T22:43:23.374 INFO:tasks.workunit.client.0.smithi043.stdout: 1 423625 7.38 MB/sec execute 174 sec latency 27.807 ms 2024-03-28T22:43:23.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:22 smithi172 ceph-mon[36566]: pgmap v190: 129 pgs: 129 active+clean; 45 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 1.8 MiB/s wr, 144 op/s 2024-03-28T22:43:23.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:22 smithi082 ceph-mon[36471]: pgmap v190: 129 pgs: 129 active+clean; 45 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 1.8 MiB/s wr, 144 op/s 2024-03-28T22:43:24.374 INFO:tasks.workunit.client.0.smithi043.stdout: 1 425051 7.38 MB/sec execute 175 sec latency 23.600 ms 2024-03-28T22:43:25.250 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:24 smithi082 ceph-mon[36471]: pgmap v191: 129 pgs: 129 active+clean; 49 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 220 op/s 2024-03-28T22:43:25.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:24 smithi043 ceph-mon[31823]: pgmap v191: 129 pgs: 129 active+clean; 49 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 220 op/s 2024-03-28T22:43:25.374 INFO:tasks.workunit.client.0.smithi043.stdout: 1 426231 7.38 MB/sec execute 176 sec latency 26.702 ms 2024-03-28T22:43:25.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:24 smithi172 ceph-mon[36566]: pgmap v191: 129 pgs: 129 active+clean; 49 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 220 op/s 2024-03-28T22:43:26.374 INFO:tasks.workunit.client.0.smithi043.stdout: 1 427179 7.36 MB/sec execute 177 sec latency 85.383 ms 2024-03-28T22:43:27.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:26 smithi043 ceph-mon[31823]: pgmap v192: 129 pgs: 129 active+clean; 49 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 191 op/s 2024-03-28T22:43:27.374 INFO:tasks.workunit.client.0.smithi043.stdout: 1 428568 7.36 MB/sec execute 178 sec latency 39.193 ms 2024-03-28T22:43:27.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:26 smithi172 ceph-mon[36566]: pgmap v192: 129 pgs: 129 active+clean; 49 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 191 op/s 2024-03-28T22:43:27.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:26 smithi082 ceph-mon[36471]: pgmap v192: 129 pgs: 129 active+clean; 49 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 191 op/s 2024-03-28T22:43:28.374 INFO:tasks.workunit.client.0.smithi043.stdout: 1 429842 7.36 MB/sec execute 179 sec latency 18.358 ms 2024-03-28T22:43:28.428 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.429+0000 7fd5e2600640 1 -- 172.21.15.43:0/3620983545 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5dc109940 msgr2=0x7fd5dc109d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:28.428 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.429+0000 7fd5e2600640 1 --2- 172.21.15.43:0/3620983545 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5dc109940 0x7fd5dc109d20 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc009930 tx=0x7fd5cc02bc20 comp rx=0 tx=0).stop 2024-03-28T22:43:28.428 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.429+0000 7fd5e2600640 1 -- 172.21.15.43:0/3620983545 shutdown_connections 2024-03-28T22:43:28.428 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.429+0000 7fd5e2600640 1 --2- 172.21.15.43:0/3620983545 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd5dc1042f0 0x7fd5dc110800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.428 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.429+0000 7fd5e2600640 1 --2- 172.21.15.43:0/3620983545 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5dc103930 0x7fd5dc103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.428 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.429+0000 7fd5e2600640 1 --2- 172.21.15.43:0/3620983545 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5dc109940 0x7fd5dc109d20 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.428 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.429+0000 7fd5e2600640 1 -- 172.21.15.43:0/3620983545 >> 172.21.15.43:0/3620983545 conn(0x7fd5dc0ff400 msgr2=0x7fd5dc101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:28.428 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.429+0000 7fd5e2600640 1 -- 172.21.15.43:0/3620983545 shutdown_connections 2024-03-28T22:43:28.428 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.429+0000 7fd5e2600640 1 -- 172.21.15.43:0/3620983545 wait complete. 2024-03-28T22:43:28.429 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.430+0000 7fd5e2600640 1 Processor -- start 2024-03-28T22:43:28.430 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.430+0000 7fd5e2600640 1 -- start start 2024-03-28T22:43:28.430 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.431+0000 7fd5e2600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd5dc103930 0x7fd5dc19a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:28.430 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.431+0000 7fd5e2600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5dc1042f0 0x7fd5dc19a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:28.430 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.431+0000 7fd5e2600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5dc109940 0x7fd5dc19eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:28.430 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.431+0000 7fd5e2600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd5dc113da0 con 0x7fd5dc109940 2024-03-28T22:43:28.430 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.431+0000 7fd5e2600640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd5dc113c20 con 0x7fd5dc103930 2024-03-28T22:43:28.430 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.431+0000 7fd5e2600640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd5dc113f20 con 0x7fd5dc1042f0 2024-03-28T22:43:28.430 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.431+0000 7fd5db400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd5dc103930 0x7fd5dc19a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:28.430 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.431+0000 7fd5db400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd5dc103930 0x7fd5dc19a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:60968/0 (socket says 172.21.15.43:60968) 2024-03-28T22:43:28.430 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.431+0000 7fd5db400640 1 -- 172.21.15.43:0/2491547709 learned_addr learned my addr 172.21.15.43:0/2491547709 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:28.430 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.431+0000 7fd5daa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5dc1042f0 0x7fd5dc19a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:28.430 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.431+0000 7fd5dbe00640 1 --2- 172.21.15.43:0/2491547709 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5dc109940 0x7fd5dc19eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:28.432 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.432+0000 7fd5db400640 1 -- 172.21.15.43:0/2491547709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5dc1042f0 msgr2=0x7fd5dc19a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:28.432 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.432+0000 7fd5db400640 1 --2- 172.21.15.43:0/2491547709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5dc1042f0 0x7fd5dc19a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.432 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.432+0000 7fd5db400640 1 -- 172.21.15.43:0/2491547709 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5dc109940 msgr2=0x7fd5dc19eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:28.432 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.432+0000 7fd5db400640 1 --2- 172.21.15.43:0/2491547709 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5dc109940 0x7fd5dc19eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.432 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.432+0000 7fd5db400640 1 -- 172.21.15.43:0/2491547709 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5dc19f1d0 con 0x7fd5dc103930 2024-03-28T22:43:28.432 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.432+0000 7fd5daa00640 1 --2- 172.21.15.43:0/2491547709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5dc1042f0 0x7fd5dc19a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:28.432 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.432+0000 7fd5db400640 1 --2- 172.21.15.43:0/2491547709 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd5dc103930 0x7fd5dc19a1e0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc005ab0 tx=0x7fd5cc0042e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:28.432 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.432+0000 7fd5d3e00640 1 -- 172.21.15.43:0/2491547709 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc043020 con 0x7fd5dc103930 2024-03-28T22:43:28.432 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.432+0000 7fd5d3e00640 1 -- 172.21.15.43:0/2491547709 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5cc0044d0 con 0x7fd5dc103930 2024-03-28T22:43:28.432 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.432+0000 7fd5e2600640 1 -- 172.21.15.43:0/2491547709 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5dc19f460 con 0x7fd5dc103930 2024-03-28T22:43:28.432 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.433+0000 7fd5d3e00640 1 -- 172.21.15.43:0/2491547709 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc003bb0 con 0x7fd5dc103930 2024-03-28T22:43:28.432 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.433+0000 7fd5e2600640 1 -- 172.21.15.43:0/2491547709 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5dc1a6d50 con 0x7fd5dc103930 2024-03-28T22:43:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.436+0000 7fd5e5384740 1 Processor -- start 2024-03-28T22:43:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.436+0000 7fd5e5384740 1 -- start start 2024-03-28T22:43:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.436+0000 7fd5d3e00640 1 -- 172.21.15.43:0/2491547709 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fd5cc046020 con 0x7fd5dc103930 2024-03-28T22:43:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.436+0000 7fd5e5384740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55788b4ce3a0 0x55788b4ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.436+0000 7fd5e5384740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55788b4cecc0 0x55788b4d7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.437+0000 7fd5e5384740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55788b4d7a90 0x55788b4d9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.437+0000 7fd5e5384740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55788b431640 con 0x55788b4d7a90 2024-03-28T22:43:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.437+0000 7fd5e5384740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55788b3da810 con 0x55788b4cecc0 2024-03-28T22:43:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.437+0000 7fd5e5384740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55788b42d6e0 con 0x55788b4ce3a0 2024-03-28T22:43:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.437+0000 7fd5dbe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55788b4ce3a0 0x55788b4ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:28.438 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.437+0000 7fd5db400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55788b4cecc0 0x55788b4d7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:28.438 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.437+0000 7fd5dbe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55788b4ce3a0 0x55788b4ce780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:41830/0 (socket says 172.21.15.43:41830) 2024-03-28T22:43:28.438 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.437+0000 7fd5dbe00640 1 -- 172.21.15.43:0/600392637 learned_addr learned my addr 172.21.15.43:0/600392637 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:28.438 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.437+0000 7fd5daa00640 1 --2- 172.21.15.43:0/600392637 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55788b4d7a90 0x55788b4d9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:28.438 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.437+0000 7fd5db400640 1 -- 172.21.15.43:0/600392637 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55788b4ce3a0 msgr2=0x55788b4ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:28.438 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.437+0000 7fd5db400640 1 --2- 172.21.15.43:0/600392637 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55788b4ce3a0 0x55788b4ce780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.438 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.437+0000 7fd5db400640 1 -- 172.21.15.43:0/600392637 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55788b4d7a90 msgr2=0x55788b4d9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:28.438 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.437+0000 7fd5db400640 1 --2- 172.21.15.43:0/600392637 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55788b4d7a90 0x55788b4d9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.438 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.437+0000 7fd5db400640 1 -- 172.21.15.43:0/600392637 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55788b146210 con 0x55788b4cecc0 2024-03-28T22:43:28.438 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.438+0000 7fd5db400640 1 --2- 172.21.15.43:0/600392637 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55788b4cecc0 0x55788b4d7550 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc0ed820 tx=0x7fd5cc0f84b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbc9b3c03f4ce915 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:28.438 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.438+0000 7fd5d0c00640 1 -- 172.21.15.43:0/600392637 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc0f2310 con 0x55788b4cecc0 2024-03-28T22:43:28.438 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.438+0000 7fd5d0c00640 1 -- 172.21.15.43:0/600392637 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5cc0f24b0 con 0x55788b4cecc0 2024-03-28T22:43:28.438 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.438+0000 7fd5d0c00640 1 -- 172.21.15.43:0/600392637 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc101930 con 0x55788b4cecc0 2024-03-28T22:43:28.438 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.438+0000 7fd5d3e00640 1 --2- 172.21.15.43:0/2491547709 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd5c00a2010 0x7fd5c00a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:28.439 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.438+0000 7fd5d3e00640 1 -- 172.21.15.43:0/2491547709 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fd5cc0f03c0 con 0x7fd5dc103930 2024-03-28T22:43:28.439 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.439+0000 7fd5dbe00640 1 --2- 172.21.15.43:0/2491547709 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd5c00a2010 0x7fd5c00a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:28.439 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.439+0000 7fd5e5384740 1 -- 172.21.15.43:0/600392637 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55788b4cecc0 msgr2=0x55788b4d7550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:28.439 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.439+0000 7fd5e5384740 1 --2- 172.21.15.43:0/600392637 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55788b4cecc0 0x55788b4d7550 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc0ed820 tx=0x7fd5cc0f84b0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.441 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.439+0000 7fd5e5384740 1 -- 172.21.15.43:0/600392637 shutdown_connections 2024-03-28T22:43:28.441 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.439+0000 7fd5e5384740 1 --2- 172.21.15.43:0/600392637 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55788b4cecc0 0x55788b4d7550 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.442 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.439+0000 7fd5e5384740 1 --2- 172.21.15.43:0/600392637 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55788b4d7a90 0x55788b4d9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.442 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.439+0000 7fd5e5384740 1 --2- 172.21.15.43:0/600392637 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55788b4ce3a0 0x55788b4ce780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.442 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.439+0000 7fd5e5384740 1 -- 172.21.15.43:0/600392637 >> 172.21.15.43:0/600392637 conn(0x55788b4c8cc0 msgr2=0x55788b4c90a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:28.442 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.439+0000 7fd5e5384740 1 -- 172.21.15.43:0/600392637 shutdown_connections 2024-03-28T22:43:28.442 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.439+0000 7fd5e5384740 1 -- 172.21.15.43:0/600392637 wait complete. 2024-03-28T22:43:28.442 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.440+0000 7fd5dbe00640 1 --2- 172.21.15.43:0/2491547709 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd5c00a2010 0x7fd5c00a44d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4011630 tx=0x7fd5d4009500 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:28.442 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.440+0000 7fd5e5384740 1 -- start start 2024-03-28T22:43:28.442 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.440+0000 7fd5e5384740 1 Processor -- start 2024-03-28T22:43:28.442 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.441+0000 7fd5e5384740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55788b4ce3a0 0x55788b4d4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:28.442 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.441+0000 7fd5e5384740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55788b4cecc0 0x55788b4d4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:28.442 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.441+0000 7fd5e5384740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55788b4d7a90 0x55788b4d5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:28.442 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.441+0000 7fd5e5384740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55788b4da480 con 0x55788b4d7a90 2024-03-28T22:43:28.442 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.441+0000 7fd5e5384740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55788b3cfd00 con 0x55788b4cecc0 2024-03-28T22:43:28.442 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.441+0000 7fd5db400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55788b4ce3a0 0x55788b4d4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:28.443 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.441+0000 7fd5daa00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55788b4cecc0 0x55788b4d4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:28.443 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.441+0000 7fd5db400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55788b4ce3a0 0x55788b4d4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:41844/0 (socket says 172.21.15.43:41844) 2024-03-28T22:43:28.443 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.441+0000 7fd5db400640 1 -- 172.21.15.43:0/741818414 learned_addr learned my addr 172.21.15.43:0/741818414 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:28.443 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.441+0000 7fd5e5384740 1 -- 172.21.15.43:0/741818414 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55788b4da600 con 0x55788b4ce3a0 2024-03-28T22:43:28.443 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.441+0000 7fd5dbe00640 1 --2- 172.21.15.43:0/741818414 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55788b4d7a90 0x55788b4d5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:28.443 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.441+0000 7fd5e5384740 10 client.0 _ll_register_callbacks cb 0x55788b3ba490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:28.443 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.442+0000 7fd5db400640 1 -- 172.21.15.43:0/741818414 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55788b4cecc0 msgr2=0x55788b4d4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:28.443 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.442+0000 7fd5db400640 1 --2- 172.21.15.43:0/741818414 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55788b4cecc0 0x55788b4d4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.443 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.442+0000 7fd5db400640 1 -- 172.21.15.43:0/741818414 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55788b4d7a90 msgr2=0x55788b4d5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:28.444 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.442+0000 7fd5db400640 1 --2- 172.21.15.43:0/741818414 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55788b4d7a90 0x55788b4d5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.444 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.442+0000 7fd5db400640 1 -- 172.21.15.43:0/741818414 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55788b212620 con 0x55788b4ce3a0 2024-03-28T22:43:28.444 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.442+0000 7fd5db400640 1 --2- 172.21.15.43:0/741818414 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55788b4ce3a0 0x55788b4d4a90 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc0bca50 tx=0x7fd5cc02bc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b96b85b91e855f82 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:28.444 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.442+0000 7fd5c5600640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:43:28.444 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.442+0000 7fd5c5600640 1 -- 172.21.15.43:0/741818414 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc0ede60 con 0x55788b4ce3a0 2024-03-28T22:43:28.444 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.442+0000 7fd5c5600640 1 -- 172.21.15.43:0/741818414 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fd5cc0badc0 con 0x55788b4ce3a0 2024-03-28T22:43:28.444 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.442+0000 7fd5e5384740 1 -- 172.21.15.43:0/741818414 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5cc0f24b0 con 0x55788b4ce3a0 2024-03-28T22:43:28.444 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.443+0000 7fd5c5600640 1 -- 172.21.15.43:0/741818414 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd5cc0b8d60 con 0x55788b4ce3a0 2024-03-28T22:43:28.444 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.443+0000 7fd5e5384740 10 client.27928 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:28.444 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.443+0000 7fd5e5384740 1 -- 172.21.15.43:0/741818414 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55788b4da480 con 0x55788b4ce3a0 2024-03-28T22:43:28.444 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.443+0000 7fd5c5600640 1 -- 172.21.15.43:0/741818414 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5cc0f24b0 con 0x55788b4ce3a0 2024-03-28T22:43:28.446 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.443+0000 7fd5c5600640 1 -- 172.21.15.43:0/741818414 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd5cc02cb80 con 0x55788b4ce3a0 2024-03-28T22:43:28.446 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.444+0000 7fd5e5384740 10 client.27928 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:28.446 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.444+0000 7fd5e5384740 10 client.27928 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:43:28.446 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.444+0000 7fd5e5384740 20 client.27928 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:28.446 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.444+0000 7fd5e5384740 1 --2- 172.21.15.43:0/741818414 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55788b4e74c0 0x55788b4e98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:28.446 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.444+0000 7fd5e5384740 4 client.27928 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:43:28.446 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.444+0000 7fd5e5384740 1 -- 172.21.15.43:0/741818414 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55788b445030 con 0x55788b4e74c0 2024-03-28T22:43:28.447 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.444+0000 7fd5daa00640 1 --2- 172.21.15.43:0/741818414 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55788b4e74c0 0x55788b4e98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:28.447 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.445+0000 7fd5daa00640 1 --2- 172.21.15.43:0/741818414 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55788b4e74c0 0x55788b4e98b0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8834ee3e753930d3 server_cookie=174ef43723cb581b in_seq=0 out_seq=0 2024-03-28T22:43:28.447 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.445+0000 7fd5c5600640 10 client.27928 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:28.447 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.446+0000 7fd5c5600640 1 -- 172.21.15.43:0/741818414 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55788b445030 con 0x55788b4e74c0 2024-03-28T22:43:28.447 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.446+0000 7fd5c5600640 10 client.27928 handle_command_reply: tid=0 2024-03-28T22:43:28.447 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.446+0000 7fd5e5384740 1 client.27928 shutdown 2024-03-28T22:43:28.447 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.446+0000 7fd5e5384740 1 -- 172.21.15.43:0/741818414 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55788b4ce3a0 msgr2=0x55788b4d4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:28.447 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.446+0000 7fd5e5384740 1 --2- 172.21.15.43:0/741818414 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55788b4ce3a0 0x55788b4d4a90 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc0bca50 tx=0x7fd5cc02bc20 comp rx=0 tx=0).stop 2024-03-28T22:43:28.447 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.446+0000 7fd5e5384740 1 -- 172.21.15.43:0/741818414 shutdown_connections 2024-03-28T22:43:28.447 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.446+0000 7fd5e5384740 1 --2- 172.21.15.43:0/741818414 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55788b4e74c0 0x55788b4e98b0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.447 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.446+0000 7fd5e5384740 1 --2- 172.21.15.43:0/741818414 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55788b4cecc0 0x55788b4d4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.447 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.446+0000 7fd5e5384740 1 --2- 172.21.15.43:0/741818414 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55788b4d7a90 0x55788b4d5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.447 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.446+0000 7fd5e5384740 1 --2- 172.21.15.43:0/741818414 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55788b4ce3a0 0x55788b4d4a90 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.447 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.446+0000 7fd5e5384740 1 -- 172.21.15.43:0/741818414 >> 172.21.15.43:0/741818414 conn(0x55788b4c8cc0 msgr2=0x55788b4caa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:28.448 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.446+0000 7fd5c5600640 0 client.27928 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:28.448 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.446+0000 7fd5e5384740 1 -- 172.21.15.43:0/741818414 shutdown_connections 2024-03-28T22:43:28.448 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.447+0000 7fd5e5384740 1 -- 172.21.15.43:0/741818414 wait complete. 2024-03-28T22:43:28.448 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.447+0000 7fd5e5384740 20 client.27928 trim_cache size 0 max 16384 2024-03-28T22:43:28.464 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.466+0000 7fd5e5384740 1 Processor -- start 2024-03-28T22:43:28.464 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.466+0000 7fd5e5384740 1 -- start start 2024-03-28T22:43:28.465 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.466+0000 7fd5e5384740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55788b4ce3a0 0x55788b4e3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:28.465 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.466+0000 7fd5e5384740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55788b4cecc0 0x55788b4c38d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:28.465 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.466+0000 7fd5e5384740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55788b4d7a90 0x55788b4ef1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:28.466 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.466+0000 7fd5e5384740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd5dc1b0db0 con 0x55788b4ce3a0 2024-03-28T22:43:28.466 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.466+0000 7fd5e5384740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd5dc1b28f0 con 0x55788b4d7a90 2024-03-28T22:43:28.466 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.466+0000 7fd5e5384740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd5dc1b1480 con 0x55788b4cecc0 2024-03-28T22:43:28.466 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.466+0000 7fd5daa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55788b4cecc0 0x55788b4c38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:28.466 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.466+0000 7fd5daa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55788b4cecc0 0x55788b4c38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:41852/0 (socket says 172.21.15.43:41852) 2024-03-28T22:43:28.466 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.466+0000 7fd5daa00640 1 -- 172.21.15.43:0/2597312553 learned_addr learned my addr 172.21.15.43:0/2597312553 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:28.466 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.466+0000 7fd5db400640 1 --2- 172.21.15.43:0/2597312553 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55788b4ce3a0 0x55788b4e3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:28.466 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.467+0000 7fd5dbe00640 1 --2- 172.21.15.43:0/2597312553 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55788b4d7a90 0x55788b4ef1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:28.466 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.467+0000 7fd5daa00640 1 -- 172.21.15.43:0/2597312553 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55788b4d7a90 msgr2=0x55788b4ef1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:28.466 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.467+0000 7fd5daa00640 1 --2- 172.21.15.43:0/2597312553 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55788b4d7a90 0x55788b4ef1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.466 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.467+0000 7fd5daa00640 1 -- 172.21.15.43:0/2597312553 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55788b4ce3a0 msgr2=0x55788b4e3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:28.466 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.467+0000 7fd5daa00640 1 --2- 172.21.15.43:0/2597312553 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55788b4ce3a0 0x55788b4e3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.466 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.467+0000 7fd5daa00640 1 -- 172.21.15.43:0/2597312553 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55788b4ceaf0 con 0x55788b4cecc0 2024-03-28T22:43:28.466 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.467+0000 7fd5daa00640 1 --2- 172.21.15.43:0/2597312553 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55788b4cecc0 0x55788b4c38d0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fd5c800baf0 tx=0x7fd5c8012410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e79d36f166933606 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:28.468 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.467+0000 7fd5c7e00640 1 -- 172.21.15.43:0/2597312553 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c8013d50 con 0x55788b4cecc0 2024-03-28T22:43:28.468 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.468+0000 7fd5c7e00640 1 -- 172.21.15.43:0/2597312553 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5c8004020 con 0x55788b4cecc0 2024-03-28T22:43:28.469 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.468+0000 7fd5c7e00640 1 -- 172.21.15.43:0/2597312553 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c8005440 con 0x55788b4cecc0 2024-03-28T22:43:28.469 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.468+0000 7fd5e5384740 1 -- 172.21.15.43:0/2597312553 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55788b4cecc0 msgr2=0x55788b4c38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:28.469 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.468+0000 7fd5e5384740 1 --2- 172.21.15.43:0/2597312553 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55788b4cecc0 0x55788b4c38d0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fd5c800baf0 tx=0x7fd5c8012410 comp rx=0 tx=0).stop 2024-03-28T22:43:28.469 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.468+0000 7fd5e5384740 1 -- 172.21.15.43:0/2597312553 shutdown_connections 2024-03-28T22:43:28.469 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.468+0000 7fd5e5384740 1 --2- 172.21.15.43:0/2597312553 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55788b4d7a90 0x55788b4ef1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.469 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.468+0000 7fd5e5384740 1 --2- 172.21.15.43:0/2597312553 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55788b4cecc0 0x55788b4c38d0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.469 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.468+0000 7fd5e5384740 1 --2- 172.21.15.43:0/2597312553 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55788b4ce3a0 0x55788b4e3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.469 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.468+0000 7fd5e5384740 1 -- 172.21.15.43:0/2597312553 >> 172.21.15.43:0/2597312553 conn(0x55788b4c8cc0 msgr2=0x55788b4cfd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:28.469 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.468+0000 7fd5e5384740 1 -- 172.21.15.43:0/2597312553 shutdown_connections 2024-03-28T22:43:28.469 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.469+0000 7fd5e5384740 1 -- 172.21.15.43:0/2597312553 wait complete. 2024-03-28T22:43:28.469 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.469+0000 7fd5e5384740 1 -- start start 2024-03-28T22:43:28.469 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.469+0000 7fd5e5384740 1 Processor -- start 2024-03-28T22:43:28.469 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.469+0000 7fd5e5384740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55788b4ce3a0 0x55788b4db3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:28.470 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.470+0000 7fd5e5384740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55788b4cecc0 0x55788b4db8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:28.470 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.470+0000 7fd5e5384740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55788b4d7a90 0x55788b4dbe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:28.470 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.470+0000 7fd5e5384740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd5dc1b2220 con 0x55788b4ce3a0 2024-03-28T22:43:28.470 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.470+0000 7fd5e5384740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd5dc1b1b50 con 0x55788b4cecc0 2024-03-28T22:43:28.470 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.470+0000 7fd5e5384740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd5dc1aed90 con 0x55788b4d7a90 2024-03-28T22:43:28.470 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.470+0000 7fd5e5384740 10 client.0 _ll_register_callbacks cb 0x7fd5d4009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:28.470 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.470+0000 7fd5daa00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55788b4cecc0 0x55788b4db8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:28.470 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.470+0000 7fd5dbe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55788b4d7a90 0x55788b4dbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:28.470 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.470+0000 7fd5daa00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55788b4cecc0 0x55788b4db8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:32774/0 (socket says 172.21.15.43:32774) 2024-03-28T22:43:28.470 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.470+0000 7fd5daa00640 1 -- 172.21.15.43:0/1921388181 learned_addr learned my addr 172.21.15.43:0/1921388181 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:28.470 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.470+0000 7fd5db400640 1 --2- 172.21.15.43:0/1921388181 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55788b4ce3a0 0x55788b4db3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:28.470 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.470+0000 7fd5daa00640 1 -- 172.21.15.43:0/1921388181 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55788b4d7a90 msgr2=0x55788b4dbe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:28.470 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.470+0000 7fd5daa00640 1 --2- 172.21.15.43:0/1921388181 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55788b4d7a90 0x55788b4dbe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.470 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.470+0000 7fd5daa00640 1 -- 172.21.15.43:0/1921388181 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55788b4ce3a0 msgr2=0x55788b4db3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:28.470 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.470+0000 7fd5daa00640 1 --2- 172.21.15.43:0/1921388181 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55788b4ce3a0 0x55788b4db3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.471 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.470+0000 7fd5daa00640 1 -- 172.21.15.43:0/1921388181 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55788b5552d0 con 0x55788b4cecc0 2024-03-28T22:43:28.472 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.471+0000 7fd5daa00640 1 --2- 172.21.15.43:0/1921388181 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55788b4cecc0 0x55788b4db8e0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fd5c8004700 tx=0x7fd5c80042e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94221239c12f5642 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:28.472 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.471+0000 7fd5c5600640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:43:28.472 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.471+0000 7fd5c5600640 1 -- 172.21.15.43:0/1921388181 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c8013d50 con 0x55788b4cecc0 2024-03-28T22:43:28.472 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.471+0000 7fd5c5600640 1 -- 172.21.15.43:0/1921388181 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fd5c801f3c0 con 0x55788b4cecc0 2024-03-28T22:43:28.472 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.471+0000 7fd5e5384740 1 -- 172.21.15.43:0/1921388181 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5c8004020 con 0x55788b4cecc0 2024-03-28T22:43:28.472 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.471+0000 7fd5c5600640 1 -- 172.21.15.43:0/1921388181 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd5c801f5c0 con 0x55788b4cecc0 2024-03-28T22:43:28.472 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.471+0000 7fd5e5384740 10 client.27992 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:28.472 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.471+0000 7fd5e5384740 1 -- 172.21.15.43:0/1921388181 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55788b556610 con 0x55788b4cecc0 2024-03-28T22:43:28.473 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.472+0000 7fd5c5600640 1 -- 172.21.15.43:0/1921388181 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5c8004020 con 0x55788b4cecc0 2024-03-28T22:43:28.473 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.472+0000 7fd5c5600640 1 -- 172.21.15.43:0/1921388181 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd5c8004920 con 0x55788b4cecc0 2024-03-28T22:43:28.473 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.472+0000 7fd5e5384740 10 client.27992 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:28.473 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.472+0000 7fd5e5384740 10 client.27992 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:43:28.473 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.472+0000 7fd5e5384740 20 client.27992 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:28.473 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.472+0000 7fd5e5384740 1 --2- 172.21.15.43:0/1921388181 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55788b556db0 0x55788b5591d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:28.473 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.472+0000 7fd5e5384740 4 client.27992 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T22:43:28.473 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.472+0000 7fd5e5384740 1 -- 172.21.15.43:0/1921388181 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55788b43eb80 con 0x55788b556db0 2024-03-28T22:43:28.473 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.472+0000 7fd5db400640 1 --2- 172.21.15.43:0/1921388181 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55788b556db0 0x55788b5591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:28.473 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.473+0000 7fd5db400640 1 --2- 172.21.15.43:0/1921388181 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55788b556db0 0x55788b5591d0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e86cf8a4f2c3e80 server_cookie=f1cf0bca58e5d235 in_seq=0 out_seq=0 2024-03-28T22:43:28.473 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.473+0000 7fd5c5600640 10 client.27992 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:28.711 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.712+0000 7fd5c5600640 1 -- 172.21.15.43:0/1921388181 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55788b43eb80 con 0x55788b556db0 2024-03-28T22:43:28.711 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.712+0000 7fd5c5600640 10 client.27992 handle_command_reply: tid=0 2024-03-28T22:43:28.711 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.712+0000 7fd5e5384740 1 client.27992 shutdown 2024-03-28T22:43:28.711 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.712+0000 7fd5e5384740 1 -- 172.21.15.43:0/1921388181 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55788b4cecc0 msgr2=0x55788b4db8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:28.711 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.712+0000 7fd5e5384740 1 --2- 172.21.15.43:0/1921388181 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55788b4cecc0 0x55788b4db8e0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fd5c8004700 tx=0x7fd5c80042e0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.711 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.712+0000 7fd5e5384740 1 -- 172.21.15.43:0/1921388181 shutdown_connections 2024-03-28T22:43:28.711 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.712+0000 7fd5e5384740 1 --2- 172.21.15.43:0/1921388181 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55788b556db0 0x55788b5591d0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.711 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.712+0000 7fd5e5384740 1 --2- 172.21.15.43:0/1921388181 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55788b4cecc0 0x55788b4db8e0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.712 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.712+0000 7fd5e5384740 1 --2- 172.21.15.43:0/1921388181 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55788b4d7a90 0x55788b4dbe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.712 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.712+0000 7fd5e5384740 1 --2- 172.21.15.43:0/1921388181 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55788b4ce3a0 0x55788b4db3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.712 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.712+0000 7fd5c5600640 0 client.27992 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:28.712 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.712+0000 7fd5e5384740 1 -- 172.21.15.43:0/1921388181 >> 172.21.15.43:0/1921388181 conn(0x55788b4c8cc0 msgr2=0x55788b4d4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:28.712 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.712+0000 7fd5e5384740 1 -- 172.21.15.43:0/1921388181 shutdown_connections 2024-03-28T22:43:28.712 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.712+0000 7fd5e5384740 1 -- 172.21.15.43:0/1921388181 wait complete. 2024-03-28T22:43:28.712 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.712+0000 7fd5e5384740 20 client.27992 trim_cache size 0 max 16384 2024-03-28T22:43:28.712 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:{ 2024-03-28T22:43:28.712 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "message": "", 2024-03-28T22:43:28.713 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "return_code": 0 2024-03-28T22:43:28.713 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:} 2024-03-28T22:43:28.713 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.713+0000 7fd5e2600640 1 -- 172.21.15.43:0/2491547709 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd5c00a2010 msgr2=0x7fd5c00a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:28.713 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.713+0000 7fd5e2600640 1 --2- 172.21.15.43:0/2491547709 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd5c00a2010 0x7fd5c00a44d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4011630 tx=0x7fd5d4009500 comp rx=0 tx=0).stop 2024-03-28T22:43:28.713 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.713+0000 7fd5e2600640 1 -- 172.21.15.43:0/2491547709 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd5dc103930 msgr2=0x7fd5dc19a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:28.713 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.713+0000 7fd5e2600640 1 --2- 172.21.15.43:0/2491547709 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd5dc103930 0x7fd5dc19a1e0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc005ab0 tx=0x7fd5cc0042e0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.713 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.713+0000 7fd5e2600640 1 -- 172.21.15.43:0/2491547709 shutdown_connections 2024-03-28T22:43:28.713 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.713+0000 7fd5e2600640 1 --2- 172.21.15.43:0/2491547709 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd5c00a2010 0x7fd5c00a44d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.714 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.713+0000 7fd5e2600640 1 --2- 172.21.15.43:0/2491547709 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd5dc109940 0x7fd5dc19eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.714 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.713+0000 7fd5e2600640 1 --2- 172.21.15.43:0/2491547709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd5dc1042f0 0x7fd5dc19a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.714 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.713+0000 7fd5e2600640 1 --2- 172.21.15.43:0/2491547709 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd5dc103930 0x7fd5dc19a1e0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:28.714 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.713+0000 7fd5e2600640 1 -- 172.21.15.43:0/2491547709 >> 172.21.15.43:0/2491547709 conn(0x7fd5dc0ff400 msgr2=0x7fd5dc0ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:28.714 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.714+0000 7fd5e2600640 1 -- 172.21.15.43:0/2491547709 shutdown_connections 2024-03-28T22:43:28.714 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:28.714+0000 7fd5e2600640 1 -- 172.21.15.43:0/2491547709 wait complete. 2024-03-28T22:43:29.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:28 smithi043 ceph-mon[31823]: pgmap v193: 129 pgs: 129 active+clean; 52 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 259 op/s 2024-03-28T22:43:29.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:28 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:43:29.374 INFO:tasks.workunit.client.0.smithi043.stdout: 1 430881 7.35 MB/sec execute 180 sec latency 70.519 ms 2024-03-28T22:43:29.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:28 smithi172 ceph-mon[36566]: pgmap v193: 129 pgs: 129 active+clean; 52 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 259 op/s 2024-03-28T22:43:29.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:28 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:43:29.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:28 smithi082 ceph-mon[36471]: pgmap v193: 129 pgs: 129 active+clean; 52 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.8 MiB/s wr, 259 op/s 2024-03-28T22:43:29.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:28 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:43:30.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:29 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:43:30.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:29 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:43:30.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:29 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:43:30.375 INFO:tasks.workunit.client.0.smithi043.stdout: 1 432730 7.37 MB/sec execute 181 sec latency 23.534 ms 2024-03-28T22:43:30.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:29 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:43:30.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:29 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:43:30.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:29 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:43:30.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:29 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:43:30.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:29 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:43:30.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:29 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:43:31.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:30 smithi043 ceph-mon[31823]: pgmap v194: 129 pgs: 129 active+clean; 52 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 144 op/s 2024-03-28T22:43:31.376 INFO:tasks.workunit.client.0.smithi043.stdout: 1 433668 7.35 MB/sec execute 182 sec latency 20.475 ms 2024-03-28T22:43:31.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:30 smithi172 ceph-mon[36566]: pgmap v194: 129 pgs: 129 active+clean; 52 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 144 op/s 2024-03-28T22:43:31.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:30 smithi082 ceph-mon[36471]: pgmap v194: 129 pgs: 129 active+clean; 52 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 144 op/s 2024-03-28T22:43:32.376 INFO:tasks.workunit.client.0.smithi043.stdout: 1 434857 7.33 MB/sec execute 183 sec latency 45.313 ms 2024-03-28T22:43:33.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:33 smithi043 ceph-mon[31823]: pgmap v195: 129 pgs: 129 active+clean; 52 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 144 op/s 2024-03-28T22:43:33.376 INFO:tasks.workunit.client.0.smithi043.stdout: 1 436554 7.37 MB/sec execute 184 sec latency 38.348 ms 2024-03-28T22:43:33.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:33 smithi172 ceph-mon[36566]: pgmap v195: 129 pgs: 129 active+clean; 52 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 144 op/s 2024-03-28T22:43:33.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:33 smithi082 ceph-mon[36471]: pgmap v195: 129 pgs: 129 active+clean; 52 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 144 op/s 2024-03-28T22:43:34.376 INFO:tasks.workunit.client.0.smithi043.stdout: 1 437560 7.34 MB/sec execute 185 sec latency 20.806 ms 2024-03-28T22:43:35.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:35 smithi043 ceph-mon[31823]: pgmap v196: 129 pgs: 129 active+clean; 48 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 296 op/s 2024-03-28T22:43:35.376 INFO:tasks.workunit.client.0.smithi043.stdout: 1 438909 7.33 MB/sec execute 186 sec latency 41.956 ms 2024-03-28T22:43:35.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:35 smithi172 ceph-mon[36566]: pgmap v196: 129 pgs: 129 active+clean; 48 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 296 op/s 2024-03-28T22:43:35.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:35 smithi082 ceph-mon[36471]: pgmap v196: 129 pgs: 129 active+clean; 48 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 296 op/s 2024-03-28T22:43:36.376 INFO:tasks.workunit.client.0.smithi043.stdout: 1 440449 7.35 MB/sec execute 187 sec latency 14.034 ms 2024-03-28T22:43:37.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:37 smithi043 ceph-mon[31823]: pgmap v197: 129 pgs: 129 active+clean; 48 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 220 op/s 2024-03-28T22:43:37.376 INFO:tasks.workunit.client.0.smithi043.stdout: 1 441547 7.34 MB/sec execute 188 sec latency 48.852 ms 2024-03-28T22:43:37.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:37 smithi172 ceph-mon[36566]: pgmap v197: 129 pgs: 129 active+clean; 48 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 220 op/s 2024-03-28T22:43:37.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:37 smithi082 ceph-mon[36471]: pgmap v197: 129 pgs: 129 active+clean; 48 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 220 op/s 2024-03-28T22:43:38.377 INFO:tasks.workunit.client.0.smithi043.stdout: 1 443192 7.35 MB/sec execute 189 sec latency 36.984 ms 2024-03-28T22:43:39.377 INFO:tasks.workunit.client.0.smithi043.stdout: 1 444006 7.34 MB/sec execute 190 sec latency 18.675 ms 2024-03-28T22:43:39.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:39 smithi172 ceph-mon[36566]: pgmap v198: 129 pgs: 129 active+clean; 53 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.5 MiB/s wr, 288 op/s 2024-03-28T22:43:39.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:39 smithi082 ceph-mon[36471]: pgmap v198: 129 pgs: 129 active+clean; 53 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.5 MiB/s wr, 288 op/s 2024-03-28T22:43:39.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:39 smithi043 ceph-mon[31823]: pgmap v198: 129 pgs: 129 active+clean; 53 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.5 MiB/s wr, 288 op/s 2024-03-28T22:43:40.377 INFO:tasks.workunit.client.0.smithi043.stdout: 1 444963 7.32 MB/sec execute 191 sec latency 63.356 ms 2024-03-28T22:43:41.377 INFO:tasks.workunit.client.0.smithi043.stdout: 1 446318 7.32 MB/sec execute 192 sec latency 50.180 ms 2024-03-28T22:43:41.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:41 smithi172 ceph-mon[36566]: pgmap v199: 129 pgs: 129 active+clean; 53 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 220 op/s 2024-03-28T22:43:41.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:41 smithi082 ceph-mon[36471]: pgmap v199: 129 pgs: 129 active+clean; 53 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 220 op/s 2024-03-28T22:43:41.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:41 smithi043 ceph-mon[31823]: pgmap v199: 129 pgs: 129 active+clean; 53 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 220 op/s 2024-03-28T22:43:41.857 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.858+0000 7fd3dbe00640 1 -- 172.21.15.43:0/2468854106 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd3d4068dd0 msgr2=0x7fd3d40691b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:41.857 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.858+0000 7fd3dbe00640 1 --2- 172.21.15.43:0/2468854106 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd3d4068dd0 0x7fd3d40691b0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8009ca0 tx=0x7fd3c802c180 comp rx=0 tx=0).stop 2024-03-28T22:43:41.857 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.858+0000 7fd3dbe00640 1 -- 172.21.15.43:0/2468854106 shutdown_connections 2024-03-28T22:43:41.857 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.858+0000 7fd3dbe00640 1 --2- 172.21.15.43:0/2468854106 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3d410e450 0x7fd3d4110840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:41.857 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.858+0000 7fd3dbe00640 1 --2- 172.21.15.43:0/2468854106 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3d40696f0 0x7fd3d410df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:41.858 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.858+0000 7fd3dbe00640 1 --2- 172.21.15.43:0/2468854106 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd3d4068dd0 0x7fd3d40691b0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:41.858 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.858+0000 7fd3dbe00640 1 -- 172.21.15.43:0/2468854106 >> 172.21.15.43:0/2468854106 conn(0x7fd3d40774a0 msgr2=0x7fd3d40778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:41.858 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.859+0000 7fd3dbe00640 1 -- 172.21.15.43:0/2468854106 shutdown_connections 2024-03-28T22:43:41.858 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.859+0000 7fd3dbe00640 1 -- 172.21.15.43:0/2468854106 wait complete. 2024-03-28T22:43:41.858 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.860+0000 7fd3dbe00640 1 Processor -- start 2024-03-28T22:43:41.858 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.860+0000 7fd3dbe00640 1 -- start start 2024-03-28T22:43:41.859 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.860+0000 7fd3dbe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3d4068dd0 0x7fd3d419a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:41.859 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.860+0000 7fd3dbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3d40696f0 0x7fd3d419a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:41.859 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.860+0000 7fd3dbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd3d410e450 0x7fd3d419e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:41.859 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.860+0000 7fd3dbe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d4113d20 con 0x7fd3d40696f0 2024-03-28T22:43:41.859 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.860+0000 7fd3dbe00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d4113ba0 con 0x7fd3d410e450 2024-03-28T22:43:41.859 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.860+0000 7fd3dbe00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d4113ea0 con 0x7fd3d4068dd0 2024-03-28T22:43:41.860 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.861+0000 7fd3d9400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3d4068dd0 0x7fd3d419a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:41.860 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.861+0000 7fd3d8a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3d40696f0 0x7fd3d419a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:41.861 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.861+0000 7fd3d9e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd3d410e450 0x7fd3d419e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:41.861 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.861+0000 7fd3d9400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3d4068dd0 0x7fd3d419a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:45982/0 (socket says 172.21.15.43:45982) 2024-03-28T22:43:41.861 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.861+0000 7fd3d9400640 1 -- 172.21.15.43:0/1466793144 learned_addr learned my addr 172.21.15.43:0/1466793144 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:41.861 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.861+0000 7fd3d9e00640 1 -- 172.21.15.43:0/1466793144 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3d4068dd0 msgr2=0x7fd3d419a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:41.861 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.861+0000 7fd3d9e00640 1 --2- 172.21.15.43:0/1466793144 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3d4068dd0 0x7fd3d419a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:41.861 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.861+0000 7fd3d9e00640 1 -- 172.21.15.43:0/1466793144 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3d40696f0 msgr2=0x7fd3d419a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:41.861 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.861+0000 7fd3d9e00640 1 --2- 172.21.15.43:0/1466793144 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3d40696f0 0x7fd3d419a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:41.861 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.861+0000 7fd3d9e00640 1 -- 172.21.15.43:0/1466793144 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3d419f0b0 con 0x7fd3d410e450 2024-03-28T22:43:41.861 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.861+0000 7fd3d9400640 1 --2- 172.21.15.43:0/1466793144 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3d4068dd0 0x7fd3d419a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:41.861 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.861+0000 7fd3d9e00640 1 --2- 172.21.15.43:0/1466793144 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd3d410e450 0x7fd3d419e980 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc00e710 tx=0x7fd3cc00ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:41.861 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.862+0000 7fd3d2000640 1 -- 172.21.15.43:0/1466793144 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc004220 con 0x7fd3d410e450 2024-03-28T22:43:41.861 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.862+0000 7fd3d2000640 1 -- 172.21.15.43:0/1466793144 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3cc00cd90 con 0x7fd3d410e450 2024-03-28T22:43:41.861 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.862+0000 7fd3dbe00640 1 -- 172.21.15.43:0/1466793144 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3d419f3a0 con 0x7fd3d410e450 2024-03-28T22:43:41.861 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.862+0000 7fd3d2000640 1 -- 172.21.15.43:0/1466793144 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc004a00 con 0x7fd3d410e450 2024-03-28T22:43:41.862 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.862+0000 7fd3dbe00640 1 -- 172.21.15.43:0/1466793144 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3d41a6c30 con 0x7fd3d410e450 2024-03-28T22:43:41.863 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.865+0000 7fd3d2000640 1 -- 172.21.15.43:0/1466793144 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fd3cc00c8b0 con 0x7fd3d410e450 2024-03-28T22:43:41.864 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.866+0000 7fd3dea67740 1 Processor -- start 2024-03-28T22:43:41.864 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.866+0000 7fd3dea67740 1 -- start start 2024-03-28T22:43:41.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.866+0000 7fd3dea67740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c8fb05d3a0 0x55c8fb05d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:41.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.866+0000 7fd3dea67740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c8fb05dcc0 0x55c8fb066550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:41.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.866+0000 7fd3dea67740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c8fb066a90 0x55c8fb068e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:41.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.866+0000 7fd3dea67740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c8fafc0640 con 0x55c8fb05d3a0 2024-03-28T22:43:41.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.866+0000 7fd3dea67740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55c8faf69810 con 0x55c8fb05dcc0 2024-03-28T22:43:41.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.866+0000 7fd3dea67740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55c8fafbc6e0 con 0x55c8fb066a90 2024-03-28T22:43:41.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.866+0000 7fd3d9400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c8fb05dcc0 0x55c8fb066550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:41.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.867+0000 7fd3d9400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c8fb05dcc0 0x55c8fb066550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:60796/0 (socket says 172.21.15.43:60796) 2024-03-28T22:43:41.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.867+0000 7fd3d9400640 1 -- 172.21.15.43:0/389071371 learned_addr learned my addr 172.21.15.43:0/389071371 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:41.865 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.867+0000 7fd3d2000640 1 --2- 172.21.15.43:0/1466793144 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd3bc0a2010 0x7fd3bc0a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:41.866 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.867+0000 7fd3d8a00640 1 --2- 172.21.15.43:0/389071371 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c8fb05d3a0 0x55c8fb05d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:41.866 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.867+0000 7fd3d8a00640 1 --2- 172.21.15.43:0/389071371 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c8fb066a90 0x55c8fb068e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:41.867 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.867+0000 7fd3d2000640 1 -- 172.21.15.43:0/1466793144 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fd3cc0ce520 con 0x7fd3d410e450 2024-03-28T22:43:41.867 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.867+0000 7fd3d9e00640 1 --2- 172.21.15.43:0/1466793144 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd3bc0a2010 0x7fd3bc0a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:41.867 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.867+0000 7fd3d9400640 1 -- 172.21.15.43:0/389071371 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c8fb066a90 msgr2=0x55c8fb068e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:41.867 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.867+0000 7fd3d9400640 1 --2- 172.21.15.43:0/389071371 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c8fb066a90 0x55c8fb068e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:41.867 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.867+0000 7fd3d9400640 1 -- 172.21.15.43:0/389071371 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c8fb05d3a0 msgr2=0x55c8fb05d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:41.867 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.867+0000 7fd3d9400640 1 --2- 172.21.15.43:0/389071371 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c8fb05d3a0 0x55c8fb05d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:41.867 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.867+0000 7fd3d9400640 1 -- 172.21.15.43:0/389071371 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8facd5210 con 0x55c8fb05dcc0 2024-03-28T22:43:41.867 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.867+0000 7fd3d8a00640 1 --2- 172.21.15.43:0/389071371 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c8fb05d3a0 0x55c8fb05d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:43:41.867 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.868+0000 7fd3d9400640 1 --2- 172.21.15.43:0/389071371 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c8fb05dcc0 0x55c8fb066550 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8037040 tx=0x7fd3c80098a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18d9a7378f7c17af server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:41.867 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.868+0000 7fd3d9e00640 1 --2- 172.21.15.43:0/1466793144 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd3bc0a2010 0x7fd3bc0a44d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc00cfd0 tx=0x7fd3cc00cf40 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:41.867 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.868+0000 7fd3c6600640 1 -- 172.21.15.43:0/389071371 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c8039040 con 0x55c8fb05dcc0 2024-03-28T22:43:41.867 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.868+0000 7fd3c6600640 1 -- 172.21.15.43:0/389071371 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3c8003970 con 0x55c8fb05dcc0 2024-03-28T22:43:41.867 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.868+0000 7fd3c6600640 1 -- 172.21.15.43:0/389071371 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c8004040 con 0x55c8fb05dcc0 2024-03-28T22:43:41.869 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.869+0000 7fd3dea67740 1 -- 172.21.15.43:0/389071371 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c8fb05dcc0 msgr2=0x55c8fb066550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:41.870 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.869+0000 7fd3dea67740 1 --2- 172.21.15.43:0/389071371 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c8fb05dcc0 0x55c8fb066550 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8037040 tx=0x7fd3c80098a0 comp rx=0 tx=0).stop 2024-03-28T22:43:41.870 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.869+0000 7fd3dea67740 1 -- 172.21.15.43:0/389071371 shutdown_connections 2024-03-28T22:43:41.870 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.869+0000 7fd3dea67740 1 --2- 172.21.15.43:0/389071371 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c8fb05dcc0 0x55c8fb066550 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:41.870 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.869+0000 7fd3dea67740 1 --2- 172.21.15.43:0/389071371 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c8fb066a90 0x55c8fb068e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:41.870 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.869+0000 7fd3dea67740 1 --2- 172.21.15.43:0/389071371 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c8fb05d3a0 0x55c8fb05d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:41.870 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.869+0000 7fd3dea67740 1 -- 172.21.15.43:0/389071371 >> 172.21.15.43:0/389071371 conn(0x55c8fb057cc0 msgr2=0x55c8fb0580a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:41.870 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.869+0000 7fd3dea67740 1 -- 172.21.15.43:0/389071371 shutdown_connections 2024-03-28T22:43:41.870 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.869+0000 7fd3dea67740 1 -- 172.21.15.43:0/389071371 wait complete. 2024-03-28T22:43:41.870 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.870+0000 7fd3dea67740 1 -- start start 2024-03-28T22:43:41.870 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.870+0000 7fd3dea67740 1 Processor -- start 2024-03-28T22:43:41.870 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.870+0000 7fd3dea67740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c8fb05d3a0 0x55c8fb063a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:41.870 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.870+0000 7fd3dea67740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c8fb05dcc0 0x55c8fb063fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:41.870 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.870+0000 7fd3dea67740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c8fb066a90 0x55c8fb064500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:41.871 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.870+0000 7fd3dea67740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c8fb069660 con 0x55c8fb05dcc0 2024-03-28T22:43:41.871 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.870+0000 7fd3dea67740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55c8faf5ed00 con 0x55c8fb05d3a0 2024-03-28T22:43:41.871 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.870+0000 7fd3dea67740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55c8fb0697e0 con 0x55c8fb066a90 2024-03-28T22:43:41.871 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.871+0000 7fd3dea67740 10 client.0 _ll_register_callbacks cb 0x55c8faf49490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:41.872 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.871+0000 7fd3d8a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c8fb05d3a0 0x55c8fb063a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:41.872 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.871+0000 7fd3d9400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c8fb05dcc0 0x55c8fb063fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:41.872 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.871+0000 7fd3d8a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c8fb05d3a0 0x55c8fb063a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:60812/0 (socket says 172.21.15.43:60812) 2024-03-28T22:43:41.872 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.871+0000 7fd3d8a00640 1 -- 172.21.15.43:0/4001388457 learned_addr learned my addr 172.21.15.43:0/4001388457 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:41.872 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.871+0000 7fd3d8a00640 1 --2- 172.21.15.43:0/4001388457 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c8fb066a90 0x55c8fb064500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:41.872 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.871+0000 7fd3d8a00640 1 -- 172.21.15.43:0/4001388457 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c8fb066a90 msgr2=0x55c8fb064500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:41.872 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.871+0000 7fd3d8a00640 1 --2- 172.21.15.43:0/4001388457 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c8fb066a90 0x55c8fb064500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:41.872 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.871+0000 7fd3d8a00640 1 -- 172.21.15.43:0/4001388457 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c8fb05dcc0 msgr2=0x55c8fb063fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:41.872 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.871+0000 7fd3d8a00640 1 --2- 172.21.15.43:0/4001388457 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c8fb05dcc0 0x55c8fb063fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:41.872 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.871+0000 7fd3d8a00640 1 -- 172.21.15.43:0/4001388457 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8fada1620 con 0x55c8fb05d3a0 2024-03-28T22:43:41.875 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.871+0000 7fd3d8a00640 1 --2- 172.21.15.43:0/4001388457 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c8fb05d3a0 0x55c8fb063a80 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fd3c0011d10 tx=0x7fd3c0015040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b87faa71292223c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:41.875 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.871+0000 7fd3baa00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:43:41.875 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.872+0000 7fd3baa00640 1 -- 172.21.15.43:0/4001388457 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c000fe60 con 0x55c8fb05d3a0 2024-03-28T22:43:41.875 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.872+0000 7fd3baa00640 1 -- 172.21.15.43:0/4001388457 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fd3c0005020 con 0x55c8fb05d3a0 2024-03-28T22:43:41.875 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.872+0000 7fd3dea67740 1 -- 172.21.15.43:0/4001388457 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3c8003970 con 0x55c8fb05d3a0 2024-03-28T22:43:41.875 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.872+0000 7fd3baa00640 1 -- 172.21.15.43:0/4001388457 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd3c002a050 con 0x55c8fb05d3a0 2024-03-28T22:43:41.875 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.872+0000 7fd3dea67740 10 client.28004 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:41.875 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.872+0000 7fd3dea67740 1 -- 172.21.15.43:0/4001388457 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c8fb075540 con 0x55c8fb05d3a0 2024-03-28T22:43:41.875 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.872+0000 7fd3baa00640 1 -- 172.21.15.43:0/4001388457 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3c8003970 con 0x55c8fb05d3a0 2024-03-28T22:43:41.875 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.872+0000 7fd3baa00640 1 -- 172.21.15.43:0/4001388457 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd3c0022420 con 0x55c8fb05d3a0 2024-03-28T22:43:41.875 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.873+0000 7fd3dea67740 10 client.28004 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:41.875 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.873+0000 7fd3dea67740 10 client.28004 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:43:41.875 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.873+0000 7fd3dea67740 20 client.28004 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:41.875 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.873+0000 7fd3dea67740 1 --2- 172.21.15.43:0/4001388457 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c8fb0769b0 0x55c8fb078de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:41.876 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.873+0000 7fd3dea67740 4 client.28004 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:43:41.876 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.873+0000 7fd3dea67740 1 -- 172.21.15.43:0/4001388457 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8fafd4030 con 0x55c8fb0769b0 2024-03-28T22:43:41.876 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.873+0000 7fd3d9400640 1 --2- 172.21.15.43:0/4001388457 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c8fb0769b0 0x55c8fb078de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:41.876 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.874+0000 7fd3d9400640 1 --2- 172.21.15.43:0/4001388457 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c8fb0769b0 0x55c8fb078de0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=933144fcc28dcaa9 server_cookie=3cd708b69f9eeb9b in_seq=0 out_seq=0 2024-03-28T22:43:41.876 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.874+0000 7fd3baa00640 10 client.28004 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:41.876 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.874+0000 7fd3baa00640 1 -- 172.21.15.43:0/4001388457 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c8fafd4030 con 0x55c8fb0769b0 2024-03-28T22:43:41.876 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.874+0000 7fd3baa00640 10 client.28004 handle_command_reply: tid=0 2024-03-28T22:43:41.876 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.874+0000 7fd3dea67740 1 client.28004 shutdown 2024-03-28T22:43:41.876 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.875+0000 7fd3dea67740 1 -- 172.21.15.43:0/4001388457 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c8fb05d3a0 msgr2=0x55c8fb063a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:41.876 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.875+0000 7fd3dea67740 1 --2- 172.21.15.43:0/4001388457 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c8fb05d3a0 0x55c8fb063a80 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fd3c0011d10 tx=0x7fd3c0015040 comp rx=0 tx=0).stop 2024-03-28T22:43:41.876 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.875+0000 7fd3dea67740 1 -- 172.21.15.43:0/4001388457 shutdown_connections 2024-03-28T22:43:41.876 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.875+0000 7fd3dea67740 1 --2- 172.21.15.43:0/4001388457 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c8fb066a90 0x55c8fb064500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:41.876 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.875+0000 7fd3dea67740 1 --2- 172.21.15.43:0/4001388457 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c8fb05dcc0 0x55c8fb063fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:41.876 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.875+0000 7fd3dea67740 1 --2- 172.21.15.43:0/4001388457 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c8fb0769b0 0x55c8fb078de0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:41.877 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.875+0000 7fd3dea67740 1 --2- 172.21.15.43:0/4001388457 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c8fb05d3a0 0x55c8fb063a80 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:41.877 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.875+0000 7fd3dea67740 1 -- 172.21.15.43:0/4001388457 >> 172.21.15.43:0/4001388457 conn(0x55c8fb057cc0 msgr2=0x55c8fb059a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:41.877 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.875+0000 7fd3baa00640 0 client.28004 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:41.877 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.875+0000 7fd3dea67740 1 -- 172.21.15.43:0/4001388457 shutdown_connections 2024-03-28T22:43:41.877 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.875+0000 7fd3dea67740 1 -- 172.21.15.43:0/4001388457 wait complete. 2024-03-28T22:43:41.877 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.875+0000 7fd3dea67740 20 client.28004 trim_cache size 0 max 16384 2024-03-28T22:43:41.891 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.892+0000 7fd3dea67740 1 Processor -- start 2024-03-28T22:43:41.891 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.892+0000 7fd3dea67740 1 -- start start 2024-03-28T22:43:41.891 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.892+0000 7fd3dea67740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c8fb05d3a0 0x55c8fb072400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:41.891 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.892+0000 7fd3dea67740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c8fb05dcc0 0x55c8fb05e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:41.891 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.892+0000 7fd3dea67740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c8fb066a90 0x55c8fb07e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:41.891 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.892+0000 7fd3dea67740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d41ac8d0 con 0x55c8fb066a90 2024-03-28T22:43:41.891 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.892+0000 7fd3dea67740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d41ae410 con 0x55c8fb05d3a0 2024-03-28T22:43:41.891 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.892+0000 7fd3dea67740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d41acfa0 con 0x55c8fb05dcc0 2024-03-28T22:43:41.892 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.893+0000 7fd3d9400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c8fb05dcc0 0x55c8fb05e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:41.893 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.893+0000 7fd3d8a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c8fb05d3a0 0x55c8fb072400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:41.893 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.893+0000 7fd3d9400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c8fb05dcc0 0x55c8fb05e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:46018/0 (socket says 172.21.15.43:46018) 2024-03-28T22:43:41.893 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.893+0000 7fd3d9400640 1 -- 172.21.15.43:0/1552455950 learned_addr learned my addr 172.21.15.43:0/1552455950 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:41.893 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.893+0000 7fd3d8a00640 1 --2- 172.21.15.43:0/1552455950 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c8fb05d3a0 0x55c8fb072400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:60826/0 (socket says 172.21.15.43:60826) 2024-03-28T22:43:41.893 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.893+0000 7fd3d9400640 1 -- 172.21.15.43:0/1552455950 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c8fb05d3a0 msgr2=0x55c8fb072400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:41.893 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.893+0000 7fd3d9400640 1 --2- 172.21.15.43:0/1552455950 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c8fb05d3a0 0x55c8fb072400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:41.893 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.893+0000 7fd3d9400640 1 -- 172.21.15.43:0/1552455950 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c8fb066a90 msgr2=0x55c8fb07e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:41.893 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.893+0000 7fd3d9400640 1 --2- 172.21.15.43:0/1552455950 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c8fb066a90 0x55c8fb07e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:41.893 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.893+0000 7fd3d9400640 1 -- 172.21.15.43:0/1552455950 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8fb072940 con 0x55c8fb05dcc0 2024-03-28T22:43:41.893 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.893+0000 7fd3d8a00640 1 --2- 172.21.15.43:0/1552455950 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c8fb05d3a0 0x55c8fb072400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:43:41.893 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.894+0000 7fd3d9400640 1 --2- 172.21.15.43:0/1552455950 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c8fb05dcc0 0x55c8fb05e220 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fd3c802c790 tx=0x7fd3c8046040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf7dfe5bd3976c35 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:41.893 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.894+0000 7fd3c5c00640 1 -- 172.21.15.43:0/1552455950 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c8039040 con 0x55c8fb05dcc0 2024-03-28T22:43:41.893 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.894+0000 7fd3c5c00640 1 -- 172.21.15.43:0/1552455950 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3c8034ad0 con 0x55c8fb05dcc0 2024-03-28T22:43:41.893 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.894+0000 7fd3c5c00640 1 -- 172.21.15.43:0/1552455950 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c8003cf0 con 0x55c8fb05dcc0 2024-03-28T22:43:41.898 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.895+0000 7fd3dea67740 1 -- 172.21.15.43:0/1552455950 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c8fb05dcc0 msgr2=0x55c8fb05e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:41.898 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.895+0000 7fd3dea67740 1 --2- 172.21.15.43:0/1552455950 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c8fb05dcc0 0x55c8fb05e220 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fd3c802c790 tx=0x7fd3c8046040 comp rx=0 tx=0).stop 2024-03-28T22:43:41.898 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.895+0000 7fd3dea67740 1 -- 172.21.15.43:0/1552455950 shutdown_connections 2024-03-28T22:43:41.898 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.895+0000 7fd3dea67740 1 --2- 172.21.15.43:0/1552455950 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c8fb066a90 0x55c8fb07e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:41.898 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.895+0000 7fd3dea67740 1 --2- 172.21.15.43:0/1552455950 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c8fb05dcc0 0x55c8fb05e220 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:41.898 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.895+0000 7fd3dea67740 1 --2- 172.21.15.43:0/1552455950 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c8fb05d3a0 0x55c8fb072400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:41.898 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.895+0000 7fd3dea67740 1 -- 172.21.15.43:0/1552455950 >> 172.21.15.43:0/1552455950 conn(0x55c8fb057cc0 msgr2=0x55c8fb0580a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:41.899 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.895+0000 7fd3dea67740 1 -- 172.21.15.43:0/1552455950 shutdown_connections 2024-03-28T22:43:41.899 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.895+0000 7fd3dea67740 1 -- 172.21.15.43:0/1552455950 wait complete. 2024-03-28T22:43:41.899 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.896+0000 7fd3dea67740 1 -- start start 2024-03-28T22:43:41.899 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.896+0000 7fd3dea67740 1 Processor -- start 2024-03-28T22:43:41.899 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.896+0000 7fd3dea67740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c8fb05d3a0 0x55c8fb06a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:41.899 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.896+0000 7fd3dea67740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c8fb05dcc0 0x55c8fb06a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:41.899 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.896+0000 7fd3dea67740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c8fb066a90 0x55c8fb06ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:41.899 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.896+0000 7fd3dea67740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d41add40 con 0x55c8fb05dcc0 2024-03-28T22:43:41.899 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.896+0000 7fd3dea67740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d41ad670 con 0x55c8fb05d3a0 2024-03-28T22:43:41.899 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.896+0000 7fd3dea67740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d41aa8f0 con 0x55c8fb066a90 2024-03-28T22:43:41.899 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.896+0000 7fd3dea67740 10 client.0 _ll_register_callbacks cb 0x7fd3c0009ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:41.899 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.897+0000 7fd3d8a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c8fb05d3a0 0x55c8fb06a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:41.899 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.897+0000 7fd3d8a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c8fb05d3a0 0x55c8fb06a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:60828/0 (socket says 172.21.15.43:60828) 2024-03-28T22:43:41.899 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.897+0000 7fd3d8a00640 1 -- 172.21.15.43:0/3302273809 learned_addr learned my addr 172.21.15.43:0/3302273809 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:41.899 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.897+0000 7fd3d8a00640 1 --2- 172.21.15.43:0/3302273809 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c8fb066a90 0x55c8fb06ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:41.902 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.897+0000 7fd3d9400640 1 --2- 172.21.15.43:0/3302273809 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c8fb05dcc0 0x55c8fb06a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:41.902 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.897+0000 7fd3d8a00640 1 -- 172.21.15.43:0/3302273809 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c8fb066a90 msgr2=0x55c8fb06ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:41.902 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.897+0000 7fd3d8a00640 1 --2- 172.21.15.43:0/3302273809 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c8fb066a90 0x55c8fb06ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:41.902 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.897+0000 7fd3d8a00640 1 -- 172.21.15.43:0/3302273809 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c8fb05dcc0 msgr2=0x55c8fb06a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:41.902 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.897+0000 7fd3d8a00640 1 --2- 172.21.15.43:0/3302273809 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c8fb05dcc0 0x55c8fb06a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:41.902 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.897+0000 7fd3d8a00640 1 -- 172.21.15.43:0/3302273809 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8fb0dbdd0 con 0x55c8fb05d3a0 2024-03-28T22:43:41.902 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.897+0000 7fd3d8a00640 1 --2- 172.21.15.43:0/3302273809 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c8fb05d3a0 0x55c8fb06a410 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fd3c0022fd0 tx=0x7fd3c0015040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccccdbe387c05cd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:41.902 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.897+0000 7fd3baa00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:43:41.902 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.898+0000 7fd3baa00640 1 -- 172.21.15.43:0/3302273809 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c0022c20 con 0x55c8fb05d3a0 2024-03-28T22:43:41.902 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.898+0000 7fd3baa00640 1 -- 172.21.15.43:0/3302273809 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fd3c0022420 con 0x55c8fb05d3a0 2024-03-28T22:43:41.902 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.898+0000 7fd3dea67740 1 -- 172.21.15.43:0/3302273809 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3c8034ad0 con 0x55c8fb05d3a0 2024-03-28T22:43:41.902 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.898+0000 7fd3baa00640 1 -- 172.21.15.43:0/3302273809 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd3c0022620 con 0x55c8fb05d3a0 2024-03-28T22:43:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.898+0000 7fd3dea67740 10 client.28010 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.898+0000 7fd3dea67740 1 -- 172.21.15.43:0/3302273809 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c8fb0dd0a0 con 0x55c8fb05d3a0 2024-03-28T22:43:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.898+0000 7fd3baa00640 1 -- 172.21.15.43:0/3302273809 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3c8034ad0 con 0x55c8fb05d3a0 2024-03-28T22:43:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.898+0000 7fd3baa00640 1 -- 172.21.15.43:0/3302273809 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd3c0003a30 con 0x55c8fb05d3a0 2024-03-28T22:43:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.898+0000 7fd3dea67740 10 client.28010 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.899+0000 7fd3dea67740 10 client.28010 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:43:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.899+0000 7fd3dea67740 20 client.28010 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.899+0000 7fd3dea67740 1 --2- 172.21.15.43:0/3302273809 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c8fb0dd700 0x55c8fb0dfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.899+0000 7fd3dea67740 4 client.28010 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T22:43:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.899+0000 7fd3dea67740 1 -- 172.21.15.43:0/3302273809 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c8fafcdb80 con 0x55c8fb0dd700 2024-03-28T22:43:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.899+0000 7fd3d9400640 1 --2- 172.21.15.43:0/3302273809 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c8fb0dd700 0x55c8fb0dfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.899+0000 7fd3d9400640 1 --2- 172.21.15.43:0/3302273809 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c8fb0dd700 0x55c8fb0dfb20 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=853a4bd6d7ab03b4 server_cookie=75a2981534d4b7a0 in_seq=0 out_seq=0 2024-03-28T22:43:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:41.899+0000 7fd3baa00640 10 client.28010 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:42.091 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:42.092+0000 7fd3baa00640 1 -- 172.21.15.43:0/3302273809 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c8fafcdb80 con 0x55c8fb0dd700 2024-03-28T22:43:42.091 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:42.092+0000 7fd3baa00640 10 client.28010 handle_command_reply: tid=0 2024-03-28T22:43:42.091 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:42.092+0000 7fd3dea67740 1 client.28010 shutdown 2024-03-28T22:43:42.091 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:42.092+0000 7fd3dea67740 1 -- 172.21.15.43:0/3302273809 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c8fb05d3a0 msgr2=0x55c8fb06a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:42.091 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:42.092+0000 7fd3dea67740 1 --2- 172.21.15.43:0/3302273809 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c8fb05d3a0 0x55c8fb06a410 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fd3c0022fd0 tx=0x7fd3c0015040 comp rx=0 tx=0).stop 2024-03-28T22:43:42.091 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:42.092+0000 7fd3dea67740 1 -- 172.21.15.43:0/3302273809 shutdown_connections 2024-03-28T22:43:42.091 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:42.092+0000 7fd3dea67740 1 --2- 172.21.15.43:0/3302273809 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c8fb066a90 0x55c8fb06ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:42.091 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:42.092+0000 7fd3dea67740 1 --2- 172.21.15.43:0/3302273809 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c8fb05dcc0 0x55c8fb06a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:42.091 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:42.092+0000 7fd3dea67740 1 --2- 172.21.15.43:0/3302273809 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c8fb0dd700 0x55c8fb0dfb20 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:42.091 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:42.092+0000 7fd3dea67740 1 --2- 172.21.15.43:0/3302273809 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c8fb05d3a0 0x55c8fb06a410 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:42.091 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:42.092+0000 7fd3dea67740 1 -- 172.21.15.43:0/3302273809 >> 172.21.15.43:0/3302273809 conn(0x55c8fb057cc0 msgr2=0x55c8fb05e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:42.092 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:42.092+0000 7fd3baa00640 0 client.28010 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:42.092 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:42.092+0000 7fd3dea67740 1 -- 172.21.15.43:0/3302273809 shutdown_connections 2024-03-28T22:43:42.092 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:42.092+0000 7fd3dea67740 1 -- 172.21.15.43:0/3302273809 wait complete. 2024-03-28T22:43:42.092 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:42.092+0000 7fd3dea67740 20 client.28010 trim_cache size 0 max 16384 2024-03-28T22:43:42.092 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:{ 2024-03-28T22:43:42.092 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "message": "", 2024-03-28T22:43:42.092 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "return_code": 0 2024-03-28T22:43:42.092 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:} 2024-03-28T22:43:42.093 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:42.093+0000 7fd3dbe00640 1 -- 172.21.15.43:0/1466793144 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd3bc0a2010 msgr2=0x7fd3bc0a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:42.093 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:42.093+0000 7fd3dbe00640 1 --2- 172.21.15.43:0/1466793144 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd3bc0a2010 0x7fd3bc0a44d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc00cfd0 tx=0x7fd3cc00cf40 comp rx=0 tx=0).stop 2024-03-28T22:43:42.093 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:42.093+0000 7fd3dbe00640 1 -- 172.21.15.43:0/1466793144 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd3d410e450 msgr2=0x7fd3d419e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:42.093 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:42.093+0000 7fd3dbe00640 1 --2- 172.21.15.43:0/1466793144 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd3d410e450 0x7fd3d419e980 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc00e710 tx=0x7fd3cc00ebd0 comp rx=0 tx=0).stop 2024-03-28T22:43:42.093 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:42.093+0000 7fd3dbe00640 1 -- 172.21.15.43:0/1466793144 shutdown_connections 2024-03-28T22:43:42.093 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:42.093+0000 7fd3dbe00640 1 --2- 172.21.15.43:0/1466793144 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd3bc0a2010 0x7fd3bc0a44d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:42.094 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:42.093+0000 7fd3dbe00640 1 --2- 172.21.15.43:0/1466793144 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd3d410e450 0x7fd3d419e980 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:42.094 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:42.093+0000 7fd3dbe00640 1 --2- 172.21.15.43:0/1466793144 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3d40696f0 0x7fd3d419a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:42.094 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:42.093+0000 7fd3dbe00640 1 --2- 172.21.15.43:0/1466793144 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3d4068dd0 0x7fd3d419a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:42.094 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:42.093+0000 7fd3dbe00640 1 -- 172.21.15.43:0/1466793144 >> 172.21.15.43:0/1466793144 conn(0x7fd3d40774a0 msgr2=0x7fd3d410f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:42.094 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:42.093+0000 7fd3dbe00640 1 -- 172.21.15.43:0/1466793144 shutdown_connections 2024-03-28T22:43:42.094 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:42.093+0000 7fd3dbe00640 1 -- 172.21.15.43:0/1466793144 wait complete. 2024-03-28T22:43:42.377 INFO:tasks.workunit.client.0.smithi043.stdout: 1 447329 7.32 MB/sec execute 193 sec latency 128.679 ms 2024-03-28T22:43:43.377 INFO:tasks.workunit.client.0.smithi043.stdout: 1 448204 7.29 MB/sec execute 194 sec latency 19.528 ms 2024-03-28T22:43:43.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:43 smithi172 ceph-mon[36566]: pgmap v200: 129 pgs: 129 active+clean; 53 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 220 op/s 2024-03-28T22:43:43.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:43 smithi082 ceph-mon[36471]: pgmap v200: 129 pgs: 129 active+clean; 53 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 220 op/s 2024-03-28T22:43:43.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:43 smithi043 ceph-mon[31823]: pgmap v200: 129 pgs: 129 active+clean; 53 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 220 op/s 2024-03-28T22:43:44.377 INFO:tasks.workunit.client.0.smithi043.stdout: 1 449414 7.29 MB/sec execute 195 sec latency 37.485 ms 2024-03-28T22:43:45.377 INFO:tasks.workunit.client.0.smithi043.stdout: 1 450902 7.31 MB/sec execute 196 sec latency 40.181 ms 2024-03-28T22:43:45.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:45 smithi172 ceph-mon[36566]: pgmap v201: 129 pgs: 129 active+clean; 47 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 357 op/s 2024-03-28T22:43:45.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:45 smithi082 ceph-mon[36471]: pgmap v201: 129 pgs: 129 active+clean; 47 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 357 op/s 2024-03-28T22:43:45.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:45 smithi043 ceph-mon[31823]: pgmap v201: 129 pgs: 129 active+clean; 47 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.2 MiB/s wr, 357 op/s 2024-03-28T22:43:46.378 INFO:tasks.workunit.client.0.smithi043.stdout: 1 452025 7.30 MB/sec execute 197 sec latency 25.602 ms 2024-03-28T22:43:47.378 INFO:tasks.workunit.client.0.smithi043.stdout: 1 453499 7.30 MB/sec execute 198 sec latency 40.332 ms 2024-03-28T22:43:47.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:47 smithi172 ceph-mon[36566]: pgmap v202: 129 pgs: 129 active+clean; 47 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 205 op/s 2024-03-28T22:43:47.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:47 smithi082 ceph-mon[36471]: pgmap v202: 129 pgs: 129 active+clean; 47 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 205 op/s 2024-03-28T22:43:47.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:47 smithi043 ceph-mon[31823]: pgmap v202: 129 pgs: 129 active+clean; 47 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 205 op/s 2024-03-28T22:43:47.985 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:43:47.985 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:43:48.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.234+0000 7f2310a00640 1 -- 172.21.15.43:0/4288420334 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f230c107f60 msgr2=0x7f230c108340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:48.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.234+0000 7f2310a00640 1 --2- 172.21.15.43:0/4288420334 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f230c107f60 0x7f230c108340 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f22fc009900 tx=0x7f22fc02bc30 comp rx=0 tx=0).stop 2024-03-28T22:43:48.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.234+0000 7f2310a00640 1 -- 172.21.15.43:0/4288420334 shutdown_connections 2024-03-28T22:43:48.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.234+0000 7f2310a00640 1 --2- 172.21.15.43:0/4288420334 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f230c10d030 0x7f230c114c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.234+0000 7f2310a00640 1 --2- 172.21.15.43:0/4288420334 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f230c108910 0x7f230c10c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.234+0000 7f2310a00640 1 --2- 172.21.15.43:0/4288420334 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f230c107f60 0x7f230c108340 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.234+0000 7f2310a00640 1 -- 172.21.15.43:0/4288420334 >> 172.21.15.43:0/4288420334 conn(0x7f230c075c90 msgr2=0x7f230c0780b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:48.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.234+0000 7f2310a00640 1 -- 172.21.15.43:0/4288420334 shutdown_connections 2024-03-28T22:43:48.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.234+0000 7f2310a00640 1 -- 172.21.15.43:0/4288420334 wait complete. 2024-03-28T22:43:48.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.235+0000 7f2310a00640 1 Processor -- start 2024-03-28T22:43:48.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.235+0000 7f2310a00640 1 -- start start 2024-03-28T22:43:48.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.235+0000 7f2310a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f230c107f60 0x7f230c19e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:48.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.235+0000 7f2310a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f230c10d030 0x7f230c19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:48.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.235+0000 7f2310a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f230c1a2f10 0x7f230c1a33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:48.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.235+0000 7f2310a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f230c118130 con 0x7f230c107f60 2024-03-28T22:43:48.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.235+0000 7f2310a00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f230c117fb0 con 0x7f230c1a2f10 2024-03-28T22:43:48.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.235+0000 7f2310a00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f230c1182b0 con 0x7f230c10d030 2024-03-28T22:43:48.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.236+0000 7f2309400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f230c10d030 0x7f230c19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:48.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.236+0000 7f2309e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f230c107f60 0x7f230c19e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:48.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.236+0000 7f2309400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f230c10d030 0x7f230c19eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:60818/0 (socket says 172.21.15.43:60818) 2024-03-28T22:43:48.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.236+0000 7f2309400640 1 -- 172.21.15.43:0/4108106246 learned_addr learned my addr 172.21.15.43:0/4108106246 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:48.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.236+0000 7f230a800640 1 --2- 172.21.15.43:0/4108106246 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f230c1a2f10 0x7f230c1a33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:48.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.236+0000 7f2309e00640 1 -- 172.21.15.43:0/4108106246 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f230c10d030 msgr2=0x7f230c19eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:48.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.236+0000 7f2309e00640 1 --2- 172.21.15.43:0/4108106246 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f230c10d030 0x7f230c19eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.236+0000 7f2309e00640 1 -- 172.21.15.43:0/4108106246 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f230c1a2f10 msgr2=0x7f230c1a33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:48.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.236+0000 7f2309e00640 1 --2- 172.21.15.43:0/4108106246 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f230c1a2f10 0x7f230c1a33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.236+0000 7f2309e00640 1 -- 172.21.15.43:0/4108106246 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f230c1a3a00 con 0x7f230c107f60 2024-03-28T22:43:48.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.236+0000 7f2309400640 1 --2- 172.21.15.43:0/4108106246 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f230c10d030 0x7f230c19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:48.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.237+0000 7f2309e00640 1 --2- 172.21.15.43:0/4108106246 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f230c107f60 0x7f230c19e650 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f22fc02c170 tx=0x7f22fc004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:48.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.237+0000 7f2302a00640 1 -- 172.21.15.43:0/4108106246 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22fc043020 con 0x7f230c107f60 2024-03-28T22:43:48.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.237+0000 7f2310a00640 1 -- 172.21.15.43:0/4108106246 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f230c1a3c90 con 0x7f230c107f60 2024-03-28T22:43:48.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.237+0000 7f2302a00640 1 -- 172.21.15.43:0/4108106246 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22fc0054e0 con 0x7f230c107f60 2024-03-28T22:43:48.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.237+0000 7f2302a00640 1 -- 172.21.15.43:0/4108106246 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22fc003b50 con 0x7f230c107f60 2024-03-28T22:43:48.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.237+0000 7f2310a00640 1 -- 172.21.15.43:0/4108106246 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f230c1ab450 con 0x7f230c107f60 2024-03-28T22:43:48.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.240+0000 7f2302a00640 1 -- 172.21.15.43:0/4108106246 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f22fc046020 con 0x7f230c107f60 2024-03-28T22:43:48.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.241+0000 7f23137c5740 1 Processor -- start 2024-03-28T22:43:48.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.241+0000 7f23137c5740 1 -- start start 2024-03-28T22:43:48.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.242+0000 7f23137c5740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563415817bc0 0x563415816910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:48.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.242+0000 7f23137c5740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563415816ee0 0x563415817360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:48.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.242+0000 7f23137c5740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563415820250 0x563415822640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.242+0000 7f23137c5740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56341577b100 con 0x563415817bc0 2024-03-28T22:43:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.242+0000 7f23137c5740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x563415723ba0 con 0x563415816ee0 2024-03-28T22:43:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.242+0000 7f23137c5740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5634157771a0 con 0x563415820250 2024-03-28T22:43:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.242+0000 7f2309e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563415816ee0 0x563415817360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.242+0000 7f230a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563415817bc0 0x563415816910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.242+0000 7f230a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563415817bc0 0x563415816910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53808/0 (socket says 172.21.15.43:53808) 2024-03-28T22:43:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.242+0000 7f2309400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563415820250 0x563415822640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.242+0000 7f230a800640 1 -- 172.21.15.43:0/3271694821 learned_addr learned my addr 172.21.15.43:0/3271694821 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:48.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.242+0000 7f2309e00640 1 -- 172.21.15.43:0/3271694821 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563415820250 msgr2=0x563415822640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:48.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.242+0000 7f2309e00640 1 --2- 172.21.15.43:0/3271694821 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563415820250 0x563415822640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.242+0000 7f2309e00640 1 -- 172.21.15.43:0/3271694821 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563415817bc0 msgr2=0x563415816910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:48.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.242+0000 7f2309e00640 1 --2- 172.21.15.43:0/3271694821 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563415817bc0 0x563415816910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.243+0000 7f2309e00640 1 -- 172.21.15.43:0/3271694821 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634157541a0 con 0x563415816ee0 2024-03-28T22:43:48.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.243+0000 7f230a800640 1 --2- 172.21.15.43:0/3271694821 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563415817bc0 0x563415816910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:48.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.243+0000 7f2302a00640 1 --2- 172.21.15.43:0/4108106246 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f22f00a1ef0 0x7f22f00a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:48.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.243+0000 7f2309e00640 1 --2- 172.21.15.43:0/3271694821 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563415816ee0 0x563415817360 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f22fc0ed940 tx=0x7f22fc0f6550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcadc0eab3f577c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:48.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.243+0000 7f2302a00640 1 -- 172.21.15.43:0/4108106246 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f22fc0f04e0 con 0x7f230c107f60 2024-03-28T22:43:48.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.243+0000 7f230a800640 1 --2- 172.21.15.43:0/4108106246 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f22f00a1ef0 0x7f22f00a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:48.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.243+0000 7f22f7000640 1 -- 172.21.15.43:0/3271694821 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22fc0b8e80 con 0x563415816ee0 2024-03-28T22:43:48.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.243+0000 7f22f7000640 1 -- 172.21.15.43:0/3271694821 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22fc004c80 con 0x563415816ee0 2024-03-28T22:43:48.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.243+0000 7f22f7000640 1 -- 172.21.15.43:0/3271694821 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22fc0bc540 con 0x563415816ee0 2024-03-28T22:43:48.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.244+0000 7f23137c5740 1 -- 172.21.15.43:0/3271694821 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563415816ee0 msgr2=0x563415817360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:48.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.244+0000 7f23137c5740 1 --2- 172.21.15.43:0/3271694821 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563415816ee0 0x563415817360 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f22fc0ed940 tx=0x7f22fc0f6550 comp rx=0 tx=0).stop 2024-03-28T22:43:48.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.244+0000 7f230a800640 1 --2- 172.21.15.43:0/4108106246 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f22f00a1ef0 0x7f22f00a43b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2304005f40 tx=0x7f2304009600 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:48.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.244+0000 7f23137c5740 1 -- 172.21.15.43:0/3271694821 shutdown_connections 2024-03-28T22:43:48.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.244+0000 7f23137c5740 1 --2- 172.21.15.43:0/3271694821 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563415816ee0 0x563415817360 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.244+0000 7f23137c5740 1 --2- 172.21.15.43:0/3271694821 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563415820250 0x563415822640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.244+0000 7f23137c5740 1 --2- 172.21.15.43:0/3271694821 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563415817bc0 0x563415816910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.244+0000 7f23137c5740 1 -- 172.21.15.43:0/3271694821 >> 172.21.15.43:0/3271694821 conn(0x5634157dda10 msgr2=0x563415811930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:48.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.244+0000 7f23137c5740 1 -- 172.21.15.43:0/3271694821 shutdown_connections 2024-03-28T22:43:48.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.244+0000 7f23137c5740 1 -- 172.21.15.43:0/3271694821 wait complete. 2024-03-28T22:43:48.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.245+0000 7f23137c5740 1 -- start start 2024-03-28T22:43:48.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.245+0000 7f23137c5740 1 Processor -- start 2024-03-28T22:43:48.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.245+0000 7f23137c5740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563415816ee0 0x56341581d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:48.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.245+0000 7f23137c5740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563415817bc0 0x56341581d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:48.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.245+0000 7f23137c5740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563415820250 0x56341581dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:48.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.245+0000 7f23137c5740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x563415822e80 con 0x563415816ee0 2024-03-28T22:43:48.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.245+0000 7f23137c5740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x563415719250 con 0x563415820250 2024-03-28T22:43:48.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.245+0000 7f23137c5740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x563415823000 con 0x563415817bc0 2024-03-28T22:43:48.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.246+0000 7f23137c5740 10 client.0 _ll_register_callbacks cb 0x56341548ebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:48.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.246+0000 7f2309400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563415817bc0 0x56341581d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:48.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.246+0000 7f230a800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563415820250 0x56341581dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:48.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.246+0000 7f2309400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563415817bc0 0x56341581d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:60836/0 (socket says 172.21.15.43:60836) 2024-03-28T22:43:48.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.246+0000 7f2309400640 1 -- 172.21.15.43:0/24581671 learned_addr learned my addr 172.21.15.43:0/24581671 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:48.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.246+0000 7f2309e00640 1 --2- 172.21.15.43:0/24581671 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563415816ee0 0x56341581d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:48.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.246+0000 7f230a800640 1 -- 172.21.15.43:0/24581671 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563415817bc0 msgr2=0x56341581d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:48.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.246+0000 7f230a800640 1 --2- 172.21.15.43:0/24581671 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563415817bc0 0x56341581d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.246+0000 7f230a800640 1 -- 172.21.15.43:0/24581671 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563415816ee0 msgr2=0x56341581d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:48.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.246+0000 7f230a800640 1 --2- 172.21.15.43:0/24581671 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563415816ee0 0x56341581d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.246+0000 7f230a800640 1 -- 172.21.15.43:0/24581671 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563415735960 con 0x563415820250 2024-03-28T22:43:48.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.246+0000 7f2309400640 1 --2- 172.21.15.43:0/24581671 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563415817bc0 0x56341581d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:48.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.246+0000 7f230a800640 1 --2- 172.21.15.43:0/24581671 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563415820250 0x56341581dc10 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f2304013fb0 tx=0x7f2304004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=583cfa14b59a6a2e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:48.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.246+0000 7f22eb400640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:43:48.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.247+0000 7f22eb400640 1 -- 172.21.15.43:0/24581671 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230401ed30 con 0x563415820250 2024-03-28T22:43:48.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.247+0000 7f22eb400640 1 -- 172.21.15.43:0/24581671 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f2304003970 con 0x563415820250 2024-03-28T22:43:48.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.247+0000 7f23137c5740 1 -- 172.21.15.43:0/24581671 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22fc004c80 con 0x563415820250 2024-03-28T22:43:48.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.247+0000 7f22eb400640 1 -- 172.21.15.43:0/24581671 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2304009bb0 con 0x563415820250 2024-03-28T22:43:48.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.247+0000 7f23137c5740 10 client.28022 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:48.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.247+0000 7f23137c5740 1 -- 172.21.15.43:0/24581671 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5634155597d0 con 0x563415820250 2024-03-28T22:43:48.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.247+0000 7f22eb400640 1 -- 172.21.15.43:0/24581671 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2304032070 con 0x563415820250 2024-03-28T22:43:48.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.247+0000 7f22eb400640 1 -- 172.21.15.43:0/24581671 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f230400f040 con 0x563415820250 2024-03-28T22:43:48.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.248+0000 7f23137c5740 10 client.28022 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:48.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.248+0000 7f23137c5740 10 client.28022 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:43:48.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.248+0000 7f23137c5740 20 client.28022 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:48.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.248+0000 7f23137c5740 1 --2- 172.21.15.43:0/24581671 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x563415830420 0x563415832880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:48.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.248+0000 7f23137c5740 4 client.28022 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:43:48.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.248+0000 7f23137c5740 1 -- 172.21.15.43:0/24581671 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563415788640 con 0x563415830420 2024-03-28T22:43:48.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.248+0000 7f2309e00640 1 --2- 172.21.15.43:0/24581671 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x563415830420 0x563415832880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:48.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.248+0000 7f2309e00640 1 --2- 172.21.15.43:0/24581671 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x563415830420 0x563415832880 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3fcef03b56f8d18 server_cookie=70dcbb9993d1902 in_seq=0 out_seq=0 2024-03-28T22:43:48.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.248+0000 7f22eb400640 10 client.28022 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:48.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.252+0000 7f22eb400640 1 -- 172.21.15.43:0/24581671 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563415788640 con 0x563415830420 2024-03-28T22:43:48.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.252+0000 7f22eb400640 10 client.28022 handle_command_reply: tid=0 2024-03-28T22:43:48.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.252+0000 7f23137c5740 1 client.28022 shutdown 2024-03-28T22:43:48.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.252+0000 7f23137c5740 1 -- 172.21.15.43:0/24581671 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563415820250 msgr2=0x56341581dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:48.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.252+0000 7f23137c5740 1 --2- 172.21.15.43:0/24581671 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563415820250 0x56341581dc10 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f2304013fb0 tx=0x7f2304004a40 comp rx=0 tx=0).stop 2024-03-28T22:43:48.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.252+0000 7f23137c5740 1 -- 172.21.15.43:0/24581671 shutdown_connections 2024-03-28T22:43:48.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.252+0000 7f23137c5740 1 --2- 172.21.15.43:0/24581671 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x563415830420 0x563415832880 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.252+0000 7f23137c5740 1 --2- 172.21.15.43:0/24581671 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563415820250 0x56341581dc10 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.252+0000 7f23137c5740 1 --2- 172.21.15.43:0/24581671 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563415817bc0 0x56341581d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.252+0000 7f23137c5740 1 --2- 172.21.15.43:0/24581671 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563415816ee0 0x56341581d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.252+0000 7f23137c5740 1 -- 172.21.15.43:0/24581671 >> 172.21.15.43:0/24581671 conn(0x5634157dda10 msgr2=0x563415813680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:48.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.252+0000 7f22eb400640 0 client.28022 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:48.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.253+0000 7f23137c5740 1 -- 172.21.15.43:0/24581671 shutdown_connections 2024-03-28T22:43:48.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.253+0000 7f23137c5740 1 -- 172.21.15.43:0/24581671 wait complete. 2024-03-28T22:43:48.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.253+0000 7f23137c5740 20 client.28022 trim_cache size 0 max 16384 2024-03-28T22:43:48.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.273+0000 7f23137c5740 1 Processor -- start 2024-03-28T22:43:48.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.273+0000 7f23137c5740 1 -- start start 2024-03-28T22:43:48.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.273+0000 7f23137c5740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563415816ee0 0x56341581f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:48.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.273+0000 7f23137c5740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563415817bc0 0x56341581c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:48.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.273+0000 7f23137c5740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563415820250 0x563415811460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:48.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.273+0000 7f23137c5740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f230c1b5150 con 0x563415816ee0 2024-03-28T22:43:48.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.273+0000 7f23137c5740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f230c1b6c90 con 0x563415820250 2024-03-28T22:43:48.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.273+0000 7f23137c5740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f230c1b5820 con 0x563415817bc0 2024-03-28T22:43:48.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.273+0000 7f2309400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563415817bc0 0x56341581c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:48.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.273+0000 7f2309e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563415816ee0 0x56341581f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:48.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.273+0000 7f230a800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563415820250 0x563415811460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:48.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.273+0000 7f2309400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563415817bc0 0x56341581c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:60852/0 (socket says 172.21.15.43:60852) 2024-03-28T22:43:48.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.273+0000 7f2309400640 1 -- 172.21.15.43:0/3512422978 learned_addr learned my addr 172.21.15.43:0/3512422978 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:48.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.274+0000 7f2309400640 1 -- 172.21.15.43:0/3512422978 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563415820250 msgr2=0x563415811460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:48.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.274+0000 7f2309400640 1 --2- 172.21.15.43:0/3512422978 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563415820250 0x563415811460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.274+0000 7f2309400640 1 -- 172.21.15.43:0/3512422978 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563415816ee0 msgr2=0x56341581f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:48.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.274+0000 7f2309400640 1 --2- 172.21.15.43:0/3512422978 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563415816ee0 0x56341581f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.274+0000 7f2309400640 1 -- 172.21.15.43:0/3512422978 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56341548f3f0 con 0x563415817bc0 2024-03-28T22:43:48.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.274+0000 7f2309400640 1 --2- 172.21.15.43:0/3512422978 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563415817bc0 0x56341581c0d0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f22f8012ab0 tx=0x7f22f8005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcf64ef929a0563c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:48.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.274+0000 7f22f6600640 1 -- 172.21.15.43:0/3512422978 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f80120d0 con 0x563415817bc0 2024-03-28T22:43:48.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.274+0000 7f22f6600640 1 -- 172.21.15.43:0/3512422978 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22f8012270 con 0x563415817bc0 2024-03-28T22:43:48.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.274+0000 7f22f6600640 1 -- 172.21.15.43:0/3512422978 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f8004ad0 con 0x563415817bc0 2024-03-28T22:43:48.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.275+0000 7f23137c5740 1 -- 172.21.15.43:0/3512422978 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563415817bc0 msgr2=0x56341581c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:48.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.275+0000 7f23137c5740 1 --2- 172.21.15.43:0/3512422978 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563415817bc0 0x56341581c0d0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f22f8012ab0 tx=0x7f22f8005f90 comp rx=0 tx=0).stop 2024-03-28T22:43:48.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.275+0000 7f23137c5740 1 -- 172.21.15.43:0/3512422978 shutdown_connections 2024-03-28T22:43:48.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.275+0000 7f23137c5740 1 --2- 172.21.15.43:0/3512422978 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563415820250 0x563415811460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.275+0000 7f23137c5740 1 --2- 172.21.15.43:0/3512422978 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563415817bc0 0x56341581c0d0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.275+0000 7f23137c5740 1 --2- 172.21.15.43:0/3512422978 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563415816ee0 0x56341581f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.275+0000 7f23137c5740 1 -- 172.21.15.43:0/3512422978 >> 172.21.15.43:0/3512422978 conn(0x5634157dda10 msgr2=0x563415813610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:48.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.275+0000 7f23137c5740 1 -- 172.21.15.43:0/3512422978 shutdown_connections 2024-03-28T22:43:48.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.275+0000 7f23137c5740 1 -- 172.21.15.43:0/3512422978 wait complete. 2024-03-28T22:43:48.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.276+0000 7f23137c5740 1 -- start start 2024-03-28T22:43:48.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.276+0000 7f23137c5740 1 Processor -- start 2024-03-28T22:43:48.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.276+0000 7f23137c5740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563415816ee0 0x563415823c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:48.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.276+0000 7f23137c5740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563415817bc0 0x563415824160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:48.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.276+0000 7f23137c5740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563415820250 0x5634158246a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:48.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.276+0000 7f23137c5740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f230c1b65c0 con 0x563415816ee0 2024-03-28T22:43:48.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.276+0000 7f23137c5740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f230c1b5ef0 con 0x563415820250 2024-03-28T22:43:48.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.276+0000 7f23137c5740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f230c1b3130 con 0x563415817bc0 2024-03-28T22:43:48.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.276+0000 7f23137c5740 10 client.0 _ll_register_callbacks cb 0x7f2304009ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:48.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.277+0000 7f2309400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563415817bc0 0x563415824160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:48.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.277+0000 7f230a800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563415820250 0x5634158246a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:48.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.277+0000 7f2309400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563415817bc0 0x563415824160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:60862/0 (socket says 172.21.15.43:60862) 2024-03-28T22:43:48.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.277+0000 7f2309400640 1 -- 172.21.15.43:0/1524954206 learned_addr learned my addr 172.21.15.43:0/1524954206 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:48.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.277+0000 7f2309e00640 1 --2- 172.21.15.43:0/1524954206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563415816ee0 0x563415823c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:48.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.277+0000 7f230a800640 1 -- 172.21.15.43:0/1524954206 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563415817bc0 msgr2=0x563415824160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:48.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.277+0000 7f230a800640 1 --2- 172.21.15.43:0/1524954206 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563415817bc0 0x563415824160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.277+0000 7f230a800640 1 -- 172.21.15.43:0/1524954206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563415816ee0 msgr2=0x563415823c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:48.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.277+0000 7f230a800640 1 --2- 172.21.15.43:0/1524954206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563415816ee0 0x563415823c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.277+0000 7f230a800640 1 -- 172.21.15.43:0/1524954206 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56341544f0f0 con 0x563415820250 2024-03-28T22:43:48.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.277+0000 7f230a800640 1 --2- 172.21.15.43:0/1524954206 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563415820250 0x5634158246a0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f2304000f80 tx=0x7f2304004d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3a5495f62558782 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:48.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.277+0000 7f22eb400640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:43:48.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.277+0000 7f22eb400640 1 -- 172.21.15.43:0/1524954206 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230403a020 con 0x563415820250 2024-03-28T22:43:48.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.278+0000 7f23137c5740 1 -- 172.21.15.43:0/1524954206 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22f8012270 con 0x563415820250 2024-03-28T22:43:48.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.278+0000 7f22eb400640 1 -- 172.21.15.43:0/1524954206 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f230401fc90 con 0x563415820250 2024-03-28T22:43:48.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.278+0000 7f22eb400640 1 -- 172.21.15.43:0/1524954206 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2304045050 con 0x563415820250 2024-03-28T22:43:48.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.278+0000 7f23137c5740 10 client.28028 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:48.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.278+0000 7f23137c5740 1 -- 172.21.15.43:0/1524954206 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563415896990 con 0x563415820250 2024-03-28T22:43:48.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.278+0000 7f22eb400640 1 -- 172.21.15.43:0/1524954206 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22f8012270 con 0x563415820250 2024-03-28T22:43:48.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.278+0000 7f22eb400640 1 -- 172.21.15.43:0/1524954206 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f230400f040 con 0x563415820250 2024-03-28T22:43:48.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.278+0000 7f23137c5740 10 client.28028 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:48.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.278+0000 7f23137c5740 10 client.28028 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:43:48.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.278+0000 7f23137c5740 20 client.28028 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:48.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.278+0000 7f23137c5740 1 --2- 172.21.15.43:0/1524954206 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x563415897130 0x563415899550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:48.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.278+0000 7f23137c5740 4 client.28028 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:43:48.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.278+0000 7f23137c5740 1 -- 172.21.15.43:0/1524954206 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563415782990 con 0x563415897130 2024-03-28T22:43:48.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.279+0000 7f2309e00640 1 --2- 172.21.15.43:0/1524954206 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x563415897130 0x563415899550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:48.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.279+0000 7f2309e00640 1 --2- 172.21.15.43:0/1524954206 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x563415897130 0x563415899550 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6a0e76480ee08a9 server_cookie=e6dd20ae63a83b3d in_seq=0 out_seq=0 2024-03-28T22:43:48.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.279+0000 7f22eb400640 10 client.28028 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:48.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.280+0000 7f22eb400640 1 -- 172.21.15.43:0/1524954206 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563415782990 con 0x563415897130 2024-03-28T22:43:48.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.280+0000 7f22eb400640 10 client.28028 handle_command_reply: tid=0 2024-03-28T22:43:48.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.280+0000 7f23137c5740 1 client.28028 shutdown 2024-03-28T22:43:48.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.280+0000 7f23137c5740 1 -- 172.21.15.43:0/1524954206 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563415820250 msgr2=0x5634158246a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:48.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.280+0000 7f23137c5740 1 --2- 172.21.15.43:0/1524954206 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563415820250 0x5634158246a0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f2304000f80 tx=0x7f2304004d60 comp rx=0 tx=0).stop 2024-03-28T22:43:48.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.280+0000 7f23137c5740 1 -- 172.21.15.43:0/1524954206 shutdown_connections 2024-03-28T22:43:48.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.280+0000 7f23137c5740 1 --2- 172.21.15.43:0/1524954206 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x563415897130 0x563415899550 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.280+0000 7f23137c5740 1 --2- 172.21.15.43:0/1524954206 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563415820250 0x5634158246a0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.280+0000 7f23137c5740 1 --2- 172.21.15.43:0/1524954206 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563415817bc0 0x563415824160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.280+0000 7f23137c5740 1 --2- 172.21.15.43:0/1524954206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563415816ee0 0x563415823c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.280+0000 7f23137c5740 1 -- 172.21.15.43:0/1524954206 >> 172.21.15.43:0/1524954206 conn(0x5634157dda10 msgr2=0x56341581d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:48.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.280+0000 7f22eb400640 0 client.28028 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:48.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.280+0000 7f23137c5740 1 -- 172.21.15.43:0/1524954206 shutdown_connections 2024-03-28T22:43:48.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.281+0000 7f23137c5740 1 -- 172.21.15.43:0/1524954206 wait complete. 2024-03-28T22:43:48.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.281+0000 7f23137c5740 20 client.28028 trim_cache size 0 max 16384 2024-03-28T22:43:48.283 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:43:48.283 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-03-28T22:43:48.283 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-03-28T22:43:48.283 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:43:48.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.281+0000 7f2310a00640 1 -- 172.21.15.43:0/4108106246 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f22f00a1ef0 msgr2=0x7f22f00a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:48.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.281+0000 7f2310a00640 1 --2- 172.21.15.43:0/4108106246 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f22f00a1ef0 0x7f22f00a43b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2304005f40 tx=0x7f2304009600 comp rx=0 tx=0).stop 2024-03-28T22:43:48.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.281+0000 7f2310a00640 1 -- 172.21.15.43:0/4108106246 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f230c107f60 msgr2=0x7f230c19e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:48.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.281+0000 7f2310a00640 1 --2- 172.21.15.43:0/4108106246 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f230c107f60 0x7f230c19e650 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f22fc02c170 tx=0x7f22fc004360 comp rx=0 tx=0).stop 2024-03-28T22:43:48.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.281+0000 7f2310a00640 1 -- 172.21.15.43:0/4108106246 shutdown_connections 2024-03-28T22:43:48.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.281+0000 7f2310a00640 1 --2- 172.21.15.43:0/4108106246 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f22f00a1ef0 0x7f22f00a43b0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.281+0000 7f2310a00640 1 --2- 172.21.15.43:0/4108106246 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f230c1a2f10 0x7f230c1a33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.281+0000 7f2310a00640 1 --2- 172.21.15.43:0/4108106246 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f230c10d030 0x7f230c19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.281+0000 7f2310a00640 1 --2- 172.21.15.43:0/4108106246 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f230c107f60 0x7f230c19e650 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.281+0000 7f2310a00640 1 -- 172.21.15.43:0/4108106246 >> 172.21.15.43:0/4108106246 conn(0x7f230c075c90 msgr2=0x7f230c077ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:48.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.282+0000 7f2310a00640 1 -- 172.21.15.43:0/4108106246 shutdown_connections 2024-03-28T22:43:48.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.282+0000 7f2310a00640 1 -- 172.21.15.43:0/4108106246 wait complete. 2024-03-28T22:43:48.300 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:43:48.300 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:43:48.378 INFO:tasks.workunit.client.0.smithi043.stdout: 1 454702 7.29 MB/sec execute 199 sec latency 13.411 ms 2024-03-28T22:43:48.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.537+0000 7f0807000640 1 -- 172.21.15.43:0/2457147985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f080010dbf0 msgr2=0x7f080010dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:48.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.537+0000 7f0807000640 1 --2- 172.21.15.43:0/2457147985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f080010dbf0 0x7f080010dfd0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f07f4009930 tx=0x7f07f402bc30 comp rx=0 tx=0).stop 2024-03-28T22:43:48.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.537+0000 7f0807000640 1 -- 172.21.15.43:0/2457147985 shutdown_connections 2024-03-28T22:43:48.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.537+0000 7f0807000640 1 --2- 172.21.15.43:0/2457147985 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0800107b30 0x7f0800107fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.537+0000 7f0807000640 1 --2- 172.21.15.43:0/2457147985 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08001084f0 0x7f08001149b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.537+0000 7f0807000640 1 --2- 172.21.15.43:0/2457147985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f080010dbf0 0x7f080010dfd0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.537+0000 7f0807000640 1 -- 172.21.15.43:0/2457147985 >> 172.21.15.43:0/2457147985 conn(0x7f0800077270 msgr2=0x7f0800079690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:48.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.537+0000 7f0807000640 1 -- 172.21.15.43:0/2457147985 shutdown_connections 2024-03-28T22:43:48.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.537+0000 7f0807000640 1 -- 172.21.15.43:0/2457147985 wait complete. 2024-03-28T22:43:48.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.538+0000 7f0807000640 1 Processor -- start 2024-03-28T22:43:48.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.538+0000 7f0807000640 1 -- start start 2024-03-28T22:43:48.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.539+0000 7f0807000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0800107b30 0x7f080019e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:48.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.539+0000 7f0807000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08001084f0 0x7f080019e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:48.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.539+0000 7f0807000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f080010dbf0 0x7f08001a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:48.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.539+0000 7f0807000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0800117fa0 con 0x7f080010dbf0 2024-03-28T22:43:48.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.539+0000 7f0807000640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f0800117e20 con 0x7f0800107b30 2024-03-28T22:43:48.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.539+0000 7f0807000640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f0800118120 con 0x7f08001084f0 2024-03-28T22:43:48.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.539+0000 7f0805200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08001084f0 0x7f080019e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:48.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.539+0000 7f0805c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0800107b30 0x7f080019e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:48.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.539+0000 7f0805200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08001084f0 0x7f080019e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:60882/0 (socket says 172.21.15.43:60882) 2024-03-28T22:43:48.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.539+0000 7f0805c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0800107b30 0x7f080019e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47266/0 (socket says 172.21.15.43:47266) 2024-03-28T22:43:48.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.539+0000 7f0805200640 1 -- 172.21.15.43:0/806575601 learned_addr learned my addr 172.21.15.43:0/806575601 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:48.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.539+0000 7f0806600640 1 --2- 172.21.15.43:0/806575601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f080010dbf0 0x7f08001a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:48.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.540+0000 7f0805c00640 1 -- 172.21.15.43:0/806575601 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08001084f0 msgr2=0x7f080019e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:48.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.540+0000 7f0805c00640 1 --2- 172.21.15.43:0/806575601 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08001084f0 0x7f080019e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.540+0000 7f0805c00640 1 -- 172.21.15.43:0/806575601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f080010dbf0 msgr2=0x7f08001a2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:48.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.540+0000 7f0805c00640 1 --2- 172.21.15.43:0/806575601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f080010dbf0 0x7f08001a2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.540+0000 7f0805c00640 1 -- 172.21.15.43:0/806575601 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08001a33f0 con 0x7f0800107b30 2024-03-28T22:43:48.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.540+0000 7f0805c00640 1 --2- 172.21.15.43:0/806575601 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0800107b30 0x7f080019e400 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f07f4036040 tx=0x7f07f4004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:48.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.540+0000 7f07fe000640 1 -- 172.21.15.43:0/806575601 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f4043020 con 0x7f0800107b30 2024-03-28T22:43:48.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.540+0000 7f07fe000640 1 -- 172.21.15.43:0/806575601 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07f40054e0 con 0x7f0800107b30 2024-03-28T22:43:48.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.540+0000 7f0807000640 1 -- 172.21.15.43:0/806575601 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08001a3680 con 0x7f0800107b30 2024-03-28T22:43:48.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.540+0000 7f07fe000640 1 -- 172.21.15.43:0/806575601 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f4003b80 con 0x7f0800107b30 2024-03-28T22:43:48.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.541+0000 7f0807000640 1 -- 172.21.15.43:0/806575601 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08001aaf70 con 0x7f0800107b30 2024-03-28T22:43:48.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.542+0000 7f0807000640 1 -- 172.21.15.43:0/806575601 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08001091e0 con 0x7f0800107b30 2024-03-28T22:43:48.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.544+0000 7f07fe000640 1 -- 172.21.15.43:0/806575601 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f07f4046020 con 0x7f0800107b30 2024-03-28T22:43:48.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.545+0000 7f07fe000640 1 --2- 172.21.15.43:0/806575601 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f07e80a1ef0 0x7f07e80a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:48.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.545+0000 7f07fe000640 1 -- 172.21.15.43:0/806575601 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f07f40f04b0 con 0x7f0800107b30 2024-03-28T22:43:48.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.546+0000 7f0805200640 1 --2- 172.21.15.43:0/806575601 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f07e80a1ef0 0x7f07e80a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:48.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.546+0000 7f0805200640 1 --2- 172.21.15.43:0/806575601 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f07e80a1ef0 0x7f07e80a43b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f080010d230 tx=0x7f07f0005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:48.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.549+0000 7f07fe000640 1 -- 172.21.15.43:0/806575601 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f07f404d020 con 0x7f0800107b30 2024-03-28T22:43:48.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.808+0000 7f0807000640 1 -- 172.21.15.43:0/806575601 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f080010ffe0 con 0x7f0800107b30 2024-03-28T22:43:48.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.809+0000 7f07fe000640 1 -- 172.21.15.43:0/806575601 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11931 (secure 0 0 0) 0x7f07f40bd2b0 con 0x7f0800107b30 2024-03-28T22:43:48.808 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:43:48.808 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:33:28.906396+0000","modified":"2024-03-28T22:40:08.672168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24572,"mds_1":24556,"mds_2":24535,"mds_3":24523,"mds_4":24518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26717":{"gid":26717,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/215779342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":215779342},{"type":"v1","addr":"172.21.15.82:6835","nonce":215779342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24535,24556,24518,24523,24572]},"id":1}]} 2024-03-28T22:43:48.808 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 32 2024-03-28T22:43:48.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.812+0000 7f0807000640 1 -- 172.21.15.43:0/806575601 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f07e80a1ef0 msgr2=0x7f07e80a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:48.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.812+0000 7f0807000640 1 --2- 172.21.15.43:0/806575601 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f07e80a1ef0 0x7f07e80a43b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f080010d230 tx=0x7f07f0005f90 comp rx=0 tx=0).stop 2024-03-28T22:43:48.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.812+0000 7f0807000640 1 -- 172.21.15.43:0/806575601 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0800107b30 msgr2=0x7f080019e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:48.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.812+0000 7f0807000640 1 --2- 172.21.15.43:0/806575601 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0800107b30 0x7f080019e400 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f07f4036040 tx=0x7f07f4004360 comp rx=0 tx=0).stop 2024-03-28T22:43:48.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.812+0000 7f0807000640 1 -- 172.21.15.43:0/806575601 shutdown_connections 2024-03-28T22:43:48.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.812+0000 7f0807000640 1 --2- 172.21.15.43:0/806575601 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f07e80a1ef0 0x7f07e80a43b0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.812+0000 7f0807000640 1 --2- 172.21.15.43:0/806575601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f080010dbf0 0x7f08001a2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.812+0000 7f0807000640 1 --2- 172.21.15.43:0/806575601 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f08001084f0 0x7f080019e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.812+0000 7f0807000640 1 --2- 172.21.15.43:0/806575601 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0800107b30 0x7f080019e400 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:48.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.812+0000 7f0807000640 1 -- 172.21.15.43:0/806575601 >> 172.21.15.43:0/806575601 conn(0x7f0800077270 msgr2=0x7f0800078990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:48.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.812+0000 7f0807000640 1 -- 172.21.15.43:0/806575601 shutdown_connections 2024-03-28T22:43:48.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:48.812+0000 7f0807000640 1 -- 172.21.15.43:0/806575601 wait complete. 2024-03-28T22:43:48.832 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T22:43:49.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.062+0000 7fa894e00640 1 -- 172.21.15.43:0/2433995893 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa890073b80 msgr2=0x7fa890073f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.062+0000 7fa894e00640 1 --2- 172.21.15.43:0/2433995893 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa890073b80 0x7fa890073f60 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa880009900 tx=0x7fa88002bcd0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.062+0000 7fa894e00640 1 -- 172.21.15.43:0/2433995893 shutdown_connections 2024-03-28T22:43:49.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.062+0000 7fa894e00640 1 --2- 172.21.15.43:0/2433995893 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa89010ddd0 0x7fa890110220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.062+0000 7fa894e00640 1 --2- 172.21.15.43:0/2433995893 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8900744a0 0x7fa89010d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.062+0000 7fa894e00640 1 --2- 172.21.15.43:0/2433995893 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa890073b80 0x7fa890073f60 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.062+0000 7fa894e00640 1 -- 172.21.15.43:0/2433995893 >> 172.21.15.43:0/2433995893 conn(0x7fa8900fd0c0 msgr2=0x7fa8900ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:49.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.062+0000 7fa894e00640 1 -- 172.21.15.43:0/2433995893 shutdown_connections 2024-03-28T22:43:49.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.062+0000 7fa894e00640 1 -- 172.21.15.43:0/2433995893 wait complete. 2024-03-28T22:43:49.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.063+0000 7fa894e00640 1 Processor -- start 2024-03-28T22:43:49.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.064+0000 7fa894e00640 1 -- start start 2024-03-28T22:43:49.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.064+0000 7fa894e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa890073b80 0x7fa89019a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.064+0000 7fa894e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8900744a0 0x7fa89019a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.064+0000 7fa894e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa89010ddd0 0x7fa89019e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.064+0000 7fa894e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa890113b70 con 0x7fa8900744a0 2024-03-28T22:43:49.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.064+0000 7fa894e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa8901139f0 con 0x7fa89010ddd0 2024-03-28T22:43:49.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.064+0000 7fa894e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fa890113cf0 con 0x7fa890073b80 2024-03-28T22:43:49.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.064+0000 7fa88f400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa890073b80 0x7fa89019a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.064+0000 7fa88f400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa890073b80 0x7fa89019a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:60902/0 (socket says 172.21.15.43:60902) 2024-03-28T22:43:49.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.064+0000 7fa88f400640 1 -- 172.21.15.43:0/1995509099 learned_addr learned my addr 172.21.15.43:0/1995509099 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:49.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.064+0000 7fa88fe00640 1 --2- 172.21.15.43:0/1995509099 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa89010ddd0 0x7fa89019e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.064+0000 7fa88ea00640 1 --2- 172.21.15.43:0/1995509099 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8900744a0 0x7fa89019a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.065+0000 7fa88f400640 1 -- 172.21.15.43:0/1995509099 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa89010ddd0 msgr2=0x7fa89019e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.065+0000 7fa88f400640 1 --2- 172.21.15.43:0/1995509099 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa89010ddd0 0x7fa89019e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.065+0000 7fa88f400640 1 -- 172.21.15.43:0/1995509099 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8900744a0 msgr2=0x7fa89019a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.065+0000 7fa88f400640 1 --2- 172.21.15.43:0/1995509099 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8900744a0 0x7fa89019a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.065+0000 7fa88f400640 1 -- 172.21.15.43:0/1995509099 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa89019f0c0 con 0x7fa890073b80 2024-03-28T22:43:49.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.065+0000 7fa88fe00640 1 --2- 172.21.15.43:0/1995509099 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa89010ddd0 0x7fa89019e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:49.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.065+0000 7fa88f400640 1 --2- 172.21.15.43:0/1995509099 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa890073b80 0x7fa89019a0d0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fa880005940 tx=0x7fa88002cbd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:49.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.066+0000 7fa887e00640 1 -- 172.21.15.43:0/1995509099 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa880041070 con 0x7fa890073b80 2024-03-28T22:43:49.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.066+0000 7fa887e00640 1 -- 172.21.15.43:0/1995509099 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa880004450 con 0x7fa890073b80 2024-03-28T22:43:49.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.066+0000 7fa887e00640 1 -- 172.21.15.43:0/1995509099 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa880003d80 con 0x7fa890073b80 2024-03-28T22:43:49.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.066+0000 7fa894e00640 1 -- 172.21.15.43:0/1995509099 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa89019f350 con 0x7fa890073b80 2024-03-28T22:43:49.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.066+0000 7fa894e00640 1 -- 172.21.15.43:0/1995509099 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8901a6c40 con 0x7fa890073b80 2024-03-28T22:43:49.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.069+0000 7fa887e00640 1 -- 172.21.15.43:0/1995509099 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fa880044070 con 0x7fa890073b80 2024-03-28T22:43:49.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.069+0000 7fa8991eb740 1 Processor -- start 2024-03-28T22:43:49.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.069+0000 7fa8991eb740 1 -- start start 2024-03-28T22:43:49.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.070+0000 7fa8991eb740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560589422d60 0x560589423140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.070+0000 7fa8991eb740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560589423680 0x56058942bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.070+0000 7fa8991eb740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56058942c260 0x56058942e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.070+0000 7fa8991eb740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56058937e050 con 0x56058942c260 2024-03-28T22:43:49.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.070+0000 7fa8991eb740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x560589326ce0 con 0x560589422d60 2024-03-28T22:43:49.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.070+0000 7fa8991eb740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x56058937a0f0 con 0x560589423680 2024-03-28T22:43:49.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.070+0000 7fa88fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560589422d60 0x560589423140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.070+0000 7fa88fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560589422d60 0x560589423140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47292/0 (socket says 172.21.15.43:47292) 2024-03-28T22:43:49.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.070+0000 7fa88fe00640 1 -- 172.21.15.43:0/1554934459 learned_addr learned my addr 172.21.15.43:0/1554934459 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:49.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.070+0000 7fa88f400640 1 --2- 172.21.15.43:0/1554934459 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560589423680 0x56058942bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.070+0000 7fa88ea00640 1 --2- 172.21.15.43:0/1554934459 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56058942c260 0x56058942e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.070+0000 7fa88fe00640 1 -- 172.21.15.43:0/1554934459 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560589423680 msgr2=0x56058942bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:49.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.070+0000 7fa88fe00640 1 --2- 172.21.15.43:0/1554934459 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560589423680 0x56058942bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.070+0000 7fa88fe00640 1 -- 172.21.15.43:0/1554934459 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56058942c260 msgr2=0x56058942e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:49.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.070+0000 7fa88fe00640 1 --2- 172.21.15.43:0/1554934459 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56058942c260 0x56058942e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.070+0000 7fa88fe00640 1 -- 172.21.15.43:0/1554934459 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605890923f0 con 0x560589422d60 2024-03-28T22:43:49.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.071+0000 7fa88fe00640 1 --2- 172.21.15.43:0/1554934459 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560589422d60 0x560589423140 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fa888011790 tx=0x7fa888011c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6018d6b9fd10a8ef server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:49.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.071+0000 7fa884c00640 1 -- 172.21.15.43:0/1554934459 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa888003c30 con 0x560589422d60 2024-03-28T22:43:49.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.071+0000 7fa884c00640 1 -- 172.21.15.43:0/1554934459 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa88800fe80 con 0x560589422d60 2024-03-28T22:43:49.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.071+0000 7fa884c00640 1 -- 172.21.15.43:0/1554934459 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8880193e0 con 0x560589422d60 2024-03-28T22:43:49.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.071+0000 7fa887e00640 1 --2- 172.21.15.43:0/1995509099 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa8740a1f70 0x7fa8740a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.071+0000 7fa887e00640 1 -- 172.21.15.43:0/1995509099 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fa8800eff30 con 0x7fa890073b80 2024-03-28T22:43:49.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.071+0000 7fa88f400640 1 --2- 172.21.15.43:0/1995509099 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa8740a1f70 0x7fa8740a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.072+0000 7fa8991eb740 1 -- 172.21.15.43:0/1554934459 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560589422d60 msgr2=0x560589423140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.072+0000 7fa8991eb740 1 --2- 172.21.15.43:0/1554934459 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560589422d60 0x560589423140 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fa888011790 tx=0x7fa888011c50 comp rx=0 tx=0).stop 2024-03-28T22:43:49.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.072+0000 7fa8991eb740 1 -- 172.21.15.43:0/1554934459 shutdown_connections 2024-03-28T22:43:49.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.072+0000 7fa8991eb740 1 --2- 172.21.15.43:0/1554934459 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56058942c260 0x56058942e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.072+0000 7fa88f400640 1 --2- 172.21.15.43:0/1995509099 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa8740a1f70 0x7fa8740a4430 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x560589422830 tx=0x7fa8800f39e0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:49.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.072+0000 7fa8991eb740 1 --2- 172.21.15.43:0/1554934459 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560589423680 0x56058942bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.072+0000 7fa8991eb740 1 --2- 172.21.15.43:0/1554934459 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560589422d60 0x560589423140 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.072+0000 7fa8991eb740 1 -- 172.21.15.43:0/1554934459 >> 172.21.15.43:0/1554934459 conn(0x56058941d490 msgr2=0x56058941d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:49.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.072+0000 7fa8991eb740 1 -- 172.21.15.43:0/1554934459 shutdown_connections 2024-03-28T22:43:49.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.072+0000 7fa8991eb740 1 -- 172.21.15.43:0/1554934459 wait complete. 2024-03-28T22:43:49.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.073+0000 7fa8991eb740 1 -- start start 2024-03-28T22:43:49.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.073+0000 7fa8991eb740 1 Processor -- start 2024-03-28T22:43:49.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.073+0000 7fa8991eb740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560589422d60 0x560589429320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.074+0000 7fa8991eb740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560589423680 0x560589429860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.074+0000 7fa8991eb740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56058942c260 0x560589429da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.074+0000 7fa8991eb740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56058942f050 con 0x56058942c260 2024-03-28T22:43:49.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.074+0000 7fa8991eb740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x56058931c2f0 con 0x560589422d60 2024-03-28T22:43:49.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.074+0000 7fa8991eb740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x56058942f1d0 con 0x560589423680 2024-03-28T22:43:49.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.074+0000 7fa8991eb740 10 client.0 _ll_register_callbacks cb 0x560589091be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:49.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.074+0000 7fa88ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560589423680 0x560589429860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.074+0000 7fa88ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560589423680 0x560589429860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:60908/0 (socket says 172.21.15.43:60908) 2024-03-28T22:43:49.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.074+0000 7fa88ea00640 1 -- 172.21.15.43:0/2805430069 learned_addr learned my addr 172.21.15.43:0/2805430069 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:49.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.074+0000 7fa88fe00640 1 --2- 172.21.15.43:0/2805430069 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560589422d60 0x560589429320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.074+0000 7fa88fe00640 1 --2- 172.21.15.43:0/2805430069 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56058942c260 0x560589429da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.074+0000 7fa88ea00640 1 -- 172.21.15.43:0/2805430069 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560589422d60 msgr2=0x560589429320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:49.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.074+0000 7fa88ea00640 1 --2- 172.21.15.43:0/2805430069 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560589422d60 0x560589429320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.074+0000 7fa88ea00640 1 -- 172.21.15.43:0/2805430069 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56058942c260 msgr2=0x560589429da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:49.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.074+0000 7fa88ea00640 1 --2- 172.21.15.43:0/2805430069 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56058942c260 0x560589429da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.074+0000 7fa88ea00640 1 -- 172.21.15.43:0/2805430069 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56058915c7d0 con 0x560589423680 2024-03-28T22:43:49.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.074+0000 7fa88fe00640 1 --2- 172.21.15.43:0/2805430069 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56058942c260 0x560589429da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:43:49.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.075+0000 7fa88ea00640 1 --2- 172.21.15.43:0/2805430069 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560589423680 0x560589429860 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa87c012700 tx=0x7fa87c012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bdafe519d7175747 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:49.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.075+0000 7fa879600640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:43:49.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.075+0000 7fa879600640 1 -- 172.21.15.43:0/2805430069 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa87c00f8c0 con 0x560589423680 2024-03-28T22:43:49.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.075+0000 7fa8991eb740 1 -- 172.21.15.43:0/2805430069 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560589327680 con 0x560589423680 2024-03-28T22:43:49.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.075+0000 7fa879600640 1 -- 172.21.15.43:0/2805430069 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fa87c00fa60 con 0x560589423680 2024-03-28T22:43:49.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.075+0000 7fa879600640 1 -- 172.21.15.43:0/2805430069 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa87c025050 con 0x560589423680 2024-03-28T22:43:49.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.075+0000 7fa8991eb740 10 client.27988 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:49.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.075+0000 7fa8991eb740 1 -- 172.21.15.43:0/2805430069 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56058943b3e0 con 0x560589423680 2024-03-28T22:43:49.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.075+0000 7fa879600640 1 -- 172.21.15.43:0/2805430069 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x560589327680 con 0x560589423680 2024-03-28T22:43:49.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.076+0000 7fa879600640 1 -- 172.21.15.43:0/2805430069 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa87c010430 con 0x560589423680 2024-03-28T22:43:49.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.076+0000 7fa8991eb740 10 client.27988 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:49.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.076+0000 7fa8991eb740 10 client.27988 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:43:49.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.076+0000 7fa8991eb740 20 client.27988 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:49.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.076+0000 7fa8991eb740 1 --2- 172.21.15.43:0/2805430069 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x56058943c8f0 0x56058943ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.076+0000 7fa8991eb740 4 client.27988 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:43:49.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.076+0000 7fa8991eb740 1 -- 172.21.15.43:0/2805430069 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56058938b590 con 0x56058943c8f0 2024-03-28T22:43:49.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.076+0000 7fa88fe00640 1 --2- 172.21.15.43:0/2805430069 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x56058943c8f0 0x56058943ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.077+0000 7fa88fe00640 1 --2- 172.21.15.43:0/2805430069 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x56058943c8f0 0x56058943ed30 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ae54dbab50f30216 server_cookie=c9fdc1d6bd8e2b2f in_seq=0 out_seq=0 2024-03-28T22:43:49.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.077+0000 7fa879600640 10 client.27988 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:43:49.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.077+0000 7fa879600640 1 -- 172.21.15.43:0/2805430069 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56058938b590 con 0x56058943c8f0 2024-03-28T22:43:49.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.077+0000 7fa879600640 10 client.27988 handle_command_reply: tid=0 2024-03-28T22:43:49.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.077+0000 7fa8991eb740 1 client.27988 shutdown 2024-03-28T22:43:49.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.078+0000 7fa8991eb740 1 -- 172.21.15.43:0/2805430069 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560589423680 msgr2=0x560589429860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.078+0000 7fa8991eb740 1 --2- 172.21.15.43:0/2805430069 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560589423680 0x560589429860 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa87c012700 tx=0x7fa87c012bc0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.078+0000 7fa8991eb740 1 -- 172.21.15.43:0/2805430069 shutdown_connections 2024-03-28T22:43:49.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.078+0000 7fa8991eb740 1 --2- 172.21.15.43:0/2805430069 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x56058943c8f0 0x56058943ed30 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.078+0000 7fa8991eb740 1 --2- 172.21.15.43:0/2805430069 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56058942c260 0x560589429da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.078+0000 7fa8991eb740 1 --2- 172.21.15.43:0/2805430069 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560589423680 0x560589429860 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.078+0000 7fa8991eb740 1 --2- 172.21.15.43:0/2805430069 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560589422d60 0x560589429320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.078+0000 7fa8991eb740 1 -- 172.21.15.43:0/2805430069 >> 172.21.15.43:0/2805430069 conn(0x56058941d490 msgr2=0x56058941f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:49.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.078+0000 7fa879600640 0 client.27988 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:43:49.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.078+0000 7fa8991eb740 1 -- 172.21.15.43:0/2805430069 shutdown_connections 2024-03-28T22:43:49.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.078+0000 7fa8991eb740 1 -- 172.21.15.43:0/2805430069 wait complete. 2024-03-28T22:43:49.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.078+0000 7fa8991eb740 20 client.27988 trim_cache size 0 max 16384 2024-03-28T22:43:49.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.094+0000 7fa8991eb740 1 Processor -- start 2024-03-28T22:43:49.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.094+0000 7fa8991eb740 1 -- start start 2024-03-28T22:43:49.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.094+0000 7fa8991eb740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560589422d60 0x56058942ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.094+0000 7fa8991eb740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560589423680 0x560589437df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.095+0000 7fa8991eb740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56058942c260 0x560589444600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.095+0000 7fa8991eb740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa8901ac810 con 0x560589423680 2024-03-28T22:43:49.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.095+0000 7fa8991eb740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa8901ae350 con 0x560589422d60 2024-03-28T22:43:49.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.095+0000 7fa8991eb740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fa8901acee0 con 0x56058942c260 2024-03-28T22:43:49.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.095+0000 7fa88ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560589423680 0x560589437df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.095+0000 7fa88fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560589422d60 0x56058942ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.095+0000 7fa88fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560589422d60 0x56058942ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47320/0 (socket says 172.21.15.43:47320) 2024-03-28T22:43:49.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.095+0000 7fa88fe00640 1 -- 172.21.15.43:0/3766088608 learned_addr learned my addr 172.21.15.43:0/3766088608 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:49.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.095+0000 7fa88ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560589423680 0x560589437df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53910/0 (socket says 172.21.15.43:53910) 2024-03-28T22:43:49.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.095+0000 7fa88fe00640 1 --2- 172.21.15.43:0/3766088608 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56058942c260 0x560589444600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.095+0000 7fa88fe00640 1 -- 172.21.15.43:0/3766088608 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56058942c260 msgr2=0x560589444600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:49.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.095+0000 7fa88fe00640 1 --2- 172.21.15.43:0/3766088608 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56058942c260 0x560589444600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.095+0000 7fa88fe00640 1 -- 172.21.15.43:0/3766088608 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560589423680 msgr2=0x560589437df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:49.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.095+0000 7fa88fe00640 1 --2- 172.21.15.43:0/3766088608 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560589423680 0x560589437df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.095+0000 7fa88fe00640 1 -- 172.21.15.43:0/3766088608 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560589438330 con 0x560589422d60 2024-03-28T22:43:49.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.095+0000 7fa88ea00640 1 --2- 172.21.15.43:0/3766088608 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560589423680 0x560589437df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:49.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.096+0000 7fa88fe00640 1 --2- 172.21.15.43:0/3766088608 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560589422d60 0x56058942ba50 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fa888009fd0 tx=0x7fa88801c040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=187903d1ffefeb75 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:49.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.096+0000 7fa87be00640 1 -- 172.21.15.43:0/3766088608 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa888011030 con 0x560589422d60 2024-03-28T22:43:49.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.096+0000 7fa87be00640 1 -- 172.21.15.43:0/3766088608 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa888005a10 con 0x560589422d60 2024-03-28T22:43:49.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.096+0000 7fa87be00640 1 -- 172.21.15.43:0/3766088608 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa888003bd0 con 0x560589422d60 2024-03-28T22:43:49.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.097+0000 7fa8991eb740 1 -- 172.21.15.43:0/3766088608 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560589422d60 msgr2=0x56058942ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.097+0000 7fa8991eb740 1 --2- 172.21.15.43:0/3766088608 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560589422d60 0x56058942ba50 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fa888009fd0 tx=0x7fa88801c040 comp rx=0 tx=0).stop 2024-03-28T22:43:49.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.097+0000 7fa8991eb740 1 -- 172.21.15.43:0/3766088608 shutdown_connections 2024-03-28T22:43:49.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.097+0000 7fa8991eb740 1 --2- 172.21.15.43:0/3766088608 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56058942c260 0x560589444600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.097+0000 7fa8991eb740 1 --2- 172.21.15.43:0/3766088608 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560589423680 0x560589437df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.097+0000 7fa8991eb740 1 --2- 172.21.15.43:0/3766088608 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560589422d60 0x56058942ba50 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.097+0000 7fa8991eb740 1 -- 172.21.15.43:0/3766088608 >> 172.21.15.43:0/3766088608 conn(0x56058941d490 msgr2=0x560589413dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:49.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.097+0000 7fa8991eb740 1 -- 172.21.15.43:0/3766088608 shutdown_connections 2024-03-28T22:43:49.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.097+0000 7fa8991eb740 1 -- 172.21.15.43:0/3766088608 wait complete. 2024-03-28T22:43:49.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.097+0000 7fa8991eb740 1 -- start start 2024-03-28T22:43:49.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.097+0000 7fa8991eb740 1 Processor -- start 2024-03-28T22:43:49.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.098+0000 7fa8991eb740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560589422d60 0x56058942fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.098+0000 7fa8991eb740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560589423680 0x560589430320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.098+0000 7fa8991eb740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56058942c260 0x560589430860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.098+0000 7fa8991eb740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa8901adc80 con 0x56058942c260 2024-03-28T22:43:49.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.098+0000 7fa8991eb740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa8901ad5b0 con 0x560589422d60 2024-03-28T22:43:49.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.098+0000 7fa8991eb740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fa8901aa7f0 con 0x560589423680 2024-03-28T22:43:49.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.098+0000 7fa8991eb740 10 client.0 _ll_register_callbacks cb 0x7fa87c009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:49.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.098+0000 7fa88fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560589422d60 0x56058942fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.098+0000 7fa88ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560589423680 0x560589430320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.098+0000 7fa88fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56058942c260 0x560589430860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.098+0000 7fa88ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560589423680 0x560589430320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:60926/0 (socket says 172.21.15.43:60926) 2024-03-28T22:43:49.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.098+0000 7fa88ea00640 1 -- 172.21.15.43:0/2186227719 learned_addr learned my addr 172.21.15.43:0/2186227719 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:49.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.098+0000 7fa88fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56058942c260 0x560589430860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53916/0 (socket says 172.21.15.43:53916) 2024-03-28T22:43:49.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.099+0000 7fa88ea00640 1 -- 172.21.15.43:0/2186227719 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560589422d60 msgr2=0x56058942fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:49.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.099+0000 7fa88ea00640 1 --2- 172.21.15.43:0/2186227719 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560589422d60 0x56058942fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.099+0000 7fa88ea00640 1 -- 172.21.15.43:0/2186227719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56058942c260 msgr2=0x560589430860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:49.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.099+0000 7fa88ea00640 1 --2- 172.21.15.43:0/2186227719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56058942c260 0x560589430860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.099+0000 7fa88ea00640 1 -- 172.21.15.43:0/2186227719 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605894a1c60 con 0x560589423680 2024-03-28T22:43:49.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.099+0000 7fa88ea00640 1 --2- 172.21.15.43:0/2186227719 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560589423680 0x560589430320 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fa87c012420 tx=0x7fa87c009f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c06e08bbee49ce92 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:49.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.099+0000 7fa879600640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:43:49.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.099+0000 7fa879600640 1 -- 172.21.15.43:0/2186227719 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa87c010a40 con 0x560589423680 2024-03-28T22:43:49.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.099+0000 7fa879600640 1 -- 172.21.15.43:0/2186227719 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fa87c022040 con 0x560589423680 2024-03-28T22:43:49.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.099+0000 7fa8991eb740 1 -- 172.21.15.43:0/2186227719 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5605893858e0 con 0x560589423680 2024-03-28T22:43:49.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.100+0000 7fa879600640 1 -- 172.21.15.43:0/2186227719 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa87c032050 con 0x560589423680 2024-03-28T22:43:49.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.100+0000 7fa8991eb740 10 client.27994 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:49.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.100+0000 7fa8991eb740 1 -- 172.21.15.43:0/2186227719 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5605894a2f70 con 0x560589423680 2024-03-28T22:43:49.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.100+0000 7fa879600640 1 -- 172.21.15.43:0/2186227719 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5605893858e0 con 0x560589423680 2024-03-28T22:43:49.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.100+0000 7fa879600640 1 -- 172.21.15.43:0/2186227719 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa87c00f920 con 0x560589423680 2024-03-28T22:43:49.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.100+0000 7fa8991eb740 10 client.27994 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:49.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.100+0000 7fa8991eb740 10 client.27994 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:43:49.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.100+0000 7fa8991eb740 20 client.27994 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:49.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.100+0000 7fa8991eb740 1 --2- 172.21.15.43:0/2186227719 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x5605894a3720 0x5605894a5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.100+0000 7fa8991eb740 4 client.27994 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:43:49.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.100+0000 7fa8991eb740 1 -- 172.21.15.43:0/2186227719 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560589303560 con 0x5605894a3720 2024-03-28T22:43:49.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.100+0000 7fa88fe00640 1 --2- 172.21.15.43:0/2186227719 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x5605894a3720 0x5605894a5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.101+0000 7fa88fe00640 1 --2- 172.21.15.43:0/2186227719 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x5605894a3720 0x5605894a5b40 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8a18e667a91be99d server_cookie=32e178029596afb6 in_seq=0 out_seq=0 2024-03-28T22:43:49.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.101+0000 7fa879600640 10 client.27994 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:43:49.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.101+0000 7fa879600640 1 -- 172.21.15.43:0/2186227719 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560589303560 con 0x5605894a3720 2024-03-28T22:43:49.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.101+0000 7fa879600640 10 client.27994 handle_command_reply: tid=0 2024-03-28T22:43:49.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.101+0000 7fa8991eb740 1 client.27994 shutdown 2024-03-28T22:43:49.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.101+0000 7fa8991eb740 1 -- 172.21.15.43:0/2186227719 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560589423680 msgr2=0x560589430320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.101+0000 7fa8991eb740 1 --2- 172.21.15.43:0/2186227719 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560589423680 0x560589430320 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fa87c012420 tx=0x7fa87c009f70 comp rx=0 tx=0).stop 2024-03-28T22:43:49.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.101+0000 7fa8991eb740 1 -- 172.21.15.43:0/2186227719 shutdown_connections 2024-03-28T22:43:49.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.101+0000 7fa8991eb740 1 --2- 172.21.15.43:0/2186227719 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x5605894a3720 0x5605894a5b40 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.101+0000 7fa8991eb740 1 --2- 172.21.15.43:0/2186227719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56058942c260 0x560589430860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.101+0000 7fa8991eb740 1 --2- 172.21.15.43:0/2186227719 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560589423680 0x560589430320 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.101+0000 7fa8991eb740 1 --2- 172.21.15.43:0/2186227719 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560589422d60 0x56058942fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.101+0000 7fa879600640 0 client.27994 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:43:49.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.101+0000 7fa8991eb740 1 -- 172.21.15.43:0/2186227719 >> 172.21.15.43:0/2186227719 conn(0x56058941d490 msgr2=0x560589413530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:49.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.102+0000 7fa8991eb740 1 -- 172.21.15.43:0/2186227719 shutdown_connections 2024-03-28T22:43:49.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.102+0000 7fa8991eb740 1 -- 172.21.15.43:0/2186227719 wait complete. 2024-03-28T22:43:49.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.102+0000 7fa8991eb740 20 client.27994 trim_cache size 0 max 16384 2024-03-28T22:43:49.105 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:43:49.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.102+0000 7fa894e00640 1 -- 172.21.15.43:0/1995509099 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa8740a1f70 msgr2=0x7fa8740a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.102+0000 7fa894e00640 1 --2- 172.21.15.43:0/1995509099 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa8740a1f70 0x7fa8740a4430 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x560589422830 tx=0x7fa8800f39e0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.102+0000 7fa894e00640 1 -- 172.21.15.43:0/1995509099 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa890073b80 msgr2=0x7fa89019a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.103+0000 7fa894e00640 1 --2- 172.21.15.43:0/1995509099 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa890073b80 0x7fa89019a0d0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fa880005940 tx=0x7fa88002cbd0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.103+0000 7fa894e00640 1 -- 172.21.15.43:0/1995509099 shutdown_connections 2024-03-28T22:43:49.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.103+0000 7fa894e00640 1 --2- 172.21.15.43:0/1995509099 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa8740a1f70 0x7fa8740a4430 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.103+0000 7fa894e00640 1 --2- 172.21.15.43:0/1995509099 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa89010ddd0 0x7fa89019e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.103+0000 7fa894e00640 1 --2- 172.21.15.43:0/1995509099 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8900744a0 0x7fa89019a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.103+0000 7fa894e00640 1 --2- 172.21.15.43:0/1995509099 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa890073b80 0x7fa89019a0d0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.103+0000 7fa894e00640 1 -- 172.21.15.43:0/1995509099 >> 172.21.15.43:0/1995509099 conn(0x7fa8900fd0c0 msgr2=0x7fa8900fe980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:49.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.103+0000 7fa894e00640 1 -- 172.21.15.43:0/1995509099 shutdown_connections 2024-03-28T22:43:49.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.103+0000 7fa894e00640 1 -- 172.21.15.43:0/1995509099 wait complete. 2024-03-28T22:43:49.120 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:49 smithi043 ceph-mon[31823]: pgmap v203: 129 pgs: 129 active+clean; 51 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 262 op/s 2024-03-28T22:43:49.120 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:49 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/806575601' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:43:49.123 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T22:43:49.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.358+0000 7fcd89600640 1 -- 172.21.15.43:0/1752427086 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd84074410 msgr2=0x7fcd841121a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.358+0000 7fcd89600640 1 --2- 172.21.15.43:0/1752427086 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd84074410 0x7fcd841121a0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fcd70009860 tx=0x7fcd7002bc00 comp rx=0 tx=0).stop 2024-03-28T22:43:49.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.358+0000 7fcd89600640 1 -- 172.21.15.43:0/1752427086 shutdown_connections 2024-03-28T22:43:49.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.358+0000 7fcd89600640 1 --2- 172.21.15.43:0/1752427086 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd84074410 0x7fcd841121a0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.358+0000 7fcd89600640 1 --2- 172.21.15.43:0/1752427086 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd841126e0 0x7fcd84114ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.358+0000 7fcd89600640 1 --2- 172.21.15.43:0/1752427086 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd84073af0 0x7fcd84073ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.358+0000 7fcd89600640 1 -- 172.21.15.43:0/1752427086 >> 172.21.15.43:0/1752427086 conn(0x7fcd840fd1e0 msgr2=0x7fcd840ff600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:49.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.358+0000 7fcd89600640 1 -- 172.21.15.43:0/1752427086 shutdown_connections 2024-03-28T22:43:49.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.358+0000 7fcd89600640 1 -- 172.21.15.43:0/1752427086 wait complete. 2024-03-28T22:43:49.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.360+0000 7fcd89600640 1 Processor -- start 2024-03-28T22:43:49.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.360+0000 7fcd89600640 1 -- start start 2024-03-28T22:43:49.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.360+0000 7fcd89600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd84073af0 0x7fcd8419e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.360+0000 7fcd89600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd84074410 0x7fcd8419eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.360+0000 7fcd89600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd841126e0 0x7fcd841a2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.360+0000 7fcd89600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcd841180a0 con 0x7fcd84073af0 2024-03-28T22:43:49.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.360+0000 7fcd89600640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fcd84117f20 con 0x7fcd841126e0 2024-03-28T22:43:49.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.360+0000 7fcd89600640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fcd84118220 con 0x7fcd84074410 2024-03-28T22:43:49.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.361+0000 7fcd82000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd84074410 0x7fcd8419eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.361+0000 7fcd82000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd84074410 0x7fcd8419eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:60944/0 (socket says 172.21.15.43:60944) 2024-03-28T22:43:49.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.361+0000 7fcd82000640 1 -- 172.21.15.43:0/3545701552 learned_addr learned my addr 172.21.15.43:0/3545701552 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:49.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.361+0000 7fcd82a00640 1 --2- 172.21.15.43:0/3545701552 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd84073af0 0x7fcd8419e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.361+0000 7fcd83400640 1 --2- 172.21.15.43:0/3545701552 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd841126e0 0x7fcd841a2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.361+0000 7fcd82000640 1 -- 172.21.15.43:0/3545701552 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd841126e0 msgr2=0x7fcd841a2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.361+0000 7fcd82000640 1 --2- 172.21.15.43:0/3545701552 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd841126e0 0x7fcd841a2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.361+0000 7fcd82000640 1 -- 172.21.15.43:0/3545701552 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd84073af0 msgr2=0x7fcd8419e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.361+0000 7fcd82000640 1 --2- 172.21.15.43:0/3545701552 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd84073af0 0x7fcd8419e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.361+0000 7fcd82000640 1 -- 172.21.15.43:0/3545701552 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd841a3650 con 0x7fcd84074410 2024-03-28T22:43:49.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.362+0000 7fcd82000640 1 --2- 172.21.15.43:0/3545701552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd84074410 0x7fcd8419eba0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fcd70037040 tx=0x7fcd700344c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:49.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.362+0000 7fcd7b400640 1 -- 172.21.15.43:0/3545701552 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd70041020 con 0x7fcd84074410 2024-03-28T22:43:49.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.362+0000 7fcd7b400640 1 -- 172.21.15.43:0/3545701552 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd7002ca80 con 0x7fcd84074410 2024-03-28T22:43:49.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.362+0000 7fcd89600640 1 -- 172.21.15.43:0/3545701552 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd841a38e0 con 0x7fcd84074410 2024-03-28T22:43:49.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.362+0000 7fcd7b400640 1 -- 172.21.15.43:0/3545701552 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd70039040 con 0x7fcd84074410 2024-03-28T22:43:49.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.362+0000 7fcd89600640 1 -- 172.21.15.43:0/3545701552 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd841ab1d0 con 0x7fcd84074410 2024-03-28T22:43:49.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.366+0000 7fcd8c223740 1 Processor -- start 2024-03-28T22:43:49.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.366+0000 7fcd8c223740 1 -- start start 2024-03-28T22:43:49.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.366+0000 7fcd8c223740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fc1321380 0x562fc1321760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.366+0000 7fcd8c223740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fc1321ca0 0x562fc132a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.366+0000 7fcd8c223740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fc132a880 0x562fc132cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.366+0000 7fcd8c223740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562fc1285050 con 0x562fc132a880 2024-03-28T22:43:49.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.366+0000 7fcd8c223740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x562fc122dce0 con 0x562fc1321ca0 2024-03-28T22:43:49.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.366+0000 7fcd8c223740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x562fc12810f0 con 0x562fc1321380 2024-03-28T22:43:49.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.366+0000 7fcd82a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fc1321ca0 0x562fc132a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.366+0000 7fcd7b400640 1 -- 172.21.15.43:0/3545701552 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fcd70044020 con 0x7fcd84074410 2024-03-28T22:43:49.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.366+0000 7fcd82a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fc1321ca0 0x562fc132a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47358/0 (socket says 172.21.15.43:47358) 2024-03-28T22:43:49.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.366+0000 7fcd83400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fc1321380 0x562fc1321760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.366+0000 7fcd82a00640 1 -- 172.21.15.43:0/385104953 learned_addr learned my addr 172.21.15.43:0/385104953 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:49.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.367+0000 7fcd82000640 1 --2- 172.21.15.43:0/385104953 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fc132a880 0x562fc132cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.367+0000 7fcd82a00640 1 -- 172.21.15.43:0/385104953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fc1321380 msgr2=0x562fc1321760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:49.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.367+0000 7fcd82a00640 1 --2- 172.21.15.43:0/385104953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fc1321380 0x562fc1321760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.367+0000 7fcd82a00640 1 -- 172.21.15.43:0/385104953 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fc132a880 msgr2=0x562fc132cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:49.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.367+0000 7fcd82a00640 1 --2- 172.21.15.43:0/385104953 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fc132a880 0x562fc132cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.367+0000 7fcd82a00640 1 -- 172.21.15.43:0/385104953 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fc0f993f0 con 0x562fc1321ca0 2024-03-28T22:43:49.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.367+0000 7fcd82000640 1 --2- 172.21.15.43:0/385104953 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fc132a880 0x562fc132cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:43:49.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.367+0000 7fcd82a00640 1 --2- 172.21.15.43:0/385104953 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fc1321ca0 0x562fc132a340 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fcd74009cf0 tx=0x7fcd74011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5be1e1d525dced10 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:49.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.368+0000 7fcd6f400640 1 -- 172.21.15.43:0/385104953 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd74004230 con 0x562fc1321ca0 2024-03-28T22:43:49.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.368+0000 7fcd6f400640 1 -- 172.21.15.43:0/385104953 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd740043d0 con 0x562fc1321ca0 2024-03-28T22:43:49.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.368+0000 7fcd6f400640 1 -- 172.21.15.43:0/385104953 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd74019960 con 0x562fc1321ca0 2024-03-28T22:43:49.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.368+0000 7fcd7b400640 1 --2- 172.21.15.43:0/3545701552 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcd680a1f70 0x7fcd680a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.368+0000 7fcd8c223740 1 -- 172.21.15.43:0/385104953 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fc1321ca0 msgr2=0x562fc132a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.368+0000 7fcd8c223740 1 --2- 172.21.15.43:0/385104953 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fc1321ca0 0x562fc132a340 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fcd74009cf0 tx=0x7fcd74011810 comp rx=0 tx=0).stop 2024-03-28T22:43:49.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.368+0000 7fcd7b400640 1 -- 172.21.15.43:0/3545701552 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fcd700efb00 con 0x7fcd84074410 2024-03-28T22:43:49.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.368+0000 7fcd8c223740 1 -- 172.21.15.43:0/385104953 shutdown_connections 2024-03-28T22:43:49.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.368+0000 7fcd8c223740 1 --2- 172.21.15.43:0/385104953 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fc1321ca0 0x562fc132a340 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.368+0000 7fcd8c223740 1 --2- 172.21.15.43:0/385104953 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fc132a880 0x562fc132cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.368+0000 7fcd8c223740 1 --2- 172.21.15.43:0/385104953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fc1321380 0x562fc1321760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.368+0000 7fcd8c223740 1 -- 172.21.15.43:0/385104953 >> 172.21.15.43:0/385104953 conn(0x562fc131bab0 msgr2=0x562fc131be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:49.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.369+0000 7fcd83400640 1 --2- 172.21.15.43:0/3545701552 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcd680a1f70 0x7fcd680a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.369+0000 7fcd8c223740 1 -- 172.21.15.43:0/385104953 shutdown_connections 2024-03-28T22:43:49.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.369+0000 7fcd8c223740 1 -- 172.21.15.43:0/385104953 wait complete. 2024-03-28T22:43:49.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.369+0000 7fcd8c223740 1 -- start start 2024-03-28T22:43:49.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.369+0000 7fcd83400640 1 --2- 172.21.15.43:0/3545701552 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcd680a1f70 0x7fcd680a4430 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcd8406adc0 tx=0x7fcd7c010040 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:49.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.369+0000 7fcd8c223740 1 Processor -- start 2024-03-28T22:43:49.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.370+0000 7fcd8c223740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fc1321380 0x562fc1327840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.370+0000 7fcd8c223740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fc1321ca0 0x562fc1327d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.370+0000 7fcd8c223740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fc132a880 0x562fc13282c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.370+0000 7fcd8c223740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562fc132d670 con 0x562fc1321380 2024-03-28T22:43:49.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.370+0000 7fcd8c223740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x562fc12232f0 con 0x562fc132a880 2024-03-28T22:43:49.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.370+0000 7fcd8c223740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x562fc132d7f0 con 0x562fc1321ca0 2024-03-28T22:43:49.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.370+0000 7fcd8c223740 10 client.0 _ll_register_callbacks cb 0x562fc0f98be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:49.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.370+0000 7fcd83400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fc132a880 0x562fc13282c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.370+0000 7fcd82a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fc1321380 0x562fc1327840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.370+0000 7fcd82000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fc1321ca0 0x562fc1327d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.370+0000 7fcd83400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fc132a880 0x562fc13282c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47370/0 (socket says 172.21.15.43:47370) 2024-03-28T22:43:49.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.370+0000 7fcd83400640 1 -- 172.21.15.43:0/622813712 learned_addr learned my addr 172.21.15.43:0/622813712 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:49.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.370+0000 7fcd82000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fc1321ca0 0x562fc1327d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:60962/0 (socket says 172.21.15.43:60962) 2024-03-28T22:43:49.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.371+0000 7fcd83400640 1 -- 172.21.15.43:0/622813712 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fc1321ca0 msgr2=0x562fc1327d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:49.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.371+0000 7fcd83400640 1 --2- 172.21.15.43:0/622813712 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fc1321ca0 0x562fc1327d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.371+0000 7fcd83400640 1 -- 172.21.15.43:0/622813712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fc1321380 msgr2=0x562fc1327840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:49.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.371+0000 7fcd83400640 1 --2- 172.21.15.43:0/622813712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fc1321380 0x562fc1327840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.371+0000 7fcd83400640 1 -- 172.21.15.43:0/622813712 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fc10637d0 con 0x562fc132a880 2024-03-28T22:43:49.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.371+0000 7fcd82000640 1 --2- 172.21.15.43:0/622813712 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fc1321ca0 0x562fc1327d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:49.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.371+0000 7fcd82a00640 1 --2- 172.21.15.43:0/622813712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fc1321380 0x562fc1327840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:49.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.371+0000 7fcd83400640 1 --2- 172.21.15.43:0/622813712 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fc132a880 0x562fc13282c0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c00df40 tx=0x7fcd7c004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20b787ec5f9bb313 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:49.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.371+0000 7fcd63e00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:43:49.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.371+0000 7fcd63e00640 1 -- 172.21.15.43:0/622813712 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c018630 con 0x562fc132a880 2024-03-28T22:43:49.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.371+0000 7fcd8c223740 1 -- 172.21.15.43:0/622813712 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd740043d0 con 0x562fc132a880 2024-03-28T22:43:49.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.371+0000 7fcd63e00640 1 -- 172.21.15.43:0/622813712 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fcd7c0187d0 con 0x562fc132a880 2024-03-28T22:43:49.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.372+0000 7fcd63e00640 1 -- 172.21.15.43:0/622813712 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcd7c033050 con 0x562fc132a880 2024-03-28T22:43:49.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.372+0000 7fcd8c223740 10 client.28064 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:49.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.372+0000 7fcd8c223740 1 -- 172.21.15.43:0/622813712 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562fc13398a0 con 0x562fc132a880 2024-03-28T22:43:49.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.372+0000 7fcd63e00640 1 -- 172.21.15.43:0/622813712 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd740043d0 con 0x562fc132a880 2024-03-28T22:43:49.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.372+0000 7fcd63e00640 1 -- 172.21.15.43:0/622813712 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fcd7c024070 con 0x562fc132a880 2024-03-28T22:43:49.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.372+0000 7fcd8c223740 10 client.28064 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:49.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.372+0000 7fcd8c223740 10 client.28064 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:43:49.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.372+0000 7fcd8c223740 20 client.28064 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:49.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.372+0000 7fcd8c223740 1 --2- 172.21.15.43:0/622813712 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x562fc133ada0 0x562fc133d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.372+0000 7fcd8c223740 4 client.28064 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:43:49.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.372+0000 7fcd8c223740 1 -- 172.21.15.43:0/622813712 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562fc122e680 con 0x562fc133ada0 2024-03-28T22:43:49.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.373+0000 7fcd82a00640 1 --2- 172.21.15.43:0/622813712 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x562fc133ada0 0x562fc133d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.373+0000 7fcd82a00640 1 --2- 172.21.15.43:0/622813712 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x562fc133ada0 0x562fc133d1f0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4345457fe49af67 server_cookie=fabb893c71737450 in_seq=0 out_seq=0 2024-03-28T22:43:49.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.373+0000 7fcd63e00640 10 client.28064 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:43:49.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.374+0000 7fcd63e00640 1 -- 172.21.15.43:0/622813712 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562fc122e680 con 0x562fc133ada0 2024-03-28T22:43:49.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.374+0000 7fcd63e00640 10 client.28064 handle_command_reply: tid=0 2024-03-28T22:43:49.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.374+0000 7fcd8c223740 1 client.28064 shutdown 2024-03-28T22:43:49.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.374+0000 7fcd8c223740 1 -- 172.21.15.43:0/622813712 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fc132a880 msgr2=0x562fc13282c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.374+0000 7fcd8c223740 1 --2- 172.21.15.43:0/622813712 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fc132a880 0x562fc13282c0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c00df40 tx=0x7fcd7c004ab0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.374+0000 7fcd8c223740 1 -- 172.21.15.43:0/622813712 shutdown_connections 2024-03-28T22:43:49.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.374+0000 7fcd8c223740 1 --2- 172.21.15.43:0/622813712 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x562fc133ada0 0x562fc133d1f0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.374+0000 7fcd8c223740 1 --2- 172.21.15.43:0/622813712 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fc132a880 0x562fc13282c0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.374+0000 7fcd8c223740 1 --2- 172.21.15.43:0/622813712 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fc1321ca0 0x562fc1327d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.374+0000 7fcd63e00640 0 client.28064 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:43:49.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.374+0000 7fcd8c223740 1 --2- 172.21.15.43:0/622813712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fc1321380 0x562fc1327840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.374+0000 7fcd8c223740 1 -- 172.21.15.43:0/622813712 >> 172.21.15.43:0/622813712 conn(0x562fc131bab0 msgr2=0x562fc131d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:49.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.375+0000 7fcd8c223740 1 -- 172.21.15.43:0/622813712 shutdown_connections 2024-03-28T22:43:49.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.375+0000 7fcd8c223740 1 -- 172.21.15.43:0/622813712 wait complete. 2024-03-28T22:43:49.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.375+0000 7fcd8c223740 20 client.28064 trim_cache size 0 max 16384 2024-03-28T22:43:49.378 INFO:tasks.workunit.client.0.smithi043.stdout: 1 456006 7.28 MB/sec execute 200 sec latency 22.226 ms 2024-03-28T22:43:49.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.390+0000 7fcd8c223740 1 Processor -- start 2024-03-28T22:43:49.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.390+0000 7fcd8c223740 1 -- start start 2024-03-28T22:43:49.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.391+0000 7fcd8c223740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fc1321380 0x562fc1329f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.391+0000 7fcd8c223740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fc1321ca0 0x562fc1336440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.391+0000 7fcd8c223740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fc132a880 0x562fc1342ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.391+0000 7fcd8c223740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcd841b5210 con 0x562fc132a880 2024-03-28T22:43:49.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.391+0000 7fcd8c223740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fcd841b6d50 con 0x562fc1321380 2024-03-28T22:43:49.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.391+0000 7fcd8c223740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fcd841b58e0 con 0x562fc1321ca0 2024-03-28T22:43:49.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.391+0000 7fcd82a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fc1321380 0x562fc1329f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.391+0000 7fcd82a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fc1321380 0x562fc1329f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47376/0 (socket says 172.21.15.43:47376) 2024-03-28T22:43:49.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.391+0000 7fcd82a00640 1 -- 172.21.15.43:0/748377117 learned_addr learned my addr 172.21.15.43:0/748377117 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:49.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.391+0000 7fcd82000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fc1321ca0 0x562fc1336440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.391+0000 7fcd83400640 1 --2- 172.21.15.43:0/748377117 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fc132a880 0x562fc1342ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.391+0000 7fcd82000640 1 -- 172.21.15.43:0/748377117 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fc1321380 msgr2=0x562fc1329f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:49.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.391+0000 7fcd82000640 1 --2- 172.21.15.43:0/748377117 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fc1321380 0x562fc1329f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.391+0000 7fcd82000640 1 -- 172.21.15.43:0/748377117 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fc132a880 msgr2=0x562fc1342ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:49.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.391+0000 7fcd82000640 1 --2- 172.21.15.43:0/748377117 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fc132a880 0x562fc1342ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.392+0000 7fcd82000640 1 -- 172.21.15.43:0/748377117 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fc131b8e0 con 0x562fc1321ca0 2024-03-28T22:43:49.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.392+0000 7fcd83400640 1 --2- 172.21.15.43:0/748377117 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fc132a880 0x562fc1342ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:43:49.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.392+0000 7fcd82000640 1 --2- 172.21.15.43:0/748377117 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fc1321ca0 0x562fc1336440 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fcd700ecf60 tx=0x7fcd70005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=291bebae2bacd7c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:49.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.392+0000 7fcd6ea00640 1 -- 172.21.15.43:0/748377117 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd700c0130 con 0x562fc1321ca0 2024-03-28T22:43:49.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.392+0000 7fcd6ea00640 1 -- 172.21.15.43:0/748377117 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd700bc200 con 0x562fc1321ca0 2024-03-28T22:43:49.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.392+0000 7fcd6ea00640 1 -- 172.21.15.43:0/748377117 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd700bc460 con 0x562fc1321ca0 2024-03-28T22:43:49.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.393+0000 7fcd8c223740 1 -- 172.21.15.43:0/748377117 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fc1321ca0 msgr2=0x562fc1336440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.393+0000 7fcd8c223740 1 --2- 172.21.15.43:0/748377117 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fc1321ca0 0x562fc1336440 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fcd700ecf60 tx=0x7fcd70005740 comp rx=0 tx=0).stop 2024-03-28T22:43:49.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.393+0000 7fcd8c223740 1 -- 172.21.15.43:0/748377117 shutdown_connections 2024-03-28T22:43:49.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.393+0000 7fcd8c223740 1 --2- 172.21.15.43:0/748377117 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fc132a880 0x562fc1342ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.393+0000 7fcd8c223740 1 --2- 172.21.15.43:0/748377117 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fc1321ca0 0x562fc1336440 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.393+0000 7fcd8c223740 1 --2- 172.21.15.43:0/748377117 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fc1321380 0x562fc1329f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.393+0000 7fcd8c223740 1 -- 172.21.15.43:0/748377117 >> 172.21.15.43:0/748377117 conn(0x562fc131bab0 msgr2=0x562fc1327800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:49.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.393+0000 7fcd8c223740 1 -- 172.21.15.43:0/748377117 shutdown_connections 2024-03-28T22:43:49.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.393+0000 7fcd8c223740 1 -- 172.21.15.43:0/748377117 wait complete. 2024-03-28T22:43:49.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.393+0000 7fcd8c223740 1 -- start start 2024-03-28T22:43:49.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.393+0000 7fcd8c223740 1 Processor -- start 2024-03-28T22:43:49.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.394+0000 7fcd8c223740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fc1321380 0x562fc132e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.394+0000 7fcd8c223740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fc1321ca0 0x562fc132e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.394+0000 7fcd8c223740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fc132a880 0x562fc132edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.394+0000 7fcd8c223740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcd841b6680 con 0x562fc1321ca0 2024-03-28T22:43:49.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.394+0000 7fcd8c223740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fcd841b5fb0 con 0x562fc1321380 2024-03-28T22:43:49.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.394+0000 7fcd8c223740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fcd841b31f0 con 0x562fc132a880 2024-03-28T22:43:49.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.394+0000 7fcd8c223740 10 client.0 _ll_register_callbacks cb 0x7fcd7c00dbc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:49.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.394+0000 7fcd82a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fc1321380 0x562fc132e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.394+0000 7fcd82a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fc1321380 0x562fc132e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47382/0 (socket says 172.21.15.43:47382) 2024-03-28T22:43:49.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.394+0000 7fcd82a00640 1 -- 172.21.15.43:0/2082995409 learned_addr learned my addr 172.21.15.43:0/2082995409 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:49.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.394+0000 7fcd83400640 1 --2- 172.21.15.43:0/2082995409 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fc132a880 0x562fc132edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.394+0000 7fcd82000640 1 --2- 172.21.15.43:0/2082995409 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fc1321ca0 0x562fc132e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.395+0000 7fcd82a00640 1 -- 172.21.15.43:0/2082995409 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fc132a880 msgr2=0x562fc132edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:49.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.395+0000 7fcd82a00640 1 --2- 172.21.15.43:0/2082995409 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fc132a880 0x562fc132edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.395+0000 7fcd82a00640 1 -- 172.21.15.43:0/2082995409 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fc1321ca0 msgr2=0x562fc132e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:49.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.395+0000 7fcd82a00640 1 --2- 172.21.15.43:0/2082995409 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fc1321ca0 0x562fc132e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.395+0000 7fcd82a00640 1 -- 172.21.15.43:0/2082995409 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fc13a00f0 con 0x562fc1321380 2024-03-28T22:43:49.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.395+0000 7fcd82a00640 1 --2- 172.21.15.43:0/2082995409 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fc1321380 0x562fc132e350 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fcd74019460 tx=0x7fcd74003f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2de66d722ea03be server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:49.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.395+0000 7fcd63e00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:43:49.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.395+0000 7fcd63e00640 1 -- 172.21.15.43:0/2082995409 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd7402bc30 con 0x562fc1321380 2024-03-28T22:43:49.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.395+0000 7fcd63e00640 1 -- 172.21.15.43:0/2082995409 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fcd7402bdd0 con 0x562fc1321380 2024-03-28T22:43:49.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.395+0000 7fcd8c223740 1 -- 172.21.15.43:0/2082995409 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd700bc200 con 0x562fc1321380 2024-03-28T22:43:49.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.395+0000 7fcd63e00640 1 -- 172.21.15.43:0/2082995409 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcd7403b050 con 0x562fc1321380 2024-03-28T22:43:49.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.395+0000 7fcd8c223740 10 client.28070 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:49.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.396+0000 7fcd8c223740 1 -- 172.21.15.43:0/2082995409 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562fc13a13c0 con 0x562fc1321380 2024-03-28T22:43:49.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.396+0000 7fcd63e00640 1 -- 172.21.15.43:0/2082995409 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd700bc200 con 0x562fc1321380 2024-03-28T22:43:49.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.396+0000 7fcd63e00640 1 -- 172.21.15.43:0/2082995409 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fcd740035f0 con 0x562fc1321380 2024-03-28T22:43:49.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.396+0000 7fcd8c223740 10 client.28070 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:49.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.396+0000 7fcd8c223740 10 client.28070 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:43:49.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.396+0000 7fcd8c223740 20 client.28070 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:49.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.396+0000 7fcd8c223740 1 --2- 172.21.15.43:0/2082995409 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x562fc13a1980 0x562fc13a3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.396+0000 7fcd8c223740 4 client.28070 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:43:49.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.396+0000 7fcd8c223740 1 -- 172.21.15.43:0/2082995409 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562fc1292590 con 0x562fc13a1980 2024-03-28T22:43:49.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.396+0000 7fcd82000640 1 --2- 172.21.15.43:0/2082995409 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x562fc13a1980 0x562fc13a3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.397+0000 7fcd82000640 1 --2- 172.21.15.43:0/2082995409 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x562fc13a1980 0x562fc13a3da0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=179d75d866d233f5 server_cookie=24d222972c71edb7 in_seq=0 out_seq=0 2024-03-28T22:43:49.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.397+0000 7fcd63e00640 10 client.28070 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:43:49.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.397+0000 7fcd63e00640 1 -- 172.21.15.43:0/2082995409 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562fc1292590 con 0x562fc13a1980 2024-03-28T22:43:49.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.397+0000 7fcd63e00640 10 client.28070 handle_command_reply: tid=0 2024-03-28T22:43:49.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.397+0000 7fcd8c223740 1 client.28070 shutdown 2024-03-28T22:43:49.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.397+0000 7fcd8c223740 1 -- 172.21.15.43:0/2082995409 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fc1321380 msgr2=0x562fc132e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.397+0000 7fcd8c223740 1 --2- 172.21.15.43:0/2082995409 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fc1321380 0x562fc132e350 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fcd74019460 tx=0x7fcd74003f80 comp rx=0 tx=0).stop 2024-03-28T22:43:49.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.397+0000 7fcd8c223740 1 -- 172.21.15.43:0/2082995409 shutdown_connections 2024-03-28T22:43:49.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.397+0000 7fcd8c223740 1 --2- 172.21.15.43:0/2082995409 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x562fc13a1980 0x562fc13a3da0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.397+0000 7fcd8c223740 1 --2- 172.21.15.43:0/2082995409 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562fc132a880 0x562fc132edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.397+0000 7fcd8c223740 1 --2- 172.21.15.43:0/2082995409 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562fc1321ca0 0x562fc132e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.397+0000 7fcd8c223740 1 --2- 172.21.15.43:0/2082995409 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562fc1321380 0x562fc132e350 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.397+0000 7fcd63e00640 0 client.28070 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:43:49.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.397+0000 7fcd8c223740 1 -- 172.21.15.43:0/2082995409 >> 172.21.15.43:0/2082995409 conn(0x562fc131bab0 msgr2=0x562fc131ab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:49.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.398+0000 7fcd8c223740 1 -- 172.21.15.43:0/2082995409 shutdown_connections 2024-03-28T22:43:49.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.398+0000 7fcd8c223740 1 -- 172.21.15.43:0/2082995409 wait complete. 2024-03-28T22:43:49.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.398+0000 7fcd8c223740 20 client.28070 trim_cache size 0 max 16384 2024-03-28T22:43:49.400 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:43:49.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.398+0000 7fcd89600640 1 -- 172.21.15.43:0/3545701552 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcd680a1f70 msgr2=0x7fcd680a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.398+0000 7fcd89600640 1 --2- 172.21.15.43:0/3545701552 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcd680a1f70 0x7fcd680a4430 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcd8406adc0 tx=0x7fcd7c010040 comp rx=0 tx=0).stop 2024-03-28T22:43:49.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.398+0000 7fcd89600640 1 -- 172.21.15.43:0/3545701552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd84074410 msgr2=0x7fcd8419eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.398+0000 7fcd89600640 1 --2- 172.21.15.43:0/3545701552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd84074410 0x7fcd8419eba0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fcd70037040 tx=0x7fcd700344c0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.398+0000 7fcd89600640 1 -- 172.21.15.43:0/3545701552 shutdown_connections 2024-03-28T22:43:49.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.398+0000 7fcd89600640 1 --2- 172.21.15.43:0/3545701552 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcd680a1f70 0x7fcd680a4430 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.398+0000 7fcd89600640 1 --2- 172.21.15.43:0/3545701552 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd841126e0 0x7fcd841a2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.398+0000 7fcd89600640 1 --2- 172.21.15.43:0/3545701552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcd84074410 0x7fcd8419eba0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.398+0000 7fcd89600640 1 --2- 172.21.15.43:0/3545701552 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcd84073af0 0x7fcd8419e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.398+0000 7fcd89600640 1 -- 172.21.15.43:0/3545701552 >> 172.21.15.43:0/3545701552 conn(0x7fcd840fd1e0 msgr2=0x7fcd840feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:49.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.399+0000 7fcd89600640 1 -- 172.21.15.43:0/3545701552 shutdown_connections 2024-03-28T22:43:49.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.399+0000 7fcd89600640 1 -- 172.21.15.43:0/3545701552 wait complete. 2024-03-28T22:43:49.402 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:49 smithi172 ceph-mon[36566]: pgmap v203: 129 pgs: 129 active+clean; 51 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 262 op/s 2024-03-28T22:43:49.402 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:49 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/806575601' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:43:49.414 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:43:49.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:49 smithi082 ceph-mon[36471]: pgmap v203: 129 pgs: 129 active+clean; 51 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 262 op/s 2024-03-28T22:43:49.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:49 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/806575601' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:43:49.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.647+0000 7f6a6d200640 1 -- 172.21.15.43:0/3835502930 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a68100010 msgr2=0x7f6a68100470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.647+0000 7f6a6d200640 1 --2- 172.21.15.43:0/3835502930 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a68100010 0x7f6a68100470 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f6a54009930 tx=0x7f6a5402bc30 comp rx=0 tx=0).stop 2024-03-28T22:43:49.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.647+0000 7f6a6d200640 1 -- 172.21.15.43:0/3835502930 shutdown_connections 2024-03-28T22:43:49.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.647+0000 7f6a6d200640 1 --2- 172.21.15.43:0/3835502930 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a68100010 0x7f6a68100470 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.647+0000 7f6a6d200640 1 --2- 172.21.15.43:0/3835502930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a681009b0 0x7f6a6810cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.647+0000 7f6a6d200640 1 --2- 172.21.15.43:0/3835502930 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a68104d80 0x7f6a68105160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.647+0000 7f6a6d200640 1 -- 172.21.15.43:0/3835502930 >> 172.21.15.43:0/3835502930 conn(0x7f6a68075990 msgr2=0x7f6a68075da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:49.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.647+0000 7f6a6d200640 1 -- 172.21.15.43:0/3835502930 shutdown_connections 2024-03-28T22:43:49.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.647+0000 7f6a6d200640 1 -- 172.21.15.43:0/3835502930 wait complete. 2024-03-28T22:43:49.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.648+0000 7f6a6d200640 1 Processor -- start 2024-03-28T22:43:49.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.648+0000 7f6a6d200640 1 -- start start 2024-03-28T22:43:49.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.649+0000 7f6a6d200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a68100010 0x7f6a6810e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.649+0000 7f6a6d200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a681009b0 0x7f6a6810ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.649+0000 7f6a6d200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a68104d80 0x7f6a6810efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.649+0000 7f6a6d200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6a68111b60 con 0x7f6a681009b0 2024-03-28T22:43:49.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.649+0000 7f6a6d200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f6a681119e0 con 0x7f6a68100010 2024-03-28T22:43:49.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.649+0000 7f6a6d200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f6a68111ce0 con 0x7f6a68104d80 2024-03-28T22:43:49.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.649+0000 7f6a65e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a68100010 0x7f6a6810e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.649+0000 7f6a65e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a68100010 0x7f6a6810e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47394/0 (socket says 172.21.15.43:47394) 2024-03-28T22:43:49.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.649+0000 7f6a65e00640 1 -- 172.21.15.43:0/1961525994 learned_addr learned my addr 172.21.15.43:0/1961525994 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:49.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.649+0000 7f6a66800640 1 --2- 172.21.15.43:0/1961525994 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a68104d80 0x7f6a6810efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.649+0000 7f6a65400640 1 --2- 172.21.15.43:0/1961525994 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a681009b0 0x7f6a6810ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.649+0000 7f6a65e00640 1 -- 172.21.15.43:0/1961525994 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a68104d80 msgr2=0x7f6a6810efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.649+0000 7f6a65e00640 1 --2- 172.21.15.43:0/1961525994 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a68104d80 0x7f6a6810efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.649+0000 7f6a65e00640 1 -- 172.21.15.43:0/1961525994 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a681009b0 msgr2=0x7f6a6810ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.649+0000 7f6a65e00640 1 --2- 172.21.15.43:0/1961525994 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a681009b0 0x7f6a6810ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.649+0000 7f6a65e00640 1 -- 172.21.15.43:0/1961525994 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a681a6480 con 0x7f6a68100010 2024-03-28T22:43:49.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.650+0000 7f6a65e00640 1 --2- 172.21.15.43:0/1961525994 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a68100010 0x7f6a6810e520 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f6a5800e9f0 tx=0x7f6a5800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:49.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.650+0000 7f6a5ea00640 1 -- 172.21.15.43:0/1961525994 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5800cd80 con 0x7f6a68100010 2024-03-28T22:43:49.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.650+0000 7f6a5ea00640 1 -- 172.21.15.43:0/1961525994 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a58010910 con 0x7f6a68100010 2024-03-28T22:43:49.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.650+0000 7f6a6d200640 1 -- 172.21.15.43:0/1961525994 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a681a6770 con 0x7f6a68100010 2024-03-28T22:43:49.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.650+0000 7f6a5ea00640 1 -- 172.21.15.43:0/1961525994 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a58018990 con 0x7f6a68100010 2024-03-28T22:43:49.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.650+0000 7f6a6d200640 1 -- 172.21.15.43:0/1961525994 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a681a6c90 con 0x7f6a68100010 2024-03-28T22:43:49.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.654+0000 7f6a6ffd4740 1 Processor -- start 2024-03-28T22:43:49.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.655+0000 7f6a6ffd4740 1 -- start start 2024-03-28T22:43:49.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.655+0000 7f6a5ea00640 1 -- 172.21.15.43:0/1961525994 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f6a58010ac0 con 0x7f6a68100010 2024-03-28T22:43:49.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.655+0000 7f6a6ffd4740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563a23497380 0x563a23497760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.655+0000 7f6a6ffd4740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563a23497ca0 0x563a234a0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.655+0000 7f6a6ffd4740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563a234a0880 0x563a234a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.655+0000 7f6a6ffd4740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x563a233fb050 con 0x563a23497380 2024-03-28T22:43:49.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.655+0000 7f6a6ffd4740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x563a233a3ce0 con 0x563a234a0880 2024-03-28T22:43:49.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.655+0000 7f6a6ffd4740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x563a233f70f0 con 0x563a23497ca0 2024-03-28T22:43:49.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.656+0000 7f6a65e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563a23497ca0 0x563a234a0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.656+0000 7f6a65400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563a234a0880 0x563a234a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.656+0000 7f6a65e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563a23497ca0 0x563a234a0340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:32772/0 (socket says 172.21.15.43:32772) 2024-03-28T22:43:49.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.656+0000 7f6a65e00640 1 -- 172.21.15.43:0/4159206962 learned_addr learned my addr 172.21.15.43:0/4159206962 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:49.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.656+0000 7f6a65400640 1 --2- 172.21.15.43:0/4159206962 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563a234a0880 0x563a234a2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47398/0 (socket says 172.21.15.43:47398) 2024-03-28T22:43:49.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.656+0000 7f6a66800640 1 --2- 172.21.15.43:0/4159206962 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563a23497380 0x563a23497760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.656+0000 7f6a65e00640 1 -- 172.21.15.43:0/4159206962 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563a234a0880 msgr2=0x563a234a2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:49.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.656+0000 7f6a65e00640 1 --2- 172.21.15.43:0/4159206962 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563a234a0880 0x563a234a2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.656+0000 7f6a65e00640 1 -- 172.21.15.43:0/4159206962 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563a23497380 msgr2=0x563a23497760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:49.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.656+0000 7f6a65e00640 1 --2- 172.21.15.43:0/4159206962 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563a23497380 0x563a23497760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.656+0000 7f6a65e00640 1 -- 172.21.15.43:0/4159206962 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a2310f3f0 con 0x563a23497ca0 2024-03-28T22:43:49.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.656+0000 7f6a65400640 1 --2- 172.21.15.43:0/4159206962 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563a234a0880 0x563a234a2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:49.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.657+0000 7f6a65e00640 1 --2- 172.21.15.43:0/4159206962 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563a23497ca0 0x563a234a0340 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f6a580cbab0 tx=0x7f6a580d6670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=355c9fb0527e348e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:49.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.657+0000 7f6a53400640 1 -- 172.21.15.43:0/4159206962 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a580dd070 con 0x563a23497ca0 2024-03-28T22:43:49.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.657+0000 7f6a53400640 1 -- 172.21.15.43:0/4159206962 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a580d05b0 con 0x563a23497ca0 2024-03-28T22:43:49.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.657+0000 7f6a53400640 1 -- 172.21.15.43:0/4159206962 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a580d9920 con 0x563a23497ca0 2024-03-28T22:43:49.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.657+0000 7f6a5ea00640 1 --2- 172.21.15.43:0/1961525994 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6a4c0a1f20 0x7f6a4c0a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.657+0000 7f6a5ea00640 1 -- 172.21.15.43:0/1961525994 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f6a580ce820 con 0x7f6a68100010 2024-03-28T22:43:49.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.658+0000 7f6a66800640 1 --2- 172.21.15.43:0/1961525994 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6a4c0a1f20 0x7f6a4c0a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.658+0000 7f6a6ffd4740 1 -- 172.21.15.43:0/4159206962 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563a23497ca0 msgr2=0x563a234a0340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.658+0000 7f6a6ffd4740 1 --2- 172.21.15.43:0/4159206962 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563a23497ca0 0x563a234a0340 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f6a580cbab0 tx=0x7f6a580d6670 comp rx=0 tx=0).stop 2024-03-28T22:43:49.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.658+0000 7f6a6ffd4740 1 -- 172.21.15.43:0/4159206962 shutdown_connections 2024-03-28T22:43:49.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.658+0000 7f6a6ffd4740 1 --2- 172.21.15.43:0/4159206962 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563a234a0880 0x563a234a2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.658+0000 7f6a6ffd4740 1 --2- 172.21.15.43:0/4159206962 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563a23497ca0 0x563a234a0340 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.658+0000 7f6a6ffd4740 1 --2- 172.21.15.43:0/4159206962 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563a23497380 0x563a23497760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.658+0000 7f6a6ffd4740 1 -- 172.21.15.43:0/4159206962 >> 172.21.15.43:0/4159206962 conn(0x563a23491ab0 msgr2=0x563a23491e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:49.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.658+0000 7f6a6ffd4740 1 -- 172.21.15.43:0/4159206962 shutdown_connections 2024-03-28T22:43:49.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.658+0000 7f6a66800640 1 --2- 172.21.15.43:0/1961525994 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6a4c0a1f20 0x7f6a4c0a43e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6a60005fd0 tx=0x7f6a60009500 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:49.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.658+0000 7f6a6ffd4740 1 -- 172.21.15.43:0/4159206962 wait complete. 2024-03-28T22:43:49.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.659+0000 7f6a6ffd4740 1 -- start start 2024-03-28T22:43:49.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.659+0000 7f6a6ffd4740 1 Processor -- start 2024-03-28T22:43:49.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.660+0000 7f6a6ffd4740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563a23497380 0x563a2349d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.660+0000 7f6a6ffd4740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563a23497ca0 0x563a2349dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.660+0000 7f6a6ffd4740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563a234a0880 0x563a2349e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.660+0000 7f6a6ffd4740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x563a234a3670 con 0x563a23497ca0 2024-03-28T22:43:49.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.660+0000 7f6a6ffd4740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x563a233992f0 con 0x563a234a0880 2024-03-28T22:43:49.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.660+0000 7f6a6ffd4740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x563a234a37f0 con 0x563a23497380 2024-03-28T22:43:49.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.660+0000 7f6a6ffd4740 10 client.0 _ll_register_callbacks cb 0x563a2310ebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:49.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.660+0000 7f6a65e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563a23497380 0x563a2349d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.660+0000 7f6a65e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563a23497380 0x563a2349d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:32778/0 (socket says 172.21.15.43:32778) 2024-03-28T22:43:49.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.660+0000 7f6a65e00640 1 -- 172.21.15.43:0/1139393532 learned_addr learned my addr 172.21.15.43:0/1139393532 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:49.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.660+0000 7f6a66800640 1 --2- 172.21.15.43:0/1139393532 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563a234a0880 0x563a2349e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.660+0000 7f6a65400640 1 --2- 172.21.15.43:0/1139393532 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563a23497ca0 0x563a2349dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.660+0000 7f6a65e00640 1 -- 172.21.15.43:0/1139393532 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563a234a0880 msgr2=0x563a2349e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:49.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.660+0000 7f6a65e00640 1 --2- 172.21.15.43:0/1139393532 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563a234a0880 0x563a2349e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.660+0000 7f6a65e00640 1 -- 172.21.15.43:0/1139393532 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563a23497ca0 msgr2=0x563a2349dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:49.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.660+0000 7f6a65e00640 1 --2- 172.21.15.43:0/1139393532 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563a23497ca0 0x563a2349dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.660+0000 7f6a65e00640 1 -- 172.21.15.43:0/1139393532 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a231d97d0 con 0x563a23497380 2024-03-28T22:43:49.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.661+0000 7f6a65e00640 1 --2- 172.21.15.43:0/1139393532 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563a23497380 0x563a2349d980 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6a58097890 tx=0x7f6a580d9560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8432082350a29642 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:49.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.661+0000 7f6a47e00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:43:49.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.661+0000 7f6a47e00640 1 -- 172.21.15.43:0/1139393532 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a580aee60 con 0x563a23497380 2024-03-28T22:43:49.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.661+0000 7f6a47e00640 1 -- 172.21.15.43:0/1139393532 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f6a580cbdf0 con 0x563a23497380 2024-03-28T22:43:49.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.661+0000 7f6a6ffd4740 1 -- 172.21.15.43:0/1139393532 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a580d05b0 con 0x563a23497380 2024-03-28T22:43:49.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.661+0000 7f6a47e00640 1 -- 172.21.15.43:0/1139393532 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6a580dd030 con 0x563a23497380 2024-03-28T22:43:49.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.661+0000 7f6a6ffd4740 10 client.28024 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:49.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.661+0000 7f6a6ffd4740 1 -- 172.21.15.43:0/1139393532 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563a234a3670 con 0x563a23497380 2024-03-28T22:43:49.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.661+0000 7f6a47e00640 1 -- 172.21.15.43:0/1139393532 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a580d05b0 con 0x563a23497380 2024-03-28T22:43:49.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.662+0000 7f6a47e00640 1 -- 172.21.15.43:0/1139393532 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6a580ce3c0 con 0x563a23497380 2024-03-28T22:43:49.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.662+0000 7f6a6ffd4740 10 client.28024 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:49.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.662+0000 7f6a6ffd4740 10 client.28024 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:43:49.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.662+0000 7f6a6ffd4740 20 client.28024 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:49.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.662+0000 7f6a6ffd4740 1 --2- 172.21.15.43:0/1139393532 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x563a234b0bc0 0x563a234b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.662+0000 7f6a6ffd4740 4 client.28024 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:43:49.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.662+0000 7f6a6ffd4740 1 -- 172.21.15.43:0/1139393532 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a233a4680 con 0x563a234b0bc0 2024-03-28T22:43:49.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.662+0000 7f6a65400640 1 --2- 172.21.15.43:0/1139393532 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x563a234b0bc0 0x563a234b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.663+0000 7f6a65400640 1 --2- 172.21.15.43:0/1139393532 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x563a234b0bc0 0x563a234b2fb0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f1b455fca624da8d server_cookie=2b81ad89ef4a77d1 in_seq=0 out_seq=0 2024-03-28T22:43:49.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.663+0000 7f6a47e00640 10 client.28024 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:43:49.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.664+0000 7f6a47e00640 1 -- 172.21.15.43:0/1139393532 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563a233a4680 con 0x563a234b0bc0 2024-03-28T22:43:49.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.664+0000 7f6a47e00640 10 client.28024 handle_command_reply: tid=0 2024-03-28T22:43:49.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.664+0000 7f6a6ffd4740 1 client.28024 shutdown 2024-03-28T22:43:49.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.664+0000 7f6a6ffd4740 1 -- 172.21.15.43:0/1139393532 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563a23497380 msgr2=0x563a2349d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.664+0000 7f6a6ffd4740 1 --2- 172.21.15.43:0/1139393532 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563a23497380 0x563a2349d980 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6a58097890 tx=0x7f6a580d9560 comp rx=0 tx=0).stop 2024-03-28T22:43:49.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.664+0000 7f6a6ffd4740 1 -- 172.21.15.43:0/1139393532 shutdown_connections 2024-03-28T22:43:49.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.664+0000 7f6a6ffd4740 1 --2- 172.21.15.43:0/1139393532 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x563a234b0bc0 0x563a234b2fb0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.664+0000 7f6a6ffd4740 1 --2- 172.21.15.43:0/1139393532 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563a234a0880 0x563a2349e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.664+0000 7f6a6ffd4740 1 --2- 172.21.15.43:0/1139393532 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563a23497ca0 0x563a2349dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.664+0000 7f6a47e00640 0 client.28024 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:43:49.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.664+0000 7f6a6ffd4740 1 --2- 172.21.15.43:0/1139393532 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563a23497380 0x563a2349d980 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.664+0000 7f6a6ffd4740 1 -- 172.21.15.43:0/1139393532 >> 172.21.15.43:0/1139393532 conn(0x563a23491ab0 msgr2=0x563a23493800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:49.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.664+0000 7f6a6ffd4740 1 -- 172.21.15.43:0/1139393532 shutdown_connections 2024-03-28T22:43:49.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.665+0000 7f6a6ffd4740 1 -- 172.21.15.43:0/1139393532 wait complete. 2024-03-28T22:43:49.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.665+0000 7f6a6ffd4740 20 client.28024 trim_cache size 0 max 16384 2024-03-28T22:43:49.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.682+0000 7f6a6ffd4740 1 Processor -- start 2024-03-28T22:43:49.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.682+0000 7f6a6ffd4740 1 -- start start 2024-03-28T22:43:49.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.682+0000 7f6a6ffd4740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563a23497380 0x563a234ac410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.682+0000 7f6a6ffd4740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563a23497ca0 0x563a2349e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.682+0000 7f6a6ffd4740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563a234a0880 0x563a234b8610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.682+0000 7f6a6ffd4740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6a681b0ef0 con 0x563a234a0880 2024-03-28T22:43:49.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.682+0000 7f6a6ffd4740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f6a681b2a30 con 0x563a23497ca0 2024-03-28T22:43:49.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.682+0000 7f6a6ffd4740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f6a681b15c0 con 0x563a23497380 2024-03-28T22:43:49.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.683+0000 7f6a65400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563a23497ca0 0x563a2349e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.683+0000 7f6a65e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563a23497380 0x563a234ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.683+0000 7f6a65400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563a23497ca0 0x563a2349e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47418/0 (socket says 172.21.15.43:47418) 2024-03-28T22:43:49.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.683+0000 7f6a65400640 1 -- 172.21.15.43:0/3311716915 learned_addr learned my addr 172.21.15.43:0/3311716915 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:49.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.683+0000 7f6a65e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563a23497380 0x563a234ac410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:32788/0 (socket says 172.21.15.43:32788) 2024-03-28T22:43:49.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.683+0000 7f6a66800640 1 --2- 172.21.15.43:0/3311716915 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563a234a0880 0x563a234b8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.683+0000 7f6a65e00640 1 -- 172.21.15.43:0/3311716915 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563a23497ca0 msgr2=0x563a2349e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:49.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.683+0000 7f6a65e00640 1 --2- 172.21.15.43:0/3311716915 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563a23497ca0 0x563a2349e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.683+0000 7f6a65e00640 1 -- 172.21.15.43:0/3311716915 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563a234a0880 msgr2=0x563a234b8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:49.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.683+0000 7f6a65e00640 1 --2- 172.21.15.43:0/3311716915 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563a234a0880 0x563a234b8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.683+0000 7f6a65e00640 1 -- 172.21.15.43:0/3311716915 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a2348d370 con 0x563a23497380 2024-03-28T22:43:49.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.683+0000 7f6a65400640 1 --2- 172.21.15.43:0/3311716915 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563a23497ca0 0x563a2349e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:49.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.683+0000 7f6a66800640 1 --2- 172.21.15.43:0/3311716915 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563a234a0880 0x563a234b8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:43:49.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.683+0000 7f6a65e00640 1 --2- 172.21.15.43:0/3311716915 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563a23497380 0x563a234ac410 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f6a58097840 tx=0x7f6a580d6670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf1c19fc1f6e89b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:49.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.684+0000 7f6a52a00640 1 -- 172.21.15.43:0/3311716915 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a580aee60 con 0x563a23497380 2024-03-28T22:43:49.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.684+0000 7f6a52a00640 1 -- 172.21.15.43:0/3311716915 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a5809aec0 con 0x563a23497380 2024-03-28T22:43:49.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.684+0000 7f6a52a00640 1 -- 172.21.15.43:0/3311716915 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a580d6ea0 con 0x563a23497380 2024-03-28T22:43:49.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.684+0000 7f6a6ffd4740 1 -- 172.21.15.43:0/3311716915 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563a23497380 msgr2=0x563a234ac410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.685+0000 7f6a6ffd4740 1 --2- 172.21.15.43:0/3311716915 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563a23497380 0x563a234ac410 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f6a58097840 tx=0x7f6a580d6670 comp rx=0 tx=0).stop 2024-03-28T22:43:49.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.685+0000 7f6a6ffd4740 1 -- 172.21.15.43:0/3311716915 shutdown_connections 2024-03-28T22:43:49.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.685+0000 7f6a6ffd4740 1 --2- 172.21.15.43:0/3311716915 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563a23497ca0 0x563a2349e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.685+0000 7f6a6ffd4740 1 --2- 172.21.15.43:0/3311716915 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563a234a0880 0x563a234b8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.685+0000 7f6a6ffd4740 1 --2- 172.21.15.43:0/3311716915 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563a23497380 0x563a234ac410 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.685+0000 7f6a6ffd4740 1 -- 172.21.15.43:0/3311716915 >> 172.21.15.43:0/3311716915 conn(0x563a23491ab0 msgr2=0x563a23493700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:49.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.685+0000 7f6a6ffd4740 1 -- 172.21.15.43:0/3311716915 shutdown_connections 2024-03-28T22:43:49.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.685+0000 7f6a6ffd4740 1 -- 172.21.15.43:0/3311716915 wait complete. 2024-03-28T22:43:49.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.685+0000 7f6a6ffd4740 1 -- start start 2024-03-28T22:43:49.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.685+0000 7f6a6ffd4740 1 Processor -- start 2024-03-28T22:43:49.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.686+0000 7f6a6ffd4740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563a23497380 0x563a234a8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.686+0000 7f6a6ffd4740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563a23497ca0 0x563a234a4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.686+0000 7f6a6ffd4740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563a234a0880 0x563a234a4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.686+0000 7f6a6ffd4740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6a681b2360 con 0x563a23497380 2024-03-28T22:43:49.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.686+0000 7f6a6ffd4740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f6a681b1c90 con 0x563a23497ca0 2024-03-28T22:43:49.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.686+0000 7f6a6ffd4740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f6a681aeed0 con 0x563a234a0880 2024-03-28T22:43:49.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.686+0000 7f6a6ffd4740 10 client.0 _ll_register_callbacks cb 0x7f6a60009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:49.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.686+0000 7f6a65400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563a23497ca0 0x563a234a4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.686+0000 7f6a66800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563a234a0880 0x563a234a4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.686+0000 7f6a65400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563a23497ca0 0x563a234a4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47434/0 (socket says 172.21.15.43:47434) 2024-03-28T22:43:49.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.686+0000 7f6a65400640 1 -- 172.21.15.43:0/570496051 learned_addr learned my addr 172.21.15.43:0/570496051 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:49.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.686+0000 7f6a65e00640 1 --2- 172.21.15.43:0/570496051 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563a23497380 0x563a234a8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.687+0000 7f6a65400640 1 -- 172.21.15.43:0/570496051 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563a234a0880 msgr2=0x563a234a4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:49.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.687+0000 7f6a65400640 1 --2- 172.21.15.43:0/570496051 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563a234a0880 0x563a234a4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.687+0000 7f6a65400640 1 -- 172.21.15.43:0/570496051 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563a23497380 msgr2=0x563a234a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:49.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.687+0000 7f6a65400640 1 --2- 172.21.15.43:0/570496051 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563a23497380 0x563a234a8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.687+0000 7f6a65400640 1 -- 172.21.15.43:0/570496051 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a23515c40 con 0x563a23497ca0 2024-03-28T22:43:49.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.687+0000 7f6a65400640 1 --2- 172.21.15.43:0/570496051 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563a23497ca0 0x563a234a4200 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f6a54009230 tx=0x7f6a5402ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2133fe413458be1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:49.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.687+0000 7f6a47e00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:43:49.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.687+0000 7f6a47e00640 1 -- 172.21.15.43:0/570496051 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a54048d80 con 0x563a23497ca0 2024-03-28T22:43:49.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.687+0000 7f6a6ffd4740 1 -- 172.21.15.43:0/570496051 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a5809aec0 con 0x563a23497ca0 2024-03-28T22:43:49.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.687+0000 7f6a47e00640 1 -- 172.21.15.43:0/570496051 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f6a54048460 con 0x563a23497ca0 2024-03-28T22:43:49.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.688+0000 7f6a47e00640 1 -- 172.21.15.43:0/570496051 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6a5405a050 con 0x563a23497ca0 2024-03-28T22:43:49.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.688+0000 7f6a6ffd4740 10 client.28088 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:49.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.688+0000 7f6a6ffd4740 1 -- 172.21.15.43:0/570496051 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563a2351f5e0 con 0x563a23497ca0 2024-03-28T22:43:49.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.688+0000 7f6a47e00640 1 -- 172.21.15.43:0/570496051 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a5809aec0 con 0x563a23497ca0 2024-03-28T22:43:49.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.688+0000 7f6a47e00640 1 -- 172.21.15.43:0/570496051 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6a54004a20 con 0x563a23497ca0 2024-03-28T22:43:49.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.688+0000 7f6a6ffd4740 10 client.28088 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:49.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.688+0000 7f6a6ffd4740 10 client.28088 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:43:49.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.688+0000 7f6a6ffd4740 20 client.28088 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:49.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.688+0000 7f6a6ffd4740 1 --2- 172.21.15.43:0/570496051 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x563a2351fd90 0x563a235221b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.688+0000 7f6a6ffd4740 4 client.28088 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:43:49.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.688+0000 7f6a6ffd4740 1 -- 172.21.15.43:0/570496051 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563a23408590 con 0x563a2351fd90 2024-03-28T22:43:49.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.688+0000 7f6a65e00640 1 --2- 172.21.15.43:0/570496051 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x563a2351fd90 0x563a235221b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.689+0000 7f6a65e00640 1 --2- 172.21.15.43:0/570496051 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x563a2351fd90 0x563a235221b0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8ff41f0788742ddf server_cookie=90c22f1d363fd569 in_seq=0 out_seq=0 2024-03-28T22:43:49.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.689+0000 7f6a47e00640 10 client.28088 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:43:49.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.689+0000 7f6a47e00640 1 -- 172.21.15.43:0/570496051 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563a23408590 con 0x563a2351fd90 2024-03-28T22:43:49.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.689+0000 7f6a47e00640 10 client.28088 handle_command_reply: tid=0 2024-03-28T22:43:49.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.689+0000 7f6a6ffd4740 1 client.28088 shutdown 2024-03-28T22:43:49.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.689+0000 7f6a6ffd4740 1 -- 172.21.15.43:0/570496051 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563a23497ca0 msgr2=0x563a234a4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.689+0000 7f6a6ffd4740 1 --2- 172.21.15.43:0/570496051 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563a23497ca0 0x563a234a4200 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f6a54009230 tx=0x7f6a5402ca10 comp rx=0 tx=0).stop 2024-03-28T22:43:49.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.689+0000 7f6a6ffd4740 1 -- 172.21.15.43:0/570496051 shutdown_connections 2024-03-28T22:43:49.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.689+0000 7f6a6ffd4740 1 --2- 172.21.15.43:0/570496051 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x563a2351fd90 0x563a235221b0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.689+0000 7f6a6ffd4740 1 --2- 172.21.15.43:0/570496051 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563a23497ca0 0x563a234a4200 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.689+0000 7f6a6ffd4740 1 --2- 172.21.15.43:0/570496051 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563a234a0880 0x563a234a4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.689+0000 7f6a6ffd4740 1 --2- 172.21.15.43:0/570496051 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563a23497380 0x563a234a8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.689+0000 7f6a47e00640 0 client.28088 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:43:49.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.689+0000 7f6a6ffd4740 1 -- 172.21.15.43:0/570496051 >> 172.21.15.43:0/570496051 conn(0x563a23491ab0 msgr2=0x563a234902e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:49.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.690+0000 7f6a6ffd4740 1 -- 172.21.15.43:0/570496051 shutdown_connections 2024-03-28T22:43:49.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.690+0000 7f6a6ffd4740 1 -- 172.21.15.43:0/570496051 wait complete. 2024-03-28T22:43:49.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.690+0000 7f6a6ffd4740 20 client.28088 trim_cache size 0 max 16384 2024-03-28T22:43:49.692 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:43:49.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.690+0000 7f6a6d200640 1 -- 172.21.15.43:0/1961525994 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6a4c0a1f20 msgr2=0x7f6a4c0a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.690+0000 7f6a6d200640 1 --2- 172.21.15.43:0/1961525994 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6a4c0a1f20 0x7f6a4c0a43e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6a60005fd0 tx=0x7f6a60009500 comp rx=0 tx=0).stop 2024-03-28T22:43:49.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.690+0000 7f6a6d200640 1 -- 172.21.15.43:0/1961525994 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a68100010 msgr2=0x7f6a6810e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.690+0000 7f6a6d200640 1 --2- 172.21.15.43:0/1961525994 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a68100010 0x7f6a6810e520 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f6a5800e9f0 tx=0x7f6a5800eeb0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.691+0000 7f6a6d200640 1 -- 172.21.15.43:0/1961525994 shutdown_connections 2024-03-28T22:43:49.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.691+0000 7f6a6d200640 1 --2- 172.21.15.43:0/1961525994 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6a4c0a1f20 0x7f6a4c0a43e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.691+0000 7f6a6d200640 1 --2- 172.21.15.43:0/1961525994 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a68104d80 0x7f6a6810efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.691+0000 7f6a6d200640 1 --2- 172.21.15.43:0/1961525994 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a681009b0 0x7f6a6810ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.691+0000 7f6a6d200640 1 --2- 172.21.15.43:0/1961525994 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a68100010 0x7f6a6810e520 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.691+0000 7f6a6d200640 1 -- 172.21.15.43:0/1961525994 >> 172.21.15.43:0/1961525994 conn(0x7f6a68075990 msgr2=0x7f6a680fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:49.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.691+0000 7f6a6d200640 1 -- 172.21.15.43:0/1961525994 shutdown_connections 2024-03-28T22:43:49.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.691+0000 7f6a6d200640 1 -- 172.21.15.43:0/1961525994 wait complete. 2024-03-28T22:43:49.709 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:43:49.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.943+0000 7f1179600640 1 -- 172.21.15.43:0/2116832268 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1174107f00 msgr2=0x7f11741082e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.943+0000 7f1179600640 1 --2- 172.21.15.43:0/2116832268 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1174107f00 0x7f11741082e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f1164009ca0 tx=0x7f116402c100 comp rx=0 tx=0).stop 2024-03-28T22:43:49.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.943+0000 7f1179600640 1 -- 172.21.15.43:0/2116832268 shutdown_connections 2024-03-28T22:43:49.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.943+0000 7f1179600640 1 --2- 172.21.15.43:0/2116832268 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11741088b0 0x7f117410c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.943+0000 7f1179600640 1 --2- 172.21.15.43:0/2116832268 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f117410cf50 0x7f1174114b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.943+0000 7f1179600640 1 --2- 172.21.15.43:0/2116832268 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1174107f00 0x7f11741082e0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.943+0000 7f1179600640 1 -- 172.21.15.43:0/2116832268 >> 172.21.15.43:0/2116832268 conn(0x7f1174077570 msgr2=0x7f1174106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:49.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.943+0000 7f1179600640 1 -- 172.21.15.43:0/2116832268 shutdown_connections 2024-03-28T22:43:49.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.943+0000 7f1179600640 1 -- 172.21.15.43:0/2116832268 wait complete. 2024-03-28T22:43:49.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.944+0000 7f1179600640 1 Processor -- start 2024-03-28T22:43:49.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.944+0000 7f1179600640 1 -- start start 2024-03-28T22:43:49.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.944+0000 7f1179600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1174107f00 0x7f117419e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.944+0000 7f1179600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f117410cf50 0x7f117419eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.944+0000 7f1179600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11741a2e40 0x7f11741a32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.944+0000 7f1179600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1174118000 con 0x7f117410cf50 2024-03-28T22:43:49.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.945+0000 7f1179600640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1174117e80 con 0x7f1174107f00 2024-03-28T22:43:49.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.945+0000 7f1179600640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f1174118180 con 0x7f11741a2e40 2024-03-28T22:43:49.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.945+0000 7f1178c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11741a2e40 0x7f11741a32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.945+0000 7f1173e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1174107f00 0x7f117419e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.945+0000 7f1173400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f117410cf50 0x7f117419eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.945+0000 7f1178c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11741a2e40 0x7f11741a32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:32818/0 (socket says 172.21.15.43:32818) 2024-03-28T22:43:49.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.945+0000 7f1178c00640 1 -- 172.21.15.43:0/259865846 learned_addr learned my addr 172.21.15.43:0/259865846 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:49.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.945+0000 7f1173e00640 1 --2- 172.21.15.43:0/259865846 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1174107f00 0x7f117419e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47454/0 (socket says 172.21.15.43:47454) 2024-03-28T22:43:49.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.945+0000 7f1173e00640 1 -- 172.21.15.43:0/259865846 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11741a2e40 msgr2=0x7f11741a32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.945+0000 7f1173e00640 1 --2- 172.21.15.43:0/259865846 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11741a2e40 0x7f11741a32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.945+0000 7f1173e00640 1 -- 172.21.15.43:0/259865846 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f117410cf50 msgr2=0x7f117419eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.945+0000 7f1173e00640 1 --2- 172.21.15.43:0/259865846 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f117410cf50 0x7f117419eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.945+0000 7f1173e00640 1 -- 172.21.15.43:0/259865846 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11741a3930 con 0x7f1174107f00 2024-03-28T22:43:49.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.945+0000 7f1178c00640 1 --2- 172.21.15.43:0/259865846 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11741a2e40 0x7f11741a32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:49.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.945+0000 7f1173400640 1 --2- 172.21.15.43:0/259865846 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f117410cf50 0x7f117419eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:49.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.946+0000 7f1173e00640 1 --2- 172.21.15.43:0/259865846 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1174107f00 0x7f117419e580 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f1164006010 tx=0x7f1164035ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:49.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.946+0000 7f1170c00640 1 -- 172.21.15.43:0/259865846 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1164039040 con 0x7f1174107f00 2024-03-28T22:43:49.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.946+0000 7f1170c00640 1 -- 172.21.15.43:0/259865846 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1164035cd0 con 0x7f1174107f00 2024-03-28T22:43:49.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.946+0000 7f1179600640 1 -- 172.21.15.43:0/259865846 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11741a3bc0 con 0x7f1174107f00 2024-03-28T22:43:49.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.946+0000 7f1170c00640 1 -- 172.21.15.43:0/259865846 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11640042a0 con 0x7f1174107f00 2024-03-28T22:43:49.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.946+0000 7f1179600640 1 -- 172.21.15.43:0/259865846 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11741ab320 con 0x7f1174107f00 2024-03-28T22:43:49.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.949+0000 7f117da2c740 1 Processor -- start 2024-03-28T22:43:49.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.949+0000 7f117da2c740 1 -- start start 2024-03-28T22:43:49.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.950+0000 7f117da2c740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562b8e09c550 0x562b8e09c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.950+0000 7f1170c00640 1 -- 172.21.15.43:0/259865846 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f1164046030 con 0x7f1174107f00 2024-03-28T22:43:49.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.950+0000 7f117da2c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b8e09ce70 0x562b8e0a5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.950+0000 7f117da2c740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562b8e0a5a50 0x562b8e0a7e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.950+0000 7f117da2c740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562b8e000050 con 0x562b8e09c550 2024-03-28T22:43:49.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.950+0000 7f117da2c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x562b8dfa8ce0 con 0x562b8e09ce70 2024-03-28T22:43:49.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.950+0000 7f117da2c740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x562b8dffc0f0 con 0x562b8e0a5a50 2024-03-28T22:43:49.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.950+0000 7f1173e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b8e09ce70 0x562b8e0a5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.950+0000 7f1173400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562b8e0a5a50 0x562b8e0a7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.950+0000 7f1178c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562b8e09c550 0x562b8e09c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.950+0000 7f1173400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562b8e0a5a50 0x562b8e0a7e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:32828/0 (socket says 172.21.15.43:32828) 2024-03-28T22:43:49.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.950+0000 7f1173400640 1 -- 172.21.15.43:0/2907111928 learned_addr learned my addr 172.21.15.43:0/2907111928 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:49.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.951+0000 7f1173400640 1 -- 172.21.15.43:0/2907111928 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b8e09ce70 msgr2=0x562b8e0a5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:49.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.951+0000 7f1173400640 1 --2- 172.21.15.43:0/2907111928 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b8e09ce70 0x562b8e0a5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.951+0000 7f1173400640 1 -- 172.21.15.43:0/2907111928 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562b8e09c550 msgr2=0x562b8e09c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:49.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.951+0000 7f1173400640 1 --2- 172.21.15.43:0/2907111928 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562b8e09c550 0x562b8e09c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.951+0000 7f1173400640 1 -- 172.21.15.43:0/2907111928 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b8dd143f0 con 0x562b8e0a5a50 2024-03-28T22:43:49.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.951+0000 7f1173e00640 1 --2- 172.21.15.43:0/2907111928 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b8e09ce70 0x562b8e0a5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:49.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.951+0000 7f1173400640 1 --2- 172.21.15.43:0/2907111928 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562b8e0a5a50 0x562b8e0a7e40 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f115c011470 tx=0x7f115c011930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17de3be79ee5218d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:49.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.951+0000 7f1169200640 1 -- 172.21.15.43:0/2907111928 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f115c01e070 con 0x562b8e0a5a50 2024-03-28T22:43:49.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.951+0000 7f1169200640 1 -- 172.21.15.43:0/2907111928 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f115c0035f0 con 0x562b8e0a5a50 2024-03-28T22:43:49.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.951+0000 7f1169200640 1 -- 172.21.15.43:0/2907111928 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f115c00f860 con 0x562b8e0a5a50 2024-03-28T22:43:49.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.952+0000 7f1170c00640 1 --2- 172.21.15.43:0/259865846 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f11580a1ef0 0x7f11580a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.952+0000 7f1170c00640 1 -- 172.21.15.43:0/259865846 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f11640effe0 con 0x7f1174107f00 2024-03-28T22:43:49.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.952+0000 7f1178c00640 1 --2- 172.21.15.43:0/259865846 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f11580a1ef0 0x7f11580a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.952+0000 7f117da2c740 1 -- 172.21.15.43:0/2907111928 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562b8e0a5a50 msgr2=0x562b8e0a7e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.952+0000 7f117da2c740 1 --2- 172.21.15.43:0/2907111928 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562b8e0a5a50 0x562b8e0a7e40 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f115c011470 tx=0x7f115c011930 comp rx=0 tx=0).stop 2024-03-28T22:43:49.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.952+0000 7f117da2c740 1 -- 172.21.15.43:0/2907111928 shutdown_connections 2024-03-28T22:43:49.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.952+0000 7f117da2c740 1 --2- 172.21.15.43:0/2907111928 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b8e09ce70 0x562b8e0a5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.952+0000 7f117da2c740 1 --2- 172.21.15.43:0/2907111928 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562b8e0a5a50 0x562b8e0a7e40 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.952+0000 7f117da2c740 1 --2- 172.21.15.43:0/2907111928 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562b8e09c550 0x562b8e09c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.952+0000 7f117da2c740 1 -- 172.21.15.43:0/2907111928 >> 172.21.15.43:0/2907111928 conn(0x562b8e096c80 msgr2=0x562b8e097060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:49.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.953+0000 7f117da2c740 1 -- 172.21.15.43:0/2907111928 shutdown_connections 2024-03-28T22:43:49.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.953+0000 7f117da2c740 1 -- 172.21.15.43:0/2907111928 wait complete. 2024-03-28T22:43:49.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.953+0000 7f1178c00640 1 --2- 172.21.15.43:0/259865846 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f11580a1ef0 0x7f11580a43b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f116c005e00 tx=0x7f116c00b370 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:49.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.953+0000 7f117da2c740 1 -- start start 2024-03-28T22:43:49.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.953+0000 7f117da2c740 1 Processor -- start 2024-03-28T22:43:49.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.954+0000 7f117da2c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b8e09c550 0x562b8e0a2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.954+0000 7f117da2c740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562b8e09ce70 0x562b8e0a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.954+0000 7f117da2c740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562b8e0a5a50 0x562b8e0a3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.954+0000 7f117da2c740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x562b8e0a8840 con 0x562b8e09ce70 2024-03-28T22:43:49.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.954+0000 7f117da2c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x562b8df9e2f0 con 0x562b8e09c550 2024-03-28T22:43:49.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.954+0000 7f117da2c740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x562b8e0a89c0 con 0x562b8e0a5a50 2024-03-28T22:43:49.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.954+0000 7f117da2c740 10 client.0 _ll_register_callbacks cb 0x562b8dd13be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:49.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.954+0000 7f1173e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b8e09c550 0x562b8e0a2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.954+0000 7f1178c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562b8e0a5a50 0x562b8e0a3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.954+0000 7f1173e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b8e09c550 0x562b8e0a2aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47464/0 (socket says 172.21.15.43:47464) 2024-03-28T22:43:49.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.954+0000 7f1173e00640 1 -- 172.21.15.43:0/579356020 learned_addr learned my addr 172.21.15.43:0/579356020 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:49.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.954+0000 7f1173400640 1 --2- 172.21.15.43:0/579356020 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562b8e09ce70 0x562b8e0a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.954+0000 7f1178c00640 1 -- 172.21.15.43:0/579356020 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b8e09c550 msgr2=0x562b8e0a2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:49.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.954+0000 7f1178c00640 1 --2- 172.21.15.43:0/579356020 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b8e09c550 0x562b8e0a2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.954+0000 7f1178c00640 1 -- 172.21.15.43:0/579356020 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562b8e09ce70 msgr2=0x562b8e0a2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:49.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.954+0000 7f1178c00640 1 --2- 172.21.15.43:0/579356020 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562b8e09ce70 0x562b8e0a2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.954+0000 7f1178c00640 1 -- 172.21.15.43:0/579356020 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b8ddde7d0 con 0x562b8e0a5a50 2024-03-28T22:43:49.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.955+0000 7f1178c00640 1 --2- 172.21.15.43:0/579356020 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562b8e0a5a50 0x562b8e0a3520 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f116c005fd0 tx=0x7f116c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14fd6636bc8e0815 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:49.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.955+0000 7f1161600640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:43:49.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.955+0000 7f1161600640 1 -- 172.21.15.43:0/579356020 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116c01f880 con 0x562b8e0a5a50 2024-03-28T22:43:49.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.955+0000 7f1161600640 1 -- 172.21.15.43:0/579356020 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f116c01fa20 con 0x562b8e0a5a50 2024-03-28T22:43:49.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.955+0000 7f117da2c740 1 -- 172.21.15.43:0/579356020 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f115c0035f0 con 0x562b8e0a5a50 2024-03-28T22:43:49.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.955+0000 7f1161600640 1 -- 172.21.15.43:0/579356020 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f116c01fc20 con 0x562b8e0a5a50 2024-03-28T22:43:49.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.956+0000 7f117da2c740 10 client.28042 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:49.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.956+0000 7f117da2c740 1 -- 172.21.15.43:0/579356020 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562b8e0b4b90 con 0x562b8e0a5a50 2024-03-28T22:43:49.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.956+0000 7f1161600640 1 -- 172.21.15.43:0/579356020 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f115c0035f0 con 0x562b8e0a5a50 2024-03-28T22:43:49.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.956+0000 7f1161600640 1 -- 172.21.15.43:0/579356020 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f116c026070 con 0x562b8e0a5a50 2024-03-28T22:43:49.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.956+0000 7f117da2c740 10 client.28042 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:49.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.956+0000 7f117da2c740 10 client.28042 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:43:49.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.956+0000 7f117da2c740 20 client.28042 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:49.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.956+0000 7f117da2c740 1 --2- 172.21.15.43:0/579356020 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x562b8e0b60a0 0x562b8e0b8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.956+0000 7f117da2c740 4 client.28042 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:43:49.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.956+0000 7f117da2c740 1 -- 172.21.15.43:0/579356020 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562b8dfa9680 con 0x562b8e0b60a0 2024-03-28T22:43:49.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.957+0000 7f1173e00640 1 --2- 172.21.15.43:0/579356020 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x562b8e0b60a0 0x562b8e0b8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.957+0000 7f1173e00640 1 --2- 172.21.15.43:0/579356020 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x562b8e0b60a0 0x562b8e0b8510 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b2b7c693b3bfa5a5 server_cookie=1a3af142be7f62ed in_seq=0 out_seq=0 2024-03-28T22:43:49.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.957+0000 7f1161600640 10 client.28042 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:43:49.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.958+0000 7f1161600640 1 -- 172.21.15.43:0/579356020 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562b8dfa9680 con 0x562b8e0b60a0 2024-03-28T22:43:49.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.958+0000 7f1161600640 10 client.28042 handle_command_reply: tid=0 2024-03-28T22:43:49.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.958+0000 7f117da2c740 1 client.28042 shutdown 2024-03-28T22:43:49.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.958+0000 7f117da2c740 1 -- 172.21.15.43:0/579356020 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562b8e0a5a50 msgr2=0x562b8e0a3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.958+0000 7f117da2c740 1 --2- 172.21.15.43:0/579356020 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562b8e0a5a50 0x562b8e0a3520 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f116c005fd0 tx=0x7f116c004730 comp rx=0 tx=0).stop 2024-03-28T22:43:49.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.958+0000 7f117da2c740 1 -- 172.21.15.43:0/579356020 shutdown_connections 2024-03-28T22:43:49.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.958+0000 7f117da2c740 1 --2- 172.21.15.43:0/579356020 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x562b8e0b60a0 0x562b8e0b8510 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.958+0000 7f117da2c740 1 --2- 172.21.15.43:0/579356020 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562b8e0a5a50 0x562b8e0a3520 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.958+0000 7f117da2c740 1 --2- 172.21.15.43:0/579356020 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562b8e09ce70 0x562b8e0a2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.958+0000 7f117da2c740 1 --2- 172.21.15.43:0/579356020 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b8e09c550 0x562b8e0a2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.958+0000 7f117da2c740 1 -- 172.21.15.43:0/579356020 >> 172.21.15.43:0/579356020 conn(0x562b8e096c80 msgr2=0x562b8e0989d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:49.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.958+0000 7f1161600640 0 client.28042 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:43:49.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.958+0000 7f117da2c740 1 -- 172.21.15.43:0/579356020 shutdown_connections 2024-03-28T22:43:49.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.959+0000 7f117da2c740 1 -- 172.21.15.43:0/579356020 wait complete. 2024-03-28T22:43:49.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.959+0000 7f117da2c740 20 client.28042 trim_cache size 0 max 16384 2024-03-28T22:43:49.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.974+0000 7f117da2c740 1 Processor -- start 2024-03-28T22:43:49.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.975+0000 7f117da2c740 1 -- start start 2024-03-28T22:43:49.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.975+0000 7f117da2c740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562b8e09c550 0x562b8e0a51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.975+0000 7f117da2c740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562b8e09ce70 0x562b8e0b1610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.975+0000 7f117da2c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b8e0a5a50 0x562b8e0bde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.975+0000 7f117da2c740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f11741b4f30 con 0x562b8e09c550 2024-03-28T22:43:49.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.975+0000 7f117da2c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f11741b6a70 con 0x562b8e0a5a50 2024-03-28T22:43:49.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.975+0000 7f117da2c740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f11741b5600 con 0x562b8e09ce70 2024-03-28T22:43:49.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.975+0000 7f1173400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562b8e09ce70 0x562b8e0b1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.975+0000 7f1178c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b8e0a5a50 0x562b8e0bde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.975+0000 7f1173400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562b8e09ce70 0x562b8e0b1610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:32844/0 (socket says 172.21.15.43:32844) 2024-03-28T22:43:49.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.975+0000 7f1173e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562b8e09c550 0x562b8e0a51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.975+0000 7f1173400640 1 -- 172.21.15.43:0/863560719 learned_addr learned my addr 172.21.15.43:0/863560719 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:49.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.976+0000 7f1173400640 1 -- 172.21.15.43:0/863560719 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b8e0a5a50 msgr2=0x562b8e0bde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:49.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.976+0000 7f1173400640 1 --2- 172.21.15.43:0/863560719 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b8e0a5a50 0x562b8e0bde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.976+0000 7f1173400640 1 -- 172.21.15.43:0/863560719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562b8e09c550 msgr2=0x562b8e0a51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:49.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.976+0000 7f1173400640 1 --2- 172.21.15.43:0/863560719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562b8e09c550 0x562b8e0a51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.976+0000 7f1173400640 1 -- 172.21.15.43:0/863560719 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b8e096ab0 con 0x562b8e09ce70 2024-03-28T22:43:49.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.976+0000 7f1178c00640 1 --2- 172.21.15.43:0/863560719 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b8e0a5a50 0x562b8e0bde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:49.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.976+0000 7f1173400640 1 --2- 172.21.15.43:0/863560719 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562b8e09ce70 0x562b8e0b1610 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f115c01c040 tx=0x7f115c009ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b14c418bc5985986 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:49.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.976+0000 7f1163e00640 1 -- 172.21.15.43:0/863560719 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f115c01e040 con 0x562b8e09ce70 2024-03-28T22:43:49.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.977+0000 7f1163e00640 1 -- 172.21.15.43:0/863560719 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f115c005520 con 0x562b8e09ce70 2024-03-28T22:43:49.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.977+0000 7f1163e00640 1 -- 172.21.15.43:0/863560719 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f115c00f7a0 con 0x562b8e09ce70 2024-03-28T22:43:49.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.977+0000 7f117da2c740 1 -- 172.21.15.43:0/863560719 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562b8e09ce70 msgr2=0x562b8e0b1610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.977+0000 7f117da2c740 1 --2- 172.21.15.43:0/863560719 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562b8e09ce70 0x562b8e0b1610 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f115c01c040 tx=0x7f115c009ce0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.978+0000 7f117da2c740 1 -- 172.21.15.43:0/863560719 shutdown_connections 2024-03-28T22:43:49.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.978+0000 7f117da2c740 1 --2- 172.21.15.43:0/863560719 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b8e0a5a50 0x562b8e0bde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.978+0000 7f117da2c740 1 --2- 172.21.15.43:0/863560719 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562b8e09ce70 0x562b8e0b1610 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.978+0000 7f117da2c740 1 --2- 172.21.15.43:0/863560719 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562b8e09c550 0x562b8e0a51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.978+0000 7f117da2c740 1 -- 172.21.15.43:0/863560719 >> 172.21.15.43:0/863560719 conn(0x562b8e096c80 msgr2=0x562b8e098940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:49.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.978+0000 7f117da2c740 1 -- 172.21.15.43:0/863560719 shutdown_connections 2024-03-28T22:43:49.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.978+0000 7f117da2c740 1 -- 172.21.15.43:0/863560719 wait complete. 2024-03-28T22:43:49.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.978+0000 7f117da2c740 1 -- start start 2024-03-28T22:43:49.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.978+0000 7f117da2c740 1 Processor -- start 2024-03-28T22:43:49.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.979+0000 7f117da2c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b8e09c550 0x562b8e09f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.979+0000 7f117da2c740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562b8e09ce70 0x562b8e09fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.979+0000 7f117da2c740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562b8e0a5a50 0x562b8e0a0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.979+0000 7f117da2c740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f11741b63a0 con 0x562b8e0a5a50 2024-03-28T22:43:49.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.979+0000 7f117da2c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f11741b5cd0 con 0x562b8e09c550 2024-03-28T22:43:49.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.979+0000 7f117da2c740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f11741b2f10 con 0x562b8e09ce70 2024-03-28T22:43:49.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.979+0000 7f117da2c740 10 client.0 _ll_register_callbacks cb 0x7f116c00be60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:49.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.979+0000 7f1173e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b8e09c550 0x562b8e09f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.979+0000 7f1173400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562b8e09ce70 0x562b8e09fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.979+0000 7f1173e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b8e09c550 0x562b8e09f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47480/0 (socket says 172.21.15.43:47480) 2024-03-28T22:43:49.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.979+0000 7f1173e00640 1 -- 172.21.15.43:0/1104926847 learned_addr learned my addr 172.21.15.43:0/1104926847 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:49.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.979+0000 7f1178c00640 1 --2- 172.21.15.43:0/1104926847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562b8e0a5a50 0x562b8e0a0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.980+0000 7f1173e00640 1 -- 172.21.15.43:0/1104926847 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562b8e09ce70 msgr2=0x562b8e09fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:49.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.980+0000 7f1173e00640 1 --2- 172.21.15.43:0/1104926847 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562b8e09ce70 0x562b8e09fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.980+0000 7f1173e00640 1 -- 172.21.15.43:0/1104926847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562b8e0a5a50 msgr2=0x562b8e0a0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:49.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.980+0000 7f1173e00640 1 --2- 172.21.15.43:0/1104926847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562b8e0a5a50 0x562b8e0a0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.980+0000 7f1173e00640 1 -- 172.21.15.43:0/1104926847 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b8e0a7e40 con 0x562b8e09c550 2024-03-28T22:43:49.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.980+0000 7f1178c00640 1 --2- 172.21.15.43:0/1104926847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562b8e0a5a50 0x562b8e0a0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:43:49.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.980+0000 7f1173e00640 1 --2- 172.21.15.43:0/1104926847 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b8e09c550 0x562b8e09f910 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f116402c640 tx=0x7f1164035c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6bfc8e1173e9065 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:49.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.980+0000 7f1161600640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:43:49.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.980+0000 7f1161600640 1 -- 172.21.15.43:0/1104926847 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11640bceb0 con 0x562b8e09c550 2024-03-28T22:43:49.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.980+0000 7f1161600640 1 -- 172.21.15.43:0/1104926847 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f11640f36f0 con 0x562b8e09c550 2024-03-28T22:43:49.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.980+0000 7f117da2c740 1 -- 172.21.15.43:0/1104926847 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f115c005520 con 0x562b8e09c550 2024-03-28T22:43:49.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.980+0000 7f1161600640 1 -- 172.21.15.43:0/1104926847 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f11640bc7b0 con 0x562b8e09c550 2024-03-28T22:43:49.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.980+0000 7f117da2c740 10 client.28106 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:49.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.980+0000 7f117da2c740 1 -- 172.21.15.43:0/1104926847 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562b8e11c860 con 0x562b8e09c550 2024-03-28T22:43:49.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.981+0000 7f1161600640 1 -- 172.21.15.43:0/1104926847 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f115c005520 con 0x562b8e09c550 2024-03-28T22:43:49.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.981+0000 7f1161600640 1 -- 172.21.15.43:0/1104926847 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f116403c030 con 0x562b8e09c550 2024-03-28T22:43:49.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.981+0000 7f117da2c740 10 client.28106 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:49.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.981+0000 7f117da2c740 10 client.28106 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:43:49.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.981+0000 7f117da2c740 20 client.28106 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:49.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.981+0000 7f117da2c740 1 --2- 172.21.15.43:0/1104926847 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x562b8e11cee0 0x562b8e11f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:49.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.981+0000 7f117da2c740 4 client.28106 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:43:49.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.981+0000 7f117da2c740 1 -- 172.21.15.43:0/1104926847 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562b8e00d590 con 0x562b8e11cee0 2024-03-28T22:43:49.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.981+0000 7f1173400640 1 --2- 172.21.15.43:0/1104926847 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x562b8e11cee0 0x562b8e11f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:49.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.982+0000 7f1173400640 1 --2- 172.21.15.43:0/1104926847 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x562b8e11cee0 0x562b8e11f2d0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5623992f9daef52f server_cookie=f19c78d1e9122cac in_seq=0 out_seq=0 2024-03-28T22:43:49.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.982+0000 7f1161600640 10 client.28106 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:43:49.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.982+0000 7f1161600640 1 -- 172.21.15.43:0/1104926847 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562b8e00d590 con 0x562b8e11cee0 2024-03-28T22:43:49.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.982+0000 7f1161600640 10 client.28106 handle_command_reply: tid=0 2024-03-28T22:43:49.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.982+0000 7f117da2c740 1 client.28106 shutdown 2024-03-28T22:43:49.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.982+0000 7f117da2c740 1 -- 172.21.15.43:0/1104926847 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b8e09c550 msgr2=0x562b8e09f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.982+0000 7f117da2c740 1 --2- 172.21.15.43:0/1104926847 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b8e09c550 0x562b8e09f910 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f116402c640 tx=0x7f1164035c50 comp rx=0 tx=0).stop 2024-03-28T22:43:49.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.982+0000 7f117da2c740 1 -- 172.21.15.43:0/1104926847 shutdown_connections 2024-03-28T22:43:49.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.983+0000 7f117da2c740 1 --2- 172.21.15.43:0/1104926847 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x562b8e11cee0 0x562b8e11f2d0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.983+0000 7f117da2c740 1 --2- 172.21.15.43:0/1104926847 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x562b8e0a5a50 0x562b8e0a0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.983+0000 7f117da2c740 1 --2- 172.21.15.43:0/1104926847 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x562b8e09ce70 0x562b8e09fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.983+0000 7f117da2c740 1 --2- 172.21.15.43:0/1104926847 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b8e09c550 0x562b8e09f910 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.983+0000 7f117da2c740 1 -- 172.21.15.43:0/1104926847 >> 172.21.15.43:0/1104926847 conn(0x562b8e096c80 msgr2=0x562b8e095530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:49.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.983+0000 7f1161600640 0 client.28106 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:43:49.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.983+0000 7f117da2c740 1 -- 172.21.15.43:0/1104926847 shutdown_connections 2024-03-28T22:43:49.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.983+0000 7f117da2c740 1 -- 172.21.15.43:0/1104926847 wait complete. 2024-03-28T22:43:49.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.983+0000 7f117da2c740 20 client.28106 trim_cache size 0 max 16384 2024-03-28T22:43:49.992 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:43:49.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.983+0000 7f1179600640 1 -- 172.21.15.43:0/259865846 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f11580a1ef0 msgr2=0x7f11580a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.983+0000 7f1179600640 1 --2- 172.21.15.43:0/259865846 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f11580a1ef0 0x7f11580a43b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f116c005e00 tx=0x7f116c00b370 comp rx=0 tx=0).stop 2024-03-28T22:43:49.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.983+0000 7f1179600640 1 -- 172.21.15.43:0/259865846 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1174107f00 msgr2=0x7f117419e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:49.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.983+0000 7f1179600640 1 --2- 172.21.15.43:0/259865846 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1174107f00 0x7f117419e580 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f1164006010 tx=0x7f1164035ae0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.984+0000 7f1179600640 1 -- 172.21.15.43:0/259865846 shutdown_connections 2024-03-28T22:43:49.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.984+0000 7f1179600640 1 --2- 172.21.15.43:0/259865846 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f11580a1ef0 0x7f11580a43b0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.984+0000 7f1179600640 1 --2- 172.21.15.43:0/259865846 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f11741a2e40 0x7f11741a32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.984+0000 7f1179600640 1 --2- 172.21.15.43:0/259865846 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f117410cf50 0x7f117419eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.984+0000 7f1179600640 1 --2- 172.21.15.43:0/259865846 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1174107f00 0x7f117419e580 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:49.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.984+0000 7f1179600640 1 -- 172.21.15.43:0/259865846 >> 172.21.15.43:0/259865846 conn(0x7f1174077570 msgr2=0x7f11740791c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:49.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.984+0000 7f1179600640 1 -- 172.21.15.43:0/259865846 shutdown_connections 2024-03-28T22:43:49.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:49.984+0000 7f1179600640 1 -- 172.21.15.43:0/259865846 wait complete. 2024-03-28T22:43:50.002 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:43:50.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.229+0000 7f6638c00640 1 -- 172.21.15.43:0/4014273487 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66341032b0 msgr2=0x7f663410b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:50.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.229+0000 7f6638c00640 1 --2- 172.21.15.43:0/4014273487 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66341032b0 0x7f663410b790 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f661c009860 tx=0x7f661c02bc10 comp rx=0 tx=0).stop 2024-03-28T22:43:50.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.229+0000 7f6638c00640 1 -- 172.21.15.43:0/4014273487 shutdown_connections 2024-03-28T22:43:50.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.229+0000 7f6638c00640 1 --2- 172.21.15.43:0/4014273487 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f663410bd60 0x7f663410e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:50.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.230+0000 7f6638c00640 1 --2- 172.21.15.43:0/4014273487 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f66341032b0 0x7f663410b790 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:50.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.230+0000 7f6638c00640 1 --2- 172.21.15.43:0/4014273487 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6634102990 0x7f6634102d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:50.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.230+0000 7f6638c00640 1 -- 172.21.15.43:0/4014273487 >> 172.21.15.43:0/4014273487 conn(0x7f66340769d0 msgr2=0x7f6634076de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:50.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.230+0000 7f6638c00640 1 -- 172.21.15.43:0/4014273487 shutdown_connections 2024-03-28T22:43:50.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.230+0000 7f6638c00640 1 -- 172.21.15.43:0/4014273487 wait complete. 2024-03-28T22:43:50.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.231+0000 7f6638c00640 1 Processor -- start 2024-03-28T22:43:50.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.231+0000 7f6638c00640 1 -- start start 2024-03-28T22:43:50.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.231+0000 7f6638c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6634102990 0x7f663419a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:50.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.231+0000 7f6638c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f66341032b0 0x7f663419a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:50.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.231+0000 7f6638c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f663410bd60 0x7f663419eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:50.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.231+0000 7f6638c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6634111af0 con 0x7f6634102990 2024-03-28T22:43:50.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.231+0000 7f6638c00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f6634111970 con 0x7f66341032b0 2024-03-28T22:43:50.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.231+0000 7f6638c00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f6634111c70 con 0x7f663410bd60 2024-03-28T22:43:50.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.232+0000 7f6631400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f66341032b0 0x7f663419a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:50.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.232+0000 7f6632800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f663410bd60 0x7f663419eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:50.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.232+0000 7f6631e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6634102990 0x7f663419a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:50.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.232+0000 7f6631400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f66341032b0 0x7f663419a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47496/0 (socket says 172.21.15.43:47496) 2024-03-28T22:43:50.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.232+0000 7f6631400640 1 -- 172.21.15.43:0/407022670 learned_addr learned my addr 172.21.15.43:0/407022670 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:50.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.232+0000 7f6632800640 1 -- 172.21.15.43:0/407022670 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f66341032b0 msgr2=0x7f663419a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:50.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.232+0000 7f6632800640 1 --2- 172.21.15.43:0/407022670 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f66341032b0 0x7f663419a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:50.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.232+0000 7f6632800640 1 -- 172.21.15.43:0/407022670 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6634102990 msgr2=0x7f663419a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:50.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.232+0000 7f6632800640 1 --2- 172.21.15.43:0/407022670 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6634102990 0x7f663419a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:50.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.232+0000 7f6632800640 1 -- 172.21.15.43:0/407022670 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f663419f1f0 con 0x7f663410bd60 2024-03-28T22:43:50.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.232+0000 7f6631400640 1 --2- 172.21.15.43:0/407022670 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f66341032b0 0x7f663419a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:50.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.232+0000 7f6632800640 1 --2- 172.21.15.43:0/407022670 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f663410bd60 0x7f663419eac0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f662c00e940 tx=0x7f662c00ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:50.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.233+0000 7f662aa00640 1 -- 172.21.15.43:0/407022670 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f662c00ccc0 con 0x7f663410bd60 2024-03-28T22:43:50.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.233+0000 7f662aa00640 1 -- 172.21.15.43:0/407022670 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f662c005020 con 0x7f663410bd60 2024-03-28T22:43:50.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.233+0000 7f662aa00640 1 -- 172.21.15.43:0/407022670 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f662c004a00 con 0x7f663410bd60 2024-03-28T22:43:50.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.233+0000 7f6638c00640 1 -- 172.21.15.43:0/407022670 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f663419f4e0 con 0x7f663410bd60 2024-03-28T22:43:50.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.233+0000 7f6638c00640 1 -- 172.21.15.43:0/407022670 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66341a6d70 con 0x7f663410bd60 2024-03-28T22:43:50.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.236+0000 7f663b980740 1 Processor -- start 2024-03-28T22:43:50.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.236+0000 7f662aa00640 1 -- 172.21.15.43:0/407022670 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f662c0051d0 con 0x7f663410bd60 2024-03-28T22:43:50.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.236+0000 7f663b980740 1 -- start start 2024-03-28T22:43:50.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.237+0000 7f663b980740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556ed2425380 0x556ed2425760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:50.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.237+0000 7f663b980740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556ed2425ca0 0x556ed242e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:50.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.237+0000 7f663b980740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556ed242e880 0x556ed2430c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:50.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.237+0000 7f663b980740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x556ed2389050 con 0x556ed2425380 2024-03-28T22:43:50.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.237+0000 7f663b980740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x556ed2331ce0 con 0x556ed2425ca0 2024-03-28T22:43:50.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.237+0000 7f663b980740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x556ed23850f0 con 0x556ed242e880 2024-03-28T22:43:50.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.237+0000 7f6631e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556ed2425ca0 0x556ed242e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:50.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.237+0000 7f6631e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556ed2425ca0 0x556ed242e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47508/0 (socket says 172.21.15.43:47508) 2024-03-28T22:43:50.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.237+0000 7f6631e00640 1 -- 172.21.15.43:0/3293090683 learned_addr learned my addr 172.21.15.43:0/3293090683 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:50.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.237+0000 7f6631400640 1 --2- 172.21.15.43:0/3293090683 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556ed2425380 0x556ed2425760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:50.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.237+0000 7f6631400640 1 --2- 172.21.15.43:0/3293090683 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556ed242e880 0x556ed2430c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:50.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.237+0000 7f6631e00640 1 -- 172.21.15.43:0/3293090683 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556ed242e880 msgr2=0x556ed2430c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:50.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.237+0000 7f6631e00640 1 --2- 172.21.15.43:0/3293090683 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556ed242e880 0x556ed2430c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:50.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.237+0000 7f6631e00640 1 -- 172.21.15.43:0/3293090683 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556ed2425380 msgr2=0x556ed2425760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:50.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.237+0000 7f6631e00640 1 --2- 172.21.15.43:0/3293090683 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556ed2425380 0x556ed2425760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:50.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.237+0000 7f6631e00640 1 -- 172.21.15.43:0/3293090683 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ed209d3f0 con 0x556ed2425ca0 2024-03-28T22:43:50.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.238+0000 7f662aa00640 1 --2- 172.21.15.43:0/407022670 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f66180a1ef0 0x7f66180a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:50.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.238+0000 7f6631e00640 1 --2- 172.21.15.43:0/3293090683 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556ed2425ca0 0x556ed242e340 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f662400e5d0 tx=0x7f66240095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b9f27776d388ac server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:50.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.238+0000 7f662aa00640 1 -- 172.21.15.43:0/407022670 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f662c0ce420 con 0x7f663410bd60 2024-03-28T22:43:50.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.238+0000 7f6631400640 1 --2- 172.21.15.43:0/407022670 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f66180a1ef0 0x7f66180a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:50.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.238+0000 7f6623000640 1 -- 172.21.15.43:0/3293090683 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f662400dad0 con 0x556ed2425ca0 2024-03-28T22:43:50.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.238+0000 7f6623000640 1 -- 172.21.15.43:0/3293090683 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f662401c070 con 0x556ed2425ca0 2024-03-28T22:43:50.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.238+0000 7f6623000640 1 -- 172.21.15.43:0/3293090683 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66240165e0 con 0x556ed2425ca0 2024-03-28T22:43:50.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.239+0000 7f663b980740 1 -- 172.21.15.43:0/3293090683 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556ed2425ca0 msgr2=0x556ed242e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:50.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.239+0000 7f663b980740 1 --2- 172.21.15.43:0/3293090683 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556ed2425ca0 0x556ed242e340 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f662400e5d0 tx=0x7f66240095c0 comp rx=0 tx=0).stop 2024-03-28T22:43:50.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.239+0000 7f6631400640 1 --2- 172.21.15.43:0/407022670 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f66180a1ef0 0x7f66180a43b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f661c005e00 tx=0x7f661c02bc10 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:50.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.239+0000 7f663b980740 1 -- 172.21.15.43:0/3293090683 shutdown_connections 2024-03-28T22:43:50.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.239+0000 7f663b980740 1 --2- 172.21.15.43:0/3293090683 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556ed2425ca0 0x556ed242e340 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:50.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.239+0000 7f663b980740 1 --2- 172.21.15.43:0/3293090683 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556ed242e880 0x556ed2430c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:50.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.239+0000 7f663b980740 1 --2- 172.21.15.43:0/3293090683 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556ed2425380 0x556ed2425760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:50.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.239+0000 7f663b980740 1 -- 172.21.15.43:0/3293090683 >> 172.21.15.43:0/3293090683 conn(0x556ed241fab0 msgr2=0x556ed241fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:50.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.239+0000 7f663b980740 1 -- 172.21.15.43:0/3293090683 shutdown_connections 2024-03-28T22:43:50.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.239+0000 7f663b980740 1 -- 172.21.15.43:0/3293090683 wait complete. 2024-03-28T22:43:50.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.240+0000 7f663b980740 1 -- start start 2024-03-28T22:43:50.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.240+0000 7f663b980740 1 Processor -- start 2024-03-28T22:43:50.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.241+0000 7f663b980740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556ed2425380 0x556ed242b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:50.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.241+0000 7f663b980740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556ed2425ca0 0x556ed242bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:50.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.241+0000 7f663b980740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556ed242e880 0x556ed242c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:50.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.241+0000 7f663b980740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x556ed2431670 con 0x556ed2425380 2024-03-28T22:43:50.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.241+0000 7f663b980740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x556ed23272f0 con 0x556ed242e880 2024-03-28T22:43:50.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.241+0000 7f663b980740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x556ed24317f0 con 0x556ed2425ca0 2024-03-28T22:43:50.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.241+0000 7f663b980740 10 client.0 _ll_register_callbacks cb 0x556ed209cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:50.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.241+0000 7f6631e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556ed2425380 0x556ed242b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:50.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.241+0000 7f6631e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556ed2425380 0x556ed242b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54074/0 (socket says 172.21.15.43:54074) 2024-03-28T22:43:50.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.241+0000 7f6631400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556ed2425ca0 0x556ed242bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:50.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.241+0000 7f6631e00640 1 -- 172.21.15.43:0/454401524 learned_addr learned my addr 172.21.15.43:0/454401524 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:50.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.241+0000 7f6632800640 1 --2- 172.21.15.43:0/454401524 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556ed242e880 0x556ed242c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:50.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.241+0000 7f6631e00640 1 -- 172.21.15.43:0/454401524 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556ed2425ca0 msgr2=0x556ed242bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:50.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.241+0000 7f6631e00640 1 --2- 172.21.15.43:0/454401524 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556ed2425ca0 0x556ed242bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:50.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.241+0000 7f6631e00640 1 -- 172.21.15.43:0/454401524 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556ed242e880 msgr2=0x556ed242c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:50.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.241+0000 7f6631e00640 1 --2- 172.21.15.43:0/454401524 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556ed242e880 0x556ed242c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:50.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.241+0000 7f6631e00640 1 -- 172.21.15.43:0/454401524 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ed21677d0 con 0x556ed2425380 2024-03-28T22:43:50.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.241+0000 7f6632800640 1 --2- 172.21.15.43:0/454401524 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556ed242e880 0x556ed242c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:50.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.242+0000 7f6631e00640 1 --2- 172.21.15.43:0/454401524 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556ed2425380 0x556ed242b8a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6624005c10 tx=0x7f662400db00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abf72161ab874844 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:50.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.242+0000 7f6617400640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:43:50.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.242+0000 7f6617400640 1 -- 172.21.15.43:0/454401524 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6624017420 con 0x556ed2425380 2024-03-28T22:43:50.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.242+0000 7f6617400640 1 -- 172.21.15.43:0/454401524 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f66240175c0 con 0x556ed2425380 2024-03-28T22:43:50.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.242+0000 7f663b980740 1 -- 172.21.15.43:0/454401524 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f662401c070 con 0x556ed2425380 2024-03-28T22:43:50.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.242+0000 7f6617400640 1 -- 172.21.15.43:0/454401524 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f66240177c0 con 0x556ed2425380 2024-03-28T22:43:50.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.243+0000 7f663b980740 10 client.17871 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:50.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.243+0000 7f663b980740 1 -- 172.21.15.43:0/454401524 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556ed243d8f0 con 0x556ed2425380 2024-03-28T22:43:50.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.243+0000 7f6617400640 1 -- 172.21.15.43:0/454401524 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f662401c070 con 0x556ed2425380 2024-03-28T22:43:50.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.243+0000 7f6617400640 1 -- 172.21.15.43:0/454401524 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f662400b040 con 0x556ed2425380 2024-03-28T22:43:50.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.243+0000 7f663b980740 10 client.17871 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:50.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.243+0000 7f663b980740 10 client.17871 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:43:50.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.243+0000 7f663b980740 20 client.17871 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:50.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.243+0000 7f663b980740 1 --2- 172.21.15.43:0/454401524 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x556ed243edb0 0x556ed24411a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:50.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.243+0000 7f663b980740 4 client.17871 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:43:50.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.243+0000 7f663b980740 1 -- 172.21.15.43:0/454401524 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ed2332680 con 0x556ed243edb0 2024-03-28T22:43:50.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.244+0000 7f6631400640 1 --2- 172.21.15.43:0/454401524 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x556ed243edb0 0x556ed24411a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:50.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.244+0000 7f6631400640 1 --2- 172.21.15.43:0/454401524 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x556ed243edb0 0x556ed24411a0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fba58e9075f9f62 server_cookie=a22af28fda8c3f83 in_seq=0 out_seq=0 2024-03-28T22:43:50.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.244+0000 7f6617400640 10 client.17871 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:50.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.246+0000 7f6617400640 1 -- 172.21.15.43:0/454401524 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556ed2332680 con 0x556ed243edb0 2024-03-28T22:43:50.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.246+0000 7f6617400640 10 client.17871 handle_command_reply: tid=0 2024-03-28T22:43:50.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.246+0000 7f663b980740 1 client.17871 shutdown 2024-03-28T22:43:50.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.246+0000 7f663b980740 1 -- 172.21.15.43:0/454401524 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556ed2425380 msgr2=0x556ed242b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:50.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.246+0000 7f663b980740 1 --2- 172.21.15.43:0/454401524 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556ed2425380 0x556ed242b8a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6624005c10 tx=0x7f662400db00 comp rx=0 tx=0).stop 2024-03-28T22:43:50.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.246+0000 7f663b980740 1 -- 172.21.15.43:0/454401524 shutdown_connections 2024-03-28T22:43:50.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.246+0000 7f663b980740 1 --2- 172.21.15.43:0/454401524 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x556ed243edb0 0x556ed24411a0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:50.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.246+0000 7f663b980740 1 --2- 172.21.15.43:0/454401524 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556ed242e880 0x556ed242c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:50.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.246+0000 7f663b980740 1 --2- 172.21.15.43:0/454401524 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556ed2425ca0 0x556ed242bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:50.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.246+0000 7f6617400640 0 client.17871 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:50.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.246+0000 7f663b980740 1 --2- 172.21.15.43:0/454401524 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556ed2425380 0x556ed242b8a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:50.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.246+0000 7f663b980740 1 -- 172.21.15.43:0/454401524 >> 172.21.15.43:0/454401524 conn(0x556ed241fab0 msgr2=0x556ed2421800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:50.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.247+0000 7f663b980740 1 -- 172.21.15.43:0/454401524 shutdown_connections 2024-03-28T22:43:50.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.247+0000 7f663b980740 1 -- 172.21.15.43:0/454401524 wait complete. 2024-03-28T22:43:50.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.247+0000 7f663b980740 20 client.17871 trim_cache size 0 max 16384 2024-03-28T22:43:50.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.262+0000 7f663b980740 1 Processor -- start 2024-03-28T22:43:50.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.262+0000 7f663b980740 1 -- start start 2024-03-28T22:43:50.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.262+0000 7f663b980740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556ed2425380 0x556ed243a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:50.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.262+0000 7f663b980740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556ed2425ca0 0x556ed2446930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:50.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.262+0000 7f663b980740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556ed242e880 0x556ed2436a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:50.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.262+0000 7f663b980740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f66341b0e00 con 0x556ed2425380 2024-03-28T22:43:50.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.262+0000 7f663b980740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f66341b2940 con 0x556ed2425ca0 2024-03-28T22:43:50.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.262+0000 7f663b980740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f66341b14d0 con 0x556ed242e880 2024-03-28T22:43:50.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.262+0000 7f6631400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556ed2425ca0 0x556ed2446930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:50.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.263+0000 7f6632800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556ed242e880 0x556ed2436a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:50.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.263+0000 7f6631400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556ed2425ca0 0x556ed2446930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47520/0 (socket says 172.21.15.43:47520) 2024-03-28T22:43:50.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.263+0000 7f6631400640 1 -- 172.21.15.43:0/364873611 learned_addr learned my addr 172.21.15.43:0/364873611 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:50.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.263+0000 7f6631e00640 1 --2- 172.21.15.43:0/364873611 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556ed2425380 0x556ed243a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:50.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.263+0000 7f6632800640 1 -- 172.21.15.43:0/364873611 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556ed2425ca0 msgr2=0x556ed2446930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:50.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.263+0000 7f6632800640 1 --2- 172.21.15.43:0/364873611 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556ed2425ca0 0x556ed2446930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:50.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.263+0000 7f6632800640 1 -- 172.21.15.43:0/364873611 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556ed2425380 msgr2=0x556ed243a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:50.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.263+0000 7f6632800640 1 --2- 172.21.15.43:0/364873611 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556ed2425380 0x556ed243a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:50.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.263+0000 7f6632800640 1 -- 172.21.15.43:0/364873611 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ed241f2f0 con 0x556ed242e880 2024-03-28T22:43:50.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.263+0000 7f6631400640 1 --2- 172.21.15.43:0/364873611 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556ed2425ca0 0x556ed2446930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:50.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.263+0000 7f6631e00640 1 --2- 172.21.15.43:0/364873611 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556ed2425380 0x556ed243a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:43:50.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.263+0000 7f6632800640 1 --2- 172.21.15.43:0/364873611 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556ed242e880 0x556ed2436a30 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f662c0cb7d0 tx=0x7f662c09b170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7800d80f1efa34b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:50.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.264+0000 7f6622600640 1 -- 172.21.15.43:0/364873611 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f662c01c040 con 0x556ed242e880 2024-03-28T22:43:50.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.264+0000 7f6622600640 1 -- 172.21.15.43:0/364873611 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f662c0a3210 con 0x556ed242e880 2024-03-28T22:43:50.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.264+0000 7f6622600640 1 -- 172.21.15.43:0/364873611 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f662c0a2d80 con 0x556ed242e880 2024-03-28T22:43:50.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.264+0000 7f663b980740 1 -- 172.21.15.43:0/364873611 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556ed242e880 msgr2=0x556ed2436a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:50.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.264+0000 7f663b980740 1 --2- 172.21.15.43:0/364873611 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556ed242e880 0x556ed2436a30 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f662c0cb7d0 tx=0x7f662c09b170 comp rx=0 tx=0).stop 2024-03-28T22:43:50.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.264+0000 7f663b980740 1 -- 172.21.15.43:0/364873611 shutdown_connections 2024-03-28T22:43:50.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.264+0000 7f663b980740 1 --2- 172.21.15.43:0/364873611 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556ed2425ca0 0x556ed2446930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:50.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.264+0000 7f663b980740 1 --2- 172.21.15.43:0/364873611 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556ed242e880 0x556ed2436a30 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:50.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.264+0000 7f663b980740 1 --2- 172.21.15.43:0/364873611 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556ed2425380 0x556ed243a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:50.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.264+0000 7f663b980740 1 -- 172.21.15.43:0/364873611 >> 172.21.15.43:0/364873611 conn(0x556ed241fab0 msgr2=0x556ed2421720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:50.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.265+0000 7f663b980740 1 -- 172.21.15.43:0/364873611 shutdown_connections 2024-03-28T22:43:50.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.265+0000 7f663b980740 1 -- 172.21.15.43:0/364873611 wait complete. 2024-03-28T22:43:50.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.265+0000 7f663b980740 1 -- start start 2024-03-28T22:43:50.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.265+0000 7f663b980740 1 Processor -- start 2024-03-28T22:43:50.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.265+0000 7f663b980740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556ed2425380 0x556ed2436980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:50.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.266+0000 7f663b980740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556ed2425ca0 0x556ed2433e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:50.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.266+0000 7f663b980740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556ed242e880 0x556ed2434340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:50.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.266+0000 7f663b980740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f66341b2270 con 0x556ed242e880 2024-03-28T22:43:50.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.266+0000 7f663b980740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f66341b1ba0 con 0x556ed2425ca0 2024-03-28T22:43:50.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.266+0000 7f663b980740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f66341aede0 con 0x556ed2425380 2024-03-28T22:43:50.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.266+0000 7f663b980740 10 client.0 _ll_register_callbacks cb 0x7f662400e450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:50.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.266+0000 7f6631e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556ed2425380 0x556ed2436980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:50.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.266+0000 7f6631e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556ed2425380 0x556ed2436980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:32924/0 (socket says 172.21.15.43:32924) 2024-03-28T22:43:50.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.266+0000 7f6631e00640 1 -- 172.21.15.43:0/1860769428 learned_addr learned my addr 172.21.15.43:0/1860769428 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:50.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.266+0000 7f6632800640 1 --2- 172.21.15.43:0/1860769428 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556ed242e880 0x556ed2434340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:50.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.266+0000 7f6631400640 1 --2- 172.21.15.43:0/1860769428 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556ed2425ca0 0x556ed2433e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:50.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.266+0000 7f6631e00640 1 -- 172.21.15.43:0/1860769428 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556ed2425ca0 msgr2=0x556ed2433e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:50.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.266+0000 7f6631e00640 1 --2- 172.21.15.43:0/1860769428 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556ed2425ca0 0x556ed2433e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:50.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.266+0000 7f6631e00640 1 -- 172.21.15.43:0/1860769428 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556ed242e880 msgr2=0x556ed2434340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:50.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.266+0000 7f6631e00640 1 --2- 172.21.15.43:0/1860769428 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556ed242e880 0x556ed2434340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:50.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.266+0000 7f6631e00640 1 -- 172.21.15.43:0/1860769428 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ed2434880 con 0x556ed2425380 2024-03-28T22:43:50.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.267+0000 7f6631e00640 1 --2- 172.21.15.43:0/1860769428 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556ed2425380 0x556ed2436980 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f6624000f80 tx=0x7f662400dc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a1cbcfaa105291f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:50.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.267+0000 7f6617400640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:43:50.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.267+0000 7f6617400640 1 -- 172.21.15.43:0/1860769428 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6624017420 con 0x556ed2425380 2024-03-28T22:43:50.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.267+0000 7f663b980740 1 -- 172.21.15.43:0/1860769428 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f662c0a3210 con 0x556ed2425380 2024-03-28T22:43:50.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.267+0000 7f6617400640 1 -- 172.21.15.43:0/1860769428 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f66240175c0 con 0x556ed2425380 2024-03-28T22:43:50.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.267+0000 7f6617400640 1 -- 172.21.15.43:0/1860769428 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6624003970 con 0x556ed2425380 2024-03-28T22:43:50.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.267+0000 7f663b980740 10 client.28066 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:50.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.267+0000 7f663b980740 1 -- 172.21.15.43:0/1860769428 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556ed24a4c20 con 0x556ed2425380 2024-03-28T22:43:50.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.268+0000 7f6617400640 1 -- 172.21.15.43:0/1860769428 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f662c0a3210 con 0x556ed2425380 2024-03-28T22:43:50.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.268+0000 7f6617400640 1 -- 172.21.15.43:0/1860769428 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f662400b040 con 0x556ed2425380 2024-03-28T22:43:50.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.268+0000 7f663b980740 10 client.28066 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:50.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.268+0000 7f663b980740 10 client.28066 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:43:50.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.268+0000 7f663b980740 20 client.28066 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:50.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.268+0000 7f663b980740 1 --2- 172.21.15.43:0/1860769428 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x556ed24a5150 0x556ed24a7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:50.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.268+0000 7f663b980740 4 client.28066 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:43:50.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.268+0000 7f663b980740 1 -- 172.21.15.43:0/1860769428 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556ed2396590 con 0x556ed24a5150 2024-03-28T22:43:50.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.268+0000 7f6631400640 1 --2- 172.21.15.43:0/1860769428 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x556ed24a5150 0x556ed24a7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:50.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.269+0000 7f6631400640 1 --2- 172.21.15.43:0/1860769428 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x556ed24a5150 0x556ed24a7540 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b79073807adeee20 server_cookie=7fd7ed3172ccb030 in_seq=0 out_seq=0 2024-03-28T22:43:50.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.269+0000 7f6617400640 10 client.28066 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:50.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.271+0000 7f6617400640 1 -- 172.21.15.43:0/1860769428 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556ed2396590 con 0x556ed24a5150 2024-03-28T22:43:50.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.271+0000 7f6617400640 10 client.28066 handle_command_reply: tid=0 2024-03-28T22:43:50.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.271+0000 7f663b980740 1 client.28066 shutdown 2024-03-28T22:43:50.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.271+0000 7f663b980740 1 -- 172.21.15.43:0/1860769428 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556ed2425380 msgr2=0x556ed2436980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:50.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.271+0000 7f663b980740 1 --2- 172.21.15.43:0/1860769428 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556ed2425380 0x556ed2436980 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f6624000f80 tx=0x7f662400dc70 comp rx=0 tx=0).stop 2024-03-28T22:43:50.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.271+0000 7f663b980740 1 -- 172.21.15.43:0/1860769428 shutdown_connections 2024-03-28T22:43:50.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.271+0000 7f663b980740 1 --2- 172.21.15.43:0/1860769428 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x556ed24a5150 0x556ed24a7540 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:50.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.271+0000 7f663b980740 1 --2- 172.21.15.43:0/1860769428 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556ed2425ca0 0x556ed2433e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:50.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.271+0000 7f663b980740 1 --2- 172.21.15.43:0/1860769428 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556ed242e880 0x556ed2434340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:50.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.271+0000 7f6617400640 0 client.28066 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:50.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.271+0000 7f663b980740 1 --2- 172.21.15.43:0/1860769428 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556ed2425380 0x556ed2436980 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:50.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.271+0000 7f663b980740 1 -- 172.21.15.43:0/1860769428 >> 172.21.15.43:0/1860769428 conn(0x556ed241fab0 msgr2=0x556ed241e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:50.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.271+0000 7f663b980740 1 -- 172.21.15.43:0/1860769428 shutdown_connections 2024-03-28T22:43:50.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.271+0000 7f663b980740 1 -- 172.21.15.43:0/1860769428 wait complete. 2024-03-28T22:43:50.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.271+0000 7f663b980740 20 client.28066 trim_cache size 0 max 16384 2024-03-28T22:43:50.279 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:43:50.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.272+0000 7f6638c00640 1 -- 172.21.15.43:0/407022670 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f66180a1ef0 msgr2=0x7f66180a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:50.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.272+0000 7f6638c00640 1 --2- 172.21.15.43:0/407022670 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f66180a1ef0 0x7f66180a43b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f661c005e00 tx=0x7f661c02bc10 comp rx=0 tx=0).stop 2024-03-28T22:43:50.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.272+0000 7f6638c00640 1 -- 172.21.15.43:0/407022670 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f663410bd60 msgr2=0x7f663419eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:50.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.272+0000 7f6638c00640 1 --2- 172.21.15.43:0/407022670 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f663410bd60 0x7f663419eac0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f662c00e940 tx=0x7f662c00ee00 comp rx=0 tx=0).stop 2024-03-28T22:43:50.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.272+0000 7f6638c00640 1 -- 172.21.15.43:0/407022670 shutdown_connections 2024-03-28T22:43:50.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.272+0000 7f6638c00640 1 --2- 172.21.15.43:0/407022670 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f66180a1ef0 0x7f66180a43b0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:50.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.272+0000 7f6638c00640 1 --2- 172.21.15.43:0/407022670 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f66341032b0 0x7f663419a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:50.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.272+0000 7f6638c00640 1 --2- 172.21.15.43:0/407022670 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f663410bd60 0x7f663419eac0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:50.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.272+0000 7f6638c00640 1 --2- 172.21.15.43:0/407022670 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6634102990 0x7f663419a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:50.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.272+0000 7f6638c00640 1 -- 172.21.15.43:0/407022670 >> 172.21.15.43:0/407022670 conn(0x7f66340769d0 msgr2=0x7f66340ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:50.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.272+0000 7f6638c00640 1 -- 172.21.15.43:0/407022670 shutdown_connections 2024-03-28T22:43:50.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:50.273+0000 7f6638c00640 1 -- 172.21.15.43:0/407022670 wait complete. 2024-03-28T22:43:50.378 INFO:tasks.workunit.client.0.smithi043.stdout: 1 457826 7.31 MB/sec execute 201 sec latency 35.460 ms 2024-03-28T22:43:51.290 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:43:51.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:43:51.291 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:43:51.378 INFO:tasks.workunit.client.0.smithi043.stdout: 1 458517 7.31 MB/sec execute 202 sec latency 76.980 ms 2024-03-28T22:43:51.384 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:51 smithi043 ceph-mon[31823]: pgmap v204: 129 pgs: 129 active+clean; 51 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 194 op/s 2024-03-28T22:43:51.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:51 smithi172 ceph-mon[36566]: pgmap v204: 129 pgs: 129 active+clean; 51 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 194 op/s 2024-03-28T22:43:51.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:51 smithi082 ceph-mon[36471]: pgmap v204: 129 pgs: 129 active+clean; 51 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 194 op/s 2024-03-28T22:43:51.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.567+0000 7f45a6200640 1 -- 172.21.15.43:0/4174149690 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f45a0102840 msgr2=0x7f45a0102c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:51.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.567+0000 7f45a6200640 1 --2- 172.21.15.43:0/4174149690 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f45a0102840 0x7f45a0102c20 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f4594009900 tx=0x7f459402bcd0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.567+0000 7f45a6200640 1 -- 172.21.15.43:0/4174149690 shutdown_connections 2024-03-28T22:43:51.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.567+0000 7f45a6200640 1 --2- 172.21.15.43:0/4174149690 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45a010bc10 0x7f45a010e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.567+0000 7f45a6200640 1 --2- 172.21.15.43:0/4174149690 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45a0103160 0x7f45a010b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.567+0000 7f45a6200640 1 --2- 172.21.15.43:0/4174149690 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f45a0102840 0x7f45a0102c20 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.567+0000 7f45a6200640 1 -- 172.21.15.43:0/4174149690 >> 172.21.15.43:0/4174149690 conn(0x7f45a00fbfa0 msgr2=0x7f45a00fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:51.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.567+0000 7f45a6200640 1 -- 172.21.15.43:0/4174149690 shutdown_connections 2024-03-28T22:43:51.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.567+0000 7f45a6200640 1 -- 172.21.15.43:0/4174149690 wait complete. 2024-03-28T22:43:51.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.568+0000 7f45a6200640 1 Processor -- start 2024-03-28T22:43:51.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.568+0000 7f45a6200640 1 -- start start 2024-03-28T22:43:51.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.569+0000 7f45a6200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45a0102840 0x7f45a0100490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.569+0000 7f45a6200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45a0103160 0x7f45a01009d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.569+0000 7f45a6200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f45a010bc10 0x7f45a0100f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.569+0000 7f45a6200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f45a01119f0 con 0x7f45a0103160 2024-03-28T22:43:51.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.569+0000 7f45a6200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f45a0111870 con 0x7f45a010bc10 2024-03-28T22:43:51.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.569+0000 7f45a6200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f45a0111b70 con 0x7f45a0102840 2024-03-28T22:43:51.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.569+0000 7f45a4e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45a0102840 0x7f45a0100490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.569+0000 7f45a4e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45a0102840 0x7f45a0100490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:32954/0 (socket says 172.21.15.43:32954) 2024-03-28T22:43:51.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.569+0000 7f45a4e00640 1 -- 172.21.15.43:0/2819781132 learned_addr learned my addr 172.21.15.43:0/2819781132 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:51.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.569+0000 7f45a5800640 1 --2- 172.21.15.43:0/2819781132 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f45a010bc10 0x7f45a0100f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.569+0000 7f459fe00640 1 --2- 172.21.15.43:0/2819781132 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45a0103160 0x7f45a01009d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.569+0000 7f45a4e00640 1 -- 172.21.15.43:0/2819781132 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f45a010bc10 msgr2=0x7f45a0100f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:51.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.569+0000 7f45a4e00640 1 --2- 172.21.15.43:0/2819781132 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f45a010bc10 0x7f45a0100f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.569+0000 7f45a4e00640 1 -- 172.21.15.43:0/2819781132 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45a0103160 msgr2=0x7f45a01009d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:51.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.569+0000 7f45a4e00640 1 --2- 172.21.15.43:0/2819781132 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45a0103160 0x7f45a01009d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.569+0000 7f45a4e00640 1 -- 172.21.15.43:0/2819781132 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45a01a6370 con 0x7f45a0102840 2024-03-28T22:43:51.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.570+0000 7f45a4e00640 1 --2- 172.21.15.43:0/2819781132 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45a0102840 0x7f45a0100490 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f4594000c00 tx=0x7f4594004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:51.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.570+0000 7f459d600640 1 -- 172.21.15.43:0/2819781132 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4594042020 con 0x7f45a0102840 2024-03-28T22:43:51.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.570+0000 7f459d600640 1 -- 172.21.15.43:0/2819781132 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45940054e0 con 0x7f45a0102840 2024-03-28T22:43:51.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.570+0000 7f45a6200640 1 -- 172.21.15.43:0/2819781132 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45a01a6600 con 0x7f45a0102840 2024-03-28T22:43:51.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.570+0000 7f459d600640 1 -- 172.21.15.43:0/2819781132 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4594003af0 con 0x7f45a0102840 2024-03-28T22:43:51.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.570+0000 7f45a6200640 1 -- 172.21.15.43:0/2819781132 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45a01a6b50 con 0x7f45a0102840 2024-03-28T22:43:51.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.573+0000 7f459d600640 1 -- 172.21.15.43:0/2819781132 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f4594045020 con 0x7f45a0102840 2024-03-28T22:43:51.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.573+0000 7f45aa572740 1 Processor -- start 2024-03-28T22:43:51.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.573+0000 7f45aa572740 1 -- start start 2024-03-28T22:43:51.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.574+0000 7f45aa572740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55aa27520170 0x55aa27520550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.574+0000 7f45aa572740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aa27520a90 0x55aa27529130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.574+0000 7f45aa572740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa27529670 0x55aa2752ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.574+0000 7f45aa572740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55aa27483c40 con 0x55aa27529670 2024-03-28T22:43:51.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.574+0000 7f45aa572740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55aa2742cc40 con 0x55aa27520170 2024-03-28T22:43:51.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.574+0000 7f45aa572740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55aa2747fce0 con 0x55aa27520a90 2024-03-28T22:43:51.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.574+0000 7f45a5800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55aa27520170 0x55aa27520550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.574+0000 7f45a5800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55aa27520170 0x55aa27520550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47566/0 (socket says 172.21.15.43:47566) 2024-03-28T22:43:51.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.574+0000 7f45a5800640 1 -- 172.21.15.43:0/3173099445 learned_addr learned my addr 172.21.15.43:0/3173099445 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:51.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.574+0000 7f45a4e00640 1 --2- 172.21.15.43:0/3173099445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aa27520a90 0x55aa27529130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.574+0000 7f459fe00640 1 --2- 172.21.15.43:0/3173099445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa27529670 0x55aa2752ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.574+0000 7f45a5800640 1 -- 172.21.15.43:0/3173099445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aa27520a90 msgr2=0x55aa27529130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:51.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.574+0000 7f45a5800640 1 --2- 172.21.15.43:0/3173099445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aa27520a90 0x55aa27529130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.574+0000 7f45a5800640 1 -- 172.21.15.43:0/3173099445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa27529670 msgr2=0x55aa2752ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:51.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.574+0000 7f45a5800640 1 --2- 172.21.15.43:0/3173099445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa27529670 0x55aa2752ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.575+0000 7f45a5800640 1 -- 172.21.15.43:0/3173099445 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa272607e0 con 0x55aa27520170 2024-03-28T22:43:51.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.575+0000 7f45a5800640 1 --2- 172.21.15.43:0/3173099445 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55aa27520170 0x55aa27520550 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f4598009f30 tx=0x7f45980118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98ba2bde6a45f205 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:51.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.575+0000 7f4592000640 1 -- 172.21.15.43:0/3173099445 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4598004d20 con 0x55aa27520170 2024-03-28T22:43:51.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.575+0000 7f4592000640 1 -- 172.21.15.43:0/3173099445 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4598004050 con 0x55aa27520170 2024-03-28T22:43:51.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.575+0000 7f4592000640 1 -- 172.21.15.43:0/3173099445 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4598005220 con 0x55aa27520170 2024-03-28T22:43:51.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.575+0000 7f459d600640 1 --2- 172.21.15.43:0/2819781132 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f45880a1f20 0x7f45880a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.575+0000 7f459d600640 1 -- 172.21.15.43:0/2819781132 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f45940effc0 con 0x7f45a0102840 2024-03-28T22:43:51.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.575+0000 7f45a4e00640 1 --2- 172.21.15.43:0/2819781132 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f45880a1f20 0x7f45880a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.576+0000 7f45aa572740 1 -- 172.21.15.43:0/3173099445 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55aa27520170 msgr2=0x55aa27520550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:51.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.576+0000 7f45aa572740 1 --2- 172.21.15.43:0/3173099445 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55aa27520170 0x55aa27520550 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f4598009f30 tx=0x7f45980118d0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.576+0000 7f45aa572740 1 -- 172.21.15.43:0/3173099445 shutdown_connections 2024-03-28T22:43:51.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.576+0000 7f45aa572740 1 --2- 172.21.15.43:0/3173099445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa27529670 0x55aa2752ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.576+0000 7f45aa572740 1 --2- 172.21.15.43:0/3173099445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aa27520a90 0x55aa27529130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.576+0000 7f45aa572740 1 --2- 172.21.15.43:0/3173099445 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55aa27520170 0x55aa27520550 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.576+0000 7f45aa572740 1 -- 172.21.15.43:0/3173099445 >> 172.21.15.43:0/3173099445 conn(0x55aa2751a8a0 msgr2=0x55aa2751ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:51.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.576+0000 7f45aa572740 1 -- 172.21.15.43:0/3173099445 shutdown_connections 2024-03-28T22:43:51.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.576+0000 7f45aa572740 1 -- 172.21.15.43:0/3173099445 wait complete. 2024-03-28T22:43:51.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.576+0000 7f45a4e00640 1 --2- 172.21.15.43:0/2819781132 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f45880a1f20 0x7f45880a43e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f45940ed530 tx=0x7f45940dc040 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:51.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.577+0000 7f45aa572740 1 -- start start 2024-03-28T22:43:51.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.577+0000 7f45aa572740 1 Processor -- start 2024-03-28T22:43:51.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.577+0000 7f45aa572740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa27520170 0x55aa27526690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.577+0000 7f45aa572740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aa27520a90 0x55aa27526bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.577+0000 7f45aa572740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55aa27529670 0x55aa27527110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.577+0000 7f45aa572740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55aa2752c460 con 0x55aa27520170 2024-03-28T22:43:51.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.577+0000 7f45aa572740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55aa27422250 con 0x55aa27529670 2024-03-28T22:43:51.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.577+0000 7f45aa572740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55aa2752c5e0 con 0x55aa27520a90 2024-03-28T22:43:51.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.577+0000 7f45aa572740 10 client.0 _ll_register_callbacks cb 0x55aa27443e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:51.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.578+0000 7f459fe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aa27520a90 0x55aa27526bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.578+0000 7f459fe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aa27520a90 0x55aa27526bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:32974/0 (socket says 172.21.15.43:32974) 2024-03-28T22:43:51.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.578+0000 7f459fe00640 1 -- 172.21.15.43:0/492396850 learned_addr learned my addr 172.21.15.43:0/492396850 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:51.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.578+0000 7f45a5800640 1 --2- 172.21.15.43:0/492396850 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa27520170 0x55aa27526690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.578+0000 7f45a5800640 1 --2- 172.21.15.43:0/492396850 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55aa27529670 0x55aa27527110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.578+0000 7f459fe00640 1 -- 172.21.15.43:0/492396850 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55aa27529670 msgr2=0x55aa27527110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:51.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.578+0000 7f459fe00640 1 --2- 172.21.15.43:0/492396850 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55aa27529670 0x55aa27527110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.578+0000 7f459fe00640 1 -- 172.21.15.43:0/492396850 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa27520170 msgr2=0x55aa27526690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:51.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.578+0000 7f459fe00640 1 --2- 172.21.15.43:0/492396850 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa27520170 0x55aa27526690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.578+0000 7f459fe00640 1 -- 172.21.15.43:0/492396850 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa272c0f70 con 0x55aa27520a90 2024-03-28T22:43:51.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.578+0000 7f45a5800640 1 --2- 172.21.15.43:0/492396850 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55aa27529670 0x55aa27527110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:43:51.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.578+0000 7f459fe00640 1 --2- 172.21.15.43:0/492396850 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aa27520a90 0x55aa27526bd0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f458c012700 tx=0x7f458c012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64564a22a8e2c4a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:51.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.578+0000 7f4586a00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:43:51.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.579+0000 7f4586a00640 1 -- 172.21.15.43:0/492396850 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458c004050 con 0x55aa27520a90 2024-03-28T22:43:51.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.579+0000 7f45aa572740 1 -- 172.21.15.43:0/492396850 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4598004050 con 0x55aa27520a90 2024-03-28T22:43:51.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.579+0000 7f4586a00640 1 -- 172.21.15.43:0/492396850 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f458c0041f0 con 0x55aa27520a90 2024-03-28T22:43:51.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.579+0000 7f4586a00640 1 -- 172.21.15.43:0/492396850 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f458c025050 con 0x55aa27520a90 2024-03-28T22:43:51.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.579+0000 7f45aa572740 10 client.28078 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:51.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.579+0000 7f45aa572740 1 -- 172.21.15.43:0/492396850 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55aa27158170 con 0x55aa27520a90 2024-03-28T22:43:51.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.579+0000 7f4586a00640 1 -- 172.21.15.43:0/492396850 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4598004050 con 0x55aa27520a90 2024-03-28T22:43:51.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.580+0000 7f4586a00640 1 -- 172.21.15.43:0/492396850 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f458c003a80 con 0x55aa27520a90 2024-03-28T22:43:51.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.580+0000 7f45aa572740 10 client.28078 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:51.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.580+0000 7f45aa572740 10 client.28078 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:43:51.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.580+0000 7f45aa572740 20 client.28078 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:51.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.580+0000 7f45aa572740 1 --2- 172.21.15.43:0/492396850 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55aa275399d0 0x55aa2753bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.580+0000 7f45aa572740 4 client.28078 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:43:51.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.580+0000 7f45aa572740 1 -- 172.21.15.43:0/492396850 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aa27491180 con 0x55aa275399d0 2024-03-28T22:43:51.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.580+0000 7f45a5800640 1 --2- 172.21.15.43:0/492396850 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55aa275399d0 0x55aa2753bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.580+0000 7f45a5800640 1 --2- 172.21.15.43:0/492396850 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55aa275399d0 0x55aa2753bdc0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb8da3e9b1d4ed37 server_cookie=a22c664e993271bb in_seq=0 out_seq=0 2024-03-28T22:43:51.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.580+0000 7f4586a00640 10 client.28078 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:51.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.582+0000 7f4586a00640 1 -- 172.21.15.43:0/492396850 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55aa27491180 con 0x55aa275399d0 2024-03-28T22:43:51.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.582+0000 7f4586a00640 10 client.28078 handle_command_reply: tid=0 2024-03-28T22:43:51.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.582+0000 7f45aa572740 1 client.28078 shutdown 2024-03-28T22:43:51.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.582+0000 7f45aa572740 1 -- 172.21.15.43:0/492396850 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aa27520a90 msgr2=0x55aa27526bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:51.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.582+0000 7f45aa572740 1 --2- 172.21.15.43:0/492396850 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aa27520a90 0x55aa27526bd0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f458c012700 tx=0x7f458c012bc0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.583+0000 7f45aa572740 1 -- 172.21.15.43:0/492396850 shutdown_connections 2024-03-28T22:43:51.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.583+0000 7f45aa572740 1 --2- 172.21.15.43:0/492396850 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55aa275399d0 0x55aa2753bdc0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.583+0000 7f45aa572740 1 --2- 172.21.15.43:0/492396850 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55aa27529670 0x55aa27527110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.583+0000 7f45aa572740 1 --2- 172.21.15.43:0/492396850 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aa27520a90 0x55aa27526bd0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.583+0000 7f45aa572740 1 --2- 172.21.15.43:0/492396850 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa27520170 0x55aa27526690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.583+0000 7f45aa572740 1 -- 172.21.15.43:0/492396850 >> 172.21.15.43:0/492396850 conn(0x55aa2751a8a0 msgr2=0x55aa2751c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:51.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.583+0000 7f4586a00640 0 client.28078 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:51.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.583+0000 7f45aa572740 1 -- 172.21.15.43:0/492396850 shutdown_connections 2024-03-28T22:43:51.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.583+0000 7f45aa572740 1 -- 172.21.15.43:0/492396850 wait complete. 2024-03-28T22:43:51.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.583+0000 7f45aa572740 20 client.28078 trim_cache size 0 max 16384 2024-03-28T22:43:51.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.604+0000 7f45aa572740 1 Processor -- start 2024-03-28T22:43:51.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.604+0000 7f45aa572740 1 -- start start 2024-03-28T22:43:51.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.605+0000 7f45aa572740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55aa27520170 0x55aa27528dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.605+0000 7f45aa572740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa27520a90 0x55aa27535200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.605+0000 7f45aa572740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aa27529670 0x55aa275416d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.605+0000 7f45aa572740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f45a01b0c00 con 0x55aa27520a90 2024-03-28T22:43:51.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.605+0000 7f45aa572740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f45a01b2740 con 0x55aa27520170 2024-03-28T22:43:51.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.605+0000 7f45aa572740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f45a01b12d0 con 0x55aa27529670 2024-03-28T22:43:51.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.605+0000 7f45a5800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55aa27520170 0x55aa27528dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.605+0000 7f459fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa27520a90 0x55aa27535200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.605+0000 7f45a5800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55aa27520170 0x55aa27528dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47590/0 (socket says 172.21.15.43:47590) 2024-03-28T22:43:51.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.605+0000 7f45a5800640 1 -- 172.21.15.43:0/4205712546 learned_addr learned my addr 172.21.15.43:0/4205712546 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:51.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.605+0000 7f45a5800640 1 --2- 172.21.15.43:0/4205712546 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aa27529670 0x55aa275416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.606+0000 7f45a5800640 1 -- 172.21.15.43:0/4205712546 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aa27529670 msgr2=0x55aa275416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:51.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.606+0000 7f45a5800640 1 --2- 172.21.15.43:0/4205712546 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aa27529670 0x55aa275416d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.606+0000 7f45a5800640 1 -- 172.21.15.43:0/4205712546 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa27520a90 msgr2=0x55aa27535200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:51.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.606+0000 7f45a5800640 1 --2- 172.21.15.43:0/4205712546 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa27520a90 0x55aa27535200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.606+0000 7f45a5800640 1 -- 172.21.15.43:0/4205712546 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa2751f260 con 0x55aa27520170 2024-03-28T22:43:51.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.606+0000 7f459fe00640 1 --2- 172.21.15.43:0/4205712546 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa27520a90 0x55aa27535200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:51.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.606+0000 7f45a5800640 1 --2- 172.21.15.43:0/4205712546 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55aa27520170 0x55aa27528dc0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f4598009ed0 tx=0x7f4598009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc4c0dfad92649e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:51.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.606+0000 7f4591600640 1 -- 172.21.15.43:0/4205712546 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4598004250 con 0x55aa27520170 2024-03-28T22:43:51.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.606+0000 7f4591600640 1 -- 172.21.15.43:0/4205712546 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45980043f0 con 0x55aa27520170 2024-03-28T22:43:51.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.606+0000 7f4591600640 1 -- 172.21.15.43:0/4205712546 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4598028e00 con 0x55aa27520170 2024-03-28T22:43:51.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.607+0000 7f45aa572740 1 -- 172.21.15.43:0/4205712546 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55aa27520170 msgr2=0x55aa27528dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:51.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.607+0000 7f45aa572740 1 --2- 172.21.15.43:0/4205712546 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55aa27520170 0x55aa27528dc0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f4598009ed0 tx=0x7f4598009500 comp rx=0 tx=0).stop 2024-03-28T22:43:51.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.607+0000 7f45aa572740 1 -- 172.21.15.43:0/4205712546 shutdown_connections 2024-03-28T22:43:51.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.607+0000 7f45aa572740 1 --2- 172.21.15.43:0/4205712546 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aa27529670 0x55aa275416d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.607+0000 7f45aa572740 1 --2- 172.21.15.43:0/4205712546 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa27520a90 0x55aa27535200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.607+0000 7f45aa572740 1 --2- 172.21.15.43:0/4205712546 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55aa27520170 0x55aa27528dc0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.607+0000 7f45aa572740 1 -- 172.21.15.43:0/4205712546 >> 172.21.15.43:0/4205712546 conn(0x55aa2751a8a0 msgr2=0x55aa27526ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:51.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.607+0000 7f45aa572740 1 -- 172.21.15.43:0/4205712546 shutdown_connections 2024-03-28T22:43:51.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.607+0000 7f45aa572740 1 -- 172.21.15.43:0/4205712546 wait complete. 2024-03-28T22:43:51.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.608+0000 7f45aa572740 1 -- start start 2024-03-28T22:43:51.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.608+0000 7f45aa572740 1 Processor -- start 2024-03-28T22:43:51.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.608+0000 7f45aa572740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aa27520170 0x55aa27531b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.608+0000 7f45aa572740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55aa27520a90 0x55aa2752e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.608+0000 7f45aa572740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa27529670 0x55aa2752ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.608+0000 7f45aa572740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f45a01b2070 con 0x55aa27529670 2024-03-28T22:43:51.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.608+0000 7f45aa572740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f45a01b19a0 con 0x55aa27520a90 2024-03-28T22:43:51.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.608+0000 7f45aa572740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f45a01aebe0 con 0x55aa27520170 2024-03-28T22:43:51.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.608+0000 7f45aa572740 10 client.0 _ll_register_callbacks cb 0x7f458c009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:51.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.609+0000 7f459fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55aa27520a90 0x55aa2752e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.609+0000 7f45a5800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aa27520170 0x55aa27531b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.609+0000 7f459fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55aa27520a90 0x55aa2752e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47592/0 (socket says 172.21.15.43:47592) 2024-03-28T22:43:51.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.609+0000 7f459fe00640 1 -- 172.21.15.43:0/671646206 learned_addr learned my addr 172.21.15.43:0/671646206 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:51.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.609+0000 7f45a5800640 1 --2- 172.21.15.43:0/671646206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa27529670 0x55aa2752ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.609+0000 7f459fe00640 1 -- 172.21.15.43:0/671646206 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aa27520170 msgr2=0x55aa27531b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:51.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.609+0000 7f459fe00640 1 --2- 172.21.15.43:0/671646206 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aa27520170 0x55aa27531b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.609+0000 7f459fe00640 1 -- 172.21.15.43:0/671646206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa27529670 msgr2=0x55aa2752ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:51.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.609+0000 7f459fe00640 1 --2- 172.21.15.43:0/671646206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa27529670 0x55aa2752ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.609+0000 7f459fe00640 1 -- 172.21.15.43:0/671646206 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa2752f590 con 0x55aa27520a90 2024-03-28T22:43:51.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.609+0000 7f45a5800640 1 --2- 172.21.15.43:0/671646206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa27529670 0x55aa2752ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:43:51.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.609+0000 7f459fe00640 1 --2- 172.21.15.43:0/671646206 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55aa27520a90 0x55aa2752e800 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x55aa27525ef0 tx=0x7f458c00f800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad79cd57f94c2292 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:51.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.609+0000 7f4586a00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:43:51.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.610+0000 7f4586a00640 1 -- 172.21.15.43:0/671646206 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458c009cb0 con 0x55aa27520a90 2024-03-28T22:43:51.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.610+0000 7f4586a00640 1 -- 172.21.15.43:0/671646206 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f458c022030 con 0x55aa27520a90 2024-03-28T22:43:51.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.610+0000 7f45aa572740 1 -- 172.21.15.43:0/671646206 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45980043f0 con 0x55aa27520a90 2024-03-28T22:43:51.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.610+0000 7f4586a00640 1 -- 172.21.15.43:0/671646206 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f458c010b60 con 0x55aa27520a90 2024-03-28T22:43:51.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.610+0000 7f45aa572740 10 client.28142 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:51.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.610+0000 7f45aa572740 1 -- 172.21.15.43:0/671646206 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55aa275a0de0 con 0x55aa27520a90 2024-03-28T22:43:51.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.610+0000 7f4586a00640 1 -- 172.21.15.43:0/671646206 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45980043f0 con 0x55aa27520a90 2024-03-28T22:43:51.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.610+0000 7f4586a00640 1 -- 172.21.15.43:0/671646206 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f458c005340 con 0x55aa27520a90 2024-03-28T22:43:51.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.610+0000 7f45aa572740 10 client.28142 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:51.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.610+0000 7f45aa572740 10 client.28142 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:43:51.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.611+0000 7f45aa572740 20 client.28142 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:51.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.611+0000 7f45aa572740 1 --2- 172.21.15.43:0/671646206 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55aa275a1590 0x55aa275a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.611+0000 7f45aa572740 4 client.28142 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:43:51.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.611+0000 7f45aa572740 1 -- 172.21.15.43:0/671646206 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55aa2748b4d0 con 0x55aa275a1590 2024-03-28T22:43:51.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.611+0000 7f45a5800640 1 --2- 172.21.15.43:0/671646206 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55aa275a1590 0x55aa275a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.611+0000 7f45a5800640 1 --2- 172.21.15.43:0/671646206 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55aa275a1590 0x55aa275a39b0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3742e76735f6abb server_cookie=1be5ad2517fe9b9c in_seq=0 out_seq=0 2024-03-28T22:43:51.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.611+0000 7f4586a00640 10 client.28142 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:51.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.613+0000 7f4586a00640 1 -- 172.21.15.43:0/671646206 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55aa2748b4d0 con 0x55aa275a1590 2024-03-28T22:43:51.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.613+0000 7f4586a00640 10 client.28142 handle_command_reply: tid=0 2024-03-28T22:43:51.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.613+0000 7f45aa572740 1 client.28142 shutdown 2024-03-28T22:43:51.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.613+0000 7f45aa572740 1 -- 172.21.15.43:0/671646206 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55aa27520a90 msgr2=0x55aa2752e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:51.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.613+0000 7f45aa572740 1 --2- 172.21.15.43:0/671646206 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55aa27520a90 0x55aa2752e800 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x55aa27525ef0 tx=0x7f458c00f800 comp rx=0 tx=0).stop 2024-03-28T22:43:51.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.613+0000 7f45aa572740 1 -- 172.21.15.43:0/671646206 shutdown_connections 2024-03-28T22:43:51.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.613+0000 7f45aa572740 1 --2- 172.21.15.43:0/671646206 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55aa275a1590 0x55aa275a39b0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.613+0000 7f45aa572740 1 --2- 172.21.15.43:0/671646206 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55aa27520a90 0x55aa2752e800 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.613+0000 7f45aa572740 1 --2- 172.21.15.43:0/671646206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55aa27529670 0x55aa2752ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.613+0000 7f45aa572740 1 --2- 172.21.15.43:0/671646206 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55aa27520170 0x55aa27531b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.613+0000 7f4586a00640 0 client.28142 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:51.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.613+0000 7f45aa572740 1 -- 172.21.15.43:0/671646206 >> 172.21.15.43:0/671646206 conn(0x55aa2751a8a0 msgr2=0x55aa27519070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:51.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.614+0000 7f45aa572740 1 -- 172.21.15.43:0/671646206 shutdown_connections 2024-03-28T22:43:51.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.614+0000 7f45aa572740 1 -- 172.21.15.43:0/671646206 wait complete. 2024-03-28T22:43:51.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.614+0000 7f45aa572740 20 client.28142 trim_cache size 0 max 16384 2024-03-28T22:43:51.615 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:43:51.615 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-03-28T22:43:51.615 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "cff0e42b-1491-43ad-9800-30926e70d782", 2024-03-28T22:43:51.615 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-03-28T22:43:51.615 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:43:51.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.614+0000 7f45a6200640 1 -- 172.21.15.43:0/2819781132 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f45880a1f20 msgr2=0x7f45880a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:51.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.614+0000 7f45a6200640 1 --2- 172.21.15.43:0/2819781132 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f45880a1f20 0x7f45880a43e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f45940ed530 tx=0x7f45940dc040 comp rx=0 tx=0).stop 2024-03-28T22:43:51.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.614+0000 7f45a6200640 1 -- 172.21.15.43:0/2819781132 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45a0102840 msgr2=0x7f45a0100490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:51.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.614+0000 7f45a6200640 1 --2- 172.21.15.43:0/2819781132 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45a0102840 0x7f45a0100490 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f4594000c00 tx=0x7f4594004360 comp rx=0 tx=0).stop 2024-03-28T22:43:51.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.615+0000 7f45a6200640 1 -- 172.21.15.43:0/2819781132 shutdown_connections 2024-03-28T22:43:51.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.615+0000 7f45a6200640 1 --2- 172.21.15.43:0/2819781132 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f45880a1f20 0x7f45880a43e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.615+0000 7f45a6200640 1 --2- 172.21.15.43:0/2819781132 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f45a010bc10 0x7f45a0100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.615+0000 7f45a6200640 1 --2- 172.21.15.43:0/2819781132 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f45a0103160 0x7f45a01009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.615+0000 7f45a6200640 1 --2- 172.21.15.43:0/2819781132 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f45a0102840 0x7f45a0100490 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.615+0000 7f45a6200640 1 -- 172.21.15.43:0/2819781132 >> 172.21.15.43:0/2819781132 conn(0x7f45a00fbfa0 msgr2=0x7f45a00fccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:51.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.615+0000 7f45a6200640 1 -- 172.21.15.43:0/2819781132 shutdown_connections 2024-03-28T22:43:51.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.615+0000 7f45a6200640 1 -- 172.21.15.43:0/2819781132 wait complete. 2024-03-28T22:43:51.632 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:43:51.632 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:43:51.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.850+0000 7fa5fd000640 1 -- 172.21.15.43:0/1313124841 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5f8108930 msgr2=0x7fa5f8108d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:51.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.850+0000 7fa5fd000640 1 --2- 172.21.15.43:0/1313124841 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5f8108930 0x7fa5f8108d10 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fa5e8009900 tx=0x7fa5e802bc50 comp rx=0 tx=0).stop 2024-03-28T22:43:51.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.853+0000 7fa5fd000640 1 -- 172.21.15.43:0/1313124841 shutdown_connections 2024-03-28T22:43:51.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.853+0000 7fa5fd000640 1 --2- 172.21.15.43:0/1313124841 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa5f81148a0 0x7fa5f8116c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.853+0000 7fa5fd000640 1 --2- 172.21.15.43:0/1313124841 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5f8109250 0x7fa5f8114360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.853+0000 7fa5fd000640 1 --2- 172.21.15.43:0/1313124841 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5f8108930 0x7fa5f8108d10 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.853+0000 7fa5fd000640 1 -- 172.21.15.43:0/1313124841 >> 172.21.15.43:0/1313124841 conn(0x7fa5f806c830 msgr2=0x7fa5f806cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:51.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.854+0000 7fa5fd000640 1 -- 172.21.15.43:0/1313124841 shutdown_connections 2024-03-28T22:43:51.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.855+0000 7fa5fd000640 1 -- 172.21.15.43:0/1313124841 wait complete. 2024-03-28T22:43:51.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.857+0000 7fa5fd000640 1 Processor -- start 2024-03-28T22:43:51.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.860+0000 7fa5fd000640 1 -- start start 2024-03-28T22:43:51.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.861+0000 7fa5fd000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5f8108930 0x7fa5f81141f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.861+0000 7fa5fd000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5f8109250 0x7fa5f81112e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.861+0000 7fa5fd000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa5f81148a0 0x7fa5f8111820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.861+0000 7fa5fd000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f811c4f0 con 0x7fa5f8109250 2024-03-28T22:43:51.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.861+0000 7fa5fd000640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f811c370 con 0x7fa5f81148a0 2024-03-28T22:43:51.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.861+0000 7fa5fd000640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f811c670 con 0x7fa5f8108930 2024-03-28T22:43:51.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.862+0000 7fa5f6800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa5f81148a0 0x7fa5f8111820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.862+0000 7fa5f5400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5f8109250 0x7fa5f81112e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.862+0000 7fa5f6800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa5f81148a0 0x7fa5f8111820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47612/0 (socket says 172.21.15.43:47612) 2024-03-28T22:43:51.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.862+0000 7fa5f6800640 1 -- 172.21.15.43:0/69739211 learned_addr learned my addr 172.21.15.43:0/69739211 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:51.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.863+0000 7fa5f6800640 1 -- 172.21.15.43:0/69739211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5f8108930 msgr2=0x7fa5f81141f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:51.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.863+0000 7fa5f5e00640 1 --2- 172.21.15.43:0/69739211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5f8108930 0x7fa5f81141f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.863+0000 7fa5f6800640 1 --2- 172.21.15.43:0/69739211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5f8108930 0x7fa5f81141f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.863+0000 7fa5f6800640 1 -- 172.21.15.43:0/69739211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5f8109250 msgr2=0x7fa5f81112e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:51.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.863+0000 7fa5f6800640 1 --2- 172.21.15.43:0/69739211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5f8109250 0x7fa5f81112e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.864+0000 7fa5f6800640 1 -- 172.21.15.43:0/69739211 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5f81b3210 con 0x7fa5f81148a0 2024-03-28T22:43:51.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.864+0000 7fa5f5e00640 1 --2- 172.21.15.43:0/69739211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5f8108930 0x7fa5f81141f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:43:51.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.865+0000 7fa5f6800640 1 --2- 172.21.15.43:0/69739211 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa5f81148a0 0x7fa5f8111820 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fa5f000eb40 tx=0x7fa5f000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:51.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.866+0000 7fa5eea00640 1 -- 172.21.15.43:0/69739211 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f0004b20 con 0x7fa5f81148a0 2024-03-28T22:43:51.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.866+0000 7fa5eea00640 1 -- 172.21.15.43:0/69739211 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5f0004cc0 con 0x7fa5f81148a0 2024-03-28T22:43:51.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.866+0000 7fa5eea00640 1 -- 172.21.15.43:0/69739211 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f00187f0 con 0x7fa5f81148a0 2024-03-28T22:43:51.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.866+0000 7fa5fd000640 1 -- 172.21.15.43:0/69739211 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5f81b33e0 con 0x7fa5f81148a0 2024-03-28T22:43:51.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.866+0000 7fa5fd000640 1 -- 172.21.15.43:0/69739211 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5f81b37f0 con 0x7fa5f81148a0 2024-03-28T22:43:51.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.880+0000 7fa5ffc6a740 1 Processor -- start 2024-03-28T22:43:51.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.881+0000 7fa5ffc6a740 1 -- start start 2024-03-28T22:43:51.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.881+0000 7fa5ffc6a740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556934e87bc0 0x556934e86910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.881+0000 7fa5ffc6a740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556934e86ee0 0x556934e87360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.881+0000 7fa5ffc6a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556934e90250 0x556934e92640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.881+0000 7fa5eea00640 1 -- 172.21.15.43:0/69739211 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fa5f0005020 con 0x7fa5f81148a0 2024-03-28T22:43:51.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.882+0000 7fa5f5400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556934e87bc0 0x556934e86910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.882+0000 7fa5f5400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556934e87bc0 0x556934e86910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:33026/0 (socket says 172.21.15.43:33026) 2024-03-28T22:43:51.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.882+0000 7fa5f5400640 1 -- 172.21.15.43:0/1624566731 learned_addr learned my addr 172.21.15.43:0/1624566731 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:51.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.883+0000 7fa5f5e00640 1 --2- 172.21.15.43:0/1624566731 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556934e86ee0 0x556934e87360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.883+0000 7fa5ffc6a740 1 -- 172.21.15.43:0/1624566731 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x556934deb100 con 0x556934e86ee0 2024-03-28T22:43:51.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.883+0000 7fa5ffc6a740 1 -- 172.21.15.43:0/1624566731 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x556934d93ba0 con 0x556934e90250 2024-03-28T22:43:51.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.883+0000 7fa5ffc6a740 1 -- 172.21.15.43:0/1624566731 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x556934de71a0 con 0x556934e87bc0 2024-03-28T22:43:51.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.883+0000 7fa5f5400640 1 --2- 172.21.15.43:0/1624566731 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556934e90250 0x556934e92640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.884+0000 7fa5f5e00640 1 -- 172.21.15.43:0/1624566731 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556934e87bc0 msgr2=0x556934e86910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:51.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.884+0000 7fa5f5e00640 1 --2- 172.21.15.43:0/1624566731 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556934e87bc0 0x556934e86910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.884+0000 7fa5f5e00640 1 -- 172.21.15.43:0/1624566731 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556934e90250 msgr2=0x556934e92640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:51.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.884+0000 7fa5f5e00640 1 --2- 172.21.15.43:0/1624566731 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556934e90250 0x556934e92640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.884+0000 7fa5f5e00640 1 -- 172.21.15.43:0/1624566731 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556934dc41a0 con 0x556934e86ee0 2024-03-28T22:43:51.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.885+0000 7fa5f5e00640 1 --2- 172.21.15.43:0/1624566731 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556934e86ee0 0x556934e87360 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa5e8036040 tx=0x7fa5e8009470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=633fcc09d5db854b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:51.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.885+0000 7fa5eea00640 1 --2- 172.21.15.43:0/69739211 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa5dc0a1ff0 0x7fa5dc0a44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.885+0000 7fa5eea00640 1 -- 172.21.15.43:0/69739211 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fa5f00cec80 con 0x7fa5f81148a0 2024-03-28T22:43:51.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.885+0000 7fa5e3400640 1 -- 172.21.15.43:0/1624566731 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e8043020 con 0x556934e86ee0 2024-03-28T22:43:51.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.885+0000 7fa5e3400640 1 -- 172.21.15.43:0/1624566731 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5e8004040 con 0x556934e86ee0 2024-03-28T22:43:51.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.886+0000 7fa5e3400640 1 -- 172.21.15.43:0/1624566731 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e8005550 con 0x556934e86ee0 2024-03-28T22:43:51.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.886+0000 7fa5f5400640 1 --2- 172.21.15.43:0/69739211 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa5dc0a1ff0 0x7fa5dc0a44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.888+0000 7fa5ffc6a740 1 -- 172.21.15.43:0/1624566731 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556934e86ee0 msgr2=0x556934e87360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:51.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.888+0000 7fa5ffc6a740 1 --2- 172.21.15.43:0/1624566731 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556934e86ee0 0x556934e87360 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa5e8036040 tx=0x7fa5e8009470 comp rx=0 tx=0).stop 2024-03-28T22:43:51.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.888+0000 7fa5f5400640 1 --2- 172.21.15.43:0/69739211 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa5dc0a1ff0 0x7fa5dc0a44b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa5e400bfd0 tx=0x7fa5e4004730 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:51.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.889+0000 7fa5ffc6a740 1 -- 172.21.15.43:0/1624566731 shutdown_connections 2024-03-28T22:43:51.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.889+0000 7fa5ffc6a740 1 --2- 172.21.15.43:0/1624566731 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556934e90250 0x556934e92640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.889+0000 7fa5ffc6a740 1 --2- 172.21.15.43:0/1624566731 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556934e86ee0 0x556934e87360 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.889+0000 7fa5ffc6a740 1 --2- 172.21.15.43:0/1624566731 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556934e87bc0 0x556934e86910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.889+0000 7fa5ffc6a740 1 -- 172.21.15.43:0/1624566731 >> 172.21.15.43:0/1624566731 conn(0x556934e4da10 msgr2=0x556934e81930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:51.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.890+0000 7fa5ffc6a740 1 -- 172.21.15.43:0/1624566731 shutdown_connections 2024-03-28T22:43:51.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.890+0000 7fa5ffc6a740 1 -- 172.21.15.43:0/1624566731 wait complete. 2024-03-28T22:43:51.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.891+0000 7fa5ffc6a740 1 -- start start 2024-03-28T22:43:51.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.891+0000 7fa5ffc6a740 1 Processor -- start 2024-03-28T22:43:51.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.892+0000 7fa5ffc6a740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556934e86ee0 0x556934e8d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.892+0000 7fa5ffc6a740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556934e87bc0 0x556934e8d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.892+0000 7fa5ffc6a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556934e90250 0x556934e8dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.892+0000 7fa5ffc6a740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x556934e93060 con 0x556934e86ee0 2024-03-28T22:43:51.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.892+0000 7fa5ffc6a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x556934d89250 con 0x556934e90250 2024-03-28T22:43:51.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.892+0000 7fa5ffc6a740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x556934e931e0 con 0x556934e87bc0 2024-03-28T22:43:51.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.892+0000 7fa5ffc6a740 10 client.0 _ll_register_callbacks cb 0x556934afebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:51.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.892+0000 7fa5f5e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556934e86ee0 0x556934e8d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.893+0000 7fa5f5e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556934e86ee0 0x556934e8d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54186/0 (socket says 172.21.15.43:54186) 2024-03-28T22:43:51.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.893+0000 7fa5f5e00640 1 -- 172.21.15.43:0/3540199071 learned_addr learned my addr 172.21.15.43:0/3540199071 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:51.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.893+0000 7fa5f5400640 1 --2- 172.21.15.43:0/3540199071 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556934e87bc0 0x556934e8d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.893+0000 7fa5f6800640 1 --2- 172.21.15.43:0/3540199071 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556934e90250 0x556934e8dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.893+0000 7fa5f5e00640 1 -- 172.21.15.43:0/3540199071 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556934e87bc0 msgr2=0x556934e8d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:51.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.893+0000 7fa5f5e00640 1 --2- 172.21.15.43:0/3540199071 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556934e87bc0 0x556934e8d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.893+0000 7fa5f5e00640 1 -- 172.21.15.43:0/3540199071 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556934e90250 msgr2=0x556934e8dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:51.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.893+0000 7fa5f5e00640 1 --2- 172.21.15.43:0/3540199071 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556934e90250 0x556934e8dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.893+0000 7fa5f5e00640 1 -- 172.21.15.43:0/3540199071 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556934da5960 con 0x556934e86ee0 2024-03-28T22:43:51.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.893+0000 7fa5f6800640 1 --2- 172.21.15.43:0/3540199071 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556934e90250 0x556934e8dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:51.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.894+0000 7fa5f5e00640 1 --2- 172.21.15.43:0/3540199071 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556934e86ee0 0x556934e8d230 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa5e8000c00 tx=0x7fa5e8005500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c99554dcaf9abc80 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:51.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.894+0000 7fa5d7e00640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:43:51.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.895+0000 7fa5d7e00640 1 -- 172.21.15.43:0/3540199071 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e804f030 con 0x556934e86ee0 2024-03-28T22:43:51.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.895+0000 7fa5d7e00640 1 -- 172.21.15.43:0/3540199071 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fa5e802ca70 con 0x556934e86ee0 2024-03-28T22:43:51.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.895+0000 7fa5ffc6a740 1 -- 172.21.15.43:0/3540199071 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5e8004040 con 0x556934e86ee0 2024-03-28T22:43:51.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.895+0000 7fa5d7e00640 1 -- 172.21.15.43:0/3540199071 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa5e8043040 con 0x556934e86ee0 2024-03-28T22:43:51.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.895+0000 7fa5ffc6a740 10 client.17907 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:51.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.896+0000 7fa5ffc6a740 1 -- 172.21.15.43:0/3540199071 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556934bc97d0 con 0x556934e86ee0 2024-03-28T22:43:51.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.896+0000 7fa5d7e00640 1 -- 172.21.15.43:0/3540199071 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5e8004040 con 0x556934e86ee0 2024-03-28T22:43:51.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.897+0000 7fa5d7e00640 1 -- 172.21.15.43:0/3540199071 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa5e80035f0 con 0x556934e86ee0 2024-03-28T22:43:51.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.899+0000 7fa5ffc6a740 10 client.17907 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:51.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.899+0000 7fa5ffc6a740 10 client.17907 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:43:51.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.899+0000 7fa5ffc6a740 20 client.17907 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:51.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.899+0000 7fa5ffc6a740 1 --2- 172.21.15.43:0/3540199071 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x556934ea0510 0x556934ea2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.899+0000 7fa5ffc6a740 4 client.17907 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:43:51.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.899+0000 7fa5ffc6a740 1 -- 172.21.15.43:0/3540199071 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556934df8640 con 0x556934ea0510 2024-03-28T22:43:51.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.899+0000 7fa5f5400640 1 --2- 172.21.15.43:0/3540199071 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x556934ea0510 0x556934ea2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.901+0000 7fa5d7e00640 10 client.17907 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:51.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.902+0000 7fa5f5400640 1 --2- 172.21.15.43:0/3540199071 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x556934ea0510 0x556934ea2900 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7a0fbe3d570f365 server_cookie=d143971593dd13db in_seq=0 out_seq=0 2024-03-28T22:43:51.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.904+0000 7fa5d7e00640 1 -- 172.21.15.43:0/3540199071 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556934df8640 con 0x556934ea0510 2024-03-28T22:43:51.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.904+0000 7fa5d7e00640 10 client.17907 handle_command_reply: tid=0 2024-03-28T22:43:51.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.904+0000 7fa5ffc6a740 1 client.17907 shutdown 2024-03-28T22:43:51.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.904+0000 7fa5ffc6a740 1 -- 172.21.15.43:0/3540199071 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556934e86ee0 msgr2=0x556934e8d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:51.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.904+0000 7fa5ffc6a740 1 --2- 172.21.15.43:0/3540199071 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556934e86ee0 0x556934e8d230 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa5e8000c00 tx=0x7fa5e8005500 comp rx=0 tx=0).stop 2024-03-28T22:43:51.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.905+0000 7fa5ffc6a740 1 -- 172.21.15.43:0/3540199071 shutdown_connections 2024-03-28T22:43:51.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.905+0000 7fa5ffc6a740 1 --2- 172.21.15.43:0/3540199071 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x556934ea0510 0x556934ea2900 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.905+0000 7fa5ffc6a740 1 --2- 172.21.15.43:0/3540199071 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556934e90250 0x556934e8dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.905+0000 7fa5ffc6a740 1 --2- 172.21.15.43:0/3540199071 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556934e87bc0 0x556934e8d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.905+0000 7fa5ffc6a740 1 --2- 172.21.15.43:0/3540199071 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556934e86ee0 0x556934e8d230 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.905+0000 7fa5ffc6a740 1 -- 172.21.15.43:0/3540199071 >> 172.21.15.43:0/3540199071 conn(0x556934e4da10 msgr2=0x556934e83680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:51.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.905+0000 7fa5d7e00640 0 client.17907 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:51.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.905+0000 7fa5ffc6a740 1 -- 172.21.15.43:0/3540199071 shutdown_connections 2024-03-28T22:43:51.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.905+0000 7fa5ffc6a740 1 -- 172.21.15.43:0/3540199071 wait complete. 2024-03-28T22:43:51.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.906+0000 7fa5ffc6a740 20 client.17907 trim_cache size 0 max 16384 2024-03-28T22:43:51.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.938+0000 7fa5ffc6a740 1 Processor -- start 2024-03-28T22:43:51.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.938+0000 7fa5ffc6a740 1 -- start start 2024-03-28T22:43:51.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.938+0000 7fa5ffc6a740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556934e86ee0 0x556934e8f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.938+0000 7fa5ffc6a740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556934e87bc0 0x556934e8c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.938+0000 7fa5ffc6a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556934e90250 0x556934e9be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.938+0000 7fa5ffc6a740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f81bd9f0 con 0x556934e87bc0 2024-03-28T22:43:51.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.938+0000 7fa5ffc6a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f81bf530 con 0x556934e90250 2024-03-28T22:43:51.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.938+0000 7fa5ffc6a740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f81be0c0 con 0x556934e86ee0 2024-03-28T22:43:51.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.939+0000 7fa5f6800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556934e90250 0x556934e9be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.939+0000 7fa5f6800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556934e90250 0x556934e9be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47626/0 (socket says 172.21.15.43:47626) 2024-03-28T22:43:51.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.939+0000 7fa5f6800640 1 -- 172.21.15.43:0/785664989 learned_addr learned my addr 172.21.15.43:0/785664989 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:51.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.939+0000 7fa5f5e00640 1 --2- 172.21.15.43:0/785664989 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556934e86ee0 0x556934e8f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.939+0000 7fa5f6800640 1 -- 172.21.15.43:0/785664989 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556934e86ee0 msgr2=0x556934e8f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:51.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.939+0000 7fa5f6800640 1 --2- 172.21.15.43:0/785664989 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556934e86ee0 0x556934e8f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.939+0000 7fa5f6800640 1 -- 172.21.15.43:0/785664989 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556934e87bc0 msgr2=0x556934e8c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:51.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.939+0000 7fa5f6800640 1 --2- 172.21.15.43:0/785664989 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556934e87bc0 0x556934e8c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.939+0000 7fa5f6800640 1 -- 172.21.15.43:0/785664989 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556934aff3f0 con 0x556934e90250 2024-03-28T22:43:51.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.939+0000 7fa5f6800640 1 --2- 172.21.15.43:0/785664989 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556934e90250 0x556934e9be40 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0097bf0 tx=0x7fa5f0004f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba241ad8482f5835 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:51.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.940+0000 7fa5e2a00640 1 -- 172.21.15.43:0/785664989 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f009f3c0 con 0x556934e90250 2024-03-28T22:43:51.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.940+0000 7fa5e2a00640 1 -- 172.21.15.43:0/785664989 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5f00a3a10 con 0x556934e90250 2024-03-28T22:43:51.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.940+0000 7fa5e2a00640 1 -- 172.21.15.43:0/785664989 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f009a6c0 con 0x556934e90250 2024-03-28T22:43:51.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.940+0000 7fa5ffc6a740 1 -- 172.21.15.43:0/785664989 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556934e90250 msgr2=0x556934e9be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:51.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.940+0000 7fa5ffc6a740 1 --2- 172.21.15.43:0/785664989 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556934e90250 0x556934e9be40 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0097bf0 tx=0x7fa5f0004f50 comp rx=0 tx=0).stop 2024-03-28T22:43:51.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.940+0000 7fa5ffc6a740 1 -- 172.21.15.43:0/785664989 shutdown_connections 2024-03-28T22:43:51.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.940+0000 7fa5ffc6a740 1 --2- 172.21.15.43:0/785664989 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556934e90250 0x556934e9be40 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.940+0000 7fa5ffc6a740 1 --2- 172.21.15.43:0/785664989 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556934e87bc0 0x556934e8c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.940+0000 7fa5ffc6a740 1 --2- 172.21.15.43:0/785664989 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556934e86ee0 0x556934e8f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.940+0000 7fa5ffc6a740 1 -- 172.21.15.43:0/785664989 >> 172.21.15.43:0/785664989 conn(0x556934e4da10 msgr2=0x556934e80220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:51.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.941+0000 7fa5ffc6a740 1 -- 172.21.15.43:0/785664989 shutdown_connections 2024-03-28T22:43:51.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.941+0000 7fa5ffc6a740 1 -- 172.21.15.43:0/785664989 wait complete. 2024-03-28T22:43:51.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.941+0000 7fa5ffc6a740 1 -- start start 2024-03-28T22:43:51.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.941+0000 7fa5ffc6a740 1 Processor -- start 2024-03-28T22:43:51.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.942+0000 7fa5ffc6a740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556934e86ee0 0x556934e98680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.942+0000 7fa5ffc6a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556934e87bc0 0x556934e8a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.942+0000 7fa5ffc6a740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556934e90250 0x556934e8a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.942+0000 7fa5ffc6a740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f81bee60 con 0x556934e86ee0 2024-03-28T22:43:51.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.942+0000 7fa5ffc6a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f81be790 con 0x556934e87bc0 2024-03-28T22:43:51.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.942+0000 7fa5ffc6a740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f81bb990 con 0x556934e90250 2024-03-28T22:43:51.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.942+0000 7fa5ffc6a740 10 client.0 _ll_register_callbacks cb 0x7fa5e400be60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:51.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.942+0000 7fa5f5400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556934e87bc0 0x556934e8a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.942+0000 7fa5f5e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556934e86ee0 0x556934e98680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.942+0000 7fa5f5400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556934e87bc0 0x556934e8a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47632/0 (socket says 172.21.15.43:47632) 2024-03-28T22:43:51.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.942+0000 7fa5f6800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556934e90250 0x556934e8a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.942+0000 7fa5f5400640 1 -- 172.21.15.43:0/3273600238 learned_addr learned my addr 172.21.15.43:0/3273600238 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:51.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.942+0000 7fa5f5400640 1 -- 172.21.15.43:0/3273600238 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556934e90250 msgr2=0x556934e8a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:51.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.942+0000 7fa5f5400640 1 --2- 172.21.15.43:0/3273600238 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556934e90250 0x556934e8a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.942+0000 7fa5f5400640 1 -- 172.21.15.43:0/3273600238 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556934e86ee0 msgr2=0x556934e98680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:51.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.942+0000 7fa5f5400640 1 --2- 172.21.15.43:0/3273600238 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556934e86ee0 0x556934e98680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.942+0000 7fa5f5400640 1 -- 172.21.15.43:0/3273600238 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556934abf0f0 con 0x556934e87bc0 2024-03-28T22:43:51.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.942+0000 7fa5f6800640 1 --2- 172.21.15.43:0/3273600238 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556934e90250 0x556934e8a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:51.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.943+0000 7fa5f5400640 1 --2- 172.21.15.43:0/3273600238 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556934e87bc0 0x556934e8a4b0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x556934ea2e70 tx=0x7fa5e400bbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a6921058e4a0ae4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:51.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.943+0000 7fa5d7e00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:43:51.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.943+0000 7fa5d7e00640 1 -- 172.21.15.43:0/3273600238 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e402c740 con 0x556934e87bc0 2024-03-28T22:43:51.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.943+0000 7fa5ffc6a740 1 -- 172.21.15.43:0/3273600238 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5f00a3a10 con 0x556934e87bc0 2024-03-28T22:43:51.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.943+0000 7fa5d7e00640 1 -- 172.21.15.43:0/3273600238 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7fa5e40125e0 con 0x556934e87bc0 2024-03-28T22:43:51.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.943+0000 7fa5d7e00640 1 -- 172.21.15.43:0/3273600238 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa5e403b050 con 0x556934e87bc0 2024-03-28T22:43:51.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.943+0000 7fa5ffc6a740 10 client.28160 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:51.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.943+0000 7fa5ffc6a740 1 -- 172.21.15.43:0/3273600238 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556934e91a60 con 0x556934e87bc0 2024-03-28T22:43:51.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.944+0000 7fa5d7e00640 1 -- 172.21.15.43:0/3273600238 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5f00a3a10 con 0x556934e87bc0 2024-03-28T22:43:51.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.944+0000 7fa5d7e00640 1 -- 172.21.15.43:0/3273600238 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa5e401d070 con 0x556934e87bc0 2024-03-28T22:43:51.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.944+0000 7fa5ffc6a740 10 client.28160 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:51.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.944+0000 7fa5ffc6a740 10 client.28160 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:43:51.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.944+0000 7fa5ffc6a740 20 client.28160 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:51.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.944+0000 7fa5ffc6a740 1 --2- 172.21.15.43:0/3273600238 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x556934f06ae0 0x556934f08f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:51.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.944+0000 7fa5ffc6a740 4 client.28160 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:43:51.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.944+0000 7fa5ffc6a740 1 -- 172.21.15.43:0/3273600238 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556934df2990 con 0x556934f06ae0 2024-03-28T22:43:51.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.944+0000 7fa5f5e00640 1 --2- 172.21.15.43:0/3273600238 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x556934f06ae0 0x556934f08f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:51.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.945+0000 7fa5f5e00640 1 --2- 172.21.15.43:0/3273600238 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x556934f06ae0 0x556934f08f00 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd400a5739d1373b server_cookie=9de15dfe7640d48e in_seq=0 out_seq=0 2024-03-28T22:43:51.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.945+0000 7fa5d7e00640 10 client.28160 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:51.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.946+0000 7fa5d7e00640 1 -- 172.21.15.43:0/3273600238 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x556934df2990 con 0x556934f06ae0 2024-03-28T22:43:51.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.946+0000 7fa5d7e00640 10 client.28160 handle_command_reply: tid=0 2024-03-28T22:43:51.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.946+0000 7fa5ffc6a740 1 client.28160 shutdown 2024-03-28T22:43:51.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.946+0000 7fa5ffc6a740 1 -- 172.21.15.43:0/3273600238 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556934e87bc0 msgr2=0x556934e8a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:51.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.946+0000 7fa5ffc6a740 1 --2- 172.21.15.43:0/3273600238 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556934e87bc0 0x556934e8a4b0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x556934ea2e70 tx=0x7fa5e400bbb0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.946+0000 7fa5ffc6a740 1 -- 172.21.15.43:0/3273600238 shutdown_connections 2024-03-28T22:43:51.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.946+0000 7fa5ffc6a740 1 --2- 172.21.15.43:0/3273600238 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x556934f06ae0 0x556934f08f00 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.946+0000 7fa5ffc6a740 1 --2- 172.21.15.43:0/3273600238 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556934e87bc0 0x556934e8a4b0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.946+0000 7fa5ffc6a740 1 --2- 172.21.15.43:0/3273600238 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x556934e90250 0x556934e8a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.946+0000 7fa5ffc6a740 1 --2- 172.21.15.43:0/3273600238 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x556934e86ee0 0x556934e98680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.946+0000 7fa5ffc6a740 1 -- 172.21.15.43:0/3273600238 >> 172.21.15.43:0/3273600238 conn(0x556934e4da10 msgr2=0x556934e80ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:51.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.946+0000 7fa5d7e00640 0 client.28160 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:51.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.946+0000 7fa5ffc6a740 1 -- 172.21.15.43:0/3273600238 shutdown_connections 2024-03-28T22:43:51.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.946+0000 7fa5ffc6a740 1 -- 172.21.15.43:0/3273600238 wait complete. 2024-03-28T22:43:51.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.946+0000 7fa5ffc6a740 20 client.28160 trim_cache size 0 max 16384 2024-03-28T22:43:51.949 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:43:51.949 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:43:51.949 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-03-28T22:43:51.949 INFO:teuthology.orchestra.run.smithi043.stdout: "cff0e42b-1491-43ad-9800-30926e70d782": { 2024-03-28T22:43:51.949 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-03-28T22:43:51.949 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "cff0e42b-1491-43ad-9800-30926e70d782", 2024-03-28T22:43:51.949 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-03-28T22:43:51.949 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:43:51.949 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:43:51.949 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:43:51.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.947+0000 7fa5fd000640 1 -- 172.21.15.43:0/69739211 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa5dc0a1ff0 msgr2=0x7fa5dc0a44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:51.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.947+0000 7fa5fd000640 1 --2- 172.21.15.43:0/69739211 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa5dc0a1ff0 0x7fa5dc0a44b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa5e400bfd0 tx=0x7fa5e4004730 comp rx=0 tx=0).stop 2024-03-28T22:43:51.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.947+0000 7fa5fd000640 1 -- 172.21.15.43:0/69739211 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa5f81148a0 msgr2=0x7fa5f8111820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:51.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.947+0000 7fa5fd000640 1 --2- 172.21.15.43:0/69739211 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa5f81148a0 0x7fa5f8111820 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fa5f000eb40 tx=0x7fa5f000c2d0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.947+0000 7fa5fd000640 1 -- 172.21.15.43:0/69739211 shutdown_connections 2024-03-28T22:43:51.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.947+0000 7fa5fd000640 1 --2- 172.21.15.43:0/69739211 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa5dc0a1ff0 0x7fa5dc0a44b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.947+0000 7fa5fd000640 1 --2- 172.21.15.43:0/69739211 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa5f81148a0 0x7fa5f8111820 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.947+0000 7fa5fd000640 1 --2- 172.21.15.43:0/69739211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa5f8109250 0x7fa5f81112e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.947+0000 7fa5fd000640 1 --2- 172.21.15.43:0/69739211 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5f8108930 0x7fa5f81141f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:51.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.947+0000 7fa5fd000640 1 -- 172.21.15.43:0/69739211 >> 172.21.15.43:0/69739211 conn(0x7fa5f806c830 msgr2=0x7fa5f8070210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:51.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.947+0000 7fa5fd000640 1 -- 172.21.15.43:0/69739211 shutdown_connections 2024-03-28T22:43:51.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:43:51.947+0000 7fa5fd000640 1 -- 172.21.15.43:0/69739211 wait complete. 2024-03-28T22:43:51.963 INFO:tasks.cephfs.filesystem:scrub status for tag:cff0e42b-1491-43ad-9800-30926e70d782 - {'path': '/', 'tag': 'cff0e42b-1491-43ad-9800-30926e70d782', 'options': 'recursive,force'} 2024-03-28T22:43:52.378 INFO:tasks.workunit.client.0.smithi043.stdout: 1 459217 7.35 MB/sec execute 203 sec latency 136.258 ms 2024-03-28T22:43:53.378 INFO:tasks.workunit.client.0.smithi043.stdout: 1 460419 7.34 MB/sec execute 204 sec latency 25.629 ms 2024-03-28T22:43:53.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:53 smithi172 ceph-mon[36566]: pgmap v205: 129 pgs: 129 active+clean; 51 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 194 op/s 2024-03-28T22:43:53.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:53 smithi172 ceph-mon[36566]: scrub queued for path: / 2024-03-28T22:43:53.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:53 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:43:53.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:53 smithi172 ceph-mon[36566]: scrub summary: active paths [/] 2024-03-28T22:43:53.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:53 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:43:53.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:53 smithi082 ceph-mon[36471]: pgmap v205: 129 pgs: 129 active+clean; 51 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 194 op/s 2024-03-28T22:43:53.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:53 smithi082 ceph-mon[36471]: scrub queued for path: / 2024-03-28T22:43:53.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:53 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:43:53.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:53 smithi082 ceph-mon[36471]: scrub summary: active paths [/] 2024-03-28T22:43:53.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:53 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:43:53.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:53 smithi043 ceph-mon[31823]: pgmap v205: 129 pgs: 129 active+clean; 51 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 194 op/s 2024-03-28T22:43:53.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:53 smithi043 ceph-mon[31823]: scrub queued for path: / 2024-03-28T22:43:53.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:53 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:43:53.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:53 smithi043 ceph-mon[31823]: scrub summary: active paths [/] 2024-03-28T22:43:53.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:53 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:43:54.379 INFO:tasks.workunit.client.0.smithi043.stdout: 1 462124 7.37 MB/sec execute 205 sec latency 22.638 ms 2024-03-28T22:43:54.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:54 smithi172 ceph-mon[36566]: scrub summary: idle 2024-03-28T22:43:54.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:54 smithi082 ceph-mon[36471]: scrub summary: idle 2024-03-28T22:43:54.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:54 smithi043 ceph-mon[31823]: scrub summary: idle 2024-03-28T22:43:55.245 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.246+0000 7f7f05c00640 1 -- 172.21.15.43:0/2750663755 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7f00107710 msgr2=0x7f7f00107af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:55.245 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.246+0000 7f7f05c00640 1 --2- 172.21.15.43:0/2750663755 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7f00107710 0x7f7f00107af0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f7ef0009ca0 tx=0x7f7ef002c100 comp rx=0 tx=0).stop 2024-03-28T22:43:55.246 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.247+0000 7f7f05c00640 1 -- 172.21.15.43:0/2750663755 shutdown_connections 2024-03-28T22:43:55.246 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.247+0000 7f7f05c00640 1 --2- 172.21.15.43:0/2750663755 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f001020c0 0x7f7f0010e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.246 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.247+0000 7f7f05c00640 1 --2- 172.21.15.43:0/2750663755 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f00101700 0x7f7f00101b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.246 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.247+0000 7f7f05c00640 1 --2- 172.21.15.43:0/2750663755 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7f00107710 0x7f7f00107af0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.246 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.247+0000 7f7f05c00640 1 -- 172.21.15.43:0/2750663755 >> 172.21.15.43:0/2750663755 conn(0x7f7f000fd210 msgr2=0x7f7f000ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:55.246 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.247+0000 7f7f05c00640 1 -- 172.21.15.43:0/2750663755 shutdown_connections 2024-03-28T22:43:55.246 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.247+0000 7f7f05c00640 1 -- 172.21.15.43:0/2750663755 wait complete. 2024-03-28T22:43:55.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.248+0000 7f7f05c00640 1 Processor -- start 2024-03-28T22:43:55.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.248+0000 7f7f05c00640 1 -- start start 2024-03-28T22:43:55.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.248+0000 7f7f05c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f00101700 0x7f7f00073a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:55.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.248+0000 7f7f05c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7f001020c0 0x7f7f00073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:55.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.248+0000 7f7f05c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f00107710 0x7f7f000744d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:55.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.248+0000 7f7f05c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7f00111ae0 con 0x7f7f00107710 2024-03-28T22:43:55.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.248+0000 7f7f05c00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7f00111960 con 0x7f7f001020c0 2024-03-28T22:43:55.248 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.248+0000 7f7f05c00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f7f00111c60 con 0x7f7f00101700 2024-03-28T22:43:55.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.249+0000 7f7efea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f00101700 0x7f7f00073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:55.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.249+0000 7f7efe000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7f001020c0 0x7f7f00073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:55.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.249+0000 7f7efea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f00101700 0x7f7f00073a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57874/0 (socket says 172.21.15.43:57874) 2024-03-28T22:43:55.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.249+0000 7f7efe000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7f001020c0 0x7f7f00073f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:37430/0 (socket says 172.21.15.43:37430) 2024-03-28T22:43:55.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.249+0000 7f7efea00640 1 -- 172.21.15.43:0/3345216205 learned_addr learned my addr 172.21.15.43:0/3345216205 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:55.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.249+0000 7f7eff400640 1 --2- 172.21.15.43:0/3345216205 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f00107710 0x7f7f000744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:55.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.249+0000 7f7efea00640 1 -- 172.21.15.43:0/3345216205 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7f001020c0 msgr2=0x7f7f00073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:55.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.249+0000 7f7efea00640 1 --2- 172.21.15.43:0/3345216205 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7f001020c0 0x7f7f00073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.249+0000 7f7efea00640 1 -- 172.21.15.43:0/3345216205 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f00107710 msgr2=0x7f7f000744d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:55.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.249+0000 7f7efea00640 1 --2- 172.21.15.43:0/3345216205 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f00107710 0x7f7f000744d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.249+0000 7f7efea00640 1 -- 172.21.15.43:0/3345216205 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f001a6440 con 0x7f7f00101700 2024-03-28T22:43:55.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.249+0000 7f7eff400640 1 --2- 172.21.15.43:0/3345216205 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f00107710 0x7f7f000744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:43:55.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.249+0000 7f7efea00640 1 --2- 172.21.15.43:0/3345216205 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f00101700 0x7f7f00073a50 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f7ef0005e30 tx=0x7f7ef00346f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:55.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.250+0000 7f7ef7400640 1 -- 172.21.15.43:0/3345216205 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef0039050 con 0x7f7f00101700 2024-03-28T22:43:55.249 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.250+0000 7f7ef7400640 1 -- 172.21.15.43:0/3345216205 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ef0045070 con 0x7f7f00101700 2024-03-28T22:43:55.250 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.250+0000 7f7ef7400640 1 -- 172.21.15.43:0/3345216205 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef0004630 con 0x7f7f00101700 2024-03-28T22:43:55.250 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.250+0000 7f7f05c00640 1 -- 172.21.15.43:0/3345216205 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f001a66d0 con 0x7f7f00101700 2024-03-28T22:43:55.250 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.250+0000 7f7f05c00640 1 -- 172.21.15.43:0/3345216205 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f001a6c10 con 0x7f7f00101700 2024-03-28T22:43:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.253+0000 7f7f089bf740 1 Processor -- start 2024-03-28T22:43:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.253+0000 7f7f089bf740 1 -- start start 2024-03-28T22:43:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.253+0000 7f7ef7400640 1 -- 172.21.15.43:0/3345216205 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f7ef0003970 con 0x7f7f00101700 2024-03-28T22:43:55.253 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.254+0000 7f7f089bf740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1f07e33a0 0x55d1f07e3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:55.253 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.254+0000 7f7f089bf740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d1f07e3cc0 0x55d1f07ec550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:55.253 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.254+0000 7f7f089bf740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1f07eca90 0x55d1f07eee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:55.253 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.254+0000 7f7f089bf740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d1f0746640 con 0x55d1f07e33a0 2024-03-28T22:43:55.253 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.254+0000 7f7f089bf740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55d1f06ef810 con 0x55d1f07eca90 2024-03-28T22:43:55.253 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.254+0000 7f7f089bf740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55d1f07426e0 con 0x55d1f07e3cc0 2024-03-28T22:43:55.253 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.254+0000 7f7efea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d1f07e3cc0 0x55d1f07ec550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:55.253 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.254+0000 7f7efe000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1f07eca90 0x55d1f07eee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:55.253 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.254+0000 7f7efe000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1f07eca90 0x55d1f07eee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:37442/0 (socket says 172.21.15.43:37442) 2024-03-28T22:43:55.253 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.254+0000 7f7eff400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1f07e33a0 0x55d1f07e3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:55.254 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.254+0000 7f7efe000640 1 -- 172.21.15.43:0/2667641572 learned_addr learned my addr 172.21.15.43:0/2667641572 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:55.254 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.254+0000 7f7efe000640 1 -- 172.21.15.43:0/2667641572 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d1f07e3cc0 msgr2=0x55d1f07ec550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:55.254 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.255+0000 7f7efe000640 1 --2- 172.21.15.43:0/2667641572 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d1f07e3cc0 0x55d1f07ec550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.254 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.255+0000 7f7efe000640 1 -- 172.21.15.43:0/2667641572 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1f07e33a0 msgr2=0x55d1f07e3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:55.254 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.255+0000 7f7efe000640 1 --2- 172.21.15.43:0/2667641572 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1f07e33a0 0x55d1f07e3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.254 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.255+0000 7f7efe000640 1 -- 172.21.15.43:0/2667641572 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1f045b210 con 0x55d1f07eca90 2024-03-28T22:43:55.254 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.255+0000 7f7efea00640 1 --2- 172.21.15.43:0/2667641572 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d1f07e3cc0 0x55d1f07ec550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:55.258 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.255+0000 7f7efe000640 1 --2- 172.21.15.43:0/2667641572 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1f07eca90 0x55d1f07eee80 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8009c90 tx=0x7f7ee8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96deab730e1bce8a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:55.258 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.255+0000 7f7eefe00640 1 -- 172.21.15.43:0/2667641572 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee801e070 con 0x55d1f07eca90 2024-03-28T22:43:55.258 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.255+0000 7f7eefe00640 1 -- 172.21.15.43:0/2667641572 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ee8019b10 con 0x55d1f07eca90 2024-03-28T22:43:55.259 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.255+0000 7f7ef7400640 1 --2- 172.21.15.43:0/3345216205 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7ee40a2010 0x7f7ee40a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:55.259 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.255+0000 7f7eefe00640 1 -- 172.21.15.43:0/2667641572 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee800ea70 con 0x55d1f07eca90 2024-03-28T22:43:55.259 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.256+0000 7f7ef7400640 1 -- 172.21.15.43:0/3345216205 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f7ef00f0b40 con 0x7f7f00101700 2024-03-28T22:43:55.259 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.256+0000 7f7eff400640 1 --2- 172.21.15.43:0/3345216205 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7ee40a2010 0x7f7ee40a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:55.259 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.256+0000 7f7f089bf740 1 -- 172.21.15.43:0/2667641572 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1f07eca90 msgr2=0x55d1f07eee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:55.259 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.256+0000 7f7f089bf740 1 --2- 172.21.15.43:0/2667641572 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1f07eca90 0x55d1f07eee80 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8009c90 tx=0x7f7ee8011810 comp rx=0 tx=0).stop 2024-03-28T22:43:55.259 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.256+0000 7f7eff400640 1 --2- 172.21.15.43:0/3345216205 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7ee40a2010 0x7f7ee40a44d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8005fd0 tx=0x7f7ef800d4c0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:55.259 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.257+0000 7f7f089bf740 1 -- 172.21.15.43:0/2667641572 shutdown_connections 2024-03-28T22:43:55.259 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.257+0000 7f7f089bf740 1 --2- 172.21.15.43:0/2667641572 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1f07eca90 0x55d1f07eee80 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.259 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.257+0000 7f7f089bf740 1 --2- 172.21.15.43:0/2667641572 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d1f07e3cc0 0x55d1f07ec550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.259 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.257+0000 7f7f089bf740 1 --2- 172.21.15.43:0/2667641572 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1f07e33a0 0x55d1f07e3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.259 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.257+0000 7f7f089bf740 1 -- 172.21.15.43:0/2667641572 >> 172.21.15.43:0/2667641572 conn(0x55d1f07ddcc0 msgr2=0x55d1f07de0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:55.259 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.257+0000 7f7f089bf740 1 -- 172.21.15.43:0/2667641572 shutdown_connections 2024-03-28T22:43:55.259 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.257+0000 7f7f089bf740 1 -- 172.21.15.43:0/2667641572 wait complete. 2024-03-28T22:43:55.260 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.257+0000 7f7f089bf740 1 -- start start 2024-03-28T22:43:55.260 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.257+0000 7f7f089bf740 1 Processor -- start 2024-03-28T22:43:55.260 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.258+0000 7f7f089bf740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1f07e33a0 0x55d1f07e9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:55.260 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.258+0000 7f7f089bf740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d1f07e3cc0 0x55d1f07ea020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:55.260 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.258+0000 7f7f089bf740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1f07eca90 0x55d1f07ea560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:55.260 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.258+0000 7f7f089bf740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d1f07ef6c0 con 0x55d1f07eca90 2024-03-28T22:43:55.260 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.258+0000 7f7f089bf740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55d1f06e4d00 con 0x55d1f07e33a0 2024-03-28T22:43:55.260 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.258+0000 7f7f089bf740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55d1f07ef840 con 0x55d1f07e3cc0 2024-03-28T22:43:55.260 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.258+0000 7f7f089bf740 10 client.0 _ll_register_callbacks cb 0x55d1f06cf490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:55.260 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.258+0000 7f7efe000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d1f07e3cc0 0x55d1f07ea020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:55.260 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.258+0000 7f7efea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1f07e33a0 0x55d1f07e9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:55.260 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.258+0000 7f7efe000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d1f07e3cc0 0x55d1f07ea020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57884/0 (socket says 172.21.15.43:57884) 2024-03-28T22:43:55.260 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.258+0000 7f7efe000640 1 -- 172.21.15.43:0/2100868220 learned_addr learned my addr 172.21.15.43:0/2100868220 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:55.260 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.258+0000 7f7eff400640 1 --2- 172.21.15.43:0/2100868220 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1f07eca90 0x55d1f07ea560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.259+0000 7f7efea00640 1 -- 172.21.15.43:0/2100868220 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d1f07e3cc0 msgr2=0x55d1f07ea020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.259+0000 7f7efea00640 1 --2- 172.21.15.43:0/2100868220 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d1f07e3cc0 0x55d1f07ea020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.259+0000 7f7efea00640 1 -- 172.21.15.43:0/2100868220 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1f07eca90 msgr2=0x55d1f07ea560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.259+0000 7f7efea00640 1 --2- 172.21.15.43:0/2100868220 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1f07eca90 0x55d1f07ea560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.259+0000 7f7efea00640 1 -- 172.21.15.43:0/2100868220 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1f0527620 con 0x55d1f07e33a0 2024-03-28T22:43:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.259+0000 7f7efea00640 1 --2- 172.21.15.43:0/2100868220 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1f07e33a0 0x55d1f07e9ae0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f7ef00ee000 tx=0x7f7ef0034f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94c1e9423a6ddf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.259+0000 7f7eecc00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:43:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.259+0000 7f7eecc00640 1 -- 172.21.15.43:0/2100868220 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef00bd070 con 0x55d1f07e33a0 2024-03-28T22:43:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.259+0000 7f7eecc00640 1 -- 172.21.15.43:0/2100868220 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f7ef0003c60 con 0x55d1f07e33a0 2024-03-28T22:43:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.259+0000 7f7f089bf740 1 -- 172.21.15.43:0/2100868220 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ee8019b10 con 0x55d1f07e33a0 2024-03-28T22:43:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.259+0000 7f7eecc00640 1 -- 172.21.15.43:0/2100868220 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7ef00bcc00 con 0x55d1f07e33a0 2024-03-28T22:43:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.259+0000 7f7f089bf740 10 client.28172 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.259+0000 7f7f089bf740 1 -- 172.21.15.43:0/2100868220 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d1f07fb5b0 con 0x55d1f07e33a0 2024-03-28T22:43:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.260+0000 7f7eecc00640 1 -- 172.21.15.43:0/2100868220 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ee8019b10 con 0x55d1f07e33a0 2024-03-28T22:43:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.260+0000 7f7eecc00640 1 -- 172.21.15.43:0/2100868220 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7ef00d2020 con 0x55d1f07e33a0 2024-03-28T22:43:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.260+0000 7f7f089bf740 10 client.28172 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.260+0000 7f7f089bf740 10 client.28172 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:43:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.260+0000 7f7f089bf740 20 client.28172 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.260+0000 7f7f089bf740 1 --2- 172.21.15.43:0/2100868220 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d1f07fcab0 0x55d1f07fef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.260+0000 7f7f089bf740 4 client.28172 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:43:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.260+0000 7f7f089bf740 1 -- 172.21.15.43:0/2100868220 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d1f075a030 con 0x55d1f07fcab0 2024-03-28T22:43:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.260+0000 7f7efe000640 1 --2- 172.21.15.43:0/2100868220 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d1f07fcab0 0x55d1f07fef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.261+0000 7f7efe000640 1 --2- 172.21.15.43:0/2100868220 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d1f07fcab0 0x55d1f07fef00 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97c8a7608da92aa1 server_cookie=e124378d1ce8eed5 in_seq=0 out_seq=0 2024-03-28T22:43:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.261+0000 7f7eecc00640 10 client.28172 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.262+0000 7f7eecc00640 1 -- 172.21.15.43:0/2100868220 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d1f075a030 con 0x55d1f07fcab0 2024-03-28T22:43:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.262+0000 7f7eecc00640 10 client.28172 handle_command_reply: tid=0 2024-03-28T22:43:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.262+0000 7f7f089bf740 1 client.28172 shutdown 2024-03-28T22:43:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.262+0000 7f7f089bf740 1 -- 172.21.15.43:0/2100868220 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1f07e33a0 msgr2=0x55d1f07e9ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.262+0000 7f7f089bf740 1 --2- 172.21.15.43:0/2100868220 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1f07e33a0 0x55d1f07e9ae0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f7ef00ee000 tx=0x7f7ef0034f10 comp rx=0 tx=0).stop 2024-03-28T22:43:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.263+0000 7f7f089bf740 1 -- 172.21.15.43:0/2100868220 shutdown_connections 2024-03-28T22:43:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.263+0000 7f7f089bf740 1 --2- 172.21.15.43:0/2100868220 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1f07eca90 0x55d1f07ea560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.263+0000 7f7f089bf740 1 --2- 172.21.15.43:0/2100868220 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d1f07e3cc0 0x55d1f07ea020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.263+0000 7f7f089bf740 1 --2- 172.21.15.43:0/2100868220 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d1f07fcab0 0x55d1f07fef00 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.263+0000 7f7eecc00640 0 client.28172 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.263+0000 7f7f089bf740 1 --2- 172.21.15.43:0/2100868220 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1f07e33a0 0x55d1f07e9ae0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.263+0000 7f7f089bf740 1 -- 172.21.15.43:0/2100868220 >> 172.21.15.43:0/2100868220 conn(0x55d1f07ddcc0 msgr2=0x55d1f07dfa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.263+0000 7f7f089bf740 1 -- 172.21.15.43:0/2100868220 shutdown_connections 2024-03-28T22:43:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.263+0000 7f7f089bf740 1 -- 172.21.15.43:0/2100868220 wait complete. 2024-03-28T22:43:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.263+0000 7f7f089bf740 20 client.28172 trim_cache size 0 max 16384 2024-03-28T22:43:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.283+0000 7f7f089bf740 1 Processor -- start 2024-03-28T22:43:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.283+0000 7f7f089bf740 1 -- start start 2024-03-28T22:43:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.283+0000 7f7f089bf740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1f07e33a0 0x55d1f07ec210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.283+0000 7f7f089bf740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1f07e3cc0 0x55d1f07f8460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.283+0000 7f7f089bf740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d1f07eca90 0x55d1f0804810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:55.282 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.283+0000 7f7f089bf740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7f001b0d20 con 0x55d1f07e33a0 2024-03-28T22:43:55.282 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.283+0000 7f7f089bf740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7f001b2860 con 0x55d1f07e3cc0 2024-03-28T22:43:55.282 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.283+0000 7f7f089bf740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f7f001b13f0 con 0x55d1f07eca90 2024-03-28T22:43:55.283 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.283+0000 7f7efe000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1f07e3cc0 0x55d1f07f8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:55.283 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.283+0000 7f7efea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1f07e33a0 0x55d1f07ec210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:55.283 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.283+0000 7f7eff400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d1f07eca90 0x55d1f0804810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:55.283 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.283+0000 7f7efe000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1f07e3cc0 0x55d1f07f8460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:37464/0 (socket says 172.21.15.43:37464) 2024-03-28T22:43:55.283 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.283+0000 7f7efe000640 1 -- 172.21.15.43:0/1516880936 learned_addr learned my addr 172.21.15.43:0/1516880936 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:55.283 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.284+0000 7f7efe000640 1 -- 172.21.15.43:0/1516880936 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d1f07eca90 msgr2=0x55d1f0804810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:55.283 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.284+0000 7f7efe000640 1 --2- 172.21.15.43:0/1516880936 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d1f07eca90 0x55d1f0804810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.283 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.284+0000 7f7efe000640 1 -- 172.21.15.43:0/1516880936 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1f07e33a0 msgr2=0x55d1f07ec210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.284+0000 7f7efe000640 1 --2- 172.21.15.43:0/1516880936 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1f07e33a0 0x55d1f07ec210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.284+0000 7f7efe000640 1 -- 172.21.15.43:0/1516880936 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1f07eee80 con 0x55d1f07e3cc0 2024-03-28T22:43:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.284+0000 7f7efe000640 1 --2- 172.21.15.43:0/1516880936 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1f07e3cc0 0x55d1f07f8460 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8009230 tx=0x7f7ee800e5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0e1f38d40f598a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.284+0000 7f7eef400640 1 -- 172.21.15.43:0/1516880936 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee8031070 con 0x55d1f07e3cc0 2024-03-28T22:43:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.284+0000 7f7eef400640 1 -- 172.21.15.43:0/1516880936 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ee802a070 con 0x55d1f07e3cc0 2024-03-28T22:43:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.284+0000 7f7eef400640 1 -- 172.21.15.43:0/1516880936 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee801e040 con 0x55d1f07e3cc0 2024-03-28T22:43:55.286 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.285+0000 7f7f089bf740 1 -- 172.21.15.43:0/1516880936 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1f07e3cc0 msgr2=0x55d1f07f8460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:55.286 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.285+0000 7f7f089bf740 1 --2- 172.21.15.43:0/1516880936 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1f07e3cc0 0x55d1f07f8460 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8009230 tx=0x7f7ee800e5d0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.286 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.285+0000 7f7f089bf740 1 -- 172.21.15.43:0/1516880936 shutdown_connections 2024-03-28T22:43:55.286 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.285+0000 7f7f089bf740 1 --2- 172.21.15.43:0/1516880936 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1f07e3cc0 0x55d1f07f8460 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.286 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.285+0000 7f7f089bf740 1 --2- 172.21.15.43:0/1516880936 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d1f07eca90 0x55d1f0804810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.286 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.285+0000 7f7f089bf740 1 --2- 172.21.15.43:0/1516880936 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1f07e33a0 0x55d1f07ec210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.286 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.285+0000 7f7f089bf740 1 -- 172.21.15.43:0/1516880936 >> 172.21.15.43:0/1516880936 conn(0x55d1f07ddcc0 msgr2=0x55d1f07e4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:55.286 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.285+0000 7f7f089bf740 1 -- 172.21.15.43:0/1516880936 shutdown_connections 2024-03-28T22:43:55.286 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.285+0000 7f7f089bf740 1 -- 172.21.15.43:0/1516880936 wait complete. 2024-03-28T22:43:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.286+0000 7f7f089bf740 1 -- start start 2024-03-28T22:43:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.286+0000 7f7f089bf740 1 Processor -- start 2024-03-28T22:43:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.287+0000 7f7f089bf740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1f07e33a0 0x55d1f07f0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.287+0000 7f7f089bf740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d1f07e3cc0 0x55d1f07f0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.287+0000 7f7f089bf740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1f07eca90 0x55d1f07f0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.287+0000 7f7f089bf740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7f001b2190 con 0x55d1f07e33a0 2024-03-28T22:43:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.287+0000 7f7f089bf740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7f001b1ac0 con 0x55d1f07eca90 2024-03-28T22:43:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.287+0000 7f7f089bf740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f7f001aed00 con 0x55d1f07e3cc0 2024-03-28T22:43:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.287+0000 7f7f089bf740 10 client.0 _ll_register_callbacks cb 0x7f7ef0005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:43:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.287+0000 7f7efe000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d1f07e3cc0 0x55d1f07f0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.287+0000 7f7eff400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1f07eca90 0x55d1f07f0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.287+0000 7f7efe000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d1f07e3cc0 0x55d1f07f0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:57910/0 (socket says 172.21.15.43:57910) 2024-03-28T22:43:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.287+0000 7f7efe000640 1 -- 172.21.15.43:0/3348575640 learned_addr learned my addr 172.21.15.43:0/3348575640 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:43:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.287+0000 7f7efea00640 1 --2- 172.21.15.43:0/3348575640 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1f07e33a0 0x55d1f07f0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:55.287 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.287+0000 7f7efe000640 1 -- 172.21.15.43:0/3348575640 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1f07eca90 msgr2=0x55d1f07f0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:55.288 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.287+0000 7f7efe000640 1 --2- 172.21.15.43:0/3348575640 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1f07eca90 0x55d1f07f0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.288 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.287+0000 7f7efe000640 1 -- 172.21.15.43:0/3348575640 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1f07e33a0 msgr2=0x55d1f07f0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:43:55.288 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.287+0000 7f7efe000640 1 --2- 172.21.15.43:0/3348575640 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1f07e33a0 0x55d1f07f0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.288 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.287+0000 7f7efe000640 1 -- 172.21.15.43:0/3348575640 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1f0861ef0 con 0x55d1f07e3cc0 2024-03-28T22:43:55.288 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.287+0000 7f7eff400640 1 --2- 172.21.15.43:0/3348575640 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1f07eca90 0x55d1f07f0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:43:55.288 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.288+0000 7f7efe000640 1 --2- 172.21.15.43:0/3348575640 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d1f07e3cc0 0x55d1f07f0980 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8011850 tx=0x7f7ee80096d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1821cc704563046d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:43:55.288 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.288+0000 7f7eecc00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:43:55.289 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.288+0000 7f7eecc00640 1 -- 172.21.15.43:0/3348575640 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee8031030 con 0x55d1f07e3cc0 2024-03-28T22:43:55.290 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.288+0000 7f7eecc00640 1 -- 172.21.15.43:0/3348575640 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9451+0+0 (secure 0 0 0) 0x7f7ee8019860 con 0x55d1f07e3cc0 2024-03-28T22:43:55.290 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.288+0000 7f7f089bf740 1 -- 172.21.15.43:0/3348575640 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ee802a070 con 0x55d1f07e3cc0 2024-03-28T22:43:55.290 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.288+0000 7f7eecc00640 1 -- 172.21.15.43:0/3348575640 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7ee801e050 con 0x55d1f07e3cc0 2024-03-28T22:43:55.290 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.288+0000 7f7f089bf740 10 client.28117 fetch_fsmap learned FSMap version 32 2024-03-28T22:43:55.290 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.288+0000 7f7f089bf740 1 -- 172.21.15.43:0/3348575640 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d1f08631e0 con 0x55d1f07e3cc0 2024-03-28T22:43:55.290 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.289+0000 7f7eecc00640 1 -- 172.21.15.43:0/3348575640 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ee802a070 con 0x55d1f07e3cc0 2024-03-28T22:43:55.290 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.289+0000 7f7eecc00640 1 -- 172.21.15.43:0/3348575640 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7ee802b5e0 con 0x55d1f07e3cc0 2024-03-28T22:43:55.290 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.289+0000 7f7f089bf740 10 client.28117 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:43:55.290 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.289+0000 7f7f089bf740 10 client.28117 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T22:43:55.290 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.289+0000 7f7f089bf740 20 client.28117 populate_metadata read hostname 'smithi043' 2024-03-28T22:43:55.290 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.289+0000 7f7f089bf740 1 --2- 172.21.15.43:0/3348575640 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d1f07fcab0 0x55d1f0865600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:43:55.290 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.289+0000 7f7f089bf740 4 client.28117 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T22:43:55.290 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.289+0000 7f7f089bf740 1 -- 172.21.15.43:0/3348575640 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d1f0753b80 con 0x55d1f07fcab0 2024-03-28T22:43:55.290 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.289+0000 7f7efea00640 1 --2- 172.21.15.43:0/3348575640 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d1f07fcab0 0x55d1f0865600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:43:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.290+0000 7f7eecc00640 10 client.28117 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:55.291 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.290+0000 7f7efea00640 1 --2- 172.21.15.43:0/3348575640 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d1f07fcab0 0x55d1f0865600 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=882e1c06f5ae0641 server_cookie=80e144a707419d04 in_seq=0 out_seq=0 2024-03-28T22:43:55.349 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:55 smithi082 ceph-mon[36471]: pgmap v206: 129 pgs: 129 active+clean; 59 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 257 op/s 2024-03-28T22:43:55.379 INFO:tasks.workunit.client.0.smithi043.stdout: 1 462883 7.34 MB/sec execute 206 sec latency 44.925 ms 2024-03-28T22:43:55.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:55 smithi172 ceph-mon[36566]: pgmap v206: 129 pgs: 129 active+clean; 59 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 257 op/s 2024-03-28T22:43:55.418 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:55 smithi043 ceph-mon[31823]: pgmap v206: 129 pgs: 129 active+clean; 59 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 257 op/s 2024-03-28T22:43:55.523 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.526+0000 7f7eecc00640 1 -- 172.21.15.43:0/3348575640 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d1f0753b80 con 0x55d1f07fcab0 2024-03-28T22:43:55.524 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.526+0000 7f7eecc00640 10 client.28117 handle_command_reply: tid=0 2024-03-28T22:43:55.524 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.526+0000 7f7f089bf740 1 client.28117 shutdown 2024-03-28T22:43:55.525 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.526+0000 7f7f089bf740 1 -- 172.21.15.43:0/3348575640 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d1f07e3cc0 msgr2=0x55d1f07f0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:55.525 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.526+0000 7f7f089bf740 1 --2- 172.21.15.43:0/3348575640 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d1f07e3cc0 0x55d1f07f0980 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8011850 tx=0x7f7ee80096d0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.525 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.526+0000 7f7f089bf740 1 -- 172.21.15.43:0/3348575640 shutdown_connections 2024-03-28T22:43:55.525 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.526+0000 7f7f089bf740 1 --2- 172.21.15.43:0/3348575640 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55d1f07fcab0 0x55d1f0865600 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.525 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.526+0000 7f7f089bf740 1 --2- 172.21.15.43:0/3348575640 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1f07eca90 0x55d1f07f0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.525 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.526+0000 7f7f089bf740 1 --2- 172.21.15.43:0/3348575640 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d1f07e3cc0 0x55d1f07f0980 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.525 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.526+0000 7f7f089bf740 1 --2- 172.21.15.43:0/3348575640 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d1f07e33a0 0x55d1f07f0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.525 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.526+0000 7f7eecc00640 0 client.28117 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:43:55.525 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.526+0000 7f7f089bf740 1 -- 172.21.15.43:0/3348575640 >> 172.21.15.43:0/3348575640 conn(0x55d1f07ddcc0 msgr2=0x55d1f07df000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:55.525 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.526+0000 7f7f089bf740 1 -- 172.21.15.43:0/3348575640 shutdown_connections 2024-03-28T22:43:55.525 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.526+0000 7f7f089bf740 1 -- 172.21.15.43:0/3348575640 wait complete. 2024-03-28T22:43:55.525 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.526+0000 7f7f089bf740 20 client.28117 trim_cache size 0 max 16384 2024-03-28T22:43:55.526 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:{ 2024-03-28T22:43:55.526 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "message": "", 2024-03-28T22:43:55.526 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout: "return_code": 0 2024-03-28T22:43:55.526 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stdout:} 2024-03-28T22:43:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.527+0000 7f7f05c00640 1 -- 172.21.15.43:0/3345216205 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7ee40a2010 msgr2=0x7f7ee40a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.527+0000 7f7f05c00640 1 --2- 172.21.15.43:0/3345216205 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7ee40a2010 0x7f7ee40a44d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8005fd0 tx=0x7f7ef800d4c0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.527+0000 7f7f05c00640 1 -- 172.21.15.43:0/3345216205 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f00101700 msgr2=0x7f7f00073a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:43:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.527+0000 7f7f05c00640 1 --2- 172.21.15.43:0/3345216205 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f00101700 0x7f7f00073a50 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f7ef0005e30 tx=0x7f7ef00346f0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.527+0000 7f7f05c00640 1 -- 172.21.15.43:0/3345216205 shutdown_connections 2024-03-28T22:43:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.527+0000 7f7f05c00640 1 --2- 172.21.15.43:0/3345216205 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7ee40a2010 0x7f7ee40a44d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.527+0000 7f7f05c00640 1 --2- 172.21.15.43:0/3345216205 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7f001020c0 0x7f7f00073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.527+0000 7f7f05c00640 1 --2- 172.21.15.43:0/3345216205 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f00107710 0x7f7f000744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.527+0000 7f7f05c00640 1 --2- 172.21.15.43:0/3345216205 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7f00101700 0x7f7f00073a50 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:43:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.528+0000 7f7f05c00640 1 -- 172.21.15.43:0/3345216205 >> 172.21.15.43:0/3345216205 conn(0x7f7f000fd210 msgr2=0x7f7f000fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:43:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.528+0000 7f7f05c00640 1 -- 172.21.15.43:0/3345216205 shutdown_connections 2024-03-28T22:43:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi043.front.sepia.ceph.com.smithi043.stderr:2024-03-28T22:43:55.528+0000 7f7f05c00640 1 -- 172.21.15.43:0/3345216205 wait complete. 2024-03-28T22:43:56.379 INFO:tasks.workunit.client.0.smithi043.stdout: 1 463994 7.33 MB/sec execute 207 sec latency 181.136 ms 2024-03-28T22:43:57.379 INFO:tasks.workunit.client.0.smithi043.stdout: 1 465667 7.35 MB/sec execute 208 sec latency 28.751 ms 2024-03-28T22:43:57.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:57 smithi172 ceph-mon[36566]: pgmap v207: 129 pgs: 129 active+clean; 59 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 120 op/s 2024-03-28T22:43:57.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:57 smithi082 ceph-mon[36471]: pgmap v207: 129 pgs: 129 active+clean; 59 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 120 op/s 2024-03-28T22:43:57.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:57 smithi043 ceph-mon[31823]: pgmap v207: 129 pgs: 129 active+clean; 59 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 120 op/s 2024-03-28T22:43:58.379 INFO:tasks.workunit.client.0.smithi043.stdout: 1 466432 7.32 MB/sec execute 209 sec latency 30.435 ms 2024-03-28T22:43:59.380 INFO:tasks.workunit.client.0.smithi043.stdout: 1 467786 7.32 MB/sec execute 210 sec latency 26.329 ms 2024-03-28T22:43:59.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:43:59 smithi172 ceph-mon[36566]: pgmap v208: 129 pgs: 129 active+clean; 49 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 291 op/s 2024-03-28T22:43:59.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:43:59 smithi082 ceph-mon[36471]: pgmap v208: 129 pgs: 129 active+clean; 49 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 291 op/s 2024-03-28T22:43:59.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:43:59 smithi043 ceph-mon[31823]: pgmap v208: 129 pgs: 129 active+clean; 49 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 291 op/s 2024-03-28T22:44:00.380 INFO:tasks.workunit.client.0.smithi043.stdout: 1 469278 7.34 MB/sec execute 211 sec latency 16.187 ms 2024-03-28T22:44:01.380 INFO:tasks.workunit.client.0.smithi043.stdout: 1 470007 7.31 MB/sec execute 212 sec latency 86.516 ms 2024-03-28T22:44:01.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:01 smithi172 ceph-mon[36566]: pgmap v209: 129 pgs: 129 active+clean; 49 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 234 op/s 2024-03-28T22:44:01.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:01 smithi082 ceph-mon[36471]: pgmap v209: 129 pgs: 129 active+clean; 49 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 234 op/s 2024-03-28T22:44:01.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:01 smithi043 ceph-mon[31823]: pgmap v209: 129 pgs: 129 active+clean; 49 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 234 op/s 2024-03-28T22:44:02.380 INFO:tasks.workunit.client.0.smithi043.stdout: 1 470225 7.28 MB/sec execute 213 sec latency 137.051 ms 2024-03-28T22:44:03.380 INFO:tasks.workunit.client.0.smithi043.stdout: 1 471392 7.27 MB/sec execute 214 sec latency 42.551 ms 2024-03-28T22:44:03.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:03 smithi172 ceph-mon[36566]: pgmap v210: 129 pgs: 129 active+clean; 49 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 234 op/s 2024-03-28T22:44:03.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:03 smithi082 ceph-mon[36471]: pgmap v210: 129 pgs: 129 active+clean; 49 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 234 op/s 2024-03-28T22:44:03.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:03 smithi043 ceph-mon[31823]: pgmap v210: 129 pgs: 129 active+clean; 49 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 234 op/s 2024-03-28T22:44:04.380 INFO:tasks.workunit.client.0.smithi043.stdout: 1 472751 7.29 MB/sec execute 215 sec latency 28.084 ms 2024-03-28T22:44:05.380 INFO:tasks.workunit.client.0.smithi043.stdout: 1 473022 7.26 MB/sec execute 216 sec latency 37.099 ms 2024-03-28T22:44:05.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:05 smithi172 ceph-mon[36566]: pgmap v211: 129 pgs: 129 active+clean; 55 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 340 op/s 2024-03-28T22:44:05.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:05 smithi172 ceph-mon[36566]: osdmap e100: 12 total, 12 up, 12 in 2024-03-28T22:44:05.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:05 smithi082 ceph-mon[36471]: pgmap v211: 129 pgs: 129 active+clean; 55 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 340 op/s 2024-03-28T22:44:05.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:05 smithi082 ceph-mon[36471]: osdmap e100: 12 total, 12 up, 12 in 2024-03-28T22:44:05.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:05 smithi043 ceph-mon[31823]: pgmap v211: 129 pgs: 129 active+clean; 55 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 340 op/s 2024-03-28T22:44:05.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:05 smithi043 ceph-mon[31823]: osdmap e100: 12 total, 12 up, 12 in 2024-03-28T22:44:06.172 INFO:tasks.workunit.client.0.smithi043.stdout:[473396] write failed on handle 13099 (Resource temporarily unavailable) 2024-03-28T22:44:06.172 INFO:tasks.workunit.client.0.smithi043.stdout:Child failed with status 1 2024-03-28T22:44:06.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T22:44:06.174 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-03-28T22:44:06.174 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-28T22:44:07.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:07 smithi172 ceph-mon[36566]: pgmap v213: 129 pgs: 129 active+clean; 55 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 725 KiB/s rd, 2.1 MiB/s wr, 332 op/s 2024-03-28T22:44:07.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:07 smithi082 ceph-mon[36471]: pgmap v213: 129 pgs: 129 active+clean; 55 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 725 KiB/s rd, 2.1 MiB/s wr, 332 op/s 2024-03-28T22:44:07.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:07 smithi043 ceph-mon[31823]: pgmap v213: 129 pgs: 129 active+clean; 55 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 725 KiB/s rd, 2.1 MiB/s wr, 332 op/s 2024-03-28T22:44:07.786 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_55f5138942aa4866e49329c9103468889cec3f73/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_55f5138942aa4866e49329c9103468889cec3f73/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_55f5138942aa4866e49329c9103468889cec3f73/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi043 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=55f5138942aa4866e49329c9103468889cec3f73 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-03-28T22:44:08.032 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c5dff688b9284c948c89be7afb4501b0 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_55f5138942aa4866e49329c9103468889cec3f73/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_55f5138942aa4866e49329c9103468889cec3f73/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_55f5138942aa4866e49329c9103468889cec3f73/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi043 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=55f5138942aa4866e49329c9103468889cec3f73 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-03-28T22:44:08.034 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-03-28T22:44:08.047 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi043.front.sepia.ceph.com 2024-03-28T22:44:08.070 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-03-28T22:44:08.121 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-03-28T22:44:08.121 INFO:teuthology.task.exec:Executing custom commands... 2024-03-28T22:44:08.121 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi043.front.sepia.ceph.com 2024-03-28T22:44:08.122 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-03-28T22:44:08.232 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-03-28T22:44:09.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:09 smithi172 ceph-mon[36566]: pgmap v214: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 57 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 2.1 MiB/s wr, 186 op/s 2024-03-28T22:44:09.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:09 smithi082 ceph-mon[36471]: pgmap v214: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 57 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 2.1 MiB/s wr, 186 op/s 2024-03-28T22:44:09.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:09 smithi043 ceph-mon[31823]: pgmap v214: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 57 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 2.1 MiB/s wr, 186 op/s 2024-03-28T22:44:11.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:11 smithi172 ceph-mon[36566]: pgmap v215: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 57 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 2.1 MiB/s wr, 186 op/s 2024-03-28T22:44:11.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:11 smithi082 ceph-mon[36471]: pgmap v215: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 57 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 2.1 MiB/s wr, 186 op/s 2024-03-28T22:44:11.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:11 smithi043 ceph-mon[31823]: pgmap v215: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 57 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 2.1 MiB/s wr, 186 op/s 2024-03-28T22:44:13.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:13 smithi172 ceph-mon[36566]: pgmap v216: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 57 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 2.1 MiB/s wr, 186 op/s 2024-03-28T22:44:13.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:13 smithi172 ceph-mon[36566]: osdmap e101: 12 total, 12 up, 12 in 2024-03-28T22:44:13.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:13 smithi082 ceph-mon[36471]: pgmap v216: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 57 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 2.1 MiB/s wr, 186 op/s 2024-03-28T22:44:13.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:13 smithi082 ceph-mon[36471]: osdmap e101: 12 total, 12 up, 12 in 2024-03-28T22:44:13.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:13 smithi043 ceph-mon[31823]: pgmap v216: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 57 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 2.1 MiB/s wr, 186 op/s 2024-03-28T22:44:13.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:13 smithi043 ceph-mon[31823]: osdmap e101: 12 total, 12 up, 12 in 2024-03-28T22:44:15.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:15 smithi172 ceph-mon[36566]: pgmap v218: 129 pgs: 129 active+clean; 55 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 1.5 MiB/s wr, 102 op/s 2024-03-28T22:44:15.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:15 smithi082 ceph-mon[36471]: pgmap v218: 129 pgs: 129 active+clean; 55 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 1.5 MiB/s wr, 102 op/s 2024-03-28T22:44:15.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:15 smithi043 ceph-mon[31823]: pgmap v218: 129 pgs: 129 active+clean; 55 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 1.5 MiB/s wr, 102 op/s 2024-03-28T22:44:17.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:17 smithi172 ceph-mon[36566]: pgmap v219: 129 pgs: 129 active+clean; 55 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 1.4 MiB/s wr, 96 op/s 2024-03-28T22:44:17.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:17 smithi082 ceph-mon[36471]: pgmap v219: 129 pgs: 129 active+clean; 55 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 1.4 MiB/s wr, 96 op/s 2024-03-28T22:44:17.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:17 smithi043 ceph-mon[31823]: pgmap v219: 129 pgs: 129 active+clean; 55 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 1.4 MiB/s wr, 96 op/s 2024-03-28T22:44:19.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:18 smithi172 ceph-mon[36566]: pgmap v220: 129 pgs: 129 active+clean; 50 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 623 KiB/s wr, 64 op/s 2024-03-28T22:44:19.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:18 smithi172 ceph-mon[36566]: osdmap e102: 12 total, 12 up, 12 in 2024-03-28T22:44:19.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:18 smithi082 ceph-mon[36471]: pgmap v220: 129 pgs: 129 active+clean; 50 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 623 KiB/s wr, 64 op/s 2024-03-28T22:44:19.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:18 smithi082 ceph-mon[36471]: osdmap e102: 12 total, 12 up, 12 in 2024-03-28T22:44:19.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:18 smithi043 ceph-mon[31823]: pgmap v220: 129 pgs: 129 active+clean; 50 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 623 KiB/s wr, 64 op/s 2024-03-28T22:44:19.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:18 smithi043 ceph-mon[31823]: osdmap e102: 12 total, 12 up, 12 in 2024-03-28T22:44:21.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:20 smithi172 ceph-mon[36566]: pgmap v222: 129 pgs: 129 active+clean; 50 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 779 KiB/s wr, 80 op/s 2024-03-28T22:44:21.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:20 smithi082 ceph-mon[36471]: pgmap v222: 129 pgs: 129 active+clean; 50 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 779 KiB/s wr, 80 op/s 2024-03-28T22:44:21.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:20 smithi043 ceph-mon[31823]: pgmap v222: 129 pgs: 129 active+clean; 50 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 779 KiB/s wr, 80 op/s 2024-03-28T22:44:21.964 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:44:21.964 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:44:22.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.206+0000 7f6b73e00640 1 -- 172.21.15.43:0/3888104477 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b741015c0 msgr2=0x7f6b74101a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:22.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.206+0000 7f6b73e00640 1 --2- 172.21.15.43:0/3888104477 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b741015c0 0x7f6b74101a40 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f6b60009900 tx=0x7f6b6002bc30 comp rx=0 tx=0).stop 2024-03-28T22:44:22.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.206+0000 7f6b73e00640 1 -- 172.21.15.43:0/3888104477 shutdown_connections 2024-03-28T22:44:22.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.206+0000 7f6b73e00640 1 --2- 172.21.15.43:0/3888104477 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b741015c0 0x7f6b74101a40 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.206+0000 7f6b73e00640 1 --2- 172.21.15.43:0/3888104477 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b74101f80 0x7f6b7410e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.206+0000 7f6b73e00640 1 --2- 172.21.15.43:0/3888104477 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b74107660 0x7f6b74107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.206+0000 7f6b73e00640 1 -- 172.21.15.43:0/3888104477 >> 172.21.15.43:0/3888104477 conn(0x7f6b740fd0f0 msgr2=0x7f6b740ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:22.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.206+0000 7f6b73e00640 1 -- 172.21.15.43:0/3888104477 shutdown_connections 2024-03-28T22:44:22.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.206+0000 7f6b73e00640 1 -- 172.21.15.43:0/3888104477 wait complete. 2024-03-28T22:44:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.207+0000 7f6b73e00640 1 Processor -- start 2024-03-28T22:44:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.207+0000 7f6b73e00640 1 -- start start 2024-03-28T22:44:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.208+0000 7f6b73e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b741015c0 0x7f6b7419a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.208+0000 7f6b73e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b74101f80 0x7f6b7419a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.208+0000 7f6b73e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b74107660 0x7f6b7419e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.208+0000 7f6b73e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6b741119a0 con 0x7f6b74101f80 2024-03-28T22:44:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.208+0000 7f6b73e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f6b74111820 con 0x7f6b74107660 2024-03-28T22:44:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.208+0000 7f6b73e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f6b74111b20 con 0x7f6b741015c0 2024-03-28T22:44:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.208+0000 7f6b72a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b741015c0 0x7f6b7419a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.208+0000 7f6b72a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b741015c0 0x7f6b7419a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:47818/0 (socket says 172.21.15.43:47818) 2024-03-28T22:44:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.208+0000 7f6b72a00640 1 -- 172.21.15.43:0/1203289890 learned_addr learned my addr 172.21.15.43:0/1203289890 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.208+0000 7f6b73400640 1 --2- 172.21.15.43:0/1203289890 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b74107660 0x7f6b7419e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.208+0000 7f6b72000640 1 --2- 172.21.15.43:0/1203289890 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b74101f80 0x7f6b7419a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.209+0000 7f6b72a00640 1 -- 172.21.15.43:0/1203289890 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b74107660 msgr2=0x7f6b7419e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.209+0000 7f6b72a00640 1 --2- 172.21.15.43:0/1203289890 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b74107660 0x7f6b7419e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.209+0000 7f6b72a00640 1 -- 172.21.15.43:0/1203289890 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b74101f80 msgr2=0x7f6b7419a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.209+0000 7f6b72a00640 1 --2- 172.21.15.43:0/1203289890 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b74101f80 0x7f6b7419a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.209+0000 7f6b72a00640 1 -- 172.21.15.43:0/1203289890 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b7419f060 con 0x7f6b741015c0 2024-03-28T22:44:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.209+0000 7f6b73400640 1 --2- 172.21.15.43:0/1203289890 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b74107660 0x7f6b7419e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:44:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.209+0000 7f6b72a00640 1 --2- 172.21.15.43:0/1203289890 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b741015c0 0x7f6b7419a070 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f6b6400e9c0 tx=0x7f6b6400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.209+0000 7f6b78a00640 1 -- 172.21.15.43:0/1203289890 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6400cd80 con 0x7f6b741015c0 2024-03-28T22:44:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.209+0000 7f6b73e00640 1 -- 172.21.15.43:0/1203289890 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b7419f350 con 0x7f6b741015c0 2024-03-28T22:44:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.209+0000 7f6b78a00640 1 -- 172.21.15.43:0/1203289890 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b640035f0 con 0x7f6b741015c0 2024-03-28T22:44:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.209+0000 7f6b78a00640 1 -- 172.21.15.43:0/1203289890 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b64010690 con 0x7f6b741015c0 2024-03-28T22:44:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.209+0000 7f6b73e00640 1 -- 172.21.15.43:0/1203289890 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b741a6be0 con 0x7f6b741015c0 2024-03-28T22:44:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.212+0000 7f6b7cb6b740 1 Processor -- start 2024-03-28T22:44:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.213+0000 7f6b78a00640 1 -- 172.21.15.43:0/1203289890 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f6b640037a0 con 0x7f6b741015c0 2024-03-28T22:44:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.213+0000 7f6b7cb6b740 1 -- start start 2024-03-28T22:44:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.213+0000 7f6b7cb6b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5580b1d42db0 0x5580b1d41b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.213+0000 7f6b7cb6b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580b1d420d0 0x5580b1d42550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.213+0000 7f6b7cb6b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5580b1d4b440 0x5580b1d4d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.213+0000 7f6b7cb6b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5580b1ca6100 con 0x5580b1d420d0 2024-03-28T22:44:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.213+0000 7f6b7cb6b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5580b1c4eba0 con 0x5580b1d42db0 2024-03-28T22:44:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.213+0000 7f6b7cb6b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5580b1ca21a0 con 0x5580b1d4b440 2024-03-28T22:44:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.213+0000 7f6b73400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5580b1d42db0 0x5580b1d41b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.213+0000 7f6b72000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5580b1d4b440 0x5580b1d4d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.213+0000 7f6b73400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5580b1d42db0 0x5580b1d41b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48418/0 (socket says 172.21.15.43:48418) 2024-03-28T22:44:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.213+0000 7f6b73400640 1 -- 172.21.15.43:0/2417940136 learned_addr learned my addr 172.21.15.43:0/2417940136 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.213+0000 7f6b72000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5580b1d4b440 0x5580b1d4d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:47824/0 (socket says 172.21.15.43:47824) 2024-03-28T22:44:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.214+0000 7f6b72a00640 1 --2- 172.21.15.43:0/2417940136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580b1d420d0 0x5580b1d42550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.214+0000 7f6b73400640 1 -- 172.21.15.43:0/2417940136 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5580b1d4b440 msgr2=0x5580b1d4d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.214+0000 7f6b73400640 1 --2- 172.21.15.43:0/2417940136 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5580b1d4b440 0x5580b1d4d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.214+0000 7f6b73400640 1 -- 172.21.15.43:0/2417940136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580b1d420d0 msgr2=0x5580b1d42550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.214+0000 7f6b73400640 1 --2- 172.21.15.43:0/2417940136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580b1d420d0 0x5580b1d42550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.214+0000 7f6b73400640 1 -- 172.21.15.43:0/2417940136 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580b1c7f1a0 con 0x5580b1d42db0 2024-03-28T22:44:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.214+0000 7f6b72a00640 1 --2- 172.21.15.43:0/2417940136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580b1d420d0 0x5580b1d42550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:44:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.214+0000 7f6b73400640 1 --2- 172.21.15.43:0/2417940136 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5580b1d42db0 0x5580b1d41b00 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c010750 tx=0x7f6b6c010c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11254110051051dd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.214+0000 7f6b78a00640 1 --2- 172.21.15.43:0/1203289890 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6b580a1ef0 0x7f6b580a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.214+0000 7f6b5fe00640 1 -- 172.21.15.43:0/2417940136 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c00ead0 con 0x5580b1d42db0 2024-03-28T22:44:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.214+0000 7f6b5fe00640 1 -- 172.21.15.43:0/2417940136 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b6c010e60 con 0x5580b1d42db0 2024-03-28T22:44:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.214+0000 7f6b5fe00640 1 -- 172.21.15.43:0/2417940136 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c011640 con 0x5580b1d42db0 2024-03-28T22:44:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.215+0000 7f6b72a00640 1 --2- 172.21.15.43:0/1203289890 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6b580a1ef0 0x7f6b580a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.215+0000 7f6b78a00640 1 -- 172.21.15.43:0/1203289890 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f6b640ce940 con 0x7f6b741015c0 2024-03-28T22:44:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.215+0000 7f6b7cb6b740 1 -- 172.21.15.43:0/2417940136 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5580b1d42db0 msgr2=0x5580b1d41b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.215+0000 7f6b7cb6b740 1 --2- 172.21.15.43:0/2417940136 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5580b1d42db0 0x5580b1d41b00 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c010750 tx=0x7f6b6c010c10 comp rx=0 tx=0).stop 2024-03-28T22:44:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.215+0000 7f6b72a00640 1 --2- 172.21.15.43:0/1203289890 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6b580a1ef0 0x7f6b580a43b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x5580b1d43300 tx=0x7f6b640045f0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.215+0000 7f6b7cb6b740 1 -- 172.21.15.43:0/2417940136 shutdown_connections 2024-03-28T22:44:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.215+0000 7f6b7cb6b740 1 --2- 172.21.15.43:0/2417940136 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5580b1d4b440 0x5580b1d4d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.215+0000 7f6b7cb6b740 1 --2- 172.21.15.43:0/2417940136 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580b1d420d0 0x5580b1d42550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.215+0000 7f6b7cb6b740 1 --2- 172.21.15.43:0/2417940136 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5580b1d42db0 0x5580b1d41b00 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.215+0000 7f6b7cb6b740 1 -- 172.21.15.43:0/2417940136 >> 172.21.15.43:0/2417940136 conn(0x5580b1d08a10 msgr2=0x5580b1d3cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.215+0000 7f6b7cb6b740 1 -- 172.21.15.43:0/2417940136 shutdown_connections 2024-03-28T22:44:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.216+0000 7f6b7cb6b740 1 -- 172.21.15.43:0/2417940136 wait complete. 2024-03-28T22:44:22.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.216+0000 7f6b7cb6b740 1 -- start start 2024-03-28T22:44:22.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.216+0000 7f6b7cb6b740 1 Processor -- start 2024-03-28T22:44:22.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.217+0000 7f6b7cb6b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5580b1d420d0 0x5580b1d48420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:22.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.217+0000 7f6b7cb6b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5580b1d42db0 0x5580b1d48960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:22.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.217+0000 7f6b7cb6b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580b1d4b440 0x5580b1d48ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:22.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.217+0000 7f6b7cb6b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5580b1d4e2b0 con 0x5580b1d4b440 2024-03-28T22:44:22.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.217+0000 7f6b7cb6b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5580b1c44250 con 0x5580b1d420d0 2024-03-28T22:44:22.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.217+0000 7f6b7cb6b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5580b1d4e430 con 0x5580b1d42db0 2024-03-28T22:44:22.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.217+0000 7f6b7cb6b740 10 client.0 _ll_register_callbacks cb 0x5580b19b9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:44:22.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.217+0000 7f6b72000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5580b1d42db0 0x5580b1d48960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:22.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.217+0000 7f6b72000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5580b1d42db0 0x5580b1d48960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:47826/0 (socket says 172.21.15.43:47826) 2024-03-28T22:44:22.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.217+0000 7f6b72000640 1 -- 172.21.15.43:0/2670968217 learned_addr learned my addr 172.21.15.43:0/2670968217 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:22.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.217+0000 7f6b73400640 1 --2- 172.21.15.43:0/2670968217 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5580b1d420d0 0x5580b1d48420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:22.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.217+0000 7f6b73400640 1 --2- 172.21.15.43:0/2670968217 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580b1d4b440 0x5580b1d48ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:22.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.217+0000 7f6b72000640 1 -- 172.21.15.43:0/2670968217 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5580b1d420d0 msgr2=0x5580b1d48420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:22.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.217+0000 7f6b72000640 1 --2- 172.21.15.43:0/2670968217 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5580b1d420d0 0x5580b1d48420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.217+0000 7f6b72000640 1 -- 172.21.15.43:0/2670968217 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580b1d4b440 msgr2=0x5580b1d48ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:22.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.217+0000 7f6b72000640 1 --2- 172.21.15.43:0/2670968217 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580b1d4b440 0x5580b1d48ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.217+0000 7f6b72000640 1 -- 172.21.15.43:0/2670968217 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580b1c60960 con 0x5580b1d42db0 2024-03-28T22:44:22.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.218+0000 7f6b72000640 1 --2- 172.21.15.43:0/2670968217 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5580b1d42db0 0x5580b1d48960 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f6b60035000 tx=0x7f6b6002c7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d3c07c4ab567ac6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:22.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.218+0000 7f6b5cc00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:44:22.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.218+0000 7f6b5cc00640 1 -- 172.21.15.43:0/2670968217 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6002ccc0 con 0x5580b1d42db0 2024-03-28T22:44:22.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.218+0000 7f6b7cb6b740 1 -- 172.21.15.43:0/2670968217 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b6c010e60 con 0x5580b1d42db0 2024-03-28T22:44:22.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.218+0000 7f6b5cc00640 1 -- 172.21.15.43:0/2670968217 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f6b600046c0 con 0x5580b1d42db0 2024-03-28T22:44:22.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.218+0000 7f6b5cc00640 1 -- 172.21.15.43:0/2670968217 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6b600048c0 con 0x5580b1d42db0 2024-03-28T22:44:22.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.218+0000 7f6b7cb6b740 10 client.28129 fetch_fsmap learned FSMap version 32 2024-03-28T22:44:22.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.219+0000 7f6b7cb6b740 1 -- 172.21.15.43:0/2670968217 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5580b1a847d0 con 0x5580b1d42db0 2024-03-28T22:44:22.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.219+0000 7f6b5cc00640 1 -- 172.21.15.43:0/2670968217 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b6c010e60 con 0x5580b1d42db0 2024-03-28T22:44:22.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.219+0000 7f6b5cc00640 1 -- 172.21.15.43:0/2670968217 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6b60046070 con 0x5580b1d42db0 2024-03-28T22:44:22.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.219+0000 7f6b7cb6b740 10 client.28129 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:44:22.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.219+0000 7f6b7cb6b740 10 client.28129 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:44:22.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.219+0000 7f6b7cb6b740 20 client.28129 populate_metadata read hostname 'smithi043' 2024-03-28T22:44:22.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.219+0000 7f6b7cb6b740 1 --2- 172.21.15.43:0/2670968217 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5580b1d5b8f0 0x5580b1d5dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:22.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.219+0000 7f6b7cb6b740 4 client.28129 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:44:22.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.219+0000 7f6b7cb6b740 1 -- 172.21.15.43:0/2670968217 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5580b1cb3640 con 0x5580b1d5b8f0 2024-03-28T22:44:22.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.219+0000 7f6b73400640 1 --2- 172.21.15.43:0/2670968217 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5580b1d5b8f0 0x5580b1d5dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:22.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.220+0000 7f6b73400640 1 --2- 172.21.15.43:0/2670968217 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5580b1d5b8f0 0x5580b1d5dd30 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d33eb785a80dc643 server_cookie=adc6fb8c0e4019e9 in_seq=0 out_seq=0 2024-03-28T22:44:22.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.220+0000 7f6b5cc00640 10 client.28129 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:44:22.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.221+0000 7f6b5cc00640 1 -- 172.21.15.43:0/2670968217 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5580b1cb3640 con 0x5580b1d5b8f0 2024-03-28T22:44:22.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.221+0000 7f6b5cc00640 10 client.28129 handle_command_reply: tid=0 2024-03-28T22:44:22.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.221+0000 7f6b7cb6b740 1 client.28129 shutdown 2024-03-28T22:44:22.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.221+0000 7f6b7cb6b740 1 -- 172.21.15.43:0/2670968217 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5580b1d42db0 msgr2=0x5580b1d48960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:22.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.221+0000 7f6b7cb6b740 1 --2- 172.21.15.43:0/2670968217 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5580b1d42db0 0x5580b1d48960 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f6b60035000 tx=0x7f6b6002c7e0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.221+0000 7f6b7cb6b740 1 -- 172.21.15.43:0/2670968217 shutdown_connections 2024-03-28T22:44:22.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.221+0000 7f6b7cb6b740 1 --2- 172.21.15.43:0/2670968217 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580b1d4b440 0x5580b1d48ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.221+0000 7f6b7cb6b740 1 --2- 172.21.15.43:0/2670968217 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5580b1d42db0 0x5580b1d48960 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.221+0000 7f6b7cb6b740 1 --2- 172.21.15.43:0/2670968217 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5580b1d5b8f0 0x5580b1d5dd30 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.221+0000 7f6b7cb6b740 1 --2- 172.21.15.43:0/2670968217 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5580b1d420d0 0x5580b1d48420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.221+0000 7f6b5cc00640 0 client.28129 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:44:22.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.221+0000 7f6b7cb6b740 1 -- 172.21.15.43:0/2670968217 >> 172.21.15.43:0/2670968217 conn(0x5580b1d08a10 msgr2=0x5580b1d3e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:22.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.221+0000 7f6b7cb6b740 1 -- 172.21.15.43:0/2670968217 shutdown_connections 2024-03-28T22:44:22.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.221+0000 7f6b7cb6b740 1 -- 172.21.15.43:0/2670968217 wait complete. 2024-03-28T22:44:22.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.222+0000 7f6b7cb6b740 20 client.28129 trim_cache size 0 max 16384 2024-03-28T22:44:22.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.241+0000 7f6b7cb6b740 1 Processor -- start 2024-03-28T22:44:22.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.241+0000 7f6b7cb6b740 1 -- start start 2024-03-28T22:44:22.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.241+0000 7f6b7cb6b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5580b1d420d0 0x5580b1d47380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:22.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.241+0000 7f6b7cb6b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5580b1d42db0 0x5580b1d3c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:22.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.241+0000 7f6b7cb6b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580b1d4b440 0x5580b1d570c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:22.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.241+0000 7f6b7cb6b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6b741b0bf0 con 0x5580b1d4b440 2024-03-28T22:44:22.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.241+0000 7f6b7cb6b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f6b741b2730 con 0x5580b1d42db0 2024-03-28T22:44:22.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.241+0000 7f6b7cb6b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f6b741b12c0 con 0x5580b1d420d0 2024-03-28T22:44:22.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.242+0000 7f6b72000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5580b1d42db0 0x5580b1d3c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:22.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.242+0000 7f6b72000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5580b1d42db0 0x5580b1d3c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48442/0 (socket says 172.21.15.43:48442) 2024-03-28T22:44:22.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.242+0000 7f6b72000640 1 -- 172.21.15.43:0/2387388842 learned_addr learned my addr 172.21.15.43:0/2387388842 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:22.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.242+0000 7f6b73400640 1 --2- 172.21.15.43:0/2387388842 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5580b1d420d0 0x5580b1d47380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:22.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.242+0000 7f6b73400640 1 --2- 172.21.15.43:0/2387388842 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580b1d4b440 0x5580b1d570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:22.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.242+0000 7f6b72000640 1 -- 172.21.15.43:0/2387388842 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5580b1d420d0 msgr2=0x5580b1d47380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:22.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.242+0000 7f6b72000640 1 --2- 172.21.15.43:0/2387388842 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5580b1d420d0 0x5580b1d47380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.242+0000 7f6b72000640 1 -- 172.21.15.43:0/2387388842 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580b1d4b440 msgr2=0x5580b1d570c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:22.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.242+0000 7f6b72000640 1 --2- 172.21.15.43:0/2387388842 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580b1d4b440 0x5580b1d570c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.242+0000 7f6b72000640 1 -- 172.21.15.43:0/2387388842 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580b19ba3f0 con 0x5580b1d42db0 2024-03-28T22:44:22.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.242+0000 7f6b73400640 1 --2- 172.21.15.43:0/2387388842 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580b1d4b440 0x5580b1d570c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:44:22.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.242+0000 7f6b72000640 1 --2- 172.21.15.43:0/2387388842 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5580b1d42db0 0x5580b1d3c650 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f6b60005c40 tx=0x7f6b60005dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59edf453c2d0ff20 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:22.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.243+0000 7f6b5f400640 1 -- 172.21.15.43:0/2387388842 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b60009520 con 0x5580b1d42db0 2024-03-28T22:44:22.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.243+0000 7f6b5f400640 1 -- 172.21.15.43:0/2387388842 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b60003e80 con 0x5580b1d42db0 2024-03-28T22:44:22.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.243+0000 7f6b5f400640 1 -- 172.21.15.43:0/2387388842 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b60034970 con 0x5580b1d42db0 2024-03-28T22:44:22.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.243+0000 7f6b7cb6b740 1 -- 172.21.15.43:0/2387388842 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5580b1d42db0 msgr2=0x5580b1d3c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:22.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.243+0000 7f6b7cb6b740 1 --2- 172.21.15.43:0/2387388842 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5580b1d42db0 0x5580b1d3c650 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f6b60005c40 tx=0x7f6b60005dd0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.243+0000 7f6b7cb6b740 1 -- 172.21.15.43:0/2387388842 shutdown_connections 2024-03-28T22:44:22.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.243+0000 7f6b7cb6b740 1 --2- 172.21.15.43:0/2387388842 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5580b1d42db0 0x5580b1d3c650 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.243+0000 7f6b7cb6b740 1 --2- 172.21.15.43:0/2387388842 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580b1d4b440 0x5580b1d570c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.243+0000 7f6b7cb6b740 1 --2- 172.21.15.43:0/2387388842 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5580b1d420d0 0x5580b1d47380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.243+0000 7f6b7cb6b740 1 -- 172.21.15.43:0/2387388842 >> 172.21.15.43:0/2387388842 conn(0x5580b1d08a10 msgr2=0x5580b1d3bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:22.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.243+0000 7f6b7cb6b740 1 -- 172.21.15.43:0/2387388842 shutdown_connections 2024-03-28T22:44:22.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.244+0000 7f6b7cb6b740 1 -- 172.21.15.43:0/2387388842 wait complete. 2024-03-28T22:44:22.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.244+0000 7f6b7cb6b740 1 -- start start 2024-03-28T22:44:22.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.244+0000 7f6b7cb6b740 1 Processor -- start 2024-03-28T22:44:22.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.245+0000 7f6b7cb6b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5580b1d420d0 0x5580b1d539e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:22.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.245+0000 7f6b7cb6b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580b1d42db0 0x5580b1d456c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:22.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.245+0000 7f6b7cb6b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5580b1d4b440 0x5580b1d45c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:22.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.245+0000 7f6b7cb6b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6b741b2060 con 0x5580b1d42db0 2024-03-28T22:44:22.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.245+0000 7f6b7cb6b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f6b741b1990 con 0x5580b1d420d0 2024-03-28T22:44:22.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.245+0000 7f6b7cb6b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f6b741aebf0 con 0x5580b1d4b440 2024-03-28T22:44:22.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.245+0000 7f6b7cb6b740 10 client.0 _ll_register_callbacks cb 0x7f6b60005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:44:22.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.245+0000 7f6b73400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5580b1d420d0 0x5580b1d539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:22.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.245+0000 7f6b73400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5580b1d420d0 0x5580b1d539e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48456/0 (socket says 172.21.15.43:48456) 2024-03-28T22:44:22.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.245+0000 7f6b73400640 1 -- 172.21.15.43:0/1879048234 learned_addr learned my addr 172.21.15.43:0/1879048234 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:22.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.245+0000 7f6b72000640 1 --2- 172.21.15.43:0/1879048234 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580b1d42db0 0x5580b1d456c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:22.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.245+0000 7f6b73400640 1 --2- 172.21.15.43:0/1879048234 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5580b1d4b440 0x5580b1d45c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:22.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.245+0000 7f6b73400640 1 -- 172.21.15.43:0/1879048234 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5580b1d4b440 msgr2=0x5580b1d45c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:22.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.245+0000 7f6b73400640 1 --2- 172.21.15.43:0/1879048234 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5580b1d4b440 0x5580b1d45c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.245+0000 7f6b73400640 1 -- 172.21.15.43:0/1879048234 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580b1d42db0 msgr2=0x5580b1d456c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:22.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.245+0000 7f6b73400640 1 --2- 172.21.15.43:0/1879048234 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580b1d42db0 0x5580b1d456c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.245+0000 7f6b73400640 1 -- 172.21.15.43:0/1879048234 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580b197a0f0 con 0x5580b1d420d0 2024-03-28T22:44:22.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.246+0000 7f6b73400640 1 --2- 172.21.15.43:0/1879048234 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5580b1d420d0 0x5580b1d539e0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c012d00 tx=0x7f6b6c01e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a1831c32c77f78 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:22.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.246+0000 7f6b5cc00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:44:22.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.246+0000 7f6b5cc00640 1 -- 172.21.15.43:0/1879048234 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c025040 con 0x5580b1d420d0 2024-03-28T22:44:22.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.246+0000 7f6b7cb6b740 1 -- 172.21.15.43:0/1879048234 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5580b1cad990 con 0x5580b1d420d0 2024-03-28T22:44:22.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.246+0000 7f6b5cc00640 1 -- 172.21.15.43:0/1879048234 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f6b6c012650 con 0x5580b1d420d0 2024-03-28T22:44:22.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.246+0000 7f6b5cc00640 1 -- 172.21.15.43:0/1879048234 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6b6c012850 con 0x5580b1d420d0 2024-03-28T22:44:22.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.246+0000 7f6b7cb6b740 10 client.28196 fetch_fsmap learned FSMap version 32 2024-03-28T22:44:22.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.246+0000 7f6b7cb6b740 1 -- 172.21.15.43:0/1879048234 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5580b1dc1c30 con 0x5580b1d420d0 2024-03-28T22:44:22.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.246+0000 7f6b5cc00640 1 -- 172.21.15.43:0/1879048234 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5580b1cad990 con 0x5580b1d420d0 2024-03-28T22:44:22.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.247+0000 7f6b5cc00640 1 -- 172.21.15.43:0/1879048234 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6b6c02b040 con 0x5580b1d420d0 2024-03-28T22:44:22.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.247+0000 7f6b7cb6b740 10 client.28196 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:44:22.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.247+0000 7f6b7cb6b740 10 client.28196 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:44:22.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.247+0000 7f6b7cb6b740 20 client.28196 populate_metadata read hostname 'smithi043' 2024-03-28T22:44:22.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.247+0000 7f6b7cb6b740 1 --2- 172.21.15.43:0/1879048234 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5580b1dc2010 0x5580b1dc4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:22.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.247+0000 7f6b7cb6b740 4 client.28196 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:44:22.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.247+0000 7f6b7cb6b740 1 -- 172.21.15.43:0/1879048234 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5580b1c2b400 con 0x5580b1dc2010 2024-03-28T22:44:22.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.247+0000 7f6b72000640 1 --2- 172.21.15.43:0/1879048234 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5580b1dc2010 0x5580b1dc4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:22.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.248+0000 7f6b5cc00640 10 client.28196 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:44:22.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.248+0000 7f6b72000640 1 --2- 172.21.15.43:0/1879048234 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5580b1dc2010 0x5580b1dc4430 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9aa3e33e90df661 server_cookie=55ee96dfee2d9b7e in_seq=0 out_seq=0 2024-03-28T22:44:22.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.248+0000 7f6b5cc00640 1 -- 172.21.15.43:0/1879048234 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5580b1c2b400 con 0x5580b1dc2010 2024-03-28T22:44:22.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.248+0000 7f6b5cc00640 10 client.28196 handle_command_reply: tid=0 2024-03-28T22:44:22.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.248+0000 7f6b7cb6b740 1 client.28196 shutdown 2024-03-28T22:44:22.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.248+0000 7f6b7cb6b740 1 -- 172.21.15.43:0/1879048234 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5580b1d420d0 msgr2=0x5580b1d539e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:22.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.248+0000 7f6b7cb6b740 1 --2- 172.21.15.43:0/1879048234 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5580b1d420d0 0x5580b1d539e0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c012d00 tx=0x7f6b6c01e040 comp rx=0 tx=0).stop 2024-03-28T22:44:22.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.248+0000 7f6b7cb6b740 1 -- 172.21.15.43:0/1879048234 shutdown_connections 2024-03-28T22:44:22.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.248+0000 7f6b7cb6b740 1 --2- 172.21.15.43:0/1879048234 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5580b1d4b440 0x5580b1d45c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.248+0000 7f6b7cb6b740 1 --2- 172.21.15.43:0/1879048234 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5580b1d42db0 0x5580b1d456c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.248+0000 7f6b7cb6b740 1 --2- 172.21.15.43:0/1879048234 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5580b1dc2010 0x5580b1dc4430 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.248+0000 7f6b7cb6b740 1 --2- 172.21.15.43:0/1879048234 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5580b1d420d0 0x5580b1d539e0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.248+0000 7f6b5cc00640 0 client.28196 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:44:22.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.248+0000 7f6b7cb6b740 1 -- 172.21.15.43:0/1879048234 >> 172.21.15.43:0/1879048234 conn(0x5580b1d08a10 msgr2=0x5580b1d3df00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:22.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.248+0000 7f6b7cb6b740 1 -- 172.21.15.43:0/1879048234 shutdown_connections 2024-03-28T22:44:22.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.248+0000 7f6b7cb6b740 1 -- 172.21.15.43:0/1879048234 wait complete. 2024-03-28T22:44:22.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.248+0000 7f6b7cb6b740 20 client.28196 trim_cache size 0 max 16384 2024-03-28T22:44:22.257 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:44:22.257 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-03-28T22:44:22.257 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-03-28T22:44:22.257 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:44:22.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.249+0000 7f6b73e00640 1 -- 172.21.15.43:0/1203289890 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6b580a1ef0 msgr2=0x7f6b580a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:22.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.249+0000 7f6b73e00640 1 --2- 172.21.15.43:0/1203289890 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6b580a1ef0 0x7f6b580a43b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x5580b1d43300 tx=0x7f6b640045f0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.249+0000 7f6b73e00640 1 -- 172.21.15.43:0/1203289890 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b741015c0 msgr2=0x7f6b7419a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:22.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.249+0000 7f6b73e00640 1 --2- 172.21.15.43:0/1203289890 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b741015c0 0x7f6b7419a070 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f6b6400e9c0 tx=0x7f6b6400ee80 comp rx=0 tx=0).stop 2024-03-28T22:44:22.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.249+0000 7f6b73e00640 1 -- 172.21.15.43:0/1203289890 shutdown_connections 2024-03-28T22:44:22.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.249+0000 7f6b73e00640 1 --2- 172.21.15.43:0/1203289890 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6b580a1ef0 0x7f6b580a43b0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.249+0000 7f6b73e00640 1 --2- 172.21.15.43:0/1203289890 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b74107660 0x7f6b7419e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.249+0000 7f6b73e00640 1 --2- 172.21.15.43:0/1203289890 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6b74101f80 0x7f6b7419a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.249+0000 7f6b73e00640 1 --2- 172.21.15.43:0/1203289890 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6b741015c0 0x7f6b7419a070 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.249+0000 7f6b73e00640 1 -- 172.21.15.43:0/1203289890 >> 172.21.15.43:0/1203289890 conn(0x7f6b740fd0f0 msgr2=0x7f6b740fe520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:22.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.249+0000 7f6b73e00640 1 -- 172.21.15.43:0/1203289890 shutdown_connections 2024-03-28T22:44:22.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.249+0000 7f6b73e00640 1 -- 172.21.15.43:0/1203289890 wait complete. 2024-03-28T22:44:22.266 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:44:22.267 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:44:22.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.491+0000 7feffb200640 1 -- 172.21.15.43:0/4031820961 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feffc103720 msgr2=0x7feffc103ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:22.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.491+0000 7feffb200640 1 --2- 172.21.15.43:0/4031820961 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feffc103720 0x7feffc103ba0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fefe4009930 tx=0x7fefe402bc20 comp rx=0 tx=0).stop 2024-03-28T22:44:22.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.491+0000 7feffb200640 1 -- 172.21.15.43:0/4031820961 shutdown_connections 2024-03-28T22:44:22.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.492+0000 7feffb200640 1 --2- 172.21.15.43:0/4031820961 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feffc1040e0 0x7feffc1105f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.492+0000 7feffb200640 1 --2- 172.21.15.43:0/4031820961 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feffc103720 0x7feffc103ba0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.492+0000 7feffb200640 1 --2- 172.21.15.43:0/4031820961 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feffc109730 0x7feffc109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.492+0000 7feffb200640 1 -- 172.21.15.43:0/4031820961 >> 172.21.15.43:0/4031820961 conn(0x7feffc0ff1f0 msgr2=0x7feffc101610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:22.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.492+0000 7feffb200640 1 -- 172.21.15.43:0/4031820961 shutdown_connections 2024-03-28T22:44:22.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.492+0000 7feffb200640 1 -- 172.21.15.43:0/4031820961 wait complete. 2024-03-28T22:44:22.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.493+0000 7feffb200640 1 Processor -- start 2024-03-28T22:44:22.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.493+0000 7feffb200640 1 -- start start 2024-03-28T22:44:22.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.493+0000 7feffb200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feffc103720 0x7feffc19a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:22.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.494+0000 7feffb200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feffc1040e0 0x7feffc19a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:22.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.494+0000 7feffb200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feffc109730 0x7feffc19e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:22.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.494+0000 7feffb200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7feffc113b90 con 0x7feffc1040e0 2024-03-28T22:44:22.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.494+0000 7feffb200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7feffc113a10 con 0x7feffc103720 2024-03-28T22:44:22.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.494+0000 7feffb200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7feffc113d10 con 0x7feffc109730 2024-03-28T22:44:22.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.494+0000 7feff9e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feffc103720 0x7feffc19a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:22.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.494+0000 7feff9e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feffc103720 0x7feffc19a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48480/0 (socket says 172.21.15.43:48480) 2024-03-28T22:44:22.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.494+0000 7feff9e00640 1 -- 172.21.15.43:0/2152374617 learned_addr learned my addr 172.21.15.43:0/2152374617 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:22.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.494+0000 7feffa800640 1 --2- 172.21.15.43:0/2152374617 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feffc109730 0x7feffc19e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:22.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.494+0000 7feff3e00640 1 --2- 172.21.15.43:0/2152374617 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feffc1040e0 0x7feffc19a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:22.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.494+0000 7feff9e00640 1 -- 172.21.15.43:0/2152374617 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feffc109730 msgr2=0x7feffc19e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:22.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.494+0000 7feff9e00640 1 --2- 172.21.15.43:0/2152374617 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feffc109730 0x7feffc19e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.494+0000 7feff9e00640 1 -- 172.21.15.43:0/2152374617 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feffc1040e0 msgr2=0x7feffc19a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:22.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.494+0000 7feff9e00640 1 --2- 172.21.15.43:0/2152374617 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feffc1040e0 0x7feffc19a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.494+0000 7feff9e00640 1 -- 172.21.15.43:0/2152374617 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feffc19f0b0 con 0x7feffc103720 2024-03-28T22:44:22.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.494+0000 7feffa800640 1 --2- 172.21.15.43:0/2152374617 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feffc109730 0x7feffc19e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:44:22.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.495+0000 7feff9e00640 1 --2- 172.21.15.43:0/2152374617 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feffc103720 0x7feffc19a110 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fefec00ea60 tx=0x7fefec00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:22.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.495+0000 7feff2a00640 1 -- 172.21.15.43:0/2152374617 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefec00cde0 con 0x7feffc103720 2024-03-28T22:44:22.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.495+0000 7feff2a00640 1 -- 172.21.15.43:0/2152374617 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefec0035f0 con 0x7feffc103720 2024-03-28T22:44:22.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.495+0000 7feff2a00640 1 -- 172.21.15.43:0/2152374617 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefec010670 con 0x7feffc103720 2024-03-28T22:44:22.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.495+0000 7feffb200640 1 -- 172.21.15.43:0/2152374617 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feffc19f3a0 con 0x7feffc103720 2024-03-28T22:44:22.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.495+0000 7feffb200640 1 -- 172.21.15.43:0/2152374617 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feffc1a6c30 con 0x7feffc103720 2024-03-28T22:44:22.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.496+0000 7feffb200640 1 -- 172.21.15.43:0/2152374617 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feffc0688b0 con 0x7feffc103720 2024-03-28T22:44:22.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.498+0000 7feff2a00640 1 -- 172.21.15.43:0/2152374617 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fefec0037a0 con 0x7feffc103720 2024-03-28T22:44:22.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.500+0000 7feff2a00640 1 --2- 172.21.15.43:0/2152374617 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fefe00a1f70 0x7fefe00a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:22.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.500+0000 7feff2a00640 1 -- 172.21.15.43:0/2152374617 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7fefec0ce7d0 con 0x7feffc103720 2024-03-28T22:44:22.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.500+0000 7feff3e00640 1 --2- 172.21.15.43:0/2152374617 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fefe00a1f70 0x7fefe00a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:22.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.501+0000 7feff3e00640 1 --2- 172.21.15.43:0/2152374617 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fefe00a1f70 0x7fefe00a4430 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fefe4009900 tx=0x7fefe4005a20 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:22.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.503+0000 7feff2a00640 1 -- 172.21.15.43:0/2152374617 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fefec09b490 con 0x7feffc103720 2024-03-28T22:44:22.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.830+0000 7feffb200640 1 -- 172.21.15.43:0/2152374617 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feffc19b410 con 0x7feffc103720 2024-03-28T22:44:22.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.831+0000 7feff2a00640 1 -- 172.21.15.43:0/2152374617 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11931 (secure 0 0 0) 0x7fefec09ad90 con 0x7feffc103720 2024-03-28T22:44:22.829 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:44:22.829 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:33:28.906396+0000","modified":"2024-03-28T22:40:08.672168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24572,"mds_1":24556,"mds_2":24535,"mds_3":24523,"mds_4":24518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26717":{"gid":26717,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/215779342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":215779342},{"type":"v1","addr":"172.21.15.82:6835","nonce":215779342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24535,24556,24518,24523,24572]},"id":1}]} 2024-03-28T22:44:22.829 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 32 2024-03-28T22:44:22.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.834+0000 7feffb200640 1 -- 172.21.15.43:0/2152374617 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fefe00a1f70 msgr2=0x7fefe00a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:22.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.834+0000 7feffb200640 1 --2- 172.21.15.43:0/2152374617 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fefe00a1f70 0x7fefe00a4430 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fefe4009900 tx=0x7fefe4005a20 comp rx=0 tx=0).stop 2024-03-28T22:44:22.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.834+0000 7feffb200640 1 -- 172.21.15.43:0/2152374617 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feffc103720 msgr2=0x7feffc19a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:22.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.834+0000 7feffb200640 1 --2- 172.21.15.43:0/2152374617 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feffc103720 0x7feffc19a110 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fefec00ea60 tx=0x7fefec00ef20 comp rx=0 tx=0).stop 2024-03-28T22:44:22.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.834+0000 7feffb200640 1 -- 172.21.15.43:0/2152374617 shutdown_connections 2024-03-28T22:44:22.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.834+0000 7feffb200640 1 --2- 172.21.15.43:0/2152374617 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fefe00a1f70 0x7fefe00a4430 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.834+0000 7feffb200640 1 --2- 172.21.15.43:0/2152374617 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feffc109730 0x7feffc19e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.834+0000 7feffb200640 1 --2- 172.21.15.43:0/2152374617 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feffc1040e0 0x7feffc19a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.834+0000 7feffb200640 1 --2- 172.21.15.43:0/2152374617 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feffc103720 0x7feffc19a110 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:22.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.834+0000 7feffb200640 1 -- 172.21.15.43:0/2152374617 >> 172.21.15.43:0/2152374617 conn(0x7feffc0ff1f0 msgr2=0x7feffc0ff8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:22.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.834+0000 7feffb200640 1 -- 172.21.15.43:0/2152374617 shutdown_connections 2024-03-28T22:44:22.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:22.834+0000 7feffb200640 1 -- 172.21.15.43:0/2152374617 wait complete. 2024-03-28T22:44:22.851 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T22:44:23.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.077+0000 7f5c04a00640 1 -- 172.21.15.43:0/4043891598 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c00103300 msgr2=0x7f5c0010b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.077+0000 7f5c04a00640 1 --2- 172.21.15.43:0/4043891598 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c00103300 0x7f5c0010b7e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f5be8009900 tx=0x7f5be802bcd0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.077+0000 7f5c04a00640 1 -- 172.21.15.43:0/4043891598 shutdown_connections 2024-03-28T22:44:23.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.077+0000 7f5c04a00640 1 --2- 172.21.15.43:0/4043891598 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c0010bd20 0x7f5c0010e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.077+0000 7f5c04a00640 1 --2- 172.21.15.43:0/4043891598 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c00103300 0x7f5c0010b7e0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.077+0000 7f5c04a00640 1 --2- 172.21.15.43:0/4043891598 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5c001029e0 0x7f5c00102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.077+0000 7f5c04a00640 1 -- 172.21.15.43:0/4043891598 >> 172.21.15.43:0/4043891598 conn(0x7f5c000777b0 msgr2=0x7f5c00077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:23.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.077+0000 7f5c04a00640 1 -- 172.21.15.43:0/4043891598 shutdown_connections 2024-03-28T22:44:23.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.077+0000 7f5c04a00640 1 -- 172.21.15.43:0/4043891598 wait complete. 2024-03-28T22:44:23.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.078+0000 7f5c04a00640 1 Processor -- start 2024-03-28T22:44:23.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.078+0000 7f5c04a00640 1 -- start start 2024-03-28T22:44:23.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.079+0000 7f5c04a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5c001029e0 0x7f5c0019a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.079+0000 7f5c04a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c00103300 0x7f5c0019a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.079+0000 7f5c04a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c0010bd20 0x7f5c0019eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.079+0000 7f5c04a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5c00111b40 con 0x7f5c0010bd20 2024-03-28T22:44:23.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.079+0000 7f5c04a00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5c001119c0 con 0x7f5c001029e0 2024-03-28T22:44:23.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.079+0000 7f5c04a00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f5c00111cc0 con 0x7f5c00103300 2024-03-28T22:44:23.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.079+0000 7f5bfde00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5c001029e0 0x7f5c0019a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.079+0000 7f5bf7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c00103300 0x7f5c0019a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.079+0000 7f5bfde00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5c001029e0 0x7f5c0019a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48492/0 (socket says 172.21.15.43:48492) 2024-03-28T22:44:23.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.079+0000 7f5bfde00640 1 -- 172.21.15.43:0/1110075622 learned_addr learned my addr 172.21.15.43:0/1110075622 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:23.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.079+0000 7f5bf7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c00103300 0x7f5c0019a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:47862/0 (socket says 172.21.15.43:47862) 2024-03-28T22:44:23.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.079+0000 7f5bfe800640 1 --2- 172.21.15.43:0/1110075622 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c0010bd20 0x7f5c0019eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.080+0000 7f5bf7e00640 1 -- 172.21.15.43:0/1110075622 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5c001029e0 msgr2=0x7f5c0019a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.080+0000 7f5bf7e00640 1 --2- 172.21.15.43:0/1110075622 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5c001029e0 0x7f5c0019a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.080+0000 7f5bf7e00640 1 -- 172.21.15.43:0/1110075622 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c0010bd20 msgr2=0x7f5c0019eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.080+0000 7f5bf7e00640 1 --2- 172.21.15.43:0/1110075622 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c0010bd20 0x7f5c0019eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.080+0000 7f5bf7e00640 1 -- 172.21.15.43:0/1110075622 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c0019f1e0 con 0x7f5c00103300 2024-03-28T22:44:23.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.080+0000 7f5bfe800640 1 --2- 172.21.15.43:0/1110075622 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c0010bd20 0x7f5c0019eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:44:23.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.080+0000 7f5bfde00640 1 --2- 172.21.15.43:0/1110075622 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5c001029e0 0x7f5c0019a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:44:23.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.080+0000 7f5bf7e00640 1 --2- 172.21.15.43:0/1110075622 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c00103300 0x7f5c0019a780 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f5be802c210 tx=0x7f5be8004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:23.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.080+0000 7f5bf6a00640 1 -- 172.21.15.43:0/1110075622 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be8042020 con 0x7f5c00103300 2024-03-28T22:44:23.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.080+0000 7f5bf6a00640 1 -- 172.21.15.43:0/1110075622 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5be80054e0 con 0x7f5c00103300 2024-03-28T22:44:23.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.080+0000 7f5c04a00640 1 -- 172.21.15.43:0/1110075622 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c0019f470 con 0x7f5c00103300 2024-03-28T22:44:23.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.080+0000 7f5bf6a00640 1 -- 172.21.15.43:0/1110075622 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be8003b00 con 0x7f5c00103300 2024-03-28T22:44:23.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.080+0000 7f5c04a00640 1 -- 172.21.15.43:0/1110075622 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c001a6d60 con 0x7f5c00103300 2024-03-28T22:44:23.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.084+0000 7f5c07633740 1 Processor -- start 2024-03-28T22:44:23.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.084+0000 7f5c07633740 1 -- start start 2024-03-28T22:44:23.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.084+0000 7f5bf6a00640 1 -- 172.21.15.43:0/1110075622 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f5be8045020 con 0x7f5c00103300 2024-03-28T22:44:23.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.084+0000 7f5c07633740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5601c29cd380 0x5601c29cd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.084+0000 7f5c07633740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5601c29cdca0 0x5601c29d6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.084+0000 7f5c07633740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5601c29d6880 0x5601c29d8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.084+0000 7f5c07633740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5601c2931050 con 0x5601c29cdca0 2024-03-28T22:44:23.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.084+0000 7f5c07633740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5601c28d9ce0 con 0x5601c29cd380 2024-03-28T22:44:23.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.084+0000 7f5c07633740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5601c292d0f0 con 0x5601c29d6880 2024-03-28T22:44:23.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.084+0000 7f5bfe800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5601c29cd380 0x5601c29cd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.085+0000 7f5bf7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5601c29d6880 0x5601c29d8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.085+0000 7f5bfde00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5601c29cdca0 0x5601c29d6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.085+0000 7f5bfe800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5601c29cd380 0x5601c29cd760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48494/0 (socket says 172.21.15.43:48494) 2024-03-28T22:44:23.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.085+0000 7f5bfe800640 1 -- 172.21.15.43:0/96473681 learned_addr learned my addr 172.21.15.43:0/96473681 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:23.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.085+0000 7f5bf7e00640 1 -- 172.21.15.43:0/96473681 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5601c29cd380 msgr2=0x5601c29cd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:23.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.085+0000 7f5bf7e00640 1 --2- 172.21.15.43:0/96473681 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5601c29cd380 0x5601c29cd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.085+0000 7f5bf7e00640 1 -- 172.21.15.43:0/96473681 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5601c29cdca0 msgr2=0x5601c29d6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:23.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.085+0000 7f5bf7e00640 1 --2- 172.21.15.43:0/96473681 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5601c29cdca0 0x5601c29d6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.085+0000 7f5bf7e00640 1 -- 172.21.15.43:0/96473681 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601c26453f0 con 0x5601c29d6880 2024-03-28T22:44:23.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.085+0000 7f5bf7e00640 1 --2- 172.21.15.43:0/96473681 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5601c29d6880 0x5601c29d8c70 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f5be80ed4b0 tx=0x7f5be80f81a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=196b0b809fd4b204 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:23.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.085+0000 7f5bf6a00640 1 --2- 172.21.15.43:0/1110075622 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f5be40a1f70 0x7f5be40a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.086+0000 7f5bef000640 1 -- 172.21.15.43:0/96473681 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be80b89f0 con 0x5601c29d6880 2024-03-28T22:44:23.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.086+0000 7f5bef000640 1 -- 172.21.15.43:0/96473681 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5be80fd030 con 0x5601c29d6880 2024-03-28T22:44:23.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.086+0000 7f5bef000640 1 -- 172.21.15.43:0/96473681 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be80bbba0 con 0x5601c29d6880 2024-03-28T22:44:23.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.086+0000 7f5bf6a00640 1 -- 172.21.15.43:0/1110075622 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f5be80f0080 con 0x7f5c00103300 2024-03-28T22:44:23.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.086+0000 7f5bfe800640 1 --2- 172.21.15.43:0/1110075622 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f5be40a1f70 0x7f5be40a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.086+0000 7f5c07633740 1 -- 172.21.15.43:0/96473681 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5601c29d6880 msgr2=0x5601c29d8c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.086+0000 7f5c07633740 1 --2- 172.21.15.43:0/96473681 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5601c29d6880 0x5601c29d8c70 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f5be80ed4b0 tx=0x7f5be80f81a0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.087+0000 7f5c07633740 1 -- 172.21.15.43:0/96473681 shutdown_connections 2024-03-28T22:44:23.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.087+0000 7f5c07633740 1 --2- 172.21.15.43:0/96473681 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5601c29d6880 0x5601c29d8c70 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.087+0000 7f5c07633740 1 --2- 172.21.15.43:0/96473681 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5601c29cdca0 0x5601c29d6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.087+0000 7f5c07633740 1 --2- 172.21.15.43:0/96473681 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5601c29cd380 0x5601c29cd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.087+0000 7f5c07633740 1 -- 172.21.15.43:0/96473681 >> 172.21.15.43:0/96473681 conn(0x5601c29c7ab0 msgr2=0x5601c29c7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:23.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.087+0000 7f5bfe800640 1 --2- 172.21.15.43:0/1110075622 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f5be40a1f70 0x7f5be40a4430 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8013570 tx=0x7f5bf8009810 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:23.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.087+0000 7f5c07633740 1 -- 172.21.15.43:0/96473681 shutdown_connections 2024-03-28T22:44:23.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.087+0000 7f5c07633740 1 -- 172.21.15.43:0/96473681 wait complete. 2024-03-28T22:44:23.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.087+0000 7f5c07633740 1 -- start start 2024-03-28T22:44:23.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.087+0000 7f5c07633740 1 Processor -- start 2024-03-28T22:44:23.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.088+0000 7f5c07633740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5601c29cd380 0x5601c29d38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.088+0000 7f5c07633740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5601c29cdca0 0x5601c29d3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.088+0000 7f5c07633740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5601c29d6880 0x5601c29d4360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.088+0000 7f5c07633740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5601c29d9430 con 0x5601c29cdca0 2024-03-28T22:44:23.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.088+0000 7f5c07633740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5601c28cf2f0 con 0x5601c29cd380 2024-03-28T22:44:23.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.088+0000 7f5c07633740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5601c29d95b0 con 0x5601c29d6880 2024-03-28T22:44:23.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.088+0000 7f5c07633740 10 client.0 _ll_register_callbacks cb 0x5601c2644be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:44:23.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.088+0000 7f5bfde00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5601c29cd380 0x5601c29d38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.088+0000 7f5bf7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5601c29cdca0 0x5601c29d3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.088+0000 7f5bfde00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5601c29cd380 0x5601c29d38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48498/0 (socket says 172.21.15.43:48498) 2024-03-28T22:44:23.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.088+0000 7f5bfde00640 1 -- 172.21.15.43:0/128291823 learned_addr learned my addr 172.21.15.43:0/128291823 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:23.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.088+0000 7f5bfe800640 1 --2- 172.21.15.43:0/128291823 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5601c29d6880 0x5601c29d4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.089+0000 7f5bfde00640 1 -- 172.21.15.43:0/128291823 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5601c29d6880 msgr2=0x5601c29d4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:23.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.089+0000 7f5bfde00640 1 --2- 172.21.15.43:0/128291823 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5601c29d6880 0x5601c29d4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.089+0000 7f5bfde00640 1 -- 172.21.15.43:0/128291823 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5601c29cdca0 msgr2=0x5601c29d3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:23.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.089+0000 7f5bfde00640 1 --2- 172.21.15.43:0/128291823 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5601c29cdca0 0x5601c29d3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.089+0000 7f5bfde00640 1 -- 172.21.15.43:0/128291823 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601c270f7d0 con 0x5601c29cd380 2024-03-28T22:44:23.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.089+0000 7f5bf7e00640 1 --2- 172.21.15.43:0/128291823 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5601c29cdca0 0x5601c29d3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:44:23.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.089+0000 7f5bfe800640 1 --2- 172.21.15.43:0/128291823 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5601c29d6880 0x5601c29d4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:44:23.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.089+0000 7f5bfde00640 1 --2- 172.21.15.43:0/128291823 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5601c29cd380 0x5601c29d38e0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0012d20 tx=0x7f5bf000f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=faa49532abefcb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:23.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.089+0000 7f5be3400640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:44:23.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.089+0000 7f5be3400640 1 -- 172.21.15.43:0/128291823 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf0005020 con 0x5601c29cd380 2024-03-28T22:44:23.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.089+0000 7f5c07633740 1 -- 172.21.15.43:0/128291823 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5be80fd030 con 0x5601c29cd380 2024-03-28T22:44:23.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.089+0000 7f5be3400640 1 -- 172.21.15.43:0/128291823 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f5bf00051c0 con 0x5601c29cd380 2024-03-28T22:44:23.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.089+0000 7f5be3400640 1 -- 172.21.15.43:0/128291823 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5bf00053c0 con 0x5601c29cd380 2024-03-28T22:44:23.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.089+0000 7f5c07633740 10 client.28214 fetch_fsmap learned FSMap version 32 2024-03-28T22:44:23.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.089+0000 7f5c07633740 1 -- 172.21.15.43:0/128291823 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5601c29e56c0 con 0x5601c29cd380 2024-03-28T22:44:23.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.090+0000 7f5be3400640 1 -- 172.21.15.43:0/128291823 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5be80fd030 con 0x5601c29cd380 2024-03-28T22:44:23.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.090+0000 7f5be3400640 1 -- 172.21.15.43:0/128291823 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f5bf001c490 con 0x5601c29cd380 2024-03-28T22:44:23.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.090+0000 7f5c07633740 10 client.28214 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:44:23.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.090+0000 7f5c07633740 10 client.28214 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:44:23.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.090+0000 7f5c07633740 20 client.28214 populate_metadata read hostname 'smithi043' 2024-03-28T22:44:23.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.090+0000 7f5c07633740 1 --2- 172.21.15.43:0/128291823 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x5601c29e6b80 0x5601c29e8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.090+0000 7f5c07633740 4 client.28214 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:44:23.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.090+0000 7f5c07633740 1 -- 172.21.15.43:0/128291823 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5601c28da680 con 0x5601c29e6b80 2024-03-28T22:44:23.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.090+0000 7f5bf7e00640 1 --2- 172.21.15.43:0/128291823 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x5601c29e6b80 0x5601c29e8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.091+0000 7f5bf7e00640 1 --2- 172.21.15.43:0/128291823 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x5601c29e6b80 0x5601c29e8fb0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=78070b234cbe2620 server_cookie=5eb3b206540c311e in_seq=0 out_seq=0 2024-03-28T22:44:23.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.091+0000 7f5be3400640 10 client.28214 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:44:23.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.092+0000 7f5be3400640 1 -- 172.21.15.43:0/128291823 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5601c28da680 con 0x5601c29e6b80 2024-03-28T22:44:23.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.092+0000 7f5be3400640 10 client.28214 handle_command_reply: tid=0 2024-03-28T22:44:23.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.092+0000 7f5c07633740 1 client.28214 shutdown 2024-03-28T22:44:23.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.092+0000 7f5c07633740 1 -- 172.21.15.43:0/128291823 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5601c29cd380 msgr2=0x5601c29d38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.092+0000 7f5c07633740 1 --2- 172.21.15.43:0/128291823 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5601c29cd380 0x5601c29d38e0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0012d20 tx=0x7f5bf000f790 comp rx=0 tx=0).stop 2024-03-28T22:44:23.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.092+0000 7f5c07633740 1 -- 172.21.15.43:0/128291823 shutdown_connections 2024-03-28T22:44:23.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.092+0000 7f5c07633740 1 --2- 172.21.15.43:0/128291823 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x5601c29e6b80 0x5601c29e8fb0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.092+0000 7f5c07633740 1 --2- 172.21.15.43:0/128291823 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5601c29d6880 0x5601c29d4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.092+0000 7f5c07633740 1 --2- 172.21.15.43:0/128291823 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5601c29cdca0 0x5601c29d3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.092+0000 7f5c07633740 1 --2- 172.21.15.43:0/128291823 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5601c29cd380 0x5601c29d38e0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.092+0000 7f5c07633740 1 -- 172.21.15.43:0/128291823 >> 172.21.15.43:0/128291823 conn(0x5601c29c7ab0 msgr2=0x5601c29c9800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:23.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.092+0000 7f5be3400640 0 client.28214 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:44:23.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.093+0000 7f5c07633740 1 -- 172.21.15.43:0/128291823 shutdown_connections 2024-03-28T22:44:23.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.093+0000 7f5c07633740 1 -- 172.21.15.43:0/128291823 wait complete. 2024-03-28T22:44:23.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.093+0000 7f5c07633740 20 client.28214 trim_cache size 0 max 16384 2024-03-28T22:44:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.108+0000 7f5c07633740 1 Processor -- start 2024-03-28T22:44:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.108+0000 7f5c07633740 1 -- start start 2024-03-28T22:44:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.108+0000 7f5c07633740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5601c29cd380 0x5601c29e20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.108+0000 7f5c07633740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5601c29cdca0 0x5601c29c7e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.108+0000 7f5c07633740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5601c29d6880 0x5601c29ee880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.108+0000 7f5c07633740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5c001b0e10 con 0x5601c29cd380 2024-03-28T22:44:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.108+0000 7f5c07633740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5c001b2950 con 0x5601c29d6880 2024-03-28T22:44:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.108+0000 7f5c07633740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f5c001b14e0 con 0x5601c29cdca0 2024-03-28T22:44:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.108+0000 7f5bfde00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5601c29cd380 0x5601c29e20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.108+0000 7f5bfe800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5601c29d6880 0x5601c29ee880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.108+0000 7f5bfde00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5601c29cd380 0x5601c29e20c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44222/0 (socket says 172.21.15.43:44222) 2024-03-28T22:44:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.108+0000 7f5bfde00640 1 -- 172.21.15.43:0/3167273303 learned_addr learned my addr 172.21.15.43:0/3167273303 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.108+0000 7f5bf7e00640 1 --2- 172.21.15.43:0/3167273303 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5601c29cdca0 0x5601c29c7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.109+0000 7f5bfde00640 1 -- 172.21.15.43:0/3167273303 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5601c29cdca0 msgr2=0x5601c29c7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:23.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.109+0000 7f5bfde00640 1 --2- 172.21.15.43:0/3167273303 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5601c29cdca0 0x5601c29c7e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.109+0000 7f5bfde00640 1 -- 172.21.15.43:0/3167273303 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5601c29d6880 msgr2=0x5601c29ee880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:23.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.109+0000 7f5bfde00640 1 --2- 172.21.15.43:0/3167273303 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5601c29d6880 0x5601c29ee880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.109+0000 7f5bfde00640 1 -- 172.21.15.43:0/3167273303 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601c29d2890 con 0x5601c29cd380 2024-03-28T22:44:23.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.109+0000 7f5bfde00640 1 --2- 172.21.15.43:0/3167273303 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5601c29cd380 0x5601c29e20c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5bf001c8b0 tx=0x7f5bf000fd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcdd7b012dfac5fa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:23.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.109+0000 7f5bee600640 1 -- 172.21.15.43:0/3167273303 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf002e070 con 0x5601c29cd380 2024-03-28T22:44:23.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.109+0000 7f5bee600640 1 -- 172.21.15.43:0/3167273303 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bf0009e30 con 0x5601c29cd380 2024-03-28T22:44:23.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.109+0000 7f5bee600640 1 -- 172.21.15.43:0/3167273303 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf0029b80 con 0x5601c29cd380 2024-03-28T22:44:23.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.110+0000 7f5c07633740 1 -- 172.21.15.43:0/3167273303 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5601c29cd380 msgr2=0x5601c29e20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.110+0000 7f5c07633740 1 --2- 172.21.15.43:0/3167273303 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5601c29cd380 0x5601c29e20c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5bf001c8b0 tx=0x7f5bf000fd10 comp rx=0 tx=0).stop 2024-03-28T22:44:23.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.110+0000 7f5c07633740 1 -- 172.21.15.43:0/3167273303 shutdown_connections 2024-03-28T22:44:23.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.110+0000 7f5c07633740 1 --2- 172.21.15.43:0/3167273303 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5601c29d6880 0x5601c29ee880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.110+0000 7f5c07633740 1 --2- 172.21.15.43:0/3167273303 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5601c29cdca0 0x5601c29c7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.110+0000 7f5c07633740 1 --2- 172.21.15.43:0/3167273303 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5601c29cd380 0x5601c29e20c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.110+0000 7f5c07633740 1 -- 172.21.15.43:0/3167273303 >> 172.21.15.43:0/3167273303 conn(0x5601c29c7ab0 msgr2=0x5601c29c6b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:23.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.110+0000 7f5c07633740 1 -- 172.21.15.43:0/3167273303 shutdown_connections 2024-03-28T22:44:23.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.110+0000 7f5c07633740 1 -- 172.21.15.43:0/3167273303 wait complete. 2024-03-28T22:44:23.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.111+0000 7f5c07633740 1 -- start start 2024-03-28T22:44:23.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.111+0000 7f5c07633740 1 Processor -- start 2024-03-28T22:44:23.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.111+0000 7f5c07633740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5601c29cd380 0x5601c29deb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.111+0000 7f5c07633740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5601c29cdca0 0x5601c29d9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.111+0000 7f5c07633740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5601c29d6880 0x5601c29da590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.111+0000 7f5c07633740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5c001b2280 con 0x5601c29d6880 2024-03-28T22:44:23.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.111+0000 7f5c07633740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5c001b1bb0 con 0x5601c29cdca0 2024-03-28T22:44:23.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.111+0000 7f5c07633740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f5c001aee10 con 0x5601c29cd380 2024-03-28T22:44:23.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.111+0000 7f5c07633740 10 client.0 _ll_register_callbacks cb 0x7f5bf0012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:44:23.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.112+0000 7f5bfde00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5601c29cd380 0x5601c29deb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.112+0000 7f5bfde00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5601c29cd380 0x5601c29deb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:47896/0 (socket says 172.21.15.43:47896) 2024-03-28T22:44:23.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.112+0000 7f5bfde00640 1 -- 172.21.15.43:0/2323215366 learned_addr learned my addr 172.21.15.43:0/2323215366 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:23.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.112+0000 7f5bf7e00640 1 --2- 172.21.15.43:0/2323215366 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5601c29cdca0 0x5601c29d9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.112+0000 7f5bfe800640 1 --2- 172.21.15.43:0/2323215366 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5601c29d6880 0x5601c29da590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.112+0000 7f5bfde00640 1 -- 172.21.15.43:0/2323215366 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5601c29cdca0 msgr2=0x5601c29d9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:23.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.112+0000 7f5bfde00640 1 --2- 172.21.15.43:0/2323215366 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5601c29cdca0 0x5601c29d9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.112+0000 7f5bfde00640 1 -- 172.21.15.43:0/2323215366 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5601c29d6880 msgr2=0x5601c29da590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:23.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.112+0000 7f5bfde00640 1 --2- 172.21.15.43:0/2323215366 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5601c29d6880 0x5601c29da590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.112+0000 7f5bfde00640 1 -- 172.21.15.43:0/2323215366 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601c29cdad0 con 0x5601c29cd380 2024-03-28T22:44:23.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.112+0000 7f5bfde00640 1 --2- 172.21.15.43:0/2323215366 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5601c29cd380 0x5601c29deb00 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0005fd0 tx=0x7f5bf00128a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a50686f20b428b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:23.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.112+0000 7f5be3400640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:44:23.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.113+0000 7f5be3400640 1 -- 172.21.15.43:0/2323215366 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf002e040 con 0x5601c29cd380 2024-03-28T22:44:23.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.113+0000 7f5c07633740 1 -- 172.21.15.43:0/2323215366 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bf0009e30 con 0x5601c29cd380 2024-03-28T22:44:23.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.113+0000 7f5be3400640 1 -- 172.21.15.43:0/2323215366 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f5bf0029440 con 0x5601c29cd380 2024-03-28T22:44:23.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.113+0000 7f5be3400640 1 -- 172.21.15.43:0/2323215366 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5bf0029660 con 0x5601c29cd380 2024-03-28T22:44:23.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.113+0000 7f5c07633740 10 client.28159 fetch_fsmap learned FSMap version 32 2024-03-28T22:44:23.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.113+0000 7f5c07633740 1 -- 172.21.15.43:0/2323215366 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5601c2a4ce60 con 0x5601c29cd380 2024-03-28T22:44:23.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.113+0000 7f5be3400640 1 -- 172.21.15.43:0/2323215366 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bf0009e30 con 0x5601c29cd380 2024-03-28T22:44:23.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.113+0000 7f5be3400640 1 -- 172.21.15.43:0/2323215366 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f5bf0031070 con 0x5601c29cd380 2024-03-28T22:44:23.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.113+0000 7f5c07633740 10 client.28159 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:44:23.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.113+0000 7f5c07633740 10 client.28159 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:44:23.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.113+0000 7f5c07633740 20 client.28159 populate_metadata read hostname 'smithi043' 2024-03-28T22:44:23.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.113+0000 7f5c07633740 1 --2- 172.21.15.43:0/2323215366 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x5601c2a4d4c0 0x5601c2a4f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.114+0000 7f5c07633740 4 client.28159 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:44:23.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.114+0000 7f5c07633740 1 -- 172.21.15.43:0/2323215366 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5601c293e590 con 0x5601c2a4d4c0 2024-03-28T22:44:23.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.114+0000 7f5bf7e00640 1 --2- 172.21.15.43:0/2323215366 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x5601c2a4d4c0 0x5601c2a4f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.114+0000 7f5bf7e00640 1 --2- 172.21.15.43:0/2323215366 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x5601c2a4d4c0 0x5601c2a4f8e0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=cf2de07fdd27ca01 server_cookie=3d91c9846643d041 in_seq=0 out_seq=0 2024-03-28T22:44:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.114+0000 7f5be3400640 10 client.28159 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:44:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.114+0000 7f5be3400640 1 -- 172.21.15.43:0/2323215366 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5601c293e590 con 0x5601c2a4d4c0 2024-03-28T22:44:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.114+0000 7f5be3400640 10 client.28159 handle_command_reply: tid=0 2024-03-28T22:44:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.114+0000 7f5c07633740 1 client.28159 shutdown 2024-03-28T22:44:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.115+0000 7f5c07633740 1 -- 172.21.15.43:0/2323215366 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5601c29cd380 msgr2=0x5601c29deb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.115+0000 7f5c07633740 1 --2- 172.21.15.43:0/2323215366 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5601c29cd380 0x5601c29deb00 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0005fd0 tx=0x7f5bf00128a0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.115+0000 7f5c07633740 1 -- 172.21.15.43:0/2323215366 shutdown_connections 2024-03-28T22:44:23.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.115+0000 7f5c07633740 1 --2- 172.21.15.43:0/2323215366 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x5601c2a4d4c0 0x5601c2a4f8e0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.115+0000 7f5c07633740 1 --2- 172.21.15.43:0/2323215366 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5601c29cdca0 0x5601c29d9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.115+0000 7f5c07633740 1 --2- 172.21.15.43:0/2323215366 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5601c29d6880 0x5601c29da590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.115+0000 7f5be3400640 0 client.28159 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:44:23.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.115+0000 7f5c07633740 1 --2- 172.21.15.43:0/2323215366 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5601c29cd380 0x5601c29deb00 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.115+0000 7f5c07633740 1 -- 172.21.15.43:0/2323215366 >> 172.21.15.43:0/2323215366 conn(0x5601c29c7ab0 msgr2=0x5601c29c6b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:23.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.115+0000 7f5c07633740 1 -- 172.21.15.43:0/2323215366 shutdown_connections 2024-03-28T22:44:23.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.115+0000 7f5c07633740 1 -- 172.21.15.43:0/2323215366 wait complete. 2024-03-28T22:44:23.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.115+0000 7f5c07633740 20 client.28159 trim_cache size 0 max 16384 2024-03-28T22:44:23.120 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:44:23.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.116+0000 7f5c04a00640 1 -- 172.21.15.43:0/1110075622 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f5be40a1f70 msgr2=0x7f5be40a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.116+0000 7f5c04a00640 1 --2- 172.21.15.43:0/1110075622 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f5be40a1f70 0x7f5be40a4430 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8013570 tx=0x7f5bf8009810 comp rx=0 tx=0).stop 2024-03-28T22:44:23.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.116+0000 7f5c04a00640 1 -- 172.21.15.43:0/1110075622 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c00103300 msgr2=0x7f5c0019a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.116+0000 7f5c04a00640 1 --2- 172.21.15.43:0/1110075622 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c00103300 0x7f5c0019a780 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f5be802c210 tx=0x7f5be8004360 comp rx=0 tx=0).stop 2024-03-28T22:44:23.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.116+0000 7f5c04a00640 1 -- 172.21.15.43:0/1110075622 shutdown_connections 2024-03-28T22:44:23.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.116+0000 7f5c04a00640 1 --2- 172.21.15.43:0/1110075622 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f5be40a1f70 0x7f5be40a4430 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.116+0000 7f5c04a00640 1 --2- 172.21.15.43:0/1110075622 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5c0010bd20 0x7f5c0019eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.116+0000 7f5c04a00640 1 --2- 172.21.15.43:0/1110075622 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5c00103300 0x7f5c0019a780 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.116+0000 7f5c04a00640 1 --2- 172.21.15.43:0/1110075622 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5c001029e0 0x7f5c0019a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.116+0000 7f5c04a00640 1 -- 172.21.15.43:0/1110075622 >> 172.21.15.43:0/1110075622 conn(0x7f5c000777b0 msgr2=0x7f5c00107c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:23.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.116+0000 7f5c04a00640 1 -- 172.21.15.43:0/1110075622 shutdown_connections 2024-03-28T22:44:23.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.116+0000 7f5c04a00640 1 -- 172.21.15.43:0/1110075622 wait complete. 2024-03-28T22:44:23.129 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:22 smithi043 ceph-mon[31823]: pgmap v223: 129 pgs: 129 active+clean; 50 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 130 KiB/s wr, 32 op/s 2024-03-28T22:44:23.133 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T22:44:23.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:22 smithi172 ceph-mon[36566]: pgmap v223: 129 pgs: 129 active+clean; 50 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 130 KiB/s wr, 32 op/s 2024-03-28T22:44:23.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:22 smithi082 ceph-mon[36471]: pgmap v223: 129 pgs: 129 active+clean; 50 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 130 KiB/s wr, 32 op/s 2024-03-28T22:44:23.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.341+0000 7f1d4fe00640 1 -- 172.21.15.43:0/3010822187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d501088b0 msgr2=0x7f1d5010c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.341+0000 7f1d4fe00640 1 --2- 172.21.15.43:0/3010822187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d501088b0 0x7f1d5010c870 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f1d38009900 tx=0x7f1d3802bc30 comp rx=0 tx=0).stop 2024-03-28T22:44:23.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.341+0000 7f1d4fe00640 1 -- 172.21.15.43:0/3010822187 shutdown_connections 2024-03-28T22:44:23.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.341+0000 7f1d4fe00640 1 --2- 172.21.15.43:0/3010822187 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d5010cf50 0x7f1d50114b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.341+0000 7f1d4fe00640 1 --2- 172.21.15.43:0/3010822187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d501088b0 0x7f1d5010c870 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.341+0000 7f1d4fe00640 1 --2- 172.21.15.43:0/3010822187 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1d50107f00 0x7f1d501082e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.341+0000 7f1d4fe00640 1 -- 172.21.15.43:0/3010822187 >> 172.21.15.43:0/3010822187 conn(0x7f1d50077570 msgr2=0x7f1d50106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:23.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.341+0000 7f1d4fe00640 1 -- 172.21.15.43:0/3010822187 shutdown_connections 2024-03-28T22:44:23.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.341+0000 7f1d4fe00640 1 -- 172.21.15.43:0/3010822187 wait complete. 2024-03-28T22:44:23.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.342+0000 7f1d4fe00640 1 Processor -- start 2024-03-28T22:44:23.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.342+0000 7f1d4fe00640 1 -- start start 2024-03-28T22:44:23.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.342+0000 7f1d4fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d50107f00 0x7f1d5019e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.342+0000 7f1d4fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1d5010cf50 0x7f1d5019ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.342+0000 7f1d4fe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d501a2dc0 0x7f1d501a3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.342+0000 7f1d4fe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1d50118050 con 0x7f1d50107f00 2024-03-28T22:44:23.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.342+0000 7f1d4fe00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1d50117ed0 con 0x7f1d5010cf50 2024-03-28T22:44:23.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.342+0000 7f1d4fe00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f1d501181d0 con 0x7f1d501a2dc0 2024-03-28T22:44:23.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.343+0000 7f1d4f400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d501a2dc0 0x7f1d501a3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.343+0000 7f1d46000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1d5010cf50 0x7f1d5019ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.343+0000 7f1d4ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d50107f00 0x7f1d5019e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.343+0000 7f1d46000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1d5010cf50 0x7f1d5019ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48532/0 (socket says 172.21.15.43:48532) 2024-03-28T22:44:23.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.343+0000 7f1d4f400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d501a2dc0 0x7f1d501a3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:47912/0 (socket says 172.21.15.43:47912) 2024-03-28T22:44:23.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.343+0000 7f1d46000640 1 -- 172.21.15.43:0/1966551705 learned_addr learned my addr 172.21.15.43:0/1966551705 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:23.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.343+0000 7f1d46000640 1 -- 172.21.15.43:0/1966551705 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d501a2dc0 msgr2=0x7f1d501a3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.343+0000 7f1d46000640 1 --2- 172.21.15.43:0/1966551705 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d501a2dc0 0x7f1d501a3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.343+0000 7f1d46000640 1 -- 172.21.15.43:0/1966551705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d50107f00 msgr2=0x7f1d5019e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.343+0000 7f1d46000640 1 --2- 172.21.15.43:0/1966551705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d50107f00 0x7f1d5019e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.343+0000 7f1d46000640 1 -- 172.21.15.43:0/1966551705 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d501a38b0 con 0x7f1d5010cf50 2024-03-28T22:44:23.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.343+0000 7f1d4f400640 1 --2- 172.21.15.43:0/1966551705 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d501a2dc0 0x7f1d501a3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:44:23.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.344+0000 7f1d46000640 1 --2- 172.21.15.43:0/1966551705 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1d5010cf50 0x7f1d5019ea90 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f1d3802c240 tx=0x7f1d38004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:23.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.344+0000 7f1d47e00640 1 -- 172.21.15.43:0/1966551705 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d38043020 con 0x7f1d5010cf50 2024-03-28T22:44:23.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.344+0000 7f1d4fe00640 1 -- 172.21.15.43:0/1966551705 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d501a3b40 con 0x7f1d5010cf50 2024-03-28T22:44:23.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.344+0000 7f1d47e00640 1 -- 172.21.15.43:0/1966551705 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d380054e0 con 0x7f1d5010cf50 2024-03-28T22:44:23.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.344+0000 7f1d47e00640 1 -- 172.21.15.43:0/1966551705 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d38003b80 con 0x7f1d5010cf50 2024-03-28T22:44:23.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.344+0000 7f1d4fe00640 1 -- 172.21.15.43:0/1966551705 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d501ab390 con 0x7f1d5010cf50 2024-03-28T22:44:23.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.347+0000 7f1d47e00640 1 -- 172.21.15.43:0/1966551705 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f1d38046020 con 0x7f1d5010cf50 2024-03-28T22:44:23.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.347+0000 7f1d585c8740 1 Processor -- start 2024-03-28T22:44:23.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.347+0000 7f1d585c8740 1 -- start start 2024-03-28T22:44:23.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.348+0000 7f1d585c8740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55711acfd550 0x55711acfd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.348+0000 7f1d585c8740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711acfde70 0x55711ad06510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.348+0000 7f1d585c8740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711ad06a50 0x55711ad08e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.348+0000 7f1d585c8740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55711ac61050 con 0x55711acfd550 2024-03-28T22:44:23.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.348+0000 7f1d585c8740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55711ac09ce0 con 0x55711ad06a50 2024-03-28T22:44:23.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.348+0000 7f1d585c8740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55711ac5d0f0 con 0x55711acfde70 2024-03-28T22:44:23.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.348+0000 7f1d4ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711acfde70 0x55711ad06510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.348+0000 7f1d4ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711acfde70 0x55711ad06510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:47916/0 (socket says 172.21.15.43:47916) 2024-03-28T22:44:23.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.348+0000 7f1d4ea00640 1 -- 172.21.15.43:0/3238759628 learned_addr learned my addr 172.21.15.43:0/3238759628 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:23.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.348+0000 7f1d4f400640 1 --2- 172.21.15.43:0/3238759628 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55711acfd550 0x55711acfd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.348+0000 7f1d46000640 1 --2- 172.21.15.43:0/3238759628 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711ad06a50 0x55711ad08e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.348+0000 7f1d4ea00640 1 -- 172.21.15.43:0/3238759628 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711ad06a50 msgr2=0x55711ad08e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:23.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.348+0000 7f1d4ea00640 1 --2- 172.21.15.43:0/3238759628 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711ad06a50 0x55711ad08e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.348+0000 7f1d4ea00640 1 -- 172.21.15.43:0/3238759628 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55711acfd550 msgr2=0x55711acfd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:23.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.348+0000 7f1d4ea00640 1 --2- 172.21.15.43:0/3238759628 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55711acfd550 0x55711acfd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.348+0000 7f1d4ea00640 1 -- 172.21.15.43:0/3238759628 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55711a9753f0 con 0x55711acfde70 2024-03-28T22:44:23.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.348+0000 7f1d46000640 1 --2- 172.21.15.43:0/3238759628 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711ad06a50 0x55711ad08e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:44:23.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.349+0000 7f1d47e00640 1 --2- 172.21.15.43:0/1966551705 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1d340a1f70 0x7f1d340a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.349+0000 7f1d4ea00640 1 --2- 172.21.15.43:0/3238759628 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711acfde70 0x55711ad06510 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f1d400114f0 tx=0x7f1d400119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b570678cf77d59d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:23.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.349+0000 7f1d47e00640 1 -- 172.21.15.43:0/1966551705 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f1d380f0550 con 0x7f1d5010cf50 2024-03-28T22:44:23.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.349+0000 7f1d4f400640 1 --2- 172.21.15.43:0/1966551705 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1d340a1f70 0x7f1d340a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.349+0000 7f1d3fe00640 1 -- 172.21.15.43:0/3238759628 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d4001e070 con 0x55711acfde70 2024-03-28T22:44:23.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.349+0000 7f1d3fe00640 1 -- 172.21.15.43:0/3238759628 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d4000fb00 con 0x55711acfde70 2024-03-28T22:44:23.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.349+0000 7f1d3fe00640 1 -- 172.21.15.43:0/3238759628 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d40019bf0 con 0x55711acfde70 2024-03-28T22:44:23.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.350+0000 7f1d585c8740 1 -- 172.21.15.43:0/3238759628 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711acfde70 msgr2=0x55711ad06510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.350+0000 7f1d585c8740 1 --2- 172.21.15.43:0/3238759628 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711acfde70 0x55711ad06510 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f1d400114f0 tx=0x7f1d400119b0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.350+0000 7f1d4f400640 1 --2- 172.21.15.43:0/1966551705 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1d340a1f70 0x7f1d340a4430 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1d48005fd0 tx=0x7f1d48005cc0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:23.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.350+0000 7f1d585c8740 1 -- 172.21.15.43:0/3238759628 shutdown_connections 2024-03-28T22:44:23.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.350+0000 7f1d585c8740 1 --2- 172.21.15.43:0/3238759628 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711ad06a50 0x55711ad08e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.350+0000 7f1d585c8740 1 --2- 172.21.15.43:0/3238759628 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711acfde70 0x55711ad06510 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.350+0000 7f1d585c8740 1 --2- 172.21.15.43:0/3238759628 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55711acfd550 0x55711acfd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.350+0000 7f1d585c8740 1 -- 172.21.15.43:0/3238759628 >> 172.21.15.43:0/3238759628 conn(0x55711acf7c80 msgr2=0x55711acf8060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:23.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.350+0000 7f1d585c8740 1 -- 172.21.15.43:0/3238759628 shutdown_connections 2024-03-28T22:44:23.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.350+0000 7f1d585c8740 1 -- 172.21.15.43:0/3238759628 wait complete. 2024-03-28T22:44:23.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.350+0000 7f1d585c8740 1 -- start start 2024-03-28T22:44:23.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.351+0000 7f1d585c8740 1 Processor -- start 2024-03-28T22:44:23.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.351+0000 7f1d585c8740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711acfd550 0x55711ad03a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.351+0000 7f1d585c8740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55711acfde70 0x55711ad03fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.351+0000 7f1d585c8740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711ad06a50 0x55711ad04510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.351+0000 7f1d585c8740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55711ad09840 con 0x55711acfde70 2024-03-28T22:44:23.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.351+0000 7f1d585c8740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55711abff2f0 con 0x55711acfd550 2024-03-28T22:44:23.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.351+0000 7f1d585c8740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55711ad099c0 con 0x55711ad06a50 2024-03-28T22:44:23.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.351+0000 7f1d585c8740 10 client.0 _ll_register_callbacks cb 0x55711a974be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:44:23.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.351+0000 7f1d4ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711acfd550 0x55711ad03a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.351+0000 7f1d46000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55711acfde70 0x55711ad03fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.351+0000 7f1d4ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711acfd550 0x55711ad03a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48544/0 (socket says 172.21.15.43:48544) 2024-03-28T22:44:23.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.351+0000 7f1d4ea00640 1 -- 172.21.15.43:0/1917973760 learned_addr learned my addr 172.21.15.43:0/1917973760 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:23.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.351+0000 7f1d4f400640 1 --2- 172.21.15.43:0/1917973760 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711ad06a50 0x55711ad04510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.352+0000 7f1d4ea00640 1 -- 172.21.15.43:0/1917973760 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711ad06a50 msgr2=0x55711ad04510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:23.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.352+0000 7f1d4ea00640 1 --2- 172.21.15.43:0/1917973760 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711ad06a50 0x55711ad04510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.352+0000 7f1d4ea00640 1 -- 172.21.15.43:0/1917973760 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55711acfde70 msgr2=0x55711ad03fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:23.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.352+0000 7f1d4ea00640 1 --2- 172.21.15.43:0/1917973760 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55711acfde70 0x55711ad03fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.352+0000 7f1d4ea00640 1 -- 172.21.15.43:0/1917973760 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55711aa3f7d0 con 0x55711acfd550 2024-03-28T22:44:23.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.352+0000 7f1d46000640 1 --2- 172.21.15.43:0/1917973760 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55711acfde70 0x55711ad03fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:44:23.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.352+0000 7f1d4ea00640 1 --2- 172.21.15.43:0/1917973760 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711acfd550 0x55711ad03a90 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f1d400099d0 tx=0x7f1d40019410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cd1bd63a88d574f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:23.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.352+0000 7f1d3cc00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:44:23.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.352+0000 7f1d3cc00640 1 -- 172.21.15.43:0/1917973760 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d4001e040 con 0x55711acfd550 2024-03-28T22:44:23.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.353+0000 7f1d3cc00640 1 -- 172.21.15.43:0/1917973760 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f1d40004a40 con 0x55711acfd550 2024-03-28T22:44:23.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.353+0000 7f1d585c8740 1 -- 172.21.15.43:0/1917973760 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d4000fb00 con 0x55711acfd550 2024-03-28T22:44:23.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.353+0000 7f1d3cc00640 1 -- 172.21.15.43:0/1917973760 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1d400194c0 con 0x55711acfd550 2024-03-28T22:44:23.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.353+0000 7f1d585c8740 10 client.28232 fetch_fsmap learned FSMap version 32 2024-03-28T22:44:23.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.353+0000 7f1d585c8740 1 -- 172.21.15.43:0/1917973760 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55711ad15b30 con 0x55711acfd550 2024-03-28T22:44:23.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.353+0000 7f1d3cc00640 1 -- 172.21.15.43:0/1917973760 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d4000fb00 con 0x55711acfd550 2024-03-28T22:44:23.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.353+0000 7f1d3cc00640 1 -- 172.21.15.43:0/1917973760 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f1d4002c030 con 0x55711acfd550 2024-03-28T22:44:23.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.353+0000 7f1d585c8740 10 client.28232 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:44:23.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.353+0000 7f1d585c8740 10 client.28232 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:44:23.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.353+0000 7f1d585c8740 20 client.28232 populate_metadata read hostname 'smithi043' 2024-03-28T22:44:23.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.353+0000 7f1d585c8740 1 --2- 172.21.15.43:0/1917973760 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55711ad16f90 0x55711ad19380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.353+0000 7f1d585c8740 4 client.28232 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:44:23.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.353+0000 7f1d585c8740 1 -- 172.21.15.43:0/1917973760 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55711ac0a680 con 0x55711ad16f90 2024-03-28T22:44:23.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.354+0000 7f1d46000640 1 --2- 172.21.15.43:0/1917973760 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55711ad16f90 0x55711ad19380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.354+0000 7f1d46000640 1 --2- 172.21.15.43:0/1917973760 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55711ad16f90 0x55711ad19380 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f4fc2662588b4d67 server_cookie=bc2628c669e82b10 in_seq=0 out_seq=0 2024-03-28T22:44:23.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.354+0000 7f1d3cc00640 10 client.28232 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:44:23.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.355+0000 7f1d3cc00640 1 -- 172.21.15.43:0/1917973760 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55711ac0a680 con 0x55711ad16f90 2024-03-28T22:44:23.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.355+0000 7f1d3cc00640 10 client.28232 handle_command_reply: tid=0 2024-03-28T22:44:23.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.355+0000 7f1d585c8740 1 client.28232 shutdown 2024-03-28T22:44:23.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.355+0000 7f1d585c8740 1 -- 172.21.15.43:0/1917973760 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711acfd550 msgr2=0x55711ad03a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.355+0000 7f1d585c8740 1 --2- 172.21.15.43:0/1917973760 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711acfd550 0x55711ad03a90 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f1d400099d0 tx=0x7f1d40019410 comp rx=0 tx=0).stop 2024-03-28T22:44:23.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.355+0000 7f1d585c8740 1 -- 172.21.15.43:0/1917973760 shutdown_connections 2024-03-28T22:44:23.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.355+0000 7f1d585c8740 1 --2- 172.21.15.43:0/1917973760 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55711ad16f90 0x55711ad19380 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.355+0000 7f1d585c8740 1 --2- 172.21.15.43:0/1917973760 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711ad06a50 0x55711ad04510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.355+0000 7f1d585c8740 1 --2- 172.21.15.43:0/1917973760 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55711acfde70 0x55711ad03fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.355+0000 7f1d585c8740 1 --2- 172.21.15.43:0/1917973760 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711acfd550 0x55711ad03a90 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.355+0000 7f1d3cc00640 0 client.28232 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:44:23.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.355+0000 7f1d585c8740 1 -- 172.21.15.43:0/1917973760 >> 172.21.15.43:0/1917973760 conn(0x55711acf7c80 msgr2=0x55711acf99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:23.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.355+0000 7f1d585c8740 1 -- 172.21.15.43:0/1917973760 shutdown_connections 2024-03-28T22:44:23.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.356+0000 7f1d585c8740 1 -- 172.21.15.43:0/1917973760 wait complete. 2024-03-28T22:44:23.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.356+0000 7f1d585c8740 20 client.28232 trim_cache size 0 max 16384 2024-03-28T22:44:23.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.373+0000 7f1d585c8740 1 Processor -- start 2024-03-28T22:44:23.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.373+0000 7f1d585c8740 1 -- start start 2024-03-28T22:44:23.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.373+0000 7f1d585c8740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55711acfd550 0x55711ad12610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.373+0000 7f1d585c8740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711acfde70 0x55711ad04330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.373+0000 7f1d585c8740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711ad06a50 0x55711ad1eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.373+0000 7f1d585c8740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1d501b5070 con 0x55711acfd550 2024-03-28T22:44:23.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.373+0000 7f1d585c8740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1d501b6bb0 con 0x55711acfde70 2024-03-28T22:44:23.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.373+0000 7f1d585c8740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f1d501b5740 con 0x55711ad06a50 2024-03-28T22:44:23.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.373+0000 7f1d46000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711acfde70 0x55711ad04330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.374+0000 7f1d46000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711acfde70 0x55711ad04330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48550/0 (socket says 172.21.15.43:48550) 2024-03-28T22:44:23.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.374+0000 7f1d46000640 1 -- 172.21.15.43:0/1468430559 learned_addr learned my addr 172.21.15.43:0/1468430559 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:23.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.374+0000 7f1d4f400640 1 --2- 172.21.15.43:0/1468430559 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711ad06a50 0x55711ad1eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.374+0000 7f1d4ea00640 1 --2- 172.21.15.43:0/1468430559 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55711acfd550 0x55711ad12610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.374+0000 7f1d46000640 1 -- 172.21.15.43:0/1468430559 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711ad06a50 msgr2=0x55711ad1eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:23.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.374+0000 7f1d46000640 1 --2- 172.21.15.43:0/1468430559 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711ad06a50 0x55711ad1eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.374+0000 7f1d46000640 1 -- 172.21.15.43:0/1468430559 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55711acfd550 msgr2=0x55711ad12610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:23.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.374+0000 7f1d46000640 1 --2- 172.21.15.43:0/1468430559 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55711acfd550 0x55711ad12610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.374+0000 7f1d46000640 1 -- 172.21.15.43:0/1468430559 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55711acfdca0 con 0x55711acfde70 2024-03-28T22:44:23.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.374+0000 7f1d46000640 1 --2- 172.21.15.43:0/1468430559 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711acfde70 0x55711ad04330 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f1d380bc920 tx=0x7f1d38004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60df8b38e4ddd371 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:23.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.374+0000 7f1d3f400640 1 -- 172.21.15.43:0/1468430559 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d38004c80 con 0x55711acfde70 2024-03-28T22:44:23.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.374+0000 7f1d3f400640 1 -- 172.21.15.43:0/1468430559 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d38005690 con 0x55711acfde70 2024-03-28T22:44:23.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.374+0000 7f1d3f400640 1 -- 172.21.15.43:0/1468430559 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d380f3cf0 con 0x55711acfde70 2024-03-28T22:44:23.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.375+0000 7f1d585c8740 1 -- 172.21.15.43:0/1468430559 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711acfde70 msgr2=0x55711ad04330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.375+0000 7f1d585c8740 1 --2- 172.21.15.43:0/1468430559 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711acfde70 0x55711ad04330 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f1d380bc920 tx=0x7f1d38004610 comp rx=0 tx=0).stop 2024-03-28T22:44:23.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.375+0000 7f1d585c8740 1 -- 172.21.15.43:0/1468430559 shutdown_connections 2024-03-28T22:44:23.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.375+0000 7f1d585c8740 1 --2- 172.21.15.43:0/1468430559 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711acfde70 0x55711ad04330 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.375+0000 7f1d585c8740 1 --2- 172.21.15.43:0/1468430559 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711ad06a50 0x55711ad1eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.375+0000 7f1d585c8740 1 --2- 172.21.15.43:0/1468430559 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55711acfd550 0x55711ad12610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.375+0000 7f1d585c8740 1 -- 172.21.15.43:0/1468430559 >> 172.21.15.43:0/1468430559 conn(0x55711acf7c80 msgr2=0x55711acf6dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:23.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.375+0000 7f1d585c8740 1 -- 172.21.15.43:0/1468430559 shutdown_connections 2024-03-28T22:44:23.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.375+0000 7f1d585c8740 1 -- 172.21.15.43:0/1468430559 wait complete. 2024-03-28T22:44:23.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.376+0000 7f1d585c8740 1 -- start start 2024-03-28T22:44:23.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.376+0000 7f1d585c8740 1 Processor -- start 2024-03-28T22:44:23.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.376+0000 7f1d585c8740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55711acfd550 0x55711ad0eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.376+0000 7f1d585c8740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711acfde70 0x55711ad00300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.376+0000 7f1d585c8740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711ad06a50 0x55711ad00840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.376+0000 7f1d585c8740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1d501b64e0 con 0x55711acfd550 2024-03-28T22:44:23.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.376+0000 7f1d585c8740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1d501b5e10 con 0x55711ad06a50 2024-03-28T22:44:23.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.376+0000 7f1d585c8740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f1d501b3070 con 0x55711acfde70 2024-03-28T22:44:23.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.376+0000 7f1d585c8740 10 client.0 _ll_register_callbacks cb 0x7f1d4001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:44:23.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.377+0000 7f1d46000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711acfde70 0x55711ad00300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.377+0000 7f1d46000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711acfde70 0x55711ad00300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:47932/0 (socket says 172.21.15.43:47932) 2024-03-28T22:44:23.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.377+0000 7f1d46000640 1 -- 172.21.15.43:0/1699295309 learned_addr learned my addr 172.21.15.43:0/1699295309 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:23.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.377+0000 7f1d4ea00640 1 --2- 172.21.15.43:0/1699295309 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55711acfd550 0x55711ad0eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.377+0000 7f1d4f400640 1 --2- 172.21.15.43:0/1699295309 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711ad06a50 0x55711ad00840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.377+0000 7f1d46000640 1 -- 172.21.15.43:0/1699295309 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711ad06a50 msgr2=0x55711ad00840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:23.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.377+0000 7f1d46000640 1 --2- 172.21.15.43:0/1699295309 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711ad06a50 0x55711ad00840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.377+0000 7f1d46000640 1 -- 172.21.15.43:0/1699295309 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55711acfd550 msgr2=0x55711ad0eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:23.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.377+0000 7f1d46000640 1 --2- 172.21.15.43:0/1699295309 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55711acfd550 0x55711ad0eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.377+0000 7f1d46000640 1 -- 172.21.15.43:0/1699295309 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55711ad01070 con 0x55711acfde70 2024-03-28T22:44:23.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.377+0000 7f1d46000640 1 --2- 172.21.15.43:0/1699295309 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711acfde70 0x55711ad00300 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f1d3802c210 tx=0x7f1d380035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=993bc0905ed203fe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:23.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.377+0000 7f1d3cc00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:44:23.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.377+0000 7f1d3cc00640 1 -- 172.21.15.43:0/1699295309 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d38004c80 con 0x55711acfde70 2024-03-28T22:44:23.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.378+0000 7f1d3cc00640 1 -- 172.21.15.43:0/1699295309 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f1d380d5690 con 0x55711acfde70 2024-03-28T22:44:23.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.378+0000 7f1d585c8740 1 -- 172.21.15.43:0/1699295309 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55711ac6e590 con 0x55711acfde70 2024-03-28T22:44:23.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.378+0000 7f1d3cc00640 1 -- 172.21.15.43:0/1699295309 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1d380c0d70 con 0x55711acfde70 2024-03-28T22:44:23.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.378+0000 7f1d585c8740 10 client.28177 fetch_fsmap learned FSMap version 32 2024-03-28T22:44:23.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.378+0000 7f1d585c8740 1 -- 172.21.15.43:0/1699295309 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55711ad7d150 con 0x55711acfde70 2024-03-28T22:44:23.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.378+0000 7f1d3cc00640 1 -- 172.21.15.43:0/1699295309 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55711ac6e590 con 0x55711acfde70 2024-03-28T22:44:23.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.378+0000 7f1d3cc00640 1 -- 172.21.15.43:0/1699295309 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f1d380edaf0 con 0x55711acfde70 2024-03-28T22:44:23.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.378+0000 7f1d585c8740 10 client.28177 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:44:23.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.378+0000 7f1d585c8740 10 client.28177 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:44:23.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.378+0000 7f1d585c8740 20 client.28177 populate_metadata read hostname 'smithi043' 2024-03-28T22:44:23.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.378+0000 7f1d585c8740 1 --2- 172.21.15.43:0/1699295309 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55711ad7d810 0x55711ad7fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.378+0000 7f1d585c8740 4 client.28177 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:44:23.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.378+0000 7f1d585c8740 1 -- 172.21.15.43:0/1699295309 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55711ac688e0 con 0x55711ad7d810 2024-03-28T22:44:23.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.379+0000 7f1d4ea00640 1 --2- 172.21.15.43:0/1699295309 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55711ad7d810 0x55711ad7fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.379+0000 7f1d4ea00640 1 --2- 172.21.15.43:0/1699295309 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55711ad7d810 0x55711ad7fc30 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=83e963258f352c31 server_cookie=86acbcdbd2953fa2 in_seq=0 out_seq=0 2024-03-28T22:44:23.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.379+0000 7f1d3cc00640 10 client.28177 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:44:23.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.379+0000 7f1d3cc00640 1 -- 172.21.15.43:0/1699295309 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55711ac688e0 con 0x55711ad7d810 2024-03-28T22:44:23.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.379+0000 7f1d3cc00640 10 client.28177 handle_command_reply: tid=0 2024-03-28T22:44:23.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.379+0000 7f1d585c8740 1 client.28177 shutdown 2024-03-28T22:44:23.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.379+0000 7f1d585c8740 1 -- 172.21.15.43:0/1699295309 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711acfde70 msgr2=0x55711ad00300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.379+0000 7f1d585c8740 1 --2- 172.21.15.43:0/1699295309 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711acfde70 0x55711ad00300 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f1d3802c210 tx=0x7f1d380035f0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.380+0000 7f1d585c8740 1 -- 172.21.15.43:0/1699295309 shutdown_connections 2024-03-28T22:44:23.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.380+0000 7f1d585c8740 1 --2- 172.21.15.43:0/1699295309 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55711ad7d810 0x55711ad7fc30 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.380+0000 7f1d585c8740 1 --2- 172.21.15.43:0/1699295309 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55711ad06a50 0x55711ad00840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.380+0000 7f1d585c8740 1 --2- 172.21.15.43:0/1699295309 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55711acfde70 0x55711ad00300 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.380+0000 7f1d585c8740 1 --2- 172.21.15.43:0/1699295309 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55711acfd550 0x55711ad0eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.380+0000 7f1d585c8740 1 -- 172.21.15.43:0/1699295309 >> 172.21.15.43:0/1699295309 conn(0x55711acf7c80 msgr2=0x55711acf9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:23.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.380+0000 7f1d3cc00640 0 client.28177 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:44:23.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.380+0000 7f1d585c8740 1 -- 172.21.15.43:0/1699295309 shutdown_connections 2024-03-28T22:44:23.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.380+0000 7f1d585c8740 1 -- 172.21.15.43:0/1699295309 wait complete. 2024-03-28T22:44:23.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.380+0000 7f1d585c8740 20 client.28177 trim_cache size 0 max 16384 2024-03-28T22:44:23.386 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:44:23.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.381+0000 7f1d4fe00640 1 -- 172.21.15.43:0/1966551705 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1d340a1f70 msgr2=0x7f1d340a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.381+0000 7f1d4fe00640 1 --2- 172.21.15.43:0/1966551705 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1d340a1f70 0x7f1d340a4430 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1d48005fd0 tx=0x7f1d48005cc0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.381+0000 7f1d4fe00640 1 -- 172.21.15.43:0/1966551705 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1d5010cf50 msgr2=0x7f1d5019ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.381+0000 7f1d4fe00640 1 --2- 172.21.15.43:0/1966551705 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1d5010cf50 0x7f1d5019ea90 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f1d3802c240 tx=0x7f1d38004360 comp rx=0 tx=0).stop 2024-03-28T22:44:23.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.381+0000 7f1d4fe00640 1 -- 172.21.15.43:0/1966551705 shutdown_connections 2024-03-28T22:44:23.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.381+0000 7f1d4fe00640 1 --2- 172.21.15.43:0/1966551705 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1d340a1f70 0x7f1d340a4430 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.381+0000 7f1d4fe00640 1 --2- 172.21.15.43:0/1966551705 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1d5010cf50 0x7f1d5019ea90 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.381+0000 7f1d4fe00640 1 --2- 172.21.15.43:0/1966551705 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1d501a2dc0 0x7f1d501a3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.381+0000 7f1d4fe00640 1 --2- 172.21.15.43:0/1966551705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d50107f00 0x7f1d5019e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.381+0000 7f1d4fe00640 1 -- 172.21.15.43:0/1966551705 >> 172.21.15.43:0/1966551705 conn(0x7f1d50077570 msgr2=0x7f1d500791c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:23.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.381+0000 7f1d4fe00640 1 -- 172.21.15.43:0/1966551705 shutdown_connections 2024-03-28T22:44:23.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.381+0000 7f1d4fe00640 1 -- 172.21.15.43:0/1966551705 wait complete. 2024-03-28T22:44:23.396 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:44:23.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.647+0000 7f0252a00640 1 -- 172.21.15.43:0/3167572574 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f024c1037d0 msgr2=0x7f024c103c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.647+0000 7f0252a00640 1 --2- 172.21.15.43:0/3167572574 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f024c1037d0 0x7f024c103c50 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f0238009900 tx=0x7f023802bc30 comp rx=0 tx=0).stop 2024-03-28T22:44:23.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.647+0000 7f0252a00640 1 -- 172.21.15.43:0/3167572574 shutdown_connections 2024-03-28T22:44:23.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.647+0000 7f0252a00640 1 --2- 172.21.15.43:0/3167572574 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f024c1037d0 0x7f024c103c50 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.647+0000 7f0252a00640 1 --2- 172.21.15.43:0/3167572574 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f024c104190 0x7f024c1106a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.647+0000 7f0252a00640 1 --2- 172.21.15.43:0/3167572574 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f024c1097e0 0x7f024c109bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.647+0000 7f0252a00640 1 -- 172.21.15.43:0/3167572574 >> 172.21.15.43:0/3167572574 conn(0x7f024c0ff260 msgr2=0x7f024c101680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:23.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.648+0000 7f0252a00640 1 -- 172.21.15.43:0/3167572574 shutdown_connections 2024-03-28T22:44:23.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.648+0000 7f0252a00640 1 -- 172.21.15.43:0/3167572574 wait complete. 2024-03-28T22:44:23.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.648+0000 7f0252a00640 1 Processor -- start 2024-03-28T22:44:23.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.649+0000 7f0252a00640 1 -- start start 2024-03-28T22:44:23.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.649+0000 7f0252a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f024c1037d0 0x7f024c19a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.649+0000 7f0252a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f024c104190 0x7f024c19a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.649+0000 7f0252a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f024c1097e0 0x7f024c19e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.649+0000 7f0252a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f024c113c40 con 0x7f024c104190 2024-03-28T22:44:23.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.649+0000 7f0252a00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f024c113ac0 con 0x7f024c1037d0 2024-03-28T22:44:23.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.649+0000 7f0252a00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f024c113dc0 con 0x7f024c1097e0 2024-03-28T22:44:23.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.649+0000 7f0251600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f024c1037d0 0x7f024c19a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.649+0000 7f0251600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f024c1037d0 0x7f024c19a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48568/0 (socket says 172.21.15.43:48568) 2024-03-28T22:44:23.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.649+0000 7f0251600640 1 -- 172.21.15.43:0/2762000835 learned_addr learned my addr 172.21.15.43:0/2762000835 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:23.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.649+0000 7f0250c00640 1 --2- 172.21.15.43:0/2762000835 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f024c104190 0x7f024c19a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.649+0000 7f0252000640 1 --2- 172.21.15.43:0/2762000835 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f024c1097e0 0x7f024c19e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.650+0000 7f0251600640 1 -- 172.21.15.43:0/2762000835 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f024c1097e0 msgr2=0x7f024c19e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.650+0000 7f0251600640 1 --2- 172.21.15.43:0/2762000835 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f024c1097e0 0x7f024c19e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.650+0000 7f0251600640 1 -- 172.21.15.43:0/2762000835 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f024c104190 msgr2=0x7f024c19a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.650+0000 7f0251600640 1 --2- 172.21.15.43:0/2762000835 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f024c104190 0x7f024c19a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.650+0000 7f0251600640 1 -- 172.21.15.43:0/2762000835 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f024c19f0a0 con 0x7f024c1037d0 2024-03-28T22:44:23.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.650+0000 7f0251600640 1 --2- 172.21.15.43:0/2762000835 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f024c1037d0 0x7f024c19a100 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f024000e9f0 tx=0x7f024000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:23.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.650+0000 7f024a000640 1 -- 172.21.15.43:0/2762000835 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f024000cd80 con 0x7f024c1037d0 2024-03-28T22:44:23.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.650+0000 7f024a000640 1 -- 172.21.15.43:0/2762000835 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0240005500 con 0x7f024c1037d0 2024-03-28T22:44:23.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.650+0000 7f0252a00640 1 -- 172.21.15.43:0/2762000835 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f024c19f390 con 0x7f024c1037d0 2024-03-28T22:44:23.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.650+0000 7f024a000640 1 -- 172.21.15.43:0/2762000835 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0240010690 con 0x7f024c1037d0 2024-03-28T22:44:23.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.650+0000 7f0252a00640 1 -- 172.21.15.43:0/2762000835 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f024c1a6c20 con 0x7f024c1037d0 2024-03-28T22:44:23.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.653+0000 7f024a000640 1 -- 172.21.15.43:0/2762000835 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f02400046c0 con 0x7f024c1037d0 2024-03-28T22:44:23.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.654+0000 7f0256e1b740 1 Processor -- start 2024-03-28T22:44:23.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.654+0000 7f0256e1b740 1 -- start start 2024-03-28T22:44:23.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.654+0000 7f0256e1b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563e89a07550 0x563e89a07930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.654+0000 7f0256e1b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e89a07e70 0x563e89a10510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.654+0000 7f0256e1b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563e89a10a50 0x563e89a12e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.654+0000 7f0256e1b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x563e8996b050 con 0x563e89a07e70 2024-03-28T22:44:23.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.654+0000 7f0256e1b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x563e89913ce0 con 0x563e89a07550 2024-03-28T22:44:23.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.654+0000 7f0256e1b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x563e899670f0 con 0x563e89a10a50 2024-03-28T22:44:23.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.654+0000 7f0252000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563e89a07550 0x563e89a07930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.654+0000 7f0251600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e89a07e70 0x563e89a10510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.654+0000 7f0250c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563e89a10a50 0x563e89a12e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.654+0000 7f0252000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563e89a07550 0x563e89a07930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48584/0 (socket says 172.21.15.43:48584) 2024-03-28T22:44:23.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.654+0000 7f0252000640 1 -- 172.21.15.43:0/1348338831 learned_addr learned my addr 172.21.15.43:0/1348338831 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:23.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.655+0000 7f0252000640 1 -- 172.21.15.43:0/1348338831 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563e89a10a50 msgr2=0x563e89a12e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:23.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.655+0000 7f0252000640 1 --2- 172.21.15.43:0/1348338831 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563e89a10a50 0x563e89a12e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.655+0000 7f0252000640 1 -- 172.21.15.43:0/1348338831 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e89a07e70 msgr2=0x563e89a10510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:23.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.655+0000 7f0252000640 1 --2- 172.21.15.43:0/1348338831 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e89a07e70 0x563e89a10510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.655+0000 7f0252000640 1 -- 172.21.15.43:0/1348338831 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e8967f3f0 con 0x563e89a07550 2024-03-28T22:44:23.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.655+0000 7f0252000640 1 --2- 172.21.15.43:0/1348338831 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563e89a07550 0x563e89a07930 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f0244009fd0 tx=0x7f02440118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f22f812dd0d9abdb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:23.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.655+0000 7f023e600640 1 -- 172.21.15.43:0/1348338831 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0244004d20 con 0x563e89a07550 2024-03-28T22:44:23.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.655+0000 7f023e600640 1 -- 172.21.15.43:0/1348338831 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0244004050 con 0x563e89a07550 2024-03-28T22:44:23.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.655+0000 7f024a000640 1 --2- 172.21.15.43:0/2762000835 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f02340a1ef0 0x7f02340a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.655+0000 7f023e600640 1 -- 172.21.15.43:0/1348338831 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0244005260 con 0x563e89a07550 2024-03-28T22:44:23.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.656+0000 7f024a000640 1 -- 172.21.15.43:0/2762000835 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f02400ce990 con 0x7f024c1037d0 2024-03-28T22:44:23.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.656+0000 7f0251600640 1 --2- 172.21.15.43:0/2762000835 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f02340a1ef0 0x7f02340a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.656+0000 7f0256e1b740 1 -- 172.21.15.43:0/1348338831 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563e89a07550 msgr2=0x563e89a07930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.656+0000 7f0256e1b740 1 --2- 172.21.15.43:0/1348338831 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563e89a07550 0x563e89a07930 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f0244009fd0 tx=0x7f02440118d0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.656+0000 7f0256e1b740 1 -- 172.21.15.43:0/1348338831 shutdown_connections 2024-03-28T22:44:23.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.656+0000 7f0256e1b740 1 --2- 172.21.15.43:0/1348338831 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563e89a10a50 0x563e89a12e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.656+0000 7f0256e1b740 1 --2- 172.21.15.43:0/1348338831 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e89a07e70 0x563e89a10510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.656+0000 7f0256e1b740 1 --2- 172.21.15.43:0/1348338831 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563e89a07550 0x563e89a07930 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.656+0000 7f0256e1b740 1 -- 172.21.15.43:0/1348338831 >> 172.21.15.43:0/1348338831 conn(0x563e89a01c80 msgr2=0x563e89a02060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:23.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.656+0000 7f0251600640 1 --2- 172.21.15.43:0/2762000835 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f02340a1ef0 0x7f02340a43b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x563e89a07020 tx=0x7f02400d4530 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:23.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.656+0000 7f0256e1b740 1 -- 172.21.15.43:0/1348338831 shutdown_connections 2024-03-28T22:44:23.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.657+0000 7f0256e1b740 1 -- 172.21.15.43:0/1348338831 wait complete. 2024-03-28T22:44:23.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.657+0000 7f0256e1b740 1 -- start start 2024-03-28T22:44:23.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.657+0000 7f0256e1b740 1 Processor -- start 2024-03-28T22:44:23.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.658+0000 7f0256e1b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563e89a07550 0x563e89a0daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.658+0000 7f0256e1b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e89a07e70 0x563e89a0e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.658+0000 7f0256e1b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563e89a10a50 0x563e89a0e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.658+0000 7f0256e1b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x563e89a13840 con 0x563e89a07e70 2024-03-28T22:44:23.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.658+0000 7f0256e1b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x563e899092f0 con 0x563e89a07550 2024-03-28T22:44:23.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.658+0000 7f0256e1b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x563e89a139c0 con 0x563e89a10a50 2024-03-28T22:44:23.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.658+0000 7f0256e1b740 10 client.0 _ll_register_callbacks cb 0x563e8967ebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:44:23.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.658+0000 7f0252000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563e89a07550 0x563e89a0daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.658+0000 7f0252000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563e89a07550 0x563e89a0daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48594/0 (socket says 172.21.15.43:48594) 2024-03-28T22:44:23.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.658+0000 7f0252000640 1 -- 172.21.15.43:0/3497568806 learned_addr learned my addr 172.21.15.43:0/3497568806 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:23.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.658+0000 7f0252000640 1 --2- 172.21.15.43:0/3497568806 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563e89a10a50 0x563e89a0e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.658+0000 7f0250c00640 1 --2- 172.21.15.43:0/3497568806 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e89a07e70 0x563e89a0e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.658+0000 7f0252000640 1 -- 172.21.15.43:0/3497568806 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563e89a10a50 msgr2=0x563e89a0e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:23.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.658+0000 7f0252000640 1 --2- 172.21.15.43:0/3497568806 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563e89a10a50 0x563e89a0e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.658+0000 7f0252000640 1 -- 172.21.15.43:0/3497568806 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e89a07e70 msgr2=0x563e89a0e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:23.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.658+0000 7f0252000640 1 --2- 172.21.15.43:0/3497568806 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e89a07e70 0x563e89a0e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.658+0000 7f0252000640 1 -- 172.21.15.43:0/3497568806 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e897497d0 con 0x563e89a07550 2024-03-28T22:44:23.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.659+0000 7f0252000640 1 --2- 172.21.15.43:0/3497568806 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563e89a07550 0x563e89a0daf0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f02440094d0 tx=0x7f0244005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3b93ae5fcb22226 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:23.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.659+0000 7f0232a00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:44:23.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.659+0000 7f0232a00640 1 -- 172.21.15.43:0/3497568806 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0244004770 con 0x563e89a07550 2024-03-28T22:44:23.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.659+0000 7f0232a00640 1 -- 172.21.15.43:0/3497568806 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f024400f4e0 con 0x563e89a07550 2024-03-28T22:44:23.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.659+0000 7f0256e1b740 1 -- 172.21.15.43:0/3497568806 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0244004050 con 0x563e89a07550 2024-03-28T22:44:23.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.659+0000 7f0232a00640 1 -- 172.21.15.43:0/3497568806 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0244036050 con 0x563e89a07550 2024-03-28T22:44:23.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.659+0000 7f0256e1b740 10 client.28250 fetch_fsmap learned FSMap version 32 2024-03-28T22:44:23.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.659+0000 7f0256e1b740 1 -- 172.21.15.43:0/3497568806 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563e89a1fb40 con 0x563e89a07550 2024-03-28T22:44:23.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.660+0000 7f0232a00640 1 -- 172.21.15.43:0/3497568806 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0244004050 con 0x563e89a07550 2024-03-28T22:44:23.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.660+0000 7f0232a00640 1 -- 172.21.15.43:0/3497568806 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f0244025040 con 0x563e89a07550 2024-03-28T22:44:23.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.660+0000 7f0256e1b740 10 client.28250 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:44:23.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.660+0000 7f0256e1b740 10 client.28250 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:44:23.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.660+0000 7f0256e1b740 20 client.28250 populate_metadata read hostname 'smithi043' 2024-03-28T22:44:23.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.660+0000 7f0256e1b740 1 --2- 172.21.15.43:0/3497568806 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x563e89a20f40 0x563e89a23330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.660+0000 7f0256e1b740 4 client.28250 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:44:23.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.660+0000 7f0256e1b740 1 -- 172.21.15.43:0/3497568806 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e89914680 con 0x563e89a20f40 2024-03-28T22:44:23.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.661+0000 7f0250c00640 1 --2- 172.21.15.43:0/3497568806 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x563e89a20f40 0x563e89a23330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.661+0000 7f0250c00640 1 --2- 172.21.15.43:0/3497568806 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x563e89a20f40 0x563e89a23330 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9d66ab4709554485 server_cookie=39e0a4fdaf5e6a17 in_seq=0 out_seq=0 2024-03-28T22:44:23.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.661+0000 7f0232a00640 10 client.28250 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:44:23.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.662+0000 7f0232a00640 1 -- 172.21.15.43:0/3497568806 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563e89914680 con 0x563e89a20f40 2024-03-28T22:44:23.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.662+0000 7f0232a00640 10 client.28250 handle_command_reply: tid=0 2024-03-28T22:44:23.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.662+0000 7f0256e1b740 1 client.28250 shutdown 2024-03-28T22:44:23.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.663+0000 7f0256e1b740 1 -- 172.21.15.43:0/3497568806 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563e89a07550 msgr2=0x563e89a0daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.663+0000 7f0256e1b740 1 --2- 172.21.15.43:0/3497568806 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563e89a07550 0x563e89a0daf0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f02440094d0 tx=0x7f0244005bd0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.663+0000 7f0256e1b740 1 -- 172.21.15.43:0/3497568806 shutdown_connections 2024-03-28T22:44:23.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.663+0000 7f0256e1b740 1 --2- 172.21.15.43:0/3497568806 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563e89a10a50 0x563e89a0e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.663+0000 7f0256e1b740 1 --2- 172.21.15.43:0/3497568806 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e89a07e70 0x563e89a0e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.663+0000 7f0256e1b740 1 --2- 172.21.15.43:0/3497568806 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x563e89a20f40 0x563e89a23330 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.663+0000 7f0256e1b740 1 --2- 172.21.15.43:0/3497568806 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563e89a07550 0x563e89a0daf0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.663+0000 7f0256e1b740 1 -- 172.21.15.43:0/3497568806 >> 172.21.15.43:0/3497568806 conn(0x563e89a01c80 msgr2=0x563e89a039d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:23.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.663+0000 7f0232a00640 0 client.28250 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:44:23.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.663+0000 7f0256e1b740 1 -- 172.21.15.43:0/3497568806 shutdown_connections 2024-03-28T22:44:23.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.663+0000 7f0256e1b740 1 -- 172.21.15.43:0/3497568806 wait complete. 2024-03-28T22:44:23.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.663+0000 7f0256e1b740 20 client.28250 trim_cache size 0 max 16384 2024-03-28T22:44:23.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.678+0000 7f0256e1b740 1 Processor -- start 2024-03-28T22:44:23.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.678+0000 7f0256e1b740 1 -- start start 2024-03-28T22:44:23.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.678+0000 7f0256e1b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e89a07550 0x563e89a01000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.678+0000 7f0256e1b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563e89a07e70 0x563e89a0d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.678+0000 7f0256e1b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563e89a10a50 0x563e89a1c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.678+0000 7f0256e1b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f024c1b0d10 con 0x563e89a07550 2024-03-28T22:44:23.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.678+0000 7f0256e1b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f024c1b2850 con 0x563e89a07e70 2024-03-28T22:44:23.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.679+0000 7f0256e1b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f024c1b13e0 con 0x563e89a10a50 2024-03-28T22:44:23.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.679+0000 7f0250c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563e89a07e70 0x563e89a0d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.679+0000 7f0252000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e89a07550 0x563e89a01000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.679+0000 7f0250c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563e89a07e70 0x563e89a0d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48598/0 (socket says 172.21.15.43:48598) 2024-03-28T22:44:23.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.679+0000 7f0250c00640 1 -- 172.21.15.43:0/1226555949 learned_addr learned my addr 172.21.15.43:0/1226555949 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:23.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.679+0000 7f0252000640 1 --2- 172.21.15.43:0/1226555949 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563e89a10a50 0x563e89a1c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.679+0000 7f0250c00640 1 -- 172.21.15.43:0/1226555949 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563e89a10a50 msgr2=0x563e89a1c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:23.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.679+0000 7f0250c00640 1 --2- 172.21.15.43:0/1226555949 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563e89a10a50 0x563e89a1c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.679+0000 7f0250c00640 1 -- 172.21.15.43:0/1226555949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e89a07550 msgr2=0x563e89a01000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:23.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.679+0000 7f0250c00640 1 --2- 172.21.15.43:0/1226555949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e89a07550 0x563e89a01000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.679+0000 7f0250c00640 1 -- 172.21.15.43:0/1226555949 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e89a13720 con 0x563e89a07e70 2024-03-28T22:44:23.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.679+0000 7f0252000640 1 --2- 172.21.15.43:0/1226555949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e89a07550 0x563e89a01000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:44:23.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.679+0000 7f0250c00640 1 --2- 172.21.15.43:0/1226555949 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563e89a07e70 0x563e89a0d3d0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f0238009900 tx=0x7f0238000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b362b3745c5f425f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:23.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.680+0000 7f023dc00640 1 -- 172.21.15.43:0/1226555949 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0238050070 con 0x563e89a07e70 2024-03-28T22:44:23.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.680+0000 7f023dc00640 1 -- 172.21.15.43:0/1226555949 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f023804a620 con 0x563e89a07e70 2024-03-28T22:44:23.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.680+0000 7f023dc00640 1 -- 172.21.15.43:0/1226555949 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0238004920 con 0x563e89a07e70 2024-03-28T22:44:23.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.680+0000 7f0256e1b740 1 -- 172.21.15.43:0/1226555949 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563e89a07e70 msgr2=0x563e89a0d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.680+0000 7f0256e1b740 1 --2- 172.21.15.43:0/1226555949 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563e89a07e70 0x563e89a0d3d0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f0238009900 tx=0x7f0238000bc0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.680+0000 7f0256e1b740 1 -- 172.21.15.43:0/1226555949 shutdown_connections 2024-03-28T22:44:23.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.680+0000 7f0256e1b740 1 --2- 172.21.15.43:0/1226555949 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563e89a07e70 0x563e89a0d3d0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.680+0000 7f0256e1b740 1 --2- 172.21.15.43:0/1226555949 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563e89a10a50 0x563e89a1c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.680+0000 7f0256e1b740 1 --2- 172.21.15.43:0/1226555949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e89a07550 0x563e89a01000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.680+0000 7f0256e1b740 1 -- 172.21.15.43:0/1226555949 >> 172.21.15.43:0/1226555949 conn(0x563e89a01c80 msgr2=0x563e89a0e340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:23.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.680+0000 7f0256e1b740 1 -- 172.21.15.43:0/1226555949 shutdown_connections 2024-03-28T22:44:23.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.680+0000 7f0256e1b740 1 -- 172.21.15.43:0/1226555949 wait complete. 2024-03-28T22:44:23.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.681+0000 7f0256e1b740 1 -- start start 2024-03-28T22:44:23.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.681+0000 7f0256e1b740 1 Processor -- start 2024-03-28T22:44:23.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.681+0000 7f0256e1b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563e89a07550 0x563e89a18c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.682+0000 7f0256e1b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e89a07e70 0x563e89a13fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.682+0000 7f0256e1b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563e89a10a50 0x563e89a14520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.682+0000 7f0256e1b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f024c1b2180 con 0x563e89a07e70 2024-03-28T22:44:23.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.682+0000 7f0256e1b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f024c1b1ab0 con 0x563e89a07550 2024-03-28T22:44:23.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.682+0000 7f0256e1b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f024c1aed10 con 0x563e89a10a50 2024-03-28T22:44:23.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.682+0000 7f0256e1b740 10 client.0 _ll_register_callbacks cb 0x7f02400d4400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:44:23.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.682+0000 7f0250c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e89a07e70 0x563e89a13fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.682+0000 7f0252000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563e89a07550 0x563e89a18c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.682+0000 7f0250c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e89a07e70 0x563e89a13fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44354/0 (socket says 172.21.15.43:44354) 2024-03-28T22:44:23.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.682+0000 7f0250c00640 1 -- 172.21.15.43:0/3747664966 learned_addr learned my addr 172.21.15.43:0/3747664966 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:23.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.682+0000 7f0252000640 1 --2- 172.21.15.43:0/3747664966 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563e89a10a50 0x563e89a14520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.682+0000 7f0250c00640 1 -- 172.21.15.43:0/3747664966 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563e89a10a50 msgr2=0x563e89a14520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:23.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.682+0000 7f0250c00640 1 --2- 172.21.15.43:0/3747664966 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563e89a10a50 0x563e89a14520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.682+0000 7f0250c00640 1 -- 172.21.15.43:0/3747664966 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563e89a07550 msgr2=0x563e89a18c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:23.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.682+0000 7f0250c00640 1 --2- 172.21.15.43:0/3747664966 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563e89a07550 0x563e89a18c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.682+0000 7f0250c00640 1 -- 172.21.15.43:0/3747664966 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e89a14a60 con 0x563e89a07e70 2024-03-28T22:44:23.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.682+0000 7f0252000640 1 --2- 172.21.15.43:0/3747664966 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563e89a10a50 0x563e89a14520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:44:23.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.682+0000 7f0250c00640 1 --2- 172.21.15.43:0/3747664966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e89a07e70 0x563e89a13fe0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0238036040 tx=0x7f0238005900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc0b059dff1247bf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:23.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.682+0000 7f0232a00640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:44:23.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.683+0000 7f0232a00640 1 -- 172.21.15.43:0/3747664966 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f023805b060 con 0x563e89a07e70 2024-03-28T22:44:23.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.683+0000 7f0232a00640 1 -- 172.21.15.43:0/3747664966 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f023802cc40 con 0x563e89a07e70 2024-03-28T22:44:23.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.683+0000 7f0256e1b740 1 -- 172.21.15.43:0/3747664966 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f023804a620 con 0x563e89a07e70 2024-03-28T22:44:23.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.683+0000 7f0232a00640 1 -- 172.21.15.43:0/3747664966 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0238062050 con 0x563e89a07e70 2024-03-28T22:44:23.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.683+0000 7f0256e1b740 10 client.18003 fetch_fsmap learned FSMap version 32 2024-03-28T22:44:23.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.683+0000 7f0256e1b740 1 -- 172.21.15.43:0/3747664966 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563e89a86e00 con 0x563e89a07e70 2024-03-28T22:44:23.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.683+0000 7f0232a00640 1 -- 172.21.15.43:0/3747664966 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f023804a620 con 0x563e89a07e70 2024-03-28T22:44:23.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.684+0000 7f0232a00640 1 -- 172.21.15.43:0/3747664966 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f0238047070 con 0x563e89a07e70 2024-03-28T22:44:23.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.684+0000 7f0256e1b740 10 client.18003 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:44:23.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.684+0000 7f0256e1b740 10 client.18003 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:44:23.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.684+0000 7f0256e1b740 20 client.18003 populate_metadata read hostname 'smithi043' 2024-03-28T22:44:23.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.684+0000 7f0256e1b740 1 --2- 172.21.15.43:0/3747664966 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x563e89a87560 0x563e89a89980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.684+0000 7f0256e1b740 4 client.18003 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:44:23.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.684+0000 7f0256e1b740 1 -- 172.21.15.43:0/3747664966 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563e89978590 con 0x563e89a87560 2024-03-28T22:44:23.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.684+0000 7f0252000640 1 --2- 172.21.15.43:0/3747664966 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x563e89a87560 0x563e89a89980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.684+0000 7f0252000640 1 --2- 172.21.15.43:0/3747664966 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x563e89a87560 0x563e89a89980 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=68c001098809dcf0 server_cookie=3c4bb938adac76a0 in_seq=0 out_seq=0 2024-03-28T22:44:23.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.684+0000 7f0232a00640 10 client.18003 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:44:23.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.685+0000 7f0232a00640 1 -- 172.21.15.43:0/3747664966 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563e89978590 con 0x563e89a87560 2024-03-28T22:44:23.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.685+0000 7f0232a00640 10 client.18003 handle_command_reply: tid=0 2024-03-28T22:44:23.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.685+0000 7f0256e1b740 1 client.18003 shutdown 2024-03-28T22:44:23.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.685+0000 7f0256e1b740 1 -- 172.21.15.43:0/3747664966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e89a07e70 msgr2=0x563e89a13fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.685+0000 7f0256e1b740 1 --2- 172.21.15.43:0/3747664966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e89a07e70 0x563e89a13fe0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0238036040 tx=0x7f0238005900 comp rx=0 tx=0).stop 2024-03-28T22:44:23.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.685+0000 7f0256e1b740 1 -- 172.21.15.43:0/3747664966 shutdown_connections 2024-03-28T22:44:23.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.685+0000 7f0256e1b740 1 --2- 172.21.15.43:0/3747664966 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563e89a10a50 0x563e89a14520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.685+0000 7f0256e1b740 1 --2- 172.21.15.43:0/3747664966 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563e89a07e70 0x563e89a13fe0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.685+0000 7f0256e1b740 1 --2- 172.21.15.43:0/3747664966 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x563e89a87560 0x563e89a89980 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.685+0000 7f0256e1b740 1 --2- 172.21.15.43:0/3747664966 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563e89a07550 0x563e89a18c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.685+0000 7f0256e1b740 1 -- 172.21.15.43:0/3747664966 >> 172.21.15.43:0/3747664966 conn(0x563e89a01c80 msgr2=0x563e89a02c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:23.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.685+0000 7f0232a00640 0 client.18003 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:44:23.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.685+0000 7f0256e1b740 1 -- 172.21.15.43:0/3747664966 shutdown_connections 2024-03-28T22:44:23.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.685+0000 7f0256e1b740 1 -- 172.21.15.43:0/3747664966 wait complete. 2024-03-28T22:44:23.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.685+0000 7f0256e1b740 20 client.18003 trim_cache size 0 max 16384 2024-03-28T22:44:23.691 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:44:23.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.686+0000 7f0252a00640 1 -- 172.21.15.43:0/2762000835 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f02340a1ef0 msgr2=0x7f02340a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.686+0000 7f0252a00640 1 --2- 172.21.15.43:0/2762000835 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f02340a1ef0 0x7f02340a43b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x563e89a07020 tx=0x7f02400d4530 comp rx=0 tx=0).stop 2024-03-28T22:44:23.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.686+0000 7f0252a00640 1 -- 172.21.15.43:0/2762000835 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f024c1037d0 msgr2=0x7f024c19a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.686+0000 7f0252a00640 1 --2- 172.21.15.43:0/2762000835 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f024c1037d0 0x7f024c19a100 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f024000e9f0 tx=0x7f024000edf0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.686+0000 7f0252a00640 1 -- 172.21.15.43:0/2762000835 shutdown_connections 2024-03-28T22:44:23.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.686+0000 7f0252a00640 1 --2- 172.21.15.43:0/2762000835 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f02340a1ef0 0x7f02340a43b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.686+0000 7f0252a00640 1 --2- 172.21.15.43:0/2762000835 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f024c1097e0 0x7f024c19e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.686+0000 7f0252a00640 1 --2- 172.21.15.43:0/2762000835 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f024c104190 0x7f024c19a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.686+0000 7f0252a00640 1 --2- 172.21.15.43:0/2762000835 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f024c1037d0 0x7f024c19a100 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.686+0000 7f0252a00640 1 -- 172.21.15.43:0/2762000835 >> 172.21.15.43:0/2762000835 conn(0x7f024c0ff260 msgr2=0x7f024c0ff930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:23.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.686+0000 7f0252a00640 1 -- 172.21.15.43:0/2762000835 shutdown_connections 2024-03-28T22:44:23.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.686+0000 7f0252a00640 1 -- 172.21.15.43:0/2762000835 wait complete. 2024-03-28T22:44:23.702 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:44:23.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.916+0000 7f0fc6e00640 1 -- 172.21.15.43:0/2116856426 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0fc0073b00 msgr2=0x7f0fc0073ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.916+0000 7f0fc6e00640 1 --2- 172.21.15.43:0/2116856426 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0fc0073b00 0x7f0fc0073ee0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0009900 tx=0x7f0fb002bcd0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.916+0000 7f0fc6e00640 1 -- 172.21.15.43:0/2116856426 shutdown_connections 2024-03-28T22:44:23.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.916+0000 7f0fc6e00640 1 --2- 172.21.15.43:0/2116856426 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0fc010de80 0x7f0fc0110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.916+0000 7f0fc6e00640 1 --2- 172.21.15.43:0/2116856426 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0fc0074420 0x7f0fc010d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.916+0000 7f0fc6e00640 1 --2- 172.21.15.43:0/2116856426 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0fc0073b00 0x7f0fc0073ee0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.916+0000 7f0fc6e00640 1 -- 172.21.15.43:0/2116856426 >> 172.21.15.43:0/2116856426 conn(0x7f0fc00fd0f0 msgr2=0x7f0fc00ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:23.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.917+0000 7f0fc6e00640 1 -- 172.21.15.43:0/2116856426 shutdown_connections 2024-03-28T22:44:23.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.917+0000 7f0fc6e00640 1 -- 172.21.15.43:0/2116856426 wait complete. 2024-03-28T22:44:23.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.918+0000 7f0fc6e00640 1 Processor -- start 2024-03-28T22:44:23.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.918+0000 7f0fc6e00640 1 -- start start 2024-03-28T22:44:23.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.918+0000 7f0fc6e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0fc0073b00 0x7f0fc019a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.918+0000 7f0fc6e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0fc0074420 0x7f0fc019a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.918+0000 7f0fc6e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0fc010de80 0x7f0fc019e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.918+0000 7f0fc6e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc0113bb0 con 0x7f0fc0074420 2024-03-28T22:44:23.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.918+0000 7f0fc6e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc0113a30 con 0x7f0fc0073b00 2024-03-28T22:44:23.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.918+0000 7f0fc6e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc0113d30 con 0x7f0fc010de80 2024-03-28T22:44:23.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.918+0000 7f0fc5a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0fc0073b00 0x7f0fc019a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.918+0000 7f0fc5a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0fc0073b00 0x7f0fc019a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48626/0 (socket says 172.21.15.43:48626) 2024-03-28T22:44:23.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.918+0000 7f0fc5a00640 1 -- 172.21.15.43:0/2737450194 learned_addr learned my addr 172.21.15.43:0/2737450194 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:23.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.918+0000 7f0fc6400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0fc010de80 0x7f0fc019e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.918+0000 7f0fc5000640 1 --2- 172.21.15.43:0/2737450194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0fc0074420 0x7f0fc019a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.919+0000 7f0fc5a00640 1 -- 172.21.15.43:0/2737450194 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0fc010de80 msgr2=0x7f0fc019e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.919+0000 7f0fc5a00640 1 --2- 172.21.15.43:0/2737450194 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0fc010de80 0x7f0fc019e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.919+0000 7f0fc5a00640 1 -- 172.21.15.43:0/2737450194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0fc0074420 msgr2=0x7f0fc019a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.919+0000 7f0fc5a00640 1 --2- 172.21.15.43:0/2737450194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0fc0074420 0x7f0fc019a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.919+0000 7f0fc5a00640 1 -- 172.21.15.43:0/2737450194 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fc019f080 con 0x7f0fc0073b00 2024-03-28T22:44:23.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.919+0000 7f0fc5a00640 1 --2- 172.21.15.43:0/2737450194 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0fc0073b00 0x7f0fc019a090 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0000c00 tx=0x7f0fb0004430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:23.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.919+0000 7f0fbe000640 1 -- 172.21.15.43:0/2737450194 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb0042020 con 0x7f0fc0073b00 2024-03-28T22:44:23.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.919+0000 7f0fbe000640 1 -- 172.21.15.43:0/2737450194 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fb00054e0 con 0x7f0fc0073b00 2024-03-28T22:44:23.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.919+0000 7f0fc6e00640 1 -- 172.21.15.43:0/2737450194 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fc019f310 con 0x7f0fc0073b00 2024-03-28T22:44:23.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.919+0000 7f0fbe000640 1 -- 172.21.15.43:0/2737450194 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb0003c90 con 0x7f0fc0073b00 2024-03-28T22:44:23.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.920+0000 7f0fc6e00640 1 -- 172.21.15.43:0/2737450194 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fc01a6c00 con 0x7f0fc0073b00 2024-03-28T22:44:23.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.922+0000 7f0fbe000640 1 -- 172.21.15.43:0/2737450194 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f0fb0045020 con 0x7f0fc0073b00 2024-03-28T22:44:23.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.923+0000 7f0fcb06f740 1 Processor -- start 2024-03-28T22:44:23.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.923+0000 7f0fcb06f740 1 -- start start 2024-03-28T22:44:23.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.923+0000 7f0fcb06f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56417089d550 0x56417089d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.923+0000 7f0fcb06f740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56417089de70 0x5641708a6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.923+0000 7f0fcb06f740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641708a6a50 0x5641708a8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.923+0000 7f0fcb06f740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x564170801050 con 0x56417089de70 2024-03-28T22:44:23.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.923+0000 7f0fcb06f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5641707a9ce0 con 0x56417089d550 2024-03-28T22:44:23.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.923+0000 7f0fcb06f740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5641707fd0f0 con 0x5641708a6a50 2024-03-28T22:44:23.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.924+0000 7f0fc6400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56417089d550 0x56417089d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.924+0000 7f0fc5a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56417089de70 0x5641708a6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.924+0000 7f0fc5000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641708a6a50 0x5641708a8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.924+0000 7f0fc6400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56417089d550 0x56417089d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48628/0 (socket says 172.21.15.43:48628) 2024-03-28T22:44:23.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.924+0000 7f0fc6400640 1 -- 172.21.15.43:0/2632756977 learned_addr learned my addr 172.21.15.43:0/2632756977 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:23.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.924+0000 7f0fc6400640 1 -- 172.21.15.43:0/2632756977 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641708a6a50 msgr2=0x5641708a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:23.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.924+0000 7f0fc6400640 1 --2- 172.21.15.43:0/2632756977 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641708a6a50 0x5641708a8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.924+0000 7f0fc6400640 1 -- 172.21.15.43:0/2632756977 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56417089de70 msgr2=0x5641708a6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:23.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.924+0000 7f0fc6400640 1 --2- 172.21.15.43:0/2632756977 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56417089de70 0x5641708a6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.924+0000 7f0fc6400640 1 -- 172.21.15.43:0/2632756977 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641705153f0 con 0x56417089d550 2024-03-28T22:44:23.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.924+0000 7f0fbe000640 1 --2- 172.21.15.43:0/2737450194 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0fa40a1ef0 0x7f0fa40a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.924+0000 7f0fbe000640 1 -- 172.21.15.43:0/2737450194 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f0fb00f01f0 con 0x7f0fc0073b00 2024-03-28T22:44:23.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.924+0000 7f0fc6400640 1 --2- 172.21.15.43:0/2632756977 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56417089d550 0x56417089d930 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8009f70 tx=0x7f0fb8010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4aceec1c82a42912 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:23.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.925+0000 7f0fc6400640 1 --2- 172.21.15.43:0/2737450194 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0fa40a1ef0 0x7f0fa40a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.925+0000 7f0fb6a00640 1 -- 172.21.15.43:0/2632756977 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8016070 con 0x56417089d550 2024-03-28T22:44:23.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.925+0000 7f0fb6a00640 1 -- 172.21.15.43:0/2632756977 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fb801bc40 con 0x56417089d550 2024-03-28T22:44:23.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.925+0000 7f0fb6a00640 1 -- 172.21.15.43:0/2632756977 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb80113e0 con 0x56417089d550 2024-03-28T22:44:23.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.925+0000 7f0fcb06f740 1 -- 172.21.15.43:0/2632756977 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56417089d550 msgr2=0x56417089d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.925+0000 7f0fc6400640 1 --2- 172.21.15.43:0/2737450194 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0fa40a1ef0 0x7f0fa40a43b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0fb801bfd0 tx=0x7f0fb8011680 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:23.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.925+0000 7f0fcb06f740 1 --2- 172.21.15.43:0/2632756977 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56417089d550 0x56417089d930 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8009f70 tx=0x7f0fb8010800 comp rx=0 tx=0).stop 2024-03-28T22:44:23.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.925+0000 7f0fcb06f740 1 -- 172.21.15.43:0/2632756977 shutdown_connections 2024-03-28T22:44:23.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.926+0000 7f0fcb06f740 1 --2- 172.21.15.43:0/2632756977 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641708a6a50 0x5641708a8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.926+0000 7f0fcb06f740 1 --2- 172.21.15.43:0/2632756977 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56417089de70 0x5641708a6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.926+0000 7f0fcb06f740 1 --2- 172.21.15.43:0/2632756977 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56417089d550 0x56417089d930 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.926+0000 7f0fcb06f740 1 -- 172.21.15.43:0/2632756977 >> 172.21.15.43:0/2632756977 conn(0x564170897c80 msgr2=0x564170898060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:23.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.926+0000 7f0fcb06f740 1 -- 172.21.15.43:0/2632756977 shutdown_connections 2024-03-28T22:44:23.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.926+0000 7f0fcb06f740 1 -- 172.21.15.43:0/2632756977 wait complete. 2024-03-28T22:44:23.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.926+0000 7f0fcb06f740 1 -- start start 2024-03-28T22:44:23.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.926+0000 7f0fcb06f740 1 Processor -- start 2024-03-28T22:44:23.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.927+0000 7f0fcb06f740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56417089d550 0x5641708a3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.927+0000 7f0fcb06f740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56417089de70 0x5641708a3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.927+0000 7f0fcb06f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641708a6a50 0x5641708a44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.927+0000 7f0fcb06f740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5641708a9840 con 0x56417089d550 2024-03-28T22:44:23.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.927+0000 7f0fcb06f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x56417079f2f0 con 0x5641708a6a50 2024-03-28T22:44:23.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.927+0000 7f0fcb06f740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5641708a99c0 con 0x56417089de70 2024-03-28T22:44:23.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.927+0000 7f0fcb06f740 10 client.0 _ll_register_callbacks cb 0x564170514be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:44:23.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.927+0000 7f0fc5000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56417089de70 0x5641708a3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.927+0000 7f0fc6400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641708a6a50 0x5641708a44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.927+0000 7f0fc5a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56417089d550 0x5641708a3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.927+0000 7f0fc5000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56417089de70 0x5641708a3fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:48000/0 (socket says 172.21.15.43:48000) 2024-03-28T22:44:23.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.927+0000 7f0fc5000640 1 -- 172.21.15.43:0/1193245302 learned_addr learned my addr 172.21.15.43:0/1193245302 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:23.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.927+0000 7f0fc6400640 1 -- 172.21.15.43:0/1193245302 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56417089de70 msgr2=0x5641708a3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:23.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.927+0000 7f0fc6400640 1 --2- 172.21.15.43:0/1193245302 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56417089de70 0x5641708a3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.927+0000 7f0fc6400640 1 -- 172.21.15.43:0/1193245302 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56417089d550 msgr2=0x5641708a3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:23.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.927+0000 7f0fc6400640 1 --2- 172.21.15.43:0/1193245302 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56417089d550 0x5641708a3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.927+0000 7f0fc6400640 1 -- 172.21.15.43:0/1193245302 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641705df7d0 con 0x5641708a6a50 2024-03-28T22:44:23.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.928+0000 7f0fc6400640 1 --2- 172.21.15.43:0/1193245302 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641708a6a50 0x5641708a44f0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8011010 tx=0x7f0fb801b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95d2b606a210569a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:23.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.928+0000 7f0fab400640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:44:23.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.928+0000 7f0fab400640 1 -- 172.21.15.43:0/1193245302 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8035060 con 0x5641708a6a50 2024-03-28T22:44:23.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.928+0000 7f0fab400640 1 -- 172.21.15.43:0/1193245302 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f0fb8012450 con 0x5641708a6a50 2024-03-28T22:44:23.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.928+0000 7f0fcb06f740 1 -- 172.21.15.43:0/1193245302 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fb801bc40 con 0x5641708a6a50 2024-03-28T22:44:23.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.928+0000 7f0fab400640 1 -- 172.21.15.43:0/1193245302 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0fb803c050 con 0x5641708a6a50 2024-03-28T22:44:23.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.928+0000 7f0fcb06f740 10 client.28268 fetch_fsmap learned FSMap version 32 2024-03-28T22:44:23.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.928+0000 7f0fcb06f740 1 -- 172.21.15.43:0/1193245302 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5641708b5ac0 con 0x5641708a6a50 2024-03-28T22:44:23.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.929+0000 7f0fab400640 1 -- 172.21.15.43:0/1193245302 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fb801bc40 con 0x5641708a6a50 2024-03-28T22:44:23.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.929+0000 7f0fab400640 1 -- 172.21.15.43:0/1193245302 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f0fb800f040 con 0x5641708a6a50 2024-03-28T22:44:23.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.929+0000 7f0fcb06f740 10 client.28268 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:44:23.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.929+0000 7f0fcb06f740 10 client.28268 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:44:23.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.929+0000 7f0fcb06f740 20 client.28268 populate_metadata read hostname 'smithi043' 2024-03-28T22:44:23.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.929+0000 7f0fcb06f740 1 --2- 172.21.15.43:0/1193245302 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x5641708b6f20 0x5641708b9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.929+0000 7f0fcb06f740 4 client.28268 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:44:23.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.929+0000 7f0fcb06f740 1 -- 172.21.15.43:0/1193245302 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5641707aa680 con 0x5641708b6f20 2024-03-28T22:44:23.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.929+0000 7f0fc5a00640 1 --2- 172.21.15.43:0/1193245302 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x5641708b6f20 0x5641708b9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.930+0000 7f0fc5a00640 1 --2- 172.21.15.43:0/1193245302 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x5641708b6f20 0x5641708b9310 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=915b65304935722a server_cookie=176fc74679490792 in_seq=0 out_seq=0 2024-03-28T22:44:23.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.930+0000 7f0fab400640 10 client.28268 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:44:23.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.930+0000 7f0fab400640 1 -- 172.21.15.43:0/1193245302 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5641707aa680 con 0x5641708b6f20 2024-03-28T22:44:23.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.931+0000 7f0fab400640 10 client.28268 handle_command_reply: tid=0 2024-03-28T22:44:23.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.931+0000 7f0fcb06f740 1 client.28268 shutdown 2024-03-28T22:44:23.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.931+0000 7f0fcb06f740 1 -- 172.21.15.43:0/1193245302 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641708a6a50 msgr2=0x5641708a44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.931+0000 7f0fcb06f740 1 --2- 172.21.15.43:0/1193245302 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641708a6a50 0x5641708a44f0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8011010 tx=0x7f0fb801b000 comp rx=0 tx=0).stop 2024-03-28T22:44:23.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.931+0000 7f0fcb06f740 1 -- 172.21.15.43:0/1193245302 shutdown_connections 2024-03-28T22:44:23.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.931+0000 7f0fcb06f740 1 --2- 172.21.15.43:0/1193245302 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x5641708b6f20 0x5641708b9310 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.931+0000 7f0fcb06f740 1 --2- 172.21.15.43:0/1193245302 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641708a6a50 0x5641708a44f0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.931+0000 7f0fab400640 0 client.28268 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:44:23.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.931+0000 7f0fcb06f740 1 --2- 172.21.15.43:0/1193245302 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56417089de70 0x5641708a3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.931+0000 7f0fcb06f740 1 --2- 172.21.15.43:0/1193245302 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56417089d550 0x5641708a3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.931+0000 7f0fcb06f740 1 -- 172.21.15.43:0/1193245302 >> 172.21.15.43:0/1193245302 conn(0x564170897c80 msgr2=0x5641708999d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:23.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.931+0000 7f0fcb06f740 1 -- 172.21.15.43:0/1193245302 shutdown_connections 2024-03-28T22:44:23.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.931+0000 7f0fcb06f740 1 -- 172.21.15.43:0/1193245302 wait complete. 2024-03-28T22:44:23.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.931+0000 7f0fcb06f740 20 client.28268 trim_cache size 0 max 16384 2024-03-28T22:44:23.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.946+0000 7f0fcb06f740 1 Processor -- start 2024-03-28T22:44:23.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.946+0000 7f0fcb06f740 1 -- start start 2024-03-28T22:44:23.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.947+0000 7f0fcb06f740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56417089d550 0x5641708b25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.947+0000 7f0fcb06f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56417089de70 0x5641708be970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.947+0000 7f0fcb06f740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641708a6a50 0x5641708aec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.947+0000 7f0fcb06f740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc01b0d20 con 0x56417089d550 2024-03-28T22:44:23.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.947+0000 7f0fcb06f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc01b2860 con 0x56417089de70 2024-03-28T22:44:23.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.947+0000 7f0fcb06f740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc01b13f0 con 0x5641708a6a50 2024-03-28T22:44:23.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.947+0000 7f0fc5000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56417089de70 0x5641708be970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.947+0000 7f0fc6400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641708a6a50 0x5641708aec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.947+0000 7f0fc5a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56417089d550 0x5641708b25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.947+0000 7f0fc5000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56417089de70 0x5641708be970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48646/0 (socket says 172.21.15.43:48646) 2024-03-28T22:44:23.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.947+0000 7f0fc6400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641708a6a50 0x5641708aec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:48016/0 (socket says 172.21.15.43:48016) 2024-03-28T22:44:23.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.947+0000 7f0fc5000640 1 -- 172.21.15.43:0/2308484424 learned_addr learned my addr 172.21.15.43:0/2308484424 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:23.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.947+0000 7f0fc6400640 1 -- 172.21.15.43:0/2308484424 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56417089de70 msgr2=0x5641708be970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:23.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.947+0000 7f0fc6400640 1 --2- 172.21.15.43:0/2308484424 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56417089de70 0x5641708be970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.947+0000 7f0fc6400640 1 -- 172.21.15.43:0/2308484424 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56417089d550 msgr2=0x5641708b25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:23.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.947+0000 7f0fc6400640 1 --2- 172.21.15.43:0/2308484424 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56417089d550 0x5641708b25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.948+0000 7f0fc6400640 1 -- 172.21.15.43:0/2308484424 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641708985c0 con 0x5641708a6a50 2024-03-28T22:44:23.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.948+0000 7f0fc6400640 1 --2- 172.21.15.43:0/2308484424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641708a6a50 0x5641708aec00 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8011b00 tx=0x7f0fb8010d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38acd996554fa670 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:23.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.948+0000 7f0fb6000640 1 -- 172.21.15.43:0/2308484424 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8045070 con 0x5641708a6a50 2024-03-28T22:44:23.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.948+0000 7f0fb6000640 1 -- 172.21.15.43:0/2308484424 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fb8009a00 con 0x5641708a6a50 2024-03-28T22:44:23.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.948+0000 7f0fb6000640 1 -- 172.21.15.43:0/2308484424 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8003e60 con 0x5641708a6a50 2024-03-28T22:44:23.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.949+0000 7f0fcb06f740 1 -- 172.21.15.43:0/2308484424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641708a6a50 msgr2=0x5641708aec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.949+0000 7f0fcb06f740 1 --2- 172.21.15.43:0/2308484424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641708a6a50 0x5641708aec00 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8011b00 tx=0x7f0fb8010d00 comp rx=0 tx=0).stop 2024-03-28T22:44:23.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.949+0000 7f0fcb06f740 1 -- 172.21.15.43:0/2308484424 shutdown_connections 2024-03-28T22:44:23.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.949+0000 7f0fcb06f740 1 --2- 172.21.15.43:0/2308484424 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56417089de70 0x5641708be970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.949+0000 7f0fcb06f740 1 --2- 172.21.15.43:0/2308484424 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5641708a6a50 0x5641708aec00 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.949+0000 7f0fcb06f740 1 --2- 172.21.15.43:0/2308484424 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56417089d550 0x5641708b25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.949+0000 7f0fcb06f740 1 -- 172.21.15.43:0/2308484424 >> 172.21.15.43:0/2308484424 conn(0x564170897c80 msgr2=0x5641708a3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:23.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.949+0000 7f0fcb06f740 1 -- 172.21.15.43:0/2308484424 shutdown_connections 2024-03-28T22:44:23.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.949+0000 7f0fcb06f740 1 -- 172.21.15.43:0/2308484424 wait complete. 2024-03-28T22:44:23.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.949+0000 7f0fcb06f740 1 -- start start 2024-03-28T22:44:23.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.950+0000 7f0fcb06f740 1 Processor -- start 2024-03-28T22:44:23.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.950+0000 7f0fcb06f740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56417089d550 0x5641708a09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.950+0000 7f0fcb06f740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56417089de70 0x5641708a0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.950+0000 7f0fcb06f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641708a6a50 0x56417091bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.950+0000 7f0fcb06f740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc01b2190 con 0x56417089d550 2024-03-28T22:44:23.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.950+0000 7f0fcb06f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc01b1ac0 con 0x5641708a6a50 2024-03-28T22:44:23.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.950+0000 7f0fcb06f740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc01aed20 con 0x56417089de70 2024-03-28T22:44:23.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.950+0000 7f0fcb06f740 10 client.0 _ll_register_callbacks cb 0x7f0fb00f3b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:44:23.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.950+0000 7f0fc5000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56417089de70 0x5641708a0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.950+0000 7f0fc5000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56417089de70 0x5641708a0f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:48030/0 (socket says 172.21.15.43:48030) 2024-03-28T22:44:23.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.950+0000 7f0fc5000640 1 -- 172.21.15.43:0/955853059 learned_addr learned my addr 172.21.15.43:0/955853059 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:23.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.950+0000 7f0fc6400640 1 --2- 172.21.15.43:0/955853059 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641708a6a50 0x56417091bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.951+0000 7f0fc5a00640 1 --2- 172.21.15.43:0/955853059 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56417089d550 0x5641708a09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.951+0000 7f0fc5000640 1 -- 172.21.15.43:0/955853059 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641708a6a50 msgr2=0x56417091bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:23.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.951+0000 7f0fc5000640 1 --2- 172.21.15.43:0/955853059 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641708a6a50 0x56417091bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.951+0000 7f0fc5000640 1 -- 172.21.15.43:0/955853059 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56417089d550 msgr2=0x5641708a09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:23.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.951+0000 7f0fc5000640 1 --2- 172.21.15.43:0/955853059 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56417089d550 0x5641708a09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.951+0000 7f0fc5000640 1 -- 172.21.15.43:0/955853059 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641708a1800 con 0x56417089de70 2024-03-28T22:44:23.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.951+0000 7f0fc6400640 1 --2- 172.21.15.43:0/955853059 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641708a6a50 0x56417091bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:44:23.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.951+0000 7f0fc5a00640 1 --2- 172.21.15.43:0/955853059 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56417089d550 0x5641708a09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:44:23.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.951+0000 7f0fc5000640 1 --2- 172.21.15.43:0/955853059 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56417089de70 0x5641708a0f20 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f0fac00bb80 tx=0x7f0fac0039c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6cdc21dc1d29170 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:23.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.951+0000 7f0fab400640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:44:23.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.951+0000 7f0fab400640 1 -- 172.21.15.43:0/955853059 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fac005220 con 0x56417089de70 2024-03-28T22:44:23.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.952+0000 7f0fab400640 1 -- 172.21.15.43:0/955853059 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f0fac0053c0 con 0x56417089de70 2024-03-28T22:44:23.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.952+0000 7f0fcb06f740 1 -- 172.21.15.43:0/955853059 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fb8009a00 con 0x56417089de70 2024-03-28T22:44:23.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.952+0000 7f0fab400640 1 -- 172.21.15.43:0/955853059 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0fac028050 con 0x56417089de70 2024-03-28T22:44:23.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.952+0000 7f0fcb06f740 10 client.28213 fetch_fsmap learned FSMap version 32 2024-03-28T22:44:23.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.952+0000 7f0fcb06f740 1 -- 172.21.15.43:0/955853059 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56417091d650 con 0x56417089de70 2024-03-28T22:44:23.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.952+0000 7f0fab400640 1 -- 172.21.15.43:0/955853059 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fb8009a00 con 0x56417089de70 2024-03-28T22:44:23.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.952+0000 7f0fab400640 1 -- 172.21.15.43:0/955853059 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f0fac01d830 con 0x56417089de70 2024-03-28T22:44:23.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.952+0000 7f0fcb06f740 10 client.28213 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:44:23.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.952+0000 7f0fcb06f740 10 client.28213 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:44:23.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.952+0000 7f0fcb06f740 20 client.28213 populate_metadata read hostname 'smithi043' 2024-03-28T22:44:23.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.952+0000 7f0fcb06f740 1 --2- 172.21.15.43:0/955853059 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x56417091de80 0x5641709202a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:23.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.952+0000 7f0fcb06f740 4 client.28213 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:44:23.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.952+0000 7f0fcb06f740 1 -- 172.21.15.43:0/955853059 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56417080e590 con 0x56417091de80 2024-03-28T22:44:23.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.953+0000 7f0fc5a00640 1 --2- 172.21.15.43:0/955853059 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x56417091de80 0x5641709202a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:23.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.953+0000 7f0fc5a00640 1 --2- 172.21.15.43:0/955853059 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x56417091de80 0x5641709202a0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1128352888d36a75 server_cookie=dd1a161242d94c22 in_seq=0 out_seq=0 2024-03-28T22:44:23.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.953+0000 7f0fab400640 10 client.28213 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:44:23.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.953+0000 7f0fab400640 1 -- 172.21.15.43:0/955853059 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56417080e590 con 0x56417091de80 2024-03-28T22:44:23.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.953+0000 7f0fab400640 10 client.28213 handle_command_reply: tid=0 2024-03-28T22:44:23.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.953+0000 7f0fcb06f740 1 client.28213 shutdown 2024-03-28T22:44:23.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.954+0000 7f0fcb06f740 1 -- 172.21.15.43:0/955853059 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56417089de70 msgr2=0x5641708a0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.954+0000 7f0fcb06f740 1 --2- 172.21.15.43:0/955853059 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56417089de70 0x5641708a0f20 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f0fac00bb80 tx=0x7f0fac0039c0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.954+0000 7f0fcb06f740 1 -- 172.21.15.43:0/955853059 shutdown_connections 2024-03-28T22:44:23.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.954+0000 7f0fcb06f740 1 --2- 172.21.15.43:0/955853059 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x56417091de80 0x5641709202a0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.954+0000 7f0fcb06f740 1 --2- 172.21.15.43:0/955853059 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5641708a6a50 0x56417091bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.954+0000 7f0fcb06f740 1 --2- 172.21.15.43:0/955853059 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56417089de70 0x5641708a0f20 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.954+0000 7f0fab400640 0 client.28213 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:44:23.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.954+0000 7f0fcb06f740 1 --2- 172.21.15.43:0/955853059 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56417089d550 0x5641708a09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.954+0000 7f0fcb06f740 1 -- 172.21.15.43:0/955853059 >> 172.21.15.43:0/955853059 conn(0x564170897c80 msgr2=0x564170896dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:23.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.954+0000 7f0fcb06f740 1 -- 172.21.15.43:0/955853059 shutdown_connections 2024-03-28T22:44:23.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.954+0000 7f0fcb06f740 1 -- 172.21.15.43:0/955853059 wait complete. 2024-03-28T22:44:23.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.954+0000 7f0fcb06f740 20 client.28213 trim_cache size 0 max 16384 2024-03-28T22:44:23.959 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:44:23.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.955+0000 7f0fc6e00640 1 -- 172.21.15.43:0/2737450194 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0fa40a1ef0 msgr2=0x7f0fa40a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.955+0000 7f0fc6e00640 1 --2- 172.21.15.43:0/2737450194 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0fa40a1ef0 0x7f0fa40a43b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0fb801bfd0 tx=0x7f0fb8011680 comp rx=0 tx=0).stop 2024-03-28T22:44:23.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.955+0000 7f0fc6e00640 1 -- 172.21.15.43:0/2737450194 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0fc0073b00 msgr2=0x7f0fc019a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:23.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.955+0000 7f0fc6e00640 1 --2- 172.21.15.43:0/2737450194 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0fc0073b00 0x7f0fc019a090 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0000c00 tx=0x7f0fb0004430 comp rx=0 tx=0).stop 2024-03-28T22:44:23.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.955+0000 7f0fc6e00640 1 -- 172.21.15.43:0/2737450194 shutdown_connections 2024-03-28T22:44:23.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.955+0000 7f0fc6e00640 1 --2- 172.21.15.43:0/2737450194 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0fa40a1ef0 0x7f0fa40a43b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.955+0000 7f0fc6e00640 1 --2- 172.21.15.43:0/2737450194 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0fc010de80 0x7f0fc019e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.955+0000 7f0fc6e00640 1 --2- 172.21.15.43:0/2737450194 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0fc0074420 0x7f0fc019a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.955+0000 7f0fc6e00640 1 --2- 172.21.15.43:0/2737450194 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0fc0073b00 0x7f0fc019a090 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:23.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.955+0000 7f0fc6e00640 1 -- 172.21.15.43:0/2737450194 >> 172.21.15.43:0/2737450194 conn(0x7f0fc00fd0f0 msgr2=0x7f0fc00fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:23.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.955+0000 7f0fc6e00640 1 -- 172.21.15.43:0/2737450194 shutdown_connections 2024-03-28T22:44:23.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:23.955+0000 7f0fc6e00640 1 -- 172.21.15.43:0/2737450194 wait complete. 2024-03-28T22:44:23.968 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:23 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2152374617' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:44:23.970 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:44:24.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:23 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2152374617' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:44:24.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.199+0000 7f7bafa00640 1 -- 172.21.15.43:0/1293137573 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ba810d030 msgr2=0x7f7ba8114c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:24.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.199+0000 7f7bafa00640 1 --2- 172.21.15.43:0/1293137573 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ba810d030 0x7f7ba8114c00 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0009ca0 tx=0x7f7ba002c0f0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.199+0000 7f7bafa00640 1 -- 172.21.15.43:0/1293137573 shutdown_connections 2024-03-28T22:44:24.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.199+0000 7f7bafa00640 1 --2- 172.21.15.43:0/1293137573 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ba8108910 0x7f7ba810c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.199+0000 7f7bafa00640 1 --2- 172.21.15.43:0/1293137573 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ba810d030 0x7f7ba8114c00 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.199+0000 7f7bafa00640 1 --2- 172.21.15.43:0/1293137573 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ba8107f60 0x7f7ba8108340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.199+0000 7f7bafa00640 1 -- 172.21.15.43:0/1293137573 >> 172.21.15.43:0/1293137573 conn(0x7f7ba8075c90 msgr2=0x7f7ba80780b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:24.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.199+0000 7f7bafa00640 1 -- 172.21.15.43:0/1293137573 shutdown_connections 2024-03-28T22:44:24.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.199+0000 7f7bafa00640 1 -- 172.21.15.43:0/1293137573 wait complete. 2024-03-28T22:44:24.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:23 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2152374617' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:44:24.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.200+0000 7f7bafa00640 1 Processor -- start 2024-03-28T22:44:24.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.200+0000 7f7bafa00640 1 -- start start 2024-03-28T22:44:24.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.200+0000 7f7bafa00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ba8107f60 0x7f7ba819e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:24.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.200+0000 7f7bafa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ba810d030 0x7f7ba819eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:24.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.200+0000 7f7bafa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ba81a2e40 0x7f7ba81a32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:24.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.200+0000 7f7bafa00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba81180c0 con 0x7f7ba810d030 2024-03-28T22:44:24.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.200+0000 7f7bafa00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba8117f40 con 0x7f7ba8107f60 2024-03-28T22:44:24.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.200+0000 7f7bafa00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba8118240 con 0x7f7ba81a2e40 2024-03-28T22:44:24.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.201+0000 7f7bad000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ba8107f60 0x7f7ba819e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:24.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.201+0000 7f7bad000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ba8107f60 0x7f7ba819e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48666/0 (socket says 172.21.15.43:48666) 2024-03-28T22:44:24.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.201+0000 7f7bad000640 1 -- 172.21.15.43:0/1527165158 learned_addr learned my addr 172.21.15.43:0/1527165158 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:24.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.201+0000 7f7ba7e00640 1 --2- 172.21.15.43:0/1527165158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ba810d030 0x7f7ba819eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:24.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.201+0000 7f7bada00640 1 --2- 172.21.15.43:0/1527165158 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ba81a2e40 0x7f7ba81a32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:24.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.201+0000 7f7bad000640 1 -- 172.21.15.43:0/1527165158 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ba81a2e40 msgr2=0x7f7ba81a32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:24.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.201+0000 7f7bad000640 1 --2- 172.21.15.43:0/1527165158 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ba81a2e40 0x7f7ba81a32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.201+0000 7f7bad000640 1 -- 172.21.15.43:0/1527165158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ba810d030 msgr2=0x7f7ba819eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:24.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.201+0000 7f7bad000640 1 --2- 172.21.15.43:0/1527165158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ba810d030 0x7f7ba819eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.201+0000 7f7bad000640 1 -- 172.21.15.43:0/1527165158 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ba81a3930 con 0x7f7ba8107f60 2024-03-28T22:44:24.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.201+0000 7f7bad000640 1 --2- 172.21.15.43:0/1527165158 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ba8107f60 0x7f7ba819e580 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c00e9f0 tx=0x7f7b9c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:24.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.202+0000 7f7ba5600640 1 -- 172.21.15.43:0/1527165158 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b9c00cd80 con 0x7f7ba8107f60 2024-03-28T22:44:24.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.202+0000 7f7ba5600640 1 -- 172.21.15.43:0/1527165158 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b9c010910 con 0x7f7ba8107f60 2024-03-28T22:44:24.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.202+0000 7f7ba5600640 1 -- 172.21.15.43:0/1527165158 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b9c018990 con 0x7f7ba8107f60 2024-03-28T22:44:24.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.202+0000 7f7bafa00640 1 -- 172.21.15.43:0/1527165158 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ba81a3c20 con 0x7f7ba8107f60 2024-03-28T22:44:24.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.202+0000 7f7bafa00640 1 -- 172.21.15.43:0/1527165158 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ba81ab3c0 con 0x7f7ba8107f60 2024-03-28T22:44:24.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.205+0000 7f7bb280c740 1 Processor -- start 2024-03-28T22:44:24.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.205+0000 7f7ba5600640 1 -- 172.21.15.43:0/1527165158 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f7b9c010ac0 con 0x7f7ba8107f60 2024-03-28T22:44:24.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.205+0000 7f7bb280c740 1 -- start start 2024-03-28T22:44:24.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.205+0000 7f7bb280c740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5638b1b16380 0x5638b1b16760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:24.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.205+0000 7f7bb280c740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5638b1b16ca0 0x5638b1b1f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:24.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.205+0000 7f7bb280c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638b1b1f880 0x5638b1b21c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:24.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.205+0000 7f7bb280c740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5638b1a7a050 con 0x5638b1b16380 2024-03-28T22:44:24.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.205+0000 7f7bb280c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5638b1a22ce0 con 0x5638b1b1f880 2024-03-28T22:44:24.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.205+0000 7f7bb280c740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5638b1a760f0 con 0x5638b1b16ca0 2024-03-28T22:44:24.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.206+0000 7f7ba7e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638b1b1f880 0x5638b1b21c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:24.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.206+0000 7f7bad000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5638b1b16ca0 0x5638b1b1f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:24.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.206+0000 7f7bada00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5638b1b16380 0x5638b1b16760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:24.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.206+0000 7f7ba7e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638b1b1f880 0x5638b1b21c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48674/0 (socket says 172.21.15.43:48674) 2024-03-28T22:44:24.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.206+0000 7f7ba7e00640 1 -- 172.21.15.43:0/1489030159 learned_addr learned my addr 172.21.15.43:0/1489030159 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:24.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.206+0000 7f7ba7e00640 1 -- 172.21.15.43:0/1489030159 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5638b1b16ca0 msgr2=0x5638b1b1f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:24.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.206+0000 7f7ba7e00640 1 --2- 172.21.15.43:0/1489030159 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5638b1b16ca0 0x5638b1b1f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.206+0000 7f7ba7e00640 1 -- 172.21.15.43:0/1489030159 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5638b1b16380 msgr2=0x5638b1b16760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:24.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.206+0000 7f7ba7e00640 1 --2- 172.21.15.43:0/1489030159 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5638b1b16380 0x5638b1b16760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.206+0000 7f7ba7e00640 1 -- 172.21.15.43:0/1489030159 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638b178e3f0 con 0x5638b1b1f880 2024-03-28T22:44:24.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.206+0000 7f7ba7e00640 1 --2- 172.21.15.43:0/1489030159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638b1b1f880 0x5638b1b21c70 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f7b94009c20 tx=0x7f7b94011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54885705b3f2970b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:24.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.206+0000 7f7ba5600640 1 --2- 172.21.15.43:0/1527165158 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7b900a1f70 0x7f7b900a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:24.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.207+0000 7f7b9a000640 1 -- 172.21.15.43:0/1489030159 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b9401e070 con 0x5638b1b1f880 2024-03-28T22:44:24.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.207+0000 7f7b9a000640 1 -- 172.21.15.43:0/1489030159 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b94009dd0 con 0x5638b1b1f880 2024-03-28T22:44:24.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.207+0000 7f7b9a000640 1 -- 172.21.15.43:0/1489030159 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b940187a0 con 0x5638b1b1f880 2024-03-28T22:44:24.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.207+0000 7f7ba5600640 1 -- 172.21.15.43:0/1527165158 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f7b9c0ce890 con 0x7f7ba8107f60 2024-03-28T22:44:24.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.207+0000 7f7bada00640 1 --2- 172.21.15.43:0/1527165158 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7b900a1f70 0x7f7b900a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:24.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.207+0000 7f7bb280c740 1 -- 172.21.15.43:0/1489030159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638b1b1f880 msgr2=0x5638b1b21c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:24.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.207+0000 7f7bb280c740 1 --2- 172.21.15.43:0/1489030159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638b1b1f880 0x5638b1b21c70 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f7b94009c20 tx=0x7f7b94011810 comp rx=0 tx=0).stop 2024-03-28T22:44:24.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.207+0000 7f7bb280c740 1 -- 172.21.15.43:0/1489030159 shutdown_connections 2024-03-28T22:44:24.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.207+0000 7f7bb280c740 1 --2- 172.21.15.43:0/1489030159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638b1b1f880 0x5638b1b21c70 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.207+0000 7f7bb280c740 1 --2- 172.21.15.43:0/1489030159 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5638b1b16ca0 0x5638b1b1f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.207+0000 7f7bb280c740 1 --2- 172.21.15.43:0/1489030159 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5638b1b16380 0x5638b1b16760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.207+0000 7f7bb280c740 1 -- 172.21.15.43:0/1489030159 >> 172.21.15.43:0/1489030159 conn(0x5638b1b10ab0 msgr2=0x5638b1b10e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:24.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.208+0000 7f7bada00640 1 --2- 172.21.15.43:0/1527165158 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7b900a1f70 0x7f7b900a4430 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0005e30 tx=0x7f7ba0005d80 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:24.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.208+0000 7f7bb280c740 1 -- 172.21.15.43:0/1489030159 shutdown_connections 2024-03-28T22:44:24.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.208+0000 7f7bb280c740 1 -- 172.21.15.43:0/1489030159 wait complete. 2024-03-28T22:44:24.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.208+0000 7f7bb280c740 1 -- start start 2024-03-28T22:44:24.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.208+0000 7f7bb280c740 1 Processor -- start 2024-03-28T22:44:24.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.209+0000 7f7bb280c740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5638b1b16380 0x5638b1b1c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:24.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.209+0000 7f7bb280c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638b1b16ca0 0x5638b1b1cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:24.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.209+0000 7f7bb280c740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5638b1b1f880 0x5638b1b1d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:24.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.209+0000 7f7bb280c740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5638b1b22670 con 0x5638b1b16380 2024-03-28T22:44:24.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.209+0000 7f7bb280c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5638b1a182f0 con 0x5638b1b16ca0 2024-03-28T22:44:24.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.209+0000 7f7bb280c740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5638b1b227f0 con 0x5638b1b1f880 2024-03-28T22:44:24.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.209+0000 7f7bb280c740 10 client.0 _ll_register_callbacks cb 0x5638b178dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:44:24.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.209+0000 7f7ba7e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638b1b16ca0 0x5638b1b1cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:24.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.209+0000 7f7ba7e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638b1b16ca0 0x5638b1b1cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48680/0 (socket says 172.21.15.43:48680) 2024-03-28T22:44:24.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.209+0000 7f7bada00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5638b1b1f880 0x5638b1b1d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:24.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.209+0000 7f7ba7e00640 1 -- 172.21.15.43:0/486292023 learned_addr learned my addr 172.21.15.43:0/486292023 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:24.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.209+0000 7f7bad000640 1 --2- 172.21.15.43:0/486292023 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5638b1b16380 0x5638b1b1c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:24.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.209+0000 7f7ba7e00640 1 -- 172.21.15.43:0/486292023 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5638b1b1f880 msgr2=0x5638b1b1d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:24.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.209+0000 7f7ba7e00640 1 --2- 172.21.15.43:0/486292023 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5638b1b1f880 0x5638b1b1d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.209+0000 7f7ba7e00640 1 -- 172.21.15.43:0/486292023 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5638b1b16380 msgr2=0x5638b1b1c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:24.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.209+0000 7f7ba7e00640 1 --2- 172.21.15.43:0/486292023 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5638b1b16380 0x5638b1b1c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.209+0000 7f7ba7e00640 1 -- 172.21.15.43:0/486292023 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638b18587d0 con 0x5638b1b16ca0 2024-03-28T22:44:24.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.210+0000 7f7ba7e00640 1 --2- 172.21.15.43:0/486292023 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638b1b16ca0 0x5638b1b1cdc0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f7b94009bf0 tx=0x7f7b94019640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=615328272f973c59 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:24.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.210+0000 7f7b8ea00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:44:24.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.210+0000 7f7b8ea00640 1 -- 172.21.15.43:0/486292023 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b9401e070 con 0x5638b1b16ca0 2024-03-28T22:44:24.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.210+0000 7f7b8ea00640 1 -- 172.21.15.43:0/486292023 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f7b94019a40 con 0x5638b1b16ca0 2024-03-28T22:44:24.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.210+0000 7f7bb280c740 1 -- 172.21.15.43:0/486292023 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b94009dd0 con 0x5638b1b16ca0 2024-03-28T22:44:24.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.210+0000 7f7b8ea00640 1 -- 172.21.15.43:0/486292023 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7b94031050 con 0x5638b1b16ca0 2024-03-28T22:44:24.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.210+0000 7f7bb280c740 10 client.28286 fetch_fsmap learned FSMap version 32 2024-03-28T22:44:24.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.210+0000 7f7bb280c740 1 -- 172.21.15.43:0/486292023 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5638b1b2e8d0 con 0x5638b1b16ca0 2024-03-28T22:44:24.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.211+0000 7f7b8ea00640 1 -- 172.21.15.43:0/486292023 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b94009dd0 con 0x5638b1b16ca0 2024-03-28T22:44:24.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.211+0000 7f7b8ea00640 1 -- 172.21.15.43:0/486292023 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7b94004750 con 0x5638b1b16ca0 2024-03-28T22:44:24.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.211+0000 7f7bb280c740 10 client.28286 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:44:24.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.211+0000 7f7bb280c740 10 client.28286 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:44:24.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.211+0000 7f7bb280c740 20 client.28286 populate_metadata read hostname 'smithi043' 2024-03-28T22:44:24.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.211+0000 7f7bb280c740 1 --2- 172.21.15.43:0/486292023 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5638b1b2fd30 0x5638b1b32120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:24.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.211+0000 7f7bb280c740 4 client.28286 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:44:24.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.211+0000 7f7bb280c740 1 -- 172.21.15.43:0/486292023 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5638b1a23680 con 0x5638b1b2fd30 2024-03-28T22:44:24.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.211+0000 7f7bad000640 1 --2- 172.21.15.43:0/486292023 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5638b1b2fd30 0x5638b1b32120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:24.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.212+0000 7f7bad000640 1 --2- 172.21.15.43:0/486292023 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5638b1b2fd30 0x5638b1b32120 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f7fb0cdf262e782 server_cookie=119fde19e6e0a77 in_seq=0 out_seq=0 2024-03-28T22:44:24.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.212+0000 7f7b8ea00640 10 client.28286 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:44:24.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.213+0000 7f7b8ea00640 1 -- 172.21.15.43:0/486292023 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5638b1a23680 con 0x5638b1b2fd30 2024-03-28T22:44:24.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.213+0000 7f7b8ea00640 10 client.28286 handle_command_reply: tid=0 2024-03-28T22:44:24.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.213+0000 7f7bb280c740 1 client.28286 shutdown 2024-03-28T22:44:24.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.213+0000 7f7bb280c740 1 -- 172.21.15.43:0/486292023 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638b1b16ca0 msgr2=0x5638b1b1cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:24.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.213+0000 7f7bb280c740 1 --2- 172.21.15.43:0/486292023 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638b1b16ca0 0x5638b1b1cdc0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f7b94009bf0 tx=0x7f7b94019640 comp rx=0 tx=0).stop 2024-03-28T22:44:24.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.213+0000 7f7bb280c740 1 -- 172.21.15.43:0/486292023 shutdown_connections 2024-03-28T22:44:24.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.213+0000 7f7bb280c740 1 --2- 172.21.15.43:0/486292023 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5638b1b2fd30 0x5638b1b32120 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.213+0000 7f7bb280c740 1 --2- 172.21.15.43:0/486292023 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638b1b16ca0 0x5638b1b1cdc0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.213+0000 7f7bb280c740 1 --2- 172.21.15.43:0/486292023 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5638b1b1f880 0x5638b1b1d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.213+0000 7f7b8ea00640 0 client.28286 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:44:24.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.213+0000 7f7bb280c740 1 --2- 172.21.15.43:0/486292023 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5638b1b16380 0x5638b1b1c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.213+0000 7f7bb280c740 1 -- 172.21.15.43:0/486292023 >> 172.21.15.43:0/486292023 conn(0x5638b1b10ab0 msgr2=0x5638b1b12800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:24.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.213+0000 7f7bb280c740 1 -- 172.21.15.43:0/486292023 shutdown_connections 2024-03-28T22:44:24.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.213+0000 7f7bb280c740 1 -- 172.21.15.43:0/486292023 wait complete. 2024-03-28T22:44:24.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.213+0000 7f7bb280c740 20 client.28286 trim_cache size 0 max 16384 2024-03-28T22:44:24.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.228+0000 7f7bb280c740 1 Processor -- start 2024-03-28T22:44:24.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.228+0000 7f7bb280c740 1 -- start start 2024-03-28T22:44:24.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.229+0000 7f7bb280c740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5638b1b16380 0x5638b1b1efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:24.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.229+0000 7f7bb280c740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5638b1b16ca0 0x5638b1b2b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:24.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.229+0000 7f7bb280c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638b1b1f880 0x5638b1b12580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:24.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.229+0000 7f7bb280c740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba81b52d0 con 0x5638b1b16380 2024-03-28T22:44:24.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.229+0000 7f7bb280c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba81b6e10 con 0x5638b1b1f880 2024-03-28T22:44:24.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.229+0000 7f7bb280c740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba81b59a0 con 0x5638b1b16ca0 2024-03-28T22:44:24.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.229+0000 7f7bad000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5638b1b16380 0x5638b1b1efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:24.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.229+0000 7f7bada00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638b1b1f880 0x5638b1b12580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:24.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.229+0000 7f7ba7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5638b1b16ca0 0x5638b1b2b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:24.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.229+0000 7f7ba7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5638b1b16ca0 0x5638b1b2b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:48076/0 (socket says 172.21.15.43:48076) 2024-03-28T22:44:24.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.229+0000 7f7ba7e00640 1 -- 172.21.15.43:0/2547253999 learned_addr learned my addr 172.21.15.43:0/2547253999 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:24.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.229+0000 7f7bada00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638b1b1f880 0x5638b1b12580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48692/0 (socket says 172.21.15.43:48692) 2024-03-28T22:44:24.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.229+0000 7f7ba7e00640 1 -- 172.21.15.43:0/2547253999 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638b1b1f880 msgr2=0x5638b1b12580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:24.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.229+0000 7f7ba7e00640 1 --2- 172.21.15.43:0/2547253999 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638b1b1f880 0x5638b1b12580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.229+0000 7f7ba7e00640 1 -- 172.21.15.43:0/2547253999 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5638b1b16380 msgr2=0x5638b1b1efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:24.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.229+0000 7f7ba7e00640 1 --2- 172.21.15.43:0/2547253999 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5638b1b16380 0x5638b1b1efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.229+0000 7f7ba7e00640 1 -- 172.21.15.43:0/2547253999 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638b1b22550 con 0x5638b1b16ca0 2024-03-28T22:44:24.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.230+0000 7f7ba7e00640 1 --2- 172.21.15.43:0/2547253999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5638b1b16ca0 0x5638b1b2b410 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f7b940097e0 tx=0x7f7b94011e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29731e7363dc95b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:24.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.230+0000 7f7b99600640 1 -- 172.21.15.43:0/2547253999 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b9401e070 con 0x5638b1b16ca0 2024-03-28T22:44:24.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.230+0000 7f7b99600640 1 -- 172.21.15.43:0/2547253999 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b94028070 con 0x5638b1b16ca0 2024-03-28T22:44:24.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.230+0000 7f7b99600640 1 -- 172.21.15.43:0/2547253999 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b94003a30 con 0x5638b1b16ca0 2024-03-28T22:44:24.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.230+0000 7f7bb280c740 1 -- 172.21.15.43:0/2547253999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5638b1b16ca0 msgr2=0x5638b1b2b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:24.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.230+0000 7f7bb280c740 1 --2- 172.21.15.43:0/2547253999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5638b1b16ca0 0x5638b1b2b410 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f7b940097e0 tx=0x7f7b94011e40 comp rx=0 tx=0).stop 2024-03-28T22:44:24.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.230+0000 7f7bb280c740 1 -- 172.21.15.43:0/2547253999 shutdown_connections 2024-03-28T22:44:24.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.230+0000 7f7bb280c740 1 --2- 172.21.15.43:0/2547253999 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638b1b1f880 0x5638b1b12580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.230+0000 7f7bb280c740 1 --2- 172.21.15.43:0/2547253999 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5638b1b16ca0 0x5638b1b2b410 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.230+0000 7f7bb280c740 1 --2- 172.21.15.43:0/2547253999 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5638b1b16380 0x5638b1b1efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.230+0000 7f7bb280c740 1 -- 172.21.15.43:0/2547253999 >> 172.21.15.43:0/2547253999 conn(0x5638b1b10ab0 msgr2=0x5638b1b11e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:24.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.231+0000 7f7bb280c740 1 -- 172.21.15.43:0/2547253999 shutdown_connections 2024-03-28T22:44:24.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.231+0000 7f7bb280c740 1 -- 172.21.15.43:0/2547253999 wait complete. 2024-03-28T22:44:24.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.231+0000 7f7bb280c740 1 -- start start 2024-03-28T22:44:24.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.231+0000 7f7bb280c740 1 Processor -- start 2024-03-28T22:44:24.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.232+0000 7f7bb280c740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5638b1b16380 0x5638b1b23520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:24.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.232+0000 7f7bb280c740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5638b1b16ca0 0x5638b1b23a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:24.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.232+0000 7f7bb280c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638b1b1f880 0x5638b1b23fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:24.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.232+0000 7f7bb280c740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba81b6740 con 0x5638b1b16ca0 2024-03-28T22:44:24.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.232+0000 7f7bb280c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba81b6070 con 0x5638b1b1f880 2024-03-28T22:44:24.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.232+0000 7f7bb280c740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba81b32d0 con 0x5638b1b16380 2024-03-28T22:44:24.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.232+0000 7f7bb280c740 10 client.0 _ll_register_callbacks cb 0x7f7b9401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:44:24.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.232+0000 7f7bada00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638b1b1f880 0x5638b1b23fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:24.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.232+0000 7f7ba7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5638b1b16ca0 0x5638b1b23a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:24.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.232+0000 7f7bad000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5638b1b16380 0x5638b1b23520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:24.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.232+0000 7f7bada00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638b1b1f880 0x5638b1b23fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:48694/0 (socket says 172.21.15.43:48694) 2024-03-28T22:44:24.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.232+0000 7f7bada00640 1 -- 172.21.15.43:0/2920933994 learned_addr learned my addr 172.21.15.43:0/2920933994 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:24.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.232+0000 7f7bada00640 1 -- 172.21.15.43:0/2920933994 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5638b1b16380 msgr2=0x5638b1b23520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:24.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.232+0000 7f7bada00640 1 --2- 172.21.15.43:0/2920933994 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5638b1b16380 0x5638b1b23520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.232+0000 7f7bada00640 1 -- 172.21.15.43:0/2920933994 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5638b1b16ca0 msgr2=0x5638b1b23a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:24.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.232+0000 7f7bada00640 1 --2- 172.21.15.43:0/2920933994 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5638b1b16ca0 0x5638b1b23a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.232+0000 7f7bada00640 1 -- 172.21.15.43:0/2920933994 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638b1b950e0 con 0x5638b1b1f880 2024-03-28T22:44:24.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.232+0000 7f7bad000640 1 --2- 172.21.15.43:0/2920933994 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5638b1b16380 0x5638b1b23520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:44:24.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.232+0000 7f7ba7e00640 1 --2- 172.21.15.43:0/2920933994 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5638b1b16ca0 0x5638b1b23a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:44:24.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.233+0000 7f7bada00640 1 --2- 172.21.15.43:0/2920933994 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638b1b1f880 0x5638b1b23fa0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f7ba002c700 tx=0x7f7ba00095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1459894bb34cb0a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:24.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.233+0000 7f7b8ea00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:44:24.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.233+0000 7f7b8ea00640 1 -- 172.21.15.43:0/2920933994 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0039040 con 0x5638b1b1f880 2024-03-28T22:44:24.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.233+0000 7f7b8ea00640 1 -- 172.21.15.43:0/2920933994 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f7ba004f440 con 0x5638b1b1f880 2024-03-28T22:44:24.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.233+0000 7f7bb280c740 1 -- 172.21.15.43:0/2920933994 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b94028070 con 0x5638b1b1f880 2024-03-28T22:44:24.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.233+0000 7f7b8ea00640 1 -- 172.21.15.43:0/2920933994 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7ba004f640 con 0x5638b1b1f880 2024-03-28T22:44:24.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.233+0000 7f7bb280c740 10 client.28292 fetch_fsmap learned FSMap version 32 2024-03-28T22:44:24.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.233+0000 7f7bb280c740 1 -- 172.21.15.43:0/2920933994 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5638b1b963c0 con 0x5638b1b1f880 2024-03-28T22:44:24.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.233+0000 7f7b8ea00640 1 -- 172.21.15.43:0/2920933994 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b94028070 con 0x5638b1b1f880 2024-03-28T22:44:24.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.234+0000 7f7b8ea00640 1 -- 172.21.15.43:0/2920933994 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7ba0034ab0 con 0x5638b1b1f880 2024-03-28T22:44:24.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.234+0000 7f7bb280c740 10 client.28292 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:44:24.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.234+0000 7f7bb280c740 10 client.28292 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:44:24.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.234+0000 7f7bb280c740 20 client.28292 populate_metadata read hostname 'smithi043' 2024-03-28T22:44:24.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.234+0000 7f7bb280c740 1 --2- 172.21.15.43:0/2920933994 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5638b1b96b40 0x5638b1b98f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:24.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.234+0000 7f7bb280c740 4 client.28292 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:44:24.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.234+0000 7f7bb280c740 1 -- 172.21.15.43:0/2920933994 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5638b1a87590 con 0x5638b1b96b40 2024-03-28T22:44:24.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.234+0000 7f7bad000640 1 --2- 172.21.15.43:0/2920933994 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5638b1b96b40 0x5638b1b98f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:24.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.234+0000 7f7bad000640 1 --2- 172.21.15.43:0/2920933994 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5638b1b96b40 0x5638b1b98f60 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b84a525bcda3acd2 server_cookie=22ff1fa02aeeab72 in_seq=0 out_seq=0 2024-03-28T22:44:24.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.234+0000 7f7b8ea00640 10 client.28292 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:44:24.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.235+0000 7f7b8ea00640 1 -- 172.21.15.43:0/2920933994 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5638b1a87590 con 0x5638b1b96b40 2024-03-28T22:44:24.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.235+0000 7f7b8ea00640 10 client.28292 handle_command_reply: tid=0 2024-03-28T22:44:24.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.235+0000 7f7bb280c740 1 client.28292 shutdown 2024-03-28T22:44:24.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.235+0000 7f7bb280c740 1 -- 172.21.15.43:0/2920933994 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638b1b1f880 msgr2=0x5638b1b23fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:24.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.235+0000 7f7bb280c740 1 --2- 172.21.15.43:0/2920933994 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638b1b1f880 0x5638b1b23fa0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f7ba002c700 tx=0x7f7ba00095c0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.235+0000 7f7bb280c740 1 -- 172.21.15.43:0/2920933994 shutdown_connections 2024-03-28T22:44:24.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.235+0000 7f7bb280c740 1 --2- 172.21.15.43:0/2920933994 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5638b1b96b40 0x5638b1b98f60 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.235+0000 7f7bb280c740 1 --2- 172.21.15.43:0/2920933994 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638b1b1f880 0x5638b1b23fa0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.235+0000 7f7bb280c740 1 --2- 172.21.15.43:0/2920933994 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5638b1b16ca0 0x5638b1b23a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.235+0000 7f7bb280c740 1 --2- 172.21.15.43:0/2920933994 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5638b1b16380 0x5638b1b23520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.235+0000 7f7b8ea00640 0 client.28292 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:44:24.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.235+0000 7f7bb280c740 1 -- 172.21.15.43:0/2920933994 >> 172.21.15.43:0/2920933994 conn(0x5638b1b10ab0 msgr2=0x5638b1b126e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:24.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.235+0000 7f7bb280c740 1 -- 172.21.15.43:0/2920933994 shutdown_connections 2024-03-28T22:44:24.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.235+0000 7f7bb280c740 1 -- 172.21.15.43:0/2920933994 wait complete. 2024-03-28T22:44:24.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.235+0000 7f7bb280c740 20 client.28292 trim_cache size 0 max 16384 2024-03-28T22:44:24.242 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:44:24.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.236+0000 7f7bafa00640 1 -- 172.21.15.43:0/1527165158 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7b900a1f70 msgr2=0x7f7b900a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:24.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.236+0000 7f7bafa00640 1 --2- 172.21.15.43:0/1527165158 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7b900a1f70 0x7f7b900a4430 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0005e30 tx=0x7f7ba0005d80 comp rx=0 tx=0).stop 2024-03-28T22:44:24.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.236+0000 7f7bafa00640 1 -- 172.21.15.43:0/1527165158 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ba8107f60 msgr2=0x7f7ba819e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:24.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.236+0000 7f7bafa00640 1 --2- 172.21.15.43:0/1527165158 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ba8107f60 0x7f7ba819e580 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c00e9f0 tx=0x7f7b9c00eeb0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.236+0000 7f7bafa00640 1 -- 172.21.15.43:0/1527165158 shutdown_connections 2024-03-28T22:44:24.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.236+0000 7f7bafa00640 1 --2- 172.21.15.43:0/1527165158 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7b900a1f70 0x7f7b900a4430 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.236+0000 7f7bafa00640 1 --2- 172.21.15.43:0/1527165158 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7ba81a2e40 0x7f7ba81a32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.236+0000 7f7bafa00640 1 --2- 172.21.15.43:0/1527165158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ba810d030 0x7f7ba819eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.236+0000 7f7bafa00640 1 --2- 172.21.15.43:0/1527165158 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7ba8107f60 0x7f7ba819e580 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:24.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.236+0000 7f7bafa00640 1 -- 172.21.15.43:0/1527165158 >> 172.21.15.43:0/1527165158 conn(0x7f7ba8075c90 msgr2=0x7f7ba8077f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:24.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.236+0000 7f7bafa00640 1 -- 172.21.15.43:0/1527165158 shutdown_connections 2024-03-28T22:44:24.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:24.236+0000 7f7bafa00640 1 -- 172.21.15.43:0/1527165158 wait complete. 2024-03-28T22:44:25.099 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:24 smithi082 ceph-mon[36471]: pgmap v224: 129 pgs: 129 active+clean; 48 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 750 KiB/s wr, 62 op/s 2024-03-28T22:44:25.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:24 smithi172 ceph-mon[36566]: pgmap v224: 129 pgs: 129 active+clean; 48 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 750 KiB/s wr, 62 op/s 2024-03-28T22:44:25.251 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:44:25.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:44:25.252 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:44:25.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:24 smithi043 ceph-mon[31823]: pgmap v224: 129 pgs: 129 active+clean; 48 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 750 KiB/s wr, 62 op/s 2024-03-28T22:44:25.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.494+0000 7fee0b200640 1 -- 172.21.15.43:0/1211897068 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee0c1097e0 msgr2=0x7fee0c109bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:25.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.494+0000 7fee0b200640 1 --2- 172.21.15.43:0/1211897068 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee0c1097e0 0x7fee0c109bc0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fedfc009930 tx=0x7fedfc02bc30 comp rx=0 tx=0).stop 2024-03-28T22:44:25.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.494+0000 7fee0b200640 1 -- 172.21.15.43:0/1211897068 shutdown_connections 2024-03-28T22:44:25.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.495+0000 7fee0b200640 1 --2- 172.21.15.43:0/1211897068 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fee0c1037d0 0x7fee0c103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.495+0000 7fee0b200640 1 --2- 172.21.15.43:0/1211897068 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee0c104190 0x7fee0c1106a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.495+0000 7fee0b200640 1 --2- 172.21.15.43:0/1211897068 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee0c1097e0 0x7fee0c109bc0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.495+0000 7fee0b200640 1 -- 172.21.15.43:0/1211897068 >> 172.21.15.43:0/1211897068 conn(0x7fee0c0ff260 msgr2=0x7fee0c101680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:25.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.495+0000 7fee0b200640 1 -- 172.21.15.43:0/1211897068 shutdown_connections 2024-03-28T22:44:25.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.495+0000 7fee0b200640 1 -- 172.21.15.43:0/1211897068 wait complete. 2024-03-28T22:44:25.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.496+0000 7fee0b200640 1 Processor -- start 2024-03-28T22:44:25.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.496+0000 7fee0b200640 1 -- start start 2024-03-28T22:44:25.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.496+0000 7fee0b200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee0c1037d0 0x7fee0c19a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.496+0000 7fee0b200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee0c104190 0x7fee0c19a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.496+0000 7fee0b200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fee0c1097e0 0x7fee0c19ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.496+0000 7fee0b200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c113c40 con 0x7fee0c104190 2024-03-28T22:44:25.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.496+0000 7fee0b200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c113ac0 con 0x7fee0c1097e0 2024-03-28T22:44:25.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.496+0000 7fee0b200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c113dc0 con 0x7fee0c1037d0 2024-03-28T22:44:25.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.497+0000 7fee09e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee0c1037d0 0x7fee0c19a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.497+0000 7fee09e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee0c1037d0 0x7fee0c19a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:36756/0 (socket says 172.21.15.43:36756) 2024-03-28T22:44:25.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.497+0000 7fee09e00640 1 -- 172.21.15.43:0/2195040637 learned_addr learned my addr 172.21.15.43:0/2195040637 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:25.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.497+0000 7fee0a800640 1 --2- 172.21.15.43:0/2195040637 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fee0c1097e0 0x7fee0c19ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.497+0000 7fee03e00640 1 --2- 172.21.15.43:0/2195040637 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee0c104190 0x7fee0c19a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.497+0000 7fee09e00640 1 -- 172.21.15.43:0/2195040637 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fee0c1097e0 msgr2=0x7fee0c19ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:25.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.497+0000 7fee09e00640 1 --2- 172.21.15.43:0/2195040637 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fee0c1097e0 0x7fee0c19ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.497+0000 7fee09e00640 1 -- 172.21.15.43:0/2195040637 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee0c104190 msgr2=0x7fee0c19a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:25.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.497+0000 7fee09e00640 1 --2- 172.21.15.43:0/2195040637 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee0c104190 0x7fee0c19a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.497+0000 7fee09e00640 1 -- 172.21.15.43:0/2195040637 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee0c19f3a0 con 0x7fee0c1037d0 2024-03-28T22:44:25.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.497+0000 7fee03e00640 1 --2- 172.21.15.43:0/2195040637 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee0c104190 0x7fee0c19a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:44:25.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.497+0000 7fee0a800640 1 --2- 172.21.15.43:0/2195040637 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fee0c1097e0 0x7fee0c19ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:44:25.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.497+0000 7fee09e00640 1 --2- 172.21.15.43:0/2195040637 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee0c1037d0 0x7fee0c19a3b0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fedfc009900 tx=0x7fedfc004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:25.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.498+0000 7fee02a00640 1 -- 172.21.15.43:0/2195040637 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedfc043020 con 0x7fee0c1037d0 2024-03-28T22:44:25.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.498+0000 7fee02a00640 1 -- 172.21.15.43:0/2195040637 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedfc0054e0 con 0x7fee0c1037d0 2024-03-28T22:44:25.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.498+0000 7fee0b200640 1 -- 172.21.15.43:0/2195040637 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee0c19f630 con 0x7fee0c1037d0 2024-03-28T22:44:25.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.498+0000 7fee02a00640 1 -- 172.21.15.43:0/2195040637 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedfc003b80 con 0x7fee0c1037d0 2024-03-28T22:44:25.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.498+0000 7fee0b200640 1 -- 172.21.15.43:0/2195040637 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee0c1a6f20 con 0x7fee0c1037d0 2024-03-28T22:44:25.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.501+0000 7fee13e7b740 1 Processor -- start 2024-03-28T22:44:25.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.501+0000 7fee13e7b740 1 -- start start 2024-03-28T22:44:25.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.502+0000 7fee02a00640 1 -- 172.21.15.43:0/2195040637 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fedfc046020 con 0x7fee0c1037d0 2024-03-28T22:44:25.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.502+0000 7fee13e7b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639f0f11170 0x5639f0f11550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.502+0000 7fee13e7b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639f0f11a90 0x5639f0f1a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.502+0000 7fee13e7b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639f0f1a670 0x5639f0f1ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.502+0000 7fee13e7b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5639f0e74c40 con 0x5639f0f11a90 2024-03-28T22:44:25.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.502+0000 7fee13e7b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5639f0e1dc40 con 0x5639f0f1a670 2024-03-28T22:44:25.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.502+0000 7fee13e7b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5639f0e70ce0 con 0x5639f0f11170 2024-03-28T22:44:25.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.502+0000 7fee0a800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639f0f11170 0x5639f0f11550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.502+0000 7fee0a800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639f0f11170 0x5639f0f11550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:36770/0 (socket says 172.21.15.43:36770) 2024-03-28T22:44:25.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.502+0000 7fee0a800640 1 -- 172.21.15.43:0/843481212 learned_addr learned my addr 172.21.15.43:0/843481212 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:25.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.502+0000 7fee03e00640 1 --2- 172.21.15.43:0/843481212 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639f0f1a670 0x5639f0f1ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.502+0000 7fee09e00640 1 --2- 172.21.15.43:0/843481212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639f0f11a90 0x5639f0f1a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.503+0000 7fee0a800640 1 -- 172.21.15.43:0/843481212 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639f0f1a670 msgr2=0x5639f0f1ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:25.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.503+0000 7fee0a800640 1 --2- 172.21.15.43:0/843481212 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639f0f1a670 0x5639f0f1ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.503+0000 7fee0a800640 1 -- 172.21.15.43:0/843481212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639f0f11a90 msgr2=0x5639f0f1a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:25.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.503+0000 7fee0a800640 1 --2- 172.21.15.43:0/843481212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639f0f11a90 0x5639f0f1a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.503+0000 7fee0a800640 1 -- 172.21.15.43:0/843481212 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639f0c517e0 con 0x5639f0f11170 2024-03-28T22:44:25.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.503+0000 7fee0a800640 1 --2- 172.21.15.43:0/843481212 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639f0f11170 0x5639f0f11550 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fee0400f790 tx=0x7fee0400fc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87ed5bdac09fb3a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:25.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.503+0000 7fee02a00640 1 --2- 172.21.15.43:0/2195040637 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fedf00a1ef0 0x7fedf00a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.503+0000 7fedfb400640 1 -- 172.21.15.43:0/843481212 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee04004d20 con 0x5639f0f11170 2024-03-28T22:44:25.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.503+0000 7fedfb400640 1 -- 172.21.15.43:0/843481212 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee0400de00 con 0x5639f0f11170 2024-03-28T22:44:25.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.503+0000 7fedfb400640 1 -- 172.21.15.43:0/843481212 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee04010430 con 0x5639f0f11170 2024-03-28T22:44:25.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.503+0000 7fee02a00640 1 -- 172.21.15.43:0/2195040637 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7fedfc0f0660 con 0x7fee0c1037d0 2024-03-28T22:44:25.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.504+0000 7fee09e00640 1 --2- 172.21.15.43:0/2195040637 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fedf00a1ef0 0x7fedf00a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.504+0000 7fee13e7b740 1 -- 172.21.15.43:0/843481212 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639f0f11170 msgr2=0x5639f0f11550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:25.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.504+0000 7fee13e7b740 1 --2- 172.21.15.43:0/843481212 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639f0f11170 0x5639f0f11550 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fee0400f790 tx=0x7fee0400fc50 comp rx=0 tx=0).stop 2024-03-28T22:44:25.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.504+0000 7fee09e00640 1 --2- 172.21.15.43:0/2195040637 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fedf00a1ef0 0x7fedf00a43b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x5639f0f10c40 tx=0x7fedfc004610 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:25.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.504+0000 7fee13e7b740 1 -- 172.21.15.43:0/843481212 shutdown_connections 2024-03-28T22:44:25.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.504+0000 7fee13e7b740 1 --2- 172.21.15.43:0/843481212 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639f0f1a670 0x5639f0f1ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.504+0000 7fee13e7b740 1 --2- 172.21.15.43:0/843481212 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639f0f11a90 0x5639f0f1a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.504+0000 7fee13e7b740 1 --2- 172.21.15.43:0/843481212 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639f0f11170 0x5639f0f11550 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.504+0000 7fee13e7b740 1 -- 172.21.15.43:0/843481212 >> 172.21.15.43:0/843481212 conn(0x5639f0f0b8a0 msgr2=0x5639f0f0bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:25.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.504+0000 7fee13e7b740 1 -- 172.21.15.43:0/843481212 shutdown_connections 2024-03-28T22:44:25.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.504+0000 7fee13e7b740 1 -- 172.21.15.43:0/843481212 wait complete. 2024-03-28T22:44:25.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.505+0000 7fee13e7b740 1 -- start start 2024-03-28T22:44:25.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.505+0000 7fee13e7b740 1 Processor -- start 2024-03-28T22:44:25.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.505+0000 7fee13e7b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639f0f11170 0x5639f0f176a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.505+0000 7fee13e7b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639f0f11a90 0x5639f0f17be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.506+0000 7fee13e7b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639f0f1a670 0x5639f0f18120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.506+0000 7fee13e7b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5639f0f1d460 con 0x5639f0f11170 2024-03-28T22:44:25.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.506+0000 7fee13e7b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5639f0e13250 con 0x5639f0f1a670 2024-03-28T22:44:25.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.506+0000 7fee13e7b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5639f0f1d5e0 con 0x5639f0f11a90 2024-03-28T22:44:25.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.506+0000 7fee13e7b740 10 client.0 _ll_register_callbacks cb 0x5639f0e34e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:44:25.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.506+0000 7fee03e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639f0f11a90 0x5639f0f17be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.506+0000 7fee0a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639f0f11170 0x5639f0f176a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.506+0000 7fee03e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639f0f11a90 0x5639f0f17be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:36780/0 (socket says 172.21.15.43:36780) 2024-03-28T22:44:25.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.506+0000 7fee03e00640 1 -- 172.21.15.43:0/436179285 learned_addr learned my addr 172.21.15.43:0/436179285 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:25.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.506+0000 7fee0a800640 1 --2- 172.21.15.43:0/436179285 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639f0f1a670 0x5639f0f18120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.506+0000 7fee03e00640 1 -- 172.21.15.43:0/436179285 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639f0f1a670 msgr2=0x5639f0f18120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:25.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.506+0000 7fee03e00640 1 --2- 172.21.15.43:0/436179285 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639f0f1a670 0x5639f0f18120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.506+0000 7fee03e00640 1 -- 172.21.15.43:0/436179285 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639f0f11170 msgr2=0x5639f0f176a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:25.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.506+0000 7fee03e00640 1 --2- 172.21.15.43:0/436179285 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639f0f11170 0x5639f0f176a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.506+0000 7fee03e00640 1 -- 172.21.15.43:0/436179285 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639f0cb1f70 con 0x5639f0f11a90 2024-03-28T22:44:25.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.506+0000 7fee0a800640 1 --2- 172.21.15.43:0/436179285 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639f0f11170 0x5639f0f176a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:44:25.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.506+0000 7fee03e00640 1 --2- 172.21.15.43:0/436179285 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639f0f11a90 0x5639f0f17be0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fedf4013740 tx=0x7fedf4009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ae5bae7750d2cb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:25.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.507+0000 7fedefe00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:44:25.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.507+0000 7fedefe00640 1 -- 172.21.15.43:0/436179285 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf4005020 con 0x5639f0f11a90 2024-03-28T22:44:25.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.507+0000 7fedefe00640 1 -- 172.21.15.43:0/436179285 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7fedf40051c0 con 0x5639f0f11a90 2024-03-28T22:44:25.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.507+0000 7fee13e7b740 1 -- 172.21.15.43:0/436179285 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee0400de00 con 0x5639f0f11a90 2024-03-28T22:44:25.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.507+0000 7fedefe00640 1 -- 172.21.15.43:0/436179285 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fedf4026050 con 0x5639f0f11a90 2024-03-28T22:44:25.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.507+0000 7fee13e7b740 10 client.28243 fetch_fsmap learned FSMap version 32 2024-03-28T22:44:25.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.507+0000 7fee13e7b740 1 -- 172.21.15.43:0/436179285 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5639f0b49170 con 0x5639f0f11a90 2024-03-28T22:44:25.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.507+0000 7fedefe00640 1 -- 172.21.15.43:0/436179285 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee0400de00 con 0x5639f0f11a90 2024-03-28T22:44:25.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.508+0000 7fedefe00640 1 -- 172.21.15.43:0/436179285 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fedf401a030 con 0x5639f0f11a90 2024-03-28T22:44:25.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.508+0000 7fee13e7b740 10 client.28243 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:44:25.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.508+0000 7fee13e7b740 10 client.28243 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:44:25.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.508+0000 7fee13e7b740 20 client.28243 populate_metadata read hostname 'smithi043' 2024-03-28T22:44:25.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.508+0000 7fee13e7b740 1 --2- 172.21.15.43:0/436179285 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5639f0f2aa50 0x5639f0f2ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.508+0000 7fee13e7b740 4 client.28243 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:44:25.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.508+0000 7fee13e7b740 1 -- 172.21.15.43:0/436179285 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5639f0e82180 con 0x5639f0f2aa50 2024-03-28T22:44:25.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.508+0000 7fee0a800640 1 --2- 172.21.15.43:0/436179285 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5639f0f2aa50 0x5639f0f2ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.509+0000 7fee0a800640 1 --2- 172.21.15.43:0/436179285 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5639f0f2aa50 0x5639f0f2ceb0 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c91d3d4098cf14ed server_cookie=2a0a8b320c4780e0 in_seq=0 out_seq=0 2024-03-28T22:44:25.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.509+0000 7fedefe00640 10 client.28243 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:44:25.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.510+0000 7fedefe00640 1 -- 172.21.15.43:0/436179285 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5639f0e82180 con 0x5639f0f2aa50 2024-03-28T22:44:25.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.510+0000 7fedefe00640 10 client.28243 handle_command_reply: tid=0 2024-03-28T22:44:25.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.510+0000 7fee13e7b740 1 client.28243 shutdown 2024-03-28T22:44:25.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.510+0000 7fee13e7b740 1 -- 172.21.15.43:0/436179285 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639f0f11a90 msgr2=0x5639f0f17be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:25.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.510+0000 7fee13e7b740 1 --2- 172.21.15.43:0/436179285 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639f0f11a90 0x5639f0f17be0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fedf4013740 tx=0x7fedf4009f90 comp rx=0 tx=0).stop 2024-03-28T22:44:25.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.510+0000 7fee13e7b740 1 -- 172.21.15.43:0/436179285 shutdown_connections 2024-03-28T22:44:25.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.510+0000 7fee13e7b740 1 --2- 172.21.15.43:0/436179285 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5639f0f2aa50 0x5639f0f2ceb0 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.510+0000 7fee13e7b740 1 --2- 172.21.15.43:0/436179285 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639f0f1a670 0x5639f0f18120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.510+0000 7fee13e7b740 1 --2- 172.21.15.43:0/436179285 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639f0f11a90 0x5639f0f17be0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.510+0000 7fee13e7b740 1 --2- 172.21.15.43:0/436179285 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639f0f11170 0x5639f0f176a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.510+0000 7fedefe00640 0 client.28243 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:44:25.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.510+0000 7fee13e7b740 1 -- 172.21.15.43:0/436179285 >> 172.21.15.43:0/436179285 conn(0x5639f0f0b8a0 msgr2=0x5639f0f0d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:25.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.510+0000 7fee13e7b740 1 -- 172.21.15.43:0/436179285 shutdown_connections 2024-03-28T22:44:25.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.510+0000 7fee13e7b740 1 -- 172.21.15.43:0/436179285 wait complete. 2024-03-28T22:44:25.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.510+0000 7fee13e7b740 20 client.28243 trim_cache size 0 max 16384 2024-03-28T22:44:25.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.530+0000 7fee13e7b740 1 Processor -- start 2024-03-28T22:44:25.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.530+0000 7fee13e7b740 1 -- start start 2024-03-28T22:44:25.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.530+0000 7fee13e7b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639f0f0b8a0 0x5639f0f19dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.530+0000 7fee13e7b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639f0f11170 0x5639f0f26200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.530+0000 7fee13e7b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639f0f11a90 0x5639f0f32780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.530+0000 7fee13e7b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c1b0c10 con 0x5639f0f11a90 2024-03-28T22:44:25.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.530+0000 7fee13e7b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c1b2750 con 0x5639f0f0b8a0 2024-03-28T22:44:25.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.530+0000 7fee13e7b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c1b12e0 con 0x5639f0f11170 2024-03-28T22:44:25.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.531+0000 7fee03e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639f0f11170 0x5639f0f26200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.531+0000 7fee03e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639f0f11170 0x5639f0f26200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:36790/0 (socket says 172.21.15.43:36790) 2024-03-28T22:44:25.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.531+0000 7fee03e00640 1 -- 172.21.15.43:0/1382103924 learned_addr learned my addr 172.21.15.43:0/1382103924 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:25.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.531+0000 7fee0a800640 1 --2- 172.21.15.43:0/1382103924 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639f0f0b8a0 0x5639f0f19dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.531+0000 7fee0a800640 1 --2- 172.21.15.43:0/1382103924 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639f0f11a90 0x5639f0f32780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.531+0000 7fee03e00640 1 -- 172.21.15.43:0/1382103924 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639f0f0b8a0 msgr2=0x5639f0f19dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:25.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.531+0000 7fee03e00640 1 --2- 172.21.15.43:0/1382103924 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639f0f0b8a0 0x5639f0f19dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.531+0000 7fee03e00640 1 -- 172.21.15.43:0/1382103924 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639f0f11a90 msgr2=0x5639f0f32780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:25.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.531+0000 7fee03e00640 1 --2- 172.21.15.43:0/1382103924 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639f0f11a90 0x5639f0f32780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.531+0000 7fee03e00640 1 -- 172.21.15.43:0/1382103924 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639f0f2d530 con 0x5639f0f11170 2024-03-28T22:44:25.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.531+0000 7fee0a800640 1 --2- 172.21.15.43:0/1382103924 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639f0f11a90 0x5639f0f32780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:44:25.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.531+0000 7fee03e00640 1 --2- 172.21.15.43:0/1382103924 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639f0f11170 0x5639f0f26200 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fedf401b4a0 tx=0x7fedf4009d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25fba456bfd66ec3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:25.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.532+0000 7fedfaa00640 1 -- 172.21.15.43:0/1382103924 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf4011cf0 con 0x5639f0f11170 2024-03-28T22:44:25.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.532+0000 7fedfaa00640 1 -- 172.21.15.43:0/1382103924 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedf4011e90 con 0x5639f0f11170 2024-03-28T22:44:25.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.532+0000 7fedfaa00640 1 -- 172.21.15.43:0/1382103924 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf400e3e0 con 0x5639f0f11170 2024-03-28T22:44:25.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.532+0000 7fee13e7b740 1 -- 172.21.15.43:0/1382103924 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639f0f11170 msgr2=0x5639f0f26200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:25.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.532+0000 7fee13e7b740 1 --2- 172.21.15.43:0/1382103924 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639f0f11170 0x5639f0f26200 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fedf401b4a0 tx=0x7fedf4009d30 comp rx=0 tx=0).stop 2024-03-28T22:44:25.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.532+0000 7fee13e7b740 1 -- 172.21.15.43:0/1382103924 shutdown_connections 2024-03-28T22:44:25.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.532+0000 7fee13e7b740 1 --2- 172.21.15.43:0/1382103924 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639f0f11a90 0x5639f0f32780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.532+0000 7fee13e7b740 1 --2- 172.21.15.43:0/1382103924 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639f0f11170 0x5639f0f26200 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.532+0000 7fee13e7b740 1 --2- 172.21.15.43:0/1382103924 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639f0f0b8a0 0x5639f0f19dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.532+0000 7fee13e7b740 1 -- 172.21.15.43:0/1382103924 >> 172.21.15.43:0/1382103924 conn(0x5639f0f16300 msgr2=0x5639f0f17ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:25.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.532+0000 7fee13e7b740 1 -- 172.21.15.43:0/1382103924 shutdown_connections 2024-03-28T22:44:25.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.532+0000 7fee13e7b740 1 -- 172.21.15.43:0/1382103924 wait complete. 2024-03-28T22:44:25.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.533+0000 7fee13e7b740 1 -- start start 2024-03-28T22:44:25.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.533+0000 7fee13e7b740 1 Processor -- start 2024-03-28T22:44:25.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.533+0000 7fee13e7b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639f0f0b8a0 0x5639f0f1f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.533+0000 7fee13e7b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639f0f11170 0x5639f0f1fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.533+0000 7fee13e7b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639f0f11a90 0x5639f0f20280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.533+0000 7fee13e7b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c1b2080 con 0x5639f0f11170 2024-03-28T22:44:25.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.533+0000 7fee13e7b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c1b19b0 con 0x5639f0f11a90 2024-03-28T22:44:25.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.533+0000 7fee13e7b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c1aec10 con 0x5639f0f0b8a0 2024-03-28T22:44:25.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.533+0000 7fee13e7b740 10 client.0 _ll_register_callbacks cb 0x7fedf4009860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:44:25.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.534+0000 7fee0a800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639f0f0b8a0 0x5639f0f1f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.534+0000 7fee0a800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639f0f0b8a0 0x5639f0f1f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:36794/0 (socket says 172.21.15.43:36794) 2024-03-28T22:44:25.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.534+0000 7fee03e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639f0f11170 0x5639f0f1fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.534+0000 7fee0a800640 1 -- 172.21.15.43:0/3741265482 learned_addr learned my addr 172.21.15.43:0/3741265482 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:25.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.534+0000 7fee0a800640 1 --2- 172.21.15.43:0/3741265482 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639f0f11a90 0x5639f0f20280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.534+0000 7fee0a800640 1 -- 172.21.15.43:0/3741265482 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639f0f11a90 msgr2=0x5639f0f20280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:25.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.534+0000 7fee0a800640 1 --2- 172.21.15.43:0/3741265482 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639f0f11a90 0x5639f0f20280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.534+0000 7fee0a800640 1 -- 172.21.15.43:0/3741265482 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639f0f11170 msgr2=0x5639f0f1fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:25.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.534+0000 7fee0a800640 1 --2- 172.21.15.43:0/3741265482 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639f0f11170 0x5639f0f1fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.534+0000 7fee0a800640 1 -- 172.21.15.43:0/3741265482 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639f0f99620 con 0x5639f0f0b8a0 2024-03-28T22:44:25.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.534+0000 7fee0a800640 1 --2- 172.21.15.43:0/3741265482 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639f0f0b8a0 0x5639f0f1f800 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fee04007560 tx=0x7fee04012040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7d08a04d8eabaf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:25.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.534+0000 7fedefe00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:44:25.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.534+0000 7fedefe00640 1 -- 172.21.15.43:0/3741265482 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee0401d040 con 0x5639f0f0b8a0 2024-03-28T22:44:25.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.534+0000 7fee13e7b740 1 -- 172.21.15.43:0/3741265482 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedf4011e90 con 0x5639f0f0b8a0 2024-03-28T22:44:25.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.534+0000 7fedefe00640 1 -- 172.21.15.43:0/3741265482 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7fee04003b70 con 0x5639f0f0b8a0 2024-03-28T22:44:25.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.535+0000 7fedefe00640 1 -- 172.21.15.43:0/3741265482 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fee040051a0 con 0x5639f0f0b8a0 2024-03-28T22:44:25.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.535+0000 7fee13e7b740 10 client.28249 fetch_fsmap learned FSMap version 32 2024-03-28T22:44:25.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.535+0000 7fee13e7b740 1 -- 172.21.15.43:0/3741265482 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5639f0f9a920 con 0x5639f0f0b8a0 2024-03-28T22:44:25.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.535+0000 7fedefe00640 1 -- 172.21.15.43:0/3741265482 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedf4011e90 con 0x5639f0f0b8a0 2024-03-28T22:44:25.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.535+0000 7fedefe00640 1 -- 172.21.15.43:0/3741265482 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fee04023070 con 0x5639f0f0b8a0 2024-03-28T22:44:25.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.535+0000 7fee13e7b740 10 client.28249 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:44:25.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.535+0000 7fee13e7b740 10 client.28249 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:44:25.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.535+0000 7fee13e7b740 20 client.28249 populate_metadata read hostname 'smithi043' 2024-03-28T22:44:25.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.535+0000 7fee13e7b740 1 --2- 172.21.15.43:0/3741265482 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5639f0f1a670 0x5639f0f9cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.535+0000 7fee13e7b740 4 client.28249 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:44:25.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.535+0000 7fee13e7b740 1 -- 172.21.15.43:0/3741265482 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5639f0e7c4d0 con 0x5639f0f1a670 2024-03-28T22:44:25.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.535+0000 7fee03e00640 1 --2- 172.21.15.43:0/3741265482 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5639f0f1a670 0x5639f0f9cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.536+0000 7fee03e00640 1 --2- 172.21.15.43:0/3741265482 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5639f0f1a670 0x5639f0f9cd30 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8982abe29dbcef91 server_cookie=39af7e787a72f6cd in_seq=0 out_seq=0 2024-03-28T22:44:25.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.536+0000 7fedefe00640 10 client.28249 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:44:25.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.537+0000 7fedefe00640 1 -- 172.21.15.43:0/3741265482 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5639f0e7c4d0 con 0x5639f0f1a670 2024-03-28T22:44:25.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.537+0000 7fedefe00640 10 client.28249 handle_command_reply: tid=0 2024-03-28T22:44:25.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.537+0000 7fee13e7b740 1 client.28249 shutdown 2024-03-28T22:44:25.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.538+0000 7fee13e7b740 1 -- 172.21.15.43:0/3741265482 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639f0f0b8a0 msgr2=0x5639f0f1f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:25.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.538+0000 7fee13e7b740 1 --2- 172.21.15.43:0/3741265482 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639f0f0b8a0 0x5639f0f1f800 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fee04007560 tx=0x7fee04012040 comp rx=0 tx=0).stop 2024-03-28T22:44:25.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.538+0000 7fee13e7b740 1 -- 172.21.15.43:0/3741265482 shutdown_connections 2024-03-28T22:44:25.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.538+0000 7fee13e7b740 1 --2- 172.21.15.43:0/3741265482 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5639f0f1a670 0x5639f0f9cd30 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.538+0000 7fee13e7b740 1 --2- 172.21.15.43:0/3741265482 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639f0f11a90 0x5639f0f20280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.538+0000 7fee13e7b740 1 --2- 172.21.15.43:0/3741265482 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639f0f11170 0x5639f0f1fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.538+0000 7fee13e7b740 1 --2- 172.21.15.43:0/3741265482 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639f0f0b8a0 0x5639f0f1f800 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.538+0000 7fedefe00640 0 client.28249 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:44:25.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.538+0000 7fee13e7b740 1 -- 172.21.15.43:0/3741265482 >> 172.21.15.43:0/3741265482 conn(0x5639f0f16300 msgr2=0x5639f0f0a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:25.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.538+0000 7fee13e7b740 1 -- 172.21.15.43:0/3741265482 shutdown_connections 2024-03-28T22:44:25.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.538+0000 7fee13e7b740 1 -- 172.21.15.43:0/3741265482 wait complete. 2024-03-28T22:44:25.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.538+0000 7fee13e7b740 20 client.28249 trim_cache size 0 max 16384 2024-03-28T22:44:25.544 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:44:25.544 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-03-28T22:44:25.544 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "59e64d15-9226-41b3-83d0-096cdc02381a", 2024-03-28T22:44:25.544 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-03-28T22:44:25.544 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:44:25.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.539+0000 7fee0b200640 1 -- 172.21.15.43:0/2195040637 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fedf00a1ef0 msgr2=0x7fedf00a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:25.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.539+0000 7fee0b200640 1 --2- 172.21.15.43:0/2195040637 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fedf00a1ef0 0x7fedf00a43b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x5639f0f10c40 tx=0x7fedfc004610 comp rx=0 tx=0).stop 2024-03-28T22:44:25.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.539+0000 7fee0b200640 1 -- 172.21.15.43:0/2195040637 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee0c1037d0 msgr2=0x7fee0c19a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:25.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.539+0000 7fee0b200640 1 --2- 172.21.15.43:0/2195040637 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee0c1037d0 0x7fee0c19a3b0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fedfc009900 tx=0x7fedfc004360 comp rx=0 tx=0).stop 2024-03-28T22:44:25.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.539+0000 7fee0b200640 1 -- 172.21.15.43:0/2195040637 shutdown_connections 2024-03-28T22:44:25.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.539+0000 7fee0b200640 1 --2- 172.21.15.43:0/2195040637 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fedf00a1ef0 0x7fedf00a43b0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.539+0000 7fee0b200640 1 --2- 172.21.15.43:0/2195040637 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fee0c1097e0 0x7fee0c19ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.539+0000 7fee0b200640 1 --2- 172.21.15.43:0/2195040637 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fee0c104190 0x7fee0c19a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.539+0000 7fee0b200640 1 --2- 172.21.15.43:0/2195040637 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fee0c1037d0 0x7fee0c19a3b0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.539+0000 7fee0b200640 1 -- 172.21.15.43:0/2195040637 >> 172.21.15.43:0/2195040637 conn(0x7fee0c0ff260 msgr2=0x7fee0c0ffb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:25.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.539+0000 7fee0b200640 1 -- 172.21.15.43:0/2195040637 shutdown_connections 2024-03-28T22:44:25.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.539+0000 7fee0b200640 1 -- 172.21.15.43:0/2195040637 wait complete. 2024-03-28T22:44:25.557 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:44:25.557 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:44:25.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.836+0000 7f28d0c00640 1 -- 172.21.15.43:0/413196969 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f28cc108910 msgr2=0x7f28cc10c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:25.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.836+0000 7f28d0c00640 1 --2- 172.21.15.43:0/413196969 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f28cc108910 0x7f28cc10c950 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f28b4009900 tx=0x7f28b402bc40 comp rx=0 tx=0).stop 2024-03-28T22:44:25.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.836+0000 7f28d0c00640 1 -- 172.21.15.43:0/413196969 shutdown_connections 2024-03-28T22:44:25.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.836+0000 7f28d0c00640 1 --2- 172.21.15.43:0/413196969 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f28cc108910 0x7f28cc10c950 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.836+0000 7f28d0c00640 1 --2- 172.21.15.43:0/413196969 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28cc10d030 0x7f28cc114c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.836+0000 7f28d0c00640 1 --2- 172.21.15.43:0/413196969 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28cc107f60 0x7f28cc108340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.836+0000 7f28d0c00640 1 -- 172.21.15.43:0/413196969 >> 172.21.15.43:0/413196969 conn(0x7f28cc075c90 msgr2=0x7f28cc0780b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:25.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.836+0000 7f28d0c00640 1 -- 172.21.15.43:0/413196969 shutdown_connections 2024-03-28T22:44:25.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.836+0000 7f28d0c00640 1 -- 172.21.15.43:0/413196969 wait complete. 2024-03-28T22:44:25.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.837+0000 7f28d0c00640 1 Processor -- start 2024-03-28T22:44:25.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.838+0000 7f28d0c00640 1 -- start start 2024-03-28T22:44:25.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.838+0000 7f28d0c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f28cc107f60 0x7f28cc19e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.838+0000 7f28d0c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28cc10d030 0x7f28cc19ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.838+0000 7f28d0c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28cc1a2f30 0x7f28cc1a33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.838+0000 7f28d0c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f28cc118110 con 0x7f28cc10d030 2024-03-28T22:44:25.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.838+0000 7f28d0c00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f28cc117f90 con 0x7f28cc107f60 2024-03-28T22:44:25.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.838+0000 7f28d0c00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f28cc118290 con 0x7f28cc1a2f30 2024-03-28T22:44:25.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.838+0000 7f28c9e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f28cc107f60 0x7f28cc19e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.838+0000 7f28c9e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f28cc107f60 0x7f28cc19e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:56052/0 (socket says 172.21.15.43:56052) 2024-03-28T22:44:25.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.838+0000 7f28c9e00640 1 -- 172.21.15.43:0/2914452634 learned_addr learned my addr 172.21.15.43:0/2914452634 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:25.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.838+0000 7f28ca800640 1 --2- 172.21.15.43:0/2914452634 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28cc1a2f30 0x7f28cc1a33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.839+0000 7f28c9400640 1 --2- 172.21.15.43:0/2914452634 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28cc10d030 0x7f28cc19ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.839+0000 7f28c9e00640 1 -- 172.21.15.43:0/2914452634 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28cc1a2f30 msgr2=0x7f28cc1a33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:25.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.839+0000 7f28c9e00640 1 --2- 172.21.15.43:0/2914452634 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28cc1a2f30 0x7f28cc1a33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.839+0000 7f28c9e00640 1 -- 172.21.15.43:0/2914452634 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28cc10d030 msgr2=0x7f28cc19ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:25.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.839+0000 7f28c9e00640 1 --2- 172.21.15.43:0/2914452634 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28cc10d030 0x7f28cc19ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.839+0000 7f28c9e00640 1 -- 172.21.15.43:0/2914452634 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28cc1a3a20 con 0x7f28cc107f60 2024-03-28T22:44:25.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.839+0000 7f28c9400640 1 --2- 172.21.15.43:0/2914452634 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28cc10d030 0x7f28cc19ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:44:25.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.839+0000 7f28c9e00640 1 --2- 172.21.15.43:0/2914452634 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f28cc107f60 0x7f28cc19e670 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f28bc00eb90 tx=0x7f28bc00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:25.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.839+0000 7f28c2a00640 1 -- 172.21.15.43:0/2914452634 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28bc004d20 con 0x7f28cc107f60 2024-03-28T22:44:25.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.839+0000 7f28c2a00640 1 -- 172.21.15.43:0/2914452634 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28bc00ee90 con 0x7f28cc107f60 2024-03-28T22:44:25.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.839+0000 7f28c2a00640 1 -- 172.21.15.43:0/2914452634 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28bc010830 con 0x7f28cc107f60 2024-03-28T22:44:25.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.839+0000 7f28d0c00640 1 -- 172.21.15.43:0/2914452634 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28cc1a3d10 con 0x7f28cc107f60 2024-03-28T22:44:25.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.840+0000 7f28d0c00640 1 -- 172.21.15.43:0/2914452634 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28cc1ab480 con 0x7f28cc107f60 2024-03-28T22:44:25.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.843+0000 7f28c2a00640 1 -- 172.21.15.43:0/2914452634 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f28bc005020 con 0x7f28cc107f60 2024-03-28T22:44:25.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.843+0000 7f28d3a36740 1 Processor -- start 2024-03-28T22:44:25.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.843+0000 7f28d3a36740 1 -- start start 2024-03-28T22:44:25.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.844+0000 7f28d3a36740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559ac0736bc0 0x559ac0735910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.844+0000 7f28d3a36740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ac0735ee0 0x559ac0736360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.844+0000 7f28d3a36740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559ac073f250 0x559ac0741640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.844+0000 7f28d3a36740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x559ac069a100 con 0x559ac0735ee0 2024-03-28T22:44:25.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.844+0000 7f28d3a36740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x559ac0642ba0 con 0x559ac0736bc0 2024-03-28T22:44:25.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.844+0000 7f28d3a36740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x559ac06961a0 con 0x559ac073f250 2024-03-28T22:44:25.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.844+0000 7f28ca800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559ac0736bc0 0x559ac0735910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.844+0000 7f28ca800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559ac0736bc0 0x559ac0735910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:56064/0 (socket says 172.21.15.43:56064) 2024-03-28T22:44:25.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.844+0000 7f28ca800640 1 -- 172.21.15.43:0/266943716 learned_addr learned my addr 172.21.15.43:0/266943716 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:25.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.844+0000 7f28c9e00640 1 --2- 172.21.15.43:0/266943716 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ac0735ee0 0x559ac0736360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.844+0000 7f28c9400640 1 --2- 172.21.15.43:0/266943716 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559ac073f250 0x559ac0741640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.844+0000 7f28ca800640 1 -- 172.21.15.43:0/266943716 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559ac073f250 msgr2=0x559ac0741640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:25.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.844+0000 7f28ca800640 1 --2- 172.21.15.43:0/266943716 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559ac073f250 0x559ac0741640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.845+0000 7f28ca800640 1 -- 172.21.15.43:0/266943716 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ac0735ee0 msgr2=0x559ac0736360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:25.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.845+0000 7f28ca800640 1 --2- 172.21.15.43:0/266943716 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ac0735ee0 0x559ac0736360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.845+0000 7f28ca800640 1 -- 172.21.15.43:0/266943716 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ac06731a0 con 0x559ac0736bc0 2024-03-28T22:44:25.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.845+0000 7f28c9400640 1 --2- 172.21.15.43:0/266943716 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559ac073f250 0x559ac0741640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:44:25.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.845+0000 7f28c2a00640 1 --2- 172.21.15.43:0/2914452634 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f28b00a1ef0 0x7f28b00a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.845+0000 7f28ca800640 1 --2- 172.21.15.43:0/266943716 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559ac0736bc0 0x559ac0735910 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f28c400b6b0 tx=0x7f28c4004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=174df4fc20e61e66 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:25.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.845+0000 7f28c2a00640 1 -- 172.21.15.43:0/2914452634 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f28bc0cecd0 con 0x7f28cc107f60 2024-03-28T22:44:25.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.845+0000 7f28c9e00640 1 --2- 172.21.15.43:0/2914452634 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f28b00a1ef0 0x7f28b00a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.845+0000 7f28bb000640 1 -- 172.21.15.43:0/266943716 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c4003b20 con 0x559ac0736bc0 2024-03-28T22:44:25.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.845+0000 7f28bb000640 1 -- 172.21.15.43:0/266943716 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28c4015070 con 0x559ac0736bc0 2024-03-28T22:44:25.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.845+0000 7f28bb000640 1 -- 172.21.15.43:0/266943716 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c40116a0 con 0x559ac0736bc0 2024-03-28T22:44:25.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.846+0000 7f28c9e00640 1 --2- 172.21.15.43:0/2914452634 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f28b00a1ef0 0x7f28b00a43b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x559ac0737110 tx=0x7f28bc005960 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:25.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.846+0000 7f28d3a36740 1 -- 172.21.15.43:0/266943716 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559ac0736bc0 msgr2=0x559ac0735910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:25.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.846+0000 7f28d3a36740 1 --2- 172.21.15.43:0/266943716 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559ac0736bc0 0x559ac0735910 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f28c400b6b0 tx=0x7f28c4004610 comp rx=0 tx=0).stop 2024-03-28T22:44:25.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.846+0000 7f28d3a36740 1 -- 172.21.15.43:0/266943716 shutdown_connections 2024-03-28T22:44:25.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.846+0000 7f28d3a36740 1 --2- 172.21.15.43:0/266943716 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559ac073f250 0x559ac0741640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.846+0000 7f28d3a36740 1 --2- 172.21.15.43:0/266943716 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ac0735ee0 0x559ac0736360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.846+0000 7f28d3a36740 1 --2- 172.21.15.43:0/266943716 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559ac0736bc0 0x559ac0735910 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.846+0000 7f28d3a36740 1 -- 172.21.15.43:0/266943716 >> 172.21.15.43:0/266943716 conn(0x559ac06fca10 msgr2=0x559ac0730930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:25.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.846+0000 7f28d3a36740 1 -- 172.21.15.43:0/266943716 shutdown_connections 2024-03-28T22:44:25.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.846+0000 7f28d3a36740 1 -- 172.21.15.43:0/266943716 wait complete. 2024-03-28T22:44:25.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.847+0000 7f28d3a36740 1 -- start start 2024-03-28T22:44:25.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.847+0000 7f28d3a36740 1 Processor -- start 2024-03-28T22:44:25.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.847+0000 7f28d3a36740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559ac0735ee0 0x559ac073c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.847+0000 7f28d3a36740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559ac0736bc0 0x559ac073c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.847+0000 7f28d3a36740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ac073f250 0x559ac073ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.847+0000 7f28d3a36740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x559ac07420c0 con 0x559ac073f250 2024-03-28T22:44:25.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.847+0000 7f28d3a36740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x559ac0638250 con 0x559ac0736bc0 2024-03-28T22:44:25.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.847+0000 7f28d3a36740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x559ac0742240 con 0x559ac0735ee0 2024-03-28T22:44:25.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.847+0000 7f28d3a36740 10 client.0 _ll_register_callbacks cb 0x559ac03adbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:44:25.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.848+0000 7f28c9400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559ac0736bc0 0x559ac073c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.848+0000 7f28c9400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559ac0736bc0 0x559ac073c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:56072/0 (socket says 172.21.15.43:56072) 2024-03-28T22:44:25.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.848+0000 7f28c9400640 1 -- 172.21.15.43:0/4204010497 learned_addr learned my addr 172.21.15.43:0/4204010497 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:25.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.848+0000 7f28ca800640 1 --2- 172.21.15.43:0/4204010497 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559ac0735ee0 0x559ac073c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.848+0000 7f28ca800640 1 --2- 172.21.15.43:0/4204010497 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ac073f250 0x559ac073ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.848+0000 7f28c9400640 1 -- 172.21.15.43:0/4204010497 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559ac0735ee0 msgr2=0x559ac073c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:25.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.848+0000 7f28c9400640 1 --2- 172.21.15.43:0/4204010497 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559ac0735ee0 0x559ac073c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.848+0000 7f28c9400640 1 -- 172.21.15.43:0/4204010497 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ac073f250 msgr2=0x559ac073ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:25.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.848+0000 7f28c9400640 1 --2- 172.21.15.43:0/4204010497 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ac073f250 0x559ac073ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.848+0000 7f28c9400640 1 -- 172.21.15.43:0/4204010497 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ac0654960 con 0x559ac0736bc0 2024-03-28T22:44:25.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.848+0000 7f28ca800640 1 --2- 172.21.15.43:0/4204010497 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ac073f250 0x559ac073ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:44:25.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.848+0000 7f28c9400640 1 --2- 172.21.15.43:0/4204010497 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559ac0736bc0 0x559ac073c790 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f28b4006010 tx=0x7f28b4005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69ac6654c4348690 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:25.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.848+0000 7f28af400640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:44:25.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.849+0000 7f28af400640 1 -- 172.21.15.43:0/4204010497 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b4042070 con 0x559ac0736bc0 2024-03-28T22:44:25.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.849+0000 7f28af400640 1 -- 172.21.15.43:0/4204010497 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f28b4005030 con 0x559ac0736bc0 2024-03-28T22:44:25.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.849+0000 7f28d3a36740 1 -- 172.21.15.43:0/4204010497 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28c4015070 con 0x559ac0736bc0 2024-03-28T22:44:25.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.849+0000 7f28af400640 1 -- 172.21.15.43:0/4204010497 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f28b402cd20 con 0x559ac0736bc0 2024-03-28T22:44:25.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.849+0000 7f28d3a36740 10 client.28322 fetch_fsmap learned FSMap version 32 2024-03-28T22:44:25.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.849+0000 7f28d3a36740 1 -- 172.21.15.43:0/4204010497 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559ac04787d0 con 0x559ac0736bc0 2024-03-28T22:44:25.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.849+0000 7f28af400640 1 -- 172.21.15.43:0/4204010497 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28c4015070 con 0x559ac0736bc0 2024-03-28T22:44:25.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.849+0000 7f28af400640 1 -- 172.21.15.43:0/4204010497 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f28b40035f0 con 0x559ac0736bc0 2024-03-28T22:44:25.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.850+0000 7f28d3a36740 10 client.28322 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:44:25.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.850+0000 7f28d3a36740 10 client.28322 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:44:25.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.850+0000 7f28d3a36740 20 client.28322 populate_metadata read hostname 'smithi043' 2024-03-28T22:44:25.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.850+0000 7f28d3a36740 1 --2- 172.21.15.43:0/4204010497 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559ac074f770 0x559ac0751b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.850+0000 7f28d3a36740 4 client.28322 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:44:25.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.850+0000 7f28d3a36740 1 -- 172.21.15.43:0/4204010497 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ac06a7640 con 0x559ac074f770 2024-03-28T22:44:25.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.850+0000 7f28ca800640 1 --2- 172.21.15.43:0/4204010497 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559ac074f770 0x559ac0751b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.850+0000 7f28ca800640 1 --2- 172.21.15.43:0/4204010497 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559ac074f770 0x559ac0751b60 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=788bb7545734ffd2 server_cookie=f32cbc9e2f772b3e in_seq=0 out_seq=0 2024-03-28T22:44:25.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.850+0000 7f28af400640 10 client.28322 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:44:25.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.851+0000 7f28af400640 1 -- 172.21.15.43:0/4204010497 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559ac06a7640 con 0x559ac074f770 2024-03-28T22:44:25.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.851+0000 7f28af400640 10 client.28322 handle_command_reply: tid=0 2024-03-28T22:44:25.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.852+0000 7f28d3a36740 1 client.28322 shutdown 2024-03-28T22:44:25.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.852+0000 7f28d3a36740 1 -- 172.21.15.43:0/4204010497 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559ac0736bc0 msgr2=0x559ac073c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:25.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.852+0000 7f28d3a36740 1 --2- 172.21.15.43:0/4204010497 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559ac0736bc0 0x559ac073c790 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f28b4006010 tx=0x7f28b4005d50 comp rx=0 tx=0).stop 2024-03-28T22:44:25.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.852+0000 7f28d3a36740 1 -- 172.21.15.43:0/4204010497 shutdown_connections 2024-03-28T22:44:25.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.852+0000 7f28d3a36740 1 --2- 172.21.15.43:0/4204010497 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559ac074f770 0x559ac0751b60 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.852+0000 7f28d3a36740 1 --2- 172.21.15.43:0/4204010497 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559ac0736bc0 0x559ac073c790 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.852+0000 7f28d3a36740 1 --2- 172.21.15.43:0/4204010497 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ac073f250 0x559ac073ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.852+0000 7f28d3a36740 1 --2- 172.21.15.43:0/4204010497 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559ac0735ee0 0x559ac073c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.852+0000 7f28d3a36740 1 -- 172.21.15.43:0/4204010497 >> 172.21.15.43:0/4204010497 conn(0x559ac06fca10 msgr2=0x559ac0732680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:25.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.852+0000 7f28af400640 0 client.28322 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:44:25.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.852+0000 7f28d3a36740 1 -- 172.21.15.43:0/4204010497 shutdown_connections 2024-03-28T22:44:25.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.852+0000 7f28d3a36740 1 -- 172.21.15.43:0/4204010497 wait complete. 2024-03-28T22:44:25.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.852+0000 7f28d3a36740 20 client.28322 trim_cache size 0 max 16384 2024-03-28T22:44:25.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.871+0000 7f28d3a36740 1 Processor -- start 2024-03-28T22:44:25.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.872+0000 7f28d3a36740 1 -- start start 2024-03-28T22:44:25.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.872+0000 7f28d3a36740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559ac0735ee0 0x559ac073b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.872+0000 7f28d3a36740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559ac0736bc0 0x559ac074aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.872+0000 7f28d3a36740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ac073f250 0x559ac073caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.872+0000 7f28d3a36740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f28cc1b5360 con 0x559ac073f250 2024-03-28T22:44:25.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.872+0000 7f28d3a36740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f28cc1b6ea0 con 0x559ac0736bc0 2024-03-28T22:44:25.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.872+0000 7f28d3a36740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f28cc1b5a30 con 0x559ac0735ee0 2024-03-28T22:44:25.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.872+0000 7f28c9400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559ac0736bc0 0x559ac074aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.872+0000 7f28ca800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559ac0735ee0 0x559ac073b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.872+0000 7f28c9400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559ac0736bc0 0x559ac074aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:56078/0 (socket says 172.21.15.43:56078) 2024-03-28T22:44:25.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.872+0000 7f28c9400640 1 -- 172.21.15.43:0/3833963170 learned_addr learned my addr 172.21.15.43:0/3833963170 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:25.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.872+0000 7f28ca800640 1 --2- 172.21.15.43:0/3833963170 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ac073f250 0x559ac073caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.872+0000 7f28c9400640 1 -- 172.21.15.43:0/3833963170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559ac0735ee0 msgr2=0x559ac073b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:25.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.873+0000 7f28c9400640 1 --2- 172.21.15.43:0/3833963170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559ac0735ee0 0x559ac073b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.873+0000 7f28c9400640 1 -- 172.21.15.43:0/3833963170 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ac073f250 msgr2=0x559ac073caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:25.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.873+0000 7f28c9400640 1 --2- 172.21.15.43:0/3833963170 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ac073f250 0x559ac073caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.873+0000 7f28c9400640 1 -- 172.21.15.43:0/3833963170 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ac03ae3f0 con 0x559ac0736bc0 2024-03-28T22:44:25.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.873+0000 7f28ca800640 1 --2- 172.21.15.43:0/3833963170 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ac073f250 0x559ac073caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:44:25.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.873+0000 7f28c9400640 1 --2- 172.21.15.43:0/3833963170 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559ac0736bc0 0x559ac074aed0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f28b402c180 tx=0x7f28b4005b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f03690bb5bee987 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:25.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.873+0000 7f28ba600640 1 -- 172.21.15.43:0/3833963170 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b4042040 con 0x559ac0736bc0 2024-03-28T22:44:25.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.873+0000 7f28ba600640 1 -- 172.21.15.43:0/3833963170 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28b4009780 con 0x559ac0736bc0 2024-03-28T22:44:25.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.873+0000 7f28ba600640 1 -- 172.21.15.43:0/3833963170 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b40046c0 con 0x559ac0736bc0 2024-03-28T22:44:25.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.874+0000 7f28d3a36740 1 -- 172.21.15.43:0/3833963170 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559ac0736bc0 msgr2=0x559ac074aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:25.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.874+0000 7f28d3a36740 1 --2- 172.21.15.43:0/3833963170 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559ac0736bc0 0x559ac074aed0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f28b402c180 tx=0x7f28b4005b50 comp rx=0 tx=0).stop 2024-03-28T22:44:25.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.874+0000 7f28d3a36740 1 -- 172.21.15.43:0/3833963170 shutdown_connections 2024-03-28T22:44:25.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.874+0000 7f28d3a36740 1 --2- 172.21.15.43:0/3833963170 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559ac0736bc0 0x559ac074aed0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.874+0000 7f28d3a36740 1 --2- 172.21.15.43:0/3833963170 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ac073f250 0x559ac073caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.874+0000 7f28d3a36740 1 --2- 172.21.15.43:0/3833963170 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559ac0735ee0 0x559ac073b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.874+0000 7f28d3a36740 1 -- 172.21.15.43:0/3833963170 >> 172.21.15.43:0/3833963170 conn(0x559ac06fca10 msgr2=0x559ac0732610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:25.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.874+0000 7f28d3a36740 1 -- 172.21.15.43:0/3833963170 shutdown_connections 2024-03-28T22:44:25.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.874+0000 7f28d3a36740 1 -- 172.21.15.43:0/3833963170 wait complete. 2024-03-28T22:44:25.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.874+0000 7f28d3a36740 1 -- start start 2024-03-28T22:44:25.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.874+0000 7f28d3a36740 1 Processor -- start 2024-03-28T22:44:25.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.875+0000 7f28d3a36740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ac0735ee0 0x559ac0747900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.875+0000 7f28d3a36740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559ac0736bc0 0x559ac0739580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.875+0000 7f28d3a36740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559ac073f250 0x559ac0739cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.875+0000 7f28d3a36740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f28cc1b67d0 con 0x559ac0735ee0 2024-03-28T22:44:25.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.875+0000 7f28d3a36740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f28cc1b6100 con 0x559ac0736bc0 2024-03-28T22:44:25.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.875+0000 7f28d3a36740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f28cc1b3360 con 0x559ac073f250 2024-03-28T22:44:25.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.875+0000 7f28d3a36740 10 client.0 _ll_register_callbacks cb 0x7f28bc0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:44:25.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.875+0000 7f28c9400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559ac0736bc0 0x559ac0739580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.875+0000 7f28ca800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ac0735ee0 0x559ac0747900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.875+0000 7f28c9400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559ac0736bc0 0x559ac0739580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:56088/0 (socket says 172.21.15.43:56088) 2024-03-28T22:44:25.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.875+0000 7f28c9400640 1 -- 172.21.15.43:0/2854692238 learned_addr learned my addr 172.21.15.43:0/2854692238 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:25.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.875+0000 7f28ca800640 1 --2- 172.21.15.43:0/2854692238 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559ac073f250 0x559ac0739cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.875+0000 7f28c9400640 1 -- 172.21.15.43:0/2854692238 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559ac073f250 msgr2=0x559ac0739cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:25.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.875+0000 7f28c9400640 1 --2- 172.21.15.43:0/2854692238 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559ac073f250 0x559ac0739cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.875+0000 7f28c9400640 1 -- 172.21.15.43:0/2854692238 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ac0735ee0 msgr2=0x559ac0747900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:25.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.875+0000 7f28c9400640 1 --2- 172.21.15.43:0/2854692238 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ac0735ee0 0x559ac0747900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.875+0000 7f28c9400640 1 -- 172.21.15.43:0/2854692238 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ac036e0f0 con 0x559ac0736bc0 2024-03-28T22:44:25.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.875+0000 7f28ca800640 1 --2- 172.21.15.43:0/2854692238 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ac0735ee0 0x559ac0747900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:44:25.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.876+0000 7f28c9400640 1 --2- 172.21.15.43:0/2854692238 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559ac0736bc0 0x559ac0739580 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f28b4003ae0 tx=0x7f28b4003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c971bc6aee4e970d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:25.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.876+0000 7f28af400640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:44:25.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.876+0000 7f28af400640 1 -- 172.21.15.43:0/2854692238 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b4042070 con 0x559ac0736bc0 2024-03-28T22:44:25.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.876+0000 7f28d3a36740 1 -- 172.21.15.43:0/2854692238 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28b4009780 con 0x559ac0736bc0 2024-03-28T22:44:25.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.876+0000 7f28af400640 1 -- 172.21.15.43:0/2854692238 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f28b4005440 con 0x559ac0736bc0 2024-03-28T22:44:25.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.876+0000 7f28af400640 1 -- 172.21.15.43:0/2854692238 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f28b4064050 con 0x559ac0736bc0 2024-03-28T22:44:25.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.876+0000 7f28d3a36740 10 client.28328 fetch_fsmap learned FSMap version 32 2024-03-28T22:44:25.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.876+0000 7f28d3a36740 1 -- 172.21.15.43:0/2854692238 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559ac07b5b40 con 0x559ac0736bc0 2024-03-28T22:44:25.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.876+0000 7f28af400640 1 -- 172.21.15.43:0/2854692238 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28b4009780 con 0x559ac0736bc0 2024-03-28T22:44:25.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.877+0000 7f28af400640 1 -- 172.21.15.43:0/2854692238 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f28b40345d0 con 0x559ac0736bc0 2024-03-28T22:44:25.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.877+0000 7f28d3a36740 10 client.28328 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:44:25.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.877+0000 7f28d3a36740 10 client.28328 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:44:25.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.877+0000 7f28d3a36740 20 client.28328 populate_metadata read hostname 'smithi043' 2024-03-28T22:44:25.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.877+0000 7f28d3a36740 1 --2- 172.21.15.43:0/2854692238 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559ac07b61a0 0x559ac07b8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:25.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.877+0000 7f28d3a36740 4 client.28328 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:44:25.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.877+0000 7f28d3a36740 1 -- 172.21.15.43:0/2854692238 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559ac06a1990 con 0x559ac07b61a0 2024-03-28T22:44:25.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.877+0000 7f28ca800640 1 --2- 172.21.15.43:0/2854692238 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559ac07b61a0 0x559ac07b8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:25.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.877+0000 7f28ca800640 1 --2- 172.21.15.43:0/2854692238 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559ac07b61a0 0x559ac07b8590 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11b656a96d0e1d51 server_cookie=5048c066ef7dabb9 in_seq=0 out_seq=0 2024-03-28T22:44:25.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.877+0000 7f28af400640 10 client.28328 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:44:25.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.878+0000 7f28af400640 1 -- 172.21.15.43:0/2854692238 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x559ac06a1990 con 0x559ac07b61a0 2024-03-28T22:44:25.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.878+0000 7f28af400640 10 client.28328 handle_command_reply: tid=0 2024-03-28T22:44:25.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.878+0000 7f28d3a36740 1 client.28328 shutdown 2024-03-28T22:44:25.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.878+0000 7f28d3a36740 1 -- 172.21.15.43:0/2854692238 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559ac0736bc0 msgr2=0x559ac0739580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:25.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.878+0000 7f28d3a36740 1 --2- 172.21.15.43:0/2854692238 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559ac0736bc0 0x559ac0739580 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f28b4003ae0 tx=0x7f28b4003820 comp rx=0 tx=0).stop 2024-03-28T22:44:25.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.878+0000 7f28d3a36740 1 -- 172.21.15.43:0/2854692238 shutdown_connections 2024-03-28T22:44:25.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.878+0000 7f28d3a36740 1 --2- 172.21.15.43:0/2854692238 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559ac07b61a0 0x559ac07b8590 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.878+0000 7f28d3a36740 1 --2- 172.21.15.43:0/2854692238 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559ac0736bc0 0x559ac0739580 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.878+0000 7f28d3a36740 1 --2- 172.21.15.43:0/2854692238 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559ac073f250 0x559ac0739cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.878+0000 7f28d3a36740 1 --2- 172.21.15.43:0/2854692238 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559ac0735ee0 0x559ac0747900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.878+0000 7f28af400640 0 client.28328 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:44:25.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.878+0000 7f28d3a36740 1 -- 172.21.15.43:0/2854692238 >> 172.21.15.43:0/2854692238 conn(0x559ac06fca10 msgr2=0x559ac0731b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:25.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.878+0000 7f28d3a36740 1 -- 172.21.15.43:0/2854692238 shutdown_connections 2024-03-28T22:44:25.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.878+0000 7f28d3a36740 1 -- 172.21.15.43:0/2854692238 wait complete. 2024-03-28T22:44:25.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.878+0000 7f28d3a36740 20 client.28328 trim_cache size 0 max 16384 2024-03-28T22:44:25.888 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:44:25.888 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:44:25.888 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-03-28T22:44:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: "59e64d15-9226-41b3-83d0-096cdc02381a": { 2024-03-28T22:44:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-03-28T22:44:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "59e64d15-9226-41b3-83d0-096cdc02381a", 2024-03-28T22:44:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-03-28T22:44:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:44:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:44:25.889 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:44:25.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.879+0000 7f28d0c00640 1 -- 172.21.15.43:0/2914452634 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f28b00a1ef0 msgr2=0x7f28b00a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:25.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.879+0000 7f28d0c00640 1 --2- 172.21.15.43:0/2914452634 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f28b00a1ef0 0x7f28b00a43b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x559ac0737110 tx=0x7f28bc005960 comp rx=0 tx=0).stop 2024-03-28T22:44:25.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.879+0000 7f28d0c00640 1 -- 172.21.15.43:0/2914452634 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f28cc107f60 msgr2=0x7f28cc19e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:25.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.879+0000 7f28d0c00640 1 --2- 172.21.15.43:0/2914452634 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f28cc107f60 0x7f28cc19e670 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f28bc00eb90 tx=0x7f28bc00c6a0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.879+0000 7f28d0c00640 1 -- 172.21.15.43:0/2914452634 shutdown_connections 2024-03-28T22:44:25.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.879+0000 7f28d0c00640 1 --2- 172.21.15.43:0/2914452634 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f28b00a1ef0 0x7f28b00a43b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.879+0000 7f28d0c00640 1 --2- 172.21.15.43:0/2914452634 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f28cc1a2f30 0x7f28cc1a33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.879+0000 7f28d0c00640 1 --2- 172.21.15.43:0/2914452634 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f28cc10d030 0x7f28cc19ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.879+0000 7f28d0c00640 1 --2- 172.21.15.43:0/2914452634 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f28cc107f60 0x7f28cc19e670 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:25.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.879+0000 7f28d0c00640 1 -- 172.21.15.43:0/2914452634 >> 172.21.15.43:0/2914452634 conn(0x7f28cc075c90 msgr2=0x7f28cc077fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:25.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.879+0000 7f28d0c00640 1 -- 172.21.15.43:0/2914452634 shutdown_connections 2024-03-28T22:44:25.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:25.879+0000 7f28d0c00640 1 -- 172.21.15.43:0/2914452634 wait complete. 2024-03-28T22:44:25.894 INFO:tasks.cephfs.filesystem:scrub status for tag:59e64d15-9226-41b3-83d0-096cdc02381a - {'path': '/', 'tag': '59e64d15-9226-41b3-83d0-096cdc02381a', 'options': 'recursive,force'} 2024-03-28T22:44:27.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:26 smithi172 ceph-mon[36566]: scrub queued for path: / 2024-03-28T22:44:27.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:26 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:44:27.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:26 smithi172 ceph-mon[36566]: scrub summary: active paths [/] 2024-03-28T22:44:27.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:26 smithi172 ceph-mon[36566]: pgmap v225: 129 pgs: 129 active+clean; 48 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 750 KiB/s wr, 62 op/s 2024-03-28T22:44:27.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:26 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:44:27.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:26 smithi082 ceph-mon[36471]: scrub queued for path: / 2024-03-28T22:44:27.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:26 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:44:27.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:26 smithi082 ceph-mon[36471]: scrub summary: active paths [/] 2024-03-28T22:44:27.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:26 smithi082 ceph-mon[36471]: pgmap v225: 129 pgs: 129 active+clean; 48 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 750 KiB/s wr, 62 op/s 2024-03-28T22:44:27.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:26 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:44:27.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:26 smithi043 ceph-mon[31823]: scrub queued for path: / 2024-03-28T22:44:27.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:26 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:44:27.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:26 smithi043 ceph-mon[31823]: scrub summary: active paths [/] 2024-03-28T22:44:27.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:26 smithi043 ceph-mon[31823]: pgmap v225: 129 pgs: 129 active+clean; 48 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 750 KiB/s wr, 62 op/s 2024-03-28T22:44:27.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:26 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:44:29.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:28 smithi172 ceph-mon[36566]: pgmap v226: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 2.9 MiB/s wr, 74 op/s 2024-03-28T22:44:29.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:28 smithi172 ceph-mon[36566]: scrub summary: idle 2024-03-28T22:44:29.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:28 smithi082 ceph-mon[36471]: pgmap v226: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 2.9 MiB/s wr, 74 op/s 2024-03-28T22:44:29.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:28 smithi082 ceph-mon[36471]: scrub summary: idle 2024-03-28T22:44:29.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:28 smithi043 ceph-mon[31823]: pgmap v226: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 2.9 MiB/s wr, 74 op/s 2024-03-28T22:44:29.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:28 smithi043 ceph-mon[31823]: scrub summary: idle 2024-03-28T22:44:31.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:30 smithi172 ceph-mon[36566]: pgmap v227: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 2.5 MiB/s wr, 63 op/s 2024-03-28T22:44:31.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:30 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:44:31.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:30 smithi082 ceph-mon[36471]: pgmap v227: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 2.5 MiB/s wr, 63 op/s 2024-03-28T22:44:31.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:30 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:44:31.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:30 smithi043 ceph-mon[31823]: pgmap v227: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 2.5 MiB/s wr, 63 op/s 2024-03-28T22:44:31.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:30 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:44:32.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:31 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:44:32.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:31 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:44:32.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:31 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:44:32.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:31 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:44:32.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:31 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:44:32.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:31 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:44:32.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:31 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:44:32.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:31 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:44:32.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:31 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:44:33.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:32 smithi172 ceph-mon[36566]: pgmap v228: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 2.4 MiB/s wr, 62 op/s 2024-03-28T22:44:33.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:32 smithi082 ceph-mon[36471]: pgmap v228: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 2.4 MiB/s wr, 62 op/s 2024-03-28T22:44:33.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:32 smithi043 ceph-mon[31823]: pgmap v228: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 2.4 MiB/s wr, 62 op/s 2024-03-28T22:44:35.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:34 smithi172 ceph-mon[36566]: pgmap v229: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 2.9 MiB/s wr, 87 op/s 2024-03-28T22:44:35.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:34 smithi082 ceph-mon[36471]: pgmap v229: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 2.9 MiB/s wr, 87 op/s 2024-03-28T22:44:35.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:34 smithi043 ceph-mon[31823]: pgmap v229: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 2.9 MiB/s wr, 87 op/s 2024-03-28T22:44:37.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:36 smithi172 ceph-mon[36566]: pgmap v230: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 2.4 MiB/s wr, 58 op/s 2024-03-28T22:44:37.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:36 smithi082 ceph-mon[36471]: pgmap v230: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 2.4 MiB/s wr, 58 op/s 2024-03-28T22:44:37.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:36 smithi043 ceph-mon[31823]: pgmap v230: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 2.4 MiB/s wr, 58 op/s 2024-03-28T22:44:39.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:38 smithi172 ceph-mon[36566]: pgmap v231: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 2.6 MiB/s wr, 75 op/s 2024-03-28T22:44:39.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:38 smithi082 ceph-mon[36471]: pgmap v231: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 2.6 MiB/s wr, 75 op/s 2024-03-28T22:44:39.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:38 smithi043 ceph-mon[31823]: pgmap v231: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 2.6 MiB/s wr, 75 op/s 2024-03-28T22:44:41.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:40 smithi172 ceph-mon[36566]: pgmap v232: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 755 KiB/s wr, 41 op/s 2024-03-28T22:44:41.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:40 smithi082 ceph-mon[36471]: pgmap v232: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 755 KiB/s wr, 41 op/s 2024-03-28T22:44:41.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:40 smithi043 ceph-mon[31823]: pgmap v232: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 755 KiB/s wr, 41 op/s 2024-03-28T22:44:43.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:42 smithi172 ceph-mon[36566]: pgmap v233: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 755 KiB/s wr, 41 op/s 2024-03-28T22:44:43.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:42 smithi082 ceph-mon[36471]: pgmap v233: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 755 KiB/s wr, 41 op/s 2024-03-28T22:44:43.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:42 smithi043 ceph-mon[31823]: pgmap v233: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 755 KiB/s wr, 41 op/s 2024-03-28T22:44:45.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:44 smithi172 ceph-mon[36566]: pgmap v234: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 1.0 MiB/s wr, 52 op/s 2024-03-28T22:44:45.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:44 smithi082 ceph-mon[36471]: pgmap v234: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 1.0 MiB/s wr, 52 op/s 2024-03-28T22:44:45.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:44 smithi043 ceph-mon[31823]: pgmap v234: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 1.0 MiB/s wr, 52 op/s 2024-03-28T22:44:47.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:46 smithi172 ceph-mon[36566]: pgmap v235: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 578 KiB/s wr, 27 op/s 2024-03-28T22:44:47.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:46 smithi082 ceph-mon[36471]: pgmap v235: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 578 KiB/s wr, 27 op/s 2024-03-28T22:44:47.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:46 smithi043 ceph-mon[31823]: pgmap v235: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 578 KiB/s wr, 27 op/s 2024-03-28T22:44:49.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:48 smithi172 ceph-mon[36566]: pgmap v236: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 592 KiB/s wr, 32 op/s 2024-03-28T22:44:49.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:48 smithi082 ceph-mon[36471]: pgmap v236: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 592 KiB/s wr, 32 op/s 2024-03-28T22:44:49.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:48 smithi043 ceph-mon[31823]: pgmap v236: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 592 KiB/s wr, 32 op/s 2024-03-28T22:44:51.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:50 smithi172 ceph-mon[36566]: pgmap v237: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 327 KiB/s wr, 15 op/s 2024-03-28T22:44:51.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:50 smithi082 ceph-mon[36471]: pgmap v237: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 327 KiB/s wr, 15 op/s 2024-03-28T22:44:51.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:50 smithi043 ceph-mon[31823]: pgmap v237: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 327 KiB/s wr, 15 op/s 2024-03-28T22:44:53.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:52 smithi172 ceph-mon[36566]: pgmap v238: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 327 KiB/s wr, 15 op/s 2024-03-28T22:44:53.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:52 smithi082 ceph-mon[36471]: pgmap v238: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 327 KiB/s wr, 15 op/s 2024-03-28T22:44:53.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:52 smithi043 ceph-mon[31823]: pgmap v238: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 327 KiB/s wr, 15 op/s 2024-03-28T22:44:55.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:54 smithi172 ceph-mon[36566]: pgmap v239: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 339 KiB/s wr, 20 op/s 2024-03-28T22:44:55.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:54 smithi082 ceph-mon[36471]: pgmap v239: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 339 KiB/s wr, 20 op/s 2024-03-28T22:44:55.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:54 smithi043 ceph-mon[31823]: pgmap v239: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 339 KiB/s wr, 20 op/s 2024-03-28T22:44:55.895 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:44:55.895 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:44:56.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.136+0000 7f27a2600640 1 -- 172.21.15.43:0/2485170226 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f279c1088b0 msgr2=0x7f279c10c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:56.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.136+0000 7f27a2600640 1 --2- 172.21.15.43:0/2485170226 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f279c1088b0 0x7f279c10c870 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f2784009900 tx=0x7f278402bc40 comp rx=0 tx=0).stop 2024-03-28T22:44:56.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.136+0000 7f27a2600640 1 -- 172.21.15.43:0/2485170226 shutdown_connections 2024-03-28T22:44:56.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.136+0000 7f27a2600640 1 --2- 172.21.15.43:0/2485170226 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f279c10cf50 0x7f279c114b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.136+0000 7f27a2600640 1 --2- 172.21.15.43:0/2485170226 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f279c1088b0 0x7f279c10c870 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.136+0000 7f27a2600640 1 --2- 172.21.15.43:0/2485170226 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f279c107f00 0x7f279c1082e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.136+0000 7f27a2600640 1 -- 172.21.15.43:0/2485170226 >> 172.21.15.43:0/2485170226 conn(0x7f279c077570 msgr2=0x7f279c106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:56.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.136+0000 7f27a2600640 1 -- 172.21.15.43:0/2485170226 shutdown_connections 2024-03-28T22:44:56.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.136+0000 7f27a2600640 1 -- 172.21.15.43:0/2485170226 wait complete. 2024-03-28T22:44:56.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.137+0000 7f27a2600640 1 Processor -- start 2024-03-28T22:44:56.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.137+0000 7f27a2600640 1 -- start start 2024-03-28T22:44:56.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.138+0000 7f27a2600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f279c107f00 0x7f279c19e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.138+0000 7f27a2600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f279c10cf50 0x7f279c19ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.138+0000 7f27a2600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f279c1a2db0 0x7f279c1a3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.138+0000 7f27a2600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f279c118010 con 0x7f279c1a2db0 2024-03-28T22:44:56.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.138+0000 7f27a2600640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f279c117e90 con 0x7f279c107f00 2024-03-28T22:44:56.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.138+0000 7f27a2600640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f279c118190 con 0x7f279c10cf50 2024-03-28T22:44:56.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.138+0000 7f27a1200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f279c107f00 0x7f279c19e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.138+0000 7f2793e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f279c10cf50 0x7f279c19ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.138+0000 7f27a1c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f279c1a2db0 0x7f279c1a3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.138+0000 7f27a1200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f279c107f00 0x7f279c19e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47072/0 (socket says 172.21.15.43:47072) 2024-03-28T22:44:56.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.138+0000 7f27a1200640 1 -- 172.21.15.43:0/31299574 learned_addr learned my addr 172.21.15.43:0/31299574 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:56.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.138+0000 7f2793e00640 1 -- 172.21.15.43:0/31299574 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f279c107f00 msgr2=0x7f279c19e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:56.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.138+0000 7f2793e00640 1 --2- 172.21.15.43:0/31299574 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f279c107f00 0x7f279c19e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.138+0000 7f2793e00640 1 -- 172.21.15.43:0/31299574 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f279c1a2db0 msgr2=0x7f279c1a3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:56.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.138+0000 7f2793e00640 1 --2- 172.21.15.43:0/31299574 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f279c1a2db0 0x7f279c1a3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.138+0000 7f2793e00640 1 -- 172.21.15.43:0/31299574 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f279c1a38a0 con 0x7f279c10cf50 2024-03-28T22:44:56.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.139+0000 7f2793e00640 1 --2- 172.21.15.43:0/31299574 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f279c10cf50 0x7f279c19ea30 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f2784009230 tx=0x7f27840042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:56.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.139+0000 7f279a000640 1 -- 172.21.15.43:0/31299574 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2784041070 con 0x7f279c10cf50 2024-03-28T22:44:56.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.139+0000 7f279a000640 1 -- 172.21.15.43:0/31299574 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f278402cca0 con 0x7f279c10cf50 2024-03-28T22:44:56.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.139+0000 7f27a2600640 1 -- 172.21.15.43:0/31299574 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f279c1a3b30 con 0x7f279c10cf50 2024-03-28T22:44:56.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.139+0000 7f279a000640 1 -- 172.21.15.43:0/31299574 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2784003df0 con 0x7f279c10cf50 2024-03-28T22:44:56.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.139+0000 7f27a2600640 1 -- 172.21.15.43:0/31299574 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f279c1ab2f0 con 0x7f279c10cf50 2024-03-28T22:44:56.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.142+0000 7f279a000640 1 -- 172.21.15.43:0/31299574 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f2784044070 con 0x7f279c10cf50 2024-03-28T22:44:56.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.142+0000 7f27a69c8740 1 Processor -- start 2024-03-28T22:44:56.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.143+0000 7f27a69c8740 1 -- start start 2024-03-28T22:44:56.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.143+0000 7f27a69c8740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c137357db0 0x55c137356b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.143+0000 7f27a69c8740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c1373570d0 0x55c137357550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.143+0000 7f27a69c8740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c137360440 0x55c137362830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.143+0000 7f27a69c8740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c1372bb100 con 0x55c137357db0 2024-03-28T22:44:56.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.143+0000 7f27a69c8740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55c137263ba0 con 0x55c137360440 2024-03-28T22:44:56.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.143+0000 7f27a69c8740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55c1372b71a0 con 0x55c1373570d0 2024-03-28T22:44:56.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.143+0000 7f27a1200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c1373570d0 0x55c137357550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.143+0000 7f27a1200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c1373570d0 0x55c137357550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:39904/0 (socket says 172.21.15.43:39904) 2024-03-28T22:44:56.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.143+0000 7f27a1200640 1 -- 172.21.15.43:0/1658912108 learned_addr learned my addr 172.21.15.43:0/1658912108 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:56.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.143+0000 7f2793e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c137360440 0x55c137362830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.143+0000 7f27a1c00640 1 --2- 172.21.15.43:0/1658912108 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c137357db0 0x55c137356b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.144+0000 7f27a1200640 1 -- 172.21.15.43:0/1658912108 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c137360440 msgr2=0x55c137362830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:56.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.144+0000 7f27a1200640 1 --2- 172.21.15.43:0/1658912108 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c137360440 0x55c137362830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.144+0000 7f27a1200640 1 -- 172.21.15.43:0/1658912108 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c137357db0 msgr2=0x55c137356b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:56.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.144+0000 7f27a1200640 1 --2- 172.21.15.43:0/1658912108 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c137357db0 0x55c137356b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.144+0000 7f27a1200640 1 -- 172.21.15.43:0/1658912108 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1372941a0 con 0x55c1373570d0 2024-03-28T22:44:56.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.144+0000 7f27a1200640 1 --2- 172.21.15.43:0/1658912108 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c1373570d0 0x55c137357550 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f278c009e10 tx=0x7f278c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a281b3715876c91 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:56.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.144+0000 7f279a000640 1 --2- 172.21.15.43:0/31299574 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f27800a1f70 0x7f27800a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.144+0000 7f2792000640 1 -- 172.21.15.43:0/1658912108 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f278c01e070 con 0x55c1373570d0 2024-03-28T22:44:56.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.144+0000 7f2792000640 1 -- 172.21.15.43:0/1658912108 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f278c019b10 con 0x55c1373570d0 2024-03-28T22:44:56.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.144+0000 7f2792000640 1 -- 172.21.15.43:0/1658912108 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f278c00fbd0 con 0x55c1373570d0 2024-03-28T22:44:56.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.144+0000 7f279a000640 1 -- 172.21.15.43:0/31299574 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f27840f00f0 con 0x7f279c10cf50 2024-03-28T22:44:56.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.145+0000 7f27a1c00640 1 --2- 172.21.15.43:0/31299574 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f27800a1f70 0x7f27800a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.145+0000 7f27a69c8740 1 -- 172.21.15.43:0/1658912108 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c1373570d0 msgr2=0x55c137357550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:56.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.145+0000 7f27a69c8740 1 --2- 172.21.15.43:0/1658912108 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c1373570d0 0x55c137357550 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f278c009e10 tx=0x7f278c011810 comp rx=0 tx=0).stop 2024-03-28T22:44:56.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.145+0000 7f27a69c8740 1 -- 172.21.15.43:0/1658912108 shutdown_connections 2024-03-28T22:44:56.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.145+0000 7f27a69c8740 1 --2- 172.21.15.43:0/1658912108 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c137360440 0x55c137362830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.145+0000 7f27a69c8740 1 --2- 172.21.15.43:0/1658912108 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c1373570d0 0x55c137357550 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.145+0000 7f27a69c8740 1 --2- 172.21.15.43:0/1658912108 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c137357db0 0x55c137356b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.145+0000 7f27a69c8740 1 -- 172.21.15.43:0/1658912108 >> 172.21.15.43:0/1658912108 conn(0x55c13731da10 msgr2=0x55c137351b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:56.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.145+0000 7f27a69c8740 1 -- 172.21.15.43:0/1658912108 shutdown_connections 2024-03-28T22:44:56.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.145+0000 7f27a1c00640 1 --2- 172.21.15.43:0/31299574 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f27800a1f70 0x7f27800a4430 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2794005fd0 tx=0x7f2794009500 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:56.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.145+0000 7f27a69c8740 1 -- 172.21.15.43:0/1658912108 wait complete. 2024-03-28T22:44:56.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.146+0000 7f27a69c8740 1 -- start start 2024-03-28T22:44:56.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.146+0000 7f27a69c8740 1 Processor -- start 2024-03-28T22:44:56.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.147+0000 7f27a69c8740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c1373570d0 0x55c13735d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.147+0000 7f27a69c8740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c137357db0 0x55c13735d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.147+0000 7f27a69c8740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c137360440 0x55c13735deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.147+0000 7f27a69c8740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c1373632b0 con 0x55c137360440 2024-03-28T22:44:56.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.147+0000 7f27a69c8740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55c137259250 con 0x55c137357db0 2024-03-28T22:44:56.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.147+0000 7f27a69c8740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55c137363430 con 0x55c1373570d0 2024-03-28T22:44:56.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.147+0000 7f27a69c8740 10 client.0 _ll_register_callbacks cb 0x55c136fcebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:44:56.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.147+0000 7f27a1200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c1373570d0 0x55c13735d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.147+0000 7f2793e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c137357db0 0x55c13735d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.147+0000 7f27a1200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c1373570d0 0x55c13735d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:39906/0 (socket says 172.21.15.43:39906) 2024-03-28T22:44:56.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.147+0000 7f27a1200640 1 -- 172.21.15.43:0/1222658016 learned_addr learned my addr 172.21.15.43:0/1222658016 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:56.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.147+0000 7f27a1c00640 1 --2- 172.21.15.43:0/1222658016 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c137360440 0x55c13735deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.147+0000 7f2793e00640 1 -- 172.21.15.43:0/1222658016 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c1373570d0 msgr2=0x55c13735d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:56.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.147+0000 7f2793e00640 1 --2- 172.21.15.43:0/1222658016 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c1373570d0 0x55c13735d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.147+0000 7f2793e00640 1 -- 172.21.15.43:0/1222658016 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c137360440 msgr2=0x55c13735deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:56.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.147+0000 7f2793e00640 1 --2- 172.21.15.43:0/1222658016 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c137360440 0x55c13735deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.147+0000 7f2793e00640 1 -- 172.21.15.43:0/1222658016 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c137275960 con 0x55c137357db0 2024-03-28T22:44:56.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.147+0000 7f27a1200640 1 --2- 172.21.15.43:0/1222658016 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c1373570d0 0x55c13735d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:44:56.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.147+0000 7f2793e00640 1 --2- 172.21.15.43:0/1222658016 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c137357db0 0x55c13735d970 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f27840ed4f0 tx=0x7f27840b8450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8419b2fa5605fd4f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:56.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.147+0000 7f278aa00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:44:56.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.148+0000 7f278aa00640 1 -- 172.21.15.43:0/1222658016 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f278404b030 con 0x55c137357db0 2024-03-28T22:44:56.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.148+0000 7f278aa00640 1 -- 172.21.15.43:0/1222658016 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f27840a70b0 con 0x55c137357db0 2024-03-28T22:44:56.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.148+0000 7f27a69c8740 1 -- 172.21.15.43:0/1222658016 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f278c019b10 con 0x55c137357db0 2024-03-28T22:44:56.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.148+0000 7f278aa00640 1 -- 172.21.15.43:0/1222658016 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f27840bc640 con 0x55c137357db0 2024-03-28T22:44:56.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.148+0000 7f27a69c8740 10 client.28340 fetch_fsmap learned FSMap version 32 2024-03-28T22:44:56.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.148+0000 7f27a69c8740 1 -- 172.21.15.43:0/1222658016 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c1370997d0 con 0x55c137357db0 2024-03-28T22:44:56.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.148+0000 7f278aa00640 1 -- 172.21.15.43:0/1222658016 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f278c019b10 con 0x55c137357db0 2024-03-28T22:44:56.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.148+0000 7f278aa00640 1 -- 172.21.15.43:0/1222658016 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f27840ed690 con 0x55c137357db0 2024-03-28T22:44:56.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.148+0000 7f27a69c8740 10 client.28340 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:44:56.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.148+0000 7f27a69c8740 10 client.28340 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:44:56.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.148+0000 7f27a69c8740 20 client.28340 populate_metadata read hostname 'smithi043' 2024-03-28T22:44:56.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.148+0000 7f27a69c8740 1 --2- 172.21.15.43:0/1222658016 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c137370850 0x55c137372c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.148+0000 7f27a69c8740 4 client.28340 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:44:56.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.148+0000 7f27a69c8740 1 -- 172.21.15.43:0/1222658016 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c1372c8640 con 0x55c137370850 2024-03-28T22:44:56.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.149+0000 7f27a1200640 1 --2- 172.21.15.43:0/1222658016 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c137370850 0x55c137372c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.149+0000 7f27a1200640 1 --2- 172.21.15.43:0/1222658016 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c137370850 0x55c137372c40 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b18dec7573093dc8 server_cookie=16aa69bc415e5ee4 in_seq=0 out_seq=0 2024-03-28T22:44:56.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.149+0000 7f278aa00640 10 client.28340 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:44:56.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.150+0000 7f278aa00640 1 -- 172.21.15.43:0/1222658016 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c1372c8640 con 0x55c137370850 2024-03-28T22:44:56.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.150+0000 7f278aa00640 10 client.28340 handle_command_reply: tid=0 2024-03-28T22:44:56.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.150+0000 7f27a69c8740 1 client.28340 shutdown 2024-03-28T22:44:56.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.150+0000 7f27a69c8740 1 -- 172.21.15.43:0/1222658016 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c137357db0 msgr2=0x55c13735d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:56.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.150+0000 7f27a69c8740 1 --2- 172.21.15.43:0/1222658016 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c137357db0 0x55c13735d970 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f27840ed4f0 tx=0x7f27840b8450 comp rx=0 tx=0).stop 2024-03-28T22:44:56.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.150+0000 7f27a69c8740 1 -- 172.21.15.43:0/1222658016 shutdown_connections 2024-03-28T22:44:56.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.150+0000 7f27a69c8740 1 --2- 172.21.15.43:0/1222658016 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c137370850 0x55c137372c40 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.150+0000 7f27a69c8740 1 --2- 172.21.15.43:0/1222658016 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c137357db0 0x55c13735d970 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.150+0000 7f27a69c8740 1 --2- 172.21.15.43:0/1222658016 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c137360440 0x55c13735deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.150+0000 7f27a69c8740 1 --2- 172.21.15.43:0/1222658016 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c1373570d0 0x55c13735d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.150+0000 7f278aa00640 0 client.28340 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:44:56.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.150+0000 7f27a69c8740 1 -- 172.21.15.43:0/1222658016 >> 172.21.15.43:0/1222658016 conn(0x55c13731da10 msgr2=0x55c137353870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:56.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.151+0000 7f27a69c8740 1 -- 172.21.15.43:0/1222658016 shutdown_connections 2024-03-28T22:44:56.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.151+0000 7f27a69c8740 1 -- 172.21.15.43:0/1222658016 wait complete. 2024-03-28T22:44:56.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.151+0000 7f27a69c8740 20 client.28340 trim_cache size 0 max 16384 2024-03-28T22:44:56.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.170+0000 7f27a69c8740 1 Processor -- start 2024-03-28T22:44:56.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.170+0000 7f27a69c8740 1 -- start start 2024-03-28T22:44:56.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.170+0000 7f27a69c8740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c1373570d0 0x55c13735c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.170+0000 7f27a69c8740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c137357db0 0x55c137351650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.170+0000 7f27a69c8740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c137360440 0x55c13736c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.170+0000 7f27a69c8740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f279c1b4f40 con 0x55c1373570d0 2024-03-28T22:44:56.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.170+0000 7f27a69c8740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f279c1b6a80 con 0x55c137357db0 2024-03-28T22:44:56.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.170+0000 7f27a69c8740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f279c1b5610 con 0x55c137360440 2024-03-28T22:44:56.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.171+0000 7f27a1200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c1373570d0 0x55c13735c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.171+0000 7f2793e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c137357db0 0x55c137351650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.171+0000 7f27a1200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c1373570d0 0x55c13735c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44978/0 (socket says 172.21.15.43:44978) 2024-03-28T22:44:56.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.171+0000 7f2793e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c137357db0 0x55c137351650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47096/0 (socket says 172.21.15.43:47096) 2024-03-28T22:44:56.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.171+0000 7f27a1c00640 1 --2- 172.21.15.43:0/3918691314 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c137360440 0x55c13736c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.171+0000 7f27a1200640 1 -- 172.21.15.43:0/3918691314 learned_addr learned my addr 172.21.15.43:0/3918691314 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:56.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.171+0000 7f2793e00640 1 -- 172.21.15.43:0/3918691314 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c137360440 msgr2=0x55c13736c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:56.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.171+0000 7f2793e00640 1 --2- 172.21.15.43:0/3918691314 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c137360440 0x55c13736c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.171+0000 7f2793e00640 1 -- 172.21.15.43:0/3918691314 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c1373570d0 msgr2=0x55c13735c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:56.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.171+0000 7f2793e00640 1 --2- 172.21.15.43:0/3918691314 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c1373570d0 0x55c13735c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.171+0000 7f2793e00640 1 -- 172.21.15.43:0/3918691314 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c136fcf3f0 con 0x55c137357db0 2024-03-28T22:44:56.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.171+0000 7f2793e00640 1 --2- 172.21.15.43:0/3918691314 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c137357db0 0x55c137351650 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f2784004650 tx=0x7f2784003f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d83ba58b86f7ab1d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:56.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.172+0000 7f2791600640 1 -- 172.21.15.43:0/3918691314 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f278404b070 con 0x55c137357db0 2024-03-28T22:44:56.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.172+0000 7f2791600640 1 -- 172.21.15.43:0/3918691314 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2784004c80 con 0x55c137357db0 2024-03-28T22:44:56.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.172+0000 7f2791600640 1 -- 172.21.15.43:0/3918691314 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27840bc1d0 con 0x55c137357db0 2024-03-28T22:44:56.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.172+0000 7f27a69c8740 1 -- 172.21.15.43:0/3918691314 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c137357db0 msgr2=0x55c137351650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:56.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.172+0000 7f27a69c8740 1 --2- 172.21.15.43:0/3918691314 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c137357db0 0x55c137351650 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f2784004650 tx=0x7f2784003f90 comp rx=0 tx=0).stop 2024-03-28T22:44:56.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.172+0000 7f27a69c8740 1 -- 172.21.15.43:0/3918691314 shutdown_connections 2024-03-28T22:44:56.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.172+0000 7f27a69c8740 1 --2- 172.21.15.43:0/3918691314 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c137357db0 0x55c137351650 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.172+0000 7f27a69c8740 1 --2- 172.21.15.43:0/3918691314 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c137360440 0x55c13736c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.172+0000 7f27a69c8740 1 --2- 172.21.15.43:0/3918691314 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c1373570d0 0x55c13735c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.172+0000 7f27a69c8740 1 -- 172.21.15.43:0/3918691314 >> 172.21.15.43:0/3918691314 conn(0x55c13731da10 msgr2=0x55c137353800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:56.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.173+0000 7f27a69c8740 1 -- 172.21.15.43:0/3918691314 shutdown_connections 2024-03-28T22:44:56.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.173+0000 7f27a69c8740 1 -- 172.21.15.43:0/3918691314 wait complete. 2024-03-28T22:44:56.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.173+0000 7f27a69c8740 1 -- start start 2024-03-28T22:44:56.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.173+0000 7f27a69c8740 1 Processor -- start 2024-03-28T22:44:56.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.174+0000 7f27a69c8740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c1373570d0 0x55c137368930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.174+0000 7f27a69c8740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c137357db0 0x55c137360bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.174+0000 7f27a69c8740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c137360440 0x55c137361130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.174+0000 7f27a69c8740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f279c1b63b0 con 0x55c137357db0 2024-03-28T22:44:56.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.174+0000 7f27a69c8740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f279c1b5ce0 con 0x55c137360440 2024-03-28T22:44:56.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.174+0000 7f27a69c8740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f279c1b2f40 con 0x55c1373570d0 2024-03-28T22:44:56.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.174+0000 7f27a69c8740 10 client.0 _ll_register_callbacks cb 0x7f2784009260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:44:56.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.174+0000 7f27a1200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c1373570d0 0x55c137368930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.174+0000 7f27a1200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c1373570d0 0x55c137368930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:39926/0 (socket says 172.21.15.43:39926) 2024-03-28T22:44:56.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.174+0000 7f27a1200640 1 -- 172.21.15.43:0/1735516760 learned_addr learned my addr 172.21.15.43:0/1735516760 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:56.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.174+0000 7f27a1c00640 1 --2- 172.21.15.43:0/1735516760 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c137360440 0x55c137361130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.174+0000 7f2793e00640 1 --2- 172.21.15.43:0/1735516760 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c137357db0 0x55c137360bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.174+0000 7f27a1200640 1 -- 172.21.15.43:0/1735516760 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c137360440 msgr2=0x55c137361130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:56.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.174+0000 7f27a1200640 1 --2- 172.21.15.43:0/1735516760 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c137360440 0x55c137361130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.174+0000 7f27a1200640 1 -- 172.21.15.43:0/1735516760 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c137357db0 msgr2=0x55c137360bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:56.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.174+0000 7f27a1200640 1 --2- 172.21.15.43:0/1735516760 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c137357db0 0x55c137360bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.174+0000 7f27a1200640 1 -- 172.21.15.43:0/1735516760 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c136f8f0f0 con 0x55c1373570d0 2024-03-28T22:44:56.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.174+0000 7f2793e00640 1 --2- 172.21.15.43:0/1735516760 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c137357db0 0x55c137360bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:44:56.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.175+0000 7f27a1200640 1 --2- 172.21.15.43:0/1735516760 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c1373570d0 0x55c137368930 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f278c009250 tx=0x7f278c0094e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=deec0018b4484761 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:56.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.175+0000 7f278aa00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:44:56.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.175+0000 7f278aa00640 1 -- 172.21.15.43:0/1735516760 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f278c01e040 con 0x55c1373570d0 2024-03-28T22:44:56.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.175+0000 7f278aa00640 1 -- 172.21.15.43:0/1735516760 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f278c0054d0 con 0x55c1373570d0 2024-03-28T22:44:56.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.175+0000 7f27a69c8740 1 -- 172.21.15.43:0/1735516760 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2784004c80 con 0x55c1373570d0 2024-03-28T22:44:56.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.175+0000 7f278aa00640 1 -- 172.21.15.43:0/1735516760 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f278c02c250 con 0x55c1373570d0 2024-03-28T22:44:56.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.175+0000 7f27a69c8740 10 client.28285 fetch_fsmap learned FSMap version 32 2024-03-28T22:44:56.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.175+0000 7f27a69c8740 1 -- 172.21.15.43:0/1735516760 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c1373684e0 con 0x55c1373570d0 2024-03-28T22:44:56.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.175+0000 7f278aa00640 1 -- 172.21.15.43:0/1735516760 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2784004c80 con 0x55c1373570d0 2024-03-28T22:44:56.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.176+0000 7f278aa00640 1 -- 172.21.15.43:0/1735516760 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f278c00fb70 con 0x55c1373570d0 2024-03-28T22:44:56.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.176+0000 7f27a69c8740 10 client.28285 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:44:56.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.176+0000 7f27a69c8740 10 client.28285 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:44:56.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.176+0000 7f27a69c8740 20 client.28285 populate_metadata read hostname 'smithi043' 2024-03-28T22:44:56.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.176+0000 7f27a69c8740 1 --2- 172.21.15.43:0/1735516760 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c1373d6f00 0x55c1373d9320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.176+0000 7f27a69c8740 4 client.28285 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:44:56.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.176+0000 7f27a69c8740 1 -- 172.21.15.43:0/1735516760 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c1372c2990 con 0x55c1373d6f00 2024-03-28T22:44:56.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.176+0000 7f2793e00640 1 --2- 172.21.15.43:0/1735516760 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c1373d6f00 0x55c1373d9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.176+0000 7f2793e00640 1 --2- 172.21.15.43:0/1735516760 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c1373d6f00 0x55c1373d9320 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b7af2eb3bdad38b server_cookie=fcb9f2b9b4f92d23 in_seq=0 out_seq=0 2024-03-28T22:44:56.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.176+0000 7f278aa00640 10 client.28285 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:44:56.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.177+0000 7f278aa00640 1 -- 172.21.15.43:0/1735516760 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c1372c2990 con 0x55c1373d6f00 2024-03-28T22:44:56.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.177+0000 7f278aa00640 10 client.28285 handle_command_reply: tid=0 2024-03-28T22:44:56.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.177+0000 7f27a69c8740 1 client.28285 shutdown 2024-03-28T22:44:56.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.177+0000 7f27a69c8740 1 -- 172.21.15.43:0/1735516760 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c1373570d0 msgr2=0x55c137368930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:56.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.177+0000 7f27a69c8740 1 --2- 172.21.15.43:0/1735516760 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c1373570d0 0x55c137368930 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f278c009250 tx=0x7f278c0094e0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.177+0000 7f27a69c8740 1 -- 172.21.15.43:0/1735516760 shutdown_connections 2024-03-28T22:44:56.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.177+0000 7f27a69c8740 1 --2- 172.21.15.43:0/1735516760 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55c1373d6f00 0x55c1373d9320 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.177+0000 7f27a69c8740 1 --2- 172.21.15.43:0/1735516760 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c137360440 0x55c137361130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.177+0000 7f27a69c8740 1 --2- 172.21.15.43:0/1735516760 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c137357db0 0x55c137360bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.177+0000 7f27a69c8740 1 --2- 172.21.15.43:0/1735516760 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c1373570d0 0x55c137368930 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.177+0000 7f27a69c8740 1 -- 172.21.15.43:0/1735516760 >> 172.21.15.43:0/1735516760 conn(0x55c13731da10 msgr2=0x55c137352de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:56.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.177+0000 7f278aa00640 0 client.28285 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:44:56.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.177+0000 7f27a69c8740 1 -- 172.21.15.43:0/1735516760 shutdown_connections 2024-03-28T22:44:56.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.177+0000 7f27a69c8740 1 -- 172.21.15.43:0/1735516760 wait complete. 2024-03-28T22:44:56.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.177+0000 7f27a69c8740 20 client.28285 trim_cache size 0 max 16384 2024-03-28T22:44:56.189 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:44:56.189 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-03-28T22:44:56.189 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-03-28T22:44:56.189 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:44:56.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.178+0000 7f27a2600640 1 -- 172.21.15.43:0/31299574 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f27800a1f70 msgr2=0x7f27800a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:56.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.178+0000 7f27a2600640 1 --2- 172.21.15.43:0/31299574 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f27800a1f70 0x7f27800a4430 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2794005fd0 tx=0x7f2794009500 comp rx=0 tx=0).stop 2024-03-28T22:44:56.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.178+0000 7f27a2600640 1 -- 172.21.15.43:0/31299574 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f279c10cf50 msgr2=0x7f279c19ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:56.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.178+0000 7f27a2600640 1 --2- 172.21.15.43:0/31299574 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f279c10cf50 0x7f279c19ea30 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f2784009230 tx=0x7f27840042f0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.178+0000 7f27a2600640 1 -- 172.21.15.43:0/31299574 shutdown_connections 2024-03-28T22:44:56.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.178+0000 7f27a2600640 1 --2- 172.21.15.43:0/31299574 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f27800a1f70 0x7f27800a4430 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.178+0000 7f27a2600640 1 --2- 172.21.15.43:0/31299574 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f279c1a2db0 0x7f279c1a3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.178+0000 7f27a2600640 1 --2- 172.21.15.43:0/31299574 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f279c10cf50 0x7f279c19ea30 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.178+0000 7f27a2600640 1 --2- 172.21.15.43:0/31299574 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f279c107f00 0x7f279c19e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.178+0000 7f27a2600640 1 -- 172.21.15.43:0/31299574 >> 172.21.15.43:0/31299574 conn(0x7f279c077570 msgr2=0x7f279c079130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:56.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.178+0000 7f27a2600640 1 -- 172.21.15.43:0/31299574 shutdown_connections 2024-03-28T22:44:56.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.178+0000 7f27a2600640 1 -- 172.21.15.43:0/31299574 wait complete. 2024-03-28T22:44:56.195 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:44:56.195 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:44:56.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.418+0000 7f75e6c00640 1 -- 172.21.15.43:0/4125113655 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f75e00765a0 msgr2=0x7f75e0112060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:56.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.418+0000 7f75e6c00640 1 --2- 172.21.15.43:0/4125113655 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f75e00765a0 0x7f75e0112060 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f75c8009930 tx=0x7f75c802bc30 comp rx=0 tx=0).stop 2024-03-28T22:44:56.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.418+0000 7f75e6c00640 1 -- 172.21.15.43:0/4125113655 shutdown_connections 2024-03-28T22:44:56.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.418+0000 7f75e6c00640 1 --2- 172.21.15.43:0/4125113655 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f75e00765a0 0x7f75e0112060 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.418+0000 7f75e6c00640 1 --2- 172.21.15.43:0/4125113655 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75e01125a0 0x7f75e0114990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.418+0000 7f75e6c00640 1 --2- 172.21.15.43:0/4125113655 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75e0075c80 0x7f75e0076060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.418+0000 7f75e6c00640 1 -- 172.21.15.43:0/4125113655 >> 172.21.15.43:0/4125113655 conn(0x7f75e00ff390 msgr2=0x7f75e01017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:56.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.418+0000 7f75e6c00640 1 -- 172.21.15.43:0/4125113655 shutdown_connections 2024-03-28T22:44:56.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.418+0000 7f75e6c00640 1 -- 172.21.15.43:0/4125113655 wait complete. 2024-03-28T22:44:56.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.419+0000 7f75e6c00640 1 Processor -- start 2024-03-28T22:44:56.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.419+0000 7f75e6c00640 1 -- start start 2024-03-28T22:44:56.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.419+0000 7f75e6c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f75e0075c80 0x7f75e019e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.419+0000 7f75e6c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75e00765a0 0x7f75e019ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.419+0000 7f75e6c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75e01125a0 0x7f75e01a2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.419+0000 7f75e6c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f75e0118140 con 0x7f75e00765a0 2024-03-28T22:44:56.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.419+0000 7f75e6c00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f75e0117fc0 con 0x7f75e0075c80 2024-03-28T22:44:56.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.419+0000 7f75e6c00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f75e01182c0 con 0x7f75e01125a0 2024-03-28T22:44:56.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.420+0000 7f75dfe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f75e0075c80 0x7f75e019e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.420+0000 7f75dfe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f75e0075c80 0x7f75e019e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47116/0 (socket says 172.21.15.43:47116) 2024-03-28T22:44:56.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.420+0000 7f75dfe00640 1 -- 172.21.15.43:0/2336824896 learned_addr learned my addr 172.21.15.43:0/2336824896 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:56.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.420+0000 7f75e4c00640 1 --2- 172.21.15.43:0/2336824896 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75e01125a0 0x7f75e01a2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.420+0000 7f75d7e00640 1 --2- 172.21.15.43:0/2336824896 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75e00765a0 0x7f75e019ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.420+0000 7f75dfe00640 1 -- 172.21.15.43:0/2336824896 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75e01125a0 msgr2=0x7f75e01a2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:56.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.420+0000 7f75dfe00640 1 --2- 172.21.15.43:0/2336824896 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75e01125a0 0x7f75e01a2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.420+0000 7f75dfe00640 1 -- 172.21.15.43:0/2336824896 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75e00765a0 msgr2=0x7f75e019ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:56.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.420+0000 7f75dfe00640 1 --2- 172.21.15.43:0/2336824896 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75e00765a0 0x7f75e019ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.420+0000 7f75dfe00640 1 -- 172.21.15.43:0/2336824896 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75e01a3690 con 0x7f75e0075c80 2024-03-28T22:44:56.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.420+0000 7f75d7e00640 1 --2- 172.21.15.43:0/2336824896 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75e00765a0 0x7f75e019ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:44:56.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.420+0000 7f75dfe00640 1 --2- 172.21.15.43:0/2336824896 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f75e0075c80 0x7f75e019e6a0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f75d000e9f0 tx=0x7f75d000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:56.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.420+0000 7f75dd600640 1 -- 172.21.15.43:0/2336824896 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d000cd80 con 0x7f75e0075c80 2024-03-28T22:44:56.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.421+0000 7f75dd600640 1 -- 172.21.15.43:0/2336824896 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75d0010910 con 0x7f75e0075c80 2024-03-28T22:44:56.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.421+0000 7f75dd600640 1 -- 172.21.15.43:0/2336824896 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d0018990 con 0x7f75e0075c80 2024-03-28T22:44:56.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.421+0000 7f75e6c00640 1 -- 172.21.15.43:0/2336824896 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75e01a3980 con 0x7f75e0075c80 2024-03-28T22:44:56.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.421+0000 7f75e6c00640 1 -- 172.21.15.43:0/2336824896 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75e01ab210 con 0x7f75e0075c80 2024-03-28T22:44:56.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.422+0000 7f75e6c00640 1 -- 172.21.15.43:0/2336824896 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75e0068540 con 0x7f75e0075c80 2024-03-28T22:44:56.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.424+0000 7f75dd600640 1 -- 172.21.15.43:0/2336824896 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f75d0010ac0 con 0x7f75e0075c80 2024-03-28T22:44:56.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.425+0000 7f75dd600640 1 --2- 172.21.15.43:0/2336824896 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f75c40a1f70 0x7f75c40a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.425+0000 7f75dd600640 1 -- 172.21.15.43:0/2336824896 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f75d00ce940 con 0x7f75e0075c80 2024-03-28T22:44:56.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.425+0000 7f75d7e00640 1 --2- 172.21.15.43:0/2336824896 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f75c40a1f70 0x7f75c40a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.426+0000 7f75d7e00640 1 --2- 172.21.15.43:0/2336824896 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f75c40a1f70 0x7f75c40a4430 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f75c8009900 tx=0x7f75c802df90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:56.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.429+0000 7f75dd600640 1 -- 172.21.15.43:0/2336824896 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f75d009b600 con 0x7f75e0075c80 2024-03-28T22:44:56.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.681+0000 7f75e6c00640 1 -- 172.21.15.43:0/2336824896 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f75e002d170 con 0x7f75e0075c80 2024-03-28T22:44:56.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.682+0000 7f75dd600640 1 -- 172.21.15.43:0/2336824896 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11931 (secure 0 0 0) 0x7f75d009af00 con 0x7f75e0075c80 2024-03-28T22:44:56.680 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:44:56.681 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:33:28.906396+0000","modified":"2024-03-28T22:40:08.672168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24572,"mds_1":24556,"mds_2":24535,"mds_3":24523,"mds_4":24518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26717":{"gid":26717,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/215779342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":215779342},{"type":"v1","addr":"172.21.15.82:6835","nonce":215779342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24535,24556,24518,24523,24572]},"id":1}]} 2024-03-28T22:44:56.681 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 32 2024-03-28T22:44:56.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.685+0000 7f75e6c00640 1 -- 172.21.15.43:0/2336824896 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f75c40a1f70 msgr2=0x7f75c40a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:56.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.685+0000 7f75e6c00640 1 --2- 172.21.15.43:0/2336824896 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f75c40a1f70 0x7f75c40a4430 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f75c8009900 tx=0x7f75c802df90 comp rx=0 tx=0).stop 2024-03-28T22:44:56.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.685+0000 7f75e6c00640 1 -- 172.21.15.43:0/2336824896 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f75e0075c80 msgr2=0x7f75e019e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:56.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.685+0000 7f75e6c00640 1 --2- 172.21.15.43:0/2336824896 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f75e0075c80 0x7f75e019e6a0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f75d000e9f0 tx=0x7f75d000eeb0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.685+0000 7f75e6c00640 1 -- 172.21.15.43:0/2336824896 shutdown_connections 2024-03-28T22:44:56.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.685+0000 7f75e6c00640 1 --2- 172.21.15.43:0/2336824896 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f75c40a1f70 0x7f75c40a4430 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.685+0000 7f75e6c00640 1 --2- 172.21.15.43:0/2336824896 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f75e01125a0 0x7f75e01a2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.685+0000 7f75e6c00640 1 --2- 172.21.15.43:0/2336824896 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75e00765a0 0x7f75e019ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.685+0000 7f75e6c00640 1 --2- 172.21.15.43:0/2336824896 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f75e0075c80 0x7f75e019e6a0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.685+0000 7f75e6c00640 1 -- 172.21.15.43:0/2336824896 >> 172.21.15.43:0/2336824896 conn(0x7f75e00ff390 msgr2=0x7f75e0113cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:56.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.685+0000 7f75e6c00640 1 -- 172.21.15.43:0/2336824896 shutdown_connections 2024-03-28T22:44:56.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.685+0000 7f75e6c00640 1 -- 172.21.15.43:0/2336824896 wait complete. 2024-03-28T22:44:56.704 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T22:44:56.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.939+0000 7f2cda600640 1 -- 172.21.15.43:0/3644744308 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2cd4108910 msgr2=0x7f2cd410c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:56.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.939+0000 7f2cda600640 1 --2- 172.21.15.43:0/3644744308 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2cd4108910 0x7f2cd410c950 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0009900 tx=0x7f2cc002bc30 comp rx=0 tx=0).stop 2024-03-28T22:44:56.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.939+0000 7f2cda600640 1 -- 172.21.15.43:0/3644744308 shutdown_connections 2024-03-28T22:44:56.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.939+0000 7f2cda600640 1 --2- 172.21.15.43:0/3644744308 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2cd4108910 0x7f2cd410c950 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.939+0000 7f2cda600640 1 --2- 172.21.15.43:0/3644744308 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2cd410d030 0x7f2cd4114c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.939+0000 7f2cda600640 1 --2- 172.21.15.43:0/3644744308 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2cd4107f60 0x7f2cd4108340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.939+0000 7f2cda600640 1 -- 172.21.15.43:0/3644744308 >> 172.21.15.43:0/3644744308 conn(0x7f2cd4075c90 msgr2=0x7f2cd40780b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:56.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.939+0000 7f2cda600640 1 -- 172.21.15.43:0/3644744308 shutdown_connections 2024-03-28T22:44:56.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.939+0000 7f2cda600640 1 -- 172.21.15.43:0/3644744308 wait complete. 2024-03-28T22:44:56.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.940+0000 7f2cda600640 1 Processor -- start 2024-03-28T22:44:56.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.941+0000 7f2cda600640 1 -- start start 2024-03-28T22:44:56.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.941+0000 7f2cda600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2cd4107f60 0x7f2cd419e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.941+0000 7f2cda600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2cd410d030 0x7f2cd419eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.941+0000 7f2cda600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2cd41a2f10 0x7f2cd41a33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.941+0000 7f2cda600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd4118110 con 0x7f2cd410d030 2024-03-28T22:44:56.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.941+0000 7f2cda600640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd4117f90 con 0x7f2cd4107f60 2024-03-28T22:44:56.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.941+0000 7f2cda600640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd4118290 con 0x7f2cd41a2f10 2024-03-28T22:44:56.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.941+0000 7f2cd3e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2cd41a2f10 0x7f2cd41a33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.941+0000 7f2cd3e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2cd41a2f10 0x7f2cd41a33c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:39970/0 (socket says 172.21.15.43:39970) 2024-03-28T22:44:56.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.941+0000 7f2cd3400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2cd4107f60 0x7f2cd419e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.941+0000 7f2cd3e00640 1 -- 172.21.15.43:0/2357055626 learned_addr learned my addr 172.21.15.43:0/2357055626 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:56.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.942+0000 7f2cd3e00640 1 -- 172.21.15.43:0/2357055626 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2cd4107f60 msgr2=0x7f2cd419e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:56.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.942+0000 7f2cd3e00640 1 --2- 172.21.15.43:0/2357055626 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2cd4107f60 0x7f2cd419e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.942+0000 7f2cd3e00640 1 -- 172.21.15.43:0/2357055626 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2cd410d030 msgr2=0x7f2cd419eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:56.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.942+0000 7f2cd2a00640 1 --2- 172.21.15.43:0/2357055626 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2cd410d030 0x7f2cd419eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.942+0000 7f2cd3e00640 1 --2- 172.21.15.43:0/2357055626 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2cd410d030 0x7f2cd419eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.942+0000 7f2cd3e00640 1 -- 172.21.15.43:0/2357055626 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cd41a3a00 con 0x7f2cd41a2f10 2024-03-28T22:44:56.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.942+0000 7f2cd2a00640 1 --2- 172.21.15.43:0/2357055626 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2cd410d030 0x7f2cd419eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:44:56.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.942+0000 7f2cd3e00640 1 --2- 172.21.15.43:0/2357055626 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2cd41a2f10 0x7f2cd41a33c0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc00eb10 tx=0x7f2ccc00c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:56.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.942+0000 7f2ccbe00640 1 -- 172.21.15.43:0/2357055626 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc004b20 con 0x7f2cd41a2f10 2024-03-28T22:44:56.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.943+0000 7f2ccbe00640 1 -- 172.21.15.43:0/2357055626 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ccc004cc0 con 0x7f2cd41a2f10 2024-03-28T22:44:56.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.943+0000 7f2cda600640 1 -- 172.21.15.43:0/2357055626 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cd41a3cf0 con 0x7f2cd41a2f10 2024-03-28T22:44:56.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.943+0000 7f2ccbe00640 1 -- 172.21.15.43:0/2357055626 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc010760 con 0x7f2cd41a2f10 2024-03-28T22:44:56.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.943+0000 7f2cda600640 1 -- 172.21.15.43:0/2357055626 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cd41ab4e0 con 0x7f2cd41a2f10 2024-03-28T22:44:56.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.946+0000 7f2cdd3ae740 1 Processor -- start 2024-03-28T22:44:56.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.946+0000 7f2ccbe00640 1 -- 172.21.15.43:0/2357055626 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f2ccc005020 con 0x7f2cd41a2f10 2024-03-28T22:44:56.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.946+0000 7f2cdd3ae740 1 -- start start 2024-03-28T22:44:56.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.947+0000 7f2cdd3ae740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4ef1a5380 0x55e4ef1a5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.947+0000 7f2cdd3ae740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4ef1a5ca0 0x55e4ef1ae340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.947+0000 7f2cdd3ae740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4ef1ae880 0x55e4ef1b0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.947+0000 7f2cdd3ae740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e4ef109050 con 0x55e4ef1ae880 2024-03-28T22:44:56.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.947+0000 7f2cdd3ae740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55e4ef0b1ce0 con 0x55e4ef1a5380 2024-03-28T22:44:56.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.947+0000 7f2cdd3ae740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55e4ef1050f0 con 0x55e4ef1a5ca0 2024-03-28T22:44:56.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.947+0000 7f2cd3400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4ef1a5ca0 0x55e4ef1ae340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.947+0000 7f2cd3400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4ef1a5ca0 0x55e4ef1ae340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:39974/0 (socket says 172.21.15.43:39974) 2024-03-28T22:44:56.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.947+0000 7f2cd3400640 1 -- 172.21.15.43:0/2626709651 learned_addr learned my addr 172.21.15.43:0/2626709651 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:56.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.947+0000 7f2cd2a00640 1 --2- 172.21.15.43:0/2626709651 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4ef1a5380 0x55e4ef1a5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.947+0000 7f2cd2a00640 1 --2- 172.21.15.43:0/2626709651 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4ef1ae880 0x55e4ef1b0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.948+0000 7f2cd3400640 1 -- 172.21.15.43:0/2626709651 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4ef1a5380 msgr2=0x55e4ef1a5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:56.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.948+0000 7f2cd3400640 1 --2- 172.21.15.43:0/2626709651 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4ef1a5380 0x55e4ef1a5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.948+0000 7f2cd3400640 1 -- 172.21.15.43:0/2626709651 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4ef1ae880 msgr2=0x55e4ef1b0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:56.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.948+0000 7f2cd3400640 1 --2- 172.21.15.43:0/2626709651 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4ef1ae880 0x55e4ef1b0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.948+0000 7f2cd3400640 1 -- 172.21.15.43:0/2626709651 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4eee1d3f0 con 0x55e4ef1a5ca0 2024-03-28T22:44:56.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.948+0000 7f2ccbe00640 1 --2- 172.21.15.43:0/2357055626 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f2cb80a1ef0 0x7f2cb80a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.948+0000 7f2cd3400640 1 --2- 172.21.15.43:0/2626709651 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4ef1a5ca0 0x55e4ef1ae340 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f2cc400bcd0 tx=0x7f2cc40107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e15642d84749af0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:56.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.948+0000 7f2ccbe00640 1 -- 172.21.15.43:0/2357055626 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f2ccc0cecd0 con 0x7f2cd41a2f10 2024-03-28T22:44:56.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.948+0000 7f2cd2a00640 1 --2- 172.21.15.43:0/2357055626 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f2cb80a1ef0 0x7f2cb80a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.948+0000 7f2cc8c00640 1 -- 172.21.15.43:0/2626709651 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc4005220 con 0x55e4ef1a5ca0 2024-03-28T22:44:56.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.948+0000 7f2cc8c00640 1 -- 172.21.15.43:0/2626709651 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cc4004c80 con 0x55e4ef1a5ca0 2024-03-28T22:44:56.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.948+0000 7f2cc8c00640 1 -- 172.21.15.43:0/2626709651 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc40117e0 con 0x55e4ef1a5ca0 2024-03-28T22:44:56.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.949+0000 7f2cd2a00640 1 --2- 172.21.15.43:0/2357055626 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f2cb80a1ef0 0x7f2cb80a43b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0036040 tx=0x7f2cc002e040 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:56.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.949+0000 7f2cdd3ae740 1 -- 172.21.15.43:0/2626709651 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4ef1a5ca0 msgr2=0x55e4ef1ae340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:56.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.949+0000 7f2cdd3ae740 1 --2- 172.21.15.43:0/2626709651 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4ef1a5ca0 0x55e4ef1ae340 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f2cc400bcd0 tx=0x7f2cc40107a0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.949+0000 7f2cdd3ae740 1 -- 172.21.15.43:0/2626709651 shutdown_connections 2024-03-28T22:44:56.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.949+0000 7f2cdd3ae740 1 --2- 172.21.15.43:0/2626709651 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4ef1ae880 0x55e4ef1b0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.949+0000 7f2cdd3ae740 1 --2- 172.21.15.43:0/2626709651 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4ef1a5ca0 0x55e4ef1ae340 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.949+0000 7f2cdd3ae740 1 --2- 172.21.15.43:0/2626709651 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4ef1a5380 0x55e4ef1a5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.949+0000 7f2cdd3ae740 1 -- 172.21.15.43:0/2626709651 >> 172.21.15.43:0/2626709651 conn(0x55e4ef19fab0 msgr2=0x55e4ef19fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:56.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.949+0000 7f2cdd3ae740 1 -- 172.21.15.43:0/2626709651 shutdown_connections 2024-03-28T22:44:56.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.949+0000 7f2cdd3ae740 1 -- 172.21.15.43:0/2626709651 wait complete. 2024-03-28T22:44:56.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.950+0000 7f2cdd3ae740 1 -- start start 2024-03-28T22:44:56.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.950+0000 7f2cdd3ae740 1 Processor -- start 2024-03-28T22:44:56.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.951+0000 7f2cdd3ae740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4ef1a5380 0x55e4ef1ab8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.951+0000 7f2cdd3ae740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4ef1a5ca0 0x55e4ef1abe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.951+0000 7f2cdd3ae740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4ef1ae880 0x55e4ef1ac360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.951+0000 7f2cdd3ae740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e4ef1b1670 con 0x55e4ef1ae880 2024-03-28T22:44:56.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.951+0000 7f2cdd3ae740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55e4ef0a72f0 con 0x55e4ef1a5380 2024-03-28T22:44:56.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.951+0000 7f2cdd3ae740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55e4ef1b17f0 con 0x55e4ef1a5ca0 2024-03-28T22:44:56.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.951+0000 7f2cd3400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4ef1a5380 0x55e4ef1ab8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.951+0000 7f2cdd3ae740 10 client.0 _ll_register_callbacks cb 0x55e4eee1cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:44:56.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.951+0000 7f2cd3400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4ef1a5380 0x55e4ef1ab8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47146/0 (socket says 172.21.15.43:47146) 2024-03-28T22:44:56.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.951+0000 7f2cd3400640 1 -- 172.21.15.43:0/1862772481 learned_addr learned my addr 172.21.15.43:0/1862772481 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:56.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.951+0000 7f2cd2a00640 1 --2- 172.21.15.43:0/1862772481 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4ef1a5ca0 0x55e4ef1abe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.951+0000 7f2cd3e00640 1 --2- 172.21.15.43:0/1862772481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4ef1ae880 0x55e4ef1ac360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.951+0000 7f2cd3400640 1 -- 172.21.15.43:0/1862772481 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4ef1a5ca0 msgr2=0x55e4ef1abe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.951+0000 7f2cd3400640 1 --2- 172.21.15.43:0/1862772481 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4ef1a5ca0 0x55e4ef1abe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.951+0000 7f2cd3400640 1 -- 172.21.15.43:0/1862772481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4ef1ae880 msgr2=0x55e4ef1ac360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.951+0000 7f2cd3400640 1 --2- 172.21.15.43:0/1862772481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4ef1ae880 0x55e4ef1ac360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.951+0000 7f2cd3400640 1 -- 172.21.15.43:0/1862772481 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4eeee77d0 con 0x55e4ef1a5380 2024-03-28T22:44:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.951+0000 7f2cd3e00640 1 --2- 172.21.15.43:0/1862772481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4ef1ae880 0x55e4ef1ac360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:44:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.952+0000 7f2cd3400640 1 --2- 172.21.15.43:0/1862772481 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4ef1a5380 0x55e4ef1ab8e0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4005d00 tx=0x7f2cc40048f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b008be46dbe81748 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.952+0000 7f2cbd600640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:44:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.952+0000 7f2cbd600640 1 -- 172.21.15.43:0/1862772481 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc4004ae0 con 0x55e4ef1a5380 2024-03-28T22:44:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.952+0000 7f2cbd600640 1 -- 172.21.15.43:0/1862772481 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f2cc4005240 con 0x55e4ef1a5380 2024-03-28T22:44:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.952+0000 7f2cdd3ae740 1 -- 172.21.15.43:0/1862772481 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2cc4004c80 con 0x55e4ef1a5380 2024-03-28T22:44:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.952+0000 7f2cbd600640 1 -- 172.21.15.43:0/1862772481 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2cc400be80 con 0x55e4ef1a5380 2024-03-28T22:44:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.952+0000 7f2cdd3ae740 10 client.28364 fetch_fsmap learned FSMap version 32 2024-03-28T22:44:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.952+0000 7f2cdd3ae740 1 -- 172.21.15.43:0/1862772481 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e4ef1bd920 con 0x55e4ef1a5380 2024-03-28T22:44:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.952+0000 7f2cbd600640 1 -- 172.21.15.43:0/1862772481 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cc4004c80 con 0x55e4ef1a5380 2024-03-28T22:44:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.952+0000 7f2cbd600640 1 -- 172.21.15.43:0/1862772481 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f2cc4025830 con 0x55e4ef1a5380 2024-03-28T22:44:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.953+0000 7f2cdd3ae740 10 client.28364 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:44:56.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.953+0000 7f2cdd3ae740 10 client.28364 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:44:56.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.953+0000 7f2cdd3ae740 20 client.28364 populate_metadata read hostname 'smithi043' 2024-03-28T22:44:56.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.953+0000 7f2cdd3ae740 1 --2- 172.21.15.43:0/1862772481 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55e4ef1bed80 0x55e4ef1c1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.953+0000 7f2cdd3ae740 4 client.28364 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:44:56.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.953+0000 7f2cdd3ae740 1 -- 172.21.15.43:0/1862772481 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e4ef0b2680 con 0x55e4ef1bed80 2024-03-28T22:44:56.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.953+0000 7f2cd2a00640 1 --2- 172.21.15.43:0/1862772481 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55e4ef1bed80 0x55e4ef1c1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.954+0000 7f2cd2a00640 1 --2- 172.21.15.43:0/1862772481 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55e4ef1bed80 0x55e4ef1c1170 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=27235e6869896f27 server_cookie=70edb4dcc8451af7 in_seq=0 out_seq=0 2024-03-28T22:44:56.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.954+0000 7f2cbd600640 10 client.28364 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:44:56.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.955+0000 7f2cbd600640 1 -- 172.21.15.43:0/1862772481 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e4ef0b2680 con 0x55e4ef1bed80 2024-03-28T22:44:56.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.955+0000 7f2cbd600640 10 client.28364 handle_command_reply: tid=0 2024-03-28T22:44:56.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.955+0000 7f2cdd3ae740 1 client.28364 shutdown 2024-03-28T22:44:56.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.955+0000 7f2cdd3ae740 1 -- 172.21.15.43:0/1862772481 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4ef1a5380 msgr2=0x55e4ef1ab8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:56.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.955+0000 7f2cdd3ae740 1 --2- 172.21.15.43:0/1862772481 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4ef1a5380 0x55e4ef1ab8e0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4005d00 tx=0x7f2cc40048f0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.955+0000 7f2cdd3ae740 1 -- 172.21.15.43:0/1862772481 shutdown_connections 2024-03-28T22:44:56.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.955+0000 7f2cdd3ae740 1 --2- 172.21.15.43:0/1862772481 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55e4ef1bed80 0x55e4ef1c1170 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.955+0000 7f2cdd3ae740 1 --2- 172.21.15.43:0/1862772481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4ef1ae880 0x55e4ef1ac360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.955+0000 7f2cdd3ae740 1 --2- 172.21.15.43:0/1862772481 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4ef1a5ca0 0x55e4ef1abe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.955+0000 7f2cdd3ae740 1 --2- 172.21.15.43:0/1862772481 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4ef1a5380 0x55e4ef1ab8e0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.955+0000 7f2cbd600640 0 client.28364 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:44:56.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.955+0000 7f2cdd3ae740 1 -- 172.21.15.43:0/1862772481 >> 172.21.15.43:0/1862772481 conn(0x55e4ef19fab0 msgr2=0x55e4ef1a1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:56.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.955+0000 7f2cdd3ae740 1 -- 172.21.15.43:0/1862772481 shutdown_connections 2024-03-28T22:44:56.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.955+0000 7f2cdd3ae740 1 -- 172.21.15.43:0/1862772481 wait complete. 2024-03-28T22:44:56.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.955+0000 7f2cdd3ae740 20 client.28364 trim_cache size 0 max 16384 2024-03-28T22:44:56.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.972+0000 7f2cdd3ae740 1 Processor -- start 2024-03-28T22:44:56.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.972+0000 7f2cdd3ae740 1 -- start start 2024-03-28T22:44:56.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.972+0000 7f2cdd3ae740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4ef1a5380 0x55e4ef1ba410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.972+0000 7f2cdd3ae740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4ef1a5ca0 0x55e4ef1ac180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.972+0000 7f2cdd3ae740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4ef1ae880 0x55e4ef1c6930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.972+0000 7f2cdd3ae740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd41b5400 con 0x55e4ef1a5380 2024-03-28T22:44:56.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.972+0000 7f2cdd3ae740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd41b6f40 con 0x55e4ef1ae880 2024-03-28T22:44:56.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.972+0000 7f2cdd3ae740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd41b5ad0 con 0x55e4ef1a5ca0 2024-03-28T22:44:56.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.972+0000 7f2cd3e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4ef1ae880 0x55e4ef1c6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.972+0000 7f2cd3e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4ef1ae880 0x55e4ef1c6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47152/0 (socket says 172.21.15.43:47152) 2024-03-28T22:44:56.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.972+0000 7f2cd3e00640 1 -- 172.21.15.43:0/795401888 learned_addr learned my addr 172.21.15.43:0/795401888 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:56.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.972+0000 7f2cd2a00640 1 --2- 172.21.15.43:0/795401888 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4ef1a5ca0 0x55e4ef1ac180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.973+0000 7f2cd3400640 1 --2- 172.21.15.43:0/795401888 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4ef1a5380 0x55e4ef1ba410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.973+0000 7f2cd3e00640 1 -- 172.21.15.43:0/795401888 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4ef1a5ca0 msgr2=0x55e4ef1ac180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:56.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.973+0000 7f2cd3e00640 1 --2- 172.21.15.43:0/795401888 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4ef1a5ca0 0x55e4ef1ac180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.973+0000 7f2cd3e00640 1 -- 172.21.15.43:0/795401888 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4ef1a5380 msgr2=0x55e4ef1ba410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:56.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.973+0000 7f2cd3e00640 1 --2- 172.21.15.43:0/795401888 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4ef1a5380 0x55e4ef1ba410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.973+0000 7f2cd3e00640 1 -- 172.21.15.43:0/795401888 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4ef1a5ad0 con 0x55e4ef1ae880 2024-03-28T22:44:56.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.973+0000 7f2cd2a00640 1 --2- 172.21.15.43:0/795401888 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4ef1a5ca0 0x55e4ef1ac180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:44:56.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.973+0000 7f2cd3400640 1 --2- 172.21.15.43:0/795401888 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4ef1a5380 0x55e4ef1ba410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:44:56.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.973+0000 7f2cd3e00640 1 --2- 172.21.15.43:0/795401888 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4ef1ae880 0x55e4ef1c6930 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc097c60 tx=0x7f2ccc09ae90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=356e732af6dd0589 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:56.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.973+0000 7f2cbfe00640 1 -- 172.21.15.43:0/795401888 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc09b290 con 0x55e4ef1ae880 2024-03-28T22:44:56.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.974+0000 7f2cbfe00640 1 -- 172.21.15.43:0/795401888 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ccc09f3e0 con 0x55e4ef1ae880 2024-03-28T22:44:56.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.974+0000 7f2cbfe00640 1 -- 172.21.15.43:0/795401888 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc005310 con 0x55e4ef1ae880 2024-03-28T22:44:56.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.974+0000 7f2cdd3ae740 1 -- 172.21.15.43:0/795401888 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4ef1ae880 msgr2=0x55e4ef1c6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:56.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.974+0000 7f2cdd3ae740 1 --2- 172.21.15.43:0/795401888 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4ef1ae880 0x55e4ef1c6930 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc097c60 tx=0x7f2ccc09ae90 comp rx=0 tx=0).stop 2024-03-28T22:44:56.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.974+0000 7f2cdd3ae740 1 -- 172.21.15.43:0/795401888 shutdown_connections 2024-03-28T22:44:56.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.974+0000 7f2cdd3ae740 1 --2- 172.21.15.43:0/795401888 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4ef1ae880 0x55e4ef1c6930 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.974+0000 7f2cdd3ae740 1 --2- 172.21.15.43:0/795401888 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4ef1a5ca0 0x55e4ef1ac180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.974+0000 7f2cdd3ae740 1 --2- 172.21.15.43:0/795401888 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4ef1a5380 0x55e4ef1ba410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.974+0000 7f2cdd3ae740 1 -- 172.21.15.43:0/795401888 >> 172.21.15.43:0/795401888 conn(0x55e4ef19fab0 msgr2=0x55e4ef19eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:56.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.974+0000 7f2cdd3ae740 1 -- 172.21.15.43:0/795401888 shutdown_connections 2024-03-28T22:44:56.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.975+0000 7f2cdd3ae740 1 -- 172.21.15.43:0/795401888 wait complete. 2024-03-28T22:44:56.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.975+0000 7f2cdd3ae740 1 -- start start 2024-03-28T22:44:56.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.975+0000 7f2cdd3ae740 1 Processor -- start 2024-03-28T22:44:56.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.976+0000 7f2cdd3ae740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4ef1a5380 0x55e4ef1b6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.976+0000 7f2cdd3ae740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4ef1a5ca0 0x55e4ef1a8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.976+0000 7f2cdd3ae740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4ef1ae880 0x55e4ef1a86b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.976+0000 7f2cdd3ae740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd41b6870 con 0x55e4ef1a5380 2024-03-28T22:44:56.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.976+0000 7f2cdd3ae740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd41b61a0 con 0x55e4ef1ae880 2024-03-28T22:44:56.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.976+0000 7f2cdd3ae740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd41b3400 con 0x55e4ef1a5ca0 2024-03-28T22:44:56.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.976+0000 7f2cdd3ae740 10 client.0 _ll_register_callbacks cb 0x7f2cc400bb20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:44:56.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.976+0000 7f2cd2a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4ef1a5ca0 0x55e4ef1a8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.976+0000 7f2cd3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4ef1a5380 0x55e4ef1b6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.976+0000 7f2cd2a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4ef1a5ca0 0x55e4ef1a8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40018/0 (socket says 172.21.15.43:40018) 2024-03-28T22:44:56.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.976+0000 7f2cd2a00640 1 -- 172.21.15.43:0/2889583326 learned_addr learned my addr 172.21.15.43:0/2889583326 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:56.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.976+0000 7f2cd3e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4ef1ae880 0x55e4ef1a86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.976+0000 7f2cd2a00640 1 -- 172.21.15.43:0/2889583326 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4ef1ae880 msgr2=0x55e4ef1a86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:56.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.976+0000 7f2cd2a00640 1 --2- 172.21.15.43:0/2889583326 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4ef1ae880 0x55e4ef1a86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.976+0000 7f2cd2a00640 1 -- 172.21.15.43:0/2889583326 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4ef1a5380 msgr2=0x55e4ef1b6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:56.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.976+0000 7f2cd2a00640 1 --2- 172.21.15.43:0/2889583326 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4ef1a5380 0x55e4ef1b6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.976+0000 7f2cd2a00640 1 -- 172.21.15.43:0/2889583326 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4ef1a8ca0 con 0x55e4ef1a5ca0 2024-03-28T22:44:56.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.976+0000 7f2cd3e00640 1 --2- 172.21.15.43:0/2889583326 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4ef1ae880 0x55e4ef1a86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:44:56.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.977+0000 7f2cd2a00640 1 --2- 172.21.15.43:0/2889583326 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4ef1a5ca0 0x55e4ef1a8170 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f2cc002e010 tx=0x7f2cc0000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6651b32162495bbc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:56.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.977+0000 7f2cbd600640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:44:56.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.977+0000 7f2cbd600640 1 -- 172.21.15.43:0/2889583326 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc0054440 con 0x55e4ef1a5ca0 2024-03-28T22:44:56.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.977+0000 7f2cbd600640 1 -- 172.21.15.43:0/2889583326 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f2cc00545e0 con 0x55e4ef1a5ca0 2024-03-28T22:44:56.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.977+0000 7f2cdd3ae740 1 -- 172.21.15.43:0/2889583326 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ccc09f3e0 con 0x55e4ef1a5ca0 2024-03-28T22:44:56.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.977+0000 7f2cbd600640 1 -- 172.21.15.43:0/2889583326 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2cc0063050 con 0x55e4ef1a5ca0 2024-03-28T22:44:56.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.977+0000 7f2cdd3ae740 10 client.28309 fetch_fsmap learned FSMap version 32 2024-03-28T22:44:56.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.977+0000 7f2cdd3ae740 1 -- 172.21.15.43:0/2889583326 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e4ef224d50 con 0x55e4ef1a5ca0 2024-03-28T22:44:56.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.978+0000 7f2cbd600640 1 -- 172.21.15.43:0/2889583326 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ccc09f3e0 con 0x55e4ef1a5ca0 2024-03-28T22:44:56.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.978+0000 7f2cbd600640 1 -- 172.21.15.43:0/2889583326 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f2cc0004bf0 con 0x55e4ef1a5ca0 2024-03-28T22:44:56.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.978+0000 7f2cdd3ae740 10 client.28309 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:44:56.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.978+0000 7f2cdd3ae740 10 client.28309 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:44:56.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.978+0000 7f2cdd3ae740 20 client.28309 populate_metadata read hostname 'smithi043' 2024-03-28T22:44:56.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.978+0000 7f2cdd3ae740 1 --2- 172.21.15.43:0/2889583326 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55e4ef225480 0x55e4ef2278a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:56.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.978+0000 7f2cdd3ae740 4 client.28309 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:44:56.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.978+0000 7f2cdd3ae740 1 -- 172.21.15.43:0/2889583326 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e4ef116590 con 0x55e4ef225480 2024-03-28T22:44:56.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.978+0000 7f2cd3400640 1 --2- 172.21.15.43:0/2889583326 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55e4ef225480 0x55e4ef2278a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:56.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.979+0000 7f2cd3400640 1 --2- 172.21.15.43:0/2889583326 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55e4ef225480 0x55e4ef2278a0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=892ddf7f20a2eff7 server_cookie=a01d4f6e4820f139 in_seq=0 out_seq=0 2024-03-28T22:44:56.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.979+0000 7f2cbd600640 10 client.28309 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:44:56.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.979+0000 7f2cbd600640 1 -- 172.21.15.43:0/2889583326 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e4ef116590 con 0x55e4ef225480 2024-03-28T22:44:56.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.979+0000 7f2cbd600640 10 client.28309 handle_command_reply: tid=0 2024-03-28T22:44:56.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.979+0000 7f2cdd3ae740 1 client.28309 shutdown 2024-03-28T22:44:56.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.979+0000 7f2cdd3ae740 1 -- 172.21.15.43:0/2889583326 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4ef1a5ca0 msgr2=0x55e4ef1a8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:56.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.979+0000 7f2cdd3ae740 1 --2- 172.21.15.43:0/2889583326 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4ef1a5ca0 0x55e4ef1a8170 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f2cc002e010 tx=0x7f2cc0000bc0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.979+0000 7f2cdd3ae740 1 -- 172.21.15.43:0/2889583326 shutdown_connections 2024-03-28T22:44:56.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.979+0000 7f2cdd3ae740 1 --2- 172.21.15.43:0/2889583326 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55e4ef225480 0x55e4ef2278a0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.979+0000 7f2cdd3ae740 1 --2- 172.21.15.43:0/2889583326 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4ef1ae880 0x55e4ef1a86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.979+0000 7f2cdd3ae740 1 --2- 172.21.15.43:0/2889583326 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4ef1a5ca0 0x55e4ef1a8170 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.979+0000 7f2cdd3ae740 1 --2- 172.21.15.43:0/2889583326 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4ef1a5380 0x55e4ef1b6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.979+0000 7f2cdd3ae740 1 -- 172.21.15.43:0/2889583326 >> 172.21.15.43:0/2889583326 conn(0x55e4ef19fab0 msgr2=0x55e4ef1a0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:56.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.980+0000 7f2cbd600640 0 client.28309 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:44:56.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.980+0000 7f2cdd3ae740 1 -- 172.21.15.43:0/2889583326 shutdown_connections 2024-03-28T22:44:56.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.980+0000 7f2cdd3ae740 1 -- 172.21.15.43:0/2889583326 wait complete. 2024-03-28T22:44:56.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.980+0000 7f2cdd3ae740 20 client.28309 trim_cache size 0 max 16384 2024-03-28T22:44:56.992 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:44:56.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.980+0000 7f2cda600640 1 -- 172.21.15.43:0/2357055626 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f2cb80a1ef0 msgr2=0x7f2cb80a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:56.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.980+0000 7f2cda600640 1 --2- 172.21.15.43:0/2357055626 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f2cb80a1ef0 0x7f2cb80a43b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0036040 tx=0x7f2cc002e040 comp rx=0 tx=0).stop 2024-03-28T22:44:56.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.980+0000 7f2cda600640 1 -- 172.21.15.43:0/2357055626 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2cd41a2f10 msgr2=0x7f2cd41a33c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:56.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.980+0000 7f2cda600640 1 --2- 172.21.15.43:0/2357055626 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2cd41a2f10 0x7f2cd41a33c0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc00eb10 tx=0x7f2ccc00c2d0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.981+0000 7f2cda600640 1 -- 172.21.15.43:0/2357055626 shutdown_connections 2024-03-28T22:44:56.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.981+0000 7f2cda600640 1 --2- 172.21.15.43:0/2357055626 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f2cb80a1ef0 0x7f2cb80a43b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.981+0000 7f2cda600640 1 --2- 172.21.15.43:0/2357055626 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2cd41a2f10 0x7f2cd41a33c0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.981+0000 7f2cda600640 1 --2- 172.21.15.43:0/2357055626 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2cd410d030 0x7f2cd419eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.981+0000 7f2cda600640 1 --2- 172.21.15.43:0/2357055626 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2cd4107f60 0x7f2cd419e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:56.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.981+0000 7f2cda600640 1 -- 172.21.15.43:0/2357055626 >> 172.21.15.43:0/2357055626 conn(0x7f2cd4075c90 msgr2=0x7f2cd4077fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:56.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.981+0000 7f2cda600640 1 -- 172.21.15.43:0/2357055626 shutdown_connections 2024-03-28T22:44:56.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:56.981+0000 7f2cda600640 1 -- 172.21.15.43:0/2357055626 wait complete. 2024-03-28T22:44:56.995 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:56 smithi043 ceph-mon[31823]: pgmap v240: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 26 KiB/s wr, 9 op/s 2024-03-28T22:44:56.996 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:56 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2336824896' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:44:56.998 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T22:44:57.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:56 smithi172 ceph-mon[36566]: pgmap v240: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 26 KiB/s wr, 9 op/s 2024-03-28T22:44:57.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:56 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2336824896' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:44:57.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:56 smithi082 ceph-mon[36471]: pgmap v240: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 26 KiB/s wr, 9 op/s 2024-03-28T22:44:57.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:56 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2336824896' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:44:57.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.223+0000 7f8658600640 1 -- 172.21.15.43:0/110764346 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86501076f0 msgr2=0x7f8650107ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:57.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.223+0000 7f8658600640 1 --2- 172.21.15.43:0/110764346 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86501076f0 0x7f8650107ad0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8644009930 tx=0x7f864402bc30 comp rx=0 tx=0).stop 2024-03-28T22:44:57.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.223+0000 7f8658600640 1 -- 172.21.15.43:0/110764346 shutdown_connections 2024-03-28T22:44:57.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.223+0000 7f8658600640 1 --2- 172.21.15.43:0/110764346 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86501020a0 0x7f865010e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.223+0000 7f8658600640 1 --2- 172.21.15.43:0/110764346 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86501016e0 0x7f8650101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.223+0000 7f8658600640 1 --2- 172.21.15.43:0/110764346 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86501076f0 0x7f8650107ad0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.223+0000 7f8658600640 1 -- 172.21.15.43:0/110764346 >> 172.21.15.43:0/110764346 conn(0x7f8650078b50 msgr2=0x7f8650078f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:57.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.223+0000 7f8658600640 1 -- 172.21.15.43:0/110764346 shutdown_connections 2024-03-28T22:44:57.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.223+0000 7f8658600640 1 -- 172.21.15.43:0/110764346 wait complete. 2024-03-28T22:44:57.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.224+0000 7f8658600640 1 Processor -- start 2024-03-28T22:44:57.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.224+0000 7f8658600640 1 -- start start 2024-03-28T22:44:57.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.225+0000 7f8658600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86501016e0 0x7f8650073a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.225+0000 7f8658600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86501020a0 0x7f8650073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.225+0000 7f8658600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86501076f0 0x7f86500744d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.225+0000 7f8658600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8650111b10 con 0x7f86501016e0 2024-03-28T22:44:57.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.225+0000 7f8658600640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8650111990 con 0x7f86501020a0 2024-03-28T22:44:57.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.225+0000 7f8658600640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f8650111c90 con 0x7f86501076f0 2024-03-28T22:44:57.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.225+0000 7f8655c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86501016e0 0x7f8650073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.225+0000 7f8655200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86501020a0 0x7f8650073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.225+0000 7f8655200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86501020a0 0x7f8650073f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47164/0 (socket says 172.21.15.43:47164) 2024-03-28T22:44:57.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.225+0000 7f8655c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86501016e0 0x7f8650073a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45090/0 (socket says 172.21.15.43:45090) 2024-03-28T22:44:57.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.225+0000 7f8656600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86501076f0 0x7f86500744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.225+0000 7f8655200640 1 -- 172.21.15.43:0/1698657247 learned_addr learned my addr 172.21.15.43:0/1698657247 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:57.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.225+0000 7f8655200640 1 -- 172.21.15.43:0/1698657247 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86501076f0 msgr2=0x7f86500744d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:57.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.225+0000 7f8655200640 1 --2- 172.21.15.43:0/1698657247 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86501076f0 0x7f86500744d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.225+0000 7f8655200640 1 -- 172.21.15.43:0/1698657247 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86501016e0 msgr2=0x7f8650073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:57.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.225+0000 7f8655200640 1 --2- 172.21.15.43:0/1698657247 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86501016e0 0x7f8650073a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.225+0000 7f8655200640 1 -- 172.21.15.43:0/1698657247 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86501a64b0 con 0x7f86501020a0 2024-03-28T22:44:57.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.226+0000 7f8655200640 1 --2- 172.21.15.43:0/1698657247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86501020a0 0x7f8650073f90 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f863c00e9f0 tx=0x7f863c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:57.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.226+0000 7f864e000640 1 -- 172.21.15.43:0/1698657247 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863c00cd80 con 0x7f86501020a0 2024-03-28T22:44:57.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.226+0000 7f864e000640 1 -- 172.21.15.43:0/1698657247 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f863c010910 con 0x7f86501020a0 2024-03-28T22:44:57.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.226+0000 7f864e000640 1 -- 172.21.15.43:0/1698657247 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863c018990 con 0x7f86501020a0 2024-03-28T22:44:57.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.226+0000 7f8658600640 1 -- 172.21.15.43:0/1698657247 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86501a67a0 con 0x7f86501020a0 2024-03-28T22:44:57.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.226+0000 7f8658600640 1 -- 172.21.15.43:0/1698657247 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86501a6d40 con 0x7f86501020a0 2024-03-28T22:44:57.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.229+0000 7f865b1e6740 1 Processor -- start 2024-03-28T22:44:57.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.229+0000 7f864e000640 1 -- 172.21.15.43:0/1698657247 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f863c010ac0 con 0x7f86501020a0 2024-03-28T22:44:57.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.229+0000 7f865b1e6740 1 -- start start 2024-03-28T22:44:57.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.230+0000 7f865b1e6740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a870f7380 0x564a870f7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.230+0000 7f865b1e6740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a870f7ca0 0x564a87100340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.230+0000 7f865b1e6740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a87100880 0x564a87102c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.230+0000 7f865b1e6740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x564a8705b050 con 0x564a870f7380 2024-03-28T22:44:57.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.230+0000 7f865b1e6740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x564a87003ce0 con 0x564a87100880 2024-03-28T22:44:57.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.230+0000 7f865b1e6740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x564a870570f0 con 0x564a870f7ca0 2024-03-28T22:44:57.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.230+0000 7f8655c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a870f7ca0 0x564a87100340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.230+0000 7f8655c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a870f7ca0 0x564a87100340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40042/0 (socket says 172.21.15.43:40042) 2024-03-28T22:44:57.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.230+0000 7f8655c00640 1 -- 172.21.15.43:0/3600655731 learned_addr learned my addr 172.21.15.43:0/3600655731 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:57.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.230+0000 7f8656600640 1 --2- 172.21.15.43:0/3600655731 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a870f7380 0x564a870f7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.230+0000 7f8655200640 1 --2- 172.21.15.43:0/3600655731 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a87100880 0x564a87102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.231+0000 7f8655c00640 1 -- 172.21.15.43:0/3600655731 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a87100880 msgr2=0x564a87102c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:57.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.231+0000 7f8655c00640 1 --2- 172.21.15.43:0/3600655731 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a87100880 0x564a87102c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.231+0000 7f8655c00640 1 -- 172.21.15.43:0/3600655731 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a870f7380 msgr2=0x564a870f7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:57.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.231+0000 7f8655c00640 1 --2- 172.21.15.43:0/3600655731 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a870f7380 0x564a870f7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.231+0000 7f8655c00640 1 -- 172.21.15.43:0/3600655731 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a86d6f3f0 con 0x564a870f7ca0 2024-03-28T22:44:57.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.231+0000 7f8656600640 1 --2- 172.21.15.43:0/3600655731 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a870f7380 0x564a870f7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:44:57.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.231+0000 7f8655c00640 1 --2- 172.21.15.43:0/3600655731 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a870f7ca0 0x564a87100340 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f8644000c00 tx=0x7f8644009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbcb140ef3faeb47 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:57.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.231+0000 7f864e000640 1 --2- 172.21.15.43:0/1698657247 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f86380a1fa0 0x7f86380a4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.231+0000 7f864e000640 1 -- 172.21.15.43:0/1698657247 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f863c0ce830 con 0x7f86501020a0 2024-03-28T22:44:57.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.231+0000 7f8642a00640 1 -- 172.21.15.43:0/3600655731 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8644045070 con 0x564a870f7ca0 2024-03-28T22:44:57.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.231+0000 7f8656600640 1 --2- 172.21.15.43:0/1698657247 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f86380a1fa0 0x7f86380a4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.231+0000 7f8642a00640 1 -- 172.21.15.43:0/3600655731 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f864402c650 con 0x564a870f7ca0 2024-03-28T22:44:57.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.231+0000 7f8642a00640 1 -- 172.21.15.43:0/3600655731 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8644004b50 con 0x564a870f7ca0 2024-03-28T22:44:57.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.232+0000 7f865b1e6740 1 -- 172.21.15.43:0/3600655731 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a870f7ca0 msgr2=0x564a87100340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:57.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.232+0000 7f865b1e6740 1 --2- 172.21.15.43:0/3600655731 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a870f7ca0 0x564a87100340 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f8644000c00 tx=0x7f8644009560 comp rx=0 tx=0).stop 2024-03-28T22:44:57.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.232+0000 7f8656600640 1 --2- 172.21.15.43:0/1698657247 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f86380a1fa0 0x7f86380a4460 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8648005fd0 tx=0x7f8648009500 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:57.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.232+0000 7f865b1e6740 1 -- 172.21.15.43:0/3600655731 shutdown_connections 2024-03-28T22:44:57.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.232+0000 7f865b1e6740 1 --2- 172.21.15.43:0/3600655731 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a87100880 0x564a87102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.232+0000 7f865b1e6740 1 --2- 172.21.15.43:0/3600655731 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a870f7ca0 0x564a87100340 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.232+0000 7f865b1e6740 1 --2- 172.21.15.43:0/3600655731 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a870f7380 0x564a870f7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.232+0000 7f865b1e6740 1 -- 172.21.15.43:0/3600655731 >> 172.21.15.43:0/3600655731 conn(0x564a870f1ab0 msgr2=0x564a870f1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:57.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.233+0000 7f865b1e6740 1 -- 172.21.15.43:0/3600655731 shutdown_connections 2024-03-28T22:44:57.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.233+0000 7f865b1e6740 1 -- 172.21.15.43:0/3600655731 wait complete. 2024-03-28T22:44:57.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.233+0000 7f865b1e6740 1 -- start start 2024-03-28T22:44:57.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.233+0000 7f865b1e6740 1 Processor -- start 2024-03-28T22:44:57.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.234+0000 7f865b1e6740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a870f7380 0x564a870fd920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.234+0000 7f865b1e6740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a870f7ca0 0x564a870fde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.234+0000 7f865b1e6740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a87100880 0x564a870fe3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.234+0000 7f865b1e6740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x564a87103610 con 0x564a870f7380 2024-03-28T22:44:57.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.234+0000 7f865b1e6740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x564a86ff92f0 con 0x564a870f7ca0 2024-03-28T22:44:57.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.234+0000 7f865b1e6740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x564a87103790 con 0x564a87100880 2024-03-28T22:44:57.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.234+0000 7f865b1e6740 10 client.0 _ll_register_callbacks cb 0x564a86d6ebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:44:57.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.234+0000 7f8655200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a870f7ca0 0x564a870fde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.234+0000 7f8656600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a87100880 0x564a870fe3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.234+0000 7f8655c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a870f7380 0x564a870fd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.234+0000 7f8655200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a870f7ca0 0x564a870fde60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47176/0 (socket says 172.21.15.43:47176) 2024-03-28T22:44:57.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.234+0000 7f8655200640 1 -- 172.21.15.43:0/1531642468 learned_addr learned my addr 172.21.15.43:0/1531642468 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:57.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.234+0000 7f8656600640 1 -- 172.21.15.43:0/1531642468 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a870f7ca0 msgr2=0x564a870fde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:57.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.234+0000 7f8656600640 1 --2- 172.21.15.43:0/1531642468 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a870f7ca0 0x564a870fde60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.235+0000 7f8656600640 1 -- 172.21.15.43:0/1531642468 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a870f7380 msgr2=0x564a870fd920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:57.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.235+0000 7f8656600640 1 --2- 172.21.15.43:0/1531642468 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a870f7380 0x564a870fd920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.235+0000 7f8656600640 1 -- 172.21.15.43:0/1531642468 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a86e397d0 con 0x564a87100880 2024-03-28T22:44:57.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.235+0000 7f8655200640 1 --2- 172.21.15.43:0/1531642468 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a870f7ca0 0x564a870fde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:44:57.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.235+0000 7f8656600640 1 --2- 172.21.15.43:0/1531642468 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a87100880 0x564a870fe3a0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f8648003f00 tx=0x7f8648004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99ab0e2573b1b082 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:57.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.235+0000 7f8637400640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:44:57.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.235+0000 7f8637400640 1 -- 172.21.15.43:0/1531642468 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8648004b20 con 0x564a87100880 2024-03-28T22:44:57.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.235+0000 7f865b1e6740 1 -- 172.21.15.43:0/1531642468 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f864402c650 con 0x564a87100880 2024-03-28T22:44:57.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.235+0000 7f8637400640 1 -- 172.21.15.43:0/1531642468 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f8648004cc0 con 0x564a87100880 2024-03-28T22:44:57.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.235+0000 7f8637400640 1 -- 172.21.15.43:0/1531642468 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f864800f810 con 0x564a87100880 2024-03-28T22:44:57.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.235+0000 7f865b1e6740 10 client.28321 fetch_fsmap learned FSMap version 32 2024-03-28T22:44:57.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.235+0000 7f865b1e6740 1 -- 172.21.15.43:0/1531642468 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564a8710f9a0 con 0x564a87100880 2024-03-28T22:44:57.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.236+0000 7f8637400640 1 -- 172.21.15.43:0/1531642468 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f864402c650 con 0x564a87100880 2024-03-28T22:44:57.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.236+0000 7f8637400640 1 -- 172.21.15.43:0/1531642468 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f8648027070 con 0x564a87100880 2024-03-28T22:44:57.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.236+0000 7f865b1e6740 10 client.28321 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:44:57.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.236+0000 7f865b1e6740 10 client.28321 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:44:57.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.236+0000 7f865b1e6740 20 client.28321 populate_metadata read hostname 'smithi043' 2024-03-28T22:44:57.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.236+0000 7f865b1e6740 1 --2- 172.21.15.43:0/1531642468 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x564a87110eb0 0x564a871132c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.236+0000 7f865b1e6740 4 client.28321 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:44:57.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.236+0000 7f865b1e6740 1 -- 172.21.15.43:0/1531642468 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a87004680 con 0x564a87110eb0 2024-03-28T22:44:57.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.236+0000 7f8655c00640 1 --2- 172.21.15.43:0/1531642468 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x564a87110eb0 0x564a871132c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.237+0000 7f8655c00640 1 --2- 172.21.15.43:0/1531642468 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x564a87110eb0 0x564a871132c0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=be58e931850a8d3f server_cookie=407dbb7f7b4b7875 in_seq=0 out_seq=0 2024-03-28T22:44:57.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.237+0000 7f8637400640 10 client.28321 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:44:57.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.238+0000 7f8637400640 1 -- 172.21.15.43:0/1531642468 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564a87004680 con 0x564a87110eb0 2024-03-28T22:44:57.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.238+0000 7f8637400640 10 client.28321 handle_command_reply: tid=0 2024-03-28T22:44:57.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.238+0000 7f865b1e6740 1 client.28321 shutdown 2024-03-28T22:44:57.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.238+0000 7f865b1e6740 1 -- 172.21.15.43:0/1531642468 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a87100880 msgr2=0x564a870fe3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:57.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.238+0000 7f865b1e6740 1 --2- 172.21.15.43:0/1531642468 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a87100880 0x564a870fe3a0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f8648003f00 tx=0x7f8648004a40 comp rx=0 tx=0).stop 2024-03-28T22:44:57.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.238+0000 7f865b1e6740 1 -- 172.21.15.43:0/1531642468 shutdown_connections 2024-03-28T22:44:57.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.238+0000 7f865b1e6740 1 --2- 172.21.15.43:0/1531642468 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x564a87110eb0 0x564a871132c0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.238+0000 7f865b1e6740 1 --2- 172.21.15.43:0/1531642468 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a870f7ca0 0x564a870fde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.238+0000 7f865b1e6740 1 --2- 172.21.15.43:0/1531642468 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a87100880 0x564a870fe3a0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.238+0000 7f865b1e6740 1 --2- 172.21.15.43:0/1531642468 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a870f7380 0x564a870fd920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.238+0000 7f865b1e6740 1 -- 172.21.15.43:0/1531642468 >> 172.21.15.43:0/1531642468 conn(0x564a870f1ab0 msgr2=0x564a870f3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:57.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.238+0000 7f8637400640 0 client.28321 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:44:57.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.239+0000 7f865b1e6740 1 -- 172.21.15.43:0/1531642468 shutdown_connections 2024-03-28T22:44:57.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.239+0000 7f865b1e6740 1 -- 172.21.15.43:0/1531642468 wait complete. 2024-03-28T22:44:57.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.239+0000 7f865b1e6740 20 client.28321 trim_cache size 0 max 16384 2024-03-28T22:44:57.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.253+0000 7f865b1e6740 1 Processor -- start 2024-03-28T22:44:57.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.253+0000 7f865b1e6740 1 -- start start 2024-03-28T22:44:57.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.254+0000 7f865b1e6740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a870f7380 0x564a87100050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.254+0000 7f865b1e6740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a870f7ca0 0x564a8710c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.254+0000 7f865b1e6740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a87100880 0x564a870f09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.254+0000 7f865b1e6740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f86501b0e50 con 0x564a870f7ca0 2024-03-28T22:44:57.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.254+0000 7f865b1e6740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f86501b2990 con 0x564a87100880 2024-03-28T22:44:57.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.254+0000 7f865b1e6740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f86501b1520 con 0x564a870f7380 2024-03-28T22:44:57.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.254+0000 7f8655c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a870f7380 0x564a87100050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.254+0000 7f8655c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a870f7380 0x564a87100050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40054/0 (socket says 172.21.15.43:40054) 2024-03-28T22:44:57.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.254+0000 7f8655c00640 1 -- 172.21.15.43:0/497139493 learned_addr learned my addr 172.21.15.43:0/497139493 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:57.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.254+0000 7f8656600640 1 --2- 172.21.15.43:0/497139493 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a87100880 0x564a870f09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.254+0000 7f8655200640 1 --2- 172.21.15.43:0/497139493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a870f7ca0 0x564a8710c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.254+0000 7f8655c00640 1 -- 172.21.15.43:0/497139493 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a87100880 msgr2=0x564a870f09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:57.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.254+0000 7f8655c00640 1 --2- 172.21.15.43:0/497139493 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a87100880 0x564a870f09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.254+0000 7f8655c00640 1 -- 172.21.15.43:0/497139493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a870f7ca0 msgr2=0x564a8710c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:57.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.254+0000 7f8655c00640 1 --2- 172.21.15.43:0/497139493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a870f7ca0 0x564a8710c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.254+0000 7f8655c00640 1 -- 172.21.15.43:0/497139493 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a870f15e0 con 0x564a870f7380 2024-03-28T22:44:57.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.254+0000 7f8655200640 1 --2- 172.21.15.43:0/497139493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a870f7ca0 0x564a8710c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:44:57.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.255+0000 7f8655c00640 1 --2- 172.21.15.43:0/497139493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a870f7380 0x564a87100050 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f864402c820 tx=0x7f864402bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adf690d8756767b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:57.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.255+0000 7f8642000640 1 -- 172.21.15.43:0/497139493 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8644045040 con 0x564a870f7380 2024-03-28T22:44:57.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.255+0000 7f8642000640 1 -- 172.21.15.43:0/497139493 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8644004ad0 con 0x564a870f7380 2024-03-28T22:44:57.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.255+0000 7f8642000640 1 -- 172.21.15.43:0/497139493 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86440054a0 con 0x564a870f7380 2024-03-28T22:44:57.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.255+0000 7f865b1e6740 1 -- 172.21.15.43:0/497139493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a870f7380 msgr2=0x564a87100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:57.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.255+0000 7f865b1e6740 1 --2- 172.21.15.43:0/497139493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a870f7380 0x564a87100050 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f864402c820 tx=0x7f864402bc30 comp rx=0 tx=0).stop 2024-03-28T22:44:57.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.256+0000 7f865b1e6740 1 -- 172.21.15.43:0/497139493 shutdown_connections 2024-03-28T22:44:57.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.256+0000 7f865b1e6740 1 --2- 172.21.15.43:0/497139493 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a87100880 0x564a870f09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.256+0000 7f865b1e6740 1 --2- 172.21.15.43:0/497139493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a870f7ca0 0x564a8710c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.256+0000 7f865b1e6740 1 --2- 172.21.15.43:0/497139493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a870f7380 0x564a87100050 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.256+0000 7f865b1e6740 1 -- 172.21.15.43:0/497139493 >> 172.21.15.43:0/497139493 conn(0x564a870f1ab0 msgr2=0x564a870f02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:57.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.256+0000 7f865b1e6740 1 -- 172.21.15.43:0/497139493 shutdown_connections 2024-03-28T22:44:57.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.256+0000 7f865b1e6740 1 -- 172.21.15.43:0/497139493 wait complete. 2024-03-28T22:44:57.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.256+0000 7f865b1e6740 1 -- start start 2024-03-28T22:44:57.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.256+0000 7f865b1e6740 1 Processor -- start 2024-03-28T22:44:57.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.257+0000 7f865b1e6740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a870f7380 0x564a871042b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.257+0000 7f865b1e6740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a870f7ca0 0x564a871047f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.257+0000 7f865b1e6740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a87100880 0x564a87104d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.257+0000 7f865b1e6740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f86501b22c0 con 0x564a870f7ca0 2024-03-28T22:44:57.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.257+0000 7f865b1e6740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f86501b1bf0 con 0x564a87100880 2024-03-28T22:44:57.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.257+0000 7f865b1e6740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f86501aee50 con 0x564a870f7380 2024-03-28T22:44:57.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.257+0000 7f865b1e6740 10 client.0 _ll_register_callbacks cb 0x7f8648009d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:44:57.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.257+0000 7f8655c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a870f7380 0x564a871042b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.257+0000 7f8655200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a870f7ca0 0x564a871047f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.257+0000 7f8656600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a87100880 0x564a87104d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.257+0000 7f8655c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a870f7380 0x564a871042b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40062/0 (socket says 172.21.15.43:40062) 2024-03-28T22:44:57.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.257+0000 7f8655c00640 1 -- 172.21.15.43:0/897704290 learned_addr learned my addr 172.21.15.43:0/897704290 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:57.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.257+0000 7f8655200640 1 -- 172.21.15.43:0/897704290 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a870f7380 msgr2=0x564a871042b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:57.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.257+0000 7f8655200640 1 --2- 172.21.15.43:0/897704290 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a870f7380 0x564a871042b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.257+0000 7f8655200640 1 -- 172.21.15.43:0/897704290 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a87100880 msgr2=0x564a87104d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:57.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.257+0000 7f8655200640 1 --2- 172.21.15.43:0/897704290 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a87100880 0x564a87104d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.257+0000 7f8655200640 1 -- 172.21.15.43:0/897704290 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a871761c0 con 0x564a870f7ca0 2024-03-28T22:44:57.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.257+0000 7f8656600640 1 --2- 172.21.15.43:0/897704290 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a87100880 0x564a87104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:44:57.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.257+0000 7f8655c00640 1 --2- 172.21.15.43:0/897704290 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a870f7380 0x564a871042b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:44:57.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.258+0000 7f8655200640 1 --2- 172.21.15.43:0/897704290 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a870f7ca0 0x564a871047f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f863c000c00 tx=0x7f863c009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c2d02322d24e5d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:57.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.258+0000 7f8637400640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:44:57.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.258+0000 7f8637400640 1 -- 172.21.15.43:0/897704290 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863c09e060 con 0x564a870f7ca0 2024-03-28T22:44:57.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.258+0000 7f8637400640 1 -- 172.21.15.43:0/897704290 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f863c018d30 con 0x564a870f7ca0 2024-03-28T22:44:57.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.258+0000 7f865b1e6740 1 -- 172.21.15.43:0/897704290 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8644004ad0 con 0x564a870f7ca0 2024-03-28T22:44:57.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.258+0000 7f8637400640 1 -- 172.21.15.43:0/897704290 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f863c09a800 con 0x564a870f7ca0 2024-03-28T22:44:57.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.258+0000 7f865b1e6740 10 client.18123 fetch_fsmap learned FSMap version 32 2024-03-28T22:44:57.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.258+0000 7f865b1e6740 1 -- 172.21.15.43:0/897704290 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564a871774f0 con 0x564a870f7ca0 2024-03-28T22:44:57.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.259+0000 7f8637400640 1 -- 172.21.15.43:0/897704290 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8644004ad0 con 0x564a870f7ca0 2024-03-28T22:44:57.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.259+0000 7f8637400640 1 -- 172.21.15.43:0/897704290 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f863c09a380 con 0x564a870f7ca0 2024-03-28T22:44:57.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.259+0000 7f865b1e6740 10 client.18123 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:44:57.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.259+0000 7f865b1e6740 10 client.18123 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:44:57.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.259+0000 7f865b1e6740 20 client.18123 populate_metadata read hostname 'smithi043' 2024-03-28T22:44:57.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.259+0000 7f865b1e6740 1 --2- 172.21.15.43:0/897704290 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x564a87177c70 0x564a8717a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.259+0000 7f865b1e6740 4 client.18123 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:44:57.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.259+0000 7f865b1e6740 1 -- 172.21.15.43:0/897704290 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564a87068590 con 0x564a87177c70 2024-03-28T22:44:57.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.259+0000 7f8655c00640 1 --2- 172.21.15.43:0/897704290 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x564a87177c70 0x564a8717a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.260+0000 7f8655c00640 1 --2- 172.21.15.43:0/897704290 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x564a87177c70 0x564a8717a090 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=93f31da82c783cc server_cookie=3e8073d0cc6630d3 in_seq=0 out_seq=0 2024-03-28T22:44:57.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.260+0000 7f8637400640 10 client.18123 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:44:57.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.260+0000 7f8637400640 1 -- 172.21.15.43:0/897704290 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564a87068590 con 0x564a87177c70 2024-03-28T22:44:57.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.260+0000 7f8637400640 10 client.18123 handle_command_reply: tid=0 2024-03-28T22:44:57.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.260+0000 7f865b1e6740 1 client.18123 shutdown 2024-03-28T22:44:57.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.260+0000 7f865b1e6740 1 -- 172.21.15.43:0/897704290 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a870f7ca0 msgr2=0x564a871047f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:57.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.260+0000 7f865b1e6740 1 --2- 172.21.15.43:0/897704290 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a870f7ca0 0x564a871047f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f863c000c00 tx=0x7f863c009310 comp rx=0 tx=0).stop 2024-03-28T22:44:57.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.260+0000 7f865b1e6740 1 -- 172.21.15.43:0/897704290 shutdown_connections 2024-03-28T22:44:57.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.260+0000 7f865b1e6740 1 --2- 172.21.15.43:0/897704290 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x564a87177c70 0x564a8717a090 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.260+0000 7f865b1e6740 1 --2- 172.21.15.43:0/897704290 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564a87100880 0x564a87104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.260+0000 7f865b1e6740 1 --2- 172.21.15.43:0/897704290 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564a870f7ca0 0x564a871047f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.260+0000 7f865b1e6740 1 --2- 172.21.15.43:0/897704290 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564a870f7380 0x564a871042b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.260+0000 7f8637400640 0 client.18123 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:44:57.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.260+0000 7f865b1e6740 1 -- 172.21.15.43:0/897704290 >> 172.21.15.43:0/897704290 conn(0x564a870f1ab0 msgr2=0x564a870f0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:57.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.260+0000 7f865b1e6740 1 -- 172.21.15.43:0/897704290 shutdown_connections 2024-03-28T22:44:57.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.260+0000 7f865b1e6740 1 -- 172.21.15.43:0/897704290 wait complete. 2024-03-28T22:44:57.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.260+0000 7f865b1e6740 20 client.18123 trim_cache size 0 max 16384 2024-03-28T22:44:57.274 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:44:57.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.261+0000 7f8658600640 1 -- 172.21.15.43:0/1698657247 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f86380a1fa0 msgr2=0x7f86380a4460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:57.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.261+0000 7f8658600640 1 --2- 172.21.15.43:0/1698657247 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f86380a1fa0 0x7f86380a4460 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8648005fd0 tx=0x7f8648009500 comp rx=0 tx=0).stop 2024-03-28T22:44:57.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.261+0000 7f8658600640 1 -- 172.21.15.43:0/1698657247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86501020a0 msgr2=0x7f8650073f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:57.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.261+0000 7f8658600640 1 --2- 172.21.15.43:0/1698657247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86501020a0 0x7f8650073f90 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f863c00e9f0 tx=0x7f863c00eeb0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.261+0000 7f8658600640 1 -- 172.21.15.43:0/1698657247 shutdown_connections 2024-03-28T22:44:57.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.261+0000 7f8658600640 1 --2- 172.21.15.43:0/1698657247 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f86380a1fa0 0x7f86380a4460 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.261+0000 7f8658600640 1 --2- 172.21.15.43:0/1698657247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86501020a0 0x7f8650073f90 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.261+0000 7f8658600640 1 --2- 172.21.15.43:0/1698657247 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86501076f0 0x7f86500744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.261+0000 7f8658600640 1 --2- 172.21.15.43:0/1698657247 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86501016e0 0x7f8650073a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.261+0000 7f8658600640 1 -- 172.21.15.43:0/1698657247 >> 172.21.15.43:0/1698657247 conn(0x7f8650078b50 msgr2=0x7f8650104110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:57.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.261+0000 7f8658600640 1 -- 172.21.15.43:0/1698657247 shutdown_connections 2024-03-28T22:44:57.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.261+0000 7f8658600640 1 -- 172.21.15.43:0/1698657247 wait complete. 2024-03-28T22:44:57.278 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:44:57.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.509+0000 7f47a6200640 1 -- 172.21.15.43:0/3052845768 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f47a0107f00 msgr2=0x7f47a01082e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:57.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.509+0000 7f47a6200640 1 --2- 172.21.15.43:0/3052845768 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f47a0107f00 0x7f47a01082e0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f4790009900 tx=0x7f479002bc60 comp rx=0 tx=0).stop 2024-03-28T22:44:57.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.509+0000 7f47a6200640 1 -- 172.21.15.43:0/3052845768 shutdown_connections 2024-03-28T22:44:57.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.509+0000 7f47a6200640 1 --2- 172.21.15.43:0/3052845768 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47a010cf50 0x7f47a0114b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.509+0000 7f47a6200640 1 --2- 172.21.15.43:0/3052845768 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f47a01088b0 0x7f47a010c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.509+0000 7f47a6200640 1 --2- 172.21.15.43:0/3052845768 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f47a0107f00 0x7f47a01082e0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.509+0000 7f47a6200640 1 -- 172.21.15.43:0/3052845768 >> 172.21.15.43:0/3052845768 conn(0x7f47a0077570 msgr2=0x7f47a0106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:57.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.509+0000 7f47a6200640 1 -- 172.21.15.43:0/3052845768 shutdown_connections 2024-03-28T22:44:57.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.509+0000 7f47a6200640 1 -- 172.21.15.43:0/3052845768 wait complete. 2024-03-28T22:44:57.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.511+0000 7f47a6200640 1 Processor -- start 2024-03-28T22:44:57.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.511+0000 7f47a6200640 1 -- start start 2024-03-28T22:44:57.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.511+0000 7f47a6200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f47a0107f00 0x7f47a019e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.511+0000 7f47a6200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47a010cf50 0x7f47a019ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.511+0000 7f47a6200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f47a01a2dc0 0x7f47a01a3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.511+0000 7f47a6200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f47a01180d0 con 0x7f47a0107f00 2024-03-28T22:44:57.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.511+0000 7f47a6200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f47a0117f50 con 0x7f47a01a2dc0 2024-03-28T22:44:57.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.511+0000 7f47a6200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f47a0118250 con 0x7f47a010cf50 2024-03-28T22:44:57.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.512+0000 7f47a4e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f47a0107f00 0x7f47a019e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.512+0000 7f47a5800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f47a01a2dc0 0x7f47a01a3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.512+0000 7f4797e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47a010cf50 0x7f47a019ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.512+0000 7f47a4e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f47a0107f00 0x7f47a019e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45144/0 (socket says 172.21.15.43:45144) 2024-03-28T22:44:57.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.512+0000 7f47a4e00640 1 -- 172.21.15.43:0/111021703 learned_addr learned my addr 172.21.15.43:0/111021703 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:57.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.512+0000 7f47a5800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f47a01a2dc0 0x7f47a01a3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47194/0 (socket says 172.21.15.43:47194) 2024-03-28T22:44:57.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.512+0000 7f47a5800640 1 -- 172.21.15.43:0/111021703 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47a010cf50 msgr2=0x7f47a019ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:57.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.512+0000 7f47a5800640 1 --2- 172.21.15.43:0/111021703 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47a010cf50 0x7f47a019ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.512+0000 7f47a5800640 1 -- 172.21.15.43:0/111021703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f47a0107f00 msgr2=0x7f47a019e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:57.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.512+0000 7f47a5800640 1 --2- 172.21.15.43:0/111021703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f47a0107f00 0x7f47a019e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.512+0000 7f47a5800640 1 -- 172.21.15.43:0/111021703 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47a01a38b0 con 0x7f47a01a2dc0 2024-03-28T22:44:57.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.512+0000 7f47a5800640 1 --2- 172.21.15.43:0/111021703 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f47a01a2dc0 0x7f47a01a3270 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f479800e960 tx=0x7f479800ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:57.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.513+0000 7f479e000640 1 -- 172.21.15.43:0/111021703 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4798004d20 con 0x7f47a01a2dc0 2024-03-28T22:44:57.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.513+0000 7f479e000640 1 -- 172.21.15.43:0/111021703 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4798005020 con 0x7f47a01a2dc0 2024-03-28T22:44:57.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.513+0000 7f47a6200640 1 -- 172.21.15.43:0/111021703 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47a01a3ba0 con 0x7f47a01a2dc0 2024-03-28T22:44:57.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.513+0000 7f479e000640 1 -- 172.21.15.43:0/111021703 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47980116b0 con 0x7f47a01a2dc0 2024-03-28T22:44:57.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.513+0000 7f47a6200640 1 -- 172.21.15.43:0/111021703 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47a01ab390 con 0x7f47a01a2dc0 2024-03-28T22:44:57.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.516+0000 7f479e000640 1 -- 172.21.15.43:0/111021703 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f47980054d0 con 0x7f47a01a2dc0 2024-03-28T22:44:57.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.516+0000 7f47aa5a5740 1 Processor -- start 2024-03-28T22:44:57.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.516+0000 7f47aa5a5740 1 -- start start 2024-03-28T22:44:57.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.517+0000 7f47aa5a5740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d634000550 0x55d634000930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.517+0000 7f47aa5a5740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d634000e70 0x55d634009510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.517+0000 7f47aa5a5740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d634009a50 0x55d63400be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.517+0000 7f47aa5a5740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d633f64050 con 0x55d634000e70 2024-03-28T22:44:57.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.517+0000 7f47aa5a5740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55d633f0cce0 con 0x55d634009a50 2024-03-28T22:44:57.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.517+0000 7f47aa5a5740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55d633f600f0 con 0x55d634000550 2024-03-28T22:44:57.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.517+0000 7f4797e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d634000550 0x55d634000930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.517+0000 7f47a4e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d634000e70 0x55d634009510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.517+0000 7f4797e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d634009a50 0x55d63400be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.517+0000 7f47a4e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d634000e70 0x55d634009510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45158/0 (socket says 172.21.15.43:45158) 2024-03-28T22:44:57.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.517+0000 7f47a4e00640 1 -- 172.21.15.43:0/2708817120 learned_addr learned my addr 172.21.15.43:0/2708817120 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:57.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.518+0000 7f4797e00640 1 -- 172.21.15.43:0/2708817120 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d634000550 msgr2=0x55d634000930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:57.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.518+0000 7f4797e00640 1 --2- 172.21.15.43:0/2708817120 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d634000550 0x55d634000930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.518+0000 7f4797e00640 1 -- 172.21.15.43:0/2708817120 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d634000e70 msgr2=0x55d634009510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:57.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.518+0000 7f4797e00640 1 --2- 172.21.15.43:0/2708817120 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d634000e70 0x55d634009510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.518+0000 7f479e000640 1 --2- 172.21.15.43:0/111021703 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f47800a1ef0 0x7f47800a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.518+0000 7f4797e00640 1 -- 172.21.15.43:0/2708817120 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d633c783f0 con 0x55d634009a50 2024-03-28T22:44:57.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.518+0000 7f479e000640 1 -- 172.21.15.43:0/111021703 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f47980ce8e0 con 0x7f47a01a2dc0 2024-03-28T22:44:57.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.518+0000 7f4797e00640 1 --2- 172.21.15.43:0/2708817120 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d634009a50 0x55d63400be40 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f47a006ac80 tx=0x7f4788009110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1dd4eff29657157f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:57.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.518+0000 7f47a5800640 1 --2- 172.21.15.43:0/111021703 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f47800a1ef0 0x7f47800a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.518+0000 7f4796000640 1 -- 172.21.15.43:0/2708817120 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478801c020 con 0x55d634009a50 2024-03-28T22:44:57.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.518+0000 7f4796000640 1 -- 172.21.15.43:0/2708817120 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4788015e10 con 0x55d634009a50 2024-03-28T22:44:57.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.518+0000 7f4796000640 1 -- 172.21.15.43:0/2708817120 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4788018c40 con 0x55d634009a50 2024-03-28T22:44:57.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.519+0000 7f47aa5a5740 1 -- 172.21.15.43:0/2708817120 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d634009a50 msgr2=0x55d63400be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:57.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.519+0000 7f47a5800640 1 --2- 172.21.15.43:0/111021703 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f47800a1ef0 0x7f47800a43b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4798005fd0 tx=0x7f479800ef50 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:57.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.519+0000 7f47aa5a5740 1 --2- 172.21.15.43:0/2708817120 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d634009a50 0x55d63400be40 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f47a006ac80 tx=0x7f4788009110 comp rx=0 tx=0).stop 2024-03-28T22:44:57.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.519+0000 7f47aa5a5740 1 -- 172.21.15.43:0/2708817120 shutdown_connections 2024-03-28T22:44:57.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.519+0000 7f47aa5a5740 1 --2- 172.21.15.43:0/2708817120 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d634009a50 0x55d63400be40 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.519+0000 7f47aa5a5740 1 --2- 172.21.15.43:0/2708817120 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d634000e70 0x55d634009510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.519+0000 7f47aa5a5740 1 --2- 172.21.15.43:0/2708817120 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d634000550 0x55d634000930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.519+0000 7f47aa5a5740 1 -- 172.21.15.43:0/2708817120 >> 172.21.15.43:0/2708817120 conn(0x55d633ffac80 msgr2=0x55d633ffb060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:57.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.519+0000 7f47aa5a5740 1 -- 172.21.15.43:0/2708817120 shutdown_connections 2024-03-28T22:44:57.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.519+0000 7f47aa5a5740 1 -- 172.21.15.43:0/2708817120 wait complete. 2024-03-28T22:44:57.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.520+0000 7f47aa5a5740 1 -- start start 2024-03-28T22:44:57.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.520+0000 7f47aa5a5740 1 Processor -- start 2024-03-28T22:44:57.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.520+0000 7f47aa5a5740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d634000550 0x55d634006b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.520+0000 7f47aa5a5740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d634000e70 0x55d6340070b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.520+0000 7f47aa5a5740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d634009a50 0x55d6340075f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.521+0000 7f47aa5a5740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55d63400c840 con 0x55d634000550 2024-03-28T22:44:57.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.521+0000 7f47aa5a5740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55d633f022f0 con 0x55d634009a50 2024-03-28T22:44:57.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.521+0000 7f47aa5a5740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55d63400c9c0 con 0x55d634000e70 2024-03-28T22:44:57.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.521+0000 7f47aa5a5740 10 client.0 _ll_register_callbacks cb 0x55d633c77be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:44:57.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.521+0000 7f4797e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d634000550 0x55d634006b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.521+0000 7f47a4e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d634000e70 0x55d6340070b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.521+0000 7f47a4e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d634000e70 0x55d6340070b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40106/0 (socket says 172.21.15.43:40106) 2024-03-28T22:44:57.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.521+0000 7f47a4e00640 1 -- 172.21.15.43:0/3035174891 learned_addr learned my addr 172.21.15.43:0/3035174891 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:57.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.521+0000 7f4797e00640 1 --2- 172.21.15.43:0/3035174891 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d634009a50 0x55d6340075f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.521+0000 7f47a4e00640 1 -- 172.21.15.43:0/3035174891 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d634009a50 msgr2=0x55d6340075f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:57.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.521+0000 7f47a4e00640 1 --2- 172.21.15.43:0/3035174891 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d634009a50 0x55d6340075f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.521+0000 7f47a4e00640 1 -- 172.21.15.43:0/3035174891 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d634000550 msgr2=0x55d634006b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:57.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.521+0000 7f47a4e00640 1 --2- 172.21.15.43:0/3035174891 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d634000550 0x55d634006b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.521+0000 7f47a4e00640 1 -- 172.21.15.43:0/3035174891 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d633d427d0 con 0x55d634000e70 2024-03-28T22:44:57.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.521+0000 7f4797e00640 1 --2- 172.21.15.43:0/3035174891 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d634009a50 0x55d6340075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:44:57.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.522+0000 7f47a4e00640 1 --2- 172.21.15.43:0/3035174891 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d634000e70 0x55d6340070b0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f4790000c00 tx=0x7f479002c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88b8f01a28550d4d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:57.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.522+0000 7f478ea00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:44:57.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.522+0000 7f478ea00640 1 -- 172.21.15.43:0/3035174891 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47900048c0 con 0x55d634000e70 2024-03-28T22:44:57.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.522+0000 7f478ea00640 1 -- 172.21.15.43:0/3035174891 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f4790004a60 con 0x55d634000e70 2024-03-28T22:44:57.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.522+0000 7f47aa5a5740 1 -- 172.21.15.43:0/3035174891 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4788015e10 con 0x55d634000e70 2024-03-28T22:44:57.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.522+0000 7f478ea00640 1 -- 172.21.15.43:0/3035174891 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4790004c60 con 0x55d634000e70 2024-03-28T22:44:57.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.522+0000 7f47aa5a5740 10 client.28339 fetch_fsmap learned FSMap version 32 2024-03-28T22:44:57.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.522+0000 7f47aa5a5740 1 -- 172.21.15.43:0/3035174891 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d634018bd0 con 0x55d634000e70 2024-03-28T22:44:57.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.522+0000 7f478ea00640 1 -- 172.21.15.43:0/3035174891 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4788015e10 con 0x55d634000e70 2024-03-28T22:44:57.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.523+0000 7f478ea00640 1 -- 172.21.15.43:0/3035174891 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4790046070 con 0x55d634000e70 2024-03-28T22:44:57.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.523+0000 7f47aa5a5740 10 client.28339 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:44:57.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.523+0000 7f47aa5a5740 10 client.28339 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:44:57.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.523+0000 7f47aa5a5740 20 client.28339 populate_metadata read hostname 'smithi043' 2024-03-28T22:44:57.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.523+0000 7f47aa5a5740 1 --2- 172.21.15.43:0/3035174891 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55d634019fe0 0x55d63401c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.523+0000 7f47aa5a5740 4 client.28339 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:44:57.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.523+0000 7f47aa5a5740 1 -- 172.21.15.43:0/3035174891 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d633f0d680 con 0x55d634019fe0 2024-03-28T22:44:57.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.523+0000 7f4797e00640 1 --2- 172.21.15.43:0/3035174891 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55d634019fe0 0x55d63401c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.524+0000 7f4797e00640 1 --2- 172.21.15.43:0/3035174891 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55d634019fe0 0x55d63401c460 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2aedfeb0b2209fda server_cookie=d2a84abe02e83d7e in_seq=0 out_seq=0 2024-03-28T22:44:57.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.524+0000 7f478ea00640 10 client.28339 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:44:57.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.524+0000 7f478ea00640 1 -- 172.21.15.43:0/3035174891 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d633f0d680 con 0x55d634019fe0 2024-03-28T22:44:57.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.524+0000 7f478ea00640 10 client.28339 handle_command_reply: tid=0 2024-03-28T22:44:57.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.524+0000 7f47aa5a5740 1 client.28339 shutdown 2024-03-28T22:44:57.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.525+0000 7f47aa5a5740 1 -- 172.21.15.43:0/3035174891 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d634000e70 msgr2=0x55d6340070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:57.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.525+0000 7f47aa5a5740 1 --2- 172.21.15.43:0/3035174891 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d634000e70 0x55d6340070b0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f4790000c00 tx=0x7f479002c8b0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.525+0000 7f47aa5a5740 1 -- 172.21.15.43:0/3035174891 shutdown_connections 2024-03-28T22:44:57.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.525+0000 7f47aa5a5740 1 --2- 172.21.15.43:0/3035174891 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55d634019fe0 0x55d63401c460 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.525+0000 7f47aa5a5740 1 --2- 172.21.15.43:0/3035174891 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d634009a50 0x55d6340075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.525+0000 7f47aa5a5740 1 --2- 172.21.15.43:0/3035174891 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d634000e70 0x55d6340070b0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.525+0000 7f47aa5a5740 1 --2- 172.21.15.43:0/3035174891 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d634000550 0x55d634006b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.525+0000 7f47aa5a5740 1 -- 172.21.15.43:0/3035174891 >> 172.21.15.43:0/3035174891 conn(0x55d633ffac80 msgr2=0x55d633ffc9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:57.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.525+0000 7f478ea00640 0 client.28339 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:44:57.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.525+0000 7f47aa5a5740 1 -- 172.21.15.43:0/3035174891 shutdown_connections 2024-03-28T22:44:57.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.525+0000 7f47aa5a5740 1 -- 172.21.15.43:0/3035174891 wait complete. 2024-03-28T22:44:57.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.525+0000 7f47aa5a5740 20 client.28339 trim_cache size 0 max 16384 2024-03-28T22:44:57.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.540+0000 7f47aa5a5740 1 Processor -- start 2024-03-28T22:44:57.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.540+0000 7f47aa5a5740 1 -- start start 2024-03-28T22:44:57.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.540+0000 7f47aa5a5740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d634000550 0x55d6340092a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.540+0000 7f47aa5a5740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d634000e70 0x55d6340155e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.540+0000 7f47aa5a5740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d634009a50 0x55d634021d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.540+0000 7f47aa5a5740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f47a01b4fb0 con 0x55d634000e70 2024-03-28T22:44:57.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.540+0000 7f47aa5a5740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f47a01b6af0 con 0x55d634000550 2024-03-28T22:44:57.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.540+0000 7f47aa5a5740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f47a01b5680 con 0x55d634009a50 2024-03-28T22:44:57.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.540+0000 7f47a4e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d634000e70 0x55d6340155e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.540+0000 7f4797e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d634009a50 0x55d634021d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.540+0000 7f4797e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d634009a50 0x55d634021d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40114/0 (socket says 172.21.15.43:40114) 2024-03-28T22:44:57.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.540+0000 7f4797e00640 1 -- 172.21.15.43:0/210990324 learned_addr learned my addr 172.21.15.43:0/210990324 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:57.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.540+0000 7f47a4e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d634000e70 0x55d6340155e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45168/0 (socket says 172.21.15.43:45168) 2024-03-28T22:44:57.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.541+0000 7f4797e00640 1 --2- 172.21.15.43:0/210990324 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d634000550 0x55d6340092a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.541+0000 7f4797e00640 1 -- 172.21.15.43:0/210990324 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d634000550 msgr2=0x55d6340092a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:57.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.541+0000 7f4797e00640 1 --2- 172.21.15.43:0/210990324 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d634000550 0x55d6340092a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.541+0000 7f4797e00640 1 -- 172.21.15.43:0/210990324 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d634000e70 msgr2=0x55d6340155e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:57.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.541+0000 7f4797e00640 1 --2- 172.21.15.43:0/210990324 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d634000e70 0x55d6340155e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.541+0000 7f4797e00640 1 -- 172.21.15.43:0/210990324 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d633ffa7b0 con 0x55d634009a50 2024-03-28T22:44:57.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.541+0000 7f47a4e00640 1 --2- 172.21.15.43:0/210990324 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d634000e70 0x55d6340155e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:44:57.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.541+0000 7f4797e00640 1 --2- 172.21.15.43:0/210990324 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d634009a50 0x55d634021d30 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f4788008010 tx=0x7f4788024040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2eb235be3104761e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:57.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.541+0000 7f4795600640 1 -- 172.21.15.43:0/210990324 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478801c040 con 0x55d634009a50 2024-03-28T22:44:57.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.542+0000 7f4795600640 1 -- 172.21.15.43:0/210990324 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4788035070 con 0x55d634009a50 2024-03-28T22:44:57.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.542+0000 7f4795600640 1 -- 172.21.15.43:0/210990324 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4788015bd0 con 0x55d634009a50 2024-03-28T22:44:57.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.542+0000 7f47aa5a5740 1 -- 172.21.15.43:0/210990324 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d634009a50 msgr2=0x55d634021d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:57.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.542+0000 7f47aa5a5740 1 --2- 172.21.15.43:0/210990324 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d634009a50 0x55d634021d30 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f4788008010 tx=0x7f4788024040 comp rx=0 tx=0).stop 2024-03-28T22:44:57.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.542+0000 7f47aa5a5740 1 -- 172.21.15.43:0/210990324 shutdown_connections 2024-03-28T22:44:57.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.542+0000 7f47aa5a5740 1 --2- 172.21.15.43:0/210990324 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d634009a50 0x55d634021d30 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.542+0000 7f47aa5a5740 1 --2- 172.21.15.43:0/210990324 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d634000e70 0x55d6340155e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.542+0000 7f47aa5a5740 1 --2- 172.21.15.43:0/210990324 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d634000550 0x55d6340092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.542+0000 7f47aa5a5740 1 -- 172.21.15.43:0/210990324 >> 172.21.15.43:0/210990324 conn(0x55d633ffac80 msgr2=0x55d634005440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:57.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.542+0000 7f47aa5a5740 1 -- 172.21.15.43:0/210990324 shutdown_connections 2024-03-28T22:44:57.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.542+0000 7f47aa5a5740 1 -- 172.21.15.43:0/210990324 wait complete. 2024-03-28T22:44:57.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.543+0000 7f47aa5a5740 1 -- start start 2024-03-28T22:44:57.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.543+0000 7f47aa5a5740 1 Processor -- start 2024-03-28T22:44:57.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.543+0000 7f47aa5a5740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d634000550 0x55d6340036f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.543+0000 7f47aa5a5740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d634000e70 0x55d634003c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.543+0000 7f47aa5a5740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d634009a50 0x55d634004170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.543+0000 7f47aa5a5740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f47a01b6420 con 0x55d634000550 2024-03-28T22:44:57.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.543+0000 7f47aa5a5740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f47a01b5d50 con 0x55d634000e70 2024-03-28T22:44:57.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.543+0000 7f47aa5a5740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f47a01b2fb0 con 0x55d634009a50 2024-03-28T22:44:57.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.543+0000 7f47aa5a5740 10 client.0 _ll_register_callbacks cb 0x7f4790009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:44:57.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.544+0000 7f47a4e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d634000e70 0x55d634003c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.544+0000 7f47a4e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d634000e70 0x55d634003c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47246/0 (socket says 172.21.15.43:47246) 2024-03-28T22:44:57.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.544+0000 7f47a4e00640 1 -- 172.21.15.43:0/456173895 learned_addr learned my addr 172.21.15.43:0/456173895 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:57.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.544+0000 7f4797e00640 1 --2- 172.21.15.43:0/456173895 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d634000550 0x55d6340036f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.544+0000 7f47a4e00640 1 -- 172.21.15.43:0/456173895 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d634009a50 msgr2=0x55d634004170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:57.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.544+0000 7f47a4e00640 1 --2- 172.21.15.43:0/456173895 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d634009a50 0x55d634004170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.544+0000 7f47a4e00640 1 -- 172.21.15.43:0/456173895 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d634000550 msgr2=0x55d6340036f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:57.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.544+0000 7f47a4e00640 1 --2- 172.21.15.43:0/456173895 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d634000550 0x55d6340036f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.544+0000 7f47a4e00640 1 -- 172.21.15.43:0/456173895 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d63407f360 con 0x55d634000e70 2024-03-28T22:44:57.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.544+0000 7f4797e00640 1 --2- 172.21.15.43:0/456173895 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d634000550 0x55d6340036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:44:57.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.544+0000 7f47a4e00640 1 --2- 172.21.15.43:0/456173895 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d634000e70 0x55d634003c30 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x55d6340063d0 tx=0x7f4790004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38d8043578995592 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:57.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.544+0000 7f478ea00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:44:57.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.544+0000 7f478ea00640 1 -- 172.21.15.43:0/456173895 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4790000e60 con 0x55d634000e70 2024-03-28T22:44:57.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.544+0000 7f47aa5a5740 1 -- 172.21.15.43:0/456173895 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4788035070 con 0x55d634000e70 2024-03-28T22:44:57.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.545+0000 7f478ea00640 1 -- 172.21.15.43:0/456173895 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f47900048c0 con 0x55d634000e70 2024-03-28T22:44:57.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.545+0000 7f478ea00640 1 -- 172.21.15.43:0/456173895 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4790003800 con 0x55d634000e70 2024-03-28T22:44:57.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.545+0000 7f47aa5a5740 10 client.28406 fetch_fsmap learned FSMap version 32 2024-03-28T22:44:57.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.545+0000 7f47aa5a5740 1 -- 172.21.15.43:0/456173895 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d634080620 con 0x55d634000e70 2024-03-28T22:44:57.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.545+0000 7f478ea00640 1 -- 172.21.15.43:0/456173895 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4788035070 con 0x55d634000e70 2024-03-28T22:44:57.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.545+0000 7f478ea00640 1 -- 172.21.15.43:0/456173895 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4790046040 con 0x55d634000e70 2024-03-28T22:44:57.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.545+0000 7f47aa5a5740 10 client.28406 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:44:57.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.545+0000 7f47aa5a5740 10 client.28406 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:44:57.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.545+0000 7f47aa5a5740 20 client.28406 populate_metadata read hostname 'smithi043' 2024-03-28T22:44:57.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.545+0000 7f47aa5a5740 1 --2- 172.21.15.43:0/456173895 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55d634080d90 0x55d6340831b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.545+0000 7f47aa5a5740 4 client.28406 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:44:57.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.545+0000 7f47aa5a5740 1 -- 172.21.15.43:0/456173895 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d633f71590 con 0x55d634080d90 2024-03-28T22:44:57.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.546+0000 7f4797e00640 1 --2- 172.21.15.43:0/456173895 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55d634080d90 0x55d6340831b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.546+0000 7f4797e00640 1 --2- 172.21.15.43:0/456173895 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55d634080d90 0x55d6340831b0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b95aa3b0f5f9af7b server_cookie=9275962901d6c4a0 in_seq=0 out_seq=0 2024-03-28T22:44:57.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.546+0000 7f478ea00640 10 client.28406 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:44:57.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.546+0000 7f478ea00640 1 -- 172.21.15.43:0/456173895 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d633f71590 con 0x55d634080d90 2024-03-28T22:44:57.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.546+0000 7f478ea00640 10 client.28406 handle_command_reply: tid=0 2024-03-28T22:44:57.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.546+0000 7f47aa5a5740 1 client.28406 shutdown 2024-03-28T22:44:57.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.546+0000 7f47aa5a5740 1 -- 172.21.15.43:0/456173895 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d634000e70 msgr2=0x55d634003c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:57.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.546+0000 7f47aa5a5740 1 --2- 172.21.15.43:0/456173895 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d634000e70 0x55d634003c30 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x55d6340063d0 tx=0x7f4790004610 comp rx=0 tx=0).stop 2024-03-28T22:44:57.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.546+0000 7f47aa5a5740 1 -- 172.21.15.43:0/456173895 shutdown_connections 2024-03-28T22:44:57.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.546+0000 7f47aa5a5740 1 --2- 172.21.15.43:0/456173895 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55d634080d90 0x55d6340831b0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.546+0000 7f47aa5a5740 1 --2- 172.21.15.43:0/456173895 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d634000e70 0x55d634003c30 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.546+0000 7f47aa5a5740 1 --2- 172.21.15.43:0/456173895 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55d634009a50 0x55d634004170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.546+0000 7f478ea00640 0 client.28406 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:44:57.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.546+0000 7f47aa5a5740 1 --2- 172.21.15.43:0/456173895 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55d634000550 0x55d6340036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.546+0000 7f47aa5a5740 1 -- 172.21.15.43:0/456173895 >> 172.21.15.43:0/456173895 conn(0x55d633ffac80 msgr2=0x55d633ff9530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:57.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.547+0000 7f47aa5a5740 1 -- 172.21.15.43:0/456173895 shutdown_connections 2024-03-28T22:44:57.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.547+0000 7f47aa5a5740 1 -- 172.21.15.43:0/456173895 wait complete. 2024-03-28T22:44:57.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.547+0000 7f47aa5a5740 20 client.28406 trim_cache size 0 max 16384 2024-03-28T22:44:57.554 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:44:57.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.547+0000 7f47a6200640 1 -- 172.21.15.43:0/111021703 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f47800a1ef0 msgr2=0x7f47800a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:57.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.547+0000 7f47a6200640 1 --2- 172.21.15.43:0/111021703 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f47800a1ef0 0x7f47800a43b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4798005fd0 tx=0x7f479800ef50 comp rx=0 tx=0).stop 2024-03-28T22:44:57.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.547+0000 7f47a6200640 1 -- 172.21.15.43:0/111021703 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f47a01a2dc0 msgr2=0x7f47a01a3270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:57.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.547+0000 7f47a6200640 1 --2- 172.21.15.43:0/111021703 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f47a01a2dc0 0x7f47a01a3270 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f479800e960 tx=0x7f479800ed60 comp rx=0 tx=0).stop 2024-03-28T22:44:57.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.548+0000 7f47a6200640 1 -- 172.21.15.43:0/111021703 shutdown_connections 2024-03-28T22:44:57.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.548+0000 7f47a6200640 1 --2- 172.21.15.43:0/111021703 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f47800a1ef0 0x7f47800a43b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.548+0000 7f47a6200640 1 --2- 172.21.15.43:0/111021703 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f47a01a2dc0 0x7f47a01a3270 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.548+0000 7f47a6200640 1 --2- 172.21.15.43:0/111021703 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f47a010cf50 0x7f47a019ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.548+0000 7f47a6200640 1 --2- 172.21.15.43:0/111021703 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f47a0107f00 0x7f47a019e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.548+0000 7f47a6200640 1 -- 172.21.15.43:0/111021703 >> 172.21.15.43:0/111021703 conn(0x7f47a0077570 msgr2=0x7f47a0077b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:57.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.548+0000 7f47a6200640 1 -- 172.21.15.43:0/111021703 shutdown_connections 2024-03-28T22:44:57.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.548+0000 7f47a6200640 1 -- 172.21.15.43:0/111021703 wait complete. 2024-03-28T22:44:57.565 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:44:57.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.797+0000 7f9e95c00640 1 -- 172.21.15.43:0/2828638414 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e901002f0 msgr2=0x7f9e9010c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:57.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.797+0000 7f9e95c00640 1 --2- 172.21.15.43:0/2828638414 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e901002f0 0x7f9e9010c800 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f9e8800adb0 tx=0x7f9e8802cff0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.797+0000 7f9e95c00640 1 -- 172.21.15.43:0/2828638414 shutdown_connections 2024-03-28T22:44:57.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.797+0000 7f9e95c00640 1 --2- 172.21.15.43:0/2828638414 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e901002f0 0x7f9e9010c800 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.797+0000 7f9e95c00640 1 --2- 172.21.15.43:0/2828638414 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e900ff930 0x7f9e900ffdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.797+0000 7f9e95c00640 1 --2- 172.21.15.43:0/2828638414 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e90105940 0x7f9e90105d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.797+0000 7f9e95c00640 1 -- 172.21.15.43:0/2828638414 >> 172.21.15.43:0/2828638414 conn(0x7f9e900767a0 msgr2=0x7f9e900fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:57.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.797+0000 7f9e95c00640 1 -- 172.21.15.43:0/2828638414 shutdown_connections 2024-03-28T22:44:57.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.797+0000 7f9e95c00640 1 -- 172.21.15.43:0/2828638414 wait complete. 2024-03-28T22:44:57.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.799+0000 7f9e95c00640 1 Processor -- start 2024-03-28T22:44:57.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.799+0000 7f9e95c00640 1 -- start start 2024-03-28T22:44:57.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.799+0000 7f9e95c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e900ff930 0x7f9e900fbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.799+0000 7f9e95c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e901002f0 0x7f9e900fc130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.799+0000 7f9e95c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e90105940 0x7f9e900fc670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.799+0000 7f9e95c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9010f920 con 0x7f9e900ff930 2024-03-28T22:44:57.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.799+0000 7f9e95c00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9010f7a0 con 0x7f9e90105940 2024-03-28T22:44:57.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.799+0000 7f9e95c00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9010faa0 con 0x7f9e901002f0 2024-03-28T22:44:57.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.799+0000 7f9e87e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e901002f0 0x7f9e900fc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.799+0000 7f9e8ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e900ff930 0x7f9e900fbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.800+0000 7f9e8f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e90105940 0x7f9e900fc670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.800+0000 7f9e87e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e901002f0 0x7f9e900fc130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40128/0 (socket says 172.21.15.43:40128) 2024-03-28T22:44:57.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.800+0000 7f9e87e00640 1 -- 172.21.15.43:0/12838699 learned_addr learned my addr 172.21.15.43:0/12838699 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:57.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.800+0000 7f9e87e00640 1 -- 172.21.15.43:0/12838699 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e90105940 msgr2=0x7f9e900fc670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:57.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.800+0000 7f9e87e00640 1 --2- 172.21.15.43:0/12838699 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e90105940 0x7f9e900fc670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.800+0000 7f9e87e00640 1 -- 172.21.15.43:0/12838699 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e900ff930 msgr2=0x7f9e900fbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:57.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.800+0000 7f9e87e00640 1 --2- 172.21.15.43:0/12838699 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e900ff930 0x7f9e900fbbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.800+0000 7f9e87e00640 1 -- 172.21.15.43:0/12838699 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e901a6540 con 0x7f9e901002f0 2024-03-28T22:44:57.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.800+0000 7f9e8f400640 1 --2- 172.21.15.43:0/12838699 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e90105940 0x7f9e900fc670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:44:57.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.800+0000 7f9e87e00640 1 --2- 172.21.15.43:0/12838699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e901002f0 0x7f9e900fc130 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f9e7800e9f0 tx=0x7f9e7800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:57.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.800+0000 7f9e87400640 1 -- 172.21.15.43:0/12838699 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e7800cd80 con 0x7f9e901002f0 2024-03-28T22:44:57.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.801+0000 7f9e87400640 1 -- 172.21.15.43:0/12838699 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e78005500 con 0x7f9e901002f0 2024-03-28T22:44:57.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.801+0000 7f9e87400640 1 -- 172.21.15.43:0/12838699 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e78010690 con 0x7f9e901002f0 2024-03-28T22:44:57.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.801+0000 7f9e95c00640 1 -- 172.21.15.43:0/12838699 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e901a6830 con 0x7f9e901002f0 2024-03-28T22:44:57.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.801+0000 7f9e95c00640 1 -- 172.21.15.43:0/12838699 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e901a6d30 con 0x7f9e901002f0 2024-03-28T22:44:57.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.804+0000 7f9e989c6740 1 Processor -- start 2024-03-28T22:44:57.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.804+0000 7f9e87400640 1 -- 172.21.15.43:0/12838699 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f9e780046c0 con 0x7f9e901002f0 2024-03-28T22:44:57.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.804+0000 7f9e989c6740 1 -- start start 2024-03-28T22:44:57.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.805+0000 7f9e989c6740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557560f97380 0x557560f97760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.805+0000 7f9e989c6740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557560f97ca0 0x557560fa0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.805+0000 7f9e989c6740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557560fa0880 0x557560fa2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.805+0000 7f9e989c6740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x557560efb050 con 0x557560f97380 2024-03-28T22:44:57.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.805+0000 7f9e989c6740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x557560ea3ce0 con 0x557560fa0880 2024-03-28T22:44:57.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.805+0000 7f9e989c6740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x557560ef70f0 con 0x557560f97ca0 2024-03-28T22:44:57.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.805+0000 7f9e8ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557560f97ca0 0x557560fa0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.805+0000 7f9e8f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557560f97380 0x557560f97760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.805+0000 7f9e8ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557560f97ca0 0x557560fa0340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40132/0 (socket says 172.21.15.43:40132) 2024-03-28T22:44:57.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.805+0000 7f9e8ea00640 1 -- 172.21.15.43:0/3083448089 learned_addr learned my addr 172.21.15.43:0/3083448089 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:57.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.805+0000 7f9e87e00640 1 --2- 172.21.15.43:0/3083448089 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557560fa0880 0x557560fa2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.805+0000 7f9e8ea00640 1 -- 172.21.15.43:0/3083448089 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557560fa0880 msgr2=0x557560fa2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:57.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.805+0000 7f9e8ea00640 1 --2- 172.21.15.43:0/3083448089 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557560fa0880 0x557560fa2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.805+0000 7f9e8ea00640 1 -- 172.21.15.43:0/3083448089 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557560f97380 msgr2=0x557560f97760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:57.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.805+0000 7f9e8ea00640 1 --2- 172.21.15.43:0/3083448089 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557560f97380 0x557560f97760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.805+0000 7f9e8ea00640 1 -- 172.21.15.43:0/3083448089 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557560c0f3f0 con 0x557560f97ca0 2024-03-28T22:44:57.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.806+0000 7f9e8ea00640 1 --2- 172.21.15.43:0/3083448089 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557560f97ca0 0x557560fa0340 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f9e8000bc90 tx=0x7f9e800107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6316a074b8a7ad0f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:57.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.806+0000 7f9e7fe00640 1 -- 172.21.15.43:0/3083448089 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e8001c480 con 0x557560f97ca0 2024-03-28T22:44:57.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.806+0000 7f9e87400640 1 --2- 172.21.15.43:0/12838699 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9e740a1ef0 0x7f9e740a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.806+0000 7f9e7fe00640 1 -- 172.21.15.43:0/3083448089 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e8001cb00 con 0x557560f97ca0 2024-03-28T22:44:57.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.806+0000 7f9e7fe00640 1 -- 172.21.15.43:0/3083448089 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e80011c60 con 0x557560f97ca0 2024-03-28T22:44:57.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.806+0000 7f9e87400640 1 -- 172.21.15.43:0/12838699 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f9e780ce980 con 0x7f9e901002f0 2024-03-28T22:44:57.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.806+0000 7f9e8f400640 1 --2- 172.21.15.43:0/12838699 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9e740a1ef0 0x7f9e740a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.807+0000 7f9e989c6740 1 -- 172.21.15.43:0/3083448089 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557560f97ca0 msgr2=0x557560fa0340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:57.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.807+0000 7f9e989c6740 1 --2- 172.21.15.43:0/3083448089 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557560f97ca0 0x557560fa0340 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f9e8000bc90 tx=0x7f9e800107a0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.807+0000 7f9e989c6740 1 -- 172.21.15.43:0/3083448089 shutdown_connections 2024-03-28T22:44:57.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.807+0000 7f9e989c6740 1 --2- 172.21.15.43:0/3083448089 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557560fa0880 0x557560fa2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.807+0000 7f9e989c6740 1 --2- 172.21.15.43:0/3083448089 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557560f97ca0 0x557560fa0340 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.807+0000 7f9e989c6740 1 --2- 172.21.15.43:0/3083448089 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557560f97380 0x557560f97760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.807+0000 7f9e989c6740 1 -- 172.21.15.43:0/3083448089 >> 172.21.15.43:0/3083448089 conn(0x557560f91ab0 msgr2=0x557560f91e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:57.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.807+0000 7f9e989c6740 1 -- 172.21.15.43:0/3083448089 shutdown_connections 2024-03-28T22:44:57.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.807+0000 7f9e8f400640 1 --2- 172.21.15.43:0/12838699 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9e740a1ef0 0x7f9e740a43b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9e88005e30 tx=0x7f9e8802ff70 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:57.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.807+0000 7f9e989c6740 1 -- 172.21.15.43:0/3083448089 wait complete. 2024-03-28T22:44:57.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.808+0000 7f9e989c6740 1 -- start start 2024-03-28T22:44:57.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.808+0000 7f9e989c6740 1 Processor -- start 2024-03-28T22:44:57.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.808+0000 7f9e989c6740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557560f97380 0x557560f9d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.808+0000 7f9e989c6740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557560f97ca0 0x557560f9de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.808+0000 7f9e989c6740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557560fa0880 0x557560f9e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.808+0000 7f9e989c6740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x557560fa3670 con 0x557560fa0880 2024-03-28T22:44:57.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.808+0000 7f9e989c6740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x557560e992f0 con 0x557560f97380 2024-03-28T22:44:57.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.808+0000 7f9e989c6740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x557560fa37f0 con 0x557560f97ca0 2024-03-28T22:44:57.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.809+0000 7f9e8ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557560f97380 0x557560f9d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.809+0000 7f9e989c6740 10 client.0 _ll_register_callbacks cb 0x557560c0ebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:44:57.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.809+0000 7f9e8ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557560f97380 0x557560f9d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47270/0 (socket says 172.21.15.43:47270) 2024-03-28T22:44:57.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.809+0000 7f9e8ea00640 1 -- 172.21.15.43:0/2538390054 learned_addr learned my addr 172.21.15.43:0/2538390054 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:57.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.809+0000 7f9e8f400640 1 --2- 172.21.15.43:0/2538390054 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557560fa0880 0x557560f9e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.809+0000 7f9e87e00640 1 --2- 172.21.15.43:0/2538390054 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557560f97ca0 0x557560f9de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.809+0000 7f9e8ea00640 1 -- 172.21.15.43:0/2538390054 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557560f97ca0 msgr2=0x557560f9de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:57.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.809+0000 7f9e8ea00640 1 --2- 172.21.15.43:0/2538390054 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557560f97ca0 0x557560f9de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.809+0000 7f9e8ea00640 1 -- 172.21.15.43:0/2538390054 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557560fa0880 msgr2=0x557560f9e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:57.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.809+0000 7f9e8ea00640 1 --2- 172.21.15.43:0/2538390054 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557560fa0880 0x557560f9e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.809+0000 7f9e8ea00640 1 -- 172.21.15.43:0/2538390054 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557560cd97d0 con 0x557560f97380 2024-03-28T22:44:57.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.809+0000 7f9e87e00640 1 --2- 172.21.15.43:0/2538390054 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557560f97ca0 0x557560f9de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:44:57.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.809+0000 7f9e8ea00640 1 --2- 172.21.15.43:0/2538390054 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557560f97380 0x557560f9d8f0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f9e8001def0 tx=0x7f9e8001df20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acef47e86efb4ce server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:57.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.809+0000 7f9e7cc00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:44:57.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.810+0000 7f9e7cc00640 1 -- 172.21.15.43:0/2538390054 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e8000bd80 con 0x557560f97380 2024-03-28T22:44:57.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.810+0000 7f9e7cc00640 1 -- 172.21.15.43:0/2538390054 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f9e80011bb0 con 0x557560f97380 2024-03-28T22:44:57.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.810+0000 7f9e989c6740 1 -- 172.21.15.43:0/2538390054 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e8001cb00 con 0x557560f97380 2024-03-28T22:44:57.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.810+0000 7f9e7cc00640 1 -- 172.21.15.43:0/2538390054 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9e80031050 con 0x557560f97380 2024-03-28T22:44:57.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.810+0000 7f9e989c6740 10 client.28418 fetch_fsmap learned FSMap version 32 2024-03-28T22:44:57.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.810+0000 7f9e989c6740 1 -- 172.21.15.43:0/2538390054 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557560faf960 con 0x557560f97380 2024-03-28T22:44:57.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.810+0000 7f9e7cc00640 1 -- 172.21.15.43:0/2538390054 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e8001cb00 con 0x557560f97380 2024-03-28T22:44:57.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.810+0000 7f9e7cc00640 1 -- 172.21.15.43:0/2538390054 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f9e80005040 con 0x557560f97380 2024-03-28T22:44:57.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.810+0000 7f9e989c6740 10 client.28418 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:44:57.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.810+0000 7f9e989c6740 10 client.28418 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:44:57.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.810+0000 7f9e989c6740 20 client.28418 populate_metadata read hostname 'smithi043' 2024-03-28T22:44:57.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.811+0000 7f9e989c6740 1 --2- 172.21.15.43:0/2538390054 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x557560fb0d60 0x557560fb3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.811+0000 7f9e989c6740 4 client.28418 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:44:57.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.811+0000 7f9e989c6740 1 -- 172.21.15.43:0/2538390054 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557560ea4680 con 0x557560fb0d60 2024-03-28T22:44:57.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.811+0000 7f9e87e00640 1 --2- 172.21.15.43:0/2538390054 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x557560fb0d60 0x557560fb3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.811+0000 7f9e87e00640 1 --2- 172.21.15.43:0/2538390054 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x557560fb0d60 0x557560fb3150 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f73ace368e894efc server_cookie=b3d1da565d662625 in_seq=0 out_seq=0 2024-03-28T22:44:57.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.811+0000 7f9e7cc00640 10 client.28418 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:44:57.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.812+0000 7f9e7cc00640 1 -- 172.21.15.43:0/2538390054 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557560ea4680 con 0x557560fb0d60 2024-03-28T22:44:57.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.812+0000 7f9e7cc00640 10 client.28418 handle_command_reply: tid=0 2024-03-28T22:44:57.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.813+0000 7f9e989c6740 1 client.28418 shutdown 2024-03-28T22:44:57.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.813+0000 7f9e989c6740 1 -- 172.21.15.43:0/2538390054 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557560f97380 msgr2=0x557560f9d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:57.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.813+0000 7f9e989c6740 1 --2- 172.21.15.43:0/2538390054 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557560f97380 0x557560f9d8f0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f9e8001def0 tx=0x7f9e8001df20 comp rx=0 tx=0).stop 2024-03-28T22:44:57.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.813+0000 7f9e989c6740 1 -- 172.21.15.43:0/2538390054 shutdown_connections 2024-03-28T22:44:57.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.813+0000 7f9e989c6740 1 --2- 172.21.15.43:0/2538390054 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x557560fb0d60 0x557560fb3150 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.813+0000 7f9e989c6740 1 --2- 172.21.15.43:0/2538390054 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557560fa0880 0x557560f9e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.813+0000 7f9e989c6740 1 --2- 172.21.15.43:0/2538390054 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557560f97ca0 0x557560f9de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.813+0000 7f9e989c6740 1 --2- 172.21.15.43:0/2538390054 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557560f97380 0x557560f9d8f0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.813+0000 7f9e7cc00640 0 client.28418 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:44:57.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.813+0000 7f9e989c6740 1 -- 172.21.15.43:0/2538390054 >> 172.21.15.43:0/2538390054 conn(0x557560f91ab0 msgr2=0x557560f93800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:57.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.813+0000 7f9e989c6740 1 -- 172.21.15.43:0/2538390054 shutdown_connections 2024-03-28T22:44:57.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.813+0000 7f9e989c6740 1 -- 172.21.15.43:0/2538390054 wait complete. 2024-03-28T22:44:57.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.813+0000 7f9e989c6740 20 client.28418 trim_cache size 0 max 16384 2024-03-28T22:44:57.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.828+0000 7f9e989c6740 1 Processor -- start 2024-03-28T22:44:57.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.828+0000 7f9e989c6740 1 -- start start 2024-03-28T22:44:57.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.829+0000 7f9e989c6740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557560f97380 0x557560fa0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.829+0000 7f9e989c6740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557560f97ca0 0x557560fac410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.829+0000 7f9e989c6740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557560fa0880 0x557560f9e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.829+0000 7f9e989c6740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9e901b0ea0 con 0x557560fa0880 2024-03-28T22:44:57.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.829+0000 7f9e989c6740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9e901b29e0 con 0x557560f97380 2024-03-28T22:44:57.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.829+0000 7f9e989c6740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9e901b1570 con 0x557560f97ca0 2024-03-28T22:44:57.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.829+0000 7f9e8ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557560f97380 0x557560fa0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.829+0000 7f9e8ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557560f97380 0x557560fa0020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47286/0 (socket says 172.21.15.43:47286) 2024-03-28T22:44:57.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.829+0000 7f9e8ea00640 1 -- 172.21.15.43:0/3407877254 learned_addr learned my addr 172.21.15.43:0/3407877254 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:57.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.829+0000 7f9e87e00640 1 --2- 172.21.15.43:0/3407877254 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557560f97ca0 0x557560fac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.829+0000 7f9e8f400640 1 --2- 172.21.15.43:0/3407877254 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557560fa0880 0x557560f9e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.829+0000 7f9e8ea00640 1 -- 172.21.15.43:0/3407877254 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557560f97ca0 msgr2=0x557560fac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:57.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.829+0000 7f9e8ea00640 1 --2- 172.21.15.43:0/3407877254 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557560f97ca0 0x557560fac410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.829+0000 7f9e8ea00640 1 -- 172.21.15.43:0/3407877254 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557560fa0880 msgr2=0x557560f9e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:57.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.829+0000 7f9e8ea00640 1 --2- 172.21.15.43:0/3407877254 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557560fa0880 0x557560f9e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.829+0000 7f9e8ea00640 1 -- 172.21.15.43:0/3407877254 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557560f994b0 con 0x557560f97380 2024-03-28T22:44:57.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.830+0000 7f9e8ea00640 1 --2- 172.21.15.43:0/3407877254 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557560f97380 0x557560fa0020 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f9e8000bc90 tx=0x7f9e8000b410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6e0705996620554 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:57.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.830+0000 7f9e7f400640 1 -- 172.21.15.43:0/3407877254 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e8003a070 con 0x557560f97380 2024-03-28T22:44:57.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.830+0000 7f9e7f400640 1 -- 172.21.15.43:0/3407877254 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e80005630 con 0x557560f97380 2024-03-28T22:44:57.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.830+0000 7f9e7f400640 1 -- 172.21.15.43:0/3407877254 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e80011bb0 con 0x557560f97380 2024-03-28T22:44:57.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.831+0000 7f9e989c6740 1 -- 172.21.15.43:0/3407877254 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557560f97380 msgr2=0x557560fa0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:57.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.831+0000 7f9e989c6740 1 --2- 172.21.15.43:0/3407877254 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557560f97380 0x557560fa0020 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f9e8000bc90 tx=0x7f9e8000b410 comp rx=0 tx=0).stop 2024-03-28T22:44:57.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.831+0000 7f9e989c6740 1 -- 172.21.15.43:0/3407877254 shutdown_connections 2024-03-28T22:44:57.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.831+0000 7f9e989c6740 1 --2- 172.21.15.43:0/3407877254 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557560fa0880 0x557560f9e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.831+0000 7f9e989c6740 1 --2- 172.21.15.43:0/3407877254 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557560f97ca0 0x557560fac410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.831+0000 7f9e989c6740 1 --2- 172.21.15.43:0/3407877254 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557560f97380 0x557560fa0020 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.831+0000 7f9e989c6740 1 -- 172.21.15.43:0/3407877254 >> 172.21.15.43:0/3407877254 conn(0x557560f91ab0 msgr2=0x557560f91e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:57.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.831+0000 7f9e989c6740 1 -- 172.21.15.43:0/3407877254 shutdown_connections 2024-03-28T22:44:57.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.831+0000 7f9e989c6740 1 -- 172.21.15.43:0/3407877254 wait complete. 2024-03-28T22:44:57.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.831+0000 7f9e989c6740 1 -- start start 2024-03-28T22:44:57.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.831+0000 7f9e989c6740 1 Processor -- start 2024-03-28T22:44:57.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.832+0000 7f9e989c6740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557560f97380 0x557560fa89c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.832+0000 7f9e989c6740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557560f97ca0 0x557560fa3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.832+0000 7f9e989c6740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557560fa0880 0x557560fa44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.832+0000 7f9e989c6740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9e901b2310 con 0x557560f97ca0 2024-03-28T22:44:57.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.832+0000 7f9e989c6740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9e901b1c40 con 0x557560fa0880 2024-03-28T22:44:57.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.832+0000 7f9e989c6740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f9e901aeea0 con 0x557560f97380 2024-03-28T22:44:57.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.832+0000 7f9e989c6740 10 client.0 _ll_register_callbacks cb 0x7f9e78005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:44:57.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.832+0000 7f9e8ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557560f97380 0x557560fa89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.832+0000 7f9e8f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557560fa0880 0x557560fa44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.832+0000 7f9e8f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557560fa0880 0x557560fa44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47294/0 (socket says 172.21.15.43:47294) 2024-03-28T22:44:57.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.832+0000 7f9e8ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557560f97380 0x557560fa89c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40168/0 (socket says 172.21.15.43:40168) 2024-03-28T22:44:57.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.832+0000 7f9e8f400640 1 -- 172.21.15.43:0/4147479365 learned_addr learned my addr 172.21.15.43:0/4147479365 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:57.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.832+0000 7f9e87e00640 1 --2- 172.21.15.43:0/4147479365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557560f97ca0 0x557560fa3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.833+0000 7f9e8ea00640 1 -- 172.21.15.43:0/4147479365 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557560fa0880 msgr2=0x557560fa44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:57.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.833+0000 7f9e8ea00640 1 --2- 172.21.15.43:0/4147479365 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557560fa0880 0x557560fa44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.833+0000 7f9e8ea00640 1 -- 172.21.15.43:0/4147479365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557560f97ca0 msgr2=0x557560fa3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:57.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.833+0000 7f9e8ea00640 1 --2- 172.21.15.43:0/4147479365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557560f97ca0 0x557560fa3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.833+0000 7f9e8ea00640 1 -- 172.21.15.43:0/4147479365 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557560fa4a30 con 0x557560f97380 2024-03-28T22:44:57.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.833+0000 7f9e8f400640 1 --2- 172.21.15.43:0/4147479365 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557560fa0880 0x557560fa44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:44:57.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.833+0000 7f9e87e00640 1 --2- 172.21.15.43:0/4147479365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557560f97ca0 0x557560fa3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:44:57.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.833+0000 7f9e8ea00640 1 --2- 172.21.15.43:0/4147479365 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557560f97380 0x557560fa89c0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f9e80010ca0 tx=0x7f9e800057e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7d025e9a59583f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:57.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.833+0000 7f9e7cc00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:44:57.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.833+0000 7f9e7cc00640 1 -- 172.21.15.43:0/4147479365 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e8003a070 con 0x557560f97380 2024-03-28T22:44:57.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.833+0000 7f9e7cc00640 1 -- 172.21.15.43:0/4147479365 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f9e800035f0 con 0x557560f97380 2024-03-28T22:44:57.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.833+0000 7f9e989c6740 1 -- 172.21.15.43:0/4147479365 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e80005630 con 0x557560f97380 2024-03-28T22:44:57.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.833+0000 7f9e7cc00640 1 -- 172.21.15.43:0/4147479365 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9e80011bb0 con 0x557560f97380 2024-03-28T22:44:57.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.833+0000 7f9e989c6740 10 client.28360 fetch_fsmap learned FSMap version 32 2024-03-28T22:44:57.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.833+0000 7f9e989c6740 1 -- 172.21.15.43:0/4147479365 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557561016b00 con 0x557560f97380 2024-03-28T22:44:57.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.834+0000 7f9e7cc00640 1 -- 172.21.15.43:0/4147479365 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e80005630 con 0x557560f97380 2024-03-28T22:44:57.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.834+0000 7f9e7cc00640 1 -- 172.21.15.43:0/4147479365 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f9e8001c900 con 0x557560f97380 2024-03-28T22:44:57.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.834+0000 7f9e989c6740 10 client.28360 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:44:57.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.834+0000 7f9e989c6740 10 client.28360 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:44:57.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.834+0000 7f9e989c6740 20 client.28360 populate_metadata read hostname 'smithi043' 2024-03-28T22:44:57.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.834+0000 7f9e989c6740 1 --2- 172.21.15.43:0/4147479365 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x557561017150 0x557561019540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:57.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.834+0000 7f9e989c6740 4 client.28360 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:44:57.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.834+0000 7f9e989c6740 1 -- 172.21.15.43:0/4147479365 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557560f08590 con 0x557561017150 2024-03-28T22:44:57.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.834+0000 7f9e87e00640 1 --2- 172.21.15.43:0/4147479365 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x557561017150 0x557561019540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:57.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.835+0000 7f9e87e00640 1 --2- 172.21.15.43:0/4147479365 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x557561017150 0x557561019540 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=151a515111384bb6 server_cookie=ba27d14eafa0b9fa in_seq=0 out_seq=0 2024-03-28T22:44:57.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.835+0000 7f9e7cc00640 10 client.28360 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:44:57.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.835+0000 7f9e7cc00640 1 -- 172.21.15.43:0/4147479365 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557560f08590 con 0x557561017150 2024-03-28T22:44:57.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.835+0000 7f9e7cc00640 10 client.28360 handle_command_reply: tid=0 2024-03-28T22:44:57.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.835+0000 7f9e989c6740 1 client.28360 shutdown 2024-03-28T22:44:57.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.835+0000 7f9e989c6740 1 -- 172.21.15.43:0/4147479365 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557560f97380 msgr2=0x557560fa89c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:57.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.835+0000 7f9e989c6740 1 --2- 172.21.15.43:0/4147479365 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557560f97380 0x557560fa89c0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f9e80010ca0 tx=0x7f9e800057e0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.835+0000 7f9e989c6740 1 -- 172.21.15.43:0/4147479365 shutdown_connections 2024-03-28T22:44:57.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.835+0000 7f9e989c6740 1 --2- 172.21.15.43:0/4147479365 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x557561017150 0x557561019540 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.835+0000 7f9e989c6740 1 --2- 172.21.15.43:0/4147479365 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557560fa0880 0x557560fa44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.835+0000 7f9e989c6740 1 --2- 172.21.15.43:0/4147479365 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557560f97ca0 0x557560fa3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.835+0000 7f9e989c6740 1 --2- 172.21.15.43:0/4147479365 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557560f97380 0x557560fa89c0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.835+0000 7f9e7cc00640 0 client.28360 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:44:57.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.835+0000 7f9e989c6740 1 -- 172.21.15.43:0/4147479365 >> 172.21.15.43:0/4147479365 conn(0x557560f91ab0 msgr2=0x557560f92e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:57.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.835+0000 7f9e989c6740 1 -- 172.21.15.43:0/4147479365 shutdown_connections 2024-03-28T22:44:57.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.835+0000 7f9e989c6740 1 -- 172.21.15.43:0/4147479365 wait complete. 2024-03-28T22:44:57.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.836+0000 7f9e989c6740 20 client.28360 trim_cache size 0 max 16384 2024-03-28T22:44:57.843 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:44:57.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.836+0000 7f9e95c00640 1 -- 172.21.15.43:0/12838699 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9e740a1ef0 msgr2=0x7f9e740a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:57.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.836+0000 7f9e95c00640 1 --2- 172.21.15.43:0/12838699 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9e740a1ef0 0x7f9e740a43b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9e88005e30 tx=0x7f9e8802ff70 comp rx=0 tx=0).stop 2024-03-28T22:44:57.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.836+0000 7f9e95c00640 1 -- 172.21.15.43:0/12838699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e901002f0 msgr2=0x7f9e900fc130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:57.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.836+0000 7f9e95c00640 1 --2- 172.21.15.43:0/12838699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e901002f0 0x7f9e900fc130 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f9e7800e9f0 tx=0x7f9e7800edf0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.836+0000 7f9e95c00640 1 -- 172.21.15.43:0/12838699 shutdown_connections 2024-03-28T22:44:57.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.836+0000 7f9e95c00640 1 --2- 172.21.15.43:0/12838699 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f9e740a1ef0 0x7f9e740a43b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.836+0000 7f9e95c00640 1 --2- 172.21.15.43:0/12838699 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e90105940 0x7f9e900fc670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.836+0000 7f9e95c00640 1 --2- 172.21.15.43:0/12838699 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9e901002f0 0x7f9e900fc130 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.836+0000 7f9e95c00640 1 --2- 172.21.15.43:0/12838699 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e900ff930 0x7f9e900fbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:57.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.836+0000 7f9e95c00640 1 -- 172.21.15.43:0/12838699 >> 172.21.15.43:0/12838699 conn(0x7f9e900767a0 msgr2=0x7f9e90102c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:57.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.836+0000 7f9e95c00640 1 -- 172.21.15.43:0/12838699 shutdown_connections 2024-03-28T22:44:57.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:57.836+0000 7f9e95c00640 1 -- 172.21.15.43:0/12838699 wait complete. 2024-03-28T22:44:57.852 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:44:58.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.072+0000 7fd3de400640 1 -- 172.21.15.43:0/349662799 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd3d8107f00 msgr2=0x7fd3d81082e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:58.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.072+0000 7fd3de400640 1 --2- 172.21.15.43:0/349662799 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd3d8107f00 0x7fd3d81082e0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc009900 tx=0x7fd3cc02bc40 comp rx=0 tx=0).stop 2024-03-28T22:44:58.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.072+0000 7fd3de400640 1 -- 172.21.15.43:0/349662799 shutdown_connections 2024-03-28T22:44:58.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.072+0000 7fd3de400640 1 --2- 172.21.15.43:0/349662799 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3d810cf50 0x7fd3d8114b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:58.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.072+0000 7fd3de400640 1 --2- 172.21.15.43:0/349662799 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3d81088b0 0x7fd3d810c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:58.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.072+0000 7fd3de400640 1 --2- 172.21.15.43:0/349662799 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd3d8107f00 0x7fd3d81082e0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:58.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.072+0000 7fd3de400640 1 -- 172.21.15.43:0/349662799 >> 172.21.15.43:0/349662799 conn(0x7fd3d8077570 msgr2=0x7fd3d8106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:58.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.072+0000 7fd3de400640 1 -- 172.21.15.43:0/349662799 shutdown_connections 2024-03-28T22:44:58.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.072+0000 7fd3de400640 1 -- 172.21.15.43:0/349662799 wait complete. 2024-03-28T22:44:58.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.073+0000 7fd3de400640 1 Processor -- start 2024-03-28T22:44:58.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.074+0000 7fd3de400640 1 -- start start 2024-03-28T22:44:58.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.074+0000 7fd3de400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3d8107f00 0x7fd3d819e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:58.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.074+0000 7fd3de400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd3d810cf50 0x7fd3d819e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:58.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.074+0000 7fd3de400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3d81a2d70 0x7fd3d81a3220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:58.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.074+0000 7fd3de400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d8118070 con 0x7fd3d81a2d70 2024-03-28T22:44:58.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.074+0000 7fd3de400640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d8117ef0 con 0x7fd3d810cf50 2024-03-28T22:44:58.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.074+0000 7fd3de400640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d81181f0 con 0x7fd3d8107f00 2024-03-28T22:44:58.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.074+0000 7fd3dd000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3d8107f00 0x7fd3d819e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:58.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.074+0000 7fd3dd000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3d8107f00 0x7fd3d819e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40196/0 (socket says 172.21.15.43:40196) 2024-03-28T22:44:58.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.074+0000 7fd3dd000640 1 -- 172.21.15.43:0/2405150624 learned_addr learned my addr 172.21.15.43:0/2405150624 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:58.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.074+0000 7fd3d7e00640 1 --2- 172.21.15.43:0/2405150624 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd3d810cf50 0x7fd3d819e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:58.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.074+0000 7fd3dda00640 1 --2- 172.21.15.43:0/2405150624 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3d81a2d70 0x7fd3d81a3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:58.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.075+0000 7fd3dd000640 1 -- 172.21.15.43:0/2405150624 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd3d810cf50 msgr2=0x7fd3d819e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:58.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.075+0000 7fd3dd000640 1 --2- 172.21.15.43:0/2405150624 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd3d810cf50 0x7fd3d819e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:58.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.075+0000 7fd3dd000640 1 -- 172.21.15.43:0/2405150624 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3d81a2d70 msgr2=0x7fd3d81a3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:58.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.075+0000 7fd3dd000640 1 --2- 172.21.15.43:0/2405150624 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3d81a2d70 0x7fd3d81a3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:58.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.075+0000 7fd3dd000640 1 -- 172.21.15.43:0/2405150624 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3d81a3860 con 0x7fd3d8107f00 2024-03-28T22:44:58.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.075+0000 7fd3dda00640 1 --2- 172.21.15.43:0/2405150624 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3d81a2d70 0x7fd3d81a3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:44:58.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.075+0000 7fd3dd000640 1 --2- 172.21.15.43:0/2405150624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3d8107f00 0x7fd3d819e4b0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc005ab0 tx=0x7fd3cc004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:58.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.075+0000 7fd3d5600640 1 -- 172.21.15.43:0/2405150624 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc041070 con 0x7fd3d8107f00 2024-03-28T22:44:58.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.075+0000 7fd3d5600640 1 -- 172.21.15.43:0/2405150624 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3cc02cca0 con 0x7fd3d8107f00 2024-03-28T22:44:58.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.075+0000 7fd3d5600640 1 -- 172.21.15.43:0/2405150624 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc003e50 con 0x7fd3d8107f00 2024-03-28T22:44:58.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.075+0000 7fd3de400640 1 -- 172.21.15.43:0/2405150624 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3d81a3af0 con 0x7fd3d8107f00 2024-03-28T22:44:58.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.075+0000 7fd3de400640 1 -- 172.21.15.43:0/2405150624 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3d81ab2b0 con 0x7fd3d8107f00 2024-03-28T22:44:58.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.080+0000 7fd3e260a740 1 Processor -- start 2024-03-28T22:44:58.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.080+0000 7fd3d5600640 1 -- 172.21.15.43:0/2405150624 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fd3cc044070 con 0x7fd3d8107f00 2024-03-28T22:44:58.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.080+0000 7fd3e260a740 1 -- start start 2024-03-28T22:44:58.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.080+0000 7fd3e260a740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55908926f550 0x55908926f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:58.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.080+0000 7fd3e260a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55908926fe70 0x559089278510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:58.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.080+0000 7fd3e260a740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559089278a50 0x55908927ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:58.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.080+0000 7fd3e260a740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5590891d3050 con 0x559089278a50 2024-03-28T22:44:58.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.080+0000 7fd3e260a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55908917bce0 con 0x55908926fe70 2024-03-28T22:44:58.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.080+0000 7fd3e260a740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5590891cf0f0 con 0x55908926f550 2024-03-28T22:44:58.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.081+0000 7fd3dda00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55908926f550 0x55908926f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:58.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.081+0000 7fd3dd000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55908926fe70 0x559089278510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:58.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.081+0000 7fd3dda00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55908926f550 0x55908926f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40202/0 (socket says 172.21.15.43:40202) 2024-03-28T22:44:58.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.081+0000 7fd3dda00640 1 -- 172.21.15.43:0/4098457106 learned_addr learned my addr 172.21.15.43:0/4098457106 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:58.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.081+0000 7fd3d7e00640 1 --2- 172.21.15.43:0/4098457106 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559089278a50 0x55908927ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:58.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.081+0000 7fd3dda00640 1 -- 172.21.15.43:0/4098457106 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55908926fe70 msgr2=0x559089278510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:58.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.081+0000 7fd3dda00640 1 --2- 172.21.15.43:0/4098457106 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55908926fe70 0x559089278510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:58.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.081+0000 7fd3dda00640 1 -- 172.21.15.43:0/4098457106 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559089278a50 msgr2=0x55908927ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:58.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.081+0000 7fd3dda00640 1 --2- 172.21.15.43:0/4098457106 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559089278a50 0x55908927ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:58.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.081+0000 7fd3dda00640 1 -- 172.21.15.43:0/4098457106 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559088ee73f0 con 0x55908926f550 2024-03-28T22:44:58.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.081+0000 7fd3dda00640 1 --2- 172.21.15.43:0/4098457106 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55908926f550 0x55908926f930 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0009f20 tx=0x7fd3d0010800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=776a9a2c11b6d774 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:58.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.082+0000 7fd3d5600640 1 --2- 172.21.15.43:0/2405150624 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd3c00a2110 0x7fd3c00a45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:58.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.082+0000 7fd3ca000640 1 -- 172.21.15.43:0/4098457106 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d0016070 con 0x55908926f550 2024-03-28T22:44:58.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.082+0000 7fd3ca000640 1 -- 172.21.15.43:0/4098457106 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3d001bb40 con 0x55908926f550 2024-03-28T22:44:58.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.082+0000 7fd3ca000640 1 -- 172.21.15.43:0/4098457106 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d0012cb0 con 0x55908926f550 2024-03-28T22:44:58.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.082+0000 7fd3d5600640 1 -- 172.21.15.43:0/2405150624 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7fd3cc0efed0 con 0x7fd3d8107f00 2024-03-28T22:44:58.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.082+0000 7fd3dd000640 1 --2- 172.21.15.43:0/2405150624 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd3c00a2110 0x7fd3c00a45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:58.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.082+0000 7fd3e260a740 1 -- 172.21.15.43:0/4098457106 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55908926f550 msgr2=0x55908926f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:58.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.082+0000 7fd3e260a740 1 --2- 172.21.15.43:0/4098457106 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55908926f550 0x55908926f930 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0009f20 tx=0x7fd3d0010800 comp rx=0 tx=0).stop 2024-03-28T22:44:58.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.083+0000 7fd3e260a740 1 -- 172.21.15.43:0/4098457106 shutdown_connections 2024-03-28T22:44:58.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.083+0000 7fd3e260a740 1 --2- 172.21.15.43:0/4098457106 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55908926fe70 0x559089278510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:58.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.083+0000 7fd3e260a740 1 --2- 172.21.15.43:0/4098457106 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559089278a50 0x55908927ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:58.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.083+0000 7fd3e260a740 1 --2- 172.21.15.43:0/4098457106 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55908926f550 0x55908926f930 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:58.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.083+0000 7fd3e260a740 1 -- 172.21.15.43:0/4098457106 >> 172.21.15.43:0/4098457106 conn(0x559089269c80 msgr2=0x55908926a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:58.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.083+0000 7fd3dd000640 1 --2- 172.21.15.43:0/2405150624 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd3c00a2110 0x7fd3c00a45d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc0ed410 tx=0x7fd3cc0dc040 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:58.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.083+0000 7fd3e260a740 1 -- 172.21.15.43:0/4098457106 shutdown_connections 2024-03-28T22:44:58.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.083+0000 7fd3e260a740 1 -- 172.21.15.43:0/4098457106 wait complete. 2024-03-28T22:44:58.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.083+0000 7fd3e260a740 1 -- start start 2024-03-28T22:44:58.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.083+0000 7fd3e260a740 1 Processor -- start 2024-03-28T22:44:58.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.084+0000 7fd3e260a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55908926f550 0x559089275ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:58.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.084+0000 7fd3e260a740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55908926fe70 0x559089276020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:58.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.084+0000 7fd3e260a740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559089278a50 0x559089276560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:58.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.084+0000 7fd3e260a740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55908927b840 con 0x55908926fe70 2024-03-28T22:44:58.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.084+0000 7fd3e260a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5590891712f0 con 0x55908926f550 2024-03-28T22:44:58.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.084+0000 7fd3e260a740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55908927b9c0 con 0x559089278a50 2024-03-28T22:44:58.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.084+0000 7fd3e260a740 10 client.0 _ll_register_callbacks cb 0x559088ee6be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:44:58.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.084+0000 7fd3dda00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55908926f550 0x559089275ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:58.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.084+0000 7fd3dda00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55908926f550 0x559089275ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47334/0 (socket says 172.21.15.43:47334) 2024-03-28T22:44:58.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.084+0000 7fd3dda00640 1 -- 172.21.15.43:0/4188365187 learned_addr learned my addr 172.21.15.43:0/4188365187 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:58.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.084+0000 7fd3dda00640 1 --2- 172.21.15.43:0/4188365187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559089278a50 0x559089276560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:58.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.084+0000 7fd3d7e00640 1 --2- 172.21.15.43:0/4188365187 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55908926fe70 0x559089276020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:58.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.085+0000 7fd3dda00640 1 -- 172.21.15.43:0/4188365187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559089278a50 msgr2=0x559089276560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:58.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.085+0000 7fd3dda00640 1 --2- 172.21.15.43:0/4188365187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559089278a50 0x559089276560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:58.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.085+0000 7fd3dda00640 1 -- 172.21.15.43:0/4188365187 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55908926fe70 msgr2=0x559089276020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:58.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.085+0000 7fd3dda00640 1 --2- 172.21.15.43:0/4188365187 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55908926fe70 0x559089276020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:58.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.085+0000 7fd3dda00640 1 -- 172.21.15.43:0/4188365187 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559088fb17d0 con 0x55908926f550 2024-03-28T22:44:58.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.085+0000 7fd3dda00640 1 --2- 172.21.15.43:0/4188365187 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55908926f550 0x559089275ae0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0012990 tx=0x7fd3d0012890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7220b4a7ba1bfc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:58.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.085+0000 7fd3bea00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:44:58.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.085+0000 7fd3bea00640 1 -- 172.21.15.43:0/4188365187 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d0016070 con 0x55908926f550 2024-03-28T22:44:58.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.085+0000 7fd3e260a740 1 -- 172.21.15.43:0/4188365187 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3d001bb40 con 0x55908926f550 2024-03-28T22:44:58.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.085+0000 7fd3bea00640 1 -- 172.21.15.43:0/4188365187 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7fd3d001b460 con 0x55908926f550 2024-03-28T22:44:58.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.086+0000 7fd3bea00640 1 -- 172.21.15.43:0/4188365187 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd3d001bde0 con 0x55908926f550 2024-03-28T22:44:58.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.086+0000 7fd3e260a740 10 client.28436 fetch_fsmap learned FSMap version 32 2024-03-28T22:44:58.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.086+0000 7fd3e260a740 1 -- 172.21.15.43:0/4188365187 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559089287b40 con 0x55908926f550 2024-03-28T22:44:58.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.086+0000 7fd3bea00640 1 -- 172.21.15.43:0/4188365187 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3d001bb40 con 0x55908926f550 2024-03-28T22:44:58.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.086+0000 7fd3bea00640 1 -- 172.21.15.43:0/4188365187 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd3d000f040 con 0x55908926f550 2024-03-28T22:44:58.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.086+0000 7fd3e260a740 10 client.28436 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:44:58.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.086+0000 7fd3e260a740 10 client.28436 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:44:58.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.086+0000 7fd3e260a740 20 client.28436 populate_metadata read hostname 'smithi043' 2024-03-28T22:44:58.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.086+0000 7fd3e260a740 1 --2- 172.21.15.43:0/4188365187 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559089288ef0 0x55908928b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:58.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.086+0000 7fd3e260a740 4 client.28436 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:44:58.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.086+0000 7fd3e260a740 1 -- 172.21.15.43:0/4188365187 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55908917c680 con 0x559089288ef0 2024-03-28T22:44:58.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.086+0000 7fd3d7e00640 1 --2- 172.21.15.43:0/4188365187 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559089288ef0 0x55908928b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:58.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.087+0000 7fd3d7e00640 1 --2- 172.21.15.43:0/4188365187 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559089288ef0 0x55908928b2e0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b39b6e09743f70ab server_cookie=7e920b80521bdeb8 in_seq=0 out_seq=0 2024-03-28T22:44:58.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.087+0000 7fd3bea00640 10 client.28436 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:44:58.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.088+0000 7fd3bea00640 1 -- 172.21.15.43:0/4188365187 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55908917c680 con 0x559089288ef0 2024-03-28T22:44:58.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.088+0000 7fd3bea00640 10 client.28436 handle_command_reply: tid=0 2024-03-28T22:44:58.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.088+0000 7fd3e260a740 1 client.28436 shutdown 2024-03-28T22:44:58.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.088+0000 7fd3e260a740 1 -- 172.21.15.43:0/4188365187 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55908926f550 msgr2=0x559089275ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:58.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.088+0000 7fd3e260a740 1 --2- 172.21.15.43:0/4188365187 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55908926f550 0x559089275ae0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0012990 tx=0x7fd3d0012890 comp rx=0 tx=0).stop 2024-03-28T22:44:58.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.088+0000 7fd3e260a740 1 -- 172.21.15.43:0/4188365187 shutdown_connections 2024-03-28T22:44:58.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.088+0000 7fd3e260a740 1 --2- 172.21.15.43:0/4188365187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559089278a50 0x559089276560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:58.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.088+0000 7fd3e260a740 1 --2- 172.21.15.43:0/4188365187 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55908926fe70 0x559089276020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:58.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.088+0000 7fd3e260a740 1 --2- 172.21.15.43:0/4188365187 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559089288ef0 0x55908928b2e0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:58.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.088+0000 7fd3e260a740 1 --2- 172.21.15.43:0/4188365187 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55908926f550 0x559089275ae0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:58.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.088+0000 7fd3e260a740 1 -- 172.21.15.43:0/4188365187 >> 172.21.15.43:0/4188365187 conn(0x559089269c80 msgr2=0x55908926b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:58.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.088+0000 7fd3bea00640 0 client.28436 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:44:58.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.088+0000 7fd3e260a740 1 -- 172.21.15.43:0/4188365187 shutdown_connections 2024-03-28T22:44:58.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.089+0000 7fd3e260a740 1 -- 172.21.15.43:0/4188365187 wait complete. 2024-03-28T22:44:58.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.089+0000 7fd3e260a740 20 client.28436 trim_cache size 0 max 16384 2024-03-28T22:44:58.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.103+0000 7fd3e260a740 1 Processor -- start 2024-03-28T22:44:58.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.103+0000 7fd3e260a740 1 -- start start 2024-03-28T22:44:58.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.104+0000 7fd3e260a740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55908926f550 0x5590892845e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:58.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.104+0000 7fd3e260a740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55908926fe70 0x559089276380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:58.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.104+0000 7fd3e260a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559089278a50 0x559089290a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:58.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.104+0000 7fd3e260a740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d81b5010 con 0x55908926fe70 2024-03-28T22:44:58.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.104+0000 7fd3e260a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d81b6b50 con 0x559089278a50 2024-03-28T22:44:58.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.104+0000 7fd3e260a740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d81b56e0 con 0x55908926f550 2024-03-28T22:44:58.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.104+0000 7fd3dda00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55908926f550 0x5590892845e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:58.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.104+0000 7fd3dda00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55908926f550 0x5590892845e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40218/0 (socket says 172.21.15.43:40218) 2024-03-28T22:44:58.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.104+0000 7fd3dda00640 1 -- 172.21.15.43:0/1082139438 learned_addr learned my addr 172.21.15.43:0/1082139438 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:58.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.104+0000 7fd3dda00640 1 --2- 172.21.15.43:0/1082139438 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559089278a50 0x559089290a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:58.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.104+0000 7fd3d7e00640 1 --2- 172.21.15.43:0/1082139438 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55908926fe70 0x559089276380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:58.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.105+0000 7fd3dda00640 1 -- 172.21.15.43:0/1082139438 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559089278a50 msgr2=0x559089290a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:58.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.105+0000 7fd3dda00640 1 --2- 172.21.15.43:0/1082139438 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559089278a50 0x559089290a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:58.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.105+0000 7fd3dda00640 1 -- 172.21.15.43:0/1082139438 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55908926fe70 msgr2=0x559089276380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:58.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.105+0000 7fd3dda00640 1 --2- 172.21.15.43:0/1082139438 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55908926fe70 0x559089276380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:58.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.105+0000 7fd3dda00640 1 -- 172.21.15.43:0/1082139438 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55908926fca0 con 0x55908926f550 2024-03-28T22:44:58.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.105+0000 7fd3dda00640 1 --2- 172.21.15.43:0/1082139438 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55908926f550 0x5590892845e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fd3d001e040 tx=0x7fd3d0010800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afa11ac2b5460510 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:58.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.106+0000 7fd3c9600640 1 -- 172.21.15.43:0/1082139438 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d0016040 con 0x55908926f550 2024-03-28T22:44:58.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.106+0000 7fd3c9600640 1 -- 172.21.15.43:0/1082139438 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3d0010d00 con 0x55908926f550 2024-03-28T22:44:58.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.106+0000 7fd3c9600640 1 -- 172.21.15.43:0/1082139438 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d001b750 con 0x55908926f550 2024-03-28T22:44:58.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.106+0000 7fd3e260a740 1 -- 172.21.15.43:0/1082139438 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55908926f550 msgr2=0x5590892845e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:58.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.106+0000 7fd3e260a740 1 --2- 172.21.15.43:0/1082139438 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55908926f550 0x5590892845e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fd3d001e040 tx=0x7fd3d0010800 comp rx=0 tx=0).stop 2024-03-28T22:44:58.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.106+0000 7fd3e260a740 1 -- 172.21.15.43:0/1082139438 shutdown_connections 2024-03-28T22:44:58.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.106+0000 7fd3e260a740 1 --2- 172.21.15.43:0/1082139438 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559089278a50 0x559089290a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:58.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.106+0000 7fd3e260a740 1 --2- 172.21.15.43:0/1082139438 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55908926fe70 0x559089276380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:58.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.106+0000 7fd3e260a740 1 --2- 172.21.15.43:0/1082139438 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55908926f550 0x5590892845e0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:58.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.106+0000 7fd3e260a740 1 -- 172.21.15.43:0/1082139438 >> 172.21.15.43:0/1082139438 conn(0x559089269c80 msgr2=0x559089268dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:58.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.106+0000 7fd3e260a740 1 -- 172.21.15.43:0/1082139438 shutdown_connections 2024-03-28T22:44:58.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.106+0000 7fd3e260a740 1 -- 172.21.15.43:0/1082139438 wait complete. 2024-03-28T22:44:58.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.107+0000 7fd3e260a740 1 -- start start 2024-03-28T22:44:58.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.107+0000 7fd3e260a740 1 Processor -- start 2024-03-28T22:44:58.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.107+0000 7fd3e260a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55908926f550 0x559089281010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:58.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.107+0000 7fd3e260a740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55908926fe70 0x55908927c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:58.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.107+0000 7fd3e260a740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559089278a50 0x55908927cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:58.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.107+0000 7fd3e260a740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d81b6480 con 0x55908926fe70 2024-03-28T22:44:58.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.107+0000 7fd3e260a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d81b5db0 con 0x55908926f550 2024-03-28T22:44:58.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.107+0000 7fd3e260a740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d81b3010 con 0x559089278a50 2024-03-28T22:44:58.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.107+0000 7fd3e260a740 10 client.0 _ll_register_callbacks cb 0x7fd3cc0f38f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:44:58.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.108+0000 7fd3dda00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55908926f550 0x559089281010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:58.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.108+0000 7fd3d7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55908926fe70 0x55908927c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:58.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.108+0000 7fd3dda00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55908926f550 0x559089281010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47354/0 (socket says 172.21.15.43:47354) 2024-03-28T22:44:58.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.108+0000 7fd3dda00640 1 -- 172.21.15.43:0/4052458412 learned_addr learned my addr 172.21.15.43:0/4052458412 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:58.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.108+0000 7fd3dda00640 1 --2- 172.21.15.43:0/4052458412 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559089278a50 0x55908927cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:58.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.108+0000 7fd3d7e00640 1 -- 172.21.15.43:0/4052458412 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559089278a50 msgr2=0x55908927cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:58.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.108+0000 7fd3d7e00640 1 --2- 172.21.15.43:0/4052458412 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559089278a50 0x55908927cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:58.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.108+0000 7fd3d7e00640 1 -- 172.21.15.43:0/4052458412 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55908926f550 msgr2=0x559089281010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:58.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.108+0000 7fd3d7e00640 1 --2- 172.21.15.43:0/4052458412 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55908926f550 0x559089281010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:58.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.108+0000 7fd3d7e00640 1 -- 172.21.15.43:0/4052458412 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590892ee050 con 0x55908926fe70 2024-03-28T22:44:58.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.108+0000 7fd3dda00640 1 --2- 172.21.15.43:0/4052458412 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559089278a50 0x55908927cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:44:58.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.108+0000 7fd3d7e00640 1 --2- 172.21.15.43:0/4052458412 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55908926fe70 0x55908927c380 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x559089275340 tx=0x7fd3c40204d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd2b90f4f9c8150d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:58.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.108+0000 7fd3bea00640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:44:58.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.109+0000 7fd3bea00640 1 -- 172.21.15.43:0/4052458412 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c4024030 con 0x55908926fe70 2024-03-28T22:44:58.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.109+0000 7fd3bea00640 1 -- 172.21.15.43:0/4052458412 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7fd3c4005020 con 0x55908926fe70 2024-03-28T22:44:58.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.109+0000 7fd3e260a740 1 -- 172.21.15.43:0/4052458412 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3d0010d00 con 0x55908926fe70 2024-03-28T22:44:58.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.109+0000 7fd3bea00640 1 -- 172.21.15.43:0/4052458412 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd3c402e050 con 0x55908926fe70 2024-03-28T22:44:58.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.109+0000 7fd3e260a740 10 client.18174 fetch_fsmap learned FSMap version 32 2024-03-28T22:44:58.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.109+0000 7fd3e260a740 1 -- 172.21.15.43:0/4052458412 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5590892ef210 con 0x55908926fe70 2024-03-28T22:44:58.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.109+0000 7fd3bea00640 1 -- 172.21.15.43:0/4052458412 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3d0010d00 con 0x55908926fe70 2024-03-28T22:44:58.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.109+0000 7fd3bea00640 1 -- 172.21.15.43:0/4052458412 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd3c40035f0 con 0x55908926fe70 2024-03-28T22:44:58.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.110+0000 7fd3e260a740 10 client.18174 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:44:58.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.110+0000 7fd3e260a740 10 client.18174 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:44:58.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.110+0000 7fd3e260a740 20 client.18174 populate_metadata read hostname 'smithi043' 2024-03-28T22:44:58.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.110+0000 7fd3e260a740 1 --2- 172.21.15.43:0/4052458412 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5590892ef960 0x5590892f1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:58.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.110+0000 7fd3e260a740 4 client.18174 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:44:58.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.110+0000 7fd3e260a740 1 -- 172.21.15.43:0/4052458412 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5590891e0590 con 0x5590892ef960 2024-03-28T22:44:58.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.110+0000 7fd3dda00640 1 --2- 172.21.15.43:0/4052458412 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5590892ef960 0x5590892f1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:58.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.110+0000 7fd3dda00640 1 --2- 172.21.15.43:0/4052458412 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5590892ef960 0x5590892f1d80 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5948d60a14986c5 server_cookie=a3cfee604118bcf8 in_seq=0 out_seq=0 2024-03-28T22:44:58.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.110+0000 7fd3bea00640 10 client.18174 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:44:58.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.110+0000 7fd3bea00640 1 -- 172.21.15.43:0/4052458412 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5590891e0590 con 0x5590892ef960 2024-03-28T22:44:58.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.110+0000 7fd3bea00640 10 client.18174 handle_command_reply: tid=0 2024-03-28T22:44:58.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.111+0000 7fd3e260a740 1 client.18174 shutdown 2024-03-28T22:44:58.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.111+0000 7fd3e260a740 1 -- 172.21.15.43:0/4052458412 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55908926fe70 msgr2=0x55908927c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:58.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.111+0000 7fd3e260a740 1 --2- 172.21.15.43:0/4052458412 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55908926fe70 0x55908927c380 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x559089275340 tx=0x7fd3c40204d0 comp rx=0 tx=0).stop 2024-03-28T22:44:58.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.111+0000 7fd3e260a740 1 -- 172.21.15.43:0/4052458412 shutdown_connections 2024-03-28T22:44:58.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.111+0000 7fd3e260a740 1 --2- 172.21.15.43:0/4052458412 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559089278a50 0x55908927cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:58.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.111+0000 7fd3e260a740 1 --2- 172.21.15.43:0/4052458412 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55908926fe70 0x55908927c380 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:58.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.111+0000 7fd3e260a740 1 --2- 172.21.15.43:0/4052458412 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5590892ef960 0x5590892f1d80 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:58.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.111+0000 7fd3e260a740 1 --2- 172.21.15.43:0/4052458412 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55908926f550 0x559089281010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:58.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.111+0000 7fd3e260a740 1 -- 172.21.15.43:0/4052458412 >> 172.21.15.43:0/4052458412 conn(0x559089269c80 msgr2=0x55908926af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:58.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.111+0000 7fd3bea00640 0 client.18174 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:44:58.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.111+0000 7fd3e260a740 1 -- 172.21.15.43:0/4052458412 shutdown_connections 2024-03-28T22:44:58.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.111+0000 7fd3e260a740 1 -- 172.21.15.43:0/4052458412 wait complete. 2024-03-28T22:44:58.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.111+0000 7fd3e260a740 20 client.18174 trim_cache size 0 max 16384 2024-03-28T22:44:58.119 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:44:58.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.112+0000 7fd3de400640 1 -- 172.21.15.43:0/2405150624 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd3c00a2110 msgr2=0x7fd3c00a45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:58.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.112+0000 7fd3de400640 1 --2- 172.21.15.43:0/2405150624 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd3c00a2110 0x7fd3c00a45d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc0ed410 tx=0x7fd3cc0dc040 comp rx=0 tx=0).stop 2024-03-28T22:44:58.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.112+0000 7fd3de400640 1 -- 172.21.15.43:0/2405150624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3d8107f00 msgr2=0x7fd3d819e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:58.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.112+0000 7fd3de400640 1 --2- 172.21.15.43:0/2405150624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3d8107f00 0x7fd3d819e4b0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc005ab0 tx=0x7fd3cc004360 comp rx=0 tx=0).stop 2024-03-28T22:44:58.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.112+0000 7fd3de400640 1 -- 172.21.15.43:0/2405150624 shutdown_connections 2024-03-28T22:44:58.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.112+0000 7fd3de400640 1 --2- 172.21.15.43:0/2405150624 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd3c00a2110 0x7fd3c00a45d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:58.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.112+0000 7fd3de400640 1 --2- 172.21.15.43:0/2405150624 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd3d810cf50 0x7fd3d819e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:58.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.112+0000 7fd3de400640 1 --2- 172.21.15.43:0/2405150624 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3d81a2d70 0x7fd3d81a3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:58.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.112+0000 7fd3de400640 1 --2- 172.21.15.43:0/2405150624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd3d8107f00 0x7fd3d819e4b0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:58.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.112+0000 7fd3de400640 1 -- 172.21.15.43:0/2405150624 >> 172.21.15.43:0/2405150624 conn(0x7fd3d8077570 msgr2=0x7fd3d8077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:58.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.112+0000 7fd3de400640 1 -- 172.21.15.43:0/2405150624 shutdown_connections 2024-03-28T22:44:58.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:58.112+0000 7fd3de400640 1 -- 172.21.15.43:0/2405150624 wait complete. 2024-03-28T22:44:59.130 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:44:59.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:44:59.131 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:44:59.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:44:58 smithi172 ceph-mon[36566]: pgmap v241: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 14 op/s 2024-03-28T22:44:59.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:44:58 smithi082 ceph-mon[36471]: pgmap v241: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 14 op/s 2024-03-28T22:44:59.224 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:44:58 smithi043 ceph-mon[31823]: pgmap v241: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 14 op/s 2024-03-28T22:44:59.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.374+0000 7feef7200640 1 -- 172.21.15.43:0/3525970945 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feef8075c80 msgr2=0x7feef8076060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.374+0000 7feef7200640 1 --2- 172.21.15.43:0/3525970945 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feef8075c80 0x7feef8076060 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7feee8009930 tx=0x7feee802bc20 comp rx=0 tx=0).stop 2024-03-28T22:44:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.374+0000 7feef7200640 1 -- 172.21.15.43:0/3525970945 shutdown_connections 2024-03-28T22:44:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.374+0000 7feef7200640 1 --2- 172.21.15.43:0/3525970945 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feef8112470 0x7feef8114860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.374+0000 7feef7200640 1 --2- 172.21.15.43:0/3525970945 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feef80765a0 0x7feef8111f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.374+0000 7feef7200640 1 --2- 172.21.15.43:0/3525970945 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feef8075c80 0x7feef8076060 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.374+0000 7feef7200640 1 -- 172.21.15.43:0/3525970945 >> 172.21.15.43:0/3525970945 conn(0x7feef80ff260 msgr2=0x7feef8101680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.374+0000 7feef7200640 1 -- 172.21.15.43:0/3525970945 shutdown_connections 2024-03-28T22:44:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.374+0000 7feef7200640 1 -- 172.21.15.43:0/3525970945 wait complete. 2024-03-28T22:44:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.375+0000 7feef7200640 1 Processor -- start 2024-03-28T22:44:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.375+0000 7feef7200640 1 -- start start 2024-03-28T22:44:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.375+0000 7feef7200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feef8075c80 0x7feef81a45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.376+0000 7feef7200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feef80765a0 0x7feef81a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.376+0000 7feef7200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feef8112470 0x7feef819e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.376+0000 7feef7200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7feef8117fd0 con 0x7feef8112470 2024-03-28T22:44:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.376+0000 7feef7200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7feef8117e50 con 0x7feef80765a0 2024-03-28T22:44:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.376+0000 7feef7200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7feef8118150 con 0x7feef8075c80 2024-03-28T22:44:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.376+0000 7feef5400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feef80765a0 0x7feef81a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.376+0000 7feef5e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feef8075c80 0x7feef81a45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.376+0000 7feef5400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feef80765a0 0x7feef81a4b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47362/0 (socket says 172.21.15.43:47362) 2024-03-28T22:44:59.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.376+0000 7feef5400640 1 -- 172.21.15.43:0/928049481 learned_addr learned my addr 172.21.15.43:0/928049481 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:59.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.376+0000 7feef6800640 1 --2- 172.21.15.43:0/928049481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feef8112470 0x7feef819e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.376+0000 7feef5400640 1 -- 172.21.15.43:0/928049481 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feef8075c80 msgr2=0x7feef81a45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:59.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.376+0000 7feef5400640 1 --2- 172.21.15.43:0/928049481 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feef8075c80 0x7feef81a45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.376+0000 7feef5400640 1 -- 172.21.15.43:0/928049481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feef8112470 msgr2=0x7feef819e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.376+0000 7feef5400640 1 --2- 172.21.15.43:0/928049481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feef8112470 0x7feef819e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.376+0000 7feef5400640 1 -- 172.21.15.43:0/928049481 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feef819ef70 con 0x7feef80765a0 2024-03-28T22:44:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.376+0000 7feef6800640 1 --2- 172.21.15.43:0/928049481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feef8112470 0x7feef819e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:44:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.376+0000 7feef5400640 1 --2- 172.21.15.43:0/928049481 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feef80765a0 0x7feef81a4b30 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7feee400d950 tx=0x7feee400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.377+0000 7feeeea00640 1 -- 172.21.15.43:0/928049481 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee4017030 con 0x7feef80765a0 2024-03-28T22:44:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.377+0000 7feeeea00640 1 -- 172.21.15.43:0/928049481 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feee4004020 con 0x7feef80765a0 2024-03-28T22:44:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.377+0000 7feeeea00640 1 -- 172.21.15.43:0/928049481 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee40047a0 con 0x7feef80765a0 2024-03-28T22:44:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.377+0000 7feef7200640 1 -- 172.21.15.43:0/928049481 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feef819f260 con 0x7feef80765a0 2024-03-28T22:44:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.377+0000 7feef7200640 1 -- 172.21.15.43:0/928049481 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feef81ab570 con 0x7feef80765a0 2024-03-28T22:44:59.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.380+0000 7feeeea00640 1 -- 172.21.15.43:0/928049481 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7feee40041d0 con 0x7feef80765a0 2024-03-28T22:44:59.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.380+0000 7feeffb83740 1 Processor -- start 2024-03-28T22:44:59.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.380+0000 7feeffb83740 1 -- start start 2024-03-28T22:44:59.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.381+0000 7feeffb83740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b2a657c170 0x55b2a657c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.381+0000 7feeffb83740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2a657ca90 0x55b2a6585130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.381+0000 7feeffb83740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b2a6585670 0x55b2a6587a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.381+0000 7feeffb83740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b2a64dfc40 con 0x55b2a657ca90 2024-03-28T22:44:59.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.381+0000 7feeffb83740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55b2a6488c40 con 0x55b2a6585670 2024-03-28T22:44:59.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.381+0000 7feeffb83740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55b2a64dbce0 con 0x55b2a657c170 2024-03-28T22:44:59.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.381+0000 7feef5400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b2a6585670 0x55b2a6587a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.381+0000 7feef5e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2a657ca90 0x55b2a6585130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.381+0000 7feef6800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b2a657c170 0x55b2a657c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.381+0000 7feef5e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2a657ca90 0x55b2a6585130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45318/0 (socket says 172.21.15.43:45318) 2024-03-28T22:44:59.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.381+0000 7feef5e00640 1 -- 172.21.15.43:0/355817169 learned_addr learned my addr 172.21.15.43:0/355817169 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:59.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.381+0000 7feef5400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b2a6585670 0x55b2a6587a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47366/0 (socket says 172.21.15.43:47366) 2024-03-28T22:44:59.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.381+0000 7feef5400640 1 -- 172.21.15.43:0/355817169 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b2a657c170 msgr2=0x55b2a657c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:59.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.381+0000 7feef5400640 1 --2- 172.21.15.43:0/355817169 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b2a657c170 0x55b2a657c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.381+0000 7feef5400640 1 -- 172.21.15.43:0/355817169 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2a657ca90 msgr2=0x55b2a6585130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:59.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.381+0000 7feef5400640 1 --2- 172.21.15.43:0/355817169 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2a657ca90 0x55b2a6585130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.381+0000 7feeeea00640 1 --2- 172.21.15.43:0/928049481 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7feedc0a1f70 0x7feedc0a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.381+0000 7feef5400640 1 -- 172.21.15.43:0/355817169 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2a62bc7e0 con 0x55b2a6585670 2024-03-28T22:44:59.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.382+0000 7feeeea00640 1 -- 172.21.15.43:0/928049481 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7feee40cdf60 con 0x7feef80765a0 2024-03-28T22:44:59.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.382+0000 7feef5400640 1 --2- 172.21.15.43:0/355817169 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b2a6585670 0x55b2a6587a60 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7feee40cb280 tx=0x7feee40d1f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4317fdceac0d980 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:59.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.382+0000 7feef6800640 1 --2- 172.21.15.43:0/928049481 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7feedc0a1f70 0x7feedc0a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.382+0000 7feee3400640 1 -- 172.21.15.43:0/355817169 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee409a2f0 con 0x55b2a6585670 2024-03-28T22:44:59.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.382+0000 7feee3400640 1 -- 172.21.15.43:0/355817169 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feee409ac20 con 0x55b2a6585670 2024-03-28T22:44:59.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.382+0000 7feee3400640 1 -- 172.21.15.43:0/355817169 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee4099e80 con 0x55b2a6585670 2024-03-28T22:44:59.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.383+0000 7feeffb83740 1 -- 172.21.15.43:0/355817169 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b2a6585670 msgr2=0x55b2a6587a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:59.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.383+0000 7feef6800640 1 --2- 172.21.15.43:0/928049481 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7feedc0a1f70 0x7feedc0a4430 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7feef806ab80 tx=0x7feef0009500 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:59.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.383+0000 7feeffb83740 1 --2- 172.21.15.43:0/355817169 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b2a6585670 0x55b2a6587a60 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7feee40cb280 tx=0x7feee40d1f70 comp rx=0 tx=0).stop 2024-03-28T22:44:59.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.383+0000 7feeffb83740 1 -- 172.21.15.43:0/355817169 shutdown_connections 2024-03-28T22:44:59.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.383+0000 7feeffb83740 1 --2- 172.21.15.43:0/355817169 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b2a6585670 0x55b2a6587a60 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.383+0000 7feeffb83740 1 --2- 172.21.15.43:0/355817169 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2a657ca90 0x55b2a6585130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.383+0000 7feeffb83740 1 --2- 172.21.15.43:0/355817169 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b2a657c170 0x55b2a657c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.383+0000 7feeffb83740 1 -- 172.21.15.43:0/355817169 >> 172.21.15.43:0/355817169 conn(0x55b2a65768a0 msgr2=0x55b2a6576c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:59.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.383+0000 7feeffb83740 1 -- 172.21.15.43:0/355817169 shutdown_connections 2024-03-28T22:44:59.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.383+0000 7feeffb83740 1 -- 172.21.15.43:0/355817169 wait complete. 2024-03-28T22:44:59.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.383+0000 7feeffb83740 1 -- start start 2024-03-28T22:44:59.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.384+0000 7feeffb83740 1 Processor -- start 2024-03-28T22:44:59.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.384+0000 7feeffb83740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2a657c170 0x55b2a65826f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.384+0000 7feeffb83740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b2a657ca90 0x55b2a6582c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.384+0000 7feeffb83740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b2a6585670 0x55b2a6583170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.384+0000 7feeffb83740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b2a6588460 con 0x55b2a657c170 2024-03-28T22:44:59.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.384+0000 7feeffb83740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55b2a647e250 con 0x55b2a657ca90 2024-03-28T22:44:59.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.384+0000 7feeffb83740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55b2a65885e0 con 0x55b2a6585670 2024-03-28T22:44:59.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.384+0000 7feeffb83740 10 client.0 _ll_register_callbacks cb 0x55b2a649fe70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:44:59.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.384+0000 7feef5400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b2a657ca90 0x55b2a6582c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.385+0000 7feef6800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b2a6585670 0x55b2a6583170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.385+0000 7feef5e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2a657c170 0x55b2a65826f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.385+0000 7feef5400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b2a657ca90 0x55b2a6582c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47374/0 (socket says 172.21.15.43:47374) 2024-03-28T22:44:59.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.385+0000 7feef5400640 1 -- 172.21.15.43:0/4013999712 learned_addr learned my addr 172.21.15.43:0/4013999712 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:59.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.385+0000 7feef6800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b2a6585670 0x55b2a6583170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40258/0 (socket says 172.21.15.43:40258) 2024-03-28T22:44:59.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.385+0000 7feef5400640 1 -- 172.21.15.43:0/4013999712 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b2a6585670 msgr2=0x55b2a6583170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:59.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.385+0000 7feef5400640 1 --2- 172.21.15.43:0/4013999712 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b2a6585670 0x55b2a6583170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.385+0000 7feef5400640 1 -- 172.21.15.43:0/4013999712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2a657c170 msgr2=0x55b2a65826f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:59.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.385+0000 7feef5400640 1 --2- 172.21.15.43:0/4013999712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2a657c170 0x55b2a65826f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.385+0000 7feef5400640 1 -- 172.21.15.43:0/4013999712 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2a631cf70 con 0x55b2a657ca90 2024-03-28T22:44:59.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.385+0000 7feef6800640 1 --2- 172.21.15.43:0/4013999712 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b2a6585670 0x55b2a6583170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:44:59.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.385+0000 7feef5400640 1 --2- 172.21.15.43:0/4013999712 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b2a657ca90 0x55b2a6582c30 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7feee409a4b0 tx=0x7feee4096dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2859b1a08fd32695 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:59.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.385+0000 7feed7e00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:44:59.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.385+0000 7feed7e00640 1 -- 172.21.15.43:0/4013999712 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee409a060 con 0x55b2a657ca90 2024-03-28T22:44:59.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.385+0000 7feed7e00640 1 -- 172.21.15.43:0/4013999712 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7feee40cb500 con 0x55b2a657ca90 2024-03-28T22:44:59.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.386+0000 7feeffb83740 1 -- 172.21.15.43:0/4013999712 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feee409ac20 con 0x55b2a657ca90 2024-03-28T22:44:59.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.386+0000 7feed7e00640 1 -- 172.21.15.43:0/4013999712 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7feee40b3020 con 0x55b2a657ca90 2024-03-28T22:44:59.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.386+0000 7feeffb83740 10 client.28454 fetch_fsmap learned FSMap version 32 2024-03-28T22:44:59.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.386+0000 7feeffb83740 1 -- 172.21.15.43:0/4013999712 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b2a61b4170 con 0x55b2a657ca90 2024-03-28T22:44:59.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.386+0000 7feed7e00640 1 -- 172.21.15.43:0/4013999712 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feee409ac20 con 0x55b2a657ca90 2024-03-28T22:44:59.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.386+0000 7feed7e00640 1 -- 172.21.15.43:0/4013999712 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7feee409a780 con 0x55b2a657ca90 2024-03-28T22:44:59.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.386+0000 7feeffb83740 10 client.28454 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:44:59.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.386+0000 7feeffb83740 10 client.28454 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:44:59.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.386+0000 7feeffb83740 20 client.28454 populate_metadata read hostname 'smithi043' 2024-03-28T22:44:59.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.386+0000 7feeffb83740 1 --2- 172.21.15.43:0/4013999712 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b2a6595900 0x55b2a6597cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.386+0000 7feeffb83740 4 client.28454 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:44:59.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.386+0000 7feeffb83740 1 -- 172.21.15.43:0/4013999712 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b2a64ed180 con 0x55b2a6595900 2024-03-28T22:44:59.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.387+0000 7feef5e00640 1 --2- 172.21.15.43:0/4013999712 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b2a6595900 0x55b2a6597cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.387+0000 7feef5e00640 1 --2- 172.21.15.43:0/4013999712 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b2a6595900 0x55b2a6597cf0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5508075fb054177 server_cookie=f4af46046b8257ec in_seq=0 out_seq=0 2024-03-28T22:44:59.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.387+0000 7feed7e00640 10 client.28454 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:44:59.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.388+0000 7feed7e00640 1 -- 172.21.15.43:0/4013999712 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b2a64ed180 con 0x55b2a6595900 2024-03-28T22:44:59.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.388+0000 7feed7e00640 10 client.28454 handle_command_reply: tid=0 2024-03-28T22:44:59.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.388+0000 7feeffb83740 1 client.28454 shutdown 2024-03-28T22:44:59.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.388+0000 7feeffb83740 1 -- 172.21.15.43:0/4013999712 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b2a657ca90 msgr2=0x55b2a6582c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:59.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.388+0000 7feeffb83740 1 --2- 172.21.15.43:0/4013999712 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b2a657ca90 0x55b2a6582c30 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7feee409a4b0 tx=0x7feee4096dc0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.388+0000 7feeffb83740 1 -- 172.21.15.43:0/4013999712 shutdown_connections 2024-03-28T22:44:59.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.388+0000 7feeffb83740 1 --2- 172.21.15.43:0/4013999712 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b2a6595900 0x55b2a6597cf0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.388+0000 7feeffb83740 1 --2- 172.21.15.43:0/4013999712 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b2a657ca90 0x55b2a6582c30 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.388+0000 7feeffb83740 1 --2- 172.21.15.43:0/4013999712 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b2a6585670 0x55b2a6583170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.388+0000 7feeffb83740 1 --2- 172.21.15.43:0/4013999712 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2a657c170 0x55b2a65826f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.388+0000 7feed7e00640 0 client.28454 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:44:59.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.388+0000 7feeffb83740 1 -- 172.21.15.43:0/4013999712 >> 172.21.15.43:0/4013999712 conn(0x55b2a65768a0 msgr2=0x55b2a65785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:59.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.389+0000 7feeffb83740 1 -- 172.21.15.43:0/4013999712 shutdown_connections 2024-03-28T22:44:59.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.389+0000 7feeffb83740 1 -- 172.21.15.43:0/4013999712 wait complete. 2024-03-28T22:44:59.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.389+0000 7feeffb83740 20 client.28454 trim_cache size 0 max 16384 2024-03-28T22:44:59.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.408+0000 7feeffb83740 1 Processor -- start 2024-03-28T22:44:59.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.408+0000 7feeffb83740 1 -- start start 2024-03-28T22:44:59.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.408+0000 7feeffb83740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b2a657c170 0x55b2a6584db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.408+0000 7feeffb83740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b2a657ca90 0x55b2a659ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.408+0000 7feeffb83740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2a6585670 0x55b2a657f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.408+0000 7feeffb83740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7feef81b5140 con 0x55b2a6585670 2024-03-28T22:44:59.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.408+0000 7feeffb83740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7feef81b6c80 con 0x55b2a657c170 2024-03-28T22:44:59.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.408+0000 7feeffb83740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7feef81b5810 con 0x55b2a657ca90 2024-03-28T22:44:59.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.408+0000 7feef5400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b2a657ca90 0x55b2a659ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.408+0000 7feef5400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b2a657ca90 0x55b2a659ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40266/0 (socket says 172.21.15.43:40266) 2024-03-28T22:44:59.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.408+0000 7feef5400640 1 -- 172.21.15.43:0/1260240967 learned_addr learned my addr 172.21.15.43:0/1260240967 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:59.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.408+0000 7feef5e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b2a657c170 0x55b2a6584db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.409+0000 7feef6800640 1 --2- 172.21.15.43:0/1260240967 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2a6585670 0x55b2a657f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.409+0000 7feef5400640 1 -- 172.21.15.43:0/1260240967 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b2a657c170 msgr2=0x55b2a6584db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:59.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.409+0000 7feef5400640 1 --2- 172.21.15.43:0/1260240967 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b2a657c170 0x55b2a6584db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.409+0000 7feef5400640 1 -- 172.21.15.43:0/1260240967 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2a6585670 msgr2=0x55b2a657f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:59.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.409+0000 7feef5400640 1 --2- 172.21.15.43:0/1260240967 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2a6585670 0x55b2a657f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.409+0000 7feef5400640 1 -- 172.21.15.43:0/1260240967 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2a6567870 con 0x55b2a657ca90 2024-03-28T22:44:59.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.409+0000 7feef6800640 1 --2- 172.21.15.43:0/1260240967 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2a6585670 0x55b2a657f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:44:59.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.409+0000 7feef5e00640 1 --2- 172.21.15.43:0/1260240967 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b2a657c170 0x55b2a6584db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:44:59.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.409+0000 7feef5400640 1 --2- 172.21.15.43:0/1260240967 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b2a657ca90 0x55b2a659ad10 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7feee4009610 tx=0x7feee409a370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43f515310eb17f78 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:59.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.409+0000 7feee2a00640 1 -- 172.21.15.43:0/1260240967 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee409a060 con 0x55b2a657ca90 2024-03-28T22:44:59.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.409+0000 7feee2a00640 1 -- 172.21.15.43:0/1260240967 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feee409ac20 con 0x55b2a657ca90 2024-03-28T22:44:59.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.409+0000 7feee2a00640 1 -- 172.21.15.43:0/1260240967 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee40d18e0 con 0x55b2a657ca90 2024-03-28T22:44:59.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.410+0000 7feeffb83740 1 -- 172.21.15.43:0/1260240967 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b2a657ca90 msgr2=0x55b2a659ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:59.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.410+0000 7feeffb83740 1 --2- 172.21.15.43:0/1260240967 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b2a657ca90 0x55b2a659ad10 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7feee4009610 tx=0x7feee409a370 comp rx=0 tx=0).stop 2024-03-28T22:44:59.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.410+0000 7feeffb83740 1 -- 172.21.15.43:0/1260240967 shutdown_connections 2024-03-28T22:44:59.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.410+0000 7feeffb83740 1 --2- 172.21.15.43:0/1260240967 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2a6585670 0x55b2a657f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.410+0000 7feeffb83740 1 --2- 172.21.15.43:0/1260240967 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b2a657ca90 0x55b2a659ad10 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.410+0000 7feeffb83740 1 --2- 172.21.15.43:0/1260240967 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b2a657c170 0x55b2a6584db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.410+0000 7feeffb83740 1 -- 172.21.15.43:0/1260240967 >> 172.21.15.43:0/1260240967 conn(0x55b2a65768a0 msgr2=0x55b2a6575900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:59.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.410+0000 7feeffb83740 1 -- 172.21.15.43:0/1260240967 shutdown_connections 2024-03-28T22:44:59.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.410+0000 7feeffb83740 1 -- 172.21.15.43:0/1260240967 wait complete. 2024-03-28T22:44:59.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.410+0000 7feeffb83740 1 -- start start 2024-03-28T22:44:59.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.411+0000 7feeffb83740 1 Processor -- start 2024-03-28T22:44:59.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.411+0000 7feeffb83740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2a657c170 0x55b2a659aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.411+0000 7feeffb83740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b2a657ca90 0x55b2a6599260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.411+0000 7feeffb83740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b2a6585670 0x55b2a65997a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.411+0000 7feeffb83740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7feef81b65b0 con 0x55b2a657c170 2024-03-28T22:44:59.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.411+0000 7feeffb83740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7feef81b5ee0 con 0x55b2a657ca90 2024-03-28T22:44:59.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.411+0000 7feeffb83740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7feef81b3140 con 0x55b2a6585670 2024-03-28T22:44:59.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.411+0000 7feeffb83740 10 client.0 _ll_register_callbacks cb 0x7feee8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:44:59.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.411+0000 7feef5400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b2a657ca90 0x55b2a6599260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.411+0000 7feef5e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2a657c170 0x55b2a659aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.411+0000 7feef5400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b2a657ca90 0x55b2a6599260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47396/0 (socket says 172.21.15.43:47396) 2024-03-28T22:44:59.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.411+0000 7feef5400640 1 -- 172.21.15.43:0/767429084 learned_addr learned my addr 172.21.15.43:0/767429084 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:59.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.411+0000 7feef6800640 1 --2- 172.21.15.43:0/767429084 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b2a6585670 0x55b2a65997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.412+0000 7feef5400640 1 -- 172.21.15.43:0/767429084 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b2a6585670 msgr2=0x55b2a65997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:59.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.412+0000 7feef5400640 1 --2- 172.21.15.43:0/767429084 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b2a6585670 0x55b2a65997a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.412+0000 7feef5400640 1 -- 172.21.15.43:0/767429084 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2a657c170 msgr2=0x55b2a659aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:59.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.412+0000 7feef5400640 1 --2- 172.21.15.43:0/767429084 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2a657c170 0x55b2a659aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.412+0000 7feef5400640 1 -- 172.21.15.43:0/767429084 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2a6599ee0 con 0x55b2a657ca90 2024-03-28T22:44:59.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.412+0000 7feef5400640 1 --2- 172.21.15.43:0/767429084 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b2a657ca90 0x55b2a6599260 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7feee4096ef0 tx=0x7feee40cb360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1ab1798288b017e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:59.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.412+0000 7feed7e00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:44:59.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.412+0000 7feed7e00640 1 -- 172.21.15.43:0/767429084 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee4099e80 con 0x55b2a657ca90 2024-03-28T22:44:59.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.412+0000 7feed7e00640 1 -- 172.21.15.43:0/767429084 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7feee40cb500 con 0x55b2a657ca90 2024-03-28T22:44:59.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.412+0000 7feeffb83740 1 -- 172.21.15.43:0/767429084 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feee409ac20 con 0x55b2a657ca90 2024-03-28T22:44:59.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.412+0000 7feed7e00640 1 -- 172.21.15.43:0/767429084 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7feee409e6c0 con 0x55b2a657ca90 2024-03-28T22:44:59.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.412+0000 7feeffb83740 10 client.28460 fetch_fsmap learned FSMap version 32 2024-03-28T22:44:59.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.412+0000 7feeffb83740 1 -- 172.21.15.43:0/767429084 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b2a65fc860 con 0x55b2a657ca90 2024-03-28T22:44:59.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.413+0000 7feed7e00640 1 -- 172.21.15.43:0/767429084 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feee409ac20 con 0x55b2a657ca90 2024-03-28T22:44:59.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.413+0000 7feed7e00640 1 -- 172.21.15.43:0/767429084 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7feee40a2830 con 0x55b2a657ca90 2024-03-28T22:44:59.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.413+0000 7feeffb83740 10 client.28460 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:44:59.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.413+0000 7feeffb83740 10 client.28460 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:44:59.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.413+0000 7feeffb83740 20 client.28460 populate_metadata read hostname 'smithi043' 2024-03-28T22:44:59.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.413+0000 7feeffb83740 1 --2- 172.21.15.43:0/767429084 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b2a65fcda0 0x55b2a65ff1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.413+0000 7feeffb83740 4 client.28460 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:44:59.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.413+0000 7feeffb83740 1 -- 172.21.15.43:0/767429084 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b2a64e74d0 con 0x55b2a65fcda0 2024-03-28T22:44:59.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.413+0000 7feef5e00640 1 --2- 172.21.15.43:0/767429084 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b2a65fcda0 0x55b2a65ff1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.414+0000 7feef5e00640 1 --2- 172.21.15.43:0/767429084 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b2a65fcda0 0x55b2a65ff1c0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99a88ea12bb84858 server_cookie=b5a9a7fdd40450f4 in_seq=0 out_seq=0 2024-03-28T22:44:59.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.414+0000 7feed7e00640 10 client.28460 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:44:59.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.415+0000 7feed7e00640 1 -- 172.21.15.43:0/767429084 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b2a64e74d0 con 0x55b2a65fcda0 2024-03-28T22:44:59.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.415+0000 7feed7e00640 10 client.28460 handle_command_reply: tid=0 2024-03-28T22:44:59.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.415+0000 7feeffb83740 1 client.28460 shutdown 2024-03-28T22:44:59.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.415+0000 7feeffb83740 1 -- 172.21.15.43:0/767429084 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b2a657ca90 msgr2=0x55b2a6599260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:59.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.415+0000 7feeffb83740 1 --2- 172.21.15.43:0/767429084 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b2a657ca90 0x55b2a6599260 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7feee4096ef0 tx=0x7feee40cb360 comp rx=0 tx=0).stop 2024-03-28T22:44:59.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.415+0000 7feeffb83740 1 -- 172.21.15.43:0/767429084 shutdown_connections 2024-03-28T22:44:59.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.415+0000 7feeffb83740 1 --2- 172.21.15.43:0/767429084 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b2a65fcda0 0x55b2a65ff1c0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.415+0000 7feeffb83740 1 --2- 172.21.15.43:0/767429084 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b2a657ca90 0x55b2a6599260 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.415+0000 7feeffb83740 1 --2- 172.21.15.43:0/767429084 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b2a6585670 0x55b2a65997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.415+0000 7feeffb83740 1 --2- 172.21.15.43:0/767429084 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b2a657c170 0x55b2a659aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.415+0000 7feed7e00640 0 client.28460 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:44:59.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.415+0000 7feeffb83740 1 -- 172.21.15.43:0/767429084 >> 172.21.15.43:0/767429084 conn(0x55b2a65768a0 msgr2=0x55b2a6582690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:59.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.415+0000 7feeffb83740 1 -- 172.21.15.43:0/767429084 shutdown_connections 2024-03-28T22:44:59.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.415+0000 7feeffb83740 1 -- 172.21.15.43:0/767429084 wait complete. 2024-03-28T22:44:59.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.416+0000 7feeffb83740 20 client.28460 trim_cache size 0 max 16384 2024-03-28T22:44:59.424 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:44:59.424 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-03-28T22:44:59.424 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "6d472f11-6ed4-405f-82bb-c5b3adc62474", 2024-03-28T22:44:59.425 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-03-28T22:44:59.425 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:44:59.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.416+0000 7feef7200640 1 -- 172.21.15.43:0/928049481 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7feedc0a1f70 msgr2=0x7feedc0a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:59.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.416+0000 7feef7200640 1 --2- 172.21.15.43:0/928049481 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7feedc0a1f70 0x7feedc0a4430 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7feef806ab80 tx=0x7feef0009500 comp rx=0 tx=0).stop 2024-03-28T22:44:59.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.416+0000 7feef7200640 1 -- 172.21.15.43:0/928049481 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feef80765a0 msgr2=0x7feef81a4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:59.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.416+0000 7feef7200640 1 --2- 172.21.15.43:0/928049481 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feef80765a0 0x7feef81a4b30 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7feee400d950 tx=0x7feee400de10 comp rx=0 tx=0).stop 2024-03-28T22:44:59.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.416+0000 7feef7200640 1 -- 172.21.15.43:0/928049481 shutdown_connections 2024-03-28T22:44:59.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.416+0000 7feef7200640 1 --2- 172.21.15.43:0/928049481 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7feedc0a1f70 0x7feedc0a4430 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.416+0000 7feef7200640 1 --2- 172.21.15.43:0/928049481 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feef80765a0 0x7feef81a4b30 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.416+0000 7feef7200640 1 --2- 172.21.15.43:0/928049481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feef8112470 0x7feef819e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.416+0000 7feef7200640 1 --2- 172.21.15.43:0/928049481 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feef8075c80 0x7feef81a45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.416+0000 7feef7200640 1 -- 172.21.15.43:0/928049481 >> 172.21.15.43:0/928049481 conn(0x7feef80ff260 msgr2=0x7feef8101650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:59.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.416+0000 7feef7200640 1 -- 172.21.15.43:0/928049481 shutdown_connections 2024-03-28T22:44:59.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.416+0000 7feef7200640 1 -- 172.21.15.43:0/928049481 wait complete. 2024-03-28T22:44:59.437 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:44:59.438 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:44:59.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.664+0000 7f1111c00640 1 -- 172.21.15.43:0/2829914977 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f110c1030a0 msgr2=0x7f110c10b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:59.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.664+0000 7f1111c00640 1 --2- 172.21.15.43:0/2829914977 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f110c1030a0 0x7f110c10b580 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f10f8009900 tx=0x7f10f802bcc0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.664+0000 7f1111c00640 1 -- 172.21.15.43:0/2829914977 shutdown_connections 2024-03-28T22:44:59.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.664+0000 7f1111c00640 1 --2- 172.21.15.43:0/2829914977 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f110c1030a0 0x7f110c10b580 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.664+0000 7f1111c00640 1 --2- 172.21.15.43:0/2829914977 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f110c10bba0 0x7f110c10e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.664+0000 7f1111c00640 1 --2- 172.21.15.43:0/2829914977 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f110c102780 0x7f110c102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.664+0000 7f1111c00640 1 -- 172.21.15.43:0/2829914977 >> 172.21.15.43:0/2829914977 conn(0x7f110c0fbfa0 msgr2=0x7f110c0fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:59.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.665+0000 7f1111c00640 1 -- 172.21.15.43:0/2829914977 shutdown_connections 2024-03-28T22:44:59.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.665+0000 7f1111c00640 1 -- 172.21.15.43:0/2829914977 wait complete. 2024-03-28T22:44:59.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.666+0000 7f1111c00640 1 Processor -- start 2024-03-28T22:44:59.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.666+0000 7f1111c00640 1 -- start start 2024-03-28T22:44:59.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.666+0000 7f1111c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f110c102780 0x7f110c197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.666+0000 7f1111c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f110c1030a0 0x7f110c1983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.666+0000 7f1111c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f110c10bba0 0x7f110c19c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.667+0000 7f1111c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f110c1119f0 con 0x7f110c102780 2024-03-28T22:44:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.667+0000 7f1111c00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f110c111870 con 0x7f110c10bba0 2024-03-28T22:44:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.667+0000 7f1111c00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f110c111b70 con 0x7f110c1030a0 2024-03-28T22:44:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.667+0000 7f1111200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f110c10bba0 0x7f110c19c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.667+0000 7f110b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f110c1030a0 0x7f110c1983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.667+0000 7f1111200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f110c10bba0 0x7f110c19c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47418/0 (socket says 172.21.15.43:47418) 2024-03-28T22:44:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.667+0000 7f1111200640 1 -- 172.21.15.43:0/3584099745 learned_addr learned my addr 172.21.15.43:0/3584099745 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.667+0000 7f110be00640 1 --2- 172.21.15.43:0/3584099745 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f110c102780 0x7f110c197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.667+0000 7f110b400640 1 -- 172.21.15.43:0/3584099745 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f110c10bba0 msgr2=0x7f110c19c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.667+0000 7f110b400640 1 --2- 172.21.15.43:0/3584099745 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f110c10bba0 0x7f110c19c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.667+0000 7f110b400640 1 -- 172.21.15.43:0/3584099745 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f110c102780 msgr2=0x7f110c197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.667+0000 7f110b400640 1 --2- 172.21.15.43:0/3584099745 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f110c102780 0x7f110c197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.667+0000 7f110b400640 1 -- 172.21.15.43:0/3584099745 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f110c19ce80 con 0x7f110c1030a0 2024-03-28T22:44:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.667+0000 7f110be00640 1 --2- 172.21.15.43:0/3584099745 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f110c102780 0x7f110c197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:44:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.668+0000 7f110b400640 1 --2- 172.21.15.43:0/3584099745 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f110c1030a0 0x7f110c1983d0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f10f802c200 tx=0x7f10f8005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:59.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.668+0000 7f1108c00640 1 -- 172.21.15.43:0/3584099745 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f8042020 con 0x7f110c1030a0 2024-03-28T22:44:59.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.668+0000 7f1108c00640 1 -- 172.21.15.43:0/3584099745 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10f802cdf0 con 0x7f110c1030a0 2024-03-28T22:44:59.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.668+0000 7f1108c00640 1 -- 172.21.15.43:0/3584099745 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f8003a50 con 0x7f110c1030a0 2024-03-28T22:44:59.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.668+0000 7f1111c00640 1 -- 172.21.15.43:0/3584099745 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f110c19d110 con 0x7f110c1030a0 2024-03-28T22:44:59.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.668+0000 7f1111c00640 1 -- 172.21.15.43:0/3584099745 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f110c1a4a00 con 0x7f110c1030a0 2024-03-28T22:44:59.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.672+0000 7f1108c00640 1 -- 172.21.15.43:0/3584099745 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f10f8045020 con 0x7f110c1030a0 2024-03-28T22:44:59.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.672+0000 7f1116017740 1 Processor -- start 2024-03-28T22:44:59.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.672+0000 7f1116017740 1 -- start start 2024-03-28T22:44:59.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.672+0000 7f1116017740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ff6e74db0 0x563ff6e73b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.672+0000 7f1116017740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ff6e740d0 0x563ff6e74550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.672+0000 7f1116017740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ff6e7d440 0x563ff6e7f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.672+0000 7f1116017740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x563ff6dd8100 con 0x563ff6e74db0 2024-03-28T22:44:59.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.672+0000 7f1116017740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x563ff6d80ba0 con 0x563ff6e7d440 2024-03-28T22:44:59.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.672+0000 7f1116017740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x563ff6dd41a0 con 0x563ff6e740d0 2024-03-28T22:44:59.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.672+0000 7f1111200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ff6e74db0 0x563ff6e73b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.672+0000 7f110be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ff6e740d0 0x563ff6e74550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.672+0000 7f110b400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ff6e7d440 0x563ff6e7f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.672+0000 7f110be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ff6e740d0 0x563ff6e74550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40310/0 (socket says 172.21.15.43:40310) 2024-03-28T22:44:59.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.672+0000 7f110be00640 1 -- 172.21.15.43:0/70782086 learned_addr learned my addr 172.21.15.43:0/70782086 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:59.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.672+0000 7f1111200640 1 --2- 172.21.15.43:0/70782086 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ff6e74db0 0x563ff6e73b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45404/0 (socket says 172.21.15.43:45404) 2024-03-28T22:44:59.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.673+0000 7f110be00640 1 -- 172.21.15.43:0/70782086 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ff6e7d440 msgr2=0x563ff6e7f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:59.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.673+0000 7f110be00640 1 --2- 172.21.15.43:0/70782086 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ff6e7d440 0x563ff6e7f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.673+0000 7f110be00640 1 -- 172.21.15.43:0/70782086 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ff6e74db0 msgr2=0x563ff6e73b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:59.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.673+0000 7f110be00640 1 --2- 172.21.15.43:0/70782086 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ff6e74db0 0x563ff6e73b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.673+0000 7f110be00640 1 -- 172.21.15.43:0/70782086 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ff6db11a0 con 0x563ff6e740d0 2024-03-28T22:44:59.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.673+0000 7f110be00640 1 --2- 172.21.15.43:0/70782086 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ff6e740d0 0x563ff6e74550 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f10fc0104a0 tx=0x7f10fc0108a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2ce737294c45f52 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:59.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.673+0000 7f1108c00640 1 --2- 172.21.15.43:0/3584099745 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f10f00a1f70 0x7f10f00a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.673+0000 7f1101600640 1 -- 172.21.15.43:0/70782086 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10fc011760 con 0x563ff6e740d0 2024-03-28T22:44:59.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.673+0000 7f1101600640 1 -- 172.21.15.43:0/70782086 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10fc011de0 con 0x563ff6e740d0 2024-03-28T22:44:59.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.673+0000 7f1101600640 1 -- 172.21.15.43:0/70782086 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10fc00e3c0 con 0x563ff6e740d0 2024-03-28T22:44:59.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.674+0000 7f1108c00640 1 -- 172.21.15.43:0/3584099745 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f10f80eff80 con 0x7f110c1030a0 2024-03-28T22:44:59.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.674+0000 7f1111200640 1 --2- 172.21.15.43:0/3584099745 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f10f00a1f70 0x7f10f00a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.674+0000 7f1116017740 1 -- 172.21.15.43:0/70782086 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ff6e740d0 msgr2=0x563ff6e74550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:59.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.674+0000 7f1116017740 1 --2- 172.21.15.43:0/70782086 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ff6e740d0 0x563ff6e74550 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f10fc0104a0 tx=0x7f10fc0108a0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.674+0000 7f1116017740 1 -- 172.21.15.43:0/70782086 shutdown_connections 2024-03-28T22:44:59.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.674+0000 7f1116017740 1 --2- 172.21.15.43:0/70782086 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ff6e7d440 0x563ff6e7f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.674+0000 7f1116017740 1 --2- 172.21.15.43:0/70782086 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ff6e740d0 0x563ff6e74550 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.674+0000 7f1116017740 1 --2- 172.21.15.43:0/70782086 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ff6e74db0 0x563ff6e73b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.675+0000 7f1116017740 1 -- 172.21.15.43:0/70782086 >> 172.21.15.43:0/70782086 conn(0x563ff6e3aa10 msgr2=0x563ff6e6eb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:59.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.675+0000 7f1111200640 1 --2- 172.21.15.43:0/3584099745 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f10f00a1f70 0x7f10f00a4430 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f110400bd40 tx=0x7f110400b390 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:59.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.675+0000 7f1116017740 1 -- 172.21.15.43:0/70782086 shutdown_connections 2024-03-28T22:44:59.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.675+0000 7f1116017740 1 -- 172.21.15.43:0/70782086 wait complete. 2024-03-28T22:44:59.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.675+0000 7f1116017740 1 -- start start 2024-03-28T22:44:59.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.675+0000 7f1116017740 1 Processor -- start 2024-03-28T22:44:59.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.676+0000 7f1116017740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ff6e740d0 0x563ff6e7a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.676+0000 7f1116017740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ff6e74db0 0x563ff6e7a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.676+0000 7f1116017740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ff6e7d440 0x563ff6e7aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.676+0000 7f1116017740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x563ff6e802b0 con 0x563ff6e74db0 2024-03-28T22:44:59.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.676+0000 7f1116017740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x563ff6d76250 con 0x563ff6e7d440 2024-03-28T22:44:59.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.676+0000 7f1116017740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x563ff6e80430 con 0x563ff6e740d0 2024-03-28T22:44:59.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.676+0000 7f1116017740 10 client.0 _ll_register_callbacks cb 0x563ff6aebbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:44:59.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.676+0000 7f110be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ff6e740d0 0x563ff6e7a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.676+0000 7f1111200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ff6e7d440 0x563ff6e7aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.676+0000 7f110be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ff6e740d0 0x563ff6e7a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40322/0 (socket says 172.21.15.43:40322) 2024-03-28T22:44:59.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.676+0000 7f110be00640 1 -- 172.21.15.43:0/3763796653 learned_addr learned my addr 172.21.15.43:0/3763796653 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:59.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.676+0000 7f110b400640 1 --2- 172.21.15.43:0/3763796653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ff6e74db0 0x563ff6e7a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.677+0000 7f1111200640 1 -- 172.21.15.43:0/3763796653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ff6e740d0 msgr2=0x563ff6e7a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:59.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.677+0000 7f1111200640 1 --2- 172.21.15.43:0/3763796653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ff6e740d0 0x563ff6e7a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.677+0000 7f1111200640 1 -- 172.21.15.43:0/3763796653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ff6e74db0 msgr2=0x563ff6e7a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:59.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.677+0000 7f1111200640 1 --2- 172.21.15.43:0/3763796653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ff6e74db0 0x563ff6e7a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.677+0000 7f1111200640 1 -- 172.21.15.43:0/3763796653 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ff6d92960 con 0x563ff6e7d440 2024-03-28T22:44:59.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.677+0000 7f110be00640 1 --2- 172.21.15.43:0/3763796653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ff6e740d0 0x563ff6e7a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:44:59.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.677+0000 7f1111200640 1 --2- 172.21.15.43:0/3763796653 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ff6e7d440 0x563ff6e7aeb0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f110401f4d0 tx=0x7f1104004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=159d3a8034630278 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:59.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.677+0000 7f10f6000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:44:59.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.677+0000 7f10f6000640 1 -- 172.21.15.43:0/3763796653 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110402c060 con 0x563ff6e7d440 2024-03-28T22:44:59.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.677+0000 7f1116017740 1 -- 172.21.15.43:0/3763796653 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10fc011de0 con 0x563ff6e7d440 2024-03-28T22:44:59.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.677+0000 7f10f6000640 1 -- 172.21.15.43:0/3763796653 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f1104017030 con 0x563ff6e7d440 2024-03-28T22:44:59.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.677+0000 7f10f6000640 1 -- 172.21.15.43:0/3763796653 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1104033050 con 0x563ff6e7d440 2024-03-28T22:44:59.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.677+0000 7f1116017740 10 client.28472 fetch_fsmap learned FSMap version 32 2024-03-28T22:44:59.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.677+0000 7f1116017740 1 -- 172.21.15.43:0/3763796653 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563ff6bb67d0 con 0x563ff6e7d440 2024-03-28T22:44:59.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.678+0000 7f10f6000640 1 -- 172.21.15.43:0/3763796653 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10fc011de0 con 0x563ff6e7d440 2024-03-28T22:44:59.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.678+0000 7f10f6000640 1 -- 172.21.15.43:0/3763796653 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f1104027970 con 0x563ff6e7d440 2024-03-28T22:44:59.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.678+0000 7f1116017740 10 client.28472 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:44:59.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.678+0000 7f1116017740 10 client.28472 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:44:59.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.678+0000 7f1116017740 20 client.28472 populate_metadata read hostname 'smithi043' 2024-03-28T22:44:59.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.678+0000 7f1116017740 1 --2- 172.21.15.43:0/3763796653 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x563ff6e8d8d0 0x563ff6e8fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.678+0000 7f1116017740 4 client.28472 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:44:59.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.678+0000 7f1116017740 1 -- 172.21.15.43:0/3763796653 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ff6de5640 con 0x563ff6e8d8d0 2024-03-28T22:44:59.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.678+0000 7f110be00640 1 --2- 172.21.15.43:0/3763796653 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x563ff6e8d8d0 0x563ff6e8fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.679+0000 7f110be00640 1 --2- 172.21.15.43:0/3763796653 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x563ff6e8d8d0 0x563ff6e8fd20 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4936b95a2ae66b9 server_cookie=9c62ad8db3584101 in_seq=0 out_seq=0 2024-03-28T22:44:59.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.679+0000 7f10f6000640 10 client.28472 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:44:59.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.679+0000 7f10f6000640 1 -- 172.21.15.43:0/3763796653 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563ff6de5640 con 0x563ff6e8d8d0 2024-03-28T22:44:59.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.679+0000 7f10f6000640 10 client.28472 handle_command_reply: tid=0 2024-03-28T22:44:59.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.679+0000 7f1116017740 1 client.28472 shutdown 2024-03-28T22:44:59.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.680+0000 7f1116017740 1 -- 172.21.15.43:0/3763796653 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ff6e7d440 msgr2=0x563ff6e7aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:59.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.680+0000 7f1116017740 1 --2- 172.21.15.43:0/3763796653 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ff6e7d440 0x563ff6e7aeb0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f110401f4d0 tx=0x7f1104004730 comp rx=0 tx=0).stop 2024-03-28T22:44:59.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.680+0000 7f1116017740 1 -- 172.21.15.43:0/3763796653 shutdown_connections 2024-03-28T22:44:59.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.680+0000 7f1116017740 1 --2- 172.21.15.43:0/3763796653 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x563ff6e8d8d0 0x563ff6e8fd20 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.680+0000 7f1116017740 1 --2- 172.21.15.43:0/3763796653 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ff6e7d440 0x563ff6e7aeb0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.680+0000 7f1116017740 1 --2- 172.21.15.43:0/3763796653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ff6e74db0 0x563ff6e7a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.680+0000 7f10f6000640 0 client.28472 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:44:59.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.680+0000 7f1116017740 1 --2- 172.21.15.43:0/3763796653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ff6e740d0 0x563ff6e7a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.680+0000 7f1116017740 1 -- 172.21.15.43:0/3763796653 >> 172.21.15.43:0/3763796653 conn(0x563ff6e3aa10 msgr2=0x563ff6e70870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:59.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.680+0000 7f1116017740 1 -- 172.21.15.43:0/3763796653 shutdown_connections 2024-03-28T22:44:59.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.680+0000 7f1116017740 1 -- 172.21.15.43:0/3763796653 wait complete. 2024-03-28T22:44:59.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.680+0000 7f1116017740 20 client.28472 trim_cache size 0 max 16384 2024-03-28T22:44:59.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.699+0000 7f1116017740 1 Processor -- start 2024-03-28T22:44:59.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.699+0000 7f1116017740 1 -- start start 2024-03-28T22:44:59.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.700+0000 7f1116017740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ff6e740d0 0x563ff6e7cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.700+0000 7f1116017740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ff6e74db0 0x563ff6e89090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.700+0000 7f1116017740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ff6e7d440 0x563ff6e70680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.700+0000 7f1116017740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f110c1aea20 con 0x563ff6e740d0 2024-03-28T22:44:59.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.700+0000 7f1116017740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f110c1b0560 con 0x563ff6e7d440 2024-03-28T22:44:59.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.700+0000 7f1116017740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f110c1af0f0 con 0x563ff6e74db0 2024-03-28T22:44:59.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.700+0000 7f110b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ff6e74db0 0x563ff6e89090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.700+0000 7f110be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ff6e740d0 0x563ff6e7cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.700+0000 7f110b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ff6e74db0 0x563ff6e89090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40324/0 (socket says 172.21.15.43:40324) 2024-03-28T22:44:59.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.700+0000 7f1111200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ff6e7d440 0x563ff6e70680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.700+0000 7f110b400640 1 -- 172.21.15.43:0/810493445 learned_addr learned my addr 172.21.15.43:0/810493445 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:59.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.700+0000 7f110b400640 1 -- 172.21.15.43:0/810493445 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ff6e7d440 msgr2=0x563ff6e70680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:59.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.700+0000 7f110b400640 1 --2- 172.21.15.43:0/810493445 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ff6e7d440 0x563ff6e70680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.700+0000 7f110b400640 1 -- 172.21.15.43:0/810493445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ff6e740d0 msgr2=0x563ff6e7cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:59.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.700+0000 7f110b400640 1 --2- 172.21.15.43:0/810493445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ff6e740d0 0x563ff6e7cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.700+0000 7f110b400640 1 -- 172.21.15.43:0/810493445 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ff6aec3f0 con 0x563ff6e74db0 2024-03-28T22:44:59.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.700+0000 7f110be00640 1 --2- 172.21.15.43:0/810493445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ff6e740d0 0x563ff6e7cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:44:59.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.701+0000 7f110b400640 1 --2- 172.21.15.43:0/810493445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ff6e74db0 0x563ff6e89090 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f10f80bc350 tx=0x7f10f80f3a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ab893a9eceb6cac server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:59.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.701+0000 7f1100c00640 1 -- 172.21.15.43:0/810493445 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f80b88f0 con 0x563ff6e74db0 2024-03-28T22:44:59.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.701+0000 7f1100c00640 1 -- 172.21.15.43:0/810493445 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10f80bcd50 con 0x563ff6e74db0 2024-03-28T22:44:59.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.701+0000 7f1100c00640 1 -- 172.21.15.43:0/810493445 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f80bbfb0 con 0x563ff6e74db0 2024-03-28T22:44:59.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.701+0000 7f1116017740 1 -- 172.21.15.43:0/810493445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ff6e74db0 msgr2=0x563ff6e89090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:59.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.701+0000 7f1116017740 1 --2- 172.21.15.43:0/810493445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ff6e74db0 0x563ff6e89090 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f10f80bc350 tx=0x7f10f80f3a30 comp rx=0 tx=0).stop 2024-03-28T22:44:59.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.702+0000 7f1116017740 1 -- 172.21.15.43:0/810493445 shutdown_connections 2024-03-28T22:44:59.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.702+0000 7f1116017740 1 --2- 172.21.15.43:0/810493445 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ff6e7d440 0x563ff6e70680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.702+0000 7f1116017740 1 --2- 172.21.15.43:0/810493445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ff6e74db0 0x563ff6e89090 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.702+0000 7f1116017740 1 --2- 172.21.15.43:0/810493445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ff6e740d0 0x563ff6e7cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.702+0000 7f1116017740 1 -- 172.21.15.43:0/810493445 >> 172.21.15.43:0/810493445 conn(0x563ff6e3aa10 msgr2=0x563ff6e6ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:59.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.702+0000 7f1116017740 1 -- 172.21.15.43:0/810493445 shutdown_connections 2024-03-28T22:44:59.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.702+0000 7f1116017740 1 -- 172.21.15.43:0/810493445 wait complete. 2024-03-28T22:44:59.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.702+0000 7f1116017740 1 -- start start 2024-03-28T22:44:59.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.702+0000 7f1116017740 1 Processor -- start 2024-03-28T22:44:59.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.703+0000 7f1116017740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ff6e740d0 0x563ff6e85af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.703+0000 7f1116017740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ff6e74db0 0x563ff6e7de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.703+0000 7f1116017740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ff6e7d440 0x563ff6e7e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.703+0000 7f1116017740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f110c1afe90 con 0x563ff6e74db0 2024-03-28T22:44:59.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.703+0000 7f1116017740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f110c1af7c0 con 0x563ff6e7d440 2024-03-28T22:44:59.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.703+0000 7f1116017740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f110c1aca20 con 0x563ff6e740d0 2024-03-28T22:44:59.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.703+0000 7f1116017740 10 client.0 _ll_register_callbacks cb 0x7f110400bf30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:44:59.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.703+0000 7f110be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ff6e740d0 0x563ff6e85af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.703+0000 7f110b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ff6e74db0 0x563ff6e7de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.703+0000 7f1111200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ff6e7d440 0x563ff6e7e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.703+0000 7f110b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ff6e74db0 0x563ff6e7de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45430/0 (socket says 172.21.15.43:45430) 2024-03-28T22:44:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.703+0000 7f110be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ff6e740d0 0x563ff6e85af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40332/0 (socket says 172.21.15.43:40332) 2024-03-28T22:44:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.703+0000 7f110b400640 1 -- 172.21.15.43:0/2389194765 learned_addr learned my addr 172.21.15.43:0/2389194765 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:44:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.703+0000 7f1111200640 1 -- 172.21.15.43:0/2389194765 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ff6e740d0 msgr2=0x563ff6e85af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.703+0000 7f1111200640 1 --2- 172.21.15.43:0/2389194765 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ff6e740d0 0x563ff6e85af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.703+0000 7f1111200640 1 -- 172.21.15.43:0/2389194765 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ff6e74db0 msgr2=0x563ff6e7de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:44:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.703+0000 7f1111200640 1 --2- 172.21.15.43:0/2389194765 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ff6e74db0 0x563ff6e7de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.703+0000 7f1111200640 1 -- 172.21.15.43:0/2389194765 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ff6aac0f0 con 0x563ff6e7d440 2024-03-28T22:44:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.704+0000 7f1111200640 1 --2- 172.21.15.43:0/2389194765 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ff6e7d440 0x563ff6e7e560 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f1104027f30 tx=0x7f1104011da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f305b282aa022ca3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:44:59.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.704+0000 7f10f6000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:44:59.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.704+0000 7f10f6000640 1 -- 172.21.15.43:0/2389194765 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110403b020 con 0x563ff6e7d440 2024-03-28T22:44:59.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.704+0000 7f1116017740 1 -- 172.21.15.43:0/2389194765 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10f80bcd50 con 0x563ff6e7d440 2024-03-28T22:44:59.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.704+0000 7f10f6000640 1 -- 172.21.15.43:0/2389194765 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9495+0+0 (secure 0 0 0) 0x7f11040273e0 con 0x563ff6e7d440 2024-03-28T22:44:59.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.704+0000 7f10f6000640 1 -- 172.21.15.43:0/2389194765 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f110401cb90 con 0x563ff6e7d440 2024-03-28T22:44:59.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.704+0000 7f1116017740 10 client.28478 fetch_fsmap learned FSMap version 32 2024-03-28T22:44:59.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.704+0000 7f1116017740 1 -- 172.21.15.43:0/2389194765 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563ff6ef3c20 con 0x563ff6e7d440 2024-03-28T22:44:59.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.705+0000 7f10f6000640 1 -- 172.21.15.43:0/2389194765 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10f80bcd50 con 0x563ff6e7d440 2024-03-28T22:44:59.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.705+0000 7f10f6000640 1 -- 172.21.15.43:0/2389194765 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f1104037810 con 0x563ff6e7d440 2024-03-28T22:44:59.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.705+0000 7f1116017740 10 client.28478 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:44:59.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.705+0000 7f1116017740 10 client.28478 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:44:59.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.705+0000 7f1116017740 20 client.28478 populate_metadata read hostname 'smithi043' 2024-03-28T22:44:59.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.705+0000 7f1116017740 1 --2- 172.21.15.43:0/2389194765 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x563ff6e8d8d0 0x563ff6ef6360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:44:59.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.705+0000 7f1116017740 4 client.28478 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:44:59.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.705+0000 7f1116017740 1 -- 172.21.15.43:0/2389194765 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563ff6ddf990 con 0x563ff6e8d8d0 2024-03-28T22:44:59.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.705+0000 7f110be00640 1 --2- 172.21.15.43:0/2389194765 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x563ff6e8d8d0 0x563ff6ef6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:44:59.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.706+0000 7f110be00640 1 --2- 172.21.15.43:0/2389194765 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x563ff6e8d8d0 0x563ff6ef6360 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8bfd67902340fac3 server_cookie=756fccc2281a36c8 in_seq=0 out_seq=0 2024-03-28T22:44:59.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.706+0000 7f10f6000640 10 client.28478 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:44:59.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.706+0000 7f10f6000640 1 -- 172.21.15.43:0/2389194765 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x563ff6ddf990 con 0x563ff6e8d8d0 2024-03-28T22:44:59.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.706+0000 7f10f6000640 10 client.28478 handle_command_reply: tid=0 2024-03-28T22:44:59.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.706+0000 7f1116017740 1 client.28478 shutdown 2024-03-28T22:44:59.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.706+0000 7f1116017740 1 -- 172.21.15.43:0/2389194765 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ff6e7d440 msgr2=0x563ff6e7e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:59.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.706+0000 7f1116017740 1 --2- 172.21.15.43:0/2389194765 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ff6e7d440 0x563ff6e7e560 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f1104027f30 tx=0x7f1104011da0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.706+0000 7f1116017740 1 -- 172.21.15.43:0/2389194765 shutdown_connections 2024-03-28T22:44:59.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.706+0000 7f1116017740 1 --2- 172.21.15.43:0/2389194765 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x563ff6e8d8d0 0x563ff6ef6360 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.706+0000 7f1116017740 1 --2- 172.21.15.43:0/2389194765 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ff6e7d440 0x563ff6e7e560 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.706+0000 7f1116017740 1 --2- 172.21.15.43:0/2389194765 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ff6e74db0 0x563ff6e7de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.706+0000 7f10f6000640 0 client.28478 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:44:59.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.706+0000 7f1116017740 1 --2- 172.21.15.43:0/2389194765 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ff6e740d0 0x563ff6e85af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.706+0000 7f1116017740 1 -- 172.21.15.43:0/2389194765 >> 172.21.15.43:0/2389194765 conn(0x563ff6e3aa10 msgr2=0x563ff6e6d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:59.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.706+0000 7f1116017740 1 -- 172.21.15.43:0/2389194765 shutdown_connections 2024-03-28T22:44:59.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.706+0000 7f1116017740 1 -- 172.21.15.43:0/2389194765 wait complete. 2024-03-28T22:44:59.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.706+0000 7f1116017740 20 client.28478 trim_cache size 0 max 16384 2024-03-28T22:44:59.714 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:44:59.714 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:44:59.714 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-03-28T22:44:59.714 INFO:teuthology.orchestra.run.smithi043.stdout: "6d472f11-6ed4-405f-82bb-c5b3adc62474": { 2024-03-28T22:44:59.714 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-03-28T22:44:59.714 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "6d472f11-6ed4-405f-82bb-c5b3adc62474", 2024-03-28T22:44:59.715 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-03-28T22:44:59.715 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:44:59.715 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:44:59.715 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:44:59.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.707+0000 7f1111c00640 1 -- 172.21.15.43:0/3584099745 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f10f00a1f70 msgr2=0x7f10f00a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:59.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.707+0000 7f1111c00640 1 --2- 172.21.15.43:0/3584099745 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f10f00a1f70 0x7f10f00a4430 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f110400bd40 tx=0x7f110400b390 comp rx=0 tx=0).stop 2024-03-28T22:44:59.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.707+0000 7f1111c00640 1 -- 172.21.15.43:0/3584099745 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f110c1030a0 msgr2=0x7f110c1983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:44:59.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.707+0000 7f1111c00640 1 --2- 172.21.15.43:0/3584099745 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f110c1030a0 0x7f110c1983d0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f10f802c200 tx=0x7f10f8005e80 comp rx=0 tx=0).stop 2024-03-28T22:44:59.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.707+0000 7f1111c00640 1 -- 172.21.15.43:0/3584099745 shutdown_connections 2024-03-28T22:44:59.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.707+0000 7f1111c00640 1 --2- 172.21.15.43:0/3584099745 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f10f00a1f70 0x7f10f00a4430 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.707+0000 7f1111c00640 1 --2- 172.21.15.43:0/3584099745 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f110c10bba0 0x7f110c19c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.707+0000 7f1111c00640 1 --2- 172.21.15.43:0/3584099745 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f110c1030a0 0x7f110c1983d0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.707+0000 7f1111c00640 1 --2- 172.21.15.43:0/3584099745 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f110c102780 0x7f110c197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:44:59.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.707+0000 7f1111c00640 1 -- 172.21.15.43:0/3584099745 >> 172.21.15.43:0/3584099745 conn(0x7f110c0fbfa0 msgr2=0x7f110c0fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:44:59.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.707+0000 7f1111c00640 1 -- 172.21.15.43:0/3584099745 shutdown_connections 2024-03-28T22:44:59.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:44:59.707+0000 7f1111c00640 1 -- 172.21.15.43:0/3584099745 wait complete. 2024-03-28T22:44:59.724 INFO:tasks.cephfs.filesystem:scrub status for tag:6d472f11-6ed4-405f-82bb-c5b3adc62474 - {'path': '/', 'tag': '6d472f11-6ed4-405f-82bb-c5b3adc62474', 'options': 'recursive,force'} 2024-03-28T22:45:01.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:00 smithi172 ceph-mon[36566]: scrub queued for path: / 2024-03-28T22:45:01.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:00 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:45:01.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:00 smithi172 ceph-mon[36566]: scrub summary: active paths [/] 2024-03-28T22:45:01.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:00 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:45:01.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:00 smithi172 ceph-mon[36566]: pgmap v242: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 10 op/s 2024-03-28T22:45:01.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:00 smithi082 ceph-mon[36471]: scrub queued for path: / 2024-03-28T22:45:01.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:00 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:45:01.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:00 smithi082 ceph-mon[36471]: scrub summary: active paths [/] 2024-03-28T22:45:01.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:00 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:45:01.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:00 smithi082 ceph-mon[36471]: pgmap v242: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 10 op/s 2024-03-28T22:45:01.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:00 smithi043 ceph-mon[31823]: scrub queued for path: / 2024-03-28T22:45:01.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:00 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:45:01.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:00 smithi043 ceph-mon[31823]: scrub summary: active paths [/] 2024-03-28T22:45:01.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:00 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:45:01.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:00 smithi043 ceph-mon[31823]: pgmap v242: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 10 op/s 2024-03-28T22:45:03.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:02 smithi172 ceph-mon[36566]: pgmap v243: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 10 op/s 2024-03-28T22:45:03.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:02 smithi082 ceph-mon[36471]: pgmap v243: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 10 op/s 2024-03-28T22:45:03.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:02 smithi043 ceph-mon[31823]: pgmap v243: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 KiB/s wr, 10 op/s 2024-03-28T22:45:04.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:03 smithi172 ceph-mon[36566]: scrub summary: idle 2024-03-28T22:45:04.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:03 smithi082 ceph-mon[36471]: scrub summary: idle 2024-03-28T22:45:04.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:03 smithi043 ceph-mon[31823]: scrub summary: idle 2024-03-28T22:45:05.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:04 smithi082 ceph-mon[36471]: pgmap v244: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 80 KiB/s wr, 38 op/s 2024-03-28T22:45:05.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:04 smithi043 ceph-mon[31823]: pgmap v244: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 80 KiB/s wr, 38 op/s 2024-03-28T22:45:05.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:04 smithi172 ceph-mon[36566]: pgmap v244: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 80 KiB/s wr, 38 op/s 2024-03-28T22:45:07.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:06 smithi082 ceph-mon[36471]: pgmap v245: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 67 KiB/s wr, 34 op/s 2024-03-28T22:45:07.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:06 smithi043 ceph-mon[31823]: pgmap v245: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 67 KiB/s wr, 34 op/s 2024-03-28T22:45:07.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:06 smithi172 ceph-mon[36566]: pgmap v245: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 67 KiB/s wr, 34 op/s 2024-03-28T22:45:09.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:08 smithi082 ceph-mon[36471]: pgmap v246: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 68 KiB/s wr, 39 op/s 2024-03-28T22:45:09.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:08 smithi043 ceph-mon[31823]: pgmap v246: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 68 KiB/s wr, 39 op/s 2024-03-28T22:45:09.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:08 smithi172 ceph-mon[36566]: pgmap v246: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 68 KiB/s wr, 39 op/s 2024-03-28T22:45:10.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:09 smithi082 ceph-mon[36471]: osdmap e103: 12 total, 12 up, 12 in 2024-03-28T22:45:10.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:09 smithi043 ceph-mon[31823]: osdmap e103: 12 total, 12 up, 12 in 2024-03-28T22:45:10.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:09 smithi172 ceph-mon[36566]: osdmap e103: 12 total, 12 up, 12 in 2024-03-28T22:45:11.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:10 smithi082 ceph-mon[36471]: pgmap v248: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 75 KiB/s wr, 39 op/s 2024-03-28T22:45:11.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:10 smithi043 ceph-mon[31823]: pgmap v248: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 75 KiB/s wr, 39 op/s 2024-03-28T22:45:11.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:10 smithi172 ceph-mon[36566]: pgmap v248: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 75 KiB/s wr, 39 op/s 2024-03-28T22:45:13.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:12 smithi082 ceph-mon[36471]: pgmap v249: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 75 KiB/s wr, 39 op/s 2024-03-28T22:45:13.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:12 smithi043 ceph-mon[31823]: pgmap v249: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 75 KiB/s wr, 39 op/s 2024-03-28T22:45:13.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:12 smithi172 ceph-mon[36566]: pgmap v249: 129 pgs: 129 active+clean; 67 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 75 KiB/s wr, 39 op/s 2024-03-28T22:45:15.197 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:14 smithi172 ceph-mon[36566]: pgmap v250: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-03-28T22:45:15.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:14 smithi082 ceph-mon[36471]: pgmap v250: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-03-28T22:45:15.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:14 smithi043 ceph-mon[31823]: pgmap v250: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-03-28T22:45:17.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:16 smithi043 ceph-mon[31823]: pgmap v251: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-03-28T22:45:17.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:16 smithi172 ceph-mon[36566]: pgmap v251: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-03-28T22:45:17.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:16 smithi082 ceph-mon[36471]: pgmap v251: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-03-28T22:45:19.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:18 smithi172 ceph-mon[36566]: pgmap v252: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-03-28T22:45:19.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:18 smithi172 ceph-mon[36566]: osdmap e104: 12 total, 12 up, 12 in 2024-03-28T22:45:19.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:18 smithi082 ceph-mon[36471]: pgmap v252: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-03-28T22:45:19.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:18 smithi082 ceph-mon[36471]: osdmap e104: 12 total, 12 up, 12 in 2024-03-28T22:45:19.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:18 smithi043 ceph-mon[31823]: pgmap v252: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-03-28T22:45:19.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:18 smithi043 ceph-mon[31823]: osdmap e104: 12 total, 12 up, 12 in 2024-03-28T22:45:21.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:20 smithi172 ceph-mon[36566]: pgmap v254: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-03-28T22:45:21.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:20 smithi082 ceph-mon[36471]: pgmap v254: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-03-28T22:45:21.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:20 smithi043 ceph-mon[31823]: pgmap v254: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-03-28T22:45:23.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:22 smithi172 ceph-mon[36566]: pgmap v255: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-03-28T22:45:23.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:22 smithi082 ceph-mon[36471]: pgmap v255: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-03-28T22:45:23.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:22 smithi043 ceph-mon[31823]: pgmap v255: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-03-28T22:45:25.068 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:24 smithi043 ceph-mon[31823]: pgmap v256: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-28T22:45:25.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:24 smithi172 ceph-mon[36566]: pgmap v256: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-28T22:45:25.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:24 smithi082 ceph-mon[36471]: pgmap v256: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-28T22:45:27.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:26 smithi172 ceph-mon[36566]: pgmap v257: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-28T22:45:27.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:26 smithi082 ceph-mon[36471]: pgmap v257: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-28T22:45:27.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:26 smithi043 ceph-mon[31823]: pgmap v257: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-28T22:45:29.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:28 smithi172 ceph-mon[36566]: pgmap v258: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:45:29.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:28 smithi082 ceph-mon[36471]: pgmap v258: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:45:29.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:28 smithi043 ceph-mon[31823]: pgmap v258: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:45:29.726 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:45:29.726 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:45:29.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.974+0000 7faa7fe00640 1 -- 172.21.15.43:0/246402586 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faa80074420 msgr2=0x7faa8010d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:29.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.974+0000 7faa7fe00640 1 --2- 172.21.15.43:0/246402586 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faa80074420 0x7faa8010d940 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7faa6c009900 tx=0x7faa6c02bcd0 comp rx=0 tx=0).stop 2024-03-28T22:45:29.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.974+0000 7faa7fe00640 1 -- 172.21.15.43:0/246402586 shutdown_connections 2024-03-28T22:45:29.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.974+0000 7faa7fe00640 1 --2- 172.21.15.43:0/246402586 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faa80074420 0x7faa8010d940 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:29.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.974+0000 7faa7fe00640 1 --2- 172.21.15.43:0/246402586 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa8010de80 0x7faa80110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:29.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.974+0000 7faa7fe00640 1 --2- 172.21.15.43:0/246402586 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa80073b00 0x7faa80073ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:29.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.974+0000 7faa7fe00640 1 -- 172.21.15.43:0/246402586 >> 172.21.15.43:0/246402586 conn(0x7faa800fd0b0 msgr2=0x7faa800ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:29.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.975+0000 7faa7fe00640 1 -- 172.21.15.43:0/246402586 shutdown_connections 2024-03-28T22:45:29.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.975+0000 7faa7fe00640 1 -- 172.21.15.43:0/246402586 wait complete. 2024-03-28T22:45:29.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.976+0000 7faa7fe00640 1 Processor -- start 2024-03-28T22:45:29.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.976+0000 7faa7fe00640 1 -- start start 2024-03-28T22:45:29.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.976+0000 7faa7fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa80073b00 0x7faa8019a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:29.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.976+0000 7faa7fe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa80074420 0x7faa8019a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:29.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.976+0000 7faa7fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faa8010de80 0x7faa8019ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:29.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.976+0000 7faa7fe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7faa80113bf0 con 0x7faa80073b00 2024-03-28T22:45:29.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.976+0000 7faa7fe00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7faa80113a70 con 0x7faa8010de80 2024-03-28T22:45:29.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.976+0000 7faa7fe00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7faa80113d70 con 0x7faa80074420 2024-03-28T22:45:29.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.977+0000 7faa7f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faa8010de80 0x7faa8019ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:29.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.977+0000 7faa7e000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa80074420 0x7faa8019a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:29.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.977+0000 7faa7ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa80073b00 0x7faa8019a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:29.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.977+0000 7faa7f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faa8010de80 0x7faa8019ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33732/0 (socket says 172.21.15.43:33732) 2024-03-28T22:45:29.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.977+0000 7faa7e000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa80074420 0x7faa8019a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43468/0 (socket says 172.21.15.43:43468) 2024-03-28T22:45:29.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.977+0000 7faa7f400640 1 -- 172.21.15.43:0/1349938423 learned_addr learned my addr 172.21.15.43:0/1349938423 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:29.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.977+0000 7faa7f400640 1 -- 172.21.15.43:0/1349938423 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa80074420 msgr2=0x7faa8019a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:29.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.977+0000 7faa7f400640 1 --2- 172.21.15.43:0/1349938423 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa80074420 0x7faa8019a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:29.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.977+0000 7faa7f400640 1 -- 172.21.15.43:0/1349938423 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa80073b00 msgr2=0x7faa8019a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:29.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.977+0000 7faa7f400640 1 --2- 172.21.15.43:0/1349938423 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa80073b00 0x7faa8019a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:29.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.977+0000 7faa7f400640 1 -- 172.21.15.43:0/1349938423 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa8019f160 con 0x7faa8010de80 2024-03-28T22:45:29.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.977+0000 7faa7e000640 1 --2- 172.21.15.43:0/1349938423 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa80074420 0x7faa8019a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:45:29.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.977+0000 7faa7ea00640 1 --2- 172.21.15.43:0/1349938423 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa80073b00 0x7faa8019a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:45:29.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.978+0000 7faa7f400640 1 --2- 172.21.15.43:0/1349938423 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faa8010de80 0x7faa8019ea30 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7faa7800eb10 tx=0x7faa7800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:29.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.978+0000 7faa77400640 1 -- 172.21.15.43:0/1349938423 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa78004b20 con 0x7faa8010de80 2024-03-28T22:45:29.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.978+0000 7faa77400640 1 -- 172.21.15.43:0/1349938423 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa78004cc0 con 0x7faa8010de80 2024-03-28T22:45:29.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.978+0000 7faa77400640 1 -- 172.21.15.43:0/1349938423 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa78010760 con 0x7faa8010de80 2024-03-28T22:45:29.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.978+0000 7faa7fe00640 1 -- 172.21.15.43:0/1349938423 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa8019f450 con 0x7faa8010de80 2024-03-28T22:45:29.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.978+0000 7faa7fe00640 1 -- 172.21.15.43:0/1349938423 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa801a6ce0 con 0x7faa8010de80 2024-03-28T22:45:29.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.981+0000 7faa88671740 1 Processor -- start 2024-03-28T22:45:29.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.981+0000 7faa77400640 1 -- 172.21.15.43:0/1349938423 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7faa78005020 con 0x7faa8010de80 2024-03-28T22:45:29.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.982+0000 7faa88671740 1 -- start start 2024-03-28T22:45:29.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.982+0000 7faa88671740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a0a2aedb0 0x560a0a2adb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:29.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.982+0000 7faa88671740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a0a2ae0d0 0x560a0a2ae550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:29.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.982+0000 7faa88671740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a0a2b7440 0x560a0a2b9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:29.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.982+0000 7faa88671740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x560a0a212100 con 0x560a0a2aedb0 2024-03-28T22:45:29.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.982+0000 7faa88671740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x560a0a1baba0 con 0x560a0a2ae0d0 2024-03-28T22:45:29.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.982+0000 7faa88671740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x560a0a20e1a0 con 0x560a0a2b7440 2024-03-28T22:45:29.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.982+0000 7faa7ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a0a2ae0d0 0x560a0a2ae550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:29.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.982+0000 7faa7ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a0a2ae0d0 0x560a0a2ae550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33734/0 (socket says 172.21.15.43:33734) 2024-03-28T22:45:29.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.982+0000 7faa7ea00640 1 -- 172.21.15.43:0/3249267930 learned_addr learned my addr 172.21.15.43:0/3249267930 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:29.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.982+0000 7faa7e000640 1 --2- 172.21.15.43:0/3249267930 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a0a2aedb0 0x560a0a2adb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:29.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.983+0000 7faa7e000640 1 --2- 172.21.15.43:0/3249267930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a0a2b7440 0x560a0a2b9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:29.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.983+0000 7faa7ea00640 1 -- 172.21.15.43:0/3249267930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a0a2b7440 msgr2=0x560a0a2b9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:29.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.983+0000 7faa7ea00640 1 --2- 172.21.15.43:0/3249267930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a0a2b7440 0x560a0a2b9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:29.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.983+0000 7faa77400640 1 --2- 172.21.15.43:0/1349938423 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7faa640a1f70 0x7faa640a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:29.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.983+0000 7faa7ea00640 1 -- 172.21.15.43:0/3249267930 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a0a2aedb0 msgr2=0x560a0a2adb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:29.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.983+0000 7faa7ea00640 1 --2- 172.21.15.43:0/3249267930 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a0a2aedb0 0x560a0a2adb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:29.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.983+0000 7faa7ea00640 1 -- 172.21.15.43:0/3249267930 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a0a1eb1a0 con 0x560a0a2ae0d0 2024-03-28T22:45:29.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.983+0000 7faa7e000640 1 --2- 172.21.15.43:0/3249267930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a0a2b7440 0x560a0a2b9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:45:29.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.983+0000 7faa77400640 1 -- 172.21.15.43:0/1349938423 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7faa780ceaa0 con 0x7faa8010de80 2024-03-28T22:45:29.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.983+0000 7faa7f400640 1 --2- 172.21.15.43:0/1349938423 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7faa640a1f70 0x7faa640a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:29.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.983+0000 7faa7ea00640 1 --2- 172.21.15.43:0/3249267930 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a0a2ae0d0 0x560a0a2ae550 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7faa700115c0 tx=0x7faa70011a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a05eae5dce9af137 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:29.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.983+0000 7faa6be00640 1 -- 172.21.15.43:0/3249267930 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa7000f640 con 0x560a0a2ae0d0 2024-03-28T22:45:29.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.983+0000 7faa6be00640 1 -- 172.21.15.43:0/3249267930 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa7000fcc0 con 0x560a0a2ae0d0 2024-03-28T22:45:29.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.983+0000 7faa6be00640 1 -- 172.21.15.43:0/3249267930 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa70019dc0 con 0x560a0a2ae0d0 2024-03-28T22:45:29.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.984+0000 7faa88671740 1 -- 172.21.15.43:0/3249267930 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a0a2ae0d0 msgr2=0x560a0a2ae550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:29.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.984+0000 7faa88671740 1 --2- 172.21.15.43:0/3249267930 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a0a2ae0d0 0x560a0a2ae550 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7faa700115c0 tx=0x7faa70011a80 comp rx=0 tx=0).stop 2024-03-28T22:45:29.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.984+0000 7faa7f400640 1 --2- 172.21.15.43:0/1349938423 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7faa640a1f70 0x7faa640a4430 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7faa780cbdf0 tx=0x7faa78005b40 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:29.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.984+0000 7faa88671740 1 -- 172.21.15.43:0/3249267930 shutdown_connections 2024-03-28T22:45:29.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.984+0000 7faa88671740 1 --2- 172.21.15.43:0/3249267930 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a0a2ae0d0 0x560a0a2ae550 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:29.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.984+0000 7faa88671740 1 --2- 172.21.15.43:0/3249267930 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a0a2b7440 0x560a0a2b9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:29.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.984+0000 7faa88671740 1 --2- 172.21.15.43:0/3249267930 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a0a2aedb0 0x560a0a2adb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:29.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.984+0000 7faa88671740 1 -- 172.21.15.43:0/3249267930 >> 172.21.15.43:0/3249267930 conn(0x560a0a274a10 msgr2=0x560a0a2a8b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:29.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.984+0000 7faa88671740 1 -- 172.21.15.43:0/3249267930 shutdown_connections 2024-03-28T22:45:29.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.984+0000 7faa88671740 1 -- 172.21.15.43:0/3249267930 wait complete. 2024-03-28T22:45:29.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.985+0000 7faa88671740 1 -- start start 2024-03-28T22:45:29.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.985+0000 7faa88671740 1 Processor -- start 2024-03-28T22:45:29.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.985+0000 7faa88671740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a0a2ae0d0 0x560a0a2b43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:29.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.985+0000 7faa88671740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a0a2aedb0 0x560a0a2b4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:29.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.985+0000 7faa88671740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a0a2b7440 0x560a0a2b4e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:29.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.985+0000 7faa88671740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x560a0a2ba2b0 con 0x560a0a2b7440 2024-03-28T22:45:29.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.985+0000 7faa88671740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x560a0a1b0250 con 0x560a0a2ae0d0 2024-03-28T22:45:29.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.985+0000 7faa88671740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x560a0a2ba430 con 0x560a0a2aedb0 2024-03-28T22:45:29.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.985+0000 7faa88671740 10 client.0 _ll_register_callbacks cb 0x560a09f25be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:45:29.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.986+0000 7faa7ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a0a2aedb0 0x560a0a2b4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:29.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.986+0000 7faa7e000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a0a2ae0d0 0x560a0a2b43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:29.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.986+0000 7faa7ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a0a2aedb0 0x560a0a2b4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43484/0 (socket says 172.21.15.43:43484) 2024-03-28T22:45:29.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.986+0000 7faa7ea00640 1 -- 172.21.15.43:0/1362423816 learned_addr learned my addr 172.21.15.43:0/1362423816 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:29.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.986+0000 7faa7e000640 1 --2- 172.21.15.43:0/1362423816 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a0a2b7440 0x560a0a2b4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:29.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.986+0000 7faa7ea00640 1 -- 172.21.15.43:0/1362423816 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a0a2ae0d0 msgr2=0x560a0a2b43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:29.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.986+0000 7faa7ea00640 1 --2- 172.21.15.43:0/1362423816 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a0a2ae0d0 0x560a0a2b43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:29.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.986+0000 7faa7ea00640 1 -- 172.21.15.43:0/1362423816 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a0a2b7440 msgr2=0x560a0a2b4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:29.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.986+0000 7faa7ea00640 1 --2- 172.21.15.43:0/1362423816 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a0a2b7440 0x560a0a2b4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:29.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.986+0000 7faa7ea00640 1 -- 172.21.15.43:0/1362423816 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a0a1cc960 con 0x560a0a2aedb0 2024-03-28T22:45:29.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.986+0000 7faa7ea00640 1 --2- 172.21.15.43:0/1362423816 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a0a2aedb0 0x560a0a2b4930 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7faa70019a20 tx=0x7faa700056d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d28854d29ee970f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:29.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.986+0000 7faa68c00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:45:29.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.987+0000 7faa68c00640 1 -- 172.21.15.43:0/1362423816 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa70026020 con 0x560a0a2aedb0 2024-03-28T22:45:29.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.987+0000 7faa88671740 1 -- 172.21.15.43:0/1362423816 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa7000fcc0 con 0x560a0a2aedb0 2024-03-28T22:45:29.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.987+0000 7faa68c00640 1 -- 172.21.15.43:0/1362423816 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7faa70004040 con 0x560a0a2aedb0 2024-03-28T22:45:29.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.987+0000 7faa68c00640 1 -- 172.21.15.43:0/1362423816 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7faa70031050 con 0x560a0a2aedb0 2024-03-28T22:45:29.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.987+0000 7faa88671740 10 client.28423 fetch_fsmap learned FSMap version 32 2024-03-28T22:45:29.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.987+0000 7faa88671740 1 -- 172.21.15.43:0/1362423816 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560a09ff07d0 con 0x560a0a2aedb0 2024-03-28T22:45:29.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.987+0000 7faa68c00640 1 -- 172.21.15.43:0/1362423816 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa7000fcc0 con 0x560a0a2aedb0 2024-03-28T22:45:29.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.988+0000 7faa68c00640 1 -- 172.21.15.43:0/1362423816 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7faa70003820 con 0x560a0a2aedb0 2024-03-28T22:45:29.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.988+0000 7faa88671740 10 client.28423 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:45:29.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.988+0000 7faa88671740 10 client.28423 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:45:29.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.988+0000 7faa88671740 20 client.28423 populate_metadata read hostname 'smithi043' 2024-03-28T22:45:29.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.988+0000 7faa88671740 1 --2- 172.21.15.43:0/1362423816 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560a0a2c7770 0x560a0a2c9b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:29.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.988+0000 7faa88671740 4 client.28423 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:45:29.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.988+0000 7faa88671740 1 -- 172.21.15.43:0/1362423816 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560a0a21f640 con 0x560a0a2c7770 2024-03-28T22:45:29.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.988+0000 7faa7e000640 1 --2- 172.21.15.43:0/1362423816 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560a0a2c7770 0x560a0a2c9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:29.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.989+0000 7faa7e000640 1 --2- 172.21.15.43:0/1362423816 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560a0a2c7770 0x560a0a2c9b60 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53f940d3523a545f server_cookie=928c768bbd464a56 in_seq=0 out_seq=0 2024-03-28T22:45:29.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.989+0000 7faa68c00640 10 client.28423 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:45:29.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.990+0000 7faa68c00640 1 -- 172.21.15.43:0/1362423816 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560a0a21f640 con 0x560a0a2c7770 2024-03-28T22:45:29.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.990+0000 7faa68c00640 10 client.28423 handle_command_reply: tid=0 2024-03-28T22:45:29.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.990+0000 7faa88671740 1 client.28423 shutdown 2024-03-28T22:45:29.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.990+0000 7faa88671740 1 -- 172.21.15.43:0/1362423816 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a0a2aedb0 msgr2=0x560a0a2b4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:29.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.990+0000 7faa88671740 1 --2- 172.21.15.43:0/1362423816 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a0a2aedb0 0x560a0a2b4930 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7faa70019a20 tx=0x7faa700056d0 comp rx=0 tx=0).stop 2024-03-28T22:45:29.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.990+0000 7faa88671740 1 -- 172.21.15.43:0/1362423816 shutdown_connections 2024-03-28T22:45:29.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.990+0000 7faa88671740 1 --2- 172.21.15.43:0/1362423816 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a0a2b7440 0x560a0a2b4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:29.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.990+0000 7faa88671740 1 --2- 172.21.15.43:0/1362423816 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a0a2aedb0 0x560a0a2b4930 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:29.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.990+0000 7faa88671740 1 --2- 172.21.15.43:0/1362423816 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560a0a2c7770 0x560a0a2c9b60 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:29.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.990+0000 7faa88671740 1 --2- 172.21.15.43:0/1362423816 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a0a2ae0d0 0x560a0a2b43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:29.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.990+0000 7faa68c00640 0 client.28423 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:45:29.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.990+0000 7faa88671740 1 -- 172.21.15.43:0/1362423816 >> 172.21.15.43:0/1362423816 conn(0x560a0a274a10 msgr2=0x560a0a2aa870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:29.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.990+0000 7faa88671740 1 -- 172.21.15.43:0/1362423816 shutdown_connections 2024-03-28T22:45:29.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.990+0000 7faa88671740 1 -- 172.21.15.43:0/1362423816 wait complete. 2024-03-28T22:45:29.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:29.990+0000 7faa88671740 20 client.28423 trim_cache size 0 max 16384 2024-03-28T22:45:30.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.009+0000 7faa88671740 1 Processor -- start 2024-03-28T22:45:30.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.009+0000 7faa88671740 1 -- start start 2024-03-28T22:45:30.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.010+0000 7faa88671740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a0a2ae0d0 0x560a0a2b4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:30.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.010+0000 7faa88671740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a0a2aedb0 0x560a0a2c30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:30.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.010+0000 7faa88671740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a0a2b7440 0x560a0a2cf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:30.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.010+0000 7faa88671740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7faa801b0d60 con 0x560a0a2b7440 2024-03-28T22:45:30.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.010+0000 7faa88671740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7faa801b28a0 con 0x560a0a2ae0d0 2024-03-28T22:45:30.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.010+0000 7faa88671740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7faa801b1430 con 0x560a0a2aedb0 2024-03-28T22:45:30.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.010+0000 7faa7e000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a0a2ae0d0 0x560a0a2b4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:30.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.010+0000 7faa7e000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a0a2ae0d0 0x560a0a2b4fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33738/0 (socket says 172.21.15.43:33738) 2024-03-28T22:45:30.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.010+0000 7faa7e000640 1 -- 172.21.15.43:0/1453502959 learned_addr learned my addr 172.21.15.43:0/1453502959 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:30.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.010+0000 7faa7ea00640 1 --2- 172.21.15.43:0/1453502959 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a0a2aedb0 0x560a0a2c30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:30.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.010+0000 7faa7e000640 1 --2- 172.21.15.43:0/1453502959 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a0a2b7440 0x560a0a2cf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:30.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.011+0000 7faa7e000640 1 -- 172.21.15.43:0/1453502959 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a0a2aedb0 msgr2=0x560a0a2c30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:30.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.011+0000 7faa7e000640 1 --2- 172.21.15.43:0/1453502959 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a0a2aedb0 0x560a0a2c30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.011+0000 7faa7e000640 1 -- 172.21.15.43:0/1453502959 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a0a2b7440 msgr2=0x560a0a2cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:30.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.011+0000 7faa7e000640 1 --2- 172.21.15.43:0/1453502959 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a0a2b7440 0x560a0a2cf320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.011+0000 7faa7e000640 1 -- 172.21.15.43:0/1453502959 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a09f263f0 con 0x560a0a2ae0d0 2024-03-28T22:45:30.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.011+0000 7faa7ea00640 1 --2- 172.21.15.43:0/1453502959 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a0a2aedb0 0x560a0a2c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:45:30.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.011+0000 7faa7e000640 1 --2- 172.21.15.43:0/1453502959 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a0a2ae0d0 0x560a0a2b4fc0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7faa6c000c00 tx=0x7faa6c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39d6ac219223a4cf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:30.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.011+0000 7faa6b400640 1 -- 172.21.15.43:0/1453502959 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6c04fde0 con 0x560a0a2ae0d0 2024-03-28T22:45:30.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.011+0000 7faa6b400640 1 -- 172.21.15.43:0/1453502959 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa6c04eb30 con 0x560a0a2ae0d0 2024-03-28T22:45:30.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.011+0000 7faa6b400640 1 -- 172.21.15.43:0/1453502959 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6c04e7b0 con 0x560a0a2ae0d0 2024-03-28T22:45:30.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.012+0000 7faa88671740 1 -- 172.21.15.43:0/1453502959 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a0a2ae0d0 msgr2=0x560a0a2b4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:30.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.012+0000 7faa88671740 1 --2- 172.21.15.43:0/1453502959 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a0a2ae0d0 0x560a0a2b4fc0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7faa6c000c00 tx=0x7faa6c009260 comp rx=0 tx=0).stop 2024-03-28T22:45:30.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.012+0000 7faa88671740 1 -- 172.21.15.43:0/1453502959 shutdown_connections 2024-03-28T22:45:30.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.012+0000 7faa88671740 1 --2- 172.21.15.43:0/1453502959 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a0a2b7440 0x560a0a2cf320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.012+0000 7faa88671740 1 --2- 172.21.15.43:0/1453502959 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a0a2aedb0 0x560a0a2c30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.012+0000 7faa88671740 1 --2- 172.21.15.43:0/1453502959 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a0a2ae0d0 0x560a0a2b4fc0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.012+0000 7faa88671740 1 -- 172.21.15.43:0/1453502959 >> 172.21.15.43:0/1453502959 conn(0x560a0a274a10 msgr2=0x560a0a2aa800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:30.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.012+0000 7faa88671740 1 -- 172.21.15.43:0/1453502959 shutdown_connections 2024-03-28T22:45:30.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.012+0000 7faa88671740 1 -- 172.21.15.43:0/1453502959 wait complete. 2024-03-28T22:45:30.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.012+0000 7faa88671740 1 -- start start 2024-03-28T22:45:30.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.013+0000 7faa88671740 1 Processor -- start 2024-03-28T22:45:30.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.013+0000 7faa88671740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a0a2ae0d0 0x560a0a2bf860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:30.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.013+0000 7faa88671740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a0a2aedb0 0x560a0a2bcbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:30.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.013+0000 7faa88671740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a0a2b7440 0x560a0a2bd130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:30.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.013+0000 7faa88671740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7faa801b21d0 con 0x560a0a2ae0d0 2024-03-28T22:45:30.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.013+0000 7faa88671740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7faa801b1b00 con 0x560a0a2aedb0 2024-03-28T22:45:30.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.013+0000 7faa88671740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7faa801aed60 con 0x560a0a2b7440 2024-03-28T22:45:30.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.013+0000 7faa88671740 10 client.0 _ll_register_callbacks cb 0x7faa70011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:45:30.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.014+0000 7faa7e000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a0a2ae0d0 0x560a0a2bf860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:30.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.014+0000 7faa7e000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a0a2ae0d0 0x560a0a2bf860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50574/0 (socket says 172.21.15.43:50574) 2024-03-28T22:45:30.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.014+0000 7faa7e000640 1 -- 172.21.15.43:0/2341994022 learned_addr learned my addr 172.21.15.43:0/2341994022 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:30.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.014+0000 7faa7ea00640 1 --2- 172.21.15.43:0/2341994022 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a0a2aedb0 0x560a0a2bcbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:30.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.014+0000 7faa7e000640 1 --2- 172.21.15.43:0/2341994022 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a0a2b7440 0x560a0a2bd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:30.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.014+0000 7faa7e000640 1 -- 172.21.15.43:0/2341994022 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a0a2b7440 msgr2=0x560a0a2bd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:30.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.014+0000 7faa7e000640 1 --2- 172.21.15.43:0/2341994022 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a0a2b7440 0x560a0a2bd130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.014+0000 7faa7e000640 1 -- 172.21.15.43:0/2341994022 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a0a2aedb0 msgr2=0x560a0a2bcbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:30.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.014+0000 7faa7e000640 1 --2- 172.21.15.43:0/2341994022 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a0a2aedb0 0x560a0a2bcbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.014+0000 7faa7e000640 1 -- 172.21.15.43:0/2341994022 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a09ee60f0 con 0x560a0a2ae0d0 2024-03-28T22:45:30.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.014+0000 7faa7ea00640 1 --2- 172.21.15.43:0/2341994022 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a0a2aedb0 0x560a0a2bcbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:45:30.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.014+0000 7faa7e000640 1 --2- 172.21.15.43:0/2341994022 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a0a2ae0d0 0x560a0a2bf860 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7faa6c04e690 tx=0x7faa6c04e590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efd3010d4c8bfe8a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:30.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.014+0000 7faa68c00640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:45:30.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.015+0000 7faa68c00640 1 -- 172.21.15.43:0/2341994022 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6c04e2c0 con 0x560a0a2ae0d0 2024-03-28T22:45:30.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.015+0000 7faa68c00640 1 -- 172.21.15.43:0/2341994022 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7faa6c02cb00 con 0x560a0a2ae0d0 2024-03-28T22:45:30.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.015+0000 7faa88671740 1 -- 172.21.15.43:0/2341994022 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa6c04eb30 con 0x560a0a2ae0d0 2024-03-28T22:45:30.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.015+0000 7faa68c00640 1 -- 172.21.15.43:0/2341994022 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7faa6c02cd00 con 0x560a0a2ae0d0 2024-03-28T22:45:30.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.015+0000 7faa88671740 10 client.18228 fetch_fsmap learned FSMap version 32 2024-03-28T22:45:30.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.015+0000 7faa88671740 1 -- 172.21.15.43:0/2341994022 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560a0a32d660 con 0x560a0a2ae0d0 2024-03-28T22:45:30.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.015+0000 7faa68c00640 1 -- 172.21.15.43:0/2341994022 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa6c04eb30 con 0x560a0a2ae0d0 2024-03-28T22:45:30.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.015+0000 7faa68c00640 1 -- 172.21.15.43:0/2341994022 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7faa6c04c030 con 0x560a0a2ae0d0 2024-03-28T22:45:30.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.015+0000 7faa88671740 10 client.18228 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:45:30.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.015+0000 7faa88671740 10 client.18228 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:45:30.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.015+0000 7faa88671740 20 client.18228 populate_metadata read hostname 'smithi043' 2024-03-28T22:45:30.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.015+0000 7faa88671740 1 --2- 172.21.15.43:0/2341994022 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560a0a32dd00 0x560a0a330120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:30.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.016+0000 7faa88671740 4 client.18228 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:45:30.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.016+0000 7faa88671740 1 -- 172.21.15.43:0/2341994022 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560a0a219990 con 0x560a0a32dd00 2024-03-28T22:45:30.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.016+0000 7faa7ea00640 1 --2- 172.21.15.43:0/2341994022 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560a0a32dd00 0x560a0a330120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:30.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.016+0000 7faa7ea00640 1 --2- 172.21.15.43:0/2341994022 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560a0a32dd00 0x560a0a330120 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50d33330703ff706 server_cookie=d8090d4014690296 in_seq=0 out_seq=0 2024-03-28T22:45:30.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.016+0000 7faa68c00640 10 client.18228 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:45:30.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.016+0000 7faa68c00640 1 -- 172.21.15.43:0/2341994022 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560a0a219990 con 0x560a0a32dd00 2024-03-28T22:45:30.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.016+0000 7faa68c00640 10 client.18228 handle_command_reply: tid=0 2024-03-28T22:45:30.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.016+0000 7faa88671740 1 client.18228 shutdown 2024-03-28T22:45:30.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.017+0000 7faa88671740 1 -- 172.21.15.43:0/2341994022 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a0a2ae0d0 msgr2=0x560a0a2bf860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:30.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.017+0000 7faa88671740 1 --2- 172.21.15.43:0/2341994022 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a0a2ae0d0 0x560a0a2bf860 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7faa6c04e690 tx=0x7faa6c04e590 comp rx=0 tx=0).stop 2024-03-28T22:45:30.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.017+0000 7faa88671740 1 -- 172.21.15.43:0/2341994022 shutdown_connections 2024-03-28T22:45:30.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.017+0000 7faa88671740 1 --2- 172.21.15.43:0/2341994022 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560a0a32dd00 0x560a0a330120 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.017+0000 7faa88671740 1 --2- 172.21.15.43:0/2341994022 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560a0a2aedb0 0x560a0a2bcbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.017+0000 7faa88671740 1 --2- 172.21.15.43:0/2341994022 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560a0a2b7440 0x560a0a2bd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.017+0000 7faa68c00640 0 client.18228 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:45:30.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.017+0000 7faa88671740 1 --2- 172.21.15.43:0/2341994022 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560a0a2ae0d0 0x560a0a2bf860 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.017+0000 7faa88671740 1 -- 172.21.15.43:0/2341994022 >> 172.21.15.43:0/2341994022 conn(0x560a0a274a10 msgr2=0x560a0a2a9d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:30.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.017+0000 7faa88671740 1 -- 172.21.15.43:0/2341994022 shutdown_connections 2024-03-28T22:45:30.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.017+0000 7faa88671740 1 -- 172.21.15.43:0/2341994022 wait complete. 2024-03-28T22:45:30.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.017+0000 7faa88671740 20 client.18228 trim_cache size 0 max 16384 2024-03-28T22:45:30.024 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:45:30.024 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-03-28T22:45:30.024 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-03-28T22:45:30.024 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:45:30.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.018+0000 7faa7fe00640 1 -- 172.21.15.43:0/1349938423 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7faa640a1f70 msgr2=0x7faa640a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:30.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.018+0000 7faa7fe00640 1 --2- 172.21.15.43:0/1349938423 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7faa640a1f70 0x7faa640a4430 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7faa780cbdf0 tx=0x7faa78005b40 comp rx=0 tx=0).stop 2024-03-28T22:45:30.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.018+0000 7faa7fe00640 1 -- 172.21.15.43:0/1349938423 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faa8010de80 msgr2=0x7faa8019ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:30.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.018+0000 7faa7fe00640 1 --2- 172.21.15.43:0/1349938423 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faa8010de80 0x7faa8019ea30 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7faa7800eb10 tx=0x7faa7800c2d0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.018+0000 7faa7fe00640 1 -- 172.21.15.43:0/1349938423 shutdown_connections 2024-03-28T22:45:30.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.018+0000 7faa7fe00640 1 --2- 172.21.15.43:0/1349938423 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7faa640a1f70 0x7faa640a4430 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.018+0000 7faa7fe00640 1 --2- 172.21.15.43:0/1349938423 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faa8010de80 0x7faa8019ea30 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.018+0000 7faa7fe00640 1 --2- 172.21.15.43:0/1349938423 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faa80074420 0x7faa8019a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.018+0000 7faa7fe00640 1 --2- 172.21.15.43:0/1349938423 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa80073b00 0x7faa8019a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.018+0000 7faa7fe00640 1 -- 172.21.15.43:0/1349938423 >> 172.21.15.43:0/1349938423 conn(0x7faa800fd0b0 msgr2=0x7faa800fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:30.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.018+0000 7faa7fe00640 1 -- 172.21.15.43:0/1349938423 shutdown_connections 2024-03-28T22:45:30.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.018+0000 7faa7fe00640 1 -- 172.21.15.43:0/1349938423 wait complete. 2024-03-28T22:45:30.033 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:45:30.033 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:45:30.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.252+0000 7f87f3400640 1 -- 172.21.15.43:0/3104476813 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f87ec1098f0 msgr2=0x7f87ec109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:30.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.252+0000 7f87f3400640 1 --2- 172.21.15.43:0/3104476813 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f87ec1098f0 0x7f87ec109cd0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f87dc009ca0 tx=0x7f87dc02c0f0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.252+0000 7f87f3400640 1 -- 172.21.15.43:0/3104476813 shutdown_connections 2024-03-28T22:45:30.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.252+0000 7f87f3400640 1 --2- 172.21.15.43:0/3104476813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87ec1042a0 0x7f87ec1107b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.252+0000 7f87f3400640 1 --2- 172.21.15.43:0/3104476813 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f87ec1038e0 0x7f87ec103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.252+0000 7f87f3400640 1 --2- 172.21.15.43:0/3104476813 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f87ec1098f0 0x7f87ec109cd0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.252+0000 7f87f3400640 1 -- 172.21.15.43:0/3104476813 >> 172.21.15.43:0/3104476813 conn(0x7f87ec0ff3b0 msgr2=0x7f87ec1017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:30.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.252+0000 7f87f3400640 1 -- 172.21.15.43:0/3104476813 shutdown_connections 2024-03-28T22:45:30.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.252+0000 7f87f3400640 1 -- 172.21.15.43:0/3104476813 wait complete. 2024-03-28T22:45:30.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.253+0000 7f87f3400640 1 Processor -- start 2024-03-28T22:45:30.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.253+0000 7f87f3400640 1 -- start start 2024-03-28T22:45:30.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.254+0000 7f87f3400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f87ec1038e0 0x7f87ec19a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:30.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.254+0000 7f87f3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f87ec1042a0 0x7f87ec19a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:30.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.254+0000 7f87f3400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87ec1098f0 0x7f87ec19ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:30.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.254+0000 7f87f3400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f87ec113d00 con 0x7f87ec1042a0 2024-03-28T22:45:30.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.254+0000 7f87f3400640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f87ec113b80 con 0x7f87ec1038e0 2024-03-28T22:45:30.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.254+0000 7f87f3400640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f87ec113e80 con 0x7f87ec1098f0 2024-03-28T22:45:30.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.254+0000 7f87f0a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f87ec1038e0 0x7f87ec19a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:30.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.254+0000 7f87f0a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f87ec1038e0 0x7f87ec19a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33772/0 (socket says 172.21.15.43:33772) 2024-03-28T22:45:30.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.254+0000 7f87f0a00640 1 -- 172.21.15.43:0/1447068464 learned_addr learned my addr 172.21.15.43:0/1447068464 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:30.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.254+0000 7f87ebe00640 1 --2- 172.21.15.43:0/1447068464 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f87ec1042a0 0x7f87ec19a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:30.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.254+0000 7f87f1400640 1 --2- 172.21.15.43:0/1447068464 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87ec1098f0 0x7f87ec19ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:30.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.255+0000 7f87f0a00640 1 -- 172.21.15.43:0/1447068464 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87ec1098f0 msgr2=0x7f87ec19ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:30.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.255+0000 7f87f0a00640 1 --2- 172.21.15.43:0/1447068464 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87ec1098f0 0x7f87ec19ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.255+0000 7f87f0a00640 1 -- 172.21.15.43:0/1447068464 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f87ec1042a0 msgr2=0x7f87ec19a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:30.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.255+0000 7f87f0a00640 1 --2- 172.21.15.43:0/1447068464 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f87ec1042a0 0x7f87ec19a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.255+0000 7f87f0a00640 1 -- 172.21.15.43:0/1447068464 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87ec19f190 con 0x7f87ec1038e0 2024-03-28T22:45:30.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.255+0000 7f87ebe00640 1 --2- 172.21.15.43:0/1447068464 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f87ec1042a0 0x7f87ec19a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:45:30.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.255+0000 7f87f0a00640 1 --2- 172.21.15.43:0/1447068464 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f87ec1038e0 0x7f87ec19a1a0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f87dc005e30 tx=0x7f87dc032f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:30.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.256+0000 7f87e9600640 1 -- 172.21.15.43:0/1447068464 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87dc042020 con 0x7f87ec1038e0 2024-03-28T22:45:30.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.256+0000 7f87e9600640 1 -- 172.21.15.43:0/1447068464 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87dc034640 con 0x7f87ec1038e0 2024-03-28T22:45:30.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.256+0000 7f87e9600640 1 -- 172.21.15.43:0/1447068464 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87dc039040 con 0x7f87ec1038e0 2024-03-28T22:45:30.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.256+0000 7f87f3400640 1 -- 172.21.15.43:0/1447068464 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87ec19f420 con 0x7f87ec1038e0 2024-03-28T22:45:30.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.256+0000 7f87f3400640 1 -- 172.21.15.43:0/1447068464 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87ec1a6d10 con 0x7f87ec1038e0 2024-03-28T22:45:30.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.257+0000 7f87f3400640 1 -- 172.21.15.43:0/1447068464 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87ec068990 con 0x7f87ec1038e0 2024-03-28T22:45:30.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.259+0000 7f87e9600640 1 -- 172.21.15.43:0/1447068464 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f87dc045020 con 0x7f87ec1038e0 2024-03-28T22:45:30.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.260+0000 7f87e9600640 1 --2- 172.21.15.43:0/1447068464 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f87d00a1f70 0x7f87d00a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:30.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.260+0000 7f87e9600640 1 -- 172.21.15.43:0/1447068464 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7f87dc0f00e0 con 0x7f87ec1038e0 2024-03-28T22:45:30.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.261+0000 7f87ebe00640 1 --2- 172.21.15.43:0/1447068464 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f87d00a1f70 0x7f87d00a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:30.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.261+0000 7f87ebe00640 1 --2- 172.21.15.43:0/1447068464 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f87d00a1f70 0x7f87d00a4430 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f87ec1019b0 tx=0x7f87d4005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:30.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.262+0000 7f87e9600640 1 -- 172.21.15.43:0/1447068464 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f87dc04c020 con 0x7f87ec1038e0 2024-03-28T22:45:30.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.504+0000 7f87f3400640 1 -- 172.21.15.43:0/1447068464 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f87ec10be50 con 0x7f87ec1038e0 2024-03-28T22:45:30.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.506+0000 7f87e9600640 1 -- 172.21.15.43:0/1447068464 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11931 (secure 0 0 0) 0x7f87dc0bcf80 con 0x7f87ec1038e0 2024-03-28T22:45:30.507 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:45:30.507 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:33:28.906396+0000","modified":"2024-03-28T22:40:08.672168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24572,"mds_1":24556,"mds_2":24535,"mds_3":24523,"mds_4":24518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26717":{"gid":26717,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/215779342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":215779342},{"type":"v1","addr":"172.21.15.82:6835","nonce":215779342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24535,24556,24518,24523,24572]},"id":1}]} 2024-03-28T22:45:30.508 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 32 2024-03-28T22:45:30.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.509+0000 7f87f3400640 1 -- 172.21.15.43:0/1447068464 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f87d00a1f70 msgr2=0x7f87d00a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:30.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.509+0000 7f87f3400640 1 --2- 172.21.15.43:0/1447068464 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f87d00a1f70 0x7f87d00a4430 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f87ec1019b0 tx=0x7f87d4005f90 comp rx=0 tx=0).stop 2024-03-28T22:45:30.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.509+0000 7f87f3400640 1 -- 172.21.15.43:0/1447068464 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f87ec1038e0 msgr2=0x7f87ec19a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:30.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.509+0000 7f87f3400640 1 --2- 172.21.15.43:0/1447068464 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f87ec1038e0 0x7f87ec19a1a0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f87dc005e30 tx=0x7f87dc032f20 comp rx=0 tx=0).stop 2024-03-28T22:45:30.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.509+0000 7f87f3400640 1 -- 172.21.15.43:0/1447068464 shutdown_connections 2024-03-28T22:45:30.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.509+0000 7f87f3400640 1 --2- 172.21.15.43:0/1447068464 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f87d00a1f70 0x7f87d00a4430 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.509+0000 7f87f3400640 1 --2- 172.21.15.43:0/1447068464 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f87ec1098f0 0x7f87ec19ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.509+0000 7f87f3400640 1 --2- 172.21.15.43:0/1447068464 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f87ec1042a0 0x7f87ec19a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.509+0000 7f87f3400640 1 --2- 172.21.15.43:0/1447068464 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f87ec1038e0 0x7f87ec19a1a0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.509+0000 7f87f3400640 1 -- 172.21.15.43:0/1447068464 >> 172.21.15.43:0/1447068464 conn(0x7f87ec0ff3b0 msgr2=0x7f87ec0ffa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:30.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.509+0000 7f87f3400640 1 -- 172.21.15.43:0/1447068464 shutdown_connections 2024-03-28T22:45:30.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.509+0000 7f87f3400640 1 -- 172.21.15.43:0/1447068464 wait complete. 2024-03-28T22:45:30.526 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T22:45:30.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.757+0000 7f81d3200640 1 -- 172.21.15.43:0/1898753557 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81cc107f60 msgr2=0x7f81cc108340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:30.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.757+0000 7f81d3200640 1 --2- 172.21.15.43:0/1898753557 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81cc107f60 0x7f81cc108340 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f81bc009900 tx=0x7f81bc02bc30 comp rx=0 tx=0).stop 2024-03-28T22:45:30.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.757+0000 7f81d3200640 1 -- 172.21.15.43:0/1898753557 shutdown_connections 2024-03-28T22:45:30.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.757+0000 7f81d3200640 1 --2- 172.21.15.43:0/1898753557 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f81cc108910 0x7f81cc10c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.757+0000 7f81d3200640 1 --2- 172.21.15.43:0/1898753557 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81cc10d030 0x7f81cc114c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.757+0000 7f81d3200640 1 --2- 172.21.15.43:0/1898753557 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81cc107f60 0x7f81cc108340 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.757+0000 7f81d3200640 1 -- 172.21.15.43:0/1898753557 >> 172.21.15.43:0/1898753557 conn(0x7f81cc075c90 msgr2=0x7f81cc0780b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:30.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.757+0000 7f81d3200640 1 -- 172.21.15.43:0/1898753557 shutdown_connections 2024-03-28T22:45:30.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.757+0000 7f81d3200640 1 -- 172.21.15.43:0/1898753557 wait complete. 2024-03-28T22:45:30.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.759+0000 7f81d3200640 1 Processor -- start 2024-03-28T22:45:30.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.759+0000 7f81d3200640 1 -- start start 2024-03-28T22:45:30.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.759+0000 7f81d3200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81cc107f60 0x7f81cc19e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:30.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.759+0000 7f81d3200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81cc10d030 0x7f81cc19eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:30.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.759+0000 7f81d3200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f81cc1a2ef0 0x7f81cc1a33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:30.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.759+0000 7f81d3200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f81cc1180d0 con 0x7f81cc107f60 2024-03-28T22:45:30.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.759+0000 7f81d3200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f81cc117f50 con 0x7f81cc1a2ef0 2024-03-28T22:45:30.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.759+0000 7f81d3200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f81cc118250 con 0x7f81cc10d030 2024-03-28T22:45:30.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.760+0000 7f81cb400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81cc10d030 0x7f81cc19eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:30.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.760+0000 7f81cb400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81cc10d030 0x7f81cc19eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43532/0 (socket says 172.21.15.43:43532) 2024-03-28T22:45:30.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.760+0000 7f81cb400640 1 -- 172.21.15.43:0/1347494187 learned_addr learned my addr 172.21.15.43:0/1347494187 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:30.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.760+0000 7f81cbe00640 1 --2- 172.21.15.43:0/1347494187 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81cc107f60 0x7f81cc19e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:30.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.760+0000 7f81d1200640 1 --2- 172.21.15.43:0/1347494187 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f81cc1a2ef0 0x7f81cc1a33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:30.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.760+0000 7f81cb400640 1 -- 172.21.15.43:0/1347494187 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f81cc1a2ef0 msgr2=0x7f81cc1a33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:30.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.760+0000 7f81cb400640 1 --2- 172.21.15.43:0/1347494187 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f81cc1a2ef0 0x7f81cc1a33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.760+0000 7f81cb400640 1 -- 172.21.15.43:0/1347494187 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81cc107f60 msgr2=0x7f81cc19e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:30.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.760+0000 7f81cb400640 1 --2- 172.21.15.43:0/1347494187 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81cc107f60 0x7f81cc19e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.760+0000 7f81cb400640 1 -- 172.21.15.43:0/1347494187 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81cc1a39e0 con 0x7f81cc10d030 2024-03-28T22:45:30.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.760+0000 7f81cb400640 1 --2- 172.21.15.43:0/1347494187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81cc10d030 0x7f81cc19eb70 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f81b800e9a0 tx=0x7f81b800ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:30.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.761+0000 7f81c8c00640 1 -- 172.21.15.43:0/1347494187 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81b800cd80 con 0x7f81cc10d030 2024-03-28T22:45:30.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.761+0000 7f81c8c00640 1 -- 172.21.15.43:0/1347494187 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81b80035f0 con 0x7f81cc10d030 2024-03-28T22:45:30.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.761+0000 7f81c8c00640 1 -- 172.21.15.43:0/1347494187 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81b8010670 con 0x7f81cc10d030 2024-03-28T22:45:30.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.761+0000 7f81d3200640 1 -- 172.21.15.43:0/1347494187 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81cc1a3cd0 con 0x7f81cc10d030 2024-03-28T22:45:30.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.761+0000 7f81d3200640 1 -- 172.21.15.43:0/1347494187 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81cc1ab470 con 0x7f81cc10d030 2024-03-28T22:45:30.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.765+0000 7f81d602f740 1 Processor -- start 2024-03-28T22:45:30.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.765+0000 7f81d602f740 1 -- start start 2024-03-28T22:45:30.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.766+0000 7f81d602f740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc020e0380 0x55fc020e0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:30.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.766+0000 7f81d602f740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc020e0ca0 0x55fc020e9340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:30.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.766+0000 7f81d602f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc020e9880 0x55fc020ebc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:30.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.766+0000 7f81d602f740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55fc02044050 con 0x55fc020e0380 2024-03-28T22:45:30.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.766+0000 7f81d602f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55fc01fecce0 con 0x55fc020e9880 2024-03-28T22:45:30.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.766+0000 7f81d602f740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55fc020400f0 con 0x55fc020e0ca0 2024-03-28T22:45:30.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.766+0000 7f81cbe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc020e0ca0 0x55fc020e9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:30.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.766+0000 7f81cbe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc020e0ca0 0x55fc020e9340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43540/0 (socket says 172.21.15.43:43540) 2024-03-28T22:45:30.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.766+0000 7f81cbe00640 1 -- 172.21.15.43:0/2364296283 learned_addr learned my addr 172.21.15.43:0/2364296283 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:30.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.766+0000 7f81d1200640 1 --2- 172.21.15.43:0/2364296283 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc020e0380 0x55fc020e0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:30.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.766+0000 7f81c8c00640 1 -- 172.21.15.43:0/1347494187 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f81b80037a0 con 0x7f81cc10d030 2024-03-28T22:45:30.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.767+0000 7f81cbe00640 1 -- 172.21.15.43:0/2364296283 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc020e9880 msgr2=0x55fc020ebc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:30.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.767+0000 7f81cb400640 1 --2- 172.21.15.43:0/2364296283 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc020e9880 0x55fc020ebc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:30.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.767+0000 7f81cbe00640 1 --2- 172.21.15.43:0/2364296283 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc020e9880 0x55fc020ebc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.767+0000 7f81cbe00640 1 -- 172.21.15.43:0/2364296283 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc020e0380 msgr2=0x55fc020e0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:30.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.767+0000 7f81cbe00640 1 --2- 172.21.15.43:0/2364296283 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc020e0380 0x55fc020e0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.767+0000 7f81cbe00640 1 -- 172.21.15.43:0/2364296283 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc01d583f0 con 0x55fc020e0ca0 2024-03-28T22:45:30.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.767+0000 7f81cb400640 1 --2- 172.21.15.43:0/2364296283 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc020e9880 0x55fc020ebc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:45:30.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.767+0000 7f81d1200640 1 --2- 172.21.15.43:0/2364296283 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc020e0380 0x55fc020e0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:45:30.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.767+0000 7f81cbe00640 1 --2- 172.21.15.43:0/2364296283 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc020e0ca0 0x55fc020e9340 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f81bc02cfd0 tx=0x7f81bc005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48f2bf274dfd87b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:30.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.767+0000 7f81c1600640 1 -- 172.21.15.43:0/2364296283 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81bc042070 con 0x55fc020e0ca0 2024-03-28T22:45:30.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.767+0000 7f81c1600640 1 -- 172.21.15.43:0/2364296283 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81bc02ca80 con 0x55fc020e0ca0 2024-03-28T22:45:30.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.767+0000 7f81c1600640 1 -- 172.21.15.43:0/2364296283 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81bc004f00 con 0x55fc020e0ca0 2024-03-28T22:45:30.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.768+0000 7f81c8c00640 1 --2- 172.21.15.43:0/1347494187 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f81b00a1f70 0x7f81b00a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:30.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.768+0000 7f81d602f740 1 -- 172.21.15.43:0/2364296283 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc020e0ca0 msgr2=0x55fc020e9340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:30.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.768+0000 7f81d602f740 1 --2- 172.21.15.43:0/2364296283 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc020e0ca0 0x55fc020e9340 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f81bc02cfd0 tx=0x7f81bc005c00 comp rx=0 tx=0).stop 2024-03-28T22:45:30.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.768+0000 7f81c8c00640 1 -- 172.21.15.43:0/1347494187 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7f81b80ce7a0 con 0x7f81cc10d030 2024-03-28T22:45:30.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.768+0000 7f81d602f740 1 -- 172.21.15.43:0/2364296283 shutdown_connections 2024-03-28T22:45:30.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.768+0000 7f81d602f740 1 --2- 172.21.15.43:0/2364296283 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc020e9880 0x55fc020ebc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.768+0000 7f81d602f740 1 --2- 172.21.15.43:0/2364296283 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc020e0ca0 0x55fc020e9340 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.768+0000 7f81d602f740 1 --2- 172.21.15.43:0/2364296283 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc020e0380 0x55fc020e0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.768+0000 7f81d602f740 1 -- 172.21.15.43:0/2364296283 >> 172.21.15.43:0/2364296283 conn(0x55fc020daab0 msgr2=0x55fc020dae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:30.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.768+0000 7f81d1200640 1 --2- 172.21.15.43:0/1347494187 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f81b00a1f70 0x7f81b00a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:30.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.768+0000 7f81d602f740 1 -- 172.21.15.43:0/2364296283 shutdown_connections 2024-03-28T22:45:30.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.768+0000 7f81d602f740 1 -- 172.21.15.43:0/2364296283 wait complete. 2024-03-28T22:45:30.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.769+0000 7f81d602f740 1 -- start start 2024-03-28T22:45:30.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.769+0000 7f81d1200640 1 --2- 172.21.15.43:0/1347494187 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f81b00a1f70 0x7f81b00a4430 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f81c4005fd0 tx=0x7f81c4009500 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:30.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.769+0000 7f81d602f740 1 Processor -- start 2024-03-28T22:45:30.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.769+0000 7f81d602f740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc020e0380 0x55fc020e6930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:30.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.770+0000 7f81d602f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc020e0ca0 0x55fc020e6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:30.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.770+0000 7f81d602f740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc020e9880 0x55fc020e73b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:30.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.770+0000 7f81d602f740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55fc020ec610 con 0x55fc020e0380 2024-03-28T22:45:30.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.770+0000 7f81d602f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55fc01fe22f0 con 0x55fc020e0ca0 2024-03-28T22:45:30.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.770+0000 7f81d602f740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55fc020ec790 con 0x55fc020e9880 2024-03-28T22:45:30.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.770+0000 7f81d602f740 10 client.0 _ll_register_callbacks cb 0x55fc01d57be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:45:30.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.770+0000 7f81cb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc020e0ca0 0x55fc020e6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:30.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.770+0000 7f81d1200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc020e9880 0x55fc020e73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:30.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.770+0000 7f81cb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc020e0ca0 0x55fc020e6e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33804/0 (socket says 172.21.15.43:33804) 2024-03-28T22:45:30.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.770+0000 7f81d1200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc020e9880 0x55fc020e73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43548/0 (socket says 172.21.15.43:43548) 2024-03-28T22:45:30.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.770+0000 7f81cb400640 1 -- 172.21.15.43:0/2511533284 learned_addr learned my addr 172.21.15.43:0/2511533284 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:30.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.770+0000 7f81cbe00640 1 --2- 172.21.15.43:0/2511533284 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc020e0380 0x55fc020e6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:30.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.770+0000 7f81cb400640 1 -- 172.21.15.43:0/2511533284 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc020e9880 msgr2=0x55fc020e73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:30.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.770+0000 7f81cb400640 1 --2- 172.21.15.43:0/2511533284 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc020e9880 0x55fc020e73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.770+0000 7f81cb400640 1 -- 172.21.15.43:0/2511533284 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc020e0380 msgr2=0x55fc020e6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:30.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.770+0000 7f81cb400640 1 --2- 172.21.15.43:0/2511533284 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc020e0380 0x55fc020e6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.770+0000 7f81cb400640 1 -- 172.21.15.43:0/2511533284 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc01e227d0 con 0x55fc020e0ca0 2024-03-28T22:45:30.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.770+0000 7f81d1200640 1 --2- 172.21.15.43:0/2511533284 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc020e9880 0x55fc020e73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:45:30.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.771+0000 7f81cb400640 1 --2- 172.21.15.43:0/2511533284 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc020e0ca0 0x55fc020e6e70 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f81b800c6a0 tx=0x7f81b809a6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3de8ccd9061f0487 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:30.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.771+0000 7f81b6000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:45:30.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.771+0000 7f81b6000640 1 -- 172.21.15.43:0/2511533284 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81b809eee0 con 0x55fc020e0ca0 2024-03-28T22:45:30.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.771+0000 7f81b6000640 1 -- 172.21.15.43:0/2511533284 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7f81b80cbd70 con 0x55fc020e0ca0 2024-03-28T22:45:30.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.771+0000 7f81d602f740 1 -- 172.21.15.43:0/2511533284 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f81bc02ca80 con 0x55fc020e0ca0 2024-03-28T22:45:30.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.771+0000 7f81b6000640 1 -- 172.21.15.43:0/2511533284 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f81b80cbf70 con 0x55fc020e0ca0 2024-03-28T22:45:30.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.771+0000 7f81d602f740 10 client.28511 fetch_fsmap learned FSMap version 32 2024-03-28T22:45:30.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.771+0000 7f81d602f740 1 -- 172.21.15.43:0/2511533284 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55fc020f8910 con 0x55fc020e0ca0 2024-03-28T22:45:30.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.771+0000 7f81b6000640 1 -- 172.21.15.43:0/2511533284 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81bc02ca80 con 0x55fc020e0ca0 2024-03-28T22:45:30.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.771+0000 7f81b6000640 1 -- 172.21.15.43:0/2511533284 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f81b809a1e0 con 0x55fc020e0ca0 2024-03-28T22:45:30.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.772+0000 7f81d602f740 10 client.28511 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:45:30.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.772+0000 7f81d602f740 10 client.28511 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:45:30.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.772+0000 7f81d602f740 20 client.28511 populate_metadata read hostname 'smithi043' 2024-03-28T22:45:30.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.772+0000 7f81d602f740 1 --2- 172.21.15.43:0/2511533284 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55fc020f9dd0 0x55fc020fc240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:30.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.772+0000 7f81d602f740 4 client.28511 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:45:30.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.772+0000 7f81d602f740 1 -- 172.21.15.43:0/2511533284 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fc01fed680 con 0x55fc020f9dd0 2024-03-28T22:45:30.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.772+0000 7f81cbe00640 1 --2- 172.21.15.43:0/2511533284 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55fc020f9dd0 0x55fc020fc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:30.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.772+0000 7f81cbe00640 1 --2- 172.21.15.43:0/2511533284 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55fc020f9dd0 0x55fc020fc240 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e0c4bc12a51f8d8a server_cookie=d4cc98d7355ddf90 in_seq=0 out_seq=0 2024-03-28T22:45:30.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.772+0000 7f81b6000640 10 client.28511 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:45:30.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.773+0000 7f81b6000640 1 -- 172.21.15.43:0/2511533284 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fc01fed680 con 0x55fc020f9dd0 2024-03-28T22:45:30.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.773+0000 7f81b6000640 10 client.28511 handle_command_reply: tid=0 2024-03-28T22:45:30.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.773+0000 7f81d602f740 1 client.28511 shutdown 2024-03-28T22:45:30.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.774+0000 7f81d602f740 1 -- 172.21.15.43:0/2511533284 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc020e0ca0 msgr2=0x55fc020e6e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.774+0000 7f81d602f740 1 --2- 172.21.15.43:0/2511533284 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc020e0ca0 0x55fc020e6e70 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f81b800c6a0 tx=0x7f81b809a6f0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.774+0000 7f81d602f740 1 -- 172.21.15.43:0/2511533284 shutdown_connections 2024-03-28T22:45:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.774+0000 7f81d602f740 1 --2- 172.21.15.43:0/2511533284 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55fc020f9dd0 0x55fc020fc240 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.774+0000 7f81d602f740 1 --2- 172.21.15.43:0/2511533284 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc020e0ca0 0x55fc020e6e70 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.774+0000 7f81d602f740 1 --2- 172.21.15.43:0/2511533284 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc020e9880 0x55fc020e73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.774+0000 7f81b6000640 0 client.28511 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:45:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.774+0000 7f81d602f740 1 --2- 172.21.15.43:0/2511533284 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc020e0380 0x55fc020e6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.774+0000 7f81d602f740 1 -- 172.21.15.43:0/2511533284 >> 172.21.15.43:0/2511533284 conn(0x55fc020daab0 msgr2=0x55fc020dc800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.774+0000 7f81d602f740 1 -- 172.21.15.43:0/2511533284 shutdown_connections 2024-03-28T22:45:30.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.774+0000 7f81d602f740 1 -- 172.21.15.43:0/2511533284 wait complete. 2024-03-28T22:45:30.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.774+0000 7f81d602f740 20 client.28511 trim_cache size 0 max 16384 2024-03-28T22:45:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.789+0000 7f81d602f740 1 Processor -- start 2024-03-28T22:45:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.789+0000 7f81d602f740 1 -- start start 2024-03-28T22:45:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.789+0000 7f81d602f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc020e0380 0x55fc020e9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.789+0000 7f81d602f740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc020e0ca0 0x55fc020f53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.789+0000 7f81d602f740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc020e9880 0x55fc02101b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.789+0000 7f81d602f740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f81cc1b5340 con 0x55fc020e0ca0 2024-03-28T22:45:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.790+0000 7f81d602f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f81cc1b6e80 con 0x55fc020e0380 2024-03-28T22:45:30.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.790+0000 7f81d602f740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f81cc1b5a10 con 0x55fc020e9880 2024-03-28T22:45:30.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.790+0000 7f81cb400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc020e0ca0 0x55fc020f53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:30.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.790+0000 7f81d1200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc020e9880 0x55fc02101b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:30.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.790+0000 7f81cbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc020e0380 0x55fc020e9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:30.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.790+0000 7f81d1200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc020e9880 0x55fc02101b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43562/0 (socket says 172.21.15.43:43562) 2024-03-28T22:45:30.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.790+0000 7f81d1200640 1 -- 172.21.15.43:0/4136676408 learned_addr learned my addr 172.21.15.43:0/4136676408 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:30.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.790+0000 7f81d1200640 1 -- 172.21.15.43:0/4136676408 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc020e0380 msgr2=0x55fc020e9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:30.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.790+0000 7f81d1200640 1 --2- 172.21.15.43:0/4136676408 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc020e0380 0x55fc020e9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.790+0000 7f81d1200640 1 -- 172.21.15.43:0/4136676408 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc020e0ca0 msgr2=0x55fc020f53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:30.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.790+0000 7f81d1200640 1 --2- 172.21.15.43:0/4136676408 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc020e0ca0 0x55fc020f53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.790+0000 7f81d1200640 1 -- 172.21.15.43:0/4136676408 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc020e2550 con 0x55fc020e9880 2024-03-28T22:45:30.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.790+0000 7f81cb400640 1 --2- 172.21.15.43:0/4136676408 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc020e0ca0 0x55fc020f53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:45:30.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.791+0000 7f81d1200640 1 --2- 172.21.15.43:0/4136676408 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc020e9880 0x55fc02101b10 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f81c4004230 tx=0x7f81c401a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6f51ec2319be91e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:30.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.791+0000 7f81c0c00640 1 -- 172.21.15.43:0/4136676408 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c400fa00 con 0x55fc020e9880 2024-03-28T22:45:30.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.791+0000 7f81c0c00640 1 -- 172.21.15.43:0/4136676408 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81c400fba0 con 0x55fc020e9880 2024-03-28T22:45:30.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.791+0000 7f81c0c00640 1 -- 172.21.15.43:0/4136676408 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c401c400 con 0x55fc020e9880 2024-03-28T22:45:30.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.791+0000 7f81d602f740 1 -- 172.21.15.43:0/4136676408 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc020e9880 msgr2=0x55fc02101b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:30.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.791+0000 7f81d602f740 1 --2- 172.21.15.43:0/4136676408 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc020e9880 0x55fc02101b10 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f81c4004230 tx=0x7f81c401a000 comp rx=0 tx=0).stop 2024-03-28T22:45:30.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.791+0000 7f81d602f740 1 -- 172.21.15.43:0/4136676408 shutdown_connections 2024-03-28T22:45:30.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.792+0000 7f81d602f740 1 --2- 172.21.15.43:0/4136676408 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc020e9880 0x55fc02101b10 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.792+0000 7f81d602f740 1 --2- 172.21.15.43:0/4136676408 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc020e0ca0 0x55fc020f53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.792+0000 7f81d602f740 1 --2- 172.21.15.43:0/4136676408 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc020e0380 0x55fc020e9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.792+0000 7f81d602f740 1 -- 172.21.15.43:0/4136676408 >> 172.21.15.43:0/4136676408 conn(0x55fc020daab0 msgr2=0x55fc020d5ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:30.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.792+0000 7f81d602f740 1 -- 172.21.15.43:0/4136676408 shutdown_connections 2024-03-28T22:45:30.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.792+0000 7f81d602f740 1 -- 172.21.15.43:0/4136676408 wait complete. 2024-03-28T22:45:30.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.792+0000 7f81d602f740 1 -- start start 2024-03-28T22:45:30.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.792+0000 7f81d602f740 1 Processor -- start 2024-03-28T22:45:30.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.793+0000 7f81d602f740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc020e0380 0x55fc020f1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:30.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.793+0000 7f81d602f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc020e0ca0 0x55fc020e31c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:30.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.793+0000 7f81d602f740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc020e9880 0x55fc020e3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:30.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.793+0000 7f81d602f740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f81cc1b67b0 con 0x55fc020e0380 2024-03-28T22:45:30.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.793+0000 7f81d602f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f81cc1b60e0 con 0x55fc020e0ca0 2024-03-28T22:45:30.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.793+0000 7f81d602f740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f81cc1b3340 con 0x55fc020e9880 2024-03-28T22:45:30.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.793+0000 7f81d602f740 10 client.0 _ll_register_callbacks cb 0x7f81b800e8f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:45:30.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.793+0000 7f81cb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc020e0ca0 0x55fc020e31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:30.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.793+0000 7f81cb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc020e0ca0 0x55fc020e31c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33824/0 (socket says 172.21.15.43:33824) 2024-03-28T22:45:30.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.793+0000 7f81cb400640 1 -- 172.21.15.43:0/2182206018 learned_addr learned my addr 172.21.15.43:0/2182206018 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:30.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.793+0000 7f81cbe00640 1 --2- 172.21.15.43:0/2182206018 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc020e0380 0x55fc020f1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:30.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.793+0000 7f81d1200640 1 --2- 172.21.15.43:0/2182206018 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc020e9880 0x55fc020e3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:30.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.793+0000 7f81cb400640 1 -- 172.21.15.43:0/2182206018 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc020e9880 msgr2=0x55fc020e3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:30.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.793+0000 7f81cb400640 1 --2- 172.21.15.43:0/2182206018 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc020e9880 0x55fc020e3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.793+0000 7f81cb400640 1 -- 172.21.15.43:0/2182206018 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc020e0380 msgr2=0x55fc020f1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:30.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.793+0000 7f81cb400640 1 --2- 172.21.15.43:0/2182206018 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc020e0380 0x55fc020f1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.793+0000 7f81cb400640 1 -- 172.21.15.43:0/2182206018 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc020e3f30 con 0x55fc020e0ca0 2024-03-28T22:45:30.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.794+0000 7f81cb400640 1 --2- 172.21.15.43:0/2182206018 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc020e0ca0 0x55fc020e31c0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f81b8097710 tx=0x7f81b8000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a85ab80f166e2bc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:30.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.794+0000 7f81b6000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:45:30.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.794+0000 7f81b6000640 1 -- 172.21.15.43:0/2182206018 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81b809b490 con 0x55fc020e0ca0 2024-03-28T22:45:30.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.794+0000 7f81d602f740 1 -- 172.21.15.43:0/2182206018 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f81c400fba0 con 0x55fc020e0ca0 2024-03-28T22:45:30.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.794+0000 7f81b6000640 1 -- 172.21.15.43:0/2182206018 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7f81b80cbd70 con 0x55fc020e0ca0 2024-03-28T22:45:30.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.794+0000 7f81b6000640 1 -- 172.21.15.43:0/2182206018 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f81b80cbf70 con 0x55fc020e0ca0 2024-03-28T22:45:30.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.794+0000 7f81d602f740 10 client.28517 fetch_fsmap learned FSMap version 32 2024-03-28T22:45:30.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.794+0000 7f81d602f740 1 -- 172.21.15.43:0/2182206018 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55fc021600e0 con 0x55fc020e0ca0 2024-03-28T22:45:30.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.794+0000 7f81b6000640 1 -- 172.21.15.43:0/2182206018 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81c400fba0 con 0x55fc020e0ca0 2024-03-28T22:45:30.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.794+0000 7f81b6000640 1 -- 172.21.15.43:0/2182206018 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f81b8081060 con 0x55fc020e0ca0 2024-03-28T22:45:30.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.795+0000 7f81d602f740 10 client.28517 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:45:30.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.795+0000 7f81d602f740 10 client.28517 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:45:30.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.795+0000 7f81d602f740 20 client.28517 populate_metadata read hostname 'smithi043' 2024-03-28T22:45:30.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.795+0000 7f81d602f740 1 --2- 172.21.15.43:0/2182206018 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55fc021606b0 0x55fc02162ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:30.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.795+0000 7f81d602f740 4 client.28517 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:45:30.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.795+0000 7f81d602f740 1 -- 172.21.15.43:0/2182206018 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fc02051590 con 0x55fc021606b0 2024-03-28T22:45:30.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.795+0000 7f81cbe00640 1 --2- 172.21.15.43:0/2182206018 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55fc021606b0 0x55fc02162ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:30.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.795+0000 7f81cbe00640 1 --2- 172.21.15.43:0/2182206018 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55fc021606b0 0x55fc02162ad0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=657cd921393456e2 server_cookie=ae8c07681c471c4b in_seq=0 out_seq=0 2024-03-28T22:45:30.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.795+0000 7f81b6000640 10 client.28517 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:45:30.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.795+0000 7f81b6000640 1 -- 172.21.15.43:0/2182206018 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fc02051590 con 0x55fc021606b0 2024-03-28T22:45:30.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.795+0000 7f81b6000640 10 client.28517 handle_command_reply: tid=0 2024-03-28T22:45:30.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.796+0000 7f81d602f740 1 client.28517 shutdown 2024-03-28T22:45:30.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.796+0000 7f81d602f740 1 -- 172.21.15.43:0/2182206018 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc020e0ca0 msgr2=0x55fc020e31c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:30.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.796+0000 7f81d602f740 1 --2- 172.21.15.43:0/2182206018 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc020e0ca0 0x55fc020e31c0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f81b8097710 tx=0x7f81b8000bc0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.796+0000 7f81d602f740 1 -- 172.21.15.43:0/2182206018 shutdown_connections 2024-03-28T22:45:30.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.796+0000 7f81d602f740 1 --2- 172.21.15.43:0/2182206018 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55fc021606b0 0x55fc02162ad0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.796+0000 7f81d602f740 1 --2- 172.21.15.43:0/2182206018 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fc020e0ca0 0x55fc020e31c0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.796+0000 7f81d602f740 1 --2- 172.21.15.43:0/2182206018 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fc020e9880 0x55fc020e3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.796+0000 7f81b6000640 0 client.28517 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:45:30.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.796+0000 7f81d602f740 1 --2- 172.21.15.43:0/2182206018 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fc020e0380 0x55fc020f1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.796+0000 7f81d602f740 1 -- 172.21.15.43:0/2182206018 >> 172.21.15.43:0/2182206018 conn(0x55fc020daab0 msgr2=0x55fc020e68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:30.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.796+0000 7f81d602f740 1 -- 172.21.15.43:0/2182206018 shutdown_connections 2024-03-28T22:45:30.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.796+0000 7f81d602f740 1 -- 172.21.15.43:0/2182206018 wait complete. 2024-03-28T22:45:30.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.796+0000 7f81d602f740 20 client.28517 trim_cache size 0 max 16384 2024-03-28T22:45:30.802 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:45:30.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.797+0000 7f81d3200640 1 -- 172.21.15.43:0/1347494187 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f81b00a1f70 msgr2=0x7f81b00a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:30.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.797+0000 7f81d3200640 1 --2- 172.21.15.43:0/1347494187 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f81b00a1f70 0x7f81b00a4430 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f81c4005fd0 tx=0x7f81c4009500 comp rx=0 tx=0).stop 2024-03-28T22:45:30.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.797+0000 7f81d3200640 1 -- 172.21.15.43:0/1347494187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81cc10d030 msgr2=0x7f81cc19eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:30.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.797+0000 7f81d3200640 1 --2- 172.21.15.43:0/1347494187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81cc10d030 0x7f81cc19eb70 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f81b800e9a0 tx=0x7f81b800ee60 comp rx=0 tx=0).stop 2024-03-28T22:45:30.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.797+0000 7f81d3200640 1 -- 172.21.15.43:0/1347494187 shutdown_connections 2024-03-28T22:45:30.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.797+0000 7f81d3200640 1 --2- 172.21.15.43:0/1347494187 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f81b00a1f70 0x7f81b00a4430 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.797+0000 7f81d3200640 1 --2- 172.21.15.43:0/1347494187 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f81cc1a2ef0 0x7f81cc1a33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.797+0000 7f81d3200640 1 --2- 172.21.15.43:0/1347494187 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81cc10d030 0x7f81cc19eb70 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.797+0000 7f81d3200640 1 --2- 172.21.15.43:0/1347494187 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81cc107f60 0x7f81cc19e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:30.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.797+0000 7f81d3200640 1 -- 172.21.15.43:0/1347494187 >> 172.21.15.43:0/1347494187 conn(0x7f81cc075c90 msgr2=0x7f81cc076fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:30.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.797+0000 7f81d3200640 1 -- 172.21.15.43:0/1347494187 shutdown_connections 2024-03-28T22:45:30.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:30.797+0000 7f81d3200640 1 -- 172.21.15.43:0/1347494187 wait complete. 2024-03-28T22:45:30.811 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T22:45:31.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.039+0000 7f0d1fe00640 1 -- 172.21.15.43:0/2874764086 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0d2010cf50 msgr2=0x7f0d20114b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.039+0000 7f0d1fe00640 1 --2- 172.21.15.43:0/2874764086 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0d2010cf50 0x7f0d20114b20 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f0d18009ca0 tx=0x7f0d1802c0f0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.040+0000 7f0d1fe00640 1 -- 172.21.15.43:0/2874764086 shutdown_connections 2024-03-28T22:45:31.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.040+0000 7f0d1fe00640 1 --2- 172.21.15.43:0/2874764086 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0d2010cf50 0x7f0d20114b20 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.040+0000 7f0d1fe00640 1 --2- 172.21.15.43:0/2874764086 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d201088b0 0x7f0d2010c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.040+0000 7f0d1fe00640 1 --2- 172.21.15.43:0/2874764086 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0d20107f00 0x7f0d201082e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.040+0000 7f0d1fe00640 1 -- 172.21.15.43:0/2874764086 >> 172.21.15.43:0/2874764086 conn(0x7f0d20077570 msgr2=0x7f0d20106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:31.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.040+0000 7f0d1fe00640 1 -- 172.21.15.43:0/2874764086 shutdown_connections 2024-03-28T22:45:31.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.040+0000 7f0d1fe00640 1 -- 172.21.15.43:0/2874764086 wait complete. 2024-03-28T22:45:31.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.041+0000 7f0d1fe00640 1 Processor -- start 2024-03-28T22:45:31.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.041+0000 7f0d1fe00640 1 -- start start 2024-03-28T22:45:31.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.042+0000 7f0d1fe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d20107f00 0x7f0d2019e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.042+0000 7f0d1fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0d2010cf50 0x7f0d2019e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.042+0000 7f0d1fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0d201a2d00 0x7f0d201a31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.042+0000 7f0d1fe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0d20118020 con 0x7f0d201a2d00 2024-03-28T22:45:31.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.042+0000 7f0d1fe00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f0d20117ea0 con 0x7f0d2010cf50 2024-03-28T22:45:31.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.042+0000 7f0d1fe00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f0d201181a0 con 0x7f0d20107f00 2024-03-28T22:45:31.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.042+0000 7f0d1ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d20107f00 0x7f0d2019e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.042+0000 7f0d1ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d20107f00 0x7f0d2019e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43588/0 (socket says 172.21.15.43:43588) 2024-03-28T22:45:31.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.042+0000 7f0d1ea00640 1 -- 172.21.15.43:0/2512393181 learned_addr learned my addr 172.21.15.43:0/2512393181 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:31.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.042+0000 7f0d17e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0d2010cf50 0x7f0d2019e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.042+0000 7f0d1f400640 1 --2- 172.21.15.43:0/2512393181 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0d201a2d00 0x7f0d201a31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.043+0000 7f0d1ea00640 1 -- 172.21.15.43:0/2512393181 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0d2010cf50 msgr2=0x7f0d2019e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.043+0000 7f0d1ea00640 1 --2- 172.21.15.43:0/2512393181 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0d2010cf50 0x7f0d2019e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.043+0000 7f0d1ea00640 1 -- 172.21.15.43:0/2512393181 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0d201a2d00 msgr2=0x7f0d201a31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.043+0000 7f0d1ea00640 1 --2- 172.21.15.43:0/2512393181 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0d201a2d00 0x7f0d201a31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.043+0000 7f0d1ea00640 1 -- 172.21.15.43:0/2512393181 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d201a37f0 con 0x7f0d20107f00 2024-03-28T22:45:31.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.043+0000 7f0d17e00640 1 --2- 172.21.15.43:0/2512393181 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0d2010cf50 0x7f0d2019e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:45:31.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.043+0000 7f0d1ea00640 1 --2- 172.21.15.43:0/2512393181 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d20107f00 0x7f0d2019e440 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f0d1000d950 tx=0x7f0d1000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:31.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.043+0000 7f0d24a00640 1 -- 172.21.15.43:0/2512393181 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d10017070 con 0x7f0d20107f00 2024-03-28T22:45:31.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.043+0000 7f0d24a00640 1 -- 172.21.15.43:0/2512393181 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d10009c30 con 0x7f0d20107f00 2024-03-28T22:45:31.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.043+0000 7f0d24a00640 1 -- 172.21.15.43:0/2512393181 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d1000f6b0 con 0x7f0d20107f00 2024-03-28T22:45:31.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.043+0000 7f0d1fe00640 1 -- 172.21.15.43:0/2512393181 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d201a3ae0 con 0x7f0d20107f00 2024-03-28T22:45:31.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.044+0000 7f0d1fe00640 1 -- 172.21.15.43:0/2512393181 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d201ab220 con 0x7f0d20107f00 2024-03-28T22:45:31.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.046+0000 7f0d24a00640 1 -- 172.21.15.43:0/2512393181 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f0d100109d0 con 0x7f0d20107f00 2024-03-28T22:45:31.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.047+0000 7f0d28bbc740 1 Processor -- start 2024-03-28T22:45:31.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.047+0000 7f0d28bbc740 1 -- start start 2024-03-28T22:45:31.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.048+0000 7f0d28bbc740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5644dc94a550 0x5644dc94a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.048+0000 7f0d28bbc740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5644dc94ae70 0x5644dc953510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.048+0000 7f0d28bbc740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5644dc953a50 0x5644dc955e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.048+0000 7f0d28bbc740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5644dc8ae050 con 0x5644dc94ae70 2024-03-28T22:45:31.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.048+0000 7f0d28bbc740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5644dc856ce0 con 0x5644dc953a50 2024-03-28T22:45:31.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.048+0000 7f0d28bbc740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5644dc8aa0f0 con 0x5644dc94a550 2024-03-28T22:45:31.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.048+0000 7f0d1f400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5644dc94a550 0x5644dc94a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.048+0000 7f0d1f400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5644dc94a550 0x5644dc94a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43596/0 (socket says 172.21.15.43:43596) 2024-03-28T22:45:31.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.048+0000 7f0d1f400640 1 -- 172.21.15.43:0/506503285 learned_addr learned my addr 172.21.15.43:0/506503285 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:31.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.048+0000 7f0d1ea00640 1 --2- 172.21.15.43:0/506503285 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5644dc94ae70 0x5644dc953510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.048+0000 7f0d17e00640 1 --2- 172.21.15.43:0/506503285 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5644dc953a50 0x5644dc955e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.049+0000 7f0d24a00640 1 --2- 172.21.15.43:0/2512393181 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0d040a1f70 0x7f0d040a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.049+0000 7f0d17e00640 1 -- 172.21.15.43:0/506503285 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5644dc94a550 msgr2=0x5644dc94a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:31.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.049+0000 7f0d17e00640 1 --2- 172.21.15.43:0/506503285 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5644dc94a550 0x5644dc94a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.049+0000 7f0d24a00640 1 -- 172.21.15.43:0/2512393181 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7f0d100ce110 con 0x7f0d20107f00 2024-03-28T22:45:31.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.049+0000 7f0d17e00640 1 -- 172.21.15.43:0/506503285 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5644dc94ae70 msgr2=0x5644dc953510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:31.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.049+0000 7f0d17e00640 1 --2- 172.21.15.43:0/506503285 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5644dc94ae70 0x5644dc953510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.049+0000 7f0d17e00640 1 -- 172.21.15.43:0/506503285 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644dc5c23f0 con 0x5644dc953a50 2024-03-28T22:45:31.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.049+0000 7f0d1f400640 1 --2- 172.21.15.43:0/506503285 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5644dc94a550 0x5644dc94a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:45:31.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.049+0000 7f0d1ea00640 1 --2- 172.21.15.43:0/506503285 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5644dc94ae70 0x5644dc953510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:45:31.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.049+0000 7f0d1f400640 1 --2- 172.21.15.43:0/2512393181 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0d040a1f70 0x7f0d040a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.049+0000 7f0d17e00640 1 --2- 172.21.15.43:0/506503285 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5644dc953a50 0x5644dc955e40 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f0d2006ac80 tx=0x7f0d080118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3af40789ea668ba9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:31.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.049+0000 7f0d0fe00640 1 -- 172.21.15.43:0/506503285 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d080043d0 con 0x5644dc953a50 2024-03-28T22:45:31.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.050+0000 7f0d0fe00640 1 -- 172.21.15.43:0/506503285 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d08005500 con 0x5644dc953a50 2024-03-28T22:45:31.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.050+0000 7f0d0fe00640 1 -- 172.21.15.43:0/506503285 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d08003850 con 0x5644dc953a50 2024-03-28T22:45:31.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.050+0000 7f0d28bbc740 1 -- 172.21.15.43:0/506503285 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5644dc953a50 msgr2=0x5644dc955e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.050+0000 7f0d1f400640 1 --2- 172.21.15.43:0/2512393181 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0d040a1f70 0x7f0d040a4430 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0d18009a60 tx=0x7f0d1802e040 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:31.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.050+0000 7f0d28bbc740 1 --2- 172.21.15.43:0/506503285 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5644dc953a50 0x5644dc955e40 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f0d2006ac80 tx=0x7f0d080118d0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.050+0000 7f0d28bbc740 1 -- 172.21.15.43:0/506503285 shutdown_connections 2024-03-28T22:45:31.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.050+0000 7f0d28bbc740 1 --2- 172.21.15.43:0/506503285 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5644dc953a50 0x5644dc955e40 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.050+0000 7f0d28bbc740 1 --2- 172.21.15.43:0/506503285 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5644dc94ae70 0x5644dc953510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.050+0000 7f0d28bbc740 1 --2- 172.21.15.43:0/506503285 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5644dc94a550 0x5644dc94a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.050+0000 7f0d28bbc740 1 -- 172.21.15.43:0/506503285 >> 172.21.15.43:0/506503285 conn(0x5644dc944c80 msgr2=0x5644dc945060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:31.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.050+0000 7f0d28bbc740 1 -- 172.21.15.43:0/506503285 shutdown_connections 2024-03-28T22:45:31.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.050+0000 7f0d28bbc740 1 -- 172.21.15.43:0/506503285 wait complete. 2024-03-28T22:45:31.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.051+0000 7f0d28bbc740 1 -- start start 2024-03-28T22:45:31.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.051+0000 7f0d28bbc740 1 Processor -- start 2024-03-28T22:45:31.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.052+0000 7f0d28bbc740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5644dc94a550 0x5644dc950ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.052+0000 7f0d28bbc740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5644dc94ae70 0x5644dc951000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.052+0000 7f0d28bbc740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5644dc953a50 0x5644dc951540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.052+0000 7f0d28bbc740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5644dc956840 con 0x5644dc94a550 2024-03-28T22:45:31.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.052+0000 7f0d28bbc740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5644dc84c2f0 con 0x5644dc953a50 2024-03-28T22:45:31.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.052+0000 7f0d28bbc740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5644dc9569c0 con 0x5644dc94ae70 2024-03-28T22:45:31.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.052+0000 7f0d28bbc740 10 client.0 _ll_register_callbacks cb 0x5644dc5c1be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:45:31.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.052+0000 7f0d1ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5644dc94a550 0x5644dc950ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.052+0000 7f0d1f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5644dc953a50 0x5644dc951540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.052+0000 7f0d1f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5644dc953a50 0x5644dc951540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33860/0 (socket says 172.21.15.43:33860) 2024-03-28T22:45:31.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.052+0000 7f0d1f400640 1 -- 172.21.15.43:0/117724713 learned_addr learned my addr 172.21.15.43:0/117724713 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:31.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.052+0000 7f0d17e00640 1 --2- 172.21.15.43:0/117724713 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5644dc94ae70 0x5644dc951000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.052+0000 7f0d17e00640 1 -- 172.21.15.43:0/117724713 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5644dc953a50 msgr2=0x5644dc951540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:31.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.052+0000 7f0d17e00640 1 --2- 172.21.15.43:0/117724713 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5644dc953a50 0x5644dc951540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.052+0000 7f0d17e00640 1 -- 172.21.15.43:0/117724713 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5644dc94a550 msgr2=0x5644dc950ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:31.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.052+0000 7f0d17e00640 1 --2- 172.21.15.43:0/117724713 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5644dc94a550 0x5644dc950ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.052+0000 7f0d17e00640 1 -- 172.21.15.43:0/117724713 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644dc68c7d0 con 0x5644dc94ae70 2024-03-28T22:45:31.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.053+0000 7f0d17e00640 1 --2- 172.21.15.43:0/117724713 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5644dc94ae70 0x5644dc951000 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f0d2006ac80 tx=0x7f0d08004400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b3ccc6ddc31bc2a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:31.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.053+0000 7f0d0cc00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:45:31.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.053+0000 7f0d0cc00640 1 -- 172.21.15.43:0/117724713 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d08028030 con 0x5644dc94ae70 2024-03-28T22:45:31.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.053+0000 7f0d0cc00640 1 -- 172.21.15.43:0/117724713 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7f0d08005060 con 0x5644dc94ae70 2024-03-28T22:45:31.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.053+0000 7f0d28bbc740 1 -- 172.21.15.43:0/117724713 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d08005500 con 0x5644dc94ae70 2024-03-28T22:45:31.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.053+0000 7f0d0cc00640 1 -- 172.21.15.43:0/117724713 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0d08035050 con 0x5644dc94ae70 2024-03-28T22:45:31.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.053+0000 7f0d28bbc740 10 client.28465 fetch_fsmap learned FSMap version 32 2024-03-28T22:45:31.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.053+0000 7f0d28bbc740 1 -- 172.21.15.43:0/117724713 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5644dc962b30 con 0x5644dc94ae70 2024-03-28T22:45:31.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.054+0000 7f0d0cc00640 1 -- 172.21.15.43:0/117724713 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d08005500 con 0x5644dc94ae70 2024-03-28T22:45:31.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.054+0000 7f0d0cc00640 1 -- 172.21.15.43:0/117724713 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f0d0802f4d0 con 0x5644dc94ae70 2024-03-28T22:45:31.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.054+0000 7f0d28bbc740 10 client.28465 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:45:31.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.054+0000 7f0d28bbc740 10 client.28465 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:45:31.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.054+0000 7f0d28bbc740 20 client.28465 populate_metadata read hostname 'smithi043' 2024-03-28T22:45:31.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.054+0000 7f0d28bbc740 1 --2- 172.21.15.43:0/117724713 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5644dc963e80 0x5644dc966270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.054+0000 7f0d28bbc740 4 client.28465 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:45:31.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.054+0000 7f0d28bbc740 1 -- 172.21.15.43:0/117724713 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5644dc857680 con 0x5644dc963e80 2024-03-28T22:45:31.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.054+0000 7f0d1ea00640 1 --2- 172.21.15.43:0/117724713 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5644dc963e80 0x5644dc966270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.055+0000 7f0d1ea00640 1 --2- 172.21.15.43:0/117724713 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5644dc963e80 0x5644dc966270 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5c3064c6e9905baf server_cookie=c30b4be5a91db965 in_seq=0 out_seq=0 2024-03-28T22:45:31.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.055+0000 7f0d0cc00640 10 client.28465 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:45:31.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.056+0000 7f0d0cc00640 1 -- 172.21.15.43:0/117724713 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5644dc857680 con 0x5644dc963e80 2024-03-28T22:45:31.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.056+0000 7f0d0cc00640 10 client.28465 handle_command_reply: tid=0 2024-03-28T22:45:31.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.056+0000 7f0d28bbc740 1 client.28465 shutdown 2024-03-28T22:45:31.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.056+0000 7f0d28bbc740 1 -- 172.21.15.43:0/117724713 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5644dc94ae70 msgr2=0x5644dc951000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.056+0000 7f0d28bbc740 1 --2- 172.21.15.43:0/117724713 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5644dc94ae70 0x5644dc951000 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f0d2006ac80 tx=0x7f0d08004400 comp rx=0 tx=0).stop 2024-03-28T22:45:31.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.056+0000 7f0d28bbc740 1 -- 172.21.15.43:0/117724713 shutdown_connections 2024-03-28T22:45:31.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.056+0000 7f0d28bbc740 1 --2- 172.21.15.43:0/117724713 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5644dc963e80 0x5644dc966270 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.056+0000 7f0d28bbc740 1 --2- 172.21.15.43:0/117724713 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5644dc953a50 0x5644dc951540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.056+0000 7f0d28bbc740 1 --2- 172.21.15.43:0/117724713 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5644dc94ae70 0x5644dc951000 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.056+0000 7f0d0cc00640 0 client.28465 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:45:31.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.056+0000 7f0d28bbc740 1 --2- 172.21.15.43:0/117724713 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5644dc94a550 0x5644dc950ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.056+0000 7f0d28bbc740 1 -- 172.21.15.43:0/117724713 >> 172.21.15.43:0/117724713 conn(0x5644dc944c80 msgr2=0x5644dc9469d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:31.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.056+0000 7f0d28bbc740 1 -- 172.21.15.43:0/117724713 shutdown_connections 2024-03-28T22:45:31.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.056+0000 7f0d28bbc740 1 -- 172.21.15.43:0/117724713 wait complete. 2024-03-28T22:45:31.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.057+0000 7f0d28bbc740 20 client.28465 trim_cache size 0 max 16384 2024-03-28T22:45:31.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.071+0000 7f0d28bbc740 1 Processor -- start 2024-03-28T22:45:31.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.072+0000 7f0d28bbc740 1 -- start start 2024-03-28T22:45:31.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.072+0000 7f0d28bbc740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5644dc94a550 0x5644dc95f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.072+0000 7f0d28bbc740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5644dc94ae70 0x5644dc96b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.072+0000 7f0d28bbc740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5644dc953a50 0x5644dc95bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.072+0000 7f0d28bbc740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0d201b50d0 con 0x5644dc94a550 2024-03-28T22:45:31.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.072+0000 7f0d28bbc740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f0d201b6c10 con 0x5644dc94ae70 2024-03-28T22:45:31.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.072+0000 7f0d28bbc740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f0d201b57a0 con 0x5644dc953a50 2024-03-28T22:45:31.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.072+0000 7f0d1f400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5644dc953a50 0x5644dc95bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.072+0000 7f0d17e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5644dc94ae70 0x5644dc96b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.072+0000 7f0d1f400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5644dc953a50 0x5644dc95bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43620/0 (socket says 172.21.15.43:43620) 2024-03-28T22:45:31.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.072+0000 7f0d1f400640 1 -- 172.21.15.43:0/398298458 learned_addr learned my addr 172.21.15.43:0/398298458 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:31.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.072+0000 7f0d1ea00640 1 --2- 172.21.15.43:0/398298458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5644dc94a550 0x5644dc95f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.073+0000 7f0d17e00640 1 -- 172.21.15.43:0/398298458 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5644dc953a50 msgr2=0x5644dc95bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:31.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.073+0000 7f0d17e00640 1 --2- 172.21.15.43:0/398298458 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5644dc953a50 0x5644dc95bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.073+0000 7f0d17e00640 1 -- 172.21.15.43:0/398298458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5644dc94a550 msgr2=0x5644dc95f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:31.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.073+0000 7f0d17e00640 1 --2- 172.21.15.43:0/398298458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5644dc94a550 0x5644dc95f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.073+0000 7f0d17e00640 1 -- 172.21.15.43:0/398298458 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644dc94aca0 con 0x5644dc94ae70 2024-03-28T22:45:31.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.073+0000 7f0d17e00640 1 --2- 172.21.15.43:0/398298458 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5644dc94ae70 0x5644dc96b990 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f0d0802fa10 tx=0x7f0d08004110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa9055192947205f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:31.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.073+0000 7f0d0f400640 1 -- 172.21.15.43:0/398298458 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d08028040 con 0x5644dc94ae70 2024-03-28T22:45:31.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.073+0000 7f0d0f400640 1 -- 172.21.15.43:0/398298458 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d08005540 con 0x5644dc94ae70 2024-03-28T22:45:31.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.073+0000 7f0d0f400640 1 -- 172.21.15.43:0/398298458 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d0802f340 con 0x5644dc94ae70 2024-03-28T22:45:31.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.074+0000 7f0d28bbc740 1 -- 172.21.15.43:0/398298458 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5644dc94ae70 msgr2=0x5644dc96b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.074+0000 7f0d28bbc740 1 --2- 172.21.15.43:0/398298458 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5644dc94ae70 0x5644dc96b990 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f0d0802fa10 tx=0x7f0d08004110 comp rx=0 tx=0).stop 2024-03-28T22:45:31.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.074+0000 7f0d28bbc740 1 -- 172.21.15.43:0/398298458 shutdown_connections 2024-03-28T22:45:31.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.074+0000 7f0d28bbc740 1 --2- 172.21.15.43:0/398298458 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5644dc94ae70 0x5644dc96b990 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.074+0000 7f0d28bbc740 1 --2- 172.21.15.43:0/398298458 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5644dc953a50 0x5644dc95bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.074+0000 7f0d28bbc740 1 --2- 172.21.15.43:0/398298458 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5644dc94a550 0x5644dc95f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.074+0000 7f0d28bbc740 1 -- 172.21.15.43:0/398298458 >> 172.21.15.43:0/398298458 conn(0x5644dc944c80 msgr2=0x5644dc943530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:31.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.074+0000 7f0d28bbc740 1 -- 172.21.15.43:0/398298458 shutdown_connections 2024-03-28T22:45:31.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.074+0000 7f0d28bbc740 1 -- 172.21.15.43:0/398298458 wait complete. 2024-03-28T22:45:31.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.075+0000 7f0d28bbc740 1 -- start start 2024-03-28T22:45:31.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.075+0000 7f0d28bbc740 1 Processor -- start 2024-03-28T22:45:31.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.075+0000 7f0d28bbc740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5644dc94a550 0x5644dc957610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.075+0000 7f0d28bbc740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5644dc94ae70 0x5644dc957b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.075+0000 7f0d28bbc740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5644dc953a50 0x5644dc958090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.075+0000 7f0d28bbc740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0d201b6540 con 0x5644dc953a50 2024-03-28T22:45:31.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.075+0000 7f0d28bbc740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f0d201b5e70 con 0x5644dc94a550 2024-03-28T22:45:31.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.075+0000 7f0d28bbc740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f0d201b30d0 con 0x5644dc94ae70 2024-03-28T22:45:31.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.075+0000 7f0d28bbc740 10 client.0 _ll_register_callbacks cb 0x7f0d10096a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:45:31.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.075+0000 7f0d1ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5644dc94a550 0x5644dc957610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.076+0000 7f0d1f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5644dc953a50 0x5644dc958090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.076+0000 7f0d17e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5644dc94ae70 0x5644dc957b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.076+0000 7f0d1ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5644dc94a550 0x5644dc957610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33864/0 (socket says 172.21.15.43:33864) 2024-03-28T22:45:31.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.076+0000 7f0d1ea00640 1 -- 172.21.15.43:0/4147369864 learned_addr learned my addr 172.21.15.43:0/4147369864 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:31.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.076+0000 7f0d1ea00640 1 -- 172.21.15.43:0/4147369864 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5644dc94ae70 msgr2=0x5644dc957b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:31.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.076+0000 7f0d1ea00640 1 --2- 172.21.15.43:0/4147369864 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5644dc94ae70 0x5644dc957b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.076+0000 7f0d1ea00640 1 -- 172.21.15.43:0/4147369864 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5644dc953a50 msgr2=0x5644dc958090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:31.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.076+0000 7f0d1ea00640 1 --2- 172.21.15.43:0/4147369864 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5644dc953a50 0x5644dc958090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.076+0000 7f0d1ea00640 1 -- 172.21.15.43:0/4147369864 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644dc9c9010 con 0x5644dc94a550 2024-03-28T22:45:31.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.076+0000 7f0d1ea00640 1 --2- 172.21.15.43:0/4147369864 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5644dc94a550 0x5644dc957610 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f0d1009a450 tx=0x7f0d100ba040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7295f13859c9619d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:31.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.076+0000 7f0d0cc00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:45:31.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.076+0000 7f0d0cc00640 1 -- 172.21.15.43:0/4147369864 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d10020040 con 0x5644dc94a550 2024-03-28T22:45:31.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.076+0000 7f0d0cc00640 1 -- 172.21.15.43:0/4147369864 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7f0d10096dc0 con 0x5644dc94a550 2024-03-28T22:45:31.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.076+0000 7f0d28bbc740 1 -- 172.21.15.43:0/4147369864 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d08005540 con 0x5644dc94a550 2024-03-28T22:45:31.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.077+0000 7f0d0cc00640 1 -- 172.21.15.43:0/4147369864 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0d1009e7f0 con 0x5644dc94a550 2024-03-28T22:45:31.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.077+0000 7f0d28bbc740 10 client.28535 fetch_fsmap learned FSMap version 32 2024-03-28T22:45:31.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.077+0000 7f0d28bbc740 1 -- 172.21.15.43:0/4147369864 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5644dc9ca310 con 0x5644dc94a550 2024-03-28T22:45:31.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.077+0000 7f0d0cc00640 1 -- 172.21.15.43:0/4147369864 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d08005540 con 0x5644dc94a550 2024-03-28T22:45:31.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.077+0000 7f0d0cc00640 1 -- 172.21.15.43:0/4147369864 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f0d100ba0b0 con 0x5644dc94a550 2024-03-28T22:45:31.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.077+0000 7f0d28bbc740 10 client.28535 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:45:31.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.077+0000 7f0d28bbc740 10 client.28535 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:45:31.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.077+0000 7f0d28bbc740 20 client.28535 populate_metadata read hostname 'smithi043' 2024-03-28T22:45:31.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.077+0000 7f0d28bbc740 1 --2- 172.21.15.43:0/4147369864 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5644dc9caa10 0x5644dc9cce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.077+0000 7f0d28bbc740 4 client.28535 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:45:31.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.077+0000 7f0d28bbc740 1 -- 172.21.15.43:0/4147369864 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5644dc8bb590 con 0x5644dc9caa10 2024-03-28T22:45:31.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.077+0000 7f0d17e00640 1 --2- 172.21.15.43:0/4147369864 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5644dc9caa10 0x5644dc9cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.078+0000 7f0d17e00640 1 --2- 172.21.15.43:0/4147369864 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5644dc9caa10 0x5644dc9cce30 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b943e58d4c1a8ea1 server_cookie=b396c23a8efad9d0 in_seq=0 out_seq=0 2024-03-28T22:45:31.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.078+0000 7f0d0cc00640 10 client.28535 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:45:31.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.078+0000 7f0d0cc00640 1 -- 172.21.15.43:0/4147369864 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5644dc8bb590 con 0x5644dc9caa10 2024-03-28T22:45:31.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.078+0000 7f0d0cc00640 10 client.28535 handle_command_reply: tid=0 2024-03-28T22:45:31.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.078+0000 7f0d28bbc740 1 client.28535 shutdown 2024-03-28T22:45:31.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.078+0000 7f0d28bbc740 1 -- 172.21.15.43:0/4147369864 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5644dc94a550 msgr2=0x5644dc957610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.078+0000 7f0d28bbc740 1 --2- 172.21.15.43:0/4147369864 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5644dc94a550 0x5644dc957610 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f0d1009a450 tx=0x7f0d100ba040 comp rx=0 tx=0).stop 2024-03-28T22:45:31.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.078+0000 7f0d28bbc740 1 -- 172.21.15.43:0/4147369864 shutdown_connections 2024-03-28T22:45:31.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.078+0000 7f0d28bbc740 1 --2- 172.21.15.43:0/4147369864 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5644dc9caa10 0x5644dc9cce30 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.078+0000 7f0d28bbc740 1 --2- 172.21.15.43:0/4147369864 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5644dc953a50 0x5644dc958090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.078+0000 7f0d28bbc740 1 --2- 172.21.15.43:0/4147369864 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5644dc94ae70 0x5644dc957b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.078+0000 7f0d28bbc740 1 --2- 172.21.15.43:0/4147369864 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5644dc94a550 0x5644dc957610 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.078+0000 7f0d0cc00640 0 client.28535 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:45:31.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.078+0000 7f0d28bbc740 1 -- 172.21.15.43:0/4147369864 >> 172.21.15.43:0/4147369864 conn(0x5644dc944c80 msgr2=0x5644dc943dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:31.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.078+0000 7f0d28bbc740 1 -- 172.21.15.43:0/4147369864 shutdown_connections 2024-03-28T22:45:31.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.078+0000 7f0d28bbc740 1 -- 172.21.15.43:0/4147369864 wait complete. 2024-03-28T22:45:31.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.079+0000 7f0d28bbc740 20 client.28535 trim_cache size 0 max 16384 2024-03-28T22:45:31.086 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:45:31.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.079+0000 7f0d1fe00640 1 -- 172.21.15.43:0/2512393181 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0d040a1f70 msgr2=0x7f0d040a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.079+0000 7f0d1fe00640 1 --2- 172.21.15.43:0/2512393181 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0d040a1f70 0x7f0d040a4430 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0d18009a60 tx=0x7f0d1802e040 comp rx=0 tx=0).stop 2024-03-28T22:45:31.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.079+0000 7f0d1fe00640 1 -- 172.21.15.43:0/2512393181 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d20107f00 msgr2=0x7f0d2019e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.079+0000 7f0d1fe00640 1 --2- 172.21.15.43:0/2512393181 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d20107f00 0x7f0d2019e440 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f0d1000d950 tx=0x7f0d1000de10 comp rx=0 tx=0).stop 2024-03-28T22:45:31.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.079+0000 7f0d1fe00640 1 -- 172.21.15.43:0/2512393181 shutdown_connections 2024-03-28T22:45:31.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.079+0000 7f0d1fe00640 1 --2- 172.21.15.43:0/2512393181 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0d040a1f70 0x7f0d040a4430 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.079+0000 7f0d1fe00640 1 --2- 172.21.15.43:0/2512393181 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0d2010cf50 0x7f0d2019e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.079+0000 7f0d1fe00640 1 --2- 172.21.15.43:0/2512393181 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0d201a2d00 0x7f0d201a31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.079+0000 7f0d1fe00640 1 --2- 172.21.15.43:0/2512393181 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0d20107f00 0x7f0d2019e440 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.079+0000 7f0d1fe00640 1 -- 172.21.15.43:0/2512393181 >> 172.21.15.43:0/2512393181 conn(0x7f0d20077570 msgr2=0x7f0d20077af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:31.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.079+0000 7f0d1fe00640 1 -- 172.21.15.43:0/2512393181 shutdown_connections 2024-03-28T22:45:31.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.079+0000 7f0d1fe00640 1 -- 172.21.15.43:0/2512393181 wait complete. 2024-03-28T22:45:31.092 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:30 smithi043 ceph-mon[31823]: pgmap v259: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:45:31.092 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:30 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1447068464' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:45:31.094 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:45:31.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:30 smithi172 ceph-mon[36566]: pgmap v259: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:45:31.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:30 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1447068464' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:45:31.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:30 smithi082 ceph-mon[36471]: pgmap v259: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:45:31.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:30 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1447068464' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:45:31.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.315+0000 7fb62e600640 1 -- 172.21.15.43:0/1768325507 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb6281038f0 msgr2=0x7fb628103d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.315+0000 7fb62e600640 1 --2- 172.21.15.43:0/1768325507 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb6281038f0 0x7fb628103d70 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fb614009900 tx=0x7fb61402bc30 comp rx=0 tx=0).stop 2024-03-28T22:45:31.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.316+0000 7fb62e600640 1 -- 172.21.15.43:0/1768325507 shutdown_connections 2024-03-28T22:45:31.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.316+0000 7fb62e600640 1 --2- 172.21.15.43:0/1768325507 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb6281038f0 0x7fb628103d70 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.316+0000 7fb62e600640 1 --2- 172.21.15.43:0/1768325507 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6281042b0 0x7fb6281107c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.316+0000 7fb62e600640 1 --2- 172.21.15.43:0/1768325507 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb628109900 0x7fb628109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.316+0000 7fb62e600640 1 -- 172.21.15.43:0/1768325507 >> 172.21.15.43:0/1768325507 conn(0x7fb6280ff3a0 msgr2=0x7fb6281017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:31.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.316+0000 7fb62e600640 1 -- 172.21.15.43:0/1768325507 shutdown_connections 2024-03-28T22:45:31.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.316+0000 7fb62e600640 1 -- 172.21.15.43:0/1768325507 wait complete. 2024-03-28T22:45:31.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.317+0000 7fb62e600640 1 Processor -- start 2024-03-28T22:45:31.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.317+0000 7fb62e600640 1 -- start start 2024-03-28T22:45:31.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.317+0000 7fb62e600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6281038f0 0x7fb62819a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.318+0000 7fb62e600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb6281042b0 0x7fb62819a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.318+0000 7fb62e600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb628109900 0x7fb62819eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.318+0000 7fb62e600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb628113d60 con 0x7fb628109900 2024-03-28T22:45:31.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.318+0000 7fb62e600640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb628113be0 con 0x7fb6281042b0 2024-03-28T22:45:31.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.318+0000 7fb62e600640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fb628113ee0 con 0x7fb6281038f0 2024-03-28T22:45:31.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.318+0000 7fb626a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb6281042b0 0x7fb62819a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.318+0000 7fb627400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6281038f0 0x7fb62819a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.318+0000 7fb626a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb6281042b0 0x7fb62819a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33886/0 (socket says 172.21.15.43:33886) 2024-03-28T22:45:31.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.318+0000 7fb626a00640 1 -- 172.21.15.43:0/1051814493 learned_addr learned my addr 172.21.15.43:0/1051814493 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:31.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.318+0000 7fb627e00640 1 --2- 172.21.15.43:0/1051814493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb628109900 0x7fb62819eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.318+0000 7fb627400640 1 -- 172.21.15.43:0/1051814493 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb6281042b0 msgr2=0x7fb62819a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.318+0000 7fb627400640 1 --2- 172.21.15.43:0/1051814493 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb6281042b0 0x7fb62819a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.318+0000 7fb627400640 1 -- 172.21.15.43:0/1051814493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb628109900 msgr2=0x7fb62819eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.318+0000 7fb627400640 1 --2- 172.21.15.43:0/1051814493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb628109900 0x7fb62819eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.318+0000 7fb627400640 1 -- 172.21.15.43:0/1051814493 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb62819f1f0 con 0x7fb6281038f0 2024-03-28T22:45:31.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.318+0000 7fb626a00640 1 --2- 172.21.15.43:0/1051814493 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb6281042b0 0x7fb62819a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:45:31.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.319+0000 7fb627400640 1 --2- 172.21.15.43:0/1051814493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6281038f0 0x7fb62819a200 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fb61800e9f0 tx=0x7fb61800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:31.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.319+0000 7fb61fe00640 1 -- 172.21.15.43:0/1051814493 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61800ccc0 con 0x7fb6281038f0 2024-03-28T22:45:31.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.319+0000 7fb62e600640 1 -- 172.21.15.43:0/1051814493 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb62819f4e0 con 0x7fb6281038f0 2024-03-28T22:45:31.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.319+0000 7fb61fe00640 1 -- 172.21.15.43:0/1051814493 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb618010910 con 0x7fb6281038f0 2024-03-28T22:45:31.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.319+0000 7fb61fe00640 1 -- 172.21.15.43:0/1051814493 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb618018960 con 0x7fb6281038f0 2024-03-28T22:45:31.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.319+0000 7fb62e600640 1 -- 172.21.15.43:0/1051814493 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6281a6d70 con 0x7fb6281038f0 2024-03-28T22:45:31.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.322+0000 7fb61fe00640 1 -- 172.21.15.43:0/1051814493 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fb618010ac0 con 0x7fb6281038f0 2024-03-28T22:45:31.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.322+0000 7fb631366740 1 Processor -- start 2024-03-28T22:45:31.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.322+0000 7fb631366740 1 -- start start 2024-03-28T22:45:31.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.323+0000 7fb631366740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3b7498380 0x55b3b7498760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.323+0000 7fb631366740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3b7498ca0 0x55b3b74a1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.323+0000 7fb631366740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3b74a1880 0x55b3b74a3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.323+0000 7fb631366740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b3b73fc050 con 0x55b3b7498ca0 2024-03-28T22:45:31.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.323+0000 7fb631366740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55b3b73a4ce0 con 0x55b3b74a1880 2024-03-28T22:45:31.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.323+0000 7fb631366740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55b3b73f80f0 con 0x55b3b7498380 2024-03-28T22:45:31.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.323+0000 7fb627e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3b7498380 0x55b3b7498760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.323+0000 7fb627e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3b7498380 0x55b3b7498760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43646/0 (socket says 172.21.15.43:43646) 2024-03-28T22:45:31.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.323+0000 7fb627e00640 1 -- 172.21.15.43:0/3239998070 learned_addr learned my addr 172.21.15.43:0/3239998070 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:31.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.323+0000 7fb627400640 1 --2- 172.21.15.43:0/3239998070 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3b7498ca0 0x55b3b74a1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.323+0000 7fb626a00640 1 --2- 172.21.15.43:0/3239998070 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3b74a1880 0x55b3b74a3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.324+0000 7fb627e00640 1 -- 172.21.15.43:0/3239998070 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3b74a1880 msgr2=0x55b3b74a3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:31.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.324+0000 7fb627e00640 1 --2- 172.21.15.43:0/3239998070 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3b74a1880 0x55b3b74a3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.324+0000 7fb627e00640 1 -- 172.21.15.43:0/3239998070 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3b7498ca0 msgr2=0x55b3b74a1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:31.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.324+0000 7fb627e00640 1 --2- 172.21.15.43:0/3239998070 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3b7498ca0 0x55b3b74a1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.324+0000 7fb627e00640 1 -- 172.21.15.43:0/3239998070 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3b71103f0 con 0x55b3b7498380 2024-03-28T22:45:31.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.324+0000 7fb626a00640 1 --2- 172.21.15.43:0/3239998070 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3b74a1880 0x55b3b74a3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:45:31.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.324+0000 7fb61fe00640 1 --2- 172.21.15.43:0/1051814493 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb60c0a1ef0 0x7fb60c0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.324+0000 7fb627e00640 1 --2- 172.21.15.43:0/3239998070 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3b7498380 0x55b3b7498760 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fb620009fd0 tx=0x7fb6200118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80875d8cec023f7e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:31.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.324+0000 7fb61fe00640 1 -- 172.21.15.43:0/1051814493 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7fb6180ce900 con 0x7fb6281038f0 2024-03-28T22:45:31.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.324+0000 7fb61cc00640 1 -- 172.21.15.43:0/3239998070 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb620004d20 con 0x55b3b7498380 2024-03-28T22:45:31.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.324+0000 7fb61cc00640 1 -- 172.21.15.43:0/3239998070 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb620004050 con 0x55b3b7498380 2024-03-28T22:45:31.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.324+0000 7fb61cc00640 1 -- 172.21.15.43:0/3239998070 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb620005240 con 0x55b3b7498380 2024-03-28T22:45:31.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.324+0000 7fb627400640 1 --2- 172.21.15.43:0/1051814493 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb60c0a1ef0 0x7fb60c0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.325+0000 7fb631366740 1 -- 172.21.15.43:0/3239998070 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3b7498380 msgr2=0x55b3b7498760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.325+0000 7fb631366740 1 --2- 172.21.15.43:0/3239998070 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3b7498380 0x55b3b7498760 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fb620009fd0 tx=0x7fb6200118d0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.325+0000 7fb627400640 1 --2- 172.21.15.43:0/1051814493 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb60c0a1ef0 0x7fb60c0a43b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x55b3b7497e50 tx=0x7fb6180d4530 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:31.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.325+0000 7fb631366740 1 -- 172.21.15.43:0/3239998070 shutdown_connections 2024-03-28T22:45:31.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.325+0000 7fb631366740 1 --2- 172.21.15.43:0/3239998070 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3b74a1880 0x55b3b74a3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.325+0000 7fb631366740 1 --2- 172.21.15.43:0/3239998070 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3b7498ca0 0x55b3b74a1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.325+0000 7fb631366740 1 --2- 172.21.15.43:0/3239998070 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3b7498380 0x55b3b7498760 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.325+0000 7fb631366740 1 -- 172.21.15.43:0/3239998070 >> 172.21.15.43:0/3239998070 conn(0x55b3b7492ab0 msgr2=0x55b3b7492e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:31.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.325+0000 7fb631366740 1 -- 172.21.15.43:0/3239998070 shutdown_connections 2024-03-28T22:45:31.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.325+0000 7fb631366740 1 -- 172.21.15.43:0/3239998070 wait complete. 2024-03-28T22:45:31.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.326+0000 7fb631366740 1 -- start start 2024-03-28T22:45:31.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.326+0000 7fb631366740 1 Processor -- start 2024-03-28T22:45:31.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.326+0000 7fb631366740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3b7498380 0x55b3b749e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.326+0000 7fb631366740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3b7498ca0 0x55b3b749eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.326+0000 7fb631366740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3b74a1880 0x55b3b749f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.326+0000 7fb631366740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b3b74a4670 con 0x55b3b7498ca0 2024-03-28T22:45:31.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.326+0000 7fb631366740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55b3b739a2f0 con 0x55b3b7498380 2024-03-28T22:45:31.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.326+0000 7fb631366740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55b3b74a47f0 con 0x55b3b74a1880 2024-03-28T22:45:31.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.327+0000 7fb631366740 10 client.0 _ll_register_callbacks cb 0x55b3b710fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:45:31.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.327+0000 7fb627e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3b7498380 0x55b3b749e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.327+0000 7fb626a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3b7498ca0 0x55b3b749eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.327+0000 7fb627e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3b7498380 0x55b3b749e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33904/0 (socket says 172.21.15.43:33904) 2024-03-28T22:45:31.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.327+0000 7fb627e00640 1 -- 172.21.15.43:0/1130375171 learned_addr learned my addr 172.21.15.43:0/1130375171 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:31.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.327+0000 7fb627e00640 1 --2- 172.21.15.43:0/1130375171 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3b74a1880 0x55b3b749f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.327+0000 7fb626a00640 1 -- 172.21.15.43:0/1130375171 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3b74a1880 msgr2=0x55b3b749f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:31.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.327+0000 7fb626a00640 1 --2- 172.21.15.43:0/1130375171 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3b74a1880 0x55b3b749f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.327+0000 7fb626a00640 1 -- 172.21.15.43:0/1130375171 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3b7498380 msgr2=0x55b3b749e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:31.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.327+0000 7fb626a00640 1 --2- 172.21.15.43:0/1130375171 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3b7498380 0x55b3b749e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.327+0000 7fb626a00640 1 -- 172.21.15.43:0/1130375171 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3b71da7d0 con 0x55b3b7498ca0 2024-03-28T22:45:31.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.327+0000 7fb627e00640 1 --2- 172.21.15.43:0/1130375171 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3b7498380 0x55b3b749e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:45:31.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.328+0000 7fb626a00640 1 --2- 172.21.15.43:0/1130375171 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3b7498ca0 0x55b3b749eec0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb614000c00 tx=0x7fb61402c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=905e09e1da41294 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:31.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.328+0000 7fb611600640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:45:31.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.328+0000 7fb611600640 1 -- 172.21.15.43:0/1130375171 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb614004480 con 0x55b3b7498ca0 2024-03-28T22:45:31.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.328+0000 7fb611600640 1 -- 172.21.15.43:0/1130375171 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7fb61402c8b0 con 0x55b3b7498ca0 2024-03-28T22:45:31.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.328+0000 7fb631366740 1 -- 172.21.15.43:0/1130375171 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb620004050 con 0x55b3b7498ca0 2024-03-28T22:45:31.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.328+0000 7fb611600640 1 -- 172.21.15.43:0/1130375171 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb61402cab0 con 0x55b3b7498ca0 2024-03-28T22:45:31.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.328+0000 7fb631366740 10 client.18282 fetch_fsmap learned FSMap version 32 2024-03-28T22:45:31.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.328+0000 7fb631366740 1 -- 172.21.15.43:0/1130375171 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b3b74b0a40 con 0x55b3b7498ca0 2024-03-28T22:45:31.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.329+0000 7fb611600640 1 -- 172.21.15.43:0/1130375171 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb620004050 con 0x55b3b7498ca0 2024-03-28T22:45:31.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.329+0000 7fb611600640 1 -- 172.21.15.43:0/1130375171 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb614003860 con 0x55b3b7498ca0 2024-03-28T22:45:31.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.329+0000 7fb631366740 10 client.18282 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:45:31.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.329+0000 7fb631366740 10 client.18282 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:45:31.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.329+0000 7fb631366740 20 client.18282 populate_metadata read hostname 'smithi043' 2024-03-28T22:45:31.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.329+0000 7fb631366740 1 --2- 172.21.15.43:0/1130375171 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55b3b74b1f00 0x55b3b74b4360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.329+0000 7fb631366740 4 client.18282 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:45:31.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.329+0000 7fb631366740 1 -- 172.21.15.43:0/1130375171 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b3b73a5680 con 0x55b3b74b1f00 2024-03-28T22:45:31.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.329+0000 7fb627e00640 1 --2- 172.21.15.43:0/1130375171 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55b3b74b1f00 0x55b3b74b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.330+0000 7fb627e00640 1 --2- 172.21.15.43:0/1130375171 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55b3b74b1f00 0x55b3b74b4360 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bb978fc948667a5c server_cookie=df0c0c36a3bf222e in_seq=0 out_seq=0 2024-03-28T22:45:31.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.330+0000 7fb611600640 10 client.18282 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:45:31.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.331+0000 7fb611600640 1 -- 172.21.15.43:0/1130375171 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b3b73a5680 con 0x55b3b74b1f00 2024-03-28T22:45:31.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.331+0000 7fb611600640 10 client.18282 handle_command_reply: tid=0 2024-03-28T22:45:31.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.331+0000 7fb631366740 1 client.18282 shutdown 2024-03-28T22:45:31.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.331+0000 7fb631366740 1 -- 172.21.15.43:0/1130375171 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3b7498ca0 msgr2=0x55b3b749eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.331+0000 7fb631366740 1 --2- 172.21.15.43:0/1130375171 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3b7498ca0 0x55b3b749eec0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb614000c00 tx=0x7fb61402c800 comp rx=0 tx=0).stop 2024-03-28T22:45:31.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.331+0000 7fb631366740 1 -- 172.21.15.43:0/1130375171 shutdown_connections 2024-03-28T22:45:31.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.331+0000 7fb631366740 1 --2- 172.21.15.43:0/1130375171 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3b74a1880 0x55b3b749f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.331+0000 7fb631366740 1 --2- 172.21.15.43:0/1130375171 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3b7498ca0 0x55b3b749eec0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.331+0000 7fb631366740 1 --2- 172.21.15.43:0/1130375171 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55b3b74b1f00 0x55b3b74b4360 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.331+0000 7fb631366740 1 --2- 172.21.15.43:0/1130375171 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3b7498380 0x55b3b749e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.331+0000 7fb631366740 1 -- 172.21.15.43:0/1130375171 >> 172.21.15.43:0/1130375171 conn(0x55b3b7492ab0 msgr2=0x55b3b7494800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:31.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.331+0000 7fb611600640 0 client.18282 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:45:31.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.332+0000 7fb631366740 1 -- 172.21.15.43:0/1130375171 shutdown_connections 2024-03-28T22:45:31.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.332+0000 7fb631366740 1 -- 172.21.15.43:0/1130375171 wait complete. 2024-03-28T22:45:31.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.332+0000 7fb631366740 20 client.18282 trim_cache size 0 max 16384 2024-03-28T22:45:31.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.347+0000 7fb631366740 1 Processor -- start 2024-03-28T22:45:31.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.347+0000 7fb631366740 1 -- start start 2024-03-28T22:45:31.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.347+0000 7fb631366740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3b7498380 0x55b3b74a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.347+0000 7fb631366740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3b7498ca0 0x55b3b74ad410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.347+0000 7fb631366740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3b74a1880 0x55b3b74b9c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.347+0000 7fb631366740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb6281b0ec0 con 0x55b3b7498ca0 2024-03-28T22:45:31.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.347+0000 7fb631366740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb6281b2a00 con 0x55b3b7498380 2024-03-28T22:45:31.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.347+0000 7fb631366740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fb6281b1590 con 0x55b3b74a1880 2024-03-28T22:45:31.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.347+0000 7fb626a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3b7498ca0 0x55b3b74ad410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.347+0000 7fb627e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3b7498380 0x55b3b74a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.347+0000 7fb626a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3b7498ca0 0x55b3b74ad410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50730/0 (socket says 172.21.15.43:50730) 2024-03-28T22:45:31.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.347+0000 7fb626a00640 1 -- 172.21.15.43:0/2787591070 learned_addr learned my addr 172.21.15.43:0/2787591070 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:31.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.347+0000 7fb627e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3b74a1880 0x55b3b74b9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.348+0000 7fb626a00640 1 -- 172.21.15.43:0/2787591070 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3b74a1880 msgr2=0x55b3b74b9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:31.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.348+0000 7fb626a00640 1 --2- 172.21.15.43:0/2787591070 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3b74a1880 0x55b3b74b9c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.348+0000 7fb626a00640 1 -- 172.21.15.43:0/2787591070 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3b7498380 msgr2=0x55b3b74a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:31.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.348+0000 7fb626a00640 1 --2- 172.21.15.43:0/2787591070 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3b7498380 0x55b3b74a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.348+0000 7fb626a00640 1 -- 172.21.15.43:0/2787591070 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3b749ced0 con 0x55b3b7498ca0 2024-03-28T22:45:31.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.348+0000 7fb626a00640 1 --2- 172.21.15.43:0/2787591070 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3b7498ca0 0x55b3b74ad410 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb614036040 tx=0x7fb614004e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a98f39a44de85085 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:31.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.349+0000 7fb613e00640 1 -- 172.21.15.43:0/2787591070 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61405a020 con 0x55b3b7498ca0 2024-03-28T22:45:31.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.349+0000 7fb613e00640 1 -- 172.21.15.43:0/2787591070 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb614009520 con 0x55b3b7498ca0 2024-03-28T22:45:31.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.349+0000 7fb613e00640 1 -- 172.21.15.43:0/2787591070 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb614004f40 con 0x55b3b7498ca0 2024-03-28T22:45:31.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.349+0000 7fb631366740 1 -- 172.21.15.43:0/2787591070 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3b7498ca0 msgr2=0x55b3b74ad410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.349+0000 7fb631366740 1 --2- 172.21.15.43:0/2787591070 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3b7498ca0 0x55b3b74ad410 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb614036040 tx=0x7fb614004e10 comp rx=0 tx=0).stop 2024-03-28T22:45:31.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.349+0000 7fb631366740 1 -- 172.21.15.43:0/2787591070 shutdown_connections 2024-03-28T22:45:31.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.349+0000 7fb631366740 1 --2- 172.21.15.43:0/2787591070 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3b74a1880 0x55b3b74b9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.349+0000 7fb631366740 1 --2- 172.21.15.43:0/2787591070 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3b7498ca0 0x55b3b74ad410 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.349+0000 7fb631366740 1 --2- 172.21.15.43:0/2787591070 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3b7498380 0x55b3b74a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.349+0000 7fb631366740 1 -- 172.21.15.43:0/2787591070 >> 172.21.15.43:0/2787591070 conn(0x55b3b7492ab0 msgr2=0x55b3b7493e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:31.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.350+0000 7fb631366740 1 -- 172.21.15.43:0/2787591070 shutdown_connections 2024-03-28T22:45:31.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.350+0000 7fb631366740 1 -- 172.21.15.43:0/2787591070 wait complete. 2024-03-28T22:45:31.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.350+0000 7fb631366740 1 -- start start 2024-03-28T22:45:31.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.350+0000 7fb631366740 1 Processor -- start 2024-03-28T22:45:31.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.351+0000 7fb631366740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3b7498380 0x55b3b74a9c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.351+0000 7fb631366740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3b7498ca0 0x55b3b74a5000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.351+0000 7fb631366740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3b74a1880 0x55b3b74a5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.351+0000 7fb631366740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb6281b2330 con 0x55b3b7498380 2024-03-28T22:45:31.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.351+0000 7fb631366740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb6281b1c60 con 0x55b3b74a1880 2024-03-28T22:45:31.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.351+0000 7fb631366740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fb6281aeec0 con 0x55b3b7498ca0 2024-03-28T22:45:31.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.351+0000 7fb631366740 10 client.0 _ll_register_callbacks cb 0x7fb614009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:45:31.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.351+0000 7fb627e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3b7498380 0x55b3b74a9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.351+0000 7fb627e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3b7498380 0x55b3b74a9c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50732/0 (socket says 172.21.15.43:50732) 2024-03-28T22:45:31.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.351+0000 7fb627e00640 1 -- 172.21.15.43:0/2804850231 learned_addr learned my addr 172.21.15.43:0/2804850231 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:31.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.351+0000 7fb626a00640 1 --2- 172.21.15.43:0/2804850231 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3b7498ca0 0x55b3b74a5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.351+0000 7fb627e00640 1 --2- 172.21.15.43:0/2804850231 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3b74a1880 0x55b3b74a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.351+0000 7fb626a00640 1 -- 172.21.15.43:0/2804850231 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3b74a1880 msgr2=0x55b3b74a5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:31.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.351+0000 7fb626a00640 1 --2- 172.21.15.43:0/2804850231 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3b74a1880 0x55b3b74a5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.351+0000 7fb626a00640 1 -- 172.21.15.43:0/2804850231 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3b7498380 msgr2=0x55b3b74a9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:31.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.351+0000 7fb626a00640 1 --2- 172.21.15.43:0/2804850231 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3b7498380 0x55b3b74a9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.351+0000 7fb626a00640 1 -- 172.21.15.43:0/2804850231 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3b74a5d70 con 0x55b3b7498ca0 2024-03-28T22:45:31.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.351+0000 7fb627e00640 1 --2- 172.21.15.43:0/2804850231 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3b7498380 0x55b3b74a9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:45:31.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.352+0000 7fb626a00640 1 --2- 172.21.15.43:0/2804850231 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3b7498ca0 0x55b3b74a5000 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fb61402c320 tx=0x7fb614005e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0e8463fdf1c0fa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:31.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.352+0000 7fb611600640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:45:31.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.352+0000 7fb611600640 1 -- 172.21.15.43:0/2804850231 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb614063070 con 0x55b3b7498ca0 2024-03-28T22:45:31.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.352+0000 7fb631366740 1 -- 172.21.15.43:0/2804850231 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb614009520 con 0x55b3b7498ca0 2024-03-28T22:45:31.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.352+0000 7fb611600640 1 -- 172.21.15.43:0/2804850231 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7fb614004480 con 0x55b3b7498ca0 2024-03-28T22:45:31.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.352+0000 7fb611600640 1 -- 172.21.15.43:0/2804850231 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb61406f050 con 0x55b3b7498ca0 2024-03-28T22:45:31.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.352+0000 7fb631366740 10 client.28489 fetch_fsmap learned FSMap version 32 2024-03-28T22:45:31.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.352+0000 7fb631366740 1 -- 172.21.15.43:0/2804850231 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b3b7518230 con 0x55b3b7498ca0 2024-03-28T22:45:31.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.353+0000 7fb611600640 1 -- 172.21.15.43:0/2804850231 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb614009520 con 0x55b3b7498ca0 2024-03-28T22:45:31.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.353+0000 7fb611600640 1 -- 172.21.15.43:0/2804850231 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb6140054a0 con 0x55b3b7498ca0 2024-03-28T22:45:31.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.353+0000 7fb631366740 10 client.28489 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:45:31.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.353+0000 7fb631366740 10 client.28489 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:45:31.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.353+0000 7fb631366740 20 client.28489 populate_metadata read hostname 'smithi043' 2024-03-28T22:45:31.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.353+0000 7fb631366740 1 --2- 172.21.15.43:0/2804850231 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55b3b7518790 0x55b3b751abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.353+0000 7fb631366740 4 client.28489 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:45:31.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.353+0000 7fb631366740 1 -- 172.21.15.43:0/2804850231 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b3b7409590 con 0x55b3b7518790 2024-03-28T22:45:31.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.353+0000 7fb627e00640 1 --2- 172.21.15.43:0/2804850231 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55b3b7518790 0x55b3b751abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.354+0000 7fb627e00640 1 --2- 172.21.15.43:0/2804850231 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55b3b7518790 0x55b3b751abb0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ceada5b49a00a82a server_cookie=10dbc875571c6569 in_seq=0 out_seq=0 2024-03-28T22:45:31.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.354+0000 7fb611600640 10 client.28489 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:45:31.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.354+0000 7fb611600640 1 -- 172.21.15.43:0/2804850231 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b3b7409590 con 0x55b3b7518790 2024-03-28T22:45:31.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.354+0000 7fb611600640 10 client.28489 handle_command_reply: tid=0 2024-03-28T22:45:31.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.354+0000 7fb631366740 1 client.28489 shutdown 2024-03-28T22:45:31.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.354+0000 7fb631366740 1 -- 172.21.15.43:0/2804850231 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3b7498ca0 msgr2=0x55b3b74a5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.354+0000 7fb631366740 1 --2- 172.21.15.43:0/2804850231 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3b7498ca0 0x55b3b74a5000 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fb61402c320 tx=0x7fb614005e50 comp rx=0 tx=0).stop 2024-03-28T22:45:31.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.354+0000 7fb631366740 1 -- 172.21.15.43:0/2804850231 shutdown_connections 2024-03-28T22:45:31.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.354+0000 7fb631366740 1 --2- 172.21.15.43:0/2804850231 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55b3b7518790 0x55b3b751abb0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.354+0000 7fb631366740 1 --2- 172.21.15.43:0/2804850231 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3b74a1880 0x55b3b74a5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.354+0000 7fb631366740 1 --2- 172.21.15.43:0/2804850231 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3b7498ca0 0x55b3b74a5000 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.354+0000 7fb611600640 0 client.28489 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:45:31.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.354+0000 7fb631366740 1 --2- 172.21.15.43:0/2804850231 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3b7498380 0x55b3b74a9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.354+0000 7fb631366740 1 -- 172.21.15.43:0/2804850231 >> 172.21.15.43:0/2804850231 conn(0x55b3b7492ab0 msgr2=0x55b3b749f1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:31.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.354+0000 7fb631366740 1 -- 172.21.15.43:0/2804850231 shutdown_connections 2024-03-28T22:45:31.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.354+0000 7fb631366740 1 -- 172.21.15.43:0/2804850231 wait complete. 2024-03-28T22:45:31.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.355+0000 7fb631366740 20 client.28489 trim_cache size 0 max 16384 2024-03-28T22:45:31.360 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:45:31.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.355+0000 7fb62e600640 1 -- 172.21.15.43:0/1051814493 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb60c0a1ef0 msgr2=0x7fb60c0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.355+0000 7fb62e600640 1 --2- 172.21.15.43:0/1051814493 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb60c0a1ef0 0x7fb60c0a43b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x55b3b7497e50 tx=0x7fb6180d4530 comp rx=0 tx=0).stop 2024-03-28T22:45:31.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.355+0000 7fb62e600640 1 -- 172.21.15.43:0/1051814493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6281038f0 msgr2=0x7fb62819a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.355+0000 7fb62e600640 1 --2- 172.21.15.43:0/1051814493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6281038f0 0x7fb62819a200 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fb61800e9f0 tx=0x7fb61800eeb0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.355+0000 7fb62e600640 1 -- 172.21.15.43:0/1051814493 shutdown_connections 2024-03-28T22:45:31.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.355+0000 7fb62e600640 1 --2- 172.21.15.43:0/1051814493 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb60c0a1ef0 0x7fb60c0a43b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.355+0000 7fb62e600640 1 --2- 172.21.15.43:0/1051814493 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb6281042b0 0x7fb62819a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.355+0000 7fb62e600640 1 --2- 172.21.15.43:0/1051814493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb628109900 0x7fb62819eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.355+0000 7fb62e600640 1 --2- 172.21.15.43:0/1051814493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb6281038f0 0x7fb62819a200 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.355+0000 7fb62e600640 1 -- 172.21.15.43:0/1051814493 >> 172.21.15.43:0/1051814493 conn(0x7fb6280ff3a0 msgr2=0x7fb6280ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:31.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.356+0000 7fb62e600640 1 -- 172.21.15.43:0/1051814493 shutdown_connections 2024-03-28T22:45:31.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.356+0000 7fb62e600640 1 -- 172.21.15.43:0/1051814493 wait complete. 2024-03-28T22:45:31.372 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:45:31.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.602+0000 7fd838a00640 1 -- 172.21.15.43:0/2218111277 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd834107f00 msgr2=0x7fd8341082e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.602+0000 7fd838a00640 1 --2- 172.21.15.43:0/2218111277 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd834107f00 0x7fd8341082e0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fd824009900 tx=0x7fd82402bc30 comp rx=0 tx=0).stop 2024-03-28T22:45:31.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.602+0000 7fd838a00640 1 -- 172.21.15.43:0/2218111277 shutdown_connections 2024-03-28T22:45:31.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.602+0000 7fd838a00640 1 --2- 172.21.15.43:0/2218111277 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd83410cf50 0x7fd834114b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.602+0000 7fd838a00640 1 --2- 172.21.15.43:0/2218111277 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd8341088b0 0x7fd83410c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.602+0000 7fd838a00640 1 --2- 172.21.15.43:0/2218111277 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd834107f00 0x7fd8341082e0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.602+0000 7fd838a00640 1 -- 172.21.15.43:0/2218111277 >> 172.21.15.43:0/2218111277 conn(0x7fd834077570 msgr2=0x7fd834106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:31.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.602+0000 7fd838a00640 1 -- 172.21.15.43:0/2218111277 shutdown_connections 2024-03-28T22:45:31.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.602+0000 7fd838a00640 1 -- 172.21.15.43:0/2218111277 wait complete. 2024-03-28T22:45:31.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.603+0000 7fd838a00640 1 Processor -- start 2024-03-28T22:45:31.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.603+0000 7fd838a00640 1 -- start start 2024-03-28T22:45:31.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.604+0000 7fd838a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd834107f00 0x7fd83419e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.604+0000 7fd838a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd83410cf50 0x7fd83419e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.604+0000 7fd838a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd8341a2d50 0x7fd8341a3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.604+0000 7fd838a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd834118050 con 0x7fd834107f00 2024-03-28T22:45:31.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.604+0000 7fd838a00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd834117ed0 con 0x7fd83410cf50 2024-03-28T22:45:31.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.604+0000 7fd838a00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd8341181d0 con 0x7fd8341a2d50 2024-03-28T22:45:31.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.604+0000 7fd832a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd83410cf50 0x7fd83419e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.604+0000 7fd833400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd834107f00 0x7fd83419e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.604+0000 7fd832a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd83410cf50 0x7fd83419e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33940/0 (socket says 172.21.15.43:33940) 2024-03-28T22:45:31.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.604+0000 7fd832a00640 1 -- 172.21.15.43:0/3887725048 learned_addr learned my addr 172.21.15.43:0/3887725048 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:31.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.604+0000 7fd833e00640 1 --2- 172.21.15.43:0/3887725048 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd8341a2d50 0x7fd8341a3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.605+0000 7fd832a00640 1 -- 172.21.15.43:0/3887725048 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd8341a2d50 msgr2=0x7fd8341a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.605+0000 7fd832a00640 1 --2- 172.21.15.43:0/3887725048 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd8341a2d50 0x7fd8341a3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.605+0000 7fd832a00640 1 -- 172.21.15.43:0/3887725048 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd834107f00 msgr2=0x7fd83419e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.605+0000 7fd832a00640 1 --2- 172.21.15.43:0/3887725048 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd834107f00 0x7fd83419e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.605+0000 7fd832a00640 1 -- 172.21.15.43:0/3887725048 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8341a3840 con 0x7fd83410cf50 2024-03-28T22:45:31.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.605+0000 7fd832a00640 1 --2- 172.21.15.43:0/3887725048 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd83410cf50 0x7fd83419e9d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fd82000e970 tx=0x7fd82000ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:31.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.605+0000 7fd82be00640 1 -- 172.21.15.43:0/3887725048 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd82000cd80 con 0x7fd83410cf50 2024-03-28T22:45:31.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.605+0000 7fd82be00640 1 -- 172.21.15.43:0/3887725048 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8200035f0 con 0x7fd83410cf50 2024-03-28T22:45:31.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.605+0000 7fd82be00640 1 -- 172.21.15.43:0/3887725048 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd820010690 con 0x7fd83410cf50 2024-03-28T22:45:31.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.605+0000 7fd838a00640 1 -- 172.21.15.43:0/3887725048 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8341a3b30 con 0x7fd83410cf50 2024-03-28T22:45:31.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.606+0000 7fd838a00640 1 -- 172.21.15.43:0/3887725048 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8341ab2d0 con 0x7fd83410cf50 2024-03-28T22:45:31.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.609+0000 7fd83cc0f740 1 Processor -- start 2024-03-28T22:45:31.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.609+0000 7fd83cc0f740 1 -- start start 2024-03-28T22:45:31.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.609+0000 7fd83cc0f740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55671f959550 0x55671f959930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.609+0000 7fd83cc0f740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55671f959e70 0x55671f962510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.609+0000 7fd83cc0f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55671f962a50 0x55671f964e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.609+0000 7fd83cc0f740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55671f8bd050 con 0x55671f959e70 2024-03-28T22:45:31.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.609+0000 7fd83cc0f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55671f865ce0 con 0x55671f962a50 2024-03-28T22:45:31.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.609+0000 7fd83cc0f740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55671f8b90f0 con 0x55671f959550 2024-03-28T22:45:31.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.609+0000 7fd833e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55671f959550 0x55671f959930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.609+0000 7fd833e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55671f959550 0x55671f959930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43682/0 (socket says 172.21.15.43:43682) 2024-03-28T22:45:31.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.609+0000 7fd833e00640 1 -- 172.21.15.43:0/1139060602 learned_addr learned my addr 172.21.15.43:0/1139060602 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:31.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.609+0000 7fd82be00640 1 -- 172.21.15.43:0/3887725048 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fd8200037a0 con 0x7fd83410cf50 2024-03-28T22:45:31.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.609+0000 7fd833400640 1 --2- 172.21.15.43:0/1139060602 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55671f959e70 0x55671f962510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.610+0000 7fd833e00640 1 -- 172.21.15.43:0/1139060602 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55671f962a50 msgr2=0x55671f964e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:31.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.610+0000 7fd833e00640 1 --2- 172.21.15.43:0/1139060602 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55671f962a50 0x55671f964e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.610+0000 7fd833e00640 1 -- 172.21.15.43:0/1139060602 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55671f959e70 msgr2=0x55671f962510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:31.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.610+0000 7fd833e00640 1 --2- 172.21.15.43:0/1139060602 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55671f959e70 0x55671f962510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.610+0000 7fd833e00640 1 -- 172.21.15.43:0/1139060602 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55671f5d13f0 con 0x55671f959550 2024-03-28T22:45:31.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.610+0000 7fd833400640 1 --2- 172.21.15.43:0/1139060602 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55671f959e70 0x55671f962510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:45:31.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.610+0000 7fd833e00640 1 --2- 172.21.15.43:0/1139060602 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55671f959550 0x55671f959930 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fd82c009f60 tx=0x7fd82c009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8257c05495028b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:31.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.610+0000 7fd81fe00640 1 -- 172.21.15.43:0/1139060602 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd82c016070 con 0x55671f959550 2024-03-28T22:45:31.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.610+0000 7fd81fe00640 1 -- 172.21.15.43:0/1139060602 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd82c004d20 con 0x55671f959550 2024-03-28T22:45:31.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.610+0000 7fd81fe00640 1 -- 172.21.15.43:0/1139060602 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd82c0128f0 con 0x55671f959550 2024-03-28T22:45:31.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.611+0000 7fd83cc0f740 1 -- 172.21.15.43:0/1139060602 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55671f959550 msgr2=0x55671f959930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.611+0000 7fd83cc0f740 1 --2- 172.21.15.43:0/1139060602 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55671f959550 0x55671f959930 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fd82c009f60 tx=0x7fd82c009f90 comp rx=0 tx=0).stop 2024-03-28T22:45:31.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.611+0000 7fd82be00640 1 --2- 172.21.15.43:0/3887725048 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd8180a1ef0 0x7fd8180a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.611+0000 7fd83cc0f740 1 -- 172.21.15.43:0/1139060602 shutdown_connections 2024-03-28T22:45:31.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.611+0000 7fd83cc0f740 1 --2- 172.21.15.43:0/1139060602 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55671f962a50 0x55671f964e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.611+0000 7fd83cc0f740 1 --2- 172.21.15.43:0/1139060602 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55671f959e70 0x55671f962510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.611+0000 7fd83cc0f740 1 --2- 172.21.15.43:0/1139060602 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55671f959550 0x55671f959930 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.611+0000 7fd83cc0f740 1 -- 172.21.15.43:0/1139060602 >> 172.21.15.43:0/1139060602 conn(0x55671f953c80 msgr2=0x55671f954060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:31.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.611+0000 7fd82be00640 1 -- 172.21.15.43:0/3887725048 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7fd8200ce890 con 0x7fd83410cf50 2024-03-28T22:45:31.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.611+0000 7fd833400640 1 --2- 172.21.15.43:0/3887725048 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd8180a1ef0 0x7fd8180a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.611+0000 7fd83cc0f740 1 -- 172.21.15.43:0/1139060602 shutdown_connections 2024-03-28T22:45:31.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.611+0000 7fd83cc0f740 1 -- 172.21.15.43:0/1139060602 wait complete. 2024-03-28T22:45:31.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.612+0000 7fd83cc0f740 1 -- start start 2024-03-28T22:45:31.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.612+0000 7fd83cc0f740 1 Processor -- start 2024-03-28T22:45:31.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.612+0000 7fd833400640 1 --2- 172.21.15.43:0/3887725048 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd8180a1ef0 0x7fd8180a43b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd824000c00 tx=0x7fd82402df90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:31.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.613+0000 7fd83cc0f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55671f959550 0x55671f95fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.613+0000 7fd83cc0f740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55671f959e70 0x55671f9600f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.613+0000 7fd83cc0f740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55671f962a50 0x55671f960630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.613+0000 7fd83cc0f740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55671f965840 con 0x55671f959e70 2024-03-28T22:45:31.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.613+0000 7fd83cc0f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55671f85b2f0 con 0x55671f959550 2024-03-28T22:45:31.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.613+0000 7fd83cc0f740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55671f9659c0 con 0x55671f962a50 2024-03-28T22:45:31.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.613+0000 7fd83cc0f740 10 client.0 _ll_register_callbacks cb 0x55671f5d0be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:45:31.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.613+0000 7fd833400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55671f959550 0x55671f95fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.613+0000 7fd832a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55671f959e70 0x55671f9600f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.613+0000 7fd833e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55671f962a50 0x55671f960630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.613+0000 7fd833400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55671f959550 0x55671f95fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33954/0 (socket says 172.21.15.43:33954) 2024-03-28T22:45:31.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.613+0000 7fd833400640 1 -- 172.21.15.43:0/801936133 learned_addr learned my addr 172.21.15.43:0/801936133 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:31.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.613+0000 7fd833e00640 1 -- 172.21.15.43:0/801936133 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55671f959550 msgr2=0x55671f95fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:31.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.613+0000 7fd833e00640 1 --2- 172.21.15.43:0/801936133 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55671f959550 0x55671f95fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.613+0000 7fd833e00640 1 -- 172.21.15.43:0/801936133 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55671f959e70 msgr2=0x55671f9600f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:31.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.613+0000 7fd833e00640 1 --2- 172.21.15.43:0/801936133 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55671f959e70 0x55671f9600f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.613+0000 7fd833e00640 1 -- 172.21.15.43:0/801936133 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55671f69b7d0 con 0x55671f962a50 2024-03-28T22:45:31.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.613+0000 7fd832a00640 1 --2- 172.21.15.43:0/801936133 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55671f959e70 0x55671f9600f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:45:31.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.613+0000 7fd833400640 1 --2- 172.21.15.43:0/801936133 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55671f959550 0x55671f95fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:45:31.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.614+0000 7fd833e00640 1 --2- 172.21.15.43:0/801936133 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55671f962a50 0x55671f960630 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fd82c012530 tx=0x7fd82c0128c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9fd7adc9927b8eb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:31.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.614+0000 7fd81cc00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:45:31.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.614+0000 7fd81cc00640 1 -- 172.21.15.43:0/801936133 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd82c029060 con 0x55671f962a50 2024-03-28T22:45:31.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.614+0000 7fd83cc0f740 1 -- 172.21.15.43:0/801936133 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd82c004d20 con 0x55671f962a50 2024-03-28T22:45:31.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.614+0000 7fd81cc00640 1 -- 172.21.15.43:0/801936133 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7fd82c012c40 con 0x55671f962a50 2024-03-28T22:45:31.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.614+0000 7fd81cc00640 1 -- 172.21.15.43:0/801936133 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd82c030050 con 0x55671f962a50 2024-03-28T22:45:31.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.614+0000 7fd83cc0f740 10 client.28501 fetch_fsmap learned FSMap version 32 2024-03-28T22:45:31.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.614+0000 7fd83cc0f740 1 -- 172.21.15.43:0/801936133 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55671f971c10 con 0x55671f962a50 2024-03-28T22:45:31.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.615+0000 7fd81cc00640 1 -- 172.21.15.43:0/801936133 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd82c004d20 con 0x55671f962a50 2024-03-28T22:45:31.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.615+0000 7fd81cc00640 1 -- 172.21.15.43:0/801936133 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd82c00f040 con 0x55671f962a50 2024-03-28T22:45:31.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.615+0000 7fd83cc0f740 10 client.28501 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:45:31.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.615+0000 7fd83cc0f740 10 client.28501 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:45:31.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.615+0000 7fd83cc0f740 20 client.28501 populate_metadata read hostname 'smithi043' 2024-03-28T22:45:31.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.615+0000 7fd83cc0f740 1 --2- 172.21.15.43:0/801936133 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55671f973070 0x55671f975460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.615+0000 7fd83cc0f740 4 client.28501 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:45:31.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.615+0000 7fd83cc0f740 1 -- 172.21.15.43:0/801936133 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55671f866680 con 0x55671f973070 2024-03-28T22:45:31.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.615+0000 7fd833400640 1 --2- 172.21.15.43:0/801936133 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55671f973070 0x55671f975460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.616+0000 7fd833400640 1 --2- 172.21.15.43:0/801936133 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55671f973070 0x55671f975460 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=107fc3d226236347 server_cookie=1b6230a7a948664e in_seq=0 out_seq=0 2024-03-28T22:45:31.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.616+0000 7fd81cc00640 10 client.28501 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:45:31.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.617+0000 7fd81cc00640 1 -- 172.21.15.43:0/801936133 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55671f866680 con 0x55671f973070 2024-03-28T22:45:31.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.617+0000 7fd81cc00640 10 client.28501 handle_command_reply: tid=0 2024-03-28T22:45:31.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.617+0000 7fd83cc0f740 1 client.28501 shutdown 2024-03-28T22:45:31.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.617+0000 7fd83cc0f740 1 -- 172.21.15.43:0/801936133 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55671f962a50 msgr2=0x55671f960630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.617+0000 7fd83cc0f740 1 --2- 172.21.15.43:0/801936133 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55671f962a50 0x55671f960630 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fd82c012530 tx=0x7fd82c0128c0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.617+0000 7fd83cc0f740 1 -- 172.21.15.43:0/801936133 shutdown_connections 2024-03-28T22:45:31.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.617+0000 7fd83cc0f740 1 --2- 172.21.15.43:0/801936133 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55671f973070 0x55671f975460 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.617+0000 7fd83cc0f740 1 --2- 172.21.15.43:0/801936133 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55671f962a50 0x55671f960630 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.617+0000 7fd83cc0f740 1 --2- 172.21.15.43:0/801936133 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55671f959e70 0x55671f9600f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.617+0000 7fd83cc0f740 1 --2- 172.21.15.43:0/801936133 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55671f959550 0x55671f95fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.617+0000 7fd83cc0f740 1 -- 172.21.15.43:0/801936133 >> 172.21.15.43:0/801936133 conn(0x55671f953c80 msgr2=0x55671f955140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:31.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.617+0000 7fd81cc00640 0 client.28501 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:45:31.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.617+0000 7fd83cc0f740 1 -- 172.21.15.43:0/801936133 shutdown_connections 2024-03-28T22:45:31.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.617+0000 7fd83cc0f740 1 -- 172.21.15.43:0/801936133 wait complete. 2024-03-28T22:45:31.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.617+0000 7fd83cc0f740 20 client.28501 trim_cache size 0 max 16384 2024-03-28T22:45:31.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.632+0000 7fd83cc0f740 1 Processor -- start 2024-03-28T22:45:31.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.632+0000 7fd83cc0f740 1 -- start start 2024-03-28T22:45:31.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.633+0000 7fd83cc0f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55671f959550 0x55671f9622e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.633+0000 7fd83cc0f740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55671f959e70 0x55671f952dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.633+0000 7fd83cc0f740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55671f962a50 0x55671f95eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.633+0000 7fd83cc0f740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd8341b5150 con 0x55671f959e70 2024-03-28T22:45:31.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.633+0000 7fd83cc0f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd8341b6c90 con 0x55671f959550 2024-03-28T22:45:31.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.633+0000 7fd83cc0f740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd8341b5820 con 0x55671f962a50 2024-03-28T22:45:31.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.633+0000 7fd833400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55671f959550 0x55671f9622e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.633+0000 7fd833400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55671f959550 0x55671f9622e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33956/0 (socket says 172.21.15.43:33956) 2024-03-28T22:45:31.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.633+0000 7fd833e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55671f962a50 0x55671f95eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.633+0000 7fd832a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55671f959e70 0x55671f952dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.633+0000 7fd833400640 1 -- 172.21.15.43:0/3482514289 learned_addr learned my addr 172.21.15.43:0/3482514289 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:31.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.633+0000 7fd833400640 1 -- 172.21.15.43:0/3482514289 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55671f962a50 msgr2=0x55671f95eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:31.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.633+0000 7fd833400640 1 --2- 172.21.15.43:0/3482514289 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55671f962a50 0x55671f95eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.633+0000 7fd833400640 1 -- 172.21.15.43:0/3482514289 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55671f959e70 msgr2=0x55671f952dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:31.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.633+0000 7fd833400640 1 --2- 172.21.15.43:0/3482514289 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55671f959e70 0x55671f952dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.633+0000 7fd833400640 1 -- 172.21.15.43:0/3482514289 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55671f959c60 con 0x55671f959550 2024-03-28T22:45:31.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.634+0000 7fd833400640 1 --2- 172.21.15.43:0/3482514289 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55671f959550 0x55671f9622e0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fd82402c240 tx=0x7fd824041000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=911e1275d6261882 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:31.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.634+0000 7fd81f400640 1 -- 172.21.15.43:0/3482514289 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd824045070 con 0x55671f959550 2024-03-28T22:45:31.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.634+0000 7fd81f400640 1 -- 172.21.15.43:0/3482514289 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd82402ccb0 con 0x55671f959550 2024-03-28T22:45:31.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.634+0000 7fd81f400640 1 -- 172.21.15.43:0/3482514289 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd824057820 con 0x55671f959550 2024-03-28T22:45:31.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.635+0000 7fd83cc0f740 1 -- 172.21.15.43:0/3482514289 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55671f959550 msgr2=0x55671f9622e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.635+0000 7fd83cc0f740 1 --2- 172.21.15.43:0/3482514289 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55671f959550 0x55671f9622e0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fd82402c240 tx=0x7fd824041000 comp rx=0 tx=0).stop 2024-03-28T22:45:31.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.635+0000 7fd83cc0f740 1 -- 172.21.15.43:0/3482514289 shutdown_connections 2024-03-28T22:45:31.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.635+0000 7fd83cc0f740 1 --2- 172.21.15.43:0/3482514289 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55671f962a50 0x55671f95eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.635+0000 7fd83cc0f740 1 --2- 172.21.15.43:0/3482514289 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55671f959e70 0x55671f952dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.635+0000 7fd83cc0f740 1 --2- 172.21.15.43:0/3482514289 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55671f959550 0x55671f9622e0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.635+0000 7fd83cc0f740 1 -- 172.21.15.43:0/3482514289 >> 172.21.15.43:0/3482514289 conn(0x55671f953c80 msgr2=0x55671f95e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:31.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.635+0000 7fd83cc0f740 1 -- 172.21.15.43:0/3482514289 shutdown_connections 2024-03-28T22:45:31.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.635+0000 7fd83cc0f740 1 -- 172.21.15.43:0/3482514289 wait complete. 2024-03-28T22:45:31.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.635+0000 7fd83cc0f740 1 -- start start 2024-03-28T22:45:31.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.635+0000 7fd83cc0f740 1 Processor -- start 2024-03-28T22:45:31.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.636+0000 7fd83cc0f740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55671f959550 0x55671f978850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.636+0000 7fd83cc0f740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55671f959e70 0x55671f975c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.636+0000 7fd83cc0f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55671f962a50 0x55671f976240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.636+0000 7fd83cc0f740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd8341b65c0 con 0x55671f959e70 2024-03-28T22:45:31.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.636+0000 7fd83cc0f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd8341b5ef0 con 0x55671f962a50 2024-03-28T22:45:31.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.636+0000 7fd83cc0f740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd8341b3150 con 0x55671f959550 2024-03-28T22:45:31.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.636+0000 7fd83cc0f740 10 client.0 _ll_register_callbacks cb 0x7fd82c009de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:45:31.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.636+0000 7fd833e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55671f962a50 0x55671f976240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.636+0000 7fd833400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55671f959550 0x55671f978850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.636+0000 7fd833e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55671f962a50 0x55671f976240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33960/0 (socket says 172.21.15.43:33960) 2024-03-28T22:45:31.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.636+0000 7fd833e00640 1 -- 172.21.15.43:0/3983625387 learned_addr learned my addr 172.21.15.43:0/3983625387 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:31.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.636+0000 7fd832a00640 1 --2- 172.21.15.43:0/3983625387 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55671f959e70 0x55671f975c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.637+0000 7fd833e00640 1 -- 172.21.15.43:0/3983625387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55671f959550 msgr2=0x55671f978850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:31.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.637+0000 7fd833e00640 1 --2- 172.21.15.43:0/3983625387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55671f959550 0x55671f978850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.637+0000 7fd833e00640 1 -- 172.21.15.43:0/3983625387 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55671f959e70 msgr2=0x55671f975c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:31.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.637+0000 7fd833e00640 1 --2- 172.21.15.43:0/3983625387 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55671f959e70 0x55671f975c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.637+0000 7fd833e00640 1 -- 172.21.15.43:0/3983625387 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55671f976a70 con 0x55671f962a50 2024-03-28T22:45:31.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.637+0000 7fd832a00640 1 --2- 172.21.15.43:0/3983625387 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55671f959e70 0x55671f975c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:45:31.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.637+0000 7fd833e00640 1 --2- 172.21.15.43:0/3983625387 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55671f962a50 0x55671f976240 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fd82c005b00 tx=0x7fd82c01a6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15291fe0a9336fc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:31.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.637+0000 7fd81cc00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:45:31.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.637+0000 7fd81cc00640 1 -- 172.21.15.43:0/3983625387 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd82c029040 con 0x55671f962a50 2024-03-28T22:45:31.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.637+0000 7fd83cc0f740 1 -- 172.21.15.43:0/3983625387 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd82402ccb0 con 0x55671f962a50 2024-03-28T22:45:31.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.637+0000 7fd81cc00640 1 -- 172.21.15.43:0/3983625387 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7fd82c0099f0 con 0x55671f962a50 2024-03-28T22:45:31.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.637+0000 7fd81cc00640 1 -- 172.21.15.43:0/3983625387 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd82c016070 con 0x55671f962a50 2024-03-28T22:45:31.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.637+0000 7fd83cc0f740 10 client.28568 fetch_fsmap learned FSMap version 32 2024-03-28T22:45:31.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.637+0000 7fd83cc0f740 1 -- 172.21.15.43:0/3983625387 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55671f9d9140 con 0x55671f962a50 2024-03-28T22:45:31.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.638+0000 7fd81cc00640 1 -- 172.21.15.43:0/3983625387 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd82402ccb0 con 0x55671f962a50 2024-03-28T22:45:31.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.638+0000 7fd81cc00640 1 -- 172.21.15.43:0/3983625387 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd82c025ac0 con 0x55671f962a50 2024-03-28T22:45:31.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.638+0000 7fd83cc0f740 10 client.28568 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:45:31.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.638+0000 7fd83cc0f740 10 client.28568 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:45:31.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.638+0000 7fd83cc0f740 20 client.28568 populate_metadata read hostname 'smithi043' 2024-03-28T22:45:31.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.638+0000 7fd83cc0f740 1 --2- 172.21.15.43:0/3983625387 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55671f9d97f0 0x55671f9dbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.638+0000 7fd83cc0f740 4 client.28568 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:45:31.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.638+0000 7fd83cc0f740 1 -- 172.21.15.43:0/3983625387 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55671f8ca590 con 0x55671f9d97f0 2024-03-28T22:45:31.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.638+0000 7fd833400640 1 --2- 172.21.15.43:0/3983625387 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55671f9d97f0 0x55671f9dbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.639+0000 7fd833400640 1 --2- 172.21.15.43:0/3983625387 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55671f9d97f0 0x55671f9dbbe0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=53d93f66e83f6561 server_cookie=b96acfd3413c1905 in_seq=0 out_seq=0 2024-03-28T22:45:31.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.639+0000 7fd81cc00640 10 client.28568 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:45:31.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.639+0000 7fd81cc00640 1 -- 172.21.15.43:0/3983625387 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55671f8ca590 con 0x55671f9d97f0 2024-03-28T22:45:31.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.639+0000 7fd81cc00640 10 client.28568 handle_command_reply: tid=0 2024-03-28T22:45:31.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.639+0000 7fd83cc0f740 1 client.28568 shutdown 2024-03-28T22:45:31.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.639+0000 7fd83cc0f740 1 -- 172.21.15.43:0/3983625387 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55671f962a50 msgr2=0x55671f976240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.639+0000 7fd83cc0f740 1 --2- 172.21.15.43:0/3983625387 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55671f962a50 0x55671f976240 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fd82c005b00 tx=0x7fd82c01a6c0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.639+0000 7fd83cc0f740 1 -- 172.21.15.43:0/3983625387 shutdown_connections 2024-03-28T22:45:31.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.639+0000 7fd83cc0f740 1 --2- 172.21.15.43:0/3983625387 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55671f9d97f0 0x55671f9dbbe0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.639+0000 7fd83cc0f740 1 --2- 172.21.15.43:0/3983625387 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55671f962a50 0x55671f976240 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.639+0000 7fd81cc00640 0 client.28568 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:45:31.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.639+0000 7fd83cc0f740 1 --2- 172.21.15.43:0/3983625387 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55671f959e70 0x55671f975c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.639+0000 7fd83cc0f740 1 --2- 172.21.15.43:0/3983625387 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55671f959550 0x55671f978850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.639+0000 7fd83cc0f740 1 -- 172.21.15.43:0/3983625387 >> 172.21.15.43:0/3983625387 conn(0x55671f953c80 msgr2=0x55671f95ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:31.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.639+0000 7fd83cc0f740 1 -- 172.21.15.43:0/3983625387 shutdown_connections 2024-03-28T22:45:31.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.639+0000 7fd83cc0f740 1 -- 172.21.15.43:0/3983625387 wait complete. 2024-03-28T22:45:31.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.639+0000 7fd83cc0f740 20 client.28568 trim_cache size 0 max 16384 2024-03-28T22:45:31.645 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:45:31.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.640+0000 7fd838a00640 1 -- 172.21.15.43:0/3887725048 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd8180a1ef0 msgr2=0x7fd8180a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.640+0000 7fd838a00640 1 --2- 172.21.15.43:0/3887725048 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd8180a1ef0 0x7fd8180a43b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd824000c00 tx=0x7fd82402df90 comp rx=0 tx=0).stop 2024-03-28T22:45:31.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.640+0000 7fd838a00640 1 -- 172.21.15.43:0/3887725048 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd83410cf50 msgr2=0x7fd83419e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.640+0000 7fd838a00640 1 --2- 172.21.15.43:0/3887725048 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd83410cf50 0x7fd83419e9d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fd82000e970 tx=0x7fd82000ee30 comp rx=0 tx=0).stop 2024-03-28T22:45:31.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.640+0000 7fd838a00640 1 -- 172.21.15.43:0/3887725048 shutdown_connections 2024-03-28T22:45:31.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.640+0000 7fd838a00640 1 --2- 172.21.15.43:0/3887725048 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd8180a1ef0 0x7fd8180a43b0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.640+0000 7fd838a00640 1 --2- 172.21.15.43:0/3887725048 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd83410cf50 0x7fd83419e9d0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.640+0000 7fd838a00640 1 --2- 172.21.15.43:0/3887725048 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd8341a2d50 0x7fd8341a3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.640+0000 7fd838a00640 1 --2- 172.21.15.43:0/3887725048 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd834107f00 0x7fd83419e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.640+0000 7fd838a00640 1 -- 172.21.15.43:0/3887725048 >> 172.21.15.43:0/3887725048 conn(0x7fd834077570 msgr2=0x7fd834077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:31.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.640+0000 7fd838a00640 1 -- 172.21.15.43:0/3887725048 shutdown_connections 2024-03-28T22:45:31.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.640+0000 7fd838a00640 1 -- 172.21.15.43:0/3887725048 wait complete. 2024-03-28T22:45:31.655 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:45:31.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.876+0000 7f32a4e00640 1 -- 172.21.15.43:0/1888433522 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32a0112590 msgr2=0x7f32a0114980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.876+0000 7f32a4e00640 1 --2- 172.21.15.43:0/1888433522 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32a0112590 0x7f32a0114980 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f3298009cd0 tx=0x7f329802c0f0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.877+0000 7f32a4e00640 1 -- 172.21.15.43:0/1888433522 shutdown_connections 2024-03-28T22:45:31.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.877+0000 7f32a4e00640 1 --2- 172.21.15.43:0/1888433522 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f32a00765a0 0x7f32a0112050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.877+0000 7f32a4e00640 1 --2- 172.21.15.43:0/1888433522 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32a0112590 0x7f32a0114980 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.877+0000 7f32a4e00640 1 --2- 172.21.15.43:0/1888433522 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32a0075c80 0x7f32a0076060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.877+0000 7f32a4e00640 1 -- 172.21.15.43:0/1888433522 >> 172.21.15.43:0/1888433522 conn(0x7f32a00ff3a0 msgr2=0x7f32a01017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:31.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.877+0000 7f32a4e00640 1 -- 172.21.15.43:0/1888433522 shutdown_connections 2024-03-28T22:45:31.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.877+0000 7f32a4e00640 1 -- 172.21.15.43:0/1888433522 wait complete. 2024-03-28T22:45:31.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.878+0000 7f32a4e00640 1 Processor -- start 2024-03-28T22:45:31.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.878+0000 7f32a4e00640 1 -- start start 2024-03-28T22:45:31.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.878+0000 7f32a4e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32a0075c80 0x7f32a019e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.878+0000 7f32a4e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f32a00765a0 0x7f32a019ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.878+0000 7f32a4e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32a0112590 0x7f32a01a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.878+0000 7f32a4e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f32a01180b0 con 0x7f32a0075c80 2024-03-28T22:45:31.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.878+0000 7f32a4e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f32a0117f30 con 0x7f32a00765a0 2024-03-28T22:45:31.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.878+0000 7f32a4e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f32a0118230 con 0x7f32a0112590 2024-03-28T22:45:31.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.879+0000 7f329de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32a0075c80 0x7f32a019e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.879+0000 7f329e800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32a0112590 0x7f32a01a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.879+0000 7f3297e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f32a00765a0 0x7f32a019ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.879+0000 7f3297e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f32a00765a0 0x7f32a019ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33972/0 (socket says 172.21.15.43:33972) 2024-03-28T22:45:31.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.879+0000 7f3297e00640 1 -- 172.21.15.43:0/3222581698 learned_addr learned my addr 172.21.15.43:0/3222581698 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:31.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.879+0000 7f329e800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32a0112590 0x7f32a01a2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43716/0 (socket says 172.21.15.43:43716) 2024-03-28T22:45:31.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.879+0000 7f3297e00640 1 -- 172.21.15.43:0/3222581698 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32a0112590 msgr2=0x7f32a01a2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.879+0000 7f3297e00640 1 --2- 172.21.15.43:0/3222581698 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32a0112590 0x7f32a01a2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.879+0000 7f3297e00640 1 -- 172.21.15.43:0/3222581698 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32a0075c80 msgr2=0x7f32a019e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.879+0000 7f3297e00640 1 --2- 172.21.15.43:0/3222581698 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32a0075c80 0x7f32a019e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.879+0000 7f3297e00640 1 -- 172.21.15.43:0/3222581698 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32a01a3680 con 0x7f32a00765a0 2024-03-28T22:45:31.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.879+0000 7f3297e00640 1 --2- 172.21.15.43:0/3222581698 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f32a00765a0 0x7f32a019ebd0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f328800e9f0 tx=0x7f328800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:31.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.880+0000 7f3296a00640 1 -- 172.21.15.43:0/3222581698 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328800cd80 con 0x7f32a00765a0 2024-03-28T22:45:31.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.880+0000 7f3296a00640 1 -- 172.21.15.43:0/3222581698 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3288010910 con 0x7f32a00765a0 2024-03-28T22:45:31.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.880+0000 7f32a4e00640 1 -- 172.21.15.43:0/3222581698 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32a01a3970 con 0x7f32a00765a0 2024-03-28T22:45:31.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.880+0000 7f3296a00640 1 -- 172.21.15.43:0/3222581698 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3288018990 con 0x7f32a00765a0 2024-03-28T22:45:31.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.880+0000 7f32a4e00640 1 -- 172.21.15.43:0/3222581698 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32a01ab200 con 0x7f32a00765a0 2024-03-28T22:45:31.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.883+0000 7f32a7b64740 1 Processor -- start 2024-03-28T22:45:31.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.883+0000 7f32a7b64740 1 -- start start 2024-03-28T22:45:31.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.884+0000 7f32a7b64740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558b2ca2e380 0x558b2ca2e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.884+0000 7f32a7b64740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558b2ca2eca0 0x558b2ca37340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.884+0000 7f32a7b64740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558b2ca37880 0x558b2ca39c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.884+0000 7f32a7b64740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x558b2c992050 con 0x558b2ca2e380 2024-03-28T22:45:31.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.884+0000 7f32a7b64740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x558b2c93ace0 con 0x558b2ca37880 2024-03-28T22:45:31.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.884+0000 7f32a7b64740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x558b2c98e0f0 con 0x558b2ca2eca0 2024-03-28T22:45:31.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.884+0000 7f329de00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558b2ca2eca0 0x558b2ca37340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.884+0000 7f329e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558b2ca2e380 0x558b2ca2e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.884+0000 7f329de00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558b2ca2eca0 0x558b2ca37340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43724/0 (socket says 172.21.15.43:43724) 2024-03-28T22:45:31.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.884+0000 7f329de00640 1 -- 172.21.15.43:0/2047276496 learned_addr learned my addr 172.21.15.43:0/2047276496 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:31.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.885+0000 7f329de00640 1 -- 172.21.15.43:0/2047276496 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558b2ca37880 msgr2=0x558b2ca39c70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:45:31.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.885+0000 7f329de00640 1 --2- 172.21.15.43:0/2047276496 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558b2ca37880 0x558b2ca39c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.885+0000 7f329de00640 1 -- 172.21.15.43:0/2047276496 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558b2ca2e380 msgr2=0x558b2ca2e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:31.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.885+0000 7f329de00640 1 --2- 172.21.15.43:0/2047276496 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558b2ca2e380 0x558b2ca2e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.885+0000 7f329de00640 1 -- 172.21.15.43:0/2047276496 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b2c6a63f0 con 0x558b2ca2eca0 2024-03-28T22:45:31.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.885+0000 7f3296a00640 1 -- 172.21.15.43:0/3222581698 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f3288010ac0 con 0x7f32a00765a0 2024-03-28T22:45:31.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.885+0000 7f329de00640 1 --2- 172.21.15.43:0/2047276496 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558b2ca2eca0 0x558b2ca37340 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f3290009cf0 tx=0x7f3290011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4929c3f98585edb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:31.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.885+0000 7f328f400640 1 -- 172.21.15.43:0/2047276496 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3290004230 con 0x558b2ca2eca0 2024-03-28T22:45:31.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.885+0000 7f328f400640 1 -- 172.21.15.43:0/2047276496 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32900043d0 con 0x558b2ca2eca0 2024-03-28T22:45:31.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.885+0000 7f328f400640 1 -- 172.21.15.43:0/2047276496 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3290019960 con 0x558b2ca2eca0 2024-03-28T22:45:31.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.886+0000 7f32a7b64740 1 -- 172.21.15.43:0/2047276496 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558b2ca2eca0 msgr2=0x558b2ca37340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.886+0000 7f32a7b64740 1 --2- 172.21.15.43:0/2047276496 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558b2ca2eca0 0x558b2ca37340 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f3290009cf0 tx=0x7f3290011810 comp rx=0 tx=0).stop 2024-03-28T22:45:31.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.886+0000 7f32a7b64740 1 -- 172.21.15.43:0/2047276496 shutdown_connections 2024-03-28T22:45:31.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.886+0000 7f32a7b64740 1 --2- 172.21.15.43:0/2047276496 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558b2ca37880 0x558b2ca39c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.886+0000 7f32a7b64740 1 --2- 172.21.15.43:0/2047276496 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558b2ca2eca0 0x558b2ca37340 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.886+0000 7f32a7b64740 1 --2- 172.21.15.43:0/2047276496 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558b2ca2e380 0x558b2ca2e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.886+0000 7f32a7b64740 1 -- 172.21.15.43:0/2047276496 >> 172.21.15.43:0/2047276496 conn(0x558b2ca28ab0 msgr2=0x558b2ca28e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:31.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.886+0000 7f32a7b64740 1 -- 172.21.15.43:0/2047276496 shutdown_connections 2024-03-28T22:45:31.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.886+0000 7f32a7b64740 1 -- 172.21.15.43:0/2047276496 wait complete. 2024-03-28T22:45:31.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.887+0000 7f3296a00640 1 --2- 172.21.15.43:0/3222581698 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f32840a1e80 0x7f32840a4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.887+0000 7f3296a00640 1 -- 172.21.15.43:0/3222581698 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7f32880ce810 con 0x7f32a00765a0 2024-03-28T22:45:31.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.887+0000 7f32a7b64740 1 -- start start 2024-03-28T22:45:31.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.887+0000 7f329de00640 1 --2- 172.21.15.43:0/3222581698 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f32840a1e80 0x7f32840a4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.887+0000 7f32a7b64740 1 Processor -- start 2024-03-28T22:45:31.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.888+0000 7f32a7b64740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558b2ca2e380 0x558b2ca34960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.888+0000 7f32a7b64740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558b2ca2eca0 0x558b2ca34ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.888+0000 7f32a7b64740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558b2ca37880 0x558b2ca353e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.888+0000 7f32a7b64740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x558b2ca3a670 con 0x558b2ca37880 2024-03-28T22:45:31.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.888+0000 7f32a7b64740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x558b2c9302f0 con 0x558b2ca2e380 2024-03-28T22:45:31.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.888+0000 7f32a7b64740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x558b2ca3a7f0 con 0x558b2ca2eca0 2024-03-28T22:45:31.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.888+0000 7f329de00640 1 --2- 172.21.15.43:0/3222581698 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f32840a1e80 0x7f32840a4340 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3290003fa0 tx=0x7f3290000bc0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:31.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.888+0000 7f32a7b64740 10 client.0 _ll_register_callbacks cb 0x558b2c6a5be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:45:31.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.888+0000 7f329e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558b2ca37880 0x558b2ca353e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.888+0000 7f3297e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558b2ca2eca0 0x558b2ca34ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.888+0000 7f329e800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558b2ca37880 0x558b2ca353e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50830/0 (socket says 172.21.15.43:50830) 2024-03-28T22:45:31.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.888+0000 7f329e800640 1 -- 172.21.15.43:0/200062590 learned_addr learned my addr 172.21.15.43:0/200062590 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:31.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.888+0000 7f329de00640 1 --2- 172.21.15.43:0/200062590 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558b2ca2e380 0x558b2ca34960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.889+0000 7f3297e00640 1 -- 172.21.15.43:0/200062590 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558b2ca2e380 msgr2=0x558b2ca34960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:31.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.889+0000 7f3297e00640 1 --2- 172.21.15.43:0/200062590 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558b2ca2e380 0x558b2ca34960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.889+0000 7f3297e00640 1 -- 172.21.15.43:0/200062590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558b2ca37880 msgr2=0x558b2ca353e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:31.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.889+0000 7f3297e00640 1 --2- 172.21.15.43:0/200062590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558b2ca37880 0x558b2ca353e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.889+0000 7f3297e00640 1 -- 172.21.15.43:0/200062590 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b2c7707d0 con 0x558b2ca2eca0 2024-03-28T22:45:31.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.889+0000 7f3297e00640 1 --2- 172.21.15.43:0/200062590 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558b2ca2eca0 0x558b2ca34ea0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f32880cbb00 tx=0x7f328809ae50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f5faf67886c547e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:31.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.889+0000 7f3283e00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:45:31.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.889+0000 7f3283e00640 1 -- 172.21.15.43:0/200062590 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328809e070 con 0x558b2ca2eca0 2024-03-28T22:45:31.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.889+0000 7f3283e00640 1 -- 172.21.15.43:0/200062590 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7f32880ce370 con 0x558b2ca2eca0 2024-03-28T22:45:31.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.889+0000 7f32a7b64740 1 -- 172.21.15.43:0/200062590 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32900043d0 con 0x558b2ca2eca0 2024-03-28T22:45:31.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.890+0000 7f3283e00640 1 -- 172.21.15.43:0/200062590 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f32880a3540 con 0x558b2ca2eca0 2024-03-28T22:45:31.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.890+0000 7f32a7b64740 10 client.28519 fetch_fsmap learned FSMap version 32 2024-03-28T22:45:31.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.890+0000 7f32a7b64740 1 -- 172.21.15.43:0/200062590 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558b2ca469d0 con 0x558b2ca2eca0 2024-03-28T22:45:31.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.890+0000 7f3283e00640 1 -- 172.21.15.43:0/200062590 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32900043d0 con 0x558b2ca2eca0 2024-03-28T22:45:31.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.890+0000 7f3283e00640 1 -- 172.21.15.43:0/200062590 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f32880ce050 con 0x558b2ca2eca0 2024-03-28T22:45:31.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.890+0000 7f32a7b64740 10 client.28519 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:45:31.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.890+0000 7f32a7b64740 10 client.28519 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:45:31.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.890+0000 7f32a7b64740 20 client.28519 populate_metadata read hostname 'smithi043' 2024-03-28T22:45:31.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.890+0000 7f32a7b64740 1 --2- 172.21.15.43:0/200062590 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x558b2ca47e90 0x558b2ca4a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.890+0000 7f32a7b64740 4 client.28519 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:45:31.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.890+0000 7f32a7b64740 1 -- 172.21.15.43:0/200062590 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b2c93b680 con 0x558b2ca47e90 2024-03-28T22:45:31.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.891+0000 7f329de00640 1 --2- 172.21.15.43:0/200062590 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x558b2ca47e90 0x558b2ca4a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.891+0000 7f329de00640 1 --2- 172.21.15.43:0/200062590 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x558b2ca47e90 0x558b2ca4a280 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b39721243fbd505 server_cookie=f4266615b95a90ef in_seq=0 out_seq=0 2024-03-28T22:45:31.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.891+0000 7f3283e00640 10 client.28519 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:45:31.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.892+0000 7f3283e00640 1 -- 172.21.15.43:0/200062590 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558b2c93b680 con 0x558b2ca47e90 2024-03-28T22:45:31.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.892+0000 7f3283e00640 10 client.28519 handle_command_reply: tid=0 2024-03-28T22:45:31.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.892+0000 7f32a7b64740 1 client.28519 shutdown 2024-03-28T22:45:31.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.893+0000 7f32a7b64740 1 -- 172.21.15.43:0/200062590 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558b2ca2eca0 msgr2=0x558b2ca34ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.893+0000 7f32a7b64740 1 --2- 172.21.15.43:0/200062590 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558b2ca2eca0 0x558b2ca34ea0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f32880cbb00 tx=0x7f328809ae50 comp rx=0 tx=0).stop 2024-03-28T22:45:31.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.893+0000 7f32a7b64740 1 -- 172.21.15.43:0/200062590 shutdown_connections 2024-03-28T22:45:31.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.893+0000 7f32a7b64740 1 --2- 172.21.15.43:0/200062590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558b2ca37880 0x558b2ca353e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.893+0000 7f32a7b64740 1 --2- 172.21.15.43:0/200062590 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558b2ca2eca0 0x558b2ca34ea0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.893+0000 7f32a7b64740 1 --2- 172.21.15.43:0/200062590 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x558b2ca47e90 0x558b2ca4a280 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.893+0000 7f32a7b64740 1 --2- 172.21.15.43:0/200062590 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558b2ca2e380 0x558b2ca34960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.893+0000 7f3283e00640 0 client.28519 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:45:31.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.893+0000 7f32a7b64740 1 -- 172.21.15.43:0/200062590 >> 172.21.15.43:0/200062590 conn(0x558b2ca28ab0 msgr2=0x558b2ca2a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:31.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.893+0000 7f32a7b64740 1 -- 172.21.15.43:0/200062590 shutdown_connections 2024-03-28T22:45:31.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.893+0000 7f32a7b64740 1 -- 172.21.15.43:0/200062590 wait complete. 2024-03-28T22:45:31.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.893+0000 7f32a7b64740 20 client.28519 trim_cache size 0 max 16384 2024-03-28T22:45:31.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.908+0000 7f32a7b64740 1 Processor -- start 2024-03-28T22:45:31.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.908+0000 7f32a7b64740 1 -- start start 2024-03-28T22:45:31.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.908+0000 7f32a7b64740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558b2ca2e380 0x558b2ca37090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.908+0000 7f32a7b64740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558b2ca2eca0 0x558b2ca43440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.908+0000 7f32a7b64740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558b2ca37880 0x558b2ca2a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.908+0000 7f32a7b64740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f32a01b5430 con 0x558b2ca2eca0 2024-03-28T22:45:31.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.908+0000 7f32a7b64740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f32a01b6f70 con 0x558b2ca2e380 2024-03-28T22:45:31.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.908+0000 7f32a7b64740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f32a01b5b00 con 0x558b2ca37880 2024-03-28T22:45:31.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.909+0000 7f329de00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558b2ca2e380 0x558b2ca37090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.909+0000 7f3297e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558b2ca2eca0 0x558b2ca43440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.909+0000 7f329de00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558b2ca2e380 0x558b2ca37090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:33992/0 (socket says 172.21.15.43:33992) 2024-03-28T22:45:31.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.909+0000 7f329e800640 1 --2- 172.21.15.43:0/2620079547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558b2ca37880 0x558b2ca2a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.909+0000 7f329de00640 1 -- 172.21.15.43:0/2620079547 learned_addr learned my addr 172.21.15.43:0/2620079547 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:31.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.909+0000 7f329de00640 1 -- 172.21.15.43:0/2620079547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558b2ca37880 msgr2=0x558b2ca2a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:31.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.909+0000 7f329de00640 1 --2- 172.21.15.43:0/2620079547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558b2ca37880 0x558b2ca2a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.909+0000 7f329de00640 1 -- 172.21.15.43:0/2620079547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558b2ca2eca0 msgr2=0x558b2ca43440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:31.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.909+0000 7f329de00640 1 --2- 172.21.15.43:0/2620079547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558b2ca2eca0 0x558b2ca43440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.909+0000 7f329de00640 1 -- 172.21.15.43:0/2620079547 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b2ca39fe0 con 0x558b2ca2e380 2024-03-28T22:45:31.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.910+0000 7f329de00640 1 --2- 172.21.15.43:0/2620079547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558b2ca2e380 0x558b2ca37090 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f3290003e30 tx=0x7f3290005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae7b0890e4f56d14 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:31.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.910+0000 7f328ea00640 1 -- 172.21.15.43:0/2620079547 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3290037070 con 0x558b2ca2e380 2024-03-28T22:45:31.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.910+0000 7f328ea00640 1 -- 172.21.15.43:0/2620079547 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f329003d070 con 0x558b2ca2e380 2024-03-28T22:45:31.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.910+0000 7f328ea00640 1 -- 172.21.15.43:0/2620079547 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32900316b0 con 0x558b2ca2e380 2024-03-28T22:45:31.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.911+0000 7f32a7b64740 1 -- 172.21.15.43:0/2620079547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558b2ca2e380 msgr2=0x558b2ca37090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.911+0000 7f32a7b64740 1 --2- 172.21.15.43:0/2620079547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558b2ca2e380 0x558b2ca37090 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f3290003e30 tx=0x7f3290005f90 comp rx=0 tx=0).stop 2024-03-28T22:45:31.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.911+0000 7f32a7b64740 1 -- 172.21.15.43:0/2620079547 shutdown_connections 2024-03-28T22:45:31.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.911+0000 7f32a7b64740 1 --2- 172.21.15.43:0/2620079547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558b2ca37880 0x558b2ca2a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.911+0000 7f32a7b64740 1 --2- 172.21.15.43:0/2620079547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558b2ca2eca0 0x558b2ca43440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.911+0000 7f32a7b64740 1 --2- 172.21.15.43:0/2620079547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558b2ca2e380 0x558b2ca37090 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.911+0000 7f32a7b64740 1 -- 172.21.15.43:0/2620079547 >> 172.21.15.43:0/2620079547 conn(0x558b2ca28ab0 msgr2=0x558b2ca29f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:31.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.911+0000 7f32a7b64740 1 -- 172.21.15.43:0/2620079547 shutdown_connections 2024-03-28T22:45:31.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.911+0000 7f32a7b64740 1 -- 172.21.15.43:0/2620079547 wait complete. 2024-03-28T22:45:31.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.911+0000 7f32a7b64740 1 -- start start 2024-03-28T22:45:31.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.912+0000 7f32a7b64740 1 Processor -- start 2024-03-28T22:45:31.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.912+0000 7f32a7b64740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558b2ca2e380 0x558b2ca31cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.912+0000 7f32a7b64740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558b2ca2eca0 0x558b2ca32210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.912+0000 7f32a7b64740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558b2ca37880 0x558b2caad020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.912+0000 7f32a7b64740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f32a01b68a0 con 0x558b2ca2eca0 2024-03-28T22:45:31.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.912+0000 7f32a7b64740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f32a01b61d0 con 0x558b2ca37880 2024-03-28T22:45:31.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.912+0000 7f32a7b64740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f32a01b3430 con 0x558b2ca2e380 2024-03-28T22:45:31.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.912+0000 7f32a7b64740 10 client.0 _ll_register_callbacks cb 0x7f328809a990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:45:31.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.912+0000 7f329de00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558b2ca2e380 0x558b2ca31cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.912+0000 7f3297e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558b2ca2eca0 0x558b2ca32210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.913+0000 7f329e800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558b2ca37880 0x558b2caad020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.913+0000 7f329de00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558b2ca2e380 0x558b2ca31cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43758/0 (socket says 172.21.15.43:43758) 2024-03-28T22:45:31.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.913+0000 7f329de00640 1 -- 172.21.15.43:0/1570881033 learned_addr learned my addr 172.21.15.43:0/1570881033 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:31.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.913+0000 7f329de00640 1 -- 172.21.15.43:0/1570881033 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558b2ca37880 msgr2=0x558b2caad020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:31.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.913+0000 7f329de00640 1 --2- 172.21.15.43:0/1570881033 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558b2ca37880 0x558b2caad020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.913+0000 7f329de00640 1 -- 172.21.15.43:0/1570881033 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558b2ca2eca0 msgr2=0x558b2ca32210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:31.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.913+0000 7f329de00640 1 --2- 172.21.15.43:0/1570881033 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558b2ca2eca0 0x558b2ca32210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.913+0000 7f329de00640 1 -- 172.21.15.43:0/1570881033 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b2caad5b0 con 0x558b2ca2e380 2024-03-28T22:45:31.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.913+0000 7f3297e00640 1 --2- 172.21.15.43:0/1570881033 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558b2ca2eca0 0x558b2ca32210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:45:31.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.913+0000 7f329de00640 1 --2- 172.21.15.43:0/1570881033 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558b2ca2e380 0x558b2ca31cd0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3290011d10 tx=0x7f3290009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e31ba05049090237 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:31.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.913+0000 7f3283e00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:45:31.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.914+0000 7f3283e00640 1 -- 172.21.15.43:0/1570881033 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3290045020 con 0x558b2ca2e380 2024-03-28T22:45:31.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.914+0000 7f3283e00640 1 -- 172.21.15.43:0/1570881033 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7f3290019420 con 0x558b2ca2e380 2024-03-28T22:45:31.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.914+0000 7f32a7b64740 1 -- 172.21.15.43:0/1570881033 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f329003d070 con 0x558b2ca2e380 2024-03-28T22:45:31.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.914+0000 7f3283e00640 1 -- 172.21.15.43:0/1570881033 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3290050050 con 0x558b2ca2e380 2024-03-28T22:45:31.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.914+0000 7f32a7b64740 10 client.28525 fetch_fsmap learned FSMap version 32 2024-03-28T22:45:31.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.914+0000 7f32a7b64740 1 -- 172.21.15.43:0/1570881033 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558b2caae8b0 con 0x558b2ca2e380 2024-03-28T22:45:31.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.914+0000 7f3283e00640 1 -- 172.21.15.43:0/1570881033 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f329003d070 con 0x558b2ca2e380 2024-03-28T22:45:31.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.915+0000 7f3283e00640 1 -- 172.21.15.43:0/1570881033 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f3290031750 con 0x558b2ca2e380 2024-03-28T22:45:31.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.915+0000 7f32a7b64740 10 client.28525 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:45:31.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.915+0000 7f32a7b64740 10 client.28525 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:45:31.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.915+0000 7f32a7b64740 20 client.28525 populate_metadata read hostname 'smithi043' 2024-03-28T22:45:31.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.915+0000 7f32a7b64740 1 --2- 172.21.15.43:0/1570881033 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x558b2caaee20 0x558b2cab1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:31.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.915+0000 7f32a7b64740 4 client.28525 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:45:31.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.915+0000 7f32a7b64740 1 -- 172.21.15.43:0/1570881033 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558b2c99f590 con 0x558b2caaee20 2024-03-28T22:45:31.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.915+0000 7f3297e00640 1 --2- 172.21.15.43:0/1570881033 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x558b2caaee20 0x558b2cab1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:31.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.916+0000 7f3297e00640 1 --2- 172.21.15.43:0/1570881033 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x558b2caaee20 0x558b2cab1210 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25a6db76c489bdac server_cookie=7c0f4ef031d3aa85 in_seq=0 out_seq=0 2024-03-28T22:45:31.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.916+0000 7f3283e00640 10 client.28525 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:45:31.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.916+0000 7f3283e00640 1 -- 172.21.15.43:0/1570881033 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558b2c99f590 con 0x558b2caaee20 2024-03-28T22:45:31.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.916+0000 7f3283e00640 10 client.28525 handle_command_reply: tid=0 2024-03-28T22:45:31.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.916+0000 7f32a7b64740 1 client.28525 shutdown 2024-03-28T22:45:31.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.916+0000 7f32a7b64740 1 -- 172.21.15.43:0/1570881033 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558b2ca2e380 msgr2=0x558b2ca31cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.916+0000 7f32a7b64740 1 --2- 172.21.15.43:0/1570881033 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558b2ca2e380 0x558b2ca31cd0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3290011d10 tx=0x7f3290009f90 comp rx=0 tx=0).stop 2024-03-28T22:45:31.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.916+0000 7f32a7b64740 1 -- 172.21.15.43:0/1570881033 shutdown_connections 2024-03-28T22:45:31.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.916+0000 7f32a7b64740 1 --2- 172.21.15.43:0/1570881033 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x558b2caaee20 0x558b2cab1210 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.916+0000 7f32a7b64740 1 --2- 172.21.15.43:0/1570881033 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558b2ca37880 0x558b2caad020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.916+0000 7f32a7b64740 1 --2- 172.21.15.43:0/1570881033 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x558b2ca2eca0 0x558b2ca32210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.916+0000 7f32a7b64740 1 --2- 172.21.15.43:0/1570881033 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x558b2ca2e380 0x558b2ca31cd0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.916+0000 7f3283e00640 0 client.28525 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:45:31.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.916+0000 7f32a7b64740 1 -- 172.21.15.43:0/1570881033 >> 172.21.15.43:0/1570881033 conn(0x558b2ca28ab0 msgr2=0x558b2ca2a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:31.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.916+0000 7f32a7b64740 1 -- 172.21.15.43:0/1570881033 shutdown_connections 2024-03-28T22:45:31.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.916+0000 7f32a7b64740 1 -- 172.21.15.43:0/1570881033 wait complete. 2024-03-28T22:45:31.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.916+0000 7f32a7b64740 20 client.28525 trim_cache size 0 max 16384 2024-03-28T22:45:31.920 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:45:31.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.917+0000 7f32a4e00640 1 -- 172.21.15.43:0/3222581698 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f32840a1e80 msgr2=0x7f32840a4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.917+0000 7f32a4e00640 1 --2- 172.21.15.43:0/3222581698 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f32840a1e80 0x7f32840a4340 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3290003fa0 tx=0x7f3290000bc0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.917+0000 7f32a4e00640 1 -- 172.21.15.43:0/3222581698 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f32a00765a0 msgr2=0x7f32a019ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:31.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.917+0000 7f32a4e00640 1 --2- 172.21.15.43:0/3222581698 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f32a00765a0 0x7f32a019ebd0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f328800e9f0 tx=0x7f328800eeb0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.917+0000 7f32a4e00640 1 -- 172.21.15.43:0/3222581698 shutdown_connections 2024-03-28T22:45:31.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.917+0000 7f32a4e00640 1 --2- 172.21.15.43:0/3222581698 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f32840a1e80 0x7f32840a4340 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.917+0000 7f32a4e00640 1 --2- 172.21.15.43:0/3222581698 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f32a00765a0 0x7f32a019ebd0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.917+0000 7f32a4e00640 1 --2- 172.21.15.43:0/3222581698 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f32a0112590 0x7f32a01a2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.917+0000 7f32a4e00640 1 --2- 172.21.15.43:0/3222581698 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32a0075c80 0x7f32a019e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:31.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.917+0000 7f32a4e00640 1 -- 172.21.15.43:0/3222581698 >> 172.21.15.43:0/3222581698 conn(0x7f32a00ff3a0 msgr2=0x7f32a0101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:31.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.917+0000 7f32a4e00640 1 -- 172.21.15.43:0/3222581698 shutdown_connections 2024-03-28T22:45:31.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:31.917+0000 7f32a4e00640 1 -- 172.21.15.43:0/3222581698 wait complete. 2024-03-28T22:45:32.935 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:45:32.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:45:32.935 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:45:33.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:32 smithi172 ceph-mon[36566]: pgmap v260: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:45:33.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:32 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:45:33.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.164+0000 7fc77b200640 1 -- 172.21.15.43:0/1244131730 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc77c107660 msgr2=0x7fc77c107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:33.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.164+0000 7fc77b200640 1 --2- 172.21.15.43:0/1244131730 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc77c107660 0x7fc77c107a40 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fc76c009900 tx=0x7fc76c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:45:33.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.164+0000 7fc77b200640 1 -- 172.21.15.43:0/1244131730 shutdown_connections 2024-03-28T22:45:33.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.164+0000 7fc77b200640 1 --2- 172.21.15.43:0/1244131730 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc77c101f80 0x7fc77c10e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.164+0000 7fc77b200640 1 --2- 172.21.15.43:0/1244131730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc77c1015c0 0x7fc77c101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.164+0000 7fc77b200640 1 --2- 172.21.15.43:0/1244131730 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc77c107660 0x7fc77c107a40 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.164+0000 7fc77b200640 1 -- 172.21.15.43:0/1244131730 >> 172.21.15.43:0/1244131730 conn(0x7fc77c0fd0f0 msgr2=0x7fc77c0ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:33.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.164+0000 7fc77b200640 1 -- 172.21.15.43:0/1244131730 shutdown_connections 2024-03-28T22:45:33.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.164+0000 7fc77b200640 1 -- 172.21.15.43:0/1244131730 wait complete. 2024-03-28T22:45:33.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.165+0000 7fc77b200640 1 Processor -- start 2024-03-28T22:45:33.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.166+0000 7fc77b200640 1 -- start start 2024-03-28T22:45:33.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.166+0000 7fc77b200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc77c1015c0 0x7fc77c19a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.166+0000 7fc77b200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc77c101f80 0x7fc77c19a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.166+0000 7fc77b200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc77c107660 0x7fc77c19e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.166+0000 7fc77b200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc77c1119f0 con 0x7fc77c1015c0 2024-03-28T22:45:33.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.166+0000 7fc77b200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fc77c111870 con 0x7fc77c101f80 2024-03-28T22:45:33.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.166+0000 7fc77b200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fc77c111b70 con 0x7fc77c107660 2024-03-28T22:45:33.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.166+0000 7fc77a800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc77c107660 0x7fc77c19e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.166+0000 7fc771400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc77c101f80 0x7fc77c19a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.166+0000 7fc779e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc77c1015c0 0x7fc77c19a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.166+0000 7fc77a800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc77c107660 0x7fc77c19e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43776/0 (socket says 172.21.15.43:43776) 2024-03-28T22:45:33.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.166+0000 7fc771400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc77c101f80 0x7fc77c19a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34012/0 (socket says 172.21.15.43:34012) 2024-03-28T22:45:33.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.166+0000 7fc77a800640 1 -- 172.21.15.43:0/3095061760 learned_addr learned my addr 172.21.15.43:0/3095061760 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:33.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.167+0000 7fc77a800640 1 -- 172.21.15.43:0/3095061760 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc77c101f80 msgr2=0x7fc77c19a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:33.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.167+0000 7fc77a800640 1 --2- 172.21.15.43:0/3095061760 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc77c101f80 0x7fc77c19a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.167+0000 7fc77a800640 1 -- 172.21.15.43:0/3095061760 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc77c1015c0 msgr2=0x7fc77c19a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:33.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.167+0000 7fc77a800640 1 --2- 172.21.15.43:0/3095061760 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc77c1015c0 0x7fc77c19a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.167+0000 7fc77a800640 1 -- 172.21.15.43:0/3095061760 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc77c19f040 con 0x7fc77c107660 2024-03-28T22:45:33.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.167+0000 7fc771400640 1 --2- 172.21.15.43:0/3095061760 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc77c101f80 0x7fc77c19a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:45:33.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.167+0000 7fc77a800640 1 --2- 172.21.15.43:0/3095061760 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc77c107660 0x7fc77c19e910 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fc774009fd0 tx=0x7fc77400b260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:33.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.167+0000 7fc773400640 1 -- 172.21.15.43:0/3095061760 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc774017070 con 0x7fc77c107660 2024-03-28T22:45:33.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.168+0000 7fc77b200640 1 -- 172.21.15.43:0/3095061760 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc77c19f330 con 0x7fc77c107660 2024-03-28T22:45:33.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.168+0000 7fc773400640 1 -- 172.21.15.43:0/3095061760 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc774003e50 con 0x7fc77c107660 2024-03-28T22:45:33.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.168+0000 7fc773400640 1 -- 172.21.15.43:0/3095061760 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7740106c0 con 0x7fc77c107660 2024-03-28T22:45:33.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.168+0000 7fc77b200640 1 -- 172.21.15.43:0/3095061760 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc77c1a6bc0 con 0x7fc77c107660 2024-03-28T22:45:33.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.171+0000 7fc773400640 1 -- 172.21.15.43:0/3095061760 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fc774003970 con 0x7fc77c107660 2024-03-28T22:45:33.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.171+0000 7fc783e55740 1 Processor -- start 2024-03-28T22:45:33.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.172+0000 7fc783e55740 1 -- start start 2024-03-28T22:45:33.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.172+0000 7fc783e55740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55923e002170 0x55923e002550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.172+0000 7fc783e55740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55923e002a90 0x55923e00b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.172+0000 7fc783e55740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55923e00b670 0x55923e00da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.172+0000 7fc783e55740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55923df65c40 con 0x55923e002a90 2024-03-28T22:45:33.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.172+0000 7fc783e55740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55923df0ec40 con 0x55923e00b670 2024-03-28T22:45:33.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.172+0000 7fc783e55740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55923df61ce0 con 0x55923e002170 2024-03-28T22:45:33.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.172+0000 7fc771400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55923e002170 0x55923e002550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.172+0000 7fc771400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55923e002170 0x55923e002550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43784/0 (socket says 172.21.15.43:43784) 2024-03-28T22:45:33.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.172+0000 7fc771400640 1 -- 172.21.15.43:0/3270233757 learned_addr learned my addr 172.21.15.43:0/3270233757 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:33.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.173+0000 7fc779e00640 1 --2- 172.21.15.43:0/3270233757 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55923e002a90 0x55923e00b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.173+0000 7fc771400640 1 --2- 172.21.15.43:0/3270233757 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55923e00b670 0x55923e00da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.173+0000 7fc771400640 1 -- 172.21.15.43:0/3270233757 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55923e00b670 msgr2=0x55923e00da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:33.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.173+0000 7fc771400640 1 --2- 172.21.15.43:0/3270233757 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55923e00b670 0x55923e00da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.173+0000 7fc771400640 1 -- 172.21.15.43:0/3270233757 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55923e002a90 msgr2=0x55923e00b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:33.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.173+0000 7fc771400640 1 --2- 172.21.15.43:0/3270233757 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55923e002a90 0x55923e00b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.173+0000 7fc773400640 1 --2- 172.21.15.43:0/3095061760 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc7600a1ef0 0x7fc7600a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.173+0000 7fc771400640 1 -- 172.21.15.43:0/3270233757 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55923dd427e0 con 0x55923e002170 2024-03-28T22:45:33.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.173+0000 7fc773400640 1 -- 172.21.15.43:0/3095061760 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7fc7740ce2f0 con 0x7fc77c107660 2024-03-28T22:45:33.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.174+0000 7fc771400640 1 --2- 172.21.15.43:0/3270233757 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55923e002170 0x55923e002550 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fc7640058c0 tx=0x7fc764009770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ad74bb60dfbba72 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:33.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.174+0000 7fc771400640 1 --2- 172.21.15.43:0/3095061760 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc7600a1ef0 0x7fc7600a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.174+0000 7fc76b400640 1 -- 172.21.15.43:0/3270233757 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc76400fc70 con 0x55923e002170 2024-03-28T22:45:33.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.174+0000 7fc76b400640 1 -- 172.21.15.43:0/3270233757 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc76400fe10 con 0x55923e002170 2024-03-28T22:45:33.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.174+0000 7fc76b400640 1 -- 172.21.15.43:0/3270233757 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc764022680 con 0x55923e002170 2024-03-28T22:45:33.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.175+0000 7fc783e55740 1 -- 172.21.15.43:0/3270233757 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55923e002170 msgr2=0x55923e002550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:33.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.175+0000 7fc771400640 1 --2- 172.21.15.43:0/3095061760 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc7600a1ef0 0x7fc7600a43b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc764003fd0 tx=0x7fc764003f40 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:33.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.175+0000 7fc783e55740 1 --2- 172.21.15.43:0/3270233757 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55923e002170 0x55923e002550 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fc7640058c0 tx=0x7fc764009770 comp rx=0 tx=0).stop 2024-03-28T22:45:33.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.175+0000 7fc783e55740 1 -- 172.21.15.43:0/3270233757 shutdown_connections 2024-03-28T22:45:33.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.175+0000 7fc783e55740 1 --2- 172.21.15.43:0/3270233757 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55923e00b670 0x55923e00da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.175+0000 7fc783e55740 1 --2- 172.21.15.43:0/3270233757 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55923e002a90 0x55923e00b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.175+0000 7fc783e55740 1 --2- 172.21.15.43:0/3270233757 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55923e002170 0x55923e002550 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.175+0000 7fc783e55740 1 -- 172.21.15.43:0/3270233757 >> 172.21.15.43:0/3270233757 conn(0x55923dffc8a0 msgr2=0x55923dffcc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:33.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.175+0000 7fc783e55740 1 -- 172.21.15.43:0/3270233757 shutdown_connections 2024-03-28T22:45:33.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.175+0000 7fc783e55740 1 -- 172.21.15.43:0/3270233757 wait complete. 2024-03-28T22:45:33.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.176+0000 7fc783e55740 1 -- start start 2024-03-28T22:45:33.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.176+0000 7fc783e55740 1 Processor -- start 2024-03-28T22:45:33.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.176+0000 7fc783e55740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55923e002170 0x55923e008640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.176+0000 7fc783e55740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55923e002a90 0x55923e008b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.176+0000 7fc783e55740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55923e00b670 0x55923e0090c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.176+0000 7fc783e55740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55923e00e460 con 0x55923e00b670 2024-03-28T22:45:33.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.176+0000 7fc783e55740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55923df04250 con 0x55923e002a90 2024-03-28T22:45:33.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.176+0000 7fc783e55740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55923e00e5e0 con 0x55923e002170 2024-03-28T22:45:33.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.177+0000 7fc783e55740 10 client.0 _ll_register_callbacks cb 0x55923df25e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:45:33.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.177+0000 7fc779e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55923e002170 0x55923e008640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.177+0000 7fc771400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55923e002a90 0x55923e008b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.177+0000 7fc779e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55923e002170 0x55923e008640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43794/0 (socket says 172.21.15.43:43794) 2024-03-28T22:45:33.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.177+0000 7fc771400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55923e002a90 0x55923e008b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34024/0 (socket says 172.21.15.43:34024) 2024-03-28T22:45:33.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.177+0000 7fc779e00640 1 -- 172.21.15.43:0/3606781350 learned_addr learned my addr 172.21.15.43:0/3606781350 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:33.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.177+0000 7fc77a800640 1 --2- 172.21.15.43:0/3606781350 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55923e00b670 0x55923e0090c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.177+0000 7fc771400640 1 -- 172.21.15.43:0/3606781350 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55923e002170 msgr2=0x55923e008640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:33.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.177+0000 7fc771400640 1 --2- 172.21.15.43:0/3606781350 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55923e002170 0x55923e008640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.177+0000 7fc771400640 1 -- 172.21.15.43:0/3606781350 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55923e00b670 msgr2=0x55923e0090c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:33.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.177+0000 7fc771400640 1 --2- 172.21.15.43:0/3606781350 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55923e00b670 0x55923e0090c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.177+0000 7fc771400640 1 -- 172.21.15.43:0/3606781350 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55923dda2f70 con 0x55923e002a90 2024-03-28T22:45:33.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.178+0000 7fc771400640 1 --2- 172.21.15.43:0/3606781350 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55923e002a90 0x55923e008b80 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fc76401e000 tx=0x7fc764016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92d8f1ef7685dfe7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:33.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.178+0000 7fc75fe00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:45:33.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.178+0000 7fc75fe00640 1 -- 172.21.15.43:0/3606781350 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc76400fc20 con 0x55923e002a90 2024-03-28T22:45:33.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.178+0000 7fc783e55740 1 -- 172.21.15.43:0/3606781350 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc76400fe10 con 0x55923e002a90 2024-03-28T22:45:33.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.178+0000 7fc75fe00640 1 -- 172.21.15.43:0/3606781350 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7fc764028030 con 0x55923e002a90 2024-03-28T22:45:33.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.178+0000 7fc75fe00640 1 -- 172.21.15.43:0/3606781350 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc76403e050 con 0x55923e002a90 2024-03-28T22:45:33.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.178+0000 7fc783e55740 10 client.28595 fetch_fsmap learned FSMap version 32 2024-03-28T22:45:33.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.178+0000 7fc783e55740 1 -- 172.21.15.43:0/3606781350 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55923dc3a170 con 0x55923e002a90 2024-03-28T22:45:33.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.179+0000 7fc75fe00640 1 -- 172.21.15.43:0/3606781350 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc76400fe10 con 0x55923e002a90 2024-03-28T22:45:33.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.179+0000 7fc75fe00640 1 -- 172.21.15.43:0/3606781350 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc764005430 con 0x55923e002a90 2024-03-28T22:45:33.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.179+0000 7fc783e55740 10 client.28595 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:45:33.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.179+0000 7fc783e55740 10 client.28595 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:45:33.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.179+0000 7fc783e55740 20 client.28595 populate_metadata read hostname 'smithi043' 2024-03-28T22:45:33.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.179+0000 7fc783e55740 1 --2- 172.21.15.43:0/3606781350 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55923e01b880 0x55923e01dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.179+0000 7fc783e55740 4 client.28595 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:45:33.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.179+0000 7fc783e55740 1 -- 172.21.15.43:0/3606781350 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55923df73180 con 0x55923e01b880 2024-03-28T22:45:33.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.179+0000 7fc779e00640 1 --2- 172.21.15.43:0/3606781350 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55923e01b880 0x55923e01dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.180+0000 7fc779e00640 1 --2- 172.21.15.43:0/3606781350 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55923e01b880 0x55923e01dc70 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1de9ced54aaaf63 server_cookie=1f77b09ad7f8bfbc in_seq=0 out_seq=0 2024-03-28T22:45:33.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.180+0000 7fc75fe00640 10 client.28595 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:45:33.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.180+0000 7fc75fe00640 1 -- 172.21.15.43:0/3606781350 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55923df73180 con 0x55923e01b880 2024-03-28T22:45:33.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.180+0000 7fc75fe00640 10 client.28595 handle_command_reply: tid=0 2024-03-28T22:45:33.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.180+0000 7fc783e55740 1 client.28595 shutdown 2024-03-28T22:45:33.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.181+0000 7fc783e55740 1 -- 172.21.15.43:0/3606781350 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55923e002a90 msgr2=0x55923e008b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:33.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.181+0000 7fc783e55740 1 --2- 172.21.15.43:0/3606781350 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55923e002a90 0x55923e008b80 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fc76401e000 tx=0x7fc764016000 comp rx=0 tx=0).stop 2024-03-28T22:45:33.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.181+0000 7fc783e55740 1 -- 172.21.15.43:0/3606781350 shutdown_connections 2024-03-28T22:45:33.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.181+0000 7fc783e55740 1 --2- 172.21.15.43:0/3606781350 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55923e01b880 0x55923e01dc70 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.181+0000 7fc783e55740 1 --2- 172.21.15.43:0/3606781350 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55923e002a90 0x55923e008b80 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.181+0000 7fc783e55740 1 --2- 172.21.15.43:0/3606781350 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55923e00b670 0x55923e0090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.181+0000 7fc783e55740 1 --2- 172.21.15.43:0/3606781350 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55923e002170 0x55923e008640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.181+0000 7fc783e55740 1 -- 172.21.15.43:0/3606781350 >> 172.21.15.43:0/3606781350 conn(0x55923dffc8a0 msgr2=0x55923dffe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:33.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.181+0000 7fc75fe00640 0 client.28595 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:45:33.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.181+0000 7fc783e55740 1 -- 172.21.15.43:0/3606781350 shutdown_connections 2024-03-28T22:45:33.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.181+0000 7fc783e55740 1 -- 172.21.15.43:0/3606781350 wait complete. 2024-03-28T22:45:33.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.181+0000 7fc783e55740 20 client.28595 trim_cache size 0 max 16384 2024-03-28T22:45:33.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.201+0000 7fc783e55740 1 Processor -- start 2024-03-28T22:45:33.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.201+0000 7fc783e55740 1 -- start start 2024-03-28T22:45:33.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.201+0000 7fc783e55740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55923e002170 0x55923e00ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.201+0000 7fc783e55740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55923e002a90 0x55923e017200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.201+0000 7fc783e55740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55923e00b670 0x55923dffb780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.201+0000 7fc783e55740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc77c1b0c00 con 0x55923e00b670 2024-03-28T22:45:33.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.201+0000 7fc783e55740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fc77c1b2740 con 0x55923e002170 2024-03-28T22:45:33.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.201+0000 7fc783e55740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fc77c1b12d0 con 0x55923e002a90 2024-03-28T22:45:33.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.201+0000 7fc779e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55923e002170 0x55923e00ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.201+0000 7fc771400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55923e002a90 0x55923e017200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.201+0000 7fc779e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55923e002170 0x55923e00ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34038/0 (socket says 172.21.15.43:34038) 2024-03-28T22:45:33.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.201+0000 7fc779e00640 1 -- 172.21.15.43:0/1690007230 learned_addr learned my addr 172.21.15.43:0/1690007230 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:33.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.201+0000 7fc77a800640 1 --2- 172.21.15.43:0/1690007230 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55923e00b670 0x55923dffb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.202+0000 7fc779e00640 1 -- 172.21.15.43:0/1690007230 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55923e002a90 msgr2=0x55923e017200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:33.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.202+0000 7fc779e00640 1 --2- 172.21.15.43:0/1690007230 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55923e002a90 0x55923e017200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.202+0000 7fc779e00640 1 -- 172.21.15.43:0/1690007230 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55923e00b670 msgr2=0x55923dffb780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:33.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.202+0000 7fc779e00640 1 --2- 172.21.15.43:0/1690007230 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55923e00b670 0x55923dffb780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.202+0000 7fc779e00640 1 -- 172.21.15.43:0/1690007230 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55923e017740 con 0x55923e002170 2024-03-28T22:45:33.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.202+0000 7fc77a800640 1 --2- 172.21.15.43:0/1690007230 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55923e00b670 0x55923dffb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:45:33.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.202+0000 7fc779e00640 1 --2- 172.21.15.43:0/1690007230 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55923e002170 0x55923e00ad70 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fc76c036040 tx=0x7fc76c000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e393204412185ed server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:33.202 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:32 smithi082 ceph-mon[36471]: pgmap v260: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:45:33.202 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:32 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:45:33.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.202+0000 7fc76aa00640 1 -- 172.21.15.43:0/1690007230 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc76c050070 con 0x55923e002170 2024-03-28T22:45:33.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.202+0000 7fc76aa00640 1 -- 172.21.15.43:0/1690007230 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc76c004300 con 0x55923e002170 2024-03-28T22:45:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.202+0000 7fc76aa00640 1 -- 172.21.15.43:0/1690007230 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc76c0051c0 con 0x55923e002170 2024-03-28T22:45:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.203+0000 7fc783e55740 1 -- 172.21.15.43:0/1690007230 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55923e002170 msgr2=0x55923e00ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.203+0000 7fc783e55740 1 --2- 172.21.15.43:0/1690007230 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55923e002170 0x55923e00ad70 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fc76c036040 tx=0x7fc76c000bc0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.203+0000 7fc783e55740 1 -- 172.21.15.43:0/1690007230 shutdown_connections 2024-03-28T22:45:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.203+0000 7fc783e55740 1 --2- 172.21.15.43:0/1690007230 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55923e00b670 0x55923dffb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.203+0000 7fc783e55740 1 --2- 172.21.15.43:0/1690007230 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55923e002a90 0x55923e017200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.203+0000 7fc783e55740 1 --2- 172.21.15.43:0/1690007230 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55923e002170 0x55923e00ad70 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.203+0000 7fc783e55740 1 -- 172.21.15.43:0/1690007230 >> 172.21.15.43:0/1690007230 conn(0x55923dffc8a0 msgr2=0x55923dffb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.203+0000 7fc783e55740 1 -- 172.21.15.43:0/1690007230 shutdown_connections 2024-03-28T22:45:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.203+0000 7fc783e55740 1 -- 172.21.15.43:0/1690007230 wait complete. 2024-03-28T22:45:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.204+0000 7fc783e55740 1 -- start start 2024-03-28T22:45:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.204+0000 7fc783e55740 1 Processor -- start 2024-03-28T22:45:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.204+0000 7fc783e55740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55923e002170 0x55923e013c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.204+0000 7fc783e55740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55923e002a90 0x55923e00ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.204+0000 7fc783e55740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55923e00b670 0x55923e00f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.204+0000 7fc783e55740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc77c1b2070 con 0x55923e002170 2024-03-28T22:45:33.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.204+0000 7fc783e55740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fc77c1b19a0 con 0x55923e00b670 2024-03-28T22:45:33.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.204+0000 7fc783e55740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fc77c1aec00 con 0x55923e002a90 2024-03-28T22:45:33.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.204+0000 7fc783e55740 10 client.0 _ll_register_callbacks cb 0x7fc76c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:45:33.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.205+0000 7fc771400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55923e002a90 0x55923e00ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.205+0000 7fc779e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55923e002170 0x55923e013c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.205+0000 7fc771400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55923e002a90 0x55923e00ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43812/0 (socket says 172.21.15.43:43812) 2024-03-28T22:45:33.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.205+0000 7fc771400640 1 -- 172.21.15.43:0/2152752923 learned_addr learned my addr 172.21.15.43:0/2152752923 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:33.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.205+0000 7fc77a800640 1 --2- 172.21.15.43:0/2152752923 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55923e00b670 0x55923e00f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.205+0000 7fc771400640 1 -- 172.21.15.43:0/2152752923 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55923e00b670 msgr2=0x55923e00f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.205+0000 7fc771400640 1 --2- 172.21.15.43:0/2152752923 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55923e00b670 0x55923e00f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.205+0000 7fc771400640 1 -- 172.21.15.43:0/2152752923 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55923e002170 msgr2=0x55923e013c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.205+0000 7fc771400640 1 --2- 172.21.15.43:0/2152752923 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55923e002170 0x55923e013c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.205+0000 7fc771400640 1 -- 172.21.15.43:0/2152752923 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55923e080a50 con 0x55923e002a90 2024-03-28T22:45:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.205+0000 7fc77a800640 1 --2- 172.21.15.43:0/2152752923 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55923e00b670 0x55923e00f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:45:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.205+0000 7fc771400640 1 --2- 172.21.15.43:0/2152752923 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55923e002a90 0x55923e00ef80 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x55923e007ea0 tx=0x7fc7640221b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=329cb03b4f7a1aea server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.205+0000 7fc75fe00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:45:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.206+0000 7fc75fe00640 1 -- 172.21.15.43:0/2152752923 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc764009ea0 con 0x55923e002a90 2024-03-28T22:45:33.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.206+0000 7fc783e55740 1 -- 172.21.15.43:0/2152752923 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc76c004300 con 0x55923e002a90 2024-03-28T22:45:33.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.206+0000 7fc75fe00640 1 -- 172.21.15.43:0/2152752923 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7fc7640114c0 con 0x55923e002a90 2024-03-28T22:45:33.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.206+0000 7fc75fe00640 1 -- 172.21.15.43:0/2152752923 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc76404f050 con 0x55923e002a90 2024-03-28T22:45:33.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.206+0000 7fc783e55740 10 client.28543 fetch_fsmap learned FSMap version 32 2024-03-28T22:45:33.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.206+0000 7fc783e55740 1 -- 172.21.15.43:0/2152752923 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55923e081bc0 con 0x55923e002a90 2024-03-28T22:45:33.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.206+0000 7fc75fe00640 1 -- 172.21.15.43:0/2152752923 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc76c004300 con 0x55923e002a90 2024-03-28T22:45:33.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.206+0000 7fc75fe00640 1 -- 172.21.15.43:0/2152752923 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc76404a030 con 0x55923e002a90 2024-03-28T22:45:33.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.206+0000 7fc783e55740 10 client.28543 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:45:33.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.206+0000 7fc783e55740 10 client.28543 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:45:33.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.206+0000 7fc783e55740 20 client.28543 populate_metadata read hostname 'smithi043' 2024-03-28T22:45:33.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.206+0000 7fc783e55740 1 --2- 172.21.15.43:0/2152752923 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55923e082360 0x55923e084780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.206+0000 7fc783e55740 4 client.28543 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:45:33.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.206+0000 7fc783e55740 1 -- 172.21.15.43:0/2152752923 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55923df6d4d0 con 0x55923e082360 2024-03-28T22:45:33.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.207+0000 7fc779e00640 1 --2- 172.21.15.43:0/2152752923 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55923e082360 0x55923e084780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.207+0000 7fc779e00640 1 --2- 172.21.15.43:0/2152752923 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55923e082360 0x55923e084780 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41ab17bd889c1f3c server_cookie=be0c99db62629c15 in_seq=0 out_seq=0 2024-03-28T22:45:33.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.207+0000 7fc75fe00640 10 client.28543 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:45:33.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.208+0000 7fc75fe00640 1 -- 172.21.15.43:0/2152752923 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55923df6d4d0 con 0x55923e082360 2024-03-28T22:45:33.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.208+0000 7fc75fe00640 10 client.28543 handle_command_reply: tid=0 2024-03-28T22:45:33.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.208+0000 7fc783e55740 1 client.28543 shutdown 2024-03-28T22:45:33.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.208+0000 7fc783e55740 1 -- 172.21.15.43:0/2152752923 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55923e002a90 msgr2=0x55923e00ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:33.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.208+0000 7fc783e55740 1 --2- 172.21.15.43:0/2152752923 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55923e002a90 0x55923e00ef80 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x55923e007ea0 tx=0x7fc7640221b0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.209+0000 7fc783e55740 1 -- 172.21.15.43:0/2152752923 shutdown_connections 2024-03-28T22:45:33.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.209+0000 7fc783e55740 1 --2- 172.21.15.43:0/2152752923 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55923e082360 0x55923e084780 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.209+0000 7fc783e55740 1 --2- 172.21.15.43:0/2152752923 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55923e00b670 0x55923e00f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.209+0000 7fc783e55740 1 --2- 172.21.15.43:0/2152752923 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55923e002a90 0x55923e00ef80 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.209+0000 7fc783e55740 1 --2- 172.21.15.43:0/2152752923 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55923e002170 0x55923e013c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.209+0000 7fc783e55740 1 -- 172.21.15.43:0/2152752923 >> 172.21.15.43:0/2152752923 conn(0x55923dffc8a0 msgr2=0x55923dffb900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:33.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.209+0000 7fc75fe00640 0 client.28543 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:45:33.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.209+0000 7fc783e55740 1 -- 172.21.15.43:0/2152752923 shutdown_connections 2024-03-28T22:45:33.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.209+0000 7fc783e55740 1 -- 172.21.15.43:0/2152752923 wait complete. 2024-03-28T22:45:33.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.209+0000 7fc783e55740 20 client.28543 trim_cache size 0 max 16384 2024-03-28T22:45:33.215 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:45:33.215 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-03-28T22:45:33.215 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "3218a2a5-8467-4b27-9533-fc3b6c4540ac", 2024-03-28T22:45:33.215 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-03-28T22:45:33.215 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:45:33.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.210+0000 7fc77b200640 1 -- 172.21.15.43:0/3095061760 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc7600a1ef0 msgr2=0x7fc7600a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:33.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.210+0000 7fc77b200640 1 --2- 172.21.15.43:0/3095061760 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc7600a1ef0 0x7fc7600a43b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc764003fd0 tx=0x7fc764003f40 comp rx=0 tx=0).stop 2024-03-28T22:45:33.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.210+0000 7fc77b200640 1 -- 172.21.15.43:0/3095061760 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc77c107660 msgr2=0x7fc77c19e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:33.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.210+0000 7fc77b200640 1 --2- 172.21.15.43:0/3095061760 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc77c107660 0x7fc77c19e910 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fc774009fd0 tx=0x7fc77400b260 comp rx=0 tx=0).stop 2024-03-28T22:45:33.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.210+0000 7fc77b200640 1 -- 172.21.15.43:0/3095061760 shutdown_connections 2024-03-28T22:45:33.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.210+0000 7fc77b200640 1 --2- 172.21.15.43:0/3095061760 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc7600a1ef0 0x7fc7600a43b0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.210+0000 7fc77b200640 1 --2- 172.21.15.43:0/3095061760 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc77c101f80 0x7fc77c19a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.210+0000 7fc77b200640 1 --2- 172.21.15.43:0/3095061760 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc77c107660 0x7fc77c19e910 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.210+0000 7fc77b200640 1 --2- 172.21.15.43:0/3095061760 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc77c1015c0 0x7fc77c19a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.210+0000 7fc77b200640 1 -- 172.21.15.43:0/3095061760 >> 172.21.15.43:0/3095061760 conn(0x7fc77c0fd0f0 msgr2=0x7fc77c10d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:33.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.210+0000 7fc77b200640 1 -- 172.21.15.43:0/3095061760 shutdown_connections 2024-03-28T22:45:33.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.210+0000 7fc77b200640 1 -- 172.21.15.43:0/3095061760 wait complete. 2024-03-28T22:45:33.224 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:32 smithi043 ceph-mon[31823]: pgmap v260: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:45:33.225 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:32 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:45:33.227 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:45:33.227 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:45:33.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.437+0000 7f24e6800640 1 -- 172.21.15.43:0/58625431 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24e00744a0 msgr2=0x7f24e010d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:33.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.437+0000 7f24e6800640 1 --2- 172.21.15.43:0/58625431 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24e00744a0 0x7f24e010d9e0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f24cc009900 tx=0x7f24cc02bcd0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.437+0000 7f24e6800640 1 -- 172.21.15.43:0/58625431 shutdown_connections 2024-03-28T22:45:33.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.437+0000 7f24e6800640 1 --2- 172.21.15.43:0/58625431 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24e00744a0 0x7f24e010d9e0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.437+0000 7f24e6800640 1 --2- 172.21.15.43:0/58625431 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24e010df20 0x7f24e0110370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.437+0000 7f24e6800640 1 --2- 172.21.15.43:0/58625431 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24e0073b80 0x7f24e0073f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.437+0000 7f24e6800640 1 -- 172.21.15.43:0/58625431 >> 172.21.15.43:0/58625431 conn(0x7f24e0078b50 msgr2=0x7f24e0078f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:33.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.437+0000 7f24e6800640 1 -- 172.21.15.43:0/58625431 shutdown_connections 2024-03-28T22:45:33.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.437+0000 7f24e6800640 1 -- 172.21.15.43:0/58625431 wait complete. 2024-03-28T22:45:33.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.438+0000 7f24e6800640 1 Processor -- start 2024-03-28T22:45:33.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.438+0000 7f24e6800640 1 -- start start 2024-03-28T22:45:33.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.439+0000 7f24e6800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24e0073b80 0x7f24e019a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.439+0000 7f24e6800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24e00744a0 0x7f24e019a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.439+0000 7f24e6800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24e010df20 0x7f24e019ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.439+0000 7f24e6800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f24e0113d10 con 0x7f24e010df20 2024-03-28T22:45:33.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.439+0000 7f24e6800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f24e0113b90 con 0x7f24e00744a0 2024-03-28T22:45:33.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.439+0000 7f24e6800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f24e0113e90 con 0x7f24e0073b80 2024-03-28T22:45:33.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.439+0000 7f24dea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24e00744a0 0x7f24e019a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.439+0000 7f24dfe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24e010df20 0x7f24e019ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.439+0000 7f24dea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24e00744a0 0x7f24e019a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34054/0 (socket says 172.21.15.43:34054) 2024-03-28T22:45:33.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.439+0000 7f24dea00640 1 -- 172.21.15.43:0/2460899777 learned_addr learned my addr 172.21.15.43:0/2460899777 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:33.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.439+0000 7f24df400640 1 --2- 172.21.15.43:0/2460899777 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24e0073b80 0x7f24e019a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.440+0000 7f24dea00640 1 -- 172.21.15.43:0/2460899777 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24e0073b80 msgr2=0x7f24e019a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:33.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.440+0000 7f24dea00640 1 --2- 172.21.15.43:0/2460899777 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24e0073b80 0x7f24e019a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.440+0000 7f24dea00640 1 -- 172.21.15.43:0/2460899777 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24e010df20 msgr2=0x7f24e019ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:33.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.440+0000 7f24dea00640 1 --2- 172.21.15.43:0/2460899777 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24e010df20 0x7f24e019ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.440+0000 7f24dea00640 1 -- 172.21.15.43:0/2460899777 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24e019f190 con 0x7f24e00744a0 2024-03-28T22:45:33.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.440+0000 7f24dfe00640 1 --2- 172.21.15.43:0/2460899777 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24e010df20 0x7f24e019ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:45:33.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.440+0000 7f24dea00640 1 --2- 172.21.15.43:0/2460899777 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24e00744a0 0x7f24e019a730 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f24cc000c00 tx=0x7f24cc004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:33.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.440+0000 7f24d7e00640 1 -- 172.21.15.43:0/2460899777 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24cc042020 con 0x7f24e00744a0 2024-03-28T22:45:33.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.440+0000 7f24d7e00640 1 -- 172.21.15.43:0/2460899777 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24cc0054e0 con 0x7f24e00744a0 2024-03-28T22:45:33.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.440+0000 7f24e6800640 1 -- 172.21.15.43:0/2460899777 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24e019f420 con 0x7f24e00744a0 2024-03-28T22:45:33.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.440+0000 7f24d7e00640 1 -- 172.21.15.43:0/2460899777 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24cc003af0 con 0x7f24e00744a0 2024-03-28T22:45:33.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.440+0000 7f24e6800640 1 -- 172.21.15.43:0/2460899777 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24e01a6d10 con 0x7f24e00744a0 2024-03-28T22:45:33.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.443+0000 7f24d7e00640 1 -- 172.21.15.43:0/2460899777 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f24cc045020 con 0x7f24e00744a0 2024-03-28T22:45:33.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.444+0000 7f24e9459740 1 Processor -- start 2024-03-28T22:45:33.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.444+0000 7f24e9459740 1 -- start start 2024-03-28T22:45:33.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.444+0000 7f24e9459740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559a35e32bc0 0x559a35e31910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.444+0000 7f24e9459740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559a35e31ee0 0x559a35e32360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.444+0000 7f24e9459740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a35e3b250 0x559a35e3d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.444+0000 7f24e9459740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x559a35d96100 con 0x559a35e3b250 2024-03-28T22:45:33.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.444+0000 7f24e9459740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x559a35d3eba0 con 0x559a35e31ee0 2024-03-28T22:45:33.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.444+0000 7f24e9459740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x559a35d921a0 con 0x559a35e32bc0 2024-03-28T22:45:33.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.445+0000 7f24df400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559a35e31ee0 0x559a35e32360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.445+0000 7f24dfe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559a35e32bc0 0x559a35e31910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.445+0000 7f24dea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a35e3b250 0x559a35e3d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.445+0000 7f24dfe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559a35e32bc0 0x559a35e31910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43840/0 (socket says 172.21.15.43:43840) 2024-03-28T22:45:33.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.445+0000 7f24dfe00640 1 -- 172.21.15.43:0/48203678 learned_addr learned my addr 172.21.15.43:0/48203678 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:33.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.445+0000 7f24df400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559a35e31ee0 0x559a35e32360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34058/0 (socket says 172.21.15.43:34058) 2024-03-28T22:45:33.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.445+0000 7f24dfe00640 1 -- 172.21.15.43:0/48203678 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559a35e31ee0 msgr2=0x559a35e32360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:33.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.445+0000 7f24dfe00640 1 --2- 172.21.15.43:0/48203678 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559a35e31ee0 0x559a35e32360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.445+0000 7f24dfe00640 1 -- 172.21.15.43:0/48203678 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a35e3b250 msgr2=0x559a35e3d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:33.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.445+0000 7f24dfe00640 1 --2- 172.21.15.43:0/48203678 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a35e3b250 0x559a35e3d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.445+0000 7f24dfe00640 1 -- 172.21.15.43:0/48203678 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a35d6f1a0 con 0x559a35e32bc0 2024-03-28T22:45:33.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.445+0000 7f24df400640 1 --2- 172.21.15.43:0/48203678 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559a35e31ee0 0x559a35e32360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:45:33.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.445+0000 7f24d7e00640 1 --2- 172.21.15.43:0/2460899777 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f24c40a1f70 0x7f24c40a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.445+0000 7f24dfe00640 1 --2- 172.21.15.43:0/48203678 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559a35e32bc0 0x559a35e31910 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f24d8011790 tx=0x7f24d8011c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=530d9d93799362ed server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:33.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.445+0000 7f24d7e00640 1 -- 172.21.15.43:0/2460899777 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7f24cc0efff0 con 0x7f24e00744a0 2024-03-28T22:45:33.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.446+0000 7f24d4c00640 1 -- 172.21.15.43:0/48203678 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d8003c30 con 0x559a35e32bc0 2024-03-28T22:45:33.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.446+0000 7f24df400640 1 --2- 172.21.15.43:0/2460899777 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f24c40a1f70 0x7f24c40a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.446+0000 7f24d4c00640 1 -- 172.21.15.43:0/48203678 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24d800fe80 con 0x559a35e32bc0 2024-03-28T22:45:33.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.446+0000 7f24d4c00640 1 -- 172.21.15.43:0/48203678 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d80193e0 con 0x559a35e32bc0 2024-03-28T22:45:33.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.446+0000 7f24e9459740 1 -- 172.21.15.43:0/48203678 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559a35e32bc0 msgr2=0x559a35e31910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:33.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.446+0000 7f24e9459740 1 --2- 172.21.15.43:0/48203678 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559a35e32bc0 0x559a35e31910 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f24d8011790 tx=0x7f24d8011c50 comp rx=0 tx=0).stop 2024-03-28T22:45:33.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.446+0000 7f24df400640 1 --2- 172.21.15.43:0/2460899777 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f24c40a1f70 0x7f24c40a4430 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x559a35e33110 tx=0x7f24d0005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:33.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.446+0000 7f24e9459740 1 -- 172.21.15.43:0/48203678 shutdown_connections 2024-03-28T22:45:33.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.446+0000 7f24e9459740 1 --2- 172.21.15.43:0/48203678 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559a35e31ee0 0x559a35e32360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.446+0000 7f24e9459740 1 --2- 172.21.15.43:0/48203678 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a35e3b250 0x559a35e3d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.446+0000 7f24e9459740 1 --2- 172.21.15.43:0/48203678 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559a35e32bc0 0x559a35e31910 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.446+0000 7f24e9459740 1 -- 172.21.15.43:0/48203678 >> 172.21.15.43:0/48203678 conn(0x559a35df8a10 msgr2=0x559a35e2c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:33.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.446+0000 7f24e9459740 1 -- 172.21.15.43:0/48203678 shutdown_connections 2024-03-28T22:45:33.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.447+0000 7f24e9459740 1 -- 172.21.15.43:0/48203678 wait complete. 2024-03-28T22:45:33.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.447+0000 7f24e9459740 1 -- start start 2024-03-28T22:45:33.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.447+0000 7f24e9459740 1 Processor -- start 2024-03-28T22:45:33.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.448+0000 7f24e9459740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559a35e31ee0 0x559a35e38220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.448+0000 7f24e9459740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a35e32bc0 0x559a35e38760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.448+0000 7f24e9459740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559a35e3b250 0x559a35e38ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.448+0000 7f24e9459740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x559a35e3e0c0 con 0x559a35e32bc0 2024-03-28T22:45:33.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.448+0000 7f24e9459740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x559a35d34250 con 0x559a35e3b250 2024-03-28T22:45:33.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.448+0000 7f24e9459740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x559a35e3e240 con 0x559a35e31ee0 2024-03-28T22:45:33.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.448+0000 7f24e9459740 10 client.0 _ll_register_callbacks cb 0x559a35aa9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:45:33.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.448+0000 7f24df400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559a35e31ee0 0x559a35e38220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.448+0000 7f24df400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559a35e31ee0 0x559a35e38220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43856/0 (socket says 172.21.15.43:43856) 2024-03-28T22:45:33.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.448+0000 7f24df400640 1 -- 172.21.15.43:0/2389169421 learned_addr learned my addr 172.21.15.43:0/2389169421 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:33.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.448+0000 7f24dfe00640 1 --2- 172.21.15.43:0/2389169421 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559a35e3b250 0x559a35e38ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.448+0000 7f24dea00640 1 --2- 172.21.15.43:0/2389169421 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a35e32bc0 0x559a35e38760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.448+0000 7f24df400640 1 -- 172.21.15.43:0/2389169421 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559a35e3b250 msgr2=0x559a35e38ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:33.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.448+0000 7f24df400640 1 --2- 172.21.15.43:0/2389169421 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559a35e3b250 0x559a35e38ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.448+0000 7f24df400640 1 -- 172.21.15.43:0/2389169421 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a35e32bc0 msgr2=0x559a35e38760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:33.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.448+0000 7f24df400640 1 --2- 172.21.15.43:0/2389169421 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a35e32bc0 0x559a35e38760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.448+0000 7f24df400640 1 -- 172.21.15.43:0/2389169421 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a35d50960 con 0x559a35e31ee0 2024-03-28T22:45:33.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.449+0000 7f24df400640 1 --2- 172.21.15.43:0/2389169421 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559a35e31ee0 0x559a35e38220 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f24d0004630 tx=0x7f24d0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f307e0eac863105b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:33.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.449+0000 7f24c9600640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:45:33.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.449+0000 7f24c9600640 1 -- 172.21.15.43:0/2389169421 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d0009be0 con 0x559a35e31ee0 2024-03-28T22:45:33.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.449+0000 7f24c9600640 1 -- 172.21.15.43:0/2389169421 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7f24d001f640 con 0x559a35e31ee0 2024-03-28T22:45:33.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.449+0000 7f24e9459740 1 -- 172.21.15.43:0/2389169421 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559a35da3640 con 0x559a35e31ee0 2024-03-28T22:45:33.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.449+0000 7f24c9600640 1 -- 172.21.15.43:0/2389169421 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f24d0032050 con 0x559a35e31ee0 2024-03-28T22:45:33.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.449+0000 7f24e9459740 10 client.28555 fetch_fsmap learned FSMap version 32 2024-03-28T22:45:33.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.449+0000 7f24e9459740 1 -- 172.21.15.43:0/2389169421 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559a35b747d0 con 0x559a35e31ee0 2024-03-28T22:45:33.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.450+0000 7f24c9600640 1 -- 172.21.15.43:0/2389169421 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x559a35da3640 con 0x559a35e31ee0 2024-03-28T22:45:33.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.450+0000 7f24c9600640 1 -- 172.21.15.43:0/2389169421 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f24d0027020 con 0x559a35e31ee0 2024-03-28T22:45:33.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.450+0000 7f24e9459740 10 client.28555 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:45:33.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.450+0000 7f24e9459740 10 client.28555 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:45:33.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.450+0000 7f24e9459740 20 client.28555 populate_metadata read hostname 'smithi043' 2024-03-28T22:45:33.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.450+0000 7f24e9459740 1 --2- 172.21.15.43:0/2389169421 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559a35e4b780 0x559a35e4dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.450+0000 7f24e9459740 4 client.28555 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:45:33.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.450+0000 7f24e9459740 1 -- 172.21.15.43:0/2389169421 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559a35d9d990 con 0x559a35e4b780 2024-03-28T22:45:33.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.450+0000 7f24dea00640 1 --2- 172.21.15.43:0/2389169421 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559a35e4b780 0x559a35e4dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.451+0000 7f24dea00640 1 --2- 172.21.15.43:0/2389169421 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559a35e4b780 0x559a35e4dbd0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a03cb1af607e324 server_cookie=3b6ceb06e9c14547 in_seq=0 out_seq=0 2024-03-28T22:45:33.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.451+0000 7f24c9600640 10 client.28555 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:45:33.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.452+0000 7f24c9600640 1 -- 172.21.15.43:0/2389169421 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559a35d9d990 con 0x559a35e4b780 2024-03-28T22:45:33.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.452+0000 7f24c9600640 10 client.28555 handle_command_reply: tid=0 2024-03-28T22:45:33.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.452+0000 7f24e9459740 1 client.28555 shutdown 2024-03-28T22:45:33.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.452+0000 7f24e9459740 1 -- 172.21.15.43:0/2389169421 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559a35e31ee0 msgr2=0x559a35e38220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:33.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.452+0000 7f24e9459740 1 --2- 172.21.15.43:0/2389169421 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559a35e31ee0 0x559a35e38220 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f24d0004630 tx=0x7f24d0004730 comp rx=0 tx=0).stop 2024-03-28T22:45:33.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.452+0000 7f24e9459740 1 -- 172.21.15.43:0/2389169421 shutdown_connections 2024-03-28T22:45:33.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.452+0000 7f24e9459740 1 --2- 172.21.15.43:0/2389169421 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559a35e4b780 0x559a35e4dbd0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.452+0000 7f24e9459740 1 --2- 172.21.15.43:0/2389169421 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559a35e3b250 0x559a35e38ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.452+0000 7f24e9459740 1 --2- 172.21.15.43:0/2389169421 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a35e32bc0 0x559a35e38760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.452+0000 7f24c9600640 0 client.28555 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:45:33.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.452+0000 7f24e9459740 1 --2- 172.21.15.43:0/2389169421 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559a35e31ee0 0x559a35e38220 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.452+0000 7f24e9459740 1 -- 172.21.15.43:0/2389169421 >> 172.21.15.43:0/2389169421 conn(0x559a35df8a10 msgr2=0x559a35e2e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:33.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.452+0000 7f24e9459740 1 -- 172.21.15.43:0/2389169421 shutdown_connections 2024-03-28T22:45:33.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.452+0000 7f24e9459740 1 -- 172.21.15.43:0/2389169421 wait complete. 2024-03-28T22:45:33.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.453+0000 7f24e9459740 20 client.28555 trim_cache size 0 max 16384 2024-03-28T22:45:33.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.472+0000 7f24e9459740 1 Processor -- start 2024-03-28T22:45:33.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.472+0000 7f24e9459740 1 -- start start 2024-03-28T22:45:33.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.472+0000 7f24e9459740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559a35e31ee0 0x559a35e3a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.472+0000 7f24e9459740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559a35e32bc0 0x559a35e37160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.472+0000 7f24e9459740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a35e3b250 0x559a35e2c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.472+0000 7f24e9459740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f24e01b0ec0 con 0x559a35e3b250 2024-03-28T22:45:33.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.472+0000 7f24e9459740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f24e01b2a00 con 0x559a35e32bc0 2024-03-28T22:45:33.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.472+0000 7f24e9459740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f24e01b1590 con 0x559a35e31ee0 2024-03-28T22:45:33.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.472+0000 7f24df400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559a35e31ee0 0x559a35e3a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.472+0000 7f24dea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559a35e32bc0 0x559a35e37160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.472+0000 7f24df400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559a35e31ee0 0x559a35e3a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43868/0 (socket says 172.21.15.43:43868) 2024-03-28T22:45:33.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.472+0000 7f24df400640 1 -- 172.21.15.43:0/3863636942 learned_addr learned my addr 172.21.15.43:0/3863636942 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:33.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.473+0000 7f24dfe00640 1 --2- 172.21.15.43:0/3863636942 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a35e3b250 0x559a35e2c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.473+0000 7f24df400640 1 -- 172.21.15.43:0/3863636942 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559a35e32bc0 msgr2=0x559a35e37160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:33.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.473+0000 7f24df400640 1 --2- 172.21.15.43:0/3863636942 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559a35e32bc0 0x559a35e37160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.473+0000 7f24df400640 1 -- 172.21.15.43:0/3863636942 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a35e3b250 msgr2=0x559a35e2c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:33.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.473+0000 7f24df400640 1 --2- 172.21.15.43:0/3863636942 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a35e3b250 0x559a35e2c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.473+0000 7f24df400640 1 -- 172.21.15.43:0/3863636942 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a35aaa3f0 con 0x559a35e31ee0 2024-03-28T22:45:33.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.473+0000 7f24dea00640 1 --2- 172.21.15.43:0/3863636942 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559a35e32bc0 0x559a35e37160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:45:33.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.473+0000 7f24dfe00640 1 --2- 172.21.15.43:0/3863636942 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a35e3b250 0x559a35e2c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:45:33.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.473+0000 7f24df400640 1 --2- 172.21.15.43:0/3863636942 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559a35e31ee0 0x559a35e3a950 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f24d001db20 tx=0x7f24d00092a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd84d524417e341 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:33.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.473+0000 7f24cbe00640 1 -- 172.21.15.43:0/3863636942 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d001dd10 con 0x559a35e31ee0 2024-03-28T22:45:33.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.473+0000 7f24cbe00640 1 -- 172.21.15.43:0/3863636942 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24d001f640 con 0x559a35e31ee0 2024-03-28T22:45:33.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.474+0000 7f24cbe00640 1 -- 172.21.15.43:0/3863636942 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d000f680 con 0x559a35e31ee0 2024-03-28T22:45:33.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.474+0000 7f24e9459740 1 -- 172.21.15.43:0/3863636942 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559a35e31ee0 msgr2=0x559a35e3a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:33.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.474+0000 7f24e9459740 1 --2- 172.21.15.43:0/3863636942 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559a35e31ee0 0x559a35e3a950 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f24d001db20 tx=0x7f24d00092a0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.474+0000 7f24e9459740 1 -- 172.21.15.43:0/3863636942 shutdown_connections 2024-03-28T22:45:33.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.474+0000 7f24e9459740 1 --2- 172.21.15.43:0/3863636942 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559a35e32bc0 0x559a35e37160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.474+0000 7f24e9459740 1 --2- 172.21.15.43:0/3863636942 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a35e3b250 0x559a35e2c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.474+0000 7f24e9459740 1 --2- 172.21.15.43:0/3863636942 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559a35e31ee0 0x559a35e3a950 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.474+0000 7f24e9459740 1 -- 172.21.15.43:0/3863636942 >> 172.21.15.43:0/3863636942 conn(0x559a35df8a10 msgr2=0x559a35e2e610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:33.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.474+0000 7f24e9459740 1 -- 172.21.15.43:0/3863636942 shutdown_connections 2024-03-28T22:45:33.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.474+0000 7f24e9459740 1 -- 172.21.15.43:0/3863636942 wait complete. 2024-03-28T22:45:33.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.475+0000 7f24e9459740 1 -- start start 2024-03-28T22:45:33.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.475+0000 7f24e9459740 1 Processor -- start 2024-03-28T22:45:33.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.475+0000 7f24e9459740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559a35e31ee0 0x559a35eaade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.475+0000 7f24e9459740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559a35e32bc0 0x559a35eab320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.475+0000 7f24e9459740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a35e3b250 0x559a35e3f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.475+0000 7f24e9459740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f24e01b2330 con 0x559a35e3b250 2024-03-28T22:45:33.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.475+0000 7f24e9459740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f24e01b1c60 con 0x559a35e32bc0 2024-03-28T22:45:33.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.475+0000 7f24e9459740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f24e01aeec0 con 0x559a35e31ee0 2024-03-28T22:45:33.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.475+0000 7f24e9459740 10 client.0 _ll_register_callbacks cb 0x7f24d0009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:45:33.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.476+0000 7f24df400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559a35e31ee0 0x559a35eaade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.476+0000 7f24dea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559a35e32bc0 0x559a35eab320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.476+0000 7f24df400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559a35e31ee0 0x559a35eaade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:43878/0 (socket says 172.21.15.43:43878) 2024-03-28T22:45:33.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.476+0000 7f24df400640 1 -- 172.21.15.43:0/4267838019 learned_addr learned my addr 172.21.15.43:0/4267838019 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:45:33.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.476+0000 7f24dfe00640 1 --2- 172.21.15.43:0/4267838019 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a35e3b250 0x559a35e3f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.476+0000 7f24df400640 1 -- 172.21.15.43:0/4267838019 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559a35e32bc0 msgr2=0x559a35eab320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:33.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.476+0000 7f24df400640 1 --2- 172.21.15.43:0/4267838019 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559a35e32bc0 0x559a35eab320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.476+0000 7f24df400640 1 -- 172.21.15.43:0/4267838019 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a35e3b250 msgr2=0x559a35e3f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:45:33.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.476+0000 7f24df400640 1 --2- 172.21.15.43:0/4267838019 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a35e3b250 0x559a35e3f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.476+0000 7f24df400640 1 -- 172.21.15.43:0/4267838019 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a35a6a0f0 con 0x559a35e31ee0 2024-03-28T22:45:33.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.476+0000 7f24dea00640 1 --2- 172.21.15.43:0/4267838019 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559a35e32bc0 0x559a35eab320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:45:33.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.476+0000 7f24df400640 1 --2- 172.21.15.43:0/4267838019 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559a35e31ee0 0x559a35eaade0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f24d001dbc0 tx=0x7f24d001dec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a3f40f9348933ab server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:45:33.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.476+0000 7f24c9600640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:45:33.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.476+0000 7f24c9600640 1 -- 172.21.15.43:0/4267838019 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d0042020 con 0x559a35e31ee0 2024-03-28T22:45:33.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.477+0000 7f24c9600640 1 -- 172.21.15.43:0/4267838019 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7f24d001f7f0 con 0x559a35e31ee0 2024-03-28T22:45:33.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.477+0000 7f24e9459740 1 -- 172.21.15.43:0/4267838019 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24d001f640 con 0x559a35e31ee0 2024-03-28T22:45:33.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.477+0000 7f24c9600640 1 -- 172.21.15.43:0/4267838019 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f24d004d050 con 0x559a35e31ee0 2024-03-28T22:45:33.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.477+0000 7f24e9459740 10 client.28561 fetch_fsmap learned FSMap version 32 2024-03-28T22:45:33.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.477+0000 7f24e9459740 1 -- 172.21.15.43:0/4267838019 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559a35eb23c0 con 0x559a35e31ee0 2024-03-28T22:45:33.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.477+0000 7f24c9600640 1 -- 172.21.15.43:0/4267838019 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24d001f640 con 0x559a35e31ee0 2024-03-28T22:45:33.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.477+0000 7f24c9600640 1 -- 172.21.15.43:0/4267838019 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f24d0027040 con 0x559a35e31ee0 2024-03-28T22:45:33.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.477+0000 7f24e9459740 10 client.28561 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:45:33.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.477+0000 7f24e9459740 10 client.28561 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:45:33.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.477+0000 7f24e9459740 20 client.28561 populate_metadata read hostname 'smithi043' 2024-03-28T22:45:33.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.477+0000 7f24e9459740 1 --2- 172.21.15.43:0/4267838019 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559a35eb2a10 0x559a35eb4e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:45:33.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.477+0000 7f24e9459740 4 client.28561 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:45:33.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.477+0000 7f24e9459740 1 -- 172.21.15.43:0/4267838019 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559a35d1b400 con 0x559a35eb2a10 2024-03-28T22:45:33.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.478+0000 7f24dea00640 1 --2- 172.21.15.43:0/4267838019 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559a35eb2a10 0x559a35eb4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:45:33.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.478+0000 7f24dea00640 1 --2- 172.21.15.43:0/4267838019 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559a35eb2a10 0x559a35eb4e30 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db0eadf0d5440bda server_cookie=bcdb8ffff22a6ee9 in_seq=0 out_seq=0 2024-03-28T22:45:33.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.478+0000 7f24c9600640 10 client.28561 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:45:33.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.478+0000 7f24c9600640 1 -- 172.21.15.43:0/4267838019 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x559a35d1b400 con 0x559a35eb2a10 2024-03-28T22:45:33.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.478+0000 7f24c9600640 10 client.28561 handle_command_reply: tid=0 2024-03-28T22:45:33.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.478+0000 7f24e9459740 1 client.28561 shutdown 2024-03-28T22:45:33.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.478+0000 7f24e9459740 1 -- 172.21.15.43:0/4267838019 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559a35e31ee0 msgr2=0x559a35eaade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:33.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.478+0000 7f24e9459740 1 --2- 172.21.15.43:0/4267838019 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559a35e31ee0 0x559a35eaade0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f24d001dbc0 tx=0x7f24d001dec0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.479+0000 7f24e9459740 1 -- 172.21.15.43:0/4267838019 shutdown_connections 2024-03-28T22:45:33.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.479+0000 7f24e9459740 1 --2- 172.21.15.43:0/4267838019 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x559a35eb2a10 0x559a35eb4e30 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.479+0000 7f24e9459740 1 --2- 172.21.15.43:0/4267838019 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559a35e32bc0 0x559a35eab320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.479+0000 7f24e9459740 1 --2- 172.21.15.43:0/4267838019 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x559a35e3b250 0x559a35e3f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.479+0000 7f24e9459740 1 --2- 172.21.15.43:0/4267838019 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x559a35e31ee0 0x559a35eaade0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.479+0000 7f24e9459740 1 -- 172.21.15.43:0/4267838019 >> 172.21.15.43:0/4267838019 conn(0x559a35df8a10 msgr2=0x559a35e2b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:33.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.479+0000 7f24c9600640 0 client.28561 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:45:33.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.479+0000 7f24e9459740 1 -- 172.21.15.43:0/4267838019 shutdown_connections 2024-03-28T22:45:33.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.479+0000 7f24e9459740 1 -- 172.21.15.43:0/4267838019 wait complete. 2024-03-28T22:45:33.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.479+0000 7f24e9459740 20 client.28561 trim_cache size 0 max 16384 2024-03-28T22:45:33.486 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:45:33.486 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:45:33.486 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-03-28T22:45:33.486 INFO:teuthology.orchestra.run.smithi043.stdout: "3218a2a5-8467-4b27-9533-fc3b6c4540ac": { 2024-03-28T22:45:33.486 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-03-28T22:45:33.486 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3218a2a5-8467-4b27-9533-fc3b6c4540ac", 2024-03-28T22:45:33.486 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-03-28T22:45:33.486 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:45:33.487 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:45:33.487 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:45:33.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.479+0000 7f24e6800640 1 -- 172.21.15.43:0/2460899777 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f24c40a1f70 msgr2=0x7f24c40a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:33.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.479+0000 7f24e6800640 1 --2- 172.21.15.43:0/2460899777 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f24c40a1f70 0x7f24c40a4430 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x559a35e33110 tx=0x7f24d0005f90 comp rx=0 tx=0).stop 2024-03-28T22:45:33.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.479+0000 7f24e6800640 1 -- 172.21.15.43:0/2460899777 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24e00744a0 msgr2=0x7f24e019a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:45:33.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.479+0000 7f24e6800640 1 --2- 172.21.15.43:0/2460899777 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24e00744a0 0x7f24e019a730 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f24cc000c00 tx=0x7f24cc004360 comp rx=0 tx=0).stop 2024-03-28T22:45:33.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.480+0000 7f24e6800640 1 -- 172.21.15.43:0/2460899777 shutdown_connections 2024-03-28T22:45:33.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.480+0000 7f24e6800640 1 --2- 172.21.15.43:0/2460899777 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f24c40a1f70 0x7f24c40a4430 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.480+0000 7f24e6800640 1 --2- 172.21.15.43:0/2460899777 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24e00744a0 0x7f24e019a730 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.480+0000 7f24e6800640 1 --2- 172.21.15.43:0/2460899777 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24e010df20 0x7f24e019ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.480+0000 7f24e6800640 1 --2- 172.21.15.43:0/2460899777 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24e0073b80 0x7f24e019a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:45:33.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.480+0000 7f24e6800640 1 -- 172.21.15.43:0/2460899777 >> 172.21.15.43:0/2460899777 conn(0x7f24e0078b50 msgr2=0x7f24e00fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:45:33.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.480+0000 7f24e6800640 1 -- 172.21.15.43:0/2460899777 shutdown_connections 2024-03-28T22:45:33.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:45:33.480+0000 7f24e6800640 1 -- 172.21.15.43:0/2460899777 wait complete. 2024-03-28T22:45:33.495 INFO:tasks.cephfs.filesystem:scrub status for tag:3218a2a5-8467-4b27-9533-fc3b6c4540ac - {'path': '/', 'tag': '3218a2a5-8467-4b27-9533-fc3b6c4540ac', 'options': 'recursive,force'} 2024-03-28T22:45:34.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:33 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:45:34.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:33 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:45:34.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:33 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:45:34.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:33 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:45:34.202 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:33 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:45:34.202 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:33 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:45:34.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:33 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:45:34.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:33 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:45:34.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:33 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:45:35.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:35 smithi082 ceph-mon[36471]: scrub queued for path: / 2024-03-28T22:45:35.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:35 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:45:35.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:35 smithi082 ceph-mon[36471]: scrub summary: active paths [/] 2024-03-28T22:45:35.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:35 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:45:35.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:35 smithi082 ceph-mon[36471]: pgmap v261: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:45:35.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:35 smithi043 ceph-mon[31823]: scrub queued for path: / 2024-03-28T22:45:35.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:35 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:45:35.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:35 smithi043 ceph-mon[31823]: scrub summary: active paths [/] 2024-03-28T22:45:35.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:35 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:45:35.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:35 smithi043 ceph-mon[31823]: pgmap v261: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:45:35.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:35 smithi172 ceph-mon[36566]: scrub queued for path: / 2024-03-28T22:45:35.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:35 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:45:35.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:35 smithi172 ceph-mon[36566]: scrub summary: active paths [/] 2024-03-28T22:45:35.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:35 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:45:35.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:35 smithi172 ceph-mon[36566]: pgmap v261: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:45:37.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:37 smithi082 ceph-mon[36471]: pgmap v262: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:45:37.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:37 smithi043 ceph-mon[31823]: pgmap v262: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:45:37.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:37 smithi172 ceph-mon[36566]: pgmap v262: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:45:39.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:39 smithi082 ceph-mon[36471]: pgmap v263: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 19 op/s 2024-03-28T22:45:39.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:39 smithi082 ceph-mon[36471]: scrub summary: idle 2024-03-28T22:45:39.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:39 smithi043 ceph-mon[31823]: pgmap v263: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 19 op/s 2024-03-28T22:45:39.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:39 smithi043 ceph-mon[31823]: scrub summary: idle 2024-03-28T22:45:39.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:39 smithi172 ceph-mon[36566]: pgmap v263: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 19 op/s 2024-03-28T22:45:39.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:39 smithi172 ceph-mon[36566]: scrub summary: idle 2024-03-28T22:45:41.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:41 smithi082 ceph-mon[36471]: pgmap v264: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 15 op/s 2024-03-28T22:45:41.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:41 smithi043 ceph-mon[31823]: pgmap v264: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 15 op/s 2024-03-28T22:45:41.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:41 smithi172 ceph-mon[36566]: pgmap v264: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 15 op/s 2024-03-28T22:45:43.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:43 smithi082 ceph-mon[36471]: pgmap v265: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 15 op/s 2024-03-28T22:45:43.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:43 smithi043 ceph-mon[31823]: pgmap v265: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 15 op/s 2024-03-28T22:45:43.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:43 smithi172 ceph-mon[36566]: pgmap v265: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 15 op/s 2024-03-28T22:45:45.446 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:45 smithi172 ceph-mon[36566]: pgmap v266: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 19 op/s 2024-03-28T22:45:45.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:45 smithi082 ceph-mon[36471]: pgmap v266: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 19 op/s 2024-03-28T22:45:45.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:45 smithi043 ceph-mon[31823]: pgmap v266: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 19 op/s 2024-03-28T22:45:47.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:47 smithi082 ceph-mon[36471]: pgmap v267: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 15 op/s 2024-03-28T22:45:47.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:47 smithi043 ceph-mon[31823]: pgmap v267: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 15 op/s 2024-03-28T22:45:47.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:47 smithi172 ceph-mon[36566]: pgmap v267: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 15 op/s 2024-03-28T22:45:49.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:49 smithi082 ceph-mon[36471]: pgmap v268: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 19 op/s 2024-03-28T22:45:49.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:49 smithi043 ceph-mon[31823]: pgmap v268: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 19 op/s 2024-03-28T22:45:49.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:49 smithi172 ceph-mon[36566]: pgmap v268: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 19 op/s 2024-03-28T22:45:51.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:51 smithi043 ceph-mon[31823]: pgmap v269: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:45:51.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:51 smithi172 ceph-mon[36566]: pgmap v269: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:45:51.698 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:51 smithi082 ceph-mon[36471]: pgmap v269: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:45:53.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:53 smithi043 ceph-mon[31823]: pgmap v270: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:45:53.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:53 smithi172 ceph-mon[36566]: pgmap v270: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:45:53.698 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:53 smithi082 ceph-mon[36471]: pgmap v270: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:45:55.500 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:55 smithi082 ceph-mon[36471]: pgmap v271: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:45:55.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:55 smithi043 ceph-mon[31823]: pgmap v271: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:45:55.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:55 smithi172 ceph-mon[36566]: pgmap v271: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:45:57.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:57 smithi043 ceph-mon[31823]: pgmap v272: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:45:57.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:57 smithi172 ceph-mon[36566]: pgmap v272: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:45:57.698 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:57 smithi082 ceph-mon[36471]: pgmap v272: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:45:59.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:45:59 smithi043 ceph-mon[31823]: pgmap v273: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:45:59.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:45:59 smithi172 ceph-mon[36566]: pgmap v273: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:45:59.698 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:45:59 smithi082 ceph-mon[36471]: pgmap v273: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:46:01.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:01 smithi043 ceph-mon[31823]: pgmap v274: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:46:01.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:01 smithi172 ceph-mon[36566]: pgmap v274: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:46:01.698 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:01 smithi082 ceph-mon[36471]: pgmap v274: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:46:03.496 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:46:03.497 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:46:03.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:03 smithi043 ceph-mon[31823]: pgmap v275: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:46:03.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:03 smithi172 ceph-mon[36566]: pgmap v275: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:46:03.698 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:03 smithi082 ceph-mon[36471]: pgmap v275: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:46:03.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.739+0000 7f12bf000640 1 -- 172.21.15.43:0/224311821 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f12b80765a0 msgr2=0x7f12b8112050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:03.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.739+0000 7f12bf000640 1 --2- 172.21.15.43:0/224311821 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f12b80765a0 0x7f12b8112050 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f12a4009930 tx=0x7f12a402bc20 comp rx=0 tx=0).stop 2024-03-28T22:46:03.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.739+0000 7f12bf000640 1 -- 172.21.15.43:0/224311821 shutdown_connections 2024-03-28T22:46:03.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.739+0000 7f12bf000640 1 --2- 172.21.15.43:0/224311821 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f12b80765a0 0x7f12b8112050 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.739+0000 7f12bf000640 1 --2- 172.21.15.43:0/224311821 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12b8112590 0x7f12b8114980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.739+0000 7f12bf000640 1 --2- 172.21.15.43:0/224311821 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12b8075c80 0x7f12b8076060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.739+0000 7f12bf000640 1 -- 172.21.15.43:0/224311821 >> 172.21.15.43:0/224311821 conn(0x7f12b80ff3a0 msgr2=0x7f12b81017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:03.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.739+0000 7f12bf000640 1 -- 172.21.15.43:0/224311821 shutdown_connections 2024-03-28T22:46:03.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.739+0000 7f12bf000640 1 -- 172.21.15.43:0/224311821 wait complete. 2024-03-28T22:46:03.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.740+0000 7f12bf000640 1 Processor -- start 2024-03-28T22:46:03.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.740+0000 7f12bf000640 1 -- start start 2024-03-28T22:46:03.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.741+0000 7f12bf000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12b8075c80 0x7f12b81a4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:03.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.741+0000 7f12bf000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12b80765a0 0x7f12b81a4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:03.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.741+0000 7f12bf000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f12b8112590 0x7f12b819e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:03.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.741+0000 7f12bf000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f12b8118130 con 0x7f12b8075c80 2024-03-28T22:46:03.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.741+0000 7f12bf000640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f12b8117fb0 con 0x7f12b8112590 2024-03-28T22:46:03.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.741+0000 7f12bf000640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f12b81182b0 con 0x7f12b80765a0 2024-03-28T22:46:03.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.741+0000 7f12bd000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f12b8112590 0x7f12b819e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:03.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.741+0000 7f12b7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12b8075c80 0x7f12b81a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:03.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.741+0000 7f12b7400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12b80765a0 0x7f12b81a4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:03.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.741+0000 7f12bd000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f12b8112590 0x7f12b819e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:46360/0 (socket says 172.21.15.43:46360) 2024-03-28T22:46:03.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.741+0000 7f12bd000640 1 -- 172.21.15.43:0/2674106109 learned_addr learned my addr 172.21.15.43:0/2674106109 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:03.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.741+0000 7f12bd000640 1 -- 172.21.15.43:0/2674106109 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12b80765a0 msgr2=0x7f12b81a4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:03.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.741+0000 7f12bd000640 1 --2- 172.21.15.43:0/2674106109 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12b80765a0 0x7f12b81a4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.741+0000 7f12bd000640 1 -- 172.21.15.43:0/2674106109 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12b8075c80 msgr2=0x7f12b81a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:03.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.741+0000 7f12bd000640 1 --2- 172.21.15.43:0/2674106109 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12b8075c80 0x7f12b81a4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.741+0000 7f12bd000640 1 -- 172.21.15.43:0/2674106109 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12b819ed30 con 0x7f12b8112590 2024-03-28T22:46:03.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.742+0000 7f12bd000640 1 --2- 172.21.15.43:0/2674106109 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f12b8112590 0x7f12b819e5b0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f12b000d950 tx=0x7f12b000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:03.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.742+0000 7f12b4c00640 1 -- 172.21.15.43:0/2674106109 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b0017070 con 0x7f12b8112590 2024-03-28T22:46:03.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.742+0000 7f12b4c00640 1 -- 172.21.15.43:0/2674106109 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12b0009c30 con 0x7f12b8112590 2024-03-28T22:46:03.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.742+0000 7f12bf000640 1 -- 172.21.15.43:0/2674106109 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12b819f020 con 0x7f12b8112590 2024-03-28T22:46:03.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.742+0000 7f12b4c00640 1 -- 172.21.15.43:0/2674106109 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b0003d60 con 0x7f12b8112590 2024-03-28T22:46:03.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.742+0000 7f12bf000640 1 -- 172.21.15.43:0/2674106109 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12b81ab2c0 con 0x7f12b8112590 2024-03-28T22:46:03.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.745+0000 7f12c1c0a740 1 Processor -- start 2024-03-28T22:46:03.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.745+0000 7f12b4c00640 1 -- 172.21.15.43:0/2674106109 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f12b0003f00 con 0x7f12b8112590 2024-03-28T22:46:03.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.746+0000 7f12c1c0a740 1 -- start start 2024-03-28T22:46:03.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.746+0000 7f12c1c0a740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640a82ffbc0 0x5640a82fe910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:03.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.746+0000 7f12c1c0a740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640a82feee0 0x5640a82ff360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:03.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.746+0000 7f12c1c0a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640a8308250 0x5640a830a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:03.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.746+0000 7f12c1c0a740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5640a8263100 con 0x5640a82ffbc0 2024-03-28T22:46:03.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.746+0000 7f12c1c0a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5640a820bba0 con 0x5640a8308250 2024-03-28T22:46:03.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.746+0000 7f12c1c0a740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5640a825f1a0 con 0x5640a82feee0 2024-03-28T22:46:03.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.746+0000 7f12b7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640a82ffbc0 0x5640a82fe910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:03.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.746+0000 7f12b7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640a82ffbc0 0x5640a82fe910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56598/0 (socket says 172.21.15.43:56598) 2024-03-28T22:46:03.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.746+0000 7f12b7400640 1 -- 172.21.15.43:0/821189165 learned_addr learned my addr 172.21.15.43:0/821189165 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:03.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.746+0000 7f12b7e00640 1 --2- 172.21.15.43:0/821189165 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640a82feee0 0x5640a82ff360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:03.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.746+0000 7f12b7400640 1 --2- 172.21.15.43:0/821189165 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640a8308250 0x5640a830a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:03.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.747+0000 7f12b7e00640 1 -- 172.21.15.43:0/821189165 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640a8308250 msgr2=0x5640a830a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:03.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.747+0000 7f12b7e00640 1 --2- 172.21.15.43:0/821189165 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640a8308250 0x5640a830a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.747+0000 7f12b7e00640 1 -- 172.21.15.43:0/821189165 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640a82ffbc0 msgr2=0x5640a82fe910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:03.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.747+0000 7f12b7e00640 1 --2- 172.21.15.43:0/821189165 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640a82ffbc0 0x5640a82fe910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.747+0000 7f12b7e00640 1 -- 172.21.15.43:0/821189165 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640a823c1a0 con 0x5640a82feee0 2024-03-28T22:46:03.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.747+0000 7f12b4c00640 1 --2- 172.21.15.43:0/2674106109 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f129c0a1ef0 0x7f129c0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:03.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.747+0000 7f12b7e00640 1 --2- 172.21.15.43:0/821189165 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640a82feee0 0x5640a82ff360 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f12a8009fa0 tx=0x7f12a80118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1eb5276b93b2d08 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:03.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.747+0000 7f12b4c00640 1 -- 172.21.15.43:0/2674106109 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7f12b00cde30 con 0x7f12b8112590 2024-03-28T22:46:03.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.748+0000 7f12ad600640 1 -- 172.21.15.43:0/821189165 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a8003c30 con 0x5640a82feee0 2024-03-28T22:46:03.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.748+0000 7f12b7400640 1 --2- 172.21.15.43:0/2674106109 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f129c0a1ef0 0x7f129c0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:03.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.748+0000 7f12ad600640 1 -- 172.21.15.43:0/821189165 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12a8003dd0 con 0x5640a82feee0 2024-03-28T22:46:03.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.748+0000 7f12ad600640 1 -- 172.21.15.43:0/821189165 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a800fb20 con 0x5640a82feee0 2024-03-28T22:46:03.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.748+0000 7f12b7400640 1 --2- 172.21.15.43:0/2674106109 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f129c0a1ef0 0x7f129c0a43b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f12a4005ab0 tx=0x7f12a4005fd0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:03.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.748+0000 7f12c1c0a740 1 -- 172.21.15.43:0/821189165 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640a82feee0 msgr2=0x5640a82ff360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:03.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.748+0000 7f12c1c0a740 1 --2- 172.21.15.43:0/821189165 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640a82feee0 0x5640a82ff360 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f12a8009fa0 tx=0x7f12a80118d0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.748+0000 7f12c1c0a740 1 -- 172.21.15.43:0/821189165 shutdown_connections 2024-03-28T22:46:03.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.748+0000 7f12c1c0a740 1 --2- 172.21.15.43:0/821189165 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640a8308250 0x5640a830a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.748+0000 7f12c1c0a740 1 --2- 172.21.15.43:0/821189165 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640a82feee0 0x5640a82ff360 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.748+0000 7f12c1c0a740 1 --2- 172.21.15.43:0/821189165 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640a82ffbc0 0x5640a82fe910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.748+0000 7f12c1c0a740 1 -- 172.21.15.43:0/821189165 >> 172.21.15.43:0/821189165 conn(0x5640a82c5a10 msgr2=0x5640a82f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:03.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.749+0000 7f12c1c0a740 1 -- 172.21.15.43:0/821189165 shutdown_connections 2024-03-28T22:46:03.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.749+0000 7f12c1c0a740 1 -- 172.21.15.43:0/821189165 wait complete. 2024-03-28T22:46:03.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.749+0000 7f12c1c0a740 1 -- start start 2024-03-28T22:46:03.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.749+0000 7f12c1c0a740 1 Processor -- start 2024-03-28T22:46:03.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.750+0000 7f12c1c0a740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640a82feee0 0x5640a83051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:03.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.750+0000 7f12c1c0a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640a82ffbc0 0x5640a8305720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:03.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.750+0000 7f12c1c0a740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640a8308250 0x5640a8305c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:03.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.750+0000 7f12c1c0a740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5640a830b0c0 con 0x5640a8308250 2024-03-28T22:46:03.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.750+0000 7f12c1c0a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5640a8201250 con 0x5640a82ffbc0 2024-03-28T22:46:03.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.750+0000 7f12c1c0a740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5640a830b240 con 0x5640a82feee0 2024-03-28T22:46:03.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.750+0000 7f12c1c0a740 10 client.0 _ll_register_callbacks cb 0x5640a7f76be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:46:03.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.750+0000 7f12b7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640a82feee0 0x5640a83051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:03.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.750+0000 7f12b7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640a82feee0 0x5640a83051e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:33030/0 (socket says 172.21.15.43:33030) 2024-03-28T22:46:03.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.750+0000 7f12b7e00640 1 -- 172.21.15.43:0/148741959 learned_addr learned my addr 172.21.15.43:0/148741959 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:03.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.750+0000 7f12b7400640 1 --2- 172.21.15.43:0/148741959 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640a82ffbc0 0x5640a8305720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:03.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.750+0000 7f12bd000640 1 --2- 172.21.15.43:0/148741959 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640a8308250 0x5640a8305c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:03.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.750+0000 7f12b7e00640 1 -- 172.21.15.43:0/148741959 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640a82ffbc0 msgr2=0x5640a8305720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:03.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.750+0000 7f12b7e00640 1 --2- 172.21.15.43:0/148741959 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640a82ffbc0 0x5640a8305720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.750+0000 7f12b7e00640 1 -- 172.21.15.43:0/148741959 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640a8308250 msgr2=0x5640a8305c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:03.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.750+0000 7f12b7e00640 1 --2- 172.21.15.43:0/148741959 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640a8308250 0x5640a8305c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.750+0000 7f12b7e00640 1 -- 172.21.15.43:0/148741959 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640a821d960 con 0x5640a82feee0 2024-03-28T22:46:03.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.751+0000 7f12b7e00640 1 --2- 172.21.15.43:0/148741959 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640a82feee0 0x5640a83051e0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f12a8000f80 tx=0x7f12a8004c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c67e683e1009a22 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:03.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.751+0000 7f12a2000640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:46:03.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.751+0000 7f12a2000640 1 -- 172.21.15.43:0/148741959 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a8027070 con 0x5640a82feee0 2024-03-28T22:46:03.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.751+0000 7f12a2000640 1 -- 172.21.15.43:0/148741959 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7f12a800fc70 con 0x5640a82feee0 2024-03-28T22:46:03.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.751+0000 7f12c1c0a740 1 -- 172.21.15.43:0/148741959 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12a8003dd0 con 0x5640a82feee0 2024-03-28T22:46:03.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.751+0000 7f12a2000640 1 -- 172.21.15.43:0/148741959 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f12a8033050 con 0x5640a82feee0 2024-03-28T22:46:03.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.751+0000 7f12c1c0a740 10 client.28573 fetch_fsmap learned FSMap version 32 2024-03-28T22:46:03.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.751+0000 7f12c1c0a740 1 -- 172.21.15.43:0/148741959 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5640a80417d0 con 0x5640a82feee0 2024-03-28T22:46:03.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.752+0000 7f12a2000640 1 -- 172.21.15.43:0/148741959 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12a8003dd0 con 0x5640a82feee0 2024-03-28T22:46:03.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.752+0000 7f12a2000640 1 -- 172.21.15.43:0/148741959 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f12a8005040 con 0x5640a82feee0 2024-03-28T22:46:03.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.752+0000 7f12c1c0a740 10 client.28573 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:46:03.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.752+0000 7f12c1c0a740 10 client.28573 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:46:03.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.752+0000 7f12c1c0a740 20 client.28573 populate_metadata read hostname 'smithi043' 2024-03-28T22:46:03.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.752+0000 7f12c1c0a740 1 --2- 172.21.15.43:0/148741959 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5640a8318590 0x5640a831a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:03.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.752+0000 7f12c1c0a740 4 client.28573 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:46:03.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.752+0000 7f12c1c0a740 1 -- 172.21.15.43:0/148741959 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640a8270640 con 0x5640a8318590 2024-03-28T22:46:03.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.752+0000 7f12b7400640 1 --2- 172.21.15.43:0/148741959 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5640a8318590 0x5640a831a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:03.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.753+0000 7f12b7400640 1 --2- 172.21.15.43:0/148741959 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5640a8318590 0x5640a831a980 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bc8501fc8e95077 server_cookie=1c6f70563eb31b53 in_seq=0 out_seq=0 2024-03-28T22:46:03.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.753+0000 7f12a2000640 10 client.28573 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:46:03.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.753+0000 7f12a2000640 1 -- 172.21.15.43:0/148741959 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5640a8270640 con 0x5640a8318590 2024-03-28T22:46:03.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.753+0000 7f12a2000640 10 client.28573 handle_command_reply: tid=0 2024-03-28T22:46:03.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.753+0000 7f12c1c0a740 1 client.28573 shutdown 2024-03-28T22:46:03.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.754+0000 7f12c1c0a740 1 -- 172.21.15.43:0/148741959 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640a82feee0 msgr2=0x5640a83051e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:03.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.754+0000 7f12c1c0a740 1 --2- 172.21.15.43:0/148741959 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640a82feee0 0x5640a83051e0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f12a8000f80 tx=0x7f12a8004c70 comp rx=0 tx=0).stop 2024-03-28T22:46:03.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.754+0000 7f12c1c0a740 1 -- 172.21.15.43:0/148741959 shutdown_connections 2024-03-28T22:46:03.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.754+0000 7f12c1c0a740 1 --2- 172.21.15.43:0/148741959 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5640a8318590 0x5640a831a980 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.754+0000 7f12c1c0a740 1 --2- 172.21.15.43:0/148741959 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640a82ffbc0 0x5640a8305720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.754+0000 7f12c1c0a740 1 --2- 172.21.15.43:0/148741959 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640a8308250 0x5640a8305c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.754+0000 7f12a2000640 0 client.28573 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:46:03.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.754+0000 7f12c1c0a740 1 --2- 172.21.15.43:0/148741959 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640a82feee0 0x5640a83051e0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.754+0000 7f12c1c0a740 1 -- 172.21.15.43:0/148741959 >> 172.21.15.43:0/148741959 conn(0x5640a82c5a10 msgr2=0x5640a82fb680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:03.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.754+0000 7f12c1c0a740 1 -- 172.21.15.43:0/148741959 shutdown_connections 2024-03-28T22:46:03.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.754+0000 7f12c1c0a740 1 -- 172.21.15.43:0/148741959 wait complete. 2024-03-28T22:46:03.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.754+0000 7f12c1c0a740 20 client.28573 trim_cache size 0 max 16384 2024-03-28T22:46:03.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.773+0000 7f12c1c0a740 1 Processor -- start 2024-03-28T22:46:03.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.773+0000 7f12c1c0a740 1 -- start start 2024-03-28T22:46:03.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.774+0000 7f12c1c0a740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640a82feee0 0x5640a8313ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:03.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.774+0000 7f12c1c0a740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640a82ffbc0 0x5640a8304990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:03.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.774+0000 7f12c1c0a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640a8308250 0x5640a8320140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:03.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.774+0000 7f12c1c0a740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f12b81b5410 con 0x5640a82ffbc0 2024-03-28T22:46:03.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.774+0000 7f12c1c0a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f12b81b6f50 con 0x5640a8308250 2024-03-28T22:46:03.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.774+0000 7f12c1c0a740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f12b81b5ae0 con 0x5640a82feee0 2024-03-28T22:46:03.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.774+0000 7f12b7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640a82ffbc0 0x5640a8304990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:03.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.774+0000 7f12b7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640a82feee0 0x5640a8313ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:03.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.774+0000 7f12b7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640a82ffbc0 0x5640a8304990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56606/0 (socket says 172.21.15.43:56606) 2024-03-28T22:46:03.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.774+0000 7f12b7400640 1 -- 172.21.15.43:0/163399488 learned_addr learned my addr 172.21.15.43:0/163399488 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:03.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.774+0000 7f12bd000640 1 --2- 172.21.15.43:0/163399488 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640a8308250 0x5640a8320140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:03.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.774+0000 7f12b7400640 1 -- 172.21.15.43:0/163399488 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640a82feee0 msgr2=0x5640a8313ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:03.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.774+0000 7f12b7400640 1 --2- 172.21.15.43:0/163399488 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640a82feee0 0x5640a8313ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.774+0000 7f12b7400640 1 -- 172.21.15.43:0/163399488 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640a8308250 msgr2=0x5640a8320140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:03.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.774+0000 7f12b7400640 1 --2- 172.21.15.43:0/163399488 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640a8308250 0x5640a8320140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.774+0000 7f12b7400640 1 -- 172.21.15.43:0/163399488 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640a7f773f0 con 0x5640a82ffbc0 2024-03-28T22:46:03.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.774+0000 7f12bd000640 1 --2- 172.21.15.43:0/163399488 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640a8308250 0x5640a8320140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:46:03.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.775+0000 7f12b7400640 1 --2- 172.21.15.43:0/163399488 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640a82ffbc0 0x5640a8304990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f12a4009930 tx=0x7f12a4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aa56e6fbb564de5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:03.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.775+0000 7f12acc00640 1 -- 172.21.15.43:0/163399488 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a4058c90 con 0x5640a82ffbc0 2024-03-28T22:46:03.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.775+0000 7f12acc00640 1 -- 172.21.15.43:0/163399488 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12a4058810 con 0x5640a82ffbc0 2024-03-28T22:46:03.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.775+0000 7f12acc00640 1 -- 172.21.15.43:0/163399488 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a4004ee0 con 0x5640a82ffbc0 2024-03-28T22:46:03.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.776+0000 7f12c1c0a740 1 -- 172.21.15.43:0/163399488 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640a82ffbc0 msgr2=0x5640a8304990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:03.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.776+0000 7f12c1c0a740 1 --2- 172.21.15.43:0/163399488 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640a82ffbc0 0x5640a8304990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f12a4009930 tx=0x7f12a4009260 comp rx=0 tx=0).stop 2024-03-28T22:46:03.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.776+0000 7f12c1c0a740 1 -- 172.21.15.43:0/163399488 shutdown_connections 2024-03-28T22:46:03.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.776+0000 7f12c1c0a740 1 --2- 172.21.15.43:0/163399488 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640a8308250 0x5640a8320140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.776+0000 7f12c1c0a740 1 --2- 172.21.15.43:0/163399488 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640a82ffbc0 0x5640a8304990 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.776+0000 7f12c1c0a740 1 --2- 172.21.15.43:0/163399488 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640a82feee0 0x5640a8313ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.776+0000 7f12c1c0a740 1 -- 172.21.15.43:0/163399488 >> 172.21.15.43:0/163399488 conn(0x5640a82c5a10 msgr2=0x5640a82fb610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:03.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.776+0000 7f12c1c0a740 1 -- 172.21.15.43:0/163399488 shutdown_connections 2024-03-28T22:46:03.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.776+0000 7f12c1c0a740 1 -- 172.21.15.43:0/163399488 wait complete. 2024-03-28T22:46:03.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.776+0000 7f12c1c0a740 1 -- start start 2024-03-28T22:46:03.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.776+0000 7f12c1c0a740 1 Processor -- start 2024-03-28T22:46:03.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.777+0000 7f12c1c0a740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640a82feee0 0x5640a83040f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:03.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.777+0000 7f12c1c0a740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640a82ffbc0 0x5640a8304630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:03.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.777+0000 7f12c1c0a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640a8308250 0x5640a8304b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:03.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.777+0000 7f12c1c0a740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f12b81b6880 con 0x5640a82ffbc0 2024-03-28T22:46:03.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.777+0000 7f12c1c0a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f12b81b61b0 con 0x5640a8308250 2024-03-28T22:46:03.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.777+0000 7f12c1c0a740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f12b81b3410 con 0x5640a82feee0 2024-03-28T22:46:03.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.777+0000 7f12c1c0a740 10 client.0 _ll_register_callbacks cb 0x7f12a4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:46:03.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.777+0000 7f12bd000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640a8308250 0x5640a8304b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:03.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.777+0000 7f12b7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640a82feee0 0x5640a83040f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:03.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.777+0000 7f12bd000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640a8308250 0x5640a8304b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:46402/0 (socket says 172.21.15.43:46402) 2024-03-28T22:46:03.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.777+0000 7f12b7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640a82feee0 0x5640a83040f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:33052/0 (socket says 172.21.15.43:33052) 2024-03-28T22:46:03.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.777+0000 7f12bd000640 1 -- 172.21.15.43:0/473928695 learned_addr learned my addr 172.21.15.43:0/473928695 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:03.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.777+0000 7f12b7400640 1 --2- 172.21.15.43:0/473928695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640a82ffbc0 0x5640a8304630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:03.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.778+0000 7f12bd000640 1 -- 172.21.15.43:0/473928695 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640a82feee0 msgr2=0x5640a83040f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:03.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.778+0000 7f12bd000640 1 --2- 172.21.15.43:0/473928695 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640a82feee0 0x5640a83040f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.778+0000 7f12bd000640 1 -- 172.21.15.43:0/473928695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640a82ffbc0 msgr2=0x5640a8304630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:03.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.778+0000 7f12bd000640 1 --2- 172.21.15.43:0/473928695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640a82ffbc0 0x5640a8304630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.778+0000 7f12bd000640 1 -- 172.21.15.43:0/473928695 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640a7f370f0 con 0x5640a8308250 2024-03-28T22:46:03.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.778+0000 7f12b7e00640 1 --2- 172.21.15.43:0/473928695 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640a82feee0 0x5640a83040f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:03.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.778+0000 7f12bd000640 1 --2- 172.21.15.43:0/473928695 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640a8308250 0x5640a8304b70 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f12b0096e40 tx=0x7f12b009a070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b39b2d8e535287e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:03.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.778+0000 7f12a2000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:46:03.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.778+0000 7f12a2000640 1 -- 172.21.15.43:0/473928695 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b0020040 con 0x5640a8308250 2024-03-28T22:46:03.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.778+0000 7f12a2000640 1 -- 172.21.15.43:0/473928695 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7f12b009e170 con 0x5640a8308250 2024-03-28T22:46:03.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.778+0000 7f12c1c0a740 1 -- 172.21.15.43:0/473928695 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12a4058810 con 0x5640a8308250 2024-03-28T22:46:03.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.778+0000 7f12a2000640 1 -- 172.21.15.43:0/473928695 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f12b00a2b60 con 0x5640a8308250 2024-03-28T22:46:03.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.778+0000 7f12c1c0a740 10 client.28637 fetch_fsmap learned FSMap version 32 2024-03-28T22:46:03.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.778+0000 7f12c1c0a740 1 -- 172.21.15.43:0/473928695 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5640a8310620 con 0x5640a8308250 2024-03-28T22:46:03.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.779+0000 7f12a2000640 1 -- 172.21.15.43:0/473928695 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12a4058810 con 0x5640a8308250 2024-03-28T22:46:03.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.779+0000 7f12a2000640 1 -- 172.21.15.43:0/473928695 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f12b00b8b30 con 0x5640a8308250 2024-03-28T22:46:03.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.779+0000 7f12c1c0a740 10 client.28637 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:46:03.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.779+0000 7f12c1c0a740 10 client.28637 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:46:03.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.779+0000 7f12c1c0a740 20 client.28637 populate_metadata read hostname 'smithi043' 2024-03-28T22:46:03.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.779+0000 7f12c1c0a740 1 --2- 172.21.15.43:0/473928695 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5640a837e770 0x5640a8380b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:03.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.779+0000 7f12c1c0a740 4 client.28637 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:46:03.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.779+0000 7f12c1c0a740 1 -- 172.21.15.43:0/473928695 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5640a826a990 con 0x5640a837e770 2024-03-28T22:46:03.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.779+0000 7f12b7e00640 1 --2- 172.21.15.43:0/473928695 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5640a837e770 0x5640a8380b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:03.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.779+0000 7f12b7e00640 1 --2- 172.21.15.43:0/473928695 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5640a837e770 0x5640a8380b60 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9b1b6febe068071 server_cookie=bd9e0fb331bc3414 in_seq=0 out_seq=0 2024-03-28T22:46:03.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.779+0000 7f12a2000640 10 client.28637 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:46:03.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.780+0000 7f12a2000640 1 -- 172.21.15.43:0/473928695 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5640a826a990 con 0x5640a837e770 2024-03-28T22:46:03.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.780+0000 7f12a2000640 10 client.28637 handle_command_reply: tid=0 2024-03-28T22:46:03.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.780+0000 7f12c1c0a740 1 client.28637 shutdown 2024-03-28T22:46:03.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.780+0000 7f12c1c0a740 1 -- 172.21.15.43:0/473928695 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640a8308250 msgr2=0x5640a8304b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:03.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.780+0000 7f12c1c0a740 1 --2- 172.21.15.43:0/473928695 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640a8308250 0x5640a8304b70 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f12b0096e40 tx=0x7f12b009a070 comp rx=0 tx=0).stop 2024-03-28T22:46:03.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.780+0000 7f12c1c0a740 1 -- 172.21.15.43:0/473928695 shutdown_connections 2024-03-28T22:46:03.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.780+0000 7f12c1c0a740 1 --2- 172.21.15.43:0/473928695 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5640a837e770 0x5640a8380b60 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.780+0000 7f12c1c0a740 1 --2- 172.21.15.43:0/473928695 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5640a8308250 0x5640a8304b70 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.780+0000 7f12c1c0a740 1 --2- 172.21.15.43:0/473928695 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5640a82ffbc0 0x5640a8304630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.780+0000 7f12c1c0a740 1 --2- 172.21.15.43:0/473928695 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5640a82feee0 0x5640a83040f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.780+0000 7f12a2000640 0 client.28637 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:46:03.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.780+0000 7f12c1c0a740 1 -- 172.21.15.43:0/473928695 >> 172.21.15.43:0/473928695 conn(0x5640a82c5a10 msgr2=0x5640a82fac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:03.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.780+0000 7f12c1c0a740 1 -- 172.21.15.43:0/473928695 shutdown_connections 2024-03-28T22:46:03.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.780+0000 7f12c1c0a740 1 -- 172.21.15.43:0/473928695 wait complete. 2024-03-28T22:46:03.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.780+0000 7f12c1c0a740 20 client.28637 trim_cache size 0 max 16384 2024-03-28T22:46:03.788 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:46:03.788 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-03-28T22:46:03.788 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-03-28T22:46:03.788 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:46:03.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.781+0000 7f12bf000640 1 -- 172.21.15.43:0/2674106109 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f129c0a1ef0 msgr2=0x7f129c0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:03.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.781+0000 7f12bf000640 1 --2- 172.21.15.43:0/2674106109 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f129c0a1ef0 0x7f129c0a43b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f12a4005ab0 tx=0x7f12a4005fd0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.781+0000 7f12bf000640 1 -- 172.21.15.43:0/2674106109 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f12b8112590 msgr2=0x7f12b819e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:03.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.781+0000 7f12bf000640 1 --2- 172.21.15.43:0/2674106109 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f12b8112590 0x7f12b819e5b0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f12b000d950 tx=0x7f12b000de10 comp rx=0 tx=0).stop 2024-03-28T22:46:03.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.781+0000 7f12bf000640 1 -- 172.21.15.43:0/2674106109 shutdown_connections 2024-03-28T22:46:03.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.781+0000 7f12bf000640 1 --2- 172.21.15.43:0/2674106109 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f129c0a1ef0 0x7f129c0a43b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.781+0000 7f12bf000640 1 --2- 172.21.15.43:0/2674106109 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f12b8112590 0x7f12b819e5b0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.781+0000 7f12bf000640 1 --2- 172.21.15.43:0/2674106109 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f12b80765a0 0x7f12b81a4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.781+0000 7f12bf000640 1 --2- 172.21.15.43:0/2674106109 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f12b8075c80 0x7f12b81a4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:03.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.781+0000 7f12bf000640 1 -- 172.21.15.43:0/2674106109 >> 172.21.15.43:0/2674106109 conn(0x7f12b80ff3a0 msgr2=0x7f12b8101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:03.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.781+0000 7f12bf000640 1 -- 172.21.15.43:0/2674106109 shutdown_connections 2024-03-28T22:46:03.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:03.781+0000 7f12bf000640 1 -- 172.21.15.43:0/2674106109 wait complete. 2024-03-28T22:46:03.800 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:46:03.800 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:46:04.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.074+0000 7efc9ec00640 1 -- 172.21.15.43:0/2559914374 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc981125a0 msgr2=0x7efc98114990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:04.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.074+0000 7efc9ec00640 1 --2- 172.21.15.43:0/2559914374 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc981125a0 0x7efc98114990 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7efc90009cd0 tx=0x7efc9002c0e0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.075+0000 7efc9ec00640 1 -- 172.21.15.43:0/2559914374 shutdown_connections 2024-03-28T22:46:04.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.075+0000 7efc9ec00640 1 --2- 172.21.15.43:0/2559914374 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc981125a0 0x7efc98114990 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.075+0000 7efc9ec00640 1 --2- 172.21.15.43:0/2559914374 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc980765a0 0x7efc98112060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.075+0000 7efc9ec00640 1 --2- 172.21.15.43:0/2559914374 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efc98075c80 0x7efc98076060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.075+0000 7efc9ec00640 1 -- 172.21.15.43:0/2559914374 >> 172.21.15.43:0/2559914374 conn(0x7efc980ff390 msgr2=0x7efc981017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:04.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.075+0000 7efc9ec00640 1 -- 172.21.15.43:0/2559914374 shutdown_connections 2024-03-28T22:46:04.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.075+0000 7efc9ec00640 1 -- 172.21.15.43:0/2559914374 wait complete. 2024-03-28T22:46:04.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.076+0000 7efc9ec00640 1 Processor -- start 2024-03-28T22:46:04.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.076+0000 7efc9ec00640 1 -- start start 2024-03-28T22:46:04.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.076+0000 7efc9ec00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc98075c80 0x7efc9819e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.076+0000 7efc9ec00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efc980765a0 0x7efc9819ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.076+0000 7efc9ec00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc981125a0 0x7efc981a2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.076+0000 7efc9ec00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7efc981180c0 con 0x7efc981125a0 2024-03-28T22:46:04.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.076+0000 7efc9ec00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7efc98117f40 con 0x7efc980765a0 2024-03-28T22:46:04.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.076+0000 7efc9ec00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7efc98118240 con 0x7efc98075c80 2024-03-28T22:46:04.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.076+0000 7efc97e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc98075c80 0x7efc9819e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.077+0000 7efc97e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc98075c80 0x7efc9819e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:33070/0 (socket says 172.21.15.43:33070) 2024-03-28T22:46:04.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.077+0000 7efc97e00640 1 -- 172.21.15.43:0/402101151 learned_addr learned my addr 172.21.15.43:0/402101151 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:04.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.077+0000 7efc97400640 1 --2- 172.21.15.43:0/402101151 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efc980765a0 0x7efc9819ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.077+0000 7efc9cc00640 1 --2- 172.21.15.43:0/402101151 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc981125a0 0x7efc981a2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.077+0000 7efc97e00640 1 -- 172.21.15.43:0/402101151 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efc980765a0 msgr2=0x7efc9819ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:04.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.077+0000 7efc97e00640 1 --2- 172.21.15.43:0/402101151 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efc980765a0 0x7efc9819ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.077+0000 7efc97e00640 1 -- 172.21.15.43:0/402101151 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc981125a0 msgr2=0x7efc981a2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:04.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.077+0000 7efc97e00640 1 --2- 172.21.15.43:0/402101151 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc981125a0 0x7efc981a2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.077+0000 7efc97e00640 1 -- 172.21.15.43:0/402101151 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc981a36e0 con 0x7efc98075c80 2024-03-28T22:46:04.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.077+0000 7efc97e00640 1 --2- 172.21.15.43:0/402101151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc98075c80 0x7efc9819e6f0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7efc8800e9f0 tx=0x7efc8800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:04.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.078+0000 7efc94c00640 1 -- 172.21.15.43:0/402101151 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc8800cd80 con 0x7efc98075c80 2024-03-28T22:46:04.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.078+0000 7efc94c00640 1 -- 172.21.15.43:0/402101151 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc88010910 con 0x7efc98075c80 2024-03-28T22:46:04.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.078+0000 7efc9ec00640 1 -- 172.21.15.43:0/402101151 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc981a39d0 con 0x7efc98075c80 2024-03-28T22:46:04.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.078+0000 7efc94c00640 1 -- 172.21.15.43:0/402101151 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc88018990 con 0x7efc98075c80 2024-03-28T22:46:04.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.078+0000 7efc9ec00640 1 -- 172.21.15.43:0/402101151 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc981ab260 con 0x7efc98075c80 2024-03-28T22:46:04.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.079+0000 7efc9ec00640 1 -- 172.21.15.43:0/402101151 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc98068540 con 0x7efc98075c80 2024-03-28T22:46:04.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.081+0000 7efc94c00640 1 -- 172.21.15.43:0/402101151 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7efc88010ac0 con 0x7efc98075c80 2024-03-28T22:46:04.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.083+0000 7efc94c00640 1 --2- 172.21.15.43:0/402101151 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7efc7c0a1f70 0x7efc7c0a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.083+0000 7efc94c00640 1 -- 172.21.15.43:0/402101151 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7efc880cecc0 con 0x7efc98075c80 2024-03-28T22:46:04.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.083+0000 7efc97400640 1 --2- 172.21.15.43:0/402101151 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7efc7c0a1f70 0x7efc7c0a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.084+0000 7efc97400640 1 --2- 172.21.15.43:0/402101151 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7efc7c0a1f70 0x7efc7c0a4430 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7efc98101860 tx=0x7efc84005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:04.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.089+0000 7efc94c00640 1 -- 172.21.15.43:0/402101151 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7efc8809b300 con 0x7efc98075c80 2024-03-28T22:46:04.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.333+0000 7efc9ec00640 1 -- 172.21.15.43:0/402101151 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efc9819fa40 con 0x7efc98075c80 2024-03-28T22:46:04.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.334+0000 7efc94c00640 1 -- 172.21.15.43:0/402101151 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11931 (secure 0 0 0) 0x7efc8809f3a0 con 0x7efc98075c80 2024-03-28T22:46:04.333 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:46:04.333 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:33:28.906396+0000","modified":"2024-03-28T22:40:08.672168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24572,"mds_1":24556,"mds_2":24535,"mds_3":24523,"mds_4":24518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26717":{"gid":26717,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/215779342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":215779342},{"type":"v1","addr":"172.21.15.82:6835","nonce":215779342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24535,24556,24518,24523,24572]},"id":1}]} 2024-03-28T22:46:04.333 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 32 2024-03-28T22:46:04.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.338+0000 7efc9ec00640 1 -- 172.21.15.43:0/402101151 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7efc7c0a1f70 msgr2=0x7efc7c0a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:04.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.338+0000 7efc9ec00640 1 --2- 172.21.15.43:0/402101151 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7efc7c0a1f70 0x7efc7c0a4430 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7efc98101860 tx=0x7efc84005f90 comp rx=0 tx=0).stop 2024-03-28T22:46:04.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.338+0000 7efc9ec00640 1 -- 172.21.15.43:0/402101151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc98075c80 msgr2=0x7efc9819e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:04.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.338+0000 7efc9ec00640 1 --2- 172.21.15.43:0/402101151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc98075c80 0x7efc9819e6f0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7efc8800e9f0 tx=0x7efc8800eeb0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.338+0000 7efc9ec00640 1 -- 172.21.15.43:0/402101151 shutdown_connections 2024-03-28T22:46:04.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.338+0000 7efc9ec00640 1 --2- 172.21.15.43:0/402101151 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7efc7c0a1f70 0x7efc7c0a4430 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.338+0000 7efc9ec00640 1 --2- 172.21.15.43:0/402101151 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efc980765a0 0x7efc9819ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.338+0000 7efc9ec00640 1 --2- 172.21.15.43:0/402101151 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc981125a0 0x7efc981a2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.338+0000 7efc9ec00640 1 --2- 172.21.15.43:0/402101151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7efc98075c80 0x7efc9819e6f0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.338+0000 7efc9ec00640 1 -- 172.21.15.43:0/402101151 >> 172.21.15.43:0/402101151 conn(0x7efc980ff390 msgr2=0x7efc98113d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:04.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.338+0000 7efc9ec00640 1 -- 172.21.15.43:0/402101151 shutdown_connections 2024-03-28T22:46:04.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.338+0000 7efc9ec00640 1 -- 172.21.15.43:0/402101151 wait complete. 2024-03-28T22:46:04.357 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T22:46:04.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.579+0000 7f203b200640 1 -- 172.21.15.43:0/4290078993 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2034100850 msgr2=0x7f2034111e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:04.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.579+0000 7f203b200640 1 --2- 172.21.15.43:0/4290078993 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2034100850 0x7f2034111e80 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f2024009900 tx=0x7f202402bcd0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.579+0000 7f203b200640 1 -- 172.21.15.43:0/4290078993 shutdown_connections 2024-03-28T22:46:04.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.579+0000 7f203b200640 1 --2- 172.21.15.43:0/4290078993 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2034100850 0x7f2034111e80 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.579+0000 7f203b200640 1 --2- 172.21.15.43:0/4290078993 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f20341123c0 0x7f20341147b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.579+0000 7f203b200640 1 --2- 172.21.15.43:0/4290078993 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20340fff30 0x7f2034100310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.579+0000 7f203b200640 1 -- 172.21.15.43:0/4290078993 >> 172.21.15.43:0/4290078993 conn(0x7f20340757c0 msgr2=0x7f2034075bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:04.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.579+0000 7f203b200640 1 -- 172.21.15.43:0/4290078993 shutdown_connections 2024-03-28T22:46:04.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.579+0000 7f203b200640 1 -- 172.21.15.43:0/4290078993 wait complete. 2024-03-28T22:46:04.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.580+0000 7f203b200640 1 Processor -- start 2024-03-28T22:46:04.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.580+0000 7f203b200640 1 -- start start 2024-03-28T22:46:04.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.581+0000 7f203b200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f20340fff30 0x7f203419e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.581+0000 7f203b200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2034100850 0x7f203419ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.581+0000 7f203b200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20341123c0 0x7f20341a2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.581+0000 7f203b200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2034118010 con 0x7f2034100850 2024-03-28T22:46:04.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.581+0000 7f203b200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f2034117e90 con 0x7f20340fff30 2024-03-28T22:46:04.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.581+0000 7f203b200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f2034118190 con 0x7f20341123c0 2024-03-28T22:46:04.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.581+0000 7f2039e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f20340fff30 0x7f203419e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.581+0000 7f203a800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20341123c0 0x7f20341a2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.581+0000 7f2039e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f20340fff30 0x7f203419e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:46438/0 (socket says 172.21.15.43:46438) 2024-03-28T22:46:04.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.581+0000 7f2039e00640 1 -- 172.21.15.43:0/984989431 learned_addr learned my addr 172.21.15.43:0/984989431 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:04.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.581+0000 7f2039400640 1 --2- 172.21.15.43:0/984989431 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2034100850 0x7f203419ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.581+0000 7f2039e00640 1 -- 172.21.15.43:0/984989431 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20341123c0 msgr2=0x7f20341a2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:04.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.581+0000 7f2039e00640 1 --2- 172.21.15.43:0/984989431 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20341123c0 0x7f20341a2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.581+0000 7f2039e00640 1 -- 172.21.15.43:0/984989431 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2034100850 msgr2=0x7f203419ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:04.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.581+0000 7f2039e00640 1 --2- 172.21.15.43:0/984989431 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2034100850 0x7f203419ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.581+0000 7f2039e00640 1 -- 172.21.15.43:0/984989431 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20341a3500 con 0x7f20340fff30 2024-03-28T22:46:04.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.581+0000 7f203a800640 1 --2- 172.21.15.43:0/984989431 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20341123c0 0x7f20341a2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:04.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.582+0000 7f2039e00640 1 --2- 172.21.15.43:0/984989431 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f20340fff30 0x7f203419e510 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f202800e9c0 tx=0x7f202800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:04.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.582+0000 7f202ea00640 1 -- 172.21.15.43:0/984989431 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f202800cd80 con 0x7f20340fff30 2024-03-28T22:46:04.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.582+0000 7f202ea00640 1 -- 172.21.15.43:0/984989431 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20280035f0 con 0x7f20340fff30 2024-03-28T22:46:04.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.582+0000 7f202ea00640 1 -- 172.21.15.43:0/984989431 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2028010690 con 0x7f20340fff30 2024-03-28T22:46:04.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.582+0000 7f203b200640 1 -- 172.21.15.43:0/984989431 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20341a37f0 con 0x7f20340fff30 2024-03-28T22:46:04.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.582+0000 7f203b200640 1 -- 172.21.15.43:0/984989431 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20341ab080 con 0x7f20340fff30 2024-03-28T22:46:04.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.585+0000 7f203f618740 1 Processor -- start 2024-03-28T22:46:04.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.585+0000 7f202ea00640 1 -- 172.21.15.43:0/984989431 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f20280037a0 con 0x7f20340fff30 2024-03-28T22:46:04.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.585+0000 7f203f618740 1 -- start start 2024-03-28T22:46:04.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.586+0000 7f203f618740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccb9b18550 0x55ccb9b18930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.586+0000 7f203f618740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb9b18e70 0x55ccb9b21510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.586+0000 7f203f618740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccb9b21a50 0x55ccb9b23e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.586+0000 7f203f618740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ccb9a7c050 con 0x55ccb9b18e70 2024-03-28T22:46:04.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.586+0000 7f203f618740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55ccb9a24ce0 con 0x55ccb9b21a50 2024-03-28T22:46:04.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.586+0000 7f203f618740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55ccb9a780f0 con 0x55ccb9b18550 2024-03-28T22:46:04.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.586+0000 7f203a800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccb9b18550 0x55ccb9b18930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.586+0000 7f2039400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccb9b21a50 0x55ccb9b23e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.586+0000 7f203a800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccb9b18550 0x55ccb9b18930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:33088/0 (socket says 172.21.15.43:33088) 2024-03-28T22:46:04.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.586+0000 7f203a800640 1 -- 172.21.15.43:0/3604273899 learned_addr learned my addr 172.21.15.43:0/3604273899 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:04.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.586+0000 7f2039e00640 1 --2- 172.21.15.43:0/3604273899 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb9b18e70 0x55ccb9b21510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.586+0000 7f2039400640 1 -- 172.21.15.43:0/3604273899 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccb9b18550 msgr2=0x55ccb9b18930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:04.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.587+0000 7f2039400640 1 --2- 172.21.15.43:0/3604273899 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccb9b18550 0x55ccb9b18930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.587+0000 7f2039400640 1 -- 172.21.15.43:0/3604273899 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb9b18e70 msgr2=0x55ccb9b21510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:04.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.587+0000 7f2039400640 1 --2- 172.21.15.43:0/3604273899 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb9b18e70 0x55ccb9b21510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.587+0000 7f2039400640 1 -- 172.21.15.43:0/3604273899 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ccb97903f0 con 0x55ccb9b21a50 2024-03-28T22:46:04.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.587+0000 7f2039400640 1 --2- 172.21.15.43:0/3604273899 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccb9b21a50 0x55ccb9b23e40 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f202402c210 tx=0x7f20240095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=460f22937f12c799 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:04.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.587+0000 7f202ea00640 1 --2- 172.21.15.43:0/984989431 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f201c0a1f70 0x7f201c0a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.587+0000 7f2023000640 1 -- 172.21.15.43:0/3604273899 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2024043070 con 0x55ccb9b21a50 2024-03-28T22:46:04.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.587+0000 7f2023000640 1 -- 172.21.15.43:0/3604273899 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f202402c920 con 0x55ccb9b21a50 2024-03-28T22:46:04.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.587+0000 7f2023000640 1 -- 172.21.15.43:0/3604273899 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2024004ae0 con 0x55ccb9b21a50 2024-03-28T22:46:04.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.587+0000 7f202ea00640 1 -- 172.21.15.43:0/984989431 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7f20280ce8f0 con 0x7f20340fff30 2024-03-28T22:46:04.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.587+0000 7f203a800640 1 --2- 172.21.15.43:0/984989431 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f201c0a1f70 0x7f201c0a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.588+0000 7f203f618740 1 -- 172.21.15.43:0/3604273899 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccb9b21a50 msgr2=0x55ccb9b23e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:04.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.588+0000 7f203f618740 1 --2- 172.21.15.43:0/3604273899 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccb9b21a50 0x55ccb9b23e40 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f202402c210 tx=0x7f20240095e0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.588+0000 7f203f618740 1 -- 172.21.15.43:0/3604273899 shutdown_connections 2024-03-28T22:46:04.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.588+0000 7f203f618740 1 --2- 172.21.15.43:0/3604273899 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccb9b21a50 0x55ccb9b23e40 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.588+0000 7f203f618740 1 --2- 172.21.15.43:0/3604273899 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb9b18e70 0x55ccb9b21510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.588+0000 7f203f618740 1 --2- 172.21.15.43:0/3604273899 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccb9b18550 0x55ccb9b18930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.588+0000 7f203f618740 1 -- 172.21.15.43:0/3604273899 >> 172.21.15.43:0/3604273899 conn(0x55ccb9b12c80 msgr2=0x55ccb9b13060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:04.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.588+0000 7f203a800640 1 --2- 172.21.15.43:0/984989431 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f201c0a1f70 0x7f201c0a4430 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2030011910 tx=0x7f20300096a0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:04.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.588+0000 7f203f618740 1 -- 172.21.15.43:0/3604273899 shutdown_connections 2024-03-28T22:46:04.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.588+0000 7f203f618740 1 -- 172.21.15.43:0/3604273899 wait complete. 2024-03-28T22:46:04.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.589+0000 7f203f618740 1 -- start start 2024-03-28T22:46:04.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.589+0000 7f203f618740 1 Processor -- start 2024-03-28T22:46:04.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.589+0000 7f203f618740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccb9b18550 0x55ccb9b1ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.589+0000 7f203f618740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccb9b18e70 0x55ccb9b1efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.589+0000 7f203f618740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb9b21a50 0x55ccb9b1f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.589+0000 7f203f618740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ccb9b247e0 con 0x55ccb9b21a50 2024-03-28T22:46:04.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.589+0000 7f203f618740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55ccb9a1a2f0 con 0x55ccb9b18e70 2024-03-28T22:46:04.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.589+0000 7f203f618740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55ccb9b24960 con 0x55ccb9b18550 2024-03-28T22:46:04.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.590+0000 7f2039e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccb9b18550 0x55ccb9b1ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.590+0000 7f2039e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccb9b18550 0x55ccb9b1ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:33092/0 (socket says 172.21.15.43:33092) 2024-03-28T22:46:04.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.590+0000 7f203f618740 10 client.0 _ll_register_callbacks cb 0x55ccb978fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:46:04.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.590+0000 7f2039e00640 1 -- 172.21.15.43:0/476382749 learned_addr learned my addr 172.21.15.43:0/476382749 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:04.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.590+0000 7f2039400640 1 --2- 172.21.15.43:0/476382749 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccb9b18e70 0x55ccb9b1efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.590+0000 7f203a800640 1 --2- 172.21.15.43:0/476382749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb9b21a50 0x55ccb9b1f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.590+0000 7f2039e00640 1 -- 172.21.15.43:0/476382749 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccb9b18e70 msgr2=0x55ccb9b1efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:04.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.590+0000 7f2039e00640 1 --2- 172.21.15.43:0/476382749 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccb9b18e70 0x55ccb9b1efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.590+0000 7f2039e00640 1 -- 172.21.15.43:0/476382749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb9b21a50 msgr2=0x55ccb9b1f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:04.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.590+0000 7f2039e00640 1 --2- 172.21.15.43:0/476382749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb9b21a50 0x55ccb9b1f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.590+0000 7f2039e00640 1 -- 172.21.15.43:0/476382749 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ccb985a7d0 con 0x55ccb9b18550 2024-03-28T22:46:04.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.590+0000 7f2039400640 1 --2- 172.21.15.43:0/476382749 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccb9b18e70 0x55ccb9b1efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:04.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.590+0000 7f2039e00640 1 --2- 172.21.15.43:0/476382749 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccb9b18550 0x55ccb9b1ea80 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f202809ab80 tx=0x7f202809acd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2032b7cb42b8b34 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:04.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.590+0000 7f2017400640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:46:04.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.591+0000 7f2017400640 1 -- 172.21.15.43:0/476382749 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20280cc060 con 0x55ccb9b18550 2024-03-28T22:46:04.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.591+0000 7f2017400640 1 -- 172.21.15.43:0/476382749 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7f20280a3190 con 0x55ccb9b18550 2024-03-28T22:46:04.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.591+0000 7f203f618740 1 -- 172.21.15.43:0/476382749 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f202402c920 con 0x55ccb9b18550 2024-03-28T22:46:04.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.591+0000 7f2017400640 1 -- 172.21.15.43:0/476382749 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f202809b580 con 0x55ccb9b18550 2024-03-28T22:46:04.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.591+0000 7f203f618740 10 client.28597 fetch_fsmap learned FSMap version 32 2024-03-28T22:46:04.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.591+0000 7f203f618740 1 -- 172.21.15.43:0/476382749 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ccb9b30ae0 con 0x55ccb9b18550 2024-03-28T22:46:04.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.591+0000 7f2017400640 1 -- 172.21.15.43:0/476382749 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f202402c920 con 0x55ccb9b18550 2024-03-28T22:46:04.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.591+0000 7f2017400640 1 -- 172.21.15.43:0/476382749 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f20280cece0 con 0x55ccb9b18550 2024-03-28T22:46:04.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.591+0000 7f203f618740 10 client.28597 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:46:04.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.591+0000 7f203f618740 10 client.28597 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:46:04.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.591+0000 7f203f618740 20 client.28597 populate_metadata read hostname 'smithi043' 2024-03-28T22:46:04.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.591+0000 7f203f618740 1 --2- 172.21.15.43:0/476382749 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55ccb9b31ef0 0x55ccb9b34360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.591+0000 7f203f618740 4 client.28597 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:46:04.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.591+0000 7f203f618740 1 -- 172.21.15.43:0/476382749 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ccb9a25680 con 0x55ccb9b31ef0 2024-03-28T22:46:04.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.592+0000 7f2039400640 1 --2- 172.21.15.43:0/476382749 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55ccb9b31ef0 0x55ccb9b34360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.592+0000 7f2039400640 1 --2- 172.21.15.43:0/476382749 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55ccb9b31ef0 0x55ccb9b34360 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=981e63cd05dd27d6 server_cookie=5bc65802065fdb50 in_seq=0 out_seq=0 2024-03-28T22:46:04.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.592+0000 7f2017400640 10 client.28597 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:46:04.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.593+0000 7f2017400640 1 -- 172.21.15.43:0/476382749 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ccb9a25680 con 0x55ccb9b31ef0 2024-03-28T22:46:04.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.593+0000 7f2017400640 10 client.28597 handle_command_reply: tid=0 2024-03-28T22:46:04.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.593+0000 7f203f618740 1 client.28597 shutdown 2024-03-28T22:46:04.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.593+0000 7f203f618740 1 -- 172.21.15.43:0/476382749 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccb9b18550 msgr2=0x55ccb9b1ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:04.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.593+0000 7f203f618740 1 --2- 172.21.15.43:0/476382749 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccb9b18550 0x55ccb9b1ea80 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f202809ab80 tx=0x7f202809acd0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.594+0000 7f203f618740 1 -- 172.21.15.43:0/476382749 shutdown_connections 2024-03-28T22:46:04.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.594+0000 7f203f618740 1 --2- 172.21.15.43:0/476382749 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55ccb9b31ef0 0x55ccb9b34360 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.594+0000 7f203f618740 1 --2- 172.21.15.43:0/476382749 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccb9b18e70 0x55ccb9b1efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.594+0000 7f203f618740 1 --2- 172.21.15.43:0/476382749 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb9b21a50 0x55ccb9b1f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.594+0000 7f203f618740 1 --2- 172.21.15.43:0/476382749 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccb9b18550 0x55ccb9b1ea80 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.594+0000 7f203f618740 1 -- 172.21.15.43:0/476382749 >> 172.21.15.43:0/476382749 conn(0x55ccb9b12c80 msgr2=0x55ccb9b149d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:04.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.594+0000 7f2017400640 0 client.28597 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:46:04.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.594+0000 7f203f618740 1 -- 172.21.15.43:0/476382749 shutdown_connections 2024-03-28T22:46:04.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.594+0000 7f203f618740 1 -- 172.21.15.43:0/476382749 wait complete. 2024-03-28T22:46:04.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.594+0000 7f203f618740 20 client.28597 trim_cache size 0 max 16384 2024-03-28T22:46:04.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.608+0000 7f203f618740 1 Processor -- start 2024-03-28T22:46:04.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.608+0000 7f203f618740 1 -- start start 2024-03-28T22:46:04.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.609+0000 7f203f618740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccb9b18e70 0x55ccb9b2d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.609+0000 7f203f618740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccb9b21a50 0x55ccb9b1f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.609+0000 7f203f618740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb9b11c40 0x55ccb9b39c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.609+0000 7f203f618740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f20341b5040 con 0x55ccb9b11c40 2024-03-28T22:46:04.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.609+0000 7f203f618740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f20341b6b80 con 0x55ccb9b21a50 2024-03-28T22:46:04.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.609+0000 7f203f618740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f20341b5710 con 0x55ccb9b18e70 2024-03-28T22:46:04.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.609+0000 7f2039e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccb9b18e70 0x55ccb9b2d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.609+0000 7f2039400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccb9b21a50 0x55ccb9b1f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.609+0000 7f203a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb9b11c40 0x55ccb9b39c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.609+0000 7f2039400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccb9b21a50 0x55ccb9b1f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:46452/0 (socket says 172.21.15.43:46452) 2024-03-28T22:46:04.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.609+0000 7f2039e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccb9b18e70 0x55ccb9b2d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:33102/0 (socket says 172.21.15.43:33102) 2024-03-28T22:46:04.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.609+0000 7f2039400640 1 -- 172.21.15.43:0/1294455009 learned_addr learned my addr 172.21.15.43:0/1294455009 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:04.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.609+0000 7f2039400640 1 -- 172.21.15.43:0/1294455009 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccb9b18e70 msgr2=0x55ccb9b2d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:04.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.609+0000 7f2039400640 1 --2- 172.21.15.43:0/1294455009 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccb9b18e70 0x55ccb9b2d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.609+0000 7f2039400640 1 -- 172.21.15.43:0/1294455009 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb9b11c40 msgr2=0x55ccb9b39c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:04.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.609+0000 7f2039400640 1 --2- 172.21.15.43:0/1294455009 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb9b11c40 0x55ccb9b39c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.609+0000 7f2039400640 1 -- 172.21.15.43:0/1294455009 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ccb9b12ab0 con 0x55ccb9b21a50 2024-03-28T22:46:04.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.610+0000 7f2039400640 1 --2- 172.21.15.43:0/1294455009 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccb9b21a50 0x55ccb9b1f350 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f202402c2e0 tx=0x7f202402c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3eef532628bb807c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:04.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.610+0000 7f2022600640 1 -- 172.21.15.43:0/1294455009 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2024043040 con 0x55ccb9b21a50 2024-03-28T22:46:04.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.610+0000 7f2022600640 1 -- 172.21.15.43:0/1294455009 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2024055070 con 0x55ccb9b21a50 2024-03-28T22:46:04.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.610+0000 7f2022600640 1 -- 172.21.15.43:0/1294455009 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20240043a0 con 0x55ccb9b21a50 2024-03-28T22:46:04.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.611+0000 7f203f618740 1 -- 172.21.15.43:0/1294455009 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccb9b21a50 msgr2=0x55ccb9b1f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:04.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.611+0000 7f203f618740 1 --2- 172.21.15.43:0/1294455009 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccb9b21a50 0x55ccb9b1f350 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f202402c2e0 tx=0x7f202402c810 comp rx=0 tx=0).stop 2024-03-28T22:46:04.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.611+0000 7f203f618740 1 -- 172.21.15.43:0/1294455009 shutdown_connections 2024-03-28T22:46:04.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.611+0000 7f203f618740 1 --2- 172.21.15.43:0/1294455009 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccb9b21a50 0x55ccb9b1f350 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.611+0000 7f203f618740 1 --2- 172.21.15.43:0/1294455009 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb9b11c40 0x55ccb9b39c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.611+0000 7f203f618740 1 --2- 172.21.15.43:0/1294455009 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccb9b18e70 0x55ccb9b2d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.611+0000 7f203f618740 1 -- 172.21.15.43:0/1294455009 >> 172.21.15.43:0/1294455009 conn(0x55ccb9b12c80 msgr2=0x55ccb9b11530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:04.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.611+0000 7f203f618740 1 -- 172.21.15.43:0/1294455009 shutdown_connections 2024-03-28T22:46:04.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.611+0000 7f203f618740 1 -- 172.21.15.43:0/1294455009 wait complete. 2024-03-28T22:46:04.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.611+0000 7f203f618740 1 -- start start 2024-03-28T22:46:04.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.611+0000 7f203f618740 1 Processor -- start 2024-03-28T22:46:04.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.612+0000 7f203f618740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccb9b11c40 0x55ccb9b25800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.612+0000 7f203f618740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb9b18e70 0x55ccb9b25d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.612+0000 7f203f618740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccb9b21a50 0x55ccb9b97230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.612+0000 7f203f618740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f20341b64b0 con 0x55ccb9b18e70 2024-03-28T22:46:04.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.612+0000 7f203f618740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f20341b5de0 con 0x55ccb9b11c40 2024-03-28T22:46:04.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.612+0000 7f203f618740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f20341b3040 con 0x55ccb9b21a50 2024-03-28T22:46:04.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.612+0000 7f203f618740 10 client.0 _ll_register_callbacks cb 0x7f2028003e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:46:04.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.612+0000 7f2039400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb9b18e70 0x55ccb9b25d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.612+0000 7f2039e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccb9b11c40 0x55ccb9b25800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.612+0000 7f203a800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccb9b21a50 0x55ccb9b97230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.612+0000 7f2039e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccb9b11c40 0x55ccb9b25800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:46460/0 (socket says 172.21.15.43:46460) 2024-03-28T22:46:04.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.612+0000 7f2039e00640 1 -- 172.21.15.43:0/734378721 learned_addr learned my addr 172.21.15.43:0/734378721 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:04.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.613+0000 7f2039400640 1 -- 172.21.15.43:0/734378721 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccb9b21a50 msgr2=0x55ccb9b97230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:04.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.613+0000 7f2039400640 1 --2- 172.21.15.43:0/734378721 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccb9b21a50 0x55ccb9b97230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.613+0000 7f2039400640 1 -- 172.21.15.43:0/734378721 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccb9b11c40 msgr2=0x55ccb9b25800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:04.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.613+0000 7f2039400640 1 --2- 172.21.15.43:0/734378721 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccb9b11c40 0x55ccb9b25800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.613+0000 7f2039400640 1 -- 172.21.15.43:0/734378721 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ccb9b23e40 con 0x55ccb9b18e70 2024-03-28T22:46:04.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.613+0000 7f2039400640 1 --2- 172.21.15.43:0/734378721 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb9b18e70 0x55ccb9b25d40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x55ccb9b0e840 tx=0x7f202402bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed6fcd9cd88c8c12 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:04.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.613+0000 7f2017400640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:46:04.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.613+0000 7f2017400640 1 -- 172.21.15.43:0/734378721 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f202405e060 con 0x55ccb9b18e70 2024-03-28T22:46:04.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.613+0000 7f2017400640 1 -- 172.21.15.43:0/734378721 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7f2024004120 con 0x55ccb9b18e70 2024-03-28T22:46:04.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.613+0000 7f203f618740 1 -- 172.21.15.43:0/734378721 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2024055070 con 0x55ccb9b18e70 2024-03-28T22:46:04.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.613+0000 7f2017400640 1 -- 172.21.15.43:0/734378721 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2024043040 con 0x55ccb9b18e70 2024-03-28T22:46:04.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.614+0000 7f203f618740 10 client.18402 fetch_fsmap learned FSMap version 32 2024-03-28T22:46:04.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.614+0000 7f203f618740 1 -- 172.21.15.43:0/734378721 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ccb9b986a0 con 0x55ccb9b18e70 2024-03-28T22:46:04.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.614+0000 7f2017400640 1 -- 172.21.15.43:0/734378721 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2024055070 con 0x55ccb9b18e70 2024-03-28T22:46:04.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.614+0000 7f2017400640 1 -- 172.21.15.43:0/734378721 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f20240035f0 con 0x55ccb9b18e70 2024-03-28T22:46:04.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.614+0000 7f203f618740 10 client.18402 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:46:04.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.614+0000 7f203f618740 10 client.18402 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:46:04.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.614+0000 7f203f618740 20 client.18402 populate_metadata read hostname 'smithi043' 2024-03-28T22:46:04.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.614+0000 7f203f618740 1 --2- 172.21.15.43:0/734378721 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55ccb9b989a0 0x55ccb9b9adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.614+0000 7f203f618740 4 client.18402 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:46:04.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.614+0000 7f203f618740 1 -- 172.21.15.43:0/734378721 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ccb9a89590 con 0x55ccb9b989a0 2024-03-28T22:46:04.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.614+0000 7f2039e00640 1 --2- 172.21.15.43:0/734378721 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55ccb9b989a0 0x55ccb9b9adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.615+0000 7f2039e00640 1 --2- 172.21.15.43:0/734378721 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55ccb9b989a0 0x55ccb9b9adc0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=22d548b0b5d9fb8c server_cookie=fd4c924366fea7f8 in_seq=0 out_seq=0 2024-03-28T22:46:04.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.615+0000 7f2017400640 10 client.18402 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:46:04.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.615+0000 7f2017400640 1 -- 172.21.15.43:0/734378721 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ccb9a89590 con 0x55ccb9b989a0 2024-03-28T22:46:04.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.615+0000 7f2017400640 10 client.18402 handle_command_reply: tid=0 2024-03-28T22:46:04.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.615+0000 7f203f618740 1 client.18402 shutdown 2024-03-28T22:46:04.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.615+0000 7f203f618740 1 -- 172.21.15.43:0/734378721 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb9b18e70 msgr2=0x55ccb9b25d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:04.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.615+0000 7f203f618740 1 --2- 172.21.15.43:0/734378721 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb9b18e70 0x55ccb9b25d40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x55ccb9b0e840 tx=0x7f202402bcd0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.615+0000 7f203f618740 1 -- 172.21.15.43:0/734378721 shutdown_connections 2024-03-28T22:46:04.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.615+0000 7f203f618740 1 --2- 172.21.15.43:0/734378721 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55ccb9b989a0 0x55ccb9b9adc0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.615+0000 7f203f618740 1 --2- 172.21.15.43:0/734378721 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ccb9b21a50 0x55ccb9b97230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.615+0000 7f203f618740 1 --2- 172.21.15.43:0/734378721 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ccb9b18e70 0x55ccb9b25d40 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.615+0000 7f203f618740 1 --2- 172.21.15.43:0/734378721 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ccb9b11c40 0x55ccb9b25800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.615+0000 7f2017400640 0 client.18402 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:46:04.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.615+0000 7f203f618740 1 -- 172.21.15.43:0/734378721 >> 172.21.15.43:0/734378721 conn(0x55ccb9b12c80 msgr2=0x55ccb9b13df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:04.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.615+0000 7f203f618740 1 -- 172.21.15.43:0/734378721 shutdown_connections 2024-03-28T22:46:04.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.616+0000 7f203f618740 1 -- 172.21.15.43:0/734378721 wait complete. 2024-03-28T22:46:04.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.616+0000 7f203f618740 20 client.18402 trim_cache size 0 max 16384 2024-03-28T22:46:04.622 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:46:04.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.616+0000 7f203b200640 1 -- 172.21.15.43:0/984989431 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f201c0a1f70 msgr2=0x7f201c0a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:04.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.616+0000 7f203b200640 1 --2- 172.21.15.43:0/984989431 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f201c0a1f70 0x7f201c0a4430 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2030011910 tx=0x7f20300096a0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.616+0000 7f203b200640 1 -- 172.21.15.43:0/984989431 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f20340fff30 msgr2=0x7f203419e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:04.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.616+0000 7f203b200640 1 --2- 172.21.15.43:0/984989431 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f20340fff30 0x7f203419e510 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f202800e9c0 tx=0x7f202800ee80 comp rx=0 tx=0).stop 2024-03-28T22:46:04.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.616+0000 7f203b200640 1 -- 172.21.15.43:0/984989431 shutdown_connections 2024-03-28T22:46:04.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.616+0000 7f203b200640 1 --2- 172.21.15.43:0/984989431 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f201c0a1f70 0x7f201c0a4430 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.616+0000 7f203b200640 1 --2- 172.21.15.43:0/984989431 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20341123c0 0x7f20341a2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.616+0000 7f203b200640 1 --2- 172.21.15.43:0/984989431 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2034100850 0x7f203419ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.616+0000 7f203b200640 1 --2- 172.21.15.43:0/984989431 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f20340fff30 0x7f203419e510 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.616+0000 7f203b200640 1 -- 172.21.15.43:0/984989431 >> 172.21.15.43:0/984989431 conn(0x7f20340757c0 msgr2=0x7f203410fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:04.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.617+0000 7f203b200640 1 -- 172.21.15.43:0/984989431 shutdown_connections 2024-03-28T22:46:04.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.617+0000 7f203b200640 1 -- 172.21.15.43:0/984989431 wait complete. 2024-03-28T22:46:04.633 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T22:46:04.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.863+0000 7f4b6fe00640 1 -- 172.21.15.43:0/2597569132 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b701097e0 msgr2=0x7f4b70109bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:04.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.863+0000 7f4b6fe00640 1 --2- 172.21.15.43:0/2597569132 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b701097e0 0x7f4b70109bc0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f4b60009ca0 tx=0x7f4b6002c0f0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.863+0000 7f4b6fe00640 1 -- 172.21.15.43:0/2597569132 shutdown_connections 2024-03-28T22:46:04.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.863+0000 7f4b6fe00640 1 --2- 172.21.15.43:0/2597569132 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b70104190 0x7f4b701106a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.863+0000 7f4b6fe00640 1 --2- 172.21.15.43:0/2597569132 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4b701037d0 0x7f4b70103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.863+0000 7f4b6fe00640 1 --2- 172.21.15.43:0/2597569132 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b701097e0 0x7f4b70109bc0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.863+0000 7f4b6fe00640 1 -- 172.21.15.43:0/2597569132 >> 172.21.15.43:0/2597569132 conn(0x7f4b700ff280 msgr2=0x7f4b701016a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:04.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.863+0000 7f4b6fe00640 1 -- 172.21.15.43:0/2597569132 shutdown_connections 2024-03-28T22:46:04.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.863+0000 7f4b6fe00640 1 -- 172.21.15.43:0/2597569132 wait complete. 2024-03-28T22:46:04.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.864+0000 7f4b6fe00640 1 Processor -- start 2024-03-28T22:46:04.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.864+0000 7f4b6fe00640 1 -- start start 2024-03-28T22:46:04.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.865+0000 7f4b6fe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b701037d0 0x7f4b7019a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.865+0000 7f4b6fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4b70104190 0x7f4b7019a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.865+0000 7f4b6fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b701097e0 0x7f4b7019e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.865+0000 7f4b6fe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4b70113bb0 con 0x7f4b70104190 2024-03-28T22:46:04.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.865+0000 7f4b6fe00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4b70113a30 con 0x7f4b701097e0 2024-03-28T22:46:04.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.865+0000 7f4b6fe00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4b70113d30 con 0x7f4b701037d0 2024-03-28T22:46:04.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.865+0000 7f4b6ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b701037d0 0x7f4b7019a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.865+0000 7f4b6f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b701097e0 0x7f4b7019e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.865+0000 7f4b6ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b701037d0 0x7f4b7019a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:33114/0 (socket says 172.21.15.43:33114) 2024-03-28T22:46:04.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.865+0000 7f4b6ea00640 1 -- 172.21.15.43:0/2953006619 learned_addr learned my addr 172.21.15.43:0/2953006619 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:04.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.865+0000 7f4b6e000640 1 --2- 172.21.15.43:0/2953006619 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4b70104190 0x7f4b7019a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.866+0000 7f4b6ea00640 1 -- 172.21.15.43:0/2953006619 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b701097e0 msgr2=0x7f4b7019e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:04.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.866+0000 7f4b6ea00640 1 --2- 172.21.15.43:0/2953006619 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b701097e0 0x7f4b7019e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.866+0000 7f4b6ea00640 1 -- 172.21.15.43:0/2953006619 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4b70104190 msgr2=0x7f4b7019a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:04.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.866+0000 7f4b6ea00640 1 --2- 172.21.15.43:0/2953006619 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4b70104190 0x7f4b7019a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.866+0000 7f4b6ea00640 1 -- 172.21.15.43:0/2953006619 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b7019f050 con 0x7f4b701037d0 2024-03-28T22:46:04.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.866+0000 7f4b6f400640 1 --2- 172.21.15.43:0/2953006619 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b701097e0 0x7f4b7019e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:04.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.866+0000 7f4b6ea00640 1 --2- 172.21.15.43:0/2953006619 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b701037d0 0x7f4b7019a0b0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f4b60005e30 tx=0x7f4b60032f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:04.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.866+0000 7f4b67400640 1 -- 172.21.15.43:0/2953006619 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b60042020 con 0x7f4b701037d0 2024-03-28T22:46:04.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.866+0000 7f4b67400640 1 -- 172.21.15.43:0/2953006619 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b60034640 con 0x7f4b701037d0 2024-03-28T22:46:04.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.866+0000 7f4b6fe00640 1 -- 172.21.15.43:0/2953006619 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b7019f2e0 con 0x7f4b701037d0 2024-03-28T22:46:04.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.866+0000 7f4b67400640 1 -- 172.21.15.43:0/2953006619 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b60039040 con 0x7f4b701037d0 2024-03-28T22:46:04.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.867+0000 7f4b6fe00640 1 -- 172.21.15.43:0/2953006619 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b701a6bd0 con 0x7f4b701037d0 2024-03-28T22:46:04.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.870+0000 7f4b67400640 1 -- 172.21.15.43:0/2953006619 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f4b60045020 con 0x7f4b701037d0 2024-03-28T22:46:04.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.870+0000 7f4b78775740 1 Processor -- start 2024-03-28T22:46:04.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.870+0000 7f4b78775740 1 -- start start 2024-03-28T22:46:04.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.870+0000 7f4b78775740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55faa3bb5550 0x55faa3bb5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.870+0000 7f4b78775740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faa3bb5e70 0x55faa3bbe510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.870+0000 7f4b78775740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55faa3bbea50 0x55faa3bc0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.870+0000 7f4b78775740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55faa3b19050 con 0x55faa3bb5550 2024-03-28T22:46:04.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.870+0000 7f4b78775740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55faa3ac1ce0 con 0x55faa3bb5e70 2024-03-28T22:46:04.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.870+0000 7f4b78775740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55faa3b150f0 con 0x55faa3bbea50 2024-03-28T22:46:04.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.871+0000 7f4b6ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faa3bb5e70 0x55faa3bbe510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.871+0000 7f4b6f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55faa3bb5550 0x55faa3bb5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.871+0000 7f4b6e000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55faa3bbea50 0x55faa3bc0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.871+0000 7f4b6e000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55faa3bbea50 0x55faa3bc0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:33124/0 (socket says 172.21.15.43:33124) 2024-03-28T22:46:04.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.871+0000 7f4b6f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55faa3bb5550 0x55faa3bb5930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:56692/0 (socket says 172.21.15.43:56692) 2024-03-28T22:46:04.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.871+0000 7f4b6e000640 1 -- 172.21.15.43:0/3526117957 learned_addr learned my addr 172.21.15.43:0/3526117957 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:04.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.871+0000 7f4b6e000640 1 -- 172.21.15.43:0/3526117957 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faa3bb5e70 msgr2=0x55faa3bbe510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:04.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.871+0000 7f4b6e000640 1 --2- 172.21.15.43:0/3526117957 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faa3bb5e70 0x55faa3bbe510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.871+0000 7f4b6e000640 1 -- 172.21.15.43:0/3526117957 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55faa3bb5550 msgr2=0x55faa3bb5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:04.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.871+0000 7f4b6e000640 1 --2- 172.21.15.43:0/3526117957 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55faa3bb5550 0x55faa3bb5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.871+0000 7f4b6e000640 1 -- 172.21.15.43:0/3526117957 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55faa382d3f0 con 0x55faa3bbea50 2024-03-28T22:46:04.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.871+0000 7f4b67400640 1 --2- 172.21.15.43:0/2953006619 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4b540a1ef0 0x7f4b540a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.871+0000 7f4b6e000640 1 --2- 172.21.15.43:0/3526117957 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55faa3bbea50 0x55faa3bc0e40 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f4b58009c90 tx=0x7f4b58011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=860dbc86477365c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:04.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.872+0000 7f4b67400640 1 -- 172.21.15.43:0/2953006619 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7f4b600f04d0 con 0x7f4b701037d0 2024-03-28T22:46:04.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.872+0000 7f4b6f400640 1 --2- 172.21.15.43:0/2953006619 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4b540a1ef0 0x7f4b540a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.872+0000 7f4b5fe00640 1 -- 172.21.15.43:0/3526117957 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b5801e070 con 0x55faa3bbea50 2024-03-28T22:46:04.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.872+0000 7f4b5fe00640 1 -- 172.21.15.43:0/3526117957 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b58019b10 con 0x55faa3bbea50 2024-03-28T22:46:04.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.872+0000 7f4b5fe00640 1 -- 172.21.15.43:0/3526117957 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b5800eac0 con 0x55faa3bbea50 2024-03-28T22:46:04.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.873+0000 7f4b78775740 1 -- 172.21.15.43:0/3526117957 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55faa3bbea50 msgr2=0x55faa3bc0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:04.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.873+0000 7f4b6f400640 1 --2- 172.21.15.43:0/2953006619 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4b540a1ef0 0x7f4b540a43b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4b68012560 tx=0x7f4b68005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:04.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.873+0000 7f4b78775740 1 --2- 172.21.15.43:0/3526117957 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55faa3bbea50 0x55faa3bc0e40 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f4b58009c90 tx=0x7f4b58011810 comp rx=0 tx=0).stop 2024-03-28T22:46:04.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.873+0000 7f4b78775740 1 -- 172.21.15.43:0/3526117957 shutdown_connections 2024-03-28T22:46:04.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.873+0000 7f4b78775740 1 --2- 172.21.15.43:0/3526117957 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faa3bb5e70 0x55faa3bbe510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.873+0000 7f4b78775740 1 --2- 172.21.15.43:0/3526117957 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55faa3bbea50 0x55faa3bc0e40 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.873+0000 7f4b78775740 1 --2- 172.21.15.43:0/3526117957 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55faa3bb5550 0x55faa3bb5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.873+0000 7f4b78775740 1 -- 172.21.15.43:0/3526117957 >> 172.21.15.43:0/3526117957 conn(0x55faa3bafc80 msgr2=0x55faa3bb0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:04.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.873+0000 7f4b78775740 1 -- 172.21.15.43:0/3526117957 shutdown_connections 2024-03-28T22:46:04.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.873+0000 7f4b78775740 1 -- 172.21.15.43:0/3526117957 wait complete. 2024-03-28T22:46:04.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.873+0000 7f4b78775740 1 -- start start 2024-03-28T22:46:04.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.873+0000 7f4b78775740 1 Processor -- start 2024-03-28T22:46:04.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.874+0000 7f4b78775740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faa3bb5550 0x55faa3bbbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.874+0000 7f4b78775740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55faa3bb5e70 0x55faa3bbbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.874+0000 7f4b78775740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55faa3bbea50 0x55faa3bbc520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.874+0000 7f4b78775740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55faa3bc1840 con 0x55faa3bb5e70 2024-03-28T22:46:04.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.874+0000 7f4b78775740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55faa3ab72f0 con 0x55faa3bb5550 2024-03-28T22:46:04.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.874+0000 7f4b78775740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55faa3bc19c0 con 0x55faa3bbea50 2024-03-28T22:46:04.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.874+0000 7f4b78775740 10 client.0 _ll_register_callbacks cb 0x55faa382cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:46:04.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.874+0000 7f4b6ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faa3bb5550 0x55faa3bbbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.874+0000 7f4b6ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faa3bb5550 0x55faa3bbbaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:46490/0 (socket says 172.21.15.43:46490) 2024-03-28T22:46:04.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.874+0000 7f4b6ea00640 1 -- 172.21.15.43:0/3618550266 learned_addr learned my addr 172.21.15.43:0/3618550266 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:04.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.874+0000 7f4b6e000640 1 --2- 172.21.15.43:0/3618550266 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55faa3bb5e70 0x55faa3bbbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.874+0000 7f4b6f400640 1 --2- 172.21.15.43:0/3618550266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55faa3bbea50 0x55faa3bbc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.875+0000 7f4b6ea00640 1 -- 172.21.15.43:0/3618550266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55faa3bbea50 msgr2=0x55faa3bbc520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:04.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.875+0000 7f4b6ea00640 1 --2- 172.21.15.43:0/3618550266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55faa3bbea50 0x55faa3bbc520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.875+0000 7f4b6ea00640 1 -- 172.21.15.43:0/3618550266 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55faa3bb5e70 msgr2=0x55faa3bbbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:04.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.875+0000 7f4b6ea00640 1 --2- 172.21.15.43:0/3618550266 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55faa3bb5e70 0x55faa3bbbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.875+0000 7f4b6ea00640 1 -- 172.21.15.43:0/3618550266 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55faa38f77d0 con 0x55faa3bb5550 2024-03-28T22:46:04.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.875+0000 7f4b6ea00640 1 --2- 172.21.15.43:0/3618550266 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faa3bb5550 0x55faa3bbbaa0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f4b60006010 tx=0x7f4b600324b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23cc95a409b6d56d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:04.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.875+0000 7f4b5cc00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:46:04.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.875+0000 7f4b5cc00640 1 -- 172.21.15.43:0/3618550266 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b600bc940 con 0x55faa3bb5550 2024-03-28T22:46:04.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.875+0000 7f4b5cc00640 1 -- 172.21.15.43:0/3618550266 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7f4b60003970 con 0x55faa3bb5550 2024-03-28T22:46:04.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.875+0000 7f4b78775740 1 -- 172.21.15.43:0/3618550266 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b58019b10 con 0x55faa3bb5550 2024-03-28T22:46:04.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.875+0000 7f4b5cc00640 1 -- 172.21.15.43:0/3618550266 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4b600d5030 con 0x55faa3bb5550 2024-03-28T22:46:04.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.875+0000 7f4b78775740 10 client.28673 fetch_fsmap learned FSMap version 32 2024-03-28T22:46:04.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.875+0000 7f4b78775740 1 -- 172.21.15.43:0/3618550266 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55faa3bcdb00 con 0x55faa3bb5550 2024-03-28T22:46:04.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.876+0000 7f4b5cc00640 1 -- 172.21.15.43:0/3618550266 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b58019b10 con 0x55faa3bb5550 2024-03-28T22:46:04.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.876+0000 7f4b5cc00640 1 -- 172.21.15.43:0/3618550266 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4b600b91e0 con 0x55faa3bb5550 2024-03-28T22:46:04.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.876+0000 7f4b78775740 10 client.28673 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:46:04.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.876+0000 7f4b78775740 10 client.28673 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:46:04.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.876+0000 7f4b78775740 20 client.28673 populate_metadata read hostname 'smithi043' 2024-03-28T22:46:04.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.876+0000 7f4b78775740 1 --2- 172.21.15.43:0/3618550266 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55faa3bcef10 0x55faa3bd13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.876+0000 7f4b78775740 4 client.28673 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:46:04.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.876+0000 7f4b78775740 1 -- 172.21.15.43:0/3618550266 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55faa3ac2680 con 0x55faa3bcef10 2024-03-28T22:46:04.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.876+0000 7f4b6e000640 1 --2- 172.21.15.43:0/3618550266 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55faa3bcef10 0x55faa3bd13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.877+0000 7f4b6e000640 1 --2- 172.21.15.43:0/3618550266 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55faa3bcef10 0x55faa3bd13d0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3fdb916e50e45a3e server_cookie=57aceab454179635 in_seq=0 out_seq=0 2024-03-28T22:46:04.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.877+0000 7f4b5cc00640 10 client.28673 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:46:04.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.877+0000 7f4b5cc00640 1 -- 172.21.15.43:0/3618550266 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55faa3ac2680 con 0x55faa3bcef10 2024-03-28T22:46:04.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.877+0000 7f4b5cc00640 10 client.28673 handle_command_reply: tid=0 2024-03-28T22:46:04.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.878+0000 7f4b78775740 1 client.28673 shutdown 2024-03-28T22:46:04.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.878+0000 7f4b78775740 1 -- 172.21.15.43:0/3618550266 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faa3bb5550 msgr2=0x55faa3bbbaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:04.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.878+0000 7f4b78775740 1 --2- 172.21.15.43:0/3618550266 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faa3bb5550 0x55faa3bbbaa0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f4b60006010 tx=0x7f4b600324b0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.878+0000 7f4b78775740 1 -- 172.21.15.43:0/3618550266 shutdown_connections 2024-03-28T22:46:04.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.878+0000 7f4b78775740 1 --2- 172.21.15.43:0/3618550266 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55faa3bcef10 0x55faa3bd13d0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.878+0000 7f4b78775740 1 --2- 172.21.15.43:0/3618550266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55faa3bbea50 0x55faa3bbc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.878+0000 7f4b78775740 1 --2- 172.21.15.43:0/3618550266 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55faa3bb5e70 0x55faa3bbbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.878+0000 7f4b78775740 1 --2- 172.21.15.43:0/3618550266 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faa3bb5550 0x55faa3bbbaa0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.878+0000 7f4b5cc00640 0 client.28673 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:46:04.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.878+0000 7f4b78775740 1 -- 172.21.15.43:0/3618550266 >> 172.21.15.43:0/3618550266 conn(0x55faa3bafc80 msgr2=0x55faa3bb19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:04.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.878+0000 7f4b78775740 1 -- 172.21.15.43:0/3618550266 shutdown_connections 2024-03-28T22:46:04.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.878+0000 7f4b78775740 1 -- 172.21.15.43:0/3618550266 wait complete. 2024-03-28T22:46:04.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.878+0000 7f4b78775740 20 client.28673 trim_cache size 0 max 16384 2024-03-28T22:46:04.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.893+0000 7f4b78775740 1 Processor -- start 2024-03-28T22:46:04.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.893+0000 7f4b78775740 1 -- start start 2024-03-28T22:46:04.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.893+0000 7f4b78775740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faa3bb5550 0x55faa3bbe1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.893+0000 7f4b78775740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55faa3bb5e70 0x55faa3bb1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.893+0000 7f4b78775740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55faa3bbea50 0x55faa3bbc340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.893+0000 7f4b78775740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4b701b0ca0 con 0x55faa3bb5e70 2024-03-28T22:46:04.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.893+0000 7f4b78775740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4b701b27e0 con 0x55faa3bb5550 2024-03-28T22:46:04.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.893+0000 7f4b78775740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4b701b1370 con 0x55faa3bbea50 2024-03-28T22:46:04.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.894+0000 7f4b6ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faa3bb5550 0x55faa3bbe1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.894+0000 7f4b6e000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55faa3bb5e70 0x55faa3bb1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.894+0000 7f4b6ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faa3bb5550 0x55faa3bbe1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34198/0 (socket says 172.21.15.43:34198) 2024-03-28T22:46:04.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.894+0000 7f4b6ea00640 1 -- 172.21.15.43:0/758141661 learned_addr learned my addr 172.21.15.43:0/758141661 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:04.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.894+0000 7f4b6f400640 1 --2- 172.21.15.43:0/758141661 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55faa3bbea50 0x55faa3bbc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.894+0000 7f4b6ea00640 1 -- 172.21.15.43:0/758141661 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55faa3bbea50 msgr2=0x55faa3bbc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:04.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.894+0000 7f4b6ea00640 1 --2- 172.21.15.43:0/758141661 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55faa3bbea50 0x55faa3bbc340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.894+0000 7f4b6ea00640 1 -- 172.21.15.43:0/758141661 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55faa3bb5e70 msgr2=0x55faa3bb1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:04.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.894+0000 7f4b6ea00640 1 --2- 172.21.15.43:0/758141661 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55faa3bb5e70 0x55faa3bb1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.894+0000 7f4b6ea00640 1 -- 172.21.15.43:0/758141661 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55faa3bc1720 con 0x55faa3bb5550 2024-03-28T22:46:04.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.894+0000 7f4b6f400640 1 --2- 172.21.15.43:0/758141661 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55faa3bbea50 0x55faa3bbc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:46:04.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.894+0000 7f4b6ea00640 1 --2- 172.21.15.43:0/758141661 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faa3bb5550 0x55faa3bbe1d0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f4b600b8fd0 tx=0x7f4b600f3b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15791973ad080f36 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:04.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.895+0000 7f4b5f400640 1 -- 172.21.15.43:0/758141661 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b600bc4d0 con 0x55faa3bb5550 2024-03-28T22:46:04.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.895+0000 7f4b5f400640 1 -- 172.21.15.43:0/758141661 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b600098e0 con 0x55faa3bb5550 2024-03-28T22:46:04.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.895+0000 7f4b5f400640 1 -- 172.21.15.43:0/758141661 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b600bc6b0 con 0x55faa3bb5550 2024-03-28T22:46:04.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.895+0000 7f4b78775740 1 -- 172.21.15.43:0/758141661 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faa3bb5550 msgr2=0x55faa3bbe1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:04.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.895+0000 7f4b78775740 1 --2- 172.21.15.43:0/758141661 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faa3bb5550 0x55faa3bbe1d0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f4b600b8fd0 tx=0x7f4b600f3b10 comp rx=0 tx=0).stop 2024-03-28T22:46:04.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.895+0000 7f4b78775740 1 -- 172.21.15.43:0/758141661 shutdown_connections 2024-03-28T22:46:04.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.895+0000 7f4b78775740 1 --2- 172.21.15.43:0/758141661 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55faa3bbea50 0x55faa3bbc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.895+0000 7f4b78775740 1 --2- 172.21.15.43:0/758141661 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55faa3bb5e70 0x55faa3bb1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.895+0000 7f4b78775740 1 --2- 172.21.15.43:0/758141661 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faa3bb5550 0x55faa3bbe1d0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.895+0000 7f4b78775740 1 -- 172.21.15.43:0/758141661 >> 172.21.15.43:0/758141661 conn(0x55faa3bafc80 msgr2=0x55faa3bae530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:04.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.895+0000 7f4b78775740 1 -- 172.21.15.43:0/758141661 shutdown_connections 2024-03-28T22:46:04.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.896+0000 7f4b78775740 1 -- 172.21.15.43:0/758141661 wait complete. 2024-03-28T22:46:04.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.896+0000 7f4b78775740 1 -- start start 2024-03-28T22:46:04.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.896+0000 7f4b78775740 1 Processor -- start 2024-03-28T22:46:04.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.897+0000 7f4b78775740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55faa3bb5550 0x55faa3bb87a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.897+0000 7f4b78775740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faa3bb5e70 0x55faa3bb8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.897+0000 7f4b78775740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55faa3bbea50 0x55faa3bb9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.897+0000 7f4b78775740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4b701b2110 con 0x55faa3bbea50 2024-03-28T22:46:04.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.897+0000 7f4b78775740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4b701b1a40 con 0x55faa3bb5e70 2024-03-28T22:46:04.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.897+0000 7f4b78775740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4b701aeca0 con 0x55faa3bb5550 2024-03-28T22:46:04.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.897+0000 7f4b78775740 10 client.0 _ll_register_callbacks cb 0x7f4b60005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:46:04.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.897+0000 7f4b6ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55faa3bb5550 0x55faa3bb87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.897+0000 7f4b6ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55faa3bb5550 0x55faa3bb87a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:38564/0 (socket says 172.21.15.43:38564) 2024-03-28T22:46:04.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.897+0000 7f4b6ea00640 1 -- 172.21.15.43:0/2735416003 learned_addr learned my addr 172.21.15.43:0/2735416003 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:04.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.897+0000 7f4b6e000640 1 --2- 172.21.15.43:0/2735416003 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faa3bb5e70 0x55faa3bb8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.897+0000 7f4b6f400640 1 --2- 172.21.15.43:0/2735416003 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55faa3bbea50 0x55faa3bb9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.897+0000 7f4b6ea00640 1 -- 172.21.15.43:0/2735416003 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faa3bb5e70 msgr2=0x55faa3bb8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:04.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.897+0000 7f4b6ea00640 1 --2- 172.21.15.43:0/2735416003 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faa3bb5e70 0x55faa3bb8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.898+0000 7f4b6ea00640 1 -- 172.21.15.43:0/2735416003 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55faa3bbea50 msgr2=0x55faa3bb9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:04.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.898+0000 7f4b6ea00640 1 --2- 172.21.15.43:0/2735416003 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55faa3bbea50 0x55faa3bb9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.898+0000 7f4b6ea00640 1 -- 172.21.15.43:0/2735416003 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55faa3c343d0 con 0x55faa3bb5550 2024-03-28T22:46:04.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.898+0000 7f4b6f400640 1 --2- 172.21.15.43:0/2735416003 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55faa3bbea50 0x55faa3bb9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:46:04.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.898+0000 7f4b6e000640 1 --2- 172.21.15.43:0/2735416003 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faa3bb5e70 0x55faa3bb8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:04.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.898+0000 7f4b6ea00640 1 --2- 172.21.15.43:0/2735416003 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55faa3bb5550 0x55faa3bb87a0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f4b60034aa0 tx=0x7f4b600eda70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d6a385cfb8ddc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:04.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.898+0000 7f4b5cc00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:46:04.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.898+0000 7f4b5cc00640 1 -- 172.21.15.43:0/2735416003 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b600bc940 con 0x55faa3bb5550 2024-03-28T22:46:04.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.898+0000 7f4b5cc00640 1 -- 172.21.15.43:0/2735416003 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7f4b600c4e80 con 0x55faa3bb5550 2024-03-28T22:46:04.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.898+0000 7f4b78775740 1 -- 172.21.15.43:0/2735416003 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b600098e0 con 0x55faa3bb5550 2024-03-28T22:46:04.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.898+0000 7f4b5cc00640 1 -- 172.21.15.43:0/2735416003 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4b600a1050 con 0x55faa3bb5550 2024-03-28T22:46:04.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.898+0000 7f4b78775740 10 client.28621 fetch_fsmap learned FSMap version 32 2024-03-28T22:46:04.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.898+0000 7f4b78775740 1 -- 172.21.15.43:0/2735416003 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55faa3c343d0 con 0x55faa3bb5550 2024-03-28T22:46:04.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.898+0000 7f4b5cc00640 1 -- 172.21.15.43:0/2735416003 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b600098e0 con 0x55faa3bb5550 2024-03-28T22:46:04.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.899+0000 7f4b5cc00640 1 -- 172.21.15.43:0/2735416003 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4b60004b40 con 0x55faa3bb5550 2024-03-28T22:46:04.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.899+0000 7f4b78775740 10 client.28621 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:46:04.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.899+0000 7f4b78775740 10 client.28621 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:46:04.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.899+0000 7f4b78775740 20 client.28621 populate_metadata read hostname 'smithi043' 2024-03-28T22:46:04.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.899+0000 7f4b78775740 1 --2- 172.21.15.43:0/2735416003 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55faa3c358e0 0x55faa3c37cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:04.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.899+0000 7f4b78775740 4 client.28621 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:46:04.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.899+0000 7f4b78775740 1 -- 172.21.15.43:0/2735416003 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55faa3b26590 con 0x55faa3c358e0 2024-03-28T22:46:04.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.899+0000 7f4b6e000640 1 --2- 172.21.15.43:0/2735416003 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55faa3c358e0 0x55faa3c37cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:04.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.899+0000 7f4b6e000640 1 --2- 172.21.15.43:0/2735416003 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55faa3c358e0 0x55faa3c37cd0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b84ae1b91729b4b4 server_cookie=b9fa9252fb5190f3 in_seq=0 out_seq=0 2024-03-28T22:46:04.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.899+0000 7f4b5cc00640 10 client.28621 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:46:04.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.900+0000 7f4b5cc00640 1 -- 172.21.15.43:0/2735416003 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55faa3b26590 con 0x55faa3c358e0 2024-03-28T22:46:04.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.900+0000 7f4b5cc00640 10 client.28621 handle_command_reply: tid=0 2024-03-28T22:46:04.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.900+0000 7f4b78775740 1 client.28621 shutdown 2024-03-28T22:46:04.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.900+0000 7f4b78775740 1 -- 172.21.15.43:0/2735416003 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55faa3bb5550 msgr2=0x55faa3bb87a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:04.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.900+0000 7f4b78775740 1 --2- 172.21.15.43:0/2735416003 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55faa3bb5550 0x55faa3bb87a0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f4b60034aa0 tx=0x7f4b600eda70 comp rx=0 tx=0).stop 2024-03-28T22:46:04.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.900+0000 7f4b78775740 1 -- 172.21.15.43:0/2735416003 shutdown_connections 2024-03-28T22:46:04.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.900+0000 7f4b78775740 1 --2- 172.21.15.43:0/2735416003 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55faa3c358e0 0x55faa3c37cd0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.900+0000 7f4b78775740 1 --2- 172.21.15.43:0/2735416003 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faa3bb5e70 0x55faa3bb8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.900+0000 7f4b78775740 1 --2- 172.21.15.43:0/2735416003 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55faa3bbea50 0x55faa3bb9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.900+0000 7f4b78775740 1 --2- 172.21.15.43:0/2735416003 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55faa3bb5550 0x55faa3bb87a0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.900+0000 7f4b78775740 1 -- 172.21.15.43:0/2735416003 >> 172.21.15.43:0/2735416003 conn(0x55faa3bafc80 msgr2=0x55faa3bb17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:04.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.900+0000 7f4b5cc00640 0 client.28621 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:46:04.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.900+0000 7f4b78775740 1 -- 172.21.15.43:0/2735416003 shutdown_connections 2024-03-28T22:46:04.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.900+0000 7f4b78775740 1 -- 172.21.15.43:0/2735416003 wait complete. 2024-03-28T22:46:04.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.900+0000 7f4b78775740 20 client.28621 trim_cache size 0 max 16384 2024-03-28T22:46:04.909 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:46:04.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.901+0000 7f4b6fe00640 1 -- 172.21.15.43:0/2953006619 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4b540a1ef0 msgr2=0x7f4b540a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:04.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.901+0000 7f4b6fe00640 1 --2- 172.21.15.43:0/2953006619 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4b540a1ef0 0x7f4b540a43b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4b68012560 tx=0x7f4b68005f90 comp rx=0 tx=0).stop 2024-03-28T22:46:04.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.901+0000 7f4b6fe00640 1 -- 172.21.15.43:0/2953006619 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b701037d0 msgr2=0x7f4b7019a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:04.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.901+0000 7f4b6fe00640 1 --2- 172.21.15.43:0/2953006619 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b701037d0 0x7f4b7019a0b0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f4b60005e30 tx=0x7f4b60032f20 comp rx=0 tx=0).stop 2024-03-28T22:46:04.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.901+0000 7f4b6fe00640 1 -- 172.21.15.43:0/2953006619 shutdown_connections 2024-03-28T22:46:04.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.901+0000 7f4b6fe00640 1 --2- 172.21.15.43:0/2953006619 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4b540a1ef0 0x7f4b540a43b0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.901+0000 7f4b6fe00640 1 --2- 172.21.15.43:0/2953006619 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b701097e0 0x7f4b7019e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.901+0000 7f4b6fe00640 1 --2- 172.21.15.43:0/2953006619 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4b70104190 0x7f4b7019a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.901+0000 7f4b6fe00640 1 --2- 172.21.15.43:0/2953006619 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b701037d0 0x7f4b7019a0b0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:04.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.901+0000 7f4b6fe00640 1 -- 172.21.15.43:0/2953006619 >> 172.21.15.43:0/2953006619 conn(0x7f4b700ff280 msgr2=0x7f4b700ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:04.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.901+0000 7f4b6fe00640 1 -- 172.21.15.43:0/2953006619 shutdown_connections 2024-03-28T22:46:04.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:04.901+0000 7f4b6fe00640 1 -- 172.21.15.43:0/2953006619 wait complete. 2024-03-28T22:46:04.916 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:46:05.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.143+0000 7f8f9e200640 1 -- 172.21.15.43:0/2337326899 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8f98073b80 msgr2=0x7f8f98073f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:05.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.143+0000 7f8f9e200640 1 --2- 172.21.15.43:0/2337326899 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8f98073b80 0x7f8f98073f60 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f8f88009ca0 tx=0x7f8f8802c180 comp rx=0 tx=0).stop 2024-03-28T22:46:05.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.143+0000 7f8f9e200640 1 -- 172.21.15.43:0/2337326899 shutdown_connections 2024-03-28T22:46:05.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.143+0000 7f8f9e200640 1 --2- 172.21.15.43:0/2337326899 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f9810ddf0 0x7f8f98110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.143+0000 7f8f9e200640 1 --2- 172.21.15.43:0/2337326899 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f980744a0 0x7f8f9810d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.143+0000 7f8f9e200640 1 --2- 172.21.15.43:0/2337326899 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8f98073b80 0x7f8f98073f60 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.143+0000 7f8f9e200640 1 -- 172.21.15.43:0/2337326899 >> 172.21.15.43:0/2337326899 conn(0x7f8f980fd0c0 msgr2=0x7f8f980ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:05.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.143+0000 7f8f9e200640 1 -- 172.21.15.43:0/2337326899 shutdown_connections 2024-03-28T22:46:05.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.143+0000 7f8f9e200640 1 -- 172.21.15.43:0/2337326899 wait complete. 2024-03-28T22:46:05.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.144+0000 7f8f9e200640 1 Processor -- start 2024-03-28T22:46:05.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.144+0000 7f8f9e200640 1 -- start start 2024-03-28T22:46:05.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.145+0000 7f8f9e200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f98073b80 0x7f8f9819a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.145+0000 7f8f9e200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8f980744a0 0x7f8f9819a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.145+0000 7f8f9e200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f9810ddf0 0x7f8f9819e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.145+0000 7f8f9e200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8f98113b90 con 0x7f8f9810ddf0 2024-03-28T22:46:05.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.145+0000 7f8f9e200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8f98113a10 con 0x7f8f980744a0 2024-03-28T22:46:05.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.145+0000 7f8f9e200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f8f98113d10 con 0x7f8f98073b80 2024-03-28T22:46:05.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.145+0000 7f8f9ce00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f98073b80 0x7f8f9819a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.145+0000 7f8f8fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8f980744a0 0x7f8f9819a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.145+0000 7f8f9d800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f9810ddf0 0x7f8f9819e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.145+0000 7f8f9ce00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f98073b80 0x7f8f9819a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:38578/0 (socket says 172.21.15.43:38578) 2024-03-28T22:46:05.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.145+0000 7f8f9ce00640 1 -- 172.21.15.43:0/129667124 learned_addr learned my addr 172.21.15.43:0/129667124 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:05.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.145+0000 7f8f9ce00640 1 -- 172.21.15.43:0/129667124 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8f980744a0 msgr2=0x7f8f9819a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:05.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.146+0000 7f8f9ce00640 1 --2- 172.21.15.43:0/129667124 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8f980744a0 0x7f8f9819a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.146+0000 7f8f9ce00640 1 -- 172.21.15.43:0/129667124 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f9810ddf0 msgr2=0x7f8f9819e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:05.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.146+0000 7f8f9ce00640 1 --2- 172.21.15.43:0/129667124 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f9810ddf0 0x7f8f9819e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.146+0000 7f8f9ce00640 1 -- 172.21.15.43:0/129667124 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f9819f0c0 con 0x7f8f98073b80 2024-03-28T22:46:05.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.146+0000 7f8f8fe00640 1 --2- 172.21.15.43:0/129667124 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8f980744a0 0x7f8f9819a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:05.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.146+0000 7f8f9d800640 1 --2- 172.21.15.43:0/129667124 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f9810ddf0 0x7f8f9819e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:05.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.146+0000 7f8f9ce00640 1 --2- 172.21.15.43:0/129667124 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f98073b80 0x7f8f9819a0d0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f8f88005e30 tx=0x7f8f88035d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:05.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.146+0000 7f8f96000640 1 -- 172.21.15.43:0/129667124 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f88039040 con 0x7f8f98073b80 2024-03-28T22:46:05.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.146+0000 7f8f96000640 1 -- 172.21.15.43:0/129667124 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f88044070 con 0x7f8f98073b80 2024-03-28T22:46:05.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.146+0000 7f8f96000640 1 -- 172.21.15.43:0/129667124 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f880328d0 con 0x7f8f98073b80 2024-03-28T22:46:05.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.146+0000 7f8f9e200640 1 -- 172.21.15.43:0/129667124 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f9819f350 con 0x7f8f98073b80 2024-03-28T22:46:05.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.146+0000 7f8f9e200640 1 -- 172.21.15.43:0/129667124 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f981a6c40 con 0x7f8f98073b80 2024-03-28T22:46:05.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.150+0000 7f8f96000640 1 -- 172.21.15.43:0/129667124 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f8f88003d00 con 0x7f8f98073b80 2024-03-28T22:46:05.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.150+0000 7f8fa258a740 1 Processor -- start 2024-03-28T22:46:05.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.150+0000 7f8fa258a740 1 -- start start 2024-03-28T22:46:05.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.150+0000 7f8fa258a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf5e78550 0x55adf5e78930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.150+0000 7f8fa258a740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55adf5e78e70 0x55adf5e81510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.150+0000 7f8fa258a740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55adf5e81a50 0x55adf5e83e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.150+0000 7f8fa258a740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55adf5ddc050 con 0x55adf5e78e70 2024-03-28T22:46:05.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.150+0000 7f8fa258a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55adf5d84ce0 con 0x55adf5e78550 2024-03-28T22:46:05.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.150+0000 7f8fa258a740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55adf5dd80f0 con 0x55adf5e81a50 2024-03-28T22:46:05.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.151+0000 7f8f9d800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf5e78550 0x55adf5e78930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.151+0000 7f8f8fe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55adf5e81a50 0x55adf5e83e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.151+0000 7f8f9d800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf5e78550 0x55adf5e78930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34238/0 (socket says 172.21.15.43:34238) 2024-03-28T22:46:05.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.151+0000 7f8f9d800640 1 -- 172.21.15.43:0/4034505444 learned_addr learned my addr 172.21.15.43:0/4034505444 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:05.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.151+0000 7f8f9ce00640 1 --2- 172.21.15.43:0/4034505444 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55adf5e78e70 0x55adf5e81510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.151+0000 7f8f8fe00640 1 -- 172.21.15.43:0/4034505444 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf5e78550 msgr2=0x55adf5e78930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:05.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.151+0000 7f8f8fe00640 1 --2- 172.21.15.43:0/4034505444 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf5e78550 0x55adf5e78930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.151+0000 7f8f8fe00640 1 -- 172.21.15.43:0/4034505444 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55adf5e78e70 msgr2=0x55adf5e81510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:05.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.151+0000 7f8f8fe00640 1 --2- 172.21.15.43:0/4034505444 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55adf5e78e70 0x55adf5e81510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.151+0000 7f8f8fe00640 1 -- 172.21.15.43:0/4034505444 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adf5af03f0 con 0x55adf5e81a50 2024-03-28T22:46:05.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.151+0000 7f8f9d800640 1 --2- 172.21.15.43:0/4034505444 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf5e78550 0x55adf5e78930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:05.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.151+0000 7f8f8fe00640 1 --2- 172.21.15.43:0/4034505444 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55adf5e81a50 0x55adf5e83e40 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f8f80009fa0 tx=0x7f8f800118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ab89408ad1cc942 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:05.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.151+0000 7f8f96000640 1 --2- 172.21.15.43:0/129667124 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f8f7c0a1ef0 0x7f8f7c0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.152+0000 7f8f8e000640 1 -- 172.21.15.43:0/4034505444 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f80004030 con 0x55adf5e81a50 2024-03-28T22:46:05.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.152+0000 7f8f8e000640 1 -- 172.21.15.43:0/4034505444 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f80005500 con 0x55adf5e81a50 2024-03-28T22:46:05.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.152+0000 7f8f8e000640 1 -- 172.21.15.43:0/4034505444 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f80004e10 con 0x55adf5e81a50 2024-03-28T22:46:05.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.152+0000 7f8f96000640 1 -- 172.21.15.43:0/129667124 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7f8f880f08c0 con 0x7f8f98073b80 2024-03-28T22:46:05.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.152+0000 7f8f9d800640 1 --2- 172.21.15.43:0/129667124 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f8f7c0a1ef0 0x7f8f7c0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.152+0000 7f8fa258a740 1 -- 172.21.15.43:0/4034505444 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55adf5e81a50 msgr2=0x55adf5e83e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:05.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.152+0000 7f8fa258a740 1 --2- 172.21.15.43:0/4034505444 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55adf5e81a50 0x55adf5e83e40 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f8f80009fa0 tx=0x7f8f800118d0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.152+0000 7f8fa258a740 1 -- 172.21.15.43:0/4034505444 shutdown_connections 2024-03-28T22:46:05.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.152+0000 7f8fa258a740 1 --2- 172.21.15.43:0/4034505444 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55adf5e81a50 0x55adf5e83e40 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.152+0000 7f8fa258a740 1 --2- 172.21.15.43:0/4034505444 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55adf5e78e70 0x55adf5e81510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.152+0000 7f8fa258a740 1 --2- 172.21.15.43:0/4034505444 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf5e78550 0x55adf5e78930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.152+0000 7f8fa258a740 1 -- 172.21.15.43:0/4034505444 >> 172.21.15.43:0/4034505444 conn(0x55adf5e72c80 msgr2=0x55adf5e73060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:05.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.153+0000 7f8fa258a740 1 -- 172.21.15.43:0/4034505444 shutdown_connections 2024-03-28T22:46:05.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.153+0000 7f8fa258a740 1 -- 172.21.15.43:0/4034505444 wait complete. 2024-03-28T22:46:05.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.153+0000 7f8f9d800640 1 --2- 172.21.15.43:0/129667124 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f8f7c0a1ef0 0x7f8f7c0a43b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8f90012880 tx=0x7f8f90005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:05.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.153+0000 7f8fa258a740 1 -- start start 2024-03-28T22:46:05.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.153+0000 7f8fa258a740 1 Processor -- start 2024-03-28T22:46:05.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.154+0000 7f8fa258a740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55adf5e78550 0x55adf5e7ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.154+0000 7f8fa258a740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55adf5e78e70 0x55adf5e7f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.154+0000 7f8fa258a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf5e81a50 0x55adf5e7f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.154+0000 7f8fa258a740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55adf5e84840 con 0x55adf5e78e70 2024-03-28T22:46:05.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.154+0000 7f8fa258a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55adf5d7a2f0 con 0x55adf5e81a50 2024-03-28T22:46:05.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.154+0000 7f8fa258a740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55adf5e849c0 con 0x55adf5e78550 2024-03-28T22:46:05.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.154+0000 7f8fa258a740 10 client.0 _ll_register_callbacks cb 0x55adf5aefbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:46:05.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.154+0000 7f8f9d800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf5e81a50 0x55adf5e7f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.154+0000 7f8f9ce00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55adf5e78550 0x55adf5e7ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.154+0000 7f8f9d800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf5e81a50 0x55adf5e7f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34246/0 (socket says 172.21.15.43:34246) 2024-03-28T22:46:05.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.154+0000 7f8f9d800640 1 -- 172.21.15.43:0/4056197828 learned_addr learned my addr 172.21.15.43:0/4056197828 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:05.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.154+0000 7f8f8fe00640 1 --2- 172.21.15.43:0/4056197828 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55adf5e78e70 0x55adf5e7f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.155+0000 7f8f9d800640 1 -- 172.21.15.43:0/4056197828 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55adf5e78550 msgr2=0x55adf5e7ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:05.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.155+0000 7f8f9d800640 1 --2- 172.21.15.43:0/4056197828 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55adf5e78550 0x55adf5e7ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.155+0000 7f8f9d800640 1 -- 172.21.15.43:0/4056197828 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55adf5e78e70 msgr2=0x55adf5e7f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:05.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.155+0000 7f8f9d800640 1 --2- 172.21.15.43:0/4056197828 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55adf5e78e70 0x55adf5e7f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.155+0000 7f8f9d800640 1 -- 172.21.15.43:0/4056197828 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adf5bba7d0 con 0x55adf5e81a50 2024-03-28T22:46:05.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.155+0000 7f8f9ce00640 1 --2- 172.21.15.43:0/4056197828 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55adf5e78550 0x55adf5e7ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:05.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.155+0000 7f8f9d800640 1 --2- 172.21.15.43:0/4056197828 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf5e81a50 0x55adf5e7f550 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f8f900042e0 tx=0x7f8f90004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70b4651263df6ab3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:05.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.155+0000 7f8f86a00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:46:05.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.155+0000 7f8f86a00640 1 -- 172.21.15.43:0/4056197828 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f90004310 con 0x55adf5e81a50 2024-03-28T22:46:05.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.155+0000 7f8f86a00640 1 -- 172.21.15.43:0/4056197828 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7f8f90003af0 con 0x55adf5e81a50 2024-03-28T22:46:05.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.155+0000 7f8fa258a740 1 -- 172.21.15.43:0/4056197828 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f80005500 con 0x55adf5e81a50 2024-03-28T22:46:05.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.155+0000 7f8f86a00640 1 -- 172.21.15.43:0/4056197828 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8f900044b0 con 0x55adf5e81a50 2024-03-28T22:46:05.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.156+0000 7f8fa258a740 10 client.28691 fetch_fsmap learned FSMap version 32 2024-03-28T22:46:05.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.156+0000 7f8fa258a740 1 -- 172.21.15.43:0/4056197828 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55adf5e90b90 con 0x55adf5e81a50 2024-03-28T22:46:05.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.156+0000 7f8f86a00640 1 -- 172.21.15.43:0/4056197828 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f80005500 con 0x55adf5e81a50 2024-03-28T22:46:05.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.156+0000 7f8f86a00640 1 -- 172.21.15.43:0/4056197828 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f8f90027020 con 0x55adf5e81a50 2024-03-28T22:46:05.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.156+0000 7f8fa258a740 10 client.28691 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:46:05.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.156+0000 7f8fa258a740 10 client.28691 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:46:05.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.156+0000 7f8fa258a740 20 client.28691 populate_metadata read hostname 'smithi043' 2024-03-28T22:46:05.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.156+0000 7f8fa258a740 1 --2- 172.21.15.43:0/4056197828 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55adf5e92050 0x55adf5e944a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.156+0000 7f8fa258a740 4 client.28691 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:46:05.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.156+0000 7f8fa258a740 1 -- 172.21.15.43:0/4056197828 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55adf5d85680 con 0x55adf5e92050 2024-03-28T22:46:05.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.156+0000 7f8f9ce00640 1 --2- 172.21.15.43:0/4056197828 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55adf5e92050 0x55adf5e944a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.157+0000 7f8f9ce00640 1 --2- 172.21.15.43:0/4056197828 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55adf5e92050 0x55adf5e944a0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=69f1f91b726cc95e server_cookie=8e133b5668eec7c8 in_seq=0 out_seq=0 2024-03-28T22:46:05.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.157+0000 7f8f86a00640 10 client.28691 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:46:05.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.158+0000 7f8f86a00640 1 -- 172.21.15.43:0/4056197828 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55adf5d85680 con 0x55adf5e92050 2024-03-28T22:46:05.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.158+0000 7f8f86a00640 10 client.28691 handle_command_reply: tid=0 2024-03-28T22:46:05.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.158+0000 7f8fa258a740 1 client.28691 shutdown 2024-03-28T22:46:05.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.158+0000 7f8fa258a740 1 -- 172.21.15.43:0/4056197828 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf5e81a50 msgr2=0x55adf5e7f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:05.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.158+0000 7f8fa258a740 1 --2- 172.21.15.43:0/4056197828 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf5e81a50 0x55adf5e7f550 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f8f900042e0 tx=0x7f8f90004730 comp rx=0 tx=0).stop 2024-03-28T22:46:05.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.158+0000 7f8fa258a740 1 -- 172.21.15.43:0/4056197828 shutdown_connections 2024-03-28T22:46:05.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.158+0000 7f8fa258a740 1 --2- 172.21.15.43:0/4056197828 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55adf5e92050 0x55adf5e944a0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.158+0000 7f8fa258a740 1 --2- 172.21.15.43:0/4056197828 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf5e81a50 0x55adf5e7f550 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.158+0000 7f8fa258a740 1 --2- 172.21.15.43:0/4056197828 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55adf5e78e70 0x55adf5e7f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.158+0000 7f8fa258a740 1 --2- 172.21.15.43:0/4056197828 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55adf5e78550 0x55adf5e7ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.158+0000 7f8fa258a740 1 -- 172.21.15.43:0/4056197828 >> 172.21.15.43:0/4056197828 conn(0x55adf5e72c80 msgr2=0x55adf5e749d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:05.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.158+0000 7f8f86a00640 0 client.28691 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:46:05.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.159+0000 7f8fa258a740 1 -- 172.21.15.43:0/4056197828 shutdown_connections 2024-03-28T22:46:05.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.159+0000 7f8fa258a740 1 -- 172.21.15.43:0/4056197828 wait complete. 2024-03-28T22:46:05.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.159+0000 7f8fa258a740 20 client.28691 trim_cache size 0 max 16384 2024-03-28T22:46:05.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.173+0000 7f8fa258a740 1 Processor -- start 2024-03-28T22:46:05.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.174+0000 7f8fa258a740 1 -- start start 2024-03-28T22:46:05.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.174+0000 7f8fa258a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf5e78550 0x55adf5e81200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.174+0000 7f8fa258a740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55adf5e78e70 0x55adf5e8d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.174+0000 7f8fa258a740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55adf5e81a50 0x55adf5e99d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.174+0000 7f8fa258a740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8f981b0cf0 con 0x55adf5e78e70 2024-03-28T22:46:05.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.174+0000 7f8fa258a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8f981b2830 con 0x55adf5e78550 2024-03-28T22:46:05.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.174+0000 7f8fa258a740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f8f981b13c0 con 0x55adf5e81a50 2024-03-28T22:46:05.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.174+0000 7f8f9ce00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf5e78550 0x55adf5e81200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.174+0000 7f8f8fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55adf5e78e70 0x55adf5e8d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.174+0000 7f8f9ce00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf5e78550 0x55adf5e81200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34250/0 (socket says 172.21.15.43:34250) 2024-03-28T22:46:05.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.174+0000 7f8f9ce00640 1 -- 172.21.15.43:0/2960637227 learned_addr learned my addr 172.21.15.43:0/2960637227 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:05.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.174+0000 7f8f9d800640 1 --2- 172.21.15.43:0/2960637227 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55adf5e81a50 0x55adf5e99d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.174+0000 7f8f9ce00640 1 -- 172.21.15.43:0/2960637227 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55adf5e81a50 msgr2=0x55adf5e99d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:05.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.174+0000 7f8f9ce00640 1 --2- 172.21.15.43:0/2960637227 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55adf5e81a50 0x55adf5e99d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.175+0000 7f8f9ce00640 1 -- 172.21.15.43:0/2960637227 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55adf5e78e70 msgr2=0x55adf5e8d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:05.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.175+0000 7f8f9ce00640 1 --2- 172.21.15.43:0/2960637227 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55adf5e78e70 0x55adf5e8d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.175+0000 7f8f9ce00640 1 -- 172.21.15.43:0/2960637227 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adf5e6d1d0 con 0x55adf5e78550 2024-03-28T22:46:05.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.175+0000 7f8f8fe00640 1 --2- 172.21.15.43:0/2960637227 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55adf5e78e70 0x55adf5e8d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:05.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.175+0000 7f8f9d800640 1 --2- 172.21.15.43:0/2960637227 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55adf5e81a50 0x55adf5e99d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:46:05.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.175+0000 7f8f9ce00640 1 --2- 172.21.15.43:0/2960637227 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf5e78550 0x55adf5e81200 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f8f880b9990 tx=0x7f8f880f6550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d399604b269fa037 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:05.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.175+0000 7f8f8d600640 1 -- 172.21.15.43:0/2960637227 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f880bcd90 con 0x55adf5e78550 2024-03-28T22:46:05.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.175+0000 7f8f8d600640 1 -- 172.21.15.43:0/2960637227 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f880bd6c0 con 0x55adf5e78550 2024-03-28T22:46:05.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.175+0000 7f8f8d600640 1 -- 172.21.15.43:0/2960637227 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f880bc920 con 0x55adf5e78550 2024-03-28T22:46:05.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.176+0000 7f8fa258a740 1 -- 172.21.15.43:0/2960637227 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf5e78550 msgr2=0x55adf5e81200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:05.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.176+0000 7f8fa258a740 1 --2- 172.21.15.43:0/2960637227 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf5e78550 0x55adf5e81200 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f8f880b9990 tx=0x7f8f880f6550 comp rx=0 tx=0).stop 2024-03-28T22:46:05.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.176+0000 7f8fa258a740 1 -- 172.21.15.43:0/2960637227 shutdown_connections 2024-03-28T22:46:05.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.176+0000 7f8fa258a740 1 --2- 172.21.15.43:0/2960637227 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55adf5e81a50 0x55adf5e99d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.176+0000 7f8fa258a740 1 --2- 172.21.15.43:0/2960637227 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55adf5e78e70 0x55adf5e8d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.176+0000 7f8fa258a740 1 --2- 172.21.15.43:0/2960637227 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf5e78550 0x55adf5e81200 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.176+0000 7f8fa258a740 1 -- 172.21.15.43:0/2960637227 >> 172.21.15.43:0/2960637227 conn(0x55adf5e72c80 msgr2=0x55adf5e71dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:05.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.176+0000 7f8fa258a740 1 -- 172.21.15.43:0/2960637227 shutdown_connections 2024-03-28T22:46:05.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.176+0000 7f8fa258a740 1 -- 172.21.15.43:0/2960637227 wait complete. 2024-03-28T22:46:05.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.176+0000 7f8fa258a740 1 -- start start 2024-03-28T22:46:05.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.176+0000 7f8fa258a740 1 Processor -- start 2024-03-28T22:46:05.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.177+0000 7f8fa258a740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55adf5e78550 0x55adf5e89e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.177+0000 7f8fa258a740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55adf5e78e70 0x55adf5e851f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.177+0000 7f8fa258a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf5e81a50 0x55adf5e85730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.177+0000 7f8fa258a740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8f981b2160 con 0x55adf5e78550 2024-03-28T22:46:05.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.177+0000 7f8fa258a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8f981b1a90 con 0x55adf5e81a50 2024-03-28T22:46:05.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.177+0000 7f8fa258a740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f8f981aecf0 con 0x55adf5e78e70 2024-03-28T22:46:05.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.177+0000 7f8fa258a740 10 client.0 _ll_register_callbacks cb 0x7f8f90012e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:46:05.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.177+0000 7f8f9ce00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55adf5e78550 0x55adf5e89e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.177+0000 7f8f8fe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55adf5e78e70 0x55adf5e851f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.177+0000 7f8f8fe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55adf5e78e70 0x55adf5e851f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:38606/0 (socket says 172.21.15.43:38606) 2024-03-28T22:46:05.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.177+0000 7f8f9d800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf5e81a50 0x55adf5e85730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.177+0000 7f8f9ce00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55adf5e78550 0x55adf5e89e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34442/0 (socket says 172.21.15.43:34442) 2024-03-28T22:46:05.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.177+0000 7f8f8fe00640 1 -- 172.21.15.43:0/3694229309 learned_addr learned my addr 172.21.15.43:0/3694229309 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:05.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.178+0000 7f8f8fe00640 1 -- 172.21.15.43:0/3694229309 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf5e81a50 msgr2=0x55adf5e85730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:05.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.178+0000 7f8f8fe00640 1 --2- 172.21.15.43:0/3694229309 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf5e81a50 0x55adf5e85730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.178+0000 7f8f8fe00640 1 -- 172.21.15.43:0/3694229309 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55adf5e78550 msgr2=0x55adf5e89e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:05.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.178+0000 7f8f8fe00640 1 --2- 172.21.15.43:0/3694229309 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55adf5e78550 0x55adf5e89e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.178+0000 7f8f8fe00640 1 -- 172.21.15.43:0/3694229309 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adf5e85f40 con 0x55adf5e78e70 2024-03-28T22:46:05.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.178+0000 7f8f9d800640 1 --2- 172.21.15.43:0/3694229309 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf5e81a50 0x55adf5e85730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:05.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.178+0000 7f8f9ce00640 1 --2- 172.21.15.43:0/3694229309 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55adf5e78550 0x55adf5e89e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:05.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.178+0000 7f8f8fe00640 1 --2- 172.21.15.43:0/3694229309 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55adf5e78e70 0x55adf5e851f0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x55adf5e73e10 tx=0x7f8f800045a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aedc7c505f5be153 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:05.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.178+0000 7f8f86a00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:46:05.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.178+0000 7f8f86a00640 1 -- 172.21.15.43:0/3694229309 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f8001d040 con 0x55adf5e78e70 2024-03-28T22:46:05.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.178+0000 7f8fa258a740 1 -- 172.21.15.43:0/3694229309 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f880bd6c0 con 0x55adf5e78e70 2024-03-28T22:46:05.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.178+0000 7f8f86a00640 1 -- 172.21.15.43:0/3694229309 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7f8f80004690 con 0x55adf5e78e70 2024-03-28T22:46:05.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.178+0000 7f8f86a00640 1 -- 172.21.15.43:0/3694229309 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8f800035f0 con 0x55adf5e78e70 2024-03-28T22:46:05.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.179+0000 7f8fa258a740 10 client.28639 fetch_fsmap learned FSMap version 32 2024-03-28T22:46:05.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.179+0000 7f8fa258a740 1 -- 172.21.15.43:0/3694229309 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55adf5ef8370 con 0x55adf5e78e70 2024-03-28T22:46:05.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.179+0000 7f8f86a00640 1 -- 172.21.15.43:0/3694229309 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f880bd6c0 con 0x55adf5e78e70 2024-03-28T22:46:05.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.179+0000 7f8f86a00640 1 -- 172.21.15.43:0/3694229309 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f8f8000fab0 con 0x55adf5e78e70 2024-03-28T22:46:05.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.179+0000 7f8fa258a740 10 client.28639 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:46:05.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.179+0000 7f8fa258a740 10 client.28639 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:46:05.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.179+0000 7f8fa258a740 20 client.28639 populate_metadata read hostname 'smithi043' 2024-03-28T22:46:05.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.179+0000 7f8fa258a740 1 --2- 172.21.15.43:0/3694229309 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55adf5ef8910 0x55adf5efad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.179+0000 7f8fa258a740 4 client.28639 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:46:05.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.179+0000 7f8fa258a740 1 -- 172.21.15.43:0/3694229309 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55adf5de9590 con 0x55adf5ef8910 2024-03-28T22:46:05.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.179+0000 7f8f9ce00640 1 --2- 172.21.15.43:0/3694229309 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55adf5ef8910 0x55adf5efad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.180+0000 7f8f9ce00640 1 --2- 172.21.15.43:0/3694229309 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55adf5ef8910 0x55adf5efad30 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=edc60cfa93e7b5f1 server_cookie=f2de132057fef427 in_seq=0 out_seq=0 2024-03-28T22:46:05.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.180+0000 7f8f86a00640 10 client.28639 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:46:05.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.180+0000 7f8f86a00640 1 -- 172.21.15.43:0/3694229309 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55adf5de9590 con 0x55adf5ef8910 2024-03-28T22:46:05.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.180+0000 7f8f86a00640 10 client.28639 handle_command_reply: tid=0 2024-03-28T22:46:05.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.180+0000 7f8fa258a740 1 client.28639 shutdown 2024-03-28T22:46:05.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.180+0000 7f8fa258a740 1 -- 172.21.15.43:0/3694229309 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55adf5e78e70 msgr2=0x55adf5e851f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:05.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.180+0000 7f8fa258a740 1 --2- 172.21.15.43:0/3694229309 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55adf5e78e70 0x55adf5e851f0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x55adf5e73e10 tx=0x7f8f800045a0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.180+0000 7f8fa258a740 1 -- 172.21.15.43:0/3694229309 shutdown_connections 2024-03-28T22:46:05.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.180+0000 7f8fa258a740 1 --2- 172.21.15.43:0/3694229309 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55adf5ef8910 0x55adf5efad30 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.180+0000 7f8fa258a740 1 --2- 172.21.15.43:0/3694229309 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf5e81a50 0x55adf5e85730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.180+0000 7f8fa258a740 1 --2- 172.21.15.43:0/3694229309 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55adf5e78e70 0x55adf5e851f0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.180+0000 7f8fa258a740 1 --2- 172.21.15.43:0/3694229309 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55adf5e78550 0x55adf5e89e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.180+0000 7f8fa258a740 1 -- 172.21.15.43:0/3694229309 >> 172.21.15.43:0/3694229309 conn(0x55adf5e72c80 msgr2=0x55adf5e71530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:05.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.180+0000 7f8f86a00640 0 client.28639 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:46:05.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.180+0000 7f8fa258a740 1 -- 172.21.15.43:0/3694229309 shutdown_connections 2024-03-28T22:46:05.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.181+0000 7f8fa258a740 1 -- 172.21.15.43:0/3694229309 wait complete. 2024-03-28T22:46:05.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.181+0000 7f8fa258a740 20 client.28639 trim_cache size 0 max 16384 2024-03-28T22:46:05.189 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:46:05.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.181+0000 7f8f9e200640 1 -- 172.21.15.43:0/129667124 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f8f7c0a1ef0 msgr2=0x7f8f7c0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:05.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.181+0000 7f8f9e200640 1 --2- 172.21.15.43:0/129667124 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f8f7c0a1ef0 0x7f8f7c0a43b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8f90012880 tx=0x7f8f90005f90 comp rx=0 tx=0).stop 2024-03-28T22:46:05.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.181+0000 7f8f9e200640 1 -- 172.21.15.43:0/129667124 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f98073b80 msgr2=0x7f8f9819a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:05.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.181+0000 7f8f9e200640 1 --2- 172.21.15.43:0/129667124 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f98073b80 0x7f8f9819a0d0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f8f88005e30 tx=0x7f8f88035d90 comp rx=0 tx=0).stop 2024-03-28T22:46:05.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.181+0000 7f8f9e200640 1 -- 172.21.15.43:0/129667124 shutdown_connections 2024-03-28T22:46:05.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.181+0000 7f8f9e200640 1 --2- 172.21.15.43:0/129667124 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f8f7c0a1ef0 0x7f8f7c0a43b0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.181+0000 7f8f9e200640 1 --2- 172.21.15.43:0/129667124 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8f980744a0 0x7f8f9819a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.181+0000 7f8f9e200640 1 --2- 172.21.15.43:0/129667124 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8f9810ddf0 0x7f8f9819e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.181+0000 7f8f9e200640 1 --2- 172.21.15.43:0/129667124 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8f98073b80 0x7f8f9819a0d0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.181+0000 7f8f9e200640 1 -- 172.21.15.43:0/129667124 >> 172.21.15.43:0/129667124 conn(0x7f8f980fd0c0 msgr2=0x7f8f980fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:05.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.182+0000 7f8f9e200640 1 -- 172.21.15.43:0/129667124 shutdown_connections 2024-03-28T22:46:05.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.182+0000 7f8f9e200640 1 -- 172.21.15.43:0/129667124 wait complete. 2024-03-28T22:46:05.197 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:46:05.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.417+0000 7fd2fe200640 1 -- 172.21.15.43:0/2038776364 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2f8102780 msgr2=0x7fd2f8102b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:05.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.417+0000 7fd2fe200640 1 --2- 172.21.15.43:0/2038776364 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2f8102780 0x7fd2f8102b60 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8009900 tx=0x7fd2e802bcd0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.418+0000 7fd2fe200640 1 -- 172.21.15.43:0/2038776364 shutdown_connections 2024-03-28T22:46:05.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.418+0000 7fd2fe200640 1 --2- 172.21.15.43:0/2038776364 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd2f810bba0 0x7fd2f810e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.418+0000 7fd2fe200640 1 --2- 172.21.15.43:0/2038776364 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd2f81030a0 0x7fd2f810b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.418+0000 7fd2fe200640 1 --2- 172.21.15.43:0/2038776364 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2f8102780 0x7fd2f8102b60 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.418+0000 7fd2fe200640 1 -- 172.21.15.43:0/2038776364 >> 172.21.15.43:0/2038776364 conn(0x7fd2f80fbfa0 msgr2=0x7fd2f80fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:05.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.418+0000 7fd2fe200640 1 -- 172.21.15.43:0/2038776364 shutdown_connections 2024-03-28T22:46:05.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.418+0000 7fd2fe200640 1 -- 172.21.15.43:0/2038776364 wait complete. 2024-03-28T22:46:05.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.419+0000 7fd2fe200640 1 Processor -- start 2024-03-28T22:46:05.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.419+0000 7fd2fe200640 1 -- start start 2024-03-28T22:46:05.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.419+0000 7fd2fe200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2f8102780 0x7fd2f819a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.420+0000 7fd2fe200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd2f81030a0 0x7fd2f819a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.420+0000 7fd2fe200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd2f810bba0 0x7fd2f819e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.420+0000 7fd2fe200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f81119f0 con 0x7fd2f810bba0 2024-03-28T22:46:05.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.420+0000 7fd2fe200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f8111870 con 0x7fd2f81030a0 2024-03-28T22:46:05.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.420+0000 7fd2fe200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f8111b70 con 0x7fd2f8102780 2024-03-28T22:46:05.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.420+0000 7fd2fce00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2f8102780 0x7fd2f819a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.420+0000 7fd2fce00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2f8102780 0x7fd2f819a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:38630/0 (socket says 172.21.15.43:38630) 2024-03-28T22:46:05.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.420+0000 7fd2fce00640 1 -- 172.21.15.43:0/1764876708 learned_addr learned my addr 172.21.15.43:0/1764876708 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:05.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.420+0000 7fd2f7e00640 1 --2- 172.21.15.43:0/1764876708 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd2f81030a0 0x7fd2f819a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.420+0000 7fd2fd800640 1 --2- 172.21.15.43:0/1764876708 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd2f810bba0 0x7fd2f819e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.420+0000 7fd2fce00640 1 -- 172.21.15.43:0/1764876708 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd2f81030a0 msgr2=0x7fd2f819a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:05.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.420+0000 7fd2fce00640 1 --2- 172.21.15.43:0/1764876708 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd2f81030a0 0x7fd2f819a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.420+0000 7fd2fce00640 1 -- 172.21.15.43:0/1764876708 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd2f810bba0 msgr2=0x7fd2f819e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:05.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.420+0000 7fd2fce00640 1 --2- 172.21.15.43:0/1764876708 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd2f810bba0 0x7fd2f819e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.420+0000 7fd2fce00640 1 -- 172.21.15.43:0/1764876708 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2f819f0a0 con 0x7fd2f8102780 2024-03-28T22:46:05.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.421+0000 7fd2fce00640 1 --2- 172.21.15.43:0/1764876708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2f8102780 0x7fd2f819a0b0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8000c00 tx=0x7fd2e80043c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:05.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.421+0000 7fd2f5600640 1 -- 172.21.15.43:0/1764876708 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e8042020 con 0x7fd2f8102780 2024-03-28T22:46:05.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.421+0000 7fd2f5600640 1 -- 172.21.15.43:0/1764876708 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2e80054e0 con 0x7fd2f8102780 2024-03-28T22:46:05.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.421+0000 7fd2fe200640 1 -- 172.21.15.43:0/1764876708 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2f819f330 con 0x7fd2f8102780 2024-03-28T22:46:05.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.421+0000 7fd2f5600640 1 -- 172.21.15.43:0/1764876708 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e8003a80 con 0x7fd2f8102780 2024-03-28T22:46:05.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.421+0000 7fd2fe200640 1 -- 172.21.15.43:0/1764876708 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2f81a6c20 con 0x7fd2f8102780 2024-03-28T22:46:05.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.424+0000 7fd3025a0740 1 Processor -- start 2024-03-28T22:46:05.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.424+0000 7fd2f5600640 1 -- 172.21.15.43:0/1764876708 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fd2e8045020 con 0x7fd2f8102780 2024-03-28T22:46:05.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.424+0000 7fd3025a0740 1 -- start start 2024-03-28T22:46:05.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.425+0000 7fd3025a0740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56096a8e1550 0x56096a8e1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.425+0000 7fd3025a0740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56096a8e1e70 0x56096a8ea510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.425+0000 7fd3025a0740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56096a8eaa50 0x56096a8ece40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.425+0000 7fd3025a0740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56096a845050 con 0x56096a8e1550 2024-03-28T22:46:05.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.425+0000 7fd3025a0740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x56096a7edce0 con 0x56096a8e1e70 2024-03-28T22:46:05.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.425+0000 7fd3025a0740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x56096a8410f0 con 0x56096a8eaa50 2024-03-28T22:46:05.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.425+0000 7fd2fce00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56096a8e1e70 0x56096a8ea510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.425+0000 7fd2f7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56096a8eaa50 0x56096a8ece40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.425+0000 7fd2fd800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56096a8e1550 0x56096a8e1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.425+0000 7fd2f7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56096a8eaa50 0x56096a8ece40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:38632/0 (socket says 172.21.15.43:38632) 2024-03-28T22:46:05.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.425+0000 7fd2f7e00640 1 -- 172.21.15.43:0/2155815925 learned_addr learned my addr 172.21.15.43:0/2155815925 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:05.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.425+0000 7fd2fce00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56096a8e1e70 0x56096a8ea510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34282/0 (socket says 172.21.15.43:34282) 2024-03-28T22:46:05.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.426+0000 7fd2f7e00640 1 -- 172.21.15.43:0/2155815925 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56096a8e1e70 msgr2=0x56096a8ea510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:05.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.426+0000 7fd2f7e00640 1 --2- 172.21.15.43:0/2155815925 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56096a8e1e70 0x56096a8ea510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.426+0000 7fd2f7e00640 1 -- 172.21.15.43:0/2155815925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56096a8e1550 msgr2=0x56096a8e1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:05.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.426+0000 7fd2f7e00640 1 --2- 172.21.15.43:0/2155815925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56096a8e1550 0x56096a8e1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.426+0000 7fd2f7e00640 1 -- 172.21.15.43:0/2155815925 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56096a5593f0 con 0x56096a8eaa50 2024-03-28T22:46:05.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.426+0000 7fd2f7e00640 1 --2- 172.21.15.43:0/2155815925 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56096a8eaa50 0x56096a8ece40 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fd2e00039a0 tx=0x7fd2e0003e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4b0133214bb8d1b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:05.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.426+0000 7fd2ee000640 1 -- 172.21.15.43:0/2155815925 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e0004b20 con 0x56096a8eaa50 2024-03-28T22:46:05.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.426+0000 7fd2ee000640 1 -- 172.21.15.43:0/2155815925 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2e0004cc0 con 0x56096a8eaa50 2024-03-28T22:46:05.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.426+0000 7fd2ee000640 1 -- 172.21.15.43:0/2155815925 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e0012850 con 0x56096a8eaa50 2024-03-28T22:46:05.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.426+0000 7fd2f5600640 1 --2- 172.21.15.43:0/1764876708 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd2dc0a1ef0 0x7fd2dc0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.427+0000 7fd2f5600640 1 -- 172.21.15.43:0/1764876708 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7fd2e80f0190 con 0x7fd2f8102780 2024-03-28T22:46:05.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.427+0000 7fd2fd800640 1 --2- 172.21.15.43:0/1764876708 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd2dc0a1ef0 0x7fd2dc0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.427+0000 7fd3025a0740 1 -- 172.21.15.43:0/2155815925 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56096a8eaa50 msgr2=0x56096a8ece40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:05.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.427+0000 7fd3025a0740 1 --2- 172.21.15.43:0/2155815925 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56096a8eaa50 0x56096a8ece40 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fd2e00039a0 tx=0x7fd2e0003e60 comp rx=0 tx=0).stop 2024-03-28T22:46:05.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.427+0000 7fd3025a0740 1 -- 172.21.15.43:0/2155815925 shutdown_connections 2024-03-28T22:46:05.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.427+0000 7fd3025a0740 1 --2- 172.21.15.43:0/2155815925 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56096a8e1e70 0x56096a8ea510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.427+0000 7fd3025a0740 1 --2- 172.21.15.43:0/2155815925 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56096a8eaa50 0x56096a8ece40 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.427+0000 7fd3025a0740 1 --2- 172.21.15.43:0/2155815925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56096a8e1550 0x56096a8e1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.427+0000 7fd3025a0740 1 -- 172.21.15.43:0/2155815925 >> 172.21.15.43:0/2155815925 conn(0x56096a8dbc80 msgr2=0x56096a8dc060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:05.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.427+0000 7fd3025a0740 1 -- 172.21.15.43:0/2155815925 shutdown_connections 2024-03-28T22:46:05.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.427+0000 7fd2fd800640 1 --2- 172.21.15.43:0/1764876708 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd2dc0a1ef0 0x7fd2dc0a43b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0009d30 tx=0x7fd2f0005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:05.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.427+0000 7fd3025a0740 1 -- 172.21.15.43:0/2155815925 wait complete. 2024-03-28T22:46:05.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.428+0000 7fd3025a0740 1 -- start start 2024-03-28T22:46:05.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.428+0000 7fd3025a0740 1 Processor -- start 2024-03-28T22:46:05.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.429+0000 7fd3025a0740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56096a8e1550 0x56096a8e7a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.429+0000 7fd3025a0740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56096a8e1e70 0x56096a8e7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.429+0000 7fd3025a0740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56096a8eaa50 0x56096a8e8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.429+0000 7fd3025a0740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56096a8ed840 con 0x56096a8e1e70 2024-03-28T22:46:05.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.429+0000 7fd3025a0740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x56096a7e32f0 con 0x56096a8e1550 2024-03-28T22:46:05.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.429+0000 7fd3025a0740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x56096a8ed9c0 con 0x56096a8eaa50 2024-03-28T22:46:05.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.429+0000 7fd3025a0740 10 client.0 _ll_register_callbacks cb 0x56096a558be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:46:05.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.429+0000 7fd2fce00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56096a8e1550 0x56096a8e7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.429+0000 7fd2fce00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56096a8e1550 0x56096a8e7a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34288/0 (socket says 172.21.15.43:34288) 2024-03-28T22:46:05.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.429+0000 7fd2fce00640 1 -- 172.21.15.43:0/2247890044 learned_addr learned my addr 172.21.15.43:0/2247890044 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:05.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.429+0000 7fd2fd800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56096a8eaa50 0x56096a8e8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.429+0000 7fd2f7e00640 1 --2- 172.21.15.43:0/2247890044 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56096a8e1e70 0x56096a8e7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.429+0000 7fd2fce00640 1 -- 172.21.15.43:0/2247890044 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56096a8eaa50 msgr2=0x56096a8e8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:05.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.429+0000 7fd2fce00640 1 --2- 172.21.15.43:0/2247890044 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56096a8eaa50 0x56096a8e8510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.429+0000 7fd2fce00640 1 -- 172.21.15.43:0/2247890044 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56096a8e1e70 msgr2=0x56096a8e7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:05.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.429+0000 7fd2fce00640 1 --2- 172.21.15.43:0/2247890044 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56096a8e1e70 0x56096a8e7fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.429+0000 7fd2fce00640 1 -- 172.21.15.43:0/2247890044 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56096a6237d0 con 0x56096a8e1550 2024-03-28T22:46:05.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.429+0000 7fd2f7e00640 1 --2- 172.21.15.43:0/2247890044 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56096a8e1e70 0x56096a8e7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:46:05.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.429+0000 7fd2fce00640 1 --2- 172.21.15.43:0/2247890044 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56096a8e1550 0x56096a8e7a90 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fd2e80bc530 tx=0x7fd2e80b84d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e880560ce138831 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:05.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.430+0000 7fd2e6a00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:46:05.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.430+0000 7fd2e6a00640 1 -- 172.21.15.43:0/2247890044 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e80bc370 con 0x56096a8e1550 2024-03-28T22:46:05.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.430+0000 7fd2e6a00640 1 -- 172.21.15.43:0/2247890044 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7fd2e80ed700 con 0x56096a8e1550 2024-03-28T22:46:05.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.430+0000 7fd3025a0740 1 -- 172.21.15.43:0/2247890044 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2e0004cc0 con 0x56096a8e1550 2024-03-28T22:46:05.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.430+0000 7fd2e6a00640 1 -- 172.21.15.43:0/2247890044 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd2e80bc190 con 0x56096a8e1550 2024-03-28T22:46:05.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.430+0000 7fd3025a0740 10 client.28709 fetch_fsmap learned FSMap version 32 2024-03-28T22:46:05.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.430+0000 7fd3025a0740 1 -- 172.21.15.43:0/2247890044 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56096a8f9b00 con 0x56096a8e1550 2024-03-28T22:46:05.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.430+0000 7fd2e6a00640 1 -- 172.21.15.43:0/2247890044 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2e0004cc0 con 0x56096a8e1550 2024-03-28T22:46:05.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.430+0000 7fd2e6a00640 1 -- 172.21.15.43:0/2247890044 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd2e80d5140 con 0x56096a8e1550 2024-03-28T22:46:05.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.431+0000 7fd3025a0740 10 client.28709 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:46:05.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.431+0000 7fd3025a0740 10 client.28709 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:46:05.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.431+0000 7fd3025a0740 20 client.28709 populate_metadata read hostname 'smithi043' 2024-03-28T22:46:05.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.431+0000 7fd3025a0740 1 --2- 172.21.15.43:0/2247890044 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x56096a8faf60 0x56096a8fd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.431+0000 7fd3025a0740 4 client.28709 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:46:05.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.431+0000 7fd3025a0740 1 -- 172.21.15.43:0/2247890044 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56096a7ee680 con 0x56096a8faf60 2024-03-28T22:46:05.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.431+0000 7fd2f7e00640 1 --2- 172.21.15.43:0/2247890044 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x56096a8faf60 0x56096a8fd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.431+0000 7fd2f7e00640 1 --2- 172.21.15.43:0/2247890044 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x56096a8faf60 0x56096a8fd3f0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=59fd3974979a9c14 server_cookie=5e55a4a2180225d6 in_seq=0 out_seq=0 2024-03-28T22:46:05.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.431+0000 7fd2e6a00640 10 client.28709 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:46:05.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.432+0000 7fd2e6a00640 1 -- 172.21.15.43:0/2247890044 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56096a7ee680 con 0x56096a8faf60 2024-03-28T22:46:05.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.432+0000 7fd2e6a00640 10 client.28709 handle_command_reply: tid=0 2024-03-28T22:46:05.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.432+0000 7fd3025a0740 1 client.28709 shutdown 2024-03-28T22:46:05.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.432+0000 7fd3025a0740 1 -- 172.21.15.43:0/2247890044 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56096a8e1550 msgr2=0x56096a8e7a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:05.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.432+0000 7fd3025a0740 1 --2- 172.21.15.43:0/2247890044 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56096a8e1550 0x56096a8e7a90 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fd2e80bc530 tx=0x7fd2e80b84d0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.432+0000 7fd3025a0740 1 -- 172.21.15.43:0/2247890044 shutdown_connections 2024-03-28T22:46:05.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.433+0000 7fd3025a0740 1 --2- 172.21.15.43:0/2247890044 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x56096a8faf60 0x56096a8fd3f0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.433+0000 7fd3025a0740 1 --2- 172.21.15.43:0/2247890044 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56096a8eaa50 0x56096a8e8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.433+0000 7fd3025a0740 1 --2- 172.21.15.43:0/2247890044 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56096a8e1e70 0x56096a8e7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.433+0000 7fd3025a0740 1 --2- 172.21.15.43:0/2247890044 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56096a8e1550 0x56096a8e7a90 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.433+0000 7fd3025a0740 1 -- 172.21.15.43:0/2247890044 >> 172.21.15.43:0/2247890044 conn(0x56096a8dbc80 msgr2=0x56096a8dd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:05.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.433+0000 7fd2e6a00640 0 client.28709 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:46:05.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.433+0000 7fd3025a0740 1 -- 172.21.15.43:0/2247890044 shutdown_connections 2024-03-28T22:46:05.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.433+0000 7fd3025a0740 1 -- 172.21.15.43:0/2247890044 wait complete. 2024-03-28T22:46:05.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.433+0000 7fd3025a0740 20 client.28709 trim_cache size 0 max 16384 2024-03-28T22:46:05.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.448+0000 7fd3025a0740 1 Processor -- start 2024-03-28T22:46:05.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.448+0000 7fd3025a0740 1 -- start start 2024-03-28T22:46:05.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.448+0000 7fd3025a0740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56096a8e1550 0x56096a8f65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.448+0000 7fd3025a0740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56096a8e1e70 0x56096a8e8300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.448+0000 7fd3025a0740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56096a8eaa50 0x56096a902cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.448+0000 7fd3025a0740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f81b0d60 con 0x56096a8e1e70 2024-03-28T22:46:05.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.448+0000 7fd3025a0740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f81b28a0 con 0x56096a8eaa50 2024-03-28T22:46:05.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.448+0000 7fd3025a0740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f81b1430 con 0x56096a8e1550 2024-03-28T22:46:05.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.448+0000 7fd2fce00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56096a8e1550 0x56096a8f65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.448+0000 7fd2fce00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56096a8e1550 0x56096a8f65e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:38640/0 (socket says 172.21.15.43:38640) 2024-03-28T22:46:05.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.448+0000 7fd2fce00640 1 -- 172.21.15.43:0/3482246924 learned_addr learned my addr 172.21.15.43:0/3482246924 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:05.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.448+0000 7fd2fd800640 1 --2- 172.21.15.43:0/3482246924 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56096a8eaa50 0x56096a902cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.448+0000 7fd2f7e00640 1 --2- 172.21.15.43:0/3482246924 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56096a8e1e70 0x56096a8e8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.449+0000 7fd2fce00640 1 -- 172.21.15.43:0/3482246924 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56096a8eaa50 msgr2=0x56096a902cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:05.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.449+0000 7fd2fce00640 1 --2- 172.21.15.43:0/3482246924 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56096a8eaa50 0x56096a902cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.449+0000 7fd2fce00640 1 -- 172.21.15.43:0/3482246924 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56096a8e1e70 msgr2=0x56096a8e8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:05.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.449+0000 7fd2fce00640 1 --2- 172.21.15.43:0/3482246924 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56096a8e1e70 0x56096a8e8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.449+0000 7fd2fce00640 1 -- 172.21.15.43:0/3482246924 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56096a8e6a40 con 0x56096a8e1550 2024-03-28T22:46:05.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.449+0000 7fd2fce00640 1 --2- 172.21.15.43:0/3482246924 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56096a8e1550 0x56096a8f65e0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8006010 tx=0x7fd2e8005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc04bc4e45bd683e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:05.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.449+0000 7fd2ed600640 1 -- 172.21.15.43:0/3482246924 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e80bc370 con 0x56096a8e1550 2024-03-28T22:46:05.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.449+0000 7fd2ed600640 1 -- 172.21.15.43:0/3482246924 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2e8004c80 con 0x56096a8e1550 2024-03-28T22:46:05.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.449+0000 7fd2ed600640 1 -- 172.21.15.43:0/3482246924 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e80c4fd0 con 0x56096a8e1550 2024-03-28T22:46:05.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.450+0000 7fd3025a0740 1 -- 172.21.15.43:0/3482246924 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56096a8e1550 msgr2=0x56096a8f65e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:05.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.450+0000 7fd3025a0740 1 --2- 172.21.15.43:0/3482246924 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56096a8e1550 0x56096a8f65e0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8006010 tx=0x7fd2e8005d50 comp rx=0 tx=0).stop 2024-03-28T22:46:05.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.450+0000 7fd3025a0740 1 -- 172.21.15.43:0/3482246924 shutdown_connections 2024-03-28T22:46:05.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.450+0000 7fd3025a0740 1 --2- 172.21.15.43:0/3482246924 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56096a8eaa50 0x56096a902cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.450+0000 7fd3025a0740 1 --2- 172.21.15.43:0/3482246924 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56096a8e1e70 0x56096a8e8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.450+0000 7fd3025a0740 1 --2- 172.21.15.43:0/3482246924 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56096a8e1550 0x56096a8f65e0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.450+0000 7fd3025a0740 1 -- 172.21.15.43:0/3482246924 >> 172.21.15.43:0/3482246924 conn(0x56096a8dbc80 msgr2=0x56096a8dadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:05.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.450+0000 7fd3025a0740 1 -- 172.21.15.43:0/3482246924 shutdown_connections 2024-03-28T22:46:05.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.450+0000 7fd3025a0740 1 -- 172.21.15.43:0/3482246924 wait complete. 2024-03-28T22:46:05.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.450+0000 7fd3025a0740 1 -- start start 2024-03-28T22:46:05.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.450+0000 7fd3025a0740 1 Processor -- start 2024-03-28T22:46:05.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.451+0000 7fd3025a0740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56096a8e1550 0x56096a8ee790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.451+0000 7fd3025a0740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56096a8e1e70 0x56096a8eecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.451+0000 7fd3025a0740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56096a8eaa50 0x56096a8ef210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.451+0000 7fd3025a0740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f81b21d0 con 0x56096a8e1e70 2024-03-28T22:46:05.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.451+0000 7fd3025a0740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f81b1b00 con 0x56096a8eaa50 2024-03-28T22:46:05.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.451+0000 7fd3025a0740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f81aed60 con 0x56096a8e1550 2024-03-28T22:46:05.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.451+0000 7fd3025a0740 10 client.0 _ll_register_callbacks cb 0x7fd2e8005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:46:05.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.451+0000 7fd2fce00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56096a8e1550 0x56096a8ee790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.451+0000 7fd2fd800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56096a8eaa50 0x56096a8ef210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.451+0000 7fd2fce00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56096a8e1550 0x56096a8ee790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:38650/0 (socket says 172.21.15.43:38650) 2024-03-28T22:46:05.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.451+0000 7fd2fce00640 1 -- 172.21.15.43:0/1513691252 learned_addr learned my addr 172.21.15.43:0/1513691252 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:05.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.451+0000 7fd2fd800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56096a8eaa50 0x56096a8ef210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34300/0 (socket says 172.21.15.43:34300) 2024-03-28T22:46:05.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.451+0000 7fd2f7e00640 1 --2- 172.21.15.43:0/1513691252 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56096a8e1e70 0x56096a8eecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.452+0000 7fd2fce00640 1 -- 172.21.15.43:0/1513691252 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56096a8eaa50 msgr2=0x56096a8ef210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:05.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.452+0000 7fd2fce00640 1 --2- 172.21.15.43:0/1513691252 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56096a8eaa50 0x56096a8ef210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.452+0000 7fd2fce00640 1 -- 172.21.15.43:0/1513691252 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56096a8e1e70 msgr2=0x56096a8eecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:05.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.452+0000 7fd2fce00640 1 --2- 172.21.15.43:0/1513691252 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56096a8e1e70 0x56096a8eecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.452+0000 7fd2fce00640 1 -- 172.21.15.43:0/1513691252 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56096a9605a0 con 0x56096a8e1550 2024-03-28T22:46:05.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.452+0000 7fd2fce00640 1 --2- 172.21.15.43:0/1513691252 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56096a8e1550 0x56096a8ee790 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fd2e80b8c90 tx=0x7fd2e8005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3c1de028aa92ffb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:05.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.452+0000 7fd2e6a00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:46:05.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.452+0000 7fd2e6a00640 1 -- 172.21.15.43:0/1513691252 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e80bc370 con 0x56096a8e1550 2024-03-28T22:46:05.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.452+0000 7fd2e6a00640 1 -- 172.21.15.43:0/1513691252 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7fd2e80c4d40 con 0x56096a8e1550 2024-03-28T22:46:05.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.452+0000 7fd3025a0740 1 -- 172.21.15.43:0/1513691252 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2e8004c80 con 0x56096a8e1550 2024-03-28T22:46:05.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.452+0000 7fd2e6a00640 1 -- 172.21.15.43:0/1513691252 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd2e80f3770 con 0x56096a8e1550 2024-03-28T22:46:05.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.452+0000 7fd3025a0740 10 client.28657 fetch_fsmap learned FSMap version 32 2024-03-28T22:46:05.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.452+0000 7fd3025a0740 1 -- 172.21.15.43:0/1513691252 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56096a9605a0 con 0x56096a8e1550 2024-03-28T22:46:05.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.453+0000 7fd2e6a00640 1 -- 172.21.15.43:0/1513691252 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2e8004c80 con 0x56096a8e1550 2024-03-28T22:46:05.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.453+0000 7fd2e6a00640 1 -- 172.21.15.43:0/1513691252 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd2e80bcb90 con 0x56096a8e1550 2024-03-28T22:46:05.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.453+0000 7fd3025a0740 10 client.28657 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:46:05.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.453+0000 7fd3025a0740 10 client.28657 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:46:05.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.453+0000 7fd3025a0740 20 client.28657 populate_metadata read hostname 'smithi043' 2024-03-28T22:46:05.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.453+0000 7fd3025a0740 1 --2- 172.21.15.43:0/1513691252 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x56096a961ae0 0x56096a963ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.453+0000 7fd3025a0740 4 client.28657 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:46:05.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.453+0000 7fd3025a0740 1 -- 172.21.15.43:0/1513691252 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56096a852590 con 0x56096a961ae0 2024-03-28T22:46:05.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.453+0000 7fd2f7e00640 1 --2- 172.21.15.43:0/1513691252 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x56096a961ae0 0x56096a963ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.454+0000 7fd2f7e00640 1 --2- 172.21.15.43:0/1513691252 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x56096a961ae0 0x56096a963ed0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=933f03cfcf2cecb5 server_cookie=ea162eb98456305f in_seq=0 out_seq=0 2024-03-28T22:46:05.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.454+0000 7fd2e6a00640 10 client.28657 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:46:05.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.454+0000 7fd2e6a00640 1 -- 172.21.15.43:0/1513691252 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56096a852590 con 0x56096a961ae0 2024-03-28T22:46:05.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.454+0000 7fd2e6a00640 10 client.28657 handle_command_reply: tid=0 2024-03-28T22:46:05.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.454+0000 7fd3025a0740 1 client.28657 shutdown 2024-03-28T22:46:05.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.454+0000 7fd3025a0740 1 -- 172.21.15.43:0/1513691252 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56096a8e1550 msgr2=0x56096a8ee790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:05.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.454+0000 7fd3025a0740 1 --2- 172.21.15.43:0/1513691252 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56096a8e1550 0x56096a8ee790 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fd2e80b8c90 tx=0x7fd2e8005c00 comp rx=0 tx=0).stop 2024-03-28T22:46:05.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.454+0000 7fd3025a0740 1 -- 172.21.15.43:0/1513691252 shutdown_connections 2024-03-28T22:46:05.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.454+0000 7fd3025a0740 1 --2- 172.21.15.43:0/1513691252 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x56096a961ae0 0x56096a963ed0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.454+0000 7fd3025a0740 1 --2- 172.21.15.43:0/1513691252 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56096a8eaa50 0x56096a8ef210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.454+0000 7fd3025a0740 1 --2- 172.21.15.43:0/1513691252 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56096a8e1e70 0x56096a8eecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.454+0000 7fd3025a0740 1 --2- 172.21.15.43:0/1513691252 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56096a8e1550 0x56096a8ee790 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.454+0000 7fd3025a0740 1 -- 172.21.15.43:0/1513691252 >> 172.21.15.43:0/1513691252 conn(0x56096a8dbc80 msgr2=0x56096a8dce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:05.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.454+0000 7fd2e6a00640 0 client.28657 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:46:05.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.454+0000 7fd3025a0740 1 -- 172.21.15.43:0/1513691252 shutdown_connections 2024-03-28T22:46:05.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.454+0000 7fd3025a0740 1 -- 172.21.15.43:0/1513691252 wait complete. 2024-03-28T22:46:05.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.454+0000 7fd3025a0740 20 client.28657 trim_cache size 0 max 16384 2024-03-28T22:46:05.462 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:46:05.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.455+0000 7fd2fe200640 1 -- 172.21.15.43:0/1764876708 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd2dc0a1ef0 msgr2=0x7fd2dc0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:05.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.455+0000 7fd2fe200640 1 --2- 172.21.15.43:0/1764876708 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd2dc0a1ef0 0x7fd2dc0a43b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0009d30 tx=0x7fd2f0005f90 comp rx=0 tx=0).stop 2024-03-28T22:46:05.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.455+0000 7fd2fe200640 1 -- 172.21.15.43:0/1764876708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2f8102780 msgr2=0x7fd2f819a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:05.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.455+0000 7fd2fe200640 1 --2- 172.21.15.43:0/1764876708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2f8102780 0x7fd2f819a0b0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8000c00 tx=0x7fd2e80043c0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.455+0000 7fd2fe200640 1 -- 172.21.15.43:0/1764876708 shutdown_connections 2024-03-28T22:46:05.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.455+0000 7fd2fe200640 1 --2- 172.21.15.43:0/1764876708 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd2dc0a1ef0 0x7fd2dc0a43b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.455+0000 7fd2fe200640 1 --2- 172.21.15.43:0/1764876708 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd2f81030a0 0x7fd2f819a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.455+0000 7fd2fe200640 1 --2- 172.21.15.43:0/1764876708 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd2f810bba0 0x7fd2f819e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.455+0000 7fd2fe200640 1 --2- 172.21.15.43:0/1764876708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd2f8102780 0x7fd2f819a0b0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.455+0000 7fd2fe200640 1 -- 172.21.15.43:0/1764876708 >> 172.21.15.43:0/1764876708 conn(0x7fd2f80fbfa0 msgr2=0x7fd2f80fe390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:05.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.455+0000 7fd2fe200640 1 -- 172.21.15.43:0/1764876708 shutdown_connections 2024-03-28T22:46:05.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.455+0000 7fd2fe200640 1 -- 172.21.15.43:0/1764876708 wait complete. 2024-03-28T22:46:05.469 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:05 smithi043 ceph-mon[31823]: pgmap v276: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-03-28T22:46:05.469 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:05 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/402101151' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:46:05.473 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:46:05.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:05 smithi172 ceph-mon[36566]: pgmap v276: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-03-28T22:46:05.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:05 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/402101151' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:46:05.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.697+0000 7fd7f8200640 1 -- 172.21.15.43:0/2848024456 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7f0112590 msgr2=0x7fd7f0114980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:05.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.697+0000 7fd7f8200640 1 --2- 172.21.15.43:0/2848024456 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7f0112590 0x7fd7f0114980 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fd7e8009930 tx=0x7fd7e802bc20 comp rx=0 tx=0).stop 2024-03-28T22:46:05.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.697+0000 7fd7f8200640 1 -- 172.21.15.43:0/2848024456 shutdown_connections 2024-03-28T22:46:05.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.697+0000 7fd7f8200640 1 --2- 172.21.15.43:0/2848024456 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7f0112590 0x7fd7f0114980 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.697+0000 7fd7f8200640 1 --2- 172.21.15.43:0/2848024456 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7f00765a0 0x7fd7f0112050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.697+0000 7fd7f8200640 1 --2- 172.21.15.43:0/2848024456 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7f0075c80 0x7fd7f0076060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.697+0000 7fd7f8200640 1 -- 172.21.15.43:0/2848024456 >> 172.21.15.43:0/2848024456 conn(0x7fd7f00ff3a0 msgr2=0x7fd7f01017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:05.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.698+0000 7fd7f8200640 1 -- 172.21.15.43:0/2848024456 shutdown_connections 2024-03-28T22:46:05.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.698+0000 7fd7f8200640 1 -- 172.21.15.43:0/2848024456 wait complete. 2024-03-28T22:46:05.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.698+0000 7fd7f8200640 1 Processor -- start 2024-03-28T22:46:05.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.699+0000 7fd7f8200640 1 -- start start 2024-03-28T22:46:05.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.699+0000 7fd7f8200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7f0075c80 0x7fd7f019e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.699+0000 7fd7f8200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7f00765a0 0x7fd7f019ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.699+0000 7fd7f8200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7f0112590 0x7fd7f01a2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.699+0000 7fd7f8200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f0118120 con 0x7fd7f0112590 2024-03-28T22:46:05.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.699+0000 7fd7f8200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f0117fa0 con 0x7fd7f0075c80 2024-03-28T22:46:05.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.699+0000 7fd7f8200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f01182a0 con 0x7fd7f00765a0 2024-03-28T22:46:05.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.699+0000 7fd7f5800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7f0075c80 0x7fd7f019e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.699+0000 7fd7f5800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7f0075c80 0x7fd7f019e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34318/0 (socket says 172.21.15.43:34318) 2024-03-28T22:46:05.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.699+0000 7fd7f4e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7f00765a0 0x7fd7f019ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.699+0000 7fd7f5800640 1 -- 172.21.15.43:0/4261680978 learned_addr learned my addr 172.21.15.43:0/4261680978 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:05.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.699+0000 7fd7f6200640 1 --2- 172.21.15.43:0/4261680978 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7f0112590 0x7fd7f01a2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.700+0000 7fd7f5800640 1 -- 172.21.15.43:0/4261680978 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7f00765a0 msgr2=0x7fd7f019ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:05.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.700+0000 7fd7f5800640 1 --2- 172.21.15.43:0/4261680978 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7f00765a0 0x7fd7f019ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.700+0000 7fd7f5800640 1 -- 172.21.15.43:0/4261680978 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7f0112590 msgr2=0x7fd7f01a2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:05.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.700+0000 7fd7f5800640 1 --2- 172.21.15.43:0/4261680978 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7f0112590 0x7fd7f01a2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.700+0000 7fd7f5800640 1 -- 172.21.15.43:0/4261680978 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7f01a3670 con 0x7fd7f0075c80 2024-03-28T22:46:05.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.700+0000 7fd7f6200640 1 --2- 172.21.15.43:0/4261680978 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7f0112590 0x7fd7f01a2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:46:05.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.700+0000 7fd7f4e00640 1 --2- 172.21.15.43:0/4261680978 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7f00765a0 0x7fd7f019ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:05.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.700+0000 7fd7f5800640 1 --2- 172.21.15.43:0/4261680978 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7f0075c80 0x7fd7f019e680 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fd7e400eb90 tx=0x7fd7e400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:05.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.700+0000 7fd7ee000640 1 -- 172.21.15.43:0/4261680978 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e4004d20 con 0x7fd7f0075c80 2024-03-28T22:46:05.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.700+0000 7fd7ee000640 1 -- 172.21.15.43:0/4261680978 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7e400ee90 con 0x7fd7f0075c80 2024-03-28T22:46:05.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.700+0000 7fd7f8200640 1 -- 172.21.15.43:0/4261680978 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7f01a3960 con 0x7fd7f0075c80 2024-03-28T22:46:05.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.700+0000 7fd7ee000640 1 -- 172.21.15.43:0/4261680978 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e4010830 con 0x7fd7f0075c80 2024-03-28T22:46:05.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.700+0000 7fd7f8200640 1 -- 172.21.15.43:0/4261680978 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7f01ab1f0 con 0x7fd7f0075c80 2024-03-28T22:46:05.703 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:05 smithi082 ceph-mon[36471]: pgmap v276: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-03-28T22:46:05.703 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:05 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/402101151' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:46:05.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.703+0000 7fd7ee000640 1 -- 172.21.15.43:0/4261680978 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fd7e4005020 con 0x7fd7f0075c80 2024-03-28T22:46:05.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.704+0000 7fd7fb01e740 1 Processor -- start 2024-03-28T22:46:05.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.704+0000 7fd7fb01e740 1 -- start start 2024-03-28T22:46:05.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.704+0000 7fd7fb01e740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f2b5648380 0x55f2b5648760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.704+0000 7fd7fb01e740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2b5648ca0 0x55f2b5651340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.704+0000 7fd7fb01e740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f2b5651880 0x55f2b5653c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.704+0000 7fd7fb01e740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55f2b55ac050 con 0x55f2b5648ca0 2024-03-28T22:46:05.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.704+0000 7fd7fb01e740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55f2b5554ce0 con 0x55f2b5648380 2024-03-28T22:46:05.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.704+0000 7fd7fb01e740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55f2b55a80f0 con 0x55f2b5651880 2024-03-28T22:46:05.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.705+0000 7fd7f4e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f2b5651880 0x55f2b5653c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.705+0000 7fd7f6200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f2b5648380 0x55f2b5648760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.705+0000 7fd7f5800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2b5648ca0 0x55f2b5651340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.705+0000 7fd7f4e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f2b5651880 0x55f2b5653c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:38664/0 (socket says 172.21.15.43:38664) 2024-03-28T22:46:05.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.705+0000 7fd7f4e00640 1 -- 172.21.15.43:0/81893815 learned_addr learned my addr 172.21.15.43:0/81893815 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:05.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.705+0000 7fd7ee000640 1 --2- 172.21.15.43:0/4261680978 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd7d80a1ef0 0x7fd7d80a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.705+0000 7fd7f4e00640 1 -- 172.21.15.43:0/81893815 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f2b5648380 msgr2=0x55f2b5648760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:05.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.705+0000 7fd7f4e00640 1 --2- 172.21.15.43:0/81893815 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f2b5648380 0x55f2b5648760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.705+0000 7fd7f4e00640 1 -- 172.21.15.43:0/81893815 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2b5648ca0 msgr2=0x55f2b5651340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:05.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.705+0000 7fd7f4e00640 1 --2- 172.21.15.43:0/81893815 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2b5648ca0 0x55f2b5651340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.705+0000 7fd7f4e00640 1 -- 172.21.15.43:0/81893815 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2b52c03f0 con 0x55f2b5651880 2024-03-28T22:46:05.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.705+0000 7fd7ee000640 1 -- 172.21.15.43:0/4261680978 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7fd7e40ced10 con 0x7fd7f0075c80 2024-03-28T22:46:05.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.705+0000 7fd7f6200640 1 --2- 172.21.15.43:0/4261680978 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd7d80a1ef0 0x7fd7d80a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.705+0000 7fd7f4e00640 1 --2- 172.21.15.43:0/81893815 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f2b5651880 0x55f2b5653c70 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc0103d0 tx=0x7fd7dc010890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74941444792ea920 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:05.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.706+0000 7fd7e2a00640 1 -- 172.21.15.43:0/81893815 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc005220 con 0x55f2b5651880 2024-03-28T22:46:05.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.706+0000 7fd7e2a00640 1 -- 172.21.15.43:0/81893815 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7dc004c80 con 0x55f2b5651880 2024-03-28T22:46:05.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.706+0000 7fd7e2a00640 1 -- 172.21.15.43:0/81893815 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc011b30 con 0x55f2b5651880 2024-03-28T22:46:05.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.706+0000 7fd7fb01e740 1 -- 172.21.15.43:0/81893815 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f2b5651880 msgr2=0x55f2b5653c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:05.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.706+0000 7fd7f6200640 1 --2- 172.21.15.43:0/4261680978 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd7d80a1ef0 0x7fd7d80a43b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd7e8009230 tx=0x7fd7e8009260 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:05.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.706+0000 7fd7fb01e740 1 --2- 172.21.15.43:0/81893815 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f2b5651880 0x55f2b5653c70 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc0103d0 tx=0x7fd7dc010890 comp rx=0 tx=0).stop 2024-03-28T22:46:05.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.707+0000 7fd7fb01e740 1 -- 172.21.15.43:0/81893815 shutdown_connections 2024-03-28T22:46:05.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.707+0000 7fd7fb01e740 1 --2- 172.21.15.43:0/81893815 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f2b5651880 0x55f2b5653c70 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.707+0000 7fd7fb01e740 1 --2- 172.21.15.43:0/81893815 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2b5648ca0 0x55f2b5651340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.707+0000 7fd7fb01e740 1 --2- 172.21.15.43:0/81893815 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f2b5648380 0x55f2b5648760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.707+0000 7fd7fb01e740 1 -- 172.21.15.43:0/81893815 >> 172.21.15.43:0/81893815 conn(0x55f2b5642ab0 msgr2=0x55f2b5642e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:05.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.707+0000 7fd7fb01e740 1 -- 172.21.15.43:0/81893815 shutdown_connections 2024-03-28T22:46:05.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.707+0000 7fd7fb01e740 1 -- 172.21.15.43:0/81893815 wait complete. 2024-03-28T22:46:05.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.707+0000 7fd7fb01e740 1 -- start start 2024-03-28T22:46:05.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.707+0000 7fd7fb01e740 1 Processor -- start 2024-03-28T22:46:05.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.708+0000 7fd7fb01e740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2b5648380 0x55f2b564e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.708+0000 7fd7fb01e740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f2b5648ca0 0x55f2b564ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.708+0000 7fd7fb01e740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f2b5651880 0x55f2b564f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.708+0000 7fd7fb01e740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55f2b5654670 con 0x55f2b5648380 2024-03-28T22:46:05.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.708+0000 7fd7fb01e740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55f2b554a2f0 con 0x55f2b5648ca0 2024-03-28T22:46:05.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.708+0000 7fd7fb01e740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55f2b56547f0 con 0x55f2b5651880 2024-03-28T22:46:05.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.708+0000 7fd7fb01e740 10 client.0 _ll_register_callbacks cb 0x55f2b52bfbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:46:05.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.708+0000 7fd7f6200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f2b5651880 0x55f2b564f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.708+0000 7fd7f5800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2b5648380 0x55f2b564e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.708+0000 7fd7f4e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f2b5648ca0 0x55f2b564ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.708+0000 7fd7f4e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f2b5648ca0 0x55f2b564ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34346/0 (socket says 172.21.15.43:34346) 2024-03-28T22:46:05.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.708+0000 7fd7f4e00640 1 -- 172.21.15.43:0/1531692985 learned_addr learned my addr 172.21.15.43:0/1531692985 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:05.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.708+0000 7fd7f6200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f2b5651880 0x55f2b564f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:38678/0 (socket says 172.21.15.43:38678) 2024-03-28T22:46:05.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.709+0000 7fd7f4e00640 1 -- 172.21.15.43:0/1531692985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f2b5651880 msgr2=0x55f2b564f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:05.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.709+0000 7fd7f4e00640 1 --2- 172.21.15.43:0/1531692985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f2b5651880 0x55f2b564f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.709+0000 7fd7f4e00640 1 -- 172.21.15.43:0/1531692985 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2b5648380 msgr2=0x55f2b564e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:05.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.709+0000 7fd7f4e00640 1 --2- 172.21.15.43:0/1531692985 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2b5648380 0x55f2b564e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.709+0000 7fd7f4e00640 1 -- 172.21.15.43:0/1531692985 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2b538a7d0 con 0x55f2b5648ca0 2024-03-28T22:46:05.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.709+0000 7fd7f4e00640 1 --2- 172.21.15.43:0/1531692985 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f2b5648ca0 0x55f2b564ed80 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc011530 tx=0x7fd7dc011560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c49b51504f9ce64 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:05.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.709+0000 7fd7d7400640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:46:05.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.709+0000 7fd7d7400640 1 -- 172.21.15.43:0/1531692985 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc004020 con 0x55f2b5648ca0 2024-03-28T22:46:05.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.709+0000 7fd7d7400640 1 -- 172.21.15.43:0/1531692985 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7fd7dc0041c0 con 0x55f2b5648ca0 2024-03-28T22:46:05.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.709+0000 7fd7fb01e740 1 -- 172.21.15.43:0/1531692985 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd7dc004c80 con 0x55f2b5648ca0 2024-03-28T22:46:05.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.709+0000 7fd7d7400640 1 -- 172.21.15.43:0/1531692985 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd7dc0043c0 con 0x55f2b5648ca0 2024-03-28T22:46:05.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.710+0000 7fd7fb01e740 10 client.28727 fetch_fsmap learned FSMap version 32 2024-03-28T22:46:05.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.710+0000 7fd7fb01e740 1 -- 172.21.15.43:0/1531692985 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f2b56608b0 con 0x55f2b5648ca0 2024-03-28T22:46:05.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.710+0000 7fd7d7400640 1 -- 172.21.15.43:0/1531692985 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7dc004c80 con 0x55f2b5648ca0 2024-03-28T22:46:05.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.710+0000 7fd7d7400640 1 -- 172.21.15.43:0/1531692985 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd7dc0035f0 con 0x55f2b5648ca0 2024-03-28T22:46:05.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.710+0000 7fd7fb01e740 10 client.28727 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:46:05.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.710+0000 7fd7fb01e740 10 client.28727 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:46:05.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.710+0000 7fd7fb01e740 20 client.28727 populate_metadata read hostname 'smithi043' 2024-03-28T22:46:05.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.710+0000 7fd7fb01e740 1 --2- 172.21.15.43:0/1531692985 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f2b5661cb0 0x55f2b56640a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.710+0000 7fd7fb01e740 4 client.28727 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:46:05.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.710+0000 7fd7fb01e740 1 -- 172.21.15.43:0/1531692985 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2b5555680 con 0x55f2b5661cb0 2024-03-28T22:46:05.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.710+0000 7fd7f5800640 1 --2- 172.21.15.43:0/1531692985 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f2b5661cb0 0x55f2b56640a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.711+0000 7fd7f5800640 1 --2- 172.21.15.43:0/1531692985 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f2b5661cb0 0x55f2b56640a0 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38583a666f641239 server_cookie=8fdc05df2ca000c in_seq=0 out_seq=0 2024-03-28T22:46:05.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.711+0000 7fd7d7400640 10 client.28727 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:46:05.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.712+0000 7fd7d7400640 1 -- 172.21.15.43:0/1531692985 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f2b5555680 con 0x55f2b5661cb0 2024-03-28T22:46:05.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.712+0000 7fd7d7400640 10 client.28727 handle_command_reply: tid=0 2024-03-28T22:46:05.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.712+0000 7fd7fb01e740 1 client.28727 shutdown 2024-03-28T22:46:05.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.712+0000 7fd7fb01e740 1 -- 172.21.15.43:0/1531692985 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f2b5648ca0 msgr2=0x55f2b564ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:05.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.712+0000 7fd7fb01e740 1 --2- 172.21.15.43:0/1531692985 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f2b5648ca0 0x55f2b564ed80 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc011530 tx=0x7fd7dc011560 comp rx=0 tx=0).stop 2024-03-28T22:46:05.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.712+0000 7fd7fb01e740 1 -- 172.21.15.43:0/1531692985 shutdown_connections 2024-03-28T22:46:05.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.712+0000 7fd7fb01e740 1 --2- 172.21.15.43:0/1531692985 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f2b5661cb0 0x55f2b56640a0 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.712+0000 7fd7fb01e740 1 --2- 172.21.15.43:0/1531692985 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f2b5648ca0 0x55f2b564ed80 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.712+0000 7fd7fb01e740 1 --2- 172.21.15.43:0/1531692985 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f2b5651880 0x55f2b564f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.712+0000 7fd7d7400640 0 client.28727 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:46:05.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.712+0000 7fd7fb01e740 1 --2- 172.21.15.43:0/1531692985 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2b5648380 0x55f2b564e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.712+0000 7fd7fb01e740 1 -- 172.21.15.43:0/1531692985 >> 172.21.15.43:0/1531692985 conn(0x55f2b5642ab0 msgr2=0x55f2b5644800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:05.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.712+0000 7fd7fb01e740 1 -- 172.21.15.43:0/1531692985 shutdown_connections 2024-03-28T22:46:05.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.712+0000 7fd7fb01e740 1 -- 172.21.15.43:0/1531692985 wait complete. 2024-03-28T22:46:05.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.712+0000 7fd7fb01e740 20 client.28727 trim_cache size 0 max 16384 2024-03-28T22:46:05.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.727+0000 7fd7fb01e740 1 Processor -- start 2024-03-28T22:46:05.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.727+0000 7fd7fb01e740 1 -- start start 2024-03-28T22:46:05.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.727+0000 7fd7fb01e740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f2b5648380 0x55f2b5650f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.727+0000 7fd7fb01e740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2b5648ca0 0x55f2b565d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.727+0000 7fd7fb01e740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f2b5651880 0x55f2b56445a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.727+0000 7fd7fb01e740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f01b53b0 con 0x55f2b5648ca0 2024-03-28T22:46:05.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.727+0000 7fd7fb01e740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f01b6ef0 con 0x55f2b5648380 2024-03-28T22:46:05.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.727+0000 7fd7fb01e740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f01b5a80 con 0x55f2b5651880 2024-03-28T22:46:05.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.728+0000 7fd7f5800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f2b5648380 0x55f2b5650f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.728+0000 7fd7f6200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f2b5651880 0x55f2b56445a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.728+0000 7fd7f5800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f2b5648380 0x55f2b5650f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34354/0 (socket says 172.21.15.43:34354) 2024-03-28T22:46:05.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.728+0000 7fd7f4e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2b5648ca0 0x55f2b565d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.728+0000 7fd7f5800640 1 -- 172.21.15.43:0/1834760159 learned_addr learned my addr 172.21.15.43:0/1834760159 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:05.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.728+0000 7fd7f5800640 1 -- 172.21.15.43:0/1834760159 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f2b5651880 msgr2=0x55f2b56445a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:05.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.728+0000 7fd7f5800640 1 --2- 172.21.15.43:0/1834760159 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f2b5651880 0x55f2b56445a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.728+0000 7fd7f5800640 1 -- 172.21.15.43:0/1834760159 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2b5648ca0 msgr2=0x55f2b565d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:05.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.728+0000 7fd7f5800640 1 --2- 172.21.15.43:0/1834760159 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2b5648ca0 0x55f2b565d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.728+0000 7fd7f5800640 1 -- 172.21.15.43:0/1834760159 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2b565d950 con 0x55f2b5648380 2024-03-28T22:46:05.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.728+0000 7fd7f6200640 1 --2- 172.21.15.43:0/1834760159 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f2b5651880 0x55f2b56445a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:05.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.728+0000 7fd7f5800640 1 --2- 172.21.15.43:0/1834760159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f2b5648380 0x55f2b5650f70 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fd7e40cc060 tx=0x7fd7e4097670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b6d2fa9ecf57b94 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:05.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.729+0000 7fd7e2000640 1 -- 172.21.15.43:0/1834760159 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e409ac00 con 0x55f2b5648380 2024-03-28T22:46:05.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.729+0000 7fd7e2000640 1 -- 172.21.15.43:0/1834760159 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7e409b9a0 con 0x55f2b5648380 2024-03-28T22:46:05.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.729+0000 7fd7e2000640 1 -- 172.21.15.43:0/1834760159 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e40d2240 con 0x55f2b5648380 2024-03-28T22:46:05.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.729+0000 7fd7fb01e740 1 -- 172.21.15.43:0/1834760159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f2b5648380 msgr2=0x55f2b5650f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:05.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.729+0000 7fd7fb01e740 1 --2- 172.21.15.43:0/1834760159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f2b5648380 0x55f2b5650f70 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fd7e40cc060 tx=0x7fd7e4097670 comp rx=0 tx=0).stop 2024-03-28T22:46:05.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.729+0000 7fd7fb01e740 1 -- 172.21.15.43:0/1834760159 shutdown_connections 2024-03-28T22:46:05.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.729+0000 7fd7fb01e740 1 --2- 172.21.15.43:0/1834760159 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f2b5651880 0x55f2b56445a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.729+0000 7fd7fb01e740 1 --2- 172.21.15.43:0/1834760159 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2b5648ca0 0x55f2b565d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.729+0000 7fd7fb01e740 1 --2- 172.21.15.43:0/1834760159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f2b5648380 0x55f2b5650f70 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.729+0000 7fd7fb01e740 1 -- 172.21.15.43:0/1834760159 >> 172.21.15.43:0/1834760159 conn(0x55f2b5642ab0 msgr2=0x55f2b5643e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:05.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.729+0000 7fd7fb01e740 1 -- 172.21.15.43:0/1834760159 shutdown_connections 2024-03-28T22:46:05.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.729+0000 7fd7fb01e740 1 -- 172.21.15.43:0/1834760159 wait complete. 2024-03-28T22:46:05.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.730+0000 7fd7fb01e740 1 -- start start 2024-03-28T22:46:05.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.730+0000 7fd7fb01e740 1 Processor -- start 2024-03-28T22:46:05.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.730+0000 7fd7fb01e740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f2b5648380 0x55f2b5659a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.730+0000 7fd7fb01e740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f2b5648ca0 0x55f2b5654d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.731+0000 7fd7fb01e740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2b5651880 0x55f2b56552d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.731+0000 7fd7fb01e740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f01b6820 con 0x55f2b5651880 2024-03-28T22:46:05.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.731+0000 7fd7fb01e740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f01b6150 con 0x55f2b5648ca0 2024-03-28T22:46:05.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.731+0000 7fd7fb01e740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f01b33b0 con 0x55f2b5648380 2024-03-28T22:46:05.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.731+0000 7fd7fb01e740 10 client.0 _ll_register_callbacks cb 0x7fd7dc00be60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:46:05.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.731+0000 7fd7f5800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f2b5648380 0x55f2b5659a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.731+0000 7fd7f4e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f2b5648ca0 0x55f2b5654d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.731+0000 7fd7f5800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f2b5648380 0x55f2b5659a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:38704/0 (socket says 172.21.15.43:38704) 2024-03-28T22:46:05.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.731+0000 7fd7f5800640 1 -- 172.21.15.43:0/1723501953 learned_addr learned my addr 172.21.15.43:0/1723501953 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:05.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.731+0000 7fd7f6200640 1 --2- 172.21.15.43:0/1723501953 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2b5651880 0x55f2b56552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.731+0000 7fd7f4e00640 1 -- 172.21.15.43:0/1723501953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f2b5648380 msgr2=0x55f2b5659a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:05.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.731+0000 7fd7f4e00640 1 --2- 172.21.15.43:0/1723501953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f2b5648380 0x55f2b5659a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.731+0000 7fd7f4e00640 1 -- 172.21.15.43:0/1723501953 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2b5651880 msgr2=0x55f2b56552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:05.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.731+0000 7fd7f4e00640 1 --2- 172.21.15.43:0/1723501953 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2b5651880 0x55f2b56552d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.731+0000 7fd7f4e00640 1 -- 172.21.15.43:0/1723501953 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2b5655810 con 0x55f2b5648ca0 2024-03-28T22:46:05.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.731+0000 7fd7f4e00640 1 --2- 172.21.15.43:0/1723501953 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f2b5648ca0 0x55f2b5654d90 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc003a10 tx=0x7fd7dc003f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4855cf0851c0411b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:05.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.731+0000 7fd7d7400640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:46:05.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.732+0000 7fd7d7400640 1 -- 172.21.15.43:0/1723501953 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc0116a0 con 0x55f2b5648ca0 2024-03-28T22:46:05.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.732+0000 7fd7d7400640 1 -- 172.21.15.43:0/1723501953 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7fd7dc011840 con 0x55f2b5648ca0 2024-03-28T22:46:05.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.732+0000 7fd7fb01e740 1 -- 172.21.15.43:0/1723501953 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd7e409b9a0 con 0x55f2b5648ca0 2024-03-28T22:46:05.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.732+0000 7fd7d7400640 1 -- 172.21.15.43:0/1723501953 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd7dc042050 con 0x55f2b5648ca0 2024-03-28T22:46:05.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.732+0000 7fd7fb01e740 10 client.28733 fetch_fsmap learned FSMap version 32 2024-03-28T22:46:05.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.732+0000 7fd7fb01e740 1 -- 172.21.15.43:0/1723501953 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f2b56c7b40 con 0x55f2b5648ca0 2024-03-28T22:46:05.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.732+0000 7fd7d7400640 1 -- 172.21.15.43:0/1723501953 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7e409b9a0 con 0x55f2b5648ca0 2024-03-28T22:46:05.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.732+0000 7fd7d7400640 1 -- 172.21.15.43:0/1723501953 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd7dc025940 con 0x55f2b5648ca0 2024-03-28T22:46:05.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.732+0000 7fd7fb01e740 10 client.28733 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:46:05.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.732+0000 7fd7fb01e740 10 client.28733 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:46:05.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.732+0000 7fd7fb01e740 20 client.28733 populate_metadata read hostname 'smithi043' 2024-03-28T22:46:05.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.732+0000 7fd7fb01e740 1 --2- 172.21.15.43:0/1723501953 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f2b56c8330 0x55f2b56ca750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:05.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.732+0000 7fd7fb01e740 4 client.28733 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:46:05.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.732+0000 7fd7fb01e740 1 -- 172.21.15.43:0/1723501953 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f2b55b9590 con 0x55f2b56c8330 2024-03-28T22:46:05.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.733+0000 7fd7f5800640 1 --2- 172.21.15.43:0/1723501953 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f2b56c8330 0x55f2b56ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:05.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.733+0000 7fd7f5800640 1 --2- 172.21.15.43:0/1723501953 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f2b56c8330 0x55f2b56ca750 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b76c6ec78214628 server_cookie=d6b04547c26b4b85 in_seq=0 out_seq=0 2024-03-28T22:46:05.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.733+0000 7fd7d7400640 10 client.28733 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:46:05.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.733+0000 7fd7d7400640 1 -- 172.21.15.43:0/1723501953 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f2b55b9590 con 0x55f2b56c8330 2024-03-28T22:46:05.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.733+0000 7fd7d7400640 10 client.28733 handle_command_reply: tid=0 2024-03-28T22:46:05.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.733+0000 7fd7fb01e740 1 client.28733 shutdown 2024-03-28T22:46:05.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.733+0000 7fd7fb01e740 1 -- 172.21.15.43:0/1723501953 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f2b5648ca0 msgr2=0x55f2b5654d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:05.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.733+0000 7fd7fb01e740 1 --2- 172.21.15.43:0/1723501953 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f2b5648ca0 0x55f2b5654d90 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc003a10 tx=0x7fd7dc003f30 comp rx=0 tx=0).stop 2024-03-28T22:46:05.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.734+0000 7fd7fb01e740 1 -- 172.21.15.43:0/1723501953 shutdown_connections 2024-03-28T22:46:05.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.734+0000 7fd7fb01e740 1 --2- 172.21.15.43:0/1723501953 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f2b56c8330 0x55f2b56ca750 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.734+0000 7fd7fb01e740 1 --2- 172.21.15.43:0/1723501953 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f2b5648ca0 0x55f2b5654d90 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.734+0000 7fd7fb01e740 1 --2- 172.21.15.43:0/1723501953 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f2b5651880 0x55f2b56552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.734+0000 7fd7d7400640 0 client.28733 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:46:05.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.734+0000 7fd7fb01e740 1 --2- 172.21.15.43:0/1723501953 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f2b5648380 0x55f2b5659a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.734+0000 7fd7fb01e740 1 -- 172.21.15.43:0/1723501953 >> 172.21.15.43:0/1723501953 conn(0x55f2b5642ab0 msgr2=0x55f2b5641b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:05.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.734+0000 7fd7fb01e740 1 -- 172.21.15.43:0/1723501953 shutdown_connections 2024-03-28T22:46:05.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.734+0000 7fd7fb01e740 1 -- 172.21.15.43:0/1723501953 wait complete. 2024-03-28T22:46:05.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.734+0000 7fd7fb01e740 20 client.28733 trim_cache size 0 max 16384 2024-03-28T22:46:05.743 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:46:05.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.734+0000 7fd7f8200640 1 -- 172.21.15.43:0/4261680978 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd7d80a1ef0 msgr2=0x7fd7d80a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:05.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.734+0000 7fd7f8200640 1 --2- 172.21.15.43:0/4261680978 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd7d80a1ef0 0x7fd7d80a43b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd7e8009230 tx=0x7fd7e8009260 comp rx=0 tx=0).stop 2024-03-28T22:46:05.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.734+0000 7fd7f8200640 1 -- 172.21.15.43:0/4261680978 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7f0075c80 msgr2=0x7fd7f019e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:05.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.734+0000 7fd7f8200640 1 --2- 172.21.15.43:0/4261680978 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7f0075c80 0x7fd7f019e680 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fd7e400eb90 tx=0x7fd7e400c6a0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.735+0000 7fd7f8200640 1 -- 172.21.15.43:0/4261680978 shutdown_connections 2024-03-28T22:46:05.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.735+0000 7fd7f8200640 1 --2- 172.21.15.43:0/4261680978 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fd7d80a1ef0 0x7fd7d80a43b0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.735+0000 7fd7f8200640 1 --2- 172.21.15.43:0/4261680978 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7f0112590 0x7fd7f01a2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.735+0000 7fd7f8200640 1 --2- 172.21.15.43:0/4261680978 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd7f00765a0 0x7fd7f019ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.735+0000 7fd7f8200640 1 --2- 172.21.15.43:0/4261680978 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7f0075c80 0x7fd7f019e680 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:05.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.735+0000 7fd7f8200640 1 -- 172.21.15.43:0/4261680978 >> 172.21.15.43:0/4261680978 conn(0x7fd7f00ff3a0 msgr2=0x7fd7f0101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:05.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.735+0000 7fd7f8200640 1 -- 172.21.15.43:0/4261680978 shutdown_connections 2024-03-28T22:46:05.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:05.735+0000 7fd7f8200640 1 -- 172.21.15.43:0/4261680978 wait complete. 2024-03-28T22:46:06.751 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:46:06.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:46:06.752 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:46:06.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.990+0000 7f93c3e00640 1 -- 172.21.15.43:0/2421065799 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f93c4074410 msgr2=0x7f93c4112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:06.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.990+0000 7f93c3e00640 1 --2- 172.21.15.43:0/2421065799 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f93c4074410 0x7f93c4112090 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f93ac009900 tx=0x7f93ac02bcc0 comp rx=0 tx=0).stop 2024-03-28T22:46:06.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.990+0000 7f93c3e00640 1 -- 172.21.15.43:0/2421065799 shutdown_connections 2024-03-28T22:46:06.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.990+0000 7f93c3e00640 1 --2- 172.21.15.43:0/2421065799 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f93c4074410 0x7f93c4112090 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:06.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.990+0000 7f93c3e00640 1 --2- 172.21.15.43:0/2421065799 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93c41125d0 0x7f93c41149c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:06.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.990+0000 7f93c3e00640 1 --2- 172.21.15.43:0/2421065799 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93c4073af0 0x7f93c4073ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:06.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.990+0000 7f93c3e00640 1 -- 172.21.15.43:0/2421065799 >> 172.21.15.43:0/2421065799 conn(0x7f93c40fd0b0 msgr2=0x7f93c40ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:06.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.990+0000 7f93c3e00640 1 -- 172.21.15.43:0/2421065799 shutdown_connections 2024-03-28T22:46:06.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.991+0000 7f93c3e00640 1 -- 172.21.15.43:0/2421065799 wait complete. 2024-03-28T22:46:06.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.991+0000 7f93c3e00640 1 Processor -- start 2024-03-28T22:46:06.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.992+0000 7f93c3e00640 1 -- start start 2024-03-28T22:46:06.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.992+0000 7f93c3e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93c4073af0 0x7f93c419e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:06.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.992+0000 7f93c3e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f93c4074410 0x7f93c419ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:06.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.992+0000 7f93c3e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93c41125d0 0x7f93c41a2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:06.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.992+0000 7f93c3e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f93c4118020 con 0x7f93c4073af0 2024-03-28T22:46:06.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.992+0000 7f93c3e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f93c4117ea0 con 0x7f93c4074410 2024-03-28T22:46:06.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.992+0000 7f93c3e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f93c41181a0 con 0x7f93c41125d0 2024-03-28T22:46:06.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.992+0000 7f93c2a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93c4073af0 0x7f93c419e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:06.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.992+0000 7f93ba000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f93c4074410 0x7f93c419ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:06.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.992+0000 7f93ba000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f93c4074410 0x7f93c419ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34386/0 (socket says 172.21.15.43:34386) 2024-03-28T22:46:06.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.992+0000 7f93ba000640 1 -- 172.21.15.43:0/4023368709 learned_addr learned my addr 172.21.15.43:0/4023368709 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:06.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.992+0000 7f93c3400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93c41125d0 0x7f93c41a2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:06.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.993+0000 7f93ba000640 1 -- 172.21.15.43:0/4023368709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93c41125d0 msgr2=0x7f93c41a2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:06.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.993+0000 7f93ba000640 1 --2- 172.21.15.43:0/4023368709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93c41125d0 0x7f93c41a2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:06.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.993+0000 7f93ba000640 1 -- 172.21.15.43:0/4023368709 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93c4073af0 msgr2=0x7f93c419e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:06.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.993+0000 7f93ba000640 1 --2- 172.21.15.43:0/4023368709 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93c4073af0 0x7f93c419e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:06.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.993+0000 7f93ba000640 1 -- 172.21.15.43:0/4023368709 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93c41a34f0 con 0x7f93c4074410 2024-03-28T22:46:06.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.993+0000 7f93c2a00640 1 --2- 172.21.15.43:0/4023368709 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93c4073af0 0x7f93c419e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:06.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.993+0000 7f93ba000640 1 --2- 172.21.15.43:0/4023368709 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f93c4074410 0x7f93c419ea40 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f93ac000c00 tx=0x7f93ac004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:06.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.993+0000 7f93bbe00640 1 -- 172.21.15.43:0/4023368709 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93ac042020 con 0x7f93c4074410 2024-03-28T22:46:06.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.993+0000 7f93bbe00640 1 -- 172.21.15.43:0/4023368709 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93ac005000 con 0x7f93c4074410 2024-03-28T22:46:06.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.993+0000 7f93bbe00640 1 -- 172.21.15.43:0/4023368709 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93ac003b20 con 0x7f93c4074410 2024-03-28T22:46:06.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.993+0000 7f93c3e00640 1 -- 172.21.15.43:0/4023368709 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93c41a3780 con 0x7f93c4074410 2024-03-28T22:46:06.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.994+0000 7f93c3e00640 1 -- 172.21.15.43:0/4023368709 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93c41ab070 con 0x7f93c4074410 2024-03-28T22:46:06.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.997+0000 7f93bbe00640 1 -- 172.21.15.43:0/4023368709 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f93ac045020 con 0x7f93c4074410 2024-03-28T22:46:06.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.997+0000 7f93cc3f7740 1 Processor -- start 2024-03-28T22:46:06.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.997+0000 7f93cc3f7740 1 -- start start 2024-03-28T22:46:06.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.997+0000 7f93cc3f7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564442993170 0x564442993550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:06.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.997+0000 7f93cc3f7740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564442993a90 0x56444299c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:06.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.997+0000 7f93cc3f7740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56444299c670 0x56444299ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:06.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.997+0000 7f93cc3f7740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5644428f6c40 con 0x56444299c670 2024-03-28T22:46:06.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.997+0000 7f93cc3f7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x56444289fc40 con 0x564442993170 2024-03-28T22:46:06.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.997+0000 7f93cc3f7740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5644428f2ce0 con 0x564442993a90 2024-03-28T22:46:06.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.998+0000 7f93c2a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564442993a90 0x56444299c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:06.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.998+0000 7f93c3400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564442993170 0x564442993550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:06.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.998+0000 7f93c2a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564442993a90 0x56444299c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:38740/0 (socket says 172.21.15.43:38740) 2024-03-28T22:46:06.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.998+0000 7f93c3400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564442993170 0x564442993550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34388/0 (socket says 172.21.15.43:34388) 2024-03-28T22:46:06.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.998+0000 7f93c2a00640 1 -- 172.21.15.43:0/337115922 learned_addr learned my addr 172.21.15.43:0/337115922 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:06.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.998+0000 7f93ba000640 1 --2- 172.21.15.43:0/337115922 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56444299c670 0x56444299ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:06.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.998+0000 7f93c2a00640 1 -- 172.21.15.43:0/337115922 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564442993170 msgr2=0x564442993550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:06.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.998+0000 7f93c2a00640 1 --2- 172.21.15.43:0/337115922 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564442993170 0x564442993550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:06.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.998+0000 7f93c2a00640 1 -- 172.21.15.43:0/337115922 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56444299c670 msgr2=0x56444299ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:06.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.998+0000 7f93c2a00640 1 --2- 172.21.15.43:0/337115922 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56444299c670 0x56444299ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:06.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.998+0000 7f93c2a00640 1 -- 172.21.15.43:0/337115922 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644426d37e0 con 0x564442993a90 2024-03-28T22:46:07.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.998+0000 7f93bbe00640 1 --2- 172.21.15.43:0/4023368709 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f93a80a1ef0 0x7f93a80a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:07.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.998+0000 7f93c2a00640 1 --2- 172.21.15.43:0/337115922 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564442993a90 0x56444299c130 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f93b4011860 tx=0x7f93b4011d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6eba822172827c02 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:07.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.999+0000 7f93bbe00640 1 -- 172.21.15.43:0/4023368709 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7f93ac0f00b0 con 0x7f93c4074410 2024-03-28T22:46:07.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.999+0000 7f93b3400640 1 -- 172.21.15.43:0/337115922 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b40112f0 con 0x564442993a90 2024-03-28T22:46:07.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.999+0000 7f93c3400640 1 --2- 172.21.15.43:0/4023368709 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f93a80a1ef0 0x7f93a80a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:07.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.999+0000 7f93b3400640 1 -- 172.21.15.43:0/337115922 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93b400f9a0 con 0x564442993a90 2024-03-28T22:46:07.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.999+0000 7f93b3400640 1 -- 172.21.15.43:0/337115922 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b4005770 con 0x564442993a90 2024-03-28T22:46:07.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.999+0000 7f93cc3f7740 1 -- 172.21.15.43:0/337115922 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564442993a90 msgr2=0x56444299c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:07.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:06.999+0000 7f93cc3f7740 1 --2- 172.21.15.43:0/337115922 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564442993a90 0x56444299c130 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f93b4011860 tx=0x7f93b4011d20 comp rx=0 tx=0).stop 2024-03-28T22:46:07.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.000+0000 7f93c3400640 1 --2- 172.21.15.43:0/4023368709 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f93a80a1ef0 0x7f93a80a43b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f93c406ad00 tx=0x7f93bc005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:07.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.000+0000 7f93cc3f7740 1 -- 172.21.15.43:0/337115922 shutdown_connections 2024-03-28T22:46:07.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.000+0000 7f93cc3f7740 1 --2- 172.21.15.43:0/337115922 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56444299c670 0x56444299ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.000+0000 7f93cc3f7740 1 --2- 172.21.15.43:0/337115922 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564442993a90 0x56444299c130 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.000+0000 7f93cc3f7740 1 --2- 172.21.15.43:0/337115922 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564442993170 0x564442993550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.000+0000 7f93cc3f7740 1 -- 172.21.15.43:0/337115922 >> 172.21.15.43:0/337115922 conn(0x56444298d8a0 msgr2=0x56444298dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:07.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.000+0000 7f93cc3f7740 1 -- 172.21.15.43:0/337115922 shutdown_connections 2024-03-28T22:46:07.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.000+0000 7f93cc3f7740 1 -- 172.21.15.43:0/337115922 wait complete. 2024-03-28T22:46:07.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.000+0000 7f93cc3f7740 1 -- start start 2024-03-28T22:46:07.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.000+0000 7f93cc3f7740 1 Processor -- start 2024-03-28T22:46:07.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.001+0000 7f93cc3f7740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564442993170 0x5644429996a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:07.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.001+0000 7f93cc3f7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564442993a90 0x564442999be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:07.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.001+0000 7f93cc3f7740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56444299c670 0x56444299a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:07.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.001+0000 7f93cc3f7740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56444299f460 con 0x56444299c670 2024-03-28T22:46:07.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.001+0000 7f93cc3f7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x564442895250 con 0x564442993a90 2024-03-28T22:46:07.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.001+0000 7f93cc3f7740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x56444299f5e0 con 0x564442993170 2024-03-28T22:46:07.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.001+0000 7f93cc3f7740 10 client.0 _ll_register_callbacks cb 0x5644428b6e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:46:07.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.001+0000 7f93c2a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564442993170 0x5644429996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:07.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.001+0000 7f93c2a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564442993170 0x5644429996a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:38756/0 (socket says 172.21.15.43:38756) 2024-03-28T22:46:07.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.001+0000 7f93c2a00640 1 -- 172.21.15.43:0/2371424035 learned_addr learned my addr 172.21.15.43:0/2371424035 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:07.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.001+0000 7f93ba000640 1 --2- 172.21.15.43:0/2371424035 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564442993a90 0x564442999be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:07.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.001+0000 7f93c3400640 1 --2- 172.21.15.43:0/2371424035 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56444299c670 0x56444299a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:07.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.002+0000 7f93c2a00640 1 -- 172.21.15.43:0/2371424035 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564442993a90 msgr2=0x564442999be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:07.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.002+0000 7f93c2a00640 1 --2- 172.21.15.43:0/2371424035 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564442993a90 0x564442999be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.002+0000 7f93c2a00640 1 -- 172.21.15.43:0/2371424035 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56444299c670 msgr2=0x56444299a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:07.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.002+0000 7f93c2a00640 1 --2- 172.21.15.43:0/2371424035 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56444299c670 0x56444299a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.002+0000 7f93c2a00640 1 -- 172.21.15.43:0/2371424035 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564442733f70 con 0x564442993170 2024-03-28T22:46:07.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.002+0000 7f93ba000640 1 --2- 172.21.15.43:0/2371424035 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564442993a90 0x564442999be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:07.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.002+0000 7f93c2a00640 1 --2- 172.21.15.43:0/2371424035 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564442993170 0x5644429996a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f93b400fdb0 tx=0x7f93b4011420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=299f0e976c726c49 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:07.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.002+0000 7f93a7e00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:46:07.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.002+0000 7f93a7e00640 1 -- 172.21.15.43:0/2371424035 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b4028030 con 0x564442993170 2024-03-28T22:46:07.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.002+0000 7f93a7e00640 1 -- 172.21.15.43:0/2371424035 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7f93b4005680 con 0x564442993170 2024-03-28T22:46:07.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.002+0000 7f93cc3f7740 1 -- 172.21.15.43:0/2371424035 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93b400f9a0 con 0x564442993170 2024-03-28T22:46:07.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.003+0000 7f93a7e00640 1 -- 172.21.15.43:0/2371424035 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f93b4032050 con 0x564442993170 2024-03-28T22:46:07.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.003+0000 7f93cc3f7740 10 client.28687 fetch_fsmap learned FSMap version 32 2024-03-28T22:46:07.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.003+0000 7f93cc3f7740 1 -- 172.21.15.43:0/2371424035 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5644425cb170 con 0x564442993170 2024-03-28T22:46:07.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.003+0000 7f93a7e00640 1 -- 172.21.15.43:0/2371424035 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93b400f9a0 con 0x564442993170 2024-03-28T22:46:07.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.003+0000 7f93a7e00640 1 -- 172.21.15.43:0/2371424035 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f93b4018070 con 0x564442993170 2024-03-28T22:46:07.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.003+0000 7f93cc3f7740 10 client.28687 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:46:07.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.003+0000 7f93cc3f7740 10 client.28687 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:46:07.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.003+0000 7f93cc3f7740 20 client.28687 populate_metadata read hostname 'smithi043' 2024-03-28T22:46:07.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.003+0000 7f93cc3f7740 1 --2- 172.21.15.43:0/2371424035 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5644429ac9f0 0x5644429aede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:07.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.003+0000 7f93cc3f7740 4 client.28687 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:46:07.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.003+0000 7f93cc3f7740 1 -- 172.21.15.43:0/2371424035 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564442904180 con 0x5644429ac9f0 2024-03-28T22:46:07.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.003+0000 7f93ba000640 1 --2- 172.21.15.43:0/2371424035 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5644429ac9f0 0x5644429aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:07.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.004+0000 7f93ba000640 1 --2- 172.21.15.43:0/2371424035 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5644429ac9f0 0x5644429aede0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=283eeda955ceda6e server_cookie=b63dc026473006c1 in_seq=0 out_seq=0 2024-03-28T22:46:07.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.004+0000 7f93a7e00640 10 client.28687 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:46:07.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.005+0000 7f93a7e00640 1 -- 172.21.15.43:0/2371424035 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564442904180 con 0x5644429ac9f0 2024-03-28T22:46:07.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.005+0000 7f93a7e00640 10 client.28687 handle_command_reply: tid=0 2024-03-28T22:46:07.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.005+0000 7f93cc3f7740 1 client.28687 shutdown 2024-03-28T22:46:07.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.005+0000 7f93cc3f7740 1 -- 172.21.15.43:0/2371424035 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564442993170 msgr2=0x5644429996a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:07.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.005+0000 7f93cc3f7740 1 --2- 172.21.15.43:0/2371424035 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564442993170 0x5644429996a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f93b400fdb0 tx=0x7f93b4011420 comp rx=0 tx=0).stop 2024-03-28T22:46:07.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.005+0000 7f93cc3f7740 1 -- 172.21.15.43:0/2371424035 shutdown_connections 2024-03-28T22:46:07.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.005+0000 7f93cc3f7740 1 --2- 172.21.15.43:0/2371424035 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5644429ac9f0 0x5644429aede0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.005+0000 7f93cc3f7740 1 --2- 172.21.15.43:0/2371424035 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564442993a90 0x564442999be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.005+0000 7f93cc3f7740 1 --2- 172.21.15.43:0/2371424035 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56444299c670 0x56444299a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.005+0000 7f93a7e00640 0 client.28687 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:46:07.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.005+0000 7f93cc3f7740 1 --2- 172.21.15.43:0/2371424035 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564442993170 0x5644429996a0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.005+0000 7f93cc3f7740 1 -- 172.21.15.43:0/2371424035 >> 172.21.15.43:0/2371424035 conn(0x56444298d8a0 msgr2=0x56444298f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:07.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.005+0000 7f93cc3f7740 1 -- 172.21.15.43:0/2371424035 shutdown_connections 2024-03-28T22:46:07.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.005+0000 7f93cc3f7740 1 -- 172.21.15.43:0/2371424035 wait complete. 2024-03-28T22:46:07.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.005+0000 7f93cc3f7740 20 client.28687 trim_cache size 0 max 16384 2024-03-28T22:46:07.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.025+0000 7f93cc3f7740 1 Processor -- start 2024-03-28T22:46:07.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.025+0000 7f93cc3f7740 1 -- start start 2024-03-28T22:46:07.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.025+0000 7f93cc3f7740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564442993170 0x56444299bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:07.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.025+0000 7f93cc3f7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564442993a90 0x5644429a8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:07.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.025+0000 7f93cc3f7740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56444299c670 0x5644429b45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:07.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.025+0000 7f93cc3f7740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f93c41b5190 con 0x564442993170 2024-03-28T22:46:07.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.025+0000 7f93cc3f7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f93c41b6cd0 con 0x564442993a90 2024-03-28T22:46:07.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.025+0000 7f93cc3f7740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f93c41b5860 con 0x56444299c670 2024-03-28T22:46:07.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.026+0000 7f93c3400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56444299c670 0x5644429b45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:07.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.026+0000 7f93c2a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564442993170 0x56444299bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:07.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.026+0000 7f93c3400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56444299c670 0x5644429b45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:38762/0 (socket says 172.21.15.43:38762) 2024-03-28T22:46:07.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.026+0000 7f93c2a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564442993170 0x56444299bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34598/0 (socket says 172.21.15.43:34598) 2024-03-28T22:46:07.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.026+0000 7f93c3400640 1 -- 172.21.15.43:0/1632956849 learned_addr learned my addr 172.21.15.43:0/1632956849 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:07.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.026+0000 7f93ba000640 1 --2- 172.21.15.43:0/1632956849 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564442993a90 0x5644429a8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:07.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.026+0000 7f93c3400640 1 -- 172.21.15.43:0/1632956849 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564442993a90 msgr2=0x5644429a8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:07.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.026+0000 7f93c3400640 1 --2- 172.21.15.43:0/1632956849 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564442993a90 0x5644429a8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.026+0000 7f93c3400640 1 -- 172.21.15.43:0/1632956849 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564442993170 msgr2=0x56444299bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:07.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.026+0000 7f93c3400640 1 --2- 172.21.15.43:0/1632956849 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564442993170 0x56444299bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.026+0000 7f93c3400640 1 -- 172.21.15.43:0/1632956849 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644429af150 con 0x56444299c670 2024-03-28T22:46:07.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.026+0000 7f93c2a00640 1 --2- 172.21.15.43:0/1632956849 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564442993170 0x56444299bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:07.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.026+0000 7f93ba000640 1 --2- 172.21.15.43:0/1632956849 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564442993a90 0x5644429a8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:07.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.027+0000 7f93c3400640 1 --2- 172.21.15.43:0/1632956849 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56444299c670 0x5644429b45a0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f93bc009b00 tx=0x7f93bc01b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=508347c7f3af70ff server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:07.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.027+0000 7f93b2a00640 1 -- 172.21.15.43:0/1632956849 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93bc02e070 con 0x56444299c670 2024-03-28T22:46:07.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.027+0000 7f93b2a00640 1 -- 172.21.15.43:0/1632956849 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93bc004410 con 0x56444299c670 2024-03-28T22:46:07.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.027+0000 7f93b2a00640 1 -- 172.21.15.43:0/1632956849 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93bc029890 con 0x56444299c670 2024-03-28T22:46:07.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.027+0000 7f93cc3f7740 1 -- 172.21.15.43:0/1632956849 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56444299c670 msgr2=0x5644429b45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:07.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.027+0000 7f93cc3f7740 1 --2- 172.21.15.43:0/1632956849 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56444299c670 0x5644429b45a0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f93bc009b00 tx=0x7f93bc01b000 comp rx=0 tx=0).stop 2024-03-28T22:46:07.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.027+0000 7f93cc3f7740 1 -- 172.21.15.43:0/1632956849 shutdown_connections 2024-03-28T22:46:07.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.028+0000 7f93cc3f7740 1 --2- 172.21.15.43:0/1632956849 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564442993a90 0x5644429a8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.028+0000 7f93cc3f7740 1 --2- 172.21.15.43:0/1632956849 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56444299c670 0x5644429b45a0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.028+0000 7f93cc3f7740 1 --2- 172.21.15.43:0/1632956849 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564442993170 0x56444299bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.028+0000 7f93cc3f7740 1 -- 172.21.15.43:0/1632956849 >> 172.21.15.43:0/1632956849 conn(0x56444298d8a0 msgr2=0x56444298c900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:07.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.028+0000 7f93cc3f7740 1 -- 172.21.15.43:0/1632956849 shutdown_connections 2024-03-28T22:46:07.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.028+0000 7f93cc3f7740 1 -- 172.21.15.43:0/1632956849 wait complete. 2024-03-28T22:46:07.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.028+0000 7f93cc3f7740 1 -- start start 2024-03-28T22:46:07.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.028+0000 7f93cc3f7740 1 Processor -- start 2024-03-28T22:46:07.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.029+0000 7f93cc3f7740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564442993170 0x5644429a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:07.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.029+0000 7f93cc3f7740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564442993a90 0x56444299fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:07.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.029+0000 7f93cc3f7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56444299c670 0x5644429a0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:07.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.029+0000 7f93cc3f7740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f93c41b6600 con 0x564442993170 2024-03-28T22:46:07.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.029+0000 7f93cc3f7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f93c41b5f30 con 0x56444299c670 2024-03-28T22:46:07.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.029+0000 7f93cc3f7740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f93c41b3190 con 0x564442993a90 2024-03-28T22:46:07.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.029+0000 7f93cc3f7740 10 client.0 _ll_register_callbacks cb 0x7f93bc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:46:07.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.029+0000 7f93c2a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564442993170 0x5644429a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:07.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.029+0000 7f93c3400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56444299c670 0x5644429a0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:07.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.029+0000 7f93c2a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564442993170 0x5644429a49e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34606/0 (socket says 172.21.15.43:34606) 2024-03-28T22:46:07.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.029+0000 7f93c2a00640 1 -- 172.21.15.43:0/1042816547 learned_addr learned my addr 172.21.15.43:0/1042816547 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:07.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.029+0000 7f93ba000640 1 --2- 172.21.15.43:0/1042816547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564442993a90 0x56444299fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:07.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.029+0000 7f93c2a00640 1 -- 172.21.15.43:0/1042816547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564442993a90 msgr2=0x56444299fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:07.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.029+0000 7f93c2a00640 1 --2- 172.21.15.43:0/1042816547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564442993a90 0x56444299fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.029+0000 7f93c2a00640 1 -- 172.21.15.43:0/1042816547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56444299c670 msgr2=0x5644429a0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:07.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.029+0000 7f93c2a00640 1 --2- 172.21.15.43:0/1042816547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56444299c670 0x5644429a0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.029+0000 7f93c2a00640 1 -- 172.21.15.43:0/1042816547 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644429a0a50 con 0x564442993170 2024-03-28T22:46:07.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.029+0000 7f93c3400640 1 --2- 172.21.15.43:0/1042816547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56444299c670 0x5644429a0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:07.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.030+0000 7f93c2a00640 1 --2- 172.21.15.43:0/1042816547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564442993170 0x5644429a49e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f93b4006010 tx=0x7f93b4009480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46ff30129b5eac45 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:07.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.030+0000 7f93a7e00640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:46:07.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.030+0000 7f93a7e00640 1 -- 172.21.15.43:0/1042816547 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b4028040 con 0x564442993170 2024-03-28T22:46:07.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.030+0000 7f93a7e00640 1 -- 172.21.15.43:0/1042816547 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7f93b40050d0 con 0x564442993170 2024-03-28T22:46:07.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.030+0000 7f93cc3f7740 1 -- 172.21.15.43:0/1042816547 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93bc004410 con 0x564442993170 2024-03-28T22:46:07.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.030+0000 7f93a7e00640 1 -- 172.21.15.43:0/1042816547 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f93b40052d0 con 0x564442993170 2024-03-28T22:46:07.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.030+0000 7f93cc3f7740 10 client.18492 fetch_fsmap learned FSMap version 32 2024-03-28T22:46:07.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.030+0000 7f93cc3f7740 1 -- 172.21.15.43:0/1042816547 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564442a12b00 con 0x564442993170 2024-03-28T22:46:07.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.031+0000 7f93a7e00640 1 -- 172.21.15.43:0/1042816547 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93bc004410 con 0x564442993170 2024-03-28T22:46:07.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.031+0000 7f93a7e00640 1 -- 172.21.15.43:0/1042816547 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f93b4018040 con 0x564442993170 2024-03-28T22:46:07.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.031+0000 7f93cc3f7740 10 client.18492 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:46:07.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.031+0000 7f93cc3f7740 10 client.18492 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:46:07.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.031+0000 7f93cc3f7740 20 client.18492 populate_metadata read hostname 'smithi043' 2024-03-28T22:46:07.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.031+0000 7f93cc3f7740 1 --2- 172.21.15.43:0/1042816547 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x564442a13160 0x564442a15580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:07.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.031+0000 7f93cc3f7740 4 client.18492 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:46:07.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.031+0000 7f93cc3f7740 1 -- 172.21.15.43:0/1042816547 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5644428fe4d0 con 0x564442a13160 2024-03-28T22:46:07.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.031+0000 7f93ba000640 1 --2- 172.21.15.43:0/1042816547 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x564442a13160 0x564442a15580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:07.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.032+0000 7f93ba000640 1 --2- 172.21.15.43:0/1042816547 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x564442a13160 0x564442a15580 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b534d51e6a7aa29 server_cookie=cf89c4817d64774e in_seq=0 out_seq=0 2024-03-28T22:46:07.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.032+0000 7f93a7e00640 10 client.18492 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:46:07.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.033+0000 7f93a7e00640 1 -- 172.21.15.43:0/1042816547 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5644428fe4d0 con 0x564442a13160 2024-03-28T22:46:07.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.033+0000 7f93a7e00640 10 client.18492 handle_command_reply: tid=0 2024-03-28T22:46:07.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.033+0000 7f93cc3f7740 1 client.18492 shutdown 2024-03-28T22:46:07.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.033+0000 7f93cc3f7740 1 -- 172.21.15.43:0/1042816547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564442993170 msgr2=0x5644429a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:07.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.033+0000 7f93cc3f7740 1 --2- 172.21.15.43:0/1042816547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564442993170 0x5644429a49e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f93b4006010 tx=0x7f93b4009480 comp rx=0 tx=0).stop 2024-03-28T22:46:07.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.033+0000 7f93cc3f7740 1 -- 172.21.15.43:0/1042816547 shutdown_connections 2024-03-28T22:46:07.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.033+0000 7f93cc3f7740 1 --2- 172.21.15.43:0/1042816547 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x564442a13160 0x564442a15580 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.033+0000 7f93cc3f7740 1 --2- 172.21.15.43:0/1042816547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56444299c670 0x5644429a0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.033+0000 7f93cc3f7740 1 --2- 172.21.15.43:0/1042816547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564442993a90 0x56444299fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.033+0000 7f93cc3f7740 1 --2- 172.21.15.43:0/1042816547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564442993170 0x5644429a49e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.033+0000 7f93cc3f7740 1 -- 172.21.15.43:0/1042816547 >> 172.21.15.43:0/1042816547 conn(0x56444298d8a0 msgr2=0x564442999640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:07.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.033+0000 7f93a7e00640 0 client.18492 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:46:07.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.033+0000 7f93cc3f7740 1 -- 172.21.15.43:0/1042816547 shutdown_connections 2024-03-28T22:46:07.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.033+0000 7f93cc3f7740 1 -- 172.21.15.43:0/1042816547 wait complete. 2024-03-28T22:46:07.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.033+0000 7f93cc3f7740 20 client.18492 trim_cache size 0 max 16384 2024-03-28T22:46:07.039 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:46:07.039 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-03-28T22:46:07.039 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "9fa76a05-e3b0-47c7-a9b0-01be8ba103a3", 2024-03-28T22:46:07.039 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-03-28T22:46:07.040 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:46:07.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.034+0000 7f93c3e00640 1 -- 172.21.15.43:0/4023368709 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f93a80a1ef0 msgr2=0x7f93a80a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:07.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.034+0000 7f93c3e00640 1 --2- 172.21.15.43:0/4023368709 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f93a80a1ef0 0x7f93a80a43b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f93c406ad00 tx=0x7f93bc005f90 comp rx=0 tx=0).stop 2024-03-28T22:46:07.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.034+0000 7f93c3e00640 1 -- 172.21.15.43:0/4023368709 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f93c4074410 msgr2=0x7f93c419ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:07.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.034+0000 7f93c3e00640 1 --2- 172.21.15.43:0/4023368709 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f93c4074410 0x7f93c419ea40 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f93ac000c00 tx=0x7f93ac004360 comp rx=0 tx=0).stop 2024-03-28T22:46:07.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.034+0000 7f93c3e00640 1 -- 172.21.15.43:0/4023368709 shutdown_connections 2024-03-28T22:46:07.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.034+0000 7f93c3e00640 1 --2- 172.21.15.43:0/4023368709 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f93a80a1ef0 0x7f93a80a43b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.034+0000 7f93c3e00640 1 --2- 172.21.15.43:0/4023368709 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f93c4074410 0x7f93c419ea40 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.034+0000 7f93c3e00640 1 --2- 172.21.15.43:0/4023368709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f93c41125d0 0x7f93c41a2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.034+0000 7f93c3e00640 1 --2- 172.21.15.43:0/4023368709 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93c4073af0 0x7f93c419e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.034+0000 7f93c3e00640 1 -- 172.21.15.43:0/4023368709 >> 172.21.15.43:0/4023368709 conn(0x7f93c40fd0b0 msgr2=0x7f93c40fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:07.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.034+0000 7f93c3e00640 1 -- 172.21.15.43:0/4023368709 shutdown_connections 2024-03-28T22:46:07.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.034+0000 7f93c3e00640 1 -- 172.21.15.43:0/4023368709 wait complete. 2024-03-28T22:46:07.054 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:46:07.054 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:46:07.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.272+0000 7f8d72200640 1 -- 172.21.15.43:0/594238162 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8d6c1125d0 msgr2=0x7f8d6c1149c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:07.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.272+0000 7f8d72200640 1 --2- 172.21.15.43:0/594238162 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8d6c1125d0 0x7f8d6c1149c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f8d64009ca0 tx=0x7f8d6402c180 comp rx=0 tx=0).stop 2024-03-28T22:46:07.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.272+0000 7f8d72200640 1 -- 172.21.15.43:0/594238162 shutdown_connections 2024-03-28T22:46:07.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.272+0000 7f8d72200640 1 --2- 172.21.15.43:0/594238162 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8d6c1125d0 0x7f8d6c1149c0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.272+0000 7f8d72200640 1 --2- 172.21.15.43:0/594238162 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d6c074410 0x7f8d6c112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.272+0000 7f8d72200640 1 --2- 172.21.15.43:0/594238162 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8d6c073af0 0x7f8d6c073ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.272+0000 7f8d72200640 1 -- 172.21.15.43:0/594238162 >> 172.21.15.43:0/594238162 conn(0x7f8d6c0fd0b0 msgr2=0x7f8d6c0ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:07.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.272+0000 7f8d72200640 1 -- 172.21.15.43:0/594238162 shutdown_connections 2024-03-28T22:46:07.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.272+0000 7f8d72200640 1 -- 172.21.15.43:0/594238162 wait complete. 2024-03-28T22:46:07.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.273+0000 7f8d72200640 1 Processor -- start 2024-03-28T22:46:07.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.274+0000 7f8d72200640 1 -- start start 2024-03-28T22:46:07.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.274+0000 7f8d72200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d6c073af0 0x7f8d6c19e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:07.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.274+0000 7f8d72200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8d6c074410 0x7f8d6c19e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:07.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.274+0000 7f8d72200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8d6c1125d0 0x7f8d6c1a2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:07.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.274+0000 7f8d72200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8d6c117fd0 con 0x7f8d6c073af0 2024-03-28T22:46:07.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.274+0000 7f8d72200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8d6c117e50 con 0x7f8d6c1125d0 2024-03-28T22:46:07.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.274+0000 7f8d72200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f8d6c118150 con 0x7f8d6c074410 2024-03-28T22:46:07.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.274+0000 7f8d70e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d6c073af0 0x7f8d6c19e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:07.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.274+0000 7f8d70e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d6c073af0 0x7f8d6c19e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34628/0 (socket says 172.21.15.43:34628) 2024-03-28T22:46:07.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.274+0000 7f8d70e00640 1 -- 172.21.15.43:0/720101445 learned_addr learned my addr 172.21.15.43:0/720101445 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:07.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.274+0000 7f8d71800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8d6c1125d0 0x7f8d6c1a2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:07.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.274+0000 7f8d6be00640 1 --2- 172.21.15.43:0/720101445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8d6c074410 0x7f8d6c19e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:07.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.275+0000 7f8d70e00640 1 -- 172.21.15.43:0/720101445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8d6c074410 msgr2=0x7f8d6c19e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:07.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.275+0000 7f8d70e00640 1 --2- 172.21.15.43:0/720101445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8d6c074410 0x7f8d6c19e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.275+0000 7f8d70e00640 1 -- 172.21.15.43:0/720101445 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8d6c1125d0 msgr2=0x7f8d6c1a2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:07.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.275+0000 7f8d70e00640 1 --2- 172.21.15.43:0/720101445 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8d6c1125d0 0x7f8d6c1a2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.275+0000 7f8d70e00640 1 -- 172.21.15.43:0/720101445 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d6c1a3460 con 0x7f8d6c073af0 2024-03-28T22:46:07.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.275+0000 7f8d71800640 1 --2- 172.21.15.43:0/720101445 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8d6c1125d0 0x7f8d6c1a2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:07.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.275+0000 7f8d70e00640 1 --2- 172.21.15.43:0/720101445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d6c073af0 0x7f8d6c19e470 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8d6000e9f0 tx=0x7f8d6000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:07.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.275+0000 7f8d69600640 1 -- 172.21.15.43:0/720101445 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d6000cd80 con 0x7f8d6c073af0 2024-03-28T22:46:07.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.276+0000 7f8d72200640 1 -- 172.21.15.43:0/720101445 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d6c1a3750 con 0x7f8d6c073af0 2024-03-28T22:46:07.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.276+0000 7f8d69600640 1 -- 172.21.15.43:0/720101445 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d60010910 con 0x7f8d6c073af0 2024-03-28T22:46:07.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.276+0000 7f8d69600640 1 -- 172.21.15.43:0/720101445 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d600188f0 con 0x7f8d6c073af0 2024-03-28T22:46:07.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.276+0000 7f8d72200640 1 -- 172.21.15.43:0/720101445 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d6c1aafe0 con 0x7f8d6c073af0 2024-03-28T22:46:07.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.279+0000 7f8d69600640 1 -- 172.21.15.43:0/720101445 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f8d60010ac0 con 0x7f8d6c073af0 2024-03-28T22:46:07.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.279+0000 7f8d7641b740 1 Processor -- start 2024-03-28T22:46:07.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.279+0000 7f8d7641b740 1 -- start start 2024-03-28T22:46:07.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.280+0000 7f8d7641b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564f319aadb0 0x564f319a9b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:07.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.280+0000 7f8d7641b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564f319aa0d0 0x564f319aa550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:07.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.280+0000 7f8d7641b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564f319b3440 0x564f319b5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:07.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.280+0000 7f8d7641b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x564f3190e100 con 0x564f319aadb0 2024-03-28T22:46:07.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.280+0000 7f8d7641b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x564f318b6ba0 con 0x564f319aa0d0 2024-03-28T22:46:07.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.280+0000 7f8d7641b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x564f3190a1a0 con 0x564f319b3440 2024-03-28T22:46:07.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.280+0000 7f8d70e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564f319aa0d0 0x564f319aa550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:07.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.280+0000 7f8d70e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564f319aa0d0 0x564f319aa550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34442/0 (socket says 172.21.15.43:34442) 2024-03-28T22:46:07.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.280+0000 7f8d6be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564f319b3440 0x564f319b5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:07.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.280+0000 7f8d70e00640 1 -- 172.21.15.43:0/2503542863 learned_addr learned my addr 172.21.15.43:0/2503542863 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:07.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.280+0000 7f8d71800640 1 --2- 172.21.15.43:0/2503542863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564f319aadb0 0x564f319a9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:07.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.280+0000 7f8d6be00640 1 -- 172.21.15.43:0/2503542863 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564f319aa0d0 msgr2=0x564f319aa550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:07.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.280+0000 7f8d6be00640 1 --2- 172.21.15.43:0/2503542863 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564f319aa0d0 0x564f319aa550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.280+0000 7f8d6be00640 1 -- 172.21.15.43:0/2503542863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564f319aadb0 msgr2=0x564f319a9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:07.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.280+0000 7f8d6be00640 1 --2- 172.21.15.43:0/2503542863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564f319aadb0 0x564f319a9b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.280+0000 7f8d6be00640 1 -- 172.21.15.43:0/2503542863 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f318e71a0 con 0x564f319b3440 2024-03-28T22:46:07.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.281+0000 7f8d69600640 1 --2- 172.21.15.43:0/720101445 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f8d540a1f70 0x7f8d540a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:07.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.281+0000 7f8d6be00640 1 --2- 172.21.15.43:0/2503542863 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564f319b3440 0x564f319b5830 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f8d58009bf0 tx=0x7f8d58011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8093deb855b8d89e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:07.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.281+0000 7f8d69600640 1 -- 172.21.15.43:0/720101445 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7f8d600ce920 con 0x7f8d6c073af0 2024-03-28T22:46:07.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.281+0000 7f8d71800640 1 --2- 172.21.15.43:0/720101445 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f8d540a1f70 0x7f8d540a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:07.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.281+0000 7f8d5e000640 1 -- 172.21.15.43:0/2503542863 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d58004050 con 0x564f319b3440 2024-03-28T22:46:07.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.281+0000 7f8d5e000640 1 -- 172.21.15.43:0/2503542863 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d58005500 con 0x564f319b3440 2024-03-28T22:46:07.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.281+0000 7f8d5e000640 1 -- 172.21.15.43:0/2503542863 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d58004dd0 con 0x564f319b3440 2024-03-28T22:46:07.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.282+0000 7f8d7641b740 1 -- 172.21.15.43:0/2503542863 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564f319b3440 msgr2=0x564f319b5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:07.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.282+0000 7f8d7641b740 1 --2- 172.21.15.43:0/2503542863 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564f319b3440 0x564f319b5830 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f8d58009bf0 tx=0x7f8d58011810 comp rx=0 tx=0).stop 2024-03-28T22:46:07.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.282+0000 7f8d71800640 1 --2- 172.21.15.43:0/720101445 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f8d540a1f70 0x7f8d540a4430 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8d6402c6c0 tx=0x7f8d64005d80 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:07.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.282+0000 7f8d7641b740 1 -- 172.21.15.43:0/2503542863 shutdown_connections 2024-03-28T22:46:07.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.282+0000 7f8d7641b740 1 --2- 172.21.15.43:0/2503542863 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564f319aa0d0 0x564f319aa550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.282+0000 7f8d7641b740 1 --2- 172.21.15.43:0/2503542863 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564f319b3440 0x564f319b5830 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.282+0000 7f8d7641b740 1 --2- 172.21.15.43:0/2503542863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564f319aadb0 0x564f319a9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.282+0000 7f8d7641b740 1 -- 172.21.15.43:0/2503542863 >> 172.21.15.43:0/2503542863 conn(0x564f31970a10 msgr2=0x564f319a4b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:07.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.282+0000 7f8d7641b740 1 -- 172.21.15.43:0/2503542863 shutdown_connections 2024-03-28T22:46:07.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.282+0000 7f8d7641b740 1 -- 172.21.15.43:0/2503542863 wait complete. 2024-03-28T22:46:07.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.283+0000 7f8d7641b740 1 -- start start 2024-03-28T22:46:07.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.283+0000 7f8d7641b740 1 Processor -- start 2024-03-28T22:46:07.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.283+0000 7f8d7641b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564f319aa0d0 0x564f319b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:07.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.283+0000 7f8d7641b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564f319aadb0 0x564f319b0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:07.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.283+0000 7f8d7641b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564f319b3440 0x564f319b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:07.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.283+0000 7f8d7641b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x564f319b62b0 con 0x564f319b3440 2024-03-28T22:46:07.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.283+0000 7f8d7641b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x564f318ac250 con 0x564f319aa0d0 2024-03-28T22:46:07.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.283+0000 7f8d7641b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x564f319b6430 con 0x564f319aadb0 2024-03-28T22:46:07.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.284+0000 7f8d6be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564f319aadb0 0x564f319b0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:07.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.284+0000 7f8d70e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564f319aa0d0 0x564f319b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:07.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.284+0000 7f8d7641b740 10 client.0 _ll_register_callbacks cb 0x564f31621be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:46:07.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.284+0000 7f8d6be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564f319aadb0 0x564f319b0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:38798/0 (socket says 172.21.15.43:38798) 2024-03-28T22:46:07.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.284+0000 7f8d70e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564f319aa0d0 0x564f319b03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:34452/0 (socket says 172.21.15.43:34452) 2024-03-28T22:46:07.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.284+0000 7f8d6be00640 1 -- 172.21.15.43:0/198768934 learned_addr learned my addr 172.21.15.43:0/198768934 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:07.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.284+0000 7f8d71800640 1 --2- 172.21.15.43:0/198768934 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564f319b3440 0x564f319b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:07.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.284+0000 7f8d70e00640 1 -- 172.21.15.43:0/198768934 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564f319aadb0 msgr2=0x564f319b0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:07.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.284+0000 7f8d70e00640 1 --2- 172.21.15.43:0/198768934 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564f319aadb0 0x564f319b0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.284+0000 7f8d70e00640 1 -- 172.21.15.43:0/198768934 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564f319b3440 msgr2=0x564f319b0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:07.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.284+0000 7f8d70e00640 1 --2- 172.21.15.43:0/198768934 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564f319b3440 0x564f319b0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.284+0000 7f8d70e00640 1 -- 172.21.15.43:0/198768934 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f318c8960 con 0x564f319aa0d0 2024-03-28T22:46:07.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.284+0000 7f8d6be00640 1 --2- 172.21.15.43:0/198768934 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564f319aadb0 0x564f319b0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:07.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.284+0000 7f8d70e00640 1 --2- 172.21.15.43:0/198768934 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564f319aa0d0 0x564f319b03c0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f8d6009ac60 tx=0x7f8d600972f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5e750b5f32e44fb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:07.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.284+0000 7f8d52a00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:46:07.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.285+0000 7f8d52a00640 1 -- 172.21.15.43:0/198768934 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d600cc0a0 con 0x564f319aa0d0 2024-03-28T22:46:07.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.285+0000 7f8d52a00640 1 -- 172.21.15.43:0/198768934 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7f8d6009a5b0 con 0x564f319aa0d0 2024-03-28T22:46:07.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.285+0000 7f8d7641b740 1 -- 172.21.15.43:0/198768934 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d58005500 con 0x564f319aa0d0 2024-03-28T22:46:07.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.285+0000 7f8d52a00640 1 -- 172.21.15.43:0/198768934 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8d6009af60 con 0x564f319aa0d0 2024-03-28T22:46:07.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.285+0000 7f8d7641b740 10 client.28763 fetch_fsmap learned FSMap version 32 2024-03-28T22:46:07.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.285+0000 7f8d7641b740 1 -- 172.21.15.43:0/198768934 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564f316ec7d0 con 0x564f319aa0d0 2024-03-28T22:46:07.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.285+0000 7f8d52a00640 1 -- 172.21.15.43:0/198768934 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d58005500 con 0x564f319aa0d0 2024-03-28T22:46:07.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.285+0000 7f8d52a00640 1 -- 172.21.15.43:0/198768934 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f8d60018b60 con 0x564f319aa0d0 2024-03-28T22:46:07.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.285+0000 7f8d7641b740 10 client.28763 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:46:07.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.285+0000 7f8d7641b740 10 client.28763 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:46:07.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.285+0000 7f8d7641b740 20 client.28763 populate_metadata read hostname 'smithi043' 2024-03-28T22:46:07.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.285+0000 7f8d7641b740 1 --2- 172.21.15.43:0/198768934 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x564f319c3890 0x564f319c5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:07.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.285+0000 7f8d7641b740 4 client.28763 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:46:07.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.285+0000 7f8d7641b740 1 -- 172.21.15.43:0/198768934 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f3191b640 con 0x564f319c3890 2024-03-28T22:46:07.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.286+0000 7f8d6be00640 1 --2- 172.21.15.43:0/198768934 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x564f319c3890 0x564f319c5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:07.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.286+0000 7f8d6be00640 1 --2- 172.21.15.43:0/198768934 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x564f319c3890 0x564f319c5d10 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=178e81c4bb99abc3 server_cookie=84c8969c070fb0d7 in_seq=0 out_seq=0 2024-03-28T22:46:07.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.286+0000 7f8d52a00640 10 client.28763 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:46:07.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.287+0000 7f8d52a00640 1 -- 172.21.15.43:0/198768934 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564f3191b640 con 0x564f319c3890 2024-03-28T22:46:07.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.287+0000 7f8d52a00640 10 client.28763 handle_command_reply: tid=0 2024-03-28T22:46:07.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.287+0000 7f8d7641b740 1 client.28763 shutdown 2024-03-28T22:46:07.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.288+0000 7f8d7641b740 1 -- 172.21.15.43:0/198768934 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564f319aa0d0 msgr2=0x564f319b03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:07.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.288+0000 7f8d7641b740 1 --2- 172.21.15.43:0/198768934 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564f319aa0d0 0x564f319b03c0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f8d6009ac60 tx=0x7f8d600972f0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.288+0000 7f8d7641b740 1 -- 172.21.15.43:0/198768934 shutdown_connections 2024-03-28T22:46:07.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.288+0000 7f8d7641b740 1 --2- 172.21.15.43:0/198768934 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564f319b3440 0x564f319b0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.288+0000 7f8d7641b740 1 --2- 172.21.15.43:0/198768934 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564f319aadb0 0x564f319b0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.288+0000 7f8d7641b740 1 --2- 172.21.15.43:0/198768934 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x564f319c3890 0x564f319c5d10 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.288+0000 7f8d7641b740 1 --2- 172.21.15.43:0/198768934 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564f319aa0d0 0x564f319b03c0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.288+0000 7f8d7641b740 1 -- 172.21.15.43:0/198768934 >> 172.21.15.43:0/198768934 conn(0x564f31970a10 msgr2=0x564f319a6870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:07.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.288+0000 7f8d52a00640 0 client.28763 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:46:07.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.288+0000 7f8d7641b740 1 -- 172.21.15.43:0/198768934 shutdown_connections 2024-03-28T22:46:07.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.288+0000 7f8d7641b740 1 -- 172.21.15.43:0/198768934 wait complete. 2024-03-28T22:46:07.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.288+0000 7f8d7641b740 20 client.28763 trim_cache size 0 max 16384 2024-03-28T22:46:07.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.307+0000 7f8d7641b740 1 Processor -- start 2024-03-28T22:46:07.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.307+0000 7f8d7641b740 1 -- start start 2024-03-28T22:46:07.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.307+0000 7f8d7641b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564f319aa0d0 0x564f319b2af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:07.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.307+0000 7f8d7641b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564f319aadb0 0x564f319bf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:07.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.307+0000 7f8d7641b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564f319b3440 0x564f319cb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:07.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.307+0000 7f8d7641b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8d6c1b5120 con 0x564f319aa0d0 2024-03-28T22:46:07.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.307+0000 7f8d7641b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8d6c1b6c60 con 0x564f319aadb0 2024-03-28T22:46:07.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.307+0000 7f8d7641b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f8d6c1b57f0 con 0x564f319b3440 2024-03-28T22:46:07.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.308+0000 7f8d70e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564f319aa0d0 0x564f319b2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:07.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.308+0000 7f8d71800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564f319b3440 0x564f319cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:07.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.308+0000 7f8d6be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564f319aadb0 0x564f319bf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:07.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.308+0000 7f8d70e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564f319aa0d0 0x564f319b2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34658/0 (socket says 172.21.15.43:34658) 2024-03-28T22:46:07.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.308+0000 7f8d70e00640 1 -- 172.21.15.43:0/4093581981 learned_addr learned my addr 172.21.15.43:0/4093581981 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:07.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.308+0000 7f8d71800640 1 -- 172.21.15.43:0/4093581981 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564f319aadb0 msgr2=0x564f319bf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:07.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.308+0000 7f8d71800640 1 --2- 172.21.15.43:0/4093581981 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564f319aadb0 0x564f319bf0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.308+0000 7f8d71800640 1 -- 172.21.15.43:0/4093581981 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564f319aa0d0 msgr2=0x564f319b2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:07.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.308+0000 7f8d71800640 1 --2- 172.21.15.43:0/4093581981 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564f319aa0d0 0x564f319b2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.308+0000 7f8d71800640 1 -- 172.21.15.43:0/4093581981 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f316223f0 con 0x564f319b3440 2024-03-28T22:46:07.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.308+0000 7f8d6be00640 1 --2- 172.21.15.43:0/4093581981 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564f319aadb0 0x564f319bf0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:07.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.308+0000 7f8d70e00640 1 --2- 172.21.15.43:0/4093581981 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564f319aa0d0 0x564f319b2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:07.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.309+0000 7f8d71800640 1 --2- 172.21.15.43:0/4093581981 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564f319b3440 0x564f319cb630 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f8d64037040 tx=0x7f8d64041000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78fbf49967010537 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:07.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.309+0000 7f8d5d600640 1 -- 172.21.15.43:0/4093581981 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d64039040 con 0x564f319b3440 2024-03-28T22:46:07.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.309+0000 7f8d5d600640 1 -- 172.21.15.43:0/4093581981 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d64034d90 con 0x564f319b3440 2024-03-28T22:46:07.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.309+0000 7f8d5d600640 1 -- 172.21.15.43:0/4093581981 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d640328d0 con 0x564f319b3440 2024-03-28T22:46:07.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.309+0000 7f8d7641b740 1 -- 172.21.15.43:0/4093581981 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564f319b3440 msgr2=0x564f319cb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:07.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.309+0000 7f8d7641b740 1 --2- 172.21.15.43:0/4093581981 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564f319b3440 0x564f319cb630 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f8d64037040 tx=0x7f8d64041000 comp rx=0 tx=0).stop 2024-03-28T22:46:07.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.310+0000 7f8d7641b740 1 -- 172.21.15.43:0/4093581981 shutdown_connections 2024-03-28T22:46:07.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.310+0000 7f8d7641b740 1 --2- 172.21.15.43:0/4093581981 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564f319aadb0 0x564f319bf0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.310+0000 7f8d7641b740 1 --2- 172.21.15.43:0/4093581981 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564f319b3440 0x564f319cb630 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.310+0000 7f8d7641b740 1 --2- 172.21.15.43:0/4093581981 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564f319aa0d0 0x564f319b2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.310+0000 7f8d7641b740 1 -- 172.21.15.43:0/4093581981 >> 172.21.15.43:0/4093581981 conn(0x564f31970a10 msgr2=0x564f319aec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:07.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.310+0000 7f8d7641b740 1 -- 172.21.15.43:0/4093581981 shutdown_connections 2024-03-28T22:46:07.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.310+0000 7f8d7641b740 1 -- 172.21.15.43:0/4093581981 wait complete. 2024-03-28T22:46:07.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.310+0000 7f8d7641b740 1 -- start start 2024-03-28T22:46:07.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.310+0000 7f8d7641b740 1 Processor -- start 2024-03-28T22:46:07.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.311+0000 7f8d7641b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564f319aa0d0 0x564f319bb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:07.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.311+0000 7f8d7641b740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564f319aadb0 0x564f319b4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:07.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.311+0000 7f8d7641b740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564f319b3440 0x564f319b50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:07.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.311+0000 7f8d7641b740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8d6c1b6590 con 0x564f319b3440 2024-03-28T22:46:07.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.311+0000 7f8d7641b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8d6c1b5ec0 con 0x564f319aa0d0 2024-03-28T22:46:07.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.311+0000 7f8d7641b740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f8d6c1b3120 con 0x564f319aadb0 2024-03-28T22:46:07.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.311+0000 7f8d7641b740 10 client.0 _ll_register_callbacks cb 0x7f8d600d4220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:46:07.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.311+0000 7f8d70e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564f319aa0d0 0x564f319bb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:07.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.311+0000 7f8d6be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564f319aadb0 0x564f319b4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:07.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.311+0000 7f8d6be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564f319aadb0 0x564f319b4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:38818/0 (socket says 172.21.15.43:38818) 2024-03-28T22:46:07.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.311+0000 7f8d6be00640 1 -- 172.21.15.43:0/3746066496 learned_addr learned my addr 172.21.15.43:0/3746066496 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:07.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.311+0000 7f8d71800640 1 --2- 172.21.15.43:0/3746066496 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564f319b3440 0x564f319b50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:07.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.311+0000 7f8d6be00640 1 -- 172.21.15.43:0/3746066496 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564f319aa0d0 msgr2=0x564f319bb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:07.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.311+0000 7f8d6be00640 1 --2- 172.21.15.43:0/3746066496 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564f319aa0d0 0x564f319bb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.311+0000 7f8d6be00640 1 -- 172.21.15.43:0/3746066496 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564f319b3440 msgr2=0x564f319b50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:07.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.311+0000 7f8d6be00640 1 --2- 172.21.15.43:0/3746066496 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564f319b3440 0x564f319b50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.311+0000 7f8d6be00640 1 -- 172.21.15.43:0/3746066496 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f315e20f0 con 0x564f319aadb0 2024-03-28T22:46:07.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.311+0000 7f8d70e00640 1 --2- 172.21.15.43:0/3746066496 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564f319aa0d0 0x564f319bb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:07.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.312+0000 7f8d6be00640 1 --2- 172.21.15.43:0/3746066496 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564f319aadb0 0x564f319b4b80 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f8d5801b040 tx=0x7f8d580047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab983998ff7cae6e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:07.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.312+0000 7f8d52a00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:46:07.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.312+0000 7f8d52a00640 1 -- 172.21.15.43:0/3746066496 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d58034060 con 0x564f319aadb0 2024-03-28T22:46:07.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.312+0000 7f8d52a00640 1 -- 172.21.15.43:0/3746066496 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9539+0+0 (secure 0 0 0) 0x7f8d5802eab0 con 0x564f319aadb0 2024-03-28T22:46:07.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.312+0000 7f8d7641b740 1 -- 172.21.15.43:0/3746066496 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d64034d90 con 0x564f319aadb0 2024-03-28T22:46:07.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.312+0000 7f8d52a00640 1 -- 172.21.15.43:0/3746066496 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8d5803b050 con 0x564f319aadb0 2024-03-28T22:46:07.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.312+0000 7f8d7641b740 10 client.28711 fetch_fsmap learned FSMap version 32 2024-03-28T22:46:07.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.312+0000 7f8d7641b740 1 -- 172.21.15.43:0/3746066496 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564f319bab60 con 0x564f319aadb0 2024-03-28T22:46:07.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.313+0000 7f8d52a00640 1 -- 172.21.15.43:0/3746066496 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d64034d90 con 0x564f319aadb0 2024-03-28T22:46:07.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.313+0000 7f8d52a00640 1 -- 172.21.15.43:0/3746066496 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f8d580035f0 con 0x564f319aadb0 2024-03-28T22:46:07.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.313+0000 7f8d7641b740 10 client.28711 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:46:07.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.313+0000 7f8d7641b740 10 client.28711 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:46:07.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.313+0000 7f8d7641b740 20 client.28711 populate_metadata read hostname 'smithi043' 2024-03-28T22:46:07.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.313+0000 7f8d7641b740 1 --2- 172.21.15.43:0/3746066496 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x564f319bb0a0 0x564f31a2bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:07.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.313+0000 7f8d7641b740 4 client.28711 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:46:07.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.313+0000 7f8d7641b740 1 -- 172.21.15.43:0/3746066496 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564f31915990 con 0x564f319bb0a0 2024-03-28T22:46:07.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.313+0000 7f8d70e00640 1 --2- 172.21.15.43:0/3746066496 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x564f319bb0a0 0x564f31a2bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:07.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.313+0000 7f8d70e00640 1 --2- 172.21.15.43:0/3746066496 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x564f319bb0a0 0x564f31a2bc70 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b3239206fc27f38 server_cookie=b394523c4ef78062 in_seq=0 out_seq=0 2024-03-28T22:46:07.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.313+0000 7f8d52a00640 10 client.28711 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:46:07.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.314+0000 7f8d52a00640 1 -- 172.21.15.43:0/3746066496 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564f31915990 con 0x564f319bb0a0 2024-03-28T22:46:07.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.314+0000 7f8d52a00640 10 client.28711 handle_command_reply: tid=0 2024-03-28T22:46:07.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.314+0000 7f8d7641b740 1 client.28711 shutdown 2024-03-28T22:46:07.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.314+0000 7f8d7641b740 1 -- 172.21.15.43:0/3746066496 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564f319aadb0 msgr2=0x564f319b4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:07.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.314+0000 7f8d7641b740 1 --2- 172.21.15.43:0/3746066496 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564f319aadb0 0x564f319b4b80 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f8d5801b040 tx=0x7f8d580047c0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.314+0000 7f8d7641b740 1 -- 172.21.15.43:0/3746066496 shutdown_connections 2024-03-28T22:46:07.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.314+0000 7f8d7641b740 1 --2- 172.21.15.43:0/3746066496 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564f319b3440 0x564f319b50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.314+0000 7f8d7641b740 1 --2- 172.21.15.43:0/3746066496 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564f319aadb0 0x564f319b4b80 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.314+0000 7f8d7641b740 1 --2- 172.21.15.43:0/3746066496 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x564f319bb0a0 0x564f31a2bc70 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.314+0000 7f8d7641b740 1 --2- 172.21.15.43:0/3746066496 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564f319aa0d0 0x564f319bb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.314+0000 7f8d7641b740 1 -- 172.21.15.43:0/3746066496 >> 172.21.15.43:0/3746066496 conn(0x564f31970a10 msgr2=0x564f319a3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:07.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.314+0000 7f8d52a00640 0 client.28711 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:46:07.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.314+0000 7f8d7641b740 1 -- 172.21.15.43:0/3746066496 shutdown_connections 2024-03-28T22:46:07.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.314+0000 7f8d7641b740 1 -- 172.21.15.43:0/3746066496 wait complete. 2024-03-28T22:46:07.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.314+0000 7f8d7641b740 20 client.28711 trim_cache size 0 max 16384 2024-03-28T22:46:07.326 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:46:07.326 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:46:07.326 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-03-28T22:46:07.326 INFO:teuthology.orchestra.run.smithi043.stdout: "9fa76a05-e3b0-47c7-a9b0-01be8ba103a3": { 2024-03-28T22:46:07.326 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-03-28T22:46:07.326 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "9fa76a05-e3b0-47c7-a9b0-01be8ba103a3", 2024-03-28T22:46:07.327 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-03-28T22:46:07.327 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:46:07.327 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:46:07.327 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:46:07.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.315+0000 7f8d72200640 1 -- 172.21.15.43:0/720101445 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f8d540a1f70 msgr2=0x7f8d540a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:07.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.315+0000 7f8d72200640 1 --2- 172.21.15.43:0/720101445 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f8d540a1f70 0x7f8d540a4430 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8d6402c6c0 tx=0x7f8d64005d80 comp rx=0 tx=0).stop 2024-03-28T22:46:07.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.315+0000 7f8d72200640 1 -- 172.21.15.43:0/720101445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d6c073af0 msgr2=0x7f8d6c19e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:07.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.315+0000 7f8d72200640 1 --2- 172.21.15.43:0/720101445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d6c073af0 0x7f8d6c19e470 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8d6000e9f0 tx=0x7f8d6000eeb0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.315+0000 7f8d72200640 1 -- 172.21.15.43:0/720101445 shutdown_connections 2024-03-28T22:46:07.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.315+0000 7f8d72200640 1 --2- 172.21.15.43:0/720101445 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f8d540a1f70 0x7f8d540a4430 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.315+0000 7f8d72200640 1 --2- 172.21.15.43:0/720101445 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8d6c1125d0 0x7f8d6c1a2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.315+0000 7f8d72200640 1 --2- 172.21.15.43:0/720101445 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8d6c074410 0x7f8d6c19e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.315+0000 7f8d72200640 1 --2- 172.21.15.43:0/720101445 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d6c073af0 0x7f8d6c19e470 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:07.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.315+0000 7f8d72200640 1 -- 172.21.15.43:0/720101445 >> 172.21.15.43:0/720101445 conn(0x7f8d6c0fd0b0 msgr2=0x7f8d6c0fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:07.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.315+0000 7f8d72200640 1 -- 172.21.15.43:0/720101445 shutdown_connections 2024-03-28T22:46:07.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:07.315+0000 7f8d72200640 1 -- 172.21.15.43:0/720101445 wait complete. 2024-03-28T22:46:07.331 INFO:tasks.cephfs.filesystem:scrub status for tag:9fa76a05-e3b0-47c7-a9b0-01be8ba103a3 - {'path': '/', 'tag': '9fa76a05-e3b0-47c7-a9b0-01be8ba103a3', 'options': 'recursive,force'} 2024-03-28T22:46:07.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:07 smithi043 ceph-mon[31823]: pgmap v277: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-03-28T22:46:07.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:07 smithi043 ceph-mon[31823]: scrub queued for path: / 2024-03-28T22:46:07.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:07 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:46:07.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:07 smithi043 ceph-mon[31823]: scrub summary: active paths [/] 2024-03-28T22:46:07.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:07 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:46:07.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:07 smithi172 ceph-mon[36566]: pgmap v277: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-03-28T22:46:07.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:07 smithi172 ceph-mon[36566]: scrub queued for path: / 2024-03-28T22:46:07.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:07 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:46:07.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:07 smithi172 ceph-mon[36566]: scrub summary: active paths [/] 2024-03-28T22:46:07.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:07 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:46:07.698 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:07 smithi082 ceph-mon[36471]: pgmap v277: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-03-28T22:46:07.698 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:07 smithi082 ceph-mon[36471]: scrub queued for path: / 2024-03-28T22:46:07.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:07 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:46:07.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:07 smithi082 ceph-mon[36471]: scrub summary: active paths [/] 2024-03-28T22:46:07.699 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:07 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:46:08.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:08 smithi043 ceph-mon[31823]: scrub summary: idle 2024-03-28T22:46:08.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:08 smithi172 ceph-mon[36566]: scrub summary: idle 2024-03-28T22:46:08.698 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:08 smithi082 ceph-mon[36471]: scrub summary: idle 2024-03-28T22:46:09.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:09 smithi043 ceph-mon[31823]: pgmap v278: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 45 KiB/s wr, 36 op/s 2024-03-28T22:46:09.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:09 smithi043 ceph-mon[31823]: osdmap e105: 12 total, 12 up, 12 in 2024-03-28T22:46:09.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:09 smithi172 ceph-mon[36566]: pgmap v278: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 45 KiB/s wr, 36 op/s 2024-03-28T22:46:09.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:09 smithi172 ceph-mon[36566]: osdmap e105: 12 total, 12 up, 12 in 2024-03-28T22:46:09.698 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:09 smithi082 ceph-mon[36471]: pgmap v278: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 45 KiB/s wr, 36 op/s 2024-03-28T22:46:09.698 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:09 smithi082 ceph-mon[36471]: osdmap e105: 12 total, 12 up, 12 in 2024-03-28T22:46:11.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:11 smithi043 ceph-mon[31823]: pgmap v280: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 54 KiB/s wr, 39 op/s 2024-03-28T22:46:11.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:11 smithi172 ceph-mon[36566]: pgmap v280: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 54 KiB/s wr, 39 op/s 2024-03-28T22:46:11.698 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:11 smithi082 ceph-mon[36471]: pgmap v280: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 54 KiB/s wr, 39 op/s 2024-03-28T22:46:13.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:13 smithi043 ceph-mon[31823]: pgmap v281: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 54 KiB/s wr, 39 op/s 2024-03-28T22:46:13.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:13 smithi172 ceph-mon[36566]: pgmap v281: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 54 KiB/s wr, 39 op/s 2024-03-28T22:46:13.698 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:13 smithi082 ceph-mon[36471]: pgmap v281: 129 pgs: 129 active+clean; 63 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 54 KiB/s wr, 39 op/s 2024-03-28T22:46:15.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:15 smithi043 ceph-mon[31823]: pgmap v282: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 204 B/s wr, 18 op/s 2024-03-28T22:46:15.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:15 smithi172 ceph-mon[36566]: pgmap v282: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 204 B/s wr, 18 op/s 2024-03-28T22:46:15.698 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:15 smithi082 ceph-mon[36471]: pgmap v282: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 204 B/s wr, 18 op/s 2024-03-28T22:46:17.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:17 smithi172 ceph-mon[36566]: pgmap v283: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 204 B/s wr, 18 op/s 2024-03-28T22:46:17.698 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:17 smithi082 ceph-mon[36471]: pgmap v283: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 204 B/s wr, 18 op/s 2024-03-28T22:46:17.765 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:17 smithi043 ceph-mon[31823]: pgmap v283: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 204 B/s wr, 18 op/s 2024-03-28T22:46:19.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:18 smithi172 ceph-mon[36566]: pgmap v284: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:46:19.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:18 smithi172 ceph-mon[36566]: osdmap e106: 12 total, 12 up, 12 in 2024-03-28T22:46:19.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:18 smithi082 ceph-mon[36471]: pgmap v284: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:46:19.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:18 smithi082 ceph-mon[36471]: osdmap e106: 12 total, 12 up, 12 in 2024-03-28T22:46:19.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:18 smithi043 ceph-mon[31823]: pgmap v284: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:46:19.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:18 smithi043 ceph-mon[31823]: osdmap e106: 12 total, 12 up, 12 in 2024-03-28T22:46:21.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:20 smithi172 ceph-mon[36566]: pgmap v286: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:46:21.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:20 smithi082 ceph-mon[36471]: pgmap v286: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:46:21.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:20 smithi043 ceph-mon[31823]: pgmap v286: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:46:23.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:22 smithi172 ceph-mon[36566]: pgmap v287: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:46:23.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:22 smithi082 ceph-mon[36471]: pgmap v287: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:46:23.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:22 smithi043 ceph-mon[31823]: pgmap v287: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:46:25.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:24 smithi172 ceph-mon[36566]: pgmap v288: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T22:46:25.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:24 smithi082 ceph-mon[36471]: pgmap v288: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T22:46:25.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:24 smithi043 ceph-mon[31823]: pgmap v288: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T22:46:27.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:26 smithi172 ceph-mon[36566]: pgmap v289: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T22:46:27.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:26 smithi082 ceph-mon[36471]: pgmap v289: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T22:46:27.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:26 smithi043 ceph-mon[31823]: pgmap v289: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T22:46:29.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:28 smithi172 ceph-mon[36566]: pgmap v290: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:46:29.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:28 smithi082 ceph-mon[36471]: pgmap v290: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:46:29.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:28 smithi043 ceph-mon[31823]: pgmap v290: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:46:31.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:30 smithi172 ceph-mon[36566]: pgmap v291: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:46:31.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:30 smithi082 ceph-mon[36471]: pgmap v291: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:46:31.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:30 smithi043 ceph-mon[31823]: pgmap v291: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:46:33.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:32 smithi172 ceph-mon[36566]: pgmap v292: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:46:33.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:32 smithi082 ceph-mon[36471]: pgmap v292: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:46:33.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:32 smithi043 ceph-mon[31823]: pgmap v292: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:46:34.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:33 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:46:34.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:33 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:46:34.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:33 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:46:35.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:35 smithi043 ceph-mon[31823]: pgmap v293: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:46:35.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:35 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:46:35.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:35 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:46:35.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:35 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:46:35.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:35 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:46:35.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:35 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:46:35.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:35 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:46:35.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:35 smithi172 ceph-mon[36566]: pgmap v293: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:46:35.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:35 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:46:35.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:35 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:46:35.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:35 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:46:35.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:35 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:46:35.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:35 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:46:35.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:35 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:46:35.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:35 smithi082 ceph-mon[36471]: pgmap v293: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:46:35.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:35 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:46:35.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:35 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:46:35.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:35 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:46:35.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:35 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:46:35.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:35 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:46:35.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:35 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:46:36.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:36 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:46:36.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:36 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:46:36.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:36 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:46:36.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:36 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:46:36.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:36 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:46:36.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:36 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:46:36.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:36 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:46:36.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:36 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:46:36.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:36 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:46:37.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:37 smithi043 ceph-mon[31823]: pgmap v294: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:46:37.332 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:46:37.332 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:46:37.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:37 smithi172 ceph-mon[36566]: pgmap v294: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:46:37.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:37 smithi082 ceph-mon[36471]: pgmap v294: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:46:37.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.572+0000 7f811f800640 1 -- 172.21.15.43:0/1653356380 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8118109900 msgr2=0x7f8118109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:37.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.572+0000 7f811f800640 1 --2- 172.21.15.43:0/1653356380 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8118109900 0x7f8118109ce0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f810c009930 tx=0x7f810c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:46:37.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.572+0000 7f811f800640 1 -- 172.21.15.43:0/1653356380 shutdown_connections 2024-03-28T22:46:37.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.572+0000 7f811f800640 1 --2- 172.21.15.43:0/1653356380 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f81181042b0 0x7f81181107c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.572+0000 7f811f800640 1 --2- 172.21.15.43:0/1653356380 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81181038f0 0x7f8118103d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.572+0000 7f811f800640 1 --2- 172.21.15.43:0/1653356380 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8118109900 0x7f8118109ce0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.572+0000 7f811f800640 1 -- 172.21.15.43:0/1653356380 >> 172.21.15.43:0/1653356380 conn(0x7f81180ff3a0 msgr2=0x7f81181017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:37.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.572+0000 7f811f800640 1 -- 172.21.15.43:0/1653356380 shutdown_connections 2024-03-28T22:46:37.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.572+0000 7f811f800640 1 -- 172.21.15.43:0/1653356380 wait complete. 2024-03-28T22:46:37.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.574+0000 7f811f800640 1 Processor -- start 2024-03-28T22:46:37.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.574+0000 7f811f800640 1 -- start start 2024-03-28T22:46:37.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.574+0000 7f811f800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81181038f0 0x7f811819a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:37.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.574+0000 7f811f800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81181042b0 0x7f811819a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:37.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.574+0000 7f811f800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8118109900 0x7f811819eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:37.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.574+0000 7f811f800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8118113d60 con 0x7f81181038f0 2024-03-28T22:46:37.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.574+0000 7f811f800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8118113be0 con 0x7f8118109900 2024-03-28T22:46:37.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.574+0000 7f811f800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f8118113ee0 con 0x7f81181042b0 2024-03-28T22:46:37.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.575+0000 7f8117e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81181042b0 0x7f811819a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:37.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.575+0000 7f811ce00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81181038f0 0x7f811819a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:37.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.575+0000 7f8117e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81181042b0 0x7f811819a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:39920/0 (socket says 172.21.15.43:39920) 2024-03-28T22:46:37.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.575+0000 7f8117e00640 1 -- 172.21.15.43:0/1362877485 learned_addr learned my addr 172.21.15.43:0/1362877485 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:37.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.575+0000 7f811d800640 1 --2- 172.21.15.43:0/1362877485 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8118109900 0x7f811819eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:37.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.575+0000 7f8117e00640 1 -- 172.21.15.43:0/1362877485 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8118109900 msgr2=0x7f811819eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:37.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.575+0000 7f8117e00640 1 --2- 172.21.15.43:0/1362877485 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8118109900 0x7f811819eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.575+0000 7f8117e00640 1 -- 172.21.15.43:0/1362877485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81181038f0 msgr2=0x7f811819a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:37.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.575+0000 7f8117e00640 1 --2- 172.21.15.43:0/1362877485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81181038f0 0x7f811819a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.575+0000 7f8117e00640 1 -- 172.21.15.43:0/1362877485 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f811819f230 con 0x7f81181042b0 2024-03-28T22:46:37.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.576+0000 7f8117e00640 1 --2- 172.21.15.43:0/1362877485 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81181042b0 0x7f811819a780 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f810400e9f0 tx=0x7f810400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:37.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.576+0000 7f8115600640 1 -- 172.21.15.43:0/1362877485 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f810400cd80 con 0x7f81181042b0 2024-03-28T22:46:37.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.576+0000 7f8115600640 1 -- 172.21.15.43:0/1362877485 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8104010910 con 0x7f81181042b0 2024-03-28T22:46:37.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.576+0000 7f8115600640 1 -- 172.21.15.43:0/1362877485 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8104018990 con 0x7f81181042b0 2024-03-28T22:46:37.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.576+0000 7f811f800640 1 -- 172.21.15.43:0/1362877485 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f811819f520 con 0x7f81181042b0 2024-03-28T22:46:37.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.576+0000 7f811f800640 1 -- 172.21.15.43:0/1362877485 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81181a6db0 con 0x7f81181042b0 2024-03-28T22:46:37.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.579+0000 7f8115600640 1 -- 172.21.15.43:0/1362877485 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f8104010ac0 con 0x7f81181042b0 2024-03-28T22:46:37.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.579+0000 7f81223df740 1 Processor -- start 2024-03-28T22:46:37.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.579+0000 7f81223df740 1 -- start start 2024-03-28T22:46:37.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.580+0000 7f81223df740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639d7e6dbc0 0x5639d7e6c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:37.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.580+0000 7f81223df740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639d7e6cee0 0x5639d7e6d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:37.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.580+0000 7f81223df740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639d7e76250 0x5639d7e78640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:37.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.580+0000 7f81223df740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5639d7dd1100 con 0x5639d7e6cee0 2024-03-28T22:46:37.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.580+0000 7f81223df740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5639d7d79ba0 con 0x5639d7e6dbc0 2024-03-28T22:46:37.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.580+0000 7f81223df740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5639d7dcd1a0 con 0x5639d7e76250 2024-03-28T22:46:37.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.580+0000 7f811d800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639d7e6dbc0 0x5639d7e6c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:37.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.580+0000 7f811d800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639d7e6dbc0 0x5639d7e6c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:42446/0 (socket says 172.21.15.43:42446) 2024-03-28T22:46:37.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.580+0000 7f811d800640 1 -- 172.21.15.43:0/2609946529 learned_addr learned my addr 172.21.15.43:0/2609946529 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:37.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.580+0000 7f811ce00640 1 --2- 172.21.15.43:0/2609946529 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639d7e6cee0 0x5639d7e6d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:37.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.580+0000 7f8117e00640 1 --2- 172.21.15.43:0/2609946529 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639d7e76250 0x5639d7e78640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:37.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.580+0000 7f811d800640 1 -- 172.21.15.43:0/2609946529 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639d7e76250 msgr2=0x5639d7e78640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:37.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.580+0000 7f811d800640 1 --2- 172.21.15.43:0/2609946529 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639d7e76250 0x5639d7e78640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.580+0000 7f811d800640 1 -- 172.21.15.43:0/2609946529 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639d7e6cee0 msgr2=0x5639d7e6d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:37.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.580+0000 7f811d800640 1 --2- 172.21.15.43:0/2609946529 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639d7e6cee0 0x5639d7e6d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.580+0000 7f811d800640 1 -- 172.21.15.43:0/2609946529 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639d7daa1a0 con 0x5639d7e6dbc0 2024-03-28T22:46:37.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.580+0000 7f8117e00640 1 --2- 172.21.15.43:0/2609946529 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639d7e76250 0x5639d7e78640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:46:37.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.581+0000 7f811d800640 1 --2- 172.21.15.43:0/2609946529 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639d7e6dbc0 0x5639d7e6c910 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f8110009f50 tx=0x7f81100118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d0d29b6c22723c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:37.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.581+0000 7f8115600640 1 --2- 172.21.15.43:0/1362877485 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f81000a1f70 0x7f81000a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:37.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.581+0000 7f810a000640 1 -- 172.21.15.43:0/2609946529 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8110004d20 con 0x5639d7e6dbc0 2024-03-28T22:46:37.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.581+0000 7f810a000640 1 -- 172.21.15.43:0/2609946529 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8110004050 con 0x5639d7e6dbc0 2024-03-28T22:46:37.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.581+0000 7f8115600640 1 -- 172.21.15.43:0/1362877485 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9539+0+0 (secure 0 0 0) 0x7f81040ce8c0 con 0x7f81181042b0 2024-03-28T22:46:37.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.581+0000 7f810a000640 1 -- 172.21.15.43:0/2609946529 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8110005280 con 0x5639d7e6dbc0 2024-03-28T22:46:37.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.581+0000 7f811ce00640 1 --2- 172.21.15.43:0/1362877485 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f81000a1f70 0x7f81000a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:37.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.582+0000 7f81223df740 1 -- 172.21.15.43:0/2609946529 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639d7e6dbc0 msgr2=0x5639d7e6c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:37.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.582+0000 7f81223df740 1 --2- 172.21.15.43:0/2609946529 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639d7e6dbc0 0x5639d7e6c910 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f8110009f50 tx=0x7f81100118d0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.582+0000 7f81223df740 1 -- 172.21.15.43:0/2609946529 shutdown_connections 2024-03-28T22:46:37.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.582+0000 7f81223df740 1 --2- 172.21.15.43:0/2609946529 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639d7e76250 0x5639d7e78640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.582+0000 7f81223df740 1 --2- 172.21.15.43:0/2609946529 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639d7e6cee0 0x5639d7e6d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.582+0000 7f81223df740 1 --2- 172.21.15.43:0/2609946529 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639d7e6dbc0 0x5639d7e6c910 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.582+0000 7f81223df740 1 -- 172.21.15.43:0/2609946529 >> 172.21.15.43:0/2609946529 conn(0x5639d7e33a10 msgr2=0x5639d7e67930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:37.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.582+0000 7f81223df740 1 -- 172.21.15.43:0/2609946529 shutdown_connections 2024-03-28T22:46:37.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.582+0000 7f811ce00640 1 --2- 172.21.15.43:0/1362877485 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f81000a1f70 0x7f81000a4430 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f810c009900 tx=0x7f810c02df90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:37.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.582+0000 7f81223df740 1 -- 172.21.15.43:0/2609946529 wait complete. 2024-03-28T22:46:37.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.582+0000 7f81223df740 1 -- start start 2024-03-28T22:46:37.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.583+0000 7f81223df740 1 Processor -- start 2024-03-28T22:46:37.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.583+0000 7f81223df740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639d7e6cee0 0x5639d7e73280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:37.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.583+0000 7f81223df740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639d7e6dbc0 0x5639d7e737c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:37.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.583+0000 7f81223df740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639d7e76250 0x5639d7e73d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:37.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.583+0000 7f81223df740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5639d7e790c0 con 0x5639d7e76250 2024-03-28T22:46:37.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.583+0000 7f81223df740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5639d7d6f250 con 0x5639d7e6cee0 2024-03-28T22:46:37.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.583+0000 7f81223df740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5639d7e79240 con 0x5639d7e6dbc0 2024-03-28T22:46:37.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.583+0000 7f81223df740 10 client.0 _ll_register_callbacks cb 0x5639d7ae4be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:46:37.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.583+0000 7f811ce00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639d7e6cee0 0x5639d7e73280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:37.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.583+0000 7f8117e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639d7e6dbc0 0x5639d7e737c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:37.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.584+0000 7f8117e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639d7e6dbc0 0x5639d7e737c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:39940/0 (socket says 172.21.15.43:39940) 2024-03-28T22:46:37.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.584+0000 7f811ce00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639d7e6cee0 0x5639d7e73280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:42452/0 (socket says 172.21.15.43:42452) 2024-03-28T22:46:37.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.584+0000 7f8117e00640 1 -- 172.21.15.43:0/3581178910 learned_addr learned my addr 172.21.15.43:0/3581178910 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:37.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.584+0000 7f811d800640 1 --2- 172.21.15.43:0/3581178910 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639d7e76250 0x5639d7e73d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:37.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.584+0000 7f8117e00640 1 -- 172.21.15.43:0/3581178910 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639d7e6cee0 msgr2=0x5639d7e73280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:37.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.584+0000 7f8117e00640 1 --2- 172.21.15.43:0/3581178910 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639d7e6cee0 0x5639d7e73280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.584+0000 7f8117e00640 1 -- 172.21.15.43:0/3581178910 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639d7e76250 msgr2=0x5639d7e73d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:37.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.584+0000 7f8117e00640 1 --2- 172.21.15.43:0/3581178910 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639d7e76250 0x5639d7e73d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.584+0000 7f8117e00640 1 -- 172.21.15.43:0/3581178910 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639d7d8b960 con 0x5639d7e6dbc0 2024-03-28T22:46:37.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.584+0000 7f811d800640 1 --2- 172.21.15.43:0/3581178910 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639d7e76250 0x5639d7e73d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:46:37.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.584+0000 7f8117e00640 1 --2- 172.21.15.43:0/3581178910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639d7e6dbc0 0x5639d7e737c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f810400c6a0 tx=0x7f810409aa40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91c5a14c44ad26af server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:37.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.584+0000 7f80fea00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:46:37.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.584+0000 7f80fea00640 1 -- 172.21.15.43:0/3581178910 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81040cc040 con 0x5639d7e6dbc0 2024-03-28T22:46:37.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.584+0000 7f80fea00640 1 -- 172.21.15.43:0/3581178910 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9539+0+0 (secure 0 0 0) 0x7f81040ce400 con 0x5639d7e6dbc0 2024-03-28T22:46:37.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.584+0000 7f81223df740 1 -- 172.21.15.43:0/3581178910 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8110004050 con 0x5639d7e6dbc0 2024-03-28T22:46:37.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.585+0000 7f80fea00640 1 -- 172.21.15.43:0/3581178910 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f810409efa0 con 0x5639d7e6dbc0 2024-03-28T22:46:37.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.585+0000 7f81223df740 10 client.28723 fetch_fsmap learned FSMap version 32 2024-03-28T22:46:37.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.585+0000 7f81223df740 1 -- 172.21.15.43:0/3581178910 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5639d7baf7d0 con 0x5639d7e6dbc0 2024-03-28T22:46:37.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.585+0000 7f80fea00640 1 -- 172.21.15.43:0/3581178910 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8110004050 con 0x5639d7e6dbc0 2024-03-28T22:46:37.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.585+0000 7f80fea00640 1 -- 172.21.15.43:0/3581178910 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f8104018b30 con 0x5639d7e6dbc0 2024-03-28T22:46:37.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.585+0000 7f81223df740 10 client.28723 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:46:37.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.585+0000 7f81223df740 10 client.28723 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:46:37.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.585+0000 7f81223df740 20 client.28723 populate_metadata read hostname 'smithi043' 2024-03-28T22:46:37.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.585+0000 7f81223df740 1 --2- 172.21.15.43:0/3581178910 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5639d7e86720 0x5639d7e88b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:37.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.585+0000 7f81223df740 4 client.28723 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:46:37.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.585+0000 7f81223df740 1 -- 172.21.15.43:0/3581178910 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5639d7dde640 con 0x5639d7e86720 2024-03-28T22:46:37.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.585+0000 7f811ce00640 1 --2- 172.21.15.43:0/3581178910 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5639d7e86720 0x5639d7e88b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:37.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.586+0000 7f811ce00640 1 --2- 172.21.15.43:0/3581178910 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5639d7e86720 0x5639d7e88b10 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=613989c18c218268 server_cookie=23c97ce9f32a81a6 in_seq=0 out_seq=0 2024-03-28T22:46:37.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.586+0000 7f80fea00640 10 client.28723 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:46:37.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.587+0000 7f80fea00640 1 -- 172.21.15.43:0/3581178910 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5639d7dde640 con 0x5639d7e86720 2024-03-28T22:46:37.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.587+0000 7f80fea00640 10 client.28723 handle_command_reply: tid=0 2024-03-28T22:46:37.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.587+0000 7f81223df740 1 client.28723 shutdown 2024-03-28T22:46:37.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.587+0000 7f81223df740 1 -- 172.21.15.43:0/3581178910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639d7e6dbc0 msgr2=0x5639d7e737c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:37.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.587+0000 7f81223df740 1 --2- 172.21.15.43:0/3581178910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639d7e6dbc0 0x5639d7e737c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f810400c6a0 tx=0x7f810409aa40 comp rx=0 tx=0).stop 2024-03-28T22:46:37.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.587+0000 7f81223df740 1 -- 172.21.15.43:0/3581178910 shutdown_connections 2024-03-28T22:46:37.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.587+0000 7f81223df740 1 --2- 172.21.15.43:0/3581178910 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639d7e76250 0x5639d7e73d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.587+0000 7f81223df740 1 --2- 172.21.15.43:0/3581178910 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639d7e6dbc0 0x5639d7e737c0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.587+0000 7f81223df740 1 --2- 172.21.15.43:0/3581178910 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5639d7e86720 0x5639d7e88b10 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.587+0000 7f81223df740 1 --2- 172.21.15.43:0/3581178910 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639d7e6cee0 0x5639d7e73280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.587+0000 7f81223df740 1 -- 172.21.15.43:0/3581178910 >> 172.21.15.43:0/3581178910 conn(0x5639d7e33a10 msgr2=0x5639d7e69680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:37.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.587+0000 7f80fea00640 0 client.28723 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:46:37.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.587+0000 7f81223df740 1 -- 172.21.15.43:0/3581178910 shutdown_connections 2024-03-28T22:46:37.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.587+0000 7f81223df740 1 -- 172.21.15.43:0/3581178910 wait complete. 2024-03-28T22:46:37.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.587+0000 7f81223df740 20 client.28723 trim_cache size 0 max 16384 2024-03-28T22:46:37.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.607+0000 7f81223df740 1 Processor -- start 2024-03-28T22:46:37.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.607+0000 7f81223df740 1 -- start start 2024-03-28T22:46:37.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.607+0000 7f81223df740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639d7e6cee0 0x5639d7e75940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:37.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.607+0000 7f81223df740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639d7e6dbc0 0x5639d7e81ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:37.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.607+0000 7f81223df740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639d7e76250 0x5639d7e69420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:37.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.607+0000 7f81223df740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f81181b0e90 con 0x5639d7e6dbc0 2024-03-28T22:46:37.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.607+0000 7f81223df740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f81181b29d0 con 0x5639d7e76250 2024-03-28T22:46:37.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.607+0000 7f81223df740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f81181b1560 con 0x5639d7e6cee0 2024-03-28T22:46:37.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.607+0000 7f811ce00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639d7e6cee0 0x5639d7e75940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:37.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.607+0000 7f811ce00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639d7e6cee0 0x5639d7e75940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:39942/0 (socket says 172.21.15.43:39942) 2024-03-28T22:46:37.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.607+0000 7f811ce00640 1 -- 172.21.15.43:0/3901869040 learned_addr learned my addr 172.21.15.43:0/3901869040 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:37.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.607+0000 7f811d800640 1 --2- 172.21.15.43:0/3901869040 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639d7e76250 0x5639d7e69420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:37.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.608+0000 7f8117e00640 1 --2- 172.21.15.43:0/3901869040 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639d7e6dbc0 0x5639d7e81ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:37.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.608+0000 7f811ce00640 1 -- 172.21.15.43:0/3901869040 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639d7e76250 msgr2=0x5639d7e69420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:37.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.608+0000 7f811ce00640 1 --2- 172.21.15.43:0/3901869040 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639d7e76250 0x5639d7e69420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.608+0000 7f811ce00640 1 -- 172.21.15.43:0/3901869040 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639d7e6dbc0 msgr2=0x5639d7e81ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:37.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.608+0000 7f811ce00640 1 --2- 172.21.15.43:0/3901869040 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639d7e6dbc0 0x5639d7e81ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.608+0000 7f811ce00640 1 -- 172.21.15.43:0/3901869040 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639d7ae53f0 con 0x5639d7e6cee0 2024-03-28T22:46:37.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.608+0000 7f8117e00640 1 --2- 172.21.15.43:0/3901869040 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639d7e6dbc0 0x5639d7e81ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:46:37.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.608+0000 7f811ce00640 1 --2- 172.21.15.43:0/3901869040 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639d7e6cee0 0x5639d7e75940 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f810c02c240 tx=0x7f810c042000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b426541db02270a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:37.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.608+0000 7f8109600640 1 -- 172.21.15.43:0/3901869040 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f810c046070 con 0x5639d7e6cee0 2024-03-28T22:46:37.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.608+0000 7f8109600640 1 -- 172.21.15.43:0/3901869040 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f810c0046c0 con 0x5639d7e6cee0 2024-03-28T22:46:37.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.608+0000 7f8109600640 1 -- 172.21.15.43:0/3901869040 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f810c005280 con 0x5639d7e6cee0 2024-03-28T22:46:37.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.609+0000 7f81223df740 1 -- 172.21.15.43:0/3901869040 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639d7e6cee0 msgr2=0x5639d7e75940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:37.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.609+0000 7f81223df740 1 --2- 172.21.15.43:0/3901869040 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639d7e6cee0 0x5639d7e75940 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f810c02c240 tx=0x7f810c042000 comp rx=0 tx=0).stop 2024-03-28T22:46:37.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.609+0000 7f81223df740 1 -- 172.21.15.43:0/3901869040 shutdown_connections 2024-03-28T22:46:37.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.609+0000 7f81223df740 1 --2- 172.21.15.43:0/3901869040 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639d7e76250 0x5639d7e69420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.609+0000 7f81223df740 1 --2- 172.21.15.43:0/3901869040 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639d7e6dbc0 0x5639d7e81ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.609+0000 7f81223df740 1 --2- 172.21.15.43:0/3901869040 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639d7e6cee0 0x5639d7e75940 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.609+0000 7f81223df740 1 -- 172.21.15.43:0/3901869040 >> 172.21.15.43:0/3901869040 conn(0x5639d7e33a10 msgr2=0x5639d7e68d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:37.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.609+0000 7f81223df740 1 -- 172.21.15.43:0/3901869040 shutdown_connections 2024-03-28T22:46:37.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.609+0000 7f81223df740 1 -- 172.21.15.43:0/3901869040 wait complete. 2024-03-28T22:46:37.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.610+0000 7f81223df740 1 -- start start 2024-03-28T22:46:37.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.610+0000 7f81223df740 1 Processor -- start 2024-03-28T22:46:37.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.610+0000 7f81223df740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639d7e6cee0 0x5639d7e76ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:37.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.610+0000 7f81223df740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639d7e6dbc0 0x5639d7e77400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:37.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.610+0000 7f81223df740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639d7e76250 0x5639d7e77940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:37.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.610+0000 7f81223df740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f81181b2300 con 0x5639d7e76250 2024-03-28T22:46:37.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.610+0000 7f81223df740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f81181b1c30 con 0x5639d7e6dbc0 2024-03-28T22:46:37.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.610+0000 7f81223df740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f81181aee90 con 0x5639d7e6cee0 2024-03-28T22:46:37.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.610+0000 7f81223df740 10 client.0 _ll_register_callbacks cb 0x7f810400cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:46:37.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.611+0000 7f811ce00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639d7e6cee0 0x5639d7e76ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:37.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.611+0000 7f811ce00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639d7e6cee0 0x5639d7e76ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:39944/0 (socket says 172.21.15.43:39944) 2024-03-28T22:46:37.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.611+0000 7f811ce00640 1 -- 172.21.15.43:0/1761723540 learned_addr learned my addr 172.21.15.43:0/1761723540 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:37.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.611+0000 7f8117e00640 1 --2- 172.21.15.43:0/1761723540 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639d7e6dbc0 0x5639d7e77400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:37.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.611+0000 7f811d800640 1 --2- 172.21.15.43:0/1761723540 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639d7e76250 0x5639d7e77940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:37.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.611+0000 7f811ce00640 1 -- 172.21.15.43:0/1761723540 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639d7e6dbc0 msgr2=0x5639d7e77400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:37.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.611+0000 7f811ce00640 1 --2- 172.21.15.43:0/1761723540 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639d7e6dbc0 0x5639d7e77400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.611+0000 7f811ce00640 1 -- 172.21.15.43:0/1761723540 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639d7e76250 msgr2=0x5639d7e77940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:37.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.611+0000 7f811ce00640 1 --2- 172.21.15.43:0/1761723540 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639d7e76250 0x5639d7e77940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.611+0000 7f811ce00640 1 -- 172.21.15.43:0/1761723540 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639d7aa50f0 con 0x5639d7e6cee0 2024-03-28T22:46:37.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.611+0000 7f811ce00640 1 --2- 172.21.15.43:0/1761723540 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639d7e6cee0 0x5639d7e76ec0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f810c004350 tx=0x7f810c02ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9d4b6a910f813a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:37.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.611+0000 7f80fea00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:46:37.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.612+0000 7f80fea00640 1 -- 172.21.15.43:0/1761723540 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f810c004c80 con 0x5639d7e6cee0 2024-03-28T22:46:37.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.612+0000 7f80fea00640 1 -- 172.21.15.43:0/1761723540 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9539+0+0 (secure 0 0 0) 0x7f810c004ee0 con 0x5639d7e6cee0 2024-03-28T22:46:37.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.612+0000 7f81223df740 1 -- 172.21.15.43:0/1761723540 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f810c0046c0 con 0x5639d7e6cee0 2024-03-28T22:46:37.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.612+0000 7f80fea00640 1 -- 172.21.15.43:0/1761723540 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f810c02cd10 con 0x5639d7e6cee0 2024-03-28T22:46:37.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.612+0000 7f81223df740 10 client.28729 fetch_fsmap learned FSMap version 32 2024-03-28T22:46:37.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.612+0000 7f81223df740 1 -- 172.21.15.43:0/1761723540 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5639d7eecd40 con 0x5639d7e6cee0 2024-03-28T22:46:37.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.612+0000 7f80fea00640 1 -- 172.21.15.43:0/1761723540 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f810c0046c0 con 0x5639d7e6cee0 2024-03-28T22:46:37.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.612+0000 7f80fea00640 1 -- 172.21.15.43:0/1761723540 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f810c058560 con 0x5639d7e6cee0 2024-03-28T22:46:37.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.612+0000 7f81223df740 10 client.28729 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:46:37.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.612+0000 7f81223df740 10 client.28729 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:46:37.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.612+0000 7f81223df740 20 client.28729 populate_metadata read hostname 'smithi043' 2024-03-28T22:46:37.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.612+0000 7f81223df740 1 --2- 172.21.15.43:0/1761723540 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5639d7eed430 0x5639d7eef850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:37.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.612+0000 7f81223df740 4 client.28729 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:46:37.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.612+0000 7f81223df740 1 -- 172.21.15.43:0/1761723540 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5639d7dd8990 con 0x5639d7eed430 2024-03-28T22:46:37.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.613+0000 7f8117e00640 1 --2- 172.21.15.43:0/1761723540 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5639d7eed430 0x5639d7eef850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:37.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.613+0000 7f8117e00640 1 --2- 172.21.15.43:0/1761723540 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5639d7eed430 0x5639d7eef850 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4477caf117b53e1 server_cookie=dca95d1c739df292 in_seq=0 out_seq=0 2024-03-28T22:46:37.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.613+0000 7f80fea00640 10 client.28729 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:46:37.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.613+0000 7f80fea00640 1 -- 172.21.15.43:0/1761723540 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5639d7dd8990 con 0x5639d7eed430 2024-03-28T22:46:37.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.613+0000 7f80fea00640 10 client.28729 handle_command_reply: tid=0 2024-03-28T22:46:37.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.613+0000 7f81223df740 1 client.28729 shutdown 2024-03-28T22:46:37.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.614+0000 7f81223df740 1 -- 172.21.15.43:0/1761723540 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639d7e6cee0 msgr2=0x5639d7e76ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:37.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.614+0000 7f81223df740 1 --2- 172.21.15.43:0/1761723540 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639d7e6cee0 0x5639d7e76ec0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f810c004350 tx=0x7f810c02ca30 comp rx=0 tx=0).stop 2024-03-28T22:46:37.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.614+0000 7f81223df740 1 -- 172.21.15.43:0/1761723540 shutdown_connections 2024-03-28T22:46:37.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.614+0000 7f81223df740 1 --2- 172.21.15.43:0/1761723540 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5639d7eed430 0x5639d7eef850 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.614+0000 7f81223df740 1 --2- 172.21.15.43:0/1761723540 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5639d7e6dbc0 0x5639d7e77400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.614+0000 7f81223df740 1 --2- 172.21.15.43:0/1761723540 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5639d7e76250 0x5639d7e77940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.614+0000 7f80fea00640 0 client.28729 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:46:37.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.614+0000 7f81223df740 1 --2- 172.21.15.43:0/1761723540 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5639d7e6cee0 0x5639d7e76ec0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.614+0000 7f81223df740 1 -- 172.21.15.43:0/1761723540 >> 172.21.15.43:0/1761723540 conn(0x5639d7e33a10 msgr2=0x5639d7e695a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:37.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.614+0000 7f81223df740 1 -- 172.21.15.43:0/1761723540 shutdown_connections 2024-03-28T22:46:37.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.614+0000 7f81223df740 1 -- 172.21.15.43:0/1761723540 wait complete. 2024-03-28T22:46:37.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.614+0000 7f81223df740 20 client.28729 trim_cache size 0 max 16384 2024-03-28T22:46:37.623 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:46:37.623 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-03-28T22:46:37.624 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-03-28T22:46:37.624 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:46:37.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.615+0000 7f811f800640 1 -- 172.21.15.43:0/1362877485 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f81000a1f70 msgr2=0x7f81000a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:37.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.615+0000 7f811f800640 1 --2- 172.21.15.43:0/1362877485 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f81000a1f70 0x7f81000a4430 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f810c009900 tx=0x7f810c02df90 comp rx=0 tx=0).stop 2024-03-28T22:46:37.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.615+0000 7f811f800640 1 -- 172.21.15.43:0/1362877485 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81181042b0 msgr2=0x7f811819a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:37.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.615+0000 7f811f800640 1 --2- 172.21.15.43:0/1362877485 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81181042b0 0x7f811819a780 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f810400e9f0 tx=0x7f810400eeb0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.615+0000 7f811f800640 1 -- 172.21.15.43:0/1362877485 shutdown_connections 2024-03-28T22:46:37.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.615+0000 7f811f800640 1 --2- 172.21.15.43:0/1362877485 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f81000a1f70 0x7f81000a4430 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.615+0000 7f811f800640 1 --2- 172.21.15.43:0/1362877485 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8118109900 0x7f811819eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.615+0000 7f811f800640 1 --2- 172.21.15.43:0/1362877485 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f81181042b0 0x7f811819a780 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.615+0000 7f811f800640 1 --2- 172.21.15.43:0/1362877485 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f81181038f0 0x7f811819a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.615+0000 7f811f800640 1 -- 172.21.15.43:0/1362877485 >> 172.21.15.43:0/1362877485 conn(0x7f81180ff3a0 msgr2=0x7f81180ffb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:37.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.615+0000 7f811f800640 1 -- 172.21.15.43:0/1362877485 shutdown_connections 2024-03-28T22:46:37.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.615+0000 7f811f800640 1 -- 172.21.15.43:0/1362877485 wait complete. 2024-03-28T22:46:37.631 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:46:37.631 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:46:37.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.847+0000 7fbf25e00640 1 -- 172.21.15.43:0/4113156456 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf20109910 msgr2=0x7fbf20109cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:37.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.847+0000 7fbf25e00640 1 --2- 172.21.15.43:0/4113156456 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf20109910 0x7fbf20109cf0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fbf10009930 tx=0x7fbf1002bc20 comp rx=0 tx=0).stop 2024-03-28T22:46:37.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.847+0000 7fbf25e00640 1 -- 172.21.15.43:0/4113156456 shutdown_connections 2024-03-28T22:46:37.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.847+0000 7fbf25e00640 1 --2- 172.21.15.43:0/4113156456 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbf20103900 0x7fbf20103d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.847+0000 7fbf25e00640 1 --2- 172.21.15.43:0/4113156456 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf201042c0 0x7fbf201107d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.847+0000 7fbf25e00640 1 --2- 172.21.15.43:0/4113156456 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf20109910 0x7fbf20109cf0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.847+0000 7fbf25e00640 1 -- 172.21.15.43:0/4113156456 >> 172.21.15.43:0/4113156456 conn(0x7fbf200ff390 msgr2=0x7fbf201017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:37.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.847+0000 7fbf25e00640 1 -- 172.21.15.43:0/4113156456 shutdown_connections 2024-03-28T22:46:37.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.847+0000 7fbf25e00640 1 -- 172.21.15.43:0/4113156456 wait complete. 2024-03-28T22:46:37.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.848+0000 7fbf25e00640 1 Processor -- start 2024-03-28T22:46:37.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.848+0000 7fbf25e00640 1 -- start start 2024-03-28T22:46:37.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.848+0000 7fbf25e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbf20103900 0x7fbf2019a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:37.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.848+0000 7fbf25e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf201042c0 0x7fbf2019a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:37.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.848+0000 7fbf25e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf20109910 0x7fbf2019eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:37.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.848+0000 7fbf25e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbf20113d70 con 0x7fbf20109910 2024-03-28T22:46:37.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.849+0000 7fbf25e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fbf20113bf0 con 0x7fbf20103900 2024-03-28T22:46:37.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.849+0000 7fbf25e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fbf20113ef0 con 0x7fbf201042c0 2024-03-28T22:46:37.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.849+0000 7fbf1ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbf20103900 0x7fbf2019a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:37.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.849+0000 7fbf1e000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf201042c0 0x7fbf2019a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:37.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.849+0000 7fbf1e000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf201042c0 0x7fbf2019a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:39954/0 (socket says 172.21.15.43:39954) 2024-03-28T22:46:37.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.849+0000 7fbf1e000640 1 -- 172.21.15.43:0/230204113 learned_addr learned my addr 172.21.15.43:0/230204113 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:37.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.849+0000 7fbf1ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbf20103900 0x7fbf2019a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:42482/0 (socket says 172.21.15.43:42482) 2024-03-28T22:46:37.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.849+0000 7fbf1f400640 1 --2- 172.21.15.43:0/230204113 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf20109910 0x7fbf2019eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:37.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.849+0000 7fbf1ea00640 1 -- 172.21.15.43:0/230204113 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf201042c0 msgr2=0x7fbf2019a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:37.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.849+0000 7fbf1ea00640 1 --2- 172.21.15.43:0/230204113 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf201042c0 0x7fbf2019a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.849+0000 7fbf1ea00640 1 -- 172.21.15.43:0/230204113 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf20109910 msgr2=0x7fbf2019eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:37.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.849+0000 7fbf1ea00640 1 --2- 172.21.15.43:0/230204113 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf20109910 0x7fbf2019eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:37.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.849+0000 7fbf1ea00640 1 -- 172.21.15.43:0/230204113 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf2019f220 con 0x7fbf20103900 2024-03-28T22:46:37.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.849+0000 7fbf1e000640 1 --2- 172.21.15.43:0/230204113 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf201042c0 0x7fbf2019a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:37.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.850+0000 7fbf1ea00640 1 --2- 172.21.15.43:0/230204113 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbf20103900 0x7fbf2019a230 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fbf10036040 tx=0x7fbf10004250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:37.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.850+0000 7fbf24a00640 1 -- 172.21.15.43:0/230204113 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf10043020 con 0x7fbf20103900 2024-03-28T22:46:37.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.850+0000 7fbf24a00640 1 -- 172.21.15.43:0/230204113 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf10005000 con 0x7fbf20103900 2024-03-28T22:46:37.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.850+0000 7fbf25e00640 1 -- 172.21.15.43:0/230204113 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf2019f4b0 con 0x7fbf20103900 2024-03-28T22:46:37.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.850+0000 7fbf24a00640 1 -- 172.21.15.43:0/230204113 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf10003a80 con 0x7fbf20103900 2024-03-28T22:46:37.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.850+0000 7fbf25e00640 1 -- 172.21.15.43:0/230204113 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf201a6da0 con 0x7fbf20103900 2024-03-28T22:46:37.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.851+0000 7fbf25e00640 1 -- 172.21.15.43:0/230204113 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf20068540 con 0x7fbf20103900 2024-03-28T22:46:37.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.853+0000 7fbf24a00640 1 -- 172.21.15.43:0/230204113 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fbf10046020 con 0x7fbf20103900 2024-03-28T22:46:37.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.855+0000 7fbf24a00640 1 --2- 172.21.15.43:0/230204113 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fbf040a1f70 0x7fbf040a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:37.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.855+0000 7fbf24a00640 1 -- 172.21.15.43:0/230204113 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9539+0+0 (secure 0 0 0) 0x7fbf100f02b0 con 0x7fbf20103900 2024-03-28T22:46:37.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.855+0000 7fbf1e000640 1 --2- 172.21.15.43:0/230204113 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fbf040a1f70 0x7fbf040a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:37.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.856+0000 7fbf1e000640 1 --2- 172.21.15.43:0/230204113 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fbf040a1f70 0x7fbf040a4430 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbf20101a20 tx=0x7fbf0c005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:37.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:37.860+0000 7fbf24a00640 1 -- 172.21.15.43:0/230204113 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbf1004d020 con 0x7fbf20103900 2024-03-28T22:46:38.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.113+0000 7fbf25e00640 1 -- 172.21.15.43:0/230204113 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf2010be70 con 0x7fbf20103900 2024-03-28T22:46:38.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.114+0000 7fbf24a00640 1 -- 172.21.15.43:0/230204113 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11931 (secure 0 0 0) 0x7fbf100bd050 con 0x7fbf20103900 2024-03-28T22:46:38.113 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:46:38.113 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:33:28.906396+0000","modified":"2024-03-28T22:40:08.672168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24572,"mds_1":24556,"mds_2":24535,"mds_3":24523,"mds_4":24518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26717":{"gid":26717,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/215779342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":215779342},{"type":"v1","addr":"172.21.15.82:6835","nonce":215779342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24535,24556,24518,24523,24572]},"id":1}]} 2024-03-28T22:46:38.113 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 32 2024-03-28T22:46:38.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.117+0000 7fbf25e00640 1 -- 172.21.15.43:0/230204113 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fbf040a1f70 msgr2=0x7fbf040a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:38.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.117+0000 7fbf25e00640 1 --2- 172.21.15.43:0/230204113 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fbf040a1f70 0x7fbf040a4430 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbf20101a20 tx=0x7fbf0c005f90 comp rx=0 tx=0).stop 2024-03-28T22:46:38.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.117+0000 7fbf25e00640 1 -- 172.21.15.43:0/230204113 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbf20103900 msgr2=0x7fbf2019a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:38.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.117+0000 7fbf25e00640 1 --2- 172.21.15.43:0/230204113 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbf20103900 0x7fbf2019a230 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fbf10036040 tx=0x7fbf10004250 comp rx=0 tx=0).stop 2024-03-28T22:46:38.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.117+0000 7fbf25e00640 1 -- 172.21.15.43:0/230204113 shutdown_connections 2024-03-28T22:46:38.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.117+0000 7fbf25e00640 1 --2- 172.21.15.43:0/230204113 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fbf040a1f70 0x7fbf040a4430 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.117+0000 7fbf25e00640 1 --2- 172.21.15.43:0/230204113 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbf20109910 0x7fbf2019eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.117+0000 7fbf25e00640 1 --2- 172.21.15.43:0/230204113 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fbf201042c0 0x7fbf2019a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.117+0000 7fbf25e00640 1 --2- 172.21.15.43:0/230204113 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbf20103900 0x7fbf2019a230 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.117+0000 7fbf25e00640 1 -- 172.21.15.43:0/230204113 >> 172.21.15.43:0/230204113 conn(0x7fbf200ff390 msgr2=0x7fbf200ffb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:38.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.117+0000 7fbf25e00640 1 -- 172.21.15.43:0/230204113 shutdown_connections 2024-03-28T22:46:38.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.117+0000 7fbf25e00640 1 -- 172.21.15.43:0/230204113 wait complete. 2024-03-28T22:46:38.133 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T22:46:38.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.360+0000 7fcbd3c00640 1 -- 172.21.15.43:0/931783631 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcbcc112590 msgr2=0x7fcbcc114980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:38.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.360+0000 7fcbd3c00640 1 --2- 172.21.15.43:0/931783631 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcbcc112590 0x7fcbcc114980 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4009c10 tx=0x7fcbc402bef0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.360+0000 7fcbd3c00640 1 -- 172.21.15.43:0/931783631 shutdown_connections 2024-03-28T22:46:38.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.360+0000 7fcbd3c00640 1 --2- 172.21.15.43:0/931783631 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcbcc112590 0x7fcbcc114980 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.360+0000 7fcbd3c00640 1 --2- 172.21.15.43:0/931783631 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbcc0765a0 0x7fcbcc112050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.360+0000 7fcbd3c00640 1 --2- 172.21.15.43:0/931783631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcbcc075c80 0x7fcbcc076060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.360+0000 7fcbd3c00640 1 -- 172.21.15.43:0/931783631 >> 172.21.15.43:0/931783631 conn(0x7fcbcc0ff3a0 msgr2=0x7fcbcc1017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:38.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.360+0000 7fcbd3c00640 1 -- 172.21.15.43:0/931783631 shutdown_connections 2024-03-28T22:46:38.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.360+0000 7fcbd3c00640 1 -- 172.21.15.43:0/931783631 wait complete. 2024-03-28T22:46:38.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.361+0000 7fcbd3c00640 1 Processor -- start 2024-03-28T22:46:38.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.363+0000 7fcbd3c00640 1 -- start start 2024-03-28T22:46:38.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.363+0000 7fcbd3c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbcc075c80 0x7fcbcc19e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.363+0000 7fcbd3c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcbcc0765a0 0x7fcbcc19ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.363+0000 7fcbd3c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcbcc112590 0x7fcbcc1a2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.363+0000 7fcbd3c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc118100 con 0x7fcbcc112590 2024-03-28T22:46:38.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.363+0000 7fcbd3c00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc117f80 con 0x7fcbcc0765a0 2024-03-28T22:46:38.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.363+0000 7fcbd3c00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc118280 con 0x7fcbcc075c80 2024-03-28T22:46:38.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.363+0000 7fcbd1200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbcc075c80 0x7fcbcc19e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.363+0000 7fcbcbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcbcc0765a0 0x7fcbcc19ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.363+0000 7fcbcbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcbcc0765a0 0x7fcbcc19ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:42502/0 (socket says 172.21.15.43:42502) 2024-03-28T22:46:38.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.363+0000 7fcbcbe00640 1 -- 172.21.15.43:0/4213808809 learned_addr learned my addr 172.21.15.43:0/4213808809 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:38.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.363+0000 7fcbd1200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbcc075c80 0x7fcbcc19e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:39980/0 (socket says 172.21.15.43:39980) 2024-03-28T22:46:38.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.363+0000 7fcbd1c00640 1 --2- 172.21.15.43:0/4213808809 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcbcc112590 0x7fcbcc1a2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.363+0000 7fcbcbe00640 1 -- 172.21.15.43:0/4213808809 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbcc075c80 msgr2=0x7fcbcc19e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:38.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.363+0000 7fcbcbe00640 1 --2- 172.21.15.43:0/4213808809 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbcc075c80 0x7fcbcc19e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.363+0000 7fcbcbe00640 1 -- 172.21.15.43:0/4213808809 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcbcc112590 msgr2=0x7fcbcc1a2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:38.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.363+0000 7fcbcbe00640 1 --2- 172.21.15.43:0/4213808809 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcbcc112590 0x7fcbcc1a2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.363+0000 7fcbcbe00640 1 -- 172.21.15.43:0/4213808809 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbcc1a3670 con 0x7fcbcc0765a0 2024-03-28T22:46:38.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.363+0000 7fcbd1c00640 1 --2- 172.21.15.43:0/4213808809 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcbcc112590 0x7fcbcc1a2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:46:38.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.363+0000 7fcbd1200640 1 --2- 172.21.15.43:0/4213808809 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbcc075c80 0x7fcbcc19e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:38.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.363+0000 7fcbcbe00640 1 --2- 172.21.15.43:0/4213808809 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcbcc0765a0 0x7fcbcc19ebc0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fcbb800e9c0 tx=0x7fcbb800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:38.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.363+0000 7fcbc9600640 1 -- 172.21.15.43:0/4213808809 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb800cde0 con 0x7fcbcc0765a0 2024-03-28T22:46:38.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.363+0000 7fcbc9600640 1 -- 172.21.15.43:0/4213808809 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbb80035f0 con 0x7fcbcc0765a0 2024-03-28T22:46:38.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.363+0000 7fcbc9600640 1 -- 172.21.15.43:0/4213808809 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb8010690 con 0x7fcbcc0765a0 2024-03-28T22:46:38.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.363+0000 7fcbd3c00640 1 -- 172.21.15.43:0/4213808809 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbcc1a3960 con 0x7fcbcc0765a0 2024-03-28T22:46:38.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.364+0000 7fcbd3c00640 1 -- 172.21.15.43:0/4213808809 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbcc1ab1f0 con 0x7fcbcc0765a0 2024-03-28T22:46:38.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.368+0000 7fcbd69b9740 1 Processor -- start 2024-03-28T22:46:38.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.368+0000 7fcbd69b9740 1 -- start start 2024-03-28T22:46:38.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.368+0000 7fcbd69b9740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb335cd380 0x55fb335cd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.368+0000 7fcbd69b9740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb335cdca0 0x55fb335d6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.368+0000 7fcbc9600640 1 -- 172.21.15.43:0/4213808809 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fcbb80037a0 con 0x7fcbcc0765a0 2024-03-28T22:46:38.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.368+0000 7fcbd69b9740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb335d6880 0x55fb335d8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.368+0000 7fcbd69b9740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55fb33531050 con 0x55fb335d6880 2024-03-28T22:46:38.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.368+0000 7fcbd69b9740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55fb334d9ce0 con 0x55fb335cdca0 2024-03-28T22:46:38.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.368+0000 7fcbd69b9740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55fb3352d0f0 con 0x55fb335cd380 2024-03-28T22:46:38.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.368+0000 7fcbd1200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb335cdca0 0x55fb335d6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.368+0000 7fcbd1200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb335cdca0 0x55fb335d6340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:42518/0 (socket says 172.21.15.43:42518) 2024-03-28T22:46:38.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.368+0000 7fcbd1c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb335cd380 0x55fb335cd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.368+0000 7fcbd1200640 1 -- 172.21.15.43:0/2388872521 learned_addr learned my addr 172.21.15.43:0/2388872521 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:38.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.368+0000 7fcbcbe00640 1 --2- 172.21.15.43:0/2388872521 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb335d6880 0x55fb335d8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.368+0000 7fcbd1200640 1 -- 172.21.15.43:0/2388872521 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb335cd380 msgr2=0x55fb335cd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:38.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.368+0000 7fcbd1200640 1 --2- 172.21.15.43:0/2388872521 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb335cd380 0x55fb335cd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.368+0000 7fcbd1200640 1 -- 172.21.15.43:0/2388872521 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb335d6880 msgr2=0x55fb335d8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:38.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.368+0000 7fcbd1200640 1 --2- 172.21.15.43:0/2388872521 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb335d6880 0x55fb335d8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.368+0000 7fcbd1200640 1 -- 172.21.15.43:0/2388872521 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb332453f0 con 0x55fb335cdca0 2024-03-28T22:46:38.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.368+0000 7fcbd1200640 1 --2- 172.21.15.43:0/2388872521 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb335cdca0 0x55fb335d6340 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fcbc00115f0 tx=0x7fcbc0011ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6086c760df003f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:38.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.368+0000 7fcbbe000640 1 -- 172.21.15.43:0/2388872521 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc001e070 con 0x55fb335cdca0 2024-03-28T22:46:38.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.369+0000 7fcbbe000640 1 -- 172.21.15.43:0/2388872521 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbc000fb20 con 0x55fb335cdca0 2024-03-28T22:46:38.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.369+0000 7fcbbe000640 1 -- 172.21.15.43:0/2388872521 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0019c40 con 0x55fb335cdca0 2024-03-28T22:46:38.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.369+0000 7fcbc9600640 1 --2- 172.21.15.43:0/4213808809 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcbb40a1f70 0x7fcbb40a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.369+0000 7fcbd69b9740 1 -- 172.21.15.43:0/2388872521 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb335cdca0 msgr2=0x55fb335d6340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:38.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.369+0000 7fcbd69b9740 1 --2- 172.21.15.43:0/2388872521 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb335cdca0 0x55fb335d6340 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fcbc00115f0 tx=0x7fcbc0011ab0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.369+0000 7fcbc9600640 1 -- 172.21.15.43:0/4213808809 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9539+0+0 (secure 0 0 0) 0x7fcbb80ce8d0 con 0x7fcbcc0765a0 2024-03-28T22:46:38.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.369+0000 7fcbd1c00640 1 --2- 172.21.15.43:0/4213808809 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcbb40a1f70 0x7fcbb40a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.369+0000 7fcbd69b9740 1 -- 172.21.15.43:0/2388872521 shutdown_connections 2024-03-28T22:46:38.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.369+0000 7fcbd69b9740 1 --2- 172.21.15.43:0/2388872521 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb335cdca0 0x55fb335d6340 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.369+0000 7fcbd69b9740 1 --2- 172.21.15.43:0/2388872521 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb335d6880 0x55fb335d8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.369+0000 7fcbd69b9740 1 --2- 172.21.15.43:0/2388872521 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb335cd380 0x55fb335cd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.369+0000 7fcbd69b9740 1 -- 172.21.15.43:0/2388872521 >> 172.21.15.43:0/2388872521 conn(0x55fb335c7ab0 msgr2=0x55fb335c7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:38.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.370+0000 7fcbd69b9740 1 -- 172.21.15.43:0/2388872521 shutdown_connections 2024-03-28T22:46:38.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.370+0000 7fcbd69b9740 1 -- 172.21.15.43:0/2388872521 wait complete. 2024-03-28T22:46:38.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.370+0000 7fcbd69b9740 1 -- start start 2024-03-28T22:46:38.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.370+0000 7fcbd1c00640 1 --2- 172.21.15.43:0/4213808809 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcbb40a1f70 0x7fcbb40a4430 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcbc402c430 tx=0x7fcbc4004a40 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:38.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.370+0000 7fcbd69b9740 1 Processor -- start 2024-03-28T22:46:38.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.371+0000 7fcbd69b9740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb335cd380 0x55fb335d3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.371+0000 7fcbd69b9740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb335cdca0 0x55fb335d3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.371+0000 7fcbd69b9740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb335d6880 0x55fb335d42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.371+0000 7fcbd69b9740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55fb335d9670 con 0x55fb335d6880 2024-03-28T22:46:38.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.371+0000 7fcbd69b9740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55fb334cf2f0 con 0x55fb335cdca0 2024-03-28T22:46:38.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.371+0000 7fcbd69b9740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55fb335d97f0 con 0x55fb335cd380 2024-03-28T22:46:38.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.371+0000 7fcbd69b9740 10 client.0 _ll_register_callbacks cb 0x55fb33244be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:46:38.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.371+0000 7fcbcbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb335cdca0 0x55fb335d3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.371+0000 7fcbd1200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb335cd380 0x55fb335d3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.371+0000 7fcbd1c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb335d6880 0x55fb335d42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.371+0000 7fcbcbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb335cdca0 0x55fb335d3d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:42534/0 (socket says 172.21.15.43:42534) 2024-03-28T22:46:38.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.371+0000 7fcbd1200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb335cd380 0x55fb335d3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:39990/0 (socket says 172.21.15.43:39990) 2024-03-28T22:46:38.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.371+0000 7fcbcbe00640 1 -- 172.21.15.43:0/2914163352 learned_addr learned my addr 172.21.15.43:0/2914163352 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:38.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.372+0000 7fcbcbe00640 1 -- 172.21.15.43:0/2914163352 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb335cd380 msgr2=0x55fb335d3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:38.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.372+0000 7fcbcbe00640 1 --2- 172.21.15.43:0/2914163352 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb335cd380 0x55fb335d3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.372+0000 7fcbcbe00640 1 -- 172.21.15.43:0/2914163352 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb335d6880 msgr2=0x55fb335d42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:38.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.372+0000 7fcbcbe00640 1 --2- 172.21.15.43:0/2914163352 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb335d6880 0x55fb335d42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.372+0000 7fcbcbe00640 1 -- 172.21.15.43:0/2914163352 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb3330f7d0 con 0x55fb335cdca0 2024-03-28T22:46:38.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.372+0000 7fcbd1200640 1 --2- 172.21.15.43:0/2914163352 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb335cd380 0x55fb335d3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:38.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.372+0000 7fcbcbe00640 1 --2- 172.21.15.43:0/2914163352 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb335cdca0 0x55fb335d3d80 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fcbb800c6a0 tx=0x7fcbb809aa60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bf786f3df9d9586 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:38.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.372+0000 7fcbb2a00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:46:38.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.372+0000 7fcbb2a00640 1 -- 172.21.15.43:0/2914163352 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb809ae60 con 0x55fb335cdca0 2024-03-28T22:46:38.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.372+0000 7fcbb2a00640 1 -- 172.21.15.43:0/2914163352 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9539+0+0 (secure 0 0 0) 0x7fcbb80cbe40 con 0x55fb335cdca0 2024-03-28T22:46:38.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.372+0000 7fcbd69b9740 1 -- 172.21.15.43:0/2914163352 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbc000fb20 con 0x55fb335cdca0 2024-03-28T22:46:38.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.372+0000 7fcbb2a00640 1 -- 172.21.15.43:0/2914163352 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcbb80b8050 con 0x55fb335cdca0 2024-03-28T22:46:38.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.372+0000 7fcbd69b9740 10 client.28805 fetch_fsmap learned FSMap version 32 2024-03-28T22:46:38.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.372+0000 7fcbd69b9740 1 -- 172.21.15.43:0/2914163352 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55fb335e58e0 con 0x55fb335cdca0 2024-03-28T22:46:38.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.373+0000 7fcbb2a00640 1 -- 172.21.15.43:0/2914163352 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbc000fb20 con 0x55fb335cdca0 2024-03-28T22:46:38.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.373+0000 7fcbb2a00640 1 -- 172.21.15.43:0/2914163352 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fcbb80a3170 con 0x55fb335cdca0 2024-03-28T22:46:38.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.373+0000 7fcbd69b9740 10 client.28805 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:46:38.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.373+0000 7fcbd69b9740 10 client.28805 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:46:38.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.373+0000 7fcbd69b9740 20 client.28805 populate_metadata read hostname 'smithi043' 2024-03-28T22:46:38.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.373+0000 7fcbd69b9740 1 --2- 172.21.15.43:0/2914163352 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55fb335e6ce0 0x55fb335e9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.373+0000 7fcbd69b9740 4 client.28805 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:46:38.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.373+0000 7fcbd69b9740 1 -- 172.21.15.43:0/2914163352 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb334da680 con 0x55fb335e6ce0 2024-03-28T22:46:38.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.373+0000 7fcbd1200640 1 --2- 172.21.15.43:0/2914163352 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55fb335e6ce0 0x55fb335e9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.374+0000 7fcbd1200640 1 --2- 172.21.15.43:0/2914163352 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55fb335e6ce0 0x55fb335e9130 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=650acfffbd918e3e server_cookie=19c4f31dbb56fcc2 in_seq=0 out_seq=0 2024-03-28T22:46:38.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.374+0000 7fcbb2a00640 10 client.28805 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:46:38.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.375+0000 7fcbb2a00640 1 -- 172.21.15.43:0/2914163352 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fb334da680 con 0x55fb335e6ce0 2024-03-28T22:46:38.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.375+0000 7fcbb2a00640 10 client.28805 handle_command_reply: tid=0 2024-03-28T22:46:38.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.375+0000 7fcbd69b9740 1 client.28805 shutdown 2024-03-28T22:46:38.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.375+0000 7fcbd69b9740 1 -- 172.21.15.43:0/2914163352 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb335cdca0 msgr2=0x55fb335d3d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:38.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.375+0000 7fcbd69b9740 1 --2- 172.21.15.43:0/2914163352 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb335cdca0 0x55fb335d3d80 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fcbb800c6a0 tx=0x7fcbb809aa60 comp rx=0 tx=0).stop 2024-03-28T22:46:38.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.375+0000 7fcbd69b9740 1 -- 172.21.15.43:0/2914163352 shutdown_connections 2024-03-28T22:46:38.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.375+0000 7fcbd69b9740 1 --2- 172.21.15.43:0/2914163352 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55fb335e6ce0 0x55fb335e9130 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.375+0000 7fcbd69b9740 1 --2- 172.21.15.43:0/2914163352 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb335cdca0 0x55fb335d3d80 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.375+0000 7fcbd69b9740 1 --2- 172.21.15.43:0/2914163352 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb335d6880 0x55fb335d42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.375+0000 7fcbd69b9740 1 --2- 172.21.15.43:0/2914163352 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb335cd380 0x55fb335d3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.375+0000 7fcbd69b9740 1 -- 172.21.15.43:0/2914163352 >> 172.21.15.43:0/2914163352 conn(0x55fb335c7ab0 msgr2=0x55fb335c9800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:38.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.375+0000 7fcbb2a00640 0 client.28805 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:46:38.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.375+0000 7fcbd69b9740 1 -- 172.21.15.43:0/2914163352 shutdown_connections 2024-03-28T22:46:38.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.375+0000 7fcbd69b9740 1 -- 172.21.15.43:0/2914163352 wait complete. 2024-03-28T22:46:38.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.375+0000 7fcbd69b9740 20 client.28805 trim_cache size 0 max 16384 2024-03-28T22:46:38.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.390+0000 7fcbd69b9740 1 Processor -- start 2024-03-28T22:46:38.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.390+0000 7fcbd69b9740 1 -- start start 2024-03-28T22:46:38.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.390+0000 7fcbd69b9740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb335cd380 0x55fb335d5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.390+0000 7fcbd69b9740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb335cdca0 0x55fb335e2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.390+0000 7fcbd69b9740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb335d6880 0x55fb335c69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.390+0000 7fcbd69b9740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc1b52f0 con 0x55fb335cd380 2024-03-28T22:46:38.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.390+0000 7fcbd69b9740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc1b6e30 con 0x55fb335cdca0 2024-03-28T22:46:38.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.390+0000 7fcbd69b9740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc1b59c0 con 0x55fb335d6880 2024-03-28T22:46:38.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.391+0000 7fcbcbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb335cdca0 0x55fb335e2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.391+0000 7fcbcbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb335cdca0 0x55fb335e2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:42542/0 (socket says 172.21.15.43:42542) 2024-03-28T22:46:38.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.391+0000 7fcbcbe00640 1 -- 172.21.15.43:0/2684573747 learned_addr learned my addr 172.21.15.43:0/2684573747 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:38.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.391+0000 7fcbd1200640 1 --2- 172.21.15.43:0/2684573747 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb335cd380 0x55fb335d5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.391+0000 7fcbd1c00640 1 --2- 172.21.15.43:0/2684573747 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb335d6880 0x55fb335c69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.391+0000 7fcbcbe00640 1 -- 172.21.15.43:0/2684573747 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb335d6880 msgr2=0x55fb335c69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:38.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.391+0000 7fcbcbe00640 1 --2- 172.21.15.43:0/2684573747 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb335d6880 0x55fb335c69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.391+0000 7fcbcbe00640 1 -- 172.21.15.43:0/2684573747 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb335cd380 msgr2=0x55fb335d5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:38.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.391+0000 7fcbcbe00640 1 --2- 172.21.15.43:0/2684573747 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb335cd380 0x55fb335d5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.391+0000 7fcbcbe00640 1 -- 172.21.15.43:0/2684573747 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb335c78e0 con 0x55fb335cdca0 2024-03-28T22:46:38.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.391+0000 7fcbcbe00640 1 --2- 172.21.15.43:0/2684573747 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb335cdca0 0x55fb335e2440 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fcbb8097830 tx=0x7fcbb8097100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59ba473bc24dadfb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:38.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.392+0000 7fcbbd600640 1 -- 172.21.15.43:0/2684573747 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb809ae60 con 0x55fb335cdca0 2024-03-28T22:46:38.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.392+0000 7fcbbd600640 1 -- 172.21.15.43:0/2684573747 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbb807e020 con 0x55fb335cdca0 2024-03-28T22:46:38.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.392+0000 7fcbbd600640 1 -- 172.21.15.43:0/2684573747 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb80ce530 con 0x55fb335cdca0 2024-03-28T22:46:38.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.392+0000 7fcbd69b9740 1 -- 172.21.15.43:0/2684573747 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb335cdca0 msgr2=0x55fb335e2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:38.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.392+0000 7fcbd69b9740 1 --2- 172.21.15.43:0/2684573747 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb335cdca0 0x55fb335e2440 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fcbb8097830 tx=0x7fcbb8097100 comp rx=0 tx=0).stop 2024-03-28T22:46:38.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.392+0000 7fcbd69b9740 1 -- 172.21.15.43:0/2684573747 shutdown_connections 2024-03-28T22:46:38.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.392+0000 7fcbd69b9740 1 --2- 172.21.15.43:0/2684573747 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb335cdca0 0x55fb335e2440 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.392+0000 7fcbd69b9740 1 --2- 172.21.15.43:0/2684573747 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb335d6880 0x55fb335c69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.392+0000 7fcbd69b9740 1 --2- 172.21.15.43:0/2684573747 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb335cd380 0x55fb335d5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.392+0000 7fcbd69b9740 1 -- 172.21.15.43:0/2684573747 >> 172.21.15.43:0/2684573747 conn(0x55fb335c7ab0 msgr2=0x55fb335c62e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:38.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.392+0000 7fcbd69b9740 1 -- 172.21.15.43:0/2684573747 shutdown_connections 2024-03-28T22:46:38.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.392+0000 7fcbd69b9740 1 -- 172.21.15.43:0/2684573747 wait complete. 2024-03-28T22:46:38.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.393+0000 7fcbd69b9740 1 -- start start 2024-03-28T22:46:38.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.393+0000 7fcbd69b9740 1 Processor -- start 2024-03-28T22:46:38.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.393+0000 7fcbd69b9740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb335cd380 0x55fb335da570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.393+0000 7fcbd69b9740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb335cdca0 0x55fb335daab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.393+0000 7fcbd69b9740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb335d6880 0x55fb335daff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.393+0000 7fcbd69b9740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc1b6760 con 0x55fb335cd380 2024-03-28T22:46:38.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.393+0000 7fcbd69b9740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc1b6090 con 0x55fb335cdca0 2024-03-28T22:46:38.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.393+0000 7fcbd69b9740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc1b32f0 con 0x55fb335d6880 2024-03-28T22:46:38.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.393+0000 7fcbd69b9740 10 client.0 _ll_register_callbacks cb 0x7fcbb809ac30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:46:38.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.394+0000 7fcbd1c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb335d6880 0x55fb335daff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.394+0000 7fcbcbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb335cdca0 0x55fb335daab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.394+0000 7fcbd1200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb335cd380 0x55fb335da570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.394+0000 7fcbd1c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb335d6880 0x55fb335daff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:39996/0 (socket says 172.21.15.43:39996) 2024-03-28T22:46:38.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.394+0000 7fcbcbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb335cdca0 0x55fb335daab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:42554/0 (socket says 172.21.15.43:42554) 2024-03-28T22:46:38.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.394+0000 7fcbd1c00640 1 -- 172.21.15.43:0/3702654333 learned_addr learned my addr 172.21.15.43:0/3702654333 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:38.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.394+0000 7fcbcbe00640 1 -- 172.21.15.43:0/3702654333 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb335d6880 msgr2=0x55fb335daff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:38.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.394+0000 7fcbcbe00640 1 --2- 172.21.15.43:0/3702654333 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb335d6880 0x55fb335daff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.394+0000 7fcbcbe00640 1 -- 172.21.15.43:0/3702654333 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb335cd380 msgr2=0x55fb335da570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:38.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.394+0000 7fcbcbe00640 1 --2- 172.21.15.43:0/3702654333 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb335cd380 0x55fb335da570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.394+0000 7fcbcbe00640 1 -- 172.21.15.43:0/3702654333 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb3364c230 con 0x55fb335cdca0 2024-03-28T22:46:38.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.394+0000 7fcbd1c00640 1 --2- 172.21.15.43:0/3702654333 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb335d6880 0x55fb335daff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:38.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.394+0000 7fcbcbe00640 1 --2- 172.21.15.43:0/3702654333 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb335cdca0 0x55fb335daab0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x55fb3359d340 tx=0x7fcbb80ce760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14c021e450e39a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:38.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.394+0000 7fcbb2a00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:46:38.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.395+0000 7fcbb2a00640 1 -- 172.21.15.43:0/3702654333 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb80cc050 con 0x55fb335cdca0 2024-03-28T22:46:38.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.395+0000 7fcbb2a00640 1 -- 172.21.15.43:0/3702654333 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9539+0+0 (secure 0 0 0) 0x7fcbb809a2b0 con 0x55fb335cdca0 2024-03-28T22:46:38.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.395+0000 7fcbd69b9740 1 -- 172.21.15.43:0/3702654333 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbb807e020 con 0x55fb335cdca0 2024-03-28T22:46:38.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.395+0000 7fcbb2a00640 1 -- 172.21.15.43:0/3702654333 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcbb80ac050 con 0x55fb335cdca0 2024-03-28T22:46:38.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.395+0000 7fcbd69b9740 10 client.28811 fetch_fsmap learned FSMap version 32 2024-03-28T22:46:38.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.395+0000 7fcbd69b9740 1 -- 172.21.15.43:0/3702654333 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55fb3364d4d0 con 0x55fb335cdca0 2024-03-28T22:46:38.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.395+0000 7fcbb2a00640 1 -- 172.21.15.43:0/3702654333 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbb807e020 con 0x55fb335cdca0 2024-03-28T22:46:38.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.395+0000 7fcbb2a00640 1 -- 172.21.15.43:0/3702654333 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fcbb8004000 con 0x55fb335cdca0 2024-03-28T22:46:38.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.395+0000 7fcbd69b9740 10 client.28811 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:46:38.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.395+0000 7fcbd69b9740 10 client.28811 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:46:38.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.395+0000 7fcbd69b9740 20 client.28811 populate_metadata read hostname 'smithi043' 2024-03-28T22:46:38.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.395+0000 7fcbd69b9740 1 --2- 172.21.15.43:0/3702654333 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55fb3364d9a0 0x55fb3364fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.395+0000 7fcbd69b9740 4 client.28811 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:46:38.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.395+0000 7fcbd69b9740 1 -- 172.21.15.43:0/3702654333 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fb3353e590 con 0x55fb3364d9a0 2024-03-28T22:46:38.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.396+0000 7fcbd1200640 1 --2- 172.21.15.43:0/3702654333 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55fb3364d9a0 0x55fb3364fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.396+0000 7fcbd1200640 1 --2- 172.21.15.43:0/3702654333 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55fb3364d9a0 0x55fb3364fd90 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=908e0a18f83c160c server_cookie=3990c38fb7ebf5f9 in_seq=0 out_seq=0 2024-03-28T22:46:38.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.396+0000 7fcbb2a00640 10 client.28811 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:46:38.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.396+0000 7fcbb2a00640 1 -- 172.21.15.43:0/3702654333 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fb3353e590 con 0x55fb3364d9a0 2024-03-28T22:46:38.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.396+0000 7fcbb2a00640 10 client.28811 handle_command_reply: tid=0 2024-03-28T22:46:38.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.396+0000 7fcbd69b9740 1 client.28811 shutdown 2024-03-28T22:46:38.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.396+0000 7fcbd69b9740 1 -- 172.21.15.43:0/3702654333 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb335cdca0 msgr2=0x55fb335daab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:38.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.396+0000 7fcbd69b9740 1 --2- 172.21.15.43:0/3702654333 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb335cdca0 0x55fb335daab0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x55fb3359d340 tx=0x7fcbb80ce760 comp rx=0 tx=0).stop 2024-03-28T22:46:38.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.396+0000 7fcbd69b9740 1 -- 172.21.15.43:0/3702654333 shutdown_connections 2024-03-28T22:46:38.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.396+0000 7fcbd69b9740 1 --2- 172.21.15.43:0/3702654333 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55fb3364d9a0 0x55fb3364fd90 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.396+0000 7fcbd69b9740 1 --2- 172.21.15.43:0/3702654333 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb335cdca0 0x55fb335daab0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.396+0000 7fcbd69b9740 1 --2- 172.21.15.43:0/3702654333 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb335d6880 0x55fb335daff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.396+0000 7fcbd69b9740 1 --2- 172.21.15.43:0/3702654333 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb335cd380 0x55fb335da570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.396+0000 7fcbb2a00640 0 client.28811 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:46:38.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.396+0000 7fcbd69b9740 1 -- 172.21.15.43:0/3702654333 >> 172.21.15.43:0/3702654333 conn(0x55fb335c7ab0 msgr2=0x55fb335c6b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:38.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.397+0000 7fcbd69b9740 1 -- 172.21.15.43:0/3702654333 shutdown_connections 2024-03-28T22:46:38.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.397+0000 7fcbd69b9740 1 -- 172.21.15.43:0/3702654333 wait complete. 2024-03-28T22:46:38.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.397+0000 7fcbd69b9740 20 client.28811 trim_cache size 0 max 16384 2024-03-28T22:46:38.405 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:46:38.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.397+0000 7fcbd3c00640 1 -- 172.21.15.43:0/4213808809 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcbb40a1f70 msgr2=0x7fcbb40a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:38.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.397+0000 7fcbd3c00640 1 --2- 172.21.15.43:0/4213808809 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcbb40a1f70 0x7fcbb40a4430 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcbc402c430 tx=0x7fcbc4004a40 comp rx=0 tx=0).stop 2024-03-28T22:46:38.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.397+0000 7fcbd3c00640 1 -- 172.21.15.43:0/4213808809 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcbcc0765a0 msgr2=0x7fcbcc19ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:38.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.397+0000 7fcbd3c00640 1 --2- 172.21.15.43:0/4213808809 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcbcc0765a0 0x7fcbcc19ebc0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fcbb800e9c0 tx=0x7fcbb800ee80 comp rx=0 tx=0).stop 2024-03-28T22:46:38.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.397+0000 7fcbd3c00640 1 -- 172.21.15.43:0/4213808809 shutdown_connections 2024-03-28T22:46:38.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.397+0000 7fcbd3c00640 1 --2- 172.21.15.43:0/4213808809 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcbb40a1f70 0x7fcbb40a4430 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.397+0000 7fcbd3c00640 1 --2- 172.21.15.43:0/4213808809 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcbcc0765a0 0x7fcbcc19ebc0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.397+0000 7fcbd3c00640 1 --2- 172.21.15.43:0/4213808809 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcbcc112590 0x7fcbcc1a2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.397+0000 7fcbd3c00640 1 --2- 172.21.15.43:0/4213808809 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcbcc075c80 0x7fcbcc19e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.397+0000 7fcbd3c00640 1 -- 172.21.15.43:0/4213808809 >> 172.21.15.43:0/4213808809 conn(0x7fcbcc0ff3a0 msgr2=0x7fcbcc101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:38.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.398+0000 7fcbd3c00640 1 -- 172.21.15.43:0/4213808809 shutdown_connections 2024-03-28T22:46:38.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.398+0000 7fcbd3c00640 1 -- 172.21.15.43:0/4213808809 wait complete. 2024-03-28T22:46:38.413 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T22:46:38.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.634+0000 7fab01a00640 1 -- 172.21.15.43:0/3793417260 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faafc1015d0 msgr2=0x7faafc101a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:38.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.634+0000 7fab01a00640 1 --2- 172.21.15.43:0/3793417260 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faafc1015d0 0x7faafc101a50 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7faae8009930 tx=0x7faae802bc30 comp rx=0 tx=0).stop 2024-03-28T22:46:38.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.634+0000 7fab01a00640 1 -- 172.21.15.43:0/3793417260 shutdown_connections 2024-03-28T22:46:38.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.634+0000 7fab01a00640 1 --2- 172.21.15.43:0/3793417260 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faafc1015d0 0x7faafc101a50 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.634+0000 7fab01a00640 1 --2- 172.21.15.43:0/3793417260 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faafc101f90 0x7faafc10e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.634+0000 7fab01a00640 1 --2- 172.21.15.43:0/3793417260 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faafc107690 0x7faafc107a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.634+0000 7fab01a00640 1 -- 172.21.15.43:0/3793417260 >> 172.21.15.43:0/3793417260 conn(0x7faafc0fd0c0 msgr2=0x7faafc0ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:38.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.634+0000 7fab01a00640 1 -- 172.21.15.43:0/3793417260 shutdown_connections 2024-03-28T22:46:38.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.634+0000 7fab01a00640 1 -- 172.21.15.43:0/3793417260 wait complete. 2024-03-28T22:46:38.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.635+0000 7fab01a00640 1 Processor -- start 2024-03-28T22:46:38.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.635+0000 7fab01a00640 1 -- start start 2024-03-28T22:46:38.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.636+0000 7fab01a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faafc1015d0 0x7faafc197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.636+0000 7fab01a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faafc101f90 0x7faafc198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.636+0000 7fab01a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faafc107690 0x7faafc19c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.636+0000 7fab01a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7faafc111a20 con 0x7faafc107690 2024-03-28T22:46:38.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.636+0000 7fab01a00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7faafc1118a0 con 0x7faafc1015d0 2024-03-28T22:46:38.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.636+0000 7fab01a00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7faafc111ba0 con 0x7faafc101f90 2024-03-28T22:46:38.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.636+0000 7faafbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faafc1015d0 0x7faafc197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.636+0000 7faafb400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faafc101f90 0x7faafc198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.636+0000 7faafbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faafc1015d0 0x7faafc197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:42572/0 (socket says 172.21.15.43:42572) 2024-03-28T22:46:38.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.636+0000 7faafbe00640 1 -- 172.21.15.43:0/2328535647 learned_addr learned my addr 172.21.15.43:0/2328535647 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:38.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.636+0000 7fab01000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faafc107690 0x7faafc19c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.636+0000 7faafbe00640 1 -- 172.21.15.43:0/2328535647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faafc101f90 msgr2=0x7faafc198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:38.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.636+0000 7faafbe00640 1 --2- 172.21.15.43:0/2328535647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faafc101f90 0x7faafc198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.636+0000 7faafbe00640 1 -- 172.21.15.43:0/2328535647 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faafc107690 msgr2=0x7faafc19c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:38.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.636+0000 7faafbe00640 1 --2- 172.21.15.43:0/2328535647 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faafc107690 0x7faafc19c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.636+0000 7faafbe00640 1 -- 172.21.15.43:0/2328535647 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faafc19ceb0 con 0x7faafc1015d0 2024-03-28T22:46:38.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.637+0000 7faafbe00640 1 --2- 172.21.15.43:0/2328535647 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faafc1015d0 0x7faafc197ec0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7faaec00e9f0 tx=0x7faaec00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:38.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.637+0000 7faaf8c00640 1 -- 172.21.15.43:0/2328535647 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaec00cd80 con 0x7faafc1015d0 2024-03-28T22:46:38.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.637+0000 7fab01a00640 1 -- 172.21.15.43:0/2328535647 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faafc19d1a0 con 0x7faafc1015d0 2024-03-28T22:46:38.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.637+0000 7faaf8c00640 1 -- 172.21.15.43:0/2328535647 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faaec010910 con 0x7faafc1015d0 2024-03-28T22:46:38.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.637+0000 7faaf8c00640 1 -- 172.21.15.43:0/2328535647 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaec0188f0 con 0x7faafc1015d0 2024-03-28T22:46:38.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.637+0000 7fab01a00640 1 -- 172.21.15.43:0/2328535647 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faafc073980 con 0x7faafc1015d0 2024-03-28T22:46:38.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.640+0000 7fab05dd8740 1 Processor -- start 2024-03-28T22:46:38.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.640+0000 7faaf8c00640 1 -- 172.21.15.43:0/2328535647 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7faaec010ac0 con 0x7faafc1015d0 2024-03-28T22:46:38.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.640+0000 7fab05dd8740 1 -- start start 2024-03-28T22:46:38.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.641+0000 7fab05dd8740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5623436ab550 0x5623436ab930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.641+0000 7fab05dd8740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5623436abe70 0x5623436b4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.641+0000 7fab05dd8740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5623436b4a50 0x5623436b6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.641+0000 7fab05dd8740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56234360f050 con 0x5623436abe70 2024-03-28T22:46:38.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.641+0000 7fab05dd8740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5623435b7ce0 con 0x5623436ab550 2024-03-28T22:46:38.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.641+0000 7fab05dd8740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x56234360b0f0 con 0x5623436b4a50 2024-03-28T22:46:38.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.641+0000 7fab01000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5623436ab550 0x5623436ab930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.641+0000 7faafbe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5623436abe70 0x5623436b4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.641+0000 7fab01000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5623436ab550 0x5623436ab930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:42580/0 (socket says 172.21.15.43:42580) 2024-03-28T22:46:38.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.641+0000 7fab01000640 1 -- 172.21.15.43:0/3641665653 learned_addr learned my addr 172.21.15.43:0/3641665653 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:38.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.641+0000 7faafb400640 1 --2- 172.21.15.43:0/3641665653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5623436b4a50 0x5623436b6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.641+0000 7fab01000640 1 -- 172.21.15.43:0/3641665653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5623436b4a50 msgr2=0x5623436b6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:38.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.641+0000 7fab01000640 1 --2- 172.21.15.43:0/3641665653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5623436b4a50 0x5623436b6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.641+0000 7fab01000640 1 -- 172.21.15.43:0/3641665653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5623436abe70 msgr2=0x5623436b4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:38.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.641+0000 7fab01000640 1 --2- 172.21.15.43:0/3641665653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5623436abe70 0x5623436b4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.641+0000 7fab01000640 1 -- 172.21.15.43:0/3641665653 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623433233f0 con 0x5623436ab550 2024-03-28T22:46:38.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.642+0000 7fab01000640 1 --2- 172.21.15.43:0/3641665653 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5623436ab550 0x5623436ab930 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7faaf4009fd0 tx=0x7faaf4009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76107b860ffb0815 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:38.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.642+0000 7faaf1600640 1 -- 172.21.15.43:0/3641665653 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf4003c30 con 0x5623436ab550 2024-03-28T22:46:38.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.642+0000 7faaf8c00640 1 --2- 172.21.15.43:0/2328535647 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7faae00a1ef0 0x7faae00a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.642+0000 7faaf1600640 1 -- 172.21.15.43:0/3641665653 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faaf401f070 con 0x5623436ab550 2024-03-28T22:46:38.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.642+0000 7faaf1600640 1 -- 172.21.15.43:0/3641665653 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf400b9c0 con 0x5623436ab550 2024-03-28T22:46:38.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.642+0000 7faaf8c00640 1 -- 172.21.15.43:0/2328535647 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9539+0+0 (secure 0 0 0) 0x7faaec0ce940 con 0x7faafc1015d0 2024-03-28T22:46:38.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.642+0000 7faafbe00640 1 --2- 172.21.15.43:0/2328535647 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7faae00a1ef0 0x7faae00a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.643+0000 7fab05dd8740 1 -- 172.21.15.43:0/3641665653 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5623436ab550 msgr2=0x5623436ab930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:38.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.643+0000 7fab05dd8740 1 --2- 172.21.15.43:0/3641665653 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5623436ab550 0x5623436ab930 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7faaf4009fd0 tx=0x7faaf4009500 comp rx=0 tx=0).stop 2024-03-28T22:46:38.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.643+0000 7fab05dd8740 1 -- 172.21.15.43:0/3641665653 shutdown_connections 2024-03-28T22:46:38.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.643+0000 7fab05dd8740 1 --2- 172.21.15.43:0/3641665653 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5623436b4a50 0x5623436b6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.643+0000 7fab05dd8740 1 --2- 172.21.15.43:0/3641665653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5623436abe70 0x5623436b4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.643+0000 7fab05dd8740 1 --2- 172.21.15.43:0/3641665653 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5623436ab550 0x5623436ab930 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.643+0000 7fab05dd8740 1 -- 172.21.15.43:0/3641665653 >> 172.21.15.43:0/3641665653 conn(0x5623436a5c80 msgr2=0x5623436a6060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:38.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.643+0000 7fab05dd8740 1 -- 172.21.15.43:0/3641665653 shutdown_connections 2024-03-28T22:46:38.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.643+0000 7fab05dd8740 1 -- 172.21.15.43:0/3641665653 wait complete. 2024-03-28T22:46:38.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.643+0000 7faafbe00640 1 --2- 172.21.15.43:0/2328535647 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7faae00a1ef0 0x7faae00a43b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x5623436ab020 tx=0x7faaec0d4530 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:38.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.643+0000 7fab05dd8740 1 -- start start 2024-03-28T22:46:38.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.644+0000 7fab05dd8740 1 Processor -- start 2024-03-28T22:46:38.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.644+0000 7fab05dd8740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5623436ab550 0x5623436b1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.644+0000 7fab05dd8740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5623436abe70 0x5623436b1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.644+0000 7fab05dd8740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5623436b4a50 0x5623436b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.644+0000 7fab05dd8740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5623436b7840 con 0x5623436ab550 2024-03-28T22:46:38.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.644+0000 7fab05dd8740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5623435ad2f0 con 0x5623436b4a50 2024-03-28T22:46:38.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.644+0000 7fab05dd8740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5623436b79c0 con 0x5623436abe70 2024-03-28T22:46:38.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.644+0000 7fab05dd8740 10 client.0 _ll_register_callbacks cb 0x562343322be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:46:38.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.644+0000 7faafb400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5623436abe70 0x5623436b1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.644+0000 7faafb400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5623436abe70 0x5623436b1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40020/0 (socket says 172.21.15.43:40020) 2024-03-28T22:46:38.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.644+0000 7faafb400640 1 -- 172.21.15.43:0/1995638406 learned_addr learned my addr 172.21.15.43:0/1995638406 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:38.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.644+0000 7fab01000640 1 --2- 172.21.15.43:0/1995638406 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5623436ab550 0x5623436b1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.645+0000 7fab01000640 1 --2- 172.21.15.43:0/1995638406 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5623436b4a50 0x5623436b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.645+0000 7faafb400640 1 -- 172.21.15.43:0/1995638406 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5623436b4a50 msgr2=0x5623436b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:38.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.645+0000 7faafb400640 1 --2- 172.21.15.43:0/1995638406 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5623436b4a50 0x5623436b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.645+0000 7faafb400640 1 -- 172.21.15.43:0/1995638406 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5623436ab550 msgr2=0x5623436b1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:38.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.645+0000 7faafb400640 1 --2- 172.21.15.43:0/1995638406 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5623436ab550 0x5623436b1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.645+0000 7faafb400640 1 -- 172.21.15.43:0/1995638406 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623433ed7d0 con 0x5623436abe70 2024-03-28T22:46:38.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.645+0000 7fab01000640 1 --2- 172.21.15.43:0/1995638406 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5623436ab550 0x5623436b1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:46:38.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.645+0000 7faafb400640 1 --2- 172.21.15.43:0/1995638406 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5623436abe70 0x5623436b1fc0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7faae802c240 tx=0x7faae8005df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e8d95a76883b290 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:38.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.645+0000 7faae6000640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:46:38.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.645+0000 7faae6000640 1 -- 172.21.15.43:0/1995638406 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae802cb50 con 0x5623436abe70 2024-03-28T22:46:38.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.645+0000 7fab05dd8740 1 -- 172.21.15.43:0/1995638406 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faaf401f070 con 0x5623436abe70 2024-03-28T22:46:38.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.646+0000 7faae6000640 1 -- 172.21.15.43:0/1995638406 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9539+0+0 (secure 0 0 0) 0x7faae80046c0 con 0x5623436abe70 2024-03-28T22:46:38.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.646+0000 7faae6000640 1 -- 172.21.15.43:0/1995638406 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7faae802ccf0 con 0x5623436abe70 2024-03-28T22:46:38.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.646+0000 7fab05dd8740 10 client.28765 fetch_fsmap learned FSMap version 32 2024-03-28T22:46:38.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.646+0000 7fab05dd8740 1 -- 172.21.15.43:0/1995638406 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5623436c3b30 con 0x5623436abe70 2024-03-28T22:46:38.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.646+0000 7faae6000640 1 -- 172.21.15.43:0/1995638406 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faaf401f070 con 0x5623436abe70 2024-03-28T22:46:38.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.646+0000 7faae6000640 1 -- 172.21.15.43:0/1995638406 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7faae80035f0 con 0x5623436abe70 2024-03-28T22:46:38.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.646+0000 7fab05dd8740 10 client.28765 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:46:38.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.646+0000 7fab05dd8740 10 client.28765 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:46:38.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.646+0000 7fab05dd8740 20 client.28765 populate_metadata read hostname 'smithi043' 2024-03-28T22:46:38.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.646+0000 7fab05dd8740 1 --2- 172.21.15.43:0/1995638406 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5623436c4f40 0x5623436c7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.646+0000 7fab05dd8740 4 client.28765 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:46:38.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.646+0000 7fab05dd8740 1 -- 172.21.15.43:0/1995638406 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5623435b8680 con 0x5623436c4f40 2024-03-28T22:46:38.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.647+0000 7fab01000640 1 --2- 172.21.15.43:0/1995638406 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5623436c4f40 0x5623436c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.647+0000 7fab01000640 1 --2- 172.21.15.43:0/1995638406 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5623436c4f40 0x5623436c7330 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a2b46bffdb6daa70 server_cookie=1196db7947eaf151 in_seq=0 out_seq=0 2024-03-28T22:46:38.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.647+0000 7faae6000640 10 client.28765 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:46:38.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.648+0000 7faae6000640 1 -- 172.21.15.43:0/1995638406 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5623435b8680 con 0x5623436c4f40 2024-03-28T22:46:38.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.648+0000 7faae6000640 10 client.28765 handle_command_reply: tid=0 2024-03-28T22:46:38.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.648+0000 7fab05dd8740 1 client.28765 shutdown 2024-03-28T22:46:38.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.649+0000 7fab05dd8740 1 -- 172.21.15.43:0/1995638406 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5623436abe70 msgr2=0x5623436b1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:38.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.649+0000 7fab05dd8740 1 --2- 172.21.15.43:0/1995638406 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5623436abe70 0x5623436b1fc0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7faae802c240 tx=0x7faae8005df0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.649+0000 7fab05dd8740 1 -- 172.21.15.43:0/1995638406 shutdown_connections 2024-03-28T22:46:38.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.649+0000 7fab05dd8740 1 --2- 172.21.15.43:0/1995638406 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x5623436c4f40 0x5623436c7330 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.649+0000 7fab05dd8740 1 --2- 172.21.15.43:0/1995638406 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5623436b4a50 0x5623436b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.649+0000 7fab05dd8740 1 --2- 172.21.15.43:0/1995638406 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5623436abe70 0x5623436b1fc0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.649+0000 7faae6000640 0 client.28765 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:46:38.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.649+0000 7fab05dd8740 1 --2- 172.21.15.43:0/1995638406 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5623436ab550 0x5623436b1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.649+0000 7fab05dd8740 1 -- 172.21.15.43:0/1995638406 >> 172.21.15.43:0/1995638406 conn(0x5623436a5c80 msgr2=0x5623436a79d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:38.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.649+0000 7fab05dd8740 1 -- 172.21.15.43:0/1995638406 shutdown_connections 2024-03-28T22:46:38.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.649+0000 7fab05dd8740 1 -- 172.21.15.43:0/1995638406 wait complete. 2024-03-28T22:46:38.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.649+0000 7fab05dd8740 20 client.28765 trim_cache size 0 max 16384 2024-03-28T22:46:38.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.664+0000 7fab05dd8740 1 Processor -- start 2024-03-28T22:46:38.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.664+0000 7fab05dd8740 1 -- start start 2024-03-28T22:46:38.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.664+0000 7fab05dd8740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5623436ab550 0x5623436b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.664+0000 7fab05dd8740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5623436abe70 0x5623436c05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.664+0000 7fab05dd8740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5623436b4a50 0x5623436ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.664+0000 7fab05dd8740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7faafc1ac930 con 0x5623436ab550 2024-03-28T22:46:38.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.664+0000 7fab05dd8740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7faafc1ae470 con 0x5623436abe70 2024-03-28T22:46:38.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.664+0000 7fab05dd8740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7faafc1ad000 con 0x5623436b4a50 2024-03-28T22:46:38.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.664+0000 7faafb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5623436abe70 0x5623436c05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.664+0000 7faafb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5623436abe70 0x5623436c05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:42594/0 (socket says 172.21.15.43:42594) 2024-03-28T22:46:38.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.664+0000 7faafb400640 1 -- 172.21.15.43:0/3041551318 learned_addr learned my addr 172.21.15.43:0/3041551318 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:38.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.665+0000 7fab01000640 1 --2- 172.21.15.43:0/3041551318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5623436ab550 0x5623436b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.665+0000 7faafb400640 1 -- 172.21.15.43:0/3041551318 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5623436b4a50 msgr2=0x5623436ccc00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T22:46:38.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.665+0000 7fab01000640 1 --2- 172.21.15.43:0/3041551318 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5623436b4a50 0x5623436ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.665+0000 7faafb400640 1 --2- 172.21.15.43:0/3041551318 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5623436b4a50 0x5623436ccc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.665+0000 7faafb400640 1 -- 172.21.15.43:0/3041551318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5623436ab550 msgr2=0x5623436b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:38.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.665+0000 7faafb400640 1 --2- 172.21.15.43:0/3041551318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5623436ab550 0x5623436b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.665+0000 7faafb400640 1 -- 172.21.15.43:0/3041551318 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623436affd0 con 0x5623436abe70 2024-03-28T22:46:38.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.665+0000 7fab01000640 1 --2- 172.21.15.43:0/3041551318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5623436ab550 0x5623436b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:46:38.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.665+0000 7faafb400640 1 --2- 172.21.15.43:0/3041551318 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5623436abe70 0x5623436c05e0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7faae802c170 tx=0x7faae802bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a240952172fdb228 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:38.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.666+0000 7faaf0c00640 1 -- 172.21.15.43:0/3041551318 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae802cb50 con 0x5623436abe70 2024-03-28T22:46:38.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.666+0000 7faaf0c00640 1 -- 172.21.15.43:0/3041551318 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faae8009520 con 0x5623436abe70 2024-03-28T22:46:38.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.666+0000 7faaf0c00640 1 -- 172.21.15.43:0/3041551318 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae8005210 con 0x5623436abe70 2024-03-28T22:46:38.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.666+0000 7fab05dd8740 1 -- 172.21.15.43:0/3041551318 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5623436abe70 msgr2=0x5623436c05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:38.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.666+0000 7fab05dd8740 1 --2- 172.21.15.43:0/3041551318 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5623436abe70 0x5623436c05e0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7faae802c170 tx=0x7faae802bc30 comp rx=0 tx=0).stop 2024-03-28T22:46:38.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.666+0000 7fab05dd8740 1 -- 172.21.15.43:0/3041551318 shutdown_connections 2024-03-28T22:46:38.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.666+0000 7fab05dd8740 1 --2- 172.21.15.43:0/3041551318 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5623436abe70 0x5623436c05e0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.666+0000 7fab05dd8740 1 --2- 172.21.15.43:0/3041551318 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5623436b4a50 0x5623436ccc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.666+0000 7fab05dd8740 1 --2- 172.21.15.43:0/3041551318 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5623436ab550 0x5623436b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.666+0000 7fab05dd8740 1 -- 172.21.15.43:0/3041551318 >> 172.21.15.43:0/3041551318 conn(0x5623436a5c80 msgr2=0x5623436b22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:38.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.666+0000 7fab05dd8740 1 -- 172.21.15.43:0/3041551318 shutdown_connections 2024-03-28T22:46:38.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.666+0000 7fab05dd8740 1 -- 172.21.15.43:0/3041551318 wait complete. 2024-03-28T22:46:38.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.667+0000 7fab05dd8740 1 -- start start 2024-03-28T22:46:38.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.667+0000 7fab05dd8740 1 Processor -- start 2024-03-28T22:46:38.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.667+0000 7fab05dd8740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5623436ab550 0x5623436b8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.667+0000 7fab05dd8740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5623436abe70 0x5623436b8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.667+0000 7fab05dd8740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5623436b4a50 0x5623436b8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.667+0000 7fab05dd8740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7faafc1adda0 con 0x5623436ab550 2024-03-28T22:46:38.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.667+0000 7fab05dd8740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7faafc1ad6d0 con 0x5623436abe70 2024-03-28T22:46:38.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.667+0000 7fab05dd8740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7faafc1aa930 con 0x5623436b4a50 2024-03-28T22:46:38.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.668+0000 7fab05dd8740 10 client.0 _ll_register_callbacks cb 0x7faae8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:46:38.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.668+0000 7fab01000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5623436ab550 0x5623436b8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.668+0000 7faafb400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5623436abe70 0x5623436b8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.668+0000 7fab01000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5623436ab550 0x5623436b8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58698/0 (socket says 172.21.15.43:58698) 2024-03-28T22:46:38.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.668+0000 7fab01000640 1 -- 172.21.15.43:0/640936728 learned_addr learned my addr 172.21.15.43:0/640936728 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:38.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.668+0000 7fab01000640 1 --2- 172.21.15.43:0/640936728 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5623436b4a50 0x5623436b8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.668+0000 7faafb400640 1 -- 172.21.15.43:0/640936728 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5623436b4a50 msgr2=0x5623436b8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:38.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.668+0000 7faafb400640 1 --2- 172.21.15.43:0/640936728 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5623436b4a50 0x5623436b8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.668+0000 7faafb400640 1 -- 172.21.15.43:0/640936728 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5623436ab550 msgr2=0x5623436b8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:38.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.668+0000 7faafb400640 1 --2- 172.21.15.43:0/640936728 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5623436ab550 0x5623436b8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.668+0000 7faafb400640 1 -- 172.21.15.43:0/640936728 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56234372a230 con 0x5623436abe70 2024-03-28T22:46:38.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.668+0000 7faafb400640 1 --2- 172.21.15.43:0/640936728 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5623436abe70 0x5623436b8970 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x5623436c1730 tx=0x7faae8000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fdad7e151bb8012 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:38.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.668+0000 7faae6000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:46:38.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.669+0000 7faae6000640 1 -- 172.21.15.43:0/640936728 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae802cb50 con 0x5623436abe70 2024-03-28T22:46:38.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.669+0000 7fab05dd8740 1 -- 172.21.15.43:0/640936728 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faae8009520 con 0x5623436abe70 2024-03-28T22:46:38.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.669+0000 7faae6000640 1 -- 172.21.15.43:0/640936728 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9539+0+0 (secure 0 0 0) 0x7faae80050e0 con 0x5623436abe70 2024-03-28T22:46:38.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.669+0000 7faae6000640 1 -- 172.21.15.43:0/640936728 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7faae8069050 con 0x5623436abe70 2024-03-28T22:46:38.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.669+0000 7fab05dd8740 10 client.28829 fetch_fsmap learned FSMap version 32 2024-03-28T22:46:38.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.669+0000 7fab05dd8740 1 -- 172.21.15.43:0/640936728 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56234372b440 con 0x5623436abe70 2024-03-28T22:46:38.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.669+0000 7faae6000640 1 -- 172.21.15.43:0/640936728 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faae8009520 con 0x5623436abe70 2024-03-28T22:46:38.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.669+0000 7faae6000640 1 -- 172.21.15.43:0/640936728 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7faae804b850 con 0x5623436abe70 2024-03-28T22:46:38.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.669+0000 7fab05dd8740 10 client.28829 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:46:38.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.669+0000 7fab05dd8740 10 client.28829 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:46:38.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.669+0000 7fab05dd8740 20 client.28829 populate_metadata read hostname 'smithi043' 2024-03-28T22:46:38.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.670+0000 7fab05dd8740 1 --2- 172.21.15.43:0/640936728 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x56234372bb60 0x56234372df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.670+0000 7fab05dd8740 4 client.28829 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:46:38.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.670+0000 7fab05dd8740 1 -- 172.21.15.43:0/640936728 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56234361c590 con 0x56234372bb60 2024-03-28T22:46:38.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.670+0000 7fab01000640 1 --2- 172.21.15.43:0/640936728 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x56234372bb60 0x56234372df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.670+0000 7fab01000640 1 --2- 172.21.15.43:0/640936728 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x56234372bb60 0x56234372df80 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ea48beaef2a039a server_cookie=75d6647045605b23 in_seq=0 out_seq=0 2024-03-28T22:46:38.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.670+0000 7faae6000640 10 client.28829 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:46:38.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.671+0000 7faae6000640 1 -- 172.21.15.43:0/640936728 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56234361c590 con 0x56234372bb60 2024-03-28T22:46:38.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.671+0000 7faae6000640 10 client.28829 handle_command_reply: tid=0 2024-03-28T22:46:38.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.671+0000 7fab05dd8740 1 client.28829 shutdown 2024-03-28T22:46:38.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.671+0000 7fab05dd8740 1 -- 172.21.15.43:0/640936728 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5623436abe70 msgr2=0x5623436b8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:38.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.671+0000 7fab05dd8740 1 --2- 172.21.15.43:0/640936728 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5623436abe70 0x5623436b8970 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x5623436c1730 tx=0x7faae8000bc0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.671+0000 7fab05dd8740 1 -- 172.21.15.43:0/640936728 shutdown_connections 2024-03-28T22:46:38.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.671+0000 7fab05dd8740 1 --2- 172.21.15.43:0/640936728 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x56234372bb60 0x56234372df80 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.671+0000 7fab05dd8740 1 --2- 172.21.15.43:0/640936728 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5623436abe70 0x5623436b8970 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.671+0000 7fab05dd8740 1 --2- 172.21.15.43:0/640936728 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5623436b4a50 0x5623436b8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.671+0000 7fab05dd8740 1 --2- 172.21.15.43:0/640936728 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5623436ab550 0x5623436b8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.671+0000 7faae6000640 0 client.28829 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:46:38.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.671+0000 7fab05dd8740 1 -- 172.21.15.43:0/640936728 >> 172.21.15.43:0/640936728 conn(0x5623436a5c80 msgr2=0x5623436b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:38.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.671+0000 7fab05dd8740 1 -- 172.21.15.43:0/640936728 shutdown_connections 2024-03-28T22:46:38.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.671+0000 7fab05dd8740 1 -- 172.21.15.43:0/640936728 wait complete. 2024-03-28T22:46:38.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.671+0000 7fab05dd8740 20 client.28829 trim_cache size 0 max 16384 2024-03-28T22:46:38.676 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:46:38.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.672+0000 7fab01a00640 1 -- 172.21.15.43:0/2328535647 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7faae00a1ef0 msgr2=0x7faae00a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:38.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.672+0000 7fab01a00640 1 --2- 172.21.15.43:0/2328535647 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7faae00a1ef0 0x7faae00a43b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x5623436ab020 tx=0x7faaec0d4530 comp rx=0 tx=0).stop 2024-03-28T22:46:38.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.672+0000 7fab01a00640 1 -- 172.21.15.43:0/2328535647 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faafc1015d0 msgr2=0x7faafc197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:38.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.672+0000 7fab01a00640 1 --2- 172.21.15.43:0/2328535647 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faafc1015d0 0x7faafc197ec0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7faaec00e9f0 tx=0x7faaec00eeb0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.672+0000 7fab01a00640 1 -- 172.21.15.43:0/2328535647 shutdown_connections 2024-03-28T22:46:38.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.672+0000 7fab01a00640 1 --2- 172.21.15.43:0/2328535647 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7faae00a1ef0 0x7faae00a43b0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.672+0000 7fab01a00640 1 --2- 172.21.15.43:0/2328535647 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faafc107690 0x7faafc19c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.672+0000 7fab01a00640 1 --2- 172.21.15.43:0/2328535647 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faafc101f90 0x7faafc198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.672+0000 7fab01a00640 1 --2- 172.21.15.43:0/2328535647 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faafc1015d0 0x7faafc197ec0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.672+0000 7fab01a00640 1 -- 172.21.15.43:0/2328535647 >> 172.21.15.43:0/2328535647 conn(0x7faafc0fd0c0 msgr2=0x7faafc0fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:38.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.672+0000 7fab01a00640 1 -- 172.21.15.43:0/2328535647 shutdown_connections 2024-03-28T22:46:38.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.672+0000 7fab01a00640 1 -- 172.21.15.43:0/2328535647 wait complete. 2024-03-28T22:46:38.689 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:46:38.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.922+0000 7f5a0de00640 1 -- 172.21.15.43:0/80555583 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a08107690 msgr2=0x7f5a08107a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:38.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.922+0000 7f5a0de00640 1 --2- 172.21.15.43:0/80555583 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a08107690 0x7f5a08107a70 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f59f8009930 tx=0x7f59f802bc20 comp rx=0 tx=0).stop 2024-03-28T22:46:38.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.922+0000 7f5a0de00640 1 -- 172.21.15.43:0/80555583 shutdown_connections 2024-03-28T22:46:38.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.922+0000 7f5a0de00640 1 --2- 172.21.15.43:0/80555583 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a08101f90 0x7f5a0810e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.922+0000 7f5a0de00640 1 --2- 172.21.15.43:0/80555583 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a081015d0 0x7f5a08101a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.922+0000 7f5a0de00640 1 --2- 172.21.15.43:0/80555583 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a08107690 0x7f5a08107a70 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.922+0000 7f5a0de00640 1 -- 172.21.15.43:0/80555583 >> 172.21.15.43:0/80555583 conn(0x7f5a080fd0c0 msgr2=0x7f5a080ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:38.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.922+0000 7f5a0de00640 1 -- 172.21.15.43:0/80555583 shutdown_connections 2024-03-28T22:46:38.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.922+0000 7f5a0de00640 1 -- 172.21.15.43:0/80555583 wait complete. 2024-03-28T22:46:38.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.923+0000 7f5a0de00640 1 Processor -- start 2024-03-28T22:46:38.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.923+0000 7f5a0de00640 1 -- start start 2024-03-28T22:46:38.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.924+0000 7f5a0de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a081015d0 0x7f5a08197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.924+0000 7f5a0de00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a08101f90 0x7f5a08198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.924+0000 7f5a0de00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a08107690 0x7f5a0819c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.924+0000 7f5a0de00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5a081119e0 con 0x7f5a081015d0 2024-03-28T22:46:38.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.924+0000 7f5a0de00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5a08111860 con 0x7f5a08101f90 2024-03-28T22:46:38.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.924+0000 7f5a0de00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f5a08111b60 con 0x7f5a08107690 2024-03-28T22:46:38.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.924+0000 7f5a0ca00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a081015d0 0x7f5a08197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.924+0000 7f5a07e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a08101f90 0x7f5a08198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.924+0000 7f5a0d400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a08107690 0x7f5a0819c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.924+0000 7f5a07e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a08101f90 0x7f5a08198360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:42626/0 (socket says 172.21.15.43:42626) 2024-03-28T22:46:38.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.924+0000 7f5a0ca00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a081015d0 0x7f5a08197e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58714/0 (socket says 172.21.15.43:58714) 2024-03-28T22:46:38.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.924+0000 7f5a0d400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a08107690 0x7f5a0819c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40062/0 (socket says 172.21.15.43:40062) 2024-03-28T22:46:38.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.924+0000 7f5a07e00640 1 -- 172.21.15.43:0/464786932 learned_addr learned my addr 172.21.15.43:0/464786932 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:38.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.924+0000 7f5a07e00640 1 -- 172.21.15.43:0/464786932 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a08107690 msgr2=0x7f5a0819c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:38.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.924+0000 7f5a07e00640 1 --2- 172.21.15.43:0/464786932 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a08107690 0x7f5a0819c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.924+0000 7f5a07e00640 1 -- 172.21.15.43:0/464786932 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a081015d0 msgr2=0x7f5a08197e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:38.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.924+0000 7f5a07e00640 1 --2- 172.21.15.43:0/464786932 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a081015d0 0x7f5a08197e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.924+0000 7f5a07e00640 1 -- 172.21.15.43:0/464786932 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a0819ce10 con 0x7f5a08101f90 2024-03-28T22:46:38.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.925+0000 7f5a07e00640 1 --2- 172.21.15.43:0/464786932 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a08101f90 0x7f5a08198360 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f59f000d950 tx=0x7f59f000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:38.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.925+0000 7f5a05600640 1 -- 172.21.15.43:0/464786932 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f0017070 con 0x7f5a08101f90 2024-03-28T22:46:38.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.925+0000 7f5a05600640 1 -- 172.21.15.43:0/464786932 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59f0009c30 con 0x7f5a08101f90 2024-03-28T22:46:38.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.925+0000 7f5a05600640 1 -- 172.21.15.43:0/464786932 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f000f6b0 con 0x7f5a08101f90 2024-03-28T22:46:38.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.925+0000 7f5a0de00640 1 -- 172.21.15.43:0/464786932 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a0819d100 con 0x7f5a08101f90 2024-03-28T22:46:38.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.925+0000 7f5a0de00640 1 -- 172.21.15.43:0/464786932 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a081a4990 con 0x7f5a08101f90 2024-03-28T22:46:38.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.929+0000 7f5a121a7740 1 Processor -- start 2024-03-28T22:46:38.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.929+0000 7f5a121a7740 1 -- start start 2024-03-28T22:46:38.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.929+0000 7f5a121a7740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561229287960 0x561229287d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.929+0000 7f5a121a7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561229288280 0x561229290920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.929+0000 7f5a05600640 1 -- 172.21.15.43:0/464786932 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f59f00109d0 con 0x7f5a08101f90 2024-03-28T22:46:38.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.929+0000 7f5a121a7740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561229290e60 0x561229293250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.929+0000 7f5a121a7740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5612291e7050 con 0x561229287960 2024-03-28T22:46:38.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.929+0000 7f5a121a7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x56122918fce0 con 0x561229288280 2024-03-28T22:46:38.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.929+0000 7f5a121a7740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5612291e30f0 con 0x561229290e60 2024-03-28T22:46:38.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.929+0000 7f5a0ca00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561229288280 0x561229290920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.929+0000 7f5a0d400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561229287960 0x561229287d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.929+0000 7f5a0ca00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561229288280 0x561229290920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:42640/0 (socket says 172.21.15.43:42640) 2024-03-28T22:46:38.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.929+0000 7f5a0ca00640 1 -- 172.21.15.43:0/716568573 learned_addr learned my addr 172.21.15.43:0/716568573 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:38.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.929+0000 7f5a07e00640 1 --2- 172.21.15.43:0/716568573 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561229290e60 0x561229293250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.929+0000 7f5a0ca00640 1 -- 172.21.15.43:0/716568573 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561229290e60 msgr2=0x561229293250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:38.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.929+0000 7f5a0ca00640 1 --2- 172.21.15.43:0/716568573 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561229290e60 0x561229293250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.929+0000 7f5a0ca00640 1 -- 172.21.15.43:0/716568573 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561229287960 msgr2=0x561229287d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:38.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.929+0000 7f5a0ca00640 1 --2- 172.21.15.43:0/716568573 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561229287960 0x561229287d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.930+0000 7f5a0ca00640 1 -- 172.21.15.43:0/716568573 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561228efb3f0 con 0x561229288280 2024-03-28T22:46:38.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.930+0000 7f5a0d400640 1 --2- 172.21.15.43:0/716568573 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561229287960 0x561229287d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:46:38.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.930+0000 7f5a0ca00640 1 --2- 172.21.15.43:0/716568573 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561229288280 0x561229290920 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f59f8005ab0 tx=0x7f59f8005e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28d72bb569913028 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:38.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.930+0000 7f59fdc00640 1 -- 172.21.15.43:0/716568573 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f8045070 con 0x561229288280 2024-03-28T22:46:38.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.930+0000 7f59fdc00640 1 -- 172.21.15.43:0/716568573 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59f802c870 con 0x561229288280 2024-03-28T22:46:38.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.930+0000 7f59fdc00640 1 -- 172.21.15.43:0/716568573 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f8004c90 con 0x561229288280 2024-03-28T22:46:38.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.931+0000 7f5a05600640 1 --2- 172.21.15.43:0/464786932 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f59ec0a1f70 0x7f59ec0a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.931+0000 7f5a05600640 1 -- 172.21.15.43:0/464786932 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9539+0+0 (secure 0 0 0) 0x7f59f00ce110 con 0x7f5a08101f90 2024-03-28T22:46:38.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.931+0000 7f5a0d400640 1 --2- 172.21.15.43:0/464786932 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f59ec0a1f70 0x7f59ec0a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.931+0000 7f5a121a7740 1 -- 172.21.15.43:0/716568573 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561229288280 msgr2=0x561229290920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.931+0000 7f5a121a7740 1 --2- 172.21.15.43:0/716568573 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561229288280 0x561229290920 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f59f8005ab0 tx=0x7f59f8005e70 comp rx=0 tx=0).stop 2024-03-28T22:46:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.931+0000 7f5a121a7740 1 -- 172.21.15.43:0/716568573 shutdown_connections 2024-03-28T22:46:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.931+0000 7f5a121a7740 1 --2- 172.21.15.43:0/716568573 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561229288280 0x561229290920 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.931+0000 7f5a121a7740 1 --2- 172.21.15.43:0/716568573 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561229290e60 0x561229293250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.931+0000 7f5a121a7740 1 --2- 172.21.15.43:0/716568573 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561229287960 0x561229287d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.931+0000 7f5a121a7740 1 -- 172.21.15.43:0/716568573 >> 172.21.15.43:0/716568573 conn(0x561229282090 msgr2=0x561229282470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.931+0000 7f5a121a7740 1 -- 172.21.15.43:0/716568573 shutdown_connections 2024-03-28T22:46:38.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.931+0000 7f5a121a7740 1 -- 172.21.15.43:0/716568573 wait complete. 2024-03-28T22:46:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.932+0000 7f5a0d400640 1 --2- 172.21.15.43:0/464786932 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f59ec0a1f70 0x7f59ec0a4430 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5a00005fd0 tx=0x7f5a00009600 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.932+0000 7f5a121a7740 1 -- start start 2024-03-28T22:46:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.932+0000 7f5a121a7740 1 Processor -- start 2024-03-28T22:46:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.932+0000 7f5a121a7740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561229287960 0x56122928de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.932+0000 7f5a121a7740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561229288280 0x56122928e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.932+0000 7f5a121a7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561229290e60 0x56122928e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.933+0000 7f5a121a7740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x561229293bf0 con 0x561229288280 2024-03-28T22:46:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.933+0000 7f5a121a7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5612291852f0 con 0x561229290e60 2024-03-28T22:46:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.933+0000 7f5a121a7740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x561229293d70 con 0x561229287960 2024-03-28T22:46:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.933+0000 7f5a121a7740 10 client.0 _ll_register_callbacks cb 0x561228efabe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:46:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.933+0000 7f5a0ca00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561229287960 0x56122928de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.933+0000 7f5a07e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561229288280 0x56122928e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.933+0000 7f5a0d400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561229290e60 0x56122928e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.933+0000 7f5a0ca00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561229287960 0x56122928de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40068/0 (socket says 172.21.15.43:40068) 2024-03-28T22:46:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.933+0000 7f5a0ca00640 1 -- 172.21.15.43:0/3647444234 learned_addr learned my addr 172.21.15.43:0/3647444234 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.933+0000 7f5a0ca00640 1 -- 172.21.15.43:0/3647444234 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561229290e60 msgr2=0x56122928e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.933+0000 7f5a0ca00640 1 --2- 172.21.15.43:0/3647444234 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561229290e60 0x56122928e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.933+0000 7f5a0ca00640 1 -- 172.21.15.43:0/3647444234 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561229288280 msgr2=0x56122928e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.933+0000 7f5a0ca00640 1 --2- 172.21.15.43:0/3647444234 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561229288280 0x56122928e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.933+0000 7f5a0ca00640 1 -- 172.21.15.43:0/3647444234 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561228fc57d0 con 0x561229287960 2024-03-28T22:46:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.933+0000 7f5a07e00640 1 --2- 172.21.15.43:0/3647444234 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561229288280 0x56122928e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.934+0000 7f5a0ca00640 1 --2- 172.21.15.43:0/3647444234 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561229287960 0x56122928de80 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f59f8009900 tx=0x7f59f802cc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1fd5f927b284b33 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.934+0000 7f59f6000640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:46:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.934+0000 7f59f6000640 1 -- 172.21.15.43:0/3647444234 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f804e070 con 0x561229287960 2024-03-28T22:46:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.934+0000 7f59f6000640 1 -- 172.21.15.43:0/3647444234 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9539+0+0 (secure 0 0 0) 0x7f59f80052f0 con 0x561229287960 2024-03-28T22:46:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.934+0000 7f5a121a7740 1 -- 172.21.15.43:0/3647444234 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59f802c870 con 0x561229287960 2024-03-28T22:46:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.934+0000 7f59f6000640 1 -- 172.21.15.43:0/3647444234 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f59f8059050 con 0x561229287960 2024-03-28T22:46:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.934+0000 7f5a121a7740 10 client.28780 fetch_fsmap learned FSMap version 32 2024-03-28T22:46:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.934+0000 7f5a121a7740 1 -- 172.21.15.43:0/3647444234 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561229293bf0 con 0x561229287960 2024-03-28T22:46:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.934+0000 7f59f6000640 1 -- 172.21.15.43:0/3647444234 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59f802c870 con 0x561229287960 2024-03-28T22:46:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.935+0000 7f59f6000640 1 -- 172.21.15.43:0/3647444234 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f59f803f440 con 0x561229287960 2024-03-28T22:46:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.935+0000 7f5a121a7740 10 client.28780 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:46:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.935+0000 7f5a121a7740 10 client.28780 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:46:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.935+0000 7f5a121a7740 20 client.28780 populate_metadata read hostname 'smithi043' 2024-03-28T22:46:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.935+0000 7f5a121a7740 1 --2- 172.21.15.43:0/3647444234 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5612292a1020 0x5612292a3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.935+0000 7f5a121a7740 4 client.28780 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:46:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.935+0000 7f5a121a7740 1 -- 172.21.15.43:0/3647444234 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561229190680 con 0x5612292a1020 2024-03-28T22:46:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.935+0000 7f5a07e00640 1 --2- 172.21.15.43:0/3647444234 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5612292a1020 0x5612292a3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.936+0000 7f5a07e00640 1 --2- 172.21.15.43:0/3647444234 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5612292a1020 0x5612292a3410 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7b3b7a160eac5302 server_cookie=582032a9f33d0422 in_seq=0 out_seq=0 2024-03-28T22:46:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.936+0000 7f59f6000640 10 client.28780 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:46:38.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.937+0000 7f59f6000640 1 -- 172.21.15.43:0/3647444234 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561229190680 con 0x5612292a1020 2024-03-28T22:46:38.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.937+0000 7f59f6000640 10 client.28780 handle_command_reply: tid=0 2024-03-28T22:46:38.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.937+0000 7f5a121a7740 1 client.28780 shutdown 2024-03-28T22:46:38.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.937+0000 7f5a121a7740 1 -- 172.21.15.43:0/3647444234 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561229287960 msgr2=0x56122928de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:38.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.937+0000 7f5a121a7740 1 --2- 172.21.15.43:0/3647444234 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561229287960 0x56122928de80 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f59f8009900 tx=0x7f59f802cc30 comp rx=0 tx=0).stop 2024-03-28T22:46:38.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.937+0000 7f5a121a7740 1 -- 172.21.15.43:0/3647444234 shutdown_connections 2024-03-28T22:46:38.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.937+0000 7f5a121a7740 1 --2- 172.21.15.43:0/3647444234 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x5612292a1020 0x5612292a3410 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.937+0000 7f5a121a7740 1 --2- 172.21.15.43:0/3647444234 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561229290e60 0x56122928e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.937+0000 7f5a121a7740 1 --2- 172.21.15.43:0/3647444234 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561229288280 0x56122928e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.937+0000 7f5a121a7740 1 --2- 172.21.15.43:0/3647444234 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561229287960 0x56122928de80 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.937+0000 7f5a121a7740 1 -- 172.21.15.43:0/3647444234 >> 172.21.15.43:0/3647444234 conn(0x561229282090 msgr2=0x561229283de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:38.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.937+0000 7f59f6000640 0 client.28780 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:46:38.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.937+0000 7f5a121a7740 1 -- 172.21.15.43:0/3647444234 shutdown_connections 2024-03-28T22:46:38.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.937+0000 7f5a121a7740 1 -- 172.21.15.43:0/3647444234 wait complete. 2024-03-28T22:46:38.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.937+0000 7f5a121a7740 20 client.28780 trim_cache size 0 max 16384 2024-03-28T22:46:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.953+0000 7f5a121a7740 1 Processor -- start 2024-03-28T22:46:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.953+0000 7f5a121a7740 1 -- start start 2024-03-28T22:46:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.953+0000 7f5a121a7740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561229287960 0x5612292905b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.953+0000 7f5a121a7740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561229288280 0x56122929c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.953+0000 7f5a121a7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561229290e60 0x5612292a8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.953+0000 7f5a121a7740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5a081ac8b0 con 0x561229287960 2024-03-28T22:46:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.953+0000 7f5a121a7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5a081ae3f0 con 0x561229290e60 2024-03-28T22:46:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.953+0000 7f5a121a7740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f5a081acf80 con 0x561229288280 2024-03-28T22:46:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.953+0000 7f5a0ca00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561229287960 0x5612292905b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.953+0000 7f5a07e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561229288280 0x56122929c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.953+0000 7f5a0d400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561229290e60 0x5612292a8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.953+0000 7f5a07e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561229288280 0x56122929c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40078/0 (socket says 172.21.15.43:40078) 2024-03-28T22:46:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.953+0000 7f5a07e00640 1 -- 172.21.15.43:0/3430744464 learned_addr learned my addr 172.21.15.43:0/3430744464 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.953+0000 7f5a0d400640 1 -- 172.21.15.43:0/3430744464 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561229288280 msgr2=0x56122929c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.953+0000 7f5a0d400640 1 --2- 172.21.15.43:0/3430744464 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561229288280 0x56122929c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.953+0000 7f5a0d400640 1 -- 172.21.15.43:0/3430744464 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561229287960 msgr2=0x5612292905b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.953+0000 7f5a0d400640 1 --2- 172.21.15.43:0/3430744464 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561229287960 0x5612292905b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.953+0000 7f5a0d400640 1 -- 172.21.15.43:0/3430744464 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561229279540 con 0x561229290e60 2024-03-28T22:46:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.953+0000 7f5a0d400640 1 --2- 172.21.15.43:0/3430744464 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561229290e60 0x5612292a8b80 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f5a0001e5c0 tx=0x7f5a00015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ba52ff9c54e9357 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:38.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.954+0000 7f59fd200640 1 -- 172.21.15.43:0/3430744464 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a0001ec70 con 0x561229290e60 2024-03-28T22:46:38.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.954+0000 7f59fd200640 1 -- 172.21.15.43:0/3430744464 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a0001ee10 con 0x561229290e60 2024-03-28T22:46:38.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.954+0000 7f59fd200640 1 -- 172.21.15.43:0/3430744464 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a00010430 con 0x561229290e60 2024-03-28T22:46:38.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.954+0000 7f5a121a7740 1 -- 172.21.15.43:0/3430744464 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561229290e60 msgr2=0x5612292a8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:38.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.954+0000 7f5a121a7740 1 --2- 172.21.15.43:0/3430744464 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561229290e60 0x5612292a8b80 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f5a0001e5c0 tx=0x7f5a00015000 comp rx=0 tx=0).stop 2024-03-28T22:46:38.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.954+0000 7f5a121a7740 1 -- 172.21.15.43:0/3430744464 shutdown_connections 2024-03-28T22:46:38.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.954+0000 7f5a121a7740 1 --2- 172.21.15.43:0/3430744464 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561229290e60 0x5612292a8b80 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.954+0000 7f5a121a7740 1 --2- 172.21.15.43:0/3430744464 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561229288280 0x56122929c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.954+0000 7f5a121a7740 1 --2- 172.21.15.43:0/3430744464 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561229287960 0x5612292905b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.954+0000 7f5a121a7740 1 -- 172.21.15.43:0/3430744464 >> 172.21.15.43:0/3430744464 conn(0x561229282090 msgr2=0x56122927cdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:38.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.954+0000 7f5a121a7740 1 -- 172.21.15.43:0/3430744464 shutdown_connections 2024-03-28T22:46:38.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.954+0000 7f5a121a7740 1 -- 172.21.15.43:0/3430744464 wait complete. 2024-03-28T22:46:38.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.955+0000 7f5a121a7740 1 -- start start 2024-03-28T22:46:38.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.955+0000 7f5a121a7740 1 Processor -- start 2024-03-28T22:46:38.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.955+0000 7f5a121a7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561229287960 0x56122928aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.955+0000 7f5a121a7740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561229288280 0x56122928aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.955+0000 7f5a121a7740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561229290e60 0x56122928b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.955+0000 7f5a121a7740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5a081add20 con 0x561229288280 2024-03-28T22:46:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.955+0000 7f5a121a7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5a081ad650 con 0x561229287960 2024-03-28T22:46:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.955+0000 7f5a121a7740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f5a081aa8b0 con 0x561229290e60 2024-03-28T22:46:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.956+0000 7f5a121a7740 10 client.0 _ll_register_callbacks cb 0x7f59f00d1a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:46:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.956+0000 7f5a0ca00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561229287960 0x56122928aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.956+0000 7f5a0ca00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561229287960 0x56122928aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:42650/0 (socket says 172.21.15.43:42650) 2024-03-28T22:46:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.956+0000 7f5a0ca00640 1 -- 172.21.15.43:0/3817985479 learned_addr learned my addr 172.21.15.43:0/3817985479 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.956+0000 7f5a0d400640 1 --2- 172.21.15.43:0/3817985479 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561229290e60 0x56122928b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.956+0000 7f5a07e00640 1 --2- 172.21.15.43:0/3817985479 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561229288280 0x56122928aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.956+0000 7f5a0ca00640 1 -- 172.21.15.43:0/3817985479 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561229290e60 msgr2=0x56122928b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.956+0000 7f5a0ca00640 1 --2- 172.21.15.43:0/3817985479 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561229290e60 0x56122928b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.956+0000 7f5a0ca00640 1 -- 172.21.15.43:0/3817985479 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561229288280 msgr2=0x56122928aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.956+0000 7f5a0ca00640 1 --2- 172.21.15.43:0/3817985479 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561229288280 0x56122928aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.956+0000 7f5a0ca00640 1 -- 172.21.15.43:0/3817985479 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561229306230 con 0x561229287960 2024-03-28T22:46:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.956+0000 7f5a0ca00640 1 --2- 172.21.15.43:0/3817985479 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561229287960 0x56122928aab0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f59f8009930 tx=0x7f59f8009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7890f80488d3c2b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.956+0000 7f59f6000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:46:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.956+0000 7f59f6000640 1 -- 172.21.15.43:0/3817985479 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f804e040 con 0x561229287960 2024-03-28T22:46:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.957+0000 7f59f6000640 1 -- 172.21.15.43:0/3817985479 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9539+0+0 (secure 0 0 0) 0x7f59f803fc00 con 0x561229287960 2024-03-28T22:46:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.957+0000 7f5a121a7740 1 -- 172.21.15.43:0/3817985479 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a0001ee10 con 0x561229287960 2024-03-28T22:46:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.957+0000 7f59f6000640 1 -- 172.21.15.43:0/3817985479 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f59f8045050 con 0x561229287960 2024-03-28T22:46:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.957+0000 7f5a121a7740 10 client.28847 fetch_fsmap learned FSMap version 32 2024-03-28T22:46:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.957+0000 7f5a121a7740 1 -- 172.21.15.43:0/3817985479 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561229307510 con 0x561229287960 2024-03-28T22:46:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.957+0000 7f59f6000640 1 -- 172.21.15.43:0/3817985479 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a0001ee10 con 0x561229287960 2024-03-28T22:46:38.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.957+0000 7f59f6000640 1 -- 172.21.15.43:0/3817985479 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f59f804a450 con 0x561229287960 2024-03-28T22:46:38.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.957+0000 7f5a121a7740 10 client.28847 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:46:38.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.957+0000 7f5a121a7740 10 client.28847 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:46:38.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.957+0000 7f5a121a7740 20 client.28847 populate_metadata read hostname 'smithi043' 2024-03-28T22:46:38.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.957+0000 7f5a121a7740 1 --2- 172.21.15.43:0/3817985479 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x561229307c00 0x56122930a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:38.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.957+0000 7f5a121a7740 4 client.28847 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:46:38.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.957+0000 7f5a121a7740 1 -- 172.21.15.43:0/3817985479 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5612291f4590 con 0x561229307c00 2024-03-28T22:46:38.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.958+0000 7f5a07e00640 1 --2- 172.21.15.43:0/3817985479 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x561229307c00 0x56122930a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:38.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.958+0000 7f5a07e00640 1 --2- 172.21.15.43:0/3817985479 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x561229307c00 0x56122930a020 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e2e0f8d83462314b server_cookie=d0e11ed013dabedf in_seq=0 out_seq=0 2024-03-28T22:46:38.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.958+0000 7f59f6000640 10 client.28847 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:46:38.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.958+0000 7f59f6000640 1 -- 172.21.15.43:0/3817985479 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5612291f4590 con 0x561229307c00 2024-03-28T22:46:38.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.958+0000 7f59f6000640 10 client.28847 handle_command_reply: tid=0 2024-03-28T22:46:38.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.958+0000 7f5a121a7740 1 client.28847 shutdown 2024-03-28T22:46:38.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.959+0000 7f5a121a7740 1 -- 172.21.15.43:0/3817985479 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561229287960 msgr2=0x56122928aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:38.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.959+0000 7f5a121a7740 1 --2- 172.21.15.43:0/3817985479 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561229287960 0x56122928aab0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f59f8009930 tx=0x7f59f8009680 comp rx=0 tx=0).stop 2024-03-28T22:46:38.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.959+0000 7f5a121a7740 1 -- 172.21.15.43:0/3817985479 shutdown_connections 2024-03-28T22:46:38.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.959+0000 7f5a121a7740 1 --2- 172.21.15.43:0/3817985479 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x561229290e60 0x56122928b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.959+0000 7f5a121a7740 1 --2- 172.21.15.43:0/3817985479 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x561229288280 0x56122928aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.959+0000 7f5a121a7740 1 --2- 172.21.15.43:0/3817985479 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x561229307c00 0x56122930a020 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.959+0000 7f5a121a7740 1 --2- 172.21.15.43:0/3817985479 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561229287960 0x56122928aab0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.959+0000 7f5a121a7740 1 -- 172.21.15.43:0/3817985479 >> 172.21.15.43:0/3817985479 conn(0x561229282090 msgr2=0x56122928de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:38.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.959+0000 7f59f6000640 0 client.28847 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:46:38.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.959+0000 7f5a121a7740 1 -- 172.21.15.43:0/3817985479 shutdown_connections 2024-03-28T22:46:38.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.959+0000 7f5a121a7740 1 -- 172.21.15.43:0/3817985479 wait complete. 2024-03-28T22:46:38.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.959+0000 7f5a121a7740 20 client.28847 trim_cache size 0 max 16384 2024-03-28T22:46:38.966 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:46:38.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.960+0000 7f5a0de00640 1 -- 172.21.15.43:0/464786932 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f59ec0a1f70 msgr2=0x7f59ec0a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:38.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.960+0000 7f5a0de00640 1 --2- 172.21.15.43:0/464786932 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f59ec0a1f70 0x7f59ec0a4430 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5a00005fd0 tx=0x7f5a00009600 comp rx=0 tx=0).stop 2024-03-28T22:46:38.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.960+0000 7f5a0de00640 1 -- 172.21.15.43:0/464786932 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a08101f90 msgr2=0x7f5a08198360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:38.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.960+0000 7f5a0de00640 1 --2- 172.21.15.43:0/464786932 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a08101f90 0x7f5a08198360 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f59f000d950 tx=0x7f59f000de10 comp rx=0 tx=0).stop 2024-03-28T22:46:38.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.960+0000 7f5a0de00640 1 -- 172.21.15.43:0/464786932 shutdown_connections 2024-03-28T22:46:38.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.960+0000 7f5a0de00640 1 --2- 172.21.15.43:0/464786932 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f59ec0a1f70 0x7f59ec0a4430 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.960+0000 7f5a0de00640 1 --2- 172.21.15.43:0/464786932 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a08101f90 0x7f5a08198360 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.960+0000 7f5a0de00640 1 --2- 172.21.15.43:0/464786932 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a08107690 0x7f5a0819c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.960+0000 7f5a0de00640 1 --2- 172.21.15.43:0/464786932 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a081015d0 0x7f5a08197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:38.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.960+0000 7f5a0de00640 1 -- 172.21.15.43:0/464786932 >> 172.21.15.43:0/464786932 conn(0x7f5a080fd0c0 msgr2=0x7f5a080fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:38.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.960+0000 7f5a0de00640 1 -- 172.21.15.43:0/464786932 shutdown_connections 2024-03-28T22:46:38.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:38.960+0000 7f5a0de00640 1 -- 172.21.15.43:0/464786932 wait complete. 2024-03-28T22:46:38.975 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:46:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.205+0000 7fc8a7e00640 1 -- 172.21.15.43:0/3776881920 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8a0103300 msgr2=0x7fc8a010b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.205+0000 7fc8a7e00640 1 --2- 172.21.15.43:0/3776881920 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8a0103300 0x7fc8a010b7e0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fc88c009900 tx=0x7fc88c02bcc0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.205+0000 7fc8a7e00640 1 -- 172.21.15.43:0/3776881920 shutdown_connections 2024-03-28T22:46:39.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.205+0000 7fc8a7e00640 1 --2- 172.21.15.43:0/3776881920 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8a010bdb0 0x7fc8a010e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.205+0000 7fc8a7e00640 1 --2- 172.21.15.43:0/3776881920 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8a0103300 0x7fc8a010b7e0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.205+0000 7fc8a7e00640 1 --2- 172.21.15.43:0/3776881920 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8a01029e0 0x7fc8a0102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.205+0000 7fc8a7e00640 1 -- 172.21.15.43:0/3776881920 >> 172.21.15.43:0/3776881920 conn(0x7fc8a00777b0 msgr2=0x7fc8a0077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:39.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.205+0000 7fc8a7e00640 1 -- 172.21.15.43:0/3776881920 shutdown_connections 2024-03-28T22:46:39.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.205+0000 7fc8a7e00640 1 -- 172.21.15.43:0/3776881920 wait complete. 2024-03-28T22:46:39.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.206+0000 7fc8a7e00640 1 Processor -- start 2024-03-28T22:46:39.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.206+0000 7fc8a7e00640 1 -- start start 2024-03-28T22:46:39.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.207+0000 7fc8a7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8a01029e0 0x7fc8a0100630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.207+0000 7fc8a7e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8a0103300 0x7fc8a0100b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.207+0000 7fc8a7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8a010bdb0 0x7fc8a01010b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.207+0000 7fc8a7e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc8a0111b90 con 0x7fc8a01029e0 2024-03-28T22:46:39.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.207+0000 7fc8a7e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fc8a0111a10 con 0x7fc8a0103300 2024-03-28T22:46:39.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.207+0000 7fc8a7e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fc8a0111d10 con 0x7fc8a010bdb0 2024-03-28T22:46:39.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.207+0000 7fc8a4a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8a0103300 0x7fc8a0100b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.207+0000 7fc8a5400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8a01029e0 0x7fc8a0100630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.207+0000 7fc8a5e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8a010bdb0 0x7fc8a01010b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.207+0000 7fc8a4a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8a0103300 0x7fc8a0100b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:42668/0 (socket says 172.21.15.43:42668) 2024-03-28T22:46:39.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.207+0000 7fc8a4a00640 1 -- 172.21.15.43:0/3838368780 learned_addr learned my addr 172.21.15.43:0/3838368780 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:39.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.208+0000 7fc8a4a00640 1 -- 172.21.15.43:0/3838368780 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8a010bdb0 msgr2=0x7fc8a01010b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:39.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.208+0000 7fc8a4a00640 1 --2- 172.21.15.43:0/3838368780 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8a010bdb0 0x7fc8a01010b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.208+0000 7fc8a4a00640 1 -- 172.21.15.43:0/3838368780 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8a01029e0 msgr2=0x7fc8a0100630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:39.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.208+0000 7fc8a4a00640 1 --2- 172.21.15.43:0/3838368780 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8a01029e0 0x7fc8a0100630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.208+0000 7fc8a4a00640 1 -- 172.21.15.43:0/3838368780 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8a01a64d0 con 0x7fc8a0103300 2024-03-28T22:46:39.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.208+0000 7fc8a4a00640 1 --2- 172.21.15.43:0/3838368780 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8a0103300 0x7fc8a0100b70 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fc88c000c00 tx=0x7fc88c004430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:39.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.208+0000 7fc89e000640 1 -- 172.21.15.43:0/3838368780 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc88c042020 con 0x7fc8a0103300 2024-03-28T22:46:39.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.208+0000 7fc89e000640 1 -- 172.21.15.43:0/3838368780 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc88c0054e0 con 0x7fc8a0103300 2024-03-28T22:46:39.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.208+0000 7fc8a7e00640 1 -- 172.21.15.43:0/3838368780 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8a01a6760 con 0x7fc8a0103300 2024-03-28T22:46:39.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.208+0000 7fc89e000640 1 -- 172.21.15.43:0/3838368780 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc88c003cc0 con 0x7fc8a0103300 2024-03-28T22:46:39.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.208+0000 7fc8a7e00640 1 -- 172.21.15.43:0/3838368780 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8a01a6d00 con 0x7fc8a0103300 2024-03-28T22:46:39.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.211+0000 7fc89e000640 1 -- 172.21.15.43:0/3838368780 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fc88c045020 con 0x7fc8a0103300 2024-03-28T22:46:39.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.212+0000 7fc8aaa18740 1 Processor -- start 2024-03-28T22:46:39.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.212+0000 7fc8aaa18740 1 -- start start 2024-03-28T22:46:39.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.212+0000 7fc8aaa18740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3640d0380 0x55c3640d0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.212+0000 7fc8aaa18740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c3640d0ca0 0x55c3640d9340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.212+0000 7fc8aaa18740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3640d9880 0x55c3640dbc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.212+0000 7fc8aaa18740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c364034050 con 0x55c3640d9880 2024-03-28T22:46:39.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.212+0000 7fc8aaa18740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55c363fdcce0 con 0x55c3640d0380 2024-03-28T22:46:39.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.212+0000 7fc8aaa18740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55c3640300f0 con 0x55c3640d0ca0 2024-03-28T22:46:39.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.212+0000 7fc8a5e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3640d0380 0x55c3640d0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.213+0000 7fc8a5e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3640d0380 0x55c3640d0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:42674/0 (socket says 172.21.15.43:42674) 2024-03-28T22:46:39.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.213+0000 7fc8a5400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c3640d0ca0 0x55c3640d9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.213+0000 7fc8a5e00640 1 -- 172.21.15.43:0/4156909524 learned_addr learned my addr 172.21.15.43:0/4156909524 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:39.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.213+0000 7fc8a4a00640 1 --2- 172.21.15.43:0/4156909524 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3640d9880 0x55c3640dbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.213+0000 7fc8a5e00640 1 -- 172.21.15.43:0/4156909524 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c3640d0ca0 msgr2=0x55c3640d9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:39.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.213+0000 7fc8a5e00640 1 --2- 172.21.15.43:0/4156909524 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c3640d0ca0 0x55c3640d9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.213+0000 7fc8a5e00640 1 -- 172.21.15.43:0/4156909524 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3640d9880 msgr2=0x55c3640dbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:39.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.213+0000 7fc8a5e00640 1 --2- 172.21.15.43:0/4156909524 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3640d9880 0x55c3640dbc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.213+0000 7fc8a5e00640 1 -- 172.21.15.43:0/4156909524 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c363d483f0 con 0x55c3640d0380 2024-03-28T22:46:39.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.213+0000 7fc8a5400640 1 --2- 172.21.15.43:0/4156909524 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c3640d0ca0 0x55c3640d9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:39.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.213+0000 7fc89e000640 1 --2- 172.21.15.43:0/3838368780 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc8880a1fa0 0x7fc8880a4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.213+0000 7fc89e000640 1 -- 172.21.15.43:0/3838368780 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9539+0+0 (secure 0 0 0) 0x7fc88c0f01a0 con 0x7fc8a0103300 2024-03-28T22:46:39.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.213+0000 7fc8a5400640 1 --2- 172.21.15.43:0/3838368780 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc8880a1fa0 0x7fc8880a4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.213+0000 7fc8a5e00640 1 --2- 172.21.15.43:0/4156909524 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3640d0380 0x55c3640d0760 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fc898007ec0 tx=0x7fc89800f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a19aa5b0d9921e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:39.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.214+0000 7fc892600640 1 -- 172.21.15.43:0/4156909524 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc898010840 con 0x55c3640d0380 2024-03-28T22:46:39.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.214+0000 7fc892600640 1 -- 172.21.15.43:0/4156909524 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8980183f0 con 0x55c3640d0380 2024-03-28T22:46:39.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.214+0000 7fc892600640 1 -- 172.21.15.43:0/4156909524 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc898017600 con 0x55c3640d0380 2024-03-28T22:46:39.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.214+0000 7fc8aaa18740 1 -- 172.21.15.43:0/4156909524 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3640d0380 msgr2=0x55c3640d0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:39.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.214+0000 7fc8aaa18740 1 --2- 172.21.15.43:0/4156909524 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3640d0380 0x55c3640d0760 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fc898007ec0 tx=0x7fc89800f790 comp rx=0 tx=0).stop 2024-03-28T22:46:39.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.214+0000 7fc8a5400640 1 --2- 172.21.15.43:0/3838368780 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc8880a1fa0 0x7fc8880a4460 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x55c3640cfe50 tx=0x7fc89400b390 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:39.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.215+0000 7fc8aaa18740 1 -- 172.21.15.43:0/4156909524 shutdown_connections 2024-03-28T22:46:39.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.215+0000 7fc8aaa18740 1 --2- 172.21.15.43:0/4156909524 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3640d9880 0x55c3640dbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.215+0000 7fc8aaa18740 1 --2- 172.21.15.43:0/4156909524 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c3640d0ca0 0x55c3640d9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.215+0000 7fc8aaa18740 1 --2- 172.21.15.43:0/4156909524 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3640d0380 0x55c3640d0760 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.215+0000 7fc8aaa18740 1 -- 172.21.15.43:0/4156909524 >> 172.21.15.43:0/4156909524 conn(0x55c3640caab0 msgr2=0x55c3640cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:39.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.215+0000 7fc8aaa18740 1 -- 172.21.15.43:0/4156909524 shutdown_connections 2024-03-28T22:46:39.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.215+0000 7fc8aaa18740 1 -- 172.21.15.43:0/4156909524 wait complete. 2024-03-28T22:46:39.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.215+0000 7fc8aaa18740 1 -- start start 2024-03-28T22:46:39.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.215+0000 7fc8aaa18740 1 Processor -- start 2024-03-28T22:46:39.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.216+0000 7fc8aaa18740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3640d0380 0x55c3640d68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.216+0000 7fc8aaa18740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c3640d0ca0 0x55c3640d6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.216+0000 7fc8aaa18740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3640d9880 0x55c3640d7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.216+0000 7fc8aaa18740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55c3640dc670 con 0x55c3640d9880 2024-03-28T22:46:39.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.216+0000 7fc8aaa18740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55c363fd22f0 con 0x55c3640d0380 2024-03-28T22:46:39.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.216+0000 7fc8aaa18740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55c3640dc7f0 con 0x55c3640d0ca0 2024-03-28T22:46:39.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.216+0000 7fc8aaa18740 10 client.0 _ll_register_callbacks cb 0x55c363d47be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:46:39.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.216+0000 7fc8a5400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3640d0380 0x55c3640d68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.216+0000 7fc8a5400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3640d0380 0x55c3640d68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:42684/0 (socket says 172.21.15.43:42684) 2024-03-28T22:46:39.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.216+0000 7fc8a4a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c3640d0ca0 0x55c3640d6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.216+0000 7fc8a5400640 1 -- 172.21.15.43:0/2470375737 learned_addr learned my addr 172.21.15.43:0/2470375737 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:39.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.216+0000 7fc8a5e00640 1 --2- 172.21.15.43:0/2470375737 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3640d9880 0x55c3640d7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.216+0000 7fc8a4a00640 1 -- 172.21.15.43:0/2470375737 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3640d0380 msgr2=0x55c3640d68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:39.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.216+0000 7fc8a4a00640 1 --2- 172.21.15.43:0/2470375737 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3640d0380 0x55c3640d68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.216+0000 7fc8a4a00640 1 -- 172.21.15.43:0/2470375737 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3640d9880 msgr2=0x55c3640d7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:39.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.216+0000 7fc8a4a00640 1 --2- 172.21.15.43:0/2470375737 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3640d9880 0x55c3640d7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.216+0000 7fc8a4a00640 1 -- 172.21.15.43:0/2470375737 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c363e127d0 con 0x55c3640d0ca0 2024-03-28T22:46:39.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.216+0000 7fc8a5400640 1 --2- 172.21.15.43:0/2470375737 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3640d0380 0x55c3640d68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:39.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.217+0000 7fc8a4a00640 1 --2- 172.21.15.43:0/2470375737 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c3640d0ca0 0x55c3640d6df0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fc88c0ed5a0 tx=0x7fc88c0b8420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf696d48c768c3a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:39.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.217+0000 7fc886a00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:46:39.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.217+0000 7fc886a00640 1 -- 172.21.15.43:0/2470375737 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc88c0b8ae0 con 0x55c3640d0ca0 2024-03-28T22:46:39.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.217+0000 7fc886a00640 1 -- 172.21.15.43:0/2470375737 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9539+0+0 (secure 0 0 0) 0x7fc88c0ed710 con 0x55c3640d0ca0 2024-03-28T22:46:39.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.217+0000 7fc8aaa18740 1 -- 172.21.15.43:0/2470375737 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc8980183f0 con 0x55c3640d0ca0 2024-03-28T22:46:39.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.217+0000 7fc886a00640 1 -- 172.21.15.43:0/2470375737 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc88c003e60 con 0x55c3640d0ca0 2024-03-28T22:46:39.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.217+0000 7fc8aaa18740 10 client.28798 fetch_fsmap learned FSMap version 32 2024-03-28T22:46:39.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.217+0000 7fc8aaa18740 1 -- 172.21.15.43:0/2470375737 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c3640dc6d0 con 0x55c3640d0ca0 2024-03-28T22:46:39.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.217+0000 7fc886a00640 1 -- 172.21.15.43:0/2470375737 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8980183f0 con 0x55c3640d0ca0 2024-03-28T22:46:39.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.218+0000 7fc886a00640 1 -- 172.21.15.43:0/2470375737 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc88c0edbe0 con 0x55c3640d0ca0 2024-03-28T22:46:39.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.218+0000 7fc8aaa18740 10 client.28798 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:46:39.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.218+0000 7fc8aaa18740 10 client.28798 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:46:39.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.218+0000 7fc8aaa18740 20 client.28798 populate_metadata read hostname 'smithi043' 2024-03-28T22:46:39.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.218+0000 7fc8aaa18740 1 --2- 172.21.15.43:0/2470375737 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55c3640e9cd0 0x55c3640ec130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.218+0000 7fc8aaa18740 4 client.28798 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:46:39.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.218+0000 7fc8aaa18740 1 -- 172.21.15.43:0/2470375737 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c363fdd680 con 0x55c3640e9cd0 2024-03-28T22:46:39.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.218+0000 7fc8a5400640 1 --2- 172.21.15.43:0/2470375737 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55c3640e9cd0 0x55c3640ec130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.219+0000 7fc8a5400640 1 --2- 172.21.15.43:0/2470375737 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55c3640e9cd0 0x55c3640ec130 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7ed519ac35be7bb5 server_cookie=f9f7ec8a1fb9122 in_seq=0 out_seq=0 2024-03-28T22:46:39.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.219+0000 7fc886a00640 10 client.28798 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:46:39.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.219+0000 7fc886a00640 1 -- 172.21.15.43:0/2470375737 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c363fdd680 con 0x55c3640e9cd0 2024-03-28T22:46:39.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.221+0000 7fc886a00640 10 client.28798 handle_command_reply: tid=0 2024-03-28T22:46:39.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.221+0000 7fc8aaa18740 1 client.28798 shutdown 2024-03-28T22:46:39.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.221+0000 7fc8aaa18740 1 -- 172.21.15.43:0/2470375737 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c3640d0ca0 msgr2=0x55c3640d6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:39.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.221+0000 7fc8aaa18740 1 --2- 172.21.15.43:0/2470375737 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c3640d0ca0 0x55c3640d6df0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fc88c0ed5a0 tx=0x7fc88c0b8420 comp rx=0 tx=0).stop 2024-03-28T22:46:39.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.221+0000 7fc8aaa18740 1 -- 172.21.15.43:0/2470375737 shutdown_connections 2024-03-28T22:46:39.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.221+0000 7fc8aaa18740 1 --2- 172.21.15.43:0/2470375737 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55c3640e9cd0 0x55c3640ec130 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.221+0000 7fc8aaa18740 1 --2- 172.21.15.43:0/2470375737 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3640d9880 0x55c3640d7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.221+0000 7fc8aaa18740 1 --2- 172.21.15.43:0/2470375737 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c3640d0ca0 0x55c3640d6df0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.221+0000 7fc8aaa18740 1 --2- 172.21.15.43:0/2470375737 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3640d0380 0x55c3640d68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.221+0000 7fc8aaa18740 1 -- 172.21.15.43:0/2470375737 >> 172.21.15.43:0/2470375737 conn(0x55c3640caab0 msgr2=0x55c3640cc800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:39.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.221+0000 7fc886a00640 0 client.28798 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:46:39.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.221+0000 7fc8aaa18740 1 -- 172.21.15.43:0/2470375737 shutdown_connections 2024-03-28T22:46:39.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.221+0000 7fc8aaa18740 1 -- 172.21.15.43:0/2470375737 wait complete. 2024-03-28T22:46:39.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.221+0000 7fc8aaa18740 20 client.28798 trim_cache size 0 max 16384 2024-03-28T22:46:39.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.235+0000 7fc8aaa18740 1 Processor -- start 2024-03-28T22:46:39.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.235+0000 7fc8aaa18740 1 -- start start 2024-03-28T22:46:39.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.235+0000 7fc8aaa18740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3640d0380 0x55c3640d8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.235+0000 7fc8aaa18740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3640d0ca0 0x55c3640e5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.235+0000 7fc8aaa18740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c3640d9880 0x55c3640f1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.235+0000 7fc8aaa18740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc8a01b0e00 con 0x55c3640d0ca0 2024-03-28T22:46:39.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.235+0000 7fc8aaa18740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fc8a01b2940 con 0x55c3640d0380 2024-03-28T22:46:39.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.235+0000 7fc8aaa18740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fc8a01b14d0 con 0x55c3640d9880 2024-03-28T22:46:39.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.236+0000 7fc8a5400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3640d0380 0x55c3640d8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.236+0000 7fc8a5400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3640d0380 0x55c3640d8f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:42700/0 (socket says 172.21.15.43:42700) 2024-03-28T22:46:39.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.236+0000 7fc8a5400640 1 -- 172.21.15.43:0/2978600897 learned_addr learned my addr 172.21.15.43:0/2978600897 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:39.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.236+0000 7fc8a4a00640 1 --2- 172.21.15.43:0/2978600897 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3640d0ca0 0x55c3640e5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.236+0000 7fc8a5e00640 1 --2- 172.21.15.43:0/2978600897 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c3640d9880 0x55c3640f1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.236+0000 7fc8a5400640 1 -- 172.21.15.43:0/2978600897 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c3640d9880 msgr2=0x55c3640f1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:39.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.236+0000 7fc8a5400640 1 --2- 172.21.15.43:0/2978600897 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c3640d9880 0x55c3640f1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.236+0000 7fc8a5400640 1 -- 172.21.15.43:0/2978600897 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3640d0ca0 msgr2=0x55c3640e5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:39.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.236+0000 7fc8a5400640 1 --2- 172.21.15.43:0/2978600897 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3640d0ca0 0x55c3640e5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.236+0000 7fc8a5400640 1 -- 172.21.15.43:0/2978600897 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3640ec7b0 con 0x55c3640d0380 2024-03-28T22:46:39.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.236+0000 7fc8a5e00640 1 --2- 172.21.15.43:0/2978600897 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c3640d9880 0x55c3640f1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:46:39.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.236+0000 7fc8a5400640 1 --2- 172.21.15.43:0/2978600897 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3640d0380 0x55c3640d8f70 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x55c3640ec9b0 tx=0x7fc894019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e83a7f158ff47d0f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:39.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.237+0000 7fc891c00640 1 -- 172.21.15.43:0/2978600897 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc89402a420 con 0x55c3640d0380 2024-03-28T22:46:39.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.237+0000 7fc891c00640 1 -- 172.21.15.43:0/2978600897 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc89402f070 con 0x55c3640d0380 2024-03-28T22:46:39.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.237+0000 7fc891c00640 1 -- 172.21.15.43:0/2978600897 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc89401da70 con 0x55c3640d0380 2024-03-28T22:46:39.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.237+0000 7fc8aaa18740 1 -- 172.21.15.43:0/2978600897 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3640d0380 msgr2=0x55c3640d8f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:39.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.237+0000 7fc8aaa18740 1 --2- 172.21.15.43:0/2978600897 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3640d0380 0x55c3640d8f70 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x55c3640ec9b0 tx=0x7fc894019000 comp rx=0 tx=0).stop 2024-03-28T22:46:39.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.237+0000 7fc8aaa18740 1 -- 172.21.15.43:0/2978600897 shutdown_connections 2024-03-28T22:46:39.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.237+0000 7fc8aaa18740 1 --2- 172.21.15.43:0/2978600897 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c3640d9880 0x55c3640f1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.237+0000 7fc8aaa18740 1 --2- 172.21.15.43:0/2978600897 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3640d0ca0 0x55c3640e5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.237+0000 7fc8aaa18740 1 --2- 172.21.15.43:0/2978600897 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3640d0380 0x55c3640d8f70 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.237+0000 7fc8aaa18740 1 -- 172.21.15.43:0/2978600897 >> 172.21.15.43:0/2978600897 conn(0x55c3640caab0 msgr2=0x55c3640c92e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:39.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.237+0000 7fc8aaa18740 1 -- 172.21.15.43:0/2978600897 shutdown_connections 2024-03-28T22:46:39.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.237+0000 7fc8aaa18740 1 -- 172.21.15.43:0/2978600897 wait complete. 2024-03-28T22:46:39.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.238+0000 7fc8aaa18740 1 -- start start 2024-03-28T22:46:39.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.238+0000 7fc8aaa18740 1 Processor -- start 2024-03-28T22:46:39.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.238+0000 7fc8aaa18740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c3640d0380 0x55c3640dd510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.238+0000 7fc8aaa18740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3640d0ca0 0x55c3640dda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.238+0000 7fc8aaa18740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3640d9880 0x55c3640ddf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.238+0000 7fc8aaa18740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc8a01b2270 con 0x55c3640d0ca0 2024-03-28T22:46:39.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.238+0000 7fc8aaa18740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fc8a01b1ba0 con 0x55c3640d9880 2024-03-28T22:46:39.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.238+0000 7fc8aaa18740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fc8a01aee00 con 0x55c3640d0380 2024-03-28T22:46:39.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.238+0000 7fc8aaa18740 10 client.0 _ll_register_callbacks cb 0x7fc88c005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:46:39.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.239+0000 7fc8a5400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c3640d0380 0x55c3640dd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.239+0000 7fc8a4a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3640d0ca0 0x55c3640dda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.239+0000 7fc8a5e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3640d9880 0x55c3640ddf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.239+0000 7fc8a5400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c3640d0380 0x55c3640dd510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40146/0 (socket says 172.21.15.43:40146) 2024-03-28T22:46:39.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.239+0000 7fc8a5e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3640d9880 0x55c3640ddf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:42710/0 (socket says 172.21.15.43:42710) 2024-03-28T22:46:39.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.239+0000 7fc8a5400640 1 -- 172.21.15.43:0/4057939796 learned_addr learned my addr 172.21.15.43:0/4057939796 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:39.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.239+0000 7fc8a5400640 1 -- 172.21.15.43:0/4057939796 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3640d9880 msgr2=0x55c3640ddf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:39.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.239+0000 7fc8a5400640 1 --2- 172.21.15.43:0/4057939796 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3640d9880 0x55c3640ddf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.239+0000 7fc8a5400640 1 -- 172.21.15.43:0/4057939796 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3640d0ca0 msgr2=0x55c3640dda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:39.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.239+0000 7fc8a5400640 1 --2- 172.21.15.43:0/4057939796 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3640d0ca0 0x55c3640dda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.239+0000 7fc8a5400640 1 -- 172.21.15.43:0/4057939796 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c364157a50 con 0x55c3640d0380 2024-03-28T22:46:39.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.239+0000 7fc8a5400640 1 --2- 172.21.15.43:0/4057939796 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c3640d0380 0x55c3640dd510 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fc894003cb0 tx=0x7fc89401d410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b4fe7f8c8cdd6a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:39.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.239+0000 7fc886a00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:46:39.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.240+0000 7fc886a00640 1 -- 172.21.15.43:0/4057939796 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc89402a7e0 con 0x55c3640d0380 2024-03-28T22:46:39.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.240+0000 7fc8aaa18740 1 -- 172.21.15.43:0/4057939796 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc89402f070 con 0x55c3640d0380 2024-03-28T22:46:39.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.240+0000 7fc886a00640 1 -- 172.21.15.43:0/4057939796 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9539+0+0 (secure 0 0 0) 0x7fc89401e9b0 con 0x55c3640d0380 2024-03-28T22:46:39.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.240+0000 7fc886a00640 1 -- 172.21.15.43:0/4057939796 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc89401ebb0 con 0x55c3640d0380 2024-03-28T22:46:39.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.240+0000 7fc8aaa18740 10 client.28804 fetch_fsmap learned FSMap version 32 2024-03-28T22:46:39.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.240+0000 7fc8aaa18740 1 -- 172.21.15.43:0/4057939796 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c364158d10 con 0x55c3640d0380 2024-03-28T22:46:39.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.240+0000 7fc886a00640 1 -- 172.21.15.43:0/4057939796 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc89402f070 con 0x55c3640d0380 2024-03-28T22:46:39.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.241+0000 7fc886a00640 1 -- 172.21.15.43:0/4057939796 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fc89403d7f0 con 0x55c3640d0380 2024-03-28T22:46:39.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.241+0000 7fc8aaa18740 10 client.28804 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:46:39.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.241+0000 7fc8aaa18740 10 client.28804 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:46:39.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.241+0000 7fc8aaa18740 20 client.28804 populate_metadata read hostname 'smithi043' 2024-03-28T22:46:39.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.241+0000 7fc8aaa18740 1 --2- 172.21.15.43:0/4057939796 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55c3641593c0 0x55c36415b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.241+0000 7fc8aaa18740 4 client.28804 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:46:39.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.241+0000 7fc8aaa18740 1 -- 172.21.15.43:0/4057939796 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c364041590 con 0x55c3641593c0 2024-03-28T22:46:39.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.241+0000 7fc8a4a00640 1 --2- 172.21.15.43:0/4057939796 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55c3641593c0 0x55c36415b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.241+0000 7fc8a4a00640 1 --2- 172.21.15.43:0/4057939796 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55c3641593c0 0x55c36415b7e0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=772846298d8d46f7 server_cookie=b4de8e92ab4cf490 in_seq=0 out_seq=0 2024-03-28T22:46:39.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.241+0000 7fc886a00640 10 client.28804 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:46:39.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.242+0000 7fc886a00640 1 -- 172.21.15.43:0/4057939796 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c364041590 con 0x55c3641593c0 2024-03-28T22:46:39.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.242+0000 7fc886a00640 10 client.28804 handle_command_reply: tid=0 2024-03-28T22:46:39.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.242+0000 7fc8aaa18740 1 client.28804 shutdown 2024-03-28T22:46:39.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.242+0000 7fc8aaa18740 1 -- 172.21.15.43:0/4057939796 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c3640d0380 msgr2=0x55c3640dd510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:39.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.242+0000 7fc8aaa18740 1 --2- 172.21.15.43:0/4057939796 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c3640d0380 0x55c3640dd510 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fc894003cb0 tx=0x7fc89401d410 comp rx=0 tx=0).stop 2024-03-28T22:46:39.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.242+0000 7fc8aaa18740 1 -- 172.21.15.43:0/4057939796 shutdown_connections 2024-03-28T22:46:39.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.242+0000 7fc8aaa18740 1 --2- 172.21.15.43:0/4057939796 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55c3641593c0 0x55c36415b7e0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.242+0000 7fc8aaa18740 1 --2- 172.21.15.43:0/4057939796 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c3640d9880 0x55c3640ddf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.242+0000 7fc886a00640 0 client.28804 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:46:39.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.242+0000 7fc8aaa18740 1 --2- 172.21.15.43:0/4057939796 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55c3640d0ca0 0x55c3640dda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.242+0000 7fc8aaa18740 1 --2- 172.21.15.43:0/4057939796 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55c3640d0380 0x55c3640dd510 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.242+0000 7fc8aaa18740 1 -- 172.21.15.43:0/4057939796 >> 172.21.15.43:0/4057939796 conn(0x55c3640caab0 msgr2=0x55c3640d7100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:39.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.242+0000 7fc8aaa18740 1 -- 172.21.15.43:0/4057939796 shutdown_connections 2024-03-28T22:46:39.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.242+0000 7fc8aaa18740 1 -- 172.21.15.43:0/4057939796 wait complete. 2024-03-28T22:46:39.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.242+0000 7fc8aaa18740 20 client.28804 trim_cache size 0 max 16384 2024-03-28T22:46:39.248 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:46:39.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.243+0000 7fc8a7e00640 1 -- 172.21.15.43:0/3838368780 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc8880a1fa0 msgr2=0x7fc8880a4460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:39.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.243+0000 7fc8a7e00640 1 --2- 172.21.15.43:0/3838368780 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc8880a1fa0 0x7fc8880a4460 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x55c3640cfe50 tx=0x7fc89400b390 comp rx=0 tx=0).stop 2024-03-28T22:46:39.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.243+0000 7fc8a7e00640 1 -- 172.21.15.43:0/3838368780 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8a0103300 msgr2=0x7fc8a0100b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:39.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.243+0000 7fc8a7e00640 1 --2- 172.21.15.43:0/3838368780 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8a0103300 0x7fc8a0100b70 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fc88c000c00 tx=0x7fc88c004430 comp rx=0 tx=0).stop 2024-03-28T22:46:39.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.243+0000 7fc8a7e00640 1 -- 172.21.15.43:0/3838368780 shutdown_connections 2024-03-28T22:46:39.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.243+0000 7fc8a7e00640 1 --2- 172.21.15.43:0/3838368780 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc8880a1fa0 0x7fc8880a4460 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.243+0000 7fc8a7e00640 1 --2- 172.21.15.43:0/3838368780 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8a0103300 0x7fc8a0100b70 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.243+0000 7fc8a7e00640 1 --2- 172.21.15.43:0/3838368780 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8a010bdb0 0x7fc8a01010b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.243+0000 7fc8a7e00640 1 --2- 172.21.15.43:0/3838368780 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8a01029e0 0x7fc8a0100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.243+0000 7fc8a7e00640 1 -- 172.21.15.43:0/3838368780 >> 172.21.15.43:0/3838368780 conn(0x7fc8a00777b0 msgr2=0x7fc8a00fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:39.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.243+0000 7fc8a7e00640 1 -- 172.21.15.43:0/3838368780 shutdown_connections 2024-03-28T22:46:39.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.243+0000 7fc8a7e00640 1 -- 172.21.15.43:0/3838368780 wait complete. 2024-03-28T22:46:39.256 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:39 smithi043 ceph-mon[31823]: pgmap v295: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:46:39.256 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:39 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/230204113' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:46:39.258 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:46:39.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:39 smithi172 ceph-mon[36566]: pgmap v295: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:46:39.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:39 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/230204113' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:46:39.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:39 smithi082 ceph-mon[36471]: pgmap v295: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:46:39.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:39 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/230204113' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:46:39.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.489+0000 7f7e77000640 1 -- 172.21.15.43:0/1058832722 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7e70100d10 msgr2=0x7f7e7010d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:39.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.489+0000 7f7e77000640 1 --2- 172.21.15.43:0/1058832722 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7e70100d10 0x7f7e7010d9d0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c009900 tx=0x7f7e5c02bc40 comp rx=0 tx=0).stop 2024-03-28T22:46:39.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.489+0000 7f7e77000640 1 -- 172.21.15.43:0/1058832722 shutdown_connections 2024-03-28T22:46:39.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.489+0000 7f7e77000640 1 --2- 172.21.15.43:0/1058832722 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7e70100d10 0x7f7e7010d9d0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.489+0000 7f7e77000640 1 --2- 172.21.15.43:0/1058832722 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e7010df10 0x7f7e70110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.489+0000 7f7e77000640 1 --2- 172.21.15.43:0/1058832722 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e70100360 0x7f7e70100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.489+0000 7f7e77000640 1 -- 172.21.15.43:0/1058832722 >> 172.21.15.43:0/1058832722 conn(0x7f7e700fbfa0 msgr2=0x7f7e700fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:39.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.490+0000 7f7e77000640 1 -- 172.21.15.43:0/1058832722 shutdown_connections 2024-03-28T22:46:39.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.490+0000 7f7e77000640 1 -- 172.21.15.43:0/1058832722 wait complete. 2024-03-28T22:46:39.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.491+0000 7f7e77000640 1 Processor -- start 2024-03-28T22:46:39.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.491+0000 7f7e77000640 1 -- start start 2024-03-28T22:46:39.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.491+0000 7f7e77000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e70100360 0x7f7e7019a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.491+0000 7f7e77000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e70100d10 0x7f7e7019a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.491+0000 7f7e77000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7e7010df10 0x7f7e7019e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.491+0000 7f7e77000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7e70113c10 con 0x7f7e70100360 2024-03-28T22:46:39.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.491+0000 7f7e77000640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7e70113a90 con 0x7f7e7010df10 2024-03-28T22:46:39.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.491+0000 7f7e77000640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f7e70113d90 con 0x7f7e70100d10 2024-03-28T22:46:39.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.492+0000 7f7e76600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7e7010df10 0x7f7e7019e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.492+0000 7f7e75200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e70100d10 0x7f7e7019a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.492+0000 7f7e76600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7e7010df10 0x7f7e7019e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:42738/0 (socket says 172.21.15.43:42738) 2024-03-28T22:46:39.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.492+0000 7f7e75200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e70100d10 0x7f7e7019a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40172/0 (socket says 172.21.15.43:40172) 2024-03-28T22:46:39.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.492+0000 7f7e76600640 1 -- 172.21.15.43:0/3797216652 learned_addr learned my addr 172.21.15.43:0/3797216652 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:39.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.492+0000 7f7e75c00640 1 --2- 172.21.15.43:0/3797216652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e70100360 0x7f7e7019a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.492+0000 7f7e76600640 1 -- 172.21.15.43:0/3797216652 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e70100d10 msgr2=0x7f7e7019a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:39.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.492+0000 7f7e76600640 1 --2- 172.21.15.43:0/3797216652 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e70100d10 0x7f7e7019a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.492+0000 7f7e76600640 1 -- 172.21.15.43:0/3797216652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e70100360 msgr2=0x7f7e7019a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:39.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.492+0000 7f7e76600640 1 --2- 172.21.15.43:0/3797216652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e70100360 0x7f7e7019a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.492+0000 7f7e76600640 1 -- 172.21.15.43:0/3797216652 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e7019f0f0 con 0x7f7e7010df10 2024-03-28T22:46:39.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.492+0000 7f7e75200640 1 --2- 172.21.15.43:0/3797216652 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e70100d10 0x7f7e7019a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:39.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.493+0000 7f7e76600640 1 --2- 172.21.15.43:0/3797216652 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7e7010df10 0x7f7e7019e9c0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f7e6800e9d0 tx=0x7f7e6800ee90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:39.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.493+0000 7f7e6e000640 1 -- 172.21.15.43:0/3797216652 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6800cd80 con 0x7f7e7010df10 2024-03-28T22:46:39.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.493+0000 7f7e6e000640 1 -- 172.21.15.43:0/3797216652 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e68005020 con 0x7f7e7010df10 2024-03-28T22:46:39.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.493+0000 7f7e6e000640 1 -- 172.21.15.43:0/3797216652 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e680116d0 con 0x7f7e7010df10 2024-03-28T22:46:39.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.493+0000 7f7e77000640 1 -- 172.21.15.43:0/3797216652 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e7019f3e0 con 0x7f7e7010df10 2024-03-28T22:46:39.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.493+0000 7f7e77000640 1 -- 172.21.15.43:0/3797216652 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e701a6c70 con 0x7f7e7010df10 2024-03-28T22:46:39.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.496+0000 7f7e7b3dc740 1 Processor -- start 2024-03-28T22:46:39.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.496+0000 7f7e6e000640 1 -- 172.21.15.43:0/3797216652 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f7e680051d0 con 0x7f7e7010df10 2024-03-28T22:46:39.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.496+0000 7f7e7b3dc740 1 -- start start 2024-03-28T22:46:39.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.497+0000 7f7e7b3dc740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625db016550 0x5625db016930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.497+0000 7f7e7b3dc740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625db016e70 0x5625db01f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.497+0000 7f7e7b3dc740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625db01fa50 0x5625db021e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.497+0000 7f7e7b3dc740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5625daf7a050 con 0x5625db01fa50 2024-03-28T22:46:39.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.497+0000 7f7e7b3dc740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5625daf22ce0 con 0x5625db016550 2024-03-28T22:46:39.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.497+0000 7f7e7b3dc740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5625daf760f0 con 0x5625db016e70 2024-03-28T22:46:39.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.497+0000 7f7e75c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625db016e70 0x5625db01f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.497+0000 7f7e75c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625db016e70 0x5625db01f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40186/0 (socket says 172.21.15.43:40186) 2024-03-28T22:46:39.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.497+0000 7f7e75c00640 1 -- 172.21.15.43:0/1184059450 learned_addr learned my addr 172.21.15.43:0/1184059450 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:39.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.497+0000 7f7e75200640 1 --2- 172.21.15.43:0/1184059450 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625db016550 0x5625db016930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.497+0000 7f7e75200640 1 --2- 172.21.15.43:0/1184059450 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625db01fa50 0x5625db021e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.498+0000 7f7e75c00640 1 -- 172.21.15.43:0/1184059450 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625db016550 msgr2=0x5625db016930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:39.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.498+0000 7f7e75c00640 1 --2- 172.21.15.43:0/1184059450 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625db016550 0x5625db016930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.498+0000 7f7e75c00640 1 -- 172.21.15.43:0/1184059450 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625db01fa50 msgr2=0x5625db021e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:39.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.498+0000 7f7e75c00640 1 --2- 172.21.15.43:0/1184059450 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625db01fa50 0x5625db021e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.498+0000 7f7e75c00640 1 -- 172.21.15.43:0/1184059450 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625dac8e3f0 con 0x5625db016e70 2024-03-28T22:46:39.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.498+0000 7f7e75200640 1 --2- 172.21.15.43:0/1184059450 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625db01fa50 0x5625db021e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:46:39.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.498+0000 7f7e6e000640 1 --2- 172.21.15.43:0/3797216652 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7e540a1f70 0x7f7e540a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.498+0000 7f7e75c00640 1 --2- 172.21.15.43:0/1184059450 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625db016e70 0x5625db01f510 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f7e6000bcd0 tx=0x7f7e600107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c95fd616404713f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:39.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.498+0000 7f7e6e000640 1 -- 172.21.15.43:0/3797216652 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9539+0+0 (secure 0 0 0) 0x7f7e680ce9d0 con 0x7f7e7010df10 2024-03-28T22:46:39.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.498+0000 7f7e75200640 1 --2- 172.21.15.43:0/3797216652 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7e540a1f70 0x7f7e540a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.498+0000 7f7e66a00640 1 -- 172.21.15.43:0/1184059450 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e60005220 con 0x5625db016e70 2024-03-28T22:46:39.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.498+0000 7f7e66a00640 1 -- 172.21.15.43:0/1184059450 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e60004c80 con 0x5625db016e70 2024-03-28T22:46:39.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.498+0000 7f7e66a00640 1 -- 172.21.15.43:0/1184059450 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e600117e0 con 0x5625db016e70 2024-03-28T22:46:39.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.499+0000 7f7e7b3dc740 1 -- 172.21.15.43:0/1184059450 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625db016e70 msgr2=0x5625db01f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:39.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.499+0000 7f7e75200640 1 --2- 172.21.15.43:0/3797216652 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7e540a1f70 0x7f7e540a4430 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c02efd0 tx=0x7f7e5c000bc0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:39.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.499+0000 7f7e7b3dc740 1 --2- 172.21.15.43:0/1184059450 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625db016e70 0x5625db01f510 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f7e6000bcd0 tx=0x7f7e600107a0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.499+0000 7f7e7b3dc740 1 -- 172.21.15.43:0/1184059450 shutdown_connections 2024-03-28T22:46:39.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.499+0000 7f7e7b3dc740 1 --2- 172.21.15.43:0/1184059450 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625db01fa50 0x5625db021e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.499+0000 7f7e7b3dc740 1 --2- 172.21.15.43:0/1184059450 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625db016e70 0x5625db01f510 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.499+0000 7f7e7b3dc740 1 --2- 172.21.15.43:0/1184059450 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625db016550 0x5625db016930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.499+0000 7f7e7b3dc740 1 -- 172.21.15.43:0/1184059450 >> 172.21.15.43:0/1184059450 conn(0x5625db010c80 msgr2=0x5625db011060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:39.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.499+0000 7f7e7b3dc740 1 -- 172.21.15.43:0/1184059450 shutdown_connections 2024-03-28T22:46:39.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.499+0000 7f7e7b3dc740 1 -- 172.21.15.43:0/1184059450 wait complete. 2024-03-28T22:46:39.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.500+0000 7f7e7b3dc740 1 -- start start 2024-03-28T22:46:39.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.500+0000 7f7e7b3dc740 1 Processor -- start 2024-03-28T22:46:39.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.500+0000 7f7e7b3dc740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625db016550 0x5625db01cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.500+0000 7f7e7b3dc740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625db016e70 0x5625db01d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.500+0000 7f7e7b3dc740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625db01fa50 0x5625db01d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.500+0000 7f7e7b3dc740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5625db022840 con 0x5625db016e70 2024-03-28T22:46:39.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.501+0000 7f7e7b3dc740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5625daf182f0 con 0x5625db01fa50 2024-03-28T22:46:39.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.501+0000 7f7e7b3dc740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5625db0229c0 con 0x5625db016550 2024-03-28T22:46:39.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.501+0000 7f7e7b3dc740 10 client.0 _ll_register_callbacks cb 0x5625dac8dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:46:39.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.501+0000 7f7e75c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625db016550 0x5625db01cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.501+0000 7f7e75c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625db016550 0x5625db01cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40194/0 (socket says 172.21.15.43:40194) 2024-03-28T22:46:39.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.501+0000 7f7e75c00640 1 -- 172.21.15.43:0/3086572292 learned_addr learned my addr 172.21.15.43:0/3086572292 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:39.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.501+0000 7f7e75200640 1 --2- 172.21.15.43:0/3086572292 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625db016e70 0x5625db01d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.501+0000 7f7e76600640 1 --2- 172.21.15.43:0/3086572292 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625db01fa50 0x5625db01d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.501+0000 7f7e75c00640 1 -- 172.21.15.43:0/3086572292 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625db01fa50 msgr2=0x5625db01d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:39.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.501+0000 7f7e75c00640 1 --2- 172.21.15.43:0/3086572292 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625db01fa50 0x5625db01d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.501+0000 7f7e75c00640 1 -- 172.21.15.43:0/3086572292 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625db016e70 msgr2=0x5625db01d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:39.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.501+0000 7f7e75c00640 1 --2- 172.21.15.43:0/3086572292 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625db016e70 0x5625db01d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.501+0000 7f7e75c00640 1 -- 172.21.15.43:0/3086572292 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625dad587d0 con 0x5625db016550 2024-03-28T22:46:39.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.501+0000 7f7e76600640 1 --2- 172.21.15.43:0/3086572292 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625db01fa50 0x5625db01d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:39.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.502+0000 7f7e75c00640 1 --2- 172.21.15.43:0/3086572292 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625db016550 0x5625db01cac0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f7e60005d00 tx=0x7f7e600048f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa00e480b933e2de server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:39.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.502+0000 7f7e5b400640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:46:39.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.502+0000 7f7e5b400640 1 -- 172.21.15.43:0/3086572292 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e60004ae0 con 0x5625db016550 2024-03-28T22:46:39.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.502+0000 7f7e7b3dc740 1 -- 172.21.15.43:0/3086572292 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e60004c80 con 0x5625db016550 2024-03-28T22:46:39.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.502+0000 7f7e5b400640 1 -- 172.21.15.43:0/3086572292 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9539+0+0 (secure 0 0 0) 0x7f7e60005240 con 0x5625db016550 2024-03-28T22:46:39.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.502+0000 7f7e5b400640 1 -- 172.21.15.43:0/3086572292 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7e60004020 con 0x5625db016550 2024-03-28T22:46:39.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.502+0000 7f7e7b3dc740 10 client.28816 fetch_fsmap learned FSMap version 32 2024-03-28T22:46:39.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.502+0000 7f7e7b3dc740 1 -- 172.21.15.43:0/3086572292 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5625db02eaf0 con 0x5625db016550 2024-03-28T22:46:39.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.502+0000 7f7e5b400640 1 -- 172.21.15.43:0/3086572292 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e60004c80 con 0x5625db016550 2024-03-28T22:46:39.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.503+0000 7f7e5b400640 1 -- 172.21.15.43:0/3086572292 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7e60011520 con 0x5625db016550 2024-03-28T22:46:39.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.503+0000 7f7e7b3dc740 10 client.28816 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:46:39.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.503+0000 7f7e7b3dc740 10 client.28816 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:46:39.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.503+0000 7f7e7b3dc740 20 client.28816 populate_metadata read hostname 'smithi043' 2024-03-28T22:46:39.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.503+0000 7f7e7b3dc740 1 --2- 172.21.15.43:0/3086572292 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5625db02fea0 0x5625db032290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.503+0000 7f7e7b3dc740 4 client.28816 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:46:39.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.503+0000 7f7e7b3dc740 1 -- 172.21.15.43:0/3086572292 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625daf23680 con 0x5625db02fea0 2024-03-28T22:46:39.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.503+0000 7f7e75200640 1 --2- 172.21.15.43:0/3086572292 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5625db02fea0 0x5625db032290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.504+0000 7f7e75200640 1 --2- 172.21.15.43:0/3086572292 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5625db02fea0 0x5625db032290 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84926042b9942e40 server_cookie=dfd09d55f2d3f2ed in_seq=0 out_seq=0 2024-03-28T22:46:39.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.504+0000 7f7e5b400640 10 client.28816 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:46:39.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.504+0000 7f7e5b400640 1 -- 172.21.15.43:0/3086572292 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5625daf23680 con 0x5625db02fea0 2024-03-28T22:46:39.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.504+0000 7f7e5b400640 10 client.28816 handle_command_reply: tid=0 2024-03-28T22:46:39.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.504+0000 7f7e7b3dc740 1 client.28816 shutdown 2024-03-28T22:46:39.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.505+0000 7f7e7b3dc740 1 -- 172.21.15.43:0/3086572292 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625db016550 msgr2=0x5625db01cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:39.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.505+0000 7f7e7b3dc740 1 --2- 172.21.15.43:0/3086572292 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625db016550 0x5625db01cac0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f7e60005d00 tx=0x7f7e600048f0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.505+0000 7f7e7b3dc740 1 -- 172.21.15.43:0/3086572292 shutdown_connections 2024-03-28T22:46:39.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.505+0000 7f7e7b3dc740 1 --2- 172.21.15.43:0/3086572292 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5625db02fea0 0x5625db032290 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.505+0000 7f7e7b3dc740 1 --2- 172.21.15.43:0/3086572292 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625db01fa50 0x5625db01d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.505+0000 7f7e7b3dc740 1 --2- 172.21.15.43:0/3086572292 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625db016e70 0x5625db01d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.505+0000 7f7e7b3dc740 1 --2- 172.21.15.43:0/3086572292 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625db016550 0x5625db01cac0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.505+0000 7f7e7b3dc740 1 -- 172.21.15.43:0/3086572292 >> 172.21.15.43:0/3086572292 conn(0x5625db010c80 msgr2=0x5625db0129d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:39.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.505+0000 7f7e5b400640 0 client.28816 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:46:39.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.505+0000 7f7e7b3dc740 1 -- 172.21.15.43:0/3086572292 shutdown_connections 2024-03-28T22:46:39.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.505+0000 7f7e7b3dc740 1 -- 172.21.15.43:0/3086572292 wait complete. 2024-03-28T22:46:39.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.505+0000 7f7e7b3dc740 20 client.28816 trim_cache size 0 max 16384 2024-03-28T22:46:39.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.520+0000 7f7e7b3dc740 1 Processor -- start 2024-03-28T22:46:39.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.520+0000 7f7e7b3dc740 1 -- start start 2024-03-28T22:46:39.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.520+0000 7f7e7b3dc740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625db016550 0x5625db02b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.520+0000 7f7e7b3dc740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625db016e70 0x5625db00fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.520+0000 7f7e7b3dc740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625db01fa50 0x5625db037a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.520+0000 7f7e7b3dc740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7e701b0d60 con 0x5625db016550 2024-03-28T22:46:39.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.520+0000 7f7e7b3dc740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7e701b28a0 con 0x5625db016e70 2024-03-28T22:46:39.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.520+0000 7f7e7b3dc740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f7e701b1430 con 0x5625db01fa50 2024-03-28T22:46:39.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.521+0000 7f7e76600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625db01fa50 0x5625db037a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.521+0000 7f7e75200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625db016e70 0x5625db00fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.521+0000 7f7e76600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625db01fa50 0x5625db037a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40204/0 (socket says 172.21.15.43:40204) 2024-03-28T22:46:39.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.521+0000 7f7e76600640 1 -- 172.21.15.43:0/1377160323 learned_addr learned my addr 172.21.15.43:0/1377160323 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:39.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.521+0000 7f7e75200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625db016e70 0x5625db00fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:42760/0 (socket says 172.21.15.43:42760) 2024-03-28T22:46:39.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.521+0000 7f7e75c00640 1 --2- 172.21.15.43:0/1377160323 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625db016550 0x5625db02b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.521+0000 7f7e75200640 1 -- 172.21.15.43:0/1377160323 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625db01fa50 msgr2=0x5625db037a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:39.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.521+0000 7f7e75200640 1 --2- 172.21.15.43:0/1377160323 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625db01fa50 0x5625db037a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.521+0000 7f7e75200640 1 -- 172.21.15.43:0/1377160323 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625db016550 msgr2=0x5625db02b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:39.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.521+0000 7f7e75200640 1 --2- 172.21.15.43:0/1377160323 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625db016550 0x5625db02b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.521+0000 7f7e75200640 1 -- 172.21.15.43:0/1377160323 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625db011850 con 0x5625db016e70 2024-03-28T22:46:39.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.521+0000 7f7e76600640 1 --2- 172.21.15.43:0/1377160323 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625db01fa50 0x5625db037a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:39.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.521+0000 7f7e75200640 1 --2- 172.21.15.43:0/1377160323 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625db016e70 0x5625db00fc40 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c02c180 tx=0x7f7e5c041040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f01f132b4fd95ac server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:39.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.522+0000 7f7e66000640 1 -- 172.21.15.43:0/1377160323 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c004490 con 0x5625db016e70 2024-03-28T22:46:39.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.522+0000 7f7e66000640 1 -- 172.21.15.43:0/1377160323 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e5c004f60 con 0x5625db016e70 2024-03-28T22:46:39.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.522+0000 7f7e66000640 1 -- 172.21.15.43:0/1377160323 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c055450 con 0x5625db016e70 2024-03-28T22:46:39.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.522+0000 7f7e7b3dc740 1 -- 172.21.15.43:0/1377160323 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625db016e70 msgr2=0x5625db00fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:39.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.522+0000 7f7e7b3dc740 1 --2- 172.21.15.43:0/1377160323 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625db016e70 0x5625db00fc40 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c02c180 tx=0x7f7e5c041040 comp rx=0 tx=0).stop 2024-03-28T22:46:39.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.522+0000 7f7e7b3dc740 1 -- 172.21.15.43:0/1377160323 shutdown_connections 2024-03-28T22:46:39.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.522+0000 7f7e7b3dc740 1 --2- 172.21.15.43:0/1377160323 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625db016e70 0x5625db00fc40 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.522+0000 7f7e7b3dc740 1 --2- 172.21.15.43:0/1377160323 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625db01fa50 0x5625db037a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.523+0000 7f7e7b3dc740 1 --2- 172.21.15.43:0/1377160323 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625db016550 0x5625db02b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.523+0000 7f7e7b3dc740 1 -- 172.21.15.43:0/1377160323 >> 172.21.15.43:0/1377160323 conn(0x5625db010c80 msgr2=0x5625db00f530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:39.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.523+0000 7f7e7b3dc740 1 -- 172.21.15.43:0/1377160323 shutdown_connections 2024-03-28T22:46:39.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.523+0000 7f7e7b3dc740 1 -- 172.21.15.43:0/1377160323 wait complete. 2024-03-28T22:46:39.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.523+0000 7f7e7b3dc740 1 -- start start 2024-03-28T22:46:39.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.523+0000 7f7e7b3dc740 1 Processor -- start 2024-03-28T22:46:39.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.524+0000 7f7e7b3dc740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625db016550 0x5625db022dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.524+0000 7f7e7b3dc740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625db016e70 0x5625db023300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.524+0000 7f7e7b3dc740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625db01fa50 0x5625db09c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.524+0000 7f7e7b3dc740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7e701b21d0 con 0x5625db01fa50 2024-03-28T22:46:39.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.524+0000 7f7e7b3dc740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7e701b1b00 con 0x5625db016550 2024-03-28T22:46:39.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.524+0000 7f7e7b3dc740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f7e701aed60 con 0x5625db016e70 2024-03-28T22:46:39.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.524+0000 7f7e7b3dc740 10 client.0 _ll_register_callbacks cb 0x7f7e6000bb20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:46:39.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.524+0000 7f7e75200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625db016e70 0x5625db023300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.524+0000 7f7e76600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625db01fa50 0x5625db09c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.524+0000 7f7e75c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625db016550 0x5625db022dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.524+0000 7f7e75200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625db016e70 0x5625db023300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40212/0 (socket says 172.21.15.43:40212) 2024-03-28T22:46:39.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.524+0000 7f7e75200640 1 -- 172.21.15.43:0/2209739182 learned_addr learned my addr 172.21.15.43:0/2209739182 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:39.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.524+0000 7f7e75c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625db016550 0x5625db022dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:42776/0 (socket says 172.21.15.43:42776) 2024-03-28T22:46:39.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.525+0000 7f7e75200640 1 -- 172.21.15.43:0/2209739182 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625db016550 msgr2=0x5625db022dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:39.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.525+0000 7f7e75200640 1 --2- 172.21.15.43:0/2209739182 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625db016550 0x5625db022dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.525+0000 7f7e75200640 1 -- 172.21.15.43:0/2209739182 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625db01fa50 msgr2=0x5625db09c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:39.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.525+0000 7f7e75200640 1 --2- 172.21.15.43:0/2209739182 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625db01fa50 0x5625db09c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.525+0000 7f7e75200640 1 -- 172.21.15.43:0/2209739182 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625db09cd90 con 0x5625db016e70 2024-03-28T22:46:39.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.525+0000 7f7e75200640 1 --2- 172.21.15.43:0/2209739182 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625db016e70 0x5625db023300 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c036040 tx=0x7f7e5c02ce10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbb23204711e971d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:39.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.525+0000 7f7e5b400640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:46:39.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.525+0000 7f7e5b400640 1 -- 172.21.15.43:0/2209739182 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c061070 con 0x5625db016e70 2024-03-28T22:46:39.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.525+0000 7f7e7b3dc740 1 -- 172.21.15.43:0/2209739182 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e5c004f60 con 0x5625db016e70 2024-03-28T22:46:39.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.525+0000 7f7e5b400640 1 -- 172.21.15.43:0/2209739182 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9539+0+0 (secure 0 0 0) 0x7f7e5c005110 con 0x5625db016e70 2024-03-28T22:46:39.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.526+0000 7f7e5b400640 1 -- 172.21.15.43:0/2209739182 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7e5c06d050 con 0x5625db016e70 2024-03-28T22:46:39.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.526+0000 7f7e7b3dc740 10 client.28822 fetch_fsmap learned FSMap version 32 2024-03-28T22:46:39.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.526+0000 7f7e7b3dc740 1 -- 172.21.15.43:0/2209739182 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5625db09e040 con 0x5625db016e70 2024-03-28T22:46:39.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.526+0000 7f7e5b400640 1 -- 172.21.15.43:0/2209739182 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e5c004f60 con 0x5625db016e70 2024-03-28T22:46:39.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.526+0000 7f7e5b400640 1 -- 172.21.15.43:0/2209739182 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7e5c046040 con 0x5625db016e70 2024-03-28T22:46:39.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.526+0000 7f7e7b3dc740 10 client.28822 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:46:39.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.526+0000 7f7e7b3dc740 10 client.28822 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:46:39.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.526+0000 7f7e7b3dc740 20 client.28822 populate_metadata read hostname 'smithi043' 2024-03-28T22:46:39.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.526+0000 7f7e7b3dc740 1 --2- 172.21.15.43:0/2209739182 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5625db09e7a0 0x5625db0a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:39.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.526+0000 7f7e7b3dc740 4 client.28822 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:46:39.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.526+0000 7f7e7b3dc740 1 -- 172.21.15.43:0/2209739182 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5625daf87590 con 0x5625db09e7a0 2024-03-28T22:46:39.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.526+0000 7f7e75c00640 1 --2- 172.21.15.43:0/2209739182 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5625db09e7a0 0x5625db0a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:39.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.527+0000 7f7e75c00640 1 --2- 172.21.15.43:0/2209739182 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5625db09e7a0 0x5625db0a0bc0 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9caf943b239db7f7 server_cookie=66b926a1f04bc271 in_seq=0 out_seq=0 2024-03-28T22:46:39.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.527+0000 7f7e5b400640 10 client.28822 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:46:39.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.527+0000 7f7e5b400640 1 -- 172.21.15.43:0/2209739182 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5625daf87590 con 0x5625db09e7a0 2024-03-28T22:46:39.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.527+0000 7f7e5b400640 10 client.28822 handle_command_reply: tid=0 2024-03-28T22:46:39.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.527+0000 7f7e7b3dc740 1 client.28822 shutdown 2024-03-28T22:46:39.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.527+0000 7f7e7b3dc740 1 -- 172.21.15.43:0/2209739182 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625db016e70 msgr2=0x5625db023300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:39.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.527+0000 7f7e7b3dc740 1 --2- 172.21.15.43:0/2209739182 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625db016e70 0x5625db023300 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c036040 tx=0x7f7e5c02ce10 comp rx=0 tx=0).stop 2024-03-28T22:46:39.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.527+0000 7f7e7b3dc740 1 -- 172.21.15.43:0/2209739182 shutdown_connections 2024-03-28T22:46:39.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.527+0000 7f7e7b3dc740 1 --2- 172.21.15.43:0/2209739182 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5625db01fa50 0x5625db09c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.527+0000 7f7e7b3dc740 1 --2- 172.21.15.43:0/2209739182 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5625db016e70 0x5625db023300 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.527+0000 7f7e7b3dc740 1 --2- 172.21.15.43:0/2209739182 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5625db09e7a0 0x5625db0a0bc0 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.527+0000 7f7e7b3dc740 1 --2- 172.21.15.43:0/2209739182 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5625db016550 0x5625db022dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.527+0000 7f7e7b3dc740 1 -- 172.21.15.43:0/2209739182 >> 172.21.15.43:0/2209739182 conn(0x5625db010c80 msgr2=0x5625db011cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:39.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.527+0000 7f7e5b400640 0 client.28822 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:46:39.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.528+0000 7f7e7b3dc740 1 -- 172.21.15.43:0/2209739182 shutdown_connections 2024-03-28T22:46:39.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.528+0000 7f7e7b3dc740 1 -- 172.21.15.43:0/2209739182 wait complete. 2024-03-28T22:46:39.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.528+0000 7f7e7b3dc740 20 client.28822 trim_cache size 0 max 16384 2024-03-28T22:46:39.533 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:46:39.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.528+0000 7f7e77000640 1 -- 172.21.15.43:0/3797216652 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7e540a1f70 msgr2=0x7f7e540a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:39.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.528+0000 7f7e77000640 1 --2- 172.21.15.43:0/3797216652 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7e540a1f70 0x7f7e540a4430 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c02efd0 tx=0x7f7e5c000bc0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.528+0000 7f7e77000640 1 -- 172.21.15.43:0/3797216652 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7e7010df10 msgr2=0x7f7e7019e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:39.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.528+0000 7f7e77000640 1 --2- 172.21.15.43:0/3797216652 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7e7010df10 0x7f7e7019e9c0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f7e6800e9d0 tx=0x7f7e6800ee90 comp rx=0 tx=0).stop 2024-03-28T22:46:39.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.528+0000 7f7e77000640 1 -- 172.21.15.43:0/3797216652 shutdown_connections 2024-03-28T22:46:39.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.528+0000 7f7e77000640 1 --2- 172.21.15.43:0/3797216652 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f7e540a1f70 0x7f7e540a4430 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.528+0000 7f7e77000640 1 --2- 172.21.15.43:0/3797216652 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7e7010df10 0x7f7e7019e9c0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.528+0000 7f7e77000640 1 --2- 172.21.15.43:0/3797216652 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7e70100d10 0x7f7e7019a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.528+0000 7f7e77000640 1 --2- 172.21.15.43:0/3797216652 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7e70100360 0x7f7e7019a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:39.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.528+0000 7f7e77000640 1 -- 172.21.15.43:0/3797216652 >> 172.21.15.43:0/3797216652 conn(0x7f7e700fbfa0 msgr2=0x7f7e7010f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:39.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.529+0000 7f7e77000640 1 -- 172.21.15.43:0/3797216652 shutdown_connections 2024-03-28T22:46:39.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:39.529+0000 7f7e77000640 1 -- 172.21.15.43:0/3797216652 wait complete. 2024-03-28T22:46:40.543 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:46:40.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:46:40.544 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:46:40.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.789+0000 7fb158a00640 1 -- 172.21.15.43:0/2555440154 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb154106590 msgr2=0x7fb154106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:40.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.789+0000 7fb158a00640 1 --2- 172.21.15.43:0/2555440154 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb154106590 0x7fb154106970 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fb144009cd0 tx=0x7fb14402c0e0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.789+0000 7fb158a00640 1 -- 172.21.15.43:0/2555440154 shutdown_connections 2024-03-28T22:46:40.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.789+0000 7fb158a00640 1 --2- 172.21.15.43:0/2555440154 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb154100e90 0x7fb15410d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.789+0000 7fb158a00640 1 --2- 172.21.15.43:0/2555440154 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1541004d0 0x7fb154100950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.789+0000 7fb158a00640 1 --2- 172.21.15.43:0/2555440154 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb154106590 0x7fb154106970 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.789+0000 7fb158a00640 1 -- 172.21.15.43:0/2555440154 >> 172.21.15.43:0/2555440154 conn(0x7fb1540fbfa0 msgr2=0x7fb1540fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:40.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.789+0000 7fb158a00640 1 -- 172.21.15.43:0/2555440154 shutdown_connections 2024-03-28T22:46:40.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.789+0000 7fb158a00640 1 -- 172.21.15.43:0/2555440154 wait complete. 2024-03-28T22:46:40.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.790+0000 7fb158a00640 1 Processor -- start 2024-03-28T22:46:40.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.791+0000 7fb158a00640 1 -- start start 2024-03-28T22:46:40.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.791+0000 7fb158a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1541004d0 0x7fb154102140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:40.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.791+0000 7fb158a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb154100e90 0x7fb154102680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:40.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.791+0000 7fb158a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb154106590 0x7fb154102bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:40.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.791+0000 7fb158a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb154110850 con 0x7fb1541004d0 2024-03-28T22:46:40.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.791+0000 7fb158a00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb1541106d0 con 0x7fb154106590 2024-03-28T22:46:40.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.791+0000 7fb158a00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fb1541109d0 con 0x7fb154100e90 2024-03-28T22:46:40.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.791+0000 7fb153400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1541004d0 0x7fb154102140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:40.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.791+0000 7fb153e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb154106590 0x7fb154102bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:40.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.791+0000 7fb153400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1541004d0 0x7fb154102140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58856/0 (socket says 172.21.15.43:58856) 2024-03-28T22:46:40.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.791+0000 7fb153400640 1 -- 172.21.15.43:0/3010953946 learned_addr learned my addr 172.21.15.43:0/3010953946 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:40.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.792+0000 7fb152a00640 1 --2- 172.21.15.43:0/3010953946 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb154100e90 0x7fb154102680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:40.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.792+0000 7fb153e00640 1 -- 172.21.15.43:0/3010953946 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb154100e90 msgr2=0x7fb154102680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:40.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.792+0000 7fb153e00640 1 --2- 172.21.15.43:0/3010953946 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb154100e90 0x7fb154102680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.792+0000 7fb153e00640 1 -- 172.21.15.43:0/3010953946 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1541004d0 msgr2=0x7fb154102140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:40.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.792+0000 7fb153e00640 1 --2- 172.21.15.43:0/3010953946 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1541004d0 0x7fb154102140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.792+0000 7fb153e00640 1 -- 172.21.15.43:0/3010953946 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1541a6300 con 0x7fb154106590 2024-03-28T22:46:40.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.792+0000 7fb153400640 1 --2- 172.21.15.43:0/3010953946 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1541004d0 0x7fb154102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:40.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.792+0000 7fb153e00640 1 --2- 172.21.15.43:0/3010953946 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb154106590 0x7fb154102bc0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fb14c00e970 tx=0x7fb14c00ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:40.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.792+0000 7fb14be00640 1 -- 172.21.15.43:0/3010953946 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb14c00ccc0 con 0x7fb154106590 2024-03-28T22:46:40.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.792+0000 7fb158a00640 1 -- 172.21.15.43:0/3010953946 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1541a65f0 con 0x7fb154106590 2024-03-28T22:46:40.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.793+0000 7fb14be00640 1 -- 172.21.15.43:0/3010953946 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb14c0035f0 con 0x7fb154106590 2024-03-28T22:46:40.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.793+0000 7fb14be00640 1 -- 172.21.15.43:0/3010953946 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb14c00ce60 con 0x7fb154106590 2024-03-28T22:46:40.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.793+0000 7fb158a00640 1 -- 172.21.15.43:0/3010953946 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1541a6af0 con 0x7fb154106590 2024-03-28T22:46:40.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.796+0000 7fb14be00640 1 -- 172.21.15.43:0/3010953946 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fb14c0037a0 con 0x7fb154106590 2024-03-28T22:46:40.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.796+0000 7fb15ce22740 1 Processor -- start 2024-03-28T22:46:40.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.796+0000 7fb15ce22740 1 -- start start 2024-03-28T22:46:40.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.797+0000 7fb15ce22740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a31266c170 0x55a31266c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:40.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.797+0000 7fb15ce22740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a31266ca90 0x55a312675130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:40.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.797+0000 7fb15ce22740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a312675670 0x55a312677a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:40.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.797+0000 7fb15ce22740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55a3125cfc40 con 0x55a31266c170 2024-03-28T22:46:40.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.797+0000 7fb15ce22740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55a312578c40 con 0x55a31266ca90 2024-03-28T22:46:40.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.797+0000 7fb15ce22740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55a3125cbce0 con 0x55a312675670 2024-03-28T22:46:40.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.797+0000 7fb153400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a31266ca90 0x55a312675130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:40.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.797+0000 7fb152a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a31266c170 0x55a31266c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:40.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.797+0000 7fb153400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a31266ca90 0x55a312675130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:42810/0 (socket says 172.21.15.43:42810) 2024-03-28T22:46:40.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.797+0000 7fb153400640 1 -- 172.21.15.43:0/3711727151 learned_addr learned my addr 172.21.15.43:0/3711727151 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:40.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.797+0000 7fb152a00640 1 --2- 172.21.15.43:0/3711727151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a312675670 0x55a312677a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:40.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.797+0000 7fb153400640 1 -- 172.21.15.43:0/3711727151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a312675670 msgr2=0x55a312677a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:40.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.797+0000 7fb153400640 1 --2- 172.21.15.43:0/3711727151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a312675670 0x55a312677a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.797+0000 7fb153400640 1 -- 172.21.15.43:0/3711727151 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a31266c170 msgr2=0x55a31266c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:40.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.797+0000 7fb153400640 1 --2- 172.21.15.43:0/3711727151 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a31266c170 0x55a31266c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.797+0000 7fb153400640 1 -- 172.21.15.43:0/3711727151 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3123ac7e0 con 0x55a31266ca90 2024-03-28T22:46:40.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.797+0000 7fb152a00640 1 --2- 172.21.15.43:0/3711727151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a312675670 0x55a312677a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:46:40.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.798+0000 7fb153400640 1 --2- 172.21.15.43:0/3711727151 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a31266ca90 0x55a312675130 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fb144037040 tx=0x7fb144009970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4aa468a1b5123658 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:40.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.798+0000 7fb14be00640 1 --2- 172.21.15.43:0/3010953946 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb1380a1ef0 0x7fb1380a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:40.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.798+0000 7fb13fe00640 1 -- 172.21.15.43:0/3711727151 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb144043070 con 0x55a31266ca90 2024-03-28T22:46:40.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.798+0000 7fb13fe00640 1 -- 172.21.15.43:0/3711727151 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb144003e50 con 0x55a31266ca90 2024-03-28T22:46:40.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.798+0000 7fb13fe00640 1 -- 172.21.15.43:0/3711727151 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb144039040 con 0x55a31266ca90 2024-03-28T22:46:40.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.798+0000 7fb14be00640 1 -- 172.21.15.43:0/3010953946 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9539+0+0 (secure 0 0 0) 0x7fb14c0ce7c0 con 0x7fb154106590 2024-03-28T22:46:40.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.798+0000 7fb152a00640 1 --2- 172.21.15.43:0/3010953946 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb1380a1ef0 0x7fb1380a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:40.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.799+0000 7fb15ce22740 1 -- 172.21.15.43:0/3711727151 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a31266ca90 msgr2=0x55a312675130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:40.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.799+0000 7fb15ce22740 1 --2- 172.21.15.43:0/3711727151 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a31266ca90 0x55a312675130 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fb144037040 tx=0x7fb144009970 comp rx=0 tx=0).stop 2024-03-28T22:46:40.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.799+0000 7fb152a00640 1 --2- 172.21.15.43:0/3010953946 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb1380a1ef0 0x7fb1380a43b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x55a312667a30 tx=0x7fb140014040 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:40.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.799+0000 7fb15ce22740 1 -- 172.21.15.43:0/3711727151 shutdown_connections 2024-03-28T22:46:40.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.799+0000 7fb15ce22740 1 --2- 172.21.15.43:0/3711727151 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a31266ca90 0x55a312675130 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.799+0000 7fb15ce22740 1 --2- 172.21.15.43:0/3711727151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a312675670 0x55a312677a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.799+0000 7fb15ce22740 1 --2- 172.21.15.43:0/3711727151 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a31266c170 0x55a31266c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.799+0000 7fb15ce22740 1 -- 172.21.15.43:0/3711727151 >> 172.21.15.43:0/3711727151 conn(0x55a3126668a0 msgr2=0x55a312666c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:40.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.799+0000 7fb15ce22740 1 -- 172.21.15.43:0/3711727151 shutdown_connections 2024-03-28T22:46:40.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.799+0000 7fb15ce22740 1 -- 172.21.15.43:0/3711727151 wait complete. 2024-03-28T22:46:40.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.800+0000 7fb15ce22740 1 -- start start 2024-03-28T22:46:40.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.800+0000 7fb15ce22740 1 Processor -- start 2024-03-28T22:46:40.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.800+0000 7fb15ce22740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a31266c170 0x55a312672660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:40.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.800+0000 7fb15ce22740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a31266ca90 0x55a312672ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:40.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.800+0000 7fb15ce22740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a312675670 0x55a3126730e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:40.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.800+0000 7fb15ce22740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55a312678400 con 0x55a31266ca90 2024-03-28T22:46:40.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.800+0000 7fb15ce22740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55a31256e250 con 0x55a312675670 2024-03-28T22:46:40.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.800+0000 7fb15ce22740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55a312678580 con 0x55a31266c170 2024-03-28T22:46:40.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.801+0000 7fb15ce22740 10 client.0 _ll_register_callbacks cb 0x55a31258fe70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:46:40.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.801+0000 7fb153400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a31266c170 0x55a312672660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:40.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.801+0000 7fb152a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a31266ca90 0x55a312672ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:40.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.801+0000 7fb153400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a31266c170 0x55a312672660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40244/0 (socket says 172.21.15.43:40244) 2024-03-28T22:46:40.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.801+0000 7fb153400640 1 -- 172.21.15.43:0/3884768844 learned_addr learned my addr 172.21.15.43:0/3884768844 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:40.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.801+0000 7fb153e00640 1 --2- 172.21.15.43:0/3884768844 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a312675670 0x55a3126730e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:40.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.801+0000 7fb153400640 1 -- 172.21.15.43:0/3884768844 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a312675670 msgr2=0x55a3126730e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:40.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.801+0000 7fb153400640 1 --2- 172.21.15.43:0/3884768844 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a312675670 0x55a3126730e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.801+0000 7fb153400640 1 -- 172.21.15.43:0/3884768844 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a31266ca90 msgr2=0x55a312672ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:40.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.801+0000 7fb153400640 1 --2- 172.21.15.43:0/3884768844 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a31266ca90 0x55a312672ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.801+0000 7fb153400640 1 -- 172.21.15.43:0/3884768844 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a31240cf70 con 0x55a31266c170 2024-03-28T22:46:40.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.801+0000 7fb153e00640 1 --2- 172.21.15.43:0/3884768844 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a312675670 0x55a3126730e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:40.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.801+0000 7fb153400640 1 --2- 172.21.15.43:0/3884768844 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a31266c170 0x55a312672660 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fb144009ca0 tx=0x7fb14402c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50bb17fb935bfa9b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:40.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.801+0000 7fb13cc00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:46:40.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.802+0000 7fb13cc00640 1 -- 172.21.15.43:0/3884768844 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb14404c070 con 0x55a31266c170 2024-03-28T22:46:40.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.802+0000 7fb15ce22740 1 -- 172.21.15.43:0/3884768844 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb144003e50 con 0x55a31266c170 2024-03-28T22:46:40.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.802+0000 7fb13cc00640 1 -- 172.21.15.43:0/3884768844 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9539+0+0 (secure 0 0 0) 0x7fb1440047e0 con 0x55a31266c170 2024-03-28T22:46:40.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.802+0000 7fb13cc00640 1 -- 172.21.15.43:0/3884768844 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb144058050 con 0x55a31266c170 2024-03-28T22:46:40.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.802+0000 7fb15ce22740 10 client.28831 fetch_fsmap learned FSMap version 32 2024-03-28T22:46:40.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.802+0000 7fb15ce22740 1 -- 172.21.15.43:0/3884768844 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a3122a4170 con 0x55a31266c170 2024-03-28T22:46:40.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.802+0000 7fb13cc00640 1 -- 172.21.15.43:0/3884768844 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb144003e50 con 0x55a31266c170 2024-03-28T22:46:40.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.802+0000 7fb13cc00640 1 -- 172.21.15.43:0/3884768844 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb144048b00 con 0x55a31266c170 2024-03-28T22:46:40.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.803+0000 7fb15ce22740 10 client.28831 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:46:40.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.803+0000 7fb15ce22740 10 client.28831 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:46:40.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.803+0000 7fb15ce22740 20 client.28831 populate_metadata read hostname 'smithi043' 2024-03-28T22:46:40.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.803+0000 7fb15ce22740 1 --2- 172.21.15.43:0/3884768844 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55a312685740 0x55a312687b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:40.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.803+0000 7fb15ce22740 4 client.28831 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:46:40.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.803+0000 7fb15ce22740 1 -- 172.21.15.43:0/3884768844 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a3125dd180 con 0x55a312685740 2024-03-28T22:46:40.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.803+0000 7fb152a00640 1 --2- 172.21.15.43:0/3884768844 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55a312685740 0x55a312687b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:40.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.803+0000 7fb152a00640 1 --2- 172.21.15.43:0/3884768844 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55a312685740 0x55a312687b30 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=853ff3023dd898cd server_cookie=161e0bd7c4bf4cc7 in_seq=0 out_seq=0 2024-03-28T22:46:40.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.803+0000 7fb13cc00640 10 client.28831 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:46:40.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.804+0000 7fb13cc00640 1 -- 172.21.15.43:0/3884768844 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a3125dd180 con 0x55a312685740 2024-03-28T22:46:40.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.805+0000 7fb13cc00640 10 client.28831 handle_command_reply: tid=0 2024-03-28T22:46:40.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.805+0000 7fb15ce22740 1 client.28831 shutdown 2024-03-28T22:46:40.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.805+0000 7fb15ce22740 1 -- 172.21.15.43:0/3884768844 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a31266c170 msgr2=0x55a312672660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:40.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.805+0000 7fb15ce22740 1 --2- 172.21.15.43:0/3884768844 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a31266c170 0x55a312672660 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fb144009ca0 tx=0x7fb14402c820 comp rx=0 tx=0).stop 2024-03-28T22:46:40.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.805+0000 7fb15ce22740 1 -- 172.21.15.43:0/3884768844 shutdown_connections 2024-03-28T22:46:40.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.805+0000 7fb15ce22740 1 --2- 172.21.15.43:0/3884768844 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55a312685740 0x55a312687b30 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.805+0000 7fb15ce22740 1 --2- 172.21.15.43:0/3884768844 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a312675670 0x55a3126730e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.805+0000 7fb15ce22740 1 --2- 172.21.15.43:0/3884768844 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a31266ca90 0x55a312672ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.805+0000 7fb15ce22740 1 --2- 172.21.15.43:0/3884768844 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a31266c170 0x55a312672660 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.805+0000 7fb15ce22740 1 -- 172.21.15.43:0/3884768844 >> 172.21.15.43:0/3884768844 conn(0x55a3126668a0 msgr2=0x55a3126685f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:40.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.805+0000 7fb13cc00640 0 client.28831 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:46:40.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.805+0000 7fb15ce22740 1 -- 172.21.15.43:0/3884768844 shutdown_connections 2024-03-28T22:46:40.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.805+0000 7fb15ce22740 1 -- 172.21.15.43:0/3884768844 wait complete. 2024-03-28T22:46:40.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.805+0000 7fb15ce22740 20 client.28831 trim_cache size 0 max 16384 2024-03-28T22:46:40.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.824+0000 7fb15ce22740 1 Processor -- start 2024-03-28T22:46:40.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.825+0000 7fb15ce22740 1 -- start start 2024-03-28T22:46:40.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.825+0000 7fb15ce22740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a31266c170 0x55a312674d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:40.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.825+0000 7fb15ce22740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a31266ca90 0x55a3126811a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:40.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.825+0000 7fb15ce22740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a312675670 0x55a31268d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:40.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.825+0000 7fb15ce22740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb1541b0c20 con 0x55a31266c170 2024-03-28T22:46:40.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.825+0000 7fb15ce22740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb1541b2760 con 0x55a31266ca90 2024-03-28T22:46:40.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.825+0000 7fb15ce22740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fb1541b12f0 con 0x55a312675670 2024-03-28T22:46:40.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.825+0000 7fb152a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a31266ca90 0x55a3126811a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:40.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.825+0000 7fb153e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a312675670 0x55a31268d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:40.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.825+0000 7fb152a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a31266ca90 0x55a3126811a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:42834/0 (socket says 172.21.15.43:42834) 2024-03-28T22:46:40.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.825+0000 7fb153400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a31266c170 0x55a312674d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:40.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.825+0000 7fb152a00640 1 -- 172.21.15.43:0/2102395501 learned_addr learned my addr 172.21.15.43:0/2102395501 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:40.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.825+0000 7fb153e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a312675670 0x55a31268d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40254/0 (socket says 172.21.15.43:40254) 2024-03-28T22:46:40.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.826+0000 7fb152a00640 1 -- 172.21.15.43:0/2102395501 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a312675670 msgr2=0x55a31268d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:40.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.826+0000 7fb152a00640 1 --2- 172.21.15.43:0/2102395501 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a312675670 0x55a31268d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.826+0000 7fb152a00640 1 -- 172.21.15.43:0/2102395501 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a31266c170 msgr2=0x55a312674d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:40.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.826+0000 7fb152a00640 1 --2- 172.21.15.43:0/2102395501 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a31266c170 0x55a312674d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.826+0000 7fb152a00640 1 -- 172.21.15.43:0/2102395501 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a312657870 con 0x55a31266ca90 2024-03-28T22:46:40.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.826+0000 7fb153e00640 1 --2- 172.21.15.43:0/2102395501 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a312675670 0x55a31268d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:40.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.826+0000 7fb153400640 1 --2- 172.21.15.43:0/2102395501 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a31266c170 0x55a312674d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:40.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.826+0000 7fb152a00640 1 --2- 172.21.15.43:0/2102395501 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a31266ca90 0x55a3126811a0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fb154103bd0 tx=0x7fb1400046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c5f370accb19d78 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:40.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.826+0000 7fb13f400640 1 -- 172.21.15.43:0/2102395501 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1400293c0 con 0x55a31266ca90 2024-03-28T22:46:40.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.826+0000 7fb13f400640 1 -- 172.21.15.43:0/2102395501 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb140012db0 con 0x55a31266ca90 2024-03-28T22:46:40.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.826+0000 7fb13f400640 1 -- 172.21.15.43:0/2102395501 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1400053a0 con 0x55a31266ca90 2024-03-28T22:46:40.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.827+0000 7fb15ce22740 1 -- 172.21.15.43:0/2102395501 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a31266ca90 msgr2=0x55a3126811a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:40.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.827+0000 7fb15ce22740 1 --2- 172.21.15.43:0/2102395501 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a31266ca90 0x55a3126811a0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fb154103bd0 tx=0x7fb1400046c0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.827+0000 7fb15ce22740 1 -- 172.21.15.43:0/2102395501 shutdown_connections 2024-03-28T22:46:40.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.827+0000 7fb15ce22740 1 --2- 172.21.15.43:0/2102395501 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a31266ca90 0x55a3126811a0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.827+0000 7fb15ce22740 1 --2- 172.21.15.43:0/2102395501 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a312675670 0x55a31268d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.827+0000 7fb15ce22740 1 --2- 172.21.15.43:0/2102395501 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a31266c170 0x55a312674d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.827+0000 7fb15ce22740 1 -- 172.21.15.43:0/2102395501 >> 172.21.15.43:0/2102395501 conn(0x55a3126668a0 msgr2=0x55a312667c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:40.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.827+0000 7fb15ce22740 1 -- 172.21.15.43:0/2102395501 shutdown_connections 2024-03-28T22:46:40.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.827+0000 7fb15ce22740 1 -- 172.21.15.43:0/2102395501 wait complete. 2024-03-28T22:46:40.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.827+0000 7fb15ce22740 1 -- start start 2024-03-28T22:46:40.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.828+0000 7fb15ce22740 1 Processor -- start 2024-03-28T22:46:40.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.828+0000 7fb15ce22740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a31266c170 0x55a31266f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:40.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.828+0000 7fb15ce22740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a31266ca90 0x55a31266fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:40.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.828+0000 7fb15ce22740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a312675670 0x55a3126ea790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:40.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.828+0000 7fb15ce22740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb1541b2090 con 0x55a31266c170 2024-03-28T22:46:40.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.828+0000 7fb15ce22740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb1541b19c0 con 0x55a31266ca90 2024-03-28T22:46:40.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.828+0000 7fb15ce22740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fb1541aec20 con 0x55a312675670 2024-03-28T22:46:40.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.828+0000 7fb15ce22740 10 client.0 _ll_register_callbacks cb 0x7fb144009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:46:40.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.828+0000 7fb153400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a31266c170 0x55a31266f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:40.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.828+0000 7fb152a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a31266ca90 0x55a31266fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:40.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.828+0000 7fb153400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a31266c170 0x55a31266f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58880/0 (socket says 172.21.15.43:58880) 2024-03-28T22:46:40.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.828+0000 7fb152a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a31266ca90 0x55a31266fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:42842/0 (socket says 172.21.15.43:42842) 2024-03-28T22:46:40.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.828+0000 7fb153400640 1 -- 172.21.15.43:0/2040128750 learned_addr learned my addr 172.21.15.43:0/2040128750 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:40.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.828+0000 7fb153e00640 1 --2- 172.21.15.43:0/2040128750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a312675670 0x55a3126ea790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:40.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.829+0000 7fb152a00640 1 -- 172.21.15.43:0/2040128750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a312675670 msgr2=0x55a3126ea790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:40.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.829+0000 7fb152a00640 1 --2- 172.21.15.43:0/2040128750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a312675670 0x55a3126ea790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.829+0000 7fb152a00640 1 -- 172.21.15.43:0/2040128750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a31266c170 msgr2=0x55a31266f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:40.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.829+0000 7fb152a00640 1 --2- 172.21.15.43:0/2040128750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a31266c170 0x55a31266f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.829+0000 7fb152a00640 1 -- 172.21.15.43:0/2040128750 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3126ead00 con 0x55a31266ca90 2024-03-28T22:46:40.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.829+0000 7fb152a00640 1 --2- 172.21.15.43:0/2040128750 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a31266ca90 0x55a31266fef0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x55a312667970 tx=0x7fb140012f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b84791c9644d0af6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:40.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.829+0000 7fb13cc00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:46:40.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.829+0000 7fb13cc00640 1 -- 172.21.15.43:0/2040128750 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1400128d0 con 0x55a31266ca90 2024-03-28T22:46:40.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.829+0000 7fb15ce22740 1 -- 172.21.15.43:0/2040128750 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb140012db0 con 0x55a31266ca90 2024-03-28T22:46:40.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.829+0000 7fb13cc00640 1 -- 172.21.15.43:0/2040128750 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9539+0+0 (secure 0 0 0) 0x7fb140012a70 con 0x55a31266ca90 2024-03-28T22:46:40.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.830+0000 7fb13cc00640 1 -- 172.21.15.43:0/2040128750 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb140042050 con 0x55a31266ca90 2024-03-28T22:46:40.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.830+0000 7fb15ce22740 10 client.28901 fetch_fsmap learned FSMap version 32 2024-03-28T22:46:40.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.830+0000 7fb15ce22740 1 -- 172.21.15.43:0/2040128750 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a3126ec040 con 0x55a31266ca90 2024-03-28T22:46:40.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.830+0000 7fb13cc00640 1 -- 172.21.15.43:0/2040128750 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb140012db0 con 0x55a31266ca90 2024-03-28T22:46:40.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.830+0000 7fb13cc00640 1 -- 172.21.15.43:0/2040128750 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb140004ad0 con 0x55a31266ca90 2024-03-28T22:46:40.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.830+0000 7fb15ce22740 10 client.28901 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:46:40.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.830+0000 7fb15ce22740 10 client.28901 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:46:40.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.830+0000 7fb15ce22740 20 client.28901 populate_metadata read hostname 'smithi043' 2024-03-28T22:46:40.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.830+0000 7fb15ce22740 1 --2- 172.21.15.43:0/2040128750 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55a3126ec750 0x55a3126eeb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:40.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.830+0000 7fb15ce22740 4 client.28901 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:46:40.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.830+0000 7fb15ce22740 1 -- 172.21.15.43:0/2040128750 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a3125d74d0 con 0x55a3126ec750 2024-03-28T22:46:40.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.830+0000 7fb153400640 1 --2- 172.21.15.43:0/2040128750 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55a3126ec750 0x55a3126eeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:40.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.831+0000 7fb153400640 1 --2- 172.21.15.43:0/2040128750 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55a3126ec750 0x55a3126eeb70 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fd666f4eaa98421 server_cookie=67e7463661c9e214 in_seq=0 out_seq=0 2024-03-28T22:46:40.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.831+0000 7fb13cc00640 10 client.28901 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:46:40.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.832+0000 7fb13cc00640 1 -- 172.21.15.43:0/2040128750 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a3125d74d0 con 0x55a3126ec750 2024-03-28T22:46:40.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.832+0000 7fb13cc00640 10 client.28901 handle_command_reply: tid=0 2024-03-28T22:46:40.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.832+0000 7fb15ce22740 1 client.28901 shutdown 2024-03-28T22:46:40.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.832+0000 7fb15ce22740 1 -- 172.21.15.43:0/2040128750 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a31266ca90 msgr2=0x55a31266fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:40.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.832+0000 7fb15ce22740 1 --2- 172.21.15.43:0/2040128750 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a31266ca90 0x55a31266fef0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x55a312667970 tx=0x7fb140012f60 comp rx=0 tx=0).stop 2024-03-28T22:46:40.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.833+0000 7fb15ce22740 1 -- 172.21.15.43:0/2040128750 shutdown_connections 2024-03-28T22:46:40.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.833+0000 7fb15ce22740 1 --2- 172.21.15.43:0/2040128750 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55a3126ec750 0x55a3126eeb70 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.833+0000 7fb15ce22740 1 --2- 172.21.15.43:0/2040128750 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a31266ca90 0x55a31266fef0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.833+0000 7fb15ce22740 1 --2- 172.21.15.43:0/2040128750 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55a312675670 0x55a3126ea790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.833+0000 7fb15ce22740 1 --2- 172.21.15.43:0/2040128750 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55a31266c170 0x55a31266f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.833+0000 7fb13cc00640 0 client.28901 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:46:40.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.833+0000 7fb15ce22740 1 -- 172.21.15.43:0/2040128750 >> 172.21.15.43:0/2040128750 conn(0x55a3126668a0 msgr2=0x55a312672e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:40.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.833+0000 7fb15ce22740 1 -- 172.21.15.43:0/2040128750 shutdown_connections 2024-03-28T22:46:40.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.833+0000 7fb15ce22740 1 -- 172.21.15.43:0/2040128750 wait complete. 2024-03-28T22:46:40.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.833+0000 7fb15ce22740 20 client.28901 trim_cache size 0 max 16384 2024-03-28T22:46:40.838 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:46:40.838 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-03-28T22:46:40.838 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "0b0ec336-7f83-4e0f-b3eb-f386a553681f", 2024-03-28T22:46:40.838 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-03-28T22:46:40.838 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:46:40.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.834+0000 7fb158a00640 1 -- 172.21.15.43:0/3010953946 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb1380a1ef0 msgr2=0x7fb1380a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:40.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.834+0000 7fb158a00640 1 --2- 172.21.15.43:0/3010953946 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb1380a1ef0 0x7fb1380a43b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x55a312667a30 tx=0x7fb140014040 comp rx=0 tx=0).stop 2024-03-28T22:46:40.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.834+0000 7fb158a00640 1 -- 172.21.15.43:0/3010953946 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb154106590 msgr2=0x7fb154102bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:40.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.834+0000 7fb158a00640 1 --2- 172.21.15.43:0/3010953946 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb154106590 0x7fb154102bc0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fb14c00e970 tx=0x7fb14c00ee30 comp rx=0 tx=0).stop 2024-03-28T22:46:40.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.834+0000 7fb158a00640 1 -- 172.21.15.43:0/3010953946 shutdown_connections 2024-03-28T22:46:40.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.834+0000 7fb158a00640 1 --2- 172.21.15.43:0/3010953946 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb1380a1ef0 0x7fb1380a43b0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.834+0000 7fb158a00640 1 --2- 172.21.15.43:0/3010953946 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb154106590 0x7fb154102bc0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.834+0000 7fb158a00640 1 --2- 172.21.15.43:0/3010953946 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb154100e90 0x7fb154102680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.834+0000 7fb158a00640 1 --2- 172.21.15.43:0/3010953946 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb1541004d0 0x7fb154102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:40.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.834+0000 7fb158a00640 1 -- 172.21.15.43:0/3010953946 >> 172.21.15.43:0/3010953946 conn(0x7fb1540fbfa0 msgr2=0x7fb1540fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:40.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.834+0000 7fb158a00640 1 -- 172.21.15.43:0/3010953946 shutdown_connections 2024-03-28T22:46:40.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:40.834+0000 7fb158a00640 1 -- 172.21.15.43:0/3010953946 wait complete. 2024-03-28T22:46:40.849 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:46:40.850 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:46:41.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.084+0000 7f290fe00640 1 -- 172.21.15.43:0/1219354449 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2910074410 msgr2=0x7f2910112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:41.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.084+0000 7f290fe00640 1 --2- 172.21.15.43:0/1219354449 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2910074410 0x7f2910112090 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f28fc009900 tx=0x7f28fc02bcd0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.084+0000 7f290fe00640 1 -- 172.21.15.43:0/1219354449 shutdown_connections 2024-03-28T22:46:41.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.084+0000 7f290fe00640 1 --2- 172.21.15.43:0/1219354449 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2910074410 0x7f2910112090 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.084+0000 7f290fe00640 1 --2- 172.21.15.43:0/1219354449 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f29101125d0 0x7f29101149c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.084+0000 7f290fe00640 1 --2- 172.21.15.43:0/1219354449 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2910073af0 0x7f2910073ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.084+0000 7f290fe00640 1 -- 172.21.15.43:0/1219354449 >> 172.21.15.43:0/1219354449 conn(0x7f29100fd0b0 msgr2=0x7f29100ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:41.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.084+0000 7f290fe00640 1 -- 172.21.15.43:0/1219354449 shutdown_connections 2024-03-28T22:46:41.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.084+0000 7f290fe00640 1 -- 172.21.15.43:0/1219354449 wait complete. 2024-03-28T22:46:41.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.085+0000 7f290fe00640 1 Processor -- start 2024-03-28T22:46:41.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.086+0000 7f290fe00640 1 -- start start 2024-03-28T22:46:41.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.086+0000 7f290fe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2910073af0 0x7f291019e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:41.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.086+0000 7f290fe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2910074410 0x7f291019ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:41.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.086+0000 7f290fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f29101125d0 0x7f29101a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:41.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.086+0000 7f290fe00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2910118020 con 0x7f2910074410 2024-03-28T22:46:41.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.086+0000 7f290fe00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f2910117ea0 con 0x7f29101125d0 2024-03-28T22:46:41.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.086+0000 7f290fe00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f29101181a0 con 0x7f2910073af0 2024-03-28T22:46:41.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.086+0000 7f290ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2910073af0 0x7f291019e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:41.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.086+0000 7f290e000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2910074410 0x7f291019ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:41.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.086+0000 7f290e000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2910074410 0x7f291019ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58900/0 (socket says 172.21.15.43:58900) 2024-03-28T22:46:41.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.086+0000 7f290e000640 1 -- 172.21.15.43:0/37269490 learned_addr learned my addr 172.21.15.43:0/37269490 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:41.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.086+0000 7f290f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f29101125d0 0x7f29101a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:41.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.086+0000 7f290ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2910073af0 0x7f291019e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40286/0 (socket says 172.21.15.43:40286) 2024-03-28T22:46:41.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.087+0000 7f290e000640 1 -- 172.21.15.43:0/37269490 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2910073af0 msgr2=0x7f291019e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:41.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.087+0000 7f290e000640 1 --2- 172.21.15.43:0/37269490 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2910073af0 0x7f291019e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.087+0000 7f290e000640 1 -- 172.21.15.43:0/37269490 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f29101125d0 msgr2=0x7f29101a2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:41.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.087+0000 7f290e000640 1 --2- 172.21.15.43:0/37269490 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f29101125d0 0x7f29101a2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.087+0000 7f290e000640 1 -- 172.21.15.43:0/37269490 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29101a34d0 con 0x7f2910074410 2024-03-28T22:46:41.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.087+0000 7f290ea00640 1 --2- 172.21.15.43:0/37269490 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2910073af0 0x7f291019e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:41.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.087+0000 7f290f400640 1 --2- 172.21.15.43:0/37269490 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f29101125d0 0x7f29101a2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:41.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.087+0000 7f290e000640 1 --2- 172.21.15.43:0/37269490 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2910074410 0x7f291019ea20 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f28fc000c00 tx=0x7f28fc0043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:41.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.088+0000 7f2914a00640 1 -- 172.21.15.43:0/37269490 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28fc042020 con 0x7f2910074410 2024-03-28T22:46:41.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.088+0000 7f290fe00640 1 -- 172.21.15.43:0/37269490 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29101a3760 con 0x7f2910074410 2024-03-28T22:46:41.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.088+0000 7f2914a00640 1 -- 172.21.15.43:0/37269490 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28fc0054e0 con 0x7f2910074410 2024-03-28T22:46:41.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.088+0000 7f2914a00640 1 -- 172.21.15.43:0/37269490 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28fc003af0 con 0x7f2910074410 2024-03-28T22:46:41.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.088+0000 7f290fe00640 1 -- 172.21.15.43:0/37269490 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29101ab050 con 0x7f2910074410 2024-03-28T22:46:41.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.091+0000 7f2918a43740 1 Processor -- start 2024-03-28T22:46:41.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.091+0000 7f2918a43740 1 -- start start 2024-03-28T22:46:41.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.092+0000 7f2918a43740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56165b7aadb0 0x56165b7a9b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:41.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.092+0000 7f2914a00640 1 -- 172.21.15.43:0/37269490 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f28fc045020 con 0x7f2910074410 2024-03-28T22:46:41.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.092+0000 7f2918a43740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56165b7aa0d0 0x56165b7aa550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:41.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.092+0000 7f2918a43740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56165b7b3440 0x56165b7b5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:41.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.092+0000 7f2918a43740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56165b70e100 con 0x56165b7aadb0 2024-03-28T22:46:41.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.092+0000 7f2918a43740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x56165b6b6ba0 con 0x56165b7aa0d0 2024-03-28T22:46:41.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.092+0000 7f2918a43740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x56165b70a1a0 con 0x56165b7b3440 2024-03-28T22:46:41.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.092+0000 7f290e000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56165b7b3440 0x56165b7b5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:41.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.092+0000 7f290ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56165b7aa0d0 0x56165b7aa550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:41.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.092+0000 7f290e000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56165b7b3440 0x56165b7b5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40290/0 (socket says 172.21.15.43:40290) 2024-03-28T22:46:41.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.092+0000 7f290ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56165b7aa0d0 0x56165b7aa550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:42864/0 (socket says 172.21.15.43:42864) 2024-03-28T22:46:41.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.092+0000 7f290e000640 1 -- 172.21.15.43:0/1664784748 learned_addr learned my addr 172.21.15.43:0/1664784748 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:41.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.092+0000 7f290f400640 1 --2- 172.21.15.43:0/1664784748 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56165b7aadb0 0x56165b7a9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:41.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.093+0000 7f290e000640 1 -- 172.21.15.43:0/1664784748 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56165b7aa0d0 msgr2=0x56165b7aa550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:41.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.093+0000 7f290e000640 1 --2- 172.21.15.43:0/1664784748 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56165b7aa0d0 0x56165b7aa550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.093+0000 7f290e000640 1 -- 172.21.15.43:0/1664784748 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56165b7aadb0 msgr2=0x56165b7a9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:41.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.093+0000 7f290e000640 1 --2- 172.21.15.43:0/1664784748 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56165b7aadb0 0x56165b7a9b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.093+0000 7f290e000640 1 -- 172.21.15.43:0/1664784748 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56165b6e71a0 con 0x56165b7b3440 2024-03-28T22:46:41.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.093+0000 7f290ea00640 1 --2- 172.21.15.43:0/1664784748 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56165b7aa0d0 0x56165b7aa550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:41.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.093+0000 7f290e000640 1 --2- 172.21.15.43:0/1664784748 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56165b7b3440 0x56165b7b5830 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f28fc0ed5a0 tx=0x7f28fc0f83b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42e5cdcdeebef11e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:41.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.093+0000 7f28fbe00640 1 -- 172.21.15.43:0/1664784748 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28fc0f20d0 con 0x56165b7b3440 2024-03-28T22:46:41.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.093+0000 7f28fbe00640 1 -- 172.21.15.43:0/1664784748 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28fc0fd030 con 0x56165b7b3440 2024-03-28T22:46:41.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.093+0000 7f28fbe00640 1 -- 172.21.15.43:0/1664784748 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28fc101610 con 0x56165b7b3440 2024-03-28T22:46:41.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.093+0000 7f2914a00640 1 --2- 172.21.15.43:0/37269490 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f28f40a1ef0 0x7f28f40a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:41.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.094+0000 7f2914a00640 1 -- 172.21.15.43:0/37269490 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9539+0+0 (secure 0 0 0) 0x7f28fc0f0260 con 0x7f2910074410 2024-03-28T22:46:41.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.094+0000 7f290f400640 1 --2- 172.21.15.43:0/37269490 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f28f40a1ef0 0x7f28f40a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:41.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.094+0000 7f2918a43740 1 -- 172.21.15.43:0/1664784748 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56165b7b3440 msgr2=0x56165b7b5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:41.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.094+0000 7f2918a43740 1 --2- 172.21.15.43:0/1664784748 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56165b7b3440 0x56165b7b5830 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f28fc0ed5a0 tx=0x7f28fc0f83b0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.094+0000 7f2918a43740 1 -- 172.21.15.43:0/1664784748 shutdown_connections 2024-03-28T22:46:41.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.094+0000 7f2918a43740 1 --2- 172.21.15.43:0/1664784748 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56165b7aa0d0 0x56165b7aa550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.094+0000 7f2918a43740 1 --2- 172.21.15.43:0/1664784748 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56165b7b3440 0x56165b7b5830 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.094+0000 7f2918a43740 1 --2- 172.21.15.43:0/1664784748 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56165b7aadb0 0x56165b7a9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.094+0000 7f2918a43740 1 -- 172.21.15.43:0/1664784748 >> 172.21.15.43:0/1664784748 conn(0x56165b770a10 msgr2=0x56165b7a4b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:41.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.094+0000 7f2918a43740 1 -- 172.21.15.43:0/1664784748 shutdown_connections 2024-03-28T22:46:41.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.094+0000 7f290f400640 1 --2- 172.21.15.43:0/37269490 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f28f40a1ef0 0x7f28f40a43b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2908009fd0 tx=0x7f2908009500 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:41.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.095+0000 7f2918a43740 1 -- 172.21.15.43:0/1664784748 wait complete. 2024-03-28T22:46:41.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.095+0000 7f2918a43740 1 -- start start 2024-03-28T22:46:41.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.095+0000 7f2918a43740 1 Processor -- start 2024-03-28T22:46:41.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.096+0000 7f2918a43740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56165b7aa0d0 0x56165b7b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:41.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.096+0000 7f2918a43740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56165b7aadb0 0x56165b7b0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:41.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.096+0000 7f2918a43740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56165b7b3440 0x56165b7b0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:41.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.096+0000 7f2918a43740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x56165b7b6070 con 0x56165b7aa0d0 2024-03-28T22:46:41.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.096+0000 7f2918a43740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x56165b6ac250 con 0x56165b7b3440 2024-03-28T22:46:41.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.096+0000 7f2918a43740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x56165b7b61f0 con 0x56165b7aadb0 2024-03-28T22:46:41.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.096+0000 7f2918a43740 10 client.0 _ll_register_callbacks cb 0x56165b421be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:46:41.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.096+0000 7f290e000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56165b7aadb0 0x56165b7b0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:41.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.096+0000 7f290f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56165b7b3440 0x56165b7b0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:41.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.096+0000 7f290e000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56165b7aadb0 0x56165b7b0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40304/0 (socket says 172.21.15.43:40304) 2024-03-28T22:46:41.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.096+0000 7f290ea00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56165b7aa0d0 0x56165b7b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:41.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.096+0000 7f290e000640 1 -- 172.21.15.43:0/3502763236 learned_addr learned my addr 172.21.15.43:0/3502763236 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:41.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.096+0000 7f290e000640 1 -- 172.21.15.43:0/3502763236 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56165b7b3440 msgr2=0x56165b7b0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:41.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.096+0000 7f290e000640 1 --2- 172.21.15.43:0/3502763236 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56165b7b3440 0x56165b7b0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.096+0000 7f290e000640 1 -- 172.21.15.43:0/3502763236 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56165b7aa0d0 msgr2=0x56165b7b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:41.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.096+0000 7f290e000640 1 --2- 172.21.15.43:0/3502763236 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56165b7aa0d0 0x56165b7b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.096+0000 7f290e000640 1 -- 172.21.15.43:0/3502763236 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56165b6c8960 con 0x56165b7aadb0 2024-03-28T22:46:41.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.097+0000 7f290e000640 1 --2- 172.21.15.43:0/3502763236 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56165b7aadb0 0x56165b7b0930 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f28fc0b92d0 tx=0x7f28fc005e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=884e573a7d4a7dc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:41.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.097+0000 7f28f8c00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:46:41.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.097+0000 7f28f8c00640 1 -- 172.21.15.43:0/3502763236 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28fc0b8ba0 con 0x56165b7aadb0 2024-03-28T22:46:41.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.097+0000 7f2918a43740 1 -- 172.21.15.43:0/3502763236 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28fc0fd030 con 0x56165b7aadb0 2024-03-28T22:46:41.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.097+0000 7f28f8c00640 1 -- 172.21.15.43:0/3502763236 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9539+0+0 (secure 0 0 0) 0x7f28fc0f8d90 con 0x56165b7aadb0 2024-03-28T22:46:41.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.097+0000 7f28f8c00640 1 -- 172.21.15.43:0/3502763236 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f28fc0bc260 con 0x56165b7aadb0 2024-03-28T22:46:41.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.097+0000 7f2918a43740 10 client.28849 fetch_fsmap learned FSMap version 32 2024-03-28T22:46:41.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.097+0000 7f2918a43740 1 -- 172.21.15.43:0/3502763236 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56165b4ec7d0 con 0x56165b7aadb0 2024-03-28T22:46:41.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.097+0000 7f28f8c00640 1 -- 172.21.15.43:0/3502763236 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28fc0fd030 con 0x56165b7aadb0 2024-03-28T22:46:41.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.097+0000 7f28f8c00640 1 -- 172.21.15.43:0/3502763236 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f28fc003c90 con 0x56165b7aadb0 2024-03-28T22:46:41.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.098+0000 7f2918a43740 10 client.28849 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:46:41.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.098+0000 7f2918a43740 10 client.28849 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:46:41.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.098+0000 7f2918a43740 20 client.28849 populate_metadata read hostname 'smithi043' 2024-03-28T22:46:41.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.098+0000 7f2918a43740 1 --2- 172.21.15.43:0/3502763236 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56165b7c3470 0x56165b7c5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:41.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.098+0000 7f2918a43740 4 client.28849 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:46:41.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.098+0000 7f2918a43740 1 -- 172.21.15.43:0/3502763236 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56165b71b640 con 0x56165b7c3470 2024-03-28T22:46:41.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.098+0000 7f290ea00640 1 --2- 172.21.15.43:0/3502763236 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56165b7c3470 0x56165b7c5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:41.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.098+0000 7f290ea00640 1 --2- 172.21.15.43:0/3502763236 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56165b7c3470 0x56165b7c5860 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a66cc99e46a0d992 server_cookie=177a5ff0e8786866 in_seq=0 out_seq=0 2024-03-28T22:46:41.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.098+0000 7f28f8c00640 10 client.28849 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:46:41.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.099+0000 7f28f8c00640 1 -- 172.21.15.43:0/3502763236 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56165b71b640 con 0x56165b7c3470 2024-03-28T22:46:41.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.099+0000 7f28f8c00640 10 client.28849 handle_command_reply: tid=0 2024-03-28T22:46:41.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.099+0000 7f2918a43740 1 client.28849 shutdown 2024-03-28T22:46:41.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.100+0000 7f2918a43740 1 -- 172.21.15.43:0/3502763236 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56165b7aadb0 msgr2=0x56165b7b0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:41.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.100+0000 7f2918a43740 1 --2- 172.21.15.43:0/3502763236 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56165b7aadb0 0x56165b7b0930 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f28fc0b92d0 tx=0x7f28fc005e60 comp rx=0 tx=0).stop 2024-03-28T22:46:41.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.100+0000 7f2918a43740 1 -- 172.21.15.43:0/3502763236 shutdown_connections 2024-03-28T22:46:41.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.100+0000 7f2918a43740 1 --2- 172.21.15.43:0/3502763236 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56165b7c3470 0x56165b7c5860 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.100+0000 7f2918a43740 1 --2- 172.21.15.43:0/3502763236 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56165b7b3440 0x56165b7b0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.100+0000 7f2918a43740 1 --2- 172.21.15.43:0/3502763236 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56165b7aadb0 0x56165b7b0930 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.100+0000 7f2918a43740 1 --2- 172.21.15.43:0/3502763236 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56165b7aa0d0 0x56165b7b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.100+0000 7f28f8c00640 0 client.28849 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:46:41.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.100+0000 7f2918a43740 1 -- 172.21.15.43:0/3502763236 >> 172.21.15.43:0/3502763236 conn(0x56165b770a10 msgr2=0x56165b7a6870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:41.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.100+0000 7f2918a43740 1 -- 172.21.15.43:0/3502763236 shutdown_connections 2024-03-28T22:46:41.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.100+0000 7f2918a43740 1 -- 172.21.15.43:0/3502763236 wait complete. 2024-03-28T22:46:41.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.100+0000 7f2918a43740 20 client.28849 trim_cache size 0 max 16384 2024-03-28T22:46:41.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.119+0000 7f2918a43740 1 Processor -- start 2024-03-28T22:46:41.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.119+0000 7f2918a43740 1 -- start start 2024-03-28T22:46:41.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.120+0000 7f2918a43740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56165b7aa0d0 0x56165b7b2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:41.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.120+0000 7f2918a43740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56165b7aadb0 0x56165b7bee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:41.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.120+0000 7f2918a43740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56165b7b3440 0x56165b7a3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:41.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.120+0000 7f2918a43740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f29101b5160 con 0x56165b7aadb0 2024-03-28T22:46:41.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.120+0000 7f2918a43740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f29101b6ca0 con 0x56165b7aa0d0 2024-03-28T22:46:41.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.120+0000 7f2918a43740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f29101b5830 con 0x56165b7b3440 2024-03-28T22:46:41.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.120+0000 7f290f400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56165b7b3440 0x56165b7a3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:41.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.120+0000 7f290f400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56165b7b3440 0x56165b7a3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40312/0 (socket says 172.21.15.43:40312) 2024-03-28T22:46:41.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.120+0000 7f290f400640 1 -- 172.21.15.43:0/379282939 learned_addr learned my addr 172.21.15.43:0/379282939 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:41.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.120+0000 7f290ea00640 1 --2- 172.21.15.43:0/379282939 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56165b7aa0d0 0x56165b7b2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:41.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.120+0000 7f290f400640 1 -- 172.21.15.43:0/379282939 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56165b7aa0d0 msgr2=0x56165b7b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:41.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.120+0000 7f290f400640 1 --2- 172.21.15.43:0/379282939 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56165b7aa0d0 0x56165b7b2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.120+0000 7f290f400640 1 -- 172.21.15.43:0/379282939 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56165b7aadb0 msgr2=0x56165b7bee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:41.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.120+0000 7f290e000640 1 --2- 172.21.15.43:0/379282939 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56165b7aadb0 0x56165b7bee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:41.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.121+0000 7f290f400640 1 --2- 172.21.15.43:0/379282939 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56165b7aadb0 0x56165b7bee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.121+0000 7f290f400640 1 -- 172.21.15.43:0/379282939 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56165b4223f0 con 0x56165b7b3440 2024-03-28T22:46:41.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.121+0000 7f290e000640 1 --2- 172.21.15.43:0/379282939 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56165b7aadb0 0x56165b7bee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:46:41.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.121+0000 7f290f400640 1 --2- 172.21.15.43:0/379282939 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56165b7b3440 0x56165b7a3b80 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f2908003dc0 tx=0x7f290801b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7271a21cbacf7105 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:41.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.121+0000 7f28fb400640 1 -- 172.21.15.43:0/379282939 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f290800fda0 con 0x56165b7b3440 2024-03-28T22:46:41.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.121+0000 7f28fb400640 1 -- 172.21.15.43:0/379282939 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2908004530 con 0x56165b7b3440 2024-03-28T22:46:41.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.121+0000 7f28fb400640 1 -- 172.21.15.43:0/379282939 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2908005800 con 0x56165b7b3440 2024-03-28T22:46:41.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.122+0000 7f2918a43740 1 -- 172.21.15.43:0/379282939 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56165b7b3440 msgr2=0x56165b7a3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:41.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.122+0000 7f2918a43740 1 --2- 172.21.15.43:0/379282939 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56165b7b3440 0x56165b7a3b80 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f2908003dc0 tx=0x7f290801b000 comp rx=0 tx=0).stop 2024-03-28T22:46:41.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.122+0000 7f2918a43740 1 -- 172.21.15.43:0/379282939 shutdown_connections 2024-03-28T22:46:41.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.122+0000 7f2918a43740 1 --2- 172.21.15.43:0/379282939 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56165b7b3440 0x56165b7a3b80 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.122+0000 7f2918a43740 1 --2- 172.21.15.43:0/379282939 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56165b7aadb0 0x56165b7bee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.122+0000 7f2918a43740 1 --2- 172.21.15.43:0/379282939 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56165b7aa0d0 0x56165b7b2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.122+0000 7f2918a43740 1 -- 172.21.15.43:0/379282939 >> 172.21.15.43:0/379282939 conn(0x56165b770a10 msgr2=0x56165b7a3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:41.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.122+0000 7f2918a43740 1 -- 172.21.15.43:0/379282939 shutdown_connections 2024-03-28T22:46:41.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.122+0000 7f2918a43740 1 -- 172.21.15.43:0/379282939 wait complete. 2024-03-28T22:46:41.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.122+0000 7f2918a43740 1 -- start start 2024-03-28T22:46:41.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.123+0000 7f2918a43740 1 Processor -- start 2024-03-28T22:46:41.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.123+0000 7f2918a43740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56165b7aa0d0 0x56165b7ad8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:41.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.123+0000 7f2918a43740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56165b7aadb0 0x56165b7ade00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:41.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.123+0000 7f2918a43740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56165b7b3440 0x56165b7b40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:41.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.123+0000 7f2918a43740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f29101b65d0 con 0x56165b7b3440 2024-03-28T22:46:41.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.123+0000 7f2918a43740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f29101b5f00 con 0x56165b7aa0d0 2024-03-28T22:46:41.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.123+0000 7f2918a43740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f29101b3160 con 0x56165b7aadb0 2024-03-28T22:46:41.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.123+0000 7f2918a43740 10 client.0 _ll_register_callbacks cb 0x7f2908009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:46:41.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.123+0000 7f290e000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56165b7aadb0 0x56165b7ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:41.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.123+0000 7f290ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56165b7aa0d0 0x56165b7ad8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:41.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.124+0000 7f290f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56165b7b3440 0x56165b7b40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:41.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.124+0000 7f290e000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56165b7aadb0 0x56165b7ade00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:40322/0 (socket says 172.21.15.43:40322) 2024-03-28T22:46:41.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.124+0000 7f290e000640 1 -- 172.21.15.43:0/2750767481 learned_addr learned my addr 172.21.15.43:0/2750767481 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:46:41.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.124+0000 7f290ea00640 1 -- 172.21.15.43:0/2750767481 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56165b7aadb0 msgr2=0x56165b7ade00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:41.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.124+0000 7f290ea00640 1 --2- 172.21.15.43:0/2750767481 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56165b7aadb0 0x56165b7ade00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.124+0000 7f290ea00640 1 -- 172.21.15.43:0/2750767481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56165b7b3440 msgr2=0x56165b7b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:46:41.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.124+0000 7f290ea00640 1 --2- 172.21.15.43:0/2750767481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56165b7b3440 0x56165b7b40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.124+0000 7f290ea00640 1 -- 172.21.15.43:0/2750767481 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56165b3e20f0 con 0x56165b7aa0d0 2024-03-28T22:46:41.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.124+0000 7f290e000640 1 --2- 172.21.15.43:0/2750767481 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56165b7aadb0 0x56165b7ade00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:46:41.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.124+0000 7f290ea00640 1 --2- 172.21.15.43:0/2750767481 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56165b7aa0d0 0x56165b7ad8c0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x56165b7b61d0 tx=0x7f2900012cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c950cad30927f8fb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:46:41.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.124+0000 7f28f8c00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:46:41.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.125+0000 7f28f8c00640 1 -- 172.21.15.43:0/2750767481 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2900010e30 con 0x56165b7aa0d0 2024-03-28T22:46:41.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.125+0000 7f28f8c00640 1 -- 172.21.15.43:0/2750767481 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9539+0+0 (secure 0 0 0) 0x7f2900004020 con 0x56165b7aa0d0 2024-03-28T22:46:41.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.125+0000 7f2918a43740 1 -- 172.21.15.43:0/2750767481 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2908004530 con 0x56165b7aa0d0 2024-03-28T22:46:41.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.125+0000 7f28f8c00640 1 -- 172.21.15.43:0/2750767481 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2900030050 con 0x56165b7aa0d0 2024-03-28T22:46:41.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.125+0000 7f2918a43740 10 client.28919 fetch_fsmap learned FSMap version 32 2024-03-28T22:46:41.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.125+0000 7f2918a43740 1 -- 172.21.15.43:0/2750767481 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56165b829970 con 0x56165b7aa0d0 2024-03-28T22:46:41.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.125+0000 7f28f8c00640 1 -- 172.21.15.43:0/2750767481 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2908004530 con 0x56165b7aa0d0 2024-03-28T22:46:41.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.125+0000 7f28f8c00640 1 -- 172.21.15.43:0/2750767481 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f2900004960 con 0x56165b7aa0d0 2024-03-28T22:46:41.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.125+0000 7f2918a43740 10 client.28919 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:46:41.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.125+0000 7f2918a43740 10 client.28919 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:46:41.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.125+0000 7f2918a43740 20 client.28919 populate_metadata read hostname 'smithi043' 2024-03-28T22:46:41.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.125+0000 7f2918a43740 1 --2- 172.21.15.43:0/2750767481 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56165b82a040 0x56165b82c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:46:41.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.125+0000 7f2918a43740 4 client.28919 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:46:41.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.125+0000 7f2918a43740 1 -- 172.21.15.43:0/2750767481 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56165b715990 con 0x56165b82a040 2024-03-28T22:46:41.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.126+0000 7f290e000640 1 --2- 172.21.15.43:0/2750767481 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56165b82a040 0x56165b82c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:46:41.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.126+0000 7f290e000640 1 --2- 172.21.15.43:0/2750767481 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56165b82a040 0x56165b82c460 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4cb6425370d9e84 server_cookie=96918e4f81e83778 in_seq=0 out_seq=0 2024-03-28T22:46:41.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.126+0000 7f28f8c00640 10 client.28919 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:46:41.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.126+0000 7f28f8c00640 1 -- 172.21.15.43:0/2750767481 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56165b715990 con 0x56165b82a040 2024-03-28T22:46:41.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.126+0000 7f28f8c00640 10 client.28919 handle_command_reply: tid=0 2024-03-28T22:46:41.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.126+0000 7f2918a43740 1 client.28919 shutdown 2024-03-28T22:46:41.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.126+0000 7f2918a43740 1 -- 172.21.15.43:0/2750767481 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56165b7aa0d0 msgr2=0x56165b7ad8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:41.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.126+0000 7f2918a43740 1 --2- 172.21.15.43:0/2750767481 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56165b7aa0d0 0x56165b7ad8c0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x56165b7b61d0 tx=0x7f2900012cf0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.126+0000 7f2918a43740 1 -- 172.21.15.43:0/2750767481 shutdown_connections 2024-03-28T22:46:41.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.126+0000 7f2918a43740 1 --2- 172.21.15.43:0/2750767481 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x56165b7b3440 0x56165b7b40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.126+0000 7f2918a43740 1 --2- 172.21.15.43:0/2750767481 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x56165b7aadb0 0x56165b7ade00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.126+0000 7f2918a43740 1 --2- 172.21.15.43:0/2750767481 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56165b82a040 0x56165b82c460 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.126+0000 7f2918a43740 1 --2- 172.21.15.43:0/2750767481 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56165b7aa0d0 0x56165b7ad8c0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.126+0000 7f2918a43740 1 -- 172.21.15.43:0/2750767481 >> 172.21.15.43:0/2750767481 conn(0x56165b770a10 msgr2=0x56165b7b0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:41.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.126+0000 7f28f8c00640 0 client.28919 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:46:41.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.127+0000 7f2918a43740 1 -- 172.21.15.43:0/2750767481 shutdown_connections 2024-03-28T22:46:41.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.127+0000 7f2918a43740 1 -- 172.21.15.43:0/2750767481 wait complete. 2024-03-28T22:46:41.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.127+0000 7f2918a43740 20 client.28919 trim_cache size 0 max 16384 2024-03-28T22:46:41.132 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:46:41.132 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:46:41.132 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-03-28T22:46:41.132 INFO:teuthology.orchestra.run.smithi043.stdout: "0b0ec336-7f83-4e0f-b3eb-f386a553681f": { 2024-03-28T22:46:41.133 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-03-28T22:46:41.133 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "0b0ec336-7f83-4e0f-b3eb-f386a553681f", 2024-03-28T22:46:41.133 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-03-28T22:46:41.133 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:46:41.133 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:46:41.133 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:46:41.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.127+0000 7f290fe00640 1 -- 172.21.15.43:0/37269490 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f28f40a1ef0 msgr2=0x7f28f40a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:41.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.127+0000 7f290fe00640 1 --2- 172.21.15.43:0/37269490 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f28f40a1ef0 0x7f28f40a43b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2908009fd0 tx=0x7f2908009500 comp rx=0 tx=0).stop 2024-03-28T22:46:41.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.127+0000 7f290fe00640 1 -- 172.21.15.43:0/37269490 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2910074410 msgr2=0x7f291019ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:46:41.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.127+0000 7f290fe00640 1 --2- 172.21.15.43:0/37269490 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2910074410 0x7f291019ea20 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f28fc000c00 tx=0x7f28fc0043c0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.127+0000 7f290fe00640 1 -- 172.21.15.43:0/37269490 shutdown_connections 2024-03-28T22:46:41.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.127+0000 7f290fe00640 1 --2- 172.21.15.43:0/37269490 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f28f40a1ef0 0x7f28f40a43b0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.127+0000 7f290fe00640 1 --2- 172.21.15.43:0/37269490 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f29101125d0 0x7f29101a2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.127+0000 7f290fe00640 1 --2- 172.21.15.43:0/37269490 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2910074410 0x7f291019ea20 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.127+0000 7f290fe00640 1 --2- 172.21.15.43:0/37269490 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2910073af0 0x7f291019e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:46:41.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.127+0000 7f290fe00640 1 -- 172.21.15.43:0/37269490 >> 172.21.15.43:0/37269490 conn(0x7f29100fd0b0 msgr2=0x7f29100fe800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:46:41.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.128+0000 7f290fe00640 1 -- 172.21.15.43:0/37269490 shutdown_connections 2024-03-28T22:46:41.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:46:41.128+0000 7f290fe00640 1 -- 172.21.15.43:0/37269490 wait complete. 2024-03-28T22:46:41.142 INFO:tasks.cephfs.filesystem:scrub status for tag:0b0ec336-7f83-4e0f-b3eb-f386a553681f - {'path': '/', 'tag': '0b0ec336-7f83-4e0f-b3eb-f386a553681f', 'options': 'recursive,force'} 2024-03-28T22:46:41.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:41 smithi043 ceph-mon[31823]: pgmap v296: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:46:41.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:41 smithi172 ceph-mon[36566]: pgmap v296: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:46:41.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:41 smithi082 ceph-mon[36471]: pgmap v296: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:46:42.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:42 smithi172 ceph-mon[36566]: scrub queued for path: / 2024-03-28T22:46:42.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:42 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:46:42.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:42 smithi172 ceph-mon[36566]: scrub summary: active paths [/] 2024-03-28T22:46:42.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:42 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:46:42.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:42 smithi082 ceph-mon[36471]: scrub queued for path: / 2024-03-28T22:46:42.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:42 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:46:42.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:42 smithi082 ceph-mon[36471]: scrub summary: active paths [/] 2024-03-28T22:46:42.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:42 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:46:42.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:42 smithi043 ceph-mon[31823]: scrub queued for path: / 2024-03-28T22:46:42.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:42 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:46:42.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:42 smithi043 ceph-mon[31823]: scrub summary: active paths [/] 2024-03-28T22:46:42.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:42 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:46:43.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:43 smithi172 ceph-mon[36566]: pgmap v297: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:46:43.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:43 smithi082 ceph-mon[36471]: pgmap v297: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:46:43.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:43 smithi043 ceph-mon[31823]: pgmap v297: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:46:44.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:44 smithi172 ceph-mon[36566]: scrub summary: idle 2024-03-28T22:46:44.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:44 smithi082 ceph-mon[36471]: scrub summary: idle 2024-03-28T22:46:44.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:44 smithi043 ceph-mon[31823]: scrub summary: idle 2024-03-28T22:46:45.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:45 smithi172 ceph-mon[36566]: pgmap v298: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 19 op/s 2024-03-28T22:46:45.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:45 smithi082 ceph-mon[36471]: pgmap v298: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 19 op/s 2024-03-28T22:46:45.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:45 smithi043 ceph-mon[31823]: pgmap v298: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 19 op/s 2024-03-28T22:46:47.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:47 smithi172 ceph-mon[36566]: pgmap v299: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 15 op/s 2024-03-28T22:46:47.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:47 smithi082 ceph-mon[36471]: pgmap v299: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 15 op/s 2024-03-28T22:46:47.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:47 smithi043 ceph-mon[31823]: pgmap v299: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 15 op/s 2024-03-28T22:46:49.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:49 smithi172 ceph-mon[36566]: pgmap v300: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 19 op/s 2024-03-28T22:46:49.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:49 smithi082 ceph-mon[36471]: pgmap v300: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 19 op/s 2024-03-28T22:46:49.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:49 smithi043 ceph-mon[31823]: pgmap v300: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 19 op/s 2024-03-28T22:46:51.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:51 smithi172 ceph-mon[36566]: pgmap v301: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 15 op/s 2024-03-28T22:46:51.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:51 smithi082 ceph-mon[36471]: pgmap v301: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 15 op/s 2024-03-28T22:46:51.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:51 smithi043 ceph-mon[31823]: pgmap v301: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 15 op/s 2024-03-28T22:46:53.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:53 smithi172 ceph-mon[36566]: pgmap v302: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 15 op/s 2024-03-28T22:46:53.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:53 smithi082 ceph-mon[36471]: pgmap v302: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 15 op/s 2024-03-28T22:46:53.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:53 smithi043 ceph-mon[31823]: pgmap v302: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 15 op/s 2024-03-28T22:46:55.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:55 smithi172 ceph-mon[36566]: pgmap v303: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 19 op/s 2024-03-28T22:46:55.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:55 smithi082 ceph-mon[36471]: pgmap v303: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 19 op/s 2024-03-28T22:46:55.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:55 smithi043 ceph-mon[31823]: pgmap v303: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 19 op/s 2024-03-28T22:46:57.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:57 smithi172 ceph-mon[36566]: pgmap v304: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:46:57.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:57 smithi082 ceph-mon[36471]: pgmap v304: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:46:57.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:57 smithi043 ceph-mon[31823]: pgmap v304: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:46:59.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:46:59 smithi172 ceph-mon[36566]: pgmap v305: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:46:59.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:46:59 smithi082 ceph-mon[36471]: pgmap v305: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:46:59.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:46:59 smithi043 ceph-mon[31823]: pgmap v305: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:47:01.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:01 smithi172 ceph-mon[36566]: pgmap v306: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:47:01.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:01 smithi082 ceph-mon[36471]: pgmap v306: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:47:01.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:01 smithi043 ceph-mon[31823]: pgmap v306: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:47:03.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:03 smithi172 ceph-mon[36566]: pgmap v307: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:47:03.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:03 smithi082 ceph-mon[36471]: pgmap v307: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:47:03.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:03 smithi043 ceph-mon[31823]: pgmap v307: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:47:05.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:05 smithi172 ceph-mon[36566]: pgmap v308: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-03-28T22:47:05.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:05 smithi082 ceph-mon[36471]: pgmap v308: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-03-28T22:47:05.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:05 smithi043 ceph-mon[31823]: pgmap v308: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-03-28T22:47:07.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:07 smithi172 ceph-mon[36566]: pgmap v309: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-03-28T22:47:07.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:07 smithi082 ceph-mon[36471]: pgmap v309: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-03-28T22:47:07.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:07 smithi043 ceph-mon[31823]: pgmap v309: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-03-28T22:47:09.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:09 smithi172 ceph-mon[36566]: pgmap v310: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-03-28T22:47:09.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:09 smithi082 ceph-mon[36471]: pgmap v310: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-03-28T22:47:09.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:09 smithi043 ceph-mon[31823]: pgmap v310: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-03-28T22:47:10.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:10 smithi082 ceph-mon[36471]: osdmap e107: 12 total, 12 up, 12 in 2024-03-28T22:47:10.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:10 smithi043 ceph-mon[31823]: osdmap e107: 12 total, 12 up, 12 in 2024-03-28T22:47:10.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:10 smithi172 ceph-mon[36566]: osdmap e107: 12 total, 12 up, 12 in 2024-03-28T22:47:11.144 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:47:11.144 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:47:11.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.356+0000 7f165be00640 1 -- 172.21.15.43:0/3933829523 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f165c075c80 msgr2=0x7f165c076060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:11.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.356+0000 7f165be00640 1 --2- 172.21.15.43:0/3933829523 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f165c075c80 0x7f165c076060 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f164c009900 tx=0x7f164c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:47:11.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.356+0000 7f165be00640 1 -- 172.21.15.43:0/3933829523 shutdown_connections 2024-03-28T22:47:11.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.356+0000 7f165be00640 1 --2- 172.21.15.43:0/3933829523 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f165c112470 0x7f165c114860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.356+0000 7f165be00640 1 --2- 172.21.15.43:0/3933829523 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f165c0765a0 0x7f165c111f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.356+0000 7f165be00640 1 --2- 172.21.15.43:0/3933829523 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f165c075c80 0x7f165c076060 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.356+0000 7f165be00640 1 -- 172.21.15.43:0/3933829523 >> 172.21.15.43:0/3933829523 conn(0x7f165c0ff260 msgr2=0x7f165c101680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:11.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.356+0000 7f165be00640 1 -- 172.21.15.43:0/3933829523 shutdown_connections 2024-03-28T22:47:11.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.356+0000 7f165be00640 1 -- 172.21.15.43:0/3933829523 wait complete. 2024-03-28T22:47:11.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.357+0000 7f165be00640 1 Processor -- start 2024-03-28T22:47:11.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.357+0000 7f165be00640 1 -- start start 2024-03-28T22:47:11.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.357+0000 7f165be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f165c075c80 0x7f165c1a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:11.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.358+0000 7f165be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f165c0765a0 0x7f165c1a4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:11.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.358+0000 7f165be00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f165c112470 0x7f165c19e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:11.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.358+0000 7f165be00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f165c118020 con 0x7f165c112470 2024-03-28T22:47:11.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.358+0000 7f165be00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f165c117ea0 con 0x7f165c0765a0 2024-03-28T22:47:11.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.358+0000 7f165be00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f165c1181a0 con 0x7f165c075c80 2024-03-28T22:47:11.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.358+0000 7f165aa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f165c075c80 0x7f165c1a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:11.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.358+0000 7f165aa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f165c075c80 0x7f165c1a4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:34890/0 (socket says 172.21.15.43:34890) 2024-03-28T22:47:11.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.358+0000 7f165aa00640 1 -- 172.21.15.43:0/208442761 learned_addr learned my addr 172.21.15.43:0/208442761 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:11.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.358+0000 7f165a000640 1 --2- 172.21.15.43:0/208442761 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f165c0765a0 0x7f165c1a4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:11.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.358+0000 7f165aa00640 1 -- 172.21.15.43:0/208442761 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f165c0765a0 msgr2=0x7f165c1a4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:11.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.358+0000 7f165aa00640 1 --2- 172.21.15.43:0/208442761 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f165c0765a0 0x7f165c1a4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.358+0000 7f165aa00640 1 -- 172.21.15.43:0/208442761 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f165c112470 msgr2=0x7f165c19e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:11.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.358+0000 7f165aa00640 1 --2- 172.21.15.43:0/208442761 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f165c112470 0x7f165c19e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.358+0000 7f165aa00640 1 -- 172.21.15.43:0/208442761 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f165c19ee00 con 0x7f165c075c80 2024-03-28T22:47:11.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.358+0000 7f165aa00640 1 --2- 172.21.15.43:0/208442761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f165c075c80 0x7f165c1a4430 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f164c02efd0 tx=0x7f164c0042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:11.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.359+0000 7f1653400640 1 -- 172.21.15.43:0/208442761 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f164c041070 con 0x7f165c075c80 2024-03-28T22:47:11.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.359+0000 7f1653400640 1 -- 172.21.15.43:0/208442761 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f164c02ccc0 con 0x7f165c075c80 2024-03-28T22:47:11.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.359+0000 7f165be00640 1 -- 172.21.15.43:0/208442761 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f165c19f090 con 0x7f165c075c80 2024-03-28T22:47:11.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.359+0000 7f1653400640 1 -- 172.21.15.43:0/208442761 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f164c003d10 con 0x7f165c075c80 2024-03-28T22:47:11.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.359+0000 7f165be00640 1 -- 172.21.15.43:0/208442761 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f165c1ab2f0 con 0x7f165c075c80 2024-03-28T22:47:11.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.363+0000 7f1664a35740 1 Processor -- start 2024-03-28T22:47:11.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.363+0000 7f1653400640 1 -- 172.21.15.43:0/208442761 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f164c044070 con 0x7f165c075c80 2024-03-28T22:47:11.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.363+0000 7f1664a35740 1 -- start start 2024-03-28T22:47:11.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.363+0000 7f1664a35740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5643adbb1db0 0x5643adbb0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:11.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.364+0000 7f1664a35740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5643adbb10d0 0x5643adbb1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:11.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.364+0000 7f1664a35740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5643adbba440 0x5643adbbc830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:11.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.364+0000 7f1664a35740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5643adb15100 con 0x5643adbb10d0 2024-03-28T22:47:11.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.364+0000 7f1664a35740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5643adabdba0 con 0x5643adbba440 2024-03-28T22:47:11.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.364+0000 7f1664a35740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5643adb111a0 con 0x5643adbb1db0 2024-03-28T22:47:11.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.364+0000 7f165aa00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5643adbb10d0 0x5643adbb1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:11.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.364+0000 7f165b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5643adbb1db0 0x5643adbb0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:11.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.364+0000 7f165b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5643adbb1db0 0x5643adbb0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:34906/0 (socket says 172.21.15.43:34906) 2024-03-28T22:47:11.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.364+0000 7f165b400640 1 -- 172.21.15.43:0/3650415713 learned_addr learned my addr 172.21.15.43:0/3650415713 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:11.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.364+0000 7f165a000640 1 --2- 172.21.15.43:0/3650415713 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5643adbba440 0x5643adbbc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:11.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.364+0000 7f165b400640 1 -- 172.21.15.43:0/3650415713 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5643adbba440 msgr2=0x5643adbbc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:11.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.364+0000 7f165b400640 1 --2- 172.21.15.43:0/3650415713 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5643adbba440 0x5643adbbc830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.364+0000 7f165b400640 1 -- 172.21.15.43:0/3650415713 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5643adbb10d0 msgr2=0x5643adbb1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:11.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.364+0000 7f165b400640 1 --2- 172.21.15.43:0/3650415713 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5643adbb10d0 0x5643adbb1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.364+0000 7f165b400640 1 -- 172.21.15.43:0/3650415713 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643adaee1a0 con 0x5643adbb1db0 2024-03-28T22:47:11.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.365+0000 7f165b400640 1 --2- 172.21.15.43:0/3650415713 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5643adbb1db0 0x5643adbb0b00 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f1654009e00 tx=0x7f165400e720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5707fa9a3dbfe82 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:11.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.365+0000 7f1653400640 1 --2- 172.21.15.43:0/208442761 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f16400a1f70 0x7f16400a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:11.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.365+0000 7f1647e00640 1 -- 172.21.15.43:0/3650415713 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1654004d20 con 0x5643adbb1db0 2024-03-28T22:47:11.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.365+0000 7f1647e00640 1 -- 172.21.15.43:0/3650415713 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1654003e50 con 0x5643adbb1db0 2024-03-28T22:47:11.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.365+0000 7f1647e00640 1 -- 172.21.15.43:0/3650415713 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1654005020 con 0x5643adbb1db0 2024-03-28T22:47:11.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.365+0000 7f165aa00640 1 --2- 172.21.15.43:0/208442761 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f16400a1f70 0x7f16400a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:11.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.365+0000 7f1653400640 1 -- 172.21.15.43:0/208442761 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9627+0+0 (secure 0 0 0) 0x7f164c0effb0 con 0x7f165c075c80 2024-03-28T22:47:11.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.365+0000 7f1664a35740 1 -- 172.21.15.43:0/3650415713 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5643adbb1db0 msgr2=0x5643adbb0b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:11.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.365+0000 7f1664a35740 1 --2- 172.21.15.43:0/3650415713 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5643adbb1db0 0x5643adbb0b00 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f1654009e00 tx=0x7f165400e720 comp rx=0 tx=0).stop 2024-03-28T22:47:11.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.366+0000 7f1664a35740 1 -- 172.21.15.43:0/3650415713 shutdown_connections 2024-03-28T22:47:11.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.366+0000 7f1664a35740 1 --2- 172.21.15.43:0/3650415713 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5643adbba440 0x5643adbbc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.366+0000 7f1664a35740 1 --2- 172.21.15.43:0/3650415713 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5643adbb10d0 0x5643adbb1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.366+0000 7f1664a35740 1 --2- 172.21.15.43:0/3650415713 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5643adbb1db0 0x5643adbb0b00 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.366+0000 7f165aa00640 1 --2- 172.21.15.43:0/208442761 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f16400a1f70 0x7f16400a4430 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x5643adbb2300 tx=0x7f164c0dc040 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:11.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.366+0000 7f1664a35740 1 -- 172.21.15.43:0/3650415713 >> 172.21.15.43:0/3650415713 conn(0x5643adb77a10 msgr2=0x5643adbabb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:11.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.366+0000 7f1664a35740 1 -- 172.21.15.43:0/3650415713 shutdown_connections 2024-03-28T22:47:11.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.366+0000 7f1664a35740 1 -- 172.21.15.43:0/3650415713 wait complete. 2024-03-28T22:47:11.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.366+0000 7f1664a35740 1 -- start start 2024-03-28T22:47:11.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.366+0000 7f1664a35740 1 Processor -- start 2024-03-28T22:47:11.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.367+0000 7f1664a35740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5643adbb10d0 0x5643adbb7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:11.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.367+0000 7f1664a35740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5643adbb1db0 0x5643adbb7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:11.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.367+0000 7f1664a35740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5643adbba440 0x5643adbb7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:11.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.367+0000 7f1664a35740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5643adbbd2b0 con 0x5643adbb1db0 2024-03-28T22:47:11.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.367+0000 7f1664a35740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5643adab3250 con 0x5643adbba440 2024-03-28T22:47:11.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.367+0000 7f1664a35740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5643adbbd430 con 0x5643adbb10d0 2024-03-28T22:47:11.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.367+0000 7f1664a35740 10 client.0 _ll_register_callbacks cb 0x5643ad828be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:47:11.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.367+0000 7f165b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5643adbb10d0 0x5643adbb7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:11.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.367+0000 7f165b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5643adbb10d0 0x5643adbb7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:34916/0 (socket says 172.21.15.43:34916) 2024-03-28T22:47:11.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.367+0000 7f165b400640 1 -- 172.21.15.43:0/882881470 learned_addr learned my addr 172.21.15.43:0/882881470 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:11.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.367+0000 7f165a000640 1 --2- 172.21.15.43:0/882881470 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5643adbb1db0 0x5643adbb7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:11.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.367+0000 7f165b400640 1 --2- 172.21.15.43:0/882881470 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5643adbba440 0x5643adbb7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:11.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.368+0000 7f165b400640 1 -- 172.21.15.43:0/882881470 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5643adbba440 msgr2=0x5643adbb7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:11.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.368+0000 7f165b400640 1 --2- 172.21.15.43:0/882881470 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5643adbba440 0x5643adbb7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.368+0000 7f165b400640 1 -- 172.21.15.43:0/882881470 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5643adbb1db0 msgr2=0x5643adbb7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:11.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.368+0000 7f165b400640 1 --2- 172.21.15.43:0/882881470 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5643adbb1db0 0x5643adbb7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.368+0000 7f165b400640 1 -- 172.21.15.43:0/882881470 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643adacf960 con 0x5643adbb10d0 2024-03-28T22:47:11.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.368+0000 7f165b400640 1 --2- 172.21.15.43:0/882881470 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5643adbb10d0 0x5643adbb7330 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f1654009560 tx=0x7f1654004490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31d25732096c80db server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:11.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.368+0000 7f1644c00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:47:11.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.368+0000 7f1644c00640 1 -- 172.21.15.43:0/882881470 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1654003970 con 0x5643adbb10d0 2024-03-28T22:47:11.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.368+0000 7f1664a35740 1 -- 172.21.15.43:0/882881470 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1654003e50 con 0x5643adbb10d0 2024-03-28T22:47:11.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.368+0000 7f1644c00640 1 -- 172.21.15.43:0/882881470 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9627+0+0 (secure 0 0 0) 0x7f1654003b10 con 0x5643adbb10d0 2024-03-28T22:47:11.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.368+0000 7f1644c00640 1 -- 172.21.15.43:0/882881470 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1654005180 con 0x5643adbb10d0 2024-03-28T22:47:11.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.368+0000 7f1664a35740 10 client.28867 fetch_fsmap learned FSMap version 32 2024-03-28T22:47:11.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.368+0000 7f1664a35740 1 -- 172.21.15.43:0/882881470 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5643ad8f37d0 con 0x5643adbb10d0 2024-03-28T22:47:11.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.369+0000 7f1644c00640 1 -- 172.21.15.43:0/882881470 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1654003e50 con 0x5643adbb10d0 2024-03-28T22:47:11.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.369+0000 7f1644c00640 1 -- 172.21.15.43:0/882881470 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f1654010670 con 0x5643adbb10d0 2024-03-28T22:47:11.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.369+0000 7f1664a35740 10 client.28867 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:47:11.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.369+0000 7f1664a35740 10 client.28867 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:47:11.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.369+0000 7f1664a35740 20 client.28867 populate_metadata read hostname 'smithi043' 2024-03-28T22:47:11.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.369+0000 7f1664a35740 1 --2- 172.21.15.43:0/882881470 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5643adbca710 0x5643adbccb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:11.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.369+0000 7f1664a35740 4 client.28867 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:47:11.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.369+0000 7f1664a35740 1 -- 172.21.15.43:0/882881470 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5643adb22640 con 0x5643adbca710 2024-03-28T22:47:11.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.369+0000 7f165a000640 1 --2- 172.21.15.43:0/882881470 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5643adbca710 0x5643adbccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:11.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.370+0000 7f165a000640 1 --2- 172.21.15.43:0/882881470 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5643adbca710 0x5643adbccb00 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d91bf0760e23534c server_cookie=88767c6693eac0a6 in_seq=0 out_seq=0 2024-03-28T22:47:11.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.370+0000 7f1644c00640 10 client.28867 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:47:11.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.371+0000 7f1644c00640 1 -- 172.21.15.43:0/882881470 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5643adb22640 con 0x5643adbca710 2024-03-28T22:47:11.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.371+0000 7f1644c00640 10 client.28867 handle_command_reply: tid=0 2024-03-28T22:47:11.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.371+0000 7f1664a35740 1 client.28867 shutdown 2024-03-28T22:47:11.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.371+0000 7f1664a35740 1 -- 172.21.15.43:0/882881470 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5643adbb10d0 msgr2=0x5643adbb7330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:11.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.371+0000 7f1664a35740 1 --2- 172.21.15.43:0/882881470 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5643adbb10d0 0x5643adbb7330 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f1654009560 tx=0x7f1654004490 comp rx=0 tx=0).stop 2024-03-28T22:47:11.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.371+0000 7f1664a35740 1 -- 172.21.15.43:0/882881470 shutdown_connections 2024-03-28T22:47:11.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.371+0000 7f1664a35740 1 --2- 172.21.15.43:0/882881470 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5643adbca710 0x5643adbccb00 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.371+0000 7f1664a35740 1 --2- 172.21.15.43:0/882881470 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5643adbba440 0x5643adbb7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.371+0000 7f1664a35740 1 --2- 172.21.15.43:0/882881470 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5643adbb1db0 0x5643adbb7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.371+0000 7f1644c00640 0 client.28867 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:47:11.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.371+0000 7f1664a35740 1 --2- 172.21.15.43:0/882881470 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5643adbb10d0 0x5643adbb7330 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.371+0000 7f1664a35740 1 -- 172.21.15.43:0/882881470 >> 172.21.15.43:0/882881470 conn(0x5643adb77a10 msgr2=0x5643adbad870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:11.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.371+0000 7f1664a35740 1 -- 172.21.15.43:0/882881470 shutdown_connections 2024-03-28T22:47:11.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.371+0000 7f1664a35740 1 -- 172.21.15.43:0/882881470 wait complete. 2024-03-28T22:47:11.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.372+0000 7f1664a35740 20 client.28867 trim_cache size 0 max 16384 2024-03-28T22:47:11.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.390+0000 7f1664a35740 1 Processor -- start 2024-03-28T22:47:11.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.390+0000 7f1664a35740 1 -- start start 2024-03-28T22:47:11.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.390+0000 7f1664a35740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5643adbb10d0 0x5643adbacf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:11.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.390+0000 7f1664a35740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5643adbb1db0 0x5643adbc6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:11.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.390+0000 7f1664a35740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5643adbba440 0x5643adbb7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:11.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.390+0000 7f1664a35740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f165c1b5240 con 0x5643adbba440 2024-03-28T22:47:11.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.390+0000 7f1664a35740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f165c1b6d80 con 0x5643adbb10d0 2024-03-28T22:47:11.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.390+0000 7f1664a35740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f165c1b5910 con 0x5643adbb1db0 2024-03-28T22:47:11.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.390+0000 7f165b400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5643adbb10d0 0x5643adbacf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:11.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.391+0000 7f165b400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5643adbb10d0 0x5643adbacf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47344/0 (socket says 172.21.15.43:47344) 2024-03-28T22:47:11.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.391+0000 7f165b400640 1 -- 172.21.15.43:0/189330586 learned_addr learned my addr 172.21.15.43:0/189330586 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:11.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.391+0000 7f165a000640 1 --2- 172.21.15.43:0/189330586 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5643adbb1db0 0x5643adbc6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:11.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.391+0000 7f165b400640 1 --2- 172.21.15.43:0/189330586 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5643adbba440 0x5643adbb7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:11.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.391+0000 7f165a000640 1 -- 172.21.15.43:0/189330586 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5643adbb10d0 msgr2=0x5643adbacf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:11.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.391+0000 7f165a000640 1 --2- 172.21.15.43:0/189330586 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5643adbb10d0 0x5643adbacf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.391+0000 7f165a000640 1 -- 172.21.15.43:0/189330586 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5643adbba440 msgr2=0x5643adbb7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:11.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.391+0000 7f165a000640 1 --2- 172.21.15.43:0/189330586 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5643adbba440 0x5643adbb7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.391+0000 7f165a000640 1 -- 172.21.15.43:0/189330586 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643adbade20 con 0x5643adbb1db0 2024-03-28T22:47:11.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.391+0000 7f165b400640 1 --2- 172.21.15.43:0/189330586 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5643adbb10d0 0x5643adbacf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:11.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.391+0000 7f165a000640 1 --2- 172.21.15.43:0/189330586 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5643adbb1db0 0x5643adbc6090 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x5643adbcd250 tx=0x7f1648005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af085b13df085083 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:11.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.392+0000 7f1647400640 1 -- 172.21.15.43:0/189330586 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1648021360 con 0x5643adbb1db0 2024-03-28T22:47:11.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.392+0000 7f1647400640 1 -- 172.21.15.43:0/189330586 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1648025030 con 0x5643adbb1db0 2024-03-28T22:47:11.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.392+0000 7f1647400640 1 -- 172.21.15.43:0/189330586 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f164801e3f0 con 0x5643adbb1db0 2024-03-28T22:47:11.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.392+0000 7f1664a35740 1 -- 172.21.15.43:0/189330586 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5643adbb1db0 msgr2=0x5643adbc6090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:11.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.392+0000 7f1664a35740 1 --2- 172.21.15.43:0/189330586 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5643adbb1db0 0x5643adbc6090 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x5643adbcd250 tx=0x7f1648005f90 comp rx=0 tx=0).stop 2024-03-28T22:47:11.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.392+0000 7f1664a35740 1 -- 172.21.15.43:0/189330586 shutdown_connections 2024-03-28T22:47:11.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.392+0000 7f1664a35740 1 --2- 172.21.15.43:0/189330586 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5643adbba440 0x5643adbb7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.392+0000 7f1664a35740 1 --2- 172.21.15.43:0/189330586 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5643adbb1db0 0x5643adbc6090 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.392+0000 7f1664a35740 1 --2- 172.21.15.43:0/189330586 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5643adbb10d0 0x5643adbacf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.392+0000 7f1664a35740 1 -- 172.21.15.43:0/189330586 >> 172.21.15.43:0/189330586 conn(0x5643adb77a10 msgr2=0x5643adbaad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:11.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.392+0000 7f1664a35740 1 -- 172.21.15.43:0/189330586 shutdown_connections 2024-03-28T22:47:11.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.392+0000 7f1664a35740 1 -- 172.21.15.43:0/189330586 wait complete. 2024-03-28T22:47:11.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.393+0000 7f1664a35740 1 -- start start 2024-03-28T22:47:11.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.393+0000 7f1664a35740 1 Processor -- start 2024-03-28T22:47:11.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.393+0000 7f1664a35740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5643adbb10d0 0x5643adbc2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:11.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.393+0000 7f1664a35740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5643adbb1db0 0x5643adbbdc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:11.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.393+0000 7f1664a35740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5643adbba440 0x5643adbbe1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:11.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.393+0000 7f1664a35740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f165c1b66b0 con 0x5643adbb10d0 2024-03-28T22:47:11.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.393+0000 7f1664a35740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f165c1b5fe0 con 0x5643adbba440 2024-03-28T22:47:11.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.393+0000 7f1664a35740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f165c1b3240 con 0x5643adbb1db0 2024-03-28T22:47:11.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.393+0000 7f1664a35740 10 client.0 _ll_register_callbacks cb 0x7f16540040c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:47:11.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.394+0000 7f165b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5643adbb10d0 0x5643adbc2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:11.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.394+0000 7f165a000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5643adbb1db0 0x5643adbbdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:11.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.394+0000 7f165b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5643adbb10d0 0x5643adbc2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33868/0 (socket says 172.21.15.43:33868) 2024-03-28T22:47:11.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.394+0000 7f165a000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5643adbb1db0 0x5643adbbdc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:34932/0 (socket says 172.21.15.43:34932) 2024-03-28T22:47:11.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.394+0000 7f165b400640 1 -- 172.21.15.43:0/3988800418 learned_addr learned my addr 172.21.15.43:0/3988800418 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:11.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.394+0000 7f165b400640 1 --2- 172.21.15.43:0/3988800418 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5643adbba440 0x5643adbbe1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:11.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.394+0000 7f165a000640 1 -- 172.21.15.43:0/3988800418 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5643adbba440 msgr2=0x5643adbbe1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:11.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.394+0000 7f165a000640 1 --2- 172.21.15.43:0/3988800418 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5643adbba440 0x5643adbbe1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.394+0000 7f165a000640 1 -- 172.21.15.43:0/3988800418 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5643adbb10d0 msgr2=0x5643adbc2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:11.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.394+0000 7f165a000640 1 --2- 172.21.15.43:0/3988800418 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5643adbb10d0 0x5643adbc2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.394+0000 7f165a000640 1 -- 172.21.15.43:0/3988800418 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643ad8293f0 con 0x5643adbb1db0 2024-03-28T22:47:11.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.394+0000 7f165a000640 1 --2- 172.21.15.43:0/3988800418 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5643adbb1db0 0x5643adbbdc60 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f1648011be0 tx=0x7f1648009b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fcce3d4e9d1b66 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:11.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.394+0000 7f1644c00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:47:11.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.394+0000 7f1644c00640 1 -- 172.21.15.43:0/3988800418 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f164802e020 con 0x5643adbb1db0 2024-03-28T22:47:11.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.395+0000 7f1664a35740 1 -- 172.21.15.43:0/3988800418 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1648025030 con 0x5643adbb1db0 2024-03-28T22:47:11.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.395+0000 7f1644c00640 1 -- 172.21.15.43:0/3988800418 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9627+0+0 (secure 0 0 0) 0x7f1648011640 con 0x5643adbb1db0 2024-03-28T22:47:11.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.395+0000 7f1644c00640 1 -- 172.21.15.43:0/3988800418 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1648039050 con 0x5643adbb1db0 2024-03-28T22:47:11.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.395+0000 7f1664a35740 10 client.28873 fetch_fsmap learned FSMap version 32 2024-03-28T22:47:11.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.395+0000 7f1664a35740 1 -- 172.21.15.43:0/3988800418 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5643ad7e90f0 con 0x5643adbb1db0 2024-03-28T22:47:11.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.395+0000 7f1644c00640 1 -- 172.21.15.43:0/3988800418 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1648025030 con 0x5643adbb1db0 2024-03-28T22:47:11.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.395+0000 7f1644c00640 1 -- 172.21.15.43:0/3988800418 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f1648021510 con 0x5643adbb1db0 2024-03-28T22:47:11.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.395+0000 7f1664a35740 10 client.28873 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:47:11.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.395+0000 7f1664a35740 10 client.28873 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:47:11.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.395+0000 7f1664a35740 20 client.28873 populate_metadata read hostname 'smithi043' 2024-03-28T22:47:11.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.395+0000 7f1664a35740 1 --2- 172.21.15.43:0/3988800418 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5643adc30b50 0x5643adc32f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:11.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.395+0000 7f1664a35740 4 client.28873 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:47:11.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.395+0000 7f1664a35740 1 -- 172.21.15.43:0/3988800418 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5643adb1c990 con 0x5643adc30b50 2024-03-28T22:47:11.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.396+0000 7f165b400640 1 --2- 172.21.15.43:0/3988800418 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5643adc30b50 0x5643adc32f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:11.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.396+0000 7f165b400640 1 --2- 172.21.15.43:0/3988800418 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5643adc30b50 0x5643adc32f70 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53dbb8232cbe7fbb server_cookie=1ee95cd5d9f6b938 in_seq=0 out_seq=0 2024-03-28T22:47:11.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.396+0000 7f1644c00640 10 client.28873 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:47:11.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.396+0000 7f1644c00640 1 -- 172.21.15.43:0/3988800418 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5643adb1c990 con 0x5643adc30b50 2024-03-28T22:47:11.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.396+0000 7f1644c00640 10 client.28873 handle_command_reply: tid=0 2024-03-28T22:47:11.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.396+0000 7f1664a35740 1 client.28873 shutdown 2024-03-28T22:47:11.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.396+0000 7f1664a35740 1 -- 172.21.15.43:0/3988800418 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5643adbb1db0 msgr2=0x5643adbbdc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:11.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.396+0000 7f1664a35740 1 --2- 172.21.15.43:0/3988800418 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5643adbb1db0 0x5643adbbdc60 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f1648011be0 tx=0x7f1648009b70 comp rx=0 tx=0).stop 2024-03-28T22:47:11.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.396+0000 7f1664a35740 1 -- 172.21.15.43:0/3988800418 shutdown_connections 2024-03-28T22:47:11.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.396+0000 7f1664a35740 1 --2- 172.21.15.43:0/3988800418 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x5643adc30b50 0x5643adc32f70 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.396+0000 7f1664a35740 1 --2- 172.21.15.43:0/3988800418 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5643adbba440 0x5643adbbe1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.396+0000 7f1664a35740 1 --2- 172.21.15.43:0/3988800418 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5643adbb1db0 0x5643adbbdc60 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.396+0000 7f1664a35740 1 --2- 172.21.15.43:0/3988800418 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5643adbb10d0 0x5643adbc2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.396+0000 7f1664a35740 1 -- 172.21.15.43:0/3988800418 >> 172.21.15.43:0/3988800418 conn(0x5643adb77a10 msgr2=0x5643adbb7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:11.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.396+0000 7f1644c00640 0 client.28873 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:47:11.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.396+0000 7f1664a35740 1 -- 172.21.15.43:0/3988800418 shutdown_connections 2024-03-28T22:47:11.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.397+0000 7f1664a35740 1 -- 172.21.15.43:0/3988800418 wait complete. 2024-03-28T22:47:11.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.397+0000 7f1664a35740 20 client.28873 trim_cache size 0 max 16384 2024-03-28T22:47:11.403 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:47:11.403 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-03-28T22:47:11.403 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-03-28T22:47:11.404 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:47:11.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.397+0000 7f165be00640 1 -- 172.21.15.43:0/208442761 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f16400a1f70 msgr2=0x7f16400a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:11.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.397+0000 7f165be00640 1 --2- 172.21.15.43:0/208442761 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f16400a1f70 0x7f16400a4430 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x5643adbb2300 tx=0x7f164c0dc040 comp rx=0 tx=0).stop 2024-03-28T22:47:11.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.397+0000 7f165be00640 1 -- 172.21.15.43:0/208442761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f165c075c80 msgr2=0x7f165c1a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:11.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.397+0000 7f165be00640 1 --2- 172.21.15.43:0/208442761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f165c075c80 0x7f165c1a4430 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f164c02efd0 tx=0x7f164c0042f0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.397+0000 7f165be00640 1 -- 172.21.15.43:0/208442761 shutdown_connections 2024-03-28T22:47:11.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.397+0000 7f165be00640 1 --2- 172.21.15.43:0/208442761 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f16400a1f70 0x7f16400a4430 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.397+0000 7f165be00640 1 --2- 172.21.15.43:0/208442761 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f165c0765a0 0x7f165c1a4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.397+0000 7f165be00640 1 --2- 172.21.15.43:0/208442761 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f165c112470 0x7f165c19e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.397+0000 7f165be00640 1 --2- 172.21.15.43:0/208442761 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f165c075c80 0x7f165c1a4430 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.397+0000 7f165be00640 1 -- 172.21.15.43:0/208442761 >> 172.21.15.43:0/208442761 conn(0x7f165c0ff260 msgr2=0x7f165c101650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:11.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.397+0000 7f165be00640 1 -- 172.21.15.43:0/208442761 shutdown_connections 2024-03-28T22:47:11.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.398+0000 7f165be00640 1 -- 172.21.15.43:0/208442761 wait complete. 2024-03-28T22:47:11.407 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:11 smithi043 ceph-mon[31823]: pgmap v312: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-28T22:47:11.411 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:47:11.411 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:47:11.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:11 smithi082 ceph-mon[36471]: pgmap v312: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-28T22:47:11.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.618+0000 7fa074600640 1 -- 172.21.15.43:0/359247226 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa06c107780 msgr2=0x7fa06c107b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:11.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.618+0000 7fa074600640 1 --2- 172.21.15.43:0/359247226 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa06c107780 0x7fa06c107b60 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fa060009900 tx=0x7fa06002bc30 comp rx=0 tx=0).stop 2024-03-28T22:47:11.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.618+0000 7fa074600640 1 -- 172.21.15.43:0/359247226 shutdown_connections 2024-03-28T22:47:11.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.618+0000 7fa074600640 1 --2- 172.21.15.43:0/359247226 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa06c102130 0x7fa06c10e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.618+0000 7fa074600640 1 --2- 172.21.15.43:0/359247226 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa06c101770 0x7fa06c101bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.618+0000 7fa074600640 1 --2- 172.21.15.43:0/359247226 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa06c107780 0x7fa06c107b60 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.618+0000 7fa074600640 1 -- 172.21.15.43:0/359247226 >> 172.21.15.43:0/359247226 conn(0x7fa06c0fd220 msgr2=0x7fa06c0ff640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:11.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.618+0000 7fa074600640 1 -- 172.21.15.43:0/359247226 shutdown_connections 2024-03-28T22:47:11.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.618+0000 7fa074600640 1 -- 172.21.15.43:0/359247226 wait complete. 2024-03-28T22:47:11.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.620+0000 7fa074600640 1 Processor -- start 2024-03-28T22:47:11.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.620+0000 7fa074600640 1 -- start start 2024-03-28T22:47:11.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.620+0000 7fa074600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa06c101770 0x7fa06c19a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:11.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.620+0000 7fa074600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa06c102130 0x7fa06c19a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:11.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.620+0000 7fa074600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa06c107780 0x7fa06c19eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:11.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.620+0000 7fa074600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa06c111ba0 con 0x7fa06c101770 2024-03-28T22:47:11.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.620+0000 7fa074600640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa06c111a20 con 0x7fa06c102130 2024-03-28T22:47:11.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.620+0000 7fa074600640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fa06c111d20 con 0x7fa06c107780 2024-03-28T22:47:11.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.621+0000 7fa071200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa06c102130 0x7fa06c19a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:11.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.621+0000 7fa071c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa06c101770 0x7fa06c19a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:11.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.621+0000 7fa071200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa06c102130 0x7fa06c19a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47370/0 (socket says 172.21.15.43:47370) 2024-03-28T22:47:11.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.621+0000 7fa071200640 1 -- 172.21.15.43:0/2992193547 learned_addr learned my addr 172.21.15.43:0/2992193547 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:11.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.621+0000 7fa071c00640 1 --2- 172.21.15.43:0/2992193547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa06c101770 0x7fa06c19a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33884/0 (socket says 172.21.15.43:33884) 2024-03-28T22:47:11.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.621+0000 7fa072600640 1 --2- 172.21.15.43:0/2992193547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa06c107780 0x7fa06c19eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:11.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.621+0000 7fa071200640 1 -- 172.21.15.43:0/2992193547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa06c107780 msgr2=0x7fa06c19eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:11.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.621+0000 7fa071200640 1 --2- 172.21.15.43:0/2992193547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa06c107780 0x7fa06c19eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.621+0000 7fa071200640 1 -- 172.21.15.43:0/2992193547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa06c101770 msgr2=0x7fa06c19a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:11.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.621+0000 7fa071200640 1 --2- 172.21.15.43:0/2992193547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa06c101770 0x7fa06c19a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.621+0000 7fa071200640 1 -- 172.21.15.43:0/2992193547 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa06c19f250 con 0x7fa06c102130 2024-03-28T22:47:11.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.621+0000 7fa071200640 1 --2- 172.21.15.43:0/2992193547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa06c102130 0x7fa06c19a7a0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fa05800ea60 tx=0x7fa05800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:11.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.621+0000 7fa06a000640 1 -- 172.21.15.43:0/2992193547 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa05800cde0 con 0x7fa06c102130 2024-03-28T22:47:11.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.622+0000 7fa06a000640 1 -- 172.21.15.43:0/2992193547 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0580035f0 con 0x7fa06c102130 2024-03-28T22:47:11.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.622+0000 7fa074600640 1 -- 172.21.15.43:0/2992193547 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa06c19f540 con 0x7fa06c102130 2024-03-28T22:47:11.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.622+0000 7fa06a000640 1 -- 172.21.15.43:0/2992193547 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa058010670 con 0x7fa06c102130 2024-03-28T22:47:11.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.622+0000 7fa074600640 1 -- 172.21.15.43:0/2992193547 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa06c1a6dd0 con 0x7fa06c102130 2024-03-28T22:47:11.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.623+0000 7fa074600640 1 -- 172.21.15.43:0/2992193547 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa06c067cb0 con 0x7fa06c102130 2024-03-28T22:47:11.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.625+0000 7fa06a000640 1 -- 172.21.15.43:0/2992193547 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fa0580037a0 con 0x7fa06c102130 2024-03-28T22:47:11.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.626+0000 7fa06a000640 1 --2- 172.21.15.43:0/2992193547 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa0540a1ef0 0x7fa0540a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:11.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.627+0000 7fa06a000640 1 -- 172.21.15.43:0/2992193547 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9627+0+0 (secure 0 0 0) 0x7fa0580ce870 con 0x7fa06c102130 2024-03-28T22:47:11.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.627+0000 7fa071c00640 1 --2- 172.21.15.43:0/2992193547 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa0540a1ef0 0x7fa0540a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:11.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.628+0000 7fa071c00640 1 --2- 172.21.15.43:0/2992193547 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa0540a1ef0 0x7fa0540a43b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa06002efd0 tx=0x7fa060000bc0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:11.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.635+0000 7fa06a000640 1 -- 172.21.15.43:0/2992193547 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa05809b510 con 0x7fa06c102130 2024-03-28T22:47:11.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:11 smithi172 ceph-mon[36566]: pgmap v312: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-28T22:47:11.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.867+0000 7fa074600640 1 -- 172.21.15.43:0/2992193547 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa06c1a7070 con 0x7fa06c102130 2024-03-28T22:47:11.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.868+0000 7fa06a000640 1 -- 172.21.15.43:0/2992193547 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11931 (secure 0 0 0) 0x7fa05809ae10 con 0x7fa06c102130 2024-03-28T22:47:11.866 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:47:11.866 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:33:28.906396+0000","modified":"2024-03-28T22:40:08.672168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24572,"mds_1":24556,"mds_2":24535,"mds_3":24523,"mds_4":24518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26717":{"gid":26717,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/215779342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":215779342},{"type":"v1","addr":"172.21.15.82:6835","nonce":215779342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24535,24556,24518,24523,24572]},"id":1}]} 2024-03-28T22:47:11.866 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 32 2024-03-28T22:47:11.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.870+0000 7fa074600640 1 -- 172.21.15.43:0/2992193547 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa0540a1ef0 msgr2=0x7fa0540a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:11.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.870+0000 7fa074600640 1 --2- 172.21.15.43:0/2992193547 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa0540a1ef0 0x7fa0540a43b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa06002efd0 tx=0x7fa060000bc0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.870+0000 7fa074600640 1 -- 172.21.15.43:0/2992193547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa06c102130 msgr2=0x7fa06c19a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:11.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.870+0000 7fa074600640 1 --2- 172.21.15.43:0/2992193547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa06c102130 0x7fa06c19a7a0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fa05800ea60 tx=0x7fa05800ef20 comp rx=0 tx=0).stop 2024-03-28T22:47:11.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.870+0000 7fa074600640 1 -- 172.21.15.43:0/2992193547 shutdown_connections 2024-03-28T22:47:11.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.870+0000 7fa074600640 1 --2- 172.21.15.43:0/2992193547 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa0540a1ef0 0x7fa0540a43b0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.870+0000 7fa074600640 1 --2- 172.21.15.43:0/2992193547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa06c102130 0x7fa06c19a7a0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.870+0000 7fa074600640 1 --2- 172.21.15.43:0/2992193547 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa06c107780 0x7fa06c19eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.870+0000 7fa074600640 1 --2- 172.21.15.43:0/2992193547 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa06c101770 0x7fa06c19a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:11.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.870+0000 7fa074600640 1 -- 172.21.15.43:0/2992193547 >> 172.21.15.43:0/2992193547 conn(0x7fa06c0fd220 msgr2=0x7fa06c10cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:11.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.871+0000 7fa074600640 1 -- 172.21.15.43:0/2992193547 shutdown_connections 2024-03-28T22:47:11.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:11.871+0000 7fa074600640 1 -- 172.21.15.43:0/2992193547 wait complete. 2024-03-28T22:47:11.887 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T22:47:12.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.108+0000 7f6f0d800640 1 -- 172.21.15.43:0/3113373973 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f08104190 msgr2=0x7f6f081106a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.108+0000 7f6f0d800640 1 --2- 172.21.15.43:0/3113373973 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f08104190 0x7f6f081106a0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f6f00009ca0 tx=0x7f6f0002c0f0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.108+0000 7f6f0d800640 1 -- 172.21.15.43:0/3113373973 shutdown_connections 2024-03-28T22:47:12.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.108+0000 7f6f0d800640 1 --2- 172.21.15.43:0/3113373973 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f08104190 0x7f6f081106a0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.108+0000 7f6f0d800640 1 --2- 172.21.15.43:0/3113373973 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f081037d0 0x7f6f08103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.108+0000 7f6f0d800640 1 --2- 172.21.15.43:0/3113373973 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f081097e0 0x7f6f08109bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.108+0000 7f6f0d800640 1 -- 172.21.15.43:0/3113373973 >> 172.21.15.43:0/3113373973 conn(0x7f6f080ff280 msgr2=0x7f6f081016a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:12.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.108+0000 7f6f0d800640 1 -- 172.21.15.43:0/3113373973 shutdown_connections 2024-03-28T22:47:12.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.108+0000 7f6f0d800640 1 -- 172.21.15.43:0/3113373973 wait complete. 2024-03-28T22:47:12.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.109+0000 7f6f0d800640 1 Processor -- start 2024-03-28T22:47:12.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.110+0000 7f6f0d800640 1 -- start start 2024-03-28T22:47:12.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.110+0000 7f6f0d800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f081037d0 0x7f6f0819a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.110+0000 7f6f0d800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f08104190 0x7f6f0819a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.110+0000 7f6f0d800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f081097e0 0x7f6f0819e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.110+0000 7f6f0d800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6f08113bf0 con 0x7f6f081037d0 2024-03-28T22:47:12.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.110+0000 7f6f0d800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f6f08113a70 con 0x7f6f08104190 2024-03-28T22:47:12.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.110+0000 7f6f0d800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f6f08113d70 con 0x7f6f081097e0 2024-03-28T22:47:12.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.110+0000 7f6f07400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f08104190 0x7f6f0819a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.110+0000 7f6f07e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f081037d0 0x7f6f0819a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.110+0000 7f6f0ce00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f081097e0 0x7f6f0819e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.110+0000 7f6f07400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f08104190 0x7f6f0819a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47390/0 (socket says 172.21.15.43:47390) 2024-03-28T22:47:12.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.110+0000 7f6f07400640 1 -- 172.21.15.43:0/3062832370 learned_addr learned my addr 172.21.15.43:0/3062832370 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:12.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.111+0000 7f6f07400640 1 -- 172.21.15.43:0/3062832370 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f081097e0 msgr2=0x7f6f0819e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.111+0000 7f6f07400640 1 --2- 172.21.15.43:0/3062832370 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f081097e0 0x7f6f0819e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.111+0000 7f6f07400640 1 -- 172.21.15.43:0/3062832370 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f081037d0 msgr2=0x7f6f0819a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.111+0000 7f6f07400640 1 --2- 172.21.15.43:0/3062832370 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f081037d0 0x7f6f0819a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.111+0000 7f6f07400640 1 -- 172.21.15.43:0/3062832370 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f0819f0c0 con 0x7f6f08104190 2024-03-28T22:47:12.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.111+0000 7f6f07400640 1 --2- 172.21.15.43:0/3062832370 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f08104190 0x7f6f0819a610 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f6ef400e9f0 tx=0x7f6ef400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:12.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.111+0000 7f6f04c00640 1 -- 172.21.15.43:0/3062832370 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef400cd80 con 0x7f6f08104190 2024-03-28T22:47:12.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.111+0000 7f6f0d800640 1 -- 172.21.15.43:0/3062832370 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f0819f3b0 con 0x7f6f08104190 2024-03-28T22:47:12.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.111+0000 7f6f04c00640 1 -- 172.21.15.43:0/3062832370 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ef4005500 con 0x7f6f08104190 2024-03-28T22:47:12.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.111+0000 7f6f04c00640 1 -- 172.21.15.43:0/3062832370 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef4010690 con 0x7f6f08104190 2024-03-28T22:47:12.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.111+0000 7f6f0d800640 1 -- 172.21.15.43:0/3062832370 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f081a6c40 con 0x7f6f08104190 2024-03-28T22:47:12.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.115+0000 7f6f11b91740 1 Processor -- start 2024-03-28T22:47:12.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.115+0000 7f6f11b91740 1 -- start start 2024-03-28T22:47:12.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.115+0000 7f6f11b91740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ea919a550 0x563ea919a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.115+0000 7f6f11b91740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ea919ae70 0x563ea91a3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.115+0000 7f6f11b91740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ea91a3a50 0x563ea91a5e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.115+0000 7f6f11b91740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x563ea90fe050 con 0x563ea919a550 2024-03-28T22:47:12.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.115+0000 7f6f11b91740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x563ea90a6ce0 con 0x563ea91a3a50 2024-03-28T22:47:12.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.115+0000 7f6f11b91740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x563ea90fa0f0 con 0x563ea919ae70 2024-03-28T22:47:12.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.115+0000 7f6f04c00640 1 -- 172.21.15.43:0/3062832370 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f6ef40046c0 con 0x7f6f08104190 2024-03-28T22:47:12.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.115+0000 7f6f07e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ea919ae70 0x563ea91a3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.115+0000 7f6f0ce00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ea919a550 0x563ea919a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.115+0000 7f6f07e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ea919ae70 0x563ea91a3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:34978/0 (socket says 172.21.15.43:34978) 2024-03-28T22:47:12.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.115+0000 7f6f07e00640 1 -- 172.21.15.43:0/4080442724 learned_addr learned my addr 172.21.15.43:0/4080442724 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:12.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.116+0000 7f6f07400640 1 --2- 172.21.15.43:0/4080442724 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ea91a3a50 0x563ea91a5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.116+0000 7f6f07e00640 1 -- 172.21.15.43:0/4080442724 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ea91a3a50 msgr2=0x563ea91a5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:12.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.116+0000 7f6f07e00640 1 --2- 172.21.15.43:0/4080442724 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ea91a3a50 0x563ea91a5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.116+0000 7f6f07e00640 1 -- 172.21.15.43:0/4080442724 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ea919a550 msgr2=0x563ea919a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:12.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.116+0000 7f6f07e00640 1 --2- 172.21.15.43:0/4080442724 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ea919a550 0x563ea919a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.116+0000 7f6f07e00640 1 -- 172.21.15.43:0/4080442724 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ea8e123f0 con 0x563ea919ae70 2024-03-28T22:47:12.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.116+0000 7f6f07400640 1 --2- 172.21.15.43:0/4080442724 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ea91a3a50 0x563ea91a5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:47:12.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.116+0000 7f6f07e00640 1 --2- 172.21.15.43:0/4080442724 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ea919ae70 0x563ea91a3510 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8009c70 tx=0x7f6ef8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c27fbd1d96b61b06 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:12.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.116+0000 7f6efd600640 1 -- 172.21.15.43:0/4080442724 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8004230 con 0x563ea919ae70 2024-03-28T22:47:12.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.116+0000 7f6efd600640 1 -- 172.21.15.43:0/4080442724 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ef80043d0 con 0x563ea919ae70 2024-03-28T22:47:12.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.116+0000 7f6efd600640 1 -- 172.21.15.43:0/4080442724 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8019820 con 0x563ea919ae70 2024-03-28T22:47:12.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.117+0000 7f6f04c00640 1 --2- 172.21.15.43:0/3062832370 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6eec0a1ef0 0x7f6eec0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.117+0000 7f6f04c00640 1 -- 172.21.15.43:0/3062832370 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9627+0+0 (secure 0 0 0) 0x7f6ef40cea60 con 0x7f6f08104190 2024-03-28T22:47:12.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.117+0000 7f6f0ce00640 1 --2- 172.21.15.43:0/3062832370 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6eec0a1ef0 0x7f6eec0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.117+0000 7f6f11b91740 1 -- 172.21.15.43:0/4080442724 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ea919ae70 msgr2=0x563ea91a3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.117+0000 7f6f11b91740 1 --2- 172.21.15.43:0/4080442724 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ea919ae70 0x563ea91a3510 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8009c70 tx=0x7f6ef8011810 comp rx=0 tx=0).stop 2024-03-28T22:47:12.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.118+0000 7f6f11b91740 1 -- 172.21.15.43:0/4080442724 shutdown_connections 2024-03-28T22:47:12.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.118+0000 7f6f11b91740 1 --2- 172.21.15.43:0/4080442724 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ea91a3a50 0x563ea91a5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.118+0000 7f6f11b91740 1 --2- 172.21.15.43:0/4080442724 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ea919ae70 0x563ea91a3510 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.118+0000 7f6f11b91740 1 --2- 172.21.15.43:0/4080442724 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ea919a550 0x563ea919a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.118+0000 7f6f11b91740 1 -- 172.21.15.43:0/4080442724 >> 172.21.15.43:0/4080442724 conn(0x563ea9194c80 msgr2=0x563ea9195060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:12.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.118+0000 7f6f11b91740 1 -- 172.21.15.43:0/4080442724 shutdown_connections 2024-03-28T22:47:12.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.118+0000 7f6f11b91740 1 -- 172.21.15.43:0/4080442724 wait complete. 2024-03-28T22:47:12.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.118+0000 7f6f0ce00640 1 --2- 172.21.15.43:0/3062832370 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6eec0a1ef0 0x7f6eec0a43b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6f00005e30 tx=0x7f6f00005d80 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:12.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.118+0000 7f6f11b91740 1 -- start start 2024-03-28T22:47:12.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.118+0000 7f6f11b91740 1 Processor -- start 2024-03-28T22:47:12.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.119+0000 7f6f11b91740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ea919a550 0x563ea91a09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.119+0000 7f6f11b91740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ea919ae70 0x563ea91a0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.119+0000 7f6f11b91740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ea91a3a50 0x563ea91a1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.119+0000 7f6f11b91740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x563ea91a6840 con 0x563ea91a3a50 2024-03-28T22:47:12.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.119+0000 7f6f11b91740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x563ea909c2f0 con 0x563ea919a550 2024-03-28T22:47:12.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.119+0000 7f6f11b91740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x563ea91a69c0 con 0x563ea919ae70 2024-03-28T22:47:12.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.119+0000 7f6f11b91740 10 client.0 _ll_register_callbacks cb 0x563ea8e11be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:47:12.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.119+0000 7f6f07e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ea919a550 0x563ea91a09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.119+0000 7f6f07e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ea919a550 0x563ea91a09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47398/0 (socket says 172.21.15.43:47398) 2024-03-28T22:47:12.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.119+0000 7f6f07e00640 1 -- 172.21.15.43:0/3983024901 learned_addr learned my addr 172.21.15.43:0/3983024901 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:12.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.119+0000 7f6f07400640 1 --2- 172.21.15.43:0/3983024901 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ea919ae70 0x563ea91a0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.119+0000 7f6f0ce00640 1 --2- 172.21.15.43:0/3983024901 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ea91a3a50 0x563ea91a1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.119+0000 7f6f07400640 1 -- 172.21.15.43:0/3983024901 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ea919a550 msgr2=0x563ea91a09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:12.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.119+0000 7f6f07400640 1 --2- 172.21.15.43:0/3983024901 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ea919a550 0x563ea91a09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.119+0000 7f6f07400640 1 -- 172.21.15.43:0/3983024901 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ea91a3a50 msgr2=0x563ea91a1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:12.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.119+0000 7f6f07400640 1 --2- 172.21.15.43:0/3983024901 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ea91a3a50 0x563ea91a1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.119+0000 7f6f07400640 1 -- 172.21.15.43:0/3983024901 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ea8edc7d0 con 0x563ea919ae70 2024-03-28T22:47:12.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.119+0000 7f6f07e00640 1 --2- 172.21.15.43:0/3983024901 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ea919a550 0x563ea91a09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:12.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.119+0000 7f6f0ce00640 1 --2- 172.21.15.43:0/3983024901 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ea91a3a50 0x563ea91a1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:47:12.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.120+0000 7f6f07400640 1 --2- 172.21.15.43:0/3983024901 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ea919ae70 0x563ea91a0f20 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f6ef40cbd00 tx=0x7f6ef409adf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff7129ff3f37ba83 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:12.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.120+0000 7f6ef2000640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:47:12.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.120+0000 7f6ef2000640 1 -- 172.21.15.43:0/3983024901 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef40cc190 con 0x563ea919ae70 2024-03-28T22:47:12.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.120+0000 7f6ef2000640 1 -- 172.21.15.43:0/3983024901 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9627+0+0 (secure 0 0 0) 0x7f6ef40a34b0 con 0x563ea919ae70 2024-03-28T22:47:12.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.120+0000 7f6f11b91740 1 -- 172.21.15.43:0/3983024901 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ef80043d0 con 0x563ea919ae70 2024-03-28T22:47:12.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.120+0000 7f6ef2000640 1 -- 172.21.15.43:0/3983024901 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6ef40b8050 con 0x563ea919ae70 2024-03-28T22:47:12.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.120+0000 7f6f11b91740 10 client.28891 fetch_fsmap learned FSMap version 32 2024-03-28T22:47:12.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.120+0000 7f6f11b91740 1 -- 172.21.15.43:0/3983024901 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563ea91b2a70 con 0x563ea919ae70 2024-03-28T22:47:12.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.120+0000 7f6ef2000640 1 -- 172.21.15.43:0/3983024901 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ef80043d0 con 0x563ea919ae70 2024-03-28T22:47:12.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.121+0000 7f6ef2000640 1 -- 172.21.15.43:0/3983024901 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6ef409a3f0 con 0x563ea919ae70 2024-03-28T22:47:12.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.121+0000 7f6f11b91740 10 client.28891 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:47:12.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.121+0000 7f6f11b91740 10 client.28891 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:47:12.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.121+0000 7f6f11b91740 20 client.28891 populate_metadata read hostname 'smithi043' 2024-03-28T22:47:12.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.121+0000 7f6f11b91740 1 --2- 172.21.15.43:0/3983024901 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x563ea91b3ed0 0x563ea91b62f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.121+0000 7f6f11b91740 4 client.28891 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:47:12.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.121+0000 7f6f11b91740 1 -- 172.21.15.43:0/3983024901 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ea90a7680 con 0x563ea91b3ed0 2024-03-28T22:47:12.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.121+0000 7f6f07e00640 1 --2- 172.21.15.43:0/3983024901 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x563ea91b3ed0 0x563ea91b62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.121+0000 7f6f07e00640 1 --2- 172.21.15.43:0/3983024901 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x563ea91b3ed0 0x563ea91b62f0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9a407fbd18054f0c server_cookie=67415f46cca85009 in_seq=0 out_seq=0 2024-03-28T22:47:12.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.121+0000 7f6ef2000640 10 client.28891 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:47:12.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.122+0000 7f6ef2000640 1 -- 172.21.15.43:0/3983024901 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563ea90a7680 con 0x563ea91b3ed0 2024-03-28T22:47:12.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.122+0000 7f6ef2000640 10 client.28891 handle_command_reply: tid=0 2024-03-28T22:47:12.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.122+0000 7f6f11b91740 1 client.28891 shutdown 2024-03-28T22:47:12.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.122+0000 7f6f11b91740 1 -- 172.21.15.43:0/3983024901 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ea919ae70 msgr2=0x563ea91a0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.122+0000 7f6f11b91740 1 --2- 172.21.15.43:0/3983024901 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ea919ae70 0x563ea91a0f20 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f6ef40cbd00 tx=0x7f6ef409adf0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.122+0000 7f6f11b91740 1 -- 172.21.15.43:0/3983024901 shutdown_connections 2024-03-28T22:47:12.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.122+0000 7f6f11b91740 1 --2- 172.21.15.43:0/3983024901 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x563ea91b3ed0 0x563ea91b62f0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.122+0000 7f6f11b91740 1 --2- 172.21.15.43:0/3983024901 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ea91a3a50 0x563ea91a1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.122+0000 7f6f11b91740 1 --2- 172.21.15.43:0/3983024901 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ea919ae70 0x563ea91a0f20 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.122+0000 7f6ef2000640 0 client.28891 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:47:12.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.122+0000 7f6f11b91740 1 --2- 172.21.15.43:0/3983024901 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ea919a550 0x563ea91a09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.122+0000 7f6f11b91740 1 -- 172.21.15.43:0/3983024901 >> 172.21.15.43:0/3983024901 conn(0x563ea9194c80 msgr2=0x563ea91969d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:12.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.123+0000 7f6f11b91740 1 -- 172.21.15.43:0/3983024901 shutdown_connections 2024-03-28T22:47:12.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.123+0000 7f6f11b91740 1 -- 172.21.15.43:0/3983024901 wait complete. 2024-03-28T22:47:12.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.123+0000 7f6f11b91740 20 client.28891 trim_cache size 0 max 16384 2024-03-28T22:47:12.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.138+0000 7f6f11b91740 1 Processor -- start 2024-03-28T22:47:12.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.138+0000 7f6f11b91740 1 -- start start 2024-03-28T22:47:12.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.138+0000 7f6f11b91740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ea919a550 0x563ea91a3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.138+0000 7f6f11b91740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ea919ae70 0x563ea91a1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.138+0000 7f6f11b91740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ea91a3a50 0x563ea91bbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.138+0000 7f6f11b91740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6f081b0e80 con 0x563ea91a3a50 2024-03-28T22:47:12.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.138+0000 7f6f11b91740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f6f081b29c0 con 0x563ea919ae70 2024-03-28T22:47:12.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.138+0000 7f6f11b91740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f6f081b1550 con 0x563ea919a550 2024-03-28T22:47:12.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.138+0000 7f6f07e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ea919a550 0x563ea91a3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.138+0000 7f6f07e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ea919a550 0x563ea91a3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:34992/0 (socket says 172.21.15.43:34992) 2024-03-28T22:47:12.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.138+0000 7f6f07e00640 1 -- 172.21.15.43:0/2272338882 learned_addr learned my addr 172.21.15.43:0/2272338882 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:12.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.138+0000 7f6f07400640 1 --2- 172.21.15.43:0/2272338882 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ea919ae70 0x563ea91a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.139+0000 7f6f0ce00640 1 --2- 172.21.15.43:0/2272338882 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ea91a3a50 0x563ea91bbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.139+0000 7f6f07e00640 1 -- 172.21.15.43:0/2272338882 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ea919ae70 msgr2=0x563ea91a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:12.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.139+0000 7f6f07e00640 1 --2- 172.21.15.43:0/2272338882 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ea919ae70 0x563ea91a1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.139+0000 7f6f07e00640 1 -- 172.21.15.43:0/2272338882 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ea91a3a50 msgr2=0x563ea91bbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:12.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.139+0000 7f6f07e00640 1 --2- 172.21.15.43:0/2272338882 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ea91a3a50 0x563ea91bbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.139+0000 7f6f07e00640 1 -- 172.21.15.43:0/2272338882 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ea918f1d0 con 0x563ea919a550 2024-03-28T22:47:12.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.139+0000 7f6f07e00640 1 --2- 172.21.15.43:0/2272338882 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ea919a550 0x563ea91a3110 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8003e40 tx=0x7f6ef8019480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f6a0254b3d848a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:12.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.139+0000 7f6efcc00640 1 -- 172.21.15.43:0/2272338882 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8033070 con 0x563ea919a550 2024-03-28T22:47:12.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.139+0000 7f6efcc00640 1 -- 172.21.15.43:0/2272338882 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ef802b070 con 0x563ea919a550 2024-03-28T22:47:12.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.139+0000 7f6efcc00640 1 -- 172.21.15.43:0/2272338882 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8003a90 con 0x563ea919a550 2024-03-28T22:47:12.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.140+0000 7f6f11b91740 1 -- 172.21.15.43:0/2272338882 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ea919a550 msgr2=0x563ea91a3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.140+0000 7f6f11b91740 1 --2- 172.21.15.43:0/2272338882 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ea919a550 0x563ea91a3110 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8003e40 tx=0x7f6ef8019480 comp rx=0 tx=0).stop 2024-03-28T22:47:12.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.140+0000 7f6f11b91740 1 -- 172.21.15.43:0/2272338882 shutdown_connections 2024-03-28T22:47:12.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.140+0000 7f6f11b91740 1 --2- 172.21.15.43:0/2272338882 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ea919ae70 0x563ea91a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.140+0000 7f6f11b91740 1 --2- 172.21.15.43:0/2272338882 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ea91a3a50 0x563ea91bbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.140+0000 7f6f11b91740 1 --2- 172.21.15.43:0/2272338882 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ea919a550 0x563ea91a3110 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.140+0000 7f6f11b91740 1 -- 172.21.15.43:0/2272338882 >> 172.21.15.43:0/2272338882 conn(0x563ea9194c80 msgr2=0x563ea9195eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:12.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.140+0000 7f6f11b91740 1 -- 172.21.15.43:0/2272338882 shutdown_connections 2024-03-28T22:47:12.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.140+0000 7f6f11b91740 1 -- 172.21.15.43:0/2272338882 wait complete. 2024-03-28T22:47:12.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.141+0000 7f6f11b91740 1 -- start start 2024-03-28T22:47:12.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.141+0000 7f6f11b91740 1 Processor -- start 2024-03-28T22:47:12.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.141+0000 7f6f11b91740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ea919a550 0x563ea91a7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.141+0000 7f6f11b91740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ea919ae70 0x563ea91a7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.141+0000 7f6f11b91740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ea91a3a50 0x563ea91a7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.141+0000 7f6f11b91740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6f081b22f0 con 0x563ea919ae70 2024-03-28T22:47:12.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.141+0000 7f6f11b91740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f6f081b1c20 con 0x563ea919a550 2024-03-28T22:47:12.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.141+0000 7f6f11b91740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f6f081aee80 con 0x563ea91a3a50 2024-03-28T22:47:12.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.141+0000 7f6f11b91740 10 client.0 _ll_register_callbacks cb 0x7f6ef400e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:47:12.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.142+0000 7f6f07e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ea919a550 0x563ea91a7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.142+0000 7f6f07400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ea919ae70 0x563ea91a7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.142+0000 7f6f0ce00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ea91a3a50 0x563ea91a7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.142+0000 7f6f07e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ea919a550 0x563ea91a7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47422/0 (socket says 172.21.15.43:47422) 2024-03-28T22:47:12.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.142+0000 7f6f07e00640 1 -- 172.21.15.43:0/1103502452 learned_addr learned my addr 172.21.15.43:0/1103502452 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:12.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.142+0000 7f6f07e00640 1 -- 172.21.15.43:0/1103502452 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ea91a3a50 msgr2=0x563ea91a7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:12.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.142+0000 7f6f07e00640 1 --2- 172.21.15.43:0/1103502452 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ea91a3a50 0x563ea91a7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.142+0000 7f6f07e00640 1 -- 172.21.15.43:0/1103502452 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ea919ae70 msgr2=0x563ea91a7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:12.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.142+0000 7f6f07e00640 1 --2- 172.21.15.43:0/1103502452 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ea919ae70 0x563ea91a7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.142+0000 7f6f07e00640 1 -- 172.21.15.43:0/1103502452 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ea92191f0 con 0x563ea919a550 2024-03-28T22:47:12.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.142+0000 7f6f07400640 1 --2- 172.21.15.43:0/1103502452 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ea919ae70 0x563ea91a7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:12.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.142+0000 7f6f07e00640 1 --2- 172.21.15.43:0/1103502452 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ea919a550 0x563ea91a7540 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8003630 tx=0x7f6ef8005a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0f57436416e3df6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:12.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.142+0000 7f6ef2000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:47:12.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.143+0000 7f6ef2000640 1 -- 172.21.15.43:0/1103502452 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8033070 con 0x563ea919a550 2024-03-28T22:47:12.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.143+0000 7f6ef2000640 1 -- 172.21.15.43:0/1103502452 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9627+0+0 (secure 0 0 0) 0x7f6ef8009e20 con 0x563ea919a550 2024-03-28T22:47:12.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.143+0000 7f6f11b91740 1 -- 172.21.15.43:0/1103502452 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ef802b070 con 0x563ea919a550 2024-03-28T22:47:12.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.143+0000 7f6ef2000640 1 -- 172.21.15.43:0/1103502452 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6ef8005340 con 0x563ea919a550 2024-03-28T22:47:12.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.143+0000 7f6f11b91740 10 client.28961 fetch_fsmap learned FSMap version 32 2024-03-28T22:47:12.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.143+0000 7f6f11b91740 1 -- 172.21.15.43:0/1103502452 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563ea921a4d0 con 0x563ea919a550 2024-03-28T22:47:12.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.143+0000 7f6ef2000640 1 -- 172.21.15.43:0/1103502452 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ef802b070 con 0x563ea919a550 2024-03-28T22:47:12.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.143+0000 7f6ef2000640 1 -- 172.21.15.43:0/1103502452 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6ef8005020 con 0x563ea919a550 2024-03-28T22:47:12.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.143+0000 7f6f11b91740 10 client.28961 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:47:12.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.143+0000 7f6f11b91740 10 client.28961 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:47:12.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.143+0000 7f6f11b91740 20 client.28961 populate_metadata read hostname 'smithi043' 2024-03-28T22:47:12.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.143+0000 7f6f11b91740 1 --2- 172.21.15.43:0/1103502452 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x563ea921a9f0 0x563ea921cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.143+0000 7f6f11b91740 4 client.28961 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:47:12.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.143+0000 7f6f11b91740 1 -- 172.21.15.43:0/1103502452 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563ea910b590 con 0x563ea921a9f0 2024-03-28T22:47:12.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.144+0000 7f6f07400640 1 --2- 172.21.15.43:0/1103502452 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x563ea921a9f0 0x563ea921cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.144+0000 7f6f07400640 1 --2- 172.21.15.43:0/1103502452 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x563ea921a9f0 0x563ea921cde0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9f9941ad8eeb758d server_cookie=3da3033bbbc765de in_seq=0 out_seq=0 2024-03-28T22:47:12.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.144+0000 7f6ef2000640 10 client.28961 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:47:12.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.144+0000 7f6ef2000640 1 -- 172.21.15.43:0/1103502452 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563ea910b590 con 0x563ea921a9f0 2024-03-28T22:47:12.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.144+0000 7f6ef2000640 10 client.28961 handle_command_reply: tid=0 2024-03-28T22:47:12.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.144+0000 7f6f11b91740 1 client.28961 shutdown 2024-03-28T22:47:12.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.144+0000 7f6f11b91740 1 -- 172.21.15.43:0/1103502452 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ea919a550 msgr2=0x563ea91a7540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.144+0000 7f6f11b91740 1 --2- 172.21.15.43:0/1103502452 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ea919a550 0x563ea91a7540 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8003630 tx=0x7f6ef8005a70 comp rx=0 tx=0).stop 2024-03-28T22:47:12.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.144+0000 7f6f11b91740 1 -- 172.21.15.43:0/1103502452 shutdown_connections 2024-03-28T22:47:12.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.144+0000 7f6f11b91740 1 --2- 172.21.15.43:0/1103502452 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x563ea921a9f0 0x563ea921cde0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.144+0000 7f6f11b91740 1 --2- 172.21.15.43:0/1103502452 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x563ea91a3a50 0x563ea91a7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.144+0000 7f6f11b91740 1 --2- 172.21.15.43:0/1103502452 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x563ea919ae70 0x563ea91a7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.144+0000 7f6f11b91740 1 --2- 172.21.15.43:0/1103502452 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ea919a550 0x563ea91a7540 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.145+0000 7f6f11b91740 1 -- 172.21.15.43:0/1103502452 >> 172.21.15.43:0/1103502452 conn(0x563ea9194c80 msgr2=0x563ea9196510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:12.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.145+0000 7f6ef2000640 0 client.28961 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:47:12.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.145+0000 7f6f11b91740 1 -- 172.21.15.43:0/1103502452 shutdown_connections 2024-03-28T22:47:12.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.145+0000 7f6f11b91740 1 -- 172.21.15.43:0/1103502452 wait complete. 2024-03-28T22:47:12.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.145+0000 7f6f11b91740 20 client.28961 trim_cache size 0 max 16384 2024-03-28T22:47:12.154 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:47:12.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.145+0000 7f6f0d800640 1 -- 172.21.15.43:0/3062832370 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6eec0a1ef0 msgr2=0x7f6eec0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.145+0000 7f6f0d800640 1 --2- 172.21.15.43:0/3062832370 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6eec0a1ef0 0x7f6eec0a43b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6f00005e30 tx=0x7f6f00005d80 comp rx=0 tx=0).stop 2024-03-28T22:47:12.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.145+0000 7f6f0d800640 1 -- 172.21.15.43:0/3062832370 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f08104190 msgr2=0x7f6f0819a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.145+0000 7f6f0d800640 1 --2- 172.21.15.43:0/3062832370 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f08104190 0x7f6f0819a610 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f6ef400e9f0 tx=0x7f6ef400edf0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.145+0000 7f6f0d800640 1 -- 172.21.15.43:0/3062832370 shutdown_connections 2024-03-28T22:47:12.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.145+0000 7f6f0d800640 1 --2- 172.21.15.43:0/3062832370 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6eec0a1ef0 0x7f6eec0a43b0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.145+0000 7f6f0d800640 1 --2- 172.21.15.43:0/3062832370 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6f08104190 0x7f6f0819a610 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.145+0000 7f6f0d800640 1 --2- 172.21.15.43:0/3062832370 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6f081097e0 0x7f6f0819e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.145+0000 7f6f0d800640 1 --2- 172.21.15.43:0/3062832370 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6f081037d0 0x7f6f0819a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.145+0000 7f6f0d800640 1 -- 172.21.15.43:0/3062832370 >> 172.21.15.43:0/3062832370 conn(0x7f6f080ff280 msgr2=0x7f6f080ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:12.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.146+0000 7f6f0d800640 1 -- 172.21.15.43:0/3062832370 shutdown_connections 2024-03-28T22:47:12.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.146+0000 7f6f0d800640 1 -- 172.21.15.43:0/3062832370 wait complete. 2024-03-28T22:47:12.160 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T22:47:12.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.375+0000 7fdfb9800640 1 -- 172.21.15.43:0/594673103 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfb41076d0 msgr2=0x7fdfb4107ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.375+0000 7fdfb9800640 1 --2- 172.21.15.43:0/594673103 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfb41076d0 0x7fdfb4107ab0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4009930 tx=0x7fdfa402bc30 comp rx=0 tx=0).stop 2024-03-28T22:47:12.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.375+0000 7fdfb9800640 1 -- 172.21.15.43:0/594673103 shutdown_connections 2024-03-28T22:47:12.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.375+0000 7fdfb9800640 1 --2- 172.21.15.43:0/594673103 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdfb40692f0 0x7fdfb410e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.375+0000 7fdfb9800640 1 --2- 172.21.15.43:0/594673103 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdfb4068930 0x7fdfb4068db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.375+0000 7fdfb9800640 1 --2- 172.21.15.43:0/594673103 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfb41076d0 0x7fdfb4107ab0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.375+0000 7fdfb9800640 1 -- 172.21.15.43:0/594673103 >> 172.21.15.43:0/594673103 conn(0x7fdfb4077750 msgr2=0x7fdfb4077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:12.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.375+0000 7fdfb9800640 1 -- 172.21.15.43:0/594673103 shutdown_connections 2024-03-28T22:47:12.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.376+0000 7fdfb9800640 1 -- 172.21.15.43:0/594673103 wait complete. 2024-03-28T22:47:12.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.377+0000 7fdfb9800640 1 Processor -- start 2024-03-28T22:47:12.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.377+0000 7fdfb9800640 1 -- start start 2024-03-28T22:47:12.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.377+0000 7fdfb9800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfb4068930 0x7fdfb419a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.377+0000 7fdfb9800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdfb40692f0 0x7fdfb419a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.377+0000 7fdfb9800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdfb41076d0 0x7fdfb419eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.377+0000 7fdfb9800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb4111af0 con 0x7fdfb40692f0 2024-03-28T22:47:12.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.377+0000 7fdfb9800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb4111970 con 0x7fdfb41076d0 2024-03-28T22:47:12.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.377+0000 7fdfb9800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb4111c70 con 0x7fdfb4068930 2024-03-28T22:47:12.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.378+0000 7fdfb2a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfb4068930 0x7fdfb419a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.378+0000 7fdfb2a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfb4068930 0x7fdfb419a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35028/0 (socket says 172.21.15.43:35028) 2024-03-28T22:47:12.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.378+0000 7fdfb2a00640 1 -- 172.21.15.43:0/2275051159 learned_addr learned my addr 172.21.15.43:0/2275051159 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:12.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.378+0000 7fdfb2000640 1 --2- 172.21.15.43:0/2275051159 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdfb40692f0 0x7fdfb419a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.378+0000 7fdfb3400640 1 --2- 172.21.15.43:0/2275051159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdfb41076d0 0x7fdfb419eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.378+0000 7fdfb2a00640 1 -- 172.21.15.43:0/2275051159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdfb41076d0 msgr2=0x7fdfb419eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.378+0000 7fdfb2a00640 1 --2- 172.21.15.43:0/2275051159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdfb41076d0 0x7fdfb419eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.378+0000 7fdfb2a00640 1 -- 172.21.15.43:0/2275051159 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdfb40692f0 msgr2=0x7fdfb419a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.378+0000 7fdfb2a00640 1 --2- 172.21.15.43:0/2275051159 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdfb40692f0 0x7fdfb419a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.378+0000 7fdfb2a00640 1 -- 172.21.15.43:0/2275051159 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfb419f220 con 0x7fdfb4068930 2024-03-28T22:47:12.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.378+0000 7fdfb2000640 1 --2- 172.21.15.43:0/2275051159 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdfb40692f0 0x7fdfb419a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:12.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.379+0000 7fdfb2a00640 1 --2- 172.21.15.43:0/2275051159 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfb4068930 0x7fdfb419a230 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4009900 tx=0x7fdfa4004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:12.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.379+0000 7fdfab400640 1 -- 172.21.15.43:0/2275051159 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa4043020 con 0x7fdfb4068930 2024-03-28T22:47:12.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.379+0000 7fdfab400640 1 -- 172.21.15.43:0/2275051159 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfa40054e0 con 0x7fdfb4068930 2024-03-28T22:47:12.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.379+0000 7fdfab400640 1 -- 172.21.15.43:0/2275051159 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa4003bb0 con 0x7fdfb4068930 2024-03-28T22:47:12.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.379+0000 7fdfb9800640 1 -- 172.21.15.43:0/2275051159 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfb419f4b0 con 0x7fdfb4068930 2024-03-28T22:47:12.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.379+0000 7fdfb9800640 1 -- 172.21.15.43:0/2275051159 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfb41a6da0 con 0x7fdfb4068930 2024-03-28T22:47:12.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.382+0000 7fdfab400640 1 -- 172.21.15.43:0/2275051159 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fdfa4046020 con 0x7fdfb4068930 2024-03-28T22:47:12.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.382+0000 7fdfbc481740 1 Processor -- start 2024-03-28T22:47:12.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.382+0000 7fdfbc481740 1 -- start start 2024-03-28T22:47:12.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.383+0000 7fdfbc481740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b636ebe380 0x55b636ebe760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.383+0000 7fdfbc481740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b636ebeca0 0x55b636ec7340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.383+0000 7fdfbc481740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b636ec7880 0x55b636ec9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.383+0000 7fdfbc481740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b636e22050 con 0x55b636ebe380 2024-03-28T22:47:12.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.383+0000 7fdfbc481740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55b636dcace0 con 0x55b636ebeca0 2024-03-28T22:47:12.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.383+0000 7fdfbc481740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55b636e1e0f0 con 0x55b636ec7880 2024-03-28T22:47:12.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.383+0000 7fdfb3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b636ebe380 0x55b636ebe760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.383+0000 7fdfb2a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b636ebeca0 0x55b636ec7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.383+0000 7fdfb2000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b636ec7880 0x55b636ec9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.383+0000 7fdfb3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b636ebe380 0x55b636ebe760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33946/0 (socket says 172.21.15.43:33946) 2024-03-28T22:47:12.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.383+0000 7fdfb2a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b636ebeca0 0x55b636ec7340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47450/0 (socket says 172.21.15.43:47450) 2024-03-28T22:47:12.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.383+0000 7fdfb3400640 1 -- 172.21.15.43:0/843830579 learned_addr learned my addr 172.21.15.43:0/843830579 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:12.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.383+0000 7fdfb3400640 1 -- 172.21.15.43:0/843830579 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b636ec7880 msgr2=0x55b636ec9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:12.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.383+0000 7fdfb3400640 1 --2- 172.21.15.43:0/843830579 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b636ec7880 0x55b636ec9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.383+0000 7fdfb3400640 1 -- 172.21.15.43:0/843830579 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b636ebeca0 msgr2=0x55b636ec7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:12.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.383+0000 7fdfb3400640 1 --2- 172.21.15.43:0/843830579 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b636ebeca0 0x55b636ec7340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.383+0000 7fdfb3400640 1 -- 172.21.15.43:0/843830579 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b636b363f0 con 0x55b636ebe380 2024-03-28T22:47:12.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.383+0000 7fdfb2a00640 1 --2- 172.21.15.43:0/843830579 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b636ebeca0 0x55b636ec7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:12.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.384+0000 7fdfab400640 1 --2- 172.21.15.43:0/2275051159 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fdf940a1ef0 0x7fdf940a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.384+0000 7fdfb3400640 1 --2- 172.21.15.43:0/843830579 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b636ebe380 0x55b636ebe760 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fdfac009c20 tx=0x7fdfac011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2baa23f5af95b58d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:12.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.384+0000 7fdfab400640 1 -- 172.21.15.43:0/2275051159 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9627+0+0 (secure 0 0 0) 0x7fdfa40f0560 con 0x7fdfb4068930 2024-03-28T22:47:12.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.384+0000 7fdfb2a00640 1 --2- 172.21.15.43:0/2275051159 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fdf940a1ef0 0x7fdf940a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.384+0000 7fdfa3e00640 1 -- 172.21.15.43:0/843830579 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfac01e070 con 0x55b636ebe380 2024-03-28T22:47:12.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.384+0000 7fdfa3e00640 1 -- 172.21.15.43:0/843830579 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfac019b10 con 0x55b636ebe380 2024-03-28T22:47:12.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.384+0000 7fdfa3e00640 1 -- 172.21.15.43:0/843830579 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfac00ea80 con 0x55b636ebe380 2024-03-28T22:47:12.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.385+0000 7fdfbc481740 1 -- 172.21.15.43:0/843830579 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b636ebe380 msgr2=0x55b636ebe760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.385+0000 7fdfb2a00640 1 --2- 172.21.15.43:0/2275051159 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fdf940a1ef0 0x7fdf940a43b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x55b636ebde50 tx=0x7fdfa40f3f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:12.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.385+0000 7fdfbc481740 1 --2- 172.21.15.43:0/843830579 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b636ebe380 0x55b636ebe760 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fdfac009c20 tx=0x7fdfac011810 comp rx=0 tx=0).stop 2024-03-28T22:47:12.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.385+0000 7fdfbc481740 1 -- 172.21.15.43:0/843830579 shutdown_connections 2024-03-28T22:47:12.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.385+0000 7fdfbc481740 1 --2- 172.21.15.43:0/843830579 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b636ebeca0 0x55b636ec7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.385+0000 7fdfbc481740 1 --2- 172.21.15.43:0/843830579 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b636ec7880 0x55b636ec9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.385+0000 7fdfbc481740 1 --2- 172.21.15.43:0/843830579 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b636ebe380 0x55b636ebe760 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.385+0000 7fdfbc481740 1 -- 172.21.15.43:0/843830579 >> 172.21.15.43:0/843830579 conn(0x55b636eb8ab0 msgr2=0x55b636eb8e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:12.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.385+0000 7fdfbc481740 1 -- 172.21.15.43:0/843830579 shutdown_connections 2024-03-28T22:47:12.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.385+0000 7fdfbc481740 1 -- 172.21.15.43:0/843830579 wait complete. 2024-03-28T22:47:12.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.386+0000 7fdfbc481740 1 -- start start 2024-03-28T22:47:12.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.386+0000 7fdfbc481740 1 Processor -- start 2024-03-28T22:47:12.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.386+0000 7fdfbc481740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b636ebe380 0x55b636ec48f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.386+0000 7fdfbc481740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b636ebeca0 0x55b636ec4e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.386+0000 7fdfbc481740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b636ec7880 0x55b636ec5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.386+0000 7fdfbc481740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b636eca670 con 0x55b636ebeca0 2024-03-28T22:47:12.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.386+0000 7fdfbc481740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55b636dc02f0 con 0x55b636ec7880 2024-03-28T22:47:12.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.386+0000 7fdfbc481740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55b636eca7f0 con 0x55b636ebe380 2024-03-28T22:47:12.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.386+0000 7fdfbc481740 10 client.0 _ll_register_callbacks cb 0x55b636b35be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:47:12.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.386+0000 7fdfb2000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b636ebeca0 0x55b636ec4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.386+0000 7fdfb3400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b636ebe380 0x55b636ec48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.387+0000 7fdfb2000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b636ebeca0 0x55b636ec4e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33956/0 (socket says 172.21.15.43:33956) 2024-03-28T22:47:12.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.387+0000 7fdfb2000640 1 -- 172.21.15.43:0/2125331294 learned_addr learned my addr 172.21.15.43:0/2125331294 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:12.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.387+0000 7fdfb3400640 1 --2- 172.21.15.43:0/2125331294 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b636ec7880 0x55b636ec5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.387+0000 7fdfb2000640 1 -- 172.21.15.43:0/2125331294 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b636ebe380 msgr2=0x55b636ec48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:12.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.387+0000 7fdfb2000640 1 --2- 172.21.15.43:0/2125331294 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b636ebe380 0x55b636ec48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.387+0000 7fdfb2000640 1 -- 172.21.15.43:0/2125331294 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b636ec7880 msgr2=0x55b636ec5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:12.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.387+0000 7fdfb2000640 1 --2- 172.21.15.43:0/2125331294 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b636ec7880 0x55b636ec5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.387+0000 7fdfb2000640 1 -- 172.21.15.43:0/2125331294 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b636c007d0 con 0x55b636ebeca0 2024-03-28T22:47:12.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.387+0000 7fdfb3400640 1 --2- 172.21.15.43:0/2125331294 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b636ec7880 0x55b636ec5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:47:12.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.387+0000 7fdfb2000640 1 --2- 172.21.15.43:0/2125331294 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b636ebeca0 0x55b636ec4e30 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c00fe20 tx=0x7fdf9c012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1477c5c834d51c1b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:12.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.387+0000 7fdfa0c00640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:47:12.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.387+0000 7fdfa0c00640 1 -- 172.21.15.43:0/2125331294 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c003970 con 0x55b636ebeca0 2024-03-28T22:47:12.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.387+0000 7fdfa0c00640 1 -- 172.21.15.43:0/2125331294 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9627+0+0 (secure 0 0 0) 0x7fdf9c003b10 con 0x55b636ebeca0 2024-03-28T22:47:12.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.387+0000 7fdfbc481740 1 -- 172.21.15.43:0/2125331294 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfac019b10 con 0x55b636ebeca0 2024-03-28T22:47:12.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.388+0000 7fdfa0c00640 1 -- 172.21.15.43:0/2125331294 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdf9c025050 con 0x55b636ebeca0 2024-03-28T22:47:12.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.388+0000 7fdfbc481740 10 client.18702 fetch_fsmap learned FSMap version 32 2024-03-28T22:47:12.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.388+0000 7fdfbc481740 1 -- 172.21.15.43:0/2125331294 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b636ed69c0 con 0x55b636ebeca0 2024-03-28T22:47:12.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.388+0000 7fdfa0c00640 1 -- 172.21.15.43:0/2125331294 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfac019b10 con 0x55b636ebeca0 2024-03-28T22:47:12.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.388+0000 7fdfa0c00640 1 -- 172.21.15.43:0/2125331294 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fdf9c0105c0 con 0x55b636ebeca0 2024-03-28T22:47:12.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.388+0000 7fdfbc481740 10 client.18702 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:47:12.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.388+0000 7fdfbc481740 10 client.18702 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:47:12.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.388+0000 7fdfbc481740 20 client.18702 populate_metadata read hostname 'smithi043' 2024-03-28T22:47:12.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.388+0000 7fdfbc481740 1 --2- 172.21.15.43:0/2125331294 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55b636ed7ed0 0x55b636eda2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.389+0000 7fdfbc481740 4 client.18702 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:47:12.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.389+0000 7fdfbc481740 1 -- 172.21.15.43:0/2125331294 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b636dcb680 con 0x55b636ed7ed0 2024-03-28T22:47:12.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.389+0000 7fdfb3400640 1 --2- 172.21.15.43:0/2125331294 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55b636ed7ed0 0x55b636eda2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.389+0000 7fdfb3400640 1 --2- 172.21.15.43:0/2125331294 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55b636ed7ed0 0x55b636eda2f0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2effd2c9fed0ffe6 server_cookie=dd77e1bf5d2fb384 in_seq=0 out_seq=0 2024-03-28T22:47:12.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.389+0000 7fdfa0c00640 10 client.18702 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:47:12.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.390+0000 7fdfa0c00640 1 -- 172.21.15.43:0/2125331294 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b636dcb680 con 0x55b636ed7ed0 2024-03-28T22:47:12.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.390+0000 7fdfa0c00640 10 client.18702 handle_command_reply: tid=0 2024-03-28T22:47:12.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.390+0000 7fdfbc481740 1 client.18702 shutdown 2024-03-28T22:47:12.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.390+0000 7fdfbc481740 1 -- 172.21.15.43:0/2125331294 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b636ebeca0 msgr2=0x55b636ec4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.390+0000 7fdfbc481740 1 --2- 172.21.15.43:0/2125331294 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b636ebeca0 0x55b636ec4e30 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c00fe20 tx=0x7fdf9c012820 comp rx=0 tx=0).stop 2024-03-28T22:47:12.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.390+0000 7fdfbc481740 1 -- 172.21.15.43:0/2125331294 shutdown_connections 2024-03-28T22:47:12.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.390+0000 7fdfbc481740 1 --2- 172.21.15.43:0/2125331294 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55b636ed7ed0 0x55b636eda2f0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.390+0000 7fdfbc481740 1 --2- 172.21.15.43:0/2125331294 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b636ec7880 0x55b636ec5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.390+0000 7fdfbc481740 1 --2- 172.21.15.43:0/2125331294 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b636ebeca0 0x55b636ec4e30 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.390+0000 7fdfa0c00640 0 client.18702 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:47:12.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.390+0000 7fdfbc481740 1 --2- 172.21.15.43:0/2125331294 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b636ebe380 0x55b636ec48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.390+0000 7fdfbc481740 1 -- 172.21.15.43:0/2125331294 >> 172.21.15.43:0/2125331294 conn(0x55b636eb8ab0 msgr2=0x55b636eba800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:12.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.391+0000 7fdfbc481740 1 -- 172.21.15.43:0/2125331294 shutdown_connections 2024-03-28T22:47:12.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.391+0000 7fdfbc481740 1 -- 172.21.15.43:0/2125331294 wait complete. 2024-03-28T22:47:12.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.391+0000 7fdfbc481740 20 client.18702 trim_cache size 0 max 16384 2024-03-28T22:47:12.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.405+0000 7fdfbc481740 1 Processor -- start 2024-03-28T22:47:12.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.406+0000 7fdfbc481740 1 -- start start 2024-03-28T22:47:12.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.406+0000 7fdfbc481740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b636ebe380 0x55b636ec7020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.406+0000 7fdfbc481740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b636ebeca0 0x55b636ec3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.406+0000 7fdfbc481740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b636ec7880 0x55b636ed3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.406+0000 7fdfbc481740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb41b1070 con 0x55b636ebe380 2024-03-28T22:47:12.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.406+0000 7fdfbc481740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb41b2bb0 con 0x55b636ec7880 2024-03-28T22:47:12.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.406+0000 7fdfbc481740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb41b1740 con 0x55b636ebeca0 2024-03-28T22:47:12.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.406+0000 7fdfb3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b636ebe380 0x55b636ec7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.406+0000 7fdfb2000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b636ebeca0 0x55b636ec3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.406+0000 7fdfb3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b636ebe380 0x55b636ec7020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:33968/0 (socket says 172.21.15.43:33968) 2024-03-28T22:47:12.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.406+0000 7fdfb3400640 1 -- 172.21.15.43:0/3570935121 learned_addr learned my addr 172.21.15.43:0/3570935121 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:12.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.406+0000 7fdfb3400640 1 --2- 172.21.15.43:0/3570935121 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b636ec7880 0x55b636ed3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.407+0000 7fdfb2000640 1 -- 172.21.15.43:0/3570935121 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b636ec7880 msgr2=0x55b636ed3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:12.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.407+0000 7fdfb2000640 1 --2- 172.21.15.43:0/3570935121 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b636ec7880 0x55b636ed3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.407+0000 7fdfb2000640 1 -- 172.21.15.43:0/3570935121 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b636ebe380 msgr2=0x55b636ec7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:12.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.407+0000 7fdfb2000640 1 --2- 172.21.15.43:0/3570935121 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b636ebe380 0x55b636ec7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.407+0000 7fdfb2000640 1 -- 172.21.15.43:0/3570935121 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b636ec2e40 con 0x55b636ebeca0 2024-03-28T22:47:12.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.407+0000 7fdfb3400640 1 --2- 172.21.15.43:0/3570935121 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b636ebe380 0x55b636ec7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:12.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.407+0000 7fdfb2000640 1 --2- 172.21.15.43:0/3570935121 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b636ebeca0 0x55b636ec3a50 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c010a10 tx=0x7fdf9c009990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f30fae2148bd537 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:12.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.407+0000 7fdfa3400640 1 -- 172.21.15.43:0/3570935121 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c010430 con 0x55b636ebeca0 2024-03-28T22:47:12.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.407+0000 7fdfa3400640 1 -- 172.21.15.43:0/3570935121 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf9c003e50 con 0x55b636ebeca0 2024-03-28T22:47:12.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.407+0000 7fdfa3400640 1 -- 172.21.15.43:0/3570935121 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c0055a0 con 0x55b636ebeca0 2024-03-28T22:47:12.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.408+0000 7fdfbc481740 1 -- 172.21.15.43:0/3570935121 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b636ebeca0 msgr2=0x55b636ec3a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.408+0000 7fdfbc481740 1 --2- 172.21.15.43:0/3570935121 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b636ebeca0 0x55b636ec3a50 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c010a10 tx=0x7fdf9c009990 comp rx=0 tx=0).stop 2024-03-28T22:47:12.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.408+0000 7fdfbc481740 1 -- 172.21.15.43:0/3570935121 shutdown_connections 2024-03-28T22:47:12.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.408+0000 7fdfbc481740 1 --2- 172.21.15.43:0/3570935121 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b636ec7880 0x55b636ed3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.408+0000 7fdfbc481740 1 --2- 172.21.15.43:0/3570935121 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b636ebeca0 0x55b636ec3a50 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.408+0000 7fdfbc481740 1 --2- 172.21.15.43:0/3570935121 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b636ebe380 0x55b636ec7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.408+0000 7fdfbc481740 1 -- 172.21.15.43:0/3570935121 >> 172.21.15.43:0/3570935121 conn(0x55b636eb8ab0 msgr2=0x55b636eb72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:12.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.408+0000 7fdfbc481740 1 -- 172.21.15.43:0/3570935121 shutdown_connections 2024-03-28T22:47:12.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.408+0000 7fdfbc481740 1 -- 172.21.15.43:0/3570935121 wait complete. 2024-03-28T22:47:12.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.408+0000 7fdfbc481740 1 -- start start 2024-03-28T22:47:12.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.409+0000 7fdfbc481740 1 Processor -- start 2024-03-28T22:47:12.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.409+0000 7fdfbc481740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b636ebe380 0x55b636f364a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.409+0000 7fdfbc481740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b636ebeca0 0x55b636f369e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.409+0000 7fdfbc481740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b636ec7880 0x55b636ecbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.409+0000 7fdfbc481740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb41b24e0 con 0x55b636ec7880 2024-03-28T22:47:12.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.409+0000 7fdfbc481740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb41b1e10 con 0x55b636ebe380 2024-03-28T22:47:12.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.409+0000 7fdfbc481740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb41af070 con 0x55b636ebeca0 2024-03-28T22:47:12.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.409+0000 7fdfbc481740 10 client.0 _ll_register_callbacks cb 0x7fdf9c00f4a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:47:12.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.409+0000 7fdfb2000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b636ebeca0 0x55b636f369e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.409+0000 7fdfb2000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b636ebeca0 0x55b636f369e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35066/0 (socket says 172.21.15.43:35066) 2024-03-28T22:47:12.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.409+0000 7fdfb2000640 1 -- 172.21.15.43:0/3854644051 learned_addr learned my addr 172.21.15.43:0/3854644051 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:12.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.409+0000 7fdfb3400640 1 --2- 172.21.15.43:0/3854644051 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b636ebe380 0x55b636f364a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.410+0000 7fdfb3400640 1 --2- 172.21.15.43:0/3854644051 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b636ec7880 0x55b636ecbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.410+0000 7fdfb2000640 1 -- 172.21.15.43:0/3854644051 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b636ebe380 msgr2=0x55b636f364a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:12.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.410+0000 7fdfb2000640 1 --2- 172.21.15.43:0/3854644051 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b636ebe380 0x55b636f364a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.410+0000 7fdfb2000640 1 -- 172.21.15.43:0/3854644051 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b636ec7880 msgr2=0x55b636ecbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:12.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.410+0000 7fdfb2000640 1 --2- 172.21.15.43:0/3854644051 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b636ec7880 0x55b636ecbbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.410+0000 7fdfb2000640 1 -- 172.21.15.43:0/3854644051 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b636ecc2d0 con 0x55b636ebeca0 2024-03-28T22:47:12.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.410+0000 7fdfb3400640 1 --2- 172.21.15.43:0/3854644051 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b636ec7880 0x55b636ecbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:47:12.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.410+0000 7fdfb2000640 1 --2- 172.21.15.43:0/3854644051 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b636ebeca0 0x55b636f369e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c010990 tx=0x7fdf9c003c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81d676564691dd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:12.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.410+0000 7fdfa0c00640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:47:12.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.410+0000 7fdfa0c00640 1 -- 172.21.15.43:0/3854644051 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c010640 con 0x55b636ebeca0 2024-03-28T22:47:12.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.410+0000 7fdfbc481740 1 -- 172.21.15.43:0/3854644051 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf9c003e50 con 0x55b636ebeca0 2024-03-28T22:47:12.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.410+0000 7fdfa0c00640 1 -- 172.21.15.43:0/3854644051 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9627+0+0 (secure 0 0 0) 0x7fdf9c022040 con 0x55b636ebeca0 2024-03-28T22:47:12.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.410+0000 7fdfa0c00640 1 -- 172.21.15.43:0/3854644051 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdf9c005110 con 0x55b636ebeca0 2024-03-28T22:47:12.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.411+0000 7fdfbc481740 10 client.28915 fetch_fsmap learned FSMap version 32 2024-03-28T22:47:12.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.411+0000 7fdfbc481740 1 -- 172.21.15.43:0/3854644051 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b636f3ed70 con 0x55b636ebeca0 2024-03-28T22:47:12.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.411+0000 7fdfa0c00640 1 -- 172.21.15.43:0/3854644051 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf9c003e50 con 0x55b636ebeca0 2024-03-28T22:47:12.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.411+0000 7fdfa0c00640 1 -- 172.21.15.43:0/3854644051 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fdf9c0317b0 con 0x55b636ebeca0 2024-03-28T22:47:12.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.411+0000 7fdfbc481740 10 client.28915 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:47:12.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.411+0000 7fdfbc481740 10 client.28915 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:47:12.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.411+0000 7fdfbc481740 20 client.28915 populate_metadata read hostname 'smithi043' 2024-03-28T22:47:12.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.411+0000 7fdfbc481740 1 --2- 172.21.15.43:0/3854644051 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55b636f3f380 0x55b636f41770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.411+0000 7fdfbc481740 4 client.28915 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:47:12.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.411+0000 7fdfbc481740 1 -- 172.21.15.43:0/3854644051 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b636e2f590 con 0x55b636f3f380 2024-03-28T22:47:12.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.411+0000 7fdfb3400640 1 --2- 172.21.15.43:0/3854644051 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55b636f3f380 0x55b636f41770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.412+0000 7fdfb3400640 1 --2- 172.21.15.43:0/3854644051 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55b636f3f380 0x55b636f41770 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=591618108eb4c83f server_cookie=11d9b5a790e38cf in_seq=0 out_seq=0 2024-03-28T22:47:12.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.412+0000 7fdfa0c00640 10 client.28915 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:47:12.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.412+0000 7fdfa0c00640 1 -- 172.21.15.43:0/3854644051 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b636e2f590 con 0x55b636f3f380 2024-03-28T22:47:12.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.412+0000 7fdfa0c00640 10 client.28915 handle_command_reply: tid=0 2024-03-28T22:47:12.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.412+0000 7fdfbc481740 1 client.28915 shutdown 2024-03-28T22:47:12.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.412+0000 7fdfbc481740 1 -- 172.21.15.43:0/3854644051 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b636ebeca0 msgr2=0x55b636f369e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.412+0000 7fdfbc481740 1 --2- 172.21.15.43:0/3854644051 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b636ebeca0 0x55b636f369e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c010990 tx=0x7fdf9c003c20 comp rx=0 tx=0).stop 2024-03-28T22:47:12.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.412+0000 7fdfbc481740 1 -- 172.21.15.43:0/3854644051 shutdown_connections 2024-03-28T22:47:12.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.412+0000 7fdfbc481740 1 --2- 172.21.15.43:0/3854644051 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x55b636f3f380 0x55b636f41770 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.412+0000 7fdfbc481740 1 --2- 172.21.15.43:0/3854644051 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b636ec7880 0x55b636ecbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.412+0000 7fdfbc481740 1 --2- 172.21.15.43:0/3854644051 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b636ebeca0 0x55b636f369e0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.412+0000 7fdfa0c00640 0 client.28915 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:47:12.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.412+0000 7fdfbc481740 1 --2- 172.21.15.43:0/3854644051 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b636ebe380 0x55b636f364a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.412+0000 7fdfbc481740 1 -- 172.21.15.43:0/3854644051 >> 172.21.15.43:0/3854644051 conn(0x55b636eb8ab0 msgr2=0x55b636eb7b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:12.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.412+0000 7fdfbc481740 1 -- 172.21.15.43:0/3854644051 shutdown_connections 2024-03-28T22:47:12.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.413+0000 7fdfbc481740 1 -- 172.21.15.43:0/3854644051 wait complete. 2024-03-28T22:47:12.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.413+0000 7fdfbc481740 20 client.28915 trim_cache size 0 max 16384 2024-03-28T22:47:12.418 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:47:12.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.413+0000 7fdfb9800640 1 -- 172.21.15.43:0/2275051159 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fdf940a1ef0 msgr2=0x7fdf940a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.413+0000 7fdfb9800640 1 --2- 172.21.15.43:0/2275051159 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fdf940a1ef0 0x7fdf940a43b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x55b636ebde50 tx=0x7fdfa40f3f90 comp rx=0 tx=0).stop 2024-03-28T22:47:12.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.413+0000 7fdfb9800640 1 -- 172.21.15.43:0/2275051159 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfb4068930 msgr2=0x7fdfb419a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.413+0000 7fdfb9800640 1 --2- 172.21.15.43:0/2275051159 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfb4068930 0x7fdfb419a230 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4009900 tx=0x7fdfa4004360 comp rx=0 tx=0).stop 2024-03-28T22:47:12.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.413+0000 7fdfb9800640 1 -- 172.21.15.43:0/2275051159 shutdown_connections 2024-03-28T22:47:12.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.413+0000 7fdfb9800640 1 --2- 172.21.15.43:0/2275051159 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fdf940a1ef0 0x7fdf940a43b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.413+0000 7fdfb9800640 1 --2- 172.21.15.43:0/2275051159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdfb41076d0 0x7fdfb419eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.413+0000 7fdfb9800640 1 --2- 172.21.15.43:0/2275051159 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdfb40692f0 0x7fdfb419a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.413+0000 7fdfb9800640 1 --2- 172.21.15.43:0/2275051159 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdfb4068930 0x7fdfb419a230 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.413+0000 7fdfb9800640 1 -- 172.21.15.43:0/2275051159 >> 172.21.15.43:0/2275051159 conn(0x7fdfb4077750 msgr2=0x7fdfb410ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:12.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.413+0000 7fdfb9800640 1 -- 172.21.15.43:0/2275051159 shutdown_connections 2024-03-28T22:47:12.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.413+0000 7fdfb9800640 1 -- 172.21.15.43:0/2275051159 wait complete. 2024-03-28T22:47:12.425 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:12 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2992193547' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:47:12.429 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:47:12.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:12 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2992193547' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:47:12.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:12 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2992193547' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:47:12.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.659+0000 7f4e12e00640 1 -- 172.21.15.43:0/3530246578 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4e0c100390 msgr2=0x7f4e0c100770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.659+0000 7f4e12e00640 1 --2- 172.21.15.43:0/3530246578 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4e0c100390 0x7f4e0c100770 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4e00009900 tx=0x7f4e0002bc30 comp rx=0 tx=0).stop 2024-03-28T22:47:12.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.659+0000 7f4e12e00640 1 -- 172.21.15.43:0/3530246578 shutdown_connections 2024-03-28T22:47:12.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.659+0000 7f4e12e00640 1 --2- 172.21.15.43:0/3530246578 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4e0c10df40 0x7f4e0c110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.659+0000 7f4e12e00640 1 --2- 172.21.15.43:0/3530246578 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e0c100d40 0x7f4e0c10d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.659+0000 7f4e12e00640 1 --2- 172.21.15.43:0/3530246578 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4e0c100390 0x7f4e0c100770 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.659+0000 7f4e12e00640 1 -- 172.21.15.43:0/3530246578 >> 172.21.15.43:0/3530246578 conn(0x7f4e0c0fbfa0 msgr2=0x7f4e0c0fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:12.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.659+0000 7f4e12e00640 1 -- 172.21.15.43:0/3530246578 shutdown_connections 2024-03-28T22:47:12.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.659+0000 7f4e12e00640 1 -- 172.21.15.43:0/3530246578 wait complete. 2024-03-28T22:47:12.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.660+0000 7f4e12e00640 1 Processor -- start 2024-03-28T22:47:12.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.660+0000 7f4e12e00640 1 -- start start 2024-03-28T22:47:12.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.661+0000 7f4e12e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4e0c100390 0x7f4e0c19a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.661+0000 7f4e12e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4e0c100d40 0x7f4e0c19a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.661+0000 7f4e12e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e0c10df40 0x7f4e0c19ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.661+0000 7f4e12e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4e0c113c20 con 0x7f4e0c10df40 2024-03-28T22:47:12.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.661+0000 7f4e12e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4e0c113aa0 con 0x7f4e0c100390 2024-03-28T22:47:12.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.661+0000 7f4e12e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4e0c113da0 con 0x7f4e0c100d40 2024-03-28T22:47:12.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.661+0000 7f4e11a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4e0c100390 0x7f4e0c19a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.661+0000 7f4e11a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4e0c100390 0x7f4e0c19a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47474/0 (socket says 172.21.15.43:47474) 2024-03-28T22:47:12.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.661+0000 7f4e11000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4e0c100d40 0x7f4e0c19a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.661+0000 7f4e11a00640 1 -- 172.21.15.43:0/3668688578 learned_addr learned my addr 172.21.15.43:0/3668688578 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:12.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.661+0000 7f4e12400640 1 --2- 172.21.15.43:0/3668688578 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e0c10df40 0x7f4e0c19ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.662+0000 7f4e11a00640 1 -- 172.21.15.43:0/3668688578 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4e0c100d40 msgr2=0x7f4e0c19a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.662+0000 7f4e11a00640 1 --2- 172.21.15.43:0/3668688578 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4e0c100d40 0x7f4e0c19a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.662+0000 7f4e11a00640 1 -- 172.21.15.43:0/3668688578 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e0c10df40 msgr2=0x7f4e0c19ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.662+0000 7f4e11a00640 1 --2- 172.21.15.43:0/3668688578 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e0c10df40 0x7f4e0c19ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.662+0000 7f4e11a00640 1 -- 172.21.15.43:0/3668688578 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e0c19f170 con 0x7f4e0c100390 2024-03-28T22:47:12.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.662+0000 7f4e12400640 1 --2- 172.21.15.43:0/3668688578 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e0c10df40 0x7f4e0c19ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:47:12.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.662+0000 7f4e11000640 1 --2- 172.21.15.43:0/3668688578 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4e0c100d40 0x7f4e0c19a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:12.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.662+0000 7f4e11a00640 1 --2- 172.21.15.43:0/3668688578 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4e0c100390 0x7f4e0c19a180 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f4e0002c240 tx=0x7f4e000042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:12.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.662+0000 7f4e0a000640 1 -- 172.21.15.43:0/3668688578 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e00043020 con 0x7f4e0c100390 2024-03-28T22:47:12.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.662+0000 7f4e0a000640 1 -- 172.21.15.43:0/3668688578 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e000054e0 con 0x7f4e0c100390 2024-03-28T22:47:12.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.662+0000 7f4e12e00640 1 -- 172.21.15.43:0/3668688578 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e0c19f400 con 0x7f4e0c100390 2024-03-28T22:47:12.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.662+0000 7f4e0a000640 1 -- 172.21.15.43:0/3668688578 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e00003b20 con 0x7f4e0c100390 2024-03-28T22:47:12.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.663+0000 7f4e12e00640 1 -- 172.21.15.43:0/3668688578 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e0c1a6cf0 con 0x7f4e0c100390 2024-03-28T22:47:12.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.666+0000 7f4e17163740 1 Processor -- start 2024-03-28T22:47:12.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.666+0000 7f4e17163740 1 -- start start 2024-03-28T22:47:12.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.666+0000 7f4e17163740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb8612b550 0x55fb8612b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.666+0000 7f4e17163740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb8612be70 0x55fb86134510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.666+0000 7f4e17163740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb86134a50 0x55fb86136e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.666+0000 7f4e17163740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55fb8608f050 con 0x55fb8612b550 2024-03-28T22:47:12.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.666+0000 7f4e17163740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55fb86037ce0 con 0x55fb8612be70 2024-03-28T22:47:12.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.666+0000 7f4e17163740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55fb8608b0f0 con 0x55fb86134a50 2024-03-28T22:47:12.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.666+0000 7f4e0a000640 1 -- 172.21.15.43:0/3668688578 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f4e00046020 con 0x7f4e0c100390 2024-03-28T22:47:12.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.666+0000 7f4e12400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb8612b550 0x55fb8612b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.666+0000 7f4e11000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb86134a50 0x55fb86136e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.667+0000 7f4e12400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb8612b550 0x55fb8612b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34010/0 (socket says 172.21.15.43:34010) 2024-03-28T22:47:12.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.667+0000 7f4e12400640 1 -- 172.21.15.43:0/2776137782 learned_addr learned my addr 172.21.15.43:0/2776137782 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:12.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.667+0000 7f4e11a00640 1 --2- 172.21.15.43:0/2776137782 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb8612be70 0x55fb86134510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.667+0000 7f4e11000640 1 -- 172.21.15.43:0/2776137782 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb8612be70 msgr2=0x55fb86134510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:12.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.667+0000 7f4e11000640 1 --2- 172.21.15.43:0/2776137782 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb8612be70 0x55fb86134510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.667+0000 7f4e11000640 1 -- 172.21.15.43:0/2776137782 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb8612b550 msgr2=0x55fb8612b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:12.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.667+0000 7f4e11000640 1 --2- 172.21.15.43:0/2776137782 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb8612b550 0x55fb8612b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.667+0000 7f4e11000640 1 -- 172.21.15.43:0/2776137782 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb85da33f0 con 0x55fb86134a50 2024-03-28T22:47:12.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.667+0000 7f4e11000640 1 --2- 172.21.15.43:0/2776137782 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb86134a50 0x55fb86136e40 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f4df8009fa0 tx=0x7f4df80118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3db6b954d7a8d42c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:12.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.667+0000 7f4dfea00640 1 -- 172.21.15.43:0/2776137782 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df8004030 con 0x55fb86134a50 2024-03-28T22:47:12.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.668+0000 7f4dfea00640 1 -- 172.21.15.43:0/2776137782 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4df8005500 con 0x55fb86134a50 2024-03-28T22:47:12.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.668+0000 7f4dfea00640 1 -- 172.21.15.43:0/2776137782 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df8004e10 con 0x55fb86134a50 2024-03-28T22:47:12.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.668+0000 7f4e17163740 1 -- 172.21.15.43:0/2776137782 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb86134a50 msgr2=0x55fb86136e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.668+0000 7f4e17163740 1 --2- 172.21.15.43:0/2776137782 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb86134a50 0x55fb86136e40 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f4df8009fa0 tx=0x7f4df80118d0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.668+0000 7f4e17163740 1 -- 172.21.15.43:0/2776137782 shutdown_connections 2024-03-28T22:47:12.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.668+0000 7f4e17163740 1 --2- 172.21.15.43:0/2776137782 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb8612be70 0x55fb86134510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.668+0000 7f4e17163740 1 --2- 172.21.15.43:0/2776137782 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb86134a50 0x55fb86136e40 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.668+0000 7f4e17163740 1 --2- 172.21.15.43:0/2776137782 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb8612b550 0x55fb8612b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.668+0000 7f4e17163740 1 -- 172.21.15.43:0/2776137782 >> 172.21.15.43:0/2776137782 conn(0x55fb86125c80 msgr2=0x55fb86126060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:12.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.668+0000 7f4e17163740 1 -- 172.21.15.43:0/2776137782 shutdown_connections 2024-03-28T22:47:12.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.668+0000 7f4e0a000640 1 --2- 172.21.15.43:0/3668688578 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4df40a1ea0 0x7f4df40a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.669+0000 7f4e17163740 1 -- 172.21.15.43:0/2776137782 wait complete. 2024-03-28T22:47:12.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.669+0000 7f4e0a000640 1 -- 172.21.15.43:0/3668688578 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9627+0+0 (secure 0 0 0) 0x7f4e000f05e0 con 0x7f4e0c100390 2024-03-28T22:47:12.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.669+0000 7f4e11000640 1 --2- 172.21.15.43:0/3668688578 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4df40a1ea0 0x7f4df40a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.669+0000 7f4e17163740 1 -- start start 2024-03-28T22:47:12.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.669+0000 7f4e17163740 1 Processor -- start 2024-03-28T22:47:12.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.670+0000 7f4e11000640 1 --2- 172.21.15.43:0/3668688578 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4df40a1ea0 0x7f4df40a4360 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4df8009d60 tx=0x7f4df8009fd0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:12.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.670+0000 7f4e17163740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb8612b550 0x55fb86131a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.670+0000 7f4e17163740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb8612be70 0x55fb86131fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.670+0000 7f4e17163740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb86134a50 0x55fb86132500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.670+0000 7f4e17163740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55fb86137840 con 0x55fb8612be70 2024-03-28T22:47:12.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.670+0000 7f4e17163740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55fb8602d2f0 con 0x55fb86134a50 2024-03-28T22:47:12.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.670+0000 7f4e17163740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55fb861379c0 con 0x55fb8612b550 2024-03-28T22:47:12.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.670+0000 7f4e17163740 10 client.0 _ll_register_callbacks cb 0x55fb85da2be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:47:12.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.670+0000 7f4e12400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb86134a50 0x55fb86132500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.670+0000 7f4e11000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb8612be70 0x55fb86131fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.670+0000 7f4e11a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb8612b550 0x55fb86131a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.670+0000 7f4e12400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb86134a50 0x55fb86132500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47488/0 (socket says 172.21.15.43:47488) 2024-03-28T22:47:12.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.670+0000 7f4e11000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb8612be70 0x55fb86131fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34016/0 (socket says 172.21.15.43:34016) 2024-03-28T22:47:12.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.670+0000 7f4e12400640 1 -- 172.21.15.43:0/4158983619 learned_addr learned my addr 172.21.15.43:0/4158983619 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:12.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.671+0000 7f4e12400640 1 -- 172.21.15.43:0/4158983619 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb8612b550 msgr2=0x55fb86131a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:12.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.671+0000 7f4e12400640 1 --2- 172.21.15.43:0/4158983619 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb8612b550 0x55fb86131a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.671+0000 7f4e12400640 1 -- 172.21.15.43:0/4158983619 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb8612be70 msgr2=0x55fb86131fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:12.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.671+0000 7f4e12400640 1 --2- 172.21.15.43:0/4158983619 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb8612be70 0x55fb86131fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.671+0000 7f4e12400640 1 -- 172.21.15.43:0/4158983619 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb85e6d7d0 con 0x55fb86134a50 2024-03-28T22:47:12.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.671+0000 7f4e12400640 1 --2- 172.21.15.43:0/4158983619 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb86134a50 0x55fb86132500 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f4e04011d10 tx=0x7f4e040107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=741b57ad8dc6dffe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:12.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.671+0000 7f4df3400640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:47:12.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.671+0000 7f4df3400640 1 -- 172.21.15.43:0/4158983619 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e0401f060 con 0x55fb86134a50 2024-03-28T22:47:12.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.671+0000 7f4df3400640 1 -- 172.21.15.43:0/4158983619 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9627+0+0 (secure 0 0 0) 0x7f4e0401a410 con 0x55fb86134a50 2024-03-28T22:47:12.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.671+0000 7f4e17163740 1 -- 172.21.15.43:0/4158983619 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4df8005500 con 0x55fb86134a50 2024-03-28T22:47:12.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.671+0000 7f4df3400640 1 -- 172.21.15.43:0/4158983619 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4e04026050 con 0x55fb86134a50 2024-03-28T22:47:12.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.672+0000 7f4e17163740 10 client.28991 fetch_fsmap learned FSMap version 32 2024-03-28T22:47:12.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.672+0000 7f4e17163740 1 -- 172.21.15.43:0/4158983619 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55fb86143b40 con 0x55fb86134a50 2024-03-28T22:47:12.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.672+0000 7f4df3400640 1 -- 172.21.15.43:0/4158983619 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4df8005500 con 0x55fb86134a50 2024-03-28T22:47:12.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.672+0000 7f4df3400640 1 -- 172.21.15.43:0/4158983619 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4e0400e8e0 con 0x55fb86134a50 2024-03-28T22:47:12.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.672+0000 7f4e17163740 10 client.28991 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:47:12.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.672+0000 7f4e17163740 10 client.28991 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:47:12.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.672+0000 7f4e17163740 20 client.28991 populate_metadata read hostname 'smithi043' 2024-03-28T22:47:12.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.672+0000 7f4e17163740 1 --2- 172.21.15.43:0/4158983619 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55fb86145050 0x55fb861474b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.672+0000 7f4e17163740 4 client.28991 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:47:12.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.672+0000 7f4e17163740 1 -- 172.21.15.43:0/4158983619 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb86038680 con 0x55fb86145050 2024-03-28T22:47:12.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.673+0000 7f4e11a00640 1 --2- 172.21.15.43:0/4158983619 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55fb86145050 0x55fb861474b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.673+0000 7f4e11a00640 1 --2- 172.21.15.43:0/4158983619 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55fb86145050 0x55fb861474b0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=59aaf560e6fcf39d server_cookie=57a1d195a307b7d9 in_seq=0 out_seq=0 2024-03-28T22:47:12.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.673+0000 7f4df3400640 10 client.28991 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:47:12.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.674+0000 7f4df3400640 1 -- 172.21.15.43:0/4158983619 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fb86038680 con 0x55fb86145050 2024-03-28T22:47:12.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.674+0000 7f4df3400640 10 client.28991 handle_command_reply: tid=0 2024-03-28T22:47:12.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.674+0000 7f4e17163740 1 client.28991 shutdown 2024-03-28T22:47:12.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.675+0000 7f4e17163740 1 -- 172.21.15.43:0/4158983619 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb86134a50 msgr2=0x55fb86132500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.675+0000 7f4e17163740 1 --2- 172.21.15.43:0/4158983619 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb86134a50 0x55fb86132500 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f4e04011d10 tx=0x7f4e040107a0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.675+0000 7f4e17163740 1 -- 172.21.15.43:0/4158983619 shutdown_connections 2024-03-28T22:47:12.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.675+0000 7f4e17163740 1 --2- 172.21.15.43:0/4158983619 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55fb86145050 0x55fb861474b0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.675+0000 7f4e17163740 1 --2- 172.21.15.43:0/4158983619 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb86134a50 0x55fb86132500 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.675+0000 7f4e17163740 1 --2- 172.21.15.43:0/4158983619 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb8612be70 0x55fb86131fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.675+0000 7f4e17163740 1 --2- 172.21.15.43:0/4158983619 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb8612b550 0x55fb86131a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.675+0000 7f4e17163740 1 -- 172.21.15.43:0/4158983619 >> 172.21.15.43:0/4158983619 conn(0x55fb86125c80 msgr2=0x55fb861279d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:12.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.675+0000 7f4df3400640 0 client.28991 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:47:12.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.675+0000 7f4e17163740 1 -- 172.21.15.43:0/4158983619 shutdown_connections 2024-03-28T22:47:12.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.675+0000 7f4e17163740 1 -- 172.21.15.43:0/4158983619 wait complete. 2024-03-28T22:47:12.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.675+0000 7f4e17163740 20 client.28991 trim_cache size 0 max 16384 2024-03-28T22:47:12.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.690+0000 7f4e17163740 1 Processor -- start 2024-03-28T22:47:12.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.690+0000 7f4e17163740 1 -- start start 2024-03-28T22:47:12.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.690+0000 7f4e17163740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb8612b550 0x55fb861341b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.690+0000 7f4e17163740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb8612be70 0x55fb861405e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.690+0000 7f4e17163740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb86134a50 0x55fb8614cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.690+0000 7f4e17163740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4e0c1b0d90 con 0x55fb8612be70 2024-03-28T22:47:12.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.690+0000 7f4e17163740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4e0c1b28d0 con 0x55fb86134a50 2024-03-28T22:47:12.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.690+0000 7f4e17163740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4e0c1b1460 con 0x55fb8612b550 2024-03-28T22:47:12.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.690+0000 7f4e12400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb86134a50 0x55fb8614cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.691+0000 7f4e11a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb8612b550 0x55fb861341b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.691+0000 7f4e12400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb86134a50 0x55fb8614cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47496/0 (socket says 172.21.15.43:47496) 2024-03-28T22:47:12.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.691+0000 7f4e12400640 1 -- 172.21.15.43:0/2896257858 learned_addr learned my addr 172.21.15.43:0/2896257858 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:12.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.691+0000 7f4e11000640 1 --2- 172.21.15.43:0/2896257858 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb8612be70 0x55fb861405e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.691+0000 7f4e12400640 1 -- 172.21.15.43:0/2896257858 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb8612b550 msgr2=0x55fb861341b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:12.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.691+0000 7f4e12400640 1 --2- 172.21.15.43:0/2896257858 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb8612b550 0x55fb861341b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.691+0000 7f4e12400640 1 -- 172.21.15.43:0/2896257858 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb8612be70 msgr2=0x55fb861405e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:12.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.691+0000 7f4e12400640 1 --2- 172.21.15.43:0/2896257858 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb8612be70 0x55fb861405e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.691+0000 7f4e12400640 1 -- 172.21.15.43:0/2896257858 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb8612d660 con 0x55fb86134a50 2024-03-28T22:47:12.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.691+0000 7f4e12400640 1 --2- 172.21.15.43:0/2896257858 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb86134a50 0x55fb8614cd80 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f4e04000f80 tx=0x7f4e04011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25949469e4e58cd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:12.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.691+0000 7f4dfe000640 1 -- 172.21.15.43:0/2896257858 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e0402f070 con 0x55fb86134a50 2024-03-28T22:47:12.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.691+0000 7f4dfe000640 1 -- 172.21.15.43:0/2896257858 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e04009a70 con 0x55fb86134a50 2024-03-28T22:47:12.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.691+0000 7f4dfe000640 1 -- 172.21.15.43:0/2896257858 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e0402a9d0 con 0x55fb86134a50 2024-03-28T22:47:12.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.692+0000 7f4e17163740 1 -- 172.21.15.43:0/2896257858 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb86134a50 msgr2=0x55fb8614cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.692+0000 7f4e17163740 1 --2- 172.21.15.43:0/2896257858 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb86134a50 0x55fb8614cd80 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f4e04000f80 tx=0x7f4e04011900 comp rx=0 tx=0).stop 2024-03-28T22:47:12.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.692+0000 7f4e17163740 1 -- 172.21.15.43:0/2896257858 shutdown_connections 2024-03-28T22:47:12.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.692+0000 7f4e17163740 1 --2- 172.21.15.43:0/2896257858 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb86134a50 0x55fb8614cd80 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.692+0000 7f4e17163740 1 --2- 172.21.15.43:0/2896257858 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb8612be70 0x55fb861405e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.692+0000 7f4e17163740 1 --2- 172.21.15.43:0/2896257858 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb8612b550 0x55fb861341b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.692+0000 7f4e17163740 1 -- 172.21.15.43:0/2896257858 >> 172.21.15.43:0/2896257858 conn(0x55fb86125c80 msgr2=0x55fb86127940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:12.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.692+0000 7f4e17163740 1 -- 172.21.15.43:0/2896257858 shutdown_connections 2024-03-28T22:47:12.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.692+0000 7f4e17163740 1 -- 172.21.15.43:0/2896257858 wait complete. 2024-03-28T22:47:12.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.693+0000 7f4e17163740 1 -- start start 2024-03-28T22:47:12.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.693+0000 7f4e17163740 1 Processor -- start 2024-03-28T22:47:12.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.693+0000 7f4e17163740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb8612b550 0x55fb86138630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.693+0000 7f4e17163740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb8612be70 0x55fb86138b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.693+0000 7f4e17163740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb86134a50 0x55fb861390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.693+0000 7f4e17163740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4e0c1b2200 con 0x55fb8612be70 2024-03-28T22:47:12.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.693+0000 7f4e17163740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4e0c1b1b30 con 0x55fb8612b550 2024-03-28T22:47:12.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.693+0000 7f4e17163740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f4e0c1aed90 con 0x55fb86134a50 2024-03-28T22:47:12.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.693+0000 7f4e17163740 10 client.0 _ll_register_callbacks cb 0x7f4e04009c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:47:12.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.694+0000 7f4e11a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb8612b550 0x55fb86138630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.694+0000 7f4e11a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb8612b550 0x55fb86138630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47508/0 (socket says 172.21.15.43:47508) 2024-03-28T22:47:12.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.694+0000 7f4e11a00640 1 -- 172.21.15.43:0/1427538776 learned_addr learned my addr 172.21.15.43:0/1427538776 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:12.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.694+0000 7f4e12400640 1 --2- 172.21.15.43:0/1427538776 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb86134a50 0x55fb861390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.694+0000 7f4e11000640 1 --2- 172.21.15.43:0/1427538776 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb8612be70 0x55fb86138b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.694+0000 7f4e11a00640 1 -- 172.21.15.43:0/1427538776 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb86134a50 msgr2=0x55fb861390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:12.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.694+0000 7f4e11a00640 1 --2- 172.21.15.43:0/1427538776 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb86134a50 0x55fb861390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.694+0000 7f4e11a00640 1 -- 172.21.15.43:0/1427538776 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb8612be70 msgr2=0x55fb86138b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:12.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.694+0000 7f4e11a00640 1 --2- 172.21.15.43:0/1427538776 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb8612be70 0x55fb86138b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.694+0000 7f4e11a00640 1 -- 172.21.15.43:0/1427538776 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb861aa4e0 con 0x55fb8612b550 2024-03-28T22:47:12.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.694+0000 7f4e11000640 1 --2- 172.21.15.43:0/1427538776 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb8612be70 0x55fb86138b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:47:12.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.694+0000 7f4e11a00640 1 --2- 172.21.15.43:0/1427538776 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb8612b550 0x55fb86138630 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f4e000bc930 tx=0x7f4e0002cdf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb19c0f8a334b49f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:12.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.694+0000 7f4df3400640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:47:12.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.694+0000 7f4df3400640 1 -- 172.21.15.43:0/1427538776 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e000a7040 con 0x55fb8612b550 2024-03-28T22:47:12.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.694+0000 7f4df3400640 1 -- 172.21.15.43:0/1427538776 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9627+0+0 (secure 0 0 0) 0x7f4e000baf30 con 0x55fb8612b550 2024-03-28T22:47:12.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.694+0000 7f4e17163740 1 -- 172.21.15.43:0/1427538776 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fb8609c590 con 0x55fb8612b550 2024-03-28T22:47:12.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.695+0000 7f4df3400640 1 -- 172.21.15.43:0/1427538776 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4e000bd330 con 0x55fb8612b550 2024-03-28T22:47:12.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.695+0000 7f4e17163740 10 client.28997 fetch_fsmap learned FSMap version 32 2024-03-28T22:47:12.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.695+0000 7f4e17163740 1 -- 172.21.15.43:0/1427538776 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55fb861ab820 con 0x55fb8612b550 2024-03-28T22:47:12.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.695+0000 7f4df3400640 1 -- 172.21.15.43:0/1427538776 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55fb8609c590 con 0x55fb8612b550 2024-03-28T22:47:12.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.695+0000 7f4df3400640 1 -- 172.21.15.43:0/1427538776 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4e00003cc0 con 0x55fb8612b550 2024-03-28T22:47:12.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.695+0000 7f4e17163740 10 client.28997 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:47:12.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.695+0000 7f4e17163740 10 client.28997 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:47:12.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.695+0000 7f4e17163740 20 client.28997 populate_metadata read hostname 'smithi043' 2024-03-28T22:47:12.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.695+0000 7f4e17163740 1 --2- 172.21.15.43:0/1427538776 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55fb861abef0 0x55fb861ae310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.695+0000 7f4e17163740 4 client.28997 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:47:12.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.695+0000 7f4e17163740 1 -- 172.21.15.43:0/1427538776 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fb860968e0 con 0x55fb861abef0 2024-03-28T22:47:12.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.695+0000 7f4e11000640 1 --2- 172.21.15.43:0/1427538776 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55fb861abef0 0x55fb861ae310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.696+0000 7f4e11000640 1 --2- 172.21.15.43:0/1427538776 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55fb861abef0 0x55fb861ae310 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9d97f2f84f43d287 server_cookie=d6dbe6b3a2a1f744 in_seq=0 out_seq=0 2024-03-28T22:47:12.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.696+0000 7f4df3400640 10 client.28997 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:47:12.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.696+0000 7f4df3400640 1 -- 172.21.15.43:0/1427538776 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fb860968e0 con 0x55fb861abef0 2024-03-28T22:47:12.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.696+0000 7f4df3400640 10 client.28997 handle_command_reply: tid=0 2024-03-28T22:47:12.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.696+0000 7f4e17163740 1 client.28997 shutdown 2024-03-28T22:47:12.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.696+0000 7f4e17163740 1 -- 172.21.15.43:0/1427538776 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb8612b550 msgr2=0x55fb86138630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.696+0000 7f4e17163740 1 --2- 172.21.15.43:0/1427538776 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb8612b550 0x55fb86138630 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f4e000bc930 tx=0x7f4e0002cdf0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.696+0000 7f4e17163740 1 -- 172.21.15.43:0/1427538776 shutdown_connections 2024-03-28T22:47:12.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.696+0000 7f4e17163740 1 --2- 172.21.15.43:0/1427538776 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55fb86134a50 0x55fb861390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.696+0000 7f4e17163740 1 --2- 172.21.15.43:0/1427538776 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55fb8612be70 0x55fb86138b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.696+0000 7f4e17163740 1 --2- 172.21.15.43:0/1427538776 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55fb861abef0 0x55fb861ae310 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.696+0000 7f4e17163740 1 --2- 172.21.15.43:0/1427538776 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb8612b550 0x55fb86138630 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.696+0000 7f4e17163740 1 -- 172.21.15.43:0/1427538776 >> 172.21.15.43:0/1427538776 conn(0x55fb86125c80 msgr2=0x55fb861270b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:12.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.696+0000 7f4df3400640 0 client.28997 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:47:12.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.697+0000 7f4e17163740 1 -- 172.21.15.43:0/1427538776 shutdown_connections 2024-03-28T22:47:12.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.697+0000 7f4e17163740 1 -- 172.21.15.43:0/1427538776 wait complete. 2024-03-28T22:47:12.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.697+0000 7f4e17163740 20 client.28997 trim_cache size 0 max 16384 2024-03-28T22:47:12.701 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:47:12.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.697+0000 7f4e12e00640 1 -- 172.21.15.43:0/3668688578 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4df40a1ea0 msgr2=0x7f4df40a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.697+0000 7f4e12e00640 1 --2- 172.21.15.43:0/3668688578 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4df40a1ea0 0x7f4df40a4360 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4df8009d60 tx=0x7f4df8009fd0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.697+0000 7f4e12e00640 1 -- 172.21.15.43:0/3668688578 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4e0c100390 msgr2=0x7f4e0c19a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.697+0000 7f4e12e00640 1 --2- 172.21.15.43:0/3668688578 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4e0c100390 0x7f4e0c19a180 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f4e0002c240 tx=0x7f4e000042f0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.697+0000 7f4e12e00640 1 -- 172.21.15.43:0/3668688578 shutdown_connections 2024-03-28T22:47:12.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.697+0000 7f4e12e00640 1 --2- 172.21.15.43:0/3668688578 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f4df40a1ea0 0x7f4df40a4360 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.697+0000 7f4e12e00640 1 --2- 172.21.15.43:0/3668688578 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4e0c10df40 0x7f4e0c19ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.697+0000 7f4e12e00640 1 --2- 172.21.15.43:0/3668688578 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4e0c100d40 0x7f4e0c19a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.697+0000 7f4e12e00640 1 --2- 172.21.15.43:0/3668688578 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4e0c100390 0x7f4e0c19a180 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.697+0000 7f4e12e00640 1 -- 172.21.15.43:0/3668688578 >> 172.21.15.43:0/3668688578 conn(0x7f4e0c0fbfa0 msgr2=0x7f4e0c0fe390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:12.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.697+0000 7f4e12e00640 1 -- 172.21.15.43:0/3668688578 shutdown_connections 2024-03-28T22:47:12.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.698+0000 7f4e12e00640 1 -- 172.21.15.43:0/3668688578 wait complete. 2024-03-28T22:47:12.712 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:47:12.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.925+0000 7f7397000640 1 -- 172.21.15.43:0/391949831 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7390075c80 msgr2=0x7f7390076060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.925+0000 7f7397000640 1 --2- 172.21.15.43:0/391949831 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7390075c80 0x7f7390076060 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f7380009cd0 tx=0x7f738002c0e0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.925+0000 7f7397000640 1 -- 172.21.15.43:0/391949831 shutdown_connections 2024-03-28T22:47:12.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.925+0000 7f7397000640 1 --2- 172.21.15.43:0/391949831 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7390112590 0x7f7390114980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.925+0000 7f7397000640 1 --2- 172.21.15.43:0/391949831 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73900765a0 0x7f7390112050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.925+0000 7f7397000640 1 --2- 172.21.15.43:0/391949831 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7390075c80 0x7f7390076060 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.925+0000 7f7397000640 1 -- 172.21.15.43:0/391949831 >> 172.21.15.43:0/391949831 conn(0x7f73900ff3a0 msgr2=0x7f73901017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:12.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.925+0000 7f7397000640 1 -- 172.21.15.43:0/391949831 shutdown_connections 2024-03-28T22:47:12.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.925+0000 7f7397000640 1 -- 172.21.15.43:0/391949831 wait complete. 2024-03-28T22:47:12.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.927+0000 7f7397000640 1 Processor -- start 2024-03-28T22:47:12.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.927+0000 7f7397000640 1 -- start start 2024-03-28T22:47:12.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.927+0000 7f7397000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7390075c80 0x7f739019e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.927+0000 7f7397000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73900765a0 0x7f739019ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.927+0000 7f7397000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7390112590 0x7f73901a2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.927+0000 7f7397000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f73901180e0 con 0x7f73900765a0 2024-03-28T22:47:12.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.927+0000 7f7397000640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7390117f60 con 0x7f7390075c80 2024-03-28T22:47:12.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.927+0000 7f7397000640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f7390118260 con 0x7f7390112590 2024-03-28T22:47:12.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.928+0000 7f738fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7390075c80 0x7f739019e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.928+0000 7f738fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7390075c80 0x7f739019e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47536/0 (socket says 172.21.15.43:47536) 2024-03-28T22:47:12.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.928+0000 7f738fe00640 1 -- 172.21.15.43:0/1629015206 learned_addr learned my addr 172.21.15.43:0/1629015206 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:12.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.928+0000 7f7395000640 1 --2- 172.21.15.43:0/1629015206 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7390112590 0x7f73901a2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.928+0000 7f738f400640 1 --2- 172.21.15.43:0/1629015206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73900765a0 0x7f739019ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.928+0000 7f738fe00640 1 -- 172.21.15.43:0/1629015206 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7390112590 msgr2=0x7f73901a2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.928+0000 7f738fe00640 1 --2- 172.21.15.43:0/1629015206 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7390112590 0x7f73901a2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.928+0000 7f738fe00640 1 -- 172.21.15.43:0/1629015206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73900765a0 msgr2=0x7f739019ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.928+0000 7f738fe00640 1 --2- 172.21.15.43:0/1629015206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73900765a0 0x7f739019ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.928+0000 7f738fe00640 1 -- 172.21.15.43:0/1629015206 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73901a3640 con 0x7f7390075c80 2024-03-28T22:47:12.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.928+0000 7f738f400640 1 --2- 172.21.15.43:0/1629015206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73900765a0 0x7f739019ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:47:12.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.928+0000 7f738fe00640 1 --2- 172.21.15.43:0/1629015206 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7390075c80 0x7f739019e6a0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f7380000f80 tx=0x7f7380035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:12.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.929+0000 7f738cc00640 1 -- 172.21.15.43:0/1629015206 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7380039040 con 0x7f7390075c80 2024-03-28T22:47:12.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.929+0000 7f738cc00640 1 -- 172.21.15.43:0/1629015206 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7380044070 con 0x7f7390075c80 2024-03-28T22:47:12.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.929+0000 7f7397000640 1 -- 172.21.15.43:0/1629015206 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73901a38d0 con 0x7f7390075c80 2024-03-28T22:47:12.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.929+0000 7f738cc00640 1 -- 172.21.15.43:0/1629015206 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7380004330 con 0x7f7390075c80 2024-03-28T22:47:12.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.929+0000 7f7397000640 1 -- 172.21.15.43:0/1629015206 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73901ab1c0 con 0x7f7390075c80 2024-03-28T22:47:12.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.934+0000 7f738cc00640 1 -- 172.21.15.43:0/1629015206 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f73800044d0 con 0x7f7390075c80 2024-03-28T22:47:12.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.934+0000 7f7399c22740 1 Processor -- start 2024-03-28T22:47:12.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.934+0000 7f7399c22740 1 -- start start 2024-03-28T22:47:12.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.934+0000 7f7399c22740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4897fd380 0x55e4897fd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.934+0000 7f7399c22740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4897fdca0 0x55e489806340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.934+0000 7f7399c22740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e489806880 0x55e489808c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.934+0000 7f7399c22740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e489761050 con 0x55e489806880 2024-03-28T22:47:12.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.934+0000 7f7399c22740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55e489709ce0 con 0x55e4897fdca0 2024-03-28T22:47:12.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.934+0000 7f7399c22740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55e48975d0f0 con 0x55e4897fd380 2024-03-28T22:47:12.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.935+0000 7f7395000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4897fd380 0x55e4897fd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.935+0000 7f738fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4897fdca0 0x55e489806340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.935+0000 7f7395000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4897fd380 0x55e4897fd760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35178/0 (socket says 172.21.15.43:35178) 2024-03-28T22:47:12.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.935+0000 7f7395000640 1 -- 172.21.15.43:0/3209242599 learned_addr learned my addr 172.21.15.43:0/3209242599 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:12.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.935+0000 7f738f400640 1 --2- 172.21.15.43:0/3209242599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e489806880 0x55e489808c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.935+0000 7f7395000640 1 -- 172.21.15.43:0/3209242599 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4897fdca0 msgr2=0x55e489806340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:12.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.935+0000 7f7395000640 1 --2- 172.21.15.43:0/3209242599 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4897fdca0 0x55e489806340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.935+0000 7f7395000640 1 -- 172.21.15.43:0/3209242599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e489806880 msgr2=0x55e489808c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:12.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.935+0000 7f7395000640 1 --2- 172.21.15.43:0/3209242599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e489806880 0x55e489808c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.935+0000 7f7395000640 1 -- 172.21.15.43:0/3209242599 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4894753f0 con 0x55e4897fd380 2024-03-28T22:47:12.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.935+0000 7f738f400640 1 --2- 172.21.15.43:0/3209242599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e489806880 0x55e489808c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:47:12.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.935+0000 7f738fe00640 1 --2- 172.21.15.43:0/3209242599 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4897fdca0 0x55e489806340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:12.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.935+0000 7f7395000640 1 --2- 172.21.15.43:0/3209242599 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4897fd380 0x55e4897fd760 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f7388009ba0 tx=0x7f7388011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5738a4173258d55c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:12.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.936+0000 7f738cc00640 1 --2- 172.21.15.43:0/1629015206 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f73740a1ef0 0x7f73740a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.936+0000 7f7385600640 1 -- 172.21.15.43:0/3209242599 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7388004230 con 0x55e4897fd380 2024-03-28T22:47:12.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.936+0000 7f7385600640 1 -- 172.21.15.43:0/3209242599 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7388009e10 con 0x55e4897fd380 2024-03-28T22:47:12.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.936+0000 7f7385600640 1 -- 172.21.15.43:0/3209242599 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7388004d50 con 0x55e4897fd380 2024-03-28T22:47:12.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.936+0000 7f738cc00640 1 -- 172.21.15.43:0/1629015206 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9627+0+0 (secure 0 0 0) 0x7f73800f0a10 con 0x7f7390075c80 2024-03-28T22:47:12.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.936+0000 7f738fe00640 1 --2- 172.21.15.43:0/1629015206 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f73740a1ef0 0x7f73740a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.937+0000 7f7399c22740 1 -- 172.21.15.43:0/3209242599 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4897fd380 msgr2=0x55e4897fd760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.937+0000 7f7399c22740 1 --2- 172.21.15.43:0/3209242599 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4897fd380 0x55e4897fd760 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f7388009ba0 tx=0x7f7388011810 comp rx=0 tx=0).stop 2024-03-28T22:47:12.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.937+0000 7f7399c22740 1 -- 172.21.15.43:0/3209242599 shutdown_connections 2024-03-28T22:47:12.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.937+0000 7f7399c22740 1 --2- 172.21.15.43:0/3209242599 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4897fdca0 0x55e489806340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.937+0000 7f7399c22740 1 --2- 172.21.15.43:0/3209242599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e489806880 0x55e489808c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.937+0000 7f7399c22740 1 --2- 172.21.15.43:0/3209242599 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4897fd380 0x55e4897fd760 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.937+0000 7f7399c22740 1 -- 172.21.15.43:0/3209242599 >> 172.21.15.43:0/3209242599 conn(0x55e4897f7ab0 msgr2=0x55e4897f7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:12.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.937+0000 7f738fe00640 1 --2- 172.21.15.43:0/1629015206 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f73740a1ef0 0x7f73740a43b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f73800eded0 tx=0x7f73800f4560 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:12.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.937+0000 7f7399c22740 1 -- 172.21.15.43:0/3209242599 shutdown_connections 2024-03-28T22:47:12.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.937+0000 7f7399c22740 1 -- 172.21.15.43:0/3209242599 wait complete. 2024-03-28T22:47:12.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.937+0000 7f7399c22740 1 -- start start 2024-03-28T22:47:12.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.937+0000 7f7399c22740 1 Processor -- start 2024-03-28T22:47:12.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.938+0000 7f7399c22740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4897fd380 0x55e489803840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.938+0000 7f7399c22740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4897fdca0 0x55e489803d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.938+0000 7f7399c22740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e489806880 0x55e4898042c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.938+0000 7f7399c22740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55e489809670 con 0x55e489806880 2024-03-28T22:47:12.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.938+0000 7f7399c22740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55e4896ff2f0 con 0x55e4897fd380 2024-03-28T22:47:12.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.938+0000 7f7399c22740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55e4898097f0 con 0x55e4897fdca0 2024-03-28T22:47:12.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.938+0000 7f7399c22740 10 client.0 _ll_register_callbacks cb 0x55e489474be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:47:12.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.938+0000 7f7395000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4897fd380 0x55e489803840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.938+0000 7f7395000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4897fd380 0x55e489803840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47550/0 (socket says 172.21.15.43:47550) 2024-03-28T22:47:12.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.938+0000 7f7395000640 1 -- 172.21.15.43:0/3191563493 learned_addr learned my addr 172.21.15.43:0/3191563493 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:12.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.938+0000 7f738f400640 1 --2- 172.21.15.43:0/3191563493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4897fdca0 0x55e489803d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.938+0000 7f7395000640 1 --2- 172.21.15.43:0/3191563493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e489806880 0x55e4898042c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.939+0000 7f738f400640 1 -- 172.21.15.43:0/3191563493 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4897fd380 msgr2=0x55e489803840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:12.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.939+0000 7f738f400640 1 --2- 172.21.15.43:0/3191563493 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4897fd380 0x55e489803840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.939+0000 7f738f400640 1 -- 172.21.15.43:0/3191563493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e489806880 msgr2=0x55e4898042c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:12.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.939+0000 7f738f400640 1 --2- 172.21.15.43:0/3191563493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e489806880 0x55e4898042c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.939+0000 7f738f400640 1 -- 172.21.15.43:0/3191563493 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e48953f7d0 con 0x55e4897fdca0 2024-03-28T22:47:12.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.939+0000 7f7395000640 1 --2- 172.21.15.43:0/3191563493 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4897fd380 0x55e489803840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:12.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.939+0000 7f738f400640 1 --2- 172.21.15.43:0/3191563493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4897fdca0 0x55e489803d80 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x55e4897f8c40 tx=0x7f73780117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=849bc15688f77a12 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:12.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.939+0000 7f737e000640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:47:12.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.939+0000 7f737e000640 1 -- 172.21.15.43:0/3191563493 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7378012650 con 0x55e4897fdca0 2024-03-28T22:47:12.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.940+0000 7f7399c22740 1 -- 172.21.15.43:0/3191563493 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7388009e10 con 0x55e4897fdca0 2024-03-28T22:47:12.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.940+0000 7f737e000640 1 -- 172.21.15.43:0/3191563493 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9627+0+0 (secure 0 0 0) 0x7f73780127f0 con 0x55e4897fdca0 2024-03-28T22:47:12.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.940+0000 7f737e000640 1 -- 172.21.15.43:0/3191563493 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f73780129f0 con 0x55e4897fdca0 2024-03-28T22:47:12.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.940+0000 7f7399c22740 10 client.28945 fetch_fsmap learned FSMap version 32 2024-03-28T22:47:12.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.940+0000 7f7399c22740 1 -- 172.21.15.43:0/3191563493 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e489815960 con 0x55e4897fdca0 2024-03-28T22:47:12.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.940+0000 7f737e000640 1 -- 172.21.15.43:0/3191563493 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7388009e10 con 0x55e4897fdca0 2024-03-28T22:47:12.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.940+0000 7f737e000640 1 -- 172.21.15.43:0/3191563493 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f737801a3d0 con 0x55e4897fdca0 2024-03-28T22:47:12.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.941+0000 7f7399c22740 10 client.28945 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:47:12.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.941+0000 7f7399c22740 10 client.28945 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:47:12.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.941+0000 7f7399c22740 20 client.28945 populate_metadata read hostname 'smithi043' 2024-03-28T22:47:12.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.941+0000 7f7399c22740 1 --2- 172.21.15.43:0/3191563493 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55e489816e60 0x55e489819280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.941+0000 7f7399c22740 4 client.28945 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:47:12.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.941+0000 7f7399c22740 1 -- 172.21.15.43:0/3191563493 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e48970a680 con 0x55e489816e60 2024-03-28T22:47:12.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.941+0000 7f7395000640 1 --2- 172.21.15.43:0/3191563493 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55e489816e60 0x55e489819280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.941+0000 7f7395000640 1 --2- 172.21.15.43:0/3191563493 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55e489816e60 0x55e489819280 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8a67e525ac4cc6da server_cookie=beb3f0ec344f7aab in_seq=0 out_seq=0 2024-03-28T22:47:12.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.941+0000 7f737e000640 10 client.28945 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:47:12.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.942+0000 7f737e000640 1 -- 172.21.15.43:0/3191563493 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e48970a680 con 0x55e489816e60 2024-03-28T22:47:12.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.942+0000 7f737e000640 10 client.28945 handle_command_reply: tid=0 2024-03-28T22:47:12.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.943+0000 7f7399c22740 1 client.28945 shutdown 2024-03-28T22:47:12.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.943+0000 7f7399c22740 1 -- 172.21.15.43:0/3191563493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4897fdca0 msgr2=0x55e489803d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.943+0000 7f7399c22740 1 --2- 172.21.15.43:0/3191563493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4897fdca0 0x55e489803d80 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x55e4897f8c40 tx=0x7f73780117b0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.943+0000 7f7399c22740 1 -- 172.21.15.43:0/3191563493 shutdown_connections 2024-03-28T22:47:12.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.943+0000 7f7399c22740 1 --2- 172.21.15.43:0/3191563493 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55e489816e60 0x55e489819280 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.943+0000 7f7399c22740 1 --2- 172.21.15.43:0/3191563493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e489806880 0x55e4898042c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.943+0000 7f7399c22740 1 --2- 172.21.15.43:0/3191563493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e4897fdca0 0x55e489803d80 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.943+0000 7f737e000640 0 client.28945 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:47:12.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.943+0000 7f7399c22740 1 --2- 172.21.15.43:0/3191563493 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4897fd380 0x55e489803840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.943+0000 7f7399c22740 1 -- 172.21.15.43:0/3191563493 >> 172.21.15.43:0/3191563493 conn(0x55e4897f7ab0 msgr2=0x55e4897f9800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:12.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.943+0000 7f7399c22740 1 -- 172.21.15.43:0/3191563493 shutdown_connections 2024-03-28T22:47:12.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.943+0000 7f7399c22740 1 -- 172.21.15.43:0/3191563493 wait complete. 2024-03-28T22:47:12.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.943+0000 7f7399c22740 20 client.28945 trim_cache size 0 max 16384 2024-03-28T22:47:12.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.959+0000 7f7399c22740 1 Processor -- start 2024-03-28T22:47:12.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.959+0000 7f7399c22740 1 -- start start 2024-03-28T22:47:12.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.959+0000 7f7399c22740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4897fd380 0x55e489805f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.959+0000 7f7399c22740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4897fdca0 0x55e4898029a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.959+0000 7f7399c22740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e489806880 0x55e489812410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.959+0000 7f7399c22740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f73901b5480 con 0x55e4897fd380 2024-03-28T22:47:12.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.959+0000 7f7399c22740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f73901b6fc0 con 0x55e4897fdca0 2024-03-28T22:47:12.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.959+0000 7f7399c22740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f73901b5b50 con 0x55e489806880 2024-03-28T22:47:12.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.960+0000 7f738f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4897fdca0 0x55e4898029a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.960+0000 7f7395000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4897fd380 0x55e489805f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.960+0000 7f738f400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4897fdca0 0x55e4898029a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47560/0 (socket says 172.21.15.43:47560) 2024-03-28T22:47:12.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.960+0000 7f738f400640 1 -- 172.21.15.43:0/311242156 learned_addr learned my addr 172.21.15.43:0/311242156 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:12.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.960+0000 7f7395000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4897fd380 0x55e489805f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34086/0 (socket says 172.21.15.43:34086) 2024-03-28T22:47:12.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.960+0000 7f7395000640 1 --2- 172.21.15.43:0/311242156 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e489806880 0x55e489812410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.960+0000 7f738f400640 1 -- 172.21.15.43:0/311242156 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e489806880 msgr2=0x55e489812410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:12.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.960+0000 7f738f400640 1 --2- 172.21.15.43:0/311242156 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e489806880 0x55e489812410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.960+0000 7f738f400640 1 -- 172.21.15.43:0/311242156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4897fd380 msgr2=0x55e489805f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:12.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.960+0000 7f738f400640 1 --2- 172.21.15.43:0/311242156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4897fd380 0x55e489805f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.960+0000 7f738f400640 1 -- 172.21.15.43:0/311242156 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4897fc670 con 0x55e4897fdca0 2024-03-28T22:47:12.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.960+0000 7f738f400640 1 --2- 172.21.15.43:0/311242156 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4897fdca0 0x55e4898029a0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f7378000c00 tx=0x7f737801ad90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b97ade8bfd68c3ea server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:12.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.961+0000 7f7384c00640 1 -- 172.21.15.43:0/311242156 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7378012b90 con 0x55e4897fdca0 2024-03-28T22:47:12.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.961+0000 7f7384c00640 1 -- 172.21.15.43:0/311242156 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f737802e020 con 0x55e4897fdca0 2024-03-28T22:47:12.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.961+0000 7f7384c00640 1 -- 172.21.15.43:0/311242156 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f737801a630 con 0x55e4897fdca0 2024-03-28T22:47:12.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.961+0000 7f7399c22740 1 -- 172.21.15.43:0/311242156 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4897fdca0 msgr2=0x55e4898029a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.961+0000 7f7399c22740 1 --2- 172.21.15.43:0/311242156 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4897fdca0 0x55e4898029a0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f7378000c00 tx=0x7f737801ad90 comp rx=0 tx=0).stop 2024-03-28T22:47:12.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.962+0000 7f7399c22740 1 -- 172.21.15.43:0/311242156 shutdown_connections 2024-03-28T22:47:12.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.962+0000 7f7399c22740 1 --2- 172.21.15.43:0/311242156 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4897fdca0 0x55e4898029a0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.962+0000 7f7399c22740 1 --2- 172.21.15.43:0/311242156 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e489806880 0x55e489812410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.962+0000 7f7399c22740 1 --2- 172.21.15.43:0/311242156 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4897fd380 0x55e489805f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.962+0000 7f7399c22740 1 -- 172.21.15.43:0/311242156 >> 172.21.15.43:0/311242156 conn(0x55e4897f7ab0 msgr2=0x55e489803800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:12.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.962+0000 7f7399c22740 1 -- 172.21.15.43:0/311242156 shutdown_connections 2024-03-28T22:47:12.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.962+0000 7f7399c22740 1 -- 172.21.15.43:0/311242156 wait complete. 2024-03-28T22:47:12.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.962+0000 7f7399c22740 1 -- start start 2024-03-28T22:47:12.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.962+0000 7f7399c22740 1 Processor -- start 2024-03-28T22:47:12.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.963+0000 7f7399c22740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4897fd380 0x55e489800300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.963+0000 7f7399c22740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4897fdca0 0x55e489800840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.963+0000 7f7399c22740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e489806880 0x55e489800d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.963+0000 7f7399c22740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f73901b68f0 con 0x55e4897fd380 2024-03-28T22:47:12.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.963+0000 7f7399c22740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f73901b6220 con 0x55e4897fdca0 2024-03-28T22:47:12.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.963+0000 7f7399c22740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f73901b3480 con 0x55e489806880 2024-03-28T22:47:12.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.963+0000 7f7399c22740 10 client.0 _ll_register_callbacks cb 0x7f737800b520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:47:12.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.963+0000 7f7395000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4897fd380 0x55e489800300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.963+0000 7f7395000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4897fd380 0x55e489800300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34090/0 (socket says 172.21.15.43:34090) 2024-03-28T22:47:12.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.963+0000 7f7395000640 1 -- 172.21.15.43:0/2166818681 learned_addr learned my addr 172.21.15.43:0/2166818681 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:12.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.963+0000 7f7395000640 1 --2- 172.21.15.43:0/2166818681 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e489806880 0x55e489800d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.963+0000 7f738f400640 1 --2- 172.21.15.43:0/2166818681 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4897fdca0 0x55e489800840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.964+0000 7f7395000640 1 -- 172.21.15.43:0/2166818681 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e489806880 msgr2=0x55e489800d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:12.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.964+0000 7f7395000640 1 --2- 172.21.15.43:0/2166818681 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e489806880 0x55e489800d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.964+0000 7f7395000640 1 -- 172.21.15.43:0/2166818681 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4897fdca0 msgr2=0x55e489800840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:12.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.964+0000 7f7395000640 1 --2- 172.21.15.43:0/2166818681 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4897fdca0 0x55e489800840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.964+0000 7f7395000640 1 -- 172.21.15.43:0/2166818681 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4897f72f0 con 0x55e4897fd380 2024-03-28T22:47:12.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.964+0000 7f7395000640 1 --2- 172.21.15.43:0/2166818681 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4897fd380 0x55e489800300 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f73880057c0 tx=0x7f73880048b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7f49aa437d958e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:12.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.964+0000 7f737e000640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:47:12.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.964+0000 7f737e000640 1 -- 172.21.15.43:0/2166818681 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73880045c0 con 0x55e4897fd380 2024-03-28T22:47:12.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.964+0000 7f737e000640 1 -- 172.21.15.43:0/2166818681 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9627+0+0 (secure 0 0 0) 0x7f738801a850 con 0x55e4897fd380 2024-03-28T22:47:12.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.964+0000 7f7399c22740 1 -- 172.21.15.43:0/2166818681 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f737802e020 con 0x55e4897fd380 2024-03-28T22:47:12.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.964+0000 7f737e000640 1 -- 172.21.15.43:0/2166818681 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f738801aa50 con 0x55e4897fd380 2024-03-28T22:47:12.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.964+0000 7f7399c22740 10 client.18741 fetch_fsmap learned FSMap version 32 2024-03-28T22:47:12.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.964+0000 7f7399c22740 1 -- 172.21.15.43:0/2166818681 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e489808c70 con 0x55e4897fd380 2024-03-28T22:47:12.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.965+0000 7f737e000640 1 -- 172.21.15.43:0/2166818681 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f737802e020 con 0x55e4897fd380 2024-03-28T22:47:12.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.965+0000 7f737e000640 1 -- 172.21.15.43:0/2166818681 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f7388024040 con 0x55e4897fd380 2024-03-28T22:47:12.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.965+0000 7f7399c22740 10 client.18741 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:47:12.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.965+0000 7f7399c22740 10 client.18741 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:47:12.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.965+0000 7f7399c22740 20 client.18741 populate_metadata read hostname 'smithi043' 2024-03-28T22:47:12.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.965+0000 7f7399c22740 1 --2- 172.21.15.43:0/2166818681 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55e48987d580 0x55e48987f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:12.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.965+0000 7f7399c22740 4 client.18741 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:47:12.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.965+0000 7f7399c22740 1 -- 172.21.15.43:0/2166818681 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e48976e590 con 0x55e48987d580 2024-03-28T22:47:12.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.965+0000 7f738f400640 1 --2- 172.21.15.43:0/2166818681 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55e48987d580 0x55e48987f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:12.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.966+0000 7f738f400640 1 --2- 172.21.15.43:0/2166818681 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55e48987d580 0x55e48987f9a0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=29814ff82b4db38f server_cookie=486ced572cf17109 in_seq=0 out_seq=0 2024-03-28T22:47:12.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.966+0000 7f737e000640 10 client.18741 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:47:12.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.966+0000 7f737e000640 1 -- 172.21.15.43:0/2166818681 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e48976e590 con 0x55e48987d580 2024-03-28T22:47:12.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.966+0000 7f737e000640 10 client.18741 handle_command_reply: tid=0 2024-03-28T22:47:12.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.966+0000 7f7399c22740 1 client.18741 shutdown 2024-03-28T22:47:12.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.966+0000 7f7399c22740 1 -- 172.21.15.43:0/2166818681 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4897fd380 msgr2=0x55e489800300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.966+0000 7f7399c22740 1 --2- 172.21.15.43:0/2166818681 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4897fd380 0x55e489800300 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f73880057c0 tx=0x7f73880048b0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.966+0000 7f7399c22740 1 -- 172.21.15.43:0/2166818681 shutdown_connections 2024-03-28T22:47:12.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.966+0000 7f7399c22740 1 --2- 172.21.15.43:0/2166818681 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55e48987d580 0x55e48987f9a0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.966+0000 7f7399c22740 1 --2- 172.21.15.43:0/2166818681 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e4897fdca0 0x55e489800840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.966+0000 7f7399c22740 1 --2- 172.21.15.43:0/2166818681 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55e489806880 0x55e489800d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.966+0000 7f7399c22740 1 --2- 172.21.15.43:0/2166818681 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55e4897fd380 0x55e489800300 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.966+0000 7f737e000640 0 client.18741 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:47:12.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.966+0000 7f7399c22740 1 -- 172.21.15.43:0/2166818681 >> 172.21.15.43:0/2166818681 conn(0x55e4897f7ab0 msgr2=0x55e4897f62e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:12.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.967+0000 7f7399c22740 1 -- 172.21.15.43:0/2166818681 shutdown_connections 2024-03-28T22:47:12.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.967+0000 7f7399c22740 1 -- 172.21.15.43:0/2166818681 wait complete. 2024-03-28T22:47:12.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.967+0000 7f7399c22740 20 client.18741 trim_cache size 0 max 16384 2024-03-28T22:47:12.971 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:47:12.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.967+0000 7f7397000640 1 -- 172.21.15.43:0/1629015206 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f73740a1ef0 msgr2=0x7f73740a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.967+0000 7f7397000640 1 --2- 172.21.15.43:0/1629015206 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f73740a1ef0 0x7f73740a43b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f73800eded0 tx=0x7f73800f4560 comp rx=0 tx=0).stop 2024-03-28T22:47:12.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.967+0000 7f7397000640 1 -- 172.21.15.43:0/1629015206 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7390075c80 msgr2=0x7f739019e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:12.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.967+0000 7f7397000640 1 --2- 172.21.15.43:0/1629015206 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7390075c80 0x7f739019e6a0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f7380000f80 tx=0x7f7380035d10 comp rx=0 tx=0).stop 2024-03-28T22:47:12.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.967+0000 7f7397000640 1 -- 172.21.15.43:0/1629015206 shutdown_connections 2024-03-28T22:47:12.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.967+0000 7f7397000640 1 --2- 172.21.15.43:0/1629015206 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f73740a1ef0 0x7f73740a43b0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.967+0000 7f7397000640 1 --2- 172.21.15.43:0/1629015206 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7390112590 0x7f73901a2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.967+0000 7f7397000640 1 --2- 172.21.15.43:0/1629015206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f73900765a0 0x7f739019ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.967+0000 7f7397000640 1 --2- 172.21.15.43:0/1629015206 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7390075c80 0x7f739019e6a0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:12.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.968+0000 7f7397000640 1 -- 172.21.15.43:0/1629015206 >> 172.21.15.43:0/1629015206 conn(0x7f73900ff3a0 msgr2=0x7f7390101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:12.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.968+0000 7f7397000640 1 -- 172.21.15.43:0/1629015206 shutdown_connections 2024-03-28T22:47:12.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:12.968+0000 7f7397000640 1 -- 172.21.15.43:0/1629015206 wait complete. 2024-03-28T22:47:12.983 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:47:13.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.189+0000 7f20d1a00640 1 -- 172.21.15.43:0/3988116322 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f20cc0765a0 msgr2=0x7f20cc111f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:13.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.189+0000 7f20d1a00640 1 --2- 172.21.15.43:0/3988116322 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f20cc0765a0 0x7f20cc111f30 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f20b4009930 tx=0x7f20b402bc20 comp rx=0 tx=0).stop 2024-03-28T22:47:13.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.189+0000 7f20d1a00640 1 -- 172.21.15.43:0/3988116322 shutdown_connections 2024-03-28T22:47:13.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.189+0000 7f20d1a00640 1 --2- 172.21.15.43:0/3988116322 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f20cc0765a0 0x7f20cc111f30 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.189+0000 7f20d1a00640 1 --2- 172.21.15.43:0/3988116322 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20cc112470 0x7f20cc114860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.189+0000 7f20d1a00640 1 --2- 172.21.15.43:0/3988116322 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f20cc075c80 0x7f20cc076060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.189+0000 7f20d1a00640 1 -- 172.21.15.43:0/3988116322 >> 172.21.15.43:0/3988116322 conn(0x7f20cc0ff260 msgr2=0x7f20cc101680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:13.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.189+0000 7f20d1a00640 1 -- 172.21.15.43:0/3988116322 shutdown_connections 2024-03-28T22:47:13.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.189+0000 7f20d1a00640 1 -- 172.21.15.43:0/3988116322 wait complete. 2024-03-28T22:47:13.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.190+0000 7f20d1a00640 1 Processor -- start 2024-03-28T22:47:13.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.191+0000 7f20d1a00640 1 -- start start 2024-03-28T22:47:13.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.191+0000 7f20d1a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20cc075c80 0x7f20cc1a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:13.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.191+0000 7f20d1a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f20cc0765a0 0x7f20cc1a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:13.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.191+0000 7f20d1a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f20cc112470 0x7f20cc19e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:13.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.191+0000 7f20d1a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f20cc118020 con 0x7f20cc0765a0 2024-03-28T22:47:13.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.191+0000 7f20d1a00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f20cc117ea0 con 0x7f20cc112470 2024-03-28T22:47:13.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.191+0000 7f20d1a00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f20cc1181a0 con 0x7f20cc075c80 2024-03-28T22:47:13.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.191+0000 7f20d1000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f20cc112470 0x7f20cc19e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:13.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.191+0000 7f20cbe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20cc075c80 0x7f20cc1a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:13.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.191+0000 7f20c3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f20cc0765a0 0x7f20cc1a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:13.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.191+0000 7f20d1000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f20cc112470 0x7f20cc19e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47588/0 (socket says 172.21.15.43:47588) 2024-03-28T22:47:13.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.191+0000 7f20d1000640 1 -- 172.21.15.43:0/2366529236 learned_addr learned my addr 172.21.15.43:0/2366529236 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:13.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.192+0000 7f20d1000640 1 -- 172.21.15.43:0/2366529236 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20cc075c80 msgr2=0x7f20cc1a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:13.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.192+0000 7f20d1000640 1 --2- 172.21.15.43:0/2366529236 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20cc075c80 0x7f20cc1a45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.192+0000 7f20d1000640 1 -- 172.21.15.43:0/2366529236 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f20cc0765a0 msgr2=0x7f20cc1a4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:13.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.192+0000 7f20d1000640 1 --2- 172.21.15.43:0/2366529236 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f20cc0765a0 0x7f20cc1a4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.192+0000 7f20d1000640 1 -- 172.21.15.43:0/2366529236 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20cc19ef20 con 0x7f20cc112470 2024-03-28T22:47:13.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.192+0000 7f20d1000640 1 --2- 172.21.15.43:0/2366529236 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f20cc112470 0x7f20cc19e660 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f20c400eb40 tx=0x7f20c400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:13.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.192+0000 7f20c9600640 1 -- 172.21.15.43:0/2366529236 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c4004b20 con 0x7f20cc112470 2024-03-28T22:47:13.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.192+0000 7f20c9600640 1 -- 172.21.15.43:0/2366529236 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20c4004cc0 con 0x7f20cc112470 2024-03-28T22:47:13.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.192+0000 7f20d1a00640 1 -- 172.21.15.43:0/2366529236 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20cc19f210 con 0x7f20cc112470 2024-03-28T22:47:13.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.192+0000 7f20c9600640 1 -- 172.21.15.43:0/2366529236 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c4010850 con 0x7f20cc112470 2024-03-28T22:47:13.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.192+0000 7f20d1a00640 1 -- 172.21.15.43:0/2366529236 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20cc1ab520 con 0x7f20cc112470 2024-03-28T22:47:13.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.197+0000 7f20d5d92740 1 Processor -- start 2024-03-28T22:47:13.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.197+0000 7f20c9600640 1 -- 172.21.15.43:0/2366529236 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f20c4005020 con 0x7f20cc112470 2024-03-28T22:47:13.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.197+0000 7f20d5d92740 1 -- start start 2024-03-28T22:47:13.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.198+0000 7f20d5d92740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560f5f335550 0x560f5f335930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:13.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.198+0000 7f20d5d92740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560f5f335e70 0x560f5f33e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:13.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.198+0000 7f20d5d92740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f5f33ea50 0x560f5f340e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:13.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.198+0000 7f20d5d92740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x560f5f299050 con 0x560f5f33ea50 2024-03-28T22:47:13.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.198+0000 7f20d5d92740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x560f5f241ce0 con 0x560f5f335e70 2024-03-28T22:47:13.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.198+0000 7f20d5d92740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x560f5f2950f0 con 0x560f5f335550 2024-03-28T22:47:13.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.198+0000 7f20cbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560f5f335e70 0x560f5f33e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:13.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.198+0000 7f20cbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560f5f335e70 0x560f5f33e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47594/0 (socket says 172.21.15.43:47594) 2024-03-28T22:47:13.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.198+0000 7f20cbe00640 1 -- 172.21.15.43:0/178726597 learned_addr learned my addr 172.21.15.43:0/178726597 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:13.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.198+0000 7f20c3400640 1 --2- 172.21.15.43:0/178726597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560f5f335550 0x560f5f335930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:13.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.198+0000 7f20c3400640 1 --2- 172.21.15.43:0/178726597 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f5f33ea50 0x560f5f340e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:13.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.198+0000 7f20cbe00640 1 -- 172.21.15.43:0/178726597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560f5f335550 msgr2=0x560f5f335930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:13.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.198+0000 7f20cbe00640 1 --2- 172.21.15.43:0/178726597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560f5f335550 0x560f5f335930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.198+0000 7f20cbe00640 1 -- 172.21.15.43:0/178726597 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f5f33ea50 msgr2=0x560f5f340e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:13.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.198+0000 7f20cbe00640 1 --2- 172.21.15.43:0/178726597 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f5f33ea50 0x560f5f340e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.198+0000 7f20cbe00640 1 -- 172.21.15.43:0/178726597 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f5efad3f0 con 0x560f5f335e70 2024-03-28T22:47:13.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.199+0000 7f20cbe00640 1 --2- 172.21.15.43:0/178726597 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560f5f335e70 0x560f5f33e510 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f20bc009cf0 tx=0x7f20bc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2dddd6fbc761da2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:13.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.199+0000 7f20c9600640 1 --2- 172.21.15.43:0/2366529236 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f20b00a1ef0 0x7f20b00a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:13.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.199+0000 7f20c9600640 1 -- 172.21.15.43:0/2366529236 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9627+0+0 (secure 0 0 0) 0x7f20c40cecd0 con 0x7f20cc112470 2024-03-28T22:47:13.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.199+0000 7f20c1600640 1 -- 172.21.15.43:0/178726597 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20bc004230 con 0x560f5f335e70 2024-03-28T22:47:13.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.199+0000 7f20c1600640 1 -- 172.21.15.43:0/178726597 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20bc0043d0 con 0x560f5f335e70 2024-03-28T22:47:13.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.199+0000 7f20c1600640 1 -- 172.21.15.43:0/178726597 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20bc019900 con 0x560f5f335e70 2024-03-28T22:47:13.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.199+0000 7f20c3400640 1 --2- 172.21.15.43:0/2366529236 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f20b00a1ef0 0x7f20b00a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:13.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.200+0000 7f20d5d92740 1 -- 172.21.15.43:0/178726597 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560f5f335e70 msgr2=0x560f5f33e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:13.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.200+0000 7f20d5d92740 1 --2- 172.21.15.43:0/178726597 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560f5f335e70 0x560f5f33e510 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f20bc009cf0 tx=0x7f20bc011810 comp rx=0 tx=0).stop 2024-03-28T22:47:13.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.200+0000 7f20c3400640 1 --2- 172.21.15.43:0/2366529236 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f20b00a1ef0 0x7f20b00a43b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f20b4009900 tx=0x7f20b4005a20 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:13.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.200+0000 7f20d5d92740 1 -- 172.21.15.43:0/178726597 shutdown_connections 2024-03-28T22:47:13.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.200+0000 7f20d5d92740 1 --2- 172.21.15.43:0/178726597 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560f5f335e70 0x560f5f33e510 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.200+0000 7f20d5d92740 1 --2- 172.21.15.43:0/178726597 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f5f33ea50 0x560f5f340e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.200+0000 7f20d5d92740 1 --2- 172.21.15.43:0/178726597 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560f5f335550 0x560f5f335930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.200+0000 7f20d5d92740 1 -- 172.21.15.43:0/178726597 >> 172.21.15.43:0/178726597 conn(0x560f5f32fc80 msgr2=0x560f5f330060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:13.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.200+0000 7f20d5d92740 1 -- 172.21.15.43:0/178726597 shutdown_connections 2024-03-28T22:47:13.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.200+0000 7f20d5d92740 1 -- 172.21.15.43:0/178726597 wait complete. 2024-03-28T22:47:13.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.201+0000 7f20d5d92740 1 -- start start 2024-03-28T22:47:13.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.201+0000 7f20d5d92740 1 Processor -- start 2024-03-28T22:47:13.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.201+0000 7f20d5d92740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560f5f335550 0x560f5f33b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:13.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.201+0000 7f20d5d92740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560f5f335e70 0x560f5f33bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:13.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.201+0000 7f20d5d92740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f5f33ea50 0x560f5f33c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:13.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.201+0000 7f20d5d92740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x560f5f341840 con 0x560f5f33ea50 2024-03-28T22:47:13.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.201+0000 7f20d5d92740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x560f5f2372f0 con 0x560f5f335550 2024-03-28T22:47:13.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.201+0000 7f20d5d92740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x560f5f3419c0 con 0x560f5f335e70 2024-03-28T22:47:13.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.201+0000 7f20d5d92740 10 client.0 _ll_register_callbacks cb 0x560f5efacbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:47:13.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.202+0000 7f20cbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560f5f335550 0x560f5f33b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:13.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.202+0000 7f20c3400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560f5f335e70 0x560f5f33bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:13.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.202+0000 7f20cbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560f5f335550 0x560f5f33b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47608/0 (socket says 172.21.15.43:47608) 2024-03-28T22:47:13.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.202+0000 7f20cbe00640 1 -- 172.21.15.43:0/714107795 learned_addr learned my addr 172.21.15.43:0/714107795 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:13.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.202+0000 7f20c3400640 1 --2- 172.21.15.43:0/714107795 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560f5f335e70 0x560f5f33bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35234/0 (socket says 172.21.15.43:35234) 2024-03-28T22:47:13.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.202+0000 7f20d1000640 1 --2- 172.21.15.43:0/714107795 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f5f33ea50 0x560f5f33c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:13.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.202+0000 7f20cbe00640 1 -- 172.21.15.43:0/714107795 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560f5f335e70 msgr2=0x560f5f33bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:13.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.202+0000 7f20cbe00640 1 --2- 172.21.15.43:0/714107795 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560f5f335e70 0x560f5f33bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.202+0000 7f20cbe00640 1 -- 172.21.15.43:0/714107795 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f5f33ea50 msgr2=0x560f5f33c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:13.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.202+0000 7f20cbe00640 1 --2- 172.21.15.43:0/714107795 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f5f33ea50 0x560f5f33c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.202+0000 7f20cbe00640 1 -- 172.21.15.43:0/714107795 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f5f0777d0 con 0x560f5f335550 2024-03-28T22:47:13.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.202+0000 7f20cbe00640 1 --2- 172.21.15.43:0/714107795 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560f5f335550 0x560f5f33b9e0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f20bc0195c0 tx=0x7f20bc0038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bce600bf4ce104cf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:13.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.202+0000 7f20ba000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:47:13.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.203+0000 7f20ba000640 1 -- 172.21.15.43:0/714107795 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20bc027070 con 0x560f5f335550 2024-03-28T22:47:13.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.203+0000 7f20ba000640 1 -- 172.21.15.43:0/714107795 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9627+0+0 (secure 0 0 0) 0x7f20bc005240 con 0x560f5f335550 2024-03-28T22:47:13.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.203+0000 7f20d5d92740 1 -- 172.21.15.43:0/714107795 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20bc0043d0 con 0x560f5f335550 2024-03-28T22:47:13.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.203+0000 7f20ba000640 1 -- 172.21.15.43:0/714107795 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f20bc033050 con 0x560f5f335550 2024-03-28T22:47:13.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.203+0000 7f20d5d92740 10 client.29027 fetch_fsmap learned FSMap version 32 2024-03-28T22:47:13.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.203+0000 7f20d5d92740 1 -- 172.21.15.43:0/714107795 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560f5f34da40 con 0x560f5f335550 2024-03-28T22:47:13.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.203+0000 7f20ba000640 1 -- 172.21.15.43:0/714107795 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20bc0043d0 con 0x560f5f335550 2024-03-28T22:47:13.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.203+0000 7f20ba000640 1 -- 172.21.15.43:0/714107795 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f20bc004c30 con 0x560f5f335550 2024-03-28T22:47:13.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.204+0000 7f20d5d92740 10 client.29027 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:47:13.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.204+0000 7f20d5d92740 10 client.29027 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:47:13.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.204+0000 7f20d5d92740 20 client.29027 populate_metadata read hostname 'smithi043' 2024-03-28T22:47:13.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.204+0000 7f20d5d92740 1 --2- 172.21.15.43:0/714107795 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560f5f34ee50 0x560f5f351240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:13.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.204+0000 7f20d5d92740 4 client.29027 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:47:13.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.204+0000 7f20d5d92740 1 -- 172.21.15.43:0/714107795 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f5f242680 con 0x560f5f34ee50 2024-03-28T22:47:13.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.204+0000 7f20c3400640 1 --2- 172.21.15.43:0/714107795 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560f5f34ee50 0x560f5f351240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:13.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.204+0000 7f20c3400640 1 --2- 172.21.15.43:0/714107795 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560f5f34ee50 0x560f5f351240 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=822eab1c030f1b2c server_cookie=22e0b0d30f6fbd11 in_seq=0 out_seq=0 2024-03-28T22:47:13.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.204+0000 7f20ba000640 10 client.29027 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:47:13.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.205+0000 7f20ba000640 1 -- 172.21.15.43:0/714107795 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560f5f242680 con 0x560f5f34ee50 2024-03-28T22:47:13.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.205+0000 7f20ba000640 10 client.29027 handle_command_reply: tid=0 2024-03-28T22:47:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.206+0000 7f20d5d92740 1 client.29027 shutdown 2024-03-28T22:47:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.206+0000 7f20d5d92740 1 -- 172.21.15.43:0/714107795 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560f5f335550 msgr2=0x560f5f33b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:13.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.206+0000 7f20d5d92740 1 --2- 172.21.15.43:0/714107795 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560f5f335550 0x560f5f33b9e0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f20bc0195c0 tx=0x7f20bc0038a0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.206+0000 7f20d5d92740 1 -- 172.21.15.43:0/714107795 shutdown_connections 2024-03-28T22:47:13.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.206+0000 7f20d5d92740 1 --2- 172.21.15.43:0/714107795 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f5f33ea50 0x560f5f33c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.206+0000 7f20d5d92740 1 --2- 172.21.15.43:0/714107795 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560f5f335e70 0x560f5f33bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.206+0000 7f20d5d92740 1 --2- 172.21.15.43:0/714107795 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560f5f34ee50 0x560f5f351240 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.206+0000 7f20d5d92740 1 --2- 172.21.15.43:0/714107795 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560f5f335550 0x560f5f33b9e0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.206+0000 7f20ba000640 0 client.29027 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:47:13.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.206+0000 7f20d5d92740 1 -- 172.21.15.43:0/714107795 >> 172.21.15.43:0/714107795 conn(0x560f5f32fc80 msgr2=0x560f5f3319d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:13.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.206+0000 7f20d5d92740 1 -- 172.21.15.43:0/714107795 shutdown_connections 2024-03-28T22:47:13.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.206+0000 7f20d5d92740 1 -- 172.21.15.43:0/714107795 wait complete. 2024-03-28T22:47:13.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.206+0000 7f20d5d92740 20 client.29027 trim_cache size 0 max 16384 2024-03-28T22:47:13.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.221+0000 7f20d5d92740 1 Processor -- start 2024-03-28T22:47:13.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.221+0000 7f20d5d92740 1 -- start start 2024-03-28T22:47:13.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.222+0000 7f20d5d92740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f5f335550 0x560f5f331b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:13.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.222+0000 7f20d5d92740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560f5f335e70 0x560f5f34a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:13.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.222+0000 7f20d5d92740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560f5f33ea50 0x560f5f33c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:13.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.222+0000 7f20d5d92740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f20cc1b52a0 con 0x560f5f335550 2024-03-28T22:47:13.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.222+0000 7f20d5d92740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f20cc1b6de0 con 0x560f5f335e70 2024-03-28T22:47:13.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.222+0000 7f20d5d92740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f20cc1b5970 con 0x560f5f33ea50 2024-03-28T22:47:13.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.222+0000 7f20c3400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560f5f335e70 0x560f5f34a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:13.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.222+0000 7f20c3400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560f5f335e70 0x560f5f34a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47610/0 (socket says 172.21.15.43:47610) 2024-03-28T22:47:13.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.222+0000 7f20c3400640 1 -- 172.21.15.43:0/354629251 learned_addr learned my addr 172.21.15.43:0/354629251 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:13.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.222+0000 7f20d1000640 1 --2- 172.21.15.43:0/354629251 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560f5f33ea50 0x560f5f33c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:13.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.222+0000 7f20cbe00640 1 --2- 172.21.15.43:0/354629251 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f5f335550 0x560f5f331b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:13.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.222+0000 7f20c3400640 1 -- 172.21.15.43:0/354629251 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560f5f33ea50 msgr2=0x560f5f33c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:13.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.222+0000 7f20c3400640 1 --2- 172.21.15.43:0/354629251 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560f5f33ea50 0x560f5f33c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.222+0000 7f20c3400640 1 -- 172.21.15.43:0/354629251 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f5f335550 msgr2=0x560f5f331b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:13.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.222+0000 7f20c3400640 1 --2- 172.21.15.43:0/354629251 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f5f335550 0x560f5f331b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.222+0000 7f20c3400640 1 -- 172.21.15.43:0/354629251 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f5f335ca0 con 0x560f5f335e70 2024-03-28T22:47:13.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.222+0000 7f20cbe00640 1 --2- 172.21.15.43:0/354629251 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f5f335550 0x560f5f331b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:47:13.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.223+0000 7f20c3400640 1 --2- 172.21.15.43:0/354629251 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560f5f335e70 0x560f5f34a5e0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f20b40043a0 tx=0x7f20b403c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4edfb160446ebd40 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:13.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.223+0000 7f20c0c00640 1 -- 172.21.15.43:0/354629251 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b405e070 con 0x560f5f335e70 2024-03-28T22:47:13.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.223+0000 7f20c0c00640 1 -- 172.21.15.43:0/354629251 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20b4049040 con 0x560f5f335e70 2024-03-28T22:47:13.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.223+0000 7f20c0c00640 1 -- 172.21.15.43:0/354629251 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b4004920 con 0x560f5f335e70 2024-03-28T22:47:13.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.223+0000 7f20d5d92740 1 -- 172.21.15.43:0/354629251 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560f5f335e70 msgr2=0x560f5f34a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:13.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.223+0000 7f20d5d92740 1 --2- 172.21.15.43:0/354629251 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560f5f335e70 0x560f5f34a5e0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f20b40043a0 tx=0x7f20b403c000 comp rx=0 tx=0).stop 2024-03-28T22:47:13.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.224+0000 7f20d5d92740 1 -- 172.21.15.43:0/354629251 shutdown_connections 2024-03-28T22:47:13.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.224+0000 7f20d5d92740 1 --2- 172.21.15.43:0/354629251 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560f5f335e70 0x560f5f34a5e0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.224+0000 7f20d5d92740 1 --2- 172.21.15.43:0/354629251 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560f5f33ea50 0x560f5f33c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.224+0000 7f20d5d92740 1 --2- 172.21.15.43:0/354629251 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f5f335550 0x560f5f331b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.224+0000 7f20d5d92740 1 -- 172.21.15.43:0/354629251 >> 172.21.15.43:0/354629251 conn(0x560f5f32fc80 msgr2=0x560f5f33aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:13.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.224+0000 7f20d5d92740 1 -- 172.21.15.43:0/354629251 shutdown_connections 2024-03-28T22:47:13.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.224+0000 7f20d5d92740 1 -- 172.21.15.43:0/354629251 wait complete. 2024-03-28T22:47:13.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.224+0000 7f20d5d92740 1 -- start start 2024-03-28T22:47:13.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.224+0000 7f20d5d92740 1 Processor -- start 2024-03-28T22:47:13.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.225+0000 7f20d5d92740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560f5f335550 0x560f5f346a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:13.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.225+0000 7f20d5d92740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f5f335e70 0x560f5f346f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:13.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.225+0000 7f20d5d92740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560f5f33ea50 0x560f5f341e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:13.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.225+0000 7f20d5d92740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f20cc1b6710 con 0x560f5f335e70 2024-03-28T22:47:13.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.225+0000 7f20d5d92740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f20cc1b6040 con 0x560f5f335550 2024-03-28T22:47:13.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.225+0000 7f20d5d92740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f20cc1b32a0 con 0x560f5f33ea50 2024-03-28T22:47:13.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.225+0000 7f20d5d92740 10 client.0 _ll_register_callbacks cb 0x7f20bc009b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:47:13.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.225+0000 7f20c3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f5f335e70 0x560f5f346f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:13.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.225+0000 7f20cbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560f5f335550 0x560f5f346a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:13.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.225+0000 7f20d1000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560f5f33ea50 0x560f5f341e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:13.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.225+0000 7f20cbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560f5f335550 0x560f5f346a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47612/0 (socket says 172.21.15.43:47612) 2024-03-28T22:47:13.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.225+0000 7f20cbe00640 1 -- 172.21.15.43:0/4044063469 learned_addr learned my addr 172.21.15.43:0/4044063469 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:13.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.225+0000 7f20c3400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f5f335e70 0x560f5f346f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34142/0 (socket says 172.21.15.43:34142) 2024-03-28T22:47:13.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.225+0000 7f20d1000640 1 -- 172.21.15.43:0/4044063469 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560f5f335550 msgr2=0x560f5f346a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:13.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.226+0000 7f20d1000640 1 --2- 172.21.15.43:0/4044063469 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560f5f335550 0x560f5f346a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.226+0000 7f20d1000640 1 -- 172.21.15.43:0/4044063469 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f5f335e70 msgr2=0x560f5f346f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:13.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.226+0000 7f20d1000640 1 --2- 172.21.15.43:0/4044063469 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f5f335e70 0x560f5f346f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.226+0000 7f20d1000640 1 -- 172.21.15.43:0/4044063469 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f5f342680 con 0x560f5f33ea50 2024-03-28T22:47:13.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.226+0000 7f20cbe00640 1 --2- 172.21.15.43:0/4044063469 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560f5f335550 0x560f5f346a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:13.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.226+0000 7f20c3400640 1 --2- 172.21.15.43:0/4044063469 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f5f335e70 0x560f5f346f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:13.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.226+0000 7f20d1000640 1 --2- 172.21.15.43:0/4044063469 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560f5f33ea50 0x560f5f341e80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f20c40cbfd0 tx=0x7f20c40056e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b579cc2b48b20e6a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:13.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.226+0000 7f20ba000640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:47:13.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.226+0000 7f20ba000640 1 -- 172.21.15.43:0/4044063469 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c40b7020 con 0x560f5f33ea50 2024-03-28T22:47:13.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.226+0000 7f20ba000640 1 -- 172.21.15.43:0/4044063469 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9627+0+0 (secure 0 0 0) 0x7f20c40978e0 con 0x560f5f33ea50 2024-03-28T22:47:13.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.226+0000 7f20d5d92740 1 -- 172.21.15.43:0/4044063469 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20b4049040 con 0x560f5f33ea50 2024-03-28T22:47:13.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.226+0000 7f20ba000640 1 -- 172.21.15.43:0/4044063469 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f20c409f3c0 con 0x560f5f33ea50 2024-03-28T22:47:13.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.226+0000 7f20d5d92740 10 client.28969 fetch_fsmap learned FSMap version 32 2024-03-28T22:47:13.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.226+0000 7f20d5d92740 1 -- 172.21.15.43:0/4044063469 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560f5f3b4c20 con 0x560f5f33ea50 2024-03-28T22:47:13.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.227+0000 7f20ba000640 1 -- 172.21.15.43:0/4044063469 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20b4049040 con 0x560f5f33ea50 2024-03-28T22:47:13.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.227+0000 7f20ba000640 1 -- 172.21.15.43:0/4044063469 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f20c4084360 con 0x560f5f33ea50 2024-03-28T22:47:13.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.227+0000 7f20d5d92740 10 client.28969 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:47:13.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.227+0000 7f20d5d92740 10 client.28969 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:47:13.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.227+0000 7f20d5d92740 20 client.28969 populate_metadata read hostname 'smithi043' 2024-03-28T22:47:13.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.227+0000 7f20d5d92740 1 --2- 172.21.15.43:0/4044063469 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560f5f3b5390 0x560f5f3b77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:13.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.227+0000 7f20d5d92740 4 client.28969 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:47:13.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.227+0000 7f20d5d92740 1 -- 172.21.15.43:0/4044063469 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560f5f2a6590 con 0x560f5f3b5390 2024-03-28T22:47:13.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.227+0000 7f20cbe00640 1 --2- 172.21.15.43:0/4044063469 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560f5f3b5390 0x560f5f3b77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:13.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.228+0000 7f20cbe00640 1 --2- 172.21.15.43:0/4044063469 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560f5f3b5390 0x560f5f3b77b0 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4048818baf411d5c server_cookie=492ae4a699689d47 in_seq=0 out_seq=0 2024-03-28T22:47:13.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.228+0000 7f20ba000640 10 client.28969 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:47:13.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.228+0000 7f20ba000640 1 -- 172.21.15.43:0/4044063469 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560f5f2a6590 con 0x560f5f3b5390 2024-03-28T22:47:13.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.228+0000 7f20ba000640 10 client.28969 handle_command_reply: tid=0 2024-03-28T22:47:13.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.228+0000 7f20d5d92740 1 client.28969 shutdown 2024-03-28T22:47:13.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.228+0000 7f20d5d92740 1 -- 172.21.15.43:0/4044063469 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560f5f33ea50 msgr2=0x560f5f341e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:13.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.228+0000 7f20d5d92740 1 --2- 172.21.15.43:0/4044063469 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560f5f33ea50 0x560f5f341e80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f20c40cbfd0 tx=0x7f20c40056e0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.228+0000 7f20d5d92740 1 -- 172.21.15.43:0/4044063469 shutdown_connections 2024-03-28T22:47:13.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.228+0000 7f20d5d92740 1 --2- 172.21.15.43:0/4044063469 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560f5f33ea50 0x560f5f341e80 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.228+0000 7f20d5d92740 1 --2- 172.21.15.43:0/4044063469 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560f5f335e70 0x560f5f346f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.228+0000 7f20d5d92740 1 --2- 172.21.15.43:0/4044063469 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x560f5f3b5390 0x560f5f3b77b0 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.228+0000 7f20d5d92740 1 --2- 172.21.15.43:0/4044063469 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560f5f335550 0x560f5f346a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.228+0000 7f20d5d92740 1 -- 172.21.15.43:0/4044063469 >> 172.21.15.43:0/4044063469 conn(0x560f5f32fc80 msgr2=0x560f5f33aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:13.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.228+0000 7f20ba000640 0 client.28969 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:47:13.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.228+0000 7f20d5d92740 1 -- 172.21.15.43:0/4044063469 shutdown_connections 2024-03-28T22:47:13.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.228+0000 7f20d5d92740 1 -- 172.21.15.43:0/4044063469 wait complete. 2024-03-28T22:47:13.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.228+0000 7f20d5d92740 20 client.28969 trim_cache size 0 max 16384 2024-03-28T22:47:13.234 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:47:13.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.229+0000 7f20d1a00640 1 -- 172.21.15.43:0/2366529236 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f20b00a1ef0 msgr2=0x7f20b00a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:13.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.229+0000 7f20d1a00640 1 --2- 172.21.15.43:0/2366529236 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f20b00a1ef0 0x7f20b00a43b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f20b4009900 tx=0x7f20b4005a20 comp rx=0 tx=0).stop 2024-03-28T22:47:13.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.229+0000 7f20d1a00640 1 -- 172.21.15.43:0/2366529236 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f20cc112470 msgr2=0x7f20cc19e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:13.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.229+0000 7f20d1a00640 1 --2- 172.21.15.43:0/2366529236 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f20cc112470 0x7f20cc19e660 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f20c400eb40 tx=0x7f20c400c2d0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.229+0000 7f20d1a00640 1 -- 172.21.15.43:0/2366529236 shutdown_connections 2024-03-28T22:47:13.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.229+0000 7f20d1a00640 1 --2- 172.21.15.43:0/2366529236 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f20b00a1ef0 0x7f20b00a43b0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.229+0000 7f20d1a00640 1 --2- 172.21.15.43:0/2366529236 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f20cc112470 0x7f20cc19e660 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.229+0000 7f20d1a00640 1 --2- 172.21.15.43:0/2366529236 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f20cc0765a0 0x7f20cc1a4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.229+0000 7f20d1a00640 1 --2- 172.21.15.43:0/2366529236 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20cc075c80 0x7f20cc1a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:13.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.229+0000 7f20d1a00640 1 -- 172.21.15.43:0/2366529236 >> 172.21.15.43:0/2366529236 conn(0x7f20cc0ff260 msgr2=0x7f20cc101650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:13.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.229+0000 7f20d1a00640 1 -- 172.21.15.43:0/2366529236 shutdown_connections 2024-03-28T22:47:13.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:13.229+0000 7f20d1a00640 1 -- 172.21.15.43:0/2366529236 wait complete. 2024-03-28T22:47:13.241 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:13 smithi043 ceph-mon[31823]: pgmap v313: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-28T22:47:13.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:13 smithi082 ceph-mon[36471]: pgmap v313: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-28T22:47:13.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:13 smithi172 ceph-mon[36566]: pgmap v313: 129 pgs: 129 active+clean; 62 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-28T22:47:14.245 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T22:47:14.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:47:14.246 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T22:47:14.344 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-28T22:47:14.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.487+0000 7fe070800640 1 -- 172.21.15.43:0/22896824 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe0681042b0 msgr2=0x7fe0681107c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:14.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.487+0000 7fe070800640 1 --2- 172.21.15.43:0/22896824 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe0681042b0 0x7fe0681107c0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fe064009900 tx=0x7fe06402bc30 comp rx=0 tx=0).stop 2024-03-28T22:47:14.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.487+0000 7fe070800640 1 -- 172.21.15.43:0/22896824 shutdown_connections 2024-03-28T22:47:14.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.487+0000 7fe070800640 1 --2- 172.21.15.43:0/22896824 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe0681042b0 0x7fe0681107c0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.487+0000 7fe070800640 1 --2- 172.21.15.43:0/22896824 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0681038f0 0x7fe068103d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.487+0000 7fe070800640 1 --2- 172.21.15.43:0/22896824 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe068109900 0x7fe068109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.487+0000 7fe070800640 1 -- 172.21.15.43:0/22896824 >> 172.21.15.43:0/22896824 conn(0x7fe0680ff3c0 msgr2=0x7fe0681017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:14.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.487+0000 7fe070800640 1 -- 172.21.15.43:0/22896824 shutdown_connections 2024-03-28T22:47:14.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.487+0000 7fe070800640 1 -- 172.21.15.43:0/22896824 wait complete. 2024-03-28T22:47:14.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.489+0000 7fe070800640 1 Processor -- start 2024-03-28T22:47:14.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.489+0000 7fe070800640 1 -- start start 2024-03-28T22:47:14.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.489+0000 7fe070800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0681038f0 0x7fe06819a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.489+0000 7fe070800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe0681042b0 0x7fe06819a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.489+0000 7fe070800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe068109900 0x7fe06819eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.489+0000 7fe070800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe068113d60 con 0x7fe068109900 2024-03-28T22:47:14.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.489+0000 7fe070800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe068113be0 con 0x7fe0681042b0 2024-03-28T22:47:14.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.489+0000 7fe070800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe068113ee0 con 0x7fe0681038f0 2024-03-28T22:47:14.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.490+0000 7fe06de00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0681038f0 0x7fe06819a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.490+0000 7fe06d400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe0681042b0 0x7fe06819a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.490+0000 7fe06de00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0681038f0 0x7fe06819a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35276/0 (socket says 172.21.15.43:35276) 2024-03-28T22:47:14.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.490+0000 7fe06de00640 1 -- 172.21.15.43:0/1164217546 learned_addr learned my addr 172.21.15.43:0/1164217546 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:14.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.490+0000 7fe06e800640 1 --2- 172.21.15.43:0/1164217546 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe068109900 0x7fe06819eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.490+0000 7fe06d400640 1 -- 172.21.15.43:0/1164217546 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0681038f0 msgr2=0x7fe06819a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:14.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.490+0000 7fe06d400640 1 --2- 172.21.15.43:0/1164217546 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0681038f0 0x7fe06819a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.490+0000 7fe06d400640 1 -- 172.21.15.43:0/1164217546 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe068109900 msgr2=0x7fe06819eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:14.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.490+0000 7fe06d400640 1 --2- 172.21.15.43:0/1164217546 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe068109900 0x7fe06819eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.490+0000 7fe06d400640 1 -- 172.21.15.43:0/1164217546 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe06819f1f0 con 0x7fe0681042b0 2024-03-28T22:47:14.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.490+0000 7fe06de00640 1 --2- 172.21.15.43:0/1164217546 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0681038f0 0x7fe06819a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:14.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.491+0000 7fe06d400640 1 --2- 172.21.15.43:0/1164217546 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe0681042b0 0x7fe06819a740 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fe05800e9f0 tx=0x7fe05800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:14.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.491+0000 7fe05ea00640 1 -- 172.21.15.43:0/1164217546 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe05800ccc0 con 0x7fe0681042b0 2024-03-28T22:47:14.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.491+0000 7fe05ea00640 1 -- 172.21.15.43:0/1164217546 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe058005020 con 0x7fe0681042b0 2024-03-28T22:47:14.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.491+0000 7fe05ea00640 1 -- 172.21.15.43:0/1164217546 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0580049e0 con 0x7fe0681042b0 2024-03-28T22:47:14.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.491+0000 7fe070800640 1 -- 172.21.15.43:0/1164217546 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe06819f4e0 con 0x7fe0681042b0 2024-03-28T22:47:14.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.491+0000 7fe070800640 1 -- 172.21.15.43:0/1164217546 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0681a6d70 con 0x7fe0681042b0 2024-03-28T22:47:14.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.494+0000 7fe0735b2740 1 Processor -- start 2024-03-28T22:47:14.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.494+0000 7fe0735b2740 1 -- start start 2024-03-28T22:47:14.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.495+0000 7fe0735b2740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bbbc7cf70 0x557bbbc7d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.495+0000 7fe0735b2740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bbbc7d890 0x557bbbc85f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.495+0000 7fe05ea00640 1 -- 172.21.15.43:0/1164217546 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fe058005560 con 0x7fe0681042b0 2024-03-28T22:47:14.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.495+0000 7fe0735b2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bbbc86470 0x557bbbc88860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.495+0000 7fe0735b2740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x557bbbbe0c40 con 0x557bbbc7d890 2024-03-28T22:47:14.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.495+0000 7fe0735b2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x557bbbb89c40 con 0x557bbbc86470 2024-03-28T22:47:14.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.495+0000 7fe0735b2740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x557bbbbdcce0 con 0x557bbbc7cf70 2024-03-28T22:47:14.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.495+0000 7fe06e800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bbbc7cf70 0x557bbbc7d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.495+0000 7fe06e800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bbbc7cf70 0x557bbbc7d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35288/0 (socket says 172.21.15.43:35288) 2024-03-28T22:47:14.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.495+0000 7fe06e800640 1 -- 172.21.15.43:0/2453095736 learned_addr learned my addr 172.21.15.43:0/2453095736 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:14.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.495+0000 7fe06de00640 1 --2- 172.21.15.43:0/2453095736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bbbc7d890 0x557bbbc85f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.495+0000 7fe06d400640 1 --2- 172.21.15.43:0/2453095736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bbbc86470 0x557bbbc88860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.495+0000 7fe06e800640 1 -- 172.21.15.43:0/2453095736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bbbc86470 msgr2=0x557bbbc88860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:14.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.495+0000 7fe06e800640 1 --2- 172.21.15.43:0/2453095736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bbbc86470 0x557bbbc88860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.495+0000 7fe06e800640 1 -- 172.21.15.43:0/2453095736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bbbc7d890 msgr2=0x557bbbc85f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:14.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.496+0000 7fe06e800640 1 --2- 172.21.15.43:0/2453095736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bbbc7d890 0x557bbbc85f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.496+0000 7fe06e800640 1 -- 172.21.15.43:0/2453095736 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bbb9bd7e0 con 0x557bbbc7cf70 2024-03-28T22:47:14.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.496+0000 7fe06e800640 1 --2- 172.21.15.43:0/2453095736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bbbc7cf70 0x557bbbc7d350 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fe064009900 tx=0x7fe064005be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0488a5953c73082 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:14.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.496+0000 7fe053000640 1 -- 172.21.15.43:0/2453095736 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe064042070 con 0x557bbbc7cf70 2024-03-28T22:47:14.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.496+0000 7fe053000640 1 -- 172.21.15.43:0/2453095736 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe06402c8b0 con 0x557bbbc7cf70 2024-03-28T22:47:14.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.496+0000 7fe053000640 1 -- 172.21.15.43:0/2453095736 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe064004f20 con 0x557bbbc7cf70 2024-03-28T22:47:14.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.497+0000 7fe0735b2740 1 -- 172.21.15.43:0/2453095736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bbbc7cf70 msgr2=0x557bbbc7d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:14.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.497+0000 7fe0735b2740 1 --2- 172.21.15.43:0/2453095736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bbbc7cf70 0x557bbbc7d350 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fe064009900 tx=0x7fe064005be0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.498+0000 7fe0735b2740 1 -- 172.21.15.43:0/2453095736 shutdown_connections 2024-03-28T22:47:14.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.498+0000 7fe0735b2740 1 --2- 172.21.15.43:0/2453095736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bbbc86470 0x557bbbc88860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.498+0000 7fe0735b2740 1 --2- 172.21.15.43:0/2453095736 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bbbc7d890 0x557bbbc85f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.498+0000 7fe0735b2740 1 --2- 172.21.15.43:0/2453095736 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bbbc7cf70 0x557bbbc7d350 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.498+0000 7fe0735b2740 1 -- 172.21.15.43:0/2453095736 >> 172.21.15.43:0/2453095736 conn(0x557bbbc776a0 msgr2=0x557bbbc77a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:14.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.498+0000 7fe05ea00640 1 --2- 172.21.15.43:0/1164217546 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe0540a1f70 0x7fe0540a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.498+0000 7fe0735b2740 1 -- 172.21.15.43:0/2453095736 shutdown_connections 2024-03-28T22:47:14.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.498+0000 7fe05ea00640 1 -- 172.21.15.43:0/1164217546 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9627+0+0 (secure 0 0 0) 0x7fe0580ce540 con 0x7fe0681042b0 2024-03-28T22:47:14.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.498+0000 7fe0735b2740 1 -- 172.21.15.43:0/2453095736 wait complete. 2024-03-28T22:47:14.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.498+0000 7fe06de00640 1 --2- 172.21.15.43:0/1164217546 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe0540a1f70 0x7fe0540a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.498+0000 7fe0735b2740 1 -- start start 2024-03-28T22:47:14.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.499+0000 7fe0735b2740 1 Processor -- start 2024-03-28T22:47:14.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.499+0000 7fe06de00640 1 --2- 172.21.15.43:0/1164217546 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe0540a1f70 0x7fe0540a4430 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x557bbbc7ca40 tx=0x7fe060009500 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:14.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.499+0000 7fe0735b2740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bbbc7cf70 0x557bbbc83460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.499+0000 7fe0735b2740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bbbc7d890 0x557bbbc839a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.499+0000 7fe0735b2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bbbc86470 0x557bbbc83ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.499+0000 7fe0735b2740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x557bbbc89200 con 0x557bbbc7cf70 2024-03-28T22:47:14.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.499+0000 7fe0735b2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x557bbbb7f250 con 0x557bbbc86470 2024-03-28T22:47:14.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.499+0000 7fe0735b2740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x557bbbc89380 con 0x557bbbc7d890 2024-03-28T22:47:14.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.500+0000 7fe0735b2740 10 client.0 _ll_register_callbacks cb 0x557bbbba0e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:47:14.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.500+0000 7fe06d400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bbbc7d890 0x557bbbc839a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.500+0000 7fe06d400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bbbc7d890 0x557bbbc839a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35302/0 (socket says 172.21.15.43:35302) 2024-03-28T22:47:14.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.500+0000 7fe06d400640 1 -- 172.21.15.43:0/997144372 learned_addr learned my addr 172.21.15.43:0/997144372 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:14.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.500+0000 7fe06de00640 1 --2- 172.21.15.43:0/997144372 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bbbc7cf70 0x557bbbc83460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.500+0000 7fe06e800640 1 --2- 172.21.15.43:0/997144372 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bbbc86470 0x557bbbc83ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.500+0000 7fe06d400640 1 -- 172.21.15.43:0/997144372 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bbbc86470 msgr2=0x557bbbc83ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:14.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.500+0000 7fe06d400640 1 --2- 172.21.15.43:0/997144372 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bbbc86470 0x557bbbc83ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.500+0000 7fe06d400640 1 -- 172.21.15.43:0/997144372 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bbbc7cf70 msgr2=0x557bbbc83460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:14.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.500+0000 7fe06d400640 1 --2- 172.21.15.43:0/997144372 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bbbc7cf70 0x557bbbc83460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.500+0000 7fe06d400640 1 -- 172.21.15.43:0/997144372 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bbba1df70 con 0x557bbbc7d890 2024-03-28T22:47:14.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.500+0000 7fe06d400640 1 --2- 172.21.15.43:0/997144372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bbbc7d890 0x557bbbc839a0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fe05801e000 tx=0x7fe05809a5c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=778184d4d26b2766 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:14.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.500+0000 7fe047400640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:47:14.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.501+0000 7fe047400640 1 -- 172.21.15.43:0/997144372 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe05801c040 con 0x557bbbc7d890 2024-03-28T22:47:14.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.501+0000 7fe047400640 1 -- 172.21.15.43:0/997144372 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9627+0+0 (secure 0 0 0) 0x7fe05809ace0 con 0x557bbbc7d890 2024-03-28T22:47:14.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.501+0000 7fe0735b2740 1 -- 172.21.15.43:0/997144372 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe06402c8b0 con 0x557bbbc7d890 2024-03-28T22:47:14.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.501+0000 7fe047400640 1 -- 172.21.15.43:0/997144372 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe0580cbc70 con 0x557bbbc7d890 2024-03-28T22:47:14.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.501+0000 7fe0735b2740 10 client.28981 fetch_fsmap learned FSMap version 32 2024-03-28T22:47:14.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.501+0000 7fe0735b2740 1 -- 172.21.15.43:0/997144372 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557bbb8b5170 con 0x557bbbc7d890 2024-03-28T22:47:14.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.501+0000 7fe047400640 1 -- 172.21.15.43:0/997144372 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe06402c8b0 con 0x557bbbc7d890 2024-03-28T22:47:14.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.501+0000 7fe047400640 1 -- 172.21.15.43:0/997144372 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe0580a2d90 con 0x557bbbc7d890 2024-03-28T22:47:14.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.501+0000 7fe0735b2740 10 client.28981 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:47:14.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.501+0000 7fe0735b2740 10 client.28981 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:47:14.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.501+0000 7fe0735b2740 20 client.28981 populate_metadata read hostname 'smithi043' 2024-03-28T22:47:14.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.501+0000 7fe0735b2740 1 --2- 172.21.15.43:0/997144372 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557bbbc967d0 0x557bbbc98c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.501+0000 7fe0735b2740 4 client.28981 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:47:14.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.501+0000 7fe0735b2740 1 -- 172.21.15.43:0/997144372 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557bbbbee180 con 0x557bbbc967d0 2024-03-28T22:47:14.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.502+0000 7fe06de00640 1 --2- 172.21.15.43:0/997144372 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557bbbc967d0 0x557bbbc98c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.502+0000 7fe06de00640 1 --2- 172.21.15.43:0/997144372 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557bbbc967d0 0x557bbbc98c30 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c23cab1b311b9e88 server_cookie=72b9b270affee3f1 in_seq=0 out_seq=0 2024-03-28T22:47:14.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.502+0000 7fe047400640 10 client.28981 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:47:14.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.503+0000 7fe047400640 1 -- 172.21.15.43:0/997144372 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557bbbbee180 con 0x557bbbc967d0 2024-03-28T22:47:14.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.503+0000 7fe047400640 10 client.28981 handle_command_reply: tid=0 2024-03-28T22:47:14.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.503+0000 7fe0735b2740 1 client.28981 shutdown 2024-03-28T22:47:14.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.503+0000 7fe0735b2740 1 -- 172.21.15.43:0/997144372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bbbc7d890 msgr2=0x557bbbc839a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:14.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.503+0000 7fe0735b2740 1 --2- 172.21.15.43:0/997144372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bbbc7d890 0x557bbbc839a0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fe05801e000 tx=0x7fe05809a5c0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.503+0000 7fe0735b2740 1 -- 172.21.15.43:0/997144372 shutdown_connections 2024-03-28T22:47:14.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.503+0000 7fe0735b2740 1 --2- 172.21.15.43:0/997144372 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557bbbc967d0 0x557bbbc98c30 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.503+0000 7fe0735b2740 1 --2- 172.21.15.43:0/997144372 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bbbc86470 0x557bbbc83ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.503+0000 7fe0735b2740 1 --2- 172.21.15.43:0/997144372 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bbbc7d890 0x557bbbc839a0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.503+0000 7fe0735b2740 1 --2- 172.21.15.43:0/997144372 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bbbc7cf70 0x557bbbc83460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.503+0000 7fe047400640 0 client.28981 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:47:14.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.503+0000 7fe0735b2740 1 -- 172.21.15.43:0/997144372 >> 172.21.15.43:0/997144372 conn(0x557bbbc776a0 msgr2=0x557bbbc793f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:14.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.503+0000 7fe0735b2740 1 -- 172.21.15.43:0/997144372 shutdown_connections 2024-03-28T22:47:14.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.504+0000 7fe0735b2740 1 -- 172.21.15.43:0/997144372 wait complete. 2024-03-28T22:47:14.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.504+0000 7fe0735b2740 20 client.28981 trim_cache size 0 max 16384 2024-03-28T22:47:14.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.523+0000 7fe0735b2740 1 Processor -- start 2024-03-28T22:47:14.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.523+0000 7fe0735b2740 1 -- start start 2024-03-28T22:47:14.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.523+0000 7fe0735b2740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bbbc7cf70 0x557bbbc85b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.523+0000 7fe0735b2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bbbc7d890 0x557bbbc92030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.523+0000 7fe0735b2740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bbbc86470 0x557bbbc9e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.523+0000 7fe0735b2740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe0681b0f60 con 0x557bbbc7cf70 2024-03-28T22:47:14.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.523+0000 7fe0735b2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe0681b2aa0 con 0x557bbbc7d890 2024-03-28T22:47:14.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.523+0000 7fe0735b2740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe0681b1630 con 0x557bbbc86470 2024-03-28T22:47:14.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.524+0000 7fe06e800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bbbc86470 0x557bbbc9e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.524+0000 7fe06de00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bbbc7cf70 0x557bbbc85b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.524+0000 7fe06d400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bbbc7d890 0x557bbbc92030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.524+0000 7fe06e800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bbbc86470 0x557bbbc9e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35316/0 (socket says 172.21.15.43:35316) 2024-03-28T22:47:14.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.524+0000 7fe06e800640 1 -- 172.21.15.43:0/4083546596 learned_addr learned my addr 172.21.15.43:0/4083546596 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:14.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.524+0000 7fe06d400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bbbc7d890 0x557bbbc92030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47650/0 (socket says 172.21.15.43:47650) 2024-03-28T22:47:14.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.524+0000 7fe06e800640 1 -- 172.21.15.43:0/4083546596 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bbbc7d890 msgr2=0x557bbbc92030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:14.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.524+0000 7fe06e800640 1 --2- 172.21.15.43:0/4083546596 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bbbc7d890 0x557bbbc92030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.524+0000 7fe06e800640 1 -- 172.21.15.43:0/4083546596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bbbc7cf70 msgr2=0x557bbbc85b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:14.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.524+0000 7fe06e800640 1 --2- 172.21.15.43:0/4083546596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bbbc7cf70 0x557bbbc85b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.524+0000 7fe06e800640 1 -- 172.21.15.43:0/4083546596 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bbbc79990 con 0x557bbbc86470 2024-03-28T22:47:14.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.524+0000 7fe06d400640 1 --2- 172.21.15.43:0/4083546596 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bbbc7d890 0x557bbbc92030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:14.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.524+0000 7fe06de00640 1 --2- 172.21.15.43:0/4083546596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bbbc7cf70 0x557bbbc85b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:14.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.525+0000 7fe06e800640 1 --2- 172.21.15.43:0/4083546596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bbbc86470 0x557bbbc9e500 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fe064009900 tx=0x7fe06402bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=389383d0d718cabf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:14.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.525+0000 7fe052600640 1 -- 172.21.15.43:0/4083546596 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe064042070 con 0x557bbbc86470 2024-03-28T22:47:14.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.525+0000 7fe052600640 1 -- 172.21.15.43:0/4083546596 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe06404b030 con 0x557bbbc86470 2024-03-28T22:47:14.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.525+0000 7fe052600640 1 -- 172.21.15.43:0/4083546596 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe064005340 con 0x557bbbc86470 2024-03-28T22:47:14.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.526+0000 7fe0735b2740 1 -- 172.21.15.43:0/4083546596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bbbc86470 msgr2=0x557bbbc9e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:14.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.526+0000 7fe0735b2740 1 --2- 172.21.15.43:0/4083546596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bbbc86470 0x557bbbc9e500 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fe064009900 tx=0x7fe06402bc30 comp rx=0 tx=0).stop 2024-03-28T22:47:14.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.526+0000 7fe0735b2740 1 -- 172.21.15.43:0/4083546596 shutdown_connections 2024-03-28T22:47:14.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.526+0000 7fe0735b2740 1 --2- 172.21.15.43:0/4083546596 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bbbc7d890 0x557bbbc92030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.526+0000 7fe0735b2740 1 --2- 172.21.15.43:0/4083546596 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bbbc86470 0x557bbbc9e500 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.526+0000 7fe0735b2740 1 --2- 172.21.15.43:0/4083546596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bbbc7cf70 0x557bbbc85b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.526+0000 7fe0735b2740 1 -- 172.21.15.43:0/4083546596 >> 172.21.15.43:0/4083546596 conn(0x557bbbc776a0 msgr2=0x557bbbc79280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:14.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.526+0000 7fe0735b2740 1 -- 172.21.15.43:0/4083546596 shutdown_connections 2024-03-28T22:47:14.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.526+0000 7fe0735b2740 1 -- 172.21.15.43:0/4083546596 wait complete. 2024-03-28T22:47:14.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.526+0000 7fe0735b2740 1 -- start start 2024-03-28T22:47:14.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.526+0000 7fe0735b2740 1 Processor -- start 2024-03-28T22:47:14.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.527+0000 7fe0735b2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bbbc7cf70 0x557bbbc8e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.527+0000 7fe0735b2740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bbbc7d890 0x557bbbc8b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.527+0000 7fe0735b2740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bbbc86470 0x557bbbc8bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.527+0000 7fe0735b2740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe0681b23d0 con 0x557bbbc86470 2024-03-28T22:47:14.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.527+0000 7fe0735b2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe0681b1d00 con 0x557bbbc7cf70 2024-03-28T22:47:14.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.527+0000 7fe0735b2740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe0681aef60 con 0x557bbbc7d890 2024-03-28T22:47:14.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.527+0000 7fe0735b2740 10 client.0 _ll_register_callbacks cb 0x7fe0580046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:47:14.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.527+0000 7fe06de00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bbbc7cf70 0x557bbbc8e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.527+0000 7fe06d400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bbbc7d890 0x557bbbc8b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.527+0000 7fe06de00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bbbc7cf70 0x557bbbc8e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47658/0 (socket says 172.21.15.43:47658) 2024-03-28T22:47:14.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.527+0000 7fe06de00640 1 -- 172.21.15.43:0/599848088 learned_addr learned my addr 172.21.15.43:0/599848088 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:14.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.527+0000 7fe06e800640 1 --2- 172.21.15.43:0/599848088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bbbc86470 0x557bbbc8bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.528+0000 7fe06de00640 1 -- 172.21.15.43:0/599848088 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bbbc7d890 msgr2=0x557bbbc8b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:14.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.528+0000 7fe06de00640 1 --2- 172.21.15.43:0/599848088 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bbbc7d890 0x557bbbc8b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.528+0000 7fe06de00640 1 -- 172.21.15.43:0/599848088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bbbc86470 msgr2=0x557bbbc8bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:14.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.528+0000 7fe06de00640 1 --2- 172.21.15.43:0/599848088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bbbc86470 0x557bbbc8bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.528+0000 7fe06de00640 1 -- 172.21.15.43:0/599848088 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bbbc8c0b0 con 0x557bbbc7cf70 2024-03-28T22:47:14.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.528+0000 7fe06d400640 1 --2- 172.21.15.43:0/599848088 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bbbc7d890 0x557bbbc8b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:14.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.528+0000 7fe06e800640 1 --2- 172.21.15.43:0/599848088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bbbc86470 0x557bbbc8bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:47:14.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.528+0000 7fe06de00640 1 --2- 172.21.15.43:0/599848088 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bbbc7cf70 0x557bbbc8e660 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x557bbbc99470 tx=0x7fe060004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec9d7337f77cbb08 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:14.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.528+0000 7fe047400640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:47:14.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.528+0000 7fe047400640 1 -- 172.21.15.43:0/599848088 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe06002ad10 con 0x557bbbc7cf70 2024-03-28T22:47:14.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.528+0000 7fe047400640 1 -- 172.21.15.43:0/599848088 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9627+0+0 (secure 0 0 0) 0x7fe06002a380 con 0x557bbbc7cf70 2024-03-28T22:47:14.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.528+0000 7fe0735b2740 1 -- 172.21.15.43:0/599848088 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe06404b030 con 0x557bbbc7cf70 2024-03-28T22:47:14.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.528+0000 7fe047400640 1 -- 172.21.15.43:0/599848088 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe06003b050 con 0x557bbbc7cf70 2024-03-28T22:47:14.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.529+0000 7fe0735b2740 10 client.29051 fetch_fsmap learned FSMap version 32 2024-03-28T22:47:14.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.529+0000 7fe0735b2740 1 -- 172.21.15.43:0/599848088 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557bbbcfd890 con 0x557bbbc7cf70 2024-03-28T22:47:14.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.529+0000 7fe047400640 1 -- 172.21.15.43:0/599848088 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe06404b030 con 0x557bbbc7cf70 2024-03-28T22:47:14.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.529+0000 7fe047400640 1 -- 172.21.15.43:0/599848088 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe060032400 con 0x557bbbc7cf70 2024-03-28T22:47:14.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.529+0000 7fe0735b2740 10 client.29051 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:47:14.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.529+0000 7fe0735b2740 10 client.29051 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:47:14.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.529+0000 7fe0735b2740 20 client.29051 populate_metadata read hostname 'smithi043' 2024-03-28T22:47:14.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.529+0000 7fe0735b2740 1 --2- 172.21.15.43:0/599848088 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557bbbcfdf90 0x557bbbd003d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.529+0000 7fe0735b2740 4 client.29051 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T22:47:14.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.529+0000 7fe0735b2740 1 -- 172.21.15.43:0/599848088 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557bbbbe84d0 con 0x557bbbcfdf90 2024-03-28T22:47:14.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.529+0000 7fe06d400640 1 --2- 172.21.15.43:0/599848088 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557bbbcfdf90 0x557bbbd003d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.530+0000 7fe06d400640 1 --2- 172.21.15.43:0/599848088 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557bbbcfdf90 0x557bbbd003d0 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd05e623687c23f7 server_cookie=814f0aa9d35021ad in_seq=0 out_seq=0 2024-03-28T22:47:14.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.530+0000 7fe047400640 10 client.29051 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:47:14.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.531+0000 7fe047400640 1 -- 172.21.15.43:0/599848088 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557bbbbe84d0 con 0x557bbbcfdf90 2024-03-28T22:47:14.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.531+0000 7fe047400640 10 client.29051 handle_command_reply: tid=0 2024-03-28T22:47:14.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.531+0000 7fe0735b2740 1 client.29051 shutdown 2024-03-28T22:47:14.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.531+0000 7fe0735b2740 1 -- 172.21.15.43:0/599848088 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bbbc7cf70 msgr2=0x557bbbc8e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:14.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.531+0000 7fe0735b2740 1 --2- 172.21.15.43:0/599848088 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bbbc7cf70 0x557bbbc8e660 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x557bbbc99470 tx=0x7fe060004a40 comp rx=0 tx=0).stop 2024-03-28T22:47:14.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.532+0000 7fe0735b2740 1 -- 172.21.15.43:0/599848088 shutdown_connections 2024-03-28T22:47:14.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.532+0000 7fe0735b2740 1 --2- 172.21.15.43:0/599848088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x557bbbc86470 0x557bbbc8bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.532+0000 7fe0735b2740 1 --2- 172.21.15.43:0/599848088 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x557bbbc7d890 0x557bbbc8b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.532+0000 7fe0735b2740 1 --2- 172.21.15.43:0/599848088 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x557bbbcfdf90 0x557bbbd003d0 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.532+0000 7fe0735b2740 1 --2- 172.21.15.43:0/599848088 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557bbbc7cf70 0x557bbbc8e660 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.532+0000 7fe0735b2740 1 -- 172.21.15.43:0/599848088 >> 172.21.15.43:0/599848088 conn(0x557bbbc776a0 msgr2=0x557bbbc83420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:14.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.532+0000 7fe047400640 0 client.29051 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:47:14.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.532+0000 7fe0735b2740 1 -- 172.21.15.43:0/599848088 shutdown_connections 2024-03-28T22:47:14.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.532+0000 7fe0735b2740 1 -- 172.21.15.43:0/599848088 wait complete. 2024-03-28T22:47:14.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.532+0000 7fe0735b2740 20 client.29051 trim_cache size 0 max 16384 2024-03-28T22:47:14.535 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:47:14.536 INFO:teuthology.orchestra.run.smithi043.stdout: "return_code": 0, 2024-03-28T22:47:14.536 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_tag": "6a3c741a-6fde-4aff-9b56-98d86f5ad922", 2024-03-28T22:47:14.536 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "asynchronous" 2024-03-28T22:47:14.536 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:47:14.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.533+0000 7fe070800640 1 -- 172.21.15.43:0/1164217546 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe0540a1f70 msgr2=0x7fe0540a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:14.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.533+0000 7fe070800640 1 --2- 172.21.15.43:0/1164217546 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe0540a1f70 0x7fe0540a4430 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x557bbbc7ca40 tx=0x7fe060009500 comp rx=0 tx=0).stop 2024-03-28T22:47:14.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.533+0000 7fe070800640 1 -- 172.21.15.43:0/1164217546 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe0681042b0 msgr2=0x7fe06819a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:14.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.533+0000 7fe070800640 1 --2- 172.21.15.43:0/1164217546 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe0681042b0 0x7fe06819a740 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fe05800e9f0 tx=0x7fe05800eeb0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.533+0000 7fe070800640 1 -- 172.21.15.43:0/1164217546 shutdown_connections 2024-03-28T22:47:14.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.533+0000 7fe070800640 1 --2- 172.21.15.43:0/1164217546 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe0540a1f70 0x7fe0540a4430 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.533+0000 7fe070800640 1 --2- 172.21.15.43:0/1164217546 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe0681042b0 0x7fe06819a740 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.533+0000 7fe070800640 1 --2- 172.21.15.43:0/1164217546 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe068109900 0x7fe06819eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.533+0000 7fe070800640 1 --2- 172.21.15.43:0/1164217546 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe0681038f0 0x7fe06819a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.533+0000 7fe070800640 1 -- 172.21.15.43:0/1164217546 >> 172.21.15.43:0/1164217546 conn(0x7fe0680ff3c0 msgr2=0x7fe0680ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:14.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.533+0000 7fe070800640 1 -- 172.21.15.43:0/1164217546 shutdown_connections 2024-03-28T22:47:14.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.533+0000 7fe070800640 1 -- 172.21.15.43:0/1164217546 wait complete. 2024-03-28T22:47:14.549 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:47:14.549 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:47:14.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.552+0000 7fa214a00640 1 -- 172.21.15.43:0/3860217718 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2101077c0 msgr2=0x7fa210107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:14.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.552+0000 7fa214a00640 1 --2- 172.21.15.43:0/3860217718 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2101077c0 0x7fa210107ba0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fa200009900 tx=0x7fa20002bc30 comp rx=0 tx=0).stop 2024-03-28T22:47:14.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.552+0000 7fa214a00640 1 -- 172.21.15.43:0/3860217718 shutdown_connections 2024-03-28T22:47:14.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.552+0000 7fa214a00640 1 --2- 172.21.15.43:0/3860217718 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa210068dd0 0x7fa210069230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.552+0000 7fa214a00640 1 --2- 172.21.15.43:0/3860217718 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa210069770 0x7fa21010e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.552+0000 7fa214a00640 1 --2- 172.21.15.43:0/3860217718 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2101077c0 0x7fa210107ba0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.552+0000 7fa214a00640 1 -- 172.21.15.43:0/3860217718 >> 172.21.15.43:0/3860217718 conn(0x7fa2100774a0 msgr2=0x7fa2100778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:14.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.552+0000 7fa214a00640 1 -- 172.21.15.43:0/3860217718 shutdown_connections 2024-03-28T22:47:14.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.552+0000 7fa214a00640 1 -- 172.21.15.43:0/3860217718 wait complete. 2024-03-28T22:47:14.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.553+0000 7fa214a00640 1 Processor -- start 2024-03-28T22:47:14.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.553+0000 7fa214a00640 1 -- start start 2024-03-28T22:47:14.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.554+0000 7fa214a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa210068dd0 0x7fa210100970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.554+0000 7fa214a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa210069770 0x7fa210100eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.554+0000 7fa214a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2101077c0 0x7fa2101013f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.554+0000 7fa214a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa210111af0 con 0x7fa210068dd0 2024-03-28T22:47:14.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.554+0000 7fa214a00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa210111970 con 0x7fa210069770 2024-03-28T22:47:14.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.554+0000 7fa214a00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fa210111c70 con 0x7fa2101077c0 2024-03-28T22:47:14.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.554+0000 7fa20e800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2101077c0 0x7fa2101013f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.554+0000 7fa20e800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2101077c0 0x7fa2101013f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35356/0 (socket says 172.21.15.43:35356) 2024-03-28T22:47:14.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.554+0000 7fa20e800640 1 -- 172.21.15.43:0/1089898078 learned_addr learned my addr 172.21.15.43:0/1089898078 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:14.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.554+0000 7fa20d400640 1 --2- 172.21.15.43:0/1089898078 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa210069770 0x7fa210100eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.554+0000 7fa20de00640 1 --2- 172.21.15.43:0/1089898078 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa210068dd0 0x7fa210100970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.554+0000 7fa20d400640 1 -- 172.21.15.43:0/1089898078 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2101077c0 msgr2=0x7fa2101013f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:14.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.554+0000 7fa20d400640 1 --2- 172.21.15.43:0/1089898078 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2101077c0 0x7fa2101013f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.554+0000 7fa20d400640 1 -- 172.21.15.43:0/1089898078 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa210068dd0 msgr2=0x7fa210100970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:14.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.554+0000 7fa20d400640 1 --2- 172.21.15.43:0/1089898078 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa210068dd0 0x7fa210100970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.554+0000 7fa20d400640 1 -- 172.21.15.43:0/1089898078 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2101aa8f0 con 0x7fa210069770 2024-03-28T22:47:14.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.554+0000 7fa20e800640 1 --2- 172.21.15.43:0/1089898078 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2101077c0 0x7fa2101013f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:14.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.555+0000 7fa20d400640 1 --2- 172.21.15.43:0/1089898078 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa210069770 0x7fa210100eb0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc00e9f0 tx=0x7fa1fc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:14.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.555+0000 7fa206a00640 1 -- 172.21.15.43:0/1089898078 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc00cd80 con 0x7fa210069770 2024-03-28T22:47:14.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.555+0000 7fa206a00640 1 -- 172.21.15.43:0/1089898078 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1fc0035f0 con 0x7fa210069770 2024-03-28T22:47:14.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.555+0000 7fa206a00640 1 -- 172.21.15.43:0/1089898078 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc010690 con 0x7fa210069770 2024-03-28T22:47:14.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.555+0000 7fa214a00640 1 -- 172.21.15.43:0/1089898078 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2101aab20 con 0x7fa210069770 2024-03-28T22:47:14.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.555+0000 7fa214a00640 1 -- 172.21.15.43:0/1089898078 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2101ab020 con 0x7fa210069770 2024-03-28T22:47:14.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.556+0000 7fa214a00640 1 -- 172.21.15.43:0/1089898078 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa210102e50 con 0x7fa210069770 2024-03-28T22:47:14.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.558+0000 7fa206a00640 1 -- 172.21.15.43:0/1089898078 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fa1fc003b30 con 0x7fa210069770 2024-03-28T22:47:14.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.560+0000 7fa206a00640 1 --2- 172.21.15.43:0/1089898078 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa1f40a1f90 0x7fa1f40a4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.560+0000 7fa206a00640 1 -- 172.21.15.43:0/1089898078 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9627+0+0 (secure 0 0 0) 0x7fa1fc0ce8f0 con 0x7fa210069770 2024-03-28T22:47:14.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.561+0000 7fa20de00640 1 --2- 172.21.15.43:0/1089898078 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa1f40a1f90 0x7fa1f40a4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.561+0000 7fa20de00640 1 --2- 172.21.15.43:0/1089898078 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa1f40a1f90 0x7fa1f40a4450 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa200000c00 tx=0x7fa20002df90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:14.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.568+0000 7fa206a00640 1 -- 172.21.15.43:0/1089898078 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa1fc09b530 con 0x7fa210069770 2024-03-28T22:47:14.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.743+0000 7fa82a000640 1 -- 172.21.15.43:0/3327958860 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa824073b80 msgr2=0x7fa824073f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:14.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.743+0000 7fa82a000640 1 --2- 172.21.15.43:0/3327958860 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa824073b80 0x7fa824073f60 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fa818009900 tx=0x7fa81802bcc0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.743+0000 7fa82a000640 1 -- 172.21.15.43:0/3327958860 shutdown_connections 2024-03-28T22:47:14.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.743+0000 7fa82a000640 1 --2- 172.21.15.43:0/3327958860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa82410ddf0 0x7fa824110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.743+0000 7fa82a000640 1 --2- 172.21.15.43:0/3327958860 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa8240744a0 0x7fa82410d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.743+0000 7fa82a000640 1 --2- 172.21.15.43:0/3327958860 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa824073b80 0x7fa824073f60 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.743+0000 7fa82a000640 1 -- 172.21.15.43:0/3327958860 >> 172.21.15.43:0/3327958860 conn(0x7fa8240fd0c0 msgr2=0x7fa8240ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:14.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.743+0000 7fa82a000640 1 -- 172.21.15.43:0/3327958860 shutdown_connections 2024-03-28T22:47:14.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.743+0000 7fa82a000640 1 -- 172.21.15.43:0/3327958860 wait complete. 2024-03-28T22:47:14.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.744+0000 7fa82a000640 1 Processor -- start 2024-03-28T22:47:14.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.745+0000 7fa82a000640 1 -- start start 2024-03-28T22:47:14.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.745+0000 7fa82a000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa8240744a0 0x7fa82419a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.745+0000 7fa82a000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa82410ddf0 0x7fa82419a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.745+0000 7fa82a000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa82419ebe0 0x7fa82419f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.745+0000 7fa82a000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa824113ba0 con 0x7fa82410ddf0 2024-03-28T22:47:14.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.745+0000 7fa82a000640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa824113a20 con 0x7fa8240744a0 2024-03-28T22:47:14.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.745+0000 7fa82a000640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fa824113d20 con 0x7fa82419ebe0 2024-03-28T22:47:14.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.745+0000 7fa828c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa8240744a0 0x7fa82419a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.745+0000 7fa828c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa8240744a0 0x7fa82419a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47692/0 (socket says 172.21.15.43:47692) 2024-03-28T22:47:14.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.745+0000 7fa828c00640 1 -- 172.21.15.43:0/2851934831 learned_addr learned my addr 172.21.15.43:0/2851934831 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:14.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.745+0000 7fa823e00640 1 --2- 172.21.15.43:0/2851934831 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa82410ddf0 0x7fa82419a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.745+0000 7fa829600640 1 --2- 172.21.15.43:0/2851934831 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa82419ebe0 0x7fa82419f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.746+0000 7fa828c00640 1 -- 172.21.15.43:0/2851934831 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa82419ebe0 msgr2=0x7fa82419f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:14.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.746+0000 7fa828c00640 1 --2- 172.21.15.43:0/2851934831 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa82419ebe0 0x7fa82419f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.746+0000 7fa828c00640 1 -- 172.21.15.43:0/2851934831 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa82410ddf0 msgr2=0x7fa82419a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:14.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.746+0000 7fa828c00640 1 --2- 172.21.15.43:0/2851934831 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa82410ddf0 0x7fa82419a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.746+0000 7fa828c00640 1 -- 172.21.15.43:0/2851934831 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa82419f6d0 con 0x7fa8240744a0 2024-03-28T22:47:14.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.746+0000 7fa828c00640 1 --2- 172.21.15.43:0/2851934831 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa8240744a0 0x7fa82419a370 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fa818005ab0 tx=0x7fa81802c870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:14.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.746+0000 7fa821600640 1 -- 172.21.15.43:0/2851934831 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa818041020 con 0x7fa8240744a0 2024-03-28T22:47:14.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.746+0000 7fa82a000640 1 -- 172.21.15.43:0/2851934831 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa82419f960 con 0x7fa8240744a0 2024-03-28T22:47:14.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.746+0000 7fa821600640 1 -- 172.21.15.43:0/2851934831 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa818005000 con 0x7fa8240744a0 2024-03-28T22:47:14.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.746+0000 7fa821600640 1 -- 172.21.15.43:0/2851934831 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa818003a80 con 0x7fa8240744a0 2024-03-28T22:47:14.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.747+0000 7fa82a000640 1 -- 172.21.15.43:0/2851934831 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8241a7180 con 0x7fa8240744a0 2024-03-28T22:47:14.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.751+0000 7fa82e230740 1 Processor -- start 2024-03-28T22:47:14.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.751+0000 7fa821600640 1 -- 172.21.15.43:0/2851934831 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fa818044020 con 0x7fa8240744a0 2024-03-28T22:47:14.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.751+0000 7fa82e230740 1 -- start start 2024-03-28T22:47:14.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.752+0000 7fa82e230740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f84e1d0db0 0x55f84e1cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.752+0000 7fa82e230740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f84e1d00d0 0x55f84e1d0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.752+0000 7fa82e230740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f84e1d9440 0x55f84e1db830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.752+0000 7fa82e230740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55f84e134100 con 0x55f84e1d9440 2024-03-28T22:47:14.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.752+0000 7fa82e230740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55f84e0dcba0 con 0x55f84e1d00d0 2024-03-28T22:47:14.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.752+0000 7fa82e230740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55f84e1301a0 con 0x55f84e1d0db0 2024-03-28T22:47:14.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.752+0000 7fa828c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f84e1d00d0 0x55f84e1d0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.752+0000 7fa829600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f84e1d0db0 0x55f84e1cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.752+0000 7fa829600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f84e1d0db0 0x55f84e1cfb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35382/0 (socket says 172.21.15.43:35382) 2024-03-28T22:47:14.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.752+0000 7fa828c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f84e1d00d0 0x55f84e1d0550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:47700/0 (socket says 172.21.15.43:47700) 2024-03-28T22:47:14.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.752+0000 7fa829600640 1 -- 172.21.15.43:0/1511163771 learned_addr learned my addr 172.21.15.43:0/1511163771 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:14.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.752+0000 7fa823e00640 1 --2- 172.21.15.43:0/1511163771 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f84e1d9440 0x55f84e1db830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.753+0000 7fa828c00640 1 -- 172.21.15.43:0/1511163771 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f84e1d0db0 msgr2=0x55f84e1cfb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:14.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.753+0000 7fa828c00640 1 --2- 172.21.15.43:0/1511163771 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f84e1d0db0 0x55f84e1cfb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.753+0000 7fa828c00640 1 -- 172.21.15.43:0/1511163771 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f84e1d9440 msgr2=0x55f84e1db830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:14.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.753+0000 7fa828c00640 1 --2- 172.21.15.43:0/1511163771 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f84e1d9440 0x55f84e1db830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.753+0000 7fa828c00640 1 -- 172.21.15.43:0/1511163771 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f84e10d1a0 con 0x55f84e1d00d0 2024-03-28T22:47:14.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.753+0000 7fa829600640 1 --2- 172.21.15.43:0/1511163771 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f84e1d0db0 0x55f84e1cfb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:14.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.753+0000 7fa828c00640 1 --2- 172.21.15.43:0/1511163771 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f84e1d00d0 0x55f84e1d0550 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fa8180ecf40 tx=0x7fa8180f3c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3157e0263b4cb2c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:14.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.753+0000 7fa821600640 1 --2- 172.21.15.43:0/2851934831 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa80c0a1f70 0x7fa80c0a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.753+0000 7fa815c00640 1 -- 172.21.15.43:0/1511163771 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8180ff020 con 0x55f84e1d00d0 2024-03-28T22:47:14.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.753+0000 7fa821600640 1 -- 172.21.15.43:0/2851934831 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9627+0+0 (secure 0 0 0) 0x7fa8180efb30 con 0x7fa8240744a0 2024-03-28T22:47:14.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.753+0000 7fa815c00640 1 -- 172.21.15.43:0/1511163771 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8180bc8e0 con 0x55f84e1d00d0 2024-03-28T22:47:14.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.753+0000 7fa815c00640 1 -- 172.21.15.43:0/1511163771 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8180b8480 con 0x55f84e1d00d0 2024-03-28T22:47:14.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.754+0000 7fa829600640 1 --2- 172.21.15.43:0/2851934831 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa80c0a1f70 0x7fa80c0a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.754+0000 7fa82e230740 1 -- 172.21.15.43:0/1511163771 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f84e1d00d0 msgr2=0x55f84e1d0550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:14.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.754+0000 7fa82e230740 1 --2- 172.21.15.43:0/1511163771 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f84e1d00d0 0x55f84e1d0550 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fa8180ecf40 tx=0x7fa8180f3c70 comp rx=0 tx=0).stop 2024-03-28T22:47:14.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.754+0000 7fa829600640 1 --2- 172.21.15.43:0/2851934831 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa80c0a1f70 0x7fa80c0a4430 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa81c009fd0 tx=0x7fa81c009500 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:14.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.754+0000 7fa82e230740 1 -- 172.21.15.43:0/1511163771 shutdown_connections 2024-03-28T22:47:14.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.754+0000 7fa82e230740 1 --2- 172.21.15.43:0/1511163771 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f84e1d00d0 0x55f84e1d0550 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.754+0000 7fa82e230740 1 --2- 172.21.15.43:0/1511163771 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f84e1d9440 0x55f84e1db830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.754+0000 7fa82e230740 1 --2- 172.21.15.43:0/1511163771 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f84e1d0db0 0x55f84e1cfb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.754+0000 7fa82e230740 1 -- 172.21.15.43:0/1511163771 >> 172.21.15.43:0/1511163771 conn(0x55f84e196a10 msgr2=0x55f84e1cab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:14.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.754+0000 7fa82e230740 1 -- 172.21.15.43:0/1511163771 shutdown_connections 2024-03-28T22:47:14.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.754+0000 7fa82e230740 1 -- 172.21.15.43:0/1511163771 wait complete. 2024-03-28T22:47:14.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.755+0000 7fa82e230740 1 -- start start 2024-03-28T22:47:14.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.755+0000 7fa82e230740 1 Processor -- start 2024-03-28T22:47:14.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.756+0000 7fa82e230740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f84e1d00d0 0x55f84e1d6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.756+0000 7fa82e230740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f84e1d0db0 0x55f84e1d68d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.756+0000 7fa82e230740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f84e1d9440 0x55f84e1d6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.756+0000 7fa82e230740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55f84e1dc070 con 0x55f84e1d9440 2024-03-28T22:47:14.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.756+0000 7fa82e230740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55f84e0d2250 con 0x55f84e1d0db0 2024-03-28T22:47:14.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.756+0000 7fa82e230740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55f84e1dc1f0 con 0x55f84e1d00d0 2024-03-28T22:47:14.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.756+0000 7fa82e230740 10 client.0 _ll_register_callbacks cb 0x55f84de47be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:47:14.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.756+0000 7fa828c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f84e1d00d0 0x55f84e1d6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.756+0000 7fa823e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f84e1d0db0 0x55f84e1d68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.756+0000 7fa828c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f84e1d00d0 0x55f84e1d6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35390/0 (socket says 172.21.15.43:35390) 2024-03-28T22:47:14.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.756+0000 7fa828c00640 1 -- 172.21.15.43:0/3916269861 learned_addr learned my addr 172.21.15.43:0/3916269861 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:14.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.756+0000 7fa829600640 1 --2- 172.21.15.43:0/3916269861 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f84e1d9440 0x55f84e1d6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.756+0000 7fa828c00640 1 -- 172.21.15.43:0/3916269861 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f84e1d0db0 msgr2=0x55f84e1d68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:14.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.756+0000 7fa828c00640 1 --2- 172.21.15.43:0/3916269861 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f84e1d0db0 0x55f84e1d68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.756+0000 7fa828c00640 1 -- 172.21.15.43:0/3916269861 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f84e1d9440 msgr2=0x55f84e1d6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:14.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.756+0000 7fa828c00640 1 --2- 172.21.15.43:0/3916269861 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f84e1d9440 0x55f84e1d6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.756+0000 7fa828c00640 1 -- 172.21.15.43:0/3916269861 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f84e0ee960 con 0x55f84e1d00d0 2024-03-28T22:47:14.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.756+0000 7fa823e00640 1 --2- 172.21.15.43:0/3916269861 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f84e1d0db0 0x55f84e1d68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:14.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.756+0000 7fa829600640 1 --2- 172.21.15.43:0/3916269861 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f84e1d9440 0x55f84e1d6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:47:14.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.757+0000 7fa828c00640 1 --2- 172.21.15.43:0/3916269861 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f84e1d00d0 0x55f84e1d6390 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fa8180ecff0 tx=0x7fa818005c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58fd917a7abf1eca server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:14.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.757+0000 7fa80a000640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:47:14.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.757+0000 7fa80a000640 1 -- 172.21.15.43:0/3916269861 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8180ff070 con 0x55f84e1d00d0 2024-03-28T22:47:14.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.757+0000 7fa80a000640 1 -- 172.21.15.43:0/3916269861 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9627+0+0 (secure 0 0 0) 0x7fa8180051b0 con 0x55f84e1d00d0 2024-03-28T22:47:14.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.757+0000 7fa82e230740 1 -- 172.21.15.43:0/3916269861 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8180bc8e0 con 0x55f84e1d00d0 2024-03-28T22:47:14.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.757+0000 7fa80a000640 1 -- 172.21.15.43:0/3916269861 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa8180bbb40 con 0x55f84e1d00d0 2024-03-28T22:47:14.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.757+0000 7fa82e230740 10 client.29005 fetch_fsmap learned FSMap version 32 2024-03-28T22:47:14.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.757+0000 7fa82e230740 1 -- 172.21.15.43:0/3916269861 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f84df127d0 con 0x55f84e1d00d0 2024-03-28T22:47:14.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.757+0000 7fa80a000640 1 -- 172.21.15.43:0/3916269861 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8180bc8e0 con 0x55f84e1d00d0 2024-03-28T22:47:14.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.757+0000 7fa80a000640 1 -- 172.21.15.43:0/3916269861 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa8180b8850 con 0x55f84e1d00d0 2024-03-28T22:47:14.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.758+0000 7fa82e230740 10 client.29005 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:47:14.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.758+0000 7fa82e230740 10 client.29005 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:47:14.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.758+0000 7fa82e230740 20 client.29005 populate_metadata read hostname 'smithi043' 2024-03-28T22:47:14.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.758+0000 7fa82e230740 1 --2- 172.21.15.43:0/3916269861 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f84e1e93c0 0x55f84e1eb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.758+0000 7fa82e230740 4 client.29005 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:47:14.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.758+0000 7fa82e230740 1 -- 172.21.15.43:0/3916269861 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f84e141640 con 0x55f84e1e93c0 2024-03-28T22:47:14.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.758+0000 7fa823e00640 1 --2- 172.21.15.43:0/3916269861 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f84e1e93c0 0x55f84e1eb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.758+0000 7fa823e00640 1 --2- 172.21.15.43:0/3916269861 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f84e1e93c0 0x55f84e1eb7b0 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9faa8e586e61564 server_cookie=da8c1e3f5ffead85 in_seq=0 out_seq=0 2024-03-28T22:47:14.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.758+0000 7fa80a000640 10 client.29005 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:47:14.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.759+0000 7fa80a000640 1 -- 172.21.15.43:0/3916269861 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f84e141640 con 0x55f84e1e93c0 2024-03-28T22:47:14.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.759+0000 7fa80a000640 10 client.29005 handle_command_reply: tid=0 2024-03-28T22:47:14.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.759+0000 7fa82e230740 1 client.29005 shutdown 2024-03-28T22:47:14.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.759+0000 7fa82e230740 1 -- 172.21.15.43:0/3916269861 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f84e1d00d0 msgr2=0x55f84e1d6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:14.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.759+0000 7fa82e230740 1 --2- 172.21.15.43:0/3916269861 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f84e1d00d0 0x55f84e1d6390 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fa8180ecff0 tx=0x7fa818005c90 comp rx=0 tx=0).stop 2024-03-28T22:47:14.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.759+0000 7fa82e230740 1 -- 172.21.15.43:0/3916269861 shutdown_connections 2024-03-28T22:47:14.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.759+0000 7fa82e230740 1 --2- 172.21.15.43:0/3916269861 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f84e1e93c0 0x55f84e1eb7b0 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.759+0000 7fa82e230740 1 --2- 172.21.15.43:0/3916269861 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f84e1d0db0 0x55f84e1d68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.759+0000 7fa82e230740 1 --2- 172.21.15.43:0/3916269861 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f84e1d9440 0x55f84e1d6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.759+0000 7fa82e230740 1 --2- 172.21.15.43:0/3916269861 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f84e1d00d0 0x55f84e1d6390 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.759+0000 7fa80a000640 0 client.29005 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:47:14.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.759+0000 7fa82e230740 1 -- 172.21.15.43:0/3916269861 >> 172.21.15.43:0/3916269861 conn(0x55f84e196a10 msgr2=0x55f84e1cc870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:14.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.760+0000 7fa82e230740 1 -- 172.21.15.43:0/3916269861 shutdown_connections 2024-03-28T22:47:14.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.760+0000 7fa82e230740 1 -- 172.21.15.43:0/3916269861 wait complete. 2024-03-28T22:47:14.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.760+0000 7fa82e230740 20 client.29005 trim_cache size 0 max 16384 2024-03-28T22:47:14.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.779+0000 7fa82e230740 1 Processor -- start 2024-03-28T22:47:14.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.779+0000 7fa82e230740 1 -- start start 2024-03-28T22:47:14.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.779+0000 7fa82e230740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f84e1d00d0 0x55f84e1e4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.779+0000 7fa82e230740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f84e1d0db0 0x55f84e1f0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.779+0000 7fa82e230740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f84e1d9440 0x55f84e1d7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.779+0000 7fa82e230740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa8241b1420 con 0x55f84e1d0db0 2024-03-28T22:47:14.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.779+0000 7fa82e230740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa8241b2f60 con 0x55f84e1d9440 2024-03-28T22:47:14.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.779+0000 7fa82e230740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fa8241b1af0 con 0x55f84e1d00d0 2024-03-28T22:47:14.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.780+0000 7fa828c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f84e1d00d0 0x55f84e1e4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.780+0000 7fa829600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f84e1d9440 0x55f84e1d7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.780+0000 7fa828c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f84e1d00d0 0x55f84e1e4e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35400/0 (socket says 172.21.15.43:35400) 2024-03-28T22:47:14.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.780+0000 7fa828c00640 1 -- 172.21.15.43:0/3842971735 learned_addr learned my addr 172.21.15.43:0/3842971735 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:14.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.780+0000 7fa823e00640 1 --2- 172.21.15.43:0/3842971735 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f84e1d0db0 0x55f84e1f0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.780+0000 7fa829600640 1 -- 172.21.15.43:0/3842971735 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f84e1d00d0 msgr2=0x55f84e1e4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:14.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.780+0000 7fa829600640 1 --2- 172.21.15.43:0/3842971735 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f84e1d00d0 0x55f84e1e4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.780+0000 7fa829600640 1 -- 172.21.15.43:0/3842971735 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f84e1d0db0 msgr2=0x55f84e1f0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:14.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.780+0000 7fa829600640 1 --2- 172.21.15.43:0/3842971735 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f84e1d0db0 0x55f84e1f0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.780+0000 7fa829600640 1 -- 172.21.15.43:0/3842971735 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f84de483f0 con 0x55f84e1d9440 2024-03-28T22:47:14.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.780+0000 7fa828c00640 1 --2- 172.21.15.43:0/3842971735 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f84e1d00d0 0x55f84e1e4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:14.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.780+0000 7fa829600640 1 --2- 172.21.15.43:0/3842971735 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f84e1d9440 0x55f84e1d7ab0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fa81c00f790 tx=0x7fa81c016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e097da5ea0ca139 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:14.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.781+0000 7fa815200640 1 -- 172.21.15.43:0/3842971735 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81c004b20 con 0x55f84e1d9440 2024-03-28T22:47:14.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.781+0000 7fa815200640 1 -- 172.21.15.43:0/3842971735 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa81c004cc0 con 0x55f84e1d9440 2024-03-28T22:47:14.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.781+0000 7fa815200640 1 -- 172.21.15.43:0/3842971735 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81c005110 con 0x55f84e1d9440 2024-03-28T22:47:14.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.781+0000 7fa82e230740 1 -- 172.21.15.43:0/3842971735 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f84e1d9440 msgr2=0x55f84e1d7ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:14.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.781+0000 7fa82e230740 1 --2- 172.21.15.43:0/3842971735 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f84e1d9440 0x55f84e1d7ab0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fa81c00f790 tx=0x7fa81c016000 comp rx=0 tx=0).stop 2024-03-28T22:47:14.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.782+0000 7fa82e230740 1 -- 172.21.15.43:0/3842971735 shutdown_connections 2024-03-28T22:47:14.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.782+0000 7fa82e230740 1 --2- 172.21.15.43:0/3842971735 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f84e1d9440 0x55f84e1d7ab0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.782+0000 7fa82e230740 1 --2- 172.21.15.43:0/3842971735 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f84e1d0db0 0x55f84e1f0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.782+0000 7fa82e230740 1 --2- 172.21.15.43:0/3842971735 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f84e1d00d0 0x55f84e1e4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.782+0000 7fa82e230740 1 -- 172.21.15.43:0/3842971735 >> 172.21.15.43:0/3842971735 conn(0x55f84e196a10 msgr2=0x55f84e1d53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:14.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.782+0000 7fa82e230740 1 -- 172.21.15.43:0/3842971735 shutdown_connections 2024-03-28T22:47:14.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.782+0000 7fa82e230740 1 -- 172.21.15.43:0/3842971735 wait complete. 2024-03-28T22:47:14.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.782+0000 7fa82e230740 1 -- start start 2024-03-28T22:47:14.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.782+0000 7fa82e230740 1 Processor -- start 2024-03-28T22:47:14.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.783+0000 7fa82e230740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f84e1d00d0 0x55f84e1e18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.783+0000 7fa82e230740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f84e1d0db0 0x55f84e1d9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.783+0000 7fa82e230740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f84e1d9440 0x55f84e1da560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.783+0000 7fa82e230740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa8241b2890 con 0x55f84e1d0db0 2024-03-28T22:47:14.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.783+0000 7fa82e230740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa8241b21c0 con 0x55f84e1d9440 2024-03-28T22:47:14.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.783+0000 7fa82e230740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fa8241af420 con 0x55f84e1d00d0 2024-03-28T22:47:14.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.783+0000 7fa82e230740 10 client.0 _ll_register_callbacks cb 0x7fa81000b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:47:14.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.783+0000 7fa828c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f84e1d00d0 0x55f84e1e18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.783+0000 7fa828c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f84e1d00d0 0x55f84e1e18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:35408/0 (socket says 172.21.15.43:35408) 2024-03-28T22:47:14.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.783+0000 7fa828c00640 1 -- 172.21.15.43:0/4126165840 learned_addr learned my addr 172.21.15.43:0/4126165840 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:14.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.783+0000 7fa829600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f84e1d9440 0x55f84e1da560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.783+0000 7fa823e00640 1 --2- 172.21.15.43:0/4126165840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f84e1d0db0 0x55f84e1d9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.783+0000 7fa829600640 1 -- 172.21.15.43:0/4126165840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f84e1d00d0 msgr2=0x55f84e1e18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:14.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.783+0000 7fa829600640 1 --2- 172.21.15.43:0/4126165840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f84e1d00d0 0x55f84e1e18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.783+0000 7fa829600640 1 -- 172.21.15.43:0/4126165840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f84e1d0db0 msgr2=0x55f84e1d9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:14.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.783+0000 7fa829600640 1 --2- 172.21.15.43:0/4126165840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f84e1d0db0 0x55f84e1d9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.783+0000 7fa829600640 1 -- 172.21.15.43:0/4126165840 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f84de080f0 con 0x55f84e1d9440 2024-03-28T22:47:14.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.783+0000 7fa828c00640 1 --2- 172.21.15.43:0/4126165840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f84e1d00d0 0x55f84e1e18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:14.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.784+0000 7fa829600640 1 --2- 172.21.15.43:0/4126165840 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f84e1d9440 0x55f84e1da560 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fa81c018040 tx=0x7fa81c01c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=277f3220016b8910 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:14.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.784+0000 7fa80a000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:47:14.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.784+0000 7fa80a000640 1 -- 172.21.15.43:0/4126165840 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81c035020 con 0x55f84e1d9440 2024-03-28T22:47:14.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.784+0000 7fa80a000640 1 -- 172.21.15.43:0/4126165840 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9627+0+0 (secure 0 0 0) 0x7fa81c005090 con 0x55f84e1d9440 2024-03-28T22:47:14.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.784+0000 7fa82e230740 1 -- 172.21.15.43:0/4126165840 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa81c004cc0 con 0x55f84e1d9440 2024-03-28T22:47:14.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.784+0000 7fa80a000640 1 -- 172.21.15.43:0/4126165840 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa81c040050 con 0x55f84e1d9440 2024-03-28T22:47:14.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.784+0000 7fa82e230740 10 client.29075 fetch_fsmap learned FSMap version 32 2024-03-28T22:47:14.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.784+0000 7fa82e230740 1 -- 172.21.15.43:0/4126165840 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f84e24f620 con 0x55f84e1d9440 2024-03-28T22:47:14.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.785+0000 7fa80a000640 1 -- 172.21.15.43:0/4126165840 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa81c004cc0 con 0x55f84e1d9440 2024-03-28T22:47:14.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.785+0000 7fa80a000640 1 -- 172.21.15.43:0/4126165840 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa81c005800 con 0x55f84e1d9440 2024-03-28T22:47:14.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.785+0000 7fa82e230740 10 client.29075 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:47:14.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.785+0000 7fa82e230740 10 client.29075 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:47:14.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.785+0000 7fa82e230740 20 client.29075 populate_metadata read hostname 'smithi043' 2024-03-28T22:47:14.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.785+0000 7fa82e230740 1 --2- 172.21.15.43:0/4126165840 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f84e24fce0 0x55f84e2520d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:14.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.785+0000 7fa82e230740 4 client.29075 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:47:14.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.785+0000 7fa82e230740 1 -- 172.21.15.43:0/4126165840 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f84e13b990 con 0x55f84e24fce0 2024-03-28T22:47:14.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.785+0000 7fa828c00640 1 --2- 172.21.15.43:0/4126165840 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f84e24fce0 0x55f84e2520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:14.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.786+0000 7fa828c00640 1 --2- 172.21.15.43:0/4126165840 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f84e24fce0 0x55f84e2520d0 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0b3df0d3f2576e9 server_cookie=dd35e030834f12ea in_seq=0 out_seq=0 2024-03-28T22:47:14.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.786+0000 7fa80a000640 10 client.29075 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:47:14.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.786+0000 7fa80a000640 1 -- 172.21.15.43:0/4126165840 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f84e13b990 con 0x55f84e24fce0 2024-03-28T22:47:14.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.786+0000 7fa80a000640 10 client.29075 handle_command_reply: tid=0 2024-03-28T22:47:14.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.786+0000 7fa82e230740 1 client.29075 shutdown 2024-03-28T22:47:14.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.786+0000 7fa82e230740 1 -- 172.21.15.43:0/4126165840 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f84e1d9440 msgr2=0x55f84e1da560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:14.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.786+0000 7fa82e230740 1 --2- 172.21.15.43:0/4126165840 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f84e1d9440 0x55f84e1da560 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fa81c018040 tx=0x7fa81c01c970 comp rx=0 tx=0).stop 2024-03-28T22:47:14.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.786+0000 7fa82e230740 1 -- 172.21.15.43:0/4126165840 shutdown_connections 2024-03-28T22:47:14.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.786+0000 7fa82e230740 1 --2- 172.21.15.43:0/4126165840 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55f84e24fce0 0x55f84e2520d0 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.786+0000 7fa82e230740 1 --2- 172.21.15.43:0/4126165840 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f84e1d9440 0x55f84e1da560 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.786+0000 7fa82e230740 1 --2- 172.21.15.43:0/4126165840 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55f84e1d0db0 0x55f84e1d9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.786+0000 7fa82e230740 1 --2- 172.21.15.43:0/4126165840 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55f84e1d00d0 0x55f84e1e18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.786+0000 7fa80a000640 0 client.29075 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:47:14.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.786+0000 7fa82e230740 1 -- 172.21.15.43:0/4126165840 >> 172.21.15.43:0/4126165840 conn(0x55f84e196a10 msgr2=0x55f84e1c9470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:14.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.786+0000 7fa82e230740 1 -- 172.21.15.43:0/4126165840 shutdown_connections 2024-03-28T22:47:14.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.786+0000 7fa82e230740 1 -- 172.21.15.43:0/4126165840 wait complete. 2024-03-28T22:47:14.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.786+0000 7fa82e230740 20 client.29075 trim_cache size 0 max 16384 2024-03-28T22:47:14.792 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:47:14.792 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T22:47:14.792 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": { 2024-03-28T22:47:14.792 INFO:teuthology.orchestra.run.smithi043.stdout: "6a3c741a-6fde-4aff-9b56-98d86f5ad922": { 2024-03-28T22:47:14.792 INFO:teuthology.orchestra.run.smithi043.stdout: "path": "/", 2024-03-28T22:47:14.792 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "6a3c741a-6fde-4aff-9b56-98d86f5ad922", 2024-03-28T22:47:14.792 INFO:teuthology.orchestra.run.smithi043.stdout: "options": "recursive,force" 2024-03-28T22:47:14.793 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:14.793 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:14.793 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:47:14.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.787+0000 7fa82a000640 1 -- 172.21.15.43:0/2851934831 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa80c0a1f70 msgr2=0x7fa80c0a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:14.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.787+0000 7fa82a000640 1 --2- 172.21.15.43:0/2851934831 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa80c0a1f70 0x7fa80c0a4430 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa81c009fd0 tx=0x7fa81c009500 comp rx=0 tx=0).stop 2024-03-28T22:47:14.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.787+0000 7fa82a000640 1 -- 172.21.15.43:0/2851934831 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa8240744a0 msgr2=0x7fa82419a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:14.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.787+0000 7fa82a000640 1 --2- 172.21.15.43:0/2851934831 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa8240744a0 0x7fa82419a370 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fa818005ab0 tx=0x7fa81802c870 comp rx=0 tx=0).stop 2024-03-28T22:47:14.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.787+0000 7fa82a000640 1 -- 172.21.15.43:0/2851934831 shutdown_connections 2024-03-28T22:47:14.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.787+0000 7fa82a000640 1 --2- 172.21.15.43:0/2851934831 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa80c0a1f70 0x7fa80c0a4430 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.787+0000 7fa82a000640 1 --2- 172.21.15.43:0/2851934831 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa82419ebe0 0x7fa82419f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.787+0000 7fa82a000640 1 --2- 172.21.15.43:0/2851934831 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa82410ddf0 0x7fa82419a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.787+0000 7fa82a000640 1 --2- 172.21.15.43:0/2851934831 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa8240744a0 0x7fa82419a370 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.787+0000 7fa82a000640 1 -- 172.21.15.43:0/2851934831 >> 172.21.15.43:0/2851934831 conn(0x7fa8240fd0c0 msgr2=0x7fa8240fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:14.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.787+0000 7fa82a000640 1 -- 172.21.15.43:0/2851934831 shutdown_connections 2024-03-28T22:47:14.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.787+0000 7fa82a000640 1 -- 172.21.15.43:0/2851934831 wait complete. 2024-03-28T22:47:14.801 INFO:tasks.cephfs.filesystem:scrub status for tag:6a3c741a-6fde-4aff-9b56-98d86f5ad922 - {'path': '/', 'tag': '6a3c741a-6fde-4aff-9b56-98d86f5ad922', 'options': 'recursive,force'} 2024-03-28T22:47:14.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.805+0000 7fa214a00640 1 -- 172.21.15.43:0/1089898078 --> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fa210061b90 con 0x7fa1f40a1f90 2024-03-28T22:47:14.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.815+0000 7fa206a00640 1 -- 172.21.15.43:0/1089898078 <== mgr.25663 v2:172.21.15.43:6800/1685312925 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7fa210061b90 con 0x7fa1f40a1f90 2024-03-28T22:47:14.812 INFO:teuthology.orchestra.run.smithi043.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-28T00:00:00", "created": "2024-03-28T22:37:12", "first": "2024-03-28T22:38:00", "last": "2024-03-28T22:47:00", "last_pruned": "2024-03-28T22:47:00", "created_count": 10, "pruned_count": 4, "active": true} 2024-03-28T22:47:14.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.818+0000 7fa214a00640 1 -- 172.21.15.43:0/1089898078 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa1f40a1f90 msgr2=0x7fa1f40a4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:14.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.818+0000 7fa214a00640 1 --2- 172.21.15.43:0/1089898078 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa1f40a1f90 0x7fa1f40a4450 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa200000c00 tx=0x7fa20002df90 comp rx=0 tx=0).stop 2024-03-28T22:47:14.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.818+0000 7fa214a00640 1 -- 172.21.15.43:0/1089898078 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa210069770 msgr2=0x7fa210100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:14.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.818+0000 7fa214a00640 1 --2- 172.21.15.43:0/1089898078 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa210069770 0x7fa210100eb0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc00e9f0 tx=0x7fa1fc00eeb0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.818+0000 7fa214a00640 1 -- 172.21.15.43:0/1089898078 shutdown_connections 2024-03-28T22:47:14.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.818+0000 7fa214a00640 1 --2- 172.21.15.43:0/1089898078 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa1f40a1f90 0x7fa1f40a4450 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.818+0000 7fa214a00640 1 --2- 172.21.15.43:0/1089898078 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa210069770 0x7fa210100eb0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.818+0000 7fa214a00640 1 --2- 172.21.15.43:0/1089898078 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa2101077c0 0x7fa2101013f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.818+0000 7fa214a00640 1 --2- 172.21.15.43:0/1089898078 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa210068dd0 0x7fa210100970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:14.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.818+0000 7fa214a00640 1 -- 172.21.15.43:0/1089898078 >> 172.21.15.43:0/1089898078 conn(0x7fa2100774a0 msgr2=0x7fa2100ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:14.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.818+0000 7fa214a00640 1 -- 172.21.15.43:0/1089898078 shutdown_connections 2024-03-28T22:47:14.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:14.818+0000 7fa214a00640 1 -- 172.21.15.43:0/1089898078 wait complete. 2024-03-28T22:47:14.835 DEBUG:teuthology.orchestra.run.smithi043:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-28T22:47:15.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.098+0000 7f27e0a00640 1 -- 172.21.15.43:0/3360472654 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f27dc068dd0 msgr2=0x7f27dc069230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:15.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.098+0000 7f27e0a00640 1 --2- 172.21.15.43:0/3360472654 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f27dc068dd0 0x7f27dc069230 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f27c8009930 tx=0x7f27c802bc30 comp rx=0 tx=0).stop 2024-03-28T22:47:15.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.098+0000 7f27e0a00640 1 -- 172.21.15.43:0/3360472654 shutdown_connections 2024-03-28T22:47:15.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.098+0000 7f27e0a00640 1 --2- 172.21.15.43:0/3360472654 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f27dc068dd0 0x7f27dc069230 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:15.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.098+0000 7f27e0a00640 1 --2- 172.21.15.43:0/3360472654 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27dc069770 0x7f27dc10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:15.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.098+0000 7f27e0a00640 1 --2- 172.21.15.43:0/3360472654 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27dc107820 0x7f27dc107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:15.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.098+0000 7f27e0a00640 1 -- 172.21.15.43:0/3360472654 >> 172.21.15.43:0/3360472654 conn(0x7f27dc0774a0 msgr2=0x7f27dc0778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:15.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.098+0000 7f27e0a00640 1 -- 172.21.15.43:0/3360472654 shutdown_connections 2024-03-28T22:47:15.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.098+0000 7f27e0a00640 1 -- 172.21.15.43:0/3360472654 wait complete. 2024-03-28T22:47:15.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.099+0000 7f27e0a00640 1 Processor -- start 2024-03-28T22:47:15.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.099+0000 7f27e0a00640 1 -- start start 2024-03-28T22:47:15.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.099+0000 7f27e0a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27dc068dd0 0x7f27dc19a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:15.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.099+0000 7f27e0a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27dc069770 0x7f27dc19a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:15.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.099+0000 7f27e0a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f27dc107820 0x7f27dc19eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:15.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.099+0000 7f27e0a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f27dc111b60 con 0x7f27dc069770 2024-03-28T22:47:15.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.099+0000 7f27e0a00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f27dc1119e0 con 0x7f27dc107820 2024-03-28T22:47:15.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.099+0000 7f27e0a00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f27dc111ce0 con 0x7f27dc068dd0 2024-03-28T22:47:15.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.100+0000 7f27d9e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27dc068dd0 0x7f27dc19a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:15.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.100+0000 7f27d9e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27dc068dd0 0x7f27dc19a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:44692/0 (socket says 172.21.15.43:44692) 2024-03-28T22:47:15.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.100+0000 7f27d9e00640 1 -- 172.21.15.43:0/2928272753 learned_addr learned my addr 172.21.15.43:0/2928272753 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:15.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.100+0000 7f27d9400640 1 --2- 172.21.15.43:0/2928272753 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27dc069770 0x7f27dc19a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:15.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.100+0000 7f27da800640 1 --2- 172.21.15.43:0/2928272753 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f27dc107820 0x7f27dc19eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:15.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.100+0000 7f27d9e00640 1 -- 172.21.15.43:0/2928272753 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f27dc107820 msgr2=0x7f27dc19eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:15.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.100+0000 7f27d9e00640 1 --2- 172.21.15.43:0/2928272753 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f27dc107820 0x7f27dc19eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:15.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.100+0000 7f27d9e00640 1 -- 172.21.15.43:0/2928272753 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27dc069770 msgr2=0x7f27dc19a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:15.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.100+0000 7f27d9e00640 1 --2- 172.21.15.43:0/2928272753 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27dc069770 0x7f27dc19a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:15.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.100+0000 7f27d9e00640 1 -- 172.21.15.43:0/2928272753 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27dc19f230 con 0x7f27dc068dd0 2024-03-28T22:47:15.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.101+0000 7f27d9e00640 1 --2- 172.21.15.43:0/2928272753 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27dc068dd0 0x7f27dc19a240 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f27cc00e9f0 tx=0x7f27cc00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:15.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.101+0000 7f27d2a00640 1 -- 172.21.15.43:0/2928272753 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27cc00cd80 con 0x7f27dc068dd0 2024-03-28T22:47:15.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.101+0000 7f27d2a00640 1 -- 172.21.15.43:0/2928272753 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27cc005500 con 0x7f27dc068dd0 2024-03-28T22:47:15.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.101+0000 7f27d2a00640 1 -- 172.21.15.43:0/2928272753 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27cc010690 con 0x7f27dc068dd0 2024-03-28T22:47:15.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.101+0000 7f27e0a00640 1 -- 172.21.15.43:0/2928272753 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27dc19f520 con 0x7f27dc068dd0 2024-03-28T22:47:15.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.101+0000 7f27e0a00640 1 -- 172.21.15.43:0/2928272753 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27dc1a6db0 con 0x7f27dc068dd0 2024-03-28T22:47:15.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.102+0000 7f27e0a00640 1 -- 172.21.15.43:0/2928272753 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27dc04f4b0 con 0x7f27dc068dd0 2024-03-28T22:47:15.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.104+0000 7f27d2a00640 1 -- 172.21.15.43:0/2928272753 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f27cc0046c0 con 0x7f27dc068dd0 2024-03-28T22:47:15.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.105+0000 7f27d2a00640 1 --2- 172.21.15.43:0/2928272753 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f27c00a1f90 0x7f27c00a4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:15.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.105+0000 7f27d2a00640 1 -- 172.21.15.43:0/2928272753 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9627+0+0 (secure 0 0 0) 0x7f27cc0cea10 con 0x7f27dc068dd0 2024-03-28T22:47:15.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.106+0000 7f27d9400640 1 --2- 172.21.15.43:0/2928272753 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f27c00a1f90 0x7f27c00a4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:15.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.106+0000 7f27d9400640 1 --2- 172.21.15.43:0/2928272753 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f27c00a1f90 0x7f27c00a4450 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f27c8009900 tx=0x7f27c8000bc0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:15.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.109+0000 7f27d2a00640 1 -- 172.21.15.43:0/2928272753 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f27cc09b650 con 0x7f27dc068dd0 2024-03-28T22:47:15.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:15 smithi043 ceph-mon[31823]: pgmap v314: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:47:15.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.334+0000 7f27e0a00640 1 -- 172.21.15.43:0/2928272753 --> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f27dc061b90 con 0x7f27c00a1f90 2024-03-28T22:47:15.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.348+0000 7f27d2a00640 1 -- 172.21.15.43:0/2928272753 <== mgr.25663 v2:172.21.15.43:6800/1685312925 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f27dc061b90 con 0x7f27c00a1f90 2024-03-28T22:47:15.346 INFO:teuthology.orchestra.run.smithi043.stdout:/ 1m 6m3h 2024-03-28T22:47:15.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.351+0000 7f27e0a00640 1 -- 172.21.15.43:0/2928272753 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f27c00a1f90 msgr2=0x7f27c00a4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:15.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.351+0000 7f27e0a00640 1 --2- 172.21.15.43:0/2928272753 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f27c00a1f90 0x7f27c00a4450 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f27c8009900 tx=0x7f27c8000bc0 comp rx=0 tx=0).stop 2024-03-28T22:47:15.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.351+0000 7f27e0a00640 1 -- 172.21.15.43:0/2928272753 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27dc068dd0 msgr2=0x7f27dc19a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:15.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.351+0000 7f27e0a00640 1 --2- 172.21.15.43:0/2928272753 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27dc068dd0 0x7f27dc19a240 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f27cc00e9f0 tx=0x7f27cc00edf0 comp rx=0 tx=0).stop 2024-03-28T22:47:15.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.351+0000 7f27e0a00640 1 -- 172.21.15.43:0/2928272753 shutdown_connections 2024-03-28T22:47:15.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.351+0000 7f27e0a00640 1 --2- 172.21.15.43:0/2928272753 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f27c00a1f90 0x7f27c00a4450 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:15.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.351+0000 7f27e0a00640 1 --2- 172.21.15.43:0/2928272753 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f27dc107820 0x7f27dc19eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:15.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.351+0000 7f27e0a00640 1 --2- 172.21.15.43:0/2928272753 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f27dc069770 0x7f27dc19a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:15.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.351+0000 7f27e0a00640 1 --2- 172.21.15.43:0/2928272753 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f27dc068dd0 0x7f27dc19a240 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:15.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.351+0000 7f27e0a00640 1 -- 172.21.15.43:0/2928272753 >> 172.21.15.43:0/2928272753 conn(0x7f27dc0774a0 msgr2=0x7f27dc10cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:15.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.351+0000 7f27e0a00640 1 -- 172.21.15.43:0/2928272753 shutdown_connections 2024-03-28T22:47:15.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:15.351+0000 7f27e0a00640 1 -- 172.21.15.43:0/2928272753 wait complete. 2024-03-28T22:47:15.367 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-28T22:47:15.378 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-28T22:47:15.445 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:15 smithi172 ceph-mon[36566]: pgmap v314: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:47:15.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:15 smithi082 ceph-mon[36471]: pgmap v314: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:47:16.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:16 smithi082 ceph-mon[36471]: scrub queued for path: / 2024-03-28T22:47:16.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:16 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:47:16.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:16 smithi082 ceph-mon[36471]: scrub summary: active paths [/] 2024-03-28T22:47:16.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:16 smithi082 ceph-mon[36471]: scrub summary: idle+waiting paths [/] 2024-03-28T22:47:16.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:16 smithi082 ceph-mon[36471]: from='client.29057 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:47:16.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:16 smithi043 ceph-mon[31823]: scrub queued for path: / 2024-03-28T22:47:16.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:16 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:47:16.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:16 smithi043 ceph-mon[31823]: scrub summary: active paths [/] 2024-03-28T22:47:16.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:16 smithi043 ceph-mon[31823]: scrub summary: idle+waiting paths [/] 2024-03-28T22:47:16.516 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:16 smithi043 ceph-mon[31823]: from='client.29057 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:47:16.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:16 smithi172 ceph-mon[36566]: scrub queued for path: / 2024-03-28T22:47:16.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:16 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:47:16.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:16 smithi172 ceph-mon[36566]: scrub summary: active paths [/] 2024-03-28T22:47:16.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:16 smithi172 ceph-mon[36566]: scrub summary: idle+waiting paths [/] 2024-03-28T22:47:16.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:16 smithi172 ceph-mon[36566]: from='client.29057 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:47:17.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:17 smithi082 ceph-mon[36471]: from='client.29017 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:47:17.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:17 smithi082 ceph-mon[36471]: pgmap v315: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:47:17.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:17 smithi043 ceph-mon[31823]: from='client.29017 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:47:17.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:17 smithi043 ceph-mon[31823]: pgmap v315: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:47:17.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:17 smithi172 ceph-mon[36566]: from='client.29017 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T22:47:17.645 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:17 smithi172 ceph-mon[36566]: pgmap v315: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:47:19.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:18 smithi172 ceph-mon[36566]: pgmap v316: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 204 B/s wr, 21 op/s 2024-03-28T22:47:19.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:18 smithi172 ceph-mon[36566]: osdmap e108: 12 total, 12 up, 12 in 2024-03-28T22:47:19.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:18 smithi172 ceph-mon[36566]: scrub summary: idle 2024-03-28T22:47:19.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:18 smithi082 ceph-mon[36471]: pgmap v316: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 204 B/s wr, 21 op/s 2024-03-28T22:47:19.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:18 smithi082 ceph-mon[36471]: osdmap e108: 12 total, 12 up, 12 in 2024-03-28T22:47:19.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:18 smithi082 ceph-mon[36471]: scrub summary: idle 2024-03-28T22:47:19.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:18 smithi043 ceph-mon[31823]: pgmap v316: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 204 B/s wr, 21 op/s 2024-03-28T22:47:19.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:18 smithi043 ceph-mon[31823]: osdmap e108: 12 total, 12 up, 12 in 2024-03-28T22:47:19.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:18 smithi043 ceph-mon[31823]: scrub summary: idle 2024-03-28T22:47:21.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:20 smithi172 ceph-mon[36566]: pgmap v318: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 204 B/s wr, 21 op/s 2024-03-28T22:47:21.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:20 smithi082 ceph-mon[36471]: pgmap v318: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 204 B/s wr, 21 op/s 2024-03-28T22:47:21.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:20 smithi043 ceph-mon[31823]: pgmap v318: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 204 B/s wr, 21 op/s 2024-03-28T22:47:23.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:22 smithi172 ceph-mon[36566]: pgmap v319: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 204 B/s wr, 21 op/s 2024-03-28T22:47:23.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:22 smithi082 ceph-mon[36471]: pgmap v319: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 204 B/s wr, 21 op/s 2024-03-28T22:47:23.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:22 smithi043 ceph-mon[31823]: pgmap v319: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 204 B/s wr, 21 op/s 2024-03-28T22:47:25.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:24 smithi172 ceph-mon[36566]: pgmap v320: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 102 B/s wr, 21 op/s 2024-03-28T22:47:25.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:24 smithi082 ceph-mon[36471]: pgmap v320: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 102 B/s wr, 21 op/s 2024-03-28T22:47:25.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:24 smithi043 ceph-mon[31823]: pgmap v320: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 102 B/s wr, 21 op/s 2024-03-28T22:47:27.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:26 smithi172 ceph-mon[36566]: pgmap v321: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 102 B/s wr, 21 op/s 2024-03-28T22:47:27.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:26 smithi082 ceph-mon[36471]: pgmap v321: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 102 B/s wr, 21 op/s 2024-03-28T22:47:27.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:26 smithi043 ceph-mon[31823]: pgmap v321: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 102 B/s wr, 21 op/s 2024-03-28T22:47:29.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:28 smithi172 ceph-mon[36566]: pgmap v322: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-28T22:47:29.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:28 smithi082 ceph-mon[36471]: pgmap v322: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-28T22:47:29.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:28 smithi043 ceph-mon[31823]: pgmap v322: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-28T22:47:31.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:30 smithi172 ceph-mon[36566]: pgmap v323: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:47:31.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:30 smithi082 ceph-mon[36471]: pgmap v323: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:47:31.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:30 smithi043 ceph-mon[31823]: pgmap v323: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:47:33.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:32 smithi172 ceph-mon[36566]: pgmap v324: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-28T22:47:33.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:32 smithi082 ceph-mon[36471]: pgmap v324: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-28T22:47:33.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:32 smithi043 ceph-mon[31823]: pgmap v324: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-28T22:47:35.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:34 smithi172 ceph-mon[36566]: pgmap v325: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:47:35.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:34 smithi082 ceph-mon[36471]: pgmap v325: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:47:35.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:34 smithi043 ceph-mon[31823]: pgmap v325: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:47:37.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:36 smithi172 ceph-mon[36566]: pgmap v326: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-28T22:47:37.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:36 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:47:37.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:36 smithi082 ceph-mon[36471]: pgmap v326: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-28T22:47:37.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:36 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:47:37.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:36 smithi043 ceph-mon[31823]: pgmap v326: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-28T22:47:37.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:36 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:47:39.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:38 smithi043 ceph-mon[31823]: pgmap v327: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:47:39.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:38 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:47:39.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:38 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:47:39.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:38 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:47:39.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:38 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:47:39.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:38 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:39.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:38 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:39.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:38 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:39.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:38 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:39.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:38 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:39.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:38 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:39.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:38 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:39.017 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:38 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:39.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:38 smithi172 ceph-mon[36566]: pgmap v327: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:47:39.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:38 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:47:39.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:38 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:47:39.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:38 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:47:39.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:38 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:47:39.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:38 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:39.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:38 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:39.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:38 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:39.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:38 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:39.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:38 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:39.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:38 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:39.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:38 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:39.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:38 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:39.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:38 smithi082 ceph-mon[36471]: pgmap v327: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:47:39.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:38 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:47:39.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:38 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:47:39.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:38 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:47:39.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:38 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:47:39.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:38 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:39.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:38 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:39.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:38 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:39.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:38 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:39.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:38 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:39.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:38 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:39.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:38 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:39.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:38 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:40.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:39 smithi172 ceph-mon[36566]: Adjusting osd_memory_target on smithi082 to 174.1M 2024-03-28T22:47:40.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:39 smithi172 ceph-mon[36566]: Unable to set osd_memory_target on smithi082 to 182611251: error parsing value: Value '182611251' is below minimum 939524096 2024-03-28T22:47:40.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:39 smithi172 ceph-mon[36566]: Adjusting osd_memory_target on smithi172 to 174.1M 2024-03-28T22:47:40.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:39 smithi172 ceph-mon[36566]: Unable to set osd_memory_target on smithi172 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T22:47:40.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:39 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:47:40.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:39 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:47:40.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:39 smithi082 ceph-mon[36471]: Adjusting osd_memory_target on smithi082 to 174.1M 2024-03-28T22:47:40.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:39 smithi082 ceph-mon[36471]: Unable to set osd_memory_target on smithi082 to 182611251: error parsing value: Value '182611251' is below minimum 939524096 2024-03-28T22:47:40.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:39 smithi082 ceph-mon[36471]: Adjusting osd_memory_target on smithi172 to 174.1M 2024-03-28T22:47:40.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:39 smithi082 ceph-mon[36471]: Unable to set osd_memory_target on smithi172 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T22:47:40.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:39 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:47:40.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:39 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:47:40.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:39 smithi043 ceph-mon[31823]: Adjusting osd_memory_target on smithi082 to 174.1M 2024-03-28T22:47:40.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:39 smithi043 ceph-mon[31823]: Unable to set osd_memory_target on smithi082 to 182611251: error parsing value: Value '182611251' is below minimum 939524096 2024-03-28T22:47:40.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:39 smithi043 ceph-mon[31823]: Adjusting osd_memory_target on smithi172 to 174.1M 2024-03-28T22:47:40.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:39 smithi043 ceph-mon[31823]: Unable to set osd_memory_target on smithi172 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-28T22:47:40.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:39 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:47:40.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:39 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:47:41.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:40 smithi082 ceph-mon[36471]: pgmap v328: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:47:41.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:40 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:41.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:40 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:41.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:40 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:41.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:40 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:41.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:40 smithi082 ceph-mon[36471]: Adjusting osd_memory_target on smithi043 to 177.8M 2024-03-28T22:47:41.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:40 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:47:41.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:40 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:47:41.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:40 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:47:41.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:40 smithi043 ceph-mon[31823]: pgmap v328: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:47:41.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:40 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:41.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:40 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:41.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:40 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:41.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:40 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:41.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:40 smithi043 ceph-mon[31823]: Adjusting osd_memory_target on smithi043 to 177.8M 2024-03-28T22:47:41.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:40 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:47:41.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:40 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:47:41.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:40 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:47:41.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:40 smithi172 ceph-mon[36566]: pgmap v328: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:47:41.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:40 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:41.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:40 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:41.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:40 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:41.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:40 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T22:47:41.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:40 smithi172 ceph-mon[36566]: Adjusting osd_memory_target on smithi043 to 177.8M 2024-03-28T22:47:41.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:40 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:47:41.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:40 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:47:41.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:40 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:47:42.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:41 smithi082 ceph-mon[36471]: Unable to set osd_memory_target on smithi043 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T22:47:42.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:41 smithi043 ceph-mon[31823]: Unable to set osd_memory_target on smithi043 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T22:47:42.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:41 smithi172 ceph-mon[36566]: Unable to set osd_memory_target on smithi043 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-28T22:47:43.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:42 smithi082 ceph-mon[36471]: pgmap v329: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:47:43.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:42 smithi043 ceph-mon[31823]: pgmap v329: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:47:43.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:42 smithi172 ceph-mon[36566]: pgmap v329: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:47:44.802 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T22:47:44.802 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T22:47:45.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.047+0000 7fb792000640 1 -- 172.21.15.43:0/2207194162 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb78c107740 msgr2=0x7fb78c107b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:45.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.047+0000 7fb792000640 1 --2- 172.21.15.43:0/2207194162 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb78c107740 0x7fb78c107b20 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fb77c009900 tx=0x7fb77c02bcd0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.048+0000 7fb792000640 1 -- 172.21.15.43:0/2207194162 shutdown_connections 2024-03-28T22:47:45.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.048+0000 7fb792000640 1 --2- 172.21.15.43:0/2207194162 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb78c06a6c0 0x7fb78c10e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.048+0000 7fb792000640 1 --2- 172.21.15.43:0/2207194162 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb78c069d20 0x7fb78c06a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.048+0000 7fb792000640 1 --2- 172.21.15.43:0/2207194162 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb78c107740 0x7fb78c107b20 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.048+0000 7fb792000640 1 -- 172.21.15.43:0/2207194162 >> 172.21.15.43:0/2207194162 conn(0x7fb78c0776e0 msgr2=0x7fb78c077af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:45.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.048+0000 7fb792000640 1 -- 172.21.15.43:0/2207194162 shutdown_connections 2024-03-28T22:47:45.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.048+0000 7fb792000640 1 -- 172.21.15.43:0/2207194162 wait complete. 2024-03-28T22:47:45.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.049+0000 7fb792000640 1 Processor -- start 2024-03-28T22:47:45.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.049+0000 7fb792000640 1 -- start start 2024-03-28T22:47:45.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.050+0000 7fb792000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb78c069d20 0x7fb78c198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.050+0000 7fb792000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb78c06a6c0 0x7fb78c1985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.050+0000 7fb792000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb78c107740 0x7fb78c19c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.050+0000 7fb792000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb78c111b50 con 0x7fb78c06a6c0 2024-03-28T22:47:45.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.050+0000 7fb792000640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb78c1119d0 con 0x7fb78c107740 2024-03-28T22:47:45.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.050+0000 7fb792000640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fb78c111cd0 con 0x7fb78c069d20 2024-03-28T22:47:45.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.050+0000 7fb78b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb78c069d20 0x7fb78c198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.050+0000 7fb78b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb78c069d20 0x7fb78c198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:34144/0 (socket says 172.21.15.43:34144) 2024-03-28T22:47:45.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.050+0000 7fb78b400640 1 -- 172.21.15.43:0/3560936777 learned_addr learned my addr 172.21.15.43:0/3560936777 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:45.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.050+0000 7fb78aa00640 1 --2- 172.21.15.43:0/3560936777 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb78c06a6c0 0x7fb78c1985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.050+0000 7fb78be00640 1 --2- 172.21.15.43:0/3560936777 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb78c107740 0x7fb78c19c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.050+0000 7fb78b400640 1 -- 172.21.15.43:0/3560936777 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb78c107740 msgr2=0x7fb78c19c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:45.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.050+0000 7fb78b400640 1 --2- 172.21.15.43:0/3560936777 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb78c107740 0x7fb78c19c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.050+0000 7fb78b400640 1 -- 172.21.15.43:0/3560936777 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb78c06a6c0 msgr2=0x7fb78c1985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:45.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.050+0000 7fb78b400640 1 --2- 172.21.15.43:0/3560936777 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb78c06a6c0 0x7fb78c1985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.050+0000 7fb78b400640 1 -- 172.21.15.43:0/3560936777 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb78c19d080 con 0x7fb78c069d20 2024-03-28T22:47:45.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.050+0000 7fb78be00640 1 --2- 172.21.15.43:0/3560936777 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb78c107740 0x7fb78c19c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:45.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.051+0000 7fb78aa00640 1 --2- 172.21.15.43:0/3560936777 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb78c06a6c0 0x7fb78c1985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:47:45.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.051+0000 7fb78b400640 1 --2- 172.21.15.43:0/3560936777 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb78c069d20 0x7fb78c198090 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fb77c000c00 tx=0x7fb77c004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:45.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.051+0000 7fb783e00640 1 -- 172.21.15.43:0/3560936777 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77c042020 con 0x7fb78c069d20 2024-03-28T22:47:45.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.051+0000 7fb792000640 1 -- 172.21.15.43:0/3560936777 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb78c19d310 con 0x7fb78c069d20 2024-03-28T22:47:45.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.051+0000 7fb783e00640 1 -- 172.21.15.43:0/3560936777 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb77c0054e0 con 0x7fb78c069d20 2024-03-28T22:47:45.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.051+0000 7fb783e00640 1 -- 172.21.15.43:0/3560936777 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77c003a50 con 0x7fb78c069d20 2024-03-28T22:47:45.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.051+0000 7fb792000640 1 -- 172.21.15.43:0/3560936777 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb78c1a4c00 con 0x7fb78c069d20 2024-03-28T22:47:45.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.054+0000 7fb783e00640 1 -- 172.21.15.43:0/3560936777 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fb77c045020 con 0x7fb78c069d20 2024-03-28T22:47:45.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.055+0000 7fb794c25740 1 Processor -- start 2024-03-28T22:47:45.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.055+0000 7fb794c25740 1 -- start start 2024-03-28T22:47:45.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.055+0000 7fb794c25740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3abf69bc0 0x55b3abf68910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.055+0000 7fb794c25740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3abf68ee0 0x55b3abf69360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.055+0000 7fb794c25740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3abf72250 0x55b3abf74640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.055+0000 7fb794c25740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b3abecd100 con 0x55b3abf72250 2024-03-28T22:47:45.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.055+0000 7fb794c25740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55b3abe75ba0 con 0x55b3abf68ee0 2024-03-28T22:47:45.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.055+0000 7fb794c25740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55b3abec91a0 con 0x55b3abf69bc0 2024-03-28T22:47:45.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.055+0000 7fb78b400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3abf68ee0 0x55b3abf69360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.055+0000 7fb78be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3abf69bc0 0x55b3abf68910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.055+0000 7fb78be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3abf69bc0 0x55b3abf68910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:34154/0 (socket says 172.21.15.43:34154) 2024-03-28T22:47:45.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.055+0000 7fb78b400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3abf68ee0 0x55b3abf69360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:58828/0 (socket says 172.21.15.43:58828) 2024-03-28T22:47:45.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.055+0000 7fb78be00640 1 -- 172.21.15.43:0/404139428 learned_addr learned my addr 172.21.15.43:0/404139428 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:45.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.056+0000 7fb78aa00640 1 --2- 172.21.15.43:0/404139428 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3abf72250 0x55b3abf74640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.056+0000 7fb78b400640 1 -- 172.21.15.43:0/404139428 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3abf69bc0 msgr2=0x55b3abf68910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:45.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.056+0000 7fb78b400640 1 --2- 172.21.15.43:0/404139428 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3abf69bc0 0x55b3abf68910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.056+0000 7fb78b400640 1 -- 172.21.15.43:0/404139428 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3abf72250 msgr2=0x55b3abf74640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:45.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.056+0000 7fb78b400640 1 --2- 172.21.15.43:0/404139428 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3abf72250 0x55b3abf74640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.056+0000 7fb78b400640 1 -- 172.21.15.43:0/404139428 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3abea61a0 con 0x55b3abf68ee0 2024-03-28T22:47:45.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.056+0000 7fb78be00640 1 --2- 172.21.15.43:0/404139428 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3abf69bc0 0x55b3abf68910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:45.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.056+0000 7fb78b400640 1 --2- 172.21.15.43:0/404139428 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3abf68ee0 0x55b3abf69360 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fb77c0ed4e0 tx=0x7fb77c0fa4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da444130051b623 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:45.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.056+0000 7fb783e00640 1 --2- 172.21.15.43:0/3560936777 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb7700a1ef0 0x7fb7700a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.056+0000 7fb777e00640 1 -- 172.21.15.43:0/404139428 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77c0bbbd0 con 0x55b3abf68ee0 2024-03-28T22:47:45.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.056+0000 7fb777e00640 1 -- 172.21.15.43:0/404139428 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb77c0bce80 con 0x55b3abf68ee0 2024-03-28T22:47:45.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.056+0000 7fb777e00640 1 -- 172.21.15.43:0/404139428 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77c0bbf00 con 0x55b3abf68ee0 2024-03-28T22:47:45.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.057+0000 7fb783e00640 1 -- 172.21.15.43:0/3560936777 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9539+0+0 (secure 0 0 0) 0x7fb77c0f00e0 con 0x7fb78c069d20 2024-03-28T22:47:45.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.057+0000 7fb78be00640 1 --2- 172.21.15.43:0/3560936777 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb7700a1ef0 0x7fb7700a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.057+0000 7fb794c25740 1 -- 172.21.15.43:0/404139428 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3abf68ee0 msgr2=0x55b3abf69360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:45.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.057+0000 7fb794c25740 1 --2- 172.21.15.43:0/404139428 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3abf68ee0 0x55b3abf69360 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fb77c0ed4e0 tx=0x7fb77c0fa4c0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.057+0000 7fb794c25740 1 -- 172.21.15.43:0/404139428 shutdown_connections 2024-03-28T22:47:45.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.057+0000 7fb794c25740 1 --2- 172.21.15.43:0/404139428 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3abf68ee0 0x55b3abf69360 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.057+0000 7fb794c25740 1 --2- 172.21.15.43:0/404139428 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3abf72250 0x55b3abf74640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.057+0000 7fb794c25740 1 --2- 172.21.15.43:0/404139428 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3abf69bc0 0x55b3abf68910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.057+0000 7fb794c25740 1 -- 172.21.15.43:0/404139428 >> 172.21.15.43:0/404139428 conn(0x55b3abf2fa10 msgr2=0x55b3abf63930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:45.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.057+0000 7fb78be00640 1 --2- 172.21.15.43:0/3560936777 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb7700a1ef0 0x7fb7700a43b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb784012bb0 tx=0x7fb784009500 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:45.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.057+0000 7fb794c25740 1 -- 172.21.15.43:0/404139428 shutdown_connections 2024-03-28T22:47:45.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.057+0000 7fb794c25740 1 -- 172.21.15.43:0/404139428 wait complete. 2024-03-28T22:47:45.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.058+0000 7fb794c25740 1 -- start start 2024-03-28T22:47:45.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.058+0000 7fb794c25740 1 Processor -- start 2024-03-28T22:47:45.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.058+0000 7fb794c25740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3abf68ee0 0x55b3abf6f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.059+0000 7fb794c25740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3abf69bc0 0x55b3abf6f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.059+0000 7fb794c25740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3abf72250 0x55b3abf6fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.059+0000 7fb794c25740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55b3abf74e80 con 0x55b3abf69bc0 2024-03-28T22:47:45.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.059+0000 7fb794c25740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55b3abe6b250 con 0x55b3abf68ee0 2024-03-28T22:47:45.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.059+0000 7fb794c25740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55b3abf75000 con 0x55b3abf72250 2024-03-28T22:47:45.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.059+0000 7fb794c25740 10 client.0 _ll_register_callbacks cb 0x55b3abbe0be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:47:45.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.059+0000 7fb78b400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3abf68ee0 0x55b3abf6f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.059+0000 7fb78b400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3abf68ee0 0x55b3abf6f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:58838/0 (socket says 172.21.15.43:58838) 2024-03-28T22:47:45.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.059+0000 7fb78b400640 1 -- 172.21.15.43:0/2040219234 learned_addr learned my addr 172.21.15.43:0/2040219234 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:45.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.059+0000 7fb78be00640 1 --2- 172.21.15.43:0/2040219234 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3abf72250 0x55b3abf6fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.059+0000 7fb78aa00640 1 --2- 172.21.15.43:0/2040219234 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3abf69bc0 0x55b3abf6f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.059+0000 7fb78b400640 1 -- 172.21.15.43:0/2040219234 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3abf72250 msgr2=0x55b3abf6fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:45.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.059+0000 7fb78b400640 1 --2- 172.21.15.43:0/2040219234 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3abf72250 0x55b3abf6fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.059+0000 7fb78b400640 1 -- 172.21.15.43:0/2040219234 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3abf69bc0 msgr2=0x55b3abf6f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:45.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.059+0000 7fb78b400640 1 --2- 172.21.15.43:0/2040219234 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3abf69bc0 0x55b3abf6f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.059+0000 7fb78b400640 1 -- 172.21.15.43:0/2040219234 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3abe87960 con 0x55b3abf68ee0 2024-03-28T22:47:45.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.059+0000 7fb78aa00640 1 --2- 172.21.15.43:0/2040219234 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3abf69bc0 0x55b3abf6f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:47:45.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.059+0000 7fb78b400640 1 --2- 172.21.15.43:0/2040219234 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3abf68ee0 0x55b3abf6f210 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fb77c0bc710 tx=0x7fb77c005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47bf742235c64805 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:45.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.059+0000 7fb774c00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:47:45.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.060+0000 7fb774c00640 1 -- 172.21.15.43:0/2040219234 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77c0bc0e0 con 0x55b3abf68ee0 2024-03-28T22:47:45.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.060+0000 7fb774c00640 1 -- 172.21.15.43:0/2040219234 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9539+0+0 (secure 0 0 0) 0x7fb77c004c20 con 0x55b3abf68ee0 2024-03-28T22:47:45.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.060+0000 7fb794c25740 1 -- 172.21.15.43:0/2040219234 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb77c0bce80 con 0x55b3abf68ee0 2024-03-28T22:47:45.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.060+0000 7fb774c00640 1 -- 172.21.15.43:0/2040219234 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb77c0bc2c0 con 0x55b3abf68ee0 2024-03-28T22:47:45.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.060+0000 7fb794c25740 10 client.29093 fetch_fsmap learned FSMap version 32 2024-03-28T22:47:45.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.060+0000 7fb794c25740 1 -- 172.21.15.43:0/2040219234 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b3abcab7d0 con 0x55b3abf68ee0 2024-03-28T22:47:45.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.060+0000 7fb774c00640 1 -- 172.21.15.43:0/2040219234 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb77c0bce80 con 0x55b3abf68ee0 2024-03-28T22:47:45.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.060+0000 7fb774c00640 1 -- 172.21.15.43:0/2040219234 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb77c0b8040 con 0x55b3abf68ee0 2024-03-28T22:47:45.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.060+0000 7fb794c25740 10 client.29093 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:47:45.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.060+0000 7fb794c25740 10 client.29093 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:47:45.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.060+0000 7fb794c25740 20 client.29093 populate_metadata read hostname 'smithi043' 2024-03-28T22:47:45.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.060+0000 7fb794c25740 1 --2- 172.21.15.43:0/2040219234 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b3abf82240 0x55b3abf84630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.060+0000 7fb794c25740 4 client.29093 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:47:45.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.060+0000 7fb794c25740 1 -- 172.21.15.43:0/2040219234 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b3abeda640 con 0x55b3abf82240 2024-03-28T22:47:45.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.061+0000 7fb78aa00640 1 --2- 172.21.15.43:0/2040219234 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b3abf82240 0x55b3abf84630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.061+0000 7fb78aa00640 1 --2- 172.21.15.43:0/2040219234 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b3abf82240 0x55b3abf84630 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c847aee293403c2 server_cookie=72b3553d6c76da2d in_seq=0 out_seq=0 2024-03-28T22:47:45.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.061+0000 7fb774c00640 10 client.29093 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:47:45.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.062+0000 7fb774c00640 1 -- 172.21.15.43:0/2040219234 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b3abeda640 con 0x55b3abf82240 2024-03-28T22:47:45.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.062+0000 7fb774c00640 10 client.29093 handle_command_reply: tid=0 2024-03-28T22:47:45.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.062+0000 7fb794c25740 1 client.29093 shutdown 2024-03-28T22:47:45.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.062+0000 7fb794c25740 1 -- 172.21.15.43:0/2040219234 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3abf68ee0 msgr2=0x55b3abf6f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:45.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.062+0000 7fb794c25740 1 --2- 172.21.15.43:0/2040219234 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3abf68ee0 0x55b3abf6f210 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fb77c0bc710 tx=0x7fb77c005d50 comp rx=0 tx=0).stop 2024-03-28T22:47:45.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.062+0000 7fb794c25740 1 -- 172.21.15.43:0/2040219234 shutdown_connections 2024-03-28T22:47:45.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.062+0000 7fb794c25740 1 --2- 172.21.15.43:0/2040219234 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3abf72250 0x55b3abf6fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.062+0000 7fb794c25740 1 --2- 172.21.15.43:0/2040219234 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3abf69bc0 0x55b3abf6f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.062+0000 7fb794c25740 1 --2- 172.21.15.43:0/2040219234 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b3abf82240 0x55b3abf84630 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.062+0000 7fb794c25740 1 --2- 172.21.15.43:0/2040219234 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3abf68ee0 0x55b3abf6f210 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.062+0000 7fb794c25740 1 -- 172.21.15.43:0/2040219234 >> 172.21.15.43:0/2040219234 conn(0x55b3abf2fa10 msgr2=0x55b3abf65680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:45.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.062+0000 7fb774c00640 0 client.29093 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:47:45.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.062+0000 7fb794c25740 1 -- 172.21.15.43:0/2040219234 shutdown_connections 2024-03-28T22:47:45.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.062+0000 7fb794c25740 1 -- 172.21.15.43:0/2040219234 wait complete. 2024-03-28T22:47:45.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.062+0000 7fb794c25740 20 client.29093 trim_cache size 0 max 16384 2024-03-28T22:47:45.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.082+0000 7fb794c25740 1 Processor -- start 2024-03-28T22:47:45.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.082+0000 7fb794c25740 1 -- start start 2024-03-28T22:47:45.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.082+0000 7fb794c25740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3abf68ee0 0x55b3abf71940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.082+0000 7fb794c25740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3abf69bc0 0x55b3abf6e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.082+0000 7fb794c25740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3abf72250 0x55b3abf63460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.082+0000 7fb794c25740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb78c1acb30 con 0x55b3abf68ee0 2024-03-28T22:47:45.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.082+0000 7fb794c25740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb78c1ae670 con 0x55b3abf72250 2024-03-28T22:47:45.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.082+0000 7fb794c25740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fb78c1ad200 con 0x55b3abf69bc0 2024-03-28T22:47:45.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.082+0000 7fb78be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3abf72250 0x55b3abf63460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.082+0000 7fb78b400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3abf68ee0 0x55b3abf71940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.082+0000 7fb78be00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3abf72250 0x55b3abf63460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:58840/0 (socket says 172.21.15.43:58840) 2024-03-28T22:47:45.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.082+0000 7fb78be00640 1 -- 172.21.15.43:0/2360128926 learned_addr learned my addr 172.21.15.43:0/2360128926 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:45.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.082+0000 7fb78aa00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3abf69bc0 0x55b3abf6e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.083+0000 7fb78be00640 1 -- 172.21.15.43:0/2360128926 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3abf69bc0 msgr2=0x55b3abf6e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:45.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.083+0000 7fb78be00640 1 --2- 172.21.15.43:0/2360128926 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3abf69bc0 0x55b3abf6e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.083+0000 7fb78be00640 1 -- 172.21.15.43:0/2360128926 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3abf68ee0 msgr2=0x55b3abf71940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:45.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.083+0000 7fb78be00640 1 --2- 172.21.15.43:0/2360128926 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3abf68ee0 0x55b3abf71940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.083+0000 7fb78be00640 1 -- 172.21.15.43:0/2360128926 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3abbe13f0 con 0x55b3abf72250 2024-03-28T22:47:45.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.083+0000 7fb78be00640 1 --2- 172.21.15.43:0/2360128926 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3abf72250 0x55b3abf63460 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fb7840120a0 tx=0x7fb78401a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ead6c99ecb462656 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:45.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.083+0000 7fb777400640 1 -- 172.21.15.43:0/2360128926 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7840119c0 con 0x55b3abf72250 2024-03-28T22:47:45.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.084+0000 7fb777400640 1 -- 172.21.15.43:0/2360128926 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb784003e50 con 0x55b3abf72250 2024-03-28T22:47:45.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.084+0000 7fb777400640 1 -- 172.21.15.43:0/2360128926 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7840047c0 con 0x55b3abf72250 2024-03-28T22:47:45.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.084+0000 7fb794c25740 1 -- 172.21.15.43:0/2360128926 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3abf72250 msgr2=0x55b3abf63460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:45.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.084+0000 7fb794c25740 1 --2- 172.21.15.43:0/2360128926 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3abf72250 0x55b3abf63460 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fb7840120a0 tx=0x7fb78401a000 comp rx=0 tx=0).stop 2024-03-28T22:47:45.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.084+0000 7fb794c25740 1 -- 172.21.15.43:0/2360128926 shutdown_connections 2024-03-28T22:47:45.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.084+0000 7fb794c25740 1 --2- 172.21.15.43:0/2360128926 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3abf72250 0x55b3abf63460 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.084+0000 7fb794c25740 1 --2- 172.21.15.43:0/2360128926 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3abf69bc0 0x55b3abf6e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.084+0000 7fb794c25740 1 --2- 172.21.15.43:0/2360128926 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3abf68ee0 0x55b3abf71940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.084+0000 7fb794c25740 1 -- 172.21.15.43:0/2360128926 >> 172.21.15.43:0/2360128926 conn(0x55b3abf2fa10 msgr2=0x55b3abf62ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:45.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.084+0000 7fb794c25740 1 -- 172.21.15.43:0/2360128926 shutdown_connections 2024-03-28T22:47:45.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.084+0000 7fb794c25740 1 -- 172.21.15.43:0/2360128926 wait complete. 2024-03-28T22:47:45.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.085+0000 7fb794c25740 1 -- start start 2024-03-28T22:47:45.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.085+0000 7fb794c25740 1 Processor -- start 2024-03-28T22:47:45.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.085+0000 7fb794c25740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3abf68ee0 0x55b3abf75c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.085+0000 7fb794c25740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3abf69bc0 0x55b3abf761c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.085+0000 7fb794c25740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3abf72250 0x55b3abf76700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.085+0000 7fb794c25740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb78c1adfa0 con 0x55b3abf72250 2024-03-28T22:47:45.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.085+0000 7fb794c25740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb78c1ad8d0 con 0x55b3abf69bc0 2024-03-28T22:47:45.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.085+0000 7fb794c25740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fb78c1aab10 con 0x55b3abf68ee0 2024-03-28T22:47:45.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.085+0000 7fb794c25740 10 client.0 _ll_register_callbacks cb 0x7fb784009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:47:45.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.086+0000 7fb78aa00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3abf69bc0 0x55b3abf761c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.086+0000 7fb78aa00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3abf69bc0 0x55b3abf761c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:58854/0 (socket says 172.21.15.43:58854) 2024-03-28T22:47:45.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.086+0000 7fb78b400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3abf68ee0 0x55b3abf75c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.086+0000 7fb78aa00640 1 -- 172.21.15.43:0/3972976612 learned_addr learned my addr 172.21.15.43:0/3972976612 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:45.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.086+0000 7fb78be00640 1 --2- 172.21.15.43:0/3972976612 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3abf72250 0x55b3abf76700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.086+0000 7fb78aa00640 1 -- 172.21.15.43:0/3972976612 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3abf68ee0 msgr2=0x55b3abf75c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:45.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.086+0000 7fb78aa00640 1 --2- 172.21.15.43:0/3972976612 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3abf68ee0 0x55b3abf75c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.086+0000 7fb78aa00640 1 -- 172.21.15.43:0/3972976612 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3abf72250 msgr2=0x55b3abf76700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:45.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.086+0000 7fb78aa00640 1 --2- 172.21.15.43:0/3972976612 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3abf72250 0x55b3abf76700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.086+0000 7fb78aa00640 1 -- 172.21.15.43:0/3972976612 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3abba10f0 con 0x55b3abf69bc0 2024-03-28T22:47:45.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.086+0000 7fb78be00640 1 --2- 172.21.15.43:0/3972976612 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3abf72250 0x55b3abf76700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:47:45.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.086+0000 7fb78aa00640 1 --2- 172.21.15.43:0/3972976612 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3abf69bc0 0x55b3abf761c0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fb778005d00 tx=0x7fb778014840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ce91d33c5558358 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:45.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.086+0000 7fb774c00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:47:45.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.086+0000 7fb774c00640 1 -- 172.21.15.43:0/3972976612 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7780142d0 con 0x55b3abf69bc0 2024-03-28T22:47:45.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.086+0000 7fb794c25740 1 -- 172.21.15.43:0/3972976612 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb784003e50 con 0x55b3abf69bc0 2024-03-28T22:47:45.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.086+0000 7fb774c00640 1 -- 172.21.15.43:0/3972976612 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9539+0+0 (secure 0 0 0) 0x7fb778004030 con 0x55b3abf69bc0 2024-03-28T22:47:45.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.087+0000 7fb774c00640 1 -- 172.21.15.43:0/3972976612 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb77802f050 con 0x55b3abf69bc0 2024-03-28T22:47:45.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.087+0000 7fb794c25740 10 client.29099 fetch_fsmap learned FSMap version 32 2024-03-28T22:47:45.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.087+0000 7fb794c25740 1 -- 172.21.15.43:0/3972976612 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b3abfe86f0 con 0x55b3abf69bc0 2024-03-28T22:47:45.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.087+0000 7fb774c00640 1 -- 172.21.15.43:0/3972976612 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb784003e50 con 0x55b3abf69bc0 2024-03-28T22:47:45.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.087+0000 7fb774c00640 1 -- 172.21.15.43:0/3972976612 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb778003960 con 0x55b3abf69bc0 2024-03-28T22:47:45.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.087+0000 7fb794c25740 10 client.29099 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:47:45.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.087+0000 7fb794c25740 10 client.29099 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:47:45.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.087+0000 7fb794c25740 20 client.29099 populate_metadata read hostname 'smithi043' 2024-03-28T22:47:45.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.087+0000 7fb794c25740 1 --2- 172.21.15.43:0/3972976612 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b3abfe8e50 0x55b3abfeb270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.087+0000 7fb794c25740 4 client.29099 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T22:47:45.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.087+0000 7fb794c25740 1 -- 172.21.15.43:0/3972976612 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b3abed4990 con 0x55b3abfe8e50 2024-03-28T22:47:45.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.087+0000 7fb78b400640 1 --2- 172.21.15.43:0/3972976612 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b3abfe8e50 0x55b3abfeb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.088+0000 7fb78b400640 1 --2- 172.21.15.43:0/3972976612 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b3abfe8e50 0x55b3abfeb270 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8790c359c3d19af8 server_cookie=feb55a706db78d6c in_seq=0 out_seq=0 2024-03-28T22:47:45.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.088+0000 7fb774c00640 10 client.29099 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:47:45.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.088+0000 7fb774c00640 1 -- 172.21.15.43:0/3972976612 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b3abed4990 con 0x55b3abfe8e50 2024-03-28T22:47:45.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.088+0000 7fb774c00640 10 client.29099 handle_command_reply: tid=0 2024-03-28T22:47:45.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.088+0000 7fb794c25740 1 client.29099 shutdown 2024-03-28T22:47:45.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.088+0000 7fb794c25740 1 -- 172.21.15.43:0/3972976612 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3abf69bc0 msgr2=0x55b3abf761c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:45.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.088+0000 7fb794c25740 1 --2- 172.21.15.43:0/3972976612 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3abf69bc0 0x55b3abf761c0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fb778005d00 tx=0x7fb778014840 comp rx=0 tx=0).stop 2024-03-28T22:47:45.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.088+0000 7fb794c25740 1 -- 172.21.15.43:0/3972976612 shutdown_connections 2024-03-28T22:47:45.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.088+0000 7fb794c25740 1 --2- 172.21.15.43:0/3972976612 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x55b3abfe8e50 0x55b3abfeb270 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.088+0000 7fb794c25740 1 --2- 172.21.15.43:0/3972976612 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b3abf69bc0 0x55b3abf761c0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.088+0000 7fb794c25740 1 --2- 172.21.15.43:0/3972976612 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55b3abf72250 0x55b3abf76700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.088+0000 7fb794c25740 1 --2- 172.21.15.43:0/3972976612 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55b3abf68ee0 0x55b3abf75c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.088+0000 7fb774c00640 0 client.29099 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:47:45.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.088+0000 7fb794c25740 1 -- 172.21.15.43:0/3972976612 >> 172.21.15.43:0/3972976612 conn(0x55b3abf2fa10 msgr2=0x55b3abf6f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:45.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.089+0000 7fb794c25740 1 -- 172.21.15.43:0/3972976612 shutdown_connections 2024-03-28T22:47:45.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.089+0000 7fb794c25740 1 -- 172.21.15.43:0/3972976612 wait complete. 2024-03-28T22:47:45.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.089+0000 7fb794c25740 20 client.29099 trim_cache size 0 max 16384 2024-03-28T22:47:45.095 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:47:45.096 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "no active scrubs running", 2024-03-28T22:47:45.096 INFO:teuthology.orchestra.run.smithi043.stdout: "scrubs": {} 2024-03-28T22:47:45.096 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:47:45.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.089+0000 7fb792000640 1 -- 172.21.15.43:0/3560936777 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb7700a1ef0 msgr2=0x7fb7700a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:45.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.089+0000 7fb792000640 1 --2- 172.21.15.43:0/3560936777 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb7700a1ef0 0x7fb7700a43b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb784012bb0 tx=0x7fb784009500 comp rx=0 tx=0).stop 2024-03-28T22:47:45.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.089+0000 7fb792000640 1 -- 172.21.15.43:0/3560936777 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb78c069d20 msgr2=0x7fb78c198090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:45.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.089+0000 7fb792000640 1 --2- 172.21.15.43:0/3560936777 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb78c069d20 0x7fb78c198090 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fb77c000c00 tx=0x7fb77c004360 comp rx=0 tx=0).stop 2024-03-28T22:47:45.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.089+0000 7fb792000640 1 -- 172.21.15.43:0/3560936777 shutdown_connections 2024-03-28T22:47:45.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.089+0000 7fb792000640 1 --2- 172.21.15.43:0/3560936777 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fb7700a1ef0 0x7fb7700a43b0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.089+0000 7fb792000640 1 --2- 172.21.15.43:0/3560936777 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb78c107740 0x7fb78c19c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.089+0000 7fb792000640 1 --2- 172.21.15.43:0/3560936777 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb78c06a6c0 0x7fb78c1985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.089+0000 7fb792000640 1 --2- 172.21.15.43:0/3560936777 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fb78c069d20 0x7fb78c198090 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.089+0000 7fb792000640 1 -- 172.21.15.43:0/3560936777 >> 172.21.15.43:0/3560936777 conn(0x7fb78c0776e0 msgr2=0x7fb78c0ff290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:45.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.090+0000 7fb792000640 1 -- 172.21.15.43:0/3560936777 shutdown_connections 2024-03-28T22:47:45.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.090+0000 7fb792000640 1 -- 172.21.15.43:0/3560936777 wait complete. 2024-03-28T22:47:45.107 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T22:47:45.108 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:47:45.187 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:44 smithi043 ceph-mon[31823]: pgmap v330: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:47:45.197 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:44 smithi172 ceph-mon[36566]: pgmap v330: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:47:45.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:44 smithi082 ceph-mon[36471]: pgmap v330: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:47:45.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.330+0000 7f6daba00640 1 -- 172.21.15.43:0/2041973771 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6da4108920 msgr2=0x7f6da410c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:45.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.330+0000 7f6daba00640 1 --2- 172.21.15.43:0/2041973771 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6da4108920 0x7f6da410c960 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f6d90009900 tx=0x7f6d9002bc40 comp rx=0 tx=0).stop 2024-03-28T22:47:45.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.330+0000 7f6daba00640 1 -- 172.21.15.43:0/2041973771 shutdown_connections 2024-03-28T22:47:45.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.330+0000 7f6daba00640 1 --2- 172.21.15.43:0/2041973771 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6da4108920 0x7f6da410c960 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.330+0000 7f6daba00640 1 --2- 172.21.15.43:0/2041973771 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6da410d040 0x7f6da4114c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.330+0000 7f6daba00640 1 --2- 172.21.15.43:0/2041973771 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6da4107f70 0x7f6da4108350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.330+0000 7f6daba00640 1 -- 172.21.15.43:0/2041973771 >> 172.21.15.43:0/2041973771 conn(0x7f6da4075ca0 msgr2=0x7f6da40780c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:45.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.330+0000 7f6daba00640 1 -- 172.21.15.43:0/2041973771 shutdown_connections 2024-03-28T22:47:45.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.330+0000 7f6daba00640 1 -- 172.21.15.43:0/2041973771 wait complete. 2024-03-28T22:47:45.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.331+0000 7f6daba00640 1 Processor -- start 2024-03-28T22:47:45.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.331+0000 7f6daba00640 1 -- start start 2024-03-28T22:47:45.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.331+0000 7f6daba00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6da4107f70 0x7f6da419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.331+0000 7f6daba00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6da410d040 0x7f6da419ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.331+0000 7f6daba00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6da41a2f10 0x7f6da41a33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.331+0000 7f6daba00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6da4118120 con 0x7f6da4107f70 2024-03-28T22:47:45.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.331+0000 7f6daba00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f6da4117fa0 con 0x7f6da410d040 2024-03-28T22:47:45.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.331+0000 7f6daba00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f6da41182a0 con 0x7f6da41a2f10 2024-03-28T22:47:45.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.332+0000 7f6da9000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6da4107f70 0x7f6da419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.332+0000 7f6da3e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6da410d040 0x7f6da419ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.332+0000 7f6da3e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6da410d040 0x7f6da419ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:58874/0 (socket says 172.21.15.43:58874) 2024-03-28T22:47:45.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.332+0000 7f6da3e00640 1 -- 172.21.15.43:0/3948370862 learned_addr learned my addr 172.21.15.43:0/3948370862 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:45.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.332+0000 7f6da9a00640 1 --2- 172.21.15.43:0/3948370862 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6da41a2f10 0x7f6da41a33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.332+0000 7f6da3e00640 1 -- 172.21.15.43:0/3948370862 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6da41a2f10 msgr2=0x7f6da41a33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:45.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.332+0000 7f6da3e00640 1 --2- 172.21.15.43:0/3948370862 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6da41a2f10 0x7f6da41a33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.332+0000 7f6da3e00640 1 -- 172.21.15.43:0/3948370862 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6da4107f70 msgr2=0x7f6da419e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:45.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.332+0000 7f6da3e00640 1 --2- 172.21.15.43:0/3948370862 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6da4107f70 0x7f6da419e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.332+0000 7f6da3e00640 1 -- 172.21.15.43:0/3948370862 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6da41a3a00 con 0x7f6da410d040 2024-03-28T22:47:45.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.332+0000 7f6da3e00640 1 --2- 172.21.15.43:0/3948370862 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6da410d040 0x7f6da419ebe0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f6d9002c250 tx=0x7f6d90004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:45.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.333+0000 7f6da1600640 1 -- 172.21.15.43:0/3948370862 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d90041070 con 0x7f6da410d040 2024-03-28T22:47:45.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.333+0000 7f6da1600640 1 -- 172.21.15.43:0/3948370862 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d9002cca0 con 0x7f6da410d040 2024-03-28T22:47:45.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.333+0000 7f6daba00640 1 -- 172.21.15.43:0/3948370862 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6da41a3c90 con 0x7f6da410d040 2024-03-28T22:47:45.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.333+0000 7f6da1600640 1 -- 172.21.15.43:0/3948370862 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d90003da0 con 0x7f6da410d040 2024-03-28T22:47:45.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.333+0000 7f6daba00640 1 -- 172.21.15.43:0/3948370862 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6da41ab4b0 con 0x7f6da410d040 2024-03-28T22:47:45.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.334+0000 7f6daba00640 1 -- 172.21.15.43:0/3948370862 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6da4068990 con 0x7f6da410d040 2024-03-28T22:47:45.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.335+0000 7f6da1600640 1 -- 172.21.15.43:0/3948370862 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f6d90044070 con 0x7f6da410d040 2024-03-28T22:47:45.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.338+0000 7f6da1600640 1 --2- 172.21.15.43:0/3948370862 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6d8c0a1f70 0x7f6d8c0a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.338+0000 7f6da1600640 1 -- 172.21.15.43:0/3948370862 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9539+0+0 (secure 0 0 0) 0x7f6d900f0140 con 0x7f6da410d040 2024-03-28T22:47:45.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.338+0000 7f6da9000640 1 --2- 172.21.15.43:0/3948370862 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6d8c0a1f70 0x7f6d8c0a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.339+0000 7f6da9000640 1 --2- 172.21.15.43:0/3948370862 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6d8c0a1f70 0x7f6d8c0a4430 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6d98009bd0 tx=0x7f6d98005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:45.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.341+0000 7f6da1600640 1 -- 172.21.15.43:0/3948370862 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6d900bc890 con 0x7f6da410d040 2024-03-28T22:47:45.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.583+0000 7f6daba00640 1 -- 172.21.15.43:0/3948370862 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6da4108920 con 0x7f6da410d040 2024-03-28T22:47:45.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.584+0000 7f6da1600640 1 -- 172.21.15.43:0/3948370862 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11931 (secure 0 0 0) 0x7f6d900c0930 con 0x7f6da410d040 2024-03-28T22:47:45.583 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:47:45.583 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:33:28.906396+0000","modified":"2024-03-28T22:40:08.672168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24572,"mds_1":24556,"mds_2":24535,"mds_3":24523,"mds_4":24518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26717":{"gid":26717,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/215779342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":215779342},{"type":"v1","addr":"172.21.15.82:6835","nonce":215779342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24535,24556,24518,24523,24572]},"id":1}]} 2024-03-28T22:47:45.583 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 32 2024-03-28T22:47:45.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.587+0000 7f6daba00640 1 -- 172.21.15.43:0/3948370862 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6d8c0a1f70 msgr2=0x7f6d8c0a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:45.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.587+0000 7f6daba00640 1 --2- 172.21.15.43:0/3948370862 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6d8c0a1f70 0x7f6d8c0a4430 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6d98009bd0 tx=0x7f6d98005f90 comp rx=0 tx=0).stop 2024-03-28T22:47:45.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.587+0000 7f6daba00640 1 -- 172.21.15.43:0/3948370862 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6da410d040 msgr2=0x7f6da419ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:45.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.587+0000 7f6daba00640 1 --2- 172.21.15.43:0/3948370862 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6da410d040 0x7f6da419ebe0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f6d9002c250 tx=0x7f6d90004360 comp rx=0 tx=0).stop 2024-03-28T22:47:45.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.587+0000 7f6daba00640 1 -- 172.21.15.43:0/3948370862 shutdown_connections 2024-03-28T22:47:45.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.587+0000 7f6daba00640 1 --2- 172.21.15.43:0/3948370862 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6d8c0a1f70 0x7f6d8c0a4430 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.587+0000 7f6daba00640 1 --2- 172.21.15.43:0/3948370862 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6da410d040 0x7f6da419ebe0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.587+0000 7f6daba00640 1 --2- 172.21.15.43:0/3948370862 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6da41a2f10 0x7f6da41a33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.587+0000 7f6daba00640 1 --2- 172.21.15.43:0/3948370862 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6da4107f70 0x7f6da419e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.587+0000 7f6daba00640 1 -- 172.21.15.43:0/3948370862 >> 172.21.15.43:0/3948370862 conn(0x7f6da4075ca0 msgr2=0x7f6da4077010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:45.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.587+0000 7f6daba00640 1 -- 172.21.15.43:0/3948370862 shutdown_connections 2024-03-28T22:47:45.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.587+0000 7f6daba00640 1 -- 172.21.15.43:0/3948370862 wait complete. 2024-03-28T22:47:45.606 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T22:47:45.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.836+0000 7f5ea8200640 1 -- 172.21.15.43:0/1849048813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ea01076f0 msgr2=0x7f5ea0107ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:45.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.836+0000 7f5ea8200640 1 --2- 172.21.15.43:0/1849048813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ea01076f0 0x7f5ea0107ad0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f5e94009930 tx=0x7f5e9402bc30 comp rx=0 tx=0).stop 2024-03-28T22:47:45.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.836+0000 7f5ea8200640 1 -- 172.21.15.43:0/1849048813 shutdown_connections 2024-03-28T22:47:45.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.836+0000 7f5ea8200640 1 --2- 172.21.15.43:0/1849048813 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ea01016e0 0x7f5ea0101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.836+0000 7f5ea8200640 1 --2- 172.21.15.43:0/1849048813 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ea01020a0 0x7f5ea010e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.836+0000 7f5ea8200640 1 --2- 172.21.15.43:0/1849048813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ea01076f0 0x7f5ea0107ad0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.836+0000 7f5ea8200640 1 -- 172.21.15.43:0/1849048813 >> 172.21.15.43:0/1849048813 conn(0x7f5ea0078b50 msgr2=0x7f5ea0078f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:45.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.836+0000 7f5ea8200640 1 -- 172.21.15.43:0/1849048813 shutdown_connections 2024-03-28T22:47:45.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.836+0000 7f5ea8200640 1 -- 172.21.15.43:0/1849048813 wait complete. 2024-03-28T22:47:45.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.837+0000 7f5ea8200640 1 Processor -- start 2024-03-28T22:47:45.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.837+0000 7f5ea8200640 1 -- start start 2024-03-28T22:47:45.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.838+0000 7f5ea8200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ea01016e0 0x7f5ea0073a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.838+0000 7f5ea8200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ea01020a0 0x7f5ea0073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.838+0000 7f5ea8200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ea01076f0 0x7f5ea00744d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.838+0000 7f5ea8200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5ea0111b10 con 0x7f5ea01020a0 2024-03-28T22:47:45.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.838+0000 7f5ea8200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5ea0111990 con 0x7f5ea01016e0 2024-03-28T22:47:45.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.838+0000 7f5ea8200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f5ea0111c90 con 0x7f5ea01076f0 2024-03-28T22:47:45.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.838+0000 7f5ea5800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ea01016e0 0x7f5ea0073a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.838+0000 7f5ea5800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ea01016e0 0x7f5ea0073a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:58898/0 (socket says 172.21.15.43:58898) 2024-03-28T22:47:45.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.838+0000 7f5ea5800640 1 -- 172.21.15.43:0/2041724661 learned_addr learned my addr 172.21.15.43:0/2041724661 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:45.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.838+0000 7f5ea4e00640 1 --2- 172.21.15.43:0/2041724661 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ea01020a0 0x7f5ea0073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.838+0000 7f5ea6200640 1 --2- 172.21.15.43:0/2041724661 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ea01076f0 0x7f5ea00744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.839+0000 7f5ea5800640 1 -- 172.21.15.43:0/2041724661 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ea01076f0 msgr2=0x7f5ea00744d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:45.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.839+0000 7f5ea5800640 1 --2- 172.21.15.43:0/2041724661 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ea01076f0 0x7f5ea00744d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.839+0000 7f5ea5800640 1 -- 172.21.15.43:0/2041724661 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ea01020a0 msgr2=0x7f5ea0073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:45.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.839+0000 7f5ea5800640 1 --2- 172.21.15.43:0/2041724661 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ea01020a0 0x7f5ea0073f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.839+0000 7f5ea5800640 1 -- 172.21.15.43:0/2041724661 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ea01a64f0 con 0x7f5ea01016e0 2024-03-28T22:47:45.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.839+0000 7f5ea5800640 1 --2- 172.21.15.43:0/2041724661 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ea01016e0 0x7f5ea0073a50 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f5e94009900 tx=0x7f5e94004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:45.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.839+0000 7f5e9e000640 1 -- 172.21.15.43:0/2041724661 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e94043020 con 0x7f5ea01016e0 2024-03-28T22:47:45.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.839+0000 7f5e9e000640 1 -- 172.21.15.43:0/2041724661 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e940054e0 con 0x7f5ea01016e0 2024-03-28T22:47:45.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.839+0000 7f5e9e000640 1 -- 172.21.15.43:0/2041724661 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e94003b80 con 0x7f5ea01016e0 2024-03-28T22:47:45.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.839+0000 7f5ea8200640 1 -- 172.21.15.43:0/2041724661 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ea01a6780 con 0x7f5ea01016e0 2024-03-28T22:47:45.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.840+0000 7f5ea8200640 1 -- 172.21.15.43:0/2041724661 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ea01a6d20 con 0x7f5ea01016e0 2024-03-28T22:47:45.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.843+0000 7f5eaae4c740 1 Processor -- start 2024-03-28T22:47:45.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.843+0000 7f5eaae4c740 1 -- start start 2024-03-28T22:47:45.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.843+0000 7f5eaae4c740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ae730f8380 0x55ae730f8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.843+0000 7f5eaae4c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ae730f8ca0 0x55ae73101340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.843+0000 7f5e9e000640 1 -- 172.21.15.43:0/2041724661 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f5e94046020 con 0x7f5ea01016e0 2024-03-28T22:47:45.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.843+0000 7f5eaae4c740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ae73101880 0x55ae73103c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.843+0000 7f5eaae4c740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ae7305c050 con 0x55ae73101880 2024-03-28T22:47:45.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.843+0000 7f5eaae4c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55ae73004ce0 con 0x55ae730f8ca0 2024-03-28T22:47:45.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.843+0000 7f5eaae4c740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55ae730580f0 con 0x55ae730f8380 2024-03-28T22:47:45.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.844+0000 7f5ea6200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ae730f8380 0x55ae730f8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.844+0000 7f5ea6200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ae730f8380 0x55ae730f8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:34252/0 (socket says 172.21.15.43:34252) 2024-03-28T22:47:45.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.844+0000 7f5ea6200640 1 -- 172.21.15.43:0/2858747466 learned_addr learned my addr 172.21.15.43:0/2858747466 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:45.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.844+0000 7f5ea5800640 1 --2- 172.21.15.43:0/2858747466 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ae730f8ca0 0x55ae73101340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.844+0000 7f5ea4e00640 1 --2- 172.21.15.43:0/2858747466 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ae73101880 0x55ae73103c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.844+0000 7f5ea6200640 1 -- 172.21.15.43:0/2858747466 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ae730f8ca0 msgr2=0x55ae73101340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:45.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.844+0000 7f5ea6200640 1 --2- 172.21.15.43:0/2858747466 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ae730f8ca0 0x55ae73101340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.844+0000 7f5ea6200640 1 -- 172.21.15.43:0/2858747466 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ae73101880 msgr2=0x55ae73103c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:45.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.844+0000 7f5ea6200640 1 --2- 172.21.15.43:0/2858747466 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ae73101880 0x55ae73103c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.844+0000 7f5ea6200640 1 -- 172.21.15.43:0/2858747466 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae72d703f0 con 0x55ae730f8380 2024-03-28T22:47:45.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.844+0000 7f5ea4e00640 1 --2- 172.21.15.43:0/2858747466 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ae73101880 0x55ae73103c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:47:45.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.844+0000 7f5ea6200640 1 --2- 172.21.15.43:0/2858747466 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ae730f8380 0x55ae730f8760 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f5e98009f70 tx=0x7f5e98010800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36eb7a2d5f975549 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:45.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.845+0000 7f5e92a00640 1 -- 172.21.15.43:0/2858747466 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e98016070 con 0x55ae730f8380 2024-03-28T22:47:45.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.845+0000 7f5e92a00640 1 -- 172.21.15.43:0/2858747466 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e98004d20 con 0x55ae730f8380 2024-03-28T22:47:45.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.845+0000 7f5e92a00640 1 -- 172.21.15.43:0/2858747466 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e980128f0 con 0x55ae730f8380 2024-03-28T22:47:45.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.845+0000 7f5e9e000640 1 --2- 172.21.15.43:0/2041724661 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f5e880a1fa0 0x7f5e880a4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.845+0000 7f5e9e000640 1 -- 172.21.15.43:0/2041724661 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9539+0+0 (secure 0 0 0) 0x7f5e940f0540 con 0x7f5ea01016e0 2024-03-28T22:47:45.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.845+0000 7f5eaae4c740 1 -- 172.21.15.43:0/2858747466 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ae730f8380 msgr2=0x55ae730f8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:45.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.845+0000 7f5eaae4c740 1 --2- 172.21.15.43:0/2858747466 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ae730f8380 0x55ae730f8760 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f5e98009f70 tx=0x7f5e98010800 comp rx=0 tx=0).stop 2024-03-28T22:47:45.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.845+0000 7f5ea5800640 1 --2- 172.21.15.43:0/2041724661 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f5e880a1fa0 0x7f5e880a4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.846+0000 7f5eaae4c740 1 -- 172.21.15.43:0/2858747466 shutdown_connections 2024-03-28T22:47:45.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.846+0000 7f5eaae4c740 1 --2- 172.21.15.43:0/2858747466 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ae730f8ca0 0x55ae73101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.846+0000 7f5eaae4c740 1 --2- 172.21.15.43:0/2858747466 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ae73101880 0x55ae73103c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.846+0000 7f5eaae4c740 1 --2- 172.21.15.43:0/2858747466 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ae730f8380 0x55ae730f8760 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.846+0000 7f5eaae4c740 1 -- 172.21.15.43:0/2858747466 >> 172.21.15.43:0/2858747466 conn(0x55ae730f2ab0 msgr2=0x55ae730f2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:45.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.846+0000 7f5eaae4c740 1 -- 172.21.15.43:0/2858747466 shutdown_connections 2024-03-28T22:47:45.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.846+0000 7f5eaae4c740 1 -- 172.21.15.43:0/2858747466 wait complete. 2024-03-28T22:47:45.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.846+0000 7f5ea5800640 1 --2- 172.21.15.43:0/2041724661 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f5e880a1fa0 0x7f5e880a4460 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x55ae730f7e50 tx=0x7f5e94004610 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:45.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.846+0000 7f5eaae4c740 1 -- start start 2024-03-28T22:47:45.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.847+0000 7f5eaae4c740 1 Processor -- start 2024-03-28T22:47:45.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.847+0000 7f5eaae4c740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ae730f8380 0x55ae730fe880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.847+0000 7f5eaae4c740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ae730f8ca0 0x55ae730fedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.847+0000 7f5eaae4c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ae73101880 0x55ae730ff300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.847+0000 7f5eaae4c740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ae73104670 con 0x55ae730f8380 2024-03-28T22:47:45.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.847+0000 7f5eaae4c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55ae72ffa2f0 con 0x55ae73101880 2024-03-28T22:47:45.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.847+0000 7f5eaae4c740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55ae731047f0 con 0x55ae730f8ca0 2024-03-28T22:47:45.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.847+0000 7f5eaae4c740 10 client.0 _ll_register_callbacks cb 0x55ae72d6fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:47:45.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.848+0000 7f5ea4e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ae730f8ca0 0x55ae730fedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.848+0000 7f5ea4e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ae730f8ca0 0x55ae730fedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:34256/0 (socket says 172.21.15.43:34256) 2024-03-28T22:47:45.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.848+0000 7f5ea4e00640 1 -- 172.21.15.43:0/3312902167 learned_addr learned my addr 172.21.15.43:0/3312902167 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:45.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.848+0000 7f5ea6200640 1 --2- 172.21.15.43:0/3312902167 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ae730f8380 0x55ae730fe880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.848+0000 7f5ea6200640 1 --2- 172.21.15.43:0/3312902167 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ae73101880 0x55ae730ff300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.848+0000 7f5ea4e00640 1 -- 172.21.15.43:0/3312902167 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ae73101880 msgr2=0x55ae730ff300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:45.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.848+0000 7f5ea4e00640 1 --2- 172.21.15.43:0/3312902167 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ae73101880 0x55ae730ff300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.848+0000 7f5ea4e00640 1 -- 172.21.15.43:0/3312902167 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ae730f8380 msgr2=0x55ae730fe880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:45.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.848+0000 7f5ea4e00640 1 --2- 172.21.15.43:0/3312902167 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ae730f8380 0x55ae730fe880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.848+0000 7f5ea4e00640 1 -- 172.21.15.43:0/3312902167 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae72e3a7d0 con 0x55ae730f8ca0 2024-03-28T22:47:45.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.848+0000 7f5ea6200640 1 --2- 172.21.15.43:0/3312902167 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ae730f8380 0x55ae730fe880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:47:45.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.848+0000 7f5ea4e00640 1 --2- 172.21.15.43:0/3312902167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ae730f8ca0 0x55ae730fedc0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c012700 tx=0x7f5e8c012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a53cc8adde48d6d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:45.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.848+0000 7f5e87400640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:47:45.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.848+0000 7f5e87400640 1 -- 172.21.15.43:0/3312902167 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c004050 con 0x55ae730f8ca0 2024-03-28T22:47:45.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.849+0000 7f5eaae4c740 1 -- 172.21.15.43:0/3312902167 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e98004d20 con 0x55ae730f8ca0 2024-03-28T22:47:45.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.849+0000 7f5e87400640 1 -- 172.21.15.43:0/3312902167 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9539+0+0 (secure 0 0 0) 0x7f5e8c0041f0 con 0x55ae730f8ca0 2024-03-28T22:47:45.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.849+0000 7f5e87400640 1 -- 172.21.15.43:0/3312902167 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5e8c025050 con 0x55ae730f8ca0 2024-03-28T22:47:45.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.849+0000 7f5eaae4c740 10 client.29053 fetch_fsmap learned FSMap version 32 2024-03-28T22:47:45.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.849+0000 7f5eaae4c740 1 -- 172.21.15.43:0/3312902167 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ae731109e0 con 0x55ae730f8ca0 2024-03-28T22:47:45.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.849+0000 7f5e87400640 1 -- 172.21.15.43:0/3312902167 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e98004d20 con 0x55ae730f8ca0 2024-03-28T22:47:45.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.849+0000 7f5e87400640 1 -- 172.21.15.43:0/3312902167 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f5e8c003aa0 con 0x55ae730f8ca0 2024-03-28T22:47:45.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.849+0000 7f5eaae4c740 10 client.29053 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:47:45.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.849+0000 7f5eaae4c740 10 client.29053 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:47:45.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.849+0000 7f5eaae4c740 20 client.29053 populate_metadata read hostname 'smithi043' 2024-03-28T22:47:45.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.849+0000 7f5eaae4c740 1 --2- 172.21.15.43:0/3312902167 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55ae73111df0 0x55ae731141e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.849+0000 7f5eaae4c740 4 client.29053 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:47:45.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.849+0000 7f5eaae4c740 1 -- 172.21.15.43:0/3312902167 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae73005680 con 0x55ae73111df0 2024-03-28T22:47:45.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.850+0000 7f5ea6200640 1 --2- 172.21.15.43:0/3312902167 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55ae73111df0 0x55ae731141e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.850+0000 7f5ea6200640 1 --2- 172.21.15.43:0/3312902167 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55ae73111df0 0x55ae731141e0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b46eb9da511f4eee server_cookie=24886f6c607521fd in_seq=0 out_seq=0 2024-03-28T22:47:45.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.850+0000 7f5e87400640 10 client.29053 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:47:45.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.851+0000 7f5e87400640 1 -- 172.21.15.43:0/3312902167 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ae73005680 con 0x55ae73111df0 2024-03-28T22:47:45.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.851+0000 7f5e87400640 10 client.29053 handle_command_reply: tid=0 2024-03-28T22:47:45.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.851+0000 7f5eaae4c740 1 client.29053 shutdown 2024-03-28T22:47:45.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.851+0000 7f5eaae4c740 1 -- 172.21.15.43:0/3312902167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ae730f8ca0 msgr2=0x55ae730fedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:45.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.851+0000 7f5eaae4c740 1 --2- 172.21.15.43:0/3312902167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ae730f8ca0 0x55ae730fedc0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c012700 tx=0x7f5e8c012bc0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.851+0000 7f5eaae4c740 1 -- 172.21.15.43:0/3312902167 shutdown_connections 2024-03-28T22:47:45.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.851+0000 7f5eaae4c740 1 --2- 172.21.15.43:0/3312902167 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55ae73111df0 0x55ae731141e0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.851+0000 7f5eaae4c740 1 --2- 172.21.15.43:0/3312902167 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ae73101880 0x55ae730ff300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.851+0000 7f5eaae4c740 1 --2- 172.21.15.43:0/3312902167 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ae730f8ca0 0x55ae730fedc0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.851+0000 7f5eaae4c740 1 --2- 172.21.15.43:0/3312902167 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ae730f8380 0x55ae730fe880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.851+0000 7f5e87400640 0 client.29053 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:47:45.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.851+0000 7f5eaae4c740 1 -- 172.21.15.43:0/3312902167 >> 172.21.15.43:0/3312902167 conn(0x55ae730f2ab0 msgr2=0x55ae730f4800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:45.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.852+0000 7f5eaae4c740 1 -- 172.21.15.43:0/3312902167 shutdown_connections 2024-03-28T22:47:45.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.852+0000 7f5eaae4c740 1 -- 172.21.15.43:0/3312902167 wait complete. 2024-03-28T22:47:45.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.852+0000 7f5eaae4c740 20 client.29053 trim_cache size 0 max 16384 2024-03-28T22:47:45.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.869+0000 7f5eaae4c740 1 Processor -- start 2024-03-28T22:47:45.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.869+0000 7f5eaae4c740 1 -- start start 2024-03-28T22:47:45.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.870+0000 7f5eaae4c740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ae730f8380 0x55ae73100fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.870+0000 7f5eaae4c740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ae730f8ca0 0x55ae7310d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.870+0000 7f5eaae4c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ae73101880 0x55ae73119ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.870+0000 7f5eaae4c740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5ea01b0fb0 con 0x55ae730f8ca0 2024-03-28T22:47:45.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.870+0000 7f5eaae4c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5ea01b2af0 con 0x55ae73101880 2024-03-28T22:47:45.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.870+0000 7f5eaae4c740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f5ea01b1680 con 0x55ae730f8380 2024-03-28T22:47:45.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.870+0000 7f5ea4e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ae730f8ca0 0x55ae7310d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.870+0000 7f5ea4e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ae730f8ca0 0x55ae7310d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37258/0 (socket says 172.21.15.43:37258) 2024-03-28T22:47:45.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.870+0000 7f5ea4e00640 1 -- 172.21.15.43:0/15937601 learned_addr learned my addr 172.21.15.43:0/15937601 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:45.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.870+0000 7f5ea6200640 1 --2- 172.21.15.43:0/15937601 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ae730f8380 0x55ae73100fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.871+0000 7f5ea4e00640 1 -- 172.21.15.43:0/15937601 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ae730f8380 msgr2=0x55ae73100fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:45.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.871+0000 7f5ea4e00640 1 --2- 172.21.15.43:0/15937601 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ae730f8380 0x55ae73100fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.871+0000 7f5ea4e00640 1 -- 172.21.15.43:0/15937601 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ae73101880 msgr2=0x55ae73119ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:45.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.871+0000 7f5ea6200640 1 --2- 172.21.15.43:0/15937601 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ae73101880 0x55ae73119ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.871+0000 7f5ea4e00640 1 --2- 172.21.15.43:0/15937601 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ae73101880 0x55ae73119ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.871+0000 7f5ea4e00640 1 -- 172.21.15.43:0/15937601 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae730f28e0 con 0x55ae730f8ca0 2024-03-28T22:47:45.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.871+0000 7f5ea4e00640 1 --2- 172.21.15.43:0/15937601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ae730f8ca0 0x55ae7310d470 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c012420 tx=0x7f5e8c009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd2e7a497805c4fd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:45.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.871+0000 7f5e92000640 1 -- 172.21.15.43:0/15937601 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c010550 con 0x55ae730f8ca0 2024-03-28T22:47:45.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.871+0000 7f5e92000640 1 -- 172.21.15.43:0/15937601 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e8c02e070 con 0x55ae730f8ca0 2024-03-28T22:47:45.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.871+0000 7f5e92000640 1 -- 172.21.15.43:0/15937601 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c005070 con 0x55ae730f8ca0 2024-03-28T22:47:45.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.872+0000 7f5eaae4c740 1 -- 172.21.15.43:0/15937601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ae730f8ca0 msgr2=0x55ae7310d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:45.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.872+0000 7f5eaae4c740 1 --2- 172.21.15.43:0/15937601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ae730f8ca0 0x55ae7310d470 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c012420 tx=0x7f5e8c009240 comp rx=0 tx=0).stop 2024-03-28T22:47:45.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.872+0000 7f5eaae4c740 1 -- 172.21.15.43:0/15937601 shutdown_connections 2024-03-28T22:47:45.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.872+0000 7f5eaae4c740 1 --2- 172.21.15.43:0/15937601 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ae73101880 0x55ae73119ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.872+0000 7f5eaae4c740 1 --2- 172.21.15.43:0/15937601 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ae730f8ca0 0x55ae7310d470 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.872+0000 7f5eaae4c740 1 --2- 172.21.15.43:0/15937601 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ae730f8380 0x55ae73100fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.872+0000 7f5eaae4c740 1 -- 172.21.15.43:0/15937601 >> 172.21.15.43:0/15937601 conn(0x55ae730f2ab0 msgr2=0x55ae730f1b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:45.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.872+0000 7f5eaae4c740 1 -- 172.21.15.43:0/15937601 shutdown_connections 2024-03-28T22:47:45.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.872+0000 7f5eaae4c740 1 -- 172.21.15.43:0/15937601 wait complete. 2024-03-28T22:47:45.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.873+0000 7f5eaae4c740 1 -- start start 2024-03-28T22:47:45.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.873+0000 7f5eaae4c740 1 Processor -- start 2024-03-28T22:47:45.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.873+0000 7f5eaae4c740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ae730f8380 0x55ae73105f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.873+0000 7f5eaae4c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ae730f8ca0 0x55ae73106450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.873+0000 7f5eaae4c740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ae73101880 0x55ae731700c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.873+0000 7f5eaae4c740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5ea01b2420 con 0x55ae73101880 2024-03-28T22:47:45.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.873+0000 7f5eaae4c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5ea01b1d50 con 0x55ae730f8ca0 2024-03-28T22:47:45.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.873+0000 7f5eaae4c740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f5ea01aefb0 con 0x55ae730f8380 2024-03-28T22:47:45.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.873+0000 7f5eaae4c740 10 client.0 _ll_register_callbacks cb 0x7f5e8c009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:47:45.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.874+0000 7f5ea6200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ae730f8380 0x55ae73105f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.874+0000 7f5ea4e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ae730f8ca0 0x55ae73106450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.874+0000 7f5ea6200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ae730f8380 0x55ae73105f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:34272/0 (socket says 172.21.15.43:34272) 2024-03-28T22:47:45.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.874+0000 7f5ea6200640 1 -- 172.21.15.43:0/2873704603 learned_addr learned my addr 172.21.15.43:0/2873704603 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:45.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.874+0000 7f5ea6200640 1 --2- 172.21.15.43:0/2873704603 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ae73101880 0x55ae731700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.874+0000 7f5ea4e00640 1 -- 172.21.15.43:0/2873704603 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ae730f8380 msgr2=0x55ae73105f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:45.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.874+0000 7f5ea4e00640 1 --2- 172.21.15.43:0/2873704603 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ae730f8380 0x55ae73105f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.874+0000 7f5ea4e00640 1 -- 172.21.15.43:0/2873704603 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ae73101880 msgr2=0x55ae731700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:45.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.874+0000 7f5ea4e00640 1 --2- 172.21.15.43:0/2873704603 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ae73101880 0x55ae731700c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.874+0000 7f5ea4e00640 1 -- 172.21.15.43:0/2873704603 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae73103c70 con 0x55ae730f8ca0 2024-03-28T22:47:45.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.874+0000 7f5ea4e00640 1 --2- 172.21.15.43:0/2873704603 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ae730f8ca0 0x55ae73106450 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x55ae730fd060 tx=0x7f5e8c00fa20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1b2d7cec224c6d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:45.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.874+0000 7f5e87400640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:47:45.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.875+0000 7f5e87400640 1 -- 172.21.15.43:0/2873704603 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c00fe20 con 0x55ae730f8ca0 2024-03-28T22:47:45.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.875+0000 7f5eaae4c740 1 -- 172.21.15.43:0/2873704603 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e8c02e070 con 0x55ae730f8ca0 2024-03-28T22:47:45.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.875+0000 7f5e87400640 1 -- 172.21.15.43:0/2873704603 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9539+0+0 (secure 0 0 0) 0x7f5e8c022040 con 0x55ae730f8ca0 2024-03-28T22:47:45.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.875+0000 7f5e87400640 1 -- 172.21.15.43:0/2873704603 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5e8c010900 con 0x55ae730f8ca0 2024-03-28T22:47:45.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.875+0000 7f5eaae4c740 10 client.29120 fetch_fsmap learned FSMap version 32 2024-03-28T22:47:45.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.875+0000 7f5eaae4c740 1 -- 172.21.15.43:0/2873704603 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ae73178aa0 con 0x55ae730f8ca0 2024-03-28T22:47:45.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.875+0000 7f5e87400640 1 -- 172.21.15.43:0/2873704603 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e8c02e070 con 0x55ae730f8ca0 2024-03-28T22:47:45.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.875+0000 7f5e87400640 1 -- 172.21.15.43:0/2873704603 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f5e8c005190 con 0x55ae730f8ca0 2024-03-28T22:47:45.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.875+0000 7f5eaae4c740 10 client.29120 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:47:45.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.875+0000 7f5eaae4c740 10 client.29120 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-03-28T22:47:45.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.875+0000 7f5eaae4c740 20 client.29120 populate_metadata read hostname 'smithi043' 2024-03-28T22:47:45.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.875+0000 7f5eaae4c740 1 --2- 172.21.15.43:0/2873704603 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55ae73179150 0x55ae7317b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:45.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.875+0000 7f5eaae4c740 4 client.29120 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:47:45.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.875+0000 7f5eaae4c740 1 -- 172.21.15.43:0/2873704603 --> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ae73069590 con 0x55ae73179150 2024-03-28T22:47:45.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.876+0000 7f5ea6200640 1 --2- 172.21.15.43:0/2873704603 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55ae73179150 0x55ae7317b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:45.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.876+0000 7f5ea6200640 1 --2- 172.21.15.43:0/2873704603 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55ae73179150 0x55ae7317b540 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f4df306a7e4e4b99 server_cookie=5096f5b48b6081f in_seq=0 out_seq=0 2024-03-28T22:47:45.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.876+0000 7f5e87400640 10 client.29120 ms_handle_connect on v2:172.21.15.82:6836/757432882 2024-03-28T22:47:45.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.876+0000 7f5e87400640 1 -- 172.21.15.43:0/2873704603 <== mds.4 v2:172.21.15.82:6836/757432882 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ae73069590 con 0x55ae73179150 2024-03-28T22:47:45.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.876+0000 7f5e87400640 10 client.29120 handle_command_reply: tid=0 2024-03-28T22:47:45.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.876+0000 7f5eaae4c740 1 client.29120 shutdown 2024-03-28T22:47:45.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.877+0000 7f5eaae4c740 1 -- 172.21.15.43:0/2873704603 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ae730f8ca0 msgr2=0x55ae73106450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:45.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.877+0000 7f5eaae4c740 1 --2- 172.21.15.43:0/2873704603 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ae730f8ca0 0x55ae73106450 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x55ae730fd060 tx=0x7f5e8c00fa20 comp rx=0 tx=0).stop 2024-03-28T22:47:45.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.877+0000 7f5eaae4c740 1 -- 172.21.15.43:0/2873704603 shutdown_connections 2024-03-28T22:47:45.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.877+0000 7f5eaae4c740 1 --2- 172.21.15.43:0/2873704603 >> [v2:172.21.15.82:6836/757432882,v1:172.21.15.82:6837/757432882] conn(0x55ae73179150 0x55ae7317b540 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.877+0000 7f5eaae4c740 1 --2- 172.21.15.43:0/2873704603 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ae730f8ca0 0x55ae73106450 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.877+0000 7f5e87400640 0 client.29120 ms_handle_reset on v2:172.21.15.82:6836/757432882 2024-03-28T22:47:45.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.877+0000 7f5eaae4c740 1 --2- 172.21.15.43:0/2873704603 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ae73101880 0x55ae731700c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.877+0000 7f5eaae4c740 1 --2- 172.21.15.43:0/2873704603 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ae730f8380 0x55ae73105f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.877+0000 7f5eaae4c740 1 -- 172.21.15.43:0/2873704603 >> 172.21.15.43:0/2873704603 conn(0x55ae730f2ab0 msgr2=0x55ae730fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:45.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.877+0000 7f5eaae4c740 1 -- 172.21.15.43:0/2873704603 shutdown_connections 2024-03-28T22:47:45.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.877+0000 7f5eaae4c740 1 -- 172.21.15.43:0/2873704603 wait complete. 2024-03-28T22:47:45.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.877+0000 7f5eaae4c740 20 client.29120 trim_cache size 0 max 16384 2024-03-28T22:47:45.884 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:47:45.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.878+0000 7f5ea8200640 1 -- 172.21.15.43:0/2041724661 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f5e880a1fa0 msgr2=0x7f5e880a4460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:45.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.878+0000 7f5ea8200640 1 --2- 172.21.15.43:0/2041724661 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f5e880a1fa0 0x7f5e880a4460 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x55ae730f7e50 tx=0x7f5e94004610 comp rx=0 tx=0).stop 2024-03-28T22:47:45.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.878+0000 7f5ea8200640 1 -- 172.21.15.43:0/2041724661 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ea01016e0 msgr2=0x7f5ea0073a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:45.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.878+0000 7f5ea8200640 1 --2- 172.21.15.43:0/2041724661 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ea01016e0 0x7f5ea0073a50 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f5e94009900 tx=0x7f5e94004360 comp rx=0 tx=0).stop 2024-03-28T22:47:45.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.878+0000 7f5ea8200640 1 -- 172.21.15.43:0/2041724661 shutdown_connections 2024-03-28T22:47:45.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.878+0000 7f5ea8200640 1 --2- 172.21.15.43:0/2041724661 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f5e880a1fa0 0x7f5e880a4460 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.878+0000 7f5ea8200640 1 --2- 172.21.15.43:0/2041724661 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5ea01076f0 0x7f5ea00744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.878+0000 7f5ea8200640 1 --2- 172.21.15.43:0/2041724661 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5ea01020a0 0x7f5ea0073f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.878+0000 7f5ea8200640 1 --2- 172.21.15.43:0/2041724661 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5ea01016e0 0x7f5ea0073a50 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:45.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.878+0000 7f5ea8200640 1 -- 172.21.15.43:0/2041724661 >> 172.21.15.43:0/2041724661 conn(0x7f5ea0078b50 msgr2=0x7f5ea0104160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:45.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.878+0000 7f5ea8200640 1 -- 172.21.15.43:0/2041724661 shutdown_connections 2024-03-28T22:47:45.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:45.878+0000 7f5ea8200640 1 -- 172.21.15.43:0/2041724661 wait complete. 2024-03-28T22:47:45.895 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T22:47:46.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.116+0000 7f6cb5800640 1 -- 172.21.15.43:0/3379105992 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6cb0112590 msgr2=0x7f6cb0114980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.116+0000 7f6cb5800640 1 --2- 172.21.15.43:0/3379105992 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6cb0112590 0x7f6cb0114980 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8009930 tx=0x7f6ca802bc30 comp rx=0 tx=0).stop 2024-03-28T22:47:46.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.116+0000 7f6cb5800640 1 -- 172.21.15.43:0/3379105992 shutdown_connections 2024-03-28T22:47:46.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.116+0000 7f6cb5800640 1 --2- 172.21.15.43:0/3379105992 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6cb0112590 0x7f6cb0114980 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.116+0000 7f6cb5800640 1 --2- 172.21.15.43:0/3379105992 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6cb00765a0 0x7f6cb0112050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.116+0000 7f6cb5800640 1 --2- 172.21.15.43:0/3379105992 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6cb0075c80 0x7f6cb0076060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.116+0000 7f6cb5800640 1 -- 172.21.15.43:0/3379105992 >> 172.21.15.43:0/3379105992 conn(0x7f6cb00ff3a0 msgr2=0x7f6cb01017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:46.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.116+0000 7f6cb5800640 1 -- 172.21.15.43:0/3379105992 shutdown_connections 2024-03-28T22:47:46.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.117+0000 7f6cb5800640 1 -- 172.21.15.43:0/3379105992 wait complete. 2024-03-28T22:47:46.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.117+0000 7f6cb5800640 1 Processor -- start 2024-03-28T22:47:46.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.118+0000 7f6cb5800640 1 -- start start 2024-03-28T22:47:46.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.118+0000 7f6cb5800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6cb0075c80 0x7f6cb019e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.118+0000 7f6cb5800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6cb00765a0 0x7f6cb019ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.118+0000 7f6cb5800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6cb0112590 0x7f6cb01a2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.118+0000 7f6cb5800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb0118120 con 0x7f6cb0112590 2024-03-28T22:47:46.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.118+0000 7f6cb5800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb0117fa0 con 0x7f6cb0075c80 2024-03-28T22:47:46.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.118+0000 7f6cb5800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb01182a0 con 0x7f6cb00765a0 2024-03-28T22:47:46.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.118+0000 7f6caea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6cb0075c80 0x7f6cb019e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.118+0000 7f6caea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6cb0075c80 0x7f6cb019e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:58936/0 (socket says 172.21.15.43:58936) 2024-03-28T22:47:46.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.118+0000 7f6caea00640 1 -- 172.21.15.43:0/3231251705 learned_addr learned my addr 172.21.15.43:0/3231251705 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:46.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.118+0000 7f6ca6000640 1 --2- 172.21.15.43:0/3231251705 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6cb00765a0 0x7f6cb019ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.118+0000 7f6caf400640 1 --2- 172.21.15.43:0/3231251705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6cb0112590 0x7f6cb01a2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.119+0000 7f6caea00640 1 -- 172.21.15.43:0/3231251705 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6cb00765a0 msgr2=0x7f6cb019ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.119+0000 7f6caea00640 1 --2- 172.21.15.43:0/3231251705 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6cb00765a0 0x7f6cb019ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.119+0000 7f6caea00640 1 -- 172.21.15.43:0/3231251705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6cb0112590 msgr2=0x7f6cb01a2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.119+0000 7f6caea00640 1 --2- 172.21.15.43:0/3231251705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6cb0112590 0x7f6cb01a2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.119+0000 7f6caea00640 1 -- 172.21.15.43:0/3231251705 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cb01a3670 con 0x7f6cb0075c80 2024-03-28T22:47:46.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.119+0000 7f6caea00640 1 --2- 172.21.15.43:0/3231251705 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6cb0075c80 0x7f6cb019e680 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f6ca000ebc0 tx=0x7f6ca000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:46.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.119+0000 7f6ca7e00640 1 -- 172.21.15.43:0/3231251705 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca0010630 con 0x7f6cb0075c80 2024-03-28T22:47:46.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.119+0000 7f6ca7e00640 1 -- 172.21.15.43:0/3231251705 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ca0010cb0 con 0x7f6cb0075c80 2024-03-28T22:47:46.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.119+0000 7f6ca7e00640 1 -- 172.21.15.43:0/3231251705 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca0018bd0 con 0x7f6cb0075c80 2024-03-28T22:47:46.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.119+0000 7f6cb5800640 1 -- 172.21.15.43:0/3231251705 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cb01a3960 con 0x7f6cb0075c80 2024-03-28T22:47:46.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.120+0000 7f6cb5800640 1 -- 172.21.15.43:0/3231251705 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cb01ab1f0 con 0x7f6cb0075c80 2024-03-28T22:47:46.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.123+0000 7f6cb8409740 1 Processor -- start 2024-03-28T22:47:46.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.123+0000 7f6cb8409740 1 -- start start 2024-03-28T22:47:46.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.123+0000 7f6ca7e00640 1 -- 172.21.15.43:0/3231251705 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f6ca00107d0 con 0x7f6cb0075c80 2024-03-28T22:47:46.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.123+0000 7f6cb8409740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560747a4e380 0x560747a4e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.123+0000 7f6cb8409740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560747a4eca0 0x560747a57340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.123+0000 7f6cb8409740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560747a57880 0x560747a59c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.123+0000 7f6cb8409740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x5607479b2050 con 0x560747a57880 2024-03-28T22:47:46.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.123+0000 7f6cb8409740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x56074795ace0 con 0x560747a4e380 2024-03-28T22:47:46.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.123+0000 7f6cb8409740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x5607479ae0f0 con 0x560747a4eca0 2024-03-28T22:47:46.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.123+0000 7f6caf400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560747a4e380 0x560747a4e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.123+0000 7f6caf400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560747a4e380 0x560747a4e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:58944/0 (socket says 172.21.15.43:58944) 2024-03-28T22:47:46.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.123+0000 7f6caf400640 1 -- 172.21.15.43:0/169030128 learned_addr learned my addr 172.21.15.43:0/169030128 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:46.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.123+0000 7f6caea00640 1 --2- 172.21.15.43:0/169030128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560747a4eca0 0x560747a57340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.123+0000 7f6ca6000640 1 --2- 172.21.15.43:0/169030128 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560747a57880 0x560747a59c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.124+0000 7f6caf400640 1 -- 172.21.15.43:0/169030128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560747a4eca0 msgr2=0x560747a57340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:46.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.124+0000 7f6caf400640 1 --2- 172.21.15.43:0/169030128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560747a4eca0 0x560747a57340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.124+0000 7f6caf400640 1 -- 172.21.15.43:0/169030128 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560747a57880 msgr2=0x560747a59c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:46.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.124+0000 7f6caf400640 1 --2- 172.21.15.43:0/169030128 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560747a57880 0x560747a59c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.124+0000 7f6caf400640 1 -- 172.21.15.43:0/169030128 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607476c63f0 con 0x560747a4e380 2024-03-28T22:47:46.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.124+0000 7f6caf400640 1 --2- 172.21.15.43:0/169030128 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560747a4e380 0x560747a4e760 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f6cb006ab10 tx=0x7f6ca8005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19ed22c6d7583ec5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:46.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.124+0000 7f6c9fe00640 1 -- 172.21.15.43:0/169030128 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca8044020 con 0x560747a4e380 2024-03-28T22:47:46.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.124+0000 7f6ca7e00640 1 --2- 172.21.15.43:0/3231251705 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6c940a1f70 0x7f6c940a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.125+0000 7f6c9fe00640 1 -- 172.21.15.43:0/169030128 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ca8004240 con 0x560747a4e380 2024-03-28T22:47:46.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.125+0000 7f6c9fe00640 1 -- 172.21.15.43:0/169030128 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca80046c0 con 0x560747a4e380 2024-03-28T22:47:46.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.125+0000 7f6ca7e00640 1 -- 172.21.15.43:0/3231251705 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9539+0+0 (secure 0 0 0) 0x7f6ca00cec20 con 0x7f6cb0075c80 2024-03-28T22:47:46.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.125+0000 7f6caea00640 1 --2- 172.21.15.43:0/3231251705 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6c940a1f70 0x7f6c940a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.125+0000 7f6cb8409740 1 -- 172.21.15.43:0/169030128 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560747a4e380 msgr2=0x560747a4e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.125+0000 7f6cb8409740 1 --2- 172.21.15.43:0/169030128 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560747a4e380 0x560747a4e760 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f6cb006ab10 tx=0x7f6ca8005d50 comp rx=0 tx=0).stop 2024-03-28T22:47:46.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.125+0000 7f6cb8409740 1 -- 172.21.15.43:0/169030128 shutdown_connections 2024-03-28T22:47:46.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.125+0000 7f6cb8409740 1 --2- 172.21.15.43:0/169030128 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560747a57880 0x560747a59c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.125+0000 7f6cb8409740 1 --2- 172.21.15.43:0/169030128 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560747a4eca0 0x560747a57340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.125+0000 7f6cb8409740 1 --2- 172.21.15.43:0/169030128 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560747a4e380 0x560747a4e760 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.125+0000 7f6cb8409740 1 -- 172.21.15.43:0/169030128 >> 172.21.15.43:0/169030128 conn(0x560747a48ab0 msgr2=0x560747a48e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:46.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.125+0000 7f6cb8409740 1 -- 172.21.15.43:0/169030128 shutdown_connections 2024-03-28T22:47:46.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.126+0000 7f6cb8409740 1 -- 172.21.15.43:0/169030128 wait complete. 2024-03-28T22:47:46.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.126+0000 7f6caea00640 1 --2- 172.21.15.43:0/3231251705 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6c940a1f70 0x7f6c940a4430 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x560747a4de50 tx=0x7f6ca00d26d0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:46.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.126+0000 7f6cb8409740 1 -- start start 2024-03-28T22:47:46.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.126+0000 7f6cb8409740 1 Processor -- start 2024-03-28T22:47:46.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.127+0000 7f6cb8409740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560747a4e380 0x560747a54860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.127+0000 7f6cb8409740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560747a4eca0 0x560747a54da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.127+0000 7f6cb8409740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560747a57880 0x560747a552e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.127+0000 7f6cb8409740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x560747a5a610 con 0x560747a57880 2024-03-28T22:47:46.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.127+0000 7f6cb8409740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x5607479502f0 con 0x560747a4eca0 2024-03-28T22:47:46.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.127+0000 7f6cb8409740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x560747a5a790 con 0x560747a4e380 2024-03-28T22:47:46.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.127+0000 7f6cb8409740 10 client.0 _ll_register_callbacks cb 0x5607476c5be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:47:46.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.127+0000 7f6ca6000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560747a4eca0 0x560747a54da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.127+0000 7f6caf400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560747a4e380 0x560747a54860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.127+0000 7f6ca6000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560747a4eca0 0x560747a54da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:58950/0 (socket says 172.21.15.43:58950) 2024-03-28T22:47:46.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.127+0000 7f6ca6000640 1 -- 172.21.15.43:0/3757308694 learned_addr learned my addr 172.21.15.43:0/3757308694 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:46.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.127+0000 7f6caf400640 1 --2- 172.21.15.43:0/3757308694 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560747a57880 0x560747a552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.127+0000 7f6ca6000640 1 -- 172.21.15.43:0/3757308694 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560747a4e380 msgr2=0x560747a54860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:46.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.127+0000 7f6ca6000640 1 --2- 172.21.15.43:0/3757308694 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560747a4e380 0x560747a54860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.127+0000 7f6ca6000640 1 -- 172.21.15.43:0/3757308694 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560747a57880 msgr2=0x560747a552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:46.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.127+0000 7f6ca6000640 1 --2- 172.21.15.43:0/3757308694 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560747a57880 0x560747a552e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.127+0000 7f6ca6000640 1 -- 172.21.15.43:0/3757308694 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607477907d0 con 0x560747a4eca0 2024-03-28T22:47:46.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.127+0000 7f6caf400640 1 --2- 172.21.15.43:0/3757308694 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560747a57880 0x560747a552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:47:46.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.128+0000 7f6ca6000640 1 --2- 172.21.15.43:0/3757308694 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560747a4eca0 0x560747a54da0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f6c98012700 tx=0x7f6c98012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=937e9fff6e7c6131 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:46.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.128+0000 7f6c9cc00640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:47:46.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.128+0000 7f6c9cc00640 1 -- 172.21.15.43:0/3757308694 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c9800f8c0 con 0x560747a4eca0 2024-03-28T22:47:46.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.128+0000 7f6c9cc00640 1 -- 172.21.15.43:0/3757308694 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9539+0+0 (secure 0 0 0) 0x7f6c9800fa60 con 0x560747a4eca0 2024-03-28T22:47:46.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.128+0000 7f6cb8409740 1 -- 172.21.15.43:0/3757308694 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ca8004240 con 0x560747a4eca0 2024-03-28T22:47:46.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.128+0000 7f6c9cc00640 1 -- 172.21.15.43:0/3757308694 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6c98025050 con 0x560747a4eca0 2024-03-28T22:47:46.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.128+0000 7f6cb8409740 10 client.29132 fetch_fsmap learned FSMap version 32 2024-03-28T22:47:46.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.128+0000 7f6cb8409740 1 -- 172.21.15.43:0/3757308694 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560747a668f0 con 0x560747a4eca0 2024-03-28T22:47:46.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.129+0000 7f6c9cc00640 1 -- 172.21.15.43:0/3757308694 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ca8004240 con 0x560747a4eca0 2024-03-28T22:47:46.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.129+0000 7f6c9cc00640 1 -- 172.21.15.43:0/3757308694 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6c98010430 con 0x560747a4eca0 2024-03-28T22:47:46.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.129+0000 7f6cb8409740 10 client.29132 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:47:46.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.129+0000 7f6cb8409740 10 client.29132 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:47:46.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.129+0000 7f6cb8409740 20 client.29132 populate_metadata read hostname 'smithi043' 2024-03-28T22:47:46.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.129+0000 7f6cb8409740 1 --2- 172.21.15.43:0/3757308694 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x560747a67e00 0x560747a6a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.129+0000 7f6cb8409740 4 client.29132 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:47:46.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.129+0000 7f6cb8409740 1 -- 172.21.15.43:0/3757308694 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56074795b680 con 0x560747a67e00 2024-03-28T22:47:46.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.129+0000 7f6caf400640 1 --2- 172.21.15.43:0/3757308694 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x560747a67e00 0x560747a6a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.130+0000 7f6caf400640 1 --2- 172.21.15.43:0/3757308694 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x560747a67e00 0x560747a6a220 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=993b51e0a60e36c3 server_cookie=1502cc0f813b422f in_seq=0 out_seq=0 2024-03-28T22:47:46.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.130+0000 7f6c9cc00640 10 client.29132 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:47:46.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.131+0000 7f6c9cc00640 1 -- 172.21.15.43:0/3757308694 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56074795b680 con 0x560747a67e00 2024-03-28T22:47:46.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.131+0000 7f6c9cc00640 10 client.29132 handle_command_reply: tid=0 2024-03-28T22:47:46.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.131+0000 7f6cb8409740 1 client.29132 shutdown 2024-03-28T22:47:46.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.131+0000 7f6cb8409740 1 -- 172.21.15.43:0/3757308694 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560747a4eca0 msgr2=0x560747a54da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.131+0000 7f6cb8409740 1 --2- 172.21.15.43:0/3757308694 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560747a4eca0 0x560747a54da0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f6c98012700 tx=0x7f6c98012bc0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.131+0000 7f6cb8409740 1 -- 172.21.15.43:0/3757308694 shutdown_connections 2024-03-28T22:47:46.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.131+0000 7f6cb8409740 1 --2- 172.21.15.43:0/3757308694 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x560747a67e00 0x560747a6a220 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.131+0000 7f6cb8409740 1 --2- 172.21.15.43:0/3757308694 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560747a4eca0 0x560747a54da0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.131+0000 7f6cb8409740 1 --2- 172.21.15.43:0/3757308694 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560747a57880 0x560747a552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.131+0000 7f6cb8409740 1 --2- 172.21.15.43:0/3757308694 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560747a4e380 0x560747a54860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.131+0000 7f6c9cc00640 0 client.29132 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:47:46.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.131+0000 7f6cb8409740 1 -- 172.21.15.43:0/3757308694 >> 172.21.15.43:0/3757308694 conn(0x560747a48ab0 msgr2=0x560747a4a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:46.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.132+0000 7f6cb8409740 1 -- 172.21.15.43:0/3757308694 shutdown_connections 2024-03-28T22:47:46.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.132+0000 7f6cb8409740 1 -- 172.21.15.43:0/3757308694 wait complete. 2024-03-28T22:47:46.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.132+0000 7f6cb8409740 20 client.29132 trim_cache size 0 max 16384 2024-03-28T22:47:46.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.146+0000 7f6cb8409740 1 Processor -- start 2024-03-28T22:47:46.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.147+0000 7f6cb8409740 1 -- start start 2024-03-28T22:47:46.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.147+0000 7f6cb8409740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560747a4e380 0x560747a56f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.147+0000 7f6cb8409740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560747a4eca0 0x560747a633e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.147+0000 7f6cb8409740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560747a57880 0x560747a539c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.147+0000 7f6cb8409740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb01b53e0 con 0x560747a4eca0 2024-03-28T22:47:46.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.147+0000 7f6cb8409740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb01b6f20 con 0x560747a57880 2024-03-28T22:47:46.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.147+0000 7f6cb8409740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb01b5ab0 con 0x560747a4e380 2024-03-28T22:47:46.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.147+0000 7f6ca6000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560747a4eca0 0x560747a633e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.147+0000 7f6ca6000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560747a4eca0 0x560747a633e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37316/0 (socket says 172.21.15.43:37316) 2024-03-28T22:47:46.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.147+0000 7f6ca6000640 1 -- 172.21.15.43:0/2633640357 learned_addr learned my addr 172.21.15.43:0/2633640357 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:46.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.147+0000 7f6caf400640 1 --2- 172.21.15.43:0/2633640357 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560747a4e380 0x560747a56f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.147+0000 7f6caf400640 1 --2- 172.21.15.43:0/2633640357 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560747a57880 0x560747a539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.147+0000 7f6ca6000640 1 -- 172.21.15.43:0/2633640357 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560747a4e380 msgr2=0x560747a56f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:46.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.147+0000 7f6ca6000640 1 --2- 172.21.15.43:0/2633640357 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560747a4e380 0x560747a56f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.148+0000 7f6ca6000640 1 -- 172.21.15.43:0/2633640357 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560747a57880 msgr2=0x560747a539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:46.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.148+0000 7f6ca6000640 1 --2- 172.21.15.43:0/2633640357 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560747a57880 0x560747a539c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.148+0000 7f6ca6000640 1 -- 172.21.15.43:0/2633640357 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560747a4ead0 con 0x560747a4eca0 2024-03-28T22:47:46.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.148+0000 7f6caf400640 1 --2- 172.21.15.43:0/2633640357 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560747a57880 0x560747a539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:47:46.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.148+0000 7f6ca6000640 1 --2- 172.21.15.43:0/2633640357 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560747a4eca0 0x560747a633e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6c98010800 tx=0x7f6c98009620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=838d295e42fb61a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:46.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.148+0000 7f6c9f400640 1 -- 172.21.15.43:0/2633640357 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c98010430 con 0x560747a4eca0 2024-03-28T22:47:46.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.148+0000 7f6c9f400640 1 -- 172.21.15.43:0/2633640357 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c980105d0 con 0x560747a4eca0 2024-03-28T22:47:46.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.148+0000 7f6c9f400640 1 -- 172.21.15.43:0/2633640357 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c9800f7d0 con 0x560747a4eca0 2024-03-28T22:47:46.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.149+0000 7f6cb8409740 1 -- 172.21.15.43:0/2633640357 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560747a4eca0 msgr2=0x560747a633e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.149+0000 7f6cb8409740 1 --2- 172.21.15.43:0/2633640357 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560747a4eca0 0x560747a633e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6c98010800 tx=0x7f6c98009620 comp rx=0 tx=0).stop 2024-03-28T22:47:46.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.149+0000 7f6cb8409740 1 -- 172.21.15.43:0/2633640357 shutdown_connections 2024-03-28T22:47:46.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.149+0000 7f6cb8409740 1 --2- 172.21.15.43:0/2633640357 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560747a57880 0x560747a539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.149+0000 7f6cb8409740 1 --2- 172.21.15.43:0/2633640357 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560747a4eca0 0x560747a633e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.149+0000 7f6cb8409740 1 --2- 172.21.15.43:0/2633640357 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560747a4e380 0x560747a56f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.149+0000 7f6cb8409740 1 -- 172.21.15.43:0/2633640357 >> 172.21.15.43:0/2633640357 conn(0x560747a48ab0 msgr2=0x560747a472e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:46.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.149+0000 7f6cb8409740 1 -- 172.21.15.43:0/2633640357 shutdown_connections 2024-03-28T22:47:46.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.149+0000 7f6cb8409740 1 -- 172.21.15.43:0/2633640357 wait complete. 2024-03-28T22:47:46.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.149+0000 7f6cb8409740 1 -- start start 2024-03-28T22:47:46.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.150+0000 7f6cb8409740 1 Processor -- start 2024-03-28T22:47:46.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.150+0000 7f6cb8409740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560747a4e380 0x560747a512e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.150+0000 7f6cb8409740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560747a57880 0x560747a51820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.150+0000 7f6cb8409740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560747a51d60 0x560747acd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.150+0000 7f6cb8409740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb01b6850 con 0x560747a57880 2024-03-28T22:47:46.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.150+0000 7f6cb8409740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb01b6180 con 0x560747a51d60 2024-03-28T22:47:46.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.150+0000 7f6cb8409740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb01b33e0 con 0x560747a4e380 2024-03-28T22:47:46.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.150+0000 7f6cb8409740 10 client.0 _ll_register_callbacks cb 0x7f6c98009830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:47:46.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.150+0000 7f6ca6000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560747a57880 0x560747a51820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.150+0000 7f6caf400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560747a4e380 0x560747a512e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.150+0000 7f6ca6000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560747a57880 0x560747a51820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37320/0 (socket says 172.21.15.43:37320) 2024-03-28T22:47:46.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.150+0000 7f6ca6000640 1 -- 172.21.15.43:0/4119631321 learned_addr learned my addr 172.21.15.43:0/4119631321 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:46.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.150+0000 7f6caf400640 1 --2- 172.21.15.43:0/4119631321 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560747a51d60 0x560747acd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.151+0000 7f6ca6000640 1 -- 172.21.15.43:0/4119631321 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560747a4e380 msgr2=0x560747a512e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:46.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.151+0000 7f6ca6000640 1 --2- 172.21.15.43:0/4119631321 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560747a4e380 0x560747a512e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.151+0000 7f6ca6000640 1 -- 172.21.15.43:0/4119631321 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560747a51d60 msgr2=0x560747acd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:46.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.151+0000 7f6ca6000640 1 --2- 172.21.15.43:0/4119631321 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560747a51d60 0x560747acd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.151+0000 7f6ca6000640 1 -- 172.21.15.43:0/4119631321 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560747acd640 con 0x560747a57880 2024-03-28T22:47:46.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.151+0000 7f6ca6000640 1 --2- 172.21.15.43:0/4119631321 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560747a57880 0x560747a51820 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6c98010780 tx=0x7f6c98005fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a9c47ade9dc884f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:46.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.151+0000 7f6c9cc00640 10 client.0 ms_handle_connect on v2:172.21.15.43:3300/0 2024-03-28T22:47:46.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.151+0000 7f6c9cc00640 1 -- 172.21.15.43:0/4119631321 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c980052e0 con 0x560747a57880 2024-03-28T22:47:46.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.151+0000 7f6c9cc00640 1 -- 172.21.15.43:0/4119631321 <== mon.0 v2:172.21.15.43:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9539+0+0 (secure 0 0 0) 0x7f6c98005480 con 0x560747a57880 2024-03-28T22:47:46.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.151+0000 7f6cb8409740 1 -- 172.21.15.43:0/4119631321 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c980105d0 con 0x560747a57880 2024-03-28T22:47:46.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.152+0000 7f6c9cc00640 1 -- 172.21.15.43:0/4119631321 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6c98005680 con 0x560747a57880 2024-03-28T22:47:46.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.152+0000 7f6cb8409740 10 client.18864 fetch_fsmap learned FSMap version 32 2024-03-28T22:47:46.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.152+0000 7f6cb8409740 1 -- 172.21.15.43:0/4119631321 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560747ace670 con 0x560747a57880 2024-03-28T22:47:46.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.152+0000 7f6c9cc00640 1 -- 172.21.15.43:0/4119631321 <== mon.0 v2:172.21.15.43:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c980105d0 con 0x560747a57880 2024-03-28T22:47:46.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.152+0000 7f6c9cc00640 1 -- 172.21.15.43:0/4119631321 <== mon.0 v2:172.21.15.43:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6c98003a10 con 0x560747a57880 2024-03-28T22:47:46.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.152+0000 7f6cb8409740 10 client.18864 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:47:46.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.152+0000 7f6cb8409740 10 client.18864 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-28T22:47:46.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.152+0000 7f6cb8409740 20 client.18864 populate_metadata read hostname 'smithi043' 2024-03-28T22:47:46.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.152+0000 7f6cb8409740 1 --2- 172.21.15.43:0/4119631321 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x560747aced00 0x560747ad1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.152+0000 7f6cb8409740 4 client.18864 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:47:46.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.152+0000 7f6cb8409740 1 -- 172.21.15.43:0/4119631321 --> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5607479bf590 con 0x560747aced00 2024-03-28T22:47:46.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.152+0000 7f6caf400640 1 --2- 172.21.15.43:0/4119631321 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x560747aced00 0x560747ad1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.153+0000 7f6caf400640 1 --2- 172.21.15.43:0/4119631321 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x560747aced00 0x560747ad1120 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=258bdc83ec45f3f8 server_cookie=1fc9f8fa0aca8a2c in_seq=0 out_seq=0 2024-03-28T22:47:46.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.153+0000 7f6c9cc00640 10 client.18864 ms_handle_connect on v2:172.21.15.172:6834/318854103 2024-03-28T22:47:46.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.153+0000 7f6c9cc00640 1 -- 172.21.15.43:0/4119631321 <== mds.3 v2:172.21.15.172:6834/318854103 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5607479bf590 con 0x560747aced00 2024-03-28T22:47:46.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.153+0000 7f6c9cc00640 10 client.18864 handle_command_reply: tid=0 2024-03-28T22:47:46.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.153+0000 7f6cb8409740 1 client.18864 shutdown 2024-03-28T22:47:46.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.153+0000 7f6cb8409740 1 -- 172.21.15.43:0/4119631321 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560747a57880 msgr2=0x560747a51820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.153+0000 7f6cb8409740 1 --2- 172.21.15.43:0/4119631321 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560747a57880 0x560747a51820 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6c98010780 tx=0x7f6c98005fa0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.153+0000 7f6cb8409740 1 -- 172.21.15.43:0/4119631321 shutdown_connections 2024-03-28T22:47:46.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.153+0000 7f6cb8409740 1 --2- 172.21.15.43:0/4119631321 >> [v2:172.21.15.172:6834/318854103,v1:172.21.15.172:6835/318854103] conn(0x560747aced00 0x560747ad1120 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.153+0000 7f6cb8409740 1 --2- 172.21.15.43:0/4119631321 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560747a51d60 0x560747acd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.153+0000 7f6cb8409740 1 --2- 172.21.15.43:0/4119631321 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x560747a57880 0x560747a51820 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.153+0000 7f6cb8409740 1 --2- 172.21.15.43:0/4119631321 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x560747a4e380 0x560747a512e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.153+0000 7f6c9cc00640 0 client.18864 ms_handle_reset on v2:172.21.15.172:6834/318854103 2024-03-28T22:47:46.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.153+0000 7f6cb8409740 1 -- 172.21.15.43:0/4119631321 >> 172.21.15.43:0/4119631321 conn(0x560747a48ab0 msgr2=0x560747a472e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:46.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.154+0000 7f6cb8409740 1 -- 172.21.15.43:0/4119631321 shutdown_connections 2024-03-28T22:47:46.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.154+0000 7f6cb8409740 1 -- 172.21.15.43:0/4119631321 wait complete. 2024-03-28T22:47:46.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.154+0000 7f6cb8409740 20 client.18864 trim_cache size 0 max 16384 2024-03-28T22:47:46.161 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:47:46.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.154+0000 7f6cb5800640 1 -- 172.21.15.43:0/3231251705 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6c940a1f70 msgr2=0x7f6c940a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.154+0000 7f6cb5800640 1 --2- 172.21.15.43:0/3231251705 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6c940a1f70 0x7f6c940a4430 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x560747a4de50 tx=0x7f6ca00d26d0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.154+0000 7f6cb5800640 1 -- 172.21.15.43:0/3231251705 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6cb0075c80 msgr2=0x7f6cb019e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.154+0000 7f6cb5800640 1 --2- 172.21.15.43:0/3231251705 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6cb0075c80 0x7f6cb019e680 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f6ca000ebc0 tx=0x7f6ca000c6a0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.155+0000 7f6cb5800640 1 -- 172.21.15.43:0/3231251705 shutdown_connections 2024-03-28T22:47:46.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.155+0000 7f6cb5800640 1 --2- 172.21.15.43:0/3231251705 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f6c940a1f70 0x7f6c940a4430 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.155+0000 7f6cb5800640 1 --2- 172.21.15.43:0/3231251705 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6cb0112590 0x7f6cb01a2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.155+0000 7f6cb5800640 1 --2- 172.21.15.43:0/3231251705 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6cb00765a0 0x7f6cb019ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.155+0000 7f6cb5800640 1 --2- 172.21.15.43:0/3231251705 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6cb0075c80 0x7f6cb019e680 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.155+0000 7f6cb5800640 1 -- 172.21.15.43:0/3231251705 >> 172.21.15.43:0/3231251705 conn(0x7f6cb00ff3a0 msgr2=0x7f6cb0101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:46.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.155+0000 7f6cb5800640 1 -- 172.21.15.43:0/3231251705 shutdown_connections 2024-03-28T22:47:46.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.155+0000 7f6cb5800640 1 -- 172.21.15.43:0/3231251705 wait complete. 2024-03-28T22:47:46.168 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:45 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3948370862' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:47:46.171 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T22:47:46.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:45 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3948370862' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:47:46.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.392+0000 7fce14e00640 1 -- 172.21.15.43:0/1059139501 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fce10073b00 msgr2=0x7fce10073ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.392+0000 7fce14e00640 1 --2- 172.21.15.43:0/1059139501 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fce10073b00 0x7fce10073ee0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fce00009ca0 tx=0x7fce0002c180 comp rx=0 tx=0).stop 2024-03-28T22:47:46.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.392+0000 7fce14e00640 1 -- 172.21.15.43:0/1059139501 shutdown_connections 2024-03-28T22:47:46.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.392+0000 7fce14e00640 1 --2- 172.21.15.43:0/1059139501 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fce10074420 0x7fce1010d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.392+0000 7fce14e00640 1 --2- 172.21.15.43:0/1059139501 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce1010de80 0x7fce10110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.392+0000 7fce14e00640 1 --2- 172.21.15.43:0/1059139501 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fce10073b00 0x7fce10073ee0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.392+0000 7fce14e00640 1 -- 172.21.15.43:0/1059139501 >> 172.21.15.43:0/1059139501 conn(0x7fce100fd0f0 msgr2=0x7fce100ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:46.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.393+0000 7fce14e00640 1 -- 172.21.15.43:0/1059139501 shutdown_connections 2024-03-28T22:47:46.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.393+0000 7fce14e00640 1 -- 172.21.15.43:0/1059139501 wait complete. 2024-03-28T22:47:46.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.394+0000 7fce14e00640 1 Processor -- start 2024-03-28T22:47:46.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.394+0000 7fce14e00640 1 -- start start 2024-03-28T22:47:46.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.394+0000 7fce14e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce10073b00 0x7fce10199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.394+0000 7fce14e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fce10074420 0x7fce1019a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.394+0000 7fce14e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fce1010de80 0x7fce1019e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.395+0000 7fce14e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fce10113ba0 con 0x7fce10073b00 2024-03-28T22:47:46.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.395+0000 7fce14e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fce10113a20 con 0x7fce1010de80 2024-03-28T22:47:46.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.395+0000 7fce14e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fce10113d20 con 0x7fce10074420 2024-03-28T22:47:46.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.395+0000 7fce0f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce10073b00 0x7fce10199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.395+0000 7fce0ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fce10074420 0x7fce1019a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.395+0000 7fce0fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fce1010de80 0x7fce1019e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.395+0000 7fce0f400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce10073b00 0x7fce10199fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37336/0 (socket says 172.21.15.43:37336) 2024-03-28T22:47:46.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.395+0000 7fce0f400640 1 -- 172.21.15.43:0/315650880 learned_addr learned my addr 172.21.15.43:0/315650880 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:46.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.395+0000 7fce0f400640 1 -- 172.21.15.43:0/315650880 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fce10074420 msgr2=0x7fce1019a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.395+0000 7fce0f400640 1 --2- 172.21.15.43:0/315650880 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fce10074420 0x7fce1019a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.395+0000 7fce0f400640 1 -- 172.21.15.43:0/315650880 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fce1010de80 msgr2=0x7fce1019e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.395+0000 7fce0f400640 1 --2- 172.21.15.43:0/315650880 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fce1010de80 0x7fce1019e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.395+0000 7fce0f400640 1 -- 172.21.15.43:0/315650880 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce1019ef90 con 0x7fce10073b00 2024-03-28T22:47:46.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.395+0000 7fce0fe00640 1 --2- 172.21.15.43:0/315650880 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fce1010de80 0x7fce1019e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:46.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.396+0000 7fce0f400640 1 --2- 172.21.15.43:0/315650880 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce10073b00 0x7fce10199fa0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fce00000f80 tx=0x7fce00003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:46.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.396+0000 7fce07e00640 1 -- 172.21.15.43:0/315650880 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce00039040 con 0x7fce10073b00 2024-03-28T22:47:46.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.396+0000 7fce14e00640 1 -- 172.21.15.43:0/315650880 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce1019f220 con 0x7fce10073b00 2024-03-28T22:47:46.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.396+0000 7fce07e00640 1 -- 172.21.15.43:0/315650880 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce00044070 con 0x7fce10073b00 2024-03-28T22:47:46.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.396+0000 7fce07e00640 1 -- 172.21.15.43:0/315650880 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce00032b10 con 0x7fce10073b00 2024-03-28T22:47:46.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.396+0000 7fce14e00640 1 -- 172.21.15.43:0/315650880 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce101a6b10 con 0x7fce10073b00 2024-03-28T22:47:46.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:45 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3948370862' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:47:46.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.400+0000 7fce18ff7740 1 Processor -- start 2024-03-28T22:47:46.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.400+0000 7fce07e00640 1 -- 172.21.15.43:0/315650880 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fce00032cb0 con 0x7fce10073b00 2024-03-28T22:47:46.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.400+0000 7fce18ff7740 1 -- start start 2024-03-28T22:47:46.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.400+0000 7fce18ff7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac14d4e550 0x55ac14d4e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.400+0000 7fce18ff7740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac14d4ee70 0x55ac14d57510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.400+0000 7fce18ff7740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac14d57a50 0x55ac14d59e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.400+0000 7fce18ff7740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ac14cb2050 con 0x55ac14d57a50 2024-03-28T22:47:46.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.400+0000 7fce18ff7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55ac14c5ace0 con 0x55ac14d4e550 2024-03-28T22:47:46.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.400+0000 7fce18ff7740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55ac14cae0f0 con 0x55ac14d4ee70 2024-03-28T22:47:46.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.400+0000 7fce0fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac14d4e550 0x55ac14d4e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.400+0000 7fce0f400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac14d4ee70 0x55ac14d57510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.400+0000 7fce0fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac14d4e550 0x55ac14d4e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59002/0 (socket says 172.21.15.43:59002) 2024-03-28T22:47:46.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.401+0000 7fce0fe00640 1 -- 172.21.15.43:0/4006309215 learned_addr learned my addr 172.21.15.43:0/4006309215 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:46.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.401+0000 7fce0ea00640 1 --2- 172.21.15.43:0/4006309215 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac14d57a50 0x55ac14d59e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.401+0000 7fce0fe00640 1 -- 172.21.15.43:0/4006309215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac14d4ee70 msgr2=0x55ac14d57510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:46.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.401+0000 7fce0fe00640 1 --2- 172.21.15.43:0/4006309215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac14d4ee70 0x55ac14d57510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.401+0000 7fce0fe00640 1 -- 172.21.15.43:0/4006309215 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac14d57a50 msgr2=0x55ac14d59e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:46.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.401+0000 7fce0fe00640 1 --2- 172.21.15.43:0/4006309215 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac14d57a50 0x55ac14d59e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.401+0000 7fce0fe00640 1 -- 172.21.15.43:0/4006309215 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac149c63f0 con 0x55ac14d4e550 2024-03-28T22:47:46.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.401+0000 7fce0ea00640 1 --2- 172.21.15.43:0/4006309215 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac14d57a50 0x55ac14d59e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:47:46.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.401+0000 7fce0fe00640 1 --2- 172.21.15.43:0/4006309215 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac14d4e550 0x55ac14d4e930 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fce0800af90 tx=0x7fce0800f850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbc3d124faa43814 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:46.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.402+0000 7fce04c00640 1 -- 172.21.15.43:0/4006309215 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce08005220 con 0x55ac14d4e550 2024-03-28T22:47:46.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.402+0000 7fce07e00640 1 --2- 172.21.15.43:0/315650880 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcdf40a1ef0 0x7fcdf40a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.402+0000 7fce04c00640 1 -- 172.21.15.43:0/4006309215 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce08004c80 con 0x55ac14d4e550 2024-03-28T22:47:46.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.402+0000 7fce04c00640 1 -- 172.21.15.43:0/4006309215 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce08010bf0 con 0x55ac14d4e550 2024-03-28T22:47:46.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.402+0000 7fce07e00640 1 -- 172.21.15.43:0/315650880 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9539+0+0 (secure 0 0 0) 0x7fce000f0bf0 con 0x7fce10073b00 2024-03-28T22:47:46.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.402+0000 7fce0f400640 1 --2- 172.21.15.43:0/315650880 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcdf40a1ef0 0x7fcdf40a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.402+0000 7fce18ff7740 1 -- 172.21.15.43:0/4006309215 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac14d4e550 msgr2=0x55ac14d4e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.402+0000 7fce18ff7740 1 --2- 172.21.15.43:0/4006309215 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac14d4e550 0x55ac14d4e930 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fce0800af90 tx=0x7fce0800f850 comp rx=0 tx=0).stop 2024-03-28T22:47:46.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.402+0000 7fce18ff7740 1 -- 172.21.15.43:0/4006309215 shutdown_connections 2024-03-28T22:47:46.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.402+0000 7fce18ff7740 1 --2- 172.21.15.43:0/4006309215 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac14d57a50 0x55ac14d59e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.403+0000 7fce18ff7740 1 --2- 172.21.15.43:0/4006309215 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac14d4ee70 0x55ac14d57510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.403+0000 7fce18ff7740 1 --2- 172.21.15.43:0/4006309215 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac14d4e550 0x55ac14d4e930 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.403+0000 7fce18ff7740 1 -- 172.21.15.43:0/4006309215 >> 172.21.15.43:0/4006309215 conn(0x55ac14d48c80 msgr2=0x55ac14d49060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:46.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.403+0000 7fce0f400640 1 --2- 172.21.15.43:0/315650880 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcdf40a1ef0 0x7fcdf40a43b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x55ac14d4e020 tx=0x7fce000f46e0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:46.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.403+0000 7fce18ff7740 1 -- 172.21.15.43:0/4006309215 shutdown_connections 2024-03-28T22:47:46.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.403+0000 7fce18ff7740 1 -- 172.21.15.43:0/4006309215 wait complete. 2024-03-28T22:47:46.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.403+0000 7fce18ff7740 1 -- start start 2024-03-28T22:47:46.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.403+0000 7fce18ff7740 1 Processor -- start 2024-03-28T22:47:46.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.404+0000 7fce18ff7740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac14d4e550 0x55ac14d54b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.404+0000 7fce18ff7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac14d4ee70 0x55ac14d55070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.404+0000 7fce18ff7740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac14d57a50 0x55ac14d555b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.404+0000 7fce18ff7740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55ac14d5a840 con 0x55ac14d57a50 2024-03-28T22:47:46.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.404+0000 7fce18ff7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55ac14c502f0 con 0x55ac14d4ee70 2024-03-28T22:47:46.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.404+0000 7fce18ff7740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55ac14d5a9c0 con 0x55ac14d4e550 2024-03-28T22:47:46.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.404+0000 7fce18ff7740 10 client.0 _ll_register_callbacks cb 0x55ac149c5be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:47:46.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.404+0000 7fce0ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac14d4ee70 0x55ac14d55070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.404+0000 7fce0fe00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac14d4e550 0x55ac14d54b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.404+0000 7fce0ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac14d4ee70 0x55ac14d55070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59016/0 (socket says 172.21.15.43:59016) 2024-03-28T22:47:46.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.404+0000 7fce0ea00640 1 -- 172.21.15.43:0/312978096 learned_addr learned my addr 172.21.15.43:0/312978096 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:46.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.404+0000 7fce0fe00640 1 --2- 172.21.15.43:0/312978096 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac14d57a50 0x55ac14d555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.404+0000 7fce0ea00640 1 -- 172.21.15.43:0/312978096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac14d4e550 msgr2=0x55ac14d54b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:46.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.404+0000 7fce0ea00640 1 --2- 172.21.15.43:0/312978096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac14d4e550 0x55ac14d54b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.405+0000 7fce0ea00640 1 -- 172.21.15.43:0/312978096 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac14d57a50 msgr2=0x55ac14d555b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:46.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.405+0000 7fce0ea00640 1 --2- 172.21.15.43:0/312978096 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac14d57a50 0x55ac14d555b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.405+0000 7fce0ea00640 1 -- 172.21.15.43:0/312978096 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac14a907d0 con 0x55ac14d4ee70 2024-03-28T22:47:46.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.405+0000 7fce0fe00640 1 --2- 172.21.15.43:0/312978096 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac14d57a50 0x55ac14d555b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:47:46.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.405+0000 7fce0ea00640 1 --2- 172.21.15.43:0/312978096 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac14d4ee70 0x55ac14d55070 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8012780 tx=0x7fcdf8012c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1991bcacf7aeb53 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:46.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.405+0000 7fcdfd600640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:47:46.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.405+0000 7fcdfd600640 1 -- 172.21.15.43:0/312978096 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf801f060 con 0x55ac14d4ee70 2024-03-28T22:47:46.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.405+0000 7fce18ff7740 1 -- 172.21.15.43:0/312978096 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce08004c80 con 0x55ac14d4ee70 2024-03-28T22:47:46.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.405+0000 7fcdfd600640 1 -- 172.21.15.43:0/312978096 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9539+0+0 (secure 0 0 0) 0x7fcdf8010d60 con 0x55ac14d4ee70 2024-03-28T22:47:46.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.405+0000 7fcdfd600640 1 -- 172.21.15.43:0/312978096 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcdf8026050 con 0x55ac14d4ee70 2024-03-28T22:47:46.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.405+0000 7fce18ff7740 10 client.29144 fetch_fsmap learned FSMap version 32 2024-03-28T22:47:46.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.405+0000 7fce18ff7740 1 -- 172.21.15.43:0/312978096 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ac14d66c50 con 0x55ac14d4ee70 2024-03-28T22:47:46.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.405+0000 7fcdfd600640 1 -- 172.21.15.43:0/312978096 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce08004c80 con 0x55ac14d4ee70 2024-03-28T22:47:46.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.406+0000 7fcdfd600640 1 -- 172.21.15.43:0/312978096 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fcdf800e5e0 con 0x55ac14d4ee70 2024-03-28T22:47:46.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.406+0000 7fce18ff7740 10 client.29144 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:47:46.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.406+0000 7fce18ff7740 10 client.29144 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:47:46.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.406+0000 7fce18ff7740 20 client.29144 populate_metadata read hostname 'smithi043' 2024-03-28T22:47:46.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.406+0000 7fce18ff7740 1 --2- 172.21.15.43:0/312978096 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55ac14d680b0 0x55ac14d6a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.406+0000 7fce18ff7740 4 client.29144 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:47:46.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.406+0000 7fce18ff7740 1 -- 172.21.15.43:0/312978096 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac14c5b680 con 0x55ac14d680b0 2024-03-28T22:47:46.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.406+0000 7fce0fe00640 1 --2- 172.21.15.43:0/312978096 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55ac14d680b0 0x55ac14d6a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.407+0000 7fce0fe00640 1 --2- 172.21.15.43:0/312978096 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55ac14d680b0 0x55ac14d6a4d0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f538eb6f02f1a8c2 server_cookie=9fa745f2bf28008d in_seq=0 out_seq=0 2024-03-28T22:47:46.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.407+0000 7fcdfd600640 10 client.29144 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:47:46.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.408+0000 7fcdfd600640 1 -- 172.21.15.43:0/312978096 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ac14c5b680 con 0x55ac14d680b0 2024-03-28T22:47:46.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.408+0000 7fcdfd600640 10 client.29144 handle_command_reply: tid=0 2024-03-28T22:47:46.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.408+0000 7fce18ff7740 1 client.29144 shutdown 2024-03-28T22:47:46.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.408+0000 7fce18ff7740 1 -- 172.21.15.43:0/312978096 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac14d4ee70 msgr2=0x55ac14d55070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.408+0000 7fce18ff7740 1 --2- 172.21.15.43:0/312978096 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac14d4ee70 0x55ac14d55070 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8012780 tx=0x7fcdf8012c40 comp rx=0 tx=0).stop 2024-03-28T22:47:46.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.408+0000 7fce18ff7740 1 -- 172.21.15.43:0/312978096 shutdown_connections 2024-03-28T22:47:46.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.408+0000 7fce18ff7740 1 --2- 172.21.15.43:0/312978096 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55ac14d680b0 0x55ac14d6a4d0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.408+0000 7fce18ff7740 1 --2- 172.21.15.43:0/312978096 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac14d4ee70 0x55ac14d55070 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.408+0000 7fcdfd600640 0 client.29144 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:47:46.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.408+0000 7fce18ff7740 1 --2- 172.21.15.43:0/312978096 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac14d57a50 0x55ac14d555b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.408+0000 7fce18ff7740 1 --2- 172.21.15.43:0/312978096 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac14d4e550 0x55ac14d54b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.408+0000 7fce18ff7740 1 -- 172.21.15.43:0/312978096 >> 172.21.15.43:0/312978096 conn(0x55ac14d48c80 msgr2=0x55ac14d4a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:46.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.408+0000 7fce18ff7740 1 -- 172.21.15.43:0/312978096 shutdown_connections 2024-03-28T22:47:46.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.408+0000 7fce18ff7740 1 -- 172.21.15.43:0/312978096 wait complete. 2024-03-28T22:47:46.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.408+0000 7fce18ff7740 20 client.29144 trim_cache size 0 max 16384 2024-03-28T22:47:46.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.423+0000 7fce18ff7740 1 Processor -- start 2024-03-28T22:47:46.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.423+0000 7fce18ff7740 1 -- start start 2024-03-28T22:47:46.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.423+0000 7fce18ff7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac14d4e550 0x55ac14d57260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.423+0000 7fce18ff7740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac14d4ee70 0x55ac14d53c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.423+0000 7fce18ff7740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac14d57a50 0x55ac14d635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.424+0000 7fce18ff7740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fce101b0c40 con 0x55ac14d4ee70 2024-03-28T22:47:46.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.424+0000 7fce18ff7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fce101b2780 con 0x55ac14d4e550 2024-03-28T22:47:46.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.424+0000 7fce18ff7740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fce101b1310 con 0x55ac14d57a50 2024-03-28T22:47:46.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.424+0000 7fce0fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac14d4e550 0x55ac14d57260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.424+0000 7fce0fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac14d4e550 0x55ac14d57260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59020/0 (socket says 172.21.15.43:59020) 2024-03-28T22:47:46.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.424+0000 7fce0fe00640 1 -- 172.21.15.43:0/2722193537 learned_addr learned my addr 172.21.15.43:0/2722193537 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:46.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.424+0000 7fce0fe00640 1 --2- 172.21.15.43:0/2722193537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac14d57a50 0x55ac14d635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.424+0000 7fce0ea00640 1 --2- 172.21.15.43:0/2722193537 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac14d4ee70 0x55ac14d53c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.424+0000 7fce0fe00640 1 -- 172.21.15.43:0/2722193537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac14d57a50 msgr2=0x55ac14d635e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:46.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.424+0000 7fce0fe00640 1 --2- 172.21.15.43:0/2722193537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac14d57a50 0x55ac14d635e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.424+0000 7fce0fe00640 1 -- 172.21.15.43:0/2722193537 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac14d4ee70 msgr2=0x55ac14d53c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:46.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.424+0000 7fce0fe00640 1 --2- 172.21.15.43:0/2722193537 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac14d4ee70 0x55ac14d53c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.424+0000 7fce0fe00640 1 -- 172.21.15.43:0/2722193537 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac14d4eca0 con 0x55ac14d4e550 2024-03-28T22:47:46.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.424+0000 7fce0ea00640 1 --2- 172.21.15.43:0/2722193537 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac14d4ee70 0x55ac14d53c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:47:46.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.424+0000 7fce0fe00640 1 --2- 172.21.15.43:0/2722193537 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac14d4e550 0x55ac14d57260 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fce08000c00 tx=0x7fce08005d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6dc6e694037e1ba1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:46.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.425+0000 7fcdffe00640 1 -- 172.21.15.43:0/2722193537 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce08005100 con 0x55ac14d4e550 2024-03-28T22:47:46.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.425+0000 7fcdffe00640 1 -- 172.21.15.43:0/2722193537 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce080052a0 con 0x55ac14d4e550 2024-03-28T22:47:46.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.425+0000 7fcdffe00640 1 -- 172.21.15.43:0/2722193537 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce08003b50 con 0x55ac14d4e550 2024-03-28T22:47:46.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.425+0000 7fce18ff7740 1 -- 172.21.15.43:0/2722193537 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac14d4e550 msgr2=0x55ac14d57260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.425+0000 7fce18ff7740 1 --2- 172.21.15.43:0/2722193537 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac14d4e550 0x55ac14d57260 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fce08000c00 tx=0x7fce08005d20 comp rx=0 tx=0).stop 2024-03-28T22:47:46.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.425+0000 7fce18ff7740 1 -- 172.21.15.43:0/2722193537 shutdown_connections 2024-03-28T22:47:46.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.425+0000 7fce18ff7740 1 --2- 172.21.15.43:0/2722193537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac14d57a50 0x55ac14d635e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.425+0000 7fce18ff7740 1 --2- 172.21.15.43:0/2722193537 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac14d4ee70 0x55ac14d53c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.425+0000 7fce18ff7740 1 --2- 172.21.15.43:0/2722193537 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac14d4e550 0x55ac14d57260 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.425+0000 7fce18ff7740 1 -- 172.21.15.43:0/2722193537 >> 172.21.15.43:0/2722193537 conn(0x55ac14d48c80 msgr2=0x55ac14d47530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:46.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.425+0000 7fce18ff7740 1 -- 172.21.15.43:0/2722193537 shutdown_connections 2024-03-28T22:47:46.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.425+0000 7fce18ff7740 1 -- 172.21.15.43:0/2722193537 wait complete. 2024-03-28T22:47:46.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.426+0000 7fce18ff7740 1 -- start start 2024-03-28T22:47:46.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.426+0000 7fce18ff7740 1 Processor -- start 2024-03-28T22:47:46.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.426+0000 7fce18ff7740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac14d4e550 0x55ac14d5fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.426+0000 7fce18ff7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac14d4ee70 0x55ac14d5b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.427+0000 7fce18ff7740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac14d57a50 0x55ac14d5baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.427+0000 7fce18ff7740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fce101b20b0 con 0x55ac14d4e550 2024-03-28T22:47:46.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.427+0000 7fce18ff7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fce101b19e0 con 0x55ac14d4ee70 2024-03-28T22:47:46.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.427+0000 7fce18ff7740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fce101aec40 con 0x55ac14d57a50 2024-03-28T22:47:46.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.427+0000 7fce18ff7740 10 client.0 _ll_register_callbacks cb 0x7fcdf800ae60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:47:46.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.427+0000 7fce0ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac14d4ee70 0x55ac14d5b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.427+0000 7fce0ea00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac14d4ee70 0x55ac14d5b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59024/0 (socket says 172.21.15.43:59024) 2024-03-28T22:47:46.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.427+0000 7fce0ea00640 1 -- 172.21.15.43:0/3043506388 learned_addr learned my addr 172.21.15.43:0/3043506388 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:46.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.427+0000 7fce0fe00640 1 --2- 172.21.15.43:0/3043506388 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac14d4e550 0x55ac14d5fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.427+0000 7fce0fe00640 1 --2- 172.21.15.43:0/3043506388 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac14d57a50 0x55ac14d5baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.427+0000 7fce0ea00640 1 -- 172.21.15.43:0/3043506388 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac14d57a50 msgr2=0x55ac14d5baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:46.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.427+0000 7fce0ea00640 1 --2- 172.21.15.43:0/3043506388 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac14d57a50 0x55ac14d5baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.427+0000 7fce0ea00640 1 -- 172.21.15.43:0/3043506388 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac14d4e550 msgr2=0x55ac14d5fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:46.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.427+0000 7fce0ea00640 1 --2- 172.21.15.43:0/3043506388 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac14d4e550 0x55ac14d5fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.427+0000 7fce0ea00640 1 -- 172.21.15.43:0/3043506388 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac14dd5ba0 con 0x55ac14d4ee70 2024-03-28T22:47:46.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.427+0000 7fce0fe00640 1 --2- 172.21.15.43:0/3043506388 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac14d57a50 0x55ac14d5baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T22:47:46.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.427+0000 7fce0ea00640 1 --2- 172.21.15.43:0/3043506388 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac14d4ee70 0x55ac14d5b360 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x55ac14d56100 tx=0x7fcdf800eed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c64289335b7f662 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:46.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.427+0000 7fcdfd600640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:47:46.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.428+0000 7fcdfd600640 1 -- 172.21.15.43:0/3043506388 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf800e3b0 con 0x55ac14d4ee70 2024-03-28T22:47:46.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.428+0000 7fcdfd600640 1 -- 172.21.15.43:0/3043506388 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9539+0+0 (secure 0 0 0) 0x7fcdf800e550 con 0x55ac14d4ee70 2024-03-28T22:47:46.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.428+0000 7fce18ff7740 1 -- 172.21.15.43:0/3043506388 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce080052a0 con 0x55ac14d4ee70 2024-03-28T22:47:46.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.428+0000 7fcdfd600640 1 -- 172.21.15.43:0/3043506388 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcdf801f030 con 0x55ac14d4ee70 2024-03-28T22:47:46.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.428+0000 7fce18ff7740 10 client.29150 fetch_fsmap learned FSMap version 32 2024-03-28T22:47:46.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.428+0000 7fce18ff7740 1 -- 172.21.15.43:0/3043506388 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ac14dd6cd0 con 0x55ac14d4ee70 2024-03-28T22:47:46.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.428+0000 7fcdfd600640 1 -- 172.21.15.43:0/3043506388 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce080052a0 con 0x55ac14d4ee70 2024-03-28T22:47:46.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.428+0000 7fcdfd600640 1 -- 172.21.15.43:0/3043506388 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fcdf8030020 con 0x55ac14d4ee70 2024-03-28T22:47:46.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.428+0000 7fce18ff7740 10 client.29150 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:47:46.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.428+0000 7fce18ff7740 10 client.29150 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T22:47:46.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.428+0000 7fce18ff7740 20 client.29150 populate_metadata read hostname 'smithi043' 2024-03-28T22:47:46.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.429+0000 7fce18ff7740 1 --2- 172.21.15.43:0/3043506388 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55ac14dd7480 0x55ac14dd98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.429+0000 7fce18ff7740 4 client.29150 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:47:46.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.429+0000 7fce18ff7740 1 -- 172.21.15.43:0/3043506388 --> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ac14cbf590 con 0x55ac14dd7480 2024-03-28T22:47:46.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.429+0000 7fce0fe00640 1 --2- 172.21.15.43:0/3043506388 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55ac14dd7480 0x55ac14dd98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.429+0000 7fce0fe00640 1 --2- 172.21.15.43:0/3043506388 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55ac14dd7480 0x55ac14dd98a0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1950b1f91c667a3e server_cookie=e3a3f922b5e6671c in_seq=0 out_seq=0 2024-03-28T22:47:46.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.429+0000 7fcdfd600640 10 client.29150 ms_handle_connect on v2:172.21.15.172:6836/3745415724 2024-03-28T22:47:46.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.429+0000 7fcdfd600640 1 -- 172.21.15.43:0/3043506388 <== mds.2 v2:172.21.15.172:6836/3745415724 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ac14cbf590 con 0x55ac14dd7480 2024-03-28T22:47:46.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.429+0000 7fcdfd600640 10 client.29150 handle_command_reply: tid=0 2024-03-28T22:47:46.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.429+0000 7fce18ff7740 1 client.29150 shutdown 2024-03-28T22:47:46.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.430+0000 7fce18ff7740 1 -- 172.21.15.43:0/3043506388 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac14d4ee70 msgr2=0x55ac14d5b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.430+0000 7fce18ff7740 1 --2- 172.21.15.43:0/3043506388 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac14d4ee70 0x55ac14d5b360 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x55ac14d56100 tx=0x7fcdf800eed0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.430+0000 7fce18ff7740 1 -- 172.21.15.43:0/3043506388 shutdown_connections 2024-03-28T22:47:46.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.430+0000 7fce18ff7740 1 --2- 172.21.15.43:0/3043506388 >> [v2:172.21.15.172:6836/3745415724,v1:172.21.15.172:6837/3745415724] conn(0x55ac14dd7480 0x55ac14dd98a0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.430+0000 7fce18ff7740 1 --2- 172.21.15.43:0/3043506388 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ac14d4ee70 0x55ac14d5b360 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.430+0000 7fce18ff7740 1 --2- 172.21.15.43:0/3043506388 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55ac14d57a50 0x55ac14d5baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.430+0000 7fce18ff7740 1 --2- 172.21.15.43:0/3043506388 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55ac14d4e550 0x55ac14d5fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.430+0000 7fce18ff7740 1 -- 172.21.15.43:0/3043506388 >> 172.21.15.43:0/3043506388 conn(0x55ac14d48c80 msgr2=0x55ac14d55360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:46.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.430+0000 7fcdfd600640 0 client.29150 ms_handle_reset on v2:172.21.15.172:6836/3745415724 2024-03-28T22:47:46.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.430+0000 7fce18ff7740 1 -- 172.21.15.43:0/3043506388 shutdown_connections 2024-03-28T22:47:46.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.430+0000 7fce18ff7740 1 -- 172.21.15.43:0/3043506388 wait complete. 2024-03-28T22:47:46.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.430+0000 7fce18ff7740 20 client.29150 trim_cache size 0 max 16384 2024-03-28T22:47:46.437 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:47:46.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.431+0000 7fce14e00640 1 -- 172.21.15.43:0/315650880 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcdf40a1ef0 msgr2=0x7fcdf40a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.431+0000 7fce14e00640 1 --2- 172.21.15.43:0/315650880 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcdf40a1ef0 0x7fcdf40a43b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x55ac14d4e020 tx=0x7fce000f46e0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.431+0000 7fce14e00640 1 -- 172.21.15.43:0/315650880 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce10073b00 msgr2=0x7fce10199fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.431+0000 7fce14e00640 1 --2- 172.21.15.43:0/315650880 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce10073b00 0x7fce10199fa0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fce00000f80 tx=0x7fce00003ba0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.431+0000 7fce14e00640 1 -- 172.21.15.43:0/315650880 shutdown_connections 2024-03-28T22:47:46.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.431+0000 7fce14e00640 1 --2- 172.21.15.43:0/315650880 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcdf40a1ef0 0x7fcdf40a43b0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.431+0000 7fce14e00640 1 --2- 172.21.15.43:0/315650880 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fce1010de80 0x7fce1019e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.431+0000 7fce14e00640 1 --2- 172.21.15.43:0/315650880 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fce10074420 0x7fce1019a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.431+0000 7fce14e00640 1 --2- 172.21.15.43:0/315650880 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fce10073b00 0x7fce10199fa0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.431+0000 7fce14e00640 1 -- 172.21.15.43:0/315650880 >> 172.21.15.43:0/315650880 conn(0x7fce100fd0f0 msgr2=0x7fce10110080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:46.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.431+0000 7fce14e00640 1 -- 172.21.15.43:0/315650880 shutdown_connections 2024-03-28T22:47:46.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.431+0000 7fce14e00640 1 -- 172.21.15.43:0/315650880 wait complete. 2024-03-28T22:47:46.447 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T22:47:46.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.675+0000 7fe2ef600640 1 -- 172.21.15.43:0/3819987685 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe2e810df10 msgr2=0x7fe2e8110360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.675+0000 7fe2ef600640 1 --2- 172.21.15.43:0/3819987685 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe2e810df10 0x7fe2e8110360 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0009900 tx=0x7fe2e002bcd0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.676+0000 7fe2ef600640 1 -- 172.21.15.43:0/3819987685 shutdown_connections 2024-03-28T22:47:46.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.676+0000 7fe2ef600640 1 --2- 172.21.15.43:0/3819987685 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe2e80744a0 0x7fe2e810d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.676+0000 7fe2ef600640 1 --2- 172.21.15.43:0/3819987685 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe2e810df10 0x7fe2e8110360 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.676+0000 7fe2ef600640 1 --2- 172.21.15.43:0/3819987685 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2e8073b80 0x7fe2e8073f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.676+0000 7fe2ef600640 1 -- 172.21.15.43:0/3819987685 >> 172.21.15.43:0/3819987685 conn(0x7fe2e80fd1e0 msgr2=0x7fe2e80ff600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:46.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.676+0000 7fe2ef600640 1 -- 172.21.15.43:0/3819987685 shutdown_connections 2024-03-28T22:47:46.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.676+0000 7fe2ef600640 1 -- 172.21.15.43:0/3819987685 wait complete. 2024-03-28T22:47:46.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.677+0000 7fe2ef600640 1 Processor -- start 2024-03-28T22:47:46.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.677+0000 7fe2ef600640 1 -- start start 2024-03-28T22:47:46.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.678+0000 7fe2ef600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe2e8073b80 0x7fe2e819a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.678+0000 7fe2ef600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2e80744a0 0x7fe2e819a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.678+0000 7fe2ef600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe2e810df10 0x7fe2e819ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.678+0000 7fe2ef600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e8113d00 con 0x7fe2e80744a0 2024-03-28T22:47:46.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.678+0000 7fe2ef600640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e8113b80 con 0x7fe2e810df10 2024-03-28T22:47:46.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.678+0000 7fe2ef600640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e8113e80 con 0x7fe2e8073b80 2024-03-28T22:47:46.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.678+0000 7fe2ecc00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe2e8073b80 0x7fe2e819a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.678+0000 7fe2ecc00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe2e8073b80 0x7fe2e819a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:34390/0 (socket says 172.21.15.43:34390) 2024-03-28T22:47:46.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.678+0000 7fe2ecc00640 1 -- 172.21.15.43:0/2611870207 learned_addr learned my addr 172.21.15.43:0/2611870207 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:46.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.678+0000 7fe2ed600640 1 --2- 172.21.15.43:0/2611870207 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe2e810df10 0x7fe2e819ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.678+0000 7fe2e7e00640 1 --2- 172.21.15.43:0/2611870207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2e80744a0 0x7fe2e819a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.678+0000 7fe2ecc00640 1 -- 172.21.15.43:0/2611870207 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe2e810df10 msgr2=0x7fe2e819ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.678+0000 7fe2ecc00640 1 --2- 172.21.15.43:0/2611870207 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe2e810df10 0x7fe2e819ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.678+0000 7fe2ecc00640 1 -- 172.21.15.43:0/2611870207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2e80744a0 msgr2=0x7fe2e819a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.678+0000 7fe2ecc00640 1 --2- 172.21.15.43:0/2611870207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2e80744a0 0x7fe2e819a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.678+0000 7fe2ecc00640 1 -- 172.21.15.43:0/2611870207 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2e819f1c0 con 0x7fe2e8073b80 2024-03-28T22:47:46.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.679+0000 7fe2ecc00640 1 --2- 172.21.15.43:0/2611870207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe2e8073b80 0x7fe2e819a220 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fe2d800ebc0 tx=0x7fe2d800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:46.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.679+0000 7fe2e5600640 1 -- 172.21.15.43:0/2611870207 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d8010630 con 0x7fe2e8073b80 2024-03-28T22:47:46.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.679+0000 7fe2e5600640 1 -- 172.21.15.43:0/2611870207 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2d8010cb0 con 0x7fe2e8073b80 2024-03-28T22:47:46.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.679+0000 7fe2ef600640 1 -- 172.21.15.43:0/2611870207 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2e819f4b0 con 0x7fe2e8073b80 2024-03-28T22:47:46.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.679+0000 7fe2e5600640 1 -- 172.21.15.43:0/2611870207 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d8018bd0 con 0x7fe2e8073b80 2024-03-28T22:47:46.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.679+0000 7fe2ef600640 1 -- 172.21.15.43:0/2611870207 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2e81a6d40 con 0x7fe2e8073b80 2024-03-28T22:47:46.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.682+0000 7fe2f239f740 1 Processor -- start 2024-03-28T22:47:46.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.682+0000 7fe2e5600640 1 -- 172.21.15.43:0/2611870207 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fe2d80107d0 con 0x7fe2e8073b80 2024-03-28T22:47:46.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.683+0000 7fe2f239f740 1 -- start start 2024-03-28T22:47:46.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.683+0000 7fe2f239f740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dec5bfa380 0x55dec5bfa760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.683+0000 7fe2f239f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dec5bfaca0 0x55dec5c03340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.683+0000 7fe2f239f740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dec5c03880 0x55dec5c05c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.683+0000 7fe2f239f740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55dec5b5e050 con 0x55dec5bfa380 2024-03-28T22:47:46.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.683+0000 7fe2f239f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55dec5b06ce0 con 0x55dec5bfaca0 2024-03-28T22:47:46.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.683+0000 7fe2f239f740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55dec5b5a0f0 con 0x55dec5c03880 2024-03-28T22:47:46.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.683+0000 7fe2e7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dec5c03880 0x55dec5c05c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.683+0000 7fe2ecc00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dec5bfaca0 0x55dec5c03340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.683+0000 7fe2e7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dec5c03880 0x55dec5c05c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:34392/0 (socket says 172.21.15.43:34392) 2024-03-28T22:47:46.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.683+0000 7fe2e7e00640 1 -- 172.21.15.43:0/4007171334 learned_addr learned my addr 172.21.15.43:0/4007171334 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:46.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.683+0000 7fe2ed600640 1 --2- 172.21.15.43:0/4007171334 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dec5bfa380 0x55dec5bfa760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.684+0000 7fe2e7e00640 1 -- 172.21.15.43:0/4007171334 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dec5bfaca0 msgr2=0x55dec5c03340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:46.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.684+0000 7fe2e7e00640 1 --2- 172.21.15.43:0/4007171334 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dec5bfaca0 0x55dec5c03340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.684+0000 7fe2e7e00640 1 -- 172.21.15.43:0/4007171334 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dec5bfa380 msgr2=0x55dec5bfa760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:46.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.684+0000 7fe2e7e00640 1 --2- 172.21.15.43:0/4007171334 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dec5bfa380 0x55dec5bfa760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.684+0000 7fe2e7e00640 1 -- 172.21.15.43:0/4007171334 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dec58723f0 con 0x55dec5c03880 2024-03-28T22:47:46.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.684+0000 7fe2ecc00640 1 --2- 172.21.15.43:0/4007171334 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dec5bfaca0 0x55dec5c03340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:46.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.684+0000 7fe2e7e00640 1 --2- 172.21.15.43:0/4007171334 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dec5c03880 0x55dec5c05c70 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fe2d4009c10 tx=0x7fe2d4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10684b781cc48b2b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:46.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.684+0000 7fe2e5600640 1 --2- 172.21.15.43:0/2611870207 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe2cc0a1f70 0x7fe2cc0a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.684+0000 7fe2ddc00640 1 -- 172.21.15.43:0/4007171334 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d4004230 con 0x55dec5c03880 2024-03-28T22:47:46.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.684+0000 7fe2ddc00640 1 -- 172.21.15.43:0/4007171334 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2d40043d0 con 0x55dec5c03880 2024-03-28T22:47:46.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.684+0000 7fe2ddc00640 1 -- 172.21.15.43:0/4007171334 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d4004e10 con 0x55dec5c03880 2024-03-28T22:47:46.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.684+0000 7fe2e5600640 1 -- 172.21.15.43:0/2611870207 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9539+0+0 (secure 0 0 0) 0x7fe2d80ceb70 con 0x7fe2e8073b80 2024-03-28T22:47:46.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.684+0000 7fe2ed600640 1 --2- 172.21.15.43:0/2611870207 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe2cc0a1f70 0x7fe2cc0a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.685+0000 7fe2f239f740 1 -- 172.21.15.43:0/4007171334 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dec5c03880 msgr2=0x55dec5c05c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.685+0000 7fe2f239f740 1 --2- 172.21.15.43:0/4007171334 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dec5c03880 0x55dec5c05c70 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fe2d4009c10 tx=0x7fe2d4011810 comp rx=0 tx=0).stop 2024-03-28T22:47:46.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.685+0000 7fe2f239f740 1 -- 172.21.15.43:0/4007171334 shutdown_connections 2024-03-28T22:47:46.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.685+0000 7fe2f239f740 1 --2- 172.21.15.43:0/4007171334 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dec5bfaca0 0x55dec5c03340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.685+0000 7fe2f239f740 1 --2- 172.21.15.43:0/4007171334 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dec5c03880 0x55dec5c05c70 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.685+0000 7fe2f239f740 1 --2- 172.21.15.43:0/4007171334 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dec5bfa380 0x55dec5bfa760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.685+0000 7fe2f239f740 1 -- 172.21.15.43:0/4007171334 >> 172.21.15.43:0/4007171334 conn(0x55dec5bf4ab0 msgr2=0x55dec5bf4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:46.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.685+0000 7fe2f239f740 1 -- 172.21.15.43:0/4007171334 shutdown_connections 2024-03-28T22:47:46.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.685+0000 7fe2f239f740 1 -- 172.21.15.43:0/4007171334 wait complete. 2024-03-28T22:47:46.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.685+0000 7fe2ed600640 1 --2- 172.21.15.43:0/2611870207 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe2cc0a1f70 0x7fe2cc0a4430 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0005ab0 tx=0x7fe2e0005a40 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:46.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.686+0000 7fe2f239f740 1 -- start start 2024-03-28T22:47:46.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.686+0000 7fe2f239f740 1 Processor -- start 2024-03-28T22:47:46.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.686+0000 7fe2f239f740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dec5bfa380 0x55dec5c00810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.686+0000 7fe2f239f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dec5bfaca0 0x55dec5c00d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.686+0000 7fe2f239f740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dec5c03880 0x55dec5c01290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.686+0000 7fe2f239f740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x55dec5c06670 con 0x55dec5c03880 2024-03-28T22:47:46.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.686+0000 7fe2f239f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55dec5afc2f0 con 0x55dec5bfaca0 2024-03-28T22:47:46.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.686+0000 7fe2f239f740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x55dec5c067f0 con 0x55dec5bfa380 2024-03-28T22:47:46.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.686+0000 7fe2f239f740 10 client.0 _ll_register_callbacks cb 0x55dec5871be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:47:46.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.687+0000 7fe2e7e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dec5bfaca0 0x55dec5c00d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.687+0000 7fe2ecc00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dec5bfa380 0x55dec5c00810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.687+0000 7fe2e7e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dec5bfaca0 0x55dec5c00d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59062/0 (socket says 172.21.15.43:59062) 2024-03-28T22:47:46.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.687+0000 7fe2e7e00640 1 -- 172.21.15.43:0/1301616823 learned_addr learned my addr 172.21.15.43:0/1301616823 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:46.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.687+0000 7fe2ed600640 1 --2- 172.21.15.43:0/1301616823 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dec5c03880 0x55dec5c01290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.687+0000 7fe2e7e00640 1 -- 172.21.15.43:0/1301616823 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dec5bfa380 msgr2=0x55dec5c00810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:46.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.687+0000 7fe2e7e00640 1 --2- 172.21.15.43:0/1301616823 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dec5bfa380 0x55dec5c00810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.687+0000 7fe2e7e00640 1 -- 172.21.15.43:0/1301616823 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dec5c03880 msgr2=0x55dec5c01290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:46.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.687+0000 7fe2e7e00640 1 --2- 172.21.15.43:0/1301616823 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dec5c03880 0x55dec5c01290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.687+0000 7fe2e7e00640 1 -- 172.21.15.43:0/1301616823 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dec593c7d0 con 0x55dec5bfaca0 2024-03-28T22:47:46.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.687+0000 7fe2ecc00640 1 --2- 172.21.15.43:0/1301616823 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dec5bfa380 0x55dec5c00810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:46.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.687+0000 7fe2ed600640 1 --2- 172.21.15.43:0/1301616823 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dec5c03880 0x55dec5c01290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:47:46.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.687+0000 7fe2e7e00640 1 --2- 172.21.15.43:0/1301616823 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dec5bfaca0 0x55dec5c00d50 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fe2d40051e0 tx=0x7fe2d4005210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc9491910972158a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:46.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.687+0000 7fe2d2000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:47:46.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.688+0000 7fe2d2000640 1 -- 172.21.15.43:0/1301616823 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d4024620 con 0x55dec5bfaca0 2024-03-28T22:47:46.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.688+0000 7fe2d2000640 1 -- 172.21.15.43:0/1301616823 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9539+0+0 (secure 0 0 0) 0x7fe2d40247c0 con 0x55dec5bfaca0 2024-03-28T22:47:46.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.688+0000 7fe2f239f740 1 -- 172.21.15.43:0/1301616823 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2d40043d0 con 0x55dec5bfaca0 2024-03-28T22:47:46.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.688+0000 7fe2d2000640 1 -- 172.21.15.43:0/1301616823 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe2d4032050 con 0x55dec5bfaca0 2024-03-28T22:47:46.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.688+0000 7fe2f239f740 10 client.29162 fetch_fsmap learned FSMap version 32 2024-03-28T22:47:46.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.688+0000 7fe2f239f740 1 -- 172.21.15.43:0/1301616823 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55dec5c12870 con 0x55dec5bfaca0 2024-03-28T22:47:46.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.688+0000 7fe2d2000640 1 -- 172.21.15.43:0/1301616823 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2d40043d0 con 0x55dec5bfaca0 2024-03-28T22:47:46.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.688+0000 7fe2d2000640 1 -- 172.21.15.43:0/1301616823 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe2d4018070 con 0x55dec5bfaca0 2024-03-28T22:47:46.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.688+0000 7fe2f239f740 10 client.29162 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:47:46.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.688+0000 7fe2f239f740 10 client.29162 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:47:46.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.689+0000 7fe2f239f740 20 client.29162 populate_metadata read hostname 'smithi043' 2024-03-28T22:47:46.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.689+0000 7fe2f239f740 1 --2- 172.21.15.43:0/1301616823 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55dec5c13cd0 0x55dec5c160c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.689+0000 7fe2f239f740 4 client.29162 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:47:46.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.689+0000 7fe2f239f740 1 -- 172.21.15.43:0/1301616823 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dec5b07680 con 0x55dec5c13cd0 2024-03-28T22:47:46.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.689+0000 7fe2ecc00640 1 --2- 172.21.15.43:0/1301616823 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55dec5c13cd0 0x55dec5c160c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.689+0000 7fe2ecc00640 1 --2- 172.21.15.43:0/1301616823 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55dec5c13cd0 0x55dec5c160c0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=267e1ee3aa44030b server_cookie=95259bf7dbb16238 in_seq=0 out_seq=0 2024-03-28T22:47:46.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.689+0000 7fe2d2000640 10 client.29162 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:47:46.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.690+0000 7fe2d2000640 1 -- 172.21.15.43:0/1301616823 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dec5b07680 con 0x55dec5c13cd0 2024-03-28T22:47:46.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.690+0000 7fe2d2000640 10 client.29162 handle_command_reply: tid=0 2024-03-28T22:47:46.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.690+0000 7fe2f239f740 1 client.29162 shutdown 2024-03-28T22:47:46.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.690+0000 7fe2f239f740 1 -- 172.21.15.43:0/1301616823 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dec5bfaca0 msgr2=0x55dec5c00d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.690+0000 7fe2f239f740 1 --2- 172.21.15.43:0/1301616823 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dec5bfaca0 0x55dec5c00d50 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fe2d40051e0 tx=0x7fe2d4005210 comp rx=0 tx=0).stop 2024-03-28T22:47:46.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.690+0000 7fe2f239f740 1 -- 172.21.15.43:0/1301616823 shutdown_connections 2024-03-28T22:47:46.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.690+0000 7fe2f239f740 1 --2- 172.21.15.43:0/1301616823 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55dec5c13cd0 0x55dec5c160c0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.690+0000 7fe2f239f740 1 --2- 172.21.15.43:0/1301616823 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dec5bfaca0 0x55dec5c00d50 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.690+0000 7fe2f239f740 1 --2- 172.21.15.43:0/1301616823 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dec5c03880 0x55dec5c01290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.690+0000 7fe2f239f740 1 --2- 172.21.15.43:0/1301616823 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dec5bfa380 0x55dec5c00810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.690+0000 7fe2f239f740 1 -- 172.21.15.43:0/1301616823 >> 172.21.15.43:0/1301616823 conn(0x55dec5bf4ab0 msgr2=0x55dec5bf6800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:46.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.690+0000 7fe2d2000640 0 client.29162 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:47:46.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.691+0000 7fe2f239f740 1 -- 172.21.15.43:0/1301616823 shutdown_connections 2024-03-28T22:47:46.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.691+0000 7fe2f239f740 1 -- 172.21.15.43:0/1301616823 wait complete. 2024-03-28T22:47:46.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.691+0000 7fe2f239f740 20 client.29162 trim_cache size 0 max 16384 2024-03-28T22:47:46.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.705+0000 7fe2f239f740 1 Processor -- start 2024-03-28T22:47:46.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.705+0000 7fe2f239f740 1 -- start start 2024-03-28T22:47:46.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.706+0000 7fe2f239f740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dec5bfa380 0x55dec5c02f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.706+0000 7fe2f239f740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dec5bfaca0 0x55dec5c0f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.706+0000 7fe2f239f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dec5c03880 0x55dec5c1b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.706+0000 7fe2f239f740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e81b0f80 con 0x55dec5bfa380 2024-03-28T22:47:46.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.706+0000 7fe2f239f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e81b2ac0 con 0x55dec5c03880 2024-03-28T22:47:46.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.706+0000 7fe2f239f740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e81b1650 con 0x55dec5bfaca0 2024-03-28T22:47:46.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.706+0000 7fe2e7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dec5bfaca0 0x55dec5c0f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.706+0000 7fe2ed600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dec5c03880 0x55dec5c1b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.706+0000 7fe2e7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dec5bfaca0 0x55dec5c0f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:34406/0 (socket says 172.21.15.43:34406) 2024-03-28T22:47:46.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.706+0000 7fe2e7e00640 1 -- 172.21.15.43:0/775965982 learned_addr learned my addr 172.21.15.43:0/775965982 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:46.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.706+0000 7fe2ecc00640 1 --2- 172.21.15.43:0/775965982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dec5bfa380 0x55dec5c02f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.706+0000 7fe2e7e00640 1 -- 172.21.15.43:0/775965982 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dec5c03880 msgr2=0x55dec5c1b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:46.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.706+0000 7fe2e7e00640 1 --2- 172.21.15.43:0/775965982 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dec5c03880 0x55dec5c1b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.706+0000 7fe2e7e00640 1 -- 172.21.15.43:0/775965982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dec5bfa380 msgr2=0x55dec5c02f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:46.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.706+0000 7fe2e7e00640 1 --2- 172.21.15.43:0/775965982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dec5bfa380 0x55dec5c02f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.706+0000 7fe2e7e00640 1 -- 172.21.15.43:0/775965982 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dec5bf48e0 con 0x55dec5bfaca0 2024-03-28T22:47:46.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.706+0000 7fe2ed600640 1 --2- 172.21.15.43:0/775965982 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dec5c03880 0x55dec5c1b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:46.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.707+0000 7fe2e7e00640 1 --2- 172.21.15.43:0/775965982 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dec5bfaca0 0x55dec5c0f410 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fe2d4000c00 tx=0x7fe2d4004ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76d33787ee7a6771 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:46.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.707+0000 7fe2dd200640 1 -- 172.21.15.43:0/775965982 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d4005280 con 0x55dec5bfaca0 2024-03-28T22:47:46.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.707+0000 7fe2dd200640 1 -- 172.21.15.43:0/775965982 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2d4029070 con 0x55dec5bfaca0 2024-03-28T22:47:46.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.707+0000 7fe2dd200640 1 -- 172.21.15.43:0/775965982 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d4004520 con 0x55dec5bfaca0 2024-03-28T22:47:46.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.708+0000 7fe2f239f740 1 -- 172.21.15.43:0/775965982 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dec5bfaca0 msgr2=0x55dec5c0f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.708+0000 7fe2f239f740 1 --2- 172.21.15.43:0/775965982 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dec5bfaca0 0x55dec5c0f410 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fe2d4000c00 tx=0x7fe2d4004ec0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.708+0000 7fe2f239f740 1 -- 172.21.15.43:0/775965982 shutdown_connections 2024-03-28T22:47:46.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.708+0000 7fe2f239f740 1 --2- 172.21.15.43:0/775965982 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dec5c03880 0x55dec5c1b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.708+0000 7fe2f239f740 1 --2- 172.21.15.43:0/775965982 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dec5bfaca0 0x55dec5c0f410 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.708+0000 7fe2f239f740 1 --2- 172.21.15.43:0/775965982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dec5bfa380 0x55dec5c02f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.708+0000 7fe2f239f740 1 -- 172.21.15.43:0/775965982 >> 172.21.15.43:0/775965982 conn(0x55dec5bf4ab0 msgr2=0x55dec5bff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:46.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.708+0000 7fe2f239f740 1 -- 172.21.15.43:0/775965982 shutdown_connections 2024-03-28T22:47:46.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.708+0000 7fe2f239f740 1 -- 172.21.15.43:0/775965982 wait complete. 2024-03-28T22:47:46.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.708+0000 7fe2f239f740 1 -- start start 2024-03-28T22:47:46.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.708+0000 7fe2f239f740 1 Processor -- start 2024-03-28T22:47:46.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.709+0000 7fe2f239f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dec5bfa380 0x55dec5c06a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.709+0000 7fe2f239f740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dec5bfaca0 0x55dec5c06f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.709+0000 7fe2f239f740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dec5c03880 0x55dec5c07480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.709+0000 7fe2f239f740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e81b23f0 con 0x55dec5bfaca0 2024-03-28T22:47:46.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.709+0000 7fe2f239f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e81b1d20 con 0x55dec5bfa380 2024-03-28T22:47:46.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.709+0000 7fe2f239f740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e81aef80 con 0x55dec5c03880 2024-03-28T22:47:46.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.709+0000 7fe2f239f740 10 client.0 _ll_register_callbacks cb 0x7fe2d80d24f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:47:46.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.709+0000 7fe2ecc00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dec5bfa380 0x55dec5c06a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.709+0000 7fe2ed600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dec5c03880 0x55dec5c07480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.709+0000 7fe2ecc00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dec5bfa380 0x55dec5c06a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59068/0 (socket says 172.21.15.43:59068) 2024-03-28T22:47:46.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.709+0000 7fe2ecc00640 1 -- 172.21.15.43:0/2760465960 learned_addr learned my addr 172.21.15.43:0/2760465960 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:46.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.709+0000 7fe2ed600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dec5c03880 0x55dec5c07480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:34412/0 (socket says 172.21.15.43:34412) 2024-03-28T22:47:46.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.709+0000 7fe2e7e00640 1 --2- 172.21.15.43:0/2760465960 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dec5bfaca0 0x55dec5c06f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.710+0000 7fe2ecc00640 1 -- 172.21.15.43:0/2760465960 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dec5c03880 msgr2=0x55dec5c07480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:46.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.710+0000 7fe2ecc00640 1 --2- 172.21.15.43:0/2760465960 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dec5c03880 0x55dec5c07480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.710+0000 7fe2ecc00640 1 -- 172.21.15.43:0/2760465960 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dec5bfaca0 msgr2=0x55dec5c06f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:46.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.710+0000 7fe2ecc00640 1 --2- 172.21.15.43:0/2760465960 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dec5bfaca0 0x55dec5c06f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.710+0000 7fe2ecc00640 1 -- 172.21.15.43:0/2760465960 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dec5c781e0 con 0x55dec5bfa380 2024-03-28T22:47:46.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.710+0000 7fe2ecc00640 1 --2- 172.21.15.43:0/2760465960 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dec5bfa380 0x55dec5c06a00 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fe2d809aeb0 tx=0x7fe2d800cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aab967af39a941e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:46.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.710+0000 7fe2d2000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:47:46.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.710+0000 7fe2d2000640 1 -- 172.21.15.43:0/2760465960 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d8010e60 con 0x55dec5bfa380 2024-03-28T22:47:46.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.710+0000 7fe2d2000640 1 -- 172.21.15.43:0/2760465960 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9539+0+0 (secure 0 0 0) 0x7fe2d80994b0 con 0x55dec5bfa380 2024-03-28T22:47:46.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.710+0000 7fe2f239f740 1 -- 172.21.15.43:0/2760465960 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2d4029070 con 0x55dec5bfa380 2024-03-28T22:47:46.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.710+0000 7fe2d2000640 1 -- 172.21.15.43:0/2760465960 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe2d809b1b0 con 0x55dec5bfa380 2024-03-28T22:47:46.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.710+0000 7fe2f239f740 10 client.29168 fetch_fsmap learned FSMap version 32 2024-03-28T22:47:46.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.710+0000 7fe2f239f740 1 -- 172.21.15.43:0/2760465960 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55dec5c794c0 con 0x55dec5bfa380 2024-03-28T22:47:46.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.711+0000 7fe2d2000640 1 -- 172.21.15.43:0/2760465960 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2d4029070 con 0x55dec5bfa380 2024-03-28T22:47:46.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.711+0000 7fe2d2000640 1 -- 172.21.15.43:0/2760465960 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe2d809a600 con 0x55dec5bfa380 2024-03-28T22:47:46.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.711+0000 7fe2f239f740 10 client.29168 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:47:46.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.711+0000 7fe2f239f740 10 client.29168 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T22:47:46.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.711+0000 7fe2f239f740 20 client.29168 populate_metadata read hostname 'smithi043' 2024-03-28T22:47:46.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.711+0000 7fe2f239f740 1 --2- 172.21.15.43:0/2760465960 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55dec5c79b90 0x55dec5c7bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.711+0000 7fe2f239f740 4 client.29168 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:47:46.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.711+0000 7fe2f239f740 1 -- 172.21.15.43:0/2760465960 --> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dec5b6b590 con 0x55dec5c79b90 2024-03-28T22:47:46.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.711+0000 7fe2e7e00640 1 --2- 172.21.15.43:0/2760465960 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55dec5c79b90 0x55dec5c7bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.712+0000 7fe2e7e00640 1 --2- 172.21.15.43:0/2760465960 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55dec5c79b90 0x55dec5c7bf80 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=19140263fd023848 server_cookie=b328ec53bac6113f in_seq=0 out_seq=0 2024-03-28T22:47:46.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.712+0000 7fe2d2000640 10 client.29168 ms_handle_connect on v2:172.21.15.172:6838/3592465067 2024-03-28T22:47:46.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.712+0000 7fe2d2000640 1 -- 172.21.15.43:0/2760465960 <== mds.1 v2:172.21.15.172:6838/3592465067 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dec5b6b590 con 0x55dec5c79b90 2024-03-28T22:47:46.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.712+0000 7fe2d2000640 10 client.29168 handle_command_reply: tid=0 2024-03-28T22:47:46.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.712+0000 7fe2f239f740 1 client.29168 shutdown 2024-03-28T22:47:46.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.712+0000 7fe2f239f740 1 -- 172.21.15.43:0/2760465960 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dec5bfa380 msgr2=0x55dec5c06a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.712+0000 7fe2f239f740 1 --2- 172.21.15.43:0/2760465960 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dec5bfa380 0x55dec5c06a00 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fe2d809aeb0 tx=0x7fe2d800cf90 comp rx=0 tx=0).stop 2024-03-28T22:47:46.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.712+0000 7fe2f239f740 1 -- 172.21.15.43:0/2760465960 shutdown_connections 2024-03-28T22:47:46.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.712+0000 7fe2f239f740 1 --2- 172.21.15.43:0/2760465960 >> [v2:172.21.15.172:6838/3592465067,v1:172.21.15.172:6839/3592465067] conn(0x55dec5c79b90 0x55dec5c7bf80 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.712+0000 7fe2f239f740 1 --2- 172.21.15.43:0/2760465960 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x55dec5c03880 0x55dec5c07480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.712+0000 7fe2f239f740 1 --2- 172.21.15.43:0/2760465960 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x55dec5bfaca0 0x55dec5c06f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.712+0000 7fe2f239f740 1 --2- 172.21.15.43:0/2760465960 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dec5bfa380 0x55dec5c06a00 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.712+0000 7fe2f239f740 1 -- 172.21.15.43:0/2760465960 >> 172.21.15.43:0/2760465960 conn(0x55dec5bf4ab0 msgr2=0x55dec5bf32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:46.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.712+0000 7fe2d2000640 0 client.29168 ms_handle_reset on v2:172.21.15.172:6838/3592465067 2024-03-28T22:47:46.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.712+0000 7fe2f239f740 1 -- 172.21.15.43:0/2760465960 shutdown_connections 2024-03-28T22:47:46.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.712+0000 7fe2f239f740 1 -- 172.21.15.43:0/2760465960 wait complete. 2024-03-28T22:47:46.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.712+0000 7fe2f239f740 20 client.29168 trim_cache size 0 max 16384 2024-03-28T22:47:46.719 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:47:46.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.713+0000 7fe2ef600640 1 -- 172.21.15.43:0/2611870207 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe2cc0a1f70 msgr2=0x7fe2cc0a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.713+0000 7fe2ef600640 1 --2- 172.21.15.43:0/2611870207 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe2cc0a1f70 0x7fe2cc0a4430 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0005ab0 tx=0x7fe2e0005a40 comp rx=0 tx=0).stop 2024-03-28T22:47:46.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.713+0000 7fe2ef600640 1 -- 172.21.15.43:0/2611870207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe2e8073b80 msgr2=0x7fe2e819a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.713+0000 7fe2ef600640 1 --2- 172.21.15.43:0/2611870207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe2e8073b80 0x7fe2e819a220 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fe2d800ebc0 tx=0x7fe2d800c6a0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.713+0000 7fe2ef600640 1 -- 172.21.15.43:0/2611870207 shutdown_connections 2024-03-28T22:47:46.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.713+0000 7fe2ef600640 1 --2- 172.21.15.43:0/2611870207 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fe2cc0a1f70 0x7fe2cc0a4430 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.713+0000 7fe2ef600640 1 --2- 172.21.15.43:0/2611870207 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe2e810df10 0x7fe2e819ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.713+0000 7fe2ef600640 1 --2- 172.21.15.43:0/2611870207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe2e80744a0 0x7fe2e819a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.713+0000 7fe2ef600640 1 --2- 172.21.15.43:0/2611870207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe2e8073b80 0x7fe2e819a220 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.713+0000 7fe2ef600640 1 -- 172.21.15.43:0/2611870207 >> 172.21.15.43:0/2611870207 conn(0x7fe2e80fd1e0 msgr2=0x7fe2e80fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:46.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.713+0000 7fe2ef600640 1 -- 172.21.15.43:0/2611870207 shutdown_connections 2024-03-28T22:47:46.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.713+0000 7fe2ef600640 1 -- 172.21.15.43:0/2611870207 wait complete. 2024-03-28T22:47:46.729 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T22:47:46.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.957+0000 7f8e8f600640 1 -- 172.21.15.43:0/2899666227 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e8806a6c0 msgr2=0x7f8e8810e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.957+0000 7f8e8f600640 1 --2- 172.21.15.43:0/2899666227 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e8806a6c0 0x7f8e8810e590 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8e80009ca0 tx=0x7f8e8002c180 comp rx=0 tx=0).stop 2024-03-28T22:47:46.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.957+0000 7f8e8f600640 1 -- 172.21.15.43:0/2899666227 shutdown_connections 2024-03-28T22:47:46.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.957+0000 7f8e8f600640 1 --2- 172.21.15.43:0/2899666227 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e8806a6c0 0x7f8e8810e590 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.957+0000 7f8e8f600640 1 --2- 172.21.15.43:0/2899666227 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e88069d20 0x7f8e8806a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.957+0000 7f8e8f600640 1 --2- 172.21.15.43:0/2899666227 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8e88107740 0x7f8e88107b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.957+0000 7f8e8f600640 1 -- 172.21.15.43:0/2899666227 >> 172.21.15.43:0/2899666227 conn(0x7f8e880776e0 msgr2=0x7f8e88077af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:46.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.957+0000 7f8e8f600640 1 -- 172.21.15.43:0/2899666227 shutdown_connections 2024-03-28T22:47:46.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.957+0000 7f8e8f600640 1 -- 172.21.15.43:0/2899666227 wait complete. 2024-03-28T22:47:46.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.958+0000 7f8e8f600640 1 Processor -- start 2024-03-28T22:47:46.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.959+0000 7f8e8f600640 1 -- start start 2024-03-28T22:47:46.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.959+0000 7f8e8f600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8e88069d20 0x7f8e8819a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.959+0000 7f8e8f600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e8806a6c0 0x7f8e8819a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.959+0000 7f8e8f600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e88107740 0x7f8e8819ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.959+0000 7f8e8f600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8e88111b00 con 0x7f8e8806a6c0 2024-03-28T22:47:46.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.959+0000 7f8e8f600640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8e88111980 con 0x7f8e88069d20 2024-03-28T22:47:46.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.959+0000 7f8e8f600640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f8e88111c80 con 0x7f8e88107740 2024-03-28T22:47:46.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.959+0000 7f8e8d600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e88107740 0x7f8e8819ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.959+0000 7f8e8cc00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8e88069d20 0x7f8e8819a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.959+0000 7f8e87e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e8806a6c0 0x7f8e8819a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.959+0000 7f8e8d600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e88107740 0x7f8e8819ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:34438/0 (socket says 172.21.15.43:34438) 2024-03-28T22:47:46.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.959+0000 7f8e8d600640 1 -- 172.21.15.43:0/283289282 learned_addr learned my addr 172.21.15.43:0/283289282 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:46.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.959+0000 7f8e8cc00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8e88069d20 0x7f8e8819a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59096/0 (socket says 172.21.15.43:59096) 2024-03-28T22:47:46.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.960+0000 7f8e8cc00640 1 -- 172.21.15.43:0/283289282 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e88107740 msgr2=0x7f8e8819ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.960+0000 7f8e8cc00640 1 --2- 172.21.15.43:0/283289282 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e88107740 0x7f8e8819ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.960+0000 7f8e8cc00640 1 -- 172.21.15.43:0/283289282 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e8806a6c0 msgr2=0x7f8e8819a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.960+0000 7f8e8cc00640 1 --2- 172.21.15.43:0/283289282 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e8806a6c0 0x7f8e8819a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.960+0000 7f8e8cc00640 1 -- 172.21.15.43:0/283289282 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e8819f160 con 0x7f8e88069d20 2024-03-28T22:47:46.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.960+0000 7f8e8d600640 1 --2- 172.21.15.43:0/283289282 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e88107740 0x7f8e8819ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:46.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.960+0000 7f8e87e00640 1 --2- 172.21.15.43:0/283289282 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e8806a6c0 0x7f8e8819a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:46.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.960+0000 7f8e8cc00640 1 --2- 172.21.15.43:0/283289282 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8e88069d20 0x7f8e8819a170 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c00e9f0 tx=0x7f8e7c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:46.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.960+0000 7f8e85600640 1 -- 172.21.15.43:0/283289282 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c00ccc0 con 0x7f8e88069d20 2024-03-28T22:47:46.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.960+0000 7f8e85600640 1 -- 172.21.15.43:0/283289282 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e7c010910 con 0x7f8e88069d20 2024-03-28T22:47:46.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.960+0000 7f8e85600640 1 -- 172.21.15.43:0/283289282 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c018960 con 0x7f8e88069d20 2024-03-28T22:47:46.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.960+0000 7f8e8f600640 1 -- 172.21.15.43:0/283289282 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e8819f450 con 0x7f8e88069d20 2024-03-28T22:47:46.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.960+0000 7f8e8f600640 1 -- 172.21.15.43:0/283289282 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e881a6ce0 con 0x7f8e88069d20 2024-03-28T22:47:46.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.963+0000 7f8e85600640 1 -- 172.21.15.43:0/283289282 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f8e7c010ac0 con 0x7f8e88069d20 2024-03-28T22:47:46.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.964+0000 7f8e9241c740 1 Processor -- start 2024-03-28T22:47:46.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.964+0000 7f8e9241c740 1 -- start start 2024-03-28T22:47:46.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.965+0000 7f8e9241c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5647170fd380 0x5647170fd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.965+0000 7f8e9241c740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5647170fdca0 0x564717106340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.965+0000 7f8e9241c740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564717106880 0x564717108c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.965+0000 7f8e9241c740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x564717061050 con 0x564717106880 2024-03-28T22:47:46.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.965+0000 7f8e9241c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x564717009ce0 con 0x5647170fd380 2024-03-28T22:47:46.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.965+0000 7f8e9241c740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x56471705d0f0 con 0x5647170fdca0 2024-03-28T22:47:46.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.965+0000 7f8e8d600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5647170fd380 0x5647170fd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.965+0000 7f8e8cc00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5647170fdca0 0x564717106340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.965+0000 7f8e8d600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5647170fd380 0x5647170fd760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59110/0 (socket says 172.21.15.43:59110) 2024-03-28T22:47:46.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.965+0000 7f8e8d600640 1 -- 172.21.15.43:0/2281013060 learned_addr learned my addr 172.21.15.43:0/2281013060 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:46.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.965+0000 7f8e87e00640 1 --2- 172.21.15.43:0/2281013060 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564717106880 0x564717108c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.965+0000 7f8e8d600640 1 -- 172.21.15.43:0/2281013060 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5647170fdca0 msgr2=0x564717106340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:46.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.965+0000 7f8e8d600640 1 --2- 172.21.15.43:0/2281013060 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5647170fdca0 0x564717106340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.965+0000 7f8e8d600640 1 -- 172.21.15.43:0/2281013060 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564717106880 msgr2=0x564717108c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:46.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.965+0000 7f8e8d600640 1 --2- 172.21.15.43:0/2281013060 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564717106880 0x564717108c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.966+0000 7f8e8d600640 1 -- 172.21.15.43:0/2281013060 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564716d753f0 con 0x5647170fd380 2024-03-28T22:47:46.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.966+0000 7f8e87e00640 1 --2- 172.21.15.43:0/2281013060 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564717106880 0x564717108c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:47:46.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.966+0000 7f8e8d600640 1 --2- 172.21.15.43:0/2281013060 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5647170fd380 0x5647170fd760 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f8e80005f80 tx=0x7f8e800098c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ca586325b85f751 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:46.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.966+0000 7f8e85600640 1 --2- 172.21.15.43:0/283289282 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f8e700a1ef0 0x7f8e700a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.966+0000 7f8e85600640 1 -- 172.21.15.43:0/283289282 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9539+0+0 (secure 0 0 0) 0x7f8e7c0ce900 con 0x7f8e88069d20 2024-03-28T22:47:46.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.966+0000 7f8e8cc00640 1 --2- 172.21.15.43:0/283289282 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f8e700a1ef0 0x7f8e700a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.966+0000 7f8e79c00640 1 -- 172.21.15.43:0/2281013060 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e80043070 con 0x5647170fd380 2024-03-28T22:47:46.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.966+0000 7f8e79c00640 1 -- 172.21.15.43:0/2281013060 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e80003e50 con 0x5647170fd380 2024-03-28T22:47:46.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.966+0000 7f8e79c00640 1 -- 172.21.15.43:0/2281013060 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e80039040 con 0x5647170fd380 2024-03-28T22:47:46.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.967+0000 7f8e9241c740 1 -- 172.21.15.43:0/2281013060 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5647170fd380 msgr2=0x5647170fd760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.967+0000 7f8e8cc00640 1 --2- 172.21.15.43:0/283289282 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f8e700a1ef0 0x7f8e700a43b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x5647170fce50 tx=0x7f8e7c0d4530 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:46.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.967+0000 7f8e9241c740 1 --2- 172.21.15.43:0/2281013060 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5647170fd380 0x5647170fd760 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f8e80005f80 tx=0x7f8e800098c0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.967+0000 7f8e9241c740 1 -- 172.21.15.43:0/2281013060 shutdown_connections 2024-03-28T22:47:46.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.967+0000 7f8e9241c740 1 --2- 172.21.15.43:0/2281013060 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x564717106880 0x564717108c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.967+0000 7f8e9241c740 1 --2- 172.21.15.43:0/2281013060 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5647170fdca0 0x564717106340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.967+0000 7f8e9241c740 1 --2- 172.21.15.43:0/2281013060 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5647170fd380 0x5647170fd760 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.967+0000 7f8e9241c740 1 -- 172.21.15.43:0/2281013060 >> 172.21.15.43:0/2281013060 conn(0x5647170f7ab0 msgr2=0x5647170f7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:46.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.967+0000 7f8e9241c740 1 -- 172.21.15.43:0/2281013060 shutdown_connections 2024-03-28T22:47:46.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.967+0000 7f8e9241c740 1 -- 172.21.15.43:0/2281013060 wait complete. 2024-03-28T22:47:46.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.968+0000 7f8e9241c740 1 -- start start 2024-03-28T22:47:46.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.968+0000 7f8e9241c740 1 Processor -- start 2024-03-28T22:47:46.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.968+0000 7f8e9241c740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5647170fd380 0x564717103870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.968+0000 7f8e9241c740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5647170fdca0 0x564717103db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.968+0000 7f8e9241c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564717106880 0x5647171042f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.968+0000 7f8e9241c740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x564717109610 con 0x5647170fdca0 2024-03-28T22:47:46.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.968+0000 7f8e9241c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x564716fff2f0 con 0x564717106880 2024-03-28T22:47:46.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.968+0000 7f8e9241c740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x564717109790 con 0x5647170fd380 2024-03-28T22:47:46.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.968+0000 7f8e9241c740 10 client.0 _ll_register_callbacks cb 0x564716d74be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:47:46.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.969+0000 7f8e8d600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5647170fd380 0x564717103870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.969+0000 7f8e87e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5647170fdca0 0x564717103db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.969+0000 7f8e8d600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5647170fd380 0x564717103870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:34448/0 (socket says 172.21.15.43:34448) 2024-03-28T22:47:46.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.969+0000 7f8e8d600640 1 -- 172.21.15.43:0/3958116029 learned_addr learned my addr 172.21.15.43:0/3958116029 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:46.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.969+0000 7f8e8d600640 1 --2- 172.21.15.43:0/3958116029 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564717106880 0x5647171042f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.969+0000 7f8e8d600640 1 -- 172.21.15.43:0/3958116029 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564717106880 msgr2=0x5647171042f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:46.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.969+0000 7f8e8d600640 1 --2- 172.21.15.43:0/3958116029 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564717106880 0x5647171042f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.969+0000 7f8e8d600640 1 -- 172.21.15.43:0/3958116029 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5647170fdca0 msgr2=0x564717103db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:46.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.969+0000 7f8e8d600640 1 --2- 172.21.15.43:0/3958116029 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5647170fdca0 0x564717103db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.969+0000 7f8e8d600640 1 -- 172.21.15.43:0/3958116029 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564716e3f7d0 con 0x5647170fd380 2024-03-28T22:47:46.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.969+0000 7f8e87e00640 1 --2- 172.21.15.43:0/3958116029 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5647170fdca0 0x564717103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:46.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.969+0000 7f8e8d600640 1 --2- 172.21.15.43:0/3958116029 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5647170fd380 0x564717103870 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f8e80009930 tx=0x7f8e8002e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ddc17f161e032a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:46.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.969+0000 7f8e6e000640 10 client.0 ms_handle_connect on v2:172.21.15.172:3300/0 2024-03-28T22:47:46.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.969+0000 7f8e6e000640 1 -- 172.21.15.43:0/3958116029 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e80039030 con 0x5647170fd380 2024-03-28T22:47:46.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.970+0000 7f8e6e000640 1 -- 172.21.15.43:0/3958116029 <== mon.1 v2:172.21.15.172:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9539+0+0 (secure 0 0 0) 0x7f8e80004600 con 0x5647170fd380 2024-03-28T22:47:46.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.970+0000 7f8e9241c740 1 -- 172.21.15.43:0/3958116029 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e80003e50 con 0x5647170fd380 2024-03-28T22:47:46.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.970+0000 7f8e6e000640 1 -- 172.21.15.43:0/3958116029 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8e80043040 con 0x5647170fd380 2024-03-28T22:47:46.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.970+0000 7f8e9241c740 10 client.29122 fetch_fsmap learned FSMap version 32 2024-03-28T22:47:46.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.970+0000 7f8e9241c740 1 -- 172.21.15.43:0/3958116029 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564717109610 con 0x5647170fd380 2024-03-28T22:47:46.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.970+0000 7f8e6e000640 1 -- 172.21.15.43:0/3958116029 <== mon.1 v2:172.21.15.172:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e80003e50 con 0x5647170fd380 2024-03-28T22:47:46.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.970+0000 7f8e6e000640 1 -- 172.21.15.43:0/3958116029 <== mon.1 v2:172.21.15.172:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f8e80032b30 con 0x5647170fd380 2024-03-28T22:47:46.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.970+0000 7f8e9241c740 10 client.29122 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:47:46.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.970+0000 7f8e9241c740 10 client.29122 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:47:46.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.970+0000 7f8e9241c740 20 client.29122 populate_metadata read hostname 'smithi043' 2024-03-28T22:47:46.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.970+0000 7f8e9241c740 1 --2- 172.21.15.43:0/3958116029 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x564717116a60 0x564717118e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.970+0000 7f8e9241c740 4 client.29122 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T22:47:46.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.970+0000 7f8e9241c740 1 -- 172.21.15.43:0/3958116029 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56471700a680 con 0x564717116a60 2024-03-28T22:47:46.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.971+0000 7f8e87e00640 1 --2- 172.21.15.43:0/3958116029 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x564717116a60 0x564717118e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.971+0000 7f8e87e00640 1 --2- 172.21.15.43:0/3958116029 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x564717116a60 0x564717118e50 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=605a096eecdbf084 server_cookie=ec0b908b2652ad70 in_seq=0 out_seq=0 2024-03-28T22:47:46.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.971+0000 7f8e6e000640 10 client.29122 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:47:46.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.972+0000 7f8e6e000640 1 -- 172.21.15.43:0/3958116029 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56471700a680 con 0x564717116a60 2024-03-28T22:47:46.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.972+0000 7f8e6e000640 10 client.29122 handle_command_reply: tid=0 2024-03-28T22:47:46.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.972+0000 7f8e9241c740 1 client.29122 shutdown 2024-03-28T22:47:46.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.972+0000 7f8e9241c740 1 -- 172.21.15.43:0/3958116029 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5647170fd380 msgr2=0x564717103870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.972+0000 7f8e9241c740 1 --2- 172.21.15.43:0/3958116029 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5647170fd380 0x564717103870 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f8e80009930 tx=0x7f8e8002e040 comp rx=0 tx=0).stop 2024-03-28T22:47:46.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.972+0000 7f8e9241c740 1 -- 172.21.15.43:0/3958116029 shutdown_connections 2024-03-28T22:47:46.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.972+0000 7f8e9241c740 1 --2- 172.21.15.43:0/3958116029 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x564717116a60 0x564717118e50 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.972+0000 7f8e9241c740 1 --2- 172.21.15.43:0/3958116029 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564717106880 0x5647171042f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.972+0000 7f8e9241c740 1 --2- 172.21.15.43:0/3958116029 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5647170fdca0 0x564717103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.972+0000 7f8e9241c740 1 --2- 172.21.15.43:0/3958116029 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x5647170fd380 0x564717103870 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.972+0000 7f8e9241c740 1 -- 172.21.15.43:0/3958116029 >> 172.21.15.43:0/3958116029 conn(0x5647170f7ab0 msgr2=0x5647170f9800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:46.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.972+0000 7f8e6e000640 0 client.29122 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:47:46.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.973+0000 7f8e9241c740 1 -- 172.21.15.43:0/3958116029 shutdown_connections 2024-03-28T22:47:46.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.973+0000 7f8e9241c740 1 -- 172.21.15.43:0/3958116029 wait complete. 2024-03-28T22:47:46.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.973+0000 7f8e9241c740 20 client.29122 trim_cache size 0 max 16384 2024-03-28T22:47:46.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.987+0000 7f8e9241c740 1 Processor -- start 2024-03-28T22:47:46.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.988+0000 7f8e9241c740 1 -- start start 2024-03-28T22:47:46.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.988+0000 7f8e9241c740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5647170fd380 0x5647170f6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.988+0000 7f8e9241c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5647170fdca0 0x5647171123e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.988+0000 7f8e9241c740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564717106880 0x564717104110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.988+0000 7f8e9241c740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8e881b0f40 con 0x5647170fd380 2024-03-28T22:47:46.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.988+0000 7f8e9241c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8e881b2a80 con 0x5647170fdca0 2024-03-28T22:47:46.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.988+0000 7f8e9241c740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f8e881b1610 con 0x564717106880 2024-03-28T22:47:46.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.988+0000 7f8e87e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5647170fdca0 0x5647171123e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.988+0000 7f8e8d600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5647170fd380 0x5647170f6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.988+0000 7f8e8d600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5647170fd380 0x5647170f6db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37470/0 (socket says 172.21.15.43:37470) 2024-03-28T22:47:46.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.988+0000 7f8e87e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5647170fdca0 0x5647171123e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59124/0 (socket says 172.21.15.43:59124) 2024-03-28T22:47:46.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.988+0000 7f8e8d600640 1 -- 172.21.15.43:0/190338403 learned_addr learned my addr 172.21.15.43:0/190338403 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:46.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.988+0000 7f8e8d600640 1 --2- 172.21.15.43:0/190338403 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564717106880 0x564717104110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.989+0000 7f8e87e00640 1 -- 172.21.15.43:0/190338403 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564717106880 msgr2=0x564717104110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:46.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.989+0000 7f8e87e00640 1 --2- 172.21.15.43:0/190338403 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564717106880 0x564717104110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.989+0000 7f8e87e00640 1 -- 172.21.15.43:0/190338403 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5647170fd380 msgr2=0x5647170f6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:46.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.989+0000 7f8e87e00640 1 --2- 172.21.15.43:0/190338403 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5647170fd380 0x5647170f6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.989+0000 7f8e87e00640 1 -- 172.21.15.43:0/190338403 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564717112920 con 0x5647170fdca0 2024-03-28T22:47:46.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.989+0000 7f8e8d600640 1 --2- 172.21.15.43:0/190338403 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564717106880 0x564717104110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:46.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.989+0000 7f8e87e00640 1 --2- 172.21.15.43:0/190338403 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5647170fdca0 0x5647171123e0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x564717119480 tx=0x7f8e74005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c20753e3d6244aa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:46.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.989+0000 7f8e79200640 1 -- 172.21.15.43:0/190338403 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e74028070 con 0x5647170fdca0 2024-03-28T22:47:46.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.989+0000 7f8e79200640 1 -- 172.21.15.43:0/190338403 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e74012110 con 0x5647170fdca0 2024-03-28T22:47:46.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.989+0000 7f8e79200640 1 -- 172.21.15.43:0/190338403 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e740035f0 con 0x5647170fdca0 2024-03-28T22:47:46.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.990+0000 7f8e9241c740 1 -- 172.21.15.43:0/190338403 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5647170fdca0 msgr2=0x5647171123e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:46.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.990+0000 7f8e9241c740 1 --2- 172.21.15.43:0/190338403 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5647170fdca0 0x5647171123e0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x564717119480 tx=0x7f8e74005f90 comp rx=0 tx=0).stop 2024-03-28T22:47:46.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.990+0000 7f8e9241c740 1 -- 172.21.15.43:0/190338403 shutdown_connections 2024-03-28T22:47:46.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.990+0000 7f8e9241c740 1 --2- 172.21.15.43:0/190338403 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5647170fdca0 0x5647171123e0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.990+0000 7f8e9241c740 1 --2- 172.21.15.43:0/190338403 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564717106880 0x564717104110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.990+0000 7f8e9241c740 1 --2- 172.21.15.43:0/190338403 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5647170fd380 0x5647170f6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.990+0000 7f8e9241c740 1 -- 172.21.15.43:0/190338403 >> 172.21.15.43:0/190338403 conn(0x5647170f7ab0 msgr2=0x5647171028a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:46.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.990+0000 7f8e9241c740 1 -- 172.21.15.43:0/190338403 shutdown_connections 2024-03-28T22:47:46.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.990+0000 7f8e9241c740 1 -- 172.21.15.43:0/190338403 wait complete. 2024-03-28T22:47:46.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.990+0000 7f8e9241c740 1 -- start start 2024-03-28T22:47:46.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.991+0000 7f8e9241c740 1 Processor -- start 2024-03-28T22:47:46.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.991+0000 7f8e9241c740 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5647170fd380 0x564717100620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.991+0000 7f8e9241c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5647170fdca0 0x564717100b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.991+0000 7f8e9241c740 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564717106880 0x5647171010a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.991+0000 7f8e9241c740 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8e881b23b0 con 0x5647170fd380 2024-03-28T22:47:46.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.991+0000 7f8e9241c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8e881b1ce0 con 0x5647170fdca0 2024-03-28T22:47:46.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.991+0000 7f8e9241c740 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f8e881aef40 con 0x564717106880 2024-03-28T22:47:46.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.991+0000 7f8e9241c740 10 client.0 _ll_register_callbacks cb 0x7f8e7c0d4400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T22:47:46.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.991+0000 7f8e87e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5647170fdca0 0x564717100b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.991+0000 7f8e87e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5647170fdca0 0x564717100b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59126/0 (socket says 172.21.15.43:59126) 2024-03-28T22:47:46.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.991+0000 7f8e8d600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5647170fd380 0x564717100620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.991+0000 7f8e87e00640 1 -- 172.21.15.43:0/3370041278 learned_addr learned my addr 172.21.15.43:0/3370041278 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:46.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.991+0000 7f8e8d600640 1 --2- 172.21.15.43:0/3370041278 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564717106880 0x5647171010a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.992+0000 7f8e87e00640 1 -- 172.21.15.43:0/3370041278 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564717106880 msgr2=0x5647171010a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:46.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.992+0000 7f8e87e00640 1 --2- 172.21.15.43:0/3370041278 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564717106880 0x5647171010a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.992+0000 7f8e87e00640 1 -- 172.21.15.43:0/3370041278 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5647170fd380 msgr2=0x564717100620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T22:47:46.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.992+0000 7f8e87e00640 1 --2- 172.21.15.43:0/3370041278 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5647170fd380 0x564717100620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:46.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.992+0000 7f8e87e00640 1 -- 172.21.15.43:0/3370041278 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56471717bd60 con 0x5647170fdca0 2024-03-28T22:47:46.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.992+0000 7f8e8d600640 1 --2- 172.21.15.43:0/3370041278 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5647170fd380 0x564717100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T22:47:46.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.992+0000 7f8e87e00640 1 --2- 172.21.15.43:0/3370041278 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5647170fdca0 0x564717100b60 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f8e74005d00 tx=0x7f8e74005780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ee39005207b35e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:46.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.992+0000 7f8e6e000640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-28T22:47:46.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.992+0000 7f8e6e000640 1 -- 172.21.15.43:0/3370041278 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e74032060 con 0x5647170fdca0 2024-03-28T22:47:46.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.992+0000 7f8e6e000640 1 -- 172.21.15.43:0/3370041278 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9539+0+0 (secure 0 0 0) 0x7f8e74022550 con 0x5647170fdca0 2024-03-28T22:47:46.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.992+0000 7f8e9241c740 1 -- 172.21.15.43:0/3370041278 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e74012110 con 0x5647170fdca0 2024-03-28T22:47:46.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.993+0000 7f8e6e000640 1 -- 172.21.15.43:0/3370041278 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8e74039050 con 0x5647170fdca0 2024-03-28T22:47:46.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.993+0000 7f8e9241c740 10 client.29186 fetch_fsmap learned FSMap version 32 2024-03-28T22:47:46.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.993+0000 7f8e9241c740 1 -- 172.21.15.43:0/3370041278 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56471717d020 con 0x5647170fdca0 2024-03-28T22:47:46.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.993+0000 7f8e6e000640 1 -- 172.21.15.43:0/3370041278 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e74012110 con 0x5647170fdca0 2024-03-28T22:47:46.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.993+0000 7f8e6e000640 1 -- 172.21.15.43:0/3370041278 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f8e74003980 con 0x5647170fdca0 2024-03-28T22:47:46.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.993+0000 7f8e9241c740 10 client.29186 fetch_fsmap finished waiting for FSMap version 32 2024-03-28T22:47:46.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.993+0000 7f8e9241c740 10 client.29186 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T22:47:46.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.993+0000 7f8e9241c740 20 client.29186 populate_metadata read hostname 'smithi043' 2024-03-28T22:47:46.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.993+0000 7f8e9241c740 1 --2- 172.21.15.43:0/3370041278 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56471717d660 0x56471717fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:46.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.993+0000 7f8e9241c740 4 client.29186 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T22:47:46.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.993+0000 7f8e9241c740 1 -- 172.21.15.43:0/3370041278 --> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56471706e590 con 0x56471717d660 2024-03-28T22:47:46.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.993+0000 7f8e8d600640 1 --2- 172.21.15.43:0/3370041278 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56471717d660 0x56471717fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:46.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.994+0000 7f8e8d600640 1 --2- 172.21.15.43:0/3370041278 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56471717d660 0x56471717fa80 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be2af89175a11ed0 server_cookie=5495842ff5de233c in_seq=0 out_seq=0 2024-03-28T22:47:46.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.994+0000 7f8e6e000640 10 client.29186 ms_handle_connect on v2:172.21.15.82:6838/2220892357 2024-03-28T22:47:46.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.994+0000 7f8e6e000640 1 -- 172.21.15.43:0/3370041278 <== mds.0 v2:172.21.15.82:6838/2220892357 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56471706e590 con 0x56471717d660 2024-03-28T22:47:46.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.994+0000 7f8e6e000640 10 client.29186 handle_command_reply: tid=0 2024-03-28T22:47:46.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.994+0000 7f8e9241c740 1 client.29186 shutdown 2024-03-28T22:47:47.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.994+0000 7f8e9241c740 1 -- 172.21.15.43:0/3370041278 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5647170fdca0 msgr2=0x564717100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:47.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.994+0000 7f8e9241c740 1 --2- 172.21.15.43:0/3370041278 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5647170fdca0 0x564717100b60 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f8e74005d00 tx=0x7f8e74005780 comp rx=0 tx=0).stop 2024-03-28T22:47:47.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.994+0000 7f8e9241c740 1 -- 172.21.15.43:0/3370041278 shutdown_connections 2024-03-28T22:47:47.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.994+0000 7f8e9241c740 1 --2- 172.21.15.43:0/3370041278 >> [v2:172.21.15.82:6838/2220892357,v1:172.21.15.82:6839/2220892357] conn(0x56471717d660 0x56471717fa80 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:47.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.994+0000 7f8e9241c740 1 --2- 172.21.15.43:0/3370041278 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5647170fdca0 0x564717100b60 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:47.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.994+0000 7f8e9241c740 1 --2- 172.21.15.43:0/3370041278 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x564717106880 0x5647171010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:47.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.994+0000 7f8e6e000640 0 client.29186 ms_handle_reset on v2:172.21.15.82:6838/2220892357 2024-03-28T22:47:47.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.994+0000 7f8e9241c740 1 --2- 172.21.15.43:0/3370041278 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x5647170fd380 0x564717100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:47.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.994+0000 7f8e9241c740 1 -- 172.21.15.43:0/3370041278 >> 172.21.15.43:0/3370041278 conn(0x5647170f7ab0 msgr2=0x5647170f8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:47.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.994+0000 7f8e9241c740 1 -- 172.21.15.43:0/3370041278 shutdown_connections 2024-03-28T22:47:47.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.994+0000 7f8e9241c740 1 -- 172.21.15.43:0/3370041278 wait complete. 2024-03-28T22:47:47.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.995+0000 7f8e9241c740 20 client.29186 trim_cache size 0 max 16384 2024-03-28T22:47:47.001 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-03-28T22:47:47.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.995+0000 7f8e8f600640 1 -- 172.21.15.43:0/283289282 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f8e700a1ef0 msgr2=0x7f8e700a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:47.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.995+0000 7f8e8f600640 1 --2- 172.21.15.43:0/283289282 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f8e700a1ef0 0x7f8e700a43b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x5647170fce50 tx=0x7f8e7c0d4530 comp rx=0 tx=0).stop 2024-03-28T22:47:47.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.995+0000 7f8e8f600640 1 -- 172.21.15.43:0/283289282 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8e88069d20 msgr2=0x7f8e8819a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:47.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.995+0000 7f8e8f600640 1 --2- 172.21.15.43:0/283289282 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8e88069d20 0x7f8e8819a170 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c00e9f0 tx=0x7f8e7c00eeb0 comp rx=0 tx=0).stop 2024-03-28T22:47:47.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.995+0000 7f8e8f600640 1 -- 172.21.15.43:0/283289282 shutdown_connections 2024-03-28T22:47:47.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.995+0000 7f8e8f600640 1 --2- 172.21.15.43:0/283289282 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f8e700a1ef0 0x7f8e700a43b0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:47.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.995+0000 7f8e8f600640 1 --2- 172.21.15.43:0/283289282 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8e88107740 0x7f8e8819ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:47.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.995+0000 7f8e8f600640 1 --2- 172.21.15.43:0/283289282 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e8806a6c0 0x7f8e8819a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:47.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.995+0000 7f8e8f600640 1 --2- 172.21.15.43:0/283289282 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8e88069d20 0x7f8e8819a170 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:47.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.995+0000 7f8e8f600640 1 -- 172.21.15.43:0/283289282 >> 172.21.15.43:0/283289282 conn(0x7f8e880776e0 msgr2=0x7f8e880fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:47.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.995+0000 7f8e8f600640 1 -- 172.21.15.43:0/283289282 shutdown_connections 2024-03-28T22:47:47.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:46.995+0000 7f8e8f600640 1 -- 172.21.15.43:0/283289282 wait complete. 2024-03-28T22:47:47.009 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:46 smithi043 ceph-mon[31823]: pgmap v331: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:47:47.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:46 smithi172 ceph-mon[36566]: pgmap v331: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:47:47.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:46 smithi082 ceph-mon[36471]: pgmap v331: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:47:48.012 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-03-28T22:47:48.012 INFO:tasks.fwd_scrub:done joining 2024-03-28T22:47:48.013 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-28T22:47:48.023 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:47:48.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.260+0000 7fa863200640 1 -- 172.21.15.43:0/2295493683 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa85c107710 msgr2=0x7fa85c107af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:48.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.260+0000 7fa863200640 1 --2- 172.21.15.43:0/2295493683 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa85c107710 0x7fa85c107af0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fa84c009ca0 tx=0x7fa84c02c0f0 comp rx=0 tx=0).stop 2024-03-28T22:47:48.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.260+0000 7fa863200640 1 -- 172.21.15.43:0/2295493683 shutdown_connections 2024-03-28T22:47:48.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.260+0000 7fa863200640 1 --2- 172.21.15.43:0/2295493683 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa85c06a6a0 0x7fa85c10e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:48.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.260+0000 7fa863200640 1 --2- 172.21.15.43:0/2295493683 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa85c069d20 0x7fa85c06a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:48.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.260+0000 7fa863200640 1 --2- 172.21.15.43:0/2295493683 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa85c107710 0x7fa85c107af0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:48.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.260+0000 7fa863200640 1 -- 172.21.15.43:0/2295493683 >> 172.21.15.43:0/2295493683 conn(0x7fa85c077630 msgr2=0x7fa85c077a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:48.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.260+0000 7fa863200640 1 -- 172.21.15.43:0/2295493683 shutdown_connections 2024-03-28T22:47:48.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.261+0000 7fa863200640 1 -- 172.21.15.43:0/2295493683 wait complete. 2024-03-28T22:47:48.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.262+0000 7fa863200640 1 Processor -- start 2024-03-28T22:47:48.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.262+0000 7fa863200640 1 -- start start 2024-03-28T22:47:48.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.263+0000 7fa863200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa85c069d20 0x7fa85c197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:48.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.263+0000 7fa863200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa85c06a6a0 0x7fa85c198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:48.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.263+0000 7fa863200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa85c107710 0x7fa85c19c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:48.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.263+0000 7fa863200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa85c111b10 con 0x7fa85c06a6a0 2024-03-28T22:47:48.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.263+0000 7fa863200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa85c111990 con 0x7fa85c107710 2024-03-28T22:47:48.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.263+0000 7fa863200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fa85c111c90 con 0x7fa85c069d20 2024-03-28T22:47:48.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.263+0000 7fa85be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa85c069d20 0x7fa85c197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:48.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.263+0000 7fa85be00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa85c069d20 0x7fa85c197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:34494/0 (socket says 172.21.15.43:34494) 2024-03-28T22:47:48.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.263+0000 7fa85be00640 1 -- 172.21.15.43:0/1397241373 learned_addr learned my addr 172.21.15.43:0/1397241373 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:48.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.263+0000 7fa853400640 1 --2- 172.21.15.43:0/1397241373 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa85c06a6a0 0x7fa85c198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:48.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.263+0000 7fa861200640 1 --2- 172.21.15.43:0/1397241373 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa85c107710 0x7fa85c19c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:48.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.263+0000 7fa85be00640 1 -- 172.21.15.43:0/1397241373 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa85c107710 msgr2=0x7fa85c19c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:48.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.263+0000 7fa85be00640 1 --2- 172.21.15.43:0/1397241373 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa85c107710 0x7fa85c19c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:48.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.263+0000 7fa85be00640 1 -- 172.21.15.43:0/1397241373 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa85c06a6a0 msgr2=0x7fa85c198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:48.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.263+0000 7fa85be00640 1 --2- 172.21.15.43:0/1397241373 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa85c06a6a0 0x7fa85c198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:48.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.263+0000 7fa85be00640 1 -- 172.21.15.43:0/1397241373 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa85c19cfe0 con 0x7fa85c069d20 2024-03-28T22:47:48.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.264+0000 7fa85be00640 1 --2- 172.21.15.43:0/1397241373 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa85c069d20 0x7fa85c197ff0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fa84c005e30 tx=0x7fa84c032f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:48.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.264+0000 7fa859600640 1 -- 172.21.15.43:0/1397241373 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa84c042020 con 0x7fa85c069d20 2024-03-28T22:47:48.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.264+0000 7fa863200640 1 -- 172.21.15.43:0/1397241373 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa85c19d270 con 0x7fa85c069d20 2024-03-28T22:47:48.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.264+0000 7fa859600640 1 -- 172.21.15.43:0/1397241373 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa84c034640 con 0x7fa85c069d20 2024-03-28T22:47:48.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.264+0000 7fa859600640 1 -- 172.21.15.43:0/1397241373 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa84c039040 con 0x7fa85c069d20 2024-03-28T22:47:48.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.264+0000 7fa863200640 1 -- 172.21.15.43:0/1397241373 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa85c1a4b60 con 0x7fa85c069d20 2024-03-28T22:47:48.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.265+0000 7fa863200640 1 -- 172.21.15.43:0/1397241373 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa85c102e40 con 0x7fa85c069d20 2024-03-28T22:47:48.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.267+0000 7fa859600640 1 -- 172.21.15.43:0/1397241373 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fa84c045020 con 0x7fa85c069d20 2024-03-28T22:47:48.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.268+0000 7fa859600640 1 --2- 172.21.15.43:0/1397241373 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa8400a1ef0 0x7fa8400a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:48.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.269+0000 7fa859600640 1 -- 172.21.15.43:0/1397241373 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9539+0+0 (secure 0 0 0) 0x7fa84c0f0100 con 0x7fa85c069d20 2024-03-28T22:47:48.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.269+0000 7fa853400640 1 --2- 172.21.15.43:0/1397241373 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa8400a1ef0 0x7fa8400a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:48.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.269+0000 7fa853400640 1 --2- 172.21.15.43:0/1397241373 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa8400a1ef0 0x7fa8400a43b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa85c100470 tx=0x7fa844005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:48.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.273+0000 7fa859600640 1 -- 172.21.15.43:0/1397241373 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa84c04c020 con 0x7fa85c069d20 2024-03-28T22:47:48.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.477+0000 7fa863200640 1 -- 172.21.15.43:0/1397241373 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa85c109c20 con 0x7fa85c069d20 2024-03-28T22:47:48.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.481+0000 7fa859600640 1 -- 172.21.15.43:0/1397241373 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) ==== 74+0+25334 (secure 0 0 0) 0x7fa84c0bcea0 con 0x7fa85c069d20 2024-03-28T22:47:48.479 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:47:48.479 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":108,"fsid":"c2276428-ed51-11ee-95d0-87774f69a715","created":"2024-03-28T22:24:19.950084+0000","modified":"2024-03-28T22:47:17.836542+0000","last_up_change":"2024-03-28T22:32:31.078737+0000","last_in_change":"2024-03-28T22:32:10.147097+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:28:14.713290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:33:27.167435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:33:28.411825+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T22:33:32.523122+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b855adcd-786c-4e42-b8fe-a515a77ce979","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6803","nonce":2826124409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6805","nonce":2826124409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6809","nonce":2826124409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6807","nonce":2826124409}]},"public_addr":"172.21.15.43:6803/2826124409","cluster_addr":"172.21.15.43:6805/2826124409","heartbeat_back_addr":"172.21.15.43:6809/2826124409","heartbeat_front_addr":"172.21.15.43:6807/2826124409","state":["exists","up"]},{"osd":1,"uuid":"031c143d-8ad1-4b68-8db9-3c717137e3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6801","nonce":3043263644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6803","nonce":3043263644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6807","nonce":3043263644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6805","nonce":3043263644}]},"public_addr":"172.21.15.82:6801/3043263644","cluster_addr":"172.21.15.82:6803/3043263644","heartbeat_back_addr":"172.21.15.82:6807/3043263644","heartbeat_front_addr":"172.21.15.82:6805/3043263644","state":["exists","up"]},{"osd":2,"uuid":"20632bbb-f07a-4ca1-b0c4-ff9d508e211c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6801","nonce":4213953645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6803","nonce":4213953645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6807","nonce":4213953645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6805","nonce":4213953645}]},"public_addr":"172.21.15.172:6801/4213953645","cluster_addr":"172.21.15.172:6803/4213953645","heartbeat_back_addr":"172.21.15.172:6807/4213953645","heartbeat_front_addr":"172.21.15.172:6805/4213953645","state":["exists","up"]},{"osd":3,"uuid":"9a1f2312-4dbd-475b-8736-d5ebc8788e09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6811","nonce":1597280920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6813","nonce":1597280920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6817","nonce":1597280920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6815","nonce":1597280920}]},"public_addr":"172.21.15.43:6811/1597280920","cluster_addr":"172.21.15.43:6813/1597280920","heartbeat_back_addr":"172.21.15.43:6817/1597280920","heartbeat_front_addr":"172.21.15.43:6815/1597280920","state":["exists","up"]},{"osd":4,"uuid":"29140ead-7075-42b2-90c1-27cae0b29ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6809","nonce":215451895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6811","nonce":215451895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6815","nonce":215451895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6813","nonce":215451895}]},"public_addr":"172.21.15.82:6809/215451895","cluster_addr":"172.21.15.82:6811/215451895","heartbeat_back_addr":"172.21.15.82:6815/215451895","heartbeat_front_addr":"172.21.15.82:6813/215451895","state":["exists","up"]},{"osd":5,"uuid":"59644c61-93dc-4536-b2ad-0256b24e5cb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6809","nonce":1576148933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6811","nonce":1576148933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6815","nonce":1576148933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6813","nonce":1576148933}]},"public_addr":"172.21.15.172:6809/1576148933","cluster_addr":"172.21.15.172:6811/1576148933","heartbeat_back_addr":"172.21.15.172:6815/1576148933","heartbeat_front_addr":"172.21.15.172:6813/1576148933","state":["exists","up"]},{"osd":6,"uuid":"22d54e57-c3f1-4ecf-9921-8e815b48f92a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6819","nonce":791044279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6821","nonce":791044279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6825","nonce":791044279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6823","nonce":791044279}]},"public_addr":"172.21.15.43:6819/791044279","cluster_addr":"172.21.15.43:6821/791044279","heartbeat_back_addr":"172.21.15.43:6825/791044279","heartbeat_front_addr":"172.21.15.43:6823/791044279","state":["exists","up"]},{"osd":7,"uuid":"248cd1e2-0d6b-452f-a0a8-c51fe1b975ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6817","nonce":2157777323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6819","nonce":2157777323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6823","nonce":2157777323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6821","nonce":2157777323}]},"public_addr":"172.21.15.82:6817/2157777323","cluster_addr":"172.21.15.82:6819/2157777323","heartbeat_back_addr":"172.21.15.82:6823/2157777323","heartbeat_front_addr":"172.21.15.82:6821/2157777323","state":["exists","up"]},{"osd":8,"uuid":"d98aab6e-8bae-4c24-906c-4f492f047b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6817","nonce":1095328248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6819","nonce":1095328248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6823","nonce":1095328248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6821","nonce":1095328248}]},"public_addr":"172.21.15.172:6817/1095328248","cluster_addr":"172.21.15.172:6819/1095328248","heartbeat_back_addr":"172.21.15.172:6823/1095328248","heartbeat_front_addr":"172.21.15.172:6821/1095328248","state":["exists","up"]},{"osd":9,"uuid":"19aa383e-a0f6-45ea-96cc-e7956535caa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6827","nonce":3657791498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6829","nonce":3657791498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6833","nonce":3657791498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6831","nonce":3657791498}]},"public_addr":"172.21.15.43:6827/3657791498","cluster_addr":"172.21.15.43:6829/3657791498","heartbeat_back_addr":"172.21.15.43:6833/3657791498","heartbeat_front_addr":"172.21.15.43:6831/3657791498","state":["exists","up"]},{"osd":10,"uuid":"666977e5-eaf7-41b9-a18f-c380e149345d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6825","nonce":3454954047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6827","nonce":3454954047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6831","nonce":3454954047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6829","nonce":3454954047}]},"public_addr":"172.21.15.82:6825/3454954047","cluster_addr":"172.21.15.82:6827/3454954047","heartbeat_back_addr":"172.21.15.82:6831/3454954047","heartbeat_front_addr":"172.21.15.82:6829/3454954047","state":["exists","up"]},{"osd":11,"uuid":"615cf3be-8076-470e-ad09-c880e7593717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6825","nonce":2563517144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6827","nonce":2563517144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6831","nonce":2563517144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6829","nonce":2563517144}]},"public_addr":"172.21.15.172:6825/2563517144","cluster_addr":"172.21.15.172:6827/2563517144","heartbeat_back_addr":"172.21.15.172:6831/2563517144","heartbeat_front_addr":"172.21.15.172:6829/2563517144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:21.727405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:47.100259+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:10.615438+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:43.632144+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:09.487940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:34.604463+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:09.594452+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:35.709795+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:01.184202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:35.455322+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:02.413166+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:28.550754+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.9","mappings":[{"from":5,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/1567355374":"2024-03-29T22:37:11.308878+0000","172.21.15.43:6800/3517712438":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6801/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/686050369":"2024-03-29T22:37:11.308878+0000","172.21.15.43:0/2710117652":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/1375099638":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6800/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/3215634168":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6801/3517712438":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/4029313215":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/1343671266":"2024-03-29T22:37:11.308878+0000","172.21.15.43:0/578791143":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/4024174130":"2024-03-29T22:37:11.308878+0000","172.21.15.43:0/1574178482":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/141623093":"2024-03-29T22:37:11.308878+0000","172.21.15.43:6800/369798199":"2024-03-29T22:37:11.308878+0000","172.21.15.43:6801/369798199":"2024-03-29T22:37:11.308878+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:47:48.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.483+0000 7fa863200640 1 -- 172.21.15.43:0/1397241373 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa8400a1ef0 msgr2=0x7fa8400a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:48.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.483+0000 7fa863200640 1 --2- 172.21.15.43:0/1397241373 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa8400a1ef0 0x7fa8400a43b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa85c100470 tx=0x7fa844005f90 comp rx=0 tx=0).stop 2024-03-28T22:47:48.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.483+0000 7fa863200640 1 -- 172.21.15.43:0/1397241373 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa85c069d20 msgr2=0x7fa85c197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:48.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.483+0000 7fa863200640 1 --2- 172.21.15.43:0/1397241373 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa85c069d20 0x7fa85c197ff0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fa84c005e30 tx=0x7fa84c032f20 comp rx=0 tx=0).stop 2024-03-28T22:47:48.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.483+0000 7fa863200640 1 -- 172.21.15.43:0/1397241373 shutdown_connections 2024-03-28T22:47:48.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.483+0000 7fa863200640 1 --2- 172.21.15.43:0/1397241373 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fa8400a1ef0 0x7fa8400a43b0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:48.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.483+0000 7fa863200640 1 --2- 172.21.15.43:0/1397241373 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa85c107710 0x7fa85c19c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:48.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.483+0000 7fa863200640 1 --2- 172.21.15.43:0/1397241373 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa85c06a6a0 0x7fa85c198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:48.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.483+0000 7fa863200640 1 --2- 172.21.15.43:0/1397241373 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa85c069d20 0x7fa85c197ff0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:48.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.483+0000 7fa863200640 1 -- 172.21.15.43:0/1397241373 >> 172.21.15.43:0/1397241373 conn(0x7fa85c077630 msgr2=0x7fa85c0fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:48.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.483+0000 7fa863200640 1 -- 172.21.15.43:0/1397241373 shutdown_connections 2024-03-28T22:47:48.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.483+0000 7fa863200640 1 -- 172.21.15.43:0/1397241373 wait complete. 2024-03-28T22:47:48.503 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T22:28:14.713290+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T22:33:27.167435+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T22:33:28.411825+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 107, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T22:33:32.523122+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 107, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T22:47:48.503 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T22:47:48.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.736+0000 7fc8df200640 1 -- 172.21.15.43:0/301003634 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8d81038f0 msgr2=0x7fc8d8103d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:48.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.736+0000 7fc8df200640 1 --2- 172.21.15.43:0/301003634 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8d81038f0 0x7fc8d8103d70 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4009900 tx=0x7fc8c402bc30 comp rx=0 tx=0).stop 2024-03-28T22:47:48.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.736+0000 7fc8df200640 1 -- 172.21.15.43:0/301003634 shutdown_connections 2024-03-28T22:47:48.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.736+0000 7fc8df200640 1 --2- 172.21.15.43:0/301003634 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8d81038f0 0x7fc8d8103d70 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:48.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.736+0000 7fc8df200640 1 --2- 172.21.15.43:0/301003634 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8d81042b0 0x7fc8d81107c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:48.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.736+0000 7fc8df200640 1 --2- 172.21.15.43:0/301003634 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8d8109900 0x7fc8d8109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:48.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.736+0000 7fc8df200640 1 -- 172.21.15.43:0/301003634 >> 172.21.15.43:0/301003634 conn(0x7fc8d80ff3a0 msgr2=0x7fc8d81017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:48.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.737+0000 7fc8df200640 1 -- 172.21.15.43:0/301003634 shutdown_connections 2024-03-28T22:47:48.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.737+0000 7fc8df200640 1 -- 172.21.15.43:0/301003634 wait complete. 2024-03-28T22:47:48.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.738+0000 7fc8df200640 1 Processor -- start 2024-03-28T22:47:48.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.738+0000 7fc8df200640 1 -- start start 2024-03-28T22:47:48.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.739+0000 7fc8df200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8d81038f0 0x7fc8d819a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:48.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.739+0000 7fc8df200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8d81042b0 0x7fc8d819a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:48.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.739+0000 7fc8df200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8d8109900 0x7fc8d819ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:48.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.739+0000 7fc8df200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d8113d30 con 0x7fc8d81042b0 2024-03-28T22:47:48.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.739+0000 7fc8df200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d8113bb0 con 0x7fc8d8109900 2024-03-28T22:47:48.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.739+0000 7fc8df200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d8113eb0 con 0x7fc8d81038f0 2024-03-28T22:47:48.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.739+0000 7fc8d7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8d81038f0 0x7fc8d819a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:48.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.739+0000 7fc8d7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8d81042b0 0x7fc8d819a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:48.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.739+0000 7fc8d7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8d81038f0 0x7fc8d819a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:34510/0 (socket says 172.21.15.43:34510) 2024-03-28T22:47:48.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.739+0000 7fc8d7e00640 1 -- 172.21.15.43:0/2827836376 learned_addr learned my addr 172.21.15.43:0/2827836376 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:48.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.739+0000 7fc8dd200640 1 --2- 172.21.15.43:0/2827836376 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8d8109900 0x7fc8d819ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:48.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.739+0000 7fc8dd200640 1 -- 172.21.15.43:0/2827836376 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8d81038f0 msgr2=0x7fc8d819a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:48.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.739+0000 7fc8dd200640 1 --2- 172.21.15.43:0/2827836376 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8d81038f0 0x7fc8d819a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:48.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.739+0000 7fc8dd200640 1 -- 172.21.15.43:0/2827836376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8d81042b0 msgr2=0x7fc8d819a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:48.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.739+0000 7fc8dd200640 1 --2- 172.21.15.43:0/2827836376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8d81042b0 0x7fc8d819a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:48.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.739+0000 7fc8dd200640 1 -- 172.21.15.43:0/2827836376 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8d819f1b0 con 0x7fc8d8109900 2024-03-28T22:47:48.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.739+0000 7fc8d7e00640 1 --2- 172.21.15.43:0/2827836376 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8d81038f0 0x7fc8d819a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:48.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.740+0000 7fc8dd200640 1 --2- 172.21.15.43:0/2827836376 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8d8109900 0x7fc8d819ea80 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0009fd0 tx=0x7fc8d000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:48.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.740+0000 7fc8d4c00640 1 -- 172.21.15.43:0/2827836376 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d000ce20 con 0x7fc8d8109900 2024-03-28T22:47:48.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.740+0000 7fc8d4c00640 1 -- 172.21.15.43:0/2827836376 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8d0005020 con 0x7fc8d8109900 2024-03-28T22:47:48.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.740+0000 7fc8d4c00640 1 -- 172.21.15.43:0/2827836376 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d0010720 con 0x7fc8d8109900 2024-03-28T22:47:48.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.740+0000 7fc8df200640 1 -- 172.21.15.43:0/2827836376 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8d819f4a0 con 0x7fc8d8109900 2024-03-28T22:47:48.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.740+0000 7fc8df200640 1 -- 172.21.15.43:0/2827836376 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8d81a6d30 con 0x7fc8d8109900 2024-03-28T22:47:48.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.741+0000 7fc8df200640 1 -- 172.21.15.43:0/2827836376 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8d8068990 con 0x7fc8d8109900 2024-03-28T22:47:48.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.743+0000 7fc8d4c00640 1 -- 172.21.15.43:0/2827836376 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fc8d0005540 con 0x7fc8d8109900 2024-03-28T22:47:48.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.744+0000 7fc8d4c00640 1 --2- 172.21.15.43:0/2827836376 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc8bc0a1f70 0x7fc8bc0a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:48.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.745+0000 7fc8d4c00640 1 -- 172.21.15.43:0/2827836376 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9539+0+0 (secure 0 0 0) 0x7fc8d00cec50 con 0x7fc8d8109900 2024-03-28T22:47:48.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.745+0000 7fc8d7e00640 1 --2- 172.21.15.43:0/2827836376 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc8bc0a1f70 0x7fc8bc0a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:48.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.745+0000 7fc8d7e00640 1 --2- 172.21.15.43:0/2827836376 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc8bc0a1f70 0x7fc8bc0a4430 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8011440 tx=0x7fc8c8005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:48.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.748+0000 7fc8d4c00640 1 -- 172.21.15.43:0/2827836376 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc8d009b8e0 con 0x7fc8d8109900 2024-03-28T22:47:48.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.954+0000 7fc8df200640 1 -- 172.21.15.43:0/2827836376 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc8d802d170 con 0x7fc8d8109900 2024-03-28T22:47:48.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.954+0000 7fc8d4c00640 1 -- 172.21.15.43:0/2827836376 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v108) ==== 93+0+10 (secure 0 0 0) 0x7fc8d009b1e0 con 0x7fc8d8109900 2024-03-28T22:47:48.953 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-03-28T22:47:48.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.957+0000 7fc8df200640 1 -- 172.21.15.43:0/2827836376 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc8bc0a1f70 msgr2=0x7fc8bc0a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:48.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.957+0000 7fc8df200640 1 --2- 172.21.15.43:0/2827836376 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc8bc0a1f70 0x7fc8bc0a4430 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8011440 tx=0x7fc8c8005f90 comp rx=0 tx=0).stop 2024-03-28T22:47:48.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.957+0000 7fc8df200640 1 -- 172.21.15.43:0/2827836376 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8d8109900 msgr2=0x7fc8d819ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:48.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.957+0000 7fc8df200640 1 --2- 172.21.15.43:0/2827836376 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8d8109900 0x7fc8d819ea80 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0009fd0 tx=0x7fc8d000ef10 comp rx=0 tx=0).stop 2024-03-28T22:47:48.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.957+0000 7fc8df200640 1 -- 172.21.15.43:0/2827836376 shutdown_connections 2024-03-28T22:47:48.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.957+0000 7fc8df200640 1 --2- 172.21.15.43:0/2827836376 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fc8bc0a1f70 0x7fc8bc0a4430 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:48.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.957+0000 7fc8df200640 1 --2- 172.21.15.43:0/2827836376 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8d8109900 0x7fc8d819ea80 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:48.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.957+0000 7fc8df200640 1 --2- 172.21.15.43:0/2827836376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc8d81042b0 0x7fc8d819a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:48.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.957+0000 7fc8df200640 1 --2- 172.21.15.43:0/2827836376 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc8d81038f0 0x7fc8d819a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:48.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.957+0000 7fc8df200640 1 -- 172.21.15.43:0/2827836376 >> 172.21.15.43:0/2827836376 conn(0x7fc8d80ff3a0 msgr2=0x7fc8d80ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:48.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.957+0000 7fc8df200640 1 -- 172.21.15.43:0/2827836376 shutdown_connections 2024-03-28T22:47:48.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:48.957+0000 7fc8df200640 1 -- 172.21.15.43:0/2827836376 wait complete. 2024-03-28T22:47:48.975 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T22:47:49.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:48 smithi082 ceph-mon[36471]: pgmap v332: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:47:49.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:48 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1397241373' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:47:49.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.208+0000 7f0cb1a00640 1 -- 172.21.15.43:0/357199557 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0cac1015c0 msgr2=0x7f0cac101a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:49.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.208+0000 7f0cb1a00640 1 --2- 172.21.15.43:0/357199557 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0cac1015c0 0x7f0cac101a40 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f0c98009860 tx=0x7f0c9802bb70 comp rx=0 tx=0).stop 2024-03-28T22:47:49.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.208+0000 7f0cb1a00640 1 -- 172.21.15.43:0/357199557 shutdown_connections 2024-03-28T22:47:49.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.208+0000 7f0cb1a00640 1 --2- 172.21.15.43:0/357199557 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0cac1015c0 0x7f0cac101a40 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:49.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.208+0000 7f0cb1a00640 1 --2- 172.21.15.43:0/357199557 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0cac101f80 0x7f0cac10e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:49.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.208+0000 7f0cb1a00640 1 --2- 172.21.15.43:0/357199557 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cac107660 0x7f0cac107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:49.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.208+0000 7f0cb1a00640 1 -- 172.21.15.43:0/357199557 >> 172.21.15.43:0/357199557 conn(0x7f0cac0fd0f0 msgr2=0x7f0cac0ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:49.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.209+0000 7f0cb1a00640 1 -- 172.21.15.43:0/357199557 shutdown_connections 2024-03-28T22:47:49.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.209+0000 7f0cb1a00640 1 -- 172.21.15.43:0/357199557 wait complete. 2024-03-28T22:47:49.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.210+0000 7f0cb1a00640 1 Processor -- start 2024-03-28T22:47:49.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.210+0000 7f0cb1a00640 1 -- start start 2024-03-28T22:47:49.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.211+0000 7f0cb1a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0cac1015c0 0x7f0cac197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:49.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.211+0000 7f0cb1a00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cac101f80 0x7f0cac198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:49.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.211+0000 7f0cb1a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0cac107660 0x7f0cac19c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:49.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.211+0000 7f0cb1a00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0cac1119a0 con 0x7f0cac1015c0 2024-03-28T22:47:49.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.211+0000 7f0cb1a00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f0cac111820 con 0x7f0cac107660 2024-03-28T22:47:49.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.211+0000 7f0cb1a00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f0cac111b20 con 0x7f0cac101f80 2024-03-28T22:47:49.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.211+0000 7f0cb1000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0cac107660 0x7f0cac19c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:49.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.211+0000 7f0cab400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cac101f80 0x7f0cac198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:49.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.211+0000 7f0cabe00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0cac1015c0 0x7f0cac197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:49.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.211+0000 7f0cb1000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0cac107660 0x7f0cac19c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:59176/0 (socket says 172.21.15.43:59176) 2024-03-28T22:47:49.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.211+0000 7f0cb1000640 1 -- 172.21.15.43:0/2288510804 learned_addr learned my addr 172.21.15.43:0/2288510804 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:49.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.212+0000 7f0cab400640 1 -- 172.21.15.43:0/2288510804 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0cac107660 msgr2=0x7f0cac19c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:49.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.212+0000 7f0cab400640 1 --2- 172.21.15.43:0/2288510804 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0cac107660 0x7f0cac19c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:49.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.212+0000 7f0cab400640 1 -- 172.21.15.43:0/2288510804 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0cac1015c0 msgr2=0x7f0cac197de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:49.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.212+0000 7f0cab400640 1 --2- 172.21.15.43:0/2288510804 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0cac1015c0 0x7f0cac197de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:49.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.212+0000 7f0cab400640 1 -- 172.21.15.43:0/2288510804 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cac19cdd0 con 0x7f0cac101f80 2024-03-28T22:47:49.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.212+0000 7f0cb1000640 1 --2- 172.21.15.43:0/2288510804 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0cac107660 0x7f0cac19c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:49.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.212+0000 7f0cab400640 1 --2- 172.21.15.43:0/2288510804 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cac101f80 0x7f0cac198320 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f0c9802c0b0 tx=0x7f0c980045b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:49.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.212+0000 7f0ca8c00640 1 -- 172.21.15.43:0/2288510804 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c98041020 con 0x7f0cac101f80 2024-03-28T22:47:49.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.212+0000 7f0ca8c00640 1 -- 172.21.15.43:0/2288510804 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c9802c9f0 con 0x7f0cac101f80 2024-03-28T22:47:49.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.212+0000 7f0ca8c00640 1 -- 172.21.15.43:0/2288510804 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c98039040 con 0x7f0cac101f80 2024-03-28T22:47:49.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.212+0000 7f0cb1a00640 1 -- 172.21.15.43:0/2288510804 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cac19d060 con 0x7f0cac101f80 2024-03-28T22:47:49.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.213+0000 7f0cb1a00640 1 -- 172.21.15.43:0/2288510804 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cac1a4950 con 0x7f0cac101f80 2024-03-28T22:47:49.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.214+0000 7f0cb1a00640 1 -- 172.21.15.43:0/2288510804 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cac068460 con 0x7f0cac101f80 2024-03-28T22:47:49.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.215+0000 7f0ca8c00640 1 -- 172.21.15.43:0/2288510804 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f0c98044020 con 0x7f0cac101f80 2024-03-28T22:47:49.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.217+0000 7f0ca8c00640 1 --2- 172.21.15.43:0/2288510804 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0c900a1f70 0x7f0c900a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:49.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.217+0000 7f0ca8c00640 1 -- 172.21.15.43:0/2288510804 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9539+0+0 (secure 0 0 0) 0x7f0c980efb30 con 0x7f0cac101f80 2024-03-28T22:47:49.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.217+0000 7f0cabe00640 1 --2- 172.21.15.43:0/2288510804 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0c900a1f70 0x7f0c900a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:49.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.218+0000 7f0cabe00640 1 --2- 172.21.15.43:0/2288510804 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0c900a1f70 0x7f0c900a4430 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0cac102700 tx=0x7f0c9c009260 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:49.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.221+0000 7f0ca8c00640 1 -- 172.21.15.43:0/2288510804 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0c9804b020 con 0x7f0cac101f80 2024-03-28T22:47:49.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:48 smithi043 ceph-mon[31823]: pgmap v332: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:47:49.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:48 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1397241373' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:47:49.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:48 smithi172 ceph-mon[36566]: pgmap v332: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:47:49.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:48 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1397241373' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:47:49.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.422+0000 7f0cb1a00640 1 -- 172.21.15.43:0/2288510804 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0cac1095a0 con 0x7f0cac101f80 2024-03-28T22:47:49.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.423+0000 7f0ca8c00640 1 -- 172.21.15.43:0/2288510804 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v108) ==== 104+0+11 (secure 0 0 0) 0x7f0c980bc8d0 con 0x7f0cac101f80 2024-03-28T22:47:49.422 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-03-28T22:47:49.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.426+0000 7f0cb1a00640 1 -- 172.21.15.43:0/2288510804 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0c900a1f70 msgr2=0x7f0c900a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:49.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.426+0000 7f0cb1a00640 1 --2- 172.21.15.43:0/2288510804 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0c900a1f70 0x7f0c900a4430 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0cac102700 tx=0x7f0c9c009260 comp rx=0 tx=0).stop 2024-03-28T22:47:49.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.426+0000 7f0cb1a00640 1 -- 172.21.15.43:0/2288510804 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cac101f80 msgr2=0x7f0cac198320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:49.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.426+0000 7f0cb1a00640 1 --2- 172.21.15.43:0/2288510804 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cac101f80 0x7f0cac198320 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f0c9802c0b0 tx=0x7f0c980045b0 comp rx=0 tx=0).stop 2024-03-28T22:47:49.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.426+0000 7f0cb1a00640 1 -- 172.21.15.43:0/2288510804 shutdown_connections 2024-03-28T22:47:49.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.426+0000 7f0cb1a00640 1 --2- 172.21.15.43:0/2288510804 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0c900a1f70 0x7f0c900a4430 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:49.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.426+0000 7f0cb1a00640 1 --2- 172.21.15.43:0/2288510804 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0cac107660 0x7f0cac19c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:49.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.426+0000 7f0cb1a00640 1 --2- 172.21.15.43:0/2288510804 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cac101f80 0x7f0cac198320 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:49.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.427+0000 7f0cb1a00640 1 --2- 172.21.15.43:0/2288510804 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0cac1015c0 0x7f0cac197de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:49.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.427+0000 7f0cb1a00640 1 -- 172.21.15.43:0/2288510804 >> 172.21.15.43:0/2288510804 conn(0x7f0cac0fd0f0 msgr2=0x7f0cac0fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:49.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.427+0000 7f0cb1a00640 1 -- 172.21.15.43:0/2288510804 shutdown_connections 2024-03-28T22:47:49.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.427+0000 7f0cb1a00640 1 -- 172.21.15.43:0/2288510804 wait complete. 2024-03-28T22:47:49.445 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T22:47:49.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.675+0000 7f1a86600640 1 -- 172.21.15.43:0/105263283 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1a801037d0 msgr2=0x7f1a80103c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:49.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.675+0000 7f1a86600640 1 --2- 172.21.15.43:0/105263283 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1a801037d0 0x7f1a80103c50 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c009930 tx=0x7f1a6c02bc20 comp rx=0 tx=0).stop 2024-03-28T22:47:49.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.675+0000 7f1a86600640 1 -- 172.21.15.43:0/105263283 shutdown_connections 2024-03-28T22:47:49.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.675+0000 7f1a86600640 1 --2- 172.21.15.43:0/105263283 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1a80104190 0x7f1a801106a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:49.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.675+0000 7f1a86600640 1 --2- 172.21.15.43:0/105263283 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1a801037d0 0x7f1a80103c50 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:49.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.675+0000 7f1a86600640 1 --2- 172.21.15.43:0/105263283 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1a801097e0 0x7f1a80109bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:49.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.675+0000 7f1a86600640 1 -- 172.21.15.43:0/105263283 >> 172.21.15.43:0/105263283 conn(0x7f1a800ff260 msgr2=0x7f1a80101680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:49.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.675+0000 7f1a86600640 1 -- 172.21.15.43:0/105263283 shutdown_connections 2024-03-28T22:47:49.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.675+0000 7f1a86600640 1 -- 172.21.15.43:0/105263283 wait complete. 2024-03-28T22:47:49.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.676+0000 7f1a86600640 1 Processor -- start 2024-03-28T22:47:49.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.676+0000 7f1a86600640 1 -- start start 2024-03-28T22:47:49.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.676+0000 7f1a86600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1a801037d0 0x7f1a8019a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:49.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.676+0000 7f1a86600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1a80104190 0x7f1a8019a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:49.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.676+0000 7f1a86600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1a801097e0 0x7f1a8019ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:49.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.676+0000 7f1a86600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1a80113bf0 con 0x7f1a801037d0 2024-03-28T22:47:49.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.676+0000 7f1a86600640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1a80113a70 con 0x7f1a80104190 2024-03-28T22:47:49.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.676+0000 7f1a86600640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f1a80113d70 con 0x7f1a801097e0 2024-03-28T22:47:49.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.677+0000 7f1a85200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1a801037d0 0x7f1a8019a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:49.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.677+0000 7f1a85c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1a801097e0 0x7f1a8019ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:49.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.677+0000 7f1a7fe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1a80104190 0x7f1a8019a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:49.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.677+0000 7f1a85200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1a801037d0 0x7f1a8019a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:37536/0 (socket says 172.21.15.43:37536) 2024-03-28T22:47:49.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.677+0000 7f1a85c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1a801097e0 0x7f1a8019ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:34552/0 (socket says 172.21.15.43:34552) 2024-03-28T22:47:49.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.677+0000 7f1a85200640 1 -- 172.21.15.43:0/2289376627 learned_addr learned my addr 172.21.15.43:0/2289376627 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:49.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.677+0000 7f1a7fe00640 1 -- 172.21.15.43:0/2289376627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1a801097e0 msgr2=0x7f1a8019ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:49.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.677+0000 7f1a7fe00640 1 --2- 172.21.15.43:0/2289376627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1a801097e0 0x7f1a8019ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:49.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.677+0000 7f1a7fe00640 1 -- 172.21.15.43:0/2289376627 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1a801037d0 msgr2=0x7f1a8019a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:49.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.677+0000 7f1a7fe00640 1 --2- 172.21.15.43:0/2289376627 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1a801037d0 0x7f1a8019a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:49.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.677+0000 7f1a7fe00640 1 -- 172.21.15.43:0/2289376627 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a8019f3c0 con 0x7f1a80104190 2024-03-28T22:47:49.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.677+0000 7f1a85200640 1 --2- 172.21.15.43:0/2289376627 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1a801037d0 0x7f1a8019a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:49.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.678+0000 7f1a7fe00640 1 --2- 172.21.15.43:0/2289376627 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1a80104190 0x7f1a8019a910 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c009230 tx=0x7f1a6c02cb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:49.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.678+0000 7f1a7d600640 1 -- 172.21.15.43:0/2289376627 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c041070 con 0x7f1a80104190 2024-03-28T22:47:49.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.678+0000 7f1a7d600640 1 -- 172.21.15.43:0/2289376627 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a6c004450 con 0x7f1a80104190 2024-03-28T22:47:49.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.678+0000 7f1a7d600640 1 -- 172.21.15.43:0/2289376627 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c003cd0 con 0x7f1a80104190 2024-03-28T22:47:49.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.678+0000 7f1a86600640 1 -- 172.21.15.43:0/2289376627 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a8019f650 con 0x7f1a80104190 2024-03-28T22:47:49.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.678+0000 7f1a86600640 1 -- 172.21.15.43:0/2289376627 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a801a6f40 con 0x7f1a80104190 2024-03-28T22:47:49.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.679+0000 7f1a86600640 1 -- 172.21.15.43:0/2289376627 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a80104f10 con 0x7f1a80104190 2024-03-28T22:47:49.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.681+0000 7f1a7d600640 1 -- 172.21.15.43:0/2289376627 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f1a6c044070 con 0x7f1a80104190 2024-03-28T22:47:49.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.682+0000 7f1a7d600640 1 --2- 172.21.15.43:0/2289376627 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1a680a1ef0 0x7f1a680a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:49.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.683+0000 7f1a7d600640 1 -- 172.21.15.43:0/2289376627 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9539+0+0 (secure 0 0 0) 0x7f1a6c0efdf0 con 0x7f1a80104190 2024-03-28T22:47:49.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.683+0000 7f1a85200640 1 --2- 172.21.15.43:0/2289376627 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1a680a1ef0 0x7f1a680a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:49.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.683+0000 7f1a85200640 1 --2- 172.21.15.43:0/2289376627 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1a680a1ef0 0x7f1a680a43b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1a74011440 tx=0x7f1a74005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:49.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.686+0000 7f1a7d600640 1 -- 172.21.15.43:0/2289376627 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1a6c0bc540 con 0x7f1a80104190 2024-03-28T22:47:49.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.889+0000 7f1a86600640 1 -- 172.21.15.43:0/2289376627 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1a8010bd40 con 0x7f1a80104190 2024-03-28T22:47:49.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.890+0000 7f1a7d600640 1 -- 172.21.15.43:0/2289376627 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v108) ==== 100+0+11 (secure 0 0 0) 0x7f1a6c0c05e0 con 0x7f1a80104190 2024-03-28T22:47:49.888 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-03-28T22:47:49.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.893+0000 7f1a86600640 1 -- 172.21.15.43:0/2289376627 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1a680a1ef0 msgr2=0x7f1a680a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:49.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.893+0000 7f1a86600640 1 --2- 172.21.15.43:0/2289376627 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1a680a1ef0 0x7f1a680a43b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1a74011440 tx=0x7f1a74005f90 comp rx=0 tx=0).stop 2024-03-28T22:47:49.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.893+0000 7f1a86600640 1 -- 172.21.15.43:0/2289376627 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1a80104190 msgr2=0x7f1a8019a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:49.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.893+0000 7f1a86600640 1 --2- 172.21.15.43:0/2289376627 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1a80104190 0x7f1a8019a910 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c009230 tx=0x7f1a6c02cb20 comp rx=0 tx=0).stop 2024-03-28T22:47:49.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.893+0000 7f1a86600640 1 -- 172.21.15.43:0/2289376627 shutdown_connections 2024-03-28T22:47:49.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.893+0000 7f1a86600640 1 --2- 172.21.15.43:0/2289376627 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1a680a1ef0 0x7f1a680a43b0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:49.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.893+0000 7f1a86600640 1 --2- 172.21.15.43:0/2289376627 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1a80104190 0x7f1a8019a910 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:49.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.893+0000 7f1a86600640 1 --2- 172.21.15.43:0/2289376627 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1a801097e0 0x7f1a8019ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:49.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.893+0000 7f1a86600640 1 --2- 172.21.15.43:0/2289376627 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1a801037d0 0x7f1a8019a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:49.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.893+0000 7f1a86600640 1 -- 172.21.15.43:0/2289376627 >> 172.21.15.43:0/2289376627 conn(0x7f1a800ff260 msgr2=0x7f1a800ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:49.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.893+0000 7f1a86600640 1 -- 172.21.15.43:0/2289376627 shutdown_connections 2024-03-28T22:47:49.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:49.893+0000 7f1a86600640 1 -- 172.21.15.43:0/2289376627 wait complete. 2024-03-28T22:47:49.912 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T22:47:50.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.147+0000 7f8620000640 1 -- 172.21.15.43:0/844950865 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8618109900 msgr2=0x7f8618109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:50.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.147+0000 7f8620000640 1 --2- 172.21.15.43:0/844950865 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8618109900 0x7f8618109ce0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f860c009930 tx=0x7f860c02bc30 comp rx=0 tx=0).stop 2024-03-28T22:47:50.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.147+0000 7f8620000640 1 -- 172.21.15.43:0/844950865 shutdown_connections 2024-03-28T22:47:50.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.147+0000 7f8620000640 1 --2- 172.21.15.43:0/844950865 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86181042b0 0x7f86181107c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:50.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.147+0000 7f8620000640 1 --2- 172.21.15.43:0/844950865 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86181038f0 0x7f8618103d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:50.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.147+0000 7f8620000640 1 --2- 172.21.15.43:0/844950865 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8618109900 0x7f8618109ce0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:50.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.147+0000 7f8620000640 1 -- 172.21.15.43:0/844950865 >> 172.21.15.43:0/844950865 conn(0x7f86180ff3a0 msgr2=0x7f86181017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:50.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.147+0000 7f8620000640 1 -- 172.21.15.43:0/844950865 shutdown_connections 2024-03-28T22:47:50.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.147+0000 7f8620000640 1 -- 172.21.15.43:0/844950865 wait complete. 2024-03-28T22:47:50.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.148+0000 7f8620000640 1 Processor -- start 2024-03-28T22:47:50.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.148+0000 7f8620000640 1 -- start start 2024-03-28T22:47:50.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.149+0000 7f8620000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86181038f0 0x7f861819a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:50.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.149+0000 7f8620000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86181042b0 0x7f861819a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:50.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.149+0000 7f8620000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8618109900 0x7f861819eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:50.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.149+0000 7f8620000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8618113d60 con 0x7f8618109900 2024-03-28T22:47:50.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.149+0000 7f8620000640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8618113be0 con 0x7f86181042b0 2024-03-28T22:47:50.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.149+0000 7f8620000640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f8618113ee0 con 0x7f86181038f0 2024-03-28T22:47:50.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.149+0000 7f861d600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86181038f0 0x7f861819a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:50.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.149+0000 7f861cc00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86181042b0 0x7f861819a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:50.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.149+0000 7f861d600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86181038f0 0x7f861819a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:34564/0 (socket says 172.21.15.43:34564) 2024-03-28T22:47:50.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.149+0000 7f861d600640 1 -- 172.21.15.43:0/2214579207 learned_addr learned my addr 172.21.15.43:0/2214579207 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:50.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.149+0000 7f861e000640 1 --2- 172.21.15.43:0/2214579207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8618109900 0x7f861819eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:50.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.150+0000 7f861d600640 1 -- 172.21.15.43:0/2214579207 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86181042b0 msgr2=0x7f861819a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:50.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.150+0000 7f861d600640 1 --2- 172.21.15.43:0/2214579207 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86181042b0 0x7f861819a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:50.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.150+0000 7f861d600640 1 -- 172.21.15.43:0/2214579207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8618109900 msgr2=0x7f861819eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:50.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.150+0000 7f861d600640 1 --2- 172.21.15.43:0/2214579207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8618109900 0x7f861819eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:50.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.150+0000 7f861d600640 1 -- 172.21.15.43:0/2214579207 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f861819f230 con 0x7f86181038f0 2024-03-28T22:47:50.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.150+0000 7f861d600640 1 --2- 172.21.15.43:0/2214579207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86181038f0 0x7f861819a240 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f860c009900 tx=0x7f860c004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:50.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.150+0000 7f8616000640 1 -- 172.21.15.43:0/2214579207 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860c043020 con 0x7f86181038f0 2024-03-28T22:47:50.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.150+0000 7f8616000640 1 -- 172.21.15.43:0/2214579207 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f860c0054e0 con 0x7f86181038f0 2024-03-28T22:47:50.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.150+0000 7f8616000640 1 -- 172.21.15.43:0/2214579207 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860c003b80 con 0x7f86181038f0 2024-03-28T22:47:50.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.151+0000 7f8620000640 1 -- 172.21.15.43:0/2214579207 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f861819f4c0 con 0x7f86181038f0 2024-03-28T22:47:50.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.151+0000 7f8620000640 1 -- 172.21.15.43:0/2214579207 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86181a6db0 con 0x7f86181038f0 2024-03-28T22:47:50.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.152+0000 7f8620000640 1 -- 172.21.15.43:0/2214579207 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8618068990 con 0x7f86181038f0 2024-03-28T22:47:50.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.154+0000 7f8616000640 1 -- 172.21.15.43:0/2214579207 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f860c046020 con 0x7f86181038f0 2024-03-28T22:47:50.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.155+0000 7f8616000640 1 --2- 172.21.15.43:0/2214579207 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f86000a1f70 0x7f86000a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:50.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.155+0000 7f8616000640 1 -- 172.21.15.43:0/2214579207 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9539+0+0 (secure 0 0 0) 0x7f860c0f05b0 con 0x7f86181038f0 2024-03-28T22:47:50.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.155+0000 7f861cc00640 1 --2- 172.21.15.43:0/2214579207 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f86000a1f70 0x7f86000a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:50.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.156+0000 7f861cc00640 1 --2- 172.21.15.43:0/2214579207 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f86000a1f70 0x7f86000a4430 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8618101a00 tx=0x7f860400b410 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:50.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.159+0000 7f8616000640 1 -- 172.21.15.43:0/2214579207 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f860c04d020 con 0x7f86181038f0 2024-03-28T22:47:50.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:49 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2827836376' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:47:50.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:49 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2288510804' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:47:50.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:49 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2289376627' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:47:50.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:49 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2827836376' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:47:50.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:49 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2288510804' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:47:50.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:49 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2289376627' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:47:50.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.362+0000 7f8620000640 1 -- 172.21.15.43:0/2214579207 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f861810be60 con 0x7f86181038f0 2024-03-28T22:47:50.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.363+0000 7f8616000640 1 -- 172.21.15.43:0/2214579207 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v108) ==== 103+0+11 (secure 0 0 0) 0x7f860c0bd350 con 0x7f86181038f0 2024-03-28T22:47:50.362 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-03-28T22:47:50.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.366+0000 7f8620000640 1 -- 172.21.15.43:0/2214579207 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f86000a1f70 msgr2=0x7f86000a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:50.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.366+0000 7f8620000640 1 --2- 172.21.15.43:0/2214579207 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f86000a1f70 0x7f86000a4430 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8618101a00 tx=0x7f860400b410 comp rx=0 tx=0).stop 2024-03-28T22:47:50.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.366+0000 7f8620000640 1 -- 172.21.15.43:0/2214579207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86181038f0 msgr2=0x7f861819a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:50.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.366+0000 7f8620000640 1 --2- 172.21.15.43:0/2214579207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86181038f0 0x7f861819a240 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f860c009900 tx=0x7f860c004360 comp rx=0 tx=0).stop 2024-03-28T22:47:50.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.366+0000 7f8620000640 1 -- 172.21.15.43:0/2214579207 shutdown_connections 2024-03-28T22:47:50.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.366+0000 7f8620000640 1 --2- 172.21.15.43:0/2214579207 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f86000a1f70 0x7f86000a4430 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:50.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.366+0000 7f8620000640 1 --2- 172.21.15.43:0/2214579207 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f86181042b0 0x7f861819a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:50.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.366+0000 7f8620000640 1 --2- 172.21.15.43:0/2214579207 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8618109900 0x7f861819eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:50.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.366+0000 7f8620000640 1 --2- 172.21.15.43:0/2214579207 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f86181038f0 0x7f861819a240 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:50.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.366+0000 7f8620000640 1 -- 172.21.15.43:0/2214579207 >> 172.21.15.43:0/2214579207 conn(0x7f86180ff3a0 msgr2=0x7f86180ffb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:50.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.366+0000 7f8620000640 1 -- 172.21.15.43:0/2214579207 shutdown_connections 2024-03-28T22:47:50.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.366+0000 7f8620000640 1 -- 172.21.15.43:0/2214579207 wait complete. 2024-03-28T22:47:50.384 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-28T22:47:50.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:49 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2827836376' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:47:50.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:49 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2288510804' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:47:50.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:49 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2289376627' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:47:50.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.609+0000 7f0a3a800640 1 -- 172.21.15.43:0/3715061518 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a341042c0 msgr2=0x7f0a341107d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:50.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.609+0000 7f0a3a800640 1 --2- 172.21.15.43:0/3715061518 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a341042c0 0x7f0a341107d0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c009be0 tx=0x7f0a2c02bef0 comp rx=0 tx=0).stop 2024-03-28T22:47:50.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.609+0000 7f0a3a800640 1 -- 172.21.15.43:0/3715061518 shutdown_connections 2024-03-28T22:47:50.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.609+0000 7f0a3a800640 1 --2- 172.21.15.43:0/3715061518 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a341042c0 0x7f0a341107d0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:50.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.609+0000 7f0a3a800640 1 --2- 172.21.15.43:0/3715061518 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a34103900 0x7f0a34103d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:50.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.609+0000 7f0a3a800640 1 --2- 172.21.15.43:0/3715061518 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0a34109910 0x7f0a34109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:50.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.609+0000 7f0a3a800640 1 -- 172.21.15.43:0/3715061518 >> 172.21.15.43:0/3715061518 conn(0x7f0a340ff390 msgr2=0x7f0a341017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:50.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.609+0000 7f0a3a800640 1 -- 172.21.15.43:0/3715061518 shutdown_connections 2024-03-28T22:47:50.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.609+0000 7f0a3a800640 1 -- 172.21.15.43:0/3715061518 wait complete. 2024-03-28T22:47:50.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.610+0000 7f0a3a800640 1 Processor -- start 2024-03-28T22:47:50.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.610+0000 7f0a3a800640 1 -- start start 2024-03-28T22:47:50.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.611+0000 7f0a3a800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a34103900 0x7f0a34075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:50.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.611+0000 7f0a3a800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0a341042c0 0x7f0a340761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:50.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.611+0000 7f0a3a800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a34109910 0x7f0a340766e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:50.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.611+0000 7f0a3a800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0a34113d70 con 0x7f0a341042c0 2024-03-28T22:47:50.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.611+0000 7f0a3a800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f0a34113bf0 con 0x7f0a34109910 2024-03-28T22:47:50.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.611+0000 7f0a3a800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f0a34113ef0 con 0x7f0a34103900 2024-03-28T22:47:50.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.611+0000 7f0a33400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a34103900 0x7f0a34075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:50.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.611+0000 7f0a32a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0a341042c0 0x7f0a340761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:50.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.611+0000 7f0a33e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a34109910 0x7f0a340766e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:50.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.611+0000 7f0a33400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a34103900 0x7f0a34075c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:34568/0 (socket says 172.21.15.43:34568) 2024-03-28T22:47:50.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.611+0000 7f0a33400640 1 -- 172.21.15.43:0/559071493 learned_addr learned my addr 172.21.15.43:0/559071493 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:47:50.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.612+0000 7f0a33e00640 1 -- 172.21.15.43:0/559071493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a34103900 msgr2=0x7f0a34075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:50.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.612+0000 7f0a33e00640 1 --2- 172.21.15.43:0/559071493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a34103900 0x7f0a34075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:50.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.612+0000 7f0a33e00640 1 -- 172.21.15.43:0/559071493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0a341042c0 msgr2=0x7f0a340761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:50.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.612+0000 7f0a33e00640 1 --2- 172.21.15.43:0/559071493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0a341042c0 0x7f0a340761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:50.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.612+0000 7f0a33e00640 1 -- 172.21.15.43:0/559071493 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a341aa840 con 0x7f0a34109910 2024-03-28T22:47:50.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.612+0000 7f0a33400640 1 --2- 172.21.15.43:0/559071493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a34103900 0x7f0a34075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:47:50.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.612+0000 7f0a33e00640 1 --2- 172.21.15.43:0/559071493 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a34109910 0x7f0a340766e0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c02c430 tx=0x7f0a2c0336b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:50.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.612+0000 7f0a2be00640 1 -- 172.21.15.43:0/559071493 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c043020 con 0x7f0a34109910 2024-03-28T22:47:50.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.612+0000 7f0a2be00640 1 -- 172.21.15.43:0/559071493 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a2c003e50 con 0x7f0a34109910 2024-03-28T22:47:50.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.612+0000 7f0a2be00640 1 -- 172.21.15.43:0/559071493 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c03e6b0 con 0x7f0a34109910 2024-03-28T22:47:50.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.612+0000 7f0a3a800640 1 -- 172.21.15.43:0/559071493 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a341aaad0 con 0x7f0a34109910 2024-03-28T22:47:50.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.613+0000 7f0a3a800640 1 -- 172.21.15.43:0/559071493 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a341aafd0 con 0x7f0a34109910 2024-03-28T22:47:50.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.614+0000 7f0a3a800640 1 -- 172.21.15.43:0/559071493 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a34105040 con 0x7f0a34109910 2024-03-28T22:47:50.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.616+0000 7f0a2be00640 1 -- 172.21.15.43:0/559071493 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f0a2c046020 con 0x7f0a34109910 2024-03-28T22:47:50.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.617+0000 7f0a2be00640 1 --2- 172.21.15.43:0/559071493 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0a180a1f70 0x7f0a180a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:47:50.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.617+0000 7f0a2be00640 1 -- 172.21.15.43:0/559071493 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9539+0+0 (secure 0 0 0) 0x7f0a2c0f0ad0 con 0x7f0a34109910 2024-03-28T22:47:50.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.617+0000 7f0a33400640 1 --2- 172.21.15.43:0/559071493 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0a180a1f70 0x7f0a180a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:47:50.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.618+0000 7f0a33400640 1 --2- 172.21.15.43:0/559071493 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0a180a1f70 0x7f0a180a4430 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0a24011440 tx=0x7f0a24005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:47:50.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.619+0000 7f0a2be00640 1 -- 172.21.15.43:0/559071493 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0a2c04d030 con 0x7f0a34109910 2024-03-28T22:47:50.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.874+0000 7f0a3a800640 1 -- 172.21.15.43:0/559071493 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f0a34068540 con 0x7f0a34109910 2024-03-28T22:47:50.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.890+0000 7f0a2be00640 1 -- 172.21.15.43:0/559071493 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v28) ==== 81+0+736928 (secure 0 0 0) 0x7f0a2c0bd870 con 0x7f0a34109910 2024-03-28T22:47:50.900 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:47:50.900 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:47:50.901 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 28, 2024-03-28T22:47:50.901 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:50.901 INFO:teuthology.orchestra.run.smithi043.stdout: "active_gid": 25663, 2024-03-28T22:47:50.901 INFO:teuthology.orchestra.run.smithi043.stdout: "active_name": "x", 2024-03-28T22:47:50.901 INFO:teuthology.orchestra.run.smithi043.stdout: "active_addrs": { 2024-03-28T22:47:50.901 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-28T22:47:50.901 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:50.901 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-28T22:47:50.902 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6800", 2024-03-28T22:47:50.902 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1685312925 2024-03-28T22:47:50.902 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.902 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:50.902 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-03-28T22:47:50.902 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6801", 2024-03-28T22:47:50.902 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1685312925 2024-03-28T22:47:50.902 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:50.902 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-28T22:47:50.903 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.903 INFO:teuthology.orchestra.run.smithi043.stdout: "active_addr": "172.21.15.43:6801/1685312925", 2024-03-28T22:47:50.903 INFO:teuthology.orchestra.run.smithi043.stdout: "active_change": "2024-03-28T22:37:11.309417+0000", 2024-03-28T22:47:50.903 INFO:teuthology.orchestra.run.smithi043.stdout: "active_mgr_features": 4540701547738038271, 2024-03-28T22:47:50.903 INFO:teuthology.orchestra.run.smithi043.stdout: "available": true, 2024-03-28T22:47:50.903 INFO:teuthology.orchestra.run.smithi043.stdout: "standbys": [ 2024-03-28T22:47:50.903 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:50.903 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 25655, 2024-03-28T22:47:50.903 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "y", 2024-03-28T22:47:50.904 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_features": 4540701547738038271, 2024-03-28T22:47:50.904 INFO:teuthology.orchestra.run.smithi043.stdout: "available_modules": [ 2024-03-28T22:47:50.904 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:50.904 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "alerts", 2024-03-28T22:47:50.904 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:50.904 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:50.904 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:50.904 INFO:teuthology.orchestra.run.smithi043.stdout: "interval": { 2024-03-28T22:47:50.905 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "interval", 2024-03-28T22:47:50.905 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:50.905 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.905 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:50.905 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "60", 2024-03-28T22:47:50.905 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.905 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.905 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.905 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "How frequently to reexamine health status", 2024-03-28T22:47:50.906 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.906 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.906 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.906 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.906 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:50.906 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:50.906 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:50.906 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.907 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:50.907 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:50.907 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.907 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.907 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:50.907 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:50.907 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:50.907 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:50.907 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:50.908 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:50.908 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:50.908 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:50.908 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:50.908 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.908 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.908 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.908 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.908 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:50.908 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:50.909 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:50.909 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.909 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:50.909 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:50.909 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.909 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.909 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.909 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:50.909 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.910 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.910 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.910 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.910 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:50.910 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:50.910 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:50.910 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.910 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:50.910 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:50.911 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.911 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.911 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:50.911 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:50.911 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:50.911 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:50.911 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:50.911 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:50.911 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:50.912 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:50.912 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:50.912 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.912 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.912 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.912 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.912 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:50.912 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:50.912 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:50.913 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.913 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:50.913 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:50.913 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.913 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.913 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.913 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:50.913 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.913 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.914 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.914 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.914 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_destination": { 2024-03-28T22:47:50.914 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_destination", 2024-03-28T22:47:50.914 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:50.914 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.914 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:50.914 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:50.915 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.915 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.915 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.915 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Email address to send alerts to", 2024-03-28T22:47:50.915 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.915 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.915 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.915 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.915 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_from_name": { 2024-03-28T22:47:50.916 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_from_name", 2024-03-28T22:47:50.916 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:50.916 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.916 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:50.916 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "Ceph", 2024-03-28T22:47:50.916 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.916 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.916 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.916 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Email From: name", 2024-03-28T22:47:50.917 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.917 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.917 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.917 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.917 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_host": { 2024-03-28T22:47:50.917 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_host", 2024-03-28T22:47:50.917 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:50.917 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.917 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:50.917 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:50.918 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.918 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.918 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.918 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "SMTP server", 2024-03-28T22:47:50.918 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.918 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.918 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.918 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.919 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_password": { 2024-03-28T22:47:50.919 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_password", 2024-03-28T22:47:50.919 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:50.919 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.919 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:50.919 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:50.919 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.919 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.919 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.920 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Password to authenticate with", 2024-03-28T22:47:50.920 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.920 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.920 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.920 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.920 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_port": { 2024-03-28T22:47:50.920 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_port", 2024-03-28T22:47:50.920 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:50.920 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.921 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:50.921 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "465", 2024-03-28T22:47:50.921 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.921 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.921 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.921 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "SMTP port", 2024-03-28T22:47:50.921 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.921 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.921 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.922 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.922 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_sender": { 2024-03-28T22:47:50.922 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_sender", 2024-03-28T22:47:50.922 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:50.922 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.922 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:50.922 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:50.922 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.922 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.923 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.923 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "SMTP envelope sender", 2024-03-28T22:47:50.923 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.923 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.923 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.923 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.923 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_ssl": { 2024-03-28T22:47:50.924 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_ssl", 2024-03-28T22:47:50.924 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:50.924 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.924 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:50.924 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:50.924 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.924 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.924 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.924 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-28T22:47:50.924 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.925 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.925 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.925 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.925 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_user": { 2024-03-28T22:47:50.925 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_user", 2024-03-28T22:47:50.925 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:50.925 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.925 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:50.925 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:50.926 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.926 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.926 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.926 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "User to authenticate as", 2024-03-28T22:47:50.926 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.926 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.926 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.926 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:50.927 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:50.927 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.927 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:50.927 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "balancer", 2024-03-28T22:47:50.927 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:50.927 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:50.927 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:50.927 INFO:teuthology.orchestra.run.smithi043.stdout: "active": { 2024-03-28T22:47:50.927 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "active", 2024-03-28T22:47:50.928 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:50.928 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.928 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:50.928 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:50.928 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.928 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.928 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.928 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "automatically balance PGs across cluster", 2024-03-28T22:47:50.928 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.929 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.929 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.929 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.929 INFO:teuthology.orchestra.run.smithi043.stdout: "begin_time": { 2024-03-28T22:47:50.929 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "begin_time", 2024-03-28T22:47:50.929 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:50.929 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.929 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:50.930 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0000", 2024-03-28T22:47:50.930 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.930 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.930 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.930 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "beginning time of day to automatically balance", 2024-03-28T22:47:50.930 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T22:47:50.930 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.931 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.931 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.931 INFO:teuthology.orchestra.run.smithi043.stdout: "begin_weekday": { 2024-03-28T22:47:50.931 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "begin_weekday", 2024-03-28T22:47:50.931 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "uint", 2024-03-28T22:47:50.931 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.931 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:50.931 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-03-28T22:47:50.931 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "0", 2024-03-28T22:47:50.932 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "6", 2024-03-28T22:47:50.932 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.932 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-28T22:47:50.932 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T22:47:50.932 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.932 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.932 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.932 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_compat_max_iterations": { 2024-03-28T22:47:50.932 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "crush_compat_max_iterations", 2024-03-28T22:47:50.933 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "uint", 2024-03-28T22:47:50.933 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.933 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:50.933 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "25", 2024-03-28T22:47:50.933 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "1", 2024-03-28T22:47:50.933 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "250", 2024-03-28T22:47:50.933 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.933 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-28T22:47:50.933 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.933 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.934 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.934 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.934 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_compat_metrics": { 2024-03-28T22:47:50.934 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "crush_compat_metrics", 2024-03-28T22:47:50.934 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:50.934 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.934 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:50.934 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "pgs,objects,bytes", 2024-03-28T22:47:50.934 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.935 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.935 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.935 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-28T22:47:50.935 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-28T22:47:50.935 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.935 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.935 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.935 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_compat_step": { 2024-03-28T22:47:50.935 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "crush_compat_step", 2024-03-28T22:47:50.936 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-03-28T22:47:50.936 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.936 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:50.936 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0.5", 2024-03-28T22:47:50.936 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "0.001", 2024-03-28T22:47:50.936 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "0.999", 2024-03-28T22:47:50.936 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.936 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "aggressiveness of optimization", 2024-03-28T22:47:50.936 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-28T22:47:50.937 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.937 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.937 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.937 INFO:teuthology.orchestra.run.smithi043.stdout: "end_time": { 2024-03-28T22:47:50.937 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "end_time", 2024-03-28T22:47:50.937 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:50.937 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.937 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:50.937 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "2359", 2024-03-28T22:47:50.938 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.938 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.938 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.938 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "ending time of day to automatically balance", 2024-03-28T22:47:50.938 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T22:47:50.938 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.938 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.938 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.938 INFO:teuthology.orchestra.run.smithi043.stdout: "end_weekday": { 2024-03-28T22:47:50.939 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "end_weekday", 2024-03-28T22:47:50.939 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "uint", 2024-03-28T22:47:50.939 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.939 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:50.939 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-03-28T22:47:50.939 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "0", 2024-03-28T22:47:50.939 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "6", 2024-03-28T22:47:50.939 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.940 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-28T22:47:50.940 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T22:47:50.940 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.940 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.940 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.940 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:50.940 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:50.940 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:50.940 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.940 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:50.941 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:50.941 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.941 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.941 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:50.941 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:50.941 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:50.941 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:50.941 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:50.942 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:50.942 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:50.942 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:50.942 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:50.942 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.942 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.942 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.942 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.942 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:50.943 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:50.943 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:50.943 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.943 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:50.943 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:50.943 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.943 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.943 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.944 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:50.944 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.944 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.944 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.944 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.944 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:50.944 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:50.944 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:50.944 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.945 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:50.945 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:50.945 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.945 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.945 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:50.945 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:50.945 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:50.945 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:50.945 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:50.946 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:50.946 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:50.946 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:50.946 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:50.946 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.946 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.946 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.947 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:50.947 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:50.947 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:50.947 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.947 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:50.947 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:50.947 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.947 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.948 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.948 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:50.948 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.948 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.948 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.948 INFO:teuthology.orchestra.run.smithi043.stdout: "min_score": { 2024-03-28T22:47:50.948 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "min_score", 2024-03-28T22:47:50.948 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-03-28T22:47:50.949 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.949 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:50.949 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-03-28T22:47:50.949 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.949 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.949 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.949 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-28T22:47:50.949 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.949 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.949 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.950 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.950 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": { 2024-03-28T22:47:50.950 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mode", 2024-03-28T22:47:50.950 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:50.950 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.950 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:50.950 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "upmap", 2024-03-28T22:47:50.950 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.951 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.951 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:50.951 INFO:teuthology.orchestra.run.smithi043.stdout: "crush-compat", 2024-03-28T22:47:50.951 INFO:teuthology.orchestra.run.smithi043.stdout: "none", 2024-03-28T22:47:50.951 INFO:teuthology.orchestra.run.smithi043.stdout: "read", 2024-03-28T22:47:50.951 INFO:teuthology.orchestra.run.smithi043.stdout: "upmap", 2024-03-28T22:47:50.951 INFO:teuthology.orchestra.run.smithi043.stdout: "upmap-read" 2024-03-28T22:47:50.951 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:50.951 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Balancer mode", 2024-03-28T22:47:50.952 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.952 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.952 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.952 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_ids": { 2024-03-28T22:47:50.952 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "pool_ids", 2024-03-28T22:47:50.952 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:50.952 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.952 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:50.953 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:50.953 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.953 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.953 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.953 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-28T22:47:50.953 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.953 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.953 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.954 INFO:teuthology.orchestra.run.smithi043.stdout: "sleep_interval": { 2024-03-28T22:47:50.954 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sleep_interval", 2024-03-28T22:47:50.954 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:50.954 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.954 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:50.954 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "60", 2024-03-28T22:47:50.954 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.954 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.955 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.955 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-28T22:47:50.955 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.955 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.955 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.955 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.955 INFO:teuthology.orchestra.run.smithi043.stdout: "upmap_max_deviation": { 2024-03-28T22:47:50.955 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "upmap_max_deviation", 2024-03-28T22:47:50.955 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:50.956 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.956 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:50.956 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "5", 2024-03-28T22:47:50.956 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "1", 2024-03-28T22:47:50.956 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.956 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.956 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-28T22:47:50.956 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-28T22:47:50.956 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.957 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.957 INFO:teuthology.orchestra.run.smithi043.stdout: "upmap_max_optimizations": { 2024-03-28T22:47:50.957 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "upmap_max_optimizations", 2024-03-28T22:47:50.957 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "uint", 2024-03-28T22:47:50.957 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.957 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:50.957 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-03-28T22:47:50.957 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.958 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.958 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.958 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-28T22:47:50.958 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.958 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.958 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.958 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:50.958 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:50.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.959 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:50.959 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cephadm", 2024-03-28T22:47:50.959 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:50.959 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:50.959 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:50.959 INFO:teuthology.orchestra.run.smithi043.stdout: "agent_down_multiplier": { 2024-03-28T22:47:50.959 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "agent_down_multiplier", 2024-03-28T22:47:50.959 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-03-28T22:47:50.959 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.960 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:50.960 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "3.0", 2024-03-28T22:47:50.960 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.960 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.960 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.960 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-28T22:47:50.960 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.960 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.960 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.961 INFO:teuthology.orchestra.run.smithi043.stdout: "agent_refresh_rate": { 2024-03-28T22:47:50.961 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "agent_refresh_rate", 2024-03-28T22:47:50.961 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:50.961 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.961 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:50.961 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "20", 2024-03-28T22:47:50.961 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.962 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.962 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.962 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-28T22:47:50.962 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.962 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.962 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.962 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.962 INFO:teuthology.orchestra.run.smithi043.stdout: "agent_starting_port": { 2024-03-28T22:47:50.963 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "agent_starting_port", 2024-03-28T22:47:50.963 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:50.963 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.963 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:50.963 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "4721", 2024-03-28T22:47:50.963 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.963 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.963 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.963 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-28T22:47:50.964 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.964 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.964 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.964 INFO:teuthology.orchestra.run.smithi043.stdout: "allow_ptrace": { 2024-03-28T22:47:50.964 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "allow_ptrace", 2024-03-28T22:47:50.964 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:50.964 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.965 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:50.965 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:50.965 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.965 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.965 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.965 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-28T22:47:50.965 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-28T22:47:50.965 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.965 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.966 INFO:teuthology.orchestra.run.smithi043.stdout: "autotune_interval": { 2024-03-28T22:47:50.966 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "autotune_interval", 2024-03-28T22:47:50.966 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:50.966 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.966 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:50.966 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "600", 2024-03-28T22:47:50.966 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.966 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.967 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.967 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-28T22:47:50.967 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.967 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.967 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.967 INFO:teuthology.orchestra.run.smithi043.stdout: "autotune_memory_target_ratio": { 2024-03-28T22:47:50.967 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "autotune_memory_target_ratio", 2024-03-28T22:47:50.968 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-03-28T22:47:50.968 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.968 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:50.968 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0.7", 2024-03-28T22:47:50.968 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.968 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.968 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.968 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-28T22:47:50.968 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.969 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.969 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.969 INFO:teuthology.orchestra.run.smithi043.stdout: "cephadm_log_destination": { 2024-03-28T22:47:50.969 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cephadm_log_destination", 2024-03-28T22:47:50.969 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:50.969 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.969 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:50.969 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:50.969 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.970 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.970 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:50.970 INFO:teuthology.orchestra.run.smithi043.stdout: "file", 2024-03-28T22:47:50.970 INFO:teuthology.orchestra.run.smithi043.stdout: "file,syslog", 2024-03-28T22:47:50.970 INFO:teuthology.orchestra.run.smithi043.stdout: "syslog" 2024-03-28T22:47:50.970 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:50.970 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-28T22:47:50.970 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.970 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.971 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.971 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.971 INFO:teuthology.orchestra.run.smithi043.stdout: "cgroups_split": { 2024-03-28T22:47:50.971 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cgroups_split", 2024-03-28T22:47:50.971 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:50.971 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.971 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:50.971 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:50.971 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.972 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.972 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.972 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-28T22:47:50.972 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.972 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.972 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.972 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.972 INFO:teuthology.orchestra.run.smithi043.stdout: "config_checks_enabled": { 2024-03-28T22:47:50.972 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "config_checks_enabled", 2024-03-28T22:47:50.972 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:50.973 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.973 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:50.973 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:50.973 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.973 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.973 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.973 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-28T22:47:50.973 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.973 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.974 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.974 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.974 INFO:teuthology.orchestra.run.smithi043.stdout: "config_dashboard": { 2024-03-28T22:47:50.974 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "config_dashboard", 2024-03-28T22:47:50.974 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:50.974 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.974 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:50.974 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:50.974 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.975 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.975 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.975 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-28T22:47:50.975 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.975 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.975 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.975 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.975 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_alertmanager": { 2024-03-28T22:47:50.975 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_alertmanager", 2024-03-28T22:47:50.976 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:50.976 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.976 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:50.976 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-28T22:47:50.976 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.976 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.976 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.976 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Prometheus container image", 2024-03-28T22:47:50.976 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.976 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.977 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.977 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_base": { 2024-03-28T22:47:50.977 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_base", 2024-03-28T22:47:50.977 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:50.977 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.977 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:50.977 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-28T22:47:50.977 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.978 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.978 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.978 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Container image name, without the tag", 2024-03-28T22:47:50.978 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.978 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.978 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.978 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.978 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_elasticsearch": { 2024-03-28T22:47:50.978 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_elasticsearch", 2024-03-28T22:47:50.979 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:50.979 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.979 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:50.979 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-28T22:47:50.979 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.979 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.979 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.979 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "elasticsearch container image", 2024-03-28T22:47:50.979 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.980 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.980 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.980 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.980 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_grafana": { 2024-03-28T22:47:50.980 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_grafana", 2024-03-28T22:47:50.980 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:50.980 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.980 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:50.980 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-28T22:47:50.980 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.981 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.981 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.981 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Prometheus container image", 2024-03-28T22:47:50.981 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.981 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.981 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.981 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.981 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_haproxy": { 2024-03-28T22:47:50.981 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_haproxy", 2024-03-28T22:47:50.982 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:50.982 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.982 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:50.982 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-28T22:47:50.982 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.982 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.982 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.982 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "HAproxy container image", 2024-03-28T22:47:50.983 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.983 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.983 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.983 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_jaeger_agent": { 2024-03-28T22:47:50.983 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_jaeger_agent", 2024-03-28T22:47:50.983 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:50.983 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.983 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:50.983 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-28T22:47:50.984 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.984 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.984 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.984 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Jaeger agent container image", 2024-03-28T22:47:50.984 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.984 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.984 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.984 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.984 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_jaeger_collector": { 2024-03-28T22:47:50.985 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_jaeger_collector", 2024-03-28T22:47:50.985 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:50.985 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.985 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:50.985 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-28T22:47:50.985 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.985 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.985 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.985 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Jaeger collector container image", 2024-03-28T22:47:50.986 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.986 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.986 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.986 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.986 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_jaeger_query": { 2024-03-28T22:47:50.986 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_jaeger_query", 2024-03-28T22:47:50.986 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:50.986 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.987 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:50.987 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-28T22:47:50.987 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.987 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.987 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.987 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Jaeger query container image", 2024-03-28T22:47:50.987 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.987 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.987 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.988 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_keepalived": { 2024-03-28T22:47:50.988 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_keepalived", 2024-03-28T22:47:50.988 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:50.988 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.988 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:50.988 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-28T22:47:50.988 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.988 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.989 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.989 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Keepalived container image", 2024-03-28T22:47:50.989 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.989 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.989 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.989 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.989 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_loki": { 2024-03-28T22:47:50.989 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_loki", 2024-03-28T22:47:50.990 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:50.990 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.990 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:50.990 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-28T22:47:50.990 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.990 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.990 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.990 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Loki container image", 2024-03-28T22:47:50.991 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.991 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.991 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.991 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_node_exporter": { 2024-03-28T22:47:50.991 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_node_exporter", 2024-03-28T22:47:50.991 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:50.991 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.991 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:50.992 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-28T22:47:50.992 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.992 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.992 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.992 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Prometheus container image", 2024-03-28T22:47:50.992 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.992 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.992 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.993 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_nvmeof": { 2024-03-28T22:47:50.993 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_nvmeof", 2024-03-28T22:47:50.993 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:50.993 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.993 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:50.993 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-28T22:47:50.993 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.993 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.994 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.994 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Nvme-of container image", 2024-03-28T22:47:50.994 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.994 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.994 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.994 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.994 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_prometheus": { 2024-03-28T22:47:50.994 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_prometheus", 2024-03-28T22:47:50.994 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:50.995 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.995 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:50.995 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-28T22:47:50.995 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.995 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.995 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.995 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Prometheus container image", 2024-03-28T22:47:50.995 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.996 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.996 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.996 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.996 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_promtail": { 2024-03-28T22:47:50.996 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_promtail", 2024-03-28T22:47:50.996 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:50.996 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.996 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:50.996 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-28T22:47:50.997 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.997 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.997 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.997 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Promtail container image", 2024-03-28T22:47:50.997 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.997 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.997 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.997 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.997 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_snmp_gateway": { 2024-03-28T22:47:50.997 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_snmp_gateway", 2024-03-28T22:47:50.998 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:50.998 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.998 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:50.998 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-28T22:47:50.998 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:50.998 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:50.998 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:50.998 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "SNMP Gateway container image", 2024-03-28T22:47:50.999 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:50.999 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:50.999 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:50.999 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:50.999 INFO:teuthology.orchestra.run.smithi043.stdout: "container_init": { 2024-03-28T22:47:50.999 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_init", 2024-03-28T22:47:50.999 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:50.999 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:50.999 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:50.999 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.000 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.000 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.000 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.000 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Run podman/docker with `--init`", 2024-03-28T22:47:51.000 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.000 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.000 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.000 INFO:teuthology.orchestra.run.smithi043.stdout: "daemon_cache_timeout": { 2024-03-28T22:47:51.001 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "daemon_cache_timeout", 2024-03-28T22:47:51.001 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:51.001 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.001 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.001 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "600", 2024-03-28T22:47:51.001 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.001 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.001 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.001 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-28T22:47:51.002 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.002 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.002 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.002 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.002 INFO:teuthology.orchestra.run.smithi043.stdout: "default_cephadm_command_timeout": { 2024-03-28T22:47:51.002 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default_cephadm_command_timeout", 2024-03-28T22:47:51.002 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.002 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.002 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.003 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "900", 2024-03-28T22:47:51.003 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.003 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.003 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.003 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-28T22:47:51.003 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.003 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.003 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.003 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.004 INFO:teuthology.orchestra.run.smithi043.stdout: "default_registry": { 2024-03-28T22:47:51.004 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default_registry", 2024-03-28T22:47:51.004 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.004 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.004 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.004 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "docker.io", 2024-03-28T22:47:51.004 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.004 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.004 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.004 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-28T22:47:51.005 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.005 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.005 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.005 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.005 INFO:teuthology.orchestra.run.smithi043.stdout: "device_cache_timeout": { 2024-03-28T22:47:51.005 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "device_cache_timeout", 2024-03-28T22:47:51.005 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:51.005 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.006 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.006 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "1800", 2024-03-28T22:47:51.006 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.006 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.006 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.006 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "seconds to cache device inventory", 2024-03-28T22:47:51.006 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.006 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.007 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.007 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.007 INFO:teuthology.orchestra.run.smithi043.stdout: "device_enhanced_scan": { 2024-03-28T22:47:51.007 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "device_enhanced_scan", 2024-03-28T22:47:51.007 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.007 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.007 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.007 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.007 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.008 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.008 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.008 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-28T22:47:51.008 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.008 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.008 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.008 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.008 INFO:teuthology.orchestra.run.smithi043.stdout: "facts_cache_timeout": { 2024-03-28T22:47:51.008 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "facts_cache_timeout", 2024-03-28T22:47:51.009 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:51.009 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.009 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.009 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "60", 2024-03-28T22:47:51.009 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.009 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.009 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.009 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "seconds to cache host facts data", 2024-03-28T22:47:51.010 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.010 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.010 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.010 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.010 INFO:teuthology.orchestra.run.smithi043.stdout: "host_check_interval": { 2024-03-28T22:47:51.010 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "host_check_interval", 2024-03-28T22:47:51.010 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:51.010 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.011 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.011 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "600", 2024-03-28T22:47:51.011 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.011 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.011 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.011 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how frequently to perform a host check", 2024-03-28T22:47:51.011 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.011 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.011 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.012 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.012 INFO:teuthology.orchestra.run.smithi043.stdout: "hw_monitoring": { 2024-03-28T22:47:51.012 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "hw_monitoring", 2024-03-28T22:47:51.012 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.012 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.012 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.012 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.012 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.012 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.013 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.013 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-28T22:47:51.013 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.013 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.013 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.013 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.013 INFO:teuthology.orchestra.run.smithi043.stdout: "inventory_list_all": { 2024-03-28T22:47:51.013 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "inventory_list_all", 2024-03-28T22:47:51.013 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.014 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.014 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.014 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.014 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.014 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.014 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.014 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-28T22:47:51.014 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.014 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.015 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.015 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.015 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.015 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.015 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.015 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.015 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.015 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.015 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.016 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.016 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.016 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.016 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.016 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.016 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.016 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.016 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.016 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.017 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.017 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.017 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.017 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.017 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.017 INFO:teuthology.orchestra.run.smithi043.stdout: "log_refresh_metadata": { 2024-03-28T22:47:51.017 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_refresh_metadata", 2024-03-28T22:47:51.017 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.017 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.018 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.018 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.018 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.018 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.018 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.018 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-28T22:47:51.018 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.018 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.019 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.019 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.019 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.019 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.019 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.019 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.019 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.019 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.019 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.020 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.020 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-28T22:47:51.020 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.020 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.020 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.020 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.020 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.020 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.021 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.021 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.021 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.021 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.021 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.021 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.021 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.021 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.022 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.022 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.022 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.022 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.022 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.022 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.022 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.022 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.022 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.023 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.023 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.023 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.023 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.023 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.023 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.023 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.024 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.024 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.024 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.024 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.024 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.024 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.024 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.024 INFO:teuthology.orchestra.run.smithi043.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-28T22:47:51.024 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-28T22:47:51.025 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.025 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.025 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.025 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.025 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.025 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.025 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.025 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-28T22:47:51.026 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.026 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.026 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.026 INFO:teuthology.orchestra.run.smithi043.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-28T22:47:51.026 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-28T22:47:51.026 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.026 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.026 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.027 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "*", 2024-03-28T22:47:51.027 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.027 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.027 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.027 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-28T22:47:51.027 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.027 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.027 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.028 INFO:teuthology.orchestra.run.smithi043.stdout: "max_count_per_host": { 2024-03-28T22:47:51.028 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "max_count_per_host", 2024-03-28T22:47:51.028 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.028 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.028 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.028 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-03-28T22:47:51.028 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.028 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.029 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.029 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "max number of daemons per service per host", 2024-03-28T22:47:51.029 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.029 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.029 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.029 INFO:teuthology.orchestra.run.smithi043.stdout: "max_osd_draining_count": { 2024-03-28T22:47:51.029 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "max_osd_draining_count", 2024-03-28T22:47:51.029 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.030 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.030 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.030 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-03-28T22:47:51.030 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.030 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.030 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.030 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-28T22:47:51.030 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.031 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.031 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.031 INFO:teuthology.orchestra.run.smithi043.stdout: "migration_current": { 2024-03-28T22:47:51.031 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "migration_current", 2024-03-28T22:47:51.031 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.031 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.031 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.031 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.032 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.032 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.032 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.032 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "internal - do not modify", 2024-03-28T22:47:51.032 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.032 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.032 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.032 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": { 2024-03-28T22:47:51.033 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mode", 2024-03-28T22:47:51.033 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.033 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.033 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.033 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "root", 2024-03-28T22:47:51.033 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.033 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.033 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.033 INFO:teuthology.orchestra.run.smithi043.stdout: "cephadm-package", 2024-03-28T22:47:51.034 INFO:teuthology.orchestra.run.smithi043.stdout: "root" 2024-03-28T22:47:51.034 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.034 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "mode for remote execution of cephadm", 2024-03-28T22:47:51.034 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.034 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.034 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.034 INFO:teuthology.orchestra.run.smithi043.stdout: "oob_default_addr": { 2024-03-28T22:47:51.034 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "oob_default_addr", 2024-03-28T22:47:51.035 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.035 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.035 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.035 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "169.254.1.1", 2024-03-28T22:47:51.035 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.035 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.035 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.035 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-28T22:47:51.035 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.036 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.036 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.036 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.036 INFO:teuthology.orchestra.run.smithi043.stdout: "prometheus_alerts_path": { 2024-03-28T22:47:51.036 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "prometheus_alerts_path", 2024-03-28T22:47:51.036 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.036 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.036 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.036 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-28T22:47:51.037 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.037 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.037 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.037 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-28T22:47:51.037 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.037 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.037 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.037 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.037 INFO:teuthology.orchestra.run.smithi043.stdout: "registry_insecure": { 2024-03-28T22:47:51.038 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "registry_insecure", 2024-03-28T22:47:51.038 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.038 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.038 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.038 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.038 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.038 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.038 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.038 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-28T22:47:51.039 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.039 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.039 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.039 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.039 INFO:teuthology.orchestra.run.smithi043.stdout: "registry_password": { 2024-03-28T22:47:51.039 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "registry_password", 2024-03-28T22:47:51.039 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.039 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.040 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.040 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.040 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.040 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.040 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.040 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-28T22:47:51.040 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.040 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.040 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.041 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.041 INFO:teuthology.orchestra.run.smithi043.stdout: "registry_url": { 2024-03-28T22:47:51.041 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "registry_url", 2024-03-28T22:47:51.041 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.041 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.041 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.041 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.041 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.042 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.042 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.042 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-28T22:47:51.042 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.042 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.042 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.042 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.042 INFO:teuthology.orchestra.run.smithi043.stdout: "registry_username": { 2024-03-28T22:47:51.043 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "registry_username", 2024-03-28T22:47:51.043 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.043 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.043 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.043 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.043 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.043 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.043 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.043 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-28T22:47:51.044 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.044 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.044 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.044 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.044 INFO:teuthology.orchestra.run.smithi043.stdout: "secure_monitoring_stack": { 2024-03-28T22:47:51.044 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "secure_monitoring_stack", 2024-03-28T22:47:51.044 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.044 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.044 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.045 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.045 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.045 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.045 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.045 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-28T22:47:51.045 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.045 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.045 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.045 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.046 INFO:teuthology.orchestra.run.smithi043.stdout: "service_discovery_port": { 2024-03-28T22:47:51.046 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "service_discovery_port", 2024-03-28T22:47:51.046 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.046 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.046 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.046 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "8765", 2024-03-28T22:47:51.046 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.046 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.047 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.047 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "cephadm service discovery port", 2024-03-28T22:47:51.047 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.047 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.047 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.047 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.047 INFO:teuthology.orchestra.run.smithi043.stdout: "ssh_config_file": { 2024-03-28T22:47:51.047 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ssh_config_file", 2024-03-28T22:47:51.048 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.048 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.048 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.048 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.048 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.048 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.048 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.048 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-28T22:47:51.049 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.049 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.049 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.049 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.049 INFO:teuthology.orchestra.run.smithi043.stdout: "use_agent": { 2024-03-28T22:47:51.049 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "use_agent", 2024-03-28T22:47:51.049 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.049 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.050 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.050 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.050 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.050 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.050 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.050 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-28T22:47:51.050 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.050 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.051 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.051 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.051 INFO:teuthology.orchestra.run.smithi043.stdout: "use_repo_digest": { 2024-03-28T22:47:51.051 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "use_repo_digest", 2024-03-28T22:47:51.051 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.051 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.051 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.051 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.051 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.052 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.052 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.052 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-28T22:47:51.052 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.052 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.052 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.052 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.052 INFO:teuthology.orchestra.run.smithi043.stdout: "warn_on_failed_host_check": { 2024-03-28T22:47:51.053 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "warn_on_failed_host_check", 2024-03-28T22:47:51.053 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.053 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.053 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.053 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.053 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.053 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.053 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.053 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "raise a health warning if the host check fails", 2024-03-28T22:47:51.054 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.054 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.054 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.054 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.054 INFO:teuthology.orchestra.run.smithi043.stdout: "warn_on_stray_daemons": { 2024-03-28T22:47:51.054 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "warn_on_stray_daemons", 2024-03-28T22:47:51.054 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.054 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.055 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.055 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.055 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.055 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.055 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.055 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-28T22:47:51.055 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.055 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.056 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.056 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.056 INFO:teuthology.orchestra.run.smithi043.stdout: "warn_on_stray_hosts": { 2024-03-28T22:47:51.056 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "warn_on_stray_hosts", 2024-03-28T22:47:51.056 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.056 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.056 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.056 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.056 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.057 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.057 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.057 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-28T22:47:51.057 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.057 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.057 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.057 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.057 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.057 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.058 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.058 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "crash", 2024-03-28T22:47:51.058 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.058 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.058 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.058 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.058 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.058 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.058 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.059 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.059 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.059 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.059 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.059 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.059 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.059 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.059 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.060 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.060 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.060 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.060 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.060 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.060 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.060 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.060 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.060 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.061 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.061 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.061 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.061 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.061 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.061 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.061 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.061 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.061 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.062 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.062 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.062 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.062 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.062 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.062 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.062 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.063 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.063 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.063 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.063 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.063 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.063 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.063 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.063 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.063 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.064 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.064 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.064 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.064 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.064 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.064 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.064 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.064 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.064 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.065 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.065 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.065 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.065 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.065 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.065 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.065 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.065 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.065 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.066 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.066 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.066 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.066 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.066 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.066 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.066 INFO:teuthology.orchestra.run.smithi043.stdout: "retain_interval": { 2024-03-28T22:47:51.066 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "retain_interval", 2024-03-28T22:47:51.066 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:51.067 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.067 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.067 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "31536000", 2024-03-28T22:47:51.067 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.067 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.067 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.067 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-28T22:47:51.067 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.067 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.068 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.068 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.068 INFO:teuthology.orchestra.run.smithi043.stdout: "warn_recent_interval": { 2024-03-28T22:47:51.068 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "warn_recent_interval", 2024-03-28T22:47:51.068 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:51.068 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.068 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.068 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "1209600", 2024-03-28T22:47:51.069 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.069 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.069 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.069 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-28T22:47:51.069 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.069 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.069 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.069 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.069 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.070 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.070 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.070 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "dashboard", 2024-03-28T22:47:51.070 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.070 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.070 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.070 INFO:teuthology.orchestra.run.smithi043.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-28T22:47:51.070 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-28T22:47:51.071 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.071 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.071 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.071 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-03-28T22:47:51.071 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.071 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.071 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.071 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.071 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.072 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.072 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.072 INFO:teuthology.orchestra.run.smithi043.stdout: "ALERTMANAGER_API_HOST": { 2024-03-28T22:47:51.072 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-28T22:47:51.072 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.072 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.072 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.072 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.073 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.073 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.073 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.073 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.073 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.073 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.073 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.073 INFO:teuthology.orchestra.run.smithi043.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-28T22:47:51.074 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-28T22:47:51.074 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.074 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.074 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.074 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.074 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.074 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.075 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.075 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.075 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.075 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.075 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.075 INFO:teuthology.orchestra.run.smithi043.stdout: "AUDIT_API_ENABLED": { 2024-03-28T22:47:51.075 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "AUDIT_API_ENABLED", 2024-03-28T22:47:51.075 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.076 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.076 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.076 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.076 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.076 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.076 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.076 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.076 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.076 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.077 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.077 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.077 INFO:teuthology.orchestra.run.smithi043.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-28T22:47:51.077 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-28T22:47:51.077 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.077 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.077 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.077 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.077 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.078 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.078 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.078 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.078 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.078 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.078 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.078 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.078 INFO:teuthology.orchestra.run.smithi043.stdout: "ENABLE_BROWSABLE_API": { 2024-03-28T22:47:51.079 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-28T22:47:51.079 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.079 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.079 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.079 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.079 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.079 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.079 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.079 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.080 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.080 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.080 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.080 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.080 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-28T22:47:51.080 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-28T22:47:51.080 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.080 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.080 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.081 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.081 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.081 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.081 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.081 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.081 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.081 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.081 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.082 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.082 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-28T22:47:51.082 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-28T22:47:51.082 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.082 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.082 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.082 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.082 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.082 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.083 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.083 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.083 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.083 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.083 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.083 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.083 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-28T22:47:51.084 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-28T22:47:51.084 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.084 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.084 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.084 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.084 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.084 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.084 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.084 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.085 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.085 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.085 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.085 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.085 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-28T22:47:51.085 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-28T22:47:51.085 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.085 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.085 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.086 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.086 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.086 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.086 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.086 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.086 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.086 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.086 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.087 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.087 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-28T22:47:51.087 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-28T22:47:51.087 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.087 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.087 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.087 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.087 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.087 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.088 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.088 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.088 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.088 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.088 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.088 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.088 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-28T22:47:51.088 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-28T22:47:51.089 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.089 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.089 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.089 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.089 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.089 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.089 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.089 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.090 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.090 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.090 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.090 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.090 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-28T22:47:51.090 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-28T22:47:51.090 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.090 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.090 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.091 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.091 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.091 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.091 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.091 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.091 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.091 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.091 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.092 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.092 INFO:teuthology.orchestra.run.smithi043.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-28T22:47:51.092 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-28T22:47:51.092 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.092 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.092 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.092 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.092 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.092 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.093 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.093 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.093 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.093 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.093 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.093 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.093 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_API_PASSWORD": { 2024-03-28T22:47:51.093 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-28T22:47:51.093 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.094 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.094 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.094 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "admin", 2024-03-28T22:47:51.094 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.094 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.094 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.094 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.094 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.095 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.095 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.095 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.095 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-28T22:47:51.095 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-28T22:47:51.095 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.095 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.095 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.095 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.096 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.096 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.096 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.096 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.096 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.096 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.096 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.096 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.096 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_API_URL": { 2024-03-28T22:47:51.097 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_API_URL", 2024-03-28T22:47:51.097 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.097 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.097 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.097 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.097 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.097 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.097 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.097 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.098 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.098 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.098 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.098 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.098 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_API_USERNAME": { 2024-03-28T22:47:51.098 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-28T22:47:51.098 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.098 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.099 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.099 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "admin", 2024-03-28T22:47:51.099 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.099 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.099 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.099 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.099 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.099 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.099 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.100 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.100 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-28T22:47:51.100 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-28T22:47:51.100 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.100 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.100 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.100 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.100 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.100 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.100 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.101 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.101 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.101 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.101 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.101 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.101 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-28T22:47:51.101 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-28T22:47:51.101 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.101 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.102 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.102 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.102 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.102 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.102 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.102 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.102 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.102 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.103 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.103 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.103 INFO:teuthology.orchestra.run.smithi043.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-28T22:47:51.103 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-28T22:47:51.103 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.103 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.103 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.103 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.103 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.104 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.104 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.104 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.104 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.104 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.104 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.104 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.104 INFO:teuthology.orchestra.run.smithi043.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-28T22:47:51.104 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-28T22:47:51.105 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.105 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.105 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.105 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.105 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.105 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.105 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.105 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.105 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.106 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.106 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.106 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.106 INFO:teuthology.orchestra.run.smithi043.stdout: "PROMETHEUS_API_HOST": { 2024-03-28T22:47:51.106 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-28T22:47:51.106 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.106 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.106 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.107 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.107 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.107 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.107 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.107 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.107 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.107 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.107 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.107 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.108 INFO:teuthology.orchestra.run.smithi043.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-28T22:47:51.108 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-28T22:47:51.108 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.108 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.108 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.108 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.108 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.108 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.108 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.108 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.109 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.109 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.109 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.109 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.109 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-28T22:47:51.109 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-28T22:47:51.109 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.109 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.109 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.110 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.110 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.110 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.110 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.110 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.110 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.110 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.110 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.111 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.111 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-28T22:47:51.111 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-28T22:47:51.111 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.111 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.111 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.111 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.111 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.111 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.112 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.112 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.112 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.112 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.112 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.112 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.112 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-28T22:47:51.112 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-28T22:47:51.112 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.113 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.113 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.113 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.113 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.113 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.113 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.113 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.113 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.113 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.114 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.114 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.114 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-28T22:47:51.114 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-28T22:47:51.114 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.114 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.114 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.114 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.115 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.115 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.115 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.115 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.115 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.115 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.115 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.115 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.115 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-28T22:47:51.116 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-28T22:47:51.116 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.116 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.116 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.116 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.116 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.116 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.116 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.116 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.117 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.117 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.117 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.117 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.117 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-28T22:47:51.117 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-28T22:47:51.117 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.117 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.117 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.118 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.118 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.118 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.118 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.118 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.118 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.119 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.119 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.119 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.119 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-28T22:47:51.119 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-28T22:47:51.119 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.119 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.119 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.120 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.120 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.120 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.120 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.120 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.120 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.120 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.120 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.120 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.121 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_ENABLED": { 2024-03-28T22:47:51.121 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-28T22:47:51.121 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.121 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.121 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.121 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.121 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.121 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.121 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.122 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.122 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.122 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.122 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.122 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.122 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-28T22:47:51.122 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-28T22:47:51.122 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.123 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.123 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.123 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-28T22:47:51.123 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.123 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.123 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.123 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.123 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.123 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.124 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.124 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.124 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-28T22:47:51.124 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-28T22:47:51.124 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.124 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.124 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.124 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-03-28T22:47:51.125 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.125 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.125 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.125 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.125 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.125 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.125 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.125 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.125 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-28T22:47:51.126 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-28T22:47:51.126 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.126 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.126 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.126 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "8", 2024-03-28T22:47:51.126 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.126 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.127 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.127 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.127 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.127 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.127 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.127 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.127 INFO:teuthology.orchestra.run.smithi043.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-28T22:47:51.127 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-28T22:47:51.127 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.128 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.128 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.128 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "45", 2024-03-28T22:47:51.128 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.128 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.128 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.128 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.128 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.129 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.129 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.129 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.129 INFO:teuthology.orchestra.run.smithi043.stdout: "RGW_API_ACCESS_KEY": { 2024-03-28T22:47:51.129 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-28T22:47:51.129 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.129 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.129 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.129 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.130 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.130 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.130 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.130 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.130 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.130 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.130 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.130 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.131 INFO:teuthology.orchestra.run.smithi043.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-28T22:47:51.131 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-28T22:47:51.131 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.131 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.131 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.131 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "admin", 2024-03-28T22:47:51.131 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.131 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.131 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.132 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.132 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.132 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.132 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.132 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.132 INFO:teuthology.orchestra.run.smithi043.stdout: "RGW_API_SECRET_KEY": { 2024-03-28T22:47:51.132 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-28T22:47:51.132 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.133 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.133 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.133 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.133 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.133 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.133 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.133 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.133 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.133 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.134 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.134 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.134 INFO:teuthology.orchestra.run.smithi043.stdout: "RGW_API_SSL_VERIFY": { 2024-03-28T22:47:51.134 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-28T22:47:51.134 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.134 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.134 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.134 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.134 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.135 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.135 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.135 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.135 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.135 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.135 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.135 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.135 INFO:teuthology.orchestra.run.smithi043.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-28T22:47:51.135 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-28T22:47:51.136 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.136 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.136 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.136 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-03-28T22:47:51.136 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.136 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.136 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.136 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.137 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.137 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.137 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.137 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.137 INFO:teuthology.orchestra.run.smithi043.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-28T22:47:51.137 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-28T22:47:51.137 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.137 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.137 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.138 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-03-28T22:47:51.138 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.138 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.138 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.138 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.138 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.138 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.138 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.139 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.139 INFO:teuthology.orchestra.run.smithi043.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-28T22:47:51.139 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-28T22:47:51.139 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.139 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.139 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.139 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "5", 2024-03-28T22:47:51.139 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.139 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.140 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.140 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.140 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.140 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.140 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.140 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.140 INFO:teuthology.orchestra.run.smithi043.stdout: "cross_origin_url": { 2024-03-28T22:47:51.140 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cross_origin_url", 2024-03-28T22:47:51.140 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.141 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.141 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.141 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.141 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.141 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.141 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.141 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.141 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.142 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.142 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.142 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.142 INFO:teuthology.orchestra.run.smithi043.stdout: "crt_file": { 2024-03-28T22:47:51.142 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "crt_file", 2024-03-28T22:47:51.142 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.142 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.142 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.142 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.143 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.143 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.143 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.143 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.143 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.143 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.143 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.143 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.144 INFO:teuthology.orchestra.run.smithi043.stdout: "debug": { 2024-03-28T22:47:51.144 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "debug", 2024-03-28T22:47:51.144 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.144 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.144 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.144 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.144 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.144 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.144 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.145 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Enable/disable debug options", 2024-03-28T22:47:51.145 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.145 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.145 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.145 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.145 INFO:teuthology.orchestra.run.smithi043.stdout: "jwt_token_ttl": { 2024-03-28T22:47:51.145 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "jwt_token_ttl", 2024-03-28T22:47:51.145 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.145 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.145 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.146 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "28800", 2024-03-28T22:47:51.146 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.146 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.146 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.146 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.146 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.146 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.146 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.147 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.147 INFO:teuthology.orchestra.run.smithi043.stdout: "key_file": { 2024-03-28T22:47:51.147 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "key_file", 2024-03-28T22:47:51.147 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.147 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.147 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.147 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.147 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.148 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.148 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.148 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.148 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.148 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.148 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.148 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.148 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.149 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.149 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.149 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.149 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.149 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.149 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.149 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.149 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.150 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.150 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.150 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.150 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.150 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.150 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.150 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.150 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.151 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.151 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.151 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.151 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.151 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.151 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.151 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.151 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.151 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.152 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.152 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.152 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.152 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.152 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.152 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.152 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.152 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.153 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.153 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.153 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.153 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.153 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.153 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.153 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.153 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.153 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.154 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.154 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.154 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.154 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.154 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.154 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.154 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.154 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.155 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.155 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.155 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.155 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.155 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.155 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.155 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.155 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.155 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.156 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.156 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.156 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.156 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.156 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.156 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.156 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.156 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.156 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.157 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.157 INFO:teuthology.orchestra.run.smithi043.stdout: "motd": { 2024-03-28T22:47:51.157 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "motd", 2024-03-28T22:47:51.157 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.157 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.157 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.157 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.157 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.158 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.158 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.158 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "The message of the day", 2024-03-28T22:47:51.158 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.158 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.158 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.158 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.158 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_resolve_ip_addr": { 2024-03-28T22:47:51.159 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "redirect_resolve_ip_addr", 2024-03-28T22:47:51.159 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.159 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.159 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.159 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.159 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.159 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.159 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.159 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.160 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.160 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.160 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.160 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.160 INFO:teuthology.orchestra.run.smithi043.stdout: "server_addr": { 2024-03-28T22:47:51.160 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_addr", 2024-03-28T22:47:51.160 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.160 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.160 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.160 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "::", 2024-03-28T22:47:51.161 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.161 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.161 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.161 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.161 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.161 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.161 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.161 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.162 INFO:teuthology.orchestra.run.smithi043.stdout: "server_port": { 2024-03-28T22:47:51.162 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_port", 2024-03-28T22:47:51.162 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.162 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.162 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.162 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "8080", 2024-03-28T22:47:51.162 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.162 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.163 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.163 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.163 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.163 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.163 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.163 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.163 INFO:teuthology.orchestra.run.smithi043.stdout: "ssl": { 2024-03-28T22:47:51.163 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ssl", 2024-03-28T22:47:51.163 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.164 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.164 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.164 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.164 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.164 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.164 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.164 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.164 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.165 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.165 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.165 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.165 INFO:teuthology.orchestra.run.smithi043.stdout: "ssl_server_port": { 2024-03-28T22:47:51.165 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ssl_server_port", 2024-03-28T22:47:51.165 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.165 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.165 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.165 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "8443", 2024-03-28T22:47:51.166 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.166 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.166 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.166 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.166 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.166 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.166 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.166 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.166 INFO:teuthology.orchestra.run.smithi043.stdout: "standby_behaviour": { 2024-03-28T22:47:51.167 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "standby_behaviour", 2024-03-28T22:47:51.167 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.167 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.167 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.167 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "redirect", 2024-03-28T22:47:51.167 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.167 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.167 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.167 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.168 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect" 2024-03-28T22:47:51.168 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.168 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.168 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.168 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.168 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.168 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.168 INFO:teuthology.orchestra.run.smithi043.stdout: "standby_error_status_code": { 2024-03-28T22:47:51.169 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "standby_error_status_code", 2024-03-28T22:47:51.169 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.169 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.169 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.169 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "500", 2024-03-28T22:47:51.169 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "400", 2024-03-28T22:47:51.169 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "599", 2024-03-28T22:47:51.169 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.169 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.170 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.170 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.170 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.170 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.170 INFO:teuthology.orchestra.run.smithi043.stdout: "url_prefix": { 2024-03-28T22:47:51.170 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "url_prefix", 2024-03-28T22:47:51.170 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.171 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.171 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.171 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.171 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.171 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.171 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.171 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.171 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.171 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.172 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.172 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.172 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.172 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.172 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.172 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "devicehealth", 2024-03-28T22:47:51.172 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.172 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.172 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.173 INFO:teuthology.orchestra.run.smithi043.stdout: "enable_monitoring": { 2024-03-28T22:47:51.173 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "enable_monitoring", 2024-03-28T22:47:51.173 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.173 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.173 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.173 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.173 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.173 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.173 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.174 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "monitor device health metrics", 2024-03-28T22:47:51.174 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.174 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.174 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.174 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.174 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.174 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.174 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.175 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.175 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.175 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.175 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.175 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.175 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.175 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.175 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.175 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.176 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.176 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.176 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.176 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.176 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.176 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.176 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.176 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.176 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.177 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.177 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.177 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.177 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.177 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.177 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.177 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.177 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.177 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.178 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.178 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.178 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.178 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.178 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.178 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.178 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.178 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.178 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.179 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.179 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.179 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.179 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.179 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.179 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.179 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.179 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.180 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.180 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.180 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.180 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.180 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.180 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.180 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.180 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.180 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.181 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.181 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.181 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.181 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.181 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.181 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.181 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.181 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.181 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.182 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.182 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.182 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.182 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.182 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.182 INFO:teuthology.orchestra.run.smithi043.stdout: "mark_out_threshold": { 2024-03-28T22:47:51.182 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mark_out_threshold", 2024-03-28T22:47:51.182 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:51.183 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.183 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.183 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "2419200", 2024-03-28T22:47:51.183 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.183 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.183 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.183 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-28T22:47:51.183 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.183 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.184 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.184 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.184 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": { 2024-03-28T22:47:51.184 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "pool_name", 2024-03-28T22:47:51.184 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.184 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.184 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.184 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "device_health_metrics", 2024-03-28T22:47:51.185 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.185 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.185 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.185 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-28T22:47:51.185 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.185 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.185 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.185 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.185 INFO:teuthology.orchestra.run.smithi043.stdout: "retention_period": { 2024-03-28T22:47:51.186 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "retention_period", 2024-03-28T22:47:51.186 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:51.186 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.186 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.186 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "15552000", 2024-03-28T22:47:51.186 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.186 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.186 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.186 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how long to retain device health metrics", 2024-03-28T22:47:51.187 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.187 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.187 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.187 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.187 INFO:teuthology.orchestra.run.smithi043.stdout: "scrape_frequency": { 2024-03-28T22:47:51.187 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "scrape_frequency", 2024-03-28T22:47:51.187 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:51.187 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.187 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.188 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "86400", 2024-03-28T22:47:51.188 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.188 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.188 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.188 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-28T22:47:51.188 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.188 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.188 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.188 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.189 INFO:teuthology.orchestra.run.smithi043.stdout: "self_heal": { 2024-03-28T22:47:51.189 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "self_heal", 2024-03-28T22:47:51.189 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.189 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.189 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.189 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.189 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.189 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.189 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.190 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-28T22:47:51.190 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.190 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.190 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.190 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.190 INFO:teuthology.orchestra.run.smithi043.stdout: "sleep_interval": { 2024-03-28T22:47:51.190 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sleep_interval", 2024-03-28T22:47:51.190 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:51.190 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.191 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.191 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "600", 2024-03-28T22:47:51.191 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.191 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.191 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.191 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how frequently to wake up and check device health", 2024-03-28T22:47:51.191 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.191 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.192 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.192 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.192 INFO:teuthology.orchestra.run.smithi043.stdout: "warn_threshold": { 2024-03-28T22:47:51.192 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "warn_threshold", 2024-03-28T22:47:51.192 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:51.192 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.192 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.192 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "7257600", 2024-03-28T22:47:51.193 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.193 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.193 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.193 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-28T22:47:51.193 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.193 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.193 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.193 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.193 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.194 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.194 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.194 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "diskprediction_local", 2024-03-28T22:47:51.194 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.194 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.194 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.194 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.194 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.194 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.195 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.195 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.195 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.195 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.195 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.195 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.195 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.195 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.196 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.196 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.196 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.196 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.196 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.196 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.196 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.196 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.196 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.197 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.197 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.197 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.197 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.197 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.197 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.197 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.197 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.197 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.198 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.198 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.198 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.198 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.198 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.198 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.198 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.198 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.199 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.199 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.199 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.199 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.199 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.199 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.199 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.199 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.199 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.200 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.200 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.200 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.200 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.200 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.200 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.200 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.200 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.201 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.201 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.201 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.201 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.201 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.201 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.201 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.201 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.201 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.202 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.202 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.202 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.202 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.202 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.202 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.202 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.202 INFO:teuthology.orchestra.run.smithi043.stdout: "predict_interval": { 2024-03-28T22:47:51.203 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "predict_interval", 2024-03-28T22:47:51.203 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.203 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.203 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.203 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "86400", 2024-03-28T22:47:51.203 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.203 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.203 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.204 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.204 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.204 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.204 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.204 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.204 INFO:teuthology.orchestra.run.smithi043.stdout: "predictor_model": { 2024-03-28T22:47:51.204 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "predictor_model", 2024-03-28T22:47:51.204 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.204 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.205 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.205 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "prophetstor", 2024-03-28T22:47:51.205 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.205 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.205 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.205 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.205 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.205 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.205 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.206 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.206 INFO:teuthology.orchestra.run.smithi043.stdout: "sleep_interval": { 2024-03-28T22:47:51.206 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sleep_interval", 2024-03-28T22:47:51.206 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.206 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.206 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.206 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "600", 2024-03-28T22:47:51.206 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.207 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.207 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.207 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.207 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.207 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.207 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.207 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.207 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.207 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.208 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.208 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "influx", 2024-03-28T22:47:51.208 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": false, 2024-03-28T22:47:51.208 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "influxdb python module not found", 2024-03-28T22:47:51.208 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.208 INFO:teuthology.orchestra.run.smithi043.stdout: "batch_size": { 2024-03-28T22:47:51.208 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "batch_size", 2024-03-28T22:47:51.208 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.208 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.209 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.209 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "5000", 2024-03-28T22:47:51.209 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.209 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.209 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.209 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-28T22:47:51.209 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.209 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.210 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.210 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.210 INFO:teuthology.orchestra.run.smithi043.stdout: "database": { 2024-03-28T22:47:51.210 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "database", 2024-03-28T22:47:51.210 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.210 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.210 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.210 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "ceph", 2024-03-28T22:47:51.210 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.211 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.211 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.211 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-28T22:47:51.211 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.211 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.211 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.211 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.211 INFO:teuthology.orchestra.run.smithi043.stdout: "hostname": { 2024-03-28T22:47:51.212 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "hostname", 2024-03-28T22:47:51.212 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.212 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.212 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.212 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.212 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.212 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.212 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.212 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "InfluxDB server hostname", 2024-03-28T22:47:51.213 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.213 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.213 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.213 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.213 INFO:teuthology.orchestra.run.smithi043.stdout: "interval": { 2024-03-28T22:47:51.213 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "interval", 2024-03-28T22:47:51.213 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:51.213 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.214 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.214 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "30", 2024-03-28T22:47:51.214 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "5", 2024-03-28T22:47:51.214 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.214 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.214 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-28T22:47:51.214 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.214 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.215 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.215 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.215 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.215 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.215 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.215 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.215 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.215 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.215 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.216 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.216 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.216 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.216 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.216 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.216 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.216 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.216 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.216 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.216 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.217 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.217 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.217 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.217 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.217 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.217 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.217 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.217 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.217 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.218 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.218 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.218 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.218 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.218 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.218 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.218 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.218 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.219 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.219 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.219 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.219 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.219 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.219 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.219 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.219 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.219 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.220 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.220 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.220 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.220 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.220 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.220 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.220 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.220 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.220 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.221 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.221 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.221 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.221 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.221 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.221 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.221 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.221 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.221 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.221 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.222 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.222 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.222 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.222 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.222 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.222 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.222 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.222 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.223 INFO:teuthology.orchestra.run.smithi043.stdout: "password": { 2024-03-28T22:47:51.223 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "password", 2024-03-28T22:47:51.223 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.223 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.223 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.223 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.223 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.223 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.223 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.223 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "password of InfluxDB server user", 2024-03-28T22:47:51.224 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.224 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.224 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.224 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.224 INFO:teuthology.orchestra.run.smithi043.stdout: "port": { 2024-03-28T22:47:51.224 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "port", 2024-03-28T22:47:51.224 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.224 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.224 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.225 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "8086", 2024-03-28T22:47:51.225 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.225 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.225 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.225 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "InfluxDB server port", 2024-03-28T22:47:51.225 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.225 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.225 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.225 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.226 INFO:teuthology.orchestra.run.smithi043.stdout: "ssl": { 2024-03-28T22:47:51.226 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ssl", 2024-03-28T22:47:51.226 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.226 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.226 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.226 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "false", 2024-03-28T22:47:51.226 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.226 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.226 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.227 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T22:47:51.227 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.227 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.227 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.227 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.227 INFO:teuthology.orchestra.run.smithi043.stdout: "threads": { 2024-03-28T22:47:51.227 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "threads", 2024-03-28T22:47:51.227 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.227 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.228 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.228 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "5", 2024-03-28T22:47:51.228 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "1", 2024-03-28T22:47:51.228 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "32", 2024-03-28T22:47:51.228 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.228 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-28T22:47:51.228 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.228 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.228 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.229 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.229 INFO:teuthology.orchestra.run.smithi043.stdout: "username": { 2024-03-28T22:47:51.229 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "username", 2024-03-28T22:47:51.229 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.229 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.229 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.229 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.229 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.230 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.230 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.230 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "username of InfluxDB server user", 2024-03-28T22:47:51.230 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.230 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.230 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.230 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.230 INFO:teuthology.orchestra.run.smithi043.stdout: "verify_ssl": { 2024-03-28T22:47:51.230 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "verify_ssl", 2024-03-28T22:47:51.231 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.231 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.231 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.231 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "true", 2024-03-28T22:47:51.231 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.231 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.231 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.231 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T22:47:51.232 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.232 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.232 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.232 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.232 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.232 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.232 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.232 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "insights", 2024-03-28T22:47:51.232 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.233 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.233 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.233 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.233 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.233 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.233 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.233 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.233 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.233 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.234 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.234 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.234 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.234 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.234 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.234 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.234 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.235 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.235 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.235 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.235 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.235 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.235 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.235 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.235 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.235 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.236 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.236 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.236 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.236 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.236 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.236 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.236 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.236 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.237 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.237 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.237 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.237 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.237 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.237 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.237 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.237 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.237 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.238 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.238 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.238 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.238 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.238 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.238 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.238 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.238 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.238 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.239 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.239 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.239 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.239 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.239 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.239 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.239 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.239 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.240 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.240 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.240 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.240 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.240 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.240 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.240 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.240 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.241 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.241 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.241 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.241 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.241 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.241 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.241 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.241 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.241 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "iostat", 2024-03-28T22:47:51.242 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.242 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.242 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.242 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.242 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.242 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.242 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.242 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.243 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.243 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.243 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.243 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.243 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.243 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.243 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.243 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.244 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.244 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.244 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.244 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.244 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.244 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.244 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.244 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.244 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.245 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.245 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.245 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.245 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.245 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.245 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.245 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.245 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.245 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.246 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.246 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.246 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.246 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.246 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.246 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.246 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.246 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.247 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.247 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.247 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.247 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.247 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.247 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.247 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.247 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.247 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.248 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.248 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.248 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.248 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.248 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.248 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.248 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.248 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.248 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.249 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.249 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.249 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.249 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.249 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.249 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.249 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.249 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.249 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.250 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.250 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.250 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.250 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.250 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.250 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.250 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.251 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "k8sevents", 2024-03-28T22:47:51.251 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.251 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.251 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.251 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_event_retention_days": { 2024-03-28T22:47:51.251 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ceph_event_retention_days", 2024-03-28T22:47:51.251 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.251 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.251 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.251 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "7", 2024-03-28T22:47:51.252 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.252 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.252 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.252 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-28T22:47:51.252 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.252 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.252 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.252 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.252 INFO:teuthology.orchestra.run.smithi043.stdout: "config_check_secs": { 2024-03-28T22:47:51.253 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "config_check_secs", 2024-03-28T22:47:51.253 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.253 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.253 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.253 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-03-28T22:47:51.253 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "10", 2024-03-28T22:47:51.253 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.253 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.253 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-28T22:47:51.254 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.254 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.254 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.254 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.254 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.254 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.254 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.255 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.255 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.255 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.255 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.255 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.255 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.255 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.255 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.255 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.255 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.256 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.256 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.256 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.256 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.256 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.256 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.256 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.256 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.257 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.257 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.257 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.257 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.257 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.257 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.257 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.257 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.257 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.258 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.258 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.258 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.258 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.258 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.258 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.258 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.259 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.259 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.259 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.259 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.259 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.259 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.259 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.259 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.259 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.260 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.260 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.260 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.260 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.260 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.260 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.260 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.260 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.261 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.261 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.261 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.261 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.261 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.261 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.261 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.261 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.261 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.262 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.262 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.262 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.262 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.262 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.262 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.262 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.262 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.263 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "localpool", 2024-03-28T22:47:51.263 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.263 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.263 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.263 INFO:teuthology.orchestra.run.smithi043.stdout: "failure_domain": { 2024-03-28T22:47:51.263 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "failure_domain", 2024-03-28T22:47:51.263 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.263 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.264 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.264 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "host", 2024-03-28T22:47:51.264 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.264 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.264 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.264 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "failure domain for any created local pool", 2024-03-28T22:47:51.264 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-28T22:47:51.264 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.264 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.265 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.265 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.265 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.265 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.265 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.265 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.265 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.265 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.265 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.266 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.266 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.266 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.266 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.266 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.266 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.266 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.266 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.267 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.267 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.267 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.267 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.267 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.267 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.267 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.267 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.267 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.268 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.268 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.268 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.268 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.268 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.268 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.268 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.268 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.269 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.269 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.269 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.269 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.269 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.269 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.269 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.269 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.270 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.270 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.270 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.270 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.270 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.270 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.270 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.270 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.270 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.271 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.271 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.271 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.271 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.271 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.271 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.271 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.271 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.271 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.272 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.272 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.272 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.272 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.272 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.272 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.272 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.272 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.273 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.273 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.273 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": { 2024-03-28T22:47:51.273 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "min_size", 2024-03-28T22:47:51.273 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.273 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.273 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.273 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.273 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.274 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.274 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.274 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "default min_size for any created local pool", 2024-03-28T22:47:51.274 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-28T22:47:51.274 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.274 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.274 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.274 INFO:teuthology.orchestra.run.smithi043.stdout: "num_rep": { 2024-03-28T22:47:51.274 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "num_rep", 2024-03-28T22:47:51.275 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.275 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.275 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.275 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "3", 2024-03-28T22:47:51.275 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.275 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.275 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.275 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "default replica count for any created local pool", 2024-03-28T22:47:51.275 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.276 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.276 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.276 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": { 2024-03-28T22:47:51.276 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "pg_num", 2024-03-28T22:47:51.276 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.276 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.276 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.276 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "128", 2024-03-28T22:47:51.277 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.277 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.277 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.277 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "default pg_num for any created local pool", 2024-03-28T22:47:51.277 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.277 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.277 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.277 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.277 INFO:teuthology.orchestra.run.smithi043.stdout: "prefix": { 2024-03-28T22:47:51.278 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "prefix", 2024-03-28T22:47:51.278 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.278 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.278 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.278 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.278 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.278 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.278 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.278 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "name prefix for any created local pool", 2024-03-28T22:47:51.279 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.279 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.279 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.279 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.279 INFO:teuthology.orchestra.run.smithi043.stdout: "subtree": { 2024-03-28T22:47:51.279 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subtree", 2024-03-28T22:47:51.279 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.279 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.280 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.280 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "rack", 2024-03-28T22:47:51.280 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.280 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.280 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.280 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-28T22:47:51.280 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-28T22:47:51.280 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.280 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.281 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.281 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.281 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.281 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mds_autoscaler", 2024-03-28T22:47:51.281 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.281 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.281 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.281 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.281 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.282 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.282 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.282 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.282 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.282 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.282 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.282 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.282 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.283 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.283 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.283 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.283 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.283 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.283 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.283 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.283 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.283 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.284 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.284 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.284 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.284 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.284 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.284 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.284 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.284 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.284 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.285 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.285 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.285 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.285 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.285 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.285 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.285 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.285 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.286 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.286 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.286 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.286 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.286 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.286 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.286 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.286 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.286 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.287 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.287 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.287 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.287 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.287 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.287 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.287 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.287 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.287 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.288 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.288 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.288 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.288 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.288 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.288 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.288 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.288 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.288 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.289 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.289 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.289 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.289 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.289 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.289 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.289 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.289 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.290 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.290 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.290 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mirroring", 2024-03-28T22:47:51.290 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.290 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.290 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.290 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.290 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.291 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.291 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.291 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.291 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.291 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.291 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.291 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.291 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.291 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.292 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.292 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.292 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.292 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.292 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.292 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.292 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.292 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.292 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.293 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.293 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.293 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.293 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.293 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.293 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.293 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.293 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.293 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.294 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.294 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.294 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.294 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.294 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.294 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.294 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.294 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.294 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.295 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.295 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.295 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.295 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.295 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.295 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.295 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.295 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.295 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.296 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.296 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.296 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.296 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.296 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.296 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.296 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.296 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.296 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.297 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.297 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.297 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.297 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.297 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.297 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.297 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.297 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.298 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.298 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.298 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.298 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.298 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.298 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.298 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.298 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.298 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.299 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "nfs", 2024-03-28T22:47:51.299 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.299 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.299 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.299 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.299 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.299 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.299 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.300 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.300 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.300 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.300 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.300 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.300 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.300 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.300 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.300 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.301 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.301 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.301 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.301 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.301 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.301 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.301 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.302 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.302 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.302 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.302 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.302 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.302 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.302 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.302 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.303 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.303 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.303 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.303 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.303 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.303 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.303 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.303 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.303 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.304 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.304 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.304 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.304 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.304 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.304 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.304 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.304 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.304 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.305 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.305 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.305 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.305 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.305 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.305 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.305 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.305 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.305 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.306 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.306 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.306 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.306 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.306 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.306 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.306 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.306 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.307 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.307 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.307 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.307 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.307 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.307 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.307 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.307 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.308 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.308 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.308 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "orchestrator", 2024-03-28T22:47:51.308 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.308 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.308 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.308 INFO:teuthology.orchestra.run.smithi043.stdout: "fail_fs": { 2024-03-28T22:47:51.308 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "fail_fs", 2024-03-28T22:47:51.308 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.309 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.309 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.309 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.309 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.309 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.309 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.309 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-28T22:47:51.309 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.309 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.310 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.310 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.310 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.310 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.310 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.310 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.310 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.310 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.310 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.311 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.311 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.311 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.311 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.311 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.311 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.311 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.311 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.311 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.312 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.312 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.312 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.312 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.312 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.312 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.312 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.312 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.312 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.313 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.313 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.313 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.313 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.313 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.313 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.313 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.313 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.314 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.314 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.314 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.314 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.314 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.314 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.314 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.314 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.315 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.315 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.315 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.315 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.315 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.315 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.315 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.315 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.315 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.316 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.316 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.316 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.316 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.316 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.316 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.316 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.316 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.316 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.317 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.317 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.317 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.317 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.317 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.317 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.317 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.317 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.317 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.318 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.318 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.318 INFO:teuthology.orchestra.run.smithi043.stdout: "orchestrator": { 2024-03-28T22:47:51.318 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "orchestrator", 2024-03-28T22:47:51.318 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.318 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.318 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.318 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.319 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.319 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.319 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.319 INFO:teuthology.orchestra.run.smithi043.stdout: "cephadm", 2024-03-28T22:47:51.319 INFO:teuthology.orchestra.run.smithi043.stdout: "rook", 2024-03-28T22:47:51.319 INFO:teuthology.orchestra.run.smithi043.stdout: "test_orchestrator" 2024-03-28T22:47:51.319 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.319 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Orchestrator backend", 2024-03-28T22:47:51.319 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.320 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.320 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.320 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.320 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.320 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.320 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.320 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "osd_perf_query", 2024-03-28T22:47:51.320 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.320 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.321 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.321 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.321 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.321 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.321 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.321 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.321 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.321 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.321 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.322 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.322 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.322 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.322 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.322 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.322 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.322 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.322 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.323 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.323 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.323 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.323 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.323 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.323 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.323 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.323 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.323 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.324 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.324 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.324 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.324 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.324 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.324 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.324 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.324 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.325 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.325 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.325 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.325 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.325 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.325 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.325 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.325 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.325 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.325 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.326 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.326 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.326 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.326 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.326 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.326 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.326 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.326 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.327 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.327 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.327 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.327 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.327 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.327 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.327 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.327 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.327 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.328 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.328 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.328 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.328 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.328 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.328 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.328 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.328 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.328 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.329 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.329 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.329 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.329 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.329 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "osd_support", 2024-03-28T22:47:51.329 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.329 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.329 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.330 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.330 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.330 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.330 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.330 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.330 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.330 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.330 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.330 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.331 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.331 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.331 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.331 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.331 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.331 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.331 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.331 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.332 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.332 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.332 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.332 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.332 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.332 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.332 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.332 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.332 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.333 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.333 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.333 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.333 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.333 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.333 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.333 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.333 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.333 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.334 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.334 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.334 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.334 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.334 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.334 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.334 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.334 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.335 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.335 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.335 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.335 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.335 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.335 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.335 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.335 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.335 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.336 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.336 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.336 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.336 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.336 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.336 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.336 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.336 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.336 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.337 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.337 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.337 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.337 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.337 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.337 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.337 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.337 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.337 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.338 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.338 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.338 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.338 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "pg_autoscaler", 2024-03-28T22:47:51.338 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.338 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.338 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.338 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.338 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.339 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.339 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.339 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.339 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.339 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.339 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.339 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.339 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.339 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.340 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.340 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.340 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.340 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.340 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.340 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.340 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.340 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.340 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.341 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.341 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.341 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.341 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.341 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.341 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.341 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.341 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.341 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.341 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.342 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.342 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.342 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.342 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.342 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.342 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.342 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.342 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.343 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.343 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.343 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.343 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.343 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.343 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.343 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.343 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.343 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.344 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.344 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.344 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.344 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.344 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.344 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.344 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.344 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.344 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.345 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.345 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.345 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.345 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.345 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.345 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.345 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.345 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.345 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.346 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.346 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.346 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.346 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.346 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.346 INFO:teuthology.orchestra.run.smithi043.stdout: "sleep_interval": { 2024-03-28T22:47:51.346 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sleep_interval", 2024-03-28T22:47:51.346 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:51.347 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.347 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.347 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "60", 2024-03-28T22:47:51.347 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.347 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.347 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.347 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.347 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.347 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.348 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.348 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.348 INFO:teuthology.orchestra.run.smithi043.stdout: "threshold": { 2024-03-28T22:47:51.348 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "threshold", 2024-03-28T22:47:51.348 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-03-28T22:47:51.348 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.348 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.348 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "3.0", 2024-03-28T22:47:51.348 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "1.0", 2024-03-28T22:47:51.349 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.349 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.349 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "scaling threshold", 2024-03-28T22:47:51.349 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-28T22:47:51.349 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.349 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.349 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.349 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.349 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.350 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.350 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "progress", 2024-03-28T22:47:51.350 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.350 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.350 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.350 INFO:teuthology.orchestra.run.smithi043.stdout: "allow_pg_recovery_event": { 2024-03-28T22:47:51.350 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "allow_pg_recovery_event", 2024-03-28T22:47:51.350 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.351 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.351 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.351 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.351 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.351 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.351 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.351 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-28T22:47:51.351 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.351 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.352 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.352 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.352 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": { 2024-03-28T22:47:51.352 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "enabled", 2024-03-28T22:47:51.352 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.352 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.352 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.352 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.352 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.353 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.353 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.353 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.353 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.353 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.353 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.353 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.353 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.353 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.354 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.354 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.354 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.354 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.354 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.354 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.354 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.354 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.354 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.355 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.355 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.355 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.355 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.355 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.355 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.355 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.355 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.355 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.356 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.356 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.356 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.356 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.356 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.356 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.356 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.356 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.356 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.357 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.357 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.357 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.357 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.357 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.357 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.357 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.357 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.358 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.358 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.358 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.358 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.358 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.358 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.358 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.358 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.358 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.359 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.359 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.359 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.359 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.359 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.359 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.359 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.359 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.360 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.360 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.360 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.360 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.360 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.360 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.360 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.360 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.360 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.361 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.361 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.361 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.361 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.361 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.361 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.361 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.361 INFO:teuthology.orchestra.run.smithi043.stdout: "max_completed_events": { 2024-03-28T22:47:51.361 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "max_completed_events", 2024-03-28T22:47:51.362 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.362 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.362 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.362 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "50", 2024-03-28T22:47:51.362 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.362 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.362 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.362 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "number of past completed events to remember", 2024-03-28T22:47:51.363 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.363 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.363 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.363 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.363 INFO:teuthology.orchestra.run.smithi043.stdout: "sleep_interval": { 2024-03-28T22:47:51.363 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sleep_interval", 2024-03-28T22:47:51.363 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:51.363 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.363 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.363 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "5", 2024-03-28T22:47:51.364 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.364 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.364 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.364 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how long the module is going to sleep", 2024-03-28T22:47:51.364 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.364 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.364 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.364 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.364 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.365 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.365 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.365 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "prometheus", 2024-03-28T22:47:51.365 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.365 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.365 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.365 INFO:teuthology.orchestra.run.smithi043.stdout: "cache": { 2024-03-28T22:47:51.365 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cache", 2024-03-28T22:47:51.365 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.365 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.366 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.366 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.366 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.366 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.366 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.366 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.366 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.366 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.366 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.367 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.367 INFO:teuthology.orchestra.run.smithi043.stdout: "exclude_perf_counters": { 2024-03-28T22:47:51.367 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "exclude_perf_counters", 2024-03-28T22:47:51.367 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.367 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.367 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.367 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.367 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.367 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.368 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.368 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-28T22:47:51.368 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-28T22:47:51.368 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.368 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.368 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.368 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.368 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.368 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.369 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.369 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.369 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.369 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.369 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.369 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.369 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.369 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.370 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.370 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.370 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.370 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.370 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.370 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.370 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.370 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.370 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.371 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.371 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.371 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.371 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.371 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.371 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.371 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.371 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.371 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.372 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.372 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.372 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.372 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.372 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.372 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.372 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.372 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.372 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.373 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.373 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.373 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.373 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.373 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.373 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.373 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.373 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.373 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.374 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.374 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.374 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.374 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.374 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.374 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.374 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.374 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.374 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.375 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.375 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.375 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.375 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.375 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.375 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.375 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.375 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.376 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.376 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.376 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.376 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.376 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.376 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.376 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_stats_pools": { 2024-03-28T22:47:51.376 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rbd_stats_pools", 2024-03-28T22:47:51.376 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.377 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.377 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.377 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.377 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.377 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.377 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.377 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.377 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.377 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.377 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.378 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.378 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-28T22:47:51.378 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-28T22:47:51.378 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.378 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.378 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.378 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "300", 2024-03-28T22:47:51.378 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.378 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.379 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.379 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.379 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.379 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.379 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.379 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.379 INFO:teuthology.orchestra.run.smithi043.stdout: "scrape_interval": { 2024-03-28T22:47:51.379 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "scrape_interval", 2024-03-28T22:47:51.379 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-03-28T22:47:51.380 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.380 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.380 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "15.0", 2024-03-28T22:47:51.380 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.380 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.380 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.380 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.380 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.380 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.381 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.381 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.381 INFO:teuthology.orchestra.run.smithi043.stdout: "server_addr": { 2024-03-28T22:47:51.381 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_addr", 2024-03-28T22:47:51.381 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.381 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.381 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.381 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "::", 2024-03-28T22:47:51.381 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.382 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.382 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.382 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-28T22:47:51.382 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.382 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.382 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.382 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.382 INFO:teuthology.orchestra.run.smithi043.stdout: "server_port": { 2024-03-28T22:47:51.382 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_port", 2024-03-28T22:47:51.383 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.383 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.383 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.383 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "9283", 2024-03-28T22:47:51.383 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.383 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.383 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.383 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-28T22:47:51.383 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.383 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.384 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.384 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.384 INFO:teuthology.orchestra.run.smithi043.stdout: "stale_cache_strategy": { 2024-03-28T22:47:51.384 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "stale_cache_strategy", 2024-03-28T22:47:51.384 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.384 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.384 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.384 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "log", 2024-03-28T22:47:51.384 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.384 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.385 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.385 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.385 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.385 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.385 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.385 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.385 INFO:teuthology.orchestra.run.smithi043.stdout: "standby_behaviour": { 2024-03-28T22:47:51.385 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "standby_behaviour", 2024-03-28T22:47:51.385 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.386 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.386 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.386 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "default", 2024-03-28T22:47:51.386 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.386 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.386 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.386 INFO:teuthology.orchestra.run.smithi043.stdout: "default", 2024-03-28T22:47:51.386 INFO:teuthology.orchestra.run.smithi043.stdout: "error" 2024-03-28T22:47:51.386 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.387 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.387 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.387 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.387 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.387 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.387 INFO:teuthology.orchestra.run.smithi043.stdout: "standby_error_status_code": { 2024-03-28T22:47:51.387 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "standby_error_status_code", 2024-03-28T22:47:51.387 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.387 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.388 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.388 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "500", 2024-03-28T22:47:51.388 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "400", 2024-03-28T22:47:51.388 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "599", 2024-03-28T22:47:51.388 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.388 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.388 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.388 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.389 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.389 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.389 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.389 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.389 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.389 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rbd_support", 2024-03-28T22:47:51.389 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.389 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.389 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.390 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.390 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.390 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.390 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.390 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.390 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.390 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.390 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.391 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.391 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.391 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.391 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.391 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.391 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.391 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.391 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.392 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.392 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.392 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.392 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.392 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.392 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.392 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.392 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.392 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.393 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.393 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.393 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.393 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.393 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.393 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.393 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.393 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.393 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.394 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.394 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.394 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.394 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.394 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.394 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.394 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.394 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.394 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.395 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.395 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.395 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.395 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.395 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.395 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.395 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.395 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.396 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.396 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.396 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.396 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.396 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.396 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.396 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.396 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.396 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.397 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.397 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.397 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.397 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.397 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.397 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.397 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.397 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.397 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.398 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.398 INFO:teuthology.orchestra.run.smithi043.stdout: "max_concurrent_snap_create": { 2024-03-28T22:47:51.398 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "max_concurrent_snap_create", 2024-03-28T22:47:51.398 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.398 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.398 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.398 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-03-28T22:47:51.398 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.399 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.399 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.399 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.399 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.399 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.399 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.399 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.399 INFO:teuthology.orchestra.run.smithi043.stdout: "mirror_snapshot_schedule": { 2024-03-28T22:47:51.399 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mirror_snapshot_schedule", 2024-03-28T22:47:51.400 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.400 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.400 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.400 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.400 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.400 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.400 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.400 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.400 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.401 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.401 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.401 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.401 INFO:teuthology.orchestra.run.smithi043.stdout: "trash_purge_schedule": { 2024-03-28T22:47:51.401 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "trash_purge_schedule", 2024-03-28T22:47:51.401 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.401 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.401 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.401 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.402 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.402 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.402 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.402 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.402 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.402 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.402 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.402 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.403 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.403 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.403 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.403 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "restful", 2024-03-28T22:47:51.403 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.403 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.403 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.403 INFO:teuthology.orchestra.run.smithi043.stdout: "enable_auth": { 2024-03-28T22:47:51.403 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "enable_auth", 2024-03-28T22:47:51.404 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.404 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.404 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.404 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.404 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.404 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.404 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.404 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.405 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.405 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.405 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.405 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.405 INFO:teuthology.orchestra.run.smithi043.stdout: "key_file": { 2024-03-28T22:47:51.405 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "key_file", 2024-03-28T22:47:51.405 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.405 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.405 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.406 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.406 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.406 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.406 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.406 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.406 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.406 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.406 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.407 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.407 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.407 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.407 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.407 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.407 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.407 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.407 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.407 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.408 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.408 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.408 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.408 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.408 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.408 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.408 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.408 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.408 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.409 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.409 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.409 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.409 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.409 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.409 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.409 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.410 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.410 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.410 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.410 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.410 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.410 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.410 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.411 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.411 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.411 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.411 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.411 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.411 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.411 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.411 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.411 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.412 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.412 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.412 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.412 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.412 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.412 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.412 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.412 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.413 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.413 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.413 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.413 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.413 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.413 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.413 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.413 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.413 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.414 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.414 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.414 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.414 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.414 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.414 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.414 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.414 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.415 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.415 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.415 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.415 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.415 INFO:teuthology.orchestra.run.smithi043.stdout: "server_addr": { 2024-03-28T22:47:51.415 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_addr", 2024-03-28T22:47:51.415 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.415 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.415 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.416 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.416 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.416 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.416 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.416 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.416 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.416 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.416 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.417 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.417 INFO:teuthology.orchestra.run.smithi043.stdout: "server_port": { 2024-03-28T22:47:51.417 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_port", 2024-03-28T22:47:51.417 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.417 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.417 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.417 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.417 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.418 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.418 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.418 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.418 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.418 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.418 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.418 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.418 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.419 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.419 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.419 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rgw", 2024-03-28T22:47:51.419 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.419 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.419 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.419 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.419 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.419 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.420 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.420 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.420 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.420 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.420 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.420 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.420 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.420 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.420 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.421 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.421 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.421 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.421 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.421 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.421 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.421 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.421 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.422 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.422 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.422 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.422 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.422 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.422 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.422 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.422 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.423 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.423 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.423 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.423 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.423 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.423 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.423 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.424 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.424 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.424 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.424 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.424 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.424 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.424 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.424 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.425 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.425 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.425 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.425 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.425 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.425 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.425 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.425 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.425 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.426 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.426 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.426 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.426 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.426 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.426 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.426 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.426 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.427 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.427 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.427 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.427 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.427 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.427 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.427 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.427 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.427 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.428 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.428 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.428 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rook", 2024-03-28T22:47:51.428 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.428 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.428 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.428 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.428 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.429 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.429 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.429 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.429 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.429 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.429 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.429 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.429 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.429 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.430 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.430 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.430 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.430 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.430 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.430 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.430 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.430 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.431 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.431 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.431 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.431 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.431 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.431 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.431 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.431 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.431 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.432 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.432 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.432 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.432 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.432 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.432 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.432 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.432 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.432 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.433 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.433 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.433 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.433 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.433 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.433 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.433 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.433 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.433 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.434 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.434 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.434 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.434 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.434 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.434 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.434 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.435 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.435 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.435 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.435 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.435 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.435 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.435 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.435 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.435 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.436 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.436 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.436 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.436 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.436 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.436 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.436 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.436 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.436 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": { 2024-03-28T22:47:51.437 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "storage_class", 2024-03-28T22:47:51.437 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.437 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.437 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.437 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "local", 2024-03-28T22:47:51.437 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.437 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.437 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.437 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-28T22:47:51.438 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.438 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.438 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.438 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.438 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.438 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.438 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.438 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "selftest", 2024-03-28T22:47:51.439 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.439 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.439 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.439 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.439 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.439 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.439 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.439 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.440 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.440 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.440 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.440 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.440 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.440 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.440 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.440 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.440 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.441 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.441 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.441 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.441 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.441 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.441 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.441 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.441 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.441 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.442 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.442 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.442 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.442 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.442 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.442 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.442 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.443 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.443 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.443 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.443 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.443 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.443 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.443 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.443 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.443 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.444 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.444 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.444 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.444 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.444 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.444 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.444 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.444 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.444 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.445 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.445 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.445 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.445 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.445 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.445 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.445 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.445 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.445 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.446 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.446 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.446 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.446 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.446 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.446 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.446 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.446 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.447 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.447 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.447 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.447 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.447 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.447 INFO:teuthology.orchestra.run.smithi043.stdout: "roption1": { 2024-03-28T22:47:51.447 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "roption1", 2024-03-28T22:47:51.447 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.447 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.448 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.448 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.448 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.448 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.448 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.448 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.448 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.448 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.448 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.449 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.449 INFO:teuthology.orchestra.run.smithi043.stdout: "roption2": { 2024-03-28T22:47:51.449 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "roption2", 2024-03-28T22:47:51.449 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.449 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.449 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.449 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "xyz", 2024-03-28T22:47:51.449 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.449 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.450 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.450 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.450 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.450 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.450 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.450 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.450 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption1": { 2024-03-28T22:47:51.450 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption1", 2024-03-28T22:47:51.451 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.451 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.451 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.451 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.451 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.451 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.451 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.451 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.452 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.452 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.452 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.452 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.452 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption2": { 2024-03-28T22:47:51.452 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption2", 2024-03-28T22:47:51.452 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.452 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.452 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.453 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.453 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.453 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.453 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.453 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.453 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.453 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.453 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.454 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption3": { 2024-03-28T22:47:51.454 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption3", 2024-03-28T22:47:51.454 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-03-28T22:47:51.454 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.454 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.454 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.454 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.454 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.455 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.455 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.455 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.455 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.455 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.455 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.455 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption4": { 2024-03-28T22:47:51.455 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption4", 2024-03-28T22:47:51.455 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.456 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.456 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.456 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.456 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.456 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.456 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.456 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.456 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.457 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.457 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.457 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.457 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption5": { 2024-03-28T22:47:51.457 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption5", 2024-03-28T22:47:51.457 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.457 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.457 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.457 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.458 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.458 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.458 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.458 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.458 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.458 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.458 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.458 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.459 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption6": { 2024-03-28T22:47:51.459 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption6", 2024-03-28T22:47:51.459 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.459 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.459 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.459 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.459 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.459 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.460 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.460 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.460 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.460 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.460 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.460 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.460 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption7": { 2024-03-28T22:47:51.460 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption7", 2024-03-28T22:47:51.460 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.461 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.461 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.461 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.461 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "1", 2024-03-28T22:47:51.461 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "42", 2024-03-28T22:47:51.461 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.461 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.461 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.461 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.462 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.462 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.462 INFO:teuthology.orchestra.run.smithi043.stdout: "testkey": { 2024-03-28T22:47:51.462 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "testkey", 2024-03-28T22:47:51.462 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.462 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.462 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.462 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.463 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.463 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.463 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.463 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.463 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.463 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.463 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.463 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.463 INFO:teuthology.orchestra.run.smithi043.stdout: "testlkey": { 2024-03-28T22:47:51.464 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "testlkey", 2024-03-28T22:47:51.464 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.464 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.464 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.464 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.464 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.464 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.464 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.464 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.465 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.465 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.465 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.465 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.465 INFO:teuthology.orchestra.run.smithi043.stdout: "testnewline": { 2024-03-28T22:47:51.465 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "testnewline", 2024-03-28T22:47:51.465 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.465 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.465 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.466 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.466 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.466 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.466 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.466 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.466 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.466 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.466 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.466 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.467 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.467 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.467 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.467 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "snap_schedule", 2024-03-28T22:47:51.467 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.467 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.467 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.467 INFO:teuthology.orchestra.run.smithi043.stdout: "allow_m_granularity": { 2024-03-28T22:47:51.467 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "allow_m_granularity", 2024-03-28T22:47:51.468 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.468 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.468 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.468 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.468 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.468 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.468 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.468 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "allow minute scheduled snapshots", 2024-03-28T22:47:51.468 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.469 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.469 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.469 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.469 INFO:teuthology.orchestra.run.smithi043.stdout: "dump_on_update": { 2024-03-28T22:47:51.469 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "dump_on_update", 2024-03-28T22:47:51.469 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.469 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.469 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.469 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.470 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.470 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.470 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.470 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "dump database to debug log on update", 2024-03-28T22:47:51.470 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.470 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.470 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.470 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.471 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.471 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.471 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.471 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.471 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.471 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.471 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.471 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.471 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.472 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.472 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.472 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.472 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.472 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.472 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.472 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.472 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.472 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.472 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.473 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.473 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.473 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.473 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.473 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.473 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.473 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.473 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.473 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.474 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.474 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.474 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.474 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.474 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.474 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.474 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.474 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.475 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.475 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.475 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.475 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.475 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.475 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.475 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.475 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.475 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.475 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.476 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.476 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.476 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.476 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.476 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.476 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.476 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.476 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.476 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.477 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.477 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.477 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.477 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.477 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.477 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.477 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.477 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.477 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.478 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.478 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.478 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.478 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.478 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.478 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.478 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.478 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.479 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.479 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "stats", 2024-03-28T22:47:51.479 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.479 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.479 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.479 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.479 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.479 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.479 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.480 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.480 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.480 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.480 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.480 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.480 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.480 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.480 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.480 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.481 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.481 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.481 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.481 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.481 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.481 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.481 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.481 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.481 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.482 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.482 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.482 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.482 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.482 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.482 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.482 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.482 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.482 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.483 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.483 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.483 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.483 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.483 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.483 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.483 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.483 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.483 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.484 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.484 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.484 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.484 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.484 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.484 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.484 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.484 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.484 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.485 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.485 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.485 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.485 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.485 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.485 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.485 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.485 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.485 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.486 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.486 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.486 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.486 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.486 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.486 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.486 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.486 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.487 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.487 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.487 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.487 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.487 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.487 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.487 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.487 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "status", 2024-03-28T22:47:51.487 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.488 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.488 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.488 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.488 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.488 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.488 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.488 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.488 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.488 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.489 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.489 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.489 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.489 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.489 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.489 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.489 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.489 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.490 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.490 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.490 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.490 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.490 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.490 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.490 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.490 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.491 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.491 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.491 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.491 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.491 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.491 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.491 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.491 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.491 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.492 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.492 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.492 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.492 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.492 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.492 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.492 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.492 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.492 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.493 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.493 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.493 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.493 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.493 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.493 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.493 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.493 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.494 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.494 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.494 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.494 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.494 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.494 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.494 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.494 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.495 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.495 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.495 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.495 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.495 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.495 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.495 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.495 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.495 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.496 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.496 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.496 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.496 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.496 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.496 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.496 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.496 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "telegraf", 2024-03-28T22:47:51.496 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.497 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.497 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.497 INFO:teuthology.orchestra.run.smithi043.stdout: "address": { 2024-03-28T22:47:51.497 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "address", 2024-03-28T22:47:51.497 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.497 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.497 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.497 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-28T22:47:51.497 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.498 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.498 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.498 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.498 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.498 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.498 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.498 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.498 INFO:teuthology.orchestra.run.smithi043.stdout: "interval": { 2024-03-28T22:47:51.499 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "interval", 2024-03-28T22:47:51.499 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:51.499 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.499 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.499 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "15", 2024-03-28T22:47:51.499 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.499 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.499 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.499 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.500 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.500 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.500 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.500 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.500 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.500 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.500 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.500 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.501 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.501 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.501 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.501 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.501 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.501 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.501 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.501 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.501 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.502 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.502 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.502 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.502 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.502 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.502 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.502 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.502 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.502 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.503 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.503 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.503 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.503 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.503 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.503 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.503 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.503 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.503 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.504 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.504 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.504 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.504 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.504 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.504 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.504 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.504 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.504 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.505 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.505 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.505 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.505 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.505 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.505 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.505 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.505 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.505 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.505 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.506 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.506 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.506 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.506 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.506 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.506 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.506 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.506 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.507 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.507 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.507 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.507 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.507 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.507 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.507 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.507 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.507 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.508 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.508 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.508 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.508 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.508 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.508 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "telemetry", 2024-03-28T22:47:51.508 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.508 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.509 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.509 INFO:teuthology.orchestra.run.smithi043.stdout: "channel_basic": { 2024-03-28T22:47:51.509 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "channel_basic", 2024-03-28T22:47:51.509 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.509 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.509 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.509 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.509 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.509 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.510 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.510 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-28T22:47:51.510 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.510 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.510 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.510 INFO:teuthology.orchestra.run.smithi043.stdout: "channel_crash": { 2024-03-28T22:47:51.510 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "channel_crash", 2024-03-28T22:47:51.511 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.511 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.511 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.511 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.511 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.511 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.511 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.511 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-28T22:47:51.511 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.511 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.512 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.512 INFO:teuthology.orchestra.run.smithi043.stdout: "channel_device": { 2024-03-28T22:47:51.512 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "channel_device", 2024-03-28T22:47:51.512 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.512 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.512 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.512 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.512 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.513 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.513 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.513 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-28T22:47:51.513 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.513 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.513 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.513 INFO:teuthology.orchestra.run.smithi043.stdout: "channel_ident": { 2024-03-28T22:47:51.513 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "channel_ident", 2024-03-28T22:47:51.514 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.514 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.514 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.514 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.514 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.514 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.514 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.514 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-28T22:47:51.514 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.515 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.515 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.515 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.515 INFO:teuthology.orchestra.run.smithi043.stdout: "channel_perf": { 2024-03-28T22:47:51.515 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "channel_perf", 2024-03-28T22:47:51.515 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.515 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.515 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.516 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.516 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.516 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.516 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.516 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-28T22:47:51.516 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.516 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.516 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.517 INFO:teuthology.orchestra.run.smithi043.stdout: "contact": { 2024-03-28T22:47:51.517 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "contact", 2024-03-28T22:47:51.517 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.517 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.517 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.517 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.517 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.517 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.518 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.518 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.518 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.518 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.518 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.518 INFO:teuthology.orchestra.run.smithi043.stdout: "description": { 2024-03-28T22:47:51.518 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "description", 2024-03-28T22:47:51.519 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.519 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.519 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.519 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.519 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.519 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.519 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.519 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.519 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.520 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.520 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.520 INFO:teuthology.orchestra.run.smithi043.stdout: "device_url": { 2024-03-28T22:47:51.520 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "device_url", 2024-03-28T22:47:51.520 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.520 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.520 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.520 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-28T22:47:51.521 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.521 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.521 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.521 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.521 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.521 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.521 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.522 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": { 2024-03-28T22:47:51.522 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "enabled", 2024-03-28T22:47:51.522 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.522 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.522 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.522 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.522 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.522 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.523 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.523 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.523 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.523 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.523 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.523 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.523 INFO:teuthology.orchestra.run.smithi043.stdout: "interval": { 2024-03-28T22:47:51.523 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "interval", 2024-03-28T22:47:51.523 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.524 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.524 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.524 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "24", 2024-03-28T22:47:51.524 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "8", 2024-03-28T22:47:51.524 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.524 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.524 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.524 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.524 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.525 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.525 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.525 INFO:teuthology.orchestra.run.smithi043.stdout: "last_opt_revision": { 2024-03-28T22:47:51.525 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "last_opt_revision", 2024-03-28T22:47:51.525 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.525 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.525 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.525 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "1", 2024-03-28T22:47:51.526 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.526 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.526 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.526 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.526 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.526 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.526 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.526 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.527 INFO:teuthology.orchestra.run.smithi043.stdout: "leaderboard": { 2024-03-28T22:47:51.527 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "leaderboard", 2024-03-28T22:47:51.527 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.527 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.527 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.527 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.527 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.527 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.527 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.528 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.528 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.528 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.528 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.528 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.528 INFO:teuthology.orchestra.run.smithi043.stdout: "leaderboard_description": { 2024-03-28T22:47:51.528 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "leaderboard_description", 2024-03-28T22:47:51.528 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.528 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.529 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.529 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.529 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.529 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.529 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.529 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.529 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.529 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.530 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.530 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.530 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.530 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.530 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.530 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.530 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.530 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.530 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.531 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.531 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.531 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.531 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.531 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.531 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.531 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.531 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.531 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.532 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.532 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.532 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.532 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.532 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.532 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.532 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.532 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.532 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.533 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.533 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.533 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.533 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.533 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.533 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.533 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.533 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.534 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.534 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.534 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.534 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.534 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.534 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.534 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.534 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.535 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.535 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.535 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.535 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.535 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.535 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.535 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.535 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.535 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.536 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.536 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.536 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.536 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.536 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.536 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.536 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.536 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.536 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.536 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.537 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.537 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.537 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.537 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.537 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.537 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.537 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.537 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.538 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.538 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.538 INFO:teuthology.orchestra.run.smithi043.stdout: "organization": { 2024-03-28T22:47:51.538 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "organization", 2024-03-28T22:47:51.538 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.538 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.538 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.538 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.539 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.539 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.539 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.539 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.539 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.539 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.539 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.539 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.540 INFO:teuthology.orchestra.run.smithi043.stdout: "proxy": { 2024-03-28T22:47:51.540 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "proxy", 2024-03-28T22:47:51.540 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.540 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.540 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.540 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.540 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.540 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.540 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.541 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.541 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.541 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.541 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.541 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.541 INFO:teuthology.orchestra.run.smithi043.stdout: "url": { 2024-03-28T22:47:51.541 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "url", 2024-03-28T22:47:51.541 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.542 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.542 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.542 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-28T22:47:51.542 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.542 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.542 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.542 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.542 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.543 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.543 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.543 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.543 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.543 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.543 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.543 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "test_orchestrator", 2024-03-28T22:47:51.543 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.543 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.544 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.544 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.544 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.544 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.544 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.544 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.544 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.544 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.544 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.545 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.545 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.545 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.545 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.545 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.545 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.545 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.545 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.546 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.546 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.546 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.546 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.546 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.546 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.546 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.546 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.547 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.547 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.547 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.547 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.547 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.547 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.547 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.547 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.547 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.548 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.548 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.548 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.548 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.548 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.548 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.548 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.548 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.549 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.549 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.549 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.549 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.549 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.549 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.549 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.549 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.549 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.550 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.550 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.550 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.550 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.550 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.550 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.550 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.550 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.551 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.551 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.551 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.551 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.551 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.551 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.551 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.551 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.551 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.552 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.552 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.552 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.552 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.552 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.552 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.552 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "volumes", 2024-03-28T22:47:51.552 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.553 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.553 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.553 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.553 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.553 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.553 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.553 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.553 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.553 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.554 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.554 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.554 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.554 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.554 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.554 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.554 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.554 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.555 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.555 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.555 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.555 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.555 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.555 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.555 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.555 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.555 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.556 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.556 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.556 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.556 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.556 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.556 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.556 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.556 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.556 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.557 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.557 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.557 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.557 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.557 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.557 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.557 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.557 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.557 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.558 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.558 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.558 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.558 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.558 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.558 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.558 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.558 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.559 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.559 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.559 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.559 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.559 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.559 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.559 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.559 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.559 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.560 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.560 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.560 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.560 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.560 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.560 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.560 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.560 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.561 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.561 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.561 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.561 INFO:teuthology.orchestra.run.smithi043.stdout: "max_concurrent_clones": { 2024-03-28T22:47:51.561 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "max_concurrent_clones", 2024-03-28T22:47:51.561 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.561 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.561 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.561 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "4", 2024-03-28T22:47:51.562 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.562 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.562 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.562 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-28T22:47:51.562 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.562 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.562 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.562 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.563 INFO:teuthology.orchestra.run.smithi043.stdout: "periodic_async_work": { 2024-03-28T22:47:51.563 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "periodic_async_work", 2024-03-28T22:47:51.563 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.563 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.563 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.563 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.563 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.563 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.563 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.564 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Periodically check for async work", 2024-03-28T22:47:51.564 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.564 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.564 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.564 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.564 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot_clone_delay": { 2024-03-28T22:47:51.564 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "snapshot_clone_delay", 2024-03-28T22:47:51.564 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.564 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.565 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.565 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-03-28T22:47:51.565 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.565 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.565 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.565 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-28T22:47:51.565 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.565 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.565 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.565 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.566 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot_clone_no_wait": { 2024-03-28T22:47:51.566 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "snapshot_clone_no_wait", 2024-03-28T22:47:51.566 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.566 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.566 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.566 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.566 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.566 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.566 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.567 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-28T22:47:51.567 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.567 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.567 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.567 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.567 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.567 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.567 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.568 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "zabbix", 2024-03-28T22:47:51.568 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.568 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.568 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.568 INFO:teuthology.orchestra.run.smithi043.stdout: "discovery_interval": { 2024-03-28T22:47:51.568 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "discovery_interval", 2024-03-28T22:47:51.568 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "uint", 2024-03-28T22:47:51.568 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.568 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.569 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "100", 2024-03-28T22:47:51.569 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.569 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.569 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.569 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.569 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.569 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.569 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.569 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.570 INFO:teuthology.orchestra.run.smithi043.stdout: "identifier": { 2024-03-28T22:47:51.570 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "identifier", 2024-03-28T22:47:51.570 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.570 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.570 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.570 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.570 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.570 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.571 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.571 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.571 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.571 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.571 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.571 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.571 INFO:teuthology.orchestra.run.smithi043.stdout: "interval": { 2024-03-28T22:47:51.571 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "interval", 2024-03-28T22:47:51.571 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:51.572 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.572 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.572 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "60", 2024-03-28T22:47:51.572 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.572 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.572 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.572 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.572 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.572 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.573 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.573 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.573 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.573 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.573 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.573 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.573 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.573 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.573 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.573 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.574 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.574 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.574 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.574 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.574 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.574 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.574 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.574 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.574 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.575 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.575 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.575 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.575 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.575 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.575 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.575 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.575 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.575 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.576 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.576 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.576 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.576 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.576 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.576 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.576 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.576 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.576 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.576 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.577 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.577 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.577 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.577 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.577 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.577 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.577 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.577 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.577 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.577 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.578 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.578 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.578 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.578 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.578 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.578 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.578 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.578 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.578 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.579 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.579 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.579 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.579 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.579 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.579 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.579 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.579 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.579 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.580 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.580 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.580 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.580 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.580 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.580 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.580 INFO:teuthology.orchestra.run.smithi043.stdout: "zabbix_host": { 2024-03-28T22:47:51.580 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "zabbix_host", 2024-03-28T22:47:51.580 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.580 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.581 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.581 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.581 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.581 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.581 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.581 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.581 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.581 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.581 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.581 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.582 INFO:teuthology.orchestra.run.smithi043.stdout: "zabbix_port": { 2024-03-28T22:47:51.582 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "zabbix_port", 2024-03-28T22:47:51.582 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.582 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.582 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.582 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10051", 2024-03-28T22:47:51.582 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.582 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.582 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.583 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.583 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.583 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.583 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.583 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.583 INFO:teuthology.orchestra.run.smithi043.stdout: "zabbix_sender": { 2024-03-28T22:47:51.583 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "zabbix_sender", 2024-03-28T22:47:51.583 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.583 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.584 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.584 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-28T22:47:51.584 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.584 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.584 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.584 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.584 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.584 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.584 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.584 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.585 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.585 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.585 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-28T22:47:51.585 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.585 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.585 INFO:teuthology.orchestra.run.smithi043.stdout: "gid": 25660, 2024-03-28T22:47:51.585 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "z", 2024-03-28T22:47:51.585 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_features": 4540701547738038271, 2024-03-28T22:47:51.585 INFO:teuthology.orchestra.run.smithi043.stdout: "available_modules": [ 2024-03-28T22:47:51.585 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.586 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "alerts", 2024-03-28T22:47:51.586 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.586 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.586 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.586 INFO:teuthology.orchestra.run.smithi043.stdout: "interval": { 2024-03-28T22:47:51.586 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "interval", 2024-03-28T22:47:51.586 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:51.586 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.586 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.587 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "60", 2024-03-28T22:47:51.587 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.587 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.587 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.587 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "How frequently to reexamine health status", 2024-03-28T22:47:51.587 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.587 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.587 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.587 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.587 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.588 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.588 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.588 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.588 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.588 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.588 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.588 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.588 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.588 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.589 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.589 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.589 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.589 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.589 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.589 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.589 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.589 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.590 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.590 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.590 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.590 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.590 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.590 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.590 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.590 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.590 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.591 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.591 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.591 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.591 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.591 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.591 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.591 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.591 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.591 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.592 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.592 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.592 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.592 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.592 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.592 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.592 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.592 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.592 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.593 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.593 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.593 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.593 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.593 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.593 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.593 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.593 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.594 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.594 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.594 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.594 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.594 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.594 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.594 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.594 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.595 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.595 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.595 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.595 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.595 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.595 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.595 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.595 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.596 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.596 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_destination": { 2024-03-28T22:47:51.596 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_destination", 2024-03-28T22:47:51.596 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.596 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.596 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.596 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.596 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.596 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.597 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.597 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Email address to send alerts to", 2024-03-28T22:47:51.597 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.597 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.597 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.597 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.597 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_from_name": { 2024-03-28T22:47:51.597 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_from_name", 2024-03-28T22:47:51.597 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.598 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.598 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.598 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "Ceph", 2024-03-28T22:47:51.598 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.598 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.598 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.598 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Email From: name", 2024-03-28T22:47:51.598 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.599 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.599 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.599 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.599 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_host": { 2024-03-28T22:47:51.599 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_host", 2024-03-28T22:47:51.599 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.599 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.599 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.600 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.600 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.600 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.600 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.600 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "SMTP server", 2024-03-28T22:47:51.600 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.600 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.600 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.601 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.601 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_password": { 2024-03-28T22:47:51.601 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_password", 2024-03-28T22:47:51.601 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.601 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.601 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.601 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.601 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.601 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.602 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.602 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Password to authenticate with", 2024-03-28T22:47:51.602 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.602 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.602 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.602 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.602 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_port": { 2024-03-28T22:47:51.602 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_port", 2024-03-28T22:47:51.603 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.603 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.603 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.603 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "465", 2024-03-28T22:47:51.603 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.603 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.603 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.603 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "SMTP port", 2024-03-28T22:47:51.603 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.603 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.604 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.604 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.604 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_sender": { 2024-03-28T22:47:51.604 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_sender", 2024-03-28T22:47:51.604 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.604 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.604 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.604 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.604 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.605 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.605 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.605 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "SMTP envelope sender", 2024-03-28T22:47:51.605 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.605 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.605 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.605 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.605 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_ssl": { 2024-03-28T22:47:51.605 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_ssl", 2024-03-28T22:47:51.605 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.606 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.606 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.606 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.606 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.606 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.606 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.606 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-28T22:47:51.606 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.607 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.607 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.607 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.607 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_user": { 2024-03-28T22:47:51.607 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_user", 2024-03-28T22:47:51.607 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.607 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.607 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.607 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.608 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.608 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.608 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.608 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "User to authenticate as", 2024-03-28T22:47:51.608 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.608 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.608 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.608 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.608 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.609 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.609 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.609 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "balancer", 2024-03-28T22:47:51.609 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.609 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.609 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.609 INFO:teuthology.orchestra.run.smithi043.stdout: "active": { 2024-03-28T22:47:51.609 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "active", 2024-03-28T22:47:51.609 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.610 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.610 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.610 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.610 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.610 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.610 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.610 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "automatically balance PGs across cluster", 2024-03-28T22:47:51.610 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.611 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.611 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.611 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.611 INFO:teuthology.orchestra.run.smithi043.stdout: "begin_time": { 2024-03-28T22:47:51.611 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "begin_time", 2024-03-28T22:47:51.611 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.611 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.611 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.611 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0000", 2024-03-28T22:47:51.612 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.612 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.612 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.612 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "beginning time of day to automatically balance", 2024-03-28T22:47:51.612 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T22:47:51.612 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.612 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.612 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.612 INFO:teuthology.orchestra.run.smithi043.stdout: "begin_weekday": { 2024-03-28T22:47:51.613 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "begin_weekday", 2024-03-28T22:47:51.613 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "uint", 2024-03-28T22:47:51.613 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.613 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.613 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-03-28T22:47:51.613 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "0", 2024-03-28T22:47:51.613 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "6", 2024-03-28T22:47:51.613 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.613 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-28T22:47:51.614 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T22:47:51.614 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.614 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.614 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.614 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_compat_max_iterations": { 2024-03-28T22:47:51.614 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "crush_compat_max_iterations", 2024-03-28T22:47:51.614 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "uint", 2024-03-28T22:47:51.614 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.614 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.615 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "25", 2024-03-28T22:47:51.615 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "1", 2024-03-28T22:47:51.615 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "250", 2024-03-28T22:47:51.615 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.615 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-28T22:47:51.615 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.615 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.615 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.615 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.616 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_compat_metrics": { 2024-03-28T22:47:51.616 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "crush_compat_metrics", 2024-03-28T22:47:51.616 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.616 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.616 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.616 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "pgs,objects,bytes", 2024-03-28T22:47:51.616 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.616 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.616 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.617 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-28T22:47:51.617 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-28T22:47:51.617 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.617 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.617 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.617 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_compat_step": { 2024-03-28T22:47:51.617 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "crush_compat_step", 2024-03-28T22:47:51.617 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-03-28T22:47:51.617 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.618 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.618 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0.5", 2024-03-28T22:47:51.618 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "0.001", 2024-03-28T22:47:51.618 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "0.999", 2024-03-28T22:47:51.618 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.618 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "aggressiveness of optimization", 2024-03-28T22:47:51.618 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-28T22:47:51.618 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.619 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.619 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.619 INFO:teuthology.orchestra.run.smithi043.stdout: "end_time": { 2024-03-28T22:47:51.619 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "end_time", 2024-03-28T22:47:51.619 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.619 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.619 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.619 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "2359", 2024-03-28T22:47:51.619 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.620 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.620 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.620 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "ending time of day to automatically balance", 2024-03-28T22:47:51.620 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T22:47:51.620 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.620 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.620 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.620 INFO:teuthology.orchestra.run.smithi043.stdout: "end_weekday": { 2024-03-28T22:47:51.621 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "end_weekday", 2024-03-28T22:47:51.621 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "uint", 2024-03-28T22:47:51.621 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.621 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.621 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-03-28T22:47:51.621 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "0", 2024-03-28T22:47:51.621 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "6", 2024-03-28T22:47:51.621 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.621 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-28T22:47:51.622 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T22:47:51.622 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.622 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.622 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.622 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.622 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.622 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.622 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.623 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.623 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.623 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.623 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.623 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.623 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.623 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.623 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.623 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.624 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.624 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.624 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.624 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.624 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.624 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.624 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.624 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.624 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.624 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.625 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.625 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.625 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.625 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.625 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.625 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.625 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.625 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.625 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.626 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.626 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.626 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.626 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.626 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.626 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.626 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.626 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.626 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.627 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.627 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.627 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.627 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.627 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.627 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.627 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.627 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.627 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.627 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.628 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.628 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.628 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.628 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.628 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.628 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.628 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.628 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.629 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.629 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.629 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.629 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.629 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.629 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.629 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.629 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.629 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.630 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.630 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.630 INFO:teuthology.orchestra.run.smithi043.stdout: "min_score": { 2024-03-28T22:47:51.630 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "min_score", 2024-03-28T22:47:51.630 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-03-28T22:47:51.630 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.630 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.630 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-03-28T22:47:51.631 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.631 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.631 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.631 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-28T22:47:51.631 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.631 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.631 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.631 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.631 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": { 2024-03-28T22:47:51.631 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mode", 2024-03-28T22:47:51.632 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.632 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.632 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.632 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "upmap", 2024-03-28T22:47:51.632 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.632 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.632 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.632 INFO:teuthology.orchestra.run.smithi043.stdout: "crush-compat", 2024-03-28T22:47:51.632 INFO:teuthology.orchestra.run.smithi043.stdout: "none", 2024-03-28T22:47:51.632 INFO:teuthology.orchestra.run.smithi043.stdout: "read", 2024-03-28T22:47:51.633 INFO:teuthology.orchestra.run.smithi043.stdout: "upmap", 2024-03-28T22:47:51.633 INFO:teuthology.orchestra.run.smithi043.stdout: "upmap-read" 2024-03-28T22:47:51.633 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.633 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Balancer mode", 2024-03-28T22:47:51.633 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.633 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.633 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.633 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.633 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_ids": { 2024-03-28T22:47:51.634 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "pool_ids", 2024-03-28T22:47:51.634 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.634 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.634 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.634 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.634 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.634 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.634 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.635 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-28T22:47:51.635 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.635 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.635 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.635 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.635 INFO:teuthology.orchestra.run.smithi043.stdout: "sleep_interval": { 2024-03-28T22:47:51.635 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sleep_interval", 2024-03-28T22:47:51.635 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:51.635 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.635 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.636 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "60", 2024-03-28T22:47:51.636 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.636 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.636 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.636 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-28T22:47:51.636 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.636 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.636 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.636 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.637 INFO:teuthology.orchestra.run.smithi043.stdout: "upmap_max_deviation": { 2024-03-28T22:47:51.637 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "upmap_max_deviation", 2024-03-28T22:47:51.637 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.637 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.637 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.637 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "5", 2024-03-28T22:47:51.637 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "1", 2024-03-28T22:47:51.637 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.637 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.637 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-28T22:47:51.638 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-28T22:47:51.638 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.638 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.638 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.638 INFO:teuthology.orchestra.run.smithi043.stdout: "upmap_max_optimizations": { 2024-03-28T22:47:51.638 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "upmap_max_optimizations", 2024-03-28T22:47:51.638 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "uint", 2024-03-28T22:47:51.638 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.639 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.639 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-03-28T22:47:51.639 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.639 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.639 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.639 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-28T22:47:51.639 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.639 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.639 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.639 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.640 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.640 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.640 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.640 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cephadm", 2024-03-28T22:47:51.640 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.640 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.640 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.640 INFO:teuthology.orchestra.run.smithi043.stdout: "agent_down_multiplier": { 2024-03-28T22:47:51.640 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "agent_down_multiplier", 2024-03-28T22:47:51.641 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-03-28T22:47:51.641 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.641 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.641 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "3.0", 2024-03-28T22:47:51.641 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.641 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.641 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.641 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-28T22:47:51.641 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.642 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.642 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.642 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.642 INFO:teuthology.orchestra.run.smithi043.stdout: "agent_refresh_rate": { 2024-03-28T22:47:51.642 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "agent_refresh_rate", 2024-03-28T22:47:51.642 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:51.642 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.642 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.642 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "20", 2024-03-28T22:47:51.643 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.643 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.643 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.643 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-28T22:47:51.643 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.643 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.643 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.643 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.643 INFO:teuthology.orchestra.run.smithi043.stdout: "agent_starting_port": { 2024-03-28T22:47:51.643 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "agent_starting_port", 2024-03-28T22:47:51.644 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.644 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.644 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.644 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "4721", 2024-03-28T22:47:51.644 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.644 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.644 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.644 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-28T22:47:51.644 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.645 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.645 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.645 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.645 INFO:teuthology.orchestra.run.smithi043.stdout: "allow_ptrace": { 2024-03-28T22:47:51.645 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "allow_ptrace", 2024-03-28T22:47:51.645 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.645 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.645 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.645 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.646 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.646 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.646 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.646 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-28T22:47:51.646 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-28T22:47:51.646 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.646 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.646 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.647 INFO:teuthology.orchestra.run.smithi043.stdout: "autotune_interval": { 2024-03-28T22:47:51.647 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "autotune_interval", 2024-03-28T22:47:51.647 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:51.647 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.647 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.647 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "600", 2024-03-28T22:47:51.647 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.647 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.647 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.647 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-28T22:47:51.648 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.648 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.648 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.648 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.648 INFO:teuthology.orchestra.run.smithi043.stdout: "autotune_memory_target_ratio": { 2024-03-28T22:47:51.648 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "autotune_memory_target_ratio", 2024-03-28T22:47:51.648 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-03-28T22:47:51.648 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.648 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.649 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0.7", 2024-03-28T22:47:51.649 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.649 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.649 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.649 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-28T22:47:51.649 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.649 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.649 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.649 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.650 INFO:teuthology.orchestra.run.smithi043.stdout: "cephadm_log_destination": { 2024-03-28T22:47:51.650 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cephadm_log_destination", 2024-03-28T22:47:51.650 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.650 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.650 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.650 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.650 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.650 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.651 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.651 INFO:teuthology.orchestra.run.smithi043.stdout: "file", 2024-03-28T22:47:51.651 INFO:teuthology.orchestra.run.smithi043.stdout: "file,syslog", 2024-03-28T22:47:51.651 INFO:teuthology.orchestra.run.smithi043.stdout: "syslog" 2024-03-28T22:47:51.651 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.651 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-28T22:47:51.651 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.651 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.651 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.652 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.652 INFO:teuthology.orchestra.run.smithi043.stdout: "cgroups_split": { 2024-03-28T22:47:51.652 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cgroups_split", 2024-03-28T22:47:51.652 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.652 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.652 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.652 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.652 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.652 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.652 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.653 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-28T22:47:51.653 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.653 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.653 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.653 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.653 INFO:teuthology.orchestra.run.smithi043.stdout: "config_checks_enabled": { 2024-03-28T22:47:51.653 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "config_checks_enabled", 2024-03-28T22:47:51.653 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.654 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.654 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.654 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.654 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.654 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.654 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.654 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-28T22:47:51.654 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.655 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.655 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.655 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.655 INFO:teuthology.orchestra.run.smithi043.stdout: "config_dashboard": { 2024-03-28T22:47:51.655 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "config_dashboard", 2024-03-28T22:47:51.655 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.655 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.655 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.655 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.656 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.656 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.656 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.656 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-28T22:47:51.656 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.656 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.656 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.656 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.657 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_alertmanager": { 2024-03-28T22:47:51.657 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_alertmanager", 2024-03-28T22:47:51.657 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.657 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.657 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.657 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-28T22:47:51.657 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.657 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.657 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.658 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Prometheus container image", 2024-03-28T22:47:51.658 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.658 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.658 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.658 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.658 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_base": { 2024-03-28T22:47:51.658 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_base", 2024-03-28T22:47:51.658 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.659 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.659 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.659 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-28T22:47:51.659 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.659 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.659 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.659 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Container image name, without the tag", 2024-03-28T22:47:51.659 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.659 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.660 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.660 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.660 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_elasticsearch": { 2024-03-28T22:47:51.660 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_elasticsearch", 2024-03-28T22:47:51.660 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.660 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.660 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.660 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-28T22:47:51.660 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.661 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.661 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.661 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "elasticsearch container image", 2024-03-28T22:47:51.661 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.661 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.661 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.661 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.661 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_grafana": { 2024-03-28T22:47:51.662 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_grafana", 2024-03-28T22:47:51.662 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.662 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.662 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.662 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-28T22:47:51.662 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.662 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.662 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.663 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Prometheus container image", 2024-03-28T22:47:51.663 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.663 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.663 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.663 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.663 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_haproxy": { 2024-03-28T22:47:51.663 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_haproxy", 2024-03-28T22:47:51.663 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.663 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.664 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.664 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-28T22:47:51.664 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.664 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.664 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.664 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "HAproxy container image", 2024-03-28T22:47:51.664 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.664 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.664 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.664 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.665 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_jaeger_agent": { 2024-03-28T22:47:51.665 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_jaeger_agent", 2024-03-28T22:47:51.665 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.665 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.665 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.665 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-28T22:47:51.665 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.665 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.665 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.665 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Jaeger agent container image", 2024-03-28T22:47:51.666 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.666 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.666 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.666 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.666 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_jaeger_collector": { 2024-03-28T22:47:51.666 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_jaeger_collector", 2024-03-28T22:47:51.666 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.666 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.666 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.667 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-28T22:47:51.667 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.667 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.667 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.667 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Jaeger collector container image", 2024-03-28T22:47:51.667 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.667 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.667 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.667 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.667 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_jaeger_query": { 2024-03-28T22:47:51.668 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_jaeger_query", 2024-03-28T22:47:51.668 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.668 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.668 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.668 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-28T22:47:51.668 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.668 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.668 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.668 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Jaeger query container image", 2024-03-28T22:47:51.669 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.669 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.669 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.669 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.669 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_keepalived": { 2024-03-28T22:47:51.669 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_keepalived", 2024-03-28T22:47:51.669 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.669 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.669 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.669 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-28T22:47:51.670 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.670 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.670 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.670 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Keepalived container image", 2024-03-28T22:47:51.670 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.670 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.670 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.670 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.670 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_loki": { 2024-03-28T22:47:51.671 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_loki", 2024-03-28T22:47:51.671 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.671 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.671 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.671 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-28T22:47:51.671 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.671 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.671 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.671 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Loki container image", 2024-03-28T22:47:51.672 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.672 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.672 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.672 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.672 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_node_exporter": { 2024-03-28T22:47:51.672 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_node_exporter", 2024-03-28T22:47:51.672 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.672 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.672 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.672 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-28T22:47:51.673 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.673 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.673 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.673 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Prometheus container image", 2024-03-28T22:47:51.673 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.673 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.673 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.673 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.673 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_nvmeof": { 2024-03-28T22:47:51.674 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_nvmeof", 2024-03-28T22:47:51.674 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.674 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.674 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.674 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-28T22:47:51.674 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.674 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.674 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.674 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Nvme-of container image", 2024-03-28T22:47:51.674 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.675 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.675 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.675 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.675 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_prometheus": { 2024-03-28T22:47:51.675 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_prometheus", 2024-03-28T22:47:51.675 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.675 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.675 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.675 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-28T22:47:51.676 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.676 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.676 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.676 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Prometheus container image", 2024-03-28T22:47:51.676 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.676 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.676 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.676 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.676 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_promtail": { 2024-03-28T22:47:51.676 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_promtail", 2024-03-28T22:47:51.677 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.677 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.677 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.677 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-28T22:47:51.677 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.677 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.677 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.677 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Promtail container image", 2024-03-28T22:47:51.677 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.677 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.678 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.678 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.678 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_snmp_gateway": { 2024-03-28T22:47:51.678 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_snmp_gateway", 2024-03-28T22:47:51.678 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.678 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.678 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.678 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-28T22:47:51.679 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.679 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.679 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.679 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "SNMP Gateway container image", 2024-03-28T22:47:51.679 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.679 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.679 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.679 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.679 INFO:teuthology.orchestra.run.smithi043.stdout: "container_init": { 2024-03-28T22:47:51.680 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_init", 2024-03-28T22:47:51.680 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.680 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.680 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.680 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.680 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.680 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.680 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.681 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Run podman/docker with `--init`", 2024-03-28T22:47:51.681 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.681 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.681 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.681 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.681 INFO:teuthology.orchestra.run.smithi043.stdout: "daemon_cache_timeout": { 2024-03-28T22:47:51.681 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "daemon_cache_timeout", 2024-03-28T22:47:51.681 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:51.681 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.681 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.682 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "600", 2024-03-28T22:47:51.682 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.682 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.682 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.682 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-28T22:47:51.682 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.682 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.682 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.683 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.683 INFO:teuthology.orchestra.run.smithi043.stdout: "default_cephadm_command_timeout": { 2024-03-28T22:47:51.683 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default_cephadm_command_timeout", 2024-03-28T22:47:51.683 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.683 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.683 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.683 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "900", 2024-03-28T22:47:51.683 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.683 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.684 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.684 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-28T22:47:51.684 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.684 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.684 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.684 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.684 INFO:teuthology.orchestra.run.smithi043.stdout: "default_registry": { 2024-03-28T22:47:51.684 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default_registry", 2024-03-28T22:47:51.684 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.685 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.685 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.685 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "docker.io", 2024-03-28T22:47:51.685 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.685 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.685 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.685 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-28T22:47:51.685 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.685 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.686 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.686 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.686 INFO:teuthology.orchestra.run.smithi043.stdout: "device_cache_timeout": { 2024-03-28T22:47:51.686 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "device_cache_timeout", 2024-03-28T22:47:51.686 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:51.686 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.686 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.686 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "1800", 2024-03-28T22:47:51.687 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.687 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.687 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.687 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "seconds to cache device inventory", 2024-03-28T22:47:51.687 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.687 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.687 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.687 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.687 INFO:teuthology.orchestra.run.smithi043.stdout: "device_enhanced_scan": { 2024-03-28T22:47:51.688 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "device_enhanced_scan", 2024-03-28T22:47:51.688 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.688 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.688 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.688 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.688 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.688 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.688 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.689 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-28T22:47:51.689 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.689 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.689 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.689 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.689 INFO:teuthology.orchestra.run.smithi043.stdout: "facts_cache_timeout": { 2024-03-28T22:47:51.689 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "facts_cache_timeout", 2024-03-28T22:47:51.689 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:51.690 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.690 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.690 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "60", 2024-03-28T22:47:51.690 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.690 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.690 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.690 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "seconds to cache host facts data", 2024-03-28T22:47:51.690 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.690 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.691 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.691 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.691 INFO:teuthology.orchestra.run.smithi043.stdout: "host_check_interval": { 2024-03-28T22:47:51.691 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "host_check_interval", 2024-03-28T22:47:51.691 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:51.691 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.691 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.691 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "600", 2024-03-28T22:47:51.691 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.692 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.692 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.692 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how frequently to perform a host check", 2024-03-28T22:47:51.692 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.692 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.692 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.692 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.692 INFO:teuthology.orchestra.run.smithi043.stdout: "hw_monitoring": { 2024-03-28T22:47:51.692 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "hw_monitoring", 2024-03-28T22:47:51.693 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.693 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.693 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.693 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.693 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.693 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.693 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.693 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-28T22:47:51.693 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.694 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.694 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.694 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.694 INFO:teuthology.orchestra.run.smithi043.stdout: "inventory_list_all": { 2024-03-28T22:47:51.694 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "inventory_list_all", 2024-03-28T22:47:51.694 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.694 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.694 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.695 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.695 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.695 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.695 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.695 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-28T22:47:51.695 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.695 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.695 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.695 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.696 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.696 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.696 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.696 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.696 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.696 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.696 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.696 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.696 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.697 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.697 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.697 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.697 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.697 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.697 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.697 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.697 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.697 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.697 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.698 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.698 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.698 INFO:teuthology.orchestra.run.smithi043.stdout: "log_refresh_metadata": { 2024-03-28T22:47:51.698 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_refresh_metadata", 2024-03-28T22:47:51.698 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.698 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.698 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.698 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.698 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.699 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.699 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.699 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-28T22:47:51.699 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.699 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.699 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.699 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.699 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.699 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.700 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.700 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.700 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.700 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.700 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.700 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.700 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.700 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-28T22:47:51.700 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.700 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.701 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.701 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.701 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.701 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.701 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.701 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.701 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.701 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.701 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.701 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.702 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.702 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.702 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.702 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.702 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.702 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.702 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.702 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.702 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.703 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.703 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.703 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.703 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.703 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.703 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.703 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.703 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.703 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.703 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.704 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.704 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.704 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.704 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.704 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.704 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.704 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.704 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.704 INFO:teuthology.orchestra.run.smithi043.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-28T22:47:51.704 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-28T22:47:51.705 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.705 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.705 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.705 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.705 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.705 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.705 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.705 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-28T22:47:51.705 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.705 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.706 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.706 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.706 INFO:teuthology.orchestra.run.smithi043.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-28T22:47:51.706 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-28T22:47:51.706 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.706 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.706 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.706 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "*", 2024-03-28T22:47:51.706 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.707 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.707 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.707 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-28T22:47:51.707 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.707 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.707 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.707 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.707 INFO:teuthology.orchestra.run.smithi043.stdout: "max_count_per_host": { 2024-03-28T22:47:51.707 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "max_count_per_host", 2024-03-28T22:47:51.707 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.708 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.708 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.708 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-03-28T22:47:51.708 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.708 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.708 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.708 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "max number of daemons per service per host", 2024-03-28T22:47:51.708 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.708 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.708 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.709 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.709 INFO:teuthology.orchestra.run.smithi043.stdout: "max_osd_draining_count": { 2024-03-28T22:47:51.709 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "max_osd_draining_count", 2024-03-28T22:47:51.709 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.709 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.709 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.709 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-03-28T22:47:51.709 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.709 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.710 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.710 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-28T22:47:51.710 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.710 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.710 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.710 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.710 INFO:teuthology.orchestra.run.smithi043.stdout: "migration_current": { 2024-03-28T22:47:51.710 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "migration_current", 2024-03-28T22:47:51.710 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.711 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.711 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.711 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.711 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.711 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.711 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.711 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "internal - do not modify", 2024-03-28T22:47:51.711 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.711 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.711 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.712 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.712 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": { 2024-03-28T22:47:51.712 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mode", 2024-03-28T22:47:51.712 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.712 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.712 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.712 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "root", 2024-03-28T22:47:51.712 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.712 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.713 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.713 INFO:teuthology.orchestra.run.smithi043.stdout: "cephadm-package", 2024-03-28T22:47:51.713 INFO:teuthology.orchestra.run.smithi043.stdout: "root" 2024-03-28T22:47:51.713 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.713 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "mode for remote execution of cephadm", 2024-03-28T22:47:51.713 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.713 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.713 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.713 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.713 INFO:teuthology.orchestra.run.smithi043.stdout: "oob_default_addr": { 2024-03-28T22:47:51.714 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "oob_default_addr", 2024-03-28T22:47:51.714 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.714 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.714 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.714 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "169.254.1.1", 2024-03-28T22:47:51.714 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.714 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.714 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.714 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-28T22:47:51.714 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.715 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.715 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.715 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.715 INFO:teuthology.orchestra.run.smithi043.stdout: "prometheus_alerts_path": { 2024-03-28T22:47:51.715 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "prometheus_alerts_path", 2024-03-28T22:47:51.715 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.715 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.715 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.715 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-28T22:47:51.716 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.716 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.716 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.716 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-28T22:47:51.716 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.716 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.716 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.716 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.716 INFO:teuthology.orchestra.run.smithi043.stdout: "registry_insecure": { 2024-03-28T22:47:51.716 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "registry_insecure", 2024-03-28T22:47:51.717 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.717 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.717 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.717 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.717 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.717 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.717 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.717 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-28T22:47:51.717 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.717 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.718 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.718 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.718 INFO:teuthology.orchestra.run.smithi043.stdout: "registry_password": { 2024-03-28T22:47:51.718 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "registry_password", 2024-03-28T22:47:51.718 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.718 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.718 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.718 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.719 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.719 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.719 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.719 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-28T22:47:51.719 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.719 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.719 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.719 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.719 INFO:teuthology.orchestra.run.smithi043.stdout: "registry_url": { 2024-03-28T22:47:51.720 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "registry_url", 2024-03-28T22:47:51.720 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.720 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.720 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.720 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.720 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.720 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.720 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.720 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-28T22:47:51.720 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.721 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.721 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.721 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.721 INFO:teuthology.orchestra.run.smithi043.stdout: "registry_username": { 2024-03-28T22:47:51.721 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "registry_username", 2024-03-28T22:47:51.721 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.721 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.721 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.721 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.721 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.722 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.722 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.722 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-28T22:47:51.722 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.722 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.722 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.722 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.722 INFO:teuthology.orchestra.run.smithi043.stdout: "secure_monitoring_stack": { 2024-03-28T22:47:51.722 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "secure_monitoring_stack", 2024-03-28T22:47:51.722 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.723 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.723 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.723 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.723 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.723 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.723 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.723 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-28T22:47:51.723 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.723 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.724 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.724 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.724 INFO:teuthology.orchestra.run.smithi043.stdout: "service_discovery_port": { 2024-03-28T22:47:51.724 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "service_discovery_port", 2024-03-28T22:47:51.724 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.724 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.724 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.724 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "8765", 2024-03-28T22:47:51.724 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.725 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.725 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.725 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "cephadm service discovery port", 2024-03-28T22:47:51.725 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.725 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.725 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.725 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.725 INFO:teuthology.orchestra.run.smithi043.stdout: "ssh_config_file": { 2024-03-28T22:47:51.725 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ssh_config_file", 2024-03-28T22:47:51.726 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.726 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.726 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.726 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.726 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.726 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.726 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.726 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-28T22:47:51.726 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.726 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.727 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.727 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.727 INFO:teuthology.orchestra.run.smithi043.stdout: "use_agent": { 2024-03-28T22:47:51.727 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "use_agent", 2024-03-28T22:47:51.727 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.727 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.727 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.727 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.727 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.728 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.728 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.728 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-28T22:47:51.728 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.728 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.728 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.728 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.728 INFO:teuthology.orchestra.run.smithi043.stdout: "use_repo_digest": { 2024-03-28T22:47:51.728 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "use_repo_digest", 2024-03-28T22:47:51.729 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.729 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.729 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.729 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.729 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.729 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.729 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.729 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-28T22:47:51.729 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.729 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.730 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.730 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.730 INFO:teuthology.orchestra.run.smithi043.stdout: "warn_on_failed_host_check": { 2024-03-28T22:47:51.730 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "warn_on_failed_host_check", 2024-03-28T22:47:51.730 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.730 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.730 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.730 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.731 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.731 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.731 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.731 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "raise a health warning if the host check fails", 2024-03-28T22:47:51.731 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.731 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.731 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.731 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.731 INFO:teuthology.orchestra.run.smithi043.stdout: "warn_on_stray_daemons": { 2024-03-28T22:47:51.732 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "warn_on_stray_daemons", 2024-03-28T22:47:51.732 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.732 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.732 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.732 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.732 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.732 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.732 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.732 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-28T22:47:51.733 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.733 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.733 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.733 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.733 INFO:teuthology.orchestra.run.smithi043.stdout: "warn_on_stray_hosts": { 2024-03-28T22:47:51.733 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "warn_on_stray_hosts", 2024-03-28T22:47:51.733 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.733 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.733 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.734 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.734 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.734 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.734 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.734 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-28T22:47:51.734 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.734 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.734 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.735 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.735 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.735 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.735 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.735 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "crash", 2024-03-28T22:47:51.735 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.735 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.735 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.735 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.736 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.736 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.736 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.736 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.736 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.736 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.736 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.736 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.736 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.736 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.737 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.737 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.737 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.737 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.737 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.737 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.737 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.737 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.737 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.738 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.738 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.738 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.738 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.738 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.738 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.738 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.738 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.738 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.739 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.739 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.739 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.739 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.739 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.739 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.739 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.739 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.739 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.739 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.740 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.740 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.740 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.740 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.740 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.740 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.740 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.740 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.740 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.741 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.741 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.741 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.741 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.741 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.741 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.741 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.741 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.741 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.741 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.742 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.742 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.742 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.742 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.742 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.742 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.742 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.742 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.742 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.742 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.743 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.743 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.743 INFO:teuthology.orchestra.run.smithi043.stdout: "retain_interval": { 2024-03-28T22:47:51.743 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "retain_interval", 2024-03-28T22:47:51.743 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:51.743 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.743 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.743 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "31536000", 2024-03-28T22:47:51.743 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.744 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.744 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.744 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-28T22:47:51.744 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.744 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.744 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.744 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.744 INFO:teuthology.orchestra.run.smithi043.stdout: "warn_recent_interval": { 2024-03-28T22:47:51.744 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "warn_recent_interval", 2024-03-28T22:47:51.745 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:51.745 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.745 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.745 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "1209600", 2024-03-28T22:47:51.745 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.745 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.745 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.745 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-28T22:47:51.745 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.745 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.746 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.746 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.746 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.746 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.746 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.746 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "dashboard", 2024-03-28T22:47:51.746 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.746 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.746 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.747 INFO:teuthology.orchestra.run.smithi043.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-28T22:47:51.747 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-28T22:47:51.747 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.747 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.747 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.747 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-03-28T22:47:51.747 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.747 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.747 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.747 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.748 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.748 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.748 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.748 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.748 INFO:teuthology.orchestra.run.smithi043.stdout: "ALERTMANAGER_API_HOST": { 2024-03-28T22:47:51.748 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-28T22:47:51.748 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.748 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.748 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.748 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.749 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.749 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.749 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.749 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.749 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.749 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.749 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.749 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.750 INFO:teuthology.orchestra.run.smithi043.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-28T22:47:51.750 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-28T22:47:51.750 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.750 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.750 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.750 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.750 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.750 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.750 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.750 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.751 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.751 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.751 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.751 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.751 INFO:teuthology.orchestra.run.smithi043.stdout: "AUDIT_API_ENABLED": { 2024-03-28T22:47:51.751 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "AUDIT_API_ENABLED", 2024-03-28T22:47:51.751 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.751 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.751 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.751 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.752 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.752 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.752 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.752 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.752 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.752 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.752 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.752 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.753 INFO:teuthology.orchestra.run.smithi043.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-28T22:47:51.753 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-28T22:47:51.753 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.753 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.753 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.753 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.753 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.753 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.753 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.753 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.754 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.754 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.754 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.754 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.754 INFO:teuthology.orchestra.run.smithi043.stdout: "ENABLE_BROWSABLE_API": { 2024-03-28T22:47:51.754 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-28T22:47:51.754 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.754 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.754 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.755 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.755 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.755 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.755 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.755 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.755 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.755 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.755 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.755 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.756 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-28T22:47:51.756 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-28T22:47:51.756 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.756 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.756 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.756 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.756 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.756 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.756 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.757 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.757 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.757 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.757 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.757 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.757 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-28T22:47:51.757 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-28T22:47:51.757 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.757 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.757 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.758 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.758 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.758 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.758 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.758 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.758 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.758 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.758 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.758 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.758 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-28T22:47:51.759 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-28T22:47:51.759 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.759 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.759 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.759 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.759 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.759 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.759 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.759 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.760 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.760 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.760 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.760 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.760 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-28T22:47:51.760 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-28T22:47:51.760 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.760 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.760 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.761 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.761 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.761 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.761 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.761 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.761 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.761 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.761 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.762 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-28T22:47:51.762 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-28T22:47:51.762 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.762 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.762 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.762 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.762 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.762 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.762 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.763 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.763 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.763 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.763 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.763 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.763 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-28T22:47:51.763 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-28T22:47:51.763 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.764 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.764 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.764 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.764 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.764 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.764 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.764 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.764 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.764 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.764 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.765 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.765 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-28T22:47:51.765 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-28T22:47:51.765 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.765 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.765 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.765 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.765 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.765 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.766 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.766 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.766 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.766 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.766 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.766 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.766 INFO:teuthology.orchestra.run.smithi043.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-28T22:47:51.766 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-28T22:47:51.766 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.767 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.767 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.767 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.767 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.767 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.767 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.767 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.767 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.767 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.767 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.768 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.768 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_API_PASSWORD": { 2024-03-28T22:47:51.768 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-28T22:47:51.768 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.768 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.768 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.768 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "admin", 2024-03-28T22:47:51.768 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.768 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.768 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.769 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.769 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.769 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.769 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.769 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.769 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-28T22:47:51.769 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-28T22:47:51.769 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.769 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.770 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.770 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.770 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.770 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.770 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.770 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.770 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.770 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.770 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.771 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.771 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_API_URL": { 2024-03-28T22:47:51.771 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_API_URL", 2024-03-28T22:47:51.771 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.771 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.771 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.771 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.771 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.772 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.772 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.772 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.772 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.772 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.772 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.772 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.772 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_API_USERNAME": { 2024-03-28T22:47:51.772 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-28T22:47:51.772 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.773 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.773 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.773 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "admin", 2024-03-28T22:47:51.773 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.773 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.773 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.773 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.773 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.773 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.774 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.774 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.774 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-28T22:47:51.774 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-28T22:47:51.774 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.774 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.774 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.774 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.774 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.774 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.775 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.775 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.775 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.775 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.775 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.775 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.775 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-28T22:47:51.775 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-28T22:47:51.775 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.776 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.776 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.776 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.776 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.776 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.776 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.776 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.776 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.776 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.776 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.777 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.777 INFO:teuthology.orchestra.run.smithi043.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-28T22:47:51.777 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-28T22:47:51.777 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.777 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.777 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.777 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.777 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.777 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.778 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.778 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.778 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.778 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.778 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.778 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.778 INFO:teuthology.orchestra.run.smithi043.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-28T22:47:51.778 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-28T22:47:51.778 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.779 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.779 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.779 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.779 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.779 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.779 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.779 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.779 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.779 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.780 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.780 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.780 INFO:teuthology.orchestra.run.smithi043.stdout: "PROMETHEUS_API_HOST": { 2024-03-28T22:47:51.780 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-28T22:47:51.780 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.780 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.780 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.780 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.780 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.781 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.781 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.781 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.781 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.781 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.781 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.781 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.781 INFO:teuthology.orchestra.run.smithi043.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-28T22:47:51.781 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-28T22:47:51.781 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.782 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.782 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.782 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.782 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.782 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.782 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.782 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.782 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.782 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.783 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.783 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.783 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-28T22:47:51.783 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-28T22:47:51.783 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.783 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.783 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.783 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.783 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.784 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.784 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.784 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.784 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.784 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.784 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.784 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.784 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-28T22:47:51.784 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-28T22:47:51.785 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.785 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.785 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.785 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.785 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.785 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.785 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.785 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.786 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.786 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.786 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.786 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.786 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-28T22:47:51.786 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-28T22:47:51.786 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.786 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.786 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.787 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.787 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.787 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.787 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.787 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.787 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.787 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.787 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.788 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.788 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-28T22:47:51.788 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-28T22:47:51.788 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.788 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.788 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.788 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.788 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.788 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.788 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.789 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.789 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.789 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.789 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.789 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.789 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-28T22:47:51.789 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-28T22:47:51.789 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.790 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.790 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.790 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.790 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.790 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.790 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.790 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.790 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.791 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.791 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.791 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.791 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-28T22:47:51.791 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-28T22:47:51.791 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.791 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.791 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.792 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.792 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.792 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.792 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.792 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.792 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.792 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.792 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.792 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.793 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-28T22:47:51.793 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-28T22:47:51.793 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.793 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.793 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.793 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.793 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.793 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.793 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.794 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.794 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.794 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.794 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.794 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.794 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_ENABLED": { 2024-03-28T22:47:51.794 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-28T22:47:51.794 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.794 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.795 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.795 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.795 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.795 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.795 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.795 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.795 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.795 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.795 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.796 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.796 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-28T22:47:51.796 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-28T22:47:51.796 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.796 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.796 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.796 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-28T22:47:51.796 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.796 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.797 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.797 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.797 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.797 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.797 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.797 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.797 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-28T22:47:51.797 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-28T22:47:51.797 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.798 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.798 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.798 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-03-28T22:47:51.798 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.798 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.798 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.798 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.798 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.799 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.799 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.799 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.799 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-28T22:47:51.799 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-28T22:47:51.799 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.799 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.799 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.799 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "8", 2024-03-28T22:47:51.800 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.800 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.800 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.800 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.800 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.800 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.800 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.800 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.800 INFO:teuthology.orchestra.run.smithi043.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-28T22:47:51.800 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-28T22:47:51.801 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.801 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.801 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.801 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "45", 2024-03-28T22:47:51.801 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.801 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.801 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.801 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.801 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.802 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.802 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.802 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.802 INFO:teuthology.orchestra.run.smithi043.stdout: "RGW_API_ACCESS_KEY": { 2024-03-28T22:47:51.802 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-28T22:47:51.802 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.802 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.802 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.802 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.803 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.803 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.803 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.803 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.803 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.803 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.803 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.803 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.803 INFO:teuthology.orchestra.run.smithi043.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-28T22:47:51.804 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-28T22:47:51.804 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.804 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.804 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.804 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "admin", 2024-03-28T22:47:51.804 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.804 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.804 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.804 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.805 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.805 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.805 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.805 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.805 INFO:teuthology.orchestra.run.smithi043.stdout: "RGW_API_SECRET_KEY": { 2024-03-28T22:47:51.805 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-28T22:47:51.805 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.805 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.805 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.805 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.806 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.806 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.806 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.806 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.806 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.806 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.806 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.806 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.806 INFO:teuthology.orchestra.run.smithi043.stdout: "RGW_API_SSL_VERIFY": { 2024-03-28T22:47:51.807 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-28T22:47:51.807 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.807 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.807 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.807 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.807 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.807 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.807 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.807 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.807 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.808 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.808 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.808 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.808 INFO:teuthology.orchestra.run.smithi043.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-28T22:47:51.808 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-28T22:47:51.808 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.808 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.808 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.808 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-03-28T22:47:51.809 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.809 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.809 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.809 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.809 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.809 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.809 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.809 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.809 INFO:teuthology.orchestra.run.smithi043.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-28T22:47:51.809 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-28T22:47:51.810 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.810 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.810 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.810 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-03-28T22:47:51.810 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.810 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.810 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.810 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.811 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.811 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.811 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.811 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.811 INFO:teuthology.orchestra.run.smithi043.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-28T22:47:51.811 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-28T22:47:51.811 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.811 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.811 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.811 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "5", 2024-03-28T22:47:51.812 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.812 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.812 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.812 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.812 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.812 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.812 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.812 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.812 INFO:teuthology.orchestra.run.smithi043.stdout: "cross_origin_url": { 2024-03-28T22:47:51.813 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cross_origin_url", 2024-03-28T22:47:51.813 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.813 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.813 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.813 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.813 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.813 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.813 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.813 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.813 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.814 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.814 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.814 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.814 INFO:teuthology.orchestra.run.smithi043.stdout: "crt_file": { 2024-03-28T22:47:51.814 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "crt_file", 2024-03-28T22:47:51.814 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.814 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.814 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.814 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.815 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.815 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.815 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.815 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.815 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.815 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.815 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.815 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.815 INFO:teuthology.orchestra.run.smithi043.stdout: "debug": { 2024-03-28T22:47:51.815 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "debug", 2024-03-28T22:47:51.816 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.816 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.816 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.816 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.816 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.816 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.816 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.816 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Enable/disable debug options", 2024-03-28T22:47:51.816 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.817 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.817 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.817 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.817 INFO:teuthology.orchestra.run.smithi043.stdout: "jwt_token_ttl": { 2024-03-28T22:47:51.817 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "jwt_token_ttl", 2024-03-28T22:47:51.817 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.817 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.817 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.817 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "28800", 2024-03-28T22:47:51.817 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.818 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.818 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.818 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.818 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.818 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.818 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.818 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.819 INFO:teuthology.orchestra.run.smithi043.stdout: "key_file": { 2024-03-28T22:47:51.819 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "key_file", 2024-03-28T22:47:51.819 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.819 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.819 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.819 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.819 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.819 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.819 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.820 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.820 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.820 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.820 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.820 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.820 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.820 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.820 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.820 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.821 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.821 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.821 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.821 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.821 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.821 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.821 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.821 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.822 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.822 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.822 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.822 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.822 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.822 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.822 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.822 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.823 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.823 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.823 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.823 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.823 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.823 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.823 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.823 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.823 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.824 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.824 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.824 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.824 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.824 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.824 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.824 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.824 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.824 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.825 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.825 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.825 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.825 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.825 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.825 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.825 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.825 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.826 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.826 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.826 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.826 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.826 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.826 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.826 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.826 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.827 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.827 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.827 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.827 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.827 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.827 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.827 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.827 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.827 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.828 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.829 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.829 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.829 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.830 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.830 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.830 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.830 INFO:teuthology.orchestra.run.smithi043.stdout: "motd": { 2024-03-28T22:47:51.830 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "motd", 2024-03-28T22:47:51.830 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.830 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.830 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.831 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.831 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.831 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.831 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.831 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "The message of the day", 2024-03-28T22:47:51.831 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.831 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.831 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.831 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.831 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_resolve_ip_addr": { 2024-03-28T22:47:51.832 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "redirect_resolve_ip_addr", 2024-03-28T22:47:51.832 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.832 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.832 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.832 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.832 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.832 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.832 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.833 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.833 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.833 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.833 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.833 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.833 INFO:teuthology.orchestra.run.smithi043.stdout: "server_addr": { 2024-03-28T22:47:51.833 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_addr", 2024-03-28T22:47:51.833 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.833 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.834 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.834 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "::", 2024-03-28T22:47:51.834 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.834 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.834 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.834 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.834 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.834 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.835 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.835 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.835 INFO:teuthology.orchestra.run.smithi043.stdout: "server_port": { 2024-03-28T22:47:51.835 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_port", 2024-03-28T22:47:51.835 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.835 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.835 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.835 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "8080", 2024-03-28T22:47:51.835 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.836 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.836 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.836 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.836 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.836 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.836 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.836 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.836 INFO:teuthology.orchestra.run.smithi043.stdout: "ssl": { 2024-03-28T22:47:51.836 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ssl", 2024-03-28T22:47:51.837 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.837 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.837 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.837 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.837 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.837 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.837 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.837 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.837 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.838 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.838 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.838 INFO:teuthology.orchestra.run.smithi043.stdout: "ssl_server_port": { 2024-03-28T22:47:51.838 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ssl_server_port", 2024-03-28T22:47:51.838 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.838 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.838 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.839 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "8443", 2024-03-28T22:47:51.839 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.839 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.839 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.839 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.839 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.839 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.839 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.840 INFO:teuthology.orchestra.run.smithi043.stdout: "standby_behaviour": { 2024-03-28T22:47:51.840 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "standby_behaviour", 2024-03-28T22:47:51.840 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.840 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.840 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.840 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "redirect", 2024-03-28T22:47:51.840 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.840 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.840 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.841 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.841 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect" 2024-03-28T22:47:51.841 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.841 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.841 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.841 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.841 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.841 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.841 INFO:teuthology.orchestra.run.smithi043.stdout: "standby_error_status_code": { 2024-03-28T22:47:51.842 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "standby_error_status_code", 2024-03-28T22:47:51.842 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.842 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.842 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.842 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "500", 2024-03-28T22:47:51.842 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "400", 2024-03-28T22:47:51.842 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "599", 2024-03-28T22:47:51.842 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.843 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.843 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.843 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.843 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.843 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.843 INFO:teuthology.orchestra.run.smithi043.stdout: "url_prefix": { 2024-03-28T22:47:51.843 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "url_prefix", 2024-03-28T22:47:51.843 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.843 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.844 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.844 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.844 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.844 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.844 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.844 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.844 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.844 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.844 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.845 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.845 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.845 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.845 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.845 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "devicehealth", 2024-03-28T22:47:51.845 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.845 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.845 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.846 INFO:teuthology.orchestra.run.smithi043.stdout: "enable_monitoring": { 2024-03-28T22:47:51.846 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "enable_monitoring", 2024-03-28T22:47:51.846 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.846 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.846 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.846 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.846 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.846 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.847 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.847 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "monitor device health metrics", 2024-03-28T22:47:51.847 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.847 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.847 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.847 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.847 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.847 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.847 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.848 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.848 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.848 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.848 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.848 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.848 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.848 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.848 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.849 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.849 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.849 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.849 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.849 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.849 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.849 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.849 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.849 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.850 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.850 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.850 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.850 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.850 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.850 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.850 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.850 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.851 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.851 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.851 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.851 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.851 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.851 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.851 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.851 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.852 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.852 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.852 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.852 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.852 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.852 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.852 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.852 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.852 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.853 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.853 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.853 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.853 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.853 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.853 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.853 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.853 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.854 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.854 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.854 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.854 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.854 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.854 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.854 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.854 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.855 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.855 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.855 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.855 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.855 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.855 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.855 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.855 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.855 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.856 INFO:teuthology.orchestra.run.smithi043.stdout: "mark_out_threshold": { 2024-03-28T22:47:51.856 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mark_out_threshold", 2024-03-28T22:47:51.856 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:51.856 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.856 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.856 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "2419200", 2024-03-28T22:47:51.856 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.856 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.857 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.857 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-28T22:47:51.857 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.857 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.857 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.857 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.857 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": { 2024-03-28T22:47:51.857 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "pool_name", 2024-03-28T22:47:51.858 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.858 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.858 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.858 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "device_health_metrics", 2024-03-28T22:47:51.858 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.858 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.858 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.858 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-28T22:47:51.859 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.859 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.859 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.859 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.859 INFO:teuthology.orchestra.run.smithi043.stdout: "retention_period": { 2024-03-28T22:47:51.859 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "retention_period", 2024-03-28T22:47:51.859 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:51.859 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.860 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.860 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "15552000", 2024-03-28T22:47:51.860 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.860 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.860 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.860 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how long to retain device health metrics", 2024-03-28T22:47:51.860 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.860 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.860 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.861 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.861 INFO:teuthology.orchestra.run.smithi043.stdout: "scrape_frequency": { 2024-03-28T22:47:51.861 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "scrape_frequency", 2024-03-28T22:47:51.861 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:51.861 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.861 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.861 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "86400", 2024-03-28T22:47:51.861 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.862 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.862 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.862 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-28T22:47:51.862 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.862 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.862 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.862 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.862 INFO:teuthology.orchestra.run.smithi043.stdout: "self_heal": { 2024-03-28T22:47:51.863 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "self_heal", 2024-03-28T22:47:51.863 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.863 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.863 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.863 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:51.863 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.863 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.863 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.863 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-28T22:47:51.864 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.864 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.864 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.864 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.864 INFO:teuthology.orchestra.run.smithi043.stdout: "sleep_interval": { 2024-03-28T22:47:51.864 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sleep_interval", 2024-03-28T22:47:51.864 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:51.864 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.864 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.865 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "600", 2024-03-28T22:47:51.865 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.865 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.865 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.865 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how frequently to wake up and check device health", 2024-03-28T22:47:51.865 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.865 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.865 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.865 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.866 INFO:teuthology.orchestra.run.smithi043.stdout: "warn_threshold": { 2024-03-28T22:47:51.866 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "warn_threshold", 2024-03-28T22:47:51.866 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:51.866 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.866 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.866 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "7257600", 2024-03-28T22:47:51.866 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.866 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.867 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.867 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-28T22:47:51.867 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.867 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.867 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.867 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.867 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.867 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.867 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.868 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "diskprediction_local", 2024-03-28T22:47:51.868 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.868 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.868 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.868 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.868 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.868 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.868 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.868 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.869 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.869 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.869 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.869 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.869 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.869 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.869 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.869 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.869 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.870 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.870 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.870 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.870 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.870 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.870 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.870 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.870 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.871 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.871 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.871 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.871 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.871 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.871 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.871 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.871 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.871 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.871 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.872 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.872 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.872 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.872 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.872 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.872 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.872 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.872 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.873 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.873 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.873 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.873 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.873 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.873 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.873 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.873 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.873 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.874 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.874 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.874 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.874 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.874 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.874 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.874 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.875 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.875 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.875 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.875 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.875 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.875 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.875 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.875 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.875 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.876 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.876 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.876 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.876 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.876 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.876 INFO:teuthology.orchestra.run.smithi043.stdout: "predict_interval": { 2024-03-28T22:47:51.876 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "predict_interval", 2024-03-28T22:47:51.876 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.876 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.876 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.877 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "86400", 2024-03-28T22:47:51.877 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.877 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.877 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.877 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.877 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.877 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.877 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.877 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.878 INFO:teuthology.orchestra.run.smithi043.stdout: "predictor_model": { 2024-03-28T22:47:51.878 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "predictor_model", 2024-03-28T22:47:51.878 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.878 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.878 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.878 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "prophetstor", 2024-03-28T22:47:51.878 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.878 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.879 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.879 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.879 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.879 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.879 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.879 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.879 INFO:teuthology.orchestra.run.smithi043.stdout: "sleep_interval": { 2024-03-28T22:47:51.879 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sleep_interval", 2024-03-28T22:47:51.880 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.880 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.880 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.880 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "600", 2024-03-28T22:47:51.880 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.880 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.880 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.880 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.880 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.881 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.881 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.881 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.881 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.881 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.881 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.881 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "influx", 2024-03-28T22:47:51.881 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": false, 2024-03-28T22:47:51.881 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "influxdb python module not found", 2024-03-28T22:47:51.882 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.882 INFO:teuthology.orchestra.run.smithi043.stdout: "batch_size": { 2024-03-28T22:47:51.882 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "batch_size", 2024-03-28T22:47:51.882 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.882 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.882 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.882 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "5000", 2024-03-28T22:47:51.882 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.883 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.883 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.883 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-28T22:47:51.883 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.883 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.883 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.883 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.883 INFO:teuthology.orchestra.run.smithi043.stdout: "database": { 2024-03-28T22:47:51.883 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "database", 2024-03-28T22:47:51.884 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.884 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.884 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.884 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "ceph", 2024-03-28T22:47:51.884 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.884 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.884 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.884 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-28T22:47:51.884 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.885 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.885 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.885 INFO:teuthology.orchestra.run.smithi043.stdout: "hostname": { 2024-03-28T22:47:51.885 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "hostname", 2024-03-28T22:47:51.885 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.885 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.885 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.886 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.886 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.886 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.886 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.886 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "InfluxDB server hostname", 2024-03-28T22:47:51.886 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.886 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.886 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.887 INFO:teuthology.orchestra.run.smithi043.stdout: "interval": { 2024-03-28T22:47:51.887 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "interval", 2024-03-28T22:47:51.887 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:51.887 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.887 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.887 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "30", 2024-03-28T22:47:51.887 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "5", 2024-03-28T22:47:51.887 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.888 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.888 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-28T22:47:51.888 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.888 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.888 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.888 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.888 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.888 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.889 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.889 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.889 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.889 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.889 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.889 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.889 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.889 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.889 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.890 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.890 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.890 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.890 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.890 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.890 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.890 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.890 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.890 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.891 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.891 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.891 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.891 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.891 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.891 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.891 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.891 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.892 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.892 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.892 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.892 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.892 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.892 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.892 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.893 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.893 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.893 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.893 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.893 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.893 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.893 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.893 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.893 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.894 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.894 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.894 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.894 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.894 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.894 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.894 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.894 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.895 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.895 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.895 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.895 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.895 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.895 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.895 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.895 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.896 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.896 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.896 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.896 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.896 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.896 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.896 INFO:teuthology.orchestra.run.smithi043.stdout: "password": { 2024-03-28T22:47:51.897 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "password", 2024-03-28T22:47:51.897 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.897 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.897 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.897 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.897 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.897 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.897 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.897 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "password of InfluxDB server user", 2024-03-28T22:47:51.898 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.898 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.898 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.898 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.898 INFO:teuthology.orchestra.run.smithi043.stdout: "port": { 2024-03-28T22:47:51.898 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "port", 2024-03-28T22:47:51.898 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.899 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.899 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.899 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "8086", 2024-03-28T22:47:51.899 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.899 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.899 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.899 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "InfluxDB server port", 2024-03-28T22:47:51.899 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.899 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.900 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.900 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.900 INFO:teuthology.orchestra.run.smithi043.stdout: "ssl": { 2024-03-28T22:47:51.900 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ssl", 2024-03-28T22:47:51.900 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.900 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.900 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.900 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "false", 2024-03-28T22:47:51.900 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.901 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.901 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.901 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T22:47:51.901 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.901 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.901 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.901 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.901 INFO:teuthology.orchestra.run.smithi043.stdout: "threads": { 2024-03-28T22:47:51.901 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "threads", 2024-03-28T22:47:51.902 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.902 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.902 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.902 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "5", 2024-03-28T22:47:51.902 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "1", 2024-03-28T22:47:51.902 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "32", 2024-03-28T22:47:51.902 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.902 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-28T22:47:51.903 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.903 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.903 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.903 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.903 INFO:teuthology.orchestra.run.smithi043.stdout: "username": { 2024-03-28T22:47:51.903 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "username", 2024-03-28T22:47:51.903 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.903 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.903 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.904 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.904 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.904 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.904 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.904 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "username of InfluxDB server user", 2024-03-28T22:47:51.904 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.904 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.904 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.904 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.905 INFO:teuthology.orchestra.run.smithi043.stdout: "verify_ssl": { 2024-03-28T22:47:51.905 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "verify_ssl", 2024-03-28T22:47:51.905 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.905 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.905 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.905 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "true", 2024-03-28T22:47:51.905 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.905 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.905 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.906 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T22:47:51.906 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.906 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.906 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.906 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.906 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.906 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.906 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.907 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "insights", 2024-03-28T22:47:51.907 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.907 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.907 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.907 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.907 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.907 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.907 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.907 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.908 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.908 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.908 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.908 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.908 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.908 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.908 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.908 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.908 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.909 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.909 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.909 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.909 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.909 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.909 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.909 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.909 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.909 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.910 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.910 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.910 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.910 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.910 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.910 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.910 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.910 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.911 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.911 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.911 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.911 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.911 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.911 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.911 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.911 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.911 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.912 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.912 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.912 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.912 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.912 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.912 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.912 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.912 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.912 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.913 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.913 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.913 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.913 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.913 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.913 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.913 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.913 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.913 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.914 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.914 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.914 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.914 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.914 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.914 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.914 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.914 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.915 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.915 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.915 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.915 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.915 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.915 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.915 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.915 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "iostat", 2024-03-28T22:47:51.915 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.916 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.916 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.916 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.916 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.916 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.916 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.916 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.916 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.916 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.917 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.917 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.917 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.917 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.917 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.917 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.917 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.917 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.917 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.918 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.918 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.918 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.918 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.918 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.918 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.918 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.918 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.919 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.919 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.919 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.919 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.919 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.919 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.919 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.919 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.920 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.920 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.920 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.920 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.920 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.920 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.920 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.920 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.920 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.921 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.921 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.921 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.921 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.921 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.921 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.921 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.921 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.921 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.922 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.922 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.922 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.922 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.922 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.922 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.922 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.922 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.923 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.923 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.923 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.923 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.923 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.923 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.923 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.923 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.923 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.924 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.924 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.924 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.924 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.924 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.924 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.924 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "k8sevents", 2024-03-28T22:47:51.924 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.925 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.925 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.925 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_event_retention_days": { 2024-03-28T22:47:51.925 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ceph_event_retention_days", 2024-03-28T22:47:51.925 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.925 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.925 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.925 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "7", 2024-03-28T22:47:51.925 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.926 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.926 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.926 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-28T22:47:51.926 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.926 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.926 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.926 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.926 INFO:teuthology.orchestra.run.smithi043.stdout: "config_check_secs": { 2024-03-28T22:47:51.927 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "config_check_secs", 2024-03-28T22:47:51.927 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.927 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.927 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.927 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-03-28T22:47:51.927 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "10", 2024-03-28T22:47:51.927 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.927 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.928 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-28T22:47:51.928 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.928 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.928 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.928 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.928 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.928 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.928 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.929 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.929 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.929 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.929 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.929 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.929 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.929 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.929 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.929 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.930 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.930 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.930 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.930 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.930 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.930 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.930 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.930 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.931 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.931 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.931 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.931 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.931 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.931 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.931 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.931 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.931 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.932 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.932 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.932 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.932 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.932 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.932 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.932 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.932 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.933 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.933 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.933 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.933 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.933 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.933 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.933 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.933 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.933 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.934 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.934 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.934 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.934 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.934 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.934 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.934 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.934 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.935 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.935 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.935 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.935 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.935 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.935 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.935 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.935 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.935 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.936 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.936 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.936 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.936 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.936 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.936 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.936 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.936 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.937 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.937 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.937 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "localpool", 2024-03-28T22:47:51.937 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.937 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.937 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.937 INFO:teuthology.orchestra.run.smithi043.stdout: "failure_domain": { 2024-03-28T22:47:51.937 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "failure_domain", 2024-03-28T22:47:51.938 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.938 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.938 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.938 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "host", 2024-03-28T22:47:51.938 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.938 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.938 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.938 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "failure domain for any created local pool", 2024-03-28T22:47:51.938 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-28T22:47:51.939 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.939 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.939 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.939 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.939 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.939 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.939 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.939 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.939 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.940 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.940 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.940 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.940 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.940 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.940 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.940 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.940 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.941 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.941 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.941 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.941 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.941 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.941 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.941 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.941 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.941 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.942 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.942 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.942 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.942 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.942 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.942 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.942 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.942 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.943 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.943 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.943 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.943 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.943 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.943 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.943 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.943 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.943 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.943 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.944 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.944 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.944 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.944 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.944 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.944 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.944 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.944 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.944 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.945 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.945 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.945 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.945 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.945 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.945 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.945 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.945 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.945 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.946 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.946 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.946 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.946 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.946 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.946 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.946 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.946 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.947 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.947 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.947 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.947 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": { 2024-03-28T22:47:51.947 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "min_size", 2024-03-28T22:47:51.947 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.947 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.947 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.947 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.948 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.948 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.948 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.948 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "default min_size for any created local pool", 2024-03-28T22:47:51.948 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-28T22:47:51.948 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.948 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.949 INFO:teuthology.orchestra.run.smithi043.stdout: "num_rep": { 2024-03-28T22:47:51.949 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "num_rep", 2024-03-28T22:47:51.949 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.949 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.949 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.949 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "3", 2024-03-28T22:47:51.949 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.949 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.949 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.950 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "default replica count for any created local pool", 2024-03-28T22:47:51.950 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.950 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.950 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.950 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.950 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": { 2024-03-28T22:47:51.950 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "pg_num", 2024-03-28T22:47:51.950 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:51.951 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.951 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.951 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "128", 2024-03-28T22:47:51.951 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.951 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.951 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.951 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "default pg_num for any created local pool", 2024-03-28T22:47:51.951 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.952 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.952 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.952 INFO:teuthology.orchestra.run.smithi043.stdout: "prefix": { 2024-03-28T22:47:51.952 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "prefix", 2024-03-28T22:47:51.952 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.952 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.952 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.952 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.953 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.953 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.953 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.953 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "name prefix for any created local pool", 2024-03-28T22:47:51.953 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.953 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.953 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.953 INFO:teuthology.orchestra.run.smithi043.stdout: "subtree": { 2024-03-28T22:47:51.954 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subtree", 2024-03-28T22:47:51.954 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.954 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.954 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.954 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "rack", 2024-03-28T22:47:51.954 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.954 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.954 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.954 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-28T22:47:51.955 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-28T22:47:51.955 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.955 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.955 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.955 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.955 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.955 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.955 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mds_autoscaler", 2024-03-28T22:47:51.956 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.956 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.956 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.956 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.956 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.956 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.956 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.956 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.956 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.957 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.957 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.957 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.957 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.957 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.957 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.957 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.957 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.957 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.958 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.958 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.958 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.958 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.958 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.959 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.959 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.959 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.959 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.959 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.959 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.959 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.959 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.959 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.960 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.960 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.960 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.960 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.960 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.960 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.960 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.961 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.961 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.961 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.961 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.961 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.961 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.961 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.961 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.961 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.962 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.962 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.962 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.962 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.962 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.962 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.962 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.962 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.962 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.963 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.963 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.963 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.963 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.963 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.963 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.963 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.963 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.964 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.964 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.964 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.964 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.964 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.964 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.964 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.965 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mirroring", 2024-03-28T22:47:51.965 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.965 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.965 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.965 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.965 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.965 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.965 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.966 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.966 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.966 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.966 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.966 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.966 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.966 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.966 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.967 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.967 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.967 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.967 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.967 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.967 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.967 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.967 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.968 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.968 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.968 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.968 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.968 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.968 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.968 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.968 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.968 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.969 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.969 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.969 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.969 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.969 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.969 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.969 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.969 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.970 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.970 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.970 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.970 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.970 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.970 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.970 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.970 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.971 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.971 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.971 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.971 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.971 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.971 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.971 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.971 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.971 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.972 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.972 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.972 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.972 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.972 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.972 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.972 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.972 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.972 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.973 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.973 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.973 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.973 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.973 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.973 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.973 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.973 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.974 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "nfs", 2024-03-28T22:47:51.974 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.974 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.974 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.974 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.974 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.974 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.974 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.975 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.975 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.975 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.975 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.975 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.975 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.975 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.975 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.975 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.976 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.976 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.976 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.976 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.976 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.976 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.976 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.976 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.976 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.977 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.977 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.977 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.977 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.977 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.977 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.977 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.977 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.977 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.978 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.978 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.978 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.978 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.978 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.978 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.978 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.978 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.978 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.979 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.979 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.979 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.979 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.979 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.979 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.979 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.979 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.980 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.980 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.980 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.980 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.980 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.980 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.980 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.980 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.980 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.980 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.981 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.981 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.981 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.981 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.981 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.981 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.981 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.982 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.982 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.982 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.982 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.982 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.982 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.982 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.982 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.982 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "orchestrator", 2024-03-28T22:47:51.983 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.983 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.983 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.983 INFO:teuthology.orchestra.run.smithi043.stdout: "fail_fs": { 2024-03-28T22:47:51.983 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "fail_fs", 2024-03-28T22:47:51.983 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.983 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.983 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:51.984 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.984 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.984 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.984 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.984 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-28T22:47:51.984 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.984 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.984 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.984 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.985 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.985 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.985 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.985 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.985 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.985 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.985 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.985 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.985 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.986 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.986 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.986 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.986 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.986 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.986 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.986 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.986 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.986 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.987 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.987 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.987 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.987 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.987 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.987 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.987 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.987 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.988 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.988 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.988 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.988 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.988 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.988 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.988 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.988 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.989 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:51.989 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:51.989 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.989 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.989 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.989 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:51.989 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.989 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.989 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.990 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.990 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.990 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.990 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.990 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.990 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.990 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.990 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.991 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.991 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.991 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.991 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:51.991 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:51.991 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.991 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.991 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.992 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.992 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.992 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.992 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.992 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.992 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.992 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.992 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.993 INFO:teuthology.orchestra.run.smithi043.stdout: "orchestrator": { 2024-03-28T22:47:51.993 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "orchestrator", 2024-03-28T22:47:51.993 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.993 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.993 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.993 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.993 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.993 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.994 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.994 INFO:teuthology.orchestra.run.smithi043.stdout: "cephadm", 2024-03-28T22:47:51.994 INFO:teuthology.orchestra.run.smithi043.stdout: "rook", 2024-03-28T22:47:51.994 INFO:teuthology.orchestra.run.smithi043.stdout: "test_orchestrator" 2024-03-28T22:47:51.994 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.994 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Orchestrator backend", 2024-03-28T22:47:51.994 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.994 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.995 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.995 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.995 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:51.995 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.995 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:51.995 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "osd_perf_query", 2024-03-28T22:47:51.995 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:51.995 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:51.995 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:51.996 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:51.996 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:51.996 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:51.996 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.996 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.996 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:51.996 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.996 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.997 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:51.997 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:51.997 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:51.997 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:51.997 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:51.997 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:51.997 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:51.997 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:51.997 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.998 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.998 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.998 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:51.998 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:51.998 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:51.998 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:51.998 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:51.998 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:51.999 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:51.999 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:51.999 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:51.999 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:51.999 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:51.999 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:51.999 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:51.999 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:51.999 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.000 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.000 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.000 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.000 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.000 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.000 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.000 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.000 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.001 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.001 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.001 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.001 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.001 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.001 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.001 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.001 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.001 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.002 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.002 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.002 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.002 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.002 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.002 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.002 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.002 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.002 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.003 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.003 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.003 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.003 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.003 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.003 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.003 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.003 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.003 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.004 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.004 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.004 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.004 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "osd_support", 2024-03-28T22:47:52.004 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.004 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.004 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.004 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.004 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.005 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.005 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.005 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.005 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.005 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.005 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.005 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.005 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.006 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.006 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.006 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.006 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.006 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.006 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.006 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.006 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.006 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.007 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.007 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.007 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.007 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.007 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.007 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.007 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.007 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.007 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.008 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.008 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.008 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.008 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.008 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.008 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.008 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.008 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.008 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.009 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.009 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.009 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.009 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.009 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.009 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.009 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.009 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.009 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.010 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.010 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.010 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.010 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.010 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.010 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.010 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.010 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.011 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.011 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.011 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.011 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.011 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.011 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.011 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.011 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.011 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.012 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.012 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.012 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.012 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.012 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.012 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.012 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.012 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.012 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.013 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.013 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "pg_autoscaler", 2024-03-28T22:47:52.013 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.013 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.013 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.013 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.013 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.013 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.013 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.014 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.014 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.014 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.014 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.014 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.014 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.014 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.014 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.015 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.015 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.015 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.015 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.015 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.015 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.015 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.015 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.015 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.015 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.016 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.016 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.016 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.016 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.016 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.016 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.016 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.016 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.017 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.017 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.017 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.017 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.017 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.017 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.017 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.017 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.017 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.017 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.018 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.018 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.018 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.018 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.018 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.018 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.018 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.018 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.019 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.019 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.019 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.019 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.019 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.019 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.019 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.019 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.020 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.020 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.020 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.020 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.020 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.020 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.020 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.020 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.020 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.021 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.021 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.021 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.021 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.021 INFO:teuthology.orchestra.run.smithi043.stdout: "sleep_interval": { 2024-03-28T22:47:52.021 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sleep_interval", 2024-03-28T22:47:52.021 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:52.021 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.022 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.022 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "60", 2024-03-28T22:47:52.022 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.022 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.022 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.022 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.022 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.022 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.022 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.023 INFO:teuthology.orchestra.run.smithi043.stdout: "threshold": { 2024-03-28T22:47:52.023 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "threshold", 2024-03-28T22:47:52.023 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-03-28T22:47:52.023 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.023 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.023 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "3.0", 2024-03-28T22:47:52.023 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "1.0", 2024-03-28T22:47:52.023 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.024 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.024 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "scaling threshold", 2024-03-28T22:47:52.024 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-28T22:47:52.024 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.024 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.024 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.024 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.024 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.025 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "progress", 2024-03-28T22:47:52.025 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.025 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.025 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.025 INFO:teuthology.orchestra.run.smithi043.stdout: "allow_pg_recovery_event": { 2024-03-28T22:47:52.025 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "allow_pg_recovery_event", 2024-03-28T22:47:52.025 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.025 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.025 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.026 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.026 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.026 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.026 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.026 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-28T22:47:52.026 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.026 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.026 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.027 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": { 2024-03-28T22:47:52.027 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "enabled", 2024-03-28T22:47:52.027 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.027 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.027 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.027 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.027 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.027 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.028 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.028 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.028 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.028 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.028 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.028 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.028 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.028 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.029 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.029 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.029 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.029 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.029 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.029 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.029 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.029 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.029 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.030 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.030 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.030 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.030 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.030 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.030 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.030 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.030 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.031 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.031 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.031 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.031 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.031 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.031 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.031 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.032 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.032 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.032 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.032 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.032 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.032 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.033 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.033 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.033 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.033 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.034 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.034 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.034 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.034 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.034 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.034 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.034 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.034 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.035 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.035 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.035 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.035 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.035 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.035 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.035 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.035 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.035 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.036 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.036 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.036 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.036 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.036 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.036 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.036 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.036 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.036 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.037 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.037 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.037 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.037 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.037 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.037 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.037 INFO:teuthology.orchestra.run.smithi043.stdout: "max_completed_events": { 2024-03-28T22:47:52.037 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "max_completed_events", 2024-03-28T22:47:52.038 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.038 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.038 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.038 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "50", 2024-03-28T22:47:52.038 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.038 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.038 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.038 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "number of past completed events to remember", 2024-03-28T22:47:52.039 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.039 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.039 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.039 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.039 INFO:teuthology.orchestra.run.smithi043.stdout: "sleep_interval": { 2024-03-28T22:47:52.039 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sleep_interval", 2024-03-28T22:47:52.039 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:52.039 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.039 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.040 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "5", 2024-03-28T22:47:52.040 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.040 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.040 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.040 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how long the module is going to sleep", 2024-03-28T22:47:52.040 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.040 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.040 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.040 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.041 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.041 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.041 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.041 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "prometheus", 2024-03-28T22:47:52.041 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.041 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.041 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.041 INFO:teuthology.orchestra.run.smithi043.stdout: "cache": { 2024-03-28T22:47:52.042 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cache", 2024-03-28T22:47:52.042 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.042 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.042 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.042 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.042 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.042 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.042 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.042 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.043 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.043 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.043 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.043 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.043 INFO:teuthology.orchestra.run.smithi043.stdout: "exclude_perf_counters": { 2024-03-28T22:47:52.043 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "exclude_perf_counters", 2024-03-28T22:47:52.043 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.043 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.043 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.044 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.044 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.044 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.044 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.044 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-28T22:47:52.044 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-28T22:47:52.044 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.044 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.044 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.045 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.045 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.045 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.045 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.045 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.045 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.045 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.045 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.045 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.046 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.046 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.046 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.046 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.046 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.046 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.046 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.046 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.046 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.047 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.047 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.047 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.047 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.047 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.047 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.047 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.047 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.047 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.048 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.048 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.048 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.048 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.048 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.048 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.048 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.048 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.049 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.049 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.049 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.049 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.049 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.049 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.049 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.049 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.049 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.050 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.050 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.050 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.050 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.050 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.050 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.050 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.050 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.051 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.051 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.051 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.051 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.051 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.051 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.051 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.051 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.051 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.052 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.052 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.052 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.052 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.052 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.052 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.052 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.052 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.052 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.053 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_stats_pools": { 2024-03-28T22:47:52.053 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rbd_stats_pools", 2024-03-28T22:47:52.053 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.053 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.053 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.053 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.053 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.053 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.053 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.054 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.054 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.054 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.054 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.054 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.054 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-28T22:47:52.054 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-28T22:47:52.054 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.055 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.055 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.055 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "300", 2024-03-28T22:47:52.055 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.055 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.055 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.055 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.055 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.055 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.056 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.056 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.056 INFO:teuthology.orchestra.run.smithi043.stdout: "scrape_interval": { 2024-03-28T22:47:52.056 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "scrape_interval", 2024-03-28T22:47:52.056 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-03-28T22:47:52.056 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.056 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.056 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "15.0", 2024-03-28T22:47:52.056 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.057 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.057 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.057 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.057 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.057 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.057 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.057 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.057 INFO:teuthology.orchestra.run.smithi043.stdout: "server_addr": { 2024-03-28T22:47:52.058 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_addr", 2024-03-28T22:47:52.058 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.058 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.058 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.058 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "::", 2024-03-28T22:47:52.058 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.058 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.058 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.059 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-28T22:47:52.059 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.059 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.059 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.059 INFO:teuthology.orchestra.run.smithi043.stdout: "server_port": { 2024-03-28T22:47:52.059 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_port", 2024-03-28T22:47:52.059 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.059 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.060 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.060 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "9283", 2024-03-28T22:47:52.060 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.060 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.060 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.060 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-28T22:47:52.060 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.060 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.061 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.061 INFO:teuthology.orchestra.run.smithi043.stdout: "stale_cache_strategy": { 2024-03-28T22:47:52.061 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "stale_cache_strategy", 2024-03-28T22:47:52.061 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.061 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.061 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.061 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "log", 2024-03-28T22:47:52.061 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.062 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.062 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.062 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.062 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.062 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.062 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.062 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.062 INFO:teuthology.orchestra.run.smithi043.stdout: "standby_behaviour": { 2024-03-28T22:47:52.063 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "standby_behaviour", 2024-03-28T22:47:52.063 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.063 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.063 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.063 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "default", 2024-03-28T22:47:52.063 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.063 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.063 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.064 INFO:teuthology.orchestra.run.smithi043.stdout: "default", 2024-03-28T22:47:52.064 INFO:teuthology.orchestra.run.smithi043.stdout: "error" 2024-03-28T22:47:52.064 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.064 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.064 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.064 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.064 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.064 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.064 INFO:teuthology.orchestra.run.smithi043.stdout: "standby_error_status_code": { 2024-03-28T22:47:52.065 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "standby_error_status_code", 2024-03-28T22:47:52.065 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.065 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.065 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.065 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "500", 2024-03-28T22:47:52.065 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "400", 2024-03-28T22:47:52.065 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "599", 2024-03-28T22:47:52.065 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.065 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.066 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.066 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.066 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.066 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.066 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.066 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.066 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.066 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rbd_support", 2024-03-28T22:47:52.066 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.067 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.067 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.067 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.067 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.067 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.067 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.067 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.067 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.067 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.068 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.068 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.068 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.068 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.068 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.068 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.068 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.068 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.069 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.069 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.069 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.069 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.069 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.069 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.069 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.069 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.069 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.070 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.070 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.070 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.070 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.070 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.070 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.070 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.070 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.070 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.071 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.071 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.071 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.071 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.071 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.071 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.071 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.071 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.072 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.072 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.072 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.072 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.072 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.072 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.072 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.072 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.072 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.073 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.073 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.073 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.073 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.073 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.073 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.073 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.073 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.074 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.074 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.074 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.074 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.074 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.074 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.074 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.074 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.074 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.075 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.075 INFO:teuthology.orchestra.run.smithi043.stdout: "max_concurrent_snap_create": { 2024-03-28T22:47:52.075 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "max_concurrent_snap_create", 2024-03-28T22:47:52.075 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.075 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.075 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.075 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-03-28T22:47:52.076 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.076 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.076 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.076 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.076 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.076 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.076 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.076 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.076 INFO:teuthology.orchestra.run.smithi043.stdout: "mirror_snapshot_schedule": { 2024-03-28T22:47:52.077 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mirror_snapshot_schedule", 2024-03-28T22:47:52.077 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.077 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.077 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.077 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.077 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.077 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.077 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.077 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.078 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.078 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.078 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.078 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.078 INFO:teuthology.orchestra.run.smithi043.stdout: "trash_purge_schedule": { 2024-03-28T22:47:52.078 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "trash_purge_schedule", 2024-03-28T22:47:52.078 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.078 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.079 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.079 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.079 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.079 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.079 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.079 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.079 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.079 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.079 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.080 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.080 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.080 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.080 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.080 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "restful", 2024-03-28T22:47:52.080 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.080 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.080 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.081 INFO:teuthology.orchestra.run.smithi043.stdout: "enable_auth": { 2024-03-28T22:47:52.081 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "enable_auth", 2024-03-28T22:47:52.081 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.081 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.081 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.081 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.081 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.081 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.081 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.082 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.082 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.082 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.082 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.082 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.082 INFO:teuthology.orchestra.run.smithi043.stdout: "key_file": { 2024-03-28T22:47:52.082 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "key_file", 2024-03-28T22:47:52.082 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.083 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.083 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.083 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.083 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.083 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.083 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.083 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.083 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.083 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.084 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.084 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.084 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.084 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.084 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.084 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.084 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.084 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.084 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.085 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.085 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.085 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.085 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.085 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.085 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.085 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.085 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.086 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.086 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.086 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.086 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.086 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.086 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.086 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.086 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.086 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.087 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.087 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.087 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.087 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.087 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.087 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.087 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.087 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.088 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.088 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.088 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.088 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.088 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.088 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.088 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.088 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.088 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.089 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.089 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.089 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.089 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.089 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.089 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.089 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.089 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.089 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.090 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.090 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.090 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.090 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.090 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.090 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.090 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.090 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.091 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.091 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.091 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.091 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.091 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.091 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.091 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.091 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.092 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.092 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.092 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.092 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.092 INFO:teuthology.orchestra.run.smithi043.stdout: "server_addr": { 2024-03-28T22:47:52.092 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_addr", 2024-03-28T22:47:52.092 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.092 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.092 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.093 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.093 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.093 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.093 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.093 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.093 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.093 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.093 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.094 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.094 INFO:teuthology.orchestra.run.smithi043.stdout: "server_port": { 2024-03-28T22:47:52.094 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_port", 2024-03-28T22:47:52.094 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.094 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.094 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.094 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.094 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.095 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.095 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.095 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.095 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.095 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.095 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.095 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.095 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.095 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.096 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.096 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rgw", 2024-03-28T22:47:52.096 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.096 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.096 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.096 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.096 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.096 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.097 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.097 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.097 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.097 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.097 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.097 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.097 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.097 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.097 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.098 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.098 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.098 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.098 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.098 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.098 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.098 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.098 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.099 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.099 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.099 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.099 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.099 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.099 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.099 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.099 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.100 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.100 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.100 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.100 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.100 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.100 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.100 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.100 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.100 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.101 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.101 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.101 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.101 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.101 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.101 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.101 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.101 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.101 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.102 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.102 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.102 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.102 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.102 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.102 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.102 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.102 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.102 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.103 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.103 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.103 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.103 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.103 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.103 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.103 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.103 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.103 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.104 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.104 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.104 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.104 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.104 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.104 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.104 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.104 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.104 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.105 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rook", 2024-03-28T22:47:52.105 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.105 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.105 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.105 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.105 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.105 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.105 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.105 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.106 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.106 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.106 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.106 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.106 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.106 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.106 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.106 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.107 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.107 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.107 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.107 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.107 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.107 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.107 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.107 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.107 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.108 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.108 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.108 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.108 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.108 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.108 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.108 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.108 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.108 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.109 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.109 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.109 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.109 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.109 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.109 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.109 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.109 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.109 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.110 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.110 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.110 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.110 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.110 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.110 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.110 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.110 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.111 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.111 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.111 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.111 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.111 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.111 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.111 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.111 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.111 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.112 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.112 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.112 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.112 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.112 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.112 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.112 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.112 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.113 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.113 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.113 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.113 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.113 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.113 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": { 2024-03-28T22:47:52.113 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "storage_class", 2024-03-28T22:47:52.113 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.113 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.114 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.114 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "local", 2024-03-28T22:47:52.114 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.114 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.114 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.114 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-28T22:47:52.114 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.114 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.114 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.115 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.115 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.115 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.115 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.115 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "selftest", 2024-03-28T22:47:52.115 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.115 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.115 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.115 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.116 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.116 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.116 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.116 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.116 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.116 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.116 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.116 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.117 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.117 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.117 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.117 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.117 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.117 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.117 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.117 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.117 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.118 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.118 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.118 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.118 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.118 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.118 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.118 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.118 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.119 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.119 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.119 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.119 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.119 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.119 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.119 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.119 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.119 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.120 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.120 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.120 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.120 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.120 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.120 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.120 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.120 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.120 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.121 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.121 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.121 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.121 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.121 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.121 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.121 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.121 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.122 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.122 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.122 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.122 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.122 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.122 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.122 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.122 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.123 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.123 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.123 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.123 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.123 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.123 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.123 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.123 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.123 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.124 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.124 INFO:teuthology.orchestra.run.smithi043.stdout: "roption1": { 2024-03-28T22:47:52.124 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "roption1", 2024-03-28T22:47:52.124 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.124 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.124 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.124 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.124 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.125 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.125 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.125 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.125 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.125 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.125 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.125 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.125 INFO:teuthology.orchestra.run.smithi043.stdout: "roption2": { 2024-03-28T22:47:52.125 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "roption2", 2024-03-28T22:47:52.126 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.126 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.126 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.126 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "xyz", 2024-03-28T22:47:52.126 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.126 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.126 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.126 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.127 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.127 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.127 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.127 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.127 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption1": { 2024-03-28T22:47:52.127 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption1", 2024-03-28T22:47:52.127 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.127 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.127 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.128 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.128 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.128 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.128 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.128 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.128 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.128 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.128 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.128 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.129 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption2": { 2024-03-28T22:47:52.129 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption2", 2024-03-28T22:47:52.129 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.129 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.129 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.129 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.129 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.129 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.129 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.130 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.130 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.130 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.130 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.130 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.130 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption3": { 2024-03-28T22:47:52.130 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption3", 2024-03-28T22:47:52.130 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-03-28T22:47:52.131 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.131 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.131 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.131 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.131 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.131 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.131 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.131 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.131 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.132 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.132 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.132 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption4": { 2024-03-28T22:47:52.132 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption4", 2024-03-28T22:47:52.132 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.132 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.132 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.132 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.133 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.133 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.133 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.133 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.133 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.133 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.133 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.133 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.133 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption5": { 2024-03-28T22:47:52.134 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption5", 2024-03-28T22:47:52.134 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.134 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.134 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.134 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.134 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.134 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.134 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.135 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.135 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.135 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.135 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.135 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.135 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption6": { 2024-03-28T22:47:52.135 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption6", 2024-03-28T22:47:52.135 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.135 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.136 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.136 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.136 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.136 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.136 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.136 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.136 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.136 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.137 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.137 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.137 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption7": { 2024-03-28T22:47:52.137 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption7", 2024-03-28T22:47:52.137 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.137 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.137 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.137 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.137 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "1", 2024-03-28T22:47:52.138 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "42", 2024-03-28T22:47:52.138 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.138 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.138 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.138 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.138 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.138 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.138 INFO:teuthology.orchestra.run.smithi043.stdout: "testkey": { 2024-03-28T22:47:52.139 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "testkey", 2024-03-28T22:47:52.139 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.139 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.139 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.139 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.139 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.139 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.139 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.139 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.140 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.140 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.140 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.140 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.140 INFO:teuthology.orchestra.run.smithi043.stdout: "testlkey": { 2024-03-28T22:47:52.140 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "testlkey", 2024-03-28T22:47:52.140 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.140 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.141 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.141 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.141 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.141 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.141 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.141 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.141 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.141 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.141 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.142 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.142 INFO:teuthology.orchestra.run.smithi043.stdout: "testnewline": { 2024-03-28T22:47:52.142 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "testnewline", 2024-03-28T22:47:52.142 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.142 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.142 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.142 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.142 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.143 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.143 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.143 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.143 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.143 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.143 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.143 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.143 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.143 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.144 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.144 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "snap_schedule", 2024-03-28T22:47:52.144 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.144 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.144 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.144 INFO:teuthology.orchestra.run.smithi043.stdout: "allow_m_granularity": { 2024-03-28T22:47:52.144 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "allow_m_granularity", 2024-03-28T22:47:52.144 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.144 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.145 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.145 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.145 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.145 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.145 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.145 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "allow minute scheduled snapshots", 2024-03-28T22:47:52.145 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.145 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.145 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.146 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.146 INFO:teuthology.orchestra.run.smithi043.stdout: "dump_on_update": { 2024-03-28T22:47:52.146 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "dump_on_update", 2024-03-28T22:47:52.146 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.146 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.146 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.146 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.146 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.147 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.147 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.147 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "dump database to debug log on update", 2024-03-28T22:47:52.147 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.147 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.147 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.147 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.147 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.147 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.148 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.148 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.148 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.148 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.148 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.148 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.148 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.148 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.148 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.149 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.149 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.149 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.149 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.149 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.149 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.149 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.149 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.150 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.150 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.150 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.150 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.150 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.150 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.150 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.150 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.150 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.151 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.151 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.151 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.151 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.151 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.151 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.151 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.151 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.152 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.152 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.152 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.152 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.152 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.152 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.152 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.152 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.152 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.153 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.153 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.153 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.153 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.153 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.153 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.153 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.153 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.154 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.154 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.154 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.154 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.154 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.154 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.154 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.154 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.154 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.155 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.155 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.155 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.155 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.155 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.155 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.155 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.155 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.156 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.156 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.156 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.156 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "stats", 2024-03-28T22:47:52.156 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.156 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.156 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.156 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.157 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.157 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.157 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.157 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.157 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.157 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.157 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.157 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.157 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.158 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.158 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.158 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.158 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.158 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.158 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.158 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.158 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.159 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.159 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.159 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.159 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.159 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.159 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.159 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.159 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.160 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.160 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.160 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.160 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.160 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.160 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.160 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.160 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.161 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.161 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.161 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.161 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.161 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.161 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.161 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.161 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.161 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.162 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.162 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.162 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.162 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.162 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.162 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.162 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.163 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.163 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.163 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.163 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.163 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.163 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.163 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.163 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.163 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.164 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.164 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.164 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.164 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.164 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.164 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.164 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.164 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.165 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.165 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.165 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.165 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.165 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.165 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.165 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "status", 2024-03-28T22:47:52.165 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.166 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.166 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.166 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.166 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.166 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.166 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.166 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.166 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.167 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.167 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.167 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.167 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.167 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.167 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.167 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.167 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.167 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.168 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.168 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.168 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.168 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.168 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.168 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.168 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.168 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.169 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.169 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.169 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.169 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.169 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.169 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.169 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.169 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.170 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.170 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.170 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.170 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.170 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.170 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.170 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.170 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.170 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.171 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.171 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.171 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.171 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.171 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.171 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.171 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.171 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.171 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.172 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.172 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.172 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.172 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.172 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.172 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.172 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.172 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.172 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.173 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.173 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.173 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.173 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.173 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.173 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.173 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.173 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.173 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.174 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.174 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.174 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.174 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.174 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.174 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.174 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "telegraf", 2024-03-28T22:47:52.174 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.174 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.175 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.175 INFO:teuthology.orchestra.run.smithi043.stdout: "address": { 2024-03-28T22:47:52.175 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "address", 2024-03-28T22:47:52.175 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.175 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.175 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.175 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-28T22:47:52.175 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.176 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.176 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.176 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.176 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.176 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.176 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.176 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.176 INFO:teuthology.orchestra.run.smithi043.stdout: "interval": { 2024-03-28T22:47:52.176 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "interval", 2024-03-28T22:47:52.176 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:52.177 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.177 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.177 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "15", 2024-03-28T22:47:52.177 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.177 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.177 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.177 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.177 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.178 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.178 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.178 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.178 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.178 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.178 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.178 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.178 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.179 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.179 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.179 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.179 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.179 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.179 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.179 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.179 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.179 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.180 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.180 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.180 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.180 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.180 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.180 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.180 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.180 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.180 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.181 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.181 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.181 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.181 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.181 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.181 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.181 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.181 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.181 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.182 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.182 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.182 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.182 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.182 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.182 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.182 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.182 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.183 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.183 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.183 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.183 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.183 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.183 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.183 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.183 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.183 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.184 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.184 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.184 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.184 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.184 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.184 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.184 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.184 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.184 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.185 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.185 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.185 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.185 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.185 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.185 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.185 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.185 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.186 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.186 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.186 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.186 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.186 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.186 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.186 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.186 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "telemetry", 2024-03-28T22:47:52.187 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.187 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.187 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.187 INFO:teuthology.orchestra.run.smithi043.stdout: "channel_basic": { 2024-03-28T22:47:52.187 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "channel_basic", 2024-03-28T22:47:52.187 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.187 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.187 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.187 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.188 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.188 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.188 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.188 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-28T22:47:52.188 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.188 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.188 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.188 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.189 INFO:teuthology.orchestra.run.smithi043.stdout: "channel_crash": { 2024-03-28T22:47:52.189 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "channel_crash", 2024-03-28T22:47:52.189 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.189 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.189 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.189 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.189 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.189 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.190 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.190 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-28T22:47:52.190 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.190 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.190 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.190 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.190 INFO:teuthology.orchestra.run.smithi043.stdout: "channel_device": { 2024-03-28T22:47:52.191 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "channel_device", 2024-03-28T22:47:52.191 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.191 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.191 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.191 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.191 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.191 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.191 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.191 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-28T22:47:52.192 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.192 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.192 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.192 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.192 INFO:teuthology.orchestra.run.smithi043.stdout: "channel_ident": { 2024-03-28T22:47:52.192 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "channel_ident", 2024-03-28T22:47:52.192 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.192 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.193 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.193 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.193 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.193 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.193 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.193 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-28T22:47:52.193 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.193 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.194 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.194 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.194 INFO:teuthology.orchestra.run.smithi043.stdout: "channel_perf": { 2024-03-28T22:47:52.194 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "channel_perf", 2024-03-28T22:47:52.194 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.194 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.194 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.194 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.195 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.195 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.195 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.195 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-28T22:47:52.195 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.195 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.195 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.195 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.195 INFO:teuthology.orchestra.run.smithi043.stdout: "contact": { 2024-03-28T22:47:52.196 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "contact", 2024-03-28T22:47:52.196 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.196 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.196 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.196 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.196 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.196 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.196 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.196 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.197 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.197 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.197 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.197 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.197 INFO:teuthology.orchestra.run.smithi043.stdout: "description": { 2024-03-28T22:47:52.197 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "description", 2024-03-28T22:47:52.197 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.197 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.198 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.198 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.198 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.198 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.198 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.198 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.198 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.198 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.198 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.199 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.199 INFO:teuthology.orchestra.run.smithi043.stdout: "device_url": { 2024-03-28T22:47:52.199 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "device_url", 2024-03-28T22:47:52.199 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.199 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.199 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.199 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-28T22:47:52.199 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.200 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.200 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.200 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.200 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.200 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.200 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.200 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.200 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": { 2024-03-28T22:47:52.200 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "enabled", 2024-03-28T22:47:52.201 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.201 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.201 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.201 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.201 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.201 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.201 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.201 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.201 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.202 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.202 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.202 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.202 INFO:teuthology.orchestra.run.smithi043.stdout: "interval": { 2024-03-28T22:47:52.202 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "interval", 2024-03-28T22:47:52.202 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.202 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.202 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.203 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "24", 2024-03-28T22:47:52.203 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "8", 2024-03-28T22:47:52.203 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.203 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.203 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.203 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.203 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.203 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.203 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.204 INFO:teuthology.orchestra.run.smithi043.stdout: "last_opt_revision": { 2024-03-28T22:47:52.204 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "last_opt_revision", 2024-03-28T22:47:52.204 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.204 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.204 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.204 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "1", 2024-03-28T22:47:52.204 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.204 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.204 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.205 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.205 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.205 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.205 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.205 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.205 INFO:teuthology.orchestra.run.smithi043.stdout: "leaderboard": { 2024-03-28T22:47:52.205 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "leaderboard", 2024-03-28T22:47:52.205 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.205 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.206 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.206 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.206 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.206 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.206 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.206 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.206 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.206 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.207 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.207 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.207 INFO:teuthology.orchestra.run.smithi043.stdout: "leaderboard_description": { 2024-03-28T22:47:52.207 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "leaderboard_description", 2024-03-28T22:47:52.207 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.207 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.207 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.207 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.207 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.208 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.208 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.208 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.208 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.208 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.208 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.208 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.208 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.208 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.209 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.209 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.209 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.209 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.209 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.209 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.209 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.209 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.210 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.210 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.210 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.210 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.210 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.210 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.210 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.210 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.210 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.211 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.211 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.211 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.211 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.211 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.211 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.211 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.211 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.211 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.212 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.212 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.212 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.212 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.212 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.212 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.212 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.212 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.213 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.213 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.213 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.213 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.213 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.213 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.213 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.213 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.213 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.214 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.214 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.214 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.214 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.214 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.214 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.214 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.214 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.215 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.215 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.215 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.215 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.215 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.215 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.215 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.215 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.216 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.216 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.216 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.216 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.216 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.216 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.216 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.216 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.216 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.217 INFO:teuthology.orchestra.run.smithi043.stdout: "organization": { 2024-03-28T22:47:52.217 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "organization", 2024-03-28T22:47:52.217 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.217 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.217 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.217 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.217 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.217 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.218 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.218 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.218 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.218 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.218 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.218 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.218 INFO:teuthology.orchestra.run.smithi043.stdout: "proxy": { 2024-03-28T22:47:52.218 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "proxy", 2024-03-28T22:47:52.219 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.219 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.219 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.219 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.219 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.219 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.219 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.219 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.219 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.220 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.220 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.220 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.220 INFO:teuthology.orchestra.run.smithi043.stdout: "url": { 2024-03-28T22:47:52.220 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "url", 2024-03-28T22:47:52.220 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.220 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.220 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.220 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-28T22:47:52.221 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.221 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.221 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.221 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.221 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.221 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.221 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.221 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.221 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.222 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.222 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.222 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "test_orchestrator", 2024-03-28T22:47:52.222 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.222 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.222 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.222 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.222 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.223 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.223 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.223 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.223 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.223 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.223 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.223 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.223 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.224 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.224 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.224 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.224 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.224 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.224 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.224 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.224 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.224 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.225 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.225 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.225 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.225 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.225 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.225 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.225 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.225 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.225 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.226 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.226 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.226 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.226 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.226 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.226 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.226 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.226 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.227 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.227 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.227 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.227 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.227 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.227 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.227 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.227 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.227 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.228 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.228 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.228 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.228 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.228 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.228 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.228 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.228 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.228 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.229 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.229 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.229 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.229 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.229 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.229 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.229 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.229 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.230 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.230 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.230 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.230 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.230 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.230 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.230 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.230 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.230 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.231 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.231 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.231 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "volumes", 2024-03-28T22:47:52.231 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.231 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.231 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.231 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.231 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.231 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.232 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.232 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.232 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.232 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.232 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.232 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.232 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.232 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.233 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.233 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.233 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.233 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.233 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.233 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.233 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.233 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.233 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.233 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.234 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.234 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.234 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.234 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.234 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.234 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.234 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.237 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.237 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.237 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.237 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.237 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.238 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.238 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.238 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.238 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.238 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.238 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.238 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.239 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.239 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.239 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.239 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.239 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.239 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.239 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.239 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.239 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.240 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.240 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.240 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.240 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.240 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.240 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.240 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.240 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.240 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.241 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.241 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.241 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.241 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.241 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.241 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.241 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.241 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.241 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.242 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.242 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.242 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.242 INFO:teuthology.orchestra.run.smithi043.stdout: "max_concurrent_clones": { 2024-03-28T22:47:52.242 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "max_concurrent_clones", 2024-03-28T22:47:52.242 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.242 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.242 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.242 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "4", 2024-03-28T22:47:52.243 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.243 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.243 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.243 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-28T22:47:52.243 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.243 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.243 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.243 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.243 INFO:teuthology.orchestra.run.smithi043.stdout: "periodic_async_work": { 2024-03-28T22:47:52.244 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "periodic_async_work", 2024-03-28T22:47:52.244 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.244 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.244 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.244 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.244 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.244 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.244 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.244 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Periodically check for async work", 2024-03-28T22:47:52.245 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.245 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.245 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.245 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.245 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot_clone_delay": { 2024-03-28T22:47:52.245 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "snapshot_clone_delay", 2024-03-28T22:47:52.245 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.245 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.245 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.246 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-03-28T22:47:52.246 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.246 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.246 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.246 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-28T22:47:52.246 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.246 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.246 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.247 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.247 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot_clone_no_wait": { 2024-03-28T22:47:52.247 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "snapshot_clone_no_wait", 2024-03-28T22:47:52.247 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.247 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.247 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.247 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.247 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.247 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.248 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.248 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-28T22:47:52.248 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.248 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.248 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.248 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.248 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.248 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.249 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.249 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "zabbix", 2024-03-28T22:47:52.249 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.249 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.249 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.249 INFO:teuthology.orchestra.run.smithi043.stdout: "discovery_interval": { 2024-03-28T22:47:52.249 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "discovery_interval", 2024-03-28T22:47:52.249 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "uint", 2024-03-28T22:47:52.249 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.250 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.250 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "100", 2024-03-28T22:47:52.250 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.250 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.250 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.250 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.250 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.250 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.251 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.251 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.251 INFO:teuthology.orchestra.run.smithi043.stdout: "identifier": { 2024-03-28T22:47:52.251 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "identifier", 2024-03-28T22:47:52.251 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.251 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.251 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.251 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.252 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.252 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.252 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.252 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.252 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.252 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.252 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.252 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.252 INFO:teuthology.orchestra.run.smithi043.stdout: "interval": { 2024-03-28T22:47:52.253 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "interval", 2024-03-28T22:47:52.253 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:52.253 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.253 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.253 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "60", 2024-03-28T22:47:52.253 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.253 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.253 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.254 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.254 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.254 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.254 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.254 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.254 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.254 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.254 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.255 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.255 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.255 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.255 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.255 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.255 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.255 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.255 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.256 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.256 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.256 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.256 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.256 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.256 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.256 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.256 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.256 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.257 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.257 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.257 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.257 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.257 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.257 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.257 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.257 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.257 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.258 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.258 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.258 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.258 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.258 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.258 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.258 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.259 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.259 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.259 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.259 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.259 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.259 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.259 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.259 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.259 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.260 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.260 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.260 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.260 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.260 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.260 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.260 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.260 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.260 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.261 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.261 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.261 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.261 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.261 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.261 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.261 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.261 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.262 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.262 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.262 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.262 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.262 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.262 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.262 INFO:teuthology.orchestra.run.smithi043.stdout: "zabbix_host": { 2024-03-28T22:47:52.262 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "zabbix_host", 2024-03-28T22:47:52.262 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.263 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.263 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.263 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.263 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.263 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.263 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.263 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.263 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.263 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.264 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.264 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.264 INFO:teuthology.orchestra.run.smithi043.stdout: "zabbix_port": { 2024-03-28T22:47:52.264 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "zabbix_port", 2024-03-28T22:47:52.264 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.264 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.264 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.264 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10051", 2024-03-28T22:47:52.264 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.265 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.265 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.265 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.265 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.265 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.265 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.265 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.265 INFO:teuthology.orchestra.run.smithi043.stdout: "zabbix_sender": { 2024-03-28T22:47:52.265 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "zabbix_sender", 2024-03-28T22:47:52.266 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.266 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.266 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.266 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-28T22:47:52.266 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.266 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.266 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.266 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.267 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.267 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.267 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.267 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.267 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.267 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.267 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-28T22:47:52.267 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.267 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.268 INFO:teuthology.orchestra.run.smithi043.stdout: "modules": [ 2024-03-28T22:47:52.268 INFO:teuthology.orchestra.run.smithi043.stdout: "cephadm", 2024-03-28T22:47:52.268 INFO:teuthology.orchestra.run.smithi043.stdout: "dashboard", 2024-03-28T22:47:52.268 INFO:teuthology.orchestra.run.smithi043.stdout: "iostat", 2024-03-28T22:47:52.268 INFO:teuthology.orchestra.run.smithi043.stdout: "nfs", 2024-03-28T22:47:52.268 INFO:teuthology.orchestra.run.smithi043.stdout: "restful", 2024-03-28T22:47:52.268 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_schedule" 2024-03-28T22:47:52.268 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.269 INFO:teuthology.orchestra.run.smithi043.stdout: "available_modules": [ 2024-03-28T22:47:52.269 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.269 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "alerts", 2024-03-28T22:47:52.269 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.269 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.269 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.269 INFO:teuthology.orchestra.run.smithi043.stdout: "interval": { 2024-03-28T22:47:52.269 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "interval", 2024-03-28T22:47:52.270 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:52.270 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.270 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.270 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "60", 2024-03-28T22:47:52.270 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.270 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.270 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.270 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "How frequently to reexamine health status", 2024-03-28T22:47:52.271 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.271 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.271 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.271 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.271 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.271 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.271 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.271 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.271 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.272 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.272 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.272 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.272 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.272 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.272 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.272 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.272 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.272 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.273 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.273 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.273 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.273 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.273 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.273 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.273 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.273 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.273 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.274 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.274 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.274 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.274 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.274 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.274 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.274 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.274 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.275 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.275 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.275 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.275 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.275 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.275 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.275 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.275 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.276 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.276 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.276 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.276 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.276 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.276 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.276 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.276 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.276 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.277 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.277 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.277 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.277 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.277 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.277 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.277 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.277 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.278 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.278 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.278 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.278 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.278 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.278 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.278 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.278 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.279 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.279 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.279 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.279 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.279 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.279 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_destination": { 2024-03-28T22:47:52.279 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_destination", 2024-03-28T22:47:52.279 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.280 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.280 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.280 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.280 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.280 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.280 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.280 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Email address to send alerts to", 2024-03-28T22:47:52.280 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.280 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.281 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.281 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_from_name": { 2024-03-28T22:47:52.281 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_from_name", 2024-03-28T22:47:52.281 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.281 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.281 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.281 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "Ceph", 2024-03-28T22:47:52.281 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.282 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.282 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.282 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Email From: name", 2024-03-28T22:47:52.282 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.282 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.282 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.282 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.282 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_host": { 2024-03-28T22:47:52.283 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_host", 2024-03-28T22:47:52.283 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.283 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.283 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.283 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.283 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.283 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.283 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.283 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "SMTP server", 2024-03-28T22:47:52.284 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.284 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.284 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.284 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.284 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_password": { 2024-03-28T22:47:52.284 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_password", 2024-03-28T22:47:52.284 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.284 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.285 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.285 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.285 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.285 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.285 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.285 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Password to authenticate with", 2024-03-28T22:47:52.285 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.285 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.285 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.286 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.286 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_port": { 2024-03-28T22:47:52.286 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_port", 2024-03-28T22:47:52.286 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.286 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.286 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.286 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "465", 2024-03-28T22:47:52.286 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.287 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.287 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.287 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "SMTP port", 2024-03-28T22:47:52.287 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.287 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.287 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.287 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_sender": { 2024-03-28T22:47:52.287 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_sender", 2024-03-28T22:47:52.287 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.288 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.288 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.288 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.288 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.288 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.288 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.288 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "SMTP envelope sender", 2024-03-28T22:47:52.288 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.289 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.289 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.289 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.289 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_ssl": { 2024-03-28T22:47:52.289 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_ssl", 2024-03-28T22:47:52.289 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.289 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.289 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.289 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.290 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.290 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.290 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.290 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-28T22:47:52.290 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.290 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.290 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.290 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.291 INFO:teuthology.orchestra.run.smithi043.stdout: "smtp_user": { 2024-03-28T22:47:52.291 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "smtp_user", 2024-03-28T22:47:52.291 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.291 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.291 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.291 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.291 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.291 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.291 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.292 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "User to authenticate as", 2024-03-28T22:47:52.292 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.292 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.292 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.292 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.292 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.292 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.292 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.292 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "balancer", 2024-03-28T22:47:52.293 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.293 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.293 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.293 INFO:teuthology.orchestra.run.smithi043.stdout: "active": { 2024-03-28T22:47:52.293 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "active", 2024-03-28T22:47:52.293 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.293 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.293 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.293 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.294 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.294 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.294 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.294 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "automatically balance PGs across cluster", 2024-03-28T22:47:52.294 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.294 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.294 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.294 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.295 INFO:teuthology.orchestra.run.smithi043.stdout: "begin_time": { 2024-03-28T22:47:52.295 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "begin_time", 2024-03-28T22:47:52.295 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.295 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.295 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.295 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0000", 2024-03-28T22:47:52.295 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.295 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.295 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.296 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "beginning time of day to automatically balance", 2024-03-28T22:47:52.296 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T22:47:52.296 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.296 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.296 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.296 INFO:teuthology.orchestra.run.smithi043.stdout: "begin_weekday": { 2024-03-28T22:47:52.296 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "begin_weekday", 2024-03-28T22:47:52.296 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "uint", 2024-03-28T22:47:52.296 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.297 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.297 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-03-28T22:47:52.297 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "0", 2024-03-28T22:47:52.297 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "6", 2024-03-28T22:47:52.297 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.297 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-28T22:47:52.297 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T22:47:52.297 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.297 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.298 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.298 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_compat_max_iterations": { 2024-03-28T22:47:52.298 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "crush_compat_max_iterations", 2024-03-28T22:47:52.298 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "uint", 2024-03-28T22:47:52.298 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.298 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.298 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "25", 2024-03-28T22:47:52.298 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "1", 2024-03-28T22:47:52.299 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "250", 2024-03-28T22:47:52.299 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.299 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-28T22:47:52.299 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.299 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.299 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.299 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.299 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_compat_metrics": { 2024-03-28T22:47:52.299 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "crush_compat_metrics", 2024-03-28T22:47:52.300 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.300 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.300 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.300 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "pgs,objects,bytes", 2024-03-28T22:47:52.300 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.300 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.300 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.300 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-28T22:47:52.300 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-28T22:47:52.301 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.301 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.301 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.301 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_compat_step": { 2024-03-28T22:47:52.301 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "crush_compat_step", 2024-03-28T22:47:52.301 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-03-28T22:47:52.301 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.301 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.301 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0.5", 2024-03-28T22:47:52.302 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "0.001", 2024-03-28T22:47:52.302 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "0.999", 2024-03-28T22:47:52.302 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.302 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "aggressiveness of optimization", 2024-03-28T22:47:52.302 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-28T22:47:52.302 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.302 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.302 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.303 INFO:teuthology.orchestra.run.smithi043.stdout: "end_time": { 2024-03-28T22:47:52.303 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "end_time", 2024-03-28T22:47:52.303 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.303 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.303 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.303 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "2359", 2024-03-28T22:47:52.303 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.303 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.303 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.304 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "ending time of day to automatically balance", 2024-03-28T22:47:52.304 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T22:47:52.304 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.304 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.304 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.304 INFO:teuthology.orchestra.run.smithi043.stdout: "end_weekday": { 2024-03-28T22:47:52.304 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "end_weekday", 2024-03-28T22:47:52.304 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "uint", 2024-03-28T22:47:52.304 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.305 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.305 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-03-28T22:47:52.305 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "0", 2024-03-28T22:47:52.305 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "6", 2024-03-28T22:47:52.305 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.305 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-28T22:47:52.305 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T22:47:52.305 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.305 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.306 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.306 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.306 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.306 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.306 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.306 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.306 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.306 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.307 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.307 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.307 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.307 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.307 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.307 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.307 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.307 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.307 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.308 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.308 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.308 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.308 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.308 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.308 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.308 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.308 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.308 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.309 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.309 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.309 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.309 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.309 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.309 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.309 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.309 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.309 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.310 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.310 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.310 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.310 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.310 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.310 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.310 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.310 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.311 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.311 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.311 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.311 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.311 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.311 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.311 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.311 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.312 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.312 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.312 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.312 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.312 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.312 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.312 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.312 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.312 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.313 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.313 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.313 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.313 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.313 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.313 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.313 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.313 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.313 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.314 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.314 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.314 INFO:teuthology.orchestra.run.smithi043.stdout: "min_score": { 2024-03-28T22:47:52.314 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "min_score", 2024-03-28T22:47:52.314 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-03-28T22:47:52.314 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.314 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.314 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-03-28T22:47:52.315 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.315 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.315 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.315 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-28T22:47:52.315 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.315 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.315 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.315 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.315 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": { 2024-03-28T22:47:52.316 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mode", 2024-03-28T22:47:52.316 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.316 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.316 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.316 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "upmap", 2024-03-28T22:47:52.316 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.316 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.316 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.317 INFO:teuthology.orchestra.run.smithi043.stdout: "crush-compat", 2024-03-28T22:47:52.317 INFO:teuthology.orchestra.run.smithi043.stdout: "none", 2024-03-28T22:47:52.317 INFO:teuthology.orchestra.run.smithi043.stdout: "read", 2024-03-28T22:47:52.317 INFO:teuthology.orchestra.run.smithi043.stdout: "upmap", 2024-03-28T22:47:52.317 INFO:teuthology.orchestra.run.smithi043.stdout: "upmap-read" 2024-03-28T22:47:52.317 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.317 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Balancer mode", 2024-03-28T22:47:52.317 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.317 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.318 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.318 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.318 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_ids": { 2024-03-28T22:47:52.318 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "pool_ids", 2024-03-28T22:47:52.318 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.318 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.318 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.318 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.319 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.319 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.319 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.319 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-28T22:47:52.319 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.319 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.319 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.319 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.319 INFO:teuthology.orchestra.run.smithi043.stdout: "sleep_interval": { 2024-03-28T22:47:52.320 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sleep_interval", 2024-03-28T22:47:52.320 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:52.320 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.320 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.320 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "60", 2024-03-28T22:47:52.320 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.320 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.320 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.320 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-28T22:47:52.321 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.321 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.321 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.321 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.321 INFO:teuthology.orchestra.run.smithi043.stdout: "upmap_max_deviation": { 2024-03-28T22:47:52.321 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "upmap_max_deviation", 2024-03-28T22:47:52.321 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.321 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.321 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.322 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "5", 2024-03-28T22:47:52.322 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "1", 2024-03-28T22:47:52.322 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.322 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.322 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-28T22:47:52.322 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-28T22:47:52.322 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.322 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.323 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.323 INFO:teuthology.orchestra.run.smithi043.stdout: "upmap_max_optimizations": { 2024-03-28T22:47:52.323 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "upmap_max_optimizations", 2024-03-28T22:47:52.323 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "uint", 2024-03-28T22:47:52.323 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.323 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.323 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-03-28T22:47:52.323 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.323 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.324 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.324 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-28T22:47:52.324 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.324 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.324 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.324 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.324 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.324 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.325 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.325 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cephadm", 2024-03-28T22:47:52.325 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.325 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.325 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.325 INFO:teuthology.orchestra.run.smithi043.stdout: "agent_down_multiplier": { 2024-03-28T22:47:52.325 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "agent_down_multiplier", 2024-03-28T22:47:52.325 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-03-28T22:47:52.325 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.326 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.326 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "3.0", 2024-03-28T22:47:52.326 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.326 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.326 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.326 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-28T22:47:52.326 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.326 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.326 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.327 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.327 INFO:teuthology.orchestra.run.smithi043.stdout: "agent_refresh_rate": { 2024-03-28T22:47:52.327 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "agent_refresh_rate", 2024-03-28T22:47:52.327 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:52.327 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.327 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.327 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "20", 2024-03-28T22:47:52.327 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.327 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.328 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.328 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-28T22:47:52.328 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.328 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.328 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.328 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.328 INFO:teuthology.orchestra.run.smithi043.stdout: "agent_starting_port": { 2024-03-28T22:47:52.328 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "agent_starting_port", 2024-03-28T22:47:52.328 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.329 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.329 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.329 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "4721", 2024-03-28T22:47:52.329 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.329 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.329 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.329 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-28T22:47:52.329 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.330 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.330 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.330 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.330 INFO:teuthology.orchestra.run.smithi043.stdout: "allow_ptrace": { 2024-03-28T22:47:52.330 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "allow_ptrace", 2024-03-28T22:47:52.330 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.330 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.330 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.331 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.331 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.331 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.331 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.331 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-28T22:47:52.331 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-28T22:47:52.331 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.331 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.331 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.332 INFO:teuthology.orchestra.run.smithi043.stdout: "autotune_interval": { 2024-03-28T22:47:52.332 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "autotune_interval", 2024-03-28T22:47:52.332 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:52.332 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.332 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.332 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "600", 2024-03-28T22:47:52.332 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.332 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.333 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.333 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-28T22:47:52.333 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.333 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.333 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.333 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.333 INFO:teuthology.orchestra.run.smithi043.stdout: "autotune_memory_target_ratio": { 2024-03-28T22:47:52.333 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "autotune_memory_target_ratio", 2024-03-28T22:47:52.333 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-03-28T22:47:52.334 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.334 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.334 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0.7", 2024-03-28T22:47:52.334 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.334 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.334 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.334 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-28T22:47:52.334 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.335 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.335 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.335 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.335 INFO:teuthology.orchestra.run.smithi043.stdout: "cephadm_log_destination": { 2024-03-28T22:47:52.335 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cephadm_log_destination", 2024-03-28T22:47:52.335 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.335 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.335 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.335 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.336 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.336 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.336 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.336 INFO:teuthology.orchestra.run.smithi043.stdout: "file", 2024-03-28T22:47:52.336 INFO:teuthology.orchestra.run.smithi043.stdout: "file,syslog", 2024-03-28T22:47:52.336 INFO:teuthology.orchestra.run.smithi043.stdout: "syslog" 2024-03-28T22:47:52.336 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.336 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-28T22:47:52.336 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.337 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.337 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.337 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.337 INFO:teuthology.orchestra.run.smithi043.stdout: "cgroups_split": { 2024-03-28T22:47:52.337 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cgroups_split", 2024-03-28T22:47:52.337 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.337 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.337 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.338 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.338 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.338 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.338 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.338 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-28T22:47:52.338 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.338 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.338 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.338 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.339 INFO:teuthology.orchestra.run.smithi043.stdout: "config_checks_enabled": { 2024-03-28T22:47:52.339 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "config_checks_enabled", 2024-03-28T22:47:52.339 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.339 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.339 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.339 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.339 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.339 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.340 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.340 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-28T22:47:52.340 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.340 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.340 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.340 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.340 INFO:teuthology.orchestra.run.smithi043.stdout: "config_dashboard": { 2024-03-28T22:47:52.340 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "config_dashboard", 2024-03-28T22:47:52.340 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.341 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.341 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.341 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.341 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.341 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.341 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.341 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-28T22:47:52.341 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.341 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.342 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.342 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.342 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_alertmanager": { 2024-03-28T22:47:52.342 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_alertmanager", 2024-03-28T22:47:52.342 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.342 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.342 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.342 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-28T22:47:52.343 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.343 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.343 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.343 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Prometheus container image", 2024-03-28T22:47:52.343 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.343 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.343 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.343 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.343 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_base": { 2024-03-28T22:47:52.344 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_base", 2024-03-28T22:47:52.344 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.344 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.344 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.344 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-28T22:47:52.344 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.344 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.344 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.344 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Container image name, without the tag", 2024-03-28T22:47:52.345 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.345 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.345 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.345 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.345 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_elasticsearch": { 2024-03-28T22:47:52.345 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_elasticsearch", 2024-03-28T22:47:52.345 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.345 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.346 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.346 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-28T22:47:52.346 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.346 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.346 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.346 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "elasticsearch container image", 2024-03-28T22:47:52.346 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.346 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.346 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.347 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.347 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_grafana": { 2024-03-28T22:47:52.347 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_grafana", 2024-03-28T22:47:52.347 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.347 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.347 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.347 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-28T22:47:52.347 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.347 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.348 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.348 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Prometheus container image", 2024-03-28T22:47:52.348 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.348 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.348 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.348 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.348 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_haproxy": { 2024-03-28T22:47:52.348 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_haproxy", 2024-03-28T22:47:52.349 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.349 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.349 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.349 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-28T22:47:52.349 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.349 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.349 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.349 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "HAproxy container image", 2024-03-28T22:47:52.349 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.350 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.350 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.350 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.350 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_jaeger_agent": { 2024-03-28T22:47:52.350 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_jaeger_agent", 2024-03-28T22:47:52.350 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.350 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.350 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.351 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-28T22:47:52.351 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.351 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.351 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.351 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Jaeger agent container image", 2024-03-28T22:47:52.351 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.351 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.351 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.352 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.352 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_jaeger_collector": { 2024-03-28T22:47:52.352 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_jaeger_collector", 2024-03-28T22:47:52.352 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.352 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.352 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.352 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-28T22:47:52.352 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.353 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.353 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.353 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Jaeger collector container image", 2024-03-28T22:47:52.353 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.353 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.353 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.353 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.353 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_jaeger_query": { 2024-03-28T22:47:52.354 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_jaeger_query", 2024-03-28T22:47:52.354 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.354 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.354 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.354 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-28T22:47:52.354 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.354 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.354 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.355 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Jaeger query container image", 2024-03-28T22:47:52.355 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.355 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.355 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.355 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.355 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_keepalived": { 2024-03-28T22:47:52.355 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_keepalived", 2024-03-28T22:47:52.355 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.355 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.356 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.356 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-28T22:47:52.356 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.356 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.356 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.356 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Keepalived container image", 2024-03-28T22:47:52.356 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.356 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.357 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.357 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.357 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_loki": { 2024-03-28T22:47:52.357 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_loki", 2024-03-28T22:47:52.357 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.357 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.357 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.357 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-28T22:47:52.357 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.358 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.358 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.358 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Loki container image", 2024-03-28T22:47:52.358 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.358 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.358 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.358 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.358 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_node_exporter": { 2024-03-28T22:47:52.359 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_node_exporter", 2024-03-28T22:47:52.359 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.359 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.359 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.359 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-28T22:47:52.359 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.359 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.359 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.360 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Prometheus container image", 2024-03-28T22:47:52.360 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.360 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.360 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.360 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.360 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_nvmeof": { 2024-03-28T22:47:52.360 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_nvmeof", 2024-03-28T22:47:52.360 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.361 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.361 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.361 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-28T22:47:52.361 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.361 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.361 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.361 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Nvme-of container image", 2024-03-28T22:47:52.361 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.361 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.362 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.362 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.362 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_prometheus": { 2024-03-28T22:47:52.362 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_prometheus", 2024-03-28T22:47:52.362 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.362 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.362 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.362 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-28T22:47:52.363 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.363 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.363 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.363 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Prometheus container image", 2024-03-28T22:47:52.363 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.363 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.363 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.363 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.363 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_promtail": { 2024-03-28T22:47:52.364 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_promtail", 2024-03-28T22:47:52.364 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.364 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.364 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.364 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-28T22:47:52.364 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.364 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.364 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.364 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Promtail container image", 2024-03-28T22:47:52.365 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.365 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.365 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.365 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.365 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image_snmp_gateway": { 2024-03-28T22:47:52.365 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_image_snmp_gateway", 2024-03-28T22:47:52.365 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.365 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.366 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.366 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-28T22:47:52.366 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.366 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.366 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.366 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "SNMP Gateway container image", 2024-03-28T22:47:52.366 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.366 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.367 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.367 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.367 INFO:teuthology.orchestra.run.smithi043.stdout: "container_init": { 2024-03-28T22:47:52.367 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "container_init", 2024-03-28T22:47:52.367 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.367 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.367 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.367 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.367 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.368 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.368 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.368 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Run podman/docker with `--init`", 2024-03-28T22:47:52.368 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.368 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.368 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.368 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.368 INFO:teuthology.orchestra.run.smithi043.stdout: "daemon_cache_timeout": { 2024-03-28T22:47:52.369 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "daemon_cache_timeout", 2024-03-28T22:47:52.369 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:52.369 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.369 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.369 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "600", 2024-03-28T22:47:52.369 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.369 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.369 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.369 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-28T22:47:52.370 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.370 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.370 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.370 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.370 INFO:teuthology.orchestra.run.smithi043.stdout: "default_cephadm_command_timeout": { 2024-03-28T22:47:52.370 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default_cephadm_command_timeout", 2024-03-28T22:47:52.370 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.370 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.371 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.371 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "900", 2024-03-28T22:47:52.371 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.371 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.371 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.371 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-28T22:47:52.371 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.371 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.371 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.372 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.372 INFO:teuthology.orchestra.run.smithi043.stdout: "default_registry": { 2024-03-28T22:47:52.372 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default_registry", 2024-03-28T22:47:52.372 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.372 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.372 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.372 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "docker.io", 2024-03-28T22:47:52.372 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.373 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.373 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.373 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-28T22:47:52.373 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.373 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.373 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.373 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.373 INFO:teuthology.orchestra.run.smithi043.stdout: "device_cache_timeout": { 2024-03-28T22:47:52.373 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "device_cache_timeout", 2024-03-28T22:47:52.374 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:52.374 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.374 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.374 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "1800", 2024-03-28T22:47:52.374 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.374 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.374 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.374 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "seconds to cache device inventory", 2024-03-28T22:47:52.375 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.375 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.375 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.375 INFO:teuthology.orchestra.run.smithi043.stdout: "device_enhanced_scan": { 2024-03-28T22:47:52.375 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "device_enhanced_scan", 2024-03-28T22:47:52.375 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.375 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.376 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.376 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.376 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.376 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.376 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.376 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-28T22:47:52.376 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.376 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.376 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.377 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.377 INFO:teuthology.orchestra.run.smithi043.stdout: "facts_cache_timeout": { 2024-03-28T22:47:52.377 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "facts_cache_timeout", 2024-03-28T22:47:52.377 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:52.377 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.377 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.377 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "60", 2024-03-28T22:47:52.377 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.377 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.378 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.378 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "seconds to cache host facts data", 2024-03-28T22:47:52.378 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.378 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.378 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.378 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.378 INFO:teuthology.orchestra.run.smithi043.stdout: "host_check_interval": { 2024-03-28T22:47:52.378 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "host_check_interval", 2024-03-28T22:47:52.379 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:52.379 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.379 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.379 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "600", 2024-03-28T22:47:52.379 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.379 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.379 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.379 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how frequently to perform a host check", 2024-03-28T22:47:52.380 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.380 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.380 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.380 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.380 INFO:teuthology.orchestra.run.smithi043.stdout: "hw_monitoring": { 2024-03-28T22:47:52.380 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "hw_monitoring", 2024-03-28T22:47:52.380 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.380 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.380 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.381 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.381 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.381 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.381 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.381 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-28T22:47:52.381 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.381 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.381 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.381 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.382 INFO:teuthology.orchestra.run.smithi043.stdout: "inventory_list_all": { 2024-03-28T22:47:52.382 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "inventory_list_all", 2024-03-28T22:47:52.382 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.382 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.382 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.382 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.382 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.382 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.383 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.383 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-28T22:47:52.383 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.383 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.383 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.383 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.383 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.383 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.383 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.384 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.384 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.384 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.384 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.384 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.384 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.384 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.384 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.384 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.385 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.385 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.385 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.385 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.385 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.385 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.385 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.385 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.386 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.386 INFO:teuthology.orchestra.run.smithi043.stdout: "log_refresh_metadata": { 2024-03-28T22:47:52.386 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_refresh_metadata", 2024-03-28T22:47:52.386 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.386 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.386 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.386 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.386 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.387 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.387 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.387 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-28T22:47:52.387 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.387 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.387 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.387 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.387 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.388 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.388 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.388 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.388 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.388 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.388 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.388 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.388 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.388 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-28T22:47:52.389 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.389 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.389 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.389 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.389 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.389 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.389 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.389 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.390 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.390 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.390 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.390 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.390 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.390 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.390 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.390 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.391 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.391 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.391 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.391 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.391 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.391 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.391 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.391 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.391 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.391 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.392 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.392 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.392 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.392 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.392 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.392 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.392 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.392 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.393 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.393 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.393 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.393 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.393 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.393 INFO:teuthology.orchestra.run.smithi043.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-28T22:47:52.393 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-28T22:47:52.393 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.393 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.394 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.394 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.394 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.394 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.394 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.394 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-28T22:47:52.394 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.394 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.395 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.395 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.395 INFO:teuthology.orchestra.run.smithi043.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-28T22:47:52.395 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-28T22:47:52.395 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.395 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.395 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.395 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "*", 2024-03-28T22:47:52.395 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.395 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.396 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.396 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-28T22:47:52.396 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.396 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.396 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.396 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.396 INFO:teuthology.orchestra.run.smithi043.stdout: "max_count_per_host": { 2024-03-28T22:47:52.396 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "max_count_per_host", 2024-03-28T22:47:52.396 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.397 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.397 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.397 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-03-28T22:47:52.397 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.397 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.397 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.397 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "max number of daemons per service per host", 2024-03-28T22:47:52.397 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.397 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.398 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.398 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.398 INFO:teuthology.orchestra.run.smithi043.stdout: "max_osd_draining_count": { 2024-03-28T22:47:52.398 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "max_osd_draining_count", 2024-03-28T22:47:52.398 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.398 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.398 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.398 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-03-28T22:47:52.399 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.399 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.399 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.399 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-28T22:47:52.399 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.399 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.399 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.399 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.399 INFO:teuthology.orchestra.run.smithi043.stdout: "migration_current": { 2024-03-28T22:47:52.400 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "migration_current", 2024-03-28T22:47:52.400 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.400 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.400 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.400 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.400 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.400 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.400 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.401 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "internal - do not modify", 2024-03-28T22:47:52.401 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.401 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.401 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.401 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.401 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": { 2024-03-28T22:47:52.401 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mode", 2024-03-28T22:47:52.401 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.401 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.402 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.402 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "root", 2024-03-28T22:47:52.402 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.402 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.402 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.402 INFO:teuthology.orchestra.run.smithi043.stdout: "cephadm-package", 2024-03-28T22:47:52.402 INFO:teuthology.orchestra.run.smithi043.stdout: "root" 2024-03-28T22:47:52.402 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.403 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "mode for remote execution of cephadm", 2024-03-28T22:47:52.403 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.403 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.403 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.403 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.403 INFO:teuthology.orchestra.run.smithi043.stdout: "oob_default_addr": { 2024-03-28T22:47:52.403 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "oob_default_addr", 2024-03-28T22:47:52.403 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.403 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.404 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.404 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "169.254.1.1", 2024-03-28T22:47:52.404 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.404 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.404 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.404 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-28T22:47:52.404 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.404 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.405 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.405 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.405 INFO:teuthology.orchestra.run.smithi043.stdout: "prometheus_alerts_path": { 2024-03-28T22:47:52.405 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "prometheus_alerts_path", 2024-03-28T22:47:52.405 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.405 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.405 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.405 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-28T22:47:52.405 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.406 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.406 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.406 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-28T22:47:52.406 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.406 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.406 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.406 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.406 INFO:teuthology.orchestra.run.smithi043.stdout: "registry_insecure": { 2024-03-28T22:47:52.407 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "registry_insecure", 2024-03-28T22:47:52.407 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.407 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.407 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.407 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.407 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.407 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.407 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.407 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-28T22:47:52.407 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.408 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.408 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.408 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.408 INFO:teuthology.orchestra.run.smithi043.stdout: "registry_password": { 2024-03-28T22:47:52.408 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "registry_password", 2024-03-28T22:47:52.408 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.408 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.408 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.408 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.409 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.409 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.409 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.409 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-28T22:47:52.409 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.409 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.409 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.410 INFO:teuthology.orchestra.run.smithi043.stdout: "registry_url": { 2024-03-28T22:47:52.410 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "registry_url", 2024-03-28T22:47:52.410 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.410 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.410 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.410 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.410 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.410 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.410 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.411 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-28T22:47:52.411 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.411 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.411 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.411 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.411 INFO:teuthology.orchestra.run.smithi043.stdout: "registry_username": { 2024-03-28T22:47:52.411 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "registry_username", 2024-03-28T22:47:52.411 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.412 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.412 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.412 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.412 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.412 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.412 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.412 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-28T22:47:52.412 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.412 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.413 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.413 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.413 INFO:teuthology.orchestra.run.smithi043.stdout: "secure_monitoring_stack": { 2024-03-28T22:47:52.413 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "secure_monitoring_stack", 2024-03-28T22:47:52.413 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.413 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.413 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.413 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.414 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.414 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.414 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.414 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-28T22:47:52.414 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.414 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.414 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.414 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.415 INFO:teuthology.orchestra.run.smithi043.stdout: "service_discovery_port": { 2024-03-28T22:47:52.415 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "service_discovery_port", 2024-03-28T22:47:52.415 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.415 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.415 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.415 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "8765", 2024-03-28T22:47:52.415 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.415 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.415 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.416 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "cephadm service discovery port", 2024-03-28T22:47:52.416 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.416 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.416 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.416 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.416 INFO:teuthology.orchestra.run.smithi043.stdout: "ssh_config_file": { 2024-03-28T22:47:52.416 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ssh_config_file", 2024-03-28T22:47:52.416 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.416 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.417 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.417 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.417 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.417 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.417 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.417 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-28T22:47:52.417 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.417 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.418 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.418 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.418 INFO:teuthology.orchestra.run.smithi043.stdout: "use_agent": { 2024-03-28T22:47:52.418 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "use_agent", 2024-03-28T22:47:52.418 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.418 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.418 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.418 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.419 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.419 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.419 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.419 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-28T22:47:52.419 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.419 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.419 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.419 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.419 INFO:teuthology.orchestra.run.smithi043.stdout: "use_repo_digest": { 2024-03-28T22:47:52.420 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "use_repo_digest", 2024-03-28T22:47:52.420 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.420 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.420 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.420 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.420 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.420 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.420 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.420 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-28T22:47:52.421 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.421 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.421 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.421 INFO:teuthology.orchestra.run.smithi043.stdout: "warn_on_failed_host_check": { 2024-03-28T22:47:52.421 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "warn_on_failed_host_check", 2024-03-28T22:47:52.421 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.421 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.421 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.422 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.422 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.422 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.422 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.422 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "raise a health warning if the host check fails", 2024-03-28T22:47:52.422 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.422 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.422 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.423 INFO:teuthology.orchestra.run.smithi043.stdout: "warn_on_stray_daemons": { 2024-03-28T22:47:52.423 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "warn_on_stray_daemons", 2024-03-28T22:47:52.423 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.423 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.423 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.423 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.423 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.423 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.424 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.424 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-28T22:47:52.424 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.424 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.424 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.424 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.424 INFO:teuthology.orchestra.run.smithi043.stdout: "warn_on_stray_hosts": { 2024-03-28T22:47:52.424 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "warn_on_stray_hosts", 2024-03-28T22:47:52.425 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.425 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.425 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.425 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.425 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.425 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.425 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.425 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-28T22:47:52.425 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.426 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.426 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.426 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.426 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.426 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.426 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.426 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "crash", 2024-03-28T22:47:52.426 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.427 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.427 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.427 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.427 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.427 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.427 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.427 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.427 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.427 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.428 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.428 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.428 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.428 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.428 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.428 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.428 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.428 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.428 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.429 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.429 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.429 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.429 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.429 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.429 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.429 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.429 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.430 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.430 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.430 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.430 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.430 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.430 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.430 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.430 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.430 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.431 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.431 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.431 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.431 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.431 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.431 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.431 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.431 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.431 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.432 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.432 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.432 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.432 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.432 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.432 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.432 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.432 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.432 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.433 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.433 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.433 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.433 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.433 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.433 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.433 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.433 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.434 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.434 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.434 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.434 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.434 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.434 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.434 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.434 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.435 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.435 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.435 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.435 INFO:teuthology.orchestra.run.smithi043.stdout: "retain_interval": { 2024-03-28T22:47:52.435 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "retain_interval", 2024-03-28T22:47:52.435 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:52.435 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.435 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.435 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "31536000", 2024-03-28T22:47:52.436 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.436 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.436 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.436 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-28T22:47:52.436 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.436 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.436 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.436 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.437 INFO:teuthology.orchestra.run.smithi043.stdout: "warn_recent_interval": { 2024-03-28T22:47:52.437 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "warn_recent_interval", 2024-03-28T22:47:52.437 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:52.437 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.437 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.437 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "1209600", 2024-03-28T22:47:52.437 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.437 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.437 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.438 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-28T22:47:52.438 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.438 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.438 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.438 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.438 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.438 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.438 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.439 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "dashboard", 2024-03-28T22:47:52.439 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.439 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.439 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.439 INFO:teuthology.orchestra.run.smithi043.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-28T22:47:52.439 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-28T22:47:52.439 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.439 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.439 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.441 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-03-28T22:47:52.441 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.441 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.441 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.441 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.442 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.442 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.442 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.442 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.442 INFO:teuthology.orchestra.run.smithi043.stdout: "ALERTMANAGER_API_HOST": { 2024-03-28T22:47:52.442 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-28T22:47:52.442 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.442 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.442 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.443 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.443 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.443 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.443 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.443 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.443 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.443 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.443 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.444 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.444 INFO:teuthology.orchestra.run.smithi043.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-28T22:47:52.444 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-28T22:47:52.444 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.444 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.444 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.444 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.444 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.444 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.445 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.445 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.445 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.445 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.445 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.445 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.445 INFO:teuthology.orchestra.run.smithi043.stdout: "AUDIT_API_ENABLED": { 2024-03-28T22:47:52.445 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "AUDIT_API_ENABLED", 2024-03-28T22:47:52.446 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.446 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.446 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.446 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.446 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.446 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.446 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.446 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.446 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.447 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.447 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.447 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.447 INFO:teuthology.orchestra.run.smithi043.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-28T22:47:52.447 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-28T22:47:52.447 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.447 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.447 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.447 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.448 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.448 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.448 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.448 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.448 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.448 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.448 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.448 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.449 INFO:teuthology.orchestra.run.smithi043.stdout: "ENABLE_BROWSABLE_API": { 2024-03-28T22:47:52.449 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-28T22:47:52.449 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.449 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.449 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.449 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.449 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.449 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.449 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.450 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.450 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.450 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.450 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.450 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.450 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-28T22:47:52.450 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-28T22:47:52.450 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.451 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.451 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.451 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.451 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.451 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.451 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.451 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.451 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.451 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.452 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.452 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.452 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-28T22:47:52.452 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-28T22:47:52.452 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.452 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.452 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.452 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.452 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.453 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.453 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.453 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.453 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.453 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.453 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.453 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-28T22:47:52.453 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-28T22:47:52.454 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.454 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.454 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.454 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.454 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.454 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.454 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.454 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.455 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.455 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.455 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.455 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.455 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-28T22:47:52.455 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-28T22:47:52.455 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.455 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.455 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.456 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.456 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.456 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.456 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.456 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.456 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.456 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.456 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.456 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.457 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-28T22:47:52.457 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-28T22:47:52.457 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.457 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.457 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.457 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.457 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.457 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-28T22:47:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-28T22:47:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.459 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.459 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.459 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.459 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.459 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.459 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.459 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.459 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.460 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.460 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.460 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.460 INFO:teuthology.orchestra.run.smithi043.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-28T22:47:52.460 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-28T22:47:52.460 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.460 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.460 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.460 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.462 INFO:teuthology.orchestra.run.smithi043.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-28T22:47:52.462 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-28T22:47:52.462 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.462 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.462 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.462 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.462 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.462 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.463 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.463 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.463 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.463 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.463 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.463 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.463 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_API_PASSWORD": { 2024-03-28T22:47:52.463 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-28T22:47:52.464 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.464 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.464 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.464 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "admin", 2024-03-28T22:47:52.464 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.464 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.464 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.464 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.464 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.465 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.465 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.465 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.465 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-28T22:47:52.465 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-28T22:47:52.465 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.465 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.465 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.465 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.466 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.466 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.466 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.466 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.466 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.466 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.466 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.466 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.467 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_API_URL": { 2024-03-28T22:47:52.467 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_API_URL", 2024-03-28T22:47:52.467 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.467 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.467 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.467 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.467 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.467 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.467 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.468 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.468 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.468 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.468 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.468 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.468 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_API_USERNAME": { 2024-03-28T22:47:52.468 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-28T22:47:52.468 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.469 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.469 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.469 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "admin", 2024-03-28T22:47:52.469 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.469 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.469 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.469 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.469 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.469 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.470 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.470 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.470 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-28T22:47:52.470 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-28T22:47:52.470 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.470 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.470 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.470 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.470 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.471 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.471 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.471 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.471 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.471 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.471 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.471 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.471 INFO:teuthology.orchestra.run.smithi043.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-28T22:47:52.472 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-28T22:47:52.472 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.472 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.472 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.472 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.472 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.472 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.472 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.472 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.473 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.473 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.473 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.473 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.473 INFO:teuthology.orchestra.run.smithi043.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-28T22:47:52.473 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-28T22:47:52.473 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.473 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.473 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.473 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.474 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.474 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.474 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.474 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.474 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.474 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.474 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.474 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.475 INFO:teuthology.orchestra.run.smithi043.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-28T22:47:52.475 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-28T22:47:52.475 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.475 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.475 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.475 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.475 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.475 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.476 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.476 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.476 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.476 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.476 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.476 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.476 INFO:teuthology.orchestra.run.smithi043.stdout: "PROMETHEUS_API_HOST": { 2024-03-28T22:47:52.476 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-28T22:47:52.476 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.477 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.477 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.477 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.477 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.477 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.477 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.477 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.477 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.477 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.478 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.478 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.478 INFO:teuthology.orchestra.run.smithi043.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-28T22:47:52.478 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-28T22:47:52.478 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.478 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.478 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.478 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.478 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.479 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.479 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.479 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.479 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.479 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.479 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.479 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.479 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-28T22:47:52.480 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-28T22:47:52.480 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.480 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.480 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.480 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.480 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.480 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.480 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.481 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.481 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.481 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.481 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.481 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.481 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-28T22:47:52.481 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-28T22:47:52.481 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.481 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.481 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.482 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.482 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.482 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.482 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.482 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.482 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.482 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.482 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.483 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.483 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-28T22:47:52.483 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-28T22:47:52.483 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.483 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.483 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.483 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.483 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.483 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.484 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.484 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.484 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.484 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.484 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.484 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.484 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-28T22:47:52.484 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-28T22:47:52.484 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.484 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.485 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.485 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.485 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.485 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.485 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.485 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.485 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.485 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.485 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.486 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.486 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-28T22:47:52.486 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-28T22:47:52.486 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.486 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.486 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.486 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.486 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.486 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.487 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.487 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.487 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.487 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.487 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.487 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.487 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-28T22:47:52.487 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-28T22:47:52.488 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.488 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.488 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.488 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.488 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.488 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.488 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.488 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.488 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.489 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.489 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.489 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.489 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-28T22:47:52.489 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-28T22:47:52.489 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.489 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.489 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.489 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.490 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.490 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.490 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.490 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.490 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.490 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.490 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.490 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.490 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_ENABLED": { 2024-03-28T22:47:52.491 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-28T22:47:52.491 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.491 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.491 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.491 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.491 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.491 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.491 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.492 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.492 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.492 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.492 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.492 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.492 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-28T22:47:52.492 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-28T22:47:52.492 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.492 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.493 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.493 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-28T22:47:52.493 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.493 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.493 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.493 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.493 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.493 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.493 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.494 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.494 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-28T22:47:52.494 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-28T22:47:52.494 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.494 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.494 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.494 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-03-28T22:47:52.494 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.495 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.495 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.495 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.495 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.495 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.495 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.495 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.495 INFO:teuthology.orchestra.run.smithi043.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-28T22:47:52.495 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-28T22:47:52.496 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.496 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.496 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.496 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "8", 2024-03-28T22:47:52.496 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.496 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.496 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.496 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.496 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.497 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.497 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.497 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.497 INFO:teuthology.orchestra.run.smithi043.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-28T22:47:52.497 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-28T22:47:52.497 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.497 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.497 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.497 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "45", 2024-03-28T22:47:52.498 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.498 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.498 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.498 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.498 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.498 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.498 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.498 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.499 INFO:teuthology.orchestra.run.smithi043.stdout: "RGW_API_ACCESS_KEY": { 2024-03-28T22:47:52.499 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-28T22:47:52.499 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.499 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.499 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.499 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.499 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.499 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.499 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.500 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.500 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.500 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.500 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.500 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.500 INFO:teuthology.orchestra.run.smithi043.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-28T22:47:52.500 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-28T22:47:52.500 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.500 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.501 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.501 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "admin", 2024-03-28T22:47:52.501 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.501 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.501 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.501 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.501 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.501 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.502 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.502 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.502 INFO:teuthology.orchestra.run.smithi043.stdout: "RGW_API_SECRET_KEY": { 2024-03-28T22:47:52.502 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-28T22:47:52.502 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.502 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.502 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.502 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.503 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.503 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.503 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.503 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.503 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.503 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.503 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.503 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.503 INFO:teuthology.orchestra.run.smithi043.stdout: "RGW_API_SSL_VERIFY": { 2024-03-28T22:47:52.504 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-28T22:47:52.504 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.504 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.504 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.504 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.504 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.504 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.504 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.504 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.505 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.505 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.505 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.505 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.505 INFO:teuthology.orchestra.run.smithi043.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-28T22:47:52.505 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-28T22:47:52.505 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.505 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.506 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.506 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-03-28T22:47:52.506 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.506 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.506 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.506 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.506 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.506 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.507 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.507 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.507 INFO:teuthology.orchestra.run.smithi043.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-28T22:47:52.507 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-28T22:47:52.507 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.507 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.507 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.507 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-03-28T22:47:52.507 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.508 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.508 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.508 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.508 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.508 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.508 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.508 INFO:teuthology.orchestra.run.smithi043.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-28T22:47:52.509 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-28T22:47:52.509 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.509 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.509 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.509 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "5", 2024-03-28T22:47:52.509 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.509 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.509 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.509 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.510 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.510 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.510 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.510 INFO:teuthology.orchestra.run.smithi043.stdout: "cross_origin_url": { 2024-03-28T22:47:52.510 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cross_origin_url", 2024-03-28T22:47:52.510 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.510 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.511 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.511 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.511 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.511 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.511 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.511 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.511 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.511 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.512 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.512 INFO:teuthology.orchestra.run.smithi043.stdout: "crt_file": { 2024-03-28T22:47:52.512 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "crt_file", 2024-03-28T22:47:52.512 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.512 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.512 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.512 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.512 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.513 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.513 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.513 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.513 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.513 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.513 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.513 INFO:teuthology.orchestra.run.smithi043.stdout: "debug": { 2024-03-28T22:47:52.513 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "debug", 2024-03-28T22:47:52.513 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.514 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.514 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.514 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.514 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.514 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.514 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.514 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Enable/disable debug options", 2024-03-28T22:47:52.514 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.515 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.515 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.515 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.515 INFO:teuthology.orchestra.run.smithi043.stdout: "jwt_token_ttl": { 2024-03-28T22:47:52.515 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "jwt_token_ttl", 2024-03-28T22:47:52.515 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.515 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.515 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.515 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "28800", 2024-03-28T22:47:52.516 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.516 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.516 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.516 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.516 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.516 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.516 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.516 INFO:teuthology.orchestra.run.smithi043.stdout: "key_file": { 2024-03-28T22:47:52.517 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "key_file", 2024-03-28T22:47:52.517 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.517 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.517 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.517 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.517 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.517 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.517 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.517 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.518 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.518 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.518 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.518 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.518 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.518 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.519 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.519 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.519 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.519 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.519 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.519 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.519 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.519 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.519 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.520 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.520 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.520 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.520 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.520 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.520 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.520 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.520 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.521 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.521 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.521 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.521 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.521 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.521 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.521 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.521 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.522 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.522 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.522 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.522 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.522 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.522 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.522 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.522 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.523 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.523 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.523 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.523 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.523 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.523 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.523 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.523 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.524 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.524 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.524 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.524 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.524 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.524 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.524 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.524 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.524 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.525 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.525 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.525 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.525 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.525 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.525 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.525 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.525 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.525 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.526 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.526 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.526 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.526 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.526 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.526 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.526 INFO:teuthology.orchestra.run.smithi043.stdout: "motd": { 2024-03-28T22:47:52.526 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "motd", 2024-03-28T22:47:52.527 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.527 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.527 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.527 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.527 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.527 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.527 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.527 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "The message of the day", 2024-03-28T22:47:52.528 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.528 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.528 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.528 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.528 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_resolve_ip_addr": { 2024-03-28T22:47:52.528 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "redirect_resolve_ip_addr", 2024-03-28T22:47:52.528 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.528 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.528 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.529 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.529 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.529 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.529 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.529 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.529 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.529 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.529 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.529 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.530 INFO:teuthology.orchestra.run.smithi043.stdout: "server_addr": { 2024-03-28T22:47:52.530 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_addr", 2024-03-28T22:47:52.530 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.530 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.530 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.530 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "::", 2024-03-28T22:47:52.530 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.530 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.531 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.531 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.531 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.531 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.531 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.531 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.531 INFO:teuthology.orchestra.run.smithi043.stdout: "server_port": { 2024-03-28T22:47:52.531 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_port", 2024-03-28T22:47:52.531 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.532 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.532 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.532 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "8080", 2024-03-28T22:47:52.532 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.532 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.532 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.532 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.532 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.533 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.533 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.533 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.533 INFO:teuthology.orchestra.run.smithi043.stdout: "ssl": { 2024-03-28T22:47:52.533 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ssl", 2024-03-28T22:47:52.533 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.533 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.533 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.533 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.534 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.534 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.534 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.534 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.534 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.534 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.534 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.534 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: "ssl_server_port": { 2024-03-28T22:47:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ssl_server_port", 2024-03-28T22:47:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "8443", 2024-03-28T22:47:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.536 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.536 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.536 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.536 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.536 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.536 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.536 INFO:teuthology.orchestra.run.smithi043.stdout: "standby_behaviour": { 2024-03-28T22:47:52.536 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "standby_behaviour", 2024-03-28T22:47:52.536 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.537 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.537 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.537 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "redirect", 2024-03-28T22:47:52.537 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.537 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.537 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.537 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.537 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect" 2024-03-28T22:47:52.537 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.538 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.538 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.538 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.538 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.538 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.538 INFO:teuthology.orchestra.run.smithi043.stdout: "standby_error_status_code": { 2024-03-28T22:47:52.538 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "standby_error_status_code", 2024-03-28T22:47:52.538 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.539 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.539 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.539 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "500", 2024-03-28T22:47:52.539 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "400", 2024-03-28T22:47:52.539 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "599", 2024-03-28T22:47:52.539 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.539 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.539 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.539 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.540 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.540 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.540 INFO:teuthology.orchestra.run.smithi043.stdout: "url_prefix": { 2024-03-28T22:47:52.540 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "url_prefix", 2024-03-28T22:47:52.540 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.540 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.540 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.540 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.541 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.541 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.541 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.541 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.541 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.541 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.541 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.541 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.542 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.542 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.542 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.542 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "devicehealth", 2024-03-28T22:47:52.542 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.542 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.542 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.542 INFO:teuthology.orchestra.run.smithi043.stdout: "enable_monitoring": { 2024-03-28T22:47:52.542 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "enable_monitoring", 2024-03-28T22:47:52.543 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.543 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.543 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.543 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.543 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.543 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.543 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.543 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "monitor device health metrics", 2024-03-28T22:47:52.544 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.544 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.544 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.544 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.544 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.544 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.544 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.544 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.544 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.545 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.545 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.545 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.545 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.545 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.545 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.545 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.545 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.545 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.546 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.546 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.546 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.546 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.546 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.546 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.546 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.546 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.547 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.547 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.547 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.547 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.547 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.547 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.547 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.547 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.547 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.548 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.548 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.548 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.548 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.548 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.548 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.548 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.548 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.548 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.549 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.549 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.549 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.549 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.549 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.549 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.549 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.549 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.550 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.550 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.550 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.550 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.550 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.550 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.550 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.550 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.550 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.551 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.551 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.551 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.551 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.551 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.551 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.551 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.551 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.552 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.552 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.552 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.552 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.552 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.552 INFO:teuthology.orchestra.run.smithi043.stdout: "mark_out_threshold": { 2024-03-28T22:47:52.552 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mark_out_threshold", 2024-03-28T22:47:52.552 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:52.552 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.553 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.553 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "2419200", 2024-03-28T22:47:52.553 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.553 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.553 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.553 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-28T22:47:52.553 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.553 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.553 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.554 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.554 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": { 2024-03-28T22:47:52.554 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "pool_name", 2024-03-28T22:47:52.554 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.554 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.554 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.554 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "device_health_metrics", 2024-03-28T22:47:52.554 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.555 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.555 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.555 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-28T22:47:52.555 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.555 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.555 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.555 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.555 INFO:teuthology.orchestra.run.smithi043.stdout: "retention_period": { 2024-03-28T22:47:52.556 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "retention_period", 2024-03-28T22:47:52.556 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:52.556 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.556 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.556 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "15552000", 2024-03-28T22:47:52.556 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.556 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.556 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.556 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how long to retain device health metrics", 2024-03-28T22:47:52.557 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.557 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.557 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.557 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.557 INFO:teuthology.orchestra.run.smithi043.stdout: "scrape_frequency": { 2024-03-28T22:47:52.557 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "scrape_frequency", 2024-03-28T22:47:52.557 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:52.557 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.557 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.558 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "86400", 2024-03-28T22:47:52.558 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.558 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.558 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.558 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-28T22:47:52.558 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.558 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.558 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.559 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.559 INFO:teuthology.orchestra.run.smithi043.stdout: "self_heal": { 2024-03-28T22:47:52.559 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "self_heal", 2024-03-28T22:47:52.559 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.559 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.559 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.559 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.559 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.560 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.560 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.560 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-28T22:47:52.560 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.560 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.560 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.560 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.560 INFO:teuthology.orchestra.run.smithi043.stdout: "sleep_interval": { 2024-03-28T22:47:52.560 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sleep_interval", 2024-03-28T22:47:52.561 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:52.561 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.561 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.561 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "600", 2024-03-28T22:47:52.561 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.561 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.561 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.561 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how frequently to wake up and check device health", 2024-03-28T22:47:52.562 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.562 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.562 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.562 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.562 INFO:teuthology.orchestra.run.smithi043.stdout: "warn_threshold": { 2024-03-28T22:47:52.562 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "warn_threshold", 2024-03-28T22:47:52.562 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:52.562 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.562 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.563 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "7257600", 2024-03-28T22:47:52.563 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.563 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.563 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.563 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-28T22:47:52.563 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.563 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.563 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.564 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.564 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.564 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.564 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.564 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "diskprediction_local", 2024-03-28T22:47:52.564 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.564 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.564 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.565 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.565 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.565 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.565 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.565 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.565 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.565 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.565 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.565 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.566 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.566 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.566 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.566 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.566 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.566 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.566 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.566 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.567 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.567 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.567 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.567 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.567 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.567 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.567 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.567 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.567 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.568 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.568 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.568 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.568 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.568 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.568 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.568 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.568 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.568 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.569 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.569 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.569 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.569 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.569 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.569 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.569 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.569 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.569 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.570 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.570 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.570 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.570 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.570 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.570 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.570 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.570 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.571 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.571 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.571 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.571 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.571 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.571 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.571 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.571 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.571 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.572 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.572 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.572 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.572 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.572 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.572 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.572 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.572 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.572 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.573 INFO:teuthology.orchestra.run.smithi043.stdout: "predict_interval": { 2024-03-28T22:47:52.573 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "predict_interval", 2024-03-28T22:47:52.573 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.573 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.573 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.573 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "86400", 2024-03-28T22:47:52.573 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.573 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.573 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.574 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.574 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.574 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.574 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.574 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.574 INFO:teuthology.orchestra.run.smithi043.stdout: "predictor_model": { 2024-03-28T22:47:52.574 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "predictor_model", 2024-03-28T22:47:52.574 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.574 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.575 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.575 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "prophetstor", 2024-03-28T22:47:52.575 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.575 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.575 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.575 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.575 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.575 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.576 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.576 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.576 INFO:teuthology.orchestra.run.smithi043.stdout: "sleep_interval": { 2024-03-28T22:47:52.576 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sleep_interval", 2024-03-28T22:47:52.576 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.576 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.576 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.576 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "600", 2024-03-28T22:47:52.576 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.577 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.577 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.577 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.577 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.577 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.577 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.577 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.577 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.577 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.578 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.578 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "influx", 2024-03-28T22:47:52.578 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": false, 2024-03-28T22:47:52.578 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "influxdb python module not found", 2024-03-28T22:47:52.578 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.578 INFO:teuthology.orchestra.run.smithi043.stdout: "batch_size": { 2024-03-28T22:47:52.578 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "batch_size", 2024-03-28T22:47:52.578 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.579 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.579 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.579 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "5000", 2024-03-28T22:47:52.579 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.579 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.579 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.579 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-28T22:47:52.579 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.579 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.580 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.580 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.580 INFO:teuthology.orchestra.run.smithi043.stdout: "database": { 2024-03-28T22:47:52.580 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "database", 2024-03-28T22:47:52.580 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.580 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.580 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.580 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "ceph", 2024-03-28T22:47:52.580 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.581 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.581 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.581 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-28T22:47:52.581 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.581 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.581 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.581 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.581 INFO:teuthology.orchestra.run.smithi043.stdout: "hostname": { 2024-03-28T22:47:52.581 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "hostname", 2024-03-28T22:47:52.582 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.582 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.582 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.582 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.582 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.582 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.582 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.582 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "InfluxDB server hostname", 2024-03-28T22:47:52.583 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.583 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.583 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.583 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.583 INFO:teuthology.orchestra.run.smithi043.stdout: "interval": { 2024-03-28T22:47:52.583 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "interval", 2024-03-28T22:47:52.583 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:52.583 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.583 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.584 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "30", 2024-03-28T22:47:52.584 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "5", 2024-03-28T22:47:52.584 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.584 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.584 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-28T22:47:52.584 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.584 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.584 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.585 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.585 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.585 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.585 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.585 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.585 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.585 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.586 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.586 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.586 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.586 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.586 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.586 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.586 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.586 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.587 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.587 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.587 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.587 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.587 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.587 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.587 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.587 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.587 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.588 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.588 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.588 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.588 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.588 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.588 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.588 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.588 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.589 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.589 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.589 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.589 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.589 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.589 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.589 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.589 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.589 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.590 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.590 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.590 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.590 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.590 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.590 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.590 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.590 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.591 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.591 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.591 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.591 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.591 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.591 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.591 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.591 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.592 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.592 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.592 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.592 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.592 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.592 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.592 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.592 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.592 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.593 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.593 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.593 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.593 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.593 INFO:teuthology.orchestra.run.smithi043.stdout: "password": { 2024-03-28T22:47:52.593 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "password", 2024-03-28T22:47:52.593 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.593 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.594 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.594 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.594 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.594 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.594 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.594 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "password of InfluxDB server user", 2024-03-28T22:47:52.594 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.594 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.595 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.595 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.595 INFO:teuthology.orchestra.run.smithi043.stdout: "port": { 2024-03-28T22:47:52.595 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "port", 2024-03-28T22:47:52.595 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.595 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.595 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.595 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "8086", 2024-03-28T22:47:52.595 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.596 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.596 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.596 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "InfluxDB server port", 2024-03-28T22:47:52.596 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.596 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.596 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.596 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.596 INFO:teuthology.orchestra.run.smithi043.stdout: "ssl": { 2024-03-28T22:47:52.597 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ssl", 2024-03-28T22:47:52.597 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.597 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.597 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.597 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "false", 2024-03-28T22:47:52.597 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.597 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.597 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.597 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T22:47:52.598 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.598 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.598 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.598 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.598 INFO:teuthology.orchestra.run.smithi043.stdout: "threads": { 2024-03-28T22:47:52.598 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "threads", 2024-03-28T22:47:52.598 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.598 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.599 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.599 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "5", 2024-03-28T22:47:52.599 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "1", 2024-03-28T22:47:52.599 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "32", 2024-03-28T22:47:52.599 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.599 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-28T22:47:52.599 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.599 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.599 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.600 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.600 INFO:teuthology.orchestra.run.smithi043.stdout: "username": { 2024-03-28T22:47:52.600 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "username", 2024-03-28T22:47:52.600 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.600 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.600 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.600 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.600 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.601 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.601 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.601 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "username of InfluxDB server user", 2024-03-28T22:47:52.601 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.601 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.601 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.601 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.601 INFO:teuthology.orchestra.run.smithi043.stdout: "verify_ssl": { 2024-03-28T22:47:52.601 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "verify_ssl", 2024-03-28T22:47:52.602 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.602 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.602 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.602 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "true", 2024-03-28T22:47:52.602 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.602 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.602 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.602 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T22:47:52.602 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.603 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.603 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.603 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.603 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.603 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.603 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.603 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "insights", 2024-03-28T22:47:52.603 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.603 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.604 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.604 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.604 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.604 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.604 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.604 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.604 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.604 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.604 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.604 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.605 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.605 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.605 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.605 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.605 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.605 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.605 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.605 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.605 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.606 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.606 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.606 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.606 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.606 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.606 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.606 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.606 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.607 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.607 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.607 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.607 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.607 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.607 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.607 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.607 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.607 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.608 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.608 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.608 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.608 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.608 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.608 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.608 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.608 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.608 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.609 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.609 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.609 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.609 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.609 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.609 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.609 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.609 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.610 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.610 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.610 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.610 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.610 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.610 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.610 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.610 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.611 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.611 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.611 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.611 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.611 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.611 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.611 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.611 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.611 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.612 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.612 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.612 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.612 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.612 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "iostat", 2024-03-28T22:47:52.612 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.612 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.612 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.612 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.613 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.613 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.613 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.613 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.613 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.613 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.613 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.613 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.614 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.614 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.614 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.614 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.614 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.614 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.614 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.614 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.614 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.615 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.615 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.615 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.615 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.615 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.615 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.615 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.615 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.615 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.616 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.616 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.616 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.616 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.616 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.616 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.616 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.616 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.616 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.617 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.617 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.617 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.617 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.617 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.617 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.617 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.617 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.617 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.618 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.618 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.618 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.618 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.618 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.618 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.618 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.618 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.619 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.619 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.619 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.619 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.619 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.619 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.619 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.619 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.619 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.620 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.620 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.620 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.620 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.620 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.620 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.620 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.620 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.621 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.621 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.621 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.621 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "k8sevents", 2024-03-28T22:47:52.621 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.621 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.621 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.621 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_event_retention_days": { 2024-03-28T22:47:52.621 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "ceph_event_retention_days", 2024-03-28T22:47:52.621 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.622 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.622 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.622 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "7", 2024-03-28T22:47:52.622 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.622 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.622 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.622 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-28T22:47:52.622 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.623 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.623 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.623 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.623 INFO:teuthology.orchestra.run.smithi043.stdout: "config_check_secs": { 2024-03-28T22:47:52.623 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "config_check_secs", 2024-03-28T22:47:52.623 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.623 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.623 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.623 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-03-28T22:47:52.624 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "10", 2024-03-28T22:47:52.624 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.624 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.624 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-28T22:47:52.624 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.624 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.624 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.624 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.625 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.625 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.625 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.625 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.625 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.625 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.625 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.625 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.625 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.626 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.626 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.626 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.626 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.626 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.626 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.626 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.626 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.627 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.627 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.627 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.627 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.627 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.627 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.627 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.627 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.627 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.628 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.628 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.628 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.628 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.628 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.628 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.628 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.628 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.628 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.629 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.629 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.629 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.629 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.629 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.629 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.629 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.629 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.630 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.630 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.630 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.630 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.630 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.630 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.630 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.630 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.631 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.631 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.631 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.631 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.631 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.631 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.631 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.631 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.631 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.631 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.632 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.632 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.632 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.632 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.632 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.632 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.632 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.632 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.632 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.633 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.633 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.633 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.633 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "localpool", 2024-03-28T22:47:52.633 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.633 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.633 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.633 INFO:teuthology.orchestra.run.smithi043.stdout: "failure_domain": { 2024-03-28T22:47:52.633 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "failure_domain", 2024-03-28T22:47:52.634 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.634 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.634 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.634 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "host", 2024-03-28T22:47:52.634 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.634 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.634 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.634 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "failure domain for any created local pool", 2024-03-28T22:47:52.634 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-28T22:47:52.635 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.635 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.635 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.635 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.635 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.635 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.635 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.635 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.636 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.636 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.636 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.636 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.636 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.636 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.636 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.636 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.636 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.636 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.637 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.637 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.637 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.637 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.637 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.637 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.637 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.637 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.637 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.638 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.638 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.638 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.638 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.638 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.638 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.638 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.638 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.639 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.639 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.639 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.639 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.639 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.639 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.639 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.639 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.639 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.640 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.640 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.640 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.640 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.640 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.640 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.640 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.640 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.641 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.641 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.641 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.641 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.641 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.641 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.641 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.641 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.641 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.642 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.642 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.642 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.642 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.642 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.642 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.642 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.642 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.643 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.643 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.643 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.643 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.643 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": { 2024-03-28T22:47:52.643 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "min_size", 2024-03-28T22:47:52.643 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.643 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.644 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.646 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.646 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.646 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.646 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.646 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "default min_size for any created local pool", 2024-03-28T22:47:52.646 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-28T22:47:52.646 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.646 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.646 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.647 INFO:teuthology.orchestra.run.smithi043.stdout: "num_rep": { 2024-03-28T22:47:52.647 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "num_rep", 2024-03-28T22:47:52.647 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.647 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.647 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.647 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "3", 2024-03-28T22:47:52.647 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.647 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.648 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.648 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "default replica count for any created local pool", 2024-03-28T22:47:52.648 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.648 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.648 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.648 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.648 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": { 2024-03-28T22:47:52.648 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "pg_num", 2024-03-28T22:47:52.648 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.649 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.649 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.649 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "128", 2024-03-28T22:47:52.649 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.649 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.649 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.649 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "default pg_num for any created local pool", 2024-03-28T22:47:52.649 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.649 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.650 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.650 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.650 INFO:teuthology.orchestra.run.smithi043.stdout: "prefix": { 2024-03-28T22:47:52.650 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "prefix", 2024-03-28T22:47:52.650 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.650 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.650 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.650 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.651 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.651 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.651 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.651 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "name prefix for any created local pool", 2024-03-28T22:47:52.651 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.651 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.651 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.651 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.651 INFO:teuthology.orchestra.run.smithi043.stdout: "subtree": { 2024-03-28T22:47:52.652 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "subtree", 2024-03-28T22:47:52.652 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.652 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.652 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.652 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "rack", 2024-03-28T22:47:52.652 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.652 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.652 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.652 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-28T22:47:52.653 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-28T22:47:52.653 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.653 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.653 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.653 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.653 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.653 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.653 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mds_autoscaler", 2024-03-28T22:47:52.654 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.654 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.654 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.654 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.654 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.654 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.654 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.654 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.655 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.655 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.655 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.655 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.655 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.655 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.655 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.655 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.655 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.656 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.656 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.656 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.656 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.656 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.656 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.656 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.656 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.656 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.657 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.657 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.657 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.657 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.657 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.657 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.657 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.657 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.658 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.658 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.658 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.658 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.658 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.658 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.658 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.658 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.658 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.659 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.659 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.659 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.659 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.659 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.659 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.659 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.659 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.660 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.660 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.660 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.660 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.660 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.660 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.660 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.660 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.660 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.661 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.661 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.661 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.661 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.661 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.661 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.661 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.661 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.661 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.662 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.662 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.662 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.662 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.662 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.662 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.662 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.662 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mirroring", 2024-03-28T22:47:52.662 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.663 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.663 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.663 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.663 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.663 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.663 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.663 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.663 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.664 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.664 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.664 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.664 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.664 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.664 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.664 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.664 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.664 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.665 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.665 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.665 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.665 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.665 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.665 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.665 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.665 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.665 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.666 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.666 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.666 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.666 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.666 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.666 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.666 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.666 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.667 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.667 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.667 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.667 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.667 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.667 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.667 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.667 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.667 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.668 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.668 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.668 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.668 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.668 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.668 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.668 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.668 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.669 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.669 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.669 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.669 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.669 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.669 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.669 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.669 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.669 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.670 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.670 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.670 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.670 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.670 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.670 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.670 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.671 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.671 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.671 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.671 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.671 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.671 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.671 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.671 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.671 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "nfs", 2024-03-28T22:47:52.672 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.672 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.672 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.672 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.672 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.672 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.672 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.672 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.672 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.673 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.673 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.673 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.673 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.673 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.673 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.673 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.673 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.674 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.674 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.674 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.674 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.674 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.674 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.674 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.674 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.675 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.675 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.675 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.675 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.675 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.675 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.675 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.675 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.675 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.676 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.676 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.676 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.676 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.676 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.676 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.676 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.676 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.676 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.677 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.677 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.677 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.677 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.677 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.677 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.677 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.677 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.678 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.678 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.678 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.678 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.678 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.678 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.678 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.678 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.679 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.679 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.679 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.679 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.679 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.679 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.679 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.679 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.680 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.680 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.680 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.680 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.680 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.680 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.680 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.680 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.680 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.681 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "orchestrator", 2024-03-28T22:47:52.681 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.681 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.681 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.681 INFO:teuthology.orchestra.run.smithi043.stdout: "fail_fs": { 2024-03-28T22:47:52.681 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "fail_fs", 2024-03-28T22:47:52.681 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.681 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.681 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.682 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.682 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.682 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.682 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.682 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-28T22:47:52.682 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.682 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.683 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.683 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.683 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.683 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.683 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.683 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.683 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.683 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.683 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.683 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.684 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.684 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.684 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.684 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.684 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.684 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.684 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.684 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.684 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.685 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.685 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.685 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.685 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.685 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.685 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.685 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.685 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.686 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.686 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.686 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.686 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.686 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.686 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.686 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.686 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.687 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.687 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.687 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.687 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.687 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.687 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.687 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.687 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.688 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.688 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.688 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.688 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.688 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.688 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.688 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.688 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.688 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.689 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.689 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.689 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.689 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.689 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.689 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.689 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.689 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.690 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.690 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.690 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.690 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.690 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.690 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.690 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.690 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.690 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.691 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.691 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.691 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.691 INFO:teuthology.orchestra.run.smithi043.stdout: "orchestrator": { 2024-03-28T22:47:52.691 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "orchestrator", 2024-03-28T22:47:52.691 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.691 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.691 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.692 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.692 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.692 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.692 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.692 INFO:teuthology.orchestra.run.smithi043.stdout: "cephadm", 2024-03-28T22:47:52.692 INFO:teuthology.orchestra.run.smithi043.stdout: "rook", 2024-03-28T22:47:52.692 INFO:teuthology.orchestra.run.smithi043.stdout: "test_orchestrator" 2024-03-28T22:47:52.692 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.692 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Orchestrator backend", 2024-03-28T22:47:52.693 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.693 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.693 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.693 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.693 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.693 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.693 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.693 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "osd_perf_query", 2024-03-28T22:47:52.693 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.694 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.694 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.694 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.694 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.694 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.694 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.694 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.694 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.695 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.695 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.695 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.695 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.695 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.695 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.695 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.695 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.696 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.696 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.696 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.696 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.696 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.696 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.696 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.696 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.696 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.697 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.697 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.697 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.697 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.697 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.697 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.697 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.697 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.697 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.698 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.698 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.698 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.698 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.698 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.698 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.698 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.698 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.698 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.699 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.699 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.699 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.699 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.699 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.699 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.699 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.699 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.700 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.700 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.700 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.700 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.700 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.700 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.700 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.700 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.700 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.701 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.701 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.701 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.701 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.701 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.701 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.701 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.701 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.702 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.702 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.702 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.702 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.702 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.702 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.702 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.702 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "osd_support", 2024-03-28T22:47:52.703 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.703 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.703 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.703 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.703 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.703 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.703 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.703 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.703 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.704 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.704 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.704 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.704 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.704 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.704 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.704 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.704 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.704 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.705 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.705 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.705 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.705 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.705 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.705 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.705 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.705 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.705 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.705 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.706 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.706 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.706 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.706 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.706 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.706 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.706 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.706 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.707 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.707 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.707 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.707 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.707 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.707 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.707 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.707 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.707 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.708 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.708 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.708 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.708 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.708 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.708 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.708 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.708 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.708 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.709 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.709 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.709 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.709 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.709 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.709 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.709 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.709 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.709 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.710 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.710 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.710 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.710 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.710 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.710 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.710 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.710 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.711 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.711 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.711 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.711 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.711 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.711 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "pg_autoscaler", 2024-03-28T22:47:52.711 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.711 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.711 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.712 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.712 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.712 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.712 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.712 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.712 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.712 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.712 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.713 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.713 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.713 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.713 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.713 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.713 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.713 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.713 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.713 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.714 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.714 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.714 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.714 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.714 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.714 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.714 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.714 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.715 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.715 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.715 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.715 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.715 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.715 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.715 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.715 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.715 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.716 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.716 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.716 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.716 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.716 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.716 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.716 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.716 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.716 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.717 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.717 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.717 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.717 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.717 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.717 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.717 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.717 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.717 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.718 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.718 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.718 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.718 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.718 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.718 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.718 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.718 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.719 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.719 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.719 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.719 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.719 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.719 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.719 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.719 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.720 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.720 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.720 INFO:teuthology.orchestra.run.smithi043.stdout: "sleep_interval": { 2024-03-28T22:47:52.720 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sleep_interval", 2024-03-28T22:47:52.720 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:52.720 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.720 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.720 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "60", 2024-03-28T22:47:52.720 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.721 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.721 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.721 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.721 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.721 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.721 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.721 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.721 INFO:teuthology.orchestra.run.smithi043.stdout: "threshold": { 2024-03-28T22:47:52.721 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "threshold", 2024-03-28T22:47:52.722 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-03-28T22:47:52.722 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.722 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.722 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "3.0", 2024-03-28T22:47:52.722 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "1.0", 2024-03-28T22:47:52.722 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.722 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.722 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "scaling threshold", 2024-03-28T22:47:52.722 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-28T22:47:52.723 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.723 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.723 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.723 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.723 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.723 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.723 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "progress", 2024-03-28T22:47:52.723 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.724 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.724 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.724 INFO:teuthology.orchestra.run.smithi043.stdout: "allow_pg_recovery_event": { 2024-03-28T22:47:52.724 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "allow_pg_recovery_event", 2024-03-28T22:47:52.724 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.724 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.724 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.724 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.724 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.724 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.725 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.725 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-28T22:47:52.725 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.725 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.725 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.725 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.725 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": { 2024-03-28T22:47:52.725 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "enabled", 2024-03-28T22:47:52.726 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.726 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.726 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.726 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.726 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.726 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.726 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.726 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.726 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.727 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.727 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.727 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.727 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.727 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.727 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.727 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.727 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.727 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.728 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.728 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.728 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.728 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.728 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.728 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.728 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.728 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.728 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.729 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.729 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.729 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.729 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.729 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.729 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.729 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.729 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.729 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.730 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.730 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.730 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.730 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.730 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.730 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.730 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.730 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.731 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.731 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.731 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.731 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.731 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.731 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.731 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.731 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.731 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.731 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.732 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.732 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.732 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.732 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.732 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.732 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.732 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.732 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.732 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.733 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.733 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.733 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.733 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.733 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.733 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.733 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.733 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.734 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.734 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.734 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.734 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.734 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.734 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.734 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.734 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.734 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.735 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.735 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.735 INFO:teuthology.orchestra.run.smithi043.stdout: "max_completed_events": { 2024-03-28T22:47:52.735 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "max_completed_events", 2024-03-28T22:47:52.735 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.735 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.735 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.735 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "50", 2024-03-28T22:47:52.735 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.736 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.736 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.736 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "number of past completed events to remember", 2024-03-28T22:47:52.736 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.736 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.736 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.736 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.736 INFO:teuthology.orchestra.run.smithi043.stdout: "sleep_interval": { 2024-03-28T22:47:52.737 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "sleep_interval", 2024-03-28T22:47:52.737 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:52.737 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.737 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.737 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "5", 2024-03-28T22:47:52.737 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.737 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.737 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.737 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "how long the module is going to sleep", 2024-03-28T22:47:52.737 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.738 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.738 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.738 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.738 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.738 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.738 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.738 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "prometheus", 2024-03-28T22:47:52.738 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.739 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.739 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.739 INFO:teuthology.orchestra.run.smithi043.stdout: "cache": { 2024-03-28T22:47:52.739 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "cache", 2024-03-28T22:47:52.739 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.739 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.739 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.739 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.740 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.740 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.740 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.740 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.740 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.740 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.740 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.740 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.740 INFO:teuthology.orchestra.run.smithi043.stdout: "exclude_perf_counters": { 2024-03-28T22:47:52.741 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "exclude_perf_counters", 2024-03-28T22:47:52.741 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.741 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.741 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.741 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.741 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.741 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.741 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.741 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-28T22:47:52.742 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-28T22:47:52.742 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.742 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.742 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.742 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.742 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.742 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.742 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.743 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.743 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.743 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.743 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.743 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.743 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.743 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.743 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.743 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.744 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.744 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.744 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.744 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.744 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.744 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.744 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.744 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.745 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.745 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.745 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.745 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.745 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.745 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.745 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.745 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.746 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.746 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.746 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.746 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.746 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.746 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.746 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.746 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.746 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.747 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.747 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.747 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.747 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.747 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.747 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.747 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.747 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.747 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.748 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.748 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.748 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.748 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.748 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.748 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.748 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.748 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.748 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.749 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.749 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.749 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.749 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.749 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.749 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.749 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.749 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.750 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.750 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.750 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.750 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.750 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.750 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.750 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_stats_pools": { 2024-03-28T22:47:52.750 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rbd_stats_pools", 2024-03-28T22:47:52.750 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.751 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.751 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.751 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.751 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.751 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.751 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.751 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.751 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.751 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.752 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.752 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.752 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-28T22:47:52.752 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-28T22:47:52.752 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.752 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.752 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.752 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "300", 2024-03-28T22:47:52.753 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.753 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.753 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.753 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.753 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.753 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.753 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.753 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.753 INFO:teuthology.orchestra.run.smithi043.stdout: "scrape_interval": { 2024-03-28T22:47:52.754 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "scrape_interval", 2024-03-28T22:47:52.754 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-03-28T22:47:52.754 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.754 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.754 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "15.0", 2024-03-28T22:47:52.754 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.754 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.754 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.755 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.755 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.755 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.755 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.755 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.755 INFO:teuthology.orchestra.run.smithi043.stdout: "server_addr": { 2024-03-28T22:47:52.755 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_addr", 2024-03-28T22:47:52.755 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.755 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.756 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.756 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "::", 2024-03-28T22:47:52.756 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.756 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.756 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.756 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-28T22:47:52.756 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.756 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.757 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.757 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.757 INFO:teuthology.orchestra.run.smithi043.stdout: "server_port": { 2024-03-28T22:47:52.757 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_port", 2024-03-28T22:47:52.757 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.757 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.757 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.757 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "9283", 2024-03-28T22:47:52.757 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.757 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.758 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.758 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-28T22:47:52.758 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.758 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.758 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.758 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.758 INFO:teuthology.orchestra.run.smithi043.stdout: "stale_cache_strategy": { 2024-03-28T22:47:52.758 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "stale_cache_strategy", 2024-03-28T22:47:52.759 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.759 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.759 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.759 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "log", 2024-03-28T22:47:52.759 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.759 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.759 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.759 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.760 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.760 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.760 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.760 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.760 INFO:teuthology.orchestra.run.smithi043.stdout: "standby_behaviour": { 2024-03-28T22:47:52.760 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "standby_behaviour", 2024-03-28T22:47:52.760 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.760 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.760 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.761 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "default", 2024-03-28T22:47:52.761 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.761 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.761 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.761 INFO:teuthology.orchestra.run.smithi043.stdout: "default", 2024-03-28T22:47:52.761 INFO:teuthology.orchestra.run.smithi043.stdout: "error" 2024-03-28T22:47:52.761 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.761 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.761 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.762 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.762 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.762 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.762 INFO:teuthology.orchestra.run.smithi043.stdout: "standby_error_status_code": { 2024-03-28T22:47:52.762 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "standby_error_status_code", 2024-03-28T22:47:52.762 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.762 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.762 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.763 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "500", 2024-03-28T22:47:52.763 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "400", 2024-03-28T22:47:52.763 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "599", 2024-03-28T22:47:52.763 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.763 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.763 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.763 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.763 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.764 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.764 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.764 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.764 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.764 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rbd_support", 2024-03-28T22:47:52.764 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.764 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.764 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.764 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.765 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.765 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.765 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.765 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.765 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.765 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.765 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.765 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.766 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.766 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.766 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.766 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.766 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.766 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.766 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.766 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.767 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.767 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.767 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.767 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.767 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.767 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.767 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.767 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.767 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.768 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.768 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.768 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.768 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.768 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.768 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.768 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.768 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.768 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.769 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.769 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.769 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.769 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.769 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.769 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.769 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.769 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.769 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.770 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.770 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.770 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.770 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.770 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.770 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.770 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.771 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.771 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.771 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.771 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.771 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.771 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.771 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.771 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.771 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.772 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.772 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.772 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.772 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.772 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.772 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.772 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.772 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.772 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.773 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.773 INFO:teuthology.orchestra.run.smithi043.stdout: "max_concurrent_snap_create": { 2024-03-28T22:47:52.773 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "max_concurrent_snap_create", 2024-03-28T22:47:52.773 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.773 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.773 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.773 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10", 2024-03-28T22:47:52.773 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.774 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.774 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.774 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.774 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.774 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.774 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.774 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.774 INFO:teuthology.orchestra.run.smithi043.stdout: "mirror_snapshot_schedule": { 2024-03-28T22:47:52.775 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mirror_snapshot_schedule", 2024-03-28T22:47:52.775 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.775 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.775 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.775 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.775 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.775 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.775 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.775 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.776 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.776 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.776 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.776 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.776 INFO:teuthology.orchestra.run.smithi043.stdout: "trash_purge_schedule": { 2024-03-28T22:47:52.776 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "trash_purge_schedule", 2024-03-28T22:47:52.776 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.776 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.776 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.777 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.777 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.777 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.777 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.777 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.777 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.777 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.777 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.777 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.778 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.778 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.778 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.778 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "restful", 2024-03-28T22:47:52.778 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.778 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.778 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.778 INFO:teuthology.orchestra.run.smithi043.stdout: "enable_auth": { 2024-03-28T22:47:52.779 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "enable_auth", 2024-03-28T22:47:52.779 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.779 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.779 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.779 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.779 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.779 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.779 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.780 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.780 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.780 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.780 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.780 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.780 INFO:teuthology.orchestra.run.smithi043.stdout: "key_file": { 2024-03-28T22:47:52.780 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "key_file", 2024-03-28T22:47:52.780 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.780 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.781 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.781 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.781 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.781 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.781 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.781 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.781 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.781 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.782 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.782 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.782 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.782 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.782 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.782 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.782 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.782 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.782 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.783 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.783 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.783 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.783 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.783 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.783 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.783 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.783 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.783 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.784 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.784 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.784 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.784 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.784 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.784 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.784 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.784 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.784 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.784 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.785 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.785 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.785 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.785 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.785 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.785 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.785 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.785 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.786 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.786 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.786 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.786 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.786 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.786 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.786 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.786 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.786 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.787 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.787 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.787 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.787 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.787 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.787 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.787 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.787 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.787 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.788 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.788 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.788 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.788 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.788 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.788 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.788 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.788 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.789 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.789 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.789 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.789 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.789 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.789 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.789 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.789 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.789 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.790 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.790 INFO:teuthology.orchestra.run.smithi043.stdout: "server_addr": { 2024-03-28T22:47:52.790 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_addr", 2024-03-28T22:47:52.790 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.790 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.790 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.790 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.790 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.791 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.791 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.791 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.791 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.791 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.791 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.791 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.791 INFO:teuthology.orchestra.run.smithi043.stdout: "server_port": { 2024-03-28T22:47:52.791 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "server_port", 2024-03-28T22:47:52.791 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.792 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.792 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.792 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.792 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.792 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.792 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.792 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.792 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.792 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.793 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.793 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.793 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.793 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.793 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.793 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rgw", 2024-03-28T22:47:52.793 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.793 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.793 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.794 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.794 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.794 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.794 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.794 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.794 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.794 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.794 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.795 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.795 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.795 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.795 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.795 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.795 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.795 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.795 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.795 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.796 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.796 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.796 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.796 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.796 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.796 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.796 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.796 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.797 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.797 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.797 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.797 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.797 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.797 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.797 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.797 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.797 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.798 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.798 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.798 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.798 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.798 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.798 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.798 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.798 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.798 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.799 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.799 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.799 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.799 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.799 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.799 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.799 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.799 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.799 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.800 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.800 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.800 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.800 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.800 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.800 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.800 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.800 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.800 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.801 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.801 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.801 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.801 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.801 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.801 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.801 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.801 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.801 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.801 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.802 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.802 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.802 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rook", 2024-03-28T22:47:52.802 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.802 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.802 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.802 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.802 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.803 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.803 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.803 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.803 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.803 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.803 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.803 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.803 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.804 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.804 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.804 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.804 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.804 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.804 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.804 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.804 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.805 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.805 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.805 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.805 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.805 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.805 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.805 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.805 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.805 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.806 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.806 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.806 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.806 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.806 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.806 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.806 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.806 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.807 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.807 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.807 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.807 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.807 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.807 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.807 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.807 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.807 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.808 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.808 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.808 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.808 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.808 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.808 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.808 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.808 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.809 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.809 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.809 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.809 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.809 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.809 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.809 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.809 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.809 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.810 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.810 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.810 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.810 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.810 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.810 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.810 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.810 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.811 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.811 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": { 2024-03-28T22:47:52.811 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "storage_class", 2024-03-28T22:47:52.811 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.811 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.811 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.811 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "local", 2024-03-28T22:47:52.811 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.811 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.812 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.812 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-28T22:47:52.812 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.812 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.812 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.812 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.812 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.812 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.812 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.813 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "selftest", 2024-03-28T22:47:52.813 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.813 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.813 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.813 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.813 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.813 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.813 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.813 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.814 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.814 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.814 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.814 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.814 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.814 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.814 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.814 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.815 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.815 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.815 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.815 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.815 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.815 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.815 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.815 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.815 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.816 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.816 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.816 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.816 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.816 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.816 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.816 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.816 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.817 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.817 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.817 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.817 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.817 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.817 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.817 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.817 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.817 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.818 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.818 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.818 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.818 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.818 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.818 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.818 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.818 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.819 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.819 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.819 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.819 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.819 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.819 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.819 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.819 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.819 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.820 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.820 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.820 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.820 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.820 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.820 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.820 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.820 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.821 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.821 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.821 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.821 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.821 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.821 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.821 INFO:teuthology.orchestra.run.smithi043.stdout: "roption1": { 2024-03-28T22:47:52.821 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "roption1", 2024-03-28T22:47:52.821 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.821 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.822 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.822 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.822 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.822 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.822 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.822 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.822 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.822 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.823 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.823 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.823 INFO:teuthology.orchestra.run.smithi043.stdout: "roption2": { 2024-03-28T22:47:52.823 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "roption2", 2024-03-28T22:47:52.823 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.823 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.823 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.823 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "xyz", 2024-03-28T22:47:52.823 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.824 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.824 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.824 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.824 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.824 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.824 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.824 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.824 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption1": { 2024-03-28T22:47:52.824 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption1", 2024-03-28T22:47:52.825 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.825 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.825 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.825 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.825 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.825 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.825 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.825 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.825 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.826 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.826 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.826 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.826 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption2": { 2024-03-28T22:47:52.826 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption2", 2024-03-28T22:47:52.826 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.826 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.826 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.827 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.827 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.827 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.827 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.827 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.827 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.827 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.827 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.828 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.828 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption3": { 2024-03-28T22:47:52.828 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption3", 2024-03-28T22:47:52.828 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "float", 2024-03-28T22:47:52.828 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.828 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.828 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.828 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.828 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.829 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.829 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.829 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.829 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.829 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.829 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.829 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption4": { 2024-03-28T22:47:52.829 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption4", 2024-03-28T22:47:52.829 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.830 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.830 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.830 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.830 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.830 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.830 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.830 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.830 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.830 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.831 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.831 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.831 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption5": { 2024-03-28T22:47:52.831 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption5", 2024-03-28T22:47:52.831 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.831 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.831 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.831 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.831 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.832 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.832 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.832 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.832 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.832 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.832 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.832 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption6": { 2024-03-28T22:47:52.833 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption6", 2024-03-28T22:47:52.833 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.833 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.833 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.833 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.833 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.833 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.833 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.833 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.834 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.834 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.834 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.834 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.834 INFO:teuthology.orchestra.run.smithi043.stdout: "rwoption7": { 2024-03-28T22:47:52.834 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rwoption7", 2024-03-28T22:47:52.834 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.834 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.835 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.835 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.835 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "1", 2024-03-28T22:47:52.835 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "42", 2024-03-28T22:47:52.835 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.835 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.835 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.835 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.835 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.836 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.836 INFO:teuthology.orchestra.run.smithi043.stdout: "testkey": { 2024-03-28T22:47:52.836 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "testkey", 2024-03-28T22:47:52.836 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.836 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.836 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.836 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.836 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.836 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.837 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.837 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.837 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.837 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.837 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.837 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.837 INFO:teuthology.orchestra.run.smithi043.stdout: "testlkey": { 2024-03-28T22:47:52.837 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "testlkey", 2024-03-28T22:47:52.837 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.838 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.838 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.838 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.838 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.838 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.838 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.838 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.838 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.838 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.839 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.839 INFO:teuthology.orchestra.run.smithi043.stdout: "testnewline": { 2024-03-28T22:47:52.839 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "testnewline", 2024-03-28T22:47:52.839 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.839 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.839 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.839 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.840 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.840 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.840 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.840 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.840 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.840 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.840 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.840 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.840 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.840 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.841 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.841 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "snap_schedule", 2024-03-28T22:47:52.841 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.841 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.841 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.841 INFO:teuthology.orchestra.run.smithi043.stdout: "allow_m_granularity": { 2024-03-28T22:47:52.841 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "allow_m_granularity", 2024-03-28T22:47:52.841 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.842 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.842 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.842 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.842 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.842 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.842 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.842 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "allow minute scheduled snapshots", 2024-03-28T22:47:52.842 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.842 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.843 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.843 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.843 INFO:teuthology.orchestra.run.smithi043.stdout: "dump_on_update": { 2024-03-28T22:47:52.843 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "dump_on_update", 2024-03-28T22:47:52.843 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.843 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.843 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.843 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.843 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.844 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.844 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.844 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "dump database to debug log on update", 2024-03-28T22:47:52.844 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.844 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.844 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.844 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.844 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.844 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.845 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.845 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.845 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.845 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.845 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.845 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.845 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.845 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.845 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.846 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.846 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.846 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.846 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.846 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.846 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.846 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.846 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.847 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.847 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.847 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.847 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.847 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.847 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.847 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.847 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.847 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.848 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.848 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.848 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.848 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.848 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.848 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.848 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.849 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.849 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.849 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.850 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.850 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.850 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.850 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.850 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.850 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.850 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.850 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.850 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.851 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.851 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.851 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.851 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.851 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.851 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.851 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.851 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.851 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.852 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.852 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.852 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.852 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.852 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.852 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.852 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.852 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.852 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.853 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.853 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.853 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.853 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.853 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.853 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.853 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.853 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.853 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "stats", 2024-03-28T22:47:52.854 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.854 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.854 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.854 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.854 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.854 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.854 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.854 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.855 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.855 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.855 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.855 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.855 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.855 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.855 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.855 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.856 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.856 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.856 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.856 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.856 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.856 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.856 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.856 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.856 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.856 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.857 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.857 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.857 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.857 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.857 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.857 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.857 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.857 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.857 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.858 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.858 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.858 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.858 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.858 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.858 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.858 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.858 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.859 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.859 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.859 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.859 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.859 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.859 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.859 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.859 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.859 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.860 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.860 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.860 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.860 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.860 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.860 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.860 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.860 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.860 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.861 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.861 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.861 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.861 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.861 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.861 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.861 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.861 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.861 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.862 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.862 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.862 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.862 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.862 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.862 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.862 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "status", 2024-03-28T22:47:52.862 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.863 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.863 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.863 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.863 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.863 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.863 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.863 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.863 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.863 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.864 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.864 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.864 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.864 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.864 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.864 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.864 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.864 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.864 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.865 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.865 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.865 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.865 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.865 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.865 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.865 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.865 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.865 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.865 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.866 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.866 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.866 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.866 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.866 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.866 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.866 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.866 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.867 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.867 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.867 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.867 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.867 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.867 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.867 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.867 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.867 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.867 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.868 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.868 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.868 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.868 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.868 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.868 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.868 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.868 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.868 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.868 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.869 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.869 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.869 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.869 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.869 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.869 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.869 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.869 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.870 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.870 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.870 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.870 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.870 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.870 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.870 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.870 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.870 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.871 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.871 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.871 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "telegraf", 2024-03-28T22:47:52.871 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.871 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.871 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.871 INFO:teuthology.orchestra.run.smithi043.stdout: "address": { 2024-03-28T22:47:52.871 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "address", 2024-03-28T22:47:52.871 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.872 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.872 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.872 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-28T22:47:52.872 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.872 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.872 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.872 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.872 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.872 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.873 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.873 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.873 INFO:teuthology.orchestra.run.smithi043.stdout: "interval": { 2024-03-28T22:47:52.873 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "interval", 2024-03-28T22:47:52.873 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:52.873 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.873 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.873 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "15", 2024-03-28T22:47:52.873 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.874 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.874 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.874 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.874 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.874 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.874 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.874 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.874 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.874 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.874 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.875 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.875 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.875 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.875 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.875 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.875 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.875 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.875 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.875 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.876 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.876 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.876 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.876 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.876 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.876 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.876 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.876 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.876 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.876 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.877 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.877 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.877 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.877 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.877 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.877 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.877 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.877 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.877 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.878 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.878 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.878 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.878 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.878 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.878 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.878 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.878 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.878 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.879 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.879 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.879 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.879 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.879 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.879 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.879 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.879 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.879 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.879 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.880 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.880 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.880 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.880 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.880 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.880 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.880 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.880 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.880 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.881 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.881 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.881 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.881 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.881 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.881 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.881 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.881 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.881 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.882 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.882 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.882 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.882 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.882 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.882 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "telemetry", 2024-03-28T22:47:52.882 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.882 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.882 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.883 INFO:teuthology.orchestra.run.smithi043.stdout: "channel_basic": { 2024-03-28T22:47:52.883 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "channel_basic", 2024-03-28T22:47:52.883 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.883 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.883 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.883 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.883 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.883 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.883 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.883 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-28T22:47:52.884 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.884 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.884 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.884 INFO:teuthology.orchestra.run.smithi043.stdout: "channel_crash": { 2024-03-28T22:47:52.884 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "channel_crash", 2024-03-28T22:47:52.884 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.884 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.884 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.885 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.885 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.885 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.885 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.885 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-28T22:47:52.885 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.885 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.885 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.885 INFO:teuthology.orchestra.run.smithi043.stdout: "channel_device": { 2024-03-28T22:47:52.886 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "channel_device", 2024-03-28T22:47:52.886 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.886 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.886 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.886 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.886 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.886 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.886 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.886 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-28T22:47:52.887 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.887 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.887 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.887 INFO:teuthology.orchestra.run.smithi043.stdout: "channel_ident": { 2024-03-28T22:47:52.887 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "channel_ident", 2024-03-28T22:47:52.887 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.887 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.887 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.888 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.888 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.888 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.888 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.888 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-28T22:47:52.888 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.888 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.888 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.888 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.889 INFO:teuthology.orchestra.run.smithi043.stdout: "channel_perf": { 2024-03-28T22:47:52.889 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "channel_perf", 2024-03-28T22:47:52.889 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.889 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.889 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.889 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.889 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.889 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.889 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.890 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-28T22:47:52.890 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.890 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.890 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.890 INFO:teuthology.orchestra.run.smithi043.stdout: "contact": { 2024-03-28T22:47:52.890 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "contact", 2024-03-28T22:47:52.890 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.890 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.891 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.891 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.891 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.891 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.891 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.891 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.891 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.891 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.891 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.892 INFO:teuthology.orchestra.run.smithi043.stdout: "description": { 2024-03-28T22:47:52.892 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "description", 2024-03-28T22:47:52.892 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.892 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.892 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.892 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.892 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.892 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.893 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.893 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.893 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.893 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.893 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.893 INFO:teuthology.orchestra.run.smithi043.stdout: "device_url": { 2024-03-28T22:47:52.893 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "device_url", 2024-03-28T22:47:52.893 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.894 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.894 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.894 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-28T22:47:52.894 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.894 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.894 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.894 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.894 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.894 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.895 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.895 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": { 2024-03-28T22:47:52.895 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "enabled", 2024-03-28T22:47:52.895 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.895 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.895 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.895 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.895 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.896 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.896 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.896 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.896 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.896 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.896 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.896 INFO:teuthology.orchestra.run.smithi043.stdout: "interval": { 2024-03-28T22:47:52.897 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "interval", 2024-03-28T22:47:52.897 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.897 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.897 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.897 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "24", 2024-03-28T22:47:52.897 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "8", 2024-03-28T22:47:52.897 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.897 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.897 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.898 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.898 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.898 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.898 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.898 INFO:teuthology.orchestra.run.smithi043.stdout: "last_opt_revision": { 2024-03-28T22:47:52.898 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "last_opt_revision", 2024-03-28T22:47:52.898 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.898 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.899 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.899 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "1", 2024-03-28T22:47:52.899 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.899 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.899 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.899 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.899 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.899 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.899 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.900 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.900 INFO:teuthology.orchestra.run.smithi043.stdout: "leaderboard": { 2024-03-28T22:47:52.900 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "leaderboard", 2024-03-28T22:47:52.900 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.900 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.900 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.900 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.900 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.900 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.901 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.901 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.901 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.901 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.901 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.901 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.901 INFO:teuthology.orchestra.run.smithi043.stdout: "leaderboard_description": { 2024-03-28T22:47:52.901 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "leaderboard_description", 2024-03-28T22:47:52.901 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.902 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.902 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.902 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.902 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.902 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.902 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.902 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.902 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.902 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.903 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.903 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.903 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.903 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.903 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.903 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.903 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.903 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.903 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.904 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.904 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.904 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.904 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.904 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.904 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.904 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.904 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.904 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.905 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.905 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.905 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.905 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.905 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.905 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.905 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.905 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.905 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.906 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.906 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.906 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.906 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.906 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.906 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.906 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.906 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.907 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.907 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.907 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.907 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.907 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.907 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.907 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.907 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.907 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.908 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.908 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.908 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.908 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.908 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.908 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.908 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.908 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.908 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.909 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.909 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.909 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.909 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.909 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.909 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.909 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.909 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.909 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.910 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.910 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.910 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.910 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.910 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.910 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.910 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.910 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.911 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.911 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.911 INFO:teuthology.orchestra.run.smithi043.stdout: "organization": { 2024-03-28T22:47:52.911 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "organization", 2024-03-28T22:47:52.911 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.911 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.911 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.911 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.911 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.911 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.912 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.912 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.912 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.912 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.912 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.912 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.912 INFO:teuthology.orchestra.run.smithi043.stdout: "proxy": { 2024-03-28T22:47:52.912 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "proxy", 2024-03-28T22:47:52.913 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.913 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.913 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.913 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.913 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.913 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.913 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.913 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.913 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.914 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.914 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.914 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.914 INFO:teuthology.orchestra.run.smithi043.stdout: "url": { 2024-03-28T22:47:52.914 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "url", 2024-03-28T22:47:52.914 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.914 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.914 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.915 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-28T22:47:52.915 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.915 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.915 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.915 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.915 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.915 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.915 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.915 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.916 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.916 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.916 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.916 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "test_orchestrator", 2024-03-28T22:47:52.916 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.916 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.916 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.916 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.916 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.917 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.917 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.917 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.917 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.917 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.917 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.917 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.917 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.917 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.918 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.918 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.918 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.918 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.918 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.918 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.918 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.918 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.919 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.919 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.919 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.919 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.919 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.919 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.919 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.919 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.919 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.920 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.920 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.920 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.920 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.920 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.920 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.920 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.920 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.920 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.921 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.921 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.921 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.921 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.921 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.921 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.921 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.921 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.921 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.922 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.922 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.922 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.922 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.922 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.922 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.922 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.922 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.922 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.923 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.923 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.923 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.923 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.923 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.923 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.923 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.923 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.923 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.924 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.924 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.924 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.924 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.924 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.924 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.924 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.924 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.924 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.925 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "volumes", 2024-03-28T22:47:52.925 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.925 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.925 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.925 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.925 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.925 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.925 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.925 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.925 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.926 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.926 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.926 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.926 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.926 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.926 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.926 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.926 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.927 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.927 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.927 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.927 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.927 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.927 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.927 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.927 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.927 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.927 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.928 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.928 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.928 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.928 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.928 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.928 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.928 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.928 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.928 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.929 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.929 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.929 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.929 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.929 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.929 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.929 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.929 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.929 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.929 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.930 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.930 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.930 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.930 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.930 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.930 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.930 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.930 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.930 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.931 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.931 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.931 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.931 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.931 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.931 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.931 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.931 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.931 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.931 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.932 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.932 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.932 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.932 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.932 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.932 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.932 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.932 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.932 INFO:teuthology.orchestra.run.smithi043.stdout: "max_concurrent_clones": { 2024-03-28T22:47:52.933 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "max_concurrent_clones", 2024-03-28T22:47:52.933 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.933 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.933 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.933 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "4", 2024-03-28T22:47:52.933 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.933 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.933 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.933 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-28T22:47:52.933 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.934 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.934 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.934 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.934 INFO:teuthology.orchestra.run.smithi043.stdout: "periodic_async_work": { 2024-03-28T22:47:52.934 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "periodic_async_work", 2024-03-28T22:47:52.934 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.934 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.934 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.934 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.935 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.935 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.935 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.935 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Periodically check for async work", 2024-03-28T22:47:52.935 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.935 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.935 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.935 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.935 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot_clone_delay": { 2024-03-28T22:47:52.935 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "snapshot_clone_delay", 2024-03-28T22:47:52.936 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.936 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.936 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.936 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "0", 2024-03-28T22:47:52.936 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.936 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.936 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.936 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-28T22:47:52.936 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.936 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.937 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.937 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.937 INFO:teuthology.orchestra.run.smithi043.stdout: "snapshot_clone_no_wait": { 2024-03-28T22:47:52.937 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "snapshot_clone_no_wait", 2024-03-28T22:47:52.937 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.937 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.937 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.937 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "True", 2024-03-28T22:47:52.937 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.938 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.938 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.938 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-28T22:47:52.938 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.938 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.938 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.938 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.938 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.938 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.939 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.939 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "zabbix", 2024-03-28T22:47:52.939 INFO:teuthology.orchestra.run.smithi043.stdout: "can_run": true, 2024-03-28T22:47:52.939 INFO:teuthology.orchestra.run.smithi043.stdout: "error_string": "", 2024-03-28T22:47:52.939 INFO:teuthology.orchestra.run.smithi043.stdout: "module_options": { 2024-03-28T22:47:52.939 INFO:teuthology.orchestra.run.smithi043.stdout: "discovery_interval": { 2024-03-28T22:47:52.939 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "discovery_interval", 2024-03-28T22:47:52.939 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "uint", 2024-03-28T22:47:52.939 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.940 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.940 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "100", 2024-03-28T22:47:52.940 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.940 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.940 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.940 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.940 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.940 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.940 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.940 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.941 INFO:teuthology.orchestra.run.smithi043.stdout: "identifier": { 2024-03-28T22:47:52.941 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "identifier", 2024-03-28T22:47:52.941 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.941 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.941 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.941 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.941 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.941 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.941 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.942 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.942 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.942 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.942 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.942 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.942 INFO:teuthology.orchestra.run.smithi043.stdout: "interval": { 2024-03-28T22:47:52.942 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "interval", 2024-03-28T22:47:52.942 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "secs", 2024-03-28T22:47:52.943 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.943 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.943 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "60", 2024-03-28T22:47:52.943 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.943 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.943 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.943 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.943 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.943 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.943 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.944 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.944 INFO:teuthology.orchestra.run.smithi043.stdout: "log_level": { 2024-03-28T22:47:52.944 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_level", 2024-03-28T22:47:52.944 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.944 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.944 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.944 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.944 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.944 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.944 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.945 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.945 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.945 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.945 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.945 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.945 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.945 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.945 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.945 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.946 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.946 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.946 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster": { 2024-03-28T22:47:52.946 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster", 2024-03-28T22:47:52.946 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.946 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.946 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.946 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.947 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.947 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.947 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.947 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.947 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.947 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.947 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.947 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.947 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_cluster_level": { 2024-03-28T22:47:52.947 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_cluster_level", 2024-03-28T22:47:52.948 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.948 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.948 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.948 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "info", 2024-03-28T22:47:52.948 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.948 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.948 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [ 2024-03-28T22:47:52.948 INFO:teuthology.orchestra.run.smithi043.stdout: "", 2024-03-28T22:47:52.948 INFO:teuthology.orchestra.run.smithi043.stdout: "critical", 2024-03-28T22:47:52.948 INFO:teuthology.orchestra.run.smithi043.stdout: "debug", 2024-03-28T22:47:52.949 INFO:teuthology.orchestra.run.smithi043.stdout: "error", 2024-03-28T22:47:52.949 INFO:teuthology.orchestra.run.smithi043.stdout: "info", 2024-03-28T22:47:52.949 INFO:teuthology.orchestra.run.smithi043.stdout: "warning" 2024-03-28T22:47:52.949 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.949 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.949 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.949 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.949 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.950 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": { 2024-03-28T22:47:52.950 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "log_to_file", 2024-03-28T22:47:52.950 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "bool", 2024-03-28T22:47:52.950 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.950 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-03-28T22:47:52.950 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "False", 2024-03-28T22:47:52.950 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.950 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.950 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.951 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.951 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.951 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.951 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.951 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.951 INFO:teuthology.orchestra.run.smithi043.stdout: "zabbix_host": { 2024-03-28T22:47:52.951 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "zabbix_host", 2024-03-28T22:47:52.951 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.952 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.952 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.952 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "", 2024-03-28T22:47:52.952 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.952 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.952 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.952 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.952 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.952 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.953 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.953 INFO:teuthology.orchestra.run.smithi043.stdout: "zabbix_port": { 2024-03-28T22:47:52.953 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "zabbix_port", 2024-03-28T22:47:52.953 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "int", 2024-03-28T22:47:52.953 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.953 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.954 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "10051", 2024-03-28T22:47:52.954 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.954 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.954 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.954 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.954 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.954 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.954 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.954 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.955 INFO:teuthology.orchestra.run.smithi043.stdout: "zabbix_sender": { 2024-03-28T22:47:52.955 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "zabbix_sender", 2024-03-28T22:47:52.955 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "str", 2024-03-28T22:47:52.955 INFO:teuthology.orchestra.run.smithi043.stdout: "level": "advanced", 2024-03-28T22:47:52.955 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-03-28T22:47:52.955 INFO:teuthology.orchestra.run.smithi043.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-28T22:47:52.955 INFO:teuthology.orchestra.run.smithi043.stdout: "min": "", 2024-03-28T22:47:52.955 INFO:teuthology.orchestra.run.smithi043.stdout: "max": "", 2024-03-28T22:47:52.956 INFO:teuthology.orchestra.run.smithi043.stdout: "enum_allowed": [], 2024-03-28T22:47:52.956 INFO:teuthology.orchestra.run.smithi043.stdout: "desc": "", 2024-03-28T22:47:52.956 INFO:teuthology.orchestra.run.smithi043.stdout: "long_desc": "", 2024-03-28T22:47:52.956 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-03-28T22:47:52.956 INFO:teuthology.orchestra.run.smithi043.stdout: "see_also": [] 2024-03-28T22:47:52.956 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.956 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.956 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.956 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.957 INFO:teuthology.orchestra.run.smithi043.stdout: "services": { 2024-03-28T22:47:52.957 INFO:teuthology.orchestra.run.smithi043.stdout: "dashboard": "https://172.21.15.43:8443/" 2024-03-28T22:47:52.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.957 INFO:teuthology.orchestra.run.smithi043.stdout: "always_on_modules": { 2024-03-28T22:47:52.957 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus": [ 2024-03-28T22:47:52.957 INFO:teuthology.orchestra.run.smithi043.stdout: "balancer", 2024-03-28T22:47:52.957 INFO:teuthology.orchestra.run.smithi043.stdout: "crash", 2024-03-28T22:47:52.957 INFO:teuthology.orchestra.run.smithi043.stdout: "devicehealth", 2024-03-28T22:47:52.957 INFO:teuthology.orchestra.run.smithi043.stdout: "orchestrator", 2024-03-28T22:47:52.958 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscaler", 2024-03-28T22:47:52.958 INFO:teuthology.orchestra.run.smithi043.stdout: "progress", 2024-03-28T22:47:52.958 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_support", 2024-03-28T22:47:52.958 INFO:teuthology.orchestra.run.smithi043.stdout: "status", 2024-03-28T22:47:52.958 INFO:teuthology.orchestra.run.smithi043.stdout: "telemetry", 2024-03-28T22:47:52.958 INFO:teuthology.orchestra.run.smithi043.stdout: "volumes" 2024-03-28T22:47:52.958 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.958 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific": [ 2024-03-28T22:47:52.959 INFO:teuthology.orchestra.run.smithi043.stdout: "balancer", 2024-03-28T22:47:52.959 INFO:teuthology.orchestra.run.smithi043.stdout: "crash", 2024-03-28T22:47:52.959 INFO:teuthology.orchestra.run.smithi043.stdout: "devicehealth", 2024-03-28T22:47:52.959 INFO:teuthology.orchestra.run.smithi043.stdout: "orchestrator", 2024-03-28T22:47:52.959 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscaler", 2024-03-28T22:47:52.959 INFO:teuthology.orchestra.run.smithi043.stdout: "progress", 2024-03-28T22:47:52.959 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_support", 2024-03-28T22:47:52.959 INFO:teuthology.orchestra.run.smithi043.stdout: "status", 2024-03-28T22:47:52.959 INFO:teuthology.orchestra.run.smithi043.stdout: "telemetry", 2024-03-28T22:47:52.960 INFO:teuthology.orchestra.run.smithi043.stdout: "volumes" 2024-03-28T22:47:52.960 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.960 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy": [ 2024-03-28T22:47:52.960 INFO:teuthology.orchestra.run.smithi043.stdout: "balancer", 2024-03-28T22:47:52.960 INFO:teuthology.orchestra.run.smithi043.stdout: "crash", 2024-03-28T22:47:52.960 INFO:teuthology.orchestra.run.smithi043.stdout: "devicehealth", 2024-03-28T22:47:52.960 INFO:teuthology.orchestra.run.smithi043.stdout: "orchestrator", 2024-03-28T22:47:52.960 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscaler", 2024-03-28T22:47:52.960 INFO:teuthology.orchestra.run.smithi043.stdout: "progress", 2024-03-28T22:47:52.961 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_support", 2024-03-28T22:47:52.961 INFO:teuthology.orchestra.run.smithi043.stdout: "status", 2024-03-28T22:47:52.961 INFO:teuthology.orchestra.run.smithi043.stdout: "telemetry", 2024-03-28T22:47:52.961 INFO:teuthology.orchestra.run.smithi043.stdout: "volumes" 2024-03-28T22:47:52.961 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.961 INFO:teuthology.orchestra.run.smithi043.stdout: "reef": [ 2024-03-28T22:47:52.961 INFO:teuthology.orchestra.run.smithi043.stdout: "balancer", 2024-03-28T22:47:52.961 INFO:teuthology.orchestra.run.smithi043.stdout: "crash", 2024-03-28T22:47:52.961 INFO:teuthology.orchestra.run.smithi043.stdout: "devicehealth", 2024-03-28T22:47:52.962 INFO:teuthology.orchestra.run.smithi043.stdout: "orchestrator", 2024-03-28T22:47:52.962 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscaler", 2024-03-28T22:47:52.962 INFO:teuthology.orchestra.run.smithi043.stdout: "progress", 2024-03-28T22:47:52.962 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_support", 2024-03-28T22:47:52.962 INFO:teuthology.orchestra.run.smithi043.stdout: "status", 2024-03-28T22:47:52.962 INFO:teuthology.orchestra.run.smithi043.stdout: "telemetry", 2024-03-28T22:47:52.962 INFO:teuthology.orchestra.run.smithi043.stdout: "volumes" 2024-03-28T22:47:52.962 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-03-28T22:47:52.963 INFO:teuthology.orchestra.run.smithi043.stdout: "squid": [ 2024-03-28T22:47:52.963 INFO:teuthology.orchestra.run.smithi043.stdout: "balancer", 2024-03-28T22:47:52.963 INFO:teuthology.orchestra.run.smithi043.stdout: "crash", 2024-03-28T22:47:52.963 INFO:teuthology.orchestra.run.smithi043.stdout: "devicehealth", 2024-03-28T22:47:52.963 INFO:teuthology.orchestra.run.smithi043.stdout: "orchestrator", 2024-03-28T22:47:52.963 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscaler", 2024-03-28T22:47:52.963 INFO:teuthology.orchestra.run.smithi043.stdout: "progress", 2024-03-28T22:47:52.963 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_support", 2024-03-28T22:47:52.963 INFO:teuthology.orchestra.run.smithi043.stdout: "status", 2024-03-28T22:47:52.964 INFO:teuthology.orchestra.run.smithi043.stdout: "telemetry", 2024-03-28T22:47:52.964 INFO:teuthology.orchestra.run.smithi043.stdout: "volumes" 2024-03-28T22:47:52.964 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-28T22:47:52.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.964 INFO:teuthology.orchestra.run.smithi043.stdout: "last_failure_osd_epoch": 97, 2024-03-28T22:47:52.964 INFO:teuthology.orchestra.run.smithi043.stdout: "active_clients": [ 2024-03-28T22:47:52.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.964 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "devicehealth", 2024-03-28T22:47:52.965 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-28T22:47:52.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.965 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-28T22:47:52.965 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:0", 2024-03-28T22:47:52.965 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 378105091 2024-03-28T22:47:52.965 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.965 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-28T22:47:52.965 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.966 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "libcephsqlite", 2024-03-28T22:47:52.966 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-28T22:47:52.966 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.966 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-28T22:47:52.966 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:0", 2024-03-28T22:47:52.966 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 2852493049 2024-03-28T22:47:52.966 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.966 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-28T22:47:52.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.967 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "rbd_support", 2024-03-28T22:47:52.967 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-28T22:47:52.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.967 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-28T22:47:52.967 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:0", 2024-03-28T22:47:52.967 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 566721074 2024-03-28T22:47:52.967 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.967 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-28T22:47:52.968 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.968 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.968 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "snap_schedule", 2024-03-28T22:47:52.968 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-28T22:47:52.968 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.968 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-28T22:47:52.968 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:0", 2024-03-28T22:47:52.968 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1514635630 2024-03-28T22:47:52.968 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.968 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-28T22:47:52.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.969 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.969 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "snap_schedule", 2024-03-28T22:47:52.969 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-28T22:47:52.969 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.969 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-28T22:47:52.969 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:0", 2024-03-28T22:47:52.969 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 4275315579 2024-03-28T22:47:52.969 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.970 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-28T22:47:52.970 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:52.970 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.970 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "volumes", 2024-03-28T22:47:52.970 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-03-28T22:47:52.970 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-03-28T22:47:52.970 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-03-28T22:47:52.970 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:0", 2024-03-28T22:47:52.971 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3426568306 2024-03-28T22:47:52.971 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.971 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-28T22:47:52.971 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:52.971 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-03-28T22:47:52.971 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:47:52.972 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:50 smithi172 ceph-mon[36566]: pgmap v333: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:47:52.973 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:50 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2214579207' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:47:52.973 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:50 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/559071493' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-28T22:47:52.973 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:50 smithi082 ceph-mon[36471]: pgmap v333: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:47:52.973 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:50 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2214579207' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:47:52.973 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:50 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/559071493' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-28T22:47:52.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.899+0000 7f0a3a800640 1 -- 172.21.15.43:0/559071493 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0a180a1f70 msgr2=0x7f0a180a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:52.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.899+0000 7f0a3a800640 1 --2- 172.21.15.43:0/559071493 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0a180a1f70 0x7f0a180a4430 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0a24011440 tx=0x7f0a24005f90 comp rx=0 tx=0).stop 2024-03-28T22:47:52.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.899+0000 7f0a3a800640 1 -- 172.21.15.43:0/559071493 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a34109910 msgr2=0x7f0a340766e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:47:52.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.899+0000 7f0a3a800640 1 --2- 172.21.15.43:0/559071493 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a34109910 0x7f0a340766e0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c02c430 tx=0x7f0a2c0336b0 comp rx=0 tx=0).stop 2024-03-28T22:47:52.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.899+0000 7f0a3a800640 1 -- 172.21.15.43:0/559071493 shutdown_connections 2024-03-28T22:47:52.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.899+0000 7f0a3a800640 1 --2- 172.21.15.43:0/559071493 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0a180a1f70 0x7f0a180a4430 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:52.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.899+0000 7f0a3a800640 1 --2- 172.21.15.43:0/559071493 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0a34109910 0x7f0a340766e0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:52.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.899+0000 7f0a3a800640 1 --2- 172.21.15.43:0/559071493 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0a341042c0 0x7f0a340761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:52.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.899+0000 7f0a3a800640 1 --2- 172.21.15.43:0/559071493 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0a34103900 0x7f0a34075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:47:52.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.899+0000 7f0a3a800640 1 -- 172.21.15.43:0/559071493 >> 172.21.15.43:0/559071493 conn(0x7f0a340ff390 msgr2=0x7f0a340ff8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:47:52.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.899+0000 7f0a3a800640 1 -- 172.21.15.43:0/559071493 shutdown_connections 2024-03-28T22:47:52.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:47:50.900+0000 7f0a3a800640 1 -- 172.21.15.43:0/559071493 wait complete. 2024-03-28T22:47:52.983 DEBUG:tasks.check_counter:Getting stats from a 2024-03-28T22:47:52.983 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-28T22:47:52.983 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:50 smithi043 ceph-mon[31823]: pgmap v333: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:47:52.984 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:50 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2214579207' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:47:52.984 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:50 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/559071493' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-28T22:47:53.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:52 smithi082 ceph-mon[36471]: pgmap v334: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:47:53.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:52 smithi043 ceph-mon[31823]: pgmap v334: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:47:53.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:52 smithi172 ceph-mon[36566]: pgmap v334: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:47:53.437 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:47:55.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:54 smithi082 ceph-mon[36471]: pgmap v335: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:47:55.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:54 smithi043 ceph-mon[31823]: pgmap v335: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:47:55.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:54 smithi172 ceph-mon[36566]: pgmap v335: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:47:55.882 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:47:55.882 INFO:teuthology.orchestra.run.smithi043.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T22:47:55.882 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_messages": 4, 2024-03-28T22:47:55.883 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_messages": 4, 2024-03-28T22:47:55.883 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_bytes": 3072, 2024-03-28T22:47:55.883 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_bytes": 448, 2024-03-28T22:47:55.883 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_created_connections": 4, 2024-03-28T22:47:55.883 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_active_connections": 0, 2024-03-28T22:47:55.883 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_total_time": 0.004639238, 2024-03-28T22:47:55.883 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_send_time": 0.000068170, 2024-03-28T22:47:55.883 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_recv_time": 0.001262165, 2024-03-28T22:47:55.883 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T22:47:55.884 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:47:55.884 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 3, 2024-03-28T22:47:55.884 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.001572686, 2024-03-28T22:47:55.884 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000524228 2024-03-28T22:47:55.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:55.884 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:47:55.884 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:55.884 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:55.884 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:55.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:55.885 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-28T22:47:55.885 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_encrypted_bytes": 448 2024-03-28T22:47:55.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:55.885 INFO:teuthology.orchestra.run.smithi043.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T22:47:55.885 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_messages": 246, 2024-03-28T22:47:55.885 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_messages": 233, 2024-03-28T22:47:55.885 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_bytes": 1848992, 2024-03-28T22:47:55.885 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_bytes": 143904, 2024-03-28T22:47:55.885 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_created_connections": 2, 2024-03-28T22:47:55.885 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_active_connections": 1, 2024-03-28T22:47:55.886 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_total_time": 0.112346784, 2024-03-28T22:47:55.886 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_send_time": 0.036620437, 2024-03-28T22:47:55.886 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_recv_time": 0.092298801, 2024-03-28T22:47:55.886 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_fast_dispatch_time": 0.011187848, 2024-03-28T22:47:55.886 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:47:55.886 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 232, 2024-03-28T22:47:55.886 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.023319184, 2024-03-28T22:47:55.886 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000100513 2024-03-28T22:47:55.886 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:55.887 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:47:55.887 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:55.887 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:55.887 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:55.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:55.887 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_encrypted_bytes": 1848992, 2024-03-28T22:47:55.887 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_encrypted_bytes": 143904 2024-03-28T22:47:55.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:55.887 INFO:teuthology.orchestra.run.smithi043.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T22:47:55.888 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_messages": 2, 2024-03-28T22:47:55.888 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_messages": 179, 2024-03-28T22:47:55.888 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_bytes": 320, 2024-03-28T22:47:55.888 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_bytes": 187072, 2024-03-28T22:47:55.888 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_created_connections": 5, 2024-03-28T22:47:55.888 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_active_connections": 3, 2024-03-28T22:47:55.888 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_total_time": 0.040920447, 2024-03-28T22:47:55.888 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_send_time": 0.028884188, 2024-03-28T22:47:55.888 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_recv_time": 0.005867872, 2024-03-28T22:47:55.888 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T22:47:55.889 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:47:55.889 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 176, 2024-03-28T22:47:55.889 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.032012390, 2024-03-28T22:47:55.889 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000181888 2024-03-28T22:47:55.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:55.889 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:47:55.889 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:55.889 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:55.890 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:55.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:55.890 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T22:47:55.890 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_encrypted_bytes": 187072 2024-03-28T22:47:55.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:55.890 INFO:teuthology.orchestra.run.smithi043.stdout: "cct": { 2024-03-28T22:47:55.890 INFO:teuthology.orchestra.run.smithi043.stdout: "total_workers": 0, 2024-03-28T22:47:55.890 INFO:teuthology.orchestra.run.smithi043.stdout: "unhealthy_workers": 0 2024-03-28T22:47:55.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:55.891 INFO:teuthology.orchestra.run.smithi043.stdout: "mempool": { 2024-03-28T22:47:55.891 INFO:teuthology.orchestra.run.smithi043.stdout: "bloom_filter_bytes": 0, 2024-03-28T22:47:55.891 INFO:teuthology.orchestra.run.smithi043.stdout: "bloom_filter_items": 0, 2024-03-28T22:47:55.891 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T22:47:55.891 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_alloc_items": 0, 2024-03-28T22:47:55.891 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T22:47:55.891 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_data_items": 0, 2024-03-28T22:47:55.891 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T22:47:55.891 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T22:47:55.892 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T22:47:55.892 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T22:47:55.892 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T22:47:55.892 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_other_items": 0, 2024-03-28T22:47:55.892 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T22:47:55.892 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T22:47:55.892 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_extent_bytes": 0, 2024-03-28T22:47:55.892 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_extent_items": 0, 2024-03-28T22:47:55.892 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_blob_bytes": 0, 2024-03-28T22:47:55.893 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_blob_items": 0, 2024-03-28T22:47:55.893 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T22:47:55.893 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T22:47:55.893 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T22:47:55.893 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T22:47:55.893 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T22:47:55.893 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_items": 0, 2024-03-28T22:47:55.893 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_txc_bytes": 0, 2024-03-28T22:47:55.893 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_txc_items": 0, 2024-03-28T22:47:55.893 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T22:47:55.894 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T22:47:55.894 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_writing_bytes": 0, 2024-03-28T22:47:55.894 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_writing_items": 0, 2024-03-28T22:47:55.894 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_bytes": 0, 2024-03-28T22:47:55.894 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_items": 0, 2024-03-28T22:47:55.894 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T22:47:55.894 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_file_reader_items": 0, 2024-03-28T22:47:55.894 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T22:47:55.895 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_file_writer_items": 0, 2024-03-28T22:47:55.895 INFO:teuthology.orchestra.run.smithi043.stdout: "buffer_anon_bytes": 149745, 2024-03-28T22:47:55.895 INFO:teuthology.orchestra.run.smithi043.stdout: "buffer_anon_items": 31, 2024-03-28T22:47:55.895 INFO:teuthology.orchestra.run.smithi043.stdout: "buffer_meta_bytes": 0, 2024-03-28T22:47:55.895 INFO:teuthology.orchestra.run.smithi043.stdout: "buffer_meta_items": 0, 2024-03-28T22:47:55.895 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_bytes": 0, 2024-03-28T22:47:55.895 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_items": 0, 2024-03-28T22:47:55.895 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mapbl_bytes": 0, 2024-03-28T22:47:55.895 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mapbl_items": 0, 2024-03-28T22:47:55.896 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pglog_bytes": 0, 2024-03-28T22:47:55.896 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pglog_items": 0, 2024-03-28T22:47:55.896 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_bytes": 0, 2024-03-28T22:47:55.896 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_items": 0, 2024-03-28T22:47:55.896 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T22:47:55.896 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_mapping_items": 0, 2024-03-28T22:47:55.896 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap_bytes": 0, 2024-03-28T22:47:55.896 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap_items": 0, 2024-03-28T22:47:55.896 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_co_bytes": 928, 2024-03-28T22:47:55.897 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_co_items": 2, 2024-03-28T22:47:55.897 INFO:teuthology.orchestra.run.smithi043.stdout: "unittest_1_bytes": 0, 2024-03-28T22:47:55.897 INFO:teuthology.orchestra.run.smithi043.stdout: "unittest_1_items": 0, 2024-03-28T22:47:55.897 INFO:teuthology.orchestra.run.smithi043.stdout: "unittest_2_bytes": 0, 2024-03-28T22:47:55.897 INFO:teuthology.orchestra.run.smithi043.stdout: "unittest_2_items": 0 2024-03-28T22:47:55.897 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:55.897 INFO:teuthology.orchestra.run.smithi043.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T22:47:55.897 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 0, 2024-03-28T22:47:55.897 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 104857600, 2024-03-28T22:47:55.897 INFO:teuthology.orchestra.run.smithi043.stdout: "get_started": 0, 2024-03-28T22:47:55.898 INFO:teuthology.orchestra.run.smithi043.stdout: "get": 248, 2024-03-28T22:47:55.898 INFO:teuthology.orchestra.run.smithi043.stdout: "get_sum": 1814292, 2024-03-28T22:47:55.898 INFO:teuthology.orchestra.run.smithi043.stdout: "get_or_fail_fail": 0, 2024-03-28T22:47:55.898 INFO:teuthology.orchestra.run.smithi043.stdout: "get_or_fail_success": 248, 2024-03-28T22:47:55.898 INFO:teuthology.orchestra.run.smithi043.stdout: "take": 0, 2024-03-28T22:47:55.898 INFO:teuthology.orchestra.run.smithi043.stdout: "take_sum": 0, 2024-03-28T22:47:55.898 INFO:teuthology.orchestra.run.smithi043.stdout: "put": 248, 2024-03-28T22:47:55.898 INFO:teuthology.orchestra.run.smithi043.stdout: "put_sum": 1814292, 2024-03-28T22:47:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: "wait": { 2024-03-28T22:47:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:55.899 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:47:56.733 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-03-28T22:47:56.733 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-03-28T22:47:56.733 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2024-03-28T22:47:56.733 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2024-03-28T22:47:56.734 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-03-28T22:47:56.734 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-03-28T22:47:56.734 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-03-28T22:47:56.734 DEBUG:tasks.check_counter:Getting stats from d 2024-03-28T22:47:56.734 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-28T22:47:57.227 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:47:57.264 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:56 smithi043 ceph-mon[31823]: pgmap v336: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:47:57.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:56 smithi172 ceph-mon[36566]: pgmap v336: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:47:57.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:56 smithi082 ceph-mon[36471]: pgmap v336: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:47:59.216 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:47:58 smithi043 ceph-mon[31823]: pgmap v337: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:47:59.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:47:58 smithi172 ceph-mon[36566]: pgmap v337: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:47:59.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:47:58 smithi082 ceph-mon[36471]: pgmap v337: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:47:59.556 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:47:59.556 INFO:teuthology.orchestra.run.smithi043.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T22:47:59.556 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_messages": 2343, 2024-03-28T22:47:59.556 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_messages": 2345, 2024-03-28T22:47:59.556 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_bytes": 667542, 2024-03-28T22:47:59.556 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_bytes": 745081, 2024-03-28T22:47:59.557 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_created_connections": 6, 2024-03-28T22:47:59.557 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_active_connections": 2, 2024-03-28T22:47:59.557 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_total_time": 0.410830819, 2024-03-28T22:47:59.557 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_send_time": 0.129203508, 2024-03-28T22:47:59.557 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_recv_time": 0.103502722, 2024-03-28T22:47:59.557 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_fast_dispatch_time": 0.139653093, 2024-03-28T22:47:59.557 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:47:59.557 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 2343, 2024-03-28T22:47:59.558 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.229339130, 2024-03-28T22:47:59.558 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000097882 2024-03-28T22:47:59.558 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.558 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:47:59.558 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.558 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.558 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.558 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.559 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T22:47:59.559 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T22:47:59.559 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.559 INFO:teuthology.orchestra.run.smithi043.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T22:47:59.559 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_messages": 8, 2024-03-28T22:47:59.559 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_messages": 188, 2024-03-28T22:47:59.559 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_bytes": 3948, 2024-03-28T22:47:59.559 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_bytes": 474298, 2024-03-28T22:47:59.560 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_created_connections": 6, 2024-03-28T22:47:59.560 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_active_connections": 4, 2024-03-28T22:47:59.560 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_total_time": 0.039993890, 2024-03-28T22:47:59.560 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_send_time": 0.027875261, 2024-03-28T22:47:59.560 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_recv_time": 0.006217118, 2024-03-28T22:47:59.560 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_fast_dispatch_time": 0.000074139, 2024-03-28T22:47:59.560 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:47:59.561 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 186, 2024-03-28T22:47:59.561 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.026444859, 2024-03-28T22:47:59.561 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000142176 2024-03-28T22:47:59.561 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.561 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:47:59.561 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.561 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.561 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.562 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.562 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-03-28T22:47:59.562 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_encrypted_bytes": 473600 2024-03-28T22:47:59.562 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.562 INFO:teuthology.orchestra.run.smithi043.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T22:47:59.562 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_messages": 280, 2024-03-28T22:47:59.562 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_messages": 263, 2024-03-28T22:47:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_bytes": 1905964, 2024-03-28T22:47:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_bytes": 152686, 2024-03-28T22:47:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_created_connections": 3, 2024-03-28T22:47:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_active_connections": 2, 2024-03-28T22:47:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_total_time": 0.122107715, 2024-03-28T22:47:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_send_time": 0.028242668, 2024-03-28T22:47:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_recv_time": 0.125832191, 2024-03-28T22:47:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_fast_dispatch_time": 0.011620315, 2024-03-28T22:47:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:47:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 262, 2024-03-28T22:47:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.027650718, 2024-03-28T22:47:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000105537 2024-03-28T22:47:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:47:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_encrypted_bytes": 1905696, 2024-03-28T22:47:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_encrypted_bytes": 152368 2024-03-28T22:47:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: "cct": { 2024-03-28T22:47:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: "total_workers": 1, 2024-03-28T22:47:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: "unhealthy_workers": 0 2024-03-28T22:47:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.566 INFO:teuthology.orchestra.run.smithi043.stdout: "finisher-MDSRank": { 2024-03-28T22:47:59.566 INFO:teuthology.orchestra.run.smithi043.stdout: "queue_len": 0, 2024-03-28T22:47:59.566 INFO:teuthology.orchestra.run.smithi043.stdout: "complete_latency": { 2024-03-28T22:47:59.566 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 3891, 2024-03-28T22:47:59.566 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.209990271, 2024-03-28T22:47:59.566 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000053968 2024-03-28T22:47:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: "finisher-PurgeQueue": { 2024-03-28T22:47:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: "queue_len": 0, 2024-03-28T22:47:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: "complete_latency": { 2024-03-28T22:47:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": { 2024-03-28T22:47:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: "request": 0, 2024-03-28T22:47:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: "reply": 0, 2024-03-28T22:47:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: "reply_latency": { 2024-03-28T22:47:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: "slow_reply": 0, 2024-03-28T22:47:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: "forward": 0, 2024-03-28T22:47:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: "dir_fetch_complete": 0, 2024-03-28T22:47:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: "dir_fetch_keys": 0, 2024-03-28T22:47:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: "dir_commit": 0, 2024-03-28T22:47:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: "dir_split": 0, 2024-03-28T22:47:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: "dir_merge": 0, 2024-03-28T22:47:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: "inodes": 0, 2024-03-28T22:47:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: "inodes_top": 0, 2024-03-28T22:47:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: "inodes_bottom": 0, 2024-03-28T22:47:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: "inodes_pin_tail": 0, 2024-03-28T22:47:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: "inodes_pinned": 0, 2024-03-28T22:47:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: "inodes_expired": 0, 2024-03-28T22:47:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: "inodes_with_caps": 0, 2024-03-28T22:47:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": 0, 2024-03-28T22:47:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: "subtrees": 1, 2024-03-28T22:47:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: "traverse": 0, 2024-03-28T22:47:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: "traverse_hit": 0, 2024-03-28T22:47:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: "traverse_forward": 0, 2024-03-28T22:47:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: "traverse_discover": 0, 2024-03-28T22:47:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: "traverse_dir_fetch": 0, 2024-03-28T22:47:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: "traverse_remote_ino": 0, 2024-03-28T22:47:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: "traverse_lock": 0, 2024-03-28T22:47:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: "load_cent": 0, 2024-03-28T22:47:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: "q": 0, 2024-03-28T22:47:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: "exported": 0, 2024-03-28T22:47:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: "exported_inodes": 0, 2024-03-28T22:47:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: "imported": 0, 2024-03-28T22:47:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: "imported_inodes": 0, 2024-03-28T22:47:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: "openino_dir_fetch": 0, 2024-03-28T22:47:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: "openino_backtrace_fetch": 0, 2024-03-28T22:47:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: "openino_peer_discover": 0, 2024-03-28T22:47:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: "root_rfiles": 0, 2024-03-28T22:47:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: "root_rbytes": 0, 2024-03-28T22:47:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: "root_rsnaps": 0, 2024-03-28T22:47:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T22:47:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_set_tag": 0, 2024-03-28T22:47:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T22:47:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_inotable_repaired": 0, 2024-03-28T22:47:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_dir_inodes": 0, 2024-03-28T22:47:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T22:47:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T22:47:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_file_inodes": 0, 2024-03-28T22:47:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: "handle_inode_file_caps": 0, 2024-03-28T22:47:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T22:47:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_cap_op_grant": 0, 2024-03-28T22:47:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T22:47:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T22:47:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T22:47:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: "handle_client_caps": 0, 2024-03-28T22:47:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: "handle_client_caps_dirty": 0, 2024-03-28T22:47:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: "handle_client_cap_release": 0, 2024-03-28T22:47:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: "process_request_cap_release": 0 2024-03-28T22:47:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_cache": { 2024-03-28T22:47:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: "dir_update": 0, 2024-03-28T22:47:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: "dir_update_receipt": 0, 2024-03-28T22:47:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: "dir_try_discover": 0, 2024-03-28T22:47:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: "dir_send_discover": 0, 2024-03-28T22:47:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: "dir_handle_discover": 0, 2024-03-28T22:47:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: "num_strays": 0, 2024-03-28T22:47:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: "num_strays_delayed": 0, 2024-03-28T22:47:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: "num_strays_enqueuing": 0, 2024-03-28T22:47:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: "strays_created": 0, 2024-03-28T22:47:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: "strays_enqueued": 0, 2024-03-28T22:47:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: "strays_reintegrated": 0, 2024-03-28T22:47:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: "strays_migrated": 0, 2024-03-28T22:47:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: "num_recovering_processing": 0, 2024-03-28T22:47:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: "num_recovering_enqueued": 0, 2024-03-28T22:47:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: "num_recovering_prioritized": 0, 2024-03-28T22:47:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: "recovery_started": 0, 2024-03-28T22:47:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: "recovery_completed": 0, 2024-03-28T22:47:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: "ireq_quiesce_path": 0, 2024-03-28T22:47:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: "ireq_quiesce_inode": 0, 2024-03-28T22:47:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T22:47:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: "ireq_exportdir": 0, 2024-03-28T22:47:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: "ireq_flush": 0, 2024-03-28T22:47:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: "ireq_fragmentdir": 0, 2024-03-28T22:47:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: "ireq_fragstats": 0, 2024-03-28T22:47:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: "ireq_inodestats": 0 2024-03-28T22:47:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_log": { 2024-03-28T22:47:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: "evlrg": 0, 2024-03-28T22:47:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: "evadd": 0, 2024-03-28T22:47:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: "evex": 0, 2024-03-28T22:47:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: "evtrm": 0, 2024-03-28T22:47:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: "ev": 8, 2024-03-28T22:47:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: "evexg": 0, 2024-03-28T22:47:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: "evexd": 0, 2024-03-28T22:47:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: "segadd": 0, 2024-03-28T22:47:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: "segex": 0, 2024-03-28T22:47:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: "segtrm": 0, 2024-03-28T22:47:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: "seg": 2, 2024-03-28T22:47:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: "segmjr": 2, 2024-03-28T22:47:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: "segexg": 0, 2024-03-28T22:47:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: "segexd": 0, 2024-03-28T22:47:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: "expos": 4194304, 2024-03-28T22:47:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: "wrpos": 4196653, 2024-03-28T22:47:59.581 INFO:teuthology.orchestra.run.smithi043.stdout: "rdpos": 4196512, 2024-03-28T22:47:59.581 INFO:teuthology.orchestra.run.smithi043.stdout: "jlat": { 2024-03-28T22:47:59.581 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.581 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.581 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.581 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.581 INFO:teuthology.orchestra.run.smithi043.stdout: "replayed": 8 2024-03-28T22:47:59.581 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.581 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_mem": { 2024-03-28T22:47:59.582 INFO:teuthology.orchestra.run.smithi043.stdout: "ino": 3, 2024-03-28T22:47:59.582 INFO:teuthology.orchestra.run.smithi043.stdout: "ino+": 3, 2024-03-28T22:47:59.582 INFO:teuthology.orchestra.run.smithi043.stdout: "ino-": 0, 2024-03-28T22:47:59.582 INFO:teuthology.orchestra.run.smithi043.stdout: "dir": 1, 2024-03-28T22:47:59.582 INFO:teuthology.orchestra.run.smithi043.stdout: "dir+": 1, 2024-03-28T22:47:59.582 INFO:teuthology.orchestra.run.smithi043.stdout: "dir-": 0, 2024-03-28T22:47:59.582 INFO:teuthology.orchestra.run.smithi043.stdout: "dn": 0, 2024-03-28T22:47:59.582 INFO:teuthology.orchestra.run.smithi043.stdout: "dn+": 0, 2024-03-28T22:47:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: "dn-": 0, 2024-03-28T22:47:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: "cap": 0, 2024-03-28T22:47:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: "cap+": 0, 2024-03-28T22:47:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: "cap-": 0, 2024-03-28T22:47:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: "rss": 48896, 2024-03-28T22:47:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: "heap": 215420 2024-03-28T22:47:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_server": { 2024-03-28T22:47:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: "dispatch_client_request": 0, 2024-03-28T22:47:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: "dispatch_server_request": 0, 2024-03-28T22:47:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: "handle_client_request": 0, 2024-03-28T22:47:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: "handle_client_session": 0, 2024-03-28T22:47:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: "handle_peer_request": 0, 2024-03-28T22:47:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: "req_create_latency": { 2024-03-28T22:47:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.585 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.585 INFO:teuthology.orchestra.run.smithi043.stdout: "req_getattr_latency": { 2024-03-28T22:47:59.585 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.585 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.585 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.585 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.586 INFO:teuthology.orchestra.run.smithi043.stdout: "req_getfilelock_latency": { 2024-03-28T22:47:59.586 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.586 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.586 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.586 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.586 INFO:teuthology.orchestra.run.smithi043.stdout: "req_link_latency": { 2024-03-28T22:47:59.586 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.586 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.587 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.587 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.587 INFO:teuthology.orchestra.run.smithi043.stdout: "req_lookup_latency": { 2024-03-28T22:47:59.587 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.587 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.587 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.587 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.587 INFO:teuthology.orchestra.run.smithi043.stdout: "req_lookuphash_latency": { 2024-03-28T22:47:59.587 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.588 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.588 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.588 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.588 INFO:teuthology.orchestra.run.smithi043.stdout: "req_lookupino_latency": { 2024-03-28T22:47:59.588 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.588 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.588 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.588 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.588 INFO:teuthology.orchestra.run.smithi043.stdout: "req_lookupname_latency": { 2024-03-28T22:47:59.589 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.589 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.589 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.589 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.589 INFO:teuthology.orchestra.run.smithi043.stdout: "req_lookupparent_latency": { 2024-03-28T22:47:59.589 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.589 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.589 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.590 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.590 INFO:teuthology.orchestra.run.smithi043.stdout: "req_lookupsnap_latency": { 2024-03-28T22:47:59.590 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.590 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.590 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.590 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.590 INFO:teuthology.orchestra.run.smithi043.stdout: "req_lssnap_latency": { 2024-03-28T22:47:59.591 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.591 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.591 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.591 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.591 INFO:teuthology.orchestra.run.smithi043.stdout: "req_mkdir_latency": { 2024-03-28T22:47:59.591 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.592 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.592 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.592 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.592 INFO:teuthology.orchestra.run.smithi043.stdout: "req_mknod_latency": { 2024-03-28T22:47:59.592 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.593 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.593 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.593 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.593 INFO:teuthology.orchestra.run.smithi043.stdout: "req_mksnap_latency": { 2024-03-28T22:47:59.593 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.593 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.593 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.593 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.594 INFO:teuthology.orchestra.run.smithi043.stdout: "req_open_latency": { 2024-03-28T22:47:59.594 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.594 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.594 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.594 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.594 INFO:teuthology.orchestra.run.smithi043.stdout: "req_readdir_latency": { 2024-03-28T22:47:59.594 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.594 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.595 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.595 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.595 INFO:teuthology.orchestra.run.smithi043.stdout: "req_rename_latency": { 2024-03-28T22:47:59.595 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.595 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.595 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.595 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.595 INFO:teuthology.orchestra.run.smithi043.stdout: "req_renamesnap_latency": { 2024-03-28T22:47:59.595 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.596 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.596 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.596 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.596 INFO:teuthology.orchestra.run.smithi043.stdout: "req_snapdiff_latency": { 2024-03-28T22:47:59.596 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.596 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.596 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.596 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.596 INFO:teuthology.orchestra.run.smithi043.stdout: "req_rmdir_latency": { 2024-03-28T22:47:59.597 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.597 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.597 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.597 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.597 INFO:teuthology.orchestra.run.smithi043.stdout: "req_rmsnap_latency": { 2024-03-28T22:47:59.597 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.597 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.597 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.598 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.598 INFO:teuthology.orchestra.run.smithi043.stdout: "req_rmxattr_latency": { 2024-03-28T22:47:59.598 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.598 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.598 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.598 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.598 INFO:teuthology.orchestra.run.smithi043.stdout: "req_setattr_latency": { 2024-03-28T22:47:59.599 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.599 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.599 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.599 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.599 INFO:teuthology.orchestra.run.smithi043.stdout: "req_setdirlayout_latency": { 2024-03-28T22:47:59.599 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.599 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.599 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.599 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.600 INFO:teuthology.orchestra.run.smithi043.stdout: "req_setfilelock_latency": { 2024-03-28T22:47:59.600 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.600 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.600 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.600 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.600 INFO:teuthology.orchestra.run.smithi043.stdout: "req_setlayout_latency": { 2024-03-28T22:47:59.600 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.601 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.601 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.601 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.601 INFO:teuthology.orchestra.run.smithi043.stdout: "req_setxattr_latency": { 2024-03-28T22:47:59.601 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.601 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.601 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.601 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.602 INFO:teuthology.orchestra.run.smithi043.stdout: "req_symlink_latency": { 2024-03-28T22:47:59.602 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.602 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.602 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.602 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.602 INFO:teuthology.orchestra.run.smithi043.stdout: "req_unlink_latency": { 2024-03-28T22:47:59.602 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.602 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.603 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.603 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.603 INFO:teuthology.orchestra.run.smithi043.stdout: "cap_revoke_eviction": 0, 2024-03-28T22:47:59.603 INFO:teuthology.orchestra.run.smithi043.stdout: "cap_acquisition_throttle": 0, 2024-03-28T22:47:59.603 INFO:teuthology.orchestra.run.smithi043.stdout: "req_getvxattr_latency": { 2024-03-28T22:47:59.603 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.603 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.604 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.604 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:59.604 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.604 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_sessions": { 2024-03-28T22:47:59.604 INFO:teuthology.orchestra.run.smithi043.stdout: "session_count": 2, 2024-03-28T22:47:59.604 INFO:teuthology.orchestra.run.smithi043.stdout: "session_add": 4, 2024-03-28T22:47:59.604 INFO:teuthology.orchestra.run.smithi043.stdout: "session_remove": 2, 2024-03-28T22:47:59.604 INFO:teuthology.orchestra.run.smithi043.stdout: "sessions_open": 3, 2024-03-28T22:47:59.605 INFO:teuthology.orchestra.run.smithi043.stdout: "sessions_stale": 0, 2024-03-28T22:47:59.605 INFO:teuthology.orchestra.run.smithi043.stdout: "total_load": 0, 2024-03-28T22:47:59.605 INFO:teuthology.orchestra.run.smithi043.stdout: "average_load": 0, 2024-03-28T22:47:59.605 INFO:teuthology.orchestra.run.smithi043.stdout: "avg_session_uptime": 4813, 2024-03-28T22:47:59.605 INFO:teuthology.orchestra.run.smithi043.stdout: "mdthresh_evicted": 0 2024-03-28T22:47:59.605 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.605 INFO:teuthology.orchestra.run.smithi043.stdout: "mempool": { 2024-03-28T22:47:59.606 INFO:teuthology.orchestra.run.smithi043.stdout: "bloom_filter_bytes": 0, 2024-03-28T22:47:59.606 INFO:teuthology.orchestra.run.smithi043.stdout: "bloom_filter_items": 0, 2024-03-28T22:47:59.606 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T22:47:59.606 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_alloc_items": 0, 2024-03-28T22:47:59.606 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T22:47:59.606 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_data_items": 0, 2024-03-28T22:47:59.606 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T22:47:59.607 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T22:47:59.607 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T22:47:59.607 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T22:47:59.607 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T22:47:59.607 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_other_items": 0, 2024-03-28T22:47:59.607 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T22:47:59.607 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T22:47:59.608 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_extent_bytes": 0, 2024-03-28T22:47:59.608 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_extent_items": 0, 2024-03-28T22:47:59.608 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_blob_bytes": 0, 2024-03-28T22:47:59.608 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_blob_items": 0, 2024-03-28T22:47:59.608 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T22:47:59.608 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T22:47:59.609 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T22:47:59.609 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T22:47:59.609 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T22:47:59.609 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_items": 0, 2024-03-28T22:47:59.609 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_txc_bytes": 0, 2024-03-28T22:47:59.609 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_txc_items": 0, 2024-03-28T22:47:59.609 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T22:47:59.609 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T22:47:59.610 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_writing_bytes": 0, 2024-03-28T22:47:59.610 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_writing_items": 0, 2024-03-28T22:47:59.610 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_bytes": 0, 2024-03-28T22:47:59.610 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_items": 0, 2024-03-28T22:47:59.610 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T22:47:59.610 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_file_reader_items": 0, 2024-03-28T22:47:59.610 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T22:47:59.611 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_file_writer_items": 0, 2024-03-28T22:47:59.611 INFO:teuthology.orchestra.run.smithi043.stdout: "buffer_anon_bytes": 183112, 2024-03-28T22:47:59.611 INFO:teuthology.orchestra.run.smithi043.stdout: "buffer_anon_items": 49, 2024-03-28T22:47:59.611 INFO:teuthology.orchestra.run.smithi043.stdout: "buffer_meta_bytes": 0, 2024-03-28T22:47:59.611 INFO:teuthology.orchestra.run.smithi043.stdout: "buffer_meta_items": 0, 2024-03-28T22:47:59.611 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_bytes": 0, 2024-03-28T22:47:59.611 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_items": 0, 2024-03-28T22:47:59.611 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mapbl_bytes": 0, 2024-03-28T22:47:59.612 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mapbl_items": 0, 2024-03-28T22:47:59.612 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pglog_bytes": 0, 2024-03-28T22:47:59.612 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pglog_items": 0, 2024-03-28T22:47:59.612 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_bytes": 9736, 2024-03-28T22:47:59.612 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_items": 203, 2024-03-28T22:47:59.612 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T22:47:59.612 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_mapping_items": 0, 2024-03-28T22:47:59.612 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap_bytes": 0, 2024-03-28T22:47:59.613 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap_items": 0, 2024-03-28T22:47:59.613 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_co_bytes": 8464, 2024-03-28T22:47:59.613 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_co_items": 13, 2024-03-28T22:47:59.613 INFO:teuthology.orchestra.run.smithi043.stdout: "unittest_1_bytes": 0, 2024-03-28T22:47:59.613 INFO:teuthology.orchestra.run.smithi043.stdout: "unittest_1_items": 0, 2024-03-28T22:47:59.613 INFO:teuthology.orchestra.run.smithi043.stdout: "unittest_2_bytes": 0, 2024-03-28T22:47:59.613 INFO:teuthology.orchestra.run.smithi043.stdout: "unittest_2_items": 0 2024-03-28T22:47:59.614 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.614 INFO:teuthology.orchestra.run.smithi043.stdout: "objecter": { 2024-03-28T22:47:59.614 INFO:teuthology.orchestra.run.smithi043.stdout: "op_active": 0, 2024-03-28T22:47:59.614 INFO:teuthology.orchestra.run.smithi043.stdout: "op_laggy": 0, 2024-03-28T22:47:59.614 INFO:teuthology.orchestra.run.smithi043.stdout: "op_send": 2346, 2024-03-28T22:47:59.614 INFO:teuthology.orchestra.run.smithi043.stdout: "op_send_bytes": 21, 2024-03-28T22:47:59.614 INFO:teuthology.orchestra.run.smithi043.stdout: "op_resend": 3, 2024-03-28T22:47:59.614 INFO:teuthology.orchestra.run.smithi043.stdout: "op_reply": 2346, 2024-03-28T22:47:59.615 INFO:teuthology.orchestra.run.smithi043.stdout: "op_latency": { 2024-03-28T22:47:59.615 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 2346, 2024-03-28T22:47:59.615 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 1.779978075, 2024-03-28T22:47:59.615 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000758728 2024-03-28T22:47:59.615 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.615 INFO:teuthology.orchestra.run.smithi043.stdout: "op_inflight": 0, 2024-03-28T22:47:59.615 INFO:teuthology.orchestra.run.smithi043.stdout: "oplen_avg": { 2024-03-28T22:47:59.615 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 2346, 2024-03-28T22:47:59.616 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 2347 2024-03-28T22:47:59.616 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.616 INFO:teuthology.orchestra.run.smithi043.stdout: "op": 2346, 2024-03-28T22:47:59.616 INFO:teuthology.orchestra.run.smithi043.stdout: "op_r": 2346, 2024-03-28T22:47:59.616 INFO:teuthology.orchestra.run.smithi043.stdout: "op_w": 0, 2024-03-28T22:47:59.616 INFO:teuthology.orchestra.run.smithi043.stdout: "op_rmw": 0, 2024-03-28T22:47:59.616 INFO:teuthology.orchestra.run.smithi043.stdout: "op_pg": 0, 2024-03-28T22:47:59.616 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_stat": 1502, 2024-03-28T22:47:59.617 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_create": 0, 2024-03-28T22:47:59.617 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_read": 842, 2024-03-28T22:47:59.617 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_write": 0, 2024-03-28T22:47:59.617 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_writefull": 0, 2024-03-28T22:47:59.617 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_writesame": 0, 2024-03-28T22:47:59.617 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_append": 0, 2024-03-28T22:47:59.617 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_zero": 0, 2024-03-28T22:47:59.617 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_truncate": 0, 2024-03-28T22:47:59.618 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_delete": 0, 2024-03-28T22:47:59.618 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_mapext": 0, 2024-03-28T22:47:59.618 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_sparse_read": 0, 2024-03-28T22:47:59.618 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_clonerange": 0, 2024-03-28T22:47:59.618 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_getxattr": 1, 2024-03-28T22:47:59.618 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_setxattr": 0, 2024-03-28T22:47:59.619 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_cmpxattr": 0, 2024-03-28T22:47:59.619 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_rmxattr": 0, 2024-03-28T22:47:59.619 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_resetxattrs": 0, 2024-03-28T22:47:59.619 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_call": 0, 2024-03-28T22:47:59.619 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_watch": 0, 2024-03-28T22:47:59.619 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_notify": 0, 2024-03-28T22:47:59.619 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T22:47:59.619 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_pgls": 0, 2024-03-28T22:47:59.620 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_pgls_filter": 0, 2024-03-28T22:47:59.620 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_other": 0, 2024-03-28T22:47:59.620 INFO:teuthology.orchestra.run.smithi043.stdout: "linger_active": 0, 2024-03-28T22:47:59.620 INFO:teuthology.orchestra.run.smithi043.stdout: "linger_send": 0, 2024-03-28T22:47:59.620 INFO:teuthology.orchestra.run.smithi043.stdout: "linger_resend": 0, 2024-03-28T22:47:59.620 INFO:teuthology.orchestra.run.smithi043.stdout: "linger_ping": 0, 2024-03-28T22:47:59.620 INFO:teuthology.orchestra.run.smithi043.stdout: "poolop_active": 0, 2024-03-28T22:47:59.621 INFO:teuthology.orchestra.run.smithi043.stdout: "poolop_send": 0, 2024-03-28T22:47:59.621 INFO:teuthology.orchestra.run.smithi043.stdout: "poolop_resend": 0, 2024-03-28T22:47:59.621 INFO:teuthology.orchestra.run.smithi043.stdout: "poolstat_active": 0, 2024-03-28T22:47:59.621 INFO:teuthology.orchestra.run.smithi043.stdout: "poolstat_send": 0, 2024-03-28T22:47:59.621 INFO:teuthology.orchestra.run.smithi043.stdout: "poolstat_resend": 0, 2024-03-28T22:47:59.621 INFO:teuthology.orchestra.run.smithi043.stdout: "statfs_active": 0, 2024-03-28T22:47:59.621 INFO:teuthology.orchestra.run.smithi043.stdout: "statfs_send": 0, 2024-03-28T22:47:59.621 INFO:teuthology.orchestra.run.smithi043.stdout: "statfs_resend": 0, 2024-03-28T22:47:59.621 INFO:teuthology.orchestra.run.smithi043.stdout: "command_active": 0, 2024-03-28T22:47:59.622 INFO:teuthology.orchestra.run.smithi043.stdout: "command_send": 0, 2024-03-28T22:47:59.622 INFO:teuthology.orchestra.run.smithi043.stdout: "command_resend": 0, 2024-03-28T22:47:59.622 INFO:teuthology.orchestra.run.smithi043.stdout: "map_epoch": 108, 2024-03-28T22:47:59.622 INFO:teuthology.orchestra.run.smithi043.stdout: "map_full": 0, 2024-03-28T22:47:59.622 INFO:teuthology.orchestra.run.smithi043.stdout: "map_inc": 16, 2024-03-28T22:47:59.622 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_sessions": 4, 2024-03-28T22:47:59.622 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_session_open": 4, 2024-03-28T22:47:59.623 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_session_close": 0, 2024-03-28T22:47:59.623 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_laggy": 0, 2024-03-28T22:47:59.623 INFO:teuthology.orchestra.run.smithi043.stdout: "omap_wr": 0, 2024-03-28T22:47:59.623 INFO:teuthology.orchestra.run.smithi043.stdout: "omap_rd": 2, 2024-03-28T22:47:59.623 INFO:teuthology.orchestra.run.smithi043.stdout: "omap_del": 0 2024-03-28T22:47:59.623 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.623 INFO:teuthology.orchestra.run.smithi043.stdout: "oft": { 2024-03-28T22:47:59.623 INFO:teuthology.orchestra.run.smithi043.stdout: "omap_total_objs": 0, 2024-03-28T22:47:59.624 INFO:teuthology.orchestra.run.smithi043.stdout: "omap_total_kv_pairs": 0, 2024-03-28T22:47:59.624 INFO:teuthology.orchestra.run.smithi043.stdout: "omap_total_updates": 0, 2024-03-28T22:47:59.624 INFO:teuthology.orchestra.run.smithi043.stdout: "omap_total_removes": 0 2024-03-28T22:47:59.624 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.624 INFO:teuthology.orchestra.run.smithi043.stdout: "osd-slow-ops": { 2024-03-28T22:47:59.624 INFO:teuthology.orchestra.run.smithi043.stdout: "slow_ops_count": 0 2024-03-28T22:47:59.624 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.624 INFO:teuthology.orchestra.run.smithi043.stdout: "purge_queue": { 2024-03-28T22:47:59.624 INFO:teuthology.orchestra.run.smithi043.stdout: "pq_executing_ops": 0, 2024-03-28T22:47:59.625 INFO:teuthology.orchestra.run.smithi043.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T22:47:59.625 INFO:teuthology.orchestra.run.smithi043.stdout: "pq_executing": 0, 2024-03-28T22:47:59.625 INFO:teuthology.orchestra.run.smithi043.stdout: "pq_executing_high_water": 0, 2024-03-28T22:47:59.625 INFO:teuthology.orchestra.run.smithi043.stdout: "pq_executed_ops": 0, 2024-03-28T22:47:59.625 INFO:teuthology.orchestra.run.smithi043.stdout: "pq_executed": 0, 2024-03-28T22:47:59.625 INFO:teuthology.orchestra.run.smithi043.stdout: "pq_item_in_journal": 0 2024-03-28T22:47:59.625 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.625 INFO:teuthology.orchestra.run.smithi043.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T22:47:59.626 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 0, 2024-03-28T22:47:59.626 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 104857600, 2024-03-28T22:47:59.626 INFO:teuthology.orchestra.run.smithi043.stdout: "get_started": 0, 2024-03-28T22:47:59.626 INFO:teuthology.orchestra.run.smithi043.stdout: "get": 2627, 2024-03-28T22:47:59.626 INFO:teuthology.orchestra.run.smithi043.stdout: "get_sum": 2323730, 2024-03-28T22:47:59.626 INFO:teuthology.orchestra.run.smithi043.stdout: "get_or_fail_fail": 0, 2024-03-28T22:47:59.626 INFO:teuthology.orchestra.run.smithi043.stdout: "get_or_fail_success": 2627, 2024-03-28T22:47:59.626 INFO:teuthology.orchestra.run.smithi043.stdout: "take": 0, 2024-03-28T22:47:59.627 INFO:teuthology.orchestra.run.smithi043.stdout: "take_sum": 0, 2024-03-28T22:47:59.627 INFO:teuthology.orchestra.run.smithi043.stdout: "put": 2627, 2024-03-28T22:47:59.627 INFO:teuthology.orchestra.run.smithi043.stdout: "put_sum": 2323730, 2024-03-28T22:47:59.627 INFO:teuthology.orchestra.run.smithi043.stdout: "wait": { 2024-03-28T22:47:59.627 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.627 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.627 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.627 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:59.628 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.628 INFO:teuthology.orchestra.run.smithi043.stdout: "throttle-objecter_bytes": { 2024-03-28T22:47:59.628 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 0, 2024-03-28T22:47:59.628 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 104857600, 2024-03-28T22:47:59.628 INFO:teuthology.orchestra.run.smithi043.stdout: "get_started": 0, 2024-03-28T22:47:59.628 INFO:teuthology.orchestra.run.smithi043.stdout: "get": 0, 2024-03-28T22:47:59.628 INFO:teuthology.orchestra.run.smithi043.stdout: "get_sum": 0, 2024-03-28T22:47:59.628 INFO:teuthology.orchestra.run.smithi043.stdout: "get_or_fail_fail": 0, 2024-03-28T22:47:59.629 INFO:teuthology.orchestra.run.smithi043.stdout: "get_or_fail_success": 0, 2024-03-28T22:47:59.629 INFO:teuthology.orchestra.run.smithi043.stdout: "take": 2346, 2024-03-28T22:47:59.629 INFO:teuthology.orchestra.run.smithi043.stdout: "take_sum": 2354, 2024-03-28T22:47:59.629 INFO:teuthology.orchestra.run.smithi043.stdout: "put": 2346, 2024-03-28T22:47:59.629 INFO:teuthology.orchestra.run.smithi043.stdout: "put_sum": 2354, 2024-03-28T22:47:59.629 INFO:teuthology.orchestra.run.smithi043.stdout: "wait": { 2024-03-28T22:47:59.629 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.629 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.630 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.630 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:59.630 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.630 INFO:teuthology.orchestra.run.smithi043.stdout: "throttle-objecter_ops": { 2024-03-28T22:47:59.630 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 0, 2024-03-28T22:47:59.630 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1024, 2024-03-28T22:47:59.630 INFO:teuthology.orchestra.run.smithi043.stdout: "get_started": 0, 2024-03-28T22:47:59.630 INFO:teuthology.orchestra.run.smithi043.stdout: "get": 0, 2024-03-28T22:47:59.631 INFO:teuthology.orchestra.run.smithi043.stdout: "get_sum": 0, 2024-03-28T22:47:59.631 INFO:teuthology.orchestra.run.smithi043.stdout: "get_or_fail_fail": 0, 2024-03-28T22:47:59.631 INFO:teuthology.orchestra.run.smithi043.stdout: "get_or_fail_success": 0, 2024-03-28T22:47:59.631 INFO:teuthology.orchestra.run.smithi043.stdout: "take": 2346, 2024-03-28T22:47:59.631 INFO:teuthology.orchestra.run.smithi043.stdout: "take_sum": 2346, 2024-03-28T22:47:59.631 INFO:teuthology.orchestra.run.smithi043.stdout: "put": 2346, 2024-03-28T22:47:59.631 INFO:teuthology.orchestra.run.smithi043.stdout: "put_sum": 2346, 2024-03-28T22:47:59.631 INFO:teuthology.orchestra.run.smithi043.stdout: "wait": { 2024-03-28T22:47:59.631 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.632 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.632 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.632 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:59.632 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.632 INFO:teuthology.orchestra.run.smithi043.stdout: "throttle-write_buf_throttle": { 2024-03-28T22:47:59.632 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 0, 2024-03-28T22:47:59.632 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 3758096384, 2024-03-28T22:47:59.632 INFO:teuthology.orchestra.run.smithi043.stdout: "get_started": 0, 2024-03-28T22:47:59.633 INFO:teuthology.orchestra.run.smithi043.stdout: "get": 0, 2024-03-28T22:47:59.633 INFO:teuthology.orchestra.run.smithi043.stdout: "get_sum": 0, 2024-03-28T22:47:59.633 INFO:teuthology.orchestra.run.smithi043.stdout: "get_or_fail_fail": 0, 2024-03-28T22:47:59.633 INFO:teuthology.orchestra.run.smithi043.stdout: "get_or_fail_success": 0, 2024-03-28T22:47:59.633 INFO:teuthology.orchestra.run.smithi043.stdout: "take": 0, 2024-03-28T22:47:59.633 INFO:teuthology.orchestra.run.smithi043.stdout: "take_sum": 0, 2024-03-28T22:47:59.633 INFO:teuthology.orchestra.run.smithi043.stdout: "put": 0, 2024-03-28T22:47:59.633 INFO:teuthology.orchestra.run.smithi043.stdout: "put_sum": 0, 2024-03-28T22:47:59.633 INFO:teuthology.orchestra.run.smithi043.stdout: "wait": { 2024-03-28T22:47:59.634 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.634 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.634 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.634 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:59.634 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:47:59.634 INFO:teuthology.orchestra.run.smithi043.stdout: "throttle-write_buf_throttle-0x555674804780": { 2024-03-28T22:47:59.634 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 0, 2024-03-28T22:47:59.634 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 3758096384, 2024-03-28T22:47:59.635 INFO:teuthology.orchestra.run.smithi043.stdout: "get_started": 0, 2024-03-28T22:47:59.635 INFO:teuthology.orchestra.run.smithi043.stdout: "get": 0, 2024-03-28T22:47:59.635 INFO:teuthology.orchestra.run.smithi043.stdout: "get_sum": 0, 2024-03-28T22:47:59.635 INFO:teuthology.orchestra.run.smithi043.stdout: "get_or_fail_fail": 0, 2024-03-28T22:47:59.635 INFO:teuthology.orchestra.run.smithi043.stdout: "get_or_fail_success": 0, 2024-03-28T22:47:59.635 INFO:teuthology.orchestra.run.smithi043.stdout: "take": 0, 2024-03-28T22:47:59.635 INFO:teuthology.orchestra.run.smithi043.stdout: "take_sum": 0, 2024-03-28T22:47:59.635 INFO:teuthology.orchestra.run.smithi043.stdout: "put": 0, 2024-03-28T22:47:59.635 INFO:teuthology.orchestra.run.smithi043.stdout: "put_sum": 0, 2024-03-28T22:47:59.636 INFO:teuthology.orchestra.run.smithi043.stdout: "wait": { 2024-03-28T22:47:59.636 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:47:59.636 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:47:59.636 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:47:59.636 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:59.636 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:47:59.636 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:48:00.524 INFO:tasks.check_counter:Daemon mds.d mds.exported=0 2024-03-28T22:48:00.524 INFO:tasks.check_counter:Daemon mds.d mds.imported=0 2024-03-28T22:48:00.524 INFO:tasks.check_counter:Daemon mds.d mds_cache.dir_update=0 2024-03-28T22:48:00.525 INFO:tasks.check_counter:Daemon mds.d mds_cache.dir_update_receipt=0 2024-03-28T22:48:00.525 INFO:tasks.check_counter:Daemon mds.d mds.root_rsnaps=0 2024-03-28T22:48:00.525 INFO:tasks.check_counter:Daemon mds.d mds_server.req_mksnap_latency.avgcount=0 2024-03-28T22:48:00.525 INFO:tasks.check_counter:Daemon mds.d mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T22:48:00.525 DEBUG:tasks.check_counter:Getting stats from g 2024-03-28T22:48:00.525 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-28T22:48:01.027 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:48:01.248 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:00 smithi043 ceph-mon[31823]: pgmap v338: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:48:01.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:00 smithi172 ceph-mon[36566]: pgmap v338: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:48:01.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:00 smithi082 ceph-mon[36471]: pgmap v338: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:48:03.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:02 smithi043 ceph-mon[31823]: pgmap v339: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:48:03.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:02 smithi172 ceph-mon[36566]: pgmap v339: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:48:03.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:02 smithi082 ceph-mon[36471]: pgmap v339: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:48:03.755 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:48:03.756 INFO:teuthology.orchestra.run.smithi043.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T22:48:03.756 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_messages": 0, 2024-03-28T22:48:03.756 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_messages": 2, 2024-03-28T22:48:03.756 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_bytes": 0, 2024-03-28T22:48:03.756 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_bytes": 0, 2024-03-28T22:48:03.756 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_created_connections": 4, 2024-03-28T22:48:03.756 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_active_connections": 0, 2024-03-28T22:48:03.757 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_total_time": 0.003769863, 2024-03-28T22:48:03.757 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_send_time": 0.000000000, 2024-03-28T22:48:03.757 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_recv_time": 0.000631104, 2024-03-28T22:48:03.757 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T22:48:03.757 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:48:03.757 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:03.757 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:03.757 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:03.757 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:03.758 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:48:03.758 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:03.758 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:03.758 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:03.758 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:03.758 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T22:48:03.758 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T22:48:03.758 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:03.759 INFO:teuthology.orchestra.run.smithi043.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T22:48:03.759 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_messages": 249, 2024-03-28T22:48:03.759 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_messages": 236, 2024-03-28T22:48:03.759 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_bytes": 1849808, 2024-03-28T22:48:03.759 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_bytes": 145824, 2024-03-28T22:48:03.759 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_created_connections": 2, 2024-03-28T22:48:03.759 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_active_connections": 1, 2024-03-28T22:48:03.759 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_total_time": 0.108682111, 2024-03-28T22:48:03.760 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_send_time": 0.038611057, 2024-03-28T22:48:03.760 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_recv_time": 0.079459153, 2024-03-28T22:48:03.760 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_fast_dispatch_time": 0.011481705, 2024-03-28T22:48:03.760 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:48:03.760 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 235, 2024-03-28T22:48:03.760 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.024288622, 2024-03-28T22:48:03.760 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000103355 2024-03-28T22:48:03.760 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:03.761 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:48:03.761 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:03.761 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:03.761 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:03.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:03.761 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_encrypted_bytes": 1849808, 2024-03-28T22:48:03.761 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_encrypted_bytes": 145824 2024-03-28T22:48:03.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:03.761 INFO:teuthology.orchestra.run.smithi043.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T22:48:03.762 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_messages": 6, 2024-03-28T22:48:03.762 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_messages": 184, 2024-03-28T22:48:03.762 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_bytes": 3392, 2024-03-28T22:48:03.762 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_bytes": 188048, 2024-03-28T22:48:03.762 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_created_connections": 5, 2024-03-28T22:48:03.762 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_active_connections": 3, 2024-03-28T22:48:03.762 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_total_time": 0.038195260, 2024-03-28T22:48:03.762 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_send_time": 0.027369364, 2024-03-28T22:48:03.762 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_recv_time": 0.004815350, 2024-03-28T22:48:03.763 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T22:48:03.763 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:48:03.763 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 182, 2024-03-28T22:48:03.763 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.029547689, 2024-03-28T22:48:03.763 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000162349 2024-03-28T22:48:03.763 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:03.763 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:48:03.763 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:03.763 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:03.764 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:03.764 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:03.764 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-03-28T22:48:03.764 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_encrypted_bytes": 188048 2024-03-28T22:48:03.764 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:03.764 INFO:teuthology.orchestra.run.smithi043.stdout: "cct": { 2024-03-28T22:48:03.764 INFO:teuthology.orchestra.run.smithi043.stdout: "total_workers": 0, 2024-03-28T22:48:03.764 INFO:teuthology.orchestra.run.smithi043.stdout: "unhealthy_workers": 0 2024-03-28T22:48:03.764 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:03.765 INFO:teuthology.orchestra.run.smithi043.stdout: "mempool": { 2024-03-28T22:48:03.765 INFO:teuthology.orchestra.run.smithi043.stdout: "bloom_filter_bytes": 0, 2024-03-28T22:48:03.765 INFO:teuthology.orchestra.run.smithi043.stdout: "bloom_filter_items": 0, 2024-03-28T22:48:03.765 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T22:48:03.765 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_alloc_items": 0, 2024-03-28T22:48:03.765 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T22:48:03.765 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_data_items": 0, 2024-03-28T22:48:03.765 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T22:48:03.765 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T22:48:03.766 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T22:48:03.766 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T22:48:03.766 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T22:48:03.766 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_other_items": 0, 2024-03-28T22:48:03.766 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T22:48:03.766 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T22:48:03.766 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_extent_bytes": 0, 2024-03-28T22:48:03.767 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_extent_items": 0, 2024-03-28T22:48:03.767 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_blob_bytes": 0, 2024-03-28T22:48:03.767 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_blob_items": 0, 2024-03-28T22:48:03.767 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T22:48:03.767 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T22:48:03.767 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T22:48:03.767 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T22:48:03.767 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T22:48:03.768 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_items": 0, 2024-03-28T22:48:03.768 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_txc_bytes": 0, 2024-03-28T22:48:03.768 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_txc_items": 0, 2024-03-28T22:48:03.768 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T22:48:03.768 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T22:48:03.768 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_writing_bytes": 0, 2024-03-28T22:48:03.768 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_writing_items": 0, 2024-03-28T22:48:03.768 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_bytes": 0, 2024-03-28T22:48:03.768 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_items": 0, 2024-03-28T22:48:03.768 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T22:48:03.769 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_file_reader_items": 0, 2024-03-28T22:48:03.769 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T22:48:03.769 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_file_writer_items": 0, 2024-03-28T22:48:03.769 INFO:teuthology.orchestra.run.smithi043.stdout: "buffer_anon_bytes": 148714, 2024-03-28T22:48:03.769 INFO:teuthology.orchestra.run.smithi043.stdout: "buffer_anon_items": 30, 2024-03-28T22:48:03.769 INFO:teuthology.orchestra.run.smithi043.stdout: "buffer_meta_bytes": 0, 2024-03-28T22:48:03.769 INFO:teuthology.orchestra.run.smithi043.stdout: "buffer_meta_items": 0, 2024-03-28T22:48:03.769 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_bytes": 0, 2024-03-28T22:48:03.769 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_items": 0, 2024-03-28T22:48:03.770 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mapbl_bytes": 0, 2024-03-28T22:48:03.770 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mapbl_items": 0, 2024-03-28T22:48:03.770 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pglog_bytes": 0, 2024-03-28T22:48:03.770 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pglog_items": 0, 2024-03-28T22:48:03.770 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_bytes": 0, 2024-03-28T22:48:03.770 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_items": 0, 2024-03-28T22:48:03.770 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T22:48:03.771 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_mapping_items": 0, 2024-03-28T22:48:03.771 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap_bytes": 0, 2024-03-28T22:48:03.771 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap_items": 0, 2024-03-28T22:48:03.771 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_co_bytes": 928, 2024-03-28T22:48:03.771 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_co_items": 2, 2024-03-28T22:48:03.771 INFO:teuthology.orchestra.run.smithi043.stdout: "unittest_1_bytes": 0, 2024-03-28T22:48:03.771 INFO:teuthology.orchestra.run.smithi043.stdout: "unittest_1_items": 0, 2024-03-28T22:48:03.772 INFO:teuthology.orchestra.run.smithi043.stdout: "unittest_2_bytes": 0, 2024-03-28T22:48:03.772 INFO:teuthology.orchestra.run.smithi043.stdout: "unittest_2_items": 0 2024-03-28T22:48:03.772 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:03.772 INFO:teuthology.orchestra.run.smithi043.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T22:48:03.772 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 0, 2024-03-28T22:48:03.772 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 104857600, 2024-03-28T22:48:03.772 INFO:teuthology.orchestra.run.smithi043.stdout: "get_started": 0, 2024-03-28T22:48:03.773 INFO:teuthology.orchestra.run.smithi043.stdout: "get": 251, 2024-03-28T22:48:03.773 INFO:teuthology.orchestra.run.smithi043.stdout: "get_sum": 1814682, 2024-03-28T22:48:03.773 INFO:teuthology.orchestra.run.smithi043.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:03.773 INFO:teuthology.orchestra.run.smithi043.stdout: "get_or_fail_success": 251, 2024-03-28T22:48:03.773 INFO:teuthology.orchestra.run.smithi043.stdout: "take": 0, 2024-03-28T22:48:03.773 INFO:teuthology.orchestra.run.smithi043.stdout: "take_sum": 0, 2024-03-28T22:48:03.773 INFO:teuthology.orchestra.run.smithi043.stdout: "put": 251, 2024-03-28T22:48:03.773 INFO:teuthology.orchestra.run.smithi043.stdout: "put_sum": 1814682, 2024-03-28T22:48:03.774 INFO:teuthology.orchestra.run.smithi043.stdout: "wait": { 2024-03-28T22:48:03.774 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:03.774 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:03.774 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:03.774 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:48:03.774 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:48:03.774 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:48:04.601 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-03-28T22:48:04.601 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-03-28T22:48:04.601 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2024-03-28T22:48:04.601 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2024-03-28T22:48:04.601 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-03-28T22:48:04.601 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-03-28T22:48:04.602 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-03-28T22:48:04.602 DEBUG:tasks.check_counter:Getting stats from j 2024-03-28T22:48:04.602 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-28T22:48:05.067 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T22:48:05.241 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:04 smithi043 ceph-mon[31823]: pgmap v340: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-03-28T22:48:05.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:04 smithi172 ceph-mon[36566]: pgmap v340: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-03-28T22:48:05.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:04 smithi082 ceph-mon[36471]: pgmap v340: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-03-28T22:48:07.240 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:06 smithi043 ceph-mon[31823]: pgmap v341: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-03-28T22:48:07.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:06 smithi172 ceph-mon[36566]: pgmap v341: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-03-28T22:48:07.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:06 smithi082 ceph-mon[36471]: pgmap v341: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 25 op/s 2024-03-28T22:48:07.627 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-03-28T22:48:07.627 INFO:teuthology.orchestra.run.smithi043.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T22:48:07.628 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_messages": 853, 2024-03-28T22:48:07.628 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_messages": 853, 2024-03-28T22:48:07.628 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_bytes": 228402, 2024-03-28T22:48:07.628 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_bytes": 270494, 2024-03-28T22:48:07.628 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_created_connections": 6, 2024-03-28T22:48:07.628 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_active_connections": 2, 2024-03-28T22:48:07.628 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_total_time": 0.172798827, 2024-03-28T22:48:07.628 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_send_time": 0.060578705, 2024-03-28T22:48:07.628 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_recv_time": 0.045457435, 2024-03-28T22:48:07.629 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_fast_dispatch_time": 0.046479636, 2024-03-28T22:48:07.629 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:48:07.629 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 852, 2024-03-28T22:48:07.629 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.073894297, 2024-03-28T22:48:07.629 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000086730 2024-03-28T22:48:07.629 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.629 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:48:07.629 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.630 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.630 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.630 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.630 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-28T22:48:07.630 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_encrypted_bytes": 448 2024-03-28T22:48:07.630 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.630 INFO:teuthology.orchestra.run.smithi043.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T22:48:07.630 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_messages": 1042, 2024-03-28T22:48:07.630 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_messages": 1025, 2024-03-28T22:48:07.631 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_bytes": 2093494, 2024-03-28T22:48:07.631 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_bytes": 395973, 2024-03-28T22:48:07.631 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_created_connections": 4, 2024-03-28T22:48:07.631 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_active_connections": 3, 2024-03-28T22:48:07.631 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_total_time": 0.295206880, 2024-03-28T22:48:07.631 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_send_time": 0.104297149, 2024-03-28T22:48:07.631 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_recv_time": 0.155854241, 2024-03-28T22:48:07.632 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_fast_dispatch_time": 0.059368786, 2024-03-28T22:48:07.632 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:48:07.632 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 1024, 2024-03-28T22:48:07.632 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.089734072, 2024-03-28T22:48:07.632 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000087630 2024-03-28T22:48:07.632 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.632 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:48:07.632 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.633 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.633 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.633 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.633 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_encrypted_bytes": 1906512, 2024-03-28T22:48:07.633 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_encrypted_bytes": 154288 2024-03-28T22:48:07.633 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.633 INFO:teuthology.orchestra.run.smithi043.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T22:48:07.633 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_messages": 842, 2024-03-28T22:48:07.633 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_messages": 1027, 2024-03-28T22:48:07.634 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_bytes": 282560, 2024-03-28T22:48:07.634 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_bytes": 744032, 2024-03-28T22:48:07.634 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_created_connections": 6, 2024-03-28T22:48:07.634 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_active_connections": 4, 2024-03-28T22:48:07.634 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_total_time": 0.249148904, 2024-03-28T22:48:07.634 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_send_time": 0.123927099, 2024-03-28T22:48:07.634 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_recv_time": 0.050564059, 2024-03-28T22:48:07.634 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_running_fast_dispatch_time": 0.049313431, 2024-03-28T22:48:07.635 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:48:07.635 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 1024, 2024-03-28T22:48:07.635 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.120814101, 2024-03-28T22:48:07.635 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000117982 2024-03-28T22:48:07.635 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.635 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:48:07.635 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.635 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.636 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.636 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.636 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T22:48:07.636 INFO:teuthology.orchestra.run.smithi043.stdout: "msgr_send_encrypted_bytes": 476912 2024-03-28T22:48:07.636 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.636 INFO:teuthology.orchestra.run.smithi043.stdout: "cct": { 2024-03-28T22:48:07.636 INFO:teuthology.orchestra.run.smithi043.stdout: "total_workers": 1, 2024-03-28T22:48:07.636 INFO:teuthology.orchestra.run.smithi043.stdout: "unhealthy_workers": 0 2024-03-28T22:48:07.636 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.637 INFO:teuthology.orchestra.run.smithi043.stdout: "finisher-MDSRank": { 2024-03-28T22:48:07.637 INFO:teuthology.orchestra.run.smithi043.stdout: "queue_len": 0, 2024-03-28T22:48:07.637 INFO:teuthology.orchestra.run.smithi043.stdout: "complete_latency": { 2024-03-28T22:48:07.637 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 4105, 2024-03-28T22:48:07.637 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.215985240, 2024-03-28T22:48:07.637 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000052615 2024-03-28T22:48:07.637 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:48:07.637 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.637 INFO:teuthology.orchestra.run.smithi043.stdout: "finisher-PurgeQueue": { 2024-03-28T22:48:07.638 INFO:teuthology.orchestra.run.smithi043.stdout: "queue_len": 0, 2024-03-28T22:48:07.638 INFO:teuthology.orchestra.run.smithi043.stdout: "complete_latency": { 2024-03-28T22:48:07.638 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.638 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.638 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.638 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:48:07.638 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.638 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": { 2024-03-28T22:48:07.639 INFO:teuthology.orchestra.run.smithi043.stdout: "request": 0, 2024-03-28T22:48:07.639 INFO:teuthology.orchestra.run.smithi043.stdout: "reply": 0, 2024-03-28T22:48:07.639 INFO:teuthology.orchestra.run.smithi043.stdout: "reply_latency": { 2024-03-28T22:48:07.639 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.639 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.639 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.639 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.639 INFO:teuthology.orchestra.run.smithi043.stdout: "slow_reply": 0, 2024-03-28T22:48:07.640 INFO:teuthology.orchestra.run.smithi043.stdout: "forward": 0, 2024-03-28T22:48:07.640 INFO:teuthology.orchestra.run.smithi043.stdout: "dir_fetch_complete": 0, 2024-03-28T22:48:07.640 INFO:teuthology.orchestra.run.smithi043.stdout: "dir_fetch_keys": 0, 2024-03-28T22:48:07.640 INFO:teuthology.orchestra.run.smithi043.stdout: "dir_commit": 0, 2024-03-28T22:48:07.640 INFO:teuthology.orchestra.run.smithi043.stdout: "dir_split": 0, 2024-03-28T22:48:07.640 INFO:teuthology.orchestra.run.smithi043.stdout: "dir_merge": 0, 2024-03-28T22:48:07.640 INFO:teuthology.orchestra.run.smithi043.stdout: "inodes": 0, 2024-03-28T22:48:07.640 INFO:teuthology.orchestra.run.smithi043.stdout: "inodes_top": 0, 2024-03-28T22:48:07.640 INFO:teuthology.orchestra.run.smithi043.stdout: "inodes_bottom": 0, 2024-03-28T22:48:07.641 INFO:teuthology.orchestra.run.smithi043.stdout: "inodes_pin_tail": 0, 2024-03-28T22:48:07.641 INFO:teuthology.orchestra.run.smithi043.stdout: "inodes_pinned": 0, 2024-03-28T22:48:07.641 INFO:teuthology.orchestra.run.smithi043.stdout: "inodes_expired": 0, 2024-03-28T22:48:07.641 INFO:teuthology.orchestra.run.smithi043.stdout: "inodes_with_caps": 0, 2024-03-28T22:48:07.641 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": 0, 2024-03-28T22:48:07.641 INFO:teuthology.orchestra.run.smithi043.stdout: "subtrees": 1, 2024-03-28T22:48:07.641 INFO:teuthology.orchestra.run.smithi043.stdout: "traverse": 0, 2024-03-28T22:48:07.641 INFO:teuthology.orchestra.run.smithi043.stdout: "traverse_hit": 0, 2024-03-28T22:48:07.642 INFO:teuthology.orchestra.run.smithi043.stdout: "traverse_forward": 0, 2024-03-28T22:48:07.642 INFO:teuthology.orchestra.run.smithi043.stdout: "traverse_discover": 0, 2024-03-28T22:48:07.642 INFO:teuthology.orchestra.run.smithi043.stdout: "traverse_dir_fetch": 0, 2024-03-28T22:48:07.642 INFO:teuthology.orchestra.run.smithi043.stdout: "traverse_remote_ino": 0, 2024-03-28T22:48:07.642 INFO:teuthology.orchestra.run.smithi043.stdout: "traverse_lock": 0, 2024-03-28T22:48:07.642 INFO:teuthology.orchestra.run.smithi043.stdout: "load_cent": 0, 2024-03-28T22:48:07.642 INFO:teuthology.orchestra.run.smithi043.stdout: "q": 0, 2024-03-28T22:48:07.642 INFO:teuthology.orchestra.run.smithi043.stdout: "exported": 0, 2024-03-28T22:48:07.643 INFO:teuthology.orchestra.run.smithi043.stdout: "exported_inodes": 0, 2024-03-28T22:48:07.643 INFO:teuthology.orchestra.run.smithi043.stdout: "imported": 0, 2024-03-28T22:48:07.643 INFO:teuthology.orchestra.run.smithi043.stdout: "imported_inodes": 0, 2024-03-28T22:48:07.643 INFO:teuthology.orchestra.run.smithi043.stdout: "openino_dir_fetch": 0, 2024-03-28T22:48:07.643 INFO:teuthology.orchestra.run.smithi043.stdout: "openino_backtrace_fetch": 0, 2024-03-28T22:48:07.643 INFO:teuthology.orchestra.run.smithi043.stdout: "openino_peer_discover": 0, 2024-03-28T22:48:07.643 INFO:teuthology.orchestra.run.smithi043.stdout: "root_rfiles": 0, 2024-03-28T22:48:07.643 INFO:teuthology.orchestra.run.smithi043.stdout: "root_rbytes": 0, 2024-03-28T22:48:07.644 INFO:teuthology.orchestra.run.smithi043.stdout: "root_rsnaps": 0, 2024-03-28T22:48:07.644 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T22:48:07.644 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_set_tag": 0, 2024-03-28T22:48:07.644 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T22:48:07.644 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_inotable_repaired": 0, 2024-03-28T22:48:07.644 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_dir_inodes": 0, 2024-03-28T22:48:07.644 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T22:48:07.644 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T22:48:07.644 INFO:teuthology.orchestra.run.smithi043.stdout: "scrub_file_inodes": 0, 2024-03-28T22:48:07.645 INFO:teuthology.orchestra.run.smithi043.stdout: "handle_inode_file_caps": 0, 2024-03-28T22:48:07.645 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T22:48:07.645 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_cap_op_grant": 0, 2024-03-28T22:48:07.645 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T22:48:07.645 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T22:48:07.645 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T22:48:07.645 INFO:teuthology.orchestra.run.smithi043.stdout: "handle_client_caps": 0, 2024-03-28T22:48:07.645 INFO:teuthology.orchestra.run.smithi043.stdout: "handle_client_caps_dirty": 0, 2024-03-28T22:48:07.645 INFO:teuthology.orchestra.run.smithi043.stdout: "handle_client_cap_release": 0, 2024-03-28T22:48:07.646 INFO:teuthology.orchestra.run.smithi043.stdout: "process_request_cap_release": 0 2024-03-28T22:48:07.646 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.646 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_cache": { 2024-03-28T22:48:07.646 INFO:teuthology.orchestra.run.smithi043.stdout: "dir_update": 0, 2024-03-28T22:48:07.646 INFO:teuthology.orchestra.run.smithi043.stdout: "dir_update_receipt": 0, 2024-03-28T22:48:07.646 INFO:teuthology.orchestra.run.smithi043.stdout: "dir_try_discover": 0, 2024-03-28T22:48:07.646 INFO:teuthology.orchestra.run.smithi043.stdout: "dir_send_discover": 0, 2024-03-28T22:48:07.646 INFO:teuthology.orchestra.run.smithi043.stdout: "dir_handle_discover": 0, 2024-03-28T22:48:07.647 INFO:teuthology.orchestra.run.smithi043.stdout: "num_strays": 0, 2024-03-28T22:48:07.647 INFO:teuthology.orchestra.run.smithi043.stdout: "num_strays_delayed": 0, 2024-03-28T22:48:07.647 INFO:teuthology.orchestra.run.smithi043.stdout: "num_strays_enqueuing": 0, 2024-03-28T22:48:07.647 INFO:teuthology.orchestra.run.smithi043.stdout: "strays_created": 0, 2024-03-28T22:48:07.647 INFO:teuthology.orchestra.run.smithi043.stdout: "strays_enqueued": 0, 2024-03-28T22:48:07.647 INFO:teuthology.orchestra.run.smithi043.stdout: "strays_reintegrated": 0, 2024-03-28T22:48:07.647 INFO:teuthology.orchestra.run.smithi043.stdout: "strays_migrated": 0, 2024-03-28T22:48:07.647 INFO:teuthology.orchestra.run.smithi043.stdout: "num_recovering_processing": 0, 2024-03-28T22:48:07.647 INFO:teuthology.orchestra.run.smithi043.stdout: "num_recovering_enqueued": 0, 2024-03-28T22:48:07.648 INFO:teuthology.orchestra.run.smithi043.stdout: "num_recovering_prioritized": 0, 2024-03-28T22:48:07.648 INFO:teuthology.orchestra.run.smithi043.stdout: "recovery_started": 0, 2024-03-28T22:48:07.648 INFO:teuthology.orchestra.run.smithi043.stdout: "recovery_completed": 0, 2024-03-28T22:48:07.648 INFO:teuthology.orchestra.run.smithi043.stdout: "ireq_quiesce_path": 0, 2024-03-28T22:48:07.648 INFO:teuthology.orchestra.run.smithi043.stdout: "ireq_quiesce_inode": 0, 2024-03-28T22:48:07.648 INFO:teuthology.orchestra.run.smithi043.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T22:48:07.648 INFO:teuthology.orchestra.run.smithi043.stdout: "ireq_exportdir": 0, 2024-03-28T22:48:07.648 INFO:teuthology.orchestra.run.smithi043.stdout: "ireq_flush": 0, 2024-03-28T22:48:07.649 INFO:teuthology.orchestra.run.smithi043.stdout: "ireq_fragmentdir": 0, 2024-03-28T22:48:07.649 INFO:teuthology.orchestra.run.smithi043.stdout: "ireq_fragstats": 0, 2024-03-28T22:48:07.649 INFO:teuthology.orchestra.run.smithi043.stdout: "ireq_inodestats": 0 2024-03-28T22:48:07.649 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.649 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_log": { 2024-03-28T22:48:07.649 INFO:teuthology.orchestra.run.smithi043.stdout: "evlrg": 0, 2024-03-28T22:48:07.650 INFO:teuthology.orchestra.run.smithi043.stdout: "evadd": 0, 2024-03-28T22:48:07.650 INFO:teuthology.orchestra.run.smithi043.stdout: "evex": 0, 2024-03-28T22:48:07.650 INFO:teuthology.orchestra.run.smithi043.stdout: "evtrm": 0, 2024-03-28T22:48:07.650 INFO:teuthology.orchestra.run.smithi043.stdout: "ev": 8, 2024-03-28T22:48:07.650 INFO:teuthology.orchestra.run.smithi043.stdout: "evexg": 0, 2024-03-28T22:48:07.650 INFO:teuthology.orchestra.run.smithi043.stdout: "evexd": 0, 2024-03-28T22:48:07.650 INFO:teuthology.orchestra.run.smithi043.stdout: "segadd": 0, 2024-03-28T22:48:07.651 INFO:teuthology.orchestra.run.smithi043.stdout: "segex": 0, 2024-03-28T22:48:07.651 INFO:teuthology.orchestra.run.smithi043.stdout: "segtrm": 0, 2024-03-28T22:48:07.651 INFO:teuthology.orchestra.run.smithi043.stdout: "seg": 2, 2024-03-28T22:48:07.651 INFO:teuthology.orchestra.run.smithi043.stdout: "segmjr": 2, 2024-03-28T22:48:07.651 INFO:teuthology.orchestra.run.smithi043.stdout: "segexg": 0, 2024-03-28T22:48:07.651 INFO:teuthology.orchestra.run.smithi043.stdout: "segexd": 0, 2024-03-28T22:48:07.651 INFO:teuthology.orchestra.run.smithi043.stdout: "expos": 4194304, 2024-03-28T22:48:07.651 INFO:teuthology.orchestra.run.smithi043.stdout: "wrpos": 4196653, 2024-03-28T22:48:07.651 INFO:teuthology.orchestra.run.smithi043.stdout: "rdpos": 4196512, 2024-03-28T22:48:07.652 INFO:teuthology.orchestra.run.smithi043.stdout: "jlat": { 2024-03-28T22:48:07.652 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.652 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.652 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.652 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.652 INFO:teuthology.orchestra.run.smithi043.stdout: "replayed": 8 2024-03-28T22:48:07.652 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.652 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_mem": { 2024-03-28T22:48:07.652 INFO:teuthology.orchestra.run.smithi043.stdout: "ino": 3, 2024-03-28T22:48:07.653 INFO:teuthology.orchestra.run.smithi043.stdout: "ino+": 3, 2024-03-28T22:48:07.653 INFO:teuthology.orchestra.run.smithi043.stdout: "ino-": 0, 2024-03-28T22:48:07.653 INFO:teuthology.orchestra.run.smithi043.stdout: "dir": 1, 2024-03-28T22:48:07.653 INFO:teuthology.orchestra.run.smithi043.stdout: "dir+": 1, 2024-03-28T22:48:07.653 INFO:teuthology.orchestra.run.smithi043.stdout: "dir-": 0, 2024-03-28T22:48:07.653 INFO:teuthology.orchestra.run.smithi043.stdout: "dn": 0, 2024-03-28T22:48:07.653 INFO:teuthology.orchestra.run.smithi043.stdout: "dn+": 0, 2024-03-28T22:48:07.653 INFO:teuthology.orchestra.run.smithi043.stdout: "dn-": 0, 2024-03-28T22:48:07.653 INFO:teuthology.orchestra.run.smithi043.stdout: "cap": 0, 2024-03-28T22:48:07.654 INFO:teuthology.orchestra.run.smithi043.stdout: "cap+": 0, 2024-03-28T22:48:07.654 INFO:teuthology.orchestra.run.smithi043.stdout: "cap-": 0, 2024-03-28T22:48:07.654 INFO:teuthology.orchestra.run.smithi043.stdout: "rss": 49408, 2024-03-28T22:48:07.654 INFO:teuthology.orchestra.run.smithi043.stdout: "heap": 215420 2024-03-28T22:48:07.654 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.654 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_server": { 2024-03-28T22:48:07.654 INFO:teuthology.orchestra.run.smithi043.stdout: "dispatch_client_request": 0, 2024-03-28T22:48:07.655 INFO:teuthology.orchestra.run.smithi043.stdout: "dispatch_server_request": 0, 2024-03-28T22:48:07.655 INFO:teuthology.orchestra.run.smithi043.stdout: "handle_client_request": 0, 2024-03-28T22:48:07.655 INFO:teuthology.orchestra.run.smithi043.stdout: "handle_client_session": 0, 2024-03-28T22:48:07.655 INFO:teuthology.orchestra.run.smithi043.stdout: "handle_peer_request": 0, 2024-03-28T22:48:07.655 INFO:teuthology.orchestra.run.smithi043.stdout: "req_create_latency": { 2024-03-28T22:48:07.655 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.655 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.655 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.655 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.656 INFO:teuthology.orchestra.run.smithi043.stdout: "req_getattr_latency": { 2024-03-28T22:48:07.656 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.656 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.656 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.656 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.656 INFO:teuthology.orchestra.run.smithi043.stdout: "req_getfilelock_latency": { 2024-03-28T22:48:07.656 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.656 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.656 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.657 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.657 INFO:teuthology.orchestra.run.smithi043.stdout: "req_link_latency": { 2024-03-28T22:48:07.657 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.657 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.657 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.657 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.657 INFO:teuthology.orchestra.run.smithi043.stdout: "req_lookup_latency": { 2024-03-28T22:48:07.657 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.657 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.658 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.658 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.658 INFO:teuthology.orchestra.run.smithi043.stdout: "req_lookuphash_latency": { 2024-03-28T22:48:07.658 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.658 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.658 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.659 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.659 INFO:teuthology.orchestra.run.smithi043.stdout: "req_lookupino_latency": { 2024-03-28T22:48:07.659 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.659 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.659 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.659 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.659 INFO:teuthology.orchestra.run.smithi043.stdout: "req_lookupname_latency": { 2024-03-28T22:48:07.659 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.659 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.660 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.660 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.660 INFO:teuthology.orchestra.run.smithi043.stdout: "req_lookupparent_latency": { 2024-03-28T22:48:07.660 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.660 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.660 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.660 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.660 INFO:teuthology.orchestra.run.smithi043.stdout: "req_lookupsnap_latency": { 2024-03-28T22:48:07.660 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.661 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.661 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.661 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.661 INFO:teuthology.orchestra.run.smithi043.stdout: "req_lssnap_latency": { 2024-03-28T22:48:07.661 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.661 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.661 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.661 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.661 INFO:teuthology.orchestra.run.smithi043.stdout: "req_mkdir_latency": { 2024-03-28T22:48:07.662 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.662 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.663 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.663 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.663 INFO:teuthology.orchestra.run.smithi043.stdout: "req_mknod_latency": { 2024-03-28T22:48:07.663 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.663 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.663 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.663 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.663 INFO:teuthology.orchestra.run.smithi043.stdout: "req_mksnap_latency": { 2024-03-28T22:48:07.664 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.664 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.664 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.664 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.664 INFO:teuthology.orchestra.run.smithi043.stdout: "req_open_latency": { 2024-03-28T22:48:07.664 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.664 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.664 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.664 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.665 INFO:teuthology.orchestra.run.smithi043.stdout: "req_readdir_latency": { 2024-03-28T22:48:07.665 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.665 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.665 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.665 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.665 INFO:teuthology.orchestra.run.smithi043.stdout: "req_rename_latency": { 2024-03-28T22:48:07.665 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.665 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.665 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.666 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.666 INFO:teuthology.orchestra.run.smithi043.stdout: "req_renamesnap_latency": { 2024-03-28T22:48:07.666 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.666 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.666 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.666 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.666 INFO:teuthology.orchestra.run.smithi043.stdout: "req_snapdiff_latency": { 2024-03-28T22:48:07.666 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.667 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.667 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.667 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.667 INFO:teuthology.orchestra.run.smithi043.stdout: "req_rmdir_latency": { 2024-03-28T22:48:07.667 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.667 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.667 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.668 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.668 INFO:teuthology.orchestra.run.smithi043.stdout: "req_rmsnap_latency": { 2024-03-28T22:48:07.668 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.668 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.668 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.668 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.668 INFO:teuthology.orchestra.run.smithi043.stdout: "req_rmxattr_latency": { 2024-03-28T22:48:07.668 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.669 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.669 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.669 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.669 INFO:teuthology.orchestra.run.smithi043.stdout: "req_setattr_latency": { 2024-03-28T22:48:07.669 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.669 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.669 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.669 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.670 INFO:teuthology.orchestra.run.smithi043.stdout: "req_setdirlayout_latency": { 2024-03-28T22:48:07.670 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.670 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.670 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.670 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.670 INFO:teuthology.orchestra.run.smithi043.stdout: "req_setfilelock_latency": { 2024-03-28T22:48:07.670 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.670 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.670 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.671 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.671 INFO:teuthology.orchestra.run.smithi043.stdout: "req_setlayout_latency": { 2024-03-28T22:48:07.671 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.671 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.671 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.671 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.671 INFO:teuthology.orchestra.run.smithi043.stdout: "req_setxattr_latency": { 2024-03-28T22:48:07.671 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.672 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.672 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.672 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.672 INFO:teuthology.orchestra.run.smithi043.stdout: "req_symlink_latency": { 2024-03-28T22:48:07.672 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.672 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.672 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.672 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.672 INFO:teuthology.orchestra.run.smithi043.stdout: "req_unlink_latency": { 2024-03-28T22:48:07.673 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.673 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.673 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.673 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.673 INFO:teuthology.orchestra.run.smithi043.stdout: "cap_revoke_eviction": 0, 2024-03-28T22:48:07.673 INFO:teuthology.orchestra.run.smithi043.stdout: "cap_acquisition_throttle": 0, 2024-03-28T22:48:07.673 INFO:teuthology.orchestra.run.smithi043.stdout: "req_getvxattr_latency": { 2024-03-28T22:48:07.673 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.673 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.674 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.674 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:48:07.674 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.674 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_sessions": { 2024-03-28T22:48:07.674 INFO:teuthology.orchestra.run.smithi043.stdout: "session_count": 2, 2024-03-28T22:48:07.674 INFO:teuthology.orchestra.run.smithi043.stdout: "session_add": 4, 2024-03-28T22:48:07.674 INFO:teuthology.orchestra.run.smithi043.stdout: "session_remove": 2, 2024-03-28T22:48:07.675 INFO:teuthology.orchestra.run.smithi043.stdout: "sessions_open": 3, 2024-03-28T22:48:07.675 INFO:teuthology.orchestra.run.smithi043.stdout: "sessions_stale": 0, 2024-03-28T22:48:07.675 INFO:teuthology.orchestra.run.smithi043.stdout: "total_load": 0, 2024-03-28T22:48:07.675 INFO:teuthology.orchestra.run.smithi043.stdout: "average_load": 0, 2024-03-28T22:48:07.675 INFO:teuthology.orchestra.run.smithi043.stdout: "avg_session_uptime": 2751, 2024-03-28T22:48:07.675 INFO:teuthology.orchestra.run.smithi043.stdout: "mdthresh_evicted": 0 2024-03-28T22:48:07.675 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.675 INFO:teuthology.orchestra.run.smithi043.stdout: "mempool": { 2024-03-28T22:48:07.675 INFO:teuthology.orchestra.run.smithi043.stdout: "bloom_filter_bytes": 0, 2024-03-28T22:48:07.676 INFO:teuthology.orchestra.run.smithi043.stdout: "bloom_filter_items": 0, 2024-03-28T22:48:07.676 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T22:48:07.676 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_alloc_items": 0, 2024-03-28T22:48:07.676 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T22:48:07.676 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_data_items": 0, 2024-03-28T22:48:07.676 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T22:48:07.676 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T22:48:07.676 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T22:48:07.676 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T22:48:07.677 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T22:48:07.677 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_other_items": 0, 2024-03-28T22:48:07.677 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T22:48:07.677 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T22:48:07.677 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_extent_bytes": 0, 2024-03-28T22:48:07.677 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_extent_items": 0, 2024-03-28T22:48:07.677 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_blob_bytes": 0, 2024-03-28T22:48:07.677 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_blob_items": 0, 2024-03-28T22:48:07.678 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T22:48:07.678 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T22:48:07.678 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T22:48:07.678 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T22:48:07.678 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T22:48:07.678 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_items": 0, 2024-03-28T22:48:07.678 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_txc_bytes": 0, 2024-03-28T22:48:07.678 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_txc_items": 0, 2024-03-28T22:48:07.679 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T22:48:07.679 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T22:48:07.679 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_writing_bytes": 0, 2024-03-28T22:48:07.679 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_writing_items": 0, 2024-03-28T22:48:07.679 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_bytes": 0, 2024-03-28T22:48:07.679 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_items": 0, 2024-03-28T22:48:07.679 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T22:48:07.679 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_file_reader_items": 0, 2024-03-28T22:48:07.679 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T22:48:07.680 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_file_writer_items": 0, 2024-03-28T22:48:07.680 INFO:teuthology.orchestra.run.smithi043.stdout: "buffer_anon_bytes": 191192, 2024-03-28T22:48:07.680 INFO:teuthology.orchestra.run.smithi043.stdout: "buffer_anon_items": 53, 2024-03-28T22:48:07.680 INFO:teuthology.orchestra.run.smithi043.stdout: "buffer_meta_bytes": 0, 2024-03-28T22:48:07.680 INFO:teuthology.orchestra.run.smithi043.stdout: "buffer_meta_items": 0, 2024-03-28T22:48:07.680 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_bytes": 0, 2024-03-28T22:48:07.680 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_items": 0, 2024-03-28T22:48:07.680 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mapbl_bytes": 0, 2024-03-28T22:48:07.681 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mapbl_items": 0, 2024-03-28T22:48:07.681 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pglog_bytes": 0, 2024-03-28T22:48:07.681 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pglog_items": 0, 2024-03-28T22:48:07.681 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_bytes": 9736, 2024-03-28T22:48:07.681 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_items": 203, 2024-03-28T22:48:07.681 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T22:48:07.681 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap_mapping_items": 0, 2024-03-28T22:48:07.681 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap_bytes": 0, 2024-03-28T22:48:07.682 INFO:teuthology.orchestra.run.smithi043.stdout: "pgmap_items": 0, 2024-03-28T22:48:07.682 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_co_bytes": 8464, 2024-03-28T22:48:07.682 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_co_items": 13, 2024-03-28T22:48:07.682 INFO:teuthology.orchestra.run.smithi043.stdout: "unittest_1_bytes": 0, 2024-03-28T22:48:07.682 INFO:teuthology.orchestra.run.smithi043.stdout: "unittest_1_items": 0, 2024-03-28T22:48:07.682 INFO:teuthology.orchestra.run.smithi043.stdout: "unittest_2_bytes": 0, 2024-03-28T22:48:07.682 INFO:teuthology.orchestra.run.smithi043.stdout: "unittest_2_items": 0 2024-03-28T22:48:07.682 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.683 INFO:teuthology.orchestra.run.smithi043.stdout: "objecter": { 2024-03-28T22:48:07.683 INFO:teuthology.orchestra.run.smithi043.stdout: "op_active": 0, 2024-03-28T22:48:07.683 INFO:teuthology.orchestra.run.smithi043.stdout: "op_laggy": 0, 2024-03-28T22:48:07.683 INFO:teuthology.orchestra.run.smithi043.stdout: "op_send": 2449, 2024-03-28T22:48:07.683 INFO:teuthology.orchestra.run.smithi043.stdout: "op_send_bytes": 21, 2024-03-28T22:48:07.683 INFO:teuthology.orchestra.run.smithi043.stdout: "op_resend": 3, 2024-03-28T22:48:07.683 INFO:teuthology.orchestra.run.smithi043.stdout: "op_reply": 2449, 2024-03-28T22:48:07.683 INFO:teuthology.orchestra.run.smithi043.stdout: "op_latency": { 2024-03-28T22:48:07.684 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 2449, 2024-03-28T22:48:07.684 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 2.474969544, 2024-03-28T22:48:07.684 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.001010604 2024-03-28T22:48:07.684 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.684 INFO:teuthology.orchestra.run.smithi043.stdout: "op_inflight": 0, 2024-03-28T22:48:07.684 INFO:teuthology.orchestra.run.smithi043.stdout: "oplen_avg": { 2024-03-28T22:48:07.684 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 2449, 2024-03-28T22:48:07.684 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 2450 2024-03-28T22:48:07.685 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.685 INFO:teuthology.orchestra.run.smithi043.stdout: "op": 2449, 2024-03-28T22:48:07.685 INFO:teuthology.orchestra.run.smithi043.stdout: "op_r": 2449, 2024-03-28T22:48:07.685 INFO:teuthology.orchestra.run.smithi043.stdout: "op_w": 0, 2024-03-28T22:48:07.685 INFO:teuthology.orchestra.run.smithi043.stdout: "op_rmw": 0, 2024-03-28T22:48:07.685 INFO:teuthology.orchestra.run.smithi043.stdout: "op_pg": 0, 2024-03-28T22:48:07.685 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_stat": 1598, 2024-03-28T22:48:07.685 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_create": 0, 2024-03-28T22:48:07.686 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_read": 849, 2024-03-28T22:48:07.686 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_write": 0, 2024-03-28T22:48:07.686 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_writefull": 0, 2024-03-28T22:48:07.686 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_writesame": 0, 2024-03-28T22:48:07.686 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_append": 0, 2024-03-28T22:48:07.686 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_zero": 0, 2024-03-28T22:48:07.686 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_truncate": 0, 2024-03-28T22:48:07.686 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_delete": 0, 2024-03-28T22:48:07.687 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_mapext": 0, 2024-03-28T22:48:07.687 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_sparse_read": 0, 2024-03-28T22:48:07.687 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_clonerange": 0, 2024-03-28T22:48:07.687 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_getxattr": 1, 2024-03-28T22:48:07.687 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_setxattr": 0, 2024-03-28T22:48:07.687 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_cmpxattr": 0, 2024-03-28T22:48:07.687 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_rmxattr": 0, 2024-03-28T22:48:07.688 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_resetxattrs": 0, 2024-03-28T22:48:07.688 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_call": 0, 2024-03-28T22:48:07.688 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_watch": 0, 2024-03-28T22:48:07.688 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_notify": 0, 2024-03-28T22:48:07.688 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T22:48:07.688 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_pgls": 0, 2024-03-28T22:48:07.688 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_pgls_filter": 0, 2024-03-28T22:48:07.689 INFO:teuthology.orchestra.run.smithi043.stdout: "osdop_other": 0, 2024-03-28T22:48:07.689 INFO:teuthology.orchestra.run.smithi043.stdout: "linger_active": 0, 2024-03-28T22:48:07.689 INFO:teuthology.orchestra.run.smithi043.stdout: "linger_send": 0, 2024-03-28T22:48:07.689 INFO:teuthology.orchestra.run.smithi043.stdout: "linger_resend": 0, 2024-03-28T22:48:07.689 INFO:teuthology.orchestra.run.smithi043.stdout: "linger_ping": 0, 2024-03-28T22:48:07.689 INFO:teuthology.orchestra.run.smithi043.stdout: "poolop_active": 0, 2024-03-28T22:48:07.689 INFO:teuthology.orchestra.run.smithi043.stdout: "poolop_send": 0, 2024-03-28T22:48:07.689 INFO:teuthology.orchestra.run.smithi043.stdout: "poolop_resend": 0, 2024-03-28T22:48:07.689 INFO:teuthology.orchestra.run.smithi043.stdout: "poolstat_active": 0, 2024-03-28T22:48:07.690 INFO:teuthology.orchestra.run.smithi043.stdout: "poolstat_send": 0, 2024-03-28T22:48:07.690 INFO:teuthology.orchestra.run.smithi043.stdout: "poolstat_resend": 0, 2024-03-28T22:48:07.690 INFO:teuthology.orchestra.run.smithi043.stdout: "statfs_active": 0, 2024-03-28T22:48:07.690 INFO:teuthology.orchestra.run.smithi043.stdout: "statfs_send": 0, 2024-03-28T22:48:07.690 INFO:teuthology.orchestra.run.smithi043.stdout: "statfs_resend": 0, 2024-03-28T22:48:07.690 INFO:teuthology.orchestra.run.smithi043.stdout: "command_active": 0, 2024-03-28T22:48:07.690 INFO:teuthology.orchestra.run.smithi043.stdout: "command_send": 0, 2024-03-28T22:48:07.690 INFO:teuthology.orchestra.run.smithi043.stdout: "command_resend": 0, 2024-03-28T22:48:07.691 INFO:teuthology.orchestra.run.smithi043.stdout: "map_epoch": 108, 2024-03-28T22:48:07.691 INFO:teuthology.orchestra.run.smithi043.stdout: "map_full": 0, 2024-03-28T22:48:07.691 INFO:teuthology.orchestra.run.smithi043.stdout: "map_inc": 16, 2024-03-28T22:48:07.691 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_sessions": 5, 2024-03-28T22:48:07.691 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_session_open": 5, 2024-03-28T22:48:07.691 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_session_close": 0, 2024-03-28T22:48:07.691 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_laggy": 0, 2024-03-28T22:48:07.691 INFO:teuthology.orchestra.run.smithi043.stdout: "omap_wr": 0, 2024-03-28T22:48:07.692 INFO:teuthology.orchestra.run.smithi043.stdout: "omap_rd": 2, 2024-03-28T22:48:07.692 INFO:teuthology.orchestra.run.smithi043.stdout: "omap_del": 0 2024-03-28T22:48:07.692 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.692 INFO:teuthology.orchestra.run.smithi043.stdout: "oft": { 2024-03-28T22:48:07.692 INFO:teuthology.orchestra.run.smithi043.stdout: "omap_total_objs": 0, 2024-03-28T22:48:07.692 INFO:teuthology.orchestra.run.smithi043.stdout: "omap_total_kv_pairs": 0, 2024-03-28T22:48:07.692 INFO:teuthology.orchestra.run.smithi043.stdout: "omap_total_updates": 0, 2024-03-28T22:48:07.692 INFO:teuthology.orchestra.run.smithi043.stdout: "omap_total_removes": 0 2024-03-28T22:48:07.692 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.693 INFO:teuthology.orchestra.run.smithi043.stdout: "osd-slow-ops": { 2024-03-28T22:48:07.693 INFO:teuthology.orchestra.run.smithi043.stdout: "slow_ops_count": 0 2024-03-28T22:48:07.693 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.693 INFO:teuthology.orchestra.run.smithi043.stdout: "purge_queue": { 2024-03-28T22:48:07.693 INFO:teuthology.orchestra.run.smithi043.stdout: "pq_executing_ops": 0, 2024-03-28T22:48:07.693 INFO:teuthology.orchestra.run.smithi043.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T22:48:07.693 INFO:teuthology.orchestra.run.smithi043.stdout: "pq_executing": 0, 2024-03-28T22:48:07.693 INFO:teuthology.orchestra.run.smithi043.stdout: "pq_executing_high_water": 0, 2024-03-28T22:48:07.693 INFO:teuthology.orchestra.run.smithi043.stdout: "pq_executed_ops": 0, 2024-03-28T22:48:07.694 INFO:teuthology.orchestra.run.smithi043.stdout: "pq_executed": 0, 2024-03-28T22:48:07.694 INFO:teuthology.orchestra.run.smithi043.stdout: "pq_item_in_journal": 0 2024-03-28T22:48:07.694 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.694 INFO:teuthology.orchestra.run.smithi043.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T22:48:07.694 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 0, 2024-03-28T22:48:07.694 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 104857600, 2024-03-28T22:48:07.694 INFO:teuthology.orchestra.run.smithi043.stdout: "get_started": 0, 2024-03-28T22:48:07.694 INFO:teuthology.orchestra.run.smithi043.stdout: "get": 2733, 2024-03-28T22:48:07.695 INFO:teuthology.orchestra.run.smithi043.stdout: "get_sum": 2341036, 2024-03-28T22:48:07.695 INFO:teuthology.orchestra.run.smithi043.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:07.695 INFO:teuthology.orchestra.run.smithi043.stdout: "get_or_fail_success": 2733, 2024-03-28T22:48:07.695 INFO:teuthology.orchestra.run.smithi043.stdout: "take": 0, 2024-03-28T22:48:07.695 INFO:teuthology.orchestra.run.smithi043.stdout: "take_sum": 0, 2024-03-28T22:48:07.695 INFO:teuthology.orchestra.run.smithi043.stdout: "put": 2733, 2024-03-28T22:48:07.695 INFO:teuthology.orchestra.run.smithi043.stdout: "put_sum": 2341036, 2024-03-28T22:48:07.695 INFO:teuthology.orchestra.run.smithi043.stdout: "wait": { 2024-03-28T22:48:07.696 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.696 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.696 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.696 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:48:07.696 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.696 INFO:teuthology.orchestra.run.smithi043.stdout: "throttle-objecter_bytes": { 2024-03-28T22:48:07.696 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 0, 2024-03-28T22:48:07.696 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 104857600, 2024-03-28T22:48:07.696 INFO:teuthology.orchestra.run.smithi043.stdout: "get_started": 0, 2024-03-28T22:48:07.697 INFO:teuthology.orchestra.run.smithi043.stdout: "get": 0, 2024-03-28T22:48:07.697 INFO:teuthology.orchestra.run.smithi043.stdout: "get_sum": 0, 2024-03-28T22:48:07.697 INFO:teuthology.orchestra.run.smithi043.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:07.697 INFO:teuthology.orchestra.run.smithi043.stdout: "get_or_fail_success": 0, 2024-03-28T22:48:07.697 INFO:teuthology.orchestra.run.smithi043.stdout: "take": 2449, 2024-03-28T22:48:07.697 INFO:teuthology.orchestra.run.smithi043.stdout: "take_sum": 2354, 2024-03-28T22:48:07.697 INFO:teuthology.orchestra.run.smithi043.stdout: "put": 2449, 2024-03-28T22:48:07.697 INFO:teuthology.orchestra.run.smithi043.stdout: "put_sum": 2354, 2024-03-28T22:48:07.697 INFO:teuthology.orchestra.run.smithi043.stdout: "wait": { 2024-03-28T22:48:07.698 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.698 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.698 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.698 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:48:07.698 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.698 INFO:teuthology.orchestra.run.smithi043.stdout: "throttle-objecter_ops": { 2024-03-28T22:48:07.698 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 0, 2024-03-28T22:48:07.698 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 1024, 2024-03-28T22:48:07.699 INFO:teuthology.orchestra.run.smithi043.stdout: "get_started": 0, 2024-03-28T22:48:07.699 INFO:teuthology.orchestra.run.smithi043.stdout: "get": 0, 2024-03-28T22:48:07.699 INFO:teuthology.orchestra.run.smithi043.stdout: "get_sum": 0, 2024-03-28T22:48:07.699 INFO:teuthology.orchestra.run.smithi043.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:07.699 INFO:teuthology.orchestra.run.smithi043.stdout: "get_or_fail_success": 0, 2024-03-28T22:48:07.699 INFO:teuthology.orchestra.run.smithi043.stdout: "take": 2449, 2024-03-28T22:48:07.699 INFO:teuthology.orchestra.run.smithi043.stdout: "take_sum": 2449, 2024-03-28T22:48:07.699 INFO:teuthology.orchestra.run.smithi043.stdout: "put": 2449, 2024-03-28T22:48:07.700 INFO:teuthology.orchestra.run.smithi043.stdout: "put_sum": 2449, 2024-03-28T22:48:07.700 INFO:teuthology.orchestra.run.smithi043.stdout: "wait": { 2024-03-28T22:48:07.700 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.700 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.700 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.700 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:48:07.700 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.701 INFO:teuthology.orchestra.run.smithi043.stdout: "throttle-write_buf_throttle": { 2024-03-28T22:48:07.701 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 0, 2024-03-28T22:48:07.701 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 3758096384, 2024-03-28T22:48:07.701 INFO:teuthology.orchestra.run.smithi043.stdout: "get_started": 0, 2024-03-28T22:48:07.701 INFO:teuthology.orchestra.run.smithi043.stdout: "get": 0, 2024-03-28T22:48:07.701 INFO:teuthology.orchestra.run.smithi043.stdout: "get_sum": 0, 2024-03-28T22:48:07.701 INFO:teuthology.orchestra.run.smithi043.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:07.701 INFO:teuthology.orchestra.run.smithi043.stdout: "get_or_fail_success": 0, 2024-03-28T22:48:07.701 INFO:teuthology.orchestra.run.smithi043.stdout: "take": 0, 2024-03-28T22:48:07.702 INFO:teuthology.orchestra.run.smithi043.stdout: "take_sum": 0, 2024-03-28T22:48:07.702 INFO:teuthology.orchestra.run.smithi043.stdout: "put": 0, 2024-03-28T22:48:07.702 INFO:teuthology.orchestra.run.smithi043.stdout: "put_sum": 0, 2024-03-28T22:48:07.702 INFO:teuthology.orchestra.run.smithi043.stdout: "wait": { 2024-03-28T22:48:07.702 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.702 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.702 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.702 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:48:07.703 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-03-28T22:48:07.703 INFO:teuthology.orchestra.run.smithi043.stdout: "throttle-write_buf_throttle-0x56285a102780": { 2024-03-28T22:48:07.703 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 0, 2024-03-28T22:48:07.703 INFO:teuthology.orchestra.run.smithi043.stdout: "max": 3758096384, 2024-03-28T22:48:07.703 INFO:teuthology.orchestra.run.smithi043.stdout: "get_started": 0, 2024-03-28T22:48:07.703 INFO:teuthology.orchestra.run.smithi043.stdout: "get": 0, 2024-03-28T22:48:07.703 INFO:teuthology.orchestra.run.smithi043.stdout: "get_sum": 0, 2024-03-28T22:48:07.703 INFO:teuthology.orchestra.run.smithi043.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:07.704 INFO:teuthology.orchestra.run.smithi043.stdout: "get_or_fail_success": 0, 2024-03-28T22:48:07.704 INFO:teuthology.orchestra.run.smithi043.stdout: "take": 0, 2024-03-28T22:48:07.704 INFO:teuthology.orchestra.run.smithi043.stdout: "take_sum": 0, 2024-03-28T22:48:07.704 INFO:teuthology.orchestra.run.smithi043.stdout: "put": 0, 2024-03-28T22:48:07.704 INFO:teuthology.orchestra.run.smithi043.stdout: "put_sum": 0, 2024-03-28T22:48:07.704 INFO:teuthology.orchestra.run.smithi043.stdout: "wait": { 2024-03-28T22:48:07.704 INFO:teuthology.orchestra.run.smithi043.stdout: "avgcount": 0, 2024-03-28T22:48:07.704 INFO:teuthology.orchestra.run.smithi043.stdout: "sum": 0.000000000, 2024-03-28T22:48:07.704 INFO:teuthology.orchestra.run.smithi043.stdout: "avgtime": 0.000000000 2024-03-28T22:48:07.705 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:48:07.705 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-03-28T22:48:07.705 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-03-28T22:48:08.579 INFO:tasks.check_counter:Daemon mds.j mds.exported=0 2024-03-28T22:48:08.579 INFO:tasks.check_counter:Daemon mds.j mds.imported=0 2024-03-28T22:48:08.579 INFO:tasks.check_counter:Daemon mds.j mds_cache.dir_update=0 2024-03-28T22:48:08.579 INFO:tasks.check_counter:Daemon mds.j mds_cache.dir_update_receipt=0 2024-03-28T22:48:08.579 INFO:tasks.check_counter:Daemon mds.j mds.root_rsnaps=0 2024-03-28T22:48:08.579 INFO:tasks.check_counter:Daemon mds.j mds_server.req_mksnap_latency.avgcount=0 2024-03-28T22:48:08.579 INFO:tasks.check_counter:Daemon mds.j mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T22:48:08.580 DEBUG:tasks.check_counter:Getting stats from b 2024-03-28T22:48:08.580 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-28T22:48:08.838 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T22:48:09.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:08 smithi043 ceph-mon[31823]: pgmap v342: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-03-28T22:48:09.272 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:08 smithi082 ceph-mon[36471]: pgmap v342: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-03-28T22:48:09.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:08 smithi172 ceph-mon[36566]: pgmap v342: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-03-28T22:48:10.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:10 smithi043 ceph-mon[31823]: osdmap e109: 12 total, 12 up, 12 in 2024-03-28T22:48:10.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:10 smithi172 ceph-mon[36566]: osdmap e109: 12 total, 12 up, 12 in 2024-03-28T22:48:10.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:10 smithi082 ceph-mon[36471]: osdmap e109: 12 total, 12 up, 12 in 2024-03-28T22:48:11.192 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:11 smithi082 ceph-mon[36471]: pgmap v344: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-28T22:48:11.192 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-28T22:48:11.192 INFO:teuthology.orchestra.run.smithi082.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T22:48:11.192 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_messages": 62687, 2024-03-28T22:48:11.192 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages": 97762, 2024-03-28T22:48:11.193 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_bytes": 15226409, 2024-03-28T22:48:11.193 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_bytes": 101506555, 2024-03-28T22:48:11.193 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_created_connections": 15, 2024-03-28T22:48:11.193 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_active_connections": 6, 2024-03-28T22:48:11.193 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_total_time": 4.742757471, 2024-03-28T22:48:11.193 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_send_time": 1.873903971, 2024-03-28T22:48:11.193 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_recv_time": 2.113303572, 2024-03-28T22:48:11.193 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_fast_dispatch_time": 0.706821528, 2024-03-28T22:48:11.194 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:48:11.194 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 97762, 2024-03-28T22:48:11.194 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 18.737848626, 2024-03-28T22:48:11.194 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000191668 2024-03-28T22:48:11.194 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.194 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:48:11.194 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 105205, 2024-03-28T22:48:11.194 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.019793335, 2024-03-28T22:48:11.195 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000188 2024-03-28T22:48:11.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.195 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_encrypted_bytes": 1955408, 2024-03-28T22:48:11.195 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_encrypted_bytes": 507920 2024-03-28T22:48:11.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.195 INFO:teuthology.orchestra.run.smithi082.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T22:48:11.195 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_messages": 199967, 2024-03-28T22:48:11.195 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages": 317811, 2024-03-28T22:48:11.196 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_bytes": 60767575, 2024-03-28T22:48:11.196 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_bytes": 318060219, 2024-03-28T22:48:11.196 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_created_connections": 34, 2024-03-28T22:48:11.196 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_active_connections": 7, 2024-03-28T22:48:11.196 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_total_time": 13.208714604, 2024-03-28T22:48:11.196 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_send_time": 5.959618980, 2024-03-28T22:48:11.196 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_recv_time": 7.439408018, 2024-03-28T22:48:11.196 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_fast_dispatch_time": 0.862603948, 2024-03-28T22:48:11.196 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:48:11.196 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 317808, 2024-03-28T22:48:11.197 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 20.256942446, 2024-03-28T22:48:11.197 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000063739 2024-03-28T22:48:11.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.197 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:48:11.197 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 416588, 2024-03-28T22:48:11.197 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.068346043, 2024-03-28T22:48:11.197 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000164 2024-03-28T22:48:11.197 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.198 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_encrypted_bytes": 421296, 2024-03-28T22:48:11.198 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_encrypted_bytes": 372048 2024-03-28T22:48:11.198 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.198 INFO:teuthology.orchestra.run.smithi082.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T22:48:11.198 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_messages": 99476, 2024-03-28T22:48:11.198 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages": 169297, 2024-03-28T22:48:11.198 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_bytes": 19661843, 2024-03-28T22:48:11.198 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_bytes": 159010213, 2024-03-28T22:48:11.199 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_created_connections": 212, 2024-03-28T22:48:11.199 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_active_connections": 10, 2024-03-28T22:48:11.199 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_total_time": 7.222005676, 2024-03-28T22:48:11.199 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_send_time": 3.291273333, 2024-03-28T22:48:11.199 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_recv_time": 3.162421763, 2024-03-28T22:48:11.199 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_fast_dispatch_time": 0.761409136, 2024-03-28T22:48:11.199 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:48:11.199 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 169295, 2024-03-28T22:48:11.200 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 20.232847976, 2024-03-28T22:48:11.200 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000119512 2024-03-28T22:48:11.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.200 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:48:11.200 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 210774, 2024-03-28T22:48:11.200 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.035682998, 2024-03-28T22:48:11.200 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000169 2024-03-28T22:48:11.200 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.200 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_encrypted_bytes": 48736, 2024-03-28T22:48:11.201 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_encrypted_bytes": 2432 2024-03-28T22:48:11.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.201 INFO:teuthology.orchestra.run.smithi082.stdout: "cct": { 2024-03-28T22:48:11.201 INFO:teuthology.orchestra.run.smithi082.stdout: "total_workers": 1, 2024-03-28T22:48:11.201 INFO:teuthology.orchestra.run.smithi082.stdout: "unhealthy_workers": 0 2024-03-28T22:48:11.201 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.201 INFO:teuthology.orchestra.run.smithi082.stdout: "finisher-MDSRank": { 2024-03-28T22:48:11.201 INFO:teuthology.orchestra.run.smithi082.stdout: "queue_len": 0, 2024-03-28T22:48:11.201 INFO:teuthology.orchestra.run.smithi082.stdout: "complete_latency": { 2024-03-28T22:48:11.202 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 53541, 2024-03-28T22:48:11.202 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 60.342380138, 2024-03-28T22:48:11.202 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.001127031 2024-03-28T22:48:11.202 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:11.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.202 INFO:teuthology.orchestra.run.smithi082.stdout: "finisher-PurgeQueue": { 2024-03-28T22:48:11.202 INFO:teuthology.orchestra.run.smithi082.stdout: "queue_len": 0, 2024-03-28T22:48:11.202 INFO:teuthology.orchestra.run.smithi082.stdout: "complete_latency": { 2024-03-28T22:48:11.203 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 31617, 2024-03-28T22:48:11.203 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 3.508489931, 2024-03-28T22:48:11.203 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000110968 2024-03-28T22:48:11.203 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:11.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.203 INFO:teuthology.orchestra.run.smithi082.stdout: "mds": { 2024-03-28T22:48:11.203 INFO:teuthology.orchestra.run.smithi082.stdout: "request": 101645, 2024-03-28T22:48:11.203 INFO:teuthology.orchestra.run.smithi082.stdout: "reply": 101645, 2024-03-28T22:48:11.204 INFO:teuthology.orchestra.run.smithi082.stdout: "reply_latency": { 2024-03-28T22:48:11.204 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 101645, 2024-03-28T22:48:11.204 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 179.272779495, 2024-03-28T22:48:11.204 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.001763714 2024-03-28T22:48:11.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.204 INFO:teuthology.orchestra.run.smithi082.stdout: "slow_reply": 0, 2024-03-28T22:48:11.204 INFO:teuthology.orchestra.run.smithi082.stdout: "forward": 0, 2024-03-28T22:48:11.204 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_fetch_complete": 0, 2024-03-28T22:48:11.205 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_fetch_keys": 0, 2024-03-28T22:48:11.205 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_commit": 982, 2024-03-28T22:48:11.205 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_split": 4, 2024-03-28T22:48:11.205 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_merge": 6, 2024-03-28T22:48:11.205 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes": 1241, 2024-03-28T22:48:11.205 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_top": 256, 2024-03-28T22:48:11.205 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_bottom": 985, 2024-03-28T22:48:11.205 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_pin_tail": 0, 2024-03-28T22:48:11.205 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_pinned": 875, 2024-03-28T22:48:11.206 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_expired": 0, 2024-03-28T22:48:11.206 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_with_caps": 32, 2024-03-28T22:48:11.206 INFO:teuthology.orchestra.run.smithi082.stdout: "caps": 32, 2024-03-28T22:48:11.206 INFO:teuthology.orchestra.run.smithi082.stdout: "subtrees": 2, 2024-03-28T22:48:11.206 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse": 145721, 2024-03-28T22:48:11.206 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_hit": 86173, 2024-03-28T22:48:11.206 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_forward": 0, 2024-03-28T22:48:11.206 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_discover": 0, 2024-03-28T22:48:11.207 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_dir_fetch": 0, 2024-03-28T22:48:11.207 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_remote_ino": 0, 2024-03-28T22:48:11.207 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_lock": 1242, 2024-03-28T22:48:11.207 INFO:teuthology.orchestra.run.smithi082.stdout: "load_cent": 0, 2024-03-28T22:48:11.207 INFO:teuthology.orchestra.run.smithi082.stdout: "q": 0, 2024-03-28T22:48:11.207 INFO:teuthology.orchestra.run.smithi082.stdout: "exported": 0, 2024-03-28T22:48:11.207 INFO:teuthology.orchestra.run.smithi082.stdout: "exported_inodes": 0, 2024-03-28T22:48:11.207 INFO:teuthology.orchestra.run.smithi082.stdout: "imported": 0, 2024-03-28T22:48:11.208 INFO:teuthology.orchestra.run.smithi082.stdout: "imported_inodes": 0, 2024-03-28T22:48:11.208 INFO:teuthology.orchestra.run.smithi082.stdout: "openino_dir_fetch": 0, 2024-03-28T22:48:11.208 INFO:teuthology.orchestra.run.smithi082.stdout: "openino_backtrace_fetch": 1545, 2024-03-28T22:48:11.208 INFO:teuthology.orchestra.run.smithi082.stdout: "openino_peer_discover": 0, 2024-03-28T22:48:11.208 INFO:teuthology.orchestra.run.smithi082.stdout: "root_rfiles": 61, 2024-03-28T22:48:11.208 INFO:teuthology.orchestra.run.smithi082.stdout: "root_rbytes": 21056780, 2024-03-28T22:48:11.208 INFO:teuthology.orchestra.run.smithi082.stdout: "root_rsnaps": 6, 2024-03-28T22:48:11.208 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_backtrace_fetch": 1545, 2024-03-28T22:48:11.209 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_set_tag": 0, 2024-03-28T22:48:11.209 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T22:48:11.209 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_inotable_repaired": 0, 2024-03-28T22:48:11.209 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_dir_inodes": 380, 2024-03-28T22:48:11.209 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_dir_base_inodes": 20, 2024-03-28T22:48:11.209 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_dirfrag_rstats": 360, 2024-03-28T22:48:11.209 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_file_inodes": 1165, 2024-03-28T22:48:11.209 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_inode_file_caps": 0, 2024-03-28T22:48:11.209 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_revoke": 5053, 2024-03-28T22:48:11.210 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_grant": 65782, 2024-03-28T22:48:11.210 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T22:48:11.210 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T22:48:11.210 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T22:48:11.210 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_caps": 27596, 2024-03-28T22:48:11.210 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_caps_dirty": 13139, 2024-03-28T22:48:11.210 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_cap_release": 6601, 2024-03-28T22:48:11.210 INFO:teuthology.orchestra.run.smithi082.stdout: "process_request_cap_release": 64968 2024-03-28T22:48:11.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.211 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_cache": { 2024-03-28T22:48:11.211 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_update": 48, 2024-03-28T22:48:11.211 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_update_receipt": 0, 2024-03-28T22:48:11.211 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_try_discover": 0, 2024-03-28T22:48:11.211 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_send_discover": 0, 2024-03-28T22:48:11.211 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_handle_discover": 97, 2024-03-28T22:48:11.211 INFO:teuthology.orchestra.run.smithi082.stdout: "num_strays": 0, 2024-03-28T22:48:11.211 INFO:teuthology.orchestra.run.smithi082.stdout: "num_strays_delayed": 0, 2024-03-28T22:48:11.212 INFO:teuthology.orchestra.run.smithi082.stdout: "num_strays_enqueuing": 0, 2024-03-28T22:48:11.212 INFO:teuthology.orchestra.run.smithi082.stdout: "strays_created": 16558, 2024-03-28T22:48:11.212 INFO:teuthology.orchestra.run.smithi082.stdout: "strays_enqueued": 16558, 2024-03-28T22:48:11.212 INFO:teuthology.orchestra.run.smithi082.stdout: "strays_reintegrated": 0, 2024-03-28T22:48:11.212 INFO:teuthology.orchestra.run.smithi082.stdout: "strays_migrated": 0, 2024-03-28T22:48:11.212 INFO:teuthology.orchestra.run.smithi082.stdout: "num_recovering_processing": 0, 2024-03-28T22:48:11.212 INFO:teuthology.orchestra.run.smithi082.stdout: "num_recovering_enqueued": 0, 2024-03-28T22:48:11.212 INFO:teuthology.orchestra.run.smithi082.stdout: "num_recovering_prioritized": 0, 2024-03-28T22:48:11.213 INFO:teuthology.orchestra.run.smithi082.stdout: "recovery_started": 0, 2024-03-28T22:48:11.213 INFO:teuthology.orchestra.run.smithi082.stdout: "recovery_completed": 0, 2024-03-28T22:48:11.213 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_quiesce_path": 0, 2024-03-28T22:48:11.213 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_quiesce_inode": 0, 2024-03-28T22:48:11.213 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_enqueue_scrub": 20, 2024-03-28T22:48:11.213 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_exportdir": 0, 2024-03-28T22:48:11.213 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_flush": 0, 2024-03-28T22:48:11.213 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_fragmentdir": 10, 2024-03-28T22:48:11.213 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_fragstats": 0, 2024-03-28T22:48:11.214 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_inodestats": 0 2024-03-28T22:48:11.214 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.214 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_log": { 2024-03-28T22:48:11.214 INFO:teuthology.orchestra.run.smithi082.stdout: "evlrg": 0, 2024-03-28T22:48:11.214 INFO:teuthology.orchestra.run.smithi082.stdout: "evadd": 75874, 2024-03-28T22:48:11.214 INFO:teuthology.orchestra.run.smithi082.stdout: "evex": 72790, 2024-03-28T22:48:11.214 INFO:teuthology.orchestra.run.smithi082.stdout: "evtrm": 72790, 2024-03-28T22:48:11.214 INFO:teuthology.orchestra.run.smithi082.stdout: "ev": 3084, 2024-03-28T22:48:11.215 INFO:teuthology.orchestra.run.smithi082.stdout: "evexg": 0, 2024-03-28T22:48:11.215 INFO:teuthology.orchestra.run.smithi082.stdout: "evexd": 2596, 2024-03-28T22:48:11.215 INFO:teuthology.orchestra.run.smithi082.stdout: "segadd": 128, 2024-03-28T22:48:11.215 INFO:teuthology.orchestra.run.smithi082.stdout: "segex": 124, 2024-03-28T22:48:11.215 INFO:teuthology.orchestra.run.smithi082.stdout: "segtrm": 124, 2024-03-28T22:48:11.215 INFO:teuthology.orchestra.run.smithi082.stdout: "seg": 4, 2024-03-28T22:48:11.215 INFO:teuthology.orchestra.run.smithi082.stdout: "segmjr": 1, 2024-03-28T22:48:11.215 INFO:teuthology.orchestra.run.smithi082.stdout: "segexg": 0, 2024-03-28T22:48:11.216 INFO:teuthology.orchestra.run.smithi082.stdout: "segexd": 5, 2024-03-28T22:48:11.216 INFO:teuthology.orchestra.run.smithi082.stdout: "expos": 235966387, 2024-03-28T22:48:11.216 INFO:teuthology.orchestra.run.smithi082.stdout: "wrpos": 246449318, 2024-03-28T22:48:11.216 INFO:teuthology.orchestra.run.smithi082.stdout: "rdpos": 0, 2024-03-28T22:48:11.216 INFO:teuthology.orchestra.run.smithi082.stdout: "jlat": { 2024-03-28T22:48:11.216 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 26031, 2024-03-28T22:48:11.216 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 97.948186907, 2024-03-28T22:48:11.216 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.003762751 2024-03-28T22:48:11.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.217 INFO:teuthology.orchestra.run.smithi082.stdout: "replayed": 0 2024-03-28T22:48:11.217 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.217 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_mem": { 2024-03-28T22:48:11.217 INFO:teuthology.orchestra.run.smithi082.stdout: "ino": 433, 2024-03-28T22:48:11.217 INFO:teuthology.orchestra.run.smithi082.stdout: "ino+": 17011, 2024-03-28T22:48:11.217 INFO:teuthology.orchestra.run.smithi082.stdout: "ino-": 16578, 2024-03-28T22:48:11.217 INFO:teuthology.orchestra.run.smithi082.stdout: "dir": 42, 2024-03-28T22:48:11.217 INFO:teuthology.orchestra.run.smithi082.stdout: "dir+": 70, 2024-03-28T22:48:11.217 INFO:teuthology.orchestra.run.smithi082.stdout: "dir-": 28, 2024-03-28T22:48:11.218 INFO:teuthology.orchestra.run.smithi082.stdout: "dn": 1241, 2024-03-28T22:48:11.218 INFO:teuthology.orchestra.run.smithi082.stdout: "dn+": 36234, 2024-03-28T22:48:11.218 INFO:teuthology.orchestra.run.smithi082.stdout: "dn-": 34993, 2024-03-28T22:48:11.218 INFO:teuthology.orchestra.run.smithi082.stdout: "cap": 32, 2024-03-28T22:48:11.218 INFO:teuthology.orchestra.run.smithi082.stdout: "cap+": 17477, 2024-03-28T22:48:11.218 INFO:teuthology.orchestra.run.smithi082.stdout: "cap-": 17445, 2024-03-28T22:48:11.218 INFO:teuthology.orchestra.run.smithi082.stdout: "rss": 74112, 2024-03-28T22:48:11.218 INFO:teuthology.orchestra.run.smithi082.stdout: "heap": 231804 2024-03-28T22:48:11.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.219 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_server": { 2024-03-28T22:48:11.219 INFO:teuthology.orchestra.run.smithi082.stdout: "dispatch_client_request": 135526, 2024-03-28T22:48:11.219 INFO:teuthology.orchestra.run.smithi082.stdout: "dispatch_server_request": 1, 2024-03-28T22:48:11.219 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_request": 101645, 2024-03-28T22:48:11.219 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_session": 2242, 2024-03-28T22:48:11.219 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_peer_request": 2, 2024-03-28T22:48:11.219 INFO:teuthology.orchestra.run.smithi082.stdout: "req_create_latency": { 2024-03-28T22:48:11.219 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 16619, 2024-03-28T22:48:11.220 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 22.188538808, 2024-03-28T22:48:11.220 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.001335130 2024-03-28T22:48:11.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.220 INFO:teuthology.orchestra.run.smithi082.stdout: "req_getattr_latency": { 2024-03-28T22:48:11.220 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 201, 2024-03-28T22:48:11.220 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.122005103, 2024-03-28T22:48:11.220 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000606990 2024-03-28T22:48:11.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.220 INFO:teuthology.orchestra.run.smithi082.stdout: "req_getfilelock_latency": { 2024-03-28T22:48:11.221 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:11.221 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:11.221 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:11.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.221 INFO:teuthology.orchestra.run.smithi082.stdout: "req_link_latency": { 2024-03-28T22:48:11.221 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:11.221 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:11.221 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:11.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.222 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookup_latency": { 2024-03-28T22:48:11.222 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 33935, 2024-03-28T22:48:11.222 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 22.663989508, 2024-03-28T22:48:11.222 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000667864 2024-03-28T22:48:11.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.222 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookuphash_latency": { 2024-03-28T22:48:11.222 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:11.222 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:11.223 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:11.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.223 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookupino_latency": { 2024-03-28T22:48:11.223 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:11.223 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:11.223 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:11.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.223 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookupname_latency": { 2024-03-28T22:48:11.223 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:11.224 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:11.224 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:11.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.224 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookupparent_latency": { 2024-03-28T22:48:11.224 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:11.224 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:11.224 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:11.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.224 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookupsnap_latency": { 2024-03-28T22:48:11.225 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:11.225 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:11.225 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:11.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.226 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lssnap_latency": { 2024-03-28T22:48:11.226 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 11, 2024-03-28T22:48:11.226 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.007193429, 2024-03-28T22:48:11.226 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000653948 2024-03-28T22:48:11.226 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.226 INFO:teuthology.orchestra.run.smithi082.stdout: "req_mkdir_latency": { 2024-03-28T22:48:11.226 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 20, 2024-03-28T22:48:11.227 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.023673742, 2024-03-28T22:48:11.227 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.001183687 2024-03-28T22:48:11.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.227 INFO:teuthology.orchestra.run.smithi082.stdout: "req_mknod_latency": { 2024-03-28T22:48:11.227 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:11.227 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:11.227 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:11.227 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.227 INFO:teuthology.orchestra.run.smithi082.stdout: "req_mksnap_latency": { 2024-03-28T22:48:11.228 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 11, 2024-03-28T22:48:11.228 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.104901294, 2024-03-28T22:48:11.228 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.009536481 2024-03-28T22:48:11.228 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.228 INFO:teuthology.orchestra.run.smithi082.stdout: "req_open_latency": { 2024-03-28T22:48:11.228 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 1, 2024-03-28T22:48:11.228 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000222071, 2024-03-28T22:48:11.228 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000222071 2024-03-28T22:48:11.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.229 INFO:teuthology.orchestra.run.smithi082.stdout: "req_readdir_latency": { 2024-03-28T22:48:11.229 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 30278, 2024-03-28T22:48:11.229 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 100.622242406, 2024-03-28T22:48:11.229 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.003323279 2024-03-28T22:48:11.229 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.229 INFO:teuthology.orchestra.run.smithi082.stdout: "req_rename_latency": { 2024-03-28T22:48:11.229 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 3463, 2024-03-28T22:48:11.230 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 6.158578995, 2024-03-28T22:48:11.230 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.001778394 2024-03-28T22:48:11.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.230 INFO:teuthology.orchestra.run.smithi082.stdout: "req_renamesnap_latency": { 2024-03-28T22:48:11.230 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:11.230 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:11.230 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:11.230 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.230 INFO:teuthology.orchestra.run.smithi082.stdout: "req_snapdiff_latency": { 2024-03-28T22:48:11.231 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:11.231 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:11.231 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:11.231 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.231 INFO:teuthology.orchestra.run.smithi082.stdout: "req_rmdir_latency": { 2024-03-28T22:48:11.231 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:11.231 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:11.231 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:11.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.232 INFO:teuthology.orchestra.run.smithi082.stdout: "req_rmsnap_latency": { 2024-03-28T22:48:11.232 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 5, 2024-03-28T22:48:11.232 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.032612830, 2024-03-28T22:48:11.232 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.006522566 2024-03-28T22:48:11.232 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.232 INFO:teuthology.orchestra.run.smithi082.stdout: "req_rmxattr_latency": { 2024-03-28T22:48:11.232 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:11.233 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:11.233 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:11.233 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.233 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setattr_latency": { 2024-03-28T22:48:11.233 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 2, 2024-03-28T22:48:11.233 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.001114538, 2024-03-28T22:48:11.233 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000557269 2024-03-28T22:48:11.233 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.233 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setdirlayout_latency": { 2024-03-28T22:48:11.234 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:11.234 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:11.234 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:11.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.234 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setfilelock_latency": { 2024-03-28T22:48:11.234 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 532, 2024-03-28T22:48:11.234 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.288818645, 2024-03-28T22:48:11.234 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000542892 2024-03-28T22:48:11.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.235 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setlayout_latency": { 2024-03-28T22:48:11.235 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:11.235 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:11.235 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:11.235 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.235 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setxattr_latency": { 2024-03-28T22:48:11.235 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 9, 2024-03-28T22:48:11.235 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.023019957, 2024-03-28T22:48:11.236 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.002557773 2024-03-28T22:48:11.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.236 INFO:teuthology.orchestra.run.smithi082.stdout: "req_symlink_latency": { 2024-03-28T22:48:11.236 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:11.236 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:11.236 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:11.236 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.236 INFO:teuthology.orchestra.run.smithi082.stdout: "req_unlink_latency": { 2024-03-28T22:48:11.237 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 16558, 2024-03-28T22:48:11.237 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 27.035868169, 2024-03-28T22:48:11.237 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.001632797 2024-03-28T22:48:11.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.237 INFO:teuthology.orchestra.run.smithi082.stdout: "cap_revoke_eviction": 0, 2024-03-28T22:48:11.237 INFO:teuthology.orchestra.run.smithi082.stdout: "cap_acquisition_throttle": 0, 2024-03-28T22:48:11.237 INFO:teuthology.orchestra.run.smithi082.stdout: "req_getvxattr_latency": { 2024-03-28T22:48:11.237 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:11.237 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:11.238 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:11.238 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:11.238 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.238 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_sessions": { 2024-03-28T22:48:11.238 INFO:teuthology.orchestra.run.smithi082.stdout: "session_count": 2, 2024-03-28T22:48:11.238 INFO:teuthology.orchestra.run.smithi082.stdout: "session_add": 7, 2024-03-28T22:48:11.238 INFO:teuthology.orchestra.run.smithi082.stdout: "session_remove": 5, 2024-03-28T22:48:11.238 INFO:teuthology.orchestra.run.smithi082.stdout: "sessions_open": 2, 2024-03-28T22:48:11.238 INFO:teuthology.orchestra.run.smithi082.stdout: "sessions_stale": 0, 2024-03-28T22:48:11.239 INFO:teuthology.orchestra.run.smithi082.stdout: "total_load": 1666, 2024-03-28T22:48:11.239 INFO:teuthology.orchestra.run.smithi082.stdout: "average_load": 833, 2024-03-28T22:48:11.239 INFO:teuthology.orchestra.run.smithi082.stdout: "avg_session_uptime": 643, 2024-03-28T22:48:11.239 INFO:teuthology.orchestra.run.smithi082.stdout: "mdthresh_evicted": 0 2024-03-28T22:48:11.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.239 INFO:teuthology.orchestra.run.smithi082.stdout: "mempool": { 2024-03-28T22:48:11.239 INFO:teuthology.orchestra.run.smithi082.stdout: "bloom_filter_bytes": 0, 2024-03-28T22:48:11.239 INFO:teuthology.orchestra.run.smithi082.stdout: "bloom_filter_items": 0, 2024-03-28T22:48:11.239 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T22:48:11.240 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_alloc_items": 0, 2024-03-28T22:48:11.240 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T22:48:11.240 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_data_items": 0, 2024-03-28T22:48:11.240 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T22:48:11.240 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T22:48:11.240 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T22:48:11.240 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T22:48:11.240 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T22:48:11.241 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_other_items": 0, 2024-03-28T22:48:11.241 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T22:48:11.241 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T22:48:11.241 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_extent_bytes": 0, 2024-03-28T22:48:11.241 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_extent_items": 0, 2024-03-28T22:48:11.241 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_blob_bytes": 0, 2024-03-28T22:48:11.241 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_blob_items": 0, 2024-03-28T22:48:11.241 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T22:48:11.241 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T22:48:11.242 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T22:48:11.242 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T22:48:11.242 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T22:48:11.242 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_fsck_items": 0, 2024-03-28T22:48:11.242 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_txc_bytes": 0, 2024-03-28T22:48:11.242 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_txc_items": 0, 2024-03-28T22:48:11.242 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T22:48:11.242 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T22:48:11.243 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_writing_bytes": 0, 2024-03-28T22:48:11.243 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_writing_items": 0, 2024-03-28T22:48:11.243 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_bytes": 0, 2024-03-28T22:48:11.243 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_items": 0, 2024-03-28T22:48:11.243 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T22:48:11.243 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_file_reader_items": 0, 2024-03-28T22:48:11.243 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T22:48:11.244 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_file_writer_items": 0, 2024-03-28T22:48:11.244 INFO:teuthology.orchestra.run.smithi082.stdout: "buffer_anon_bytes": 336372, 2024-03-28T22:48:11.244 INFO:teuthology.orchestra.run.smithi082.stdout: "buffer_anon_items": 354, 2024-03-28T22:48:11.244 INFO:teuthology.orchestra.run.smithi082.stdout: "buffer_meta_bytes": 528, 2024-03-28T22:48:11.244 INFO:teuthology.orchestra.run.smithi082.stdout: "buffer_meta_items": 6, 2024-03-28T22:48:11.244 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_bytes": 0, 2024-03-28T22:48:11.244 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_items": 0, 2024-03-28T22:48:11.244 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_mapbl_bytes": 0, 2024-03-28T22:48:11.244 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_mapbl_items": 0, 2024-03-28T22:48:11.245 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_pglog_bytes": 0, 2024-03-28T22:48:11.245 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_pglog_items": 0, 2024-03-28T22:48:11.245 INFO:teuthology.orchestra.run.smithi082.stdout: "osdmap_bytes": 9736, 2024-03-28T22:48:11.245 INFO:teuthology.orchestra.run.smithi082.stdout: "osdmap_items": 203, 2024-03-28T22:48:11.245 INFO:teuthology.orchestra.run.smithi082.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T22:48:11.245 INFO:teuthology.orchestra.run.smithi082.stdout: "osdmap_mapping_items": 0, 2024-03-28T22:48:11.245 INFO:teuthology.orchestra.run.smithi082.stdout: "pgmap_bytes": 0, 2024-03-28T22:48:11.245 INFO:teuthology.orchestra.run.smithi082.stdout: "pgmap_items": 0, 2024-03-28T22:48:11.245 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_co_bytes": 2070517, 2024-03-28T22:48:11.246 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_co_items": 19178, 2024-03-28T22:48:11.246 INFO:teuthology.orchestra.run.smithi082.stdout: "unittest_1_bytes": 0, 2024-03-28T22:48:11.246 INFO:teuthology.orchestra.run.smithi082.stdout: "unittest_1_items": 0, 2024-03-28T22:48:11.246 INFO:teuthology.orchestra.run.smithi082.stdout: "unittest_2_bytes": 0, 2024-03-28T22:48:11.246 INFO:teuthology.orchestra.run.smithi082.stdout: "unittest_2_items": 0 2024-03-28T22:48:11.246 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.246 INFO:teuthology.orchestra.run.smithi082.stdout: "objecter": { 2024-03-28T22:48:11.246 INFO:teuthology.orchestra.run.smithi082.stdout: "op_active": 0, 2024-03-28T22:48:11.247 INFO:teuthology.orchestra.run.smithi082.stdout: "op_laggy": 0, 2024-03-28T22:48:11.247 INFO:teuthology.orchestra.run.smithi082.stdout: "op_send": 80741, 2024-03-28T22:48:11.247 INFO:teuthology.orchestra.run.smithi082.stdout: "op_send_bytes": 257001601, 2024-03-28T22:48:11.247 INFO:teuthology.orchestra.run.smithi082.stdout: "op_resend": 32, 2024-03-28T22:48:11.247 INFO:teuthology.orchestra.run.smithi082.stdout: "op_reply": 80741, 2024-03-28T22:48:11.247 INFO:teuthology.orchestra.run.smithi082.stdout: "op_latency": { 2024-03-28T22:48:11.247 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 80741, 2024-03-28T22:48:11.247 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 2024.737058796, 2024-03-28T22:48:11.248 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.025076938 2024-03-28T22:48:11.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.248 INFO:teuthology.orchestra.run.smithi082.stdout: "op_inflight": 0, 2024-03-28T22:48:11.248 INFO:teuthology.orchestra.run.smithi082.stdout: "oplen_avg": { 2024-03-28T22:48:11.248 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 80741, 2024-03-28T22:48:11.248 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 108838 2024-03-28T22:48:11.249 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.250 INFO:teuthology.orchestra.run.smithi082.stdout: "op": 80741, 2024-03-28T22:48:11.250 INFO:teuthology.orchestra.run.smithi082.stdout: "op_r": 2035, 2024-03-28T22:48:11.250 INFO:teuthology.orchestra.run.smithi082.stdout: "op_w": 78706, 2024-03-28T22:48:11.250 INFO:teuthology.orchestra.run.smithi082.stdout: "op_rmw": 0, 2024-03-28T22:48:11.250 INFO:teuthology.orchestra.run.smithi082.stdout: "op_pg": 0, 2024-03-28T22:48:11.250 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_stat": 912, 2024-03-28T22:48:11.251 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_create": 16675, 2024-03-28T22:48:11.251 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_read": 470, 2024-03-28T22:48:11.251 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_write": 26481, 2024-03-28T22:48:11.251 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_writefull": 369, 2024-03-28T22:48:11.251 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_writesame": 0, 2024-03-28T22:48:11.251 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_append": 0, 2024-03-28T22:48:11.251 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_zero": 0, 2024-03-28T22:48:11.251 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_truncate": 4, 2024-03-28T22:48:11.252 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_delete": 33764, 2024-03-28T22:48:11.252 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_mapext": 0, 2024-03-28T22:48:11.252 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_sparse_read": 0, 2024-03-28T22:48:11.252 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_clonerange": 0, 2024-03-28T22:48:11.252 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_getxattr": 1565, 2024-03-28T22:48:11.252 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_setxattr": 25175, 2024-03-28T22:48:11.252 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_cmpxattr": 0, 2024-03-28T22:48:11.252 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_rmxattr": 0, 2024-03-28T22:48:11.252 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_resetxattrs": 0, 2024-03-28T22:48:11.253 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_call": 0, 2024-03-28T22:48:11.253 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_watch": 0, 2024-03-28T22:48:11.253 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_notify": 0, 2024-03-28T22:48:11.253 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T22:48:11.253 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_pgls": 0, 2024-03-28T22:48:11.253 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_pgls_filter": 0, 2024-03-28T22:48:11.253 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_other": 0, 2024-03-28T22:48:11.253 INFO:teuthology.orchestra.run.smithi082.stdout: "linger_active": 0, 2024-03-28T22:48:11.254 INFO:teuthology.orchestra.run.smithi082.stdout: "linger_send": 0, 2024-03-28T22:48:11.254 INFO:teuthology.orchestra.run.smithi082.stdout: "linger_resend": 0, 2024-03-28T22:48:11.254 INFO:teuthology.orchestra.run.smithi082.stdout: "linger_ping": 0, 2024-03-28T22:48:11.254 INFO:teuthology.orchestra.run.smithi082.stdout: "poolop_active": 0, 2024-03-28T22:48:11.254 INFO:teuthology.orchestra.run.smithi082.stdout: "poolop_send": 0, 2024-03-28T22:48:11.254 INFO:teuthology.orchestra.run.smithi082.stdout: "poolop_resend": 0, 2024-03-28T22:48:11.254 INFO:teuthology.orchestra.run.smithi082.stdout: "poolstat_active": 0, 2024-03-28T22:48:11.254 INFO:teuthology.orchestra.run.smithi082.stdout: "poolstat_send": 0, 2024-03-28T22:48:11.255 INFO:teuthology.orchestra.run.smithi082.stdout: "poolstat_resend": 0, 2024-03-28T22:48:11.255 INFO:teuthology.orchestra.run.smithi082.stdout: "statfs_active": 0, 2024-03-28T22:48:11.255 INFO:teuthology.orchestra.run.smithi082.stdout: "statfs_send": 0, 2024-03-28T22:48:11.255 INFO:teuthology.orchestra.run.smithi082.stdout: "statfs_resend": 0, 2024-03-28T22:48:11.255 INFO:teuthology.orchestra.run.smithi082.stdout: "command_active": 0, 2024-03-28T22:48:11.255 INFO:teuthology.orchestra.run.smithi082.stdout: "command_send": 0, 2024-03-28T22:48:11.255 INFO:teuthology.orchestra.run.smithi082.stdout: "command_resend": 0, 2024-03-28T22:48:11.255 INFO:teuthology.orchestra.run.smithi082.stdout: "map_epoch": 109, 2024-03-28T22:48:11.256 INFO:teuthology.orchestra.run.smithi082.stdout: "map_full": 0, 2024-03-28T22:48:11.256 INFO:teuthology.orchestra.run.smithi082.stdout: "map_inc": 23, 2024-03-28T22:48:11.256 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_sessions": 12, 2024-03-28T22:48:11.256 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_session_open": 12, 2024-03-28T22:48:11.256 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_session_close": 0, 2024-03-28T22:48:11.256 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_laggy": 0, 2024-03-28T22:48:11.256 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_wr": 2583, 2024-03-28T22:48:11.256 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_rd": 0, 2024-03-28T22:48:11.256 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_del": 840 2024-03-28T22:48:11.257 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.257 INFO:teuthology.orchestra.run.smithi082.stdout: "oft": { 2024-03-28T22:48:11.257 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_total_objs": 1, 2024-03-28T22:48:11.257 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_total_kv_pairs": 0, 2024-03-28T22:48:11.257 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_total_updates": 0, 2024-03-28T22:48:11.257 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_total_removes": 0 2024-03-28T22:48:11.257 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.257 INFO:teuthology.orchestra.run.smithi082.stdout: "osd-slow-ops": { 2024-03-28T22:48:11.257 INFO:teuthology.orchestra.run.smithi082.stdout: "slow_ops_count": 0 2024-03-28T22:48:11.258 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.258 INFO:teuthology.orchestra.run.smithi082.stdout: "purge_queue": { 2024-03-28T22:48:11.258 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executing_ops": 0, 2024-03-28T22:48:11.258 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executing_ops_high_water": 10, 2024-03-28T22:48:11.258 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executing": 0, 2024-03-28T22:48:11.258 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executing_high_water": 5, 2024-03-28T22:48:11.258 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executed_ops": 33116, 2024-03-28T22:48:11.258 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executed": 16558, 2024-03-28T22:48:11.259 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_item_in_journal": 0 2024-03-28T22:48:11.259 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.259 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T22:48:11.259 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-28T22:48:11.259 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 104857600, 2024-03-28T22:48:11.259 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-28T22:48:11.259 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 362126, 2024-03-28T22:48:11.259 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 62993980, 2024-03-28T22:48:11.259 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:11.260 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 362126, 2024-03-28T22:48:11.260 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 0, 2024-03-28T22:48:11.260 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 0, 2024-03-28T22:48:11.260 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 362126, 2024-03-28T22:48:11.260 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 62993980, 2024-03-28T22:48:11.260 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-28T22:48:11.260 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:11.260 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:11.260 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:11.261 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:11.261 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.261 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-objecter_bytes": { 2024-03-28T22:48:11.261 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-28T22:48:11.261 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 104857600, 2024-03-28T22:48:11.261 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-28T22:48:11.261 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 0, 2024-03-28T22:48:11.261 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 0, 2024-03-28T22:48:11.262 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:11.262 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 0, 2024-03-28T22:48:11.262 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 80741, 2024-03-28T22:48:11.262 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 258809351, 2024-03-28T22:48:11.262 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 80741, 2024-03-28T22:48:11.262 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 258809351, 2024-03-28T22:48:11.262 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-28T22:48:11.262 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:11.262 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:11.263 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:11.263 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:11.263 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.263 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-objecter_ops": { 2024-03-28T22:48:11.263 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-28T22:48:11.263 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 1024, 2024-03-28T22:48:11.263 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-28T22:48:11.263 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 0, 2024-03-28T22:48:11.263 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 0, 2024-03-28T22:48:11.264 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:11.264 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 0, 2024-03-28T22:48:11.264 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 80741, 2024-03-28T22:48:11.264 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 80741, 2024-03-28T22:48:11.264 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 80741, 2024-03-28T22:48:11.264 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 80741, 2024-03-28T22:48:11.264 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-28T22:48:11.264 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:11.265 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:11.265 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:11.265 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:11.265 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.265 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-write_buf_throttle": { 2024-03-28T22:48:11.265 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-28T22:48:11.265 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 3758096384, 2024-03-28T22:48:11.265 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-28T22:48:11.265 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 16558, 2024-03-28T22:48:11.266 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 1807750, 2024-03-28T22:48:11.266 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:11.266 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 16558, 2024-03-28T22:48:11.266 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 0, 2024-03-28T22:48:11.266 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 0, 2024-03-28T22:48:11.266 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 450, 2024-03-28T22:48:11.266 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 1807750, 2024-03-28T22:48:11.266 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-28T22:48:11.267 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:11.267 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:11.267 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:11.267 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:11.267 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:11.267 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-write_buf_throttle-0x5579204d8500": { 2024-03-28T22:48:11.267 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 136, 2024-03-28T22:48:11.267 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 3758096384, 2024-03-28T22:48:11.267 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-28T22:48:11.268 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 75874, 2024-03-28T22:48:11.268 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 242255014, 2024-03-28T22:48:11.268 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:11.268 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 75874, 2024-03-28T22:48:11.268 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 0, 2024-03-28T22:48:11.268 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 0, 2024-03-28T22:48:11.268 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 26031, 2024-03-28T22:48:11.268 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 242254878, 2024-03-28T22:48:11.268 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-28T22:48:11.269 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:11.269 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:11.269 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:11.269 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:11.269 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:11.269 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-28T22:48:11.270 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:11 smithi043 ceph-mon[31823]: pgmap v344: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-28T22:48:11.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:11 smithi172 ceph-mon[36566]: pgmap v344: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-28T22:48:12.024 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-03-28T22:48:12.025 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-03-28T22:48:12.025 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=48 2024-03-28T22:48:12.025 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=0 2024-03-28T22:48:12.025 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-03-28T22:48:12.025 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=11 2024-03-28T22:48:12.025 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=5 2024-03-28T22:48:12.025 DEBUG:tasks.check_counter:Getting stats from e 2024-03-28T22:48:12.026 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-28T22:48:12.323 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T22:48:13.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:13 smithi043 ceph-mon[31823]: pgmap v345: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-28T22:48:13.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:13 smithi172 ceph-mon[36566]: pgmap v345: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-28T22:48:13.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:13 smithi082 ceph-mon[36471]: pgmap v345: 129 pgs: 129 active+clean; 59 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-28T22:48:14.148 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-28T22:48:14.148 INFO:teuthology.orchestra.run.smithi082.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T22:48:14.148 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_messages": 197, 2024-03-28T22:48:14.149 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages": 197, 2024-03-28T22:48:14.149 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_bytes": 41996137, 2024-03-28T22:48:14.149 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_bytes": 61837, 2024-03-28T22:48:14.149 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_created_connections": 8, 2024-03-28T22:48:14.149 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_active_connections": 4, 2024-03-28T22:48:14.149 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_total_time": 0.038918151, 2024-03-28T22:48:14.149 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_send_time": 0.003136255, 2024-03-28T22:48:14.149 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_recv_time": 0.029954778, 2024-03-28T22:48:14.149 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_fast_dispatch_time": 0.003816488, 2024-03-28T22:48:14.150 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:48:14.150 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 196, 2024-03-28T22:48:14.150 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.010257522, 2024-03-28T22:48:14.150 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000052334 2024-03-28T22:48:14.150 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.150 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:48:14.150 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.151 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.151 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.151 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.151 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-28T22:48:14.151 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_encrypted_bytes": 448 2024-03-28T22:48:14.151 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.151 INFO:teuthology.orchestra.run.smithi082.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T22:48:14.151 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_messages": 1369, 2024-03-28T22:48:14.151 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages": 1368, 2024-03-28T22:48:14.152 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_bytes": 100208433, 2024-03-28T22:48:14.152 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_bytes": 471461, 2024-03-28T22:48:14.152 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_created_connections": 6, 2024-03-28T22:48:14.152 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_active_connections": 5, 2024-03-28T22:48:14.152 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_total_time": 0.139764953, 2024-03-28T22:48:14.152 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_send_time": 0.031344415, 2024-03-28T22:48:14.152 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_recv_time": 0.075268666, 2024-03-28T22:48:14.152 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_fast_dispatch_time": 0.027575280, 2024-03-28T22:48:14.152 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:48:14.153 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 1367, 2024-03-28T22:48:14.153 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.029359337, 2024-03-28T22:48:14.153 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000021477 2024-03-28T22:48:14.153 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.153 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:48:14.153 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.153 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.154 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.154 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_encrypted_bytes": 203056, 2024-03-28T22:48:14.154 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_encrypted_bytes": 83440 2024-03-28T22:48:14.154 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.154 INFO:teuthology.orchestra.run.smithi082.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T22:48:14.154 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_messages": 439, 2024-03-28T22:48:14.154 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages": 539, 2024-03-28T22:48:14.155 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_bytes": 41710212, 2024-03-28T22:48:14.155 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_bytes": 400789, 2024-03-28T22:48:14.155 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_created_connections": 6, 2024-03-28T22:48:14.155 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_active_connections": 5, 2024-03-28T22:48:14.155 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_total_time": 0.058279605, 2024-03-28T22:48:14.155 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_send_time": 0.010889737, 2024-03-28T22:48:14.155 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_recv_time": 0.033569590, 2024-03-28T22:48:14.155 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_fast_dispatch_time": 0.010610225, 2024-03-28T22:48:14.155 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:48:14.156 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 537, 2024-03-28T22:48:14.156 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.023161179, 2024-03-28T22:48:14.156 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000043130 2024-03-28T22:48:14.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.156 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:48:14.156 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.156 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.156 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.157 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-28T22:48:14.157 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_encrypted_bytes": 261504 2024-03-28T22:48:14.157 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.157 INFO:teuthology.orchestra.run.smithi082.stdout: "cct": { 2024-03-28T22:48:14.157 INFO:teuthology.orchestra.run.smithi082.stdout: "total_workers": 1, 2024-03-28T22:48:14.157 INFO:teuthology.orchestra.run.smithi082.stdout: "unhealthy_workers": 0 2024-03-28T22:48:14.157 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.157 INFO:teuthology.orchestra.run.smithi082.stdout: "finisher-MDSRank": { 2024-03-28T22:48:14.157 INFO:teuthology.orchestra.run.smithi082.stdout: "queue_len": 0, 2024-03-28T22:48:14.157 INFO:teuthology.orchestra.run.smithi082.stdout: "complete_latency": { 2024-03-28T22:48:14.158 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 3053, 2024-03-28T22:48:14.158 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.147148822, 2024-03-28T22:48:14.158 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000048198 2024-03-28T22:48:14.158 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:14.158 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.158 INFO:teuthology.orchestra.run.smithi082.stdout: "finisher-PurgeQueue": { 2024-03-28T22:48:14.158 INFO:teuthology.orchestra.run.smithi082.stdout: "queue_len": 0, 2024-03-28T22:48:14.158 INFO:teuthology.orchestra.run.smithi082.stdout: "complete_latency": { 2024-03-28T22:48:14.159 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.159 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.159 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.159 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:14.159 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.159 INFO:teuthology.orchestra.run.smithi082.stdout: "mds": { 2024-03-28T22:48:14.159 INFO:teuthology.orchestra.run.smithi082.stdout: "request": 0, 2024-03-28T22:48:14.159 INFO:teuthology.orchestra.run.smithi082.stdout: "reply": 0, 2024-03-28T22:48:14.159 INFO:teuthology.orchestra.run.smithi082.stdout: "reply_latency": { 2024-03-28T22:48:14.160 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.160 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.160 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.160 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.160 INFO:teuthology.orchestra.run.smithi082.stdout: "slow_reply": 0, 2024-03-28T22:48:14.160 INFO:teuthology.orchestra.run.smithi082.stdout: "forward": 0, 2024-03-28T22:48:14.160 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_fetch_complete": 0, 2024-03-28T22:48:14.160 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_fetch_keys": 0, 2024-03-28T22:48:14.160 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_commit": 0, 2024-03-28T22:48:14.161 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_split": 0, 2024-03-28T22:48:14.161 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_merge": 0, 2024-03-28T22:48:14.161 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes": 12304, 2024-03-28T22:48:14.161 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_top": 7743, 2024-03-28T22:48:14.161 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_bottom": 4561, 2024-03-28T22:48:14.161 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_pin_tail": 0, 2024-03-28T22:48:14.161 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_pinned": 1243, 2024-03-28T22:48:14.161 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_expired": 0, 2024-03-28T22:48:14.161 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_with_caps": 0, 2024-03-28T22:48:14.162 INFO:teuthology.orchestra.run.smithi082.stdout: "caps": 0, 2024-03-28T22:48:14.162 INFO:teuthology.orchestra.run.smithi082.stdout: "subtrees": 2, 2024-03-28T22:48:14.162 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse": 0, 2024-03-28T22:48:14.162 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_hit": 0, 2024-03-28T22:48:14.162 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_forward": 0, 2024-03-28T22:48:14.162 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_discover": 0, 2024-03-28T22:48:14.162 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_dir_fetch": 0, 2024-03-28T22:48:14.162 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_remote_ino": 0, 2024-03-28T22:48:14.163 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_lock": 0, 2024-03-28T22:48:14.163 INFO:teuthology.orchestra.run.smithi082.stdout: "load_cent": 0, 2024-03-28T22:48:14.163 INFO:teuthology.orchestra.run.smithi082.stdout: "q": 0, 2024-03-28T22:48:14.163 INFO:teuthology.orchestra.run.smithi082.stdout: "exported": 0, 2024-03-28T22:48:14.163 INFO:teuthology.orchestra.run.smithi082.stdout: "exported_inodes": 0, 2024-03-28T22:48:14.163 INFO:teuthology.orchestra.run.smithi082.stdout: "imported": 0, 2024-03-28T22:48:14.163 INFO:teuthology.orchestra.run.smithi082.stdout: "imported_inodes": 0, 2024-03-28T22:48:14.163 INFO:teuthology.orchestra.run.smithi082.stdout: "openino_dir_fetch": 0, 2024-03-28T22:48:14.163 INFO:teuthology.orchestra.run.smithi082.stdout: "openino_backtrace_fetch": 0, 2024-03-28T22:48:14.164 INFO:teuthology.orchestra.run.smithi082.stdout: "openino_peer_discover": 0, 2024-03-28T22:48:14.164 INFO:teuthology.orchestra.run.smithi082.stdout: "root_rfiles": 61, 2024-03-28T22:48:14.164 INFO:teuthology.orchestra.run.smithi082.stdout: "root_rbytes": 21056780, 2024-03-28T22:48:14.164 INFO:teuthology.orchestra.run.smithi082.stdout: "root_rsnaps": 6, 2024-03-28T22:48:14.164 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T22:48:14.164 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_set_tag": 0, 2024-03-28T22:48:14.164 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T22:48:14.164 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_inotable_repaired": 0, 2024-03-28T22:48:14.164 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_dir_inodes": 0, 2024-03-28T22:48:14.165 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T22:48:14.165 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T22:48:14.165 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_file_inodes": 0, 2024-03-28T22:48:14.165 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_inode_file_caps": 0, 2024-03-28T22:48:14.165 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T22:48:14.165 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_grant": 0, 2024-03-28T22:48:14.165 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T22:48:14.165 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T22:48:14.165 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T22:48:14.166 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_caps": 0, 2024-03-28T22:48:14.166 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_caps_dirty": 0, 2024-03-28T22:48:14.166 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_cap_release": 0, 2024-03-28T22:48:14.166 INFO:teuthology.orchestra.run.smithi082.stdout: "process_request_cap_release": 0 2024-03-28T22:48:14.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.166 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_cache": { 2024-03-28T22:48:14.166 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_update": 0, 2024-03-28T22:48:14.166 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_update_receipt": 0, 2024-03-28T22:48:14.166 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_try_discover": 0, 2024-03-28T22:48:14.167 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_send_discover": 0, 2024-03-28T22:48:14.167 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_handle_discover": 0, 2024-03-28T22:48:14.167 INFO:teuthology.orchestra.run.smithi082.stdout: "num_strays": 0, 2024-03-28T22:48:14.167 INFO:teuthology.orchestra.run.smithi082.stdout: "num_strays_delayed": 0, 2024-03-28T22:48:14.167 INFO:teuthology.orchestra.run.smithi082.stdout: "num_strays_enqueuing": 0, 2024-03-28T22:48:14.167 INFO:teuthology.orchestra.run.smithi082.stdout: "strays_created": 0, 2024-03-28T22:48:14.167 INFO:teuthology.orchestra.run.smithi082.stdout: "strays_enqueued": 0, 2024-03-28T22:48:14.167 INFO:teuthology.orchestra.run.smithi082.stdout: "strays_reintegrated": 0, 2024-03-28T22:48:14.168 INFO:teuthology.orchestra.run.smithi082.stdout: "strays_migrated": 0, 2024-03-28T22:48:14.168 INFO:teuthology.orchestra.run.smithi082.stdout: "num_recovering_processing": 0, 2024-03-28T22:48:14.168 INFO:teuthology.orchestra.run.smithi082.stdout: "num_recovering_enqueued": 0, 2024-03-28T22:48:14.168 INFO:teuthology.orchestra.run.smithi082.stdout: "num_recovering_prioritized": 0, 2024-03-28T22:48:14.168 INFO:teuthology.orchestra.run.smithi082.stdout: "recovery_started": 0, 2024-03-28T22:48:14.168 INFO:teuthology.orchestra.run.smithi082.stdout: "recovery_completed": 0, 2024-03-28T22:48:14.168 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_quiesce_path": 0, 2024-03-28T22:48:14.168 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_quiesce_inode": 0, 2024-03-28T22:48:14.168 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T22:48:14.168 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_exportdir": 0, 2024-03-28T22:48:14.169 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_flush": 0, 2024-03-28T22:48:14.169 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_fragmentdir": 0, 2024-03-28T22:48:14.169 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_fragstats": 0, 2024-03-28T22:48:14.169 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_inodestats": 0 2024-03-28T22:48:14.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.169 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_log": { 2024-03-28T22:48:14.169 INFO:teuthology.orchestra.run.smithi082.stdout: "evlrg": 0, 2024-03-28T22:48:14.169 INFO:teuthology.orchestra.run.smithi082.stdout: "evadd": 0, 2024-03-28T22:48:14.170 INFO:teuthology.orchestra.run.smithi082.stdout: "evex": 0, 2024-03-28T22:48:14.170 INFO:teuthology.orchestra.run.smithi082.stdout: "evtrm": 0, 2024-03-28T22:48:14.170 INFO:teuthology.orchestra.run.smithi082.stdout: "ev": 52840, 2024-03-28T22:48:14.170 INFO:teuthology.orchestra.run.smithi082.stdout: "evexg": 0, 2024-03-28T22:48:14.170 INFO:teuthology.orchestra.run.smithi082.stdout: "evexd": 0, 2024-03-28T22:48:14.170 INFO:teuthology.orchestra.run.smithi082.stdout: "segadd": 0, 2024-03-28T22:48:14.170 INFO:teuthology.orchestra.run.smithi082.stdout: "segex": 0, 2024-03-28T22:48:14.170 INFO:teuthology.orchestra.run.smithi082.stdout: "segtrm": 0, 2024-03-28T22:48:14.170 INFO:teuthology.orchestra.run.smithi082.stdout: "seg": 4, 2024-03-28T22:48:14.171 INFO:teuthology.orchestra.run.smithi082.stdout: "segmjr": 18, 2024-03-28T22:48:14.171 INFO:teuthology.orchestra.run.smithi082.stdout: "segexg": 0, 2024-03-28T22:48:14.171 INFO:teuthology.orchestra.run.smithi082.stdout: "segexd": 0, 2024-03-28T22:48:14.171 INFO:teuthology.orchestra.run.smithi082.stdout: "expos": 235966387, 2024-03-28T22:48:14.171 INFO:teuthology.orchestra.run.smithi082.stdout: "wrpos": 246449318, 2024-03-28T22:48:14.171 INFO:teuthology.orchestra.run.smithi082.stdout: "rdpos": 246449182, 2024-03-28T22:48:14.171 INFO:teuthology.orchestra.run.smithi082.stdout: "jlat": { 2024-03-28T22:48:14.171 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.172 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.172 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.172 INFO:teuthology.orchestra.run.smithi082.stdout: "replayed": 52840 2024-03-28T22:48:14.172 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.172 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_mem": { 2024-03-28T22:48:14.172 INFO:teuthology.orchestra.run.smithi082.stdout: "ino": 337, 2024-03-28T22:48:14.172 INFO:teuthology.orchestra.run.smithi082.stdout: "ino+": 12184, 2024-03-28T22:48:14.172 INFO:teuthology.orchestra.run.smithi082.stdout: "ino-": 11847, 2024-03-28T22:48:14.173 INFO:teuthology.orchestra.run.smithi082.stdout: "dir": 40, 2024-03-28T22:48:14.173 INFO:teuthology.orchestra.run.smithi082.stdout: "dir+": 65, 2024-03-28T22:48:14.173 INFO:teuthology.orchestra.run.smithi082.stdout: "dir-": 25, 2024-03-28T22:48:14.173 INFO:teuthology.orchestra.run.smithi082.stdout: "dn": 12304, 2024-03-28T22:48:14.173 INFO:teuthology.orchestra.run.smithi082.stdout: "dn+": 12304, 2024-03-28T22:48:14.173 INFO:teuthology.orchestra.run.smithi082.stdout: "dn-": 0, 2024-03-28T22:48:14.173 INFO:teuthology.orchestra.run.smithi082.stdout: "cap": 0, 2024-03-28T22:48:14.173 INFO:teuthology.orchestra.run.smithi082.stdout: "cap+": 0, 2024-03-28T22:48:14.173 INFO:teuthology.orchestra.run.smithi082.stdout: "cap-": 0, 2024-03-28T22:48:14.174 INFO:teuthology.orchestra.run.smithi082.stdout: "rss": 62160, 2024-03-28T22:48:14.174 INFO:teuthology.orchestra.run.smithi082.stdout: "heap": 215396 2024-03-28T22:48:14.174 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.174 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_server": { 2024-03-28T22:48:14.174 INFO:teuthology.orchestra.run.smithi082.stdout: "dispatch_client_request": 0, 2024-03-28T22:48:14.174 INFO:teuthology.orchestra.run.smithi082.stdout: "dispatch_server_request": 0, 2024-03-28T22:48:14.174 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_request": 0, 2024-03-28T22:48:14.174 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_session": 0, 2024-03-28T22:48:14.175 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_peer_request": 0, 2024-03-28T22:48:14.175 INFO:teuthology.orchestra.run.smithi082.stdout: "req_create_latency": { 2024-03-28T22:48:14.175 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.175 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.175 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.175 INFO:teuthology.orchestra.run.smithi082.stdout: "req_getattr_latency": { 2024-03-28T22:48:14.175 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.175 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.176 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.176 INFO:teuthology.orchestra.run.smithi082.stdout: "req_getfilelock_latency": { 2024-03-28T22:48:14.176 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.176 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.176 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.176 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.176 INFO:teuthology.orchestra.run.smithi082.stdout: "req_link_latency": { 2024-03-28T22:48:14.176 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.177 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.177 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.177 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookup_latency": { 2024-03-28T22:48:14.177 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.177 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.177 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.177 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.177 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookuphash_latency": { 2024-03-28T22:48:14.178 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.178 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.178 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.178 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.178 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookupino_latency": { 2024-03-28T22:48:14.178 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.178 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.178 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.179 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookupname_latency": { 2024-03-28T22:48:14.179 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.179 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.179 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.179 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.179 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookupparent_latency": { 2024-03-28T22:48:14.179 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.179 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.180 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.180 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookupsnap_latency": { 2024-03-28T22:48:14.180 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.180 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.180 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.180 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.180 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lssnap_latency": { 2024-03-28T22:48:14.181 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.181 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.181 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.181 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.182 INFO:teuthology.orchestra.run.smithi082.stdout: "req_mkdir_latency": { 2024-03-28T22:48:14.182 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.182 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.182 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.182 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.182 INFO:teuthology.orchestra.run.smithi082.stdout: "req_mknod_latency": { 2024-03-28T22:48:14.183 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.183 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.183 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.183 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.183 INFO:teuthology.orchestra.run.smithi082.stdout: "req_mksnap_latency": { 2024-03-28T22:48:14.183 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.183 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.184 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.184 INFO:teuthology.orchestra.run.smithi082.stdout: "req_open_latency": { 2024-03-28T22:48:14.184 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.184 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.184 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.184 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.184 INFO:teuthology.orchestra.run.smithi082.stdout: "req_readdir_latency": { 2024-03-28T22:48:14.184 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.185 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.185 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.185 INFO:teuthology.orchestra.run.smithi082.stdout: "req_rename_latency": { 2024-03-28T22:48:14.185 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.185 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.185 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.185 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.186 INFO:teuthology.orchestra.run.smithi082.stdout: "req_renamesnap_latency": { 2024-03-28T22:48:14.186 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.186 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.186 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.186 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.186 INFO:teuthology.orchestra.run.smithi082.stdout: "req_snapdiff_latency": { 2024-03-28T22:48:14.186 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.186 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.187 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.187 INFO:teuthology.orchestra.run.smithi082.stdout: "req_rmdir_latency": { 2024-03-28T22:48:14.187 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.187 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.187 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.187 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.187 INFO:teuthology.orchestra.run.smithi082.stdout: "req_rmsnap_latency": { 2024-03-28T22:48:14.187 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.188 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.188 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.188 INFO:teuthology.orchestra.run.smithi082.stdout: "req_rmxattr_latency": { 2024-03-28T22:48:14.188 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.188 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.188 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.188 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.188 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setattr_latency": { 2024-03-28T22:48:14.189 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.189 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.189 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.189 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.189 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setdirlayout_latency": { 2024-03-28T22:48:14.189 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.189 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.189 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.190 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setfilelock_latency": { 2024-03-28T22:48:14.190 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.190 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.190 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.190 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.190 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setlayout_latency": { 2024-03-28T22:48:14.190 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.190 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.191 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.191 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setxattr_latency": { 2024-03-28T22:48:14.191 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.191 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.191 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.191 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.191 INFO:teuthology.orchestra.run.smithi082.stdout: "req_symlink_latency": { 2024-03-28T22:48:14.192 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.192 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.192 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.192 INFO:teuthology.orchestra.run.smithi082.stdout: "req_unlink_latency": { 2024-03-28T22:48:14.192 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.192 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.192 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.192 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.193 INFO:teuthology.orchestra.run.smithi082.stdout: "cap_revoke_eviction": 0, 2024-03-28T22:48:14.193 INFO:teuthology.orchestra.run.smithi082.stdout: "cap_acquisition_throttle": 0, 2024-03-28T22:48:14.193 INFO:teuthology.orchestra.run.smithi082.stdout: "req_getvxattr_latency": { 2024-03-28T22:48:14.193 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.193 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.193 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.193 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:14.193 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.194 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_sessions": { 2024-03-28T22:48:14.194 INFO:teuthology.orchestra.run.smithi082.stdout: "session_count": 2, 2024-03-28T22:48:14.194 INFO:teuthology.orchestra.run.smithi082.stdout: "session_add": 2, 2024-03-28T22:48:14.194 INFO:teuthology.orchestra.run.smithi082.stdout: "session_remove": 0, 2024-03-28T22:48:14.194 INFO:teuthology.orchestra.run.smithi082.stdout: "sessions_open": 0, 2024-03-28T22:48:14.194 INFO:teuthology.orchestra.run.smithi082.stdout: "sessions_stale": 0, 2024-03-28T22:48:14.194 INFO:teuthology.orchestra.run.smithi082.stdout: "total_load": 0, 2024-03-28T22:48:14.194 INFO:teuthology.orchestra.run.smithi082.stdout: "average_load": 0, 2024-03-28T22:48:14.195 INFO:teuthology.orchestra.run.smithi082.stdout: "avg_session_uptime": 2331, 2024-03-28T22:48:14.195 INFO:teuthology.orchestra.run.smithi082.stdout: "mdthresh_evicted": 0 2024-03-28T22:48:14.195 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.195 INFO:teuthology.orchestra.run.smithi082.stdout: "mempool": { 2024-03-28T22:48:14.195 INFO:teuthology.orchestra.run.smithi082.stdout: "bloom_filter_bytes": 0, 2024-03-28T22:48:14.195 INFO:teuthology.orchestra.run.smithi082.stdout: "bloom_filter_items": 0, 2024-03-28T22:48:14.195 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T22:48:14.196 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_alloc_items": 0, 2024-03-28T22:48:14.196 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T22:48:14.196 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_data_items": 0, 2024-03-28T22:48:14.196 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T22:48:14.196 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T22:48:14.196 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T22:48:14.196 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T22:48:14.196 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T22:48:14.197 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_other_items": 0, 2024-03-28T22:48:14.197 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T22:48:14.197 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T22:48:14.197 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_extent_bytes": 0, 2024-03-28T22:48:14.197 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_extent_items": 0, 2024-03-28T22:48:14.197 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_blob_bytes": 0, 2024-03-28T22:48:14.197 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_blob_items": 0, 2024-03-28T22:48:14.197 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T22:48:14.197 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T22:48:14.198 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T22:48:14.198 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T22:48:14.198 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T22:48:14.198 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_fsck_items": 0, 2024-03-28T22:48:14.198 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_txc_bytes": 0, 2024-03-28T22:48:14.198 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_txc_items": 0, 2024-03-28T22:48:14.198 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T22:48:14.198 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T22:48:14.199 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_writing_bytes": 0, 2024-03-28T22:48:14.199 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_writing_items": 0, 2024-03-28T22:48:14.199 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_bytes": 0, 2024-03-28T22:48:14.199 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_items": 0, 2024-03-28T22:48:14.199 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T22:48:14.199 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_file_reader_items": 0, 2024-03-28T22:48:14.199 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T22:48:14.199 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_file_writer_items": 0, 2024-03-28T22:48:14.199 INFO:teuthology.orchestra.run.smithi082.stdout: "buffer_anon_bytes": 252719, 2024-03-28T22:48:14.200 INFO:teuthology.orchestra.run.smithi082.stdout: "buffer_anon_items": 478, 2024-03-28T22:48:14.200 INFO:teuthology.orchestra.run.smithi082.stdout: "buffer_meta_bytes": 0, 2024-03-28T22:48:14.200 INFO:teuthology.orchestra.run.smithi082.stdout: "buffer_meta_items": 0, 2024-03-28T22:48:14.200 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_bytes": 0, 2024-03-28T22:48:14.200 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_items": 0, 2024-03-28T22:48:14.200 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_mapbl_bytes": 0, 2024-03-28T22:48:14.200 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_mapbl_items": 0, 2024-03-28T22:48:14.200 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_pglog_bytes": 0, 2024-03-28T22:48:14.200 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_pglog_items": 0, 2024-03-28T22:48:14.201 INFO:teuthology.orchestra.run.smithi082.stdout: "osdmap_bytes": 9976, 2024-03-28T22:48:14.201 INFO:teuthology.orchestra.run.smithi082.stdout: "osdmap_items": 209, 2024-03-28T22:48:14.201 INFO:teuthology.orchestra.run.smithi082.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T22:48:14.201 INFO:teuthology.orchestra.run.smithi082.stdout: "osdmap_mapping_items": 0, 2024-03-28T22:48:14.201 INFO:teuthology.orchestra.run.smithi082.stdout: "pgmap_bytes": 0, 2024-03-28T22:48:14.201 INFO:teuthology.orchestra.run.smithi082.stdout: "pgmap_items": 0, 2024-03-28T22:48:14.201 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_co_bytes": 9605443, 2024-03-28T22:48:14.201 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_co_items": 58751, 2024-03-28T22:48:14.201 INFO:teuthology.orchestra.run.smithi082.stdout: "unittest_1_bytes": 0, 2024-03-28T22:48:14.202 INFO:teuthology.orchestra.run.smithi082.stdout: "unittest_1_items": 0, 2024-03-28T22:48:14.202 INFO:teuthology.orchestra.run.smithi082.stdout: "unittest_2_bytes": 0, 2024-03-28T22:48:14.202 INFO:teuthology.orchestra.run.smithi082.stdout: "unittest_2_items": 0 2024-03-28T22:48:14.202 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.202 INFO:teuthology.orchestra.run.smithi082.stdout: "objecter": { 2024-03-28T22:48:14.202 INFO:teuthology.orchestra.run.smithi082.stdout: "op_active": 0, 2024-03-28T22:48:14.202 INFO:teuthology.orchestra.run.smithi082.stdout: "op_laggy": 0, 2024-03-28T22:48:14.202 INFO:teuthology.orchestra.run.smithi082.stdout: "op_send": 1851, 2024-03-28T22:48:14.203 INFO:teuthology.orchestra.run.smithi082.stdout: "op_send_bytes": 26, 2024-03-28T22:48:14.203 INFO:teuthology.orchestra.run.smithi082.stdout: "op_resend": 4, 2024-03-28T22:48:14.203 INFO:teuthology.orchestra.run.smithi082.stdout: "op_reply": 1851, 2024-03-28T22:48:14.203 INFO:teuthology.orchestra.run.smithi082.stdout: "op_latency": { 2024-03-28T22:48:14.203 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 1851, 2024-03-28T22:48:14.203 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 1.708969343, 2024-03-28T22:48:14.203 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000923268 2024-03-28T22:48:14.203 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.203 INFO:teuthology.orchestra.run.smithi082.stdout: "op_inflight": 0, 2024-03-28T22:48:14.204 INFO:teuthology.orchestra.run.smithi082.stdout: "oplen_avg": { 2024-03-28T22:48:14.204 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 1851, 2024-03-28T22:48:14.204 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 1852 2024-03-28T22:48:14.204 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.204 INFO:teuthology.orchestra.run.smithi082.stdout: "op": 1851, 2024-03-28T22:48:14.204 INFO:teuthology.orchestra.run.smithi082.stdout: "op_r": 1851, 2024-03-28T22:48:14.204 INFO:teuthology.orchestra.run.smithi082.stdout: "op_w": 0, 2024-03-28T22:48:14.204 INFO:teuthology.orchestra.run.smithi082.stdout: "op_rmw": 0, 2024-03-28T22:48:14.204 INFO:teuthology.orchestra.run.smithi082.stdout: "op_pg": 0, 2024-03-28T22:48:14.205 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_stat": 1065, 2024-03-28T22:48:14.205 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_create": 0, 2024-03-28T22:48:14.205 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_read": 783, 2024-03-28T22:48:14.205 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_write": 0, 2024-03-28T22:48:14.205 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_writefull": 0, 2024-03-28T22:48:14.205 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_writesame": 0, 2024-03-28T22:48:14.205 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_append": 0, 2024-03-28T22:48:14.205 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_zero": 0, 2024-03-28T22:48:14.205 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_truncate": 0, 2024-03-28T22:48:14.206 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_delete": 0, 2024-03-28T22:48:14.206 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_mapext": 0, 2024-03-28T22:48:14.206 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_sparse_read": 0, 2024-03-28T22:48:14.206 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_clonerange": 0, 2024-03-28T22:48:14.206 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_getxattr": 2, 2024-03-28T22:48:14.206 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_setxattr": 0, 2024-03-28T22:48:14.206 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_cmpxattr": 0, 2024-03-28T22:48:14.206 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_rmxattr": 0, 2024-03-28T22:48:14.206 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_resetxattrs": 0, 2024-03-28T22:48:14.207 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_call": 0, 2024-03-28T22:48:14.207 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_watch": 0, 2024-03-28T22:48:14.207 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_notify": 0, 2024-03-28T22:48:14.207 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T22:48:14.207 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_pgls": 0, 2024-03-28T22:48:14.207 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_pgls_filter": 0, 2024-03-28T22:48:14.207 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_other": 0, 2024-03-28T22:48:14.207 INFO:teuthology.orchestra.run.smithi082.stdout: "linger_active": 0, 2024-03-28T22:48:14.207 INFO:teuthology.orchestra.run.smithi082.stdout: "linger_send": 0, 2024-03-28T22:48:14.208 INFO:teuthology.orchestra.run.smithi082.stdout: "linger_resend": 0, 2024-03-28T22:48:14.208 INFO:teuthology.orchestra.run.smithi082.stdout: "linger_ping": 0, 2024-03-28T22:48:14.208 INFO:teuthology.orchestra.run.smithi082.stdout: "poolop_active": 0, 2024-03-28T22:48:14.208 INFO:teuthology.orchestra.run.smithi082.stdout: "poolop_send": 0, 2024-03-28T22:48:14.208 INFO:teuthology.orchestra.run.smithi082.stdout: "poolop_resend": 0, 2024-03-28T22:48:14.208 INFO:teuthology.orchestra.run.smithi082.stdout: "poolstat_active": 0, 2024-03-28T22:48:14.208 INFO:teuthology.orchestra.run.smithi082.stdout: "poolstat_send": 0, 2024-03-28T22:48:14.208 INFO:teuthology.orchestra.run.smithi082.stdout: "poolstat_resend": 0, 2024-03-28T22:48:14.208 INFO:teuthology.orchestra.run.smithi082.stdout: "statfs_active": 0, 2024-03-28T22:48:14.209 INFO:teuthology.orchestra.run.smithi082.stdout: "statfs_send": 0, 2024-03-28T22:48:14.209 INFO:teuthology.orchestra.run.smithi082.stdout: "statfs_resend": 0, 2024-03-28T22:48:14.209 INFO:teuthology.orchestra.run.smithi082.stdout: "command_active": 0, 2024-03-28T22:48:14.209 INFO:teuthology.orchestra.run.smithi082.stdout: "command_send": 0, 2024-03-28T22:48:14.209 INFO:teuthology.orchestra.run.smithi082.stdout: "command_resend": 0, 2024-03-28T22:48:14.209 INFO:teuthology.orchestra.run.smithi082.stdout: "map_epoch": 109, 2024-03-28T22:48:14.209 INFO:teuthology.orchestra.run.smithi082.stdout: "map_full": 0, 2024-03-28T22:48:14.209 INFO:teuthology.orchestra.run.smithi082.stdout: "map_inc": 10, 2024-03-28T22:48:14.210 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_sessions": 11, 2024-03-28T22:48:14.210 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_session_open": 11, 2024-03-28T22:48:14.210 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_session_close": 0, 2024-03-28T22:48:14.210 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_laggy": 0, 2024-03-28T22:48:14.210 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_wr": 0, 2024-03-28T22:48:14.210 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_rd": 2, 2024-03-28T22:48:14.210 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_del": 0 2024-03-28T22:48:14.210 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.210 INFO:teuthology.orchestra.run.smithi082.stdout: "oft": { 2024-03-28T22:48:14.211 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_total_objs": 0, 2024-03-28T22:48:14.211 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_total_kv_pairs": 0, 2024-03-28T22:48:14.211 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_total_updates": 0, 2024-03-28T22:48:14.211 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_total_removes": 0 2024-03-28T22:48:14.211 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.211 INFO:teuthology.orchestra.run.smithi082.stdout: "osd-slow-ops": { 2024-03-28T22:48:14.212 INFO:teuthology.orchestra.run.smithi082.stdout: "slow_ops_count": 0 2024-03-28T22:48:14.212 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.212 INFO:teuthology.orchestra.run.smithi082.stdout: "purge_queue": { 2024-03-28T22:48:14.212 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executing_ops": 0, 2024-03-28T22:48:14.212 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T22:48:14.212 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executing": 0, 2024-03-28T22:48:14.213 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executing_high_water": 0, 2024-03-28T22:48:14.213 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executed_ops": 0, 2024-03-28T22:48:14.213 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executed": 0, 2024-03-28T22:48:14.213 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_item_in_journal": 0 2024-03-28T22:48:14.213 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.213 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T22:48:14.213 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-28T22:48:14.213 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 104857600, 2024-03-28T22:48:14.214 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-28T22:48:14.214 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 2001, 2024-03-28T22:48:14.214 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 183723998, 2024-03-28T22:48:14.214 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:14.214 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 2001, 2024-03-28T22:48:14.214 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 0, 2024-03-28T22:48:14.214 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 0, 2024-03-28T22:48:14.215 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 2001, 2024-03-28T22:48:14.215 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 183723998, 2024-03-28T22:48:14.215 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-28T22:48:14.215 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.215 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.215 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.215 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:14.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.216 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-objecter_bytes": { 2024-03-28T22:48:14.216 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-28T22:48:14.216 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 104857600, 2024-03-28T22:48:14.216 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-28T22:48:14.216 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 0, 2024-03-28T22:48:14.216 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 0, 2024-03-28T22:48:14.217 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:14.217 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 0, 2024-03-28T22:48:14.217 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 1851, 2024-03-28T22:48:14.217 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 183193452, 2024-03-28T22:48:14.217 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 1851, 2024-03-28T22:48:14.217 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 183193452, 2024-03-28T22:48:14.217 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-28T22:48:14.218 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.218 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.218 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.218 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:14.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.218 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-objecter_ops": { 2024-03-28T22:48:14.218 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-28T22:48:14.218 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 1024, 2024-03-28T22:48:14.219 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-28T22:48:14.219 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 0, 2024-03-28T22:48:14.219 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 0, 2024-03-28T22:48:14.219 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:14.219 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 0, 2024-03-28T22:48:14.219 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 1851, 2024-03-28T22:48:14.219 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 1851, 2024-03-28T22:48:14.220 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 1851, 2024-03-28T22:48:14.220 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 1851, 2024-03-28T22:48:14.220 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-28T22:48:14.220 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.220 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.220 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.220 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:14.220 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.220 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-write_buf_throttle": { 2024-03-28T22:48:14.221 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-28T22:48:14.221 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 3758096384, 2024-03-28T22:48:14.221 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-28T22:48:14.221 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 0, 2024-03-28T22:48:14.221 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 0, 2024-03-28T22:48:14.221 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:14.221 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 0, 2024-03-28T22:48:14.221 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 0, 2024-03-28T22:48:14.221 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 0, 2024-03-28T22:48:14.222 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 0, 2024-03-28T22:48:14.222 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 0, 2024-03-28T22:48:14.222 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-28T22:48:14.222 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.222 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.222 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.222 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:14.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:14.222 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-write_buf_throttle-0x560583ba06e0": { 2024-03-28T22:48:14.223 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-28T22:48:14.223 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 3758096384, 2024-03-28T22:48:14.223 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-28T22:48:14.223 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 0, 2024-03-28T22:48:14.223 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 0, 2024-03-28T22:48:14.223 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:14.223 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 0, 2024-03-28T22:48:14.223 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 0, 2024-03-28T22:48:14.224 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 0, 2024-03-28T22:48:14.224 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 0, 2024-03-28T22:48:14.224 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 0, 2024-03-28T22:48:14.224 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-28T22:48:14.224 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:14.224 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:14.224 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:14.224 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:14.224 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:14.225 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-28T22:48:15.166 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-28T22:48:15.166 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-28T22:48:15.166 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-03-28T22:48:15.166 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-03-28T22:48:15.166 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=6 2024-03-28T22:48:15.166 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-03-28T22:48:15.166 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T22:48:15.167 DEBUG:tasks.check_counter:Getting stats from h 2024-03-28T22:48:15.167 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-28T22:48:15.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:15 smithi043 ceph-mon[31823]: pgmap v346: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:48:15.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:15 smithi172 ceph-mon[36566]: pgmap v346: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:48:15.424 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T22:48:15.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:15 smithi082 ceph-mon[36471]: pgmap v346: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:48:17.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:17 smithi082 ceph-mon[36471]: pgmap v347: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:48:17.213 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-28T22:48:17.213 INFO:teuthology.orchestra.run.smithi082.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T22:48:17.213 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_messages": 70253, 2024-03-28T22:48:17.213 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages": 35295, 2024-03-28T22:48:17.213 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_bytes": 23254917, 2024-03-28T22:48:17.214 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_bytes": 5770472, 2024-03-28T22:48:17.214 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_created_connections": 14, 2024-03-28T22:48:17.214 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_active_connections": 3, 2024-03-28T22:48:17.214 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_total_time": 3.384703529, 2024-03-28T22:48:17.214 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_send_time": 1.490500010, 2024-03-28T22:48:17.214 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_recv_time": 2.618436189, 2024-03-28T22:48:17.214 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_fast_dispatch_time": 0.002966764, 2024-03-28T22:48:17.214 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:48:17.215 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 35293, 2024-03-28T22:48:17.215 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.503788425, 2024-03-28T22:48:17.215 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000014274 2024-03-28T22:48:17.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.215 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:48:17.215 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 105393, 2024-03-28T22:48:17.215 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.016590653, 2024-03-28T22:48:17.215 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000157 2024-03-28T22:48:17.215 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.216 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T22:48:17.216 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T22:48:17.216 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.216 INFO:teuthology.orchestra.run.smithi082.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T22:48:17.216 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_messages": 361, 2024-03-28T22:48:17.216 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages": 343, 2024-03-28T22:48:17.216 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_bytes": 1928410, 2024-03-28T22:48:17.216 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_bytes": 610004, 2024-03-28T22:48:17.216 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_created_connections": 45, 2024-03-28T22:48:17.217 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_active_connections": 3, 2024-03-28T22:48:17.217 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_total_time": 0.052366506, 2024-03-28T22:48:17.217 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_send_time": 0.011460853, 2024-03-28T22:48:17.217 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_recv_time": 0.050572710, 2024-03-28T22:48:17.217 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_fast_dispatch_time": 0.004429966, 2024-03-28T22:48:17.217 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:48:17.217 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 342, 2024-03-28T22:48:17.217 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.036860083, 2024-03-28T22:48:17.217 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000107778 2024-03-28T22:48:17.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.218 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:48:17.218 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 46, 2024-03-28T22:48:17.218 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000004395, 2024-03-28T22:48:17.218 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000095 2024-03-28T22:48:17.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.218 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_encrypted_bytes": 1920416, 2024-03-28T22:48:17.218 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_encrypted_bytes": 159440 2024-03-28T22:48:17.218 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.219 INFO:teuthology.orchestra.run.smithi082.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T22:48:17.219 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_messages": 31, 2024-03-28T22:48:17.219 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages": 215, 2024-03-28T22:48:17.219 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_bytes": 10057, 2024-03-28T22:48:17.219 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_bytes": 567942, 2024-03-28T22:48:17.219 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_created_connections": 10, 2024-03-28T22:48:17.220 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_active_connections": 4, 2024-03-28T22:48:17.220 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_total_time": 0.010568116, 2024-03-28T22:48:17.220 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_send_time": 0.005567009, 2024-03-28T22:48:17.220 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_recv_time": 0.003291613, 2024-03-28T22:48:17.220 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_fast_dispatch_time": 0.000318300, 2024-03-28T22:48:17.220 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:48:17.220 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 212, 2024-03-28T22:48:17.220 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.038548092, 2024-03-28T22:48:17.220 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000181830 2024-03-28T22:48:17.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.221 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:48:17.221 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 14, 2024-03-28T22:48:17.221 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000002119, 2024-03-28T22:48:17.221 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000151 2024-03-28T22:48:17.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.221 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_encrypted_bytes": 5392, 2024-03-28T22:48:17.221 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_encrypted_bytes": 559712 2024-03-28T22:48:17.221 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.222 INFO:teuthology.orchestra.run.smithi082.stdout: "cct": { 2024-03-28T22:48:17.222 INFO:teuthology.orchestra.run.smithi082.stdout: "total_workers": 1, 2024-03-28T22:48:17.222 INFO:teuthology.orchestra.run.smithi082.stdout: "unhealthy_workers": 0 2024-03-28T22:48:17.222 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.222 INFO:teuthology.orchestra.run.smithi082.stdout: "finisher-MDSRank": { 2024-03-28T22:48:17.222 INFO:teuthology.orchestra.run.smithi082.stdout: "queue_len": 0, 2024-03-28T22:48:17.222 INFO:teuthology.orchestra.run.smithi082.stdout: "complete_latency": { 2024-03-28T22:48:17.222 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 51, 2024-03-28T22:48:17.223 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.001258766, 2024-03-28T22:48:17.223 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000024681 2024-03-28T22:48:17.223 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:17.223 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.223 INFO:teuthology.orchestra.run.smithi082.stdout: "finisher-PurgeQueue": { 2024-03-28T22:48:17.223 INFO:teuthology.orchestra.run.smithi082.stdout: "queue_len": 0, 2024-03-28T22:48:17.223 INFO:teuthology.orchestra.run.smithi082.stdout: "complete_latency": { 2024-03-28T22:48:17.223 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 2, 2024-03-28T22:48:17.223 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000006178, 2024-03-28T22:48:17.224 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000003089 2024-03-28T22:48:17.224 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:17.224 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.224 INFO:teuthology.orchestra.run.smithi082.stdout: "mds": { 2024-03-28T22:48:17.224 INFO:teuthology.orchestra.run.smithi082.stdout: "request": 3, 2024-03-28T22:48:17.224 INFO:teuthology.orchestra.run.smithi082.stdout: "reply": 2, 2024-03-28T22:48:17.224 INFO:teuthology.orchestra.run.smithi082.stdout: "reply_latency": { 2024-03-28T22:48:17.224 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 2, 2024-03-28T22:48:17.225 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000376211, 2024-03-28T22:48:17.225 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000188105 2024-03-28T22:48:17.225 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.225 INFO:teuthology.orchestra.run.smithi082.stdout: "slow_reply": 0, 2024-03-28T22:48:17.225 INFO:teuthology.orchestra.run.smithi082.stdout: "forward": 1, 2024-03-28T22:48:17.225 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_fetch_complete": 0, 2024-03-28T22:48:17.225 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_fetch_keys": 0, 2024-03-28T22:48:17.225 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_commit": 11, 2024-03-28T22:48:17.225 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_split": 0, 2024-03-28T22:48:17.226 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_merge": 0, 2024-03-28T22:48:17.226 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes": 28, 2024-03-28T22:48:17.226 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_top": 6, 2024-03-28T22:48:17.226 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_bottom": 22, 2024-03-28T22:48:17.226 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_pin_tail": 0, 2024-03-28T22:48:17.226 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_pinned": 19, 2024-03-28T22:48:17.226 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_expired": 0, 2024-03-28T22:48:17.226 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_with_caps": 0, 2024-03-28T22:48:17.226 INFO:teuthology.orchestra.run.smithi082.stdout: "caps": 0, 2024-03-28T22:48:17.227 INFO:teuthology.orchestra.run.smithi082.stdout: "subtrees": 2, 2024-03-28T22:48:17.227 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse": 28, 2024-03-28T22:48:17.227 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_hit": 15, 2024-03-28T22:48:17.227 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_forward": 1, 2024-03-28T22:48:17.227 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_discover": 11, 2024-03-28T22:48:17.227 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_dir_fetch": 0, 2024-03-28T22:48:17.227 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_remote_ino": 0, 2024-03-28T22:48:17.227 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_lock": 0, 2024-03-28T22:48:17.228 INFO:teuthology.orchestra.run.smithi082.stdout: "load_cent": 0, 2024-03-28T22:48:17.228 INFO:teuthology.orchestra.run.smithi082.stdout: "q": 0, 2024-03-28T22:48:17.228 INFO:teuthology.orchestra.run.smithi082.stdout: "exported": 0, 2024-03-28T22:48:17.228 INFO:teuthology.orchestra.run.smithi082.stdout: "exported_inodes": 0, 2024-03-28T22:48:17.228 INFO:teuthology.orchestra.run.smithi082.stdout: "imported": 0, 2024-03-28T22:48:17.228 INFO:teuthology.orchestra.run.smithi082.stdout: "imported_inodes": 0, 2024-03-28T22:48:17.228 INFO:teuthology.orchestra.run.smithi082.stdout: "openino_dir_fetch": 0, 2024-03-28T22:48:17.228 INFO:teuthology.orchestra.run.smithi082.stdout: "openino_backtrace_fetch": 0, 2024-03-28T22:48:17.228 INFO:teuthology.orchestra.run.smithi082.stdout: "openino_peer_discover": 0, 2024-03-28T22:48:17.229 INFO:teuthology.orchestra.run.smithi082.stdout: "root_rfiles": 61, 2024-03-28T22:48:17.229 INFO:teuthology.orchestra.run.smithi082.stdout: "root_rbytes": 21056780, 2024-03-28T22:48:17.229 INFO:teuthology.orchestra.run.smithi082.stdout: "root_rsnaps": 6, 2024-03-28T22:48:17.229 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T22:48:17.229 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_set_tag": 0, 2024-03-28T22:48:17.229 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T22:48:17.229 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_inotable_repaired": 0, 2024-03-28T22:48:17.229 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_dir_inodes": 0, 2024-03-28T22:48:17.229 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T22:48:17.230 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T22:48:17.230 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_file_inodes": 0, 2024-03-28T22:48:17.230 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_inode_file_caps": 0, 2024-03-28T22:48:17.230 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T22:48:17.230 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_grant": 0, 2024-03-28T22:48:17.230 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T22:48:17.230 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T22:48:17.230 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T22:48:17.231 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_caps": 0, 2024-03-28T22:48:17.231 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_caps_dirty": 0, 2024-03-28T22:48:17.231 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_cap_release": 0, 2024-03-28T22:48:17.231 INFO:teuthology.orchestra.run.smithi082.stdout: "process_request_cap_release": 0 2024-03-28T22:48:17.231 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.231 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_cache": { 2024-03-28T22:48:17.231 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_update": 0, 2024-03-28T22:48:17.231 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_update_receipt": 34, 2024-03-28T22:48:17.231 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_try_discover": 23, 2024-03-28T22:48:17.232 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_send_discover": 23, 2024-03-28T22:48:17.232 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_handle_discover": 0, 2024-03-28T22:48:17.232 INFO:teuthology.orchestra.run.smithi082.stdout: "num_strays": 0, 2024-03-28T22:48:17.232 INFO:teuthology.orchestra.run.smithi082.stdout: "num_strays_delayed": 0, 2024-03-28T22:48:17.232 INFO:teuthology.orchestra.run.smithi082.stdout: "num_strays_enqueuing": 0, 2024-03-28T22:48:17.232 INFO:teuthology.orchestra.run.smithi082.stdout: "strays_created": 0, 2024-03-28T22:48:17.232 INFO:teuthology.orchestra.run.smithi082.stdout: "strays_enqueued": 0, 2024-03-28T22:48:17.232 INFO:teuthology.orchestra.run.smithi082.stdout: "strays_reintegrated": 0, 2024-03-28T22:48:17.232 INFO:teuthology.orchestra.run.smithi082.stdout: "strays_migrated": 0, 2024-03-28T22:48:17.233 INFO:teuthology.orchestra.run.smithi082.stdout: "num_recovering_processing": 0, 2024-03-28T22:48:17.233 INFO:teuthology.orchestra.run.smithi082.stdout: "num_recovering_enqueued": 0, 2024-03-28T22:48:17.233 INFO:teuthology.orchestra.run.smithi082.stdout: "num_recovering_prioritized": 0, 2024-03-28T22:48:17.233 INFO:teuthology.orchestra.run.smithi082.stdout: "recovery_started": 0, 2024-03-28T22:48:17.233 INFO:teuthology.orchestra.run.smithi082.stdout: "recovery_completed": 0, 2024-03-28T22:48:17.233 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_quiesce_path": 0, 2024-03-28T22:48:17.233 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_quiesce_inode": 0, 2024-03-28T22:48:17.233 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T22:48:17.233 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_exportdir": 0, 2024-03-28T22:48:17.234 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_flush": 0, 2024-03-28T22:48:17.234 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_fragmentdir": 0, 2024-03-28T22:48:17.234 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_fragstats": 0, 2024-03-28T22:48:17.234 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_inodestats": 0 2024-03-28T22:48:17.234 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.234 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_log": { 2024-03-28T22:48:17.234 INFO:teuthology.orchestra.run.smithi082.stdout: "evlrg": 0, 2024-03-28T22:48:17.234 INFO:teuthology.orchestra.run.smithi082.stdout: "evadd": 6, 2024-03-28T22:48:17.235 INFO:teuthology.orchestra.run.smithi082.stdout: "evex": 0, 2024-03-28T22:48:17.235 INFO:teuthology.orchestra.run.smithi082.stdout: "evtrm": 0, 2024-03-28T22:48:17.235 INFO:teuthology.orchestra.run.smithi082.stdout: "ev": 6, 2024-03-28T22:48:17.235 INFO:teuthology.orchestra.run.smithi082.stdout: "evexg": 0, 2024-03-28T22:48:17.235 INFO:teuthology.orchestra.run.smithi082.stdout: "evexd": 0, 2024-03-28T22:48:17.235 INFO:teuthology.orchestra.run.smithi082.stdout: "segadd": 2, 2024-03-28T22:48:17.235 INFO:teuthology.orchestra.run.smithi082.stdout: "segex": 0, 2024-03-28T22:48:17.235 INFO:teuthology.orchestra.run.smithi082.stdout: "segtrm": 0, 2024-03-28T22:48:17.235 INFO:teuthology.orchestra.run.smithi082.stdout: "seg": 2, 2024-03-28T22:48:17.236 INFO:teuthology.orchestra.run.smithi082.stdout: "segmjr": 2, 2024-03-28T22:48:17.236 INFO:teuthology.orchestra.run.smithi082.stdout: "segexg": 0, 2024-03-28T22:48:17.236 INFO:teuthology.orchestra.run.smithi082.stdout: "segexd": 0, 2024-03-28T22:48:17.236 INFO:teuthology.orchestra.run.smithi082.stdout: "expos": 4194304, 2024-03-28T22:48:17.236 INFO:teuthology.orchestra.run.smithi082.stdout: "wrpos": 4195956, 2024-03-28T22:48:17.236 INFO:teuthology.orchestra.run.smithi082.stdout: "rdpos": 0, 2024-03-28T22:48:17.236 INFO:teuthology.orchestra.run.smithi082.stdout: "jlat": { 2024-03-28T22:48:17.236 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 5, 2024-03-28T22:48:17.237 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.014813689, 2024-03-28T22:48:17.237 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.002962737 2024-03-28T22:48:17.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.237 INFO:teuthology.orchestra.run.smithi082.stdout: "replayed": 0 2024-03-28T22:48:17.237 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.237 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_mem": { 2024-03-28T22:48:17.237 INFO:teuthology.orchestra.run.smithi082.stdout: "ino": 31, 2024-03-28T22:48:17.237 INFO:teuthology.orchestra.run.smithi082.stdout: "ino+": 31, 2024-03-28T22:48:17.237 INFO:teuthology.orchestra.run.smithi082.stdout: "ino-": 0, 2024-03-28T22:48:17.238 INFO:teuthology.orchestra.run.smithi082.stdout: "dir": 37, 2024-03-28T22:48:17.238 INFO:teuthology.orchestra.run.smithi082.stdout: "dir+": 39, 2024-03-28T22:48:17.238 INFO:teuthology.orchestra.run.smithi082.stdout: "dir-": 2, 2024-03-28T22:48:17.238 INFO:teuthology.orchestra.run.smithi082.stdout: "dn": 28, 2024-03-28T22:48:17.238 INFO:teuthology.orchestra.run.smithi082.stdout: "dn+": 28, 2024-03-28T22:48:17.238 INFO:teuthology.orchestra.run.smithi082.stdout: "dn-": 0, 2024-03-28T22:48:17.238 INFO:teuthology.orchestra.run.smithi082.stdout: "cap": 0, 2024-03-28T22:48:17.238 INFO:teuthology.orchestra.run.smithi082.stdout: "cap+": 2, 2024-03-28T22:48:17.239 INFO:teuthology.orchestra.run.smithi082.stdout: "cap-": 2, 2024-03-28T22:48:17.239 INFO:teuthology.orchestra.run.smithi082.stdout: "rss": 50748, 2024-03-28T22:48:17.239 INFO:teuthology.orchestra.run.smithi082.stdout: "heap": 223612 2024-03-28T22:48:17.239 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.239 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_server": { 2024-03-28T22:48:17.239 INFO:teuthology.orchestra.run.smithi082.stdout: "dispatch_client_request": 3, 2024-03-28T22:48:17.239 INFO:teuthology.orchestra.run.smithi082.stdout: "dispatch_server_request": 0, 2024-03-28T22:48:17.239 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_request": 3, 2024-03-28T22:48:17.239 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_session": 8, 2024-03-28T22:48:17.240 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_peer_request": 0, 2024-03-28T22:48:17.240 INFO:teuthology.orchestra.run.smithi082.stdout: "req_create_latency": { 2024-03-28T22:48:17.240 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.240 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.240 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.240 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.240 INFO:teuthology.orchestra.run.smithi082.stdout: "req_getattr_latency": { 2024-03-28T22:48:17.240 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 1, 2024-03-28T22:48:17.241 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000192268, 2024-03-28T22:48:17.241 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000192268 2024-03-28T22:48:17.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.241 INFO:teuthology.orchestra.run.smithi082.stdout: "req_getfilelock_latency": { 2024-03-28T22:48:17.241 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.241 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.241 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.241 INFO:teuthology.orchestra.run.smithi082.stdout: "req_link_latency": { 2024-03-28T22:48:17.242 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.242 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.242 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.242 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookup_latency": { 2024-03-28T22:48:17.242 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 1, 2024-03-28T22:48:17.242 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000183943, 2024-03-28T22:48:17.242 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000183943 2024-03-28T22:48:17.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.243 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookuphash_latency": { 2024-03-28T22:48:17.243 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.243 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.243 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.243 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.243 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookupino_latency": { 2024-03-28T22:48:17.243 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.243 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.244 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.244 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.244 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookupname_latency": { 2024-03-28T22:48:17.244 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.244 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.244 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.244 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.244 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookupparent_latency": { 2024-03-28T22:48:17.244 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.245 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.245 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.245 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookupsnap_latency": { 2024-03-28T22:48:17.245 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.245 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.245 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.246 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lssnap_latency": { 2024-03-28T22:48:17.246 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.246 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.246 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.246 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.246 INFO:teuthology.orchestra.run.smithi082.stdout: "req_mkdir_latency": { 2024-03-28T22:48:17.247 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.247 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.247 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.247 INFO:teuthology.orchestra.run.smithi082.stdout: "req_mknod_latency": { 2024-03-28T22:48:17.247 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.248 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.248 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.248 INFO:teuthology.orchestra.run.smithi082.stdout: "req_mksnap_latency": { 2024-03-28T22:48:17.248 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.248 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.248 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.249 INFO:teuthology.orchestra.run.smithi082.stdout: "req_open_latency": { 2024-03-28T22:48:17.249 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.249 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.249 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.249 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.249 INFO:teuthology.orchestra.run.smithi082.stdout: "req_readdir_latency": { 2024-03-28T22:48:17.249 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.249 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.250 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.250 INFO:teuthology.orchestra.run.smithi082.stdout: "req_rename_latency": { 2024-03-28T22:48:17.250 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.250 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.250 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.250 INFO:teuthology.orchestra.run.smithi082.stdout: "req_renamesnap_latency": { 2024-03-28T22:48:17.251 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.251 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.251 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.251 INFO:teuthology.orchestra.run.smithi082.stdout: "req_snapdiff_latency": { 2024-03-28T22:48:17.251 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.251 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.251 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.251 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.252 INFO:teuthology.orchestra.run.smithi082.stdout: "req_rmdir_latency": { 2024-03-28T22:48:17.252 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.252 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.252 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.252 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.252 INFO:teuthology.orchestra.run.smithi082.stdout: "req_rmsnap_latency": { 2024-03-28T22:48:17.252 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.252 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.252 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.253 INFO:teuthology.orchestra.run.smithi082.stdout: "req_rmxattr_latency": { 2024-03-28T22:48:17.253 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.253 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.253 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.253 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.253 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setattr_latency": { 2024-03-28T22:48:17.253 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.253 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.254 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.254 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.254 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setdirlayout_latency": { 2024-03-28T22:48:17.254 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.254 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.254 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.254 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.254 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setfilelock_latency": { 2024-03-28T22:48:17.254 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.255 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.255 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.255 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.255 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setlayout_latency": { 2024-03-28T22:48:17.255 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.255 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.255 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.255 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.255 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setxattr_latency": { 2024-03-28T22:48:17.256 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.256 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.256 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.256 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.256 INFO:teuthology.orchestra.run.smithi082.stdout: "req_symlink_latency": { 2024-03-28T22:48:17.256 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.256 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.256 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.257 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.257 INFO:teuthology.orchestra.run.smithi082.stdout: "req_unlink_latency": { 2024-03-28T22:48:17.257 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.257 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.257 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.257 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.257 INFO:teuthology.orchestra.run.smithi082.stdout: "cap_revoke_eviction": 0, 2024-03-28T22:48:17.257 INFO:teuthology.orchestra.run.smithi082.stdout: "cap_acquisition_throttle": 0, 2024-03-28T22:48:17.257 INFO:teuthology.orchestra.run.smithi082.stdout: "req_getvxattr_latency": { 2024-03-28T22:48:17.258 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.258 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.258 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.258 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:17.258 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.258 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_sessions": { 2024-03-28T22:48:17.258 INFO:teuthology.orchestra.run.smithi082.stdout: "session_count": 0, 2024-03-28T22:48:17.258 INFO:teuthology.orchestra.run.smithi082.stdout: "session_add": 2, 2024-03-28T22:48:17.258 INFO:teuthology.orchestra.run.smithi082.stdout: "session_remove": 2, 2024-03-28T22:48:17.259 INFO:teuthology.orchestra.run.smithi082.stdout: "sessions_open": 0, 2024-03-28T22:48:17.259 INFO:teuthology.orchestra.run.smithi082.stdout: "sessions_stale": 0, 2024-03-28T22:48:17.259 INFO:teuthology.orchestra.run.smithi082.stdout: "total_load": 1, 2024-03-28T22:48:17.259 INFO:teuthology.orchestra.run.smithi082.stdout: "average_load": 0, 2024-03-28T22:48:17.259 INFO:teuthology.orchestra.run.smithi082.stdout: "avg_session_uptime": 87, 2024-03-28T22:48:17.259 INFO:teuthology.orchestra.run.smithi082.stdout: "mdthresh_evicted": 0 2024-03-28T22:48:17.259 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.259 INFO:teuthology.orchestra.run.smithi082.stdout: "mempool": { 2024-03-28T22:48:17.260 INFO:teuthology.orchestra.run.smithi082.stdout: "bloom_filter_bytes": 0, 2024-03-28T22:48:17.260 INFO:teuthology.orchestra.run.smithi082.stdout: "bloom_filter_items": 0, 2024-03-28T22:48:17.260 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T22:48:17.260 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_alloc_items": 0, 2024-03-28T22:48:17.260 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T22:48:17.260 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_data_items": 0, 2024-03-28T22:48:17.260 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T22:48:17.260 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T22:48:17.261 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T22:48:17.261 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T22:48:17.261 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T22:48:17.261 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_other_items": 0, 2024-03-28T22:48:17.261 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T22:48:17.261 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T22:48:17.261 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_extent_bytes": 0, 2024-03-28T22:48:17.261 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_extent_items": 0, 2024-03-28T22:48:17.261 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_blob_bytes": 0, 2024-03-28T22:48:17.262 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_blob_items": 0, 2024-03-28T22:48:17.262 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T22:48:17.262 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T22:48:17.262 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T22:48:17.262 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T22:48:17.262 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T22:48:17.262 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_fsck_items": 0, 2024-03-28T22:48:17.262 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_txc_bytes": 0, 2024-03-28T22:48:17.263 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_txc_items": 0, 2024-03-28T22:48:17.263 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T22:48:17.263 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T22:48:17.263 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_writing_bytes": 0, 2024-03-28T22:48:17.263 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_writing_items": 0, 2024-03-28T22:48:17.263 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_bytes": 0, 2024-03-28T22:48:17.263 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_items": 0, 2024-03-28T22:48:17.263 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T22:48:17.264 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_file_reader_items": 0, 2024-03-28T22:48:17.264 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T22:48:17.264 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_file_writer_items": 0, 2024-03-28T22:48:17.264 INFO:teuthology.orchestra.run.smithi082.stdout: "buffer_anon_bytes": 255842, 2024-03-28T22:48:17.264 INFO:teuthology.orchestra.run.smithi082.stdout: "buffer_anon_items": 96, 2024-03-28T22:48:17.264 INFO:teuthology.orchestra.run.smithi082.stdout: "buffer_meta_bytes": 0, 2024-03-28T22:48:17.264 INFO:teuthology.orchestra.run.smithi082.stdout: "buffer_meta_items": 0, 2024-03-28T22:48:17.264 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_bytes": 0, 2024-03-28T22:48:17.264 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_items": 0, 2024-03-28T22:48:17.265 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_mapbl_bytes": 0, 2024-03-28T22:48:17.265 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_mapbl_items": 0, 2024-03-28T22:48:17.265 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_pglog_bytes": 0, 2024-03-28T22:48:17.265 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_pglog_items": 0, 2024-03-28T22:48:17.265 INFO:teuthology.orchestra.run.smithi082.stdout: "osdmap_bytes": 9976, 2024-03-28T22:48:17.265 INFO:teuthology.orchestra.run.smithi082.stdout: "osdmap_items": 209, 2024-03-28T22:48:17.265 INFO:teuthology.orchestra.run.smithi082.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T22:48:17.265 INFO:teuthology.orchestra.run.smithi082.stdout: "osdmap_mapping_items": 0, 2024-03-28T22:48:17.265 INFO:teuthology.orchestra.run.smithi082.stdout: "pgmap_bytes": 0, 2024-03-28T22:48:17.266 INFO:teuthology.orchestra.run.smithi082.stdout: "pgmap_items": 0, 2024-03-28T22:48:17.266 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_co_bytes": 141596, 2024-03-28T22:48:17.266 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_co_items": 757, 2024-03-28T22:48:17.266 INFO:teuthology.orchestra.run.smithi082.stdout: "unittest_1_bytes": 0, 2024-03-28T22:48:17.266 INFO:teuthology.orchestra.run.smithi082.stdout: "unittest_1_items": 0, 2024-03-28T22:48:17.266 INFO:teuthology.orchestra.run.smithi082.stdout: "unittest_2_bytes": 0, 2024-03-28T22:48:17.266 INFO:teuthology.orchestra.run.smithi082.stdout: "unittest_2_items": 0 2024-03-28T22:48:17.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.266 INFO:teuthology.orchestra.run.smithi082.stdout: "objecter": { 2024-03-28T22:48:17.267 INFO:teuthology.orchestra.run.smithi082.stdout: "op_active": 0, 2024-03-28T22:48:17.267 INFO:teuthology.orchestra.run.smithi082.stdout: "op_laggy": 0, 2024-03-28T22:48:17.267 INFO:teuthology.orchestra.run.smithi082.stdout: "op_send": 43, 2024-03-28T22:48:17.267 INFO:teuthology.orchestra.run.smithi082.stdout: "op_send_bytes": 12099, 2024-03-28T22:48:17.267 INFO:teuthology.orchestra.run.smithi082.stdout: "op_resend": 28, 2024-03-28T22:48:17.267 INFO:teuthology.orchestra.run.smithi082.stdout: "op_reply": 43, 2024-03-28T22:48:17.267 INFO:teuthology.orchestra.run.smithi082.stdout: "op_latency": { 2024-03-28T22:48:17.267 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 43, 2024-03-28T22:48:17.268 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.292994313, 2024-03-28T22:48:17.268 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.006813821 2024-03-28T22:48:17.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.268 INFO:teuthology.orchestra.run.smithi082.stdout: "op_inflight": 0, 2024-03-28T22:48:17.268 INFO:teuthology.orchestra.run.smithi082.stdout: "oplen_avg": { 2024-03-28T22:48:17.268 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 43, 2024-03-28T22:48:17.268 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 67 2024-03-28T22:48:17.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.269 INFO:teuthology.orchestra.run.smithi082.stdout: "op": 43, 2024-03-28T22:48:17.269 INFO:teuthology.orchestra.run.smithi082.stdout: "op_r": 0, 2024-03-28T22:48:17.269 INFO:teuthology.orchestra.run.smithi082.stdout: "op_w": 43, 2024-03-28T22:48:17.269 INFO:teuthology.orchestra.run.smithi082.stdout: "op_rmw": 0, 2024-03-28T22:48:17.269 INFO:teuthology.orchestra.run.smithi082.stdout: "op_pg": 0, 2024-03-28T22:48:17.269 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_stat": 0, 2024-03-28T22:48:17.269 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_create": 10, 2024-03-28T22:48:17.269 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_read": 0, 2024-03-28T22:48:17.269 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_write": 5, 2024-03-28T22:48:17.270 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_writefull": 7, 2024-03-28T22:48:17.270 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_writesame": 0, 2024-03-28T22:48:17.270 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_append": 0, 2024-03-28T22:48:17.270 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_zero": 0, 2024-03-28T22:48:17.270 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_truncate": 0, 2024-03-28T22:48:17.270 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_delete": 6, 2024-03-28T22:48:17.270 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_mapext": 0, 2024-03-28T22:48:17.270 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_sparse_read": 0, 2024-03-28T22:48:17.271 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_clonerange": 0, 2024-03-28T22:48:17.271 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_getxattr": 0, 2024-03-28T22:48:17.271 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_setxattr": 20, 2024-03-28T22:48:17.271 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_cmpxattr": 0, 2024-03-28T22:48:17.271 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_rmxattr": 0, 2024-03-28T22:48:17.271 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_resetxattrs": 0, 2024-03-28T22:48:17.271 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_call": 0, 2024-03-28T22:48:17.271 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_watch": 0, 2024-03-28T22:48:17.271 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_notify": 0, 2024-03-28T22:48:17.272 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T22:48:17.272 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_pgls": 0, 2024-03-28T22:48:17.272 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_pgls_filter": 0, 2024-03-28T22:48:17.272 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_other": 0, 2024-03-28T22:48:17.272 INFO:teuthology.orchestra.run.smithi082.stdout: "linger_active": 0, 2024-03-28T22:48:17.272 INFO:teuthology.orchestra.run.smithi082.stdout: "linger_send": 0, 2024-03-28T22:48:17.272 INFO:teuthology.orchestra.run.smithi082.stdout: "linger_resend": 0, 2024-03-28T22:48:17.272 INFO:teuthology.orchestra.run.smithi082.stdout: "linger_ping": 0, 2024-03-28T22:48:17.272 INFO:teuthology.orchestra.run.smithi082.stdout: "poolop_active": 0, 2024-03-28T22:48:17.273 INFO:teuthology.orchestra.run.smithi082.stdout: "poolop_send": 0, 2024-03-28T22:48:17.273 INFO:teuthology.orchestra.run.smithi082.stdout: "poolop_resend": 0, 2024-03-28T22:48:17.273 INFO:teuthology.orchestra.run.smithi082.stdout: "poolstat_active": 0, 2024-03-28T22:48:17.273 INFO:teuthology.orchestra.run.smithi082.stdout: "poolstat_send": 0, 2024-03-28T22:48:17.273 INFO:teuthology.orchestra.run.smithi082.stdout: "poolstat_resend": 0, 2024-03-28T22:48:17.273 INFO:teuthology.orchestra.run.smithi082.stdout: "statfs_active": 0, 2024-03-28T22:48:17.273 INFO:teuthology.orchestra.run.smithi082.stdout: "statfs_send": 0, 2024-03-28T22:48:17.273 INFO:teuthology.orchestra.run.smithi082.stdout: "statfs_resend": 0, 2024-03-28T22:48:17.273 INFO:teuthology.orchestra.run.smithi082.stdout: "command_active": 0, 2024-03-28T22:48:17.274 INFO:teuthology.orchestra.run.smithi082.stdout: "command_send": 0, 2024-03-28T22:48:17.274 INFO:teuthology.orchestra.run.smithi082.stdout: "command_resend": 0, 2024-03-28T22:48:17.274 INFO:teuthology.orchestra.run.smithi082.stdout: "map_epoch": 109, 2024-03-28T22:48:17.274 INFO:teuthology.orchestra.run.smithi082.stdout: "map_full": 0, 2024-03-28T22:48:17.274 INFO:teuthology.orchestra.run.smithi082.stdout: "map_inc": 17, 2024-03-28T22:48:17.274 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_sessions": 9, 2024-03-28T22:48:17.274 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_session_open": 9, 2024-03-28T22:48:17.274 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_session_close": 0, 2024-03-28T22:48:17.275 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_laggy": 0, 2024-03-28T22:48:17.275 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_wr": 17, 2024-03-28T22:48:17.275 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_rd": 0, 2024-03-28T22:48:17.275 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_del": 2 2024-03-28T22:48:17.275 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.275 INFO:teuthology.orchestra.run.smithi082.stdout: "oft": { 2024-03-28T22:48:17.275 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_total_objs": 1, 2024-03-28T22:48:17.275 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_total_kv_pairs": 0, 2024-03-28T22:48:17.275 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_total_updates": 0, 2024-03-28T22:48:17.276 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_total_removes": 0 2024-03-28T22:48:17.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.276 INFO:teuthology.orchestra.run.smithi082.stdout: "osd-slow-ops": { 2024-03-28T22:48:17.276 INFO:teuthology.orchestra.run.smithi082.stdout: "slow_ops_count": 0 2024-03-28T22:48:17.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.276 INFO:teuthology.orchestra.run.smithi082.stdout: "purge_queue": { 2024-03-28T22:48:17.276 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executing_ops": 0, 2024-03-28T22:48:17.276 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T22:48:17.276 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executing": 0, 2024-03-28T22:48:17.277 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executing_high_water": 0, 2024-03-28T22:48:17.277 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executed_ops": 0, 2024-03-28T22:48:17.277 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executed": 0, 2024-03-28T22:48:17.277 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_item_in_journal": 0 2024-03-28T22:48:17.277 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.277 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T22:48:17.277 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-28T22:48:17.277 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 104857600, 2024-03-28T22:48:17.277 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-28T22:48:17.278 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 70641, 2024-03-28T22:48:17.278 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 18816440, 2024-03-28T22:48:17.278 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:17.278 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 70641, 2024-03-28T22:48:17.278 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 0, 2024-03-28T22:48:17.278 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 0, 2024-03-28T22:48:17.278 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 70641, 2024-03-28T22:48:17.279 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 18816440, 2024-03-28T22:48:17.279 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-28T22:48:17.279 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.279 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.279 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.279 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:17.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.279 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-objecter_bytes": { 2024-03-28T22:48:17.279 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-28T22:48:17.280 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 104857600, 2024-03-28T22:48:17.280 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-28T22:48:17.280 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 0, 2024-03-28T22:48:17.280 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 0, 2024-03-28T22:48:17.280 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:17.280 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 0, 2024-03-28T22:48:17.280 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 43, 2024-03-28T22:48:17.280 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 12099, 2024-03-28T22:48:17.280 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 43, 2024-03-28T22:48:17.281 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 12099, 2024-03-28T22:48:17.281 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-28T22:48:17.281 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.281 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.281 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.281 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:17.281 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.281 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-objecter_ops": { 2024-03-28T22:48:17.281 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-28T22:48:17.282 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 1024, 2024-03-28T22:48:17.282 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-28T22:48:17.282 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 0, 2024-03-28T22:48:17.282 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 0, 2024-03-28T22:48:17.282 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:17.282 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 0, 2024-03-28T22:48:17.282 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 43, 2024-03-28T22:48:17.282 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 43, 2024-03-28T22:48:17.283 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 43, 2024-03-28T22:48:17.283 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 43, 2024-03-28T22:48:17.283 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-28T22:48:17.283 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.283 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.283 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.283 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:17.283 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.283 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-write_buf_throttle": { 2024-03-28T22:48:17.284 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-28T22:48:17.284 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 3758096384, 2024-03-28T22:48:17.284 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-28T22:48:17.284 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 0, 2024-03-28T22:48:17.284 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 0, 2024-03-28T22:48:17.284 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:17.284 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 0, 2024-03-28T22:48:17.284 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 0, 2024-03-28T22:48:17.284 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 0, 2024-03-28T22:48:17.285 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 0, 2024-03-28T22:48:17.285 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 0, 2024-03-28T22:48:17.285 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-28T22:48:17.285 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.285 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.285 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.285 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:17.285 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:17.285 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-write_buf_throttle-0x55d78946e500": { 2024-03-28T22:48:17.286 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-28T22:48:17.286 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 3758096384, 2024-03-28T22:48:17.286 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-28T22:48:17.286 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 6, 2024-03-28T22:48:17.286 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 1652, 2024-03-28T22:48:17.286 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:17.286 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 6, 2024-03-28T22:48:17.286 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 0, 2024-03-28T22:48:17.287 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 0, 2024-03-28T22:48:17.287 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 5, 2024-03-28T22:48:17.287 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 1652, 2024-03-28T22:48:17.287 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-28T22:48:17.287 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:17.287 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:17.287 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:17.287 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:17.288 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:17.288 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-28T22:48:17.288 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:17 smithi043 ceph-mon[31823]: pgmap v347: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:48:17.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:17 smithi172 ceph-mon[36566]: pgmap v347: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:48:17.960 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-03-28T22:48:17.961 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-03-28T22:48:17.961 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2024-03-28T22:48:17.961 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=34 2024-03-28T22:48:17.962 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-03-28T22:48:17.962 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-03-28T22:48:17.962 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T22:48:17.963 DEBUG:tasks.check_counter:Getting stats from k 2024-03-28T22:48:17.963 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-28T22:48:18.245 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T22:48:18.948 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:18 smithi082 ceph-mon[36471]: pgmap v348: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:48:18.948 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:18 smithi082 ceph-mon[36471]: osdmap e110: 12 total, 12 up, 12 in 2024-03-28T22:48:19.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:18 smithi172 ceph-mon[36566]: pgmap v348: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:48:19.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:18 smithi172 ceph-mon[36566]: osdmap e110: 12 total, 12 up, 12 in 2024-03-28T22:48:19.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:18 smithi043 ceph-mon[31823]: pgmap v348: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:48:19.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:18 smithi043 ceph-mon[31823]: osdmap e110: 12 total, 12 up, 12 in 2024-03-28T22:48:20.101 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-28T22:48:20.102 INFO:teuthology.orchestra.run.smithi082.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T22:48:20.102 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_messages": 853, 2024-03-28T22:48:20.102 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages": 855, 2024-03-28T22:48:20.102 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_bytes": 286553, 2024-03-28T22:48:20.102 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_bytes": 271255, 2024-03-28T22:48:20.102 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_created_connections": 6, 2024-03-28T22:48:20.102 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_active_connections": 2, 2024-03-28T22:48:20.103 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_total_time": 0.058708907, 2024-03-28T22:48:20.103 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_send_time": 0.017584257, 2024-03-28T22:48:20.103 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_recv_time": 0.015371523, 2024-03-28T22:48:20.103 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_fast_dispatch_time": 0.019563213, 2024-03-28T22:48:20.103 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:48:20.103 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 853, 2024-03-28T22:48:20.103 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.025758934, 2024-03-28T22:48:20.103 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000030198 2024-03-28T22:48:20.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.104 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:48:20.104 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.104 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.104 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.104 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.104 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T22:48:20.104 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T22:48:20.105 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.105 INFO:teuthology.orchestra.run.smithi082.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T22:48:20.105 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_messages": 1524, 2024-03-28T22:48:20.105 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages": 1712, 2024-03-28T22:48:20.105 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_bytes": 390080, 2024-03-28T22:48:20.105 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_bytes": 964725, 2024-03-28T22:48:20.105 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_created_connections": 7, 2024-03-28T22:48:20.105 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_active_connections": 5, 2024-03-28T22:48:20.105 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_total_time": 0.089585700, 2024-03-28T22:48:20.106 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_send_time": 0.030875199, 2024-03-28T22:48:20.106 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_recv_time": 0.020928966, 2024-03-28T22:48:20.106 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_fast_dispatch_time": 0.030531191, 2024-03-28T22:48:20.106 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:48:20.106 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 1709, 2024-03-28T22:48:20.106 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.062384753, 2024-03-28T22:48:20.106 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000036503 2024-03-28T22:48:20.106 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.107 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:48:20.107 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.107 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.107 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.107 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.107 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T22:48:20.107 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_encrypted_bytes": 480672 2024-03-28T22:48:20.107 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.108 INFO:teuthology.orchestra.run.smithi082.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T22:48:20.108 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_messages": 295, 2024-03-28T22:48:20.108 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages": 276, 2024-03-28T22:48:20.108 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_bytes": 1913844, 2024-03-28T22:48:20.108 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_bytes": 158613, 2024-03-28T22:48:20.108 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_created_connections": 4, 2024-03-28T22:48:20.108 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_active_connections": 3, 2024-03-28T22:48:20.108 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_total_time": 0.045257233, 2024-03-28T22:48:20.108 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_send_time": 0.010275686, 2024-03-28T22:48:20.109 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_recv_time": 0.043858731, 2024-03-28T22:48:20.109 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_fast_dispatch_time": 0.004621315, 2024-03-28T22:48:20.109 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:48:20.109 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 276, 2024-03-28T22:48:20.109 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.011249602, 2024-03-28T22:48:20.109 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000040759 2024-03-28T22:48:20.109 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.109 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:48:20.109 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.110 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.110 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.110 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_encrypted_bytes": 1912496, 2024-03-28T22:48:20.110 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_encrypted_bytes": 157648 2024-03-28T22:48:20.110 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.110 INFO:teuthology.orchestra.run.smithi082.stdout: "cct": { 2024-03-28T22:48:20.110 INFO:teuthology.orchestra.run.smithi082.stdout: "total_workers": 1, 2024-03-28T22:48:20.111 INFO:teuthology.orchestra.run.smithi082.stdout: "unhealthy_workers": 0 2024-03-28T22:48:20.111 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.111 INFO:teuthology.orchestra.run.smithi082.stdout: "finisher-MDSRank": { 2024-03-28T22:48:20.111 INFO:teuthology.orchestra.run.smithi082.stdout: "queue_len": 0, 2024-03-28T22:48:20.111 INFO:teuthology.orchestra.run.smithi082.stdout: "complete_latency": { 2024-03-28T22:48:20.111 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 4075, 2024-03-28T22:48:20.111 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.081692732, 2024-03-28T22:48:20.111 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000020047 2024-03-28T22:48:20.112 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:20.112 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.112 INFO:teuthology.orchestra.run.smithi082.stdout: "finisher-PurgeQueue": { 2024-03-28T22:48:20.112 INFO:teuthology.orchestra.run.smithi082.stdout: "queue_len": 0, 2024-03-28T22:48:20.112 INFO:teuthology.orchestra.run.smithi082.stdout: "complete_latency": { 2024-03-28T22:48:20.112 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.112 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.112 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.112 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:20.113 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.113 INFO:teuthology.orchestra.run.smithi082.stdout: "mds": { 2024-03-28T22:48:20.113 INFO:teuthology.orchestra.run.smithi082.stdout: "request": 0, 2024-03-28T22:48:20.113 INFO:teuthology.orchestra.run.smithi082.stdout: "reply": 0, 2024-03-28T22:48:20.113 INFO:teuthology.orchestra.run.smithi082.stdout: "reply_latency": { 2024-03-28T22:48:20.113 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.113 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.113 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.114 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.114 INFO:teuthology.orchestra.run.smithi082.stdout: "slow_reply": 0, 2024-03-28T22:48:20.114 INFO:teuthology.orchestra.run.smithi082.stdout: "forward": 0, 2024-03-28T22:48:20.114 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_fetch_complete": 0, 2024-03-28T22:48:20.114 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_fetch_keys": 0, 2024-03-28T22:48:20.114 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_commit": 0, 2024-03-28T22:48:20.114 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_split": 0, 2024-03-28T22:48:20.115 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_merge": 0, 2024-03-28T22:48:20.115 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes": 0, 2024-03-28T22:48:20.115 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_top": 0, 2024-03-28T22:48:20.115 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_bottom": 0, 2024-03-28T22:48:20.115 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_pin_tail": 0, 2024-03-28T22:48:20.115 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_pinned": 0, 2024-03-28T22:48:20.115 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_expired": 0, 2024-03-28T22:48:20.115 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_with_caps": 0, 2024-03-28T22:48:20.115 INFO:teuthology.orchestra.run.smithi082.stdout: "caps": 0, 2024-03-28T22:48:20.116 INFO:teuthology.orchestra.run.smithi082.stdout: "subtrees": 1, 2024-03-28T22:48:20.116 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse": 0, 2024-03-28T22:48:20.116 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_hit": 0, 2024-03-28T22:48:20.116 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_forward": 0, 2024-03-28T22:48:20.116 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_discover": 0, 2024-03-28T22:48:20.116 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_dir_fetch": 0, 2024-03-28T22:48:20.116 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_remote_ino": 0, 2024-03-28T22:48:20.116 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_lock": 0, 2024-03-28T22:48:20.116 INFO:teuthology.orchestra.run.smithi082.stdout: "load_cent": 0, 2024-03-28T22:48:20.117 INFO:teuthology.orchestra.run.smithi082.stdout: "q": 0, 2024-03-28T22:48:20.117 INFO:teuthology.orchestra.run.smithi082.stdout: "exported": 0, 2024-03-28T22:48:20.117 INFO:teuthology.orchestra.run.smithi082.stdout: "exported_inodes": 0, 2024-03-28T22:48:20.117 INFO:teuthology.orchestra.run.smithi082.stdout: "imported": 0, 2024-03-28T22:48:20.117 INFO:teuthology.orchestra.run.smithi082.stdout: "imported_inodes": 0, 2024-03-28T22:48:20.117 INFO:teuthology.orchestra.run.smithi082.stdout: "openino_dir_fetch": 0, 2024-03-28T22:48:20.117 INFO:teuthology.orchestra.run.smithi082.stdout: "openino_backtrace_fetch": 0, 2024-03-28T22:48:20.117 INFO:teuthology.orchestra.run.smithi082.stdout: "openino_peer_discover": 0, 2024-03-28T22:48:20.118 INFO:teuthology.orchestra.run.smithi082.stdout: "root_rfiles": 0, 2024-03-28T22:48:20.118 INFO:teuthology.orchestra.run.smithi082.stdout: "root_rbytes": 0, 2024-03-28T22:48:20.118 INFO:teuthology.orchestra.run.smithi082.stdout: "root_rsnaps": 0, 2024-03-28T22:48:20.118 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T22:48:20.118 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_set_tag": 0, 2024-03-28T22:48:20.118 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T22:48:20.118 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_inotable_repaired": 0, 2024-03-28T22:48:20.118 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_dir_inodes": 0, 2024-03-28T22:48:20.119 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T22:48:20.119 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T22:48:20.119 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_file_inodes": 0, 2024-03-28T22:48:20.119 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_inode_file_caps": 0, 2024-03-28T22:48:20.119 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T22:48:20.119 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_grant": 0, 2024-03-28T22:48:20.119 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T22:48:20.119 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T22:48:20.119 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T22:48:20.120 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_caps": 0, 2024-03-28T22:48:20.120 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_caps_dirty": 0, 2024-03-28T22:48:20.120 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_cap_release": 0, 2024-03-28T22:48:20.120 INFO:teuthology.orchestra.run.smithi082.stdout: "process_request_cap_release": 0 2024-03-28T22:48:20.120 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.120 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_cache": { 2024-03-28T22:48:20.120 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_update": 0, 2024-03-28T22:48:20.120 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_update_receipt": 0, 2024-03-28T22:48:20.120 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_try_discover": 0, 2024-03-28T22:48:20.121 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_send_discover": 0, 2024-03-28T22:48:20.121 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_handle_discover": 0, 2024-03-28T22:48:20.121 INFO:teuthology.orchestra.run.smithi082.stdout: "num_strays": 0, 2024-03-28T22:48:20.121 INFO:teuthology.orchestra.run.smithi082.stdout: "num_strays_delayed": 0, 2024-03-28T22:48:20.121 INFO:teuthology.orchestra.run.smithi082.stdout: "num_strays_enqueuing": 0, 2024-03-28T22:48:20.121 INFO:teuthology.orchestra.run.smithi082.stdout: "strays_created": 0, 2024-03-28T22:48:20.121 INFO:teuthology.orchestra.run.smithi082.stdout: "strays_enqueued": 0, 2024-03-28T22:48:20.122 INFO:teuthology.orchestra.run.smithi082.stdout: "strays_reintegrated": 0, 2024-03-28T22:48:20.122 INFO:teuthology.orchestra.run.smithi082.stdout: "strays_migrated": 0, 2024-03-28T22:48:20.122 INFO:teuthology.orchestra.run.smithi082.stdout: "num_recovering_processing": 0, 2024-03-28T22:48:20.122 INFO:teuthology.orchestra.run.smithi082.stdout: "num_recovering_enqueued": 0, 2024-03-28T22:48:20.122 INFO:teuthology.orchestra.run.smithi082.stdout: "num_recovering_prioritized": 0, 2024-03-28T22:48:20.122 INFO:teuthology.orchestra.run.smithi082.stdout: "recovery_started": 0, 2024-03-28T22:48:20.122 INFO:teuthology.orchestra.run.smithi082.stdout: "recovery_completed": 0, 2024-03-28T22:48:20.123 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_quiesce_path": 0, 2024-03-28T22:48:20.123 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_quiesce_inode": 0, 2024-03-28T22:48:20.123 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T22:48:20.123 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_exportdir": 0, 2024-03-28T22:48:20.123 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_flush": 0, 2024-03-28T22:48:20.123 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_fragmentdir": 0, 2024-03-28T22:48:20.123 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_fragstats": 0, 2024-03-28T22:48:20.123 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_inodestats": 0 2024-03-28T22:48:20.123 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.124 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_log": { 2024-03-28T22:48:20.124 INFO:teuthology.orchestra.run.smithi082.stdout: "evlrg": 0, 2024-03-28T22:48:20.124 INFO:teuthology.orchestra.run.smithi082.stdout: "evadd": 0, 2024-03-28T22:48:20.124 INFO:teuthology.orchestra.run.smithi082.stdout: "evex": 0, 2024-03-28T22:48:20.124 INFO:teuthology.orchestra.run.smithi082.stdout: "evtrm": 0, 2024-03-28T22:48:20.124 INFO:teuthology.orchestra.run.smithi082.stdout: "ev": 6, 2024-03-28T22:48:20.124 INFO:teuthology.orchestra.run.smithi082.stdout: "evexg": 0, 2024-03-28T22:48:20.125 INFO:teuthology.orchestra.run.smithi082.stdout: "evexd": 0, 2024-03-28T22:48:20.125 INFO:teuthology.orchestra.run.smithi082.stdout: "segadd": 0, 2024-03-28T22:48:20.125 INFO:teuthology.orchestra.run.smithi082.stdout: "segex": 0, 2024-03-28T22:48:20.125 INFO:teuthology.orchestra.run.smithi082.stdout: "segtrm": 0, 2024-03-28T22:48:20.125 INFO:teuthology.orchestra.run.smithi082.stdout: "seg": 2, 2024-03-28T22:48:20.125 INFO:teuthology.orchestra.run.smithi082.stdout: "segmjr": 2, 2024-03-28T22:48:20.125 INFO:teuthology.orchestra.run.smithi082.stdout: "segexg": 0, 2024-03-28T22:48:20.125 INFO:teuthology.orchestra.run.smithi082.stdout: "segexd": 0, 2024-03-28T22:48:20.126 INFO:teuthology.orchestra.run.smithi082.stdout: "expos": 4194304, 2024-03-28T22:48:20.126 INFO:teuthology.orchestra.run.smithi082.stdout: "wrpos": 4195956, 2024-03-28T22:48:20.126 INFO:teuthology.orchestra.run.smithi082.stdout: "rdpos": 4195815, 2024-03-28T22:48:20.126 INFO:teuthology.orchestra.run.smithi082.stdout: "jlat": { 2024-03-28T22:48:20.126 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.126 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.126 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.126 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.127 INFO:teuthology.orchestra.run.smithi082.stdout: "replayed": 6 2024-03-28T22:48:20.127 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.127 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_mem": { 2024-03-28T22:48:20.127 INFO:teuthology.orchestra.run.smithi082.stdout: "ino": 3, 2024-03-28T22:48:20.127 INFO:teuthology.orchestra.run.smithi082.stdout: "ino+": 3, 2024-03-28T22:48:20.127 INFO:teuthology.orchestra.run.smithi082.stdout: "ino-": 0, 2024-03-28T22:48:20.127 INFO:teuthology.orchestra.run.smithi082.stdout: "dir": 1, 2024-03-28T22:48:20.127 INFO:teuthology.orchestra.run.smithi082.stdout: "dir+": 1, 2024-03-28T22:48:20.127 INFO:teuthology.orchestra.run.smithi082.stdout: "dir-": 0, 2024-03-28T22:48:20.128 INFO:teuthology.orchestra.run.smithi082.stdout: "dn": 0, 2024-03-28T22:48:20.128 INFO:teuthology.orchestra.run.smithi082.stdout: "dn+": 0, 2024-03-28T22:48:20.128 INFO:teuthology.orchestra.run.smithi082.stdout: "dn-": 0, 2024-03-28T22:48:20.128 INFO:teuthology.orchestra.run.smithi082.stdout: "cap": 0, 2024-03-28T22:48:20.128 INFO:teuthology.orchestra.run.smithi082.stdout: "cap+": 0, 2024-03-28T22:48:20.128 INFO:teuthology.orchestra.run.smithi082.stdout: "cap-": 0, 2024-03-28T22:48:20.128 INFO:teuthology.orchestra.run.smithi082.stdout: "rss": 48776, 2024-03-28T22:48:20.128 INFO:teuthology.orchestra.run.smithi082.stdout: "heap": 215420 2024-03-28T22:48:20.129 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.129 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_server": { 2024-03-28T22:48:20.129 INFO:teuthology.orchestra.run.smithi082.stdout: "dispatch_client_request": 0, 2024-03-28T22:48:20.129 INFO:teuthology.orchestra.run.smithi082.stdout: "dispatch_server_request": 0, 2024-03-28T22:48:20.129 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_request": 0, 2024-03-28T22:48:20.129 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_session": 0, 2024-03-28T22:48:20.129 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_peer_request": 0, 2024-03-28T22:48:20.129 INFO:teuthology.orchestra.run.smithi082.stdout: "req_create_latency": { 2024-03-28T22:48:20.130 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.130 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.130 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.130 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.130 INFO:teuthology.orchestra.run.smithi082.stdout: "req_getattr_latency": { 2024-03-28T22:48:20.130 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.130 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.130 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.130 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.131 INFO:teuthology.orchestra.run.smithi082.stdout: "req_getfilelock_latency": { 2024-03-28T22:48:20.131 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.131 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.131 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.131 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.131 INFO:teuthology.orchestra.run.smithi082.stdout: "req_link_latency": { 2024-03-28T22:48:20.131 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.131 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.132 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.132 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookup_latency": { 2024-03-28T22:48:20.132 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.132 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.132 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.132 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.132 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookuphash_latency": { 2024-03-28T22:48:20.132 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.133 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.133 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.133 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookupino_latency": { 2024-03-28T22:48:20.133 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.133 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.133 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.133 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.133 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookupname_latency": { 2024-03-28T22:48:20.134 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.134 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.134 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.134 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.134 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookupparent_latency": { 2024-03-28T22:48:20.134 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.134 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.134 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.135 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.135 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookupsnap_latency": { 2024-03-28T22:48:20.135 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.135 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.135 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.135 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.135 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lssnap_latency": { 2024-03-28T22:48:20.135 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.135 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.136 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.136 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.136 INFO:teuthology.orchestra.run.smithi082.stdout: "req_mkdir_latency": { 2024-03-28T22:48:20.136 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.136 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.136 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.136 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.136 INFO:teuthology.orchestra.run.smithi082.stdout: "req_mknod_latency": { 2024-03-28T22:48:20.137 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.137 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.137 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.137 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.137 INFO:teuthology.orchestra.run.smithi082.stdout: "req_mksnap_latency": { 2024-03-28T22:48:20.137 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.138 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.138 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.138 INFO:teuthology.orchestra.run.smithi082.stdout: "req_open_latency": { 2024-03-28T22:48:20.138 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.138 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.138 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.138 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.139 INFO:teuthology.orchestra.run.smithi082.stdout: "req_readdir_latency": { 2024-03-28T22:48:20.139 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.139 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.139 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.139 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.139 INFO:teuthology.orchestra.run.smithi082.stdout: "req_rename_latency": { 2024-03-28T22:48:20.139 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.139 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.139 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.140 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.140 INFO:teuthology.orchestra.run.smithi082.stdout: "req_renamesnap_latency": { 2024-03-28T22:48:20.140 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.140 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.140 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.140 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.140 INFO:teuthology.orchestra.run.smithi082.stdout: "req_snapdiff_latency": { 2024-03-28T22:48:20.140 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.140 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.141 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.141 INFO:teuthology.orchestra.run.smithi082.stdout: "req_rmdir_latency": { 2024-03-28T22:48:20.141 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.141 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.141 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.141 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.141 INFO:teuthology.orchestra.run.smithi082.stdout: "req_rmsnap_latency": { 2024-03-28T22:48:20.141 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.142 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.142 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.142 INFO:teuthology.orchestra.run.smithi082.stdout: "req_rmxattr_latency": { 2024-03-28T22:48:20.142 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.142 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.142 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.142 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.143 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setattr_latency": { 2024-03-28T22:48:20.143 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.143 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.143 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.143 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.143 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setdirlayout_latency": { 2024-03-28T22:48:20.143 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.143 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.143 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.144 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.144 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setfilelock_latency": { 2024-03-28T22:48:20.144 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.144 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.144 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.144 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.144 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setlayout_latency": { 2024-03-28T22:48:20.144 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.145 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.145 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.145 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.145 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setxattr_latency": { 2024-03-28T22:48:20.145 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.145 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.145 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.145 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.145 INFO:teuthology.orchestra.run.smithi082.stdout: "req_symlink_latency": { 2024-03-28T22:48:20.145 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.146 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.146 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.146 INFO:teuthology.orchestra.run.smithi082.stdout: "req_unlink_latency": { 2024-03-28T22:48:20.146 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.146 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.146 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.146 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.147 INFO:teuthology.orchestra.run.smithi082.stdout: "cap_revoke_eviction": 0, 2024-03-28T22:48:20.147 INFO:teuthology.orchestra.run.smithi082.stdout: "cap_acquisition_throttle": 0, 2024-03-28T22:48:20.147 INFO:teuthology.orchestra.run.smithi082.stdout: "req_getvxattr_latency": { 2024-03-28T22:48:20.147 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.147 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.147 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.147 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:20.147 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.147 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_sessions": { 2024-03-28T22:48:20.148 INFO:teuthology.orchestra.run.smithi082.stdout: "session_count": 0, 2024-03-28T22:48:20.148 INFO:teuthology.orchestra.run.smithi082.stdout: "session_add": 2, 2024-03-28T22:48:20.148 INFO:teuthology.orchestra.run.smithi082.stdout: "session_remove": 2, 2024-03-28T22:48:20.148 INFO:teuthology.orchestra.run.smithi082.stdout: "sessions_open": 2, 2024-03-28T22:48:20.148 INFO:teuthology.orchestra.run.smithi082.stdout: "sessions_stale": 0, 2024-03-28T22:48:20.148 INFO:teuthology.orchestra.run.smithi082.stdout: "total_load": 0, 2024-03-28T22:48:20.148 INFO:teuthology.orchestra.run.smithi082.stdout: "average_load": 0, 2024-03-28T22:48:20.148 INFO:teuthology.orchestra.run.smithi082.stdout: "avg_session_uptime": 1756, 2024-03-28T22:48:20.149 INFO:teuthology.orchestra.run.smithi082.stdout: "mdthresh_evicted": 0 2024-03-28T22:48:20.149 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.149 INFO:teuthology.orchestra.run.smithi082.stdout: "mempool": { 2024-03-28T22:48:20.149 INFO:teuthology.orchestra.run.smithi082.stdout: "bloom_filter_bytes": 0, 2024-03-28T22:48:20.149 INFO:teuthology.orchestra.run.smithi082.stdout: "bloom_filter_items": 0, 2024-03-28T22:48:20.149 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T22:48:20.149 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_alloc_items": 0, 2024-03-28T22:48:20.150 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T22:48:20.150 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_data_items": 0, 2024-03-28T22:48:20.150 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T22:48:20.150 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T22:48:20.150 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T22:48:20.150 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T22:48:20.150 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T22:48:20.150 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_other_items": 0, 2024-03-28T22:48:20.151 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T22:48:20.151 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T22:48:20.151 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_extent_bytes": 0, 2024-03-28T22:48:20.151 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_extent_items": 0, 2024-03-28T22:48:20.151 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_blob_bytes": 0, 2024-03-28T22:48:20.151 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_blob_items": 0, 2024-03-28T22:48:20.151 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T22:48:20.151 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T22:48:20.152 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T22:48:20.152 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T22:48:20.152 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T22:48:20.152 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_fsck_items": 0, 2024-03-28T22:48:20.152 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_txc_bytes": 0, 2024-03-28T22:48:20.152 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_txc_items": 0, 2024-03-28T22:48:20.152 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T22:48:20.152 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T22:48:20.152 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_writing_bytes": 0, 2024-03-28T22:48:20.153 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_writing_items": 0, 2024-03-28T22:48:20.153 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_bytes": 0, 2024-03-28T22:48:20.153 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_items": 0, 2024-03-28T22:48:20.153 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T22:48:20.153 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_file_reader_items": 0, 2024-03-28T22:48:20.153 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T22:48:20.153 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_file_writer_items": 0, 2024-03-28T22:48:20.153 INFO:teuthology.orchestra.run.smithi082.stdout: "buffer_anon_bytes": 198197, 2024-03-28T22:48:20.153 INFO:teuthology.orchestra.run.smithi082.stdout: "buffer_anon_items": 56, 2024-03-28T22:48:20.154 INFO:teuthology.orchestra.run.smithi082.stdout: "buffer_meta_bytes": 0, 2024-03-28T22:48:20.154 INFO:teuthology.orchestra.run.smithi082.stdout: "buffer_meta_items": 0, 2024-03-28T22:48:20.154 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_bytes": 0, 2024-03-28T22:48:20.154 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_items": 0, 2024-03-28T22:48:20.154 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_mapbl_bytes": 0, 2024-03-28T22:48:20.154 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_mapbl_items": 0, 2024-03-28T22:48:20.154 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_pglog_bytes": 0, 2024-03-28T22:48:20.154 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_pglog_items": 0, 2024-03-28T22:48:20.155 INFO:teuthology.orchestra.run.smithi082.stdout: "osdmap_bytes": 9736, 2024-03-28T22:48:20.155 INFO:teuthology.orchestra.run.smithi082.stdout: "osdmap_items": 203, 2024-03-28T22:48:20.155 INFO:teuthology.orchestra.run.smithi082.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T22:48:20.155 INFO:teuthology.orchestra.run.smithi082.stdout: "osdmap_mapping_items": 0, 2024-03-28T22:48:20.155 INFO:teuthology.orchestra.run.smithi082.stdout: "pgmap_bytes": 0, 2024-03-28T22:48:20.155 INFO:teuthology.orchestra.run.smithi082.stdout: "pgmap_items": 0, 2024-03-28T22:48:20.155 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_co_bytes": 8464, 2024-03-28T22:48:20.155 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_co_items": 13, 2024-03-28T22:48:20.155 INFO:teuthology.orchestra.run.smithi082.stdout: "unittest_1_bytes": 0, 2024-03-28T22:48:20.156 INFO:teuthology.orchestra.run.smithi082.stdout: "unittest_1_items": 0, 2024-03-28T22:48:20.156 INFO:teuthology.orchestra.run.smithi082.stdout: "unittest_2_bytes": 0, 2024-03-28T22:48:20.156 INFO:teuthology.orchestra.run.smithi082.stdout: "unittest_2_items": 0 2024-03-28T22:48:20.156 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.156 INFO:teuthology.orchestra.run.smithi082.stdout: "objecter": { 2024-03-28T22:48:20.156 INFO:teuthology.orchestra.run.smithi082.stdout: "op_active": 0, 2024-03-28T22:48:20.156 INFO:teuthology.orchestra.run.smithi082.stdout: "op_laggy": 0, 2024-03-28T22:48:20.156 INFO:teuthology.orchestra.run.smithi082.stdout: "op_send": 2378, 2024-03-28T22:48:20.157 INFO:teuthology.orchestra.run.smithi082.stdout: "op_send_bytes": 21, 2024-03-28T22:48:20.157 INFO:teuthology.orchestra.run.smithi082.stdout: "op_resend": 3, 2024-03-28T22:48:20.157 INFO:teuthology.orchestra.run.smithi082.stdout: "op_reply": 2378, 2024-03-28T22:48:20.157 INFO:teuthology.orchestra.run.smithi082.stdout: "op_latency": { 2024-03-28T22:48:20.157 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 2378, 2024-03-28T22:48:20.157 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 2.148960691, 2024-03-28T22:48:20.157 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000903684 2024-03-28T22:48:20.157 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.157 INFO:teuthology.orchestra.run.smithi082.stdout: "op_inflight": 0, 2024-03-28T22:48:20.158 INFO:teuthology.orchestra.run.smithi082.stdout: "oplen_avg": { 2024-03-28T22:48:20.158 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 2378, 2024-03-28T22:48:20.158 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 2379 2024-03-28T22:48:20.158 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.158 INFO:teuthology.orchestra.run.smithi082.stdout: "op": 2378, 2024-03-28T22:48:20.158 INFO:teuthology.orchestra.run.smithi082.stdout: "op_r": 2378, 2024-03-28T22:48:20.158 INFO:teuthology.orchestra.run.smithi082.stdout: "op_w": 0, 2024-03-28T22:48:20.158 INFO:teuthology.orchestra.run.smithi082.stdout: "op_rmw": 0, 2024-03-28T22:48:20.159 INFO:teuthology.orchestra.run.smithi082.stdout: "op_pg": 0, 2024-03-28T22:48:20.159 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_stat": 1518, 2024-03-28T22:48:20.159 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_create": 0, 2024-03-28T22:48:20.159 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_read": 858, 2024-03-28T22:48:20.159 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_write": 0, 2024-03-28T22:48:20.159 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_writefull": 0, 2024-03-28T22:48:20.159 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_writesame": 0, 2024-03-28T22:48:20.159 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_append": 0, 2024-03-28T22:48:20.159 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_zero": 0, 2024-03-28T22:48:20.160 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_truncate": 0, 2024-03-28T22:48:20.160 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_delete": 0, 2024-03-28T22:48:20.160 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_mapext": 0, 2024-03-28T22:48:20.160 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_sparse_read": 0, 2024-03-28T22:48:20.160 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_clonerange": 0, 2024-03-28T22:48:20.160 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_getxattr": 1, 2024-03-28T22:48:20.160 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_setxattr": 0, 2024-03-28T22:48:20.160 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_cmpxattr": 0, 2024-03-28T22:48:20.161 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_rmxattr": 0, 2024-03-28T22:48:20.161 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_resetxattrs": 0, 2024-03-28T22:48:20.161 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_call": 0, 2024-03-28T22:48:20.161 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_watch": 0, 2024-03-28T22:48:20.161 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_notify": 0, 2024-03-28T22:48:20.161 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T22:48:20.161 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_pgls": 0, 2024-03-28T22:48:20.161 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_pgls_filter": 0, 2024-03-28T22:48:20.161 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_other": 0, 2024-03-28T22:48:20.162 INFO:teuthology.orchestra.run.smithi082.stdout: "linger_active": 0, 2024-03-28T22:48:20.162 INFO:teuthology.orchestra.run.smithi082.stdout: "linger_send": 0, 2024-03-28T22:48:20.162 INFO:teuthology.orchestra.run.smithi082.stdout: "linger_resend": 0, 2024-03-28T22:48:20.162 INFO:teuthology.orchestra.run.smithi082.stdout: "linger_ping": 0, 2024-03-28T22:48:20.162 INFO:teuthology.orchestra.run.smithi082.stdout: "poolop_active": 0, 2024-03-28T22:48:20.162 INFO:teuthology.orchestra.run.smithi082.stdout: "poolop_send": 0, 2024-03-28T22:48:20.162 INFO:teuthology.orchestra.run.smithi082.stdout: "poolop_resend": 0, 2024-03-28T22:48:20.162 INFO:teuthology.orchestra.run.smithi082.stdout: "poolstat_active": 0, 2024-03-28T22:48:20.163 INFO:teuthology.orchestra.run.smithi082.stdout: "poolstat_send": 0, 2024-03-28T22:48:20.163 INFO:teuthology.orchestra.run.smithi082.stdout: "poolstat_resend": 0, 2024-03-28T22:48:20.163 INFO:teuthology.orchestra.run.smithi082.stdout: "statfs_active": 0, 2024-03-28T22:48:20.163 INFO:teuthology.orchestra.run.smithi082.stdout: "statfs_send": 0, 2024-03-28T22:48:20.163 INFO:teuthology.orchestra.run.smithi082.stdout: "statfs_resend": 0, 2024-03-28T22:48:20.163 INFO:teuthology.orchestra.run.smithi082.stdout: "command_active": 0, 2024-03-28T22:48:20.163 INFO:teuthology.orchestra.run.smithi082.stdout: "command_send": 0, 2024-03-28T22:48:20.163 INFO:teuthology.orchestra.run.smithi082.stdout: "command_resend": 0, 2024-03-28T22:48:20.163 INFO:teuthology.orchestra.run.smithi082.stdout: "map_epoch": 110, 2024-03-28T22:48:20.164 INFO:teuthology.orchestra.run.smithi082.stdout: "map_full": 0, 2024-03-28T22:48:20.164 INFO:teuthology.orchestra.run.smithi082.stdout: "map_inc": 18, 2024-03-28T22:48:20.164 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_sessions": 6, 2024-03-28T22:48:20.164 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_session_open": 6, 2024-03-28T22:48:20.164 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_session_close": 0, 2024-03-28T22:48:20.164 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_laggy": 0, 2024-03-28T22:48:20.164 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_wr": 0, 2024-03-28T22:48:20.164 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_rd": 2, 2024-03-28T22:48:20.165 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_del": 0 2024-03-28T22:48:20.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.165 INFO:teuthology.orchestra.run.smithi082.stdout: "oft": { 2024-03-28T22:48:20.165 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_total_objs": 0, 2024-03-28T22:48:20.165 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_total_kv_pairs": 0, 2024-03-28T22:48:20.165 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_total_updates": 0, 2024-03-28T22:48:20.165 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_total_removes": 0 2024-03-28T22:48:20.165 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.165 INFO:teuthology.orchestra.run.smithi082.stdout: "osd-slow-ops": { 2024-03-28T22:48:20.166 INFO:teuthology.orchestra.run.smithi082.stdout: "slow_ops_count": 0 2024-03-28T22:48:20.166 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.166 INFO:teuthology.orchestra.run.smithi082.stdout: "purge_queue": { 2024-03-28T22:48:20.166 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executing_ops": 0, 2024-03-28T22:48:20.166 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T22:48:20.166 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executing": 0, 2024-03-28T22:48:20.166 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executing_high_water": 0, 2024-03-28T22:48:20.166 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executed_ops": 0, 2024-03-28T22:48:20.167 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executed": 0, 2024-03-28T22:48:20.167 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_item_in_journal": 0 2024-03-28T22:48:20.167 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.167 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T22:48:20.167 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-28T22:48:20.167 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 104857600, 2024-03-28T22:48:20.167 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-28T22:48:20.167 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 2668, 2024-03-28T22:48:20.168 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 2332603, 2024-03-28T22:48:20.168 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:20.168 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 2668, 2024-03-28T22:48:20.168 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 0, 2024-03-28T22:48:20.168 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 0, 2024-03-28T22:48:20.168 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 2668, 2024-03-28T22:48:20.168 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 2332603, 2024-03-28T22:48:20.168 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-28T22:48:20.168 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.169 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.169 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.169 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:20.169 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.169 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-objecter_bytes": { 2024-03-28T22:48:20.169 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-28T22:48:20.169 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 104857600, 2024-03-28T22:48:20.169 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-28T22:48:20.170 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 0, 2024-03-28T22:48:20.170 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 0, 2024-03-28T22:48:20.170 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:20.170 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 0, 2024-03-28T22:48:20.170 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 2378, 2024-03-28T22:48:20.170 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 1657, 2024-03-28T22:48:20.170 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 2378, 2024-03-28T22:48:20.171 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 1657, 2024-03-28T22:48:20.171 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-28T22:48:20.171 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.171 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.171 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.171 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:20.171 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.171 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-objecter_ops": { 2024-03-28T22:48:20.172 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-28T22:48:20.172 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 1024, 2024-03-28T22:48:20.172 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-28T22:48:20.172 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 0, 2024-03-28T22:48:20.172 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 0, 2024-03-28T22:48:20.172 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:20.172 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 0, 2024-03-28T22:48:20.172 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 2378, 2024-03-28T22:48:20.172 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 2378, 2024-03-28T22:48:20.173 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 2378, 2024-03-28T22:48:20.173 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 2378, 2024-03-28T22:48:20.173 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-28T22:48:20.173 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.173 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.173 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.173 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:20.173 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.173 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-write_buf_throttle": { 2024-03-28T22:48:20.173 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-28T22:48:20.174 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 3758096384, 2024-03-28T22:48:20.174 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-28T22:48:20.174 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 0, 2024-03-28T22:48:20.174 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 0, 2024-03-28T22:48:20.174 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:20.174 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 0, 2024-03-28T22:48:20.174 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 0, 2024-03-28T22:48:20.175 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 0, 2024-03-28T22:48:20.175 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 0, 2024-03-28T22:48:20.175 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 0, 2024-03-28T22:48:20.175 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-28T22:48:20.175 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.175 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.175 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.175 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:20.175 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-28T22:48:20.176 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-write_buf_throttle-0x557113cb26e0": { 2024-03-28T22:48:20.176 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-28T22:48:20.176 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 3758096384, 2024-03-28T22:48:20.176 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-28T22:48:20.176 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 0, 2024-03-28T22:48:20.176 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 0, 2024-03-28T22:48:20.176 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:20.176 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 0, 2024-03-28T22:48:20.176 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 0, 2024-03-28T22:48:20.177 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 0, 2024-03-28T22:48:20.177 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 0, 2024-03-28T22:48:20.177 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 0, 2024-03-28T22:48:20.177 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-28T22:48:20.177 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-28T22:48:20.177 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-28T22:48:20.177 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-28T22:48:20.177 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:20.177 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-28T22:48:20.178 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-28T22:48:21.020 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-03-28T22:48:21.020 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-03-28T22:48:21.020 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update=0 2024-03-28T22:48:21.020 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update_receipt=0 2024-03-28T22:48:21.021 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=0 2024-03-28T22:48:21.021 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-03-28T22:48:21.021 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T22:48:21.021 DEBUG:tasks.check_counter:Getting stats from c 2024-03-28T22:48:21.021 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-28T22:48:21.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:20 smithi172 ceph-mon[36566]: pgmap v350: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:48:21.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:20 smithi082 ceph-mon[36471]: pgmap v350: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:48:21.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:20 smithi043 ceph-mon[31823]: pgmap v350: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:48:21.294 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T22:48:23.017 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:22 smithi172 ceph-mon[36566]: pgmap v351: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:48:23.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:22 smithi082 ceph-mon[36471]: pgmap v351: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:48:23.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:22 smithi043 ceph-mon[31823]: pgmap v351: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T22:48:23.304 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-28T22:48:23.305 INFO:teuthology.orchestra.run.smithi172.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T22:48:23.305 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_messages": 42, 2024-03-28T22:48:23.305 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_messages": 179, 2024-03-28T22:48:23.305 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_bytes": 10066, 2024-03-28T22:48:23.305 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_bytes": 437700, 2024-03-28T22:48:23.305 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_created_connections": 18, 2024-03-28T22:48:23.306 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_active_connections": 2, 2024-03-28T22:48:23.306 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_total_time": 0.013716114, 2024-03-28T22:48:23.306 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_send_time": 0.004520740, 2024-03-28T22:48:23.306 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_recv_time": 0.004515440, 2024-03-28T22:48:23.306 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_fast_dispatch_time": 0.000623027, 2024-03-28T22:48:23.310 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:48:23.310 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 177, 2024-03-28T22:48:23.310 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.045883189, 2024-03-28T22:48:23.310 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000259227 2024-03-28T22:48:23.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.310 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:48:23.311 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 21, 2024-03-28T22:48:23.312 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000001897, 2024-03-28T22:48:23.312 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000090 2024-03-28T22:48:23.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.312 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_encrypted_bytes": 2096, 2024-03-28T22:48:23.312 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_encrypted_bytes": 331008 2024-03-28T22:48:23.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.312 INFO:teuthology.orchestra.run.smithi172.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T22:48:23.312 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_messages": 7627, 2024-03-28T22:48:23.312 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_messages": 7712, 2024-03-28T22:48:23.313 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_bytes": 2269034, 2024-03-28T22:48:23.313 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_bytes": 1396375, 2024-03-28T22:48:23.313 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_created_connections": 44, 2024-03-28T22:48:23.313 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_active_connections": 4, 2024-03-28T22:48:23.313 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_total_time": 0.428935929, 2024-03-28T22:48:23.313 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_send_time": 0.150088115, 2024-03-28T22:48:23.313 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_recv_time": 0.364183498, 2024-03-28T22:48:23.313 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_fast_dispatch_time": 0.000364931, 2024-03-28T22:48:23.313 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:48:23.314 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 7709, 2024-03-28T22:48:23.314 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.127386864, 2024-03-28T22:48:23.314 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000016524 2024-03-28T22:48:23.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.314 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:48:23.314 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 15203, 2024-03-28T22:48:23.314 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.002199686, 2024-03-28T22:48:23.314 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000144 2024-03-28T22:48:23.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.315 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_encrypted_bytes": 1248, 2024-03-28T22:48:23.315 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_encrypted_bytes": 233504 2024-03-28T22:48:23.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.315 INFO:teuthology.orchestra.run.smithi172.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T22:48:23.315 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_messages": 70602, 2024-03-28T22:48:23.315 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_messages": 35627, 2024-03-28T22:48:23.315 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_bytes": 25206393, 2024-03-28T22:48:23.315 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_bytes": 6106469, 2024-03-28T22:48:23.316 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_created_connections": 8, 2024-03-28T22:48:23.316 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_active_connections": 5, 2024-03-28T22:48:23.316 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_total_time": 3.110803125, 2024-03-28T22:48:23.316 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_send_time": 1.157777516, 2024-03-28T22:48:23.316 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_recv_time": 2.642384229, 2024-03-28T22:48:23.316 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_fast_dispatch_time": 0.007795543, 2024-03-28T22:48:23.316 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:48:23.316 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 35627, 2024-03-28T22:48:23.316 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.759482001, 2024-03-28T22:48:23.317 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000021317 2024-03-28T22:48:23.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.317 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:48:23.317 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 105482, 2024-03-28T22:48:23.317 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.015792277, 2024-03-28T22:48:23.317 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000149 2024-03-28T22:48:23.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.317 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_encrypted_bytes": 1937792, 2024-03-28T22:48:23.318 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_encrypted_bytes": 167040 2024-03-28T22:48:23.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.318 INFO:teuthology.orchestra.run.smithi172.stdout: "cct": { 2024-03-28T22:48:23.318 INFO:teuthology.orchestra.run.smithi172.stdout: "total_workers": 1, 2024-03-28T22:48:23.318 INFO:teuthology.orchestra.run.smithi172.stdout: "unhealthy_workers": 0 2024-03-28T22:48:23.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.318 INFO:teuthology.orchestra.run.smithi172.stdout: "finisher-MDSRank": { 2024-03-28T22:48:23.318 INFO:teuthology.orchestra.run.smithi172.stdout: "queue_len": 0, 2024-03-28T22:48:23.319 INFO:teuthology.orchestra.run.smithi172.stdout: "complete_latency": { 2024-03-28T22:48:23.319 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 62, 2024-03-28T22:48:23.319 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.001499131, 2024-03-28T22:48:23.319 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000024179 2024-03-28T22:48:23.319 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:23.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.319 INFO:teuthology.orchestra.run.smithi172.stdout: "finisher-PurgeQueue": { 2024-03-28T22:48:23.319 INFO:teuthology.orchestra.run.smithi172.stdout: "queue_len": 0, 2024-03-28T22:48:23.320 INFO:teuthology.orchestra.run.smithi172.stdout: "complete_latency": { 2024-03-28T22:48:23.320 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 2, 2024-03-28T22:48:23.320 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000012271, 2024-03-28T22:48:23.320 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000006135 2024-03-28T22:48:23.320 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:23.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.320 INFO:teuthology.orchestra.run.smithi172.stdout: "mds": { 2024-03-28T22:48:23.320 INFO:teuthology.orchestra.run.smithi172.stdout: "request": 7566, 2024-03-28T22:48:23.320 INFO:teuthology.orchestra.run.smithi172.stdout: "reply": 2, 2024-03-28T22:48:23.321 INFO:teuthology.orchestra.run.smithi172.stdout: "reply_latency": { 2024-03-28T22:48:23.321 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 2, 2024-03-28T22:48:23.321 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000232978, 2024-03-28T22:48:23.321 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000116489 2024-03-28T22:48:23.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.321 INFO:teuthology.orchestra.run.smithi172.stdout: "slow_reply": 0, 2024-03-28T22:48:23.321 INFO:teuthology.orchestra.run.smithi172.stdout: "forward": 7564, 2024-03-28T22:48:23.321 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_fetch_complete": 0, 2024-03-28T22:48:23.322 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_fetch_keys": 0, 2024-03-28T22:48:23.322 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_commit": 11, 2024-03-28T22:48:23.322 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_split": 0, 2024-03-28T22:48:23.322 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_merge": 0, 2024-03-28T22:48:23.322 INFO:teuthology.orchestra.run.smithi172.stdout: "inodes": 29, 2024-03-28T22:48:23.322 INFO:teuthology.orchestra.run.smithi172.stdout: "inodes_top": 7, 2024-03-28T22:48:23.322 INFO:teuthology.orchestra.run.smithi172.stdout: "inodes_bottom": 22, 2024-03-28T22:48:23.322 INFO:teuthology.orchestra.run.smithi172.stdout: "inodes_pin_tail": 0, 2024-03-28T22:48:23.323 INFO:teuthology.orchestra.run.smithi172.stdout: "inodes_pinned": 19, 2024-03-28T22:48:23.323 INFO:teuthology.orchestra.run.smithi172.stdout: "inodes_expired": 0, 2024-03-28T22:48:23.323 INFO:teuthology.orchestra.run.smithi172.stdout: "inodes_with_caps": 1, 2024-03-28T22:48:23.323 INFO:teuthology.orchestra.run.smithi172.stdout: "caps": 1, 2024-03-28T22:48:23.323 INFO:teuthology.orchestra.run.smithi172.stdout: "subtrees": 2, 2024-03-28T22:48:23.323 INFO:teuthology.orchestra.run.smithi172.stdout: "traverse": 7590, 2024-03-28T22:48:23.323 INFO:teuthology.orchestra.run.smithi172.stdout: "traverse_hit": 14, 2024-03-28T22:48:23.323 INFO:teuthology.orchestra.run.smithi172.stdout: "traverse_forward": 7562, 2024-03-28T22:48:23.323 INFO:teuthology.orchestra.run.smithi172.stdout: "traverse_discover": 12, 2024-03-28T22:48:23.324 INFO:teuthology.orchestra.run.smithi172.stdout: "traverse_dir_fetch": 0, 2024-03-28T22:48:23.324 INFO:teuthology.orchestra.run.smithi172.stdout: "traverse_remote_ino": 0, 2024-03-28T22:48:23.324 INFO:teuthology.orchestra.run.smithi172.stdout: "traverse_lock": 0, 2024-03-28T22:48:23.324 INFO:teuthology.orchestra.run.smithi172.stdout: "load_cent": 0, 2024-03-28T22:48:23.324 INFO:teuthology.orchestra.run.smithi172.stdout: "q": 0, 2024-03-28T22:48:23.324 INFO:teuthology.orchestra.run.smithi172.stdout: "exported": 0, 2024-03-28T22:48:23.324 INFO:teuthology.orchestra.run.smithi172.stdout: "exported_inodes": 0, 2024-03-28T22:48:23.324 INFO:teuthology.orchestra.run.smithi172.stdout: "imported": 0, 2024-03-28T22:48:23.325 INFO:teuthology.orchestra.run.smithi172.stdout: "imported_inodes": 0, 2024-03-28T22:48:23.325 INFO:teuthology.orchestra.run.smithi172.stdout: "openino_dir_fetch": 0, 2024-03-28T22:48:23.325 INFO:teuthology.orchestra.run.smithi172.stdout: "openino_backtrace_fetch": 0, 2024-03-28T22:48:23.326 INFO:teuthology.orchestra.run.smithi172.stdout: "openino_peer_discover": 0, 2024-03-28T22:48:23.326 INFO:teuthology.orchestra.run.smithi172.stdout: "root_rfiles": 61, 2024-03-28T22:48:23.326 INFO:teuthology.orchestra.run.smithi172.stdout: "root_rbytes": 21056780, 2024-03-28T22:48:23.326 INFO:teuthology.orchestra.run.smithi172.stdout: "root_rsnaps": 6, 2024-03-28T22:48:23.326 INFO:teuthology.orchestra.run.smithi172.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T22:48:23.326 INFO:teuthology.orchestra.run.smithi172.stdout: "scrub_set_tag": 0, 2024-03-28T22:48:23.326 INFO:teuthology.orchestra.run.smithi172.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T22:48:23.327 INFO:teuthology.orchestra.run.smithi172.stdout: "scrub_inotable_repaired": 0, 2024-03-28T22:48:23.327 INFO:teuthology.orchestra.run.smithi172.stdout: "scrub_dir_inodes": 0, 2024-03-28T22:48:23.327 INFO:teuthology.orchestra.run.smithi172.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T22:48:23.327 INFO:teuthology.orchestra.run.smithi172.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T22:48:23.327 INFO:teuthology.orchestra.run.smithi172.stdout: "scrub_file_inodes": 0, 2024-03-28T22:48:23.327 INFO:teuthology.orchestra.run.smithi172.stdout: "handle_inode_file_caps": 0, 2024-03-28T22:48:23.327 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T22:48:23.327 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph_cap_op_grant": 0, 2024-03-28T22:48:23.327 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T22:48:23.328 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T22:48:23.328 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T22:48:23.328 INFO:teuthology.orchestra.run.smithi172.stdout: "handle_client_caps": 0, 2024-03-28T22:48:23.328 INFO:teuthology.orchestra.run.smithi172.stdout: "handle_client_caps_dirty": 0, 2024-03-28T22:48:23.328 INFO:teuthology.orchestra.run.smithi172.stdout: "handle_client_cap_release": 0, 2024-03-28T22:48:23.328 INFO:teuthology.orchestra.run.smithi172.stdout: "process_request_cap_release": 2 2024-03-28T22:48:23.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.328 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_cache": { 2024-03-28T22:48:23.329 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_update": 0, 2024-03-28T22:48:23.329 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_update_receipt": 36, 2024-03-28T22:48:23.329 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_try_discover": 24, 2024-03-28T22:48:23.329 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_send_discover": 25, 2024-03-28T22:48:23.329 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_handle_discover": 0, 2024-03-28T22:48:23.329 INFO:teuthology.orchestra.run.smithi172.stdout: "num_strays": 0, 2024-03-28T22:48:23.329 INFO:teuthology.orchestra.run.smithi172.stdout: "num_strays_delayed": 0, 2024-03-28T22:48:23.329 INFO:teuthology.orchestra.run.smithi172.stdout: "num_strays_enqueuing": 0, 2024-03-28T22:48:23.329 INFO:teuthology.orchestra.run.smithi172.stdout: "strays_created": 0, 2024-03-28T22:48:23.330 INFO:teuthology.orchestra.run.smithi172.stdout: "strays_enqueued": 0, 2024-03-28T22:48:23.330 INFO:teuthology.orchestra.run.smithi172.stdout: "strays_reintegrated": 0, 2024-03-28T22:48:23.330 INFO:teuthology.orchestra.run.smithi172.stdout: "strays_migrated": 0, 2024-03-28T22:48:23.330 INFO:teuthology.orchestra.run.smithi172.stdout: "num_recovering_processing": 0, 2024-03-28T22:48:23.330 INFO:teuthology.orchestra.run.smithi172.stdout: "num_recovering_enqueued": 0, 2024-03-28T22:48:23.330 INFO:teuthology.orchestra.run.smithi172.stdout: "num_recovering_prioritized": 0, 2024-03-28T22:48:23.330 INFO:teuthology.orchestra.run.smithi172.stdout: "recovery_started": 0, 2024-03-28T22:48:23.330 INFO:teuthology.orchestra.run.smithi172.stdout: "recovery_completed": 0, 2024-03-28T22:48:23.331 INFO:teuthology.orchestra.run.smithi172.stdout: "ireq_quiesce_path": 0, 2024-03-28T22:48:23.331 INFO:teuthology.orchestra.run.smithi172.stdout: "ireq_quiesce_inode": 0, 2024-03-28T22:48:23.331 INFO:teuthology.orchestra.run.smithi172.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T22:48:23.331 INFO:teuthology.orchestra.run.smithi172.stdout: "ireq_exportdir": 0, 2024-03-28T22:48:23.331 INFO:teuthology.orchestra.run.smithi172.stdout: "ireq_flush": 0, 2024-03-28T22:48:23.331 INFO:teuthology.orchestra.run.smithi172.stdout: "ireq_fragmentdir": 0, 2024-03-28T22:48:23.331 INFO:teuthology.orchestra.run.smithi172.stdout: "ireq_fragstats": 0, 2024-03-28T22:48:23.331 INFO:teuthology.orchestra.run.smithi172.stdout: "ireq_inodestats": 0 2024-03-28T22:48:23.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.332 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_log": { 2024-03-28T22:48:23.332 INFO:teuthology.orchestra.run.smithi172.stdout: "evlrg": 0, 2024-03-28T22:48:23.332 INFO:teuthology.orchestra.run.smithi172.stdout: "evadd": 8, 2024-03-28T22:48:23.332 INFO:teuthology.orchestra.run.smithi172.stdout: "evex": 0, 2024-03-28T22:48:23.332 INFO:teuthology.orchestra.run.smithi172.stdout: "evtrm": 0, 2024-03-28T22:48:23.332 INFO:teuthology.orchestra.run.smithi172.stdout: "ev": 8, 2024-03-28T22:48:23.332 INFO:teuthology.orchestra.run.smithi172.stdout: "evexg": 0, 2024-03-28T22:48:23.332 INFO:teuthology.orchestra.run.smithi172.stdout: "evexd": 0, 2024-03-28T22:48:23.332 INFO:teuthology.orchestra.run.smithi172.stdout: "segadd": 2, 2024-03-28T22:48:23.333 INFO:teuthology.orchestra.run.smithi172.stdout: "segex": 0, 2024-03-28T22:48:23.333 INFO:teuthology.orchestra.run.smithi172.stdout: "segtrm": 0, 2024-03-28T22:48:23.333 INFO:teuthology.orchestra.run.smithi172.stdout: "seg": 2, 2024-03-28T22:48:23.333 INFO:teuthology.orchestra.run.smithi172.stdout: "segmjr": 2, 2024-03-28T22:48:23.333 INFO:teuthology.orchestra.run.smithi172.stdout: "segexg": 0, 2024-03-28T22:48:23.333 INFO:teuthology.orchestra.run.smithi172.stdout: "segexd": 0, 2024-03-28T22:48:23.333 INFO:teuthology.orchestra.run.smithi172.stdout: "expos": 4194304, 2024-03-28T22:48:23.333 INFO:teuthology.orchestra.run.smithi172.stdout: "wrpos": 4196653, 2024-03-28T22:48:23.334 INFO:teuthology.orchestra.run.smithi172.stdout: "rdpos": 0, 2024-03-28T22:48:23.334 INFO:teuthology.orchestra.run.smithi172.stdout: "jlat": { 2024-03-28T22:48:23.334 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 7, 2024-03-28T22:48:23.334 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.012090667, 2024-03-28T22:48:23.334 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.001727238 2024-03-28T22:48:23.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.334 INFO:teuthology.orchestra.run.smithi172.stdout: "replayed": 0 2024-03-28T22:48:23.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.334 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_mem": { 2024-03-28T22:48:23.335 INFO:teuthology.orchestra.run.smithi172.stdout: "ino": 32, 2024-03-28T22:48:23.335 INFO:teuthology.orchestra.run.smithi172.stdout: "ino+": 32, 2024-03-28T22:48:23.336 INFO:teuthology.orchestra.run.smithi172.stdout: "ino-": 0, 2024-03-28T22:48:23.336 INFO:teuthology.orchestra.run.smithi172.stdout: "dir": 38, 2024-03-28T22:48:23.336 INFO:teuthology.orchestra.run.smithi172.stdout: "dir+": 39, 2024-03-28T22:48:23.336 INFO:teuthology.orchestra.run.smithi172.stdout: "dir-": 1, 2024-03-28T22:48:23.336 INFO:teuthology.orchestra.run.smithi172.stdout: "dn": 29, 2024-03-28T22:48:23.336 INFO:teuthology.orchestra.run.smithi172.stdout: "dn+": 29, 2024-03-28T22:48:23.336 INFO:teuthology.orchestra.run.smithi172.stdout: "dn-": 0, 2024-03-28T22:48:23.337 INFO:teuthology.orchestra.run.smithi172.stdout: "cap": 1, 2024-03-28T22:48:23.337 INFO:teuthology.orchestra.run.smithi172.stdout: "cap+": 2, 2024-03-28T22:48:23.337 INFO:teuthology.orchestra.run.smithi172.stdout: "cap-": 1, 2024-03-28T22:48:23.337 INFO:teuthology.orchestra.run.smithi172.stdout: "rss": 52528, 2024-03-28T22:48:23.337 INFO:teuthology.orchestra.run.smithi172.stdout: "heap": 223612 2024-03-28T22:48:23.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.338 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_server": { 2024-03-28T22:48:23.338 INFO:teuthology.orchestra.run.smithi172.stdout: "dispatch_client_request": 7566, 2024-03-28T22:48:23.338 INFO:teuthology.orchestra.run.smithi172.stdout: "dispatch_server_request": 0, 2024-03-28T22:48:23.338 INFO:teuthology.orchestra.run.smithi172.stdout: "handle_client_request": 7566, 2024-03-28T22:48:23.338 INFO:teuthology.orchestra.run.smithi172.stdout: "handle_client_session": 27, 2024-03-28T22:48:23.338 INFO:teuthology.orchestra.run.smithi172.stdout: "handle_peer_request": 0, 2024-03-28T22:48:23.338 INFO:teuthology.orchestra.run.smithi172.stdout: "req_create_latency": { 2024-03-28T22:48:23.339 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.339 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.339 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.339 INFO:teuthology.orchestra.run.smithi172.stdout: "req_getattr_latency": { 2024-03-28T22:48:23.339 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 1, 2024-03-28T22:48:23.339 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000150177, 2024-03-28T22:48:23.339 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000150177 2024-03-28T22:48:23.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.340 INFO:teuthology.orchestra.run.smithi172.stdout: "req_getfilelock_latency": { 2024-03-28T22:48:23.340 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.340 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.340 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.340 INFO:teuthology.orchestra.run.smithi172.stdout: "req_link_latency": { 2024-03-28T22:48:23.340 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.341 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.341 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.341 INFO:teuthology.orchestra.run.smithi172.stdout: "req_lookup_latency": { 2024-03-28T22:48:23.341 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.341 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.341 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.342 INFO:teuthology.orchestra.run.smithi172.stdout: "req_lookuphash_latency": { 2024-03-28T22:48:23.342 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.342 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.342 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.342 INFO:teuthology.orchestra.run.smithi172.stdout: "req_lookupino_latency": { 2024-03-28T22:48:23.343 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 1, 2024-03-28T22:48:23.343 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000082801, 2024-03-28T22:48:23.343 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000082801 2024-03-28T22:48:23.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.343 INFO:teuthology.orchestra.run.smithi172.stdout: "req_lookupname_latency": { 2024-03-28T22:48:23.343 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.343 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.343 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.344 INFO:teuthology.orchestra.run.smithi172.stdout: "req_lookupparent_latency": { 2024-03-28T22:48:23.344 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.344 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.344 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.344 INFO:teuthology.orchestra.run.smithi172.stdout: "req_lookupsnap_latency": { 2024-03-28T22:48:23.344 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.345 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.345 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.345 INFO:teuthology.orchestra.run.smithi172.stdout: "req_lssnap_latency": { 2024-03-28T22:48:23.345 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.345 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.345 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.346 INFO:teuthology.orchestra.run.smithi172.stdout: "req_mkdir_latency": { 2024-03-28T22:48:23.347 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.347 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.347 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.347 INFO:teuthology.orchestra.run.smithi172.stdout: "req_mknod_latency": { 2024-03-28T22:48:23.347 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.348 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.348 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.348 INFO:teuthology.orchestra.run.smithi172.stdout: "req_mksnap_latency": { 2024-03-28T22:48:23.348 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.348 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.348 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.349 INFO:teuthology.orchestra.run.smithi172.stdout: "req_open_latency": { 2024-03-28T22:48:23.349 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.349 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.349 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.349 INFO:teuthology.orchestra.run.smithi172.stdout: "req_readdir_latency": { 2024-03-28T22:48:23.349 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.349 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.350 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.350 INFO:teuthology.orchestra.run.smithi172.stdout: "req_rename_latency": { 2024-03-28T22:48:23.350 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.350 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.350 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.350 INFO:teuthology.orchestra.run.smithi172.stdout: "req_renamesnap_latency": { 2024-03-28T22:48:23.350 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.351 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.351 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.351 INFO:teuthology.orchestra.run.smithi172.stdout: "req_snapdiff_latency": { 2024-03-28T22:48:23.351 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.351 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.352 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.352 INFO:teuthology.orchestra.run.smithi172.stdout: "req_rmdir_latency": { 2024-03-28T22:48:23.352 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.352 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.352 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.352 INFO:teuthology.orchestra.run.smithi172.stdout: "req_rmsnap_latency": { 2024-03-28T22:48:23.352 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.353 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.353 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.353 INFO:teuthology.orchestra.run.smithi172.stdout: "req_rmxattr_latency": { 2024-03-28T22:48:23.353 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.353 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.353 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.354 INFO:teuthology.orchestra.run.smithi172.stdout: "req_setattr_latency": { 2024-03-28T22:48:23.354 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.354 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.354 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.354 INFO:teuthology.orchestra.run.smithi172.stdout: "req_setdirlayout_latency": { 2024-03-28T22:48:23.354 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.354 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.355 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.355 INFO:teuthology.orchestra.run.smithi172.stdout: "req_setfilelock_latency": { 2024-03-28T22:48:23.355 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.355 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.355 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.355 INFO:teuthology.orchestra.run.smithi172.stdout: "req_setlayout_latency": { 2024-03-28T22:48:23.355 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.356 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.356 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.356 INFO:teuthology.orchestra.run.smithi172.stdout: "req_setxattr_latency": { 2024-03-28T22:48:23.356 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.356 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.356 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.357 INFO:teuthology.orchestra.run.smithi172.stdout: "req_symlink_latency": { 2024-03-28T22:48:23.357 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.357 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.357 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.357 INFO:teuthology.orchestra.run.smithi172.stdout: "req_unlink_latency": { 2024-03-28T22:48:23.357 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.357 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.357 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.358 INFO:teuthology.orchestra.run.smithi172.stdout: "cap_revoke_eviction": 0, 2024-03-28T22:48:23.358 INFO:teuthology.orchestra.run.smithi172.stdout: "cap_acquisition_throttle": 0, 2024-03-28T22:48:23.358 INFO:teuthology.orchestra.run.smithi172.stdout: "req_getvxattr_latency": { 2024-03-28T22:48:23.358 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.358 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.358 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.358 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:23.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.359 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_sessions": { 2024-03-28T22:48:23.359 INFO:teuthology.orchestra.run.smithi172.stdout: "session_count": 2, 2024-03-28T22:48:23.359 INFO:teuthology.orchestra.run.smithi172.stdout: "session_add": 4, 2024-03-28T22:48:23.359 INFO:teuthology.orchestra.run.smithi172.stdout: "session_remove": 2, 2024-03-28T22:48:23.359 INFO:teuthology.orchestra.run.smithi172.stdout: "sessions_open": 2, 2024-03-28T22:48:23.359 INFO:teuthology.orchestra.run.smithi172.stdout: "sessions_stale": 0, 2024-03-28T22:48:23.360 INFO:teuthology.orchestra.run.smithi172.stdout: "total_load": 1, 2024-03-28T22:48:23.360 INFO:teuthology.orchestra.run.smithi172.stdout: "average_load": 0, 2024-03-28T22:48:23.360 INFO:teuthology.orchestra.run.smithi172.stdout: "avg_session_uptime": 643, 2024-03-28T22:48:23.360 INFO:teuthology.orchestra.run.smithi172.stdout: "mdthresh_evicted": 0 2024-03-28T22:48:23.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.361 INFO:teuthology.orchestra.run.smithi172.stdout: "mempool": { 2024-03-28T22:48:23.361 INFO:teuthology.orchestra.run.smithi172.stdout: "bloom_filter_bytes": 0, 2024-03-28T22:48:23.361 INFO:teuthology.orchestra.run.smithi172.stdout: "bloom_filter_items": 0, 2024-03-28T22:48:23.361 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T22:48:23.361 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_alloc_items": 0, 2024-03-28T22:48:23.361 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T22:48:23.361 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_data_items": 0, 2024-03-28T22:48:23.362 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T22:48:23.362 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T22:48:23.362 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T22:48:23.362 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T22:48:23.362 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T22:48:23.362 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_other_items": 0, 2024-03-28T22:48:23.362 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T22:48:23.362 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T22:48:23.362 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_extent_bytes": 0, 2024-03-28T22:48:23.363 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_extent_items": 0, 2024-03-28T22:48:23.363 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_blob_bytes": 0, 2024-03-28T22:48:23.363 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_blob_items": 0, 2024-03-28T22:48:23.363 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T22:48:23.363 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T22:48:23.363 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T22:48:23.363 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T22:48:23.363 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T22:48:23.363 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_fsck_items": 0, 2024-03-28T22:48:23.364 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_txc_bytes": 0, 2024-03-28T22:48:23.364 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_txc_items": 0, 2024-03-28T22:48:23.364 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T22:48:23.364 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T22:48:23.364 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_writing_bytes": 0, 2024-03-28T22:48:23.364 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_writing_items": 0, 2024-03-28T22:48:23.364 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_bytes": 0, 2024-03-28T22:48:23.364 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_items": 0, 2024-03-28T22:48:23.364 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T22:48:23.365 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_file_reader_items": 0, 2024-03-28T22:48:23.365 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T22:48:23.365 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_file_writer_items": 0, 2024-03-28T22:48:23.365 INFO:teuthology.orchestra.run.smithi172.stdout: "buffer_anon_bytes": 251566, 2024-03-28T22:48:23.365 INFO:teuthology.orchestra.run.smithi172.stdout: "buffer_anon_items": 113, 2024-03-28T22:48:23.365 INFO:teuthology.orchestra.run.smithi172.stdout: "buffer_meta_bytes": 0, 2024-03-28T22:48:23.365 INFO:teuthology.orchestra.run.smithi172.stdout: "buffer_meta_items": 0, 2024-03-28T22:48:23.365 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_bytes": 0, 2024-03-28T22:48:23.366 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_items": 0, 2024-03-28T22:48:23.366 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mapbl_bytes": 0, 2024-03-28T22:48:23.366 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mapbl_items": 0, 2024-03-28T22:48:23.366 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pglog_bytes": 0, 2024-03-28T22:48:23.366 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pglog_items": 0, 2024-03-28T22:48:23.366 INFO:teuthology.orchestra.run.smithi172.stdout: "osdmap_bytes": 9736, 2024-03-28T22:48:23.366 INFO:teuthology.orchestra.run.smithi172.stdout: "osdmap_items": 203, 2024-03-28T22:48:23.366 INFO:teuthology.orchestra.run.smithi172.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T22:48:23.366 INFO:teuthology.orchestra.run.smithi172.stdout: "osdmap_mapping_items": 0, 2024-03-28T22:48:23.367 INFO:teuthology.orchestra.run.smithi172.stdout: "pgmap_bytes": 0, 2024-03-28T22:48:23.367 INFO:teuthology.orchestra.run.smithi172.stdout: "pgmap_items": 0, 2024-03-28T22:48:23.367 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_co_bytes": 146124, 2024-03-28T22:48:23.367 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_co_items": 770, 2024-03-28T22:48:23.367 INFO:teuthology.orchestra.run.smithi172.stdout: "unittest_1_bytes": 0, 2024-03-28T22:48:23.367 INFO:teuthology.orchestra.run.smithi172.stdout: "unittest_1_items": 0, 2024-03-28T22:48:23.367 INFO:teuthology.orchestra.run.smithi172.stdout: "unittest_2_bytes": 0, 2024-03-28T22:48:23.367 INFO:teuthology.orchestra.run.smithi172.stdout: "unittest_2_items": 0 2024-03-28T22:48:23.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.368 INFO:teuthology.orchestra.run.smithi172.stdout: "objecter": { 2024-03-28T22:48:23.368 INFO:teuthology.orchestra.run.smithi172.stdout: "op_active": 0, 2024-03-28T22:48:23.368 INFO:teuthology.orchestra.run.smithi172.stdout: "op_laggy": 0, 2024-03-28T22:48:23.368 INFO:teuthology.orchestra.run.smithi172.stdout: "op_send": 45, 2024-03-28T22:48:23.368 INFO:teuthology.orchestra.run.smithi172.stdout: "op_send_bytes": 12831, 2024-03-28T22:48:23.368 INFO:teuthology.orchestra.run.smithi172.stdout: "op_resend": 33, 2024-03-28T22:48:23.368 INFO:teuthology.orchestra.run.smithi172.stdout: "op_reply": 45, 2024-03-28T22:48:23.368 INFO:teuthology.orchestra.run.smithi172.stdout: "op_latency": { 2024-03-28T22:48:23.368 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 45, 2024-03-28T22:48:23.369 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.395990607, 2024-03-28T22:48:23.369 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.008799791 2024-03-28T22:48:23.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.369 INFO:teuthology.orchestra.run.smithi172.stdout: "op_inflight": 0, 2024-03-28T22:48:23.369 INFO:teuthology.orchestra.run.smithi172.stdout: "oplen_avg": { 2024-03-28T22:48:23.369 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 45, 2024-03-28T22:48:23.369 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 67 2024-03-28T22:48:23.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.370 INFO:teuthology.orchestra.run.smithi172.stdout: "op": 45, 2024-03-28T22:48:23.370 INFO:teuthology.orchestra.run.smithi172.stdout: "op_r": 0, 2024-03-28T22:48:23.370 INFO:teuthology.orchestra.run.smithi172.stdout: "op_w": 45, 2024-03-28T22:48:23.370 INFO:teuthology.orchestra.run.smithi172.stdout: "op_rmw": 0, 2024-03-28T22:48:23.370 INFO:teuthology.orchestra.run.smithi172.stdout: "op_pg": 0, 2024-03-28T22:48:23.370 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_stat": 0, 2024-03-28T22:48:23.370 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_create": 10, 2024-03-28T22:48:23.370 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_read": 0, 2024-03-28T22:48:23.370 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_write": 7, 2024-03-28T22:48:23.371 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_writefull": 9, 2024-03-28T22:48:23.371 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_writesame": 0, 2024-03-28T22:48:23.371 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_append": 0, 2024-03-28T22:48:23.371 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_zero": 0, 2024-03-28T22:48:23.371 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_truncate": 0, 2024-03-28T22:48:23.371 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_delete": 6, 2024-03-28T22:48:23.371 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_mapext": 0, 2024-03-28T22:48:23.371 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_sparse_read": 0, 2024-03-28T22:48:23.371 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_clonerange": 0, 2024-03-28T22:48:23.372 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_getxattr": 0, 2024-03-28T22:48:23.372 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_setxattr": 20, 2024-03-28T22:48:23.372 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_cmpxattr": 0, 2024-03-28T22:48:23.372 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_rmxattr": 0, 2024-03-28T22:48:23.372 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_resetxattrs": 0, 2024-03-28T22:48:23.372 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_call": 0, 2024-03-28T22:48:23.372 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_watch": 0, 2024-03-28T22:48:23.372 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_notify": 0, 2024-03-28T22:48:23.373 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T22:48:23.373 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_pgls": 0, 2024-03-28T22:48:23.373 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_pgls_filter": 0, 2024-03-28T22:48:23.373 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_other": 0, 2024-03-28T22:48:23.373 INFO:teuthology.orchestra.run.smithi172.stdout: "linger_active": 0, 2024-03-28T22:48:23.373 INFO:teuthology.orchestra.run.smithi172.stdout: "linger_send": 0, 2024-03-28T22:48:23.373 INFO:teuthology.orchestra.run.smithi172.stdout: "linger_resend": 0, 2024-03-28T22:48:23.373 INFO:teuthology.orchestra.run.smithi172.stdout: "linger_ping": 0, 2024-03-28T22:48:23.373 INFO:teuthology.orchestra.run.smithi172.stdout: "poolop_active": 0, 2024-03-28T22:48:23.374 INFO:teuthology.orchestra.run.smithi172.stdout: "poolop_send": 0, 2024-03-28T22:48:23.374 INFO:teuthology.orchestra.run.smithi172.stdout: "poolop_resend": 0, 2024-03-28T22:48:23.374 INFO:teuthology.orchestra.run.smithi172.stdout: "poolstat_active": 0, 2024-03-28T22:48:23.374 INFO:teuthology.orchestra.run.smithi172.stdout: "poolstat_send": 0, 2024-03-28T22:48:23.374 INFO:teuthology.orchestra.run.smithi172.stdout: "poolstat_resend": 0, 2024-03-28T22:48:23.374 INFO:teuthology.orchestra.run.smithi172.stdout: "statfs_active": 0, 2024-03-28T22:48:23.374 INFO:teuthology.orchestra.run.smithi172.stdout: "statfs_send": 0, 2024-03-28T22:48:23.374 INFO:teuthology.orchestra.run.smithi172.stdout: "statfs_resend": 0, 2024-03-28T22:48:23.374 INFO:teuthology.orchestra.run.smithi172.stdout: "command_active": 0, 2024-03-28T22:48:23.375 INFO:teuthology.orchestra.run.smithi172.stdout: "command_send": 0, 2024-03-28T22:48:23.375 INFO:teuthology.orchestra.run.smithi172.stdout: "command_resend": 0, 2024-03-28T22:48:23.375 INFO:teuthology.orchestra.run.smithi172.stdout: "map_epoch": 110, 2024-03-28T22:48:23.375 INFO:teuthology.orchestra.run.smithi172.stdout: "map_full": 0, 2024-03-28T22:48:23.375 INFO:teuthology.orchestra.run.smithi172.stdout: "map_inc": 18, 2024-03-28T22:48:23.375 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_sessions": 8, 2024-03-28T22:48:23.375 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_session_open": 8, 2024-03-28T22:48:23.375 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_session_close": 0, 2024-03-28T22:48:23.376 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_laggy": 0, 2024-03-28T22:48:23.376 INFO:teuthology.orchestra.run.smithi172.stdout: "omap_wr": 14, 2024-03-28T22:48:23.376 INFO:teuthology.orchestra.run.smithi172.stdout: "omap_rd": 0, 2024-03-28T22:48:23.376 INFO:teuthology.orchestra.run.smithi172.stdout: "omap_del": 1 2024-03-28T22:48:23.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.376 INFO:teuthology.orchestra.run.smithi172.stdout: "oft": { 2024-03-28T22:48:23.376 INFO:teuthology.orchestra.run.smithi172.stdout: "omap_total_objs": 1, 2024-03-28T22:48:23.376 INFO:teuthology.orchestra.run.smithi172.stdout: "omap_total_kv_pairs": 0, 2024-03-28T22:48:23.376 INFO:teuthology.orchestra.run.smithi172.stdout: "omap_total_updates": 0, 2024-03-28T22:48:23.377 INFO:teuthology.orchestra.run.smithi172.stdout: "omap_total_removes": 0 2024-03-28T22:48:23.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.377 INFO:teuthology.orchestra.run.smithi172.stdout: "osd-slow-ops": { 2024-03-28T22:48:23.377 INFO:teuthology.orchestra.run.smithi172.stdout: "slow_ops_count": 0 2024-03-28T22:48:23.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.377 INFO:teuthology.orchestra.run.smithi172.stdout: "purge_queue": { 2024-03-28T22:48:23.377 INFO:teuthology.orchestra.run.smithi172.stdout: "pq_executing_ops": 0, 2024-03-28T22:48:23.377 INFO:teuthology.orchestra.run.smithi172.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T22:48:23.377 INFO:teuthology.orchestra.run.smithi172.stdout: "pq_executing": 0, 2024-03-28T22:48:23.378 INFO:teuthology.orchestra.run.smithi172.stdout: "pq_executing_high_water": 0, 2024-03-28T22:48:23.378 INFO:teuthology.orchestra.run.smithi172.stdout: "pq_executed_ops": 0, 2024-03-28T22:48:23.378 INFO:teuthology.orchestra.run.smithi172.stdout: "pq_executed": 0, 2024-03-28T22:48:23.378 INFO:teuthology.orchestra.run.smithi172.stdout: "pq_item_in_journal": 0 2024-03-28T22:48:23.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.378 INFO:teuthology.orchestra.run.smithi172.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T22:48:23.378 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 0, 2024-03-28T22:48:23.378 INFO:teuthology.orchestra.run.smithi172.stdout: "max": 104857600, 2024-03-28T22:48:23.378 INFO:teuthology.orchestra.run.smithi172.stdout: "get_started": 0, 2024-03-28T22:48:23.379 INFO:teuthology.orchestra.run.smithi172.stdout: "get": 78267, 2024-03-28T22:48:23.379 INFO:teuthology.orchestra.run.smithi172.stdout: "get_sum": 20421258, 2024-03-28T22:48:23.379 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:23.379 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_success": 78267, 2024-03-28T22:48:23.379 INFO:teuthology.orchestra.run.smithi172.stdout: "take": 0, 2024-03-28T22:48:23.379 INFO:teuthology.orchestra.run.smithi172.stdout: "take_sum": 0, 2024-03-28T22:48:23.379 INFO:teuthology.orchestra.run.smithi172.stdout: "put": 78267, 2024-03-28T22:48:23.379 INFO:teuthology.orchestra.run.smithi172.stdout: "put_sum": 20421258, 2024-03-28T22:48:23.380 INFO:teuthology.orchestra.run.smithi172.stdout: "wait": { 2024-03-28T22:48:23.380 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.380 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.380 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.380 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:23.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.380 INFO:teuthology.orchestra.run.smithi172.stdout: "throttle-objecter_bytes": { 2024-03-28T22:48:23.380 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 0, 2024-03-28T22:48:23.380 INFO:teuthology.orchestra.run.smithi172.stdout: "max": 104857600, 2024-03-28T22:48:23.381 INFO:teuthology.orchestra.run.smithi172.stdout: "get_started": 0, 2024-03-28T22:48:23.381 INFO:teuthology.orchestra.run.smithi172.stdout: "get": 0, 2024-03-28T22:48:23.381 INFO:teuthology.orchestra.run.smithi172.stdout: "get_sum": 0, 2024-03-28T22:48:23.381 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:23.381 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_success": 0, 2024-03-28T22:48:23.381 INFO:teuthology.orchestra.run.smithi172.stdout: "take": 45, 2024-03-28T22:48:23.381 INFO:teuthology.orchestra.run.smithi172.stdout: "take_sum": 12831, 2024-03-28T22:48:23.381 INFO:teuthology.orchestra.run.smithi172.stdout: "put": 45, 2024-03-28T22:48:23.382 INFO:teuthology.orchestra.run.smithi172.stdout: "put_sum": 12831, 2024-03-28T22:48:23.382 INFO:teuthology.orchestra.run.smithi172.stdout: "wait": { 2024-03-28T22:48:23.382 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.382 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.382 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.382 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:23.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.382 INFO:teuthology.orchestra.run.smithi172.stdout: "throttle-objecter_ops": { 2024-03-28T22:48:23.382 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 0, 2024-03-28T22:48:23.383 INFO:teuthology.orchestra.run.smithi172.stdout: "max": 1024, 2024-03-28T22:48:23.383 INFO:teuthology.orchestra.run.smithi172.stdout: "get_started": 0, 2024-03-28T22:48:23.383 INFO:teuthology.orchestra.run.smithi172.stdout: "get": 0, 2024-03-28T22:48:23.383 INFO:teuthology.orchestra.run.smithi172.stdout: "get_sum": 0, 2024-03-28T22:48:23.383 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:23.383 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_success": 0, 2024-03-28T22:48:23.383 INFO:teuthology.orchestra.run.smithi172.stdout: "take": 45, 2024-03-28T22:48:23.383 INFO:teuthology.orchestra.run.smithi172.stdout: "take_sum": 45, 2024-03-28T22:48:23.384 INFO:teuthology.orchestra.run.smithi172.stdout: "put": 45, 2024-03-28T22:48:23.384 INFO:teuthology.orchestra.run.smithi172.stdout: "put_sum": 45, 2024-03-28T22:48:23.384 INFO:teuthology.orchestra.run.smithi172.stdout: "wait": { 2024-03-28T22:48:23.384 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.384 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.384 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.384 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:23.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.384 INFO:teuthology.orchestra.run.smithi172.stdout: "throttle-write_buf_throttle": { 2024-03-28T22:48:23.385 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 0, 2024-03-28T22:48:23.385 INFO:teuthology.orchestra.run.smithi172.stdout: "max": 3758096384, 2024-03-28T22:48:23.385 INFO:teuthology.orchestra.run.smithi172.stdout: "get_started": 0, 2024-03-28T22:48:23.385 INFO:teuthology.orchestra.run.smithi172.stdout: "get": 0, 2024-03-28T22:48:23.385 INFO:teuthology.orchestra.run.smithi172.stdout: "get_sum": 0, 2024-03-28T22:48:23.385 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:23.385 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_success": 0, 2024-03-28T22:48:23.385 INFO:teuthology.orchestra.run.smithi172.stdout: "take": 0, 2024-03-28T22:48:23.385 INFO:teuthology.orchestra.run.smithi172.stdout: "take_sum": 0, 2024-03-28T22:48:23.386 INFO:teuthology.orchestra.run.smithi172.stdout: "put": 0, 2024-03-28T22:48:23.386 INFO:teuthology.orchestra.run.smithi172.stdout: "put_sum": 0, 2024-03-28T22:48:23.386 INFO:teuthology.orchestra.run.smithi172.stdout: "wait": { 2024-03-28T22:48:23.386 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.386 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.386 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.386 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:23.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:23.387 INFO:teuthology.orchestra.run.smithi172.stdout: "throttle-write_buf_throttle-0x562846986500": { 2024-03-28T22:48:23.387 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 0, 2024-03-28T22:48:23.387 INFO:teuthology.orchestra.run.smithi172.stdout: "max": 3758096384, 2024-03-28T22:48:23.387 INFO:teuthology.orchestra.run.smithi172.stdout: "get_started": 0, 2024-03-28T22:48:23.387 INFO:teuthology.orchestra.run.smithi172.stdout: "get": 8, 2024-03-28T22:48:23.387 INFO:teuthology.orchestra.run.smithi172.stdout: "get_sum": 2349, 2024-03-28T22:48:23.387 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:23.387 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_success": 8, 2024-03-28T22:48:23.387 INFO:teuthology.orchestra.run.smithi172.stdout: "take": 0, 2024-03-28T22:48:23.388 INFO:teuthology.orchestra.run.smithi172.stdout: "take_sum": 0, 2024-03-28T22:48:23.388 INFO:teuthology.orchestra.run.smithi172.stdout: "put": 7, 2024-03-28T22:48:23.388 INFO:teuthology.orchestra.run.smithi172.stdout: "put_sum": 2349, 2024-03-28T22:48:23.388 INFO:teuthology.orchestra.run.smithi172.stdout: "wait": { 2024-03-28T22:48:23.388 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:23.388 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:23.388 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:23.388 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:23.389 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:23.389 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-28T22:48:24.076 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-28T22:48:24.077 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-03-28T22:48:24.077 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-03-28T22:48:24.077 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=36 2024-03-28T22:48:24.077 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-03-28T22:48:24.077 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-03-28T22:48:24.077 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T22:48:24.077 DEBUG:tasks.check_counter:Getting stats from f 2024-03-28T22:48:24.078 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-28T22:48:24.372 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T22:48:25.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:24 smithi172 ceph-mon[36566]: pgmap v352: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T22:48:25.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:24 smithi082 ceph-mon[36471]: pgmap v352: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T22:48:25.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:24 smithi043 ceph-mon[31823]: pgmap v352: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T22:48:26.339 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-28T22:48:26.339 INFO:teuthology.orchestra.run.smithi172.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T22:48:26.339 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_messages": 80, 2024-03-28T22:48:26.340 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_messages": 83, 2024-03-28T22:48:26.340 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_bytes": 16411, 2024-03-28T22:48:26.340 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_bytes": 482861, 2024-03-28T22:48:26.340 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_created_connections": 49, 2024-03-28T22:48:26.340 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_active_connections": 2, 2024-03-28T22:48:26.340 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_total_time": 0.018134437, 2024-03-28T22:48:26.340 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_send_time": 0.001752719, 2024-03-28T22:48:26.341 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_recv_time": 0.012533171, 2024-03-28T22:48:26.341 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_fast_dispatch_time": 0.000587040, 2024-03-28T22:48:26.341 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:48:26.341 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 81, 2024-03-28T22:48:26.341 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.054947330, 2024-03-28T22:48:26.341 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000678362 2024-03-28T22:48:26.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.342 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:48:26.342 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 62, 2024-03-28T22:48:26.342 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000006679, 2024-03-28T22:48:26.342 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000107 2024-03-28T22:48:26.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.342 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-03-28T22:48:26.343 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_encrypted_bytes": 4144 2024-03-28T22:48:26.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.343 INFO:teuthology.orchestra.run.smithi172.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T22:48:26.343 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_messages": 22, 2024-03-28T22:48:26.343 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_messages": 208, 2024-03-28T22:48:26.343 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_bytes": 7611, 2024-03-28T22:48:26.343 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_bytes": 567255, 2024-03-28T22:48:26.344 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_created_connections": 10, 2024-03-28T22:48:26.344 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_active_connections": 4, 2024-03-28T22:48:26.344 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_total_time": 0.013111395, 2024-03-28T22:48:26.344 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_send_time": 0.007089281, 2024-03-28T22:48:26.344 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_recv_time": 0.004042695, 2024-03-28T22:48:26.344 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_fast_dispatch_time": 0.000411571, 2024-03-28T22:48:26.344 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:48:26.344 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 206, 2024-03-28T22:48:26.345 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.038032384, 2024-03-28T22:48:26.345 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000184623 2024-03-28T22:48:26.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.345 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:48:26.345 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 8, 2024-03-28T22:48:26.345 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000001151, 2024-03-28T22:48:26.346 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000143 2024-03-28T22:48:26.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.346 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_encrypted_bytes": 4320, 2024-03-28T22:48:26.346 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_encrypted_bytes": 561584 2024-03-28T22:48:26.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.347 INFO:teuthology.orchestra.run.smithi172.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T22:48:26.347 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_messages": 78116, 2024-03-28T22:48:26.347 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_messages": 43154, 2024-03-28T22:48:26.347 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_bytes": 27432576, 2024-03-28T22:48:26.347 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_bytes": 6770761, 2024-03-28T22:48:26.347 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_created_connections": 10, 2024-03-28T22:48:26.348 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_active_connections": 5, 2024-03-28T22:48:26.348 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_total_time": 3.512283441, 2024-03-28T22:48:26.348 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_send_time": 1.308291921, 2024-03-28T22:48:26.348 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_recv_time": 2.976491294, 2024-03-28T22:48:26.348 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_fast_dispatch_time": 0.007685079, 2024-03-28T22:48:26.348 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:48:26.348 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 43153, 2024-03-28T22:48:26.348 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.821760655, 2024-03-28T22:48:26.348 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000019042 2024-03-28T22:48:26.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.349 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:48:26.349 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 120528, 2024-03-28T22:48:26.349 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.017914096, 2024-03-28T22:48:26.350 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000148 2024-03-28T22:48:26.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.350 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_encrypted_bytes": 1928048, 2024-03-28T22:48:26.350 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_encrypted_bytes": 161360 2024-03-28T22:48:26.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.350 INFO:teuthology.orchestra.run.smithi172.stdout: "cct": { 2024-03-28T22:48:26.350 INFO:teuthology.orchestra.run.smithi172.stdout: "total_workers": 1, 2024-03-28T22:48:26.351 INFO:teuthology.orchestra.run.smithi172.stdout: "unhealthy_workers": 0 2024-03-28T22:48:26.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.351 INFO:teuthology.orchestra.run.smithi172.stdout: "finisher-MDSRank": { 2024-03-28T22:48:26.351 INFO:teuthology.orchestra.run.smithi172.stdout: "queue_len": 0, 2024-03-28T22:48:26.351 INFO:teuthology.orchestra.run.smithi172.stdout: "complete_latency": { 2024-03-28T22:48:26.351 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 62, 2024-03-28T22:48:26.352 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.001486618, 2024-03-28T22:48:26.352 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000023977 2024-03-28T22:48:26.352 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:26.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.352 INFO:teuthology.orchestra.run.smithi172.stdout: "finisher-PurgeQueue": { 2024-03-28T22:48:26.353 INFO:teuthology.orchestra.run.smithi172.stdout: "queue_len": 0, 2024-03-28T22:48:26.353 INFO:teuthology.orchestra.run.smithi172.stdout: "complete_latency": { 2024-03-28T22:48:26.353 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 2, 2024-03-28T22:48:26.354 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000006844, 2024-03-28T22:48:26.354 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000003422 2024-03-28T22:48:26.354 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:26.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.355 INFO:teuthology.orchestra.run.smithi172.stdout: "mds": { 2024-03-28T22:48:26.355 INFO:teuthology.orchestra.run.smithi172.stdout: "request": 7533, 2024-03-28T22:48:26.355 INFO:teuthology.orchestra.run.smithi172.stdout: "reply": 3, 2024-03-28T22:48:26.356 INFO:teuthology.orchestra.run.smithi172.stdout: "reply_latency": { 2024-03-28T22:48:26.356 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 3, 2024-03-28T22:48:26.356 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.002383090, 2024-03-28T22:48:26.356 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000794363 2024-03-28T22:48:26.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.356 INFO:teuthology.orchestra.run.smithi172.stdout: "slow_reply": 0, 2024-03-28T22:48:26.357 INFO:teuthology.orchestra.run.smithi172.stdout: "forward": 7530, 2024-03-28T22:48:26.357 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_fetch_complete": 0, 2024-03-28T22:48:26.357 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_fetch_keys": 0, 2024-03-28T22:48:26.357 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_commit": 11, 2024-03-28T22:48:26.357 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_split": 0, 2024-03-28T22:48:26.358 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_merge": 0, 2024-03-28T22:48:26.358 INFO:teuthology.orchestra.run.smithi172.stdout: "inodes": 28, 2024-03-28T22:48:26.358 INFO:teuthology.orchestra.run.smithi172.stdout: "inodes_top": 6, 2024-03-28T22:48:26.358 INFO:teuthology.orchestra.run.smithi172.stdout: "inodes_bottom": 22, 2024-03-28T22:48:26.358 INFO:teuthology.orchestra.run.smithi172.stdout: "inodes_pin_tail": 0, 2024-03-28T22:48:26.359 INFO:teuthology.orchestra.run.smithi172.stdout: "inodes_pinned": 19, 2024-03-28T22:48:26.359 INFO:teuthology.orchestra.run.smithi172.stdout: "inodes_expired": 0, 2024-03-28T22:48:26.359 INFO:teuthology.orchestra.run.smithi172.stdout: "inodes_with_caps": 0, 2024-03-28T22:48:26.359 INFO:teuthology.orchestra.run.smithi172.stdout: "caps": 0, 2024-03-28T22:48:26.359 INFO:teuthology.orchestra.run.smithi172.stdout: "subtrees": 2, 2024-03-28T22:48:26.359 INFO:teuthology.orchestra.run.smithi172.stdout: "traverse": 7557, 2024-03-28T22:48:26.360 INFO:teuthology.orchestra.run.smithi172.stdout: "traverse_hit": 17, 2024-03-28T22:48:26.360 INFO:teuthology.orchestra.run.smithi172.stdout: "traverse_forward": 7529, 2024-03-28T22:48:26.360 INFO:teuthology.orchestra.run.smithi172.stdout: "traverse_discover": 11, 2024-03-28T22:48:26.360 INFO:teuthology.orchestra.run.smithi172.stdout: "traverse_dir_fetch": 0, 2024-03-28T22:48:26.360 INFO:teuthology.orchestra.run.smithi172.stdout: "traverse_remote_ino": 0, 2024-03-28T22:48:26.361 INFO:teuthology.orchestra.run.smithi172.stdout: "traverse_lock": 0, 2024-03-28T22:48:26.361 INFO:teuthology.orchestra.run.smithi172.stdout: "load_cent": 0, 2024-03-28T22:48:26.361 INFO:teuthology.orchestra.run.smithi172.stdout: "q": 0, 2024-03-28T22:48:26.361 INFO:teuthology.orchestra.run.smithi172.stdout: "exported": 0, 2024-03-28T22:48:26.361 INFO:teuthology.orchestra.run.smithi172.stdout: "exported_inodes": 0, 2024-03-28T22:48:26.361 INFO:teuthology.orchestra.run.smithi172.stdout: "imported": 0, 2024-03-28T22:48:26.361 INFO:teuthology.orchestra.run.smithi172.stdout: "imported_inodes": 0, 2024-03-28T22:48:26.362 INFO:teuthology.orchestra.run.smithi172.stdout: "openino_dir_fetch": 0, 2024-03-28T22:48:26.362 INFO:teuthology.orchestra.run.smithi172.stdout: "openino_backtrace_fetch": 0, 2024-03-28T22:48:26.362 INFO:teuthology.orchestra.run.smithi172.stdout: "openino_peer_discover": 0, 2024-03-28T22:48:26.363 INFO:teuthology.orchestra.run.smithi172.stdout: "root_rfiles": 61, 2024-03-28T22:48:26.363 INFO:teuthology.orchestra.run.smithi172.stdout: "root_rbytes": 21056780, 2024-03-28T22:48:26.364 INFO:teuthology.orchestra.run.smithi172.stdout: "root_rsnaps": 6, 2024-03-28T22:48:26.364 INFO:teuthology.orchestra.run.smithi172.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T22:48:26.364 INFO:teuthology.orchestra.run.smithi172.stdout: "scrub_set_tag": 0, 2024-03-28T22:48:26.365 INFO:teuthology.orchestra.run.smithi172.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T22:48:26.365 INFO:teuthology.orchestra.run.smithi172.stdout: "scrub_inotable_repaired": 0, 2024-03-28T22:48:26.365 INFO:teuthology.orchestra.run.smithi172.stdout: "scrub_dir_inodes": 0, 2024-03-28T22:48:26.365 INFO:teuthology.orchestra.run.smithi172.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T22:48:26.365 INFO:teuthology.orchestra.run.smithi172.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T22:48:26.365 INFO:teuthology.orchestra.run.smithi172.stdout: "scrub_file_inodes": 0, 2024-03-28T22:48:26.365 INFO:teuthology.orchestra.run.smithi172.stdout: "handle_inode_file_caps": 0, 2024-03-28T22:48:26.365 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T22:48:26.366 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph_cap_op_grant": 0, 2024-03-28T22:48:26.366 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T22:48:26.370 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T22:48:26.370 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T22:48:26.370 INFO:teuthology.orchestra.run.smithi172.stdout: "handle_client_caps": 0, 2024-03-28T22:48:26.370 INFO:teuthology.orchestra.run.smithi172.stdout: "handle_client_caps_dirty": 0, 2024-03-28T22:48:26.371 INFO:teuthology.orchestra.run.smithi172.stdout: "handle_client_cap_release": 0, 2024-03-28T22:48:26.371 INFO:teuthology.orchestra.run.smithi172.stdout: "process_request_cap_release": 0 2024-03-28T22:48:26.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.371 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_cache": { 2024-03-28T22:48:26.371 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_update": 0, 2024-03-28T22:48:26.371 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_update_receipt": 33, 2024-03-28T22:48:26.371 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_try_discover": 22, 2024-03-28T22:48:26.372 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_send_discover": 23, 2024-03-28T22:48:26.372 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_handle_discover": 0, 2024-03-28T22:48:26.372 INFO:teuthology.orchestra.run.smithi172.stdout: "num_strays": 0, 2024-03-28T22:48:26.372 INFO:teuthology.orchestra.run.smithi172.stdout: "num_strays_delayed": 0, 2024-03-28T22:48:26.372 INFO:teuthology.orchestra.run.smithi172.stdout: "num_strays_enqueuing": 0, 2024-03-28T22:48:26.372 INFO:teuthology.orchestra.run.smithi172.stdout: "strays_created": 0, 2024-03-28T22:48:26.372 INFO:teuthology.orchestra.run.smithi172.stdout: "strays_enqueued": 0, 2024-03-28T22:48:26.372 INFO:teuthology.orchestra.run.smithi172.stdout: "strays_reintegrated": 0, 2024-03-28T22:48:26.373 INFO:teuthology.orchestra.run.smithi172.stdout: "strays_migrated": 0, 2024-03-28T22:48:26.373 INFO:teuthology.orchestra.run.smithi172.stdout: "num_recovering_processing": 0, 2024-03-28T22:48:26.373 INFO:teuthology.orchestra.run.smithi172.stdout: "num_recovering_enqueued": 0, 2024-03-28T22:48:26.373 INFO:teuthology.orchestra.run.smithi172.stdout: "num_recovering_prioritized": 0, 2024-03-28T22:48:26.373 INFO:teuthology.orchestra.run.smithi172.stdout: "recovery_started": 0, 2024-03-28T22:48:26.373 INFO:teuthology.orchestra.run.smithi172.stdout: "recovery_completed": 0, 2024-03-28T22:48:26.373 INFO:teuthology.orchestra.run.smithi172.stdout: "ireq_quiesce_path": 0, 2024-03-28T22:48:26.373 INFO:teuthology.orchestra.run.smithi172.stdout: "ireq_quiesce_inode": 0, 2024-03-28T22:48:26.374 INFO:teuthology.orchestra.run.smithi172.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T22:48:26.374 INFO:teuthology.orchestra.run.smithi172.stdout: "ireq_exportdir": 0, 2024-03-28T22:48:26.374 INFO:teuthology.orchestra.run.smithi172.stdout: "ireq_flush": 0, 2024-03-28T22:48:26.374 INFO:teuthology.orchestra.run.smithi172.stdout: "ireq_fragmentdir": 0, 2024-03-28T22:48:26.374 INFO:teuthology.orchestra.run.smithi172.stdout: "ireq_fragstats": 0, 2024-03-28T22:48:26.374 INFO:teuthology.orchestra.run.smithi172.stdout: "ireq_inodestats": 0 2024-03-28T22:48:26.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.374 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_log": { 2024-03-28T22:48:26.375 INFO:teuthology.orchestra.run.smithi172.stdout: "evlrg": 0, 2024-03-28T22:48:26.375 INFO:teuthology.orchestra.run.smithi172.stdout: "evadd": 9, 2024-03-28T22:48:26.375 INFO:teuthology.orchestra.run.smithi172.stdout: "evex": 0, 2024-03-28T22:48:26.375 INFO:teuthology.orchestra.run.smithi172.stdout: "evtrm": 0, 2024-03-28T22:48:26.375 INFO:teuthology.orchestra.run.smithi172.stdout: "ev": 9, 2024-03-28T22:48:26.375 INFO:teuthology.orchestra.run.smithi172.stdout: "evexg": 0, 2024-03-28T22:48:26.375 INFO:teuthology.orchestra.run.smithi172.stdout: "evexd": 0, 2024-03-28T22:48:26.375 INFO:teuthology.orchestra.run.smithi172.stdout: "segadd": 2, 2024-03-28T22:48:26.376 INFO:teuthology.orchestra.run.smithi172.stdout: "segex": 0, 2024-03-28T22:48:26.376 INFO:teuthology.orchestra.run.smithi172.stdout: "segtrm": 0, 2024-03-28T22:48:26.376 INFO:teuthology.orchestra.run.smithi172.stdout: "seg": 2, 2024-03-28T22:48:26.376 INFO:teuthology.orchestra.run.smithi172.stdout: "segmjr": 2, 2024-03-28T22:48:26.376 INFO:teuthology.orchestra.run.smithi172.stdout: "segexg": 0, 2024-03-28T22:48:26.376 INFO:teuthology.orchestra.run.smithi172.stdout: "segexd": 0, 2024-03-28T22:48:26.377 INFO:teuthology.orchestra.run.smithi172.stdout: "expos": 4194304, 2024-03-28T22:48:26.377 INFO:teuthology.orchestra.run.smithi172.stdout: "wrpos": 4196794, 2024-03-28T22:48:26.377 INFO:teuthology.orchestra.run.smithi172.stdout: "rdpos": 0, 2024-03-28T22:48:26.377 INFO:teuthology.orchestra.run.smithi172.stdout: "jlat": { 2024-03-28T22:48:26.377 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 8, 2024-03-28T22:48:26.378 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.037472346, 2024-03-28T22:48:26.378 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.004684043 2024-03-28T22:48:26.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.378 INFO:teuthology.orchestra.run.smithi172.stdout: "replayed": 0 2024-03-28T22:48:26.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.378 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_mem": { 2024-03-28T22:48:26.379 INFO:teuthology.orchestra.run.smithi172.stdout: "ino": 31, 2024-03-28T22:48:26.379 INFO:teuthology.orchestra.run.smithi172.stdout: "ino+": 31, 2024-03-28T22:48:26.379 INFO:teuthology.orchestra.run.smithi172.stdout: "ino-": 0, 2024-03-28T22:48:26.379 INFO:teuthology.orchestra.run.smithi172.stdout: "dir": 37, 2024-03-28T22:48:26.379 INFO:teuthology.orchestra.run.smithi172.stdout: "dir+": 39, 2024-03-28T22:48:26.379 INFO:teuthology.orchestra.run.smithi172.stdout: "dir-": 2, 2024-03-28T22:48:26.379 INFO:teuthology.orchestra.run.smithi172.stdout: "dn": 28, 2024-03-28T22:48:26.379 INFO:teuthology.orchestra.run.smithi172.stdout: "dn+": 28, 2024-03-28T22:48:26.379 INFO:teuthology.orchestra.run.smithi172.stdout: "dn-": 0, 2024-03-28T22:48:26.380 INFO:teuthology.orchestra.run.smithi172.stdout: "cap": 0, 2024-03-28T22:48:26.380 INFO:teuthology.orchestra.run.smithi172.stdout: "cap+": 3, 2024-03-28T22:48:26.380 INFO:teuthology.orchestra.run.smithi172.stdout: "cap-": 3, 2024-03-28T22:48:26.380 INFO:teuthology.orchestra.run.smithi172.stdout: "rss": 53196, 2024-03-28T22:48:26.380 INFO:teuthology.orchestra.run.smithi172.stdout: "heap": 223612 2024-03-28T22:48:26.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.380 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_server": { 2024-03-28T22:48:26.380 INFO:teuthology.orchestra.run.smithi172.stdout: "dispatch_client_request": 7536, 2024-03-28T22:48:26.381 INFO:teuthology.orchestra.run.smithi172.stdout: "dispatch_server_request": 0, 2024-03-28T22:48:26.381 INFO:teuthology.orchestra.run.smithi172.stdout: "handle_client_request": 7533, 2024-03-28T22:48:26.381 INFO:teuthology.orchestra.run.smithi172.stdout: "handle_client_session": 20, 2024-03-28T22:48:26.381 INFO:teuthology.orchestra.run.smithi172.stdout: "handle_peer_request": 1, 2024-03-28T22:48:26.382 INFO:teuthology.orchestra.run.smithi172.stdout: "req_create_latency": { 2024-03-28T22:48:26.382 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.382 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.382 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.383 INFO:teuthology.orchestra.run.smithi172.stdout: "req_getattr_latency": { 2024-03-28T22:48:26.383 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 1, 2024-03-28T22:48:26.383 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.001775394, 2024-03-28T22:48:26.384 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.001775394 2024-03-28T22:48:26.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.384 INFO:teuthology.orchestra.run.smithi172.stdout: "req_getfilelock_latency": { 2024-03-28T22:48:26.384 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.384 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.385 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.385 INFO:teuthology.orchestra.run.smithi172.stdout: "req_link_latency": { 2024-03-28T22:48:26.385 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.385 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.385 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.386 INFO:teuthology.orchestra.run.smithi172.stdout: "req_lookup_latency": { 2024-03-28T22:48:26.386 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 2, 2024-03-28T22:48:26.386 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000607696, 2024-03-28T22:48:26.387 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000303848 2024-03-28T22:48:26.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.387 INFO:teuthology.orchestra.run.smithi172.stdout: "req_lookuphash_latency": { 2024-03-28T22:48:26.387 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.388 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.388 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.388 INFO:teuthology.orchestra.run.smithi172.stdout: "req_lookupino_latency": { 2024-03-28T22:48:26.389 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.389 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.389 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.389 INFO:teuthology.orchestra.run.smithi172.stdout: "req_lookupname_latency": { 2024-03-28T22:48:26.390 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.390 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.390 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.391 INFO:teuthology.orchestra.run.smithi172.stdout: "req_lookupparent_latency": { 2024-03-28T22:48:26.391 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.391 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.391 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.392 INFO:teuthology.orchestra.run.smithi172.stdout: "req_lookupsnap_latency": { 2024-03-28T22:48:26.392 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.392 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.393 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.393 INFO:teuthology.orchestra.run.smithi172.stdout: "req_lssnap_latency": { 2024-03-28T22:48:26.393 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.394 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.394 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.395 INFO:teuthology.orchestra.run.smithi172.stdout: "req_mkdir_latency": { 2024-03-28T22:48:26.396 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.396 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.396 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.397 INFO:teuthology.orchestra.run.smithi172.stdout: "req_mknod_latency": { 2024-03-28T22:48:26.397 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.397 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.397 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.398 INFO:teuthology.orchestra.run.smithi172.stdout: "req_mksnap_latency": { 2024-03-28T22:48:26.398 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.398 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.399 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.399 INFO:teuthology.orchestra.run.smithi172.stdout: "req_open_latency": { 2024-03-28T22:48:26.399 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.399 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.400 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.400 INFO:teuthology.orchestra.run.smithi172.stdout: "req_readdir_latency": { 2024-03-28T22:48:26.400 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.400 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.401 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.401 INFO:teuthology.orchestra.run.smithi172.stdout: "req_rename_latency": { 2024-03-28T22:48:26.401 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.402 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.402 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.402 INFO:teuthology.orchestra.run.smithi172.stdout: "req_renamesnap_latency": { 2024-03-28T22:48:26.402 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.402 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.402 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.403 INFO:teuthology.orchestra.run.smithi172.stdout: "req_snapdiff_latency": { 2024-03-28T22:48:26.403 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.403 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.403 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.403 INFO:teuthology.orchestra.run.smithi172.stdout: "req_rmdir_latency": { 2024-03-28T22:48:26.403 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.403 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.404 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.404 INFO:teuthology.orchestra.run.smithi172.stdout: "req_rmsnap_latency": { 2024-03-28T22:48:26.404 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.404 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.404 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.404 INFO:teuthology.orchestra.run.smithi172.stdout: "req_rmxattr_latency": { 2024-03-28T22:48:26.404 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.405 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.405 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.405 INFO:teuthology.orchestra.run.smithi172.stdout: "req_setattr_latency": { 2024-03-28T22:48:26.405 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.405 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.405 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.406 INFO:teuthology.orchestra.run.smithi172.stdout: "req_setdirlayout_latency": { 2024-03-28T22:48:26.406 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.406 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.406 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.406 INFO:teuthology.orchestra.run.smithi172.stdout: "req_setfilelock_latency": { 2024-03-28T22:48:26.406 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.406 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.407 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.407 INFO:teuthology.orchestra.run.smithi172.stdout: "req_setlayout_latency": { 2024-03-28T22:48:26.407 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.407 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.407 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.407 INFO:teuthology.orchestra.run.smithi172.stdout: "req_setxattr_latency": { 2024-03-28T22:48:26.408 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.408 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.408 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.408 INFO:teuthology.orchestra.run.smithi172.stdout: "req_symlink_latency": { 2024-03-28T22:48:26.408 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.408 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.408 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.409 INFO:teuthology.orchestra.run.smithi172.stdout: "req_unlink_latency": { 2024-03-28T22:48:26.409 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.409 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.409 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.409 INFO:teuthology.orchestra.run.smithi172.stdout: "cap_revoke_eviction": 0, 2024-03-28T22:48:26.409 INFO:teuthology.orchestra.run.smithi172.stdout: "cap_acquisition_throttle": 0, 2024-03-28T22:48:26.410 INFO:teuthology.orchestra.run.smithi172.stdout: "req_getvxattr_latency": { 2024-03-28T22:48:26.410 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.410 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.410 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.410 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:26.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.410 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_sessions": { 2024-03-28T22:48:26.410 INFO:teuthology.orchestra.run.smithi172.stdout: "session_count": 1, 2024-03-28T22:48:26.411 INFO:teuthology.orchestra.run.smithi172.stdout: "session_add": 4, 2024-03-28T22:48:26.411 INFO:teuthology.orchestra.run.smithi172.stdout: "session_remove": 3, 2024-03-28T22:48:26.411 INFO:teuthology.orchestra.run.smithi172.stdout: "sessions_open": 1, 2024-03-28T22:48:26.411 INFO:teuthology.orchestra.run.smithi172.stdout: "sessions_stale": 0, 2024-03-28T22:48:26.411 INFO:teuthology.orchestra.run.smithi172.stdout: "total_load": 2, 2024-03-28T22:48:26.411 INFO:teuthology.orchestra.run.smithi172.stdout: "average_load": 2, 2024-03-28T22:48:26.411 INFO:teuthology.orchestra.run.smithi172.stdout: "avg_session_uptime": 593, 2024-03-28T22:48:26.411 INFO:teuthology.orchestra.run.smithi172.stdout: "mdthresh_evicted": 0 2024-03-28T22:48:26.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.412 INFO:teuthology.orchestra.run.smithi172.stdout: "mempool": { 2024-03-28T22:48:26.412 INFO:teuthology.orchestra.run.smithi172.stdout: "bloom_filter_bytes": 0, 2024-03-28T22:48:26.412 INFO:teuthology.orchestra.run.smithi172.stdout: "bloom_filter_items": 0, 2024-03-28T22:48:26.412 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T22:48:26.412 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_alloc_items": 0, 2024-03-28T22:48:26.412 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T22:48:26.412 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_data_items": 0, 2024-03-28T22:48:26.412 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T22:48:26.413 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T22:48:26.413 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T22:48:26.413 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T22:48:26.413 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T22:48:26.413 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_other_items": 0, 2024-03-28T22:48:26.413 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T22:48:26.413 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T22:48:26.413 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_extent_bytes": 0, 2024-03-28T22:48:26.414 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_extent_items": 0, 2024-03-28T22:48:26.414 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_blob_bytes": 0, 2024-03-28T22:48:26.414 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_blob_items": 0, 2024-03-28T22:48:26.414 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T22:48:26.414 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T22:48:26.414 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T22:48:26.414 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T22:48:26.414 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T22:48:26.414 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_fsck_items": 0, 2024-03-28T22:48:26.415 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_txc_bytes": 0, 2024-03-28T22:48:26.415 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_txc_items": 0, 2024-03-28T22:48:26.415 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T22:48:26.415 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T22:48:26.415 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_writing_bytes": 0, 2024-03-28T22:48:26.415 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_writing_items": 0, 2024-03-28T22:48:26.415 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_bytes": 0, 2024-03-28T22:48:26.415 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_items": 0, 2024-03-28T22:48:26.416 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T22:48:26.416 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_file_reader_items": 0, 2024-03-28T22:48:26.416 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T22:48:26.416 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_file_writer_items": 0, 2024-03-28T22:48:26.416 INFO:teuthology.orchestra.run.smithi172.stdout: "buffer_anon_bytes": 251449, 2024-03-28T22:48:26.416 INFO:teuthology.orchestra.run.smithi172.stdout: "buffer_anon_items": 112, 2024-03-28T22:48:26.416 INFO:teuthology.orchestra.run.smithi172.stdout: "buffer_meta_bytes": 0, 2024-03-28T22:48:26.416 INFO:teuthology.orchestra.run.smithi172.stdout: "buffer_meta_items": 0, 2024-03-28T22:48:26.416 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_bytes": 0, 2024-03-28T22:48:26.417 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_items": 0, 2024-03-28T22:48:26.417 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mapbl_bytes": 0, 2024-03-28T22:48:26.417 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mapbl_items": 0, 2024-03-28T22:48:26.417 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pglog_bytes": 0, 2024-03-28T22:48:26.417 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pglog_items": 0, 2024-03-28T22:48:26.417 INFO:teuthology.orchestra.run.smithi172.stdout: "osdmap_bytes": 9736, 2024-03-28T22:48:26.417 INFO:teuthology.orchestra.run.smithi172.stdout: "osdmap_items": 203, 2024-03-28T22:48:26.417 INFO:teuthology.orchestra.run.smithi172.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T22:48:26.418 INFO:teuthology.orchestra.run.smithi172.stdout: "osdmap_mapping_items": 0, 2024-03-28T22:48:26.418 INFO:teuthology.orchestra.run.smithi172.stdout: "pgmap_bytes": 0, 2024-03-28T22:48:26.418 INFO:teuthology.orchestra.run.smithi172.stdout: "pgmap_items": 0, 2024-03-28T22:48:26.418 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_co_bytes": 141668, 2024-03-28T22:48:26.418 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_co_items": 766, 2024-03-28T22:48:26.418 INFO:teuthology.orchestra.run.smithi172.stdout: "unittest_1_bytes": 0, 2024-03-28T22:48:26.418 INFO:teuthology.orchestra.run.smithi172.stdout: "unittest_1_items": 0, 2024-03-28T22:48:26.418 INFO:teuthology.orchestra.run.smithi172.stdout: "unittest_2_bytes": 0, 2024-03-28T22:48:26.419 INFO:teuthology.orchestra.run.smithi172.stdout: "unittest_2_items": 0 2024-03-28T22:48:26.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.419 INFO:teuthology.orchestra.run.smithi172.stdout: "objecter": { 2024-03-28T22:48:26.419 INFO:teuthology.orchestra.run.smithi172.stdout: "op_active": 0, 2024-03-28T22:48:26.419 INFO:teuthology.orchestra.run.smithi172.stdout: "op_laggy": 0, 2024-03-28T22:48:26.419 INFO:teuthology.orchestra.run.smithi172.stdout: "op_send": 47, 2024-03-28T22:48:26.419 INFO:teuthology.orchestra.run.smithi172.stdout: "op_send_bytes": 13107, 2024-03-28T22:48:26.419 INFO:teuthology.orchestra.run.smithi172.stdout: "op_resend": 33, 2024-03-28T22:48:26.420 INFO:teuthology.orchestra.run.smithi172.stdout: "op_reply": 47, 2024-03-28T22:48:26.420 INFO:teuthology.orchestra.run.smithi172.stdout: "op_latency": { 2024-03-28T22:48:26.420 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 47, 2024-03-28T22:48:26.420 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.357991534, 2024-03-28T22:48:26.420 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.007616841 2024-03-28T22:48:26.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.420 INFO:teuthology.orchestra.run.smithi172.stdout: "op_inflight": 0, 2024-03-28T22:48:26.420 INFO:teuthology.orchestra.run.smithi172.stdout: "oplen_avg": { 2024-03-28T22:48:26.420 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 47, 2024-03-28T22:48:26.421 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 71 2024-03-28T22:48:26.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.421 INFO:teuthology.orchestra.run.smithi172.stdout: "op": 47, 2024-03-28T22:48:26.421 INFO:teuthology.orchestra.run.smithi172.stdout: "op_r": 0, 2024-03-28T22:48:26.421 INFO:teuthology.orchestra.run.smithi172.stdout: "op_w": 47, 2024-03-28T22:48:26.421 INFO:teuthology.orchestra.run.smithi172.stdout: "op_rmw": 0, 2024-03-28T22:48:26.421 INFO:teuthology.orchestra.run.smithi172.stdout: "op_pg": 0, 2024-03-28T22:48:26.421 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_stat": 0, 2024-03-28T22:48:26.421 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_create": 10, 2024-03-28T22:48:26.422 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_read": 0, 2024-03-28T22:48:26.422 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_write": 8, 2024-03-28T22:48:26.422 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_writefull": 8, 2024-03-28T22:48:26.422 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_writesame": 0, 2024-03-28T22:48:26.422 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_append": 0, 2024-03-28T22:48:26.422 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_zero": 0, 2024-03-28T22:48:26.422 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_truncate": 0, 2024-03-28T22:48:26.422 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_delete": 6, 2024-03-28T22:48:26.423 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_mapext": 0, 2024-03-28T22:48:26.423 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_sparse_read": 0, 2024-03-28T22:48:26.423 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_clonerange": 0, 2024-03-28T22:48:26.423 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_getxattr": 0, 2024-03-28T22:48:26.423 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_setxattr": 20, 2024-03-28T22:48:26.423 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_cmpxattr": 0, 2024-03-28T22:48:26.423 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_rmxattr": 0, 2024-03-28T22:48:26.423 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_resetxattrs": 0, 2024-03-28T22:48:26.423 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_call": 0, 2024-03-28T22:48:26.424 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_watch": 0, 2024-03-28T22:48:26.424 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_notify": 0, 2024-03-28T22:48:26.424 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T22:48:26.424 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_pgls": 0, 2024-03-28T22:48:26.424 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_pgls_filter": 0, 2024-03-28T22:48:26.424 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_other": 0, 2024-03-28T22:48:26.424 INFO:teuthology.orchestra.run.smithi172.stdout: "linger_active": 0, 2024-03-28T22:48:26.424 INFO:teuthology.orchestra.run.smithi172.stdout: "linger_send": 0, 2024-03-28T22:48:26.424 INFO:teuthology.orchestra.run.smithi172.stdout: "linger_resend": 0, 2024-03-28T22:48:26.425 INFO:teuthology.orchestra.run.smithi172.stdout: "linger_ping": 0, 2024-03-28T22:48:26.425 INFO:teuthology.orchestra.run.smithi172.stdout: "poolop_active": 0, 2024-03-28T22:48:26.425 INFO:teuthology.orchestra.run.smithi172.stdout: "poolop_send": 0, 2024-03-28T22:48:26.425 INFO:teuthology.orchestra.run.smithi172.stdout: "poolop_resend": 0, 2024-03-28T22:48:26.425 INFO:teuthology.orchestra.run.smithi172.stdout: "poolstat_active": 0, 2024-03-28T22:48:26.425 INFO:teuthology.orchestra.run.smithi172.stdout: "poolstat_send": 0, 2024-03-28T22:48:26.425 INFO:teuthology.orchestra.run.smithi172.stdout: "poolstat_resend": 0, 2024-03-28T22:48:26.425 INFO:teuthology.orchestra.run.smithi172.stdout: "statfs_active": 0, 2024-03-28T22:48:26.426 INFO:teuthology.orchestra.run.smithi172.stdout: "statfs_send": 0, 2024-03-28T22:48:26.426 INFO:teuthology.orchestra.run.smithi172.stdout: "statfs_resend": 0, 2024-03-28T22:48:26.426 INFO:teuthology.orchestra.run.smithi172.stdout: "command_active": 0, 2024-03-28T22:48:26.426 INFO:teuthology.orchestra.run.smithi172.stdout: "command_send": 0, 2024-03-28T22:48:26.426 INFO:teuthology.orchestra.run.smithi172.stdout: "command_resend": 0, 2024-03-28T22:48:26.426 INFO:teuthology.orchestra.run.smithi172.stdout: "map_epoch": 110, 2024-03-28T22:48:26.426 INFO:teuthology.orchestra.run.smithi172.stdout: "map_full": 0, 2024-03-28T22:48:26.426 INFO:teuthology.orchestra.run.smithi172.stdout: "map_inc": 18, 2024-03-28T22:48:26.427 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_sessions": 8, 2024-03-28T22:48:26.427 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_session_open": 8, 2024-03-28T22:48:26.427 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_session_close": 0, 2024-03-28T22:48:26.427 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_laggy": 0, 2024-03-28T22:48:26.427 INFO:teuthology.orchestra.run.smithi172.stdout: "omap_wr": 17, 2024-03-28T22:48:26.427 INFO:teuthology.orchestra.run.smithi172.stdout: "omap_rd": 0, 2024-03-28T22:48:26.427 INFO:teuthology.orchestra.run.smithi172.stdout: "omap_del": 2 2024-03-28T22:48:26.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.427 INFO:teuthology.orchestra.run.smithi172.stdout: "oft": { 2024-03-28T22:48:26.428 INFO:teuthology.orchestra.run.smithi172.stdout: "omap_total_objs": 1, 2024-03-28T22:48:26.428 INFO:teuthology.orchestra.run.smithi172.stdout: "omap_total_kv_pairs": 0, 2024-03-28T22:48:26.428 INFO:teuthology.orchestra.run.smithi172.stdout: "omap_total_updates": 0, 2024-03-28T22:48:26.428 INFO:teuthology.orchestra.run.smithi172.stdout: "omap_total_removes": 0 2024-03-28T22:48:26.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.428 INFO:teuthology.orchestra.run.smithi172.stdout: "osd-slow-ops": { 2024-03-28T22:48:26.428 INFO:teuthology.orchestra.run.smithi172.stdout: "slow_ops_count": 0 2024-03-28T22:48:26.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.429 INFO:teuthology.orchestra.run.smithi172.stdout: "purge_queue": { 2024-03-28T22:48:26.429 INFO:teuthology.orchestra.run.smithi172.stdout: "pq_executing_ops": 0, 2024-03-28T22:48:26.429 INFO:teuthology.orchestra.run.smithi172.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T22:48:26.429 INFO:teuthology.orchestra.run.smithi172.stdout: "pq_executing": 0, 2024-03-28T22:48:26.429 INFO:teuthology.orchestra.run.smithi172.stdout: "pq_executing_high_water": 0, 2024-03-28T22:48:26.429 INFO:teuthology.orchestra.run.smithi172.stdout: "pq_executed_ops": 0, 2024-03-28T22:48:26.429 INFO:teuthology.orchestra.run.smithi172.stdout: "pq_executed": 0, 2024-03-28T22:48:26.429 INFO:teuthology.orchestra.run.smithi172.stdout: "pq_item_in_journal": 0 2024-03-28T22:48:26.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.430 INFO:teuthology.orchestra.run.smithi172.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T22:48:26.430 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 0, 2024-03-28T22:48:26.430 INFO:teuthology.orchestra.run.smithi172.stdout: "max": 104857600, 2024-03-28T22:48:26.430 INFO:teuthology.orchestra.run.smithi172.stdout: "get_started": 0, 2024-03-28T22:48:26.430 INFO:teuthology.orchestra.run.smithi172.stdout: "get": 78214, 2024-03-28T22:48:26.430 INFO:teuthology.orchestra.run.smithi172.stdout: "get_sum": 20397419, 2024-03-28T22:48:26.430 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:26.430 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_success": 78214, 2024-03-28T22:48:26.431 INFO:teuthology.orchestra.run.smithi172.stdout: "take": 0, 2024-03-28T22:48:26.431 INFO:teuthology.orchestra.run.smithi172.stdout: "take_sum": 0, 2024-03-28T22:48:26.431 INFO:teuthology.orchestra.run.smithi172.stdout: "put": 78214, 2024-03-28T22:48:26.431 INFO:teuthology.orchestra.run.smithi172.stdout: "put_sum": 20397419, 2024-03-28T22:48:26.431 INFO:teuthology.orchestra.run.smithi172.stdout: "wait": { 2024-03-28T22:48:26.431 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.431 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.431 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.432 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:26.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.432 INFO:teuthology.orchestra.run.smithi172.stdout: "throttle-objecter_bytes": { 2024-03-28T22:48:26.432 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 0, 2024-03-28T22:48:26.432 INFO:teuthology.orchestra.run.smithi172.stdout: "max": 104857600, 2024-03-28T22:48:26.432 INFO:teuthology.orchestra.run.smithi172.stdout: "get_started": 0, 2024-03-28T22:48:26.432 INFO:teuthology.orchestra.run.smithi172.stdout: "get": 0, 2024-03-28T22:48:26.432 INFO:teuthology.orchestra.run.smithi172.stdout: "get_sum": 0, 2024-03-28T22:48:26.432 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:26.433 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_success": 0, 2024-03-28T22:48:26.433 INFO:teuthology.orchestra.run.smithi172.stdout: "take": 47, 2024-03-28T22:48:26.433 INFO:teuthology.orchestra.run.smithi172.stdout: "take_sum": 13107, 2024-03-28T22:48:26.433 INFO:teuthology.orchestra.run.smithi172.stdout: "put": 47, 2024-03-28T22:48:26.433 INFO:teuthology.orchestra.run.smithi172.stdout: "put_sum": 13107, 2024-03-28T22:48:26.433 INFO:teuthology.orchestra.run.smithi172.stdout: "wait": { 2024-03-28T22:48:26.433 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.434 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.434 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.434 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:26.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.434 INFO:teuthology.orchestra.run.smithi172.stdout: "throttle-objecter_ops": { 2024-03-28T22:48:26.434 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 0, 2024-03-28T22:48:26.434 INFO:teuthology.orchestra.run.smithi172.stdout: "max": 1024, 2024-03-28T22:48:26.434 INFO:teuthology.orchestra.run.smithi172.stdout: "get_started": 0, 2024-03-28T22:48:26.434 INFO:teuthology.orchestra.run.smithi172.stdout: "get": 0, 2024-03-28T22:48:26.435 INFO:teuthology.orchestra.run.smithi172.stdout: "get_sum": 0, 2024-03-28T22:48:26.435 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:26.435 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_success": 0, 2024-03-28T22:48:26.435 INFO:teuthology.orchestra.run.smithi172.stdout: "take": 47, 2024-03-28T22:48:26.435 INFO:teuthology.orchestra.run.smithi172.stdout: "take_sum": 47, 2024-03-28T22:48:26.435 INFO:teuthology.orchestra.run.smithi172.stdout: "put": 47, 2024-03-28T22:48:26.435 INFO:teuthology.orchestra.run.smithi172.stdout: "put_sum": 47, 2024-03-28T22:48:26.435 INFO:teuthology.orchestra.run.smithi172.stdout: "wait": { 2024-03-28T22:48:26.436 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.436 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.436 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.436 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:26.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.436 INFO:teuthology.orchestra.run.smithi172.stdout: "throttle-write_buf_throttle": { 2024-03-28T22:48:26.436 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 0, 2024-03-28T22:48:26.436 INFO:teuthology.orchestra.run.smithi172.stdout: "max": 3758096384, 2024-03-28T22:48:26.437 INFO:teuthology.orchestra.run.smithi172.stdout: "get_started": 0, 2024-03-28T22:48:26.437 INFO:teuthology.orchestra.run.smithi172.stdout: "get": 0, 2024-03-28T22:48:26.437 INFO:teuthology.orchestra.run.smithi172.stdout: "get_sum": 0, 2024-03-28T22:48:26.437 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:26.437 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_success": 0, 2024-03-28T22:48:26.437 INFO:teuthology.orchestra.run.smithi172.stdout: "take": 0, 2024-03-28T22:48:26.437 INFO:teuthology.orchestra.run.smithi172.stdout: "take_sum": 0, 2024-03-28T22:48:26.437 INFO:teuthology.orchestra.run.smithi172.stdout: "put": 0, 2024-03-28T22:48:26.437 INFO:teuthology.orchestra.run.smithi172.stdout: "put_sum": 0, 2024-03-28T22:48:26.438 INFO:teuthology.orchestra.run.smithi172.stdout: "wait": { 2024-03-28T22:48:26.438 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.438 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.438 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.438 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:26.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:26.438 INFO:teuthology.orchestra.run.smithi172.stdout: "throttle-write_buf_throttle-0x563c210b0500": { 2024-03-28T22:48:26.438 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 0, 2024-03-28T22:48:26.439 INFO:teuthology.orchestra.run.smithi172.stdout: "max": 3758096384, 2024-03-28T22:48:26.439 INFO:teuthology.orchestra.run.smithi172.stdout: "get_started": 0, 2024-03-28T22:48:26.439 INFO:teuthology.orchestra.run.smithi172.stdout: "get": 9, 2024-03-28T22:48:26.439 INFO:teuthology.orchestra.run.smithi172.stdout: "get_sum": 2490, 2024-03-28T22:48:26.439 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:26.439 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_success": 9, 2024-03-28T22:48:26.439 INFO:teuthology.orchestra.run.smithi172.stdout: "take": 0, 2024-03-28T22:48:26.439 INFO:teuthology.orchestra.run.smithi172.stdout: "take_sum": 0, 2024-03-28T22:48:26.439 INFO:teuthology.orchestra.run.smithi172.stdout: "put": 8, 2024-03-28T22:48:26.440 INFO:teuthology.orchestra.run.smithi172.stdout: "put_sum": 2490, 2024-03-28T22:48:26.440 INFO:teuthology.orchestra.run.smithi172.stdout: "wait": { 2024-03-28T22:48:26.440 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:26.440 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:26.440 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:26.440 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:26.440 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:26.440 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-28T22:48:27.081 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:26 smithi172 ceph-mon[36566]: pgmap v353: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T22:48:27.115 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-28T22:48:27.115 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-28T22:48:27.115 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-03-28T22:48:27.115 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=33 2024-03-28T22:48:27.115 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=6 2024-03-28T22:48:27.115 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-03-28T22:48:27.116 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T22:48:27.116 DEBUG:tasks.check_counter:Getting stats from i 2024-03-28T22:48:27.116 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-28T22:48:27.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:26 smithi082 ceph-mon[36471]: pgmap v353: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T22:48:27.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:26 smithi043 ceph-mon[31823]: pgmap v353: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T22:48:27.447 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T22:48:29.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:28 smithi172 ceph-mon[36566]: pgmap v354: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:48:29.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:28 smithi082 ceph-mon[36471]: pgmap v354: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:48:29.223 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-28T22:48:29.223 INFO:teuthology.orchestra.run.smithi172.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T22:48:29.223 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_messages": 70312, 2024-03-28T22:48:29.223 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_messages": 35387, 2024-03-28T22:48:29.223 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_bytes": 23281428, 2024-03-28T22:48:29.223 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_bytes": 5963779, 2024-03-28T22:48:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_created_connections": 11, 2024-03-28T22:48:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_active_connections": 4, 2024-03-28T22:48:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_total_time": 3.128653210, 2024-03-28T22:48:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_send_time": 1.153371664, 2024-03-28T22:48:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_recv_time": 2.678837616, 2024-03-28T22:48:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_fast_dispatch_time": 0.003584061, 2024-03-28T22:48:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:48:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 35386, 2024-03-28T22:48:29.224 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.818726048, 2024-03-28T22:48:29.225 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000023137 2024-03-28T22:48:29.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.225 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:48:29.225 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 105530, 2024-03-28T22:48:29.225 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.015245373, 2024-03-28T22:48:29.225 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000144 2024-03-28T22:48:29.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.225 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_encrypted_bytes": 6720, 2024-03-28T22:48:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_encrypted_bytes": 12784 2024-03-28T22:48:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T22:48:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_messages": 7650, 2024-03-28T22:48:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_messages": 7733, 2024-03-28T22:48:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_bytes": 2275015, 2024-03-28T22:48:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_bytes": 1484102, 2024-03-28T22:48:29.226 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_created_connections": 52, 2024-03-28T22:48:29.227 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_active_connections": 8, 2024-03-28T22:48:29.230 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_total_time": 0.434359774, 2024-03-28T22:48:29.230 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_send_time": 0.152229202, 2024-03-28T22:48:29.230 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_recv_time": 0.369142678, 2024-03-28T22:48:29.230 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_fast_dispatch_time": 0.000215358, 2024-03-28T22:48:29.230 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:48:29.231 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 7730, 2024-03-28T22:48:29.231 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.130762445, 2024-03-28T22:48:29.231 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000016916 2024-03-28T22:48:29.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.231 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:48:29.231 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 15255, 2024-03-28T22:48:29.231 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.002086540, 2024-03-28T22:48:29.231 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000136 2024-03-28T22:48:29.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.232 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_encrypted_bytes": 1088, 2024-03-28T22:48:29.232 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_encrypted_bytes": 232976 2024-03-28T22:48:29.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.232 INFO:teuthology.orchestra.run.smithi172.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T22:48:29.232 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_messages": 336, 2024-03-28T22:48:29.232 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_messages": 447, 2024-03-28T22:48:29.232 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_bytes": 1940151, 2024-03-28T22:48:29.232 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_bytes": 501502, 2024-03-28T22:48:29.233 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_created_connections": 7, 2024-03-28T22:48:29.233 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_active_connections": 5, 2024-03-28T22:48:29.233 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_total_time": 0.050657459, 2024-03-28T22:48:29.233 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_send_time": 0.014776900, 2024-03-28T22:48:29.233 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_recv_time": 0.044968841, 2024-03-28T22:48:29.233 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_fast_dispatch_time": 0.004891793, 2024-03-28T22:48:29.233 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:48:29.233 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 446, 2024-03-28T22:48:29.234 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.042820805, 2024-03-28T22:48:29.234 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000096010 2024-03-28T22:48:29.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.234 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:48:29.235 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 6, 2024-03-28T22:48:29.235 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000763, 2024-03-28T22:48:29.235 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000127 2024-03-28T22:48:29.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.235 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_encrypted_bytes": 1937632, 2024-03-28T22:48:29.235 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_encrypted_bytes": 491872 2024-03-28T22:48:29.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.235 INFO:teuthology.orchestra.run.smithi172.stdout: "cct": { 2024-03-28T22:48:29.235 INFO:teuthology.orchestra.run.smithi172.stdout: "total_workers": 1, 2024-03-28T22:48:29.236 INFO:teuthology.orchestra.run.smithi172.stdout: "unhealthy_workers": 0 2024-03-28T22:48:29.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.236 INFO:teuthology.orchestra.run.smithi172.stdout: "finisher-MDSRank": { 2024-03-28T22:48:29.236 INFO:teuthology.orchestra.run.smithi172.stdout: "queue_len": 0, 2024-03-28T22:48:29.236 INFO:teuthology.orchestra.run.smithi172.stdout: "complete_latency": { 2024-03-28T22:48:29.236 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 63, 2024-03-28T22:48:29.236 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.001784803, 2024-03-28T22:48:29.236 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000028330 2024-03-28T22:48:29.236 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:29.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.237 INFO:teuthology.orchestra.run.smithi172.stdout: "finisher-PurgeQueue": { 2024-03-28T22:48:29.237 INFO:teuthology.orchestra.run.smithi172.stdout: "queue_len": 0, 2024-03-28T22:48:29.237 INFO:teuthology.orchestra.run.smithi172.stdout: "complete_latency": { 2024-03-28T22:48:29.237 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 2, 2024-03-28T22:48:29.237 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000006757, 2024-03-28T22:48:29.237 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000003378 2024-03-28T22:48:29.237 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:29.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.238 INFO:teuthology.orchestra.run.smithi172.stdout: "mds": { 2024-03-28T22:48:29.238 INFO:teuthology.orchestra.run.smithi172.stdout: "request": 7589, 2024-03-28T22:48:29.238 INFO:teuthology.orchestra.run.smithi172.stdout: "reply": 4, 2024-03-28T22:48:29.238 INFO:teuthology.orchestra.run.smithi172.stdout: "reply_latency": { 2024-03-28T22:48:29.238 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 4, 2024-03-28T22:48:29.238 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000692450, 2024-03-28T22:48:29.238 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000173112 2024-03-28T22:48:29.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.239 INFO:teuthology.orchestra.run.smithi172.stdout: "slow_reply": 0, 2024-03-28T22:48:29.239 INFO:teuthology.orchestra.run.smithi172.stdout: "forward": 7585, 2024-03-28T22:48:29.239 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_fetch_complete": 0, 2024-03-28T22:48:29.239 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_fetch_keys": 0, 2024-03-28T22:48:29.239 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_commit": 11, 2024-03-28T22:48:29.239 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_split": 0, 2024-03-28T22:48:29.239 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_merge": 0, 2024-03-28T22:48:29.239 INFO:teuthology.orchestra.run.smithi172.stdout: "inodes": 29, 2024-03-28T22:48:29.239 INFO:teuthology.orchestra.run.smithi172.stdout: "inodes_top": 7, 2024-03-28T22:48:29.240 INFO:teuthology.orchestra.run.smithi172.stdout: "inodes_bottom": 22, 2024-03-28T22:48:29.240 INFO:teuthology.orchestra.run.smithi172.stdout: "inodes_pin_tail": 0, 2024-03-28T22:48:29.240 INFO:teuthology.orchestra.run.smithi172.stdout: "inodes_pinned": 19, 2024-03-28T22:48:29.240 INFO:teuthology.orchestra.run.smithi172.stdout: "inodes_expired": 0, 2024-03-28T22:48:29.240 INFO:teuthology.orchestra.run.smithi172.stdout: "inodes_with_caps": 2, 2024-03-28T22:48:29.240 INFO:teuthology.orchestra.run.smithi172.stdout: "caps": 2, 2024-03-28T22:48:29.240 INFO:teuthology.orchestra.run.smithi172.stdout: "subtrees": 2, 2024-03-28T22:48:29.240 INFO:teuthology.orchestra.run.smithi172.stdout: "traverse": 7614, 2024-03-28T22:48:29.241 INFO:teuthology.orchestra.run.smithi172.stdout: "traverse_hit": 19, 2024-03-28T22:48:29.241 INFO:teuthology.orchestra.run.smithi172.stdout: "traverse_forward": 7582, 2024-03-28T22:48:29.241 INFO:teuthology.orchestra.run.smithi172.stdout: "traverse_discover": 12, 2024-03-28T22:48:29.241 INFO:teuthology.orchestra.run.smithi172.stdout: "traverse_dir_fetch": 0, 2024-03-28T22:48:29.241 INFO:teuthology.orchestra.run.smithi172.stdout: "traverse_remote_ino": 0, 2024-03-28T22:48:29.241 INFO:teuthology.orchestra.run.smithi172.stdout: "traverse_lock": 0, 2024-03-28T22:48:29.241 INFO:teuthology.orchestra.run.smithi172.stdout: "load_cent": 0, 2024-03-28T22:48:29.241 INFO:teuthology.orchestra.run.smithi172.stdout: "q": 0, 2024-03-28T22:48:29.241 INFO:teuthology.orchestra.run.smithi172.stdout: "exported": 0, 2024-03-28T22:48:29.242 INFO:teuthology.orchestra.run.smithi172.stdout: "exported_inodes": 0, 2024-03-28T22:48:29.242 INFO:teuthology.orchestra.run.smithi172.stdout: "imported": 0, 2024-03-28T22:48:29.242 INFO:teuthology.orchestra.run.smithi172.stdout: "imported_inodes": 0, 2024-03-28T22:48:29.242 INFO:teuthology.orchestra.run.smithi172.stdout: "openino_dir_fetch": 0, 2024-03-28T22:48:29.242 INFO:teuthology.orchestra.run.smithi172.stdout: "openino_backtrace_fetch": 0, 2024-03-28T22:48:29.242 INFO:teuthology.orchestra.run.smithi172.stdout: "openino_peer_discover": 0, 2024-03-28T22:48:29.242 INFO:teuthology.orchestra.run.smithi172.stdout: "root_rfiles": 61, 2024-03-28T22:48:29.243 INFO:teuthology.orchestra.run.smithi172.stdout: "root_rbytes": 21056780, 2024-03-28T22:48:29.243 INFO:teuthology.orchestra.run.smithi172.stdout: "root_rsnaps": 6, 2024-03-28T22:48:29.243 INFO:teuthology.orchestra.run.smithi172.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T22:48:29.243 INFO:teuthology.orchestra.run.smithi172.stdout: "scrub_set_tag": 0, 2024-03-28T22:48:29.243 INFO:teuthology.orchestra.run.smithi172.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T22:48:29.243 INFO:teuthology.orchestra.run.smithi172.stdout: "scrub_inotable_repaired": 0, 2024-03-28T22:48:29.243 INFO:teuthology.orchestra.run.smithi172.stdout: "scrub_dir_inodes": 0, 2024-03-28T22:48:29.243 INFO:teuthology.orchestra.run.smithi172.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T22:48:29.243 INFO:teuthology.orchestra.run.smithi172.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T22:48:29.244 INFO:teuthology.orchestra.run.smithi172.stdout: "scrub_file_inodes": 0, 2024-03-28T22:48:29.244 INFO:teuthology.orchestra.run.smithi172.stdout: "handle_inode_file_caps": 0, 2024-03-28T22:48:29.244 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T22:48:29.244 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph_cap_op_grant": 0, 2024-03-28T22:48:29.244 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T22:48:29.244 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T22:48:29.244 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T22:48:29.244 INFO:teuthology.orchestra.run.smithi172.stdout: "handle_client_caps": 0, 2024-03-28T22:48:29.244 INFO:teuthology.orchestra.run.smithi172.stdout: "handle_client_caps_dirty": 0, 2024-03-28T22:48:29.245 INFO:teuthology.orchestra.run.smithi172.stdout: "handle_client_cap_release": 0, 2024-03-28T22:48:29.245 INFO:teuthology.orchestra.run.smithi172.stdout: "process_request_cap_release": 0 2024-03-28T22:48:29.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.245 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_cache": { 2024-03-28T22:48:29.245 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_update": 0, 2024-03-28T22:48:29.245 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_update_receipt": 38, 2024-03-28T22:48:29.245 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_try_discover": 25, 2024-03-28T22:48:29.245 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_send_discover": 26, 2024-03-28T22:48:29.245 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_handle_discover": 0, 2024-03-28T22:48:29.246 INFO:teuthology.orchestra.run.smithi172.stdout: "num_strays": 0, 2024-03-28T22:48:29.246 INFO:teuthology.orchestra.run.smithi172.stdout: "num_strays_delayed": 0, 2024-03-28T22:48:29.246 INFO:teuthology.orchestra.run.smithi172.stdout: "num_strays_enqueuing": 0, 2024-03-28T22:48:29.246 INFO:teuthology.orchestra.run.smithi172.stdout: "strays_created": 0, 2024-03-28T22:48:29.246 INFO:teuthology.orchestra.run.smithi172.stdout: "strays_enqueued": 0, 2024-03-28T22:48:29.246 INFO:teuthology.orchestra.run.smithi172.stdout: "strays_reintegrated": 0, 2024-03-28T22:48:29.246 INFO:teuthology.orchestra.run.smithi172.stdout: "strays_migrated": 0, 2024-03-28T22:48:29.246 INFO:teuthology.orchestra.run.smithi172.stdout: "num_recovering_processing": 0, 2024-03-28T22:48:29.247 INFO:teuthology.orchestra.run.smithi172.stdout: "num_recovering_enqueued": 0, 2024-03-28T22:48:29.247 INFO:teuthology.orchestra.run.smithi172.stdout: "num_recovering_prioritized": 0, 2024-03-28T22:48:29.247 INFO:teuthology.orchestra.run.smithi172.stdout: "recovery_started": 0, 2024-03-28T22:48:29.247 INFO:teuthology.orchestra.run.smithi172.stdout: "recovery_completed": 0, 2024-03-28T22:48:29.247 INFO:teuthology.orchestra.run.smithi172.stdout: "ireq_quiesce_path": 0, 2024-03-28T22:48:29.247 INFO:teuthology.orchestra.run.smithi172.stdout: "ireq_quiesce_inode": 0, 2024-03-28T22:48:29.247 INFO:teuthology.orchestra.run.smithi172.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T22:48:29.247 INFO:teuthology.orchestra.run.smithi172.stdout: "ireq_exportdir": 0, 2024-03-28T22:48:29.248 INFO:teuthology.orchestra.run.smithi172.stdout: "ireq_flush": 0, 2024-03-28T22:48:29.248 INFO:teuthology.orchestra.run.smithi172.stdout: "ireq_fragmentdir": 0, 2024-03-28T22:48:29.248 INFO:teuthology.orchestra.run.smithi172.stdout: "ireq_fragstats": 0, 2024-03-28T22:48:29.248 INFO:teuthology.orchestra.run.smithi172.stdout: "ireq_inodestats": 0 2024-03-28T22:48:29.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.248 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_log": { 2024-03-28T22:48:29.248 INFO:teuthology.orchestra.run.smithi172.stdout: "evlrg": 0, 2024-03-28T22:48:29.248 INFO:teuthology.orchestra.run.smithi172.stdout: "evadd": 8, 2024-03-28T22:48:29.249 INFO:teuthology.orchestra.run.smithi172.stdout: "evex": 0, 2024-03-28T22:48:29.249 INFO:teuthology.orchestra.run.smithi172.stdout: "evtrm": 0, 2024-03-28T22:48:29.249 INFO:teuthology.orchestra.run.smithi172.stdout: "ev": 8, 2024-03-28T22:48:29.249 INFO:teuthology.orchestra.run.smithi172.stdout: "evexg": 0, 2024-03-28T22:48:29.249 INFO:teuthology.orchestra.run.smithi172.stdout: "evexd": 0, 2024-03-28T22:48:29.249 INFO:teuthology.orchestra.run.smithi172.stdout: "segadd": 2, 2024-03-28T22:48:29.249 INFO:teuthology.orchestra.run.smithi172.stdout: "segex": 0, 2024-03-28T22:48:29.249 INFO:teuthology.orchestra.run.smithi172.stdout: "segtrm": 0, 2024-03-28T22:48:29.249 INFO:teuthology.orchestra.run.smithi172.stdout: "seg": 2, 2024-03-28T22:48:29.250 INFO:teuthology.orchestra.run.smithi172.stdout: "segmjr": 2, 2024-03-28T22:48:29.250 INFO:teuthology.orchestra.run.smithi172.stdout: "segexg": 0, 2024-03-28T22:48:29.250 INFO:teuthology.orchestra.run.smithi172.stdout: "segexd": 0, 2024-03-28T22:48:29.250 INFO:teuthology.orchestra.run.smithi172.stdout: "expos": 4194304, 2024-03-28T22:48:29.250 INFO:teuthology.orchestra.run.smithi172.stdout: "wrpos": 4196653, 2024-03-28T22:48:29.250 INFO:teuthology.orchestra.run.smithi172.stdout: "rdpos": 0, 2024-03-28T22:48:29.250 INFO:teuthology.orchestra.run.smithi172.stdout: "jlat": { 2024-03-28T22:48:29.250 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 7, 2024-03-28T22:48:29.251 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.020599173, 2024-03-28T22:48:29.251 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.002942739 2024-03-28T22:48:29.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.251 INFO:teuthology.orchestra.run.smithi172.stdout: "replayed": 0 2024-03-28T22:48:29.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.251 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_mem": { 2024-03-28T22:48:29.251 INFO:teuthology.orchestra.run.smithi172.stdout: "ino": 32, 2024-03-28T22:48:29.251 INFO:teuthology.orchestra.run.smithi172.stdout: "ino+": 32, 2024-03-28T22:48:29.252 INFO:teuthology.orchestra.run.smithi172.stdout: "ino-": 0, 2024-03-28T22:48:29.252 INFO:teuthology.orchestra.run.smithi172.stdout: "dir": 38, 2024-03-28T22:48:29.252 INFO:teuthology.orchestra.run.smithi172.stdout: "dir+": 41, 2024-03-28T22:48:29.252 INFO:teuthology.orchestra.run.smithi172.stdout: "dir-": 3, 2024-03-28T22:48:29.252 INFO:teuthology.orchestra.run.smithi172.stdout: "dn": 29, 2024-03-28T22:48:29.252 INFO:teuthology.orchestra.run.smithi172.stdout: "dn+": 29, 2024-03-28T22:48:29.252 INFO:teuthology.orchestra.run.smithi172.stdout: "dn-": 0, 2024-03-28T22:48:29.252 INFO:teuthology.orchestra.run.smithi172.stdout: "cap": 2, 2024-03-28T22:48:29.252 INFO:teuthology.orchestra.run.smithi172.stdout: "cap+": 2, 2024-03-28T22:48:29.253 INFO:teuthology.orchestra.run.smithi172.stdout: "cap-": 0, 2024-03-28T22:48:29.253 INFO:teuthology.orchestra.run.smithi172.stdout: "rss": 57384, 2024-03-28T22:48:29.253 INFO:teuthology.orchestra.run.smithi172.stdout: "heap": 223612 2024-03-28T22:48:29.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.253 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_server": { 2024-03-28T22:48:29.253 INFO:teuthology.orchestra.run.smithi172.stdout: "dispatch_client_request": 7589, 2024-03-28T22:48:29.253 INFO:teuthology.orchestra.run.smithi172.stdout: "dispatch_server_request": 0, 2024-03-28T22:48:29.253 INFO:teuthology.orchestra.run.smithi172.stdout: "handle_client_request": 7589, 2024-03-28T22:48:29.254 INFO:teuthology.orchestra.run.smithi172.stdout: "handle_client_session": 26, 2024-03-28T22:48:29.254 INFO:teuthology.orchestra.run.smithi172.stdout: "handle_peer_request": 0, 2024-03-28T22:48:29.254 INFO:teuthology.orchestra.run.smithi172.stdout: "req_create_latency": { 2024-03-28T22:48:29.254 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.254 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.254 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.255 INFO:teuthology.orchestra.run.smithi172.stdout: "req_getattr_latency": { 2024-03-28T22:48:29.255 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 3, 2024-03-28T22:48:29.255 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000512707, 2024-03-28T22:48:29.255 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000170902 2024-03-28T22:48:29.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.255 INFO:teuthology.orchestra.run.smithi172.stdout: "req_getfilelock_latency": { 2024-03-28T22:48:29.255 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.255 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.255 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.256 INFO:teuthology.orchestra.run.smithi172.stdout: "req_link_latency": { 2024-03-28T22:48:29.256 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.256 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.256 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.256 INFO:teuthology.orchestra.run.smithi172.stdout: "req_lookup_latency": { 2024-03-28T22:48:29.256 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.257 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.257 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.257 INFO:teuthology.orchestra.run.smithi172.stdout: "req_lookuphash_latency": { 2024-03-28T22:48:29.257 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.257 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.257 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.258 INFO:teuthology.orchestra.run.smithi172.stdout: "req_lookupino_latency": { 2024-03-28T22:48:29.258 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 1, 2024-03-28T22:48:29.258 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000179743, 2024-03-28T22:48:29.258 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000179743 2024-03-28T22:48:29.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.258 INFO:teuthology.orchestra.run.smithi172.stdout: "req_lookupname_latency": { 2024-03-28T22:48:29.258 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.258 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.258 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.259 INFO:teuthology.orchestra.run.smithi172.stdout: "req_lookupparent_latency": { 2024-03-28T22:48:29.259 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.259 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.259 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.259 INFO:teuthology.orchestra.run.smithi172.stdout: "req_lookupsnap_latency": { 2024-03-28T22:48:29.259 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.260 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.260 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.260 INFO:teuthology.orchestra.run.smithi172.stdout: "req_lssnap_latency": { 2024-03-28T22:48:29.260 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.260 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.260 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.261 INFO:teuthology.orchestra.run.smithi172.stdout: "req_mkdir_latency": { 2024-03-28T22:48:29.261 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.261 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.261 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.262 INFO:teuthology.orchestra.run.smithi172.stdout: "req_mknod_latency": { 2024-03-28T22:48:29.262 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.262 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.262 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.262 INFO:teuthology.orchestra.run.smithi172.stdout: "req_mksnap_latency": { 2024-03-28T22:48:29.262 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.263 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.263 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.263 INFO:teuthology.orchestra.run.smithi172.stdout: "req_open_latency": { 2024-03-28T22:48:29.263 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.263 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.263 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.263 INFO:teuthology.orchestra.run.smithi172.stdout: "req_readdir_latency": { 2024-03-28T22:48:29.264 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.264 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.264 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.264 INFO:teuthology.orchestra.run.smithi172.stdout: "req_rename_latency": { 2024-03-28T22:48:29.264 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.264 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.264 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.265 INFO:teuthology.orchestra.run.smithi172.stdout: "req_renamesnap_latency": { 2024-03-28T22:48:29.265 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.265 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.265 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.265 INFO:teuthology.orchestra.run.smithi172.stdout: "req_snapdiff_latency": { 2024-03-28T22:48:29.265 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.266 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.266 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.266 INFO:teuthology.orchestra.run.smithi172.stdout: "req_rmdir_latency": { 2024-03-28T22:48:29.266 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.266 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.266 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.267 INFO:teuthology.orchestra.run.smithi172.stdout: "req_rmsnap_latency": { 2024-03-28T22:48:29.267 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.267 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.267 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.267 INFO:teuthology.orchestra.run.smithi172.stdout: "req_rmxattr_latency": { 2024-03-28T22:48:29.267 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.267 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.268 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.268 INFO:teuthology.orchestra.run.smithi172.stdout: "req_setattr_latency": { 2024-03-28T22:48:29.268 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.268 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.268 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.268 INFO:teuthology.orchestra.run.smithi172.stdout: "req_setdirlayout_latency": { 2024-03-28T22:48:29.268 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.269 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.269 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.269 INFO:teuthology.orchestra.run.smithi172.stdout: "req_setfilelock_latency": { 2024-03-28T22:48:29.269 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.269 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.269 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.270 INFO:teuthology.orchestra.run.smithi172.stdout: "req_setlayout_latency": { 2024-03-28T22:48:29.270 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.270 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.270 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.270 INFO:teuthology.orchestra.run.smithi172.stdout: "req_setxattr_latency": { 2024-03-28T22:48:29.270 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.270 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.271 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.271 INFO:teuthology.orchestra.run.smithi172.stdout: "req_symlink_latency": { 2024-03-28T22:48:29.271 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.271 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.271 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.271 INFO:teuthology.orchestra.run.smithi172.stdout: "req_unlink_latency": { 2024-03-28T22:48:29.271 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.272 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.272 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.272 INFO:teuthology.orchestra.run.smithi172.stdout: "cap_revoke_eviction": 0, 2024-03-28T22:48:29.272 INFO:teuthology.orchestra.run.smithi172.stdout: "cap_acquisition_throttle": 0, 2024-03-28T22:48:29.272 INFO:teuthology.orchestra.run.smithi172.stdout: "req_getvxattr_latency": { 2024-03-28T22:48:29.272 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.272 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.273 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.273 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:29.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.273 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_sessions": { 2024-03-28T22:48:29.273 INFO:teuthology.orchestra.run.smithi172.stdout: "session_count": 2, 2024-03-28T22:48:29.273 INFO:teuthology.orchestra.run.smithi172.stdout: "session_add": 4, 2024-03-28T22:48:29.273 INFO:teuthology.orchestra.run.smithi172.stdout: "session_remove": 2, 2024-03-28T22:48:29.273 INFO:teuthology.orchestra.run.smithi172.stdout: "sessions_open": 2, 2024-03-28T22:48:29.274 INFO:teuthology.orchestra.run.smithi172.stdout: "sessions_stale": 0, 2024-03-28T22:48:29.274 INFO:teuthology.orchestra.run.smithi172.stdout: "total_load": 3, 2024-03-28T22:48:29.274 INFO:teuthology.orchestra.run.smithi172.stdout: "average_load": 1, 2024-03-28T22:48:29.274 INFO:teuthology.orchestra.run.smithi172.stdout: "avg_session_uptime": 662, 2024-03-28T22:48:29.274 INFO:teuthology.orchestra.run.smithi172.stdout: "mdthresh_evicted": 0 2024-03-28T22:48:29.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.274 INFO:teuthology.orchestra.run.smithi172.stdout: "mempool": { 2024-03-28T22:48:29.274 INFO:teuthology.orchestra.run.smithi172.stdout: "bloom_filter_bytes": 0, 2024-03-28T22:48:29.275 INFO:teuthology.orchestra.run.smithi172.stdout: "bloom_filter_items": 0, 2024-03-28T22:48:29.275 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T22:48:29.275 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_alloc_items": 0, 2024-03-28T22:48:29.275 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T22:48:29.275 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_data_items": 0, 2024-03-28T22:48:29.275 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T22:48:29.275 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T22:48:29.275 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T22:48:29.276 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T22:48:29.276 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T22:48:29.276 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_other_items": 0, 2024-03-28T22:48:29.276 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T22:48:29.276 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T22:48:29.276 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_extent_bytes": 0, 2024-03-28T22:48:29.276 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_extent_items": 0, 2024-03-28T22:48:29.276 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_blob_bytes": 0, 2024-03-28T22:48:29.276 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_blob_items": 0, 2024-03-28T22:48:29.276 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T22:48:29.277 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T22:48:29.277 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T22:48:29.277 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T22:48:29.277 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T22:48:29.277 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_fsck_items": 0, 2024-03-28T22:48:29.277 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_txc_bytes": 0, 2024-03-28T22:48:29.277 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_txc_items": 0, 2024-03-28T22:48:29.277 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T22:48:29.277 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T22:48:29.278 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_writing_bytes": 0, 2024-03-28T22:48:29.278 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_writing_items": 0, 2024-03-28T22:48:29.278 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_bytes": 0, 2024-03-28T22:48:29.278 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_items": 0, 2024-03-28T22:48:29.278 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T22:48:29.278 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_file_reader_items": 0, 2024-03-28T22:48:29.278 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T22:48:29.278 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_file_writer_items": 0, 2024-03-28T22:48:29.279 INFO:teuthology.orchestra.run.smithi172.stdout: "buffer_anon_bytes": 268340, 2024-03-28T22:48:29.279 INFO:teuthology.orchestra.run.smithi172.stdout: "buffer_anon_items": 127, 2024-03-28T22:48:29.279 INFO:teuthology.orchestra.run.smithi172.stdout: "buffer_meta_bytes": 0, 2024-03-28T22:48:29.279 INFO:teuthology.orchestra.run.smithi172.stdout: "buffer_meta_items": 0, 2024-03-28T22:48:29.279 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_bytes": 0, 2024-03-28T22:48:29.279 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_items": 0, 2024-03-28T22:48:29.279 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mapbl_bytes": 0, 2024-03-28T22:48:29.279 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mapbl_items": 0, 2024-03-28T22:48:29.280 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pglog_bytes": 0, 2024-03-28T22:48:29.280 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pglog_items": 0, 2024-03-28T22:48:29.280 INFO:teuthology.orchestra.run.smithi172.stdout: "osdmap_bytes": 9736, 2024-03-28T22:48:29.280 INFO:teuthology.orchestra.run.smithi172.stdout: "osdmap_items": 203, 2024-03-28T22:48:29.280 INFO:teuthology.orchestra.run.smithi172.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T22:48:29.280 INFO:teuthology.orchestra.run.smithi172.stdout: "osdmap_mapping_items": 0, 2024-03-28T22:48:29.280 INFO:teuthology.orchestra.run.smithi172.stdout: "pgmap_bytes": 0, 2024-03-28T22:48:29.280 INFO:teuthology.orchestra.run.smithi172.stdout: "pgmap_items": 0, 2024-03-28T22:48:29.280 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_co_bytes": 146516, 2024-03-28T22:48:29.281 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_co_items": 775, 2024-03-28T22:48:29.281 INFO:teuthology.orchestra.run.smithi172.stdout: "unittest_1_bytes": 0, 2024-03-28T22:48:29.281 INFO:teuthology.orchestra.run.smithi172.stdout: "unittest_1_items": 0, 2024-03-28T22:48:29.281 INFO:teuthology.orchestra.run.smithi172.stdout: "unittest_2_bytes": 0, 2024-03-28T22:48:29.281 INFO:teuthology.orchestra.run.smithi172.stdout: "unittest_2_items": 0 2024-03-28T22:48:29.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.281 INFO:teuthology.orchestra.run.smithi172.stdout: "objecter": { 2024-03-28T22:48:29.281 INFO:teuthology.orchestra.run.smithi172.stdout: "op_active": 0, 2024-03-28T22:48:29.282 INFO:teuthology.orchestra.run.smithi172.stdout: "op_laggy": 0, 2024-03-28T22:48:29.282 INFO:teuthology.orchestra.run.smithi172.stdout: "op_send": 46, 2024-03-28T22:48:29.282 INFO:teuthology.orchestra.run.smithi172.stdout: "op_send_bytes": 13042, 2024-03-28T22:48:29.282 INFO:teuthology.orchestra.run.smithi172.stdout: "op_resend": 33, 2024-03-28T22:48:29.282 INFO:teuthology.orchestra.run.smithi172.stdout: "op_reply": 46, 2024-03-28T22:48:29.282 INFO:teuthology.orchestra.run.smithi172.stdout: "op_latency": { 2024-03-28T22:48:29.282 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 46, 2024-03-28T22:48:29.282 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.345991800, 2024-03-28T22:48:29.283 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.007521560 2024-03-28T22:48:29.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.283 INFO:teuthology.orchestra.run.smithi172.stdout: "op_inflight": 0, 2024-03-28T22:48:29.283 INFO:teuthology.orchestra.run.smithi172.stdout: "oplen_avg": { 2024-03-28T22:48:29.283 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 46, 2024-03-28T22:48:29.283 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 69 2024-03-28T22:48:29.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.283 INFO:teuthology.orchestra.run.smithi172.stdout: "op": 46, 2024-03-28T22:48:29.284 INFO:teuthology.orchestra.run.smithi172.stdout: "op_r": 0, 2024-03-28T22:48:29.284 INFO:teuthology.orchestra.run.smithi172.stdout: "op_w": 46, 2024-03-28T22:48:29.284 INFO:teuthology.orchestra.run.smithi172.stdout: "op_rmw": 0, 2024-03-28T22:48:29.284 INFO:teuthology.orchestra.run.smithi172.stdout: "op_pg": 0, 2024-03-28T22:48:29.284 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_stat": 0, 2024-03-28T22:48:29.284 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_create": 10, 2024-03-28T22:48:29.284 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_read": 0, 2024-03-28T22:48:29.284 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_write": 7, 2024-03-28T22:48:29.284 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_writefull": 9, 2024-03-28T22:48:29.285 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_writesame": 0, 2024-03-28T22:48:29.285 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_append": 0, 2024-03-28T22:48:29.285 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_zero": 0, 2024-03-28T22:48:29.285 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_truncate": 0, 2024-03-28T22:48:29.285 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_delete": 6, 2024-03-28T22:48:29.285 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_mapext": 0, 2024-03-28T22:48:29.285 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_sparse_read": 0, 2024-03-28T22:48:29.285 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_clonerange": 0, 2024-03-28T22:48:29.286 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_getxattr": 0, 2024-03-28T22:48:29.286 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_setxattr": 20, 2024-03-28T22:48:29.286 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_cmpxattr": 0, 2024-03-28T22:48:29.286 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_rmxattr": 0, 2024-03-28T22:48:29.286 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_resetxattrs": 0, 2024-03-28T22:48:29.286 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_call": 0, 2024-03-28T22:48:29.286 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_watch": 0, 2024-03-28T22:48:29.286 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_notify": 0, 2024-03-28T22:48:29.286 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T22:48:29.287 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_pgls": 0, 2024-03-28T22:48:29.287 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_pgls_filter": 0, 2024-03-28T22:48:29.287 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_other": 0, 2024-03-28T22:48:29.287 INFO:teuthology.orchestra.run.smithi172.stdout: "linger_active": 0, 2024-03-28T22:48:29.287 INFO:teuthology.orchestra.run.smithi172.stdout: "linger_send": 0, 2024-03-28T22:48:29.287 INFO:teuthology.orchestra.run.smithi172.stdout: "linger_resend": 0, 2024-03-28T22:48:29.287 INFO:teuthology.orchestra.run.smithi172.stdout: "linger_ping": 0, 2024-03-28T22:48:29.287 INFO:teuthology.orchestra.run.smithi172.stdout: "poolop_active": 0, 2024-03-28T22:48:29.288 INFO:teuthology.orchestra.run.smithi172.stdout: "poolop_send": 0, 2024-03-28T22:48:29.288 INFO:teuthology.orchestra.run.smithi172.stdout: "poolop_resend": 0, 2024-03-28T22:48:29.288 INFO:teuthology.orchestra.run.smithi172.stdout: "poolstat_active": 0, 2024-03-28T22:48:29.288 INFO:teuthology.orchestra.run.smithi172.stdout: "poolstat_send": 0, 2024-03-28T22:48:29.288 INFO:teuthology.orchestra.run.smithi172.stdout: "poolstat_resend": 0, 2024-03-28T22:48:29.288 INFO:teuthology.orchestra.run.smithi172.stdout: "statfs_active": 0, 2024-03-28T22:48:29.288 INFO:teuthology.orchestra.run.smithi172.stdout: "statfs_send": 0, 2024-03-28T22:48:29.288 INFO:teuthology.orchestra.run.smithi172.stdout: "statfs_resend": 0, 2024-03-28T22:48:29.289 INFO:teuthology.orchestra.run.smithi172.stdout: "command_active": 0, 2024-03-28T22:48:29.289 INFO:teuthology.orchestra.run.smithi172.stdout: "command_send": 0, 2024-03-28T22:48:29.289 INFO:teuthology.orchestra.run.smithi172.stdout: "command_resend": 0, 2024-03-28T22:48:29.289 INFO:teuthology.orchestra.run.smithi172.stdout: "map_epoch": 110, 2024-03-28T22:48:29.289 INFO:teuthology.orchestra.run.smithi172.stdout: "map_full": 0, 2024-03-28T22:48:29.289 INFO:teuthology.orchestra.run.smithi172.stdout: "map_inc": 18, 2024-03-28T22:48:29.289 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_sessions": 9, 2024-03-28T22:48:29.289 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_session_open": 9, 2024-03-28T22:48:29.289 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_session_close": 0, 2024-03-28T22:48:29.290 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_laggy": 0, 2024-03-28T22:48:29.290 INFO:teuthology.orchestra.run.smithi172.stdout: "omap_wr": 16, 2024-03-28T22:48:29.290 INFO:teuthology.orchestra.run.smithi172.stdout: "omap_rd": 0, 2024-03-28T22:48:29.290 INFO:teuthology.orchestra.run.smithi172.stdout: "omap_del": 1 2024-03-28T22:48:29.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.290 INFO:teuthology.orchestra.run.smithi172.stdout: "oft": { 2024-03-28T22:48:29.290 INFO:teuthology.orchestra.run.smithi172.stdout: "omap_total_objs": 1, 2024-03-28T22:48:29.290 INFO:teuthology.orchestra.run.smithi172.stdout: "omap_total_kv_pairs": 0, 2024-03-28T22:48:29.291 INFO:teuthology.orchestra.run.smithi172.stdout: "omap_total_updates": 0, 2024-03-28T22:48:29.291 INFO:teuthology.orchestra.run.smithi172.stdout: "omap_total_removes": 0 2024-03-28T22:48:29.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.291 INFO:teuthology.orchestra.run.smithi172.stdout: "osd-slow-ops": { 2024-03-28T22:48:29.291 INFO:teuthology.orchestra.run.smithi172.stdout: "slow_ops_count": 0 2024-03-28T22:48:29.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.291 INFO:teuthology.orchestra.run.smithi172.stdout: "purge_queue": { 2024-03-28T22:48:29.291 INFO:teuthology.orchestra.run.smithi172.stdout: "pq_executing_ops": 0, 2024-03-28T22:48:29.292 INFO:teuthology.orchestra.run.smithi172.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T22:48:29.292 INFO:teuthology.orchestra.run.smithi172.stdout: "pq_executing": 0, 2024-03-28T22:48:29.292 INFO:teuthology.orchestra.run.smithi172.stdout: "pq_executing_high_water": 0, 2024-03-28T22:48:29.292 INFO:teuthology.orchestra.run.smithi172.stdout: "pq_executed_ops": 0, 2024-03-28T22:48:29.292 INFO:teuthology.orchestra.run.smithi172.stdout: "pq_executed": 0, 2024-03-28T22:48:29.292 INFO:teuthology.orchestra.run.smithi172.stdout: "pq_item_in_journal": 0 2024-03-28T22:48:29.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.292 INFO:teuthology.orchestra.run.smithi172.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T22:48:29.292 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 0, 2024-03-28T22:48:29.293 INFO:teuthology.orchestra.run.smithi172.stdout: "max": 104857600, 2024-03-28T22:48:29.293 INFO:teuthology.orchestra.run.smithi172.stdout: "get_started": 0, 2024-03-28T22:48:29.293 INFO:teuthology.orchestra.run.smithi172.stdout: "get": 78294, 2024-03-28T22:48:29.293 INFO:teuthology.orchestra.run.smithi172.stdout: "get_sum": 20429824, 2024-03-28T22:48:29.293 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:29.293 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_success": 78294, 2024-03-28T22:48:29.293 INFO:teuthology.orchestra.run.smithi172.stdout: "take": 0, 2024-03-28T22:48:29.293 INFO:teuthology.orchestra.run.smithi172.stdout: "take_sum": 0, 2024-03-28T22:48:29.294 INFO:teuthology.orchestra.run.smithi172.stdout: "put": 78294, 2024-03-28T22:48:29.294 INFO:teuthology.orchestra.run.smithi172.stdout: "put_sum": 20429824, 2024-03-28T22:48:29.294 INFO:teuthology.orchestra.run.smithi172.stdout: "wait": { 2024-03-28T22:48:29.294 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.294 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.294 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.294 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:29.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.295 INFO:teuthology.orchestra.run.smithi172.stdout: "throttle-objecter_bytes": { 2024-03-28T22:48:29.295 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 0, 2024-03-28T22:48:29.295 INFO:teuthology.orchestra.run.smithi172.stdout: "max": 104857600, 2024-03-28T22:48:29.295 INFO:teuthology.orchestra.run.smithi172.stdout: "get_started": 0, 2024-03-28T22:48:29.295 INFO:teuthology.orchestra.run.smithi172.stdout: "get": 0, 2024-03-28T22:48:29.295 INFO:teuthology.orchestra.run.smithi172.stdout: "get_sum": 0, 2024-03-28T22:48:29.295 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:29.295 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_success": 0, 2024-03-28T22:48:29.296 INFO:teuthology.orchestra.run.smithi172.stdout: "take": 46, 2024-03-28T22:48:29.296 INFO:teuthology.orchestra.run.smithi172.stdout: "take_sum": 13042, 2024-03-28T22:48:29.296 INFO:teuthology.orchestra.run.smithi172.stdout: "put": 46, 2024-03-28T22:48:29.296 INFO:teuthology.orchestra.run.smithi172.stdout: "put_sum": 13042, 2024-03-28T22:48:29.296 INFO:teuthology.orchestra.run.smithi172.stdout: "wait": { 2024-03-28T22:48:29.296 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.296 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.296 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.296 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:29.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.297 INFO:teuthology.orchestra.run.smithi172.stdout: "throttle-objecter_ops": { 2024-03-28T22:48:29.297 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 0, 2024-03-28T22:48:29.297 INFO:teuthology.orchestra.run.smithi172.stdout: "max": 1024, 2024-03-28T22:48:29.297 INFO:teuthology.orchestra.run.smithi172.stdout: "get_started": 0, 2024-03-28T22:48:29.297 INFO:teuthology.orchestra.run.smithi172.stdout: "get": 0, 2024-03-28T22:48:29.297 INFO:teuthology.orchestra.run.smithi172.stdout: "get_sum": 0, 2024-03-28T22:48:29.297 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:29.298 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_success": 0, 2024-03-28T22:48:29.298 INFO:teuthology.orchestra.run.smithi172.stdout: "take": 46, 2024-03-28T22:48:29.298 INFO:teuthology.orchestra.run.smithi172.stdout: "take_sum": 46, 2024-03-28T22:48:29.298 INFO:teuthology.orchestra.run.smithi172.stdout: "put": 46, 2024-03-28T22:48:29.298 INFO:teuthology.orchestra.run.smithi172.stdout: "put_sum": 46, 2024-03-28T22:48:29.298 INFO:teuthology.orchestra.run.smithi172.stdout: "wait": { 2024-03-28T22:48:29.298 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.298 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.298 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.299 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:29.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.299 INFO:teuthology.orchestra.run.smithi172.stdout: "throttle-write_buf_throttle": { 2024-03-28T22:48:29.299 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 0, 2024-03-28T22:48:29.299 INFO:teuthology.orchestra.run.smithi172.stdout: "max": 3758096384, 2024-03-28T22:48:29.299 INFO:teuthology.orchestra.run.smithi172.stdout: "get_started": 0, 2024-03-28T22:48:29.299 INFO:teuthology.orchestra.run.smithi172.stdout: "get": 0, 2024-03-28T22:48:29.299 INFO:teuthology.orchestra.run.smithi172.stdout: "get_sum": 0, 2024-03-28T22:48:29.299 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:29.300 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_success": 0, 2024-03-28T22:48:29.300 INFO:teuthology.orchestra.run.smithi172.stdout: "take": 0, 2024-03-28T22:48:29.300 INFO:teuthology.orchestra.run.smithi172.stdout: "take_sum": 0, 2024-03-28T22:48:29.300 INFO:teuthology.orchestra.run.smithi172.stdout: "put": 0, 2024-03-28T22:48:29.300 INFO:teuthology.orchestra.run.smithi172.stdout: "put_sum": 0, 2024-03-28T22:48:29.300 INFO:teuthology.orchestra.run.smithi172.stdout: "wait": { 2024-03-28T22:48:29.300 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.300 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.300 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.301 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:29.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:29.301 INFO:teuthology.orchestra.run.smithi172.stdout: "throttle-write_buf_throttle-0x55b6fe922500": { 2024-03-28T22:48:29.301 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 0, 2024-03-28T22:48:29.301 INFO:teuthology.orchestra.run.smithi172.stdout: "max": 3758096384, 2024-03-28T22:48:29.301 INFO:teuthology.orchestra.run.smithi172.stdout: "get_started": 0, 2024-03-28T22:48:29.301 INFO:teuthology.orchestra.run.smithi172.stdout: "get": 8, 2024-03-28T22:48:29.301 INFO:teuthology.orchestra.run.smithi172.stdout: "get_sum": 2349, 2024-03-28T22:48:29.301 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:29.302 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_success": 8, 2024-03-28T22:48:29.302 INFO:teuthology.orchestra.run.smithi172.stdout: "take": 0, 2024-03-28T22:48:29.302 INFO:teuthology.orchestra.run.smithi172.stdout: "take_sum": 0, 2024-03-28T22:48:29.302 INFO:teuthology.orchestra.run.smithi172.stdout: "put": 7, 2024-03-28T22:48:29.302 INFO:teuthology.orchestra.run.smithi172.stdout: "put_sum": 2349, 2024-03-28T22:48:29.302 INFO:teuthology.orchestra.run.smithi172.stdout: "wait": { 2024-03-28T22:48:29.302 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:29.302 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:29.303 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:29.303 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:29.303 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:29.303 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-28T22:48:29.303 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:28 smithi043 ceph-mon[31823]: pgmap v354: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T22:48:29.985 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-03-28T22:48:29.985 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-03-28T22:48:29.985 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=0 2024-03-28T22:48:29.986 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=38 2024-03-28T22:48:29.986 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-03-28T22:48:29.986 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-03-28T22:48:29.986 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T22:48:29.986 DEBUG:tasks.check_counter:Getting stats from l 2024-03-28T22:48:29.986 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be shell --fsid c2276428-ed51-11ee-95d0-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-28T22:48:30.281 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T22:48:31.136 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:30 smithi172 ceph-mon[36566]: pgmap v355: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:48:31.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:30 smithi082 ceph-mon[36471]: pgmap v355: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:48:31.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:30 smithi043 ceph-mon[31823]: pgmap v355: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:48:32.134 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-28T22:48:32.135 INFO:teuthology.orchestra.run.smithi172.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T22:48:32.135 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_messages": 1739, 2024-03-28T22:48:32.135 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_messages": 1741, 2024-03-28T22:48:32.135 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_bytes": 522115, 2024-03-28T22:48:32.135 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_bytes": 553003, 2024-03-28T22:48:32.135 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_created_connections": 6, 2024-03-28T22:48:32.135 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_active_connections": 2, 2024-03-28T22:48:32.136 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_total_time": 0.116128998, 2024-03-28T22:48:32.136 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_send_time": 0.038301398, 2024-03-28T22:48:32.136 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_recv_time": 0.029205491, 2024-03-28T22:48:32.136 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_fast_dispatch_time": 0.038787019, 2024-03-28T22:48:32.136 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:48:32.136 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 1739, 2024-03-28T22:48:32.136 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.036020969, 2024-03-28T22:48:32.136 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000020713 2024-03-28T22:48:32.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.137 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:48:32.137 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.137 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.137 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.137 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T22:48:32.137 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T22:48:32.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.138 INFO:teuthology.orchestra.run.smithi172.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T22:48:32.138 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_messages": 1000, 2024-03-28T22:48:32.138 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_messages": 981, 2024-03-28T22:48:32.138 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_bytes": 2087420, 2024-03-28T22:48:32.138 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_bytes": 384148, 2024-03-28T22:48:32.138 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_created_connections": 4, 2024-03-28T22:48:32.138 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_active_connections": 3, 2024-03-28T22:48:32.139 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_total_time": 0.099443508, 2024-03-28T22:48:32.139 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_send_time": 0.027629350, 2024-03-28T22:48:32.139 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_recv_time": 0.056012910, 2024-03-28T22:48:32.139 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_fast_dispatch_time": 0.025993379, 2024-03-28T22:48:32.139 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:48:32.139 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 981, 2024-03-28T22:48:32.139 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.024125326, 2024-03-28T22:48:32.139 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000024592 2024-03-28T22:48:32.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.140 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:48:32.140 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.140 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.140 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.140 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_encrypted_bytes": 1913584, 2024-03-28T22:48:32.140 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_encrypted_bytes": 160208 2024-03-28T22:48:32.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.141 INFO:teuthology.orchestra.run.smithi172.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T22:48:32.141 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_messages": 3, 2024-03-28T22:48:32.141 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_messages": 195, 2024-03-28T22:48:32.141 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_bytes": 588, 2024-03-28T22:48:32.141 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_bytes": 564590, 2024-03-28T22:48:32.141 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_created_connections": 7, 2024-03-28T22:48:32.141 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_active_connections": 4, 2024-03-28T22:48:32.141 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_total_time": 0.009140122, 2024-03-28T22:48:32.142 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_send_time": 0.005672972, 2024-03-28T22:48:32.142 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_recv_time": 0.001756384, 2024-03-28T22:48:32.142 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_running_fast_dispatch_time": 0.000022699, 2024-03-28T22:48:32.142 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T22:48:32.142 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 192, 2024-03-28T22:48:32.142 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.030514758, 2024-03-28T22:48:32.142 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000158931 2024-03-28T22:48:32.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.143 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_handle_ack_lat": { 2024-03-28T22:48:32.143 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.143 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.143 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.143 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T22:48:32.143 INFO:teuthology.orchestra.run.smithi172.stdout: "msgr_send_encrypted_bytes": 564272 2024-03-28T22:48:32.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.143 INFO:teuthology.orchestra.run.smithi172.stdout: "cct": { 2024-03-28T22:48:32.144 INFO:teuthology.orchestra.run.smithi172.stdout: "total_workers": 1, 2024-03-28T22:48:32.144 INFO:teuthology.orchestra.run.smithi172.stdout: "unhealthy_workers": 0 2024-03-28T22:48:32.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.144 INFO:teuthology.orchestra.run.smithi172.stdout: "finisher-MDSRank": { 2024-03-28T22:48:32.144 INFO:teuthology.orchestra.run.smithi172.stdout: "queue_len": 0, 2024-03-28T22:48:32.144 INFO:teuthology.orchestra.run.smithi172.stdout: "complete_latency": { 2024-03-28T22:48:32.144 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 4150, 2024-03-28T22:48:32.144 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.091356332, 2024-03-28T22:48:32.144 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000022013 2024-03-28T22:48:32.145 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:32.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.145 INFO:teuthology.orchestra.run.smithi172.stdout: "finisher-PurgeQueue": { 2024-03-28T22:48:32.145 INFO:teuthology.orchestra.run.smithi172.stdout: "queue_len": 0, 2024-03-28T22:48:32.145 INFO:teuthology.orchestra.run.smithi172.stdout: "complete_latency": { 2024-03-28T22:48:32.145 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.145 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.145 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.145 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:32.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.146 INFO:teuthology.orchestra.run.smithi172.stdout: "mds": { 2024-03-28T22:48:32.146 INFO:teuthology.orchestra.run.smithi172.stdout: "request": 0, 2024-03-28T22:48:32.146 INFO:teuthology.orchestra.run.smithi172.stdout: "reply": 0, 2024-03-28T22:48:32.146 INFO:teuthology.orchestra.run.smithi172.stdout: "reply_latency": { 2024-03-28T22:48:32.146 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.146 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.146 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.147 INFO:teuthology.orchestra.run.smithi172.stdout: "slow_reply": 0, 2024-03-28T22:48:32.147 INFO:teuthology.orchestra.run.smithi172.stdout: "forward": 0, 2024-03-28T22:48:32.147 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_fetch_complete": 0, 2024-03-28T22:48:32.147 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_fetch_keys": 0, 2024-03-28T22:48:32.147 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_commit": 0, 2024-03-28T22:48:32.147 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_split": 0, 2024-03-28T22:48:32.147 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_merge": 0, 2024-03-28T22:48:32.148 INFO:teuthology.orchestra.run.smithi172.stdout: "inodes": 0, 2024-03-28T22:48:32.148 INFO:teuthology.orchestra.run.smithi172.stdout: "inodes_top": 0, 2024-03-28T22:48:32.148 INFO:teuthology.orchestra.run.smithi172.stdout: "inodes_bottom": 0, 2024-03-28T22:48:32.148 INFO:teuthology.orchestra.run.smithi172.stdout: "inodes_pin_tail": 0, 2024-03-28T22:48:32.148 INFO:teuthology.orchestra.run.smithi172.stdout: "inodes_pinned": 0, 2024-03-28T22:48:32.148 INFO:teuthology.orchestra.run.smithi172.stdout: "inodes_expired": 0, 2024-03-28T22:48:32.148 INFO:teuthology.orchestra.run.smithi172.stdout: "inodes_with_caps": 0, 2024-03-28T22:48:32.148 INFO:teuthology.orchestra.run.smithi172.stdout: "caps": 0, 2024-03-28T22:48:32.148 INFO:teuthology.orchestra.run.smithi172.stdout: "subtrees": 1, 2024-03-28T22:48:32.149 INFO:teuthology.orchestra.run.smithi172.stdout: "traverse": 0, 2024-03-28T22:48:32.149 INFO:teuthology.orchestra.run.smithi172.stdout: "traverse_hit": 0, 2024-03-28T22:48:32.149 INFO:teuthology.orchestra.run.smithi172.stdout: "traverse_forward": 0, 2024-03-28T22:48:32.149 INFO:teuthology.orchestra.run.smithi172.stdout: "traverse_discover": 0, 2024-03-28T22:48:32.149 INFO:teuthology.orchestra.run.smithi172.stdout: "traverse_dir_fetch": 0, 2024-03-28T22:48:32.149 INFO:teuthology.orchestra.run.smithi172.stdout: "traverse_remote_ino": 0, 2024-03-28T22:48:32.149 INFO:teuthology.orchestra.run.smithi172.stdout: "traverse_lock": 0, 2024-03-28T22:48:32.149 INFO:teuthology.orchestra.run.smithi172.stdout: "load_cent": 0, 2024-03-28T22:48:32.149 INFO:teuthology.orchestra.run.smithi172.stdout: "q": 0, 2024-03-28T22:48:32.150 INFO:teuthology.orchestra.run.smithi172.stdout: "exported": 0, 2024-03-28T22:48:32.150 INFO:teuthology.orchestra.run.smithi172.stdout: "exported_inodes": 0, 2024-03-28T22:48:32.150 INFO:teuthology.orchestra.run.smithi172.stdout: "imported": 0, 2024-03-28T22:48:32.150 INFO:teuthology.orchestra.run.smithi172.stdout: "imported_inodes": 0, 2024-03-28T22:48:32.150 INFO:teuthology.orchestra.run.smithi172.stdout: "openino_dir_fetch": 0, 2024-03-28T22:48:32.150 INFO:teuthology.orchestra.run.smithi172.stdout: "openino_backtrace_fetch": 0, 2024-03-28T22:48:32.150 INFO:teuthology.orchestra.run.smithi172.stdout: "openino_peer_discover": 0, 2024-03-28T22:48:32.150 INFO:teuthology.orchestra.run.smithi172.stdout: "root_rfiles": 0, 2024-03-28T22:48:32.151 INFO:teuthology.orchestra.run.smithi172.stdout: "root_rbytes": 0, 2024-03-28T22:48:32.151 INFO:teuthology.orchestra.run.smithi172.stdout: "root_rsnaps": 0, 2024-03-28T22:48:32.151 INFO:teuthology.orchestra.run.smithi172.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T22:48:32.151 INFO:teuthology.orchestra.run.smithi172.stdout: "scrub_set_tag": 0, 2024-03-28T22:48:32.151 INFO:teuthology.orchestra.run.smithi172.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T22:48:32.151 INFO:teuthology.orchestra.run.smithi172.stdout: "scrub_inotable_repaired": 0, 2024-03-28T22:48:32.151 INFO:teuthology.orchestra.run.smithi172.stdout: "scrub_dir_inodes": 0, 2024-03-28T22:48:32.151 INFO:teuthology.orchestra.run.smithi172.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T22:48:32.152 INFO:teuthology.orchestra.run.smithi172.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T22:48:32.152 INFO:teuthology.orchestra.run.smithi172.stdout: "scrub_file_inodes": 0, 2024-03-28T22:48:32.152 INFO:teuthology.orchestra.run.smithi172.stdout: "handle_inode_file_caps": 0, 2024-03-28T22:48:32.152 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T22:48:32.152 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph_cap_op_grant": 0, 2024-03-28T22:48:32.152 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T22:48:32.152 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T22:48:32.152 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T22:48:32.153 INFO:teuthology.orchestra.run.smithi172.stdout: "handle_client_caps": 0, 2024-03-28T22:48:32.153 INFO:teuthology.orchestra.run.smithi172.stdout: "handle_client_caps_dirty": 0, 2024-03-28T22:48:32.153 INFO:teuthology.orchestra.run.smithi172.stdout: "handle_client_cap_release": 0, 2024-03-28T22:48:32.153 INFO:teuthology.orchestra.run.smithi172.stdout: "process_request_cap_release": 0 2024-03-28T22:48:32.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.153 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_cache": { 2024-03-28T22:48:32.153 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_update": 0, 2024-03-28T22:48:32.153 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_update_receipt": 0, 2024-03-28T22:48:32.153 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_try_discover": 0, 2024-03-28T22:48:32.154 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_send_discover": 0, 2024-03-28T22:48:32.154 INFO:teuthology.orchestra.run.smithi172.stdout: "dir_handle_discover": 0, 2024-03-28T22:48:32.154 INFO:teuthology.orchestra.run.smithi172.stdout: "num_strays": 0, 2024-03-28T22:48:32.154 INFO:teuthology.orchestra.run.smithi172.stdout: "num_strays_delayed": 0, 2024-03-28T22:48:32.154 INFO:teuthology.orchestra.run.smithi172.stdout: "num_strays_enqueuing": 0, 2024-03-28T22:48:32.154 INFO:teuthology.orchestra.run.smithi172.stdout: "strays_created": 0, 2024-03-28T22:48:32.154 INFO:teuthology.orchestra.run.smithi172.stdout: "strays_enqueued": 0, 2024-03-28T22:48:32.154 INFO:teuthology.orchestra.run.smithi172.stdout: "strays_reintegrated": 0, 2024-03-28T22:48:32.155 INFO:teuthology.orchestra.run.smithi172.stdout: "strays_migrated": 0, 2024-03-28T22:48:32.155 INFO:teuthology.orchestra.run.smithi172.stdout: "num_recovering_processing": 0, 2024-03-28T22:48:32.155 INFO:teuthology.orchestra.run.smithi172.stdout: "num_recovering_enqueued": 0, 2024-03-28T22:48:32.155 INFO:teuthology.orchestra.run.smithi172.stdout: "num_recovering_prioritized": 0, 2024-03-28T22:48:32.155 INFO:teuthology.orchestra.run.smithi172.stdout: "recovery_started": 0, 2024-03-28T22:48:32.155 INFO:teuthology.orchestra.run.smithi172.stdout: "recovery_completed": 0, 2024-03-28T22:48:32.155 INFO:teuthology.orchestra.run.smithi172.stdout: "ireq_quiesce_path": 0, 2024-03-28T22:48:32.155 INFO:teuthology.orchestra.run.smithi172.stdout: "ireq_quiesce_inode": 0, 2024-03-28T22:48:32.156 INFO:teuthology.orchestra.run.smithi172.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T22:48:32.156 INFO:teuthology.orchestra.run.smithi172.stdout: "ireq_exportdir": 0, 2024-03-28T22:48:32.156 INFO:teuthology.orchestra.run.smithi172.stdout: "ireq_flush": 0, 2024-03-28T22:48:32.156 INFO:teuthology.orchestra.run.smithi172.stdout: "ireq_fragmentdir": 0, 2024-03-28T22:48:32.156 INFO:teuthology.orchestra.run.smithi172.stdout: "ireq_fragstats": 0, 2024-03-28T22:48:32.156 INFO:teuthology.orchestra.run.smithi172.stdout: "ireq_inodestats": 0 2024-03-28T22:48:32.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.156 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_log": { 2024-03-28T22:48:32.156 INFO:teuthology.orchestra.run.smithi172.stdout: "evlrg": 0, 2024-03-28T22:48:32.157 INFO:teuthology.orchestra.run.smithi172.stdout: "evadd": 0, 2024-03-28T22:48:32.157 INFO:teuthology.orchestra.run.smithi172.stdout: "evex": 0, 2024-03-28T22:48:32.157 INFO:teuthology.orchestra.run.smithi172.stdout: "evtrm": 0, 2024-03-28T22:48:32.157 INFO:teuthology.orchestra.run.smithi172.stdout: "ev": 9, 2024-03-28T22:48:32.157 INFO:teuthology.orchestra.run.smithi172.stdout: "evexg": 0, 2024-03-28T22:48:32.157 INFO:teuthology.orchestra.run.smithi172.stdout: "evexd": 0, 2024-03-28T22:48:32.157 INFO:teuthology.orchestra.run.smithi172.stdout: "segadd": 0, 2024-03-28T22:48:32.157 INFO:teuthology.orchestra.run.smithi172.stdout: "segex": 0, 2024-03-28T22:48:32.158 INFO:teuthology.orchestra.run.smithi172.stdout: "segtrm": 0, 2024-03-28T22:48:32.158 INFO:teuthology.orchestra.run.smithi172.stdout: "seg": 2, 2024-03-28T22:48:32.158 INFO:teuthology.orchestra.run.smithi172.stdout: "segmjr": 2, 2024-03-28T22:48:32.158 INFO:teuthology.orchestra.run.smithi172.stdout: "segexg": 0, 2024-03-28T22:48:32.158 INFO:teuthology.orchestra.run.smithi172.stdout: "segexd": 0, 2024-03-28T22:48:32.158 INFO:teuthology.orchestra.run.smithi172.stdout: "expos": 4194304, 2024-03-28T22:48:32.158 INFO:teuthology.orchestra.run.smithi172.stdout: "wrpos": 4196794, 2024-03-28T22:48:32.158 INFO:teuthology.orchestra.run.smithi172.stdout: "rdpos": 4196653, 2024-03-28T22:48:32.158 INFO:teuthology.orchestra.run.smithi172.stdout: "jlat": { 2024-03-28T22:48:32.159 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.159 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.159 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.159 INFO:teuthology.orchestra.run.smithi172.stdout: "replayed": 9 2024-03-28T22:48:32.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.159 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_mem": { 2024-03-28T22:48:32.159 INFO:teuthology.orchestra.run.smithi172.stdout: "ino": 3, 2024-03-28T22:48:32.160 INFO:teuthology.orchestra.run.smithi172.stdout: "ino+": 3, 2024-03-28T22:48:32.160 INFO:teuthology.orchestra.run.smithi172.stdout: "ino-": 0, 2024-03-28T22:48:32.160 INFO:teuthology.orchestra.run.smithi172.stdout: "dir": 1, 2024-03-28T22:48:32.160 INFO:teuthology.orchestra.run.smithi172.stdout: "dir+": 1, 2024-03-28T22:48:32.160 INFO:teuthology.orchestra.run.smithi172.stdout: "dir-": 0, 2024-03-28T22:48:32.160 INFO:teuthology.orchestra.run.smithi172.stdout: "dn": 0, 2024-03-28T22:48:32.160 INFO:teuthology.orchestra.run.smithi172.stdout: "dn+": 0, 2024-03-28T22:48:32.160 INFO:teuthology.orchestra.run.smithi172.stdout: "dn-": 0, 2024-03-28T22:48:32.161 INFO:teuthology.orchestra.run.smithi172.stdout: "cap": 0, 2024-03-28T22:48:32.161 INFO:teuthology.orchestra.run.smithi172.stdout: "cap+": 0, 2024-03-28T22:48:32.161 INFO:teuthology.orchestra.run.smithi172.stdout: "cap-": 0, 2024-03-28T22:48:32.161 INFO:teuthology.orchestra.run.smithi172.stdout: "rss": 48448, 2024-03-28T22:48:32.161 INFO:teuthology.orchestra.run.smithi172.stdout: "heap": 215420 2024-03-28T22:48:32.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.161 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_server": { 2024-03-28T22:48:32.161 INFO:teuthology.orchestra.run.smithi172.stdout: "dispatch_client_request": 0, 2024-03-28T22:48:32.161 INFO:teuthology.orchestra.run.smithi172.stdout: "dispatch_server_request": 0, 2024-03-28T22:48:32.162 INFO:teuthology.orchestra.run.smithi172.stdout: "handle_client_request": 0, 2024-03-28T22:48:32.162 INFO:teuthology.orchestra.run.smithi172.stdout: "handle_client_session": 0, 2024-03-28T22:48:32.162 INFO:teuthology.orchestra.run.smithi172.stdout: "handle_peer_request": 0, 2024-03-28T22:48:32.162 INFO:teuthology.orchestra.run.smithi172.stdout: "req_create_latency": { 2024-03-28T22:48:32.162 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.162 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.162 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.163 INFO:teuthology.orchestra.run.smithi172.stdout: "req_getattr_latency": { 2024-03-28T22:48:32.163 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.163 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.163 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.163 INFO:teuthology.orchestra.run.smithi172.stdout: "req_getfilelock_latency": { 2024-03-28T22:48:32.163 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.163 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.164 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.164 INFO:teuthology.orchestra.run.smithi172.stdout: "req_link_latency": { 2024-03-28T22:48:32.164 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.164 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.164 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.165 INFO:teuthology.orchestra.run.smithi172.stdout: "req_lookup_latency": { 2024-03-28T22:48:32.165 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.165 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.165 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.165 INFO:teuthology.orchestra.run.smithi172.stdout: "req_lookuphash_latency": { 2024-03-28T22:48:32.165 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.165 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.166 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.166 INFO:teuthology.orchestra.run.smithi172.stdout: "req_lookupino_latency": { 2024-03-28T22:48:32.166 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.166 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.166 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.166 INFO:teuthology.orchestra.run.smithi172.stdout: "req_lookupname_latency": { 2024-03-28T22:48:32.167 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.167 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.167 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.167 INFO:teuthology.orchestra.run.smithi172.stdout: "req_lookupparent_latency": { 2024-03-28T22:48:32.167 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.167 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.167 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.168 INFO:teuthology.orchestra.run.smithi172.stdout: "req_lookupsnap_latency": { 2024-03-28T22:48:32.168 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.168 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.168 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.168 INFO:teuthology.orchestra.run.smithi172.stdout: "req_lssnap_latency": { 2024-03-28T22:48:32.168 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.168 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.169 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.169 INFO:teuthology.orchestra.run.smithi172.stdout: "req_mkdir_latency": { 2024-03-28T22:48:32.169 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.170 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.170 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.170 INFO:teuthology.orchestra.run.smithi172.stdout: "req_mknod_latency": { 2024-03-28T22:48:32.170 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.170 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.170 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.171 INFO:teuthology.orchestra.run.smithi172.stdout: "req_mksnap_latency": { 2024-03-28T22:48:32.171 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.171 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.171 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.171 INFO:teuthology.orchestra.run.smithi172.stdout: "req_open_latency": { 2024-03-28T22:48:32.171 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.171 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.172 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.172 INFO:teuthology.orchestra.run.smithi172.stdout: "req_readdir_latency": { 2024-03-28T22:48:32.172 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.172 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.172 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.172 INFO:teuthology.orchestra.run.smithi172.stdout: "req_rename_latency": { 2024-03-28T22:48:32.172 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.173 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.173 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.173 INFO:teuthology.orchestra.run.smithi172.stdout: "req_renamesnap_latency": { 2024-03-28T22:48:32.173 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.173 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.173 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.174 INFO:teuthology.orchestra.run.smithi172.stdout: "req_snapdiff_latency": { 2024-03-28T22:48:32.174 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.174 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.174 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.174 INFO:teuthology.orchestra.run.smithi172.stdout: "req_rmdir_latency": { 2024-03-28T22:48:32.174 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.174 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.174 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.175 INFO:teuthology.orchestra.run.smithi172.stdout: "req_rmsnap_latency": { 2024-03-28T22:48:32.175 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.175 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.175 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.175 INFO:teuthology.orchestra.run.smithi172.stdout: "req_rmxattr_latency": { 2024-03-28T22:48:32.175 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.176 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.176 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.176 INFO:teuthology.orchestra.run.smithi172.stdout: "req_setattr_latency": { 2024-03-28T22:48:32.176 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.176 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.176 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.176 INFO:teuthology.orchestra.run.smithi172.stdout: "req_setdirlayout_latency": { 2024-03-28T22:48:32.177 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.177 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.177 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.177 INFO:teuthology.orchestra.run.smithi172.stdout: "req_setfilelock_latency": { 2024-03-28T22:48:32.177 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.177 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.177 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.178 INFO:teuthology.orchestra.run.smithi172.stdout: "req_setlayout_latency": { 2024-03-28T22:48:32.178 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.178 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.178 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.178 INFO:teuthology.orchestra.run.smithi172.stdout: "req_setxattr_latency": { 2024-03-28T22:48:32.178 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.178 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.179 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.179 INFO:teuthology.orchestra.run.smithi172.stdout: "req_symlink_latency": { 2024-03-28T22:48:32.179 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.179 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.179 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.179 INFO:teuthology.orchestra.run.smithi172.stdout: "req_unlink_latency": { 2024-03-28T22:48:32.179 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.180 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.180 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.180 INFO:teuthology.orchestra.run.smithi172.stdout: "cap_revoke_eviction": 0, 2024-03-28T22:48:32.180 INFO:teuthology.orchestra.run.smithi172.stdout: "cap_acquisition_throttle": 0, 2024-03-28T22:48:32.180 INFO:teuthology.orchestra.run.smithi172.stdout: "req_getvxattr_latency": { 2024-03-28T22:48:32.180 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.180 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.180 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.181 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:32.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.181 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_sessions": { 2024-03-28T22:48:32.181 INFO:teuthology.orchestra.run.smithi172.stdout: "session_count": 1, 2024-03-28T22:48:32.181 INFO:teuthology.orchestra.run.smithi172.stdout: "session_add": 4, 2024-03-28T22:48:32.181 INFO:teuthology.orchestra.run.smithi172.stdout: "session_remove": 3, 2024-03-28T22:48:32.181 INFO:teuthology.orchestra.run.smithi172.stdout: "sessions_open": 3, 2024-03-28T22:48:32.181 INFO:teuthology.orchestra.run.smithi172.stdout: "sessions_stale": 0, 2024-03-28T22:48:32.182 INFO:teuthology.orchestra.run.smithi172.stdout: "total_load": 0, 2024-03-28T22:48:32.182 INFO:teuthology.orchestra.run.smithi172.stdout: "average_load": 0, 2024-03-28T22:48:32.182 INFO:teuthology.orchestra.run.smithi172.stdout: "avg_session_uptime": 5636, 2024-03-28T22:48:32.182 INFO:teuthology.orchestra.run.smithi172.stdout: "mdthresh_evicted": 0 2024-03-28T22:48:32.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.182 INFO:teuthology.orchestra.run.smithi172.stdout: "mempool": { 2024-03-28T22:48:32.182 INFO:teuthology.orchestra.run.smithi172.stdout: "bloom_filter_bytes": 0, 2024-03-28T22:48:32.182 INFO:teuthology.orchestra.run.smithi172.stdout: "bloom_filter_items": 0, 2024-03-28T22:48:32.183 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T22:48:32.183 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_alloc_items": 0, 2024-03-28T22:48:32.183 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T22:48:32.183 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_data_items": 0, 2024-03-28T22:48:32.183 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T22:48:32.183 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T22:48:32.183 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T22:48:32.183 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T22:48:32.183 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T22:48:32.184 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_other_items": 0, 2024-03-28T22:48:32.184 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T22:48:32.184 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T22:48:32.184 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_extent_bytes": 0, 2024-03-28T22:48:32.184 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_extent_items": 0, 2024-03-28T22:48:32.184 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_blob_bytes": 0, 2024-03-28T22:48:32.184 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_blob_items": 0, 2024-03-28T22:48:32.184 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T22:48:32.184 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T22:48:32.185 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T22:48:32.185 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T22:48:32.185 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T22:48:32.185 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_fsck_items": 0, 2024-03-28T22:48:32.185 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_txc_bytes": 0, 2024-03-28T22:48:32.185 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_txc_items": 0, 2024-03-28T22:48:32.185 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T22:48:32.185 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T22:48:32.185 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_writing_bytes": 0, 2024-03-28T22:48:32.186 INFO:teuthology.orchestra.run.smithi172.stdout: "bluestore_writing_items": 0, 2024-03-28T22:48:32.186 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_bytes": 0, 2024-03-28T22:48:32.186 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_items": 0, 2024-03-28T22:48:32.186 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T22:48:32.186 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_file_reader_items": 0, 2024-03-28T22:48:32.186 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T22:48:32.186 INFO:teuthology.orchestra.run.smithi172.stdout: "bluefs_file_writer_items": 0, 2024-03-28T22:48:32.186 INFO:teuthology.orchestra.run.smithi172.stdout: "buffer_anon_bytes": 190179, 2024-03-28T22:48:32.187 INFO:teuthology.orchestra.run.smithi172.stdout: "buffer_anon_items": 53, 2024-03-28T22:48:32.187 INFO:teuthology.orchestra.run.smithi172.stdout: "buffer_meta_bytes": 0, 2024-03-28T22:48:32.187 INFO:teuthology.orchestra.run.smithi172.stdout: "buffer_meta_items": 0, 2024-03-28T22:48:32.187 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_bytes": 0, 2024-03-28T22:48:32.187 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_items": 0, 2024-03-28T22:48:32.187 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mapbl_bytes": 0, 2024-03-28T22:48:32.187 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_mapbl_items": 0, 2024-03-28T22:48:32.187 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pglog_bytes": 0, 2024-03-28T22:48:32.187 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_pglog_items": 0, 2024-03-28T22:48:32.188 INFO:teuthology.orchestra.run.smithi172.stdout: "osdmap_bytes": 9736, 2024-03-28T22:48:32.188 INFO:teuthology.orchestra.run.smithi172.stdout: "osdmap_items": 203, 2024-03-28T22:48:32.188 INFO:teuthology.orchestra.run.smithi172.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T22:48:32.188 INFO:teuthology.orchestra.run.smithi172.stdout: "osdmap_mapping_items": 0, 2024-03-28T22:48:32.188 INFO:teuthology.orchestra.run.smithi172.stdout: "pgmap_bytes": 0, 2024-03-28T22:48:32.188 INFO:teuthology.orchestra.run.smithi172.stdout: "pgmap_items": 0, 2024-03-28T22:48:32.188 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_co_bytes": 8464, 2024-03-28T22:48:32.188 INFO:teuthology.orchestra.run.smithi172.stdout: "mds_co_items": 13, 2024-03-28T22:48:32.188 INFO:teuthology.orchestra.run.smithi172.stdout: "unittest_1_bytes": 0, 2024-03-28T22:48:32.189 INFO:teuthology.orchestra.run.smithi172.stdout: "unittest_1_items": 0, 2024-03-28T22:48:32.189 INFO:teuthology.orchestra.run.smithi172.stdout: "unittest_2_bytes": 0, 2024-03-28T22:48:32.189 INFO:teuthology.orchestra.run.smithi172.stdout: "unittest_2_items": 0 2024-03-28T22:48:32.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.189 INFO:teuthology.orchestra.run.smithi172.stdout: "objecter": { 2024-03-28T22:48:32.189 INFO:teuthology.orchestra.run.smithi172.stdout: "op_active": 0, 2024-03-28T22:48:32.189 INFO:teuthology.orchestra.run.smithi172.stdout: "op_laggy": 0, 2024-03-28T22:48:32.189 INFO:teuthology.orchestra.run.smithi172.stdout: "op_send": 2444, 2024-03-28T22:48:32.189 INFO:teuthology.orchestra.run.smithi172.stdout: "op_send_bytes": 21, 2024-03-28T22:48:32.190 INFO:teuthology.orchestra.run.smithi172.stdout: "op_resend": 3, 2024-03-28T22:48:32.190 INFO:teuthology.orchestra.run.smithi172.stdout: "op_reply": 2444, 2024-03-28T22:48:32.190 INFO:teuthology.orchestra.run.smithi172.stdout: "op_latency": { 2024-03-28T22:48:32.190 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 2444, 2024-03-28T22:48:32.190 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 1.027976215, 2024-03-28T22:48:32.190 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000420612 2024-03-28T22:48:32.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.190 INFO:teuthology.orchestra.run.smithi172.stdout: "op_inflight": 0, 2024-03-28T22:48:32.190 INFO:teuthology.orchestra.run.smithi172.stdout: "oplen_avg": { 2024-03-28T22:48:32.191 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 2444, 2024-03-28T22:48:32.191 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 2445 2024-03-28T22:48:32.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.191 INFO:teuthology.orchestra.run.smithi172.stdout: "op": 2444, 2024-03-28T22:48:32.191 INFO:teuthology.orchestra.run.smithi172.stdout: "op_r": 2444, 2024-03-28T22:48:32.191 INFO:teuthology.orchestra.run.smithi172.stdout: "op_w": 0, 2024-03-28T22:48:32.191 INFO:teuthology.orchestra.run.smithi172.stdout: "op_rmw": 0, 2024-03-28T22:48:32.191 INFO:teuthology.orchestra.run.smithi172.stdout: "op_pg": 0, 2024-03-28T22:48:32.192 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_stat": 1567, 2024-03-28T22:48:32.192 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_create": 0, 2024-03-28T22:48:32.192 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_read": 875, 2024-03-28T22:48:32.192 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_write": 0, 2024-03-28T22:48:32.192 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_writefull": 0, 2024-03-28T22:48:32.192 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_writesame": 0, 2024-03-28T22:48:32.192 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_append": 0, 2024-03-28T22:48:32.192 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_zero": 0, 2024-03-28T22:48:32.193 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_truncate": 0, 2024-03-28T22:48:32.193 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_delete": 0, 2024-03-28T22:48:32.193 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_mapext": 0, 2024-03-28T22:48:32.193 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_sparse_read": 0, 2024-03-28T22:48:32.193 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_clonerange": 0, 2024-03-28T22:48:32.193 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_getxattr": 1, 2024-03-28T22:48:32.193 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_setxattr": 0, 2024-03-28T22:48:32.193 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_cmpxattr": 0, 2024-03-28T22:48:32.193 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_rmxattr": 0, 2024-03-28T22:48:32.194 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_resetxattrs": 0, 2024-03-28T22:48:32.194 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_call": 0, 2024-03-28T22:48:32.194 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_watch": 0, 2024-03-28T22:48:32.194 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_notify": 0, 2024-03-28T22:48:32.194 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T22:48:32.194 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_pgls": 0, 2024-03-28T22:48:32.194 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_pgls_filter": 0, 2024-03-28T22:48:32.194 INFO:teuthology.orchestra.run.smithi172.stdout: "osdop_other": 0, 2024-03-28T22:48:32.195 INFO:teuthology.orchestra.run.smithi172.stdout: "linger_active": 0, 2024-03-28T22:48:32.195 INFO:teuthology.orchestra.run.smithi172.stdout: "linger_send": 0, 2024-03-28T22:48:32.195 INFO:teuthology.orchestra.run.smithi172.stdout: "linger_resend": 0, 2024-03-28T22:48:32.195 INFO:teuthology.orchestra.run.smithi172.stdout: "linger_ping": 0, 2024-03-28T22:48:32.195 INFO:teuthology.orchestra.run.smithi172.stdout: "poolop_active": 0, 2024-03-28T22:48:32.195 INFO:teuthology.orchestra.run.smithi172.stdout: "poolop_send": 0, 2024-03-28T22:48:32.195 INFO:teuthology.orchestra.run.smithi172.stdout: "poolop_resend": 0, 2024-03-28T22:48:32.195 INFO:teuthology.orchestra.run.smithi172.stdout: "poolstat_active": 0, 2024-03-28T22:48:32.195 INFO:teuthology.orchestra.run.smithi172.stdout: "poolstat_send": 0, 2024-03-28T22:48:32.196 INFO:teuthology.orchestra.run.smithi172.stdout: "poolstat_resend": 0, 2024-03-28T22:48:32.196 INFO:teuthology.orchestra.run.smithi172.stdout: "statfs_active": 0, 2024-03-28T22:48:32.196 INFO:teuthology.orchestra.run.smithi172.stdout: "statfs_send": 0, 2024-03-28T22:48:32.196 INFO:teuthology.orchestra.run.smithi172.stdout: "statfs_resend": 0, 2024-03-28T22:48:32.196 INFO:teuthology.orchestra.run.smithi172.stdout: "command_active": 0, 2024-03-28T22:48:32.196 INFO:teuthology.orchestra.run.smithi172.stdout: "command_send": 0, 2024-03-28T22:48:32.196 INFO:teuthology.orchestra.run.smithi172.stdout: "command_resend": 0, 2024-03-28T22:48:32.196 INFO:teuthology.orchestra.run.smithi172.stdout: "map_epoch": 110, 2024-03-28T22:48:32.197 INFO:teuthology.orchestra.run.smithi172.stdout: "map_full": 0, 2024-03-28T22:48:32.197 INFO:teuthology.orchestra.run.smithi172.stdout: "map_inc": 18, 2024-03-28T22:48:32.197 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_sessions": 5, 2024-03-28T22:48:32.197 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_session_open": 5, 2024-03-28T22:48:32.197 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_session_close": 0, 2024-03-28T22:48:32.197 INFO:teuthology.orchestra.run.smithi172.stdout: "osd_laggy": 0, 2024-03-28T22:48:32.197 INFO:teuthology.orchestra.run.smithi172.stdout: "omap_wr": 0, 2024-03-28T22:48:32.197 INFO:teuthology.orchestra.run.smithi172.stdout: "omap_rd": 2, 2024-03-28T22:48:32.197 INFO:teuthology.orchestra.run.smithi172.stdout: "omap_del": 0 2024-03-28T22:48:32.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.198 INFO:teuthology.orchestra.run.smithi172.stdout: "oft": { 2024-03-28T22:48:32.198 INFO:teuthology.orchestra.run.smithi172.stdout: "omap_total_objs": 0, 2024-03-28T22:48:32.198 INFO:teuthology.orchestra.run.smithi172.stdout: "omap_total_kv_pairs": 0, 2024-03-28T22:48:32.198 INFO:teuthology.orchestra.run.smithi172.stdout: "omap_total_updates": 0, 2024-03-28T22:48:32.198 INFO:teuthology.orchestra.run.smithi172.stdout: "omap_total_removes": 0 2024-03-28T22:48:32.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.198 INFO:teuthology.orchestra.run.smithi172.stdout: "osd-slow-ops": { 2024-03-28T22:48:32.199 INFO:teuthology.orchestra.run.smithi172.stdout: "slow_ops_count": 0 2024-03-28T22:48:32.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.199 INFO:teuthology.orchestra.run.smithi172.stdout: "purge_queue": { 2024-03-28T22:48:32.199 INFO:teuthology.orchestra.run.smithi172.stdout: "pq_executing_ops": 0, 2024-03-28T22:48:32.199 INFO:teuthology.orchestra.run.smithi172.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T22:48:32.199 INFO:teuthology.orchestra.run.smithi172.stdout: "pq_executing": 0, 2024-03-28T22:48:32.199 INFO:teuthology.orchestra.run.smithi172.stdout: "pq_executing_high_water": 0, 2024-03-28T22:48:32.199 INFO:teuthology.orchestra.run.smithi172.stdout: "pq_executed_ops": 0, 2024-03-28T22:48:32.199 INFO:teuthology.orchestra.run.smithi172.stdout: "pq_executed": 0, 2024-03-28T22:48:32.200 INFO:teuthology.orchestra.run.smithi172.stdout: "pq_item_in_journal": 0 2024-03-28T22:48:32.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.200 INFO:teuthology.orchestra.run.smithi172.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T22:48:32.200 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 0, 2024-03-28T22:48:32.200 INFO:teuthology.orchestra.run.smithi172.stdout: "max": 104857600, 2024-03-28T22:48:32.200 INFO:teuthology.orchestra.run.smithi172.stdout: "get_started": 0, 2024-03-28T22:48:32.200 INFO:teuthology.orchestra.run.smithi172.stdout: "get": 2738, 2024-03-28T22:48:32.200 INFO:teuthology.orchestra.run.smithi172.stdout: "get_sum": 2345741, 2024-03-28T22:48:32.200 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:32.201 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_success": 2738, 2024-03-28T22:48:32.201 INFO:teuthology.orchestra.run.smithi172.stdout: "take": 0, 2024-03-28T22:48:32.201 INFO:teuthology.orchestra.run.smithi172.stdout: "take_sum": 0, 2024-03-28T22:48:32.201 INFO:teuthology.orchestra.run.smithi172.stdout: "put": 2738, 2024-03-28T22:48:32.201 INFO:teuthology.orchestra.run.smithi172.stdout: "put_sum": 2345741, 2024-03-28T22:48:32.201 INFO:teuthology.orchestra.run.smithi172.stdout: "wait": { 2024-03-28T22:48:32.201 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.201 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.201 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.202 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:32.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.202 INFO:teuthology.orchestra.run.smithi172.stdout: "throttle-objecter_bytes": { 2024-03-28T22:48:32.202 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 0, 2024-03-28T22:48:32.202 INFO:teuthology.orchestra.run.smithi172.stdout: "max": 104857600, 2024-03-28T22:48:32.202 INFO:teuthology.orchestra.run.smithi172.stdout: "get_started": 0, 2024-03-28T22:48:32.202 INFO:teuthology.orchestra.run.smithi172.stdout: "get": 0, 2024-03-28T22:48:32.202 INFO:teuthology.orchestra.run.smithi172.stdout: "get_sum": 0, 2024-03-28T22:48:32.202 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:32.203 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_success": 0, 2024-03-28T22:48:32.203 INFO:teuthology.orchestra.run.smithi172.stdout: "take": 2444, 2024-03-28T22:48:32.203 INFO:teuthology.orchestra.run.smithi172.stdout: "take_sum": 2495, 2024-03-28T22:48:32.203 INFO:teuthology.orchestra.run.smithi172.stdout: "put": 2444, 2024-03-28T22:48:32.203 INFO:teuthology.orchestra.run.smithi172.stdout: "put_sum": 2495, 2024-03-28T22:48:32.203 INFO:teuthology.orchestra.run.smithi172.stdout: "wait": { 2024-03-28T22:48:32.203 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.203 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.204 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.204 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:32.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.204 INFO:teuthology.orchestra.run.smithi172.stdout: "throttle-objecter_ops": { 2024-03-28T22:48:32.204 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 0, 2024-03-28T22:48:32.204 INFO:teuthology.orchestra.run.smithi172.stdout: "max": 1024, 2024-03-28T22:48:32.204 INFO:teuthology.orchestra.run.smithi172.stdout: "get_started": 0, 2024-03-28T22:48:32.204 INFO:teuthology.orchestra.run.smithi172.stdout: "get": 0, 2024-03-28T22:48:32.204 INFO:teuthology.orchestra.run.smithi172.stdout: "get_sum": 0, 2024-03-28T22:48:32.205 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:32.205 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_success": 0, 2024-03-28T22:48:32.205 INFO:teuthology.orchestra.run.smithi172.stdout: "take": 2444, 2024-03-28T22:48:32.205 INFO:teuthology.orchestra.run.smithi172.stdout: "take_sum": 2444, 2024-03-28T22:48:32.205 INFO:teuthology.orchestra.run.smithi172.stdout: "put": 2444, 2024-03-28T22:48:32.205 INFO:teuthology.orchestra.run.smithi172.stdout: "put_sum": 2444, 2024-03-28T22:48:32.205 INFO:teuthology.orchestra.run.smithi172.stdout: "wait": { 2024-03-28T22:48:32.205 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.206 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.206 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.206 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:32.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.206 INFO:teuthology.orchestra.run.smithi172.stdout: "throttle-write_buf_throttle": { 2024-03-28T22:48:32.206 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 0, 2024-03-28T22:48:32.206 INFO:teuthology.orchestra.run.smithi172.stdout: "max": 3758096384, 2024-03-28T22:48:32.206 INFO:teuthology.orchestra.run.smithi172.stdout: "get_started": 0, 2024-03-28T22:48:32.206 INFO:teuthology.orchestra.run.smithi172.stdout: "get": 0, 2024-03-28T22:48:32.207 INFO:teuthology.orchestra.run.smithi172.stdout: "get_sum": 0, 2024-03-28T22:48:32.207 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:32.207 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_success": 0, 2024-03-28T22:48:32.207 INFO:teuthology.orchestra.run.smithi172.stdout: "take": 0, 2024-03-28T22:48:32.207 INFO:teuthology.orchestra.run.smithi172.stdout: "take_sum": 0, 2024-03-28T22:48:32.207 INFO:teuthology.orchestra.run.smithi172.stdout: "put": 0, 2024-03-28T22:48:32.207 INFO:teuthology.orchestra.run.smithi172.stdout: "put_sum": 0, 2024-03-28T22:48:32.207 INFO:teuthology.orchestra.run.smithi172.stdout: "wait": { 2024-03-28T22:48:32.208 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.208 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.208 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.208 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:32.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-28T22:48:32.208 INFO:teuthology.orchestra.run.smithi172.stdout: "throttle-write_buf_throttle-0x55f50e6a46e0": { 2024-03-28T22:48:32.208 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 0, 2024-03-28T22:48:32.208 INFO:teuthology.orchestra.run.smithi172.stdout: "max": 3758096384, 2024-03-28T22:48:32.208 INFO:teuthology.orchestra.run.smithi172.stdout: "get_started": 0, 2024-03-28T22:48:32.209 INFO:teuthology.orchestra.run.smithi172.stdout: "get": 0, 2024-03-28T22:48:32.209 INFO:teuthology.orchestra.run.smithi172.stdout: "get_sum": 0, 2024-03-28T22:48:32.209 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_fail": 0, 2024-03-28T22:48:32.209 INFO:teuthology.orchestra.run.smithi172.stdout: "get_or_fail_success": 0, 2024-03-28T22:48:32.209 INFO:teuthology.orchestra.run.smithi172.stdout: "take": 0, 2024-03-28T22:48:32.209 INFO:teuthology.orchestra.run.smithi172.stdout: "take_sum": 0, 2024-03-28T22:48:32.209 INFO:teuthology.orchestra.run.smithi172.stdout: "put": 0, 2024-03-28T22:48:32.209 INFO:teuthology.orchestra.run.smithi172.stdout: "put_sum": 0, 2024-03-28T22:48:32.209 INFO:teuthology.orchestra.run.smithi172.stdout: "wait": { 2024-03-28T22:48:32.210 INFO:teuthology.orchestra.run.smithi172.stdout: "avgcount": 0, 2024-03-28T22:48:32.210 INFO:teuthology.orchestra.run.smithi172.stdout: "sum": 0.000000000, 2024-03-28T22:48:32.210 INFO:teuthology.orchestra.run.smithi172.stdout: "avgtime": 0.000000000 2024-03-28T22:48:32.210 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:32.210 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-28T22:48:32.210 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-28T22:48:32.987 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-03-28T22:48:32.987 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-03-28T22:48:32.987 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update=0 2024-03-28T22:48:32.988 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update_receipt=0 2024-03-28T22:48:32.988 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=0 2024-03-28T22:48:32.988 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-03-28T22:48:32.988 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T22:48:32.988 ERROR:teuthology.run_tasks:Manager failed: check-counter Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph_55f5138942aa4866e49329c9103468889cec3f73/qa/tasks/check_counter.py", line 125, in end raise RuntimeError("The following counters failed to be set " RuntimeError: The following counters failed to be set on mds daemons: {'mds.imported', 'mds.exported'} 2024-03-28T22:48:32.999 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-28T22:48:33.010 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-28T22:48:33.010 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-28T22:48:33.010 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T22:48:33.076 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-28T22:48:33.076 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T22:48:33.132 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:48:33.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:32 smithi172 ceph-mon[36566]: pgmap v356: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:48:33.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:32 smithi082 ceph-mon[36471]: pgmap v356: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:48:33.221 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:32 smithi043 ceph-mon[31823]: pgmap v356: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:48:33.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.371+0000 7f6450200640 1 -- 172.21.15.43:0/556347556 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64481126f0 msgr2=0x7f6448114ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:33.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.371+0000 7f6450200640 1 --2- 172.21.15.43:0/556347556 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64481126f0 0x7f6448114ae0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f6440009ca0 tx=0x7f644002c180 comp rx=0 tx=0).stop 2024-03-28T22:48:33.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.371+0000 7f6450200640 1 -- 172.21.15.43:0/556347556 shutdown_connections 2024-03-28T22:48:33.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.371+0000 7f6450200640 1 --2- 172.21.15.43:0/556347556 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64481126f0 0x7f6448114ae0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:33.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.371+0000 7f6450200640 1 --2- 172.21.15.43:0/556347556 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6448074410 0x7f64481121b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:33.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.371+0000 7f6450200640 1 --2- 172.21.15.43:0/556347556 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6448073af0 0x7f6448073ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:33.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.371+0000 7f6450200640 1 -- 172.21.15.43:0/556347556 >> 172.21.15.43:0/556347556 conn(0x7f64480fd1f0 msgr2=0x7f64480ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:48:33.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.371+0000 7f6450200640 1 -- 172.21.15.43:0/556347556 shutdown_connections 2024-03-28T22:48:33.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.371+0000 7f6450200640 1 -- 172.21.15.43:0/556347556 wait complete. 2024-03-28T22:48:33.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.372+0000 7f6450200640 1 Processor -- start 2024-03-28T22:48:33.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.373+0000 7f6450200640 1 -- start start 2024-03-28T22:48:33.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.373+0000 7f6450200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6448073af0 0x7f644819e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:33.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.373+0000 7f6450200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6448074410 0x7f644819ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:33.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.373+0000 7f6450200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64481126f0 0x7f64481a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:33.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.373+0000 7f6450200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f64481180b0 con 0x7f6448073af0 2024-03-28T22:48:33.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.373+0000 7f6450200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f6448117f30 con 0x7f6448074410 2024-03-28T22:48:33.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.373+0000 7f6450200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f6448118230 con 0x7f64481126f0 2024-03-28T22:48:33.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.373+0000 7f644ce00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6448074410 0x7f644819ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:33.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.373+0000 7f644ce00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6448074410 0x7f644819ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:37062/0 (socket says 172.21.15.43:37062) 2024-03-28T22:48:33.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.373+0000 7f644ce00640 1 -- 172.21.15.43:0/1653921813 learned_addr learned my addr 172.21.15.43:0/1653921813 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:48:33.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.373+0000 7f644e200640 1 --2- 172.21.15.43:0/1653921813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64481126f0 0x7f64481a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:33.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.374+0000 7f644d800640 1 --2- 172.21.15.43:0/1653921813 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6448073af0 0x7f644819e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:33.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.374+0000 7f644ce00640 1 -- 172.21.15.43:0/1653921813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64481126f0 msgr2=0x7f64481a2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:33.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.374+0000 7f644ce00640 1 --2- 172.21.15.43:0/1653921813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64481126f0 0x7f64481a2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:33.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.374+0000 7f644ce00640 1 -- 172.21.15.43:0/1653921813 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6448073af0 msgr2=0x7f644819e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:33.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.374+0000 7f644ce00640 1 --2- 172.21.15.43:0/1653921813 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6448073af0 0x7f644819e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:33.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.374+0000 7f644ce00640 1 -- 172.21.15.43:0/1653921813 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64481a3680 con 0x7f6448074410 2024-03-28T22:48:33.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.374+0000 7f644e200640 1 --2- 172.21.15.43:0/1653921813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64481126f0 0x7f64481a2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:48:33.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.374+0000 7f644ce00640 1 --2- 172.21.15.43:0/1653921813 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6448074410 0x7f644819ebd0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f643400e9f0 tx=0x7f643400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:48:33.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.374+0000 7f6446000640 1 -- 172.21.15.43:0/1653921813 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f643400cd80 con 0x7f6448074410 2024-03-28T22:48:33.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.374+0000 7f6450200640 1 -- 172.21.15.43:0/1653921813 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64481a3970 con 0x7f6448074410 2024-03-28T22:48:33.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.374+0000 7f6446000640 1 -- 172.21.15.43:0/1653921813 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6434010910 con 0x7f6448074410 2024-03-28T22:48:33.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.374+0000 7f6446000640 1 -- 172.21.15.43:0/1653921813 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64340188f0 con 0x7f6448074410 2024-03-28T22:48:33.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.374+0000 7f6450200640 1 -- 172.21.15.43:0/1653921813 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64481ab200 con 0x7f6448074410 2024-03-28T22:48:33.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.376+0000 7f6450200640 1 -- 172.21.15.43:0/1653921813 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6448068990 con 0x7f6448074410 2024-03-28T22:48:33.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.378+0000 7f6446000640 1 -- 172.21.15.43:0/1653921813 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f6434010ac0 con 0x7f6448074410 2024-03-28T22:48:33.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.380+0000 7f6446000640 1 --2- 172.21.15.43:0/1653921813 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f642c0a1ef0 0x7f642c0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:33.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.380+0000 7f6446000640 1 -- 172.21.15.43:0/1653921813 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9539+0+0 (secure 0 0 0) 0x7f64340ceb70 con 0x7f6448074410 2024-03-28T22:48:33.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.380+0000 7f644d800640 1 --2- 172.21.15.43:0/1653921813 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f642c0a1ef0 0x7f642c0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:33.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.381+0000 7f644d800640 1 --2- 172.21.15.43:0/1653921813 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f642c0a1ef0 0x7f642c0a43b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6438009b60 tx=0x7f6438005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:48:33.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.385+0000 7f6446000640 1 -- 172.21.15.43:0/1653921813 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f643409b800 con 0x7f6448074410 2024-03-28T22:48:33.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.588+0000 7f6450200640 1 -- 172.21.15.43:0/1653921813 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f644819f9e0 con 0x7f6448074410 2024-03-28T22:48:33.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.590+0000 7f6446000640 1 -- 172.21.15.43:0/1653921813 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) ==== 74+0+25334 (secure 0 0 0) 0x7f644819f9e0 con 0x7f6448074410 2024-03-28T22:48:33.588 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:48:33.588 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":110,"fsid":"c2276428-ed51-11ee-95d0-87774f69a715","created":"2024-03-28T22:24:19.950084+0000","modified":"2024-03-28T22:48:17.858062+0000","last_up_change":"2024-03-28T22:32:31.078737+0000","last_in_change":"2024-03-28T22:32:10.147097+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:28:14.713290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:33:27.167435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:33:28.411825+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":109,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T22:33:32.523122+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":109,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b855adcd-786c-4e42-b8fe-a515a77ce979","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6803","nonce":2826124409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6805","nonce":2826124409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6809","nonce":2826124409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6807","nonce":2826124409}]},"public_addr":"172.21.15.43:6803/2826124409","cluster_addr":"172.21.15.43:6805/2826124409","heartbeat_back_addr":"172.21.15.43:6809/2826124409","heartbeat_front_addr":"172.21.15.43:6807/2826124409","state":["exists","up"]},{"osd":1,"uuid":"031c143d-8ad1-4b68-8db9-3c717137e3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6801","nonce":3043263644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6803","nonce":3043263644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6807","nonce":3043263644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6805","nonce":3043263644}]},"public_addr":"172.21.15.82:6801/3043263644","cluster_addr":"172.21.15.82:6803/3043263644","heartbeat_back_addr":"172.21.15.82:6807/3043263644","heartbeat_front_addr":"172.21.15.82:6805/3043263644","state":["exists","up"]},{"osd":2,"uuid":"20632bbb-f07a-4ca1-b0c4-ff9d508e211c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6801","nonce":4213953645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6803","nonce":4213953645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6807","nonce":4213953645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6805","nonce":4213953645}]},"public_addr":"172.21.15.172:6801/4213953645","cluster_addr":"172.21.15.172:6803/4213953645","heartbeat_back_addr":"172.21.15.172:6807/4213953645","heartbeat_front_addr":"172.21.15.172:6805/4213953645","state":["exists","up"]},{"osd":3,"uuid":"9a1f2312-4dbd-475b-8736-d5ebc8788e09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6811","nonce":1597280920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6813","nonce":1597280920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6817","nonce":1597280920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6815","nonce":1597280920}]},"public_addr":"172.21.15.43:6811/1597280920","cluster_addr":"172.21.15.43:6813/1597280920","heartbeat_back_addr":"172.21.15.43:6817/1597280920","heartbeat_front_addr":"172.21.15.43:6815/1597280920","state":["exists","up"]},{"osd":4,"uuid":"29140ead-7075-42b2-90c1-27cae0b29ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6809","nonce":215451895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6811","nonce":215451895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6815","nonce":215451895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6813","nonce":215451895}]},"public_addr":"172.21.15.82:6809/215451895","cluster_addr":"172.21.15.82:6811/215451895","heartbeat_back_addr":"172.21.15.82:6815/215451895","heartbeat_front_addr":"172.21.15.82:6813/215451895","state":["exists","up"]},{"osd":5,"uuid":"59644c61-93dc-4536-b2ad-0256b24e5cb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6809","nonce":1576148933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6811","nonce":1576148933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6815","nonce":1576148933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6813","nonce":1576148933}]},"public_addr":"172.21.15.172:6809/1576148933","cluster_addr":"172.21.15.172:6811/1576148933","heartbeat_back_addr":"172.21.15.172:6815/1576148933","heartbeat_front_addr":"172.21.15.172:6813/1576148933","state":["exists","up"]},{"osd":6,"uuid":"22d54e57-c3f1-4ecf-9921-8e815b48f92a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6819","nonce":791044279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6821","nonce":791044279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6825","nonce":791044279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6823","nonce":791044279}]},"public_addr":"172.21.15.43:6819/791044279","cluster_addr":"172.21.15.43:6821/791044279","heartbeat_back_addr":"172.21.15.43:6825/791044279","heartbeat_front_addr":"172.21.15.43:6823/791044279","state":["exists","up"]},{"osd":7,"uuid":"248cd1e2-0d6b-452f-a0a8-c51fe1b975ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6817","nonce":2157777323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6819","nonce":2157777323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6823","nonce":2157777323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6821","nonce":2157777323}]},"public_addr":"172.21.15.82:6817/2157777323","cluster_addr":"172.21.15.82:6819/2157777323","heartbeat_back_addr":"172.21.15.82:6823/2157777323","heartbeat_front_addr":"172.21.15.82:6821/2157777323","state":["exists","up"]},{"osd":8,"uuid":"d98aab6e-8bae-4c24-906c-4f492f047b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6817","nonce":1095328248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6819","nonce":1095328248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6823","nonce":1095328248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6821","nonce":1095328248}]},"public_addr":"172.21.15.172:6817/1095328248","cluster_addr":"172.21.15.172:6819/1095328248","heartbeat_back_addr":"172.21.15.172:6823/1095328248","heartbeat_front_addr":"172.21.15.172:6821/1095328248","state":["exists","up"]},{"osd":9,"uuid":"19aa383e-a0f6-45ea-96cc-e7956535caa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6827","nonce":3657791498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6829","nonce":3657791498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6833","nonce":3657791498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6831","nonce":3657791498}]},"public_addr":"172.21.15.43:6827/3657791498","cluster_addr":"172.21.15.43:6829/3657791498","heartbeat_back_addr":"172.21.15.43:6833/3657791498","heartbeat_front_addr":"172.21.15.43:6831/3657791498","state":["exists","up"]},{"osd":10,"uuid":"666977e5-eaf7-41b9-a18f-c380e149345d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6825","nonce":3454954047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6827","nonce":3454954047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6831","nonce":3454954047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6829","nonce":3454954047}]},"public_addr":"172.21.15.82:6825/3454954047","cluster_addr":"172.21.15.82:6827/3454954047","heartbeat_back_addr":"172.21.15.82:6831/3454954047","heartbeat_front_addr":"172.21.15.82:6829/3454954047","state":["exists","up"]},{"osd":11,"uuid":"615cf3be-8076-470e-ad09-c880e7593717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6825","nonce":2563517144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6827","nonce":2563517144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6831","nonce":2563517144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6829","nonce":2563517144}]},"public_addr":"172.21.15.172:6825/2563517144","cluster_addr":"172.21.15.172:6827/2563517144","heartbeat_back_addr":"172.21.15.172:6831/2563517144","heartbeat_front_addr":"172.21.15.172:6829/2563517144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:21.727405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:47.100259+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:10.615438+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:43.632144+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:09.487940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:34.604463+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:09.594452+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:35.709795+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:01.184202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:35.455322+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:02.413166+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:28.550754+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.9","mappings":[{"from":5,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/1567355374":"2024-03-29T22:37:11.308878+0000","172.21.15.43:6800/3517712438":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6801/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/686050369":"2024-03-29T22:37:11.308878+0000","172.21.15.43:0/2710117652":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/1375099638":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6800/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/3215634168":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6801/3517712438":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/4029313215":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/1343671266":"2024-03-29T22:37:11.308878+0000","172.21.15.43:0/578791143":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/4024174130":"2024-03-29T22:37:11.308878+0000","172.21.15.43:0/1574178482":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/141623093":"2024-03-29T22:37:11.308878+0000","172.21.15.43:6800/369798199":"2024-03-29T22:37:11.308878+0000","172.21.15.43:6801/369798199":"2024-03-29T22:37:11.308878+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:48:33.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.593+0000 7f6450200640 1 -- 172.21.15.43:0/1653921813 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f642c0a1ef0 msgr2=0x7f642c0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:33.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.593+0000 7f6450200640 1 --2- 172.21.15.43:0/1653921813 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f642c0a1ef0 0x7f642c0a43b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6438009b60 tx=0x7f6438005f90 comp rx=0 tx=0).stop 2024-03-28T22:48:33.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.593+0000 7f6450200640 1 -- 172.21.15.43:0/1653921813 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6448074410 msgr2=0x7f644819ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:33.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.593+0000 7f6450200640 1 --2- 172.21.15.43:0/1653921813 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6448074410 0x7f644819ebd0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f643400e9f0 tx=0x7f643400eeb0 comp rx=0 tx=0).stop 2024-03-28T22:48:33.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.594+0000 7f6450200640 1 -- 172.21.15.43:0/1653921813 shutdown_connections 2024-03-28T22:48:33.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.594+0000 7f6450200640 1 --2- 172.21.15.43:0/1653921813 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f642c0a1ef0 0x7f642c0a43b0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:33.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.594+0000 7f6450200640 1 --2- 172.21.15.43:0/1653921813 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6448074410 0x7f644819ebd0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:33.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.594+0000 7f6450200640 1 --2- 172.21.15.43:0/1653921813 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f64481126f0 0x7f64481a2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:33.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.594+0000 7f6450200640 1 --2- 172.21.15.43:0/1653921813 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6448073af0 0x7f644819e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:33.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.594+0000 7f6450200640 1 -- 172.21.15.43:0/1653921813 >> 172.21.15.43:0/1653921813 conn(0x7f64480fd1f0 msgr2=0x7f64480febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:48:33.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.594+0000 7f6450200640 1 -- 172.21.15.43:0/1653921813 shutdown_connections 2024-03-28T22:48:33.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.594+0000 7f6450200640 1 -- 172.21.15.43:0/1653921813 wait complete. 2024-03-28T22:48:33.612 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T22:28:14.713290+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T22:33:27.167435+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T22:33:28.411825+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 109, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T22:33:32.523122+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 109, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T22:48:33.612 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T22:48:33.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.838+0000 7f1ecb200640 1 -- 172.21.15.43:0/4154134793 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ec4107f00 msgr2=0x7f1ec41082e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:33.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.838+0000 7f1ecb200640 1 --2- 172.21.15.43:0/4154134793 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ec4107f00 0x7f1ec41082e0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8009900 tx=0x7f1eb802bc30 comp rx=0 tx=0).stop 2024-03-28T22:48:33.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.838+0000 7f1ecb200640 1 -- 172.21.15.43:0/4154134793 shutdown_connections 2024-03-28T22:48:33.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.838+0000 7f1ecb200640 1 --2- 172.21.15.43:0/4154134793 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1ec410cf50 0x7f1ec4114b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:33.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.838+0000 7f1ecb200640 1 --2- 172.21.15.43:0/4154134793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ec41088b0 0x7f1ec410c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:33.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.838+0000 7f1ecb200640 1 --2- 172.21.15.43:0/4154134793 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ec4107f00 0x7f1ec41082e0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:33.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.838+0000 7f1ecb200640 1 -- 172.21.15.43:0/4154134793 >> 172.21.15.43:0/4154134793 conn(0x7f1ec4077570 msgr2=0x7f1ec4106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:48:33.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.838+0000 7f1ecb200640 1 -- 172.21.15.43:0/4154134793 shutdown_connections 2024-03-28T22:48:33.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.838+0000 7f1ecb200640 1 -- 172.21.15.43:0/4154134793 wait complete. 2024-03-28T22:48:33.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.839+0000 7f1ecb200640 1 Processor -- start 2024-03-28T22:48:33.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.840+0000 7f1ecb200640 1 -- start start 2024-03-28T22:48:33.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.840+0000 7f1ecb200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1ec4107f00 0x7f1ec419e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:33.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.840+0000 7f1ecb200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ec410cf50 0x7f1ec419eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:33.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.840+0000 7f1ecb200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ec41a2de0 0x7f1ec41a3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:33.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.840+0000 7f1ecb200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec4118050 con 0x7f1ec41a2de0 2024-03-28T22:48:33.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.840+0000 7f1ecb200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec4117ed0 con 0x7f1ec4107f00 2024-03-28T22:48:33.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.840+0000 7f1ecb200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec41181d0 con 0x7f1ec410cf50 2024-03-28T22:48:33.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.840+0000 7f1ec9e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1ec4107f00 0x7f1ec419e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:33.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.840+0000 7f1ec9e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1ec4107f00 0x7f1ec419e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:37070/0 (socket says 172.21.15.43:37070) 2024-03-28T22:48:33.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.840+0000 7f1ec9e00640 1 -- 172.21.15.43:0/943919344 learned_addr learned my addr 172.21.15.43:0/943919344 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:48:33.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.840+0000 7f1eca800640 1 --2- 172.21.15.43:0/943919344 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ec41a2de0 0x7f1ec41a3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:33.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.840+0000 7f1ec9400640 1 --2- 172.21.15.43:0/943919344 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ec410cf50 0x7f1ec419eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:33.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.841+0000 7f1ec9e00640 1 -- 172.21.15.43:0/943919344 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ec410cf50 msgr2=0x7f1ec419eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:33.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.841+0000 7f1ec9e00640 1 --2- 172.21.15.43:0/943919344 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ec410cf50 0x7f1ec419eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:33.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.841+0000 7f1ec9e00640 1 -- 172.21.15.43:0/943919344 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ec41a2de0 msgr2=0x7f1ec41a3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:33.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.841+0000 7f1ec9e00640 1 --2- 172.21.15.43:0/943919344 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ec41a2de0 0x7f1ec41a3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:33.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.841+0000 7f1ec9e00640 1 -- 172.21.15.43:0/943919344 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ec41a38d0 con 0x7f1ec4107f00 2024-03-28T22:48:33.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.841+0000 7f1ec9e00640 1 --2- 172.21.15.43:0/943919344 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1ec4107f00 0x7f1ec419e570 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f1eb802c240 tx=0x7f1eb80042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:48:33.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.841+0000 7f1ebea00640 1 -- 172.21.15.43:0/943919344 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8043020 con 0x7f1ec4107f00 2024-03-28T22:48:33.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.841+0000 7f1ebea00640 1 -- 172.21.15.43:0/943919344 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1eb80054e0 con 0x7f1ec4107f00 2024-03-28T22:48:33.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.841+0000 7f1ecb200640 1 -- 172.21.15.43:0/943919344 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ec41a3b60 con 0x7f1ec4107f00 2024-03-28T22:48:33.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.841+0000 7f1ebea00640 1 -- 172.21.15.43:0/943919344 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8003b20 con 0x7f1ec4107f00 2024-03-28T22:48:33.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.842+0000 7f1ecb200640 1 -- 172.21.15.43:0/943919344 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ec41ab3b0 con 0x7f1ec4107f00 2024-03-28T22:48:33.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.843+0000 7f1ecb200640 1 -- 172.21.15.43:0/943919344 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ec40688b0 con 0x7f1ec4107f00 2024-03-28T22:48:33.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.844+0000 7f1ebea00640 1 -- 172.21.15.43:0/943919344 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f1eb8046020 con 0x7f1ec4107f00 2024-03-28T22:48:33.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.846+0000 7f1ebea00640 1 --2- 172.21.15.43:0/943919344 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1eac0a1f70 0x7f1eac0a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:33.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.846+0000 7f1ebea00640 1 -- 172.21.15.43:0/943919344 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9539+0+0 (secure 0 0 0) 0x7f1eb80f0430 con 0x7f1ec4107f00 2024-03-28T22:48:33.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.846+0000 7f1ec9400640 1 --2- 172.21.15.43:0/943919344 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1eac0a1f70 0x7f1eac0a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:33.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.847+0000 7f1ec9400640 1 --2- 172.21.15.43:0/943919344 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1eac0a1f70 0x7f1eac0a4430 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1ec40784e0 tx=0x7f1eb4005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:48:33.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:33.850+0000 7f1ebea00640 1 -- 172.21.15.43:0/943919344 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1eb804d020 con 0x7f1ec4107f00 2024-03-28T22:48:34.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.053+0000 7f1ecb200640 1 -- 172.21.15.43:0/943919344 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1ec4110150 con 0x7f1ec4107f00 2024-03-28T22:48:34.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.053+0000 7f1ebea00640 1 -- 172.21.15.43:0/943919344 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v110) ==== 93+0+10 (secure 0 0 0) 0x7f1eb80bd1d0 con 0x7f1ec4107f00 2024-03-28T22:48:34.051 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-03-28T22:48:34.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.056+0000 7f1ecb200640 1 -- 172.21.15.43:0/943919344 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1eac0a1f70 msgr2=0x7f1eac0a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:34.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.056+0000 7f1ecb200640 1 --2- 172.21.15.43:0/943919344 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1eac0a1f70 0x7f1eac0a4430 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1ec40784e0 tx=0x7f1eb4005f90 comp rx=0 tx=0).stop 2024-03-28T22:48:34.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.056+0000 7f1ecb200640 1 -- 172.21.15.43:0/943919344 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1ec4107f00 msgr2=0x7f1ec419e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:34.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.056+0000 7f1ecb200640 1 --2- 172.21.15.43:0/943919344 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1ec4107f00 0x7f1ec419e570 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f1eb802c240 tx=0x7f1eb80042f0 comp rx=0 tx=0).stop 2024-03-28T22:48:34.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.056+0000 7f1ecb200640 1 -- 172.21.15.43:0/943919344 shutdown_connections 2024-03-28T22:48:34.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.056+0000 7f1ecb200640 1 --2- 172.21.15.43:0/943919344 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f1eac0a1f70 0x7f1eac0a4430 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:34.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.056+0000 7f1ecb200640 1 --2- 172.21.15.43:0/943919344 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1ec41a2de0 0x7f1ec41a3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:34.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.056+0000 7f1ecb200640 1 --2- 172.21.15.43:0/943919344 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1ec410cf50 0x7f1ec419eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:34.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.056+0000 7f1ecb200640 1 --2- 172.21.15.43:0/943919344 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1ec4107f00 0x7f1ec419e570 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:34.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.056+0000 7f1ecb200640 1 -- 172.21.15.43:0/943919344 >> 172.21.15.43:0/943919344 conn(0x7f1ec4077570 msgr2=0x7f1ec4079130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:48:34.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.056+0000 7f1ecb200640 1 -- 172.21.15.43:0/943919344 shutdown_connections 2024-03-28T22:48:34.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.056+0000 7f1ecb200640 1 -- 172.21.15.43:0/943919344 wait complete. 2024-03-28T22:48:34.074 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T22:48:34.155 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:33 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1653921813' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:48:34.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:33 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1653921813' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:48:34.232 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:33 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1653921813' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:48:34.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.307+0000 7f23dd200640 1 -- 172.21.15.43:0/3461418989 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f23d8107f00 msgr2=0x7f23d81082e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:34.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.307+0000 7f23dd200640 1 --2- 172.21.15.43:0/3461418989 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f23d8107f00 0x7f23d81082e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f23c8009900 tx=0x7f23c802bc30 comp rx=0 tx=0).stop 2024-03-28T22:48:34.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.307+0000 7f23dd200640 1 -- 172.21.15.43:0/3461418989 shutdown_connections 2024-03-28T22:48:34.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.307+0000 7f23dd200640 1 --2- 172.21.15.43:0/3461418989 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f23d810cf50 0x7f23d8114b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:34.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.307+0000 7f23dd200640 1 --2- 172.21.15.43:0/3461418989 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23d81088b0 0x7f23d810c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:34.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.307+0000 7f23dd200640 1 --2- 172.21.15.43:0/3461418989 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f23d8107f00 0x7f23d81082e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:34.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.307+0000 7f23dd200640 1 -- 172.21.15.43:0/3461418989 >> 172.21.15.43:0/3461418989 conn(0x7f23d8077570 msgr2=0x7f23d8106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:48:34.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.307+0000 7f23dd200640 1 -- 172.21.15.43:0/3461418989 shutdown_connections 2024-03-28T22:48:34.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.307+0000 7f23dd200640 1 -- 172.21.15.43:0/3461418989 wait complete. 2024-03-28T22:48:34.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.308+0000 7f23dd200640 1 Processor -- start 2024-03-28T22:48:34.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.308+0000 7f23dd200640 1 -- start start 2024-03-28T22:48:34.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.308+0000 7f23dd200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f23d8107f00 0x7f23d819e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:34.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.308+0000 7f23dd200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f23d810cf50 0x7f23d819ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:34.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.308+0000 7f23dd200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23d81a2dc0 0x7f23d81a3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:34.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.308+0000 7f23dd200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f23d8118000 con 0x7f23d8107f00 2024-03-28T22:48:34.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.308+0000 7f23dd200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f23d8117e80 con 0x7f23d810cf50 2024-03-28T22:48:34.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.308+0000 7f23dd200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f23d8118180 con 0x7f23d81a2dc0 2024-03-28T22:48:34.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.309+0000 7f23d7400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f23d8107f00 0x7f23d819e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:34.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.309+0000 7f23d6a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f23d810cf50 0x7f23d819ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:34.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.309+0000 7f23d7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23d81a2dc0 0x7f23d81a3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:34.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.309+0000 7f23d6a00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f23d810cf50 0x7f23d819ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:37098/0 (socket says 172.21.15.43:37098) 2024-03-28T22:48:34.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.309+0000 7f23d6a00640 1 -- 172.21.15.43:0/4035761636 learned_addr learned my addr 172.21.15.43:0/4035761636 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:48:34.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.309+0000 7f23d6a00640 1 -- 172.21.15.43:0/4035761636 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23d81a2dc0 msgr2=0x7f23d81a3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:34.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.309+0000 7f23d6a00640 1 --2- 172.21.15.43:0/4035761636 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23d81a2dc0 0x7f23d81a3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:34.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.309+0000 7f23d6a00640 1 -- 172.21.15.43:0/4035761636 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f23d8107f00 msgr2=0x7f23d819e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:34.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.309+0000 7f23d6a00640 1 --2- 172.21.15.43:0/4035761636 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f23d8107f00 0x7f23d819e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:34.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.309+0000 7f23d6a00640 1 -- 172.21.15.43:0/4035761636 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23d81a38b0 con 0x7f23d810cf50 2024-03-28T22:48:34.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.309+0000 7f23d6a00640 1 --2- 172.21.15.43:0/4035761636 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f23d810cf50 0x7f23d819ea40 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f23c400e9f0 tx=0x7f23c400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:48:34.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.310+0000 7f23cfe00640 1 -- 172.21.15.43:0/4035761636 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23c400ccc0 con 0x7f23d810cf50 2024-03-28T22:48:34.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.310+0000 7f23cfe00640 1 -- 172.21.15.43:0/4035761636 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23c4010910 con 0x7f23d810cf50 2024-03-28T22:48:34.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.310+0000 7f23dd200640 1 -- 172.21.15.43:0/4035761636 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23d81a3ba0 con 0x7f23d810cf50 2024-03-28T22:48:34.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.310+0000 7f23cfe00640 1 -- 172.21.15.43:0/4035761636 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23c4018920 con 0x7f23d810cf50 2024-03-28T22:48:34.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.310+0000 7f23dd200640 1 -- 172.21.15.43:0/4035761636 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23d81ab390 con 0x7f23d810cf50 2024-03-28T22:48:34.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.311+0000 7f23dd200640 1 -- 172.21.15.43:0/4035761636 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23d80688b0 con 0x7f23d810cf50 2024-03-28T22:48:34.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.313+0000 7f23cfe00640 1 -- 172.21.15.43:0/4035761636 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f23c4010ac0 con 0x7f23d810cf50 2024-03-28T22:48:34.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.315+0000 7f23cfe00640 1 --2- 172.21.15.43:0/4035761636 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f23bc0a1ef0 0x7f23bc0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:34.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.315+0000 7f23cfe00640 1 -- 172.21.15.43:0/4035761636 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9539+0+0 (secure 0 0 0) 0x7f23c40ce8a0 con 0x7f23d810cf50 2024-03-28T22:48:34.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.315+0000 7f23d7400640 1 --2- 172.21.15.43:0/4035761636 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f23bc0a1ef0 0x7f23bc0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:34.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.316+0000 7f23d7400640 1 --2- 172.21.15.43:0/4035761636 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f23bc0a1ef0 0x7f23bc0a43b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f23c8006010 tx=0x7f23c8005dd0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:48:34.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.318+0000 7f23cfe00640 1 -- 172.21.15.43:0/4035761636 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f23c409aee0 con 0x7f23d810cf50 2024-03-28T22:48:34.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.522+0000 7f23dd200640 1 -- 172.21.15.43:0/4035761636 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f23d81088b0 con 0x7f23d810cf50 2024-03-28T22:48:34.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.523+0000 7f23cfe00640 1 -- 172.21.15.43:0/4035761636 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v110) ==== 104+0+11 (secure 0 0 0) 0x7f23c409ef80 con 0x7f23d810cf50 2024-03-28T22:48:34.521 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-03-28T22:48:34.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.525+0000 7f23dd200640 1 -- 172.21.15.43:0/4035761636 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f23bc0a1ef0 msgr2=0x7f23bc0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:34.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.525+0000 7f23dd200640 1 --2- 172.21.15.43:0/4035761636 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f23bc0a1ef0 0x7f23bc0a43b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f23c8006010 tx=0x7f23c8005dd0 comp rx=0 tx=0).stop 2024-03-28T22:48:34.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.525+0000 7f23dd200640 1 -- 172.21.15.43:0/4035761636 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f23d810cf50 msgr2=0x7f23d819ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:34.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.525+0000 7f23dd200640 1 --2- 172.21.15.43:0/4035761636 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f23d810cf50 0x7f23d819ea40 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f23c400e9f0 tx=0x7f23c400eeb0 comp rx=0 tx=0).stop 2024-03-28T22:48:34.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.526+0000 7f23dd200640 1 -- 172.21.15.43:0/4035761636 shutdown_connections 2024-03-28T22:48:34.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.526+0000 7f23dd200640 1 --2- 172.21.15.43:0/4035761636 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f23bc0a1ef0 0x7f23bc0a43b0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:34.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.526+0000 7f23dd200640 1 --2- 172.21.15.43:0/4035761636 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f23d810cf50 0x7f23d819ea40 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:34.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.526+0000 7f23dd200640 1 --2- 172.21.15.43:0/4035761636 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f23d81a2dc0 0x7f23d81a3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:34.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.526+0000 7f23dd200640 1 --2- 172.21.15.43:0/4035761636 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f23d8107f00 0x7f23d819e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:34.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.526+0000 7f23dd200640 1 -- 172.21.15.43:0/4035761636 >> 172.21.15.43:0/4035761636 conn(0x7f23d8077570 msgr2=0x7f23d8077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:48:34.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.526+0000 7f23dd200640 1 -- 172.21.15.43:0/4035761636 shutdown_connections 2024-03-28T22:48:34.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.526+0000 7f23dd200640 1 -- 172.21.15.43:0/4035761636 wait complete. 2024-03-28T22:48:34.543 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T22:48:34.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.774+0000 7fabce600640 1 -- 172.21.15.43:0/3962594819 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fabc810de80 msgr2=0x7fabc8110270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:34.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.774+0000 7fabce600640 1 --2- 172.21.15.43:0/3962594819 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fabc810de80 0x7fabc8110270 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fabc0009ca0 tx=0x7fabc002c180 comp rx=0 tx=0).stop 2024-03-28T22:48:34.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.774+0000 7fabce600640 1 -- 172.21.15.43:0/3962594819 shutdown_connections 2024-03-28T22:48:34.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.774+0000 7fabce600640 1 --2- 172.21.15.43:0/3962594819 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fabc810de80 0x7fabc8110270 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:34.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.774+0000 7fabce600640 1 --2- 172.21.15.43:0/3962594819 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fabc8074420 0x7fabc810d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:34.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.774+0000 7fabce600640 1 --2- 172.21.15.43:0/3962594819 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fabc8073b00 0x7fabc8073ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:34.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.774+0000 7fabce600640 1 -- 172.21.15.43:0/3962594819 >> 172.21.15.43:0/3962594819 conn(0x7fabc80fd0b0 msgr2=0x7fabc80ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:48:34.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.774+0000 7fabce600640 1 -- 172.21.15.43:0/3962594819 shutdown_connections 2024-03-28T22:48:34.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.774+0000 7fabce600640 1 -- 172.21.15.43:0/3962594819 wait complete. 2024-03-28T22:48:34.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.775+0000 7fabce600640 1 Processor -- start 2024-03-28T22:48:34.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.775+0000 7fabce600640 1 -- start start 2024-03-28T22:48:34.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.776+0000 7fabce600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fabc8073b00 0x7fabc819a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:34.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.776+0000 7fabce600640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fabc8074420 0x7fabc819a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:34.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.776+0000 7fabce600640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fabc810de80 0x7fabc819e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:34.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.776+0000 7fabce600640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fabc8113ba0 con 0x7fabc810de80 2024-03-28T22:48:34.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.776+0000 7fabce600640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fabc8113a20 con 0x7fabc8074420 2024-03-28T22:48:34.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.776+0000 7fabce600640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fabc8113d20 con 0x7fabc8073b00 2024-03-28T22:48:34.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.776+0000 7fabcdc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fabc810de80 0x7fabc819e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:34.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.776+0000 7fabc7e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fabc8074420 0x7fabc819a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:34.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.776+0000 7fabc7e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fabc8074420 0x7fabc819a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:37114/0 (socket says 172.21.15.43:37114) 2024-03-28T22:48:34.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.776+0000 7fabcd200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fabc8073b00 0x7fabc819a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:34.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.776+0000 7fabcdc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fabc810de80 0x7fabc819e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50308/0 (socket says 172.21.15.43:50308) 2024-03-28T22:48:34.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.776+0000 7fabc7e00640 1 -- 172.21.15.43:0/1346914264 learned_addr learned my addr 172.21.15.43:0/1346914264 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:48:34.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.777+0000 7fabc7e00640 1 -- 172.21.15.43:0/1346914264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fabc8073b00 msgr2=0x7fabc819a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:34.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.777+0000 7fabc7e00640 1 --2- 172.21.15.43:0/1346914264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fabc8073b00 0x7fabc819a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:34.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.777+0000 7fabc7e00640 1 -- 172.21.15.43:0/1346914264 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fabc810de80 msgr2=0x7fabc819e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:34.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.777+0000 7fabc7e00640 1 --2- 172.21.15.43:0/1346914264 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fabc810de80 0x7fabc819e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:34.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.777+0000 7fabc7e00640 1 -- 172.21.15.43:0/1346914264 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabc819f0a0 con 0x7fabc8074420 2024-03-28T22:48:34.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.777+0000 7fabcdc00640 1 --2- 172.21.15.43:0/1346914264 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fabc810de80 0x7fabc819e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:48:34.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.777+0000 7fabc7e00640 1 --2- 172.21.15.43:0/1346914264 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fabc8074420 0x7fabc819a640 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fabb400ea60 tx=0x7fabb400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:48:34.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.777+0000 7fabc5600640 1 -- 172.21.15.43:0/1346914264 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb400cde0 con 0x7fabc8074420 2024-03-28T22:48:34.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.777+0000 7fabce600640 1 -- 172.21.15.43:0/1346914264 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabc819f390 con 0x7fabc8074420 2024-03-28T22:48:34.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.777+0000 7fabc5600640 1 -- 172.21.15.43:0/1346914264 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabb40035f0 con 0x7fabc8074420 2024-03-28T22:48:34.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.777+0000 7fabc5600640 1 -- 172.21.15.43:0/1346914264 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb4010670 con 0x7fabc8074420 2024-03-28T22:48:34.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.777+0000 7fabce600640 1 -- 172.21.15.43:0/1346914264 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabc81a6c20 con 0x7fabc8074420 2024-03-28T22:48:34.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.779+0000 7fabce600640 1 -- 172.21.15.43:0/1346914264 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabc80688b0 con 0x7fabc8074420 2024-03-28T22:48:34.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.780+0000 7fabc5600640 1 -- 172.21.15.43:0/1346914264 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fabb40037a0 con 0x7fabc8074420 2024-03-28T22:48:34.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.782+0000 7fabc5600640 1 --2- 172.21.15.43:0/1346914264 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fabb00a1ef0 0x7fabb00a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:34.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.782+0000 7fabc5600640 1 -- 172.21.15.43:0/1346914264 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9539+0+0 (secure 0 0 0) 0x7fabb40ce860 con 0x7fabc8074420 2024-03-28T22:48:34.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.782+0000 7fabcd200640 1 --2- 172.21.15.43:0/1346914264 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fabb00a1ef0 0x7fabb00a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:34.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.783+0000 7fabcd200640 1 --2- 172.21.15.43:0/1346914264 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fabb00a1ef0 0x7fabb00a43b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fabbc00bbe0 tx=0x7fabbc00b390 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:48:34.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.790+0000 7fabc5600640 1 -- 172.21.15.43:0/1346914264 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fabb409b4f0 con 0x7fabc8074420 2024-03-28T22:48:34.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.995+0000 7fabce600640 1 -- 172.21.15.43:0/1346914264 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fabc8109000 con 0x7fabc8074420 2024-03-28T22:48:34.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.996+0000 7fabc5600640 1 -- 172.21.15.43:0/1346914264 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v110) ==== 100+0+11 (secure 0 0 0) 0x7fabb409adf0 con 0x7fabc8074420 2024-03-28T22:48:34.993 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 64 2024-03-28T22:48:34.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.998+0000 7fabce600640 1 -- 172.21.15.43:0/1346914264 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fabb00a1ef0 msgr2=0x7fabb00a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:34.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.998+0000 7fabce600640 1 --2- 172.21.15.43:0/1346914264 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fabb00a1ef0 0x7fabb00a43b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fabbc00bbe0 tx=0x7fabbc00b390 comp rx=0 tx=0).stop 2024-03-28T22:48:34.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.998+0000 7fabce600640 1 -- 172.21.15.43:0/1346914264 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fabc8074420 msgr2=0x7fabc819a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:34.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.998+0000 7fabce600640 1 --2- 172.21.15.43:0/1346914264 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fabc8074420 0x7fabc819a640 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fabb400ea60 tx=0x7fabb400ef20 comp rx=0 tx=0).stop 2024-03-28T22:48:34.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.998+0000 7fabce600640 1 -- 172.21.15.43:0/1346914264 shutdown_connections 2024-03-28T22:48:34.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.998+0000 7fabce600640 1 --2- 172.21.15.43:0/1346914264 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fabb00a1ef0 0x7fabb00a43b0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:34.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.999+0000 7fabce600640 1 --2- 172.21.15.43:0/1346914264 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fabc8074420 0x7fabc819a640 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:34.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.999+0000 7fabce600640 1 --2- 172.21.15.43:0/1346914264 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fabc810de80 0x7fabc819e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:34.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.999+0000 7fabce600640 1 --2- 172.21.15.43:0/1346914264 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fabc8073b00 0x7fabc819a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:34.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.999+0000 7fabce600640 1 -- 172.21.15.43:0/1346914264 >> 172.21.15.43:0/1346914264 conn(0x7fabc80fd0b0 msgr2=0x7fabc80fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:48:34.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.999+0000 7fabce600640 1 -- 172.21.15.43:0/1346914264 shutdown_connections 2024-03-28T22:48:34.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:34.999+0000 7fabce600640 1 -- 172.21.15.43:0/1346914264 wait complete. 2024-03-28T22:48:35.016 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T22:48:35.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:34 smithi082 ceph-mon[36471]: pgmap v357: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:48:35.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:34 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/943919344' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:48:35.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:34 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/4035761636' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:48:35.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.252+0000 7f8ae0800640 1 -- 172.21.15.43:0/1469710701 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8ad8068950 msgr2=0x7f8ad8068d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:35.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.252+0000 7f8ae0800640 1 --2- 172.21.15.43:0/1469710701 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8ad8068950 0x7f8ad8068d30 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0009900 tx=0x7f8ad002bcd0 comp rx=0 tx=0).stop 2024-03-28T22:48:35.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.252+0000 7f8ae0800640 1 -- 172.21.15.43:0/1469710701 shutdown_connections 2024-03-28T22:48:35.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.252+0000 7f8ae0800640 1 --2- 172.21.15.43:0/1469710701 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ad810e0f0 0x7f8ad81104e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:35.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.252+0000 7f8ae0800640 1 --2- 172.21.15.43:0/1469710701 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ad8069270 0x7f8ad810dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:35.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.252+0000 7f8ae0800640 1 --2- 172.21.15.43:0/1469710701 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8ad8068950 0x7f8ad8068d30 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:35.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.252+0000 7f8ae0800640 1 -- 172.21.15.43:0/1469710701 >> 172.21.15.43:0/1469710701 conn(0x7f8ad8077750 msgr2=0x7f8ad8077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:48:35.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.252+0000 7f8ae0800640 1 -- 172.21.15.43:0/1469710701 shutdown_connections 2024-03-28T22:48:35.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.252+0000 7f8ae0800640 1 -- 172.21.15.43:0/1469710701 wait complete. 2024-03-28T22:48:35.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.253+0000 7f8ae0800640 1 Processor -- start 2024-03-28T22:48:35.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.253+0000 7f8ae0800640 1 -- start start 2024-03-28T22:48:35.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.254+0000 7f8ae0800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ad8068950 0x7f8ad819a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:35.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.254+0000 7f8ae0800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ad8069270 0x7f8ad819a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:35.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.254+0000 7f8ae0800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8ad810e0f0 0x7f8ad819ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:35.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.254+0000 7f8ae0800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad8113cc0 con 0x7f8ad8068950 2024-03-28T22:48:35.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.254+0000 7f8ae0800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad8113b40 con 0x7f8ad810e0f0 2024-03-28T22:48:35.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.254+0000 7f8ae0800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad8113e40 con 0x7f8ad8069270 2024-03-28T22:48:35.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.254+0000 7f8adde00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ad8068950 0x7f8ad819a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:35.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.254+0000 7f8adde00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ad8068950 0x7f8ad819a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43830/0 (socket says 172.21.15.43:43830) 2024-03-28T22:48:35.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.254+0000 7f8adde00640 1 -- 172.21.15.43:0/510031651 learned_addr learned my addr 172.21.15.43:0/510031651 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:48:35.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.254+0000 7f8add400640 1 --2- 172.21.15.43:0/510031651 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ad8069270 0x7f8ad819a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:35.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.254+0000 7f8ade800640 1 --2- 172.21.15.43:0/510031651 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8ad810e0f0 0x7f8ad819ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:35.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.254+0000 7f8adde00640 1 -- 172.21.15.43:0/510031651 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ad8069270 msgr2=0x7f8ad819a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:35.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.254+0000 7f8adde00640 1 --2- 172.21.15.43:0/510031651 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ad8069270 0x7f8ad819a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:35.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.254+0000 7f8adde00640 1 -- 172.21.15.43:0/510031651 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8ad810e0f0 msgr2=0x7f8ad819ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:35.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.254+0000 7f8adde00640 1 --2- 172.21.15.43:0/510031651 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8ad810e0f0 0x7f8ad819ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:35.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.254+0000 7f8adde00640 1 -- 172.21.15.43:0/510031651 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ad819f130 con 0x7f8ad8068950 2024-03-28T22:48:35.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.255+0000 7f8adde00640 1 --2- 172.21.15.43:0/510031651 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ad8068950 0x7f8ad819a140 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0006010 tx=0x7f8ad002cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:48:35.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.255+0000 7f8acea00640 1 -- 172.21.15.43:0/510031651 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad0043020 con 0x7f8ad8068950 2024-03-28T22:48:35.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.255+0000 7f8acea00640 1 -- 172.21.15.43:0/510031651 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ad00054e0 con 0x7f8ad8068950 2024-03-28T22:48:35.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.255+0000 7f8ae0800640 1 -- 172.21.15.43:0/510031651 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ad819f3c0 con 0x7f8ad8068950 2024-03-28T22:48:35.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.255+0000 7f8acea00640 1 -- 172.21.15.43:0/510031651 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad0003c60 con 0x7f8ad8068950 2024-03-28T22:48:35.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.255+0000 7f8ae0800640 1 -- 172.21.15.43:0/510031651 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ad81a6cb0 con 0x7f8ad8068950 2024-03-28T22:48:35.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.257+0000 7f8ae0800640 1 -- 172.21.15.43:0/510031651 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ad8068540 con 0x7f8ad8068950 2024-03-28T22:48:35.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.259+0000 7f8acea00640 1 -- 172.21.15.43:0/510031651 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f8ad0046020 con 0x7f8ad8068950 2024-03-28T22:48:35.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.262+0000 7f8acea00640 1 --2- 172.21.15.43:0/510031651 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f8ac40a1ef0 0x7f8ac40a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:35.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.262+0000 7f8acea00640 1 -- 172.21.15.43:0/510031651 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9539+0+0 (secure 0 0 0) 0x7f8ad00f0bd0 con 0x7f8ad8068950 2024-03-28T22:48:35.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.262+0000 7f8add400640 1 --2- 172.21.15.43:0/510031651 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f8ac40a1ef0 0x7f8ac40a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:35.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.263+0000 7f8add400640 1 --2- 172.21.15.43:0/510031651 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f8ac40a1ef0 0x7f8ac40a43b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8100080 tx=0x7f8ac8005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:48:35.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.265+0000 7f8acea00640 1 -- 172.21.15.43:0/510031651 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8ad004d020 con 0x7f8ad8068950 2024-03-28T22:48:35.264 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:34 smithi043 ceph-mon[31823]: pgmap v357: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:48:35.264 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:34 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/943919344' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:48:35.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:34 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/4035761636' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:48:35.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:34 smithi172 ceph-mon[36566]: pgmap v357: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T22:48:35.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:34 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/943919344' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T22:48:35.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:34 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/4035761636' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T22:48:35.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.468+0000 7f8ae0800640 1 -- 172.21.15.43:0/510031651 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f8ad8109280 con 0x7f8ad8068950 2024-03-28T22:48:35.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.469+0000 7f8acea00640 1 -- 172.21.15.43:0/510031651 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v110) ==== 103+0+11 (secure 0 0 0) 0x7f8ad00bd970 con 0x7f8ad8068950 2024-03-28T22:48:35.466 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 32 2024-03-28T22:48:35.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.471+0000 7f8ae0800640 1 -- 172.21.15.43:0/510031651 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f8ac40a1ef0 msgr2=0x7f8ac40a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:35.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.471+0000 7f8ae0800640 1 --2- 172.21.15.43:0/510031651 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f8ac40a1ef0 0x7f8ac40a43b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8100080 tx=0x7f8ac8005f90 comp rx=0 tx=0).stop 2024-03-28T22:48:35.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.471+0000 7f8ae0800640 1 -- 172.21.15.43:0/510031651 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ad8068950 msgr2=0x7f8ad819a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:35.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.471+0000 7f8ae0800640 1 --2- 172.21.15.43:0/510031651 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ad8068950 0x7f8ad819a140 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0006010 tx=0x7f8ad002cbd0 comp rx=0 tx=0).stop 2024-03-28T22:48:35.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.472+0000 7f8ae0800640 1 -- 172.21.15.43:0/510031651 shutdown_connections 2024-03-28T22:48:35.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.472+0000 7f8ae0800640 1 --2- 172.21.15.43:0/510031651 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f8ac40a1ef0 0x7f8ac40a43b0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:35.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.472+0000 7f8ae0800640 1 --2- 172.21.15.43:0/510031651 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8ad810e0f0 0x7f8ad819ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:35.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.472+0000 7f8ae0800640 1 --2- 172.21.15.43:0/510031651 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8ad8069270 0x7f8ad819a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:35.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.472+0000 7f8ae0800640 1 --2- 172.21.15.43:0/510031651 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ad8068950 0x7f8ad819a140 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:35.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.472+0000 7f8ae0800640 1 -- 172.21.15.43:0/510031651 >> 172.21.15.43:0/510031651 conn(0x7f8ad8077750 msgr2=0x7f8ad810f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:48:35.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.472+0000 7f8ae0800640 1 -- 172.21.15.43:0/510031651 shutdown_connections 2024-03-28T22:48:35.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.472+0000 7f8ae0800640 1 -- 172.21.15.43:0/510031651 wait complete. 2024-03-28T22:48:35.488 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T22:48:35.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.720+0000 7feab7e00640 1 -- 172.21.15.43:0/3218852899 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feab81097e0 msgr2=0x7feab8109bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:35.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.720+0000 7feab7e00640 1 --2- 172.21.15.43:0/3218852899 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feab81097e0 0x7feab8109bc0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7feaa8009ca0 tx=0x7feaa802c100 comp rx=0 tx=0).stop 2024-03-28T22:48:35.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.720+0000 7feab7e00640 1 -- 172.21.15.43:0/3218852899 shutdown_connections 2024-03-28T22:48:35.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.720+0000 7feab7e00640 1 --2- 172.21.15.43:0/3218852899 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feab8104190 0x7feab81106a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:35.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.720+0000 7feab7e00640 1 --2- 172.21.15.43:0/3218852899 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feab81037d0 0x7feab8103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:35.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.720+0000 7feab7e00640 1 --2- 172.21.15.43:0/3218852899 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feab81097e0 0x7feab8109bc0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:35.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.720+0000 7feab7e00640 1 -- 172.21.15.43:0/3218852899 >> 172.21.15.43:0/3218852899 conn(0x7feab80ff260 msgr2=0x7feab8101680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:48:35.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.720+0000 7feab7e00640 1 -- 172.21.15.43:0/3218852899 shutdown_connections 2024-03-28T22:48:35.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.720+0000 7feab7e00640 1 -- 172.21.15.43:0/3218852899 wait complete. 2024-03-28T22:48:35.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.722+0000 7feab7e00640 1 Processor -- start 2024-03-28T22:48:35.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.722+0000 7feab7e00640 1 -- start start 2024-03-28T22:48:35.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.722+0000 7feab7e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feab81037d0 0x7feab819a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:35.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.722+0000 7feab7e00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feab8104190 0x7feab819a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:35.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.722+0000 7feab7e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feab81097e0 0x7feab819e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:35.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.722+0000 7feab7e00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7feab8113bb0 con 0x7feab81037d0 2024-03-28T22:48:35.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.722+0000 7feab7e00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7feab8113a30 con 0x7feab81097e0 2024-03-28T22:48:35.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.722+0000 7feab7e00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7feab8113d30 con 0x7feab8104190 2024-03-28T22:48:35.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.723+0000 7feaae000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feab8104190 0x7feab819a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:35.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.723+0000 7feaae000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feab8104190 0x7feab819a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:34932/0 (socket says 172.21.15.43:34932) 2024-03-28T22:48:35.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.723+0000 7feaae000640 1 -- 172.21.15.43:0/2772956581 learned_addr learned my addr 172.21.15.43:0/2772956581 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:48:35.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.723+0000 7feab7400640 1 --2- 172.21.15.43:0/2772956581 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feab81097e0 0x7feab819e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:35.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.723+0000 7feab6a00640 1 --2- 172.21.15.43:0/2772956581 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feab81037d0 0x7feab819a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:35.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.723+0000 7feaae000640 1 -- 172.21.15.43:0/2772956581 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feab81097e0 msgr2=0x7feab819e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:35.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.723+0000 7feaae000640 1 --2- 172.21.15.43:0/2772956581 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feab81097e0 0x7feab819e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:35.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.723+0000 7feaae000640 1 -- 172.21.15.43:0/2772956581 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feab81037d0 msgr2=0x7feab819a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:35.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.723+0000 7feaae000640 1 --2- 172.21.15.43:0/2772956581 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feab81037d0 0x7feab819a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:35.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.723+0000 7feaae000640 1 -- 172.21.15.43:0/2772956581 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feab819f0a0 con 0x7feab8104190 2024-03-28T22:48:35.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.723+0000 7feab7400640 1 --2- 172.21.15.43:0/2772956581 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feab81097e0 0x7feab819e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:48:35.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.723+0000 7feaae000640 1 --2- 172.21.15.43:0/2772956581 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feab8104190 0x7feab819a5f0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7feaa000d950 tx=0x7feaa000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:48:35.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.724+0000 7feaafe00640 1 -- 172.21.15.43:0/2772956581 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa0017070 con 0x7feab8104190 2024-03-28T22:48:35.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.724+0000 7feaafe00640 1 -- 172.21.15.43:0/2772956581 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feaa0009c30 con 0x7feab8104190 2024-03-28T22:48:35.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.724+0000 7feab7e00640 1 -- 172.21.15.43:0/2772956581 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feab819f390 con 0x7feab8104190 2024-03-28T22:48:35.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.724+0000 7feaafe00640 1 -- 172.21.15.43:0/2772956581 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa000f6b0 con 0x7feab8104190 2024-03-28T22:48:35.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.724+0000 7feab7e00640 1 -- 172.21.15.43:0/2772956581 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feab81a6c20 con 0x7feab8104190 2024-03-28T22:48:35.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.725+0000 7feab7e00640 1 -- 172.21.15.43:0/2772956581 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feab8068460 con 0x7feab8104190 2024-03-28T22:48:35.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.727+0000 7feaafe00640 1 -- 172.21.15.43:0/2772956581 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7feaa00109d0 con 0x7feab8104190 2024-03-28T22:48:35.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.728+0000 7feaafe00640 1 --2- 172.21.15.43:0/2772956581 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fea9c0a1ef0 0x7fea9c0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:35.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.729+0000 7feaafe00640 1 -- 172.21.15.43:0/2772956581 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9539+0+0 (secure 0 0 0) 0x7feaa00ce180 con 0x7feab8104190 2024-03-28T22:48:35.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.729+0000 7feab6a00640 1 --2- 172.21.15.43:0/2772956581 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fea9c0a1ef0 0x7fea9c0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:35.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.729+0000 7feab6a00640 1 --2- 172.21.15.43:0/2772956581 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fea9c0a1ef0 0x7fea9c0a43b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7feaa8000f80 tx=0x7feaa8009360 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:48:35.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.731+0000 7feaafe00640 1 -- 172.21.15.43:0/2772956581 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7feaa0009de0 con 0x7feab8104190 2024-03-28T22:48:35.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.948+0000 7feab7e00640 1 -- 172.21.15.43:0/2772956581 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7feab810bd70 con 0x7feab8104190 2024-03-28T22:48:35.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.949+0000 7feaafe00640 1 -- 172.21.15.43:0/2772956581 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 17 entries v110) ==== 81+0+999 (secure 0 0 0) 0x7feaa009a7c0 con 0x7feab8104190 2024-03-28T22:48:35.946 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/1567355374 2024-03-29T22:37:11.308878+0000 2024-03-28T22:48:35.946 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6800/3517712438 2024-03-29T22:25:28.008215+0000 2024-03-28T22:48:35.946 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6801/1532177494 2024-03-29T22:24:52.981057+0000 2024-03-28T22:48:35.947 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/686050369 2024-03-29T22:37:11.308878+0000 2024-03-28T22:48:35.947 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/2710117652 2024-03-29T22:24:52.981057+0000 2024-03-28T22:48:35.947 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/1375099638 2024-03-29T22:24:52.981057+0000 2024-03-28T22:48:35.947 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6800/1532177494 2024-03-29T22:24:52.981057+0000 2024-03-28T22:48:35.947 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/3215634168 2024-03-29T22:24:52.981057+0000 2024-03-28T22:48:35.947 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6801/3517712438 2024-03-29T22:25:28.008215+0000 2024-03-28T22:48:35.947 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/4029313215 2024-03-29T22:25:28.008215+0000 2024-03-28T22:48:35.947 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/1343671266 2024-03-29T22:37:11.308878+0000 2024-03-28T22:48:35.948 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/578791143 2024-03-29T22:25:28.008215+0000 2024-03-28T22:48:35.948 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/4024174130 2024-03-29T22:37:11.308878+0000 2024-03-28T22:48:35.948 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/1574178482 2024-03-29T22:25:28.008215+0000 2024-03-28T22:48:35.948 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:0/141623093 2024-03-29T22:37:11.308878+0000 2024-03-28T22:48:35.948 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6800/369798199 2024-03-29T22:37:11.308878+0000 2024-03-28T22:48:35.948 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43:6801/369798199 2024-03-29T22:37:11.308878+0000 2024-03-28T22:48:35.949 INFO:teuthology.orchestra.run.smithi043.stderr:listed 17 entries 2024-03-28T22:48:35.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.951+0000 7feab7e00640 1 -- 172.21.15.43:0/2772956581 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fea9c0a1ef0 msgr2=0x7fea9c0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:35.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.951+0000 7feab7e00640 1 --2- 172.21.15.43:0/2772956581 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fea9c0a1ef0 0x7fea9c0a43b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7feaa8000f80 tx=0x7feaa8009360 comp rx=0 tx=0).stop 2024-03-28T22:48:35.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.951+0000 7feab7e00640 1 -- 172.21.15.43:0/2772956581 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feab8104190 msgr2=0x7feab819a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:35.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.951+0000 7feab7e00640 1 --2- 172.21.15.43:0/2772956581 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feab8104190 0x7feab819a5f0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7feaa000d950 tx=0x7feaa000de10 comp rx=0 tx=0).stop 2024-03-28T22:48:35.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.952+0000 7feab7e00640 1 -- 172.21.15.43:0/2772956581 shutdown_connections 2024-03-28T22:48:35.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.952+0000 7feab7e00640 1 --2- 172.21.15.43:0/2772956581 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fea9c0a1ef0 0x7fea9c0a43b0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:35.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.952+0000 7feab7e00640 1 --2- 172.21.15.43:0/2772956581 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feab81097e0 0x7feab819e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:35.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.952+0000 7feab7e00640 1 --2- 172.21.15.43:0/2772956581 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7feab8104190 0x7feab819a5f0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:35.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.952+0000 7feab7e00640 1 --2- 172.21.15.43:0/2772956581 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7feab81037d0 0x7feab819a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:35.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.952+0000 7feab7e00640 1 -- 172.21.15.43:0/2772956581 >> 172.21.15.43:0/2772956581 conn(0x7feab80ff260 msgr2=0x7feab80ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:48:35.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.952+0000 7feab7e00640 1 -- 172.21.15.43:0/2772956581 shutdown_connections 2024-03-28T22:48:35.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:35.952+0000 7feab7e00640 1 -- 172.21.15.43:0/2772956581 wait complete. 2024-03-28T22:48:35.966 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-28T22:48:35.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:48:35.967 DEBUG:teuthology.orchestra.run.smithi043:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-28T22:48:36.103 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi043.front.sepia.ceph.com 2024-03-28T22:48:36.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:48:36.104 DEBUG:teuthology.orchestra.run.smithi043:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-28T22:48:36.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:48:36.167 DEBUG:teuthology.orchestra.run.smithi043:> ip netns list 2024-03-28T22:48:36.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:35 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1346914264' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:48:36.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:35 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/510031651' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:48:36.204 INFO:teuthology.orchestra.run.smithi043.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-28T22:48:36.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:48:36.206 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T22:48:36.264 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:35 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1346914264' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:48:36.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:35 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/510031651' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:48:36.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:48:36.329 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link delete ceph-brx 2024-03-28T22:48:36.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:35 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1346914264' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T22:48:36.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:35 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/510031651' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T22:48:36.498 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-28T22:48:36.498 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-03-28T22:48:36.498 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T22:48:36.542 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054203,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/fe518172546cef99a5a8c86e07cb3ceb900a53f271c97a4a51c5d72882e2d6f0/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/UVEFLZ2PQUY2B6EFRFROUNQAXO:/var/lib/containers/storage/overlay/l/WVIZQU7EWKGXBJCQ2GAGWRBJBJ,upperdir=/var/lib/containers/storage/overlay/fe518172546cef99a5a8c86e07cb3ceb900a53f271c97a4a51c5d72882e2d6f0/diff,workdir=/var/lib/containers/storage/overlay/fe518172546cef99a5a8c86e07cb3ceb900a53f271c97a4a51c5d72882e2d6f0/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/4687208e912e6d527d54f8260aac2b58a6a2162e92dbc3c8ce8cda99b750b040/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/UVEFLZ2PQUY2B6EFRFROUNQAXO:/var/lib/containers/storage/overlay/l/WVIZQU7EWKGXBJCQ2GAGWRBJBJ,upperdir=/var/lib/containers/storage/overlay/4687208e912e6d527d54f8260aac2b58a6a2162e92dbc3c8ce8cda99b750b040/diff,workdir=/var/lib/containers/storage/overlay/4687208e912e6d527d54f8260aac2b58a6a2162e92dbc3c8ce8cda99b750b040/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/22bbf09122ced5d269cff1cf1ffa641964e3219e05403855b4db1e9ea67591eb/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/UVEFLZ2PQUY2B6EFRFROUNQAXO:/var/lib/containers/storage/overlay/l/WVIZQU7EWKGXBJCQ2GAGWRBJBJ,upperdir=/var/lib/containers/storage/overlay/22bbf09122ced5d269cff1cf1ffa641964e3219e05403855b4db1e9ea67591eb/diff,workdir=/var/lib/containers/storage/overlay/22bbf09122ced5d269cff1cf1ffa641964e3219e05403855b4db1e9ea67591eb/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/f62092261c55900a04328b941b86965b516b5af10f2e9560e86752f9200655a1/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/UVEFLZ2PQUY2B6EFRFROUNQAXO:/var/lib/containers/storage/overlay/l/WVIZQU7EWKGXBJCQ2GAGWRBJBJ,upperdir=/var/lib/containers/storage/overlay/f62092261c55900a04328b941b86965b516b5af10f2e9560e86752f9200655a1/diff,workdir=/var/lib/containers/storage/overlay/f62092261c55900a04328b941b86965b516b5af10f2e9560e86752f9200655a1/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/894e112b4640db23e0e30e552fc4a3aca569b8139f45ac0f9b89d224ac08fdcf/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/UVEFLZ2PQUY2B6EFRFROUNQAXO:/var/lib/containers/storage/overlay/l/WVIZQU7EWKGXBJCQ2GAGWRBJBJ,upperdir=/var/lib/containers/storage/overlay/894e112b4640db23e0e30e552fc4a3aca569b8139f45ac0f9b89d224ac08fdcf/diff,workdir=/var/lib/containers/storage/overlay/894e112b4640db23e0e30e552fc4a3aca569b8139f45ac0f9b89d224ac08fdcf/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/ef5d0a506eadc7e15ad71dcbcd6726277852fbb09848d3bbf4f185a5104842d3/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/UVEFLZ2PQUY2B6EFRFROUNQAXO:/var/lib/containers/storage/overlay/l/WVIZQU7EWKGXBJCQ2GAGWRBJBJ,upperdir=/var/lib/containers/storage/overlay/ef5d0a506eadc7e15ad71dcbcd6726277852fbb09848d3bbf4f185a5104842d3/diff,workdir=/var/lib/containers/storage/overlay/ef5d0a506eadc7e15ad71dcbcd6726277852fbb09848d3bbf4f185a5104842d3/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/65b41b1bc08fadc010f9fdf317f491c611190b60b2c1c2cde81706d1ed91fd16/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/UVEFLZ2PQUY2B6EFRFROUNQAXO:/var/lib/containers/storage/overlay/l/WVIZQU7EWKGXBJCQ2GAGWRBJBJ,upperdir=/var/lib/containers/storage/overlay/65b41b1bc08fadc010f9fdf317f491c611190b60b2c1c2cde81706d1ed91fd16/diff,workdir=/var/lib/containers/storage/overlay/65b41b1bc08fadc010f9fdf317f491c611190b60b2c1c2cde81706d1ed91fd16/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/a92555d89a3757e9d56918cee272cadf7cf600bbf00b4ad0189819387f90aa22/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/UVEFLZ2PQUY2B6EFRFROUNQAXO:/var/lib/containers/storage/overlay/l/WVIZQU7EWKGXBJCQ2GAGWRBJBJ,upperdir=/var/lib/containers/storage/overlay/a92555d89a3757e9d56918cee272cadf7cf600bbf00b4ad0189819387f90aa22/diff,workdir=/var/lib/containers/storage/overlay/a92555d89a3757e9d56918cee272cadf7cf600bbf00b4ad0189819387f90aa22/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/d13069e2c5c6b6f7fda815bf0a9b03865dfbe7cb8329a0b1ef826f53588174dc/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/UVEFLZ2PQUY2B6EFRFROUNQAXO:/var/lib/containers/storage/overlay/l/WVIZQU7EWKGXBJCQ2GAGWRBJBJ,upperdir=/var/lib/containers/storage/overlay/d13069e2c5c6b6f7fda815bf0a9b03865dfbe7cb8329a0b1ef826f53588174dc/diff,workdir=/var/lib/containers/storage/overlay/d13069e2c5c6b6f7fda815bf0a9b03865dfbe7cb8329a0b1ef826f53588174dc/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/3c8bae2b92dd4fb04072b2be67971623901a5ba172ae2424b052427da3158989/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/UVEFLZ2PQUY2B6EFRFROUNQAXO:/var/lib/containers/storage/overlay/l/WVIZQU7EWKGXBJCQ2GAGWRBJBJ,upperdir=/var/lib/containers/storage/overlay/3c8bae2b92dd4fb04072b2be67971623901a5ba172ae2424b052427da3158989/diff,workdir=/var/lib/containers/storage/overlay/3c8bae2b92dd4fb04072b2be67971623901a5ba172ae2424b052427da3158989/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 2024-03-28T22:48:36.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:48:36.543 DEBUG:teuthology.orchestra.run.smithi043:> ip netns list 2024-03-28T22:48:36.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T22:48:36.638 DEBUG:teuthology.orchestra.run.smithi043:> sudo ip link delete ceph-brx 2024-03-28T22:48:36.728 INFO:teuthology.orchestra.run.smithi043.stderr:Cannot find device "ceph-brx" 2024-03-28T22:48:36.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T22:48:36.732 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-28T22:48:36.751 INFO:tasks.cephadm:Teardown begin 2024-03-28T22:48:36.751 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-28T22:48:36.751 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:48:36.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.986+0000 7f6a08800640 1 -- 172.21.15.43:0/1853763682 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a00075c80 msgr2=0x7f6a00076060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:36.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.986+0000 7f6a08800640 1 --2- 172.21.15.43:0/1853763682 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a00075c80 0x7f6a00076060 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f69f4009900 tx=0x7f69f402bc30 comp rx=0 tx=0).stop 2024-03-28T22:48:36.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.986+0000 7f6a08800640 1 -- 172.21.15.43:0/1853763682 shutdown_connections 2024-03-28T22:48:36.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.986+0000 7f6a08800640 1 --2- 172.21.15.43:0/1853763682 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a001125a0 0x7f6a00114990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:36.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.986+0000 7f6a08800640 1 --2- 172.21.15.43:0/1853763682 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a000765a0 0x7f6a00112060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:36.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.986+0000 7f6a08800640 1 --2- 172.21.15.43:0/1853763682 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a00075c80 0x7f6a00076060 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:36.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.986+0000 7f6a08800640 1 -- 172.21.15.43:0/1853763682 >> 172.21.15.43:0/1853763682 conn(0x7f6a000ff390 msgr2=0x7f6a001017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:48:36.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.986+0000 7f6a08800640 1 -- 172.21.15.43:0/1853763682 shutdown_connections 2024-03-28T22:48:36.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.986+0000 7f6a08800640 1 -- 172.21.15.43:0/1853763682 wait complete. 2024-03-28T22:48:36.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.987+0000 7f6a08800640 1 Processor -- start 2024-03-28T22:48:36.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.987+0000 7f6a08800640 1 -- start start 2024-03-28T22:48:36.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.987+0000 7f6a08800640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a00075c80 0x7f6a0019e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:36.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.987+0000 7f6a08800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a000765a0 0x7f6a0019eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:36.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.987+0000 7f6a08800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a001125a0 0x7f6a001a2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:36.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.987+0000 7f6a08800640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6a001180f0 con 0x7f6a00075c80 2024-03-28T22:48:36.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.987+0000 7f6a08800640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f6a00117f70 con 0x7f6a000765a0 2024-03-28T22:48:36.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.987+0000 7f6a08800640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f6a00118270 con 0x7f6a001125a0 2024-03-28T22:48:36.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.988+0000 7f6a05e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a00075c80 0x7f6a0019e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:36.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.988+0000 7f6a05400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a000765a0 0x7f6a0019eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:36.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.988+0000 7f6a05e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a00075c80 0x7f6a0019e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43872/0 (socket says 172.21.15.43:43872) 2024-03-28T22:48:36.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.988+0000 7f6a05e00640 1 -- 172.21.15.43:0/3554102394 learned_addr learned my addr 172.21.15.43:0/3554102394 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:48:36.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.988+0000 7f6a06800640 1 --2- 172.21.15.43:0/3554102394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a001125a0 0x7f6a001a2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:36.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.988+0000 7f6a05400640 1 -- 172.21.15.43:0/3554102394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a001125a0 msgr2=0x7f6a001a2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:36.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.988+0000 7f6a05400640 1 --2- 172.21.15.43:0/3554102394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a001125a0 0x7f6a001a2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:36.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.988+0000 7f6a05400640 1 -- 172.21.15.43:0/3554102394 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a00075c80 msgr2=0x7f6a0019e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:36.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.988+0000 7f6a05400640 1 --2- 172.21.15.43:0/3554102394 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a00075c80 0x7f6a0019e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:36.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.988+0000 7f6a05400640 1 -- 172.21.15.43:0/3554102394 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a001a35d0 con 0x7f6a000765a0 2024-03-28T22:48:36.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.989+0000 7f6a05400640 1 --2- 172.21.15.43:0/3554102394 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a000765a0 0x7f6a0019eb70 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f69f000e9f0 tx=0x7f69f000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:48:36.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.989+0000 7f69faa00640 1 -- 172.21.15.43:0/3554102394 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f000ccc0 con 0x7f6a000765a0 2024-03-28T22:48:36.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.989+0000 7f69faa00640 1 -- 172.21.15.43:0/3554102394 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69f0010910 con 0x7f6a000765a0 2024-03-28T22:48:36.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.989+0000 7f6a08800640 1 -- 172.21.15.43:0/3554102394 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a001a38c0 con 0x7f6a000765a0 2024-03-28T22:48:36.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.989+0000 7f69faa00640 1 -- 172.21.15.43:0/3554102394 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f0018960 con 0x7f6a000765a0 2024-03-28T22:48:36.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.989+0000 7f6a08800640 1 -- 172.21.15.43:0/3554102394 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a001ab150 con 0x7f6a000765a0 2024-03-28T22:48:36.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.990+0000 7f6a08800640 1 -- 172.21.15.43:0/3554102394 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a00068540 con 0x7f6a000765a0 2024-03-28T22:48:36.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.992+0000 7f69faa00640 1 -- 172.21.15.43:0/3554102394 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f69f0010ac0 con 0x7f6a000765a0 2024-03-28T22:48:36.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.994+0000 7f69faa00640 1 --2- 172.21.15.43:0/3554102394 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f69e80a1f70 0x7f69e80a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:36.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.994+0000 7f69faa00640 1 -- 172.21.15.43:0/3554102394 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9539+0+0 (secure 0 0 0) 0x7f69f00ce920 con 0x7f6a000765a0 2024-03-28T22:48:36.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.994+0000 7f6a05e00640 1 --2- 172.21.15.43:0/3554102394 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f69e80a1f70 0x7f69e80a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:36.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.995+0000 7f6a05e00640 1 --2- 172.21.15.43:0/3554102394 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f69e80a1f70 0x7f69e80a4430 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f69f4000c00 tx=0x7f69f40058d0 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:48:36.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:36.998+0000 7f69faa00640 1 -- 172.21.15.43:0/3554102394 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f69f009af60 con 0x7f6a000765a0 2024-03-28T22:48:37.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:36 smithi043 ceph-mon[31823]: pgmap v358: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:48:37.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:36 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2772956581' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T22:48:37.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:36 smithi082 ceph-mon[36471]: pgmap v358: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:48:37.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:36 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2772956581' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T22:48:37.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.272+0000 7f6a08800640 1 -- 172.21.15.43:0/3554102394 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6a0010d1a0 con 0x7f6a000765a0 2024-03-28T22:48:37.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.274+0000 7f69faa00640 1 -- 172.21.15.43:0/3554102394 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11931 (secure 0 0 0) 0x7f69f009f000 con 0x7f6a000765a0 2024-03-28T22:48:37.271 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:48:37.271 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:33:28.906396+0000","modified":"2024-03-28T22:40:08.672168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24572,"mds_1":24556,"mds_2":24535,"mds_3":24523,"mds_4":24518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26717":{"gid":26717,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/215779342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":215779342},{"type":"v1","addr":"172.21.15.82:6835","nonce":215779342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24535,24556,24518,24523,24572]},"id":1}]} 2024-03-28T22:48:37.272 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 32 2024-03-28T22:48:37.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.276+0000 7f6a08800640 1 -- 172.21.15.43:0/3554102394 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f69e80a1f70 msgr2=0x7f69e80a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:37.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.276+0000 7f6a08800640 1 --2- 172.21.15.43:0/3554102394 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f69e80a1f70 0x7f69e80a4430 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f69f4000c00 tx=0x7f69f40058d0 comp rx=0 tx=0).stop 2024-03-28T22:48:37.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.276+0000 7f6a08800640 1 -- 172.21.15.43:0/3554102394 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a000765a0 msgr2=0x7f6a0019eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:37.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.276+0000 7f6a08800640 1 --2- 172.21.15.43:0/3554102394 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a000765a0 0x7f6a0019eb70 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f69f000e9f0 tx=0x7f69f000eeb0 comp rx=0 tx=0).stop 2024-03-28T22:48:37.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.276+0000 7f6a08800640 1 -- 172.21.15.43:0/3554102394 shutdown_connections 2024-03-28T22:48:37.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.277+0000 7f6a08800640 1 --2- 172.21.15.43:0/3554102394 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f69e80a1f70 0x7f69e80a4430 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:37.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.277+0000 7f6a08800640 1 --2- 172.21.15.43:0/3554102394 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a000765a0 0x7f6a0019eb70 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:37.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.277+0000 7f6a08800640 1 --2- 172.21.15.43:0/3554102394 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a001125a0 0x7f6a001a2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:37.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.277+0000 7f6a08800640 1 --2- 172.21.15.43:0/3554102394 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a00075c80 0x7f6a0019e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:37.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.277+0000 7f6a08800640 1 -- 172.21.15.43:0/3554102394 >> 172.21.15.43:0/3554102394 conn(0x7f6a000ff390 msgr2=0x7f6a001140f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:48:37.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.277+0000 7f6a08800640 1 -- 172.21.15.43:0/3554102394 shutdown_connections 2024-03-28T22:48:37.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.277+0000 7f6a08800640 1 -- 172.21.15.43:0/3554102394 wait complete. 2024-03-28T22:48:37.293 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T22:48:37.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:36 smithi172 ceph-mon[36566]: pgmap v358: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T22:48:37.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:36 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2772956581' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T22:48:37.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.518+0000 7f7558000640 1 -- 172.21.15.43:0/4159204106 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7550109910 msgr2=0x7f7550109cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:37.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.518+0000 7f7558000640 1 --2- 172.21.15.43:0/4159204106 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7550109910 0x7f7550109cf0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f7544009900 tx=0x7f754402bc40 comp rx=0 tx=0).stop 2024-03-28T22:48:37.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.518+0000 7f7558000640 1 -- 172.21.15.43:0/4159204106 shutdown_connections 2024-03-28T22:48:37.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.518+0000 7f7558000640 1 --2- 172.21.15.43:0/4159204106 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f75501042c0 0x7f75501107d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:37.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.518+0000 7f7558000640 1 --2- 172.21.15.43:0/4159204106 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7550103900 0x7f7550103d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:37.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.518+0000 7f7558000640 1 --2- 172.21.15.43:0/4159204106 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7550109910 0x7f7550109cf0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:37.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.518+0000 7f7558000640 1 -- 172.21.15.43:0/4159204106 >> 172.21.15.43:0/4159204106 conn(0x7f75500ff390 msgr2=0x7f75501017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:48:37.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.518+0000 7f7558000640 1 -- 172.21.15.43:0/4159204106 shutdown_connections 2024-03-28T22:48:37.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.518+0000 7f7558000640 1 -- 172.21.15.43:0/4159204106 wait complete. 2024-03-28T22:48:37.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.520+0000 7f7558000640 1 Processor -- start 2024-03-28T22:48:37.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.520+0000 7f7558000640 1 -- start start 2024-03-28T22:48:37.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.520+0000 7f7558000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7550103900 0x7f755019a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:37.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.520+0000 7f7558000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f75501042c0 0x7f755019a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:37.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.520+0000 7f7558000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7550109910 0x7f755019ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:37.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.520+0000 7f7558000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7550113d70 con 0x7f7550109910 2024-03-28T22:48:37.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.520+0000 7f7558000640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7550113bf0 con 0x7f75501042c0 2024-03-28T22:48:37.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.520+0000 7f7558000640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f7550113ef0 con 0x7f7550103900 2024-03-28T22:48:37.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.521+0000 7f7555600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7550103900 0x7f755019a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:37.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.521+0000 7f7555600640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7550103900 0x7f755019a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:34972/0 (socket says 172.21.15.43:34972) 2024-03-28T22:48:37.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.521+0000 7f7555600640 1 -- 172.21.15.43:0/3316128260 learned_addr learned my addr 172.21.15.43:0/3316128260 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:48:37.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.521+0000 7f7554c00640 1 --2- 172.21.15.43:0/3316128260 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f75501042c0 0x7f755019a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:37.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.521+0000 7f7556000640 1 --2- 172.21.15.43:0/3316128260 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7550109910 0x7f755019ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:37.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.521+0000 7f7555600640 1 -- 172.21.15.43:0/3316128260 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f75501042c0 msgr2=0x7f755019a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:37.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.521+0000 7f7555600640 1 --2- 172.21.15.43:0/3316128260 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f75501042c0 0x7f755019a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:37.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.521+0000 7f7555600640 1 -- 172.21.15.43:0/3316128260 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7550109910 msgr2=0x7f755019ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:37.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.521+0000 7f7555600640 1 --2- 172.21.15.43:0/3316128260 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7550109910 0x7f755019ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:37.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.521+0000 7f7555600640 1 -- 172.21.15.43:0/3316128260 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f755019f200 con 0x7f7550103900 2024-03-28T22:48:37.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.521+0000 7f7554c00640 1 --2- 172.21.15.43:0/3316128260 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f75501042c0 0x7f755019a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:48:37.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.522+0000 7f7555600640 1 --2- 172.21.15.43:0/3316128260 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7550103900 0x7f755019a210 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f7544005ab0 tx=0x7f7544004430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:48:37.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.522+0000 7f754e000640 1 -- 172.21.15.43:0/3316128260 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7544041070 con 0x7f7550103900 2024-03-28T22:48:37.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.522+0000 7f754e000640 1 -- 172.21.15.43:0/3316128260 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f754402ccd0 con 0x7f7550103900 2024-03-28T22:48:37.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.522+0000 7f7558000640 1 -- 172.21.15.43:0/3316128260 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f755019f490 con 0x7f7550103900 2024-03-28T22:48:37.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.522+0000 7f754e000640 1 -- 172.21.15.43:0/3316128260 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7544003e20 con 0x7f7550103900 2024-03-28T22:48:37.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.522+0000 7f7558000640 1 -- 172.21.15.43:0/3316128260 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75501a6d80 con 0x7f7550103900 2024-03-28T22:48:37.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.523+0000 7f7558000640 1 -- 172.21.15.43:0/3316128260 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7550068540 con 0x7f7550103900 2024-03-28T22:48:37.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.525+0000 7f754e000640 1 -- 172.21.15.43:0/3316128260 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f7544044070 con 0x7f7550103900 2024-03-28T22:48:37.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.527+0000 7f754e000640 1 --2- 172.21.15.43:0/3316128260 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f75380a1ef0 0x7f75380a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:37.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.527+0000 7f754e000640 1 -- 172.21.15.43:0/3316128260 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9539+0+0 (secure 0 0 0) 0x7f75440effe0 con 0x7f7550103900 2024-03-28T22:48:37.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.527+0000 7f7554c00640 1 --2- 172.21.15.43:0/3316128260 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f75380a1ef0 0x7f75380a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:37.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.528+0000 7f7554c00640 1 --2- 172.21.15.43:0/3316128260 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f75380a1ef0 0x7f75380a43b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7550102a40 tx=0x7f753c00b390 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:48:37.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.531+0000 7f754e000640 1 -- 172.21.15.43:0/3316128260 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f75440bc770 con 0x7f7550103900 2024-03-28T22:48:37.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.774+0000 7f7558000640 1 -- 172.21.15.43:0/3316128260 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f755010be70 con 0x7f7550103900 2024-03-28T22:48:37.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.775+0000 7f754e000640 1 -- 172.21.15.43:0/3316128260 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11931 (secure 0 0 0) 0x7f75440c0810 con 0x7f7550103900 2024-03-28T22:48:37.773 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:48:37.773 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14451,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6839/2742974591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6838","nonce":2742974591},{"type":"v1","addr":"172.21.15.43:6839","nonce":2742974591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.43:6841/2371736231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6840","nonce":2371736231},{"type":"v1","addr":"172.21.15.43:6841","nonce":2371736231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T22:33:28.906396+0000","modified":"2024-03-28T22:40:08.672168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24572,"mds_1":24556,"mds_2":24535,"mds_3":24523,"mds_4":24518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6833/3363736995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3363736995},{"type":"v1","addr":"172.21.15.82:6833","nonce":3363736995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.172:6833/2579392697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":2579392697},{"type":"v1","addr":"172.21.15.172:6833","nonce":2579392697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6835/4195313768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6834","nonce":4195313768},{"type":"v1","addr":"172.21.15.43:6835","nonce":4195313768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"d","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.43:6837/1330639541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6836","nonce":1330639541},{"type":"v1","addr":"172.21.15.43:6837","nonce":1330639541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24518":{"gid":24518,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.82:6837/757432882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":757432882},{"type":"v1","addr":"172.21.15.82:6837","nonce":757432882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.172:6835/318854103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6834","nonce":318854103},{"type":"v1","addr":"172.21.15.172:6835","nonce":318854103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":16,"addr":"172.21.15.172:6837/3745415724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6836","nonce":3745415724},{"type":"v1","addr":"172.21.15.172:6837","nonce":3745415724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.172:6839/3592465067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6838","nonce":3592465067},{"type":"v1","addr":"172.21.15.172:6839","nonce":3592465067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2220892357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2220892357},{"type":"v1","addr":"172.21.15.82:6839","nonce":2220892357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26717":{"gid":26717,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/215779342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":215779342},{"type":"v1","addr":"172.21.15.82:6835","nonce":215779342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24572,"qdb_cluster":[24535,24556,24518,24523,24572]},"id":1}]} 2024-03-28T22:48:37.773 INFO:teuthology.orchestra.run.smithi043.stderr:dumped fsmap epoch 32 2024-03-28T22:48:37.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.778+0000 7f7558000640 1 -- 172.21.15.43:0/3316128260 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f75380a1ef0 msgr2=0x7f75380a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:37.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.778+0000 7f7558000640 1 --2- 172.21.15.43:0/3316128260 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f75380a1ef0 0x7f75380a43b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7550102a40 tx=0x7f753c00b390 comp rx=0 tx=0).stop 2024-03-28T22:48:37.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.778+0000 7f7558000640 1 -- 172.21.15.43:0/3316128260 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7550103900 msgr2=0x7f755019a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:37.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.778+0000 7f7558000640 1 --2- 172.21.15.43:0/3316128260 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7550103900 0x7f755019a210 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f7544005ab0 tx=0x7f7544004430 comp rx=0 tx=0).stop 2024-03-28T22:48:37.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.778+0000 7f7558000640 1 -- 172.21.15.43:0/3316128260 shutdown_connections 2024-03-28T22:48:37.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.778+0000 7f7558000640 1 --2- 172.21.15.43:0/3316128260 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f75380a1ef0 0x7f75380a43b0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:37.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.778+0000 7f7558000640 1 --2- 172.21.15.43:0/3316128260 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f75501042c0 0x7f755019a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:37.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.778+0000 7f7558000640 1 --2- 172.21.15.43:0/3316128260 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7550109910 0x7f755019ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:37.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.778+0000 7f7558000640 1 --2- 172.21.15.43:0/3316128260 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7550103900 0x7f755019a210 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:37.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.778+0000 7f7558000640 1 -- 172.21.15.43:0/3316128260 >> 172.21.15.43:0/3316128260 conn(0x7f75500ff390 msgr2=0x7f75500ffa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:48:37.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.778+0000 7f7558000640 1 -- 172.21.15.43:0/3316128260 shutdown_connections 2024-03-28T22:48:37.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:37.778+0000 7f7558000640 1 -- 172.21.15.43:0/3316128260 wait complete. 2024-03-28T22:48:37.798 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T22:48:38.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.023+0000 7fcf55c00640 1 -- 172.21.15.43:0/2150917636 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcf50109910 msgr2=0x7fcf50109cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:38.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.023+0000 7fcf55c00640 1 --2- 172.21.15.43:0/2150917636 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcf50109910 0x7fcf50109cf0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fcf40009900 tx=0x7fcf4002bc40 comp rx=0 tx=0).stop 2024-03-28T22:48:38.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.023+0000 7fcf55c00640 1 -- 172.21.15.43:0/2150917636 shutdown_connections 2024-03-28T22:48:38.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.023+0000 7fcf55c00640 1 --2- 172.21.15.43:0/2150917636 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf501042c0 0x7fcf501107d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:38.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.023+0000 7fcf55c00640 1 --2- 172.21.15.43:0/2150917636 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf50103900 0x7fcf50103d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:38.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.023+0000 7fcf55c00640 1 --2- 172.21.15.43:0/2150917636 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcf50109910 0x7fcf50109cf0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:38.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.023+0000 7fcf55c00640 1 -- 172.21.15.43:0/2150917636 >> 172.21.15.43:0/2150917636 conn(0x7fcf500ff390 msgr2=0x7fcf501017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:48:38.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.024+0000 7fcf55c00640 1 -- 172.21.15.43:0/2150917636 shutdown_connections 2024-03-28T22:48:38.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.024+0000 7fcf55c00640 1 -- 172.21.15.43:0/2150917636 wait complete. 2024-03-28T22:48:38.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.025+0000 7fcf55c00640 1 Processor -- start 2024-03-28T22:48:38.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.025+0000 7fcf55c00640 1 -- start start 2024-03-28T22:48:38.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.025+0000 7fcf55c00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf50103900 0x7fcf5019a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:38.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.025+0000 7fcf55c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf501042c0 0x7fcf5019a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:38.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.025+0000 7fcf55c00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcf50109910 0x7fcf5019ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:38.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.025+0000 7fcf55c00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fcf50113d70 con 0x7fcf501042c0 2024-03-28T22:48:38.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.025+0000 7fcf55c00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fcf50113bf0 con 0x7fcf50109910 2024-03-28T22:48:38.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.025+0000 7fcf55c00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7fcf50113ef0 con 0x7fcf50103900 2024-03-28T22:48:38.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.025+0000 7fcf4ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf50103900 0x7fcf5019a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:38.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.026+0000 7fcf4ea00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf50103900 0x7fcf5019a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.43:34992/0 (socket says 172.21.15.43:34992) 2024-03-28T22:48:38.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.026+0000 7fcf4ea00640 1 -- 172.21.15.43:0/1688533517 learned_addr learned my addr 172.21.15.43:0/1688533517 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:48:38.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.026+0000 7fcf4f400640 1 --2- 172.21.15.43:0/1688533517 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcf50109910 0x7fcf5019ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:38.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.026+0000 7fcf4e000640 1 --2- 172.21.15.43:0/1688533517 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf501042c0 0x7fcf5019a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:38.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.026+0000 7fcf4ea00640 1 -- 172.21.15.43:0/1688533517 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcf50109910 msgr2=0x7fcf5019ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:38.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.026+0000 7fcf4ea00640 1 --2- 172.21.15.43:0/1688533517 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcf50109910 0x7fcf5019ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:38.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.026+0000 7fcf4ea00640 1 -- 172.21.15.43:0/1688533517 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf501042c0 msgr2=0x7fcf5019a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:38.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.026+0000 7fcf4ea00640 1 --2- 172.21.15.43:0/1688533517 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf501042c0 0x7fcf5019a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:38.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.026+0000 7fcf4ea00640 1 -- 172.21.15.43:0/1688533517 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf5019f200 con 0x7fcf50103900 2024-03-28T22:48:38.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.026+0000 7fcf4ea00640 1 --2- 172.21.15.43:0/1688533517 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf50103900 0x7fcf5019a210 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fcf40005ab0 tx=0x7fcf400042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:48:38.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.027+0000 7fcf47400640 1 -- 172.21.15.43:0/1688533517 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf40041070 con 0x7fcf50103900 2024-03-28T22:48:38.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.027+0000 7fcf47400640 1 -- 172.21.15.43:0/1688533517 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf4002ccd0 con 0x7fcf50103900 2024-03-28T22:48:38.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.027+0000 7fcf55c00640 1 -- 172.21.15.43:0/1688533517 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf5019f490 con 0x7fcf50103900 2024-03-28T22:48:38.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.027+0000 7fcf47400640 1 -- 172.21.15.43:0/1688533517 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf40003d10 con 0x7fcf50103900 2024-03-28T22:48:38.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.027+0000 7fcf55c00640 1 -- 172.21.15.43:0/1688533517 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf501a6d80 con 0x7fcf50103900 2024-03-28T22:48:38.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.028+0000 7fcf55c00640 1 -- 172.21.15.43:0/1688533517 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf50068540 con 0x7fcf50103900 2024-03-28T22:48:38.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.030+0000 7fcf47400640 1 -- 172.21.15.43:0/1688533517 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7fcf40044070 con 0x7fcf50103900 2024-03-28T22:48:38.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.031+0000 7fcf47400640 1 --2- 172.21.15.43:0/1688533517 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcf340a1ef0 0x7fcf340a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:38.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.031+0000 7fcf47400640 1 -- 172.21.15.43:0/1688533517 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9539+0+0 (secure 0 0 0) 0x7fcf400efef0 con 0x7fcf50103900 2024-03-28T22:48:38.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.031+0000 7fcf4e000640 1 --2- 172.21.15.43:0/1688533517 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcf340a1ef0 0x7fcf340a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:38.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.032+0000 7fcf4e000640 1 --2- 172.21.15.43:0/1688533517 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcf340a1ef0 0x7fcf340a43b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fcf50102a40 tx=0x7fcf3c005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:48:38.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.039+0000 7fcf47400640 1 -- 172.21.15.43:0/1688533517 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fcf400bc640 con 0x7fcf50103900 2024-03-28T22:48:38.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:37 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3554102394' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:48:38.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:37 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3316128260' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:48:38.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.250+0000 7fcf55c00640 1 -- 172.21.15.43:0/1688533517 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcf5010be70 con 0x7fcf50103900 2024-03-28T22:48:38.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.253+0000 7fcf47400640 1 -- 172.21.15.43:0/1688533517 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) ==== 74+0+25334 (secure 0 0 0) 0x7fcf400b8020 con 0x7fcf50103900 2024-03-28T22:48:38.250 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-03-28T22:48:38.251 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":110,"fsid":"c2276428-ed51-11ee-95d0-87774f69a715","created":"2024-03-28T22:24:19.950084+0000","modified":"2024-03-28T22:48:17.858062+0000","last_up_change":"2024-03-28T22:32:31.078737+0000","last_in_change":"2024-03-28T22:32:10.147097+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T22:28:14.713290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T22:33:27.167435+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T22:33:28.411825+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":109,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T22:33:32.523122+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":109,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b855adcd-786c-4e42-b8fe-a515a77ce979","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6803","nonce":2826124409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6805","nonce":2826124409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6809","nonce":2826124409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":2826124409},{"type":"v1","addr":"172.21.15.43:6807","nonce":2826124409}]},"public_addr":"172.21.15.43:6803/2826124409","cluster_addr":"172.21.15.43:6805/2826124409","heartbeat_back_addr":"172.21.15.43:6809/2826124409","heartbeat_front_addr":"172.21.15.43:6807/2826124409","state":["exists","up"]},{"osd":1,"uuid":"031c143d-8ad1-4b68-8db9-3c717137e3e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6801","nonce":3043263644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6803","nonce":3043263644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6807","nonce":3043263644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3043263644},{"type":"v1","addr":"172.21.15.82:6805","nonce":3043263644}]},"public_addr":"172.21.15.82:6801/3043263644","cluster_addr":"172.21.15.82:6803/3043263644","heartbeat_back_addr":"172.21.15.82:6807/3043263644","heartbeat_front_addr":"172.21.15.82:6805/3043263644","state":["exists","up"]},{"osd":2,"uuid":"20632bbb-f07a-4ca1-b0c4-ff9d508e211c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6801","nonce":4213953645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6803","nonce":4213953645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6807","nonce":4213953645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":4213953645},{"type":"v1","addr":"172.21.15.172:6805","nonce":4213953645}]},"public_addr":"172.21.15.172:6801/4213953645","cluster_addr":"172.21.15.172:6803/4213953645","heartbeat_back_addr":"172.21.15.172:6807/4213953645","heartbeat_front_addr":"172.21.15.172:6805/4213953645","state":["exists","up"]},{"osd":3,"uuid":"9a1f2312-4dbd-475b-8736-d5ebc8788e09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6811","nonce":1597280920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6813","nonce":1597280920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6817","nonce":1597280920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":1597280920},{"type":"v1","addr":"172.21.15.43:6815","nonce":1597280920}]},"public_addr":"172.21.15.43:6811/1597280920","cluster_addr":"172.21.15.43:6813/1597280920","heartbeat_back_addr":"172.21.15.43:6817/1597280920","heartbeat_front_addr":"172.21.15.43:6815/1597280920","state":["exists","up"]},{"osd":4,"uuid":"29140ead-7075-42b2-90c1-27cae0b29ec1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6809","nonce":215451895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6811","nonce":215451895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6815","nonce":215451895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":215451895},{"type":"v1","addr":"172.21.15.82:6813","nonce":215451895}]},"public_addr":"172.21.15.82:6809/215451895","cluster_addr":"172.21.15.82:6811/215451895","heartbeat_back_addr":"172.21.15.82:6815/215451895","heartbeat_front_addr":"172.21.15.82:6813/215451895","state":["exists","up"]},{"osd":5,"uuid":"59644c61-93dc-4536-b2ad-0256b24e5cb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6809","nonce":1576148933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6811","nonce":1576148933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6815","nonce":1576148933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1576148933},{"type":"v1","addr":"172.21.15.172:6813","nonce":1576148933}]},"public_addr":"172.21.15.172:6809/1576148933","cluster_addr":"172.21.15.172:6811/1576148933","heartbeat_back_addr":"172.21.15.172:6815/1576148933","heartbeat_front_addr":"172.21.15.172:6813/1576148933","state":["exists","up"]},{"osd":6,"uuid":"22d54e57-c3f1-4ecf-9921-8e815b48f92a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6819","nonce":791044279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6821","nonce":791044279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6825","nonce":791044279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":791044279},{"type":"v1","addr":"172.21.15.43:6823","nonce":791044279}]},"public_addr":"172.21.15.43:6819/791044279","cluster_addr":"172.21.15.43:6821/791044279","heartbeat_back_addr":"172.21.15.43:6825/791044279","heartbeat_front_addr":"172.21.15.43:6823/791044279","state":["exists","up"]},{"osd":7,"uuid":"248cd1e2-0d6b-452f-a0a8-c51fe1b975ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6817","nonce":2157777323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6819","nonce":2157777323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6823","nonce":2157777323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2157777323},{"type":"v1","addr":"172.21.15.82:6821","nonce":2157777323}]},"public_addr":"172.21.15.82:6817/2157777323","cluster_addr":"172.21.15.82:6819/2157777323","heartbeat_back_addr":"172.21.15.82:6823/2157777323","heartbeat_front_addr":"172.21.15.82:6821/2157777323","state":["exists","up"]},{"osd":8,"uuid":"d98aab6e-8bae-4c24-906c-4f492f047b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6817","nonce":1095328248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6819","nonce":1095328248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6823","nonce":1095328248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1095328248},{"type":"v1","addr":"172.21.15.172:6821","nonce":1095328248}]},"public_addr":"172.21.15.172:6817/1095328248","cluster_addr":"172.21.15.172:6819/1095328248","heartbeat_back_addr":"172.21.15.172:6823/1095328248","heartbeat_front_addr":"172.21.15.172:6821/1095328248","state":["exists","up"]},{"osd":9,"uuid":"19aa383e-a0f6-45ea-96cc-e7956535caa6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6827","nonce":3657791498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6829","nonce":3657791498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6833","nonce":3657791498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":3657791498},{"type":"v1","addr":"172.21.15.43:6831","nonce":3657791498}]},"public_addr":"172.21.15.43:6827/3657791498","cluster_addr":"172.21.15.43:6829/3657791498","heartbeat_back_addr":"172.21.15.43:6833/3657791498","heartbeat_front_addr":"172.21.15.43:6831/3657791498","state":["exists","up"]},{"osd":10,"uuid":"666977e5-eaf7-41b9-a18f-c380e149345d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6825","nonce":3454954047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6827","nonce":3454954047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6831","nonce":3454954047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3454954047},{"type":"v1","addr":"172.21.15.82:6829","nonce":3454954047}]},"public_addr":"172.21.15.82:6825/3454954047","cluster_addr":"172.21.15.82:6827/3454954047","heartbeat_back_addr":"172.21.15.82:6831/3454954047","heartbeat_front_addr":"172.21.15.82:6829/3454954047","state":["exists","up"]},{"osd":11,"uuid":"615cf3be-8076-470e-ad09-c880e7593717","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6825","nonce":2563517144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6827","nonce":2563517144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6831","nonce":2563517144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":2563517144},{"type":"v1","addr":"172.21.15.172:6829","nonce":2563517144}]},"public_addr":"172.21.15.172:6825/2563517144","cluster_addr":"172.21.15.172:6827/2563517144","heartbeat_back_addr":"172.21.15.172:6831/2563517144","heartbeat_front_addr":"172.21.15.172:6829/2563517144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:21.727405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:27:47.100259+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:10.615438+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:28:43.632144+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:09.487940+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:29:34.604463+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:09.594452+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:30:35.709795+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:01.184202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:31:35.455322+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:02.413166+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T22:32:28.550754+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.9","mappings":[{"from":5,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/1567355374":"2024-03-29T22:37:11.308878+0000","172.21.15.43:6800/3517712438":"2024-03-29T22:25:28.008215+0000","172.21.15.43:6801/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/686050369":"2024-03-29T22:37:11.308878+0000","172.21.15.43:0/2710117652":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/1375099638":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6800/1532177494":"2024-03-29T22:24:52.981057+0000","172.21.15.43:0/3215634168":"2024-03-29T22:24:52.981057+0000","172.21.15.43:6801/3517712438":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/4029313215":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/1343671266":"2024-03-29T22:37:11.308878+0000","172.21.15.43:0/578791143":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/4024174130":"2024-03-29T22:37:11.308878+0000","172.21.15.43:0/1574178482":"2024-03-29T22:25:28.008215+0000","172.21.15.43:0/141623093":"2024-03-29T22:37:11.308878+0000","172.21.15.43:6800/369798199":"2024-03-29T22:37:11.308878+0000","172.21.15.43:6801/369798199":"2024-03-29T22:37:11.308878+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T22:48:38.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.256+0000 7fcf55c00640 1 -- 172.21.15.43:0/1688533517 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcf340a1ef0 msgr2=0x7fcf340a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:38.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.256+0000 7fcf55c00640 1 --2- 172.21.15.43:0/1688533517 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcf340a1ef0 0x7fcf340a43b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fcf50102a40 tx=0x7fcf3c005f90 comp rx=0 tx=0).stop 2024-03-28T22:48:38.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.256+0000 7fcf55c00640 1 -- 172.21.15.43:0/1688533517 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf50103900 msgr2=0x7fcf5019a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:38.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.256+0000 7fcf55c00640 1 --2- 172.21.15.43:0/1688533517 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf50103900 0x7fcf5019a210 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fcf40005ab0 tx=0x7fcf400042f0 comp rx=0 tx=0).stop 2024-03-28T22:48:38.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.256+0000 7fcf55c00640 1 -- 172.21.15.43:0/1688533517 shutdown_connections 2024-03-28T22:48:38.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.256+0000 7fcf55c00640 1 --2- 172.21.15.43:0/1688533517 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7fcf340a1ef0 0x7fcf340a43b0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:38.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.256+0000 7fcf55c00640 1 --2- 172.21.15.43:0/1688533517 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcf50109910 0x7fcf5019ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:38.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.256+0000 7fcf55c00640 1 --2- 172.21.15.43:0/1688533517 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fcf501042c0 0x7fcf5019a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:38.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.256+0000 7fcf55c00640 1 --2- 172.21.15.43:0/1688533517 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fcf50103900 0x7fcf5019a210 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:38.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.256+0000 7fcf55c00640 1 -- 172.21.15.43:0/1688533517 >> 172.21.15.43:0/1688533517 conn(0x7fcf500ff390 msgr2=0x7fcf500ffa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:48:38.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.256+0000 7fcf55c00640 1 -- 172.21.15.43:0/1688533517 shutdown_connections 2024-03-28T22:48:38.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.256+0000 7fcf55c00640 1 -- 172.21.15.43:0/1688533517 wait complete. 2024-03-28T22:48:38.264 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:37 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3554102394' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:48:38.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:37 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3316128260' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:48:38.276 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-28T22:48:38.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:37 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3554102394' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:48:38.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:37 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3316128260' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T22:48:38.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.492+0000 7f3ce3200640 1 -- 172.21.15.43:0/1843481132 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3ce4100570 msgr2=0x7f3ce4100950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:38.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.492+0000 7f3ce3200640 1 --2- 172.21.15.43:0/1843481132 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3ce4100570 0x7f3ce4100950 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f3cd4009900 tx=0x7f3cd402bcd0 comp rx=0 tx=0).stop 2024-03-28T22:48:38.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.493+0000 7f3ce3200640 1 -- 172.21.15.43:0/1843481132 shutdown_connections 2024-03-28T22:48:38.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.493+0000 7f3ce3200640 1 --2- 172.21.15.43:0/1843481132 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ce4105970 0x7f3ce4105e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:38.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.493+0000 7f3ce3200640 1 --2- 172.21.15.43:0/1843481132 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ce4100e90 0x7f3ce4105350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:38.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.493+0000 7f3ce3200640 1 --2- 172.21.15.43:0/1843481132 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3ce4100570 0x7f3ce4100950 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:38.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.493+0000 7f3ce3200640 1 -- 172.21.15.43:0/1843481132 >> 172.21.15.43:0/1843481132 conn(0x7f3ce40fbfa0 msgr2=0x7f3ce40fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:48:38.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.493+0000 7f3ce3200640 1 -- 172.21.15.43:0/1843481132 shutdown_connections 2024-03-28T22:48:38.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.493+0000 7f3ce3200640 1 -- 172.21.15.43:0/1843481132 wait complete. 2024-03-28T22:48:38.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.494+0000 7f3ce3200640 1 Processor -- start 2024-03-28T22:48:38.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.494+0000 7f3ce3200640 1 -- start start 2024-03-28T22:48:38.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.495+0000 7f3ce3200640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ce4100570 0x7f3ce419e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:38.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.495+0000 7f3ce3200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3ce4100e90 0x7f3ce419ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:38.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.495+0000 7f3ce3200640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ce4105970 0x7f3ce41a2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:38.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.495+0000 7f3ce3200640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce4117f40 con 0x7f3ce4100570 2024-03-28T22:48:38.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.495+0000 7f3ce3200640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce4117dc0 con 0x7f3ce4100e90 2024-03-28T22:48:38.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.495+0000 7f3ce3200640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce41180c0 con 0x7f3ce4105970 2024-03-28T22:48:38.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.495+0000 7f3cdbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3ce4100e90 0x7f3ce419ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:38.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.495+0000 7f3cdbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3ce4100e90 0x7f3ce419ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52406/0 (socket says 172.21.15.43:52406) 2024-03-28T22:48:38.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.495+0000 7f3ce2800640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ce4105970 0x7f3ce41a2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:38.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.495+0000 7f3cdbe00640 1 -- 172.21.15.43:0/416856501 learned_addr learned my addr 172.21.15.43:0/416856501 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:48:38.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.495+0000 7f3ce1e00640 1 --2- 172.21.15.43:0/416856501 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ce4100570 0x7f3ce419e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:38.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.496+0000 7f3cdbe00640 1 -- 172.21.15.43:0/416856501 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ce4105970 msgr2=0x7f3ce41a2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:38.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.496+0000 7f3cdbe00640 1 --2- 172.21.15.43:0/416856501 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ce4105970 0x7f3ce41a2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:38.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.496+0000 7f3cdbe00640 1 -- 172.21.15.43:0/416856501 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ce4100570 msgr2=0x7f3ce419e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:38.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.496+0000 7f3cdbe00640 1 --2- 172.21.15.43:0/416856501 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ce4100570 0x7f3ce419e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:38.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.496+0000 7f3cdbe00640 1 -- 172.21.15.43:0/416856501 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce41a34b0 con 0x7f3ce4100e90 2024-03-28T22:48:38.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.496+0000 7f3ce2800640 1 --2- 172.21.15.43:0/416856501 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ce4105970 0x7f3ce41a2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:48:38.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.496+0000 7f3cdbe00640 1 --2- 172.21.15.43:0/416856501 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3ce4100e90 0x7f3ce419ea00 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc00d950 tx=0x7f3ccc00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:48:38.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.496+0000 7f3cdaa00640 1 -- 172.21.15.43:0/416856501 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc017030 con 0x7f3ce4100e90 2024-03-28T22:48:38.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.496+0000 7f3cdaa00640 1 -- 172.21.15.43:0/416856501 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ccc004020 con 0x7f3ce4100e90 2024-03-28T22:48:38.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.496+0000 7f3cdaa00640 1 -- 172.21.15.43:0/416856501 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc0047a0 con 0x7f3ce4100e90 2024-03-28T22:48:38.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.496+0000 7f3ce3200640 1 -- 172.21.15.43:0/416856501 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ce41a37a0 con 0x7f3ce4100e90 2024-03-28T22:48:38.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.496+0000 7f3ce3200640 1 -- 172.21.15.43:0/416856501 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ce41ab030 con 0x7f3ce4100e90 2024-03-28T22:48:38.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.497+0000 7f3ce3200640 1 -- 172.21.15.43:0/416856501 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ce4068460 con 0x7f3ce4100e90 2024-03-28T22:48:38.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.499+0000 7f3cdaa00640 1 -- 172.21.15.43:0/416856501 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f3ccc0041d0 con 0x7f3ce4100e90 2024-03-28T22:48:38.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.501+0000 7f3cdaa00640 1 --2- 172.21.15.43:0/416856501 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f3cc80a1ef0 0x7f3cc80a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:38.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.501+0000 7f3cdaa00640 1 -- 172.21.15.43:0/416856501 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9539+0+0 (secure 0 0 0) 0x7f3ccc0cdf00 con 0x7f3ce4100e90 2024-03-28T22:48:38.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.501+0000 7f3ce1e00640 1 --2- 172.21.15.43:0/416856501 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f3cc80a1ef0 0x7f3cc80a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:38.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.502+0000 7f3ce1e00640 1 --2- 172.21.15.43:0/416856501 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f3cc80a1ef0 0x7f3cc80a43b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3cd4006010 tx=0x7f3cd4005a20 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:48:38.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.509+0000 7f3cdaa00640 1 -- 172.21.15.43:0/416856501 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3ccc09ab90 con 0x7f3ce4100e90 2024-03-28T22:48:38.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.752+0000 7f3ce3200640 1 -- 172.21.15.43:0/416856501 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f3ce41ab260 con 0x7f3ce4100e90 2024-03-28T22:48:38.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.760+0000 7f3cdaa00640 1 -- 172.21.15.43:0/416856501 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v33) ==== 161+0+0 (secure 0 0 0) 0x7f3ccc09a490 con 0x7f3ce4100e90 2024-03-28T22:48:38.757 INFO:teuthology.orchestra.run.smithi043.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-28T22:48:38.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.762+0000 7f3ce3200640 1 -- 172.21.15.43:0/416856501 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f3cc80a1ef0 msgr2=0x7f3cc80a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:38.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.762+0000 7f3ce3200640 1 --2- 172.21.15.43:0/416856501 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f3cc80a1ef0 0x7f3cc80a43b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3cd4006010 tx=0x7f3cd4005a20 comp rx=0 tx=0).stop 2024-03-28T22:48:38.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.763+0000 7f3ce3200640 1 -- 172.21.15.43:0/416856501 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3ce4100e90 msgr2=0x7f3ce419ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:38.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.763+0000 7f3ce3200640 1 --2- 172.21.15.43:0/416856501 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3ce4100e90 0x7f3ce419ea00 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc00d950 tx=0x7f3ccc00de10 comp rx=0 tx=0).stop 2024-03-28T22:48:38.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.763+0000 7f3ce3200640 1 -- 172.21.15.43:0/416856501 shutdown_connections 2024-03-28T22:48:38.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.763+0000 7f3ce3200640 1 --2- 172.21.15.43:0/416856501 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f3cc80a1ef0 0x7f3cc80a43b0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:38.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.763+0000 7f3ce3200640 1 --2- 172.21.15.43:0/416856501 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3ce4100e90 0x7f3ce419ea00 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:38.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.763+0000 7f3ce3200640 1 --2- 172.21.15.43:0/416856501 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3ce4105970 0x7f3ce41a2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:38.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.763+0000 7f3ce3200640 1 --2- 172.21.15.43:0/416856501 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f3ce4100570 0x7f3ce419e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:38.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.763+0000 7f3ce3200640 1 -- 172.21.15.43:0/416856501 >> 172.21.15.43:0/416856501 conn(0x7f3ce40fbfa0 msgr2=0x7f3ce4103320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:48:38.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.763+0000 7f3ce3200640 1 -- 172.21.15.43:0/416856501 shutdown_connections 2024-03-28T22:48:38.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.763+0000 7f3ce3200640 1 -- 172.21.15.43:0/416856501 wait complete. 2024-03-28T22:48:38.781 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-28T22:48:38.928 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:38 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mon-a[31819]: 2024-03-28T22:48:38.753+0000 7fa532600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T22:48:38.928 INFO:journalctl@ceph.mds.d.smithi043.stdout:Mar 28 22:48:38 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-d[70267]: ignoring --setuser ceph since I am not root 2024-03-28T22:48:38.929 INFO:journalctl@ceph.mds.d.smithi043.stdout:Mar 28 22:48:38 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-d[70267]: ignoring --setgroup ceph since I am not root 2024-03-28T22:48:38.929 INFO:journalctl@ceph.mds.d.smithi043.stdout:Mar 28 22:48:38 smithi043 ceph-mds[70271]: -- 172.21.15.43:0/841392412 <== mon.2 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d0bef4da40 con 0x55d0bef21000 2024-03-28T22:48:38.929 INFO:journalctl@ceph.mds.d.smithi043.stdout:Mar 28 22:48:38 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-d[70267]: starting mds.d at 2024-03-28T22:48:38.929 INFO:journalctl@ceph.mds.j.smithi043.stdout:Mar 28 22:48:38 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-j[69875]: ignoring --setuser ceph since I am not root 2024-03-28T22:48:38.929 INFO:journalctl@ceph.mds.j.smithi043.stdout:Mar 28 22:48:38 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-j[69875]: ignoring --setgroup ceph since I am not root 2024-03-28T22:48:38.930 INFO:journalctl@ceph.mds.j.smithi043.stdout:Mar 28 22:48:38 smithi043 ceph-mds[69879]: -- 172.21.15.43:0/2609560223 <== mon.0 v2:172.21.15.43:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556f146b5a40 con 0x556f14688c00 2024-03-28T22:48:38.930 INFO:journalctl@ceph.mds.j.smithi043.stdout:Mar 28 22:48:38 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-j[69875]: starting mds.j at 2024-03-28T22:48:38.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.997+0000 7f5dde400640 1 -- 172.21.15.43:0/3988812793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dd8069310 msgr2=0x7f5dd810e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:38.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.997+0000 7f5dde400640 1 --2- 172.21.15.43:0/3988812793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dd8069310 0x7f5dd810e5b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0009cd0 tx=0x7f5dd002c0e0 comp rx=0 tx=0).stop 2024-03-28T22:48:38.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.997+0000 7f5dde400640 1 -- 172.21.15.43:0/3988812793 shutdown_connections 2024-03-28T22:48:38.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.997+0000 7f5dde400640 1 --2- 172.21.15.43:0/3988812793 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5dd8068950 0x7f5dd8068dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:38.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.997+0000 7f5dde400640 1 --2- 172.21.15.43:0/3988812793 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dd8069310 0x7f5dd810e5b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:38.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.997+0000 7f5dde400640 1 --2- 172.21.15.43:0/3988812793 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dd81077a0 0x7f5dd8107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:38.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.997+0000 7f5dde400640 1 -- 172.21.15.43:0/3988812793 >> 172.21.15.43:0/3988812793 conn(0x7f5dd8077750 msgr2=0x7f5dd8077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:48:38.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.997+0000 7f5dde400640 1 -- 172.21.15.43:0/3988812793 shutdown_connections 2024-03-28T22:48:38.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.997+0000 7f5dde400640 1 -- 172.21.15.43:0/3988812793 wait complete. 2024-03-28T22:48:38.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.998+0000 7f5dde400640 1 Processor -- start 2024-03-28T22:48:38.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.998+0000 7f5dde400640 1 -- start start 2024-03-28T22:48:38.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.999+0000 7f5dde400640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5dd8068950 0x7f5dd819a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:38.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.999+0000 7f5dde400640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dd8069310 0x7f5dd819a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:38.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.999+0000 7f5dde400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dd81077a0 0x7f5dd819ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:38.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.999+0000 7f5dde400640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd8111af0 con 0x7f5dd8069310 2024-03-28T22:48:38.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.999+0000 7f5dde400640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd8111970 con 0x7f5dd8068950 2024-03-28T22:48:38.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.999+0000 7f5dde400640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd8111c70 con 0x7f5dd81077a0 2024-03-28T22:48:38.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.999+0000 7f5dd6a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dd8069310 0x7f5dd819a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:38.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.999+0000 7f5dd6a00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dd8069310 0x7f5dd819a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43960/0 (socket says 172.21.15.43:43960) 2024-03-28T22:48:38.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.999+0000 7f5dd6a00640 1 -- 172.21.15.43:0/3453183964 learned_addr learned my addr 172.21.15.43:0/3453183964 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:48:38.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.999+0000 7f5dd7e00640 1 --2- 172.21.15.43:0/3453183964 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dd81077a0 0x7f5dd819ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:38.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.999+0000 7f5dd7400640 1 --2- 172.21.15.43:0/3453183964 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5dd8068950 0x7f5dd819a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:38.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.999+0000 7f5dd7e00640 1 -- 172.21.15.43:0/3453183964 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5dd8068950 msgr2=0x7f5dd819a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:39.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.999+0000 7f5dd7e00640 1 --2- 172.21.15.43:0/3453183964 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5dd8068950 0x7f5dd819a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:39.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.999+0000 7f5dd7e00640 1 -- 172.21.15.43:0/3453183964 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dd8069310 msgr2=0x7f5dd819a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:39.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:38.999+0000 7f5dd7e00640 1 --2- 172.21.15.43:0/3453183964 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dd8069310 0x7f5dd819a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:39.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.000+0000 7f5dd7e00640 1 -- 172.21.15.43:0/3453183964 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dd819f140 con 0x7f5dd81077a0 2024-03-28T22:48:39.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.000+0000 7f5dd7400640 1 --2- 172.21.15.43:0/3453183964 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5dd8068950 0x7f5dd819a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:48:39.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.000+0000 7f5dd6a00640 1 --2- 172.21.15.43:0/3453183964 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dd8069310 0x7f5dd819a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T22:48:39.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.000+0000 7f5dd7e00640 1 --2- 172.21.15.43:0/3453183964 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dd81077a0 0x7f5dd819ea10 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0037040 tx=0x7f5dd0034670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:48:39.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.000+0000 7f5dcfe00640 1 -- 172.21.15.43:0/3453183964 <== mon.1 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd0039050 con 0x7f5dd81077a0 2024-03-28T22:48:39.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.000+0000 7f5dcfe00640 1 -- 172.21.15.43:0/3453183964 <== mon.1 v2:172.21.15.172:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5dd0045070 con 0x7f5dd81077a0 2024-03-28T22:48:39.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.000+0000 7f5dde400640 1 -- 172.21.15.43:0/3453183964 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dd819f3d0 con 0x7f5dd81077a0 2024-03-28T22:48:39.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.000+0000 7f5dcfe00640 1 -- 172.21.15.43:0/3453183964 <== mon.1 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd0004460 con 0x7f5dd81077a0 2024-03-28T22:48:39.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.000+0000 7f5dde400640 1 -- 172.21.15.43:0/3453183964 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dd81a6cc0 con 0x7f5dd81077a0 2024-03-28T22:48:39.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.001+0000 7f5dde400640 1 -- 172.21.15.43:0/3453183964 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dd8102e20 con 0x7f5dd81077a0 2024-03-28T22:48:39.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.003+0000 7f5dcfe00640 1 -- 172.21.15.43:0/3453183964 <== mon.1 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f5dd0004600 con 0x7f5dd81077a0 2024-03-28T22:48:39.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.004+0000 7f5dcfe00640 1 --2- 172.21.15.43:0/3453183964 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f5dbc0a1ef0 0x7f5dbc0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:39.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.005+0000 7f5dcfe00640 1 -- 172.21.15.43:0/3453183964 <== mon.1 v2:172.21.15.172:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9881+0+0 (secure 0 0 0) 0x7f5dd00f0bb0 con 0x7f5dd81077a0 2024-03-28T22:48:39.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.005+0000 7f5dd7400640 1 --2- 172.21.15.43:0/3453183964 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f5dbc0a1ef0 0x7f5dbc0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:39.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.005+0000 7f5dd7400640 1 --2- 172.21.15.43:0/3453183964 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f5dbc0a1ef0 0x7f5dbc0a43b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f5dc8009fc0 tx=0x7f5dc8005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:48:39.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.008+0000 7f5dcfe00640 1 -- 172.21.15.43:0/3453183964 <== mon.1 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5dd00bd800 con 0x7f5dd81077a0 2024-03-28T22:48:39.145 INFO:journalctl@ceph.mds.c.smithi172.stdout:Mar 28 22:48:38 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-c[66974]: ignoring --setuser ceph since I am not root 2024-03-28T22:48:39.145 INFO:journalctl@ceph.mds.c.smithi172.stdout:Mar 28 22:48:38 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-c[66974]: ignoring --setgroup ceph since I am not root 2024-03-28T22:48:39.145 INFO:journalctl@ceph.mds.c.smithi172.stdout:Mar 28 22:48:38 smithi172 ceph-mds[66978]: -- 172.21.15.172:0/1074069750 <== mon.1 v2:172.21.15.172:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560ea1641a40 con 0x560ea1615000 2024-03-28T22:48:39.146 INFO:journalctl@ceph.mds.c.smithi172.stdout:Mar 28 22:48:38 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-c[66974]: starting mds.c at 2024-03-28T22:48:39.146 INFO:journalctl@ceph.mds.i.smithi172.stdout:Mar 28 22:48:38 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-i[67293]: ignoring --setuser ceph since I am not root 2024-03-28T22:48:39.146 INFO:journalctl@ceph.mds.i.smithi172.stdout:Mar 28 22:48:38 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-i[67293]: ignoring --setgroup ceph since I am not root 2024-03-28T22:48:39.146 INFO:journalctl@ceph.mds.i.smithi172.stdout:Mar 28 22:48:38 smithi172 ceph-mds[67297]: -- 172.21.15.172:0/326578149 <== mon.2 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563a23ac7a40 con 0x563a23a9ac00 2024-03-28T22:48:39.146 INFO:journalctl@ceph.mds.i.smithi172.stdout:Mar 28 22:48:38 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-i[67293]: starting mds.i at 2024-03-28T22:48:39.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:38 smithi172 ceph-mon[36566]: pgmap v359: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 767 B/s wr, 13 op/s 2024-03-28T22:48:39.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:38 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/1688533517' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:48:39.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:38 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/416856501' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T22:48:39.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:38 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T22:48:39.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:38 smithi172 ceph-mon[36566]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T22:48:39.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:38 smithi172 ceph-mon[36566]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T22:48:39.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:38 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-28T22:48:39.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:38 smithi172 ceph-mon[36566]: osdmap e111: 12 total, 12 up, 12 in 2024-03-28T22:48:39.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:38 smithi172 ceph-mon[36566]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-28T22:48:39.148 INFO:journalctl@ceph.mds.f.smithi172.stdout:Mar 28 22:48:38 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-f[66664]: ignoring --setuser ceph since I am not root 2024-03-28T22:48:39.148 INFO:journalctl@ceph.mds.f.smithi172.stdout:Mar 28 22:48:38 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-f[66664]: ignoring --setgroup ceph since I am not root 2024-03-28T22:48:39.148 INFO:journalctl@ceph.mds.f.smithi172.stdout:Mar 28 22:48:38 smithi172 ceph-mds[66668]: -- 172.21.15.172:0/2857737343 <== mon.1 v2:172.21.15.172:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56497a2bda40 con 0x56497a290c00 2024-03-28T22:48:39.148 INFO:journalctl@ceph.mds.f.smithi172.stdout:Mar 28 22:48:38 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-f[66664]: starting mds.f at 2024-03-28T22:48:39.149 INFO:journalctl@ceph.mds.l.smithi172.stdout:Mar 28 22:48:38 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-l[66361]: ignoring --setuser ceph since I am not root 2024-03-28T22:48:39.149 INFO:journalctl@ceph.mds.l.smithi172.stdout:Mar 28 22:48:38 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-l[66361]: ignoring --setgroup ceph since I am not root 2024-03-28T22:48:39.149 INFO:journalctl@ceph.mds.l.smithi172.stdout:Mar 28 22:48:38 smithi172 ceph-mds[66365]: -- 172.21.15.172:0/816765485 <== mon.1 v2:172.21.15.172:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bef403ba40 con 0x55bef400f000 2024-03-28T22:48:39.149 INFO:journalctl@ceph.mds.l.smithi172.stdout:Mar 28 22:48:38 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-l[66361]: starting mds.l at 2024-03-28T22:48:39.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:38 smithi082 ceph-mon[36471]: pgmap v359: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 767 B/s wr, 13 op/s 2024-03-28T22:48:39.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:38 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/1688533517' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:48:39.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:38 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/416856501' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T22:48:39.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:38 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T22:48:39.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:38 smithi082 ceph-mon[36471]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T22:48:39.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:38 smithi082 ceph-mon[36471]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T22:48:39.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:38 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-28T22:48:39.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:38 smithi082 ceph-mon[36471]: osdmap e111: 12 total, 12 up, 12 in 2024-03-28T22:48:39.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:38 smithi082 ceph-mon[36471]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-28T22:48:39.200 INFO:journalctl@ceph.mds.k.smithi082.stdout:Mar 28 22:48:38 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-k[66321]: ignoring --setuser ceph since I am not root 2024-03-28T22:48:39.200 INFO:journalctl@ceph.mds.k.smithi082.stdout:Mar 28 22:48:38 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-k[66321]: ignoring --setgroup ceph since I am not root 2024-03-28T22:48:39.200 INFO:journalctl@ceph.mds.k.smithi082.stdout:Mar 28 22:48:38 smithi082 ceph-mds[66325]: -- 172.21.15.82:0/2363978712 <== mon.2 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cfdfc3fa40 con 0x55cfdfc13000 2024-03-28T22:48:39.200 INFO:journalctl@ceph.mds.k.smithi082.stdout:Mar 28 22:48:38 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-k[66321]: starting mds.k at 2024-03-28T22:48:39.200 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 28 22:48:38 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-h[66953]: ignoring --setuser ceph since I am not root 2024-03-28T22:48:39.201 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 28 22:48:38 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-h[66953]: ignoring --setgroup ceph since I am not root 2024-03-28T22:48:39.201 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 28 22:48:38 smithi082 ceph-mds[66957]: -- 172.21.15.82:0/3988404462 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x557fc596c4e0 con 0x557fc5943000 2024-03-28T22:48:39.201 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 28 22:48:38 smithi082 ceph-mds[66957]: -- 172.21.15.82:0/3988404462 <== mon.2 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557fc596fa40 con 0x557fc5943000 2024-03-28T22:48:39.201 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 28 22:48:38 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-h[66953]: starting mds.h at 2024-03-28T22:48:39.201 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 28 22:48:38 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-e[66639]: ignoring --setuser ceph since I am not root 2024-03-28T22:48:39.201 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 28 22:48:38 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-e[66639]: ignoring --setgroup ceph since I am not root 2024-03-28T22:48:39.202 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 28 22:48:38 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-e[66639]: starting mds.e at 2024-03-28T22:48:39.202 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 28 22:48:38 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-b[67270]: ignoring --setuser ceph since I am not root 2024-03-28T22:48:39.202 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 28 22:48:38 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-b[67270]: ignoring --setgroup ceph since I am not root 2024-03-28T22:48:39.202 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 28 22:48:38 smithi082 ceph-mds[67274]: -- 172.21.15.82:0/352607647 <== mon.2 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556535a7ba40 con 0x556535a4ec00 2024-03-28T22:48:39.202 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 28 22:48:38 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-b[67270]: starting mds.b at 2024-03-28T22:48:39.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.244+0000 7f5dde400640 1 -- 172.21.15.43:0/3453183964 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f5dd80ff650 con 0x7f5dd81077a0 2024-03-28T22:48:39.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:38 smithi043 ceph-mon[31823]: pgmap v359: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 767 B/s wr, 13 op/s 2024-03-28T22:48:39.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:38 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/1688533517' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T22:48:39.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:38 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/416856501' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T22:48:39.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:38 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T22:48:39.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:38 smithi043 ceph-mon[31823]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T22:48:39.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:38 smithi043 ceph-mon[31823]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T22:48:39.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:38 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-28T22:48:39.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:38 smithi043 ceph-mon[31823]: osdmap e111: 12 total, 12 up, 12 in 2024-03-28T22:48:39.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:38 smithi043 ceph-mon[31823]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-28T22:48:39.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.762+0000 7f5dcfe00640 1 -- 172.21.15.43:0/3453183964 <== mon.1 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v34) ==== 104+0+0 (secure 0 0 0) 0x7f5dd00bd100 con 0x7f5dd81077a0 2024-03-28T22:48:39.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.766+0000 7f5dde400640 1 -- 172.21.15.43:0/3453183964 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f5dbc0a1ef0 msgr2=0x7f5dbc0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:39.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.766+0000 7f5dde400640 1 --2- 172.21.15.43:0/3453183964 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f5dbc0a1ef0 0x7f5dbc0a43b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f5dc8009fc0 tx=0x7f5dc8005f90 comp rx=0 tx=0).stop 2024-03-28T22:48:39.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.767+0000 7f5dde400640 1 -- 172.21.15.43:0/3453183964 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dd81077a0 msgr2=0x7f5dd819ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:39.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.767+0000 7f5dde400640 1 --2- 172.21.15.43:0/3453183964 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dd81077a0 0x7f5dd819ea10 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0037040 tx=0x7f5dd0034670 comp rx=0 tx=0).stop 2024-03-28T22:48:39.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.767+0000 7f5dde400640 1 -- 172.21.15.43:0/3453183964 shutdown_connections 2024-03-28T22:48:39.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.767+0000 7f5dde400640 1 --2- 172.21.15.43:0/3453183964 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f5dbc0a1ef0 0x7f5dbc0a43b0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:39.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.767+0000 7f5dde400640 1 --2- 172.21.15.43:0/3453183964 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dd81077a0 0x7f5dd819ea10 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:39.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.767+0000 7f5dde400640 1 --2- 172.21.15.43:0/3453183964 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dd8069310 0x7f5dd819a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:39.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.767+0000 7f5dde400640 1 --2- 172.21.15.43:0/3453183964 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5dd8068950 0x7f5dd819a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:39.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.767+0000 7f5dde400640 1 -- 172.21.15.43:0/3453183964 >> 172.21.15.43:0/3453183964 conn(0x7f5dd8077750 msgr2=0x7f5dd810e280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:48:39.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.768+0000 7f5dde400640 1 -- 172.21.15.43:0/3453183964 shutdown_connections 2024-03-28T22:48:39.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.768+0000 7f5dde400640 1 -- 172.21.15.43:0/3453183964 wait complete. 2024-03-28T22:48:39.786 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-28T22:48:39.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.993+0000 7f06bbc00640 1 -- 172.21.15.43:0/38738590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06b4073b80 msgr2=0x7f06b4073f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:39.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.993+0000 7f06bbc00640 1 --2- 172.21.15.43:0/38738590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06b4073b80 0x7f06b4073f60 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f06a8009900 tx=0x7f06a802bcc0 comp rx=0 tx=0).stop 2024-03-28T22:48:39.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.993+0000 7f06bbc00640 1 -- 172.21.15.43:0/38738590 shutdown_connections 2024-03-28T22:48:39.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.993+0000 7f06bbc00640 1 --2- 172.21.15.43:0/38738590 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f06b40744a0 0x7f06b410d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:39.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.993+0000 7f06bbc00640 1 --2- 172.21.15.43:0/38738590 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f06b410df00 0x7f06b4110350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:39.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.993+0000 7f06bbc00640 1 --2- 172.21.15.43:0/38738590 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06b4073b80 0x7f06b4073f60 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:39.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.993+0000 7f06bbc00640 1 -- 172.21.15.43:0/38738590 >> 172.21.15.43:0/38738590 conn(0x7f06b40fd1f0 msgr2=0x7f06b40ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:48:39.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.993+0000 7f06bbc00640 1 -- 172.21.15.43:0/38738590 shutdown_connections 2024-03-28T22:48:39.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.993+0000 7f06bbc00640 1 -- 172.21.15.43:0/38738590 wait complete. 2024-03-28T22:48:39.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.994+0000 7f06bbc00640 1 Processor -- start 2024-03-28T22:48:39.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.994+0000 7f06bbc00640 1 -- start start 2024-03-28T22:48:39.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.995+0000 7f06bbc00640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f06b4073b80 0x7f06b419a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:39.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.995+0000 7f06bbc00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f06b40744a0 0x7f06b419a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:39.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.995+0000 7f06bbc00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06b410df00 0x7f06b419eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:39.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.995+0000 7f06bbc00640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f06b4113c60 con 0x7f06b410df00 2024-03-28T22:48:39.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.995+0000 7f06bbc00640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f06b4113ae0 con 0x7f06b40744a0 2024-03-28T22:48:39.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.995+0000 7f06bbc00640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f06b4113de0 con 0x7f06b4073b80 2024-03-28T22:48:39.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.995+0000 7f06b9c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06b410df00 0x7f06b419eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:39.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.995+0000 7f06b3e00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f06b40744a0 0x7f06b419a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:39.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.995+0000 7f06b9c00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06b410df00 0x7f06b419eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43968/0 (socket says 172.21.15.43:43968) 2024-03-28T22:48:39.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.995+0000 7f06b9c00640 1 -- 172.21.15.43:0/4247472252 learned_addr learned my addr 172.21.15.43:0/4247472252 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:48:39.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.995+0000 7f06b9200640 1 --2- 172.21.15.43:0/4247472252 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f06b4073b80 0x7f06b419a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:39.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.995+0000 7f06b3e00640 1 -- 172.21.15.43:0/4247472252 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f06b4073b80 msgr2=0x7f06b419a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:39.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.995+0000 7f06b3e00640 1 --2- 172.21.15.43:0/4247472252 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f06b4073b80 0x7f06b419a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:39.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.995+0000 7f06b3e00640 1 -- 172.21.15.43:0/4247472252 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06b410df00 msgr2=0x7f06b419eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:39.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.995+0000 7f06b3e00640 1 --2- 172.21.15.43:0/4247472252 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06b410df00 0x7f06b419eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:39.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.995+0000 7f06b3e00640 1 -- 172.21.15.43:0/4247472252 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06b419f210 con 0x7f06b40744a0 2024-03-28T22:48:39.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.996+0000 7f06b3e00640 1 --2- 172.21.15.43:0/4247472252 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f06b40744a0 0x7f06b419a760 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f06a000e9f0 tx=0x7f06a000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:48:39.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.996+0000 7f06b1600640 1 -- 172.21.15.43:0/4247472252 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a000cd80 con 0x7f06b40744a0 2024-03-28T22:48:39.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.996+0000 7f06b1600640 1 -- 172.21.15.43:0/4247472252 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06a00035f0 con 0x7f06b40744a0 2024-03-28T22:48:39.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.996+0000 7f06bbc00640 1 -- 172.21.15.43:0/4247472252 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06b419f500 con 0x7f06b40744a0 2024-03-28T22:48:39.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.996+0000 7f06b1600640 1 -- 172.21.15.43:0/4247472252 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a0010670 con 0x7f06b40744a0 2024-03-28T22:48:39.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.996+0000 7f06bbc00640 1 -- 172.21.15.43:0/4247472252 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06b41a6d90 con 0x7f06b40744a0 2024-03-28T22:48:39.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.997+0000 7f06bbc00640 1 -- 172.21.15.43:0/4247472252 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06b4068990 con 0x7f06b40744a0 2024-03-28T22:48:39.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:39.999+0000 7f06b1600640 1 -- 172.21.15.43:0/4247472252 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f06a00037a0 con 0x7f06b40744a0 2024-03-28T22:48:39.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:40.001+0000 7f06b1600640 1 --2- 172.21.15.43:0/4247472252 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f069c0a1ef0 0x7f069c0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:39.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:40.001+0000 7f06b1600640 1 -- 172.21.15.43:0/4247472252 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9881+0+0 (secure 0 0 0) 0x7f06a00ce9d0 con 0x7f06b40744a0 2024-03-28T22:48:39.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:40.001+0000 7f06b9200640 1 --2- 172.21.15.43:0/4247472252 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f069c0a1ef0 0x7f069c0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:39.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:40.002+0000 7f06b9200640 1 --2- 172.21.15.43:0/4247472252 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f069c0a1ef0 0x7f069c0a43b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f06a8009230 tx=0x7f06a802df90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:48:40.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:40.005+0000 7f06b1600640 1 -- 172.21.15.43:0/4247472252 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f06a009b510 con 0x7f06b40744a0 2024-03-28T22:48:40.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:39 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3453183964' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T22:48:40.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:39 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T22:48:40.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:39 smithi082 ceph-mon[36471]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T22:48:40.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:39 smithi082 ceph-mon[36471]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-28T22:48:40.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:39 smithi082 ceph-mon[36471]: Cluster is now healthy 2024-03-28T22:48:40.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:39 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-28T22:48:40.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:39 smithi082 ceph-mon[36471]: mds.? [v2:172.21.15.82:6835/996288537,v1:172.21.15.82:6837/996288537] up:boot 2024-03-28T22:48:40.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:39 smithi082 ceph-mon[36471]: mds.? [v2:172.21.15.82:6832/1786167427,v1:172.21.15.82:6833/1786167427] up:boot 2024-03-28T22:48:40.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:39 smithi082 ceph-mon[36471]: mds.? [v2:172.21.15.172:6832/4193252962,v1:172.21.15.172:6833/4193252962] up:boot 2024-03-28T22:48:40.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:39 smithi082 ceph-mon[36471]: mds.? [v2:172.21.15.82:6838/1214408456,v1:172.21.15.82:6839/1214408456] up:boot 2024-03-28T22:48:40.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:39 smithi082 ceph-mon[36471]: mds.? [v2:172.21.15.172:6836/1284880736,v1:172.21.15.172:6838/1284880736] up:boot 2024-03-28T22:48:40.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:39 smithi082 ceph-mon[36471]: mds.? [v2:172.21.15.82:6834/825264325,v1:172.21.15.82:6836/825264325] up:boot 2024-03-28T22:48:40.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:39 smithi082 ceph-mon[36471]: mds.? [v2:172.21.15.172:6834/1158518510,v1:172.21.15.172:6835/1158518510] up:boot 2024-03-28T22:48:40.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:39 smithi082 ceph-mon[36471]: mds.? [v2:172.21.15.172:6837/4120007637,v1:172.21.15.172:6839/4120007637] up:boot 2024-03-28T22:48:40.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:39 smithi082 ceph-mon[36471]: mds.? [v2:172.21.15.43:6834/60310384,v1:172.21.15.43:6835/60310384] up:boot 2024-03-28T22:48:40.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:39 smithi082 ceph-mon[36471]: mds.? [v2:172.21.15.43:6836/3813070286,v1:172.21.15.43:6837/3813070286] up:boot 2024-03-28T22:48:40.200 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:39 smithi082 ceph-mon[36471]: fsmap 12 up:standby 2024-03-28T22:48:40.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:39 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T22:48:40.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:39 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T22:48:40.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:39 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T22:48:40.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:39 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T22:48:40.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:39 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T22:48:40.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:39 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T22:48:40.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:39 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T22:48:40.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:39 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T22:48:40.201 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:39 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T22:48:40.202 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:39 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T22:48:40.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:40.205+0000 7f06bbc00640 1 -- 172.21.15.43:0/4247472252 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f06b402d170 con 0x7f06b40744a0 2024-03-28T22:48:40.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:39 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3453183964' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T22:48:40.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:39 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T22:48:40.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:39 smithi043 ceph-mon[31823]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T22:48:40.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:39 smithi043 ceph-mon[31823]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-28T22:48:40.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:39 smithi043 ceph-mon[31823]: Cluster is now healthy 2024-03-28T22:48:40.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:39 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-28T22:48:40.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:39 smithi043 ceph-mon[31823]: mds.? [v2:172.21.15.82:6835/996288537,v1:172.21.15.82:6837/996288537] up:boot 2024-03-28T22:48:40.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:39 smithi043 ceph-mon[31823]: mds.? [v2:172.21.15.82:6832/1786167427,v1:172.21.15.82:6833/1786167427] up:boot 2024-03-28T22:48:40.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:39 smithi043 ceph-mon[31823]: mds.? [v2:172.21.15.172:6832/4193252962,v1:172.21.15.172:6833/4193252962] up:boot 2024-03-28T22:48:40.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:39 smithi043 ceph-mon[31823]: mds.? [v2:172.21.15.82:6838/1214408456,v1:172.21.15.82:6839/1214408456] up:boot 2024-03-28T22:48:40.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:39 smithi043 ceph-mon[31823]: mds.? [v2:172.21.15.172:6836/1284880736,v1:172.21.15.172:6838/1284880736] up:boot 2024-03-28T22:48:40.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:39 smithi043 ceph-mon[31823]: mds.? [v2:172.21.15.82:6834/825264325,v1:172.21.15.82:6836/825264325] up:boot 2024-03-28T22:48:40.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:39 smithi043 ceph-mon[31823]: mds.? [v2:172.21.15.172:6834/1158518510,v1:172.21.15.172:6835/1158518510] up:boot 2024-03-28T22:48:40.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:39 smithi043 ceph-mon[31823]: mds.? [v2:172.21.15.172:6837/4120007637,v1:172.21.15.172:6839/4120007637] up:boot 2024-03-28T22:48:40.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:39 smithi043 ceph-mon[31823]: mds.? [v2:172.21.15.43:6834/60310384,v1:172.21.15.43:6835/60310384] up:boot 2024-03-28T22:48:40.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:39 smithi043 ceph-mon[31823]: mds.? [v2:172.21.15.43:6836/3813070286,v1:172.21.15.43:6837/3813070286] up:boot 2024-03-28T22:48:40.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:39 smithi043 ceph-mon[31823]: fsmap 12 up:standby 2024-03-28T22:48:40.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:39 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T22:48:40.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:39 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T22:48:40.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:39 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T22:48:40.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:39 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T22:48:40.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:39 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T22:48:40.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:39 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T22:48:40.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:39 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T22:48:40.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:39 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T22:48:40.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:39 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T22:48:40.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:39 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T22:48:40.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:39 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3453183964' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T22:48:40.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:39 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T22:48:40.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:39 smithi172 ceph-mon[36566]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T22:48:40.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:39 smithi172 ceph-mon[36566]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-28T22:48:40.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:39 smithi172 ceph-mon[36566]: Cluster is now healthy 2024-03-28T22:48:40.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:39 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-28T22:48:40.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:39 smithi172 ceph-mon[36566]: mds.? [v2:172.21.15.82:6835/996288537,v1:172.21.15.82:6837/996288537] up:boot 2024-03-28T22:48:40.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:39 smithi172 ceph-mon[36566]: mds.? [v2:172.21.15.82:6832/1786167427,v1:172.21.15.82:6833/1786167427] up:boot 2024-03-28T22:48:40.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:39 smithi172 ceph-mon[36566]: mds.? [v2:172.21.15.172:6832/4193252962,v1:172.21.15.172:6833/4193252962] up:boot 2024-03-28T22:48:40.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:39 smithi172 ceph-mon[36566]: mds.? [v2:172.21.15.82:6838/1214408456,v1:172.21.15.82:6839/1214408456] up:boot 2024-03-28T22:48:40.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:39 smithi172 ceph-mon[36566]: mds.? [v2:172.21.15.172:6836/1284880736,v1:172.21.15.172:6838/1284880736] up:boot 2024-03-28T22:48:40.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:39 smithi172 ceph-mon[36566]: mds.? [v2:172.21.15.82:6834/825264325,v1:172.21.15.82:6836/825264325] up:boot 2024-03-28T22:48:40.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:39 smithi172 ceph-mon[36566]: mds.? [v2:172.21.15.172:6834/1158518510,v1:172.21.15.172:6835/1158518510] up:boot 2024-03-28T22:48:40.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:39 smithi172 ceph-mon[36566]: mds.? [v2:172.21.15.172:6837/4120007637,v1:172.21.15.172:6839/4120007637] up:boot 2024-03-28T22:48:40.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:39 smithi172 ceph-mon[36566]: mds.? [v2:172.21.15.43:6834/60310384,v1:172.21.15.43:6835/60310384] up:boot 2024-03-28T22:48:40.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:39 smithi172 ceph-mon[36566]: mds.? [v2:172.21.15.43:6836/3813070286,v1:172.21.15.43:6837/3813070286] up:boot 2024-03-28T22:48:40.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:39 smithi172 ceph-mon[36566]: fsmap 12 up:standby 2024-03-28T22:48:40.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:39 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T22:48:40.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:39 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T22:48:40.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:39 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T22:48:40.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:39 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T22:48:40.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:39 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T22:48:40.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:39 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T22:48:40.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:39 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T22:48:40.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:39 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T22:48:40.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:39 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T22:48:40.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:39 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T22:48:40.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:40.947+0000 7f06b1600640 1 -- 172.21.15.43:0/4247472252 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v112) ==== 181+0+0 (secure 0 0 0) 0x7f06a009ae10 con 0x7f06b40744a0 2024-03-28T22:48:40.945 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'cephfs_metadata' removed 2024-03-28T22:48:40.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:40.955+0000 7f06bbc00640 1 -- 172.21.15.43:0/4247472252 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f069c0a1ef0 msgr2=0x7f069c0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:40.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:40.955+0000 7f06bbc00640 1 --2- 172.21.15.43:0/4247472252 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f069c0a1ef0 0x7f069c0a43b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f06a8009230 tx=0x7f06a802df90 comp rx=0 tx=0).stop 2024-03-28T22:48:40.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:40.955+0000 7f06bbc00640 1 -- 172.21.15.43:0/4247472252 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f06b40744a0 msgr2=0x7f06b419a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:40.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:40.955+0000 7f06bbc00640 1 --2- 172.21.15.43:0/4247472252 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f06b40744a0 0x7f06b419a760 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f06a000e9f0 tx=0x7f06a000eeb0 comp rx=0 tx=0).stop 2024-03-28T22:48:40.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:40.957+0000 7f06bbc00640 1 -- 172.21.15.43:0/4247472252 shutdown_connections 2024-03-28T22:48:40.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:40.957+0000 7f06bbc00640 1 --2- 172.21.15.43:0/4247472252 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f069c0a1ef0 0x7f069c0a43b0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:40.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:40.957+0000 7f06bbc00640 1 --2- 172.21.15.43:0/4247472252 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f06b40744a0 0x7f06b419a760 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:40.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:40.957+0000 7f06bbc00640 1 --2- 172.21.15.43:0/4247472252 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06b410df00 0x7f06b419eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:40.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:40.957+0000 7f06bbc00640 1 --2- 172.21.15.43:0/4247472252 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f06b4073b80 0x7f06b419a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:40.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:40.957+0000 7f06bbc00640 1 -- 172.21.15.43:0/4247472252 >> 172.21.15.43:0/4247472252 conn(0x7f06b40fd1f0 msgr2=0x7f06b40feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:48:40.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:40.957+0000 7f06bbc00640 1 -- 172.21.15.43:0/4247472252 shutdown_connections 2024-03-28T22:48:40.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:40.958+0000 7f06bbc00640 1 -- 172.21.15.43:0/4247472252 wait complete. 2024-03-28T22:48:40.982 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-28T22:48:41.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.201+0000 7f24d1000640 1 -- 172.21.15.43:0/511657392 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24cc1042b0 msgr2=0x7f24cc1107c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:41.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.201+0000 7f24d1000640 1 --2- 172.21.15.43:0/511657392 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24cc1042b0 0x7f24cc1107c0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f24c4009ca0 tx=0x7f24c402c0f0 comp rx=0 tx=0).stop 2024-03-28T22:48:41.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.201+0000 7f24d1000640 1 -- 172.21.15.43:0/511657392 shutdown_connections 2024-03-28T22:48:41.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.201+0000 7f24d1000640 1 --2- 172.21.15.43:0/511657392 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24cc1042b0 0x7f24cc1107c0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:41.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.201+0000 7f24d1000640 1 --2- 172.21.15.43:0/511657392 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24cc1038f0 0x7f24cc103d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:41.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.201+0000 7f24d1000640 1 --2- 172.21.15.43:0/511657392 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24cc109900 0x7f24cc109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:41.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.201+0000 7f24d1000640 1 -- 172.21.15.43:0/511657392 >> 172.21.15.43:0/511657392 conn(0x7f24cc0ff400 msgr2=0x7f24cc101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:48:41.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.201+0000 7f24d1000640 1 -- 172.21.15.43:0/511657392 shutdown_connections 2024-03-28T22:48:41.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.202+0000 7f24d1000640 1 -- 172.21.15.43:0/511657392 wait complete. 2024-03-28T22:48:41.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.203+0000 7f24d1000640 1 Processor -- start 2024-03-28T22:48:41.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.203+0000 7f24d1000640 1 -- start start 2024-03-28T22:48:41.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.204+0000 7f24d1000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24cc1038f0 0x7f24cc19a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:41.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.204+0000 7f24d1000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24cc1042b0 0x7f24cc19a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:41.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.204+0000 7f24d1000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24cc109900 0x7f24cc19eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:41.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.204+0000 7f24d1000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f24cc113d10 con 0x7f24cc1038f0 2024-03-28T22:48:41.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.204+0000 7f24d1000640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f24cc113b90 con 0x7f24cc109900 2024-03-28T22:48:41.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.204+0000 7f24d1000640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f24cc113e90 con 0x7f24cc1042b0 2024-03-28T22:48:41.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.204+0000 7f24c9e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24cc1038f0 0x7f24cc19a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:41.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.204+0000 7f24ca800640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24cc109900 0x7f24cc19eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:41.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.204+0000 7f24c9e00640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24cc1038f0 0x7f24cc19a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43976/0 (socket says 172.21.15.43:43976) 2024-03-28T22:48:41.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.204+0000 7f24c9400640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24cc1042b0 0x7f24cc19a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:41.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.204+0000 7f24c9e00640 1 -- 172.21.15.43:0/2055476658 learned_addr learned my addr 172.21.15.43:0/2055476658 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:48:41.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.204+0000 7f24ca800640 1 -- 172.21.15.43:0/2055476658 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24cc1042b0 msgr2=0x7f24cc19a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:41.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.205+0000 7f24ca800640 1 --2- 172.21.15.43:0/2055476658 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24cc1042b0 0x7f24cc19a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:41.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.205+0000 7f24ca800640 1 -- 172.21.15.43:0/2055476658 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24cc1038f0 msgr2=0x7f24cc19a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:41.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.205+0000 7f24ca800640 1 --2- 172.21.15.43:0/2055476658 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24cc1038f0 0x7f24cc19a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:41.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.205+0000 7f24ca800640 1 -- 172.21.15.43:0/2055476658 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24cc19f1d0 con 0x7f24cc109900 2024-03-28T22:48:41.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.205+0000 7f24ca800640 1 --2- 172.21.15.43:0/2055476658 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24cc109900 0x7f24cc19eaa0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f24c4009bf0 tx=0x7f24c4032ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:48:41.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.205+0000 7f24c2a00640 1 -- 172.21.15.43:0/2055476658 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c4042020 con 0x7f24cc109900 2024-03-28T22:48:41.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.205+0000 7f24c2a00640 1 -- 172.21.15.43:0/2055476658 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24c4034640 con 0x7f24cc109900 2024-03-28T22:48:41.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.205+0000 7f24d1000640 1 -- 172.21.15.43:0/2055476658 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24cc19f460 con 0x7f24cc109900 2024-03-28T22:48:41.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.205+0000 7f24c2a00640 1 -- 172.21.15.43:0/2055476658 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c4039040 con 0x7f24cc109900 2024-03-28T22:48:41.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.205+0000 7f24d1000640 1 -- 172.21.15.43:0/2055476658 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24cc1a6d50 con 0x7f24cc109900 2024-03-28T22:48:41.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.207+0000 7f24d1000640 1 -- 172.21.15.43:0/2055476658 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24cc105030 con 0x7f24cc109900 2024-03-28T22:48:41.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.208+0000 7f24c2a00640 1 -- 172.21.15.43:0/2055476658 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f24c4045020 con 0x7f24cc109900 2024-03-28T22:48:41.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.210+0000 7f24c2a00640 1 --2- 172.21.15.43:0/2055476658 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f24b00a1ef0 0x7f24b00a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:41.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.210+0000 7f24c2a00640 1 -- 172.21.15.43:0/2055476658 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9385+0+0 (secure 0 0 0) 0x7f24c40effe0 con 0x7f24cc109900 2024-03-28T22:48:41.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.210+0000 7f24c9e00640 1 --2- 172.21.15.43:0/2055476658 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f24b00a1ef0 0x7f24b00a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:41.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.211+0000 7f24c9e00640 1 --2- 172.21.15.43:0/2055476658 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f24b00a1ef0 0x7f24b00a43b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f24bc009c50 tx=0x7f24bc005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:48:41.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.214+0000 7f24c2a00640 1 -- 172.21.15.43:0/2055476658 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f24c404c020 con 0x7f24cc109900 2024-03-28T22:48:41.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:40 smithi043 ceph-mon[31823]: pgmap v361: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 921 B/s wr, 11 op/s 2024-03-28T22:48:41.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:40 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/4247472252' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T22:48:41.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:40 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T22:48:41.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:40 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:48:41.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:40 smithi172 ceph-mon[36566]: pgmap v361: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 921 B/s wr, 11 op/s 2024-03-28T22:48:41.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:40 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/4247472252' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T22:48:41.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:40 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T22:48:41.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:40 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:48:41.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.432+0000 7f24d1000640 1 -- 172.21.15.43:0/2055476658 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f24cc10beb0 con 0x7f24cc109900 2024-03-28T22:48:41.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:40 smithi082 ceph-mon[36471]: pgmap v361: 129 pgs: 129 active+clean; 58 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 921 B/s wr, 11 op/s 2024-03-28T22:48:41.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:40 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/4247472252' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T22:48:41.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:40 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T22:48:41.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:40 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T22:48:41.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.967+0000 7f24c2a00640 1 -- 172.21.15.43:0/2055476658 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v113) ==== 169+0+0 (secure 0 0 0) 0x7f24c40bce40 con 0x7f24cc109900 2024-03-28T22:48:41.966 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'cephfs_data' removed 2024-03-28T22:48:41.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.973+0000 7f24d1000640 1 -- 172.21.15.43:0/2055476658 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f24b00a1ef0 msgr2=0x7f24b00a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:41.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.974+0000 7f24d1000640 1 --2- 172.21.15.43:0/2055476658 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f24b00a1ef0 0x7f24b00a43b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f24bc009c50 tx=0x7f24bc005f90 comp rx=0 tx=0).stop 2024-03-28T22:48:41.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.974+0000 7f24d1000640 1 -- 172.21.15.43:0/2055476658 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24cc109900 msgr2=0x7f24cc19eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:41.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.974+0000 7f24d1000640 1 --2- 172.21.15.43:0/2055476658 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24cc109900 0x7f24cc19eaa0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f24c4009bf0 tx=0x7f24c4032ea0 comp rx=0 tx=0).stop 2024-03-28T22:48:41.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.975+0000 7f24d1000640 1 -- 172.21.15.43:0/2055476658 shutdown_connections 2024-03-28T22:48:41.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.975+0000 7f24d1000640 1 --2- 172.21.15.43:0/2055476658 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f24b00a1ef0 0x7f24b00a43b0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:41.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.975+0000 7f24d1000640 1 --2- 172.21.15.43:0/2055476658 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24cc109900 0x7f24cc19eaa0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:41.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.975+0000 7f24d1000640 1 --2- 172.21.15.43:0/2055476658 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f24cc1042b0 0x7f24cc19a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:41.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.975+0000 7f24d1000640 1 --2- 172.21.15.43:0/2055476658 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f24cc1038f0 0x7f24cc19a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:41.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.975+0000 7f24d1000640 1 -- 172.21.15.43:0/2055476658 >> 172.21.15.43:0/2055476658 conn(0x7f24cc0ff400 msgr2=0x7f24cc0ffaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:48:41.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.975+0000 7f24d1000640 1 -- 172.21.15.43:0/2055476658 shutdown_connections 2024-03-28T22:48:41.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:41.977+0000 7f24d1000640 1 -- 172.21.15.43:0/2055476658 wait complete. 2024-03-28T22:48:42.012 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-28T22:48:42.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.211+0000 7f0b02000640 1 -- 172.21.15.43:0/3606205575 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0afc1042c0 msgr2=0x7f0afc1107d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:42.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.211+0000 7f0b02000640 1 --2- 172.21.15.43:0/3606205575 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0afc1042c0 0x7f0afc1107d0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f0af4009ca0 tx=0x7f0af402c100 comp rx=0 tx=0).stop 2024-03-28T22:48:42.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.211+0000 7f0b02000640 1 -- 172.21.15.43:0/3606205575 shutdown_connections 2024-03-28T22:48:42.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.211+0000 7f0b02000640 1 --2- 172.21.15.43:0/3606205575 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0afc1042c0 0x7f0afc1107d0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:42.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.211+0000 7f0b02000640 1 --2- 172.21.15.43:0/3606205575 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0afc103900 0x7f0afc103d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:42.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.211+0000 7f0b02000640 1 --2- 172.21.15.43:0/3606205575 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0afc109910 0x7f0afc109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:42.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.211+0000 7f0b02000640 1 -- 172.21.15.43:0/3606205575 >> 172.21.15.43:0/3606205575 conn(0x7f0afc0ff3d0 msgr2=0x7f0afc1017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:48:42.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.211+0000 7f0b02000640 1 -- 172.21.15.43:0/3606205575 shutdown_connections 2024-03-28T22:48:42.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.211+0000 7f0b02000640 1 -- 172.21.15.43:0/3606205575 wait complete. 2024-03-28T22:48:42.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.212+0000 7f0b02000640 1 Processor -- start 2024-03-28T22:48:42.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.212+0000 7f0b02000640 1 -- start start 2024-03-28T22:48:42.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.212+0000 7f0b02000640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0afc103900 0x7f0afc19a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:42.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.212+0000 7f0b02000640 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0afc1042c0 0x7f0afc19a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:42.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.213+0000 7f0b02000640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0afc109910 0x7f0afc19ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:42.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.213+0000 7f0b02000640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0afc113d20 con 0x7f0afc103900 2024-03-28T22:48:42.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.213+0000 7f0b02000640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f0afc113ba0 con 0x7f0afc109910 2024-03-28T22:48:42.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.213+0000 7f0b02000640 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 -- 0x7f0afc113ea0 con 0x7f0afc1042c0 2024-03-28T22:48:42.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.213+0000 7f0afbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0afc109910 0x7f0afc19ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:42.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.213+0000 7f0afbe00640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0afc109910 0x7f0afc19ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.43:52520/0 (socket says 172.21.15.43:52520) 2024-03-28T22:48:42.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.213+0000 7f0afbe00640 1 -- 172.21.15.43:0/3452704105 learned_addr learned my addr 172.21.15.43:0/3452704105 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-03-28T22:48:42.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.213+0000 7f0afaa00640 1 --2- 172.21.15.43:0/3452704105 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0afc1042c0 0x7f0afc19a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:42.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.213+0000 7f0afb400640 1 --2- 172.21.15.43:0/3452704105 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0afc103900 0x7f0afc19a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:42.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.213+0000 7f0afbe00640 1 -- 172.21.15.43:0/3452704105 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0afc1042c0 msgr2=0x7f0afc19a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:42.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.213+0000 7f0afbe00640 1 --2- 172.21.15.43:0/3452704105 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0afc1042c0 0x7f0afc19a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:42.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.213+0000 7f0afbe00640 1 -- 172.21.15.43:0/3452704105 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0afc103900 msgr2=0x7f0afc19a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:42.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.213+0000 7f0afbe00640 1 --2- 172.21.15.43:0/3452704105 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0afc103900 0x7f0afc19a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:42.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.213+0000 7f0afbe00640 1 -- 172.21.15.43:0/3452704105 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0afc19f190 con 0x7f0afc109910 2024-03-28T22:48:42.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.214+0000 7f0afbe00640 1 --2- 172.21.15.43:0/3452704105 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0afc109910 0x7f0afc19ea60 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f0af402c710 tx=0x7f0af40346f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:48:42.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.214+0000 7f0af3e00640 1 -- 172.21.15.43:0/3452704105 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af4039050 con 0x7f0afc109910 2024-03-28T22:48:42.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.214+0000 7f0b02000640 1 -- 172.21.15.43:0/3452704105 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0afc19f420 con 0x7f0afc109910 2024-03-28T22:48:42.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.214+0000 7f0af3e00640 1 -- 172.21.15.43:0/3452704105 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0af4045070 con 0x7f0afc109910 2024-03-28T22:48:42.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.214+0000 7f0af3e00640 1 -- 172.21.15.43:0/3452704105 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af40046d0 con 0x7f0afc109910 2024-03-28T22:48:42.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.214+0000 7f0b02000640 1 -- 172.21.15.43:0/3452704105 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0afc1a6d10 con 0x7f0afc109910 2024-03-28T22:48:42.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.215+0000 7f0b02000640 1 -- 172.21.15.43:0/3452704105 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0afc068540 con 0x7f0afc109910 2024-03-28T22:48:42.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.217+0000 7f0af3e00640 1 -- 172.21.15.43:0/3452704105 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 28) ==== 136887+0+0 (secure 0 0 0) 0x7f0af4003970 con 0x7f0afc109910 2024-03-28T22:48:42.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.219+0000 7f0af3e00640 1 --2- 172.21.15.43:0/3452704105 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0ae00a1ef0 0x7f0ae00a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:48:42.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.219+0000 7f0af3e00640 1 -- 172.21.15.43:0/3452704105 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8797+0+0 (secure 0 0 0) 0x7f0af40f0990 con 0x7f0afc109910 2024-03-28T22:48:42.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.219+0000 7f0afb400640 1 --2- 172.21.15.43:0/3452704105 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0ae00a1ef0 0x7f0ae00a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T22:48:42.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.220+0000 7f0afb400640 1 --2- 172.21.15.43:0/3452704105 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0ae00a1ef0 0x7f0ae00a43b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f0aec009ae0 tx=0x7f0aec005f90 comp rx=0 tx=0).ready entity=mgr.25663 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T22:48:42.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.223+0000 7f0af3e00640 1 -- 172.21.15.43:0/3452704105 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0af40bda80 con 0x7f0afc109910 2024-03-28T22:48:42.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:41 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T22:48:42.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:41 smithi043 ceph-mon[31823]: osdmap e112: 12 total, 12 up, 12 in 2024-03-28T22:48:42.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:41 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/2055476658' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T22:48:42.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:41 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T22:48:42.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:41 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:48:42.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:41 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:48:42.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:41 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T22:48:42.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:41 smithi172 ceph-mon[36566]: osdmap e112: 12 total, 12 up, 12 in 2024-03-28T22:48:42.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:41 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/2055476658' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T22:48:42.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:41 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T22:48:42.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:41 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:48:42.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:41 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:48:42.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.438+0000 7f0b02000640 1 -- 172.21.15.43:0/3452704105 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f0afc10bea0 con 0x7f0afc109910 2024-03-28T22:48:42.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:41 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T22:48:42.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:41 smithi082 ceph-mon[36471]: osdmap e112: 12 total, 12 up, 12 in 2024-03-28T22:48:42.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:41 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/2055476658' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T22:48:42.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:41 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T22:48:42.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:41 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T22:48:42.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:41 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T22:48:42.988 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'cephfs_data_ec' removed 2024-03-28T22:48:42.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.989+0000 7f0af3e00640 1 -- 172.21.15.43:0/3452704105 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v114) ==== 178+0+0 (secure 0 0 0) 0x7f0af40bd380 con 0x7f0afc109910 2024-03-28T22:48:42.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.995+0000 7f0b02000640 1 -- 172.21.15.43:0/3452704105 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0ae00a1ef0 msgr2=0x7f0ae00a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:42.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.995+0000 7f0b02000640 1 --2- 172.21.15.43:0/3452704105 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0ae00a1ef0 0x7f0ae00a43b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f0aec009ae0 tx=0x7f0aec005f90 comp rx=0 tx=0).stop 2024-03-28T22:48:42.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.995+0000 7f0b02000640 1 -- 172.21.15.43:0/3452704105 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0afc109910 msgr2=0x7f0afc19ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T22:48:42.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.995+0000 7f0b02000640 1 --2- 172.21.15.43:0/3452704105 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0afc109910 0x7f0afc19ea60 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f0af402c710 tx=0x7f0af40346f0 comp rx=0 tx=0).stop 2024-03-28T22:48:42.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.995+0000 7f0b02000640 1 -- 172.21.15.43:0/3452704105 shutdown_connections 2024-03-28T22:48:42.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.995+0000 7f0b02000640 1 --2- 172.21.15.43:0/3452704105 >> [v2:172.21.15.43:6800/1685312925,v1:172.21.15.43:6801/1685312925] conn(0x7f0ae00a1ef0 0x7f0ae00a43b0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:42.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.995+0000 7f0b02000640 1 --2- 172.21.15.43:0/3452704105 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0afc109910 0x7f0afc19ea60 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:42.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.995+0000 7f0b02000640 1 --2- 172.21.15.43:0/3452704105 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0afc1042c0 0x7f0afc19a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:42.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.995+0000 7f0b02000640 1 --2- 172.21.15.43:0/3452704105 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0afc103900 0x7f0afc19a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:48:42.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.995+0000 7f0b02000640 1 -- 172.21.15.43:0/3452704105 >> 172.21.15.43:0/3452704105 conn(0x7f0afc0ff3d0 msgr2=0x7f0afc0ffb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T22:48:42.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.995+0000 7f0b02000640 1 -- 172.21.15.43:0/3452704105 shutdown_connections 2024-03-28T22:48:42.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-03-28T22:48:42.996+0000 7f0b02000640 1 -- 172.21.15.43:0/3452704105 wait complete. 2024-03-28T22:48:43.028 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T22:48:43.109 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T22:48:43.140 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T22:48:43.173 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-28T22:48:43.173 DEBUG:teuthology.orchestra.run.smithi043:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-28T22:48:43.243 DEBUG:teuthology.orchestra.run.smithi082:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-28T22:48:43.258 DEBUG:teuthology.orchestra.run.smithi172:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-28T22:48:43.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:42 smithi043 ceph-mon[31823]: pgmap v363: 97 pgs: 97 active+clean; 44 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:48:43.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:42 smithi043 ceph-mon[31823]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:48:43.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:42 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T22:48:43.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:42 smithi043 ceph-mon[31823]: osdmap e113: 12 total, 12 up, 12 in 2024-03-28T22:48:43.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:42 smithi043 ceph-mon[31823]: from='client.? 172.21.15.43:0/3452704105' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T22:48:43.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:42 smithi043 ceph-mon[31823]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T22:48:43.272 INFO:tasks.cephadm:Stopping all daemons... 2024-03-28T22:48:43.272 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-28T22:48:43.272 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-c2276428-ed51-11ee-95d0-87774f69a715@mon.a 2024-03-28T22:48:43.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:43 smithi172 ceph-mon[36566]: pgmap v363: 97 pgs: 97 active+clean; 44 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:48:43.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:43 smithi172 ceph-mon[36566]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:48:43.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:43 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T22:48:43.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:43 smithi172 ceph-mon[36566]: osdmap e113: 12 total, 12 up, 12 in 2024-03-28T22:48:43.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:43 smithi172 ceph-mon[36566]: from='client.? 172.21.15.43:0/3452704105' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T22:48:43.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:43 smithi172 ceph-mon[36566]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T22:48:43.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:43 smithi082 ceph-mon[36471]: pgmap v363: 97 pgs: 97 active+clean; 44 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T22:48:43.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:43 smithi082 ceph-mon[36471]: from='mgr.25663 172.21.15.43:0/1016343778' entity='mgr.x' 2024-03-28T22:48:43.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:43 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T22:48:43.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:43 smithi082 ceph-mon[36471]: osdmap e113: 12 total, 12 up, 12 in 2024-03-28T22:48:43.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:43 smithi082 ceph-mon[36471]: from='client.? 172.21.15.43:0/3452704105' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T22:48:43.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:43 smithi082 ceph-mon[36471]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T22:48:43.765 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:43 smithi043 systemd[1]: Stopping Ceph mon.a for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:48:44.170 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:43 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mon-a[31819]: 2024-03-28T22:48:43.817+0000 7fa539400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:48:44.170 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:43 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mon-a[31819]: 2024-03-28T22:48:43.817+0000 7fa539400700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-28T22:48:44.170 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:43 smithi043 podman[104155]: 2024-03-28 22:48:43.859675624 +0000 UTC m=+0.191211145 container died f2315b91a227e5922ae97fad845c259b4c5293dd04bf8a5a1b02a4cc7d4dba39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mon-a, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=squid-323d23d, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T22:48:44.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:44 smithi043 podman[104155]: 2024-03-28 22:48:44.310755241 +0000 UTC m=+0.642290784 container cleanup f2315b91a227e5922ae97fad845c259b4c5293dd04bf8a5a1b02a4cc7d4dba39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-323d23d, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T22:48:44.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:44 smithi043 bash[104155]: ceph-c2276428-ed51-11ee-95d0-87774f69a715-mon-a 2024-03-28T22:48:44.999 INFO:journalctl@ceph.mon.a.smithi043.stdout:Mar 28 22:48:44 smithi043 podman[104169]: 2024-03-28 22:48:44.545194223 +0000 UTC m=+0.669790625 container remove f2315b91a227e5922ae97fad845c259b4c5293dd04bf8a5a1b02a4cc7d4dba39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d, org.label-schema.vendor=CentOS) 2024-03-28T22:48:45.035 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mon.a.service' 2024-03-28T22:48:45.195 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:48:45.196 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-28T22:48:45.196 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-28T22:48:45.196 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-c2276428-ed51-11ee-95d0-87774f69a715@mon.b 2024-03-28T22:48:45.536 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:45 smithi082 systemd[1]: Stopping Ceph mon.b for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:48:45.937 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:45 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mon-b[36467]: 2024-03-28T22:48:45.538+0000 7f692feee700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:48:45.938 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:45 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mon-b[36467]: 2024-03-28T22:48:45.538+0000 7f692feee700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-28T22:48:45.938 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:45 smithi082 podman[73715]: 2024-03-28 22:48:45.682517949 +0000 UTC m=+0.273826262 container died 94c6712ec6de47c634a32d673f16ec7e69b4a67b01750c504bbc21bc3dd140d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mon-b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-323d23d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-28T22:48:46.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:46 smithi082 podman[73715]: 2024-03-28 22:48:46.039476815 +0000 UTC m=+0.630785126 container cleanup 94c6712ec6de47c634a32d673f16ec7e69b4a67b01750c504bbc21bc3dd140d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mon-b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240326) 2024-03-28T22:48:46.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 28 22:48:46 smithi082 bash[73715]: ceph-c2276428-ed51-11ee-95d0-87774f69a715-mon-b 2024-03-28T22:48:46.585 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mon.b.service' 2024-03-28T22:48:46.630 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:48:46.631 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-28T22:48:46.631 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-28T22:48:46.631 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-c2276428-ed51-11ee-95d0-87774f69a715@mon.c 2024-03-28T22:48:46.999 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:46 smithi172 systemd[1]: Stopping Ceph mon.c for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:48:47.267 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:47 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mon-c[36562]: 2024-03-28T22:48:47.004+0000 7f4dc7056700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:48:47.268 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:47 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mon-c[36562]: 2024-03-28T22:48:47.004+0000 7f4dc7056700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-28T22:48:47.268 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:47 smithi172 podman[73318]: 2024-03-28 22:48:47.067365513 +0000 UTC m=+0.235320669 container died 2fa7788646ace03a9eec69aa78bcc027ad37990e426a0c386cf01878fd8c0a94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mon-c, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-323d23d, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-28T22:48:47.528 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:47 smithi172 podman[73318]: 2024-03-28 22:48:47.438385351 +0000 UTC m=+0.606340490 container cleanup 2fa7788646ace03a9eec69aa78bcc027ad37990e426a0c386cf01878fd8c0a94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mon-c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-323d23d, ceph=True) 2024-03-28T22:48:47.529 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:47 smithi172 bash[73318]: ceph-c2276428-ed51-11ee-95d0-87774f69a715-mon-c 2024-03-28T22:48:47.864 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 28 22:48:47 smithi172 podman[73330]: 2024-03-28 22:48:47.73206604 +0000 UTC m=+0.662306477 container remove 2fa7788646ace03a9eec69aa78bcc027ad37990e426a0c386cf01878fd8c0a94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-323d23d, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-28T22:48:47.877 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mon.c.service' 2024-03-28T22:48:47.922 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:48:47.922 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-28T22:48:47.923 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-28T22:48:47.923 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-c2276428-ed51-11ee-95d0-87774f69a715@mgr.x 2024-03-28T22:48:48.265 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:48:48 smithi043 systemd[1]: Stopping Ceph mgr.x for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:48:49.016 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:48:48 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:48:48.686+0000 7f3620800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:48:49.016 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:48:48 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x[32051]: 2024-03-28T22:48:48.686+0000 7f3620800700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-28T22:48:49.016 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:48:48 smithi043 podman[104274]: 2024-03-28 22:48:48.802798716 +0000 UTC m=+0.386184284 container died 9d6e46f876ef3130d16aed7408ba72f3a9bda7d46d9e94aa29d6f93599a2bcd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-323d23d, ceph=True, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-28T22:48:49.515 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:48:49 smithi043 podman[104274]: 2024-03-28 22:48:49.307941235 +0000 UTC m=+0.891326819 container cleanup 9d6e46f876ef3130d16aed7408ba72f3a9bda7d46d9e94aa29d6f93599a2bcd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-323d23d, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-28T22:48:49.516 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:48:49 smithi043 bash[104274]: ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x 2024-03-28T22:48:49.995 INFO:journalctl@ceph.mgr.x.smithi043.stdout:Mar 28 22:48:49 smithi043 podman[104288]: 2024-03-28 22:48:49.525941968 +0000 UTC m=+0.706389264 container remove 9d6e46f876ef3130d16aed7408ba72f3a9bda7d46d9e94aa29d6f93599a2bcd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:323d23dc228bbcb7f16f188fdc9c0dcb801844be, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-x, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, RELEASE=squid-323d23d, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-28T22:48:50.036 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mgr.x.service' 2024-03-28T22:48:50.161 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:48:50.161 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-28T22:48:50.161 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-28T22:48:50.162 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-c2276428-ed51-11ee-95d0-87774f69a715@mgr.y 2024-03-28T22:48:50.449 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:48:50 smithi082 systemd[1]: Stopping Ceph mgr.y for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:48:50.840 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:48:50 smithi082 podman[73838]: 2024-03-28 22:48:50.506637878 +0000 UTC m=+0.150210650 container died 55e39a534d8ab5cbe4a6a6a2025794a7b4557ec5e20ffb89c1148fe1e59ce407 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-323d23d) 2024-03-28T22:48:51.198 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:48:50 smithi082 podman[73838]: 2024-03-28 22:48:50.844221981 +0000 UTC m=+0.487794742 container cleanup 55e39a534d8ab5cbe4a6a6a2025794a7b4557ec5e20ffb89c1148fe1e59ce407 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, GIT_CLEAN=True, RELEASE=squid-323d23d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-28T22:48:51.199 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:48:50 smithi082 bash[73838]: ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y 2024-03-28T22:48:51.199 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:48:51 smithi082 podman[73853]: 2024-03-28 22:48:51.037581103 +0000 UTC m=+0.527308935 container remove 55e39a534d8ab5cbe4a6a6a2025794a7b4557ec5e20ffb89c1148fe1e59ce407 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-y, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-323d23d, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-28T22:48:51.199 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 28 22:48:51 smithi082 systemd[1]: ceph-c2276428-ed51-11ee-95d0-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-03-28T22:48:51.245 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mgr.y.service' 2024-03-28T22:48:51.282 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:48:51.282 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-28T22:48:51.282 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-28T22:48:51.282 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-c2276428-ed51-11ee-95d0-87774f69a715@mgr.z 2024-03-28T22:48:51.636 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:48:51 smithi172 systemd[1]: Stopping Ceph mgr.z for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:48:51.895 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:48:51 smithi172 podman[73451]: 2024-03-28 22:48:51.638762896 +0000 UTC m=+0.158733814 container died bd4fcfee3f63124285a9a535510ca6768749c4dad4ee75bfb3cad376f820c892 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, RELEASE=squid-323d23d, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-28T22:48:52.300 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:48:52 smithi172 podman[73451]: 2024-03-28 22:48:52.052214289 +0000 UTC m=+0.572185201 container cleanup bd4fcfee3f63124285a9a535510ca6768749c4dad4ee75bfb3cad376f820c892 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z, org.label-schema.build-date=20240326, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-323d23d, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-28T22:48:52.300 INFO:journalctl@ceph.mgr.z.smithi172.stdout:Mar 28 22:48:52 smithi172 bash[73451]: ceph-c2276428-ed51-11ee-95d0-87774f69a715-mgr-z 2024-03-28T22:48:52.433 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mgr.z.service' 2024-03-28T22:48:52.471 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:48:52.471 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-28T22:48:52.472 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-28T22:48:52.472 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.0 2024-03-28T22:48:53.015 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 28 22:48:52 smithi043 systemd[1]: Stopping Ceph osd.0 for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:48:53.515 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 28 22:48:53 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-0[44038]: 2024-03-28T22:48:53.076+0000 7f6235800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:48:54.282 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 28 22:48:53 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-0[44038]: 2024-03-28T22:48:53.076+0000 7f6235800700 -1 osd.0 114 *** Got signal Terminated *** 2024-03-28T22:48:54.282 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 28 22:48:53 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-0[44038]: 2024-03-28T22:48:53.076+0000 7f6235800700 -1 osd.0 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T22:48:58.922 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 28 22:48:58 smithi043 podman[104394]: 2024-03-28 22:48:58.549963707 +0000 UTC m=+5.613641566 container died 98cb6dd46f3396b3c912eb10ad3e9c32b35650e20b877a7d86e8c80682f81bc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-323d23d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-28T22:48:59.265 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 28 22:48:59 smithi043 podman[104394]: 2024-03-28 22:48:59.087884314 +0000 UTC m=+6.151562210 container cleanup 98cb6dd46f3396b3c912eb10ad3e9c32b35650e20b877a7d86e8c80682f81bc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-0, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-323d23d) 2024-03-28T22:48:59.265 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 28 22:48:59 smithi043 bash[104394]: ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-0 2024-03-28T22:48:59.765 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 28 22:48:59 smithi043 podman[104412]: 2024-03-28 22:48:59.397866257 +0000 UTC m=+0.827045265 container remove 98cb6dd46f3396b3c912eb10ad3e9c32b35650e20b877a7d86e8c80682f81bc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-323d23d) 2024-03-28T22:49:00.329 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 28 22:49:00 smithi043 podman[104503]: 2024-03-28 22:49:00.164008726 +0000 UTC m=+0.054218420 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:49:00.684 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 28 22:49:00 smithi043 podman[104503]: 2024-03-28 22:49:00.332150482 +0000 UTC m=+0.222360206 container create 48fd4dddf71edafeae4a8070abbe29e981e705635cda6e12ea377ae8895bb130 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, RELEASE=squid-323d23d, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-28T22:49:01.015 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 28 22:49:00 smithi043 podman[104503]: 2024-03-28 22:49:00.686181853 +0000 UTC m=+0.576391584 container init 48fd4dddf71edafeae4a8070abbe29e981e705635cda6e12ea377ae8895bb130 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-323d23d, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-28T22:49:01.016 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 28 22:49:00 smithi043 podman[104503]: 2024-03-28 22:49:00.700673166 +0000 UTC m=+0.590882871 container start 48fd4dddf71edafeae4a8070abbe29e981e705635cda6e12ea377ae8895bb130 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-323d23d, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-28T22:49:01.016 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 28 22:49:00 smithi043 podman[104503]: 2024-03-28 22:49:00.793620754 +0000 UTC m=+0.683830504 container attach 48fd4dddf71edafeae4a8070abbe29e981e705635cda6e12ea377ae8895bb130 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-323d23d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5) 2024-03-28T22:49:01.484 INFO:journalctl@ceph.osd.0.smithi043.stdout:Mar 28 22:49:01 smithi043 podman[104503]: 2024-03-28 22:49:01.131224866 +0000 UTC m=+1.021434553 container died 48fd4dddf71edafeae4a8070abbe29e981e705635cda6e12ea377ae8895bb130 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-323d23d) 2024-03-28T22:49:01.872 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.0.service' 2024-03-28T22:49:01.997 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:49:01.998 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-28T22:49:01.998 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-28T22:49:01.998 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.1 2024-03-28T22:49:02.320 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 28 22:49:02 smithi082 systemd[1]: Stopping Ceph osd.1 for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:49:02.698 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 28 22:49:02 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-1[41055]: 2024-03-28T22:49:02.322+0000 7f75109ec700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:49:02.699 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 28 22:49:02 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-1[41055]: 2024-03-28T22:49:02.322+0000 7f75109ec700 -1 osd.1 114 *** Got signal Terminated *** 2024-03-28T22:49:02.699 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 28 22:49:02 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-1[41055]: 2024-03-28T22:49:02.322+0000 7f75109ec700 -1 osd.1 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T22:49:08.162 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 28 22:49:07 smithi082 podman[73974]: 2024-03-28 22:49:07.897804883 +0000 UTC m=+5.710561831 container died d4e635bc71ba0aae16adaf21813ef47b8a3ff20db7816c9cf02a01f7a674abc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-1, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-28T22:49:08.445 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 28 22:49:08 smithi082 podman[73974]: 2024-03-28 22:49:08.256035012 +0000 UTC m=+6.068791953 container cleanup d4e635bc71ba0aae16adaf21813ef47b8a3ff20db7816c9cf02a01f7a674abc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, RELEASE=squid-323d23d) 2024-03-28T22:49:08.446 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 28 22:49:08 smithi082 bash[73974]: ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-1 2024-03-28T22:49:08.698 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 28 22:49:08 smithi082 podman[73991]: 2024-03-28 22:49:08.449160149 +0000 UTC m=+0.554028409 container remove d4e635bc71ba0aae16adaf21813ef47b8a3ff20db7816c9cf02a01f7a674abc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-1, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, RELEASE=squid-323d23d, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-28T22:49:09.150 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 28 22:49:08 smithi082 podman[74086]: 2024-03-28 22:49:08.769345974 +0000 UTC m=+0.017460221 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:49:09.150 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 28 22:49:08 smithi082 podman[74086]: 2024-03-28 22:49:08.899973866 +0000 UTC m=+0.148088084 container create ddd4f0f76cd0e4ddf0d1471ba4b693f98fe658c5e3b10a624c4912a60262a4c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-323d23d, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0) 2024-03-28T22:49:09.150 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 28 22:49:09 smithi082 podman[74086]: 2024-03-28 22:49:09.153034397 +0000 UTC m=+0.401148630 container init ddd4f0f76cd0e4ddf0d1471ba4b693f98fe658c5e3b10a624c4912a60262a4c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240326, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-323d23d, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-28T22:49:09.442 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 28 22:49:09 smithi082 podman[74086]: 2024-03-28 22:49:09.156491119 +0000 UTC m=+0.404605349 container start ddd4f0f76cd0e4ddf0d1471ba4b693f98fe658c5e3b10a624c4912a60262a4c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240326, GIT_CLEAN=True) 2024-03-28T22:49:09.442 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 28 22:49:09 smithi082 podman[74086]: 2024-03-28 22:49:09.252681699 +0000 UTC m=+0.500795939 container attach ddd4f0f76cd0e4ddf0d1471ba4b693f98fe658c5e3b10a624c4912a60262a4c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-1-deactivate, RELEASE=squid-323d23d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-28T22:49:09.698 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 28 22:49:09 smithi082 podman[74086]: 2024-03-28 22:49:09.445645525 +0000 UTC m=+0.693759756 container died ddd4f0f76cd0e4ddf0d1471ba4b693f98fe658c5e3b10a624c4912a60262a4c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-323d23d, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-28T22:49:10.136 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.1.service' 2024-03-28T22:49:10.179 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:49:10.179 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-28T22:49:10.180 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-28T22:49:10.180 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.2 2024-03-28T22:49:10.511 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 28 22:49:10 smithi172 systemd[1]: Stopping Ceph osd.2 for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:49:10.895 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 28 22:49:10 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-2[41722]: 2024-03-28T22:49:10.513+0000 7feed9ace700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:49:10.895 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 28 22:49:10 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-2[41722]: 2024-03-28T22:49:10.513+0000 7feed9ace700 -1 osd.2 114 *** Got signal Terminated *** 2024-03-28T22:49:10.895 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 28 22:49:10 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-2[41722]: 2024-03-28T22:49:10.513+0000 7feed9ace700 -1 osd.2 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T22:49:16.362 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 28 22:49:16 smithi172 podman[73576]: 2024-03-28 22:49:16.057464708 +0000 UTC m=+5.678228842 container died 6ff01883a429a79c45477f12446ca20fc62c0461e8926bd65becd431df3e02cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-2, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-323d23d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-28T22:49:16.645 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 28 22:49:16 smithi172 podman[73576]: 2024-03-28 22:49:16.472970116 +0000 UTC m=+6.093734250 container cleanup 6ff01883a429a79c45477f12446ca20fc62c0461e8926bd65becd431df3e02cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-323d23d, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-28T22:49:16.646 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 28 22:49:16 smithi172 bash[73576]: ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-2 2024-03-28T22:49:17.131 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 28 22:49:16 smithi172 podman[73594]: 2024-03-28 22:49:16.724707583 +0000 UTC m=+0.666243396 container remove 6ff01883a429a79c45477f12446ca20fc62c0461e8926bd65becd431df3e02cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-323d23d, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux ) 2024-03-28T22:49:17.395 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 28 22:49:17 smithi172 podman[73684]: 2024-03-28 22:49:17.037729583 +0000 UTC m=+0.017974760 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:49:17.395 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 28 22:49:17 smithi172 podman[73684]: 2024-03-28 22:49:17.167369158 +0000 UTC m=+0.147614364 container create 7947d2cbf227348e86e89fdd17ca7a44a585999909fc8d73a73c3eef529517bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-323d23d, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-28T22:49:17.734 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 28 22:49:17 smithi172 podman[73684]: 2024-03-28 22:49:17.445430116 +0000 UTC m=+0.425675279 container init 7947d2cbf227348e86e89fdd17ca7a44a585999909fc8d73a73c3eef529517bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-323d23d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2) 2024-03-28T22:49:17.735 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 28 22:49:17 smithi172 podman[73684]: 2024-03-28 22:49:17.44881528 +0000 UTC m=+0.429060436 container start 7947d2cbf227348e86e89fdd17ca7a44a585999909fc8d73a73c3eef529517bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, RELEASE=squid-323d23d, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5) 2024-03-28T22:49:17.735 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 28 22:49:17 smithi172 podman[73684]: 2024-03-28 22:49:17.537468155 +0000 UTC m=+0.517713313 container attach 7947d2cbf227348e86e89fdd17ca7a44a585999909fc8d73a73c3eef529517bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-323d23d, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T22:49:17.997 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 28 22:49:17 smithi172 podman[73684]: 2024-03-28 22:49:17.737987974 +0000 UTC m=+0.718233146 container died 7947d2cbf227348e86e89fdd17ca7a44a585999909fc8d73a73c3eef529517bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-2-deactivate, RELEASE=squid-323d23d, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326) 2024-03-28T22:49:18.285 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.2.service' 2024-03-28T22:49:18.328 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:49:18.329 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-28T22:49:18.329 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-28T22:49:18.329 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.3 2024-03-28T22:49:18.767 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 28 22:49:18 smithi043 systemd[1]: Stopping Ceph osd.3 for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:49:19.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:18 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:18.834+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:19.265 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 28 22:49:18 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-3[50541]: 2024-03-28T22:49:18.980+0000 7ff871400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:49:19.266 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 28 22:49:18 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-3[50541]: 2024-03-28T22:49:18.980+0000 7ff871400700 -1 osd.3 114 *** Got signal Terminated *** 2024-03-28T22:49:19.266 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 28 22:49:18 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-3[50541]: 2024-03-28T22:49:18.980+0000 7ff871400700 -1 osd.3 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T22:49:19.464 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:19 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:19.169+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.764339+0000 front 2024-03-28T22:48:52.763956+0000 (oldest deadline 2024-03-28T22:49:18.663743+0000) 2024-03-28T22:49:20.187 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:19 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:19.814+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:20.265 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 28 22:49:19 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-3[50541]: 2024-03-28T22:49:19.938+0000 7ff867c00700 -1 osd.3 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:55.178136+0000 front 2024-03-28T22:48:55.178824+0000 (oldest deadline 2024-03-28T22:49:19.779397+0000) 2024-03-28T22:49:20.645 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:20 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:20.196+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.764339+0000 front 2024-03-28T22:48:52.763956+0000 (oldest deadline 2024-03-28T22:49:18.663743+0000) 2024-03-28T22:49:21.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:20 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:20.845+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:21.265 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 28 22:49:20 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-3[50541]: 2024-03-28T22:49:20.933+0000 7ff867c00700 -1 osd.3 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:55.178136+0000 front 2024-03-28T22:48:55.178824+0000 (oldest deadline 2024-03-28T22:49:19.779397+0000) 2024-03-28T22:49:21.504 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:21 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:21.211+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.764339+0000 front 2024-03-28T22:48:52.763956+0000 (oldest deadline 2024-03-28T22:49:18.663743+0000) 2024-03-28T22:49:21.504 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:21 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:21.338+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:21.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:21 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:21.506+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:22.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:21 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:21.808+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:22.266 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 28 22:49:21 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-3[50541]: 2024-03-28T22:49:21.885+0000 7ff867c00700 -1 osd.3 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:55.178136+0000 front 2024-03-28T22:48:55.178824+0000 (oldest deadline 2024-03-28T22:49:19.779397+0000) 2024-03-28T22:49:22.531 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:22 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:22.227+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.764339+0000 front 2024-03-28T22:48:52.763956+0000 (oldest deadline 2024-03-28T22:49:18.663743+0000) 2024-03-28T22:49:22.531 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:22 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:22.370+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:22.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:22 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:22.533+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:23.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:22 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:22.847+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:23.265 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 28 22:49:22 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-3[50541]: 2024-03-28T22:49:22.866+0000 7ff867c00700 -1 osd.3 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:55.178136+0000 front 2024-03-28T22:48:55.178824+0000 (oldest deadline 2024-03-28T22:49:19.779397+0000) 2024-03-28T22:49:23.493 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:23 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:23.180+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.764339+0000 front 2024-03-28T22:48:52.763956+0000 (oldest deadline 2024-03-28T22:49:18.663743+0000) 2024-03-28T22:49:23.493 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:23 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:23.349+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:23.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:23 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:23.495+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:24.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:23 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:23.861+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:24.265 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 28 22:49:23 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-3[50541]: 2024-03-28T22:49:23.888+0000 7ff867c00700 -1 osd.3 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:55.178136+0000 front 2024-03-28T22:48:55.178824+0000 (oldest deadline 2024-03-28T22:49:19.779397+0000) 2024-03-28T22:49:24.479 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:24 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:24.189+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.764339+0000 front 2024-03-28T22:48:52.763956+0000 (oldest deadline 2024-03-28T22:49:18.663743+0000) 2024-03-28T22:49:24.480 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:24 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:24.366+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:24.721 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 28 22:49:24 smithi043 podman[104618]: 2024-03-28 22:49:24.425029217 +0000 UTC m=+5.591950048 container died 6ed163cc2e4a5e3018dd07c8f1d81edb46a9f8868578f94cfd2681e14bd9a867 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-3, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T22:49:24.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:24 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:24.480+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:25.015 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 28 22:49:24 smithi043 podman[104618]: 2024-03-28 22:49:24.849965173 +0000 UTC m=+6.016886058 container cleanup 6ed163cc2e4a5e3018dd07c8f1d81edb46a9f8868578f94cfd2681e14bd9a867 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-3, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d) 2024-03-28T22:49:25.015 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 28 22:49:24 smithi043 bash[104618]: ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-3 2024-03-28T22:49:25.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:24 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:24.848+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:25.310 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 28 22:49:25 smithi043 podman[104637]: 2024-03-28 22:49:25.168253706 +0000 UTC m=+0.736126194 container remove 6ed163cc2e4a5e3018dd07c8f1d81edb46a9f8868578f94cfd2681e14bd9a867 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-3, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-323d23d, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-28T22:49:25.512 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:25 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:25.206+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.764339+0000 front 2024-03-28T22:48:52.763956+0000 (oldest deadline 2024-03-28T22:49:18.663743+0000) 2024-03-28T22:49:25.513 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:25 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:25.363+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:25.766 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:25 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:25.312+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:25.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:25 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:25.514+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:26.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:25 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:25.826+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:26.503 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:26 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:26.288+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:26.504 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 28 22:49:26 smithi043 podman[104727]: 2024-03-28 22:49:26.068198988 +0000 UTC m=+0.036176704 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:49:26.504 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 28 22:49:26 smithi043 podman[104727]: 2024-03-28 22:49:26.211008681 +0000 UTC m=+0.178986434 container create 0dd4dfd16f2233f6b93f8720e1772c630c83bdcdf3a47f42cc815b2987ae2cb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-323d23d, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-28T22:49:26.504 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 28 22:49:26 smithi043 podman[104727]: 2024-03-28 22:49:26.506242642 +0000 UTC m=+0.474220402 container init 0dd4dfd16f2233f6b93f8720e1772c630c83bdcdf3a47f42cc815b2987ae2cb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-323d23d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5) 2024-03-28T22:49:26.540 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:26 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:26.177+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.764339+0000 front 2024-03-28T22:48:52.763956+0000 (oldest deadline 2024-03-28T22:49:18.663743+0000) 2024-03-28T22:49:26.540 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:26 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:26.395+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:26.765 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 28 22:49:26 smithi043 podman[104727]: 2024-03-28 22:49:26.517967861 +0000 UTC m=+0.485945586 container start 0dd4dfd16f2233f6b93f8720e1772c630c83bdcdf3a47f42cc815b2987ae2cb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, RELEASE=squid-323d23d, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-28T22:49:26.765 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 28 22:49:26 smithi043 podman[104727]: 2024-03-28 22:49:26.621906866 +0000 UTC m=+0.589884673 container attach 0dd4dfd16f2233f6b93f8720e1772c630c83bdcdf3a47f42cc815b2987ae2cb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-3-deactivate, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=squid-323d23d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-28T22:49:26.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:26 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:26.542+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:27.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:26 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:26.816+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:27.265 INFO:journalctl@ceph.osd.3.smithi043.stdout:Mar 28 22:49:26 smithi043 podman[104727]: 2024-03-28 22:49:26.971719816 +0000 UTC m=+0.939697576 container died 0dd4dfd16f2233f6b93f8720e1772c630c83bdcdf3a47f42cc815b2987ae2cb2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-323d23d, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, io.buildah.version=1.33.5) 2024-03-28T22:49:27.569 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:27 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:27.197+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.764339+0000 front 2024-03-28T22:48:52.763956+0000 (oldest deadline 2024-03-28T22:49:18.663743+0000) 2024-03-28T22:49:27.570 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:27 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:27.375+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:27.748 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:27 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:27.283+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:27.795 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.3.service' 2024-03-28T22:49:27.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:27 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:27.572+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:27.926 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:49:27.926 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-28T22:49:27.927 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-28T22:49:27.927 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.4 2024-03-28T22:49:28.198 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 28 22:49:27 smithi082 systemd[1]: Stopping Ceph osd.4 for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:49:28.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:27 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:27.856+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:28.547 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:28 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:28.239+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.764339+0000 front 2024-03-28T22:48:52.763956+0000 (oldest deadline 2024-03-28T22:49:18.663743+0000) 2024-03-28T22:49:28.547 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:28 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:28.375+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:28.694 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 28 22:49:28 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-4[48035]: 2024-03-28T22:49:28.240+0000 7f3f1fa6a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:49:28.694 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 28 22:49:28 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-4[48035]: 2024-03-28T22:49:28.240+0000 7f3f1fa6a700 -1 osd.4 114 *** Got signal Terminated *** 2024-03-28T22:49:28.695 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 28 22:49:28 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-4[48035]: 2024-03-28T22:49:28.240+0000 7f3f1fa6a700 -1 osd.4 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T22:49:28.695 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 28 22:49:28 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-4[48035]: 2024-03-28T22:49:28.697+0000 7f3f1c05e700 -1 osd.4 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:06.200654+0000 front 2024-03-28T22:49:06.200629+0000 (oldest deadline 2024-03-28T22:49:27.900745+0000) 2024-03-28T22:49:28.765 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:28 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:28.322+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:28.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:28 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:28.550+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:28.948 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:28 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:28.814+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:28.948 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:28 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:28.814+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:49:29.265 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:28 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:28.942+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:03.036324+0000 front 2024-03-28T22:49:03.036211+0000 (oldest deadline 2024-03-28T22:49:28.336442+0000) 2024-03-28T22:49:29.374 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:29 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:29.191+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.764339+0000 front 2024-03-28T22:48:52.763956+0000 (oldest deadline 2024-03-28T22:49:18.663743+0000) 2024-03-28T22:49:29.645 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:29 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:29.511+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:29.645 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:29 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:29.511+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:49:29.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:29 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:29.376+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:29.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:29 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:29.376+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:49:29.720 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:29 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:29.358+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:29.948 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 28 22:49:29 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-4[48035]: 2024-03-28T22:49:29.668+0000 7f3f1c05e700 -1 osd.4 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:06.200654+0000 front 2024-03-28T22:49:06.200629+0000 (oldest deadline 2024-03-28T22:49:27.900745+0000) 2024-03-28T22:49:29.949 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:29 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:29.854+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:29.949 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:29 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:29.854+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:49:30.015 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:29 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:29.927+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:03.036324+0000 front 2024-03-28T22:49:03.036211+0000 (oldest deadline 2024-03-28T22:49:28.336442+0000) 2024-03-28T22:49:30.460 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:30 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:30.178+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.764339+0000 front 2024-03-28T22:48:52.763956+0000 (oldest deadline 2024-03-28T22:49:18.663743+0000) 2024-03-28T22:49:30.461 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:30 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:30.414+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:30.461 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:30 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:30.414+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:49:30.657 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:30 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:30.395+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:30.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:30 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:30.463+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:30.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:30 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:30.463+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:49:30.948 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 28 22:49:30 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-4[48035]: 2024-03-28T22:49:30.671+0000 7f3f1c05e700 -1 osd.4 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:06.200654+0000 front 2024-03-28T22:49:06.200629+0000 (oldest deadline 2024-03-28T22:49:27.900745+0000) 2024-03-28T22:49:30.949 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:30 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:30.850+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:30.949 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:30 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:30.850+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:49:31.265 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:30 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:30.946+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:03.036324+0000 front 2024-03-28T22:49:03.036211+0000 (oldest deadline 2024-03-28T22:49:28.336442+0000) 2024-03-28T22:49:31.506 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:31 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:31.202+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.764339+0000 front 2024-03-28T22:48:52.763956+0000 (oldest deadline 2024-03-28T22:49:18.663743+0000) 2024-03-28T22:49:31.507 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:31 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:31.436+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:31.507 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:31 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:31.436+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:49:31.766 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:31 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:31.391+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:31.767 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:31 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:31.391+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:49:31.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:31 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:31.509+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:31.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:31 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:31.509+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:49:31.948 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 28 22:49:31 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-4[48035]: 2024-03-28T22:49:31.692+0000 7f3f1c05e700 -1 osd.4 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:06.200654+0000 front 2024-03-28T22:49:06.200629+0000 (oldest deadline 2024-03-28T22:49:27.900745+0000) 2024-03-28T22:49:31.948 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:31 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:31.832+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:31.949 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:31 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:31.832+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:49:32.265 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:31 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:31.907+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:03.036324+0000 front 2024-03-28T22:49:03.036211+0000 (oldest deadline 2024-03-28T22:49:28.336442+0000) 2024-03-28T22:49:32.645 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:32 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:32.496+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:32.645 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:32 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:32.496+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:49:32.645 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:32 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:32.433+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:32.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:32 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:32.433+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:49:32.646 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:32 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:32.249+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.764339+0000 front 2024-03-28T22:48:52.763956+0000 (oldest deadline 2024-03-28T22:49:18.663743+0000) 2024-03-28T22:49:32.646 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:32 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:32.249+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:06.765925+0000 front 2024-03-28T22:49:06.765944+0000 (oldest deadline 2024-03-28T22:49:31.465969+0000) 2024-03-28T22:49:32.765 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:32 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:32.404+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:32.765 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:32 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:32.404+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:49:33.198 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 28 22:49:32 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-4[48035]: 2024-03-28T22:49:32.728+0000 7f3f1c05e700 -1 osd.4 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:06.200654+0000 front 2024-03-28T22:49:06.200629+0000 (oldest deadline 2024-03-28T22:49:27.900745+0000) 2024-03-28T22:49:33.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:32 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:32.870+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:33.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:32 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:32.870+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:49:33.265 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:32 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:32.894+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:03.036324+0000 front 2024-03-28T22:49:03.036211+0000 (oldest deadline 2024-03-28T22:49:28.336442+0000) 2024-03-28T22:49:33.541 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:33 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:33.384+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:33.542 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:33 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:33.384+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:49:33.542 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:33 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:33.287+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.764339+0000 front 2024-03-28T22:48:52.763956+0000 (oldest deadline 2024-03-28T22:49:18.663743+0000) 2024-03-28T22:49:33.542 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:33 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:33.287+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:06.765925+0000 front 2024-03-28T22:49:06.765944+0000 (oldest deadline 2024-03-28T22:49:31.465969+0000) 2024-03-28T22:49:33.691 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:33 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:33.409+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.580271+0000 front 2024-03-28T22:49:02.580249+0000 (oldest deadline 2024-03-28T22:49:32.580879+0000) 2024-03-28T22:49:33.765 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:33 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:33.398+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:33.765 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:33 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:33.398+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:49:33.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:33 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:33.544+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:33.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:33 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:33.544+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:49:33.948 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 28 22:49:33 smithi082 podman[74191]: 2024-03-28 22:49:33.69455894 +0000 UTC m=+5.577551912 container died 1dc40d4323eacbaddf6423b1e4d101913f1ce9e248a57d53403e881f93f5cd60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-4, GIT_CLEAN=True, RELEASE=squid-323d23d, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-28T22:49:33.948 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:33 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:33.846+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:33.948 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:33 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:33.846+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:49:34.265 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:33 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:33.911+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:03.036324+0000 front 2024-03-28T22:49:03.036211+0000 (oldest deadline 2024-03-28T22:49:28.336442+0000) 2024-03-28T22:49:34.321 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 28 22:49:34 smithi082 podman[74191]: 2024-03-28 22:49:34.06586045 +0000 UTC m=+5.948853424 container cleanup 1dc40d4323eacbaddf6423b1e4d101913f1ce9e248a57d53403e881f93f5cd60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-4, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-323d23d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326) 2024-03-28T22:49:34.322 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 28 22:49:34 smithi082 bash[74191]: ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-4 2024-03-28T22:49:34.544 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:34 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:34.383+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:34.545 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:34 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:34.383+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:49:34.545 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:34 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:34.250+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.764339+0000 front 2024-03-28T22:48:52.763956+0000 (oldest deadline 2024-03-28T22:49:18.663743+0000) 2024-03-28T22:49:34.545 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:34 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:34.250+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:06.765925+0000 front 2024-03-28T22:49:06.765944+0000 (oldest deadline 2024-03-28T22:49:31.465969+0000) 2024-03-28T22:49:34.664 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:34 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:34.426+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.580271+0000 front 2024-03-28T22:49:02.580249+0000 (oldest deadline 2024-03-28T22:49:32.580879+0000) 2024-03-28T22:49:34.664 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 28 22:49:34 smithi082 podman[74208]: 2024-03-28 22:49:34.325024878 +0000 UTC m=+0.634003796 container remove 1dc40d4323eacbaddf6423b1e4d101913f1ce9e248a57d53403e881f93f5cd60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-4, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-323d23d, org.label-schema.license=GPLv2) 2024-03-28T22:49:34.765 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:34 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:34.395+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:34.765 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:34 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:34.395+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:49:34.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:34 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:34.547+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:34.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:34 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:34.547+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:49:34.924 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:34 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:34.825+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:34.925 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:34 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:34.825+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:49:34.925 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 28 22:49:34 smithi082 podman[74300]: 2024-03-28 22:49:34.667460423 +0000 UTC m=+0.102464283 container create 25c3db35c29885b9e36c82e33fdf985a53496d5cbd7074564bd1f5113104633b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-323d23d, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, ceph=True) 2024-03-28T22:49:34.925 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 28 22:49:34 smithi082 podman[74300]: 2024-03-28 22:49:34.582612489 +0000 UTC m=+0.017616365 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:49:34.925 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 28 22:49:34 smithi082 podman[74300]: 2024-03-28 22:49:34.928292587 +0000 UTC m=+0.363296454 container init 25c3db35c29885b9e36c82e33fdf985a53496d5cbd7074564bd1f5113104633b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-4-deactivate, RELEASE=squid-323d23d, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, ceph=True) 2024-03-28T22:49:35.198 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 28 22:49:34 smithi082 podman[74300]: 2024-03-28 22:49:34.931570942 +0000 UTC m=+0.366574810 container start 25c3db35c29885b9e36c82e33fdf985a53496d5cbd7074564bd1f5113104633b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-323d23d, CEPH_POINT_RELEASE=) 2024-03-28T22:49:35.198 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 28 22:49:35 smithi082 podman[74300]: 2024-03-28 22:49:35.019943646 +0000 UTC m=+0.454947522 container attach 25c3db35c29885b9e36c82e33fdf985a53496d5cbd7074564bd1f5113104633b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-323d23d, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-28T22:49:35.265 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:34 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:34.902+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:03.036324+0000 front 2024-03-28T22:49:03.036211+0000 (oldest deadline 2024-03-28T22:49:28.336442+0000) 2024-03-28T22:49:35.473 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 28 22:49:35 smithi082 podman[74300]: 2024-03-28 22:49:35.217888191 +0000 UTC m=+0.652892056 container died 25c3db35c29885b9e36c82e33fdf985a53496d5cbd7074564bd1f5113104633b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-323d23d, ceph=True) 2024-03-28T22:49:35.540 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:35 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:35.339+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:35.540 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:35 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:35.339+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:49:35.541 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:35 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:35.243+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.764339+0000 front 2024-03-28T22:48:52.763956+0000 (oldest deadline 2024-03-28T22:49:18.663743+0000) 2024-03-28T22:49:35.541 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:35 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:35.243+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:06.765925+0000 front 2024-03-28T22:49:06.765944+0000 (oldest deadline 2024-03-28T22:49:31.465969+0000) 2024-03-28T22:49:35.765 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:35 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:35.397+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:35.765 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:35 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:35.397+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:49:35.808 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:35 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:35.475+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.580271+0000 front 2024-03-28T22:49:02.580249+0000 (oldest deadline 2024-03-28T22:49:32.580879+0000) 2024-03-28T22:49:35.894 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.4.service' 2024-03-28T22:49:35.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:35 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:35.543+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:35.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:35 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:35.543+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:49:35.936 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:49:35.936 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-28T22:49:35.936 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-28T22:49:35.936 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.5 2024-03-28T22:49:36.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:35 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:35.810+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:36.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:35 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:35.810+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:49:36.234 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:35 smithi172 systemd[1]: Stopping Ceph osd.5 for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:49:36.235 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:36 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:36.211+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.764339+0000 front 2024-03-28T22:48:52.763956+0000 (oldest deadline 2024-03-28T22:49:18.663743+0000) 2024-03-28T22:49:36.235 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:36 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:36.211+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:06.765925+0000 front 2024-03-28T22:49:06.765944+0000 (oldest deadline 2024-03-28T22:49:31.465969+0000) 2024-03-28T22:49:36.235 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:36 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:36.236+0000 7fd04a3f6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:49:36.235 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:36 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:36.236+0000 7fd04a3f6700 -1 osd.5 114 *** Got signal Terminated *** 2024-03-28T22:49:36.235 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:36 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:36.236+0000 7fd04a3f6700 -1 osd.5 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T22:49:36.265 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:35 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:35.861+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:03.036324+0000 front 2024-03-28T22:49:03.036211+0000 (oldest deadline 2024-03-28T22:49:28.336442+0000) 2024-03-28T22:49:36.515 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:36 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:36.295+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:36.516 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:36 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:36.295+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:49:36.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:36 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:36.441+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.580271+0000 front 2024-03-28T22:49:02.580249+0000 (oldest deadline 2024-03-28T22:49:32.580879+0000) 2024-03-28T22:49:36.765 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:36 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:36.398+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:36.765 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:36 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:36.398+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:49:36.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:36 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:36.518+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:36.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:36 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:36.518+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:49:37.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:36 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:36.859+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:37.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:36 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:36.859+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:49:37.265 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:36 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:36.811+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:03.036324+0000 front 2024-03-28T22:49:03.036211+0000 (oldest deadline 2024-03-28T22:49:28.336442+0000) 2024-03-28T22:49:37.645 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:37 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:37.478+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:37.645 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:37 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:37.478+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:49:37.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:37 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:37.253+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:37.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:37 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:37.253+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:49:37.646 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:37 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:37.241+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.764339+0000 front 2024-03-28T22:48:52.763956+0000 (oldest deadline 2024-03-28T22:49:18.663743+0000) 2024-03-28T22:49:37.646 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:37 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:37.241+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:06.765925+0000 front 2024-03-28T22:49:06.765944+0000 (oldest deadline 2024-03-28T22:49:31.465969+0000) 2024-03-28T22:49:37.646 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:37 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:37.241+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:11.466374+0000 front 2024-03-28T22:49:11.466356+0000 (oldest deadline 2024-03-28T22:49:36.766538+0000) 2024-03-28T22:49:37.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:37 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:37.400+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.580271+0000 front 2024-03-28T22:49:02.580249+0000 (oldest deadline 2024-03-28T22:49:32.580879+0000) 2024-03-28T22:49:37.765 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:37 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:37.428+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:37.765 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:37 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:37.428+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:49:38.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:37 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:37.868+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:38.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:37 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:37.868+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:49:38.265 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:37 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:37.797+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:03.036324+0000 front 2024-03-28T22:49:03.036211+0000 (oldest deadline 2024-03-28T22:49:28.336442+0000) 2024-03-28T22:49:38.265 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:37 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:37.797+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:13.038851+0000 front 2024-03-28T22:49:13.039078+0000 (oldest deadline 2024-03-28T22:49:37.138524+0000) 2024-03-28T22:49:38.645 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:38 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:38.526+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:38.645 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:38 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:38.526+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:49:38.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:38 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:38.294+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:38.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:38 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:38.294+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:49:38.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:38 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:38.294+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:49:38.646 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:38 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:38.279+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.764339+0000 front 2024-03-28T22:48:52.763956+0000 (oldest deadline 2024-03-28T22:49:18.663743+0000) 2024-03-28T22:49:38.646 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:38 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:38.279+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:06.765925+0000 front 2024-03-28T22:49:06.765944+0000 (oldest deadline 2024-03-28T22:49:31.465969+0000) 2024-03-28T22:49:38.647 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:38 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:38.279+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:11.466374+0000 front 2024-03-28T22:49:11.466356+0000 (oldest deadline 2024-03-28T22:49:36.766538+0000) 2024-03-28T22:49:38.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:38 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:38.437+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.580271+0000 front 2024-03-28T22:49:02.580249+0000 (oldest deadline 2024-03-28T22:49:32.580879+0000) 2024-03-28T22:49:38.764 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:38 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:38.426+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:38.764 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:38 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:38.426+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:49:39.015 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:38 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:38.765+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:03.036324+0000 front 2024-03-28T22:49:03.036211+0000 (oldest deadline 2024-03-28T22:49:28.336442+0000) 2024-03-28T22:49:39.015 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:38 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:38.765+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:13.038851+0000 front 2024-03-28T22:49:13.039078+0000 (oldest deadline 2024-03-28T22:49:37.138524+0000) 2024-03-28T22:49:39.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:38 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:38.862+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:39.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:38 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:38.862+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:49:39.645 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:39 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:39.317+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:39.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:39 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:39.317+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:49:39.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:39 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:39.317+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:49:39.646 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:39 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:39.323+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.764339+0000 front 2024-03-28T22:48:52.763956+0000 (oldest deadline 2024-03-28T22:49:18.663743+0000) 2024-03-28T22:49:39.646 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:39 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:39.323+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:06.765925+0000 front 2024-03-28T22:49:06.765944+0000 (oldest deadline 2024-03-28T22:49:31.465969+0000) 2024-03-28T22:49:39.647 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:39 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:39.323+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:11.466374+0000 front 2024-03-28T22:49:11.466356+0000 (oldest deadline 2024-03-28T22:49:36.766538+0000) 2024-03-28T22:49:39.647 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:39 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:39.498+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:39.647 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:39 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:39.498+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:49:39.647 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:39 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:39.498+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.203317+0000 front 2024-03-28T22:49:15.203271+0000 (oldest deadline 2024-03-28T22:49:38.703429+0000) 2024-03-28T22:49:39.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:39 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:39.404+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.580271+0000 front 2024-03-28T22:49:02.580249+0000 (oldest deadline 2024-03-28T22:49:32.580879+0000) 2024-03-28T22:49:39.715 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:39 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:39.426+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:39.715 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:39 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:39.426+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:49:40.015 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:39 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:39.717+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:03.036324+0000 front 2024-03-28T22:49:03.036211+0000 (oldest deadline 2024-03-28T22:49:28.336442+0000) 2024-03-28T22:49:40.015 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:39 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:39.717+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:13.038851+0000 front 2024-03-28T22:49:13.039078+0000 (oldest deadline 2024-03-28T22:49:37.138524+0000) 2024-03-28T22:49:40.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:39 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:39.838+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:40.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:39 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:39.838+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:49:40.645 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:40 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:40.514+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:40.645 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:40 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:40.514+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:49:40.645 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:40 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:40.514+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.203317+0000 front 2024-03-28T22:49:15.203271+0000 (oldest deadline 2024-03-28T22:49:38.703429+0000) 2024-03-28T22:49:40.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:40 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:40.321+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:40.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:40 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:40.321+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:49:40.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:40 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:40.321+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:49:40.646 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:40 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:40.362+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.764339+0000 front 2024-03-28T22:48:52.763956+0000 (oldest deadline 2024-03-28T22:49:18.663743+0000) 2024-03-28T22:49:40.646 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:40 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:40.362+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:06.765925+0000 front 2024-03-28T22:49:06.765944+0000 (oldest deadline 2024-03-28T22:49:31.465969+0000) 2024-03-28T22:49:40.647 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:40 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5[48270]: 2024-03-28T22:49:40.362+0000 7fd0469ea700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:11.466374+0000 front 2024-03-28T22:49:11.466356+0000 (oldest deadline 2024-03-28T22:49:36.766538+0000) 2024-03-28T22:49:40.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:40 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:40.404+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.580271+0000 front 2024-03-28T22:49:02.580249+0000 (oldest deadline 2024-03-28T22:49:32.580879+0000) 2024-03-28T22:49:40.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:40 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:40.404+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.881952+0000 front 2024-03-28T22:49:14.881972+0000 (oldest deadline 2024-03-28T22:49:39.582177+0000) 2024-03-28T22:49:40.733 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:40 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:40.473+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:40.733 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:40 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:40.473+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:49:41.015 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:40 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:40.735+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:03.036324+0000 front 2024-03-28T22:49:03.036211+0000 (oldest deadline 2024-03-28T22:49:28.336442+0000) 2024-03-28T22:49:41.015 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:40 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:40.735+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:13.038851+0000 front 2024-03-28T22:49:13.039078+0000 (oldest deadline 2024-03-28T22:49:37.138524+0000) 2024-03-28T22:49:41.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:40 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:40.812+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:41.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:40 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:40.812+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:49:41.645 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:41 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:41.561+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:41.646 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:41 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:41.561+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:49:41.646 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:41 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:41.561+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.203317+0000 front 2024-03-28T22:49:15.203271+0000 (oldest deadline 2024-03-28T22:49:38.703429+0000) 2024-03-28T22:49:41.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:41 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:41.335+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:41.647 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:41 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:41.335+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:49:41.647 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:41 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:41.335+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:49:41.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:41 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:41.428+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.580271+0000 front 2024-03-28T22:49:02.580249+0000 (oldest deadline 2024-03-28T22:49:32.580879+0000) 2024-03-28T22:49:41.699 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:41 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:41.428+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.881952+0000 front 2024-03-28T22:49:14.881972+0000 (oldest deadline 2024-03-28T22:49:39.582177+0000) 2024-03-28T22:49:41.765 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:41 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:41.688+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:03.036324+0000 front 2024-03-28T22:49:03.036211+0000 (oldest deadline 2024-03-28T22:49:28.336442+0000) 2024-03-28T22:49:41.766 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:41 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:41.688+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:13.038851+0000 front 2024-03-28T22:49:13.039078+0000 (oldest deadline 2024-03-28T22:49:37.138524+0000) 2024-03-28T22:49:41.766 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:41 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:41.506+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:41.766 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:41 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:41.506+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:49:41.766 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:41 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:41.506+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:49:42.118 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:41 smithi172 podman[73801]: 2024-03-28 22:49:41.816942842 +0000 UTC m=+5.687051367 container died bfefd97bf3eb535020b939123a86a562db4b78403a623c858ae5424078d68fea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, RELEASE=squid-323d23d, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-28T22:49:42.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:41 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:41.772+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:42.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:41 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:41.772+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:49:42.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:41 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:41.772+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:49:42.395 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:42 smithi172 podman[73801]: 2024-03-28 22:49:42.220503683 +0000 UTC m=+6.090612197 container cleanup bfefd97bf3eb535020b939123a86a562db4b78403a623c858ae5424078d68fea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-323d23d, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-28T22:49:42.395 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:42 smithi172 bash[73801]: ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5 2024-03-28T22:49:42.396 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:42 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:42.288+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:42.396 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:42 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:42.288+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:49:42.396 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:42 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:42.288+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:49:42.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:42 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:42.420+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.580271+0000 front 2024-03-28T22:49:02.580249+0000 (oldest deadline 2024-03-28T22:49:32.580879+0000) 2024-03-28T22:49:42.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:42 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:42.420+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.881952+0000 front 2024-03-28T22:49:14.881972+0000 (oldest deadline 2024-03-28T22:49:39.582177+0000) 2024-03-28T22:49:42.756 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:42 smithi172 podman[73816]: 2024-03-28 22:49:42.447003742 +0000 UTC m=+0.630214719 container remove bfefd97bf3eb535020b939123a86a562db4b78403a623c858ae5424078d68fea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-323d23d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=) 2024-03-28T22:49:42.756 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:42 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:42.581+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:42.756 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:42 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:42.581+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:49:42.757 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:42 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:42.581+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.203317+0000 front 2024-03-28T22:49:15.203271+0000 (oldest deadline 2024-03-28T22:49:38.703429+0000) 2024-03-28T22:49:43.015 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:42 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:42.706+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:03.036324+0000 front 2024-03-28T22:49:03.036211+0000 (oldest deadline 2024-03-28T22:49:28.336442+0000) 2024-03-28T22:49:43.015 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:42 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:42.706+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:13.038851+0000 front 2024-03-28T22:49:13.039078+0000 (oldest deadline 2024-03-28T22:49:37.138524+0000) 2024-03-28T22:49:43.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:42 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:42.551+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:43.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:42 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:42.551+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:49:43.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:42 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:42.551+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:49:43.055 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:42 smithi172 podman[73912]: 2024-03-28 22:49:42.662755389 +0000 UTC m=+0.018068117 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:49:43.056 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:42 smithi172 podman[73912]: 2024-03-28 22:49:42.77304919 +0000 UTC m=+0.128361962 container create ac452e3c9c282c63b6ff7cead86eb045be3999d3b053df3024761ff3e1fe7367 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-323d23d, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-28T22:49:43.056 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:43 smithi172 podman[73912]: 2024-03-28 22:49:43.058857243 +0000 UTC m=+0.414169979 container init ac452e3c9c282c63b6ff7cead86eb045be3999d3b053df3024761ff3e1fe7367 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240326, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-323d23d) 2024-03-28T22:49:43.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:42 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:42.786+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:43.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:42 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:42.786+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:49:43.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:42 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:42.786+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:49:43.319 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:43 smithi172 podman[73912]: 2024-03-28 22:49:43.062226948 +0000 UTC m=+0.417539681 container start ac452e3c9c282c63b6ff7cead86eb045be3999d3b053df3024761ff3e1fe7367 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-323d23d, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-28T22:49:43.319 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:43 smithi172 podman[73912]: 2024-03-28 22:49:43.150605524 +0000 UTC m=+0.505918267 container attach ac452e3c9c282c63b6ff7cead86eb045be3999d3b053df3024761ff3e1fe7367 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-323d23d, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD) 2024-03-28T22:49:43.645 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 28 22:49:43 smithi172 podman[73912]: 2024-03-28 22:49:43.357862138 +0000 UTC m=+0.713174872 container died ac452e3c9c282c63b6ff7cead86eb045be3999d3b053df3024761ff3e1fe7367 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, RELEASE=squid-323d23d, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-28T22:49:43.645 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:43 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:43.568+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:43.646 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:43 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:43.568+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:49:43.646 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:43 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:43.568+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.203317+0000 front 2024-03-28T22:49:15.203271+0000 (oldest deadline 2024-03-28T22:49:38.703429+0000) 2024-03-28T22:49:43.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:43 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:43.321+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:43.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:43 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:43.321+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:49:43.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:43 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:43.321+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:49:43.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:43 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:43.433+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.580271+0000 front 2024-03-28T22:49:02.580249+0000 (oldest deadline 2024-03-28T22:49:32.580879+0000) 2024-03-28T22:49:43.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:43 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:43.433+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.881952+0000 front 2024-03-28T22:49:14.881972+0000 (oldest deadline 2024-03-28T22:49:39.582177+0000) 2024-03-28T22:49:43.765 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:43 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:43.720+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:03.036324+0000 front 2024-03-28T22:49:03.036211+0000 (oldest deadline 2024-03-28T22:49:28.336442+0000) 2024-03-28T22:49:43.765 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:43 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:43.720+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:13.038851+0000 front 2024-03-28T22:49:13.039078+0000 (oldest deadline 2024-03-28T22:49:37.138524+0000) 2024-03-28T22:49:43.765 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:43 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:43.512+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:43.766 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:43 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:43.512+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:49:43.766 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:43 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:43.512+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:49:44.019 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.5.service' 2024-03-28T22:49:44.060 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:49:44.060 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-28T22:49:44.060 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-28T22:49:44.060 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.6 2024-03-28T22:49:44.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:43 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:43.791+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:44.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:43 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:43.791+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:49:44.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:43 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:43.791+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:49:44.515 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:44 smithi043 systemd[1]: Stopping Ceph osd.6 for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:49:44.645 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:44 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:44.586+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:44.645 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:44 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:44.586+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:49:44.645 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:44 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:44.586+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.203317+0000 front 2024-03-28T22:49:15.203271+0000 (oldest deadline 2024-03-28T22:49:38.703429+0000) 2024-03-28T22:49:44.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:44 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:44.357+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:44.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:44 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:44.357+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:49:44.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:44 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:44.357+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:49:44.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:44 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:44.450+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.580271+0000 front 2024-03-28T22:49:02.580249+0000 (oldest deadline 2024-03-28T22:49:32.580879+0000) 2024-03-28T22:49:44.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:44 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:44.450+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.881952+0000 front 2024-03-28T22:49:14.881972+0000 (oldest deadline 2024-03-28T22:49:39.582177+0000) 2024-03-28T22:49:45.015 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:44 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:44.681+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:03.036324+0000 front 2024-03-28T22:49:03.036211+0000 (oldest deadline 2024-03-28T22:49:28.336442+0000) 2024-03-28T22:49:45.015 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:44 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:44.681+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:13.038851+0000 front 2024-03-28T22:49:13.039078+0000 (oldest deadline 2024-03-28T22:49:37.138524+0000) 2024-03-28T22:49:45.016 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:44 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:44.694+0000 7efe01800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:49:45.016 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:44 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:44.694+0000 7efe01800700 -1 osd.6 114 *** Got signal Terminated *** 2024-03-28T22:49:45.016 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:44 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:44.694+0000 7efe01800700 -1 osd.6 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T22:49:45.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:44 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:44.530+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:45.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:44 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:44.530+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:49:45.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:44 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:44.530+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:49:45.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:44 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:44.813+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:45.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:44 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:44.813+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:49:45.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:44 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:44.813+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:49:45.645 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:45 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:45.542+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:45.646 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:45 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:45.542+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:49:45.646 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:45 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:45.542+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.203317+0000 front 2024-03-28T22:49:15.203271+0000 (oldest deadline 2024-03-28T22:49:38.703429+0000) 2024-03-28T22:49:45.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:45 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:45.386+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:45.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:45 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:45.386+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:49:45.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:45 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:45.386+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:49:45.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:45 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:45.408+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.580271+0000 front 2024-03-28T22:49:02.580249+0000 (oldest deadline 2024-03-28T22:49:32.580879+0000) 2024-03-28T22:49:45.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:45 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:45.408+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.881952+0000 front 2024-03-28T22:49:14.881972+0000 (oldest deadline 2024-03-28T22:49:39.582177+0000) 2024-03-28T22:49:45.699 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:45 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:45.408+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.684465+0000 front 2024-03-28T22:49:21.684541+0000 (oldest deadline 2024-03-28T22:49:44.584347+0000) 2024-03-28T22:49:45.765 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:45 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:45.659+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:03.036324+0000 front 2024-03-28T22:49:03.036211+0000 (oldest deadline 2024-03-28T22:49:28.336442+0000) 2024-03-28T22:49:45.766 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:45 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:45.659+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:13.038851+0000 front 2024-03-28T22:49:13.039078+0000 (oldest deadline 2024-03-28T22:49:37.138524+0000) 2024-03-28T22:49:45.766 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:45 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:45.659+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:19.440719+0000 front 2024-03-28T22:49:19.441231+0000 (oldest deadline 2024-03-28T22:49:45.340972+0000) 2024-03-28T22:49:45.766 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:45 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:45.515+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:45.766 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:45 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:45.515+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:49:45.767 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:45 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:45.515+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:49:46.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:45 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:45.802+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:46.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:45 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:45.802+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:49:46.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:45 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:45.802+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:49:46.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:46 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:46.368+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.580271+0000 front 2024-03-28T22:49:02.580249+0000 (oldest deadline 2024-03-28T22:49:32.580879+0000) 2024-03-28T22:49:46.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:46 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:46.368+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.881952+0000 front 2024-03-28T22:49:14.881972+0000 (oldest deadline 2024-03-28T22:49:39.582177+0000) 2024-03-28T22:49:46.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:46 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:46.368+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.684465+0000 front 2024-03-28T22:49:21.684541+0000 (oldest deadline 2024-03-28T22:49:44.584347+0000) 2024-03-28T22:49:46.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:46 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:46.576+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:46.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:46 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:46.576+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:49:46.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:46 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:46.576+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.203317+0000 front 2024-03-28T22:49:15.203271+0000 (oldest deadline 2024-03-28T22:49:38.703429+0000) 2024-03-28T22:49:46.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:46 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:46.405+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:46.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:46 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:46.405+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:49:46.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:46 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:46.405+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:49:47.015 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:46 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:46.665+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:03.036324+0000 front 2024-03-28T22:49:03.036211+0000 (oldest deadline 2024-03-28T22:49:28.336442+0000) 2024-03-28T22:49:47.015 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:46 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:46.665+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:13.038851+0000 front 2024-03-28T22:49:13.039078+0000 (oldest deadline 2024-03-28T22:49:37.138524+0000) 2024-03-28T22:49:47.015 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:46 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:46.665+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:19.440719+0000 front 2024-03-28T22:49:19.441231+0000 (oldest deadline 2024-03-28T22:49:45.340972+0000) 2024-03-28T22:49:47.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:46 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:46.523+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:47.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:46 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:46.523+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:49:47.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:46 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:46.523+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:49:47.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:46 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:46.523+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.656864+0000 front 2024-03-28T22:49:21.657045+0000 (oldest deadline 2024-03-28T22:49:45.757121+0000) 2024-03-28T22:49:47.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:46 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:46.763+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:47.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:46 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:46.763+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:49:47.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:46 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:46.763+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:49:47.645 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:47 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:47.539+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:47.645 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:47 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:47.539+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:49:47.645 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:47 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:47.539+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.203317+0000 front 2024-03-28T22:49:15.203271+0000 (oldest deadline 2024-03-28T22:49:38.703429+0000) 2024-03-28T22:49:47.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:47 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:47.362+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:47.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:47 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:47.362+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:49:47.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:47 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:47.362+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:49:47.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:47 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:47.335+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.580271+0000 front 2024-03-28T22:49:02.580249+0000 (oldest deadline 2024-03-28T22:49:32.580879+0000) 2024-03-28T22:49:47.699 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:47 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:47.335+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.881952+0000 front 2024-03-28T22:49:14.881972+0000 (oldest deadline 2024-03-28T22:49:39.582177+0000) 2024-03-28T22:49:47.699 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:47 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:47.335+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.684465+0000 front 2024-03-28T22:49:21.684541+0000 (oldest deadline 2024-03-28T22:49:44.584347+0000) 2024-03-28T22:49:48.015 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:47 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:47.698+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:03.036324+0000 front 2024-03-28T22:49:03.036211+0000 (oldest deadline 2024-03-28T22:49:28.336442+0000) 2024-03-28T22:49:48.015 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:47 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:47.698+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:13.038851+0000 front 2024-03-28T22:49:13.039078+0000 (oldest deadline 2024-03-28T22:49:37.138524+0000) 2024-03-28T22:49:48.015 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:47 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:47.698+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:19.440719+0000 front 2024-03-28T22:49:19.441231+0000 (oldest deadline 2024-03-28T22:49:45.340972+0000) 2024-03-28T22:49:48.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:47 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:47.544+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:48.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:47 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:47.544+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:49:48.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:47 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:47.544+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:49:48.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:47 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:47.544+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.656864+0000 front 2024-03-28T22:49:21.657045+0000 (oldest deadline 2024-03-28T22:49:45.757121+0000) 2024-03-28T22:49:48.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:47 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:47.726+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:48.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:47 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:47.726+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:49:48.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:47 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:47.726+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:49:48.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:47 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:47.726+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:49:48.645 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:48 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:48.525+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:48.645 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:48 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:48.525+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:49:48.645 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:48 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:48.525+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.203317+0000 front 2024-03-28T22:49:15.203271+0000 (oldest deadline 2024-03-28T22:49:38.703429+0000) 2024-03-28T22:49:48.646 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:48 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:48.525+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:23.305205+0000 front 2024-03-28T22:49:23.305481+0000 (oldest deadline 2024-03-28T22:49:48.004899+0000) 2024-03-28T22:49:48.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:48 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:48.374+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:48.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:48 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:48.374+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:49:48.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:48 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:48.374+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:49:48.684 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:48 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:48.294+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.580271+0000 front 2024-03-28T22:49:02.580249+0000 (oldest deadline 2024-03-28T22:49:32.580879+0000) 2024-03-28T22:49:48.684 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:48 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:48.294+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.881952+0000 front 2024-03-28T22:49:14.881972+0000 (oldest deadline 2024-03-28T22:49:39.582177+0000) 2024-03-28T22:49:48.684 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:48 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:48.294+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.684465+0000 front 2024-03-28T22:49:21.684541+0000 (oldest deadline 2024-03-28T22:49:44.584347+0000) 2024-03-28T22:49:48.948 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:48 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:48.687+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:48.949 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:48 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:48.687+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:49:48.949 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:48 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:48.687+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:49:48.949 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:48 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:48.687+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:49:49.015 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:48 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:48.657+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:03.036324+0000 front 2024-03-28T22:49:03.036211+0000 (oldest deadline 2024-03-28T22:49:28.336442+0000) 2024-03-28T22:49:49.015 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:48 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:48.657+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:13.038851+0000 front 2024-03-28T22:49:13.039078+0000 (oldest deadline 2024-03-28T22:49:37.138524+0000) 2024-03-28T22:49:49.016 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:48 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:48.657+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:19.440719+0000 front 2024-03-28T22:49:19.441231+0000 (oldest deadline 2024-03-28T22:49:45.340972+0000) 2024-03-28T22:49:49.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:48 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:48.538+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:49.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:48 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:48.538+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:49:49.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:48 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:48.538+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:49:49.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:48 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:48.538+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.656864+0000 front 2024-03-28T22:49:21.657045+0000 (oldest deadline 2024-03-28T22:49:45.757121+0000) 2024-03-28T22:49:49.645 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:49 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:49.514+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:49.645 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:49 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:49.514+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:49:49.646 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:49 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:49.514+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.203317+0000 front 2024-03-28T22:49:15.203271+0000 (oldest deadline 2024-03-28T22:49:38.703429+0000) 2024-03-28T22:49:49.646 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:49 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:49.514+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:23.305205+0000 front 2024-03-28T22:49:23.305481+0000 (oldest deadline 2024-03-28T22:49:48.004899+0000) 2024-03-28T22:49:49.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:49 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:49.363+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:49.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:49 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:49.363+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:49:49.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:49 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:49.363+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:49:49.679 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:49 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:49.294+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.580271+0000 front 2024-03-28T22:49:02.580249+0000 (oldest deadline 2024-03-28T22:49:32.580879+0000) 2024-03-28T22:49:49.680 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:49 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:49.294+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.881952+0000 front 2024-03-28T22:49:14.881972+0000 (oldest deadline 2024-03-28T22:49:39.582177+0000) 2024-03-28T22:49:49.680 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:49 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:49.294+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.684465+0000 front 2024-03-28T22:49:21.684541+0000 (oldest deadline 2024-03-28T22:49:44.584347+0000) 2024-03-28T22:49:49.948 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:49 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:49.683+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:49.948 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:49 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:49.683+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:49:49.948 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:49 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:49.683+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:49:49.948 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:49 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:49.683+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:49:50.015 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:49 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:49.659+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:03.036324+0000 front 2024-03-28T22:49:03.036211+0000 (oldest deadline 2024-03-28T22:49:28.336442+0000) 2024-03-28T22:49:50.015 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:49 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:49.659+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:13.038851+0000 front 2024-03-28T22:49:13.039078+0000 (oldest deadline 2024-03-28T22:49:37.138524+0000) 2024-03-28T22:49:50.015 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:49 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6[57270]: 2024-03-28T22:49:49.659+0000 7efdfd000700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:19.440719+0000 front 2024-03-28T22:49:19.441231+0000 (oldest deadline 2024-03-28T22:49:45.340972+0000) 2024-03-28T22:49:50.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:49 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:49.529+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:50.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:49 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:49.529+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:49:50.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:49 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:49.529+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:49:50.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:49 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:49.529+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.656864+0000 front 2024-03-28T22:49:21.657045+0000 (oldest deadline 2024-03-28T22:49:45.757121+0000) 2024-03-28T22:49:50.490 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:50 smithi043 podman[104844]: 2024-03-28 22:49:50.19378064 +0000 UTC m=+5.624492974 container died a42963d43b0dfac6732a7d0f5004d675ed27913ed7119ad0044a364d80217221 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-323d23d, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-28T22:49:50.645 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:50 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:50.530+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:50.645 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:50 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:50.530+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:49:50.646 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:50 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:50.530+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.203317+0000 front 2024-03-28T22:49:15.203271+0000 (oldest deadline 2024-03-28T22:49:38.703429+0000) 2024-03-28T22:49:50.646 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:50 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:50.530+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:23.305205+0000 front 2024-03-28T22:49:23.305481+0000 (oldest deadline 2024-03-28T22:49:48.004899+0000) 2024-03-28T22:49:50.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:50 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:50.358+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:50.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:50 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:50.358+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:49:50.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:50 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:50.358+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:49:50.669 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:50 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:50.308+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.580271+0000 front 2024-03-28T22:49:02.580249+0000 (oldest deadline 2024-03-28T22:49:32.580879+0000) 2024-03-28T22:49:50.669 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:50 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:50.308+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.881952+0000 front 2024-03-28T22:49:14.881972+0000 (oldest deadline 2024-03-28T22:49:39.582177+0000) 2024-03-28T22:49:50.670 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:50 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:50.308+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.684465+0000 front 2024-03-28T22:49:21.684541+0000 (oldest deadline 2024-03-28T22:49:44.584347+0000) 2024-03-28T22:49:50.765 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:50 smithi043 podman[104844]: 2024-03-28 22:49:50.620711476 +0000 UTC m=+6.051423840 container cleanup a42963d43b0dfac6732a7d0f5004d675ed27913ed7119ad0044a364d80217221 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326) 2024-03-28T22:49:50.765 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:50 smithi043 bash[104844]: ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6 2024-03-28T22:49:50.765 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:50 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:50.545+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:50.766 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:50 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:50.545+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:49:50.766 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:50 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:50.545+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:49:50.766 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:50 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:50.545+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.656864+0000 front 2024-03-28T22:49:21.657045+0000 (oldest deadline 2024-03-28T22:49:45.757121+0000) 2024-03-28T22:49:50.948 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:50 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:50.672+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:50.948 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:50 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:50.672+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:49:50.949 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:50 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:50.672+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:49:50.949 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:50 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:50.672+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:49:51.265 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:50 smithi043 podman[104863]: 2024-03-28 22:49:50.838865674 +0000 UTC m=+0.635192547 container remove a42963d43b0dfac6732a7d0f5004d675ed27913ed7119ad0044a364d80217221 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=squid-323d23d, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-28T22:49:51.645 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:51 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:51.542+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:51.646 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:51 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:51.542+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:49:51.646 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:51 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:51.542+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.203317+0000 front 2024-03-28T22:49:15.203271+0000 (oldest deadline 2024-03-28T22:49:38.703429+0000) 2024-03-28T22:49:51.646 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:51 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:51.542+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:23.305205+0000 front 2024-03-28T22:49:23.305481+0000 (oldest deadline 2024-03-28T22:49:48.004899+0000) 2024-03-28T22:49:51.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:51 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:51.393+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:51.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:51 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:51.393+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:49:51.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:51 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:51.393+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:49:51.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:51 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:51.287+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.580271+0000 front 2024-03-28T22:49:02.580249+0000 (oldest deadline 2024-03-28T22:49:32.580879+0000) 2024-03-28T22:49:51.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:51 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:51.287+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.881952+0000 front 2024-03-28T22:49:14.881972+0000 (oldest deadline 2024-03-28T22:49:39.582177+0000) 2024-03-28T22:49:51.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:51 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:51.287+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.684465+0000 front 2024-03-28T22:49:21.684541+0000 (oldest deadline 2024-03-28T22:49:44.584347+0000) 2024-03-28T22:49:51.857 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:51 smithi043 podman[104955]: 2024-03-28 22:49:51.559897542 +0000 UTC m=+0.033189371 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:49:51.857 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:51 smithi043 podman[104955]: 2024-03-28 22:49:51.689802419 +0000 UTC m=+0.163094282 container create bc1ff1383917c34fe36321ce29640601abcb40f6ba441865151c4e61e125a0aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-323d23d, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-28T22:49:51.858 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:51 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:51.543+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:51.858 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:51 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:51.543+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:49:51.858 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:51 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:51.543+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:49:51.858 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:51 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:51.543+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.656864+0000 front 2024-03-28T22:49:21.657045+0000 (oldest deadline 2024-03-28T22:49:45.757121+0000) 2024-03-28T22:49:52.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:51 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:51.705+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:52.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:51 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:51.705+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:49:52.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:51 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:51.705+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:49:52.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:51 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:51.705+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:49:52.239 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:51 smithi043 podman[104955]: 2024-03-28 22:49:51.985922799 +0000 UTC m=+0.459214702 container init bc1ff1383917c34fe36321ce29640601abcb40f6ba441865151c4e61e125a0aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6-deactivate, RELEASE=squid-323d23d, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-28T22:49:52.239 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:51 smithi043 podman[104955]: 2024-03-28 22:49:51.998902371 +0000 UTC m=+0.472194258 container start bc1ff1383917c34fe36321ce29640601abcb40f6ba441865151c4e61e125a0aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-323d23d, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5) 2024-03-28T22:49:52.239 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:52 smithi043 podman[104955]: 2024-03-28 22:49:52.242089754 +0000 UTC m=+0.715381590 container attach bc1ff1383917c34fe36321ce29640601abcb40f6ba441865151c4e61e125a0aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=squid-323d23d, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-28T22:49:52.515 INFO:journalctl@ceph.osd.6.smithi043.stdout:Mar 28 22:49:52 smithi043 podman[104955]: 2024-03-28 22:49:52.435726077 +0000 UTC m=+0.909017903 container died bc1ff1383917c34fe36321ce29640601abcb40f6ba441865151c4e61e125a0aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-323d23d, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-28T22:49:52.645 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:52 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:52.517+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:52.645 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:52 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:52.517+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:49:52.646 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:52 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:52.517+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.203317+0000 front 2024-03-28T22:49:15.203271+0000 (oldest deadline 2024-03-28T22:49:38.703429+0000) 2024-03-28T22:49:52.646 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:52 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:52.517+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:23.305205+0000 front 2024-03-28T22:49:23.305481+0000 (oldest deadline 2024-03-28T22:49:48.004899+0000) 2024-03-28T22:49:52.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:52 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:52.387+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:52.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:52 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:52.387+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:49:52.646 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:52 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:52.387+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:49:52.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:52 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:52.242+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.580271+0000 front 2024-03-28T22:49:02.580249+0000 (oldest deadline 2024-03-28T22:49:32.580879+0000) 2024-03-28T22:49:52.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:52 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:52.242+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.881952+0000 front 2024-03-28T22:49:14.881972+0000 (oldest deadline 2024-03-28T22:49:39.582177+0000) 2024-03-28T22:49:52.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:52 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:52.242+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.684465+0000 front 2024-03-28T22:49:21.684541+0000 (oldest deadline 2024-03-28T22:49:44.584347+0000) 2024-03-28T22:49:53.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:52 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:52.577+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:53.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:52 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:52.577+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:49:53.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:52 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:52.577+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:49:53.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:52 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:52.577+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.656864+0000 front 2024-03-28T22:49:21.657045+0000 (oldest deadline 2024-03-28T22:49:45.757121+0000) 2024-03-28T22:49:53.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:52 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:52.740+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:53.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:52 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:52.740+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:49:53.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:52 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:52.740+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:49:53.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:52 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:52.740+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:49:53.324 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.6.service' 2024-03-28T22:49:53.449 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:49:53.450 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-28T22:49:53.450 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-28T22:49:53.450 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.7 2024-03-28T22:49:53.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:53 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:53.257+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.580271+0000 front 2024-03-28T22:49:02.580249+0000 (oldest deadline 2024-03-28T22:49:32.580879+0000) 2024-03-28T22:49:53.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:53 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:53.257+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.881952+0000 front 2024-03-28T22:49:14.881972+0000 (oldest deadline 2024-03-28T22:49:39.582177+0000) 2024-03-28T22:49:53.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:53 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:53.257+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.684465+0000 front 2024-03-28T22:49:21.684541+0000 (oldest deadline 2024-03-28T22:49:44.584347+0000) 2024-03-28T22:49:53.699 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:53 smithi082 systemd[1]: Stopping Ceph osd.7 for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:49:53.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:53 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:53.563+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:53.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:53 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:53.563+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:49:53.896 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:53 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:53.563+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.203317+0000 front 2024-03-28T22:49:15.203271+0000 (oldest deadline 2024-03-28T22:49:38.703429+0000) 2024-03-28T22:49:53.896 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:53 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:53.563+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:23.305205+0000 front 2024-03-28T22:49:23.305481+0000 (oldest deadline 2024-03-28T22:49:48.004899+0000) 2024-03-28T22:49:53.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:53 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:53.423+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:53.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:53 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:53.423+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:49:53.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:53 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:53.423+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:49:54.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:53 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:53.605+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:54.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:53 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:53.605+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:49:54.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:53 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:53.605+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:49:54.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:53 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:53.605+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.656864+0000 front 2024-03-28T22:49:21.657045+0000 (oldest deadline 2024-03-28T22:49:45.757121+0000) 2024-03-28T22:49:54.198 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:53 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:53.774+0000 7f3b791c1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:49:54.198 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:53 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:53.774+0000 7f3b791c1700 -1 osd.7 114 *** Got signal Terminated *** 2024-03-28T22:49:54.198 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:53 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:53.774+0000 7f3b791c1700 -1 osd.7 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T22:49:54.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:53 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:53.744+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:54.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:53 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:53.744+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:49:54.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:53 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:53.744+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:49:54.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:53 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:53.744+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:49:54.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:54 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:54.231+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.580271+0000 front 2024-03-28T22:49:02.580249+0000 (oldest deadline 2024-03-28T22:49:32.580879+0000) 2024-03-28T22:49:54.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:54 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:54.231+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.881952+0000 front 2024-03-28T22:49:14.881972+0000 (oldest deadline 2024-03-28T22:49:39.582177+0000) 2024-03-28T22:49:54.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:54 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:54.231+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.684465+0000 front 2024-03-28T22:49:21.684541+0000 (oldest deadline 2024-03-28T22:49:44.584347+0000) 2024-03-28T22:49:54.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:54 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:54.613+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:54.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:54 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:54.613+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:49:54.896 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:54 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:54.613+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.203317+0000 front 2024-03-28T22:49:15.203271+0000 (oldest deadline 2024-03-28T22:49:38.703429+0000) 2024-03-28T22:49:54.896 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:54 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:54.613+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:23.305205+0000 front 2024-03-28T22:49:23.305481+0000 (oldest deadline 2024-03-28T22:49:48.004899+0000) 2024-03-28T22:49:54.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:54 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:54.464+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:54.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:54 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:54.464+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:49:54.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:54 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:54.464+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:49:55.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:54 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:54.609+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:55.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:54 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:54.609+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:49:55.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:54 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:54.609+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:49:55.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:54 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:54.609+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.656864+0000 front 2024-03-28T22:49:21.657045+0000 (oldest deadline 2024-03-28T22:49:45.757121+0000) 2024-03-28T22:49:55.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:54 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:54.609+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.458838+0000 front 2024-03-28T22:49:30.458960+0000 (oldest deadline 2024-03-28T22:49:53.959115+0000) 2024-03-28T22:49:55.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:54 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:54.738+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:55.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:54 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:54.738+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:49:55.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:54 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:54.738+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:49:55.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:54 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:54.738+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:49:55.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:55 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:55.247+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.580271+0000 front 2024-03-28T22:49:02.580249+0000 (oldest deadline 2024-03-28T22:49:32.580879+0000) 2024-03-28T22:49:55.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:55 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:55.247+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.881952+0000 front 2024-03-28T22:49:14.881972+0000 (oldest deadline 2024-03-28T22:49:39.582177+0000) 2024-03-28T22:49:55.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:55 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:55.247+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.684465+0000 front 2024-03-28T22:49:21.684541+0000 (oldest deadline 2024-03-28T22:49:44.584347+0000) 2024-03-28T22:49:55.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:55 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:55.643+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:55.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:55 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:55.643+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:49:55.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:55 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:55.643+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.203317+0000 front 2024-03-28T22:49:15.203271+0000 (oldest deadline 2024-03-28T22:49:38.703429+0000) 2024-03-28T22:49:55.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:55 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:55.643+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:23.305205+0000 front 2024-03-28T22:49:23.305481+0000 (oldest deadline 2024-03-28T22:49:48.004899+0000) 2024-03-28T22:49:55.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:55 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:55.468+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:55.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:55 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:55.468+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:49:55.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:55 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:55.468+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:49:55.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:55 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:55.468+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:49:56.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:55 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:55.607+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:56.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:55 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:55.607+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:49:56.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:55 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:55.607+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:49:56.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:55 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:55.607+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.656864+0000 front 2024-03-28T22:49:21.657045+0000 (oldest deadline 2024-03-28T22:49:45.757121+0000) 2024-03-28T22:49:56.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:55 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:55.607+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.458838+0000 front 2024-03-28T22:49:30.458960+0000 (oldest deadline 2024-03-28T22:49:53.959115+0000) 2024-03-28T22:49:56.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:55 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:55.769+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:56.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:55 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:55.769+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:49:56.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:55 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:55.769+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:49:56.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:55 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:55.769+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:49:56.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:55 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:55.769+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:32.267858+0000 front 2024-03-28T22:49:32.267894+0000 (oldest deadline 2024-03-28T22:49:55.167920+0000) 2024-03-28T22:49:56.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:56 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:56.218+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.580271+0000 front 2024-03-28T22:49:02.580249+0000 (oldest deadline 2024-03-28T22:49:32.580879+0000) 2024-03-28T22:49:56.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:56 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:56.218+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.881952+0000 front 2024-03-28T22:49:14.881972+0000 (oldest deadline 2024-03-28T22:49:39.582177+0000) 2024-03-28T22:49:56.699 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:56 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:56.218+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.684465+0000 front 2024-03-28T22:49:21.684541+0000 (oldest deadline 2024-03-28T22:49:44.584347+0000) 2024-03-28T22:49:56.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:56 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:56.611+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:56.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:56 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:56.611+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:49:56.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:56 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:56.611+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.203317+0000 front 2024-03-28T22:49:15.203271+0000 (oldest deadline 2024-03-28T22:49:38.703429+0000) 2024-03-28T22:49:56.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:56 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:56.611+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:23.305205+0000 front 2024-03-28T22:49:23.305481+0000 (oldest deadline 2024-03-28T22:49:48.004899+0000) 2024-03-28T22:49:56.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:56 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:56.511+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:56.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:56 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:56.511+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:49:56.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:56 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:56.511+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:49:56.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:56 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:56.511+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:49:57.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:56 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:56.648+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:57.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:56 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:56.648+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:49:57.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:56 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:56.648+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:49:57.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:56 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:56.648+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.656864+0000 front 2024-03-28T22:49:21.657045+0000 (oldest deadline 2024-03-28T22:49:45.757121+0000) 2024-03-28T22:49:57.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:56 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:56.648+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.458838+0000 front 2024-03-28T22:49:30.458960+0000 (oldest deadline 2024-03-28T22:49:53.959115+0000) 2024-03-28T22:49:57.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:56 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:56.812+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:57.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:56 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:56.812+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:49:57.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:56 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:56.812+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:49:57.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:56 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:56.812+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:49:57.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:56 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:56.812+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:32.267858+0000 front 2024-03-28T22:49:32.267894+0000 (oldest deadline 2024-03-28T22:49:55.167920+0000) 2024-03-28T22:49:57.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:57 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:57.235+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.580271+0000 front 2024-03-28T22:49:02.580249+0000 (oldest deadline 2024-03-28T22:49:32.580879+0000) 2024-03-28T22:49:57.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:57 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:57.235+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.881952+0000 front 2024-03-28T22:49:14.881972+0000 (oldest deadline 2024-03-28T22:49:39.582177+0000) 2024-03-28T22:49:57.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:57 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:57.235+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.684465+0000 front 2024-03-28T22:49:21.684541+0000 (oldest deadline 2024-03-28T22:49:44.584347+0000) 2024-03-28T22:49:57.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:57 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:57.635+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:57.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:57 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:57.635+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:49:57.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:57 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:57.635+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.203317+0000 front 2024-03-28T22:49:15.203271+0000 (oldest deadline 2024-03-28T22:49:38.703429+0000) 2024-03-28T22:49:57.896 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:57 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:57.635+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:23.305205+0000 front 2024-03-28T22:49:23.305481+0000 (oldest deadline 2024-03-28T22:49:48.004899+0000) 2024-03-28T22:49:57.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:57 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:57.497+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:57.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:57 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:57.497+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:49:57.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:57 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:57.497+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:49:57.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:57 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:57.497+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:49:58.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:57 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:57.636+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:58.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:57 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:57.636+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:49:58.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:57 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:57.636+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:49:58.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:57 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:57.636+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.656864+0000 front 2024-03-28T22:49:21.657045+0000 (oldest deadline 2024-03-28T22:49:45.757121+0000) 2024-03-28T22:49:58.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:57 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:57.636+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.458838+0000 front 2024-03-28T22:49:30.458960+0000 (oldest deadline 2024-03-28T22:49:53.959115+0000) 2024-03-28T22:49:58.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:57 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:57.771+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:58.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:57 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:57.771+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:49:58.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:57 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:57.771+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:49:58.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:57 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:57.771+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:49:58.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:57 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:57.771+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:32.267858+0000 front 2024-03-28T22:49:32.267894+0000 (oldest deadline 2024-03-28T22:49:55.167920+0000) 2024-03-28T22:49:58.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:58 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:58.241+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.580271+0000 front 2024-03-28T22:49:02.580249+0000 (oldest deadline 2024-03-28T22:49:32.580879+0000) 2024-03-28T22:49:58.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:58 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:58.241+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.881952+0000 front 2024-03-28T22:49:14.881972+0000 (oldest deadline 2024-03-28T22:49:39.582177+0000) 2024-03-28T22:49:58.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:58 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:58.241+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.684465+0000 front 2024-03-28T22:49:21.684541+0000 (oldest deadline 2024-03-28T22:49:44.584347+0000) 2024-03-28T22:49:58.698 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:58 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7[54925]: 2024-03-28T22:49:58.241+0000 7f3b717ad700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:31.386736+0000 front 2024-03-28T22:49:31.386699+0000 (oldest deadline 2024-03-28T22:49:57.286855+0000) 2024-03-28T22:49:58.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:58 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:58.596+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:58.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:58 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:58.596+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:49:58.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:58 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:58.596+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.203317+0000 front 2024-03-28T22:49:15.203271+0000 (oldest deadline 2024-03-28T22:49:38.703429+0000) 2024-03-28T22:49:58.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:58 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:58.596+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:23.305205+0000 front 2024-03-28T22:49:23.305481+0000 (oldest deadline 2024-03-28T22:49:48.004899+0000) 2024-03-28T22:49:58.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:58 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:58.464+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:58.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:58 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:58.464+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:49:58.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:58 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:58.464+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:49:58.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:58 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:58.464+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:49:58.917 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:58 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:58.655+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:58.917 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:58 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:58.655+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:49:58.918 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:58 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:58.655+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:49:58.918 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:58 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:58.655+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.656864+0000 front 2024-03-28T22:49:21.657045+0000 (oldest deadline 2024-03-28T22:49:45.757121+0000) 2024-03-28T22:49:58.918 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:58 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:58.655+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.458838+0000 front 2024-03-28T22:49:30.458960+0000 (oldest deadline 2024-03-28T22:49:53.959115+0000) 2024-03-28T22:49:59.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:58 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:58.818+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:49:59.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:58 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:58.818+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:49:59.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:58 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:58.818+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:49:59.201 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:58 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:58.818+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:49:59.201 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:58 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:58.818+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:32.267858+0000 front 2024-03-28T22:49:32.267894+0000 (oldest deadline 2024-03-28T22:49:55.167920+0000) 2024-03-28T22:49:59.515 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:59 smithi082 podman[74406]: 2024-03-28 22:49:59.331746806 +0000 UTC m=+5.685247004 container died 8fdba4227919c2c4c98e63fda21496720b2e5a8161a157d2e7b35af88cd3c770 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-323d23d, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326) 2024-03-28T22:49:59.616 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:59 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:59.482+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:49:59.616 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:59 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:59.482+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:49:59.616 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:59 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:59.482+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:49:59.616 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:49:59 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:49:59.482+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:49:59.778 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:59 smithi082 podman[74406]: 2024-03-28 22:49:59.700143251 +0000 UTC m=+6.053643441 container cleanup 8fdba4227919c2c4c98e63fda21496720b2e5a8161a157d2e7b35af88cd3c770 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d, ceph=True, io.buildah.version=1.33.5) 2024-03-28T22:49:59.778 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:49:59 smithi082 bash[74406]: ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7 2024-03-28T22:49:59.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:59 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:59.618+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:49:59.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:59 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:59.618+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:49:59.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:59 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:59.618+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.203317+0000 front 2024-03-28T22:49:15.203271+0000 (oldest deadline 2024-03-28T22:49:38.703429+0000) 2024-03-28T22:49:59.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:49:59 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:49:59.618+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:23.305205+0000 front 2024-03-28T22:49:23.305481+0000 (oldest deadline 2024-03-28T22:49:48.004899+0000) 2024-03-28T22:49:59.944 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:59 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:59.627+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:49:59.945 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:59 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:59.627+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:49:59.945 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:59 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:59.627+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:49:59.945 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:59 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:59.627+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.656864+0000 front 2024-03-28T22:49:21.657045+0000 (oldest deadline 2024-03-28T22:49:45.757121+0000) 2024-03-28T22:49:59.945 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:49:59 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:49:59.627+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.458838+0000 front 2024-03-28T22:49:30.458960+0000 (oldest deadline 2024-03-28T22:49:53.959115+0000) 2024-03-28T22:50:00.028 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:50:00 smithi082 podman[74425]: 2024-03-28 22:50:00.011322285 +0000 UTC m=+0.674904571 container remove 8fdba4227919c2c4c98e63fda21496720b2e5a8161a157d2e7b35af88cd3c770 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-323d23d, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T22:50:00.028 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:59 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:59.786+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:50:00.028 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:59 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:59.786+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:50:00.029 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:59 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:59.786+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:50:00.029 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:59 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:59.786+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:50:00.029 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:49:59 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:49:59.786+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:32.267858+0000 front 2024-03-28T22:49:32.267894+0000 (oldest deadline 2024-03-28T22:49:55.167920+0000) 2024-03-28T22:50:00.686 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:50:00 smithi082 podman[74523]: 2024-03-28 22:50:00.280718506 +0000 UTC m=+0.017371608 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:50:00.687 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:50:00 smithi082 podman[74523]: 2024-03-28 22:50:00.412723016 +0000 UTC m=+0.149376107 container create f6c23b536d2e30a21202a44a855d2155665d66edd345e2d3ed15f8905508923c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-28T22:50:00.687 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:50:00 smithi082 podman[74523]: 2024-03-28 22:50:00.606529785 +0000 UTC m=+0.343182892 container init f6c23b536d2e30a21202a44a855d2155665d66edd345e2d3ed15f8905508923c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-323d23d, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326) 2024-03-28T22:50:00.687 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:50:00 smithi082 podman[74523]: 2024-03-28 22:50:00.609577078 +0000 UTC m=+0.346230171 container start f6c23b536d2e30a21202a44a855d2155665d66edd345e2d3ed15f8905508923c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-323d23d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T22:50:00.687 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:50:00 smithi082 podman[74523]: 2024-03-28 22:50:00.689822803 +0000 UTC m=+0.426475913 container attach f6c23b536d2e30a21202a44a855d2155665d66edd345e2d3ed15f8905508923c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-323d23d, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-28T22:50:00.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:00 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:50:00.606+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:50:00.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:00 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:50:00.606+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:50:00.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:00 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:50:00.606+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.203317+0000 front 2024-03-28T22:49:15.203271+0000 (oldest deadline 2024-03-28T22:49:38.703429+0000) 2024-03-28T22:50:00.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:00 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:50:00.606+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:23.305205+0000 front 2024-03-28T22:49:23.305481+0000 (oldest deadline 2024-03-28T22:49:48.004899+0000) 2024-03-28T22:50:00.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:00 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:00.482+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:50:00.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:00 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:00.482+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:50:00.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:00 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:00.482+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:50:00.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:00 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:00.482+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:50:00.948 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 28 22:50:00 smithi082 podman[74523]: 2024-03-28 22:50:00.90901097 +0000 UTC m=+0.645664062 container died f6c23b536d2e30a21202a44a855d2155665d66edd345e2d3ed15f8905508923c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, RELEASE=squid-323d23d, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-28T22:50:00.948 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:00 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:00.774+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:50:00.949 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:00 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:00.774+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:50:00.949 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:00 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:00.774+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:50:00.949 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:00 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:00.774+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:50:00.949 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:00 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:00.774+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:32.267858+0000 front 2024-03-28T22:49:32.267894+0000 (oldest deadline 2024-03-28T22:49:55.167920+0000) 2024-03-28T22:50:01.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:00 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:00.674+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:50:01.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:00 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:00.674+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:50:01.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:00 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:00.674+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:50:01.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:00 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:00.674+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.656864+0000 front 2024-03-28T22:49:21.657045+0000 (oldest deadline 2024-03-28T22:49:45.757121+0000) 2024-03-28T22:50:01.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:00 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:00.674+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.458838+0000 front 2024-03-28T22:49:30.458960+0000 (oldest deadline 2024-03-28T22:49:53.959115+0000) 2024-03-28T22:50:01.492 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.7.service' 2024-03-28T22:50:01.536 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:50:01.536 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-28T22:50:01.537 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-28T22:50:01.537 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.8 2024-03-28T22:50:01.853 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:01 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:01.481+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:50:01.854 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:01 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:01.481+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:50:01.854 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:01 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:01.482+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:50:01.854 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:01 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:01.482+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:50:01.854 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:01 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:50:01.572+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:50:01.854 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:01 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:50:01.572+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:50:01.854 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:01 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:50:01.572+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.203317+0000 front 2024-03-28T22:49:15.203271+0000 (oldest deadline 2024-03-28T22:49:38.703429+0000) 2024-03-28T22:50:01.855 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:01 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:50:01.572+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:23.305205+0000 front 2024-03-28T22:49:23.305481+0000 (oldest deadline 2024-03-28T22:49:48.004899+0000) 2024-03-28T22:50:01.855 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:01 smithi172 systemd[1]: Stopping Ceph osd.8 for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:50:01.855 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:01 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:50:01.856+0000 7fc0a9ea1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:50:01.855 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:01 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:50:01.856+0000 7fc0a9ea1700 -1 osd.8 114 *** Got signal Terminated *** 2024-03-28T22:50:01.855 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:01 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:50:01.856+0000 7fc0a9ea1700 -1 osd.8 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T22:50:02.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:01 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:01.647+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:50:02.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:01 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:01.647+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:50:02.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:01 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:01.647+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:50:02.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:01 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:01.647+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.656864+0000 front 2024-03-28T22:49:21.657045+0000 (oldest deadline 2024-03-28T22:49:45.757121+0000) 2024-03-28T22:50:02.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:01 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:01.647+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.458838+0000 front 2024-03-28T22:49:30.458960+0000 (oldest deadline 2024-03-28T22:49:53.959115+0000) 2024-03-28T22:50:02.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:01 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:01.794+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:50:02.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:01 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:01.794+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:50:02.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:01 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:01.794+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:50:02.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:01 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:01.794+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:50:02.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:01 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:01.794+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:32.267858+0000 front 2024-03-28T22:49:32.267894+0000 (oldest deadline 2024-03-28T22:49:55.167920+0000) 2024-03-28T22:50:02.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:02 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:50:02.566+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:50:02.896 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:02 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:50:02.566+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:50:02.896 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:02 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:50:02.566+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.203317+0000 front 2024-03-28T22:49:15.203271+0000 (oldest deadline 2024-03-28T22:49:38.703429+0000) 2024-03-28T22:50:02.896 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:02 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:50:02.566+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:23.305205+0000 front 2024-03-28T22:49:23.305481+0000 (oldest deadline 2024-03-28T22:49:48.004899+0000) 2024-03-28T22:50:02.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:02 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:02.528+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:50:02.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:02 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:02.528+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:50:02.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:02 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:02.528+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:50:02.897 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:02 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:02.528+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:50:03.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:02 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:02.619+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:50:03.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:02 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:02.619+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:50:03.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:02 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:02.619+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:50:03.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:02 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:02.619+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.656864+0000 front 2024-03-28T22:49:21.657045+0000 (oldest deadline 2024-03-28T22:49:45.757121+0000) 2024-03-28T22:50:03.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:02 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:02.619+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.458838+0000 front 2024-03-28T22:49:30.458960+0000 (oldest deadline 2024-03-28T22:49:53.959115+0000) 2024-03-28T22:50:03.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:02 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:02.776+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:50:03.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:02 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:02.776+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:50:03.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:02 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:02.776+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:50:03.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:02 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:02.776+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:50:03.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:02 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:02.776+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:32.267858+0000 front 2024-03-28T22:49:32.267894+0000 (oldest deadline 2024-03-28T22:49:55.167920+0000) 2024-03-28T22:50:03.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:02 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:02.776+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:36.869153+0000 front 2024-03-28T22:49:36.869074+0000 (oldest deadline 2024-03-28T22:50:02.168902+0000) 2024-03-28T22:50:03.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:03 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:50:03.565+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:50:03.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:03 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:50:03.565+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:50:03.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:03 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:50:03.565+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.203317+0000 front 2024-03-28T22:49:15.203271+0000 (oldest deadline 2024-03-28T22:49:38.703429+0000) 2024-03-28T22:50:03.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:03 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:50:03.565+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:23.305205+0000 front 2024-03-28T22:49:23.305481+0000 (oldest deadline 2024-03-28T22:49:48.004899+0000) 2024-03-28T22:50:03.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:03 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:03.490+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:50:03.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:03 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:03.490+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:50:03.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:03 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:03.490+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:50:03.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:03 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:03.490+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:50:03.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:03 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:03.490+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:40.201551+0000 front 2024-03-28T22:49:40.201598+0000 (oldest deadline 2024-03-28T22:50:03.101602+0000) 2024-03-28T22:50:04.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:03 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:03.610+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:50:04.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:03 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:03.610+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:50:04.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:03 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:03.610+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:50:04.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:03 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:03.610+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.656864+0000 front 2024-03-28T22:49:21.657045+0000 (oldest deadline 2024-03-28T22:49:45.757121+0000) 2024-03-28T22:50:04.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:03 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:03.610+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.458838+0000 front 2024-03-28T22:49:30.458960+0000 (oldest deadline 2024-03-28T22:49:53.959115+0000) 2024-03-28T22:50:04.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:03 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:03.820+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:50:04.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:03 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:03.820+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:50:04.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:03 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:03.820+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:50:04.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:03 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:03.820+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:50:04.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:03 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:03.820+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:32.267858+0000 front 2024-03-28T22:49:32.267894+0000 (oldest deadline 2024-03-28T22:49:55.167920+0000) 2024-03-28T22:50:04.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:03 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:03.820+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:36.869153+0000 front 2024-03-28T22:49:36.869074+0000 (oldest deadline 2024-03-28T22:50:02.168902+0000) 2024-03-28T22:50:04.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:04 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:50:04.590+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:50:04.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:04 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:50:04.590+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:50:04.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:04 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:50:04.590+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.203317+0000 front 2024-03-28T22:49:15.203271+0000 (oldest deadline 2024-03-28T22:49:38.703429+0000) 2024-03-28T22:50:04.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:04 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:50:04.590+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:23.305205+0000 front 2024-03-28T22:49:23.305481+0000 (oldest deadline 2024-03-28T22:49:48.004899+0000) 2024-03-28T22:50:04.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:04 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:04.515+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:50:04.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:04 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:04.515+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:50:04.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:04 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:04.515+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:50:04.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:04 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:04.515+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:50:04.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:04 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:04.515+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:40.201551+0000 front 2024-03-28T22:49:40.201598+0000 (oldest deadline 2024-03-28T22:50:03.101602+0000) 2024-03-28T22:50:05.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:04 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:04.574+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:50:05.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:04 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:04.574+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:50:05.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:04 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:04.574+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:50:05.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:04 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:04.574+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.656864+0000 front 2024-03-28T22:49:21.657045+0000 (oldest deadline 2024-03-28T22:49:45.757121+0000) 2024-03-28T22:50:05.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:04 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:04.574+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.458838+0000 front 2024-03-28T22:49:30.458960+0000 (oldest deadline 2024-03-28T22:49:53.959115+0000) 2024-03-28T22:50:05.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:04 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:04.858+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:50:05.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:04 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:04.858+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:50:05.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:04 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:04.858+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:50:05.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:04 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:04.858+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:50:05.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:04 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:04.858+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:32.267858+0000 front 2024-03-28T22:49:32.267894+0000 (oldest deadline 2024-03-28T22:49:55.167920+0000) 2024-03-28T22:50:05.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:04 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:04.858+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:36.869153+0000 front 2024-03-28T22:49:36.869074+0000 (oldest deadline 2024-03-28T22:50:02.168902+0000) 2024-03-28T22:50:05.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:05 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:50:05.632+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:50:05.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:05 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:50:05.632+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:50:05.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:05 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:50:05.632+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.203317+0000 front 2024-03-28T22:49:15.203271+0000 (oldest deadline 2024-03-28T22:49:38.703429+0000) 2024-03-28T22:50:05.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:05 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:50:05.632+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:23.305205+0000 front 2024-03-28T22:49:23.305481+0000 (oldest deadline 2024-03-28T22:49:48.004899+0000) 2024-03-28T22:50:05.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:05 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:05.512+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:50:05.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:05 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:05.512+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:50:05.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:05 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:05.512+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:50:05.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:05 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:05.512+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:50:05.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:05 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:05.512+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:40.201551+0000 front 2024-03-28T22:49:40.201598+0000 (oldest deadline 2024-03-28T22:50:03.101602+0000) 2024-03-28T22:50:06.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:05 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:05.612+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:50:06.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:05 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:05.612+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:50:06.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:05 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:05.612+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:50:06.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:05 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:05.612+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.656864+0000 front 2024-03-28T22:49:21.657045+0000 (oldest deadline 2024-03-28T22:49:45.757121+0000) 2024-03-28T22:50:06.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:05 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:05.612+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.458838+0000 front 2024-03-28T22:49:30.458960+0000 (oldest deadline 2024-03-28T22:49:53.959115+0000) 2024-03-28T22:50:06.017 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:05 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:05.612+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:37.461601+0000 front 2024-03-28T22:49:37.461334+0000 (oldest deadline 2024-03-28T22:50:05.062479+0000) 2024-03-28T22:50:06.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:05 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:05.858+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:50:06.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:05 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:05.858+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:50:06.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:05 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:05.858+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:50:06.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:05 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:05.858+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:50:06.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:05 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:05.858+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:32.267858+0000 front 2024-03-28T22:49:32.267894+0000 (oldest deadline 2024-03-28T22:49:55.167920+0000) 2024-03-28T22:50:06.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:05 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:05.858+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:36.869153+0000 front 2024-03-28T22:49:36.869074+0000 (oldest deadline 2024-03-28T22:50:02.168902+0000) 2024-03-28T22:50:06.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:06 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:50:06.672+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.699734+0000 front 2024-03-28T22:48:56.699630+0000 (oldest deadline 2024-03-28T22:49:21.399563+0000) 2024-03-28T22:50:06.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:06 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:50:06.672+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:02.901793+0000 front 2024-03-28T22:49:02.901890+0000 (oldest deadline 2024-03-28T22:49:28.801881+0000) 2024-03-28T22:50:06.896 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:06 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:50:06.672+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.203317+0000 front 2024-03-28T22:49:15.203271+0000 (oldest deadline 2024-03-28T22:49:38.703429+0000) 2024-03-28T22:50:06.896 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:06 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8[54868]: 2024-03-28T22:50:06.672+0000 7fc0a6495700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:23.305205+0000 front 2024-03-28T22:49:23.305481+0000 (oldest deadline 2024-03-28T22:49:48.004899+0000) 2024-03-28T22:50:06.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:06 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:06.560+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:50:06.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:06 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:06.560+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:50:06.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:06 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:06.560+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:50:06.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:06 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:06.560+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:50:06.897 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:06 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:06.560+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:40.201551+0000 front 2024-03-28T22:49:40.201598+0000 (oldest deadline 2024-03-28T22:50:03.101602+0000) 2024-03-28T22:50:07.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:06 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:06.590+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:50:07.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:06 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:06.590+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:50:07.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:06 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:06.590+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:50:07.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:06 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:06.590+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.656864+0000 front 2024-03-28T22:49:21.657045+0000 (oldest deadline 2024-03-28T22:49:45.757121+0000) 2024-03-28T22:50:07.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:06 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:06.590+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.458838+0000 front 2024-03-28T22:49:30.458960+0000 (oldest deadline 2024-03-28T22:49:53.959115+0000) 2024-03-28T22:50:07.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:06 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:06.590+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:37.461601+0000 front 2024-03-28T22:49:37.461334+0000 (oldest deadline 2024-03-28T22:50:05.062479+0000) 2024-03-28T22:50:07.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:06 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:06.837+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:50:07.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:06 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:06.837+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:50:07.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:06 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:06.837+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:50:07.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:06 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:06.837+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:50:07.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:06 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:06.837+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:32.267858+0000 front 2024-03-28T22:49:32.267894+0000 (oldest deadline 2024-03-28T22:49:55.167920+0000) 2024-03-28T22:50:07.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:06 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:06.837+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:36.869153+0000 front 2024-03-28T22:49:36.869074+0000 (oldest deadline 2024-03-28T22:50:02.168902+0000) 2024-03-28T22:50:07.548 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:07 smithi172 podman[74029]: 2024-03-28 22:50:07.275496823 +0000 UTC m=+5.541042793 container died 0c117c9970f897ed0c0ec308ad5532107da4cb30c996143b97af478c7eb9df9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T22:50:07.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:07 smithi172 podman[74029]: 2024-03-28 22:50:07.672543561 +0000 UTC m=+5.938089529 container cleanup 0c117c9970f897ed0c0ec308ad5532107da4cb30c996143b97af478c7eb9df9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-28T22:50:07.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:07 smithi172 bash[74029]: ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8 2024-03-28T22:50:07.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:07 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:07.549+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:50:07.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:07 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:07.549+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:50:07.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:07 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:07.549+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:50:07.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:07 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:07.549+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:50:07.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:07 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:07.549+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:40.201551+0000 front 2024-03-28T22:49:40.201598+0000 (oldest deadline 2024-03-28T22:50:03.101602+0000) 2024-03-28T22:50:08.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:07 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:07.557+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:50:08.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:07 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:07.557+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:50:08.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:07 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:07.557+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:50:08.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:07 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:07.557+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.656864+0000 front 2024-03-28T22:49:21.657045+0000 (oldest deadline 2024-03-28T22:49:45.757121+0000) 2024-03-28T22:50:08.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:07 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:07.557+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.458838+0000 front 2024-03-28T22:49:30.458960+0000 (oldest deadline 2024-03-28T22:49:53.959115+0000) 2024-03-28T22:50:08.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:07 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:07.557+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:37.461601+0000 front 2024-03-28T22:49:37.461334+0000 (oldest deadline 2024-03-28T22:50:05.062479+0000) 2024-03-28T22:50:08.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:07 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:07.813+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:50:08.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:07 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:07.813+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:50:08.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:07 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:07.813+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:50:08.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:07 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:07.813+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:50:08.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:07 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:07.813+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:32.267858+0000 front 2024-03-28T22:49:32.267894+0000 (oldest deadline 2024-03-28T22:49:55.167920+0000) 2024-03-28T22:50:08.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:07 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:07.813+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:36.869153+0000 front 2024-03-28T22:49:36.869074+0000 (oldest deadline 2024-03-28T22:50:02.168902+0000) 2024-03-28T22:50:08.234 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:07 smithi172 podman[74046]: 2024-03-28 22:50:07.924225717 +0000 UTC m=+0.652558011 container remove 0c117c9970f897ed0c0ec308ad5532107da4cb30c996143b97af478c7eb9df9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, RELEASE=squid-323d23d, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326) 2024-03-28T22:50:08.534 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:08 smithi172 podman[74134]: 2024-03-28 22:50:08.14073457 +0000 UTC m=+0.017650231 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:50:08.535 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:08 smithi172 podman[74134]: 2024-03-28 22:50:08.241676315 +0000 UTC m=+0.118591999 container create b6941987e7c30380b56e79fbe6b5fc6a3b18154583d359c2a3e0693b19e82e57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-28T22:50:08.535 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:08 smithi172 podman[74134]: 2024-03-28 22:50:08.477474576 +0000 UTC m=+0.354390228 container init b6941987e7c30380b56e79fbe6b5fc6a3b18154583d359c2a3e0693b19e82e57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-323d23d, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326) 2024-03-28T22:50:08.535 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:08 smithi172 podman[74134]: 2024-03-28 22:50:08.480662762 +0000 UTC m=+0.357578412 container start b6941987e7c30380b56e79fbe6b5fc6a3b18154583d359c2a3e0693b19e82e57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8-deactivate, RELEASE=squid-323d23d, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-28T22:50:08.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:08 smithi172 podman[74134]: 2024-03-28 22:50:08.594229156 +0000 UTC m=+0.471144809 container attach b6941987e7c30380b56e79fbe6b5fc6a3b18154583d359c2a3e0693b19e82e57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-323d23d, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-28T22:50:08.895 INFO:journalctl@ceph.osd.8.smithi172.stdout:Mar 28 22:50:08 smithi172 podman[74134]: 2024-03-28 22:50:08.768501904 +0000 UTC m=+0.645417558 container died b6941987e7c30380b56e79fbe6b5fc6a3b18154583d359c2a3e0693b19e82e57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-8-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD) 2024-03-28T22:50:08.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:08 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:08.536+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:50:08.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:08 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:08.536+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:50:08.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:08 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:08.536+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:50:08.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:08 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:08.536+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:50:08.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:08 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:08.536+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:40.201551+0000 front 2024-03-28T22:49:40.201598+0000 (oldest deadline 2024-03-28T22:50:03.101602+0000) 2024-03-28T22:50:09.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:08 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:08.597+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:50:09.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:08 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:08.597+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:50:09.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:08 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:08.597+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:50:09.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:08 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:08.597+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.656864+0000 front 2024-03-28T22:49:21.657045+0000 (oldest deadline 2024-03-28T22:49:45.757121+0000) 2024-03-28T22:50:09.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:08 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:08.597+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.458838+0000 front 2024-03-28T22:49:30.458960+0000 (oldest deadline 2024-03-28T22:49:53.959115+0000) 2024-03-28T22:50:09.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:08 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:08.597+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:37.461601+0000 front 2024-03-28T22:49:37.461334+0000 (oldest deadline 2024-03-28T22:50:05.062479+0000) 2024-03-28T22:50:09.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:08 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:08.843+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:50:09.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:08 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:08.843+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:50:09.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:08 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:08.843+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:50:09.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:08 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:08.843+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:50:09.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:08 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:08.843+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:32.267858+0000 front 2024-03-28T22:49:32.267894+0000 (oldest deadline 2024-03-28T22:49:55.167920+0000) 2024-03-28T22:50:09.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:08 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:08.843+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:36.869153+0000 front 2024-03-28T22:49:36.869074+0000 (oldest deadline 2024-03-28T22:50:02.168902+0000) 2024-03-28T22:50:09.451 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.8.service' 2024-03-28T22:50:09.494 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:50:09.495 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-28T22:50:09.495 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-28T22:50:09.495 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.9 2024-03-28T22:50:09.705 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:09 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:09.556+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:50:09.706 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:09 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:09.556+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:50:09.706 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:09 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:09.556+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:50:09.706 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:09 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:09.556+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:50:09.706 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:09 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:09.556+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:40.201551+0000 front 2024-03-28T22:49:40.201598+0000 (oldest deadline 2024-03-28T22:50:03.101602+0000) 2024-03-28T22:50:10.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:09 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:09.635+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:50:10.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:09 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:09.635+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:50:10.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:09 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:09.635+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:50:10.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:09 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:09.635+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.656864+0000 front 2024-03-28T22:49:21.657045+0000 (oldest deadline 2024-03-28T22:49:45.757121+0000) 2024-03-28T22:50:10.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:09 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:09.635+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.458838+0000 front 2024-03-28T22:49:30.458960+0000 (oldest deadline 2024-03-28T22:49:53.959115+0000) 2024-03-28T22:50:10.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:09 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:09.635+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:37.461601+0000 front 2024-03-28T22:49:37.461334+0000 (oldest deadline 2024-03-28T22:50:05.062479+0000) 2024-03-28T22:50:10.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:09 smithi043 systemd[1]: Stopping Ceph osd.9 for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:50:10.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:09 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:09.888+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:50:10.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:09 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:09.889+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:50:10.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:09 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:09.889+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:50:10.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:09 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:09.889+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:50:10.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:09 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:09.889+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:32.267858+0000 front 2024-03-28T22:49:32.267894+0000 (oldest deadline 2024-03-28T22:49:55.167920+0000) 2024-03-28T22:50:10.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:09 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:09.889+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:36.869153+0000 front 2024-03-28T22:49:36.869074+0000 (oldest deadline 2024-03-28T22:50:02.168902+0000) 2024-03-28T22:50:10.515 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:10 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:10.079+0000 7f6d27000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:50:10.515 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:10 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:10.080+0000 7f6d27000700 -1 osd.9 114 *** Got signal Terminated *** 2024-03-28T22:50:10.515 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:10 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:10.080+0000 7f6d27000700 -1 osd.9 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T22:50:10.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:10 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:10.584+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:50:10.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:10 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:10.584+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:50:10.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:10 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:10.584+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:50:10.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:10 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:10.584+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:50:10.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:10 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:10.584+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:40.201551+0000 front 2024-03-28T22:49:40.201598+0000 (oldest deadline 2024-03-28T22:50:03.101602+0000) 2024-03-28T22:50:10.983 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:10 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:10.624+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:50:10.983 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:10 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:10.624+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:50:10.983 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:10 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:10.624+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:50:10.983 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:10 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:10.624+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.656864+0000 front 2024-03-28T22:49:21.657045+0000 (oldest deadline 2024-03-28T22:49:45.757121+0000) 2024-03-28T22:50:10.983 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:10 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:10.624+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.458838+0000 front 2024-03-28T22:49:30.458960+0000 (oldest deadline 2024-03-28T22:49:53.959115+0000) 2024-03-28T22:50:10.983 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:10 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:10.624+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:37.461601+0000 front 2024-03-28T22:49:37.461334+0000 (oldest deadline 2024-03-28T22:50:05.062479+0000) 2024-03-28T22:50:11.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:10 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:10.841+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:50:11.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:10 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:10.841+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:50:11.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:10 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:10.841+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:50:11.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:10 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:10.841+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:50:11.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:10 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:10.841+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:32.267858+0000 front 2024-03-28T22:49:32.267894+0000 (oldest deadline 2024-03-28T22:49:55.167920+0000) 2024-03-28T22:50:11.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:10 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:10.841+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:36.869153+0000 front 2024-03-28T22:49:36.869074+0000 (oldest deadline 2024-03-28T22:50:02.168902+0000) 2024-03-28T22:50:11.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:11 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:11.585+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:50:11.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:11 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:11.585+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:50:11.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:11 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:11.585+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:50:11.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:11 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:11.585+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:50:11.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:11 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:11.585+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:40.201551+0000 front 2024-03-28T22:49:40.201598+0000 (oldest deadline 2024-03-28T22:50:03.101602+0000) 2024-03-28T22:50:12.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:11 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:11.661+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:50:12.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:11 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:11.661+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:50:12.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:11 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:11.661+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:50:12.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:11 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:11.661+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.656864+0000 front 2024-03-28T22:49:21.657045+0000 (oldest deadline 2024-03-28T22:49:45.757121+0000) 2024-03-28T22:50:12.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:11 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:11.661+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.458838+0000 front 2024-03-28T22:49:30.458960+0000 (oldest deadline 2024-03-28T22:49:53.959115+0000) 2024-03-28T22:50:12.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:11 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:11.661+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:37.461601+0000 front 2024-03-28T22:49:37.461334+0000 (oldest deadline 2024-03-28T22:50:05.062479+0000) 2024-03-28T22:50:12.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:11 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:11.851+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:50:12.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:11 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:11.851+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:50:12.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:11 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:11.851+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:50:12.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:11 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:11.851+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:50:12.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:11 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:11.851+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:32.267858+0000 front 2024-03-28T22:49:32.267894+0000 (oldest deadline 2024-03-28T22:49:55.167920+0000) 2024-03-28T22:50:12.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:11 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:11.851+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:36.869153+0000 front 2024-03-28T22:49:36.869074+0000 (oldest deadline 2024-03-28T22:50:02.168902+0000) 2024-03-28T22:50:12.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:12 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:12.576+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:50:12.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:12 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:12.576+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:50:12.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:12 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:12.576+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:50:12.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:12 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:12.576+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:50:12.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:12 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:12.576+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:40.201551+0000 front 2024-03-28T22:49:40.201598+0000 (oldest deadline 2024-03-28T22:50:03.101602+0000) 2024-03-28T22:50:12.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:12 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:12.576+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-03-28T22:49:48.402804+0000 front 2024-03-28T22:49:48.402759+0000 (oldest deadline 2024-03-28T22:50:11.902489+0000) 2024-03-28T22:50:13.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:12 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:12.676+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:50:13.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:12 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:12.676+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:50:13.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:12 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:12.676+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:50:13.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:12 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:12.676+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.656864+0000 front 2024-03-28T22:49:21.657045+0000 (oldest deadline 2024-03-28T22:49:45.757121+0000) 2024-03-28T22:50:13.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:12 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:12.676+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.458838+0000 front 2024-03-28T22:49:30.458960+0000 (oldest deadline 2024-03-28T22:49:53.959115+0000) 2024-03-28T22:50:13.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:12 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:12.676+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:37.461601+0000 front 2024-03-28T22:49:37.461334+0000 (oldest deadline 2024-03-28T22:50:05.062479+0000) 2024-03-28T22:50:13.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:12 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:12.676+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-03-28T22:49:48.565509+0000 front 2024-03-28T22:49:48.565458+0000 (oldest deadline 2024-03-28T22:50:12.665111+0000) 2024-03-28T22:50:13.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:12 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:12.896+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:50:13.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:12 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:12.896+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:50:13.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:12 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:12.896+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:50:13.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:12 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:12.896+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:50:13.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:12 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:12.896+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:32.267858+0000 front 2024-03-28T22:49:32.267894+0000 (oldest deadline 2024-03-28T22:49:55.167920+0000) 2024-03-28T22:50:13.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:12 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:12.896+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:36.869153+0000 front 2024-03-28T22:49:36.869074+0000 (oldest deadline 2024-03-28T22:50:02.168902+0000) 2024-03-28T22:50:13.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:13 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:13.554+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:50:13.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:13 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:13.554+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:50:13.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:13 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:13.554+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:50:13.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:13 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:13.554+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:50:13.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:13 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:13.554+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:40.201551+0000 front 2024-03-28T22:49:40.201598+0000 (oldest deadline 2024-03-28T22:50:03.101602+0000) 2024-03-28T22:50:13.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:13 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:13.554+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-03-28T22:49:48.402804+0000 front 2024-03-28T22:49:48.402759+0000 (oldest deadline 2024-03-28T22:50:11.902489+0000) 2024-03-28T22:50:14.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:13 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:13.714+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:50:14.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:13 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:13.714+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:50:14.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:13 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:13.714+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:50:14.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:13 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:13.714+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.656864+0000 front 2024-03-28T22:49:21.657045+0000 (oldest deadline 2024-03-28T22:49:45.757121+0000) 2024-03-28T22:50:14.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:13 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:13.714+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.458838+0000 front 2024-03-28T22:49:30.458960+0000 (oldest deadline 2024-03-28T22:49:53.959115+0000) 2024-03-28T22:50:14.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:13 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:13.714+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:37.461601+0000 front 2024-03-28T22:49:37.461334+0000 (oldest deadline 2024-03-28T22:50:05.062479+0000) 2024-03-28T22:50:14.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:13 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:13.714+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-03-28T22:49:48.565509+0000 front 2024-03-28T22:49:48.565458+0000 (oldest deadline 2024-03-28T22:50:12.665111+0000) 2024-03-28T22:50:14.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:13 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:13.854+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:50:14.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:13 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:13.854+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:50:14.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:13 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:13.854+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:50:14.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:13 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:13.854+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:50:14.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:13 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:13.854+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:32.267858+0000 front 2024-03-28T22:49:32.267894+0000 (oldest deadline 2024-03-28T22:49:55.167920+0000) 2024-03-28T22:50:14.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:13 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:13.854+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:36.869153+0000 front 2024-03-28T22:49:36.869074+0000 (oldest deadline 2024-03-28T22:50:02.168902+0000) 2024-03-28T22:50:14.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:14 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:14.559+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:50:14.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:14 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:14.559+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:50:14.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:14 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:14.559+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:50:14.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:14 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:14.559+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:50:14.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:14 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:14.559+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:40.201551+0000 front 2024-03-28T22:49:40.201598+0000 (oldest deadline 2024-03-28T22:50:03.101602+0000) 2024-03-28T22:50:14.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:14 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:14.559+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-03-28T22:49:48.402804+0000 front 2024-03-28T22:49:48.402759+0000 (oldest deadline 2024-03-28T22:50:11.902489+0000) 2024-03-28T22:50:15.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:14 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:14.684+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:54.944391+0000 front 2024-03-28T22:48:54.944928+0000 (oldest deadline 2024-03-28T22:49:24.345841+0000) 2024-03-28T22:50:15.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:14 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:14.684+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.349043+0000 front 2024-03-28T22:49:05.348867+0000 (oldest deadline 2024-03-28T22:49:30.649109+0000) 2024-03-28T22:50:15.015 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:14 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:14.684+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:14.751890+0000 front 2024-03-28T22:49:14.751804+0000 (oldest deadline 2024-03-28T22:49:41.153539+0000) 2024-03-28T22:50:15.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:14 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:14.684+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.656864+0000 front 2024-03-28T22:49:21.657045+0000 (oldest deadline 2024-03-28T22:49:45.757121+0000) 2024-03-28T22:50:15.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:14 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:14.684+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.458838+0000 front 2024-03-28T22:49:30.458960+0000 (oldest deadline 2024-03-28T22:49:53.959115+0000) 2024-03-28T22:50:15.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:14 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:14.684+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:37.461601+0000 front 2024-03-28T22:49:37.461334+0000 (oldest deadline 2024-03-28T22:50:05.062479+0000) 2024-03-28T22:50:15.016 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:14 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9[64320]: 2024-03-28T22:50:14.684+0000 7f6d22800700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-03-28T22:49:48.565509+0000 front 2024-03-28T22:49:48.565458+0000 (oldest deadline 2024-03-28T22:50:12.665111+0000) 2024-03-28T22:50:15.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:14 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:14.842+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:50:15.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:14 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:14.842+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:50:15.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:14 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:14.842+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:50:15.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:14 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:14.842+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:50:15.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:14 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:14.842+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:32.267858+0000 front 2024-03-28T22:49:32.267894+0000 (oldest deadline 2024-03-28T22:49:55.167920+0000) 2024-03-28T22:50:15.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:14 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:14.842+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:36.869153+0000 front 2024-03-28T22:49:36.869074+0000 (oldest deadline 2024-03-28T22:50:02.168902+0000) 2024-03-28T22:50:15.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:15 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:15.597+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:50:15.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:15 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:15.597+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:50:15.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:15 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:15.597+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:50:15.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:15 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:15.597+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:50:15.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:15 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:15.597+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:40.201551+0000 front 2024-03-28T22:49:40.201598+0000 (oldest deadline 2024-03-28T22:50:03.101602+0000) 2024-03-28T22:50:15.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:15 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:15.597+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-03-28T22:49:48.402804+0000 front 2024-03-28T22:49:48.402759+0000 (oldest deadline 2024-03-28T22:50:11.902489+0000) 2024-03-28T22:50:16.004 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:15 smithi043 podman[105074]: 2024-03-28 22:50:15.592701057 +0000 UTC m=+5.651647636 container died 54750ead066a7fa3cf8756b7f1bb8a66bc11464ba332e3bf1bf30304651c169c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD) 2024-03-28T22:50:16.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:15 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:15.829+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:50:16.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:15 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:15.829+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:50:16.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:15 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:15.829+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:50:16.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:15 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:15.829+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:50:16.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:15 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:15.829+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:32.267858+0000 front 2024-03-28T22:49:32.267894+0000 (oldest deadline 2024-03-28T22:49:55.167920+0000) 2024-03-28T22:50:16.200 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:15 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:15.829+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:36.869153+0000 front 2024-03-28T22:49:36.869074+0000 (oldest deadline 2024-03-28T22:50:02.168902+0000) 2024-03-28T22:50:16.200 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:15 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:15.829+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-03-28T22:49:49.170749+0000 front 2024-03-28T22:49:49.170896+0000 (oldest deadline 2024-03-28T22:50:15.070453+0000) 2024-03-28T22:50:16.265 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:16 smithi043 podman[105074]: 2024-03-28 22:50:16.114259088 +0000 UTC m=+6.173205661 container cleanup 54750ead066a7fa3cf8756b7f1bb8a66bc11464ba332e3bf1bf30304651c169c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, RELEASE=squid-323d23d, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-28T22:50:16.265 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:16 smithi043 bash[105074]: ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9 2024-03-28T22:50:16.766 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:16 smithi043 podman[105093]: 2024-03-28 22:50:16.382176516 +0000 UTC m=+0.773064460 container remove 54750ead066a7fa3cf8756b7f1bb8a66bc11464ba332e3bf1bf30304651c169c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-323d23d, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-28T22:50:16.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:16 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:16.631+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:50:16.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:16 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:16.631+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:50:16.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:16 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:16.631+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:50:16.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:16 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:16.631+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:50:16.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:16 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:16.631+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:40.201551+0000 front 2024-03-28T22:49:40.201598+0000 (oldest deadline 2024-03-28T22:50:03.101602+0000) 2024-03-28T22:50:16.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:16 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:16.631+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-03-28T22:49:48.402804+0000 front 2024-03-28T22:49:48.402759+0000 (oldest deadline 2024-03-28T22:50:11.902489+0000) 2024-03-28T22:50:17.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:16 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:16.836+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:50:17.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:16 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:16.836+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:50:17.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:16 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:16.836+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:50:17.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:16 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:16.836+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:50:17.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:16 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:16.836+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:32.267858+0000 front 2024-03-28T22:49:32.267894+0000 (oldest deadline 2024-03-28T22:49:55.167920+0000) 2024-03-28T22:50:17.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:16 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:16.836+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:36.869153+0000 front 2024-03-28T22:49:36.869074+0000 (oldest deadline 2024-03-28T22:50:02.168902+0000) 2024-03-28T22:50:17.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:16 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:16.836+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-03-28T22:49:49.170749+0000 front 2024-03-28T22:49:49.170896+0000 (oldest deadline 2024-03-28T22:50:15.070453+0000) 2024-03-28T22:50:17.515 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:17 smithi043 podman[105177]: 2024-03-28 22:50:17.156480013 +0000 UTC m=+0.038016326 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:50:17.515 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:17 smithi043 podman[105177]: 2024-03-28 22:50:17.266250841 +0000 UTC m=+0.147787181 container create 1983939608e57dfc8bfe9d279ff826cd7ef2776d52bdbb4f257d3f7491132656 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS) 2024-03-28T22:50:17.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:17 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:17.617+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:50:17.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:17 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:17.617+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:50:17.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:17 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:17.617+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:50:17.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:17 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:17.617+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:50:17.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:17 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:17.617+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:40.201551+0000 front 2024-03-28T22:49:40.201598+0000 (oldest deadline 2024-03-28T22:50:03.101602+0000) 2024-03-28T22:50:17.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:17 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:17.617+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-03-28T22:49:48.402804+0000 front 2024-03-28T22:49:48.402759+0000 (oldest deadline 2024-03-28T22:50:11.902489+0000) 2024-03-28T22:50:17.982 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:17 smithi043 podman[105177]: 2024-03-28 22:50:17.678872814 +0000 UTC m=+0.560409158 container init 1983939608e57dfc8bfe9d279ff826cd7ef2776d52bdbb4f257d3f7491132656 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, RELEASE=squid-323d23d, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-28T22:50:17.982 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:17 smithi043 podman[105177]: 2024-03-28 22:50:17.690375881 +0000 UTC m=+0.571912216 container start 1983939608e57dfc8bfe9d279ff826cd7ef2776d52bdbb4f257d3f7491132656 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-323d23d, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, io.buildah.version=1.33.5) 2024-03-28T22:50:17.982 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:17 smithi043 podman[105177]: 2024-03-28 22:50:17.777104685 +0000 UTC m=+0.658641009 container attach 1983939608e57dfc8bfe9d279ff826cd7ef2776d52bdbb4f257d3f7491132656 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-323d23d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-28T22:50:18.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:17 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:17.801+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:50:18.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:17 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:17.801+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:50:18.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:17 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:17.801+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:50:18.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:17 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:17.801+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:50:18.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:17 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:17.801+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:32.267858+0000 front 2024-03-28T22:49:32.267894+0000 (oldest deadline 2024-03-28T22:49:55.167920+0000) 2024-03-28T22:50:18.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:17 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:17.801+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:36.869153+0000 front 2024-03-28T22:49:36.869074+0000 (oldest deadline 2024-03-28T22:50:02.168902+0000) 2024-03-28T22:50:18.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:17 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:17.801+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-03-28T22:49:49.170749+0000 front 2024-03-28T22:49:49.170896+0000 (oldest deadline 2024-03-28T22:50:15.070453+0000) 2024-03-28T22:50:18.265 INFO:journalctl@ceph.osd.9.smithi043.stdout:Mar 28 22:50:18 smithi043 podman[105177]: 2024-03-28 22:50:18.140954846 +0000 UTC m=+1.022491176 container died 1983939608e57dfc8bfe9d279ff826cd7ef2776d52bdbb4f257d3f7491132656 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, RELEASE=squid-323d23d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-28T22:50:18.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:18 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:18.593+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:50:18.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:18 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:18.593+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:50:18.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:18 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:18.593+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:50:18.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:18 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:18.593+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:50:18.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:18 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:18.593+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:40.201551+0000 front 2024-03-28T22:49:40.201598+0000 (oldest deadline 2024-03-28T22:50:03.101602+0000) 2024-03-28T22:50:18.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:18 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:18.593+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-03-28T22:49:48.402804+0000 front 2024-03-28T22:49:48.402759+0000 (oldest deadline 2024-03-28T22:50:11.902489+0000) 2024-03-28T22:50:19.059 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.9.service' 2024-03-28T22:50:19.187 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:50:19.188 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-28T22:50:19.188 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-28T22:50:19.188 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.10 2024-03-28T22:50:19.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:18 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:18.826+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:50:19.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:18 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:18.826+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:50:19.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:18 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:18.826+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:50:19.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:18 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:18.826+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:50:19.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:18 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:18.826+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:32.267858+0000 front 2024-03-28T22:49:32.267894+0000 (oldest deadline 2024-03-28T22:49:55.167920+0000) 2024-03-28T22:50:19.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:18 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:18.826+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:36.869153+0000 front 2024-03-28T22:49:36.869074+0000 (oldest deadline 2024-03-28T22:50:02.168902+0000) 2024-03-28T22:50:19.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:18 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:18.826+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-03-28T22:49:49.170749+0000 front 2024-03-28T22:49:49.170896+0000 (oldest deadline 2024-03-28T22:50:15.070453+0000) 2024-03-28T22:50:19.499 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:19 smithi082 systemd[1]: Stopping Ceph osd.10 for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:50:19.819 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:19 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:19.501+0000 7f8971443700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:50:19.819 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:19 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:19.501+0000 7f8971443700 -1 osd.10 114 *** Got signal Terminated *** 2024-03-28T22:50:19.820 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:19 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:19.501+0000 7f8971443700 -1 osd.10 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T22:50:19.820 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:19 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:19.822+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:50:19.820 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:19 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:19.822+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:50:19.820 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:19 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:19.822+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:50:19.820 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:19 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:19.822+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:50:19.820 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:19 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:19.822+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:32.267858+0000 front 2024-03-28T22:49:32.267894+0000 (oldest deadline 2024-03-28T22:49:55.167920+0000) 2024-03-28T22:50:19.820 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:19 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:19.822+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:36.869153+0000 front 2024-03-28T22:49:36.869074+0000 (oldest deadline 2024-03-28T22:50:02.168902+0000) 2024-03-28T22:50:19.820 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:19 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:19.822+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-03-28T22:49:49.170749+0000 front 2024-03-28T22:49:49.170896+0000 (oldest deadline 2024-03-28T22:50:15.070453+0000) 2024-03-28T22:50:19.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:19 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:19.562+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:50:19.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:19 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:19.562+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:50:19.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:19 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:19.562+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:50:19.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:19 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:19.562+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:50:19.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:19 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:19.562+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:40.201551+0000 front 2024-03-28T22:49:40.201598+0000 (oldest deadline 2024-03-28T22:50:03.101602+0000) 2024-03-28T22:50:19.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:19 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:19.562+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-03-28T22:49:48.402804+0000 front 2024-03-28T22:49:48.402759+0000 (oldest deadline 2024-03-28T22:50:11.902489+0000) 2024-03-28T22:50:20.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:20 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:20.523+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:50:20.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:20 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:20.523+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:50:20.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:20 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:20.523+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:50:20.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:20 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:20.523+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:50:20.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:20 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:20.523+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:40.201551+0000 front 2024-03-28T22:49:40.201598+0000 (oldest deadline 2024-03-28T22:50:03.101602+0000) 2024-03-28T22:50:20.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:20 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:20.523+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-03-28T22:49:48.402804+0000 front 2024-03-28T22:49:48.402759+0000 (oldest deadline 2024-03-28T22:50:11.902489+0000) 2024-03-28T22:50:21.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:20 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:20.777+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:50:21.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:20 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:20.777+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:50:21.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:20 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:20.777+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:50:21.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:20 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:20.777+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:50:21.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:20 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:20.777+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:32.267858+0000 front 2024-03-28T22:49:32.267894+0000 (oldest deadline 2024-03-28T22:49:55.167920+0000) 2024-03-28T22:50:21.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:20 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:20.777+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:36.869153+0000 front 2024-03-28T22:49:36.869074+0000 (oldest deadline 2024-03-28T22:50:02.168902+0000) 2024-03-28T22:50:21.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:20 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:20.777+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-03-28T22:49:49.170749+0000 front 2024-03-28T22:49:49.170896+0000 (oldest deadline 2024-03-28T22:50:15.070453+0000) 2024-03-28T22:50:21.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:21 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:21.569+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:50:21.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:21 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:21.569+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:50:21.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:21 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:21.569+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:50:21.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:21 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:21.569+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:50:21.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:21 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:21.569+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:40.201551+0000 front 2024-03-28T22:49:40.201598+0000 (oldest deadline 2024-03-28T22:50:03.101602+0000) 2024-03-28T22:50:21.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:21 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:21.569+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-03-28T22:49:48.402804+0000 front 2024-03-28T22:49:48.402759+0000 (oldest deadline 2024-03-28T22:50:11.902489+0000) 2024-03-28T22:50:22.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:21 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:21.741+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:50:22.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:21 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:21.741+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:50:22.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:21 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:21.741+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:50:22.200 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:21 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:21.741+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:50:22.200 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:21 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:21.741+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:32.267858+0000 front 2024-03-28T22:49:32.267894+0000 (oldest deadline 2024-03-28T22:49:55.167920+0000) 2024-03-28T22:50:22.200 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:21 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:21.741+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:36.869153+0000 front 2024-03-28T22:49:36.869074+0000 (oldest deadline 2024-03-28T22:50:02.168902+0000) 2024-03-28T22:50:22.200 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:21 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:21.741+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-03-28T22:49:49.170749+0000 front 2024-03-28T22:49:49.170896+0000 (oldest deadline 2024-03-28T22:50:15.070453+0000) 2024-03-28T22:50:22.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:22 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:22.555+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:50:22.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:22 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:22.555+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:50:22.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:22 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:22.555+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:50:22.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:22 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:22.555+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:50:22.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:22 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:22.555+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:40.201551+0000 front 2024-03-28T22:49:40.201598+0000 (oldest deadline 2024-03-28T22:50:03.101602+0000) 2024-03-28T22:50:22.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:22 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:22.555+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-03-28T22:49:48.402804+0000 front 2024-03-28T22:49:48.402759+0000 (oldest deadline 2024-03-28T22:50:11.902489+0000) 2024-03-28T22:50:22.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:22 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:22.555+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6820 osd.7 since back 2024-03-28T22:49:58.304149+0000 front 2024-03-28T22:49:58.304225+0000 (oldest deadline 2024-03-28T22:50:21.804125+0000) 2024-03-28T22:50:23.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:22 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:22.754+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:50:23.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:22 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:22.754+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:50:23.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:22 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:22.754+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:50:23.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:22 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:22.754+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:50:23.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:22 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:22.754+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:32.267858+0000 front 2024-03-28T22:49:32.267894+0000 (oldest deadline 2024-03-28T22:49:55.167920+0000) 2024-03-28T22:50:23.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:22 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:22.754+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:36.869153+0000 front 2024-03-28T22:49:36.869074+0000 (oldest deadline 2024-03-28T22:50:02.168902+0000) 2024-03-28T22:50:23.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:22 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:22.754+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-03-28T22:49:49.170749+0000 front 2024-03-28T22:49:49.170896+0000 (oldest deadline 2024-03-28T22:50:15.070453+0000) 2024-03-28T22:50:23.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:23 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:23.590+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:50:23.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:23 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:23.590+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:50:23.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:23 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:23.590+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:50:23.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:23 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:23.590+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:50:23.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:23 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:23.590+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:40.201551+0000 front 2024-03-28T22:49:40.201598+0000 (oldest deadline 2024-03-28T22:50:03.101602+0000) 2024-03-28T22:50:23.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:23 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:23.590+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-03-28T22:49:48.402804+0000 front 2024-03-28T22:49:48.402759+0000 (oldest deadline 2024-03-28T22:50:11.902489+0000) 2024-03-28T22:50:23.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:23 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:23.590+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6820 osd.7 since back 2024-03-28T22:49:58.304149+0000 front 2024-03-28T22:49:58.304225+0000 (oldest deadline 2024-03-28T22:50:21.804125+0000) 2024-03-28T22:50:24.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:23 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:23.726+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:52.760999+0000 front 2024-03-28T22:48:52.761164+0000 (oldest deadline 2024-03-28T22:49:18.660854+0000) 2024-03-28T22:50:24.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:23 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:23.726+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:05.662380+0000 front 2024-03-28T22:49:05.662501+0000 (oldest deadline 2024-03-28T22:49:28.562526+0000) 2024-03-28T22:50:24.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:23 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:23.726+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:12.965377+0000 front 2024-03-28T22:49:12.965449+0000 (oldest deadline 2024-03-28T22:49:41.765909+0000) 2024-03-28T22:50:24.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:23 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:23.726+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6814 osd.3 since back 2024-03-28T22:49:21.766654+0000 front 2024-03-28T22:49:21.766925+0000 (oldest deadline 2024-03-28T22:49:47.066474+0000) 2024-03-28T22:50:24.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:23 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:23.726+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:32.267858+0000 front 2024-03-28T22:49:32.267894+0000 (oldest deadline 2024-03-28T22:49:55.167920+0000) 2024-03-28T22:50:24.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:23 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:23.726+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:36.869153+0000 front 2024-03-28T22:49:36.869074+0000 (oldest deadline 2024-03-28T22:50:02.168902+0000) 2024-03-28T22:50:24.199 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:23 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10[62014]: 2024-03-28T22:50:23.726+0000 7f896da37700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-03-28T22:49:49.170749+0000 front 2024-03-28T22:49:49.170896+0000 (oldest deadline 2024-03-28T22:50:15.070453+0000) 2024-03-28T22:50:24.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:24 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:24.586+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:50:24.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:24 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:24.586+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:50:24.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:24 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:24.586+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:50:24.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:24 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:24.586+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:50:24.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:24 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:24.586+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:40.201551+0000 front 2024-03-28T22:49:40.201598+0000 (oldest deadline 2024-03-28T22:50:03.101602+0000) 2024-03-28T22:50:24.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:24 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:24.586+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-03-28T22:49:48.402804+0000 front 2024-03-28T22:49:48.402759+0000 (oldest deadline 2024-03-28T22:50:11.902489+0000) 2024-03-28T22:50:24.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:24 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:24.586+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6820 osd.7 since back 2024-03-28T22:49:58.304149+0000 front 2024-03-28T22:49:58.304225+0000 (oldest deadline 2024-03-28T22:50:21.804125+0000) 2024-03-28T22:50:25.190 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:24 smithi082 podman[74630]: 2024-03-28 22:50:24.933258398 +0000 UTC m=+5.553156143 container died c810979396e6a3d404e02519deffb0539f7ca9675bafda7a4b346e4dfb25d26f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-323d23d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-28T22:50:25.448 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:25 smithi082 podman[74630]: 2024-03-28 22:50:25.275911509 +0000 UTC m=+5.895809255 container cleanup c810979396e6a3d404e02519deffb0539f7ca9675bafda7a4b346e4dfb25d26f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d) 2024-03-28T22:50:25.448 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:25 smithi082 bash[74630]: ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10 2024-03-28T22:50:25.888 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:25 smithi082 podman[74647]: 2024-03-28 22:50:25.561446263 +0000 UTC m=+0.630686675 container remove c810979396e6a3d404e02519deffb0539f7ca9675bafda7a4b346e4dfb25d26f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-28T22:50:25.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:25 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:25.627+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:50:25.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:25 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:25.627+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:50:25.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:25 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:25.627+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:50:25.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:25 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:25.627+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:50:25.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:25 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:25.627+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:40.201551+0000 front 2024-03-28T22:49:40.201598+0000 (oldest deadline 2024-03-28T22:50:03.101602+0000) 2024-03-28T22:50:25.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:25 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:25.627+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-03-28T22:49:48.402804+0000 front 2024-03-28T22:49:48.402759+0000 (oldest deadline 2024-03-28T22:50:11.902489+0000) 2024-03-28T22:50:25.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:25 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:25.627+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6820 osd.7 since back 2024-03-28T22:49:58.304149+0000 front 2024-03-28T22:49:58.304225+0000 (oldest deadline 2024-03-28T22:50:21.804125+0000) 2024-03-28T22:50:26.170 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:25 smithi082 podman[74740]: 2024-03-28 22:50:25.794530221 +0000 UTC m=+0.017472301 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:50:26.170 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:25 smithi082 podman[74740]: 2024-03-28 22:50:25.928825996 +0000 UTC m=+0.151768114 container create c89fd609f14bbabbc1ec4bd9866932b6db6639aa14c4212791d7eac19e3a01ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-28T22:50:26.170 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:26 smithi082 podman[74740]: 2024-03-28 22:50:26.09878373 +0000 UTC m=+0.321725818 container init c89fd609f14bbabbc1ec4bd9866932b6db6639aa14c4212791d7eac19e3a01ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10-deactivate, org.label-schema.vendor=CentOS, RELEASE=squid-323d23d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-28T22:50:26.170 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:26 smithi082 podman[74740]: 2024-03-28 22:50:26.102168408 +0000 UTC m=+0.325110478 container start c89fd609f14bbabbc1ec4bd9866932b6db6639aa14c4212791d7eac19e3a01ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, RELEASE=squid-323d23d, org.label-schema.vendor=CentOS) 2024-03-28T22:50:26.170 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:26 smithi082 podman[74740]: 2024-03-28 22:50:26.173289472 +0000 UTC m=+0.396231565 container attach c89fd609f14bbabbc1ec4bd9866932b6db6639aa14c4212791d7eac19e3a01ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T22:50:26.448 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:26 smithi082 podman[74740]: 2024-03-28 22:50:26.392065718 +0000 UTC m=+0.615007796 container died c89fd609f14bbabbc1ec4bd9866932b6db6639aa14c4212791d7eac19e3a01ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10-deactivate, RELEASE=squid-323d23d, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240326) 2024-03-28T22:50:26.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:26 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:26.585+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:50:26.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:26 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:26.585+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:50:26.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:26 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:26.585+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:50:26.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:26 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:26.585+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:50:26.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:26 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:26.585+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:40.201551+0000 front 2024-03-28T22:49:40.201598+0000 (oldest deadline 2024-03-28T22:50:03.101602+0000) 2024-03-28T22:50:26.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:26 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:26.585+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-03-28T22:49:48.402804+0000 front 2024-03-28T22:49:48.402759+0000 (oldest deadline 2024-03-28T22:50:11.902489+0000) 2024-03-28T22:50:26.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:26 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:26.585+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6820 osd.7 since back 2024-03-28T22:49:58.304149+0000 front 2024-03-28T22:49:58.304225+0000 (oldest deadline 2024-03-28T22:50:21.804125+0000) 2024-03-28T22:50:26.922 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 28 22:50:26 smithi082 podman[74766]: 2024-03-28 22:50:26.909530755 +0000 UTC m=+0.512558694 container remove c89fd609f14bbabbc1ec4bd9866932b6db6639aa14c4212791d7eac19e3a01ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-323d23d, GIT_CLEAN=True, org.label-schema.build-date=20240326) 2024-03-28T22:50:26.926 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.10.service' 2024-03-28T22:50:27.006 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:50:27.007 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-28T22:50:27.007 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-28T22:50:27.007 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.11 2024-03-28T22:50:27.329 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:27 smithi172 systemd[1]: Stopping Ceph osd.11 for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:50:27.621 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:27 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:27.332+0000 7fdf645b0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:50:27.622 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:27 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:27.332+0000 7fdf645b0700 -1 osd.11 114 *** Got signal Terminated *** 2024-03-28T22:50:27.622 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:27 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:27.332+0000 7fdf645b0700 -1 osd.11 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T22:50:27.622 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:27 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:27.624+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:50:27.622 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:27 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:27.624+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:50:27.622 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:27 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:27.624+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:50:27.622 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:27 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:27.624+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:50:27.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:27 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:27.624+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:40.201551+0000 front 2024-03-28T22:49:40.201598+0000 (oldest deadline 2024-03-28T22:50:03.101602+0000) 2024-03-28T22:50:27.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:27 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:27.624+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-03-28T22:49:48.402804+0000 front 2024-03-28T22:49:48.402759+0000 (oldest deadline 2024-03-28T22:50:11.902489+0000) 2024-03-28T22:50:27.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:27 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:27.624+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6820 osd.7 since back 2024-03-28T22:49:58.304149+0000 front 2024-03-28T22:49:58.304225+0000 (oldest deadline 2024-03-28T22:50:21.804125+0000) 2024-03-28T22:50:29.145 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:28 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:28.654+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:50:29.145 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:28 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:28.654+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:50:29.146 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:28 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:28.654+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:50:29.146 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:28 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:28.654+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:50:29.146 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:28 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:28.654+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:40.201551+0000 front 2024-03-28T22:49:40.201598+0000 (oldest deadline 2024-03-28T22:50:03.101602+0000) 2024-03-28T22:50:29.146 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:28 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:28.654+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-03-28T22:49:48.402804+0000 front 2024-03-28T22:49:48.402759+0000 (oldest deadline 2024-03-28T22:50:11.902489+0000) 2024-03-28T22:50:29.146 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:28 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:28.654+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6820 osd.7 since back 2024-03-28T22:49:58.304149+0000 front 2024-03-28T22:49:58.304225+0000 (oldest deadline 2024-03-28T22:50:21.804125+0000) 2024-03-28T22:50:30.145 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:29 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:29.700+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:50:30.145 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:29 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:29.700+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:50:30.145 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:29 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:29.700+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:50:30.145 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:29 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:29.700+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:50:30.146 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:29 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:29.700+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:40.201551+0000 front 2024-03-28T22:49:40.201598+0000 (oldest deadline 2024-03-28T22:50:03.101602+0000) 2024-03-28T22:50:30.146 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:29 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:29.700+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-03-28T22:49:48.402804+0000 front 2024-03-28T22:49:48.402759+0000 (oldest deadline 2024-03-28T22:50:11.902489+0000) 2024-03-28T22:50:30.146 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:29 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:29.700+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6820 osd.7 since back 2024-03-28T22:49:58.304149+0000 front 2024-03-28T22:49:58.304225+0000 (oldest deadline 2024-03-28T22:50:21.804125+0000) 2024-03-28T22:50:30.146 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:29 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:29.700+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6820 osd.8 since back 2024-03-28T22:50:04.105205+0000 front 2024-03-28T22:50:04.105233+0000 (oldest deadline 2024-03-28T22:50:29.405151+0000) 2024-03-28T22:50:31.145 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:30 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:30.715+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:50:31.145 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:30 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:30.715+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:50:31.145 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:30 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:30.715+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:50:31.145 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:30 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:30.715+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:50:31.146 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:30 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:30.715+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:40.201551+0000 front 2024-03-28T22:49:40.201598+0000 (oldest deadline 2024-03-28T22:50:03.101602+0000) 2024-03-28T22:50:31.146 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:30 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:30.715+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-03-28T22:49:48.402804+0000 front 2024-03-28T22:49:48.402759+0000 (oldest deadline 2024-03-28T22:50:11.902489+0000) 2024-03-28T22:50:31.146 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:30 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:30.715+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6820 osd.7 since back 2024-03-28T22:49:58.304149+0000 front 2024-03-28T22:49:58.304225+0000 (oldest deadline 2024-03-28T22:50:21.804125+0000) 2024-03-28T22:50:31.146 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:30 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:30.715+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6820 osd.8 since back 2024-03-28T22:50:04.105205+0000 front 2024-03-28T22:50:04.105233+0000 (oldest deadline 2024-03-28T22:50:29.405151+0000) 2024-03-28T22:50:32.145 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:31 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:31.725+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-03-28T22:48:56.594397+0000 front 2024-03-28T22:48:56.594337+0000 (oldest deadline 2024-03-28T22:49:20.694268+0000) 2024-03-28T22:50:32.145 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:31 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:31.725+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-28T22:49:07.095900+0000 front 2024-03-28T22:49:07.096107+0000 (oldest deadline 2024-03-28T22:49:28.795907+0000) 2024-03-28T22:50:32.145 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:31 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:31.725+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6804 osd.2 since back 2024-03-28T22:49:15.197612+0000 front 2024-03-28T22:49:15.197582+0000 (oldest deadline 2024-03-28T22:49:37.497695+0000) 2024-03-28T22:50:32.146 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:31 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:31.725+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-28T22:49:30.399777+0000 front 2024-03-28T22:49:30.399837+0000 (oldest deadline 2024-03-28T22:49:55.099678+0000) 2024-03-28T22:50:32.146 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:31 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:31.725+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6812 osd.5 since back 2024-03-28T22:49:40.201551+0000 front 2024-03-28T22:49:40.201598+0000 (oldest deadline 2024-03-28T22:50:03.101602+0000) 2024-03-28T22:50:32.146 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:31 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:31.725+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.43:6822 osd.6 since back 2024-03-28T22:49:48.402804+0000 front 2024-03-28T22:49:48.402759+0000 (oldest deadline 2024-03-28T22:50:11.902489+0000) 2024-03-28T22:50:32.146 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:31 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:31.725+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.82:6820 osd.7 since back 2024-03-28T22:49:58.304149+0000 front 2024-03-28T22:49:58.304225+0000 (oldest deadline 2024-03-28T22:50:21.804125+0000) 2024-03-28T22:50:32.146 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:31 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11[61957]: 2024-03-28T22:50:31.725+0000 7fdf5cb9c700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.172:6820 osd.8 since back 2024-03-28T22:50:04.105205+0000 front 2024-03-28T22:50:04.105233+0000 (oldest deadline 2024-03-28T22:50:29.405151+0000) 2024-03-28T22:50:32.963 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:32 smithi172 podman[74251]: 2024-03-28 22:50:32.684041015 +0000 UTC m=+5.476701430 container died db0f58b43a9db702df163d602acdbc2603462a3aeccb08a9281d12ce3516ce2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240326) 2024-03-28T22:50:33.313 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:33 smithi172 podman[74251]: 2024-03-28 22:50:33.073636539 +0000 UTC m=+5.866296957 container cleanup db0f58b43a9db702df163d602acdbc2603462a3aeccb08a9281d12ce3516ce2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-323d23d) 2024-03-28T22:50:33.313 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:33 smithi172 bash[74251]: ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11 2024-03-28T22:50:33.623 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:33 smithi172 podman[74268]: 2024-03-28 22:50:33.316325734 +0000 UTC m=+0.628089204 container remove db0f58b43a9db702df163d602acdbc2603462a3aeccb08a9281d12ce3516ce2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-323d23d, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T22:50:33.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:33 smithi172 podman[74356]: 2024-03-28 22:50:33.530427303 +0000 UTC m=+0.018014337 image pull bed8e21b47c64c348bd50fa93e0a2db0f997a2392760f42fd1dd03442cc97c52 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15 2024-03-28T22:50:33.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:33 smithi172 podman[74356]: 2024-03-28 22:50:33.642217918 +0000 UTC m=+0.129804989 container create 5e37a987edb796060f7d3a28e2045ecf0fae02d914e3cd6b203f09542d126699 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, RELEASE=squid-323d23d, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T22:50:33.895 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:33 smithi172 podman[74356]: 2024-03-28 22:50:33.852745759 +0000 UTC m=+0.340332795 container init 5e37a987edb796060f7d3a28e2045ecf0fae02d914e3cd6b203f09542d126699 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-323d23d, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-28T22:50:33.896 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:33 smithi172 podman[74356]: 2024-03-28 22:50:33.855943889 +0000 UTC m=+0.343530917 container start 5e37a987edb796060f7d3a28e2045ecf0fae02d914e3cd6b203f09542d126699 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11-deactivate, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-323d23d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-28T22:50:34.395 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:33 smithi172 podman[74356]: 2024-03-28 22:50:33.935978352 +0000 UTC m=+0.423565400 container attach 5e37a987edb796060f7d3a28e2045ecf0fae02d914e3cd6b203f09542d126699 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-28T22:50:34.395 INFO:journalctl@ceph.osd.11.smithi172.stdout:Mar 28 22:50:34 smithi172 podman[74356]: 2024-03-28 22:50:34.14043854 +0000 UTC m=+0.628025570 container died 5e37a987edb796060f7d3a28e2045ecf0fae02d914e3cd6b203f09542d126699 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-osd-11-deactivate, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, RELEASE=squid-323d23d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-28T22:50:34.694 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@osd.11.service' 2024-03-28T22:50:34.735 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:50:34.736 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-28T22:50:34.736 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-28T22:50:34.736 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.a 2024-03-28T22:50:35.265 INFO:journalctl@ceph.mds.a.smithi043.stdout:Mar 28 22:50:34 smithi043 systemd[1]: Stopping Ceph mds.a for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:50:35.765 INFO:journalctl@ceph.mds.a.smithi043.stdout:Mar 28 22:50:35 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-a[71241]: 2024-03-28T22:50:35.364+0000 7f67b3800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:50:35.765 INFO:journalctl@ceph.mds.a.smithi043.stdout:Mar 28 22:50:35 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-a[71241]: 2024-03-28T22:50:35.364+0000 7f67b3800700 -1 mds.a *** got signal Terminated *** 2024-03-28T22:50:39.676 INFO:journalctl@ceph.mds.a.smithi043.stdout:Mar 28 22:50:39 smithi043 podman[105305]: 2024-03-28 22:50:39.38530649 +0000 UTC m=+4.170877082 container died cfd48870702c76ea810a086f63ca8c536f9d9f8fd34c481e975b0e56e0ae21e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-a, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, RELEASE=squid-323d23d, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T22:50:40.015 INFO:journalctl@ceph.mds.a.smithi043.stdout:Mar 28 22:50:39 smithi043 podman[105305]: 2024-03-28 22:50:39.807428149 +0000 UTC m=+4.592998804 container cleanup cfd48870702c76ea810a086f63ca8c536f9d9f8fd34c481e975b0e56e0ae21e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-a, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-323d23d, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326) 2024-03-28T22:50:40.015 INFO:journalctl@ceph.mds.a.smithi043.stdout:Mar 28 22:50:39 smithi043 bash[105305]: ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-a 2024-03-28T22:50:40.515 INFO:journalctl@ceph.mds.a.smithi043.stdout:Mar 28 22:50:40 smithi043 podman[105319]: 2024-03-28 22:50:40.100379337 +0000 UTC m=+0.697282038 container remove cfd48870702c76ea810a086f63ca8c536f9d9f8fd34c481e975b0e56e0ae21e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-323d23d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-28T22:50:40.650 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.a.service' 2024-03-28T22:50:40.783 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:50:40.783 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-28T22:50:40.783 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-28T22:50:40.783 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.d 2024-03-28T22:50:41.265 INFO:journalctl@ceph.mds.d.smithi043.stdout:Mar 28 22:50:40 smithi043 systemd[1]: Stopping Ceph mds.d for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:50:41.765 INFO:journalctl@ceph.mds.d.smithi043.stdout:Mar 28 22:50:41 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-d[70267]: 2024-03-28T22:50:41.400+0000 7ff5c3600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:50:41.765 INFO:journalctl@ceph.mds.d.smithi043.stdout:Mar 28 22:50:41 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-d[70267]: 2024-03-28T22:50:41.400+0000 7ff5c3600700 -1 mds.d *** got signal Terminated *** 2024-03-28T22:50:43.120 INFO:journalctl@ceph.mds.d.smithi043.stdout:Mar 28 22:50:42 smithi043 podman[105424]: 2024-03-28 22:50:42.867236004 +0000 UTC m=+1.594183970 container died 04d255d931292be2e21e582213ce2659d8e3d15400e02086d582e8748d6414fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-d, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-323d23d) 2024-03-28T22:50:43.409 INFO:journalctl@ceph.mds.d.smithi043.stdout:Mar 28 22:50:43 smithi043 podman[105424]: 2024-03-28 22:50:43.235686103 +0000 UTC m=+1.962634080 container cleanup 04d255d931292be2e21e582213ce2659d8e3d15400e02086d582e8748d6414fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-d, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, ceph=True, io.buildah.version=1.33.5) 2024-03-28T22:50:43.409 INFO:journalctl@ceph.mds.d.smithi043.stdout:Mar 28 22:50:43 smithi043 bash[105424]: ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-d 2024-03-28T22:50:43.765 INFO:journalctl@ceph.mds.d.smithi043.stdout:Mar 28 22:50:43 smithi043 podman[105438]: 2024-03-28 22:50:43.411681126 +0000 UTC m=+0.523988952 container remove 04d255d931292be2e21e582213ce2659d8e3d15400e02086d582e8748d6414fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-323d23d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-28T22:50:43.977 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.d.service' 2024-03-28T22:50:44.101 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:50:44.102 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-28T22:50:44.102 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-28T22:50:44.102 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.g 2024-03-28T22:50:44.515 INFO:journalctl@ceph.mds.g.smithi043.stdout:Mar 28 22:50:44 smithi043 systemd[1]: Stopping Ceph mds.g for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:50:45.015 INFO:journalctl@ceph.mds.g.smithi043.stdout:Mar 28 22:50:44 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-g[70858]: 2024-03-28T22:50:44.678+0000 7f8399000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:50:45.015 INFO:journalctl@ceph.mds.g.smithi043.stdout:Mar 28 22:50:44 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-g[70858]: 2024-03-28T22:50:44.678+0000 7f8399000700 -1 mds.g *** got signal Terminated *** 2024-03-28T22:50:48.217 INFO:journalctl@ceph.mds.g.smithi043.stdout:Mar 28 22:50:47 smithi043 podman[105544]: 2024-03-28 22:50:47.930964588 +0000 UTC m=+3.390180649 container died 66fac975734c4b1da9d2f05246aaeab96c7c787f10b195fd6460f113cfa845bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-g, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, RELEASE=squid-323d23d, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5) 2024-03-28T22:50:48.504 INFO:journalctl@ceph.mds.g.smithi043.stdout:Mar 28 22:50:48 smithi043 podman[105544]: 2024-03-28 22:50:48.331027544 +0000 UTC m=+3.790243623 container cleanup 66fac975734c4b1da9d2f05246aaeab96c7c787f10b195fd6460f113cfa845bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-g, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-28T22:50:48.505 INFO:journalctl@ceph.mds.g.smithi043.stdout:Mar 28 22:50:48 smithi043 bash[105544]: ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-g 2024-03-28T22:50:48.765 INFO:journalctl@ceph.mds.g.smithi043.stdout:Mar 28 22:50:48 smithi043 podman[105557]: 2024-03-28 22:50:48.506830481 +0000 UTC m=+0.557503633 container remove 66fac975734c4b1da9d2f05246aaeab96c7c787f10b195fd6460f113cfa845bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-g, RELEASE=squid-323d23d, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-28T22:50:49.033 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.g.service' 2024-03-28T22:50:49.157 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:50:49.158 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-28T22:50:49.158 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-28T22:50:49.158 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.j 2024-03-28T22:50:49.730 INFO:journalctl@ceph.mds.j.smithi043.stdout:Mar 28 22:50:49 smithi043 systemd[1]: Stopping Ceph mds.j for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:50:50.015 INFO:journalctl@ceph.mds.j.smithi043.stdout:Mar 28 22:50:49 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-j[69875]: 2024-03-28T22:50:49.732+0000 7f0726600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:50:50.015 INFO:journalctl@ceph.mds.j.smithi043.stdout:Mar 28 22:50:49 smithi043 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-j[69875]: 2024-03-28T22:50:49.732+0000 7f0726600700 -1 mds.j *** got signal Terminated *** 2024-03-28T22:50:51.153 INFO:journalctl@ceph.mds.j.smithi043.stdout:Mar 28 22:50:50 smithi043 podman[105660]: 2024-03-28 22:50:50.884422195 +0000 UTC m=+1.261327126 container died ddb67d04777208c50055825d7934b66ac7ea8d5880fd12f288e59ca9900aa3e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, RELEASE=squid-323d23d) 2024-03-28T22:50:51.515 INFO:journalctl@ceph.mds.j.smithi043.stdout:Mar 28 22:50:51 smithi043 podman[105660]: 2024-03-28 22:50:51.267668783 +0000 UTC m=+1.644573707 container cleanup ddb67d04777208c50055825d7934b66ac7ea8d5880fd12f288e59ca9900aa3e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-j, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-323d23d, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e) 2024-03-28T22:50:51.515 INFO:journalctl@ceph.mds.j.smithi043.stdout:Mar 28 22:50:51 smithi043 bash[105660]: ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-j 2024-03-28T22:50:51.816 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.j.service' 2024-03-28T22:50:51.882 INFO:journalctl@ceph.mds.j.smithi043.stdout:Mar 28 22:50:51 smithi043 podman[105678]: 2024-03-28 22:50:51.568553666 +0000 UTC m=+0.674684563 container remove ddb67d04777208c50055825d7934b66ac7ea8d5880fd12f288e59ca9900aa3e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-j, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-323d23d, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-28T22:50:51.883 INFO:journalctl@ceph.mds.j.smithi043.stdout:Mar 28 22:50:51 smithi043 systemd[1]: ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.j.service: Deactivated successfully. 2024-03-28T22:50:51.883 INFO:journalctl@ceph.mds.j.smithi043.stdout:Mar 28 22:50:51 smithi043 systemd[1]: Stopped Ceph mds.j for c2276428-ed51-11ee-95d0-87774f69a715. 2024-03-28T22:50:51.883 INFO:journalctl@ceph.mds.j.smithi043.stdout:Mar 28 22:50:51 smithi043 systemd[1]: ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.j.service: Consumed 5.625s CPU time. 2024-03-28T22:50:51.906 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:50:51.906 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-28T22:50:51.907 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-28T22:50:51.907 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.b 2024-03-28T22:50:52.198 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 28 22:50:51 smithi082 systemd[1]: Stopping Ceph mds.b for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:50:52.455 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 28 22:50:52 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-b[67270]: 2024-03-28T22:50:52.232+0000 7fca1b4e5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:50:52.456 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 28 22:50:52 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-b[67270]: 2024-03-28T22:50:52.232+0000 7fca1b4e5700 -1 mds.b *** got signal Terminated *** 2024-03-28T22:50:55.081 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 28 22:50:54 smithi082 podman[74845]: 2024-03-28 22:50:54.831425242 +0000 UTC m=+2.730883790 container died bfb1218e0d706d28fc0ef96213fc8b14174e8505da1c89ac83e2db5604389e5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-28T22:50:55.448 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 28 22:50:55 smithi082 podman[74845]: 2024-03-28 22:50:55.183662089 +0000 UTC m=+3.083120645 container cleanup bfb1218e0d706d28fc0ef96213fc8b14174e8505da1c89ac83e2db5604389e5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-323d23d, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-28T22:50:55.449 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 28 22:50:55 smithi082 bash[74845]: ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-b 2024-03-28T22:50:55.603 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.b.service' 2024-03-28T22:50:55.642 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:50:55.643 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-28T22:50:55.643 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-28T22:50:55.643 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.e 2024-03-28T22:50:55.959 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 28 22:50:55 smithi082 systemd[1]: Stopping Ceph mds.e for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:50:56.448 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 28 22:50:55 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-e[66639]: 2024-03-28T22:50:55.961+0000 7f06ca05c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:50:56.448 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 28 22:50:55 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-e[66639]: 2024-03-28T22:50:55.961+0000 7f06ca05c700 -1 mds.e *** got signal Terminated *** 2024-03-28T22:50:59.092 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 28 22:50:58 smithi082 podman[74978]: 2024-03-28 22:50:58.841381195 +0000 UTC m=+2.980546846 container died c3a5469d05e7a31f57c934c8bf833e457423ec64ad918012c008f1a3571fe039 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-e, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-323d23d, GIT_BRANCH=HEAD, ceph=True) 2024-03-28T22:50:59.448 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 28 22:50:59 smithi082 podman[74978]: 2024-03-28 22:50:59.194848536 +0000 UTC m=+3.334014185 container cleanup c3a5469d05e7a31f57c934c8bf833e457423ec64ad918012c008f1a3571fe039 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-323d23d, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-03-28T22:50:59.448 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 28 22:50:59 smithi082 bash[74978]: ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-e 2024-03-28T22:50:59.617 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.e.service' 2024-03-28T22:50:59.653 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:50:59.653 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-28T22:50:59.653 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-28T22:50:59.653 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.h 2024-03-28T22:51:00.198 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 28 22:50:59 smithi082 systemd[1]: Stopping Ceph mds.h for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:51:00.198 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 28 22:50:59 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-h[66953]: 2024-03-28T22:50:59.973+0000 7f5ec0885700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:51:00.199 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 28 22:50:59 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-h[66953]: 2024-03-28T22:50:59.973+0000 7f5ec0885700 -1 mds.h *** got signal Terminated *** 2024-03-28T22:51:03.144 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 28 22:51:02 smithi082 podman[75100]: 2024-03-28 22:51:02.833556091 +0000 UTC m=+2.962672921 container died 95924c8f41dd132a5a429bb5a8b27118b38ae6e1e8f63b9ede2df93600584e5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-h, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, RELEASE=squid-323d23d, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux ) 2024-03-28T22:51:03.145 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 28 22:51:03 smithi082 podman[75100]: 2024-03-28 22:51:03.147915547 +0000 UTC m=+3.277032378 container cleanup 95924c8f41dd132a5a429bb5a8b27118b38ae6e1e8f63b9ede2df93600584e5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-h, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-323d23d, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-28T22:51:03.436 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 28 22:51:03 smithi082 bash[75100]: ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-h 2024-03-28T22:51:03.437 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 28 22:51:03 smithi082 podman[75114]: 2024-03-28 22:51:03.357376043 +0000 UTC m=+0.522652804 container remove 95924c8f41dd132a5a429bb5a8b27118b38ae6e1e8f63b9ede2df93600584e5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-h, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-323d23d, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-28T22:51:03.463 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.h.service' 2024-03-28T22:51:03.497 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:51:03.497 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-28T22:51:03.497 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-28T22:51:03.497 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.k 2024-03-28T22:51:03.698 INFO:journalctl@ceph.mds.k.smithi082.stdout:Mar 28 22:51:03 smithi082 systemd[1]: Stopping Ceph mds.k for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:51:04.198 INFO:journalctl@ceph.mds.k.smithi082.stdout:Mar 28 22:51:03 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-k[66321]: 2024-03-28T22:51:03.826+0000 7f8405683700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:51:04.199 INFO:journalctl@ceph.mds.k.smithi082.stdout:Mar 28 22:51:03 smithi082 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-k[66321]: 2024-03-28T22:51:03.826+0000 7f8405683700 -1 mds.k *** got signal Terminated *** 2024-03-28T22:51:07.090 INFO:journalctl@ceph.mds.k.smithi082.stdout:Mar 28 22:51:06 smithi082 podman[75220]: 2024-03-28 22:51:06.819516327 +0000 UTC m=+3.098037507 container died 19590bac1035fa8e713ca00e77117c4639840f67a35cb7ede484211b8885d834 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-323d23d, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240326, io.buildah.version=1.33.5) 2024-03-28T22:51:07.440 INFO:journalctl@ceph.mds.k.smithi082.stdout:Mar 28 22:51:07 smithi082 podman[75220]: 2024-03-28 22:51:07.175904008 +0000 UTC m=+3.454425185 container cleanup 19590bac1035fa8e713ca00e77117c4639840f67a35cb7ede484211b8885d834 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-k, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-323d23d, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-28T22:51:07.440 INFO:journalctl@ceph.mds.k.smithi082.stdout:Mar 28 22:51:07 smithi082 bash[75220]: ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-k 2024-03-28T22:51:07.590 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.k.service' 2024-03-28T22:51:07.624 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:51:07.624 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-28T22:51:07.624 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-28T22:51:07.624 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.c 2024-03-28T22:51:08.001 INFO:journalctl@ceph.mds.c.smithi172.stdout:Mar 28 22:51:07 smithi172 systemd[1]: Stopping Ceph mds.c for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:51:08.271 INFO:journalctl@ceph.mds.c.smithi172.stdout:Mar 28 22:51:08 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-c[66974]: 2024-03-28T22:51:08.004+0000 7f6f8d229700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:51:08.272 INFO:journalctl@ceph.mds.c.smithi172.stdout:Mar 28 22:51:08 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-c[66974]: 2024-03-28T22:51:08.004+0000 7f6f8d229700 -1 mds.c *** got signal Terminated *** 2024-03-28T22:51:11.102 INFO:journalctl@ceph.mds.c.smithi172.stdout:Mar 28 22:51:10 smithi172 podman[74461]: 2024-03-28 22:51:10.827297239 +0000 UTC m=+2.999265410 container died 46207b93f97a5e7f24527d83d878642ccbdf938ceaf64e5f0c8bd01594d48599 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-c, CEPH_POINT_RELEASE=, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-323d23d, GIT_CLEAN=True) 2024-03-28T22:51:11.395 INFO:journalctl@ceph.mds.c.smithi172.stdout:Mar 28 22:51:11 smithi172 podman[74461]: 2024-03-28 22:51:11.204130886 +0000 UTC m=+3.376099049 container cleanup 46207b93f97a5e7f24527d83d878642ccbdf938ceaf64e5f0c8bd01594d48599 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-c, io.buildah.version=1.33.5, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d, GIT_BRANCH=HEAD) 2024-03-28T22:51:11.395 INFO:journalctl@ceph.mds.c.smithi172.stdout:Mar 28 22:51:11 smithi172 bash[74461]: ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-c 2024-03-28T22:51:11.573 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.c.service' 2024-03-28T22:51:11.612 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:51:11.612 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-28T22:51:11.612 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-28T22:51:11.612 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.f 2024-03-28T22:51:12.145 INFO:journalctl@ceph.mds.f.smithi172.stdout:Mar 28 22:51:11 smithi172 systemd[1]: Stopping Ceph mds.f for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:51:12.145 INFO:journalctl@ceph.mds.f.smithi172.stdout:Mar 28 22:51:11 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-f[66664]: 2024-03-28T22:51:11.924+0000 7f13dba84700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:51:12.145 INFO:journalctl@ceph.mds.f.smithi172.stdout:Mar 28 22:51:11 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-f[66664]: 2024-03-28T22:51:11.924+0000 7f13dba84700 -1 mds.f *** got signal Terminated *** 2024-03-28T22:51:15.105 INFO:journalctl@ceph.mds.f.smithi172.stdout:Mar 28 22:51:14 smithi172 podman[74594]: 2024-03-28 22:51:14.833318937 +0000 UTC m=+3.004670507 container died 18384718f7a0e28829c00dd7327a49743c07ae952a94149464f8dd10c949b14e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-f, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-323d23d, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0) 2024-03-28T22:51:15.395 INFO:journalctl@ceph.mds.f.smithi172.stdout:Mar 28 22:51:15 smithi172 podman[74594]: 2024-03-28 22:51:15.198905744 +0000 UTC m=+3.370257279 container cleanup 18384718f7a0e28829c00dd7327a49743c07ae952a94149464f8dd10c949b14e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-f, RELEASE=squid-323d23d, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240326, io.buildah.version=1.33.5) 2024-03-28T22:51:15.395 INFO:journalctl@ceph.mds.f.smithi172.stdout:Mar 28 22:51:15 smithi172 bash[74594]: ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-f 2024-03-28T22:51:15.621 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.f.service' 2024-03-28T22:51:15.656 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:51:15.657 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-28T22:51:15.657 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-28T22:51:15.657 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.i 2024-03-28T22:51:15.983 INFO:journalctl@ceph.mds.i.smithi172.stdout:Mar 28 22:51:15 smithi172 systemd[1]: Stopping Ceph mds.i for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:51:16.395 INFO:journalctl@ceph.mds.i.smithi172.stdout:Mar 28 22:51:15 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-i[67293]: 2024-03-28T22:51:15.985+0000 7fc524628700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:51:16.395 INFO:journalctl@ceph.mds.i.smithi172.stdout:Mar 28 22:51:15 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-i[67293]: 2024-03-28T22:51:15.985+0000 7fc524628700 -1 mds.i *** got signal Terminated *** 2024-03-28T22:51:19.145 INFO:journalctl@ceph.mds.i.smithi172.stdout:Mar 28 22:51:18 smithi172 podman[74717]: 2024-03-28 22:51:18.835486022 +0000 UTC m=+2.960807832 container died 59dc64b08c15d0d80dd2c407141af35c81d48dfb4acdba9bcf3c9c106a3a43e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-i, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-323d23d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True) 2024-03-28T22:51:19.645 INFO:journalctl@ceph.mds.i.smithi172.stdout:Mar 28 22:51:19 smithi172 podman[74717]: 2024-03-28 22:51:19.469197346 +0000 UTC m=+3.594519141 container cleanup 59dc64b08c15d0d80dd2c407141af35c81d48dfb4acdba9bcf3c9c106a3a43e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, RELEASE=squid-323d23d, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240326) 2024-03-28T22:51:19.646 INFO:journalctl@ceph.mds.i.smithi172.stdout:Mar 28 22:51:19 smithi172 bash[74717]: ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-i 2024-03-28T22:51:19.883 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.i.service' 2024-03-28T22:51:19.918 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:51:19.918 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-28T22:51:19.918 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-28T22:51:19.919 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.l 2024-03-28T22:51:20.395 INFO:journalctl@ceph.mds.l.smithi172.stdout:Mar 28 22:51:20 smithi172 systemd[1]: Stopping Ceph mds.l for c2276428-ed51-11ee-95d0-87774f69a715... 2024-03-28T22:51:20.396 INFO:journalctl@ceph.mds.l.smithi172.stdout:Mar 28 22:51:20 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-l[66361]: 2024-03-28T22:51:20.230+0000 7fe7124af700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T22:51:20.396 INFO:journalctl@ceph.mds.l.smithi172.stdout:Mar 28 22:51:20 smithi172 ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-l[66361]: 2024-03-28T22:51:20.230+0000 7fe7124af700 -1 mds.l *** got signal Terminated *** 2024-03-28T22:51:23.103 INFO:journalctl@ceph.mds.l.smithi172.stdout:Mar 28 22:51:22 smithi172 podman[74836]: 2024-03-28 22:51:22.834325693 +0000 UTC m=+2.693435261 container died 83dd91753c3874b27289692f117b8e10063dc02cd0164be15a9e3414310dc742 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-l, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-323d23d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-28T22:51:23.395 INFO:journalctl@ceph.mds.l.smithi172.stdout:Mar 28 22:51:23 smithi172 podman[74836]: 2024-03-28 22:51:23.197487418 +0000 UTC m=+3.056596967 container cleanup 83dd91753c3874b27289692f117b8e10063dc02cd0164be15a9e3414310dc742 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:159b776f7ea908ec11862e28e350001dd95c067c81ec5a746cd26e38222ced15, name=ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-l, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-323d23d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=d7c7095173c3793ef2ed0fc814eebfb88de11a2e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-28T22:51:23.395 INFO:journalctl@ceph.mds.l.smithi172.stdout:Mar 28 22:51:23 smithi172 bash[74836]: ceph-c2276428-ed51-11ee-95d0-87774f69a715-mds-l 2024-03-28T22:51:23.558 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2276428-ed51-11ee-95d0-87774f69a715@mds.l.service' 2024-03-28T22:51:23.592 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T22:51:23.592 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-28T22:51:23.593 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c2276428-ed51-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-28T22:51:23.939 INFO:teuthology.orchestra.run.smithi043.stdout:Deleting cluster with fsid: c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:51:29.189 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c2276428-ed51-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-28T22:51:29.385 INFO:teuthology.orchestra.run.smithi082.stdout:Deleting cluster with fsid: c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:51:31.916 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c2276428-ed51-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-28T22:51:32.106 INFO:teuthology.orchestra.run.smithi172.stdout:Deleting cluster with fsid: c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:51:34.544 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T22:51:34.652 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T22:51:34.693 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T22:51:34.723 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-28T22:51:34.812 DEBUG:teuthology.misc:Transferring archived files from smithi043:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627121/remote/smithi043/crash 2024-03-28T22:51:34.873 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/crash -- . 2024-03-28T22:51:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:tar: /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-28T22:51:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:tar: Error is not recoverable: exiting now 2024-03-28T22:51:35.174 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627121/remote/smithi082/crash 2024-03-28T22:51:35.245 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/crash -- . 2024-03-28T22:51:35.292 INFO:teuthology.orchestra.run.smithi082.stderr:tar: /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-28T22:51:35.292 INFO:teuthology.orchestra.run.smithi082.stderr:tar: Error is not recoverable: exiting now 2024-03-28T22:51:35.295 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627121/remote/smithi172/crash 2024-03-28T22:51:35.299 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/crash -- . 2024-03-28T22:51:35.342 INFO:teuthology.orchestra.run.smithi172.stderr:tar: /var/lib/ceph/c2276428-ed51-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-28T22:51:35.343 INFO:teuthology.orchestra.run.smithi172.stderr:tar: Error is not recoverable: exiting now 2024-03-28T22:51:35.343 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-28T22:51:35.344 DEBUG:teuthology.orchestra.run.smithi043:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-28T22:51:35.455 INFO:tasks.cephadm:Compressing logs... 2024-03-28T22:51:35.455 DEBUG:teuthology.orchestra.run.smithi043:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T22:51:35.459 DEBUG:teuthology.orchestra.run.smithi082:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T22:51:35.462 DEBUG:teuthology.orchestra.run.smithi172:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T22:51:35.492 INFO:teuthology.orchestra.run.smithi172.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-28T22:51:35.492 INFO:teuthology.orchestra.run.smithi172.stderr:: No such file or directory 2024-03-28T22:51:35.492 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.l.log 2024-03-28T22:51:35.493 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-28T22:51:35.493 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.i.log 2024-03-28T22:51:35.493 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.11.log 2024-03-28T22:51:35.493 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mgr.z.log 2024-03-28T22:51:35.493 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.i.log: /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.8.log 2024-03-28T22:51:35.493 INFO:teuthology.orchestra.run.smithi172.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-28T22:51:35.493 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.5.log 2024-03-28T22:51:35.494 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.f.log 2024-03-28T22:51:35.494 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-28T22:51:35.500 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.f.log: 90.1% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-28T22:51:35.500 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph.log 2024-03-28T22:51:35.500 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph.cephadm.log: 83.5% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-28T22:51:35.500 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.2.log 2024-03-28T22:51:35.501 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph.log: 87.1% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-28T22:51:35.501 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.c.log 2024-03-28T22:51:35.503 INFO:teuthology.orchestra.run.smithi082.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose: No such file or directory -- 2024-03-28T22:51:35.503 INFO:teuthology.orchestra.run.smithi082.stderr: /var/log/ceph/cephadm.log 2024-03-28T22:51:35.503 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.10.log 2024-03-28T22:51:35.504 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-28T22:51:35.504 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.b.log 2024-03-28T22:51:35.504 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.4.log 2024-03-28T22:51:35.504 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.1.log 2024-03-28T22:51:35.504 INFO:teuthology.orchestra.run.smithi082.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-28T22:51:35.504 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.k.log 2024-03-28T22:51:35.504 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.h.log 2024-03-28T22:51:35.504 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.7.log 2024-03-28T22:51:35.505 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.e.log 2024-03-28T22:51:35.505 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.7.log: 90.0% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-28T22:51:35.505 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mgr.y.log 2024-03-28T22:51:35.506 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.2.log: 92.0% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mgr.z.log.gz 2024-03-28T22:51:35.506 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-28T22:51:35.508 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-28T22:51:35.514 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mon.c.log 2024-03-28T22:51:35.516 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph.log 2024-03-28T22:51:35.522 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-28T22:51:35.522 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph.log: 83.6% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-28T22:51:35.525 INFO:teuthology.orchestra.run.smithi082.stderr: 91.9% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mgr.y.log.gz 2024-03-28T22:51:35.527 INFO:teuthology.orchestra.run.smithi043.stderr:find: ‘/var/log/rbd-target-api’: No such file or directorygzip 2024-03-28T22:51:35.528 INFO:teuthology.orchestra.run.smithi043.stderr: -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-28T22:51:35.528 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.6.log 2024-03-28T22:51:35.529 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mon.b.log 2024-03-28T22:51:35.529 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mon.a.log 2024-03-28T22:51:35.530 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-28T22:51:35.532 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.g.log 2024-03-28T22:51:35.533 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.j.log 2024-03-28T22:51:35.536 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.3.log 2024-03-28T22:51:35.537 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-volume.log: 87.1% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-28T22:51:35.538 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.j.log: /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.3.log: 93.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-28T22:51:35.538 INFO:teuthology.orchestra.run.smithi043.stderr: 89.9% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-28T22:51:35.539 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mgr.x.log 2024-03-28T22:51:35.540 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.0.log 2024-03-28T22:51:35.541 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-volume.log: /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mon.c.log: 90.9% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-28T22:51:35.541 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mgr.x.log: 94.4% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.g.log.gz 2024-03-28T22:51:35.543 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-28T22:51:35.544 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph.log 2024-03-28T22:51:35.546 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-28T22:51:35.547 INFO:teuthology.orchestra.run.smithi043.stderr: 83.8% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-28T22:51:35.548 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.9.log 2024-03-28T22:51:35.553 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph.log: /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.9.log: /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.d.log 2024-03-28T22:51:35.555 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.a.log 2024-03-28T22:51:35.557 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.d.log: 87.1% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-28T22:51:35.559 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-03-28T22:51:35.565 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mon.b.log: 90.9% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-28T22:51:35.566 INFO:teuthology.orchestra.run.smithi082.stderr: 94.6% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.k.log.gz 2024-03-28T22:51:35.569 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.log: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-03-28T22:51:35.573 INFO:teuthology.orchestra.run.smithi043.stderr: 94.3% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.a.log.gz 2024-03-28T22:51:35.586 INFO:teuthology.orchestra.run.smithi043.stderr: 90.8% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-28T22:51:35.611 INFO:teuthology.orchestra.run.smithi172.stderr: 94.5% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.l.log.gz 2024-03-28T22:51:35.618 INFO:teuthology.orchestra.run.smithi043.stderr: 94.4% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.j.log.gz 2024-03-28T22:51:35.668 INFO:teuthology.orchestra.run.smithi043.stderr: 94.5% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.d.log.gz 2024-03-28T22:51:35.977 INFO:teuthology.orchestra.run.smithi043.stderr: 90.1% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mgr.x.log.gz 2024-03-28T22:51:36.186 INFO:teuthology.orchestra.run.smithi082.stderr: 91.8% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mon.b.log.gz 2024-03-28T22:51:36.402 INFO:teuthology.orchestra.run.smithi172.stderr: 92.2% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mon.c.log.gz 2024-03-28T22:51:36.665 INFO:teuthology.orchestra.run.smithi043.stderr: 90.4% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mon.a.log.gz 2024-03-28T22:51:37.261 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.f.log.gz 2024-03-28T22:51:37.356 INFO:teuthology.orchestra.run.smithi082.stderr: 97.0% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.h.log.gz 2024-03-28T22:51:37.691 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.c.log.gz 2024-03-28T22:51:37.696 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.i.log.gz 2024-03-28T22:51:39.399 INFO:teuthology.orchestra.run.smithi082.stderr: 95.3% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.e.log.gz 2024-03-28T22:51:45.121 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.11.log.gz 2024-03-28T22:51:45.991 INFO:teuthology.orchestra.run.smithi082.stderr: 91.6% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.10.log.gz 2024-03-28T22:51:46.334 INFO:teuthology.orchestra.run.smithi172.stderr: 91.5% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.8.log.gz 2024-03-28T22:51:46.395 INFO:teuthology.orchestra.run.smithi043.stderr: 91.7% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.0.log.gz 2024-03-28T22:51:47.984 INFO:teuthology.orchestra.run.smithi043.stderr: 91.7% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.3.log.gz 2024-03-28T22:51:48.044 INFO:teuthology.orchestra.run.smithi043.stderr: 91.5% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.6.log.gz 2024-03-28T22:51:49.625 INFO:teuthology.orchestra.run.smithi082.stderr: 92.0% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.1.log.gz 2024-03-28T22:51:49.699 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.2.log.gz 2024-03-28T22:51:49.859 INFO:teuthology.orchestra.run.smithi082.stderr: 91.7% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.4.log.gz 2024-03-28T22:51:50.482 INFO:teuthology.orchestra.run.smithi043.stderr: 91.5% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.9.log.gz 2024-03-28T22:51:50.487 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-03-28T22:51:50.487 INFO:teuthology.orchestra.run.smithi043.stderr:real 0m14.995s 2024-03-28T22:51:50.487 INFO:teuthology.orchestra.run.smithi043.stderr:user 0m48.873s 2024-03-28T22:51:50.487 INFO:teuthology.orchestra.run.smithi043.stderr:sys 0m3.688s 2024-03-28T22:51:51.264 INFO:teuthology.orchestra.run.smithi172.stderr: 91.4% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.5.log.gz 2024-03-28T22:51:51.265 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-28T22:51:51.266 INFO:teuthology.orchestra.run.smithi172.stderr:real 0m15.789s 2024-03-28T22:51:51.266 INFO:teuthology.orchestra.run.smithi172.stderr:user 0m55.603s 2024-03-28T22:51:51.266 INFO:teuthology.orchestra.run.smithi172.stderr:sys 0m1.842s 2024-03-28T22:51:52.691 INFO:teuthology.orchestra.run.smithi082.stderr: 91.7% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-osd.7.log.gz 2024-03-28T22:52:56.428 INFO:teuthology.orchestra.run.smithi082.stderr: 93.6% -- replaced with /var/log/ceph/c2276428-ed51-11ee-95d0-87774f69a715/ceph-mds.b.log.gz 2024-03-28T22:52:56.430 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-28T22:52:56.430 INFO:teuthology.orchestra.run.smithi082.stderr:real 1m20.953s 2024-03-28T22:52:56.430 INFO:teuthology.orchestra.run.smithi082.stderr:user 2m18.996s 2024-03-28T22:52:56.431 INFO:teuthology.orchestra.run.smithi082.stderr:sys 0m4.332s 2024-03-28T22:52:56.431 INFO:tasks.cephadm:Archiving logs... 2024-03-28T22:52:56.432 DEBUG:teuthology.misc:Transferring archived files from smithi043:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627121/remote/smithi043/log 2024-03-28T22:52:56.433 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T22:53:04.636 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627121/remote/smithi082/log 2024-03-28T22:53:04.637 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T22:53:26.182 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627121/remote/smithi172/log 2024-03-28T22:53:26.184 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T22:53:34.696 INFO:tasks.cephadm:Removing cluster... 2024-03-28T22:53:34.697 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c2276428-ed51-11ee-95d0-87774f69a715 --force 2024-03-28T22:53:35.033 INFO:teuthology.orchestra.run.smithi043.stdout:Deleting cluster with fsid: c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:53:35.843 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c2276428-ed51-11ee-95d0-87774f69a715 --force 2024-03-28T22:53:36.034 INFO:teuthology.orchestra.run.smithi082.stdout:Deleting cluster with fsid: c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:53:36.665 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c2276428-ed51-11ee-95d0-87774f69a715 --force 2024-03-28T22:53:36.856 INFO:teuthology.orchestra.run.smithi172.stdout:Deleting cluster with fsid: c2276428-ed51-11ee-95d0-87774f69a715 2024-03-28T22:53:37.301 INFO:tasks.cephadm:Removing cephadm ... 2024-03-28T22:53:37.301 DEBUG:teuthology.orchestra.run.smithi043:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-28T22:53:37.364 DEBUG:teuthology.orchestra.run.smithi082:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-28T22:53:37.379 DEBUG:teuthology.orchestra.run.smithi172:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-28T22:53:37.395 INFO:tasks.cephadm:Teardown complete 2024-03-28T22:53:37.395 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-28T22:53:37.408 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-28T22:53:37.408 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T22:53:37.415 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T22:53:37.422 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T22:53:37.589 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-28T22:53:37.600 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-28T22:53:37.601 DEBUG:teuthology.orchestra.run.smithi043:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T22:53:37.610 DEBUG:teuthology.orchestra.run.smithi082:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T22:53:37.612 DEBUG:teuthology.orchestra.run.smithi172:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T22:53:37.624 INFO:teuthology.orchestra.run.smithi082.stderr:bash: line 1: ntpq: command not found 2024-03-28T22:53:37.626 INFO:teuthology.orchestra.run.smithi172.stderr:bash: line 1: ntpq: command not found 2024-03-28T22:53:37.628 INFO:teuthology.orchestra.run.smithi082.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T22:53:37.628 INFO:teuthology.orchestra.run.smithi082.stdout:=============================================================================== 2024-03-28T22:53:37.628 INFO:teuthology.orchestra.run.smithi082.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 38 -526us[ -492us] +/- 75ms 2024-03-28T22:53:37.628 INFO:teuthology.orchestra.run.smithi082.stdout:^* hv02.front.sepia.ceph.com 3 6 377 35 -558us[ -524us] +/- 53ms 2024-03-28T22:53:37.628 INFO:teuthology.orchestra.run.smithi082.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 38 +1515us[+1548us] +/- 83ms 2024-03-28T22:53:37.628 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T22:53:37.630 INFO:teuthology.orchestra.run.smithi172.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T22:53:37.630 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================== 2024-03-28T22:53:37.630 INFO:teuthology.orchestra.run.smithi172.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 36 -513us[ -513us] +/- 75ms 2024-03-28T22:53:37.630 INFO:teuthology.orchestra.run.smithi172.stdout:^* hv02.front.sepia.ceph.com 3 6 377 38 -546us[ -516us] +/- 53ms 2024-03-28T22:53:37.630 INFO:teuthology.orchestra.run.smithi172.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 38 +1529us[+1529us] +/- 83ms 2024-03-28T22:53:37.630 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T22:53:37.658 INFO:teuthology.orchestra.run.smithi043.stderr:bash: line 1: ntpq: command not found 2024-03-28T22:53:37.668 INFO:teuthology.orchestra.run.smithi043.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T22:53:37.668 INFO:teuthology.orchestra.run.smithi043.stdout:=============================================================================== 2024-03-28T22:53:37.668 INFO:teuthology.orchestra.run.smithi043.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 34 -457us[ -457us] +/- 75ms 2024-03-28T22:53:37.669 INFO:teuthology.orchestra.run.smithi043.stdout:^* hv02.front.sepia.ceph.com 3 7 377 103 -545us[ -537us] +/- 52ms 2024-03-28T22:53:37.669 INFO:teuthology.orchestra.run.smithi043.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 +1515us[+1515us] +/- 83ms 2024-03-28T22:53:37.669 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T22:53:37.670 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-28T22:53:37.681 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-28T22:53:37.681 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-28T22:53:37.691 DEBUG:teuthology.orchestra.run.smithi043:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T22:53:37.949 DEBUG:teuthology.orchestra.run.smithi082:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T22:53:38.023 DEBUG:teuthology.orchestra.run.smithi172:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T22:53:38.096 DEBUG:teuthology.orchestra.run.smithi043:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T22:53:38.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T22:53:38.210 DEBUG:teuthology.orchestra.run.smithi082:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T22:53:38.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T22:53:38.241 DEBUG:teuthology.orchestra.run.smithi172:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T22:53:38.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T22:53:38.272 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-28T22:53:38.283 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-28T22:53:38.294 INFO:teuthology.task.internal:Duration was 2351.418183 seconds 2024-03-28T22:53:38.295 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-28T22:53:38.305 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-28T22:53:38.305 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T22:53:38.311 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T22:53:38.313 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T22:53:38.355 INFO:teuthology.orchestra.run.smithi082.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T22:53:38.356 INFO:teuthology.orchestra.run.smithi172.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T22:53:38.476 INFO:teuthology.orchestra.run.smithi043.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T22:53:38.851 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-28T22:53:38.851 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi043.front.sepia.ceph.com 2024-03-28T22:53:38.852 DEBUG:teuthology.orchestra.run.smithi043:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T22:53:38.920 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi082.front.sepia.ceph.com 2024-03-28T22:53:38.921 DEBUG:teuthology.orchestra.run.smithi082:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T22:53:38.943 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi172.front.sepia.ceph.com 2024-03-28T22:53:38.944 DEBUG:teuthology.orchestra.run.smithi172:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T22:53:38.974 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-28T22:53:38.974 DEBUG:teuthology.orchestra.run.smithi043:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T22:53:38.982 DEBUG:teuthology.orchestra.run.smithi082:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T22:53:38.986 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T22:53:39.070 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-28T22:53:39.070 DEBUG:teuthology.orchestra.run.smithi043:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T22:53:39.075 DEBUG:teuthology.orchestra.run.smithi082:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T22:53:39.077 DEBUG:teuthology.orchestra.run.smithi172:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T22:53:39.638 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-28T22:53:39.650 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-28T22:53:39.650 DEBUG:teuthology.orchestra.run.smithi043:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T22:53:39.752 DEBUG:teuthology.orchestra.run.smithi082:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T22:53:39.780 DEBUG:teuthology.orchestra.run.smithi172:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T22:53:39.807 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-28T22:53:39.820 DEBUG:teuthology.orchestra.run.smithi043:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T22:53:39.829 DEBUG:teuthology.orchestra.run.smithi082:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T22:53:39.831 DEBUG:teuthology.orchestra.run.smithi172:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T22:53:39.855 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = core 2024-03-28T22:53:39.873 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = core 2024-03-28T22:53:39.938 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern = core 2024-03-28T22:53:39.996 DEBUG:teuthology.orchestra.run.smithi043:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T22:53:40.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T22:53:40.051 DEBUG:teuthology.orchestra.run.smithi082:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T22:53:40.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T22:53:40.066 DEBUG:teuthology.orchestra.run.smithi172:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T22:53:40.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T22:53:40.081 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-28T22:53:40.093 INFO:teuthology.task.internal:Transferring archived files... 2024-03-28T22:53:40.093 DEBUG:teuthology.misc:Transferring archived files from smithi043:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627121/remote/smithi043 2024-03-28T22:53:40.094 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T22:53:40.216 DEBUG:teuthology.misc:Transferring archived files from smithi082:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627121/remote/smithi082 2024-03-28T22:53:40.216 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T22:53:40.256 DEBUG:teuthology.misc:Transferring archived files from smithi172:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627121/remote/smithi172 2024-03-28T22:53:40.257 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T22:53:40.297 INFO:teuthology.task.internal:Removing archive directory... 2024-03-28T22:53:40.297 DEBUG:teuthology.orchestra.run.smithi043:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T22:53:40.305 DEBUG:teuthology.orchestra.run.smithi082:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T22:53:40.307 DEBUG:teuthology.orchestra.run.smithi172:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T22:53:40.358 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-28T22:53:40.372 INFO:teuthology.task.internal:Not uploading archives. 2024-03-28T22:53:40.372 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-28T22:53:40.384 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-28T22:53:40.384 DEBUG:teuthology.orchestra.run.smithi043:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T22:53:40.392 DEBUG:teuthology.orchestra.run.smithi082:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T22:53:40.394 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T22:53:40.407 INFO:teuthology.orchestra.run.smithi082.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 22:53 /home/ubuntu/cephtest 2024-03-28T22:53:40.408 INFO:teuthology.orchestra.run.smithi172.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 22:53 /home/ubuntu/cephtest 2024-03-28T22:53:40.445 INFO:teuthology.orchestra.run.smithi043.stdout: 393924 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 22:53 /home/ubuntu/cephtest 2024-03-28T22:53:40.448 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-28T22:53:40.461 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-28T22:53:40.486 INFO:teuthology.nuke:Checking targets against current locks 2024-03-28T22:53:40.515 DEBUG:teuthology.nuke:shortname: smithi043 2024-03-28T22:53:40.515 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T22:53:40.540 DEBUG:teuthology.nuke:shortname: smithi082 2024-03-28T22:53:40.541 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T22:53:40.545 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi043.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627121', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 22:05:11.800096', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBYr1Ssw9eRynMasJZ901KkWuiYsR9v3BvnaY7jH8jUWsowQft6ZQwe+wl3S5s9+68Pq4XNaDYR9725IG+q+Bo8='} 2024-03-28T22:53:40.566 DEBUG:teuthology.nuke:shortname: smithi172 2024-03-28T22:53:40.566 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T22:53:40.570 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi082.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627121', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 22:05:11.803401', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAbC5P4o+dEy/zQHqjuWbzSn6SgvjeI7xm29CErx33cJhzMgXCvGcQOvoOyjUq4GLYIGu2mcsT4ob00QaD8nnpI='} 2024-03-28T22:53:40.593 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-27_21:24:03-fs-squid-distro-default-smithi/7627121', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 22:05:11.802100', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIsh76rEIablDRipA8ysrOA1thB3z3+kbKqxrMns7d3K5WxYO5qyNVd6cbx0W3sOD7LwMS+s3QFlx5yayY4CYGk='} 2024-03-28T22:53:40.603 INFO:teuthology.orchestra.console.smithi043:Power off 2024-03-28T22:53:40.604 DEBUG:teuthology.orchestra.console.smithi043:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T22:53:40.627 DEBUG:teuthology.orchestra.console.smithi043:power off output: Chassis Power Control: Down/Off 2024-03-28T22:53:40.627 DEBUG:teuthology.orchestra.console.smithi043:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T22:53:40.649 INFO:teuthology.orchestra.console.smithi082:Power off 2024-03-28T22:53:40.650 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T22:53:40.659 DEBUG:teuthology.orchestra.console.smithi043:check power output: Chassis Power is on 2024-03-28T22:53:40.672 DEBUG:teuthology.orchestra.console.smithi082:power off output: Chassis Power Control: Down/Off 2024-03-28T22:53:40.672 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T22:53:40.688 INFO:teuthology.orchestra.console.smithi172:Power off 2024-03-28T22:53:40.688 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T22:53:40.698 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on 2024-03-28T22:53:40.712 DEBUG:teuthology.orchestra.console.smithi172:power off output: Chassis Power Control: Down/Off 2024-03-28T22:53:40.712 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T22:53:40.739 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2024-03-28T22:53:44.661 DEBUG:teuthology.orchestra.console.smithi043:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T22:53:44.700 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T22:53:44.740 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T22:53:44.772 DEBUG:teuthology.orchestra.console.smithi043:check power output: Chassis Power is on 2024-03-28T22:53:44.811 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on 2024-03-28T22:53:44.851 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2024-03-28T22:53:48.774 DEBUG:teuthology.orchestra.console.smithi043:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T22:53:48.811 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T22:53:48.852 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T22:53:48.886 DEBUG:teuthology.orchestra.console.smithi043:check power output: Chassis Power is off 2024-03-28T22:53:48.923 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on 2024-03-28T22:53:48.965 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2024-03-28T22:53:48.987 INFO:teuthology.orchestra.console.smithi043:Power off completed 2024-03-28T22:53:52.926 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T22:53:52.965 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T22:53:53.037 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is off 2024-03-28T22:53:53.077 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is off 2024-03-28T22:53:53.139 INFO:teuthology.orchestra.console.smithi082:Power off completed 2024-03-28T22:53:53.179 INFO:teuthology.orchestra.console.smithi172:Power off completed 2024-03-28T22:53:53.286 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} duration: 2351.418183326721 failure_reason: 'Command failed (workunit test suites/dbench.sh) on smithi043 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=55f5138942aa4866e49329c9103468889cec3f73 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c5dff688b9284c948c89be7afb4501b0 status: fail success: false 2024-03-28T22:53:53.287 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T22:53:53.374 INFO:teuthology.run:FAIL